diff --git a/.github/workflows/llvm-bugs.yml b/.github/workflows/llvm-bugs.yml
index bfa0c9af946d9..f592dd6ccd903 100644
--- a/.github/workflows/llvm-bugs.yml
+++ b/.github/workflows/llvm-bugs.yml
@@ -12,6 +12,7 @@ on:
 jobs:
   auto-subscribe:
     runs-on: ubuntu-latest
+    if: github.repository == 'llvm/llvm-project'
     steps:
       - uses: actions/setup-node@v3
         with:
diff --git a/README.md b/README.md
index 1273ba17c2fa0..eb8d624d75cec 100644
--- a/README.md
+++ b/README.md
@@ -1,27 +1,16 @@
 # The LLVM Compiler Infrastructure
 
-This directory and its sub-directories contain the source code for LLVM,
-a toolkit for the construction of highly optimized compilers,
-optimizers, and run-time environments.
-
-The README briefly describes how to get started with building LLVM.
-For more information on how to contribute to the LLVM project, please
-take a look at the
-[Contributing to LLVM](https://llvm.org/docs/Contributing.html) guide.
-
-## Getting Started with the LLVM System
-
-Taken from [here](https://llvm.org/docs/GettingStarted.html).
-
-### Overview
-
 Welcome to the LLVM project!
 
+This repository contains the source code for LLVM, a toolkit for the
+construction of highly optimized compilers, optimizers, and run-time
+environments.
+
 The LLVM project has multiple components. The core of the project is
 itself called "LLVM". This contains all of the tools, libraries, and header
 files needed to process intermediate representations and convert them into
 object files. Tools include an assembler, disassembler, bitcode analyzer, and
-bitcode optimizer. It also contains basic regression tests.
+bitcode optimizer.
 
 C-like languages use the [Clang](http://clang.llvm.org/) frontend. This
 component compiles C, C++, Objective-C, and Objective-C++ code into LLVM bitcode
@@ -31,92 +20,20 @@ Other components include:
 the [libc++ C++ standard library](https://libcxx.llvm.org),
 the [LLD linker](https://lld.llvm.org), and more.
 
-### Getting the Source Code and Building LLVM
-
-The LLVM Getting Started documentation may be out of date. The [Clang
-Getting Started](http://clang.llvm.org/get_started.html) page might have more
-accurate information.
-
-This is an example work-flow and configuration to get and build the LLVM source:
-
-1. Checkout LLVM (including related sub-projects like Clang):
-
-     * ``git clone https://github.com/llvm/llvm-project.git``
-
-     * Or, on windows, ``git clone --config core.autocrlf=false
-    https://github.com/llvm/llvm-project.git``
-
-2. Configure and build LLVM and Clang:
-
-     * ``cd llvm-project``
-
-     * ``cmake -S llvm -B build -G <generator> [options]``
-
-        Some common build system generators are:
-
-        * ``Ninja`` --- for generating [Ninja](https://ninja-build.org)
-          build files. Most llvm developers use Ninja.
-        * ``Unix Makefiles`` --- for generating make-compatible parallel makefiles.
-        * ``Visual Studio`` --- for generating Visual Studio projects and
-          solutions.
-        * ``Xcode`` --- for generating Xcode projects.
-
-        Some common options:
-
-        * ``-DLLVM_ENABLE_PROJECTS='...'`` and ``-DLLVM_ENABLE_RUNTIMES='...'`` ---
-          semicolon-separated list of the LLVM sub-projects and runtimes you'd like to
-          additionally build. ``LLVM_ENABLE_PROJECTS`` can include any of: clang,
-          clang-tools-extra, cross-project-tests, flang, libc, libclc, lld, lldb,
-          mlir, openmp, polly, or pstl. ``LLVM_ENABLE_RUNTIMES`` can include any of
-          libcxx, libcxxabi, libunwind, compiler-rt, libc or openmp. Some runtime
-          projects can be specified either in ``LLVM_ENABLE_PROJECTS`` or in
-          ``LLVM_ENABLE_RUNTIMES``.
-
-          For example, to build LLVM, Clang, libcxx, and libcxxabi, use
-          ``-DLLVM_ENABLE_PROJECTS="clang" -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi"``.
-
-        * ``-DCMAKE_INSTALL_PREFIX=directory`` --- Specify for *directory* the full
-          path name of where you want the LLVM tools and libraries to be installed
-          (default ``/usr/local``). Be careful if you install runtime libraries: if
-          your system uses those provided by LLVM (like libc++ or libc++abi), you
-          must not overwrite your system's copy of those libraries, since that
-          could render your system unusable. In general, using something like
-          ``/usr`` is not advised, but ``/usr/local`` is fine.
-
-        * ``-DCMAKE_BUILD_TYPE=type`` --- Valid options for *type* are Debug,
-          Release, RelWithDebInfo, and MinSizeRel. Default is Debug.
-
-        * ``-DLLVM_ENABLE_ASSERTIONS=On`` --- Compile with assertion checks enabled
-          (default is Yes for Debug builds, No for all other build types).
-
-      * ``cmake --build build [-- [options] <target>]`` or your build system specified above
-        directly.
-
-        * The default target (i.e. ``ninja`` or ``make``) will build all of LLVM.
-
-        * The ``check-all`` target (i.e. ``ninja check-all``) will run the
-          regression tests to ensure everything is in working order.
-
-        * CMake will generate targets for each tool and library, and most
-          LLVM sub-projects generate their own ``check-<project>`` target.
-
-        * Running a serial build will be **slow**. To improve speed, try running a
-          parallel build. That's done by default in Ninja; for ``make``, use the option
-          ``-j NNN``, where ``NNN`` is the number of parallel jobs to run.
-          In most cases, you get the best performance if you specify the number of CPU threads you have.
-          On some Unix systems, you can specify this with ``-j$(nproc)``.
-
-      * For more information see [CMake](https://llvm.org/docs/CMake.html).
+## Getting the Source Code and Building LLVM
 
 Consult the
-[Getting Started with LLVM](https://llvm.org/docs/GettingStarted.html#getting-started-with-llvm)
-page for detailed information on configuring and compiling LLVM. You can visit
-[Directory Layout](https://llvm.org/docs/GettingStarted.html#directory-layout)
-to learn about the layout of the source code tree.
+[Getting Started with LLVM](https://llvm.org/docs/GettingStarted.html#getting-the-source-code-and-building-llvm)
+page for information on building and running LLVM.
+
+For information on how to contribute to the LLVM project, please take a look at
+the [Contributing to LLVM](https://llvm.org/docs/Contributing.html) guide.
 
 ## Getting in touch
 
-Join [LLVM Discourse forums](https://discourse.llvm.org/), [discord chat](https://discord.gg/xS7Z362) or #llvm IRC channel on [OFTC](https://oftc.net/).
+Join the [LLVM Discourse forums](https://discourse.llvm.org/), [Discord
+chat](https://discord.gg/xS7Z362), or #llvm IRC channel on
+[OFTC](https://oftc.net/).
 
 The LLVM project has adopted a [code of conduct](https://llvm.org/docs/CodeOfConduct.html) for
 participants to all modes of communication within the project.
diff --git a/bolt/include/bolt/Core/BinaryBasicBlock.h b/bolt/include/bolt/Core/BinaryBasicBlock.h
index d5f52734bcf36..8c044597b5fd0 100644
--- a/bolt/include/bolt/Core/BinaryBasicBlock.h
+++ b/bolt/include/bolt/Core/BinaryBasicBlock.h
@@ -427,10 +427,6 @@ class BinaryBasicBlock {
   /// Return branch info corresponding to an edge going to \p Succ basic block.
   const BinaryBranchInfo &getBranchInfo(const BinaryBasicBlock &Succ) const;
 
-  /// Return branch info corresponding to an edge going to a basic block with
-  /// label \p Label.
-  BinaryBranchInfo &getBranchInfo(const MCSymbol *Label);
-
   /// Set branch information for the outgoing edge to block \p Succ.
   void setSuccessorBranchInfo(const BinaryBasicBlock &Succ, uint64_t Count,
                               uint64_t MispredictedCount) {
diff --git a/bolt/include/bolt/Core/BinaryFunction.h b/bolt/include/bolt/Core/BinaryFunction.h
index 051a58547007f..7a99ce2481a1a 100644
--- a/bolt/include/bolt/Core/BinaryFunction.h
+++ b/bolt/include/bolt/Core/BinaryFunction.h
@@ -167,6 +167,10 @@ class BinaryFunction {
     /// List of relocations associated with data in the constant island
     std::map<uint64_t, Relocation> Relocations;
 
+    /// Set true if constant island contains dynamic relocations, which may
+    /// happen if binary is linked with -z notext option.
+    bool HasDynamicRelocations{false};
+
     /// Offsets in function that are data values in a constant island identified
     /// after disassembling
     std::map<uint64_t, MCSymbol *> Offsets;
@@ -1229,96 +1233,11 @@ class BinaryFunction {
     return InputOffsetToAddressMap;
   }
 
-  void addRelocationAArch64(uint64_t Offset, MCSymbol *Symbol, uint64_t RelType,
-                            uint64_t Addend, uint64_t Value, bool IsCI) {
-    std::map<uint64_t, Relocation> &Rels =
-        (IsCI) ? Islands->Relocations : Relocations;
-    switch (RelType) {
-    case ELF::R_AARCH64_ABS64:
-    case ELF::R_AARCH64_ABS32:
-    case ELF::R_AARCH64_ABS16:
-    case ELF::R_AARCH64_ADD_ABS_LO12_NC:
-    case ELF::R_AARCH64_ADR_GOT_PAGE:
-    case ELF::R_AARCH64_ADR_PREL_LO21:
-    case ELF::R_AARCH64_ADR_PREL_PG_HI21:
-    case ELF::R_AARCH64_ADR_PREL_PG_HI21_NC:
-    case ELF::R_AARCH64_LD64_GOT_LO12_NC:
-    case ELF::R_AARCH64_LDST8_ABS_LO12_NC:
-    case ELF::R_AARCH64_LDST16_ABS_LO12_NC:
-    case ELF::R_AARCH64_LDST32_ABS_LO12_NC:
-    case ELF::R_AARCH64_LDST64_ABS_LO12_NC:
-    case ELF::R_AARCH64_LDST128_ABS_LO12_NC:
-    case ELF::R_AARCH64_TLSDESC_ADD_LO12:
-    case ELF::R_AARCH64_TLSDESC_ADR_PAGE21:
-    case ELF::R_AARCH64_TLSDESC_ADR_PREL21:
-    case ELF::R_AARCH64_TLSDESC_LD64_LO12:
-    case ELF::R_AARCH64_TLSIE_ADR_GOTTPREL_PAGE21:
-    case ELF::R_AARCH64_TLSIE_LD64_GOTTPREL_LO12_NC:
-    case ELF::R_AARCH64_MOVW_UABS_G0:
-    case ELF::R_AARCH64_MOVW_UABS_G0_NC:
-    case ELF::R_AARCH64_MOVW_UABS_G1:
-    case ELF::R_AARCH64_MOVW_UABS_G1_NC:
-    case ELF::R_AARCH64_MOVW_UABS_G2:
-    case ELF::R_AARCH64_MOVW_UABS_G2_NC:
-    case ELF::R_AARCH64_MOVW_UABS_G3:
-    case ELF::R_AARCH64_PREL16:
-    case ELF::R_AARCH64_PREL32:
-    case ELF::R_AARCH64_PREL64:
-      Rels[Offset] = Relocation{Offset, Symbol, RelType, Addend, Value};
-      return;
-    case ELF::R_AARCH64_CALL26:
-    case ELF::R_AARCH64_JUMP26:
-    case ELF::R_AARCH64_TSTBR14:
-    case ELF::R_AARCH64_CONDBR19:
-    case ELF::R_AARCH64_TLSDESC_CALL:
-    case ELF::R_AARCH64_TLSLE_ADD_TPREL_HI12:
-    case ELF::R_AARCH64_TLSLE_ADD_TPREL_LO12_NC:
-      return;
-    default:
-      llvm_unreachable("Unexpected AArch64 relocation type in code");
-    }
-  }
-
-  void addRelocationX86(uint64_t Offset, MCSymbol *Symbol, uint64_t RelType,
-                        uint64_t Addend, uint64_t Value) {
-    switch (RelType) {
-    case ELF::R_X86_64_8:
-    case ELF::R_X86_64_16:
-    case ELF::R_X86_64_32:
-    case ELF::R_X86_64_32S:
-    case ELF::R_X86_64_64:
-    case ELF::R_X86_64_PC8:
-    case ELF::R_X86_64_PC32:
-    case ELF::R_X86_64_PC64:
-    case ELF::R_X86_64_GOTPCRELX:
-    case ELF::R_X86_64_REX_GOTPCRELX:
-      Relocations[Offset] = Relocation{Offset, Symbol, RelType, Addend, Value};
-      return;
-    case ELF::R_X86_64_PLT32:
-    case ELF::R_X86_64_GOTPCREL:
-    case ELF::R_X86_64_TPOFF32:
-    case ELF::R_X86_64_GOTTPOFF:
-      return;
-    default:
-      llvm_unreachable("Unexpected x86 relocation type in code");
-    }
-  }
-
   /// Register relocation type \p RelType at a given \p Address in the function
   /// against \p Symbol.
   /// Assert if the \p Address is not inside this function.
   void addRelocation(uint64_t Address, MCSymbol *Symbol, uint64_t RelType,
-                     uint64_t Addend, uint64_t Value) {
-    assert(Address >= getAddress() && Address < getAddress() + getMaxSize() &&
-           "address is outside of the function");
-    uint64_t Offset = Address - getAddress();
-    if (BC.isAArch64()) {
-      return addRelocationAArch64(Offset, Symbol, RelType, Addend, Value,
-                                  isInConstantIsland(Address));
-    }
-
-    return addRelocationX86(Offset, Symbol, RelType, Addend, Value);
-  }
+                     uint64_t Addend, uint64_t Value);
 
   /// Return the name of the section this function originated from.
   std::optional<StringRef> getOriginSectionName() const {
@@ -1835,7 +1754,7 @@ class BinaryFunction {
 
   /// Returns if this function is a child of \p Other function.
   bool isChildOf(const BinaryFunction &Other) const {
-    return llvm::is_contained(ParentFragments, &Other);
+    return ParentFragments.contains(&Other);
   }
 
   /// Set the profile data for the number of times the function was called.
@@ -1945,6 +1864,28 @@ class BinaryFunction {
     return Symbol;
   }
 
+  /// Support dynamic relocations in constant islands, which may happen if
+  /// binary is linked with -z notext option.
+  void markIslandDynamicRelocationAtAddress(uint64_t Address) {
+    if (!isInConstantIsland(Address)) {
+      errs() << "BOLT-ERROR: dynamic relocation found for text section at 0x"
+             << Twine::utohexstr(Address) << "\n";
+      exit(1);
+    }
+
+    // Mark island to have dynamic relocation
+    Islands->HasDynamicRelocations = true;
+
+    // Create island access, so we would emit the label and
+    // move binary data during updateOutputValues, making us emit
+    // dynamic relocation with the right offset value.
+    getOrCreateIslandAccess(Address);
+  }
+
+  bool hasDynamicRelocationAtIsland() const {
+    return !!(Islands && Islands->HasDynamicRelocations);
+  }
+
   /// Called by an external function which wishes to emit references to constant
   /// island symbols of this function. We create a proxy for it, so we emit
   /// separate symbols when emitting our constant island on behalf of this other
diff --git a/bolt/include/bolt/Core/BinarySection.h b/bolt/include/bolt/Core/BinarySection.h
index 20b4b5c92576f..d3c3ba485c75b 100644
--- a/bolt/include/bolt/Core/BinarySection.h
+++ b/bolt/include/bolt/Core/BinarySection.h
@@ -275,6 +275,7 @@ class BinarySection {
   bool isTBSS() const { return isBSS() && isTLS(); }
   bool isVirtual() const { return ELFType == ELF::SHT_NOBITS; }
   bool isRela() const { return ELFType == ELF::SHT_RELA; }
+  bool isRelr() const { return ELFType == ELF::SHT_RELR; }
   bool isWritable() const { return (ELFFlags & ELF::SHF_WRITE); }
   bool isAllocatable() const {
     if (isELF()) {
diff --git a/bolt/include/bolt/Core/MCPlusBuilder.h b/bolt/include/bolt/Core/MCPlusBuilder.h
index dfb593e794fc3..e6eb1429d3a18 100644
--- a/bolt/include/bolt/Core/MCPlusBuilder.h
+++ b/bolt/include/bolt/Core/MCPlusBuilder.h
@@ -310,6 +310,7 @@ class MCPlusBuilder {
     MaxAllocatorId++;
     // Build alias map
     initAliases();
+    initSizeMap();
   }
 
   /// Create and return target-specific MC symbolizer for the \p Function.
@@ -414,6 +415,22 @@ class MCPlusBuilder {
     return Info->get(Inst.getOpcode()).isPseudo();
   }
 
+  /// Return true if the relocation type needs to be registered in the function.
+  /// These code relocations are used in disassembly to better understand code.
+  ///
+  /// For ARM, they help us decode instruction operands unambiguously, but
+  /// sometimes we might discard them because we already have the necessary
+  /// information in the instruction itself (e.g. we don't need to record CALL
+  /// relocs in ARM because we can fully decode the target from the call
+  /// operand).
+  ///
+  /// For X86, they might be used in scanExternalRefs when we want to skip
+  /// a function but still patch references inside it.
+  virtual bool shouldRecordCodeRelocation(uint64_t RelType) const {
+    llvm_unreachable("not implemented");
+    return false;
+  }
+
   /// Creates x86 pause instruction.
   virtual void createPause(MCInst &Inst) const {
     llvm_unreachable("not implemented");
@@ -450,10 +467,11 @@ class MCPlusBuilder {
   virtual MCPhysReg getX86R11() const { llvm_unreachable("not implemented"); }
 
   /// Create increment contents of target by 1 for Instrumentation
-  virtual void createInstrIncMemory(InstructionListType &Instrs,
-                                    const MCSymbol *Target, MCContext *Ctx,
-                                    bool IsLeaf) const {
+  virtual InstructionListType createInstrIncMemory(const MCSymbol *Target,
+                                                   MCContext *Ctx,
+                                                   bool IsLeaf) const {
     llvm_unreachable("not implemented");
+    return InstructionListType();
   }
 
   /// Return a register number that is guaranteed to not match with
@@ -496,15 +514,9 @@ class MCPlusBuilder {
     return false;
   }
 
-  virtual bool isPrefix(const MCInst &Inst) const {
-    llvm_unreachable("not implemented");
-    return false;
-  }
+  virtual bool isPrefix(const MCInst &Inst) const { return false; }
 
-  virtual bool isRep(const MCInst &Inst) const {
-    llvm_unreachable("not implemented");
-    return false;
-  }
+  virtual bool isRep(const MCInst &Inst) const { return false; }
 
   virtual bool deleteREPPrefix(MCInst &Inst) const {
     llvm_unreachable("not implemented");
@@ -515,10 +527,7 @@ class MCPlusBuilder {
     return Inst.getOpcode() == TargetOpcode::EH_LABEL;
   }
 
-  virtual bool isPop(const MCInst &Inst) const {
-    llvm_unreachable("not implemented");
-    return false;
-  }
+  virtual bool isPop(const MCInst &Inst) const { return false; }
 
   /// Return true if the instruction is used to terminate an indirect branch.
   virtual bool isTerminateBranch(const MCInst &Inst) const {
@@ -555,10 +564,7 @@ class MCPlusBuilder {
     return false;
   }
 
-  virtual bool isLeave(const MCInst &Inst) const {
-    llvm_unreachable("not implemented");
-    return false;
-  }
+  virtual bool isLeave(const MCInst &Inst) const { return false; }
 
   virtual bool isADRP(const MCInst &Inst) const {
     llvm_unreachable("not implemented");
@@ -574,10 +580,7 @@ class MCPlusBuilder {
     llvm_unreachable("not implemented");
   }
 
-  virtual bool isMoveMem2Reg(const MCInst &Inst) const {
-    llvm_unreachable("not implemented");
-    return false;
-  }
+  virtual bool isMoveMem2Reg(const MCInst &Inst) const { return false; }
 
   virtual bool isLoad(const MCInst &Inst) const {
     llvm_unreachable("not implemented");
@@ -861,10 +864,7 @@ class MCPlusBuilder {
   }
 
   /// Return true if the instruction is encoded using EVEX (AVX-512).
-  virtual bool hasEVEXEncoding(const MCInst &Inst) const {
-    llvm_unreachable("not implemented");
-    return false;
-  }
+  virtual bool hasEVEXEncoding(const MCInst &Inst) const { return false; }
 
   /// Return true if a pair of instructions represented by \p Insts
   /// could be fused into a single uop.
@@ -1193,7 +1193,10 @@ class MCPlusBuilder {
                                       bool OnlySmaller = false) const;
 
   /// Initialize aliases tables.
-  virtual void initAliases();
+  void initAliases();
+
+  /// Initialize register size table.
+  void initSizeMap();
 
   /// Change \p Regs setting all registers used to pass parameters according
   /// to the host abi. Do nothing if not implemented.
@@ -1236,7 +1239,7 @@ class MCPlusBuilder {
   }
 
   /// Return the register width in bytes (1, 2, 4 or 8)
-  virtual uint8_t getRegSize(MCPhysReg Reg) const;
+  uint8_t getRegSize(MCPhysReg Reg) const { return SizeMap[Reg]; }
 
   /// For aliased registers, return an alias of \p Reg that has the width of
   /// \p Size bytes
@@ -1986,6 +1989,8 @@ class MCPlusBuilder {
   // alias (are sub or superregs of itself, including itself).
   std::vector<BitVector> AliasMap;
   std::vector<BitVector> SmallerAliasMap;
+  // SizeMap caches a mapping of registers to their sizes.
+  std::vector<uint8_t> SizeMap;
 };
 
 MCPlusBuilder *createX86MCPlusBuilder(const MCInstrAnalysis *,
diff --git a/bolt/include/bolt/Core/Relocation.h b/bolt/include/bolt/Core/Relocation.h
index ed56d41beb05b..51dd769387565 100644
--- a/bolt/include/bolt/Core/Relocation.h
+++ b/bolt/include/bolt/Core/Relocation.h
@@ -109,6 +109,9 @@ struct Relocation {
   /// Return code for a ABS 8-byte relocation
   static uint64_t getAbs64();
 
+  /// Return code for a RELATIVE relocation
+  static uint64_t getRelative();
+
   /// Return true if this relocation is PC-relative. Return false otherwise.
   bool isPCRelative() const { return isPCRelative(Type); }
 
diff --git a/bolt/include/bolt/Profile/DataAggregator.h b/bolt/include/bolt/Profile/DataAggregator.h
index 7f96f4fa732ae..45b652c8a3707 100644
--- a/bolt/include/bolt/Profile/DataAggregator.h
+++ b/bolt/include/bolt/Profile/DataAggregator.h
@@ -315,6 +315,12 @@ class DataAggregator : public DataReader {
   /// consume it (peek only).
   bool checkNewLine();
 
+  using PerfProcessErrorCallbackTy = std::function<void(int, StringRef)>;
+  /// Prepare to parse data from a given perf script invocation.
+  /// Returns an invocation exit code.
+  int prepareToParse(StringRef Name, PerfProcessInfo &Process,
+                     PerfProcessErrorCallbackTy Callback);
+
   /// Parse a single LBR entry as output by perf script -Fbrstack
   ErrorOr<LBREntry> parseLBREntry();
 
diff --git a/bolt/include/bolt/Rewrite/RewriteInstance.h b/bolt/include/bolt/Rewrite/RewriteInstance.h
index c8d5f03d085c7..7e4f7c3e2187e 100644
--- a/bolt/include/bolt/Rewrite/RewriteInstance.h
+++ b/bolt/include/bolt/Rewrite/RewriteInstance.h
@@ -138,6 +138,9 @@ class RewriteInstance {
   /// Read relocations from a given section.
   void readDynamicRelocations(const object::SectionRef &Section, bool IsJmpRel);
 
+  /// Read relocations from a given RELR section.
+  void readDynamicRelrRelocations(BinarySection &Section);
+
   /// Print relocation information.
   void printRelocationInfo(const RelocationRef &Rel, StringRef SymbolName,
                            uint64_t SymbolAddress, uint64_t Addend,
@@ -312,6 +315,9 @@ class RewriteInstance {
   /// Patch allocatable relocation sections.
   ELF_FUNCTION(void, patchELFAllocatableRelaSections);
 
+  /// Patch allocatable relr section.
+  ELF_FUNCTION(void, patchELFAllocatableRelrSection);
+
   /// Finalize memory image of section header string table.
   ELF_FUNCTION(void, finalizeSectionStringTable);
 
@@ -474,6 +480,10 @@ class RewriteInstance {
   uint64_t NewTextSegmentOffset{0};
   uint64_t NewTextSegmentSize{0};
 
+  /// New writable segment info.
+  uint64_t NewWritableSegmentAddress{0};
+  uint64_t NewWritableSegmentSize{0};
+
   /// Track next available address for new allocatable sections.
   uint64_t NextAvailableAddress{0};
 
@@ -482,6 +492,11 @@ class RewriteInstance {
   uint64_t DynamicRelocationsSize{0};
   uint64_t DynamicRelativeRelocationsCount{0};
 
+  // Location and size of .relr.dyn relocations.
+  std::optional<uint64_t> DynamicRelrAddress;
+  uint64_t DynamicRelrSize{0};
+  uint64_t DynamicRelrEntrySize{0};
+
   /// PLT relocations are special kind of dynamic relocations stored separately.
   std::optional<uint64_t> PLTRelocationsAddress;
   uint64_t PLTRelocationsSize{0};
@@ -547,18 +562,6 @@ class RewriteInstance {
   ErrorOr<BinarySection &> LSDASection{std::errc::bad_address};
   ErrorOr<BinarySection &> EHFrameSection{std::errc::bad_address};
 
-  /// .got.plt sections.
-  ///
-  /// Contains jump slots (addresses) indirectly referenced by
-  /// instructions in .plt section.
-  ErrorOr<BinarySection &> GOTPLTSection{std::errc::bad_address};
-
-  /// .rela.plt section.
-  ///
-  /// Contains relocations against .got.plt.
-  ErrorOr<BinarySection &> RelaPLTSection{std::errc::bad_address};
-  ErrorOr<BinarySection &> RelaDynSection{std::errc::bad_address};
-
   /// .note.gnu.build-id section.
   ErrorOr<BinarySection &> BuildIDSection{std::errc::bad_address};
 
diff --git a/bolt/lib/Core/BinaryBasicBlock.cpp b/bolt/lib/Core/BinaryBasicBlock.cpp
index 0a582d7096667..b271b86ec6992 100644
--- a/bolt/lib/Core/BinaryBasicBlock.cpp
+++ b/bolt/lib/Core/BinaryBasicBlock.cpp
@@ -593,19 +593,6 @@ BinaryBasicBlock::getBranchInfo(const BinaryBasicBlock &Succ) const {
   return std::get<1>(*Result);
 }
 
-BinaryBasicBlock::BinaryBranchInfo &
-BinaryBasicBlock::getBranchInfo(const MCSymbol *Label) {
-  auto BI = branch_info_begin();
-  for (BinaryBasicBlock *BB : successors()) {
-    if (BB->getLabel() == Label)
-      return *BI;
-    ++BI;
-  }
-
-  llvm_unreachable("Invalid successor");
-  return *BI;
-}
-
 BinaryBasicBlock *BinaryBasicBlock::splitAt(iterator II) {
   assert(II != end() && "expected iterator pointing to instruction");
 
diff --git a/bolt/lib/Core/BinaryContext.cpp b/bolt/lib/Core/BinaryContext.cpp
index 19e356410198c..09f59f7ad6321 100644
--- a/bolt/lib/Core/BinaryContext.cpp
+++ b/bolt/lib/Core/BinaryContext.cpp
@@ -391,8 +391,18 @@ BinaryContext::handleAddressRef(uint64_t Address, BinaryFunction &BF,
       --IslandIter;
 
     if (IslandIter != AddressToConstantIslandMap.end()) {
-      if (MCSymbol *IslandSym =
-              IslandIter->second->getOrCreateProxyIslandAccess(Address, BF)) {
+      // Fall-back to referencing the original constant island in the presence
+      // of dynamic relocs, as we currently do not support cloning them.
+      // Notice: we might fail to link because of this, if the original constant
+      // island we are referring would be emitted too far away.
+      if (IslandIter->second->hasDynamicRelocationAtIsland()) {
+        MCSymbol *IslandSym =
+            IslandIter->second->getOrCreateIslandAccess(Address);
+        if (IslandSym)
+          return std::make_pair(IslandSym, 0);
+      } else if (MCSymbol *IslandSym =
+                     IslandIter->second->getOrCreateProxyIslandAccess(Address,
+                                                                      BF)) {
         BF.createIslandDependency(IslandSym, IslandIter->second);
         return std::make_pair(IslandSym, 0);
       }
diff --git a/bolt/lib/Core/BinaryFunction.cpp b/bolt/lib/Core/BinaryFunction.cpp
index 01e7de53da55b..1cbdd6b2721b8 100644
--- a/bolt/lib/Core/BinaryFunction.cpp
+++ b/bolt/lib/Core/BinaryFunction.cpp
@@ -987,7 +987,7 @@ size_t BinaryFunction::getSizeOfDataInCodeAt(uint64_t Offset) const {
   if (!Islands)
     return 0;
 
-  if (Islands->DataOffsets.find(Offset) == Islands->DataOffsets.end())
+  if (!llvm::is_contained(Islands->DataOffsets, Offset))
     return 0;
 
   auto Iter = Islands->CodeOffsets.upper_bound(Offset);
@@ -2589,8 +2589,7 @@ struct CFISnapshotDiff : public CFISnapshot {
       if (RestoredRegs[Reg])
         return true;
       RestoredRegs[Reg] = true;
-      const int32_t CurRegRule =
-          RegRule.find(Reg) != RegRule.end() ? RegRule[Reg] : UNKNOWN;
+      const int32_t CurRegRule = RegRule.contains(Reg) ? RegRule[Reg] : UNKNOWN;
       if (CurRegRule == UNKNOWN) {
         if (Instr.getOperation() == MCCFIInstruction::OpRestore ||
             Instr.getOperation() == MCCFIInstruction::OpSameValue)
@@ -2731,7 +2730,7 @@ BinaryFunction::unwindCFIState(int32_t FromState, int32_t ToState,
         Reg = *R;
       }
 
-      if (ToCFITable.RegRule.find(Reg) == ToCFITable.RegRule.end()) {
+      if (!ToCFITable.RegRule.contains(Reg)) {
         FrameInstructions.emplace_back(
             MCCFIInstruction::createRestore(nullptr, Reg));
         if (FromCFITable.isRedundant(FrameInstructions.back())) {
@@ -4088,6 +4087,16 @@ void BinaryFunction::updateOutputValues(const MCAsmLayout &Layout) {
       const uint64_t DataOffset =
           Layout.getSymbolOffset(*getFunctionConstantIslandLabel());
       setOutputDataAddress(BaseAddress + DataOffset);
+      for (auto It : Islands->Offsets) {
+        const uint64_t OldOffset = It.first;
+        BinaryData *BD = BC.getBinaryDataAtAddress(getAddress() + OldOffset);
+        if (!BD)
+          continue;
+
+        MCSymbol *Symbol = It.second;
+        const uint64_t NewOffset = Layout.getSymbolOffset(*Symbol);
+        BD->setOutputLocation(*getCodeSection(), NewOffset);
+      }
     }
     if (isSplit()) {
       for (FunctionFragment &FF : getLayout().getSplitFragments()) {
@@ -4507,5 +4516,21 @@ bool BinaryFunction::isAArch64Veneer() const {
   return true;
 }
 
+void BinaryFunction::addRelocation(uint64_t Address, MCSymbol *Symbol,
+                                   uint64_t RelType, uint64_t Addend,
+                                   uint64_t Value) {
+  assert(Address >= getAddress() && Address < getAddress() + getMaxSize() &&
+         "address is outside of the function");
+  uint64_t Offset = Address - getAddress();
+  LLVM_DEBUG(dbgs() << "BOLT-DEBUG: addRelocation in "
+                    << formatv("{0}@{1:x} against {2}\n", this, Offset,
+                               Symbol->getName()));
+  bool IsCI = BC.isAArch64() && isInConstantIsland(Address);
+  std::map<uint64_t, Relocation> &Rels =
+      IsCI ? Islands->Relocations : Relocations;
+  if (BC.MIB->shouldRecordCodeRelocation(RelType))
+    Rels[Offset] = Relocation{Offset, Symbol, RelType, Addend, Value};
+}
+
 } // namespace bolt
 } // namespace llvm
diff --git a/bolt/lib/Core/MCPlusBuilder.cpp b/bolt/lib/Core/MCPlusBuilder.cpp
index a8b2d12d10afc..b2a102d318313 100644
--- a/bolt/lib/Core/MCPlusBuilder.cpp
+++ b/bolt/lib/Core/MCPlusBuilder.cpp
@@ -464,17 +464,9 @@ void MCPlusBuilder::initAliases() {
   }
 
   // Propagate smaller alias info upwards. Skip reg 0 (mapped to NoRegister)
-  std::queue<MCPhysReg> Worklist;
   for (MCPhysReg I = 1, E = RegInfo->getNumRegs(); I < E; ++I)
-    Worklist.push(I);
-  while (!Worklist.empty()) {
-    MCPhysReg I = Worklist.front();
-    Worklist.pop();
     for (MCSubRegIterator SI(I, RegInfo); SI.isValid(); ++SI)
       SmallerAliasMap[I] |= SmallerAliasMap[*SI];
-    for (MCSuperRegIterator SI(I, RegInfo); SI.isValid(); ++SI)
-      Worklist.push(*SI);
-  }
 
   LLVM_DEBUG({
     dbgs() << "Dumping reg alias table:\n";
@@ -491,22 +483,12 @@ void MCPlusBuilder::initAliases() {
   });
 }
 
-uint8_t MCPlusBuilder::getRegSize(MCPhysReg Reg) const {
-  // SizeMap caches a mapping of registers to their sizes
-  static std::vector<uint8_t> SizeMap;
-
-  if (SizeMap.size() > 0) {
-    return SizeMap[Reg];
-  }
-  SizeMap = std::vector<uint8_t>(RegInfo->getNumRegs());
+void MCPlusBuilder::initSizeMap() {
+  SizeMap.resize(RegInfo->getNumRegs());
   // Build size map
-  for (auto I = RegInfo->regclass_begin(), E = RegInfo->regclass_end(); I != E;
-       ++I) {
-    for (MCPhysReg Reg : *I)
-      SizeMap[Reg] = I->getSizeInBits() / 8;
-  }
-
-  return SizeMap[Reg];
+  for (auto RC : RegInfo->regclasses())
+    for (MCPhysReg Reg : RC)
+      SizeMap[Reg] = RC.getSizeInBits() / 8;
 }
 
 bool MCPlusBuilder::setOperandToSymbolRef(MCInst &Inst, int OpNum,
diff --git a/bolt/lib/Core/Relocation.cpp b/bolt/lib/Core/Relocation.cpp
index eceecf3ef03c4..b36e321f4e342 100644
--- a/bolt/lib/Core/Relocation.cpp
+++ b/bolt/lib/Core/Relocation.cpp
@@ -642,6 +642,12 @@ uint64_t Relocation::getAbs64() {
   return ELF::R_X86_64_64;
 }
 
+uint64_t Relocation::getRelative() {
+  if (Arch == Triple::aarch64)
+    return ELF::R_AARCH64_RELATIVE;
+  return ELF::R_X86_64_RELATIVE;
+}
+
 size_t Relocation::emit(MCStreamer *Streamer) const {
   const size_t Size = getSizeForType(Type);
   MCContext &Ctx = Streamer->getContext();
diff --git a/bolt/lib/Passes/FrameAnalysis.cpp b/bolt/lib/Passes/FrameAnalysis.cpp
index 9aa5dee4c06a5..1e6be498a4790 100644
--- a/bolt/lib/Passes/FrameAnalysis.cpp
+++ b/bolt/lib/Passes/FrameAnalysis.cpp
@@ -352,7 +352,7 @@ bool FrameAnalysis::updateArgsTouchedFor(const BinaryFunction &BF, MCInst &Inst,
     // offset specially after an epilogue, where tailcalls happen. It should be
     // -8.
     for (std::pair<int64_t, uint8_t> Elem : Iter->second) {
-      if (ArgsTouchedMap[&BF].find(Elem) == ArgsTouchedMap[&BF].end()) {
+      if (!llvm::is_contained(ArgsTouchedMap[&BF], Elem)) {
         ArgsTouchedMap[&BF].emplace(Elem);
         Changed = true;
       }
diff --git a/bolt/lib/Passes/IndirectCallPromotion.cpp b/bolt/lib/Passes/IndirectCallPromotion.cpp
index 8b20fbd368c07..ea8019431cf52 100644
--- a/bolt/lib/Passes/IndirectCallPromotion.cpp
+++ b/bolt/lib/Passes/IndirectCallPromotion.cpp
@@ -257,14 +257,14 @@ IndirectCallPromotion::getCallTargets(BinaryBasicBlock &BB,
            JT->EntrySize == BC.AsmInfo->getCodePointerSize());
     for (size_t I = Range.first; I < Range.second; ++I, JI += JIAdj) {
       MCSymbol *Entry = JT->Entries[I];
-      assert(BF.getBasicBlockForLabel(Entry) ||
-             Entry == BF.getFunctionEndLabel() ||
+      const BinaryBasicBlock *ToBB = BF.getBasicBlockForLabel(Entry);
+      assert(ToBB || Entry == BF.getFunctionEndLabel() ||
              Entry == BF.getFunctionEndLabel(FragmentNum::cold()));
       if (Entry == BF.getFunctionEndLabel() ||
           Entry == BF.getFunctionEndLabel(FragmentNum::cold()))
         continue;
       const Location To(Entry);
-      const BinaryBasicBlock::BinaryBranchInfo &BI = BB.getBranchInfo(Entry);
+      const BinaryBasicBlock::BinaryBranchInfo &BI = BB.getBranchInfo(*ToBB);
       Targets.emplace_back(From, To, BI.MispredictedCount, BI.Count,
                            I - Range.first);
     }
diff --git a/bolt/lib/Passes/Instrumentation.cpp b/bolt/lib/Passes/Instrumentation.cpp
index c6e1bf9e6b16a..ddab5e6414dd4 100644
--- a/bolt/lib/Passes/Instrumentation.cpp
+++ b/bolt/lib/Passes/Instrumentation.cpp
@@ -174,12 +174,9 @@ void Instrumentation::createLeafNodeDescription(FunctionDescription &FuncDesc,
 InstructionListType
 Instrumentation::createInstrumentationSnippet(BinaryContext &BC, bool IsLeaf) {
   auto L = BC.scopeLock();
-  MCSymbol *Label;
-  Label = BC.Ctx->createNamedTempSymbol("InstrEntry");
+  MCSymbol *Label = BC.Ctx->createNamedTempSymbol("InstrEntry");
   Summary->Counters.emplace_back(Label);
-  InstructionListType CounterInstrs;
-  BC.MIB->createInstrIncMemory(CounterInstrs, Label, &*BC.Ctx, IsLeaf);
-  return CounterInstrs;
+  return BC.MIB->createInstrIncMemory(Label, BC.Ctx.get(), IsLeaf);
 }
 
 // Helper instruction sequence insertion function
@@ -231,8 +228,9 @@ bool Instrumentation::instrumentOneTarget(
     BinaryBasicBlock &FromBB, uint32_t From, BinaryFunction &ToFunc,
     BinaryBasicBlock *TargetBB, uint32_t ToOffset, bool IsLeaf, bool IsInvoke,
     FunctionDescription *FuncDesc, uint32_t FromNodeID, uint32_t ToNodeID) {
+  BinaryContext &BC = FromFunction.getBinaryContext();
   {
-    auto L = FromFunction.getBinaryContext().scopeLock();
+    auto L = BC.scopeLock();
     bool Created = true;
     if (!TargetBB)
       Created = createCallDescription(*FuncDesc, FromFunction, From, FromNodeID,
@@ -245,10 +243,8 @@ bool Instrumentation::instrumentOneTarget(
       return false;
   }
 
-  InstructionListType CounterInstrs =
-      createInstrumentationSnippet(FromFunction.getBinaryContext(), IsLeaf);
+  InstructionListType CounterInstrs = createInstrumentationSnippet(BC, IsLeaf);
 
-  BinaryContext &BC = FromFunction.getBinaryContext();
   const MCInst &Inst = *Iter;
   if (BC.MIB->isCall(Inst)) {
     // This code handles both
@@ -341,7 +337,7 @@ void Instrumentation::instrumentFunction(BinaryFunction &Function,
       const BinaryBasicBlock *Pred;
       std::tie(Pred, BB) = Stack.top();
       Stack.pop();
-      if (VisitedSet.find(BB) != VisitedSet.end())
+      if (llvm::is_contained(VisitedSet, BB))
         continue;
 
       VisitedSet.insert(BB);
@@ -409,7 +405,7 @@ void Instrumentation::instrumentFunction(BinaryFunction &Function,
       }
       if (TargetFunc) {
         // Do not instrument edges in the spanning tree
-        if (STOutSet[&BB].find(TargetBB) != STOutSet[&BB].end()) {
+        if (llvm::is_contained(STOutSet[&BB], TargetBB)) {
           auto L = BC.scopeLock();
           createEdgeDescription(*FuncDesc, Function, FromOffset, BBToID[&BB],
                                 Function, ToOffset, BBToID[TargetBB],
@@ -426,7 +422,7 @@ void Instrumentation::instrumentFunction(BinaryFunction &Function,
       if (IsJumpTable) {
         for (BinaryBasicBlock *&Succ : BB.successors()) {
           // Do not instrument edges in the spanning tree
-          if (STOutSet[&BB].find(&*Succ) != STOutSet[&BB].end()) {
+          if (llvm::is_contained(STOutSet[&BB], &*Succ)) {
             auto L = BC.scopeLock();
             createEdgeDescription(*FuncDesc, Function, FromOffset, BBToID[&BB],
                                   Function, Succ->getInputOffset(),
@@ -469,7 +465,7 @@ void Instrumentation::instrumentFunction(BinaryFunction &Function,
       FromOffset = *BC.MIB->getOffset(*LastInstr);
 
       // Do not instrument edges in the spanning tree
-      if (STOutSet[&BB].find(FTBB) != STOutSet[&BB].end()) {
+      if (llvm::is_contained(STOutSet[&BB], FTBB)) {
         auto L = BC.scopeLock();
         createEdgeDescription(*FuncDesc, Function, FromOffset, BBToID[&BB],
                               Function, FTBB->getInputOffset(), BBToID[FTBB],
diff --git a/bolt/lib/Passes/ReorderFunctions.cpp b/bolt/lib/Passes/ReorderFunctions.cpp
index 19ccf9f6fc80e..2fc99f652bf1c 100644
--- a/bolt/lib/Passes/ReorderFunctions.cpp
+++ b/bolt/lib/Passes/ReorderFunctions.cpp
@@ -360,7 +360,7 @@ void ReorderFunctions::runOnFunctions(BinaryContext &BC) {
           }
           // Strip LTO suffixes
           if (std::optional<StringRef> CommonName = getLTOCommonName(Function))
-            if (LTOCommonNameMap.find(*CommonName) != LTOCommonNameMap.end())
+            if (LTOCommonNameMap.contains(*CommonName))
               llvm::append_range(FuncAddrs, LTOCommonNameMap[*CommonName]);
         } else {
           FuncAddrs.push_back(BD->getAddress());
diff --git a/bolt/lib/Profile/DataAggregator.cpp b/bolt/lib/Profile/DataAggregator.cpp
index c0f4ce258079d..7d72c3b0e372f 100644
--- a/bolt/lib/Profile/DataAggregator.cpp
+++ b/bolt/lib/Profile/DataAggregator.cpp
@@ -467,6 +467,45 @@ void DataAggregator::filterBinaryMMapInfo() {
   }
 }
 
+int DataAggregator::prepareToParse(StringRef Name, PerfProcessInfo &Process,
+                                   PerfProcessErrorCallbackTy Callback) {
+  std::string Error;
+  outs() << "PERF2BOLT: waiting for perf " << Name
+         << " collection to finish...\n";
+  sys::ProcessInfo PI = sys::Wait(Process.PI, std::nullopt, &Error);
+
+  if (!Error.empty()) {
+    errs() << "PERF-ERROR: " << PerfPath << ": " << Error << "\n";
+    deleteTempFiles();
+    exit(1);
+  }
+
+  if (PI.ReturnCode != 0) {
+    ErrorOr<std::unique_ptr<MemoryBuffer>> ErrorMB =
+        MemoryBuffer::getFileOrSTDIN(Process.StderrPath.data());
+    StringRef ErrBuf = (*ErrorMB)->getBuffer();
+
+    deleteTempFiles();
+    Callback(PI.ReturnCode, ErrBuf);
+    return PI.ReturnCode;
+  }
+
+  ErrorOr<std::unique_ptr<MemoryBuffer>> MB =
+      MemoryBuffer::getFileOrSTDIN(Process.StdoutPath.data());
+  if (std::error_code EC = MB.getError()) {
+    errs() << "Cannot open " << Process.StdoutPath.data() << ": "
+           << EC.message() << "\n";
+    deleteTempFiles();
+    exit(1);
+  }
+
+  FileBuf = std::move(*MB);
+  ParsingBuf = FileBuf->getBuffer();
+  Col = 0;
+  Line = 1;
+  return PI.ReturnCode;
+}
+
 Error DataAggregator::preprocessProfile(BinaryContext &BC) {
   this->BC = &BC;
 
@@ -483,42 +522,16 @@ Error DataAggregator::preprocessProfile(BinaryContext &BC) {
               "not read one from input binary\n";
   }
 
-  auto prepareToParse = [&](StringRef Name, PerfProcessInfo &Process) {
-    std::string Error;
-    outs() << "PERF2BOLT: waiting for perf " << Name
-           << " collection to finish...\n";
-    sys::ProcessInfo PI = sys::Wait(Process.PI, std::nullopt, &Error);
-
-    if (!Error.empty()) {
-      errs() << "PERF-ERROR: " << PerfPath << ": " << Error << "\n";
-      deleteTempFiles();
-      exit(1);
-    }
-
-    if (PI.ReturnCode != 0) {
-      ErrorOr<std::unique_ptr<MemoryBuffer>> ErrorMB =
-          MemoryBuffer::getFileOrSTDIN(Process.StderrPath.data());
-      StringRef ErrBuf = (*ErrorMB)->getBuffer();
-
-      errs() << "PERF-ERROR: return code " << PI.ReturnCode << "\n";
-      errs() << ErrBuf;
-      deleteTempFiles();
-      exit(1);
-    }
-
-    ErrorOr<std::unique_ptr<MemoryBuffer>> MB =
-        MemoryBuffer::getFileOrSTDIN(Process.StdoutPath.data());
-    if (std::error_code EC = MB.getError()) {
-      errs() << "Cannot open " << Process.StdoutPath.data() << ": "
-             << EC.message() << "\n";
-      deleteTempFiles();
-      exit(1);
-    }
+  auto ErrorCallback = [](int ReturnCode, StringRef ErrBuf) {
+    errs() << "PERF-ERROR: return code " << ReturnCode << "\n" << ErrBuf;
+    exit(1);
+  };
 
-    FileBuf = std::move(*MB);
-    ParsingBuf = FileBuf->getBuffer();
-    Col = 0;
-    Line = 1;
+  auto MemEventsErrorCallback = [&](int ReturnCode, StringRef ErrBuf) {
+    Regex NoData("Samples for '.*' event do not have ADDR attribute set. "
+                 "Cannot print 'addr' field.");
+    if (!NoData.match(ErrBuf))
+      ErrorCallback(ReturnCode, ErrBuf);
   };
 
   if (opts::LinuxKernelMode) {
@@ -537,17 +550,17 @@ Error DataAggregator::preprocessProfile(BinaryContext &BC) {
     // in Linux kernel mode.
     opts::IgnoreInterruptLBR = false;
   } else {
-    prepareToParse("mmap events", MMapEventsPPI);
+    prepareToParse("mmap events", MMapEventsPPI, ErrorCallback);
     if (parseMMapEvents())
       errs() << "PERF2BOLT: failed to parse mmap events\n";
   }
 
-  prepareToParse("task events", TaskEventsPPI);
+  prepareToParse("task events", TaskEventsPPI, ErrorCallback);
   if (parseTaskEvents())
     errs() << "PERF2BOLT: failed to parse task events\n";
 
   filterBinaryMMapInfo();
-  prepareToParse("events", MainEventsPPI);
+  prepareToParse("events", MainEventsPPI, ErrorCallback);
 
   if (opts::HeatmapMode) {
     if (std::error_code EC = printLBRHeatMap()) {
@@ -571,38 +584,9 @@ Error DataAggregator::preprocessProfile(BinaryContext &BC) {
   }
 
   // Special handling for memory events
-  std::string Error;
-  sys::ProcessInfo PI = sys::Wait(MemEventsPPI.PI, std::nullopt, &Error);
-  if (PI.ReturnCode != 0) {
-    ErrorOr<std::unique_ptr<MemoryBuffer>> MB =
-        MemoryBuffer::getFileOrSTDIN(MemEventsPPI.StderrPath.data());
-    StringRef ErrBuf = (*MB)->getBuffer();
-
-    deleteTempFiles();
-
-    Regex NoData("Samples for '.*' event do not have ADDR attribute set. "
-                 "Cannot print 'addr' field.");
-    if (!NoData.match(ErrBuf)) {
-      errs() << "PERF-ERROR: return code " << PI.ReturnCode << "\n";
-      errs() << ErrBuf;
-      exit(1);
-    }
+  if (prepareToParse("mem events", MemEventsPPI, MemEventsErrorCallback))
     return Error::success();
-  }
 
-  ErrorOr<std::unique_ptr<MemoryBuffer>> MB =
-      MemoryBuffer::getFileOrSTDIN(MemEventsPPI.StdoutPath.data());
-  if (std::error_code EC = MB.getError()) {
-    errs() << "Cannot open " << MemEventsPPI.StdoutPath.data() << ": "
-           << EC.message() << "\n";
-    deleteTempFiles();
-    exit(1);
-  }
-
-  FileBuf = std::move(*MB);
-  ParsingBuf = FileBuf->getBuffer();
-  Col = 0;
-  Line = 1;
   if (const std::error_code EC = parseMemEvents())
     errs() << "PERF2BOLT: failed to parse memory events: " << EC.message()
            << '\n';
diff --git a/bolt/lib/Profile/YAMLProfileReader.cpp b/bolt/lib/Profile/YAMLProfileReader.cpp
index d2ccadd62a172..13f56ed51c3ac 100644
--- a/bolt/lib/Profile/YAMLProfileReader.cpp
+++ b/bolt/lib/Profile/YAMLProfileReader.cpp
@@ -285,10 +285,10 @@ Error YAMLProfileReader::preprocessProfile(BinaryContext &BC) {
 
 bool YAMLProfileReader::mayHaveProfileData(const BinaryFunction &BF) {
   for (StringRef Name : BF.getNames()) {
-    if (ProfileNameToProfile.find(Name) != ProfileNameToProfile.end())
+    if (ProfileNameToProfile.contains(Name))
       return true;
     if (const std::optional<StringRef> CommonName = getLTOCommonName(Name)) {
-      if (LTOCommonNameMap.find(*CommonName) != LTOCommonNameMap.end())
+      if (LTOCommonNameMap.contains(*CommonName))
         return true;
     }
   }
diff --git a/bolt/lib/Rewrite/RewriteInstance.cpp b/bolt/lib/Rewrite/RewriteInstance.cpp
index a4460dfeef169..04ccbcf20de11 100644
--- a/bolt/lib/Rewrite/RewriteInstance.cpp
+++ b/bolt/lib/Rewrite/RewriteInstance.cpp
@@ -1278,6 +1278,25 @@ void RewriteInstance::discoverFileObjects() {
     llvm_unreachable("Unknown marker");
   }
 
+  if (BC->isAArch64()) {
+    // Check for dynamic relocations that might be contained in
+    // constant islands.
+    for (const BinarySection &Section : BC->allocatableSections()) {
+      const uint64_t SectionAddress = Section.getAddress();
+      for (const Relocation &Rel : Section.dynamicRelocations()) {
+        const uint64_t RelAddress = SectionAddress + Rel.Offset;
+        BinaryFunction *BF =
+            BC->getBinaryFunctionContainingAddress(RelAddress,
+                                                   /*CheckPastEnd*/ false,
+                                                   /*UseMaxSize*/ true);
+        if (BF) {
+          assert(Rel.isRelative() && "Expected relative relocation for island");
+          BF->markIslandDynamicRelocationAtAddress(RelAddress);
+        }
+      }
+    }
+  }
+
   if (opts::LinuxKernelMode) {
     // Read all special linux kernel sections and their relocations
     processLKSections();
@@ -1679,9 +1698,6 @@ Error RewriteInstance::readSpecialSections() {
   HasSymbolTable = (bool)BC->getUniqueSectionByName(".symtab");
   LSDASection = BC->getUniqueSectionByName(".gcc_except_table");
   EHFrameSection = BC->getUniqueSectionByName(".eh_frame");
-  GOTPLTSection = BC->getUniqueSectionByName(".got.plt");
-  RelaPLTSection = BC->getUniqueSectionByName(".rela.plt");
-  RelaDynSection = BC->getUniqueSectionByName(".rela.dyn");
   BuildIDSection = BC->getUniqueSectionByName(".note.gnu.build-id");
   SDTSection = BC->getUniqueSectionByName(".note.stapsdt");
   PseudoProbeDescSection = BC->getUniqueSectionByName(".pseudo_probe_desc");
@@ -2058,6 +2074,19 @@ bool RewriteInstance::analyzeRelocation(
 }
 
 void RewriteInstance::processDynamicRelocations() {
+  // Read .relr.dyn section containing compressed R_*_RELATIVE relocations.
+  if (DynamicRelrSize > 0) {
+    ErrorOr<BinarySection &> DynamicRelrSectionOrErr =
+        BC->getSectionForAddress(*DynamicRelrAddress);
+    if (!DynamicRelrSectionOrErr)
+      report_error("unable to find section corresponding to DT_RELR",
+                   DynamicRelrSectionOrErr.getError());
+    if (DynamicRelrSectionOrErr->getSize() != DynamicRelrSize)
+      report_error("section size mismatch for DT_RELRSZ",
+                   errc::executable_format_error);
+    readDynamicRelrRelocations(*DynamicRelrSectionOrErr);
+  }
+
   // Read relocations for PLT - DT_JMPREL.
   if (PLTRelocationsSize > 0) {
     ErrorOr<BinarySection &> PLTRelSectionOrErr =
@@ -2356,6 +2385,60 @@ void RewriteInstance::readDynamicRelocations(const SectionRef &Section,
   }
 }
 
+void RewriteInstance::readDynamicRelrRelocations(BinarySection &Section) {
+  assert(Section.isAllocatable() && "allocatable expected");
+
+  LLVM_DEBUG({
+    StringRef SectionName = Section.getName();
+    dbgs() << "BOLT-DEBUG: reading relocations in section " << SectionName
+           << ":\n";
+  });
+
+  const uint64_t RType = Relocation::getRelative();
+  const uint8_t PSize = BC->AsmInfo->getCodePointerSize();
+  const uint64_t MaxDelta = ((CHAR_BIT * DynamicRelrEntrySize) - 1) * PSize;
+
+  auto ExtractAddendValue = [&](uint64_t Address) -> uint64_t {
+    ErrorOr<BinarySection &> Section = BC->getSectionForAddress(Address);
+    assert(Section && "cannot get section for data address from RELR");
+    DataExtractor DE = DataExtractor(Section->getContents(),
+                                     BC->AsmInfo->isLittleEndian(), PSize);
+    uint64_t Offset = Address - Section->getAddress();
+    return DE.getUnsigned(&Offset, PSize);
+  };
+
+  auto AddRelocation = [&](uint64_t Address) {
+    uint64_t Addend = ExtractAddendValue(Address);
+    LLVM_DEBUG(dbgs() << "BOLT-DEBUG: R_*_RELATIVE relocation at 0x"
+                      << Twine::utohexstr(Address) << " to 0x"
+                      << Twine::utohexstr(Addend) << '\n';);
+    BC->addDynamicRelocation(Address, nullptr, RType, Addend);
+  };
+
+  DataExtractor DE = DataExtractor(Section.getContents(),
+                                   BC->AsmInfo->isLittleEndian(), PSize);
+  uint64_t Offset = 0, Address = 0;
+  uint64_t RelrCount = DynamicRelrSize / DynamicRelrEntrySize;
+  while (RelrCount--) {
+    assert(DE.isValidOffset(Offset));
+    uint64_t Entry = DE.getUnsigned(&Offset, DynamicRelrEntrySize);
+    if ((Entry & 1) == 0) {
+      AddRelocation(Entry);
+      Address = Entry + PSize;
+    } else {
+      const uint64_t StartAddress = Address;
+      while (Entry >>= 1) {
+        if (Entry & 1)
+          AddRelocation(Address);
+
+        Address += PSize;
+      }
+
+      Address = StartAddress + MaxDelta;
+    }
+  }
+}
+
 void RewriteInstance::printRelocationInfo(const RelocationRef &Rel,
                                           StringRef SymbolName,
                                           uint64_t SymbolAddress,
@@ -3177,6 +3260,12 @@ void RewriteInstance::postProcessFunctions() {
   for (auto &BFI : BC->getBinaryFunctions()) {
     BinaryFunction &Function = BFI.second;
 
+    // Set function as non-simple if it has dynamic relocations
+    // in constant island, we don't want this function to be optimized
+    // e.g. function splitting is unsupported.
+    if (Function.hasDynamicRelocationAtIsland())
+      Function.setSimple(false);
+
     if (Function.empty())
       continue;
 
@@ -4042,6 +4131,13 @@ void RewriteInstance::mapAllocatableSections(RuntimeDyld &RTDyld) {
   // Allocate read-only sections first, then writable sections.
   enum : uint8_t { ST_READONLY, ST_READWRITE };
   for (uint8_t SType = ST_READONLY; SType <= ST_READWRITE; ++SType) {
+    const uint64_t LastNextAvailableAddress = NextAvailableAddress;
+    if (SType == ST_READWRITE) {
+      // Align R+W segment to regular page size
+      NextAvailableAddress = alignTo(NextAvailableAddress, BC->RegularPageSize);
+      NewWritableSegmentAddress = NextAvailableAddress;
+    }
+
     for (BinarySection &Section : BC->allocatableSections()) {
       if (!Section.hasValidSectionID())
         continue;
@@ -4088,6 +4184,21 @@ void RewriteInstance::mapAllocatableSections(RuntimeDyld &RTDyld) {
         NextAvailableAddress += Section.getOutputSize();
       }
     }
+
+    if (SType == ST_READONLY) {
+      if (PHDRTableAddress) {
+        // Segment size includes the size of the PHDR area.
+        NewTextSegmentSize = NextAvailableAddress - PHDRTableAddress;
+      } else {
+        // Existing PHDR table would be updated.
+        NewTextSegmentSize = NextAvailableAddress - NewTextSegmentAddress;
+      }
+    } else if (SType == ST_READWRITE) {
+      NewWritableSegmentSize = NextAvailableAddress - NewWritableSegmentAddress;
+      // Restore NextAvailableAddress if no new writable sections
+      if (!NewWritableSegmentSize)
+        NextAvailableAddress = LastNextAvailableAddress;
+    }
   }
 }
 
@@ -4108,16 +4219,32 @@ void RewriteInstance::patchELFPHDRTable() {
   // Write/re-write program headers.
   Phnum = Obj.getHeader().e_phnum;
   if (PHDRTableOffset) {
-    // Writing new pheader table.
-    Phnum += 1; // only adding one new segment
-    // Segment size includes the size of the PHDR area.
-    NewTextSegmentSize = NextAvailableAddress - PHDRTableAddress;
+    // Writing new pheader table and adding one new entry for R+X segment.
+    Phnum += 1;
+    if (NewWritableSegmentSize) {
+      // Adding one more entry for R+W segment.
+      Phnum += 1;
+    }
   } else {
     assert(!PHDRTableAddress && "unexpected address for program header table");
-    // Update existing table.
     PHDRTableOffset = Obj.getHeader().e_phoff;
-    NewTextSegmentSize = NextAvailableAddress - NewTextSegmentAddress;
+    if (NewWritableSegmentSize) {
+      errs() << "Unable to add writable segment with UseGnuStack option\n";
+      exit(1);
+    }
+  }
+
+  // NOTE Currently .eh_frame_hdr appends to the last segment, recalculate
+  // last segments size based on the NextAvailableAddress variable.
+  if (!NewWritableSegmentSize) {
+    if (PHDRTableAddress)
+      NewTextSegmentSize = NextAvailableAddress - PHDRTableAddress;
+    else
+      NewTextSegmentSize = NextAvailableAddress - NewTextSegmentAddress;
+  } else {
+    NewWritableSegmentSize = NextAvailableAddress - NewWritableSegmentAddress;
   }
+
   OS.seek(PHDRTableOffset);
 
   bool ModdedGnuStack = false;
@@ -4149,6 +4276,19 @@ void RewriteInstance::patchELFPHDRTable() {
     return NewPhdr;
   };
 
+  auto createNewWritableSectionsPhdr = [&]() {
+    ELF64LEPhdrTy NewPhdr;
+    NewPhdr.p_type = ELF::PT_LOAD;
+    NewPhdr.p_offset = getFileOffsetForAddress(NewWritableSegmentAddress);
+    NewPhdr.p_vaddr = NewWritableSegmentAddress;
+    NewPhdr.p_paddr = NewWritableSegmentAddress;
+    NewPhdr.p_filesz = NewWritableSegmentSize;
+    NewPhdr.p_memsz = NewWritableSegmentSize;
+    NewPhdr.p_align = BC->RegularPageSize;
+    NewPhdr.p_flags = ELF::PF_R | ELF::PF_W;
+    return NewPhdr;
+  };
+
   // Copy existing program headers with modifications.
   for (const ELF64LE::Phdr &Phdr : cantFail(Obj.program_headers())) {
     ELF64LE::Phdr NewPhdr = Phdr;
@@ -4177,6 +4317,11 @@ void RewriteInstance::patchELFPHDRTable() {
       ELF64LE::Phdr NewTextPhdr = createNewTextPhdr();
       OS.write(reinterpret_cast<const char *>(&NewTextPhdr),
                sizeof(NewTextPhdr));
+      if (NewWritableSegmentSize) {
+        ELF64LEPhdrTy NewWritablePhdr = createNewWritableSectionsPhdr();
+        OS.write(reinterpret_cast<const char *>(&NewWritablePhdr),
+                 sizeof(NewWritablePhdr));
+      }
       AddedSegment = true;
     }
     OS.write(reinterpret_cast<const char *>(&NewPhdr), sizeof(NewPhdr));
@@ -4186,6 +4331,11 @@ void RewriteInstance::patchELFPHDRTable() {
     // Append the new header to the end of the table.
     ELF64LE::Phdr NewTextPhdr = createNewTextPhdr();
     OS.write(reinterpret_cast<const char *>(&NewTextPhdr), sizeof(NewTextPhdr));
+    if (NewWritableSegmentSize) {
+      ELF64LEPhdrTy NewWritablePhdr = createNewWritableSectionsPhdr();
+      OS.write(reinterpret_cast<const char *>(&NewWritablePhdr),
+               sizeof(NewWritablePhdr));
+    }
   }
 
   assert((!opts::UseGnuStack || ModdedGnuStack) &&
@@ -5120,6 +5270,101 @@ void RewriteInstance::patchELFSymTabs(ELFObjectFile<ELFT> *File) {
                                   ELF::SHT_STRTAB);
 }
 
+template <typename ELFT>
+void RewriteInstance::patchELFAllocatableRelrSection(
+    ELFObjectFile<ELFT> *File) {
+  if (!DynamicRelrAddress)
+    return;
+
+  raw_fd_ostream &OS = Out->os();
+  const uint8_t PSize = BC->AsmInfo->getCodePointerSize();
+  const uint64_t MaxDelta = ((CHAR_BIT * DynamicRelrEntrySize) - 1) * PSize;
+
+  auto FixAddend = [&](const BinarySection &Section, const Relocation &Rel) {
+    // Fix relocation symbol value in place if no static relocation found
+    // on the same address
+    if (Section.getRelocationAt(Rel.Offset))
+      return;
+
+    // No fixup needed if symbol address was not changed
+    const uint64_t Addend = getNewFunctionOrDataAddress(Rel.Addend);
+    if (!Addend)
+      return;
+
+    uint64_t FileOffset = Section.getOutputFileOffset();
+    if (!FileOffset)
+      FileOffset = Section.getInputFileOffset();
+
+    FileOffset += Rel.Offset;
+    OS.pwrite(reinterpret_cast<const char *>(&Addend), PSize, FileOffset);
+  };
+
+  // Fill new relative relocation offsets set
+  std::set<uint64_t> RelOffsets;
+  for (const BinarySection &Section : BC->allocatableSections()) {
+    const uint64_t SectionInputAddress = Section.getAddress();
+    uint64_t SectionAddress = Section.getOutputAddress();
+    if (!SectionAddress)
+      SectionAddress = SectionInputAddress;
+
+    for (const Relocation &Rel : Section.dynamicRelocations()) {
+      if (!Rel.isRelative())
+        continue;
+
+      uint64_t RelOffset =
+          getNewFunctionOrDataAddress(SectionInputAddress + Rel.Offset);
+
+      RelOffset = RelOffset == 0 ? SectionAddress + Rel.Offset : RelOffset;
+      assert((RelOffset & 1) == 0 && "Wrong relocation offset");
+      RelOffsets.emplace(RelOffset);
+      FixAddend(Section, Rel);
+    }
+  }
+
+  ErrorOr<BinarySection &> Section =
+      BC->getSectionForAddress(*DynamicRelrAddress);
+  assert(Section && "cannot get .relr.dyn section");
+  assert(Section->isRelr() && "Expected section to be SHT_RELR type");
+  uint64_t RelrDynOffset = Section->getInputFileOffset();
+  const uint64_t RelrDynEndOffset = RelrDynOffset + Section->getSize();
+
+  auto WriteRelr = [&](uint64_t Value) {
+    if (RelrDynOffset + DynamicRelrEntrySize > RelrDynEndOffset) {
+      errs() << "BOLT-ERROR: Offset overflow for relr.dyn section\n";
+      exit(1);
+    }
+
+    OS.pwrite(reinterpret_cast<const char *>(&Value), DynamicRelrEntrySize,
+              RelrDynOffset);
+    RelrDynOffset += DynamicRelrEntrySize;
+  };
+
+  for (auto RelIt = RelOffsets.begin(); RelIt != RelOffsets.end();) {
+    WriteRelr(*RelIt);
+    uint64_t Base = *RelIt++ + PSize;
+    while (1) {
+      uint64_t Bitmap = 0;
+      for (; RelIt != RelOffsets.end(); ++RelIt) {
+        const uint64_t Delta = *RelIt - Base;
+        if (Delta >= MaxDelta || Delta % PSize)
+          break;
+
+        Bitmap |= (1ULL << (Delta / PSize));
+      }
+
+      if (!Bitmap)
+        break;
+
+      WriteRelr((Bitmap << 1) | 1);
+      Base += MaxDelta;
+    }
+  }
+
+  // Fill the rest of the section with empty bitmap value
+  while (RelrDynOffset != RelrDynEndOffset)
+    WriteRelr(1);
+}
+
 template <typename ELFT>
 void
 RewriteInstance::patchELFAllocatableRelaSections(ELFObjectFile<ELFT> *File) {
@@ -5133,6 +5378,7 @@ RewriteInstance::patchELFAllocatableRelaSections(ELFObjectFile<ELFT> *File) {
   auto setSectionFileOffsets = [&](uint64_t Address, uint64_t &Start,
                                    uint64_t &End) {
     ErrorOr<BinarySection &> Section = BC->getSectionForAddress(Address);
+    assert(Section && "cannot get relocation section");
     Start = Section->getInputFileOffset();
     End = Start + Section->getSize();
   };
@@ -5157,6 +5403,11 @@ RewriteInstance::patchELFAllocatableRelaSections(ELFObjectFile<ELFT> *File) {
 
   auto writeRelocations = [&](bool PatchRelative) {
     for (BinarySection &Section : BC->allocatableSections()) {
+      const uint64_t SectionInputAddress = Section.getAddress();
+      uint64_t SectionAddress = Section.getOutputAddress();
+      if (!SectionAddress)
+        SectionAddress = SectionInputAddress;
+
       for (const Relocation &Rel : Section.dynamicRelocations()) {
         const bool IsRelative = Rel.isRelative();
         if (PatchRelative != IsRelative)
@@ -5166,13 +5417,13 @@ RewriteInstance::patchELFAllocatableRelaSections(ELFObjectFile<ELFT> *File) {
           ++DynamicRelativeRelocationsCount;
 
         Elf_Rela NewRelA;
-        uint64_t SectionAddress = Section.getOutputAddress();
-        SectionAddress =
-            SectionAddress == 0 ? Section.getAddress() : SectionAddress;
         MCSymbol *Symbol = Rel.Symbol;
         uint32_t SymbolIdx = 0;
         uint64_t Addend = Rel.Addend;
+        uint64_t RelOffset =
+            getNewFunctionOrDataAddress(SectionInputAddress + Rel.Offset);
 
+        RelOffset = RelOffset == 0 ? SectionAddress + Rel.Offset : RelOffset;
         if (Rel.Symbol) {
           SymbolIdx = getOutputDynamicSymbolIndex(Symbol);
         } else {
@@ -5183,11 +5434,10 @@ RewriteInstance::patchELFAllocatableRelaSections(ELFObjectFile<ELFT> *File) {
         }
 
         NewRelA.setSymbolAndType(SymbolIdx, Rel.Type, EF.isMips64EL());
-        NewRelA.r_offset = SectionAddress + Rel.Offset;
+        NewRelA.r_offset = RelOffset;
         NewRelA.r_addend = Addend;
 
-        const bool IsJmpRel =
-            !!(IsJmpRelocation.find(Rel.Type) != IsJmpRelocation.end());
+        const bool IsJmpRel = IsJmpRelocation.contains(Rel.Type);
         uint64_t &Offset = IsJmpRel ? RelPltOffset : RelDynOffset;
         const uint64_t &EndOffset =
             IsJmpRel ? RelPltEndOffset : RelDynEndOffset;
@@ -5206,8 +5456,11 @@ RewriteInstance::patchELFAllocatableRelaSections(ELFObjectFile<ELFT> *File) {
     }
   };
 
-  // The dynamic linker expects R_*_RELATIVE relocations to be emitted first
-  writeRelocations(/* PatchRelative */ true);
+  // Place R_*_RELATIVE relocations in RELA section if RELR is not presented.
+  // The dynamic linker expects all R_*_RELATIVE relocations in RELA
+  // to be emitted first.
+  if (!DynamicRelrAddress)
+    writeRelocations(/* PatchRelative */ true);
   writeRelocations(/* PatchRelative */ false);
 
   auto fillNone = [&](uint64_t &Offset, uint64_t EndOffset) {
@@ -5413,6 +5666,15 @@ Error RewriteInstance::readELFDynamic(ELFObjectFile<ELFT> *File) {
     case ELF::DT_RELACOUNT:
       DynamicRelativeRelocationsCount = Dyn.getVal();
       break;
+    case ELF::DT_RELR:
+      DynamicRelrAddress = Dyn.getPtr();
+      break;
+    case ELF::DT_RELRSZ:
+      DynamicRelrSize = Dyn.getVal();
+      break;
+    case ELF::DT_RELRENT:
+      DynamicRelrEntrySize = Dyn.getVal();
+      break;
     }
   }
 
@@ -5425,6 +5687,20 @@ Error RewriteInstance::readELFDynamic(ELFObjectFile<ELFT> *File) {
     PLTRelocationsAddress.reset();
     PLTRelocationsSize = 0;
   }
+
+  if (!DynamicRelrAddress || !DynamicRelrSize) {
+    DynamicRelrAddress.reset();
+    DynamicRelrSize = 0;
+  } else if (!DynamicRelrEntrySize) {
+    errs() << "BOLT-ERROR: expected DT_RELRENT to be presented "
+           << "in DYNAMIC section\n";
+    exit(1);
+  } else if (DynamicRelrSize % DynamicRelrEntrySize) {
+    errs() << "BOLT-ERROR: expected RELR table size to be divisible "
+           << "by RELR entry size\n";
+    exit(1);
+  }
+
   return Error::success();
 }
 
@@ -5636,6 +5912,7 @@ void RewriteInstance::rewriteFile() {
 
   if (BC->HasRelocations) {
     patchELFAllocatableRelaSections();
+    patchELFAllocatableRelrSection();
     patchELFGOT();
   }
 
diff --git a/bolt/lib/Target/AArch64/AArch64MCPlusBuilder.cpp b/bolt/lib/Target/AArch64/AArch64MCPlusBuilder.cpp
index 422f7feebb08e..ec9f78d48981f 100644
--- a/bolt/lib/Target/AArch64/AArch64MCPlusBuilder.cpp
+++ b/bolt/lib/Target/AArch64/AArch64MCPlusBuilder.cpp
@@ -45,8 +45,6 @@ class AArch64MCPlusBuilder : public MCPlusBuilder {
                                  *AArch64ExprB.getSubExpr(), Comp);
   }
 
-  bool hasEVEXEncoding(const MCInst &) const override { return false; }
-
   bool isMacroOpFusionPair(ArrayRef<MCInst> Insts) const override {
     return false;
   }
@@ -1091,13 +1089,51 @@ class AArch64MCPlusBuilder : public MCPlusBuilder {
     return true;
   }
 
-  bool isMoveMem2Reg(const MCInst &Inst) const override { return false; }
-
-  bool isLeave(const MCInst &Inst) const override { return false; }
-
-  bool isPop(const MCInst &Inst) const override { return false; }
-
-  bool isPrefix(const MCInst &Inst) const override { return false; }
+  bool shouldRecordCodeRelocation(uint64_t RelType) const override {
+    switch (RelType) {
+    case ELF::R_AARCH64_ABS64:
+    case ELF::R_AARCH64_ABS32:
+    case ELF::R_AARCH64_ABS16:
+    case ELF::R_AARCH64_ADD_ABS_LO12_NC:
+    case ELF::R_AARCH64_ADR_GOT_PAGE:
+    case ELF::R_AARCH64_ADR_PREL_LO21:
+    case ELF::R_AARCH64_ADR_PREL_PG_HI21:
+    case ELF::R_AARCH64_ADR_PREL_PG_HI21_NC:
+    case ELF::R_AARCH64_LD64_GOT_LO12_NC:
+    case ELF::R_AARCH64_LDST8_ABS_LO12_NC:
+    case ELF::R_AARCH64_LDST16_ABS_LO12_NC:
+    case ELF::R_AARCH64_LDST32_ABS_LO12_NC:
+    case ELF::R_AARCH64_LDST64_ABS_LO12_NC:
+    case ELF::R_AARCH64_LDST128_ABS_LO12_NC:
+    case ELF::R_AARCH64_TLSDESC_ADD_LO12:
+    case ELF::R_AARCH64_TLSDESC_ADR_PAGE21:
+    case ELF::R_AARCH64_TLSDESC_ADR_PREL21:
+    case ELF::R_AARCH64_TLSDESC_LD64_LO12:
+    case ELF::R_AARCH64_TLSIE_ADR_GOTTPREL_PAGE21:
+    case ELF::R_AARCH64_TLSIE_LD64_GOTTPREL_LO12_NC:
+    case ELF::R_AARCH64_MOVW_UABS_G0:
+    case ELF::R_AARCH64_MOVW_UABS_G0_NC:
+    case ELF::R_AARCH64_MOVW_UABS_G1:
+    case ELF::R_AARCH64_MOVW_UABS_G1_NC:
+    case ELF::R_AARCH64_MOVW_UABS_G2:
+    case ELF::R_AARCH64_MOVW_UABS_G2_NC:
+    case ELF::R_AARCH64_MOVW_UABS_G3:
+    case ELF::R_AARCH64_PREL16:
+    case ELF::R_AARCH64_PREL32:
+    case ELF::R_AARCH64_PREL64:
+      return true;
+    case ELF::R_AARCH64_CALL26:
+    case ELF::R_AARCH64_JUMP26:
+    case ELF::R_AARCH64_TSTBR14:
+    case ELF::R_AARCH64_CONDBR19:
+    case ELF::R_AARCH64_TLSDESC_CALL:
+    case ELF::R_AARCH64_TLSLE_ADD_TPREL_HI12:
+    case ELF::R_AARCH64_TLSLE_ADD_TPREL_LO12_NC:
+      return false;
+    default:
+      llvm_unreachable("Unexpected AArch64 relocation type in code");
+    }
+  }
 
   bool createReturn(MCInst &Inst) const override {
     Inst.setOpcode(AArch64::RET);
diff --git a/bolt/lib/Target/X86/X86MCPlusBuilder.cpp b/bolt/lib/Target/X86/X86MCPlusBuilder.cpp
index 61a7353d85a5d..ad80255dcf35e 100644
--- a/bolt/lib/Target/X86/X86MCPlusBuilder.cpp
+++ b/bolt/lib/Target/X86/X86MCPlusBuilder.cpp
@@ -388,6 +388,29 @@ class X86MCPlusBuilder : public MCPlusBuilder {
     return (Desc.TSFlags & X86II::OpPrefixMask) == X86II::PD;
   }
 
+  bool shouldRecordCodeRelocation(uint64_t RelType) const override {
+    switch (RelType) {
+    case ELF::R_X86_64_8:
+    case ELF::R_X86_64_16:
+    case ELF::R_X86_64_32:
+    case ELF::R_X86_64_32S:
+    case ELF::R_X86_64_64:
+    case ELF::R_X86_64_PC8:
+    case ELF::R_X86_64_PC32:
+    case ELF::R_X86_64_PC64:
+    case ELF::R_X86_64_GOTPCRELX:
+    case ELF::R_X86_64_REX_GOTPCRELX:
+      return true;
+    case ELF::R_X86_64_PLT32:
+    case ELF::R_X86_64_GOTPCREL:
+    case ELF::R_X86_64_TPOFF32:
+    case ELF::R_X86_64_GOTTPOFF:
+      return false;
+    default:
+      llvm_unreachable("Unexpected x86 relocation type in code");
+    }
+  }
+
   unsigned getTrapFillValue() const override { return 0xCC; }
 
   struct IndJmpMatcherFrag1 : MCInstMatcher {
@@ -3030,11 +3053,12 @@ class X86MCPlusBuilder : public MCPlusBuilder {
     Inst.clear();
   }
 
-  void createInstrIncMemory(InstructionListType &Instrs, const MCSymbol *Target,
-                            MCContext *Ctx, bool IsLeaf) const override {
+  InstructionListType createInstrIncMemory(const MCSymbol *Target,
+                                           MCContext *Ctx,
+                                           bool IsLeaf) const override {
+    InstructionListType Instrs(IsLeaf ? 13 : 11);
     unsigned int I = 0;
 
-    Instrs.resize(IsLeaf ? 13 : 11);
     // Don't clobber application red zone (ABI dependent)
     if (IsLeaf)
       createStackPointerIncrement(Instrs[I++], 128,
@@ -3061,6 +3085,7 @@ class X86MCPlusBuilder : public MCPlusBuilder {
     if (IsLeaf)
       createStackPointerDecrement(Instrs[I], 128,
                                   /*NoFlagsClobber=*/true);
+    return Instrs;
   }
 
   void createSwap(MCInst &Inst, MCPhysReg Source, MCPhysReg MemBaseReg,
diff --git a/bolt/test/AArch64/constant_island_pie_update.s b/bolt/test/AArch64/constant_island_pie_update.s
index f4989b9c021d5..c6856988d52f7 100644
--- a/bolt/test/AArch64/constant_island_pie_update.s
+++ b/bolt/test/AArch64/constant_island_pie_update.s
@@ -1,17 +1,50 @@
-// This test checks that the constant island value is updated if it
-// has dynamic relocation.
+// This test checks that the constant island offset and value is updated if
+// it has dynamic relocation. The tests checks both .rela.dyn and relr.dyn
+// dynamic relocations.
+// Also check that we don't duplicate CI if it has dynamic relocations.
 
-# RUN: llvm-mc -filetype=obj -triple aarch64-unknown-unknown \
-# RUN:   %s -o %t.o
-# RUN: %clang %cflags -fPIC -pie %t.o -o %t.exe -Wl,-q -nostdlib -Wl,-z,notext
-# RUN: llvm-bolt %t.exe -o %t.bolt --use-old-text=0 --lite=0
-# RUN: llvm-objdump -j .text -dR %t.bolt | FileCheck %s
+# RUN: llvm-mc -filetype=obj -triple aarch64-unknown-unknown %s -o %t.o
+// .rela.dyn
+# RUN: %clang %cflags -fPIC -pie %t.o -o %t.rela.exe -nostdlib \
+# RUN:   -Wl,-q -Wl,-z,notext
+# RUN: llvm-bolt %t.rela.exe -o %t.rela.bolt --use-old-text=0 --lite=0
+# RUN: llvm-objdump -j .text -d %t.rela.bolt | FileCheck %s
+# RUN: llvm-readelf -rsW %t.rela.bolt | FileCheck --check-prefix=ELFCHECK %s
+// .relr.dyn
+# RUN: %clang %cflags -fPIC -pie %t.o -o %t.relr.exe -nostdlib \
+# RUN:   -Wl,-q -Wl,-z,notext -Wl,--pack-dyn-relocs=relr
+# RUN: llvm-bolt %t.relr.exe -o %t.relr.bolt --use-old-text=0 --lite=0
+# RUN: llvm-objdump -j .text -d %t.relr.bolt | FileCheck %s
+# RUN: llvm-readelf -rsW %t.relr.bolt | FileCheck --check-prefix=ELFCHECK %s
+# RUN: llvm-readelf -SW %t.relr.bolt | FileCheck --check-prefix=RELRSZCHECK %s
 
-# CHECK: R_AARCH64_RELATIVE *ABS*+0x[[#%x,ADDR:]]
-# CHECK: [[#ADDR]] <exitLocal>:
+// Check that the CI value was updated
+# CHECK: [[#%x,ADDR:]] <exitLocal>:
 # CHECK: {{.*}} <$d>:
 # CHECK-NEXT: {{.*}} .word 0x{{[0]+}}[[#ADDR]]
 # CHECK-NEXT: {{.*}} .word 0x00000000
+# CHECK-NEXT: {{.*}} .word 0x{{[0]+}}[[#ADDR]]
+# CHECK-NEXT: {{.*}} .word 0x00000000
+# CHECK-NEXT: {{.*}} .word 0x00000000
+# CHECK-NEXT: {{.*}} .word 0x00000000
+# CHECK-NEXT: {{.*}} .word 0x{{[0]+}}[[#ADDR]]
+# CHECK-NEXT: {{.*}} .word 0x00000000
+
+
+// Check that we've relaxed adr to adrp + add to refer external CI
+# CHECK: <addressDynCi>:
+# CHECK-NEXT: adrp
+# CHECK-NEXT: add
+
+// Check that relocation offsets were updated
+# ELFCHECK: [[#%x,OFF:]] [[#%x,INFO_DYN:]] R_AARCH64_RELATIVE
+# ELFCHECK-NEXT: [[#OFF + 8]] {{0*}}[[#INFO_DYN]] R_AARCH64_RELATIVE
+# ELFCHECK-NEXT: [[#OFF + 24]] {{0*}}[[#INFO_DYN]] R_AARCH64_RELATIVE
+# ELFCHECK: {{.*}}[[#OFF]] {{.*}} $d
+
+// Check that .relr.dyn size is 2 bytes to ensure that last 2 relocations were
+// encoded as a bitmap so the total section size for 3 relocations is 2 bytes.
+# RELRSZCHECK: .relr.dyn RELR [[#%x,ADDR:]] [[#%x,OFF:]] {{0*}}10
 
   .text
   .align 4
@@ -20,6 +53,8 @@
 exitLocal:
   add x1, x1, #1
   add x1, x1, #1
+  nop
+  nop
   ret
   .size exitLocal, .-exitLocal
 
@@ -34,5 +69,15 @@ _start:
   bl exitLocal
   nop
 .Lci:
+  .xword exitLocal
+  .xword exitLocal
+  .xword 0
   .xword exitLocal
   .size _start, .-_start
+
+  .global addressDynCi
+  .type addressDynCi, %function
+addressDynCi:
+  adr x1, .Lci
+  bl _start
+.size addressDynCi, .-addressDynCi
diff --git a/bolt/test/Inputs/reorder-data-writable-ptload.fdata b/bolt/test/Inputs/reorder-data-writable-ptload.fdata
new file mode 100644
index 0000000000000..f5dab9d1dacaf
--- /dev/null
+++ b/bolt/test/Inputs/reorder-data-writable-ptload.fdata
@@ -0,0 +1 @@
+4 _start 0 4 hot1 0 100
diff --git a/bolt/test/X86/instrumentation-eh_frame_hdr.cpp b/bolt/test/X86/instrumentation-eh_frame_hdr.cpp
new file mode 100644
index 0000000000000..f6ebd6b76f60a
--- /dev/null
+++ b/bolt/test/X86/instrumentation-eh_frame_hdr.cpp
@@ -0,0 +1,40 @@
+// This test checks that .eh_frame_hdr address is in bounds of the last LOAD
+// end address i.e. the section address is smaller then the LOAD end address.
+
+// REQUIRES: system-linux,bolt-runtime
+
+// RUN: %clangxx %cxxflags -static -Wl,-q %s -o %t.exe -Wl,--entry=_start
+// RUN: llvm-bolt %t.exe -o %t.instr -instrument \
+// RUN:   --instrumentation-file=%t.fdata -instrumentation-sleep-time=1
+// RUN: (llvm-readelf -SW %t.instr | grep -v bolt; llvm-readelf -lW %t.instr | \
+// RUN:   grep LOAD | tail -n 1) | FileCheck %s
+
+// CHECK: {{.*}} .eh_frame_hdr PROGBITS [[#%x, EH_ADDR:]]
+// CHECK: LOAD 0x[[#%x, LD_OFFSET:]] 0x[[#%x, LD_VADDR:]] 0x[[#%x, LD_FSIZE:]]
+// CHECK-SAME: 0x[[#%x, LD_MEMSIZE:]]
+//
+// If .eh_frame_hdr address bigger then last LOAD segment end address test would
+// fail with overflow error, otherwise the result of the expression is 0 that
+// could be found on this line e.g. in LOAD align field.
+// CHECK-SAME: [[#LD_VADDR + LD_MEMSIZE - max(LD_VADDR + LD_MEMSIZE,EH_ADDR)]]
+
+#include <cstdio>
+#include <stdexcept>
+
+void foo() { throw std::runtime_error("Exception from foo()"); }
+
+void bar() { foo(); }
+
+int main() {
+  try {
+    bar();
+  } catch (const std::exception &e) {
+    printf("Exception caught: %s\n", e.what());
+  }
+}
+
+extern "C" {
+void _start();
+}
+
+void _start() { main(); }
diff --git a/bolt/test/reorder-data-writable-ptload.c b/bolt/test/reorder-data-writable-ptload.c
new file mode 100644
index 0000000000000..7b384e9655a32
--- /dev/null
+++ b/bolt/test/reorder-data-writable-ptload.c
@@ -0,0 +1,21 @@
+// This test checks that reorder-data pass puts new hot .data section
+// to the writable segment.
+
+// RUN: %clang %cflags -O3 -nostdlib -Wl,-q %s -o %t.exe
+// RUN: llvm-bolt %t.exe -o %t.bolt --reorder-data=".data" \
+// RUN:   -data %S/Inputs/reorder-data-writable-ptload.fdata
+// RUN: llvm-readelf -SlW %t.bolt | FileCheck %s
+
+// CHECK: .bolt.org.data
+// CHECK: {{.*}} .data PROGBITS [[#%x,ADDR:]] [[#%x,OFF:]]
+// CHECK: LOAD 0x{{.*}}[[#OFF]] 0x{{.*}}[[#ADDR]] {{.*}} RW
+
+volatile int cold1 = 42;
+volatile int hot1 = 42;
+volatile int cold2 = 42;
+volatile int cold3 = 42;
+
+void _start() {
+  hot1++;
+  _start();
+}
diff --git a/bolt/test/runtime/X86/instrumentation-xmm.c b/bolt/test/runtime/X86/instrumentation-xmm.c
index 06002c08b1733..1ddc104bc7226 100644
--- a/bolt/test/runtime/X86/instrumentation-xmm.c
+++ b/bolt/test/runtime/X86/instrumentation-xmm.c
@@ -18,7 +18,8 @@ int main() {
 REQUIRES: system-linux,bolt-runtime
 
 RUN: %clang %cflags %s -o %t.exe -fno-inline -Wl,-q
-RUN: llvm-bolt %t.exe --instrument -o %t.instrumented
+RUN: llvm-bolt %t.exe --instrument -o %t.instrumented \
+RUN:   --instrumentation-file=%t.fdata
 RUN: %t.instrumented | FileCheck %s
 
 CHECK: a = 42.000000
diff --git a/clang-tools-extra/clang-doc/BitcodeWriter.cpp b/clang-tools-extra/clang-doc/BitcodeWriter.cpp
index 8a5647cb6a755..2ece99a3124ab 100644
--- a/clang-tools-extra/clang-doc/BitcodeWriter.cpp
+++ b/clang-tools-extra/clang-doc/BitcodeWriter.cpp
@@ -264,13 +264,13 @@ constexpr unsigned char BitCodeConstants::Signature[];
 void ClangDocBitcodeWriter::AbbreviationMap::add(RecordId RID,
                                                  unsigned AbbrevID) {
   assert(RecordIdNameMap[RID] && "Unknown RecordId.");
-  assert(Abbrevs.find(RID) == Abbrevs.end() && "Abbreviation already added.");
+  assert(!Abbrevs.contains(RID) && "Abbreviation already added.");
   Abbrevs[RID] = AbbrevID;
 }
 
 unsigned ClangDocBitcodeWriter::AbbreviationMap::get(RecordId RID) const {
   assert(RecordIdNameMap[RID] && "Unknown RecordId.");
-  assert(Abbrevs.find(RID) != Abbrevs.end() && "Unknown abbreviation.");
+  assert(Abbrevs.contains(RID) && "Unknown abbreviation.");
   return Abbrevs.lookup(RID);
 }
 
diff --git a/clang-tools-extra/clang-doc/HTMLGenerator.cpp b/clang-tools-extra/clang-doc/HTMLGenerator.cpp
index b61780ca4d971..d2466857e5110 100644
--- a/clang-tools-extra/clang-doc/HTMLGenerator.cpp
+++ b/clang-tools-extra/clang-doc/HTMLGenerator.cpp
@@ -866,7 +866,7 @@ HTMLGenerator::generateDocs(StringRef RootDir,
     llvm::SmallString<128> Path;
     llvm::sys::path::native(RootDir, Path);
     llvm::sys::path::append(Path, Info->getRelativeFilePath(""));
-    if (CreatedDirs.find(Path) == CreatedDirs.end()) {
+    if (!CreatedDirs.contains(Path)) {
       if (std::error_code Err = llvm::sys::fs::create_directories(Path);
           Err != std::error_code()) {
         return llvm::createStringError(Err, "Failed to create directory '%s'.",
diff --git a/clang-tools-extra/clang-doc/MDGenerator.cpp b/clang-tools-extra/clang-doc/MDGenerator.cpp
index 8261e14c3efb0..2f56fd99b7bdc 100644
--- a/clang-tools-extra/clang-doc/MDGenerator.cpp
+++ b/clang-tools-extra/clang-doc/MDGenerator.cpp
@@ -387,7 +387,7 @@ MDGenerator::generateDocs(StringRef RootDir,
     llvm::SmallString<128> Path;
     llvm::sys::path::native(RootDir, Path);
     llvm::sys::path::append(Path, Info->getRelativeFilePath(""));
-    if (CreatedDirs.find(Path) == CreatedDirs.end()) {
+    if (!CreatedDirs.contains(Path)) {
       if (std::error_code Err = llvm::sys::fs::create_directories(Path);
           Err != std::error_code()) {
         return llvm::createStringError(Err, "Failed to create directory '%s'.",
diff --git a/clang-tools-extra/clang-include-fixer/find-all-symbols/STLPostfixHeaderMap.cpp b/clang-tools-extra/clang-include-fixer/find-all-symbols/STLPostfixHeaderMap.cpp
index 0d0bbd9f0555b..122d2df1f1398 100644
--- a/clang-tools-extra/clang-include-fixer/find-all-symbols/STLPostfixHeaderMap.cpp
+++ b/clang-tools-extra/clang-include-fixer/find-all-symbols/STLPostfixHeaderMap.cpp
@@ -243,6 +243,7 @@ const HeaderMapCollector::RegexHeaderMap *getSTLPostfixHeaderMap() {
       {"mutex$", "<mutex>"},
       {"new$", "<new>"},
       {"numeric$", "<numeric>"},
+      {"optional$", "<optional>"},
       {"ostream$", "<ostream>"},
       {"queue$", "<queue>"},
       {"random$", "<random>"},
diff --git a/clang-tools-extra/clang-tidy/bugprone/EasilySwappableParametersCheck.cpp b/clang-tools-extra/clang-tidy/bugprone/EasilySwappableParametersCheck.cpp
index 0e21453234f2f..fa7285744be89 100644
--- a/clang-tools-extra/clang-tidy/bugprone/EasilySwappableParametersCheck.cpp
+++ b/clang-tools-extra/clang-tidy/bugprone/EasilySwappableParametersCheck.cpp
@@ -1581,7 +1581,7 @@ bool lazyMapOfSetsIntersectionExists(const MapTy &Map, const ElemTy &E1,
     return false;
 
   for (const auto &E1SetElem : E1Iterator->second)
-    if (llvm::is_contained(E2Iterator->second, E1SetElem))
+    if (E2Iterator->second.contains(E1SetElem))
       return true;
 
   return false;
diff --git a/clang-tools-extra/clang-tidy/bugprone/ForwardDeclarationNamespaceCheck.cpp b/clang-tools-extra/clang-tidy/bugprone/ForwardDeclarationNamespaceCheck.cpp
index ae874210afc1a..f578f7ea71c08 100644
--- a/clang-tools-extra/clang-tidy/bugprone/ForwardDeclarationNamespaceCheck.cpp
+++ b/clang-tools-extra/clang-tidy/bugprone/ForwardDeclarationNamespaceCheck.cpp
@@ -147,7 +147,7 @@ void ForwardDeclarationNamespaceCheck::onEndOfTranslationUnit() {
       }
       // Check if a definition in another namespace exists.
       const auto DeclName = CurDecl->getName();
-      if (DeclNameToDefinitions.find(DeclName) == DeclNameToDefinitions.end()) {
+      if (!DeclNameToDefinitions.contains(DeclName)) {
         continue; // No definition in this translation unit, we can skip it.
       }
       // Make a warning for each definition with the same name (in other
diff --git a/clang-tools-extra/clang-tidy/bugprone/TooSmallLoopVariableCheck.cpp b/clang-tools-extra/clang-tidy/bugprone/TooSmallLoopVariableCheck.cpp
index 133fcabd78392..8ba8b893e03a6 100644
--- a/clang-tools-extra/clang-tidy/bugprone/TooSmallLoopVariableCheck.cpp
+++ b/clang-tools-extra/clang-tidy/bugprone/TooSmallLoopVariableCheck.cpp
@@ -34,6 +34,11 @@ struct MagnitudeBits {
   bool operator<(const MagnitudeBits &Other) const noexcept {
     return WidthWithoutSignBit < Other.WidthWithoutSignBit;
   }
+
+  bool operator!=(const MagnitudeBits &Other) const noexcept {
+    return WidthWithoutSignBit != Other.WidthWithoutSignBit ||
+           BitFieldWidth != Other.BitFieldWidth;
+  }
 };
 
 } // namespace
@@ -184,13 +189,19 @@ void TooSmallLoopVariableCheck::check(const MatchFinder::MatchResult &Result) {
   if (LoopVar->getType() != LoopIncrement->getType())
     return;
 
-  const QualType LoopVarType = LoopVar->getType();
-  const QualType UpperBoundType = UpperBound->getType();
-
   ASTContext &Context = *Result.Context;
 
+  const QualType LoopVarType = LoopVar->getType();
   const MagnitudeBits LoopVarMagnitudeBits =
       calcMagnitudeBits(Context, LoopVarType, LoopVar);
+
+  const MagnitudeBits LoopIncrementMagnitudeBits =
+      calcMagnitudeBits(Context, LoopIncrement->getType(), LoopIncrement);
+  // We matched the loop variable incorrectly.
+  if (LoopIncrementMagnitudeBits != LoopVarMagnitudeBits)
+    return;
+
+  const QualType UpperBoundType = UpperBound->getType();
   const MagnitudeBits UpperBoundMagnitudeBits =
       calcUpperBoundMagnitudeBits(Context, UpperBound, UpperBoundType);
 
diff --git a/clang-tools-extra/clang-tidy/cppcoreguidelines/AvoidCapturingLambdaCoroutinesCheck.cpp b/clang-tools-extra/clang-tidy/cppcoreguidelines/AvoidCapturingLambdaCoroutinesCheck.cpp
new file mode 100644
index 0000000000000..3c99831f9d640
--- /dev/null
+++ b/clang-tools-extra/clang-tidy/cppcoreguidelines/AvoidCapturingLambdaCoroutinesCheck.cpp
@@ -0,0 +1,45 @@
+//===--- AvoidCapturingLambdaCoroutinesCheck.cpp - clang-tidy -------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "AvoidCapturingLambdaCoroutinesCheck.h"
+#include "clang/AST/ASTContext.h"
+#include "clang/ASTMatchers/ASTMatchFinder.h"
+
+using namespace clang::ast_matchers;
+
+namespace clang::tidy::cppcoreguidelines {
+
+namespace {
+AST_MATCHER(LambdaExpr, hasCoroutineBody) {
+  const Stmt *Body = Node.getBody();
+  return Body != nullptr && CoroutineBodyStmt::classof(Body);
+}
+
+AST_MATCHER(LambdaExpr, hasCaptures) { return Node.capture_size() != 0U; }
+} // namespace
+
+void AvoidCapturingLambdaCoroutinesCheck::registerMatchers(
+    MatchFinder *Finder) {
+  Finder->addMatcher(
+      lambdaExpr(hasCaptures(), hasCoroutineBody()).bind("lambda"), this);
+}
+
+bool AvoidCapturingLambdaCoroutinesCheck::isLanguageVersionSupported(
+    const LangOptions &LangOpts) const {
+  return LangOpts.CPlusPlus20;
+}
+
+void AvoidCapturingLambdaCoroutinesCheck::check(
+    const MatchFinder::MatchResult &Result) {
+  const auto *MatchedLambda = Result.Nodes.getNodeAs<LambdaExpr>("lambda");
+  diag(MatchedLambda->getExprLoc(),
+       "coroutine lambda may cause use-after-free, avoid captures or ensure "
+       "lambda closure object has guaranteed lifetime");
+}
+
+} // namespace clang::tidy::cppcoreguidelines
diff --git a/clang-tools-extra/clang-tidy/cppcoreguidelines/AvoidCapturingLambdaCoroutinesCheck.h b/clang-tools-extra/clang-tidy/cppcoreguidelines/AvoidCapturingLambdaCoroutinesCheck.h
new file mode 100644
index 0000000000000..b32e2662b5fba
--- /dev/null
+++ b/clang-tools-extra/clang-tidy/cppcoreguidelines/AvoidCapturingLambdaCoroutinesCheck.h
@@ -0,0 +1,33 @@
+//===--- AvoidCapturingLambdaCoroutinesCheck.h - clang-tidy -----*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_TOOLS_EXTRA_CLANG_TIDY_CPPCOREGUIDELINES_AVOIDCAPTURINGLAMBDACOROUTINESCHECK_H
+#define LLVM_CLANG_TOOLS_EXTRA_CLANG_TIDY_CPPCOREGUIDELINES_AVOIDCAPTURINGLAMBDACOROUTINESCHECK_H
+
+#include "../ClangTidyCheck.h"
+
+namespace clang::tidy::cppcoreguidelines {
+
+/// Flags C++20 coroutine lambdas with non-empty capture lists that may cause
+/// use-after-free errors and suggests avoiding captures or ensuring the lambda
+/// closure object has a guaranteed lifetime.
+///
+/// For the user-facing documentation see:
+/// https://clang.llvm.org/extra/clang-tidy/checks/cppcoreguidelines/avoid-capturing-lambda-coroutines.html
+class AvoidCapturingLambdaCoroutinesCheck : public ClangTidyCheck {
+public:
+  AvoidCapturingLambdaCoroutinesCheck(StringRef Name, ClangTidyContext *Context)
+      : ClangTidyCheck(Name, Context) {}
+  void registerMatchers(ast_matchers::MatchFinder *Finder) override;
+  void check(const ast_matchers::MatchFinder::MatchResult &Result) override;
+  bool isLanguageVersionSupported(const LangOptions &LangOpts) const override;
+};
+
+} // namespace clang::tidy::cppcoreguidelines
+
+#endif // LLVM_CLANG_TOOLS_EXTRA_CLANG_TIDY_CPPCOREGUIDELINES_AVOIDCAPTURINGLAMBDACOROUTINESCHECK_H
diff --git a/clang-tools-extra/clang-tidy/cppcoreguidelines/CMakeLists.txt b/clang-tools-extra/clang-tidy/cppcoreguidelines/CMakeLists.txt
index bd7a999a71743..0804675c63949 100644
--- a/clang-tools-extra/clang-tidy/cppcoreguidelines/CMakeLists.txt
+++ b/clang-tools-extra/clang-tidy/cppcoreguidelines/CMakeLists.txt
@@ -5,6 +5,7 @@ set(LLVM_LINK_COMPONENTS
 
 add_clang_library(clangTidyCppCoreGuidelinesModule
   AvoidCaptureDefaultWhenCapturingThisCheck.cpp
+  AvoidCapturingLambdaCoroutinesCheck.cpp
   AvoidConstOrRefDataMembersCheck.cpp
   AvoidDoWhileCheck.cpp
   AvoidGotoCheck.cpp
@@ -28,6 +29,7 @@ add_clang_library(clangTidyCppCoreGuidelinesModule
   ProTypeStaticCastDowncastCheck.cpp
   ProTypeUnionAccessCheck.cpp
   ProTypeVarargCheck.cpp
+  RvalueReferenceParamNotMovedCheck.cpp
   SlicingCheck.cpp
   SpecialMemberFunctionsCheck.cpp
   VirtualClassDestructorCheck.cpp
diff --git a/clang-tools-extra/clang-tidy/cppcoreguidelines/CppCoreGuidelinesTidyModule.cpp b/clang-tools-extra/clang-tidy/cppcoreguidelines/CppCoreGuidelinesTidyModule.cpp
index ac70fc8eaa24c..5bd9648fe8371 100644
--- a/clang-tools-extra/clang-tidy/cppcoreguidelines/CppCoreGuidelinesTidyModule.cpp
+++ b/clang-tools-extra/clang-tidy/cppcoreguidelines/CppCoreGuidelinesTidyModule.cpp
@@ -15,6 +15,7 @@
 #include "../modernize/UseOverrideCheck.h"
 #include "../readability/MagicNumbersCheck.h"
 #include "AvoidCaptureDefaultWhenCapturingThisCheck.h"
+#include "AvoidCapturingLambdaCoroutinesCheck.h"
 #include "AvoidConstOrRefDataMembersCheck.h"
 #include "AvoidDoWhileCheck.h"
 #include "AvoidGotoCheck.h"
@@ -37,6 +38,7 @@
 #include "ProTypeStaticCastDowncastCheck.h"
 #include "ProTypeUnionAccessCheck.h"
 #include "ProTypeVarargCheck.h"
+#include "RvalueReferenceParamNotMovedCheck.h"
 #include "SlicingCheck.h"
 #include "SpecialMemberFunctionsCheck.h"
 #include "VirtualClassDestructorCheck.h"
@@ -50,6 +52,8 @@ class CppCoreGuidelinesModule : public ClangTidyModule {
   void addCheckFactories(ClangTidyCheckFactories &CheckFactories) override {
     CheckFactories.registerCheck<AvoidCaptureDefaultWhenCapturingThisCheck>(
         "cppcoreguidelines-avoid-capture-default-when-capturing-this");
+    CheckFactories.registerCheck<AvoidCapturingLambdaCoroutinesCheck>(
+        "cppcoreguidelines-avoid-capturing-lambda-coroutines");
     CheckFactories.registerCheck<modernize::AvoidCArraysCheck>(
         "cppcoreguidelines-avoid-c-arrays");
     CheckFactories.registerCheck<AvoidConstOrRefDataMembersCheck>(
@@ -101,6 +105,8 @@ class CppCoreGuidelinesModule : public ClangTidyModule {
         "cppcoreguidelines-pro-type-union-access");
     CheckFactories.registerCheck<ProTypeVarargCheck>(
         "cppcoreguidelines-pro-type-vararg");
+    CheckFactories.registerCheck<RvalueReferenceParamNotMovedCheck>(
+        "cppcoreguidelines-rvalue-reference-param-not-moved");
     CheckFactories.registerCheck<SpecialMemberFunctionsCheck>(
         "cppcoreguidelines-special-member-functions");
     CheckFactories.registerCheck<SlicingCheck>("cppcoreguidelines-slicing");
diff --git a/clang-tools-extra/clang-tidy/cppcoreguidelines/RvalueReferenceParamNotMovedCheck.cpp b/clang-tools-extra/clang-tidy/cppcoreguidelines/RvalueReferenceParamNotMovedCheck.cpp
new file mode 100644
index 0000000000000..eb67df05411a0
--- /dev/null
+++ b/clang-tools-extra/clang-tidy/cppcoreguidelines/RvalueReferenceParamNotMovedCheck.cpp
@@ -0,0 +1,128 @@
+//===--- RvalueReferenceParamNotMovedCheck.cpp - clang-tidy ---------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "RvalueReferenceParamNotMovedCheck.h"
+#include "clang/AST/ASTContext.h"
+#include "clang/ASTMatchers/ASTMatchFinder.h"
+
+using namespace clang::ast_matchers;
+
+namespace clang::tidy::cppcoreguidelines {
+
+namespace {
+AST_MATCHER_P(LambdaExpr, valueCapturesVar, DeclarationMatcher, VarMatcher) {
+  return std::find_if(Node.capture_begin(), Node.capture_end(),
+                      [&](const LambdaCapture &Capture) {
+                        return Capture.capturesVariable() &&
+                               VarMatcher.matches(*Capture.getCapturedVar(),
+                                                  Finder, Builder) &&
+                               Capture.getCaptureKind() == LCK_ByCopy;
+                      }) != Node.capture_end();
+}
+AST_MATCHER_P2(Stmt, argumentOf, bool, AllowPartialMove, StatementMatcher,
+               Ref) {
+  if (AllowPartialMove) {
+    return stmt(anyOf(Ref, hasDescendant(Ref))).matches(Node, Finder, Builder);
+  } else {
+    return Ref.matches(Node, Finder, Builder);
+  }
+}
+} // namespace
+
+void RvalueReferenceParamNotMovedCheck::registerMatchers(MatchFinder *Finder) {
+  auto ToParam = hasAnyParameter(parmVarDecl(equalsBoundNode("param")));
+
+  StatementMatcher MoveCallMatcher =
+      callExpr(
+          anyOf(callee(functionDecl(hasName("::std::move"))),
+                callee(unresolvedLookupExpr(hasAnyDeclaration(
+                    namedDecl(hasUnderlyingDecl(hasName("::std::move"))))))),
+          argumentCountIs(1),
+          hasArgument(
+              0, argumentOf(
+                     AllowPartialMove,
+                     declRefExpr(to(equalsBoundNode("param"))).bind("ref"))),
+          unless(hasAncestor(
+              lambdaExpr(valueCapturesVar(equalsBoundNode("param"))))))
+          .bind("move-call");
+
+  Finder->addMatcher(
+      parmVarDecl(
+          hasType(type(rValueReferenceType())), parmVarDecl().bind("param"),
+          unless(hasType(references(qualType(
+              anyOf(isConstQualified(), substTemplateTypeParmType()))))),
+          optionally(hasType(qualType(references(templateTypeParmType(
+              hasDeclaration(templateTypeParmDecl().bind("template-type"))))))),
+          anyOf(hasAncestor(cxxConstructorDecl(
+                    ToParam, isDefinition(), unless(isMoveConstructor()),
+                    optionally(hasDescendant(MoveCallMatcher)))),
+                hasAncestor(functionDecl(
+                    unless(cxxConstructorDecl()), ToParam,
+                    unless(cxxMethodDecl(isMoveAssignmentOperator())),
+                    hasBody(optionally(hasDescendant(MoveCallMatcher))))))),
+      this);
+}
+
+void RvalueReferenceParamNotMovedCheck::check(
+    const MatchFinder::MatchResult &Result) {
+  const auto *Param = Result.Nodes.getNodeAs<ParmVarDecl>("param");
+  const auto *TemplateType =
+      Result.Nodes.getNodeAs<TemplateTypeParmDecl>("template-type");
+
+  if (!Param)
+    return;
+
+  if (IgnoreUnnamedParams && Param->getName().empty())
+    return;
+
+  const auto *Function = dyn_cast<FunctionDecl>(Param->getDeclContext());
+  if (!Function)
+    return;
+
+  if (IgnoreNonDeducedTemplateTypes && TemplateType)
+    return;
+
+  if (TemplateType) {
+    if (const FunctionTemplateDecl *FuncTemplate =
+            Function->getDescribedFunctionTemplate()) {
+      const TemplateParameterList *Params =
+          FuncTemplate->getTemplateParameters();
+      if (llvm::is_contained(*Params, TemplateType)) {
+        // Ignore forwarding reference
+        return;
+      }
+    }
+  }
+
+  const auto *MoveCall = Result.Nodes.getNodeAs<CallExpr>("move-call");
+  if (!MoveCall) {
+    diag(Param->getLocation(),
+         "rvalue reference parameter %0 is never moved from "
+         "inside the function body")
+        << Param;
+  }
+}
+
+RvalueReferenceParamNotMovedCheck::RvalueReferenceParamNotMovedCheck(
+    StringRef Name, ClangTidyContext *Context)
+    : ClangTidyCheck(Name, Context),
+      AllowPartialMove(Options.getLocalOrGlobal("AllowPartialMove", false)),
+      IgnoreUnnamedParams(
+          Options.getLocalOrGlobal("IgnoreUnnamedParams", false)),
+      IgnoreNonDeducedTemplateTypes(
+          Options.getLocalOrGlobal("IgnoreNonDeducedTemplateTypes", false)) {}
+
+void RvalueReferenceParamNotMovedCheck::storeOptions(
+    ClangTidyOptions::OptionMap &Opts) {
+  Options.store(Opts, "AllowPartialMove", AllowPartialMove);
+  Options.store(Opts, "IgnoreUnnamedParams", IgnoreUnnamedParams);
+  Options.store(Opts, "IgnoreNonDeducedTemplateTypes",
+                IgnoreNonDeducedTemplateTypes);
+}
+
+} // namespace clang::tidy::cppcoreguidelines
diff --git a/clang-tools-extra/clang-tidy/cppcoreguidelines/RvalueReferenceParamNotMovedCheck.h b/clang-tools-extra/clang-tidy/cppcoreguidelines/RvalueReferenceParamNotMovedCheck.h
new file mode 100644
index 0000000000000..d8c3d2bd4ba0e
--- /dev/null
+++ b/clang-tools-extra/clang-tidy/cppcoreguidelines/RvalueReferenceParamNotMovedCheck.h
@@ -0,0 +1,39 @@
+//===--- RvalueReferenceParamNotMovedCheck.h - clang-tidy -------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_TOOLS_EXTRA_CLANG_TIDY_CPPCOREGUIDELINES_RVALUEREFERENCEPARAMNOTMOVEDCHECK_H
+#define LLVM_CLANG_TOOLS_EXTRA_CLANG_TIDY_CPPCOREGUIDELINES_RVALUEREFERENCEPARAMNOTMOVEDCHECK_H
+
+#include "../ClangTidyCheck.h"
+
+namespace clang::tidy::cppcoreguidelines {
+
+/// Warns when an rvalue reference function parameter is never moved within
+/// the function body. This check implements CppCoreGuideline F.18.
+///
+/// For the user-facing documentation see:
+/// http://clang.llvm.org/extra/clang-tidy/checks/cppcoreguidelines/rvalue-reference-param-not-moved.html
+class RvalueReferenceParamNotMovedCheck : public ClangTidyCheck {
+public:
+  RvalueReferenceParamNotMovedCheck(StringRef Name, ClangTidyContext *Context);
+  void registerMatchers(ast_matchers::MatchFinder *Finder) override;
+  void check(const ast_matchers::MatchFinder::MatchResult &Result) override;
+  bool isLanguageVersionSupported(const LangOptions &LangOpts) const override {
+    return LangOpts.CPlusPlus11;
+  }
+  void storeOptions(ClangTidyOptions::OptionMap &Opts) override;
+
+private:
+  const bool AllowPartialMove;
+  const bool IgnoreUnnamedParams;
+  const bool IgnoreNonDeducedTemplateTypes;
+};
+
+} // namespace clang::tidy::cppcoreguidelines
+
+#endif // LLVM_CLANG_TOOLS_EXTRA_CLANG_TIDY_CPPCOREGUIDELINES_RVALUEREFERENCEPARAMNOTMOVEDCHECK_H
diff --git a/clang-tools-extra/clang-tidy/cppcoreguidelines/SlicingCheck.cpp b/clang-tools-extra/clang-tidy/cppcoreguidelines/SlicingCheck.cpp
index bc094a2f028d6..76754394de760 100644
--- a/clang-tools-extra/clang-tidy/cppcoreguidelines/SlicingCheck.cpp
+++ b/clang-tools-extra/clang-tidy/cppcoreguidelines/SlicingCheck.cpp
@@ -40,12 +40,15 @@ void SlicingCheck::registerMatchers(MatchFinder *Finder) {
   const auto HasTypeDerivedFromBaseDecl =
       anyOf(hasType(IsDerivedFromBaseDecl),
             hasType(references(IsDerivedFromBaseDecl)));
-  const auto IsWithinDerivedCtor =
-      hasParent(cxxConstructorDecl(ofClass(equalsBoundNode("DerivedDecl"))));
+  const auto IsCallToBaseClass = hasParent(cxxConstructorDecl(
+      ofClass(isSameOrDerivedFrom(equalsBoundNode("DerivedDecl"))),
+      hasAnyConstructorInitializer(allOf(
+          isBaseInitializer(), withInitializer(equalsBoundNode("Call"))))));
 
   // Assignment slicing: "a = b;" and "a = std::move(b);" variants.
   const auto SlicesObjectInAssignment =
-      callExpr(callee(cxxMethodDecl(anyOf(isCopyAssignmentOperator(),
+      callExpr(expr().bind("Call"),
+               callee(cxxMethodDecl(anyOf(isCopyAssignmentOperator(),
                                           isMoveAssignmentOperator()),
                                     OfBaseClass)),
                hasArgument(1, HasTypeDerivedFromBaseDecl));
@@ -53,17 +56,17 @@ void SlicingCheck::registerMatchers(MatchFinder *Finder) {
   // Construction slicing: "A a{b};" and "f(b);" variants. Note that in case of
   // slicing the letter will create a temporary and therefore call a ctor.
   const auto SlicesObjectInCtor = cxxConstructExpr(
+      expr().bind("Call"),
       hasDeclaration(cxxConstructorDecl(
           anyOf(isCopyConstructor(), isMoveConstructor()), OfBaseClass)),
       hasArgument(0, HasTypeDerivedFromBaseDecl),
       // We need to disable matching on the call to the base copy/move
       // constructor in DerivedDecl's constructors.
-      unless(IsWithinDerivedCtor));
+      unless(IsCallToBaseClass));
 
-  Finder->addMatcher(traverse(TK_AsIs, expr(anyOf(SlicesObjectInAssignment,
-                                                  SlicesObjectInCtor))
-                                           .bind("Call")),
-                     this);
+  Finder->addMatcher(
+      traverse(TK_AsIs, expr(SlicesObjectInAssignment).bind("Call")), this);
+  Finder->addMatcher(traverse(TK_AsIs, SlicesObjectInCtor), this);
 }
 
 /// Warns on methods overridden in DerivedDecl with respect to BaseDecl.
diff --git a/clang-tools-extra/clang-tidy/google/AvoidUnderscoreInGoogletestNameCheck.cpp b/clang-tools-extra/clang-tidy/google/AvoidUnderscoreInGoogletestNameCheck.cpp
index 8ffcf476457db..c5bd6055072aa 100644
--- a/clang-tools-extra/clang-tidy/google/AvoidUnderscoreInGoogletestNameCheck.cpp
+++ b/clang-tools-extra/clang-tidy/google/AvoidUnderscoreInGoogletestNameCheck.cpp
@@ -24,7 +24,7 @@ constexpr llvm::StringLiteral KDisabledTestPrefix = "DISABLED_";
 static bool isGoogletestTestMacro(StringRef MacroName) {
   static const llvm::StringSet<> MacroNames = {"TEST", "TEST_F", "TEST_P",
                                                "TYPED_TEST", "TYPED_TEST_P"};
-  return MacroNames.find(MacroName) != MacroNames.end();
+  return MacroNames.contains(MacroName);
 }
 
 namespace {
diff --git a/clang-tools-extra/clang-tidy/modernize/UseDefaultMemberInitCheck.cpp b/clang-tools-extra/clang-tidy/modernize/UseDefaultMemberInitCheck.cpp
index d3e7bd4b8032c..673c475db630f 100644
--- a/clang-tools-extra/clang-tidy/modernize/UseDefaultMemberInitCheck.cpp
+++ b/clang-tools-extra/clang-tidy/modernize/UseDefaultMemberInitCheck.cpp
@@ -246,8 +246,12 @@ void UseDefaultMemberInitCheck::checkDefaultInit(
   // Check whether we have multiple hand-written constructors and bomb out, as
   // it is hard to reconcile their sets of member initializers.
   const auto *ClassDecl = cast<CXXRecordDecl>(Field->getParent());
-  if (llvm::count_if(ClassDecl->ctors(), [](const CXXConstructorDecl *Ctor) {
-        return !Ctor->isCopyOrMoveConstructor();
+  if (llvm::count_if(ClassDecl->decls(), [](const Decl *D) {
+        if (const auto *FTD = dyn_cast<FunctionTemplateDecl>(D))
+          D = FTD->getTemplatedDecl();
+        if (const auto *Ctor = dyn_cast<CXXConstructorDecl>(D))
+          return !Ctor->isCopyOrMoveConstructor();
+        return false;
       }) > 1)
     return;
 
diff --git a/clang-tools-extra/clang-tidy/mpi/TypeMismatchCheck.cpp b/clang-tools-extra/clang-tidy/mpi/TypeMismatchCheck.cpp
index 903fd42d12ab1..5abe4f77d6598 100644
--- a/clang-tools-extra/clang-tidy/mpi/TypeMismatchCheck.cpp
+++ b/clang-tools-extra/clang-tidy/mpi/TypeMismatchCheck.cpp
@@ -75,7 +75,7 @@ static bool isStandardMPIDatatype(StringRef MPIDatatype) {
                                        "MPI_CXX_DOUBLE_COMPLEX",
                                        "MPI_CXX_LONG_DOUBLE_COMPLEX"};
 
-  return AllTypes.find(MPIDatatype) != AllTypes.end();
+  return AllTypes.contains(MPIDatatype);
 }
 
 /// Check if a BuiltinType matches the MPI datatype.
diff --git a/clang-tools-extra/clang-tidy/readability/IdentifierNamingCheck.cpp b/clang-tools-extra/clang-tidy/readability/IdentifierNamingCheck.cpp
index 61421a13d73ba..81d05a8a93cd9 100644
--- a/clang-tools-extra/clang-tidy/readability/IdentifierNamingCheck.cpp
+++ b/clang-tools-extra/clang-tidy/readability/IdentifierNamingCheck.cpp
@@ -230,9 +230,8 @@ static StringRef const HungarainNotationUserDefinedTypes[] = {
 // clang-format on
 
 IdentifierNamingCheck::NamingStyle::NamingStyle(
-    std::optional<IdentifierNamingCheck::CaseType> Case,
-    const std::string &Prefix, const std::string &Suffix,
-    const std::string &IgnoredRegexpStr, HungarianPrefixType HPType)
+    std::optional<IdentifierNamingCheck::CaseType> Case, StringRef Prefix,
+    StringRef Suffix, StringRef IgnoredRegexpStr, HungarianPrefixType HPType)
     : Case(Case), Prefix(Prefix), Suffix(Suffix),
       IgnoredRegexpStr(IgnoredRegexpStr), HPType(HPType) {
   if (!IgnoredRegexpStr.empty()) {
@@ -265,22 +264,21 @@ IdentifierNamingCheck::FileStyle IdentifierNamingCheck::getFileStyleFromOptions(
 
     memcpy(&StyleString[StyleSize], "IgnoredRegexp", 13);
     StyleString.truncate(StyleSize + 13);
-    StringRef IgnoredRegexpStr = Options.get(StyleString, "");
+    std::optional<StringRef> IgnoredRegexpStr = Options.get(StyleString);
     memcpy(&StyleString[StyleSize], "Prefix", 6);
     StyleString.truncate(StyleSize + 6);
-    std::string Prefix(Options.get(StyleString, ""));
+    std::optional<StringRef> Prefix(Options.get(StyleString));
     // Fast replacement of [Pre]fix -> [Suf]fix.
     memcpy(&StyleString[StyleSize], "Suf", 3);
-    std::string Postfix(Options.get(StyleString, ""));
+    std::optional<StringRef> Postfix(Options.get(StyleString));
     memcpy(&StyleString[StyleSize], "Case", 4);
     StyleString.pop_back_n(2);
-    auto CaseOptional =
+    std::optional<CaseType> CaseOptional =
         Options.get<IdentifierNamingCheck::CaseType>(StyleString);
 
-    if (CaseOptional || !Prefix.empty() || !Postfix.empty() ||
-        !IgnoredRegexpStr.empty() || HPTOpt)
-      Styles[I].emplace(std::move(CaseOptional), std::move(Prefix),
-                        std::move(Postfix), IgnoredRegexpStr.str(),
+    if (CaseOptional || Prefix || Postfix || IgnoredRegexpStr || HPTOpt)
+      Styles[I].emplace(std::move(CaseOptional), Prefix.value_or(""),
+                        Postfix.value_or(""), IgnoredRegexpStr.value_or(""),
                         HPTOpt.value_or(IdentifierNamingCheck::HPT_Off));
   }
   bool IgnoreMainLike = Options.get("IgnoreMainLikeFunctions", false);
@@ -650,7 +648,9 @@ std::string IdentifierNamingCheck::HungarianNotation::getClassPrefix(
 
 std::string IdentifierNamingCheck::HungarianNotation::getEnumPrefix(
     const EnumConstantDecl *ECD) const {
-  std::string Name = ECD->getType().getAsString();
+  const EnumDecl *ED = cast<EnumDecl>(ECD->getDeclContext());
+
+  std::string Name = ED->getName().str();
   if (std::string::npos != Name.find("enum")) {
     Name = Name.substr(strlen("enum"), Name.length() - strlen("enum"));
     Name = Name.erase(0, Name.find_first_not_of(" "));
diff --git a/clang-tools-extra/clang-tidy/readability/IdentifierNamingCheck.h b/clang-tools-extra/clang-tidy/readability/IdentifierNamingCheck.h
index cbf0653a280b6..b1db919902e22 100644
--- a/clang-tools-extra/clang-tidy/readability/IdentifierNamingCheck.h
+++ b/clang-tools-extra/clang-tidy/readability/IdentifierNamingCheck.h
@@ -68,8 +68,8 @@ class IdentifierNamingCheck final : public RenamerClangTidyCheck {
   struct NamingStyle {
     NamingStyle() = default;
 
-    NamingStyle(std::optional<CaseType> Case, const std::string &Prefix,
-                const std::string &Suffix, const std::string &IgnoredRegexpStr,
+    NamingStyle(std::optional<CaseType> Case, StringRef Prefix,
+                StringRef Suffix, StringRef IgnoredRegexpStr,
                 HungarianPrefixType HPType);
     NamingStyle(const NamingStyle &O) = delete;
     NamingStyle &operator=(NamingStyle &&O) = default;
diff --git a/clang-tools-extra/clang-tidy/readability/MagicNumbersCheck.cpp b/clang-tools-extra/clang-tidy/readability/MagicNumbersCheck.cpp
index 64940ede8e1c7..ca81b6e3c6e61 100644
--- a/clang-tools-extra/clang-tidy/readability/MagicNumbersCheck.cpp
+++ b/clang-tools-extra/clang-tidy/readability/MagicNumbersCheck.cpp
@@ -14,6 +14,8 @@
 #include "MagicNumbersCheck.h"
 #include "../utils/OptionsUtils.h"
 #include "clang/AST/ASTContext.h"
+#include "clang/AST/ASTTypeTraits.h"
+#include "clang/AST/Type.h"
 #include "clang/ASTMatchers/ASTMatchFinder.h"
 #include "llvm/ADT/STLExtras.h"
 #include <algorithm>
@@ -42,6 +44,18 @@ static bool isUsedToInitializeAConstant(const MatchFinder::MatchResult &Result,
                       });
 }
 
+static bool isUsedToDefineATypeAlias(const MatchFinder::MatchResult &Result,
+                                     const DynTypedNode &Node) {
+
+  if (Node.get<TypeAliasDecl>() || Node.get<TypedefNameDecl>())
+    return true;
+
+  return llvm::any_of(Result.Context->getParents(Node),
+                      [&Result](const DynTypedNode &Parent) {
+                        return isUsedToDefineATypeAlias(Result, Parent);
+                      });
+}
+
 static bool isUsedToDefineABitField(const MatchFinder::MatchResult &Result,
                                     const DynTypedNode &Node) {
   const auto *AsFieldDecl = Node.get<FieldDecl>();
@@ -66,6 +80,7 @@ MagicNumbersCheck::MagicNumbersCheck(StringRef Name, ClangTidyContext *Context)
       IgnoreBitFieldsWidths(Options.get("IgnoreBitFieldsWidths", true)),
       IgnorePowersOf2IntegerValues(
           Options.get("IgnorePowersOf2IntegerValues", false)),
+      IgnoreTypeAliases(Options.get("IgnoreTypeAliases", false)),
       RawIgnoredIntegerValues(
           Options.get("IgnoredIntegerValues", DefaultIgnoredIntegerValues)),
       RawIgnoredFloatingPointValues(Options.get(
@@ -114,6 +129,7 @@ void MagicNumbersCheck::storeOptions(ClangTidyOptions::OptionMap &Opts) {
   Options.store(Opts, "IgnoreBitFieldsWidths", IgnoreBitFieldsWidths);
   Options.store(Opts, "IgnorePowersOf2IntegerValues",
                 IgnorePowersOf2IntegerValues);
+  Options.store(Opts, "IgnoreTypeAliases", IgnoreTypeAliases);
   Options.store(Opts, "IgnoredIntegerValues", RawIgnoredIntegerValues);
   Options.store(Opts, "IgnoredFloatingPointValues",
                 RawIgnoredFloatingPointValues);
@@ -137,10 +153,13 @@ bool MagicNumbersCheck::isConstant(const MatchFinder::MatchResult &Result,
                                    const Expr &ExprResult) const {
   return llvm::any_of(
       Result.Context->getParents(ExprResult),
-      [&Result](const DynTypedNode &Parent) {
+      [this, &Result](const DynTypedNode &Parent) {
         if (isUsedToInitializeAConstant(Result, Parent))
           return true;
 
+        if (IgnoreTypeAliases && isUsedToDefineATypeAlias(Result, Parent))
+          return true;
+
         // Ignore this instance, because this matches an
         // expanded class enumeration value.
         if (Parent.get<CStyleCastExpr>() &&
diff --git a/clang-tools-extra/clang-tidy/readability/MagicNumbersCheck.h b/clang-tools-extra/clang-tidy/readability/MagicNumbersCheck.h
index ba39f17f4d500..39e9baea3adc8 100644
--- a/clang-tools-extra/clang-tidy/readability/MagicNumbersCheck.h
+++ b/clang-tools-extra/clang-tidy/readability/MagicNumbersCheck.h
@@ -84,6 +84,7 @@ class MagicNumbersCheck : public ClangTidyCheck {
   const bool IgnoreAllFloatingPointValues;
   const bool IgnoreBitFieldsWidths;
   const bool IgnorePowersOf2IntegerValues;
+  const bool IgnoreTypeAliases;
   const StringRef RawIgnoredIntegerValues;
   const StringRef RawIgnoredFloatingPointValues;
 
diff --git a/clang-tools-extra/clang-tidy/readability/RedundantStringCStrCheck.cpp b/clang-tools-extra/clang-tidy/readability/RedundantStringCStrCheck.cpp
index 960d5abcc912e..dfcc82c270540 100644
--- a/clang-tools-extra/clang-tidy/readability/RedundantStringCStrCheck.cpp
+++ b/clang-tools-extra/clang-tidy/readability/RedundantStringCStrCheck.cpp
@@ -11,6 +11,8 @@
 //===----------------------------------------------------------------------===//
 
 #include "RedundantStringCStrCheck.h"
+#include "../utils/Matchers.h"
+#include "../utils/OptionsUtils.h"
 #include "clang/Lex/Lexer.h"
 #include "clang/Tooling/FixIt.h"
 
@@ -52,6 +54,10 @@ formatDereference(const ast_matchers::MatchFinder::MatchResult &Result,
 
   if (Text.empty())
     return std::string();
+
+  // Remove remaining '->' from overloaded operator call
+  Text.consume_back("->");
+
   // Add leading '*'.
   if (needParensAfterUnaryOperator(ExprNode)) {
     return (llvm::Twine("*(") + Text + ")").str();
@@ -65,6 +71,17 @@ AST_MATCHER(MaterializeTemporaryExpr, isBoundToLValue) {
 
 } // end namespace
 
+RedundantStringCStrCheck::RedundantStringCStrCheck(StringRef Name,
+                                                   ClangTidyContext *Context)
+    : ClangTidyCheck(Name, Context),
+      StringParameterFunctions(utils::options::parseStringList(
+          Options.get("StringParameterFunctions", ""))) {
+  if (getLangOpts().CPlusPlus20)
+    StringParameterFunctions.push_back("::std::format");
+  if (getLangOpts().CPlusPlus2b)
+    StringParameterFunctions.push_back("::std::print");
+}
+
 void RedundantStringCStrCheck::registerMatchers(
     ast_matchers::MatchFinder *Finder) {
   // Match expressions of type 'string' or 'string*'.
@@ -178,6 +195,18 @@ void RedundantStringCStrCheck::registerMatchers(
               // directly.
               hasArgument(0, StringCStrCallExpr))),
       this);
+
+  if (!StringParameterFunctions.empty()) {
+    // Detect redundant 'c_str()' calls in parameters passed to std::format in
+    // C++20 onwards and std::print in C++23 onwards.
+    Finder->addMatcher(
+        traverse(TK_AsIs,
+                 callExpr(callee(functionDecl(matchers::matchesAnyListedName(
+                              StringParameterFunctions))),
+                          forEachArgumentWithParam(StringCStrCallExpr,
+                                                   parmVarDecl()))),
+        this);
+  }
 }
 
 void RedundantStringCStrCheck::check(const MatchFinder::MatchResult &Result) {
diff --git a/clang-tools-extra/clang-tidy/readability/RedundantStringCStrCheck.h b/clang-tools-extra/clang-tidy/readability/RedundantStringCStrCheck.h
index 662fb955fc653..e2e6ab1fd939c 100644
--- a/clang-tools-extra/clang-tidy/readability/RedundantStringCStrCheck.h
+++ b/clang-tools-extra/clang-tidy/readability/RedundantStringCStrCheck.h
@@ -16,13 +16,15 @@ namespace clang::tidy::readability {
 /// Finds unnecessary calls to `std::string::c_str()`.
 class RedundantStringCStrCheck : public ClangTidyCheck {
 public:
-  RedundantStringCStrCheck(StringRef Name, ClangTidyContext *Context)
-      : ClangTidyCheck(Name, Context) {}
+  RedundantStringCStrCheck(StringRef Name, ClangTidyContext *Context);
   bool isLanguageVersionSupported(const LangOptions &LangOpts) const override {
     return LangOpts.CPlusPlus;
   }
   void registerMatchers(ast_matchers::MatchFinder *Finder) override;
   void check(const ast_matchers::MatchFinder::MatchResult &Result) override;
+
+private:
+  std::vector<StringRef> StringParameterFunctions;
 };
 
 } // namespace clang::tidy::readability
diff --git a/clang-tools-extra/clang-tidy/readability/SuspiciousCallArgumentCheck.cpp b/clang-tools-extra/clang-tidy/readability/SuspiciousCallArgumentCheck.cpp
index 7fe0c0982dc30..14bf22a4d4200 100644
--- a/clang-tools-extra/clang-tidy/readability/SuspiciousCallArgumentCheck.cpp
+++ b/clang-tools-extra/clang-tidy/readability/SuspiciousCallArgumentCheck.cpp
@@ -137,11 +137,11 @@ static bool applyEqualityHeuristic(StringRef Arg, StringRef Param) {
 static bool applyAbbreviationHeuristic(
     const llvm::StringMap<std::string> &AbbreviationDictionary, StringRef Arg,
     StringRef Param) {
-  if (AbbreviationDictionary.find(Arg) != AbbreviationDictionary.end() &&
+  if (AbbreviationDictionary.contains(Arg) &&
       Param.equals(AbbreviationDictionary.lookup(Arg)))
     return true;
 
-  if (AbbreviationDictionary.find(Param) != AbbreviationDictionary.end() &&
+  if (AbbreviationDictionary.contains(Param) &&
       Arg.equals(AbbreviationDictionary.lookup(Param)))
     return true;
 
diff --git a/clang-tools-extra/clang-tidy/rename_check.py b/clang-tools-extra/clang-tidy/rename_check.py
index 764960e044219..f56776fdb7291 100755
--- a/clang-tools-extra/clang-tidy/rename_check.py
+++ b/clang-tools-extra/clang-tidy/rename_check.py
@@ -94,10 +94,14 @@ def deleteMatchingLines(fileName, pattern):
 
 def getListOfFiles(clang_tidy_path):
   files = glob.glob(os.path.join(clang_tidy_path, '**'), recursive=True)
+  files += [os.path.normpath(os.path.join(clang_tidy_path,
+                                           '../docs/ReleaseNotes.rst'))]
   files += glob.glob(os.path.join(clang_tidy_path, '..', 'test',
                                   'clang-tidy', 'checkers', '**'), recursive=True)
   files += glob.glob(os.path.join(clang_tidy_path, '..', 'docs',
-                                  'clang-tidy', 'checks', '*'))
+                                  'clang-tidy', 'checks', '*.rst'))
+  files += glob.glob(os.path.join(clang_tidy_path, '..', 'docs',
+                                  'clang-tidy', 'checks', "*", "*.rst"), recursive=True)
   return [filename for filename in files if os.path.isfile(filename)]
 
 
@@ -213,9 +217,11 @@ def add_release_notes(clang_tidy_path, old_check_name, new_check_name):
         if header_found and add_note_here:
           if not line.startswith('^^^^'):
             f.write("""- The '%s' check was renamed to :doc:`%s
-  <clang-tidy/checks/%s>`
+  <clang-tidy/checks/%s/%s>`
 
-""" % (old_check_name, new_check_name, new_check_name))
+                    """ % (old_check_name, new_check_name,
+                           new_check_name.split('-', 1)[0],
+                           '-'.join(new_check_name.split('-')[1:])))
             note_added = True
 
       f.write(line)
@@ -232,15 +238,18 @@ def main():
 
   old_module = args.old_check_name.split('-')[0]
   new_module = args.new_check_name.split('-')[0]
+  old_name = '-'.join(args.old_check_name.split('-')[1:])
+  new_name  = '-'.join(args.new_check_name.split('-')[1:])
+
   if args.check_class_name:
     check_name_camel = args.check_class_name
   else:
     check_name_camel = (''.join(map(lambda elem: elem.capitalize(),
-                                    args.old_check_name.split('-')[1:])) +
+                                    old_name.split('-'))) +
                         'Check')
 
   new_check_name_camel = (''.join(map(lambda elem: elem.capitalize(),
-                                      args.new_check_name.split('-')[1:])) +
+                                      new_name.split('-'))) +
                           'Check')
 
   clang_tidy_path = os.path.dirname(__file__)
@@ -255,7 +264,7 @@ def main():
   old_module_path = os.path.join(clang_tidy_path, old_module)
   new_module_path = os.path.join(clang_tidy_path, new_module)
 
-  if (args.old_check_name != args.new_check_name):
+  if (old_module != new_module):
     # Remove the check from the old module.
     cmake_lists = os.path.join(old_module_path, 'CMakeLists.txt')
     check_found = deleteMatchingLines(cmake_lists, '\\b' + check_name_camel)
@@ -270,12 +279,10 @@ def main():
     deleteMatchingLines(os.path.join(old_module_path, modulecpp),
                       '\\b' + check_name_camel + '|\\b' + args.old_check_name)
 
-  old_check_filename = '-'.join(args.old_check_name.split('-')[1:])
-  new_check_filename = '-'.join(args.new_check_name.split('-')[1:])
-
   for filename in getListOfFiles(clang_tidy_path):
     originalName = filename
-    filename = fileRename(filename, old_check_filename, new_check_filename)
+    filename = fileRename(filename, old_module + "/" + old_name, new_module + "/" + new_name)
+    filename = fileRename(filename, args.old_check_name, args.new_check_name)
     filename = fileRename(filename, check_name_camel, new_check_name_camel)
     replaceInFile(filename, generateCommentLineHeader(originalName),
                   generateCommentLineHeader(filename))
@@ -284,7 +291,7 @@ def main():
     for header_guard in header_guard_variants:
       replaceInFile(filename, header_guard, header_guard_new)
 
-    if args.new_check_name + '.rst' in filename:
+    if new_module + '/'+ new_name + '.rst' in filename:
       replaceInFile(
           filename,
           args.old_check_name + '\n' + '=' * len(args.old_check_name) + '\n',
@@ -295,6 +302,8 @@ def main():
                   new_module + '::' + new_check_name_camel)
     replaceInFile(filename, old_module + '/' + check_name_camel,
                   new_module + '/' + new_check_name_camel)
+    replaceInFile(filename, old_module + '/' + old_name,
+                  new_module + '/' + new_name)
     replaceInFile(filename, check_name_camel, new_check_name_camel)
 
   if old_module != new_module or new_module == 'llvm':
@@ -311,13 +320,12 @@ def main():
                          'namespace clang::tidy::' + old_module + '[^ \n]*',
                          'namespace clang::tidy::' + new_namespace)
 
-  if (args.old_check_name == args.new_check_name):
-    return
+  if old_module != new_module:
 
-  # Add check to the new module.
-  adapt_cmake(new_module_path, new_check_name_camel)
-  adapt_module(new_module_path, new_module, args.new_check_name,
-               new_check_name_camel)
+      # Add check to the new module.
+      adapt_cmake(new_module_path, new_check_name_camel)
+      adapt_module(new_module_path, new_module, args.new_check_name,
+                   new_check_name_camel)
 
   os.system(os.path.join(clang_tidy_path, 'add_new_check.py')
             + ' --update-docs')
diff --git a/clang-tools-extra/clang-tidy/tool/run-clang-tidy.py b/clang-tools-extra/clang-tidy/tool/run-clang-tidy.py
index e3da6fb9b0967..30e29182908d7 100755
--- a/clang-tools-extra/clang-tidy/tool/run-clang-tidy.py
+++ b/clang-tools-extra/clang-tidy/tool/run-clang-tidy.py
@@ -300,7 +300,7 @@ def main():
                                   build_path)
 
   tmpdir = None
-  if args.fix or (yaml and args.export_fixes):
+  if args.fix:
     clang_apply_replacements_binary = find_binary(
       args.clang_apply_replacements_binary, "clang-apply-replacements",
       build_path)
diff --git a/clang-tools-extra/clang-tidy/utils/ExprSequence.cpp b/clang-tools-extra/clang-tidy/utils/ExprSequence.cpp
index beb4c44467a80..f9555be57e738 100644
--- a/clang-tools-extra/clang-tidy/utils/ExprSequence.cpp
+++ b/clang-tools-extra/clang-tidy/utils/ExprSequence.cpp
@@ -8,6 +8,7 @@
 
 #include "ExprSequence.h"
 #include "clang/AST/ParentMapContext.h"
+#include "llvm/ADT/SmallVector.h"
 #include <optional>
 
 namespace clang::tidy::utils {
@@ -49,6 +50,7 @@ static SmallVector<const Stmt *, 1> getParentStmts(const Stmt *S,
 }
 
 namespace {
+
 bool isDescendantOrEqual(const Stmt *Descendant, const Stmt *Ancestor,
                          ASTContext *Context) {
   if (Descendant == Ancestor)
@@ -60,6 +62,17 @@ bool isDescendantOrEqual(const Stmt *Descendant, const Stmt *Ancestor,
 
   return false;
 }
+
+llvm::SmallVector<const InitListExpr *>
+getAllInitListForms(const InitListExpr *InitList) {
+  llvm::SmallVector<const InitListExpr *> result = {InitList};
+  if (const InitListExpr *AltForm = InitList->getSyntacticForm())
+    result.push_back(AltForm);
+  if (const InitListExpr *AltForm = InitList->getSemanticForm())
+    result.push_back(AltForm);
+  return result;
+}
+
 } // namespace
 
 ExprSequence::ExprSequence(const CFG *TheCFG, const Stmt *Root,
@@ -111,9 +124,12 @@ const Stmt *ExprSequence::getSequenceSuccessor(const Stmt *S) const {
     } else if (const auto *InitList = dyn_cast<InitListExpr>(Parent)) {
       // Initializer list: Each initializer clause is sequenced after the
       // clauses that precede it.
-      for (unsigned I = 1; I < InitList->getNumInits(); ++I) {
-        if (InitList->getInit(I - 1) == S)
-          return InitList->getInit(I);
+      for (const InitListExpr *Form : getAllInitListForms(InitList)) {
+        for (unsigned I = 1; I < Form->getNumInits(); ++I) {
+          if (Form->getInit(I - 1) == S) {
+            return Form->getInit(I);
+          }
+        }
       }
     } else if (const auto *Compound = dyn_cast<CompoundStmt>(Parent)) {
       // Compound statement: Each sub-statement is sequenced after the
diff --git a/clang-tools-extra/clangd/CodeComplete.cpp b/clang-tools-extra/clangd/CodeComplete.cpp
index 98fa285f71051..260c44b2064b5 100644
--- a/clang-tools-extra/clangd/CodeComplete.cpp
+++ b/clang-tools-extra/clangd/CodeComplete.cpp
@@ -436,9 +436,8 @@ struct CodeCompletionBuilder {
     Bundled.emplace_back();
     BundledEntry &S = Bundled.back();
     if (C.SemaResult) {
-      bool IsPattern = C.SemaResult->Kind == CodeCompletionResult::RK_Pattern;
-      getSignature(*SemaCCS, &S.Signature, &S.SnippetSuffix,
-                   &Completion.RequiredQualifier, IsPattern);
+      getSignature(*SemaCCS, &S.Signature, &S.SnippetSuffix, C.SemaResult->Kind,
+                   C.SemaResult->CursorKind, &Completion.RequiredQualifier);
       if (!C.SemaResult->FunctionCanBeCall)
         S.SnippetSuffix.clear();
       S.ReturnType = getReturnType(*SemaCCS);
diff --git a/clang-tools-extra/clangd/CodeCompletionStrings.cpp b/clang-tools-extra/clangd/CodeCompletionStrings.cpp
index 21f83451f7014..26f8e0e602b2b 100644
--- a/clang-tools-extra/clangd/CodeCompletionStrings.cpp
+++ b/clang-tools-extra/clangd/CodeCompletionStrings.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "CodeCompletionStrings.h"
+#include "clang-c/Index.h"
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/RawCommentList.h"
 #include "clang/Basic/SourceManager.h"
@@ -56,6 +57,26 @@ bool looksLikeDocComment(llvm::StringRef CommentText) {
   return CommentText.find_first_not_of("/*-= \t\r\n") != llvm::StringRef::npos;
 }
 
+// Determine whether the completion string should be patched
+// to replace the last placeholder with $0.
+bool shouldPatchPlaceholder0(CodeCompletionResult::ResultKind ResultKind,
+                             CXCursorKind CursorKind) {
+  bool CompletingPattern = ResultKind == CodeCompletionResult::RK_Pattern;
+
+  if (!CompletingPattern)
+    return false;
+
+  // If the result kind of CodeCompletionResult(CCR) is `RK_Pattern`, it doesn't
+  // always mean we're completing a chunk of statements.  Constructors defined
+  // in base class, for example, are considered as a type of pattern, with the
+  // cursor type set to CXCursor_Constructor.
+  if (CursorKind == CXCursorKind::CXCursor_Constructor ||
+      CursorKind == CXCursorKind::CXCursor_Destructor)
+    return false;
+
+  return true;
+}
+
 } // namespace
 
 std::string getDocComment(const ASTContext &Ctx,
@@ -95,17 +116,20 @@ std::string getDeclComment(const ASTContext &Ctx, const NamedDecl &Decl) {
 }
 
 void getSignature(const CodeCompletionString &CCS, std::string *Signature,
-                  std::string *Snippet, std::string *RequiredQualifiers,
-                  bool CompletingPattern) {
-  // Placeholder with this index will be ${0:…} to mark final cursor position.
+                  std::string *Snippet,
+                  CodeCompletionResult::ResultKind ResultKind,
+                  CXCursorKind CursorKind, std::string *RequiredQualifiers) {
+  // Placeholder with this index will be $0 to mark final cursor position.
   // Usually we do not add $0, so the cursor is placed at end of completed text.
   unsigned CursorSnippetArg = std::numeric_limits<unsigned>::max();
-  if (CompletingPattern) {
-    // In patterns, it's best to place the cursor at the last placeholder, to
-    // handle cases like
-    //    namespace ${1:name} {
-    //      ${0:decls}
-    //    }
+
+  // If the snippet contains a group of statements, we replace the
+  // last placeholder with $0 to leave the cursor there, e.g.
+  //    namespace ${1:name} {
+  //      ${0:decls}
+  //    }
+  // We try to identify such cases using the ResultKind and CursorKind.
+  if (shouldPatchPlaceholder0(ResultKind, CursorKind)) {
     CursorSnippetArg =
         llvm::count_if(CCS, [](const CodeCompletionString::Chunk &C) {
           return C.Kind == CodeCompletionString::CK_Placeholder;
@@ -185,7 +209,7 @@ void getSignature(const CodeCompletionString &CCS, std::string *Signature,
       *Snippet += Chunk.Text;
       break;
     case CodeCompletionString::CK_Optional:
-      assert(Chunk.Optional);      
+      assert(Chunk.Optional);
       // No need to create placeholders for default arguments in Snippet.
       appendOptionalChunk(*Chunk.Optional, Signature);
       break;
diff --git a/clang-tools-extra/clangd/CodeCompletionStrings.h b/clang-tools-extra/clangd/CodeCompletionStrings.h
index 6733d0231df49..566756ac9c8cc 100644
--- a/clang-tools-extra/clangd/CodeCompletionStrings.h
+++ b/clang-tools-extra/clangd/CodeCompletionStrings.h
@@ -42,12 +42,15 @@ std::string getDeclComment(const ASTContext &Ctx, const NamedDecl &D);
 /// If set, RequiredQualifiers is the text that must be typed before the name.
 /// e.g "Base::" when calling a base class member function that's hidden.
 ///
-/// When \p CompletingPattern is true, the last placeholder will be of the form
-/// ${0:…}, indicating the cursor should stay there.
+/// When \p ResultKind is RK_Pattern, the last placeholder will be $0,
+/// indicating the cursor should stay there.
+/// Note that for certain \p CursorKind like \p CXCursor_Constructor, $0 won't
+/// be emitted in order to avoid overlapping normal parameters.
 void getSignature(const CodeCompletionString &CCS, std::string *Signature,
                   std::string *Snippet,
-                  std::string *RequiredQualifiers = nullptr,
-                  bool CompletingPattern = false);
+                  CodeCompletionResult::ResultKind ResultKind,
+                  CXCursorKind CursorKind,
+                  std::string *RequiredQualifiers = nullptr);
 
 /// Assembles formatted documentation for a completion result. This includes
 /// documentation comments and other relevant information like annotations.
diff --git a/clang-tools-extra/clangd/CompileCommands.cpp b/clang-tools-extra/clangd/CompileCommands.cpp
index bcd39b2d38ba5..39b180e002a68 100644
--- a/clang-tools-extra/clangd/CompileCommands.cpp
+++ b/clang-tools-extra/clangd/CompileCommands.cpp
@@ -14,6 +14,7 @@
 #include "clang/Driver/Options.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Tooling/CompilationDatabase.h"
+#include "clang/Tooling/Tooling.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallVector.h"
@@ -27,6 +28,7 @@
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Program.h"
+#include "llvm/TargetParser/Host.h"
 #include <iterator>
 #include <optional>
 #include <string>
@@ -185,6 +187,12 @@ static std::string resolveDriver(llvm::StringRef Driver, bool FollowSymlink,
 
 } // namespace
 
+CommandMangler::CommandMangler() {
+  Tokenizer = llvm::Triple(llvm::sys::getProcessTriple()).isOSWindows()
+                  ? llvm::cl::TokenizeWindowsCommandLine
+                  : llvm::cl::TokenizeGNUCommandLine;
+}
+
 CommandMangler CommandMangler::detect() {
   CommandMangler Result;
   Result.ClangPath = detectClangPath();
@@ -201,9 +209,18 @@ void CommandMangler::operator()(tooling::CompileCommand &Command,
   trace::Span S("AdjustCompileFlags");
   // Most of the modifications below assumes the Cmd starts with a driver name.
   // We might consider injecting a generic driver name like "cc" or "c++", but
-  // a Cmd missing the driver is probably rare enough in practice and errnous.
+  // a Cmd missing the driver is probably rare enough in practice and erroneous.
   if (Cmd.empty())
     return;
+
+  // FS used for expanding response files.
+  // FIXME: ExpandResponseFiles appears not to provide the usual
+  // thread-safety guarantees, as the access to FS is not locked!
+  // For now, use the real FS, which is known to be threadsafe (if we don't
+  // use/change working directory, which ExpandResponseFiles doesn't).
+  auto FS = llvm::vfs::getRealFileSystem();
+  tooling::addExpandedResponseFiles(Cmd, Command.Directory, Tokenizer, *FS);
+
   auto &OptTable = clang::driver::getDriverOptTable();
   // OriginalArgs needs to outlive ArgList.
   llvm::SmallVector<const char *, 16> OriginalArgs;
@@ -212,7 +229,7 @@ void CommandMangler::operator()(tooling::CompileCommand &Command,
     OriginalArgs.push_back(S.c_str());
   bool IsCLMode = driver::IsClangCL(driver::getDriverMode(
       OriginalArgs[0], llvm::ArrayRef(OriginalArgs).slice(1)));
-  // ParseArgs propagates missig arg/opt counts on error, but preserves
+  // ParseArgs propagates missing arg/opt counts on error, but preserves
   // everything it could parse in ArgList. So we just ignore those counts.
   unsigned IgnoredCount;
   // Drop the executable name, as ParseArgs doesn't expect it. This means
@@ -307,12 +324,16 @@ void CommandMangler::operator()(tooling::CompileCommand &Command,
   //    necessary for the system include extractor to identify the file type
   //  - AFTER applying CompileFlags.Edits, because the name of the compiler
   //    that needs to be invoked may come from the CompileFlags->Compiler key
+  //  - BEFORE addTargetAndModeForProgramName(), because gcc doesn't support
+  //    the target flag that might be added.
   //  - BEFORE resolveDriver() because that can mess up the driver path,
   //    e.g. changing gcc to /path/to/clang/bin/gcc
   if (SystemIncludeExtractor) {
     SystemIncludeExtractor(Command, File);
   }
 
+  tooling::addTargetAndModeForProgramName(Cmd, Cmd.front());
+
   // Check whether the flag exists, either as -flag or -flag=*
   auto Has = [&](llvm::StringRef Flag) {
     for (llvm::StringRef Arg : Cmd) {
diff --git a/clang-tools-extra/clangd/CompileCommands.h b/clang-tools-extra/clangd/CompileCommands.h
index eb318d18baf63..1b37f44f0b9db 100644
--- a/clang-tools-extra/clangd/CompileCommands.h
+++ b/clang-tools-extra/clangd/CompileCommands.h
@@ -12,6 +12,7 @@
 #include "support/Threading.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/Support/CommandLine.h"
 #include <deque>
 #include <optional>
 #include <string>
@@ -50,10 +51,11 @@ struct CommandMangler {
                   llvm::StringRef TargetFile) const;
 
 private:
-  CommandMangler() = default;
+  CommandMangler();
 
   Memoize<llvm::StringMap<std::string>> ResolvedDrivers;
   Memoize<llvm::StringMap<std::string>> ResolvedDriversNoFollow;
+  llvm::cl::TokenizerCallback Tokenizer;
 };
 
 // Removes args from a command-line in a semantically-aware way.
diff --git a/clang-tools-extra/clangd/Config.h b/clang-tools-extra/clangd/Config.h
index dffd54b01c459..daadf0ee3d3ce 100644
--- a/clang-tools-extra/clangd/Config.h
+++ b/clang-tools-extra/clangd/Config.h
@@ -88,12 +88,10 @@ struct Config {
     bool StandardLibrary = true;
   } Index;
 
-  enum class UnusedIncludesPolicy {
-    /// Diagnose unused includes.
+  enum class IncludesPolicy {
+    /// Diagnose missing and unused includes.
     Strict,
     None,
-    /// The same as Strict, but using the include-cleaner library.
-    Experiment,
   };
   /// Controls warnings and errors when parsing code.
   struct {
@@ -107,11 +105,12 @@ struct Config {
       llvm::StringMap<std::string> CheckOptions;
     } ClangTidy;
 
-    UnusedIncludesPolicy UnusedIncludes = UnusedIncludesPolicy::None;
-
     /// Enable emitting diagnostics using stale preambles.
     bool AllowStalePreamble = false;
 
+    IncludesPolicy UnusedIncludes = IncludesPolicy::None;
+    IncludesPolicy MissingIncludes = IncludesPolicy::None;
+
     /// IncludeCleaner will not diagnose usages of these headers matched by
     /// these regexes.
     struct {
diff --git a/clang-tools-extra/clangd/ConfigCompile.cpp b/clang-tools-extra/clangd/ConfigCompile.cpp
index 18de6e4d5c3b6..fb6c6e86c1acd 100644
--- a/clang-tools-extra/clangd/ConfigCompile.cpp
+++ b/clang-tools-extra/clangd/ConfigCompile.cpp
@@ -430,17 +430,25 @@ struct FragmentCompiler {
               C.Diagnostics.Suppress.insert(N);
           });
 
-    if (F.UnusedIncludes)
-      if (auto Val =
-              compileEnum<Config::UnusedIncludesPolicy>("UnusedIncludes",
-                                                        **F.UnusedIncludes)
-                  .map("Strict", Config::UnusedIncludesPolicy::Strict)
-                  .map("Experiment", Config::UnusedIncludesPolicy::Experiment)
-                  .map("None", Config::UnusedIncludesPolicy::None)
-                  .value())
+    if (F.UnusedIncludes) {
+      auto Val = compileEnum<Config::IncludesPolicy>("UnusedIncludes",
+                                                     **F.UnusedIncludes)
+                     .map("Strict", Config::IncludesPolicy::Strict)
+                     .map("None", Config::IncludesPolicy::None)
+                     .value();
+      if (!Val && **F.UnusedIncludes == "Experiment") {
+        diag(Warning,
+             "Experiment is deprecated for UnusedIncludes, use Strict instead.",
+             F.UnusedIncludes->Range);
+        Val = Config::IncludesPolicy::Strict;
+      }
+      if (Val) {
         Out.Apply.push_back([Val](const Params &, Config &C) {
           C.Diagnostics.UnusedIncludes = *Val;
         });
+      }
+    }
+
     if (F.AllowStalePreamble) {
       if (auto Val = F.AllowStalePreamble)
         Out.Apply.push_back([Val](const Params &, Config &C) {
@@ -448,6 +456,16 @@ struct FragmentCompiler {
         });
     }
 
+    if (F.MissingIncludes)
+      if (auto Val = compileEnum<Config::IncludesPolicy>("MissingIncludes",
+                                                         **F.MissingIncludes)
+                         .map("Strict", Config::IncludesPolicy::Strict)
+                         .map("None", Config::IncludesPolicy::None)
+                         .value())
+        Out.Apply.push_back([Val](const Params &, Config &C) {
+          C.Diagnostics.MissingIncludes = *Val;
+        });
+
     compile(std::move(F.Includes));
     compile(std::move(F.ClangTidy));
   }
diff --git a/clang-tools-extra/clangd/ConfigFragment.h b/clang-tools-extra/clangd/ConfigFragment.h
index a5597596196fa..a56e919cbaf7a 100644
--- a/clang-tools-extra/clangd/ConfigFragment.h
+++ b/clang-tools-extra/clangd/ConfigFragment.h
@@ -221,7 +221,7 @@ struct Fragment {
     /// This often has other advantages, such as skipping some analysis.
     std::vector<Located<std::string>> Suppress;
 
-    /// Controls how clangd will correct "unnecessary #include directives.
+    /// Controls how clangd will correct "unnecessary" #include directives.
     /// clangd can warn if a header is `#include`d but not used, and suggest
     /// removing it.
     //
@@ -232,13 +232,26 @@ struct Fragment {
     ///
     /// Valid values are:
     /// - Strict
-    /// - Experiment
     /// - None
     std::optional<Located<std::string>> UnusedIncludes;
 
+
     /// Enable emitting diagnostics using stale preambles.
     std::optional<Located<bool>> AllowStalePreamble;
 
+    /// Controls if clangd should analyze missing #include directives.
+    /// clangd will warn if no header providing a symbol is `#include`d
+    /// (missing) directly, and suggest adding it.
+    ///
+    /// Strict means a header providing a symbol is missing if it is not
+    /// *directly #include'd. The file might still compile if the header is
+    /// included transitively.
+    ///
+    /// Valid values are:
+    /// - Strict
+    /// - None
+    std::optional<Located<std::string>> MissingIncludes;
+
     /// Controls IncludeCleaner diagnostics.
     struct IncludesBlock {
       /// Regexes that will be used to avoid diagnosing certain includes as
diff --git a/clang-tools-extra/clangd/ConfigYAML.cpp b/clang-tools-extra/clangd/ConfigYAML.cpp
index 0ec0239fc71e6..84559f5c44f86 100644
--- a/clang-tools-extra/clangd/ConfigYAML.cpp
+++ b/clang-tools-extra/clangd/ConfigYAML.cpp
@@ -128,6 +128,9 @@ class Parser {
     Dict.handle("UnusedIncludes", [&](Node &N) {
       F.UnusedIncludes = scalarValue(N, "UnusedIncludes");
     });
+    Dict.handle("MissingIncludes", [&](Node &N) {
+      F.MissingIncludes = scalarValue(N, "MissingIncludes");
+    });
     Dict.handle("Includes", [&](Node &N) { parse(F.Includes, N); });
     Dict.handle("ClangTidy", [&](Node &N) { parse(F.ClangTidy, N); });
     Dict.handle("AllowStalePreamble", [&](Node &N) {
diff --git a/clang-tools-extra/clangd/FileDistance.cpp b/clang-tools-extra/clangd/FileDistance.cpp
index cf10d158f08ca..a5fb3d2bcc35b 100644
--- a/clang-tools-extra/clangd/FileDistance.cpp
+++ b/clang-tools-extra/clangd/FileDistance.cpp
@@ -78,7 +78,7 @@ FileDistance::FileDistance(llvm::StringMap<SourceParams> Sources,
         Down.push_back(Hash);
       // We can't just break after MaxUpTraversals, must still set DownEdges.
       if (I > S.getValue().MaxUpTraversals) {
-        if (Cache.find(Hash) != Cache.end())
+        if (Cache.contains(Hash))
           break;
       } else {
         unsigned Cost = S.getValue().Cost + I * Opts.UpCost;
diff --git a/clang-tools-extra/clangd/GlobalCompilationDatabase.cpp b/clang-tools-extra/clangd/GlobalCompilationDatabase.cpp
index def43683e0ab7..d1833759917a3 100644
--- a/clang-tools-extra/clangd/GlobalCompilationDatabase.cpp
+++ b/clang-tools-extra/clangd/GlobalCompilationDatabase.cpp
@@ -244,15 +244,7 @@ static std::unique_ptr<tooling::CompilationDatabase>
 parseJSON(PathRef Path, llvm::StringRef Data, std::string &Error) {
   if (auto CDB = tooling::JSONCompilationDatabase::loadFromBuffer(
           Data, Error, tooling::JSONCommandLineSyntax::AutoDetect)) {
-    // FS used for expanding response files.
-    // FIXME: ExpandResponseFilesDatabase appears not to provide the usual
-    // thread-safety guarantees, as the access to FS is not locked!
-    // For now, use the real FS, which is known to be threadsafe (if we don't
-    // use/change working directory, which ExpandResponseFilesDatabase doesn't).
-    auto FS = llvm::vfs::getRealFileSystem();
-    return tooling::inferTargetAndDriverMode(
-        tooling::inferMissingCompileCommands(
-            expandResponseFiles(std::move(CDB), std::move(FS))));
+    return tooling::inferMissingCompileCommands(std::move(CDB));
   }
   return nullptr;
 }
diff --git a/clang-tools-extra/clangd/Headers.cpp b/clang-tools-extra/clangd/Headers.cpp
index 7000c1cbf4513..344afdba9622a 100644
--- a/clang-tools-extra/clangd/Headers.cpp
+++ b/clang-tools-extra/clangd/Headers.cpp
@@ -25,8 +25,7 @@
 namespace clang {
 namespace clangd {
 
-class IncludeStructure::RecordHeaders : public PPCallbacks,
-                                        public CommentHandler {
+class IncludeStructure::RecordHeaders : public PPCallbacks {
 public:
   RecordHeaders(const CompilerInstance &CI, IncludeStructure *Out)
       : SM(CI.getSourceManager()),
@@ -61,8 +60,6 @@ class IncludeStructure::RecordHeaders : public PPCallbacks,
           SM.getLineNumber(SM.getFileID(HashLoc), Inc.HashOffset) - 1;
       Inc.FileKind = FileKind;
       Inc.Directive = IncludeTok.getIdentifierInfo()->getPPKeywordID();
-      if (LastPragmaKeepInMainFileLine == Inc.HashLine)
-        Inc.BehindPragmaKeep = true;
       if (File) {
         IncludeStructure::HeaderID HID = Out->getOrCreateID(*File);
         Inc.HeaderID = static_cast<unsigned>(HID);
@@ -108,17 +105,6 @@ class IncludeStructure::RecordHeaders : public PPCallbacks,
       --Level;
       if (PrevFID == BuiltinFile)
         InBuiltinFile = false;
-      // At file exit time HeaderSearchInfo is valid and can be used to
-      // determine whether the file was a self-contained header or not.
-      if (const FileEntry *FE = SM.getFileEntryForID(PrevFID)) {
-        // isSelfContainedHeader only returns true once the full header-guard
-        // structure has been seen, i.e. when exiting the *outer* copy of the
-        // file. So last result wins.
-        if (tooling::isSelfContainedHeader(FE, SM, HeaderInfo))
-          Out->NonSelfContained.erase(*Out->getID(FE));
-        else
-          Out->NonSelfContained.insert(*Out->getID(FE));
-      }
       break;
     }
     case PPCallbacks::RenameFile:
@@ -127,47 +113,6 @@ class IncludeStructure::RecordHeaders : public PPCallbacks,
     }
   }
 
-  bool HandleComment(Preprocessor &PP, SourceRange Range) override {
-    auto Pragma =
-        tooling::parseIWYUPragma(SM.getCharacterData(Range.getBegin()));
-    if (!Pragma)
-      return false;
-
-    if (inMainFile()) {
-      // Given:
-      //
-      // #include "foo.h"
-      // #include "bar.h" // IWYU pragma: keep
-      //
-      // The order in which the callbacks will be triggered:
-      //
-      // 1. InclusionDirective("foo.h")
-      // 2. handleCommentInMainFile("// IWYU pragma: keep")
-      // 3. InclusionDirective("bar.h")
-      //
-      // This code stores the last location of "IWYU pragma: keep" (or export)
-      // comment in the main file, so that when InclusionDirective is called, it
-      // will know that the next inclusion is behind the IWYU pragma.
-      // FIXME: Support "IWYU pragma: begin_exports" and "IWYU pragma:
-      // end_exports".
-      if (!Pragma->startswith("export") && !Pragma->startswith("keep"))
-        return false;
-      unsigned Offset = SM.getFileOffset(Range.getBegin());
-      LastPragmaKeepInMainFileLine =
-          SM.getLineNumber(SM.getMainFileID(), Offset) - 1;
-    } else {
-      // Memorize headers that have export pragmas in them. Include Cleaner
-      // does not support them properly yet, so they will be not marked as
-      // unused.
-      // FIXME: Once IncludeCleaner supports export pragmas, remove this.
-      if (!Pragma->startswith("export") && !Pragma->startswith("begin_exports"))
-        return false;
-      Out->HasIWYUExport.insert(
-          *Out->getID(SM.getFileEntryForID(SM.getFileID(Range.getBegin()))));
-    }
-    return false;
-  }
-
 private:
   // Keeps track of include depth for the current file. It's 1 for main file.
   int Level = 0;
@@ -181,9 +126,6 @@ class IncludeStructure::RecordHeaders : public PPCallbacks,
   bool InBuiltinFile = false;
 
   IncludeStructure *Out;
-
-  // The last line "IWYU pragma: keep" was seen in the main file, 0-indexed.
-  int LastPragmaKeepInMainFileLine = -1;
 };
 
 bool isLiteralInclude(llvm::StringRef Include) {
@@ -234,7 +176,6 @@ void IncludeStructure::collect(const CompilerInstance &CI) {
   auto &SM = CI.getSourceManager();
   MainFileEntry = SM.getFileEntryForID(SM.getMainFileID());
   auto Collector = std::make_unique<RecordHeaders>(CI, this);
-  CI.getPreprocessor().addCommentHandler(Collector.get());
   CI.getPreprocessor().addPPCallbacks(std::move(Collector));
 }
 
@@ -321,7 +262,7 @@ bool IncludeInserter::shouldInsertInclude(
   if (FileName == DeclaringHeader || FileName == InsertedHeader.File)
     return false;
   auto Included = [&](llvm::StringRef Header) {
-    return IncludedHeaders.find(Header) != IncludedHeaders.end();
+    return IncludedHeaders.contains(Header);
   };
   return !Included(DeclaringHeader) && !Included(InsertedHeader.File);
 }
diff --git a/clang-tools-extra/clangd/Headers.h b/clang-tools-extra/clangd/Headers.h
index a21ea851831aa..f6aeab1dbfd35 100644
--- a/clang-tools-extra/clangd/Headers.h
+++ b/clang-tools-extra/clangd/Headers.h
@@ -73,7 +73,6 @@ struct Inclusion {
   int HashLine = 0;        // Line number containing the directive, 0-indexed.
   SrcMgr::CharacteristicKind FileKind = SrcMgr::C_User;
   std::optional<unsigned> HeaderID;
-  bool BehindPragmaKeep = false; // Has IWYU pragma: keep right after.
 };
 llvm::raw_ostream &operator<<(llvm::raw_ostream &, const Inclusion &);
 bool operator==(const Inclusion &LHS, const Inclusion &RHS);
@@ -151,12 +150,6 @@ class IncludeStructure {
     return RealPathNames[static_cast<unsigned>(ID)];
   }
 
-  bool isSelfContained(HeaderID ID) const {
-    return !NonSelfContained.contains(ID);
-  }
-
-  bool hasIWYUExport(HeaderID ID) const { return HasIWYUExport.contains(ID); }
-
   // Return all transitively reachable files.
   llvm::ArrayRef<std::string> allHeaders() const { return RealPathNames; }
 
@@ -199,12 +192,6 @@ class IncludeStructure {
   // and RealPathName and UniqueID are not preserved in
   // the preamble.
   llvm::DenseMap<llvm::sys::fs::UniqueID, HeaderID> UIDToIndex;
-  // Contains HeaderIDs of all non self-contained entries in the
-  // IncludeStructure.
-  llvm::DenseSet<HeaderID> NonSelfContained;
-  // Contains a set of headers that have either "IWYU pragma: export" or "IWYU
-  // pragma: begin_exports".
-  llvm::DenseSet<HeaderID> HasIWYUExport;
 
   // Maps written includes to indices in MainFileInclude for easier lookup by
   // spelling.
diff --git a/clang-tools-extra/clangd/IncludeCleaner.cpp b/clang-tools-extra/clangd/IncludeCleaner.cpp
index 7d523d9a63641..98135529f259b 100644
--- a/clang-tools-extra/clangd/IncludeCleaner.cpp
+++ b/clang-tools-extra/clangd/IncludeCleaner.cpp
@@ -8,32 +8,53 @@
 
 #include "IncludeCleaner.h"
 #include "Config.h"
+#include "Diagnostics.h"
 #include "Headers.h"
 #include "ParsedAST.h"
 #include "Protocol.h"
 #include "SourceCode.h"
+#include "URI.h"
 #include "clang-include-cleaner/Analysis.h"
+#include "clang-include-cleaner/Record.h"
 #include "clang-include-cleaner/Types.h"
-#include "index/CanonicalIncludes.h"
 #include "support/Logger.h"
+#include "support/Path.h"
 #include "support/Trace.h"
 #include "clang/AST/ASTContext.h"
+#include "clang/AST/DeclCXX.h"
+#include "clang/AST/Expr.h"
 #include "clang/AST/ExprCXX.h"
-#include "clang/AST/RecursiveASTVisitor.h"
+#include "clang/AST/TemplateName.h"
+#include "clang/AST/Type.h"
+#include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/LLVM.h"
 #include "clang/Basic/SourceLocation.h"
 #include "clang/Basic/SourceManager.h"
+#include "clang/Format/Format.h"
 #include "clang/Lex/HeaderSearch.h"
 #include "clang/Lex/Preprocessor.h"
+#include "clang/Tooling/Core/Replacement.h"
+#include "clang/Tooling/Inclusions/HeaderIncludes.h"
+#include "clang/Tooling/Inclusions/StandardLibrary.h"
 #include "clang/Tooling/Syntax/Tokens.h"
 #include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseSet.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/STLFunctionalExtras.h"
 #include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/StringSet.h"
+#include "llvm/Support/Casting.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Regex.h"
-#include <functional>
+#include <iterator>
 #include <optional>
+#include <string>
+#include <vector>
 
 namespace clang {
 namespace clangd {
@@ -43,181 +64,6 @@ void setIncludeCleanerAnalyzesStdlib(bool B) { AnalyzeStdlib = B; }
 
 namespace {
 
-/// Crawler traverses the AST and feeds in the locations of (sometimes
-/// implicitly) used symbols into \p Result.
-class ReferencedLocationCrawler
-    : public RecursiveASTVisitor<ReferencedLocationCrawler> {
-public:
-  ReferencedLocationCrawler(ReferencedLocations &Result,
-                            const SourceManager &SM)
-      : Result(Result), SM(SM) {}
-
-  bool VisitDeclRefExpr(DeclRefExpr *DRE) {
-    add(DRE->getDecl());
-    add(DRE->getFoundDecl());
-    return true;
-  }
-
-  bool VisitMemberExpr(MemberExpr *ME) {
-    add(ME->getMemberDecl());
-    add(ME->getFoundDecl().getDecl());
-    return true;
-  }
-
-  bool VisitTagType(TagType *TT) {
-    add(TT->getDecl());
-    return true;
-  }
-
-  bool VisitFunctionDecl(FunctionDecl *FD) {
-    // Function definition will require redeclarations to be included.
-    if (FD->isThisDeclarationADefinition())
-      add(FD);
-    return true;
-  }
-
-  bool VisitCXXConstructExpr(CXXConstructExpr *CCE) {
-    add(CCE->getConstructor());
-    return true;
-  }
-
-  bool VisitTemplateSpecializationType(TemplateSpecializationType *TST) {
-    // Using templateName case is handled by the override TraverseTemplateName.
-    if (TST->getTemplateName().getKind() == TemplateName::UsingTemplate)
-      return true;
-    add(TST->getAsCXXRecordDecl()); // Specialization
-    return true;
-  }
-
-  // There is no VisitTemplateName in RAV, thus we override the Traverse version
-  // to handle the Using TemplateName case.
-  bool TraverseTemplateName(TemplateName TN) {
-    VisitTemplateName(TN);
-    return Base::TraverseTemplateName(TN);
-  }
-  // A pseudo VisitTemplateName, dispatched by the above TraverseTemplateName!
-  bool VisitTemplateName(TemplateName TN) {
-    if (const auto *USD = TN.getAsUsingShadowDecl()) {
-      add(USD);
-      return true;
-    }
-    add(TN.getAsTemplateDecl()); // Primary template.
-    return true;
-  }
-
-  bool VisitUsingType(UsingType *UT) {
-    add(UT->getFoundDecl());
-    return true;
-  }
-
-  bool VisitTypedefType(TypedefType *TT) {
-    add(TT->getDecl());
-    return true;
-  }
-
-  // Consider types of any subexpression used, even if the type is not named.
-  // This is helpful in getFoo().bar(), where Foo must be complete.
-  // FIXME(kirillbobyrev): Should we tweak this? It may not be desirable to
-  // consider types "used" when they are not directly spelled in code.
-  bool VisitExpr(Expr *E) {
-    TraverseType(E->getType());
-    return true;
-  }
-
-  bool TraverseType(QualType T) {
-    if (isNew(T.getTypePtrOrNull())) // don't care about quals
-      Base::TraverseType(T);
-    return true;
-  }
-
-  bool VisitUsingDecl(UsingDecl *D) {
-    for (const auto *Shadow : D->shadows())
-      add(Shadow->getTargetDecl());
-    return true;
-  }
-
-  // Enums may be usefully forward-declared as *complete* types by specifying
-  // an underlying type. In this case, the definition should see the declaration
-  // so they can be checked for compatibility.
-  bool VisitEnumDecl(EnumDecl *D) {
-    if (D->isThisDeclarationADefinition() && D->getIntegerTypeSourceInfo())
-      add(D);
-    return true;
-  }
-
-  // When the overload is not resolved yet, mark all candidates as used.
-  bool VisitOverloadExpr(OverloadExpr *E) {
-    for (const auto *ResolutionDecl : E->decls())
-      add(ResolutionDecl);
-    return true;
-  }
-
-private:
-  using Base = RecursiveASTVisitor<ReferencedLocationCrawler>;
-
-  void add(const Decl *D) {
-    if (!D || !isNew(D->getCanonicalDecl()))
-      return;
-    if (auto SS = StdRecognizer(D)) {
-      Result.Stdlib.insert(*SS);
-      return;
-    }
-    // Special case RecordDecls, as it is common for them to be forward
-    // declared multiple times. The most common cases are:
-    // - Definition available in TU, only mark that one as usage. The rest is
-    //   likely to be unnecessary. This might result in false positives when an
-    //   internal definition is visible.
-    // - There's a forward declaration in the main file, no need for other
-    //   redecls.
-    if (const auto *RD = llvm::dyn_cast<RecordDecl>(D)) {
-      if (const auto *Definition = RD->getDefinition()) {
-        Result.User.insert(Definition->getLocation());
-        return;
-      }
-      if (SM.isInMainFile(RD->getMostRecentDecl()->getLocation()))
-        return;
-    }
-    for (const Decl *Redecl : D->redecls())
-      Result.User.insert(Redecl->getLocation());
-  }
-
-  bool isNew(const void *P) { return P && Visited.insert(P).second; }
-
-  ReferencedLocations &Result;
-  llvm::DenseSet<const void *> Visited;
-  const SourceManager &SM;
-  tooling::stdlib::Recognizer StdRecognizer;
-};
-
-// Given a set of referenced FileIDs, determines all the potentially-referenced
-// files and macros by traversing expansion/spelling locations of macro IDs.
-// This is used to map the referenced SourceLocations onto real files.
-struct ReferencedFilesBuilder {
-  ReferencedFilesBuilder(const SourceManager &SM) : SM(SM) {}
-  llvm::DenseSet<FileID> Files;
-  llvm::DenseSet<FileID> Macros;
-  const SourceManager &SM;
-
-  void add(SourceLocation Loc) { add(SM.getFileID(Loc), Loc); }
-
-  void add(FileID FID, SourceLocation Loc) {
-    if (FID.isInvalid())
-      return;
-    assert(SM.isInFileID(Loc, FID));
-    if (Loc.isFileID()) {
-      Files.insert(FID);
-      return;
-    }
-    // Don't process the same macro FID twice.
-    if (!Macros.insert(FID).second)
-      return;
-    const auto &Exp = SM.getSLocEntry(FID).getExpansion();
-    add(Exp.getSpellingLoc());
-    add(Exp.getExpansionLocStart());
-    add(Exp.getExpansionLocEnd());
-  }
-};
-
 // Returns the range starting at '#' and ending at EOL. Escaped newlines are not
 // handled.
 clangd::Range getDiagnosticRange(llvm::StringRef Code, unsigned HashOffset) {
@@ -232,37 +78,23 @@ clangd::Range getDiagnosticRange(llvm::StringRef Code, unsigned HashOffset) {
   return Result;
 }
 
-// Finds locations of macros referenced from within the main file. That includes
-// references that were not yet expanded, e.g `BAR` in `#define FOO BAR`.
-void findReferencedMacros(const SourceManager &SM, Preprocessor &PP,
-                          const syntax::TokenBuffer *Tokens,
-                          ReferencedLocations &Result) {
-  trace::Span Tracer("IncludeCleaner::findReferencedMacros");
-  // FIXME(kirillbobyrev): The macros from the main file are collected in
-  // ParsedAST's MainFileMacros. However, we can't use it here because it
-  // doesn't handle macro references that were not expanded, e.g. in macro
-  // definitions or preprocessor-disabled sections.
-  //
-  // Extending MainFileMacros to collect missing references and switching to
-  // this mechanism (as opposed to iterating through all tokens) will improve
-  // the performance of findReferencedMacros and also improve other features
-  // relying on MainFileMacros.
-  for (const syntax::Token &Tok : Tokens->spelledTokens(SM.getMainFileID())) {
-    auto Macro = locateMacroAt(Tok, PP);
-    if (!Macro)
-      continue;
-    auto Loc = Macro->Info->getDefinitionLoc();
-    if (Loc.isValid())
-      Result.User.insert(Loc);
-    // FIXME: support stdlib macros
+
+bool isFilteredByConfig(const Config &Cfg, llvm::StringRef HeaderPath) {
+  // Convert the path to Unix slashes and try to match against the filter.
+  llvm::SmallString<64> NormalizedPath(HeaderPath);
+  llvm::sys::path::native(NormalizedPath, llvm::sys::path::Style::posix);
+  for (auto &Filter : Cfg.Diagnostics.Includes.IgnoreHeader) {
+    if (Filter(NormalizedPath))
+      return true;
   }
+  return false;
 }
 
 static bool mayConsiderUnused(const Inclusion &Inc, ParsedAST &AST,
-                              const Config &Cfg) {
-  if (Inc.BehindPragmaKeep)
+                              const Config &Cfg,
+                              const include_cleaner::PragmaIncludes *PI) {
+  if (PI && PI->shouldKeep(Inc.HashLine + 1))
     return false;
-
   // FIXME(kirillbobyrev): We currently do not support the umbrella headers.
   // System headers are likely to be standard library headers.
   // Until we have good support for umbrella headers, don't warn about them.
@@ -273,10 +105,6 @@ static bool mayConsiderUnused(const Inclusion &Inc, ParsedAST &AST,
   }
   assert(Inc.HeaderID);
   auto HID = static_cast<IncludeStructure::HeaderID>(*Inc.HeaderID);
-  // FIXME: Ignore the headers with IWYU export pragmas for now, remove this
-  // check when we have more precise tracking of exported headers.
-  if (AST.getIncludeStructure().hasIWYUExport(HID))
-    return false;
   auto FE = AST.getSourceManager().getFileManager().getFileRef(
       AST.getIncludeStructure().getRealPath(HID));
   assert(FE);
@@ -288,120 +116,205 @@ static bool mayConsiderUnused(const Inclusion &Inc, ParsedAST &AST,
          FE->getName());
     return false;
   }
-  for (auto &Filter : Cfg.Diagnostics.Includes.IgnoreHeader) {
-    // Convert the path to Unix slashes and try to match against the filter.
-    llvm::SmallString<64> Path(Inc.Resolved);
-    llvm::sys::path::native(Path, llvm::sys::path::Style::posix);
-    if (Filter(Path)) {
-      dlog("{0} header is filtered out by the configuration", FE->getName());
-      return false;
-    }
+
+  if (isFilteredByConfig(Cfg, Inc.Resolved)) {
+    dlog("{0} header is filtered out by the configuration", FE->getName());
+    return false;
   }
   return true;
 }
 
-// In case symbols are coming from non self-contained header, we need to find
-// its first includer that is self-contained. This is the header users can
-// include, so it will be responsible for bringing the symbols from given
-// header into the scope.
-FileID headerResponsible(FileID ID, const SourceManager &SM,
-                         const IncludeStructure &Includes) {
-  // Unroll the chain of non self-contained headers until we find the one that
-  // can be included.
-  for (const FileEntry *FE = SM.getFileEntryForID(ID); ID != SM.getMainFileID();
-       FE = SM.getFileEntryForID(ID)) {
-    // If FE is nullptr, we consider it to be the responsible header.
-    if (!FE)
-      break;
-    auto HID = Includes.getID(FE);
-    assert(HID && "We're iterating over headers already existing in "
-                  "IncludeStructure");
-    if (Includes.isSelfContained(*HID))
-      break;
-    // The header is not self-contained: put the responsibility for its symbols
-    // on its includer.
-    ID = SM.getFileID(SM.getIncludeLoc(ID));
+include_cleaner::Includes
+convertIncludes(const SourceManager &SM,
+                const llvm::ArrayRef<Inclusion> MainFileIncludes) {
+  include_cleaner::Includes Includes;
+  for (const Inclusion &Inc : MainFileIncludes) {
+    include_cleaner::Include TransformedInc;
+    llvm::StringRef WrittenRef = llvm::StringRef(Inc.Written);
+    TransformedInc.Spelled = WrittenRef.trim("\"<>");
+    TransformedInc.HashLocation =
+        SM.getComposedLoc(SM.getMainFileID(), Inc.HashOffset);
+    TransformedInc.Line = Inc.HashLine + 1;
+    TransformedInc.Angled = WrittenRef.starts_with("<");
+    auto FE = SM.getFileManager().getFile(Inc.Resolved);
+    if (!FE) {
+      elog("IncludeCleaner: Failed to get an entry for resolved path {0}: {1}",
+           Inc.Resolved, FE.getError().message());
+      continue;
+    }
+    TransformedInc.Resolved = *FE;
+    Includes.add(std::move(TransformedInc));
   }
-  return ID;
+  return Includes;
 }
 
-} // namespace
+std::string spellHeader(ParsedAST &AST, const FileEntry *MainFile,
+                        include_cleaner::Header Provider) {
+  if (Provider.kind() == include_cleaner::Header::Physical) {
+    if (auto CanonicalPath =
+            getCanonicalPath(Provider.physical(), AST.getSourceManager())) {
+      std::string SpelledHeader =
+          llvm::cantFail(URI::includeSpelling(URI::create(*CanonicalPath)));
+      if (!SpelledHeader.empty())
+        return SpelledHeader;
+    }
+  }
+  return include_cleaner::spellHeader(
+      Provider, AST.getPreprocessor().getHeaderSearchInfo(), MainFile);
+}
 
-ReferencedLocations findReferencedLocations(ASTContext &Ctx, Preprocessor &PP,
-                                            const syntax::TokenBuffer *Tokens) {
-  trace::Span Tracer("IncludeCleaner::findReferencedLocations");
-  ReferencedLocations Result;
-  const auto &SM = Ctx.getSourceManager();
-  ReferencedLocationCrawler Crawler(Result, SM);
-  Crawler.TraverseAST(Ctx);
-  if (Tokens)
-    findReferencedMacros(SM, PP, Tokens, Result);
-  return Result;
+std::vector<include_cleaner::SymbolReference>
+collectMacroReferences(ParsedAST &AST) {
+  const auto &SM = AST.getSourceManager();
+  //  FIXME: !!this is a hacky way to collect macro references.
+  std::vector<include_cleaner::SymbolReference> Macros;
+  auto &PP = AST.getPreprocessor();
+  for (const syntax::Token &Tok :
+       AST.getTokens().spelledTokens(SM.getMainFileID())) {
+    auto Macro = locateMacroAt(Tok, PP);
+    if (!Macro)
+      continue;
+    if (auto DefLoc = Macro->Info->getDefinitionLoc(); DefLoc.isValid())
+      Macros.push_back(
+          {Tok.location(),
+           include_cleaner::Macro{/*Name=*/PP.getIdentifierInfo(Tok.text(SM)),
+                                  DefLoc},
+           include_cleaner::RefType::Explicit});
+  }
+  return Macros;
+}
+
+llvm::StringRef getResolvedPath(const include_cleaner::Header &SymProvider) {
+  switch (SymProvider.kind()) {
+  case include_cleaner::Header::Physical:
+    return SymProvider.physical()->tryGetRealPathName();
+  case include_cleaner::Header::Standard:
+    return SymProvider.standard().name().trim("<>\"");
+  case include_cleaner::Header::Verbatim:
+    return SymProvider.verbatim().trim("<>\"");
+  }
+  llvm_unreachable("Unknown header kind");
 }
 
-ReferencedLocations findReferencedLocations(ParsedAST &AST) {
-  return findReferencedLocations(AST.getASTContext(), AST.getPreprocessor(),
-                                 &AST.getTokens());
+std::string getSymbolName(const include_cleaner::Symbol &Sym) {
+  switch (Sym.kind()) {
+  case include_cleaner::Symbol::Macro:
+    return Sym.macro().Name->getName().str();
+  case include_cleaner::Symbol::Declaration:
+    return llvm::dyn_cast<NamedDecl>(&Sym.declaration())
+        ->getQualifiedNameAsString();
+  }
+  llvm_unreachable("Unknown symbol kind");
 }
 
-ReferencedFiles findReferencedFiles(
-    const ReferencedLocations &Locs, const SourceManager &SM,
-    llvm::function_ref<FileID(FileID)> HeaderResponsible,
-    llvm::function_ref<std::optional<StringRef>(FileID)> UmbrellaHeader) {
-  std::vector<SourceLocation> Sorted{Locs.User.begin(), Locs.User.end()};
-  llvm::sort(Sorted); // Group by FileID.
-  ReferencedFilesBuilder Builder(SM);
-  for (auto It = Sorted.begin(); It < Sorted.end();) {
-    FileID FID = SM.getFileID(*It);
-    Builder.add(FID, *It);
-    // Cheaply skip over all the other locations from the same FileID.
-    // This avoids lots of redundant Loc->File lookups for the same file.
-    do
-      ++It;
-    while (It != Sorted.end() && SM.isInFileID(*It, FID));
+std::vector<Diag> generateMissingIncludeDiagnostics(
+    ParsedAST &AST, llvm::ArrayRef<MissingIncludeDiagInfo> MissingIncludes,
+    llvm::StringRef Code) {
+  std::vector<Diag> Result;
+  const Config &Cfg = Config::current();
+  if (Cfg.Diagnostics.MissingIncludes != Config::IncludesPolicy::Strict ||
+      Cfg.Diagnostics.SuppressAll ||
+      Cfg.Diagnostics.Suppress.contains("missing-includes")) {
+    return Result;
   }
 
-  // If a header is not self-contained, we consider its symbols a logical part
-  // of the including file. Therefore, mark the parents of all used
-  // non-self-contained FileIDs as used. Perform this on FileIDs rather than
-  // HeaderIDs, as each inclusion of a non-self-contained file is distinct.
-  llvm::DenseSet<FileID> UserFiles;
-  llvm::StringSet<> PublicHeaders;
-  for (FileID ID : Builder.Files) {
-    UserFiles.insert(HeaderResponsible(ID));
-    if (auto PublicHeader = UmbrellaHeader(ID)) {
-      PublicHeaders.insert(*PublicHeader);
-    }
+  const SourceManager &SM = AST.getSourceManager();
+  const FileEntry *MainFile = SM.getFileEntryForID(SM.getMainFileID());
+
+  auto FileStyle = format::getStyle(
+      format::DefaultFormatStyle, AST.tuPath(), format::DefaultFallbackStyle,
+      Code, &SM.getFileManager().getVirtualFileSystem());
+  if (!FileStyle) {
+    elog("Couldn't infer style", FileStyle.takeError());
+    FileStyle = format::getLLVMStyle();
   }
 
-  llvm::DenseSet<tooling::stdlib::Header> StdlibFiles;
-  for (const auto &Symbol : Locs.Stdlib)
-    for (const auto &Header : Symbol.headers())
-      StdlibFiles.insert(Header);
+  tooling::HeaderIncludes HeaderIncludes(AST.tuPath(), Code,
+                                         FileStyle->IncludeStyle);
+  for (const auto &SymbolWithMissingInclude : MissingIncludes) {
+    llvm::StringRef ResolvedPath =
+        getResolvedPath(SymbolWithMissingInclude.Providers.front());
+    if (isFilteredByConfig(Cfg, ResolvedPath)) {
+      dlog("IncludeCleaner: not diagnosing missing include {0}, filtered by "
+           "config",
+           ResolvedPath);
+      continue;
+    }
+
+    std::string Spelling =
+        spellHeader(AST, MainFile, SymbolWithMissingInclude.Providers.front());
+    llvm::StringRef HeaderRef{Spelling};
+    bool Angled = HeaderRef.starts_with("<");
+    // We might suggest insertion of an existing include in edge cases, e.g.,
+    // include is present in a PP-disabled region, or spelling of the header
+    // turns out to be the same as one of the unresolved includes in the
+    // main file.
+    std::optional<tooling::Replacement> Replacement = HeaderIncludes.insert(
+        HeaderRef.trim("\"<>"), Angled, tooling::IncludeDirective::Include);
+    if (!Replacement.has_value())
+      continue;
 
-  return {std::move(UserFiles), std::move(StdlibFiles),
-          std::move(PublicHeaders)};
+    Diag &D = Result.emplace_back();
+    D.Message =
+        llvm::formatv("No header providing \"{0}\" is directly included",
+                      getSymbolName(SymbolWithMissingInclude.Symbol));
+    D.Name = "missing-includes";
+    D.Source = Diag::DiagSource::Clangd;
+    D.File = AST.tuPath();
+    D.InsideMainFile = true;
+    D.Severity = DiagnosticsEngine::Warning;
+    D.Range = clangd::Range{
+        offsetToPosition(Code,
+                         SymbolWithMissingInclude.SymRefRange.beginOffset()),
+        offsetToPosition(Code,
+                         SymbolWithMissingInclude.SymRefRange.endOffset())};
+    auto &F = D.Fixes.emplace_back();
+    F.Message = "#include " + Spelling;
+    TextEdit Edit = replacementToEdit(Code, *Replacement);
+    F.Edits.emplace_back(std::move(Edit));
+  }
+  return Result;
 }
 
-ReferencedFiles findReferencedFiles(const ReferencedLocations &Locs,
-                                    const IncludeStructure &Includes,
-                                    const CanonicalIncludes &CanonIncludes,
-                                    const SourceManager &SM) {
-  return findReferencedFiles(
-      Locs, SM,
-      [&SM, &Includes](FileID ID) {
-        return headerResponsible(ID, SM, Includes);
-      },
-      [&SM, &CanonIncludes](FileID ID) -> std::optional<StringRef> {
-        auto Entry = SM.getFileEntryRefForID(ID);
-        if (!Entry)
-          return std::nullopt;
-        auto PublicHeader = CanonIncludes.mapHeader(*Entry);
-        if (PublicHeader.empty())
-          return std::nullopt;
-        return PublicHeader;
-      });
+std::vector<Diag> generateUnusedIncludeDiagnostics(
+    PathRef FileName, llvm::ArrayRef<const Inclusion *> UnusedIncludes,
+    llvm::StringRef Code) {
+  std::vector<Diag> Result;
+  const Config &Cfg = Config::current();
+  if (Cfg.Diagnostics.UnusedIncludes == Config::IncludesPolicy::None ||
+      Cfg.Diagnostics.SuppressAll ||
+      Cfg.Diagnostics.Suppress.contains("unused-includes")) {
+    return Result;
+  }
+  for (const auto *Inc : UnusedIncludes) {
+    Diag &D = Result.emplace_back();
+    D.Message =
+        llvm::formatv("included header {0} is not used directly",
+                      llvm::sys::path::filename(
+                          Inc->Written.substr(1, Inc->Written.size() - 2),
+                          llvm::sys::path::Style::posix));
+    D.Name = "unused-includes";
+    D.Source = Diag::DiagSource::Clangd;
+    D.File = FileName;
+    D.InsideMainFile = true;
+    D.Severity = DiagnosticsEngine::Warning;
+    D.Tags.push_back(Unnecessary);
+    D.Range = getDiagnosticRange(Code, Inc->HashOffset);
+    // FIXME(kirillbobyrev): Removing inclusion might break the code if the
+    // used headers are only reachable transitively through this one. Suggest
+    // including them directly instead.
+    // FIXME(kirillbobyrev): Add fix suggestion for adding IWYU pragmas
+    // (keep/export) remove the warning once we support IWYU pragmas.
+    auto &F = D.Fixes.emplace_back();
+    F.Message = "remove #include directive";
+    F.Edits.emplace_back();
+    F.Edits.back().range.start.line = Inc->HashLine;
+    F.Edits.back().range.end.line = Inc->HashLine + 1;
+  }
+  return Result;
 }
+} // namespace
+
 
 std::vector<const Inclusion *>
 getUnused(ParsedAST &AST,
@@ -417,7 +330,7 @@ getUnused(ParsedAST &AST,
       continue;
     auto IncludeID = static_cast<IncludeStructure::HeaderID>(*MFI.HeaderID);
     bool Used = ReferencedFiles.contains(IncludeID);
-    if (!Used && !mayConsiderUnused(MFI, AST, Cfg)) {
+    if (!Used && !mayConsiderUnused(MFI, AST, Cfg, AST.getPragmaIncludes())) {
       dlog("{0} was not used, but is not eligible to be diagnosed as unused",
            MFI.Written);
       continue;
@@ -429,150 +342,80 @@ getUnused(ParsedAST &AST,
   return Unused;
 }
 
-#ifndef NDEBUG
-// Is FID a <built-in>, <scratch space> etc?
-static bool isSpecialBuffer(FileID FID, const SourceManager &SM) {
-  const SrcMgr::FileInfo &FI = SM.getSLocEntry(FID).getFile();
-  return FI.getName().startswith("<");
-}
-#endif
-
-llvm::DenseSet<IncludeStructure::HeaderID>
-translateToHeaderIDs(const ReferencedFiles &Files,
-                     const IncludeStructure &Includes,
-                     const SourceManager &SM) {
-  trace::Span Tracer("IncludeCleaner::translateToHeaderIDs");
-  llvm::DenseSet<IncludeStructure::HeaderID> TranslatedHeaderIDs;
-  TranslatedHeaderIDs.reserve(Files.User.size());
-  for (FileID FID : Files.User) {
-    const FileEntry *FE = SM.getFileEntryForID(FID);
-    if (!FE) {
-      assert(isSpecialBuffer(FID, SM));
-      continue;
-    }
-    const auto File = Includes.getID(FE);
-    assert(File);
-    TranslatedHeaderIDs.insert(*File);
-  }
-  for (tooling::stdlib::Header StdlibUsed : Files.Stdlib)
-    for (auto HID : Includes.StdlibHeaders.lookup(StdlibUsed))
-      TranslatedHeaderIDs.insert(HID);
-  return TranslatedHeaderIDs;
-}
-
-// This is the original clangd-own implementation for computing unused
-// #includes. Eventually it will be deprecated and replaced by the
-// include-cleaner-lib-based implementation.
-std::vector<const Inclusion *> computeUnusedIncludes(ParsedAST &AST) {
-  const auto &SM = AST.getSourceManager();
-
-  auto Refs = findReferencedLocations(AST);
-  auto ReferencedFiles =
-      findReferencedFiles(Refs, AST.getIncludeStructure(),
-                          AST.getCanonicalIncludes(), AST.getSourceManager());
-  auto ReferencedHeaders =
-      translateToHeaderIDs(ReferencedFiles, AST.getIncludeStructure(), SM);
-  return getUnused(AST, ReferencedHeaders, ReferencedFiles.SpelledUmbrellas);
-}
-std::vector<const Inclusion *>
-computeUnusedIncludesExperimental(ParsedAST &AST) {
+IncludeCleanerFindings computeIncludeCleanerFindings(ParsedAST &AST) {
   const auto &SM = AST.getSourceManager();
   const auto &Includes = AST.getIncludeStructure();
+  include_cleaner::Includes ConvertedIncludes =
+      convertIncludes(SM, Includes.MainFileIncludes);
+  const FileEntry *MainFile = SM.getFileEntryForID(SM.getMainFileID());
 
-  // FIXME: !!this is a hacky way to collect macro references.
-  std::vector<include_cleaner::SymbolReference> Macros;
-  auto &PP = AST.getPreprocessor();
-  for (const syntax::Token &Tok :
-       AST.getTokens().spelledTokens(SM.getMainFileID())) {
-    auto Macro = locateMacroAt(Tok, PP);
-    if (!Macro)
-      continue;
-    if (auto DefLoc = Macro->Info->getDefinitionLoc(); DefLoc.isValid())
-      Macros.push_back(
-          {Tok.location(),
-           include_cleaner::Macro{/*Name=*/PP.getIdentifierInfo(Tok.text(SM)),
-                                  DefLoc},
-           include_cleaner::RefType::Explicit});
-  }
+  std::vector<include_cleaner::SymbolReference> Macros =
+      collectMacroReferences(AST);
+  std::vector<MissingIncludeDiagInfo> MissingIncludes;
   llvm::DenseSet<IncludeStructure::HeaderID> Used;
+  trace::Span Tracer("include_cleaner::walkUsed");
   include_cleaner::walkUsed(
       AST.getLocalTopLevelDecls(), /*MacroRefs=*/Macros,
       AST.getPragmaIncludes(), SM,
       [&](const include_cleaner::SymbolReference &Ref,
           llvm::ArrayRef<include_cleaner::Header> Providers) {
+        bool Satisfied = false;
         for (const auto &H : Providers) {
-          switch (H.kind()) {
-          case include_cleaner::Header::Physical:
-            if (auto HeaderID = Includes.getID(H.physical()))
-              Used.insert(*HeaderID);
-            break;
-          case include_cleaner::Header::Standard:
-            for (auto HeaderID : Includes.StdlibHeaders.lookup(H.standard()))
-              Used.insert(HeaderID);
-            break;
-          case include_cleaner::Header::Verbatim:
-            for (auto *Inc :
-                 Includes.mainFileIncludesWithSpelling(H.verbatim())) {
-              if (!Inc->HeaderID.has_value())
-                continue;
-              IncludeStructure::HeaderID ID =
-                  static_cast<IncludeStructure::HeaderID>(*Inc->HeaderID);
-              Used.insert(ID);
-            }
-            break;
+          if (H.kind() == include_cleaner::Header::Physical &&
+              H.physical() == MainFile) {
+            Satisfied = true;
+            continue;
+          }
+          for (auto *Inc : ConvertedIncludes.match(H)) {
+            Satisfied = true;
+            auto HeaderID = Includes.getID(Inc->Resolved);
+            assert(HeaderID.has_value() &&
+                   "ConvertedIncludes only contains resolved includes.");
+            Used.insert(*HeaderID);
           }
         }
+
+        if (Satisfied || Providers.empty() ||
+            Ref.RT != include_cleaner::RefType::Explicit)
+          return;
+
+        auto &Tokens = AST.getTokens();
+        auto SpelledForExpanded =
+            Tokens.spelledForExpanded(Tokens.expandedTokens(Ref.RefLocation));
+        if (!SpelledForExpanded)
+          return;
+
+        auto Range = syntax::Token::range(SM, SpelledForExpanded->front(),
+                                          SpelledForExpanded->back());
+        MissingIncludeDiagInfo DiagInfo{Ref.Target, Range, Providers};
+        MissingIncludes.push_back(std::move(DiagInfo));
       });
-  return getUnused(AST, Used, /*ReferencedPublicHeaders*/ {});
+  std::vector<const Inclusion *> UnusedIncludes =
+      getUnused(AST, Used, /*ReferencedPublicHeaders*/ {});
+  return {std::move(UnusedIncludes), std::move(MissingIncludes)};
 }
 
-std::vector<Diag> issueUnusedIncludesDiagnostics(ParsedAST &AST,
+std::vector<Diag> issueIncludeCleanerDiagnostics(ParsedAST &AST,
                                                  llvm::StringRef Code) {
-  const Config &Cfg = Config::current();
-  if (Cfg.Diagnostics.UnusedIncludes == Config::UnusedIncludesPolicy::None ||
-      Cfg.Diagnostics.SuppressAll ||
-      Cfg.Diagnostics.Suppress.contains("unused-includes"))
-    return {};
   // Interaction is only polished for C/CPP.
   if (AST.getLangOpts().ObjC)
     return {};
-  trace::Span Tracer("IncludeCleaner::issueUnusedIncludesDiagnostics");
-  std::vector<Diag> Result;
-  std::string FileName =
-      AST.getSourceManager()
-          .getFileEntryRefForID(AST.getSourceManager().getMainFileID())
-          ->getName()
-          .str();
-  const auto &UnusedIncludes =
-      Cfg.Diagnostics.UnusedIncludes == Config::UnusedIncludesPolicy::Experiment
-          ? computeUnusedIncludesExperimental(AST)
-          : computeUnusedIncludes(AST);
-  for (const auto *Inc : UnusedIncludes) {
-    Diag D;
-    D.Message =
-        llvm::formatv("included header {0} is not used directly",
-                      llvm::sys::path::filename(
-                          Inc->Written.substr(1, Inc->Written.size() - 2),
-                          llvm::sys::path::Style::posix));
-    D.Name = "unused-includes";
-    D.Source = Diag::DiagSource::Clangd;
-    D.File = FileName;
-    D.Severity = DiagnosticsEngine::Warning;
-    D.Tags.push_back(Unnecessary);
-    D.Range = getDiagnosticRange(Code, Inc->HashOffset);
-    // FIXME(kirillbobyrev): Removing inclusion might break the code if the
-    // used headers are only reachable transitively through this one. Suggest
-    // including them directly instead.
-    // FIXME(kirillbobyrev): Add fix suggestion for adding IWYU pragmas
-    // (keep/export) remove the warning once we support IWYU pragmas.
-    D.Fixes.emplace_back();
-    D.Fixes.back().Message = "remove #include directive";
-    D.Fixes.back().Edits.emplace_back();
-    D.Fixes.back().Edits.back().range.start.line = Inc->HashLine;
-    D.Fixes.back().Edits.back().range.end.line = Inc->HashLine + 1;
-    D.InsideMainFile = true;
-    Result.push_back(std::move(D));
+
+  trace::Span Tracer("IncludeCleaner::issueIncludeCleanerDiagnostics");
+
+  const Config &Cfg = Config::current();
+  IncludeCleanerFindings Findings;
+  if (Cfg.Diagnostics.MissingIncludes == Config::IncludesPolicy::Strict ||
+      Cfg.Diagnostics.UnusedIncludes == Config::IncludesPolicy::Strict) {
+    // will need include-cleaner results, call it once
+    Findings = computeIncludeCleanerFindings(AST);
   }
+
+  std::vector<Diag> Result = generateUnusedIncludeDiagnostics(
+      AST.tuPath(), Findings.UnusedIncludes, Code);
+  llvm::move(
+      generateMissingIncludeDiagnostics(AST, Findings.MissingIncludes, Code),
+      std::back_inserter(Result));
   return Result;
 }
 
diff --git a/clang-tools-extra/clangd/IncludeCleaner.h b/clang-tools-extra/clangd/IncludeCleaner.h
index a7beb9c3c9d4b..d7edca035c965 100644
--- a/clang-tools-extra/clangd/IncludeCleaner.h
+++ b/clang-tools-extra/clangd/IncludeCleaner.h
@@ -20,74 +20,32 @@
 
 #include "Headers.h"
 #include "ParsedAST.h"
-#include "index/CanonicalIncludes.h"
-#include "clang/Basic/SourceLocation.h"
-#include "clang/Tooling/Inclusions/StandardLibrary.h"
+#include "clang-include-cleaner/Types.h"
+#include "clang/Tooling/Syntax/Tokens.h"
 #include "llvm/ADT/DenseSet.h"
-#include "llvm/ADT/STLFunctionalExtras.h"
 #include "llvm/ADT/StringSet.h"
-#include <optional>
+#include <tuple>
 #include <vector>
 
 namespace clang {
 namespace clangd {
 
-struct ReferencedLocations {
-  llvm::DenseSet<SourceLocation> User;
-  llvm::DenseSet<tooling::stdlib::Symbol> Stdlib;
-};
-
-/// Finds locations of all symbols used in the main file.
-///
-/// - RecursiveASTVisitor finds references to symbols and records their
-///   associated locations. These may be macro expansions, and are not resolved
-///   to their spelling or expansion location. These locations are later used to
-///   determine which headers should be marked as "used" and "directly used".
-/// - If \p Tokens is not nullptr, we also examine all identifier tokens in the
-///   file in case they reference macros macros.
-/// We use this to compute unused headers, so we:
-///
-/// - cover the whole file in a single traversal for efficiency
-/// - don't attempt to describe where symbols were referenced from in
-///   ambiguous cases (e.g. implicitly used symbols, multiple declarations)
-/// - err on the side of reporting all possible locations
-ReferencedLocations findReferencedLocations(ASTContext &Ctx, Preprocessor &PP,
-                                            const syntax::TokenBuffer *Tokens);
-ReferencedLocations findReferencedLocations(ParsedAST &AST);
+// Data needed for missing include diagnostics.
+struct MissingIncludeDiagInfo {
+  include_cleaner::Symbol Symbol;
+  syntax::FileRange SymRefRange;
+  std::vector<include_cleaner::Header> Providers;
 
-struct ReferencedFiles {
-  llvm::DenseSet<FileID> User;
-  llvm::DenseSet<tooling::stdlib::Header> Stdlib;
-  /// Files responsible for the symbols referenced in the main file and defined
-  /// in private headers (private headers have IWYU pragma: private, include
-  /// "public.h"). We store spelling of the public header (with quotes or angle
-  /// brackets) files here to avoid dealing with full filenames and visibility.
-  llvm::StringSet<> SpelledUmbrellas;
+  bool operator==(const MissingIncludeDiagInfo &Other) const {
+    return std::tie(SymRefRange, Providers, Symbol) ==
+           std::tie(Other.SymRefRange, Other.Providers, Other.Symbol);
+  }
 };
 
-/// Retrieves IDs of all files containing SourceLocations from \p Locs.
-/// The output only includes things SourceManager sees as files (not macro IDs).
-/// This can include <built-in>, <scratch space> etc that are not true files.
-/// \p HeaderResponsible returns the public header that should be included given
-/// symbols from a file with the given FileID (example: public headers should be
-/// preferred to non self-contained and private headers).
-/// \p UmbrellaHeader returns the public public header is responsible for
-/// providing symbols from a file with the given FileID (example: MyType.h
-/// should be included instead of MyType_impl.h).
-ReferencedFiles findReferencedFiles(
-    const ReferencedLocations &Locs, const SourceManager &SM,
-    llvm::function_ref<FileID(FileID)> HeaderResponsible,
-    llvm::function_ref<std::optional<StringRef>(FileID)> UmbrellaHeader);
-ReferencedFiles findReferencedFiles(const ReferencedLocations &Locs,
-                                    const IncludeStructure &Includes,
-                                    const CanonicalIncludes &CanonIncludes,
-                                    const SourceManager &SM);
-
-/// Maps FileIDs to the internal IncludeStructure representation (HeaderIDs).
-/// FileIDs that are not true files (<built-in> etc) are dropped.
-llvm::DenseSet<IncludeStructure::HeaderID>
-translateToHeaderIDs(const ReferencedFiles &Files,
-                     const IncludeStructure &Includes, const SourceManager &SM);
+struct IncludeCleanerFindings {
+  std::vector<const Inclusion *> UnusedIncludes;
+  std::vector<MissingIncludeDiagInfo> MissingIncludes;
+};
 
 /// Retrieves headers that are referenced from the main file but not used.
 /// In unclear cases, headers are not marked as unused.
@@ -96,13 +54,9 @@ getUnused(ParsedAST &AST,
           const llvm::DenseSet<IncludeStructure::HeaderID> &ReferencedFiles,
           const llvm::StringSet<> &ReferencedPublicHeaders);
 
-std::vector<const Inclusion *> computeUnusedIncludes(ParsedAST &AST);
-// The same as computeUnusedIncludes, but it is an experimental and
-// include-cleaner-lib-based implementation.
-std::vector<const Inclusion *>
-computeUnusedIncludesExperimental(ParsedAST &AST);
+IncludeCleanerFindings computeIncludeCleanerFindings(ParsedAST &AST);
 
-std::vector<Diag> issueUnusedIncludesDiagnostics(ParsedAST &AST,
+std::vector<Diag> issueIncludeCleanerDiagnostics(ParsedAST &AST,
                                                  llvm::StringRef Code);
 
 /// Affects whether standard library includes should be considered for
diff --git a/clang-tools-extra/clangd/ParsedAST.cpp b/clang-tools-extra/clangd/ParsedAST.cpp
index 54a0f979c7303..1671eec133b6e 100644
--- a/clang-tools-extra/clangd/ParsedAST.cpp
+++ b/clang-tools-extra/clangd/ParsedAST.cpp
@@ -605,16 +605,15 @@ ParsedAST::build(llvm::StringRef Filename, const ParseInputs &Inputs,
   // Copy over the macros in the preamble region of the main file, and combine
   // with non-preamble macros below.
   MainFileMacros Macros;
-  if (Preamble)
-    Macros = Preamble->Macros;
+  std::vector<PragmaMark> Marks;
+  if (Preamble) {
+    Macros = Patch->mainFileMacros();
+    Marks = Patch->marks();
+  }
   Clang->getPreprocessor().addPPCallbacks(
       std::make_unique<CollectMainFileMacros>(Clang->getSourceManager(),
                                               Macros));
 
-  std::vector<PragmaMark> Marks;
-  // FIXME: We need to patch the marks for stale preambles.
-  if (Preamble)
-    Marks = Preamble->Marks;
   Clang->getPreprocessor().addPPCallbacks(
       collectPragmaMarksCallback(Clang->getSourceManager(), Marks));
 
@@ -687,13 +686,9 @@ ParsedAST::build(llvm::StringRef Filename, const ParseInputs &Inputs,
                    std::move(Macros), std::move(Marks), std::move(ParsedDecls),
                    std::move(Diags), std::move(Includes),
                    std::move(CanonIncludes));
-  if (Result.Diags) {
-    auto UnusedHeadersDiags =
-        issueUnusedIncludesDiagnostics(Result, Inputs.Contents);
-    Result.Diags->insert(Result.Diags->end(),
-                         make_move_iterator(UnusedHeadersDiags.begin()),
-                         make_move_iterator(UnusedHeadersDiags.end()));
-  }
+  if (Result.Diags)
+    llvm::move(issueIncludeCleanerDiagnostics(Result, Inputs.Contents),
+               std::back_inserter(*Result.Diags));
   return std::move(Result);
 }
 
diff --git a/clang-tools-extra/clangd/Preamble.cpp b/clang-tools-extra/clangd/Preamble.cpp
index f50597f9ad209..3b0af0ab50a62 100644
--- a/clang-tools-extra/clangd/Preamble.cpp
+++ b/clang-tools-extra/clangd/Preamble.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "Preamble.h"
+#include "CollectMacros.h"
 #include "Compiler.h"
 #include "Config.h"
 #include "Diagnostics.h"
@@ -14,7 +15,9 @@
 #include "Protocol.h"
 #include "SourceCode.h"
 #include "clang-include-cleaner/Record.h"
+#include "index/CanonicalIncludes.h"
 #include "support/Logger.h"
+#include "support/Path.h"
 #include "support/ThreadsafeFS.h"
 #include "support/Trace.h"
 #include "clang/AST/DeclTemplate.h"
@@ -26,6 +29,7 @@
 #include "clang/Basic/TokenKinds.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Frontend/FrontendActions.h"
+#include "clang/Frontend/PrecompiledPreamble.h"
 #include "clang/Lex/HeaderSearch.h"
 #include "clang/Lex/Lexer.h"
 #include "clang/Lex/PPCallbacks.h"
@@ -41,6 +45,7 @@
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Casting.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/FormatVariadic.h"
@@ -49,10 +54,12 @@
 #include "llvm/Support/VirtualFileSystem.h"
 #include "llvm/Support/raw_ostream.h"
 #include <cstddef>
+#include <functional>
 #include <memory>
 #include <optional>
 #include <string>
 #include <system_error>
+#include <tuple>
 #include <utility>
 #include <vector>
 
@@ -128,7 +135,9 @@ class CppFilePreambleCallbacks : public PreambleCallbacks {
     SourceMgr = &CI.getSourceManager();
     Includes.collect(CI);
     if (Config::current().Diagnostics.UnusedIncludes ==
-        Config::UnusedIncludesPolicy::Experiment)
+                Config::IncludesPolicy::Strict ||
+            Config::current().Diagnostics.MissingIncludes ==
+            Config::IncludesPolicy::Strict)
       Pragmas.record(CI);
     if (BeforeExecuteCallback)
       BeforeExecuteCallback(CI);
@@ -315,6 +324,8 @@ struct ScannedPreamble {
   // Literal lines of the preamble contents.
   std::vector<llvm::StringRef> Lines;
   PreambleBounds Bounds = {0, false};
+  std::vector<PragmaMark> Marks;
+  MainFileMacros Macros;
 };
 
 /// Scans the preprocessor directives in the preamble section of the file by
@@ -363,12 +374,15 @@ scanPreamble(llvm::StringRef Contents, const tooling::CompileCommand &Cmd) {
   if (!Action.BeginSourceFile(*Clang, Clang->getFrontendOpts().Inputs[0]))
     return error("failed BeginSourceFile");
   Preprocessor &PP = Clang->getPreprocessor();
+  const auto &SM = PP.getSourceManager();
   IncludeStructure Includes;
   Includes.collect(*Clang);
   ScannedPreamble SP;
   SP.Bounds = Bounds;
   PP.addPPCallbacks(
       std::make_unique<DirectiveCollector>(PP, SP.TextualDirectives));
+  PP.addPPCallbacks(collectPragmaMarksCallback(SM, SP.Marks));
+  PP.addPPCallbacks(std::make_unique<CollectMainFileMacros>(SM, SP.Macros));
   if (llvm::Error Err = Action.Execute())
     return std::move(Err);
   Action.EndSourceFile();
@@ -641,7 +655,7 @@ buildPreamble(PathRef FileName, CompilerInvocation CI,
   auto BuiltPreamble = PrecompiledPreamble::Build(
       CI, ContentsBuffer.get(), Bounds, *PreambleDiagsEngine,
       Stats ? TimedFS : StatCacheFS, std::make_shared<PCHContainerOperations>(),
-      StoreInMemory, CapturedInfo);
+      StoreInMemory, /*StoragePath=*/StringRef(), CapturedInfo);
   PreambleTimer.stopTimer();
 
   // When building the AST for the main file, we do want the function
@@ -763,6 +777,7 @@ PreamblePatch PreamblePatch::create(llvm::StringRef FileName,
     return PreamblePatch::unmodified(Baseline);
 
   PreamblePatch PP;
+  PP.Baseline = &Baseline;
   // This shouldn't coincide with any real file name.
   llvm::SmallString<128> PatchName;
   llvm::sys::path::append(PatchName, llvm::sys::path::parent_path(FileName),
@@ -845,6 +860,8 @@ PreamblePatch PreamblePatch::create(llvm::StringRef FileName,
   }
 
   PP.PatchedDiags = patchDiags(Baseline.Diags, *BaselineScan, *ModifiedScan);
+  PP.PatchedMarks = std::move(ModifiedScan->Marks);
+  PP.PatchedMacros = std::move(ModifiedScan->Macros);
   dlog("Created preamble patch: {0}", Patch.str());
   Patch.flush();
   return PP;
@@ -884,6 +901,7 @@ std::vector<Inclusion> PreamblePatch::preambleIncludes() const {
 
 PreamblePatch PreamblePatch::unmodified(const PreambleData &Preamble) {
   PreamblePatch PP;
+  PP.Baseline = &Preamble;
   PP.PreambleIncludes = Preamble.Includes.MainFileIncludes;
   PP.ModifiedBounds = Preamble.Preamble.getBounds();
   PP.PatchedDiags = Preamble.Diags;
@@ -894,5 +912,16 @@ bool PreamblePatch::preserveDiagnostics() const {
   return PatchContents.empty() ||
          Config::current().Diagnostics.AllowStalePreamble;
 }
+llvm::ArrayRef<PragmaMark> PreamblePatch::marks() const {
+  if (PatchContents.empty())
+    return Baseline->Marks;
+  return PatchedMarks;
+}
+
+const MainFileMacros &PreamblePatch::mainFileMacros() const {
+  if (PatchContents.empty())
+    return Baseline->Macros;
+  return PatchedMacros;
+}
 } // namespace clangd
 } // namespace clang
diff --git a/clang-tools-extra/clangd/Preamble.h b/clang-tools-extra/clangd/Preamble.h
index b5a5c107ab48a..a16f497737aa3 100644
--- a/clang-tools-extra/clangd/Preamble.h
+++ b/clang-tools-extra/clangd/Preamble.h
@@ -163,6 +163,9 @@ class PreamblePatch {
 
   static constexpr llvm::StringLiteral HeaderName = "__preamble_patch__.h";
 
+  llvm::ArrayRef<PragmaMark> marks() const;
+  const MainFileMacros &mainFileMacros() const;
+
 private:
   static PreamblePatch create(llvm::StringRef FileName,
                               const ParseInputs &Modified,
@@ -178,6 +181,9 @@ class PreamblePatch {
   // Diags that were attached to a line preserved in Modified contents.
   std::vector<Diag> PatchedDiags;
   PreambleBounds ModifiedBounds = {0, false};
+  const PreambleData *Baseline = nullptr;
+  std::vector<PragmaMark> PatchedMarks;
+  MainFileMacros PatchedMacros;
 };
 
 } // namespace clangd
diff --git a/clang-tools-extra/clangd/SemanticHighlighting.cpp b/clang-tools-extra/clangd/SemanticHighlighting.cpp
index 19f79f793420d..2122cb86c2834 100644
--- a/clang-tools-extra/clangd/SemanticHighlighting.cpp
+++ b/clang-tools-extra/clangd/SemanticHighlighting.cpp
@@ -156,7 +156,7 @@ std::optional<HighlightingKind> kindForDecl(const NamedDecl *D,
     return HighlightingKind::Concept;
   if (const auto *UUVD = dyn_cast<UnresolvedUsingValueDecl>(D)) {
     auto Targets = Resolver->resolveUsingValueDecl(UUVD);
-    if (!Targets.empty()) {
+    if (!Targets.empty() && Targets[0] != UUVD) {
       return kindForDecl(Targets[0], Resolver);
     }
     return HighlightingKind::Unknown;
diff --git a/clang-tools-extra/clangd/TUScheduler.cpp b/clang-tools-extra/clangd/TUScheduler.cpp
index a4f6a93b616a7..9b366cdd43eb2 100644
--- a/clang-tools-extra/clangd/TUScheduler.cpp
+++ b/clang-tools-extra/clangd/TUScheduler.cpp
@@ -948,7 +948,8 @@ void ASTWorker::update(ParseInputs Inputs, WantDiagnostics WantDiags,
     // rebuild. Newly built preamble cannot emit diagnostics before this call
     // finishes (ast callbacks are called from astpeer thread), hence we
     // gurantee eventual consistency.
-    if (LatestPreamble && Config::current().Diagnostics.AllowStalePreamble)
+    if (LatestPreamble && WantDiags != WantDiagnostics::No &&
+        Config::current().Diagnostics.AllowStalePreamble)
       generateDiagnostics(std::move(Invocation), std::move(Inputs),
                           std::move(CompilerInvocationDiags));
 
diff --git a/clang-tools-extra/clangd/index/SymbolCollector.cpp b/clang-tools-extra/clangd/index/SymbolCollector.cpp
index 24d01043a42d6..3179810b1b185 100644
--- a/clang-tools-extra/clangd/index/SymbolCollector.cpp
+++ b/clang-tools-extra/clangd/index/SymbolCollector.cpp
@@ -756,7 +756,8 @@ bool SymbolCollector::handleMacroOccurrence(const IdentifierInfo *Name,
       *PP, *CompletionAllocator, *CompletionTUInfo);
   std::string Signature;
   std::string SnippetSuffix;
-  getSignature(*CCS, &Signature, &SnippetSuffix);
+  getSignature(*CCS, &Signature, &SnippetSuffix, SymbolCompletion.Kind,
+               SymbolCompletion.CursorKind);
   S.Signature = Signature;
   S.CompletionSnippetSuffix = SnippetSuffix;
 
@@ -933,7 +934,8 @@ const Symbol *SymbolCollector::addDeclaration(const NamedDecl &ND, SymbolID ID,
   S.Documentation = Documentation;
   std::string Signature;
   std::string SnippetSuffix;
-  getSignature(*CCS, &Signature, &SnippetSuffix);
+  getSignature(*CCS, &Signature, &SnippetSuffix, SymbolCompletion.Kind,
+               SymbolCompletion.CursorKind);
   S.Signature = Signature;
   S.CompletionSnippetSuffix = SnippetSuffix;
   std::string ReturnType = getReturnType(*CCS);
diff --git a/clang-tools-extra/clangd/refactor/tweaks/SpecialMembers.cpp b/clang-tools-extra/clangd/refactor/tweaks/SpecialMembers.cpp
index e15605191c752..0b86b48422707 100644
--- a/clang-tools-extra/clangd/refactor/tweaks/SpecialMembers.cpp
+++ b/clang-tools-extra/clangd/refactor/tweaks/SpecialMembers.cpp
@@ -8,10 +8,7 @@
 #include "ParsedAST.h"
 #include "refactor/InsertionPoint.h"
 #include "refactor/Tweak.h"
-#include "support/Logger.h"
 #include "clang/AST/DeclCXX.h"
-#include "clang/Basic/SourceLocation.h"
-#include "clang/Basic/SourceManager.h"
 #include "clang/Sema/Sema.h"
 #include "clang/Tooling/Core/Replacement.h"
 #include "llvm/ADT/StringRef.h"
@@ -84,7 +81,7 @@ std::string buildSpecialMemberDeclarations(const CXXRecordDecl &Class) {
 //  - to understand the implicit behavior
 //  - to avoid relying on the implicit behavior
 //  - as a baseline for explicit modification
-class DeclareCopyMove : public Tweak {
+class SpecialMembers : public Tweak {
 public:
   const char *id() const final;
   llvm::StringLiteral kind() const override {
@@ -103,7 +100,7 @@ class DeclareCopyMove : public Tweak {
     // Trigger only on class definitions.
     if (auto *N = Inputs.ASTSelection.commonAncestor())
       Class = const_cast<CXXRecordDecl *>(N->ASTNode.get<CXXRecordDecl>());
-    if (!Class || !Class->isThisDeclarationADefinition())
+    if (!Class || !Class->isThisDeclarationADefinition() || Class->isUnion())
       return false;
 
     // Tweak is only available if some members are missing.
@@ -146,7 +143,7 @@ class DeclareCopyMove : public Tweak {
   bool NeedCopy = false, NeedMove = false;
   CXXRecordDecl *Class = nullptr;
 };
-REGISTER_TWEAK(DeclareCopyMove)
+REGISTER_TWEAK(SpecialMembers)
 
 } // namespace
 } // namespace clangd
diff --git a/clang-tools-extra/clangd/unittests/BackgroundIndexTests.cpp b/clang-tools-extra/clangd/unittests/BackgroundIndexTests.cpp
index b32875fe19500..e5c9fc5408806 100644
--- a/clang-tools-extra/clangd/unittests/BackgroundIndexTests.cpp
+++ b/clang-tools-extra/clangd/unittests/BackgroundIndexTests.cpp
@@ -69,7 +69,7 @@ class MemoryShardStorage : public BackgroundIndexStorage {
   loadShard(llvm::StringRef ShardIdentifier) const override {
     std::lock_guard<std::mutex> Lock(StorageMu);
     AccessedPaths.insert(ShardIdentifier);
-    if (Storage.find(ShardIdentifier) == Storage.end()) {
+    if (!Storage.contains(ShardIdentifier)) {
       return nullptr;
     }
     auto IndexFile =
diff --git a/clang-tools-extra/clangd/unittests/CMakeLists.txt b/clang-tools-extra/clangd/unittests/CMakeLists.txt
index a68d8cb0b78ff..056e5803d711b 100644
--- a/clang-tools-extra/clangd/unittests/CMakeLists.txt
+++ b/clang-tools-extra/clangd/unittests/CMakeLists.txt
@@ -104,12 +104,13 @@ add_unittest(ClangdUnitTests ClangdTests
 
   support/CancellationTests.cpp
   support/ContextTests.cpp
+  support/FileCacheTests.cpp
   support/FunctionTests.cpp
   support/MarkupTests.cpp
   support/MemoryTreeTests.cpp
   support/PathTests.cpp
-  support/ThreadingTests.cpp
   support/TestTracer.cpp
+  support/ThreadingTests.cpp
   support/TraceTests.cpp
 
   tweaks/AddUsingTests.cpp
@@ -130,6 +131,7 @@ add_unittest(ClangdUnitTests ClangdTests
   tweaks/RawStringLiteralTests.cpp
   tweaks/RemoveUsingNamespaceTests.cpp
   tweaks/ShowSelectionTreeTests.cpp
+  tweaks/SpecialMembersTests.cpp
   tweaks/SwapIfBranchesTests.cpp
   tweaks/TweakTesting.cpp
   tweaks/TweakTests.cpp
@@ -155,6 +157,7 @@ clang_target_link_libraries(ClangdTests
   clangLex
   clangSema
   clangSerialization
+  clangTesting
   clangTooling
   clangToolingCore
   clangToolingInclusions
diff --git a/clang-tools-extra/clangd/unittests/CodeCompleteTests.cpp b/clang-tools-extra/clangd/unittests/CodeCompleteTests.cpp
index f57f266c3c511..25d6a7b189ed5 100644
--- a/clang-tools-extra/clangd/unittests/CodeCompleteTests.cpp
+++ b/clang-tools-extra/clangd/unittests/CodeCompleteTests.cpp
@@ -3450,6 +3450,22 @@ TEST(CompletionTest, CursorInSnippets) {
   EXPECT_THAT(Results.Completions,
               Contains(AllOf(named("while_foo"),
                              snippetSuffix("(${1:int a}, ${2:int b})"))));
+
+  Results = completions(R"cpp(
+    struct Base {
+      Base(int a, int b) {}
+    };
+
+    struct Derived : Base {
+      Derived() : Base^
+    };
+  )cpp",
+                        /*IndexSymbols=*/{}, Options);
+  // Constructors from base classes are a kind of pattern that shouldn't end
+  // with $0.
+  EXPECT_THAT(Results.Completions,
+              Contains(AllOf(named("Base"),
+                             snippetSuffix("(${1:int a}, ${2:int b})"))));
 }
 
 TEST(CompletionTest, WorksWithNullType) {
diff --git a/clang-tools-extra/clangd/unittests/CodeCompletionStringsTests.cpp b/clang-tools-extra/clangd/unittests/CodeCompletionStringsTests.cpp
index 329a213c66984..faab6253e2832 100644
--- a/clang-tools-extra/clangd/unittests/CodeCompletionStringsTests.cpp
+++ b/clang-tools-extra/clangd/unittests/CodeCompletionStringsTests.cpp
@@ -24,11 +24,13 @@ class CompletionStringTest : public ::testing::Test {
 
 protected:
   void computeSignature(const CodeCompletionString &CCS,
-                        bool CompletingPattern = false) {
+                        CodeCompletionResult::ResultKind ResultKind =
+                            CodeCompletionResult::ResultKind::RK_Declaration) {
     Signature.clear();
     Snippet.clear();
-    getSignature(CCS, &Signature, &Snippet, /*RequiredQualifier=*/nullptr,
-                 CompletingPattern);
+    getSignature(CCS, &Signature, &Snippet, ResultKind,
+                 /*CursorKind=*/CXCursorKind::CXCursor_NotImplemented,
+                 /*RequiredQualifiers=*/nullptr);
   }
 
   std::shared_ptr<clang::GlobalCodeCompletionAllocator> Allocator;
@@ -145,11 +147,12 @@ TEST_F(CompletionStringTest, SnippetsInPatterns) {
     Builder.AddChunk(CodeCompletionString::CK_SemiColon);
     return *Builder.TakeString();
   };
-  computeSignature(MakeCCS(), /*CompletingPattern=*/false);
+  computeSignature(MakeCCS());
   EXPECT_EQ(Snippet, " ${1:name} = ${2:target};");
 
   // When completing a pattern, the last placeholder holds the cursor position.
-  computeSignature(MakeCCS(), /*CompletingPattern=*/true);
+  computeSignature(MakeCCS(),
+                   /*ResultKind=*/CodeCompletionResult::ResultKind::RK_Pattern);
   EXPECT_EQ(Snippet, " ${1:name} = $0;");
 }
 
diff --git a/clang-tools-extra/clangd/unittests/CompileCommandsTests.cpp b/clang-tools-extra/clangd/unittests/CompileCommandsTests.cpp
index be44ce59f6657..28f0d85d332ca 100644
--- a/clang-tools-extra/clangd/unittests/CompileCommandsTests.cpp
+++ b/clang-tools-extra/clangd/unittests/CompileCommandsTests.cpp
@@ -11,6 +11,7 @@
 #include "TestFS.h"
 #include "support/Context.h"
 
+#include "clang/Testing/CommandLineArgs.h"
 #include "clang/Tooling/ArgumentsAdjusters.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/STLExtras.h"
@@ -20,6 +21,7 @@
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Process.h"
+#include "llvm/Support/TargetSelect.h"
 
 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
@@ -41,15 +43,18 @@ using ::testing::Not;
 // Make use of all features and assert the exact command we get out.
 // Other tests just verify presence/absence of certain args.
 TEST(CommandMangler, Everything) {
+  llvm::InitializeAllTargetInfos(); // As in ClangdMain
+  std::string Target = getAnyTargetForTesting();
   auto Mangler = CommandMangler::forTests();
   Mangler.ClangPath = testPath("fake/clang");
   Mangler.ResourceDir = testPath("fake/resources");
   Mangler.Sysroot = testPath("fake/sysroot");
   tooling::CompileCommand Cmd;
-  Cmd.CommandLine = {"clang++", "--", "foo.cc", "bar.cc"};
+  Cmd.CommandLine = {Target + "-clang++", "--", "foo.cc", "bar.cc"};
   Mangler(Cmd, "foo.cc");
   EXPECT_THAT(Cmd.CommandLine,
-              ElementsAre(testPath("fake/clang++"),
+              ElementsAre(testPath("fake/" + Target + "-clang++"),
+                          "--target=" + Target, "--driver-mode=g++",
                           "-resource-dir=" + testPath("fake/resources"),
                           "-isysroot", testPath("fake/sysroot"), "--",
                           "foo.cc"));
@@ -197,8 +202,26 @@ TEST(CommandMangler, ConfigEdits) {
     Mangler(Cmd, "foo.cc");
   }
   // Edits are applied in given order and before other mangling and they always
-  // go before filename.
-  EXPECT_THAT(Cmd.CommandLine, ElementsAre(_, "--hello", "--", "FOO.CC"));
+  // go before filename. `--driver-mode=g++` here is in lower case because
+  // options inserted by addTargetAndModeForProgramName are not editable,
+  // see discussion in https://reviews.llvm.org/D138546
+  EXPECT_THAT(Cmd.CommandLine,
+              ElementsAre(_, "--driver-mode=g++", "--hello", "--", "FOO.CC"));
+}
+
+TEST(CommandMangler, ExpandedResponseFiles) {
+  SmallString<1024> Path;
+  int FD;
+  ASSERT_FALSE(llvm::sys::fs::createTemporaryFile("args", "", FD, Path));
+  llvm::raw_fd_ostream OutStream(FD, true);
+  OutStream << "-Wall";
+  OutStream.close();
+
+  auto Mangler = CommandMangler::forTests();
+  tooling::CompileCommand Cmd;
+  Cmd.CommandLine = {"clang", ("@" + Path).str(), "foo.cc"};
+  Mangler(Cmd, "foo.cc");
+  EXPECT_THAT(Cmd.CommandLine, ElementsAre(_, "-Wall", "--", "foo.cc"));
 }
 
 static std::string strip(llvm::StringRef Arg, llvm::StringRef Argv) {
diff --git a/clang-tools-extra/clangd/unittests/ConfigCompileTests.cpp b/clang-tools-extra/clangd/unittests/ConfigCompileTests.cpp
index 014db131c6224..45dacba5822b7 100644
--- a/clang-tools-extra/clangd/unittests/ConfigCompileTests.cpp
+++ b/clang-tools-extra/clangd/unittests/ConfigCompileTests.cpp
@@ -249,19 +249,19 @@ TEST_F(ConfigCompileTests, DiagnosticsIncludeCleaner) {
   // Defaults to None.
   EXPECT_TRUE(compileAndApply());
   EXPECT_EQ(Conf.Diagnostics.UnusedIncludes,
-            Config::UnusedIncludesPolicy::None);
+            Config::IncludesPolicy::None);
 
   Frag = {};
   Frag.Diagnostics.UnusedIncludes.emplace("None");
   EXPECT_TRUE(compileAndApply());
   EXPECT_EQ(Conf.Diagnostics.UnusedIncludes,
-            Config::UnusedIncludesPolicy::None);
+            Config::IncludesPolicy::None);
 
   Frag = {};
   Frag.Diagnostics.UnusedIncludes.emplace("Strict");
   EXPECT_TRUE(compileAndApply());
   EXPECT_EQ(Conf.Diagnostics.UnusedIncludes,
-            Config::UnusedIncludesPolicy::Strict);
+            Config::IncludesPolicy::Strict);
 
   Frag = {};
   EXPECT_TRUE(Conf.Diagnostics.Includes.IgnoreHeader.empty())
diff --git a/clang-tools-extra/clangd/unittests/DiagnosticsTests.cpp b/clang-tools-extra/clangd/unittests/DiagnosticsTests.cpp
index a20be39722ec3..289767b0dec0f 100644
--- a/clang-tools-extra/clangd/unittests/DiagnosticsTests.cpp
+++ b/clang-tools-extra/clangd/unittests/DiagnosticsTests.cpp
@@ -1900,7 +1900,7 @@ TEST(DiagnosticsTest, IncludeCleaner) {
   // Off by default.
   EXPECT_THAT(*TU.build().getDiagnostics(), IsEmpty());
   Config Cfg;
-  Cfg.Diagnostics.UnusedIncludes = Config::UnusedIncludesPolicy::Strict;
+  Cfg.Diagnostics.UnusedIncludes = Config::IncludesPolicy::Strict;
   // Set filtering.
   Cfg.Diagnostics.Includes.IgnoreHeader.emplace_back(
       [](llvm::StringRef Header) { return Header.endswith("ignore.h"); });
diff --git a/clang-tools-extra/clangd/unittests/HeadersTests.cpp b/clang-tools-extra/clangd/unittests/HeadersTests.cpp
index bc4a5abd0e46a..8c0eae6bb3cc8 100644
--- a/clang-tools-extra/clangd/unittests/HeadersTests.cpp
+++ b/clang-tools-extra/clangd/unittests/HeadersTests.cpp
@@ -147,7 +147,6 @@ MATCHER_P(written, Name, "") { return arg.Written == Name; }
 MATCHER_P(resolved, Name, "") { return arg.Resolved == Name; }
 MATCHER_P(includeLine, N, "") { return arg.HashLine == N; }
 MATCHER_P(directive, D, "") { return arg.Directive == D; }
-MATCHER_P(hasPragmaKeep, H, "") { return arg.BehindPragmaKeep == H; }
 
 MATCHER_P2(Distance, File, D, "") {
   if (arg.getFirst() != File)
@@ -293,18 +292,6 @@ TEST_F(HeadersTest, IncludeDirective) {
                                    directive(tok::pp_include_next)));
 }
 
-TEST_F(HeadersTest, IWYUPragmaKeep) {
-  FS.Files[MainFile] = R"cpp(
-#include "bar.h" // IWYU pragma: keep
-#include "foo.h"
-)cpp";
-
-  EXPECT_THAT(
-      collectIncludes().MainFileIncludes,
-      UnorderedElementsAre(AllOf(written("\"foo.h\""), hasPragmaKeep(false)),
-                           AllOf(written("\"bar.h\""), hasPragmaKeep(true))));
-}
-
 TEST_F(HeadersTest, InsertInclude) {
   std::string Path = testPath("sub/bar.h");
   FS.Files[Path] = "";
@@ -415,75 +402,6 @@ TEST_F(HeadersTest, PresumedLocations) {
               Contains(AllOf(includeLine(2), written("<a.h>"))));
 }
 
-TEST_F(HeadersTest, SelfContainedHeaders) {
-  // Including through non-builtin file has no effects.
-  FS.Files[MainFile] = R"cpp(
-#include "includeguarded.h"
-#include "nonguarded.h"
-#include "pp_depend.h"
-#include "pragmaguarded.h"
-#include "recursive.h"
-)cpp";
-  FS.Files["pragmaguarded.h"] = R"cpp(
-#pragma once
-)cpp";
-  FS.Files["includeguarded.h"] = R"cpp(
-#ifndef INCLUDE_GUARDED_H
-#define INCLUDE_GUARDED_H
-void foo();
-#endif // INCLUDE_GUARDED_H
-)cpp";
-  FS.Files["nonguarded.h"] = R"cpp(
-)cpp";
-  FS.Files["pp_depend.h"] = R"cpp(
-  #ifndef REQUIRED_PP_DIRECTIVE
-  # error You have to have PP directive set to include this one!
-  #endif
-)cpp";
-  FS.Files["recursive.h"] = R"cpp(
-  #ifndef RECURSIVE_H
-  #define RECURSIVE_H
-
-  #include "recursive.h"
-
-  #endif // RECURSIVE_H
-)cpp";
-
-  auto Includes = collectIncludes();
-  EXPECT_TRUE(Includes.isSelfContained(getID("pragmaguarded.h", Includes)));
-  EXPECT_TRUE(Includes.isSelfContained(getID("includeguarded.h", Includes)));
-  EXPECT_TRUE(Includes.isSelfContained(getID("recursive.h", Includes)));
-  EXPECT_FALSE(Includes.isSelfContained(getID("nonguarded.h", Includes)));
-  EXPECT_FALSE(Includes.isSelfContained(getID("pp_depend.h", Includes)));
-}
-
-TEST_F(HeadersTest, HasIWYUPragmas) {
-  FS.Files[MainFile] = R"cpp(
-#include "export.h"
-#include "begin_exports.h"
-#include "none.h"
-)cpp";
-  FS.Files["export.h"] = R"cpp(
-#pragma once
-#include "none.h" // IWYU pragma: export
-)cpp";
-  FS.Files["begin_exports.h"] = R"cpp(
-#pragma once
-// IWYU pragma: begin_exports
-#include "none.h"
-// IWYU pragma: end_exports
-)cpp";
-  FS.Files["none.h"] = R"cpp(
-#pragma once
-// Not a pragma.
-)cpp";
-
-  auto Includes = collectIncludes();
-  EXPECT_TRUE(Includes.hasIWYUExport(getID("export.h", Includes)));
-  EXPECT_TRUE(Includes.hasIWYUExport(getID("begin_exports.h", Includes)));
-  EXPECT_FALSE(Includes.hasIWYUExport(getID("none.h", Includes)));
-}
-
 } // namespace
 } // namespace clangd
 } // namespace clang
diff --git a/clang-tools-extra/clangd/unittests/IncludeCleanerTests.cpp b/clang-tools-extra/clangd/unittests/IncludeCleanerTests.cpp
index 01b8e6a05dab5..409e3cee791c3 100644
--- a/clang-tools-extra/clangd/unittests/IncludeCleanerTests.cpp
+++ b/clang-tools-extra/clangd/unittests/IncludeCleanerTests.cpp
@@ -8,308 +8,59 @@
 
 #include "Annotations.h"
 #include "Config.h"
+#include "Diagnostics.h"
 #include "IncludeCleaner.h"
+#include "ParsedAST.h"
 #include "SourceCode.h"
 #include "TestFS.h"
 #include "TestTU.h"
+#include "clang-include-cleaner/Analysis.h"
+#include "clang-include-cleaner/Types.h"
 #include "support/Context.h"
+#include "clang/AST/DeclBase.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Tooling/Syntax/Tokens.h"
 #include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Casting.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/ScopedPrinter.h"
 #include "llvm/Testing/Support/SupportHelpers.h"
 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
+#include <cstddef>
+#include <string>
+#include <vector>
 
 namespace clang {
 namespace clangd {
 namespace {
 
+using ::testing::AllOf;
 using ::testing::ElementsAre;
-using ::testing::ElementsAreArray;
 using ::testing::IsEmpty;
+using ::testing::Matcher;
 using ::testing::Pointee;
 using ::testing::UnorderedElementsAre;
 
-std::string guard(llvm::StringRef Code) {
-  return "#pragma once\n" + Code.str();
+Matcher<const Diag &> withFix(::testing::Matcher<Fix> FixMatcher) {
+  return Field(&Diag::Fixes, ElementsAre(FixMatcher));
 }
 
-TEST(IncludeCleaner, ReferencedLocations) {
-  struct TestCase {
-    std::string HeaderCode;
-    std::string MainCode;
-  };
-  TestCase Cases[] = {
-      // DeclRefExpr
-      {
-          "int ^x();",
-          "int y = x();",
-      },
-      // RecordDecl
-      {
-          "class ^X;",
-          "X *y;",
-      },
-      // When definition is available, we don't need to mark forward
-      // declarations as used.
-      {
-          "class ^X {}; class X;",
-          "X *y;",
-      },
-      // We already have forward declaration in the main file, the other
-      // non-definition declarations are not needed.
-      {
-          "class ^X {}; class X;",
-          "class X; X *y;",
-      },
-      // Nested class definition can occur outside of the parent class
-      // definition. Bar declaration should be visible to its definition but
-      // it will always be because we will mark Foo definition as used.
-      {
-          "class ^Foo { class Bar; };",
-          "class Foo::Bar {};",
-      },
-      // TypedefType and UsingDecls
-      {
-          "using ^Integer = int;",
-          "Integer x;",
-      },
-      {
-          "namespace ns { void ^foo(); void ^foo() {} }",
-          "using ns::foo;",
-      },
-      {
-          "namespace ns { void foo(); void foo() {}; }",
-          "using namespace ns;",
-      },
-      {
-          // Refs from UsingTypeLoc and implicit constructor!
-          "struct ^A {}; using B = A; using ^C = B;",
-          "C a;",
-      },
-      {"namespace ns { template<typename T> class A {}; } using ns::^A;",
-       "A<int>* a;"},
-      {"namespace ns { template<typename T> class A {}; } using ns::^A;",
-       R"cpp(
-          template <template <typename> class T> class X {};
-          X<A> x;
-        )cpp"},
-      {R"cpp(
-          namespace ns { template<typename T> class A {}; }
-          namespace absl {using ns::^A;}
-       )cpp",
-       R"cpp(
-          template <template <typename> class T> class X {};
-          X<absl::A> x;
-       )cpp"},
-      {R"cpp(
-          namespace ns { template<typename T> struct ^A { ^A(T); }; }
-          using ns::^A;
-       )cpp",
-       "A CATD(123);"},
-      {
-          "typedef bool ^Y; template <typename T> struct ^X {};",
-          "X<Y> x;",
-      },
-      {
-          // https://github.com/clangd/clangd/issues/1036
-          R"cpp(
-            struct ^Base { void ^base(); };
-            template <int> struct ^Derived : Base {};
-          )cpp",
-          R"cpp(
-            class Holder {
-              void foo() { Member.base(); }
-              Derived<0> Member;
-            };
-          )cpp",
-      },
-      {
-          "struct Foo; struct ^Foo{}; typedef Foo ^Bar;",
-          "Bar b;",
-      },
-      {
-          "namespace ns { class X; }; using ns::^X;",
-          "X *y;",
-      },
-      // MemberExpr
-      {
-          "struct ^X{int ^a;}; X ^foo();",
-          "int y = foo().a;",
-      },
-      // Expr (type is traversed)
-      {
-          "class ^X{}; X ^foo();",
-          "auto bar() { return foo(); }",
-      },
-      // Redecls
-      {
-          "void ^foo(); void ^foo() {} void ^foo();",
-          "void bar() { foo(); }",
-      },
-      // Constructor
-      {
-          "struct ^X { ^X(int) {} int ^foo(); };",
-          "auto x = X(42); auto y = x.foo();",
-      },
-      // Function
-      {
-          "void ^foo();",
-          "void foo() {}",
-      },
-      {
-          "void foo() {}",
-          "void foo();",
-      },
-      {
-          "inline void ^foo() {}",
-          "void bar() { foo(); }",
-      },
-      {
-          "int ^foo(char); int ^foo(float);",
-          "template<class T> int x = foo(T{});",
-      },
-      // Static function
-      {
-          "struct ^X { static bool ^foo(); }; bool X::^foo() {}",
-          "auto b = X::foo();",
-      },
-      // TemplateRecordDecl
-      {
-          "template <typename> class ^X;",
-          "X<int> *y;",
-      },
-      // Type name not spelled out in code
-      {
-          "class ^X{}; X ^getX();",
-          "auto x = getX();",
-      },
-      // Enums
-      {
-          "enum ^Color { ^Red = 42, Green = 9000};",
-          "int MyColor = Red;",
-      },
-      {
-          "struct ^X { enum ^Language { ^CXX = 42, Python = 9000}; };",
-          "int Lang = X::CXX;",
-      },
-      // Macros
-      {
-          "#define ^CONSTANT 42",
-          "int Foo = CONSTANT;",
-      },
-      {
-          "#define ^FOO x",
-          "#define BAR FOO",
-      },
-      {
-          "#define INNER 42\n"
-          "#define ^OUTER INNER",
-          "int answer = OUTER;",
-      },
-      {
-          "#define ^ANSWER 42\n"
-          "#define ^SQUARE(X) X * X",
-          "int sq = SQUARE(ANSWER);",
-      },
-      {
-          "#define ^FOO\n"
-          "#define ^BAR",
-          "#if 0\n"
-          "#if FOO\n"
-          "BAR\n"
-          "#endif\n"
-          "#endif",
-      },
-      // Misc
-      {
-          "enum class ^Color : int;",
-          "enum class Color : int {};",
-      },
-      {
-          "enum class Color : int {};",
-          "enum class Color : int;",
-      },
-      {
-          "enum class ^Color;",
-          "Color c;",
-      },
-      {
-          "enum class ^Color : int;",
-          "Color c;",
-      },
-      {
-          "enum class ^Color : char;",
-          "Color *c;",
-      },
-      {
-          "enum class ^Color : char {};",
-          "Color *c;",
-      },
-  };
-  for (const TestCase &T : Cases) {
-    TestTU TU;
-    TU.Code = T.MainCode;
-    Annotations Header(T.HeaderCode);
-    TU.HeaderCode = Header.code().str();
-    TU.ExtraArgs.push_back("-std=c++17");
-    auto AST = TU.build();
-
-    std::vector<Position> Points;
-    for (const auto &Loc : findReferencedLocations(AST).User) {
-      if (AST.getSourceManager().getBufferName(Loc).endswith(
-              TU.HeaderFilename)) {
-        Points.push_back(offsetToPosition(
-            TU.HeaderCode, AST.getSourceManager().getFileOffset(Loc)));
-      }
-    }
-    llvm::sort(Points);
-
-    EXPECT_EQ(Points, Header.points()) << T.HeaderCode << "\n---\n"
-                                       << T.MainCode;
-  }
+MATCHER_P2(Diag, Range, Message,
+           "Diag at " + llvm::to_string(Range) + " = [" + Message + "]") {
+  return arg.Range == Range && arg.Message == Message;
 }
 
-TEST(IncludeCleaner, Stdlib) {
-  // Smoke tests only for finding used symbols/headers.
-  // Details of Decl -> stdlib::Symbol -> stdlib::Headers mapping tested there.
-  auto TU = TestTU::withHeaderCode(R"cpp(
-    namespace std { class error_code {}; }
-    class error_code {};
-    namespace nonstd { class error_code {}; }
-  )cpp");
-  struct {
-    llvm::StringRef Code;
-    std::vector<llvm::StringRef> Symbols;
-    std::vector<llvm::StringRef> Headers;
-  } Tests[] = {
-      {"std::error_code x;", {"std::error_code"}, {"<system_error>"}},
-      {"error_code x;", {}, {}},
-      {"nonstd::error_code x;", {}, {}},
-  };
-
-  for (const auto &Test : Tests) {
-    TU.Code = Test.Code.str();
-    ParsedAST AST = TU.build();
-    std::vector<tooling::stdlib::Symbol> WantSyms;
-    for (const auto &SymName : Test.Symbols) {
-      auto QName = splitQualifiedName(SymName);
-      auto Sym = tooling::stdlib::Symbol::named(QName.first, QName.second);
-      EXPECT_TRUE(Sym) << SymName;
-      WantSyms.push_back(*Sym);
-    }
-    std::vector<tooling::stdlib::Header> WantHeaders;
-    for (const auto &HeaderName : Test.Headers) {
-      auto Header = tooling::stdlib::Header::named(HeaderName);
-      EXPECT_TRUE(Header) << HeaderName;
-      WantHeaders.push_back(*Header);
-    }
+MATCHER_P3(Fix, Range, Replacement, Message,
+           "Fix " + llvm::to_string(Range) + " => " +
+               ::testing::PrintToString(Replacement) + " = [" + Message + "]") {
+  return arg.Message == Message && arg.Edits.size() == 1 &&
+         arg.Edits[0].range == Range && arg.Edits[0].newText == Replacement;
+}
 
-    ReferencedLocations Locs = findReferencedLocations(AST);
-    EXPECT_THAT(Locs.Stdlib, ElementsAreArray(WantSyms));
-    ReferencedFiles Files =
-        findReferencedFiles(Locs, AST.getIncludeStructure(),
-                            AST.getCanonicalIncludes(), AST.getSourceManager());
-    EXPECT_THAT(Files.Stdlib, ElementsAreArray(WantHeaders));
-  }
+std::string guard(llvm::StringRef Code) {
+  return "#pragma once\n" + Code.str();
 }
 
 MATCHER_P(writtenInclusion, Written, "") {
@@ -340,9 +91,8 @@ TEST(IncludeCleaner, StdlibUnused) {
   TU.AdditionalFiles["queue"] = "#include <bits>";
   TU.ExtraArgs = {"-isystem", testRoot()};
   auto AST = TU.build();
-  EXPECT_THAT(computeUnusedIncludes(AST),
-              ElementsAre(Pointee(writtenInclusion("<queue>"))));
-  EXPECT_THAT(computeUnusedIncludesExperimental(AST),
+  IncludeCleanerFindings Findings = computeIncludeCleanerFindings(AST);
+  EXPECT_THAT(Findings.UnusedIncludes,
               ElementsAre(Pointee(writtenInclusion("<queue>"))));
 }
 
@@ -375,150 +125,135 @@ TEST(IncludeCleaner, GetUnusedHeaders) {
   TU.ExtraArgs.push_back("-isystem" + testPath("system"));
   TU.Code = MainFile.str();
   ParsedAST AST = TU.build();
+  IncludeCleanerFindings Findings = computeIncludeCleanerFindings(AST);
   EXPECT_THAT(
-      computeUnusedIncludes(AST),
-      UnorderedElementsAre(Pointee(writtenInclusion("\"unused.h\"")),
-                           Pointee(writtenInclusion("\"dir/unused.h\""))));
-  EXPECT_THAT(
-      computeUnusedIncludesExperimental(AST),
+      Findings.UnusedIncludes,
       UnorderedElementsAre(Pointee(writtenInclusion("\"unused.h\"")),
                            Pointee(writtenInclusion("\"dir/unused.h\""))));
 }
 
-TEST(IncludeCleaner, VirtualBuffers) {
+TEST(IncludeCleaner, ComputeMissingHeaders) {
+  Annotations MainFile(R"cpp(
+    #include "a.h"
+
+    void foo() {
+      $b[[b]]();
+    })cpp");
   TestTU TU;
-  TU.Code = R"cpp(
-    #include "macros.h"
+  TU.Filename = "foo.cpp";
+  TU.AdditionalFiles["a.h"] = guard("#include \"b.h\"");
+  TU.AdditionalFiles["b.h"] = guard("void b();");
 
-    using flags::FLAGS_FOO;
+  TU.Code = MainFile.code();
+  ParsedAST AST = TU.build();
 
-    // CLI will come from a define, __cplusplus is a built-in. In both cases, they
-    // come from non-existent files.
-    int y = CLI + __cplusplus;
+  IncludeCleanerFindings Findings = computeIncludeCleanerFindings(AST);
+  const SourceManager &SM = AST.getSourceManager();
+  const NamedDecl *BDecl = nullptr;
+  for (Decl *D : AST.getASTContext().getTranslationUnitDecl()->decls()) {
+    const NamedDecl *CandidateDecl = llvm::dyn_cast<NamedDecl>(D);
+    std::string Name = CandidateDecl->getQualifiedNameAsString();
+    if (Name != "b")
+      continue;
+    BDecl = CandidateDecl;
+  }
+  ASSERT_TRUE(BDecl);
+  include_cleaner::Symbol B{*BDecl};
+  auto Range = MainFile.range("b");
+  size_t Start = llvm::cantFail(positionToOffset(MainFile.code(), Range.start));
+  size_t End = llvm::cantFail(positionToOffset(MainFile.code(), Range.end));
+  syntax::FileRange BRange{SM.getMainFileID(), static_cast<unsigned int>(Start),
+                           static_cast<unsigned int>(End)};
+  include_cleaner::Header Header{*SM.getFileManager().getFile("b.h")};
+  MissingIncludeDiagInfo BInfo{B, BRange, {Header}};
+  EXPECT_THAT(Findings.MissingIncludes, ElementsAre(BInfo));
+}
 
-    int concat(a, b) = 42;
-    )cpp";
-  // The pasting operator in combination with DEFINE_FLAG will create
-  // ScratchBuffer with `flags::FLAGS_FOO` that will have FileID but not
-  // FileEntry.
-  TU.AdditionalFiles["macros.h"] = R"cpp(
-    #ifndef MACROS_H
-    #define MACROS_H
+TEST(IncludeCleaner, GenerateMissingHeaderDiags) {
+  Config Cfg;
+  Cfg.Diagnostics.MissingIncludes = Config::IncludesPolicy::Strict;
+  Cfg.Diagnostics.Includes.IgnoreHeader = {
+      [](llvm::StringRef Header) { return Header.ends_with("buzz.h"); }};
+  WithContextValue Ctx(Config::Key, std::move(Cfg));
+  Annotations MainFile(R"cpp(
+#include "a.h"
+$insert_b[[]]#include "baz.h"
+#include "dir/c.h"
+$insert_d[[]]#include "fuzz.h"
+#include "header.h"
+$insert_foobar[[]]#include <e.h>
+$insert_f[[]]$insert_vector[[]]
 
-    #define DEFINE_FLAG(X) \
-    namespace flags { \
-    int FLAGS_##X; \
-    } \
+    void foo() {
+      $b[[b]]();
 
-    DEFINE_FLAG(FOO)
+      ns::$bar[[Bar]] bar;
+      bar.d();
+      $f[[f]](); 
 
-    #define ab x
-    #define concat(x, y) x##y
+      // this should not be diagnosed, because it's ignored in the config
+      buzz(); 
 
-    #endif // MACROS_H
-    )cpp";
-  TU.ExtraArgs = {"-DCLI=42"};
-  ParsedAST AST = TU.build();
-  auto &SM = AST.getSourceManager();
-  auto &Includes = AST.getIncludeStructure();
-
-  auto ReferencedFiles = findReferencedFiles(
-      findReferencedLocations(AST), Includes, AST.getCanonicalIncludes(), SM);
-  llvm::StringSet<> ReferencedFileNames;
-  for (FileID FID : ReferencedFiles.User)
-    ReferencedFileNames.insert(
-        SM.getPresumedLoc(SM.getLocForStartOfFile(FID)).getFilename());
-  // Note we deduped the names as _number_ of <built-in>s is uninteresting.
-  EXPECT_THAT(ReferencedFileNames.keys(),
-              UnorderedElementsAre("<built-in>", "<scratch space>",
-                                   testPath("macros.h")));
-
-  // Should not crash due to FileIDs that are not headers.
-  auto ReferencedHeaders = translateToHeaderIDs(ReferencedFiles, Includes, SM);
-  std::vector<llvm::StringRef> ReferencedHeaderNames;
-  for (IncludeStructure::HeaderID HID : ReferencedHeaders)
-    ReferencedHeaderNames.push_back(Includes.getRealPath(HID));
-  // Non-header files are gone at this point.
-  EXPECT_THAT(ReferencedHeaderNames, ElementsAre(testPath("macros.h")));
-
-  // Sanity check.
-  EXPECT_THAT(
-      getUnused(AST, ReferencedHeaders, ReferencedFiles.SpelledUmbrellas),
-      IsEmpty());
-}
+      $foobar[[foobar]]();
 
-TEST(IncludeCleaner, DistinctUnguardedInclusions) {
-  TestTU TU;
-  TU.Code = R"cpp(
-    #include "bar.h"
-    #include "foo.h"
+      std::$vector[[vector]] v;
+    })cpp");
 
-    int LocalFoo = foo::Variable;
-    )cpp";
-  TU.AdditionalFiles["foo.h"] = R"cpp(
-    #pragma once
-    namespace foo {
-    #include "unguarded.h"
-    }
-    )cpp";
-  TU.AdditionalFiles["bar.h"] = R"cpp(
-    #pragma once
-    namespace bar {
-    #include "unguarded.h"
-    }
-    )cpp";
-  TU.AdditionalFiles["unguarded.h"] = R"cpp(
-    constexpr int Variable = 42;
-    )cpp";
+  TestTU TU;
+  TU.Filename = "foo.cpp";
+  TU.AdditionalFiles["a.h"] = guard("#include \"b.h\"");
+  TU.AdditionalFiles["b.h"] = guard("void b();");
 
-  ParsedAST AST = TU.build();
+  TU.AdditionalFiles["dir/c.h"] = guard("#include \"d.h\"");
+  TU.AdditionalFiles["dir/d.h"] =
+      guard("namespace ns { struct Bar { void d(); }; }");
 
-  auto ReferencedFiles = findReferencedFiles(
-      findReferencedLocations(AST), AST.getIncludeStructure(),
-      AST.getCanonicalIncludes(), AST.getSourceManager());
-  llvm::StringSet<> ReferencedFileNames;
-  auto &SM = AST.getSourceManager();
-  for (FileID FID : ReferencedFiles.User)
-    ReferencedFileNames.insert(
-        SM.getPresumedLoc(SM.getLocForStartOfFile(FID)).getFilename());
-  // Note that we have uplifted the referenced files from non self-contained
-  // headers to header-guarded ones.
-  EXPECT_THAT(ReferencedFileNames.keys(),
-              UnorderedElementsAre(testPath("foo.h")));
-}
+  TU.AdditionalFiles["system/e.h"] = guard("#include <f.h>");
+  TU.AdditionalFiles["system/f.h"] = guard("void f();");
+  TU.ExtraArgs.push_back("-isystem" + testPath("system"));
 
-TEST(IncludeCleaner, NonSelfContainedHeaders) {
-  TestTU TU;
-  TU.Code = R"cpp(
-    #include "foo.h"
+  TU.AdditionalFiles["fuzz.h"] = guard("#include \"buzz.h\"");
+  TU.AdditionalFiles["buzz.h"] = guard("void buzz();");
 
-    int LocalFoo = Variable;
-    )cpp";
-  TU.AdditionalFiles["foo.h"] = R"cpp(
-    #pragma once
-    #include "indirection.h"
-    )cpp";
-  TU.AdditionalFiles["indirection.h"] = R"cpp(
-    #include "unguarded.h"
-    )cpp";
-  TU.AdditionalFiles["unguarded.h"] = R"cpp(
-    constexpr int Variable = 42;
-    )cpp";
+  TU.AdditionalFiles["baz.h"] = guard("#include \"private.h\"");
+  TU.AdditionalFiles["private.h"] = guard(R"cpp(
+    // IWYU pragma: private, include "public.h"
+    void foobar();
+  )cpp");
+  TU.AdditionalFiles["header.h"] = guard(R"cpp(
+  namespace std { class vector {}; }
+  )cpp");
 
+  TU.Code = MainFile.code();
   ParsedAST AST = TU.build();
 
-  auto ReferencedFiles = findReferencedFiles(
-      findReferencedLocations(AST), AST.getIncludeStructure(),
-      AST.getCanonicalIncludes(), AST.getSourceManager());
-  llvm::StringSet<> ReferencedFileNames;
-  auto &SM = AST.getSourceManager();
-  for (FileID FID : ReferencedFiles.User)
-    ReferencedFileNames.insert(
-        SM.getPresumedLoc(SM.getLocForStartOfFile(FID)).getFilename());
-  // Note that we have uplifted the referenced files from non self-contained
-  // headers to header-guarded ones.
-  EXPECT_THAT(ReferencedFileNames.keys(),
-              UnorderedElementsAre(testPath("foo.h")));
+  std::vector<clangd::Diag> Diags =
+      issueIncludeCleanerDiagnostics(AST, TU.Code);
+  EXPECT_THAT(
+      Diags,
+      UnorderedElementsAre(
+          AllOf(Diag(MainFile.range("b"),
+                     "No header providing \"b\" is directly included"),
+                withFix(Fix(MainFile.range("insert_b"), "#include \"b.h\"\n",
+                            "#include \"b.h\""))),
+          AllOf(Diag(MainFile.range("bar"),
+                     "No header providing \"ns::Bar\" is directly included"),
+                withFix(Fix(MainFile.range("insert_d"),
+                            "#include \"dir/d.h\"\n", "#include \"dir/d.h\""))),
+          AllOf(Diag(MainFile.range("f"),
+                     "No header providing \"f\" is directly included"),
+                withFix(Fix(MainFile.range("insert_f"), "#include <f.h>\n",
+                            "#include <f.h>"))),
+          AllOf(
+              Diag(MainFile.range("foobar"),
+                   "No header providing \"foobar\" is directly included"),
+              withFix(Fix(MainFile.range("insert_foobar"),
+                          "#include \"public.h\"\n", "#include \"public.h\""))),
+          AllOf(
+              Diag(MainFile.range("vector"),
+                   "No header providing \"std::vector\" is directly included"),
+              withFix(Fix(MainFile.range("insert_vector"),
+                          "#include <vector>\n", "#include <vector>")))));
 }
 
 TEST(IncludeCleaner, IWYUPragmas) {
@@ -538,52 +273,12 @@ TEST(IncludeCleaner, IWYUPragmas) {
     void foo() {}
   )cpp");
   Config Cfg;
-  Cfg.Diagnostics.UnusedIncludes = Config::UnusedIncludesPolicy::Experiment;
+  Cfg.Diagnostics.UnusedIncludes = Config::IncludesPolicy::Strict;
   WithContextValue Ctx(Config::Key, std::move(Cfg));
   ParsedAST AST = TU.build();
-
-  auto ReferencedFiles = findReferencedFiles(
-      findReferencedLocations(AST), AST.getIncludeStructure(),
-      AST.getCanonicalIncludes(), AST.getSourceManager());
-  EXPECT_EQ(ReferencedFiles.SpelledUmbrellas.size(), 1u);
-  EXPECT_EQ(ReferencedFiles.SpelledUmbrellas.begin()->getKey(), "\"public.h\"");
-  EXPECT_EQ(ReferencedFiles.User.size(), 2u);
-  EXPECT_TRUE(
-      ReferencedFiles.User.contains(AST.getSourceManager().getMainFileID()));
-  EXPECT_TRUE(
-      ReferencedFiles.User.contains(AST.getSourceManager().getMainFileID()));
-  EXPECT_THAT(AST.getDiagnostics(), llvm::ValueIs(IsEmpty()));
-  EXPECT_THAT(computeUnusedIncludes(AST), IsEmpty());
-  EXPECT_THAT(computeUnusedIncludesExperimental(AST), IsEmpty());
-}
-
-TEST(IncludeCleaner, RecursiveInclusion) {
-  TestTU TU;
-  TU.Code = R"cpp(
-    #include "foo.h"
-
-    void baz() {
-      foo();
-    }
-    )cpp";
-  TU.AdditionalFiles["foo.h"] = R"cpp(
-    #ifndef FOO_H
-    #define FOO_H
-
-    void foo() {}
-
-    #include "bar.h"
-
-    #endif
-  )cpp";
-  TU.AdditionalFiles["bar.h"] = guard(R"cpp(
-    #include "foo.h"
-  )cpp");
-  ParsedAST AST = TU.build();
-
   EXPECT_THAT(AST.getDiagnostics(), llvm::ValueIs(IsEmpty()));
-  EXPECT_THAT(computeUnusedIncludes(AST), IsEmpty());
-  EXPECT_THAT(computeUnusedIncludesExperimental(AST), IsEmpty());
+  IncludeCleanerFindings Findings = computeIncludeCleanerFindings(AST);
+  EXPECT_THAT(Findings.UnusedIncludes, IsEmpty());
 }
 
 TEST(IncludeCleaner, IWYUPragmaExport) {
@@ -605,10 +300,9 @@ TEST(IncludeCleaner, IWYUPragmaExport) {
   ParsedAST AST = TU.build();
 
   EXPECT_THAT(AST.getDiagnostics(), llvm::ValueIs(IsEmpty()));
-  // FIXME: This is not correct: foo.h is unused but is not diagnosed as such
-  // because we ignore headers with IWYU export pragmas for now.
-  EXPECT_THAT(computeUnusedIncludes(AST), IsEmpty());
-  EXPECT_THAT(computeUnusedIncludesExperimental(AST), IsEmpty());
+  IncludeCleanerFindings Findings = computeIncludeCleanerFindings(AST);
+  EXPECT_THAT(Findings.UnusedIncludes,
+              ElementsAre(Pointee(writtenInclusion("\"foo.h\""))));
 }
 
 TEST(IncludeCleaner, NoDiagsForObjC) {
@@ -627,7 +321,9 @@ TEST(IncludeCleaner, NoDiagsForObjC) {
   TU.ExtraArgs.emplace_back("-xobjective-c");
 
   Config Cfg;
-  Cfg.Diagnostics.UnusedIncludes = Config::UnusedIncludesPolicy::Strict;
+
+  Cfg.Diagnostics.UnusedIncludes = Config::IncludesPolicy::Strict;
+  Cfg.Diagnostics.MissingIncludes = Config::IncludesPolicy::Strict;
   WithContextValue Ctx(Config::Key, std::move(Cfg));
   ParsedAST AST = TU.build();
   EXPECT_THAT(AST.getDiagnostics(), llvm::ValueIs(IsEmpty()));
diff --git a/clang-tools-extra/clangd/unittests/PreambleTests.cpp b/clang-tools-extra/clangd/unittests/PreambleTests.cpp
index 94870abf1569b..903d9ef123eea 100644
--- a/clang-tools-extra/clangd/unittests/PreambleTests.cpp
+++ b/clang-tools-extra/clangd/unittests/PreambleTests.cpp
@@ -19,6 +19,7 @@
 #include "TestFS.h"
 #include "TestTU.h"
 #include "XRefs.h"
+#include "support/Context.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Format/Format.h"
 #include "clang/Frontend/FrontendActions.h"
@@ -29,13 +30,13 @@
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/ScopedPrinter.h"
 #include "llvm/Support/VirtualFileSystem.h"
-#include "llvm/Testing/Support/SupportHelpers.h"
 #include "gmock/gmock.h"
 #include "gtest/gtest-matchers.h"
 #include "gtest/gtest.h"
 #include <memory>
 #include <optional>
 #include <string>
+#include <utility>
 #include <vector>
 
 using testing::AllOf;
@@ -216,7 +217,6 @@ TEST(PreamblePatchTest, PatchesPreambleIncludes) {
           Field(&Inclusion::Written, "\"a.h\""),
           Field(&Inclusion::Resolved, testPath("a.h")),
           Field(&Inclusion::HeaderID, testing::Not(testing::Eq(std::nullopt))),
-          Field(&Inclusion::BehindPragmaKeep, true),
           Field(&Inclusion::FileKind, SrcMgr::CharacteristicKind::C_User))));
 }
 
@@ -665,7 +665,7 @@ TEST(PreamblePatch, DiagnosticsFromMainASTAreInRightPlace) {
 TEST(PreamblePatch, DiagnosticsToPreamble) {
   Config Cfg;
   Cfg.Diagnostics.AllowStalePreamble = true;
-  Cfg.Diagnostics.UnusedIncludes = Config::UnusedIncludesPolicy::Strict;
+  Cfg.Diagnostics.UnusedIncludes = Config::IncludesPolicy::Strict;
   WithContextValue WithCfg(Config::Key, std::move(Cfg));
 
   llvm::StringMap<std::string> AdditionalFiles;
@@ -827,6 +827,45 @@ x>)");
     EXPECT_THAT(*AST->getDiagnostics(), IsEmpty());
   }
 }
+
+MATCHER_P2(Mark, Range, Text, "") {
+  return std::tie(arg.Rng, arg.Trivia) == std::tie(Range, Text);
+}
+
+TEST(PreamblePatch, MacroAndMarkHandling) {
+  Config Cfg;
+  Cfg.Diagnostics.AllowStalePreamble = true;
+  WithContextValue WithCfg(Config::Key, std::move(Cfg));
+
+  {
+    Annotations Code(R"cpp(
+#ifndef FOO
+#define FOO
+// Some comments
+#pragma mark XX
+#define BAR
+
+#endif)cpp");
+    Annotations NewCode(R"cpp(
+#ifndef FOO
+#define FOO
+#define BAR
+#pragma $x[[mark XX
+]]
+#pragma $y[[mark YY
+]]
+#define BAZ
+
+#endif)cpp");
+    auto AST = createPatchedAST(Code.code(), NewCode.code());
+    EXPECT_THAT(AST->getMacros().Names.keys(),
+                UnorderedElementsAreArray({"FOO", "BAR", "BAZ"}));
+    EXPECT_THAT(AST->getMarks(),
+                UnorderedElementsAre(Mark(NewCode.range("x"), " XX"),
+                                     Mark(NewCode.range("y"), " YY")));
+  }
+}
+
 } // namespace
 } // namespace clangd
 } // namespace clang
diff --git a/clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp b/clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp
index f5174979a8d6e..259efcf54a6b2 100644
--- a/clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp
+++ b/clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp
@@ -892,7 +892,7 @@ sizeof...($TemplateParameter[[Elements]]);
 
         template $Bracket[[<]]typename $TemplateParameter_def[[T]]$Bracket[[>]]
         $TemplateParameter[[T]] $Variable_def[[x]] = {};
-        template <>
+        template $Bracket[[<]]$Bracket[[>]]
         int $Variable_def[[x]]$Bracket[[<]]int$Bracket[[>]] = (int)sizeof($Class[[Base]]);
       )cpp",
       // operator calls in template
@@ -977,7 +977,7 @@ sizeof...($TemplateParameter[[Elements]]);
         }
         template $Bracket[[<]]typename $TemplateParameter_def[[T]]$Bracket[[>]] constexpr int $Variable_def_readonly[[V]] = 42;
         constexpr int $Variable_def_readonly[[Y]] = $Variable_readonly[[V]]$Bracket[[<]]char$Bracket[[>]];
-        template <>
+        template $Bracket[[<]]$Bracket[[>]]
         constexpr int $Variable_def_readonly[[V]]$Bracket[[<]]int$Bracket[[>]] = 5;
         template $Bracket[[<]]typename $TemplateParameter_def[[T]]$Bracket[[>]]
         constexpr int $Variable_def_readonly[[V]]$Bracket[[<]]$TemplateParameter[[T]]*$Bracket[[>]] = 6;
@@ -1045,7 +1045,15 @@ sizeof...($TemplateParameter[[Elements]]);
           auto $LocalVariable_def[[s]] = $Operator[[new]] $Class[[Foo]]$Bracket[[<]]$TemplateParameter[[T]]$Bracket[[>]]();
           $Operator[[delete]] $LocalVariable[[s]];
         }
-      )cpp"};
+      )cpp",
+      // Recursive UsingValueDecl
+      R"cpp(
+        template $Bracket[[<]]int$Bracket[[>]] class $Class_def[[X]] {
+          template $Bracket[[<]]int$Bracket[[>]] class $Class_def[[Y]] {
+            using $Class[[Y]]$Bracket[[<]]0$Bracket[[>]]::$Unknown_dependentName[[xxx]];
+          };
+        };
+    )cpp"};
   for (const auto &TestCase : TestCases)
     // Mask off scope modifiers to keep the tests manageable.
     // They're tested separately.
diff --git a/clang-tools-extra/clangd/unittests/TUSchedulerTests.cpp b/clang-tools-extra/clangd/unittests/TUSchedulerTests.cpp
index ead85a4ce5f23..0538947b7095b 100644
--- a/clang-tools-extra/clangd/unittests/TUSchedulerTests.cpp
+++ b/clang-tools-extra/clangd/unittests/TUSchedulerTests.cpp
@@ -1309,6 +1309,13 @@ TEST_F(TUSchedulerTests, PublishWithStalePreamble) {
 
   // Make sure that we have eventual consistency.
   EXPECT_THAT(Collector.diagVersions().back(), Pair(PI.Version, PI.Version));
+
+  // Check that WantDiagnostics::No doesn't emit any diags.
+  PI.Version = "4";
+  PI.Contents = "#define FOO\n" + PI.Version;
+  S.update(File, PI, WantDiagnostics::No);
+  S.blockUntilIdle(timeoutSeconds(5));
+  EXPECT_THAT(Collector.diagVersions().back(), Pair("3", "3"));
 }
 
 // If a header file is missing from the CDB (or inferred using heuristics), and
diff --git a/clang-tools-extra/clangd/unittests/support/FileCacheTests.cpp b/clang-tools-extra/clangd/unittests/support/FileCacheTests.cpp
index 840cad623d776..3cdaceb5cf23c 100644
--- a/clang-tools-extra/clangd/unittests/support/FileCacheTests.cpp
+++ b/clang-tools-extra/clangd/unittests/support/FileCacheTests.cpp
@@ -11,9 +11,9 @@
 #include "TestFS.h"
 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
-#include <atomic>
 #include <chrono>
 #include <optional>
+#include <utility>
 
 namespace clang {
 namespace clangd {
@@ -34,10 +34,10 @@ class TestCache : public FileCache {
       FS.Files.erase(testPath("foo.cc"));
   }
 
-  std::string get(std::chrono::steady_clock::time_point FreshTime,
-                  bool ExpectParse) const {
+  std::pair<std::string, /*Parsed=*/bool>
+  get(std::chrono::steady_clock::time_point FreshTime) const {
     bool GotParse = false;
-    bool GotRead;
+    bool GotRead = false;
     std::string Result;
     read(
         FS, FreshTime,
@@ -49,12 +49,14 @@ class TestCache : public FileCache {
           GotRead = true;
           Result = Value;
         });
-    EXPECT_EQ(GotParse, ExpectParse);
     EXPECT_TRUE(GotRead);
-    return Result;
+    return {Result, GotParse};
   }
 };
 
+MATCHER_P(Parsed, Value, "") { return arg.second && arg.first == Value; }
+MATCHER_P(Cached, Value, "") { return !arg.second && arg.first == Value; }
+
 TEST(FileCacheTest, Invalidation) {
   TestCache C;
 
@@ -62,20 +64,20 @@ TEST(FileCacheTest, Invalidation) {
   auto MustBeFresh = StaleOK + std::chrono::hours(1);
 
   C.setContents("a");
-  EXPECT_EQ("a", C.get(StaleOK, /*ExpectParse=*/true)) << "Parsed first time";
-  EXPECT_EQ("a", C.get(StaleOK, /*ExpectParse=*/false)) << "Cached (time)";
-  EXPECT_EQ("a", C.get(MustBeFresh, /*ExpectParse=*/false)) << "Cached (stat)";
+  EXPECT_THAT(C.get(StaleOK), Parsed("a")) << "Parsed first time";
+  EXPECT_THAT(C.get(StaleOK), Cached("a")) << "Cached (time)";
+  EXPECT_THAT(C.get(MustBeFresh), Cached("a")) << "Cached (stat)";
   C.setContents("bb");
-  EXPECT_EQ("a", C.get(StaleOK, /*ExpectParse=*/false)) << "Cached (time)";
-  EXPECT_EQ("bb", C.get(MustBeFresh, /*ExpectParse=*/true)) << "Size changed";
-  EXPECT_EQ("bb", C.get(MustBeFresh, /*ExpectParse=*/true)) << "Cached (stat)";
+  EXPECT_THAT(C.get(StaleOK), Cached("a")) << "Cached (time)";
+  EXPECT_THAT(C.get(MustBeFresh), Parsed("bb")) << "Size changed";
+  EXPECT_THAT(C.get(MustBeFresh), Cached("bb")) << "Cached (stat)";
   C.setContents(nullptr);
-  EXPECT_EQ("bb", C.get(StaleOK, /*ExpectParse=*/false)) << "Cached (time)";
-  EXPECT_EQ("", C.get(MustBeFresh, /*ExpectParse=*/true)) << "Stat failed";
-  EXPECT_EQ("", C.get(MustBeFresh, /*ExpectParse=*/false)) << "Cached (404)";
+  EXPECT_THAT(C.get(StaleOK), Cached("bb")) << "Cached (time)";
+  EXPECT_THAT(C.get(MustBeFresh), Parsed("")) << "stat failed";
+  EXPECT_THAT(C.get(MustBeFresh), Cached("")) << "Cached (404)";
   C.setContents("bb"); // Match the previous stat values!
-  EXPECT_EQ("", C.get(StaleOK, /*ExpectParse=*/false)) << "Cached (time)";
-  EXPECT_EQ("bb", C.get(MustBeFresh, /*ExpectParse=*/true)) << "Size changed";
+  EXPECT_THAT(C.get(StaleOK), Cached("")) << "Cached (time)";
+  EXPECT_THAT(C.get(MustBeFresh), Parsed("bb")) << "Size changed";
 }
 
 } // namespace
diff --git a/clang-tools-extra/clangd/unittests/tweaks/SpecialMembersTests.cpp b/clang-tools-extra/clangd/unittests/tweaks/SpecialMembersTests.cpp
index 7fcaf41b426b1..eeb3f1d0fe4eb 100644
--- a/clang-tools-extra/clangd/unittests/tweaks/SpecialMembersTests.cpp
+++ b/clang-tools-extra/clangd/unittests/tweaks/SpecialMembersTests.cpp
@@ -7,8 +7,6 @@
 //===----------------------------------------------------------------------===//
 
 #include "TweakTesting.h"
-#include "gmock/gmock-matchers.h"
-#include "gmock/gmock.h"
 #include "gtest/gtest.h"
 
 namespace clang {
@@ -27,17 +25,17 @@ TEST_F(SpecialMembersTest, Test) {
                      "S &operator=(S&&); S &operator=(const S&);"
                      "};");
 
-  const char *Output = R"cpp(struct S{S(const S &) = default;
-  S(S &&) = default;
-  S &operator=(const S &) = default;
-  S &operator=(S &&) = default;
+  const char *Output = R"cpp(struct S{S(const S&) = default;
+S(S&&) = default;
+S &operator=(const S&) = default;
+S &operator=(S&&) = default;
 };)cpp";
   EXPECT_EQ(apply("struct ^S{};"), Output);
 
-  Output = R"cpp(struct S{S(const S &) = default;
-S(S &&) = default;
-S &operator=(const S &) = delete;
-S &operator=(S &&) = delete;
+  Output = R"cpp(struct S{S(const S&) = default;
+S(S&&) = default;
+S &operator=(const S&) = delete;
+S &operator=(S&&) = delete;
 int& ref;};)cpp";
   EXPECT_EQ(apply("struct ^S{int& ref;};"), Output);
 }
diff --git a/clang-tools-extra/clangd/unittests/tweaks/TweakTesting.cpp b/clang-tools-extra/clangd/unittests/tweaks/TweakTesting.cpp
index ee93436d708d2..51071d89a66e5 100644
--- a/clang-tools-extra/clangd/unittests/tweaks/TweakTesting.cpp
+++ b/clang-tools-extra/clangd/unittests/tweaks/TweakTesting.cpp
@@ -157,7 +157,8 @@ std::string TweakTest::decorate(llvm::StringRef Code, unsigned Point) {
 std::string TweakTest::decorate(llvm::StringRef Code,
                                 llvm::Annotations::Range Range) {
   return (Code.substr(0, Range.Begin) + "[[" +
-          Code.substr(Range.Begin, Range.End) + "]]" + Code.substr(Range.End))
+          Code.substr(Range.Begin, Range.End - Range.Begin) + "]]" +
+          Code.substr(Range.End))
       .str();
 }
 
diff --git a/clang-tools-extra/docs/ReleaseNotes.rst b/clang-tools-extra/docs/ReleaseNotes.rst
index e5c46de76cfcc..3f79e8e2a187a 100644
--- a/clang-tools-extra/docs/ReleaseNotes.rst
+++ b/clang-tools-extra/docs/ReleaseNotes.rst
@@ -114,7 +114,20 @@ New checks
 
   Warns when lambda specify a capture default and capture ``this``.
 
-- New :doc: `llvmlibc-inline-function-decl
+- New :doc:`cppcoreguidelines-avoid-capturing-lambda-coroutines
+  <clang-tidy/checks/cppcoreguidelines/avoid-capturing-lambda-coroutines>` check.
+
+  Flags C++20 coroutine lambdas with non-empty capture lists that may cause
+  use-after-free errors and suggests avoiding captures or ensuring the lambda
+  closure object has a guaranteed lifetime.
+
+- New :doc:`cppcoreguidelines-rvalue-reference-param-not-moved
+  <clang-tidy/checks/cppcoreguidelines/rvalue-reference-param-not-moved>` check.
+
+  Warns when an rvalue reference function parameter is never moved within
+  the function body.
+
+- New :doc:`llvmlibc-inline-function-decl
   <clang-tidy/checks/llvmlibc/inline-function-decl>` check.
 
   Checks that all implicit and explicit inline functions in header files are
@@ -133,6 +146,11 @@ New check aliases
 
 Changes in existing checks
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
+- Improved :doc:`readability-redundant-string-cstr
+  <clang-tidy/checks/readability/redundant-string-cstr>` check to recognise
+  unnecessary ``std::string::c_str()`` and ``std::string::data()`` calls in
+  arguments to ``std::print``, ``std::format`` or other functions listed in
+  the ``StringParameterFunction`` check option.
 
 - Deprecated check-local options `HeaderFileExtensions`
   in :doc:`bugprone-dynamic-static-initializers
@@ -169,6 +187,11 @@ Changes in existing checks
   <clang-tidy/checks/misc/unused-using-decls>` check.
   Global options of the same name should be used instead.
 
+- In :doc:`modernize-use-default-member-init
+  <clang-tidy/checks/modernize/use-default-member-init>` count template
+  constructors toward hand written constructors so that they are skipped if more
+  than one exists.
+
 - Fixed reading `HungarianNotation.CString.*` options in
   :doc:`readability-identifier-naming
   <clang-tidy/checks/readability/identifier-naming>` check.
@@ -178,6 +201,12 @@ Changes in existing checks
   :doc:`readability-identifier-naming
   <clang-tidy/checks/readability/identifier-naming>` check.
 
+- Updated the Hungarian prefixes for enums in C files to match those used in C++
+  files for improved readability, as checked by :doc:`readability-identifier-naming
+  <clang-tidy/checks/readability/identifier-naming>`. To preserve the previous
+  behavior of using `i` as the prefix for enum tags, set the `EnumConstantPrefix`
+  option to `i` instead of using `EnumConstantHungarianPrefix`.
+
 - Fixed a false positive in :doc:`readability-container-size-empty
   <clang-tidy/checks/readability/container-size-empty>` check when comparing
   ``std::array`` objects to default constructed ones. The behavior for this and
@@ -187,6 +216,24 @@ Changes in existing checks
   <clang-tidy/checks/bugprone/too-small-loop-variable>` check. Basic support
   for bit-field and integer members as a loop variable or upper limit were added.
 
+- Improved :doc:`readability-magic-numbers
+  <clang-tidy/checks/readability/magic-numbers>` check, now allows for
+  magic numbers in type aliases such as ``using`` and ``typedef`` declarations if
+  the new ``IgnoreTypeAliases`` option is set to true.
+
+- Fixed a false positive in :doc:`cppcoreguidelines-slicing
+  <clang-tidy/checks/cppcoreguidelines/slicing>` check when warning would be
+  emitted in constructor for virtual base class initialization.
+
+- Improved :doc:`bugprone-use-after-move
+  <clang-tidy/checks/bugprone/use-after-move>` to understand that there is a
+  sequence point between designated initializers.
+
+- Fixed an issue in :doc:`readability-identifier-naming
+  <clang-tidy/checks/readability/identifier-naming>` when specifying an empty
+  string for ``Prefix`` or ``Suffix`` options could result in the style not
+  being used.
+
 Removed checks
 ^^^^^^^^^^^^^^
 
diff --git a/clang-tools-extra/docs/clang-tidy/checks/cppcoreguidelines/avoid-capturing-lambda-coroutines.rst b/clang-tools-extra/docs/clang-tidy/checks/cppcoreguidelines/avoid-capturing-lambda-coroutines.rst
new file mode 100644
index 0000000000000..74ec5045a6bbf
--- /dev/null
+++ b/clang-tools-extra/docs/clang-tidy/checks/cppcoreguidelines/avoid-capturing-lambda-coroutines.rst
@@ -0,0 +1,54 @@
+.. title:: clang-tidy - cppcoreguidelines-avoid-capturing-lambda-coroutines
+
+cppcoreguidelines-avoid-capturing-lambda-coroutines
+===================================================
+
+Flags C++20 coroutine lambdas with non-empty capture lists that may cause
+use-after-free errors and suggests avoiding captures or ensuring the lambda
+closure object has a guaranteed lifetime.
+
+This check implements `CP.51
+<https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Rcoro-capture>`_
+from the C++ Core Guidelines.
+
+Using coroutine lambdas with non-empty capture lists can be risky, as capturing
+variables can lead to accessing freed memory after the first suspension point.
+This issue can occur even with refcounted smart pointers and copyable types.
+When a lambda expression creates a coroutine, it results in a closure object
+with storage, which is often on the stack and will eventually go out of scope.
+When the closure object goes out of scope, its captures also go out of scope.
+While normal lambdas finish executing before this happens, coroutine lambdas may
+resume from suspension after the closure object has been destructed, resulting
+in use-after-free memory access for all captures.
+
+Consider the following example:
+
+.. code-block:: c++
+
+    int value = get_value();
+    std::shared_ptr<Foo> sharedFoo = get_foo();
+    {
+        const auto lambda = [value, sharedFoo]() -> std::future<void>
+        {
+            co_await something();
+            // "sharedFoo" and "value" have already been destroyed
+            // the "shared" pointer didn't accomplish anything
+        };
+        lambda();
+    } // the lambda closure object has now gone out of scope
+
+In this example, the lambda object is defined with two captures: value and
+``sharedFoo``. When ``lambda()`` is called, the lambda object is created on the
+stack, and the captures are copied into the closure object. When the coroutine
+is suspended, the lambda object goes out of scope, and the closure object is
+destroyed. When the coroutine is resumed, the captured variables may have been
+destroyed, resulting in use-after-free bugs.
+
+In conclusion, the use of coroutine lambdas with non-empty capture lists can
+lead to use-after-free errors when resuming the coroutine after the closure
+object has been destroyed. This check helps prevent such errors by flagging
+C++20 coroutine lambdas with non-empty capture lists and suggesting avoiding
+captures or ensuring the lambda closure object has a guaranteed lifetime.
+
+Following these guidelines can help ensure the safe and reliable use of
+coroutine lambdas in C++ code.
diff --git a/clang-tools-extra/docs/clang-tidy/checks/cppcoreguidelines/rvalue-reference-param-not-moved.rst b/clang-tools-extra/docs/clang-tidy/checks/cppcoreguidelines/rvalue-reference-param-not-moved.rst
new file mode 100644
index 0000000000000..f4c58b3734e2f
--- /dev/null
+++ b/clang-tools-extra/docs/clang-tidy/checks/cppcoreguidelines/rvalue-reference-param-not-moved.rst
@@ -0,0 +1,73 @@
+.. title:: clang-tidy - cppcoreguidelines-rvalue-reference-param-not-moved
+
+cppcoreguidelines-rvalue-reference-param-not-moved
+==================================================
+
+Warns when an rvalue reference function parameter is never moved within
+the function body.
+
+Rvalue reference parameters indicate a parameter that should be moved with
+``std::move`` from within the function body. Any such parameter that is
+never moved is confusing and potentially indicative of a buggy program.
+
+Example:
+
+.. code-block:: c++
+
+  void logic(std::string&& Input) {
+    std::string Copy(Input); // Oops - forgot to std::move
+  }
+
+Options
+-------
+
+.. option:: AllowPartialMove
+
+   If set to `true`, the check accepts ``std::move`` calls containing any
+   subexpression containing the parameter. CppCoreGuideline F.18 officially
+   mandates that the parameter itself must be moved. Default is `false`.
+
+  .. code-block:: c++
+
+    // 'p' is flagged by this check if and only if AllowPartialMove is false
+    void move_members_of(pair<Obj, Obj>&& p) {
+      pair<Obj, Obj> other;
+      other.first = std::move(p.first);
+      other.second = std::move(p.second);
+    }
+
+    // 'p' is never flagged by this check
+    void move_whole_pair(pair<Obj, Obj>&& p) {
+      pair<Obj, Obj> other = std::move(p);
+    }
+
+.. option:: IgnoreUnnamedParams
+
+   If set to `true`, the check ignores unnamed rvalue reference parameters.
+   Default is `false`.
+
+.. option:: IgnoreNonDeducedTemplateTypes
+
+   If set to `true`, the check ignores non-deduced template type rvalue
+   reference parameters. Default is `false`.
+
+  .. code-block:: c++
+
+    template <class T>
+    struct SomeClass {
+      // Below, 'T' is not deduced and 'T&&' is an rvalue reference type.
+      // This will be flagged if and only if IgnoreNonDeducedTemplateTypes is
+      // false. One suggested fix would be to specialize the class for 'T' and
+      // 'T&' separately (e.g., see std::future), or allow only one of 'T' or
+      // 'T&' instantiations of SomeClass (e.g., see std::optional).
+      SomeClass(T&& t) { }
+    };
+
+    // Never flagged, since 'T' is a forwarding reference in a deduced context
+    template <class T>
+    void forwarding_ref(T&& t) {
+      T other = std::forward<T>(t);
+    }
+
+This check implements
+`CppCoreGuideline F.18 <http://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#f18-for-will-move-from-parameters-pass-by-x-and-stdmove-the-parameter>`_.
diff --git a/clang-tools-extra/docs/clang-tidy/checks/list.rst b/clang-tools-extra/docs/clang-tidy/checks/list.rst
index 5a2f9c877a961..f1945c1b5bf05 100644
--- a/clang-tools-extra/docs/clang-tidy/checks/list.rst
+++ b/clang-tools-extra/docs/clang-tidy/checks/list.rst
@@ -179,7 +179,8 @@ Clang-Tidy Checks
    `clang-analyzer-valist.Unterminated <clang-analyzer/valist.Unterminated.html>`_,
    `concurrency-mt-unsafe <concurrency/mt-unsafe.html>`_,
    `concurrency-thread-canceltype-asynchronous <concurrency/thread-canceltype-asynchronous.html>`_,
-   `cppcoreguidelines-avoid-capture-default-when-capturing-this <cppcoreguidelines/avoid-capture-default-when-capturing-this.html>`_,
+   `cppcoreguidelines-avoid-capture-default-when-capturing-this <cppcoreguidelines/avoid-capture-default-when-capturing-this.html>`_, "Yes"
+   `cppcoreguidelines-avoid-capturing-lambda-coroutines <cppcoreguidelines/avoid-capturing-lambda-coroutines.html>`_,
    `cppcoreguidelines-avoid-const-or-ref-data-members <cppcoreguidelines/avoid-const-or-ref-data-members.html>`_,
    `cppcoreguidelines-avoid-do-while <cppcoreguidelines/avoid-do-while.html>`_,
    `cppcoreguidelines-avoid-goto <cppcoreguidelines/avoid-goto.html>`_,
@@ -202,6 +203,7 @@ Clang-Tidy Checks
    `cppcoreguidelines-pro-type-static-cast-downcast <cppcoreguidelines/pro-type-static-cast-downcast.html>`_, "Yes"
    `cppcoreguidelines-pro-type-union-access <cppcoreguidelines/pro-type-union-access.html>`_,
    `cppcoreguidelines-pro-type-vararg <cppcoreguidelines/pro-type-vararg.html>`_,
+   `cppcoreguidelines-rvalue-reference-param-not-moved <cppcoreguidelines/rvalue-reference-param-not-moved.html>`_,
    `cppcoreguidelines-slicing <cppcoreguidelines/slicing.html>`_,
    `cppcoreguidelines-special-member-functions <cppcoreguidelines/special-member-functions.html>`_,
    `cppcoreguidelines-virtual-class-destructor <cppcoreguidelines/virtual-class-destructor.html>`_, "Yes"
diff --git a/clang-tools-extra/docs/clang-tidy/checks/readability/magic-numbers.rst b/clang-tools-extra/docs/clang-tidy/checks/readability/magic-numbers.rst
index b00ddd149cece..b5017f81d0a59 100644
--- a/clang-tools-extra/docs/clang-tidy/checks/readability/magic-numbers.rst
+++ b/clang-tools-extra/docs/clang-tidy/checks/readability/magic-numbers.rst
@@ -24,6 +24,16 @@ Examples of magic values:
 
 .. code-block:: c++
 
+   template<typename T, size_t N>
+   struct CustomType {
+      T arr[N];
+   };
+
+   struct OtherType {
+      CustomType<int, 30> container;
+   }
+   CustomType<int, 30> values;
+
    double circleArea = 3.1415926535 * radius * radius;
 
    double totalCharge = 1.08 * itemPrice;
@@ -40,6 +50,19 @@ Example with magic values refactored:
 
 .. code-block:: c++
 
+   template<typename T, size_t N>
+   struct CustomType {
+      T arr[N];
+   };
+
+   const size_t NUMBER_OF_ELEMENTS = 30;
+   using containerType = CustomType<int, NUMBER_OF_ELEMENTS>;
+   
+   struct OtherType {
+      containerType container;
+   }
+   containerType values;
+
    double circleArea = M_PI * radius * radius;
 
    const double TAX_RATE = 0.08;  // or make it variable and read from a file
@@ -116,3 +139,8 @@ Options
    Boolean value indicating whether to accept magic numbers as bit field widths
    without warning. This is useful for example for register definitions which
    are generated from hardware specifications. Default value is `true`.
+
+.. option:: IgnoreTypeAliases
+
+   Boolean value indicating whether to accept magic numbers in ``typedef`` or
+   ``using`` declarations. Default value is `false`.
diff --git a/clang-tools-extra/docs/clang-tidy/checks/readability/redundant-string-cstr.rst b/clang-tools-extra/docs/clang-tidy/checks/readability/redundant-string-cstr.rst
index e6760a41ca33c..2789f9c096ccf 100644
--- a/clang-tools-extra/docs/clang-tidy/checks/readability/redundant-string-cstr.rst
+++ b/clang-tools-extra/docs/clang-tidy/checks/readability/redundant-string-cstr.rst
@@ -5,3 +5,16 @@ readability-redundant-string-cstr
 
 
 Finds unnecessary calls to ``std::string::c_str()`` and ``std::string::data()``.
+
+Options
+-------
+
+.. option:: StringParameterFunctions
+
+   A semicolon-separated list of (fully qualified) function/method/operator
+   names, with the requirement that any parameter currently accepting a
+   ``const char*`` input should also be able to accept ``std::string``
+   inputs, or proper overload candidates that can do so should exist. This
+   can be used to configure functions such as ``fmt::format``,
+   ``spdlog::logger::info``, or wrappers around these and similar
+   functions. The default value is the empty string.
diff --git a/clang-tools-extra/include-cleaner/include/clang-include-cleaner/Analysis.h b/clang-tools-extra/include-cleaner/include/clang-include-cleaner/Analysis.h
index f6afaff09cfd9..cd11700548075 100644
--- a/clang-tools-extra/include-cleaner/include/clang-include-cleaner/Analysis.h
+++ b/clang-tools-extra/include-cleaner/include/clang-include-cleaner/Analysis.h
@@ -73,6 +73,8 @@ AnalysisResults analyze(llvm::ArrayRef<Decl *> ASTRoots,
 std::string fixIncludes(const AnalysisResults &Results, llvm::StringRef Code,
                         const format::FormatStyle &IncludeStyle);
 
+std::string spellHeader(const Header &H, HeaderSearch &HS,
+                        const FileEntry *Main);
 } // namespace include_cleaner
 } // namespace clang
 
diff --git a/clang-tools-extra/include-cleaner/include/clang-include-cleaner/Record.h b/clang-tools-extra/include-cleaner/include/clang-include-cleaner/Record.h
index 140713bf12807..27a2270f7e362 100644
--- a/clang-tools-extra/include-cleaner/include/clang-include-cleaner/Record.h
+++ b/clang-tools-extra/include-cleaner/include/clang-include-cleaner/Record.h
@@ -55,7 +55,7 @@ class PragmaIncludes {
   /// Returns true if the given #include of the main-file should never be
   /// removed.
   bool shouldKeep(unsigned HashLineNumber) const {
-    return ShouldKeep.find(HashLineNumber) != ShouldKeep.end();
+    return ShouldKeep.contains(HashLineNumber);
   }
 
   /// Returns the public mapping include for the given physical header file.
diff --git a/clang-tools-extra/include-cleaner/lib/Analysis.cpp b/clang-tools-extra/include-cleaner/lib/Analysis.cpp
index c5559db57e14c..6237bdb46babf 100644
--- a/clang-tools-extra/include-cleaner/lib/Analysis.cpp
+++ b/clang-tools-extra/include-cleaner/lib/Analysis.cpp
@@ -49,8 +49,8 @@ void walkUsed(llvm::ArrayRef<Decl *> ASTRoots,
   }
 }
 
-static std::string spellHeader(const Header &H, HeaderSearch &HS,
-                               const FileEntry *Main) {
+std::string spellHeader(const Header &H, HeaderSearch &HS,
+                        const FileEntry *Main) {
   switch (H.kind()) {
   case Header::Physical: {
     bool IsSystem = false;
diff --git a/clang-tools-extra/include-cleaner/lib/FindHeaders.cpp b/clang-tools-extra/include-cleaner/lib/FindHeaders.cpp
index ae2a3a858a38c..e273d429cd8c3 100644
--- a/clang-tools-extra/include-cleaner/lib/FindHeaders.cpp
+++ b/clang-tools-extra/include-cleaner/lib/FindHeaders.cpp
@@ -118,7 +118,7 @@ headersForSpecialSymbol(const Symbol &S, const SourceManager &SM,
   if (!FD)
     return {};
 
-  llvm::StringRef FName = FD->getName();
+  llvm::StringRef FName = symbolName(S);
   llvm::SmallVector<tooling::stdlib::Header> Headers;
   if (FName == "move") {
     if (FD->getNumParams() == 1)
diff --git a/clang-tools-extra/include-cleaner/lib/Record.cpp b/clang-tools-extra/include-cleaner/lib/Record.cpp
index 51fd39300d7c7..ff29fda8663b9 100644
--- a/clang-tools-extra/include-cleaner/lib/Record.cpp
+++ b/clang-tools-extra/include-cleaner/lib/Record.cpp
@@ -382,7 +382,7 @@ bool PragmaIncludes::isSelfContained(const FileEntry *FE) const {
 }
 
 bool PragmaIncludes::isPrivate(const FileEntry *FE) const {
-  return IWYUPublic.find(FE->getUniqueID()) != IWYUPublic.end();
+  return IWYUPublic.contains(FE->getUniqueID());
 }
 
 namespace {
diff --git a/clang-tools-extra/include-cleaner/test/nocrash.cpp b/clang-tools-extra/include-cleaner/test/nocrash.cpp
new file mode 100644
index 0000000000000..60e0f098867c6
--- /dev/null
+++ b/clang-tools-extra/include-cleaner/test/nocrash.cpp
@@ -0,0 +1,8 @@
+// RUN: clang-include-cleaner %s --
+
+namespace std {
+class Foo {};
+bool operator==(Foo, int) { return false; }
+}
+// no crash on a reference to a non-identifier symbol (operator ==).
+bool s = std::operator==(std::Foo(), 1);
diff --git a/clang-tools-extra/include-cleaner/unittests/LocateSymbolTest.cpp b/clang-tools-extra/include-cleaner/unittests/LocateSymbolTest.cpp
index 9961b013b24d8..d69e25bf8116d 100644
--- a/clang-tools-extra/include-cleaner/unittests/LocateSymbolTest.cpp
+++ b/clang-tools-extra/include-cleaner/unittests/LocateSymbolTest.cpp
@@ -16,15 +16,11 @@
 #include "clang/Testing/TestAST.h"
 #include "clang/Tooling/Inclusions/StandardLibrary.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Casting.h"
 #include "llvm/Testing/Annotations/Annotations.h"
 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
-#include <cstddef>
-#include <memory>
 #include <tuple>
-#include <unordered_map>
-#include <utility>
-#include <variant>
 #include <vector>
 
 namespace clang::include_cleaner {
@@ -66,8 +62,8 @@ struct LocateExample {
           ND = TD;
         if (ND->getName() == NameToFind) {
           EXPECT_TRUE(Out == nullptr || Out == ND->getCanonicalDecl())
-              << "Found multiple matches for " << NameToFind;
-          Out = cast<NamedDecl>(ND->getCanonicalDecl());
+              << "Found multiple matches for " << NameToFind.str();
+          Out = llvm::cast<NamedDecl>(ND->getCanonicalDecl());
         }
         return true;
       }
diff --git a/clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp b/clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp
index 7dcbf475e9869..68b6b217a2e01 100644
--- a/clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp
+++ b/clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp
@@ -171,6 +171,13 @@ TEST(WalkAST, TemplateNames) {
       namespace ns {template <typename> struct S {}; }
       using ns::$explicit^S;)cpp",
            "^S<int> x;");
+  testWalk(R"cpp(
+      namespace ns {
+        template <typename T> struct S { S(T);};
+        template <typename T> S(T t) -> S<T>;
+      }
+      using ns::$explicit^S;)cpp",
+      "^S x(123);");
   testWalk("template<typename> struct $explicit^S {};",
            R"cpp(
       template <template <typename> typename> struct X {};
diff --git a/clang-tools-extra/test/clang-tidy/checkers/Inputs/Headers/string b/clang-tools-extra/test/clang-tidy/checkers/Inputs/Headers/string
index 614480968f2aa..80b41da61c2e3 100644
--- a/clang-tools-extra/test/clang-tidy/checkers/Inputs/Headers/string
+++ b/clang-tools-extra/test/clang-tidy/checkers/Inputs/Headers/string
@@ -12,18 +12,22 @@ template <typename T>
 class allocator {};
 template <typename T>
 class char_traits {};
-template <typename C, typename T, typename A>
+template <typename C, typename T = char_traits<C>, typename A = allocator<C>>
 struct basic_string {
   typedef size_t size_type;
   typedef basic_string<C, T, A> _Type;
   basic_string();
   basic_string(const C *p, const A &a = A());
+  basic_string(const C *p, size_type count);
 
   ~basic_string();
 
   const C *c_str() const;
   const C *data() const;
 
+  bool empty() const;
+  size_type size() const;
+
   _Type& append(const C *s);
   _Type& append(const C *s, size_type n);
   _Type& assign(const C *s);
@@ -42,25 +46,29 @@ struct basic_string {
   _Type& insert(size_type pos, const C* s);
   _Type& insert(size_type pos, const C* s, size_type n);
 
+  _Type& operator[](size_type);
+  const _Type& operator[](size_type) const;
+
   _Type& operator+=(const _Type& str);
   _Type& operator+=(const C* s);
   _Type& operator=(const _Type& str);
   _Type& operator=(const C* s);
 };
 
-typedef basic_string<char, std::char_traits<char>, std::allocator<char>> string;
-typedef basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t>> wstring;
-typedef basic_string<char16, std::char_traits<char16>, std::allocator<char16>> u16string;
-typedef basic_string<char32, std::char_traits<char32>, std::allocator<char32>> u32string;
+typedef basic_string<char> string;
+typedef basic_string<wchar_t> wstring;
+typedef basic_string<char16> u16string;
+typedef basic_string<char32> u32string;
 
-template <typename C, typename T>
+template <typename C, typename T = char_traits<C>>
 struct basic_string_view {
-  basic_string_view(const C* s);
+  const C *str;
+  constexpr basic_string_view(const C* s) : str(s) {}
 };
-typedef basic_string_view<char, std::char_traits<char>> string_view;
-typedef basic_string_view<wchar_t, std::char_traits<wchar_t>> wstring_view;
-typedef basic_string_view<char16, std::char_traits<char16>> u16string_view;
-typedef basic_string_view<char32, std::char_traits<char32>> u32string_view;
+typedef basic_string_view<char> string_view;
+typedef basic_string_view<wchar_t> wstring_view;
+typedef basic_string_view<char16> u16string_view;
+typedef basic_string_view<char32> u32string_view;
 
 std::string operator+(const std::string&, const std::string&);
 std::string operator+(const std::string&, const char*);
@@ -69,6 +77,14 @@ std::string operator+(const char*, const std::string&);
 bool operator==(const std::string&, const std::string&);
 bool operator==(const std::string&, const char*);
 bool operator==(const char*, const std::string&);
+
+bool operator!=(const std::string&, const std::string&);
+bool operator!=(const std::string&, const char*);
+bool operator!=(const char*, const std::string&);
+
+bool operator==(const std::wstring&, const std::wstring&);
+bool operator==(const std::wstring&, const wchar_t*);
+bool operator==(const wchar_t*, const std::wstring&);
 }
 
 #endif // _STRING_
diff --git a/clang-tools-extra/test/clang-tidy/checkers/abseil/redundant-strcat-calls.cpp b/clang-tools-extra/test/clang-tidy/checkers/abseil/redundant-strcat-calls.cpp
index dad8ef857e2a1..ecd17bba293c5 100644
--- a/clang-tools-extra/test/clang-tidy/checkers/abseil/redundant-strcat-calls.cpp
+++ b/clang-tools-extra/test/clang-tidy/checkers/abseil/redundant-strcat-calls.cpp
@@ -1,92 +1,8 @@
-// RUN: %check_clang_tidy %s abseil-redundant-strcat-calls %t
+// RUN: %check_clang_tidy %s abseil-redundant-strcat-calls %t -- -- -isystem %clang_tidy_headers
+#include <string>
 
 int strlen(const char *);
 
-// Here we mimic the hierarchy of ::string.
-// We need to do so because we are matching on the fully qualified name of the
-// methods.
-struct __sso_string_base {};
-namespace __gnu_cxx {
-template <typename A, typename B, typename C, typename D = __sso_string_base>
-class __versa_string {
- public:
-  const char *c_str() const;
-  const char *data() const;
-  int size() const;
-  int capacity() const;
-  int length() const;
-  bool empty() const;
-  char &operator[](int);
-  void clear();
-  void resize(int);
-  int compare(const __versa_string &) const;
-};
-}  // namespace __gnu_cxx
-
-namespace std {
-template <typename T>
-class char_traits {};
-template <typename T>
-class allocator {};
-}  // namespace std
-
-template <typename A, typename B = std::char_traits<A>,
-          typename C = std::allocator<A>>
-class basic_string : public __gnu_cxx::__versa_string<A, B, C> {
- public:
-  basic_string();
-  basic_string(const basic_string &);
-  basic_string(const char *, C = C());
-  basic_string(const char *, int, C = C());
-  basic_string(const basic_string &, int, int, C = C());
-  ~basic_string();
-
-  basic_string &operator+=(const basic_string &);
-};
-
-template <typename A, typename B, typename C>
-basic_string<A, B, C> operator+(const basic_string<A, B, C> &,
-                                const basic_string<A, B, C> &);
-template <typename A, typename B, typename C>
-basic_string<A, B, C> operator+(const basic_string<A, B, C> &, const char *);
-
-typedef basic_string<char> string;
-
-bool operator==(const string &, const string &);
-bool operator==(const string &, const char *);
-bool operator==(const char *, const string &);
-
-bool operator!=(const string &, const string &);
-bool operator<(const string &, const string &);
-bool operator>(const string &, const string &);
-bool operator<=(const string &, const string &);
-bool operator>=(const string &, const string &);
-
-namespace std {
-template <typename _CharT, typename _Traits = char_traits<_CharT>,
-          typename _Alloc = allocator<_CharT>>
-class basic_string;
-
-template <typename _CharT, typename _Traits, typename _Alloc>
-class basic_string {
- public:
-  basic_string();
-  basic_string(const basic_string &);
-  basic_string(const char *, const _Alloc & = _Alloc());
-  basic_string(const char *, int, const _Alloc & = _Alloc());
-  basic_string(const basic_string &, int, int, const _Alloc & = _Alloc());
-  ~basic_string();
-
-  basic_string &operator+=(const basic_string &);
-
-  unsigned size() const;
-  unsigned length() const;
-  bool empty() const;
-};
-
-typedef basic_string<char> string;
-}  // namespace std
-
 namespace absl {
 
 class string_view {
@@ -95,12 +11,12 @@ class string_view {
 
   string_view();
   string_view(const char *);
-  string_view(const string &);
+  string_view(const std::string &);
   string_view(const char *, int);
   string_view(string_view, int);
 
   template <typename A>
-  explicit operator ::basic_string<char, traits_type, A>() const;
+  explicit operator std::basic_string<char, traits_type, A>() const;
 
   const char *data() const;
   int size() const;
@@ -113,7 +29,7 @@ struct AlphaNum {
   AlphaNum(int i);
   AlphaNum(double f);
   AlphaNum(const char *c_str);
-  AlphaNum(const string &str);
+  AlphaNum(const std::string &str);
   AlphaNum(const string_view &pc);
 
  private:
@@ -121,28 +37,28 @@ struct AlphaNum {
   AlphaNum &operator=(const AlphaNum &);
 };
 
-string StrCat();
-string StrCat(const AlphaNum &A);
-string StrCat(const AlphaNum &A, const AlphaNum &B);
-string StrCat(const AlphaNum &A, const AlphaNum &B, const AlphaNum &C);
-string StrCat(const AlphaNum &A, const AlphaNum &B, const AlphaNum &C,
-              const AlphaNum &D);
+std::string StrCat();
+std::string StrCat(const AlphaNum &A);
+std::string StrCat(const AlphaNum &A, const AlphaNum &B);
+std::string StrCat(const AlphaNum &A, const AlphaNum &B, const AlphaNum &C);
+std::string StrCat(const AlphaNum &A, const AlphaNum &B, const AlphaNum &C,
+                   const AlphaNum &D);
 
 // Support 5 or more arguments
 template <typename... AV>
-string StrCat(const AlphaNum &A, const AlphaNum &B, const AlphaNum &C,
+std::string StrCat(const AlphaNum &A, const AlphaNum &B, const AlphaNum &C,
               const AlphaNum &D, const AlphaNum &E, const AV &... args);
 
-void StrAppend(string *Dest, const AlphaNum &A);
-void StrAppend(string *Dest, const AlphaNum &A, const AlphaNum &B);
-void StrAppend(string *Dest, const AlphaNum &A, const AlphaNum &B,
-               const AlphaNum &C);
-void StrAppend(string *Dest, const AlphaNum &A, const AlphaNum &B,
-               const AlphaNum &C, const AlphaNum &D);
+void StrAppend(std::string *Dest, const AlphaNum &A);
+void StrAppend(std::string *Dest, const AlphaNum &A, const AlphaNum &B);
+void StrAppend(std::string *Dest, const AlphaNum &A, const AlphaNum &B,
+                    const AlphaNum &C);
+void StrAppend(std::string *Dest, const AlphaNum &A, const AlphaNum &B,
+                    const AlphaNum &C, const AlphaNum &D);
 
 // Support 5 or more arguments
 template <typename... AV>
-void StrAppend(string *Dest, const AlphaNum &A, const AlphaNum &B,
+void StrAppend(std::string *Dest, const AlphaNum &A, const AlphaNum &B,
                const AlphaNum &C, const AlphaNum &D, const AlphaNum &E,
                const AV &... args);
 
@@ -153,8 +69,8 @@ using absl::StrAppend;
 using absl::StrCat;
 
 void Positives() {
-  string S = StrCat(1, StrCat("A", StrCat(1.1)));
-  // CHECK-MESSAGES: [[@LINE-1]]:14: warning: multiple calls to 'absl::StrCat' can be flattened into a single call
+  std::string S = StrCat(1, StrCat("A", StrCat(1.1)));
+  // CHECK-MESSAGES: [[@LINE-1]]:19: warning: multiple calls to 'absl::StrCat' can be flattened into a single call
   // CHECK-FIXES: string S = StrCat(1, "A", 1.1);
 
   S = StrCat(StrCat(StrCat(StrCat(StrCat(1)))));
@@ -190,7 +106,7 @@ void Positives() {
 
 void Negatives() {
   // One arg. It is used for conversion. Ignore.
-  string S = StrCat(1);
+  std::string S = StrCat(1);
 
 #define A_MACRO(x, y, z) StrCat(x, y, z)
   S = A_MACRO(1, 2, StrCat("A", "B"));
diff --git a/clang-tools-extra/test/clang-tidy/checkers/bugprone/too-small-loop-variable.cpp b/clang-tools-extra/test/clang-tidy/checkers/bugprone/too-small-loop-variable.cpp
index 1505dbfef519e..7821c4144ccfe 100644
--- a/clang-tools-extra/test/clang-tidy/checkers/bugprone/too-small-loop-variable.cpp
+++ b/clang-tools-extra/test/clang-tidy/checkers/bugprone/too-small-loop-variable.cpp
@@ -373,3 +373,20 @@ void badForLoopWithBitfieldOnLoopVarAndUpperBoundOnPtr() {
   }
 }
 
+void goodForLoopWithBitfieldOnUpperBoundOnly() {
+  struct S {
+    int x : 4;
+  } s;
+
+  for (int i = 10; i > s.x; --i) {
+  }
+}
+
+void goodForLoopWithIntegersOnUpperBoundOnly() {
+  struct S {
+    short x;
+  } s;
+
+  for (int i = 10; i > s.x; --i) {
+  }
+}
diff --git a/clang-tools-extra/test/clang-tidy/checkers/bugprone/use-after-move.cpp b/clang-tools-extra/test/clang-tidy/checkers/bugprone/use-after-move.cpp
index 281f2083857ad..45cef8abfd1f6 100644
--- a/clang-tools-extra/test/clang-tidy/checkers/bugprone/use-after-move.cpp
+++ b/clang-tools-extra/test/clang-tidy/checkers/bugprone/use-after-move.cpp
@@ -1155,18 +1155,32 @@ void initializerListSequences() {
       int i;
       A a;
     };
-    A a;
-    S1 s1{a.getInt(), std::move(a)};
+    {
+      A a;
+      S1 s1{a.getInt(), std::move(a)};
+    }
+    {
+      A a;
+      S1 s1{.i = a.getInt(), .a = std::move(a)};
+    }
   }
   {
     struct S2 {
       A a;
       int i;
     };
-    A a;
-    S2 s2{std::move(a), a.getInt()};
-    // CHECK-NOTES: [[@LINE-1]]:25: warning: 'a' used after it was moved
-    // CHECK-NOTES: [[@LINE-2]]:11: note: move occurred here
+    {
+      A a;
+      S2 s2{std::move(a), a.getInt()};
+      // CHECK-NOTES: [[@LINE-1]]:27: warning: 'a' used after it was moved
+      // CHECK-NOTES: [[@LINE-2]]:13: note: move occurred here
+    }
+    {
+      A a;
+      S2 s2{.a = std::move(a), .i = a.getInt()};
+      // CHECK-NOTES: [[@LINE-1]]:37: warning: 'a' used after it was moved
+      // CHECK-NOTES: [[@LINE-2]]:13: note: move occurred here
+    }
   }
 }
 
diff --git a/clang-tools-extra/test/clang-tidy/checkers/cppcoreguidelines/Inputs/system/coroutines.h b/clang-tools-extra/test/clang-tidy/checkers/cppcoreguidelines/Inputs/system/coroutines.h
new file mode 100644
index 0000000000000..ed4373394351b
--- /dev/null
+++ b/clang-tools-extra/test/clang-tidy/checkers/cppcoreguidelines/Inputs/system/coroutines.h
@@ -0,0 +1,32 @@
+#pragma once
+
+namespace std {
+
+template <typename ret_t, typename... args_t>
+struct coroutine_traits {
+  using promise_type = typename ret_t::promise_type;
+};
+
+template <class promise_t>
+struct coroutine_handle {
+  static constexpr coroutine_handle from_address(void *addr) noexcept { return {}; };
+};
+
+} // namespace std
+
+struct never_suspend {
+  bool await_ready() noexcept { return false; }
+  template <typename coro_t>
+  void await_suspend(coro_t handle) noexcept {}
+  void await_resume() noexcept {}
+};
+
+struct task {
+  struct promise_type {
+    task get_return_object() noexcept { return {}; }
+    never_suspend initial_suspend() noexcept { return {}; }
+    never_suspend final_suspend() noexcept { return {}; }
+    void return_void() {}
+    void unhandled_exception() {}
+  };
+};
diff --git a/clang-tools-extra/test/clang-tidy/checkers/cppcoreguidelines/avoid-capturing-lambda-coroutines.cpp b/clang-tools-extra/test/clang-tidy/checkers/cppcoreguidelines/avoid-capturing-lambda-coroutines.cpp
new file mode 100644
index 0000000000000..6670f4a5420be
--- /dev/null
+++ b/clang-tools-extra/test/clang-tidy/checkers/cppcoreguidelines/avoid-capturing-lambda-coroutines.cpp
@@ -0,0 +1,36 @@
+// RUN: %check_clang_tidy -std=c++20-or-later %s cppcoreguidelines-avoid-capturing-lambda-coroutines %t -- -- -isystem %S/Inputs/system
+
+#include <coroutines.h>
+
+void Caught() {
+    int v;
+
+    [&] () -> task { int y = v; co_return; };
+    // CHECK-MESSAGES: [[@LINE-1]]:5: warning: coroutine lambda may cause use-after-free, avoid captures or ensure lambda closure object has guaranteed lifetime [cppcoreguidelines-avoid-capturing-lambda-coroutines]
+    [=] () -> task { int y = v; co_return; };
+    // CHECK-MESSAGES: [[@LINE-1]]:5: warning: coroutine lambda may cause use-after-free, avoid captures or ensure lambda closure object has guaranteed lifetime [cppcoreguidelines-avoid-capturing-lambda-coroutines]
+    [v] () -> task { co_return; };
+    // CHECK-MESSAGES: [[@LINE-1]]:5: warning: coroutine lambda may cause use-after-free, avoid captures or ensure lambda closure object has guaranteed lifetime [cppcoreguidelines-avoid-capturing-lambda-coroutines]
+    [&v] () -> task { co_return; };
+    // CHECK-MESSAGES: [[@LINE-1]]:5: warning: coroutine lambda may cause use-after-free, avoid captures or ensure lambda closure object has guaranteed lifetime [cppcoreguidelines-avoid-capturing-lambda-coroutines]
+    [y=v] () -> task { co_return; };
+    // CHECK-MESSAGES: [[@LINE-1]]:5: warning: coroutine lambda may cause use-after-free, avoid captures or ensure lambda closure object has guaranteed lifetime [cppcoreguidelines-avoid-capturing-lambda-coroutines]
+    [y{v}] () -> task { co_return; };
+    // CHECK-MESSAGES: [[@LINE-1]]:5: warning: coroutine lambda may cause use-after-free, avoid captures or ensure lambda closure object has guaranteed lifetime [cppcoreguidelines-avoid-capturing-lambda-coroutines]
+}
+
+struct S {
+    void m() {
+        [this] () -> task { co_return; };
+        // CHECK-MESSAGES: [[@LINE-1]]:9: warning: coroutine lambda may cause use-after-free, avoid captures or ensure lambda closure object has guaranteed lifetime [cppcoreguidelines-avoid-capturing-lambda-coroutines]
+    }
+};
+
+void Safe() {
+    int v;
+    [] () -> task { co_return; };
+    [&] () -> task { co_return; };
+    [=] () -> task { co_return; };
+
+    [&v]{++v;}();
+}
diff --git a/clang-tools-extra/test/clang-tidy/checkers/cppcoreguidelines/rvalue-reference-param-not-moved.cpp b/clang-tools-extra/test/clang-tidy/checkers/cppcoreguidelines/rvalue-reference-param-not-moved.cpp
new file mode 100644
index 0000000000000..9ee26e851bfc2
--- /dev/null
+++ b/clang-tools-extra/test/clang-tidy/checkers/cppcoreguidelines/rvalue-reference-param-not-moved.cpp
@@ -0,0 +1,328 @@
+// RUN: %check_clang_tidy -std=c++11 %s cppcoreguidelines-rvalue-reference-param-not-moved %t -- \
+// RUN: -config="{CheckOptions: [{key: cppcoreguidelines-rvalue-reference-param-not-moved.AllowPartialMove, value: true},{key: cppcoreguidelines-rvalue-reference-param-not-moved.IgnoreUnnamedParams, value: true},{key: cppcoreguidelines-rvalue-reference-param-not-moved.IgnoreNonDeducedTemplateTypes, value: true}]}" -- -fno-delayed-template-parsing
+// RUN: %check_clang_tidy -check-suffix=,CXX14 -std=c++14 %s cppcoreguidelines-rvalue-reference-param-not-moved %t -- \
+// RUN: -config="{CheckOptions: [{key: cppcoreguidelines-rvalue-reference-param-not-moved.AllowPartialMove, value: true},{key: cppcoreguidelines-rvalue-reference-param-not-moved.IgnoreUnnamedParams, value: true},{key: cppcoreguidelines-rvalue-reference-param-not-moved.IgnoreNonDeducedTemplateTypes, value: true}]}" -- -fno-delayed-template-parsing
+// RUN: %check_clang_tidy -check-suffix=,NOSUBEXPR -std=c++11 %s cppcoreguidelines-rvalue-reference-param-not-moved %t -- \
+// RUN: -config="{CheckOptions: [{key: cppcoreguidelines-rvalue-reference-param-not-moved.AllowPartialMove, value: false},{key: cppcoreguidelines-rvalue-reference-param-not-moved.IgnoreUnnamedParams, value: true},{key: cppcoreguidelines-rvalue-reference-param-not-moved.IgnoreNonDeducedTemplateTypes, value: true}]}" -- -fno-delayed-template-parsing
+// RUN: %check_clang_tidy -check-suffix=,UNNAMED -std=c++11 %s cppcoreguidelines-rvalue-reference-param-not-moved %t -- \
+// RUN: -config="{CheckOptions: [{key: cppcoreguidelines-rvalue-reference-param-not-moved.AllowPartialMove, value: true},{key: cppcoreguidelines-rvalue-reference-param-not-moved.IgnoreUnnamedParams, value: false},{key: cppcoreguidelines-rvalue-reference-param-not-moved.IgnoreNonDeducedTemplateTypes, value: true}]}" -- -fno-delayed-template-parsing
+// RUN: %check_clang_tidy -check-suffix=,NONDEDUCED -std=c++11 %s cppcoreguidelines-rvalue-reference-param-not-moved %t -- \
+// RUN: -config="{CheckOptions: [{key: cppcoreguidelines-rvalue-reference-param-not-moved.AllowPartialMove, value: true},{key: cppcoreguidelines-rvalue-reference-param-not-moved.IgnoreUnnamedParams, value: true},{key: cppcoreguidelines-rvalue-reference-param-not-moved.IgnoreNonDeducedTemplateTypes, value: false}]}" -- -fno-delayed-template-parsing
+
+// NOLINTBEGIN
+namespace std {
+template <typename>
+struct remove_reference;
+
+template <typename _Tp> struct remove_reference { typedef _Tp type; };
+template <typename _Tp> struct remove_reference<_Tp&> { typedef _Tp type; };
+template <typename _Tp> struct remove_reference<_Tp&&> { typedef _Tp type; };
+
+template <typename _Tp>
+constexpr typename std::remove_reference<_Tp>::type &&move(_Tp &&__t) noexcept;
+
+template <typename _Tp>
+constexpr _Tp &&
+forward(typename remove_reference<_Tp>::type &__t) noexcept;
+
+}
+// NOLINTEND
+
+struct Obj {
+  Obj();
+  Obj(const Obj&);
+  Obj& operator=(const Obj&);
+  Obj(Obj&&);
+  Obj& operator=(Obj&&);
+  void member() const;
+};
+
+void consumes_object(Obj);
+
+void never_moves_param(Obj&& o) {
+  // CHECK-MESSAGES: :[[@LINE-1]]:30: warning: rvalue reference parameter 'o' is never moved from inside the function body [cppcoreguidelines-rvalue-reference-param-not-moved]
+  o.member();
+}
+
+void just_a_declaration(Obj&& o);
+
+void copies_object(Obj&& o) {
+  // CHECK-MESSAGES: :[[@LINE-1]]:26: warning: rvalue reference parameter 'o' is never moved from inside the function body [cppcoreguidelines-rvalue-reference-param-not-moved]
+  Obj copy = o;
+}
+
+template <typename T>
+void never_moves_param_template(Obj&& o, T t) {
+  // CHECK-MESSAGES: :[[@LINE-1]]:39: warning: rvalue reference parameter 'o' is never moved from inside the function body [cppcoreguidelines-rvalue-reference-param-not-moved]
+  o.member();
+}
+
+void never_moves_params(Obj&& o1, Obj&& o2) {
+  // CHECK-MESSAGES: :[[@LINE-1]]:31: warning: rvalue reference parameter 'o1' is never moved from inside the function body [cppcoreguidelines-rvalue-reference-param-not-moved]
+  // CHECK-MESSAGES: :[[@LINE-2]]:41: warning: rvalue reference parameter 'o2' is never moved from inside the function body [cppcoreguidelines-rvalue-reference-param-not-moved]
+}
+
+void never_moves_some_params(Obj&& o1, Obj&& o2) {
+  // CHECK-MESSAGES: :[[@LINE-1]]:36: warning: rvalue reference parameter 'o1' is never moved from inside the function body [cppcoreguidelines-rvalue-reference-param-not-moved]
+
+  Obj other{std::move(o2)};
+}
+
+void never_moves_unnamed(Obj&&) {}
+// CHECK-MESSAGES-UNNAMED: :[[@LINE-1]]:31: warning: rvalue reference parameter '' is never moved from inside the function body [cppcoreguidelines-rvalue-reference-param-not-moved]
+
+void never_moves_mixed(Obj o1, Obj&& o2) {
+  // CHECK-MESSAGES: :[[@LINE-1]]:38: warning: rvalue reference parameter 'o2' is never moved from inside the function body [cppcoreguidelines-rvalue-reference-param-not-moved]
+}
+
+void lambda_captures_parameter_as_value(Obj&& o) {
+  auto f = [o]() {
+    consumes_object(std::move(o));
+  };
+  // CHECK-MESSAGES: :[[@LINE-4]]:47: warning: rvalue reference parameter 'o' is never moved from inside the function body [cppcoreguidelines-rvalue-reference-param-not-moved]
+}
+
+void lambda_captures_parameter_as_value_nested(Obj&& o) {
+  // CHECK-MESSAGES: :[[@LINE-1]]:54: warning: rvalue reference parameter 'o' is never moved from inside the function body [cppcoreguidelines-rvalue-reference-param-not-moved]
+  auto f = [&o]() {
+    auto f_nested = [o]() {
+      consumes_object(std::move(o));
+    };
+  };
+  auto f2 = [o]() {
+    auto f_nested = [&o]() {
+      consumes_object(std::move(o));
+    };
+  };
+  auto f3 = [o]() {
+    auto f_nested = [&o]() {
+      auto f_nested_inner = [&o]() {
+        consumes_object(std::move(o));
+      };
+    };
+  };
+  auto f4 = [&o]() {
+    auto f_nested = [&o]() {
+      auto f_nested_inner = [o]() {
+        consumes_object(std::move(o));
+      };
+    };
+  };
+}
+
+void misc_lambda_checks() {
+  auto never_moves = [](Obj&& o1) {
+    Obj other{o1};
+  };
+  // CHECK-MESSAGES: :[[@LINE-3]]:31: warning: rvalue reference parameter 'o1' is never moved from inside the function body [cppcoreguidelines-rvalue-reference-param-not-moved]
+
+#if __cplusplus >= 201402L
+  auto never_moves_with_auto_param = [](Obj&& o1, auto& v) {
+    Obj other{o1};
+  };
+  // CHECK-MESSAGES-CXX14: :[[@LINE-3]]:47: warning: rvalue reference parameter 'o1' is never moved from inside the function body [cppcoreguidelines-rvalue-reference-param-not-moved]
+#endif
+}
+
+template <typename T>
+void forwarding_ref(T&& t) {
+  t.member();
+}
+
+template <typename T>
+void forwarding_ref_forwarded(T&& t) {
+  forwarding_ref(std::forward<T>(t));
+}
+
+template <typename... Ts>
+void type_pack(Ts&&... ts) {
+  (forwarding_ref(std::forward<Ts>(ts)), ...);
+}
+
+void call_forwarding_functions() {
+  Obj o;
+
+  forwarding_ref(Obj{});
+  type_pack(Obj{});
+  type_pack(Obj{}, o);
+  type_pack(Obj{}, Obj{});
+}
+
+void moves_parameter(Obj&& o) {
+  Obj moved = std::move(o);
+}
+
+void moves_parameter_extra_parens(Obj&& o) {
+  Obj moved = std::move((o));
+}
+
+template <typename T1, typename T2>
+struct mypair {
+  T1 first;
+  T2 second;
+};
+
+void moves_member_of_parameter(mypair<Obj, Obj>&& pair) {
+  // CHECK-MESSAGES-NOSUBEXPR: :[[@LINE-1]]:51: warning: rvalue reference parameter 'pair' is never moved from inside the function body [cppcoreguidelines-rvalue-reference-param-not-moved]
+  Obj a = std::move(pair.first);
+  Obj b = std::move(pair.second);
+}
+
+template <typename T>
+struct myoptional {
+  T& operator*() &;
+  T&& operator*() &&;
+};
+
+void moves_deref_optional(myoptional<Obj>&& opt) {
+  // CHECK-MESSAGES-NOSUBEXPR: :[[@LINE-1]]:45: warning: rvalue reference parameter 'opt' is never moved from inside the function body [cppcoreguidelines-rvalue-reference-param-not-moved]
+  Obj other = std::move(*opt);
+}
+
+void moves_optional_then_deref_resulting_rvalue(myoptional<Obj>&& opt) {
+  Obj other = *std::move(opt);
+}
+
+void pass_by_lvalue_reference(Obj& o) {
+  o.member();
+}
+
+void pass_by_value(Obj o) {
+  o.member();
+}
+
+void pass_by_const_lvalue_reference(const Obj& o) {
+  o.member();
+}
+
+void pass_by_const_lvalue_reference(const Obj&& o) {
+  o.member();
+}
+
+void lambda_captures_parameter_as_reference(Obj&& o) {
+  auto f = [&o]() {
+    consumes_object(std::move(o));
+  };
+}
+
+void lambda_captures_parameter_as_reference_nested(Obj&& o) {
+  auto f = [&o]() {
+    auto f_nested = [&o]() {
+      auto f_nested2 = [&o]() {
+        consumes_object(std::move(o));
+      };
+    };
+  };
+}
+
+#if __cplusplus >= 201402L
+void lambda_captures_parameter_generalized(Obj&& o) {
+  auto f = [o = std::move(o)]() {
+    consumes_object(std::move(o));
+  };
+}
+#endif
+
+void negative_lambda_checks() {
+  auto never_moves_nested = [](Obj&& o1) {
+    auto nested = [&]() {
+      Obj other{std::move(o1)};
+    };
+  };
+
+#if __cplusplus >= 201402L
+  auto auto_lvalue_ref_param = [](auto& o1) {
+    Obj other{o1};
+  };
+
+  auto auto_forwarding_ref_param = [](auto&& o1) {
+    Obj other{o1};
+  };
+
+  auto does_move_auto_rvalue_ref_param = [](auto&& o1) {
+    Obj other{std::forward(o1)};
+  };
+#endif
+
+  auto does_move = [](Obj&& o1) {
+    Obj other{std::move(o1)};
+  };
+
+  auto not_rvalue_ref = [](Obj& o1) {
+    Obj other{std::move(o1)};
+  };
+
+  Obj local;
+  auto captures = [local]() { };
+}
+
+struct AClass {
+  void member_with_lambda_no_move(Obj&& o) {
+    // CHECK-MESSAGES: :[[@LINE-1]]:41: warning: rvalue reference parameter 'o' is never moved from inside the function body [cppcoreguidelines-rvalue-reference-param-not-moved]
+    auto captures_this = [=, this]() {
+      Obj other = std::move(o);
+    };
+  }
+  void member_with_lambda_that_moves(Obj&& o) {
+    auto captures_this = [&, this]() {
+      Obj other = std::move(o);
+    };
+  }
+};
+
+void useless_move(Obj&& o) {
+  // FIXME - The object is not actually moved from - this should probably be
+  // flagged by *some* check. Which one?
+  std::move(o);
+}
+
+template <typename>
+class TemplatedClass;
+
+template <typename T>
+void unresolved_lookup(TemplatedClass<T>&& o) {
+  TemplatedClass<T> moved = std::move(o);
+}
+
+struct DefinesMove {
+  DefinesMove(DefinesMove&& rhs) : o(std::move(rhs.o)) { }
+  DefinesMove& operator=(DefinesMove&& rhs) {
+    if (this != &rhs) {
+      o = std::move(rhs.o);
+    }
+    return *this;
+  }
+  Obj o;
+};
+
+struct DeclaresMove {
+  DeclaresMove(DeclaresMove&& rhs);
+  DeclaresMove& operator=(DeclaresMove&& rhs);
+};
+
+struct AnotherObj {
+  AnotherObj(Obj&& o) : o(std::move(o)) {}
+  AnotherObj(Obj&& o, int) { o = std::move(o); }
+  Obj o;
+};
+
+template <class T>
+struct AClassTemplate {
+  AClassTemplate(T&& t) {}
+  // CHECK-MESSAGES-NONDEDUCED: :[[@LINE-1]]:22: warning: rvalue reference parameter 't' is never moved from inside the function body [cppcoreguidelines-rvalue-reference-param-not-moved]
+
+  void moves(T&& t) {
+    T other = std::move(t);
+  }
+  void never_moves(T&& t) {}
+  // CHECK-MESSAGES-NONDEDUCED: :[[@LINE-1]]:24: warning: rvalue reference parameter 't' is never moved from inside the function body [cppcoreguidelines-rvalue-reference-param-not-moved]
+};
+
+void instantiate_a_class_template() {
+  Obj o;
+  AClassTemplate<Obj> withObj{std::move(o)};
+  withObj.never_moves(std::move(o));
+
+  AClassTemplate<Obj&> withObjRef(o);
+  withObjRef.never_moves(o);
+}
diff --git a/clang-tools-extra/test/clang-tidy/checkers/cppcoreguidelines/slicing.cpp b/clang-tools-extra/test/clang-tidy/checkers/cppcoreguidelines/slicing.cpp
index 6856f52f7061d..58ca644f18d43 100644
--- a/clang-tools-extra/test/clang-tidy/checkers/cppcoreguidelines/slicing.cpp
+++ b/clang-tools-extra/test/clang-tidy/checkers/cppcoreguidelines/slicing.cpp
@@ -98,3 +98,25 @@ void negatives() {
   a = h;
   // CHECK-MESSAGES: :[[@LINE-1]]:5: warning: slicing object from type 'DerivedThatOverridesH' to 'Base' discards override 'h'
 }
+
+namespace PR31187 {
+// Don't warn when calling constructor of base virtual class, from
+// initialization list of derived class constructor.
+
+struct BaseA {
+virtual ~BaseA() {}
+virtual void foo() {}
+
+int i;
+};
+
+struct BaseB : virtual BaseA {
+virtual void foo() {}
+};
+
+struct ClassWithVirtualBases : BaseB {
+  ClassWithVirtualBases(const BaseB& other) : BaseA(other), BaseB(other) {}
+  ClassWithVirtualBases(const ClassWithVirtualBases& other) : BaseA(other), BaseB(other) {}
+};
+
+}
diff --git a/clang-tools-extra/test/clang-tidy/checkers/modernize/use-default-member-init.cpp b/clang-tools-extra/test/clang-tidy/checkers/modernize/use-default-member-init.cpp
index 464cfbdeacb88..3eb6331d0a9f2 100644
--- a/clang-tools-extra/test/clang-tidy/checkers/modernize/use-default-member-init.cpp
+++ b/clang-tools-extra/test/clang-tidy/checkers/modernize/use-default-member-init.cpp
@@ -60,6 +60,12 @@ struct TwoConstructors {
     int i;
 };
 
+struct TwoConstructorsTpl {
+  TwoConstructorsTpl() : i{7} {}
+  template <typename T> TwoConstructorsTpl(T, int) : i(8) {}
+  int i;
+};
+
 struct PositiveNotDefaultOOLInt {
   PositiveNotDefaultOOLInt(int);
   int i;
diff --git a/clang-tools-extra/test/clang-tidy/checkers/readability/container-data-pointer.cpp b/clang-tools-extra/test/clang-tidy/checkers/readability/container-data-pointer.cpp
index 7f75564ad9dc9..ee2ba6a48752c 100644
--- a/clang-tools-extra/test/clang-tidy/checkers/readability/container-data-pointer.cpp
+++ b/clang-tools-extra/test/clang-tidy/checkers/readability/container-data-pointer.cpp
@@ -1,4 +1,5 @@
-// RUN: %check_clang_tidy %s readability-container-data-pointer %t -- -- -fno-delayed-template-parsing
+// RUN: %check_clang_tidy %s readability-container-data-pointer %t -- -- -isystem %clang_tidy_headers -fno-delayed-template-parsing
+#include <string>
 
 typedef __SIZE_TYPE__ size_t;
 
@@ -17,22 +18,6 @@ struct vector {
   const T &operator[](size_type) const;
 };
 
-template <typename T>
-struct basic_string {
-  using size_type = size_t;
-
-  basic_string();
-
-  T *data();
-  const T *data() const;
-
-  T &operator[](size_t);
-  const T &operator[](size_type) const;
-};
-
-typedef basic_string<char> string;
-typedef basic_string<wchar_t> wstring;
-
 template <typename T>
 struct is_integral;
 
diff --git a/clang-tools-extra/test/clang-tidy/checkers/readability/container-size-empty.cpp b/clang-tools-extra/test/clang-tidy/checkers/readability/container-size-empty.cpp
index 47606070d69c1..28ce8d5f8cd32 100644
--- a/clang-tools-extra/test/clang-tidy/checkers/readability/container-size-empty.cpp
+++ b/clang-tools-extra/test/clang-tidy/checkers/readability/container-size-empty.cpp
@@ -1,6 +1,7 @@
 // RUN: %check_clang_tidy %s readability-container-size-empty %t -- \
 // RUN: -config="{CheckOptions: [{key: readability-container-size-empty.ExcludedComparisonTypes , value: '::std::array;::IgnoredDummyType'}]}" \
-// RUN: -- -fno-delayed-template-parsing
+// RUN: -- -fno-delayed-template-parsing -isystem %clang_tidy_headers
+#include <string>
 
 namespace std {
 template <typename T> struct vector {
@@ -11,20 +12,6 @@ template <typename T> struct vector {
   bool empty() const;
 };
 
-template <typename T> struct basic_string {
-  basic_string();
-  bool operator==(const basic_string<T>& other) const;
-  bool operator!=(const basic_string<T>& other) const;
-  bool operator==(const char *) const;
-  bool operator!=(const char *) const;
-  basic_string<T> operator+(const basic_string<T>& other) const;
-  unsigned long size() const;
-  bool empty() const;
-};
-
-typedef basic_string<char> string;
-typedef basic_string<wchar_t> wstring;
-
 inline namespace __v2 {
 template <typename T> struct set {
   set();
@@ -125,12 +112,12 @@ bool returnsBool() {
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:7: warning: the 'empty' method should be used to check for emptiness instead of 'size' [readability-container-size-empty]
   // CHECK-FIXES: {{^  }}if (intSet.empty()){{$}}
-  // CHECK-MESSAGES: :34:8: note: method 'set'::empty() defined here
+  // CHECK-MESSAGES: :21:8: note: method 'set'::empty() defined here
   if (intSet == std::set<int>())
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:7: warning: the 'empty' method should be used to check for emptiness
   // CHECK-FIXES: {{^  }}if (intSet.empty()){{$}}
-  // CHECK-MESSAGES: :34:8: note: method 'set'::empty() defined here
+  // CHECK-MESSAGES: :21:8: note: method 'set'::empty() defined here
   if (s_func() == "")
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:7: warning: the 'empty' method should be used
@@ -155,7 +142,7 @@ bool returnsBool() {
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:7: warning: the 'empty' method should be used
   // CHECK-FIXES: {{^  }}if (wstr.empty()){{$}}
-  if (wstr == "")
+  if (wstr == L"")
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:7: warning: the 'empty' method should be used
   // CHECK-FIXES: {{^  }}if (wstr.empty()){{$}}
@@ -452,7 +439,7 @@ class ConstructWithBoolField {
 public:
   ConstructWithBoolField(const std::vector<int> &C) : B(C.size()) {}
 // CHECK-MESSAGES: :[[@LINE-1]]:57: warning: the 'empty' method should be used
-// CHECK-MESSAGES: :11:8: note: method 'vector'::empty() defined here
+// CHECK-MESSAGES: :12:8: note: method 'vector'::empty() defined here
 // CHECK-FIXES: {{^  }}ConstructWithBoolField(const std::vector<int> &C) : B(!C.empty()) {}
 };
 
@@ -460,21 +447,21 @@ struct StructWithNSDMI {
   std::vector<int> C;
   bool B = C.size();
 // CHECK-MESSAGES: :[[@LINE-1]]:12: warning: the 'empty' method should be used
-// CHECK-MESSAGES: :11:8: note: method 'vector'::empty() defined here
+// CHECK-MESSAGES: :12:8: note: method 'vector'::empty() defined here
 // CHECK-FIXES: {{^  }}bool B = !C.empty();
 };
 
 int func(const std::vector<int> &C) {
   return C.size() ? 0 : 1;
 // CHECK-MESSAGES: :[[@LINE-1]]:10: warning: the 'empty' method should be used
-// CHECK-MESSAGES: :11:8: note: method 'vector'::empty() defined here
+// CHECK-MESSAGES: :12:8: note: method 'vector'::empty() defined here
 // CHECK-FIXES: {{^  }}return !C.empty() ? 0 : 1;
 }
 
 constexpr Lazy L;
 static_assert(!L.size(), "");
 // CHECK-MESSAGES: :[[@LINE-1]]:16: warning: the 'empty' method should be used
-// CHECK-MESSAGES: :103:18: note: method 'Lazy'::empty() defined here
+// CHECK-MESSAGES: :90:18: note: method 'Lazy'::empty() defined here
 // CHECK-FIXES: {{^}}static_assert(L.empty(), "");
 
 struct StructWithLazyNoexcept {
@@ -489,7 +476,7 @@ template <typename T> void f() {
   if (v.size())
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:7: warning: the 'empty' method should be used to check for emptiness instead of 'size' [readability-container-size-empty]
-  // CHECK-MESSAGES: :11:8: note: method 'vector'::empty() defined here
+  // CHECK-MESSAGES: :12:8: note: method 'vector'::empty() defined here
   // CHECK-FIXES: {{^  }}if (!v.empty()){{$}}
   if (v == std::vector<T>())
     ;
@@ -498,24 +485,24 @@ template <typename T> void f() {
   // CHECK-FIXES-NEXT: ;
   CHECKSIZE(v);
   // CHECK-MESSAGES: :[[@LINE-1]]:13: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :11:8: note: method 'vector'::empty() defined here
+  // CHECK-MESSAGES: :12:8: note: method 'vector'::empty() defined here
   // CHECK-FIXES: CHECKSIZE(v);
 
   TemplatedContainer<T> templated_container;
   if (templated_container.size())
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:7: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :46:8: note: method 'TemplatedContainer'::empty() defined here
+  // CHECK-MESSAGES: :33:8: note: method 'TemplatedContainer'::empty() defined here
   // CHECK-FIXES: {{^  }}if (!templated_container.empty()){{$}}
   if (templated_container != TemplatedContainer<T>())
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:7: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :46:8: note: method 'TemplatedContainer'::empty() defined here
+  // CHECK-MESSAGES: :33:8: note: method 'TemplatedContainer'::empty() defined here
   // CHECK-FIXES: {{^  }}if (!templated_container.empty()){{$}}
   // CHECK-FIXES-NEXT: ;
   CHECKSIZE(templated_container);
   // CHECK-MESSAGES: :[[@LINE-1]]:13: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :46:8: note: method 'TemplatedContainer'::empty() defined here
+  // CHECK-MESSAGES: :33:8: note: method 'TemplatedContainer'::empty() defined here
   // CHECK-FIXES: CHECKSIZE(templated_container);
 }
 
@@ -531,7 +518,7 @@ bool neverInstantiatedTemplate() {
   if (v.size())
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:7: warning: the 'empty' method should be used to check for emptiness instead of 'size' [readability-container-size-empty]
-  // CHECK-MESSAGES: :11:8: note: method 'vector'::empty() defined here
+  // CHECK-MESSAGES: :12:8: note: method 'vector'::empty() defined here
   // CHECK-FIXES: {{^  }}if (!v.empty()){{$}}
 
   if (v == std::vector<T>())
@@ -542,22 +529,22 @@ bool neverInstantiatedTemplate() {
   if (v.size() == 0)
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:7: warning: the 'empty' method should be used to check for emptiness instead of 'size' [readability-container-size-empty]
-  // CHECK-MESSAGES: :11:8: note: method 'vector'::empty() defined here
+  // CHECK-MESSAGES: :12:8: note: method 'vector'::empty() defined here
   // CHECK-FIXES: {{^  }}if (v.empty()){{$}}
   if (v.size() != 0)
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:7: warning: the 'empty' method should be used to check for emptiness instead of 'size' [readability-container-size-empty]
-  // CHECK-MESSAGES: :11:8: note: method 'vector'::empty() defined here
+  // CHECK-MESSAGES: :12:8: note: method 'vector'::empty() defined here
   // CHECK-FIXES: {{^  }}if (!v.empty()){{$}}
   if (v.size() < 1)
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:7: warning: the 'empty' method should be used to check for emptiness instead of 'size' [readability-container-size-empty]
-  // CHECK-MESSAGES: :11:8: note: method 'vector'::empty() defined here
+  // CHECK-MESSAGES: :12:8: note: method 'vector'::empty() defined here
   // CHECK-FIXES: {{^  }}if (v.empty()){{$}}
   if (v.size() > 0)
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:7: warning: the 'empty' method should be used to check for emptiness instead of 'size' [readability-container-size-empty]
-  // CHECK-MESSAGES: :11:8: note: method 'vector'::empty() defined here
+  // CHECK-MESSAGES: :12:8: note: method 'vector'::empty() defined here
   // CHECK-FIXES: {{^  }}if (!v.empty()){{$}}
   if (v.size() == 1)
     ;
@@ -571,91 +558,91 @@ bool neverInstantiatedTemplate() {
   if (static_cast<bool>(v.size()))
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:25: warning: the 'empty' method should be used to check for emptiness instead of 'size' [readability-container-size-empty]
-  // CHECK-MESSAGES: :11:8: note: method 'vector'::empty() defined here
+  // CHECK-MESSAGES: :12:8: note: method 'vector'::empty() defined here
   // CHECK-FIXES: {{^  }}if (static_cast<bool>(!v.empty())){{$}}
   if (v.size() && false)
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:7: warning: the 'empty' method should be used to check for emptiness instead of 'size' [readability-container-size-empty]
-  // CHECK-MESSAGES: :11:8: note: method 'vector'::empty() defined here
+  // CHECK-MESSAGES: :12:8: note: method 'vector'::empty() defined here
   // CHECK-FIXES: {{^  }}if (!v.empty() && false){{$}}
   if (!v.size())
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:8: warning: the 'empty' method should be used to check for emptiness instead of 'size' [readability-container-size-empty]
-  // CHECK-MESSAGES: :11:8: note: method 'vector'::empty() defined here
+  // CHECK-MESSAGES: :12:8: note: method 'vector'::empty() defined here
   // CHECK-FIXES: {{^  }}if (v.empty()){{$}}
 
   TemplatedContainer<T> templated_container;
   if (templated_container.size())
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:7: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :46:8: note: method 'TemplatedContainer'::empty() defined here
+  // CHECK-MESSAGES: :33:8: note: method 'TemplatedContainer'::empty() defined here
   // CHECK-FIXES: {{^  }}if (!templated_container.empty()){{$}}
   if (templated_container != TemplatedContainer<T>())
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:7: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :46:8: note: method 'TemplatedContainer'::empty() defined here
+  // CHECK-MESSAGES: :33:8: note: method 'TemplatedContainer'::empty() defined here
   // CHECK-FIXES: {{^  }}if (!templated_container.empty()){{$}}
   // CHECK-FIXES-NEXT: ;
   while (templated_container.size())
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:10: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :46:8: note: method 'TemplatedContainer'::empty() defined here
+  // CHECK-MESSAGES: :33:8: note: method 'TemplatedContainer'::empty() defined here
   // CHECK-FIXES: {{^  }}while (!templated_container.empty()){{$}}
 
   do {
   }
   while (templated_container.size());
   // CHECK-MESSAGES: :[[@LINE-1]]:10: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :46:8: note: method 'TemplatedContainer'::empty() defined here
+  // CHECK-MESSAGES: :33:8: note: method 'TemplatedContainer'::empty() defined here
   // CHECK-FIXES: {{^  }}while (!templated_container.empty());
 
   for (; templated_container.size();)
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:10: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :46:8: note: method 'TemplatedContainer'::empty() defined here
+  // CHECK-MESSAGES: :33:8: note: method 'TemplatedContainer'::empty() defined here
   // CHECK-FIXES: {{^  }}for (; !templated_container.empty();){{$}}
 
   if (true && templated_container.size())
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:15: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :46:8: note: method 'TemplatedContainer'::empty() defined here
+  // CHECK-MESSAGES: :33:8: note: method 'TemplatedContainer'::empty() defined here
   // CHECK-FIXES: {{^  }}if (true && !templated_container.empty()){{$}}
 
   if (true || templated_container.size())
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:15: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :46:8: note: method 'TemplatedContainer'::empty() defined here
+  // CHECK-MESSAGES: :33:8: note: method 'TemplatedContainer'::empty() defined here
   // CHECK-FIXES: {{^  }}if (true || !templated_container.empty()){{$}}
 
   if (!templated_container.size())
     ;
   // CHECK-MESSAGES: :[[@LINE-2]]:8: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :46:8: note: method 'TemplatedContainer'::empty() defined here
+  // CHECK-MESSAGES: :33:8: note: method 'TemplatedContainer'::empty() defined here
   // CHECK-FIXES: {{^  }}if (templated_container.empty()){{$}}
 
   bool b1 = templated_container.size();
   // CHECK-MESSAGES: :[[@LINE-1]]:13: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :46:8: note: method 'TemplatedContainer'::empty() defined here
+  // CHECK-MESSAGES: :33:8: note: method 'TemplatedContainer'::empty() defined here
   // CHECK-FIXES: {{^  }}bool b1 = !templated_container.empty();
 
   bool b2(templated_container.size());
   // CHECK-MESSAGES: :[[@LINE-1]]:11: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :46:8: note: method 'TemplatedContainer'::empty() defined here
+  // CHECK-MESSAGES: :33:8: note: method 'TemplatedContainer'::empty() defined here
   // CHECK-FIXES: {{^  }}bool b2(!templated_container.empty());
 
   auto b3 = static_cast<bool>(templated_container.size());
   // CHECK-MESSAGES: :[[@LINE-1]]:31: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :46:8: note: method 'TemplatedContainer'::empty() defined here
+  // CHECK-MESSAGES: :33:8: note: method 'TemplatedContainer'::empty() defined here
   // CHECK-FIXES: {{^  }}auto b3 = static_cast<bool>(!templated_container.empty());
 
   auto b4 = (bool)templated_container.size();
   // CHECK-MESSAGES: :[[@LINE-1]]:19: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :46:8: note: method 'TemplatedContainer'::empty() defined here
+  // CHECK-MESSAGES: :33:8: note: method 'TemplatedContainer'::empty() defined here
   // CHECK-FIXES: {{^  }}auto b4 = (bool)!templated_container.empty();
 
   auto b5 = bool(templated_container.size());
   // CHECK-MESSAGES: :[[@LINE-1]]:18: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :46:8: note: method 'TemplatedContainer'::empty() defined here
+  // CHECK-MESSAGES: :33:8: note: method 'TemplatedContainer'::empty() defined here
   // CHECK-FIXES: {{^  }}auto b5 = bool(!templated_container.empty());
 
   takesBool(templated_container.size());
@@ -664,7 +651,7 @@ bool neverInstantiatedTemplate() {
 
   return templated_container.size();
   // CHECK-MESSAGES: :[[@LINE-1]]:10: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :46:8: note: method 'TemplatedContainer'::empty() defined here
+  // CHECK-MESSAGES: :33:8: note: method 'TemplatedContainer'::empty() defined here
   // CHECK-FIXES: {{^  }}return !templated_container.empty();
 }
 
@@ -710,14 +697,14 @@ struct unique_ptr {
 bool call_through_unique_ptr(const std::unique_ptr<std::vector<int>> &ptr) {
   return ptr->size() > 0;
   // CHECK-MESSAGES: :[[@LINE-1]]:10: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :11:8: note: method 'vector'::empty() defined here
+  // CHECK-MESSAGES: :12:8: note: method 'vector'::empty() defined here
   // CHECK-FIXES: {{^  }}return !ptr->empty();
 }
 
 bool call_through_unique_ptr_deref(const std::unique_ptr<std::vector<int>> &ptr) {
   return (*ptr).size() > 0;
   // CHECK-MESSAGES: :[[@LINE-1]]:10: warning: the 'empty' method should be used
-  // CHECK-MESSAGES: :11:8: note: method 'vector'::empty() defined here
+  // CHECK-MESSAGES: :12:8: note: method 'vector'::empty() defined here
   // CHECK-FIXES: {{^  }}return !(*ptr).empty();
 }
 
@@ -766,7 +753,7 @@ bool testArraySize(const Array& value) {
   return value.size() == 0U;
 // CHECK-MESSAGES: :[[@LINE-1]]:10: warning: the 'empty' method should be used to check for emptiness instead of 'size' [readability-container-size-empty]
 // CHECK-FIXES: {{^  }}return value.empty();{{$}}
-// CHECK-MESSAGES: :744:8: note: method 'array'::empty() defined here
+// CHECK-MESSAGES: :731:8: note: method 'array'::empty() defined here
 }
 
 bool testArrayCompareToEmpty(const Array& value) {
@@ -777,7 +764,7 @@ bool testDummyType(const DummyType& value) {
   return value == DummyType();
 // CHECK-MESSAGES: :[[@LINE-1]]:10: warning: the 'empty' method should be used to check for emptiness instead of comparing to an empty object [readability-container-size-empty]
 // CHECK-FIXES: {{^  }}return value.empty();{{$}}
-// CHECK-MESSAGES: :754:8: note: method 'DummyType'::empty() defined here
+// CHECK-MESSAGES: :741:8: note: method 'DummyType'::empty() defined here
 }
 
 bool testIgnoredDummyType(const IgnoredDummyType& value) {
diff --git a/clang-tools-extra/test/clang-tidy/checkers/readability/identifier-naming-empty-options.cpp b/clang-tools-extra/test/clang-tidy/checkers/readability/identifier-naming-empty-options.cpp
new file mode 100644
index 0000000000000..fa261d8bc377e
--- /dev/null
+++ b/clang-tools-extra/test/clang-tidy/checkers/readability/identifier-naming-empty-options.cpp
@@ -0,0 +1,12 @@
+// RUN: %check_clang_tidy %s readability-identifier-naming %t -- \
+// RUN:   -config='{CheckOptions: { \
+// RUN:     readability-identifier-naming.GlobalConstantPrefix: "", \
+// RUN:     readability-identifier-naming.GlobalVariablePrefix: g_ }}'
+
+int BadGlobalVariable;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'BadGlobalVariable' [readability-identifier-naming]
+// CHECK-FIXES: int g_BadGlobalVariable;
+int g_GoodGlobalVariable;
+
+const int GoodGlobalConstant = 0;
+const int g_IgnoreGlobalConstant = 0;
diff --git a/clang-tools-extra/test/clang-tidy/checkers/readability/identifier-naming-hungarian-notation-c-language.c b/clang-tools-extra/test/clang-tidy/checkers/readability/identifier-naming-hungarian-notation-c-language.c
new file mode 100644
index 0000000000000..1e4e6d7b32726
--- /dev/null
+++ b/clang-tools-extra/test/clang-tidy/checkers/readability/identifier-naming-hungarian-notation-c-language.c
@@ -0,0 +1,536 @@
+// RUN: %check_clang_tidy %s readability-identifier-naming %t -- \
+// RUN:   --config-file=%S/Inputs/identifier-naming/hungarian-notation1/.clang-tidy -- -I %S
+
+#include "identifier-naming-standard-types.h"
+
+// clang-format off
+//===----------------------------------------------------------------------===//
+// Cases to CheckOptions
+//===----------------------------------------------------------------------===//
+const int GlobalConstantCase = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:11: warning: invalid case style for global constant 'GlobalConstantCase' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}const int iGlobalConstantCase = 0;
+
+const int* GlobalConstantPointerCase = NULL;
+// CHECK-MESSAGES: :[[@LINE-1]]:12: warning: invalid case style for global pointer 'GlobalConstantPointerCase' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}const int* piGlobalConstantPointerCase = NULL;
+
+int* GlobalPointerCase = NULL;
+// CHECK-MESSAGES: :[[@LINE-1]]:6: warning: invalid case style for global pointer 'GlobalPointerCase' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int* piGlobalPointerCase = NULL;
+
+int GlobalVariableCase = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'GlobalVariableCase' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int iGlobalVariableCase = 0;
+
+void Func1(){
+  int const LocalConstantCase = 3;
+  // CHECK-MESSAGES: :[[@LINE-1]]:13: warning: invalid case style for local constant 'LocalConstantCase' [readability-identifier-naming]
+  // CHECK-FIXES: {{^}}  int const iLocalConstantCase = 3;
+
+  unsigned const ConstantCase = 1;
+  // CHECK-MESSAGES: :[[@LINE-1]]:18: warning: invalid case style for local constant 'ConstantCase' [readability-identifier-naming]
+  // CHECK-FIXES: {{^}}  unsigned const uConstantCase = 1;
+
+  int* const LocalConstantPointerCase = NULL;
+  // CHECK-MESSAGES: :[[@LINE-1]]:14: warning: invalid case style for local constant pointer 'LocalConstantPointerCase' [readability-identifier-naming]
+  // CHECK-FIXES: {{^}}  int* const piLocalConstantPointerCase = NULL;
+
+  int *LocalPointerCase = NULL;
+  // CHECK-MESSAGES: :[[@LINE-1]]:8: warning: invalid case style for local pointer 'LocalPointerCase' [readability-identifier-naming]
+  // CHECK-FIXES: {{^}}  int *piLocalPointerCase = NULL;
+
+  int LocalVariableCase = 0;
+  // CHECK-MESSAGES: :[[@LINE-1]]:7: warning: invalid case style for local variable 'LocalVariableCase' [readability-identifier-naming]
+  // CHECK-FIXES: {{^}}  int iLocalVariableCase = 0;
+}
+
+struct CMyClass2 {
+  char MemberCase;
+  // CHECK-MESSAGES: :[[@LINE-1]]:8: warning: invalid case style for public member 'MemberCase' [readability-identifier-naming]
+  // CHECK-FIXES: {{^}}  char cMemberCase;
+};
+
+static const int StaticConstantCase = 3;
+// CHECK-MESSAGES: :[[@LINE-1]]:18: warning: invalid case style for global constant 'StaticConstantCase' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}static const int iStaticConstantCase = 3;
+
+static int StaticVariableCase = 3;
+// CHECK-MESSAGES: :[[@LINE-1]]:12: warning: invalid case style for global variable 'StaticVariableCase' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}static int iStaticVariableCase = 3;
+
+struct MyStruct { int StructCase; };
+// CHECK-MESSAGES: :[[@LINE-1]]:23: warning: invalid case style for public member 'StructCase' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}struct MyStruct { int iStructCase; };
+
+union MyUnion { int UnionCase; long lUnionCase; };
+// CHECK-MESSAGES: :[[@LINE-1]]:21: warning: invalid case style for public member 'UnionCase' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}union MyUnion { int iUnionCase; long lUnionCase; };
+
+//===----------------------------------------------------------------------===//
+// C string
+//===----------------------------------------------------------------------===//
+const char *NamePtr = "Name";
+// CHECK-MESSAGES: :[[@LINE-1]]:13: warning: invalid case style for global pointer 'NamePtr' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}const char *szNamePtr = "Name";
+
+const char NameArray[] = "Name";
+// CHECK-MESSAGES: :[[@LINE-1]]:12: warning: invalid case style for global constant 'NameArray' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}const char szNameArray[] = "Name";
+
+const char *NamePtrArray[] = {"AA", "BB"};
+// CHECK-MESSAGES: :[[@LINE-1]]:13: warning: invalid case style for global variable 'NamePtrArray' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}const char *pszNamePtrArray[] = {"AA", "BB"};
+
+const wchar_t *WideNamePtr = L"Name";
+// CHECK-MESSAGES: :[[@LINE-1]]:16: warning: invalid case style for global pointer 'WideNamePtr' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}const wchar_t *wszWideNamePtr = L"Name";
+
+const wchar_t WideNameArray[] = L"Name";
+// CHECK-MESSAGES: :[[@LINE-1]]:15: warning: invalid case style for global constant 'WideNameArray' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}const wchar_t wszWideNameArray[] = L"Name";
+
+const wchar_t *WideNamePtrArray[] = {L"AA", L"BB"};
+// CHECK-MESSAGES: :[[@LINE-1]]:16: warning: invalid case style for global variable 'WideNamePtrArray' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}const wchar_t *pwszWideNamePtrArray[] = {L"AA", L"BB"};
+
+
+//===----------------------------------------------------------------------===//
+// Microsoft Windows data types
+//===----------------------------------------------------------------------===//
+DWORD MsDword = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:7: warning: invalid case style for global variable 'MsDword' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}DWORD dwMsDword = 0;
+
+BYTE MsByte = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:6: warning: invalid case style for global variable 'MsByte' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}BYTE byMsByte = 0;
+
+WORD MsWord = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:6: warning: invalid case style for global variable 'MsWord' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}WORD wMsWord = 0;
+
+BOOL MsBool = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:6: warning: invalid case style for global variable 'MsBool' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}BOOL bMsBool = 0;
+
+BOOLEAN MsBoolean = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:9: warning: invalid case style for global variable 'MsBoolean' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}BOOLEAN bMsBoolean = 0;
+
+CHAR MsValueChar = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:6: warning: invalid case style for global variable 'MsValueChar' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}CHAR cMsValueChar = 0;
+
+UCHAR MsValueUchar = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:7: warning: invalid case style for global variable 'MsValueUchar' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}UCHAR ucMsValueUchar = 0;
+
+SHORT MsValueShort = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:7: warning: invalid case style for global variable 'MsValueShort' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}SHORT sMsValueShort = 0;
+
+USHORT MsValueUshort = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:8: warning: invalid case style for global variable 'MsValueUshort' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}USHORT usMsValueUshort = 0;
+
+WORD MsValueWord = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:6: warning: invalid case style for global variable 'MsValueWord' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}WORD wMsValueWord = 0;
+
+DWORD MsValueDword = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:7: warning: invalid case style for global variable 'MsValueDword' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}DWORD dwMsValueDword = 0;
+
+DWORD32 MsValueDword32 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:9: warning: invalid case style for global variable 'MsValueDword32' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}DWORD32 dw32MsValueDword32 = 0;
+
+DWORD64 MsValueDword64 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:9: warning: invalid case style for global variable 'MsValueDword64' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}DWORD64 dw64MsValueDword64 = 0;
+
+LONG MsValueLong = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:6: warning: invalid case style for global variable 'MsValueLong' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}LONG lMsValueLong = 0;
+
+ULONG MsValueUlong = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:7: warning: invalid case style for global variable 'MsValueUlong' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}ULONG ulMsValueUlong = 0;
+
+ULONG32 MsValueUlong32 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:9: warning: invalid case style for global variable 'MsValueUlong32' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}ULONG32 ul32MsValueUlong32 = 0;
+
+ULONG64 MsValueUlong64 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:9: warning: invalid case style for global variable 'MsValueUlong64' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}ULONG64 ul64MsValueUlong64 = 0;
+
+ULONGLONG MsValueUlongLong = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:11: warning: invalid case style for global variable 'MsValueUlongLong' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}ULONGLONG ullMsValueUlongLong = 0;
+
+HANDLE MsValueHandle = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:8: warning: invalid case style for global pointer 'MsValueHandle' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}HANDLE hMsValueHandle = 0;
+
+INT MsValueInt = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'MsValueInt' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}INT iMsValueInt = 0;
+
+INT8 MsValueInt8 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:6: warning: invalid case style for global variable 'MsValueInt8' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}INT8 i8MsValueInt8 = 0;
+
+INT16 MsValueInt16 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:7: warning: invalid case style for global variable 'MsValueInt16' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}INT16 i16MsValueInt16 = 0;
+
+INT32 MsValueInt32 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:7: warning: invalid case style for global variable 'MsValueInt32' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}INT32 i32MsValueInt32 = 0;
+
+INT64 MsValueINt64 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:7: warning: invalid case style for global variable 'MsValueINt64' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}INT64 i64MsValueINt64 = 0;
+
+UINT MsValueUint = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:6: warning: invalid case style for global variable 'MsValueUint' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}UINT uiMsValueUint = 0;
+
+UINT8 MsValueUint8 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:7: warning: invalid case style for global variable 'MsValueUint8' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}UINT8 u8MsValueUint8 = 0;
+
+UINT16 MsValueUint16 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:8: warning: invalid case style for global variable 'MsValueUint16' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}UINT16 u16MsValueUint16 = 0;
+
+UINT32 MsValueUint32 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:8: warning: invalid case style for global variable 'MsValueUint32' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}UINT32 u32MsValueUint32 = 0;
+
+UINT64 MsValueUint64 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:8: warning: invalid case style for global variable 'MsValueUint64' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}UINT64 u64MsValueUint64 = 0;
+
+PVOID MsValuePvoid = NULL;
+// CHECK-MESSAGES: :[[@LINE-1]]:7: warning: invalid case style for global pointer 'MsValuePvoid' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}PVOID pMsValuePvoid = NULL;
+
+
+//===----------------------------------------------------------------------===//
+// Array
+//===----------------------------------------------------------------------===//
+unsigned GlobalUnsignedArray[] = {1, 2, 3};
+// CHECK-MESSAGES: :[[@LINE-1]]:10: warning: invalid case style for global variable 'GlobalUnsignedArray' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}unsigned aGlobalUnsignedArray[] = {1, 2, 3};
+
+int GlobalIntArray[] = {1, 2, 3};
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'GlobalIntArray' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int aGlobalIntArray[] = {1, 2, 3};
+
+int DataInt[1] = {0};
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'DataInt' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int aDataInt[1] = {0};
+
+int DataArray[2] = {0};
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'DataArray' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int aDataArray[2] = {0};
+
+
+//===----------------------------------------------------------------------===//
+// Pointer
+//===----------------------------------------------------------------------===//
+int *DataIntPtr[1] = {0};
+// CHECK-MESSAGES: :[[@LINE-1]]:6: warning: invalid case style for global variable 'DataIntPtr' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int *paDataIntPtr[1] = {0};
+
+void *BufferPtr1;
+// CHECK-MESSAGES: :[[@LINE-1]]:7: warning: invalid case style for global pointer 'BufferPtr1' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}void *pBufferPtr1;
+
+void **BufferPtr2;
+// CHECK-MESSAGES: :[[@LINE-1]]:8: warning: invalid case style for global pointer 'BufferPtr2' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}void **ppBufferPtr2;
+
+void **pBufferPtr3;
+// CHECK-MESSAGES: :[[@LINE-1]]:8: warning: invalid case style for global pointer 'pBufferPtr3' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}void **ppBufferPtr3;
+
+int *pBufferPtr4;
+// CHECK-MESSAGES: :[[@LINE-1]]:6: warning: invalid case style for global pointer 'pBufferPtr4' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int *piBufferPtr4;
+
+typedef void (*FUNC_PTR_HELLO)();
+FUNC_PTR_HELLO Hello = NULL;
+// CHECK-MESSAGES: :[[@LINE-1]]:16: warning: invalid case style for global pointer 'Hello' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}FUNC_PTR_HELLO fnHello = NULL;
+
+void *ValueVoidPtr = NULL;
+// CHECK-MESSAGES: :[[@LINE-1]]:7: warning: invalid case style for global pointer 'ValueVoidPtr' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}void *pValueVoidPtr = NULL;
+
+ptrdiff_t PtrDiff = NULL;
+// CHECK-MESSAGES: :[[@LINE-1]]:11: warning: invalid case style for global variable 'PtrDiff' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}ptrdiff_t pPtrDiff = NULL;
+
+int8_t *ValueI8Ptr;
+// CHECK-MESSAGES: :[[@LINE-1]]:9: warning: invalid case style for global pointer 'ValueI8Ptr' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int8_t *pi8ValueI8Ptr;
+
+uint8_t *ValueU8Ptr;
+// CHECK-MESSAGES: :[[@LINE-1]]:10: warning: invalid case style for global pointer 'ValueU8Ptr' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}uint8_t *pu8ValueU8Ptr;
+
+void MyFunc2(void* Val){}
+// CHECK-MESSAGES: :[[@LINE-1]]:20: warning: invalid case style for pointer parameter 'Val' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}void MyFunc2(void* pVal){}
+
+
+//===----------------------------------------------------------------------===//
+// Various types
+//===----------------------------------------------------------------------===//
+int8_t ValueI8;
+// CHECK-MESSAGES: :[[@LINE-1]]:8: warning: invalid case style for global variable 'ValueI8' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int8_t i8ValueI8;
+
+int16_t ValueI16 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:9: warning: invalid case style for global variable 'ValueI16' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int16_t i16ValueI16 = 0;
+
+int32_t ValueI32 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:9: warning: invalid case style for global variable 'ValueI32' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int32_t i32ValueI32 = 0;
+
+int64_t ValueI64 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:9: warning: invalid case style for global variable 'ValueI64' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int64_t i64ValueI64 = 0;
+
+uint8_t ValueU8 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:9: warning: invalid case style for global variable 'ValueU8' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}uint8_t u8ValueU8 = 0;
+
+uint16_t ValueU16 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:10: warning: invalid case style for global variable 'ValueU16' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}uint16_t u16ValueU16 = 0;
+
+uint32_t ValueU32 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:10: warning: invalid case style for global variable 'ValueU32' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}uint32_t u32ValueU32 = 0;
+
+uint64_t ValueU64 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:10: warning: invalid case style for global variable 'ValueU64' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}uint64_t u64ValueU64 = 0;
+
+float ValueFloat = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:7: warning: invalid case style for global variable 'ValueFloat' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}float fValueFloat = 0;
+
+double ValueDouble = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:8: warning: invalid case style for global variable 'ValueDouble' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}double dValueDouble = 0;
+
+char ValueChar = 'c';
+// CHECK-MESSAGES: :[[@LINE-1]]:6: warning: invalid case style for global variable 'ValueChar' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}char cValueChar = 'c';
+
+bool ValueBool = true;
+// CHECK-MESSAGES: :[[@LINE-1]]:6: warning: invalid case style for global variable 'ValueBool' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}bool bValueBool = true;
+
+int ValueInt = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'ValueInt' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int iValueInt = 0;
+
+size_t ValueSize = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:8: warning: invalid case style for global variable 'ValueSize' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}size_t nValueSize = 0;
+
+wchar_t ValueWchar = 'w';
+// CHECK-MESSAGES: :[[@LINE-1]]:9: warning: invalid case style for global variable 'ValueWchar' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}wchar_t wcValueWchar = 'w';
+
+short ValueShort = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:7: warning: invalid case style for global variable 'ValueShort' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}short sValueShort = 0;
+
+unsigned ValueUnsigned = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:10: warning: invalid case style for global variable 'ValueUnsigned' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}unsigned uValueUnsigned = 0;
+
+signed ValueSigned = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:8: warning: invalid case style for global variable 'ValueSigned' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}signed sValueSigned = 0;
+
+long ValueLong = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:6: warning: invalid case style for global variable 'ValueLong' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}long lValueLong = 0;
+
+long long ValueLongLong = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:11: warning: invalid case style for global variable 'ValueLongLong' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}long long llValueLongLong = 0;
+
+long long int ValueLongLongInt = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:15: warning: invalid case style for global variable 'ValueLongLongInt' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}long long int lliValueLongLongInt = 0;
+
+long double ValueLongDouble = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:13: warning: invalid case style for global variable 'ValueLongDouble' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}long double ldValueLongDouble = 0;
+
+signed int ValueSignedInt = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:12: warning: invalid case style for global variable 'ValueSignedInt' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}signed int siValueSignedInt = 0;
+
+signed short ValueSignedShort = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:14: warning: invalid case style for global variable 'ValueSignedShort' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}signed short ssValueSignedShort = 0;
+
+signed short int ValueSignedShortInt = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:18: warning: invalid case style for global variable 'ValueSignedShortInt' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}signed short int ssiValueSignedShortInt = 0;
+
+signed long long ValueSignedLongLong = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:18: warning: invalid case style for global variable 'ValueSignedLongLong' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}signed long long sllValueSignedLongLong = 0;
+
+signed long int ValueSignedLongInt = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:17: warning: invalid case style for global variable 'ValueSignedLongInt' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}signed long int sliValueSignedLongInt = 0;
+
+signed long ValueSignedLong = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:13: warning: invalid case style for global variable 'ValueSignedLong' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}signed long slValueSignedLong = 0;
+
+unsigned long long int ValueUnsignedLongLongInt = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:24: warning: invalid case style for global variable 'ValueUnsignedLongLongInt' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}unsigned long long int ulliValueUnsignedLongLongInt = 0;
+
+unsigned long long ValueUnsignedLongLong = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:20: warning: invalid case style for global variable 'ValueUnsignedLongLong' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}unsigned long long ullValueUnsignedLongLong = 0;
+
+unsigned long int ValueUnsignedLongInt = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:19: warning: invalid case style for global variable 'ValueUnsignedLongInt' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}unsigned long int uliValueUnsignedLongInt = 0;
+
+unsigned long ValueUnsignedLong = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:15: warning: invalid case style for global variable 'ValueUnsignedLong' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}unsigned long ulValueUnsignedLong = 0;
+
+unsigned short int ValueUnsignedShortInt = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:20: warning: invalid case style for global variable 'ValueUnsignedShortInt' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}unsigned short int usiValueUnsignedShortInt = 0;
+
+unsigned short ValueUnsignedShort = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:16: warning: invalid case style for global variable 'ValueUnsignedShort' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}unsigned short usValueUnsignedShort = 0;
+
+unsigned int ValueUnsignedInt = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:14: warning: invalid case style for global variable 'ValueUnsignedInt' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}unsigned int uiValueUnsignedInt = 0;
+
+unsigned char ValueUnsignedChar = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:15: warning: invalid case style for global variable 'ValueUnsignedChar' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}unsigned char ucValueUnsignedChar = 0;
+
+long int ValueLongInt = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:10: warning: invalid case style for global variable 'ValueLongInt' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}long int liValueLongInt = 0;
+
+
+//===----------------------------------------------------------------------===//
+// Specifier, Qualifier, Other keywords
+//===----------------------------------------------------------------------===//
+volatile int VolatileInt = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:14: warning: invalid case style for global variable 'VolatileInt' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}volatile int iVolatileInt = 0;
+
+extern int ExternValueInt;
+// CHECK-MESSAGES: :[[@LINE-1]]:12: warning: invalid case style for global variable 'ExternValueInt' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}extern int iExternValueInt;
+
+//===----------------------------------------------------------------------===//
+// Redefined types
+//===----------------------------------------------------------------------===//
+typedef int INDEX;
+INDEX iIndex = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:7: warning: invalid case style for global variable 'iIndex' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}INDEX Index = 0;
+
+
+//===----------------------------------------------------------------------===//
+// Other Cases
+//===----------------------------------------------------------------------===//
+int lower_case = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'lower_case' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int iLowerCase = 0;
+
+int lower_case1 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'lower_case1' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int iLowerCase1 = 0;
+
+int lower_case_2 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'lower_case_2' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int iLowerCase2 = 0;
+
+int UPPER_CASE = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'UPPER_CASE' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int iUpperCase = 0;
+
+int UPPER_CASE_1 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'UPPER_CASE_1' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int iUpperCase1 = 0;
+
+int camelBack = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'camelBack' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int iCamelBack = 0;
+
+int camelBack_1 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'camelBack_1' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int iCamelBack1 = 0;
+
+int camelBack2 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'camelBack2' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int iCamelBack2 = 0;
+
+int CamelCase = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'CamelCase' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int iCamelCase = 0;
+
+int CamelCase_1 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'CamelCase_1' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int iCamelCase1 = 0;
+
+int CamelCase2 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'CamelCase2' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int iCamelCase2 = 0;
+
+int camel_Snake_Back = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'camel_Snake_Back' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int iCamelSnakeBack = 0;
+
+int camel_Snake_Back_1 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'camel_Snake_Back_1' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int iCamelSnakeBack1 = 0;
+
+int Camel_Snake_Case = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'Camel_Snake_Case' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int iCamelSnakeCase = 0;
+
+int Camel_Snake_Case_1 = 0;
+// CHECK-MESSAGES: :[[@LINE-1]]:5: warning: invalid case style for global variable 'Camel_Snake_Case_1' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}int iCamelSnakeCase1 = 0;
+
+//===----------------------------------------------------------------------===//
+// Enum
+//===----------------------------------------------------------------------===//
+enum REV_TYPE { RevValid };
+// CHECK-MESSAGES: :[[@LINE-1]]:17: warning: invalid case style for enum constant 'RevValid' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}enum REV_TYPE { rtRevValid };
+
+enum EnumConstantCase { OneByte, TwoByte };
+// CHECK-MESSAGES: :[[@LINE-1]]:25: warning: invalid case style for enum constant 'OneByte' [readability-identifier-naming]
+// CHECK-MESSAGES: :[[@LINE-2]]:34: warning: invalid case style for enum constant 'TwoByte' [readability-identifier-naming]
+// CHECK-FIXES: {{^}}enum EnumConstantCase { eccOneByte, eccTwoByte };
diff --git a/clang-tools-extra/test/clang-tidy/checkers/readability/identifier-naming-hungarian-notation-cfgfile.cpp b/clang-tools-extra/test/clang-tidy/checkers/readability/identifier-naming-hungarian-notation-cfgfile.cpp
index d9241adb0a831..6e081500b1aa4 100644
--- a/clang-tools-extra/test/clang-tidy/checkers/readability/identifier-naming-hungarian-notation-cfgfile.cpp
+++ b/clang-tools-extra/test/clang-tidy/checkers/readability/identifier-naming-hungarian-notation-cfgfile.cpp
@@ -1,53 +1,7 @@
-// RUN: %check_clang_tidy %s readability-identifier-naming %t -- --config-file=%S/Inputs/identifier-naming/hungarian-notation2/.clang-tidy
+// RUN: %check_clang_tidy %s readability-identifier-naming %t -- \
+// RUN:   --config-file=%S/Inputs/identifier-naming/hungarian-notation2/.clang-tidy -- -I %S
 
-// clang-format off
-typedef signed char         int8_t;     // NOLINT
-typedef short               int16_t;    // NOLINT
-typedef long                int32_t;    // NOLINT
-typedef long long           int64_t;    // NOLINT
-typedef unsigned char       uint8_t;    // NOLINT
-typedef unsigned short      uint16_t;   // NOLINT
-typedef unsigned long       uint32_t;   // NOLINT
-typedef unsigned long long  uint64_t;   // NOLINT
-#ifndef _WIN32
-typedef unsigned long long  size_t;     // NOLINT
-#endif
-typedef long                intptr_t;   // NOLINT
-typedef unsigned long       uintptr_t;  // NOLINT
-typedef long int            ptrdiff_t;  // NOLINT
-typedef unsigned char       BYTE;       // NOLINT
-typedef unsigned short      WORD;       // NOLINT
-typedef unsigned long       DWORD;      // NOLINT
-typedef int                 BOOL;       // NOLINT
-typedef int                 BOOLEAN;    // NOLINT
-typedef float               FLOAT;      // NOLINT
-typedef int                 INT;        // NOLINT
-typedef unsigned int        UINT;       // NOLINT
-typedef unsigned long       ULONG;      // NOLINT
-typedef short               SHORT;      // NOLINT
-typedef unsigned short      USHORT;     // NOLINT
-typedef char                CHAR;       // NOLINT
-typedef unsigned char       UCHAR;      // NOLINT
-typedef signed char         INT8;       // NOLINT
-typedef signed short        INT16;      // NOLINT
-typedef signed int          INT32;      // NOLINT
-typedef signed long long    INT64;      // NOLINT
-typedef unsigned char       UINT8;      // NOLINT
-typedef unsigned short      UINT16;     // NOLINT
-typedef unsigned int        UINT32;     // NOLINT
-typedef unsigned long long  UINT64;     // NOLINT
-typedef long                LONG;       // NOLINT
-typedef signed int          LONG32;     // NOLINT
-typedef unsigned int        ULONG32;    // NOLINT
-typedef uint64_t            ULONG64;    // NOLINT
-typedef unsigned int        DWORD32;    // NOLINT
-typedef uint64_t            DWORD64;    // NOLINT
-typedef uint64_t            ULONGLONG;  // NOLINT
-typedef void*               PVOID;      // NOLINT
-typedef void*               HANDLE;     // NOLINT
-typedef void*               FILE;       // NOLINT
-#define NULL                (0)         // NOLINT
-// clang-format on
+#include "identifier-naming-standard-types.h"
 
 // clang-format off
 //===----------------------------------------------------------------------===//
diff --git a/clang-tools-extra/test/clang-tidy/checkers/readability/identifier-naming-hungarian-notation.cpp b/clang-tools-extra/test/clang-tidy/checkers/readability/identifier-naming-hungarian-notation.cpp
index 72dbfd4cdbbba..1808441f8210c 100644
--- a/clang-tools-extra/test/clang-tidy/checkers/readability/identifier-naming-hungarian-notation.cpp
+++ b/clang-tools-extra/test/clang-tidy/checkers/readability/identifier-naming-hungarian-notation.cpp
@@ -1,53 +1,7 @@
-// RUN: %check_clang_tidy %s readability-identifier-naming %t -- --config-file=%S/Inputs/identifier-naming/hungarian-notation1/.clang-tidy
+// RUN: %check_clang_tidy %s readability-identifier-naming %t -- \
+// RUN:   --config-file=%S/Inputs/identifier-naming/hungarian-notation1/.clang-tidy -- -I %S
 
-// clang-format off
-typedef signed char         int8_t;     // NOLINT
-typedef short               int16_t;    // NOLINT
-typedef long                int32_t;    // NOLINT
-typedef long long           int64_t;    // NOLINT
-typedef unsigned char       uint8_t;    // NOLINT
-typedef unsigned short      uint16_t;   // NOLINT
-typedef unsigned long       uint32_t;   // NOLINT
-typedef unsigned long long  uint64_t;   // NOLINT
-#ifndef _WIN32
-typedef unsigned long long  size_t;     // NOLINT
-#endif
-typedef long                intptr_t;   // NOLINT
-typedef unsigned long       uintptr_t;  // NOLINT
-typedef long int            ptrdiff_t;  // NOLINT
-typedef unsigned char       BYTE;       // NOLINT
-typedef unsigned short      WORD;       // NOLINT
-typedef unsigned long       DWORD;      // NOLINT
-typedef int                 BOOL;       // NOLINT
-typedef int                 BOOLEAN;    // NOLINT
-typedef float               FLOAT;      // NOLINT
-typedef int                 INT;        // NOLINT
-typedef unsigned int        UINT;       // NOLINT
-typedef unsigned long       ULONG;      // NOLINT
-typedef short               SHORT;      // NOLINT
-typedef unsigned short      USHORT;     // NOLINT
-typedef char                CHAR;       // NOLINT
-typedef unsigned char       UCHAR;      // NOLINT
-typedef signed char         INT8;       // NOLINT
-typedef signed short        INT16;      // NOLINT
-typedef signed int          INT32;      // NOLINT
-typedef signed long long    INT64;      // NOLINT
-typedef unsigned char       UINT8;      // NOLINT
-typedef unsigned short      UINT16;     // NOLINT
-typedef unsigned int        UINT32;     // NOLINT
-typedef unsigned long long  UINT64;     // NOLINT
-typedef long                LONG;       // NOLINT
-typedef signed int          LONG32;     // NOLINT
-typedef unsigned int        ULONG32;    // NOLINT
-typedef uint64_t            ULONG64;    // NOLINT
-typedef unsigned int        DWORD32;    // NOLINT
-typedef uint64_t            DWORD64;    // NOLINT
-typedef uint64_t            ULONGLONG;  // NOLINT
-typedef void*               PVOID;      // NOLINT
-typedef void*               HANDLE;     // NOLINT
-typedef void*               FILE;       // NOLINT
-#define NULL                (0)         // NOLINT
-// clang-format on
+#include "identifier-naming-standard-types.h"
 
 // clang-format off
 //===----------------------------------------------------------------------===//
diff --git a/clang-tools-extra/test/clang-tidy/checkers/readability/identifier-naming-standard-types.h b/clang-tools-extra/test/clang-tidy/checkers/readability/identifier-naming-standard-types.h
new file mode 100644
index 0000000000000..edec8d7380200
--- /dev/null
+++ b/clang-tools-extra/test/clang-tidy/checkers/readability/identifier-naming-standard-types.h
@@ -0,0 +1,58 @@
+#pragma once
+
+// clang-format off
+typedef signed char         int8_t;     // NOLINT
+typedef short               int16_t;    // NOLINT
+typedef long                int32_t;    // NOLINT
+typedef long long           int64_t;    // NOLINT
+typedef unsigned char       uint8_t;    // NOLINT
+typedef unsigned short      uint16_t;   // NOLINT
+typedef unsigned long       uint32_t;   // NOLINT
+typedef unsigned long long  uint64_t;   // NOLINT
+#ifndef _WIN32
+typedef unsigned long long  size_t;     // NOLINT
+#endif
+typedef long                intptr_t;   // NOLINT
+typedef unsigned long       uintptr_t;  // NOLINT
+typedef long int            ptrdiff_t;  // NOLINT
+typedef unsigned char       BYTE;       // NOLINT
+typedef unsigned short      WORD;       // NOLINT
+typedef unsigned long       DWORD;      // NOLINT
+typedef int                 BOOL;       // NOLINT
+typedef int                 BOOLEAN;    // NOLINT
+typedef float               FLOAT;      // NOLINT
+typedef int                 INT;        // NOLINT
+typedef unsigned int        UINT;       // NOLINT
+typedef unsigned long       ULONG;      // NOLINT
+typedef short               SHORT;      // NOLINT
+typedef unsigned short      USHORT;     // NOLINT
+typedef char                CHAR;       // NOLINT
+typedef unsigned char       UCHAR;      // NOLINT
+typedef signed char         INT8;       // NOLINT
+typedef signed short        INT16;      // NOLINT
+typedef signed int          INT32;      // NOLINT
+typedef signed long long    INT64;      // NOLINT
+typedef unsigned char       UINT8;      // NOLINT
+typedef unsigned short      UINT16;     // NOLINT
+typedef unsigned int        UINT32;     // NOLINT
+typedef unsigned long long  UINT64;     // NOLINT
+typedef long                LONG;       // NOLINT
+typedef signed int          LONG32;     // NOLINT
+typedef unsigned int        ULONG32;    // NOLINT
+typedef uint64_t            ULONG64;    // NOLINT
+typedef unsigned int        DWORD32;    // NOLINT
+typedef uint64_t            DWORD64;    // NOLINT
+typedef uint64_t            ULONGLONG;  // NOLINT
+typedef void*               PVOID;      // NOLINT
+typedef void*               HANDLE;     // NOLINT
+typedef void*               FILE;       // NOLINT
+
+#define NULL                (0)         // NOLINT
+
+#ifndef __cplusplus
+typedef _Bool               bool;       // NOLINT
+typedef __WCHAR_TYPE__      wchar_t;    // NOLINT
+#define true                1           // NOLINT
+#define false               0           // NOLINT
+#endif
+// clang-format on
diff --git a/clang-tools-extra/test/clang-tidy/checkers/readability/magic-numbers.cpp b/clang-tools-extra/test/clang-tidy/checkers/readability/magic-numbers.cpp
index 638b0183d8242..57547177f5230 100644
--- a/clang-tools-extra/test/clang-tidy/checkers/readability/magic-numbers.cpp
+++ b/clang-tools-extra/test/clang-tidy/checkers/readability/magic-numbers.cpp
@@ -3,7 +3,8 @@
 // RUN:  [{key: readability-magic-numbers.IgnoredIntegerValues, value: "0;1;2;10;100;"}, \
 // RUN:   {key: readability-magic-numbers.IgnoredFloatingPointValues, value: "3.14;2.71828;9.81;10000.0;101.0;0x1.2p3"}, \
 // RUN:   {key: readability-magic-numbers.IgnoreBitFieldsWidths, value: false}, \
-// RUN:   {key: readability-magic-numbers.IgnorePowersOf2IntegerValues, value: true}]}' \
+// RUN:   {key: readability-magic-numbers.IgnorePowersOf2IntegerValues, value: true}, \
+// RUN:   {key: readability-magic-numbers.IgnoreTypeAliases, value: false}]}' \
 // RUN: --
 
 template <typename T, int V>
@@ -96,6 +97,10 @@ struct HardwareGateway {
    // CHECK-MESSAGES: :[[@LINE-1]]:25: warning: 3 is a magic number; consider replacing it with a named constant [readability-magic-numbers]
 };
 
+using NumberInTypeAlias = ValueBucket<int, 25>;
+// CHECK-MESSAGES: :[[@LINE-1]]:44: warning: 25 is a magic number; consider replacing it with a named constant [readability-magic-numbers]
+typedef ValueBucket<char, 243> NumberInTypedef;
+// CHECK-MESSAGES: :[[@LINE-1]]:27: warning: 243 is a magic number; consider replacing it with a named constant [readability-magic-numbers]
 
 /*
  * Clean code
diff --git a/clang-tools-extra/test/clang-tidy/checkers/readability/redundant-string-cstr-format.cpp b/clang-tools-extra/test/clang-tidy/checkers/readability/redundant-string-cstr-format.cpp
new file mode 100644
index 0000000000000..e58c8d8b69c05
--- /dev/null
+++ b/clang-tools-extra/test/clang-tidy/checkers/readability/redundant-string-cstr-format.cpp
@@ -0,0 +1,214 @@
+// RUN: %check_clang_tidy -check-suffix=STDFORMAT -std=c++20 %s readability-redundant-string-cstr %t -- --  -isystem %clang_tidy_headers -DTEST_STDFORMAT
+// RUN: %check_clang_tidy -check-suffixes=STDFORMAT,STDPRINT -std=c++2b %s readability-redundant-string-cstr %t -- --  -isystem %clang_tidy_headers -DTEST_STDFORMAT -DTEST_STDPRINT
+#include <string>
+
+namespace std {
+  template<typename T>
+    struct type_identity { using type = T; };
+  template<typename T>
+    using type_identity_t = typename type_identity<T>::type;
+
+  template <typename CharT, typename... Args>
+  struct basic_format_string {
+    consteval basic_format_string(const CharT *format) : str(format) {}
+    basic_string_view<CharT, std::char_traits<CharT>> str;
+  };
+
+  template<typename... Args>
+    using format_string = basic_format_string<char, type_identity_t<Args>...>;
+
+  template<typename... Args>
+    using wformat_string = basic_format_string<wchar_t, type_identity_t<Args>...>;
+
+#if defined(TEST_STDFORMAT)
+  template<typename ...Args>
+  std::string format(format_string<Args...>, Args &&...);
+  template<typename ...Args>
+  std::string format(wformat_string<Args...>, Args &&...);
+#endif // TEST_STDFORMAT
+
+#if defined(TEST_STDPRINT)
+  template<typename ...Args>
+  void print(format_string<Args...>, Args &&...);
+  template<typename ...Args>
+  void print(wformat_string<Args...>, Args &&...);
+#endif // TEST_STDPRINT
+}
+
+namespace notstd {
+#if defined(TEST_STDFORMAT)
+  template<typename ...Args>
+  std::string format(const char *, Args &&...);
+  template<typename ...Args>
+  std::string format(const wchar_t *, Args &&...);
+#endif // TEST_STDFORMAT
+#if defined(TEST_STDPRINT)
+  template<typename ...Args>
+  void print(const char *, Args &&...);
+  template<typename ...Args>
+  void print(const wchar_t *, Args &&...);
+#endif // TEST_STDPRINT
+}
+
+std::string return_temporary();
+std::wstring return_wtemporary();
+
+#if defined(TEST_STDFORMAT)
+void std_format(const std::string &s1, const std::string &s2, const std::string &s3) {
+  auto r1 = std::format("One:{}\n", s1.c_str());
+  // CHECK-MESSAGES-STDFORMAT: :[[@LINE-1]]:37: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES-STDFORMAT: {{^  }}auto r1 = std::format("One:{}\n", s1);
+
+  auto r2 = std::format("One:{} Two:{} Three:{} Four:{}\n", s1.c_str(), s2, s3.c_str(), return_temporary().c_str());
+  // CHECK-MESSAGES-STDFORMAT: :[[@LINE-1]]:61: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-MESSAGES-STDFORMAT: :[[@LINE-2]]:77: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-MESSAGES-STDFORMAT: :[[@LINE-3]]:89: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES-STDFORMAT: {{^  }}auto r2 = std::format("One:{} Two:{} Three:{} Four:{}\n", s1, s2, s3, return_temporary());
+
+  using namespace std;
+  auto r3 = format("Four:{}\n", s1.c_str());
+  // CHECK-MESSAGES-STDFORMAT: :[[@LINE-1]]:33: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES-STDFORMAT: {{^  }}auto r3 = format("Four:{}\n", s1);
+}
+
+void std_format_wide(const std::wstring &s1, const std::wstring &s2, const std::wstring &s3) {
+  auto r1 = std::format(L"One:{}\n", s1.c_str());
+  // CHECK-MESSAGES-STDFORMAT: :[[@LINE-1]]:38: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES-STDFORMAT: {{^  }}auto r1 = std::format(L"One:{}\n", s1);
+
+  auto r2 = std::format(L"One:{} Two:{} Three:{} Four:{}\n", s1.c_str(), s2, s3.c_str(), return_wtemporary().c_str());
+  // CHECK-MESSAGES-STDFORMAT: :[[@LINE-1]]:62: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-MESSAGES-STDFORMAT: :[[@LINE-2]]:78: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-MESSAGES-STDFORMAT: :[[@LINE-3]]:90: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES-STDFORMAT: {{^  }}auto r2 = std::format(L"One:{} Two:{} Three:{} Four:{}\n", s1, s2, s3, return_wtemporary());
+
+  using namespace std;
+  auto r3 = format(L"Four:{}\n", s1.c_str());
+  // CHECK-MESSAGES-STDFORMAT: :[[@LINE-1]]:34: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES-STDFORMAT: {{^  }}auto r3 = format(L"Four:{}\n", s1);
+}
+
+// There's are c_str() calls here, so it shouldn't be touched.
+std::string std_format_no_cstr(const std::string &s1, const std::string &s2) {
+  return std::format("One: {}, Two: {}\n", s1, s2);
+}
+
+// There's are c_str() calls here, so it shouldn't be touched.
+std::string std_format_no_cstr_wide(const std::string &s1, const std::string &s2) {
+  return std::format(L"One: {}, Two: {}\n", s1, s2);
+}
+
+// This is not std::format, so it shouldn't be fixed.
+std::string not_std_format(const std::string &s1) {
+  return notstd::format("One: {}\n", s1.c_str());
+
+  using namespace notstd;
+  format("One: {}\n", s1.c_str());
+}
+
+// This is not std::format, so it shouldn't be fixed.
+std::string not_std_format_wide(const std::string &s1) {
+  return notstd::format(L"One: {}\n", s1.c_str());
+
+  using namespace notstd;
+  format(L"One: {}\n", s1.c_str());
+}
+#endif // TEST_STDFORMAT
+
+#if defined(TEST_STDPRINT)
+void std_print(const std::string &s1, const std::string &s2, const std::string &s3) {
+  std::print("One:{}\n", s1.c_str());
+  // CHECK-MESSAGES-STDPRINT: :[[@LINE-1]]:26: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES-STDPRINT: {{^  }}std::print("One:{}\n", s1);
+
+  std::print("One:{} Two:{} Three:{} Four:{}\n", s1.c_str(), s2, s3.c_str(), return_temporary().c_str());
+  // CHECK-MESSAGES-STDPRINT: :[[@LINE-1]]:50: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-MESSAGES-STDPRINT: :[[@LINE-2]]:66: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-MESSAGES-STDPRINT: :[[@LINE-3]]:78: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES-STDPRINT: {{^  }}std::print("One:{} Two:{} Three:{} Four:{}\n", s1, s2, s3, return_temporary());
+
+  using namespace std;
+  print("Four:{}\n", s1.c_str());
+  // CHECK-MESSAGES-STDPRINT: :[[@LINE-1]]:22: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES-STDPRINT: {{^  }}print("Four:{}\n", s1);
+}
+
+void std_print_wide(const std::wstring &s1, const std::wstring &s2, const std::wstring &s3) {
+  std::print(L"One:{}\n", s1.c_str());
+  // CHECK-MESSAGES-STDPRINT: :[[@LINE-1]]:27: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES-STDPRINT: {{^  }}std::print(L"One:{}\n", s1);
+
+  std::print(L"One:{} Two:{} Three:{} Four:{}\n", s1.c_str(), s2, s3.c_str(), return_wtemporary().c_str());
+  // CHECK-MESSAGES-STDPRINT: :[[@LINE-1]]:51: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-MESSAGES-STDPRINT: :[[@LINE-2]]:67: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-MESSAGES-STDPRINT: :[[@LINE-3]]:79: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES-STDPRINT: {{^  }}std::print(L"One:{} Two:{} Three:{} Four:{}\n", s1, s2, s3, return_wtemporary());
+
+  using namespace std;
+  print(L"Four:{}\n", s1.c_str());
+  // CHECK-MESSAGES-STDPRINT: :[[@LINE-1]]:23: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES-STDPRINT: {{^  }}print(L"Four:{}\n", s1);
+}
+
+// There's no c_str() call here, so it shouldn't be touched.
+void std_print_no_cstr(const std::string &s1, const std::string &s2) {
+  std::print("One: {}, Two: {}\n", s1, s2);
+}
+
+// There's no c_str() call here, so it shouldn't be touched.
+void std_print_no_cstr_wide(const std::wstring &s1, const std::wstring &s2) {
+  std::print(L"One: {}, Two: {}\n", s1, s2);
+}
+
+// This isn't std::print, so it shouldn't be fixed.
+void not_std_print(const std::string &s1) {
+  notstd::print("One: {}\n", s1.c_str());
+
+  using namespace notstd;
+  print("One: {}\n", s1.c_str());
+}
+
+// This isn't std::print, so it shouldn't be fixed.
+void not_std_print_wide(const std::string &s1) {
+  notstd::print(L"One: {}\n", s1.c_str());
+
+  using namespace notstd;
+  print(L"One: {}\n", s1.c_str());
+}
+#endif // TEST_STDPRINT
+
+#if defined(TEST_STDFORMAT)
+// We can't declare these earlier since they make the "using namespace std"
+// tests ambiguous.
+template<typename ...Args>
+std::string format(const char *, Args &&...);
+template<typename ...Args>
+std::string format(const wchar_t *, Args &&...);
+
+// This is not std::format, so it shouldn't be fixed.
+std::string not_std_format2(const std::wstring &s1) {
+  return format("One: {}\n", s1.c_str());
+}
+
+// This is not std::format, so it shouldn't be fixed.
+std::string not_std_format2_wide(const std::wstring &s1) {
+  return format(L"One: {}\n", s1.c_str());
+}
+#endif // TEST_STDFORMAT
+
+#if defined(TEST_STDPRINT)
+template<typename ...Args>
+void print(const char *, Args &&...);
+template<typename ...Args>
+void print(const wchar_t *, Args &&...);
+
+// This isn't std::print, so it shouldn't be fixed.
+void not_std_print2(const std::string &s1) {
+  print("One: {}\n", s1.c_str());
+}
+
+// This isn't std::print, so it shouldn't be fixed.
+void not_std_print2_wide(const std::string &s1) {
+  print(L"One: {}\n", s1.c_str());
+}
+#endif // TEST_STDPRINT
diff --git a/clang-tools-extra/test/clang-tidy/checkers/readability/redundant-string-cstr-function.cpp b/clang-tools-extra/test/clang-tidy/checkers/readability/redundant-string-cstr-function.cpp
new file mode 100644
index 0000000000000..04e035cdbb626
--- /dev/null
+++ b/clang-tools-extra/test/clang-tidy/checkers/readability/redundant-string-cstr-function.cpp
@@ -0,0 +1,148 @@
+// RUN: %check_clang_tidy %s readability-redundant-string-cstr %t -- \
+// RUN:   -config="{CheckOptions: \
+// RUN:             [{key: readability-redundant-string-cstr.StringParameterFunctions, \
+// RUN:               value: '::fmt::format; ::fmt::print; ::BaseLogger::operator(); ::BaseLogger::Log'}] \
+// RUN:             }" \
+// RUN:   -- -isystem %clang_tidy_headers
+#include <string>
+
+namespace fmt {
+  inline namespace v8 {
+    template<typename ...Args>
+    void print(const char *, Args &&...);
+    template<typename ...Args>
+    std::string format(const char *, Args &&...);
+  }
+}
+
+namespace notfmt {
+  inline namespace v8 {
+    template<typename ...Args>
+    void print(const char *, Args &&...);
+    template<typename ...Args>
+    std::string format(const char *, Args &&...);
+  }
+}
+
+void fmt_print(const std::string &s1, const std::string &s2, const std::string &s3) {
+  fmt::print("One:{}\n", s1.c_str());
+  // CHECK-MESSAGES: :[[@LINE-1]]:26: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES: {{^  }}fmt::print("One:{}\n", s1);
+
+  fmt::print("One:{} Two:{} Three:{}\n", s1.c_str(), s2, s3.c_str());
+  // CHECK-MESSAGES: :[[@LINE-1]]:42: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-MESSAGES: :[[@LINE-2]]:58: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES: {{^  }}fmt::print("One:{} Two:{} Three:{}\n", s1, s2, s3);
+}
+
+// There's no c_str() call here, so it shouldn't be touched
+void fmt_print_no_cstr(const std::string &s1, const std::string &s2) {
+    fmt::print("One: {}, Two: {}\n", s1, s2);
+}
+
+// This isn't fmt::print, so it shouldn't be fixed.
+void not_fmt_print(const std::string &s1) {
+    notfmt::print("One: {}\n", s1.c_str());
+}
+
+void fmt_format(const std::string &s1, const std::string &s2, const std::string &s3) {
+  auto r1 = fmt::format("One:{}\n", s1.c_str());
+  // CHECK-MESSAGES: :[[@LINE-1]]:37: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES: {{^  }}auto r1 = fmt::format("One:{}\n", s1);
+
+  auto r2 = fmt::format("One:{} Two:{} Three:{}\n", s1.c_str(), s2, s3.c_str());
+  // CHECK-MESSAGES: :[[@LINE-1]]:53: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-MESSAGES: :[[@LINE-2]]:69: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES: {{^  }}auto r2 = fmt::format("One:{} Two:{} Three:{}\n", s1, s2, s3);
+}
+
+// There's are c_str() calls here, so it shouldn't be touched
+void fmt_format_no_cstr(const std::string &s1, const std::string &s2) {
+    fmt::format("One: {}, Two: {}\n", s1, s2);
+}
+
+// This is not fmt::format, so it shouldn't be fixed
+std::string not_fmt_format(const std::string &s1) {
+    return notfmt::format("One: {}\n", s1.c_str());
+}
+
+class BaseLogger {
+public:
+  template <typename... Args>
+  void operator()(const char *fmt, Args &&...args) {
+  }
+
+  template <typename... Args>
+  void Log(const char *fmt, Args &&...args) {
+  }
+};
+
+class DerivedLogger : public BaseLogger {};
+class DoubleDerivedLogger : public DerivedLogger {};
+typedef DerivedLogger TypedefDerivedLogger;
+
+void logger1(const std::string &s1, const std::string &s2, const std::string &s3) {
+  BaseLogger LOGGER;
+
+  LOGGER("%s\n", s1.c_str(), s2, s3.c_str());
+  // CHECK-MESSAGES: :[[@LINE-1]]:18: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-MESSAGES: :[[@LINE-2]]:34: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES: {{^  }}LOGGER("%s\n", s1, s2, s3);
+
+  DerivedLogger LOGGER2;
+  LOGGER2("%d %s\n", 42, s1.c_str(), s2.c_str(), s3);
+  // CHECK-MESSAGES: :[[@LINE-1]]:26: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-MESSAGES: :[[@LINE-2]]:38: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES: {{^  }}LOGGER2("%d %s\n", 42, s1, s2, s3);
+
+  DoubleDerivedLogger LOGGERD;
+  LOGGERD("%d %s\n", 42, s1.c_str(), s2, s3.c_str());
+  // CHECK-MESSAGES: :[[@LINE-1]]:26: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-MESSAGES: :[[@LINE-2]]:42: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES: {{^  }}LOGGERD("%d %s\n", 42, s1, s2, s3);
+
+  TypedefDerivedLogger LOGGERT;
+  LOGGERT("%d %s\n", 42, s1.c_str(), s2, s3.c_str());
+  // CHECK-MESSAGES: :[[@LINE-1]]:26: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-MESSAGES: :[[@LINE-2]]:42: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES: {{^  }}LOGGERT("%d %s\n", 42, s1, s2, s3);
+}
+
+void logger2(const std::string &s1, const std::string &s2) {
+  BaseLogger LOGGER3;
+
+  LOGGER3.Log("%s\n", s1.c_str(), s2.c_str());
+  // CHECK-MESSAGES: :[[@LINE-1]]:23: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-MESSAGES: :[[@LINE-2]]:35: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES: {{^  }}LOGGER3.Log("%s\n", s1, s2);
+
+  DerivedLogger LOGGER4;
+  LOGGER4.Log("%d %s\n", 42, s1.c_str(), s2.c_str());
+  // CHECK-MESSAGES: :[[@LINE-1]]:30: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-MESSAGES: :[[@LINE-2]]:42: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES: {{^  }}LOGGER4.Log("%d %s\n", 42, s1, s2);
+}
+
+class NotLogger {
+public:
+  template <typename... Args>
+  void operator()(const char *fmt, Args &&...args) {
+  }
+
+  template <typename... Args>
+  void Log(const char *fmt, Args &&...args) {
+  }
+};
+
+void Log(const char *fmt, ...);
+
+void logger3(const std::string &s1)
+{
+  // Not BaseLogger or something derived from it
+  NotLogger LOGGER;
+  LOGGER("%s\n", s1.c_str());
+  LOGGER.Log("%s\n", s1.c_str());
+
+  // Free function not in StringParameterFunctions list
+  Log("%s\n", s1.c_str());
+}
diff --git a/clang-tools-extra/test/clang-tidy/checkers/readability/redundant-string-cstr.cpp b/clang-tools-extra/test/clang-tidy/checkers/readability/redundant-string-cstr.cpp
index 2b6d290a4aff6..5dc138e81eeec 100644
--- a/clang-tools-extra/test/clang-tidy/checkers/readability/redundant-string-cstr.cpp
+++ b/clang-tools-extra/test/clang-tidy/checkers/readability/redundant-string-cstr.cpp
@@ -1,6 +1,12 @@
 // RUN: %check_clang_tidy %s readability-redundant-string-cstr %t -- -- -isystem %clang_tidy_headers
 #include <string>
 
+template <typename T>
+struct iterator {
+  T *operator->();
+  T &operator*();
+};
+
 namespace llvm {
 struct StringRef {
   StringRef(const char *p);
@@ -202,6 +208,31 @@ void m1(std::string&&) {
   m1p2(s.c_str());  
 }
 
+// Test for overloaded operator->
+void it(iterator<std::string> i)
+{
+  std::string tmp;
+  tmp = i->c_str();
+  // CHECK-MESSAGES: :[[@LINE-1]]:9: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES: {{^  }}tmp = *i;{{$}}
+
+  // An unlikely situation and the outcome is not ideal, but at least the fix doesn't generate broken code.
+  tmp = i.operator->()->c_str();
+  // CHECK-MESSAGES: :[[@LINE-1]]:9: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES: {{^  }}tmp = *i.operator->();{{$}}
+
+  // The fix contains an unnecessary set of parentheses, but these have no effect.
+  iterator<std::string> *pi = &i;
+  tmp = (*pi)->c_str();
+  // CHECK-MESSAGES: :[[@LINE-1]]:9: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES: {{^  }}tmp = *(*pi);{{$}}
+
+  // An unlikely situation, but at least the fix doesn't generate broken code.
+  tmp = pi->operator->()->c_str();
+  // CHECK-MESSAGES: :[[@LINE-1]]:9: warning: redundant call to 'c_str' [readability-redundant-string-cstr]
+  // CHECK-FIXES: {{^  }}tmp = *pi->operator->();{{$}}
+}
+
 namespace PR45286 {
 struct Foo {
   void func(const std::string &) {}
diff --git a/clang-tools-extra/test/clang-tidy/checkers/readability/string-compare.cpp b/clang-tools-extra/test/clang-tidy/checkers/readability/string-compare.cpp
index 45114edf2f0b7..2c08b86cf72fa 100644
--- a/clang-tools-extra/test/clang-tidy/checkers/readability/string-compare.cpp
+++ b/clang-tools-extra/test/clang-tidy/checkers/readability/string-compare.cpp
@@ -1,25 +1,5 @@
-// RUN: %check_clang_tidy %s readability-string-compare %t
-
-namespace std {
-template <typename T>
-class allocator {};
-template <typename T>
-class char_traits {};
-template <typename C, typename T = std::char_traits<C>, typename A = std::allocator<C>>
-class basic_string {
-public:
-  basic_string();
-  basic_string(const C *, unsigned int size);
-  int compare(const basic_string<char> &str) const;
-  int compare(const C *) const;
-  int compare(int, int, const basic_string<char> &str) const;
-  bool empty();
-};
-bool operator==(const basic_string<char> &lhs, const basic_string<char> &rhs);
-bool operator!=(const basic_string<char> &lhs, const basic_string<char> &rhs);
-bool operator==(const basic_string<char> &lhs, const char *&rhs);
-typedef basic_string<char> string;
-}
+// RUN: %check_clang_tidy %s readability-string-compare %t -- -- -isystem %clang_tidy_headers
+#include <string>
 
 void func(bool b);
 
diff --git a/clang/CMakeLists.txt b/clang/CMakeLists.txt
index be59c199f05f8..1fff005d65251 100644
--- a/clang/CMakeLists.txt
+++ b/clang/CMakeLists.txt
@@ -118,12 +118,10 @@ if(CLANG_BUILT_STANDALONE)
         set(LLVM_UTILS_PROVIDED ON)
         set(CLANG_TEST_DEPS FileCheck count not)
       endif()
-      set(UNITTEST_DIR ${LLVM_THIRD_PARTY_DIR}/unittest)
-      if(EXISTS ${UNITTEST_DIR}/googletest/include/gtest/gtest.h
-          AND NOT EXISTS ${LLVM_LIBRARY_DIR}/${CMAKE_STATIC_LIBRARY_PREFIX}gtest${CMAKE_STATIC_LIBRARY_SUFFIX}
-          AND EXISTS ${UNITTEST_DIR}/CMakeLists.txt)
-        add_subdirectory(${UNITTEST_DIR} third-party/unittest)
-      endif()
+    endif()
+
+    if (NOT TARGET llvm_gtest)
+        message(FATAL_ERROR "llvm-gtest not found. Please install llvm-gtest or disable tests with -DLLVM_INCLUDE_TESTS=OFF")
     endif()
 
     if(LLVM_LIT)
@@ -506,13 +504,11 @@ endif()
 
 
 if( CLANG_INCLUDE_TESTS )
-  if(EXISTS ${LLVM_THIRD_PARTY_DIR}/unittest/googletest/include/gtest/gtest.h)
-    add_subdirectory(unittests)
-    list(APPEND CLANG_TEST_DEPS ClangUnitTests)
-    list(APPEND CLANG_TEST_PARAMS
-      clang_unit_site_config=${CMAKE_CURRENT_BINARY_DIR}/test/Unit/lit.site.cfg
-      )
-  endif()
+  add_subdirectory(unittests)
+  list(APPEND CLANG_TEST_DEPS ClangUnitTests)
+  list(APPEND CLANG_TEST_PARAMS
+    clang_unit_site_config=${CMAKE_CURRENT_BINARY_DIR}/test/Unit/lit.site.cfg
+  )
   add_subdirectory(test)
   add_subdirectory(bindings/python/tests)
 
diff --git a/clang/cmake/caches/Fuchsia-stage2.cmake b/clang/cmake/caches/Fuchsia-stage2.cmake
index 016fa9eb5f9d2..c874d8cacd197 100644
--- a/clang/cmake/caches/Fuchsia-stage2.cmake
+++ b/clang/cmake/caches/Fuchsia-stage2.cmake
@@ -23,6 +23,8 @@ set(LLVM_INCLUDE_DOCS OFF CACHE BOOL "")
 set(LLVM_INCLUDE_EXAMPLES OFF CACHE BOOL "")
 set(LLVM_STATIC_LINK_CXX_STDLIB ON CACHE BOOL "")
 set(LLVM_USE_RELATIVE_PATHS_IN_FILES ON CACHE BOOL "")
+set(LLDB_ENABLE_CURSES OFF CACHE BOOL "")
+set(LLDB_ENABLE_LIBEDIT OFF CACHE BOOL "")
 
 if(WIN32)
   set(LLVM_USE_CRT_RELEASE "MT" CACHE STRING "")
diff --git a/clang/cmake/caches/Fuchsia.cmake b/clang/cmake/caches/Fuchsia.cmake
index d9b34dd757e9a..03b96dde7fe86 100644
--- a/clang/cmake/caches/Fuchsia.cmake
+++ b/clang/cmake/caches/Fuchsia.cmake
@@ -4,7 +4,7 @@ set(LLVM_TARGETS_TO_BUILD X86;ARM;AArch64;RISCV CACHE STRING "")
 
 set(PACKAGE_VENDOR Fuchsia CACHE STRING "")
 
-set(LLVM_ENABLE_PROJECTS "bolt;clang;clang-tools-extra;lld;llvm;polly" CACHE STRING "")
+set(_FUCHSIA_ENABLE_PROJECTS "bolt;clang;clang-tools-extra;lld;llvm;polly")
 
 set(LLVM_ENABLE_DIA_SDK OFF CACHE BOOL "")
 set(LLVM_ENABLE_LIBEDIT OFF CACHE BOOL "")
@@ -17,6 +17,8 @@ set(LLVM_ENABLE_ZLIB OFF CACHE BOOL "")
 set(LLVM_INCLUDE_DOCS OFF CACHE BOOL "")
 set(LLVM_INCLUDE_EXAMPLES OFF CACHE BOOL "")
 set(LLVM_USE_RELATIVE_PATHS_IN_FILES ON CACHE BOOL "")
+set(LLDB_ENABLE_CURSES OFF CACHE BOOL "")
+set(LLDB_ENABLE_LIBEDIT OFF CACHE BOOL "")
 
 # Passthrough stage1 flags to stage1.
 set(_FUCHSIA_BOOTSTRAP_PASSTHROUGH
@@ -30,6 +32,9 @@ set(_FUCHSIA_BOOTSTRAP_PASSTHROUGH
   LLVM_ENABLE_CURL
   CURL_ROOT
   OpenSSL_ROOT
+  FUCHSIA_ENABLE_LLDB
+  LLDB_ENABLE_CURSES
+  LLDB_ENABLE_LIBEDIT
   CMAKE_FIND_PACKAGE_PREFER_CONFIG
   CMAKE_SYSROOT
   CMAKE_MODULE_LINKER_FLAGS
@@ -139,7 +144,7 @@ endif()
 set(BOOTSTRAP_LLVM_ENABLE_LLD ON CACHE BOOL "")
 set(BOOTSTRAP_LLVM_ENABLE_LTO ON CACHE BOOL "")
 
-set(CLANG_BOOTSTRAP_TARGETS
+set(_FUCHSIA_BOOTSTRAP_TARGETS
   check-all
   check-clang
   check-lld
@@ -155,7 +160,15 @@ set(CLANG_BOOTSTRAP_TARGETS
   install-distribution
   install-distribution-stripped
   install-distribution-toolchain
-  clang CACHE STRING "")
+  clang)
+
+set(FUCHSIA_ENABLE_LLDB OFF CACHE BOOL "Enable LLDB")
+if(FUCHSIA_ENABLE_LLDB)
+  list(APPEND _FUCHSIA_ENABLE_PROJECTS lldb)
+  list(APPEND _FUCHSIA_BOOTSTRAP_TARGETS check-lldb lldb-test-depends)
+endif()
+set(LLVM_ENABLE_PROJECTS ${_FUCHSIA_ENABLE_PROJECTS} CACHE STRING "")
+set(CLANG_BOOTSTRAP_TARGETS ${_FUCHSIA_BOOTSTRAP_TARGETS} CACHE STRING "")
 
 get_cmake_property(variableNames VARIABLES)
 foreach(variableName ${variableNames})
diff --git a/clang/docs/ClangOffloadBundler.rst b/clang/docs/ClangOffloadBundler.rst
index 997948a8217e5..d08bf4b97781f 100644
--- a/clang/docs/ClangOffloadBundler.rst
+++ b/clang/docs/ClangOffloadBundler.rst
@@ -174,7 +174,20 @@ Where:
       ============= ==============================================================
 
 **target-triple**
-    The target triple of the code object.
+    The target triple of the code object. See `Target Triple
+    <https://clang.llvm.org/docs/CrossCompilation.html#target-triple>`_.
+
+    The bundler accepts target triples with or without the optional environment
+    field:
+
+    ``<arch><sub>-<vendor>-<sys>``, or
+    ``<arch><sub>-<vendor>-<sys>-<env>``
+
+    However, in order to standardize outputs for tools that consume bitcode
+    bundles, bundles written by the bundler internally use only the 4-field
+    target triple:
+
+    ``<arch><sub>-<vendor>-<sys>-<env>``
 
 **target-id**
   The canonical target ID of the code object. Present only if the target
diff --git a/clang/docs/CommandGuide/clang.rst b/clang/docs/CommandGuide/clang.rst
index 7fae1b5cec53c..0722979885afb 100644
--- a/clang/docs/CommandGuide/clang.rst
+++ b/clang/docs/CommandGuide/clang.rst
@@ -473,8 +473,10 @@ Code Generation Options
 
 .. option:: -fexceptions
 
-  Enable generation of unwind information. This allows exceptions to be thrown
-  through Clang compiled stack frames.  This is on by default in x86-64.
+  Allow exceptions to be thrown through Clang compiled stack frames (on many
+  targets, this will enable unwind information for functions that might have
+  an exception thrown through them). For most targets, this is enabled by
+  default for C++.
 
 .. option:: -ftrapv
 
@@ -596,6 +598,16 @@ Driver Options
   directory (:option:`-save-stats`/"-save-stats=cwd") or the directory
   of the output file ("-save-state=obj").
 
+  You can also use environment variables to control the statistics reporting.
+  Setting ``CC_PRINT_INTERNAL_STAT`` to ``1`` enables the feature, the report
+  goes to stdout in JSON format.
+
+  Setting ``CC_PRINT_INTERNAL_STAT_FILE`` to a file path makes it report
+  statistics to the given file in the JSON format.
+
+  Note that ``-save-stats`` take precedence over ``CC_PRINT_INTERNAL_STAT``
+  and ``CC_PRINT_INTERNAL_STAT_FILE``.
+
 .. option:: -integrated-as, -no-integrated-as
 
   Used to enable and disable, respectively, the use of the integrated
diff --git a/clang/docs/CrossCompilation.rst b/clang/docs/CrossCompilation.rst
index 932e0525366e5..3578eb3e4db01 100644
--- a/clang/docs/CrossCompilation.rst
+++ b/clang/docs/CrossCompilation.rst
@@ -77,12 +77,12 @@ match (since Clang assumes the host triple), and the compilation will
 go ahead, creating code for the host platform, which will break later
 on when assembling or linking.
 
-The triple has the general format ``<arch><sub>-<vendor>-<sys>-<abi>``, where:
+The triple has the general format ``<arch><sub>-<vendor>-<sys>-<env>``, where:
  * ``arch`` = ``x86_64``, ``i386``, ``arm``, ``thumb``, ``mips``, etc.
  * ``sub`` = for ex. on ARM: ``v5``, ``v6m``, ``v7a``, ``v7m``, etc.
  * ``vendor`` = ``pc``, ``apple``, ``nvidia``, ``ibm``, etc.
  * ``sys`` = ``none``, ``linux``, ``win32``, ``darwin``, ``cuda``, etc.
- * ``abi`` = ``eabi``, ``gnu``, ``android``, ``macho``, ``elf``, etc.
+ * ``env`` = ``eabi``, ``gnu``, ``android``, ``macho``, ``elf``, etc.
 
 The sub-architecture options are available for their own architectures,
 of course, so "x86v7a" doesn't make sense. The vendor needs to be
@@ -97,8 +97,8 @@ choose ``unknown`` and the defaults will be used. If you choose a parameter
 that Clang doesn't know, like ``blerg``, it'll ignore and assume
 ``unknown``, which is not always desired, so be careful.
 
-Finally, the ABI option is something that will pick default CPU/FPU,
-define the specific behaviour of your code (PCS, extensions),
+Finally, the env (enviornment) option is something that will pick default
+CPU/FPU, define the specific behaviour of your code (PCS, extensions),
 and also choose the correct library calls, etc.
 
 CPU, FPU, ABI
@@ -198,6 +198,6 @@ forget to specify ``-ccc-gcc-name armv7l-linux-gnueabihf-gcc`` (which
 uses hard-float), Clang will pick the ``armv7l-linux-gnueabi-ld``
 (which uses soft-float) and linker errors will happen.
 
-The same is true if you're compiling for different ABIs, like ``gnueabi``
-and ``androideabi``, and might even link and run, but produce run-time
-errors, which are much harder to track down and fix.
+The same is true if you're compiling for different environments, like
+``gnueabi`` and ``androideabi``, and might even link and run, but produce
+run-time errors, which are much harder to track down and fix.
diff --git a/clang/docs/LanguageExtensions.rst b/clang/docs/LanguageExtensions.rst
index c0ea8afad6cb2..f8c83d4d6d162 100644
--- a/clang/docs/LanguageExtensions.rst
+++ b/clang/docs/LanguageExtensions.rst
@@ -639,6 +639,8 @@ Unless specified otherwise operation(±0) = ±0 and operation(±infinity) = ±in
  T __builtin_elementwise_log(T x)            return the natural logarithm of x                                floating point types
  T __builtin_elementwise_log2(T x)           return the base 2 logarithm of x                                 floating point types
  T __builtin_elementwise_log10(T x)          return the base 10 logarithm of x                                floating point types
+ T __builtin_elementwise_exp(T x)            returns the base-e exponential, e^x, of the specified value      floating point types
+ T __builtin_elementwise_exp2(T x)           returns the base-2 exponential, 2^x, of the specified value      floating point types
  T __builtin_elementwise_roundeven(T x)      round x to the nearest integer value in floating point format,   floating point types
                                              rounding halfway cases to even (that is, to the nearest value
                                              that is an even integer), regardless of the current rounding
@@ -3609,13 +3611,15 @@ Source location builtins
 
 Clang provides builtins to support C++ standard library implementation
 of ``std::source_location`` as specified in C++20.  With the exception
-of ``__builtin_COLUMN``, these builtins are also implemented by GCC.
+of ``__builtin_COLUMN`` and ``__builtin_FILE_NAME``,
+these builtins are also implemented by GCC.
 
 **Syntax**:
 
 .. code-block:: c
 
   const char *__builtin_FILE();
+  const char *__builtin_FILE_NAME(); // Clang only
   const char *__builtin_FUNCTION();
   unsigned    __builtin_LINE();
   unsigned    __builtin_COLUMN(); // Clang only
@@ -3646,11 +3650,11 @@ of ``__builtin_COLUMN``, these builtins are also implemented by GCC.
 
 **Description**:
 
-The builtins ``__builtin_LINE``, ``__builtin_FUNCTION``, and ``__builtin_FILE``
-return the values, at the "invocation point", for ``__LINE__``,
-``__FUNCTION__``, and ``__FILE__`` respectively. ``__builtin_COLUMN`` similarly
-returns the column, though there is no corresponding macro. These builtins are
-constant expressions.
+The builtins ``__builtin_LINE``, ``__builtin_FUNCTION``, ``__builtin_FILE`` and
+``__builtin_FILE_NAME`` return the values, at the "invocation point", for
+``__LINE__``, ``__FUNCTION__``, ``__FILE__`` and ``__FILE_NAME__`` respectively.
+``__builtin_COLUMN`` similarly returns the column,
+though there is no corresponding macro. These builtins are constant expressions.
 
 When the builtins appear as part of a default function argument the invocation
 point is the location of the caller. When the builtins appear as part of a
diff --git a/clang/docs/ReleaseNotes.rst b/clang/docs/ReleaseNotes.rst
index cd91fbe78bb2b..53001f651ea4b 100644
--- a/clang/docs/ReleaseNotes.rst
+++ b/clang/docs/ReleaseNotes.rst
@@ -78,6 +78,11 @@ C++ Language Changes
 
 C++20 Feature Support
 ^^^^^^^^^^^^^^^^^^^^^
+- Support for out-of-line definitions of constrained templates has been improved.
+  This partially fixes `#49620 <https://github.com/llvm/llvm-project/issues/49620>`_.
+- Lambda templates with a requires clause directly after the template parameters now parse
+  correctly if the requires clause consists of a variable with a dependent type.
+  (`#61278 <https://github.com/llvm/llvm-project/issues/61278>`_)
 
 C++2b Feature Support
 ^^^^^^^^^^^^^^^^^^^^^
@@ -93,8 +98,7 @@ Resolutions to C++ Defect Reports
 C Language Changes
 ------------------
 - Support for outputs from asm goto statements along indirect edges has been
-  added. This fixes
-  `Issue 53562 <https://github.com/llvm/llvm-project/issues/53562>`_.
+  added. (`#53562 <https://github.com/llvm/llvm-project/issues/53562>`_)
 
 C2x Feature Support
 ^^^^^^^^^^^^^^^^^^^
@@ -115,6 +119,9 @@ Non-comprehensive list of changes in this release
   optimizations.
 - Clang now supports ``__builtin_nondeterministic_value`` that returns a
   nondeterministic value of the same type as the provided argument.
+- Clang now supports ``__builtin_FILE_NAME()`` which returns the same
+  information as the ``__FILE_NAME__`` macro (the presumed file name
+  from the invocation point, with no path components included).
 
 New Compiler Flags
 ------------------
@@ -131,6 +138,9 @@ Removed Compiler Flags
   or higher to use standard C++ modules instead.
 - The deprecated flag `-fcoroutines-ts` is removed. Please use ``-std=c++20``
   or higher to use standard C++ coroutines instead.
+- The CodeGen flag `-lower-global-dtors-via-cxa-atexit` which affects how global
+  destructors are lowered for MachO is removed without replacement. The default
+  of `-lower-global-dtors-via-cxa-atexit=true` is now the only supported way.
 
 Attribute Changes in Clang
 --------------------------
@@ -146,17 +156,29 @@ Attribute Changes in Clang
   uses the optional USR value when indexing Clang's AST. This value is expected
   to be generated by an external compiler when generating C++ bindings during
   the compilation of the foreign language sources (e.g. Swift).
+- The ``__has_attribute``, ``__has_c_attribute`` and ``__has_cpp_attribute``
+  preprocessor operators now return 1 also for attributes defined by plugins.
 
 Improvements to Clang's diagnostics
 -----------------------------------
 - We now generate a diagnostic for signed integer overflow due to unary minus
   in a non-constant expression context.
-  (`#31643 <https://github.com/llvm/llvm-project/issues/31643>`)
+  (`#31643 <https://github.com/llvm/llvm-project/issues/31643>`_)
 - Clang now warns by default for C++20 and later about deprecated capture of
   ``this`` with a capture default of ``=``. This warning can be disabled with
   ``-Wno-deprecated-this-capture``.
 - Clang had failed to emit some ``-Wundefined-internal`` for members of a local
   class if that class was first introduced with a forward declaration.
+- Diagnostic notes and fix-its are now generated for ``ifunc``/``alias`` attributes
+  which point to functions whose names are mangled.
+- Diagnostics relating to macros on the command line of a preprocessed assembly
+  file are now reported as coming from the file ``<command line>`` instead of
+  ``<built-in>``.
+- Clang constexpr evaluator now provides a more concise diagnostic when calling
+  function pointer that is known to be null.
+- Clang now avoids duplicate warnings on unreachable ``[[fallthrough]];`` statements
+  previously issued from ``-Wunreachable-code`` and ``-Wunreachable-code-fallthrough``
+  by prioritizing ``-Wunreachable-code-fallthrough``.
 
 Bug Fixes in This Version
 -------------------------
@@ -180,6 +202,19 @@ Bug Fixes in This Version
   (`#60405 <https://github.com/llvm/llvm-project/issues/60405>`_)
 - Fix aggregate initialization inside lambda constexpr.
   (`#60936 <https://github.com/llvm/llvm-project/issues/60936>`_)
+- No longer issue a false positive diagnostic about a catch handler that cannot
+  be reached despite being reachable. This fixes
+  `#61177 <https://github.com/llvm/llvm-project/issues/61177>`_ in anticipation
+  of `CWG2699 <https://wg21.link/CWG2699>_` being accepted by WG21.
+- Fix crash when parsing fold expression containing a delayed typo correction.
+  (`#61326 <https://github.com/llvm/llvm-project/issues/61326>`_)
+- Fix crash when dealing with some member accesses outside of class or member
+  function context.
+  (`#37792 <https://github.com/llvm/llvm-project/issues/37792>`_) and
+  (`#48405 <https://github.com/llvm/llvm-project/issues/48405>`_)
+- Fix crash when using ``[[clang::always_inline]]`` or ``[[clang::noinline]]``
+  statement attributes on a call to a template function in the body of a
+  template function.
 
 Bug Fixes to Compiler Builtins
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -196,6 +231,11 @@ Bug Fixes to C++ Support
 - Fix crash when evaluating consteval constructor of derived class whose base
   has more than one field.
   (`#60166 <https://github.com/llvm/llvm-project/issues/60166>`_)
+- Fix an issue about ``decltype`` in the members of class templates derived from
+  templates with related parameters.
+  (`#58674 <https://github.com/llvm/llvm-project/issues/58674>`_)
+- Fix incorrect deletion of the default constructor of unions in some
+  cases. (`#48416 <https://github.com/llvm/llvm-project/issues/48416>`_)
 
 Bug Fixes to AST Handling
 ^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -209,6 +249,14 @@ Miscellaneous Clang Crashes Fixed
 Target Specific Changes
 -----------------------
 
+AMDGPU Support
+^^^^^^^^^^^^^^
+
+- Linking for AMDGPU now uses ``--no-undefined`` by default. This causes
+  undefined symbols in the created module to be a linker error. To prevent this,
+  pass ``-Wl,--undefined`` if compiling directly, or ``-Xoffload-linker
+  --undefined`` if using an offloading language.
+
 X86 Support
 ^^^^^^^^^^^
 
@@ -231,6 +279,9 @@ Windows Support
 LoongArch Support
 ^^^^^^^^^^^^^^^^^
 
+- Patchable function entry (``-fpatchable-function-entry``) is now supported
+  on LoongArch.
+
 RISC-V Support
 ^^^^^^^^^^^^^^
 - Added ``-mrvv-vector-bits=`` option to give an upper and lower bound on vector
@@ -265,6 +316,9 @@ Floating Point Support in Clang
 - Add ``__builtin_elementwise_log`` builtin for floating point types only.
 - Add ``__builtin_elementwise_log10`` builtin for floating point types only.
 - Add ``__builtin_elementwise_log2`` builtin for floating point types only.
+- Add ``__builtin_elementwise_exp`` builtin for floating point types only.
+- Add ``__builtin_elementwise_exp2`` builtin for floating point types only.
+- Add ``__builtin_set_flt_rounds`` builtin for X86, x86_64, Arm and AArch64 only.
 
 AST Matchers
 ------------
@@ -289,11 +343,23 @@ libclang
   which identifies whether a constructor or conversion function cursor
   was marked with the explicit identifier.
 
+- Introduced the new ``CXIndex`` constructor function
+  ``clang_createIndexWithOptions``, which allows storing precompiled preambles
+  in memory or overriding the precompiled preamble storage path.
+
+- Deprecated two functions ``clang_CXIndex_setGlobalOptions`` and
+  ``clang_CXIndex_setInvocationEmissionPathOption`` in favor of the new
+  function ``clang_createIndexWithOptions`` in order to improve thread safety.
+
+- Added check in ``clang_getFieldDeclBitWidth`` for whether a bit-field
+  has an evaluable bit width. Fixes undefined behavior when called on a
+  bit-field whose width depends on a template paramter.
+
 Static Analyzer
 ---------------
 - Fix incorrect alignment attribute on the this parameter of certain
   non-complete destructors when using the Microsoft ABI.
-  `Issue 60465 <https://github.com/llvm/llvm-project/issues/60465>`_.
+  (`#60465 <https://github.com/llvm/llvm-project/issues/60465>`_)
 
 .. _release-notes-sanitizers:
 
diff --git a/clang/docs/StandardCPlusPlusModules.rst b/clang/docs/StandardCPlusPlusModules.rst
index 816199052f2bc..ab34ba03ba14c 100644
--- a/clang/docs/StandardCPlusPlusModules.rst
+++ b/clang/docs/StandardCPlusPlusModules.rst
@@ -200,7 +200,7 @@ Then we are able to compile the example by the following command:
   $ clang++ -std=c++20 interface_part.cppm --precompile -o M-interface_part.pcm
   $ clang++ -std=c++20 impl_part.cppm --precompile -fprebuilt-module-path=. -o M-impl_part.pcm
   $ clang++ -std=c++20 M.cppm --precompile -fprebuilt-module-path=. -o M.pcm
-  $ clang++ -std=c++20 Impl.cpp -fmodule-file=M.pcm -c -o Impl.o
+  $ clang++ -std=c++20 Impl.cpp -fmodule-file=M=M.pcm -c -o Impl.o
 
   # Compiling the user
   $ clang++ -std=c++20 User.cpp -fprebuilt-module-path=. -c -o User.o
@@ -332,7 +332,7 @@ How to specify the dependent BMIs
 There are 3 methods to specify the dependent BMIs:
 
 * (1) ``-fprebuilt-module-path=<path/to/direcotry>``.
-* (2) ``-fmodule-file=<path/to/BMI>``.
+* (2) ``-fmodule-file=<path/to/BMI>`` (Deprecated).
 * (3) ``-fmodule-file=<module-name>=<path/to/BMI>``.
 
 The option ``-fprebuilt-module-path`` tells the compiler the path where to search for dependent BMIs.
@@ -348,7 +348,8 @@ The option ``-fmodule-file=<module-name>=<path/to/BMI>`` tells the compiler to l
 for the module specified by ``<module-name>`` when necessary. The main difference is that
 ``-fmodule-file=<path/to/BMI>`` will load the BMI eagerly, whereas
 ``-fmodule-file=<module-name>=<path/to/BMI>`` will only load the BMI lazily, which is similar
-with ``-fprebuilt-module-path``.
+with ``-fprebuilt-module-path``. The option ``-fmodule-file=<path/to/BMI>`` for named modules is deprecated
+and is planning to be removed in future versions.
 
 In case all ``-fprebuilt-module-path=<path/to/direcotry>``, ``-fmodule-file=<path/to/BMI>`` and
 ``-fmodule-file=<module-name>=<path/to/BMI>`` exist, the ``-fmodule-file=<path/to/BMI>`` option
@@ -372,7 +373,6 @@ the above example could be rewritten into:
 
 .. code-block:: console
 
-  $ clang++ -std=c++20 M.cppm --precompile -fmodule-file=M-interface_part.pcm -fmodule-file=M-impl_part.pcm -o M.pcm
   $ clang++ -std=c++20 M.cppm --precompile -fmodule-file=M:interface_part=M-interface_part.pcm -fmodule-file=M:impl_part=M-impl_part.pcm -o M.pcm
 
 ``-fprebuilt-module-path`` is more convenient and ``-fmodule-file`` is faster since
@@ -486,7 +486,7 @@ source files. For example:
 
   $ clang++ -std=c++20 M.cppm --precompile -o M.pcm
   $ rm M.cppm
-  $ clang++ -std=c++20 Use.cpp -fmodule-file=M.pcm
+  $ clang++ -std=c++20 Use.cpp -fmodule-file=M=M.pcm
 
 The compiler would reject the example since the compiler failed to find the source file to check the consistency.
 So the following example would be rejected too.
@@ -495,7 +495,7 @@ So the following example would be rejected too.
 
   $ clang++ -std=c++20 M.cppm --precompile -o M.pcm
   $ echo "int i=0;" >> M.cppm
-  $ clang++ -std=c++20 Use.cpp -fmodule-file=M.pcm
+  $ clang++ -std=c++20 Use.cpp -fmodule-file=M=M.pcm
 
 The compiler would reject it too since the compiler detected the file was changed.
 
@@ -506,7 +506,7 @@ But it is OK to move the BMI as long as the source files remain:
   $ clang++ -std=c++20 M.cppm --precompile -o M.pcm
   $ mkdir -p tmp
   $ mv M.pcm tmp/M.pcm
-  $ clang++ -std=c++20 Use.cpp -fmodule-file=tmp/M.pcm
+  $ clang++ -std=c++20 Use.cpp -fmodule-file=M=tmp/M.pcm
 
 The above example would be accepted.
 
@@ -517,7 +517,7 @@ the user could try to use ``-Xclang -fmodules-embed-all-files`` when producing B
 
   $ clang++ -std=c++20 M.cppm --precompile -Xclang -fmodules-embed-all-files -o M.pcm
   $ rm M.cppm
-  $ clang++ -std=c++20 Use.cpp -fmodule-file=M.pcm
+  $ clang++ -std=c++20 Use.cpp -fmodule-file=M=M.pcm
 
 Now the compiler would accept the above example.
 Important note: Xclang options are intended to be used by compiler internally and its semantics
@@ -553,7 +553,7 @@ Then it is problematic if we remove ``foo.h`` before import `foo` module.
   $ clang++ -std=c++20 foo.cppm --precompile  -o foo.pcm
   $ mv foo.h foo.orig.h
   # The following one is rejected
-  $ clang++ -std=c++20 Use.cpp -fmodule-file=foo.pcm -c
+  $ clang++ -std=c++20 Use.cpp -fmodule-file=foo=foo.pcm -c
 
 The above case will rejected. And we're still able to workaround it by ``-Xclang -fmodules-embed-all-files`` option:
 
@@ -561,7 +561,7 @@ The above case will rejected. And we're still able to workaround it by ``-Xclang
 
   $ clang++ -std=c++20 foo.cppm --precompile  -Xclang -fmodules-embed-all-files -o foo.pcm
   $ mv foo.h foo.orig.h
-  $ clang++ -std=c++20 Use.cpp -fmodule-file=foo.pcm -c -o Use.o
+  $ clang++ -std=c++20 Use.cpp -fmodule-file=foo=foo.pcm -c -o Use.o
   $ clang++ Use.o foo.pcm
 
 ABI Impacts
@@ -591,6 +591,19 @@ The result would be ``NS::foo@M()``, which reads as ``NS::foo()`` in module ``M`
 The ABI implies that we can't declare something in a module unit and define it in a non-module unit (or vice-versa),
 as this would result in linking errors.
 
+If we still want to implement declarations within the compatible ABI in module unit,
+we can use the language-linkage specifier. Since the declarations in the language-linkage specifier
+is attached to the global module fragments. For example:
+
+.. code-block:: c++
+
+  export module M;
+  namespace NS {
+    export extern "C++" int foo();
+  }
+
+Now the linkage name of ``NS::foo()`` will be ``_ZN2NS3fooEv``.
+
 Known Problems
 --------------
 
@@ -603,6 +616,50 @@ and add the label ``clang:modules`` (if you have permissions for that).
 
 For higher level support for proposals, you could visit https://clang.llvm.org/cxx_status.html.
 
+Including headers after import is problematic
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For example, the following example can be accept:
+
+.. code-block:: c++
+
+  #include <iostream>
+  import foo; // assume module 'foo' contain the declarations from `<iostream>`
+
+  int main(int argc, char *argv[])
+  {
+      std::cout << "Test\n";
+      return 0;
+  }
+
+but it will get rejected if we reverse the order of ``#include <iostream>`` and
+``import foo;``:
+
+.. code-block:: c++
+
+  import foo; // assume module 'foo' contain the declarations from `<iostream>`
+  #include <iostream>
+
+  int main(int argc, char *argv[])
+  {
+      std::cout << "Test\n";
+      return 0;
+  }
+
+Both of the above examples should be accepted.
+
+This is a limitation in the implementation. In the first example,
+the compiler will see and parse <iostream> first then the compiler will see the import.
+So the ODR Checking and declarations merging will happen in the deserializer.
+In the second example, the compiler will see the import first and the include second.
+As a result, the ODR Checking and declarations merging will happen in the semantic analyzer.
+
+So there is divergence in the implementation path. It might be understandable that why
+the orders matter here in the case.
+(Note that "understandable" is different from "makes sense").
+
+This is tracked in: https://github.com/llvm/llvm-project/issues/61465
+
 Ambiguous deduction guide
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/clang/docs/UsersManual.rst b/clang/docs/UsersManual.rst
index 0d5e960050c25..77b1c938c6ad4 100644
--- a/clang/docs/UsersManual.rst
+++ b/clang/docs/UsersManual.rst
@@ -2013,6 +2013,24 @@ are listed below.
    devirtualization and virtual constant propagation, for classes with
    :doc:`hidden LTO visibility <LTOVisibility>`. Requires ``-flto``.
 
+.. option:: -f[no]split-lto-unit
+
+   Controls splitting the :doc:`LTO unit <LTOVisibility>` into regular LTO and
+   :doc:`ThinLTO` portions, when compiling with -flto=thin. Defaults to false
+   unless ``-fsanitize=cfi`` or ``-fwhole-program-vtables`` are specified, in
+   which case it defaults to true. Splitting is required with ``fsanitize=cfi``,
+   and it is an error to disable via ``-fno-split-lto-unit``. Splitting is
+   optional with ``-fwhole-program-vtables``, however, it enables more
+   aggressive whole program vtable optimizations (specifically virtual constant
+   propagation).
+
+   When enabled, vtable definitions and select virtual functions are placed
+   in the split regular LTO module, enabling more aggressive whole program
+   vtable optimizations required for CFI and virtual constant propagation.
+   However, this can increase the LTO link time and memory requirements over
+   pure ThinLTO, as all split regular LTO modules are merged and LTO linked
+   with regular LTO.
+
 .. option:: -fforce-emit-vtables
 
    In order to improve devirtualization, forces emitting of vtables even in
diff --git a/clang/examples/Attribute/Attribute.cpp b/clang/examples/Attribute/Attribute.cpp
index 159b09e4b154d..24b95dde4e559 100644
--- a/clang/examples/Attribute/Attribute.cpp
+++ b/clang/examples/Attribute/Attribute.cpp
@@ -9,6 +9,8 @@
 // Example clang plugin which adds an an annotation to file-scope declarations
 // with the 'example' attribute.
 //
+// This plugin is used by clang/test/Frontend/plugin-attribute tests.
+//
 //===----------------------------------------------------------------------===//
 
 #include "clang/AST/ASTContext.h"
@@ -27,9 +29,10 @@ struct ExampleAttrInfo : public ParsedAttrInfo {
     // number of arguments. This just illustrates how many arguments a
     // `ParsedAttrInfo` can hold, we will not use that much in this example.
     OptArgs = 15;
-    // GNU-style __attribute__(("example")) and C++-style [[example]] and
+    // GNU-style __attribute__(("example")) and C++/C2x-style [[example]] and
     // [[plugin::example]] supported.
     static constexpr Spelling S[] = {{ParsedAttr::AS_GNU, "example"},
+                                     {ParsedAttr::AS_C2x, "example"},
                                      {ParsedAttr::AS_CXX11, "example"},
                                      {ParsedAttr::AS_CXX11, "plugin::example"}};
     Spellings = S;
diff --git a/clang/include/clang-c/Index.h b/clang/include/clang-c/Index.h
index a17a97721de26..c7d32e6a152ae 100644
--- a/clang/include/clang-c/Index.h
+++ b/clang/include/clang-c/Index.h
@@ -34,7 +34,7 @@
  * compatible, thus CINDEX_VERSION_MAJOR is expected to remain stable.
  */
 #define CINDEX_VERSION_MAJOR 0
-#define CINDEX_VERSION_MINOR 63
+#define CINDEX_VERSION_MINOR 64
 
 #define CINDEX_VERSION_ENCODE(major, minor) (((major)*10000) + ((minor)*1))
 
@@ -275,6 +275,22 @@ CINDEX_LINKAGE CXIndex clang_createIndex(int excludeDeclarationsFromPCH,
  */
 CINDEX_LINKAGE void clang_disposeIndex(CXIndex index);
 
+typedef enum {
+  /**
+   * Use the default value of an option that may depend on the process
+   * environment.
+   */
+  CXChoice_Default = 0,
+  /**
+   * Enable the option.
+   */
+  CXChoice_Enabled = 1,
+  /**
+   * Disable the option.
+   */
+  CXChoice_Disabled = 2
+} CXChoice;
+
 typedef enum {
   /**
    * Used to indicate that no special CXIndex options are needed.
@@ -309,9 +325,131 @@ typedef enum {
 
 } CXGlobalOptFlags;
 
+/**
+ * Index initialization options.
+ *
+ * 0 is the default value of each member of this struct except for Size.
+ * Initialize the struct in one of the following three ways to avoid adapting
+ * code each time a new member is added to it:
+ * \code
+ * CXIndexOptions Opts;
+ * memset(&Opts, 0, sizeof(Opts));
+ * Opts.Size = sizeof(CXIndexOptions);
+ * \endcode
+ * or explicitly initialize the first data member and zero-initialize the rest:
+ * \code
+ * CXIndexOptions Opts = { sizeof(CXIndexOptions) };
+ * \endcode
+ * or to prevent the -Wmissing-field-initializers warning for the above version:
+ * \code
+ * CXIndexOptions Opts{};
+ * Opts.Size = sizeof(CXIndexOptions);
+ * \endcode
+ */
+typedef struct CXIndexOptions {
+  /**
+   * The size of struct CXIndexOptions used for option versioning.
+   *
+   * Always initialize this member to sizeof(CXIndexOptions), or assign
+   * sizeof(CXIndexOptions) to it right after creating a CXIndexOptions object.
+   */
+  unsigned Size;
+  /**
+   * A CXChoice enumerator that specifies the indexing priority policy.
+   * \sa CXGlobalOpt_ThreadBackgroundPriorityForIndexing
+   */
+  unsigned char ThreadBackgroundPriorityForIndexing;
+  /**
+   * A CXChoice enumerator that specifies the editing priority policy.
+   * \sa CXGlobalOpt_ThreadBackgroundPriorityForEditing
+   */
+  unsigned char ThreadBackgroundPriorityForEditing;
+  /**
+   * \see clang_createIndex()
+   */
+  unsigned ExcludeDeclarationsFromPCH : 1;
+  /**
+   * \see clang_createIndex()
+   */
+  unsigned DisplayDiagnostics : 1;
+  /**
+   * Store PCH in memory. If zero, PCH are stored in temporary files.
+   */
+  unsigned StorePreamblesInMemory : 1;
+  unsigned /*Reserved*/ : 13;
+
+  /**
+   * The path to a directory, in which to store temporary PCH files. If null or
+   * empty, the default system temporary directory is used. These PCH files are
+   * deleted on clean exit but stay on disk if the program crashes or is killed.
+   *
+   * This option is ignored if \a StorePreamblesInMemory is non-zero.
+   *
+   * Libclang does not create the directory at the specified path in the file
+   * system. Therefore it must exist, or storing PCH files will fail.
+   */
+  const char *PreambleStoragePath;
+  /**
+   * Specifies a path which will contain log files for certain libclang
+   * invocations. A null value implies that libclang invocations are not logged.
+   */
+  const char *InvocationEmissionPath;
+} CXIndexOptions;
+
+/**
+ * Provides a shared context for creating translation units.
+ *
+ * Call this function instead of clang_createIndex() if you need to configure
+ * the additional options in CXIndexOptions.
+ *
+ * \returns The created index or null in case of error, such as an unsupported
+ * value of options->Size.
+ *
+ * For example:
+ * \code
+ * CXIndex createIndex(const char *ApplicationTemporaryPath) {
+ *   const int ExcludeDeclarationsFromPCH = 1;
+ *   const int DisplayDiagnostics = 1;
+ *   CXIndex Idx;
+ * #if CINDEX_VERSION_MINOR >= 64
+ *   CXIndexOptions Opts;
+ *   memset(&Opts, 0, sizeof(Opts));
+ *   Opts.Size = sizeof(CXIndexOptions);
+ *   Opts.ThreadBackgroundPriorityForIndexing = 1;
+ *   Opts.ExcludeDeclarationsFromPCH = ExcludeDeclarationsFromPCH;
+ *   Opts.DisplayDiagnostics = DisplayDiagnostics;
+ *   Opts.PreambleStoragePath = ApplicationTemporaryPath;
+ *   Idx = clang_createIndexWithOptions(&Opts);
+ *   if (Idx)
+ *     return Idx;
+ *   fprintf(stderr,
+ *           "clang_createIndexWithOptions() failed. "
+ *           "CINDEX_VERSION_MINOR = %d, sizeof(CXIndexOptions) = %u\n",
+ *           CINDEX_VERSION_MINOR, Opts.Size);
+ * #else
+ *   (void)ApplicationTemporaryPath;
+ * #endif
+ *   Idx = clang_createIndex(ExcludeDeclarationsFromPCH, DisplayDiagnostics);
+ *   clang_CXIndex_setGlobalOptions(
+ *       Idx, clang_CXIndex_getGlobalOptions(Idx) |
+ *                CXGlobalOpt_ThreadBackgroundPriorityForIndexing);
+ *   return Idx;
+ * }
+ * \endcode
+ *
+ * \sa clang_createIndex()
+ */
+CINDEX_LINKAGE CXIndex
+clang_createIndexWithOptions(const CXIndexOptions *options);
+
 /**
  * Sets general options associated with a CXIndex.
  *
+ * This function is DEPRECATED. Set
+ * CXIndexOptions::ThreadBackgroundPriorityForIndexing and/or
+ * CXIndexOptions::ThreadBackgroundPriorityForEditing and call
+ * clang_createIndexWithOptions() instead.
+ *
  * For example:
  * \code
  * CXIndex idx = ...;
@@ -327,6 +465,9 @@ CINDEX_LINKAGE void clang_CXIndex_setGlobalOptions(CXIndex, unsigned options);
 /**
  * Gets the general options associated with a CXIndex.
  *
+ * This function allows to obtain the final option values used by libclang after
+ * specifying the option policies via CXChoice enumerators.
+ *
  * \returns A bitmask of options, a bitwise OR of CXGlobalOpt_XXX flags that
  * are associated with the given CXIndex object.
  */
@@ -335,6 +476,9 @@ CINDEX_LINKAGE unsigned clang_CXIndex_getGlobalOptions(CXIndex);
 /**
  * Sets the invocation emission path option in a CXIndex.
  *
+ * This function is DEPRECATED. Set CXIndexOptions::InvocationEmissionPath and
+ * call clang_createIndexWithOptions() instead.
+ *
  * The invocation emission path specifies a path which will contain log
  * files for certain libclang invocations. A null value (default) implies that
  * libclang invocations are not logged..
@@ -2787,10 +2931,15 @@ enum CXTypeKind {
   CXType_OCLIntelSubgroupAVCImeResult = 169,
   CXType_OCLIntelSubgroupAVCRefResult = 170,
   CXType_OCLIntelSubgroupAVCSicResult = 171,
+  CXType_OCLIntelSubgroupAVCImeResultSingleReferenceStreamout = 172,
+  CXType_OCLIntelSubgroupAVCImeResultDualReferenceStreamout = 173,
+  CXType_OCLIntelSubgroupAVCImeSingleReferenceStreamin = 174,
+  CXType_OCLIntelSubgroupAVCImeDualReferenceStreamin = 175,
+
+  /* Old aliases for AVC OpenCL extension types. */
   CXType_OCLIntelSubgroupAVCImeResultSingleRefStreamout = 172,
   CXType_OCLIntelSubgroupAVCImeResultDualRefStreamout = 173,
   CXType_OCLIntelSubgroupAVCImeSingleRefStreamin = 174,
-
   CXType_OCLIntelSubgroupAVCImeDualRefStreamin = 175,
 
   CXType_ExtVector = 176,
@@ -2888,9 +3037,25 @@ CINDEX_LINKAGE unsigned long long
 clang_getEnumConstantDeclUnsignedValue(CXCursor C);
 
 /**
- * Retrieve the bit width of a bit field declaration as an integer.
+ * Returns non-zero if the cursor specifies a Record member that is a bit-field.
+ */
+CINDEX_LINKAGE unsigned clang_Cursor_isBitField(CXCursor C);
+
+/**
+ * Retrieve the bit width of a bit-field declaration as an integer.
  *
- * If a cursor that is not a bit field declaration is passed in, -1 is returned.
+ * If the cursor does not reference a bit-field, or if the bit-field's width
+ * expression cannot be evaluated, -1 is returned.
+ *
+ * For example:
+ * \code
+ * if (clang_Cursor_isBitField(Cursor)) {
+ *   int Width = clang_getFieldDeclBitWidth(Cursor);
+ *   if (Width != -1) {
+ *     // The bit-field width is not value-dependent.
+ *   }
+ * }
+ * \endcode
  */
 CINDEX_LINKAGE int clang_getFieldDeclBitWidth(CXCursor C);
 
@@ -3519,12 +3684,6 @@ CINDEX_LINKAGE CXType clang_Type_getTemplateArgumentAsType(CXType T,
  */
 CINDEX_LINKAGE enum CXRefQualifierKind clang_Type_getCXXRefQualifier(CXType T);
 
-/**
- * Returns non-zero if the cursor specifies a Record member that is a
- *   bitfield.
- */
-CINDEX_LINKAGE unsigned clang_Cursor_isBitField(CXCursor C);
-
 /**
  * Returns 1 if the base class specified by the cursor with kind
  *   CX_CXXBaseSpecifier is virtual.
diff --git a/clang/include/clang/AST/Decl.h b/clang/include/clang/AST/Decl.h
index e213c5fe29c68..0585fda3a566c 100644
--- a/clang/include/clang/AST/Decl.h
+++ b/clang/include/clang/AST/Decl.h
@@ -395,9 +395,7 @@ class NamedDecl : public Decl {
 
   /// Get the linkage from a semantic point of view. Entities in
   /// anonymous namespaces are external (in c++98).
-  Linkage getFormalLinkage() const {
-    return clang::getFormalLinkage(getLinkageInternal());
-  }
+  Linkage getFormalLinkage() const;
 
   /// True if this decl has external linkage.
   bool hasExternalFormalLinkage() const {
diff --git a/clang/include/clang/AST/DeclBase.h b/clang/include/clang/AST/DeclBase.h
index 7c3b755a438f1..e736f827f04b4 100644
--- a/clang/include/clang/AST/DeclBase.h
+++ b/clang/include/clang/AST/DeclBase.h
@@ -1227,6 +1227,10 @@ class alignas(8) Decl {
   /// have a FunctionType.
   const FunctionType *getFunctionType(bool BlocksToo = true) const;
 
+  // Looks through the Decl's underlying type to determine if it's a
+  // function pointer type.
+  bool isFunctionPointerType() const;
+
 private:
   void setAttrsImpl(const AttrVec& Attrs, ASTContext &Ctx);
   void setDeclContextsImpl(DeclContext *SemaDC, DeclContext *LexicalDC,
diff --git a/clang/include/clang/AST/Expr.h b/clang/include/clang/AST/Expr.h
index df2573324b752..28c23c5c0be91 100644
--- a/clang/include/clang/AST/Expr.h
+++ b/clang/include/clang/AST/Expr.h
@@ -4685,13 +4685,14 @@ class VAArgExpr : public Expr {
 };
 
 /// Represents a function call to one of __builtin_LINE(), __builtin_COLUMN(),
-/// __builtin_FUNCTION(), __builtin_FILE(), or __builtin_source_location().
+/// __builtin_FUNCTION(), __builtin_FILE(), __builtin_FILE_NAME(),
+/// or __builtin_source_location().
 class SourceLocExpr final : public Expr {
   SourceLocation BuiltinLoc, RParenLoc;
   DeclContext *ParentContext;
 
 public:
-  enum IdentKind { Function, File, Line, Column, SourceLocStruct };
+  enum IdentKind { Function, File, FileName, Line, Column, SourceLocStruct };
 
   SourceLocExpr(const ASTContext &Ctx, IdentKind Type, QualType ResultTy,
                 SourceLocation BLoc, SourceLocation RParenLoc,
@@ -4715,6 +4716,7 @@ class SourceLocExpr final : public Expr {
   bool isIntType() const {
     switch (getIdentKind()) {
     case File:
+    case FileName:
     case Function:
     case SourceLocStruct:
       return false;
diff --git a/clang/include/clang/AST/TemplateName.h b/clang/include/clang/AST/TemplateName.h
index 2f108ea00726e..d56361b500596 100644
--- a/clang/include/clang/AST/TemplateName.h
+++ b/clang/include/clang/AST/TemplateName.h
@@ -351,9 +351,7 @@ class TemplateName {
   /// error.
   void dump() const;
 
-  void Profile(llvm::FoldingSetNodeID &ID) {
-    ID.AddPointer(Storage.getOpaqueValue());
-  }
+  void Profile(llvm::FoldingSetNodeID &ID);
 
   /// Retrieve the template name as a void pointer.
   void *getAsVoidPointer() const { return Storage.getOpaqueValue(); }
diff --git a/clang/include/clang/AST/Type.h b/clang/include/clang/AST/Type.h
index 10cb4c819d7ce..9ecc29bd38fd1 100644
--- a/clang/include/clang/AST/Type.h
+++ b/clang/include/clang/AST/Type.h
@@ -2700,6 +2700,8 @@ class BuiltinType : public Type {
 
   bool isSVEBool() const { return getKind() == Kind::SveBool; }
 
+  bool isSVECount() const { return getKind() == Kind::SveCount; }
+
   /// Determines whether the given kind corresponds to a placeholder type.
   static bool isPlaceholderTypeKind(Kind K) {
     return K >= Overload;
@@ -4932,6 +4934,8 @@ class AttributedType : public Type, public llvm::FoldingSetNode {
 
   bool isMSTypeSpec() const;
 
+  bool isWebAssemblyFuncrefSpec() const;
+
   bool isCallingConv() const;
 
   std::optional<NullabilityKind> getImmediateNullability() const;
diff --git a/clang/include/clang/Analysis/Analyses/ReachableCode.h b/clang/include/clang/Analysis/Analyses/ReachableCode.h
index 514b9458d331c..f1b63f74b6c80 100644
--- a/clang/include/clang/Analysis/Analyses/ReachableCode.h
+++ b/clang/include/clang/Analysis/Analyses/ReachableCode.h
@@ -48,11 +48,9 @@ class Callback {
   virtual void anchor();
 public:
   virtual ~Callback() {}
-  virtual void HandleUnreachable(UnreachableKind UK,
-                                 SourceLocation L,
-                                 SourceRange ConditionVal,
-                                 SourceRange R1,
-                                 SourceRange R2) = 0;
+  virtual void HandleUnreachable(UnreachableKind UK, SourceLocation L,
+                                 SourceRange ConditionVal, SourceRange R1,
+                                 SourceRange R2, bool HasFallThroughAttr) = 0;
 };
 
 /// ScanReachableFromBlock - Mark all blocks reachable from Start.
diff --git a/clang/include/clang/Analysis/FlowSensitive/DataflowAnalysisContext.h b/clang/include/clang/Analysis/FlowSensitive/DataflowAnalysisContext.h
index 74748c8723ce8..970b17be224db 100644
--- a/clang/include/clang/Analysis/FlowSensitive/DataflowAnalysisContext.h
+++ b/clang/include/clang/Analysis/FlowSensitive/DataflowAnalysisContext.h
@@ -125,7 +125,7 @@ class DataflowAnalysisContext {
   ///
   ///  `D` must not be assigned a storage location.
   void setStorageLocation(const ValueDecl &D, StorageLocation &Loc) {
-    assert(DeclToLoc.find(&D) == DeclToLoc.end());
+    assert(!DeclToLoc.contains(&D));
     DeclToLoc[&D] = &Loc;
   }
 
@@ -143,7 +143,7 @@ class DataflowAnalysisContext {
   ///  `E` must not be assigned a storage location.
   void setStorageLocation(const Expr &E, StorageLocation &Loc) {
     const Expr &CanonE = ignoreCFGOmittedNodes(E);
-    assert(ExprToLoc.find(&CanonE) == ExprToLoc.end());
+    assert(!ExprToLoc.contains(&CanonE));
     ExprToLoc[&CanonE] = &Loc;
   }
 
diff --git a/clang/include/clang/Basic/AArch64SVEACLETypes.def b/clang/include/clang/Basic/AArch64SVEACLETypes.def
index b98a07436e941..56af270e1d10a 100644
--- a/clang/include/clang/Basic/AArch64SVEACLETypes.def
+++ b/clang/include/clang/Basic/AArch64SVEACLETypes.def
@@ -49,6 +49,11 @@
   SVE_TYPE(Name, Id, SingletonId)
 #endif
 
+#ifndef SVE_OPAQUE_TYPE
+#define SVE_OPAQUE_TYPE(Name, MangledName, Id, SingletonId)                    \
+  SVE_TYPE(Name, Id, SingletonId)
+#endif
+
 //===- Vector point types -----------------------------------------------===//
 
 
@@ -124,7 +129,12 @@ SVE_VECTOR_TYPE("__clang_svfloat64x4_t", "svfloat64x4_t", SveFloat64x4, SveFloat
 SVE_VECTOR_TYPE("__clang_svbfloat16x4_t", "svbfloat16x4_t", SveBFloat16x4, SveBFloat16x4Ty, 32, 16, true, false, true)
 
 SVE_PREDICATE_TYPE("__SVBool_t", "__SVBool_t", SveBool, SveBoolTy, 16)
+SVE_PREDICATE_TYPE("__clang_svboolx2_t", "svboolx2_t", SveBoolx2, SveBoolx2Ty, 32)
+SVE_PREDICATE_TYPE("__clang_svboolx4_t", "svboolx4_t", SveBoolx4, SveBoolx4Ty, 64)
+
+SVE_OPAQUE_TYPE("__SVCount_t", "__SVCount_t", SveCount, SveCountTy)
 
 #undef SVE_VECTOR_TYPE
 #undef SVE_PREDICATE_TYPE
+#undef SVE_OPAQUE_TYPE
 #undef SVE_TYPE
diff --git a/clang/include/clang/Basic/AddressSpaces.h b/clang/include/clang/Basic/AddressSpaces.h
index 2f2c5d5826bc3..7b723d508fff1 100644
--- a/clang/include/clang/Basic/AddressSpaces.h
+++ b/clang/include/clang/Basic/AddressSpaces.h
@@ -59,6 +59,9 @@ enum class LangAS : unsigned {
   // HLSL specific address spaces.
   hlsl_groupshared,
 
+  // Wasm specific address spaces.
+  wasm_funcref,
+
   // This denotes the count of language-specific address spaces and also
   // the offset added to the target-specific address spaces, which are usually
   // specified by address space attributes __attribute__(address_space(n))).
diff --git a/clang/include/clang/Basic/Attr.td b/clang/include/clang/Basic/Attr.td
index 8858bb6bec850..6c55465926bf7 100644
--- a/clang/include/clang/Basic/Attr.td
+++ b/clang/include/clang/Basic/Attr.td
@@ -168,6 +168,12 @@ def FunctionLike : SubsetSubject<DeclBase,
                                  [{S->getFunctionType(false) != nullptr}],
                                  "functions, function pointers">;
 
+// Function Pointer is a stricter version of FunctionLike that only allows function
+// pointers.
+def FunctionPointer : SubsetSubject<DeclBase,
+                                    [{S->isFunctionPointerType()}],
+                                    "functions pointers">;
+
 def OpenCLKernelFunction
     : SubsetSubject<Function, [{S->hasAttr<OpenCLKernelAttr>()}],
                     "kernel functions">;
@@ -792,7 +798,8 @@ def XRayLogArgs : InheritableAttr {
 def PatchableFunctionEntry
     : InheritableAttr,
       TargetSpecificAttr<TargetArch<
-          ["aarch64", "aarch64_be", "riscv32", "riscv64", "x86", "x86_64"]>> {
+          ["aarch64", "aarch64_be", "loongarch32", "loongarch64", "riscv32",
+           "riscv64", "x86", "x86_64"]>> {
   let Spellings = [GCC<"patchable_function_entry">];
   let Subjects = SubjectList<[Function, ObjCMethod]>;
   let Args = [UnsignedArgument<"Count">, DefaultIntArgument<"Offset", 0>];
@@ -4130,6 +4137,13 @@ def FunctionReturnThunks : InheritableAttr,
   let Subjects = SubjectList<[Function]>;
   let Documentation = [FunctionReturnThunksDocs];
 }
+
+def WebAssemblyFuncref : TypeAttr, TargetSpecificAttr<TargetWebAssembly> {
+  let Spellings = [Keyword<"__funcref">];
+  let Documentation = [WebAssemblyExportNameDocs];
+  let Subjects = SubjectList<[FunctionPointer], ErrorDiag>;
+}
+
 def ReadOnlyPlacement : InheritableAttr {
   let Spellings = [Clang<"enforce_read_only_placement">];
   let Subjects = SubjectList<[Record]>;
diff --git a/clang/include/clang/Basic/AttrDocs.td b/clang/include/clang/Basic/AttrDocs.td
index cbf28688a7408..1bb9e691725d3 100644
--- a/clang/include/clang/Basic/AttrDocs.td
+++ b/clang/include/clang/Basic/AttrDocs.td
@@ -5328,7 +5328,7 @@ takes precedence over the command line option ``-fpatchable-function-entry=N,M``
 ``M`` defaults to 0 if omitted.
 
 This attribute is only supported on
-aarch64/aarch64-be/riscv32/riscv64/i386/x86-64 targets.
+aarch64/aarch64-be/loongarch32/loongarch64/riscv32/riscv64/i386/x86-64 targets.
 }];
 }
 
@@ -6938,3 +6938,12 @@ def ReadOnlyPlacementDocs : Documentation {
      ``enforce_read_only_placement`` attribute.
   }];
 }
+
+def WebAssemblyFuncrefDocs : Documentation {
+  let Category = DocCatType;
+  let Content = [{
+Clang supports the ``__funcref`` attribute for the WebAssembly target. 
+This attribute may be attached to a function pointer type, where it modifies 
+its underlying representation to be a WebAssembly ``funcref``.
+  }];
+}
diff --git a/clang/include/clang/Basic/Builtins.def b/clang/include/clang/Basic/Builtins.def
index 6db599a3de116..957375eccb84a 100644
--- a/clang/include/clang/Basic/Builtins.def
+++ b/clang/include/clang/Basic/Builtins.def
@@ -350,6 +350,11 @@ BUILTIN(__builtin_roundf, "ff"  , "Fnc")
 BUILTIN(__builtin_roundf16, "hh"  , "Fnc")
 BUILTIN(__builtin_roundl, "LdLd"  , "Fnc")
 BUILTIN(__builtin_roundf128, "LLdLLd"  , "Fnc")
+BUILTIN(__builtin_roundeven, "dd"  , "Fnc")
+BUILTIN(__builtin_roundevenf, "ff"  , "Fnc")
+BUILTIN(__builtin_roundevenf16, "hh"  , "Fnc")
+BUILTIN(__builtin_roundevenl, "LdLd"  , "Fnc")
+BUILTIN(__builtin_roundevenf128, "LLdLLd"  , "Fnc")
 BUILTIN(__builtin_scalbln , "ddLi", "Fne")
 BUILTIN(__builtin_scalblnf, "ffLi", "Fne")
 BUILTIN(__builtin_scalblnl, "LdLdLi", "Fne")
@@ -392,6 +397,7 @@ BUILTIN(__builtin_truncf16, "hh", "Fnc")
 
 // Access to floating point environment
 BUILTIN(__builtin_flt_rounds, "i", "n")
+BUILTIN(__builtin_set_flt_rounds, "vi", "n")
 
 // C99 complex builtins
 BUILTIN(__builtin_cabs, "dXd", "Fne")
@@ -662,6 +668,8 @@ BUILTIN(__builtin_elementwise_max, "v.", "nct")
 BUILTIN(__builtin_elementwise_min, "v.", "nct")
 BUILTIN(__builtin_elementwise_ceil, "v.", "nct")
 BUILTIN(__builtin_elementwise_cos, "v.", "nct")
+BUILTIN(__builtin_elementwise_exp, "v.", "nct")
+BUILTIN(__builtin_elementwise_exp2, "v.", "nct")
 BUILTIN(__builtin_elementwise_floor, "v.", "nct")
 BUILTIN(__builtin_elementwise_log, "v.", "nct")
 BUILTIN(__builtin_elementwise_log2, "v.", "nct")
@@ -1420,6 +1428,10 @@ LIBBUILTIN(round, "dd", "fnc", MATH_H, ALL_LANGUAGES)
 LIBBUILTIN(roundf, "ff", "fnc", MATH_H, ALL_LANGUAGES)
 LIBBUILTIN(roundl, "LdLd", "fnc", MATH_H, ALL_LANGUAGES)
 
+LIBBUILTIN(roundeven, "dd", "fnc", MATH_H, ALL_LANGUAGES)
+LIBBUILTIN(roundevenf, "ff", "fnc", MATH_H, ALL_LANGUAGES)
+LIBBUILTIN(roundevenl, "LdLd", "fnc", MATH_H, ALL_LANGUAGES)
+
 LIBBUILTIN(scalbln, "ddLi", "fne", MATH_H, ALL_LANGUAGES)
 LIBBUILTIN(scalblnf, "ffLi", "fne", MATH_H, ALL_LANGUAGES)
 LIBBUILTIN(scalblnl, "LdLdLi", "fne", MATH_H, ALL_LANGUAGES)
diff --git a/clang/include/clang/Basic/BuiltinsNVPTX.def b/clang/include/clang/Basic/BuiltinsNVPTX.def
index 7fcd906c599b8..96531def77a78 100644
--- a/clang/include/clang/Basic/BuiltinsNVPTX.def
+++ b/clang/include/clang/Basic/BuiltinsNVPTX.def
@@ -782,7 +782,43 @@ TARGET_BUILTIN(__nvvm_atom_sys_cas_gen_ll, "LLiLLiD*LLiLLi", "n", SM_60)
 BUILTIN(__nvvm_compiler_error, "vcC*4", "n")
 BUILTIN(__nvvm_compiler_warn, "vcC*4", "n")
 
-// __ldg.  This is not implemented as a builtin by nvcc.
+BUILTIN(__nvvm_ldu_c, "ccC*", "")
+BUILTIN(__nvvm_ldu_s, "ssC*", "")
+BUILTIN(__nvvm_ldu_i, "iiC*", "")
+BUILTIN(__nvvm_ldu_l, "LiLiC*", "")
+BUILTIN(__nvvm_ldu_ll, "LLiLLiC*", "")
+
+BUILTIN(__nvvm_ldu_uc, "UcUcC*", "")
+BUILTIN(__nvvm_ldu_us, "UsUsC*", "")
+BUILTIN(__nvvm_ldu_ui, "UiUiC*", "")
+BUILTIN(__nvvm_ldu_ul, "ULiULiC*", "")
+BUILTIN(__nvvm_ldu_ull, "ULLiULLiC*", "")
+
+BUILTIN(__nvvm_ldu_h, "hhC*", "")
+BUILTIN(__nvvm_ldu_f, "ffC*", "")
+BUILTIN(__nvvm_ldu_d, "ddC*", "")
+
+BUILTIN(__nvvm_ldu_c2, "E2cE2cC*", "")
+BUILTIN(__nvvm_ldu_c4, "E4cE4cC*", "")
+BUILTIN(__nvvm_ldu_s2, "E2sE2sC*", "")
+BUILTIN(__nvvm_ldu_s4, "E4sE4sC*", "")
+BUILTIN(__nvvm_ldu_i2, "E2iE2iC*", "")
+BUILTIN(__nvvm_ldu_i4, "E4iE4iC*", "")
+BUILTIN(__nvvm_ldu_ll2, "E2LLiE2LLiC*", "")
+
+BUILTIN(__nvvm_ldu_uc2, "E2UcE2UcC*", "")
+BUILTIN(__nvvm_ldu_uc4, "E4UcE4UcC*", "")
+BUILTIN(__nvvm_ldu_us2, "E2UsE2UsC*", "")
+BUILTIN(__nvvm_ldu_us4, "E4UsE4UsC*", "")
+BUILTIN(__nvvm_ldu_ui2, "E2UiE2UiC*", "")
+BUILTIN(__nvvm_ldu_ui4, "E4UiE4UiC*", "")
+BUILTIN(__nvvm_ldu_ull2, "E2ULLiE2ULLiC*", "")
+
+BUILTIN(__nvvm_ldu_h2, "E2hE2hC*", "")
+BUILTIN(__nvvm_ldu_f2, "E2fE2fC*", "")
+BUILTIN(__nvvm_ldu_f4, "E4fE4fC*", "")
+BUILTIN(__nvvm_ldu_d2, "E2dE2dC*", "")
+
 BUILTIN(__nvvm_ldg_c, "ccC*", "")
 BUILTIN(__nvvm_ldg_s, "ssC*", "")
 BUILTIN(__nvvm_ldg_i, "iiC*", "")
diff --git a/clang/include/clang/Basic/BuiltinsWebAssembly.def b/clang/include/clang/Basic/BuiltinsWebAssembly.def
index d7e588c3f4d31..ddd8bc92130af 100644
--- a/clang/include/clang/Basic/BuiltinsWebAssembly.def
+++ b/clang/include/clang/Basic/BuiltinsWebAssembly.def
@@ -191,8 +191,15 @@ TARGET_BUILTIN(__builtin_wasm_relaxed_dot_i8x16_i7x16_add_s_i32x4, "V4iV16ScV16S
 TARGET_BUILTIN(__builtin_wasm_relaxed_dot_bf16x8_add_f32_f32x4, "V4fV8UsV8UsV4f", "nc", "relaxed-simd")
 
 // Reference Types builtins
+// Some builtins are custom type-checked - see 't' as part of the third argument,
+// in which case the argument spec (second argument) is unused.
 
 TARGET_BUILTIN(__builtin_wasm_ref_null_extern, "i", "nct", "reference-types")
 
+// A funcref represented as a function pointer with the funcref attribute
+// attached to the type, therefore SemaChecking will check for the right
+// return type.
+TARGET_BUILTIN(__builtin_wasm_ref_null_func, "i", "nct", "reference-types")
+
 #undef BUILTIN
 #undef TARGET_BUILTIN
diff --git a/clang/include/clang/Basic/DiagnosticASTKinds.td b/clang/include/clang/Basic/DiagnosticASTKinds.td
index 28120d13fd9e7..c283ee842e730 100644
--- a/clang/include/clang/Basic/DiagnosticASTKinds.td
+++ b/clang/include/clang/Basic/DiagnosticASTKinds.td
@@ -127,6 +127,8 @@ def note_constexpr_null_subobject : Note<
   "access array element of|perform pointer arithmetic on|"
   "access real component of|"
   "access imaginary component of}0 null pointer">;
+def note_constexpr_null_callee : Note<
+  "'%0' evaluates to a null function pointer">;
 def note_constexpr_function_param_value_unknown : Note<
   "function parameter %0 with unknown value cannot be used in a constant "
   "expression">;
diff --git a/clang/include/clang/Basic/DiagnosticDriverKinds.td b/clang/include/clang/Basic/DiagnosticDriverKinds.td
index 4c922650e100f..2c6bb0aa9331a 100644
--- a/clang/include/clang/Basic/DiagnosticDriverKinds.td
+++ b/clang/include/clang/Basic/DiagnosticDriverKinds.td
@@ -220,8 +220,8 @@ def err_drv_malformed_sanitizer_coverage_ignorelist : Error<
   "malformed sanitizer coverage ignorelist: '%0'">;
 def err_drv_malformed_sanitizer_metadata_ignorelist : Error<
   "malformed sanitizer metadata ignorelist: '%0'">;
-def err_drv_unsupported_static_ubsan_darwin : Error<
-  "static UndefinedBehaviorSanitizer runtime is not supported on darwin">;
+def err_drv_unsupported_static_sanitizer_darwin : Error<
+  "static %0 runtime is not supported on darwin">;
 def err_drv_duplicate_config : Error<
   "no more than one option '--config' is allowed">;
 def err_drv_cannot_open_config_file : Error<
@@ -716,4 +716,7 @@ def err_drv_riscv_unsupported_with_linker_relaxation : Error<
 
 def err_drv_loongarch_invalid_mfpu_EQ : Error<
   "invalid argument '%0' to -mfpu=; must be one of: 64, 32, none, 0 (alias for none)">;
+
+def err_drv_expand_response_file : Error<
+  "failed to expand response file: %0">;
 }
diff --git a/clang/include/clang/Basic/DiagnosticFrontendKinds.td b/clang/include/clang/Basic/DiagnosticFrontendKinds.td
index c2bce23e9a53f..9f788380f4337 100644
--- a/clang/include/clang/Basic/DiagnosticFrontendKinds.td
+++ b/clang/include/clang/Basic/DiagnosticFrontendKinds.td
@@ -11,7 +11,7 @@ class BackendInfo : CatBackend, ShowInSystemHeader;
 let Component = "Frontend" in {
 
 def err_fe_error_opening : Error<"error opening '%0': %1">;
-def err_fe_error_reading : Error<"error reading '%0'">;
+def err_fe_error_reading : Error<"error reading '%0': %1">;
 def err_fe_error_reading_stdin : Error<"error reading stdin: %0">;
 def err_fe_error_backend : Error<"error in backend: %0">, DefaultFatal;
 
@@ -278,6 +278,10 @@ def err_avx_calling_convention : Error<warn_avx_calling_convention.Summary>;
 def err_alias_to_undefined : Error<
   "%select{alias|ifunc}0 must point to a defined "
   "%select{variable or |}1function">;
+def note_alias_requires_mangled_name : Note<
+  "the %select{function or variable|function}0 specified in an %select{alias|ifunc}1 must refer to its mangled name">;
+def note_alias_mangled_name_alternative: Note<
+  "function by that name is mangled as \"%0\"">;
 def warn_alias_to_weak_alias : Warning<
   "%select{alias|ifunc}2 will always resolve to %0 even if weak definition of "
   "%1 is overridden">,
@@ -305,6 +309,10 @@ def warn_atomic_op_oversized : Warning<
   "; the access size (%0 bytes) exceeds the max lock-free size (%1  bytes)">,
 InGroup<AtomicAlignment>;
 
+def warn_sync_op_misaligned : Warning<
+  "__sync builtin operation MUST have natural alignment (consider using __atomic).">,
+  InGroup<SyncAlignment>;
+
 def warn_alias_with_section : Warning<
   "%select{alias|ifunc}1 will not be in section '%0' but in the same section "
   "as the %select{aliasee|resolver}2">,
diff --git a/clang/include/clang/Basic/DiagnosticGroups.td b/clang/include/clang/Basic/DiagnosticGroups.td
index d56aba34ac0a3..274f58b1a5ff0 100644
--- a/clang/include/clang/Basic/DiagnosticGroups.td
+++ b/clang/include/clang/Basic/DiagnosticGroups.td
@@ -65,7 +65,7 @@ def DeprecatedCoroutine :
 def AlwaysInlineCoroutine :
   DiagGroup<"always-inline-coroutine">;
 def CoroNonAlignedAllocationFunction :
-  DiagGroup<"coro-non-aligned-allocation-funciton">;
+  DiagGroup<"coro-non-aligned-allocation-function">;
 def Coroutine : DiagGroup<"coroutine", [CoroutineMissingUnhandledException, DeprecatedCoroutine,
                                         AlwaysInlineCoroutine, CoroNonAlignedAllocationFunction]>;
 def ObjCBoolConstantConversion : DiagGroup<"objc-bool-constant-conversion">;
@@ -801,6 +801,7 @@ def AtomicAlignment : DiagGroup<"atomic-alignment">;
 def CustomAtomic : DiagGroup<"custom-atomic-properties">;
 def AtomicProperties : DiagGroup<"atomic-properties",
                                  [ImplicitAtomic, CustomAtomic]>;
+def SyncAlignment : DiagGroup<"sync-alignment">;
 def ARCUnsafeRetainedAssign : DiagGroup<"arc-unsafe-retained-assign">;
 def ARCRetainCycles : DiagGroup<"arc-retain-cycles">;
 def ARCNonPodMemAccess : DiagGroup<"arc-non-pod-memaccess">;
diff --git a/clang/include/clang/Basic/DiagnosticSemaKinds.td b/clang/include/clang/Basic/DiagnosticSemaKinds.td
index 81e3f5e2d0623..bbab5752c9bfe 100644
--- a/clang/include/clang/Basic/DiagnosticSemaKinds.td
+++ b/clang/include/clang/Basic/DiagnosticSemaKinds.td
@@ -7385,6 +7385,8 @@ def err_attribute_arm_builtin_alias : Error<
   "'__clang_arm_builtin_alias' attribute can only be applied to an ARM builtin">;
 def err_attribute_arm_mve_polymorphism : Error<
   "'__clang_arm_mve_strict_polymorphism' attribute can only be applied to an MVE/NEON vector type">;
+def err_attribute_webassembly_funcref : Error<
+  "'__funcref' attribute can only be applied to a function pointer type">;
 
 def warn_setter_getter_impl_required : Warning<
   "property %0 requires method %1 to be defined - "
@@ -11798,4 +11800,6 @@ def err_wasm_reference_pr : Error<
   "%select{pointer|reference}0 to WebAssembly reference type is not allowed">;
 def err_wasm_ca_reference : Error<
   "cannot %select{capture|take address of}0 WebAssembly reference">;
+def err_wasm_funcref_not_wasm : Error<
+  "invalid use of '__funcref' keyword outside the WebAssembly triple">;
 } // end of sema component.
diff --git a/clang/include/clang/Basic/OpenCLExtensionTypes.def b/clang/include/clang/Basic/OpenCLExtensionTypes.def
index 84ffbe936b77d..17c72d69a0206 100644
--- a/clang/include/clang/Basic/OpenCLExtensionTypes.def
+++ b/clang/include/clang/Basic/OpenCLExtensionTypes.def
@@ -28,10 +28,10 @@ INTEL_SUBGROUP_AVC_TYPE(mce_result_t, MceResult)
 INTEL_SUBGROUP_AVC_TYPE(ime_result_t, ImeResult)
 INTEL_SUBGROUP_AVC_TYPE(ref_result_t, RefResult)
 INTEL_SUBGROUP_AVC_TYPE(sic_result_t, SicResult)
-INTEL_SUBGROUP_AVC_TYPE(ime_result_single_reference_streamout_t, ImeResultSingleRefStreamout)
-INTEL_SUBGROUP_AVC_TYPE(ime_result_dual_reference_streamout_t, ImeResultDualRefStreamout)
-INTEL_SUBGROUP_AVC_TYPE(ime_single_reference_streamin_t, ImeSingleRefStreamin)
-INTEL_SUBGROUP_AVC_TYPE(ime_dual_reference_streamin_t, ImeDualRefStreamin)
+INTEL_SUBGROUP_AVC_TYPE(ime_result_single_reference_streamout_t, ImeResultSingleReferenceStreamout)
+INTEL_SUBGROUP_AVC_TYPE(ime_result_dual_reference_streamout_t, ImeResultDualReferenceStreamout)
+INTEL_SUBGROUP_AVC_TYPE(ime_single_reference_streamin_t, ImeSingleReferenceStreamin)
+INTEL_SUBGROUP_AVC_TYPE(ime_dual_reference_streamin_t, ImeDualReferenceStreamin)
 
 #undef INTEL_SUBGROUP_AVC_TYPE
 #endif // INTEL_SUBGROUP_AVC_TYPE
diff --git a/clang/include/clang/Basic/ParsedAttrInfo.h b/clang/include/clang/Basic/ParsedAttrInfo.h
new file mode 100644
index 0000000000000..4444f12de9ce9
--- /dev/null
+++ b/clang/include/clang/Basic/ParsedAttrInfo.h
@@ -0,0 +1,152 @@
+//===- ParsedAttrInfo.h - Info needed to parse an attribute -----*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the ParsedAttrInfo class, which dictates how to
+// parse an attribute. This class is the one that plugins derive to
+// define a new attribute.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_BASIC_PARSEDATTRINFO_H
+#define LLVM_CLANG_BASIC_PARSEDATTRINFO_H
+
+#include "clang/Basic/AttrSubjectMatchRules.h"
+#include "clang/Basic/AttributeCommonInfo.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/Support/Registry.h"
+#include <climits>
+#include <list>
+
+namespace clang {
+
+class Decl;
+class LangOptions;
+class ParsedAttr;
+class Sema;
+class Stmt;
+class TargetInfo;
+
+struct ParsedAttrInfo {
+  /// Corresponds to the Kind enum.
+  unsigned AttrKind : 16;
+  /// The number of required arguments of this attribute.
+  unsigned NumArgs : 4;
+  /// The number of optional arguments of this attributes.
+  unsigned OptArgs : 4;
+  /// The number of non-fake arguments specified in the attribute definition.
+  unsigned NumArgMembers : 4;
+  /// True if the parsing does not match the semantic content.
+  unsigned HasCustomParsing : 1;
+  // True if this attribute accepts expression parameter pack expansions.
+  unsigned AcceptsExprPack : 1;
+  /// True if this attribute is only available for certain targets.
+  unsigned IsTargetSpecific : 1;
+  /// True if this attribute applies to types.
+  unsigned IsType : 1;
+  /// True if this attribute applies to statements.
+  unsigned IsStmt : 1;
+  /// True if this attribute has any spellings that are known to gcc.
+  unsigned IsKnownToGCC : 1;
+  /// True if this attribute is supported by #pragma clang attribute.
+  unsigned IsSupportedByPragmaAttribute : 1;
+  /// The syntaxes supported by this attribute and how they're spelled.
+  struct Spelling {
+    AttributeCommonInfo::Syntax Syntax;
+    const char *NormalizedFullName;
+  };
+  ArrayRef<Spelling> Spellings;
+  // The names of the known arguments of this attribute.
+  ArrayRef<const char *> ArgNames;
+
+protected:
+  constexpr ParsedAttrInfo(AttributeCommonInfo::Kind AttrKind =
+                               AttributeCommonInfo::NoSemaHandlerAttribute)
+      : AttrKind(AttrKind), NumArgs(0), OptArgs(0), NumArgMembers(0),
+        HasCustomParsing(0), AcceptsExprPack(0), IsTargetSpecific(0), IsType(0),
+        IsStmt(0), IsKnownToGCC(0), IsSupportedByPragmaAttribute(0) {}
+
+  constexpr ParsedAttrInfo(AttributeCommonInfo::Kind AttrKind, unsigned NumArgs,
+                           unsigned OptArgs, unsigned NumArgMembers,
+                           unsigned HasCustomParsing, unsigned AcceptsExprPack,
+                           unsigned IsTargetSpecific, unsigned IsType,
+                           unsigned IsStmt, unsigned IsKnownToGCC,
+                           unsigned IsSupportedByPragmaAttribute,
+                           ArrayRef<Spelling> Spellings,
+                           ArrayRef<const char *> ArgNames)
+      : AttrKind(AttrKind), NumArgs(NumArgs), OptArgs(OptArgs),
+        NumArgMembers(NumArgMembers), HasCustomParsing(HasCustomParsing),
+        AcceptsExprPack(AcceptsExprPack), IsTargetSpecific(IsTargetSpecific),
+        IsType(IsType), IsStmt(IsStmt), IsKnownToGCC(IsKnownToGCC),
+        IsSupportedByPragmaAttribute(IsSupportedByPragmaAttribute),
+        Spellings(Spellings), ArgNames(ArgNames) {}
+
+public:
+  virtual ~ParsedAttrInfo() = default;
+
+  /// Check if this attribute has specified spelling.
+  bool hasSpelling(AttributeCommonInfo::Syntax Syntax, StringRef Name) const {
+    return llvm::any_of(Spellings, [&](const Spelling &S) {
+      return (S.Syntax == Syntax && S.NormalizedFullName == Name);
+    });
+  }
+
+  /// Check if this attribute appertains to D, and issue a diagnostic if not.
+  virtual bool diagAppertainsToDecl(Sema &S, const ParsedAttr &Attr,
+                                    const Decl *D) const {
+    return true;
+  }
+  /// Check if this attribute appertains to St, and issue a diagnostic if not.
+  virtual bool diagAppertainsToStmt(Sema &S, const ParsedAttr &Attr,
+                                    const Stmt *St) const {
+    return true;
+  }
+  /// Check if the given attribute is mutually exclusive with other attributes
+  /// already applied to the given declaration.
+  virtual bool diagMutualExclusion(Sema &S, const ParsedAttr &A,
+                                   const Decl *D) const {
+    return true;
+  }
+  /// Check if this attribute is allowed by the language we are compiling.
+  virtual bool acceptsLangOpts(const LangOptions &LO) const { return true; }
+
+  /// Check if this attribute is allowed when compiling for the given target.
+  virtual bool existsInTarget(const TargetInfo &Target) const { return true; }
+  /// Convert the spelling index of Attr to a semantic spelling enum value.
+  virtual unsigned
+  spellingIndexToSemanticSpelling(const ParsedAttr &Attr) const {
+    return UINT_MAX;
+  }
+  /// Returns true if the specified parameter index for this attribute in
+  /// Attr.td is an ExprArgument or VariadicExprArgument, or a subclass thereof;
+  /// returns false otherwise.
+  virtual bool isParamExpr(size_t N) const { return false; }
+  /// Populate Rules with the match rules of this attribute.
+  virtual void getPragmaAttributeMatchRules(
+      llvm::SmallVectorImpl<std::pair<attr::SubjectMatchRule, bool>> &Rules,
+      const LangOptions &LangOpts) const {}
+
+  enum AttrHandling { NotHandled, AttributeApplied, AttributeNotApplied };
+  /// If this ParsedAttrInfo knows how to handle this ParsedAttr applied to this
+  /// Decl then do so and return either AttributeApplied if it was applied or
+  /// AttributeNotApplied if it wasn't. Otherwise return NotHandled.
+  virtual AttrHandling handleDeclAttribute(Sema &S, Decl *D,
+                                           const ParsedAttr &Attr) const {
+    return NotHandled;
+  }
+
+  static const ParsedAttrInfo &get(const AttributeCommonInfo &A);
+  static ArrayRef<const ParsedAttrInfo *> getAllBuiltin();
+};
+
+typedef llvm::Registry<ParsedAttrInfo> ParsedAttrInfoRegistry;
+
+const std::list<std::unique_ptr<ParsedAttrInfo>> &getAttributePluginInstances();
+
+} // namespace clang
+
+#endif // LLVM_CLANG_BASIC_PARSEDATTRINFO_H
diff --git a/clang/include/clang/Basic/SourceManager.h b/clang/include/clang/Basic/SourceManager.h
index a877308f04122..48c6b6611219a 100644
--- a/clang/include/clang/Basic/SourceManager.h
+++ b/clang/include/clang/Basic/SourceManager.h
@@ -992,8 +992,7 @@ class SourceManager : public RefCountedBase<SourceManager> {
     if (OverriddenFilesInfo) {
       if (OverriddenFilesInfo->OverriddenFilesWithBuffer.count(File))
         return true;
-      if (OverriddenFilesInfo->OverriddenFiles.find(File) !=
-          OverriddenFilesInfo->OverriddenFiles.end())
+      if (OverriddenFilesInfo->OverriddenFiles.contains(File))
         return true;
     }
     return false;
@@ -1686,7 +1685,7 @@ class SourceManager : public RefCountedBase<SourceManager> {
   fileinfo_iterator fileinfo_begin() const { return FileInfos.begin(); }
   fileinfo_iterator fileinfo_end() const { return FileInfos.end(); }
   bool hasFileInfo(const FileEntry *File) const {
-    return FileInfos.find(File) != FileInfos.end();
+    return FileInfos.contains(File);
   }
 
   /// Print statistics to stderr.
diff --git a/clang/include/clang/Basic/TargetBuiltins.h b/clang/include/clang/Basic/TargetBuiltins.h
index 8bac04e2a2637..8fd792054c269 100644
--- a/clang/include/clang/Basic/TargetBuiltins.h
+++ b/clang/include/clang/Basic/TargetBuiltins.h
@@ -289,6 +289,7 @@ namespace clang {
     bool isGatherPrefetch() const { return Flags & IsGatherPrefetch; }
     bool isReverseUSDOT() const { return Flags & ReverseUSDOT; }
     bool isReverseMergeAnyBinOp() const { return Flags & ReverseMergeAnyBinOp; }
+    bool isReverseMergeAnyAccOp() const { return Flags & ReverseMergeAnyAccOp; }
     bool isUndef() const { return Flags & IsUndef; }
     bool isTupleCreate() const { return Flags & IsTupleCreate; }
     bool isTupleGet() const { return Flags & IsTupleGet; }
diff --git a/clang/include/clang/Basic/TargetCXXABI.h b/clang/include/clang/Basic/TargetCXXABI.h
index 14451359581d9..c113a6a048ad4 100644
--- a/clang/include/clang/Basic/TargetCXXABI.h
+++ b/clang/include/clang/Basic/TargetCXXABI.h
@@ -60,9 +60,7 @@ class TargetCXXABI {
   static const auto &getSpelling(Kind ABIKind) {
     return getSpellingMap().find(ABIKind)->second;
   }
-  static bool isABI(StringRef Name) {
-    return getABIMap().find(Name) != getABIMap().end();
-  }
+  static bool isABI(StringRef Name) { return getABIMap().contains(Name); }
 
   // Return true if this target should use the relative vtables C++ ABI by
   // default.
diff --git a/clang/include/clang/Basic/TargetInfo.h b/clang/include/clang/Basic/TargetInfo.h
index 1bdee3f416a90..6de5d90b12634 100644
--- a/clang/include/clang/Basic/TargetInfo.h
+++ b/clang/include/clang/Basic/TargetInfo.h
@@ -1333,12 +1333,16 @@ class TargetInfo : public virtual TransferrableTargetInfo,
   }
 
   /// Returns true if feature has an impact on target code
-  /// generation and get its dependent options in second argument.
-  virtual bool getFeatureDepOptions(StringRef Feature,
-                                    std::string &Options) const {
+  /// generation.
+  virtual bool doesFeatureAffectCodeGen(StringRef Feature) const {
     return true;
   }
 
+  /// For given feature return dependent ones.
+  virtual StringRef getFeatureDependencies(StringRef Feature) const {
+    return StringRef();
+  }
+
   struct BranchProtectionInfo {
     LangOptions::SignReturnAddressScopeKind SignReturnAddr =
         LangOptions::SignReturnAddressScopeKind::None;
diff --git a/clang/include/clang/Basic/TokenKinds.def b/clang/include/clang/Basic/TokenKinds.def
index aa6951e68ebc2..28862c81e9ee1 100644
--- a/clang/include/clang/Basic/TokenKinds.def
+++ b/clang/include/clang/Basic/TokenKinds.def
@@ -436,6 +436,7 @@ KEYWORD(__attribute                 , KEYALL)
 KEYWORD(__builtin_choose_expr       , KEYALL)
 KEYWORD(__builtin_offsetof          , KEYALL)
 KEYWORD(__builtin_FILE              , KEYALL)
+KEYWORD(__builtin_FILE_NAME         , KEYALL)
 KEYWORD(__builtin_FUNCTION          , KEYALL)
 KEYWORD(__builtin_LINE              , KEYALL)
 KEYWORD(__builtin_COLUMN            , KEYALL)
@@ -679,6 +680,9 @@ KEYWORD(_Nullable                , KEYALL)
 KEYWORD(_Nullable_result         , KEYALL)
 KEYWORD(_Null_unspecified        , KEYALL)
 
+// WebAssembly Type Extension
+KEYWORD(__funcref                     , KEYALL)
+
 // Microsoft extensions which should be disabled in strict conformance mode
 KEYWORD(__ptr64                       , KEYMS)
 KEYWORD(__ptr32                       , KEYMS)
diff --git a/clang/include/clang/Basic/arm_sve.td b/clang/include/clang/Basic/arm_sve.td
index d13ba0590b353..7e2fc1328a891 100644
--- a/clang/include/clang/Basic/arm_sve.td
+++ b/clang/include/clang/Basic/arm_sve.td
@@ -206,6 +206,7 @@ def IsTupleCreate             : FlagType<0x100000000>;
 def IsTupleGet                : FlagType<0x200000000>;
 def IsTupleSet                : FlagType<0x400000000>;
 def ReverseMergeAnyBinOp      : FlagType<0x800000000>; // e.g. Implement SUBR_X using SUB_X.
+def ReverseMergeAnyAccOp      : FlagType<0x1000000000>; // e.g. Implement MSB_X using MLS_X.
 
 // These must be kept in sync with the flags in include/clang/Basic/TargetBuiltins.h
 class ImmCheckType<int val> {
@@ -788,20 +789,20 @@ defm SVSUBR   : SInstZPZZ<"svsubr", "csilUcUsUiUl", "aarch64_sve_subr",  "aarch6
 
 //------------------------------------------------------------------------------
 
-multiclass SInstZPZZZ<string name, string types, string intrinsic, list<FlagType> flags=[]> {
-  def _M   : SInst<name # "[_{d}]",   "dPddd", types, MergeOp1,  intrinsic, flags>;
-  def _X   : SInst<name # "[_{d}]",   "dPddd", types, MergeAny,  intrinsic, flags>;
-  def _Z   : SInst<name # "[_{d}]",   "dPddd", types, MergeZero, intrinsic, flags>;
+multiclass SInstZPZZZ<string name, string types, string m_intrinsic, string x_intrinsic, list<FlagType> flags=[]> {
+  def _M   : SInst<name # "[_{d}]",   "dPddd", types, MergeOp1,  m_intrinsic, flags>;
+  def _X   : SInst<name # "[_{d}]",   "dPddd", types, MergeAny,  x_intrinsic, flags>;
+  def _Z   : SInst<name # "[_{d}]",   "dPddd", types, MergeZero, m_intrinsic, flags>;
 
-  def _N_M : SInst<name # "[_n_{d}]", "dPdda", types, MergeOp1,  intrinsic, flags>;
-  def _N_X : SInst<name # "[_n_{d}]", "dPdda", types, MergeAny,  intrinsic, flags>;
-  def _N_Z : SInst<name # "[_n_{d}]", "dPdda", types, MergeZero, intrinsic, flags>;
+  def _N_M : SInst<name # "[_n_{d}]", "dPdda", types, MergeOp1,  m_intrinsic, flags>;
+  def _N_X : SInst<name # "[_n_{d}]", "dPdda", types, MergeAny,  x_intrinsic, flags>;
+  def _N_Z : SInst<name # "[_n_{d}]", "dPdda", types, MergeZero, m_intrinsic, flags>;
 }
 
-defm SVMAD : SInstZPZZZ<"svmad", "csilUcUsUiUl", "aarch64_sve_mad">;
-defm SVMLA : SInstZPZZZ<"svmla", "csilUcUsUiUl", "aarch64_sve_mla">;
-defm SVMLS : SInstZPZZZ<"svmls", "csilUcUsUiUl", "aarch64_sve_mls">;
-defm SVMSB : SInstZPZZZ<"svmsb", "csilUcUsUiUl", "aarch64_sve_msb">;
+defm SVMAD : SInstZPZZZ<"svmad", "csilUcUsUiUl", "aarch64_sve_mad", "aarch64_sve_mad">;
+defm SVMLA : SInstZPZZZ<"svmla", "csilUcUsUiUl", "aarch64_sve_mla", "aarch64_sve_mla">;
+defm SVMLS : SInstZPZZZ<"svmls", "csilUcUsUiUl", "aarch64_sve_mls", "aarch64_sve_mls">;
+defm SVMSB : SInstZPZZZ<"svmsb", "csilUcUsUiUl", "aarch64_sve_msb", "aarch64_sve_msb">;
 
 //------------------------------------------------------------------------------
 
@@ -980,18 +981,18 @@ defm SVREVW : SInstZPZ<"svrevw", "lUl",          "aarch64_sve_revw">;
 defm SVABS_F : SInstZPZ<"svabs", "hfd", "aarch64_sve_fabs">;
 defm SVNEG_F : SInstZPZ<"svneg", "hfd", "aarch64_sve_fneg">;
 
-defm SVABD_F  : SInstZPZZ<"svabd",  "hfd", "aarch64_sve_fabd",   "aarch64_sve_fabd">;
-defm SVADD_F  : SInstZPZZ<"svadd",  "hfd", "aarch64_sve_fadd",   "aarch64_sve_fadd">;
-defm SVDIV_F  : SInstZPZZ<"svdiv",  "hfd", "aarch64_sve_fdiv",   "aarch64_sve_fdiv">;
-defm SVDIVR_F : SInstZPZZ<"svdivr", "hfd", "aarch64_sve_fdivr",  "aarch64_sve_fdivr">;
-defm SVMAX_F  : SInstZPZZ<"svmax",  "hfd", "aarch64_sve_fmax",   "aarch64_sve_fmax">;
-defm SVMAXNM  : SInstZPZZ<"svmaxnm","hfd", "aarch64_sve_fmaxnm", "aarch64_sve_fmaxnm">;
-defm SVMIN_F  : SInstZPZZ<"svmin",  "hfd", "aarch64_sve_fmin",   "aarch64_sve_fmin">;
-defm SVMINNM  : SInstZPZZ<"svminnm","hfd", "aarch64_sve_fminnm", "aarch64_sve_fminnm">;
-defm SVMUL_F  : SInstZPZZ<"svmul",  "hfd", "aarch64_sve_fmul",   "aarch64_sve_fmul">;
-defm SVMULX   : SInstZPZZ<"svmulx", "hfd", "aarch64_sve_fmulx",  "aarch64_sve_fmulx">;
-defm SVSUB_F  : SInstZPZZ<"svsub",  "hfd", "aarch64_sve_fsub",   "aarch64_sve_fsub">;
-defm SVSUBR_F : SInstZPZZ<"svsubr", "hfd", "aarch64_sve_fsubr",  "aarch64_sve_fsubr">;
+defm SVABD_F  : SInstZPZZ<"svabd",  "hfd", "aarch64_sve_fabd",   "aarch64_sve_fabd_u">;
+defm SVADD_F  : SInstZPZZ<"svadd",  "hfd", "aarch64_sve_fadd",   "aarch64_sve_fadd_u">;
+defm SVDIV_F  : SInstZPZZ<"svdiv",  "hfd", "aarch64_sve_fdiv",   "aarch64_sve_fdiv_u">;
+defm SVDIVR_F : SInstZPZZ<"svdivr", "hfd", "aarch64_sve_fdivr",  "aarch64_sve_fdiv_u", [ReverseMergeAnyBinOp]>;
+defm SVMAX_F  : SInstZPZZ<"svmax",  "hfd", "aarch64_sve_fmax",   "aarch64_sve_fmax_u">;
+defm SVMAXNM  : SInstZPZZ<"svmaxnm","hfd", "aarch64_sve_fmaxnm", "aarch64_sve_fmaxnm_u">;
+defm SVMIN_F  : SInstZPZZ<"svmin",  "hfd", "aarch64_sve_fmin",   "aarch64_sve_fmin_u">;
+defm SVMINNM  : SInstZPZZ<"svminnm","hfd", "aarch64_sve_fminnm", "aarch64_sve_fminnm_u">;
+defm SVMUL_F  : SInstZPZZ<"svmul",  "hfd", "aarch64_sve_fmul",   "aarch64_sve_fmul_u">;
+defm SVMULX   : SInstZPZZ<"svmulx", "hfd", "aarch64_sve_fmulx",  "aarch64_sve_fmulx_u">;
+defm SVSUB_F  : SInstZPZZ<"svsub",  "hfd", "aarch64_sve_fsub",   "aarch64_sve_fsub_u">;
+defm SVSUBR_F : SInstZPZZ<"svsubr", "hfd", "aarch64_sve_fsubr",  "aarch64_sve_fsub_u", [ReverseMergeAnyBinOp]>;
 
 defm SVRECPX : SInstZPZ<"svrecpx", "hfd", "aarch64_sve_frecpx">;
 defm SVRINTA : SInstZPZ<"svrinta", "hfd", "aarch64_sve_frinta">;
@@ -1016,14 +1017,14 @@ def SVSCALE_N_M : SInst<"svscale[_n_{d}]", "dPdK", "hfd", MergeOp1,  "aarch64_sv
 def SVSCALE_N_X : SInst<"svscale[_n_{d}]", "dPdK", "hfd", MergeAny,  "aarch64_sve_fscale">;
 def SVSCALE_N_Z : SInst<"svscale[_n_{d}]", "dPdK", "hfd", MergeZero, "aarch64_sve_fscale">;
 
-defm SVMAD_F  : SInstZPZZZ<"svmad",  "hfd", "aarch64_sve_fmad">;
-defm SVMLA_F  : SInstZPZZZ<"svmla",  "hfd", "aarch64_sve_fmla">;
-defm SVMLS_F  : SInstZPZZZ<"svmls",  "hfd", "aarch64_sve_fmls">;
-defm SVMSB_F  : SInstZPZZZ<"svmsb",  "hfd", "aarch64_sve_fmsb">;
-defm SVNMAD_F : SInstZPZZZ<"svnmad", "hfd", "aarch64_sve_fnmad">;
-defm SVNMLA_F : SInstZPZZZ<"svnmla", "hfd", "aarch64_sve_fnmla">;
-defm SVNMLS_F : SInstZPZZZ<"svnmls", "hfd", "aarch64_sve_fnmls">;
-defm SVNMSB_F : SInstZPZZZ<"svnmsb", "hfd", "aarch64_sve_fnmsb">;
+defm SVMAD_F  : SInstZPZZZ<"svmad",  "hfd", "aarch64_sve_fmad",  "aarch64_sve_fmla_u",  [ReverseMergeAnyAccOp]>;
+defm SVMLA_F  : SInstZPZZZ<"svmla",  "hfd", "aarch64_sve_fmla",  "aarch64_sve_fmla_u">;
+defm SVMLS_F  : SInstZPZZZ<"svmls",  "hfd", "aarch64_sve_fmls",  "aarch64_sve_fmls_u">;
+defm SVMSB_F  : SInstZPZZZ<"svmsb",  "hfd", "aarch64_sve_fmsb",  "aarch64_sve_fmls_u",  [ReverseMergeAnyAccOp]>;
+defm SVNMAD_F : SInstZPZZZ<"svnmad", "hfd", "aarch64_sve_fnmad", "aarch64_sve_fnmla_u", [ReverseMergeAnyAccOp]>;
+defm SVNMLA_F : SInstZPZZZ<"svnmla", "hfd", "aarch64_sve_fnmla", "aarch64_sve_fnmla_u">;
+defm SVNMLS_F : SInstZPZZZ<"svnmls", "hfd", "aarch64_sve_fnmls", "aarch64_sve_fnmls_u">;
+defm SVNMSB_F : SInstZPZZZ<"svnmsb", "hfd", "aarch64_sve_fnmsb", "aarch64_sve_fnmls_u", [ReverseMergeAnyAccOp]>;
 
 def SVCADD_M : SInst<"svcadd[_{d}]", "dPddi",  "hfd", MergeOp1,  "aarch64_sve_fcadd", [], [ImmCheck<3, ImmCheckComplexRot90_270>]>;
 def SVCADD_X : SInst<"svcadd[_{d}]", "dPddi",  "hfd", MergeAny,  "aarch64_sve_fcadd", [], [ImmCheck<3, ImmCheckComplexRot90_270>]>;
diff --git a/clang/include/clang/Driver/Driver.h b/clang/include/clang/Driver/Driver.h
index c9136ec4ae690..e3e98bad99127 100644
--- a/clang/include/clang/Driver/Driver.h
+++ b/clang/include/clang/Driver/Driver.h
@@ -194,6 +194,9 @@ class Driver {
   /// The file to log CC_PRINT_PROC_STAT_FILE output to, if enabled.
   std::string CCPrintStatReportFilename;
 
+  /// The file to log CC_PRINT_INTERNAL_STAT_FILE output to, if enabled.
+  std::string CCPrintInternalStatReportFilename;
+
   /// The file to log CC_PRINT_OPTIONS output to, if enabled.
   std::string CCPrintOptionsFilename;
 
@@ -258,6 +261,10 @@ class Driver {
   /// performance report to CC_PRINT_PROC_STAT_FILE or to stdout.
   unsigned CCPrintProcessStats : 1;
 
+  /// Set CC_PRINT_INTERNAL_STAT mode, which causes the driver to dump internal
+  /// performance report to CC_PRINT_INTERNAL_STAT_FILE or to stdout.
+  unsigned CCPrintInternalStats : 1;
+
   /// Pointer to the ExecuteCC1Tool function, if available.
   /// When the clangDriver lib is used through clang.exe, this provides a
   /// shortcut for executing the -cc1 command-line directly, in the same
@@ -627,10 +634,19 @@ class Driver {
   /// Returns the default name for linked images (e.g., "a.out").
   const char *getDefaultImageName() const;
 
-  // Creates a temp file with $Prefix-%%%%%%.$Suffix
+  /// Creates a temp file.
+  /// 1. If \p MultipleArch is false or \p BoundArch is empty, the temp file is
+  ///    in the temporary directory with name $Prefix-%%%%%%.$Suffix.
+  /// 2. If \p MultipleArch is true and \p BoundArch is not empty,
+  ///    2a. If \p NeedUniqueDirectory is false, the temp file is in the
+  ///        temporary directory with name $Prefix-$BoundArch-%%%%%.$Suffix.
+  ///    2b. If \p NeedUniqueDirectory is true, the temp file is in a unique
+  ///        subdiretory with random name under the temporary directory, and
+  ///        the temp file itself has name $Prefix-$BoundArch.$Suffix.
   const char *CreateTempFile(Compilation &C, StringRef Prefix, StringRef Suffix,
                              bool MultipleArchs = false,
-                             StringRef BoundArch = {}) const;
+                             StringRef BoundArch = {},
+                             bool NeedUniqueDirectory = false) const;
 
   /// GetNamedOutputPath - Return the name to use for the output of
   /// the action \p JA. The result is appended to the compilation's
@@ -799,6 +815,16 @@ llvm::StringRef getDriverMode(StringRef ProgName, ArrayRef<const char *> Args);
 /// Checks whether the value produced by getDriverMode is for CL mode.
 bool IsClangCL(StringRef DriverMode);
 
+/// Expand response files from a clang driver or cc1 invocation.
+///
+/// \param Args The arguments that will be expanded.
+/// \param ClangCLMode Whether clang is in CL mode.
+/// \param Alloc Allocator for new arguments.
+/// \param FS Filesystem to use when expanding files.
+llvm::Error expandResponseFiles(SmallVectorImpl<const char *> &Args,
+                                bool ClangCLMode, llvm::BumpPtrAllocator &Alloc,
+                                llvm::vfs::FileSystem *FS = nullptr);
+
 } // end namespace driver
 } // end namespace clang
 
diff --git a/clang/include/clang/Driver/Options.td b/clang/include/clang/Driver/Options.td
index 70882e110fa52..a05e61ac0e92f 100644
--- a/clang/include/clang/Driver/Options.td
+++ b/clang/include/clang/Driver/Options.td
@@ -1167,9 +1167,9 @@ def extract_api : Flag<["-"], "extract-api">, Flags<[CC1Option]>, Group<Action_G
   HelpText<"Extract API information">;
 def product_name_EQ: Joined<["--"], "product-name=">, Flags<[CC1Option]>,
   MarshallingInfoString<FrontendOpts<"ProductName">>;
-def extract_api_ignores_EQ: Joined<["--"], "extract-api-ignores=">, Flags<[CC1Option]>,
-    HelpText<"File containing a new line separated list of API symbols to ignore when extracting API information.">,
-    MarshallingInfoString<FrontendOpts<"ExtractAPIIgnoresFile">>;
+def extract_api_ignores_EQ: CommaJoined<["--"], "extract-api-ignores=">, Flags<[CC1Option]>,
+    HelpText<"Comma separated list of files containing a new line separated list of API symbols to ignore when extracting API information.">,
+    MarshallingInfoStringVector<FrontendOpts<"ExtractAPIIgnoresFileList">>;
 def e : JoinedOrSeparate<["-"], "e">, Flags<[LinkerInput]>, Group<Link_Group>;
 def fmax_tokens_EQ : Joined<["-"], "fmax-tokens=">, Group<f_Group>, Flags<[CC1Option]>,
   HelpText<"Max total number of preprocessed tokens for -Wmax-tokens.">,
@@ -1215,7 +1215,7 @@ defm xl_pragma_pack : BoolFOption<"xl-pragma-pack",
 def shared_libsan : Flag<["-"], "shared-libsan">,
   HelpText<"Dynamically link the sanitizer runtime">;
 def static_libsan : Flag<["-"], "static-libsan">,
-  HelpText<"Statically link the sanitizer runtime">;
+  HelpText<"Statically link the sanitizer runtime (Not supported for ASan, TSan or UBSan on darwin)">;
 def : Flag<["-"], "shared-libasan">, Alias<shared_libsan>;
 def fasm : Flag<["-"], "fasm">, Group<f_Group>;
 
@@ -2237,13 +2237,13 @@ def flax_vector_conversions : Flag<["-"], "flax-vector-conversions">, Group<f_Gr
 def flimited_precision_EQ : Joined<["-"], "flimited-precision=">, Group<f_Group>;
 def fapple_link_rtlib : Flag<["-"], "fapple-link-rtlib">, Group<f_Group>,
   HelpText<"Force linking the clang builtins runtime library">;
-def flto_EQ : Joined<["-"], "flto=">, Flags<[CoreOption, CC1Option]>, Group<f_Group>,
+def flto_EQ : Joined<["-"], "flto=">, Flags<[CoreOption, CC1Option, FC1Option, FlangOption]>, Group<f_Group>,
   HelpText<"Set LTO mode">, Values<"thin,full">;
 def flto_EQ_jobserver : Flag<["-"], "flto=jobserver">, Group<f_Group>,
   Alias<flto_EQ>, AliasArgs<["full"]>, HelpText<"Enable LTO in 'full' mode">;
 def flto_EQ_auto : Flag<["-"], "flto=auto">, Group<f_Group>,
   Alias<flto_EQ>, AliasArgs<["full"]>, HelpText<"Enable LTO in 'full' mode">;
-def flto : Flag<["-"], "flto">, Flags<[CoreOption, CC1Option]>, Group<f_Group>,
+def flto : Flag<["-"], "flto">, Flags<[CoreOption, CC1Option, FC1Option, FlangOption]>, Group<f_Group>,
   Alias<flto_EQ>, AliasArgs<["full"]>, HelpText<"Enable LTO in 'full' mode">;
 def fno_lto : Flag<["-"], "fno-lto">, Flags<[CoreOption, CC1Option]>, Group<f_Group>,
   HelpText<"Disable LTO mode (default)">;
@@ -3416,10 +3416,12 @@ def mbig_endian : Flag<["-"], "mbig-endian">, Flags<[NoXarchOption]>;
 def EB : Flag<["-"], "EB">, Alias<mbig_endian>;
 def m16 : Flag<["-"], "m16">, Group<m_Group>, Flags<[NoXarchOption, CoreOption]>;
 def m32 : Flag<["-"], "m32">, Group<m_Group>, Flags<[NoXarchOption, CoreOption]>;
+def maix32 : Flag<["-"], "maix32">, Group<m_Group>, Flags<[NoXarchOption]>;
 def mqdsp6_compat : Flag<["-"], "mqdsp6-compat">, Group<m_Group>, Flags<[NoXarchOption,CC1Option]>,
   HelpText<"Enable hexagon-qdsp6 backward compatibility">,
   MarshallingInfoFlag<LangOpts<"HexagonQdsp6Compat">>;
 def m64 : Flag<["-"], "m64">, Group<m_Group>, Flags<[NoXarchOption, CoreOption]>;
+def maix64 : Flag<["-"], "maix64">, Group<m_Group>, Flags<[NoXarchOption]>;
 def mx32 : Flag<["-"], "mx32">, Group<m_Group>, Flags<[NoXarchOption, CoreOption]>;
 def mabi_EQ : Joined<["-"], "mabi=">, Group<m_Group>;
 def miamcu : Flag<["-"], "miamcu">, Group<m_Group>, Flags<[NoXarchOption, CoreOption]>,
@@ -4253,19 +4255,15 @@ def rpath : Separate<["-"], "rpath">, Flags<[LinkerInput]>, Group<Link_Group>;
 def rtlib_EQ : Joined<["-", "--"], "rtlib=">,
   HelpText<"Compiler runtime library to use">;
 def frtlib_add_rpath: Flag<["-"], "frtlib-add-rpath">, Flags<[NoArgumentUnused]>,
-  HelpText<"Add -rpath with architecture-specific resource directory to the linker flags">;
+  HelpText<"Add -rpath with architecture-specific resource directory to the linker flags. "
+  "When --hip-link is specified, also add -rpath with HIP runtime library directory to the linker flags">;
 def fno_rtlib_add_rpath: Flag<["-"], "fno-rtlib-add-rpath">, Flags<[NoArgumentUnused]>,
-  HelpText<"Do not add -rpath with architecture-specific resource directory to the linker flags">;
+  HelpText<"Do not add -rpath with architecture-specific resource directory to the linker flags. "
+  "When --hip-link is specified, do not add -rpath with HIP runtime library directory to the linker flags">;
 def offload_add_rpath: Flag<["--"], "offload-add-rpath">, Flags<[NoArgumentUnused]>,
-  HelpText<"Add -rpath with HIP runtime library directory to the linker flags">;
+  Alias<frtlib_add_rpath>;
 def no_offload_add_rpath: Flag<["--"], "no-offload-add-rpath">, Flags<[NoArgumentUnused]>,
-  HelpText<"Do not add -rpath with HIP runtime library directory to the linker flags">;
-defm openmp_implicit_rpath: BoolFOption<"openmp-implicit-rpath",
-  LangOpts<"OpenMP">,
-  DefaultTrue,
-  PosFlag<SetTrue, [], "Set rpath on OpenMP executables">,
-  NegFlag<SetFalse>,
-  BothFlags<[NoArgumentUnused]>>;
+  Alias<frtlib_add_rpath>;
 def r : Flag<["-"], "r">, Flags<[LinkerInput,NoArgumentUnused]>,
         Group<Link_Group>;
 def save_temps_EQ : Joined<["-", "--"], "save-temps=">, Flags<[CC1Option, FlangOption, NoXarchOption]>,
@@ -6149,6 +6147,9 @@ def print_stats : Flag<["-"], "print-stats">,
 def stats_file : Joined<["-"], "stats-file=">,
   HelpText<"Filename to write statistics to">,
   MarshallingInfoString<FrontendOpts<"StatsFile">>;
+def stats_file_append : Flag<["-"], "stats-file-append">,
+  HelpText<"If stats should be appended to stats-file instead of overwriting it">,
+  MarshallingInfoFlag<FrontendOpts<"AppendStats">>;
 def fdump_record_layouts_simple : Flag<["-"], "fdump-record-layouts-simple">,
   HelpText<"Dump record layout information in a simple form used for testing">,
   MarshallingInfoFlag<LangOpts<"DumpRecordLayoutsSimple">>;
@@ -6530,7 +6531,7 @@ def fno_cuda_host_device_constexpr : Flag<["-"], "fno-cuda-host-device-constexpr
 
 def fopenmp_is_device : Flag<["-"], "fopenmp-is-device">,
   HelpText<"Generate code only for an OpenMP target device.">,
-  Flags<[CC1Option, NoDriverOption]>;
+  Flags<[CC1Option, FC1Option, NoDriverOption]>;
 def fopenmp_host_ir_file_path : Separate<["-"], "fopenmp-host-ir-file-path">,
   HelpText<"Path to the IR file produced by the frontend for the host.">,
   Flags<[CC1Option, NoDriverOption]>;
@@ -7044,6 +7045,16 @@ def _SLASH_Gv : CLFlag<"Gv">,
 def _SLASH_Gregcall : CLFlag<"Gregcall">,
   HelpText<"Set __regcall as a default calling convention">;
 
+// GNU Driver aliases
+
+def : Separate<["-"], "Xmicrosoft-visualc-tools-root">, Alias<_SLASH_vctoolsdir>;
+def : Separate<["-"], "Xmicrosoft-visualc-tools-version">,
+    Alias<_SLASH_vctoolsversion>;
+def : Separate<["-"], "Xmicrosoft-windows-sdk-root">,
+    Alias<_SLASH_winsdkdir>;
+def : Separate<["-"], "Xmicrosoft-windows-sdk-version">,
+    Alias<_SLASH_winsdkversion>;
+
 // Ignored:
 
 def _SLASH_analyze_ : CLIgnoredFlag<"analyze-">;
diff --git a/clang/include/clang/ExtractAPI/APIIgnoresList.h b/clang/include/clang/ExtractAPI/APIIgnoresList.h
index 43c546102a2d6..3eee8e336cb63 100644
--- a/clang/include/clang/ExtractAPI/APIIgnoresList.h
+++ b/clang/include/clang/ExtractAPI/APIIgnoresList.h
@@ -7,7 +7,7 @@
 //===----------------------------------------------------------------------===//
 ///
 /// \file This file defines APIIgnoresList which is a type that allows querying
-/// a file containing symbols to ignore when extracting API information.
+/// files containing symbols to ignore when extracting API information.
 ///
 //===----------------------------------------------------------------------===//
 
@@ -44,11 +44,13 @@ struct IgnoresFileNotFound : public llvm::ErrorInfo<IgnoresFileNotFound> {
 /// A type that provides access to a new line separated list of symbol names to
 /// ignore when extracting API information.
 struct APIIgnoresList {
-  /// The API to use for generating from the file at \p IgnoresFilePath.
+  using FilePathList = std::vector<std::string>;
+
+  /// The API to use for generating from the files at \p IgnoresFilePathList.
   ///
   /// \returns an initialized APIIgnoresList or an Error.
-  static llvm::Expected<APIIgnoresList> create(llvm::StringRef IgnoresFilePath,
-                                               FileManager &FM);
+  static llvm::Expected<APIIgnoresList>
+  create(const FilePathList &IgnoresFilePathList, FileManager &FM);
 
   APIIgnoresList() = default;
 
@@ -58,14 +60,14 @@ struct APIIgnoresList {
 
 private:
   using SymbolNameList = llvm::SmallVector<llvm::StringRef, 32>;
+  using BufferList = llvm::SmallVector<std::unique_ptr<llvm::MemoryBuffer>>;
 
-  APIIgnoresList(SymbolNameList SymbolsToIgnore,
-                 std::unique_ptr<llvm::MemoryBuffer> Buffer)
-      : SymbolsToIgnore(std::move(SymbolsToIgnore)), Buffer(std::move(Buffer)) {
-  }
+  APIIgnoresList(SymbolNameList SymbolsToIgnore, BufferList Buffers)
+      : SymbolsToIgnore(std::move(SymbolsToIgnore)),
+        Buffers(std::move(Buffers)) {}
 
   SymbolNameList SymbolsToIgnore;
-  std::unique_ptr<llvm::MemoryBuffer> Buffer;
+  BufferList Buffers;
 };
 
 } // namespace extractapi
diff --git a/clang/include/clang/Frontend/ASTUnit.h b/clang/include/clang/Frontend/ASTUnit.h
index c792028902728..cb1ea39c17f86 100644
--- a/clang/include/clang/Frontend/ASTUnit.h
+++ b/clang/include/clang/Frontend/ASTUnit.h
@@ -119,11 +119,13 @@ class ASTUnit {
   std::shared_ptr<PreprocessorOptions>    PPOpts;
   IntrusiveRefCntPtr<ASTReader> Reader;
   bool HadModuleLoaderFatalFailure = false;
+  bool StorePreamblesInMemory = false;
 
   struct ASTWriterData;
   std::unique_ptr<ASTWriterData> WriterData;
 
   FileSystemOptions FileSystemOpts;
+  std::string PreambleStoragePath;
 
   /// The AST consumer that received information about the translation
   /// unit as it was parsed or loaded.
@@ -802,6 +804,13 @@ class ASTUnit {
   ///
   /// \param ResourceFilesPath - The path to the compiler resource files.
   ///
+  /// \param StorePreamblesInMemory - Whether to store PCH in memory. If false,
+  /// PCH are stored in temporary files.
+  ///
+  /// \param PreambleStoragePath - The path to a directory, in which to create
+  /// temporary PCH files. If empty, the default system temporary directory is
+  /// used. This parameter is ignored if \p StorePreamblesInMemory is true.
+  ///
   /// \param ModuleFormat - If provided, uses the specific module format.
   ///
   /// \param ErrAST - If non-null and parsing failed without any AST to return
@@ -820,7 +829,8 @@ class ASTUnit {
       const char **ArgBegin, const char **ArgEnd,
       std::shared_ptr<PCHContainerOperations> PCHContainerOps,
       IntrusiveRefCntPtr<DiagnosticsEngine> Diags, StringRef ResourceFilesPath,
-      bool OnlyLocalDecls = false,
+      bool StorePreamblesInMemory = false,
+      StringRef PreambleStoragePath = StringRef(), bool OnlyLocalDecls = false,
       CaptureDiagsKind CaptureDiagnostics = CaptureDiagsKind::None,
       ArrayRef<RemappedFile> RemappedFiles = std::nullopt,
       bool RemappedFilesKeepOriginalName = true,
diff --git a/clang/include/clang/Frontend/FrontendOptions.h b/clang/include/clang/Frontend/FrontendOptions.h
index 6efe3cdd58022..85183a3812b42 100644
--- a/clang/include/clang/Frontend/FrontendOptions.h
+++ b/clang/include/clang/Frontend/FrontendOptions.h
@@ -278,6 +278,8 @@ class FrontendOptions {
   /// Show frontend performance metrics and statistics.
   unsigned ShowStats : 1;
 
+  unsigned AppendStats : 1;
+
   /// print the supported cpus for the current target
   unsigned PrintSupportedCPUs : 1;
 
@@ -453,8 +455,9 @@ class FrontendOptions {
   std::string ProductName;
 
   // Currently this is only used as part of the `-extract-api` action.
-  /// The file providing a list of APIs to ignore when extracting documentation
-  std::string ExtractAPIIgnoresFile;
+  // A comma seperated list of files providing a list of APIs to
+  // ignore when extracting documentation.
+  std::vector<std::string> ExtractAPIIgnoresFileList;
 
   /// Args to pass to the plugins
   std::map<std::string, std::vector<std::string>> PluginArgs;
@@ -510,16 +513,16 @@ class FrontendOptions {
 public:
   FrontendOptions()
       : DisableFree(false), RelocatablePCH(false), ShowHelp(false),
-        ShowStats(false), TimeTrace(false), ShowVersion(false),
-        FixWhatYouCan(false), FixOnlyWarnings(false), FixAndRecompile(false),
-        FixToTemporaries(false), ARCMTMigrateEmitARCErrors(false),
-        SkipFunctionBodies(false), UseGlobalModuleIndex(true),
-        GenerateGlobalModuleIndex(true), ASTDumpDecls(false),
-        ASTDumpLookups(false), BuildingImplicitModule(false),
-        BuildingImplicitModuleUsesLock(true), ModulesEmbedAllFiles(false),
-        IncludeTimestamps(true), UseTemporary(true),
-        AllowPCMWithCompilerErrors(false), ModulesShareFileManager(true),
-        TimeTraceGranularity(500) {}
+        ShowStats(false), AppendStats(false), TimeTrace(false),
+        ShowVersion(false), FixWhatYouCan(false), FixOnlyWarnings(false),
+        FixAndRecompile(false), FixToTemporaries(false),
+        ARCMTMigrateEmitARCErrors(false), SkipFunctionBodies(false),
+        UseGlobalModuleIndex(true), GenerateGlobalModuleIndex(true),
+        ASTDumpDecls(false), ASTDumpLookups(false),
+        BuildingImplicitModule(false), BuildingImplicitModuleUsesLock(true),
+        ModulesEmbedAllFiles(false), IncludeTimestamps(true),
+        UseTemporary(true), AllowPCMWithCompilerErrors(false),
+        ModulesShareFileManager(true), TimeTraceGranularity(500) {}
 
   /// getInputKindForExtension - Return the appropriate input kind for a file
   /// extension. For example, "c" would return Language::C.
diff --git a/clang/include/clang/Frontend/PrecompiledPreamble.h b/clang/include/clang/Frontend/PrecompiledPreamble.h
index db9f33ae5961f..798870bf24fe1 100644
--- a/clang/include/clang/Frontend/PrecompiledPreamble.h
+++ b/clang/include/clang/Frontend/PrecompiledPreamble.h
@@ -75,6 +75,10 @@ class PrecompiledPreamble {
   /// \param StoreInMemory Store PCH in memory. If false, PCH will be stored in
   /// a temporary file.
   ///
+  /// \param StoragePath The path to a directory, in which to create a temporary
+  /// file to store PCH in. If empty, the default system temporary directory is
+  /// used. This parameter is ignored if \p StoreInMemory is true.
+  ///
   /// \param Callbacks A set of callbacks to be executed when building
   /// the preamble.
   static llvm::ErrorOr<PrecompiledPreamble>
@@ -83,7 +87,8 @@ class PrecompiledPreamble {
         DiagnosticsEngine &Diagnostics,
         IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS,
         std::shared_ptr<PCHContainerOperations> PCHContainerOps,
-        bool StoreInMemory, PreambleCallbacks &Callbacks);
+        bool StoreInMemory, StringRef StoragePath,
+        PreambleCallbacks &Callbacks);
 
   PrecompiledPreamble(PrecompiledPreamble &&);
   PrecompiledPreamble &operator=(PrecompiledPreamble &&);
diff --git a/clang/include/clang/Lex/Preprocessor.h b/clang/include/clang/Lex/Preprocessor.h
index d332db496fc74..a7ba486e0e50f 100644
--- a/clang/include/clang/Lex/Preprocessor.h
+++ b/clang/include/clang/Lex/Preprocessor.h
@@ -193,11 +193,6 @@ class Preprocessor {
   LangOptions::FPEvalMethodKind CurrentFPEvalMethod =
       LangOptions::FPEvalMethodKind::FEM_UnsetOnCommandLine;
 
-  // Keeps the value of the last evaluation method before a
-  // `pragma float_control (precise,off) is applied.
-  LangOptions::FPEvalMethodKind LastFPEvalMethod =
-      LangOptions::FPEvalMethodKind::FEM_UnsetOnCommandLine;
-
   // The most recent pragma location where the floating point evaluation
   // method was modified. This is used to determine whether the
   // 'pragma clang fp eval_method' was used whithin the current scope.
@@ -2335,14 +2330,6 @@ class Preprocessor {
     return LastFPEvalPragmaLocation;
   }
 
-  LangOptions::FPEvalMethodKind getLastFPEvalMethod() const {
-    return LastFPEvalMethod;
-  }
-
-  void setLastFPEvalMethod(LangOptions::FPEvalMethodKind Val) {
-    LastFPEvalMethod = Val;
-  }
-
   void setCurrentFPEvalMethod(SourceLocation PragmaLoc,
                               LangOptions::FPEvalMethodKind Val) {
     assert(Val != LangOptions::FEM_UnsetOnCommandLine &&
@@ -2855,6 +2842,11 @@ class Preprocessor {
                                       const LangOptions &LangOpts,
                                       const TargetInfo &TI);
 
+  static void processPathToFileName(SmallVectorImpl<char> &FileName,
+                                    const PresumedLoc &PLoc,
+                                    const LangOptions &LangOpts,
+                                    const TargetInfo &TI);
+
 private:
   void emitMacroDeprecationWarning(const Token &Identifier) const;
   void emitRestrictExpansionWarning(const Token &Identifier) const;
@@ -2863,7 +2855,7 @@ class Preprocessor {
   /// This boolean state keeps track if the current scanned token (by this PP)
   /// is in an "-Wunsafe-buffer-usage" opt-out region. Assuming PP scans a
   /// translation unit in a linear order.
-  bool InSafeBufferOptOutRegion = 0;
+  bool InSafeBufferOptOutRegion = false;
 
   /// Hold the start location of the current "-Wunsafe-buffer-usage" opt-out
   /// region if PP is currently in such a region.  Hold undefined value
diff --git a/clang/include/clang/Parse/Parser.h b/clang/include/clang/Parse/Parser.h
index 96963fb6aa807..2174852a94509 100644
--- a/clang/include/clang/Parse/Parser.h
+++ b/clang/include/clang/Parse/Parser.h
@@ -2513,7 +2513,8 @@ class Parser : public CodeCompletionHandler {
   /// this is a constructor declarator.
   bool isConstructorDeclarator(
       bool Unqualified, bool DeductionGuide = false,
-      DeclSpec::FriendSpecified IsFriend = DeclSpec::FriendSpecified::No);
+      DeclSpec::FriendSpecified IsFriend = DeclSpec::FriendSpecified::No,
+      const ParsedTemplateInfo *TemplateInfo = nullptr);
 
   /// Specifies the context in which type-id/expression
   /// disambiguation will occur.
@@ -2929,6 +2930,7 @@ class Parser : public CodeCompletionHandler {
                                   SourceLocation AttrNameLoc,
                                   ParsedAttributes &Attrs);
   void ParseMicrosoftTypeAttributes(ParsedAttributes &attrs);
+  void ParseWebAssemblyFuncrefTypeAttribute(ParsedAttributes &Attrs);
   void DiagnoseAndSkipExtendedMicrosoftTypeAttributes();
   SourceLocation SkipExtendedMicrosoftTypeAttributes();
   void ParseMicrosoftInheritanceClassAttributes(ParsedAttributes &attrs);
diff --git a/clang/include/clang/Sema/DeclSpec.h b/clang/include/clang/Sema/DeclSpec.h
index 69fe2c541607b..b0bf87dc18d79 100644
--- a/clang/include/clang/Sema/DeclSpec.h
+++ b/clang/include/clang/Sema/DeclSpec.h
@@ -62,9 +62,18 @@ namespace clang {
 /// often used as if it meant "present".
 ///
 /// The actual scope is described by getScopeRep().
+///
+/// If the kind of getScopeRep() is TypeSpec then TemplateParamLists may be empty
+/// or contain the template parameter lists attached to the current declaration.
+/// Consider the following example:
+/// template <class T> void SomeType<T>::some_method() {}
+/// If CXXScopeSpec refers to SomeType<T> then TemplateParamLists will contain
+/// a single element referring to template <class T>.
+
 class CXXScopeSpec {
   SourceRange Range;
   NestedNameSpecifierLocBuilder Builder;
+  ArrayRef<TemplateParameterList *> TemplateParamLists;
 
 public:
   SourceRange getRange() const { return Range; }
@@ -74,6 +83,13 @@ class CXXScopeSpec {
   SourceLocation getBeginLoc() const { return Range.getBegin(); }
   SourceLocation getEndLoc() const { return Range.getEnd(); }
 
+  void setTemplateParamLists(ArrayRef<TemplateParameterList *> L) {
+    TemplateParamLists = L;
+  }
+  ArrayRef<TemplateParameterList *> getTemplateParamLists() const {
+    return TemplateParamLists;
+  }
+
   /// Retrieve the representation of the nested-name-specifier.
   NestedNameSpecifier *getScopeRep() const {
     return Builder.getRepresentation();
diff --git a/clang/include/clang/Sema/ParsedAttr.h b/clang/include/clang/Sema/ParsedAttr.h
index f060564e6719e..e2b4be48c0bd1 100644
--- a/clang/include/clang/Sema/ParsedAttr.h
+++ b/clang/include/clang/Sema/ParsedAttr.h
@@ -17,12 +17,12 @@
 #include "clang/Basic/AttrSubjectMatchRules.h"
 #include "clang/Basic/AttributeCommonInfo.h"
 #include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/ParsedAttrInfo.h"
 #include "clang/Basic/SourceLocation.h"
 #include "clang/Sema/Ownership.h"
 #include "llvm/ADT/PointerUnion.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/Support/Allocator.h"
-#include "llvm/Support/Registry.h"
 #include "llvm/Support/VersionTuple.h"
 #include <cassert>
 #include <cstddef>
@@ -36,124 +36,10 @@ class Decl;
 class Expr;
 class IdentifierInfo;
 class LangOptions;
-class ParsedAttr;
 class Sema;
 class Stmt;
 class TargetInfo;
 
-struct ParsedAttrInfo {
-  /// Corresponds to the Kind enum.
-  unsigned AttrKind : 16;
-  /// The number of required arguments of this attribute.
-  unsigned NumArgs : 4;
-  /// The number of optional arguments of this attributes.
-  unsigned OptArgs : 4;
-  /// The number of non-fake arguments specified in the attribute definition.
-  unsigned NumArgMembers : 4;
-  /// True if the parsing does not match the semantic content.
-  unsigned HasCustomParsing : 1;
-  // True if this attribute accepts expression parameter pack expansions.
-  unsigned AcceptsExprPack : 1;
-  /// True if this attribute is only available for certain targets.
-  unsigned IsTargetSpecific : 1;
-  /// True if this attribute applies to types.
-  unsigned IsType : 1;
-  /// True if this attribute applies to statements.
-  unsigned IsStmt : 1;
-  /// True if this attribute has any spellings that are known to gcc.
-  unsigned IsKnownToGCC : 1;
-  /// True if this attribute is supported by #pragma clang attribute.
-  unsigned IsSupportedByPragmaAttribute : 1;
-  /// The syntaxes supported by this attribute and how they're spelled.
-  struct Spelling {
-    AttributeCommonInfo::Syntax Syntax;
-    const char *NormalizedFullName;
-  };
-  ArrayRef<Spelling> Spellings;
-  // The names of the known arguments of this attribute.
-  ArrayRef<const char *> ArgNames;
-
-protected:
-  constexpr ParsedAttrInfo(AttributeCommonInfo::Kind AttrKind =
-                               AttributeCommonInfo::NoSemaHandlerAttribute)
-      : AttrKind(AttrKind), NumArgs(0), OptArgs(0), NumArgMembers(0),
-        HasCustomParsing(0), AcceptsExprPack(0), IsTargetSpecific(0), IsType(0),
-        IsStmt(0), IsKnownToGCC(0), IsSupportedByPragmaAttribute(0) {}
-
-  constexpr ParsedAttrInfo(AttributeCommonInfo::Kind AttrKind, unsigned NumArgs,
-                           unsigned OptArgs, unsigned NumArgMembers,
-                           unsigned HasCustomParsing, unsigned AcceptsExprPack,
-                           unsigned IsTargetSpecific, unsigned IsType,
-                           unsigned IsStmt, unsigned IsKnownToGCC,
-                           unsigned IsSupportedByPragmaAttribute,
-                           ArrayRef<Spelling> Spellings,
-                           ArrayRef<const char *> ArgNames)
-      : AttrKind(AttrKind), NumArgs(NumArgs), OptArgs(OptArgs),
-        NumArgMembers(NumArgMembers), HasCustomParsing(HasCustomParsing),
-        AcceptsExprPack(AcceptsExprPack), IsTargetSpecific(IsTargetSpecific),
-        IsType(IsType), IsStmt(IsStmt), IsKnownToGCC(IsKnownToGCC),
-        IsSupportedByPragmaAttribute(IsSupportedByPragmaAttribute),
-        Spellings(Spellings), ArgNames(ArgNames) {}
-
-public:
-  virtual ~ParsedAttrInfo() = default;
-
-  /// Check if this attribute appertains to D, and issue a diagnostic if not.
-  virtual bool diagAppertainsToDecl(Sema &S, const ParsedAttr &Attr,
-                                    const Decl *D) const {
-    return true;
-  }
-  /// Check if this attribute appertains to St, and issue a diagnostic if not.
-  virtual bool diagAppertainsToStmt(Sema &S, const ParsedAttr &Attr,
-                                    const Stmt *St) const {
-    return true;
-  }
-  /// Check if the given attribute is mutually exclusive with other attributes
-  /// already applied to the given declaration.
-  virtual bool diagMutualExclusion(Sema &S, const ParsedAttr &A,
-                                   const Decl *D) const {
-    return true;
-  }
-  /// Check if this attribute is allowed by the language we are compiling.
-  virtual bool acceptsLangOpts(const LangOptions &LO) const { return true; }
-
-  /// Check if this attribute is allowed when compiling for the given target.
-  virtual bool existsInTarget(const TargetInfo &Target) const {
-    return true;
-  }
-  /// Convert the spelling index of Attr to a semantic spelling enum value.
-  virtual unsigned
-  spellingIndexToSemanticSpelling(const ParsedAttr &Attr) const {
-    return UINT_MAX;
-  }
-  /// Returns true if the specified parameter index for this attribute in
-  /// Attr.td is an ExprArgument or VariadicExprArgument, or a subclass thereof;
-  /// returns false otherwise.
-  virtual bool isParamExpr(size_t N) const { return false; }
-  /// Populate Rules with the match rules of this attribute.
-  virtual void getPragmaAttributeMatchRules(
-      llvm::SmallVectorImpl<std::pair<attr::SubjectMatchRule, bool>> &Rules,
-      const LangOptions &LangOpts) const {
-  }
-  enum AttrHandling {
-    NotHandled,
-    AttributeApplied,
-    AttributeNotApplied
-  };
-  /// If this ParsedAttrInfo knows how to handle this ParsedAttr applied to this
-  /// Decl then do so and return either AttributeApplied if it was applied or
-  /// AttributeNotApplied if it wasn't. Otherwise return NotHandled.
-  virtual AttrHandling handleDeclAttribute(Sema &S, Decl *D,
-                                           const ParsedAttr &Attr) const {
-    return NotHandled;
-  }
-
-  static const ParsedAttrInfo &get(const AttributeCommonInfo &A);
-  static ArrayRef<const ParsedAttrInfo *> getAllBuiltin();
-};
-
-typedef llvm::Registry<ParsedAttrInfo> ParsedAttrInfoRegistry;
-
 /// Represents information about a change in availability for
 /// an entity, which is part of the encoding of the 'availability'
 /// attribute.
diff --git a/clang/include/clang/Sema/Sema.h b/clang/include/clang/Sema/Sema.h
index d1342876f85dc..7ff10a9d52e56 100644
--- a/clang/include/clang/Sema/Sema.h
+++ b/clang/include/clang/Sema/Sema.h
@@ -238,11 +238,9 @@ namespace threadSafety {
 
 // FIXME: No way to easily map from TemplateTypeParmTypes to
 // TemplateTypeParmDecls, so we have this horrible PointerUnion.
-using UnexpandedParameterPack = std::pair<
-    llvm::PointerUnion<
-        const TemplateTypeParmType *, const SubstTemplateTypeParmPackType *,
-        const SubstNonTypeTemplateParmPackExpr *, const NamedDecl *>,
-    SourceLocation>;
+typedef std::pair<llvm::PointerUnion<const TemplateTypeParmType *, NamedDecl *>,
+                  SourceLocation>
+    UnexpandedParameterPack;
 
 /// Describes whether we've seen any nullability information for the given
 /// file.
@@ -1047,7 +1045,7 @@ class Sema final {
   /// ExpressionEvaluationContextRecord object.
   bool isConstantEvaluatedOverride;
 
-  bool isConstantEvaluated() {
+  bool isConstantEvaluated() const {
     return ExprEvalContexts.back().isConstantEvaluated() ||
            isConstantEvaluatedOverride;
   }
@@ -1488,7 +1486,7 @@ class Sema final {
   /// Determine if VD, which must be a variable or function, is an external
   /// symbol that nonetheless can't be referenced from outside this translation
   /// unit because its type has no linkage and it's not extern "C".
-  bool isExternalWithNoLinkageType(ValueDecl *VD);
+  bool isExternalWithNoLinkageType(ValueDecl *VD) const;
 
   /// Obtain a sorted list of functions that are undefined but ODR-used.
   void getUndefinedButUsed(
@@ -3550,13 +3548,13 @@ class Sema final {
   void ActOnExitFunctionContext();
 
   /// If \p AllowLambda is true, treat lambda as function.
-  DeclContext *getFunctionLevelDeclContext(bool AllowLambda = false);
+  DeclContext *getFunctionLevelDeclContext(bool AllowLambda = false) const;
 
   /// Returns a pointer to the innermost enclosing function, or nullptr if the
   /// current context is not inside a function. If \p AllowLambda is true,
   /// this can return the call operator of an enclosing lambda, otherwise
   /// lambdas are skipped when looking for an enclosing function.
-  FunctionDecl *getCurFunctionDecl(bool AllowLambda = false);
+  FunctionDecl *getCurFunctionDecl(bool AllowLambda = false) const;
 
   /// getCurMethodDecl - If inside of a method body, this returns a pointer to
   /// the method decl for the method being parsed.  If we're currently
@@ -3566,7 +3564,7 @@ class Sema final {
   /// getCurFunctionOrMethodDecl - Return the Decl for the current ObjC method
   /// or C function we're in, otherwise return null.  If we're currently
   /// in a 'block', this returns the containing context.
-  NamedDecl *getCurFunctionOrMethodDecl();
+  NamedDecl *getCurFunctionOrMethodDecl() const;
 
   /// Add this decl to the scope shadowed decl chains.
   void PushOnScopeChains(NamedDecl *D, Scope *S, bool AddToContext = true);
@@ -3579,7 +3577,7 @@ class Sema final {
   ///        enclosing namespace set of the context, rather than contained
   ///        directly within it.
   bool isDeclInScope(NamedDecl *D, DeclContext *Ctx, Scope *S = nullptr,
-                     bool AllowInlineNamespace = false);
+                     bool AllowInlineNamespace = false) const;
 
   /// Finds the scope corresponding to the given decl context, if it
   /// happens to be an enclosing scope.  Otherwise return NULL.
@@ -4351,7 +4349,7 @@ class Sema final {
     ForExternalRedeclaration
   };
 
-  RedeclarationKind forRedeclarationInCurContext() {
+  RedeclarationKind forRedeclarationInCurContext() const {
     // A declaration with an owning module for linkage can never link against
     // anything that is not visible. We don't need to check linkage here; if
     // the context has internal linkage, redeclaration lookup won't find things
@@ -13596,6 +13594,7 @@ class Sema final {
 
   // WebAssembly builtin handling.
   bool BuiltinWasmRefNullExtern(CallExpr *TheCall);
+  bool BuiltinWasmRefNullFunc(CallExpr *TheCall);
 
 public:
   enum FormatStringType {
diff --git a/clang/include/clang/Sema/SemaInternal.h b/clang/include/clang/Sema/SemaInternal.h
index 4eca50919dc7e..842eec099540c 100644
--- a/clang/include/clang/Sema/SemaInternal.h
+++ b/clang/include/clang/Sema/SemaInternal.h
@@ -62,7 +62,7 @@ inline InheritableAttr *getDLLAttr(Decl *D) {
 }
 
 /// Retrieve the depth and index of a template parameter.
-inline std::pair<unsigned, unsigned> getDepthAndIndex(const NamedDecl *ND) {
+inline std::pair<unsigned, unsigned> getDepthAndIndex(NamedDecl *ND) {
   if (const auto *TTP = dyn_cast<TemplateTypeParmDecl>(ND))
     return std::make_pair(TTP->getDepth(), TTP->getIndex());
 
@@ -79,7 +79,7 @@ getDepthAndIndex(UnexpandedParameterPack UPP) {
   if (const auto *TTP = UPP.first.dyn_cast<const TemplateTypeParmType *>())
     return std::make_pair(TTP->getDepth(), TTP->getIndex());
 
-  return getDepthAndIndex(UPP.first.get<const NamedDecl *>());
+  return getDepthAndIndex(UPP.first.get<NamedDecl *>());
 }
 
 class TypoCorrectionConsumer : public VisibleDeclConsumer {
diff --git a/clang/include/clang/Testing/CommandLineArgs.h b/clang/include/clang/Testing/CommandLineArgs.h
index e668781ee2ce1..4dd28718dfa67 100644
--- a/clang/include/clang/Testing/CommandLineArgs.h
+++ b/clang/include/clang/Testing/CommandLineArgs.h
@@ -38,6 +38,11 @@ std::vector<std::string> getCC1ArgsForTesting(TestLanguage Lang);
 
 StringRef getFilenameForTesting(TestLanguage Lang);
 
+/// Find a target name such that looking for it in TargetRegistry by that name
+/// returns the same target. We expect that there is at least one target
+/// configured with this property.
+std::string getAnyTargetForTesting();
+
 } // end namespace clang
 
 #endif
diff --git a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
index 13d42867bdd58..87a4299c7f1b9 100644
--- a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
+++ b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
@@ -143,9 +143,8 @@ class DependencyScanningTool {
 
 class FullDependencyConsumer : public DependencyConsumer {
 public:
-  FullDependencyConsumer(const llvm::StringSet<> &AlreadySeen,
-                         LookupModuleOutputCallback LookupModuleOutput)
-      : AlreadySeen(AlreadySeen), LookupModuleOutput(LookupModuleOutput) {}
+  FullDependencyConsumer(const llvm::StringSet<> &AlreadySeen)
+      : AlreadySeen(AlreadySeen) {}
 
   void handleBuildCommand(Command Cmd) override {
     Commands.push_back(std::move(Cmd));
@@ -169,11 +168,6 @@ class FullDependencyConsumer : public DependencyConsumer {
     ContextHash = std::move(Hash);
   }
 
-  std::string lookupModuleOutput(const ModuleID &ID,
-                                 ModuleOutputKind Kind) override {
-    return LookupModuleOutput(ID, Kind);
-  }
-
   TranslationUnitDeps takeTranslationUnitDeps();
   ModuleDepsGraph takeModuleGraphDeps();
 
@@ -186,6 +180,30 @@ class FullDependencyConsumer : public DependencyConsumer {
   std::string ContextHash;
   std::vector<std::string> OutputPaths;
   const llvm::StringSet<> &AlreadySeen;
+};
+
+/// A simple dependency action controller that uses a callback. If no callback
+/// is provided, it is assumed that looking up module outputs is unreachable.
+class CallbackActionController : public DependencyActionController {
+public:
+  virtual ~CallbackActionController();
+
+  CallbackActionController(LookupModuleOutputCallback LMO)
+      : LookupModuleOutput(std::move(LMO)) {
+    if (!LookupModuleOutput) {
+      LookupModuleOutput = [](const ModuleID &,
+                              ModuleOutputKind) -> std::string {
+        llvm::report_fatal_error("unexpected call to lookupModuleOutput");
+      };
+    }
+  }
+
+  std::string lookupModuleOutput(const ModuleID &ID,
+                                 ModuleOutputKind Kind) override {
+    return LookupModuleOutput(ID, Kind);
+  }
+
+private:
   LookupModuleOutputCallback LookupModuleOutput;
 };
 
diff --git a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
index 5ada65c0a8308..350acb8f8a79e 100644
--- a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
+++ b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
@@ -57,6 +57,13 @@ class DependencyConsumer {
   virtual void handleModuleDependency(ModuleDeps MD) = 0;
 
   virtual void handleContextHash(std::string Hash) = 0;
+};
+
+/// Dependency scanner callbacks that are used during scanning to influence the
+/// behaviour of the scan - for example, to customize the scanned invocations.
+class DependencyActionController {
+public:
+  virtual ~DependencyActionController();
 
   virtual std::string lookupModuleOutput(const ModuleID &ID,
                                          ModuleOutputKind Kind) = 0;
@@ -83,15 +90,15 @@ class DependencyScanningWorker {
   bool computeDependencies(StringRef WorkingDirectory,
                            const std::vector<std::string> &CommandLine,
                            DependencyConsumer &DepConsumer,
+                           DependencyActionController &Controller,
                            DiagnosticConsumer &DiagConsumer,
                            std::optional<StringRef> ModuleName = std::nullopt);
   /// \returns A \c StringError with the diagnostic output if clang errors
   /// occurred, success otherwise.
-  llvm::Error
-  computeDependencies(StringRef WorkingDirectory,
-                      const std::vector<std::string> &CommandLine,
-                      DependencyConsumer &Consumer,
-                      std::optional<StringRef> ModuleName = std::nullopt);
+  llvm::Error computeDependencies(
+      StringRef WorkingDirectory, const std::vector<std::string> &CommandLine,
+      DependencyConsumer &Consumer, DependencyActionController &Controller,
+      std::optional<StringRef> ModuleName = std::nullopt);
 
   bool shouldEagerLoadModules() const { return EagerLoadModules; }
 
diff --git a/clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h b/clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
index 238fc84ddd11c..24d7337d26e4c 100644
--- a/clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
+++ b/clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
@@ -27,6 +27,7 @@ namespace clang {
 namespace tooling {
 namespace dependencies {
 
+class DependencyActionController;
 class DependencyConsumer;
 
 /// Modular dependency that has already been built prior to the dependency scan.
@@ -201,6 +202,7 @@ class ModuleDepCollector final : public DependencyCollector {
 public:
   ModuleDepCollector(std::unique_ptr<DependencyOutputOptions> Opts,
                      CompilerInstance &ScanInstance, DependencyConsumer &C,
+                     DependencyActionController &Controller,
                      CompilerInvocation OriginalCI, bool OptimizeArgs,
                      bool EagerLoadModules, bool IsStdModuleP1689Format);
 
@@ -218,6 +220,8 @@ class ModuleDepCollector final : public DependencyCollector {
   CompilerInstance &ScanInstance;
   /// The consumer of collected dependency information.
   DependencyConsumer &Consumer;
+  /// Callbacks for computing dependency information.
+  DependencyActionController &Controller;
   /// Path to the main source file.
   std::string MainFile;
   /// Hash identifying the compilation conditions of the current TU.
diff --git a/clang/include/clang/Tooling/Tooling.h b/clang/include/clang/Tooling/Tooling.h
index 52a81cd9e7787..7a1c62e3a3d57 100644
--- a/clang/include/clang/Tooling/Tooling.h
+++ b/clang/include/clang/Tooling/Tooling.h
@@ -506,6 +506,12 @@ llvm::Expected<std::string> getAbsolutePath(llvm::vfs::FileSystem &FS,
 void addTargetAndModeForProgramName(std::vector<std::string> &CommandLine,
                                     StringRef InvokedAs);
 
+/// Helper function that expands response files in command line.
+void addExpandedResponseFiles(std::vector<std::string> &CommandLine,
+                              llvm::StringRef WorkingDir,
+                              llvm::cl::TokenizerCallback Tokenizer,
+                              llvm::vfs::FileSystem &FS);
+
 /// Creates a \c CompilerInvocation.
 CompilerInvocation *newInvocation(DiagnosticsEngine *Diagnostics,
                                   ArrayRef<const char *> CC1Args,
diff --git a/clang/lib/ARCMigrate/FileRemapper.cpp b/clang/lib/ARCMigrate/FileRemapper.cpp
index 92027fe4f1f4b..7e56e0683c129 100644
--- a/clang/lib/ARCMigrate/FileRemapper.cpp
+++ b/clang/lib/ARCMigrate/FileRemapper.cpp
@@ -251,8 +251,7 @@ const FileEntry *FileRemapper::getOriginalFile(StringRef filePath) {
     I = ToFromMappings.find(file);
   if (I != ToFromMappings.end()) {
     file = I->second;
-    assert(FromToMappings.find(file) != FromToMappings.end() &&
-           "Original file not in mappings!");
+    assert(FromToMappings.contains(file) && "Original file not in mappings!");
   }
   return file;
 }
diff --git a/clang/lib/ARCMigrate/ObjCMT.cpp b/clang/lib/ARCMigrate/ObjCMT.cpp
index ce1decd3ba3e1..c76efd78976af 100644
--- a/clang/lib/ARCMigrate/ObjCMT.cpp
+++ b/clang/lib/ARCMigrate/ObjCMT.cpp
@@ -153,8 +153,7 @@ class ObjCMigrateASTConsumer : public ASTConsumer {
   bool canModifyFile(StringRef Path) {
     if (AllowListFilenames.empty())
       return true;
-    return AllowListFilenames.find(llvm::sys::path::filename(Path)) !=
-           AllowListFilenames.end();
+    return AllowListFilenames.contains(llvm::sys::path::filename(Path));
   }
   bool canModifyFile(OptionalFileEntryRef FE) {
     if (!FE)
diff --git a/clang/lib/AST/ASTContext.cpp b/clang/lib/AST/ASTContext.cpp
index f226832966648..5196f1e3555ac 100644
--- a/clang/lib/AST/ASTContext.cpp
+++ b/clang/lib/AST/ASTContext.cpp
@@ -2322,6 +2322,11 @@ TypeInfo ASTContext::getTypeInfoImpl(const Type *T) const {
     Width = 0;                                                                 \
     Align = 16;                                                                \
     break;
+#define SVE_OPAQUE_TYPE(Name, MangledName, Id, SingletonId)                    \
+  case BuiltinType::Id:                                                        \
+    Width = 0;                                                                 \
+    Align = 16;                                                                \
+    break;
 #include "clang/Basic/AArch64SVEACLETypes.def"
 #define PPC_VECTOR_TYPE(Name, Id, Size)                                        \
   case BuiltinType::Id:                                                        \
@@ -4036,6 +4041,10 @@ ASTContext::getBuiltinVectorTypeInfo(const BuiltinType *Ty) const {
     return SVE_INT_ELTTY(64, 2, false, 4);
   case BuiltinType::SveBool:
     return SVE_ELTTY(BoolTy, 16, 1);
+  case BuiltinType::SveBoolx2:
+    return SVE_ELTTY(BoolTy, 16, 2);
+  case BuiltinType::SveBoolx4:
+    return SVE_ELTTY(BoolTy, 16, 4);
   case BuiltinType::SveFloat16:
     return SVE_ELTTY(HalfTy, 8, 1);
   case BuiltinType::SveFloat16x2:
@@ -4119,6 +4128,7 @@ QualType ASTContext::getScalableVectorType(QualType EltTy,
 #define SVE_PREDICATE_TYPE(Name, MangledName, Id, SingletonId, NumEls)         \
   if (EltTy->isBooleanType() && NumElts == NumEls)                             \
     return SingletonId;
+#define SVE_OPAQUE_TYPE(Name, MangledName, Id, SingleTonId)
 #include "clang/Basic/AArch64SVEACLETypes.def"
   } else if (Target->hasRISCVVTypes()) {
     uint64_t EltTySize = getTypeSize(EltTy);
@@ -9549,9 +9559,10 @@ bool ASTContext::areCompatibleVectorTypes(QualType FirstVec,
 /// getSVETypeSize - Return SVE vector or predicate register size.
 static uint64_t getSVETypeSize(ASTContext &Context, const BuiltinType *Ty) {
   assert(Ty->isVLSTBuiltinType() && "Invalid SVE Type");
-  return Ty->getKind() == BuiltinType::SveBool
-             ? (Context.getLangOpts().VScaleMin * 128) / Context.getCharWidth()
-             : Context.getLangOpts().VScaleMin * 128;
+  if (Ty->getKind() == BuiltinType::SveBool ||
+      Ty->getKind() == BuiltinType::SveCount)
+    return (Context.getLangOpts().VScaleMin * 128) / Context.getCharWidth();
+  return Context.getLangOpts().VScaleMin * 128;
 }
 
 bool ASTContext::areCompatibleSveTypes(QualType FirstType,
@@ -11625,9 +11636,8 @@ static GVALinkage basicGVALinkageForFunction(const ASTContext &Context,
 
   // Non-user-provided functions get emitted as weak definitions with every
   // use, no matter whether they've been explicitly instantiated etc.
-  if (const auto *MD = dyn_cast<CXXMethodDecl>(FD))
-    if (!MD->isUserProvided())
-      return GVA_DiscardableODR;
+  if (!FD->isUserProvided())
+    return GVA_DiscardableODR;
 
   GVALinkage External;
   switch (FD->getTemplateSpecializationKind()) {
@@ -11951,7 +11961,7 @@ void ASTContext::forEachMultiversionedFunctionVersion(
        FD->getDeclContext()->getRedeclContext()->lookup(FD->getDeclName())) {
     FunctionDecl *CurFD = CurDecl->getAsFunction()->getMostRecentDecl();
     if (CurFD && hasSameType(CurFD->getType(), FD->getType()) &&
-        !llvm::is_contained(SeenDecls, CurFD)) {
+        !SeenDecls.contains(CurFD)) {
       SeenDecls.insert(CurFD);
       Pred(CurFD);
     }
@@ -13450,6 +13460,7 @@ std::vector<std::string> ASTContext::filterFunctionTargetVersionAttrs(
   TV->getFeatures(Feats);
   for (auto &Feature : Feats)
     if (Target->validateCpuSupports(Feature.str()))
+      // Use '?' to mark features that came from TargetVersion.
       ResFeats.push_back("?" + Feature.str());
   return ResFeats;
 }
@@ -13519,6 +13530,7 @@ void ASTContext::getFunctionFeatureMap(llvm::StringMap<bool> &FeatureMap,
         VersionStr.split(VersionFeatures, "+");
         for (auto &VFeature : VersionFeatures) {
           VFeature = VFeature.trim();
+          // Use '?' to mark features that came from AArch64 TargetClones.
           Features.push_back((StringRef{"?"} + VFeature).str());
         }
       }
diff --git a/clang/lib/AST/Decl.cpp b/clang/lib/AST/Decl.cpp
index 240e744ee6449..56042e5fd252f 100644
--- a/clang/lib/AST/Decl.cpp
+++ b/clang/lib/AST/Decl.cpp
@@ -600,25 +600,17 @@ static bool isExportedFromModuleInterfaceUnit(const NamedDecl *D) {
   llvm_unreachable("unexpected module ownership kind");
 }
 
+static bool isDeclaredInModuleInterfaceOrPartition(const NamedDecl *D) {
+  if (auto *M = D->getOwningModule())
+    return M->isInterfaceOrPartition();
+  return false;
+}
+
 static LinkageInfo getInternalLinkageFor(const NamedDecl *D) {
   return LinkageInfo::internal();
 }
 
 static LinkageInfo getExternalLinkageFor(const NamedDecl *D) {
-  // C++ [basic.link]p4.8:
-  //   - if the declaration of the name is attached to a named module and is not
-  //   exported
-  //     the name has module linkage;
-  //
-  // [basic.namespace.general]/p2
-  //   A namespace is never attached to a named module and never has a name with
-  //   module linkage.
-  if (isInModulePurview(D) &&
-      !isExportedFromModuleInterfaceUnit(
-          cast<NamedDecl>(D->getCanonicalDecl())) &&
-      !isa<NamespaceDecl>(D))
-    return LinkageInfo(ModuleLinkage, DefaultVisibility, false);
-
   return LinkageInfo::external();
 }
 
@@ -656,15 +648,15 @@ LinkageComputer::getLVForNamespaceScopeDecl(const NamedDecl *D,
   if (const auto *Var = dyn_cast<VarDecl>(D)) {
     // - a non-template variable of non-volatile const-qualified type, unless
     //   - it is explicitly declared extern, or
-    //   - it is inline or exported, or
+    //   - it is declared in the purview of a module interface unit
+    //     (outside the private-module-fragment, if any) or module partition, or
+    //   - it is inline, or
     //   - it was previously declared and the prior declaration did not have
     //     internal linkage
     // (There is no equivalent in C99.)
-    if (Context.getLangOpts().CPlusPlus &&
-        Var->getType().isConstQualified() &&
-        !Var->getType().isVolatileQualified() &&
-        !Var->isInline() &&
-        !isExportedFromModuleInterfaceUnit(Var) &&
+    if (Context.getLangOpts().CPlusPlus && Var->getType().isConstQualified() &&
+        !Var->getType().isVolatileQualified() && !Var->isInline() &&
+        !isDeclaredInModuleInterfaceOrPartition(Var) &&
         !isa<VarTemplateSpecializationDecl>(Var) &&
         !Var->getDescribedVarTemplate()) {
       const VarDecl *PrevVar = Var->getPreviousDecl();
@@ -1165,6 +1157,29 @@ Linkage NamedDecl::getLinkageInternal() const {
       .getLinkage();
 }
 
+/// Get the linkage from a semantic point of view. Entities in
+/// anonymous namespaces are external (in c++98).
+Linkage NamedDecl::getFormalLinkage() const {
+  Linkage InternalLinkage = getLinkageInternal();
+
+  // C++ [basic.link]p4.8:
+  //   - if the declaration of the name is attached to a named module and is not
+  //   exported
+  //     the name has module linkage;
+  //
+  // [basic.namespace.general]/p2
+  //   A namespace is never attached to a named module and never has a name with
+  //   module linkage.
+  if (isInModulePurview(this) &&
+      InternalLinkage == ExternalLinkage &&
+      !isExportedFromModuleInterfaceUnit(
+          cast<NamedDecl>(this->getCanonicalDecl())) &&
+      !isa<NamespaceDecl>(this))
+    InternalLinkage = ModuleLinkage;
+
+  return clang::getFormalLinkage(InternalLinkage);
+}
+
 LinkageInfo NamedDecl::getLinkageAndVisibility() const {
   return LinkageComputer{}.getDeclLinkageAndVisibility(this);
 }
diff --git a/clang/lib/AST/DeclBase.cpp b/clang/lib/AST/DeclBase.cpp
index c94fc602155bd..f49945f434193 100644
--- a/clang/lib/AST/DeclBase.cpp
+++ b/clang/lib/AST/DeclBase.cpp
@@ -1048,6 +1048,18 @@ const FunctionType *Decl::getFunctionType(bool BlocksToo) const {
   return Ty->getAs<FunctionType>();
 }
 
+bool Decl::isFunctionPointerType() const {
+  QualType Ty;
+  if (const auto *D = dyn_cast<ValueDecl>(this))
+    Ty = D->getType();
+  else if (const auto *D = dyn_cast<TypedefNameDecl>(this))
+    Ty = D->getUnderlyingType();
+  else
+    return false;
+
+  return Ty.getCanonicalType()->isFunctionPointerType();
+}
+
 DeclContext *Decl::getNonTransparentDeclContext() {
   assert(getDeclContext());
   return getDeclContext()->getNonTransparentContext();
diff --git a/clang/lib/AST/Expr.cpp b/clang/lib/AST/Expr.cpp
index 67862a8692acd..f82587df0d70b 100644
--- a/clang/lib/AST/Expr.cpp
+++ b/clang/lib/AST/Expr.cpp
@@ -2239,6 +2239,8 @@ StringRef SourceLocExpr::getBuiltinStr() const {
   switch (getIdentKind()) {
   case File:
     return "__builtin_FILE";
+  case FileName:
+    return "__builtin_FILE_NAME";
   case Function:
     return "__builtin_FUNCTION";
   case Line:
@@ -2277,6 +2279,14 @@ APValue SourceLocExpr::EvaluateInContext(const ASTContext &Ctx,
   };
 
   switch (getIdentKind()) {
+  case SourceLocExpr::FileName: {
+    // __builtin_FILE_NAME() is a Clang-specific extension that expands to the
+    // the last part of __builtin_FILE().
+    SmallString<256> FileName;
+    clang::Preprocessor::processPathToFileName(
+        FileName, PLoc, Ctx.getLangOpts(), Ctx.getTargetInfo());
+    return MakeStringLiteral(FileName);
+  }
   case SourceLocExpr::File: {
     SmallString<256> Path(PLoc.getFilename());
     clang::Preprocessor::processPathForFileMacro(Path, Ctx.getLangOpts(),
diff --git a/clang/lib/AST/ExprConstant.cpp b/clang/lib/AST/ExprConstant.cpp
index 6b0beb1973893..fbe92d054d6bc 100644
--- a/clang/lib/AST/ExprConstant.cpp
+++ b/clang/lib/AST/ExprConstant.cpp
@@ -7668,6 +7668,11 @@ class ExprEvaluatorBase
 
       if (!CalleeLV.getLValueOffset().isZero())
         return Error(Callee);
+      if (CalleeLV.isNullPointer()) {
+        Info.FFDiag(Callee, diag::note_constexpr_null_callee)
+            << const_cast<Expr *>(Callee);
+        return false;
+      }
       FD = dyn_cast_or_null<FunctionDecl>(
           CalleeLV.getLValueBase().dyn_cast<const ValueDecl *>());
       if (!FD)
@@ -10917,7 +10922,7 @@ bool ArrayExprEvaluator::VisitCXXConstructExpr(const CXXConstructExpr *E,
         for (unsigned I = OldElts; I < N; ++I)
           Value->getArrayInitializedElt(I) = Filler;
 
-      if (HasTrivialConstructor && N == FinalSize) {
+      if (HasTrivialConstructor && N == FinalSize && FinalSize != 1) {
         // If we have a trivial constructor, only evaluate it once and copy
         // the result into all the array elements.
         APValue &FirstResult = Value->getArrayInitializedElt(0);
diff --git a/clang/lib/AST/Interp/ByteCodeExprGen.cpp b/clang/lib/AST/Interp/ByteCodeExprGen.cpp
index bc682c92c143f..0056da191765b 100644
--- a/clang/lib/AST/Interp/ByteCodeExprGen.cpp
+++ b/clang/lib/AST/Interp/ByteCodeExprGen.cpp
@@ -1139,7 +1139,7 @@ unsigned ByteCodeExprGen<Emitter>::allocateLocalPrimitive(DeclTy &&Src,
   if (const auto *VD =
           dyn_cast_if_present<ValueDecl>(Src.dyn_cast<const Decl *>())) {
     assert(!P.getGlobal(VD));
-    assert(Locals.find(VD) == Locals.end());
+    assert(!Locals.contains(VD));
   }
 
   // FIXME: There are cases where Src.is<Expr*>() is wrong, e.g.
@@ -1161,7 +1161,7 @@ ByteCodeExprGen<Emitter>::allocateLocal(DeclTy &&Src, bool IsExtended) {
   if (const auto *VD =
           dyn_cast_if_present<ValueDecl>(Src.dyn_cast<const Decl *>())) {
     assert(!P.getGlobal(VD));
-    assert(Locals.find(VD) == Locals.end());
+    assert(!Locals.contains(VD));
   }
 
   QualType Ty;
diff --git a/clang/lib/AST/Interp/ByteCodeStmtGen.cpp b/clang/lib/AST/Interp/ByteCodeStmtGen.cpp
index a4be86c0e0639..c5f87a1c9faac 100644
--- a/clang/lib/AST/Interp/ByteCodeStmtGen.cpp
+++ b/clang/lib/AST/Interp/ByteCodeStmtGen.cpp
@@ -471,7 +471,7 @@ bool ByteCodeStmtGen<Emitter>::visitSwitchStmt(const SwitchStmt *S) {
       const Expr *Value = CS->getLHS();
       PrimType ValueT = this->classifyPrim(Value->getType());
 
-      // Compare the case statment's value to the switch condition.
+      // Compare the case statement's value to the switch condition.
       if (!this->emitGetLocal(CondT, CondVar, CS))
         return false;
       if (!this->visit(Value))
diff --git a/clang/lib/AST/Interp/Descriptor.cpp b/clang/lib/AST/Interp/Descriptor.cpp
index e9fe159eccd4b..212311cfa2ae6 100644
--- a/clang/lib/AST/Interp/Descriptor.cpp
+++ b/clang/lib/AST/Interp/Descriptor.cpp
@@ -17,30 +17,32 @@ using namespace clang;
 using namespace clang::interp;
 
 template <typename T>
-static void ctorTy(Block *, char *Ptr, bool, bool, bool, Descriptor *) {
+static void ctorTy(Block *, char *Ptr, bool, bool, bool, const Descriptor *) {
   new (Ptr) T();
 }
 
-template <typename T> static void dtorTy(Block *, char *Ptr, Descriptor *) {
+template <typename T>
+static void dtorTy(Block *, char *Ptr, const Descriptor *) {
   reinterpret_cast<T *>(Ptr)->~T();
 }
 
 template <typename T>
-static void moveTy(Block *, char *Src, char *Dst, Descriptor *) {
+static void moveTy(Block *, char *Src, char *Dst, const Descriptor *) {
   auto *SrcPtr = reinterpret_cast<T *>(Src);
   auto *DstPtr = reinterpret_cast<T *>(Dst);
   new (DstPtr) T(std::move(*SrcPtr));
 }
 
 template <typename T>
-static void ctorArrayTy(Block *, char *Ptr, bool, bool, bool, Descriptor *D) {
+static void ctorArrayTy(Block *, char *Ptr, bool, bool, bool,
+                        const Descriptor *D) {
   for (unsigned I = 0, NE = D->getNumElems(); I < NE; ++I) {
     new (&reinterpret_cast<T *>(Ptr)[I]) T();
   }
 }
 
 template <typename T>
-static void dtorArrayTy(Block *, char *Ptr, Descriptor *D) {
+static void dtorArrayTy(Block *, char *Ptr, const Descriptor *D) {
   InitMap *IM = *reinterpret_cast<InitMap **>(Ptr);
   if (IM != (InitMap *)-1)
     free(IM);
@@ -52,7 +54,7 @@ static void dtorArrayTy(Block *, char *Ptr, Descriptor *D) {
 }
 
 template <typename T>
-static void moveArrayTy(Block *, char *Src, char *Dst, Descriptor *D) {
+static void moveArrayTy(Block *, char *Src, char *Dst, const Descriptor *D) {
   for (unsigned I = 0, NE = D->getNumElems(); I < NE; ++I) {
     auto *SrcPtr = &reinterpret_cast<T *>(Src)[I];
     auto *DstPtr = &reinterpret_cast<T *>(Dst)[I];
@@ -61,7 +63,7 @@ static void moveArrayTy(Block *, char *Src, char *Dst, Descriptor *D) {
 }
 
 static void ctorArrayDesc(Block *B, char *Ptr, bool IsConst, bool IsMutable,
-                          bool IsActive, Descriptor *D) {
+                          bool IsActive, const Descriptor *D) {
   const unsigned NumElems = D->getNumElems();
   const unsigned ElemSize =
       D->ElemDesc->getAllocSize() + sizeof(InlineDescriptor);
@@ -86,7 +88,7 @@ static void ctorArrayDesc(Block *B, char *Ptr, bool IsConst, bool IsMutable,
   }
 }
 
-static void dtorArrayDesc(Block *B, char *Ptr, Descriptor *D) {
+static void dtorArrayDesc(Block *B, char *Ptr, const Descriptor *D) {
   const unsigned NumElems = D->getNumElems();
   const unsigned ElemSize =
       D->ElemDesc->getAllocSize() + sizeof(InlineDescriptor);
@@ -101,7 +103,7 @@ static void dtorArrayDesc(Block *B, char *Ptr, Descriptor *D) {
   }
 }
 
-static void moveArrayDesc(Block *B, char *Src, char *Dst, Descriptor *D) {
+static void moveArrayDesc(Block *B, char *Src, char *Dst, const Descriptor *D) {
   const unsigned NumElems = D->getNumElems();
   const unsigned ElemSize =
       D->ElemDesc->getAllocSize() + sizeof(InlineDescriptor);
@@ -123,7 +125,7 @@ static void moveArrayDesc(Block *B, char *Src, char *Dst, Descriptor *D) {
 }
 
 static void ctorRecord(Block *B, char *Ptr, bool IsConst, bool IsMutable,
-                       bool IsActive, Descriptor *D) {
+                       bool IsActive, const Descriptor *D) {
   const bool IsUnion = D->ElemRecord->isUnion();
   auto CtorSub = [=](unsigned SubOff, Descriptor *F, bool IsBase) {
     auto *Desc = reinterpret_cast<InlineDescriptor *>(Ptr + SubOff) - 1;
@@ -146,7 +148,7 @@ static void ctorRecord(Block *B, char *Ptr, bool IsConst, bool IsMutable,
     CtorSub(V.Offset, V.Desc, /*isBase=*/true);
 }
 
-static void dtorRecord(Block *B, char *Ptr, Descriptor *D) {
+static void dtorRecord(Block *B, char *Ptr, const Descriptor *D) {
   auto DtorSub = [=](unsigned SubOff, Descriptor *F) {
     if (auto Fn = F->DtorFn)
       Fn(B, Ptr + SubOff, F);
@@ -159,7 +161,7 @@ static void dtorRecord(Block *B, char *Ptr, Descriptor *D) {
     DtorSub(F.Offset, F.Desc);
 }
 
-static void moveRecord(Block *B, char *Src, char *Dst, Descriptor *D) {
+static void moveRecord(Block *B, char *Src, char *Dst, const Descriptor *D) {
   for (const auto &F : D->ElemRecord->fields()) {
     auto FieldOff = F.Offset;
     auto FieldDesc = F.Desc;
diff --git a/clang/lib/AST/Interp/Descriptor.h b/clang/lib/AST/Interp/Descriptor.h
index 57cb5728c9ba8..5b86ffb7957a0 100644
--- a/clang/lib/AST/Interp/Descriptor.h
+++ b/clang/lib/AST/Interp/Descriptor.h
@@ -30,19 +30,19 @@ using DeclTy = llvm::PointerUnion<const Decl *, const Expr *>;
 /// all the fields which contain non-trivial types.
 using BlockCtorFn = void (*)(Block *Storage, char *FieldPtr, bool IsConst,
                              bool IsMutable, bool IsActive,
-                             Descriptor *FieldDesc);
+                             const Descriptor *FieldDesc);
 
 /// Invoked when a block is destroyed. Invokes the destructors of all
 /// non-trivial nested fields of arrays and records.
 using BlockDtorFn = void (*)(Block *Storage, char *FieldPtr,
-                             Descriptor *FieldDesc);
+                             const Descriptor *FieldDesc);
 
 /// Invoked when a block with pointers referencing it goes out of scope. Such
 /// blocks are persisted: the move function copies all inline descriptors and
 /// non-trivial fields, as existing pointers might need to reference those
 /// descriptors. Data is not copied since it cannot be legally read.
 using BlockMoveFn = void (*)(Block *Storage, char *SrcFieldPtr,
-                             char *DstFieldPtr, Descriptor *FieldDesc);
+                             char *DstFieldPtr, const Descriptor *FieldDesc);
 
 /// Inline descriptor embedded in structures and arrays.
 ///
diff --git a/clang/lib/AST/ItaniumMangle.cpp b/clang/lib/AST/ItaniumMangle.cpp
index fccbb09e426df..53371394730a2 100644
--- a/clang/lib/AST/ItaniumMangle.cpp
+++ b/clang/lib/AST/ItaniumMangle.cpp
@@ -3217,6 +3217,12 @@ void CXXNameMangler::mangleType(const BuiltinType *T) {
     Out << (type_name == InternalName ? "u" : "") << type_name.size()          \
         << type_name;                                                          \
     break;
+#define SVE_OPAQUE_TYPE(InternalName, MangledName, Id, SingletonId)            \
+  case BuiltinType::Id:                                                        \
+    type_name = MangledName;                                                   \
+    Out << (type_name == InternalName ? "u" : "") << type_name.size()          \
+        << type_name;                                                          \
+    break;
 #include "clang/Basic/AArch64SVEACLETypes.def"
 #define PPC_VECTOR_TYPE(Name, Id, Size) \
   case BuiltinType::Id: \
diff --git a/clang/lib/AST/TemplateName.cpp b/clang/lib/AST/TemplateName.cpp
index a6dd0fad93314..2f0e4181e9408 100644
--- a/clang/lib/AST/TemplateName.cpp
+++ b/clang/lib/AST/TemplateName.cpp
@@ -281,6 +281,15 @@ bool TemplateName::containsUnexpandedParameterPack() const {
   return getDependence() & TemplateNameDependence::UnexpandedPack;
 }
 
+void TemplateName::Profile(llvm::FoldingSetNodeID &ID) {
+  if (const auto* USD = getAsUsingShadowDecl())
+    ID.AddPointer(USD->getCanonicalDecl());
+  else if (const auto *TD = getAsTemplateDecl())
+    ID.AddPointer(TD->getCanonicalDecl());
+  else
+    ID.AddPointer(Storage.getOpaqueValue());
+}
+
 void TemplateName::print(raw_ostream &OS, const PrintingPolicy &Policy,
                          Qualified Qual) const {
   auto Kind = getKind();
diff --git a/clang/lib/AST/Type.cpp b/clang/lib/AST/Type.cpp
index 7710adbb274ae..464939427a770 100644
--- a/clang/lib/AST/Type.cpp
+++ b/clang/lib/AST/Type.cpp
@@ -2154,8 +2154,9 @@ bool Type::isFloatingType() const {
 bool Type::hasFloatingRepresentation() const {
   if (const auto *VT = dyn_cast<VectorType>(CanonicalType))
     return VT->getElementType()->isFloatingType();
-  else
-    return isFloatingType();
+  if (const auto *MT = dyn_cast<MatrixType>(CanonicalType))
+    return MT->getElementType()->isFloatingType();
+  return isFloatingType();
 }
 
 bool Type::isRealFloatingType() const {
@@ -2401,6 +2402,8 @@ bool Type::isVLSTBuiltinType() const {
     case BuiltinType::SveFloat64:
     case BuiltinType::SveBFloat16:
     case BuiltinType::SveBool:
+    case BuiltinType::SveBoolx2:
+    case BuiltinType::SveBoolx4:
       return true;
     default:
       return false;
@@ -3671,6 +3674,10 @@ bool AttributedType::isMSTypeSpec() const {
   llvm_unreachable("invalid attr kind");
 }
 
+bool AttributedType::isWebAssemblyFuncrefSpec() const {
+  return getAttrKind() == attr::WebAssemblyFuncref;
+}
+
 bool AttributedType::isCallingConv() const {
   // FIXME: Generate this with TableGen.
   switch (getAttrKind()) {
diff --git a/clang/lib/AST/TypePrinter.cpp b/clang/lib/AST/TypePrinter.cpp
index 4bd45768395a0..02a354da56a42 100644
--- a/clang/lib/AST/TypePrinter.cpp
+++ b/clang/lib/AST/TypePrinter.cpp
@@ -1651,6 +1651,9 @@ void TypePrinter::printAttributedBefore(const AttributedType *T,
     spaceBeforePlaceHolder(OS);
   }
 
+  if (T->isWebAssemblyFuncrefSpec())
+    OS << "__funcref";
+
   // Print nullability type specifiers.
   if (T->getImmediateNullability()) {
     if (T->getAttrKind() == attr::TypeNonNull)
@@ -1684,8 +1687,8 @@ void TypePrinter::printAttributedAfter(const AttributedType *T,
 
   // Some attributes are printed as qualifiers before the type, so we have
   // nothing left to do.
-  if (T->getAttrKind() == attr::ObjCKindOf ||
-      T->isMSTypeSpec() || T->getImmediateNullability())
+  if (T->getAttrKind() == attr::ObjCKindOf || T->isMSTypeSpec() ||
+      T->getImmediateNullability() || T->isWebAssemblyFuncrefSpec())
     return;
 
   // Don't print the inert __unsafe_unretained attribute at all.
@@ -1757,6 +1760,7 @@ void TypePrinter::printAttributedAfter(const AttributedType *T,
   case attr::AddressSpace:
   case attr::CmseNSCall:
   case attr::AnnotateType:
+  case attr::WebAssemblyFuncref:
     llvm_unreachable("This attribute should have been handled already");
 
   case attr::NSReturnsRetained:
@@ -2259,6 +2263,8 @@ std::string Qualifiers::getAddrSpaceAsString(LangAS AS) {
     return "__uptr __ptr32";
   case LangAS::ptr64:
     return "__ptr64";
+  case LangAS::wasm_funcref:
+    return "__funcref";
   case LangAS::hlsl_groupshared:
     return "groupshared";
   default:
diff --git a/clang/lib/ASTMatchers/Dynamic/Registry.cpp b/clang/lib/ASTMatchers/Dynamic/Registry.cpp
index 9bbb291df0639..37e1780e68a60 100644
--- a/clang/lib/ASTMatchers/Dynamic/Registry.cpp
+++ b/clang/lib/ASTMatchers/Dynamic/Registry.cpp
@@ -60,7 +60,7 @@ class RegistryMaps {
 
 void RegistryMaps::registerMatcher(
     StringRef MatcherName, std::unique_ptr<MatcherDescriptor> Callback) {
-  assert(Constructors.find(MatcherName) == Constructors.end());
+  assert(!Constructors.contains(MatcherName));
   Constructors[MatcherName] = std::move(Callback);
 }
 
diff --git a/clang/lib/Analysis/CFG.cpp b/clang/lib/Analysis/CFG.cpp
index ea8b73e81ea24..7e93247ba83fc 100644
--- a/clang/lib/Analysis/CFG.cpp
+++ b/clang/lib/Analysis/CFG.cpp
@@ -3387,8 +3387,7 @@ CFGBlock *CFGBuilder::VisitLabelStmt(LabelStmt *L) {
   if (!LabelBlock)              // This can happen when the body is empty, i.e.
     LabelBlock = createBlock(); // scopes that only contains NullStmts.
 
-  assert(LabelMap.find(L->getDecl()) == LabelMap.end() &&
-         "label already in map");
+  assert(!LabelMap.contains(L->getDecl()) && "label already in map");
   LabelMap[L->getDecl()] = JumpTarget(LabelBlock, ScopePos);
 
   // Labels partition blocks, so this is the end of the basic block we were
diff --git a/clang/lib/Analysis/ExprMutationAnalyzer.cpp b/clang/lib/Analysis/ExprMutationAnalyzer.cpp
index c876eaa6358a4..2f2997b82302a 100644
--- a/clang/lib/Analysis/ExprMutationAnalyzer.cpp
+++ b/clang/lib/Analysis/ExprMutationAnalyzer.cpp
@@ -605,7 +605,7 @@ FunctionParmMutationAnalyzer::FunctionParmMutationAnalyzer(
     for (const CXXCtorInitializer *Init : Ctor->inits()) {
       ExprMutationAnalyzer InitAnalyzer(*Init->getInit(), Context);
       for (const ParmVarDecl *Parm : Ctor->parameters()) {
-        if (Results.find(Parm) != Results.end())
+        if (Results.contains(Parm))
           continue;
         if (const Stmt *S = InitAnalyzer.findMutation(Parm))
           Results[Parm] = S;
diff --git a/clang/lib/Analysis/FlowSensitive/DataflowAnalysisContext.cpp b/clang/lib/Analysis/FlowSensitive/DataflowAnalysisContext.cpp
index 480606bdac8d8..44053246bb744 100644
--- a/clang/lib/Analysis/FlowSensitive/DataflowAnalysisContext.cpp
+++ b/clang/lib/Analysis/FlowSensitive/DataflowAnalysisContext.cpp
@@ -316,10 +316,9 @@ BoolValue &DataflowAnalysisContext::substituteBoolValue(
 BoolValue &DataflowAnalysisContext::buildAndSubstituteFlowCondition(
     AtomicBoolValue &Token,
     llvm::DenseMap<AtomicBoolValue *, BoolValue *> Substitutions) {
-  assert(
-      Substitutions.find(&getBoolLiteralValue(true)) == Substitutions.end() &&
-      Substitutions.find(&getBoolLiteralValue(false)) == Substitutions.end() &&
-      "Do not substitute true/false boolean literals");
+  assert(!Substitutions.contains(&getBoolLiteralValue(true)) &&
+         !Substitutions.contains(&getBoolLiteralValue(false)) &&
+         "Do not substitute true/false boolean literals");
   llvm::DenseMap<BoolValue *, BoolValue *> SubstitutionsCache(
       Substitutions.begin(), Substitutions.end());
   return buildAndSubstituteFlowConditionWithCache(Token, SubstitutionsCache);
diff --git a/clang/lib/Analysis/FlowSensitive/DataflowEnvironment.cpp b/clang/lib/Analysis/FlowSensitive/DataflowEnvironment.cpp
index 5d502fee30daf..46fb7bd2fd5e9 100644
--- a/clang/lib/Analysis/FlowSensitive/DataflowEnvironment.cpp
+++ b/clang/lib/Analysis/FlowSensitive/DataflowEnvironment.cpp
@@ -583,7 +583,7 @@ StorageLocation &Environment::createStorageLocation(const Expr &E) {
 }
 
 void Environment::setStorageLocation(const ValueDecl &D, StorageLocation &Loc) {
-  assert(DeclToLoc.find(&D) == DeclToLoc.end());
+  assert(!DeclToLoc.contains(&D));
   DeclToLoc[&D] = &Loc;
 }
 
@@ -595,7 +595,7 @@ StorageLocation *Environment::getStorageLocation(const ValueDecl &D,
 
 void Environment::setStorageLocation(const Expr &E, StorageLocation &Loc) {
   const Expr &CanonE = ignoreCFGOmittedNodes(E);
-  assert(ExprToLoc.find(&CanonE) == ExprToLoc.end());
+  assert(!ExprToLoc.contains(&CanonE));
   ExprToLoc[&CanonE] = &Loc;
 }
 
diff --git a/clang/lib/Analysis/ReachableCode.cpp b/clang/lib/Analysis/ReachableCode.cpp
index 5cc63bb17b09e..1bf0d9aec8620 100644
--- a/clang/lib/Analysis/ReachableCode.cpp
+++ b/clang/lib/Analysis/ReachableCode.cpp
@@ -12,6 +12,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "clang/Analysis/Analyses/ReachableCode.h"
+#include "clang/AST/Attr.h"
 #include "clang/AST/Expr.h"
 #include "clang/AST/ExprCXX.h"
 #include "clang/AST/ExprObjC.h"
@@ -629,6 +630,10 @@ void DeadCodeScan::reportDeadCode(const CFGBlock *B,
     UK = reachable_code::UK_Return;
   }
 
+  const auto *AS = dyn_cast<AttributedStmt>(S);
+  bool HasFallThroughAttr =
+      AS && hasSpecificAttr<FallThroughAttr>(AS->getAttrs());
+
   SourceRange SilenceableCondVal;
 
   if (UK == reachable_code::UK_Other) {
@@ -645,8 +650,9 @@ void DeadCodeScan::reportDeadCode(const CFGBlock *B,
         R2 = Inc->getSourceRange();
       }
 
-      CB.HandleUnreachable(reachable_code::UK_Loop_Increment,
-                           Loc, SourceRange(), SourceRange(Loc, Loc), R2);
+      CB.HandleUnreachable(reachable_code::UK_Loop_Increment, Loc,
+                           SourceRange(), SourceRange(Loc, Loc), R2,
+                           HasFallThroughAttr);
       return;
     }
 
@@ -665,7 +671,7 @@ void DeadCodeScan::reportDeadCode(const CFGBlock *B,
 
   SourceRange R1, R2;
   SourceLocation Loc = GetUnreachableLoc(S, R1, R2);
-  CB.HandleUnreachable(UK, Loc, SilenceableCondVal, R1, R2);
+  CB.HandleUnreachable(UK, Loc, SilenceableCondVal, R1, R2, HasFallThroughAttr);
 }
 
 //===----------------------------------------------------------------------===//
diff --git a/clang/lib/Analysis/ThreadSafety.cpp b/clang/lib/Analysis/ThreadSafety.cpp
index 899c6018895ef..9041000b0ff7a 100644
--- a/clang/lib/Analysis/ThreadSafety.cpp
+++ b/clang/lib/Analysis/ThreadSafety.cpp
@@ -1163,7 +1163,7 @@ void BeforeSet::checkBeforeAfter(const ValueDecl* StartVd,
       }
       // Transitively search other before sets, and warn on cycles.
       if (traverse(Vdb)) {
-        if (CycMap.find(Vd) == CycMap.end()) {
+        if (!CycMap.contains(Vd)) {
           CycMap.insert(std::make_pair(Vd, true));
           StringRef L1 = Vd->getName();
           Analyzer.Handler.handleBeforeAfterCycle(L1, Vd->getLocation());
diff --git a/clang/lib/Analysis/UnsafeBufferUsage.cpp b/clang/lib/Analysis/UnsafeBufferUsage.cpp
index d9602a41af787..116bb4954b168 100644
--- a/clang/lib/Analysis/UnsafeBufferUsage.cpp
+++ b/clang/lib/Analysis/UnsafeBufferUsage.cpp
@@ -9,8 +9,8 @@
 #include "clang/Analysis/Analyses/UnsafeBufferUsage.h"
 #include "clang/AST/RecursiveASTVisitor.h"
 #include "clang/ASTMatchers/ASTMatchFinder.h"
-#include "clang/Lex/Preprocessor.h"
 #include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
 #include "llvm/ADT/SmallVector.h"
 #include <memory>
 #include <optional>
@@ -113,13 +113,14 @@ class MatchDescendantVisitor
 
 AST_MATCHER_P(Stmt, forEveryDescendant, internal::Matcher<Stmt>, innerMatcher) {
   const DynTypedMatcher &DTM = static_cast<DynTypedMatcher>(innerMatcher);
-  
-  MatchDescendantVisitor Visitor(&DTM, Finder, Builder, ASTMatchFinder::BK_All);  
+
+  MatchDescendantVisitor Visitor(&DTM, Finder, Builder, ASTMatchFinder::BK_All);
   return Visitor.findMatch(DynTypedNode::create(Node));
 }
 
 // Matches a `Stmt` node iff the node is in a safe-buffer opt-out region
-AST_MATCHER_P(Stmt, notInSafeBufferOptOut, const UnsafeBufferUsageHandler *, Handler) {
+AST_MATCHER_P(Stmt, notInSafeBufferOptOut, const UnsafeBufferUsageHandler *,
+              Handler) {
   return !Handler->isSafeBufferOptOut(Node.getBeginLoc());
 }
 
@@ -130,7 +131,7 @@ AST_MATCHER_P(CastExpr, castSubExpr, internal::Matcher<Expr>, innerMatcher) {
 // Returns a matcher that matches any expression 'e' such that `innerMatcher`
 // matches 'e' and 'e' is in an Unspecified Lvalue Context.
 static auto isInUnspecifiedLvalueContext(internal::Matcher<Expr> innerMatcher) {
-// clang-format off
+  // clang-format off
   return
     expr(anyOf(
       implicitCastExpr(
@@ -331,7 +332,7 @@ class ArraySubscriptGadget : public WarningGadget {
               anyOf(hasPointerType(), hasArrayType()))),
             unless(hasIndex(integerLiteral(equals(0)))))
             .bind(ArraySubscrTag));
-      // clang-format on
+    // clang-format on
   }
 
   const ArraySubscriptExpr *getBaseStmt() const override { return ASE; }
@@ -354,10 +355,10 @@ class PointerArithmeticGadget : public WarningGadget {
   static constexpr const char *const PointerArithmeticTag = "ptrAdd";
   static constexpr const char *const PointerArithmeticPointerTag = "ptrAddPtr";
   const BinaryOperator *PA; // pointer arithmetic expression
-  const Expr * Ptr;         // the pointer expression in `PA`
+  const Expr *Ptr;          // the pointer expression in `PA`
 
 public:
-    PointerArithmeticGadget(const MatchFinder::MatchResult &Result)
+  PointerArithmeticGadget(const MatchFinder::MatchResult &Result)
       : WarningGadget(Kind::PointerArithmetic),
         PA(Result.Nodes.getNodeAs<BinaryOperator>(PointerArithmeticTag)),
         Ptr(Result.Nodes.getNodeAs<Expr>(PointerArithmeticPointerTag)) {}
@@ -367,43 +368,42 @@ class PointerArithmeticGadget : public WarningGadget {
   }
 
   static Matcher matcher() {
-    auto HasIntegerType = anyOf(
-          hasType(isInteger()), hasType(enumType()));
-    auto PtrAtRight = allOf(hasOperatorName("+"),
-                            hasRHS(expr(hasPointerType()).bind(PointerArithmeticPointerTag)),
-                            hasLHS(HasIntegerType));
-    auto PtrAtLeft = allOf(
-           anyOf(hasOperatorName("+"), hasOperatorName("-"),
-                 hasOperatorName("+="), hasOperatorName("-=")),
-           hasLHS(expr(hasPointerType()).bind(PointerArithmeticPointerTag)),
-           hasRHS(HasIntegerType));
-
-    return stmt(binaryOperator(anyOf(PtrAtLeft, PtrAtRight)).bind(PointerArithmeticTag));
+    auto HasIntegerType = anyOf(hasType(isInteger()), hasType(enumType()));
+    auto PtrAtRight =
+        allOf(hasOperatorName("+"),
+              hasRHS(expr(hasPointerType()).bind(PointerArithmeticPointerTag)),
+              hasLHS(HasIntegerType));
+    auto PtrAtLeft =
+        allOf(anyOf(hasOperatorName("+"), hasOperatorName("-"),
+                    hasOperatorName("+="), hasOperatorName("-=")),
+              hasLHS(expr(hasPointerType()).bind(PointerArithmeticPointerTag)),
+              hasRHS(HasIntegerType));
+
+    return stmt(binaryOperator(anyOf(PtrAtLeft, PtrAtRight))
+                    .bind(PointerArithmeticTag));
   }
 
   const Stmt *getBaseStmt() const override { return PA; }
 
   DeclUseList getClaimedVarUseSites() const override {
-    if (const auto *DRE =
-            dyn_cast<DeclRefExpr>(Ptr->IgnoreParenImpCasts())) {
+    if (const auto *DRE = dyn_cast<DeclRefExpr>(Ptr->IgnoreParenImpCasts())) {
       return {DRE};
     }
 
     return {};
   }
   // FIXME: pointer adding zero should be fine
-  //FIXME: this gadge will need a fix-it
+  // FIXME: this gadge will need a fix-it
 };
 
-
 /// A call of a function or method that performs unchecked buffer operations
 /// over one of its pointer parameters.
 class UnsafeBufferUsageAttrGadget : public WarningGadget {
-    constexpr static const char *const OpTag = "call_expr";
-    const CallExpr *Op;
+  constexpr static const char *const OpTag = "call_expr";
+  const CallExpr *Op;
 
 public:
-    UnsafeBufferUsageAttrGadget(const MatchFinder::MatchResult &Result)
+  UnsafeBufferUsageAttrGadget(const MatchFinder::MatchResult &Result)
       : WarningGadget(Kind::UnsafeBufferUsageAttr),
         Op(Result.Nodes.getNodeAs<CallExpr>(OpTag)) {}
 
@@ -413,22 +413,20 @@ class UnsafeBufferUsageAttrGadget : public WarningGadget {
 
   static Matcher matcher() {
     return stmt(callExpr(callee(functionDecl(hasAttr(attr::UnsafeBufferUsage))))
-                          .bind(OpTag));
+                    .bind(OpTag));
   }
   const Stmt *getBaseStmt() const override { return Op; }
 
-  DeclUseList getClaimedVarUseSites() const override {
-    return {};
-  }
+  DeclUseList getClaimedVarUseSites() const override { return {}; }
 };
-  
 
 // Represents expressions of the form `DRE[*]` in the Unspecified Lvalue
 // Context (see `isInUnspecifiedLvalueContext`).
 // Note here `[]` is the built-in subscript operator.
 class ULCArraySubscriptGadget : public FixableGadget {
 private:
-  static constexpr const char *const ULCArraySubscriptTag = "ArraySubscriptUnderULC";
+  static constexpr const char *const ULCArraySubscriptTag =
+      "ArraySubscriptUnderULC";
   const ArraySubscriptExpr *Node;
 
 public:
@@ -457,7 +455,8 @@ class ULCArraySubscriptGadget : public FixableGadget {
   virtual const Stmt *getBaseStmt() const override { return Node; }
 
   virtual DeclUseList getClaimedVarUseSites() const override {
-    if (const auto *DRE = dyn_cast<DeclRefExpr>(Node->getBase()->IgnoreImpCasts())) {
+    if (const auto *DRE =
+            dyn_cast<DeclRefExpr>(Node->getBase()->IgnoreImpCasts())) {
       return {DRE};
     }
     return {};
@@ -530,11 +529,11 @@ namespace {
 class Strategy {
 public:
   enum class Kind {
-    Wontfix,    // We don't plan to emit a fixit for this variable.
-    Span,       // We recommend replacing the variable with std::span.
-    Iterator,   // We recommend replacing the variable with std::span::iterator.
-    Array,      // We recommend replacing the variable with std::array.
-    Vector      // We recommend replacing the variable with std::vector.
+    Wontfix,  // We don't plan to emit a fixit for this variable.
+    Span,     // We recommend replacing the variable with std::span.
+    Iterator, // We recommend replacing the variable with std::span::iterator.
+    Array,    // We recommend replacing the variable with std::array.
+    Vector    // We recommend replacing the variable with std::vector.
   };
 
 private:
@@ -547,9 +546,7 @@ class Strategy {
   Strategy(const Strategy &) = delete; // Let's avoid copies.
   Strategy(Strategy &&) = default;
 
-  void set(const VarDecl *VD, Kind K) {
-    Map[VD] = K;
-  }
+  void set(const VarDecl *VD, Kind K) { Map[VD] = K; }
 
   Kind lookup(const VarDecl *VD) const {
     auto I = Map.find(VD);
@@ -593,17 +590,17 @@ findGadgets(const Decl *D, const UnsafeBufferUsageHandler &Handler) {
       // Figure out which matcher we've found, and call the appropriate
       // subclass constructor.
       // FIXME: Can we do this more logarithmically?
-#define FIXABLE_GADGET(name)                                                           \
-      if (Result.Nodes.getNodeAs<Stmt>(#name)) {                               \
-        FixableGadgets.push_back(std::make_unique<name ## Gadget>(Result));           \
-        NEXT;                                                                  \
-      }
+#define FIXABLE_GADGET(name)                                                   \
+  if (Result.Nodes.getNodeAs<Stmt>(#name)) {                                   \
+    FixableGadgets.push_back(std::make_unique<name##Gadget>(Result));          \
+    NEXT;                                                                      \
+  }
 #include "clang/Analysis/Analyses/UnsafeBufferUsageGadgets.def"
-#define WARNING_GADGET(name)                                                           \
-      if (Result.Nodes.getNodeAs<Stmt>(#name)) {                               \
-        WarningGadgets.push_back(std::make_unique<name ## Gadget>(Result));           \
-        NEXT;                                                                  \
-      }
+#define WARNING_GADGET(name)                                                   \
+  if (Result.Nodes.getNodeAs<Stmt>(#name)) {                                   \
+    WarningGadgets.push_back(std::make_unique<name##Gadget>(Result));          \
+    NEXT;                                                                      \
+  }
 #include "clang/Analysis/Analyses/UnsafeBufferUsageGadgets.def"
 
       assert(numFound >= 1 && "Gadgets not found in match result!");
@@ -657,11 +654,24 @@ findGadgets(const Decl *D, const UnsafeBufferUsageHandler &Handler) {
     }
   }
 
-  return {std::move(CB.FixableGadgets), std::move(CB.WarningGadgets), std::move(CB.Tracker)};
+  return {std::move(CB.FixableGadgets), std::move(CB.WarningGadgets),
+          std::move(CB.Tracker)};
 }
 
+// Compares AST nodes by source locations.
+template <typename NodeTy> struct CompareNode {
+  bool operator()(const NodeTy *N1, const NodeTy *N2) const {
+    return N1->getBeginLoc().getRawEncoding() <
+           N2->getBeginLoc().getRawEncoding();
+  }
+};
+
 struct WarningGadgetSets {
-  std::map<const VarDecl *, std::set<std::unique_ptr<WarningGadget>>> byVar;
+  std::map<const VarDecl *, std::set<std::unique_ptr<WarningGadget>>,
+           // To keep keys sorted by their locations in the map so that the
+           // order is deterministic:
+           CompareNode<VarDecl>>
+      byVar;
   // These Gadgets are not related to pointer variables (e. g. temporaries).
   llvm::SmallVector<std::unique_ptr<WarningGadget>, 16> noVar;
 };
@@ -709,8 +719,8 @@ groupFixablesByVar(FixableGadgetList &&AllFixableOperations) {
   return FixablesForUnsafeVars;
 }
 
-bool clang::internal::anyConflict(
-    const SmallVectorImpl<FixItHint> &FixIts, const SourceManager &SM) {
+bool clang::internal::anyConflict(const SmallVectorImpl<FixItHint> &FixIts,
+                                  const SourceManager &SM) {
   // A simple interval overlap detection algorithm.  Sorts all ranges by their
   // begin location then finds the first overlap in one pass.
   std::vector<const FixItHint *> All; // a copy of `FixIts`
@@ -742,7 +752,8 @@ bool clang::internal::anyConflict(
 
 std::optional<FixItList>
 ULCArraySubscriptGadget::getFixits(const Strategy &S) const {
-  if (const auto *DRE = dyn_cast<DeclRefExpr>(Node->getBase()->IgnoreImpCasts()))
+  if (const auto *DRE =
+          dyn_cast<DeclRefExpr>(Node->getBase()->IgnoreImpCasts()))
     if (const auto *VD = dyn_cast<VarDecl>(DRE->getDecl())) {
       switch (S.lookup(VD)) {
       case Strategy::Kind::Span: {
@@ -793,7 +804,7 @@ static SourceLocation getPastLoc(const NodeTy *Node, const SourceManager &SM,
 
 // Return text representation of an `Expr`.
 static StringRef getExprText(const Expr *E, const SourceManager &SM,
-                               const LangOptions &LangOpts) {
+                             const LangOptions &LangOpts) {
   SourceLocation LastCharLoc = getPastLoc(E, SM, LangOpts);
 
   return Lexer::getSourceText(
@@ -850,8 +861,8 @@ populateInitializerFixItWithSpan(const Expr *Init, const ASTContext &Ctx,
       if (AddrOfExpr->getOpcode() == UnaryOperatorKind::UO_AddrOf &&
           isa_and_present<DeclRefExpr>(AddrOfExpr->getSubExpr()))
         ExtentText = One;
-    // TODO: we can handle more cases, e.g., `&a[0]`, `&a`, `std::addressof`, and explicit casting, etc.
-    // etc.
+    // TODO: we can handle more cases, e.g., `&a[0]`, `&a`, `std::addressof`,
+    // and explicit casting, etc. etc.
   }
 
   SmallString<32> StrBuffer{};
@@ -918,7 +929,7 @@ static FixItList fixVariableWithSpan(const VarDecl *VD,
   assert(DS && "Fixing non-local variables not implemented yet!");
   if (!DS->isSingleDecl()) {
     // FIXME: to support handling multiple `VarDecl`s in a single `DeclStmt`
-    return{};
+    return {};
   }
   // Currently DS is an unused variable but we'll need it when
   // non-single decls are implemented, where the pointee type name
@@ -969,7 +980,8 @@ getFixIts(FixableGadgetSets &FixablesForUnsafeVars, const Strategy &S,
           UnsafeBufferUsageHandler &Handler) {
   std::map<const VarDecl *, FixItList> FixItsForVariable;
   for (const auto &[VD, Fixables] : FixablesForUnsafeVars.byVar) {
-    FixItsForVariable[VD] = fixVariable(VD, S.lookup(VD), Tracker, Ctx, Handler);
+    FixItsForVariable[VD] =
+        fixVariable(VD, S.lookup(VD), Tracker, Ctx, Handler);
     // If we fail to produce Fix-It for the declaration we have to skip the
     // variable entirely.
     if (FixItsForVariable[VD].empty()) {
@@ -999,7 +1011,7 @@ getFixIts(FixableGadgetSets &FixablesForUnsafeVars, const Strategy &S,
     // a fix-it conflicts with another one
     if (overlapWithMacro(FixItsForVariable[VD]) ||
         clang::internal::anyConflict(FixItsForVariable[VD],
-                                     Ctx.getSourceManager())) {      
+                                     Ctx.getSourceManager())) {
       FixItsForVariable.erase(VD);
     }
   }
diff --git a/clang/lib/Basic/Attributes.cpp b/clang/lib/Basic/Attributes.cpp
index a961e68f4ac1c..da339d5b1bab6 100644
--- a/clang/lib/Basic/Attributes.cpp
+++ b/clang/lib/Basic/Attributes.cpp
@@ -2,8 +2,18 @@
 #include "clang/Basic/AttrSubjectMatchRules.h"
 #include "clang/Basic/AttributeCommonInfo.h"
 #include "clang/Basic/IdentifierTable.h"
+#include "clang/Basic/ParsedAttrInfo.h"
 using namespace clang;
 
+static int hasAttributeImpl(AttributeCommonInfo::Syntax Syntax, StringRef Name,
+                            StringRef ScopeName, const TargetInfo &Target,
+                            const LangOptions &LangOpts) {
+
+#include "clang/Basic/AttrHasAttributeImpl.inc"
+
+  return 0;
+}
+
 int clang::hasAttribute(AttributeCommonInfo::Syntax Syntax,
                         const IdentifierInfo *Scope, const IdentifierInfo *Attr,
                         const TargetInfo &Target, const LangOptions &LangOpts) {
@@ -27,7 +37,14 @@ int clang::hasAttribute(AttributeCommonInfo::Syntax Syntax,
       ScopeName == "omp")
     return (Name == "directive" || Name == "sequence") ? 1 : 0;
 
-#include "clang/Basic/AttrHasAttributeImpl.inc"
+  int res = hasAttributeImpl(Syntax, Name, ScopeName, Target, LangOpts);
+  if (res)
+    return res;
+
+  // Check if any plugin provides this attribute.
+  for (auto &Ptr : getAttributePluginInstances())
+    if (Ptr->hasSpelling(Syntax, Name))
+      return 1;
 
   return 0;
 }
diff --git a/clang/lib/Basic/CMakeLists.txt b/clang/lib/Basic/CMakeLists.txt
index 32af83ee23f31..8c828e4c73225 100644
--- a/clang/lib/Basic/CMakeLists.txt
+++ b/clang/lib/Basic/CMakeLists.txt
@@ -61,6 +61,7 @@ add_clang_library(clangBasic
   OpenCLOptions.cpp
   OpenMPKinds.cpp
   OperatorPrecedence.cpp
+  ParsedAttrInfo.cpp
   ProfileList.cpp
   NoSanitizeList.cpp
   SanitizerSpecialCaseList.cpp
diff --git a/clang/lib/Basic/OpenCLOptions.cpp b/clang/lib/Basic/OpenCLOptions.cpp
index 44edf54025405..d39686ea688e5 100644
--- a/clang/lib/Basic/OpenCLOptions.cpp
+++ b/clang/lib/Basic/OpenCLOptions.cpp
@@ -26,7 +26,7 @@ static const std::pair<StringRef, StringRef> FeatureExtensionMap[] = {
     {"cl_khr_3d_image_writes", "__opencl_c_3d_image_writes"}};
 
 bool OpenCLOptions::isKnown(llvm::StringRef Ext) const {
-  return OptMap.find(Ext) != OptMap.end();
+  return OptMap.contains(Ext);
 }
 
 bool OpenCLOptions::isAvailableOption(llvm::StringRef Ext,
diff --git a/clang/lib/Basic/ParsedAttrInfo.cpp b/clang/lib/Basic/ParsedAttrInfo.cpp
new file mode 100644
index 0000000000000..7757f30da511d
--- /dev/null
+++ b/clang/lib/Basic/ParsedAttrInfo.cpp
@@ -0,0 +1,32 @@
+//===- ParsedAttrInfo.cpp - Registry for attribute plugins ------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file contains the Registry of attributes added by plugins which
+// derive the ParsedAttrInfo class.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Basic/ParsedAttrInfo.h"
+#include "llvm/Support/ManagedStatic.h"
+#include <list>
+#include <memory>
+
+using namespace clang;
+
+LLVM_INSTANTIATE_REGISTRY(ParsedAttrInfoRegistry)
+
+const std::list<std::unique_ptr<ParsedAttrInfo>> &
+clang::getAttributePluginInstances() {
+  static llvm::ManagedStatic<std::list<std::unique_ptr<ParsedAttrInfo>>>
+      PluginAttrInstances;
+  if (PluginAttrInstances->empty())
+    for (auto It : ParsedAttrInfoRegistry::entries())
+      PluginAttrInstances->emplace_back(It.instantiate());
+
+  return *PluginAttrInstances;
+}
diff --git a/clang/lib/Basic/TargetInfo.cpp b/clang/lib/Basic/TargetInfo.cpp
index 07af6c07031bd..e168d45020e77 100644
--- a/clang/lib/Basic/TargetInfo.cpp
+++ b/clang/lib/Basic/TargetInfo.cpp
@@ -47,6 +47,7 @@ static const LangASMap FakeAddrSpaceMap = {
     11, // ptr32_uptr
     12, // ptr64
     13, // hlsl_groupshared
+    20, // wasm_funcref
 };
 
 // TargetInfo Constructor.
diff --git a/clang/lib/Basic/Targets/AArch64.cpp b/clang/lib/Basic/Targets/AArch64.cpp
index 9b67becff9e75..1711e16b46cdc 100644
--- a/clang/lib/Basic/Targets/AArch64.cpp
+++ b/clang/lib/Basic/Targets/AArch64.cpp
@@ -373,7 +373,8 @@ void AArch64TargetInfo::getTargetDefines(const LangOptions &Opts,
   Builder.defineMacro("__ARM_ALIGN_MAX_STACK_PWR", "4");
 
   // 0xe implies support for half, single and double precision operations.
-  Builder.defineMacro("__ARM_FP", "0xE");
+  if (FPU & FPUMode)
+    Builder.defineMacro("__ARM_FP", "0xE");
 
   // PCS specifies this for SysV variants, which is all we support. Other ABIs
   // may choose __ARM_FP16_FORMAT_ALTERNATIVE.
@@ -606,16 +607,18 @@ unsigned AArch64TargetInfo::multiVersionFeatureCost() const {
   return llvm::AArch64::ExtensionInfo::MaxFMVPriority;
 }
 
-bool AArch64TargetInfo::getFeatureDepOptions(StringRef Name,
-                                             std::string &FeatureVec) const {
-  FeatureVec = "";
-  for (const auto &E : llvm::AArch64::Extensions) {
-    if (Name == E.Name) {
-      FeatureVec = E.DependentFeatures;
-      break;
-    }
-  }
-  return FeatureVec != "";
+bool AArch64TargetInfo::doesFeatureAffectCodeGen(StringRef Name) const {
+  auto F = llvm::find_if(llvm::AArch64::Extensions, [&](const auto &E) {
+    return Name == E.Name && !E.DependentFeatures.empty();
+  });
+  return F != std::end(llvm::AArch64::Extensions);
+}
+
+StringRef AArch64TargetInfo::getFeatureDependencies(StringRef Name) const {
+  auto F = llvm::find_if(llvm::AArch64::Extensions,
+                         [&](const auto &E) { return Name == E.Name; });
+  return F != std::end(llvm::AArch64::Extensions) ? F->DependentFeatures
+                                                  : StringRef();
 }
 
 bool AArch64TargetInfo::validateCpuSupports(StringRef FeatureStr) const {
@@ -707,6 +710,8 @@ void AArch64TargetInfo::setFeatureEnabled(llvm::StringMap<bool> &Features,
 bool AArch64TargetInfo::handleTargetFeatures(std::vector<std::string> &Features,
                                              DiagnosticsEngine &Diags) {
   for (const auto &Feature : Features) {
+    if (Feature == "-fp-armv8")
+      HasNoFP = true;
     if (Feature == "-neon")
       HasNoNeon = true;
     if (Feature == "-sve")
@@ -922,6 +927,8 @@ bool AArch64TargetInfo::handleTargetFeatures(std::vector<std::string> &Features,
       HasMOPS = true;
     if (Feature == "+d128")
       HasD128 = true;
+    if (Feature == "+gcs")
+      HasGCS = true;
   }
 
   // Check features that are manually disabled by command line options.
@@ -935,6 +942,11 @@ bool AArch64TargetInfo::handleTargetFeatures(std::vector<std::string> &Features,
   setDataLayout();
   setArchFeatures();
 
+  if (HasNoFP) {
+    FPU &= ~FPUMode;
+    FPU &= ~NeonMode;
+    FPU &= ~SveMode;
+  }
   if (HasNoNeon) {
     FPU &= ~NeonMode;
     FPU &= ~SveMode;
@@ -962,18 +974,18 @@ bool AArch64TargetInfo::initFeatureMap(
   }
 
   // Process target and dependent features. This is done in two loops collecting
-  // them into UpdatedFeaturesVec: first to add dependent '+'features,
-  // second to add target '+/-'features that can later disable some of
-  // features added on the first loop.
+  // them into UpdatedFeaturesVec: first to add dependent '+'features, second to
+  // add target '+/-'features that can later disable some of features added on
+  // the first loop. Function Multi Versioning features begin with '?'.
   for (const auto &Feature : FeaturesVec)
-    if ((Feature[0] == '?' || Feature[0] == '+')) {
-      std::string Options;
-      if (AArch64TargetInfo::getFeatureDepOptions(Feature.substr(1), Options)) {
-        SmallVector<StringRef, 1> AttrFeatures;
-        StringRef(Options).split(AttrFeatures, ",");
-        for (auto F : AttrFeatures)
-          UpdatedFeaturesVec.push_back(F.str());
-      }
+    if (((Feature[0] == '?' || Feature[0] == '+')) &&
+        AArch64TargetInfo::doesFeatureAffectCodeGen(Feature.substr(1))) {
+      StringRef DepFeatures =
+          AArch64TargetInfo::getFeatureDependencies(Feature.substr(1));
+      SmallVector<StringRef, 1> AttrFeatures;
+      DepFeatures.split(AttrFeatures, ",");
+      for (auto F : AttrFeatures)
+        UpdatedFeaturesVec.push_back(F.str());
     }
   for (const auto &Feature : FeaturesVec)
     if (Feature[0] != '?') {
diff --git a/clang/lib/Basic/Targets/AArch64.h b/clang/lib/Basic/Targets/AArch64.h
index 46bfb70a0d953..c973efc21bde2 100644
--- a/clang/lib/Basic/Targets/AArch64.h
+++ b/clang/lib/Basic/Targets/AArch64.h
@@ -26,7 +26,11 @@ class LLVM_LIBRARY_VISIBILITY AArch64TargetInfo : public TargetInfo {
   static const TargetInfo::GCCRegAlias GCCRegAliases[];
   static const char *const GCCRegNames[];
 
-  enum FPUModeEnum { FPUMode, NeonMode = (1 << 0), SveMode = (1 << 1) };
+  enum FPUModeEnum {
+    FPUMode = (1 << 0),
+    NeonMode = (1 << 1),
+    SveMode = (1 << 2),
+  };
 
   unsigned FPU = FPUMode;
   bool HasCRC = false;
@@ -73,9 +77,11 @@ class LLVM_LIBRARY_VISIBILITY AArch64TargetInfo : public TargetInfo {
   bool HasWFxT = false;
   bool HasJSCVT = false;
   bool HasFCMA = false;
+  bool HasNoFP = false;
   bool HasNoNeon = false;
   bool HasNoSVE = false;
   bool HasFMV = true;
+  bool HasGCS = false;
 
   const llvm::AArch64::ArchInfo *ArchInfo = &llvm::AArch64::ARMV8A;
 
@@ -143,9 +149,8 @@ class LLVM_LIBRARY_VISIBILITY AArch64TargetInfo : public TargetInfo {
 
   std::optional<std::pair<unsigned, unsigned>>
   getVScaleRange(const LangOptions &LangOpts) const override;
-
-  bool getFeatureDepOptions(StringRef Feature,
-                            std::string &Options) const override;
+  bool doesFeatureAffectCodeGen(StringRef Name) const override;
+  StringRef getFeatureDependencies(StringRef Name) const override;
   bool validateCpuSupports(StringRef FeatureStr) const override;
   bool hasFeature(StringRef Feature) const override;
   void setFeatureEnabled(llvm::StringMap<bool> &Features, StringRef Name,
diff --git a/clang/lib/Basic/Targets/AMDGPU.h b/clang/lib/Basic/Targets/AMDGPU.h
index e315e6f033a21..ab8482b3df9a6 100644
--- a/clang/lib/Basic/Targets/AMDGPU.h
+++ b/clang/lib/Basic/Targets/AMDGPU.h
@@ -447,7 +447,7 @@ class LLVM_LIBRARY_VISIBILITY AMDGPUTargetInfo final : public TargetInfo {
       StringRef Name = StringRef(F).drop_front();
       if (!llvm::is_contained(TargetIDFeatures, Name))
         continue;
-      assert(OffloadArchFeatures.find(Name) == OffloadArchFeatures.end());
+      assert(!OffloadArchFeatures.contains(Name));
       OffloadArchFeatures[Name] = IsOn;
     }
     return true;
diff --git a/clang/lib/Basic/Targets/ARM.cpp b/clang/lib/Basic/Targets/ARM.cpp
index f11751a760731..e01379ec82fb4 100644
--- a/clang/lib/Basic/Targets/ARM.cpp
+++ b/clang/lib/Basic/Targets/ARM.cpp
@@ -254,6 +254,7 @@ ARMTargetInfo::ARMTargetInfo(const llvm::Triple &Triple,
                              const TargetOptions &Opts)
     : TargetInfo(Triple), FPMath(FP_Default), IsAAPCS(true), LDREX(0),
       HW_FP(0) {
+  bool IsFreeBSD = Triple.isOSFreeBSD();
   bool IsOpenBSD = Triple.isOSOpenBSD();
   bool IsNetBSD = Triple.isOSNetBSD();
 
@@ -321,7 +322,7 @@ ARMTargetInfo::ARMTargetInfo(const llvm::Triple &Triple,
     default:
       if (IsNetBSD)
         setABI("apcs-gnu");
-      else if (IsOpenBSD)
+      else if (IsFreeBSD || IsOpenBSD)
         setABI("aapcs-linux");
       else
         setABI("aapcs");
@@ -451,7 +452,7 @@ bool ARMTargetInfo::initFeatureMap(
   }
 
   // get default FPU features
-  unsigned FPUKind = llvm::ARM::getDefaultFPU(CPU, Arch);
+  llvm::ARM::FPUKind FPUKind = llvm::ARM::getDefaultFPU(CPU, Arch);
   llvm::ARM::getFPUFeatures(FPUKind, TargetFeatures);
 
   // get default Extension features
diff --git a/clang/lib/Basic/Targets/AVR.cpp b/clang/lib/Basic/Targets/AVR.cpp
index 3d662cc3ba74f..5c75bae2d8121 100644
--- a/clang/lib/Basic/Targets/AVR.cpp
+++ b/clang/lib/Basic/Targets/AVR.cpp
@@ -428,6 +428,23 @@ bool AVRTargetInfo::setCPU(const std::string &Name) {
   return false;
 }
 
+std::optional<std::string>
+AVRTargetInfo::handleAsmEscapedChar(char EscChar) const {
+  switch (EscChar) {
+  // "%~" represents for 'r' depends on the device has long jump/call.
+  case '~':
+    return ArchHasJMPCALL(Arch) ? std::string("") : std::string(1, 'r');
+
+  // "%!" represents for 'e' depends on the PC register size.
+  case '!':
+    return ArchHas3BytePC(Arch) ? std::string(1, 'e') : std::string("");
+
+  // This is an invalid escape character for AVR.
+  default:
+    return std::nullopt;
+  }
+}
+
 void AVRTargetInfo::getTargetDefines(const LangOptions &Opts,
                                      MacroBuilder &Builder) const {
   Builder.defineMacro("AVR");
diff --git a/clang/lib/Basic/Targets/AVR.h b/clang/lib/Basic/Targets/AVR.h
index 934a01cd62173..e5d683a27c6d5 100644
--- a/clang/lib/Basic/Targets/AVR.h
+++ b/clang/lib/Basic/Targets/AVR.h
@@ -170,6 +170,7 @@ class LLVM_LIBRARY_VISIBILITY AVRTargetInfo : public TargetInfo {
   bool isValidCPUName(StringRef Name) const override;
   void fillValidCPUList(SmallVectorImpl<StringRef> &Values) const override;
   bool setCPU(const std::string &Name) override;
+  std::optional<std::string> handleAsmEscapedChar(char EscChar) const override;
   StringRef getABI() const override { return ABI; }
 
 protected:
diff --git a/clang/lib/Basic/Targets/DirectX.h b/clang/lib/Basic/Targets/DirectX.h
index 2fab674d8e7bb..cb8fad2a29066 100644
--- a/clang/lib/Basic/Targets/DirectX.h
+++ b/clang/lib/Basic/Targets/DirectX.h
@@ -42,6 +42,9 @@ static const unsigned DirectXAddrSpaceMap[] = {
     0, // ptr32_uptr
     0, // ptr64
     3, // hlsl_groupshared
+    // Wasm address space values for this target are dummy values,
+    // as it is only enabled for Wasm targets.
+    20, // wasm_funcref
 };
 
 class LLVM_LIBRARY_VISIBILITY DirectXTargetInfo : public TargetInfo {
diff --git a/clang/lib/Basic/Targets/M68k.cpp b/clang/lib/Basic/Targets/M68k.cpp
index cbc7c79837792..c83c5389e3b9d 100644
--- a/clang/lib/Basic/Targets/M68k.cpp
+++ b/clang/lib/Basic/Targets/M68k.cpp
@@ -192,6 +192,12 @@ bool M68kTargetInfo::validateAsmConstraint(
       break;
     }
     break;
+  case 'Q': // address register indirect addressing
+  case 'U': // address register indirect w/ constant offset addressing
+    // TODO: Handle 'S' (basically 'm' when pc-rel is enforced) when
+    // '-mpcrel' flag is properly handled by the driver.
+    info.setAllowsMemory();
+    return true;
   default:
     break;
   }
diff --git a/clang/lib/Basic/Targets/NVPTX.h b/clang/lib/Basic/Targets/NVPTX.h
index 51de753665706..47f61f249ace1 100644
--- a/clang/lib/Basic/Targets/NVPTX.h
+++ b/clang/lib/Basic/Targets/NVPTX.h
@@ -45,6 +45,9 @@ static const unsigned NVPTXAddrSpaceMap[] = {
     0, // ptr32_uptr
     0, // ptr64
     0, // hlsl_groupshared
+    // Wasm address space values for this target are dummy values,
+    // as it is only enabled for Wasm targets.
+    20, // wasm_funcref
 };
 
 /// The DWARF address class. Taken from
diff --git a/clang/lib/Basic/Targets/OSTargets.cpp b/clang/lib/Basic/Targets/OSTargets.cpp
index 33a5b500e2d1c..763235fb10a52 100644
--- a/clang/lib/Basic/Targets/OSTargets.cpp
+++ b/clang/lib/Basic/Targets/OSTargets.cpp
@@ -108,9 +108,16 @@ void getDarwinDefines(MacroBuilder &Builder, const LangOptions &Opts,
     Builder.defineMacro("__ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__", Str);
   }
 
-  // Tell users about the kernel if there is one.
-  if (Triple.isOSDarwin())
+  if (Triple.isOSDarwin()) {
+    // Any darwin OS defines a general darwin OS version macro in addition
+    // to the other OS specific macros.
+    assert(OsVersion.getMinor().value_or(0) < 100 &&
+           OsVersion.getSubminor().value_or(0) < 100 && "Invalid version!");
+    Builder.defineMacro("__ENVIRONMENT_OS_VERSION_MIN_REQUIRED__", Str);
+
+    // Tell users about the kernel if there is one.
     Builder.defineMacro("__MACH__");
+  }
 
   PlatformMinVersion = OsVersion;
 }
diff --git a/clang/lib/Basic/Targets/SPIR.h b/clang/lib/Basic/Targets/SPIR.h
index 0a6ab9ac6b9aa..5913719858346 100644
--- a/clang/lib/Basic/Targets/SPIR.h
+++ b/clang/lib/Basic/Targets/SPIR.h
@@ -46,6 +46,9 @@ static const unsigned SPIRDefIsPrivMap[] = {
     0, // ptr32_uptr
     0, // ptr64
     0, // hlsl_groupshared
+    // Wasm address space values for this target are dummy values,
+    // as it is only enabled for Wasm targets.
+    20, // wasm_funcref
 };
 
 // Used by both the SPIR and SPIR-V targets.
@@ -76,6 +79,9 @@ static const unsigned SPIRDefIsGenMap[] = {
     0, // ptr32_uptr
     0, // ptr64
     0, // hlsl_groupshared
+    // Wasm address space values for this target are dummy values,
+    // as it is only enabled for Wasm targets.
+    20, // wasm_funcref
 };
 
 // Base class for SPIR and SPIR-V target info.
diff --git a/clang/lib/Basic/Targets/TCE.h b/clang/lib/Basic/Targets/TCE.h
index 5ff379a771ac1..202554a336ab5 100644
--- a/clang/lib/Basic/Targets/TCE.h
+++ b/clang/lib/Basic/Targets/TCE.h
@@ -51,6 +51,9 @@ static const unsigned TCEOpenCLAddrSpaceMap[] = {
     0, // ptr32_uptr
     0, // ptr64
     0, // hlsl_groupshared
+    // Wasm address space values for this target are dummy values,
+    // as it is only enabled for Wasm targets.
+    20, // wasm_funcref
 };
 
 class LLVM_LIBRARY_VISIBILITY TCETargetInfo : public TargetInfo {
diff --git a/clang/lib/Basic/Targets/WebAssembly.h b/clang/lib/Basic/Targets/WebAssembly.h
index aeebc840a8ecd..81e3d2b648419 100644
--- a/clang/lib/Basic/Targets/WebAssembly.h
+++ b/clang/lib/Basic/Targets/WebAssembly.h
@@ -21,6 +21,30 @@
 namespace clang {
 namespace targets {
 
+static const unsigned WebAssemblyAddrSpaceMap[] = {
+    0, // Default
+    0, // opencl_global
+    0, // opencl_local
+    0, // opencl_constant
+    0, // opencl_private
+    0, // opencl_generic
+    0, // opencl_global_device
+    0, // opencl_global_host
+    0, // cuda_device
+    0, // cuda_constant
+    0, // cuda_shared
+    0, // sycl_global
+    0, // sycl_global_device
+    0, // sycl_global_host
+    0, // sycl_local
+    0, // sycl_private
+    0, // ptr32_sptr
+    0, // ptr32_uptr
+    0, // ptr64
+    0, // hlsl_groupshared
+    20, // wasm_funcref
+};
+
 class LLVM_LIBRARY_VISIBILITY WebAssemblyTargetInfo : public TargetInfo {
 
   enum SIMDEnum {
@@ -45,6 +69,7 @@ class LLVM_LIBRARY_VISIBILITY WebAssemblyTargetInfo : public TargetInfo {
 public:
   explicit WebAssemblyTargetInfo(const llvm::Triple &T, const TargetOptions &)
       : TargetInfo(T) {
+    AddrSpaceMap = &WebAssemblyAddrSpaceMap;
     NoAsmVariants = true;
     SuitableAlign = 128;
     LargeArrayMinWidth = 128;
diff --git a/clang/lib/Basic/Targets/X86.h b/clang/lib/Basic/Targets/X86.h
index ef41195a5c096..50b9eb444b6d7 100644
--- a/clang/lib/Basic/Targets/X86.h
+++ b/clang/lib/Basic/Targets/X86.h
@@ -46,6 +46,9 @@ static const unsigned X86AddrSpaceMap[] = {
     271, // ptr32_uptr
     272, // ptr64
     0,   // hlsl_groupshared
+    // Wasm address space values for this target are dummy values,
+    // as it is only enabled for Wasm targets.
+    20, // wasm_funcref
 };
 
 // X86 target abstract base class; x86-32 and x86-64 are very close, so
diff --git a/clang/lib/CodeGen/BackendUtil.cpp b/clang/lib/CodeGen/BackendUtil.cpp
index b03d326dd0654..17d0d8d653ac6 100644
--- a/clang/lib/CodeGen/BackendUtil.cpp
+++ b/clang/lib/CodeGen/BackendUtil.cpp
@@ -367,8 +367,6 @@ static bool initTargetOptions(DiagnosticsEngine &Diags,
   Options.BinutilsVersion =
       llvm::TargetMachine::parseBinutilsVersion(CodeGenOpts.BinutilsVersion);
   Options.UseInitArray = CodeGenOpts.UseInitArray;
-  Options.LowerGlobalDtorsViaCxaAtExit =
-      CodeGenOpts.RegisterGlobalDtorsWithAtExit;
   Options.DisableIntegratedAS = CodeGenOpts.DisableIntegratedAS;
   Options.CompressDebugSections = CodeGenOpts.getCompressDebugSections();
   Options.RelaxELFRelocations = CodeGenOpts.RelaxELFRelocations;
@@ -839,7 +837,7 @@ void EmitAssemblyHelper::RunOptimizationPipeline(
       TheModule->getContext(),
       (CodeGenOpts.DebugPassManager || DebugPassStructure),
       /*VerifyEach*/ false, PrintPassOpts);
-  SI.registerCallbacks(PIC, &FAM);
+  SI.registerCallbacks(PIC, &MAM);
   PassBuilder PB(TM.get(), PTO, PGOOpt, &PIC);
 
   if (CodeGenOpts.EnableAssignmentTracking) {
@@ -859,7 +857,7 @@ void EmitAssemblyHelper::RunOptimizationPipeline(
     if (!CodeGenOpts.DIBugsReportFilePath.empty())
       Debugify.setOrigDIVerifyBugsReportFilePath(
           CodeGenOpts.DIBugsReportFilePath);
-    Debugify.registerCallbacks(PIC);
+    Debugify.registerCallbacks(PIC, MAM);
   }
   // Attempt to load pass plugins and register their callbacks with PB.
   for (auto &PluginFN : CodeGenOpts.PassPlugins) {
@@ -975,9 +973,7 @@ void EmitAssemblyHelper::RunOptimizationPipeline(
             MPM.addPass(InstrProfiling(*Options, false));
           });
 
-    if (CodeGenOpts.OptimizationLevel == 0) {
-      MPM = PB.buildO0DefaultPipeline(Level, IsLTO || IsThinLTO);
-    } else if (IsThinLTO) {
+    if (IsThinLTO) {
       MPM = PB.buildThinLTOPreLinkDefaultPipeline(Level);
     } else if (IsLTO) {
       MPM = PB.buildLTOPreLinkDefaultPipeline(Level);
diff --git a/clang/lib/CodeGen/CGBuiltin.cpp b/clang/lib/CodeGen/CGBuiltin.cpp
index 07a39bca9d7a2..6381d68c161c6 100644
--- a/clang/lib/CodeGen/CGBuiltin.cpp
+++ b/clang/lib/CodeGen/CGBuiltin.cpp
@@ -28,6 +28,7 @@
 #include "clang/Basic/TargetBuiltins.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/CodeGen/CGFunctionInfo.h"
+#include "clang/Frontend/FrontendDiagnostic.h"
 #include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/APInt.h"
 #include "llvm/ADT/SmallPtrSet.h"
@@ -223,9 +224,23 @@ static Value *EmitNontemporalLoad(CodeGenFunction &CGF, const CallExpr *E) {
   return CGF.EmitLoadOfScalar(LV, E->getExprLoc());
 }
 
+static void CheckAtomicAlignment(CodeGenFunction &CGF, const CallExpr *E) {
+  ASTContext &Ctx = CGF.getContext();
+  Address Ptr = CGF.EmitPointerWithAlignment(E->getArg(0));
+  unsigned Bytes = Ptr.getElementType()->isPointerTy()
+                       ? Ctx.getTypeSizeInChars(Ctx.VoidPtrTy).getQuantity()
+                       : Ptr.getElementType()->getScalarSizeInBits() / 8;
+  unsigned Align = Ptr.getAlignment().getQuantity();
+  if (Align % Bytes != 0) {
+    DiagnosticsEngine &Diags = CGF.CGM.getDiags();
+    Diags.Report(E->getBeginLoc(), diag::warn_sync_op_misaligned);
+  }
+}
+
 static RValue EmitBinaryAtomic(CodeGenFunction &CGF,
                                llvm::AtomicRMWInst::BinOp Kind,
                                const CallExpr *E) {
+  CheckAtomicAlignment(CGF, E);
   return RValue::get(MakeBinaryAtomicValue(CGF, Kind, E));
 }
 
@@ -237,6 +252,7 @@ static RValue EmitBinaryAtomicPost(CodeGenFunction &CGF,
                                    const CallExpr *E,
                                    Instruction::BinaryOps Op,
                                    bool Invert = false) {
+  CheckAtomicAlignment(CGF, E);
   QualType T = E->getType();
   assert(E->getArg(0)->getType()->isPointerType());
   assert(CGF.getContext().hasSameUnqualifiedType(T,
@@ -284,6 +300,7 @@ static RValue EmitBinaryAtomicPost(CodeGenFunction &CGF,
 /// invoke the function EmitAtomicCmpXchgForMSIntrin.
 static Value *MakeAtomicCmpXchgValue(CodeGenFunction &CGF, const CallExpr *E,
                                      bool ReturnBool) {
+  CheckAtomicAlignment(CGF, E);
   QualType T = ReturnBool ? E->getArg(1)->getType() : E->getType();
   llvm::Value *DestPtr = CGF.EmitScalarExpr(E->getArg(0));
   unsigned AddrSpace = DestPtr->getType()->getPointerAddressSpace();
@@ -2444,6 +2461,18 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
                                    Intrinsic::round,
                                    Intrinsic::experimental_constrained_round));
 
+    case Builtin::BIroundeven:
+    case Builtin::BIroundevenf:
+    case Builtin::BIroundevenl:
+    case Builtin::BI__builtin_roundeven:
+    case Builtin::BI__builtin_roundevenf:
+    case Builtin::BI__builtin_roundevenf16:
+    case Builtin::BI__builtin_roundevenl:
+    case Builtin::BI__builtin_roundevenf128:
+      return RValue::get(emitUnaryMaybeConstrainedFPBuiltin(*this, E,
+                                   Intrinsic::roundeven,
+                                   Intrinsic::experimental_constrained_roundeven));
+                                   
     case Builtin::BIsin:
     case Builtin::BIsinf:
     case Builtin::BIsinl:
@@ -3088,6 +3117,12 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
   case Builtin::BI__builtin_elementwise_ceil:
     return RValue::get(
         emitUnaryBuiltin(*this, E, llvm::Intrinsic::ceil, "elt.ceil"));
+  case Builtin::BI__builtin_elementwise_exp:
+    return RValue::get(
+        emitUnaryBuiltin(*this, E, llvm::Intrinsic::exp, "elt.exp"));
+  case Builtin::BI__builtin_elementwise_exp2:
+    return RValue::get(
+        emitUnaryBuiltin(*this, E, llvm::Intrinsic::exp2, "elt.exp2"));
   case Builtin::BI__builtin_elementwise_log:
     return RValue::get(
         emitUnaryBuiltin(*this, E, llvm::Intrinsic::log, "elt.log"));
@@ -3363,6 +3398,14 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
     return RValue::get(Result);
   }
 
+  case Builtin::BI__builtin_set_flt_rounds: {
+    Function *F = CGM.getIntrinsic(Intrinsic::set_rounding);
+
+    Value *V = EmitScalarExpr(E->getArg(0));
+    Builder.CreateCall(F, V);
+    return RValue::get(nullptr);
+  }
+
   case Builtin::BI__builtin_fpclassify: {
     CodeGenFunction::CGFPOptionsRAII FPOptsRAII(*this, E);
     // FIXME: for strictfp/IEEE-754 we need to not trap on SNaN here.
@@ -3990,6 +4033,7 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
   case Builtin::BI__sync_lock_release_4:
   case Builtin::BI__sync_lock_release_8:
   case Builtin::BI__sync_lock_release_16: {
+    CheckAtomicAlignment(*this, E);
     Value *Ptr = EmitScalarExpr(E->getArg(0));
     QualType ElTy = E->getArg(0)->getType()->getPointeeType();
     CharUnits StoreSize = getContext().getTypeSizeInChars(ElTy);
@@ -9504,6 +9548,9 @@ Value *CodeGenFunction::EmitAArch64SVEBuiltinExpr(unsigned BuiltinID,
     else if (TypeFlags.isReverseMergeAnyBinOp() &&
              TypeFlags.getMergeType() == SVETypeFlags::MergeAny)
       std::swap(Ops[1], Ops[2]);
+    else if (TypeFlags.isReverseMergeAnyAccOp() &&
+             TypeFlags.getMergeType() == SVETypeFlags::MergeAny)
+      std::swap(Ops[1], Ops[3]);
 
     // Predicated intrinsics with _z suffix need a select w/ zeroinitializer.
     if (TypeFlags.getMergeType() == SVETypeFlags::MergeZero) {
@@ -18101,7 +18148,12 @@ static NVPTXMmaInfo getNVPTXMmaInfo(unsigned BuiltinID) {
 
 Value *
 CodeGenFunction::EmitNVPTXBuiltinExpr(unsigned BuiltinID, const CallExpr *E) {
-  auto MakeLdg = [&](unsigned IntrinsicID) {
+  auto HasHalfSupport = [&](unsigned BuiltinID) {
+    auto &Context = getContext();
+    return Context.getLangOpts().NativeHalfType ||
+           !Context.getTargetInfo().useFP16ConversionIntrinsics();
+  };
+  auto MakeLdgLdu = [&](unsigned IntrinsicID) {
     Value *Ptr = EmitScalarExpr(E->getArg(0));
     QualType ArgType = E->getArg(0)->getType();
     clang::CharUnits Align = CGM.getNaturalPointeeTypeAlignment(ArgType);
@@ -18227,15 +18279,63 @@ CodeGenFunction::EmitNVPTXBuiltinExpr(unsigned BuiltinID, const CallExpr *E) {
     // PTX Interoperability section 2.2: "For a vector with an even number of
     // elements, its alignment is set to number of elements times the alignment
     // of its member: n*alignof(t)."
-    return MakeLdg(Intrinsic::nvvm_ldg_global_i);
+    return MakeLdgLdu(Intrinsic::nvvm_ldg_global_i);
   case NVPTX::BI__nvvm_ldg_h:
-  case NVPTX::BI__nvvm_ldg_f:
   case NVPTX::BI__nvvm_ldg_h2:
+    if (!HasHalfSupport(BuiltinID)) {
+      CGM.Error(E->getExprLoc(),
+                getContext().BuiltinInfo.getName(BuiltinID).str() +
+                    " requires native half type support.");
+      return nullptr;
+    }
+    [[fallthrough]];
+  case NVPTX::BI__nvvm_ldg_f:
   case NVPTX::BI__nvvm_ldg_f2:
   case NVPTX::BI__nvvm_ldg_f4:
   case NVPTX::BI__nvvm_ldg_d:
   case NVPTX::BI__nvvm_ldg_d2:
-    return MakeLdg(Intrinsic::nvvm_ldg_global_f);
+    return MakeLdgLdu(Intrinsic::nvvm_ldg_global_f);
+
+  case NVPTX::BI__nvvm_ldu_c:
+  case NVPTX::BI__nvvm_ldu_c2:
+  case NVPTX::BI__nvvm_ldu_c4:
+  case NVPTX::BI__nvvm_ldu_s:
+  case NVPTX::BI__nvvm_ldu_s2:
+  case NVPTX::BI__nvvm_ldu_s4:
+  case NVPTX::BI__nvvm_ldu_i:
+  case NVPTX::BI__nvvm_ldu_i2:
+  case NVPTX::BI__nvvm_ldu_i4:
+  case NVPTX::BI__nvvm_ldu_l:
+  case NVPTX::BI__nvvm_ldu_ll:
+  case NVPTX::BI__nvvm_ldu_ll2:
+  case NVPTX::BI__nvvm_ldu_uc:
+  case NVPTX::BI__nvvm_ldu_uc2:
+  case NVPTX::BI__nvvm_ldu_uc4:
+  case NVPTX::BI__nvvm_ldu_us:
+  case NVPTX::BI__nvvm_ldu_us2:
+  case NVPTX::BI__nvvm_ldu_us4:
+  case NVPTX::BI__nvvm_ldu_ui:
+  case NVPTX::BI__nvvm_ldu_ui2:
+  case NVPTX::BI__nvvm_ldu_ui4:
+  case NVPTX::BI__nvvm_ldu_ul:
+  case NVPTX::BI__nvvm_ldu_ull:
+  case NVPTX::BI__nvvm_ldu_ull2:
+    return MakeLdgLdu(Intrinsic::nvvm_ldu_global_i);
+  case NVPTX::BI__nvvm_ldu_h:
+  case NVPTX::BI__nvvm_ldu_h2:
+    if (!HasHalfSupport(BuiltinID)) {
+      CGM.Error(E->getExprLoc(),
+                getContext().BuiltinInfo.getName(BuiltinID).str() +
+                    " requires native half type support.");
+      return nullptr;
+    }
+    [[fallthrough]];
+  case NVPTX::BI__nvvm_ldu_f:
+  case NVPTX::BI__nvvm_ldu_f2:
+  case NVPTX::BI__nvvm_ldu_f4:
+  case NVPTX::BI__nvvm_ldu_d:
+  case NVPTX::BI__nvvm_ldu_d2:
+    return MakeLdgLdu(Intrinsic::nvvm_ldu_global_f);
 
   case NVPTX::BI__nvvm_atom_cta_add_gen_i:
   case NVPTX::BI__nvvm_atom_cta_add_gen_l:
@@ -18855,6 +18955,10 @@ Value *CodeGenFunction::EmitWebAssemblyBuiltinExpr(unsigned BuiltinID,
     Function *Callee = CGM.getIntrinsic(Intrinsic::wasm_ref_null_extern);
     return Builder.CreateCall(Callee);
   }
+  case WebAssembly::BI__builtin_wasm_ref_null_func: {
+    Function *Callee = CGM.getIntrinsic(Intrinsic::wasm_ref_null_func);
+    return Builder.CreateCall(Callee);
+  }
   case WebAssembly::BI__builtin_wasm_swizzle_i8x16: {
     Value *Src = EmitScalarExpr(E->getArg(0));
     Value *Indices = EmitScalarExpr(E->getArg(1));
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index 319a9b448d513..aaa5d4e70d7db 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -2106,6 +2106,39 @@ static bool IsArgumentMaybeUndef(const Decl *TargetDecl,
   return false;
 }
 
+/// Test if it's legal to apply nofpclass for the given parameter type and it's
+/// lowered IR type.
+static bool canApplyNoFPClass(const ABIArgInfo &AI, QualType ParamType,
+                              bool IsReturn) {
+  // Should only apply to FP types in the source, not ABI promoted.
+  if (!ParamType->hasFloatingRepresentation())
+    return false;
+
+  // The promoted-to IR type also needs to support nofpclass.
+  llvm::Type *IRTy = AI.getCoerceToType();
+  if (llvm::AttributeFuncs::isNoFPClassCompatibleType(IRTy))
+    return true;
+
+  if (llvm::StructType *ST = dyn_cast<llvm::StructType>(IRTy)) {
+    return !IsReturn && AI.getCanBeFlattened() &&
+           llvm::all_of(ST->elements(), [](llvm::Type *Ty) {
+             return llvm::AttributeFuncs::isNoFPClassCompatibleType(Ty);
+           });
+  }
+
+  return false;
+}
+
+/// Return the nofpclass mask that can be applied to floating-point parameters.
+static llvm::FPClassTest getNoFPClassTestMask(const LangOptions &LangOpts) {
+  llvm::FPClassTest Mask = llvm::fcNone;
+  if (LangOpts.NoHonorInfs)
+    Mask |= llvm::fcInf;
+  if (LangOpts.NoHonorNaNs)
+    Mask |= llvm::fcNan;
+  return Mask;
+}
+
 /// Construct the IR attribute list of a function or call.
 ///
 /// When adding an attribute, please consider where it should be handled:
@@ -2373,6 +2406,10 @@ void CodeGenModule::ConstructAttributeList(StringRef Name,
   case ABIArgInfo::Direct:
     if (RetAI.getInReg())
       RetAttrs.addAttribute(llvm::Attribute::InReg);
+
+    if (canApplyNoFPClass(RetAI, RetTy, true))
+      RetAttrs.addNoFPClassAttr(getNoFPClassTestMask(getLangOpts()));
+
     break;
   case ABIArgInfo::Ignore:
     break;
@@ -2511,8 +2548,10 @@ void CodeGenModule::ConstructAttributeList(StringRef Name,
       else if (AI.getInReg())
         Attrs.addAttribute(llvm::Attribute::InReg);
       Attrs.addStackAlignmentAttr(llvm::MaybeAlign(AI.getDirectAlign()));
-      break;
 
+      if (canApplyNoFPClass(AI, ParamType, false))
+        Attrs.addNoFPClassAttr(getNoFPClassTestMask(getLangOpts()));
+      break;
     case ABIArgInfo::Indirect: {
       if (AI.getInReg())
         Attrs.addAttribute(llvm::Attribute::InReg);
@@ -4763,7 +4802,8 @@ RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
     // the proper cpu features (and it won't cause code generation issues due to
     // function based code generation).
     if (TargetDecl->hasAttr<AlwaysInlineAttr>() &&
-        TargetDecl->hasAttr<TargetAttr>())
+        (TargetDecl->hasAttr<TargetAttr>() ||
+         (CurFuncDecl && CurFuncDecl->hasAttr<TargetAttr>())))
       checkTargetFeatures(Loc, FD);
 
     // Some architectures (such as x86-64) have the ABI changed based on
diff --git a/clang/lib/CodeGen/CGDebugInfo.cpp b/clang/lib/CodeGen/CGDebugInfo.cpp
index 126b239042ee1..0b4c24ac2555d 100644
--- a/clang/lib/CodeGen/CGDebugInfo.cpp
+++ b/clang/lib/CodeGen/CGDebugInfo.cpp
@@ -727,24 +727,41 @@ llvm::DIType *CGDebugInfo::CreateType(const BuiltinType *BT) {
 #include "clang/Basic/AArch64SVEACLETypes.def"
     {
       ASTContext::BuiltinVectorTypeInfo Info =
-          CGM.getContext().getBuiltinVectorTypeInfo(BT);
-      unsigned NumElemsPerVG = (Info.EC.getKnownMinValue() * Info.NumVectors) / 2;
+          // For svcount_t, only the lower 2 bytes are relevant.
+          BT->getKind() == BuiltinType::SveCount
+              ? ASTContext::BuiltinVectorTypeInfo(
+                    CGM.getContext().BoolTy, llvm::ElementCount::getFixed(16),
+                    1)
+              : CGM.getContext().getBuiltinVectorTypeInfo(BT);
+
+      // A single vector of bytes may not suffice as the representation of
+      // svcount_t tuples because of the gap between the active 16bits of
+      // successive tuple members. Currently no such tuples are defined for
+      // svcount_t, so assert that NumVectors is 1.
+      assert((BT->getKind() != BuiltinType::SveCount || Info.NumVectors == 1) &&
+             "Unsupported number of vectors for svcount_t");
 
       // Debuggers can't extract 1bit from a vector, so will display a
-      // bitpattern for svbool_t instead.
+      // bitpattern for predicates instead.
+      unsigned NumElems = Info.EC.getKnownMinValue() * Info.NumVectors;
       if (Info.ElementType == CGM.getContext().BoolTy) {
-        NumElemsPerVG /= 8;
+        NumElems /= 8;
         Info.ElementType = CGM.getContext().UnsignedCharTy;
       }
 
-      auto *LowerBound =
-          llvm::ConstantAsMetadata::get(llvm::ConstantInt::getSigned(
-              llvm::Type::getInt64Ty(CGM.getLLVMContext()), 0));
-      SmallVector<uint64_t, 9> Expr(
-          {llvm::dwarf::DW_OP_constu, NumElemsPerVG, llvm::dwarf::DW_OP_bregx,
-           /* AArch64::VG */ 46, 0, llvm::dwarf::DW_OP_mul,
-           llvm::dwarf::DW_OP_constu, 1, llvm::dwarf::DW_OP_minus});
-      auto *UpperBound = DBuilder.createExpression(Expr);
+      llvm::Metadata *LowerBound, *UpperBound;
+      LowerBound = llvm::ConstantAsMetadata::get(llvm::ConstantInt::getSigned(
+          llvm::Type::getInt64Ty(CGM.getLLVMContext()), 0));
+      if (Info.EC.isScalable()) {
+        unsigned NumElemsPerVG = NumElems / 2;
+        SmallVector<uint64_t, 9> Expr(
+            {llvm::dwarf::DW_OP_constu, NumElemsPerVG, llvm::dwarf::DW_OP_bregx,
+             /* AArch64::VG */ 46, 0, llvm::dwarf::DW_OP_mul,
+             llvm::dwarf::DW_OP_constu, 1, llvm::dwarf::DW_OP_minus});
+        UpperBound = DBuilder.createExpression(Expr);
+      } else
+        UpperBound = llvm::ConstantAsMetadata::get(llvm::ConstantInt::getSigned(
+            llvm::Type::getInt64Ty(CGM.getLLVMContext()), NumElems - 1));
 
       llvm::Metadata *Subscript = DBuilder.getOrCreateSubrange(
           /*count*/ nullptr, LowerBound, UpperBound, /*stride*/ nullptr);
diff --git a/clang/lib/CodeGen/CGDecl.cpp b/clang/lib/CodeGen/CGDecl.cpp
index 0a503a237f1e2..bbbe4749cdfb0 100644
--- a/clang/lib/CodeGen/CGDecl.cpp
+++ b/clang/lib/CodeGen/CGDecl.cpp
@@ -394,13 +394,15 @@ CodeGenFunction::AddInitializerToStaticVarDecl(const VarDecl &D,
     OldGV->eraseFromParent();
   }
 
-  GV->setConstant(CGM.isTypeConstant(D.getType(), true));
+  bool NeedsDtor =
+      D.needsDestruction(getContext()) == QualType::DK_cxx_destructor;
+
+  GV->setConstant(CGM.isTypeConstant(D.getType(), true, !NeedsDtor));
   GV->setInitializer(Init);
 
   emitter.finalize(GV);
 
-  if (D.needsDestruction(getContext()) == QualType::DK_cxx_destructor &&
-      HaveInsertPoint()) {
+  if (NeedsDtor && HaveInsertPoint()) {
     // We have a constant initializer, but a nontrivial destructor. We still
     // need to perform a guarded "initialization" in order to register the
     // destructor.
@@ -1481,10 +1483,12 @@ CodeGenFunction::EmitAutoVarAlloca(const VarDecl &D) {
       // emit it as a global instead.
       // Exception is if a variable is located in non-constant address space
       // in OpenCL.
+      bool NeedsDtor =
+          D.needsDestruction(getContext()) == QualType::DK_cxx_destructor;
       if ((!getLangOpts().OpenCL ||
            Ty.getAddressSpace() == LangAS::opencl_constant) &&
           (CGM.getCodeGenOpts().MergeAllConstants && !NRVO &&
-           !isEscapingByRef && CGM.isTypeConstant(Ty, true))) {
+           !isEscapingByRef && CGM.isTypeConstant(Ty, true, !NeedsDtor))) {
         EmitStaticVarDecl(D, llvm::GlobalValue::InternalLinkage);
 
         // Signal this condition to later callbacks.
diff --git a/clang/lib/CodeGen/CGDeclCXX.cpp b/clang/lib/CodeGen/CGDeclCXX.cpp
index 416859668af98..0d0b5707e605a 100644
--- a/clang/lib/CodeGen/CGDeclCXX.cpp
+++ b/clang/lib/CodeGen/CGDeclCXX.cpp
@@ -214,9 +214,11 @@ void CodeGenFunction::EmitCXXGlobalVarDeclInit(const VarDecl &D,
           &D, DeclAddr, D.getAttr<OMPThreadPrivateDeclAttr>()->getLocation(),
           PerformInit, this);
     }
+    bool NeedsDtor =
+        D.needsDestruction(getContext()) == QualType::DK_cxx_destructor;
     if (PerformInit)
       EmitDeclInit(*this, D, DeclAddr);
-    if (CGM.isTypeConstant(D.getType(), true))
+    if (CGM.isTypeConstant(D.getType(), true, !NeedsDtor))
       EmitDeclInvariant(*this, D, DeclPtr);
     else
       EmitDeclDestroy(*this, D, DeclAddr);
diff --git a/clang/lib/CodeGen/CGExpr.cpp b/clang/lib/CodeGen/CGExpr.cpp
index a9116dd9c5b08..44b97d272634e 100644
--- a/clang/lib/CodeGen/CGExpr.cpp
+++ b/clang/lib/CodeGen/CGExpr.cpp
@@ -401,7 +401,7 @@ static Address createReferenceTemporary(CodeGenFunction &CGF,
     QualType Ty = Inner->getType();
     if (CGF.CGM.getCodeGenOpts().MergeAllConstants &&
         (Ty->isArrayType() || Ty->isRecordType()) &&
-        CGF.CGM.isTypeConstant(Ty, true))
+        CGF.CGM.isTypeConstant(Ty, true, false))
       if (auto Init = ConstantEmitter(CGF).tryEmitAbstract(Inner, Ty)) {
         auto AS = CGF.CGM.GetGlobalConstantAddressSpace();
         auto *GV = new llvm::GlobalVariable(
diff --git a/clang/lib/CodeGen/CGExprAgg.cpp b/clang/lib/CodeGen/CGExprAgg.cpp
index b7fe7fefec2f9..33d2295fa98ee 100644
--- a/clang/lib/CodeGen/CGExprAgg.cpp
+++ b/clang/lib/CodeGen/CGExprAgg.cpp
@@ -532,8 +532,8 @@ void AggExprEmitter::EmitArrayInit(Address DestPtr, llvm::ArrayType *AType,
             Emitter.tryEmitForInitializer(ExprToVisit, AS, ArrayQTy)) {
       auto GV = new llvm::GlobalVariable(
           CGM.getModule(), C->getType(),
-          CGM.isTypeConstant(ArrayQTy, /* ExcludeCtorDtor= */ true),
-          llvm::GlobalValue::PrivateLinkage, C, "constinit",
+          /* isConstant= */ true, llvm::GlobalValue::PrivateLinkage, C,
+          "constinit",
           /* InsertBefore= */ nullptr, llvm::GlobalVariable::NotThreadLocal,
           CGM.getContext().getTargetAddressSpace(AS));
       Emitter.finalize(GV);
diff --git a/clang/lib/CodeGen/CGExprConstant.cpp b/clang/lib/CodeGen/CGExprConstant.cpp
index c38feaaca35a1..329e1a27d98fc 100644
--- a/clang/lib/CodeGen/CGExprConstant.cpp
+++ b/clang/lib/CodeGen/CGExprConstant.cpp
@@ -932,12 +932,12 @@ tryEmitGlobalCompoundLiteral(ConstantEmitter &emitter,
     return ConstantAddress::invalid();
   }
 
-  auto GV = new llvm::GlobalVariable(CGM.getModule(), C->getType(),
-                                     CGM.isTypeConstant(E->getType(), true),
-                                     llvm::GlobalValue::InternalLinkage,
-                                     C, ".compoundliteral", nullptr,
-                                     llvm::GlobalVariable::NotThreadLocal,
-                    CGM.getContext().getTargetAddressSpace(addressSpace));
+  auto GV = new llvm::GlobalVariable(
+      CGM.getModule(), C->getType(),
+      CGM.isTypeConstant(E->getType(), true, false),
+      llvm::GlobalValue::InternalLinkage, C, ".compoundliteral", nullptr,
+      llvm::GlobalVariable::NotThreadLocal,
+      CGM.getContext().getTargetAddressSpace(addressSpace));
   emitter.finalize(GV);
   GV->setAlignment(Align.getAsAlign());
   CGM.setAddrOfConstantCompoundLiteral(E, GV);
@@ -1570,7 +1570,7 @@ namespace {
     }
 
     void setLocation(llvm::GlobalVariable *placeholder) {
-      assert(Locations.find(placeholder) == Locations.end() &&
+      assert(!Locations.contains(placeholder) &&
              "already found location for placeholder!");
 
       // Lazily fill in IndexValues with the values from Indices.
diff --git a/clang/lib/CodeGen/CGOpenCLRuntime.cpp b/clang/lib/CodeGen/CGOpenCLRuntime.cpp
index 5a5d7a4d18c57..dc2330a299762 100644
--- a/clang/lib/CodeGen/CGOpenCLRuntime.cpp
+++ b/clang/lib/CodeGen/CGOpenCLRuntime.cpp
@@ -31,8 +31,11 @@ void CGOpenCLRuntime::EmitWorkGroupLocalVarDecl(CodeGenFunction &CGF,
 }
 
 llvm::Type *CGOpenCLRuntime::convertOpenCLSpecificType(const Type *T) {
-  assert(T->isOpenCLSpecificType() &&
-         "Not an OpenCL specific type!");
+  assert(T->isOpenCLSpecificType() && "Not an OpenCL specific type!");
+
+  // Check if the target has a specific translation for this type first.
+  if (llvm::Type *TransTy = CGM.getTargetCodeGenInfo().getOpenCLType(CGM, T))
+    return TransTy;
 
   switch (cast<BuiltinType>(T)->getKind()) {
   default:
@@ -75,6 +78,9 @@ llvm::PointerType *CGOpenCLRuntime::getPointerType(const Type *T,
 }
 
 llvm::Type *CGOpenCLRuntime::getPipeType(const PipeType *T) {
+  if (llvm::Type *PipeTy = CGM.getTargetCodeGenInfo().getOpenCLType(CGM, T))
+    return PipeTy;
+
   if (T->isReadOnly())
     return getPipeType(T, "opencl.pipe_ro_t", PipeROTy);
   else
@@ -91,12 +97,18 @@ llvm::Type *CGOpenCLRuntime::getPipeType(const PipeType *T, StringRef Name,
   return PipeTy;
 }
 
-llvm::PointerType *CGOpenCLRuntime::getSamplerType(const Type *T) {
-  if (!SamplerTy)
-    SamplerTy = llvm::PointerType::get(llvm::StructType::create(
-      CGM.getLLVMContext(), "opencl.sampler_t"),
-      CGM.getContext().getTargetAddressSpace(
-          CGM.getContext().getOpenCLTypeAddrSpace(T)));
+llvm::Type *CGOpenCLRuntime::getSamplerType(const Type *T) {
+  if (SamplerTy)
+    return SamplerTy;
+
+  if (llvm::Type *TransTy = CGM.getTargetCodeGenInfo().getOpenCLType(
+          CGM, CGM.getContext().OCLSamplerTy.getTypePtr()))
+    SamplerTy = TransTy;
+  else
+    SamplerTy = llvm::PointerType::get(
+        llvm::StructType::create(CGM.getLLVMContext(), "opencl.sampler_t"),
+        CGM.getContext().getTargetAddressSpace(
+            CGM.getContext().getOpenCLTypeAddrSpace(T)));
   return SamplerTy;
 }
 
@@ -149,8 +161,7 @@ static const BlockExpr *getBlockExpr(const Expr *E) {
 void CGOpenCLRuntime::recordBlockInfo(const BlockExpr *E,
                                       llvm::Function *InvokeF,
                                       llvm::Value *Block, llvm::Type *BlockTy) {
-  assert(EnqueuedBlockMap.find(E) == EnqueuedBlockMap.end() &&
-         "Block expression emitted twice");
+  assert(!EnqueuedBlockMap.contains(E) && "Block expression emitted twice");
   assert(isa<llvm::Function>(InvokeF) && "Invalid invoke function");
   assert(Block->getType()->isPointerTy() && "Invalid block literal type");
   EnqueuedBlockMap[E].InvokeFunc = InvokeF;
@@ -171,8 +182,7 @@ CGOpenCLRuntime::emitOpenCLEnqueuedBlock(CodeGenFunction &CGF, const Expr *E) {
   // to get the block literal.
   const BlockExpr *Block = getBlockExpr(E);
 
-  assert(EnqueuedBlockMap.find(Block) != EnqueuedBlockMap.end() &&
-         "Block expression not emitted");
+  assert(EnqueuedBlockMap.contains(Block) && "Block expression not emitted");
 
   // Do not emit the block wrapper again if it has been emitted.
   if (EnqueuedBlockMap[Block].KernelHandle) {
diff --git a/clang/lib/CodeGen/CGOpenCLRuntime.h b/clang/lib/CodeGen/CGOpenCLRuntime.h
index b93e2041c8f16..df8084d6008be 100644
--- a/clang/lib/CodeGen/CGOpenCLRuntime.h
+++ b/clang/lib/CodeGen/CGOpenCLRuntime.h
@@ -38,7 +38,7 @@ class CGOpenCLRuntime {
   CodeGenModule &CGM;
   llvm::Type *PipeROTy;
   llvm::Type *PipeWOTy;
-  llvm::PointerType *SamplerTy;
+  llvm::Type *SamplerTy;
   llvm::StringMap<llvm::PointerType *> CachedTys;
 
   /// Structure for enqueued block information.
@@ -70,7 +70,7 @@ class CGOpenCLRuntime {
 
   virtual llvm::Type *getPipeType(const PipeType *T);
 
-  llvm::PointerType *getSamplerType(const Type *T);
+  llvm::Type *getSamplerType(const Type *T);
 
   // Returns a value which indicates the size in bytes of the pipe
   // element.
diff --git a/clang/lib/CodeGen/CGOpenMPRuntime.cpp b/clang/lib/CodeGen/CGOpenMPRuntime.cpp
index ff2e9423b6b2e..36bc7f10762d7 100644
--- a/clang/lib/CodeGen/CGOpenMPRuntime.cpp
+++ b/clang/lib/CodeGen/CGOpenMPRuntime.cpp
@@ -7163,6 +7163,7 @@ class MappableExprsHandler {
     // double d;
     // int i[100];
     // float *p;
+    // int **a = &i;
     //
     // struct S1 {
     //   int i;
@@ -7196,6 +7197,14 @@ class MappableExprsHandler {
     // in unified shared memory mode or for local pointers
     // p, &p[1], 24*sizeof(float), TARGET_PARAM | TO | FROM
     //
+    // map((*a)[0:3])
+    // &(*a), &(*a), sizeof(pointer), TARGET_PARAM | TO | FROM
+    // &(*a), &(*a)[0], 3*sizeof(int), PTR_AND_OBJ | TO | FROM
+    //
+    // map(**a)
+    // &(*a), &(*a), sizeof(pointer), TARGET_PARAM | TO | FROM
+    // &(*a), &(**a), sizeof(int), PTR_AND_OBJ | TO | FROM
+    //
     // map(s)
     // &s, &s, sizeof(S2), TARGET_PARAM | TO | FROM
     //
@@ -7488,7 +7497,9 @@ class MappableExprsHandler {
       bool IsMemberReference = isa<MemberExpr>(I->getAssociatedExpression()) &&
                                MapDecl &&
                                MapDecl->getType()->isLValueReferenceType();
-      bool IsNonDerefPointer = IsPointer && !UO && !BO && !IsNonContiguous;
+      bool IsNonDerefPointer = IsPointer &&
+                               !(UO && UO->getOpcode() != UO_Deref) && !BO &&
+                               !IsNonContiguous;
 
       if (OASE)
         ++DimSize;
@@ -8367,7 +8378,8 @@ class MappableExprsHandler {
       // individual members mapped. Emit an extra combined entry.
       if (PartialStruct.Base.isValid()) {
         CurInfo.NonContigInfo.Dims.push_back(0);
-        emitCombinedEntry(CombinedInfo, CurInfo.Types, PartialStruct, VD);
+        emitCombinedEntry(CombinedInfo, CurInfo.Types, PartialStruct,
+                          /*IsMapThis*/ !VD, VD);
       }
 
       // We need to append the results of this capture to what we already
@@ -8433,7 +8445,7 @@ class MappableExprsHandler {
   /// individual struct members.
   void emitCombinedEntry(MapCombinedInfoTy &CombinedInfo,
                          MapFlagsArrayTy &CurTypes,
-                         const StructRangeInfoTy &PartialStruct,
+                         const StructRangeInfoTy &PartialStruct, bool IsMapThis,
                          const ValueDecl *VD = nullptr,
                          bool NotTargetParams = true) const {
     if (CurTypes.size() == 1 &&
@@ -8455,8 +8467,7 @@ class MappableExprsHandler {
     const CXXMethodDecl *MD =
         CGF.CurFuncDecl ? dyn_cast<CXXMethodDecl>(CGF.CurFuncDecl) : nullptr;
     const CXXRecordDecl *RD = MD ? MD->getParent() : nullptr;
-    // When VD is not null, it is not field of class, skip generating this[:1].
-    bool HasBaseClass = RD && !VD ? RD->getNumBases() > 0 : false;
+    bool HasBaseClass = RD && IsMapThis ? RD->getNumBases() > 0 : false;
     // There should not be a mapper for a combined entry.
     if (HasBaseClass) {
       // OpenMP 5.2 148:21:
@@ -9941,6 +9952,9 @@ void CGOpenMPRuntime::emitTargetCall(
         DynCGroupMem,
     };
 
+    llvm::OpenMPIRBuilder::InsertPointTy AllocaIP(
+        CGF.AllocaInsertPt->getParent(), CGF.AllocaInsertPt->getIterator());
+
     // The target region is an outlined function launched by the runtime
     // via calls to __tgt_target_kernel().
     //
@@ -9955,7 +9969,7 @@ void CGOpenMPRuntime::emitTargetCall(
     // of teams and threads so no additional calls to the runtime are required.
     // Check the error code and execute the host version if required.
     CGF.Builder.restoreIP(OMPBuilder.emitTargetKernel(
-        CGF.Builder, Return, RTLoc, DeviceID, NumTeams, NumThreads,
+        CGF.Builder, AllocaIP, Return, RTLoc, DeviceID, NumTeams, NumThreads,
         OutlinedFnID, KernelArgs));
 
     llvm::BasicBlock *OffloadFailedBlock =
@@ -10041,8 +10055,8 @@ void CGOpenMPRuntime::emitTargetCall(
       if (PartialStruct.Base.isValid()) {
         CombinedInfo.append(PartialStruct.PreliminaryMapData);
         MEHandler.emitCombinedEntry(
-            CombinedInfo, CurInfo.Types, PartialStruct, nullptr,
-            !PartialStruct.PreliminaryMapData.BasePointers.empty());
+            CombinedInfo, CurInfo.Types, PartialStruct, CI->capturesThis(),
+            nullptr, !PartialStruct.PreliminaryMapData.BasePointers.empty());
       }
 
       // We need to append the results of this capture to what we already have.
diff --git a/clang/lib/CodeGen/CodeGenFunction.cpp b/clang/lib/CodeGen/CodeGenFunction.cpp
index af60b3166d756..a5cbf7223ad65 100644
--- a/clang/lib/CodeGen/CodeGenFunction.cpp
+++ b/clang/lib/CodeGen/CodeGenFunction.cpp
@@ -2612,6 +2612,16 @@ void CodeGenFunction::checkTargetFeatures(SourceLocation Loc,
     }))
       CGM.getDiags().Report(Loc, diag::err_function_needs_feature)
           << FD->getDeclName() << TargetDecl->getDeclName() << MissingFeature;
+  } else if (!FD->isMultiVersion() && FD->hasAttr<TargetAttr>()) {
+    llvm::StringMap<bool> CalleeFeatureMap;
+    CGM.getContext().getFunctionFeatureMap(CalleeFeatureMap, TargetDecl);
+
+    for (const auto &F : CalleeFeatureMap) {
+      if (F.getValue() && (!CallerFeatureMap.lookup(F.getKey()) ||
+                           !CallerFeatureMap.find(F.getKey())->getValue()))
+        CGM.getDiags().Report(Loc, diag::err_function_needs_feature)
+            << FD->getDeclName() << TargetDecl->getDeclName() << F.getKey();
+    }
   }
 }
 
diff --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp
index 90c9df4ea4423..0e33e9632b3eb 100644
--- a/clang/lib/CodeGen/CodeGenModule.cpp
+++ b/clang/lib/CodeGen/CodeGenModule.cpp
@@ -338,10 +338,11 @@ static const llvm::GlobalValue *getAliasedGlobal(const llvm::GlobalValue *GV) {
   return FinalGV;
 }
 
-static bool checkAliasedGlobal(DiagnosticsEngine &Diags,
-                               SourceLocation Location, bool IsIFunc,
-                               const llvm::GlobalValue *Alias,
-                               const llvm::GlobalValue *&GV) {
+static bool checkAliasedGlobal(
+    DiagnosticsEngine &Diags, SourceLocation Location, bool IsIFunc,
+    const llvm::GlobalValue *Alias, const llvm::GlobalValue *&GV,
+    const llvm::MapVector<GlobalDecl, StringRef> &MangledDeclNames,
+    SourceRange AliasRange) {
   GV = getAliasedGlobal(Alias);
   if (!GV) {
     Diags.Report(Location, diag::err_cyclic_alias) << IsIFunc;
@@ -350,6 +351,22 @@ static bool checkAliasedGlobal(DiagnosticsEngine &Diags,
 
   if (GV->isDeclaration()) {
     Diags.Report(Location, diag::err_alias_to_undefined) << IsIFunc << IsIFunc;
+    Diags.Report(Location, diag::note_alias_requires_mangled_name)
+        << IsIFunc << IsIFunc;
+    // Provide a note if the given function is not found and exists as a
+    // mangled name.
+    for (const auto &[Decl, Name] : MangledDeclNames) {
+      if (const auto *ND = dyn_cast<NamedDecl>(Decl.getDecl())) {
+        if (ND->getName() == GV->getName()) {
+          Diags.Report(Location, diag::note_alias_mangled_name_alternative)
+              << Name
+              << FixItHint::CreateReplacement(
+                     AliasRange,
+                     (Twine(IsIFunc ? "ifunc" : "alias") + "(\"" + Name + "\")")
+                         .str());
+        }
+      }
+    }
     return false;
   }
 
@@ -381,16 +398,19 @@ void CodeGenModule::checkAliases() {
   for (const GlobalDecl &GD : Aliases) {
     const auto *D = cast<ValueDecl>(GD.getDecl());
     SourceLocation Location;
+    SourceRange Range;
     bool IsIFunc = D->hasAttr<IFuncAttr>();
-    if (const Attr *A = D->getDefiningAttr())
+    if (const Attr *A = D->getDefiningAttr()) {
       Location = A->getLocation();
-    else
+      Range = A->getRange();
+    } else
       llvm_unreachable("Not an alias or ifunc?");
 
     StringRef MangledName = getMangledName(GD);
     llvm::GlobalValue *Alias = GetGlobalValue(MangledName);
     const llvm::GlobalValue *GV = nullptr;
-    if (!checkAliasedGlobal(Diags, Location, IsIFunc, Alias, GV)) {
+    if (!checkAliasedGlobal(Diags, Location, IsIFunc, Alias, GV,
+                            MangledDeclNames, Range)) {
       Error = true;
       continue;
     }
@@ -3110,7 +3130,7 @@ bool CodeGenModule::MayBeEmittedEagerly(const ValueDecl *Global) {
   // codegen for global variables, because they may be marked as threadprivate.
   if (LangOpts.OpenMP && LangOpts.OpenMPUseTLS &&
       getContext().getTargetInfo().isTLSSupported() && isa<VarDecl>(Global) &&
-      !isTypeConstant(Global->getType(), false) &&
+      !isTypeConstant(Global->getType(), false, false) &&
       !OMPDeclareTargetDeclAttr::isDeclareTargetDeclaration(Global))
     return false;
 
@@ -3218,9 +3238,13 @@ ConstantAddress CodeGenModule::GetAddrOfTemplateParamObject(
     return ConstantAddress::invalid();
   }
 
-  auto *GV = new llvm::GlobalVariable(
-      getModule(), Init->getType(),
-      /*isConstant=*/true, llvm::GlobalValue::LinkOnceODRLinkage, Init, Name);
+  llvm::GlobalValue::LinkageTypes Linkage =
+      isExternallyVisible(TPO->getLinkageAndVisibility().getLinkage())
+          ? llvm::GlobalValue::LinkOnceODRLinkage
+          : llvm::GlobalValue::InternalLinkage;
+  auto *GV = new llvm::GlobalVariable(getModule(), Init->getType(),
+                                      /*isConstant=*/true, Linkage, Init, Name);
+  setGVProperties(GV, TPO);
   if (supportsCOMDAT())
     GV->setComdat(TheModule.getOrInsertComdat(GV->getName()));
   Emitter.finalize(GV);
@@ -4311,8 +4335,9 @@ CodeGenModule::CreateRuntimeFunction(llvm::FunctionType *FTy, StringRef Name,
 ///
 /// If ExcludeCtor is true, the duration when the object's constructor runs
 /// will not be considered. The caller will need to verify that the object is
-/// not written to during its construction.
-bool CodeGenModule::isTypeConstant(QualType Ty, bool ExcludeCtor) {
+/// not written to during its construction. ExcludeDtor works similarly.
+bool CodeGenModule::isTypeConstant(QualType Ty, bool ExcludeCtor,
+                                   bool ExcludeDtor) {
   if (!Ty.isConstant(Context) && !Ty->isReferenceType())
     return false;
 
@@ -4320,7 +4345,7 @@ bool CodeGenModule::isTypeConstant(QualType Ty, bool ExcludeCtor) {
     if (const CXXRecordDecl *Record
           = Context.getBaseElementType(Ty)->getAsCXXRecordDecl())
       return ExcludeCtor && !Record->hasMutableFields() &&
-             Record->hasTrivialDestructor();
+             (Record->hasTrivialDestructor() || ExcludeDtor);
   }
 
   return true;
@@ -4433,7 +4458,7 @@ CodeGenModule::GetOrCreateLLVMGlobal(StringRef MangledName, llvm::Type *Ty,
 
     // FIXME: This code is overly simple and should be merged with other global
     // handling.
-    GV->setConstant(isTypeConstant(D->getType(), false));
+    GV->setConstant(isTypeConstant(D->getType(), false, false));
 
     GV->setAlignment(getContext().getDeclAlign(D).getAsAlign());
 
@@ -4989,7 +5014,7 @@ void CodeGenModule::EmitGlobalVarDefinition(const VarDecl *D,
 
   // If it is safe to mark the global 'constant', do so now.
   GV->setConstant(!NeedsGlobalCtor && !NeedsGlobalDtor &&
-                  isTypeConstant(D->getType(), true));
+                  isTypeConstant(D->getType(), true, true));
 
   // If it is in a read-only section, mark it 'constant'.
   if (const SectionAttr *SA = D->getAttr<SectionAttr>()) {
@@ -6063,7 +6088,8 @@ ConstantAddress CodeGenModule::GetAddrOfGlobalTemporary(
     emitter.emplace(*this);
     InitialValue = emitter->emitForInitializer(*Value, AddrSpace,
                                                MaterializedType);
-    Constant = isTypeConstant(MaterializedType, /*ExcludeCtor*/Value);
+    Constant = isTypeConstant(MaterializedType, /*ExcludeCtor*/ Value,
+                              /*ExcludeDtor*/ false);
     Type = InitialValue->getType();
   } else {
     // No initializer, the initialization will be provided when we
diff --git a/clang/lib/CodeGen/CodeGenModule.h b/clang/lib/CodeGen/CodeGenModule.h
index b3354657b2370..64c2902572376 100644
--- a/clang/lib/CodeGen/CodeGenModule.h
+++ b/clang/lib/CodeGen/CodeGenModule.h
@@ -816,7 +816,7 @@ class CodeGenModule : public CodeGenTypeCache {
     return getTBAAAccessInfo(AccessType);
   }
 
-  bool isTypeConstant(QualType QTy, bool ExcludeCtorDtor);
+  bool isTypeConstant(QualType QTy, bool ExcludeCtor, bool ExcludeDtor);
 
   bool isPaddedAtomicType(QualType type);
   bool isPaddedAtomicType(const AtomicType *type);
diff --git a/clang/lib/CodeGen/CodeGenTypes.cpp b/clang/lib/CodeGen/CodeGenTypes.cpp
index 70d954639eb01..be9516925da25 100644
--- a/clang/lib/CodeGen/CodeGenTypes.cpp
+++ b/clang/lib/CodeGen/CodeGenTypes.cpp
@@ -596,6 +596,8 @@ llvm::Type *CodeGenTypes::ConvertType(QualType T) {
     case BuiltinType::SveInt64x4:
     case BuiltinType::SveUint64x4:
     case BuiltinType::SveBool:
+    case BuiltinType::SveBoolx2:
+    case BuiltinType::SveBoolx4:
     case BuiltinType::SveFloat16:
     case BuiltinType::SveFloat16x2:
     case BuiltinType::SveFloat16x3:
@@ -618,6 +620,8 @@ llvm::Type *CodeGenTypes::ConvertType(QualType T) {
                                            Info.EC.getKnownMinValue() *
                                                Info.NumVectors);
     }
+    case BuiltinType::SveCount:
+      return llvm::TargetExtType::get(getLLVMContext(), "aarch64.svcount");
 #define PPC_VECTOR_TYPE(Name, Id, Size) \
     case BuiltinType::Id: \
       ResultType = \
diff --git a/clang/lib/CodeGen/TargetInfo.cpp b/clang/lib/CodeGen/TargetInfo.cpp
index 307d1e78b99b1..40cc2da46bd05 100644
--- a/clang/lib/CodeGen/TargetInfo.cpp
+++ b/clang/lib/CodeGen/TargetInfo.cpp
@@ -295,17 +295,13 @@ LLVM_DUMP_METHOD void ABIArgInfo::dump() const {
 static llvm::Value *emitRoundPointerUpToAlignment(CodeGenFunction &CGF,
                                                   llvm::Value *Ptr,
                                                   CharUnits Align) {
-  llvm::Value *PtrAsInt = Ptr;
   // OverflowArgArea = (OverflowArgArea + Align - 1) & -Align;
-  PtrAsInt = CGF.Builder.CreatePtrToInt(PtrAsInt, CGF.IntPtrTy);
-  PtrAsInt = CGF.Builder.CreateAdd(PtrAsInt,
-        llvm::ConstantInt::get(CGF.IntPtrTy, Align.getQuantity() - 1));
-  PtrAsInt = CGF.Builder.CreateAnd(PtrAsInt,
-           llvm::ConstantInt::get(CGF.IntPtrTy, -Align.getQuantity()));
-  PtrAsInt = CGF.Builder.CreateIntToPtr(PtrAsInt,
-                                        Ptr->getType(),
-                                        Ptr->getName() + ".aligned");
-  return PtrAsInt;
+  llvm::Value *RoundUp = CGF.Builder.CreateConstInBoundsGEP1_32(
+      CGF.Builder.getInt8Ty(), Ptr, Align.getQuantity() - 1);
+  return CGF.Builder.CreateIntrinsic(
+      llvm::Intrinsic::ptrmask, {CGF.AllocaInt8PtrTy, CGF.IntPtrTy},
+      {RoundUp, llvm::ConstantInt::get(CGF.IntPtrTy, -Align.getQuantity())},
+      nullptr, Ptr->getName() + ".aligned");
 }
 
 /// Emit va_arg for a platform using the common void* representation,
@@ -904,6 +900,10 @@ class WebAssemblyTargetCodeGenInfo final : public TargetCodeGenInfo {
   virtual llvm::Type *getWasmExternrefReferenceType() const override {
     return llvm::Type::getWasm_ExternrefTy(getABIInfo().getVMContext());
   }
+  /// Return the WebAssembly funcref reference type.
+  virtual llvm::Type *getWasmFuncrefReferenceType() const override {
+    return llvm::Type::getWasm_FuncrefTy(getABIInfo().getVMContext());
+  }
 };
 
 /// Classify argument of given type \p Ty.
@@ -9632,7 +9632,7 @@ AMDGPUTargetCodeGenInfo::getGlobalVarAddressSpace(CodeGenModule &CGM,
     return AddrSpace;
 
   // Only promote to address space 4 if VarDecl has constant initialization.
-  if (CGM.isTypeConstant(D->getType(), false) &&
+  if (CGM.isTypeConstant(D->getType(), false, false) &&
       D->hasConstantInitialization()) {
     if (auto ConstAS = CGM.getTarget().getConstantAddressSpace())
       return *ConstAS;
@@ -10559,6 +10559,7 @@ class CommonSPIRTargetCodeGenInfo : public TargetCodeGenInfo {
   }
 
   unsigned getOpenCLKernelCallingConv() const override;
+  llvm::Type *getOpenCLType(CodeGenModule &CGM, const Type *T) const override;
 };
 class SPIRVTargetCodeGenInfo : public CommonSPIRTargetCodeGenInfo {
 public:
@@ -10973,6 +10974,80 @@ static bool getTypeString(SmallStringEnc &Enc, const Decl *D,
   return false;
 }
 
+/// Construct a SPIR-V target extension type for the given OpenCL image type.
+static llvm::Type *getSPIRVImageType(llvm::LLVMContext &Ctx, StringRef BaseType,
+                                     StringRef OpenCLName,
+                                     unsigned AccessQualifier) {
+  // These parameters compare to the operands of OpTypeImage (see
+  // https://registry.khronos.org/SPIR-V/specs/unified1/SPIRV.html#OpTypeImage
+  // for more details). The first 6 integer parameters all default to 0, and
+  // will be changed to 1 only for the image type(s) that set the parameter to
+  // one. The 7th integer parameter is the access qualifier, which is tacked on
+  // at the end.
+  SmallVector<unsigned, 7> IntParams = {0, 0, 0, 0, 0, 0};
+
+  // Choose the dimension of the image--this corresponds to the Dim enum in
+  // SPIR-V (first integer parameter of OpTypeImage).
+  if (OpenCLName.startswith("image2d"))
+    IntParams[0] = 1; // 1D
+  else if (OpenCLName.startswith("image3d"))
+    IntParams[0] = 2; // 2D
+  else if (OpenCLName == "image1d_buffer")
+    IntParams[0] = 5; // Buffer
+  else
+    assert(OpenCLName.startswith("image1d") && "Unknown image type");
+
+  // Set the other integer parameters of OpTypeImage if necessary. Note that the
+  // OpenCL image types don't provide any information for the Sampled or
+  // Image Format parameters.
+  if (OpenCLName.contains("_depth"))
+    IntParams[1] = 1;
+  if (OpenCLName.contains("_array"))
+    IntParams[2] = 1;
+  if (OpenCLName.contains("_msaa"))
+    IntParams[3] = 1;
+
+  // Access qualifier
+  IntParams.push_back(AccessQualifier);
+
+  return llvm::TargetExtType::get(Ctx, BaseType, {llvm::Type::getVoidTy(Ctx)},
+                                  IntParams);
+}
+
+llvm::Type *CommonSPIRTargetCodeGenInfo::getOpenCLType(CodeGenModule &CGM,
+                                                       const Type *Ty) const {
+  llvm::LLVMContext &Ctx = CGM.getLLVMContext();
+  if (auto *PipeTy = dyn_cast<PipeType>(Ty))
+    return llvm::TargetExtType::get(Ctx, "spirv.Pipe", {},
+                                    {!PipeTy->isReadOnly()});
+  if (auto *BuiltinTy = dyn_cast<BuiltinType>(Ty)) {
+    enum AccessQualifier : unsigned { AQ_ro = 0, AQ_wo = 1, AQ_rw = 2 };
+    switch (BuiltinTy->getKind()) {
+#define IMAGE_TYPE(ImgType, Id, SingletonId, Access, Suffix)                   \
+    case BuiltinType::Id:                                                      \
+      return getSPIRVImageType(Ctx, "spirv.Image", #ImgType, AQ_##Suffix);
+#include "clang/Basic/OpenCLImageTypes.def"
+    case BuiltinType::OCLSampler:
+      return llvm::TargetExtType::get(Ctx, "spirv.Sampler");
+    case BuiltinType::OCLEvent:
+      return llvm::TargetExtType::get(Ctx, "spirv.Event");
+    case BuiltinType::OCLClkEvent:
+      return llvm::TargetExtType::get(Ctx, "spirv.DeviceEvent");
+    case BuiltinType::OCLQueue:
+      return llvm::TargetExtType::get(Ctx, "spirv.Queue");
+    case BuiltinType::OCLReserveID:
+      return llvm::TargetExtType::get(Ctx, "spirv.ReserveId");
+#define INTEL_SUBGROUP_AVC_TYPE(Name, Id)                                      \
+    case BuiltinType::OCLIntelSubgroupAVC##Id:                                 \
+      return llvm::TargetExtType::get(Ctx, "spirv.Avc" #Id "INTEL");
+#include "clang/Basic/OpenCLExtensionTypes.def"
+    default:
+      return nullptr;
+    }
+  }
+
+  return nullptr;
+}
 //===----------------------------------------------------------------------===//
 // RISCV ABI Implementation
 //===----------------------------------------------------------------------===//
diff --git a/clang/lib/CodeGen/TargetInfo.h b/clang/lib/CodeGen/TargetInfo.h
index b97c068bb9b35..3971dba70762e 100644
--- a/clang/lib/CodeGen/TargetInfo.h
+++ b/clang/lib/CodeGen/TargetInfo.h
@@ -365,6 +365,9 @@ class TargetCodeGenInfo {
   /// Return the WebAssembly externref reference type.
   virtual llvm::Type *getWasmExternrefReferenceType() const { return nullptr; }
 
+  /// Return the WebAssembly funcref reference type.
+  virtual llvm::Type *getWasmFuncrefReferenceType() const { return nullptr; }
+
   /// Emit the device-side copy of the builtin surface type.
   virtual bool emitCUDADeviceBuiltinSurfaceDeviceCopy(CodeGenFunction &CGF,
                                                       LValue Dst,
@@ -379,6 +382,11 @@ class TargetCodeGenInfo {
     // DO NOTHING by default.
     return false;
   }
+
+  /// Return an LLVM type that corresponds to an OpenCL type.
+  virtual llvm::Type *getOpenCLType(CodeGenModule &CGM, const Type *T) const {
+    return nullptr;
+  }
 };
 
 } // namespace CodeGen
diff --git a/clang/lib/CrossTU/CrossTranslationUnit.cpp b/clang/lib/CrossTU/CrossTranslationUnit.cpp
index c41da845be964..e84ef0a590e71 100644
--- a/clang/lib/CrossTU/CrossTranslationUnit.cpp
+++ b/clang/lib/CrossTU/CrossTranslationUnit.cpp
@@ -660,7 +660,7 @@ parseInvocationList(StringRef FileContent, llvm::sys::path::Style PathStyle) {
 
     StringRef InvocationKey = NativeSourcePath;
 
-    if (InvocationList.find(InvocationKey) != InvocationList.end())
+    if (InvocationList.contains(InvocationKey))
       return llvm::make_error<IndexError>(
           index_error_code::invocation_list_ambiguous);
 
diff --git a/clang/lib/Driver/Driver.cpp b/clang/lib/Driver/Driver.cpp
index 0567441225d0c..ead6252776a28 100644
--- a/clang/lib/Driver/Driver.cpp
+++ b/clang/lib/Driver/Driver.cpp
@@ -199,8 +199,9 @@ Driver::Driver(StringRef ClangExecutable, StringRef TargetTriple,
       ClangExecutable(ClangExecutable), SysRoot(DEFAULT_SYSROOT),
       DriverTitle(Title), CCCPrintBindings(false), CCPrintOptions(false),
       CCLogDiagnostics(false), CCGenDiagnostics(false),
-      CCPrintProcessStats(false), TargetTriple(TargetTriple), Saver(Alloc),
-      PrependArg(nullptr), CheckInputsExist(true), ProbePrecompiled(true),
+      CCPrintProcessStats(false), CCPrintInternalStats(false),
+      TargetTriple(TargetTriple), Saver(Alloc), PrependArg(nullptr),
+      CheckInputsExist(true), ProbePrecompiled(true),
       SuppressMissingInputWarning(false) {
   // Provide a sane fallback if no VFS is specified.
   if (!this->VFS)
@@ -594,13 +595,21 @@ static llvm::Triple computeTargetTriple(const Driver &D,
     }
   }
 
+  // The `-maix[32|64]` flags are only valid for AIX targets.
+  if (Arg *A = Args.getLastArgNoClaim(options::OPT_maix32, options::OPT_maix64);
+      A && !Target.isOSAIX())
+    D.Diag(diag::err_drv_unsupported_opt_for_target)
+        << A->getAsString(Args) << Target.str();
+
   // Handle pseudo-target flags '-m64', '-mx32', '-m32' and '-m16'.
   Arg *A = Args.getLastArg(options::OPT_m64, options::OPT_mx32,
-                           options::OPT_m32, options::OPT_m16);
+                           options::OPT_m32, options::OPT_m16,
+                           options::OPT_maix32, options::OPT_maix64);
   if (A) {
     llvm::Triple::ArchType AT = llvm::Triple::UnknownArch;
 
-    if (A->getOption().matches(options::OPT_m64)) {
+    if (A->getOption().matches(options::OPT_m64) ||
+        A->getOption().matches(options::OPT_maix64)) {
       AT = Target.get64BitArchVariant().getArch();
       if (Target.getEnvironment() == llvm::Triple::GNUX32)
         Target.setEnvironment(llvm::Triple::GNU);
@@ -613,7 +622,8 @@ static llvm::Triple computeTargetTriple(const Driver &D,
         Target.setEnvironment(llvm::Triple::MuslX32);
       else
         Target.setEnvironment(llvm::Triple::GNUX32);
-    } else if (A->getOption().matches(options::OPT_m32)) {
+    } else if (A->getOption().matches(options::OPT_m32) ||
+               A->getOption().matches(options::OPT_maix32)) {
       AT = Target.get32BitArchVariant().getArch();
       if (Target.getEnvironment() == llvm::Triple::GNUX32)
         Target.setEnvironment(llvm::Triple::GNU);
@@ -952,7 +962,7 @@ void Driver::CreateOffloadingDeviceToolChains(Compilation &C,
         } else
           TC = &getToolChain(C.getInputArgs(), TT);
         C.addOffloadDeviceToolChain(TC, Action::OFK_OpenMP);
-        if (DerivedArchs.find(TT.getTriple()) != DerivedArchs.end())
+        if (DerivedArchs.contains(TT.getTriple()))
           KnownArchs[TC] = DerivedArchs[TT.getTriple()];
       }
     }
@@ -4304,7 +4314,7 @@ Driver::getOffloadArchs(Compilation &C, const llvm::opt::DerivedArgList &Args,
                 : "--no-offload-arch");
   }
 
-  if (KnownArchs.find(TC) != KnownArchs.end())
+  if (KnownArchs.contains(TC))
     return KnownArchs.lookup(TC);
 
   llvm::DenseSet<StringRef> Archs;
@@ -5545,7 +5555,8 @@ static bool HasPreprocessOutput(const Action &JA) {
 
 const char *Driver::CreateTempFile(Compilation &C, StringRef Prefix,
                                    StringRef Suffix, bool MultipleArchs,
-                                   StringRef BoundArch) const {
+                                   StringRef BoundArch,
+                                   bool NeedUniqueDirectory) const {
   SmallString<128> TmpName;
   Arg *A = C.getArgs().getLastArg(options::OPT_fcrash_diagnostics_dir);
   std::optional<std::string> CrashDirectory =
@@ -5565,9 +5576,15 @@ const char *Driver::CreateTempFile(Compilation &C, StringRef Prefix,
     }
   } else {
     if (MultipleArchs && !BoundArch.empty()) {
-      TmpName = GetTemporaryDirectory(Prefix);
-      llvm::sys::path::append(TmpName,
-                              Twine(Prefix) + "-" + BoundArch + "." + Suffix);
+      if (NeedUniqueDirectory) {
+        TmpName = GetTemporaryDirectory(Prefix);
+        llvm::sys::path::append(TmpName,
+                                Twine(Prefix) + "-" + BoundArch + "." + Suffix);
+      } else {
+        TmpName =
+            GetTemporaryPath((Twine(Prefix) + "-" + BoundArch).str(), Suffix);
+      }
+
     } else {
       TmpName = GetTemporaryPath(Prefix, Suffix);
     }
@@ -5683,7 +5700,16 @@ const char *Driver::GetNamedOutputPath(Compilation &C, const JobAction &JA,
     StringRef Name = llvm::sys::path::filename(BaseInput);
     std::pair<StringRef, StringRef> Split = Name.split('.');
     const char *Suffix = types::getTypeTempSuffix(JA.getType(), IsCLMode());
-    return CreateTempFile(C, Split.first, Suffix, MultipleArchs, BoundArch);
+    // The non-offloading toolchain on Darwin requires deterministic input
+    // file name for binaries to be deterministic, therefore it needs unique
+    // directory.
+    llvm::Triple Triple(C.getDriver().getTargetTriple());
+    bool NeedUniqueDirectory =
+        (JA.getOffloadingDeviceKind() == Action::OFK_None ||
+         JA.getOffloadingDeviceKind() == Action::OFK_Host) &&
+        Triple.isOSDarwin();
+    return CreateTempFile(C, Split.first, Suffix, MultipleArchs, BoundArch,
+                          NeedUniqueDirectory);
   }
 
   SmallString<128> BasePath(BaseInput);
@@ -6409,3 +6435,58 @@ llvm::StringRef clang::driver::getDriverMode(StringRef ProgName,
 }
 
 bool driver::IsClangCL(StringRef DriverMode) { return DriverMode.equals("cl"); }
+
+llvm::Error driver::expandResponseFiles(SmallVectorImpl<const char *> &Args,
+                                        bool ClangCLMode,
+                                        llvm::BumpPtrAllocator &Alloc,
+                                        llvm::vfs::FileSystem *FS) {
+  // Parse response files using the GNU syntax, unless we're in CL mode. There
+  // are two ways to put clang in CL compatibility mode: ProgName is either
+  // clang-cl or cl, or --driver-mode=cl is on the command line. The normal
+  // command line parsing can't happen until after response file parsing, so we
+  // have to manually search for a --driver-mode=cl argument the hard way.
+  // Finally, our -cc1 tools don't care which tokenization mode we use because
+  // response files written by clang will tokenize the same way in either mode.
+  enum { Default, POSIX, Windows } RSPQuoting = Default;
+  for (const char *F : Args) {
+    if (strcmp(F, "--rsp-quoting=posix") == 0)
+      RSPQuoting = POSIX;
+    else if (strcmp(F, "--rsp-quoting=windows") == 0)
+      RSPQuoting = Windows;
+  }
+
+  // Determines whether we want nullptr markers in Args to indicate response
+  // files end-of-lines. We only use this for the /LINK driver argument with
+  // clang-cl.exe on Windows.
+  bool MarkEOLs = ClangCLMode;
+
+  llvm::cl::TokenizerCallback Tokenizer;
+  if (RSPQuoting == Windows || (RSPQuoting == Default && ClangCLMode))
+    Tokenizer = &llvm::cl::TokenizeWindowsCommandLine;
+  else
+    Tokenizer = &llvm::cl::TokenizeGNUCommandLine;
+
+  if (MarkEOLs && Args.size() > 1 && StringRef(Args[1]).startswith("-cc1"))
+    MarkEOLs = false;
+
+  llvm::cl::ExpansionContext ECtx(Alloc, Tokenizer);
+  ECtx.setMarkEOLs(MarkEOLs);
+  if (FS)
+    ECtx.setVFS(FS);
+
+  if (llvm::Error Err = ECtx.expandResponseFiles(Args))
+    return Err;
+
+  // If -cc1 came from a response file, remove the EOL sentinels.
+  auto FirstArg = llvm::find_if(llvm::drop_begin(Args),
+                                [](const char *A) { return A != nullptr; });
+  if (FirstArg != Args.end() && StringRef(*FirstArg).startswith("-cc1")) {
+    // If -cc1 came from a response file, remove the EOL sentinels.
+    if (MarkEOLs) {
+      auto newEnd = std::remove(Args.begin(), Args.end(), nullptr);
+      Args.resize(newEnd - Args.begin());
+    }
+  }
+
+  return llvm::Error::success();
+}
diff --git a/clang/lib/Driver/Multilib.cpp b/clang/lib/Driver/Multilib.cpp
index 1f8db753e8233..d1ab0c7b114e9 100644
--- a/clang/lib/Driver/Multilib.cpp
+++ b/clang/lib/Driver/Multilib.cpp
@@ -63,7 +63,7 @@ bool Multilib::operator==(const Multilib &Other) const {
     MyFlags.insert(Flag);
 
   for (const auto &Flag : Other.Flags)
-    if (MyFlags.find(Flag) == MyFlags.end())
+    if (!MyFlags.contains(Flag))
       return false;
 
   if (osSuffix() != Other.osSuffix())
diff --git a/clang/lib/Driver/OffloadBundler.cpp b/clang/lib/Driver/OffloadBundler.cpp
index 0145e0311145f..5163eb37c78c9 100644
--- a/clang/lib/Driver/OffloadBundler.cpp
+++ b/clang/lib/Driver/OffloadBundler.cpp
@@ -72,12 +72,22 @@ OffloadTargetInfo::OffloadTargetInfo(const StringRef Target,
   if (clang::StringToCudaArch(TripleOrGPU.second) != clang::CudaArch::UNKNOWN) {
     auto KindTriple = TripleOrGPU.first.split('-');
     this->OffloadKind = KindTriple.first;
-    this->Triple = llvm::Triple(KindTriple.second);
+
+    // Enforce optional env field to standardize bundles
+    llvm::Triple t = llvm::Triple(KindTriple.second);
+    this->Triple = llvm::Triple(t.getArchName(), t.getVendorName(),
+                                t.getOSName(), t.getEnvironmentName());
+
     this->TargetID = Target.substr(Target.find(TripleOrGPU.second));
   } else {
     auto KindTriple = TargetFeatures.first.split('-');
     this->OffloadKind = KindTriple.first;
-    this->Triple = llvm::Triple(KindTriple.second);
+
+    // Enforce optional env field to standardize bundles
+    llvm::Triple t = llvm::Triple(KindTriple.second);
+    this->Triple = llvm::Triple(t.getArchName(), t.getVendorName(),
+                                t.getOSName(), t.getEnvironmentName());
+
     this->TargetID = "";
   }
 }
@@ -395,8 +405,7 @@ class BinaryFileHandler final : public FileHandler {
       if (!Offset || Offset + Size > FC.size())
         return Error::success();
 
-      assert(BundlesInfo.find(Triple) == BundlesInfo.end() &&
-             "Triple is duplicated??");
+      assert(!BundlesInfo.contains(Triple) && "Triple is duplicated??");
       BundlesInfo[Triple] = BinaryBundleInfo(Size, Offset);
     }
     // Set the iterator to where we will start to read.
@@ -1222,8 +1231,7 @@ Error OffloadBundler::UnbundleArchive() {
 
           // For inserting <CompatibleTarget, list<CodeObject>> entry in
           // OutputArchivesMap.
-          if (OutputArchivesMap.find(CompatibleTarget) ==
-              OutputArchivesMap.end()) {
+          if (!OutputArchivesMap.contains(CompatibleTarget)) {
 
             std::vector<NewArchiveMember> ArchiveMembers;
             ArchiveMembers.push_back(NewArchiveMember(MemBufRef));
diff --git a/clang/lib/Driver/SanitizerArgs.cpp b/clang/lib/Driver/SanitizerArgs.cpp
index 390faef0dbdf1..299d4adf3333a 100644
--- a/clang/lib/Driver/SanitizerArgs.cpp
+++ b/clang/lib/Driver/SanitizerArgs.cpp
@@ -1100,13 +1100,16 @@ void SanitizerArgs::addArgs(const ToolChain &TC, const llvm::opt::ArgList &Args,
   // NVPTX doesn't currently support sanitizers.  Bailing out here means
   // that e.g. -fsanitize=address applies only to host code, which is what we
   // want for now.
-  //
-  // AMDGPU sanitizer support is experimental and controlled by -fgpu-sanitize.
-  if (TC.getTriple().isNVPTX() ||
-      (TC.getTriple().isAMDGPU() &&
-       !Args.hasFlag(options::OPT_fgpu_sanitize, options::OPT_fno_gpu_sanitize,
-                     true)))
+  if (TC.getTriple().isNVPTX())
     return;
+  // AMDGPU sanitizer support is experimental and controlled by -fgpu-sanitize.
+  bool GPUSanitize = false;
+  if (TC.getTriple().isAMDGPU()) {
+    if (!Args.hasFlag(options::OPT_fgpu_sanitize, options::OPT_fno_gpu_sanitize,
+                      true))
+      return;
+    GPUSanitize = true;
+  }
 
   // Translate available CoverageFeatures to corresponding clang-cc1 flags.
   // Do it even if Sanitizers.empty() since some forms of coverage don't require
@@ -1143,20 +1146,22 @@ void SanitizerArgs::addArgs(const ToolChain &TC, const llvm::opt::ArgList &Args,
   addSpecialCaseListOpt(Args, CmdArgs, "-fsanitize-coverage-ignorelist=",
                         CoverageIgnorelistFiles);
 
-  // Translate available BinaryMetadataFeatures to corresponding clang-cc1
-  // flags. Does not depend on any other sanitizers.
-  const std::pair<int, std::string> BinaryMetadataFlags[] = {
-      std::make_pair(BinaryMetadataCovered, "covered"),
-      std::make_pair(BinaryMetadataAtomics, "atomics"),
-      std::make_pair(BinaryMetadataUAR, "uar")};
-  for (const auto &F : BinaryMetadataFlags) {
-    if (BinaryMetadataFeatures & F.first)
-      CmdArgs.push_back(
-          Args.MakeArgString("-fexperimental-sanitize-metadata=" + F.second));
+  if (!GPUSanitize) {
+    // Translate available BinaryMetadataFeatures to corresponding clang-cc1
+    // flags. Does not depend on any other sanitizers. Unsupported on GPUs.
+    const std::pair<int, std::string> BinaryMetadataFlags[] = {
+        std::make_pair(BinaryMetadataCovered, "covered"),
+        std::make_pair(BinaryMetadataAtomics, "atomics"),
+        std::make_pair(BinaryMetadataUAR, "uar")};
+    for (const auto &F : BinaryMetadataFlags) {
+      if (BinaryMetadataFeatures & F.first)
+        CmdArgs.push_back(
+            Args.MakeArgString("-fexperimental-sanitize-metadata=" + F.second));
+    }
+    addSpecialCaseListOpt(Args, CmdArgs,
+                          "-fexperimental-sanitize-metadata-ignorelist=",
+                          BinaryMetadataIgnorelistFiles);
   }
-  addSpecialCaseListOpt(Args, CmdArgs,
-                        "-fexperimental-sanitize-metadata-ignorelist=",
-                        BinaryMetadataIgnorelistFiles);
 
   if (TC.getTriple().isOSWindows() && needsUbsanRt()) {
     // Instruct the code generator to embed linker directives in the object file
diff --git a/clang/lib/Driver/ToolChains/AIX.cpp b/clang/lib/Driver/ToolChains/AIX.cpp
index 15560e1c81bca..f41fb768fdc06 100644
--- a/clang/lib/Driver/ToolChains/AIX.cpp
+++ b/clang/lib/Driver/ToolChains/AIX.cpp
@@ -164,14 +164,14 @@ void aix::Linker::ConstructJob(Compilation &C, const JobAction &JA,
   }
 
   auto getCrt0Basename = [&Args, IsArch32Bit] {
-    // Enable gprofiling when "-pg" is specified.
-    if (Args.hasArg(options::OPT_pg))
-      return IsArch32Bit ? "gcrt0.o" : "gcrt0_64.o";
-    // Enable profiling when "-p" is specified.
-    else if (Args.hasArg(options::OPT_p))
+    if (Arg *A = Args.getLastArgNoClaim(options::OPT_p, options::OPT_pg)) {
+      // Enable gprofiling when "-pg" is specified.
+      if (A->getOption().matches(options::OPT_pg))
+        return IsArch32Bit ? "gcrt0.o" : "gcrt0_64.o";
+      // Enable profiling when "-p" is specified.
       return IsArch32Bit ? "mcrt0.o" : "mcrt0_64.o";
-    else
-      return IsArch32Bit ? "crt0.o" : "crt0_64.o";
+    }
+    return IsArch32Bit ? "crt0.o" : "crt0_64.o";
   };
 
   if (!Args.hasArg(options::OPT_nostdlib, options::OPT_nostartfiles,
@@ -271,7 +271,7 @@ void aix::Linker::ConstructJob(Compilation &C, const JobAction &JA,
 
     CmdArgs.push_back("-lc");
 
-    if (Args.hasArg(options::OPT_p, options::OPT_pg)) {
+    if (Args.hasArgNoClaim(options::OPT_p, options::OPT_pg)) {
       CmdArgs.push_back(Args.MakeArgString((llvm::Twine("-L") + D.SysRoot) +
                                            "/lib/profiled"));
       CmdArgs.push_back(Args.MakeArgString((llvm::Twine("-L") + D.SysRoot) +
diff --git a/clang/lib/Driver/ToolChains/AMDGPU.cpp b/clang/lib/Driver/ToolChains/AMDGPU.cpp
index 08496c1abbc41..99e2fb839221b 100644
--- a/clang/lib/Driver/ToolChains/AMDGPU.cpp
+++ b/clang/lib/Driver/ToolChains/AMDGPU.cpp
@@ -543,6 +543,7 @@ void amdgpu::Linker::ConstructJob(Compilation &C, const JobAction &JA,
   if (C.getDriver().isUsingLTO())
     addLTOOptions(getToolChain(), Args, CmdArgs, Output, Inputs[0],
                   C.getDriver().getLTOMode() == LTOK_Thin);
+  CmdArgs.push_back("--no-undefined");
   CmdArgs.push_back("-shared");
   CmdArgs.push_back("-o");
   CmdArgs.push_back(Output.getFilename());
diff --git a/clang/lib/Driver/ToolChains/Arch/ARM.cpp b/clang/lib/Driver/ToolChains/Arch/ARM.cpp
index e75f8a474410c..7843031a4c22f 100644
--- a/clang/lib/Driver/ToolChains/Arch/ARM.cpp
+++ b/clang/lib/Driver/ToolChains/Arch/ARM.cpp
@@ -72,25 +72,25 @@ static void getARMHWDivFeatures(const Driver &D, const Arg *A,
 }
 
 // Handle -mfpu=.
-static unsigned getARMFPUFeatures(const Driver &D, const Arg *A,
-                                  const ArgList &Args, StringRef FPU,
-                                  std::vector<StringRef> &Features) {
-  unsigned FPUID = llvm::ARM::parseFPU(FPU);
-  if (!llvm::ARM::getFPUFeatures(FPUID, Features))
+static llvm::ARM::FPUKind getARMFPUFeatures(const Driver &D, const Arg *A,
+                                            const ArgList &Args, StringRef FPU,
+                                            std::vector<StringRef> &Features) {
+  llvm::ARM::FPUKind FPUKind = llvm::ARM::parseFPU(FPU);
+  if (!llvm::ARM::getFPUFeatures(FPUKind, Features))
     D.Diag(clang::diag::err_drv_clang_unsupported) << A->getAsString(Args);
-  return FPUID;
+  return FPUKind;
 }
 
 // Decode ARM features from string like +[no]featureA+[no]featureB+...
 static bool DecodeARMFeatures(const Driver &D, StringRef text, StringRef CPU,
                               llvm::ARM::ArchKind ArchKind,
                               std::vector<StringRef> &Features,
-                              unsigned &ArgFPUID) {
+                              llvm::ARM::FPUKind &ArgFPUKind) {
   SmallVector<StringRef, 8> Split;
   text.split(Split, StringRef("+"), -1, false);
 
   for (StringRef Feature : Split) {
-    if (!appendArchExtFeatures(CPU, ArchKind, Feature, Features, ArgFPUID))
+    if (!appendArchExtFeatures(CPU, ArchKind, Feature, Features, ArgFPUKind))
       return false;
   }
   return true;
@@ -112,14 +112,16 @@ static void DecodeARMFeaturesFromCPU(const Driver &D, StringRef CPU,
 static void checkARMArchName(const Driver &D, const Arg *A, const ArgList &Args,
                              llvm::StringRef ArchName, llvm::StringRef CPUName,
                              std::vector<StringRef> &Features,
-                             const llvm::Triple &Triple, unsigned &ArgFPUID) {
+                             const llvm::Triple &Triple,
+                             llvm::ARM::FPUKind &ArgFPUKind) {
   std::pair<StringRef, StringRef> Split = ArchName.split("+");
 
   std::string MArch = arm::getARMArch(ArchName, Triple);
   llvm::ARM::ArchKind ArchKind = llvm::ARM::parseArch(MArch);
   if (ArchKind == llvm::ARM::ArchKind::INVALID ||
-      (Split.second.size() && !DecodeARMFeatures(D, Split.second, CPUName,
-                                                 ArchKind, Features, ArgFPUID)))
+      (Split.second.size() &&
+       !DecodeARMFeatures(D, Split.second, CPUName, ArchKind, Features,
+                          ArgFPUKind)))
     D.Diag(clang::diag::err_drv_unsupported_option_argument)
         << A->getSpelling() << A->getValue();
 }
@@ -128,15 +130,16 @@ static void checkARMArchName(const Driver &D, const Arg *A, const ArgList &Args,
 static void checkARMCPUName(const Driver &D, const Arg *A, const ArgList &Args,
                             llvm::StringRef CPUName, llvm::StringRef ArchName,
                             std::vector<StringRef> &Features,
-                            const llvm::Triple &Triple, unsigned &ArgFPUID) {
+                            const llvm::Triple &Triple,
+                            llvm::ARM::FPUKind &ArgFPUKind) {
   std::pair<StringRef, StringRef> Split = CPUName.split("+");
 
   std::string CPU = arm::getARMTargetCPU(CPUName, ArchName, Triple);
   llvm::ARM::ArchKind ArchKind =
     arm::getLLVMArchKindForARM(CPU, ArchName, Triple);
   if (ArchKind == llvm::ARM::ArchKind::INVALID ||
-      (Split.second.size() &&
-       !DecodeARMFeatures(D, Split.second, CPU, ArchKind, Features, ArgFPUID)))
+      (Split.second.size() && !DecodeARMFeatures(D, Split.second, CPU, ArchKind,
+                                                 Features, ArgFPUKind)))
     D.Diag(clang::diag::err_drv_unsupported_option_argument)
         << A->getSpelling() << A->getValue();
 }
@@ -498,8 +501,8 @@ void arm::getARMTargetFeatures(const Driver &D, const llvm::Triple &Triple,
   const Arg *CPUArg = Args.getLastArg(options::OPT_mcpu_EQ);
   StringRef ArchName;
   StringRef CPUName;
-  unsigned ArchArgFPUID = llvm::ARM::FK_INVALID;
-  unsigned CPUArgFPUID = llvm::ARM::FK_INVALID;
+  llvm::ARM::FPUKind ArchArgFPUKind = llvm::ARM::FK_INVALID;
+  llvm::ARM::FPUKind CPUArgFPUKind = llvm::ARM::FK_INVALID;
 
   // Check -mcpu. ClangAs gives preference to -Wa,-mcpu=.
   if (WaCPU) {
@@ -519,13 +522,13 @@ void arm::getARMTargetFeatures(const Driver &D, const llvm::Triple &Triple,
     ArchName = WaArch->second;
     // This will set any features after the base architecture.
     checkARMArchName(D, WaArch->first, Args, ArchName, CPUName,
-                     ExtensionFeatures, Triple, ArchArgFPUID);
+                     ExtensionFeatures, Triple, ArchArgFPUKind);
     // The base architecture was handled in ToolChain::ComputeLLVMTriple because
     // triple is read only by this point.
   } else if (ArchArg) {
     ArchName = ArchArg->getValue();
     checkARMArchName(D, ArchArg, Args, ArchName, CPUName, ExtensionFeatures,
-                     Triple, ArchArgFPUID);
+                     Triple, ArchArgFPUKind);
   }
 
   // Add CPU features for generic CPUs
@@ -545,14 +548,14 @@ void arm::getARMTargetFeatures(const Driver &D, const llvm::Triple &Triple,
 
   if (CPUArg)
     checkARMCPUName(D, CPUArg, Args, CPUName, ArchName, ExtensionFeatures,
-                    Triple, CPUArgFPUID);
+                    Triple, CPUArgFPUKind);
 
   // TODO Handle -mtune=. Suppress -Wunused-command-line-argument as a
   // longstanding behavior.
   (void)Args.getLastArg(options::OPT_mtune_EQ);
 
   // Honor -mfpu=. ClangAs gives preference to -Wa,-mfpu=.
-  unsigned FPUID = llvm::ARM::FK_INVALID;
+  llvm::ARM::FPUKind FPUKind = llvm::ARM::FK_INVALID;
   const Arg *FPUArg = Args.getLastArg(options::OPT_mfpu_EQ);
   if (WaFPU) {
     if (FPUArg)
@@ -560,11 +563,11 @@ void arm::getARMTargetFeatures(const Driver &D, const llvm::Triple &Triple,
           << FPUArg->getAsString(Args);
     (void)getARMFPUFeatures(D, WaFPU->first, Args, WaFPU->second, Features);
   } else if (FPUArg) {
-    FPUID = getARMFPUFeatures(D, FPUArg, Args, FPUArg->getValue(), Features);
+    FPUKind = getARMFPUFeatures(D, FPUArg, Args, FPUArg->getValue(), Features);
   } else if (Triple.isAndroid() && getARMSubArchVersionNumber(Triple) >= 7) {
     const char *AndroidFPU = "neon";
-    FPUID = llvm::ARM::parseFPU(AndroidFPU);
-    if (!llvm::ARM::getFPUFeatures(FPUID, Features))
+    FPUKind = llvm::ARM::parseFPU(AndroidFPU);
+    if (!llvm::ARM::getFPUFeatures(FPUKind, Features))
       D.Diag(clang::diag::err_drv_clang_unsupported)
           << std::string("-mfpu=") + AndroidFPU;
   } else {
@@ -572,8 +575,8 @@ void arm::getARMTargetFeatures(const Driver &D, const llvm::Triple &Triple,
       std::string CPU = arm::getARMTargetCPU(CPUName, ArchName, Triple);
       llvm::ARM::ArchKind ArchKind =
           arm::getLLVMArchKindForARM(CPU, ArchName, Triple);
-      FPUID = llvm::ARM::getDefaultFPU(CPU, ArchKind);
-      (void)llvm::ARM::getFPUFeatures(FPUID, Features);
+      FPUKind = llvm::ARM::getDefaultFPU(CPU, ArchKind);
+      (void)llvm::ARM::getFPUFeatures(FPUKind, Features);
     }
   }
 
@@ -631,9 +634,9 @@ void arm::getARMTargetFeatures(const Driver &D, const llvm::Triple &Triple,
     // above call.
     Features.insert(Features.end(), {"-dotprod", "-fp16fml", "-bf16", "-mve",
                                      "-mve.fp", "-fpregs"});
-  } else if (FPUID == llvm::ARM::FK_NONE ||
-             ArchArgFPUID == llvm::ARM::FK_NONE ||
-             CPUArgFPUID == llvm::ARM::FK_NONE) {
+  } else if (FPUKind == llvm::ARM::FK_NONE ||
+             ArchArgFPUKind == llvm::ARM::FK_NONE ||
+             CPUArgFPUKind == llvm::ARM::FK_NONE) {
     // -mfpu=none, -march=armvX+nofp or -mcpu=X+nofp is *very* similar to
     // -mfloat-abi=soft, only that it should not disable MVE-I. They disable the
     // FPU, but not the FPU registers, thus MVE-I, which depends only on the
diff --git a/clang/lib/Driver/ToolChains/Arch/PPC.cpp b/clang/lib/Driver/ToolChains/Arch/PPC.cpp
index 8824253cc9846..4287d41534693 100644
--- a/clang/lib/Driver/ToolChains/Arch/PPC.cpp
+++ b/clang/lib/Driver/ToolChains/Arch/PPC.cpp
@@ -85,7 +85,12 @@ std::string ppc::getPPCTuneCPU(const ArgList &Args, const llvm::Triple &T) {
 }
 
 /// Get the (LLVM) name of the PowerPC cpu we are targeting.
-std::string ppc::getPPCTargetCPU(const ArgList &Args, const llvm::Triple &T) {
+std::string ppc::getPPCTargetCPU(const Driver &D, const ArgList &Args,
+                                 const llvm::Triple &T) {
+  if (const Arg *A = Args.getLastArg(clang::driver::options::OPT_march_EQ)) {
+    D.Diag(diag::err_drv_unsupported_opt_for_target)
+        << A->getSpelling() << T.getTriple();
+  }
   if (Arg *A = Args.getLastArg(clang::driver::options::OPT_mcpu_EQ))
     return normalizeCPUName(A->getValue(), T);
   return getPPCGenericTargetCPU(T);
diff --git a/clang/lib/Driver/ToolChains/Arch/PPC.h b/clang/lib/Driver/ToolChains/Arch/PPC.h
index 97ac450838521..ec5b3c8140b66 100644
--- a/clang/lib/Driver/ToolChains/Arch/PPC.h
+++ b/clang/lib/Driver/ToolChains/Arch/PPC.h
@@ -35,7 +35,7 @@ enum class ReadGOTPtrMode {
 
 FloatABI getPPCFloatABI(const Driver &D, const llvm::opt::ArgList &Args);
 
-std::string getPPCTargetCPU(const llvm::opt::ArgList &Args,
+std::string getPPCTargetCPU(const Driver &D, const llvm::opt::ArgList &Args,
                             const llvm::Triple &T);
 std::string getPPCTuneCPU(const llvm::opt::ArgList &Args,
                           const llvm::Triple &T);
diff --git a/clang/lib/Driver/ToolChains/BareMetal.cpp b/clang/lib/Driver/ToolChains/BareMetal.cpp
index 3175b15a48ca7..c935758adb2ba 100644
--- a/clang/lib/Driver/ToolChains/BareMetal.cpp
+++ b/clang/lib/Driver/ToolChains/BareMetal.cpp
@@ -339,6 +339,9 @@ void baremetal::Linker::ConstructJob(Compilation &C, const JobAction &JA,
     TC.AddLinkRuntimeLib(Args, CmdArgs);
   }
 
+  if (TC.getTriple().isRISCV())
+    CmdArgs.push_back("-X");
+
   CmdArgs.push_back("-o");
   CmdArgs.push_back(Output.getFilename());
 
diff --git a/clang/lib/Driver/ToolChains/Clang.cpp b/clang/lib/Driver/ToolChains/Clang.cpp
index 79fcf4526df3e..b6d57f077e554 100644
--- a/clang/lib/Driver/ToolChains/Clang.cpp
+++ b/clang/lib/Driver/ToolChains/Clang.cpp
@@ -1993,7 +1993,7 @@ void Clang::AddMIPSTargetArgs(const ArgList &Args,
 void Clang::AddPPCTargetArgs(const ArgList &Args,
                              ArgStringList &CmdArgs) const {
   const llvm::Triple &T = getToolChain().getTriple();
-  if (const Arg *A = Args.getLastArg(options::OPT_mtune_EQ)) {
+  if (Args.getLastArg(options::OPT_mtune_EQ)) {
     CmdArgs.push_back("-tune-cpu");
     std::string CPU = ppc::getPPCTuneCPU(Args, T);
     CmdArgs.push_back(Args.MakeArgString(CPU));
@@ -5560,22 +5560,6 @@ void Clang::ConstructJob(Compilation &C, const JobAction &JA,
 
   RenderTargetOptions(Triple, Args, KernelOrKext, CmdArgs);
 
-  // FIXME: For now we want to demote any errors to warnings, when they have
-  // been raised for asking the wrong question of scalable vectors, such as
-  // asking for the fixed number of elements. This may happen because code that
-  // is not yet ported to work for scalable vectors uses the wrong interfaces,
-  // whereas the behaviour is actually correct. Emitting a warning helps bring
-  // up scalable vector support in an incremental way. When scalable vector
-  // support is stable enough, all uses of wrong interfaces should be considered
-  // as errors, but until then, we can live with a warning being emitted by the
-  // compiler. This way, Clang can be used to compile code with scalable vectors
-  // and identify possible issues.
-  if (isa<AssembleJobAction>(JA) || isa<CompileJobAction>(JA) ||
-      isa<BackendJobAction>(JA)) {
-    CmdArgs.push_back("-mllvm");
-    CmdArgs.push_back("-treat-scalable-fixed-error-as-warning");
-  }
-
   // These two are potentially updated by AddClangCLArgs.
   codegenoptions::DebugInfoKind DebugInfoKind = codegenoptions::NoDebugInfo;
   bool EmitCodeView = false;
@@ -6277,7 +6261,8 @@ void Clang::ConstructJob(Compilation &C, const JobAction &JA,
   if (Arg *A = Args.getLastArg(options::OPT_fpatchable_function_entry_EQ)) {
     StringRef S0 = A->getValue(), S = S0;
     unsigned Size, Offset = 0;
-    if (!Triple.isAArch64() && !Triple.isRISCV() && !Triple.isX86())
+    if (!Triple.isAArch64() && !Triple.isLoongArch() && !Triple.isRISCV() &&
+        !Triple.isX86())
       D.Diag(diag::err_drv_unsupported_opt_for_target)
           << A->getAsString(Args) << TripleStr;
     else if (S.consumeInteger(10, Size) ||
@@ -6322,20 +6307,26 @@ void Clang::ConstructJob(Compilation &C, const JobAction &JA,
             << A->getAsString(Args) << TripleStr;
     }
   }
-  if (Arg *A = Args.getLastArgNoClaim(options::OPT_p)) {
-    if (TC.getTriple().isOSAIX()) {
-      CmdArgs.push_back("-pg");
-    } else if (!TC.getTriple().isOSOpenBSD()) {
+
+  if (Arg *A = Args.getLastArgNoClaim(options::OPT_pg)) {
+    if (TC.getTriple().isOSzOS()) {
       D.Diag(diag::err_drv_unsupported_opt_for_target)
           << A->getAsString(Args) << TripleStr;
     }
   }
-  if (Arg *A = Args.getLastArgNoClaim(options::OPT_pg)) {
-    if (TC.getTriple().isOSzOS()) {
+  if (Arg *A = Args.getLastArgNoClaim(options::OPT_p)) {
+    if (!(TC.getTriple().isOSAIX() || TC.getTriple().isOSOpenBSD())) {
       D.Diag(diag::err_drv_unsupported_opt_for_target)
           << A->getAsString(Args) << TripleStr;
     }
   }
+  if (Arg *A = Args.getLastArgNoClaim(options::OPT_p, options::OPT_pg)) {
+    if (A->getOption().matches(options::OPT_p)) {
+      A->claim();
+      if (TC.getTriple().isOSAIX() && !Args.hasArgNoClaim(options::OPT_pg))
+        CmdArgs.push_back("-pg");
+    }
+  }
 
   if (Args.getLastArg(options::OPT_fapple_kext) ||
       (Args.hasArg(options::OPT_mkernel) && types::isCXX(InputType)))
@@ -7032,8 +7023,11 @@ void Clang::ConstructJob(Compilation &C, const JobAction &JA,
 
   // Setup statistics file output.
   SmallString<128> StatsFile = getStatsFileName(Args, Output, Input, D);
-  if (!StatsFile.empty())
+  if (!StatsFile.empty()) {
     CmdArgs.push_back(Args.MakeArgString(Twine("-stats-file=") + StatsFile));
+    if (D.CCPrintInternalStats)
+      CmdArgs.push_back("-stats-file-append");
+  }
 
   // Forward -Xclang arguments to -cc1, and -mllvm arguments to the LLVM option
   // parser.
diff --git a/clang/lib/Driver/ToolChains/CommonArgs.cpp b/clang/lib/Driver/ToolChains/CommonArgs.cpp
index c179c4152cee4..774a1843fa2c8 100644
--- a/clang/lib/Driver/ToolChains/CommonArgs.cpp
+++ b/clang/lib/Driver/ToolChains/CommonArgs.cpp
@@ -410,7 +410,7 @@ std::string tools::getCPUName(const Driver &D, const ArgList &Args,
   case llvm::Triple::ppcle:
   case llvm::Triple::ppc64:
   case llvm::Triple::ppc64le:
-    return ppc::getPPCTargetCPU(Args, T);
+    return ppc::getPPCTargetCPU(D, Args, T);
 
   case llvm::Triple::csky:
     if (const Arg *A = Args.getLastArg(options::OPT_mcpu_EQ))
@@ -795,22 +795,6 @@ void tools::addLTOOptions(const ToolChain &ToolChain, const ArgList &Args,
                          /*IsLTO=*/true, PluginOptPrefix);
 }
 
-void tools::addOpenMPRuntimeSpecificRPath(const ToolChain &TC,
-                                          const ArgList &Args,
-                                          ArgStringList &CmdArgs) {
-
-  if (Args.hasFlag(options::OPT_fopenmp_implicit_rpath,
-                   options::OPT_fno_openmp_implicit_rpath, true)) {
-    // Default to clang lib / lib64 folder, i.e. the same location as device
-    // runtime
-    SmallString<256> DefaultLibPath =
-        llvm::sys::path::parent_path(TC.getDriver().Dir);
-    llvm::sys::path::append(DefaultLibPath, CLANG_INSTALL_LIBDIR_BASENAME);
-    CmdArgs.push_back("-rpath");
-    CmdArgs.push_back(Args.MakeArgString(DefaultLibPath));
-  }
-}
-
 void tools::addOpenMPRuntimeLibraryPath(const ToolChain &TC,
                                         const ArgList &Args,
                                         ArgStringList &CmdArgs) {
@@ -881,9 +865,6 @@ bool tools::addOpenMPRuntime(ArgStringList &CmdArgs, const ToolChain &TC,
     CmdArgs.push_back("-lomptarget.devicertl");
 
   addArchSpecificRPath(TC, Args, CmdArgs);
-
-  if (RTKind == Driver::OMPRT_OMP)
-    addOpenMPRuntimeSpecificRPath(TC, Args, CmdArgs);
   addOpenMPRuntimeLibraryPath(TC, Args, CmdArgs);
 
   return true;
@@ -1785,22 +1766,29 @@ SmallString<128> tools::getStatsFileName(const llvm::opt::ArgList &Args,
                                          const InputInfo &Input,
                                          const Driver &D) {
   const Arg *A = Args.getLastArg(options::OPT_save_stats_EQ);
-  if (!A)
+  if (!A && !D.CCPrintInternalStats)
     return {};
 
-  StringRef SaveStats = A->getValue();
   SmallString<128> StatsFile;
-  if (SaveStats == "obj" && Output.isFilename()) {
-    StatsFile.assign(Output.getFilename());
-    llvm::sys::path::remove_filename(StatsFile);
-  } else if (SaveStats != "cwd") {
-    D.Diag(diag::err_drv_invalid_value) << A->getAsString(Args) << SaveStats;
-    return {};
-  }
+  if (A) {
+    StringRef SaveStats = A->getValue();
+    if (SaveStats == "obj" && Output.isFilename()) {
+      StatsFile.assign(Output.getFilename());
+      llvm::sys::path::remove_filename(StatsFile);
+    } else if (SaveStats != "cwd") {
+      D.Diag(diag::err_drv_invalid_value) << A->getAsString(Args) << SaveStats;
+      return {};
+    }
 
-  StringRef BaseName = llvm::sys::path::filename(Input.getBaseInput());
-  llvm::sys::path::append(StatsFile, BaseName);
-  llvm::sys::path::replace_extension(StatsFile, "stats");
+    StringRef BaseName = llvm::sys::path::filename(Input.getBaseInput());
+    llvm::sys::path::append(StatsFile, BaseName);
+    llvm::sys::path::replace_extension(StatsFile, "stats");
+  } else {
+    assert(D.CCPrintInternalStats);
+    StatsFile.assign(D.CCPrintInternalStatReportFilename.empty()
+                         ? "-"
+                         : D.CCPrintInternalStatReportFilename);
+  }
   return StatsFile;
 }
 
diff --git a/clang/lib/Driver/ToolChains/CommonArgs.h b/clang/lib/Driver/ToolChains/CommonArgs.h
index d44d9214c08bc..e64e9524601dd 100644
--- a/clang/lib/Driver/ToolChains/CommonArgs.h
+++ b/clang/lib/Driver/ToolChains/CommonArgs.h
@@ -117,9 +117,6 @@ void AddAssemblerKPIC(const ToolChain &ToolChain,
                       const llvm::opt::ArgList &Args,
                       llvm::opt::ArgStringList &CmdArgs);
 
-void addOpenMPRuntimeSpecificRPath(const ToolChain &TC,
-                                   const llvm::opt::ArgList &Args,
-                                   llvm::opt::ArgStringList &CmdArgs);
 void addArchSpecificRPath(const ToolChain &TC, const llvm::opt::ArgList &Args,
                           llvm::opt::ArgStringList &CmdArgs);
 void addOpenMPRuntimeLibraryPath(const ToolChain &TC,
diff --git a/clang/lib/Driver/ToolChains/Darwin.cpp b/clang/lib/Driver/ToolChains/Darwin.cpp
index b273f8ba04fda..6cee9c74c7adb 100644
--- a/clang/lib/Driver/ToolChains/Darwin.cpp
+++ b/clang/lib/Driver/ToolChains/Darwin.cpp
@@ -1426,24 +1426,42 @@ void DarwinClang::AddLinkRuntimeLibArgs(const ArgList &Args,
 
   const SanitizerArgs &Sanitize = getSanitizerArgs(Args);
 
-  if (!Sanitize.needsSharedRt() && Sanitize.needsUbsanRt()) {
-    getDriver().Diag(diag::err_drv_unsupported_static_ubsan_darwin);
-    return;
+  if (!Sanitize.needsSharedRt()) {
+    const char *sanitizer = nullptr;
+    if (Sanitize.needsUbsanRt()) {
+      sanitizer = "UndefinedBehaviorSanitizer";
+    } else if (Sanitize.needsAsanRt()) {
+      sanitizer = "AddressSanitizer";
+    } else if (Sanitize.needsTsanRt()) {
+      sanitizer = "ThreadSanitizer";
+    }
+    if (sanitizer) {
+      getDriver().Diag(diag::err_drv_unsupported_static_sanitizer_darwin)
+          << sanitizer;
+      return;
+    }
   }
 
   if (Sanitize.linkRuntimes()) {
-    if (Sanitize.needsAsanRt())
+    if (Sanitize.needsAsanRt()) {
+      assert(Sanitize.needsSharedRt() &&
+             "Static sanitizer runtimes not supported");
       AddLinkSanitizerLibArgs(Args, CmdArgs, "asan");
+    }
     if (Sanitize.needsLsanRt())
       AddLinkSanitizerLibArgs(Args, CmdArgs, "lsan");
     if (Sanitize.needsUbsanRt()) {
-      assert(Sanitize.needsSharedRt() && "Static sanitizer runtimes not supported");
-      AddLinkSanitizerLibArgs(Args, CmdArgs,
-                              Sanitize.requiresMinimalRuntime() ? "ubsan_minimal"
-                                                                : "ubsan");
+      assert(Sanitize.needsSharedRt() &&
+             "Static sanitizer runtimes not supported");
+      AddLinkSanitizerLibArgs(
+          Args, CmdArgs,
+          Sanitize.requiresMinimalRuntime() ? "ubsan_minimal" : "ubsan");
     }
-    if (Sanitize.needsTsanRt())
+    if (Sanitize.needsTsanRt()) {
+      assert(Sanitize.needsSharedRt() &&
+             "Static sanitizer runtimes not supported");
       AddLinkSanitizerLibArgs(Args, CmdArgs, "tsan");
+    }
     if (Sanitize.needsFuzzer() && !Args.hasArg(options::OPT_dynamiclib)) {
       AddLinkSanitizerLibArgs(Args, CmdArgs, "fuzzer", /*shared=*/false);
 
diff --git a/clang/lib/Driver/ToolChains/Flang.cpp b/clang/lib/Driver/ToolChains/Flang.cpp
index b50d2c5c4f8d2..0a4a0de99b89f 100644
--- a/clang/lib/Driver/ToolChains/Flang.cpp
+++ b/clang/lib/Driver/ToolChains/Flang.cpp
@@ -105,7 +105,7 @@ void Flang::addTargetOptions(const ArgList &Args,
   default:
     break;
   case llvm::Triple::aarch64:
-    [[fallthrough]];
+  case llvm::Triple::riscv64:
   case llvm::Triple::x86_64:
     getTargetFeatures(D, Triple, Args, CmdArgs, /*ForAs*/ false);
     break;
@@ -114,6 +114,35 @@ void Flang::addTargetOptions(const ArgList &Args,
   // TODO: Add target specific flags, ABI, mtune option etc.
 }
 
+void Flang::addOffloadOptions(Compilation &C, const InputInfoList &Inputs,
+                              const JobAction &JA, const ArgList &Args,
+                              ArgStringList &CmdArgs) const {
+  bool IsOpenMPDevice = JA.isDeviceOffloading(Action::OFK_OpenMP);
+  bool IsHostOffloadingAction = JA.isHostOffloading(Action::OFK_OpenMP) ||
+                                JA.isHostOffloading(C.getActiveOffloadKinds());
+
+  // Skips the primary input file, which is the input file that the compilation
+  // proccess will be executed upon (e.g. the host bitcode file) and
+  // adds the other secondary input (e.g. device bitcode files for embedding)
+  // to the embed offload object. This is condensed logic from the Clang driver
+  // for embedding offload objects during HostOffloading.
+  if (IsHostOffloadingAction) {
+    for (size_t i = 1; i < Inputs.size(); ++i) {
+      if (Inputs[i].getType() != types::TY_Nothing)
+        CmdArgs.push_back(
+            Args.MakeArgString("-fembed-offload-object=" +
+                               getToolChain().getInputFilename(Inputs[i])));
+    }
+  }
+
+  if (IsOpenMPDevice) {
+    // -fopenmp-is-device is passed along to tell the frontend that it is
+    // generating code for a device, so that only the relevant code is
+    // emitted.
+    CmdArgs.push_back("-fopenmp-is-device");
+  }
+}
+
 static void addFloatingPointOptions(const Driver &D, const ArgList &Args,
                                     ArgStringList &CmdArgs) {
   StringRef FPContract;
@@ -252,6 +281,7 @@ void Flang::ConstructJob(Compilation &C, const JobAction &JA,
 
   const Driver &D = TC.getDriver();
   ArgStringList CmdArgs;
+  DiagnosticsEngine &Diags = D.getDiags();
 
   // Invoke ourselves in -fc1 mode.
   CmdArgs.push_back("-fc1");
@@ -299,9 +329,21 @@ void Flang::ConstructJob(Compilation &C, const JobAction &JA,
   // to avoid warn_drv_unused_argument.
   Args.getLastArg(options::OPT_fcolor_diagnostics,
                   options::OPT_fno_color_diagnostics);
-  if (D.getDiags().getDiagnosticOptions().ShowColors)
+  if (Diags.getDiagnosticOptions().ShowColors)
     CmdArgs.push_back("-fcolor-diagnostics");
 
+  // LTO mode is parsed by the Clang driver library.
+  LTOKind LTOMode = D.getLTOMode(/* IsOffload */ false);
+  assert(LTOMode != LTOK_Unknown && "Unknown LTO mode.");
+  if (LTOMode == LTOK_Full)
+    CmdArgs.push_back("-flto=full");
+  else if (LTOMode == LTOK_Thin) {
+    Diags.Report(
+        Diags.getCustomDiagID(DiagnosticsEngine::Warning,
+                              "the option '-flto=thin' is a work in progress"));
+    CmdArgs.push_back("-flto=thin");
+  }
+
   // -fPIC and related options.
   addPicOptions(Args, CmdArgs);
 
@@ -314,6 +356,9 @@ void Flang::ConstructJob(Compilation &C, const JobAction &JA,
   // Add other compile options
   addOtherOptions(Args, CmdArgs);
 
+  // Offloading related options
+  addOffloadOptions(C, Inputs, JA, Args, CmdArgs);
+
   // Forward -Xflang arguments to -fc1
   Args.AddAllArgValues(CmdArgs, options::OPT_Xflang);
 
diff --git a/clang/lib/Driver/ToolChains/Flang.h b/clang/lib/Driver/ToolChains/Flang.h
index 4c85c602e2673..0dc3cb7eeadca 100644
--- a/clang/lib/Driver/ToolChains/Flang.h
+++ b/clang/lib/Driver/ToolChains/Flang.h
@@ -56,6 +56,17 @@ class LLVM_LIBRARY_VISIBILITY Flang : public Tool {
   void addTargetOptions(const llvm::opt::ArgList &Args,
                         llvm::opt::ArgStringList &CmdArgs) const;
 
+  /// Extract offload options from the driver arguments and add them to
+  /// the command arguments.
+  /// \param [in] C The current compilation for the driver invocation
+  /// \param [in] Inputs The input infomration on the current file inputs
+  /// \param [in] JA The job action
+  /// \param [in] Args The list of input driver arguments
+  /// \param [out] CmdArgs The list of output command arguments
+  void addOffloadOptions(Compilation &C, const InputInfoList &Inputs,
+                         const JobAction &JA, const llvm::opt::ArgList &Args,
+                         llvm::opt::ArgStringList &CmdArgs) const;
+
   /// Extract other compilation options from the driver arguments and add them
   /// to the command arguments.
   ///
diff --git a/clang/lib/Driver/ToolChains/FreeBSD.cpp b/clang/lib/Driver/ToolChains/FreeBSD.cpp
index 4ce0087cbe245..13b3654a1bce0 100644
--- a/clang/lib/Driver/ToolChains/FreeBSD.cpp
+++ b/clang/lib/Driver/ToolChains/FreeBSD.cpp
@@ -85,16 +85,7 @@ void freebsd::Assembler::ConstructJob(Compilation &C, const JobAction &JA,
     else
       CmdArgs.push_back("-mfpu=softvfp");
 
-    switch (getToolChain().getTriple().getEnvironment()) {
-    case llvm::Triple::GNUEABIHF:
-    case llvm::Triple::GNUEABI:
-    case llvm::Triple::EABI:
-      CmdArgs.push_back("-meabi=5");
-      break;
-
-    default:
-      CmdArgs.push_back("-matpcs");
-    }
+    CmdArgs.push_back("-meabi=5");
     break;
   }
   case llvm::Triple::sparc:
@@ -467,21 +458,6 @@ Tool *FreeBSD::buildAssembler() const {
 
 Tool *FreeBSD::buildLinker() const { return new tools::freebsd::Linker(*this); }
 
-llvm::ExceptionHandling FreeBSD::GetExceptionModel(const ArgList &Args) const {
-  // FreeBSD uses SjLj exceptions on ARM oabi.
-  switch (getTriple().getEnvironment()) {
-  case llvm::Triple::GNUEABIHF:
-  case llvm::Triple::GNUEABI:
-  case llvm::Triple::EABI:
-    return llvm::ExceptionHandling::None;
-  default:
-    if (getTriple().getArch() == llvm::Triple::arm ||
-        getTriple().getArch() == llvm::Triple::thumb)
-      return llvm::ExceptionHandling::SjLj;
-    return llvm::ExceptionHandling::None;
-  }
-}
-
 bool FreeBSD::HasNativeLLVMSupport() const { return true; }
 
 ToolChain::UnwindTableLevel
diff --git a/clang/lib/Driver/ToolChains/FreeBSD.h b/clang/lib/Driver/ToolChains/FreeBSD.h
index 9b24ef1a1e1cf..cec67d84a2cea 100644
--- a/clang/lib/Driver/ToolChains/FreeBSD.h
+++ b/clang/lib/Driver/ToolChains/FreeBSD.h
@@ -78,8 +78,6 @@ class LLVM_LIBRARY_VISIBILITY FreeBSD : public Generic_ELF {
   void AddHIPIncludeArgs(const llvm::opt::ArgList &DriverArgs,
                          llvm::opt::ArgStringList &CC1Args) const override;
 
-  llvm::ExceptionHandling
-  GetExceptionModel(const llvm::opt::ArgList &Args) const override;
   UnwindTableLevel
   getDefaultUnwindTableLevel(const llvm::opt::ArgList &Args) const override;
   bool isPIEDefault(const llvm::opt::ArgList &Args) const override;
diff --git a/clang/lib/Driver/ToolChains/Linux.cpp b/clang/lib/Driver/ToolChains/Linux.cpp
index e8890af0ab9ac..848d7247c20c0 100644
--- a/clang/lib/Driver/ToolChains/Linux.cpp
+++ b/clang/lib/Driver/ToolChains/Linux.cpp
@@ -694,8 +694,8 @@ void Linux::AddHIPRuntimeLibArgs(const ArgList &Args,
   CmdArgs.push_back(
       Args.MakeArgString(StringRef("-L") + RocmInstallation->getLibPath()));
 
-  if (Args.hasFlag(options::OPT_offload_add_rpath,
-                   options::OPT_no_offload_add_rpath, false))
+  if (Args.hasFlag(options::OPT_frtlib_add_rpath,
+                   options::OPT_fno_rtlib_add_rpath, false))
     CmdArgs.append(
         {"-rpath", Args.MakeArgString(RocmInstallation->getLibPath())});
 
@@ -776,7 +776,7 @@ SanitizerMask Linux::getSupportedSanitizers() const {
   if (IsX86 || IsX86_64)
     Res |= SanitizerKind::Function;
   if (IsX86_64 || IsMIPS64 || IsAArch64 || IsX86 || IsMIPS || IsArmArch ||
-      IsPowerPC64 || IsHexagon || IsLoongArch64)
+      IsPowerPC64 || IsHexagon || IsLoongArch64 || IsRISCV64)
     Res |= SanitizerKind::Scudo;
   if (IsX86_64 || IsAArch64 || IsRISCV64) {
     Res |= SanitizerKind::HWAddress;
diff --git a/clang/lib/Driver/ToolChains/MSVC.cpp b/clang/lib/Driver/ToolChains/MSVC.cpp
index 6f7db528e23ae..b23d0aa2f1b83 100644
--- a/clang/lib/Driver/ToolChains/MSVC.cpp
+++ b/clang/lib/Driver/ToolChains/MSVC.cpp
@@ -440,8 +440,8 @@ MSVCToolChain::MSVCToolChain(const Driver &D, const llvm::Triple &Triple,
                                       WinSysRoot, VCToolChainPath, VSLayout) ||
       llvm::findVCToolChainViaEnvironment(getVFS(), VCToolChainPath,
                                           VSLayout) ||
-      llvm::findVCToolChainViaSetupConfig(getVFS(), VCToolChainPath,
-                                          VSLayout) ||
+      llvm::findVCToolChainViaSetupConfig(getVFS(), VCToolsVersion,
+                                          VCToolChainPath, VSLayout) ||
       llvm::findVCToolChainViaRegistry(VCToolChainPath, VSLayout);
 }
 
@@ -546,6 +546,10 @@ bool MSVCToolChain::getWindowsSDKLibraryPath(const ArgList &Args,
 
   llvm::SmallString<128> libPath(sdkPath);
   llvm::sys::path::append(libPath, "Lib");
+  if (sdkMajor >= 10)
+    if (!(WinSdkDir.has_value() || WinSysRoot.has_value()) &&
+        WinSdkVersion.has_value())
+      windowsSDKLibVersion = *WinSdkVersion;
   if (sdkMajor >= 8)
     llvm::sys::path::append(libPath, windowsSDKLibVersion, "um");
   return llvm::appendArchToWindowsSDKLibPath(sdkMajor, libPath, getArch(),
@@ -567,6 +571,10 @@ bool MSVCToolChain::getUniversalCRTLibraryPath(const ArgList &Args,
                                    UCRTVersion))
     return false;
 
+  if (!(WinSdkDir.has_value() || WinSysRoot.has_value()) &&
+      WinSdkVersion.has_value())
+    UCRTVersion = *WinSdkVersion;
+
   StringRef ArchName = llvm::archToWindowsSDKArch(getArch());
   if (ArchName.empty())
     return false;
@@ -696,6 +704,9 @@ void MSVCToolChain::AddClangSystemIncludeArgs(const ArgList &DriverArgs,
       if (llvm::getUniversalCRTSdkDir(getVFS(), WinSdkDir, WinSdkVersion,
                                       WinSysRoot, UniversalCRTSdkPath,
                                       UCRTVersion)) {
+        if (!(WinSdkDir.has_value() || WinSysRoot.has_value()) &&
+            WinSdkVersion.has_value())
+          UCRTVersion = *WinSdkVersion;
         AddSystemIncludeWithSubfolder(DriverArgs, CC1Args, UniversalCRTSdkPath,
                                       "Include", UCRTVersion, "ucrt");
       }
@@ -708,6 +719,10 @@ void MSVCToolChain::AddClangSystemIncludeArgs(const ArgList &DriverArgs,
     if (llvm::getWindowsSDKDir(getVFS(), WinSdkDir, WinSdkVersion, WinSysRoot,
                                WindowsSDKDir, major, windowsSDKIncludeVersion,
                                windowsSDKLibVersion)) {
+      if (major >= 10)
+        if (!(WinSdkDir.has_value() || WinSysRoot.has_value()) &&
+            WinSdkVersion.has_value())
+          windowsSDKIncludeVersion = windowsSDKLibVersion = *WinSdkVersion;
       if (major >= 8) {
         // Note: windowsSDKIncludeVersion is empty for SDKs prior to v10.
         // Anyway, llvm::sys::path::append is able to manage it.
diff --git a/clang/lib/Driver/ToolChains/PS4CPU.cpp b/clang/lib/Driver/ToolChains/PS4CPU.cpp
index fc5d46bc25605..b3e14618f307e 100644
--- a/clang/lib/Driver/ToolChains/PS4CPU.cpp
+++ b/clang/lib/Driver/ToolChains/PS4CPU.cpp
@@ -304,19 +304,6 @@ void toolchains::PS4PS5Base::AddClangSystemIncludeArgs(
   if (DriverArgs.hasArg(options::OPT_nostdlibinc))
     return;
 
-  // Add dirs specified via 'configure --with-c-include-dirs'.
-  StringRef CIncludeDirs(C_INCLUDE_DIRS);
-  if (!CIncludeDirs.empty()) {
-    SmallVector<StringRef, 5> dirs;
-    CIncludeDirs.split(dirs, ":");
-    for (StringRef dir : dirs) {
-      StringRef Prefix =
-        llvm::sys::path::is_absolute(dir) ? StringRef(D.SysRoot) : "";
-      addExternCSystemInclude(DriverArgs, CC1Args, Prefix + dir);
-    }
-    return;
-  }
-
   addExternCSystemInclude(DriverArgs, CC1Args,
                           SDKRootDir + "/target/include");
   addExternCSystemInclude(DriverArgs, CC1Args,
diff --git a/clang/lib/Driver/Types.cpp b/clang/lib/Driver/Types.cpp
index a890cc58ee421..7d6308d757bc7 100644
--- a/clang/lib/Driver/Types.cpp
+++ b/clang/lib/Driver/Types.cpp
@@ -331,6 +331,10 @@ types::ID types::lookupTypeForExtension(llvm::StringRef Ext) {
       .Case("cui", TY_PP_CUDA)
       .Case("cxx", TY_CXX)
       .Case("CXX", TY_CXX)
+      .Case("F03", TY_Fortran)
+      .Case("f03", TY_PP_Fortran)
+      .Case("F08", TY_Fortran)
+      .Case("f08", TY_PP_Fortran)
       .Case("F90", TY_Fortran)
       .Case("f90", TY_PP_Fortran)
       .Case("F95", TY_Fortran)
diff --git a/clang/lib/ExtractAPI/APIIgnoresList.cpp b/clang/lib/ExtractAPI/APIIgnoresList.cpp
index 1d65ae2b8e31b..d6bbc6692d2bc 100644
--- a/clang/lib/ExtractAPI/APIIgnoresList.cpp
+++ b/clang/lib/ExtractAPI/APIIgnoresList.cpp
@@ -31,20 +31,29 @@ std::error_code IgnoresFileNotFound::convertToErrorCode() const {
   return llvm::inconvertibleErrorCode();
 }
 
-Expected<APIIgnoresList> APIIgnoresList::create(StringRef IgnoresFilePath,
-                                                FileManager &FM) {
-  auto BufferOrErr = FM.getBufferForFile(IgnoresFilePath);
-  if (!BufferOrErr)
-    return make_error<IgnoresFileNotFound>(IgnoresFilePath);
-
-  auto Buffer = std::move(BufferOrErr.get());
+Expected<APIIgnoresList>
+APIIgnoresList::create(const FilePathList &IgnoresFilePathList,
+                       FileManager &FM) {
   SmallVector<StringRef, 32> Lines;
-  Buffer->getBuffer().split(Lines, '\n', /*MaxSplit*/ -1, /*KeepEmpty*/ false);
-  // Symbol names don't have spaces in them, let's just remove these in case the
-  // input is slighlty malformed.
+  BufferList symbolBufferList;
+
+  for (const auto &CurrentIgnoresFilePath : IgnoresFilePathList) {
+    auto BufferOrErr = FM.getBufferForFile(CurrentIgnoresFilePath);
+
+    if (!BufferOrErr)
+      return make_error<IgnoresFileNotFound>(CurrentIgnoresFilePath);
+
+    auto Buffer = std::move(BufferOrErr.get());
+    Buffer->getBuffer().split(Lines, '\n', /*MaxSplit*/ -1,
+                              /*KeepEmpty*/ false);
+    symbolBufferList.push_back(std::move(Buffer));
+  }
+
+  // Symbol names don't have spaces in them, let's just remove these in case
+  // the input is slighlty malformed.
   transform(Lines, Lines.begin(), [](StringRef Line) { return Line.trim(); });
   sort(Lines);
-  return APIIgnoresList(std::move(Lines), std::move(Buffer));
+  return APIIgnoresList(std::move(Lines), std::move(symbolBufferList));
 }
 
 bool APIIgnoresList::shouldIgnore(StringRef SymbolName) const {
diff --git a/clang/lib/ExtractAPI/ExtractAPIConsumer.cpp b/clang/lib/ExtractAPI/ExtractAPIConsumer.cpp
index 644845efb8191..c1e47b10a39fe 100644
--- a/clang/lib/ExtractAPI/ExtractAPIConsumer.cpp
+++ b/clang/lib/ExtractAPI/ExtractAPIConsumer.cpp
@@ -339,9 +339,9 @@ ExtractAPIAction::CreateASTConsumer(CompilerInstance &CI, StringRef InFile) {
   Policy.AnonymousTagLocations = false;
   CI.getASTContext().setPrintingPolicy(Policy);
 
-  if (!CI.getFrontendOpts().ExtractAPIIgnoresFile.empty()) {
+  if (!CI.getFrontendOpts().ExtractAPIIgnoresFileList.empty()) {
     llvm::handleAllErrors(
-        APIIgnoresList::create(CI.getFrontendOpts().ExtractAPIIgnoresFile,
+        APIIgnoresList::create(CI.getFrontendOpts().ExtractAPIIgnoresFileList,
                                CI.getFileManager())
             .moveInto(IgnoresList),
         [&CI](const IgnoresFileNotFound &Err) {
diff --git a/clang/lib/Format/BreakableToken.cpp b/clang/lib/Format/BreakableToken.cpp
index b3ef2a895d7f0..16df187eadbf5 100644
--- a/clang/lib/Format/BreakableToken.cpp
+++ b/clang/lib/Format/BreakableToken.cpp
@@ -590,10 +590,8 @@ unsigned BreakableBlockComment::getContentIndent(unsigned LineIndex) const {
     ContentWithNoDecoration = ContentWithNoDecoration.substr(1).ltrim(Blanks);
   StringRef FirstWord = ContentWithNoDecoration.substr(
       0, ContentWithNoDecoration.find_first_of(Blanks));
-  if (ContentIndentingJavadocAnnotations.find(FirstWord) !=
-      ContentIndentingJavadocAnnotations.end()) {
+  if (ContentIndentingJavadocAnnotations.contains(FirstWord))
     return Style.ContinuationIndentWidth;
-  }
   return 0;
 }
 
diff --git a/clang/lib/Format/FormatToken.h b/clang/lib/Format/FormatToken.h
index 70ecf7cacccb2..5cac5776a652a 100644
--- a/clang/lib/Format/FormatToken.h
+++ b/clang/lib/Format/FormatToken.h
@@ -612,7 +612,7 @@ struct FormatToken {
     return isOneOf(tok::kw_const, tok::kw_restrict, tok::kw_volatile,
                    tok::kw___attribute, tok::kw__Nonnull, tok::kw__Nullable,
                    tok::kw__Null_unspecified, tok::kw___ptr32, tok::kw___ptr64,
-                   TT_AttributeMacro);
+                   tok::kw___funcref, TT_AttributeMacro);
   }
 
   /// Determine whether the token is a simple-type-specifier.
@@ -1801,6 +1801,13 @@ struct AdditionalKeywords {
                                     kw_input, kw_output, kw_sequence)));
   }
 
+  /// Returns whether \p Tok is a Verilog keyword that starts a
+  /// structured procedure like 'always'.
+  bool isVerilogStructuredProcedure(const FormatToken &Tok) const {
+    return Tok.isOneOf(kw_always, kw_always_comb, kw_always_ff, kw_always_latch,
+                       kw_final, kw_forever, kw_initial);
+  }
+
   bool isVerilogQualifier(const FormatToken &Tok) const {
     switch (Tok.Tok.getKind()) {
     case tok::kw_extern:
diff --git a/clang/lib/Format/FormatTokenLexer.cpp b/clang/lib/Format/FormatTokenLexer.cpp
index ae54de93daf51..f06f9fb9949d2 100644
--- a/clang/lib/Format/FormatTokenLexer.cpp
+++ b/clang/lib/Format/FormatTokenLexer.cpp
@@ -103,8 +103,6 @@ void FormatTokenLexer::tryMergePreviousTokens() {
     return;
   if (tryMergeLessLess())
     return;
-  if (tryMergeGreaterGreater())
-    return;
   if (tryMergeForEach())
     return;
   if (Style.isCpp() && tryTransformTryUsageForC())
@@ -462,11 +460,12 @@ bool FormatTokenLexer::tryMergeLessLess() {
     return false;
 
   auto X = Tokens.size() > 3 ? First[-1] : nullptr;
-  if (X && X->is(tok::less))
+  auto Y = First[2];
+  if ((X && X->is(tok::less)) || Y->is(tok::less))
     return false;
 
-  auto Y = First[2];
-  if ((!X || X->isNot(tok::kw_operator)) && Y->is(tok::less))
+  // Do not remove a whitespace between the two "<" e.g. "operator< <>".
+  if (X && X->is(tok::kw_operator) && Y->is(tok::greater))
     return false;
 
   First[0]->Tok.setKind(tok::lessless);
@@ -476,30 +475,6 @@ bool FormatTokenLexer::tryMergeLessLess() {
   return true;
 }
 
-bool FormatTokenLexer::tryMergeGreaterGreater() {
-  // Merge kw_operator,greater,greater into kw_operator,greatergreater.
-  if (Tokens.size() < 2)
-    return false;
-
-  auto First = Tokens.end() - 2;
-  if (First[0]->isNot(tok::greater) || First[1]->isNot(tok::greater))
-    return false;
-
-  // Only merge if there currently is no whitespace between the first two ">".
-  if (First[1]->hasWhitespaceBefore())
-    return false;
-
-  auto Tok = Tokens.size() > 2 ? First[-1] : nullptr;
-  if (Tok && Tok->isNot(tok::kw_operator))
-    return false;
-
-  First[0]->Tok.setKind(tok::greatergreater);
-  First[0]->TokenText = ">>";
-  First[0]->ColumnWidth += 1;
-  Tokens.erase(Tokens.end() - 1);
-  return true;
-}
-
 bool FormatTokenLexer::tryMergeTokens(ArrayRef<tok::TokenKind> Kinds,
                                       TokenType NewType) {
   if (Tokens.size() < Kinds.size())
diff --git a/clang/lib/Format/FormatTokenLexer.h b/clang/lib/Format/FormatTokenLexer.h
index 0a8123fed2934..950305a37d68b 100644
--- a/clang/lib/Format/FormatTokenLexer.h
+++ b/clang/lib/Format/FormatTokenLexer.h
@@ -51,7 +51,6 @@ class FormatTokenLexer {
   void tryMergePreviousTokens();
 
   bool tryMergeLessLess();
-  bool tryMergeGreaterGreater();
   bool tryMergeNSStringLiteral();
   bool tryMergeJSPrivateIdentifier();
   bool tryMergeCSharpStringLiteral();
diff --git a/clang/lib/Format/FormatTokenSource.h b/clang/lib/Format/FormatTokenSource.h
index d93efaf676087..ffd2fbbad02ff 100644
--- a/clang/lib/Format/FormatTokenSource.h
+++ b/clang/lib/Format/FormatTokenSource.h
@@ -162,7 +162,7 @@ class IndexedTokenSource : public FormatTokenSource {
     auto it = Jumps.find(Next);
     if (it != Jumps.end()) {
       Next = it->second;
-      assert(Jumps.find(Next) == Jumps.end());
+      assert(!Jumps.contains(Next));
     }
     return Next;
   }
diff --git a/clang/lib/Format/MacroExpander.cpp b/clang/lib/Format/MacroExpander.cpp
index e03d046114400..bc98ec4c361fe 100644
--- a/clang/lib/Format/MacroExpander.cpp
+++ b/clang/lib/Format/MacroExpander.cpp
@@ -151,18 +151,16 @@ void MacroExpander::parseDefinition(const std::string &Macro) {
 }
 
 bool MacroExpander::defined(llvm::StringRef Name) const {
-  return FunctionLike.find(Name) != FunctionLike.end() ||
-         ObjectLike.find(Name) != ObjectLike.end();
+  return FunctionLike.contains(Name) || ObjectLike.contains(Name);
 }
 
 bool MacroExpander::objectLike(llvm::StringRef Name) const {
-  return ObjectLike.find(Name) != ObjectLike.end();
+  return ObjectLike.contains(Name);
 }
 
 bool MacroExpander::hasArity(llvm::StringRef Name, unsigned Arity) const {
   auto it = FunctionLike.find(Name);
-  return it != FunctionLike.end() &&
-         (it->second.find(Arity) != it->second.end());
+  return it != FunctionLike.end() && it->second.contains(Arity);
 }
 
 llvm::SmallVector<FormatToken *, 8>
diff --git a/clang/lib/Format/TokenAnnotator.cpp b/clang/lib/Format/TokenAnnotator.cpp
index fa80588cd3b26..6b6b9eb8d31d3 100644
--- a/clang/lib/Format/TokenAnnotator.cpp
+++ b/clang/lib/Format/TokenAnnotator.cpp
@@ -591,10 +591,6 @@ class AnnotatingParser {
     return false;
   }
 
-  bool isCpp11AttributeSpecifier(const FormatToken &Tok) {
-    return isCppAttribute(Style.isCpp(), Tok);
-  }
-
   bool parseSquare() {
     if (!CurrentToken)
       return false;
@@ -617,7 +613,7 @@ class AnnotatingParser {
 
     const bool IsInnerSquare = Contexts.back().InCpp11AttributeSpecifier;
     const bool IsCpp11AttributeSpecifier =
-        isCpp11AttributeSpecifier(*Left) || IsInnerSquare;
+        isCppAttribute(Style.isCpp(), *Left) || IsInnerSquare;
 
     // Treat C# Attributes [STAThread] much like C++ attributes [[...]].
     bool IsCSharpAttributeSpecifier =
@@ -1219,25 +1215,19 @@ class AnnotatingParser {
              !CurrentToken->isOneOf(tok::l_paren, tok::semi, tok::r_paren)) {
         if (CurrentToken->isOneOf(tok::star, tok::amp))
           CurrentToken->setType(TT_PointerOrReference);
-        auto Next = CurrentToken->getNextNonComment();
-        if (!Next)
-          break;
-        if (Next->is(tok::less))
-          next();
-        else
-          consumeToken();
-        assert(CurrentToken);
-        auto Previous = CurrentToken->getPreviousNonComment();
-        assert(Previous);
-        if (CurrentToken->is(tok::comma) && Previous->isNot(tok::kw_operator))
+        consumeToken();
+        if (!CurrentToken)
+          continue;
+        if (CurrentToken->is(tok::comma) &&
+            CurrentToken->Previous->isNot(tok::kw_operator)) {
           break;
-        if (Previous->isOneOf(TT_BinaryOperator, TT_UnaryOperator, tok::comma,
-                              tok::star, tok::arrow, tok::amp, tok::ampamp) ||
+        }
+        if (CurrentToken->Previous->isOneOf(TT_BinaryOperator, TT_UnaryOperator,
+                                            tok::comma, tok::star, tok::arrow,
+                                            tok::amp, tok::ampamp) ||
             // User defined literal.
-            Previous->TokenText.startswith("\"\"")) {
-          Previous->setType(TT_OverloadedOperator);
-          if (CurrentToken->isOneOf(tok::less, tok::greater))
-            break;
+            CurrentToken->Previous->TokenText.startswith("\"\"")) {
+          CurrentToken->Previous->setType(TT_OverloadedOperator);
         }
       }
       if (CurrentToken && CurrentToken->is(tok::l_paren))
@@ -2244,7 +2234,7 @@ class AnnotatingParser {
     if (Tok.Next->isOneOf(tok::kw_noexcept, tok::kw_volatile, tok::kw_const,
                           tok::kw_requires, tok::kw_throw, tok::arrow,
                           Keywords.kw_override, Keywords.kw_final) ||
-        isCpp11AttributeSpecifier(*Tok.Next)) {
+        isCppAttribute(Style.isCpp(), *Tok.Next)) {
       return false;
     }
 
@@ -2404,7 +2394,7 @@ class AnnotatingParser {
 
     if (!NextToken ||
         NextToken->isOneOf(tok::arrow, tok::equal, tok::kw_noexcept, tok::comma,
-                           tok::r_paren) ||
+                           tok::r_paren, TT_RequiresClause) ||
         NextToken->canBePointerOrReferenceQualifier() ||
         (NextToken->is(tok::l_brace) && !NextToken->getNextNonComment())) {
       return TT_PointerOrReference;
@@ -3762,8 +3752,9 @@ bool TokenAnnotator::spaceRequiredBetween(const AnnotatedLine &Line,
     if (Right.is(TT_BlockComment))
       return true;
     // foo() -> const Bar * override/final
-    if (Right.isOneOf(Keywords.kw_override, Keywords.kw_final,
-                      tok::kw_noexcept) &&
+    // S::foo() & noexcept/requires
+    if (Right.isOneOf(Keywords.kw_override, Keywords.kw_final, tok::kw_noexcept,
+                      TT_RequiresClause) &&
         !Right.is(TT_StartOfName)) {
       return true;
     }
@@ -4057,10 +4048,6 @@ bool TokenAnnotator::spaceRequiredBefore(const AnnotatedLine &Line,
     return true;
 
   if (Style.isCpp()) {
-    if (Left.is(TT_OverloadedOperator) &&
-        Right.isOneOf(TT_TemplateOpener, TT_TemplateCloser)) {
-      return true;
-    }
     // Space between UDL and dot: auto b = 4s .count();
     if (Right.is(tok::period) && Left.is(tok::numeric_constant))
       return true;
@@ -4352,6 +4339,11 @@ bool TokenAnnotator::spaceRequiredBefore(const AnnotatedLine &Line,
         (Left.is(TT_VerilogNumberBase) && Right.is(tok::numeric_constant))) {
       return false;
     }
+    // Don't add spaces between two at signs. Like in a coverage event.
+    // Don't add spaces between at and a sensitivity list like
+    // `@(posedge clk)`.
+    if (Left.is(tok::at) && Right.isOneOf(tok::l_paren, tok::star, tok::at))
+      return false;
     // Add space between the type name and dimension like `logic [1:0]`.
     if (Right.is(tok::l_square) &&
         Left.isOneOf(TT_VerilogDimensionedTypeName, Keywords.kw_function)) {
@@ -4424,8 +4416,6 @@ bool TokenAnnotator::spaceRequiredBefore(const AnnotatedLine &Line,
          Line.First->isOneOf(tok::kw_default, tok::kw_case))) {
       return Style.SpaceBeforeCaseColon;
     }
-    if (Line.First->isOneOf(tok::kw_default, tok::kw_case))
-      return Style.SpaceBeforeCaseColon;
     const FormatToken *Next = Right.getNextNonComment();
     if (!Next || Next->is(tok::semi))
       return false;
diff --git a/clang/lib/Format/UnwrappedLineParser.cpp b/clang/lib/Format/UnwrappedLineParser.cpp
index fb7b7be453c1f..f08a51f93e2ff 100644
--- a/clang/lib/Format/UnwrappedLineParser.cpp
+++ b/clang/lib/Format/UnwrappedLineParser.cpp
@@ -1389,6 +1389,11 @@ void UnwrappedLineParser::parseStructuralElement(
   }
 
   if (Style.isVerilog()) {
+    if (Keywords.isVerilogStructuredProcedure(*FormatTok)) {
+      parseForOrWhileLoop(/*HasParens=*/false);
+      return;
+    }
+
     // Skip things that can exist before keywords like 'if' and 'case'.
     while (true) {
       if (FormatTok->isOneOf(Keywords.kw_priority, Keywords.kw_unique,
@@ -2963,8 +2968,14 @@ void UnwrappedLineParser::parseLoopBody(bool KeepBraces, bool WrapRightBrace) {
     NestedTooDeep.pop_back();
 }
 
-void UnwrappedLineParser::parseForOrWhileLoop() {
-  assert(FormatTok->isOneOf(tok::kw_for, tok::kw_while, TT_ForEachMacro) &&
+void UnwrappedLineParser::parseForOrWhileLoop(bool HasParens) {
+  assert((FormatTok->isOneOf(tok::kw_for, tok::kw_while, TT_ForEachMacro) ||
+          (Style.isVerilog() &&
+           FormatTok->isOneOf(Keywords.kw_always, Keywords.kw_always_comb,
+                              Keywords.kw_always_ff, Keywords.kw_always_latch,
+                              Keywords.kw_final, Keywords.kw_initial,
+                              Keywords.kw_foreach, Keywords.kw_forever,
+                              Keywords.kw_repeat))) &&
          "'for', 'while' or foreach macro expected");
   const bool KeepBraces = !Style.RemoveBracesLLVM ||
                           !FormatTok->isOneOf(tok::kw_for, tok::kw_while);
@@ -2975,8 +2986,11 @@ void UnwrappedLineParser::parseForOrWhileLoop() {
     nextToken();
   if (Style.isCpp() && FormatTok->is(tok::kw_co_await))
     nextToken();
-  if (FormatTok->is(tok::l_paren))
+  if (HasParens && FormatTok->is(tok::l_paren))
     parseParens();
+  // Event control.
+  if (Style.isVerilog())
+    parseVerilogSensitivityList();
 
   handleAttributes();
   parseLoopBody(KeepBraces, /*WrapRightBrace=*/true);
@@ -3441,17 +3455,6 @@ void UnwrappedLineParser::parseConstraintExpression() {
                       /*ClosingBraceKind=*/tok::greater);
       break;
 
-    case tok::kw_bool:
-      // bool is only allowed if it is directly followed by a paren for a cast:
-      // concept C = bool(...);
-      // and bool is the only type, all other types as cast must be inside a
-      // cast to bool an thus are handled by the other cases.
-      if (Tokens->peekNextToken()->isNot(tok::l_paren))
-        return;
-      nextToken();
-      parseParens();
-      break;
-
     default:
       if (!FormatTok->Tok.getIdentifierInfo()) {
         // Identifiers are part of the default case, we check for more then
@@ -3701,13 +3704,13 @@ void UnwrappedLineParser::parseJavaEnumBody() {
 void UnwrappedLineParser::parseRecord(bool ParseAsExpr) {
   const FormatToken &InitialToken = *FormatTok;
   nextToken();
+  handleAttributes();
 
   // The actual identifier can be a nested name specifier, and in macros
   // it is often token-pasted.
-  // An [[attribute]] can be before the identifier.
   while (FormatTok->isOneOf(tok::identifier, tok::coloncolon, tok::hashhash,
                             tok::kw___attribute, tok::kw___declspec,
-                            tok::kw_alignas, tok::l_square, tok::r_square) ||
+                            tok::kw_alignas) ||
          ((Style.Language == FormatStyle::LK_Java || Style.isJavaScript()) &&
           FormatTok->isOneOf(tok::period, tok::comma))) {
     if (Style.isJavaScript() &&
@@ -3725,15 +3728,10 @@ void UnwrappedLineParser::parseRecord(bool ParseAsExpr) {
         FormatTok->is(tok::identifier) &&
         FormatTok->TokenText != FormatTok->TokenText.upper();
     nextToken();
-    // We can have macros or attributes in between 'class' and the class name.
+    // We can have macros in between 'class' and the class name.
     if (!IsNonMacroIdentifier) {
       if (FormatTok->is(tok::l_paren)) {
         parseParens();
-      } else if (FormatTok->is(TT_AttributeSquare)) {
-        parseSquare();
-        // Consume the closing TT_AttributeSquare.
-        if (FormatTok->Next && FormatTok->is(TT_AttributeSquare))
-          nextToken();
       }
     }
   }
diff --git a/clang/lib/Format/UnwrappedLineParser.h b/clang/lib/Format/UnwrappedLineParser.h
index 20ea2956058b9..4a48fa856247a 100644
--- a/clang/lib/Format/UnwrappedLineParser.h
+++ b/clang/lib/Format/UnwrappedLineParser.h
@@ -166,7 +166,7 @@ class UnwrappedLineParser {
   FormatToken *parseIfThenElse(IfStmtKind *IfKind, bool KeepBraces = false);
   void parseTryCatch();
   void parseLoopBody(bool KeepBraces, bool WrapRightBrace);
-  void parseForOrWhileLoop();
+  void parseForOrWhileLoop(bool HasParens = true);
   void parseDoWhile();
   void parseLabel(bool LeftAlignLabel = false);
   void parseCaseLabel();
diff --git a/clang/lib/Frontend/ASTUnit.cpp b/clang/lib/Frontend/ASTUnit.cpp
index f2939a2e8f0ef..3269c9d0f479f 100644
--- a/clang/lib/Frontend/ASTUnit.cpp
+++ b/clang/lib/Frontend/ASTUnit.cpp
@@ -1397,7 +1397,8 @@ ASTUnit::getMainBufferWithPrecompiledPreamble(
 
     llvm::ErrorOr<PrecompiledPreamble> NewPreamble = PrecompiledPreamble::Build(
         PreambleInvocationIn, MainFileBuffer.get(), Bounds, *Diagnostics, VFS,
-        PCHContainerOps, /*StoreInMemory=*/false, Callbacks);
+        PCHContainerOps, StorePreamblesInMemory, PreambleStoragePath,
+        Callbacks);
 
     PreambleInvocationIn.getFrontendOpts().SkipFunctionBodies =
         PreviousSkipFunctionBodies;
@@ -1741,6 +1742,7 @@ ASTUnit *ASTUnit::LoadFromCommandLine(
     const char **ArgBegin, const char **ArgEnd,
     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
     IntrusiveRefCntPtr<DiagnosticsEngine> Diags, StringRef ResourceFilesPath,
+    bool StorePreamblesInMemory, StringRef PreambleStoragePath,
     bool OnlyLocalDecls, CaptureDiagsKind CaptureDiagnostics,
     ArrayRef<RemappedFile> RemappedFiles, bool RemappedFilesKeepOriginalName,
     unsigned PrecompilePreambleAfterNParses, TranslationUnitKind TUKind,
@@ -1801,6 +1803,8 @@ ASTUnit *ASTUnit::LoadFromCommandLine(
     VFS = llvm::vfs::getRealFileSystem();
   VFS = createVFSFromCompilerInvocation(*CI, *Diags, VFS);
   AST->FileMgr = new FileManager(AST->FileSystemOpts, VFS);
+  AST->StorePreamblesInMemory = StorePreamblesInMemory;
+  AST->PreambleStoragePath = PreambleStoragePath;
   AST->ModuleCache = new InMemoryModuleCache;
   AST->OnlyLocalDecls = OnlyLocalDecls;
   AST->CaptureDiagnostics = CaptureDiagnostics;
diff --git a/clang/lib/Frontend/CompilerInstance.cpp b/clang/lib/Frontend/CompilerInstance.cpp
index f3510b0c15230..691f779b5966c 100644
--- a/clang/lib/Frontend/CompilerInstance.cpp
+++ b/clang/lib/Frontend/CompilerInstance.cpp
@@ -981,10 +981,9 @@ bool CompilerInstance::InitializeSourceManager(const FrontendInputFile &Input,
                        ? FileMgr.getSTDIN()
                        : FileMgr.getFileRef(InputFile, /*OpenFile=*/true);
   if (!FileOrErr) {
-    // FIXME: include the error in the diagnostic even when it's not stdin.
     auto EC = llvm::errorToErrorCode(FileOrErr.takeError());
     if (InputFile != "-")
-      Diags.Report(diag::err_fe_error_reading) << InputFile;
+      Diags.Report(diag::err_fe_error_reading) << InputFile << EC.message();
     else
       Diags.Report(diag::err_fe_error_reading_stdin) << EC.message();
     return false;
@@ -1087,9 +1086,12 @@ bool CompilerInstance::ExecuteAction(FrontendAction &Act) {
   }
   StringRef StatsFile = getFrontendOpts().StatsFile;
   if (!StatsFile.empty()) {
+    llvm::sys::fs::OpenFlags FileFlags = llvm::sys::fs::OF_TextWithCRLF;
+    if (getFrontendOpts().AppendStats)
+      FileFlags |= llvm::sys::fs::OF_Append;
     std::error_code EC;
-    auto StatS = std::make_unique<llvm::raw_fd_ostream>(
-        StatsFile, EC, llvm::sys::fs::OF_TextWithCRLF);
+    auto StatS =
+        std::make_unique<llvm::raw_fd_ostream>(StatsFile, EC, FileFlags);
     if (EC) {
       getDiagnostics().Report(diag::warn_fe_unable_to_open_stats_file)
           << StatsFile << EC.message();
diff --git a/clang/lib/Frontend/InitPreprocessor.cpp b/clang/lib/Frontend/InitPreprocessor.cpp
index 208c6a8db1598..0b39dd1c84f7d 100644
--- a/clang/lib/Frontend/InitPreprocessor.cpp
+++ b/clang/lib/Frontend/InitPreprocessor.cpp
@@ -707,10 +707,6 @@ static void InitializeCPlusPlusFeatureTestMacros(const LangOptions &LangOpts,
   if (LangOpts.Char8)
     Builder.defineMacro("__cpp_char8_t", "202207L");
   Builder.defineMacro("__cpp_impl_destroying_delete", "201806L");
-
-  // TS features.
-  if (LangOpts.Coroutines)
-    Builder.defineMacro("__cpp_coroutines", "201703L");
 }
 
 /// InitializeOpenCLFeatureTestMacros - Define OpenCL macros based on target
@@ -1317,11 +1313,10 @@ void clang::InitializePreprocessor(
   llvm::raw_string_ostream Predefines(PredefineBuffer);
   MacroBuilder Builder(Predefines);
 
-  // Emit line markers for various builtin sections of the file.  We don't do
-  // this in asm preprocessor mode, because "# 4" is not a line marker directive
-  // in this mode.
-  if (!PP.getLangOpts().AsmPreprocessor)
-    Builder.append("# 1 \"<built-in>\" 3");
+  // Emit line markers for various builtin sections of the file. The 3 here
+  // marks <built-in> as being a system header, which suppresses warnings when
+  // the same macro is defined multiple times.
+  Builder.append("# 1 \"<built-in>\" 3");
 
   // Install things like __POWERPC__, __GNUC__, etc into the macro table.
   if (InitOpts.UsePredefines) {
@@ -1359,8 +1354,7 @@ void clang::InitializePreprocessor(
 
   // Add on the predefines from the driver.  Wrap in a #line directive to report
   // that they come from the command line.
-  if (!PP.getLangOpts().AsmPreprocessor)
-    Builder.append("# 1 \"<command line>\" 1");
+  Builder.append("# 1 \"<command line>\" 1");
 
   // Process #define's and #undef's in the order they are given.
   for (unsigned i = 0, e = InitOpts.Macros.size(); i != e; ++i) {
@@ -1372,8 +1366,7 @@ void clang::InitializePreprocessor(
   }
 
   // Exit the command line and go back to <built-in> (2 is LC_LEAVE).
-  if (!PP.getLangOpts().AsmPreprocessor)
-    Builder.append("# 1 \"<built-in>\" 2");
+  Builder.append("# 1 \"<built-in>\" 2");
 
   // If -imacros are specified, include them now.  These are processed before
   // any -include directives.
diff --git a/clang/lib/Frontend/PrecompiledPreamble.cpp b/clang/lib/Frontend/PrecompiledPreamble.cpp
index 579a0b8b614de..a47d2b0be004c 100644
--- a/clang/lib/Frontend/PrecompiledPreamble.cpp
+++ b/clang/lib/Frontend/PrecompiledPreamble.cpp
@@ -197,20 +197,32 @@ void TemporaryFiles::removeFile(StringRef File) {
 class TempPCHFile {
 public:
   // A main method used to construct TempPCHFile.
-  static std::unique_ptr<TempPCHFile> create() {
+  static std::unique_ptr<TempPCHFile> create(StringRef StoragePath) {
     // FIXME: This is a hack so that we can override the preamble file during
     // crash-recovery testing, which is the only case where the preamble files
     // are not necessarily cleaned up.
     if (const char *TmpFile = ::getenv("CINDEXTEST_PREAMBLE_FILE"))
       return std::unique_ptr<TempPCHFile>(new TempPCHFile(TmpFile));
 
-    llvm::SmallString<64> File;
-    // Using a version of createTemporaryFile with a file descriptor guarantees
+    llvm::SmallString<128> File;
+    // Using the versions of createTemporaryFile() and
+    // createUniqueFile() with a file descriptor guarantees
     // that we would never get a race condition in a multi-threaded setting
     // (i.e., multiple threads getting the same temporary path).
     int FD;
-    if (auto EC =
-            llvm::sys::fs::createTemporaryFile("preamble", "pch", FD, File))
+    std::error_code EC;
+    if (StoragePath.empty())
+      EC = llvm::sys::fs::createTemporaryFile("preamble", "pch", FD, File);
+    else {
+      llvm::SmallString<128> TempPath = StoragePath;
+      // Use the same filename model as fs::createTemporaryFile().
+      llvm::sys::path::append(TempPath, "preamble-%%%%%%.pch");
+      namespace fs = llvm::sys::fs;
+      // Use the same owner-only file permissions as fs::createTemporaryFile().
+      EC = fs::createUniqueFile(TempPath, FD, File, fs::OF_None,
+                                fs::owner_read | fs::owner_write);
+    }
+    if (EC)
       return nullptr;
     // We only needed to make sure the file exists, close the file right away.
     llvm::sys::Process::SafelyCloseFileDescriptor(FD);
@@ -403,7 +415,7 @@ llvm::ErrorOr<PrecompiledPreamble> PrecompiledPreamble::Build(
     DiagnosticsEngine &Diagnostics,
     IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS,
     std::shared_ptr<PCHContainerOperations> PCHContainerOps, bool StoreInMemory,
-    PreambleCallbacks &Callbacks) {
+    StringRef StoragePath, PreambleCallbacks &Callbacks) {
   assert(VFS && "VFS is null");
 
   auto PreambleInvocation = std::make_shared<CompilerInvocation>(Invocation);
@@ -418,7 +430,8 @@ llvm::ErrorOr<PrecompiledPreamble> PrecompiledPreamble::Build(
   } else {
     // Create a temporary file for the precompiled preamble. In rare
     // circumstances, this can fail.
-    std::unique_ptr<TempPCHFile> PreamblePCHFile = TempPCHFile::create();
+    std::unique_ptr<TempPCHFile> PreamblePCHFile =
+        TempPCHFile::create(StoragePath);
     if (!PreamblePCHFile)
       return BuildPreambleError::CouldntCreateTempFile;
     Storage = PCHStorage::file(std::move(PreamblePCHFile));
diff --git a/clang/lib/Frontend/SerializedDiagnosticPrinter.cpp b/clang/lib/Frontend/SerializedDiagnosticPrinter.cpp
index fc8fce4b42b85..b76728acb9077 100644
--- a/clang/lib/Frontend/SerializedDiagnosticPrinter.cpp
+++ b/clang/lib/Frontend/SerializedDiagnosticPrinter.cpp
@@ -37,14 +37,12 @@ class AbbreviationMap {
   AbbreviationMap() {}
 
   void set(unsigned recordID, unsigned abbrevID) {
-    assert(Abbrevs.find(recordID) == Abbrevs.end()
-           && "Abbreviation already set.");
+    assert(!Abbrevs.contains(recordID) && "Abbreviation already set.");
     Abbrevs[recordID] = abbrevID;
   }
 
   unsigned get(unsigned recordID) {
-    assert(Abbrevs.find(recordID) != Abbrevs.end() &&
-           "Abbreviation not set.");
+    assert(Abbrevs.contains(recordID) && "Abbreviation not set.");
     return Abbrevs[recordID];
   }
 };
diff --git a/clang/lib/Lex/HeaderSearch.cpp b/clang/lib/Lex/HeaderSearch.cpp
index 074c147ba3c54..5a7357a5ada43 100644
--- a/clang/lib/Lex/HeaderSearch.cpp
+++ b/clang/lib/Lex/HeaderSearch.cpp
@@ -378,15 +378,17 @@ void HeaderSearch::indexInitialHeaderMaps() {
   llvm::StringMap<unsigned, llvm::BumpPtrAllocator> Index(SearchDirs.size());
 
   // Iterate over all filename keys and associate them with the index i.
-  unsigned i = 0;
-  for (; i != SearchDirs.size(); ++i) {
+  for (unsigned i = 0; i != SearchDirs.size(); ++i) {
     auto &Dir = SearchDirs[i];
 
     // We're concerned with only the initial contiguous run of header
     // maps within SearchDirs, which can be 99% of SearchDirs when
     // SearchDirs.size() is ~10000.
-    if (!Dir.isHeaderMap())
+    if (!Dir.isHeaderMap()) {
+      SearchDirHeaderMapIndex = std::move(Index);
+      FirstNonHeaderMapSearchDirIdx = i;
       break;
+    }
 
     // Give earlier keys precedence over identical later keys.
     auto Callback = [&](StringRef Filename) {
@@ -394,9 +396,6 @@ void HeaderSearch::indexInitialHeaderMaps() {
     };
     Dir.getHeaderMap()->forEachKey(Callback);
   }
-
-  SearchDirHeaderMapIndex = std::move(Index);
-  FirstNonHeaderMapSearchDirIdx = i;
 }
 
 //===----------------------------------------------------------------------===//
@@ -1929,7 +1928,7 @@ std::string HeaderSearch::suggestPathToFileForDiagnostics(
     llvm::StringRef File, llvm::StringRef WorkingDir, llvm::StringRef MainFile,
     bool *IsSystem) {
   using namespace llvm::sys;
-  
+
   llvm::SmallString<32> FilePath = File;
   // remove_dots switches to backslashes on windows as a side-effect!
   // We always want to suggest forward slashes for includes.
diff --git a/clang/lib/Lex/Lexer.cpp b/clang/lib/Lex/Lexer.cpp
index b9b13725edbc5..bc97ca18a2ebd 100644
--- a/clang/lib/Lex/Lexer.cpp
+++ b/clang/lib/Lex/Lexer.cpp
@@ -2279,7 +2279,7 @@ void Lexer::codeCompleteIncludedFile(const char *PathStart,
     ++CompletionPoint;
     if (Next == (IsAngled ? '>' : '"'))
       break;
-    if (llvm::is_contained(SlashChars, Next))
+    if (SlashChars.contains(Next))
       break;
   }
 
diff --git a/clang/lib/Lex/ModuleMap.cpp b/clang/lib/Lex/ModuleMap.cpp
index 5973b4ae06240..8dead93b03734 100644
--- a/clang/lib/Lex/ModuleMap.cpp
+++ b/clang/lib/Lex/ModuleMap.cpp
@@ -930,27 +930,13 @@ Module *ModuleMap::createHeaderUnit(SourceLocation Loc, StringRef Name,
 
 /// For a framework module, infer the framework against which we
 /// should link.
-static void inferFrameworkLink(Module *Mod, const DirectoryEntry *FrameworkDir,
-                               FileManager &FileMgr) {
+static void inferFrameworkLink(Module *Mod) {
   assert(Mod->IsFramework && "Can only infer linking for framework modules");
   assert(!Mod->isSubFramework() &&
          "Can only infer linking for top-level frameworks");
 
-  SmallString<128> LibName;
-  LibName += FrameworkDir->getName();
-  llvm::sys::path::append(LibName, Mod->Name);
-
-  // The library name of a framework has more than one possible extension since
-  // the introduction of the text-based dynamic library format. We need to check
-  // for both before we give up.
-  for (const char *extension : {"", ".tbd"}) {
-    llvm::sys::path::replace_extension(LibName, extension);
-    if (FileMgr.getFile(LibName)) {
-      Mod->LinkLibraries.push_back(Module::LinkLibrary(Mod->Name,
-                                                       /*IsFramework=*/true));
-      return;
-    }
-  }
+  Mod->LinkLibraries.push_back(Module::LinkLibrary(Mod->Name,
+                                                   /*IsFramework=*/true));
 }
 
 Module *ModuleMap::inferFrameworkModule(const DirectoryEntry *FrameworkDir,
@@ -1129,9 +1115,8 @@ Module *ModuleMap::inferFrameworkModule(const DirectoryEntry *FrameworkDir,
 
   // If the module is a top-level framework, automatically link against the
   // framework.
-  if (!Result->isSubFramework()) {
-    inferFrameworkLink(Result, FrameworkDir, FileMgr);
-  }
+  if (!Result->isSubFramework())
+    inferFrameworkLink(Result);
 
   return Result;
 }
@@ -2185,9 +2170,8 @@ void ModuleMapParser::parseModuleDecl() {
   // If the active module is a top-level framework, and there are no link
   // libraries, automatically link against the framework.
   if (ActiveModule->IsFramework && !ActiveModule->isSubFramework() &&
-      ActiveModule->LinkLibraries.empty()) {
-    inferFrameworkLink(ActiveModule, Directory, SourceMgr.getFileManager());
-  }
+      ActiveModule->LinkLibraries.empty())
+    inferFrameworkLink(ActiveModule);
 
   // If the module meets all requirements but is still unavailable, mark the
   // whole tree as unavailable to prevent it from building.
diff --git a/clang/lib/Lex/PPDirectives.cpp b/clang/lib/Lex/PPDirectives.cpp
index 9e2392529ff53..bda9d0877612e 100644
--- a/clang/lib/Lex/PPDirectives.cpp
+++ b/clang/lib/Lex/PPDirectives.cpp
@@ -1185,8 +1185,12 @@ void Preprocessor::HandleDirective(Token &Result) {
                                     CurPPLexer->getConditionalStackDepth() > 0);
     return;
   case tok::numeric_constant:  // # 7  GNU line marker directive.
-    if (getLangOpts().AsmPreprocessor)
-      break;  // # 4 is not a preprocessor directive in .S files.
+    // In a .S file "# 4" may be a comment so don't treat it as a preprocessor
+    // directive. However do permit it in the predefines file, as we use line
+    // markers to mark the builtin macros as being in a system header.
+    if (getLangOpts().AsmPreprocessor &&
+        SourceMgr.getFileID(SavedHash.getLocation()) != getPredefinesFileID())
+      break;
     return HandleDigitDirective(Result);
   default:
     IdentifierInfo *II = Result.getIdentifierInfo();
diff --git a/clang/lib/Lex/PPMacroExpansion.cpp b/clang/lib/Lex/PPMacroExpansion.cpp
index bbc271e5611e2..a14fdcda58e02 100644
--- a/clang/lib/Lex/PPMacroExpansion.cpp
+++ b/clang/lib/Lex/PPMacroExpansion.cpp
@@ -87,7 +87,7 @@ void Preprocessor::appendMacroDirective(IdentifierInfo *II, MacroDirective *MD){
 
   // Set up the identifier as having associated macro history.
   II->setHasMacroDefinition(true);
-  if (!MD->isDefined() && LeafModuleMacros.find(II) == LeafModuleMacros.end())
+  if (!MD->isDefined() && !LeafModuleMacros.contains(II))
     II->setHasMacroDefinition(false);
   if (II->isFromAST())
     II->setChangedSinceDeserialization();
@@ -125,7 +125,7 @@ void Preprocessor::setLoadedMacroDirective(IdentifierInfo *II,
 
   // Setup the identifier as having associated macro history.
   II->setHasMacroDefinition(true);
-  if (!MD->isDefined() && LeafModuleMacros.find(II) == LeafModuleMacros.end())
+  if (!MD->isDefined() && !LeafModuleMacros.contains(II))
     II->setHasMacroDefinition(false);
 }
 
@@ -1559,17 +1559,11 @@ void Preprocessor::ExpandBuiltinMacro(Token &Tok) {
       // __FILE_NAME__ is a Clang-specific extension that expands to the
       // the last part of __FILE__.
       if (II == Ident__FILE_NAME__) {
-        // Try to get the last path component, failing that return the original
-        // presumed location.
-        StringRef PLFileName = llvm::sys::path::filename(PLoc.getFilename());
-        if (PLFileName != "")
-          FN += PLFileName;
-        else
-          FN += PLoc.getFilename();
+        processPathToFileName(FN, PLoc, getLangOpts(), getTargetInfo());
       } else {
         FN += PLoc.getFilename();
+        processPathForFileMacro(FN, getLangOpts(), getTargetInfo());
       }
-      processPathForFileMacro(FN, getLangOpts(), getTargetInfo());
       Lexer::Stringify(FN);
       OS << '"' << FN << '"';
     }
@@ -1637,35 +1631,14 @@ void Preprocessor::ExpandBuiltinMacro(Token &Tok) {
     Tok.setKind(tok::string_literal);
   } else if (II == Ident__FLT_EVAL_METHOD__) {
     // __FLT_EVAL_METHOD__ is set to the default value.
-    if (getTUFPEvalMethod() ==
-        LangOptions::FPEvalMethodKind::FEM_Indeterminable) {
-      // This is possible if `AllowFPReassoc` or `AllowReciprocal` is enabled.
-      // These modes can be triggered via the command line option `-ffast-math`
-      // or via a `pragam float_control`.
-      // __FLT_EVAL_METHOD__ expands to -1.
-      // The `minus` operator is the next token we read from the stream.
-      auto Toks = std::make_unique<Token[]>(1);
-      OS << "-";
-      Tok.setKind(tok::minus);
-      // Push the token `1` to the stream.
-      Token NumberToken;
-      NumberToken.startToken();
-      NumberToken.setKind(tok::numeric_constant);
-      NumberToken.setLiteralData("1");
-      NumberToken.setLength(1);
-      Toks[0] = NumberToken;
-      EnterTokenStream(std::move(Toks), 1, /*DisableMacroExpansion*/ false,
-                       /*IsReinject*/ false);
-    } else {
-      OS << getTUFPEvalMethod();
-      // __FLT_EVAL_METHOD__ expands to a simple numeric value.
-      Tok.setKind(tok::numeric_constant);
-      if (getLastFPEvalPragmaLocation().isValid()) {
-        // The program is ill-formed. The value of __FLT_EVAL_METHOD__ is
-        // altered by the pragma.
-        Diag(Tok, diag::err_illegal_use_of_flt_eval_macro);
-        Diag(getLastFPEvalPragmaLocation(), diag::note_pragma_entered_here);
-      }
+    OS << getTUFPEvalMethod();
+    // __FLT_EVAL_METHOD__ expands to a simple numeric value.
+    Tok.setKind(tok::numeric_constant);
+    if (getLastFPEvalPragmaLocation().isValid()) {
+      // The program is ill-formed. The value of __FLT_EVAL_METHOD__ is altered
+      // by the pragma.
+      Diag(Tok, diag::err_illegal_use_of_flt_eval_macro);
+      Diag(getLastFPEvalPragmaLocation(), diag::note_pragma_entered_here);
     }
   } else if (II == Ident__COUNTER__) {
     // __COUNTER__ expands to a simple numeric value.
@@ -1995,3 +1968,16 @@ void Preprocessor::processPathForFileMacro(SmallVectorImpl<char> &Path,
       llvm::sys::path::remove_dots(Path, false, llvm::sys::path::Style::posix);
   }
 }
+
+void Preprocessor::processPathToFileName(SmallVectorImpl<char> &FileName,
+                                         const PresumedLoc &PLoc,
+                                         const LangOptions &LangOpts,
+                                         const TargetInfo &TI) {
+  // Try to get the last path component, failing that return the original
+  // presumed location.
+  StringRef PLFileName = llvm::sys::path::filename(PLoc.getFilename());
+  if (PLFileName.empty())
+    PLFileName = PLoc.getFilename();
+  FileName.append(PLFileName.begin(), PLFileName.end());
+  processPathForFileMacro(FileName, LangOpts, TI);
+}
diff --git a/clang/lib/Lex/Preprocessor.cpp b/clang/lib/Lex/Preprocessor.cpp
index 27be7d611f595..8de78a13930ed 100644
--- a/clang/lib/Lex/Preprocessor.cpp
+++ b/clang/lib/Lex/Preprocessor.cpp
@@ -207,11 +207,6 @@ void Preprocessor::Initialize(const TargetInfo &Target,
   else
     // Set initial value of __FLT_EVAL_METHOD__ from the command line.
     setCurrentFPEvalMethod(SourceLocation(), getLangOpts().getFPEvalMethod());
-  // When `-ffast-math` option is enabled, it triggers several driver math
-  // options to be enabled. Among those, only one the following two modes
-  // affect the eval-method:  reciprocal or reassociate.
-  if (getLangOpts().AllowFPReassoc || getLangOpts().AllowRecip)
-    setCurrentFPEvalMethod(SourceLocation(), LangOptions::FEM_Indeterminable);
 }
 
 void Preprocessor::InitializeForModelFile() {
diff --git a/clang/lib/Parse/ParseDecl.cpp b/clang/lib/Parse/ParseDecl.cpp
index 3106571728692..2b5829ba0ef0e 100644
--- a/clang/lib/Parse/ParseDecl.cpp
+++ b/clang/lib/Parse/ParseDecl.cpp
@@ -852,6 +852,22 @@ void Parser::ParseMicrosoftTypeAttributes(ParsedAttributes &attrs) {
   }
 }
 
+void Parser::ParseWebAssemblyFuncrefTypeAttribute(ParsedAttributes &attrs) {
+  assert(Tok.is(tok::kw___funcref));
+  SourceLocation StartLoc = Tok.getLocation();
+  if (!getTargetInfo().getTriple().isWasm()) {
+    ConsumeToken();
+    Diag(StartLoc, diag::err_wasm_funcref_not_wasm);
+    return;
+  }
+
+  IdentifierInfo *AttrName = Tok.getIdentifierInfo();
+  SourceLocation AttrNameLoc = ConsumeToken();
+  attrs.addNew(AttrName, AttrNameLoc, /*ScopeName=*/nullptr,
+               /*ScopeLoc=*/SourceLocation{}, /*Args=*/nullptr, /*numArgs=*/0,
+               ParsedAttr::AS_Keyword);
+}
+
 void Parser::DiagnoseAndSkipExtendedMicrosoftTypeAttributes() {
   SourceLocation StartLoc = Tok.getLocation();
   SourceLocation EndLoc = SkipExtendedMicrosoftTypeAttributes();
@@ -3380,6 +3396,8 @@ void Parser::ParseDeclarationSpecifiers(
         goto DoneWithDeclSpec;
 
       CXXScopeSpec SS;
+      if (TemplateInfo.TemplateParams)
+        SS.setTemplateParamLists(*TemplateInfo.TemplateParams);
       Actions.RestoreNestedNameSpecifierAnnotation(Tok.getAnnotationValue(),
                                                    Tok.getAnnotationRange(),
                                                    SS);
@@ -3475,7 +3493,8 @@ void Parser::ParseDeclarationSpecifiers(
                                      &SS) &&
           isConstructorDeclarator(/*Unqualified=*/false,
                                   /*DeductionGuide=*/false,
-                                  DS.isFriendSpecified()))
+                                  DS.isFriendSpecified(),
+                                  &TemplateInfo))
         goto DoneWithDeclSpec;
 
       // C++20 [temp.spec] 13.9/6.
@@ -3842,6 +3861,10 @@ void Parser::ParseDeclarationSpecifiers(
       ParseMicrosoftTypeAttributes(DS.getAttributes());
       continue;
 
+    case tok::kw___funcref:
+      ParseWebAssemblyFuncrefTypeAttribute(DS.getAttributes());
+      continue;
+
     // Borland single token adornments.
     case tok::kw___pascal:
       ParseBorlandTypeAttributes(DS.getAttributes());
@@ -4957,6 +4980,7 @@ void Parser::ParseEnumSpecifier(SourceLocation StartLoc, DeclSpec &DS,
     assert(TemplateInfo.TemplateParams && "no template parameters");
     TParams = MultiTemplateParamsArg(TemplateInfo.TemplateParams->data(),
                                      TemplateInfo.TemplateParams->size());
+    SS.setTemplateParamLists(TParams);
   }
 
   if (!Name && TUK != Sema::TUK_Definition) {
@@ -5405,7 +5429,7 @@ bool Parser::isTypeSpecifierQualifier() {
   case tok::kw___read_only:
   case tok::kw___read_write:
   case tok::kw___write_only:
-
+  case tok::kw___funcref:
   case tok::kw_groupshared:
     return true;
 
@@ -5670,6 +5694,7 @@ bool Parser::isDeclarationSpecifier(
 #define GENERIC_IMAGE_TYPE(ImgType, Id) case tok::kw_##ImgType##_t:
 #include "clang/Basic/OpenCLImageTypes.def"
 
+  case tok::kw___funcref:
   case tok::kw_groupshared:
     return true;
 
@@ -5679,11 +5704,15 @@ bool Parser::isDeclarationSpecifier(
 }
 
 bool Parser::isConstructorDeclarator(bool IsUnqualified, bool DeductionGuide,
-                                     DeclSpec::FriendSpecified IsFriend) {
+                                     DeclSpec::FriendSpecified IsFriend,
+                                     const ParsedTemplateInfo *TemplateInfo) {
   TentativeParsingAction TPA(*this);
 
   // Parse the C++ scope specifier.
   CXXScopeSpec SS;
+  if (TemplateInfo && TemplateInfo->TemplateParams)
+    SS.setTemplateParamLists(*TemplateInfo->TemplateParams);
+
   if (ParseOptionalCXXScopeSpecifier(SS, /*ObjectType=*/nullptr,
                                      /*ObjectHasErrors=*/false,
                                      /*EnteringContext=*/true)) {
@@ -5931,6 +5960,12 @@ void Parser::ParseTypeQualifierListOpt(
         continue;
       }
       goto DoneWithTypeQuals;
+
+    case tok::kw___funcref:
+      ParseWebAssemblyFuncrefTypeAttribute(DS.getAttributes());
+      continue;
+      goto DoneWithTypeQuals;
+
     case tok::kw___pascal:
       if (AttrReqs & AR_VendorAttributesParsed) {
         ParseBorlandTypeAttributes(DS.getAttributes());
@@ -6075,6 +6110,7 @@ void Parser::ParseDeclaratorInternal(Declarator &D,
     bool EnteringContext = D.getContext() == DeclaratorContext::File ||
                            D.getContext() == DeclaratorContext::Member;
     CXXScopeSpec SS;
+    SS.setTemplateParamLists(D.getTemplateParameterLists());
     ParseOptionalCXXScopeSpecifier(SS, /*ObjectType=*/nullptr,
                                    /*ObjectHasErrors=*/false, EnteringContext);
 
diff --git a/clang/lib/Parse/ParseDeclCXX.cpp b/clang/lib/Parse/ParseDeclCXX.cpp
index c2403f0d6f439..037bc869c5a19 100644
--- a/clang/lib/Parse/ParseDeclCXX.cpp
+++ b/clang/lib/Parse/ParseDeclCXX.cpp
@@ -1676,6 +1676,9 @@ void Parser::ParseClassSpecifier(tok::TokenKind TagTokKind,
     ColonProtectionRAIIObject X(*this);
 
     CXXScopeSpec Spec;
+    if (TemplateInfo.TemplateParams)
+      Spec.setTemplateParamLists(*TemplateInfo.TemplateParams);
+
     bool HasValidSpec = true;
     if (ParseOptionalCXXScopeSpecifier(Spec, /*ObjectType=*/nullptr,
                                        /*ObjectHasErrors=*/false,
diff --git a/clang/lib/Parse/ParseExpr.cpp b/clang/lib/Parse/ParseExpr.cpp
index e2ada6ad9f82b..554113c5a27c2 100644
--- a/clang/lib/Parse/ParseExpr.cpp
+++ b/clang/lib/Parse/ParseExpr.cpp
@@ -789,6 +789,7 @@ class CastExpressionIdValidator final : public CorrectionCandidateCallback {
 /// [GNU]   '__builtin_choose_expr' '(' assign-expr ',' assign-expr ','
 ///                                     assign-expr ')'
 /// [GNU]   '__builtin_FILE' '(' ')'
+/// [CLANG] '__builtin_FILE_NAME' '(' ')'
 /// [GNU]   '__builtin_FUNCTION' '(' ')'
 /// [GNU]   '__builtin_LINE' '(' ')'
 /// [CLANG] '__builtin_COLUMN' '(' ')'
@@ -1317,6 +1318,7 @@ ExprResult Parser::ParseCastExpression(CastParseKind ParseKind,
   case tok::kw___builtin_convertvector:
   case tok::kw___builtin_COLUMN:
   case tok::kw___builtin_FILE:
+  case tok::kw___builtin_FILE_NAME:
   case tok::kw___builtin_FUNCTION:
   case tok::kw___builtin_LINE:
   case tok::kw___builtin_source_location:
@@ -2542,6 +2544,7 @@ ExprResult Parser::ParseUnaryExprOrTypeTraitExpression() {
 ///                                     assign-expr ')'
 /// [GNU]   '__builtin_types_compatible_p' '(' type-name ',' type-name ')'
 /// [GNU]   '__builtin_FILE' '(' ')'
+/// [CLANG] '__builtin_FILE_NAME' '(' ')'
 /// [GNU]   '__builtin_FUNCTION' '(' ')'
 /// [GNU]   '__builtin_LINE' '(' ')'
 /// [CLANG] '__builtin_COLUMN' '(' ')'
@@ -2777,6 +2780,7 @@ ExprResult Parser::ParseBuiltinPrimaryExpression() {
   }
   case tok::kw___builtin_COLUMN:
   case tok::kw___builtin_FILE:
+  case tok::kw___builtin_FILE_NAME:
   case tok::kw___builtin_FUNCTION:
   case tok::kw___builtin_LINE:
   case tok::kw___builtin_source_location: {
@@ -2790,6 +2794,8 @@ ExprResult Parser::ParseBuiltinPrimaryExpression() {
       switch (T) {
       case tok::kw___builtin_FILE:
         return SourceLocExpr::File;
+      case tok::kw___builtin_FILE_NAME:
+        return SourceLocExpr::FileName;
       case tok::kw___builtin_FUNCTION:
         return SourceLocExpr::Function;
       case tok::kw___builtin_LINE:
diff --git a/clang/lib/Parse/ParseStmt.cpp b/clang/lib/Parse/ParseStmt.cpp
index 1c8441fafc483..b6b87e356e921 100644
--- a/clang/lib/Parse/ParseStmt.cpp
+++ b/clang/lib/Parse/ParseStmt.cpp
@@ -1100,7 +1100,7 @@ StmtResult Parser::handleExprStmt(ExprResult E, ParsedStmtContext StmtCtx) {
       ++LookAhead;
     }
     // Then look to see if the next two tokens close the statement expression;
-    // if so, this expression statement is the last statement in a statment
+    // if so, this expression statement is the last statement in a statement
     // expression.
     IsStmtExprResult = GetLookAheadToken(LookAhead).is(tok::r_brace) &&
                        GetLookAheadToken(LookAhead + 1).is(tok::r_paren);
diff --git a/clang/lib/Parse/ParseTentative.cpp b/clang/lib/Parse/ParseTentative.cpp
index 785749bff65a2..b26faaff7f9f3 100644
--- a/clang/lib/Parse/ParseTentative.cpp
+++ b/clang/lib/Parse/ParseTentative.cpp
@@ -1511,6 +1511,10 @@ Parser::isCXXDeclarationSpecifier(ImplicitTypenameContext AllowImplicitTypename,
   case tok::kw___kindof:
     return TPResult::True;
 
+    // WebAssemblyFuncref
+  case tok::kw___funcref:
+    return TPResult::True;
+
     // Borland
   case tok::kw___pascal:
     return TPResult::True;
diff --git a/clang/lib/Sema/AnalysisBasedWarnings.cpp b/clang/lib/Sema/AnalysisBasedWarnings.cpp
index 436743564c258..f5c48ed89e93d 100644
--- a/clang/lib/Sema/AnalysisBasedWarnings.cpp
+++ b/clang/lib/Sema/AnalysisBasedWarnings.cpp
@@ -66,11 +66,17 @@ namespace {
   public:
     UnreachableCodeHandler(Sema &s) : S(s) {}
 
-    void HandleUnreachable(reachable_code::UnreachableKind UK,
-                           SourceLocation L,
-                           SourceRange SilenceableCondVal,
-                           SourceRange R1,
-                           SourceRange R2) override {
+    void HandleUnreachable(reachable_code::UnreachableKind UK, SourceLocation L,
+                           SourceRange SilenceableCondVal, SourceRange R1,
+                           SourceRange R2, bool HasFallThroughAttr) override {
+      // If the diagnosed code is `[[fallthrough]];` and
+      // `-Wunreachable-code-fallthrough` is  enabled, suppress `code will never
+      // be executed` warning to avoid generating diagnostic twice
+      if (HasFallThroughAttr &&
+          !S.getDiagnostics().isIgnored(diag::warn_unreachable_fallthrough_attr,
+                                        SourceLocation()))
+        return;
+
       // Avoid reporting multiple unreachable code diagnostics that are
       // triggered by the same conditional value.
       if (PreviousSilenceableCondVal.isValid() &&
diff --git a/clang/lib/Sema/ParsedAttr.cpp b/clang/lib/Sema/ParsedAttr.cpp
index c1e39acb14ec2..2af688fb58c8a 100644
--- a/clang/lib/Sema/ParsedAttr.cpp
+++ b/clang/lib/Sema/ParsedAttr.cpp
@@ -19,15 +19,12 @@
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
-#include "llvm/Support/ManagedStatic.h"
 #include <cassert>
 #include <cstddef>
 #include <utility>
 
 using namespace clang;
 
-LLVM_INSTANTIATE_REGISTRY(ParsedAttrInfoRegistry)
-
 IdentifierLoc *IdentifierLoc::create(ASTContext &Ctx, SourceLocation Loc,
                                      IdentifierInfo *Ident) {
   IdentifierLoc *Result = new (Ctx) IdentifierLoc;
@@ -120,13 +117,7 @@ const ParsedAttrInfo &ParsedAttrInfo::get(const AttributeCommonInfo &A) {
   if (A.getParsedKind() == AttributeCommonInfo::IgnoredAttribute)
     return IgnoredParsedAttrInfo;
 
-  // Otherwise this may be an attribute defined by a plugin. First instantiate
-  // all plugin attributes if we haven't already done so.
-  static llvm::ManagedStatic<std::list<std::unique_ptr<ParsedAttrInfo>>>
-      PluginAttrInstances;
-  if (PluginAttrInstances->empty())
-    for (auto It : ParsedAttrInfoRegistry::entries())
-      PluginAttrInstances->emplace_back(It.instantiate());
+  // Otherwise this may be an attribute defined by a plugin.
 
   // Search for a ParsedAttrInfo whose name and syntax match.
   std::string FullName = A.getNormalizedFullName();
@@ -134,10 +125,9 @@ const ParsedAttrInfo &ParsedAttrInfo::get(const AttributeCommonInfo &A) {
   if (SyntaxUsed == AttributeCommonInfo::AS_ContextSensitiveKeyword)
     SyntaxUsed = AttributeCommonInfo::AS_Keyword;
 
-  for (auto &Ptr : *PluginAttrInstances)
-    for (auto &S : Ptr->Spellings)
-      if (S.Syntax == SyntaxUsed && S.NormalizedFullName == FullName)
-        return *Ptr;
+  for (auto &Ptr : getAttributePluginInstances())
+    if (Ptr->hasSpelling(SyntaxUsed, FullName))
+      return *Ptr;
 
   // If we failed to find a match then return a default ParsedAttrInfo.
   static const ParsedAttrInfo DefaultParsedAttrInfo(
diff --git a/clang/lib/Sema/Sema.cpp b/clang/lib/Sema/Sema.cpp
index 72575470a78bc..89ac016f60e97 100644
--- a/clang/lib/Sema/Sema.cpp
+++ b/clang/lib/Sema/Sema.cpp
@@ -793,7 +793,7 @@ static bool isFunctionOrVarDeclExternC(NamedDecl *ND) {
 
 /// Determine whether ND is an external-linkage function or variable whose
 /// type has no linkage.
-bool Sema::isExternalWithNoLinkageType(ValueDecl *VD) {
+bool Sema::isExternalWithNoLinkageType(ValueDecl *VD) const {
   // Note: it's not quite enough to check whether VD has UniqueExternalLinkage,
   // because we also want to catch the case where its type has VisibleNoLinkage,
   // which does not affect the linkage of VD.
@@ -1441,7 +1441,7 @@ void Sema::ActOnEndOfTranslationUnit() {
 // Helper functions.
 //===----------------------------------------------------------------------===//
 
-DeclContext *Sema::getFunctionLevelDeclContext(bool AllowLambda) {
+DeclContext *Sema::getFunctionLevelDeclContext(bool AllowLambda) const {
   DeclContext *DC = CurContext;
 
   while (true) {
@@ -1461,7 +1461,7 @@ DeclContext *Sema::getFunctionLevelDeclContext(bool AllowLambda) {
 /// getCurFunctionDecl - If inside of a function body, this returns a pointer
 /// to the function decl for the function being parsed.  If we're currently
 /// in a 'block', this returns the containing context.
-FunctionDecl *Sema::getCurFunctionDecl(bool AllowLambda) {
+FunctionDecl *Sema::getCurFunctionDecl(bool AllowLambda) const {
   DeclContext *DC = getFunctionLevelDeclContext(AllowLambda);
   return dyn_cast<FunctionDecl>(DC);
 }
@@ -1473,7 +1473,7 @@ ObjCMethodDecl *Sema::getCurMethodDecl() {
   return dyn_cast<ObjCMethodDecl>(DC);
 }
 
-NamedDecl *Sema::getCurFunctionOrMethodDecl() {
+NamedDecl *Sema::getCurFunctionOrMethodDecl() const {
   DeclContext *DC = getFunctionLevelDeclContext();
   if (isa<ObjCMethodDecl>(DC) || isa<FunctionDecl>(DC))
     return cast<NamedDecl>(DC);
diff --git a/clang/lib/Sema/SemaAttr.cpp b/clang/lib/Sema/SemaAttr.cpp
index 42f582724564d..3ea97f6aa8f24 100644
--- a/clang/lib/Sema/SemaAttr.cpp
+++ b/clang/lib/Sema/SemaAttr.cpp
@@ -565,13 +565,6 @@ void Sema::ActOnPragmaFloatControl(SourceLocation Loc,
   case PFC_Precise:
     NewFPFeatures.setFPPreciseEnabled(true);
     FpPragmaStack.Act(Loc, Action, StringRef(), NewFPFeatures);
-    if (PP.getCurrentFPEvalMethod() ==
-            LangOptions::FPEvalMethodKind::FEM_Indeterminable &&
-        PP.getLastFPEvalPragmaLocation().isValid())
-      // A preceding `pragma float_control(precise,off)` has changed
-      // the value of the evaluation method.
-      // Set it back to its old value.
-      PP.setCurrentFPEvalMethod(SourceLocation(), PP.getLastFPEvalMethod());
     break;
   case PFC_NoPrecise:
     if (CurFPFeatures.getExceptionMode() == LangOptions::FPE_Strict)
@@ -581,10 +574,6 @@ void Sema::ActOnPragmaFloatControl(SourceLocation Loc,
     else
       NewFPFeatures.setFPPreciseEnabled(false);
     FpPragmaStack.Act(Loc, Action, StringRef(), NewFPFeatures);
-    PP.setLastFPEvalMethod(PP.getCurrentFPEvalMethod());
-    // `AllowFPReassoc` or `AllowReciprocal` option is enabled.
-    PP.setCurrentFPEvalMethod(
-        Loc, LangOptions::FPEvalMethodKind::FEM_Indeterminable);
     break;
   case PFC_Except:
     if (!isPreciseFPEnabled())
@@ -608,12 +597,6 @@ void Sema::ActOnPragmaFloatControl(SourceLocation Loc,
     }
     FpPragmaStack.Act(Loc, Action, StringRef(), NewFPFeatures);
     NewFPFeatures = FpPragmaStack.CurrentValue;
-    if (CurFPFeatures.getAllowFPReassociate() ||
-        CurFPFeatures.getAllowReciprocal())
-      // Since we are popping the pragma, we don't want to be passing
-      // a location here.
-      PP.setCurrentFPEvalMethod(SourceLocation(),
-                                CurFPFeatures.getFPEvalMethod());
     break;
   }
   CurFPFeatures = NewFPFeatures.applyOverrides(getLangOpts());
diff --git a/clang/lib/Sema/SemaCXXScopeSpec.cpp b/clang/lib/Sema/SemaCXXScopeSpec.cpp
index 1ef83ea52e730..759ed6f352c81 100644
--- a/clang/lib/Sema/SemaCXXScopeSpec.cpp
+++ b/clang/lib/Sema/SemaCXXScopeSpec.cpp
@@ -99,34 +99,52 @@ DeclContext *Sema::computeDeclContext(const CXXScopeSpec &SS,
         if (ClassTemplateDecl *ClassTemplate
               = dyn_cast_or_null<ClassTemplateDecl>(
                             SpecType->getTemplateName().getAsTemplateDecl())) {
-          QualType ContextType
-            = Context.getCanonicalType(QualType(SpecType, 0));
-
-          // If the type of the nested name specifier is the same as the
-          // injected class name of the named class template, we're entering
-          // into that class template definition.
-          QualType Injected
-            = ClassTemplate->getInjectedClassNameSpecialization();
-          if (Context.hasSameType(Injected, ContextType))
-            return ClassTemplate->getTemplatedDecl();
+          QualType ContextType =
+              Context.getCanonicalType(QualType(SpecType, 0));
+
+          // FIXME: The fallback on the search of partial
+          // specialization using ContextType should be eventually removed since
+          // it doesn't handle the case of constrained template parameters
+          // correctly. Currently removing this fallback would change the
+          // diagnostic output for invalid code in a number of tests.
+          ClassTemplatePartialSpecializationDecl *PartialSpec = nullptr;
+          ArrayRef<TemplateParameterList *> TemplateParamLists =
+              SS.getTemplateParamLists();
+          if (!TemplateParamLists.empty()) {
+            unsigned Depth = ClassTemplate->getTemplateParameters()->getDepth();
+            auto L = find_if(TemplateParamLists,
+                             [Depth](TemplateParameterList *TPL) {
+                               return TPL->getDepth() == Depth;
+                             });
+            if (L != TemplateParamLists.end()) {
+              void *Pos = nullptr;
+              PartialSpec = ClassTemplate->findPartialSpecialization(
+                  SpecType->template_arguments(), *L, Pos);
+            }
+          } else {
+            PartialSpec = ClassTemplate->findPartialSpecialization(ContextType);
+          }
 
-          // If the type of the nested name specifier is the same as the
-          // type of one of the class template's class template partial
-          // specializations, we're entering into the definition of that
-          // class template partial specialization.
-          if (ClassTemplatePartialSpecializationDecl *PartialSpec
-                = ClassTemplate->findPartialSpecialization(ContextType)) {
+          if (PartialSpec) {
             // A declaration of the partial specialization must be visible.
             // We can always recover here, because this only happens when we're
             // entering the context, and that can't happen in a SFINAE context.
-            assert(!isSFINAEContext() &&
-                   "partial specialization scope specifier in SFINAE context?");
+            assert(!isSFINAEContext() && "partial specialization scope "
+                                         "specifier in SFINAE context?");
             if (!hasReachableDefinition(PartialSpec))
               diagnoseMissingImport(SS.getLastQualifierNameLoc(), PartialSpec,
                                     MissingImportKind::PartialSpecialization,
-                                    /*Recover*/true);
+                                    true);
             return PartialSpec;
           }
+
+          // If the type of the nested name specifier is the same as the
+          // injected class name of the named class template, we're entering
+          // into that class template definition.
+          QualType Injected =
+              ClassTemplate->getInjectedClassNameSpecialization();
+          if (Context.hasSameType(Injected, ContextType))
+            return ClassTemplate->getTemplatedDecl();
         }
       } else if (const RecordType *RecordT = NNSType->getAs<RecordType>()) {
         // The nested name specifier refers to a member of a class template.
diff --git a/clang/lib/Sema/SemaChecking.cpp b/clang/lib/Sema/SemaChecking.cpp
index 485351f157fde..02d1cb010db50 100644
--- a/clang/lib/Sema/SemaChecking.cpp
+++ b/clang/lib/Sema/SemaChecking.cpp
@@ -2146,6 +2146,14 @@ Sema::CheckBuiltinFunctionCall(FunctionDecl *FDecl, unsigned BuiltinID,
       return ExprError();
     break;
 
+  case Builtin::BI__builtin_set_flt_rounds:
+    if (CheckBuiltinTargetInSupported(*this, BuiltinID, TheCall,
+                                      {llvm::Triple::x86, llvm::Triple::x86_64,
+                                       llvm::Triple::arm, llvm::Triple::thumb,
+                                       llvm::Triple::aarch64}))
+      return ExprError();
+    break;
+
   case Builtin::BI__builtin_isgreater:
   case Builtin::BI__builtin_isgreaterequal:
   case Builtin::BI__builtin_isless:
@@ -2614,6 +2622,8 @@ Sema::CheckBuiltinFunctionCall(FunctionDecl *FDecl, unsigned BuiltinID,
   // types only.
   case Builtin::BI__builtin_elementwise_ceil:
   case Builtin::BI__builtin_elementwise_cos:
+  case Builtin::BI__builtin_elementwise_exp:
+  case Builtin::BI__builtin_elementwise_exp2:
   case Builtin::BI__builtin_elementwise_floor:
   case Builtin::BI__builtin_elementwise_log:
   case Builtin::BI__builtin_elementwise_log2:
@@ -4723,6 +4733,8 @@ bool Sema::CheckWebAssemblyBuiltinFunctionCall(const TargetInfo &TI,
   switch (BuiltinID) {
   case WebAssembly::BI__builtin_wasm_ref_null_extern:
     return BuiltinWasmRefNullExtern(TheCall);
+  case WebAssembly::BI__builtin_wasm_ref_null_func:
+    return BuiltinWasmRefNullFunc(TheCall);
   }
 
   return false;
@@ -6757,6 +6769,25 @@ bool Sema::BuiltinWasmRefNullExtern(CallExpr *TheCall) {
   return false;
 }
 
+bool Sema::BuiltinWasmRefNullFunc(CallExpr *TheCall) {
+  if (TheCall->getNumArgs() != 0) {
+    Diag(TheCall->getBeginLoc(), diag::err_typecheck_call_too_many_args)
+         << 0 /*function call*/ << 0 << TheCall->getNumArgs();
+    return true;
+  }
+
+  // This custom type checking code ensures that the nodes are as expected
+  // in order to later on generate the necessary builtin.
+  QualType Pointee = Context.getFunctionType(Context.VoidTy, {}, {});
+  QualType Type = Context.getPointerType(Pointee);
+  Pointee = Context.getAddrSpaceQualType(Pointee, LangAS::wasm_funcref);
+  Type = Context.getAttributedType(attr::WebAssemblyFuncref, Type,
+                                   Context.getPointerType(Pointee));
+  TheCall->setType(Type);
+
+  return false;
+}
+
 /// We have a call to a function like __sync_fetch_and_add, which is an
 /// overloaded function based on the pointer type of its first argument.
 /// The main BuildCallExpr routines have already promoted the types of
diff --git a/clang/lib/Sema/SemaConcept.cpp b/clang/lib/Sema/SemaConcept.cpp
index 4a795227e6293..4ff86843dacc1 100644
--- a/clang/lib/Sema/SemaConcept.cpp
+++ b/clang/lib/Sema/SemaConcept.cpp
@@ -105,27 +105,35 @@ bool Sema::CheckConstraintExpression(const Expr *ConstraintExpression,
   QualType Type = ConstraintExpression->getType();
 
   auto CheckForNonPrimary = [&] {
-    if (PossibleNonPrimary)
-      *PossibleNonPrimary =
-          // We have the following case:
-          // template<typename> requires func(0) struct S { };
-          // The user probably isn't aware of the parentheses required around
-          // the function call, and we're only going to parse 'func' as the
-          // primary-expression, and complain that it is of non-bool type.
-          (NextToken.is(tok::l_paren) &&
-           (IsTrailingRequiresClause ||
-            (Type->isDependentType() &&
-             isa<UnresolvedLookupExpr>(ConstraintExpression)) ||
-            Type->isFunctionType() ||
-            Type->isSpecificBuiltinType(BuiltinType::Overload))) ||
-          // We have the following case:
-          // template<typename T> requires size_<T> == 0 struct S { };
-          // The user probably isn't aware of the parentheses required around
-          // the binary operator, and we're only going to parse 'func' as the
-          // first operand, and complain that it is of non-bool type.
-          getBinOpPrecedence(NextToken.getKind(),
-                             /*GreaterThanIsOperator=*/true,
-                             getLangOpts().CPlusPlus11) > prec::LogicalAnd;
+    if (!PossibleNonPrimary)
+      return;
+
+    *PossibleNonPrimary =
+        // We have the following case:
+        // template<typename> requires func(0) struct S { };
+        // The user probably isn't aware of the parentheses required around
+        // the function call, and we're only going to parse 'func' as the
+        // primary-expression, and complain that it is of non-bool type.
+        //
+        // However, if we're in a lambda, this might also be:
+        // []<typename> requires var () {};
+        // Which also looks like a function call due to the lambda parentheses,
+        // but unlike the first case, isn't an error, so this check is skipped.
+        (NextToken.is(tok::l_paren) &&
+         (IsTrailingRequiresClause ||
+          (Type->isDependentType() &&
+           isa<UnresolvedLookupExpr>(ConstraintExpression) &&
+           !dyn_cast_if_present<LambdaScopeInfo>(getCurFunction())) ||
+          Type->isFunctionType() ||
+          Type->isSpecificBuiltinType(BuiltinType::Overload))) ||
+        // We have the following case:
+        // template<typename T> requires size_<T> == 0 struct S { };
+        // The user probably isn't aware of the parentheses required around
+        // the binary operator, and we're only going to parse 'func' as the
+        // first operand, and complain that it is of non-bool type.
+        getBinOpPrecedence(NextToken.getKind(),
+                           /*GreaterThanIsOperator=*/true,
+                           getLangOpts().CPlusPlus11) > prec::LogicalAnd;
   };
 
   // An atomic constraint!
diff --git a/clang/lib/Sema/SemaDecl.cpp b/clang/lib/Sema/SemaDecl.cpp
index 09540e52892d1..64034393344f0 100644
--- a/clang/lib/Sema/SemaDecl.cpp
+++ b/clang/lib/Sema/SemaDecl.cpp
@@ -1593,7 +1593,7 @@ void Sema::PushOnScopeChains(NamedDecl *D, Scope *S, bool AddToContext) {
 }
 
 bool Sema::isDeclInScope(NamedDecl *D, DeclContext *Ctx, Scope *S,
-                         bool AllowInlineNamespace) {
+                         bool AllowInlineNamespace) const {
   return IdResolver.isDeclInScope(D, Ctx, S, AllowInlineNamespace);
 }
 
@@ -7674,7 +7674,12 @@ NamedDecl *Sema::ActOnVariableDeclarator(
 
     // If we have any template parameter lists that don't directly belong to
     // the variable (matching the scope specifier), store them.
-    unsigned VDTemplateParamLists = TemplateParams ? 1 : 0;
+    // An explicit variable template specialization does not own any template
+    // parameter lists.
+    bool IsExplicitSpecialization =
+        IsVariableTemplateSpecialization && !IsPartialSpecialization;
+    unsigned VDTemplateParamLists =
+        (TemplateParams && !IsExplicitSpecialization) ? 1 : 0;
     if (TemplateParamLists.size() > VDTemplateParamLists)
       NewVD->setTemplateParameterListsInfo(
           Context, TemplateParamLists.drop_back(VDTemplateParamLists));
@@ -9273,6 +9278,12 @@ static OpenCLParamType getOpenCLKernelParameterType(Sema &S, QualType PT) {
           ParamKind == InvalidKernelParam)
         return ParamKind;
 
+      // OpenCL v3.0 s6.11.a:
+      // A restriction to pass pointers to pointers only applies to OpenCL C
+      // v1.2 or below.
+      if (S.getLangOpts().getOpenCLCompatibleVersion() > 120)
+        return ValidKernelParam;
+
       return PtrPtrKernelParam;
     }
 
@@ -9300,6 +9311,11 @@ static OpenCLParamType getOpenCLKernelParameterType(Sema &S, QualType PT) {
       return InvalidKernelParam;
     }
 
+    // OpenCL v1.2 s6.9.p:
+    // A restriction to pass pointers only applies to OpenCL C v1.2 or below.
+    if (S.getLangOpts().getOpenCLCompatibleVersion() > 120)
+      return ValidKernelParam;
+
     return PtrKernelParam;
   }
 
@@ -9366,13 +9382,8 @@ static void checkIsValidOpenCLKernelParameter(
     // OpenCL v3.0 s6.11.a:
     // A kernel function argument cannot be declared as a pointer to a pointer
     // type. [...] This restriction only applies to OpenCL C 1.2 or below.
-    if (S.getLangOpts().getOpenCLCompatibleVersion() <= 120) {
-      S.Diag(Param->getLocation(), diag::err_opencl_ptrptr_kernel_param);
-      D.setInvalidType();
-      return;
-    }
-
-    ValidTypes.insert(PT.getTypePtr());
+    S.Diag(Param->getLocation(), diag::err_opencl_ptrptr_kernel_param);
+    D.setInvalidType();
     return;
 
   case InvalidAddrSpacePtrKernelParam:
@@ -9490,7 +9501,8 @@ static void checkIsValidOpenCLKernelParameter(
       // OpenCL v1.2 s6.9.p:
       // Arguments to kernel functions that are declared to be a struct or union
       // do not allow OpenCL objects to be passed as elements of the struct or
-      // union.
+      // union. This restriction was lifted in OpenCL v2.0 with the introduction
+      // of SVM.
       if (ParamType == PtrKernelParam || ParamType == PtrPtrKernelParam ||
           ParamType == InvalidAddrSpacePtrKernelParam) {
         S.Diag(Param->getLocation(),
@@ -14833,7 +14845,11 @@ ParmVarDecl *Sema::CheckParameter(DeclContext *DC, SourceLocation StartLoc,
       // OpenCL allows function arguments declared to be an array of a type
       // to be qualified with an address space.
       !(getLangOpts().OpenCL &&
-        (T->isArrayType() || T.getAddressSpace() == LangAS::opencl_private))) {
+        (T->isArrayType() || T.getAddressSpace() == LangAS::opencl_private)) &&
+      // WebAssembly allows reference types as parameters. Funcref in particular
+      // lives in a different address space.
+      !(T->isFunctionPointerType() &&
+        T.getAddressSpace() == LangAS::wasm_funcref)) {
     Diag(NameLoc, diag::err_arg_with_address_space);
     New->setInvalidDecl();
   }
diff --git a/clang/lib/Sema/SemaDeclAttr.cpp b/clang/lib/Sema/SemaDeclAttr.cpp
index 5a6e07c81041b..229e73618c53c 100644
--- a/clang/lib/Sema/SemaDeclAttr.cpp
+++ b/clang/lib/Sema/SemaDeclAttr.cpp
@@ -3508,6 +3508,7 @@ bool Sema::checkTargetClonesAttrString(
   enum SecondParam { None, CPU, Tune };
   enum ThirdParam { Target, TargetClones };
   HasCommas = HasCommas || Str.contains(',');
+  const TargetInfo &TInfo = Context.getTargetInfo();
   // Warn on empty at the beginning of a string.
   if (Str.size() == 0)
     return Diag(LiteralLoc, diag::warn_unsupported_target_attribute)
@@ -3517,9 +3518,9 @@ bool Sema::checkTargetClonesAttrString(
   while (!Parts.second.empty()) {
     Parts = Parts.second.split(',');
     StringRef Cur = Parts.first.trim();
-    SourceLocation CurLoc = Literal->getLocationOfByte(
-        Cur.data() - Literal->getString().data(), getSourceManager(),
-        getLangOpts(), Context.getTargetInfo());
+    SourceLocation CurLoc =
+        Literal->getLocationOfByte(Cur.data() - Literal->getString().data(),
+                                   getSourceManager(), getLangOpts(), TInfo);
 
     bool DefaultIsDupe = false;
     bool HasCodeGenImpact = false;
@@ -3527,7 +3528,7 @@ bool Sema::checkTargetClonesAttrString(
       return Diag(CurLoc, diag::warn_unsupported_target_attribute)
              << Unsupported << None << "" << TargetClones;
 
-    if (Context.getTargetInfo().getTriple().isAArch64()) {
+    if (TInfo.getTriple().isAArch64()) {
       // AArch64 target clones specific
       if (Cur == "default") {
         DefaultIsDupe = HasDefault;
@@ -3542,13 +3543,12 @@ bool Sema::checkTargetClonesAttrString(
         while (!CurParts.second.empty()) {
           CurParts = CurParts.second.split('+');
           StringRef CurFeature = CurParts.first.trim();
-          if (!Context.getTargetInfo().validateCpuSupports(CurFeature)) {
+          if (!TInfo.validateCpuSupports(CurFeature)) {
             Diag(CurLoc, diag::warn_unsupported_target_attribute)
                 << Unsupported << None << CurFeature << TargetClones;
             continue;
           }
-          std::string Options;
-          if (Context.getTargetInfo().getFeatureDepOptions(CurFeature, Options))
+          if (TInfo.doesFeatureAffectCodeGen(CurFeature))
             HasCodeGenImpact = true;
           CurFeatures.push_back(CurFeature);
         }
diff --git a/clang/lib/Sema/SemaDeclCXX.cpp b/clang/lib/Sema/SemaDeclCXX.cpp
index 21c02b7ba4aa1..16e1071366b30 100644
--- a/clang/lib/Sema/SemaDeclCXX.cpp
+++ b/clang/lib/Sema/SemaDeclCXX.cpp
@@ -2608,7 +2608,8 @@ Sema::CheckBaseSpecifier(CXXRecordDecl *Class,
   }
 
   // For the MS ABI, propagate DLL attributes to base class templates.
-  if (Context.getTargetInfo().getCXXABI().isMicrosoft()) {
+  if (Context.getTargetInfo().getCXXABI().isMicrosoft() ||
+      Context.getTargetInfo().getTriple().isPS()) {
     if (Attr *ClassAttr = getDLLAttr(Class)) {
       if (auto *BaseTemplate = dyn_cast_or_null<ClassTemplateSpecializationDecl>(
               BaseType->getAsCXXRecordDecl())) {
@@ -9157,7 +9158,18 @@ bool SpecialMemberDeletionInfo::shouldDeleteForSubobjectCall(
     // must be accessible and non-deleted, but need not be trivial. Such a
     // destructor is never actually called, but is semantically checked as
     // if it were.
-    DiagKind = 4;
+    if (CSM == Sema::CXXDefaultConstructor) {
+      // [class.default.ctor]p2:
+      //   A defaulted default constructor for class X is defined as deleted if
+      //   - X is a union that has a variant member with a non-trivial default
+      //     constructor and no variant member of X has a default member
+      //     initializer
+      const auto *RD = cast<CXXRecordDecl>(Field->getParent());
+      if (!RD->hasInClassInitializer())
+        DiagKind = 4;
+    } else {
+      DiagKind = 4;
+    }
   }
 
   if (DiagKind == -1)
diff --git a/clang/lib/Sema/SemaExpr.cpp b/clang/lib/Sema/SemaExpr.cpp
index 56abde04eaf0d..963bd43b6e128 100644
--- a/clang/lib/Sema/SemaExpr.cpp
+++ b/clang/lib/Sema/SemaExpr.cpp
@@ -5917,8 +5917,15 @@ bool Sema::CheckCXXDefaultArgExpr(SourceLocation CallLoc, FunctionDecl *FD,
     assert(!InitWithCleanup->getNumObjects() &&
            "default argument expression has capturing blocks?");
   }
+  // C++ [expr.const]p15.1:
+  //   An expression or conversion is in an immediate function context if it is
+  //   potentially evaluated and [...] its innermost enclosing non-block scope
+  //   is a function parameter scope of an immediate function.
   EnterExpressionEvaluationContext EvalContext(
-      *this, ExpressionEvaluationContext::PotentiallyEvaluated, Param);
+      *this,
+      FD->isConsteval() ? ExpressionEvaluationContext::ImmediateFunctionContext
+                        : ExpressionEvaluationContext::PotentiallyEvaluated,
+      Param);
   ExprEvalContexts.back().IsCurrentlyCheckingDefaultArgumentOrInitializer =
       SkipImmediateInvocations;
   MarkDeclarationsReferencedInExpr(Init, /*SkipLocalVariables*/ true);
@@ -6005,8 +6012,16 @@ ExprResult Sema::BuildCXXDefaultArgExpr(SourceLocation CallLoc,
     // Mark that we are replacing a default argument first.
     // If we are instantiating a template we won't have to
     // retransform immediate calls.
+    // C++ [expr.const]p15.1:
+    //   An expression or conversion is in an immediate function context if it
+    //   is potentially evaluated and [...] its innermost enclosing non-block
+    //   scope is a function parameter scope of an immediate function.
     EnterExpressionEvaluationContext EvalContext(
-        *this, ExpressionEvaluationContext::PotentiallyEvaluated, Param);
+        *this,
+        FD->isConsteval()
+            ? ExpressionEvaluationContext::ImmediateFunctionContext
+            : ExpressionEvaluationContext::PotentiallyEvaluated,
+        Param);
 
     if (Param->hasUninstantiatedDefaultArg()) {
       if (InstantiateDefaultArgument(CallLoc, FD, Param))
@@ -17045,6 +17060,7 @@ ExprResult Sema::ActOnSourceLocExpr(SourceLocExpr::IdentKind Kind,
   QualType ResultTy;
   switch (Kind) {
   case SourceLocExpr::File:
+  case SourceLocExpr::FileName:
   case SourceLocExpr::Function: {
     QualType ArrTy = Context.getStringLiteralArrayType(Context.CharTy, 0);
     ResultTy =
diff --git a/clang/lib/Sema/SemaExprMember.cpp b/clang/lib/Sema/SemaExprMember.cpp
index a3420ac6fdd2d..60b9b013d3f20 100644
--- a/clang/lib/Sema/SemaExprMember.cpp
+++ b/clang/lib/Sema/SemaExprMember.cpp
@@ -161,10 +161,13 @@ static IMAKind ClassifyImplicitMemberAccess(Sema &SemaRef,
   }
 
   CXXRecordDecl *contextClass;
-  if (CXXMethodDecl *MD = dyn_cast<CXXMethodDecl>(DC))
+  if (auto *MD = dyn_cast<CXXMethodDecl>(DC))
     contextClass = MD->getParent()->getCanonicalDecl();
+  else if (auto *RD = dyn_cast<CXXRecordDecl>(DC))
+    contextClass = RD;
   else
-    contextClass = cast<CXXRecordDecl>(DC);
+    return AbstractInstanceResult ? AbstractInstanceResult
+                                  : IMA_Error_StaticContext;
 
   // [class.mfct.non-static]p3:
   // ...is used in the body of a non-static member function of class X,
diff --git a/clang/lib/Sema/SemaOpenMP.cpp b/clang/lib/Sema/SemaOpenMP.cpp
index 20e75fb710676..e193fa3d19d5c 100644
--- a/clang/lib/Sema/SemaOpenMP.cpp
+++ b/clang/lib/Sema/SemaOpenMP.cpp
@@ -2203,11 +2203,14 @@ bool Sema::isOpenMPCapturedByRef(const ValueDecl *D, unsigned Level,
           ++EI;
           if (EI == EE)
             return false;
-
-          if (isa<ArraySubscriptExpr>(EI->getAssociatedExpression()) ||
-              isa<OMPArraySectionExpr>(EI->getAssociatedExpression()) ||
+          auto Last = std::prev(EE);
+          const auto *UO =
+              dyn_cast<UnaryOperator>(Last->getAssociatedExpression());
+          if ((UO && UO->getOpcode() == UO_Deref) ||
+              isa<ArraySubscriptExpr>(Last->getAssociatedExpression()) ||
+              isa<OMPArraySectionExpr>(Last->getAssociatedExpression()) ||
               isa<MemberExpr>(EI->getAssociatedExpression()) ||
-              isa<OMPArrayShapingExpr>(EI->getAssociatedExpression())) {
+              isa<OMPArrayShapingExpr>(Last->getAssociatedExpression())) {
             IsVariableAssociatedWithSection = true;
             // There is nothing more we need to know about this variable.
             return true;
@@ -10178,10 +10181,8 @@ checkOpenMPLoop(OpenMPDirectiveKind DKind, Expr *CollapseLoopCountExpr,
       Built.DependentInits[Cnt] = nullptr;
       Built.FinalsConditions[Cnt] = nullptr;
       if (IS.IsNonRectangularLB || IS.IsNonRectangularUB) {
-        Built.DependentCounters[Cnt] =
-            Built.Counters[NestedLoopCount - 1 - IS.LoopDependentIdx];
-        Built.DependentInits[Cnt] =
-            Built.Inits[NestedLoopCount - 1 - IS.LoopDependentIdx];
+        Built.DependentCounters[Cnt] = Built.Counters[IS.LoopDependentIdx - 1];
+        Built.DependentInits[Cnt] = Built.Inits[IS.LoopDependentIdx - 1];
         Built.FinalsConditions[Cnt] = IS.FinalCondition;
       }
     }
diff --git a/clang/lib/Sema/SemaStmt.cpp b/clang/lib/Sema/SemaStmt.cpp
index f15603fd0bd4a..3e0b3c2b953f4 100644
--- a/clang/lib/Sema/SemaStmt.cpp
+++ b/clang/lib/Sema/SemaStmt.cpp
@@ -4351,9 +4351,9 @@ class CatchHandlerType {
     if (QT->isPointerType())
       IsPointer = true;
 
+    QT = QT.getUnqualifiedType();
     if (IsPointer || QT->isReferenceType())
       QT = QT->getPointeeType();
-    QT = QT.getUnqualifiedType();
   }
 
   /// Used when creating a CatchHandlerType from a base class type; pretends the
@@ -4401,32 +4401,42 @@ template <> struct DenseMapInfo<CatchHandlerType> {
 
 namespace {
 class CatchTypePublicBases {
-  ASTContext &Ctx;
-  const llvm::DenseMap<CatchHandlerType, CXXCatchStmt *> &TypesToCheck;
-  const bool CheckAgainstPointer;
+  const llvm::DenseMap<QualType, CXXCatchStmt *> &TypesToCheck;
 
   CXXCatchStmt *FoundHandler;
-  CanQualType FoundHandlerType;
+  QualType FoundHandlerType;
+  QualType TestAgainstType;
 
 public:
-  CatchTypePublicBases(
-      ASTContext &Ctx,
-      const llvm::DenseMap<CatchHandlerType, CXXCatchStmt *> &T, bool C)
-      : Ctx(Ctx), TypesToCheck(T), CheckAgainstPointer(C),
-        FoundHandler(nullptr) {}
+  CatchTypePublicBases(const llvm::DenseMap<QualType, CXXCatchStmt *> &T,
+                       QualType QT)
+      : TypesToCheck(T), FoundHandler(nullptr), TestAgainstType(QT) {}
 
   CXXCatchStmt *getFoundHandler() const { return FoundHandler; }
-  CanQualType getFoundHandlerType() const { return FoundHandlerType; }
+  QualType getFoundHandlerType() const { return FoundHandlerType; }
 
   bool operator()(const CXXBaseSpecifier *S, CXXBasePath &) {
     if (S->getAccessSpecifier() == AccessSpecifier::AS_public) {
-      CatchHandlerType Check(S->getType(), CheckAgainstPointer);
+      QualType Check = S->getType().getCanonicalType();
       const auto &M = TypesToCheck;
       auto I = M.find(Check);
       if (I != M.end()) {
-        FoundHandler = I->second;
-        FoundHandlerType = Ctx.getCanonicalType(S->getType());
-        return true;
+        // We're pretty sure we found what we need to find. However, we still
+        // need to make sure that we properly compare for pointers and
+        // references, to handle cases like:
+        //
+        // } catch (Base *b) {
+        // } catch (Derived &d) {
+        // }
+        //
+        // where there is a qualification mismatch that disqualifies this
+        // handler as a potential problem.
+        if (I->second->getCaughtType()->isPointerType() ==
+                TestAgainstType->isPointerType()) {
+          FoundHandler = I->second;
+          FoundHandlerType = Check;
+          return true;
+        }
       }
     }
     return false;
@@ -4465,6 +4475,7 @@ StmtResult Sema::ActOnCXXTryBlock(SourceLocation TryLoc, Stmt *TryBlock,
   assert(!Handlers.empty() &&
          "The parser shouldn't call this if there are no handlers.");
 
+  llvm::DenseMap<QualType, CXXCatchStmt *> HandledBaseTypes;
   llvm::DenseMap<CatchHandlerType, CXXCatchStmt *> HandledTypes;
   for (unsigned i = 0; i < NumHandlers; ++i) {
     CXXCatchStmt *H = cast<CXXCatchStmt>(Handlers[i]);
@@ -4482,8 +4493,7 @@ StmtResult Sema::ActOnCXXTryBlock(SourceLocation TryLoc, Stmt *TryBlock,
     // Walk the type hierarchy to diagnose when this type has already been
     // handled (duplication), or cannot be handled (derivation inversion). We
     // ignore top-level cv-qualifiers, per [except.handle]p3
-    CatchHandlerType HandlerCHT =
-        (QualType)Context.getCanonicalType(H->getCaughtType());
+    CatchHandlerType HandlerCHT = H->getCaughtType().getCanonicalType();
 
     // We can ignore whether the type is a reference or a pointer; we need the
     // underlying declaration type in order to get at the underlying record
@@ -4499,10 +4509,12 @@ StmtResult Sema::ActOnCXXTryBlock(SourceLocation TryLoc, Stmt *TryBlock,
       // as the original type.
       CXXBasePaths Paths;
       Paths.setOrigin(RD);
-      CatchTypePublicBases CTPB(Context, HandledTypes, HandlerCHT.isPointer());
+      CatchTypePublicBases CTPB(HandledBaseTypes,
+                                H->getCaughtType().getCanonicalType());
       if (RD->lookupInBases(CTPB, Paths)) {
         const CXXCatchStmt *Problem = CTPB.getFoundHandler();
-        if (!Paths.isAmbiguous(CTPB.getFoundHandlerType())) {
+        if (!Paths.isAmbiguous(
+                CanQualType::CreateUnsafe(CTPB.getFoundHandlerType()))) {
           Diag(H->getExceptionDecl()->getTypeSpecStartLoc(),
                diag::warn_exception_caught_by_earlier_handler)
               << H->getCaughtType();
@@ -4511,11 +4523,16 @@ StmtResult Sema::ActOnCXXTryBlock(SourceLocation TryLoc, Stmt *TryBlock,
               << Problem->getCaughtType();
         }
       }
+      // Strip the qualifiers here because we're going to be comparing this
+      // type to the base type specifiers of a class, which are ignored in a
+      // base specifier per [class.derived.general]p2.
+      HandledBaseTypes[Underlying.getUnqualifiedType()] = H;
     }
 
     // Add the type the list of ones we have handled; diagnose if we've already
     // handled it.
-    auto R = HandledTypes.insert(std::make_pair(H->getCaughtType(), H));
+    auto R = HandledTypes.insert(
+        std::make_pair(H->getCaughtType().getCanonicalType(), H));
     if (!R.second) {
       const CXXCatchStmt *Problem = R.first->second;
       Diag(H->getExceptionDecl()->getTypeSpecStartLoc(),
diff --git a/clang/lib/Sema/SemaStmtAttr.cpp b/clang/lib/Sema/SemaStmtAttr.cpp
index 6d443837a4c57..eeef85373ccb1 100644
--- a/clang/lib/Sema/SemaStmtAttr.cpp
+++ b/clang/lib/Sema/SemaStmtAttr.cpp
@@ -233,7 +233,8 @@ static Attr *handleNoInlineAttr(Sema &S, Stmt *St, const ParsedAttr &A,
 
   for (const auto *CallExpr : CEF.getCallExprs()) {
     const Decl *Decl = CallExpr->getCalleeDecl();
-    if (Decl->hasAttr<AlwaysInlineAttr>() || Decl->hasAttr<FlattenAttr>())
+    if (Decl &&
+        (Decl->hasAttr<AlwaysInlineAttr>() || Decl->hasAttr<FlattenAttr>()))
       S.Diag(St->getBeginLoc(), diag::warn_function_stmt_attribute_precedence)
           << A << (Decl->hasAttr<AlwaysInlineAttr>() ? 0 : 1);
   }
@@ -259,7 +260,7 @@ static Attr *handleAlwaysInlineAttr(Sema &S, Stmt *St, const ParsedAttr &A,
 
   for (const auto *CallExpr : CEF.getCallExprs()) {
     const Decl *Decl = CallExpr->getCalleeDecl();
-    if (Decl->hasAttr<NoInlineAttr>() || Decl->hasAttr<FlattenAttr>())
+    if (Decl && (Decl->hasAttr<NoInlineAttr>() || Decl->hasAttr<FlattenAttr>()))
       S.Diag(St->getBeginLoc(), diag::warn_function_stmt_attribute_precedence)
           << A << (Decl->hasAttr<NoInlineAttr>() ? 2 : 1);
   }
diff --git a/clang/lib/Sema/SemaTemplateDeduction.cpp b/clang/lib/Sema/SemaTemplateDeduction.cpp
index 9e48a2a35a344..1fe2d3fac6859 100644
--- a/clang/lib/Sema/SemaTemplateDeduction.cpp
+++ b/clang/lib/Sema/SemaTemplateDeduction.cpp
@@ -756,11 +756,8 @@ class PackDeductionScope {
       SmallVector<UnexpandedParameterPack, 2> Unexpanded;
       S.collectUnexpandedParameterPacks(Pattern, Unexpanded);
       for (unsigned I = 0, N = Unexpanded.size(); I != N; ++I) {
-        UnexpandedParameterPack U = Unexpanded[I];
-        if (U.first.is<const SubstTemplateTypeParmPackType *>() ||
-            U.first.is<const SubstNonTypeTemplateParmPackExpr *>())
-          continue;
-        auto [Depth, Index] = getDepthAndIndex(U);
+        unsigned Depth, Index;
+        std::tie(Depth, Index) = getDepthAndIndex(Unexpanded[I]);
         if (Depth == Info.getDeducedDepth())
           AddPack(Index);
       }
diff --git a/clang/lib/Sema/SemaTemplateVariadic.cpp b/clang/lib/Sema/SemaTemplateVariadic.cpp
index 01a435668d883..dfcc78dafdc4c 100644
--- a/clang/lib/Sema/SemaTemplateVariadic.cpp
+++ b/clang/lib/Sema/SemaTemplateVariadic.cpp
@@ -89,23 +89,6 @@ namespace {
       return true;
     }
 
-    bool
-    VisitSubstTemplateTypeParmPackTypeLoc(SubstTemplateTypeParmPackTypeLoc TL) {
-      Unexpanded.push_back({TL.getTypePtr(), TL.getNameLoc()});
-      return true;
-    }
-
-    bool VisitSubstTemplateTypeParmPackType(SubstTemplateTypeParmPackType *T) {
-      Unexpanded.push_back({T, SourceLocation()});
-      return true;
-    }
-
-    bool
-    VisitSubstNonTypeTemplateParmPackExpr(SubstNonTypeTemplateParmPackExpr *E) {
-      Unexpanded.push_back({E, E->getParameterPackLocation()});
-      return true;
-    }
-
     /// Record occurrences of function and non-type template
     /// parameter packs in an expression.
     bool VisitDeclRefExpr(DeclRefExpr *E) {
@@ -324,8 +307,7 @@ Sema::DiagnoseUnexpandedParameterPacks(SourceLocation Loc,
           auto *TTPD = dyn_cast<TemplateTypeParmDecl>(LocalPack);
           return TTPD && TTPD->getTypeForDecl() == TTPT;
         }
-        return declaresSameEntity(Pack.first.get<const NamedDecl *>(),
-                                  LocalPack);
+        return declaresSameEntity(Pack.first.get<NamedDecl *>(), LocalPack);
       };
       if (llvm::any_of(LSI->LocalPacks, DeclaresThisPack))
         LambdaParamPackReferences.push_back(Pack);
@@ -377,7 +359,7 @@ Sema::DiagnoseUnexpandedParameterPacks(SourceLocation Loc,
           = Unexpanded[I].first.dyn_cast<const TemplateTypeParmType *>())
       Name = TTP->getIdentifier();
     else
-      Name = Unexpanded[I].first.get<const NamedDecl *>()->getIdentifier();
+      Name = Unexpanded[I].first.get<NamedDecl *>()->getIdentifier();
 
     if (Name && NamesKnown.insert(Name).second)
       Names.push_back(Name);
@@ -440,7 +422,7 @@ bool Sema::DiagnoseUnexpandedParameterPackInRequiresExpr(RequiresExpr *RE) {
   llvm::SmallPtrSet<NamedDecl*, 8> ParmSet(Parms.begin(), Parms.end());
   SmallVector<UnexpandedParameterPack, 2> UnexpandedParms;
   for (auto Parm : Unexpanded)
-    if (ParmSet.contains(Parm.first.dyn_cast<const NamedDecl *>()))
+    if (ParmSet.contains(Parm.first.dyn_cast<NamedDecl *>()))
       UnexpandedParms.push_back(Parm);
   if (UnexpandedParms.empty())
     return false;
@@ -692,95 +674,109 @@ bool Sema::CheckParameterPacksForExpansion(
     bool &RetainExpansion, std::optional<unsigned> &NumExpansions) {
   ShouldExpand = true;
   RetainExpansion = false;
-  std::pair<const IdentifierInfo *, SourceLocation> FirstPack;
-  std::optional<std::pair<unsigned, SourceLocation>> PartialExpansion;
-  std::optional<unsigned> CurNumExpansions;
+  std::pair<IdentifierInfo *, SourceLocation> FirstPack;
+  bool HaveFirstPack = false;
+  std::optional<unsigned> NumPartialExpansions;
+  SourceLocation PartiallySubstitutedPackLoc;
 
-  for (auto [P, Loc] : Unexpanded) {
+  for (UnexpandedParameterPack ParmPack : Unexpanded) {
     // Compute the depth and index for this parameter pack.
-    std::optional<std::pair<unsigned, unsigned>> Pos;
+    unsigned Depth = 0, Index = 0;
+    IdentifierInfo *Name;
+    bool IsVarDeclPack = false;
+
+    if (const TemplateTypeParmType *TTP =
+            ParmPack.first.dyn_cast<const TemplateTypeParmType *>()) {
+      Depth = TTP->getDepth();
+      Index = TTP->getIndex();
+      Name = TTP->getIdentifier();
+    } else {
+      NamedDecl *ND = ParmPack.first.get<NamedDecl *>();
+      if (isa<VarDecl>(ND))
+        IsVarDeclPack = true;
+      else
+        std::tie(Depth, Index) = getDepthAndIndex(ND);
+
+      Name = ND->getIdentifier();
+    }
+
+    // Determine the size of this argument pack.
     unsigned NewPackSize;
-    const auto *ND = P.dyn_cast<const NamedDecl *>();
-    if (ND && isa<VarDecl>(ND)) {
-      const auto *DAP =
-          CurrentInstantiationScope->findInstantiationOf(ND)
-              ->dyn_cast<LocalInstantiationScope::DeclArgumentPack *>();
-      if (!DAP) {
+    if (IsVarDeclPack) {
+      // Figure out whether we're instantiating to an argument pack or not.
+      typedef LocalInstantiationScope::DeclArgumentPack DeclArgumentPack;
+
+      llvm::PointerUnion<Decl *, DeclArgumentPack *> *Instantiation =
+          CurrentInstantiationScope->findInstantiationOf(
+              ParmPack.first.get<NamedDecl *>());
+      if (Instantiation->is<DeclArgumentPack *>()) {
+        // We could expand this function parameter pack.
+        NewPackSize = Instantiation->get<DeclArgumentPack *>()->size();
+      } else {
         // We can't expand this function parameter pack, so we can't expand
         // the pack expansion.
         ShouldExpand = false;
         continue;
       }
-      NewPackSize = DAP->size();
-    } else if (ND) {
-      Pos = getDepthAndIndex(ND);
-    } else if (const auto *TTP = P.dyn_cast<const TemplateTypeParmType *>()) {
-      Pos = {TTP->getDepth(), TTP->getIndex()};
-      ND = TTP->getDecl();
-      // FIXME: We either should have some fallback for canonical TTP, or
-      //        never have canonical TTP here.
-    } else if (const auto *STP =
-                   P.dyn_cast<const SubstTemplateTypeParmPackType *>()) {
-      NewPackSize = STP->getNumArgs();
-      ND = STP->getReplacedParameter();
     } else {
-      const auto *SEP = P.get<const SubstNonTypeTemplateParmPackExpr *>();
-      NewPackSize = SEP->getArgumentPack().pack_size();
-      ND = SEP->getParameterPack();
-    }
-
-    if (Pos) {
       // If we don't have a template argument at this depth/index, then we
       // cannot expand the pack expansion. Make a note of this, but we still
       // want to check any parameter packs we *do* have arguments for.
-      if (Pos->first >= TemplateArgs.getNumLevels() ||
-          !TemplateArgs.hasTemplateArgument(Pos->first, Pos->second)) {
+      if (Depth >= TemplateArgs.getNumLevels() ||
+          !TemplateArgs.hasTemplateArgument(Depth, Index)) {
         ShouldExpand = false;
         continue;
       }
+
       // Determine the size of the argument pack.
-      NewPackSize = TemplateArgs(Pos->first, Pos->second).pack_size();
-      // C++0x [temp.arg.explicit]p9:
-      //   Template argument deduction can extend the sequence of template
-      //   arguments corresponding to a template parameter pack, even when the
-      //   sequence contains explicitly specified template arguments.
-      if (CurrentInstantiationScope)
-        if (const NamedDecl *PartialPack =
-                CurrentInstantiationScope->getPartiallySubstitutedPack();
-            PartialPack && getDepthAndIndex(PartialPack) == *Pos) {
+      NewPackSize = TemplateArgs(Depth, Index).pack_size();
+    }
+
+    // C++0x [temp.arg.explicit]p9:
+    //   Template argument deduction can extend the sequence of template
+    //   arguments corresponding to a template parameter pack, even when the
+    //   sequence contains explicitly specified template arguments.
+    if (!IsVarDeclPack && CurrentInstantiationScope) {
+      if (NamedDecl *PartialPack =
+              CurrentInstantiationScope->getPartiallySubstitutedPack()) {
+        unsigned PartialDepth, PartialIndex;
+        std::tie(PartialDepth, PartialIndex) = getDepthAndIndex(PartialPack);
+        if (PartialDepth == Depth && PartialIndex == Index) {
           RetainExpansion = true;
           // We don't actually know the new pack size yet.
-          PartialExpansion = {NewPackSize, Loc};
+          NumPartialExpansions = NewPackSize;
+          PartiallySubstitutedPackLoc = ParmPack.second;
           continue;
         }
+      }
     }
 
-    // FIXME: Workaround for Canonical TTP.
-    const IdentifierInfo *Name = ND ? ND->getIdentifier() : nullptr;
-    if (!CurNumExpansions) {
+    if (!NumExpansions) {
       // The is the first pack we've seen for which we have an argument.
       // Record it.
-      CurNumExpansions = NewPackSize;
-      FirstPack = {Name, Loc};
-    } else if (NewPackSize != *CurNumExpansions) {
+      NumExpansions = NewPackSize;
+      FirstPack.first = Name;
+      FirstPack.second = ParmPack.second;
+      HaveFirstPack = true;
+      continue;
+    }
+
+    if (NewPackSize != *NumExpansions) {
       // C++0x [temp.variadic]p5:
       //   All of the parameter packs expanded by a pack expansion shall have
       //   the same number of arguments specified.
-      Diag(EllipsisLoc, diag::err_pack_expansion_length_conflict)
-          << FirstPack.first << Name << *CurNumExpansions << NewPackSize
-          << SourceRange(FirstPack.second) << SourceRange(Loc);
+      if (HaveFirstPack)
+        Diag(EllipsisLoc, diag::err_pack_expansion_length_conflict)
+            << FirstPack.first << Name << *NumExpansions << NewPackSize
+            << SourceRange(FirstPack.second) << SourceRange(ParmPack.second);
+      else
+        Diag(EllipsisLoc, diag::err_pack_expansion_length_conflict_multilevel)
+            << Name << *NumExpansions << NewPackSize
+            << SourceRange(ParmPack.second);
       return true;
     }
   }
 
-  if (NumExpansions && CurNumExpansions &&
-      *NumExpansions != *CurNumExpansions) {
-    Diag(EllipsisLoc, diag::err_pack_expansion_length_conflict_multilevel)
-        << FirstPack.first << *CurNumExpansions << *NumExpansions
-        << SourceRange(FirstPack.second);
-    return true;
-  }
-
   // If we're performing a partial expansion but we also have a full expansion,
   // expand to the number of common arguments. For example, given:
   //
@@ -790,18 +786,17 @@ bool Sema::CheckParameterPacksForExpansion(
   //
   // ... a call to 'A<int, int>().f<int>' should expand the pack once and
   // retain an expansion.
-  if (PartialExpansion) {
-    if (CurNumExpansions && *CurNumExpansions < PartialExpansion->first) {
+  if (NumPartialExpansions) {
+    if (NumExpansions && *NumExpansions < *NumPartialExpansions) {
       NamedDecl *PartialPack =
           CurrentInstantiationScope->getPartiallySubstitutedPack();
       Diag(EllipsisLoc, diag::err_pack_expansion_length_conflict_partial)
-          << PartialPack << PartialExpansion->first << *CurNumExpansions
-          << SourceRange(PartialExpansion->second);
+          << PartialPack << *NumPartialExpansions << *NumExpansions
+          << SourceRange(PartiallySubstitutedPackLoc);
       return true;
     }
-    NumExpansions = PartialExpansion->first;
-  } else {
-    NumExpansions = CurNumExpansions;
+
+    NumExpansions = NumPartialExpansions;
   }
 
   return false;
@@ -814,48 +809,47 @@ std::optional<unsigned> Sema::getNumArgumentsInExpansion(
   CollectUnexpandedParameterPacksVisitor(Unexpanded).TraverseType(Pattern);
 
   std::optional<unsigned> Result;
-  auto setResultSz = [&Result](unsigned Size) {
-    assert((!Result || *Result == Size) && "inconsistent pack sizes");
-    Result = Size;
-  };
-  auto setResultPos = [&](const std::pair<unsigned, unsigned> &Pos) -> bool {
-    unsigned Depth = Pos.first, Index = Pos.second;
-    if (Depth >= TemplateArgs.getNumLevels() ||
-        !TemplateArgs.hasTemplateArgument(Depth, Index))
-      // The pattern refers to an unknown template argument. We're not ready to
-      // expand this pack yet.
-      return true;
-    // Determine the size of the argument pack.
-    setResultSz(TemplateArgs(Depth, Index).pack_size());
-    return false;
-  };
+  for (unsigned I = 0, N = Unexpanded.size(); I != N; ++I) {
+    // Compute the depth and index for this parameter pack.
+    unsigned Depth;
+    unsigned Index;
 
-  for (auto [I, _] : Unexpanded) {
-    if (const auto *TTP = I.dyn_cast<const TemplateTypeParmType *>()) {
-      if (setResultPos({TTP->getDepth(), TTP->getIndex()}))
-        return std::nullopt;
-    } else if (const auto *STP =
-                   I.dyn_cast<const SubstTemplateTypeParmPackType *>()) {
-      setResultSz(STP->getNumArgs());
-    } else if (const auto *SEP =
-                   I.dyn_cast<const SubstNonTypeTemplateParmPackExpr *>()) {
-      setResultSz(SEP->getArgumentPack().pack_size());
+    if (const TemplateTypeParmType *TTP =
+            Unexpanded[I].first.dyn_cast<const TemplateTypeParmType *>()) {
+      Depth = TTP->getDepth();
+      Index = TTP->getIndex();
     } else {
-      const auto *ND = I.get<const NamedDecl *>();
-      // Function parameter pack or init-capture pack.
+      NamedDecl *ND = Unexpanded[I].first.get<NamedDecl *>();
       if (isa<VarDecl>(ND)) {
-        const auto *DAP =
-            CurrentInstantiationScope->findInstantiationOf(ND)
-                ->dyn_cast<LocalInstantiationScope::DeclArgumentPack *>();
-        if (!DAP)
+        // Function parameter pack or init-capture pack.
+        typedef LocalInstantiationScope::DeclArgumentPack DeclArgumentPack;
+
+        llvm::PointerUnion<Decl *, DeclArgumentPack *> *Instantiation =
+            CurrentInstantiationScope->findInstantiationOf(
+                Unexpanded[I].first.get<NamedDecl *>());
+        if (Instantiation->is<Decl *>())
           // The pattern refers to an unexpanded pack. We're not ready to expand
           // this pack yet.
           return std::nullopt;
-        setResultSz(DAP->size());
-      } else if (setResultPos(getDepthAndIndex(ND))) {
-        return std::nullopt;
+
+        unsigned Size = Instantiation->get<DeclArgumentPack *>()->size();
+        assert((!Result || *Result == Size) && "inconsistent pack sizes");
+        Result = Size;
+        continue;
       }
+
+      std::tie(Depth, Index) = getDepthAndIndex(ND);
     }
+    if (Depth >= TemplateArgs.getNumLevels() ||
+        !TemplateArgs.hasTemplateArgument(Depth, Index))
+      // The pattern refers to an unknown template argument. We're not ready to
+      // expand this pack yet.
+      return std::nullopt;
+
+    // Determine the size of the argument pack.
+    unsigned Size = TemplateArgs(Depth, Index).pack_size();
+    assert((!Result || *Result == Size) && "inconsistent pack sizes");
+    Result = Size;
   }
 
   return Result;
@@ -1226,10 +1220,11 @@ ExprResult Sema::ActOnCXXFoldExpr(Scope *S, SourceLocation LParenLoc, Expr *LHS,
   if (!LHS || !RHS) {
     Expr *Pack = LHS ? LHS : RHS;
     assert(Pack && "fold expression with neither LHS nor RHS");
-    DiscardOperands();
-    if (!Pack->containsUnexpandedParameterPack())
+    if (!Pack->containsUnexpandedParameterPack()) {
+      DiscardOperands();
       return Diag(EllipsisLoc, diag::err_pack_expansion_without_parameter_packs)
              << Pack->getSourceRange();
+    }
   }
 
   BinaryOperatorKind Opc = ConvertTokenKindToBinaryOpcode(Operator);
diff --git a/clang/lib/Sema/SemaType.cpp b/clang/lib/Sema/SemaType.cpp
index 54ce3ec806f57..53852dd930a71 100644
--- a/clang/lib/Sema/SemaType.cpp
+++ b/clang/lib/Sema/SemaType.cpp
@@ -7346,6 +7346,37 @@ static bool handleMSPointerTypeQualifierAttr(TypeProcessingState &State,
   return false;
 }
 
+static bool HandleWebAssemblyFuncrefAttr(TypeProcessingState &State,
+                                         QualType &QT, ParsedAttr &PAttr) {
+  assert(PAttr.getKind() == ParsedAttr::AT_WebAssemblyFuncref);
+
+  Sema &S = State.getSema();
+  Attr *A = createSimpleAttr<WebAssemblyFuncrefAttr>(S.Context, PAttr);
+
+  std::bitset<attr::LastAttr> Attrs;
+  attr::Kind NewAttrKind = A->getKind();
+  const auto *AT = dyn_cast<AttributedType>(QT);
+  while (AT) {
+    Attrs[AT->getAttrKind()] = true;
+    AT = dyn_cast<AttributedType>(AT->getModifiedType());
+  }
+
+  // You cannot specify duplicate type attributes, so if the attribute has
+  // already been applied, flag it.
+  if (Attrs[NewAttrKind]) {
+    S.Diag(PAttr.getLoc(), diag::warn_duplicate_attribute_exact) << PAttr;
+    return true;
+  }
+
+  // Add address space to type based on its attributes.
+  LangAS ASIdx = LangAS::wasm_funcref;
+  QualType Pointee = QT->getPointeeType();
+  Pointee = S.Context.getAddrSpaceQualType(
+      S.Context.removeAddrSpaceQualType(Pointee), ASIdx);
+  QT = State.getAttributedType(A, QT, S.Context.getPointerType(Pointee));
+  return false;
+}
+
 /// Map a nullability attribute kind to a nullability kind.
 static NullabilityKind mapNullabilityAttrKind(ParsedAttr::Kind kind) {
   switch (kind) {
@@ -8503,6 +8534,12 @@ static void processTypeAttrs(TypeProcessingState &state, QualType &type,
       attr.setUsedAsTypeAttr();
       break;
 
+    case ParsedAttr::AT_WebAssemblyFuncref: {
+      if (!HandleWebAssemblyFuncrefAttr(state, type, attr))
+        attr.setUsedAsTypeAttr();
+      break;
+    }
+
     MS_TYPE_ATTRS_CASELIST:
       if (!handleMSPointerTypeQualifierAttr(state, attr, type))
         attr.setUsedAsTypeAttr();
diff --git a/clang/lib/Sema/TreeTransform.h b/clang/lib/Sema/TreeTransform.h
index 44e4a14b487be..21789f96f9154 100644
--- a/clang/lib/Sema/TreeTransform.h
+++ b/clang/lib/Sema/TreeTransform.h
@@ -2803,6 +2803,21 @@ class TreeTransform {
     R.addDecl(FoundDecl);
     R.resolveKind();
 
+    if (getSema().isUnevaluatedContext() && Base->isImplicitCXXThis() &&
+        isa<FieldDecl, IndirectFieldDecl, MSPropertyDecl>(Member)) {
+      if (auto *ThisClass = cast<CXXThisExpr>(Base)
+                                ->getType()
+                                ->getPointeeType()
+                                ->getAsCXXRecordDecl()) {
+        auto *Class = cast<CXXRecordDecl>(Member->getDeclContext());
+        // In unevaluated contexts, an expression supposed to be a member access
+        // might reference a member in an unrelated class.
+        if (!ThisClass->Equals(Class) && !ThisClass->isDerivedFrom(Class))
+          return getSema().BuildDeclRefExpr(Member, Member->getType(),
+                                            VK_LValue, Member->getLocation());
+      }
+    }
+
     return getSema().BuildMemberReferenceExpr(Base, BaseType, OpLoc, isArrow,
                                               SS, TemplateKWLoc,
                                               FirstQualifierInScope,
@@ -5897,7 +5912,6 @@ bool TreeTransform<Derived>::TransformFunctionTypeParams(
                                        = dyn_cast<PackExpansionType>(OldType)) {
       // We have a function parameter pack that may need to be expanded.
       QualType Pattern = Expansion->getPattern();
-      NumExpansions = Expansion->getNumExpansions();
       SmallVector<UnexpandedParameterPack, 2> Unexpanded;
       getSema().collectUnexpandedParameterPacks(Pattern, Unexpanded);
 
diff --git a/clang/lib/Serialization/ASTReader.cpp b/clang/lib/Serialization/ASTReader.cpp
index 96ca312e7cbdf..0273fa1b839a5 100644
--- a/clang/lib/Serialization/ASTReader.cpp
+++ b/clang/lib/Serialization/ASTReader.cpp
@@ -4134,7 +4134,7 @@ void ASTReader::makeModuleVisible(Module *Mod,
       auto HiddenNames = std::move(*Hidden);
       HiddenNamesMap.erase(Hidden);
       makeNamesVisible(HiddenNames.second, HiddenNames.first);
-      assert(HiddenNamesMap.find(Mod) == HiddenNamesMap.end() &&
+      assert(!HiddenNamesMap.contains(Mod) &&
              "making names visible added hidden names");
     }
 
diff --git a/clang/lib/Serialization/ASTReaderDecl.cpp b/clang/lib/Serialization/ASTReaderDecl.cpp
index b9bbc0ec9eb28..1c449bc71bfd8 100644
--- a/clang/lib/Serialization/ASTReaderDecl.cpp
+++ b/clang/lib/Serialization/ASTReaderDecl.cpp
@@ -1977,7 +1977,7 @@ void ASTDeclReader::MergeDefinitionData(
     Reader.PendingDefinitions.erase(MergeDD.Definition);
     MergeDD.Definition->setCompleteDefinition(false);
     Reader.mergeDefinitionVisibility(DD.Definition, MergeDD.Definition);
-    assert(Reader.Lookups.find(MergeDD.Definition) == Reader.Lookups.end() &&
+    assert(!Reader.Lookups.contains(MergeDD.Definition) &&
            "already loaded pending lookups for merged definition");
   }
 
diff --git a/clang/lib/Serialization/ASTReaderStmt.cpp b/clang/lib/Serialization/ASTReaderStmt.cpp
index 46d653c7f9407..56b30d1314dc3 100644
--- a/clang/lib/Serialization/ASTReaderStmt.cpp
+++ b/clang/lib/Serialization/ASTReaderStmt.cpp
@@ -2805,7 +2805,7 @@ Stmt *ASTReader::ReadStmtFromStream(ModuleFile &F) {
 
     case STMT_REF_PTR:
       IsStmtReference = true;
-      assert(StmtEntries.find(Record[0]) != StmtEntries.end() &&
+      assert(StmtEntries.contains(Record[0]) &&
              "No stmt was recorded for this offset reference!");
       S = StmtEntries[Record.readInt()];
       break;
diff --git a/clang/lib/Serialization/ASTWriter.cpp b/clang/lib/Serialization/ASTWriter.cpp
index c1afdeb6007db..f5691e99241c1 100644
--- a/clang/lib/Serialization/ASTWriter.cpp
+++ b/clang/lib/Serialization/ASTWriter.cpp
@@ -5109,7 +5109,7 @@ ASTFileSignature ASTWriter::WriteASTCore(Sema &SemaRef, StringRef isysroot,
     };
     llvm::SmallVector<ModuleInfo, 64> Imports;
     for (const auto *I : Context.local_imports()) {
-      assert(SubmoduleIDs.find(I->getImportedModule()) != SubmoduleIDs.end());
+      assert(SubmoduleIDs.contains(I->getImportedModule()));
       Imports.push_back(ModuleInfo(SubmoduleIDs[I->getImportedModule()],
                          I->getImportedModule()));
     }
@@ -5457,7 +5457,7 @@ MacroID ASTWriter::getMacroID(MacroInfo *MI) {
   if (!MI || MI->isBuiltinMacro())
     return 0;
 
-  assert(MacroIDs.find(MI) != MacroIDs.end() && "Macro not emitted!");
+  assert(MacroIDs.contains(MI) && "Macro not emitted!");
   return MacroIDs[MI];
 }
 
@@ -5632,7 +5632,7 @@ DeclID ASTWriter::getDeclID(const Decl *D) {
   if (D->isFromASTFile())
     return D->getGlobalID();
 
-  assert(DeclIDs.find(D) != DeclIDs.end() && "Declaration not emitted!");
+  assert(DeclIDs.contains(D) && "Declaration not emitted!");
   return DeclIDs[D];
 }
 
@@ -6057,12 +6057,12 @@ void ASTWriter::SelectorRead(SelectorID ID, Selector S) {
 
 void ASTWriter::MacroDefinitionRead(serialization::PreprocessedEntityID ID,
                                     MacroDefinitionRecord *MD) {
-  assert(MacroDefinitions.find(MD) == MacroDefinitions.end());
+  assert(!MacroDefinitions.contains(MD));
   MacroDefinitions[MD] = ID;
 }
 
 void ASTWriter::ModuleRead(serialization::SubmoduleID ID, Module *Mod) {
-  assert(SubmoduleIDs.find(Mod) == SubmoduleIDs.end());
+  assert(!SubmoduleIDs.contains(Mod));
   SubmoduleIDs[Mod] = ID;
 }
 
diff --git a/clang/lib/Serialization/ASTWriterDecl.cpp b/clang/lib/Serialization/ASTWriterDecl.cpp
index 69d192612bccf..bbd3c36df2f96 100644
--- a/clang/lib/Serialization/ASTWriterDecl.cpp
+++ b/clang/lib/Serialization/ASTWriterDecl.cpp
@@ -2468,7 +2468,13 @@ void ASTWriter::WriteDeclAbbrevs() {
 /// relatively painless since they would presumably only do it for top-level
 /// decls.
 static bool isRequiredDecl(const Decl *D, ASTContext &Context,
-                           bool WritingModule) {
+                           Module *WritingModule) {
+  // Named modules have different semantics than header modules. Every named
+  // module units owns a translation unit. So the importer of named modules
+  // doesn't need to deserilize everything ahead of time.
+  if (WritingModule && WritingModule->isModulePurview())
+    return false;
+
   // An ObjCMethodDecl is never considered as "required" because its
   // implementation container always is.
 
diff --git a/clang/lib/Serialization/ASTWriterStmt.cpp b/clang/lib/Serialization/ASTWriterStmt.cpp
index b35a7cee5af20..37e0e3f60719f 100644
--- a/clang/lib/Serialization/ASTWriterStmt.cpp
+++ b/clang/lib/Serialization/ASTWriterStmt.cpp
@@ -2698,16 +2698,14 @@ void ASTStmtWriter::VisitOMPTargetParallelGenericLoopDirective(
 //===----------------------------------------------------------------------===//
 
 unsigned ASTWriter::RecordSwitchCaseID(SwitchCase *S) {
-  assert(SwitchCaseIDs.find(S) == SwitchCaseIDs.end() &&
-         "SwitchCase recorded twice");
+  assert(!SwitchCaseIDs.contains(S) && "SwitchCase recorded twice");
   unsigned NextID = SwitchCaseIDs.size();
   SwitchCaseIDs[S] = NextID;
   return NextID;
 }
 
 unsigned ASTWriter::getSwitchCaseID(SwitchCase *S) {
-  assert(SwitchCaseIDs.find(S) != SwitchCaseIDs.end() &&
-         "SwitchCase hasn't been seen yet");
+  assert(SwitchCaseIDs.contains(S) && "SwitchCase hasn't been seen yet");
   return SwitchCaseIDs[S];
 }
 
diff --git a/clang/lib/StaticAnalyzer/Checkers/StdLibraryFunctionsChecker.cpp b/clang/lib/StaticAnalyzer/Checkers/StdLibraryFunctionsChecker.cpp
index b36f72f2653c7..98f5540cef709 100644
--- a/clang/lib/StaticAnalyzer/Checkers/StdLibraryFunctionsChecker.cpp
+++ b/clang/lib/StaticAnalyzer/Checkers/StdLibraryFunctionsChecker.cpp
@@ -65,47 +65,68 @@ class StdLibraryFunctionsChecker
   class Summary;
 
   /// Specify how much the analyzer engine should entrust modeling this function
-  /// to us. If he doesn't, he performs additional invalidations.
-  enum InvalidationKind { NoEvalCall, EvalCallAsPure };
+  /// to us.
+  enum InvalidationKind {
+    /// No \c eval::Call for the function, it can be modeled elsewhere.
+    /// This checker checks only pre and post conditions.
+    NoEvalCall,
+    /// The function is modeled completely in this checker.
+    EvalCallAsPure
+  };
 
-  // The universal integral type to use in value range descriptions.
-  // Unsigned to make sure overflows are well-defined.
+  /// Given a range, should the argument stay inside or outside this range?
+  enum RangeKind { OutOfRange, WithinRange };
+
+  static RangeKind negateKind(RangeKind K) {
+    switch (K) {
+    case OutOfRange:
+      return WithinRange;
+    case WithinRange:
+      return OutOfRange;
+    }
+    llvm_unreachable("Unknown range kind");
+  }
+
+  /// The universal integral type to use in value range descriptions.
+  /// Unsigned to make sure overflows are well-defined.
   typedef uint64_t RangeInt;
 
-  /// Normally, describes a single range constraint, eg. {{0, 1}, {3, 4}} is
-  /// a non-negative integer, which less than 5 and not equal to 2. For
-  /// `ComparesToArgument', holds information about how exactly to compare to
-  /// the argument.
+  /// Describes a single range constraint. Eg. {{0, 1}, {3, 4}} is
+  /// a non-negative integer, which less than 5 and not equal to 2.
   typedef std::vector<std::pair<RangeInt, RangeInt>> IntRangeVector;
 
   /// A reference to an argument or return value by its number.
   /// ArgNo in CallExpr and CallEvent is defined as Unsigned, but
   /// obviously uint32_t should be enough for all practical purposes.
   typedef uint32_t ArgNo;
+  /// Special argument number for specifying the return value.
   static const ArgNo Ret;
 
   using DescString = SmallString<96>;
+
   /// Returns the string representation of an argument index.
   /// E.g.: (1) -> '1st arg', (2) - > '2nd arg'
   static SmallString<8> getArgDesc(ArgNo);
   /// Append textual description of a numeric range [RMin,RMax] to the string
-  /// 'Out'.
+  /// \p Out.
   static void appendInsideRangeDesc(llvm::APSInt RMin, llvm::APSInt RMax,
                                     QualType ArgT, BasicValueFactory &BVF,
                                     DescString &Out);
   /// Append textual description of a numeric range out of [RMin,RMax] to the
-  /// string 'Out'.
+  /// string \p Out.
   static void appendOutOfRangeDesc(llvm::APSInt RMin, llvm::APSInt RMax,
                                    QualType ArgT, BasicValueFactory &BVF,
                                    DescString &Out);
 
   class ValueConstraint;
 
-  // Pointer to the ValueConstraint. We need a copyable, polymorphic and
-  // default initialize able type (vector needs that). A raw pointer was good,
-  // however, we cannot default initialize that. unique_ptr makes the Summary
-  // class non-copyable, therefore not an option. Releasing the copyability
-  // requirement would render the initialization of the Summary map infeasible.
+  /// Pointer to the ValueConstraint. We need a copyable, polymorphic and
+  /// default initializable type (vector needs that). A raw pointer was good,
+  /// however, we cannot default initialize that. unique_ptr makes the Summary
+  /// class non-copyable, therefore not an option. Releasing the copyability
+  /// requirement would render the initialization of the Summary map infeasible.
+  /// Mind that a pointer to a new value constraint is created when the negate
+  /// function is used.
   using ValueConstraintPtr = std::shared_ptr<ValueConstraint>;
 
   /// Polymorphic base class that represents a constraint on a given argument
@@ -122,35 +143,12 @@ class StdLibraryFunctionsChecker
   public:
     ValueConstraint(ArgNo ArgN) : ArgN(ArgN) {}
     virtual ~ValueConstraint() {}
+
     /// Apply the effects of the constraint on the given program state. If null
     /// is returned then the constraint is not feasible.
     virtual ProgramStateRef apply(ProgramStateRef State, const CallEvent &Call,
                                   const Summary &Summary,
                                   CheckerContext &C) const = 0;
-    virtual ValueConstraintPtr negate() const {
-      llvm_unreachable("Not implemented");
-    };
-
-    /// Check whether the constraint is malformed or not. It is malformed if the
-    /// specified argument has a mismatch with the given FunctionDecl (e.g. the
-    /// arg number is out-of-range of the function's argument list).
-    bool checkValidity(const FunctionDecl *FD) const {
-      const bool ValidArg = ArgN == Ret || ArgN < FD->getNumParams();
-      assert(ValidArg && "Arg out of range!");
-      if (!ValidArg)
-        return false;
-      // Subclasses may further refine the validation.
-      return checkSpecificValidity(FD);
-    }
-    ArgNo getArgNo() const { return ArgN; }
-
-    /// Return those arguments that should be tracked when we report a bug. By
-    /// default it is the argument that is constrained, however, in some special
-    /// cases we need to track other arguments as well. E.g. a buffer size might
-    /// be encoded in another argument.
-    virtual std::vector<ArgNo> getArgsToTrack() const { return {ArgN}; }
-
-    virtual StringRef getName() const = 0;
 
     /// Represents that in which context do we require a description of the
     /// constraint.
@@ -158,6 +156,8 @@ class StdLibraryFunctionsChecker
       /// The constraint is violated.
       Violation,
       /// We assume that the constraint is satisfied.
+      /// This can be used when a precondition is satisfied, or when a summary
+      /// case is applied.
       Assumption
     };
 
@@ -174,81 +174,97 @@ class StdLibraryFunctionsChecker
           "Description not implemented for summary case constraints");
     }
 
-  protected:
-    ArgNo ArgN; // Argument to which we apply the constraint.
+    /// Return those arguments that should be tracked when we report a bug about
+    /// argument constraint violation. By default it is the argument that is
+    /// constrained, however, in some special cases we need to track other
+    /// arguments as well. E.g. a buffer size might be encoded in another
+    /// argument.
+    /// The "return value" argument number can not occur as returned value.
+    virtual std::vector<ArgNo> getArgsToTrack() const { return {ArgN}; }
 
-    /// Do polymorphic validation check on the constraint.
+    /// Get a constraint that represents exactly the opposite of the current.
+    virtual ValueConstraintPtr negate() const {
+      llvm_unreachable("Not implemented");
+    };
+
+    /// Check whether the constraint is malformed or not. It is malformed if the
+    /// specified argument has a mismatch with the given FunctionDecl (e.g. the
+    /// arg number is out-of-range of the function's argument list).
+    /// This condition can indicate if a probably wrong or unexpected function
+    /// was found where the constraint is to be applied.
+    bool checkValidity(const FunctionDecl *FD) const {
+      const bool ValidArg = ArgN == Ret || ArgN < FD->getNumParams();
+      assert(ValidArg && "Arg out of range!");
+      if (!ValidArg)
+        return false;
+      // Subclasses may further refine the validation.
+      return checkSpecificValidity(FD);
+    }
+
+    /// Return the argument number (may be placeholder for "return value").
+    ArgNo getArgNo() const { return ArgN; }
+
+  protected:
+    /// Argument to which to apply the constraint. It can be a real argument of
+    /// the function to check, or a special value to indicate the return value
+    /// of the function.
+    /// Every constraint is assigned to one main argument, even if other
+    /// arguments are involved.
+    ArgNo ArgN;
+
+    /// Do constraint-specific validation check.
     virtual bool checkSpecificValidity(const FunctionDecl *FD) const {
       return true;
     }
   };
 
-  /// Given a range, should the argument stay inside or outside this range?
-  enum RangeKind { OutOfRange, WithinRange };
-
-  /// Encapsulates a range on a single symbol.
+  /// Check if a single argument falls into a specific "range".
+  /// A range is formed as a set of intervals.
+  /// E.g. \code {['A', 'Z'], ['a', 'z'], ['_', '_']} \endcode
+  /// The intervals are closed intervals that contain one or more values.
+  ///
+  /// The default constructed RangeConstraint has an empty range, applying
+  /// such constraint does not involve any assumptions, thus the State remains
+  /// unchanged. This is meaningful, if the range is dependent on a looked up
+  /// type (e.g. [0, Socklen_tMax]). If the type is not found, then the range
+  /// is default initialized to be empty.
   class RangeConstraint : public ValueConstraint {
+    /// The constraint can be specified by allowing or disallowing the range.
+    /// WithinRange indicates allowing the range, OutOfRange indicates
+    /// disallowing it (allowing the complementary range).
     RangeKind Kind;
-    // A range is formed as a set of intervals (sub-ranges).
-    // E.g. {['A', 'Z'], ['a', 'z']}
-    //
-    // The default constructed RangeConstraint has an empty range set, applying
-    // such constraint does not involve any assumptions, thus the State remains
-    // unchanged. This is meaningful, if the range is dependent on a looked up
-    // type (e.g. [0, Socklen_tMax]). If the type is not found, then the range
-    // is default initialized to be empty.
+
+    /// A set of intervals.
     IntRangeVector Ranges;
-    // A textual description of this constraint for the specific case where the
-    // constraint is used. If empty a generated description will be used.
+
+    /// A textual description of this constraint for the specific case where the
+    /// constraint is used. If empty a generated description will be used that
+    /// is built from the range of the constraint.
     StringRef Description;
 
   public:
-    StringRef getName() const override { return "Range"; }
     RangeConstraint(ArgNo ArgN, RangeKind Kind, const IntRangeVector &Ranges,
                     StringRef Desc = "")
         : ValueConstraint(ArgN), Kind(Kind), Ranges(Ranges), Description(Desc) {
     }
 
-    std::string describe(DescriptionKind DK, const CallEvent &Call,
-                         ProgramStateRef State,
-                         const Summary &Summary) const override;
-
     const IntRangeVector &getRanges() const { return Ranges; }
 
-  private:
-    ProgramStateRef applyAsOutOfRange(ProgramStateRef State,
-                                      const CallEvent &Call,
-                                      const Summary &Summary) const;
-    ProgramStateRef applyAsWithinRange(ProgramStateRef State,
-                                       const CallEvent &Call,
-                                       const Summary &Summary) const;
-
-  public:
     ProgramStateRef apply(ProgramStateRef State, const CallEvent &Call,
                           const Summary &Summary,
-                          CheckerContext &C) const override {
-      switch (Kind) {
-      case OutOfRange:
-        return applyAsOutOfRange(State, Call, Summary);
-      case WithinRange:
-        return applyAsWithinRange(State, Call, Summary);
-      }
-      llvm_unreachable("Unknown range kind!");
-    }
+                          CheckerContext &C) const override;
+
+    std::string describe(DescriptionKind DK, const CallEvent &Call,
+                         ProgramStateRef State,
+                         const Summary &Summary) const override;
 
     ValueConstraintPtr negate() const override {
       RangeConstraint Tmp(*this);
-      switch (Kind) {
-      case OutOfRange:
-        Tmp.Kind = WithinRange;
-        break;
-      case WithinRange:
-        Tmp.Kind = OutOfRange;
-        break;
-      }
+      Tmp.Kind = negateKind(Kind);
       return std::make_shared<RangeConstraint>(Tmp);
     }
 
+  protected:
     bool checkSpecificValidity(const FunctionDecl *FD) const override {
       const bool ValidArg =
           getArgType(FD, ArgN)->isIntegralType(FD->getASTContext());
@@ -256,14 +272,52 @@ class StdLibraryFunctionsChecker
              "This constraint should be applied on an integral type");
       return ValidArg;
     }
+
+  private:
+    /// A callback function that is used when iterating over the range
+    /// intervals. It gets the begin and end (inclusive) of one interval.
+    /// This is used to make any kind of task possible that needs an iteration
+    /// over the intervals.
+    using RangeApplyFunction =
+        std::function<bool(const llvm::APSInt &Min, const llvm::APSInt &Max)>;
+
+    /// Call a function on the intervals of the range.
+    /// The function is called with all intervals in the range.
+    void applyOnWithinRange(BasicValueFactory &BVF, QualType ArgT,
+                            const RangeApplyFunction &F) const;
+    /// Call a function on all intervals in the complementary range.
+    /// The function is called with all intervals that fall out of the range.
+    /// E.g. consider an interval list [A, B] and [C, D]
+    /// \code
+    /// -------+--------+------------------+------------+----------->
+    ///        A        B                  C            D
+    /// \endcode
+    /// We get the ranges [-inf, A - 1], [D + 1, +inf], [B + 1, C - 1].
+    /// The \p ArgT is used to determine the min and max of the type that is
+    /// used as "-inf" and "+inf".
+    void applyOnOutOfRange(BasicValueFactory &BVF, QualType ArgT,
+                           const RangeApplyFunction &F) const;
+    /// Call a function on the intervals of the range or the complementary
+    /// range.
+    void applyOnRange(RangeKind Kind, BasicValueFactory &BVF, QualType ArgT,
+                      const RangeApplyFunction &F) const {
+      switch (Kind) {
+      case OutOfRange:
+        applyOnOutOfRange(BVF, ArgT, F);
+        break;
+      case WithinRange:
+        applyOnWithinRange(BVF, ArgT, F);
+        break;
+      };
+    }
   };
 
+  /// Check relation of an argument to another.
   class ComparisonConstraint : public ValueConstraint {
     BinaryOperator::Opcode Opcode;
     ArgNo OtherArgN;
 
   public:
-    StringRef getName() const override { return "Comparison"; };
     ComparisonConstraint(ArgNo ArgN, BinaryOperator::Opcode Opcode,
                          ArgNo OtherArgN)
         : ValueConstraint(ArgN), Opcode(Opcode), OtherArgN(OtherArgN) {}
@@ -274,6 +328,7 @@ class StdLibraryFunctionsChecker
                           CheckerContext &C) const override;
   };
 
+  /// Check null or non-null-ness of an argument that is of pointer type.
   class NotNullConstraint : public ValueConstraint {
     using ValueConstraint::ValueConstraint;
     // This variable has a role when we negate the constraint.
@@ -282,23 +337,14 @@ class StdLibraryFunctionsChecker
   public:
     NotNullConstraint(ArgNo ArgN, bool CannotBeNull = true)
         : ValueConstraint(ArgN), CannotBeNull(CannotBeNull) {}
-    std::string describe(DescriptionKind DK, const CallEvent &Call,
-                         ProgramStateRef State,
-                         const Summary &Summary) const override;
-    StringRef getName() const override { return "NonNull"; }
+
     ProgramStateRef apply(ProgramStateRef State, const CallEvent &Call,
                           const Summary &Summary,
-                          CheckerContext &C) const override {
-      SVal V = getArgSVal(Call, getArgNo());
-      if (V.isUndef())
-        return State;
-
-      DefinedOrUnknownSVal L = V.castAs<DefinedOrUnknownSVal>();
-      if (!isa<Loc>(L))
-        return State;
+                          CheckerContext &C) const override;
 
-      return State->assume(L, CannotBeNull);
-    }
+    std::string describe(DescriptionKind DK, const CallEvent &Call,
+                         ProgramStateRef State,
+                         const Summary &Summary) const override;
 
     ValueConstraintPtr negate() const override {
       NotNullConstraint Tmp(*this);
@@ -306,6 +352,7 @@ class StdLibraryFunctionsChecker
       return std::make_shared<NotNullConstraint>(Tmp);
     }
 
+  protected:
     bool checkSpecificValidity(const FunctionDecl *FD) const override {
       const bool ValidArg = getArgType(FD, ArgN)->isPointerType();
       assert(ValidArg &&
@@ -337,7 +384,6 @@ class StdLibraryFunctionsChecker
     BinaryOperator::Opcode Op = BO_LE;
 
   public:
-    StringRef getName() const override { return "BufferSize"; }
     BufferSizeConstraint(ArgNo Buffer, llvm::APSInt BufMinSize)
         : ValueConstraint(Buffer), ConcreteSize(BufMinSize) {}
     BufferSizeConstraint(ArgNo Buffer, ArgNo BufSize)
@@ -346,6 +392,14 @@ class StdLibraryFunctionsChecker
         : ValueConstraint(Buffer), SizeArgN(BufSize),
           SizeMultiplierArgN(BufSizeMultiplier) {}
 
+    ProgramStateRef apply(ProgramStateRef State, const CallEvent &Call,
+                          const Summary &Summary,
+                          CheckerContext &C) const override;
+
+    std::string describe(DescriptionKind DK, const CallEvent &Call,
+                         ProgramStateRef State,
+                         const Summary &Summary) const override;
+
     std::vector<ArgNo> getArgsToTrack() const override {
       std::vector<ArgNo> Result{ArgN};
       if (SizeArgN)
@@ -355,58 +409,13 @@ class StdLibraryFunctionsChecker
       return Result;
     }
 
-    std::string describe(DescriptionKind DK, const CallEvent &Call,
-                         ProgramStateRef State,
-                         const Summary &Summary) const override;
-
-    ProgramStateRef apply(ProgramStateRef State, const CallEvent &Call,
-                          const Summary &Summary,
-                          CheckerContext &C) const override {
-      SValBuilder &SvalBuilder = C.getSValBuilder();
-      // The buffer argument.
-      SVal BufV = getArgSVal(Call, getArgNo());
-
-      // Get the size constraint.
-      const SVal SizeV = [this, &State, &Call, &Summary, &SvalBuilder]() {
-        if (ConcreteSize) {
-          return SVal(SvalBuilder.makeIntVal(*ConcreteSize));
-        }
-        assert(SizeArgN && "The constraint must be either a concrete value or "
-                           "encoded in an argument.");
-        // The size argument.
-        SVal SizeV = getArgSVal(Call, *SizeArgN);
-        // Multiply with another argument if given.
-        if (SizeMultiplierArgN) {
-          SVal SizeMulV = getArgSVal(Call, *SizeMultiplierArgN);
-          SizeV = SvalBuilder.evalBinOp(State, BO_Mul, SizeV, SizeMulV,
-                                        Summary.getArgType(*SizeArgN));
-        }
-        return SizeV;
-      }();
-
-      // The dynamic size of the buffer argument, got from the analyzer engine.
-      SVal BufDynSize = getDynamicExtentWithOffset(State, BufV);
-
-      SVal Feasible = SvalBuilder.evalBinOp(State, Op, SizeV, BufDynSize,
-                                            SvalBuilder.getContext().BoolTy);
-      if (auto F = Feasible.getAs<DefinedOrUnknownSVal>())
-        return State->assume(*F, true);
-
-      // We can get here only if the size argument or the dynamic size is
-      // undefined. But the dynamic size should never be undefined, only
-      // unknown. So, here, the size of the argument is undefined, i.e. we
-      // cannot apply the constraint. Actually, other checkers like
-      // CallAndMessage should catch this situation earlier, because we call a
-      // function with an uninitialized argument.
-      llvm_unreachable("Size argument or the dynamic size is Undefined");
-    }
-
     ValueConstraintPtr negate() const override {
       BufferSizeConstraint Tmp(*this);
       Tmp.Op = BinaryOperator::negateComparisonOp(Op);
       return std::make_shared<BufferSizeConstraint>(Tmp);
     }
 
+  protected:
     bool checkSpecificValidity(const FunctionDecl *FD) const override {
       const bool ValidArg = getArgType(FD, ArgN)->isPointerType();
       assert(ValidArg &&
@@ -758,8 +767,8 @@ class StdLibraryFunctionsChecker
   void initFunctionSummaries(CheckerContext &C) const;
 
   void reportBug(const CallEvent &Call, ExplodedNode *N,
-                 const ValueConstraint *VC, const Summary &Summary,
-                 CheckerContext &C) const {
+                 const ValueConstraint *VC, const ValueConstraint *NegatedVC,
+                 const Summary &Summary, CheckerContext &C) const {
     if (!ChecksEnabled[CK_StdCLibraryFunctionArgsChecker])
       return;
     assert(Call.getDecl() &&
@@ -897,17 +906,79 @@ void StdLibraryFunctionsChecker::appendOutOfRangeDesc(llvm::APSInt RMin,
   }
 }
 
-std::string StdLibraryFunctionsChecker::NotNullConstraint::describe(
-    DescriptionKind DK, const CallEvent &Call, ProgramStateRef State,
-    const Summary &Summary) const {
-  SmallString<48> Result;
-  const auto Violation = ValueConstraint::DescriptionKind::Violation;
-  Result += "the ";
-  Result += getArgDesc(ArgN);
-  Result += " to '";
-  Result += getFunctionName(Call);
-  Result += DK == Violation ? "' should not be NULL" : "' is not NULL";
-  return Result.c_str();
+void StdLibraryFunctionsChecker::RangeConstraint::applyOnWithinRange(
+    BasicValueFactory &BVF, QualType ArgT, const RangeApplyFunction &F) const {
+  if (Ranges.empty())
+    return;
+
+  const IntRangeVector &R = getRanges();
+  size_t E = R.size();
+  for (size_t I = 0; I != E; ++I) {
+    const llvm::APSInt &Min = BVF.getValue(R[I].first, ArgT);
+    const llvm::APSInt &Max = BVF.getValue(R[I].second, ArgT);
+    assert(Min <= Max);
+    if (!F(Min, Max))
+      return;
+  }
+}
+
+void StdLibraryFunctionsChecker::RangeConstraint::applyOnOutOfRange(
+    BasicValueFactory &BVF, QualType ArgT, const RangeApplyFunction &F) const {
+  if (Ranges.empty())
+    return;
+
+  const IntRangeVector &R = getRanges();
+  size_t E = R.size();
+
+  const llvm::APSInt &MinusInf = BVF.getMinValue(ArgT);
+  const llvm::APSInt &PlusInf = BVF.getMaxValue(ArgT);
+
+  const llvm::APSInt &RangeLeft = BVF.getValue(R[0].first - 1ULL, ArgT);
+  const llvm::APSInt &RangeRight = BVF.getValue(R[E - 1].second + 1ULL, ArgT);
+
+  // Iterate over the "holes" between intervals.
+  for (size_t I = 1; I != E; ++I) {
+    const llvm::APSInt &Min = BVF.getValue(R[I - 1].second + 1ULL, ArgT);
+    const llvm::APSInt &Max = BVF.getValue(R[I].first - 1ULL, ArgT);
+    if (Min <= Max) {
+      if (!F(Min, Max))
+        return;
+    }
+  }
+  // Check the interval [T_MIN, min(R) - 1].
+  if (RangeLeft != PlusInf) {
+    assert(MinusInf <= RangeLeft);
+    if (!F(MinusInf, RangeLeft))
+      return;
+  }
+  // Check the interval [max(R) + 1, T_MAX],
+  if (RangeRight != MinusInf) {
+    assert(RangeRight <= PlusInf);
+    if (!F(RangeRight, PlusInf))
+      return;
+  }
+}
+
+ProgramStateRef StdLibraryFunctionsChecker::RangeConstraint::apply(
+    ProgramStateRef State, const CallEvent &Call, const Summary &Summary,
+    CheckerContext &C) const {
+  ConstraintManager &CM = C.getConstraintManager();
+  SVal V = getArgSVal(Call, getArgNo());
+  QualType T = Summary.getArgType(getArgNo());
+
+  if (auto N = V.getAs<NonLoc>()) {
+    auto ExcludeRangeFromArg = [&](const llvm::APSInt &Min,
+                                   const llvm::APSInt &Max) {
+      State = CM.assumeInclusiveRange(State, *N, Min, Max, false);
+      return static_cast<bool>(State);
+    };
+    // "OutOfRange R" is handled by excluding all ranges in R.
+    // "WithinRange R" is treated as "OutOfRange [T_MIN, T_MAX] \ R".
+    applyOnRange(negateKind(Kind), C.getSValBuilder().getBasicValueFactory(), T,
+                 ExcludeRangeFromArg);
+  }
+
+  return State;
 }
 
 std::string StdLibraryFunctionsChecker::RangeConstraint::describe(
@@ -915,10 +986,10 @@ std::string StdLibraryFunctionsChecker::RangeConstraint::describe(
     const Summary &Summary) const {
 
   BasicValueFactory &BVF = getBVF(State);
-
   QualType T = Summary.getArgType(getArgNo());
   DescString Result;
   const auto Violation = ValueConstraint::DescriptionKind::Violation;
+
   Result += "the ";
   Result += getArgDesc(ArgN);
   Result += " to '";
@@ -948,6 +1019,97 @@ std::string StdLibraryFunctionsChecker::RangeConstraint::describe(
   return Result.c_str();
 }
 
+ProgramStateRef StdLibraryFunctionsChecker::ComparisonConstraint::apply(
+    ProgramStateRef State, const CallEvent &Call, const Summary &Summary,
+    CheckerContext &C) const {
+
+  ProgramStateManager &Mgr = State->getStateManager();
+  SValBuilder &SVB = Mgr.getSValBuilder();
+  QualType CondT = SVB.getConditionType();
+  QualType T = Summary.getArgType(getArgNo());
+  SVal V = getArgSVal(Call, getArgNo());
+
+  BinaryOperator::Opcode Op = getOpcode();
+  ArgNo OtherArg = getOtherArgNo();
+  SVal OtherV = getArgSVal(Call, OtherArg);
+  QualType OtherT = Summary.getArgType(OtherArg);
+  // Note: we avoid integral promotion for comparison.
+  OtherV = SVB.evalCast(OtherV, T, OtherT);
+  if (auto CompV = SVB.evalBinOp(State, Op, V, OtherV, CondT)
+                       .getAs<DefinedOrUnknownSVal>())
+    State = State->assume(*CompV, true);
+  return State;
+}
+
+ProgramStateRef StdLibraryFunctionsChecker::NotNullConstraint::apply(
+    ProgramStateRef State, const CallEvent &Call, const Summary &Summary,
+    CheckerContext &C) const {
+  SVal V = getArgSVal(Call, getArgNo());
+  if (V.isUndef())
+    return State;
+
+  DefinedOrUnknownSVal L = V.castAs<DefinedOrUnknownSVal>();
+  if (!isa<Loc>(L))
+    return State;
+
+  return State->assume(L, CannotBeNull);
+}
+
+std::string StdLibraryFunctionsChecker::NotNullConstraint::describe(
+    DescriptionKind DK, const CallEvent &Call, ProgramStateRef State,
+    const Summary &Summary) const {
+  SmallString<48> Result;
+  const auto Violation = ValueConstraint::DescriptionKind::Violation;
+  Result += "the ";
+  Result += getArgDesc(ArgN);
+  Result += " to '";
+  Result += getFunctionName(Call);
+  Result += DK == Violation ? "' should not be NULL" : "' is not NULL";
+  return Result.c_str();
+}
+
+ProgramStateRef StdLibraryFunctionsChecker::BufferSizeConstraint::apply(
+    ProgramStateRef State, const CallEvent &Call, const Summary &Summary,
+    CheckerContext &C) const {
+  SValBuilder &SvalBuilder = C.getSValBuilder();
+  // The buffer argument.
+  SVal BufV = getArgSVal(Call, getArgNo());
+
+  // Get the size constraint.
+  const SVal SizeV = [this, &State, &Call, &Summary, &SvalBuilder]() {
+    if (ConcreteSize) {
+      return SVal(SvalBuilder.makeIntVal(*ConcreteSize));
+    }
+    assert(SizeArgN && "The constraint must be either a concrete value or "
+                       "encoded in an argument.");
+    // The size argument.
+    SVal SizeV = getArgSVal(Call, *SizeArgN);
+    // Multiply with another argument if given.
+    if (SizeMultiplierArgN) {
+      SVal SizeMulV = getArgSVal(Call, *SizeMultiplierArgN);
+      SizeV = SvalBuilder.evalBinOp(State, BO_Mul, SizeV, SizeMulV,
+                                    Summary.getArgType(*SizeArgN));
+    }
+    return SizeV;
+  }();
+
+  // The dynamic size of the buffer argument, got from the analyzer engine.
+  SVal BufDynSize = getDynamicExtentWithOffset(State, BufV);
+
+  SVal Feasible = SvalBuilder.evalBinOp(State, Op, SizeV, BufDynSize,
+                                        SvalBuilder.getContext().BoolTy);
+  if (auto F = Feasible.getAs<DefinedOrUnknownSVal>())
+    return State->assume(*F, true);
+
+  // We can get here only if the size argument or the dynamic size is
+  // undefined. But the dynamic size should never be undefined, only
+  // unknown. So, here, the size of the argument is undefined, i.e. we
+  // cannot apply the constraint. Actually, other checkers like
+  // CallAndMessage should catch this situation earlier, because we call a
+  // function with an uninitialized argument.
+  llvm_unreachable("Size argument or the dynamic size is Undefined");
+}
+
 std::string StdLibraryFunctionsChecker::BufferSizeConstraint::describe(
     DescriptionKind DK, const CallEvent &Call, ProgramStateRef State,
     const Summary &Summary) const {
@@ -972,116 +1134,6 @@ std::string StdLibraryFunctionsChecker::BufferSizeConstraint::describe(
   return Result.c_str();
 }
 
-ProgramStateRef StdLibraryFunctionsChecker::RangeConstraint::applyAsOutOfRange(
-    ProgramStateRef State, const CallEvent &Call,
-    const Summary &Summary) const {
-  if (Ranges.empty())
-    return State;
-
-  ProgramStateManager &Mgr = State->getStateManager();
-  SValBuilder &SVB = Mgr.getSValBuilder();
-  BasicValueFactory &BVF = SVB.getBasicValueFactory();
-  ConstraintManager &CM = Mgr.getConstraintManager();
-  QualType T = Summary.getArgType(getArgNo());
-  SVal V = getArgSVal(Call, getArgNo());
-
-  if (auto N = V.getAs<NonLoc>()) {
-    const IntRangeVector &R = getRanges();
-    size_t E = R.size();
-    for (size_t I = 0; I != E; ++I) {
-      const llvm::APSInt &Min = BVF.getValue(R[I].first, T);
-      const llvm::APSInt &Max = BVF.getValue(R[I].second, T);
-      assert(Min <= Max);
-      State = CM.assumeInclusiveRange(State, *N, Min, Max, false);
-      if (!State)
-        break;
-    }
-  }
-
-  return State;
-}
-
-ProgramStateRef StdLibraryFunctionsChecker::RangeConstraint::applyAsWithinRange(
-    ProgramStateRef State, const CallEvent &Call,
-    const Summary &Summary) const {
-  if (Ranges.empty())
-    return State;
-
-  ProgramStateManager &Mgr = State->getStateManager();
-  SValBuilder &SVB = Mgr.getSValBuilder();
-  BasicValueFactory &BVF = SVB.getBasicValueFactory();
-  ConstraintManager &CM = Mgr.getConstraintManager();
-  QualType T = Summary.getArgType(getArgNo());
-  SVal V = getArgSVal(Call, getArgNo());
-
-  // "WithinRange R" is treated as "outside [T_MIN, T_MAX] \ R".
-  // We cut off [T_MIN, min(R) - 1] and [max(R) + 1, T_MAX] if necessary,
-  // and then cut away all holes in R one by one.
-  //
-  // E.g. consider a range list R as [A, B] and [C, D]
-  // -------+--------+------------------+------------+----------->
-  //        A        B                  C            D
-  // Then we assume that the value is not in [-inf, A - 1],
-  // then not in [D + 1, +inf], then not in [B + 1, C - 1]
-  if (auto N = V.getAs<NonLoc>()) {
-    const IntRangeVector &R = getRanges();
-    size_t E = R.size();
-
-    const llvm::APSInt &MinusInf = BVF.getMinValue(T);
-    const llvm::APSInt &PlusInf = BVF.getMaxValue(T);
-
-    const llvm::APSInt &Left = BVF.getValue(R[0].first - 1ULL, T);
-    if (Left != PlusInf) {
-      assert(MinusInf <= Left);
-      State = CM.assumeInclusiveRange(State, *N, MinusInf, Left, false);
-      if (!State)
-        return nullptr;
-    }
-
-    const llvm::APSInt &Right = BVF.getValue(R[E - 1].second + 1ULL, T);
-    if (Right != MinusInf) {
-      assert(Right <= PlusInf);
-      State = CM.assumeInclusiveRange(State, *N, Right, PlusInf, false);
-      if (!State)
-        return nullptr;
-    }
-
-    for (size_t I = 1; I != E; ++I) {
-      const llvm::APSInt &Min = BVF.getValue(R[I - 1].second + 1ULL, T);
-      const llvm::APSInt &Max = BVF.getValue(R[I].first - 1ULL, T);
-      if (Min <= Max) {
-        State = CM.assumeInclusiveRange(State, *N, Min, Max, false);
-        if (!State)
-          return nullptr;
-      }
-    }
-  }
-
-  return State;
-}
-
-ProgramStateRef StdLibraryFunctionsChecker::ComparisonConstraint::apply(
-    ProgramStateRef State, const CallEvent &Call, const Summary &Summary,
-    CheckerContext &C) const {
-
-  ProgramStateManager &Mgr = State->getStateManager();
-  SValBuilder &SVB = Mgr.getSValBuilder();
-  QualType CondT = SVB.getConditionType();
-  QualType T = Summary.getArgType(getArgNo());
-  SVal V = getArgSVal(Call, getArgNo());
-
-  BinaryOperator::Opcode Op = getOpcode();
-  ArgNo OtherArg = getOtherArgNo();
-  SVal OtherV = getArgSVal(Call, OtherArg);
-  QualType OtherT = Summary.getArgType(OtherArg);
-  // Note: we avoid integral promotion for comparison.
-  OtherV = SVB.evalCast(OtherV, T, OtherT);
-  if (auto CompV = SVB.evalBinOp(State, Op, V, OtherV, CondT)
-                       .getAs<DefinedOrUnknownSVal>())
-    State = State->assume(*CompV, true);
-  return State;
-}
-
 void StdLibraryFunctionsChecker::checkPreCall(const CallEvent &Call,
                                               CheckerContext &C) const {
   std::optional<Summary> FoundSummary = findFunctionSummary(Call, C);
@@ -1094,13 +1146,15 @@ void StdLibraryFunctionsChecker::checkPreCall(const CallEvent &Call,
   ProgramStateRef NewState = State;
   ExplodedNode *NewNode = C.getPredecessor();
   for (const ValueConstraintPtr &Constraint : Summary.getArgConstraints()) {
+    ValueConstraintPtr NegatedConstraint = Constraint->negate();
     ProgramStateRef SuccessSt = Constraint->apply(NewState, Call, Summary, C);
     ProgramStateRef FailureSt =
-        Constraint->negate()->apply(NewState, Call, Summary, C);
+        NegatedConstraint->apply(NewState, Call, Summary, C);
     // The argument constraint is not satisfied.
     if (FailureSt && !SuccessSt) {
-      if (ExplodedNode *N = C.generateErrorNode(NewState, NewNode))
-        reportBug(Call, N, Constraint.get(), Summary, C);
+      if (ExplodedNode *N = C.generateErrorNode(State, NewNode))
+        reportBug(Call, N, Constraint.get(), NegatedConstraint.get(), Summary,
+                  C);
       break;
     }
     // We will apply the constraint even if we cannot reason about the
diff --git a/clang/lib/StaticAnalyzer/Core/BugReporter.cpp b/clang/lib/StaticAnalyzer/Core/BugReporter.cpp
index a7f149b87e79e..5b79e5d486286 100644
--- a/clang/lib/StaticAnalyzer/Core/BugReporter.cpp
+++ b/clang/lib/StaticAnalyzer/Core/BugReporter.cpp
@@ -766,7 +766,7 @@ PathDiagnosticPieceRef PathDiagnosticBuilder::generateDiagForSwitchOP(
     case Stmt::CaseStmtClass: {
       os << "Control jumps to 'case ";
       const auto *Case = cast<CaseStmt>(S);
-      const Expr *LHS = Case->getLHS()->IgnoreParenCasts();
+      const Expr *LHS = Case->getLHS()->IgnoreParenImpCasts();
 
       // Determine if it is an enum.
       bool GetRawInt = true;
@@ -2627,8 +2627,7 @@ BugPathInfo *BugPathGetter::getNextBugPath() {
 
   const ExplodedNode *OrigN;
   std::tie(CurrentBugPath.Report, OrigN) = ReportNodes.pop_back_val();
-  assert(PriorityMap.find(OrigN) != PriorityMap.end() &&
-         "error node not accessible from root");
+  assert(PriorityMap.contains(OrigN) && "error node not accessible from root");
 
   // Create a new graph with a single path. This is the graph that will be
   // returned to the caller.
diff --git a/clang/lib/StaticAnalyzer/Core/ExplodedGraph.cpp b/clang/lib/StaticAnalyzer/Core/ExplodedGraph.cpp
index d274d4d16db32..cac7ec3f8cf7f 100644
--- a/clang/lib/StaticAnalyzer/Core/ExplodedGraph.cpp
+++ b/clang/lib/StaticAnalyzer/Core/ExplodedGraph.cpp
@@ -488,7 +488,7 @@ ExplodedGraph::trim(ArrayRef<const NodeTy *> Sinks,
     const ExplodedNode *N = WL2.pop_back_val();
 
     // Skip this node if we have already processed it.
-    if (Pass2.find(N) != Pass2.end())
+    if (Pass2.contains(N))
       continue;
 
     // Create the corresponding node in the new graph and record the mapping
diff --git a/clang/lib/StaticAnalyzer/Core/SymbolManager.cpp b/clang/lib/StaticAnalyzer/Core/SymbolManager.cpp
index 3e97f0c95fc32..032605ffe7a23 100644
--- a/clang/lib/StaticAnalyzer/Core/SymbolManager.cpp
+++ b/clang/lib/StaticAnalyzer/Core/SymbolManager.cpp
@@ -398,7 +398,7 @@ void SymbolReaper::markDependentsLive(SymbolRef sym) {
 
   if (const SymbolRefSmallVectorTy *Deps = SymMgr.getDependentSymbols(sym)) {
     for (const auto I : *Deps) {
-      if (TheLiving.find(I) != TheLiving.end())
+      if (TheLiving.contains(I))
         continue;
       markLive(I);
     }
diff --git a/clang/lib/StaticAnalyzer/Frontend/CheckerRegistry.cpp b/clang/lib/StaticAnalyzer/Frontend/CheckerRegistry.cpp
index 4618d17577dd8..f0d3f43c414c6 100644
--- a/clang/lib/StaticAnalyzer/Frontend/CheckerRegistry.cpp
+++ b/clang/lib/StaticAnalyzer/Frontend/CheckerRegistry.cpp
@@ -233,7 +233,7 @@ void CheckerRegistry::initializeRegistry(const CheckerManager &Mgr) {
   // done recursively, its arguably cheaper, but for sure less error prone to
   // recalculate from scratch.
   auto IsEnabled = [&](const CheckerInfo *Checker) {
-    return llvm::is_contained(Tmp, Checker);
+    return Tmp.contains(Checker);
   };
   for (const CheckerInfo &Checker : Data.Checkers) {
     if (!Checker.isEnabled(Mgr))
@@ -525,4 +525,3 @@ void CheckerRegistry::validateCheckerOptions() const {
         << SuppliedCheckerOrPackage;
   }
 }
-
diff --git a/clang/lib/StaticAnalyzer/README.txt b/clang/lib/StaticAnalyzer/README.txt
index 75f20315a7ae8..ae622caf6ccb4 100644
--- a/clang/lib/StaticAnalyzer/README.txt
+++ b/clang/lib/StaticAnalyzer/README.txt
@@ -119,15 +119,17 @@ simulation results.
 
 Of course, viewing the CFG (Control-Flow Graph) is also useful:
 
-$ clang -cc1 -help | grep cfg
- -cfg-add-implicit-dtors Add C++ implicit destructors to CFGs for all analyses
- -cfg-add-initializers   Add C++ initializers to CFGs for all analyses
- -cfg-dump               Display Control-Flow Graphs
- -cfg-view               View Control-Flow Graphs using GraphViz
- -unoptimized-cfg        Generate unoptimized CFGs for all analyses
-
--cfg-dump dumps a textual representation of the CFG to the console,
-and -cfg-view creates a GraphViz representation.
+$ clang -cc1 -analyzer-checker-help-developer
+
+ -analyzer-checker=debug.DumpCFG   Display Control-Flow Graphs
+ -analyzer-checker=debug.ViewCFG   View Control-Flow Graphs using GraphViz
+(outdated below?)
+ -cfg-add-implicit-dtors           Add C++ implicit destructors to CFGs for all analyses
+ -cfg-add-initializers             Add C++ initializers to CFGs for all analyses
+ -unoptimized-cfg                  Generate unoptimized CFGs for all analyses
+
+debug.DumpCFG dumps a textual representation of the CFG to the console, and
+debug.ViewCFG creates a GraphViz representation.
 
 = References =
 
diff --git a/clang/lib/Testing/CMakeLists.txt b/clang/lib/Testing/CMakeLists.txt
index 49b6787959bc1..4a2400d6a107a 100644
--- a/clang/lib/Testing/CMakeLists.txt
+++ b/clang/lib/Testing/CMakeLists.txt
@@ -11,6 +11,7 @@ add_llvm_library(clangTesting
   BUILDTREE_ONLY
 
   LINK_COMPONENTS
+  MC
   Support
   )
 
diff --git a/clang/lib/Testing/CommandLineArgs.cpp b/clang/lib/Testing/CommandLineArgs.cpp
index bf517e2dd312e..0da087c33e3f4 100644
--- a/clang/lib/Testing/CommandLineArgs.cpp
+++ b/clang/lib/Testing/CommandLineArgs.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "clang/Testing/CommandLineArgs.h"
+#include "llvm/MC/TargetRegistry.h"
 #include "llvm/Support/ErrorHandling.h"
 
 namespace clang {
@@ -109,4 +110,18 @@ StringRef getFilenameForTesting(TestLanguage Lang) {
   llvm_unreachable("Unhandled TestLanguage enum");
 }
 
+std::string getAnyTargetForTesting() {
+  for (const auto &Target : llvm::TargetRegistry::targets()) {
+    std::string Error;
+    StringRef TargetName(Target.getName());
+    if (TargetName == "x86-64")
+      TargetName = "x86_64";
+    if (llvm::TargetRegistry::lookupTarget(std::string(TargetName), Error) ==
+        &Target) {
+      return std::string(TargetName);
+    }
+  }
+  return "";
+}
+
 } // end namespace clang
diff --git a/clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp b/clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
index 97b41fc689176..0ddb5c24c5e6c 100644
--- a/clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
+++ b/clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
@@ -181,7 +181,11 @@ static bool shouldCacheStatFailures(StringRef Filename) {
   StringRef Ext = llvm::sys::path::extension(Filename);
   if (Ext.empty())
     return false; // This may be the module cache directory.
-  // Only cache stat failures on source files.
+  // Only cache stat failures on files that are not expected to change during
+  // the build.
+  StringRef FName = llvm::sys::path::filename(Filename);
+  if (FName == "module.modulemap" || FName == "module.map")
+    return true;
   return shouldScanForDirectivesBasedOnExtension(Filename);
 }
 
diff --git a/clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp b/clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp
index c15f6e724fa14..6641518b572b1 100644
--- a/clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp
+++ b/clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp
@@ -46,11 +46,6 @@ class MakeDependencyPrinterConsumer : public DependencyConsumer {
 
   void handleContextHash(std::string Hash) override {}
 
-  std::string lookupModuleOutput(const ModuleID &ID,
-                                 ModuleOutputKind Kind) override {
-    llvm::report_fatal_error("unexpected call to lookupModuleOutput");
-  }
-
   void printDependencies(std::string &S) {
     assert(Opts && "Handled dependency output options.");
 
@@ -82,7 +77,9 @@ class MakeDependencyPrinterConsumer : public DependencyConsumer {
 llvm::Expected<std::string> DependencyScanningTool::getDependencyFile(
     const std::vector<std::string> &CommandLine, StringRef CWD) {
   MakeDependencyPrinterConsumer Consumer;
-  auto Result = Worker.computeDependencies(CWD, CommandLine, Consumer);
+  CallbackActionController Controller(nullptr);
+  auto Result =
+      Worker.computeDependencies(CWD, CommandLine, Consumer, Controller);
   if (Result)
     return std::move(Result);
   std::string Output;
@@ -117,20 +114,25 @@ llvm::Expected<P1689Rule> DependencyScanningTool::getP1689ModuleDependencyFile(
       return Opts->OutputFile;
     }
 
-    // The lookupModuleOutput is for clang modules. P1689 format don't need it.
-    std::string lookupModuleOutput(const ModuleID &ID,
-                                 ModuleOutputKind Kind) override {
-      return "";
-    }
-
   private:
     StringRef Filename;
     P1689Rule &Rule;
   };
 
+  class P1689ActionController : public DependencyActionController {
+  public:
+    // The lookupModuleOutput is for clang modules. P1689 format don't need it.
+    std::string lookupModuleOutput(const ModuleID &,
+                                   ModuleOutputKind Kind) override {
+      return "";
+    }
+  };
+
   P1689Rule Rule;
   P1689ModuleDependencyPrinterConsumer Consumer(Rule, Command);
-  auto Result = Worker.computeDependencies(CWD, Command.CommandLine, Consumer);
+  P1689ActionController Controller;
+  auto Result = Worker.computeDependencies(CWD, Command.CommandLine, Consumer,
+                                           Controller);
   if (Result)
     return std::move(Result);
 
@@ -145,8 +147,10 @@ DependencyScanningTool::getTranslationUnitDependencies(
     const std::vector<std::string> &CommandLine, StringRef CWD,
     const llvm::StringSet<> &AlreadySeen,
     LookupModuleOutputCallback LookupModuleOutput) {
-  FullDependencyConsumer Consumer(AlreadySeen, LookupModuleOutput);
-  llvm::Error Result = Worker.computeDependencies(CWD, CommandLine, Consumer);
+  FullDependencyConsumer Consumer(AlreadySeen);
+  CallbackActionController Controller(LookupModuleOutput);
+  llvm::Error Result =
+      Worker.computeDependencies(CWD, CommandLine, Consumer, Controller);
   if (Result)
     return std::move(Result);
   return Consumer.takeTranslationUnitDeps();
@@ -156,9 +160,10 @@ llvm::Expected<ModuleDepsGraph> DependencyScanningTool::getModuleDependencies(
     StringRef ModuleName, const std::vector<std::string> &CommandLine,
     StringRef CWD, const llvm::StringSet<> &AlreadySeen,
     LookupModuleOutputCallback LookupModuleOutput) {
-  FullDependencyConsumer Consumer(AlreadySeen, LookupModuleOutput);
-  llvm::Error Result =
-      Worker.computeDependencies(CWD, CommandLine, Consumer, ModuleName);
+  FullDependencyConsumer Consumer(AlreadySeen);
+  CallbackActionController Controller(LookupModuleOutput);
+  llvm::Error Result = Worker.computeDependencies(CWD, CommandLine, Consumer,
+                                                  Controller, ModuleName);
   if (Result)
     return std::move(Result);
   return Consumer.takeModuleGraphDeps();
@@ -200,3 +205,5 @@ ModuleDepsGraph FullDependencyConsumer::takeModuleGraphDeps() {
 
   return ModuleGraph;
 }
+
+CallbackActionController::~CallbackActionController() {}
diff --git a/clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp b/clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
index f7c976b4a3033..5869418b98dfe 100644
--- a/clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
+++ b/clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "clang/Tooling/DependencyScanning/DependencyScanningWorker.h"
+#include "clang/Basic/DiagnosticDriver.h"
 #include "clang/Basic/DiagnosticFrontend.h"
 #include "clang/CodeGen/ObjectFilePCHContainerOperations.h"
 #include "clang/Driver/Compilation.h"
@@ -22,6 +23,8 @@
 #include "clang/Tooling/DependencyScanning/DependencyScanningService.h"
 #include "clang/Tooling/DependencyScanning/ModuleDepCollector.h"
 #include "clang/Tooling/Tooling.h"
+#include "llvm/Support/Allocator.h"
+#include "llvm/Support/Error.h"
 #include "llvm/TargetParser/Host.h"
 #include <optional>
 
@@ -146,13 +149,14 @@ class DependencyScanningAction : public tooling::ToolAction {
 public:
   DependencyScanningAction(
       StringRef WorkingDirectory, DependencyConsumer &Consumer,
+      DependencyActionController &Controller,
       llvm::IntrusiveRefCntPtr<DependencyScanningWorkerFilesystem> DepFS,
       ScanningOutputFormat Format, bool OptimizeArgs, bool EagerLoadModules,
       bool DisableFree, std::optional<StringRef> ModuleName = std::nullopt)
       : WorkingDirectory(WorkingDirectory), Consumer(Consumer),
-        DepFS(std::move(DepFS)), Format(Format), OptimizeArgs(OptimizeArgs),
-        EagerLoadModules(EagerLoadModules), DisableFree(DisableFree),
-        ModuleName(ModuleName) {}
+        Controller(Controller), DepFS(std::move(DepFS)), Format(Format),
+        OptimizeArgs(OptimizeArgs), EagerLoadModules(EagerLoadModules),
+        DisableFree(DisableFree), ModuleName(ModuleName) {}
 
   bool runInvocation(std::shared_ptr<CompilerInvocation> Invocation,
                      FileManager *FileMgr,
@@ -250,8 +254,8 @@ class DependencyScanningAction : public tooling::ToolAction {
     case ScanningOutputFormat::P1689:
     case ScanningOutputFormat::Full:
       MDC = std::make_shared<ModuleDepCollector>(
-          std::move(Opts), ScanInstance, Consumer, OriginalInvocation,
-          OptimizeArgs, EagerLoadModules,
+          std::move(Opts), ScanInstance, Consumer, Controller,
+          OriginalInvocation, OptimizeArgs, EagerLoadModules,
           Format == ScanningOutputFormat::P1689);
       ScanInstance.addDependencyCollector(MDC);
       break;
@@ -300,6 +304,7 @@ class DependencyScanningAction : public tooling::ToolAction {
 private:
   StringRef WorkingDirectory;
   DependencyConsumer &Consumer;
+  DependencyActionController &Controller;
   llvm::IntrusiveRefCntPtr<DependencyScanningWorkerFilesystem> DepFS;
   ScanningOutputFormat Format;
   bool OptimizeArgs;
@@ -342,7 +347,8 @@ DependencyScanningWorker::DependencyScanningWorker(
 
 llvm::Error DependencyScanningWorker::computeDependencies(
     StringRef WorkingDirectory, const std::vector<std::string> &CommandLine,
-    DependencyConsumer &Consumer, std::optional<StringRef> ModuleName) {
+    DependencyConsumer &Consumer, DependencyActionController &Controller,
+    std::optional<StringRef> ModuleName) {
   std::vector<const char *> CLI;
   for (const std::string &Arg : CommandLine)
     CLI.push_back(Arg.c_str());
@@ -355,24 +361,37 @@ llvm::Error DependencyScanningWorker::computeDependencies(
   llvm::raw_string_ostream DiagnosticsOS(DiagnosticOutput);
   TextDiagnosticPrinter DiagPrinter(DiagnosticsOS, DiagOpts.release());
 
-  if (computeDependencies(WorkingDirectory, CommandLine, Consumer, DiagPrinter,
-                          ModuleName))
+  if (computeDependencies(WorkingDirectory, CommandLine, Consumer, Controller,
+                          DiagPrinter, ModuleName))
     return llvm::Error::success();
   return llvm::make_error<llvm::StringError>(DiagnosticsOS.str(),
                                              llvm::inconvertibleErrorCode());
 }
 
 static bool forEachDriverJob(
-    ArrayRef<std::string> Args, DiagnosticsEngine &Diags, FileManager &FM,
+    ArrayRef<std::string> ArgStrs, DiagnosticsEngine &Diags, FileManager &FM,
     llvm::function_ref<bool(const driver::Command &Cmd)> Callback) {
+  SmallVector<const char *, 256> Argv;
+  Argv.reserve(ArgStrs.size());
+  for (const std::string &Arg : ArgStrs)
+    Argv.push_back(Arg.c_str());
+
+  llvm::vfs::FileSystem *FS = &FM.getVirtualFileSystem();
+
   std::unique_ptr<driver::Driver> Driver = std::make_unique<driver::Driver>(
-      Args[0], llvm::sys::getDefaultTargetTriple(), Diags,
-      "clang LLVM compiler", &FM.getVirtualFileSystem());
+      Argv[0], llvm::sys::getDefaultTargetTriple(), Diags,
+      "clang LLVM compiler", FS);
   Driver->setTitle("clang_based_tool");
 
-  std::vector<const char *> Argv;
-  for (const std::string &Arg : Args)
-    Argv.push_back(Arg.c_str());
+  llvm::BumpPtrAllocator Alloc;
+  bool CLMode = driver::IsClangCL(
+      driver::getDriverMode(Argv[0], ArrayRef(Argv).slice(1)));
+
+  if (llvm::Error E = driver::expandResponseFiles(Argv, CLMode, Alloc, FS)) {
+    Diags.Report(diag::err_drv_expand_response_file)
+        << llvm::toString(std::move(E));
+    return false;
+  }
 
   const std::unique_ptr<driver::Compilation> Compilation(
       Driver->BuildCompilation(llvm::ArrayRef(Argv)));
@@ -388,8 +407,8 @@ static bool forEachDriverJob(
 
 bool DependencyScanningWorker::computeDependencies(
     StringRef WorkingDirectory, const std::vector<std::string> &CommandLine,
-    DependencyConsumer &Consumer, DiagnosticConsumer &DC,
-    std::optional<StringRef> ModuleName) {
+    DependencyConsumer &Consumer, DependencyActionController &Controller,
+    DiagnosticConsumer &DC, std::optional<StringRef> ModuleName) {
   // Reset what might have been modified in the previous worker invocation.
   BaseFS->setCurrentWorkingDirectory(WorkingDirectory);
 
@@ -445,9 +464,9 @@ bool DependencyScanningWorker::computeDependencies(
   // in-process; preserve the original value, which is
   // always true for a driver invocation.
   bool DisableFree = true;
-  DependencyScanningAction Action(WorkingDirectory, Consumer, DepFS, Format,
-                                  OptimizeArgs, EagerLoadModules, DisableFree,
-                                  ModuleName);
+  DependencyScanningAction Action(WorkingDirectory, Consumer, Controller, DepFS,
+                                  Format, OptimizeArgs, EagerLoadModules,
+                                  DisableFree, ModuleName);
   bool Success = forEachDriverJob(
       FinalCommandLine, *Diags, *FileMgr, [&](const driver::Command &Cmd) {
         if (StringRef(Cmd.getCreator().getName()) != "clang") {
@@ -485,3 +504,5 @@ bool DependencyScanningWorker::computeDependencies(
         << llvm::join(FinalCommandLine, " ");
   return Success && Action.hasScanned();
 }
+
+DependencyActionController::~DependencyActionController() {}
diff --git a/clang/lib/Tooling/DependencyScanning/ModuleDepCollector.cpp b/clang/lib/Tooling/DependencyScanning/ModuleDepCollector.cpp
index e882067dca398..8cac033742a8c 100644
--- a/clang/lib/Tooling/DependencyScanning/ModuleDepCollector.cpp
+++ b/clang/lib/Tooling/DependencyScanning/ModuleDepCollector.cpp
@@ -56,16 +56,16 @@ static std::vector<std::string> splitString(std::string S, char Separator) {
 void ModuleDepCollector::addOutputPaths(CompilerInvocation &CI,
                                         ModuleDeps &Deps) {
   CI.getFrontendOpts().OutputFile =
-      Consumer.lookupModuleOutput(Deps.ID, ModuleOutputKind::ModuleFile);
+      Controller.lookupModuleOutput(Deps.ID, ModuleOutputKind::ModuleFile);
   if (!CI.getDiagnosticOpts().DiagnosticSerializationFile.empty())
     CI.getDiagnosticOpts().DiagnosticSerializationFile =
-        Consumer.lookupModuleOutput(
+        Controller.lookupModuleOutput(
             Deps.ID, ModuleOutputKind::DiagnosticSerializationFile);
   if (!CI.getDependencyOutputOpts().OutputFile.empty()) {
-    CI.getDependencyOutputOpts().OutputFile =
-        Consumer.lookupModuleOutput(Deps.ID, ModuleOutputKind::DependencyFile);
+    CI.getDependencyOutputOpts().OutputFile = Controller.lookupModuleOutput(
+        Deps.ID, ModuleOutputKind::DependencyFile);
     CI.getDependencyOutputOpts().Targets =
-        splitString(Consumer.lookupModuleOutput(
+        splitString(Controller.lookupModuleOutput(
                         Deps.ID, ModuleOutputKind::DependencyTargets),
                     '\0');
     if (!CI.getDependencyOutputOpts().OutputFile.empty() &&
@@ -205,7 +205,7 @@ void ModuleDepCollector::addModuleFiles(
     CompilerInvocation &CI, ArrayRef<ModuleID> ClangModuleDeps) const {
   for (const ModuleID &MID : ClangModuleDeps) {
     std::string PCMPath =
-        Consumer.lookupModuleOutput(MID, ModuleOutputKind::ModuleFile);
+        Controller.lookupModuleOutput(MID, ModuleOutputKind::ModuleFile);
     if (EagerLoadModules)
       CI.getFrontendOpts().ModuleFiles.push_back(std::move(PCMPath));
     else
@@ -577,11 +577,11 @@ void ModuleDepCollectorPP::addAffectingClangModule(
 ModuleDepCollector::ModuleDepCollector(
     std::unique_ptr<DependencyOutputOptions> Opts,
     CompilerInstance &ScanInstance, DependencyConsumer &C,
-    CompilerInvocation OriginalCI, bool OptimizeArgs, bool EagerLoadModules,
-    bool IsStdModuleP1689Format)
-    : ScanInstance(ScanInstance), Consumer(C), Opts(std::move(Opts)),
-      OriginalInvocation(std::move(OriginalCI)), OptimizeArgs(OptimizeArgs),
-      EagerLoadModules(EagerLoadModules),
+    DependencyActionController &Controller, CompilerInvocation OriginalCI,
+    bool OptimizeArgs, bool EagerLoadModules, bool IsStdModuleP1689Format)
+    : ScanInstance(ScanInstance), Consumer(C), Controller(Controller),
+      Opts(std::move(Opts)), OriginalInvocation(std::move(OriginalCI)),
+      OptimizeArgs(OptimizeArgs), EagerLoadModules(EagerLoadModules),
       IsStdModuleP1689Format(IsStdModuleP1689Format) {}
 
 void ModuleDepCollector::attachToPreprocessor(Preprocessor &PP) {
diff --git a/clang/lib/Tooling/ExpandResponseFilesCompilationDatabase.cpp b/clang/lib/Tooling/ExpandResponseFilesCompilationDatabase.cpp
index dd9b9c992d846..ebf8aa2a7628c 100644
--- a/clang/lib/Tooling/ExpandResponseFilesCompilationDatabase.cpp
+++ b/clang/lib/Tooling/ExpandResponseFilesCompilationDatabase.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "clang/Tooling/CompilationDatabase.h"
+#include "clang/Tooling/Tooling.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ConvertUTF.h"
@@ -48,28 +49,9 @@ class ExpandResponseFilesDatabase : public CompilationDatabase {
 
 private:
   std::vector<CompileCommand> expand(std::vector<CompileCommand> Cmds) const {
-    for (auto &Cmd : Cmds) {
-      bool SeenRSPFile = false;
-      llvm::SmallVector<const char *, 20> Argv;
-      Argv.reserve(Cmd.CommandLine.size());
-      for (auto &Arg : Cmd.CommandLine) {
-        Argv.push_back(Arg.c_str());
-        if (!Arg.empty())
-          SeenRSPFile |= Arg.front() == '@';
-      }
-      if (!SeenRSPFile)
-        continue;
-      llvm::BumpPtrAllocator Alloc;
-      llvm::cl::ExpansionContext ECtx(Alloc, Tokenizer);
-      llvm::Error Err = ECtx.setVFS(FS.get())
-                            .setCurrentDir(Cmd.Directory)
-                            .expandResponseFiles(Argv);
-      if (Err)
-        llvm::errs() << Err;
-      // Don't assign directly, Argv aliases CommandLine.
-      std::vector<std::string> ExpandedArgv(Argv.begin(), Argv.end());
-      Cmd.CommandLine = std::move(ExpandedArgv);
-    }
+    for (auto &Cmd : Cmds)
+      tooling::addExpandedResponseFiles(Cmd.CommandLine, Cmd.Directory,
+                                        Tokenizer, *FS);
     return Cmds;
   }
 
diff --git a/clang/lib/Tooling/Inclusions/Stdlib/StdSpecialSymbolMap.inc b/clang/lib/Tooling/Inclusions/Stdlib/StdSpecialSymbolMap.inc
index 692f61a45e6f4..13a8d310a95be 100644
--- a/clang/lib/Tooling/Inclusions/Stdlib/StdSpecialSymbolMap.inc
+++ b/clang/lib/Tooling/Inclusions/Stdlib/StdSpecialSymbolMap.inc
@@ -242,11 +242,153 @@ SYMBOL(abs, None, <stdlib.h>)
 SYMBOL(abs, None, <cmath>)
 SYMBOL(abs, None, <math.h>)
 
+// Only add headers for the generic atomic template.
+// Ignore variants (std::weak_ptr, std::shared_ptr).
+SYMBOL(atomic, std::, <atomic>)
+// atomic_* family symbols. <stdatomic.h> is for C compatibility.
+SYMBOL(atomic_bool, std::, <atomic>)
+SYMBOL(atomic_bool, None, <stdatomic.h>)
+SYMBOL(atomic_char, std::, <atomic>)
+SYMBOL(atomic_char, None, <stdatomic.h>)
+SYMBOL(atomic_char16_t, std::, <atomic>)
+SYMBOL(atomic_char16_t, None, <stdatomic.h>)
+SYMBOL(atomic_char32_t, std::, <atomic>)
+SYMBOL(atomic_char32_t, None, <stdatomic.h>)
+SYMBOL(atomic_char8_t, std::, <atomic>)
+SYMBOL(atomic_char8_t, None, <stdatomic.h>)
+SYMBOL(atomic_int, std::, <atomic>)
+SYMBOL(atomic_int, None, <stdatomic.h>)
+SYMBOL(atomic_int16_t, std::, <atomic>)
+SYMBOL(atomic_int16_t, None, <stdatomic.h>)
+SYMBOL(atomic_int32_t, std::, <atomic>)
+SYMBOL(atomic_int32_t, None, <stdatomic.h>)
+SYMBOL(atomic_int64_t, std::, <atomic>)
+SYMBOL(atomic_int64_t, None, <stdatomic.h>)
+SYMBOL(atomic_int8_t, std::, <atomic>)
+SYMBOL(atomic_int8_t, None, <stdatomic.h>)
+SYMBOL(atomic_int_fast16_t, std::, <atomic>)
+SYMBOL(atomic_int_fast16_t, None, <stdatomic.h>)
+SYMBOL(atomic_int_fast32_t, std::, <atomic>)
+SYMBOL(atomic_int_fast32_t, None, <stdatomic.h>)
+SYMBOL(atomic_int_fast64_t, std::, <atomic>)
+SYMBOL(atomic_int_fast64_t, None, <stdatomic.h>)
+SYMBOL(atomic_int_fast8_t, std::, <atomic>)
+SYMBOL(atomic_int_fast8_t, None, <stdatomic.h>)
+SYMBOL(atomic_int_least16_t, std::, <atomic>)
+SYMBOL(atomic_int_least16_t, None, <stdatomic.h>)
+SYMBOL(atomic_int_least32_t, std::, <atomic>)
+SYMBOL(atomic_int_least32_t, None, <stdatomic.h>)
+SYMBOL(atomic_int_least64_t, std::, <atomic>)
+SYMBOL(atomic_int_least64_t, None, <stdatomic.h>)
+SYMBOL(atomic_int_least8_t, std::, <atomic>)
+SYMBOL(atomic_int_least8_t, None, <stdatomic.h>)
+SYMBOL(atomic_intmax_t, std::, <atomic>)
+SYMBOL(atomic_intmax_t, None, <stdatomic.h>)
+SYMBOL(atomic_intptr_t, std::, <atomic>)
+SYMBOL(atomic_intptr_t, None, <stdatomic.h>)
+SYMBOL(atomic_llong, std::, <atomic>)
+SYMBOL(atomic_llong, None, <stdatomic.h>)
+SYMBOL(atomic_long, std::, <atomic>)
+SYMBOL(atomic_long, None, <stdatomic.h>)
+SYMBOL(atomic_ptrdiff_t, std::, <atomic>)
+SYMBOL(atomic_ptrdiff_t, None, <stdatomic.h>)
+SYMBOL(atomic_schar, std::, <atomic>)
+SYMBOL(atomic_schar, None, <stdatomic.h>)
+SYMBOL(atomic_short, std::, <atomic>)
+SYMBOL(atomic_short, None, <stdatomic.h>)
+SYMBOL(atomic_signed_lock_free, std::, <atomic>)
+SYMBOL(atomic_signed_lock_free, None, <stdatomic.h>)
+SYMBOL(atomic_size_t, std::, <atomic>)
+SYMBOL(atomic_size_t, None, <stdatomic.h>)
+SYMBOL(atomic_uchar, std::, <atomic>)
+SYMBOL(atomic_uchar, None, <stdatomic.h>)
+SYMBOL(atomic_uint, std::, <atomic>)
+SYMBOL(atomic_uint, None, <stdatomic.h>)
+SYMBOL(atomic_uint16_t, std::, <atomic>)
+SYMBOL(atomic_uint16_t, None, <stdatomic.h>)
+SYMBOL(atomic_uint32_t, std::, <atomic>)
+SYMBOL(atomic_uint32_t, None, <stdatomic.h>)
+SYMBOL(atomic_uint64_t, std::, <atomic>)
+SYMBOL(atomic_uint64_t, None, <stdatomic.h>)
+SYMBOL(atomic_uint8_t, std::, <atomic>)
+SYMBOL(atomic_uint8_t, None, <stdatomic.h>)
+SYMBOL(atomic_uint_fast16_t, std::, <atomic>)
+SYMBOL(atomic_uint_fast16_t, None, <stdatomic.h>)
+SYMBOL(atomic_uint_fast32_t, std::, <atomic>)
+SYMBOL(atomic_uint_fast32_t, None, <stdatomic.h>)
+SYMBOL(atomic_uint_fast64_t, std::, <atomic>)
+SYMBOL(atomic_uint_fast64_t, None, <stdatomic.h>)
+SYMBOL(atomic_uint_fast8_t, std::, <atomic>)
+SYMBOL(atomic_uint_fast8_t, None, <stdatomic.h>)
+SYMBOL(atomic_uint_least16_t, std::, <atomic>)
+SYMBOL(atomic_uint_least16_t, None, <stdatomic.h>)
+SYMBOL(atomic_uint_least32_t, std::, <atomic>)
+SYMBOL(atomic_uint_least32_t, None, <stdatomic.h>)
+SYMBOL(atomic_uint_least64_t, std::, <atomic>)
+SYMBOL(atomic_uint_least64_t, None, <stdatomic.h>)
+SYMBOL(atomic_uint_least8_t, std::, <atomic>)
+SYMBOL(atomic_uint_least8_t, None, <stdatomic.h>)
+SYMBOL(atomic_uintmax_t, std::, <atomic>)
+SYMBOL(atomic_uintmax_t, None, <stdatomic.h>)
+SYMBOL(atomic_uintptr_t, std::, <atomic>)
+SYMBOL(atomic_uintptr_t, None, <stdatomic.h>)
+SYMBOL(atomic_ullong, std::, <atomic>)
+SYMBOL(atomic_ullong, None, <stdatomic.h>)
+SYMBOL(atomic_ulong, std::, <atomic>)
+SYMBOL(atomic_ulong, None, <stdatomic.h>)
+SYMBOL(atomic_unsigned_lock_free, std::, <atomic>)
+SYMBOL(atomic_unsigned_lock_free, None, <stdatomic.h>)
+SYMBOL(atomic_ushort, std::, <atomic>)
+SYMBOL(atomic_ushort, None, <stdatomic.h>)
+SYMBOL(atomic_wchar_t, std::, <atomic>)
+SYMBOL(atomic_wchar_t, None, <stdatomic.h>)
+
 // std::get has a few variants for different types (tuple, array, pair etc)
 // which is tricky to disambiguate without type information.
 // Don't set any header for it, as it comes with the type header.
 SYMBOL(get, std::, /*no headers*/)
 
+// cppreference symbol index page was missing these symbols.
+// Remove them when the cppreference offline archive catches up.
+SYMBOL(index_sequence, std::, <utility>)
+SYMBOL(index_sequence_for, std::, <utility>)
+SYMBOL(make_index_sequence, std::, <utility>)
+SYMBOL(make_integer_sequence, std::, <utility>)
+
+// The std::placeholder symbols (_1, ..., _N) are listed in the cppreference
+// placeholder.html, but the index only contains a single entry with "_1, _2, ..., _N"
+// text, which are not handled by the script.
+// N is an implementation-defined number (10 for libc++; 29 for libstdc++).
+SYMBOL(_1, std::placeholders::, <functional>)
+SYMBOL(_2, std::placeholders::, <functional>)
+SYMBOL(_3, std::placeholders::, <functional>)
+SYMBOL(_4, std::placeholders::, <functional>)
+SYMBOL(_5, std::placeholders::, <functional>)
+SYMBOL(_6, std::placeholders::, <functional>)
+SYMBOL(_7, std::placeholders::, <functional>)
+SYMBOL(_8, std::placeholders::, <functional>)
+SYMBOL(_9, std::placeholders::, <functional>)
+SYMBOL(_10, std::placeholders::, <functional>)
+SYMBOL(_11, std::placeholders::, <functional>)
+SYMBOL(_12, std::placeholders::, <functional>)
+SYMBOL(_13, std::placeholders::, <functional>)
+SYMBOL(_14, std::placeholders::, <functional>)
+SYMBOL(_15, std::placeholders::, <functional>)
+SYMBOL(_16, std::placeholders::, <functional>)
+SYMBOL(_17, std::placeholders::, <functional>)
+SYMBOL(_18, std::placeholders::, <functional>)
+SYMBOL(_19, std::placeholders::, <functional>)
+SYMBOL(_20, std::placeholders::, <functional>)
+SYMBOL(_21, std::placeholders::, <functional>)
+SYMBOL(_22, std::placeholders::, <functional>)
+SYMBOL(_23, std::placeholders::, <functional>)
+SYMBOL(_24, std::placeholders::, <functional>)
+SYMBOL(_25, std::placeholders::, <functional>)
+SYMBOL(_26, std::placeholders::, <functional>)
+SYMBOL(_27, std::placeholders::, <functional>)
+SYMBOL(_28, std::placeholders::, <functional>)
+SYMBOL(_29, std::placeholders::, <functional>)
+
 // Macros
 SYMBOL(NULL, None, <cstddef>)
 SYMBOL(NULL, None, <stddef.h>)
diff --git a/clang/lib/Tooling/Inclusions/Stdlib/StdSymbolMap.inc b/clang/lib/Tooling/Inclusions/Stdlib/StdSymbolMap.inc
index 1712c06c78704..a08ec11e77a4a 100644
--- a/clang/lib/Tooling/Inclusions/Stdlib/StdSymbolMap.inc
+++ b/clang/lib/Tooling/Inclusions/Stdlib/StdSymbolMap.inc
@@ -3416,6 +3416,14 @@ SYMBOL(year_month_weekday_last, std::chrono::, <chrono>)
 SYMBOL(zoned_seconds, std::chrono::, <chrono>)
 SYMBOL(zoned_time, std::chrono::, <chrono>)
 SYMBOL(zoned_traits, std::chrono::, <chrono>)
+SYMBOL(par, std::execution::, <execution>)
+SYMBOL(par_unseq, std::execution::, <execution>)
+SYMBOL(parallel_policy, std::execution::, <execution>)
+SYMBOL(parallel_unsequenced_policy, std::execution::, <execution>)
+SYMBOL(seq, std::execution::, <execution>)
+SYMBOL(sequenced_policy, std::execution::, <execution>)
+SYMBOL(unseq, std::execution::, <execution>)
+SYMBOL(unsequenced_policy, std::execution::, <execution>)
 SYMBOL(absolute, std::filesystem::, <filesystem>)
 SYMBOL(canonical, std::filesystem::, <filesystem>)
 SYMBOL(copy, std::filesystem::, <filesystem>)
@@ -3470,6 +3478,32 @@ SYMBOL(symlink_status, std::filesystem::, <filesystem>)
 SYMBOL(temp_directory_path, std::filesystem::, <filesystem>)
 SYMBOL(u8path, std::filesystem::, <filesystem>)
 SYMBOL(weakly_canonical, std::filesystem::, <filesystem>)
+SYMBOL(e, std::numbers::, <numbers>)
+SYMBOL(e_v, std::numbers::, <numbers>)
+SYMBOL(egamma, std::numbers::, <numbers>)
+SYMBOL(egamma_v, std::numbers::, <numbers>)
+SYMBOL(inv_pi, std::numbers::, <numbers>)
+SYMBOL(inv_pi_v, std::numbers::, <numbers>)
+SYMBOL(inv_sqrt3, std::numbers::, <numbers>)
+SYMBOL(inv_sqrt3_v, std::numbers::, <numbers>)
+SYMBOL(inv_sqrtpi, std::numbers::, <numbers>)
+SYMBOL(inv_sqrtpi_v, std::numbers::, <numbers>)
+SYMBOL(ln10, std::numbers::, <numbers>)
+SYMBOL(ln10_v, std::numbers::, <numbers>)
+SYMBOL(ln2, std::numbers::, <numbers>)
+SYMBOL(ln2_v, std::numbers::, <numbers>)
+SYMBOL(log10e, std::numbers::, <numbers>)
+SYMBOL(log10e_v, std::numbers::, <numbers>)
+SYMBOL(log2e, std::numbers::, <numbers>)
+SYMBOL(log2e_v, std::numbers::, <numbers>)
+SYMBOL(phi, std::numbers::, <numbers>)
+SYMBOL(phi_v, std::numbers::, <numbers>)
+SYMBOL(pi, std::numbers::, <numbers>)
+SYMBOL(pi_v, std::numbers::, <numbers>)
+SYMBOL(sqrt2, std::numbers::, <numbers>)
+SYMBOL(sqrt2_v, std::numbers::, <numbers>)
+SYMBOL(sqrt3, std::numbers::, <numbers>)
+SYMBOL(sqrt3_v, std::numbers::, <numbers>)
 SYMBOL(basic_string, std::pmr::, <string>)
 SYMBOL(cmatch, std::pmr::, <regex>)
 SYMBOL(deque, std::pmr::, <deque>)
@@ -3504,6 +3538,241 @@ SYMBOL(vector, std::pmr::, <vector>)
 SYMBOL(wcmatch, std::pmr::, <regex>)
 SYMBOL(wsmatch, std::pmr::, <regex>)
 SYMBOL(wstring, std::pmr::, <string>)
+SYMBOL(adjacent_find, std::ranges::, <algorithm>)
+SYMBOL(advance, std::ranges::, <iterator>)
+SYMBOL(all_of, std::ranges::, <algorithm>)
+SYMBOL(any_of, std::ranges::, <algorithm>)
+SYMBOL(as_const_view, std::ranges::, <ranges>)
+SYMBOL(as_rvalue_view, std::ranges::, <ranges>)
+SYMBOL(basic_istream_view, std::ranges::, <ranges>)
+SYMBOL(begin, std::ranges::, <ranges>)
+SYMBOL(bidirectional_range, std::ranges::, <ranges>)
+SYMBOL(binary_transform_result, std::ranges::, <algorithm>)
+SYMBOL(borrowed_iterator_t, std::ranges::, <ranges>)
+SYMBOL(borrowed_range, std::ranges::, <ranges>)
+SYMBOL(borrowed_subrange_t, std::ranges::, <ranges>)
+SYMBOL(cbegin, std::ranges::, <ranges>)
+SYMBOL(cdata, std::ranges::, <ranges>)
+SYMBOL(cend, std::ranges::, <ranges>)
+SYMBOL(clamp, std::ranges::, <algorithm>)
+SYMBOL(common_range, std::ranges::, <ranges>)
+SYMBOL(common_view, std::ranges::, <ranges>)
+SYMBOL(const_iterator_t, std::ranges::, <ranges>)
+SYMBOL(constant_range, std::ranges::, <ranges>)
+SYMBOL(construct_at, std::ranges::, <memory>)
+SYMBOL(contains, std::ranges::, <algorithm>)
+SYMBOL(contains_subrange, std::ranges::, <algorithm>)
+SYMBOL(contiguous_range, std::ranges::, <ranges>)
+SYMBOL(copy, std::ranges::, <algorithm>)
+SYMBOL(copy_backward, std::ranges::, <algorithm>)
+SYMBOL(copy_backward_result, std::ranges::, <algorithm>)
+SYMBOL(copy_if, std::ranges::, <algorithm>)
+SYMBOL(copy_if_result, std::ranges::, <algorithm>)
+SYMBOL(copy_n, std::ranges::, <algorithm>)
+SYMBOL(copy_n_result, std::ranges::, <algorithm>)
+SYMBOL(copy_result, std::ranges::, <algorithm>)
+SYMBOL(count, std::ranges::, <algorithm>)
+SYMBOL(count_if, std::ranges::, <algorithm>)
+SYMBOL(crbegin, std::ranges::, <ranges>)
+SYMBOL(crend, std::ranges::, <ranges>)
+SYMBOL(dangling, std::ranges::, <ranges>)
+SYMBOL(data, std::ranges::, <ranges>)
+SYMBOL(destroy, std::ranges::, <memory>)
+SYMBOL(destroy_at, std::ranges::, <memory>)
+SYMBOL(destroy_n, std::ranges::, <memory>)
+SYMBOL(disable_sized_range, std::ranges::, <ranges>)
+SYMBOL(distance, std::ranges::, <iterator>)
+SYMBOL(drop_view, std::ranges::, <ranges>)
+SYMBOL(drop_while_view, std::ranges::, <ranges>)
+SYMBOL(elements_view, std::ranges::, <ranges>)
+SYMBOL(empty, std::ranges::, <ranges>)
+SYMBOL(empty_view, std::ranges::, <ranges>)
+SYMBOL(enable_borrowed_range, std::ranges::, <ranges>)
+SYMBOL(enable_view, std::ranges::, <ranges>)
+SYMBOL(end, std::ranges::, <ranges>)
+SYMBOL(ends_with, std::ranges::, <algorithm>)
+SYMBOL(equal, std::ranges::, <algorithm>)
+SYMBOL(equal_to, std::ranges::, <functional>)
+SYMBOL(fill, std::ranges::, <algorithm>)
+SYMBOL(fill_n, std::ranges::, <algorithm>)
+SYMBOL(filter_view, std::ranges::, <ranges>)
+SYMBOL(find, std::ranges::, <algorithm>)
+SYMBOL(find_end, std::ranges::, <algorithm>)
+SYMBOL(find_first_of, std::ranges::, <algorithm>)
+SYMBOL(find_if, std::ranges::, <algorithm>)
+SYMBOL(find_if_not, std::ranges::, <algorithm>)
+SYMBOL(find_last, std::ranges::, <algorithm>)
+SYMBOL(find_last_if, std::ranges::, <algorithm>)
+SYMBOL(find_last_if_not, std::ranges::, <algorithm>)
+SYMBOL(for_each, std::ranges::, <algorithm>)
+SYMBOL(for_each_n, std::ranges::, <algorithm>)
+SYMBOL(for_each_n_result, std::ranges::, <algorithm>)
+SYMBOL(for_each_result, std::ranges::, <algorithm>)
+SYMBOL(forward_range, std::ranges::, <ranges>)
+SYMBOL(generate, std::ranges::, <algorithm>)
+SYMBOL(generate_n, std::ranges::, <algorithm>)
+SYMBOL(greater, std::ranges::, <functional>)
+SYMBOL(greater_equal, std::ranges::, <functional>)
+SYMBOL(in_found_result, std::ranges::, <algorithm>)
+SYMBOL(in_fun_result, std::ranges::, <algorithm>)
+SYMBOL(in_in_out_result, std::ranges::, <algorithm>)
+SYMBOL(in_in_result, std::ranges::, <algorithm>)
+SYMBOL(in_out_out_result, std::ranges::, <algorithm>)
+SYMBOL(in_out_result, std::ranges::, <algorithm>)
+SYMBOL(in_value_result, std::ranges::, <algorithm>)
+SYMBOL(includes, std::ranges::, <algorithm>)
+SYMBOL(inplace_merge, std::ranges::, <algorithm>)
+SYMBOL(input_range, std::ranges::, <ranges>)
+SYMBOL(iota, std::ranges::, <numeric>)
+SYMBOL(iota_result, std::ranges::, <numeric>)
+SYMBOL(iota_view, std::ranges::, <ranges>)
+SYMBOL(is_heap, std::ranges::, <algorithm>)
+SYMBOL(is_heap_until, std::ranges::, <algorithm>)
+SYMBOL(is_partitioned, std::ranges::, <algorithm>)
+SYMBOL(is_permutation, std::ranges::, <algorithm>)
+SYMBOL(is_sorted, std::ranges::, <algorithm>)
+SYMBOL(is_sorted_until, std::ranges::, <algorithm>)
+SYMBOL(istream_view, std::ranges::, <ranges>)
+SYMBOL(iter_move, std::ranges::, <iterator>)
+SYMBOL(iter_swap, std::ranges::, <iterator>)
+SYMBOL(iterator_t, std::ranges::, <ranges>)
+SYMBOL(join_view, std::ranges::, <ranges>)
+SYMBOL(join_with_view, std::ranges::, <ranges>)
+SYMBOL(keys_view, std::ranges::, <ranges>)
+SYMBOL(lazy_split_view, std::ranges::, <ranges>)
+SYMBOL(less, std::ranges::, <functional>)
+SYMBOL(less_equal, std::ranges::, <functional>)
+SYMBOL(lexicographical_compare, std::ranges::, <algorithm>)
+SYMBOL(make_heap, std::ranges::, <algorithm>)
+SYMBOL(max, std::ranges::, <algorithm>)
+SYMBOL(max_element, std::ranges::, <algorithm>)
+SYMBOL(merge, std::ranges::, <algorithm>)
+SYMBOL(merge_result, std::ranges::, <algorithm>)
+SYMBOL(min, std::ranges::, <algorithm>)
+SYMBOL(min_element, std::ranges::, <algorithm>)
+SYMBOL(min_max_result, std::ranges::, <algorithm>)
+SYMBOL(minmax, std::ranges::, <algorithm>)
+SYMBOL(minmax_element, std::ranges::, <algorithm>)
+SYMBOL(minmax_element_result, std::ranges::, <algorithm>)
+SYMBOL(minmax_result, std::ranges::, <algorithm>)
+SYMBOL(mismatch, std::ranges::, <algorithm>)
+SYMBOL(mismatch_result, std::ranges::, <algorithm>)
+SYMBOL(move, std::ranges::, <algorithm>)
+SYMBOL(move_backward, std::ranges::, <algorithm>)
+SYMBOL(move_backward_result, std::ranges::, <algorithm>)
+SYMBOL(move_result, std::ranges::, <algorithm>)
+SYMBOL(next, std::ranges::, <iterator>)
+SYMBOL(next_permutation, std::ranges::, <algorithm>)
+SYMBOL(next_permutation_result, std::ranges::, <algorithm>)
+SYMBOL(none_of, std::ranges::, <algorithm>)
+SYMBOL(not_equal_to, std::ranges::, <functional>)
+SYMBOL(nth_element, std::ranges::, <algorithm>)
+SYMBOL(out_value_result, std::ranges::, <algorithm>)
+SYMBOL(output_range, std::ranges::, <ranges>)
+SYMBOL(owning_view, std::ranges::, <ranges>)
+SYMBOL(partial_sort, std::ranges::, <algorithm>)
+SYMBOL(partial_sort_copy, std::ranges::, <algorithm>)
+SYMBOL(partial_sort_copy_result, std::ranges::, <algorithm>)
+SYMBOL(partition, std::ranges::, <algorithm>)
+SYMBOL(partition_copy, std::ranges::, <algorithm>)
+SYMBOL(partition_copy_result, std::ranges::, <algorithm>)
+SYMBOL(partition_point, std::ranges::, <algorithm>)
+SYMBOL(pop_heap, std::ranges::, <algorithm>)
+SYMBOL(prev, std::ranges::, <iterator>)
+SYMBOL(prev_permutation, std::ranges::, <algorithm>)
+SYMBOL(prev_permutation_result, std::ranges::, <algorithm>)
+SYMBOL(push_heap, std::ranges::, <algorithm>)
+SYMBOL(random_access_range, std::ranges::, <ranges>)
+SYMBOL(range, std::ranges::, <ranges>)
+SYMBOL(range_const_reference_t, std::ranges::, <ranges>)
+SYMBOL(range_difference_t, std::ranges::, <ranges>)
+SYMBOL(range_reference_t, std::ranges::, <ranges>)
+SYMBOL(range_rvalue_reference_t, std::ranges::, <ranges>)
+SYMBOL(range_size_t, std::ranges::, <ranges>)
+SYMBOL(range_value_t, std::ranges::, <ranges>)
+SYMBOL(rbegin, std::ranges::, <ranges>)
+SYMBOL(ref_view, std::ranges::, <ranges>)
+SYMBOL(remove, std::ranges::, <algorithm>)
+SYMBOL(remove_copy, std::ranges::, <algorithm>)
+SYMBOL(remove_copy_if, std::ranges::, <algorithm>)
+SYMBOL(remove_copy_if_result, std::ranges::, <algorithm>)
+SYMBOL(remove_copy_result, std::ranges::, <algorithm>)
+SYMBOL(remove_if, std::ranges::, <algorithm>)
+SYMBOL(rend, std::ranges::, <ranges>)
+SYMBOL(replace, std::ranges::, <algorithm>)
+SYMBOL(replace_copy, std::ranges::, <algorithm>)
+SYMBOL(replace_copy_if, std::ranges::, <algorithm>)
+SYMBOL(replace_copy_if_result, std::ranges::, <algorithm>)
+SYMBOL(replace_copy_result, std::ranges::, <algorithm>)
+SYMBOL(replace_if, std::ranges::, <algorithm>)
+SYMBOL(reverse, std::ranges::, <algorithm>)
+SYMBOL(reverse_copy, std::ranges::, <algorithm>)
+SYMBOL(reverse_copy_result, std::ranges::, <algorithm>)
+SYMBOL(reverse_view, std::ranges::, <ranges>)
+SYMBOL(rotate, std::ranges::, <algorithm>)
+SYMBOL(rotate_copy, std::ranges::, <algorithm>)
+SYMBOL(rotate_copy_result, std::ranges::, <algorithm>)
+SYMBOL(sample, std::ranges::, <algorithm>)
+SYMBOL(search, std::ranges::, <algorithm>)
+SYMBOL(search_n, std::ranges::, <algorithm>)
+SYMBOL(sentinel_t, std::ranges::, <ranges>)
+SYMBOL(set_difference, std::ranges::, <algorithm>)
+SYMBOL(set_difference_result, std::ranges::, <algorithm>)
+SYMBOL(set_intersection, std::ranges::, <algorithm>)
+SYMBOL(set_intersection_result, std::ranges::, <algorithm>)
+SYMBOL(set_symmetric_difference, std::ranges::, <algorithm>)
+SYMBOL(set_symmetric_difference_result, std::ranges::, <algorithm>)
+SYMBOL(set_union, std::ranges::, <algorithm>)
+SYMBOL(set_union_result, std::ranges::, <algorithm>)
+SYMBOL(shift_left, std::ranges::, <algorithm>)
+SYMBOL(shift_right, std::ranges::, <algorithm>)
+SYMBOL(shuffle, std::ranges::, <algorithm>)
+SYMBOL(single_view, std::ranges::, <ranges>)
+SYMBOL(size, std::ranges::, <ranges>)
+SYMBOL(sized_range, std::ranges::, <ranges>)
+SYMBOL(sort, std::ranges::, <algorithm>)
+SYMBOL(sort_heap, std::ranges::, <algorithm>)
+SYMBOL(split_view, std::ranges::, <ranges>)
+SYMBOL(ssize, std::ranges::, <ranges>)
+SYMBOL(stable_partition, std::ranges::, <algorithm>)
+SYMBOL(stable_sort, std::ranges::, <algorithm>)
+SYMBOL(starts_with, std::ranges::, <algorithm>)
+SYMBOL(subrange, std::ranges::, <ranges>)
+SYMBOL(subrange_kind, std::ranges::, <ranges>)
+SYMBOL(swap, std::ranges::, <concepts>)
+SYMBOL(swap_ranges, std::ranges::, <algorithm>)
+SYMBOL(swap_ranges_result, std::ranges::, <algorithm>)
+SYMBOL(take_view, std::ranges::, <ranges>)
+SYMBOL(take_while_view, std::ranges::, <ranges>)
+SYMBOL(to, std::ranges::, <ranges>)
+SYMBOL(transform, std::ranges::, <algorithm>)
+SYMBOL(transform_view, std::ranges::, <ranges>)
+SYMBOL(unary_transform_result, std::ranges::, <algorithm>)
+SYMBOL(uninitialized_copy, std::ranges::, <memory>)
+SYMBOL(uninitialized_copy_n, std::ranges::, <memory>)
+SYMBOL(uninitialized_copy_n_result, std::ranges::, <memory>)
+SYMBOL(uninitialized_copy_result, std::ranges::, <memory>)
+SYMBOL(uninitialized_default_construct, std::ranges::, <memory>)
+SYMBOL(uninitialized_default_construct_n, std::ranges::, <memory>)
+SYMBOL(uninitialized_fill, std::ranges::, <memory>)
+SYMBOL(uninitialized_fill_n, std::ranges::, <memory>)
+SYMBOL(uninitialized_move, std::ranges::, <memory>)
+SYMBOL(uninitialized_move_n, std::ranges::, <memory>)
+SYMBOL(uninitialized_move_n_result, std::ranges::, <memory>)
+SYMBOL(uninitialized_move_result, std::ranges::, <memory>)
+SYMBOL(uninitialized_value_construct, std::ranges::, <memory>)
+SYMBOL(uninitialized_value_construct_n, std::ranges::, <memory>)
+SYMBOL(unique, std::ranges::, <algorithm>)
+SYMBOL(unique_copy, std::ranges::, <algorithm>)
+SYMBOL(unique_copy_result, std::ranges::, <algorithm>)
+SYMBOL(values_view, std::ranges::, <ranges>)
+SYMBOL(view, std::ranges::, <ranges>)
+SYMBOL(view_base, std::ranges::, <ranges>)
+SYMBOL(view_interface, std::ranges::, <ranges>)
+SYMBOL(viewable_range, std::ranges::, <ranges>)
+SYMBOL(wistream_view, std::ranges::, <ranges>)
+SYMBOL(zip_transform_view, std::ranges::, <ranges>)
+SYMBOL(zip_view, std::ranges::, <ranges>)
 SYMBOL(ECMAScript, std::regex_constants::, <regex>)
 SYMBOL(awk, std::regex_constants::, <regex>)
 SYMBOL(basic, std::regex_constants::, <regex>)
diff --git a/clang/lib/Tooling/Tooling.cpp b/clang/lib/Tooling/Tooling.cpp
index 3048f69eef251..dd22cfedd0ffe 100644
--- a/clang/lib/Tooling/Tooling.cpp
+++ b/clang/lib/Tooling/Tooling.cpp
@@ -43,6 +43,7 @@
 #include "llvm/Option/OptTable.h"
 #include "llvm/Option/Option.h"
 #include "llvm/Support/Casting.h"
+#include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/FileSystem.h"
@@ -299,6 +300,31 @@ void addTargetAndModeForProgramName(std::vector<std::string> &CommandLine,
   }
 }
 
+void addExpandedResponseFiles(std::vector<std::string> &CommandLine,
+                              llvm::StringRef WorkingDir,
+                              llvm::cl::TokenizerCallback Tokenizer,
+                              llvm::vfs::FileSystem &FS) {
+  bool SeenRSPFile = false;
+  llvm::SmallVector<const char *, 20> Argv;
+  Argv.reserve(CommandLine.size());
+  for (auto &Arg : CommandLine) {
+    Argv.push_back(Arg.c_str());
+    if (!Arg.empty())
+      SeenRSPFile |= Arg.front() == '@';
+  }
+  if (!SeenRSPFile)
+    return;
+  llvm::BumpPtrAllocator Alloc;
+  llvm::cl::ExpansionContext ECtx(Alloc, Tokenizer);
+  llvm::Error Err =
+      ECtx.setVFS(&FS).setCurrentDir(WorkingDir).expandResponseFiles(Argv);
+  if (Err)
+    llvm::errs() << Err;
+  // Don't assign directly, Argv aliases CommandLine.
+  std::vector<std::string> ExpandedArgv(Argv.begin(), Argv.end());
+  CommandLine = std::move(ExpandedArgv);
+}
+
 } // namespace tooling
 } // namespace clang
 
@@ -684,7 +710,7 @@ std::unique_ptr<ASTUnit> buildASTFromCodeWithArgs(
 
   if (!Invocation.run())
     return nullptr;
- 
+
   assert(ASTs.size() == 1);
   return std::move(ASTs[0]);
 }
diff --git a/clang/test/ARCMT/verify.m b/clang/test/ARCMT/verify.m
index 9b46c1a729ac2..7480e0dc9677c 100644
--- a/clang/test/ARCMT/verify.m
+++ b/clang/test/ARCMT/verify.m
@@ -8,6 +8,9 @@
 #error should not be ignored
 // expected-error@-1 {{should not be ignored}}
 
+#error
+// expected-error@-1 {{}}
+
 //      CHECK: error: no expected directives found: consider use of 'expected-no-diagnostics'
 // CHECK-NEXT: error: 'error' diagnostics seen but not expected:
 // CHECK-NEXT:   (frontend): error reading '{{.*}}verify.m.tmp.invalid'
diff --git a/clang/test/AST/ast-dump-aarch64-sve-types.c b/clang/test/AST/ast-dump-aarch64-sve-types.c
index bfb8bc220e8d8..dd14868c42b9b 100644
--- a/clang/test/AST/ast-dump-aarch64-sve-types.c
+++ b/clang/test/AST/ast-dump-aarch64-sve-types.c
@@ -47,3 +47,6 @@
 
 // CHECK: TypedefDecl {{.*}} implicit __SVBool_t '__SVBool_t'
 // CHECK-NEXT: -BuiltinType {{.*}} '__SVBool_t'
+
+// CHECK: TypedefDecl {{.*}} implicit __SVCount_t '__SVCount_t'
+// CHECK-NEXT: -BuiltinType {{.*}} '__SVCount_t'
diff --git a/clang/test/AST/ast-dump-using-template.cpp b/clang/test/AST/ast-dump-using-template.cpp
index da18f0499f54d..de3ce277fd24f 100644
--- a/clang/test/AST/ast-dump-using-template.cpp
+++ b/clang/test/AST/ast-dump-using-template.cpp
@@ -9,8 +9,11 @@ template<typename T> class S {
  public:
    S(T);
 };
+template<typename T> struct S2 { S2(T); };
+template <typename T> S2(T t) -> S2<T>;
 }
 using ns::S;
+using ns::S2;
 
 // TemplateName in TemplateSpecializationType.
 template<typename T>
@@ -36,3 +39,10 @@ using C = decltype(DeducedTemplateSpecializationT);
 // CHECK-NEXT:  |-DeclRefExpr {{.*}}
 // CHECK-NEXT:  `-ElaboratedType {{.*}} 'S<int>' sugar
 // CHECK-NEXT:    `-DeducedTemplateSpecializationType {{.*}} 'ns::S<int>' sugar using
+
+S2 DeducedTemplateSpecializationT2(123);
+using D = decltype(DeducedTemplateSpecializationT2);
+// CHECK:      DecltypeType {{.*}}
+// CHECK-NEXT:  |-DeclRefExpr {{.*}}
+// CHECK-NEXT:  `-ElaboratedType {{.*}} 'S2<int>' sugar
+// CHECK-NEXT:    `-DeducedTemplateSpecializationType {{.*}} 'S2<int>' sugar using
diff --git a/clang/test/Analysis/bitwise-ops-nocrash.c b/clang/test/Analysis/bitwise-ops-nocrash.c
new file mode 100644
index 0000000000000..214acca78e6f4
--- /dev/null
+++ b/clang/test/Analysis/bitwise-ops-nocrash.c
@@ -0,0 +1,22 @@
+// RUN: %clang_analyze_cc1 -analyzer-checker=core -analyzer-output=text -triple x86_64-linux-gnu -Wno-shift-count-overflow -verify %s
+
+#define offsetof(type,memb) ((unsigned long)&((type*)0)->memb)
+
+typedef struct {
+  unsigned long guest_counter;
+  unsigned int guest_fpc;
+} S;
+
+// no crash
+int left_shift_overflow_no_crash(unsigned int i) {
+  unsigned shift = 323U; // expected-note{{'shift' initialized to 323}}
+  switch (i) { // expected-note{{Control jumps to 'case 8:'  at line 14}}
+  case offsetof(S, guest_fpc):
+    return 3 << shift; // expected-warning{{The result of the left shift is undefined due to shifting by '323', which is greater or equal to the width of type 'int'}}
+    // expected-note@-1{{The result of the left shift is undefined due to shifting by '323', which is greater or equal to the width of type 'int'}}
+  default:
+    break;
+  }
+
+  return 0;
+}
diff --git a/clang/test/C/C2x/n2607.c b/clang/test/C/C2x/n2607.c
new file mode 100644
index 0000000000000..763cfd9253cf2
--- /dev/null
+++ b/clang/test/C/C2x/n2607.c
@@ -0,0 +1,65 @@
+// RUN: %clang_cc1 -std=c2x -Wall -pedantic -verify %s
+// RUN: %clang_cc1 -std=c17 -Wall -pedantic -verify %s
+
+/* WG14 N2607: Partial
+ * Compatibility of Pointers to Arrays with Qualifiers
+ *
+ * FIXME: We consider this partially implemented because there are still issues
+ * with the composite type from a conditional operator. Further, we don't issue
+ * any diagnostics in C17 or earlier when we need at least a pedantic
+ * diagnostic about the type incompatibilities.
+ */
+
+void matrix_fun(int N, const float x[N][N]);
+void test1(void) {
+  int N = 100;
+  float x[N][N];
+  // FIXME: This is OK in C2x but should be diagnosed as passing incompatible
+  // pointer types in C17 and earlier.
+  matrix_fun(N, x);
+}
+
+#define TEST(Expr, Type) _Generic(Expr, Type : 1)
+
+void test2(void) {
+  typedef int array[1];
+  array reg_array;
+  const array const_array;
+
+  // An array and its elements are identically qualified. We have to test this
+  // using pointers to the array and element, because the controlling
+  // expression of _Generic will undergo lvalue conversion, which drops
+  // qualifiers.
+  (void)_Generic(&reg_array, int (*)[1] : 1);
+  (void)_Generic(&reg_array[0], int * : 1);
+
+  (void)_Generic(&const_array, const int (*)[1] : 1);
+  (void)_Generic(&const_array[0], const int * : 1);
+
+  // But identical qualification does *not* apply to the _Atomic specifier,
+  // because that's a special case. You cannot apply the _Atomic specifier or
+  // qualifier to an array type directly.
+  _Atomic array atomic_array;       // expected-error {{_Atomic cannot be applied to array type 'array'}}
+  _Atomic(array) also_atomic_array; // expected-error {{_Atomic cannot be applied to array type 'array'}}
+
+  // Ensure we support arrays of restrict-qualified pointer types.
+  int *restrict array_of_restricted_ptrs[1];
+  int *restrict md_array_of_restricted_ptrs[1][1];
+}
+
+void test3(void) {
+  // Validate that we pick the correct composite type for a conditional
+  // operator in the presence of qualifiers.
+  const int const_array[1];
+  int array[1];
+
+  // FIXME: the type here should be `const int (*)[1]`, but for some reason,
+  // Clang is deciding it's `void *`. This relates to N2607 because the
+  // conditional operator is not properly implementing 6.5.15p7 regarding
+  // qualifiers, despite that wording not being touched by this paper.
+  // However, it should get a pedantic diagnostic in C17 about use of
+  // incompatible pointer types.
+  (void)_Generic(1 ? &const_array : &array, const int (*)[1] : 1);   /* expected-error {{controlling expression type 'void *' not compatible with any generic association type}}
+                                                                        expected-warning {{pointer type mismatch ('const int (*)[1]' and 'int (*)[1]')}}
+                                                                      */
+}
diff --git a/clang/test/CMakeLists.txt b/clang/test/CMakeLists.txt
index 1d6377b5f2d80..df73c869490ce 100644
--- a/clang/test/CMakeLists.txt
+++ b/clang/test/CMakeLists.txt
@@ -71,6 +71,9 @@ list(APPEND CLANG_TEST_DEPS
   clang-refactor
   clang-diff
   clang-scan-deps
+  clang-linker-wrapper
+  clang-offload-bundler
+  clang-offload-packager
   diagtool
   hmaptool
   )
diff --git a/clang/test/CXX/drs/dr3xx.cpp b/clang/test/CXX/drs/dr3xx.cpp
index cca9470bcfd6c..e3fe8d932b30a 100644
--- a/clang/test/CXX/drs/dr3xx.cpp
+++ b/clang/test/CXX/drs/dr3xx.cpp
@@ -161,6 +161,15 @@ namespace dr308 { // dr308: yes
   struct C : A {};
   struct D : B, C {};
   void f() {
+    // NB: the warning here is correct despite being the opposite of the
+    // comments in the catch handlers. The "unreachable" comment is correct
+    // because there is an ambiguous base path to A from the D that is thrown.
+    // The warnings generated are also correct because the handlers handle
+    // const B& and const A& and we don't check to see if other derived classes
+    // exist that would cause an ambiguous base path. We issue the diagnostic
+    // despite the potential for a false positive because users are not
+    // expected to have ambiguous base paths all that often, so the false
+    // positive rate should be acceptably low.
     try {
       throw D();
     } catch (const A&) { // expected-note {{for type 'const A &'}}
diff --git a/clang/test/CXX/expr/expr.const/p6-2a.cpp b/clang/test/CXX/expr/expr.const/p6-2a.cpp
index 2ef067c549003..a937474d53b22 100644
--- a/clang/test/CXX/expr/expr.const/p6-2a.cpp
+++ b/clang/test/CXX/expr/expr.const/p6-2a.cpp
@@ -43,14 +43,12 @@ struct Temporary {
 constexpr Temporary t = {3}; // expected-error {{must have constant destruction}} expected-note {{created here}} expected-note {{in call}}
 
 namespace P1073R3 {
-consteval int f() { return 42; } // expected-note 3 {{declared here}}
+consteval int f() { return 42; } // expected-note 2 {{declared here}}
 consteval auto g() { return f; }
-// FIXME: there should be no diagnostics associated with either h() or r.
-consteval int h(int (*p)() = g()) { return p(); } // expected-error {{call to consteval function 'P1073R3::g' is not a constant expression}} \
-                                                     expected-note {{declared here}} \
-                                                     expected-note {{pointer to a consteval declaration is not a constant expression}}
-constexpr int r = h();   // expected-note {{in the default initalizer of 'p'}}
+consteval int h(int (*p)() = g()) { return p(); }
+constexpr int r = h();
 constexpr auto e = g();  // expected-error {{call to consteval function 'P1073R3::g' is not a constant expression}} \
                             expected-error {{constexpr variable 'e' must be initialized by a constant expression}} \
                             expected-note 2 {{pointer to a consteval declaration is not a constant expression}}
+static_assert(r == 42);
 } // namespace P1073R3
diff --git a/clang/test/CXX/expr/expr.const/p8-2a.cpp b/clang/test/CXX/expr/expr.const/p8-2a.cpp
index b8bee64b0aae2..c13b32a165dbd 100644
--- a/clang/test/CXX/expr/expr.const/p8-2a.cpp
+++ b/clang/test/CXX/expr/expr.const/p8-2a.cpp
@@ -2,7 +2,7 @@
 
 // expected-no-diagnostics
 
-namespace P1073R3 {
+namespace P1937R2 {
 struct N {
   constexpr N() {}
   N(N const&) = delete;
@@ -11,12 +11,22 @@ struct N {
 template<typename T> constexpr void bad_assert_copyable() { T t; T t2 = t; }
 using ineffective = decltype(bad_assert_copyable<N>());
 
-// bad_assert_copyable<N> is not needed for constant evaluation
-// (and thus not instantiated)
 template<typename T> consteval void assert_copyable() { T t; T t2 = t; }
+// Prior to P1937R2 consteval functions were evaluated even in otherwise
+// unevaluated context, now this is well-formed.
 using check = decltype(assert_copyable<N>());
-// FIXME: this should give an error because assert_copyable<N> is instantiated
-// (because it is needed for constant evaluation), but the attempt to copy t is
-// ill-formed.
-} // namespace P1073R3
+
+template<typename T>
+__add_rvalue_reference(T) declval();
+
+constexpr auto add1(auto lhs, auto rhs) {
+    return lhs + rhs;
+}
+using T = decltype(add1(declval<int>(), declval<int>()));
+
+consteval auto add2(auto lhs, auto rhs) {
+    return lhs + rhs;
+}
+using T = decltype(add2(declval<int>(), declval<int>()));
+} // namespace P1937R2
 
diff --git a/clang/test/CXX/module/basic/basic.def.odr/p4.cppm b/clang/test/CXX/module/basic/basic.def.odr/p4.cppm
index b8f24d3ed5c8a..1542e532c635a 100644
--- a/clang/test/CXX/module/basic/basic.def.odr/p4.cppm
+++ b/clang/test/CXX/module/basic/basic.def.odr/p4.cppm
@@ -5,9 +5,9 @@
 // RUN: %clang_cc1 -std=c++20 %t/Module.cppm -triple %itanium_abi_triple -emit-llvm -o - | FileCheck %t/Module.cppm --implicit-check-not unused
 //
 // RUN: %clang_cc1 -std=c++20 %t/Module.cppm -triple %itanium_abi_triple -emit-module-interface -o %t/Module.pcm
-// RUN: %clang_cc1 -std=c++20 %t/module.cpp -triple %itanium_abi_triple -fmodule-file=%t/Module.pcm -emit-llvm -o - | FileCheck %t/module.cpp --implicit-check-not=unused --implicit-check-not=global_module
+// RUN: %clang_cc1 -std=c++20 %t/module.cpp -triple %itanium_abi_triple -fmodule-file=Module=%t/Module.pcm -emit-llvm -o - | FileCheck %t/module.cpp --implicit-check-not=unused --implicit-check-not=global_module
 //
-// RUN: %clang_cc1 -std=c++20 %t/user.cpp -triple %itanium_abi_triple -fmodule-file=%t/Module.pcm -emit-llvm -o - | FileCheck %t/user.cpp --implicit-check-not=unused --implicit-check-not=global_module
+// RUN: %clang_cc1 -std=c++20 %t/user.cpp -triple %itanium_abi_triple -fmodule-file=Module=%t/Module.pcm -emit-llvm -o - | FileCheck %t/user.cpp --implicit-check-not=unused --implicit-check-not=global_module
 
 //--- Module.cppm
 // CHECK-DAG: @extern_var_global_module = external {{(dso_local )?}}global
@@ -30,7 +30,7 @@
 // permitted to run the initializer for this variable.
 // CHECK-DAG: @_ZW6Module25inline_var_module_linkage = linkonce_odr {{(dso_local )?}}global
 // CHECK-DAG: @_ZL25static_var_module_linkage = internal
-// CHECK-DAG: @_ZL24const_var_module_linkage = internal
+// CHECK-DAG: @_ZW6Module24const_var_module_linkage = {{(dso_local )?}}constant
 //
 // CHECK-DAG: @_ZW6Module25unused_var_module_linkage = {{(dso_local )?}}global i32 4
 
@@ -129,7 +129,7 @@ void f(a::b, a::c) {}
 // CHECK-DAG: @_ZW6Module25extern_var_module_linkage = external {{(dso_local )?}}global
 // CHECK-DAG: @_ZW6Module25inline_var_module_linkage = linkonce_odr {{(dso_local )?}}global
 // CHECK-DAG: @_ZL25static_var_module_linkage = internal {{(dso_local )?}}global i32 0,
-// CHECK-DAG: @_ZL24const_var_module_linkage = internal {{(dso_local )?}}constant i32 3,
+// CHECK-DAG: @_ZW6Module24const_var_module_linkage = available_externally {{(dso_local )?}}constant i32 3,
 
 module Module;
 
diff --git a/clang/test/CXX/module/basic/basic.link/p3.cppm b/clang/test/CXX/module/basic/basic.link/p3.cppm
index 60505d8ba8b4b..945cfea1cffa0 100644
--- a/clang/test/CXX/module/basic/basic.link/p3.cppm
+++ b/clang/test/CXX/module/basic/basic.link/p3.cppm
@@ -3,7 +3,7 @@
 
 export module M;
 
-// CHECK-NOT: @_ZW1M1a ={{.*}}
+// CHECK: @_ZW1M1a ={{.*}} constant i32 1
 const int a = 1;
 // CHECK: @_ZW1M1b ={{.*}} constant i32 2
 export const int b = 2;
diff --git a/clang/test/CXX/temp/temp.decls/temp.class.spec/temp.class.spec.mfunc/p1-neg.cpp b/clang/test/CXX/temp/temp.decls/temp.class.spec/temp.class.spec.mfunc/p1-neg.cpp
index 59253db3c5297..49f289dbe2d6a 100644
--- a/clang/test/CXX/temp/temp.decls/temp.class.spec/temp.class.spec.mfunc/p1-neg.cpp
+++ b/clang/test/CXX/temp/temp.decls/temp.class.spec/temp.class.spec.mfunc/p1-neg.cpp
@@ -3,7 +3,7 @@
 template<typename T, int N>
 struct A;
 
-template<typename T> // expected-note{{previous template declaration}}
+template<typename T>
 struct A<T*, 2> {
   void f0();
   void f1();
@@ -15,11 +15,10 @@ struct A<int, 1> {
   void g0();
 };
 
-// FIXME: We should probably give more precise diagnostics here, but the
-// diagnostics we give aren't terrible.
-// FIXME: why not point to the first parameter that's "too many"?
-template<typename T, int N> // expected-error{{too many template parameters}}
-void A<T*, 2>::f0() { }
+// FIXME: We should produce diagnostics pointing out the
+// non-matching candidates.
+template<typename T, int N>
+void A<T*, 2>::f0() { } // expected-error{{does not refer into a class, class template or class template partial specialization}}
 
 template<typename T, int N>
 void A<T, N>::f1() { } // expected-error{{out-of-line definition}}
diff --git a/clang/test/CXX/temp/temp.decls/temp.fct/temp.func.order/p6.cpp b/clang/test/CXX/temp/temp.decls/temp.fct/temp.func.order/p6.cpp
index d6e6d73d05ed5..55626edddf20d 100644
--- a/clang/test/CXX/temp/temp.decls/temp.fct/temp.func.order/p6.cpp
+++ b/clang/test/CXX/temp/temp.decls/temp.fct/temp.func.order/p6.cpp
@@ -89,6 +89,10 @@ void f() {
   Y3<int, 1, 1, A{}, S, int> c;
 }
 
+// Per [temp.func.order]p6.2.2, specifically "if the function parameters that
+// positionally correspond between the two templates are not of the same type",
+// this partial specialization does not work.
+// See https://github.com/llvm/llvm-project/issues/58896
 template<C T, C V> struct Y4; // expected-note {{template is declared here}}
 template<D T, C V> struct Y4<V, T>; // expected-error {{class template partial specialization is not more specialized than the primary template}}
 
diff --git a/clang/test/CXX/temp/temp.decls/temp.variadic/p5.cpp b/clang/test/CXX/temp/temp.decls/temp.variadic/p5.cpp
index 2d82a4f26b147..7542186c22c71 100644
--- a/clang/test/CXX/temp/temp.decls/temp.variadic/p5.cpp
+++ b/clang/test/CXX/temp/temp.decls/temp.variadic/p5.cpp
@@ -473,51 +473,45 @@ int fn() {
 }
 }
 
-namespace pr56094 {
-template <typename... T> struct D {
-  template <typename... U> using B = int(int (*...p)(T, U));
-  // expected-error@-1 {{pack expansion contains parameter packs 'T' and 'U' that have different lengths (1 vs. 2)}}
-  template <typename U1, typename U2> D(B<U1, U2> *);
-  // expected-note@-1 {{in instantiation of template type alias 'B' requested here}}
+namespace GH58452 {
+template <typename... As> struct A {
+  template <typename... Bs> using B = void(As...(Bs));
 };
-using t1 = D<float>::B<int>;
-// expected-note@-1 {{in instantiation of template class 'pr56094::D<float>' requested here}}
-
-template <bool...> struct F {};
-template <class...> struct G {};
-template <bool... I> struct E {
-  template <bool... U> using B = G<F<I, U>...>;
-  // expected-error@-1 {{pack expansion contains parameter packs 'I' and 'U' that have different lengths (1 vs. 2)}}
-  template <bool U1, bool U2> E(B<U1, U2> *);
-  // expected-note@-1 {{in instantiation of template type alias 'B' requested here}}
+
+template <typename... Cs> struct C {
+    template <typename... Ds> using D = typename A<Cs...>::template B<Ds...>;
 };
-using t2 = E<true>::B<false>;
-// expected-note@-1 {{in instantiation of template class 'pr56094::E<true>' requested here}}
-} // namespace pr56094
-
-namespace GH56094 {
-#if __cplusplus >= 201402L
-template <class> struct A; // expected-note {{template is declared here}}
-template <class> using B = char;
-template <class ...Cs> int C{ A<B<Cs>>{}... }; // expected-error {{implicit instantiation of undefined template}}
-#endif
-} // namespace GH56094
 
-namespace GH58679 {
-#if __cplusplus >= 201402L
-template <class> constexpr int A = 1;
+using t1 = C<int, int>::template D<float, float>;
 
-template <int> struct B;
-template <> struct B<1> { using b1 = void; };
+template <typename A, typename B>
+using ConditionalRewrite = B;
 
-template <class> using C = char;
+template <typename T>
+using SignatureType = int;
 
-template <class... Ds> int D{ B<A<C<Ds>>>{}... };
+template <typename... Args>
+struct Type1 {
+    template <typename... Params>
+        using Return = SignatureType<int(ConditionalRewrite<Args, Params>...)>;
 
-struct E {
-  template <class E1, class = typename B<A<E1>>::b1> E(E1);
 };
 
-template <typename... Es> int F{ E(C<Es>{})... };
-#endif
-} // namespace GH58679
+template <typename... Args>
+struct Type2 {
+    using T1 = Type1<Args...>;
+
+      template <typename... Params>
+          using Return = typename T1::template Return<Params...>;
+
+};
+
+template <typename T>
+typename T::template Return<int, int> InvokeMethod() {
+    return 3;
+}
+
+int Function1() {
+    return InvokeMethod<Type2<int, int>>();
+}
+}
diff --git a/clang/test/ClangScanDeps/modules-full-output-tu-order.c b/clang/test/ClangScanDeps/modules-full-output-tu-order.c
index eb96d2e99c622..04939826817fc 100644
--- a/clang/test/ClangScanDeps/modules-full-output-tu-order.c
+++ b/clang/test/ClangScanDeps/modules-full-output-tu-order.c
@@ -42,7 +42,7 @@
 // CHECK-NEXT:           "input-file": "[[PREFIX]]/tu.c"
 // CHECK-NEXT:         }
 // CHECK:            ]
-// CHECK-NEXT:     },
+// CHECK:          },
 // CHECK-NEXT:     {
 // CHECK-NEXT:       "commands": [
 // CHECK-NEXT:         {
@@ -57,8 +57,8 @@
 // CHECK-NEXT:             "[[PREFIX]]/tu.c"
 // CHECK-NEXT:           ],
 // CHECK-NEXT:           "input-file": "[[PREFIX]]/tu.c"
-// CHECK-NEXT:         }
+// CHECK:              }
 // CHECK:            ]
-// CHECK-NEXT:     }
-// CHECK-NEXT:   ]
-// CHECK-NEXT: }
+// CHECK:          }
+// CHECK:        ]
+// CHECK:      }
diff --git a/clang/test/ClangScanDeps/response-file.c b/clang/test/ClangScanDeps/response-file.c
new file mode 100644
index 0000000000000..c08105c127202
--- /dev/null
+++ b/clang/test/ClangScanDeps/response-file.c
@@ -0,0 +1,41 @@
+// Check that the scanner can handle a response file input.
+
+// RUN: rm -rf %t
+// RUN: split-file %s %t
+// RUN: sed -e "s|DIR|%/t|g" %t/cdb.json.template > %t/cdb.json
+
+// RUN: clang-scan-deps -format experimental-full -compilation-database %t/cdb.json > %t/deps.json
+
+// RUN: cat %t/deps.json | sed 's:\\\\\?:/:g' | FileCheck -DPREFIX=%/t %s
+
+// CHECK:      "command-line": [
+// CHECK:        "-fsyntax-only"
+// CHECK:        "-x"
+// CHECK-NEXT:   "c"
+// CHECK:        "tu.c"
+// CHECK:        "-I"
+// CHECK-NEXT:   "include"
+// CHECK:      ],
+// CHECK:      "file-deps": [
+// CHECK-NEXT:   "[[PREFIX]]/tu.c"
+// CHECK-NEXT:   "[[PREFIX]]/include/header.h"
+// CHECK-NEXT: ]
+
+//--- cdb.json.template
+[{
+  "file": "DIR/t.c",
+  "directory": "DIR",
+  "command": "clang @DIR/args.txt"
+}]
+
+//--- args.txt
+@args_nested.txt
+-fsyntax-only tu.c
+
+//--- args_nested.txt
+-I include
+
+//--- include/header.h
+
+//--- tu.c
+#include "header.h"
diff --git a/clang/test/CodeGen/Inputs/memprof.exe b/clang/test/CodeGen/Inputs/memprof.exe
index c3b28a8ed9d3d..bde5f3e29a3c2 100755
Binary files a/clang/test/CodeGen/Inputs/memprof.exe and b/clang/test/CodeGen/Inputs/memprof.exe differ
diff --git a/clang/test/CodeGen/Inputs/memprof.memprofraw b/clang/test/CodeGen/Inputs/memprof.memprofraw
index 7ae89bff648af..a7aa405a9c3a8 100644
Binary files a/clang/test/CodeGen/Inputs/memprof.memprofraw and b/clang/test/CodeGen/Inputs/memprof.memprofraw differ
diff --git a/clang/test/CodeGen/Inputs/update_memprof_inputs.sh b/clang/test/CodeGen/Inputs/update_memprof_inputs.sh
new file mode 100755
index 0000000000000..1d51419ed1e7f
--- /dev/null
+++ b/clang/test/CodeGen/Inputs/update_memprof_inputs.sh
@@ -0,0 +1,17 @@
+#!/bin/bash
+
+if [ -z $1 ]; then
+  echo "Path to clang required!"
+  echo "Usage: update_memprof_inputs.sh /path/to/updated/clang"
+  exit 1
+else
+  CLANG=$1
+fi
+
+# Allows the script to be invoked from other directories.
+OUTDIR=$(dirname $(realpath -s $0))
+
+DEFAULT_MEMPROF_FLAGS="-fuse-ld=lld -Wl,--no-rosegment -gmlt -fdebug-info-for-profiling -fmemory-profile -mno-omit-leaf-frame-pointer -fno-omit-frame-pointer -fno-optimize-sibling-calls -m64 -Wl,-build-id -no-pie"
+
+${CLANG} ${DEFAULT_MEMPROF_FLAGS} ${OUTDIR}/../memprof.cpp -o ${OUTDIR}/memprof.exe
+env MEMPROF_OPTIONS=log_path=stdout ${OUTDIR}/memprof.exe > ${OUTDIR}/memprof.memprofraw
diff --git a/clang/test/CodeGen/PowerPC/aix-altivec-vaargs.c b/clang/test/CodeGen/PowerPC/aix-altivec-vaargs.c
index 657d83454b0fc..03182423a422c 100644
--- a/clang/test/CodeGen/PowerPC/aix-altivec-vaargs.c
+++ b/clang/test/CodeGen/PowerPC/aix-altivec-vaargs.c
@@ -21,26 +21,22 @@ vector double vector_varargs(int count, ...) {
 
 // AIX32:       for.body:
 // AIX32-NEXT:    %argp.cur = load ptr, ptr %arg_list, align 4
-// AIX32-NEXT:    %2 = ptrtoint ptr %argp.cur to i32
-// AIX32-NEXT:    %3 = add i32 %2, 15
-// AIX32-NEXT:    %4 = and i32 %3, -16
-// AIX32-NEXT:    %argp.cur.aligned = inttoptr i32 %4 to ptr
+// AIX32-NEXT:    %2 = getelementptr inbounds i8, ptr %argp.cur, i32 15
+// AIX32-NEXT:    %argp.cur.aligned = call ptr @llvm.ptrmask.p0.i32(ptr %2, i32 -16)
 // AIX32-NEXT:    %argp.next = getelementptr inbounds i8, ptr %argp.cur.aligned, i32 16
 // AIX32-NEXT:    store ptr %argp.next, ptr %arg_list, align 4
-// AIX32-NEXT:    %5 = load <2 x double>, ptr %argp.cur.aligned, align 16
-// AIX32-NEXT:    store <2 x double> %5, ptr %ret, align 16
+// AIX32-NEXT:    %3 = load <2 x double>, ptr %argp.cur.aligned, align 16
+// AIX32-NEXT:    store <2 x double> %3, ptr %ret, align 16
 // AIX32-NEXT:    br label %for.inc
 
 // AIX64:       for.body:
 // AIX64-NEXT:    %argp.cur = load ptr, ptr %arg_list, align 8
-// AIX64-NEXT:    %2 = ptrtoint ptr %argp.cur to i64
-// AIX64-NEXT:    %3 = add i64 %2, 15
-// AIX64-NEXT:    %4 = and i64 %3, -16
-// AIX64-NEXT:    %argp.cur.aligned = inttoptr i64 %4 to ptr
+// AIX64-NEXT:    %2 = getelementptr inbounds i8, ptr %argp.cur, i32 15
+// AIX64-NEXT:    %argp.cur.aligned = call ptr @llvm.ptrmask.p0.i64(ptr %2, i64 -16)
 // AIX64-NEXT:    %argp.next = getelementptr inbounds i8, ptr %argp.cur.aligned, i64 16
 // AIX64-NEXT:    store ptr %argp.next, ptr %arg_list, align 8
-// AIX64-NEXT:    %5 = load <2 x double>, ptr %argp.cur.aligned, align 16
-// AIX64-NEXT:    store <2 x double> %5, ptr %ret, align 16
+// AIX64-NEXT:    %3 = load <2 x double>, ptr %argp.cur.aligned, align 16
+// AIX64-NEXT:    store <2 x double> %3, ptr %ret, align 16
 // AIX64-NEXT:    br label %for.inc
 
 
diff --git a/clang/test/CodeGen/PowerPC/ppc-varargs-struct.c b/clang/test/CodeGen/PowerPC/ppc-varargs-struct.c
index 6b69975870835..c8d54dd8e418f 100644
--- a/clang/test/CodeGen/PowerPC/ppc-varargs-struct.c
+++ b/clang/test/CodeGen/PowerPC/ppc-varargs-struct.c
@@ -37,10 +37,8 @@ void testva (int n, ...)
 // CHECK-PPC-NEXT:  store i8 8, ptr [[GPRPTR]], align 4
 // CHECK-PPC-NEXT:  [[OVERFLOW_AREA_P:%[0-9]+]] = getelementptr inbounds %struct.__va_list_tag, ptr [[ARRAYDECAY]], i32 0, i32 3
 // CHECK-PPC-NEXT:  [[OVERFLOW_AREA:%.+]] = load ptr, ptr [[OVERFLOW_AREA_P]], align 4
-// CHECK-PPC-NEXT:  %{{[0-9]+}} =  ptrtoint ptr %argp.cur to i32
-// CHECK-PPC-NEXT:  %{{[0-9]+}} = add i32 %{{[0-9]+}}, 7
-// CHECK-PPC-NEXT:  %{{[0-9]+}} = and i32 %{{[0-9]+}}, -8
-// CHECK-PPC-NEXT:  %argp.cur.aligned = inttoptr i32 %{{[0-9]+}} to ptr
+// CHECK-PPC-NEXT: [[GEP_ALIGN:%[0-9]+]] = getelementptr inbounds i8, ptr %argp.cur, i32 7
+// CHECK-PPC-NEXT: %argp.cur.aligned = call ptr @llvm.ptrmask.p0.i32(ptr [[GEP_ALIGN]], i32 -8)
 // CHECK-PPC-NEXT:  [[NEW_OVERFLOW_AREA:%[0-9]+]] = getelementptr inbounds i8, ptr %argp.cur.aligned, i32 4
 // CHECK-PPC-NEXT:  store ptr [[NEW_OVERFLOW_AREA:%[0-9]+]], ptr [[OVERFLOW_AREA_P]], align 4
 // CHECK-PPC-NEXT:  br label %[[CONT]]
@@ -51,7 +49,7 @@ void testva (int n, ...)
 // CHECK-PPC-NEXT:  call void @llvm.memcpy.p0.p0.i32(ptr align 8 %t, ptr align 8 [[AGGR]], i32 16, i1 false)
 
   int v = va_arg (ap, int);
-  
+
 // CHECK: getelementptr inbounds i8, ptr %{{[a-z.0-9]*}}, i64 4
 // CHECK-PPC:       [[ARRAYDECAY:%[a-z0-9]+]] = getelementptr inbounds [1 x %struct.__va_list_tag], ptr %ap, i32 0, i32 0
 // CHECK-PPC-NEXT:  [[GPRPTR:%.+]] = getelementptr inbounds %struct.__va_list_tag, ptr [[ARRAYDECAY]], i32 0, i32 0
diff --git a/clang/test/CodeGen/PowerPC/ppc64-align-struct.c b/clang/test/CodeGen/PowerPC/ppc64-align-struct.c
index b0d14546566cb..2476c7149d076 100644
--- a/clang/test/CodeGen/PowerPC/ppc64-align-struct.c
+++ b/clang/test/CodeGen/PowerPC/ppc64-align-struct.c
@@ -77,10 +77,8 @@ struct test1 test1va (int x, ...)
 
 // CHECK: define{{.*}} void @test2va(ptr noalias sret(%struct.test2) align 16 %[[AGG_RESULT:.*]], i32 noundef signext %x, ...)
 // CHECK: %[[CUR:[^ ]+]] = load ptr, ptr %ap
-// CHECK: %[[TMP0:[^ ]+]] = ptrtoint ptr %[[CUR]] to i64
-// CHECK: %[[TMP1:[^ ]+]] = add i64 %[[TMP0]], 15
-// CHECK: %[[TMP2:[^ ]+]] = and i64 %[[TMP1]], -16
-// CHECK: %[[ALIGN:[^ ]+]] = inttoptr i64 %[[TMP2]] to ptr
+// CHECK: %[[TMP0:[^ ]+]] = getelementptr inbounds i8, ptr %[[CUR]], i32 15
+// CHECK: %[[ALIGN:[^ ]+]] = call ptr @llvm.ptrmask.p0.i64(ptr %[[TMP0]], i64 -16)
 // CHECK: %[[NEXT:[^ ]+]] = getelementptr inbounds i8, ptr %[[ALIGN]], i64 16
 // CHECK: store ptr %[[NEXT]], ptr %ap
 // CHECK: call void @llvm.memcpy.p0.p0.i64(ptr align 16 %[[AGG_RESULT]], ptr align 16 %[[ALIGN]], i64 16, i1 false)
@@ -96,10 +94,8 @@ struct test2 test2va (int x, ...)
 
 // CHECK: define{{.*}} void @test3va(ptr noalias sret(%struct.test3) align 32 %[[AGG_RESULT:.*]], i32 noundef signext %x, ...)
 // CHECK: %[[CUR:[^ ]+]] = load ptr, ptr %ap
-// CHECK: %[[TMP0:[^ ]+]] = ptrtoint ptr %[[CUR]] to i64
-// CHECK: %[[TMP1:[^ ]+]] = add i64 %[[TMP0]], 15
-// CHECK: %[[TMP2:[^ ]+]] = and i64 %[[TMP1]], -16
-// CHECK: %[[ALIGN:[^ ]+]] = inttoptr i64 %[[TMP2]] to ptr
+// CHECK: %[[TMP0:[^ ]+]] = getelementptr inbounds i8, ptr %[[CUR]], i32 15
+// CHECK: %[[ALIGN:[^ ]+]] = call ptr @llvm.ptrmask.p0.i64(ptr %[[TMP0]], i64 -16)
 // CHECK: %[[NEXT:[^ ]+]] = getelementptr inbounds i8, ptr %[[ALIGN]], i64 32
 // CHECK: store ptr %[[NEXT]], ptr %ap
 // CHECK: call void @llvm.memcpy.p0.p0.i64(ptr align 32 %[[AGG_RESULT]], ptr align 16 %[[ALIGN]], i64 32, i1 false)
@@ -178,10 +174,8 @@ struct test_longdouble testva_longdouble (int x, ...)
 
 // CHECK: define{{.*}} void @testva_vector(ptr noalias sret(%struct.test_vector) align 16 %[[AGG_RESULT:.*]], i32 noundef signext %x, ...)
 // CHECK: %[[CUR:[^ ]+]] = load ptr, ptr %ap
-// CHECK: %[[TMP0:[^ ]+]] = ptrtoint ptr %[[CUR]] to i64
-// CHECK: %[[TMP1:[^ ]+]] = add i64 %[[TMP0]], 15
-// CHECK: %[[TMP2:[^ ]+]] = and i64 %[[TMP1]], -16
-// CHECK: %[[ALIGN:[^ ]+]] = inttoptr i64 %[[TMP2]] to ptr
+// CHECK: %[[TMP0:[^ ]+]] = getelementptr inbounds i8, ptr %[[CUR]], i32 15
+// CHECK: %[[ALIGN:[^ ]+]] = call ptr @llvm.ptrmask.p0.i64(ptr %[[TMP0]], i64 -16)
 // CHECK: %[[NEXT:[^ ]+]] = getelementptr inbounds i8, ptr %[[ALIGN]], i64 16
 // CHECK: store ptr %[[NEXT]], ptr %ap
 // CHECK: call void @llvm.memcpy.p0.p0.i64(ptr align 16 %[[AGG_RESULT]], ptr align 16 %[[ALIGN]], i64 16, i1 false)
diff --git a/clang/test/CodeGen/PowerPC/ppc64le-varargs-f128.c b/clang/test/CodeGen/PowerPC/ppc64le-varargs-f128.c
index 50b3c0451bd80..53cb0dbd1d610 100644
--- a/clang/test/CodeGen/PowerPC/ppc64le-varargs-f128.c
+++ b/clang/test/CodeGen/PowerPC/ppc64le-varargs-f128.c
@@ -33,10 +33,8 @@ void foo_ls(ldbl128_s);
 // OMP-HOST-LABEL: define{{.*}} void @omp(
 // OMP-HOST: call void @llvm.va_start(ptr %[[AP:[0-9a-zA-Z_.]+]])
 // OMP-HOST: %[[CUR:[0-9a-zA-Z_.]+]] = load ptr, ptr %[[AP]], align 8
-// OMP-HOST: %[[V0:[0-9a-zA-Z_.]+]] = ptrtoint ptr %[[CUR]] to i64
-// OMP-HOST: %[[V1:[0-9a-zA-Z_.]+]] = add i64 %[[V0]], 15
-// OMP-HOST: %[[V2:[0-9a-zA-Z_.]+]] = and i64 %[[V1]], -16
-// OMP-HOST: %[[ALIGN:[0-9a-zA-Z_.]+]] = inttoptr i64 %[[V2]] to ptr
+// OMP-HOST: %[[TMP0:[^ ]+]] = getelementptr inbounds i8, ptr %[[CUR]], i32 15
+// OMP-HOST: %[[ALIGN:[^ ]+]] = call ptr @llvm.ptrmask.p0.i64(ptr %[[TMP0]], i64 -16)
 // OMP-HOST: %[[V4:[0-9a-zA-Z_.]+]] = load fp128, ptr %[[ALIGN]], align 16
 // OMP-HOST: call void @foo_ld(fp128 noundef %[[V4]])
 void omp(int n, ...) {
@@ -53,10 +51,8 @@ void omp(int n, ...) {
 // IEEE-LABEL: define{{.*}} void @f128
 // IEEE: call void @llvm.va_start(ptr %[[AP:[0-9a-zA-Z_.]+]])
 // IEEE: %[[CUR:[0-9a-zA-Z_.]+]] = load ptr, ptr %[[AP]]
-// IEEE: %[[V0:[0-9a-zA-Z_.]+]] = ptrtoint ptr %[[CUR]] to i64
-// IEEE: %[[V1:[0-9a-zA-Z_.]+]] = add i64 %[[V0]], 15
-// IEEE: %[[V2:[0-9a-zA-Z_.]+]] = and i64 %[[V1]], -16
-// IEEE: %[[ALIGN:[0-9a-zA-Z_.]+]] = inttoptr i64 %[[V2]] to ptr
+// IEEE: %[[TMP0:[^ ]+]] = getelementptr inbounds i8, ptr %[[CUR]], i32 15
+// IEEE: %[[ALIGN:[^ ]+]] = call ptr @llvm.ptrmask.p0.i64(ptr %[[TMP0]], i64 -16)
 // IEEE: %[[V4:[0-9a-zA-Z_.]+]] = load fp128, ptr %[[ALIGN]], align 16
 // IEEE: call void @foo_fq(fp128 noundef %[[V4]])
 // IEEE: call void @llvm.va_end(ptr %[[AP]])
@@ -70,10 +66,8 @@ void f128(int n, ...) {
 // IEEE-LABEL: define{{.*}} void @long_double
 // IEEE: call void @llvm.va_start(ptr %[[AP:[0-9a-zA-Z_.]+]])
 // IEEE: %[[CUR:[0-9a-zA-Z_.]+]] = load ptr, ptr %[[AP]]
-// IEEE: %[[V0:[0-9a-zA-Z_.]+]] = ptrtoint ptr %[[CUR]] to i64
-// IEEE: %[[V1:[0-9a-zA-Z_.]+]] = add i64 %[[V0]], 15
-// IEEE: %[[V2:[0-9a-zA-Z_.]+]] = and i64 %[[V1]], -16
-// IEEE: %[[ALIGN:[0-9a-zA-Z_.]+]] = inttoptr i64 %[[V2]] to ptr
+// IEEE: %[[TMP0:[^ ]+]] = getelementptr inbounds i8, ptr %[[CUR]], i32 15
+// IEEE: %[[ALIGN:[^ ]+]] = call ptr @llvm.ptrmask.p0.i64(ptr %[[TMP0]], i64 -16)
 // IEEE: %[[V4:[0-9a-zA-Z_.]+]] = load fp128, ptr %[[ALIGN]], align 16
 // IEEE: call void @foo_ld(fp128 noundef %[[V4]])
 // IEEE: call void @llvm.va_end(ptr %[[AP]])
@@ -94,10 +88,8 @@ void long_double(int n, ...) {
 // IEEE-LABEL: define{{.*}} void @long_double_struct
 // IEEE: call void @llvm.va_start(ptr %[[AP:[0-9a-zA-Z_.]+]])
 // IEEE: %[[CUR:[0-9a-zA-Z_.]+]] = load ptr, ptr %[[AP]]
-// IEEE: %[[P0:[0-9a-zA-Z_.]+]] = ptrtoint ptr %[[CUR]] to i64
-// IEEE: %[[P1:[0-9a-zA-Z_.]+]] = add i64 %[[P0]], 15
-// IEEE: %[[P2:[0-9a-zA-Z_.]+]] = and i64 %[[P1]], -16
-// IEEE: %[[ALIGN:[0-9a-zA-Z_.]+]] = inttoptr i64 %[[P2]] to ptr
+// IEEE: %[[TMP0:[^ ]+]] = getelementptr inbounds i8, ptr %[[CUR]], i32 15
+// IEEE: %[[ALIGN:[^ ]+]] = call ptr @llvm.ptrmask.p0.i64(ptr %[[TMP0]], i64 -16)
 // IEEE: %[[V0:[0-9a-zA-Z_.]+]] = getelementptr inbounds i8, ptr %[[ALIGN]], i64 16
 // IEEE: store ptr %[[V0]], ptr %[[AP]], align 8
 // IEEE: call void @llvm.memcpy.p0.p0.i64(ptr align 16 %[[TMP:[0-9a-zA-Z_.]+]], ptr align 16 %[[ALIGN]], i64 16, i1 false)
diff --git a/clang/test/CodeGen/RISCV/riscv-abi.cpp b/clang/test/CodeGen/RISCV/riscv-abi.cpp
index 7e34ffef14d30..b92256206ece6 100644
--- a/clang/test/CodeGen/RISCV/riscv-abi.cpp
+++ b/clang/test/CodeGen/RISCV/riscv-abi.cpp
@@ -1,16 +1,17 @@
+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --function-signature --filter "^define |^entry:" --version 2
 // RUN: %clang_cc1 -triple riscv32 -emit-llvm %s -o - \
-// RUN:   | FileCheck -check-prefixes=ILP32,ILP32-ILP32F,ILP32-ILP32F-ILP32D %s
+// RUN:   | FileCheck -check-prefixes=ILP32-ILP32F-ILP32D,ILP32-ILP32F,ILP32 %s
 // RUN: %clang_cc1 -triple riscv32 -target-feature +f -target-abi ilp32f -emit-llvm %s -o - \
-// RUN:   | FileCheck -check-prefixes=ILP32F,ILP32-ILP32F,ILP32F-ILP32D,ILP32-ILP32F-ILP32D %s
+// RUN:   | FileCheck -check-prefixes=ILP32-ILP32F-ILP32D,ILP32F-ILP32D,ILP32-ILP32F,ILP32F %s
 // RUN: %clang_cc1 -triple riscv32 -target-feature +f -target-feature +d -target-abi ilp32d -emit-llvm %s -o - \
-// RUN:   | FileCheck -check-prefixes=ILP32D,ILP32F-ILP32D,ILP32-ILP32F-ILP32D %s
+// RUN:   | FileCheck -check-prefixes=ILP32-ILP32F-ILP32D,ILP32F-ILP32D,ILP32D %s
 
 // RUN: %clang_cc1 -triple riscv64 -emit-llvm %s -o - \
-// RUN:   | FileCheck -check-prefixes=LP64,LP64-LP64F,LP64-LP64F-LP64D %s
+// RUN:   | FileCheck -check-prefixes=LP64-LP64F-LP64D,LP64-LP64F,LP64 %s
 // RUN: %clang_cc1 -triple riscv64 -target-feature +f -target-abi lp64f -emit-llvm %s -o - \
-// RUN:   | FileCheck -check-prefixes=LP64F,LP64-LP64F,LP64F-LP64D,LP64-LP64F-LP64D %s
+// RUN:   | FileCheck -check-prefixes=LP64-LP64F-LP64D,LP64F-LP64D,LP64-LP64F,LP64F %s
 // RUN: %clang_cc1 -triple riscv64 -target-feature +f -target-feature +d -target-abi lp64d -emit-llvm %s -o - \
-// RUN:   | FileCheck -check-prefixes=LP64D,LP64F-LP64D,LP64-LP64F-LP64D %s
+// RUN:   | FileCheck -check-prefixes=LP64-LP64F-LP64D,LP64F-LP64D,LP64D %s
 
 #include <stdint.h>
 
@@ -25,8 +26,14 @@ struct child1_int32_s : parent1_int32_s {
   int32_t i2;
 };
 
-// ILP32-ILP32F-ILP32D-LABEL: define{{.*}} [2 x i32] @_Z30int32_int32_struct_inheritance14child1_int32_s([2 x i32] %a.coerce)
-// LP64-LP64F-LP64D-LABEL: define{{.*}} i64 @_Z30int32_int32_struct_inheritance14child1_int32_s(i64 %a.coerce)
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local [2 x i32] @_Z30int32_int32_struct_inheritance14child1_int32_s
+// ILP32-ILP32F-ILP32D-SAME: ([2 x i32] [[A_COERCE:%.*]]) #[[ATTR0:[0-9]+]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @_Z30int32_int32_struct_inheritance14child1_int32_s
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0:[0-9]+]] {
+// LP64-LP64F-LP64D:  entry:
+//
 struct child1_int32_s int32_int32_struct_inheritance(struct child1_int32_s a) {
   return a;
 }
@@ -39,10 +46,22 @@ struct child2_float_s : parent2_int32_s {
   float f1;
 };
 
-// ILP32: define{{.*}} [2 x i32] @_Z30int32_float_struct_inheritance14child2_float_s([2 x i32] %a.coerce)
-// ILP32F-ILP32D: define{{.*}} { i32, float } @_Z30int32_float_struct_inheritance14child2_float_s(i32 %0, float %1)
-// LP64: define{{.*}} i64 @_Z30int32_float_struct_inheritance14child2_float_s(i64 %a.coerce)
-// LP64F-LP64D: define{{.*}} { i32, float } @_Z30int32_float_struct_inheritance14child2_float_s(i32 %0, float %1)
+// ILP32-LABEL: define dso_local [2 x i32] @_Z30int32_float_struct_inheritance14child2_float_s
+// ILP32-SAME: ([2 x i32] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local { i32, float } @_Z30int32_float_struct_inheritance14child2_float_s
+// ILP32F-ILP32D-SAME: (i32 [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+// LP64-LABEL: define dso_local i64 @_Z30int32_float_struct_inheritance14child2_float_s
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local { i32, float } @_Z30int32_float_struct_inheritance14child2_float_s
+// LP64F-LP64D-SAME: (i32 [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
 struct child2_float_s int32_float_struct_inheritance(struct child2_float_s a) {
   return a;
 }
@@ -55,9 +74,18 @@ struct child3_int64_s : parent3_float_s {
   int64_t i1;
 };
 
-// ILP32-ILP32F-ILP32D-LABEL: define{{.*}} void @_Z30float_int64_struct_inheritance14child3_int64_s(ptr noalias sret(%struct.child3_int64_s)
-// LP64-LABEL: define{{.*}} [2 x i64] @_Z30float_int64_struct_inheritance14child3_int64_s([2 x i64] %a.coerce)
-// LP64F-LP64D-LABEL: define{{.*}} { float, i64 } @_Z30float_int64_struct_inheritance14child3_int64_s(float %0, i64 %1)
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @_Z30float_int64_struct_inheritance14child3_int64_s
+// ILP32-ILP32F-ILP32D-SAME: (ptr noalias sret([[STRUCT_CHILD3_INT64_S:%.*]]) align 8 [[AGG_RESULT:%.*]], ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+// LP64-LABEL: define dso_local [2 x i64] @_Z30float_int64_struct_inheritance14child3_int64_s
+// LP64-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local { float, i64 } @_Z30float_int64_struct_inheritance14child3_int64_s
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]], i64 [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
 struct child3_int64_s float_int64_struct_inheritance(struct child3_int64_s a) {
   return a;
 }
@@ -70,10 +98,22 @@ struct child4_double_s : parent4_double_s {
   double d1;
 };
 
-// ILP32-ILP32F-LABEL: define{{.*}} void @_Z32double_double_struct_inheritance15child4_double_s(ptr noalias sret(%struct.child4_double_s)
-// ILP32D-LABEL: define{{.*}} { double, double } @_Z32double_double_struct_inheritance15child4_double_s(double %0, double %1)
-// LP64-LP64F-LABEL: define{{.*}} [2 x i64] @_Z32double_double_struct_inheritance15child4_double_s([2 x i64] %a.coerce)
-// LP64D-LABEL: define{{.*}} { double, double } @_Z32double_double_struct_inheritance15child4_double_s(double %0, double %1)
+// ILP32-ILP32F-LABEL: define dso_local void @_Z32double_double_struct_inheritance15child4_double_s
+// ILP32-ILP32F-SAME: (ptr noalias sret([[STRUCT_CHILD4_DOUBLE_S:%.*]]) align 8 [[AGG_RESULT:%.*]], ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local { double, double } @_Z32double_double_struct_inheritance15child4_double_s
+// ILP32D-SAME: (double [[TMP0:%.*]], double [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+// LP64-LP64F-LABEL: define dso_local [2 x i64] @_Z32double_double_struct_inheritance15child4_double_s
+// LP64-LP64F-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local { double, double } @_Z32double_double_struct_inheritance15child4_double_s
+// LP64D-SAME: (double [[TMP0:%.*]], double [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
 struct child4_double_s double_double_struct_inheritance(struct child4_double_s a) {
   return a;
 }
@@ -89,8 +129,14 @@ struct child5_virtual_s : virtual parent5_virtual_s {
   float f1;
 };
 
-// ILP32-ILP32F-ILP32D-LABEL: define{{.*}} void @_ZN16child5_virtual_sC1EOS_(ptr noundef nonnull align 4 dereferenceable(8) %this, ptr noundef nonnull align 4 dereferenceable(8) %0)
-// LP64-LP64F-LP64D-LABEL: define{{.*}} void @_ZN16child5_virtual_sC1EOS_(ptr noundef nonnull align 8 dereferenceable(12) %this, ptr noundef nonnull align 8 dereferenceable(12) %0)
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @_Z38int32_float_virtual_struct_inheritance16child5_virtual_s
+// ILP32-ILP32F-ILP32D-SAME: (ptr noalias sret([[STRUCT_CHILD5_VIRTUAL_S:%.*]]) align 4 [[AGG_RESULT:%.*]], ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+// LP64-LP64F-LP64D-LABEL: define dso_local void @_Z38int32_float_virtual_struct_inheritance16child5_virtual_s
+// LP64-LP64F-LP64D-SAME: (ptr noalias sret([[STRUCT_CHILD5_VIRTUAL_S:%.*]]) align 8 [[AGG_RESULT:%.*]], ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
 struct child5_virtual_s int32_float_virtual_struct_inheritance(struct child5_virtual_s a) {
   return a;
 }
@@ -110,14 +156,28 @@ struct child6b_s : parent6_float_s {
 struct grandchild_6_s : child6a_s, child6b_s {
 };
 
-// ILP32: define{{.*}} [2 x i32] @_Z38float_float_diamond_struct_inheritance14grandchild_6_s([2 x i32] %a.coerce)
 // ILP32F-ILP64D: define{{.*}} { float, float } @_Z38float_float_diamond_struct_inheritance14grandchild_6_s(float %0, float %1)
-// LP64: define{{.*}} i64 @_Z38float_float_diamond_struct_inheritance14grandchild_6_s(i64 %a.coerce)
-// LP64F-LP64D: define{{.*}} { float, float } @_Z38float_float_diamond_struct_inheritance14grandchild_6_s(float %0, float %1)
+// ILP32-LABEL: define dso_local [2 x i32] @_Z38float_float_diamond_struct_inheritance14grandchild_6_s
+// ILP32-SAME: ([2 x i32] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local { float, float } @_Z38float_float_diamond_struct_inheritance14grandchild_6_s
+// ILP32F-ILP32D-SAME: (float [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+// LP64-LABEL: define dso_local i64 @_Z38float_float_diamond_struct_inheritance14grandchild_6_s
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local { float, float } @_Z38float_float_diamond_struct_inheritance14grandchild_6_s
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
 struct grandchild_6_s float_float_diamond_struct_inheritance(struct grandchild_6_s a) {
   return a;
 }
 
 // NOTE: These prefixes are unused. Do not add tests below this line:
+//// NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
 // ILP32F: {{.*}}
 // LP64F: {{.*}}
diff --git a/clang/test/CodeGen/RISCV/riscv32-abi.c b/clang/test/CodeGen/RISCV/riscv32-abi.c
new file mode 100644
index 0000000000000..5cabbf4d90a1c
--- /dev/null
+++ b/clang/test/CodeGen/RISCV/riscv32-abi.c
@@ -0,0 +1,1950 @@
+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --filter "^define |^entry:" --version 2
+// RUN: %clang_cc1 -triple riscv32 -emit-llvm %s -o - \
+// RUN:   | FileCheck -check-prefixes=ILP32-ILP32F-ILP32D,ILP32-ILP32F,ILP32 %s
+// RUN: %clang_cc1 -triple riscv32 -target-feature +f -target-abi ilp32f -emit-llvm %s -o - \
+// RUN:   | FileCheck -check-prefixes=ILP32-ILP32F-ILP32D,ILP32F-ILP32D,ILP32-ILP32F,ILP32F %s
+// RUN: %clang_cc1 -triple riscv32 -target-feature +f -target-feature +d -target-abi ilp32d -emit-llvm %s -o - \
+// RUN:   | FileCheck -check-prefixes=ILP32-ILP32F-ILP32D,ILP32F-ILP32D,ILP32D %s
+
+#include <stddef.h>
+#include <stdint.h>
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_void
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0:[0-9]+]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_void(void) {}
+
+// Scalar arguments and return values smaller than the word size are extended
+// according to the sign of their type, up to 32 bits
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local zeroext i1 @f_scalar_0
+// ILP32-ILP32F-ILP32D-SAME: (i1 noundef zeroext [[X:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+_Bool f_scalar_0(_Bool x) { return x; }
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local signext i8 @f_scalar_1
+// ILP32-ILP32F-ILP32D-SAME: (i8 noundef signext [[X:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+int8_t f_scalar_1(int8_t x) { return x; }
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local zeroext i8 @f_scalar_2
+// ILP32-ILP32F-ILP32D-SAME: (i8 noundef zeroext [[X:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+uint8_t f_scalar_2(uint8_t x) { return x; }
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_scalar_3
+// ILP32-ILP32F-ILP32D-SAME: (i32 noundef [[X:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+int32_t f_scalar_3(int32_t x) { return x; }
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i64 @f_scalar_4
+// ILP32-ILP32F-ILP32D-SAME: (i64 noundef [[X:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+int64_t f_scalar_4(int64_t x) { return x; }
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local float @f_fp_scalar_1
+// ILP32-ILP32F-ILP32D-SAME: (float noundef [[X:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+float f_fp_scalar_1(float x) { return x; }
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local double @f_fp_scalar_2
+// ILP32-ILP32F-ILP32D-SAME: (double noundef [[X:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+double f_fp_scalar_2(double x) { return x; }
+
+// Scalars larger than 2*xlen are passed/returned indirect. However, the
+// RISC-V LLVM backend can handle this fine, so the function doesn't need to
+// be modified.
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local fp128 @f_fp_scalar_3
+// ILP32-ILP32F-ILP32D-SAME: (fp128 noundef [[X:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+long double f_fp_scalar_3(long double x) { return x; }
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local half @f_fp_scalar_4
+// ILP32-ILP32F-ILP32D-SAME: (half noundef [[X:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+_Float16 f_fp_scalar_4(_Float16 x) { return x; }
+
+// Empty structs or unions are ignored.
+
+struct empty_s {};
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_agg_empty_struct
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct empty_s f_agg_empty_struct(struct empty_s x) {
+  return x;
+}
+
+union empty_u {};
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_agg_empty_union
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+union empty_u f_agg_empty_union(union empty_u x) {
+  return x;
+}
+
+// Aggregates <= 2*xlen may be passed in registers, so will be coerced to
+// integer arguments. The rules for return are the same.
+
+struct tiny {
+  uint8_t a, b, c, d;
+};
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_agg_tiny
+// ILP32-ILP32F-ILP32D-SAME: (i32 [[X_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_agg_tiny(struct tiny x) {
+  x.a += x.b;
+  x.c += x.d;
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_agg_tiny_ret
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct tiny f_agg_tiny_ret(void) {
+  return (struct tiny){1, 2, 3, 4};
+}
+
+typedef uint8_t v4i8 __attribute__((vector_size(4)));
+typedef int32_t v1i32 __attribute__((vector_size(4)));
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_vec_tiny_v4i8
+// ILP32-ILP32F-ILP32D-SAME: (i32 noundef [[X_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_vec_tiny_v4i8(v4i8 x) {
+  x[0] = x[1];
+  x[2] = x[3];
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_vec_tiny_v4i8_ret
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+v4i8 f_vec_tiny_v4i8_ret(void) {
+  return (v4i8){1, 2, 3, 4};
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_vec_tiny_v1i32
+// ILP32-ILP32F-ILP32D-SAME: (i32 noundef [[X_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_vec_tiny_v1i32(v1i32 x) {
+  x[0] = 114;
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_vec_tiny_v1i32_ret
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+v1i32 f_vec_tiny_v1i32_ret(void) {
+  return (v1i32){1};
+}
+
+struct small {
+  int32_t a, *b;
+};
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_agg_small
+// ILP32-ILP32F-ILP32D-SAME: ([2 x i32] [[X_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_agg_small(struct small x) {
+  x.a += *x.b;
+  x.b = &x.a;
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local [2 x i32] @f_agg_small_ret
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct small f_agg_small_ret(void) {
+  return (struct small){1, 0};
+}
+
+typedef uint8_t v8i8 __attribute__((vector_size(8)));
+typedef int64_t v1i64 __attribute__((vector_size(8)));
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_vec_small_v8i8
+// ILP32-ILP32F-ILP32D-SAME: (i64 noundef [[X_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_vec_small_v8i8(v8i8 x) {
+  x[0] = x[7];
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i64 @f_vec_small_v8i8_ret
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+v8i8 f_vec_small_v8i8_ret(void) {
+  return (v8i8){1, 2, 3, 4, 5, 6, 7, 8};
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_vec_small_v1i64
+// ILP32-ILP32F-ILP32D-SAME: (i64 noundef [[X_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_vec_small_v1i64(v1i64 x) {
+  x[0] = 114;
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i64 @f_vec_small_v1i64_ret
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+v1i64 f_vec_small_v1i64_ret(void) {
+  return (v1i64){1};
+}
+
+// Aggregates of 2*xlen size and 2*xlen alignment should be coerced to a
+// single 2*xlen-sized argument, to ensure that alignment can be maintained if
+// passed on the stack.
+
+struct small_aligned {
+  int64_t a;
+};
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_agg_small_aligned
+// ILP32-ILP32F-ILP32D-SAME: (i64 [[X_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_agg_small_aligned(struct small_aligned x) {
+  x.a += x.a;
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i64 @f_agg_small_aligned_ret
+// ILP32-ILP32F-ILP32D-SAME: (i64 [[X_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct small_aligned f_agg_small_aligned_ret(struct small_aligned x) {
+  return (struct small_aligned){10};
+}
+
+// Aggregates greater > 2*xlen will be passed and returned indirectly
+struct large {
+  int32_t a, b, c, d;
+};
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_agg_large
+// ILP32-ILP32F-ILP32D-SAME: (ptr noundef [[X:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_agg_large(struct large x) {
+  x.a = x.b + x.c + x.d;
+}
+
+// The address where the struct should be written to will be the first
+// argument
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_agg_large_ret
+// ILP32-ILP32F-ILP32D-SAME: (ptr noalias sret([[STRUCT_LARGE:%.*]]) align 4 [[AGG_RESULT:%.*]], i32 noundef [[I:%.*]], i8 noundef signext [[J:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct large f_agg_large_ret(int32_t i, int8_t j) {
+  return (struct large){1, 2, 3, 4};
+}
+
+typedef unsigned char v16i8 __attribute__((vector_size(16)));
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_vec_large_v16i8
+// ILP32-ILP32F-ILP32D-SAME: (ptr noundef [[TMP0:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_vec_large_v16i8(v16i8 x) {
+  x[0] = x[7];
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_vec_large_v16i8_ret
+// ILP32-ILP32F-ILP32D-SAME: (ptr noalias sret(<16 x i8>) align 16 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+v16i8 f_vec_large_v16i8_ret(void) {
+  return (v16i8){1, 2, 3, 4, 5, 6, 7, 8};
+}
+
+// Scalars passed on the stack should have signext/zeroext attributes, just as
+// if they were passed in registers.
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_scalar_stack_1
+// ILP32-ILP32F-ILP32D-SAME: (i32 [[A_COERCE:%.*]], [2 x i32] [[B_COERCE:%.*]], i64 [[C_COERCE:%.*]], ptr noundef [[D:%.*]], i8 noundef zeroext [[E:%.*]], i8 noundef signext [[F:%.*]], i8 noundef zeroext [[G:%.*]], i8 noundef signext [[H:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+int f_scalar_stack_1(struct tiny a, struct small b, struct small_aligned c,
+                     struct large d, uint8_t e, int8_t f, uint8_t g, int8_t h) {
+  return g + h;
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_scalar_stack_2
+// ILP32-ILP32F-ILP32D-SAME: (ptr noalias sret([[STRUCT_LARGE:%.*]]) align 4 [[AGG_RESULT:%.*]], i32 noundef [[A:%.*]], i64 noundef [[B:%.*]], i64 noundef [[C:%.*]], fp128 noundef [[D:%.*]], i8 noundef zeroext [[E:%.*]], i8 noundef signext [[F:%.*]], i8 noundef zeroext [[G:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct large f_scalar_stack_2(int32_t a, int64_t b, int64_t c, long double d,
+                              uint8_t e, int8_t f, uint8_t g) {
+  return (struct large){a, e, f, g};
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local fp128 @f_scalar_stack_3
+// ILP32-ILP32F-ILP32D-SAME: (i32 noundef [[A:%.*]], i64 noundef [[B:%.*]], i64 noundef [[C:%.*]], fp128 noundef [[D:%.*]], i8 noundef zeroext [[E:%.*]], i8 noundef signext [[F:%.*]], i8 noundef zeroext [[G:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+long double f_scalar_stack_3(int32_t a, int64_t b, int64_t c, long double d,
+                             uint8_t e, int8_t f, uint8_t g) {
+  return d;
+}
+
+// Aggregates and >=XLen scalars passed on the stack should be lowered just as
+// they would be if passed via registers.
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_scalar_stack_4
+// ILP32-ILP32F-ILP32D-SAME: (double noundef [[A:%.*]], i64 noundef [[B:%.*]], double noundef [[C:%.*]], i64 noundef [[D:%.*]], i32 noundef [[E:%.*]], i64 noundef [[F:%.*]], float noundef [[G:%.*]], double noundef [[H:%.*]], fp128 noundef [[I:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_scalar_stack_4(double a, int64_t b, double c, int64_t d, int e,
+                      int64_t f, float g, double h, long double i) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_scalar_stack_5
+// ILP32-ILP32F-ILP32D-SAME: (i32 noundef [[A:%.*]], i64 noundef [[B:%.*]], float noundef [[C:%.*]], double noundef [[D:%.*]], fp128 noundef [[E:%.*]], i8 noundef zeroext [[F:%.*]], i8 noundef signext [[G:%.*]], i8 noundef zeroext [[H:%.*]], half noundef [[I:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+int f_scalar_stack_5(int32_t a, int64_t b, float c, double d, long double e,
+                     uint8_t f, int8_t g, uint8_t h, _Float16 i) {
+  return g + h;
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_scalar_stack_6
+// ILP32-ILP32F-ILP32D-SAME: (ptr noalias sret([[STRUCT_LARGE:%.*]]) align 4 [[AGG_RESULT:%.*]], float noundef [[A:%.*]], i64 noundef [[B:%.*]], double noundef [[C:%.*]], fp128 noundef [[D:%.*]], i8 noundef zeroext [[E:%.*]], i8 noundef signext [[F:%.*]], i8 noundef zeroext [[G:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct large f_scalar_stack_6(float a, int64_t b, double c, long double d,
+                              uint8_t e, int8_t f, uint8_t g) {
+  return (struct large){a, e, f, g};
+}
+
+// Aggregates and >=XLen scalars passed on the stack should be lowered just as
+// they would be if passed via registers.
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_agg_stack
+// ILP32-ILP32F-ILP32D-SAME: (double noundef [[A:%.*]], i64 noundef [[B:%.*]], double noundef [[C:%.*]], i64 noundef [[D:%.*]], i32 [[E_COERCE:%.*]], [2 x i32] [[F_COERCE:%.*]], i64 [[G_COERCE:%.*]], ptr noundef [[H:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_agg_stack(double a, int64_t b, double c, int64_t d, struct tiny e,
+                 struct small f, struct small_aligned g, struct large h) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_fpr_tracking
+// ILP32-ILP32F-ILP32D-SAME: (double noundef [[A:%.*]], double noundef [[B:%.*]], double noundef [[C:%.*]], double noundef [[D:%.*]], i8 noundef signext [[E:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_fpr_tracking(double a, double b, double c, double d, int8_t e) {}
+
+// Lowering for doubles is unnmodified, as 64 > FLEN.
+
+struct double_s { double d; };
+
+struct double_double_s { double d; double e; };
+
+struct double_int8_s { double d; int64_t i; };
+
+struct int_double_s { int a; double b; };
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_int_double_s_arg
+// ILP32-ILP32F-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local void @f_int_double_s_arg
+// ILP32D-SAME: (i32 [[TMP0:%.*]], double [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+void f_int_double_s_arg(struct int_double_s a) {}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_ret_int_double_s
+// ILP32-ILP32F-SAME: (ptr noalias sret([[STRUCT_INT_DOUBLE_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local { i32, double } @f_ret_int_double_s
+// ILP32D-SAME: () #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct int_double_s f_ret_int_double_s(void) {
+  return (struct int_double_s){1, 2.0};
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_fpr_tracking2
+// ILP32-ILP32F-ILP32D-SAME: (double noundef [[A:%.*]], double noundef [[B:%.*]], double noundef [[C:%.*]], double noundef [[D:%.*]], double noundef [[E:%.*]], double noundef [[F:%.*]], double noundef [[G:%.*]], double noundef [[H:%.*]], i8 noundef zeroext [[I:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_fpr_tracking2(double a, double b, double c, double d, double e, double f,
+                     double g, double h, uint8_t i) {}
+
+// Check that fp, fp+fp, and int+fp structs are lowered correctly. These will
+// be passed in FPR, FPR+FPR, or GPR+FPR regs if sufficient registers are
+// available the widths are <= XLEN and FLEN, and should be expanded to
+// separate arguments in IR. They are passed by the same rules for returns,
+// but will be lowered to simple two-element structs if necessary (as LLVM IR
+// functions cannot return multiple values).
+
+// A struct containing just one floating-point real is passed as though it
+// were a standalone floating-point real.
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_double_s_arg
+// ILP32-ILP32F-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local void @f_double_s_arg
+// ILP32D-SAME: (double [[TMP0:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+void f_double_s_arg(struct double_s a) {}
+
+// ILP32-ILP32F-LABEL: define dso_local i64 @f_ret_double_s
+// ILP32-ILP32F-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local double @f_ret_double_s
+// ILP32D-SAME: () #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct double_s f_ret_double_s(void) {
+  return (struct double_s){1.0};
+}
+
+// A struct containing a double and any number of zero-width bitfields is
+// passed as though it were a standalone floating-point real.
+
+struct zbf_double_s { int : 0; double f; };
+struct zbf_double_zbf_s { int : 0; double f; int : 0; };
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_zbf_double_s_arg
+// ILP32-ILP32F-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local void @f_zbf_double_s_arg
+// ILP32D-SAME: (double [[TMP0:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+void f_zbf_double_s_arg(struct zbf_double_s a) {}
+
+// ILP32-ILP32F-LABEL: define dso_local i64 @f_ret_zbf_double_s
+// ILP32-ILP32F-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local double @f_ret_zbf_double_s
+// ILP32D-SAME: () #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct zbf_double_s f_ret_zbf_double_s(void) {
+  return (struct zbf_double_s){1.0};
+}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_zbf_double_zbf_s_arg
+// ILP32-ILP32F-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local void @f_zbf_double_zbf_s_arg
+// ILP32D-SAME: (double [[TMP0:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+void f_zbf_double_zbf_s_arg(struct zbf_double_zbf_s a) {}
+
+// ILP32-ILP32F-LABEL: define dso_local i64 @f_ret_zbf_double_zbf_s
+// ILP32-ILP32F-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local double @f_ret_zbf_double_zbf_s
+// ILP32D-SAME: () #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct zbf_double_zbf_s f_ret_zbf_double_zbf_s(void) {
+  return (struct zbf_double_zbf_s){1.0};
+}
+
+// Check that structs containing two floating point values (FLEN <= width) are
+// expanded provided sufficient FPRs are available.
+
+struct double_float_s { double f; float g; };
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_double_double_s_arg
+// ILP32-ILP32F-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local void @f_double_double_s_arg
+// ILP32D-SAME: (double [[TMP0:%.*]], double [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+void f_double_double_s_arg(struct double_double_s a) {}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_ret_double_double_s
+// ILP32-ILP32F-SAME: (ptr noalias sret([[STRUCT_DOUBLE_DOUBLE_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local { double, double } @f_ret_double_double_s
+// ILP32D-SAME: () #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct double_double_s f_ret_double_double_s(void) {
+  return (struct double_double_s){1.0, 2.0};
+}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_double_float_s_arg
+// ILP32-ILP32F-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local void @f_double_float_s_arg
+// ILP32D-SAME: (double [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+void f_double_float_s_arg(struct double_float_s a) {}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_ret_double_float_s
+// ILP32-ILP32F-SAME: (ptr noalias sret([[STRUCT_DOUBLE_FLOAT_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local { double, float } @f_ret_double_float_s
+// ILP32D-SAME: () #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct double_float_s f_ret_double_float_s(void) {
+  return (struct double_float_s){1.0, 2.0};
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_double_double_s_arg_insufficient_fprs
+// ILP32-ILP32F-ILP32D-SAME: (float noundef [[A:%.*]], double noundef [[B:%.*]], double noundef [[C:%.*]], double noundef [[D:%.*]], double noundef [[E:%.*]], double noundef [[F:%.*]], double noundef [[G:%.*]], ptr noundef [[H:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_double_double_s_arg_insufficient_fprs(float a, double b, double c, double d,
+    double e, double f, double g, struct double_double_s h) {}
+
+// Check that structs containing int+double values are expanded, provided
+// sufficient FPRs and GPRs are available. The integer components are neither
+// sign or zero-extended.
+
+struct double_uint8_s { double d; uint8_t i; };
+struct double_int32_s { double d; int32_t i; };
+struct double_int64_s { double d; int64_t i; };
+struct double_int64bf_s { double d; int64_t i : 32; };
+struct double_int8_zbf_s { double d; int8_t i; int : 0; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_double_int8_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_double_int8_s_arg(struct double_int8_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_ret_double_int8_s
+// ILP32-ILP32F-ILP32D-SAME: (ptr noalias sret([[STRUCT_DOUBLE_INT8_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct double_int8_s f_ret_double_int8_s(void) {
+  return (struct double_int8_s){1.0, 2};
+}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_double_uint8_s_arg
+// ILP32-ILP32F-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local void @f_double_uint8_s_arg
+// ILP32D-SAME: (double [[TMP0:%.*]], i8 [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+void f_double_uint8_s_arg(struct double_uint8_s a) {}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_ret_double_uint8_s
+// ILP32-ILP32F-SAME: (ptr noalias sret([[STRUCT_DOUBLE_UINT8_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local { double, i8 } @f_ret_double_uint8_s
+// ILP32D-SAME: () #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct double_uint8_s f_ret_double_uint8_s(void) {
+  return (struct double_uint8_s){1.0, 2};
+}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_double_int32_s_arg
+// ILP32-ILP32F-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local void @f_double_int32_s_arg
+// ILP32D-SAME: (double [[TMP0:%.*]], i32 [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+void f_double_int32_s_arg(struct double_int32_s a) {}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_ret_double_int32_s
+// ILP32-ILP32F-SAME: (ptr noalias sret([[STRUCT_DOUBLE_INT32_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local { double, i32 } @f_ret_double_int32_s
+// ILP32D-SAME: () #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct double_int32_s f_ret_double_int32_s(void) {
+  return (struct double_int32_s){1.0, 2};
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_double_int64_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_double_int64_s_arg(struct double_int64_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_ret_double_int64_s
+// ILP32-ILP32F-ILP32D-SAME: (ptr noalias sret([[STRUCT_DOUBLE_INT64_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct double_int64_s f_ret_double_int64_s(void) {
+  return (struct double_int64_s){1.0, 2};
+}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_double_int64bf_s_arg
+// ILP32-ILP32F-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local void @f_double_int64bf_s_arg
+// ILP32D-SAME: (double [[TMP0:%.*]], i32 [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+void f_double_int64bf_s_arg(struct double_int64bf_s a) {}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_ret_double_int64bf_s
+// ILP32-ILP32F-SAME: (ptr noalias sret([[STRUCT_DOUBLE_INT64BF_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local { double, i32 } @f_ret_double_int64bf_s
+// ILP32D-SAME: () #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct double_int64bf_s f_ret_double_int64bf_s(void) {
+  return (struct double_int64bf_s){1.0, 2};
+}
+
+// The zero-width bitfield means the struct can't be passed according to the
+// floating point calling convention.
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_double_int8_zbf_s
+// ILP32-ILP32F-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local void @f_double_int8_zbf_s
+// ILP32D-SAME: (double [[TMP0:%.*]], i8 [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+void f_double_int8_zbf_s(struct double_int8_zbf_s a) {}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_ret_double_int8_zbf_s
+// ILP32-ILP32F-SAME: (ptr noalias sret([[STRUCT_DOUBLE_INT8_ZBF_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local { double, i8 } @f_ret_double_int8_zbf_s
+// ILP32D-SAME: () #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct double_int8_zbf_s f_ret_double_int8_zbf_s(void) {
+  return (struct double_int8_zbf_s){1.0, 2};
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_double_int8_s_arg_insufficient_gprs
+// ILP32-ILP32F-ILP32D-SAME: (i32 noundef [[A:%.*]], i32 noundef [[B:%.*]], i32 noundef [[C:%.*]], i32 noundef [[D:%.*]], i32 noundef [[E:%.*]], i32 noundef [[F:%.*]], i32 noundef [[G:%.*]], i32 noundef [[H:%.*]], ptr noundef [[I:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_double_int8_s_arg_insufficient_gprs(int a, int b, int c, int d, int e,
+                                          int f, int g, int h, struct double_int8_s i) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_struct_double_int8_insufficient_fprs
+// ILP32-ILP32F-ILP32D-SAME: (float noundef [[A:%.*]], double noundef [[B:%.*]], double noundef [[C:%.*]], double noundef [[D:%.*]], double noundef [[E:%.*]], double noundef [[F:%.*]], double noundef [[G:%.*]], double noundef [[H:%.*]], ptr noundef [[I:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_struct_double_int8_insufficient_fprs(float a, double b, double c, double d,
+                                           double e, double f, double g, double h, struct double_int8_s i) {}
+
+// Complex floating-point values or structs containing a single complex
+// floating-point value should be passed as if it were an fp+fp struct.
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_doublecomplex
+// ILP32-ILP32F-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local void @f_doublecomplex
+// ILP32D-SAME: (double noundef [[A_COERCE0:%.*]], double noundef [[A_COERCE1:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+void f_doublecomplex(double __complex__ a) {}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_ret_doublecomplex
+// ILP32-ILP32F-SAME: (ptr noalias sret({ double, double }) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local { double, double } @f_ret_doublecomplex
+// ILP32D-SAME: () #[[ATTR0]] {
+// ILP32D:  entry:
+//
+double __complex__ f_ret_doublecomplex(void) {
+  return 1.0;
+}
+
+struct doublecomplex_s { double __complex__ c; };
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_doublecomplex_s_arg
+// ILP32-ILP32F-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local void @f_doublecomplex_s_arg
+// ILP32D-SAME: (double [[TMP0:%.*]], double [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+void f_doublecomplex_s_arg(struct doublecomplex_s a) {}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_ret_doublecomplex_s
+// ILP32-ILP32F-SAME: (ptr noalias sret([[STRUCT_DOUBLECOMPLEX_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local { double, double } @f_ret_doublecomplex_s
+// ILP32D-SAME: () #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct doublecomplex_s f_ret_doublecomplex_s(void) {
+  return (struct doublecomplex_s){1.0};
+}
+
+// Test single or two-element structs that need flattening. e.g. those
+// containing nested structs, doubles in small arrays, zero-length structs etc.
+
+struct doublearr1_s { double a[1]; };
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_doublearr1_s_arg
+// ILP32-ILP32F-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local void @f_doublearr1_s_arg
+// ILP32D-SAME: (double [[TMP0:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+void f_doublearr1_s_arg(struct doublearr1_s a) {}
+
+// ILP32-ILP32F-LABEL: define dso_local i64 @f_ret_doublearr1_s
+// ILP32-ILP32F-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local double @f_ret_doublearr1_s
+// ILP32D-SAME: () #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct doublearr1_s f_ret_doublearr1_s(void) {
+  return (struct doublearr1_s){{1.0}};
+}
+
+struct doublearr2_s { double a[2]; };
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_doublearr2_s_arg
+// ILP32-ILP32F-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local void @f_doublearr2_s_arg
+// ILP32D-SAME: (double [[TMP0:%.*]], double [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+void f_doublearr2_s_arg(struct doublearr2_s a) {}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_ret_doublearr2_s
+// ILP32-ILP32F-SAME: (ptr noalias sret([[STRUCT_DOUBLEARR2_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local { double, double } @f_ret_doublearr2_s
+// ILP32D-SAME: () #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct doublearr2_s f_ret_doublearr2_s(void) {
+  return (struct doublearr2_s){{1.0, 2.0}};
+}
+
+struct doublearr2_tricky1_s { struct { double f[1]; } g[2]; };
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_doublearr2_tricky1_s_arg
+// ILP32-ILP32F-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local void @f_doublearr2_tricky1_s_arg
+// ILP32D-SAME: (double [[TMP0:%.*]], double [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+void f_doublearr2_tricky1_s_arg(struct doublearr2_tricky1_s a) {}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_ret_doublearr2_tricky1_s
+// ILP32-ILP32F-SAME: (ptr noalias sret([[STRUCT_DOUBLEARR2_TRICKY1_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local { double, double } @f_ret_doublearr2_tricky1_s
+// ILP32D-SAME: () #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct doublearr2_tricky1_s f_ret_doublearr2_tricky1_s(void) {
+  return (struct doublearr2_tricky1_s){{{{1.0}}, {{2.0}}}};
+}
+
+struct doublearr2_tricky2_s { struct {}; struct { double f[1]; } g[2]; };
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_doublearr2_tricky2_s_arg
+// ILP32-ILP32F-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local void @f_doublearr2_tricky2_s_arg
+// ILP32D-SAME: (double [[TMP0:%.*]], double [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+void f_doublearr2_tricky2_s_arg(struct doublearr2_tricky2_s a) {}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_ret_doublearr2_tricky2_s
+// ILP32-ILP32F-SAME: (ptr noalias sret([[STRUCT_DOUBLEARR2_TRICKY2_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local { double, double } @f_ret_doublearr2_tricky2_s
+// ILP32D-SAME: () #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct doublearr2_tricky2_s f_ret_doublearr2_tricky2_s(void) {
+  return (struct doublearr2_tricky2_s){{}, {{{1.0}}, {{2.0}}}};
+}
+
+struct doublearr2_tricky3_s { union {}; struct { double f[1]; } g[2]; };
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_doublearr2_tricky3_s_arg
+// ILP32-ILP32F-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local void @f_doublearr2_tricky3_s_arg
+// ILP32D-SAME: (double [[TMP0:%.*]], double [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+void f_doublearr2_tricky3_s_arg(struct doublearr2_tricky3_s a) {}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_ret_doublearr2_tricky3_s
+// ILP32-ILP32F-SAME: (ptr noalias sret([[STRUCT_DOUBLEARR2_TRICKY3_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local { double, double } @f_ret_doublearr2_tricky3_s
+// ILP32D-SAME: () #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct doublearr2_tricky3_s f_ret_doublearr2_tricky3_s(void) {
+  return (struct doublearr2_tricky3_s){{}, {{{1.0}}, {{2.0}}}};
+}
+
+struct doublearr2_tricky4_s { union {}; struct { struct {}; double f[1]; } g[2]; };
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_doublearr2_tricky4_s_arg
+// ILP32-ILP32F-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local void @f_doublearr2_tricky4_s_arg
+// ILP32D-SAME: (double [[TMP0:%.*]], double [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+void f_doublearr2_tricky4_s_arg(struct doublearr2_tricky4_s a) {}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_ret_doublearr2_tricky4_s
+// ILP32-ILP32F-SAME: (ptr noalias sret([[STRUCT_DOUBLEARR2_TRICKY4_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local { double, double } @f_ret_doublearr2_tricky4_s
+// ILP32D-SAME: () #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct doublearr2_tricky4_s f_ret_doublearr2_tricky4_s(void) {
+  return (struct doublearr2_tricky4_s){{}, {{{}, {1.0}}, {{}, {2.0}}}};
+}
+
+// Test structs that should be passed according to the normal integer calling
+// convention.
+
+struct int_double_int_s { int a; double b; int c; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_int_double_int_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_int_double_int_s_arg(struct int_double_int_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_ret_int_double_int_s
+// ILP32-ILP32F-ILP32D-SAME: (ptr noalias sret([[STRUCT_INT_DOUBLE_INT_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct int_double_int_s f_ret_int_double_int_s(void) {
+  return (struct int_double_int_s){1, 2.0, 3};
+}
+
+struct int64_double_s { int64_t a; double b; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_int64_double_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_int64_double_s_arg(struct int64_double_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_ret_int64_double_s
+// ILP32-ILP32F-ILP32D-SAME: (ptr noalias sret([[STRUCT_INT64_DOUBLE_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct int64_double_s f_ret_int64_double_s(void) {
+  return (struct int64_double_s){1, 2.0};
+}
+
+struct char_char_double_s { char a; char b; double c; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_char_char_double_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_char_char_double_s_arg(struct char_char_double_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_ret_char_char_double_s
+// ILP32-ILP32F-ILP32D-SAME: (ptr noalias sret([[STRUCT_CHAR_CHAR_DOUBLE_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct char_char_double_s f_ret_char_char_double_s(void) {
+  return (struct char_char_double_s){1, 2, 3.0};
+}
+
+// Unions are always passed according to the integer calling convention, even
+// if they can only contain a double.
+
+union double_u { double a; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_double_u_arg
+// ILP32-ILP32F-ILP32D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_double_u_arg(union double_u a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i64 @f_ret_double_u
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+union double_u f_ret_double_u(void) {
+  return (union double_u){1.0};
+}
+
+// Test that we don't incorrectly think double+int/double+double structs will
+// be returned indirectly and thus have an off-by-one error for the number of
+// GPRs available (this is an edge case when structs > 2*XLEN are still
+// returned in registers). This includes complex doubles, which are treated as
+// double+double structs by the ABI.
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_ret_double_int32_s_double_int32_s_just_sufficient_gprs
+// ILP32-ILP32F-SAME: (ptr noalias sret([[STRUCT_DOUBLE_INT32_S:%.*]]) align 8 [[AGG_RESULT:%.*]], i32 noundef [[A:%.*]], i32 noundef [[B:%.*]], i32 noundef [[C:%.*]], i32 noundef [[D:%.*]], i32 noundef [[E:%.*]], i32 noundef [[F:%.*]], i32 noundef [[G:%.*]], ptr noundef [[H:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local { double, i32 } @f_ret_double_int32_s_double_int32_s_just_sufficient_gprs
+// ILP32D-SAME: (i32 noundef [[A:%.*]], i32 noundef [[B:%.*]], i32 noundef [[C:%.*]], i32 noundef [[D:%.*]], i32 noundef [[E:%.*]], i32 noundef [[F:%.*]], i32 noundef [[G:%.*]], double [[TMP0:%.*]], i32 [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct double_int32_s f_ret_double_int32_s_double_int32_s_just_sufficient_gprs(
+    int a, int b, int c, int d, int e, int f, int g, struct double_int32_s h) {
+  return (struct double_int32_s){1.0, 2};
+}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_ret_double_double_s_double_int32_s_just_sufficient_gprs
+// ILP32-ILP32F-SAME: (ptr noalias sret([[STRUCT_DOUBLE_DOUBLE_S:%.*]]) align 8 [[AGG_RESULT:%.*]], i32 noundef [[A:%.*]], i32 noundef [[B:%.*]], i32 noundef [[C:%.*]], i32 noundef [[D:%.*]], i32 noundef [[E:%.*]], i32 noundef [[F:%.*]], i32 noundef [[G:%.*]], ptr noundef [[H:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local { double, double } @f_ret_double_double_s_double_int32_s_just_sufficient_gprs
+// ILP32D-SAME: (i32 noundef [[A:%.*]], i32 noundef [[B:%.*]], i32 noundef [[C:%.*]], i32 noundef [[D:%.*]], i32 noundef [[E:%.*]], i32 noundef [[F:%.*]], i32 noundef [[G:%.*]], double [[TMP0:%.*]], i32 [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+struct double_double_s f_ret_double_double_s_double_int32_s_just_sufficient_gprs(
+    int a, int b, int c, int d, int e, int f, int g, struct double_int32_s h) {
+  return (struct double_double_s){1.0, 2.0};
+}
+
+// ILP32-ILP32F-LABEL: define dso_local void @f_ret_doublecomplex_double_int32_s_just_sufficient_gprs
+// ILP32-ILP32F-SAME: (ptr noalias sret({ double, double }) align 8 [[AGG_RESULT:%.*]], i32 noundef [[A:%.*]], i32 noundef [[B:%.*]], i32 noundef [[C:%.*]], i32 noundef [[D:%.*]], i32 noundef [[E:%.*]], i32 noundef [[F:%.*]], i32 noundef [[G:%.*]], ptr noundef [[H:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F:  entry:
+//
+// ILP32D-LABEL: define dso_local { double, double } @f_ret_doublecomplex_double_int32_s_just_sufficient_gprs
+// ILP32D-SAME: (i32 noundef [[A:%.*]], i32 noundef [[B:%.*]], i32 noundef [[C:%.*]], i32 noundef [[D:%.*]], i32 noundef [[E:%.*]], i32 noundef [[F:%.*]], i32 noundef [[G:%.*]], double [[TMP0:%.*]], i32 [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32D:  entry:
+//
+double __complex__ f_ret_doublecomplex_double_int32_s_just_sufficient_gprs(
+    int a, int b, int c, int d, int e, int f, int g, struct double_int32_s h) {
+  return 1.0;
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_fpr_tracking_3
+// ILP32-ILP32F-ILP32D-SAME: (float noundef [[A:%.*]], float noundef [[B:%.*]], float noundef [[C:%.*]], float noundef [[D:%.*]], float noundef [[E:%.*]], float noundef [[F:%.*]], float noundef [[G:%.*]], float noundef [[H:%.*]], i8 noundef zeroext [[I:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_fpr_tracking_3(float a, float b, float c, float d, float e, float f,
+                      float g, float h, uint8_t i) {}
+
+// Check that fp, fp+fp, and int+fp structs are lowered correctly. These will
+// be passed in FPR, FPR+FPR, or GPR+FPR regs if sufficient registers are
+// available the widths are <= XLEN and FLEN, and should be expanded to
+// separate arguments in IR. They are passed by the same rules for returns,
+// but will be lowered to simple two-element structs if necessary (as LLVM IR
+// functions cannot return multiple values).
+
+// A struct containing just one floating-point real is passed as though it
+// were a standalone floating-point real.
+
+struct float_s { float f; };
+
+// ILP32-LABEL: define dso_local void @f_float_s_arg
+// ILP32-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_float_s_arg
+// ILP32F-ILP32D-SAME: (float [[TMP0:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_float_s_arg(struct float_s a) {}
+
+// ILP32-LABEL: define dso_local i32 @f_ret_float_s
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local float @f_ret_float_s
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+struct float_s f_ret_float_s(void) {
+  return (struct float_s){1.0};
+}
+
+// A struct containing a float and any number of zero-width bitfields is
+// passed as though it were a standalone floating-point real.
+
+struct zbf_float_s { int : 0; float f; };
+struct zbf_float_zbf_s { int : 0; float f; int : 0; };
+
+// ILP32-LABEL: define dso_local void @f_zbf_float_s_arg
+// ILP32-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_zbf_float_s_arg
+// ILP32F-ILP32D-SAME: (float [[TMP0:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_zbf_float_s_arg(struct zbf_float_s a) {}
+
+// ILP32-LABEL: define dso_local i32 @f_ret_zbf_float_s
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local float @f_ret_zbf_float_s
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+struct zbf_float_s f_ret_zbf_float_s(void) {
+  return (struct zbf_float_s){1.0};
+}
+
+// ILP32-LABEL: define dso_local void @f_zbf_float_zbf_s_arg
+// ILP32-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_zbf_float_zbf_s_arg
+// ILP32F-ILP32D-SAME: (float [[TMP0:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_zbf_float_zbf_s_arg(struct zbf_float_zbf_s a) {}
+
+// ILP32-LABEL: define dso_local i32 @f_ret_zbf_float_zbf_s
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local float @f_ret_zbf_float_zbf_s
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+struct zbf_float_zbf_s f_ret_zbf_float_zbf_s(void) {
+  return (struct zbf_float_zbf_s){1.0};
+}
+
+// Check that structs containing two float values (FLEN <= width) are expanded
+// provided sufficient FPRs are available.
+
+struct float_float_s { float f; float g; };
+
+// ILP32-LABEL: define dso_local void @f_float_float_s_arg
+// ILP32-SAME: ([2 x i32] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_float_float_s_arg
+// ILP32F-ILP32D-SAME: (float [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_float_float_s_arg(struct float_float_s a) {}
+
+// ILP32-LABEL: define dso_local [2 x i32] @f_ret_float_float_s
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local { float, float } @f_ret_float_float_s
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+struct float_float_s f_ret_float_float_s(void) {
+  return (struct float_float_s){1.0, 2.0};
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float_float_s_arg_insufficient_fprs
+// ILP32-ILP32F-ILP32D-SAME: (float noundef [[A:%.*]], float noundef [[B:%.*]], float noundef [[C:%.*]], float noundef [[D:%.*]], float noundef [[E:%.*]], float noundef [[F:%.*]], float noundef [[G:%.*]], [2 x i32] [[H_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float_float_s_arg_insufficient_fprs(float a, float b, float c, float d,
+    float e, float f, float g, struct float_float_s h) {}
+
+// Check that structs containing int+float values are expanded, provided
+// sufficient FPRs and GPRs are available. The integer components are neither
+// sign or zero-extended.
+
+struct float_int8_s { float f; int8_t i; };
+struct float_uint8_s { float f; uint8_t i; };
+struct float_int32_s { float f; int32_t i; };
+struct float_int64_s { float f; int64_t i; };
+struct float_int64bf_s { float f; int64_t i : 32; };
+struct float_int8_zbf_s { float f; int8_t i; int : 0; };
+
+// ILP32-LABEL: define dso_local void @f_float_int8_s_arg
+// ILP32-SAME: ([2 x i32] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_float_int8_s_arg
+// ILP32F-ILP32D-SAME: (float [[TMP0:%.*]], i8 [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_float_int8_s_arg(struct float_int8_s a) {}
+
+// ILP32-LABEL: define dso_local [2 x i32] @f_ret_float_int8_s
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local { float, i8 } @f_ret_float_int8_s
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+struct float_int8_s f_ret_float_int8_s(void) {
+  return (struct float_int8_s){1.0, 2};
+}
+
+// ILP32-LABEL: define dso_local void @f_float_uint8_s_arg
+// ILP32-SAME: ([2 x i32] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_float_uint8_s_arg
+// ILP32F-ILP32D-SAME: (float [[TMP0:%.*]], i8 [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_float_uint8_s_arg(struct float_uint8_s a) {}
+
+// ILP32-LABEL: define dso_local [2 x i32] @f_ret_float_uint8_s
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local { float, i8 } @f_ret_float_uint8_s
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+struct float_uint8_s f_ret_float_uint8_s(void) {
+  return (struct float_uint8_s){1.0, 2};
+}
+
+// ILP32-LABEL: define dso_local void @f_float_int32_s_arg
+// ILP32-SAME: ([2 x i32] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_float_int32_s_arg
+// ILP32F-ILP32D-SAME: (float [[TMP0:%.*]], i32 [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_float_int32_s_arg(struct float_int32_s a) {}
+
+// ILP32-LABEL: define dso_local [2 x i32] @f_ret_float_int32_s
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local { float, i32 } @f_ret_float_int32_s
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+struct float_int32_s f_ret_float_int32_s(void) {
+  return (struct float_int32_s){1.0, 2};
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float_int64_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float_int64_s_arg(struct float_int64_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_ret_float_int64_s
+// ILP32-ILP32F-ILP32D-SAME: (ptr noalias sret([[STRUCT_FLOAT_INT64_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct float_int64_s f_ret_float_int64_s(void) {
+  return (struct float_int64_s){1.0, 2};
+}
+
+// ILP32-LABEL: define dso_local void @f_float_int64bf_s_arg
+// ILP32-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_float_int64bf_s_arg
+// ILP32F-ILP32D-SAME: (float [[TMP0:%.*]], i32 [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_float_int64bf_s_arg(struct float_int64bf_s a) {}
+
+// ILP32-LABEL: define dso_local i64 @f_ret_float_int64bf_s
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local { float, i32 } @f_ret_float_int64bf_s
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+struct float_int64bf_s f_ret_float_int64bf_s(void) {
+  return (struct float_int64bf_s){1.0, 2};
+}
+
+// The zero-width bitfield means the struct can't be passed according to the
+// floating point calling convention.
+
+// ILP32-LABEL: define dso_local void @f_float_int8_zbf_s
+// ILP32-SAME: ([2 x i32] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_float_int8_zbf_s
+// ILP32F-ILP32D-SAME: (float [[TMP0:%.*]], i8 [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_float_int8_zbf_s(struct float_int8_zbf_s a) {}
+
+// ILP32-LABEL: define dso_local [2 x i32] @f_ret_float_int8_zbf_s
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local { float, i8 } @f_ret_float_int8_zbf_s
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+struct float_int8_zbf_s f_ret_float_int8_zbf_s(void) {
+  return (struct float_int8_zbf_s){1.0, 2};
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float_int8_s_arg_insufficient_gprs
+// ILP32-ILP32F-ILP32D-SAME: (i32 noundef [[A:%.*]], i32 noundef [[B:%.*]], i32 noundef [[C:%.*]], i32 noundef [[D:%.*]], i32 noundef [[E:%.*]], i32 noundef [[F:%.*]], i32 noundef [[G:%.*]], i32 noundef [[H:%.*]], [2 x i32] [[I_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float_int8_s_arg_insufficient_gprs(int a, int b, int c, int d, int e,
+                                          int f, int g, int h, struct float_int8_s i) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_struct_float_int8_insufficient_fprs
+// ILP32-ILP32F-ILP32D-SAME: (float noundef [[A:%.*]], float noundef [[B:%.*]], float noundef [[C:%.*]], float noundef [[D:%.*]], float noundef [[E:%.*]], float noundef [[F:%.*]], float noundef [[G:%.*]], float noundef [[H:%.*]], [2 x i32] [[I_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_struct_float_int8_insufficient_fprs(float a, float b, float c, float d,
+                                           float e, float f, float g, float h, struct float_int8_s i) {}
+
+// Complex floating-point values or structs containing a single complex
+// floating-point value should be passed as if it were an fp+fp struct.
+
+// ILP32-LABEL: define dso_local void @f_floatcomplex
+// ILP32-SAME: ([2 x i32] noundef [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_floatcomplex
+// ILP32F-ILP32D-SAME: (float noundef [[A_COERCE0:%.*]], float noundef [[A_COERCE1:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_floatcomplex(float __complex__ a) {}
+
+// ILP32-LABEL: define dso_local [2 x i32] @f_ret_floatcomplex
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local { float, float } @f_ret_floatcomplex
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+float __complex__ f_ret_floatcomplex(void) {
+  return 1.0;
+}
+
+struct floatcomplex_s { float __complex__ c; };
+
+// ILP32-LABEL: define dso_local void @f_floatcomplex_s_arg
+// ILP32-SAME: ([2 x i32] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_floatcomplex_s_arg
+// ILP32F-ILP32D-SAME: (float [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_floatcomplex_s_arg(struct floatcomplex_s a) {}
+
+// ILP32-LABEL: define dso_local [2 x i32] @f_ret_floatcomplex_s
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local { float, float } @f_ret_floatcomplex_s
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+struct floatcomplex_s f_ret_floatcomplex_s(void) {
+  return (struct floatcomplex_s){1.0};
+}
+
+// Test single or two-element structs that need flattening. e.g. those
+// containing nested structs, floats in small arrays, zero-length structs etc.
+
+struct floatarr1_s { float a[1]; };
+
+// ILP32-LABEL: define dso_local void @f_floatarr1_s_arg
+// ILP32-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_floatarr1_s_arg
+// ILP32F-ILP32D-SAME: (float [[TMP0:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_floatarr1_s_arg(struct floatarr1_s a) {}
+
+// ILP32-LABEL: define dso_local i32 @f_ret_floatarr1_s
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local float @f_ret_floatarr1_s
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+struct floatarr1_s f_ret_floatarr1_s(void) {
+  return (struct floatarr1_s){{1.0}};
+}
+
+struct floatarr2_s { float a[2]; };
+
+// ILP32-LABEL: define dso_local void @f_floatarr2_s_arg
+// ILP32-SAME: ([2 x i32] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_floatarr2_s_arg
+// ILP32F-ILP32D-SAME: (float [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_floatarr2_s_arg(struct floatarr2_s a) {}
+
+// ILP32-LABEL: define dso_local [2 x i32] @f_ret_floatarr2_s
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local { float, float } @f_ret_floatarr2_s
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+struct floatarr2_s f_ret_floatarr2_s(void) {
+  return (struct floatarr2_s){{1.0, 2.0}};
+}
+
+struct floatarr2_tricky1_s { struct { float f[1]; } g[2]; };
+
+// ILP32-LABEL: define dso_local void @f_floatarr2_tricky1_s_arg
+// ILP32-SAME: ([2 x i32] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_floatarr2_tricky1_s_arg
+// ILP32F-ILP32D-SAME: (float [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_floatarr2_tricky1_s_arg(struct floatarr2_tricky1_s a) {}
+
+// ILP32-LABEL: define dso_local [2 x i32] @f_ret_floatarr2_tricky1_s
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local { float, float } @f_ret_floatarr2_tricky1_s
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+struct floatarr2_tricky1_s f_ret_floatarr2_tricky1_s(void) {
+  return (struct floatarr2_tricky1_s){{{{1.0}}, {{2.0}}}};
+}
+
+struct floatarr2_tricky2_s { struct {}; struct { float f[1]; } g[2]; };
+
+// ILP32-LABEL: define dso_local void @f_floatarr2_tricky2_s_arg
+// ILP32-SAME: ([2 x i32] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_floatarr2_tricky2_s_arg
+// ILP32F-ILP32D-SAME: (float [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_floatarr2_tricky2_s_arg(struct floatarr2_tricky2_s a) {}
+
+// ILP32-LABEL: define dso_local [2 x i32] @f_ret_floatarr2_tricky2_s
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local { float, float } @f_ret_floatarr2_tricky2_s
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+struct floatarr2_tricky2_s f_ret_floatarr2_tricky2_s(void) {
+  return (struct floatarr2_tricky2_s){{}, {{{1.0}}, {{2.0}}}};
+}
+
+struct floatarr2_tricky3_s { union {}; struct { float f[1]; } g[2]; };
+
+// ILP32-LABEL: define dso_local void @f_floatarr2_tricky3_s_arg
+// ILP32-SAME: ([2 x i32] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_floatarr2_tricky3_s_arg
+// ILP32F-ILP32D-SAME: (float [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_floatarr2_tricky3_s_arg(struct floatarr2_tricky3_s a) {}
+
+// ILP32-LABEL: define dso_local [2 x i32] @f_ret_floatarr2_tricky3_s
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local { float, float } @f_ret_floatarr2_tricky3_s
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+struct floatarr2_tricky3_s f_ret_floatarr2_tricky3_s(void) {
+  return (struct floatarr2_tricky3_s){{}, {{{1.0}}, {{2.0}}}};
+}
+
+struct floatarr2_tricky4_s { union {}; struct { struct {}; float f[1]; } g[2]; };
+
+// ILP32-LABEL: define dso_local void @f_floatarr2_tricky4_s_arg
+// ILP32-SAME: ([2 x i32] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_floatarr2_tricky4_s_arg
+// ILP32F-ILP32D-SAME: (float [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_floatarr2_tricky4_s_arg(struct floatarr2_tricky4_s a) {}
+
+// ILP32-LABEL: define dso_local [2 x i32] @f_ret_floatarr2_tricky4_s
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local { float, float } @f_ret_floatarr2_tricky4_s
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+struct floatarr2_tricky4_s f_ret_floatarr2_tricky4_s(void) {
+  return (struct floatarr2_tricky4_s){{}, {{{}, {1.0}}, {{}, {2.0}}}};
+}
+
+// Test structs that should be passed according to the normal integer calling
+// convention.
+
+struct int_float_int_s { int a; float b; int c; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_int_float_int_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_int_float_int_s_arg(struct int_float_int_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_ret_int_float_int_s
+// ILP32-ILP32F-ILP32D-SAME: (ptr noalias sret([[STRUCT_INT_FLOAT_INT_S:%.*]]) align 4 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct int_float_int_s f_ret_int_float_int_s(void) {
+  return (struct int_float_int_s){1, 2.0, 3};
+}
+
+struct int64_float_s { int64_t a; float b; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_int64_float_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_int64_float_s_arg(struct int64_float_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_ret_int64_float_s
+// ILP32-ILP32F-ILP32D-SAME: (ptr noalias sret([[STRUCT_INT64_FLOAT_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct int64_float_s f_ret_int64_float_s(void) {
+  return (struct int64_float_s){1, 2.0};
+}
+
+struct char_char_float_s { char a; char b; float c; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_char_char_float_s_arg
+// ILP32-ILP32F-ILP32D-SAME: ([2 x i32] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_char_char_float_s_arg(struct char_char_float_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local [2 x i32] @f_ret_char_char_float_s
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct char_char_float_s f_ret_char_char_float_s(void) {
+  return (struct char_char_float_s){1, 2, 3.0};
+}
+
+// Unions are always passed according to the integer calling convention, even
+// if they can only contain a float.
+
+union float_u { float a; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float_u_arg
+// ILP32-ILP32F-ILP32D-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float_u_arg(union float_u a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_ret_float_u
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+union float_u f_ret_float_u(void) {
+  return (union float_u){1.0};
+}
+
+// Check that fp, fp+fp, and int+fp structs are lowered correctly. These will
+// be passed in FPR, FPR+FPR, or GPR+FPR regs if sufficient registers are
+// available the widths are <= XLEN and FLEN, and should be expanded to
+// separate arguments in IR. They are passed by the same rules for returns,
+// but will be lowered to simple two-element structs if necessary (as LLVM IR
+// functions cannot return multiple values).
+// FIXME: Essentially all test cases below involving _Float16 in structs
+// aren't lowered according to the rules in the FP calling convention (i.e.
+// are incorrect for ilp32f/ilp32d).
+
+struct float16_s { _Float16 f; };
+
+// A struct containing just one floating-point real is passed as though it
+// were a standalone floating-point real.
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float16_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float16_s_arg(struct float16_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_ret_float16_s
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct float16_s f_ret_float16_s(void) {
+  return (struct float16_s){1.0};
+}
+
+// A struct containing a double and any number of zero-width bitfields is
+// passed as though it were a standalone floating-point real.
+
+struct zbf_float16_s { int : 0; _Float16 f; };
+struct zbf_float16_zbf_s { int : 0; _Float16 f; int : 0; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_zbf_float16_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_zbf_float16_s_arg(struct zbf_float16_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_ret_zbf_float16_s
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct zbf_float16_s f_ret_zbf_float16_s(void) {
+  return (struct zbf_float16_s){1.0};
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_zbf_float16_zbf_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_zbf_float16_zbf_s_arg(struct zbf_float16_zbf_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_ret_zbf_float16_zbf_s
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct zbf_float16_zbf_s f_ret_zbf_float16_zbf_s(void) {
+  return (struct zbf_float16_zbf_s){1.0};
+}
+
+// Check that structs containing two floating point values (FLEN <= width) are
+// expanded provided sufficient FPRs are available.
+
+struct double_float16_s { double f; _Float16 g; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_double_float16_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_double_float16_s_arg(struct double_float16_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_ret_double_float16_s
+// ILP32-ILP32F-ILP32D-SAME: (ptr noalias sret([[STRUCT_DOUBLE_FLOAT16_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct double_float16_s f_ret_double_float16_s(void) {
+  return (struct double_float16_s){1.0, 2.0};
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_double_float16_s_arg_insufficient_fprs
+// ILP32-ILP32F-ILP32D-SAME: (float noundef [[A:%.*]], double noundef [[B:%.*]], double noundef [[C:%.*]], double noundef [[D:%.*]], double noundef [[E:%.*]], double noundef [[F:%.*]], double noundef [[G:%.*]], ptr noundef [[H:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_double_float16_s_arg_insufficient_fprs(float a, double b, double c, double d,
+    double e, double f, double g, struct double_float16_s h) {}
+
+// Check that structs containing int+_Float16 values are expanded, provided
+// sufficient FPRs and GPRs are available. The integer components are neither
+// sign or zero-extended.
+
+struct float16_int8_s { _Float16 f; int8_t i; };
+struct float16_uint8_s { _Float16 f; uint8_t i; };
+struct float16_int32_s { _Float16 f; int32_t i; };
+struct float16_int64_s { _Float16 f; int64_t i; };
+struct float16_int64bf_s { _Float16 f; int64_t i : 32; };
+struct float16_int8_zbf_s { _Float16 f; int8_t i; int : 0; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float16_int8_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float16_int8_s_arg(struct float16_int8_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_ret_float16_int8_s
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct float16_int8_s f_ret_float16_int8_s(void) {
+  return (struct float16_int8_s){1.0, 2};
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float16_uint8_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float16_uint8_s_arg(struct float16_uint8_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_ret_float16_uint8_s
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct float16_uint8_s f_ret_float16_uint8_s(void) {
+  return (struct float16_uint8_s){1.0, 2};
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float16_int32_s_arg
+// ILP32-ILP32F-ILP32D-SAME: ([2 x i32] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float16_int32_s_arg(struct float16_int32_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local [2 x i32] @f_ret_float16_int32_s
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct float16_int32_s f_ret_float16_int32_s(void) {
+  return (struct float16_int32_s){1.0, 2};
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float16_int64_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float16_int64_s_arg(struct float16_int64_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_ret_float16_int64_s
+// ILP32-ILP32F-ILP32D-SAME: (ptr noalias sret([[STRUCT_FLOAT16_INT64_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct float16_int64_s f_ret_float16_int64_s(void) {
+  return (struct float16_int64_s){1.0, 2};
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float16_int64bf_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float16_int64bf_s_arg(struct float16_int64bf_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i64 @f_ret_float16_int64bf_s
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct float16_int64bf_s f_ret_float16_int64bf_s(void) {
+  return (struct float16_int64bf_s){1.0, 2};
+}
+
+// The zero-width bitfield means the struct can't be passed according to the
+// floating point calling convention.
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float16_int8_zbf_s
+// ILP32-ILP32F-ILP32D-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float16_int8_zbf_s(struct float16_int8_zbf_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_ret_float16_int8_zbf_s
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct float16_int8_zbf_s f_ret_float16_int8_zbf_s(void) {
+  return (struct float16_int8_zbf_s){1.0, 2};
+}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float16_int8_s_arg_insufficient_gprs
+// ILP32-ILP32F-ILP32D-SAME: (i32 noundef [[A:%.*]], i32 noundef [[B:%.*]], i32 noundef [[C:%.*]], i32 noundef [[D:%.*]], i32 noundef [[E:%.*]], i32 noundef [[F:%.*]], i32 noundef [[G:%.*]], i32 noundef [[H:%.*]], i32 [[I_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float16_int8_s_arg_insufficient_gprs(int a, int b, int c, int d, int e,
+                                            int f, int g, int h, struct float16_int8_s i) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_struct_float16_int8_insufficient_fprs
+// ILP32-ILP32F-ILP32D-SAME: (float noundef [[A:%.*]], double noundef [[B:%.*]], double noundef [[C:%.*]], double noundef [[D:%.*]], double noundef [[E:%.*]], double noundef [[F:%.*]], double noundef [[G:%.*]], double noundef [[H:%.*]], i32 [[I_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_struct_float16_int8_insufficient_fprs(float a, double b, double c, double d,
+                                             double e, double f, double g, double h, struct float16_int8_s i) {}
+
+// Complex floating-point values or structs containing a single complex
+// floating-point value should be passed as if it were an fp+fp struct.
+
+// ILP32-LABEL: define dso_local void @f_float16complex
+// ILP32-SAME: (i32 noundef [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_float16complex
+// ILP32F-ILP32D-SAME: (half noundef [[A_COERCE0:%.*]], half noundef [[A_COERCE1:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_float16complex(_Float16 __complex__ a) {}
+
+// ILP32-LABEL: define dso_local i32 @f_ret_float16complex
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local { half, half } @f_ret_float16complex
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+_Float16 __complex__ f_ret_float16complex(void) {
+  return 1.0;
+}
+
+struct float16complex_s { _Float16 __complex__ c; };
+
+// ILP32-LABEL: define dso_local void @f_float16complex_s_arg
+// ILP32-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local void @f_float16complex_s_arg
+// ILP32F-ILP32D-SAME: (half [[TMP0:%.*]], half [[TMP1:%.*]]) #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+void f_float16complex_s_arg(struct float16complex_s a) {}
+
+// ILP32-LABEL: define dso_local i32 @f_ret_float16complex_s
+// ILP32-SAME: () #[[ATTR0]] {
+// ILP32:  entry:
+//
+// ILP32F-ILP32D-LABEL: define dso_local { half, half } @f_ret_float16complex_s
+// ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32F-ILP32D:  entry:
+//
+struct float16complex_s f_ret_float16complex_s(void) {
+  return (struct float16complex_s){1.0};
+}
+
+// Test single or two-element structs that need flattening. e.g. those
+// containing nested structs, _Float16 in small arrays, zero-length structs etc.
+
+struct float16arr1_s { _Float16 a[1]; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float16arr1_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float16arr1_s_arg(struct float16arr1_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_ret_float16arr1_s
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct float16arr1_s f_ret_float16arr1_s(void) {
+  return (struct float16arr1_s){{1.0}};
+}
+
+struct float16arr2_s { _Float16 a[2]; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float16arr2_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float16arr2_s_arg(struct float16arr2_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_ret_float16arr2_s
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct float16arr2_s f_ret_float16arr2_s(void) {
+  return (struct float16arr2_s){{1.0, 2.0}};
+}
+
+struct float16arr2_tricky1_s { struct { _Float16 f[1]; } g[2]; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float16arr2_tricky1_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float16arr2_tricky1_s_arg(struct float16arr2_tricky1_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_ret_float16arr2_tricky1_s
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct float16arr2_tricky1_s f_ret_float16arr2_tricky1_s(void) {
+  return (struct float16arr2_tricky1_s){{{{1.0}}, {{2.0}}}};
+}
+
+struct float16arr2_tricky2_s { struct {}; struct { _Float16 f[1]; } g[2]; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float16arr2_tricky2_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float16arr2_tricky2_s_arg(struct float16arr2_tricky2_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_ret_float16arr2_tricky2_s
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct float16arr2_tricky2_s f_ret_float16arr2_tricky2_s(void) {
+  return (struct float16arr2_tricky2_s){{}, {{{1.0}}, {{2.0}}}};
+}
+
+struct float16arr2_tricky3_s { union {}; struct { _Float16 f[1]; } g[2]; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float16arr2_tricky3_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float16arr2_tricky3_s_arg(struct float16arr2_tricky3_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_ret_float16arr2_tricky3_s
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct float16arr2_tricky3_s f_ret_float16arr2_tricky3_s(void) {
+  return (struct float16arr2_tricky3_s){{}, {{{1.0}}, {{2.0}}}};
+}
+
+struct float16arr2_tricky4_s { union {}; struct { struct {}; _Float16 f[1]; } g[2]; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float16arr2_tricky4_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float16arr2_tricky4_s_arg(struct float16arr2_tricky4_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_ret_float16arr2_tricky4_s
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct float16arr2_tricky4_s f_ret_float16arr2_tricky4_s(void) {
+  return (struct float16arr2_tricky4_s){{}, {{{}, {1.0}}, {{}, {2.0}}}};
+}
+
+// Test structs that should be passed according to the normal integer calling
+// convention.
+
+struct int_float16_int_s { int a; _Float16 b; int c; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_int_float16_int_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_int_float16_int_s_arg(struct int_float16_int_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_ret_int_float16_int_s
+// ILP32-ILP32F-ILP32D-SAME: (ptr noalias sret([[STRUCT_INT_FLOAT16_INT_S:%.*]]) align 4 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct int_float16_int_s f_ret_int_float16_int_s(void) {
+  return (struct int_float16_int_s){1, 2.0, 3};
+}
+
+struct int64_float16_s { int64_t a; _Float16 b; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_int64_float16_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_int64_float16_s_arg(struct int64_float16_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_ret_int64_float16_s
+// ILP32-ILP32F-ILP32D-SAME: (ptr noalias sret([[STRUCT_INT64_FLOAT16_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct int64_float16_s f_ret_int64_float16_s(void) {
+  return (struct int64_float16_s){1, 2.0};
+}
+
+struct char_char_float16_s { char a; char b; _Float16 c; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_char_char_float16_s_arg
+// ILP32-ILP32F-ILP32D-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_char_char_float16_s_arg(struct char_char_float16_s a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_ret_char_char_float16_s
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+struct char_char_float16_s f_ret_char_char_float16_s(void) {
+  return (struct char_char_float16_s){1, 2, 3.0};
+}
+
+// Unions are always passed according to the integer calling convention, even
+// if they can only contain a double.
+
+union float16_u { _Float16 a; };
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local void @f_float16_u_arg
+// ILP32-ILP32F-ILP32D-SAME: (i32 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+void f_float16_u_arg(union float16_u a) {}
+
+// ILP32-ILP32F-ILP32D-LABEL: define dso_local i32 @f_ret_float16_u
+// ILP32-ILP32F-ILP32D-SAME: () #[[ATTR0]] {
+// ILP32-ILP32F-ILP32D:  entry:
+//
+union float16_u f_ret_float16_u(void) {
+  return (union float16_u){1.0};
+}
+
+//// NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+// ILP32F: {{.*}}
diff --git a/clang/test/CodeGen/RISCV/riscv32-ilp32-abi.c b/clang/test/CodeGen/RISCV/riscv32-ilp32-abi.c
deleted file mode 100644
index 9b157ad5c782e..0000000000000
--- a/clang/test/CodeGen/RISCV/riscv32-ilp32-abi.c
+++ /dev/null
@@ -1,49 +0,0 @@
-// RUN: %clang_cc1 -triple riscv32 -emit-llvm %s -o - | FileCheck %s
-
-// This file contains test cases that will have different output for ilp32 vs
-// the other 32-bit ABIs.
-
-#include <stddef.h>
-#include <stdint.h>
-
-struct tiny {
-  uint8_t a, b, c, d;
-};
-
-struct small {
-  int32_t a, *b;
-};
-
-struct small_aligned {
-  int64_t a;
-};
-
-struct large {
-  int32_t a, b, c, d;
-};
-
-// Scalars passed on the stack should have signext/zeroext attributes, just as
-// if they were passed in registers.
-
-// CHECK-LABEL: define{{.*}} i32 @f_scalar_stack_1(i32 noundef %a, i64 noundef %b, float noundef %c, double noundef %d, fp128 noundef %e, i8 noundef zeroext %f, i8 noundef signext %g, i8 noundef zeroext %h)
-int f_scalar_stack_1(int32_t a, int64_t b, float c, double d, long double e,
-                     uint8_t f, int8_t g, uint8_t h) {
-  return g + h;
-}
-
-// CHECK-LABEL: define{{.*}} void @f_scalar_stack_2(ptr noalias sret(%struct.large) align 4 %agg.result, float noundef %a, i64 noundef %b, double noundef %c, fp128 noundef %d, i8 noundef zeroext %e, i8 noundef signext %f, i8 noundef zeroext %g)
-struct large f_scalar_stack_2(float a, int64_t b, double c, long double d,
-                              uint8_t e, int8_t f, uint8_t g) {
-  return (struct large){a, e, f, g};
-}
-
-// Aggregates and >=XLen scalars passed on the stack should be lowered just as
-// they would be if passed via registers.
-
-// CHECK-LABEL: define{{.*}} void @f_scalar_stack_3(double noundef %a, i64 noundef %b, double noundef %c, i64 noundef %d, i32 noundef %e, i64 noundef %f, float noundef %g, double noundef %h, fp128 noundef %i)
-void f_scalar_stack_3(double a, int64_t b, double c, int64_t d, int e,
-                      int64_t f, float g, double h, long double i) {}
-
-// CHECK-LABEL: define{{.*}} void @f_agg_stack(double noundef %a, i64 noundef %b, double noundef %c, i64 noundef %d, i32 %e.coerce, [2 x i32] %f.coerce, i64 %g.coerce, ptr noundef %h)
-void f_agg_stack(double a, int64_t b, double c, int64_t d, struct tiny e,
-                 struct small f, struct small_aligned g, struct large h) {}
diff --git a/clang/test/CodeGen/RISCV/riscv32-ilp32-ilp32f-abi.c b/clang/test/CodeGen/RISCV/riscv32-ilp32-ilp32f-abi.c
deleted file mode 100644
index 3016feacb10c4..0000000000000
--- a/clang/test/CodeGen/RISCV/riscv32-ilp32-ilp32f-abi.c
+++ /dev/null
@@ -1,51 +0,0 @@
-// RUN: %clang_cc1 -triple riscv32 -emit-llvm %s -o - | FileCheck %s
-// RUN: %clang_cc1 -triple riscv32 -target-feature +f -target-abi ilp32f -emit-llvm %s -o - \
-// RUN:     | FileCheck %s
-
-// This file contains test cases that will have the same output for the ilp32
-// and ilp32f ABIs.
-
-#include <stddef.h>
-#include <stdint.h>
-
-struct tiny {
-  uint8_t a, b, c, d;
-};
-
-struct small {
-  int32_t a, *b;
-};
-
-struct small_aligned {
-  int64_t a;
-};
-
-struct large {
-  int32_t a, b, c, d;
-};
-
-// Scalars passed on the stack should have signext/zeroext attributes, just as
-// if they were passed in registers.
-
-// CHECK-LABEL: define{{.*}} i32 @f_scalar_stack_1(i32 noundef %a, i64 noundef %b, i32 noundef %c, double noundef %d, fp128 noundef %e, i8 noundef zeroext %f, i8 noundef signext %g, i8 noundef zeroext %h)
-int f_scalar_stack_1(int32_t a, int64_t b, int32_t c, double d, long double e,
-                     uint8_t f, int8_t g, uint8_t h) {
-  return g + h;
-}
-
-// CHECK-LABEL: define{{.*}} void @f_scalar_stack_2(ptr noalias sret(%struct.large) align 4 %agg.result, i32 noundef %a, i64 noundef %b, double noundef %c, fp128 noundef %d, i8 noundef zeroext %e, i8 noundef signext %f, i8 noundef zeroext %g)
-struct large f_scalar_stack_2(int32_t a, int64_t b, double c, long double d,
-                              uint8_t e, int8_t f, uint8_t g) {
-  return (struct large){a, e, f, g};
-}
-
-// Aggregates and >=XLen scalars passed on the stack should be lowered just as
-// they would be if passed via registers.
-
-// CHECK-LABEL: define{{.*}} void @f_scalar_stack_3(double noundef %a, i64 noundef %b, double noundef %c, i64 noundef %d, i32 noundef %e, i64 noundef %f, i32 noundef %g, double noundef %h, fp128 noundef %i)
-void f_scalar_stack_3(double a, int64_t b, double c, int64_t d, int e,
-                      int64_t f, int32_t g, double h, long double i) {}
-
-// CHECK-LABEL: define{{.*}} void @f_agg_stack(double noundef %a, i64 noundef %b, double noundef %c, i64 noundef %d, i32 %e.coerce, [2 x i32] %f.coerce, i64 %g.coerce, ptr noundef %h)
-void f_agg_stack(double a, int64_t b, double c, int64_t d, struct tiny e,
-                 struct small f, struct small_aligned g, struct large h) {}
diff --git a/clang/test/CodeGen/RISCV/riscv32-ilp32-ilp32f-ilp32d-abi.c b/clang/test/CodeGen/RISCV/riscv32-ilp32-ilp32f-ilp32d-abi.c
deleted file mode 100644
index 376828a36ef54..0000000000000
--- a/clang/test/CodeGen/RISCV/riscv32-ilp32-ilp32f-ilp32d-abi.c
+++ /dev/null
@@ -1,407 +0,0 @@
-// RUN: %clang_cc1 -triple riscv32 -emit-llvm %s -o - | FileCheck %s
-// RUN: %clang_cc1 -triple riscv32 -emit-llvm -fforce-enable-int128 %s -o - \
-// RUN:   | FileCheck %s -check-prefixes=CHECK,CHECK-FORCEINT128
-// RUN: %clang_cc1 -triple riscv32 -target-feature +f -target-abi ilp32f -emit-llvm %s -o - \
-// RUN:     | FileCheck %s
-// RUN: %clang_cc1 -triple riscv32 -target-feature +d -target-feature +f -target-abi ilp32d -emit-llvm %s -o - \
-// RUN:     | FileCheck %s
-
-// This file contains test cases that will have the same output for the ilp32,
-// ilp32f, and ilp32d ABIs.
-
-#include <stddef.h>
-#include <stdint.h>
-
-// CHECK-LABEL: define{{.*}} void @f_void()
-void f_void(void) {}
-
-// Scalar arguments and return values smaller than the word size are extended
-// according to the sign of their type, up to 32 bits
-
-// CHECK-LABEL: define{{.*}} zeroext i1 @f_scalar_0(i1 noundef zeroext %x)
-_Bool f_scalar_0(_Bool x) { return x; }
-
-// CHECK-LABEL: define{{.*}} signext i8 @f_scalar_1(i8 noundef signext %x)
-int8_t f_scalar_1(int8_t x) { return x; }
-
-// CHECK-LABEL: define{{.*}} zeroext i8 @f_scalar_2(i8 noundef zeroext %x)
-uint8_t f_scalar_2(uint8_t x) { return x; }
-
-// CHECK-LABEL: define{{.*}} i32 @f_scalar_3(i32 noundef %x)
-int32_t f_scalar_3(int32_t x) { return x; }
-
-// CHECK-LABEL: define{{.*}} i64 @f_scalar_4(i64 noundef %x)
-int64_t f_scalar_4(int64_t x) { return x; }
-
-#ifdef __SIZEOF_INT128__
-// CHECK-FORCEINT128-LABEL: define{{.*}} i128 @f_scalar_5(i128 noundef %x)
-__int128_t f_scalar_5(__int128_t x) { return x; }
-#endif
-
-// CHECK-LABEL: define{{.*}} float @f_fp_scalar_1(float noundef %x)
-float f_fp_scalar_1(float x) { return x; }
-
-// CHECK-LABEL: define{{.*}} double @f_fp_scalar_2(double noundef %x)
-double f_fp_scalar_2(double x) { return x; }
-
-// Scalars larger than 2*xlen are passed/returned indirect. However, the
-// RISC-V LLVM backend can handle this fine, so the function doesn't need to
-// be modified.
-
-// CHECK-LABEL: define{{.*}} fp128 @f_fp_scalar_3(fp128 noundef %x)
-long double f_fp_scalar_3(long double x) { return x; }
-
-// Empty structs or unions are ignored.
-
-struct empty_s {};
-
-// CHECK-LABEL: define{{.*}} void @f_agg_empty_struct()
-struct empty_s f_agg_empty_struct(struct empty_s x) {
-  return x;
-}
-
-union empty_u {};
-
-// CHECK-LABEL: define{{.*}} void @f_agg_empty_union()
-union empty_u f_agg_empty_union(union empty_u x) {
-  return x;
-}
-
-// Aggregates <= 2*xlen may be passed in registers, so will be coerced to
-// integer arguments. The rules for return are the same.
-
-struct tiny {
-  uint8_t a, b, c, d;
-};
-
-// CHECK-LABEL: define{{.*}} void @f_agg_tiny(i32 %x.coerce)
-void f_agg_tiny(struct tiny x) {
-  x.a += x.b;
-  x.c += x.d;
-}
-
-// CHECK-LABEL: define{{.*}} i32 @f_agg_tiny_ret()
-struct tiny f_agg_tiny_ret(void) {
-  return (struct tiny){1, 2, 3, 4};
-}
-
-typedef uint8_t v4i8 __attribute__((vector_size(4)));
-typedef int32_t v1i32 __attribute__((vector_size(4)));
-
-// CHECK-LABEL: define{{.*}} void @f_vec_tiny_v4i8(i32 noundef %x.coerce)
-void f_vec_tiny_v4i8(v4i8 x) {
-  x[0] = x[1];
-  x[2] = x[3];
-}
-
-// CHECK-LABEL: define{{.*}} i32 @f_vec_tiny_v4i8_ret()
-v4i8 f_vec_tiny_v4i8_ret(void) {
-  return (v4i8){1, 2, 3, 4};
-}
-
-// CHECK-LABEL: define{{.*}} void @f_vec_tiny_v1i32(i32 noundef %x.coerce)
-void f_vec_tiny_v1i32(v1i32 x) {
-  x[0] = 114;
-}
-
-// CHECK-LABEL: define{{.*}} i32 @f_vec_tiny_v1i32_ret()
-v1i32 f_vec_tiny_v1i32_ret(void) {
-  return (v1i32){1};
-}
-
-struct small {
-  int32_t a, *b;
-};
-
-// CHECK-LABEL: define{{.*}} void @f_agg_small([2 x i32] %x.coerce)
-void f_agg_small(struct small x) {
-  x.a += *x.b;
-  x.b = &x.a;
-}
-
-// CHECK-LABEL: define{{.*}} [2 x i32] @f_agg_small_ret()
-struct small f_agg_small_ret(void) {
-  return (struct small){1, 0};
-}
-
-typedef uint8_t v8i8 __attribute__((vector_size(8)));
-typedef int64_t v1i64 __attribute__((vector_size(8)));
-
-// CHECK-LABEL: define{{.*}} void @f_vec_small_v8i8(i64 noundef %x.coerce)
-void f_vec_small_v8i8(v8i8 x) {
-  x[0] = x[7];
-}
-
-// CHECK-LABEL: define{{.*}} i64 @f_vec_small_v8i8_ret()
-v8i8 f_vec_small_v8i8_ret(void) {
-  return (v8i8){1, 2, 3, 4, 5, 6, 7, 8};
-}
-
-// CHECK-LABEL: define{{.*}} void @f_vec_small_v1i64(i64 noundef %x.coerce)
-void f_vec_small_v1i64(v1i64 x) {
-  x[0] = 114;
-}
-
-// CHECK-LABEL: define{{.*}} i64 @f_vec_small_v1i64_ret()
-v1i64 f_vec_small_v1i64_ret(void) {
-  return (v1i64){1};
-}
-
-// Aggregates of 2*xlen size and 2*xlen alignment should be coerced to a
-// single 2*xlen-sized argument, to ensure that alignment can be maintained if
-// passed on the stack.
-
-struct small_aligned {
-  int64_t a;
-};
-
-// CHECK-LABEL: define{{.*}} void @f_agg_small_aligned(i64 %x.coerce)
-void f_agg_small_aligned(struct small_aligned x) {
-  x.a += x.a;
-}
-
-// CHECK-LABEL: define{{.*}} i64 @f_agg_small_aligned_ret(i64 %x.coerce)
-struct small_aligned f_agg_small_aligned_ret(struct small_aligned x) {
-  return (struct small_aligned){10};
-}
-
-// Aggregates greater > 2*xlen will be passed and returned indirectly
-struct large {
-  int32_t a, b, c, d;
-};
-
-// CHECK-LABEL: define{{.*}} void @f_agg_large(ptr noundef %x)
-void f_agg_large(struct large x) {
-  x.a = x.b + x.c + x.d;
-}
-
-// The address where the struct should be written to will be the first
-// argument
-// CHECK-LABEL: define{{.*}} void @f_agg_large_ret(ptr noalias sret(%struct.large) align 4 %agg.result, i32 noundef %i, i8 noundef signext %j)
-struct large f_agg_large_ret(int32_t i, int8_t j) {
-  return (struct large){1, 2, 3, 4};
-}
-
-typedef unsigned char v16i8 __attribute__((vector_size(16)));
-
-// CHECK-LABEL: define{{.*}} void @f_vec_large_v16i8(ptr noundef %0)
-void f_vec_large_v16i8(v16i8 x) {
-  x[0] = x[7];
-}
-
-// CHECK-LABEL: define{{.*}} void @f_vec_large_v16i8_ret(ptr noalias sret(<16 x i8>) align 16 %agg.result)
-v16i8 f_vec_large_v16i8_ret(void) {
-  return (v16i8){1, 2, 3, 4, 5, 6, 7, 8};
-}
-
-// Scalars passed on the stack should have signext/zeroext attributes, just as
-// if they were passed in registers
-
-// CHECK-LABEL: define{{.*}} i32 @f_scalar_stack_1(i32 %a.coerce, [2 x i32] %b.coerce, i64 %c.coerce, ptr noundef %d, i8 noundef zeroext %e, i8 noundef signext %f, i8 noundef zeroext %g, i8 noundef signext %h)
-int f_scalar_stack_1(struct tiny a, struct small b, struct small_aligned c,
-                     struct large d, uint8_t e, int8_t f, uint8_t g, int8_t h) {
-  return g + h;
-}
-
-// Ensure that scalars passed on the stack are still determined correctly in
-// the presence of large return values that consume a register due to the need
-// to pass a pointer.
-
-// CHECK-LABEL: define{{.*}} void @f_scalar_stack_2(ptr noalias sret(%struct.large) align 4 %agg.result, i32 noundef %a, i64 noundef %b, i64 noundef %c, fp128 noundef %d, i8 noundef zeroext %e, i8 noundef signext %f, i8 noundef zeroext %g)
-struct large f_scalar_stack_2(int32_t a, int64_t b, int64_t c, long double d,
-                              uint8_t e, int8_t f, uint8_t g) {
-  return (struct large){a, e, f, g};
-}
-
-// CHECK-LABEL: define{{.*}} fp128 @f_scalar_stack_4(i32 noundef %a, i64 noundef %b, i64 noundef %c, fp128 noundef %d, i8 noundef zeroext %e, i8 noundef signext %f, i8 noundef zeroext %g)
-long double f_scalar_stack_4(int32_t a, int64_t b, int64_t c, long double d,
-                             uint8_t e, int8_t f, uint8_t g) {
-  return d;
-}
-
-// Aggregates and >=XLen scalars passed on the stack should be lowered just as
-// they would be if passed via registers.
-
-// CHECK-LABEL: define{{.*}} void @f_scalar_stack_5(double noundef %a, i64 noundef %b, double noundef %c, i64 noundef %d, i32 noundef %e, i64 noundef %f, float noundef %g, double noundef %h, fp128 noundef %i)
-void f_scalar_stack_5(double a, int64_t b, double c, int64_t d, int e,
-                      int64_t f, float g, double h, long double i) {}
-
-// CHECK-LABEL: define{{.*}} void @f_agg_stack(double noundef %a, i64 noundef %b, double noundef %c, i64 noundef %d, i32 %e.coerce, [2 x i32] %f.coerce, i64 %g.coerce, ptr noundef %h)
-void f_agg_stack(double a, int64_t b, double c, int64_t d, struct tiny e,
-                 struct small f, struct small_aligned g, struct large h) {}
-
-// Ensure that ABI lowering happens as expected for vararg calls. For RV32
-// with the base integer calling convention there will be no observable
-// differences in the lowered IR for a call with varargs vs without.
-
-int f_va_callee(int, ...);
-
-// CHECK-LABEL: define{{.*}} void @f_va_caller()
-// CHECK: call i32 (i32, ...) @f_va_callee(i32 noundef 1, i32 noundef 2, i64 noundef 3, double noundef 4.000000e+00, double noundef 5.000000e+00, i32 {{%.*}}, [2 x i32] {{%.*}}, i64 {{%.*}}, ptr noundef {{%.*}})
-void f_va_caller(void) {
-  f_va_callee(1, 2, 3LL, 4.0f, 5.0, (struct tiny){6, 7, 8, 9},
-              (struct small){10, NULL}, (struct small_aligned){11},
-              (struct large){12, 13, 14, 15});
-}
-
-// CHECK-LABEL: define{{.*}} i32 @f_va_1(ptr noundef %fmt, ...) {{.*}} {
-// CHECK:   [[FMT_ADDR:%.*]] = alloca ptr, align 4
-// CHECK:   [[VA:%.*]] = alloca ptr, align 4
-// CHECK:   [[V:%.*]] = alloca i32, align 4
-// CHECK:   store ptr %fmt, ptr [[FMT_ADDR]], align 4
-// CHECK:   call void @llvm.va_start(ptr [[VA]])
-// CHECK:   [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 4
-// CHECK:   [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR]], i32 4
-// CHECK:   store ptr [[ARGP_NEXT]], ptr [[VA]], align 4
-// CHECK:   [[TMP1:%.*]] = load i32, ptr [[ARGP_CUR]], align 4
-// CHECK:   store i32 [[TMP1]], ptr [[V]], align 4
-// CHECK:   call void @llvm.va_end(ptr [[VA]])
-// CHECK:   [[TMP2:%.*]] = load i32, ptr [[V]], align 4
-// CHECK:   ret i32 [[TMP2]]
-// CHECK: }
-int f_va_1(char *fmt, ...) {
-  __builtin_va_list va;
-
-  __builtin_va_start(va, fmt);
-  int v = __builtin_va_arg(va, int);
-  __builtin_va_end(va);
-
-  return v;
-}
-
-// An "aligned" register pair (where the first register is even-numbered) is
-// used to pass varargs with 2x xlen alignment and 2x xlen size. Ensure the
-// correct offsets are used.
-
-// CHECK-LABEL: @f_va_2(
-// CHECK:         [[FMT_ADDR:%.*]] = alloca ptr, align 4
-// CHECK-NEXT:    [[VA:%.*]] = alloca ptr, align 4
-// CHECK-NEXT:    [[V:%.*]] = alloca double, align 8
-// CHECK-NEXT:    store ptr [[FMT:%.*]], ptr [[FMT_ADDR]], align 4
-// CHECK-NEXT:    call void @llvm.va_start(ptr [[VA]])
-// CHECK-NEXT:    [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 4
-// CHECK-NEXT:    [[TMP0:%.*]] = ptrtoint ptr [[ARGP_CUR]] to i32
-// CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[TMP0]], 7
-// CHECK-NEXT:    [[TMP2:%.*]] = and i32 [[TMP1]], -8
-// CHECK-NEXT:    [[ARGP_CUR_ALIGNED:%.*]] = inttoptr i32 [[TMP2]] to ptr
-// CHECK-NEXT:    [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR_ALIGNED]], i32 8
-// CHECK-NEXT:    store ptr [[ARGP_NEXT]], ptr [[VA]], align 4
-// CHECK-NEXT:    [[TMP4:%.*]] = load double, ptr [[ARGP_CUR_ALIGNED]], align 8
-// CHECK-NEXT:    store double [[TMP4]], ptr [[V]], align 8
-// CHECK-NEXT:    call void @llvm.va_end(ptr [[VA]])
-// CHECK-NEXT:    [[TMP5:%.*]] = load double, ptr [[V]], align 8
-// CHECK-NEXT:    ret double [[TMP5]]
-double f_va_2(char *fmt, ...) {
-  __builtin_va_list va;
-
-  __builtin_va_start(va, fmt);
-  double v = __builtin_va_arg(va, double);
-  __builtin_va_end(va);
-
-  return v;
-}
-
-// Two "aligned" register pairs.
-
-// CHECK-LABEL: @f_va_3(
-// CHECK:         [[FMT_ADDR:%.*]] = alloca ptr, align 4
-// CHECK-NEXT:    [[VA:%.*]] = alloca ptr, align 4
-// CHECK-NEXT:    [[V:%.*]] = alloca double, align 8
-// CHECK-NEXT:    [[W:%.*]] = alloca i32, align 4
-// CHECK-NEXT:    [[X:%.*]] = alloca double, align 8
-// CHECK-NEXT:    store ptr [[FMT:%.*]], ptr [[FMT_ADDR]], align 4
-// CHECK-NEXT:    call void @llvm.va_start(ptr [[VA]])
-// CHECK-NEXT:    [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 4
-// CHECK-NEXT:    [[TMP0:%.*]] = ptrtoint ptr [[ARGP_CUR]] to i32
-// CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[TMP0]], 7
-// CHECK-NEXT:    [[TMP2:%.*]] = and i32 [[TMP1]], -8
-// CHECK-NEXT:    [[ARGP_CUR_ALIGNED:%.*]] = inttoptr i32 [[TMP2]] to ptr
-// CHECK-NEXT:    [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR_ALIGNED]], i32 8
-// CHECK-NEXT:    store ptr [[ARGP_NEXT]], ptr [[VA]], align 4
-// CHECK-NEXT:    [[TMP4:%.*]] = load double, ptr [[ARGP_CUR_ALIGNED]], align 8
-// CHECK-NEXT:    store double [[TMP4]], ptr [[V]], align 8
-// CHECK-NEXT:    [[ARGP_CUR2:%.*]] = load ptr, ptr [[VA]], align 4
-// CHECK-NEXT:    [[ARGP_NEXT3:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR2]], i32 4
-// CHECK-NEXT:    store ptr [[ARGP_NEXT3]], ptr [[VA]], align 4
-// CHECK-NEXT:    [[TMP6:%.*]] = load i32, ptr [[ARGP_CUR2]], align 4
-// CHECK-NEXT:    store i32 [[TMP6]], ptr [[W]], align 4
-// CHECK-NEXT:    [[ARGP_CUR4:%.*]] = load ptr, ptr [[VA]], align 4
-// CHECK-NEXT:    [[TMP7:%.*]] = ptrtoint ptr [[ARGP_CUR4]] to i32
-// CHECK-NEXT:    [[TMP8:%.*]] = add i32 [[TMP7]], 7
-// CHECK-NEXT:    [[TMP9:%.*]] = and i32 [[TMP8]], -8
-// CHECK-NEXT:    [[ARGP_CUR4_ALIGNED:%.*]] = inttoptr i32 [[TMP9]] to ptr
-// CHECK-NEXT:    [[ARGP_NEXT5:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR4_ALIGNED]], i32 8
-// CHECK-NEXT:    store ptr [[ARGP_NEXT5]], ptr [[VA]], align 4
-// CHECK-NEXT:    [[TMP11:%.*]] = load double, ptr [[ARGP_CUR4_ALIGNED]], align 8
-// CHECK-NEXT:    store double [[TMP11]], ptr [[X]], align 8
-// CHECK-NEXT:    call void @llvm.va_end(ptr [[VA]])
-// CHECK-NEXT:    [[TMP12:%.*]] = load double, ptr [[V]], align 8
-// CHECK-NEXT:    [[TMP13:%.*]] = load double, ptr [[X]], align 8
-// CHECK-NEXT:    [[ADD:%.*]] = fadd double [[TMP12]], [[TMP13]]
-// CHECK-NEXT:    ret double [[ADD]]
-double f_va_3(char *fmt, ...) {
-  __builtin_va_list va;
-
-  __builtin_va_start(va, fmt);
-  double v = __builtin_va_arg(va, double);
-  int w = __builtin_va_arg(va, int);
-  double x = __builtin_va_arg(va, double);
-  __builtin_va_end(va);
-
-  return v + x;
-}
-
-// CHECK-LABEL: define{{.*}} i32 @f_va_4(ptr noundef %fmt, ...) {{.*}} {
-// CHECK:         [[FMT_ADDR:%.*]] = alloca ptr, align 4
-// CHECK-NEXT:    [[VA:%.*]] = alloca ptr, align 4
-// CHECK-NEXT:    [[V:%.*]] = alloca i32, align 4
-// CHECK-NEXT:    [[LD:%.*]] = alloca fp128, align 16
-// CHECK-NEXT:    [[TS:%.*]] = alloca [[STRUCT_TINY:%.*]], align 1
-// CHECK-NEXT:    [[SS:%.*]] = alloca [[STRUCT_SMALL:%.*]], align 4
-// CHECK-NEXT:    [[LS:%.*]] = alloca [[STRUCT_LARGE:%.*]], align 4
-// CHECK-NEXT:    [[RET:%.*]] = alloca i32, align 4
-// CHECK-NEXT:    store ptr [[FMT:%.*]], ptr [[FMT_ADDR]], align 4
-// CHECK-NEXT:    call void @llvm.va_start(ptr [[VA]])
-// CHECK-NEXT:    [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 4
-// CHECK-NEXT:    [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR]], i32 4
-// CHECK-NEXT:    store ptr [[ARGP_NEXT]], ptr [[VA]], align 4
-// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[ARGP_CUR]], align 4
-// CHECK-NEXT:    store i32 [[TMP1]], ptr [[V]], align 4
-// CHECK-NEXT:    [[ARGP_CUR2:%.*]] = load ptr, ptr [[VA]], align 4
-// CHECK-NEXT:    [[ARGP_NEXT3:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR2]], i32 4
-// CHECK-NEXT:    store ptr [[ARGP_NEXT3]], ptr [[VA]], align 4
-// CHECK-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[ARGP_CUR2]], align 4
-// CHECK-NEXT:    [[TMP4:%.*]] = load fp128, ptr [[TMP3]], align 16
-// CHECK-NEXT:    store fp128 [[TMP4]], ptr [[LD]], align 16
-// CHECK-NEXT:    [[ARGP_CUR4:%.*]] = load ptr, ptr [[VA]], align 4
-// CHECK-NEXT:    [[ARGP_NEXT5:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR4]], i32 4
-// CHECK-NEXT:    store ptr [[ARGP_NEXT5]], ptr [[VA]], align 4
-// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 1 [[TS]], ptr align 4 [[ARGP_CUR4]], i32 4, i1 false)
-// CHECK-NEXT:    [[ARGP_CUR6:%.*]] = load ptr, ptr [[VA]], align 4
-// CHECK-NEXT:    [[ARGP_NEXT7:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR6]], i32 8
-// CHECK-NEXT:    store ptr [[ARGP_NEXT7]], ptr [[VA]], align 4
-// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[SS]], ptr align 4 [[ARGP_CUR6]], i32 8, i1 false)
-// CHECK-NEXT:    [[ARGP_CUR8:%.*]] = load ptr, ptr [[VA]], align 4
-// CHECK-NEXT:    [[ARGP_NEXT9:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR8]], i32 4
-// CHECK-NEXT:    store ptr [[ARGP_NEXT9]], ptr [[VA]], align 4
-// CHECK-NEXT:    [[TMP12:%.*]] = load ptr, ptr [[ARGP_CUR8]], align 4
-// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[LS]], ptr align 4 [[TMP12]], i32 16, i1 false)
-// CHECK-NEXT:    call void @llvm.va_end(ptr [[VA]])
-int f_va_4(char *fmt, ...) {
-  __builtin_va_list va;
-
-  __builtin_va_start(va, fmt);
-  int v = __builtin_va_arg(va, int);
-  long double ld = __builtin_va_arg(va, long double);
-  struct tiny ts = __builtin_va_arg(va, struct tiny);
-  struct small ss = __builtin_va_arg(va, struct small);
-  struct large ls = __builtin_va_arg(va, struct large);
-  __builtin_va_end(va);
-
-  int ret = (int)((long double)v + ld);
-  ret = ret + ts.a + ts.b + ts.c + ts.d;
-  ret = ret + ss.a + (int)ss.b;
-  ret = ret + ls.a + ls.b + ls.c + ls.d;
-
-  return ret;
-}
diff --git a/clang/test/CodeGen/RISCV/riscv32-ilp32d-abi.c b/clang/test/CodeGen/RISCV/riscv32-ilp32d-abi.c
deleted file mode 100644
index 65537b57fd508..0000000000000
--- a/clang/test/CodeGen/RISCV/riscv32-ilp32d-abi.c
+++ /dev/null
@@ -1,306 +0,0 @@
-// RUN: %clang_cc1 -triple riscv32 -target-feature +d -target-feature +f -target-abi ilp32d -emit-llvm %s -o - \
-// RUN:     | FileCheck %s
-
-#include <stdint.h>
-
-// Verify that the tracking of used GPRs and FPRs works correctly by checking
-// that small integers are sign/zero extended when passed in registers.
-
-// Doubles are passed in FPRs, so argument 'i' will be passed zero-extended
-// because it will be passed in a GPR.
-
-// CHECK: define{{.*}} void @f_fpr_tracking(double noundef %a, double noundef %b, double noundef %c, double noundef %d, double noundef %e, double noundef %f, double noundef %g, double noundef %h, i8 noundef zeroext %i)
-void f_fpr_tracking(double a, double b, double c, double d, double e, double f,
-                    double g, double h, uint8_t i) {}
-
-// Check that fp, fp+fp, and int+fp structs are lowered correctly. These will
-// be passed in FPR, FPR+FPR, or GPR+FPR regs if sufficient registers are
-// available the widths are <= XLEN and FLEN, and should be expanded to
-// separate arguments in IR. They are passed by the same rules for returns,
-// but will be lowered to simple two-element structs if necessary (as LLVM IR
-// functions cannot return multiple values).
-
-// A struct containing just one floating-point real is passed as though it
-// were a standalone floating-point real.
-
-struct double_s { double f; };
-
-// CHECK: define{{.*}} void @f_double_s_arg(double %0)
-void f_double_s_arg(struct double_s a) {}
-
-// CHECK: define{{.*}} double @f_ret_double_s()
-struct double_s f_ret_double_s(void) {
-  return (struct double_s){1.0};
-}
-
-// A struct containing a double and any number of zero-width bitfields is
-// passed as though it were a standalone floating-point real.
-
-struct zbf_double_s { int : 0; double f; };
-struct zbf_double_zbf_s { int : 0; double f; int : 0; };
-
-// CHECK: define{{.*}} void @f_zbf_double_s_arg(double %0)
-void f_zbf_double_s_arg(struct zbf_double_s a) {}
-
-// CHECK: define{{.*}} double @f_ret_zbf_double_s()
-struct zbf_double_s f_ret_zbf_double_s(void) {
-  return (struct zbf_double_s){1.0};
-}
-
-// CHECK: define{{.*}} void @f_zbf_double_zbf_s_arg(double %0)
-void f_zbf_double_zbf_s_arg(struct zbf_double_zbf_s a) {}
-
-// CHECK: define{{.*}} double @f_ret_zbf_double_zbf_s()
-struct zbf_double_zbf_s f_ret_zbf_double_zbf_s(void) {
-  return (struct zbf_double_zbf_s){1.0};
-}
-
-// Check that structs containing two floating point values (FLEN <= width) are
-// expanded provided sufficient FPRs are available.
-
-struct double_double_s { double f; double g; };
-struct double_float_s { double f; float g; };
-
-// CHECK: define{{.*}} void @f_double_double_s_arg(double %0, double %1)
-void f_double_double_s_arg(struct double_double_s a) {}
-
-// CHECK: define{{.*}} { double, double } @f_ret_double_double_s()
-struct double_double_s f_ret_double_double_s(void) {
-  return (struct double_double_s){1.0, 2.0};
-}
-
-// CHECK: define{{.*}} void @f_double_float_s_arg(double %0, float %1)
-void f_double_float_s_arg(struct double_float_s a) {}
-
-// CHECK: define{{.*}} { double, float } @f_ret_double_float_s()
-struct double_float_s f_ret_double_float_s(void) {
-  return (struct double_float_s){1.0, 2.0};
-}
-
-// CHECK: define{{.*}} void @f_double_double_s_arg_insufficient_fprs(float noundef %a, double noundef %b, double noundef %c, double noundef %d, double noundef %e, double noundef %f, double noundef %g, ptr noundef %h)
-void f_double_double_s_arg_insufficient_fprs(float a, double b, double c, double d,
-    double e, double f, double g, struct double_double_s h) {}
-
-// Check that structs containing int+double values are expanded, provided
-// sufficient FPRs and GPRs are available. The integer components are neither
-// sign or zero-extended.
-
-struct double_int8_s { double f; int8_t i; };
-struct double_uint8_s { double f; uint8_t i; };
-struct double_int32_s { double f; int32_t i; };
-struct double_int64_s { double f; int64_t i; };
-struct double_int64bf_s { double f; int64_t i : 32; };
-struct double_int8_zbf_s { double f; int8_t i; int : 0; };
-
-// CHECK: define{{.*}} void @f_double_int8_s_arg(double %0, i8 %1)
-void f_double_int8_s_arg(struct double_int8_s a) {}
-
-// CHECK: define{{.*}} { double, i8 } @f_ret_double_int8_s()
-struct double_int8_s f_ret_double_int8_s(void) {
-  return (struct double_int8_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_double_uint8_s_arg(double %0, i8 %1)
-void f_double_uint8_s_arg(struct double_uint8_s a) {}
-
-// CHECK: define{{.*}} { double, i8 } @f_ret_double_uint8_s()
-struct double_uint8_s f_ret_double_uint8_s(void) {
-  return (struct double_uint8_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_double_int32_s_arg(double %0, i32 %1)
-void f_double_int32_s_arg(struct double_int32_s a) {}
-
-// CHECK: define{{.*}} { double, i32 } @f_ret_double_int32_s()
-struct double_int32_s f_ret_double_int32_s(void) {
-  return (struct double_int32_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_double_int64_s_arg(ptr noundef %a)
-void f_double_int64_s_arg(struct double_int64_s a) {}
-
-// CHECK: define{{.*}} void @f_ret_double_int64_s(ptr noalias sret(%struct.double_int64_s) align 8 %agg.result)
-struct double_int64_s f_ret_double_int64_s(void) {
-  return (struct double_int64_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_double_int64bf_s_arg(double %0, i32 %1)
-void f_double_int64bf_s_arg(struct double_int64bf_s a) {}
-
-// CHECK: define{{.*}} { double, i32 } @f_ret_double_int64bf_s()
-struct double_int64bf_s f_ret_double_int64bf_s(void) {
-  return (struct double_int64bf_s){1.0, 2};
-}
-
-// The zero-width bitfield means the struct can't be passed according to the
-// floating point calling convention.
-
-// CHECK: define{{.*}} void @f_double_int8_zbf_s(double %0, i8 %1)
-void f_double_int8_zbf_s(struct double_int8_zbf_s a) {}
-
-// CHECK: define{{.*}} { double, i8 } @f_ret_double_int8_zbf_s()
-struct double_int8_zbf_s f_ret_double_int8_zbf_s(void) {
-  return (struct double_int8_zbf_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_double_int8_s_arg_insufficient_gprs(i32 noundef %a, i32 noundef %b, i32 noundef %c, i32 noundef %d, i32 noundef %e, i32 noundef %f, i32 noundef %g, i32 noundef %h, ptr noundef %i)
-void f_double_int8_s_arg_insufficient_gprs(int a, int b, int c, int d, int e,
-                                          int f, int g, int h, struct double_int8_s i) {}
-
-// CHECK: define{{.*}} void @f_struct_double_int8_insufficient_fprs(float noundef %a, double noundef %b, double noundef %c, double noundef %d, double noundef %e, double noundef %f, double noundef %g, double noundef %h, ptr noundef %i)
-void f_struct_double_int8_insufficient_fprs(float a, double b, double c, double d,
-                                           double e, double f, double g, double h, struct double_int8_s i) {}
-
-// Complex floating-point values or structs containing a single complex
-// floating-point value should be passed as if it were an fp+fp struct.
-
-// CHECK: define{{.*}} void @f_doublecomplex(double noundef %a.coerce0, double noundef %a.coerce1)
-void f_doublecomplex(double __complex__ a) {}
-
-// CHECK: define{{.*}} { double, double } @f_ret_doublecomplex()
-double __complex__ f_ret_doublecomplex(void) {
-  return 1.0;
-}
-
-struct doublecomplex_s { double __complex__ c; };
-
-// CHECK: define{{.*}} void @f_doublecomplex_s_arg(double %0, double %1)
-void f_doublecomplex_s_arg(struct doublecomplex_s a) {}
-
-// CHECK: define{{.*}} { double, double } @f_ret_doublecomplex_s()
-struct doublecomplex_s f_ret_doublecomplex_s(void) {
-  return (struct doublecomplex_s){1.0};
-}
-
-// Test single or two-element structs that need flattening. e.g. those
-// containing nested structs, doubles in small arrays, zero-length structs etc.
-
-struct doublearr1_s { double a[1]; };
-
-// CHECK: define{{.*}} void @f_doublearr1_s_arg(double %0)
-void f_doublearr1_s_arg(struct doublearr1_s a) {}
-
-// CHECK: define{{.*}} double @f_ret_doublearr1_s()
-struct doublearr1_s f_ret_doublearr1_s(void) {
-  return (struct doublearr1_s){{1.0}};
-}
-
-struct doublearr2_s { double a[2]; };
-
-// CHECK: define{{.*}} void @f_doublearr2_s_arg(double %0, double %1)
-void f_doublearr2_s_arg(struct doublearr2_s a) {}
-
-// CHECK: define{{.*}} { double, double } @f_ret_doublearr2_s()
-struct doublearr2_s f_ret_doublearr2_s(void) {
-  return (struct doublearr2_s){{1.0, 2.0}};
-}
-
-struct doublearr2_tricky1_s { struct { double f[1]; } g[2]; };
-
-// CHECK: define{{.*}} void @f_doublearr2_tricky1_s_arg(double %0, double %1)
-void f_doublearr2_tricky1_s_arg(struct doublearr2_tricky1_s a) {}
-
-// CHECK: define{{.*}} { double, double } @f_ret_doublearr2_tricky1_s()
-struct doublearr2_tricky1_s f_ret_doublearr2_tricky1_s(void) {
-  return (struct doublearr2_tricky1_s){{{{1.0}}, {{2.0}}}};
-}
-
-struct doublearr2_tricky2_s { struct {}; struct { double f[1]; } g[2]; };
-
-// CHECK: define{{.*}} void @f_doublearr2_tricky2_s_arg(double %0, double %1)
-void f_doublearr2_tricky2_s_arg(struct doublearr2_tricky2_s a) {}
-
-// CHECK: define{{.*}} { double, double } @f_ret_doublearr2_tricky2_s()
-struct doublearr2_tricky2_s f_ret_doublearr2_tricky2_s(void) {
-  return (struct doublearr2_tricky2_s){{}, {{{1.0}}, {{2.0}}}};
-}
-
-struct doublearr2_tricky3_s { union {}; struct { double f[1]; } g[2]; };
-
-// CHECK: define{{.*}} void @f_doublearr2_tricky3_s_arg(double %0, double %1)
-void f_doublearr2_tricky3_s_arg(struct doublearr2_tricky3_s a) {}
-
-// CHECK: define{{.*}} { double, double } @f_ret_doublearr2_tricky3_s()
-struct doublearr2_tricky3_s f_ret_doublearr2_tricky3_s(void) {
-  return (struct doublearr2_tricky3_s){{}, {{{1.0}}, {{2.0}}}};
-}
-
-struct doublearr2_tricky4_s { union {}; struct { struct {}; double f[1]; } g[2]; };
-
-// CHECK: define{{.*}} void @f_doublearr2_tricky4_s_arg(double %0, double %1)
-void f_doublearr2_tricky4_s_arg(struct doublearr2_tricky4_s a) {}
-
-// CHECK: define{{.*}} { double, double } @f_ret_doublearr2_tricky4_s()
-struct doublearr2_tricky4_s f_ret_doublearr2_tricky4_s(void) {
-  return (struct doublearr2_tricky4_s){{}, {{{}, {1.0}}, {{}, {2.0}}}};
-}
-
-// Test structs that should be passed according to the normal integer calling
-// convention.
-
-struct int_double_int_s { int a; double b; int c; };
-
-// CHECK: define{{.*}} void @f_int_double_int_s_arg(ptr noundef %a)
-void f_int_double_int_s_arg(struct int_double_int_s a) {}
-
-// CHECK: define{{.*}} void @f_ret_int_double_int_s(ptr noalias sret(%struct.int_double_int_s) align 8 %agg.result)
-struct int_double_int_s f_ret_int_double_int_s(void) {
-  return (struct int_double_int_s){1, 2.0, 3};
-}
-
-struct int64_double_s { int64_t a; double b; };
-
-// CHECK: define{{.*}} void @f_int64_double_s_arg(ptr noundef %a)
-void f_int64_double_s_arg(struct int64_double_s a) {}
-
-// CHECK: define{{.*}} void @f_ret_int64_double_s(ptr noalias sret(%struct.int64_double_s) align 8 %agg.result)
-struct int64_double_s f_ret_int64_double_s(void) {
-  return (struct int64_double_s){1, 2.0};
-}
-
-struct char_char_double_s { char a; char b; double c; };
-
-// CHECK-LABEL: define{{.*}} void @f_char_char_double_s_arg(ptr noundef %a)
-void f_char_char_double_s_arg(struct char_char_double_s a) {}
-
-// CHECK: define{{.*}} void @f_ret_char_char_double_s(ptr noalias sret(%struct.char_char_double_s) align 8 %agg.result)
-struct char_char_double_s f_ret_char_char_double_s(void) {
-  return (struct char_char_double_s){1, 2, 3.0};
-}
-
-// Unions are always passed according to the integer calling convention, even
-// if they can only contain a double.
-
-union double_u { double a; };
-
-// CHECK: define{{.*}} void @f_double_u_arg(i64 %a.coerce)
-void f_double_u_arg(union double_u a) {}
-
-// CHECK: define{{.*}} i64 @f_ret_double_u()
-union double_u f_ret_double_u(void) {
-  return (union double_u){1.0};
-}
-
-// Test that we don't incorrectly think double+int/double+double structs will
-// be returned indirectly and thus have an off-by-one error for the number of
-// GPRs available (this is an edge case when structs > 2*XLEN are still
-// returned in registers). This includes complex doubles, which are treated as
-// double+double structs by the ABI.
-
-// CHECK: define{{.*}} { double, i32 } @f_ret_double_int32_s_double_int32_s_just_sufficient_gprs(i32 noundef %a, i32 noundef %b, i32 noundef %c, i32 noundef %d, i32 noundef %e, i32 noundef %f, i32 noundef %g, double %0, i32 %1)
-struct double_int32_s f_ret_double_int32_s_double_int32_s_just_sufficient_gprs(
-    int a, int b, int c, int d, int e, int f, int g, struct double_int32_s h) {
-  return (struct double_int32_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} { double, double } @f_ret_double_double_s_double_int32_s_just_sufficient_gprs(i32 noundef %a, i32 noundef %b, i32 noundef %c, i32 noundef %d, i32 noundef %e, i32 noundef %f, i32 noundef %g, double %0, i32 %1)
-struct double_double_s f_ret_double_double_s_double_int32_s_just_sufficient_gprs(
-    int a, int b, int c, int d, int e, int f, int g, struct double_int32_s h) {
-  return (struct double_double_s){1.0, 2.0};
-}
-
-// CHECK: define{{.*}} { double, double } @f_ret_doublecomplex_double_int32_s_just_sufficient_gprs(i32 noundef %a, i32 noundef %b, i32 noundef %c, i32 noundef %d, i32 noundef %e, i32 noundef %f, i32 noundef %g, double %0, i32 %1)
-double __complex__ f_ret_doublecomplex_double_int32_s_just_sufficient_gprs(
-    int a, int b, int c, int d, int e, int f, int g, struct double_int32_s h) {
-  return 1.0;
-}
diff --git a/clang/test/CodeGen/RISCV/riscv32-ilp32f-abi.c b/clang/test/CodeGen/RISCV/riscv32-ilp32f-abi.c
deleted file mode 100644
index b778afb74a1df..0000000000000
--- a/clang/test/CodeGen/RISCV/riscv32-ilp32f-abi.c
+++ /dev/null
@@ -1,42 +0,0 @@
-// RUN: %clang_cc1 -triple riscv32 -target-feature +f -target-abi ilp32f -emit-llvm %s -o - \
-// RUN:     | FileCheck %s
-
-#include <stdint.h>
-
-// CHECK: define{{.*}} void @f_fpr_tracking(double noundef %a, double noundef %b, double noundef %c, double noundef %d, i8 noundef signext %e)
-void f_fpr_tracking(double a, double b, double c, double d, int8_t e) {}
-
-// Lowering for doubles is unnmodified, as 64 > FLEN.
-
-struct double_s { double d; };
-
-// CHECK: define{{.*}} void @f_double_s_arg(i64 %a.coerce)
-void f_double_s_arg(struct double_s a) {}
-
-// CHECK: define{{.*}} i64 @f_ret_double_s()
-struct double_s f_ret_double_s(void) {
-  return (struct double_s){1.0};
-}
-
-struct double_double_s { double d; double e; };
-
-// CHECK: define{{.*}} void @f_double_double_s_arg(ptr noundef %a)
-void f_double_double_s_arg(struct double_double_s a) {}
-
-// CHECK: define{{.*}} void @f_ret_double_double_s(ptr noalias sret(%struct.double_double_s) align 8 %agg.result)
-struct double_double_s f_ret_double_double_s(void) {
-  return (struct double_double_s){1.0, 2.0};
-}
-
-struct double_int8_s { double d; int64_t i; };
-
-struct int_double_s { int a; double b; };
-
-// CHECK: define{{.*}} void @f_int_double_s_arg(ptr noundef %a)
-void f_int_double_s_arg(struct int_double_s a) {}
-
-// CHECK: define{{.*}} void @f_ret_int_double_s(ptr noalias sret(%struct.int_double_s) align 8 %agg.result)
-struct int_double_s f_ret_int_double_s(void) {
-  return (struct int_double_s){1, 2.0};
-}
-
diff --git a/clang/test/CodeGen/RISCV/riscv32-ilp32f-ilp32d-abi.c b/clang/test/CodeGen/RISCV/riscv32-ilp32f-ilp32d-abi.c
deleted file mode 100644
index ca5d6df6c6ff7..0000000000000
--- a/clang/test/CodeGen/RISCV/riscv32-ilp32f-ilp32d-abi.c
+++ /dev/null
@@ -1,275 +0,0 @@
-// RUN: %clang_cc1 -triple riscv32 -target-feature +f -target-abi ilp32f -emit-llvm %s -o - \
-// RUN:     | FileCheck %s
-// RUN: %clang_cc1 -triple riscv32 -target-feature +d -target-feature +f -target-abi ilp32d -emit-llvm %s -o - \
-// RUN:     | FileCheck %s
-
-#include <stdint.h>
-
-// Verify that the tracking of used GPRs and FPRs works correctly by checking
-// that small integers are sign/zero extended when passed in registers.
-
-// Floats are passed in FPRs, so argument 'i' will be passed zero-extended
-// because it will be passed in a GPR.
-
-// CHECK: define{{.*}} void @f_fpr_tracking(float noundef %a, float noundef %b, float noundef %c, float noundef %d, float noundef %e, float noundef %f, float noundef %g, float noundef %h, i8 noundef zeroext %i)
-void f_fpr_tracking(float a, float b, float c, float d, float e, float f,
-                    float g, float h, uint8_t i) {}
-
-// Check that fp, fp+fp, and int+fp structs are lowered correctly. These will
-// be passed in FPR, FPR+FPR, or GPR+FPR regs if sufficient registers are
-// available the widths are <= XLEN and FLEN, and should be expanded to
-// separate arguments in IR. They are passed by the same rules for returns,
-// but will be lowered to simple two-element structs if necessary (as LLVM IR
-// functions cannot return multiple values).
-
-// A struct containing just one floating-point real is passed as though it
-// were a standalone floating-point real.
-
-struct float_s { float f; };
-
-// CHECK: define{{.*}} void @f_float_s_arg(float %0)
-void f_float_s_arg(struct float_s a) {}
-
-// CHECK: define{{.*}} float @f_ret_float_s()
-struct float_s f_ret_float_s(void) {
-  return (struct float_s){1.0};
-}
-
-// A struct containing a float and any number of zero-width bitfields is
-// passed as though it were a standalone floating-point real.
-
-struct zbf_float_s { int : 0; float f; };
-struct zbf_float_zbf_s { int : 0; float f; int : 0; };
-
-// CHECK: define{{.*}} void @f_zbf_float_s_arg(float %0)
-void f_zbf_float_s_arg(struct zbf_float_s a) {}
-
-// CHECK: define{{.*}} float @f_ret_zbf_float_s()
-struct zbf_float_s f_ret_zbf_float_s(void) {
-  return (struct zbf_float_s){1.0};
-}
-
-// CHECK: define{{.*}} void @f_zbf_float_zbf_s_arg(float %0)
-void f_zbf_float_zbf_s_arg(struct zbf_float_zbf_s a) {}
-
-// CHECK: define{{.*}} float @f_ret_zbf_float_zbf_s()
-struct zbf_float_zbf_s f_ret_zbf_float_zbf_s(void) {
-  return (struct zbf_float_zbf_s){1.0};
-}
-
-// Check that structs containing two float values (FLEN <= width) are expanded
-// provided sufficient FPRs are available.
-
-struct float_float_s { float f; float g; };
-
-// CHECK: define{{.*}} void @f_float_float_s_arg(float %0, float %1)
-void f_float_float_s_arg(struct float_float_s a) {}
-
-// CHECK: define{{.*}} { float, float } @f_ret_float_float_s()
-struct float_float_s f_ret_float_float_s(void) {
-  return (struct float_float_s){1.0, 2.0};
-}
-
-// CHECK: define{{.*}} void @f_float_float_s_arg_insufficient_fprs(float noundef %a, float noundef %b, float noundef %c, float noundef %d, float noundef %e, float noundef %f, float noundef %g, [2 x i32] %h.coerce)
-void f_float_float_s_arg_insufficient_fprs(float a, float b, float c, float d,
-    float e, float f, float g, struct float_float_s h) {}
-
-// Check that structs containing int+float values are expanded, provided
-// sufficient FPRs and GPRs are available. The integer components are neither
-// sign or zero-extended.
-
-struct float_int8_s { float f; int8_t i; };
-struct float_uint8_s { float f; uint8_t i; };
-struct float_int32_s { float f; int32_t i; };
-struct float_int64_s { float f; int64_t i; };
-struct float_int64bf_s { float f; int64_t i : 32; };
-struct float_int8_zbf_s { float f; int8_t i; int : 0; };
-
-// CHECK: define{{.*}} void @f_float_int8_s_arg(float %0, i8 %1)
-void f_float_int8_s_arg(struct float_int8_s a) {}
-
-// CHECK: define{{.*}} { float, i8 } @f_ret_float_int8_s()
-struct float_int8_s f_ret_float_int8_s(void) {
-  return (struct float_int8_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_float_uint8_s_arg(float %0, i8 %1)
-void f_float_uint8_s_arg(struct float_uint8_s a) {}
-
-// CHECK: define{{.*}} { float, i8 } @f_ret_float_uint8_s()
-struct float_uint8_s f_ret_float_uint8_s(void) {
-  return (struct float_uint8_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_float_int32_s_arg(float %0, i32 %1)
-void f_float_int32_s_arg(struct float_int32_s a) {}
-
-// CHECK: define{{.*}} { float, i32 } @f_ret_float_int32_s()
-struct float_int32_s f_ret_float_int32_s(void) {
-  return (struct float_int32_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_float_int64_s_arg(ptr noundef %a)
-void f_float_int64_s_arg(struct float_int64_s a) {}
-
-// CHECK: define{{.*}} void @f_ret_float_int64_s(ptr noalias sret(%struct.float_int64_s) align 8 %agg.result)
-struct float_int64_s f_ret_float_int64_s(void) {
-  return (struct float_int64_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_float_int64bf_s_arg(float %0, i32 %1)
-void f_float_int64bf_s_arg(struct float_int64bf_s a) {}
-
-// CHECK: define{{.*}} { float, i32 } @f_ret_float_int64bf_s()
-struct float_int64bf_s f_ret_float_int64bf_s(void) {
-  return (struct float_int64bf_s){1.0, 2};
-}
-
-// The zero-width bitfield means the struct can't be passed according to the
-// floating point calling convention.
-
-// CHECK: define{{.*}} void @f_float_int8_zbf_s(float %0, i8 %1)
-void f_float_int8_zbf_s(struct float_int8_zbf_s a) {}
-
-// CHECK: define{{.*}} { float, i8 } @f_ret_float_int8_zbf_s()
-struct float_int8_zbf_s f_ret_float_int8_zbf_s(void) {
-  return (struct float_int8_zbf_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_float_int8_s_arg_insufficient_gprs(i32 noundef %a, i32 noundef %b, i32 noundef %c, i32 noundef %d, i32 noundef %e, i32 noundef %f, i32 noundef %g, i32 noundef %h, [2 x i32] %i.coerce)
-void f_float_int8_s_arg_insufficient_gprs(int a, int b, int c, int d, int e,
-                                          int f, int g, int h, struct float_int8_s i) {}
-
-// CHECK: define{{.*}} void @f_struct_float_int8_insufficient_fprs(float noundef %a, float noundef %b, float noundef %c, float noundef %d, float noundef %e, float noundef %f, float noundef %g, float noundef %h, [2 x i32] %i.coerce)
-void f_struct_float_int8_insufficient_fprs(float a, float b, float c, float d,
-                                           float e, float f, float g, float h, struct float_int8_s i) {}
-
-// Complex floating-point values or structs containing a single complex
-// floating-point value should be passed as if it were an fp+fp struct.
-
-// CHECK: define{{.*}} void @f_floatcomplex(float noundef %a.coerce0, float noundef %a.coerce1)
-void f_floatcomplex(float __complex__ a) {}
-
-// CHECK: define{{.*}} { float, float } @f_ret_floatcomplex()
-float __complex__ f_ret_floatcomplex(void) {
-  return 1.0;
-}
-
-struct floatcomplex_s { float __complex__ c; };
-
-// CHECK: define{{.*}} void @f_floatcomplex_s_arg(float %0, float %1)
-void f_floatcomplex_s_arg(struct floatcomplex_s a) {}
-
-// CHECK: define{{.*}} { float, float } @f_ret_floatcomplex_s()
-struct floatcomplex_s f_ret_floatcomplex_s(void) {
-  return (struct floatcomplex_s){1.0};
-}
-
-// Test single or two-element structs that need flattening. e.g. those
-// containing nested structs, floats in small arrays, zero-length structs etc.
-
-struct floatarr1_s { float a[1]; };
-
-// CHECK: define{{.*}} void @f_floatarr1_s_arg(float %0)
-void f_floatarr1_s_arg(struct floatarr1_s a) {}
-
-// CHECK: define{{.*}} float @f_ret_floatarr1_s()
-struct floatarr1_s f_ret_floatarr1_s(void) {
-  return (struct floatarr1_s){{1.0}};
-}
-
-struct floatarr2_s { float a[2]; };
-
-// CHECK: define{{.*}} void @f_floatarr2_s_arg(float %0, float %1)
-void f_floatarr2_s_arg(struct floatarr2_s a) {}
-
-// CHECK: define{{.*}} { float, float } @f_ret_floatarr2_s()
-struct floatarr2_s f_ret_floatarr2_s(void) {
-  return (struct floatarr2_s){{1.0, 2.0}};
-}
-
-struct floatarr2_tricky1_s { struct { float f[1]; } g[2]; };
-
-// CHECK: define{{.*}} void @f_floatarr2_tricky1_s_arg(float %0, float %1)
-void f_floatarr2_tricky1_s_arg(struct floatarr2_tricky1_s a) {}
-
-// CHECK: define{{.*}} { float, float } @f_ret_floatarr2_tricky1_s()
-struct floatarr2_tricky1_s f_ret_floatarr2_tricky1_s(void) {
-  return (struct floatarr2_tricky1_s){{{{1.0}}, {{2.0}}}};
-}
-
-struct floatarr2_tricky2_s { struct {}; struct { float f[1]; } g[2]; };
-
-// CHECK: define{{.*}} void @f_floatarr2_tricky2_s_arg(float %0, float %1)
-void f_floatarr2_tricky2_s_arg(struct floatarr2_tricky2_s a) {}
-
-// CHECK: define{{.*}} { float, float } @f_ret_floatarr2_tricky2_s()
-struct floatarr2_tricky2_s f_ret_floatarr2_tricky2_s(void) {
-  return (struct floatarr2_tricky2_s){{}, {{{1.0}}, {{2.0}}}};
-}
-
-struct floatarr2_tricky3_s { union {}; struct { float f[1]; } g[2]; };
-
-// CHECK: define{{.*}} void @f_floatarr2_tricky3_s_arg(float %0, float %1)
-void f_floatarr2_tricky3_s_arg(struct floatarr2_tricky3_s a) {}
-
-// CHECK: define{{.*}} { float, float } @f_ret_floatarr2_tricky3_s()
-struct floatarr2_tricky3_s f_ret_floatarr2_tricky3_s(void) {
-  return (struct floatarr2_tricky3_s){{}, {{{1.0}}, {{2.0}}}};
-}
-
-struct floatarr2_tricky4_s { union {}; struct { struct {}; float f[1]; } g[2]; };
-
-// CHECK: define{{.*}} void @f_floatarr2_tricky4_s_arg(float %0, float %1)
-void f_floatarr2_tricky4_s_arg(struct floatarr2_tricky4_s a) {}
-
-// CHECK: define{{.*}} { float, float } @f_ret_floatarr2_tricky4_s()
-struct floatarr2_tricky4_s f_ret_floatarr2_tricky4_s(void) {
-  return (struct floatarr2_tricky4_s){{}, {{{}, {1.0}}, {{}, {2.0}}}};
-}
-
-// Test structs that should be passed according to the normal integer calling
-// convention.
-
-struct int_float_int_s { int a; float b; int c; };
-
-// CHECK: define{{.*}} void @f_int_float_int_s_arg(ptr noundef %a)
-void f_int_float_int_s_arg(struct int_float_int_s a) {}
-
-// CHECK: define{{.*}} void @f_ret_int_float_int_s(ptr noalias sret(%struct.int_float_int_s) align 4 %agg.result)
-struct int_float_int_s f_ret_int_float_int_s(void) {
-  return (struct int_float_int_s){1, 2.0, 3};
-}
-
-struct int64_float_s { int64_t a; float b; };
-
-// CHECK: define{{.*}} void @f_int64_float_s_arg(ptr noundef %a)
-void f_int64_float_s_arg(struct int64_float_s a) {}
-
-// CHECK: define{{.*}} void @f_ret_int64_float_s(ptr noalias sret(%struct.int64_float_s) align 8 %agg.result)
-struct int64_float_s f_ret_int64_float_s(void) {
-  return (struct int64_float_s){1, 2.0};
-}
-
-struct char_char_float_s { char a; char b; float c; };
-
-// CHECK-LABEL: define{{.*}} void @f_char_char_float_s_arg([2 x i32] %a.coerce)
-void f_char_char_float_s_arg(struct char_char_float_s a) {}
-
-// CHECK: define{{.*}} [2 x i32] @f_ret_char_char_float_s()
-struct char_char_float_s f_ret_char_char_float_s(void) {
-  return (struct char_char_float_s){1, 2, 3.0};
-}
-
-// Unions are always passed according to the integer calling convention, even
-// if they can only contain a float.
-
-union float_u { float a; };
-
-// CHECK: define{{.*}} void @f_float_u_arg(i32 %a.coerce)
-void f_float_u_arg(union float_u a) {}
-
-// CHECK: define{{.*}} i32 @f_ret_float_u()
-union float_u f_ret_float_u(void) {
-  return (union float_u){1.0};
-}
diff --git a/clang/test/CodeGen/RISCV/riscv32-int128-abi.c b/clang/test/CodeGen/RISCV/riscv32-int128-abi.c
new file mode 100644
index 0000000000000..b34f7ade60db9
--- /dev/null
+++ b/clang/test/CodeGen/RISCV/riscv32-int128-abi.c
@@ -0,0 +1,9 @@
+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --filter "^define |^entry:" --version 2
+// RUN: %clang_cc1 -triple riscv32 -emit-llvm -fforce-enable-int128 %s -o - \
+// RUN:   | FileCheck %s
+
+// CHECK-LABEL: define dso_local i128 @f_int128
+// CHECK-SAME: (i128 noundef [[X:%.*]]) #[[ATTR0:[0-9]+]] {
+// CHECK:  entry:
+//
+__int128_t f_int128(__int128_t x) { return x; }
diff --git a/clang/test/CodeGen/RISCV/riscv32-vararg.c b/clang/test/CodeGen/RISCV/riscv32-vararg.c
new file mode 100644
index 0000000000000..02b1ed38e2655
--- /dev/null
+++ b/clang/test/CodeGen/RISCV/riscv32-vararg.c
@@ -0,0 +1,288 @@
+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --version 2
+// RUN: %clang_cc1 -triple riscv32 -emit-llvm %s -o - | FileCheck %s
+// RUN: %clang_cc1 -triple riscv32 -target-feature +f -target-abi ilp32f -emit-llvm %s -o - \
+// RUN:     | FileCheck %s
+// RUN: %clang_cc1 -triple riscv32 -target-feature +d -target-feature +f -target-abi ilp32d -emit-llvm %s -o - \
+// RUN:     | FileCheck %s
+
+#include <stddef.h>
+#include <stdint.h>
+
+struct tiny {
+  uint8_t a, b, c, d;
+};
+struct small {
+  int32_t a, *b;
+};
+struct small_aligned {
+  int64_t a;
+};
+struct large {
+  int32_t a, b, c, d;
+};
+
+// Ensure that ABI lowering happens as expected for vararg calls. For RV32
+// with the base integer calling convention there will be no observable
+// differences in the lowered IR for a call with varargs vs without.
+
+int f_va_callee(int, ...);
+
+// CHECK-LABEL: define dso_local void @f_va_caller
+// CHECK-SAME: () #[[ATTR0:[0-9]+]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DOTCOMPOUNDLITERAL:%.*]] = alloca [[STRUCT_TINY:%.*]], align 1
+// CHECK-NEXT:    [[DOTCOMPOUNDLITERAL1:%.*]] = alloca [[STRUCT_SMALL:%.*]], align 4
+// CHECK-NEXT:    [[DOTCOMPOUNDLITERAL4:%.*]] = alloca [[STRUCT_SMALL_ALIGNED:%.*]], align 8
+// CHECK-NEXT:    [[DOTCOMPOUNDLITERAL6:%.*]] = alloca [[STRUCT_LARGE:%.*]], align 4
+// CHECK-NEXT:    [[BYVAL_TEMP:%.*]] = alloca [[STRUCT_LARGE]], align 4
+// CHECK-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_TINY]], ptr [[DOTCOMPOUNDLITERAL]], i32 0, i32 0
+// CHECK-NEXT:    store i8 6, ptr [[A]], align 1
+// CHECK-NEXT:    [[B:%.*]] = getelementptr inbounds [[STRUCT_TINY]], ptr [[DOTCOMPOUNDLITERAL]], i32 0, i32 1
+// CHECK-NEXT:    store i8 7, ptr [[B]], align 1
+// CHECK-NEXT:    [[C:%.*]] = getelementptr inbounds [[STRUCT_TINY]], ptr [[DOTCOMPOUNDLITERAL]], i32 0, i32 2
+// CHECK-NEXT:    store i8 8, ptr [[C]], align 1
+// CHECK-NEXT:    [[D:%.*]] = getelementptr inbounds [[STRUCT_TINY]], ptr [[DOTCOMPOUNDLITERAL]], i32 0, i32 3
+// CHECK-NEXT:    store i8 9, ptr [[D]], align 1
+// CHECK-NEXT:    [[A2:%.*]] = getelementptr inbounds [[STRUCT_SMALL]], ptr [[DOTCOMPOUNDLITERAL1]], i32 0, i32 0
+// CHECK-NEXT:    store i32 10, ptr [[A2]], align 4
+// CHECK-NEXT:    [[B3:%.*]] = getelementptr inbounds [[STRUCT_SMALL]], ptr [[DOTCOMPOUNDLITERAL1]], i32 0, i32 1
+// CHECK-NEXT:    store ptr null, ptr [[B3]], align 4
+// CHECK-NEXT:    [[A5:%.*]] = getelementptr inbounds [[STRUCT_SMALL_ALIGNED]], ptr [[DOTCOMPOUNDLITERAL4]], i32 0, i32 0
+// CHECK-NEXT:    store i64 11, ptr [[A5]], align 8
+// CHECK-NEXT:    [[A7:%.*]] = getelementptr inbounds [[STRUCT_LARGE]], ptr [[DOTCOMPOUNDLITERAL6]], i32 0, i32 0
+// CHECK-NEXT:    store i32 12, ptr [[A7]], align 4
+// CHECK-NEXT:    [[B8:%.*]] = getelementptr inbounds [[STRUCT_LARGE]], ptr [[DOTCOMPOUNDLITERAL6]], i32 0, i32 1
+// CHECK-NEXT:    store i32 13, ptr [[B8]], align 4
+// CHECK-NEXT:    [[C9:%.*]] = getelementptr inbounds [[STRUCT_LARGE]], ptr [[DOTCOMPOUNDLITERAL6]], i32 0, i32 2
+// CHECK-NEXT:    store i32 14, ptr [[C9]], align 4
+// CHECK-NEXT:    [[D10:%.*]] = getelementptr inbounds [[STRUCT_LARGE]], ptr [[DOTCOMPOUNDLITERAL6]], i32 0, i32 3
+// CHECK-NEXT:    store i32 15, ptr [[D10]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[DOTCOMPOUNDLITERAL]], align 1
+// CHECK-NEXT:    [[TMP1:%.*]] = load [2 x i32], ptr [[DOTCOMPOUNDLITERAL1]], align 4
+// CHECK-NEXT:    [[COERCE_DIVE:%.*]] = getelementptr inbounds [[STRUCT_SMALL_ALIGNED]], ptr [[DOTCOMPOUNDLITERAL4]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP2:%.*]] = load i64, ptr [[COERCE_DIVE]], align 8
+// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[BYVAL_TEMP]], ptr align 4 [[DOTCOMPOUNDLITERAL6]], i32 16, i1 false)
+// CHECK-NEXT:    [[CALL:%.*]] = call i32 (i32, ...) @f_va_callee(i32 noundef 1, i32 noundef 2, i64 noundef 3, double noundef 4.000000e+00, double noundef 5.000000e+00, i32 [[TMP0]], [2 x i32] [[TMP1]], i64 [[TMP2]], ptr noundef [[BYVAL_TEMP]])
+// CHECK-NEXT:    ret void
+//
+void f_va_caller(void) {
+  f_va_callee(1, 2, 3LL, 4.0f, 5.0, (struct tiny){6, 7, 8, 9},
+              (struct small){10, NULL}, (struct small_aligned){11},
+              (struct large){12, 13, 14, 15});
+}
+
+// CHECK-LABEL: define dso_local i32 @f_va_1
+// CHECK-SAME: (ptr noundef [[FMT:%.*]], ...) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[FMT_ADDR:%.*]] = alloca ptr, align 4
+// CHECK-NEXT:    [[VA:%.*]] = alloca ptr, align 4
+// CHECK-NEXT:    [[V:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    store ptr [[FMT]], ptr [[FMT_ADDR]], align 4
+// CHECK-NEXT:    call void @llvm.va_start(ptr [[VA]])
+// CHECK-NEXT:    [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 4
+// CHECK-NEXT:    [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR]], i32 4
+// CHECK-NEXT:    store ptr [[ARGP_NEXT]], ptr [[VA]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[ARGP_CUR]], align 4
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[V]], align 4
+// CHECK-NEXT:    call void @llvm.va_end(ptr [[VA]])
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[V]], align 4
+// CHECK-NEXT:    ret i32 [[TMP1]]
+//
+int f_va_1(char *fmt, ...) {
+  __builtin_va_list va;
+
+  __builtin_va_start(va, fmt);
+  int v = __builtin_va_arg(va, int);
+  __builtin_va_end(va);
+
+  return v;
+}
+
+// An "aligned" register pair (where the first register is even-numbered) is
+// used to pass varargs with 2x xlen alignment and 2x xlen size. Ensure the
+// correct offsets are used.
+
+// CHECK-LABEL: define dso_local double @f_va_2
+// CHECK-SAME: (ptr noundef [[FMT:%.*]], ...) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[FMT_ADDR:%.*]] = alloca ptr, align 4
+// CHECK-NEXT:    [[VA:%.*]] = alloca ptr, align 4
+// CHECK-NEXT:    [[V:%.*]] = alloca double, align 8
+// CHECK-NEXT:    store ptr [[FMT]], ptr [[FMT_ADDR]], align 4
+// CHECK-NEXT:    call void @llvm.va_start(ptr [[VA]])
+// CHECK-NEXT:    [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR]], i32 7
+// CHECK-NEXT:    [[ARGP_CUR_ALIGNED:%.*]] = call ptr @llvm.ptrmask.p0.i32(ptr [[TMP0]], i32 -8)
+// CHECK-NEXT:    [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR_ALIGNED]], i32 8
+// CHECK-NEXT:    store ptr [[ARGP_NEXT]], ptr [[VA]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = load double, ptr [[ARGP_CUR_ALIGNED]], align 8
+// CHECK-NEXT:    store double [[TMP1]], ptr [[V]], align 8
+// CHECK-NEXT:    call void @llvm.va_end(ptr [[VA]])
+// CHECK-NEXT:    [[TMP2:%.*]] = load double, ptr [[V]], align 8
+// CHECK-NEXT:    ret double [[TMP2]]
+//
+double f_va_2(char *fmt, ...) {
+  __builtin_va_list va;
+
+  __builtin_va_start(va, fmt);
+  double v = __builtin_va_arg(va, double);
+  __builtin_va_end(va);
+
+  return v;
+}
+
+// Two "aligned" register pairs.
+
+// CHECK-LABEL: define dso_local double @f_va_3
+// CHECK-SAME: (ptr noundef [[FMT:%.*]], ...) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[FMT_ADDR:%.*]] = alloca ptr, align 4
+// CHECK-NEXT:    [[VA:%.*]] = alloca ptr, align 4
+// CHECK-NEXT:    [[V:%.*]] = alloca double, align 8
+// CHECK-NEXT:    [[W:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[X:%.*]] = alloca double, align 8
+// CHECK-NEXT:    store ptr [[FMT]], ptr [[FMT_ADDR]], align 4
+// CHECK-NEXT:    call void @llvm.va_start(ptr [[VA]])
+// CHECK-NEXT:    [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR]], i32 7
+// CHECK-NEXT:    [[ARGP_CUR_ALIGNED:%.*]] = call ptr @llvm.ptrmask.p0.i32(ptr [[TMP0]], i32 -8)
+// CHECK-NEXT:    [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR_ALIGNED]], i32 8
+// CHECK-NEXT:    store ptr [[ARGP_NEXT]], ptr [[VA]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = load double, ptr [[ARGP_CUR_ALIGNED]], align 8
+// CHECK-NEXT:    store double [[TMP1]], ptr [[V]], align 8
+// CHECK-NEXT:    [[ARGP_CUR1:%.*]] = load ptr, ptr [[VA]], align 4
+// CHECK-NEXT:    [[ARGP_NEXT2:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR1]], i32 4
+// CHECK-NEXT:    store ptr [[ARGP_NEXT2]], ptr [[VA]], align 4
+// CHECK-NEXT:    [[TMP2:%.*]] = load i32, ptr [[ARGP_CUR1]], align 4
+// CHECK-NEXT:    store i32 [[TMP2]], ptr [[W]], align 4
+// CHECK-NEXT:    [[ARGP_CUR3:%.*]] = load ptr, ptr [[VA]], align 4
+// CHECK-NEXT:    [[TMP3:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR3]], i32 7
+// CHECK-NEXT:    [[ARGP_CUR3_ALIGNED:%.*]] = call ptr @llvm.ptrmask.p0.i32(ptr [[TMP3]], i32 -8)
+// CHECK-NEXT:    [[ARGP_NEXT4:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR3_ALIGNED]], i32 8
+// CHECK-NEXT:    store ptr [[ARGP_NEXT4]], ptr [[VA]], align 4
+// CHECK-NEXT:    [[TMP4:%.*]] = load double, ptr [[ARGP_CUR3_ALIGNED]], align 8
+// CHECK-NEXT:    store double [[TMP4]], ptr [[X]], align 8
+// CHECK-NEXT:    call void @llvm.va_end(ptr [[VA]])
+// CHECK-NEXT:    [[TMP5:%.*]] = load double, ptr [[V]], align 8
+// CHECK-NEXT:    [[TMP6:%.*]] = load double, ptr [[X]], align 8
+// CHECK-NEXT:    [[ADD:%.*]] = fadd double [[TMP5]], [[TMP6]]
+// CHECK-NEXT:    ret double [[ADD]]
+//
+double f_va_3(char *fmt, ...) {
+  __builtin_va_list va;
+
+  __builtin_va_start(va, fmt);
+  double v = __builtin_va_arg(va, double);
+  int w = __builtin_va_arg(va, int);
+  double x = __builtin_va_arg(va, double);
+  __builtin_va_end(va);
+
+  return v + x;
+}
+
+// CHECK-LABEL: define dso_local i32 @f_va_4
+// CHECK-SAME: (ptr noundef [[FMT:%.*]], ...) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[FMT_ADDR:%.*]] = alloca ptr, align 4
+// CHECK-NEXT:    [[VA:%.*]] = alloca ptr, align 4
+// CHECK-NEXT:    [[V:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[LD:%.*]] = alloca fp128, align 16
+// CHECK-NEXT:    [[TS:%.*]] = alloca [[STRUCT_TINY:%.*]], align 1
+// CHECK-NEXT:    [[SS:%.*]] = alloca [[STRUCT_SMALL:%.*]], align 4
+// CHECK-NEXT:    [[LS:%.*]] = alloca [[STRUCT_LARGE:%.*]], align 4
+// CHECK-NEXT:    [[RET:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    store ptr [[FMT]], ptr [[FMT_ADDR]], align 4
+// CHECK-NEXT:    call void @llvm.va_start(ptr [[VA]])
+// CHECK-NEXT:    [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 4
+// CHECK-NEXT:    [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR]], i32 4
+// CHECK-NEXT:    store ptr [[ARGP_NEXT]], ptr [[VA]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[ARGP_CUR]], align 4
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[V]], align 4
+// CHECK-NEXT:    [[ARGP_CUR1:%.*]] = load ptr, ptr [[VA]], align 4
+// CHECK-NEXT:    [[ARGP_NEXT2:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR1]], i32 4
+// CHECK-NEXT:    store ptr [[ARGP_NEXT2]], ptr [[VA]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = load ptr, ptr [[ARGP_CUR1]], align 4
+// CHECK-NEXT:    [[TMP2:%.*]] = load fp128, ptr [[TMP1]], align 16
+// CHECK-NEXT:    store fp128 [[TMP2]], ptr [[LD]], align 16
+// CHECK-NEXT:    [[ARGP_CUR3:%.*]] = load ptr, ptr [[VA]], align 4
+// CHECK-NEXT:    [[ARGP_NEXT4:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR3]], i32 4
+// CHECK-NEXT:    store ptr [[ARGP_NEXT4]], ptr [[VA]], align 4
+// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 1 [[TS]], ptr align 4 [[ARGP_CUR3]], i32 4, i1 false)
+// CHECK-NEXT:    [[ARGP_CUR5:%.*]] = load ptr, ptr [[VA]], align 4
+// CHECK-NEXT:    [[ARGP_NEXT6:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR5]], i32 8
+// CHECK-NEXT:    store ptr [[ARGP_NEXT6]], ptr [[VA]], align 4
+// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[SS]], ptr align 4 [[ARGP_CUR5]], i32 8, i1 false)
+// CHECK-NEXT:    [[ARGP_CUR7:%.*]] = load ptr, ptr [[VA]], align 4
+// CHECK-NEXT:    [[ARGP_NEXT8:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR7]], i32 4
+// CHECK-NEXT:    store ptr [[ARGP_NEXT8]], ptr [[VA]], align 4
+// CHECK-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[ARGP_CUR7]], align 4
+// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[LS]], ptr align 4 [[TMP3]], i32 16, i1 false)
+// CHECK-NEXT:    call void @llvm.va_end(ptr [[VA]])
+// CHECK-NEXT:    [[TMP4:%.*]] = load i32, ptr [[V]], align 4
+// CHECK-NEXT:    [[CONV:%.*]] = sitofp i32 [[TMP4]] to fp128
+// CHECK-NEXT:    [[TMP5:%.*]] = load fp128, ptr [[LD]], align 16
+// CHECK-NEXT:    [[ADD:%.*]] = fadd fp128 [[CONV]], [[TMP5]]
+// CHECK-NEXT:    [[CONV9:%.*]] = fptosi fp128 [[ADD]] to i32
+// CHECK-NEXT:    store i32 [[CONV9]], ptr [[RET]], align 4
+// CHECK-NEXT:    [[TMP6:%.*]] = load i32, ptr [[RET]], align 4
+// CHECK-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_TINY]], ptr [[TS]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP7:%.*]] = load i8, ptr [[A]], align 1
+// CHECK-NEXT:    [[CONV10:%.*]] = zext i8 [[TMP7]] to i32
+// CHECK-NEXT:    [[ADD11:%.*]] = add nsw i32 [[TMP6]], [[CONV10]]
+// CHECK-NEXT:    [[B:%.*]] = getelementptr inbounds [[STRUCT_TINY]], ptr [[TS]], i32 0, i32 1
+// CHECK-NEXT:    [[TMP8:%.*]] = load i8, ptr [[B]], align 1
+// CHECK-NEXT:    [[CONV12:%.*]] = zext i8 [[TMP8]] to i32
+// CHECK-NEXT:    [[ADD13:%.*]] = add nsw i32 [[ADD11]], [[CONV12]]
+// CHECK-NEXT:    [[C:%.*]] = getelementptr inbounds [[STRUCT_TINY]], ptr [[TS]], i32 0, i32 2
+// CHECK-NEXT:    [[TMP9:%.*]] = load i8, ptr [[C]], align 1
+// CHECK-NEXT:    [[CONV14:%.*]] = zext i8 [[TMP9]] to i32
+// CHECK-NEXT:    [[ADD15:%.*]] = add nsw i32 [[ADD13]], [[CONV14]]
+// CHECK-NEXT:    [[D:%.*]] = getelementptr inbounds [[STRUCT_TINY]], ptr [[TS]], i32 0, i32 3
+// CHECK-NEXT:    [[TMP10:%.*]] = load i8, ptr [[D]], align 1
+// CHECK-NEXT:    [[CONV16:%.*]] = zext i8 [[TMP10]] to i32
+// CHECK-NEXT:    [[ADD17:%.*]] = add nsw i32 [[ADD15]], [[CONV16]]
+// CHECK-NEXT:    store i32 [[ADD17]], ptr [[RET]], align 4
+// CHECK-NEXT:    [[TMP11:%.*]] = load i32, ptr [[RET]], align 4
+// CHECK-NEXT:    [[A18:%.*]] = getelementptr inbounds [[STRUCT_SMALL]], ptr [[SS]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP12:%.*]] = load i32, ptr [[A18]], align 4
+// CHECK-NEXT:    [[ADD19:%.*]] = add nsw i32 [[TMP11]], [[TMP12]]
+// CHECK-NEXT:    [[B20:%.*]] = getelementptr inbounds [[STRUCT_SMALL]], ptr [[SS]], i32 0, i32 1
+// CHECK-NEXT:    [[TMP13:%.*]] = load ptr, ptr [[B20]], align 4
+// CHECK-NEXT:    [[TMP14:%.*]] = ptrtoint ptr [[TMP13]] to i32
+// CHECK-NEXT:    [[ADD21:%.*]] = add nsw i32 [[ADD19]], [[TMP14]]
+// CHECK-NEXT:    store i32 [[ADD21]], ptr [[RET]], align 4
+// CHECK-NEXT:    [[TMP15:%.*]] = load i32, ptr [[RET]], align 4
+// CHECK-NEXT:    [[A22:%.*]] = getelementptr inbounds [[STRUCT_LARGE]], ptr [[LS]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP16:%.*]] = load i32, ptr [[A22]], align 4
+// CHECK-NEXT:    [[ADD23:%.*]] = add nsw i32 [[TMP15]], [[TMP16]]
+// CHECK-NEXT:    [[B24:%.*]] = getelementptr inbounds [[STRUCT_LARGE]], ptr [[LS]], i32 0, i32 1
+// CHECK-NEXT:    [[TMP17:%.*]] = load i32, ptr [[B24]], align 4
+// CHECK-NEXT:    [[ADD25:%.*]] = add nsw i32 [[ADD23]], [[TMP17]]
+// CHECK-NEXT:    [[C26:%.*]] = getelementptr inbounds [[STRUCT_LARGE]], ptr [[LS]], i32 0, i32 2
+// CHECK-NEXT:    [[TMP18:%.*]] = load i32, ptr [[C26]], align 4
+// CHECK-NEXT:    [[ADD27:%.*]] = add nsw i32 [[ADD25]], [[TMP18]]
+// CHECK-NEXT:    [[D28:%.*]] = getelementptr inbounds [[STRUCT_LARGE]], ptr [[LS]], i32 0, i32 3
+// CHECK-NEXT:    [[TMP19:%.*]] = load i32, ptr [[D28]], align 4
+// CHECK-NEXT:    [[ADD29:%.*]] = add nsw i32 [[ADD27]], [[TMP19]]
+// CHECK-NEXT:    store i32 [[ADD29]], ptr [[RET]], align 4
+// CHECK-NEXT:    [[TMP20:%.*]] = load i32, ptr [[RET]], align 4
+// CHECK-NEXT:    ret i32 [[TMP20]]
+//
+int f_va_4(char *fmt, ...) {
+  __builtin_va_list va;
+
+  __builtin_va_start(va, fmt);
+  int v = __builtin_va_arg(va, int);
+  long double ld = __builtin_va_arg(va, long double);
+  struct tiny ts = __builtin_va_arg(va, struct tiny);
+  struct small ss = __builtin_va_arg(va, struct small);
+  struct large ls = __builtin_va_arg(va, struct large);
+  __builtin_va_end(va);
+
+  int ret = (int)((long double)v + ld);
+  ret = ret + ts.a + ts.b + ts.c + ts.d;
+  ret = ret + ss.a + (int)ss.b;
+  ret = ret + ls.a + ls.b + ls.c + ls.d;
+
+  return ret;
+}
diff --git a/clang/test/CodeGen/RISCV/riscv64-abi.c b/clang/test/CodeGen/RISCV/riscv64-abi.c
new file mode 100644
index 0000000000000..cd3ee5203bb9a
--- /dev/null
+++ b/clang/test/CodeGen/RISCV/riscv64-abi.c
@@ -0,0 +1,1909 @@
+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --filter "^define |^entry:" --version 2
+// RUN: %clang_cc1 -triple riscv64 -emit-llvm %s -o - \
+// RUN:   | FileCheck -check-prefixes=LP64-LP64F-LP64D,LP64-LP64F,LP64 %s
+// RUN: %clang_cc1 -triple riscv64 -target-feature +f -target-abi lp64f -emit-llvm %s -o - \
+// RUN:   | FileCheck -check-prefixes=LP64-LP64F-LP64D,LP64F-LP64D,LP64-LP64F,LP64F %s
+// RUN: %clang_cc1 -triple riscv64 -target-feature +f -target-feature +d -target-abi lp64d -emit-llvm %s -o - \
+// RUN:   | FileCheck -check-prefixes=LP64-LP64F-LP64D,LP64F-LP64D,LP64D %s
+
+#include <stddef.h>
+#include <stdint.h>
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_void
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0:[0-9]+]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_void(void) {}
+
+// Scalar arguments and return values smaller than the word size are extended
+// according to the sign of their type, up to 32 bits
+
+// LP64-LP64F-LP64D-LABEL: define dso_local zeroext i1 @f_scalar_0
+// LP64-LP64F-LP64D-SAME: (i1 noundef zeroext [[X:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+_Bool f_scalar_0(_Bool x) { return x; }
+
+// LP64-LP64F-LP64D-LABEL: define dso_local signext i8 @f_scalar_1
+// LP64-LP64F-LP64D-SAME: (i8 noundef signext [[X:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+int8_t f_scalar_1(int8_t x) { return x; }
+
+// LP64-LP64F-LP64D-LABEL: define dso_local zeroext i8 @f_scalar_2
+// LP64-LP64F-LP64D-SAME: (i8 noundef zeroext [[X:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+uint8_t f_scalar_2(uint8_t x) { return x; }
+
+// LP64-LP64F-LP64D-LABEL: define dso_local signext i32 @f_scalar_3
+// LP64-LP64F-LP64D-SAME: (i32 noundef signext [[X:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+uint32_t f_scalar_3(int32_t x) { return x; }
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_scalar_4
+// LP64-LP64F-LP64D-SAME: (i64 noundef [[X:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+int64_t f_scalar_4(int64_t x) { return x; }
+
+// LP64-LP64F-LP64D-LABEL: define dso_local float @f_fp_scalar_1
+// LP64-LP64F-LP64D-SAME: (float noundef [[X:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+float f_fp_scalar_1(float x) { return x; }
+
+// LP64-LP64F-LP64D-LABEL: define dso_local double @f_fp_scalar_2
+// LP64-LP64F-LP64D-SAME: (double noundef [[X:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+double f_fp_scalar_2(double x) { return x; }
+
+// LP64-LP64F-LP64D-LABEL: define dso_local fp128 @f_fp_scalar_3
+// LP64-LP64F-LP64D-SAME: (fp128 noundef [[X:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+long double f_fp_scalar_3(long double x) { return x; }
+
+// LP64-LP64F-LP64D-LABEL: define dso_local half @f_fp_scalar_4
+// LP64-LP64F-LP64D-SAME: (half noundef [[X:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+_Float16 f_fp_scalar_4(_Float16 x) { return x; }
+
+// Empty structs or unions are ignored.
+
+struct empty_s {};
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_agg_empty_struct
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct empty_s f_agg_empty_struct(struct empty_s x) {
+  return x;
+}
+
+union empty_u {};
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_agg_empty_union
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+union empty_u f_agg_empty_union(union empty_u x) {
+  return x;
+}
+
+// Aggregates <= 2*xlen may be passed in registers, so will be coerced to
+// integer arguments. The rules for return are the same.
+
+struct tiny {
+  uint16_t a, b, c, d;
+};
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_agg_tiny
+// LP64-LP64F-LP64D-SAME: (i64 [[X_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_agg_tiny(struct tiny x) {
+  x.a += x.b;
+  x.c += x.d;
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_agg_tiny_ret
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct tiny f_agg_tiny_ret(void) {
+  return (struct tiny){1, 2, 3, 4};
+}
+
+typedef uint16_t v4i16 __attribute__((vector_size(8)));
+typedef int64_t v1i64 __attribute__((vector_size(8)));
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_vec_tiny_v4i16
+// LP64-LP64F-LP64D-SAME: (i64 noundef [[X_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_vec_tiny_v4i16(v4i16 x) {
+  x[0] = x[1];
+  x[2] = x[3];
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_vec_tiny_v4i16_ret
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+v4i16 f_vec_tiny_v4i16_ret(void) {
+  return (v4i16){1, 2, 3, 4};
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_vec_tiny_v1i64
+// LP64-LP64F-LP64D-SAME: (i64 noundef [[X_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_vec_tiny_v1i64(v1i64 x) {
+  x[0] = 114;
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_vec_tiny_v1i64_ret
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+v1i64 f_vec_tiny_v1i64_ret(void) {
+  return (v1i64){1};
+}
+
+struct small {
+  int64_t a, *b;
+};
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_agg_small
+// LP64-LP64F-LP64D-SAME: ([2 x i64] [[X_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_agg_small(struct small x) {
+  x.a += *x.b;
+  x.b = &x.a;
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local [2 x i64] @f_agg_small_ret
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct small f_agg_small_ret(void) {
+  return (struct small){1, 0};
+}
+
+typedef uint16_t v8i16 __attribute__((vector_size(16)));
+typedef __int128_t v1i128 __attribute__((vector_size(16)));
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_vec_small_v8i16
+// LP64-LP64F-LP64D-SAME: (i128 noundef [[X_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_vec_small_v8i16(v8i16 x) {
+  x[0] = x[7];
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i128 @f_vec_small_v8i16_ret
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+v8i16 f_vec_small_v8i16_ret(void) {
+  return (v8i16){1, 2, 3, 4, 5, 6, 7, 8};
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_vec_small_v1i128
+// LP64-LP64F-LP64D-SAME: (i128 noundef [[X_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_vec_small_v1i128(v1i128 x) {
+  x[0] = 114;
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i128 @f_vec_small_v1i128_ret
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+v1i128 f_vec_small_v1i128_ret(void) {
+  return (v1i128){1};
+}
+
+// Aggregates of 2*xlen size and 2*xlen alignment should be coerced to a
+// single 2*xlen-sized argument, to ensure that alignment can be maintained if
+// passed on the stack.
+
+struct small_aligned {
+  __int128_t a;
+};
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_agg_small_aligned
+// LP64-LP64F-LP64D-SAME: (i128 [[X_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_agg_small_aligned(struct small_aligned x) {
+  x.a += x.a;
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i128 @f_agg_small_aligned_ret
+// LP64-LP64F-LP64D-SAME: (i128 [[X_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct small_aligned f_agg_small_aligned_ret(struct small_aligned x) {
+  return (struct small_aligned){10};
+}
+
+// Aggregates greater > 2*xlen will be passed and returned indirectly
+struct large {
+  int64_t a, b, c, d;
+};
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_agg_large
+// LP64-LP64F-LP64D-SAME: (ptr noundef [[X:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_agg_large(struct large x) {
+  x.a = x.b + x.c + x.d;
+}
+
+// The address where the struct should be written to will be the first
+// argument
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_agg_large_ret
+// LP64-LP64F-LP64D-SAME: (ptr noalias sret([[STRUCT_LARGE:%.*]]) align 8 [[AGG_RESULT:%.*]], i32 noundef signext [[I:%.*]], i8 noundef signext [[J:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct large f_agg_large_ret(int32_t i, int8_t j) {
+  return (struct large){1, 2, 3, 4};
+}
+
+typedef unsigned char v32i8 __attribute__((vector_size(32)));
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_vec_large_v32i8
+// LP64-LP64F-LP64D-SAME: (ptr noundef [[TMP0:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_vec_large_v32i8(v32i8 x) {
+  x[0] = x[7];
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_vec_large_v32i8_ret
+// LP64-LP64F-LP64D-SAME: (ptr noalias sret(<32 x i8>) align 32 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+v32i8 f_vec_large_v32i8_ret(void) {
+  return (v32i8){1, 2, 3, 4, 5, 6, 7, 8};
+}
+
+// Scalars passed on the stack should have signext/zeroext attributes, just as
+// if they were passed in registers.
+
+// LP64-LP64F-LP64D-LABEL: define dso_local signext i32 @f_scalar_stack_1
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]], [2 x i64] [[B_COERCE:%.*]], i128 [[C_COERCE:%.*]], ptr noundef [[D:%.*]], i8 noundef zeroext [[E:%.*]], i8 noundef signext [[F:%.*]], i8 noundef zeroext [[G:%.*]], i8 noundef signext [[H:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+int f_scalar_stack_1(struct tiny a, struct small b, struct small_aligned c,
+                     struct large d, uint8_t e, int8_t f, uint8_t g, int8_t h) {
+  return g + h;
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local signext i32 @f_scalar_stack_2
+// LP64-LP64F-LP64D-SAME: (i32 noundef signext [[A:%.*]], i128 noundef [[B:%.*]], i64 noundef [[C:%.*]], fp128 noundef [[D:%.*]], ptr noundef [[TMP0:%.*]], i8 noundef zeroext [[F:%.*]], i8 noundef signext [[G:%.*]], i8 noundef zeroext [[H:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+int f_scalar_stack_2(int32_t a, __int128_t b, int64_t c, long double d, v32i8 e,
+                     uint8_t f, int8_t g, uint8_t h) {
+  return g + h;
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local signext i32 @f_scalar_stack_3
+// LP64-LP64F-LP64D-SAME: (i32 noundef signext [[A:%.*]], i128 noundef [[B:%.*]], double noundef [[C:%.*]], fp128 noundef [[D:%.*]], ptr noundef [[TMP0:%.*]], i8 noundef zeroext [[F:%.*]], i8 noundef signext [[G:%.*]], i8 noundef zeroext [[H:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+int f_scalar_stack_3(int32_t a, __int128_t b, double c, long double d, v32i8 e,
+                     uint8_t f, int8_t g, uint8_t h) {
+  return g + h;
+}
+
+// Ensure that scalars passed on the stack are still determined correctly in
+// the presence of large return values that consume a register due to the need
+// to pass a pointer.
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_scalar_stack_4
+// LP64-LP64F-LP64D-SAME: (ptr noalias sret([[STRUCT_LARGE:%.*]]) align 8 [[AGG_RESULT:%.*]], i32 noundef signext [[A:%.*]], i128 noundef [[B:%.*]], fp128 noundef [[C:%.*]], ptr noundef [[TMP0:%.*]], i8 noundef zeroext [[E:%.*]], i8 noundef signext [[F:%.*]], i8 noundef zeroext [[G:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct large f_scalar_stack_4(uint32_t a, __int128_t b, long double c, v32i8 d,
+                              uint8_t e, int8_t f, uint8_t g) {
+  return (struct large){a, e, f, g};
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_scalar_stack_5
+// LP64-LP64F-LP64D-SAME: (ptr noalias sret([[STRUCT_LARGE:%.*]]) align 8 [[AGG_RESULT:%.*]], double noundef [[A:%.*]], i128 noundef [[B:%.*]], fp128 noundef [[C:%.*]], ptr noundef [[TMP0:%.*]], i8 noundef zeroext [[E:%.*]], i8 noundef signext [[F:%.*]], i8 noundef zeroext [[G:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct large f_scalar_stack_5(double a, __int128_t b, long double c, v32i8 d,
+                              uint8_t e, int8_t f, uint8_t g) {
+  return (struct large){a, e, f, g};
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local signext i32 @f_scalar_stack_6
+// LP64-LP64F-LP64D-SAME: (i32 noundef signext [[A:%.*]], i128 noundef [[B:%.*]], float noundef [[C:%.*]], fp128 noundef [[D:%.*]], ptr noundef [[TMP0:%.*]], i8 noundef zeroext [[F:%.*]], i8 noundef signext [[G:%.*]], i8 noundef zeroext [[H:%.*]], half noundef [[I:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+int f_scalar_stack_6(int32_t a, __int128_t b, float c, long double d, v32i8 e,
+                     uint8_t f, int8_t g, uint8_t h, _Float16 i) {
+  return g + h;
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_fpr_tracking
+// LP64-LP64F-LP64D-SAME: (float noundef [[A:%.*]], float noundef [[B:%.*]], float noundef [[C:%.*]], float noundef [[D:%.*]], float noundef [[E:%.*]], float noundef [[F:%.*]], float noundef [[G:%.*]], float noundef [[H:%.*]], i8 noundef zeroext [[I:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_fpr_tracking(float a, float b, float c, float d, float e, float f,
+                    float g, float h, uint8_t i) {}
+
+// Check that fp, fp+fp, and int+fp structs are lowered correctly. These will
+// be passed in FPR, FPR+FPR, or GPR+FPR regs if sufficient registers are
+// available the widths are <= XLEN and FLEN, and should be expanded to
+// separate arguments in IR. They are passed by the same rules for returns,
+// but will be lowered to simple two-element structs if necessary (as LLVM IR
+// functions cannot return multiple values).
+
+// A struct containing just one floating-point real is passed as though it
+// were a standalone floating-point real.
+
+struct float_s { float f; };
+
+// LP64-LABEL: define dso_local void @f_float_s_arg
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_float_s_arg
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_float_s_arg(struct float_s a) {}
+
+// LP64-LABEL: define dso_local i64 @f_ret_float_s
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local float @f_ret_float_s
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+struct float_s f_ret_float_s(void) {
+  return (struct float_s){1.0};
+}
+
+// A struct containing a float and any number of zero-width bitfields is
+// passed as though it were a standalone floating-point real.
+
+struct zbf_float_s { int : 0; float f; };
+struct zbf_float_zbf_s { int : 0; float f; int : 0; };
+
+// LP64-LABEL: define dso_local void @f_zbf_float_s_arg
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_zbf_float_s_arg
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_zbf_float_s_arg(struct zbf_float_s a) {}
+
+// LP64-LABEL: define dso_local i64 @f_ret_zbf_float_s
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local float @f_ret_zbf_float_s
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+struct zbf_float_s f_ret_zbf_float_s(void) {
+  return (struct zbf_float_s){1.0};
+}
+
+// LP64-LABEL: define dso_local void @f_zbf_float_zbf_s_arg
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_zbf_float_zbf_s_arg
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_zbf_float_zbf_s_arg(struct zbf_float_zbf_s a) {}
+
+// LP64-LABEL: define dso_local i64 @f_ret_zbf_float_zbf_s
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local float @f_ret_zbf_float_zbf_s
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+struct zbf_float_zbf_s f_ret_zbf_float_zbf_s(void) {
+  return (struct zbf_float_zbf_s){1.0};
+}
+
+// Check that structs containing two float values (FLEN <= width) are expanded
+// provided sufficient FPRs are available.
+
+struct float_float_s { float f; float g; };
+
+// LP64-LABEL: define dso_local void @f_float_float_s_arg
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_float_float_s_arg
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_float_float_s_arg(struct float_float_s a) {}
+
+// LP64-LABEL: define dso_local i64 @f_ret_float_float_s
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local { float, float } @f_ret_float_float_s
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+struct float_float_s f_ret_float_float_s(void) {
+  return (struct float_float_s){1.0, 2.0};
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_float_float_s_arg_insufficient_fprs
+// LP64-LP64F-LP64D-SAME: (float noundef [[A:%.*]], float noundef [[B:%.*]], float noundef [[C:%.*]], float noundef [[D:%.*]], float noundef [[E:%.*]], float noundef [[F:%.*]], float noundef [[G:%.*]], i64 [[H_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_float_float_s_arg_insufficient_fprs(float a, float b, float c, float d,
+    float e, float f, float g, struct float_float_s h) {}
+
+// Check that structs containing int+float values are expanded, provided
+// sufficient FPRs and GPRs are available. The integer components are neither
+// sign or zero-extended.
+
+struct float_int8_s { float f; int8_t i; };
+struct float_uint8_s { float f; uint8_t i; };
+struct float_int32_s { float f; int32_t i; };
+struct float_int64_s { float f; int64_t i; };
+struct float_int128bf_s { float f; __int128_t i : 64; };
+struct float_int8_zbf_s { float f; int8_t i; int : 0; };
+
+// LP64-LABEL: define dso_local void @f_float_int8_s_arg
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_float_int8_s_arg
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]], i8 [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_float_int8_s_arg(struct float_int8_s a) {}
+
+// LP64-LABEL: define dso_local i64 @f_ret_float_int8_s
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local { float, i8 } @f_ret_float_int8_s
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+struct float_int8_s f_ret_float_int8_s(void) {
+  return (struct float_int8_s){1.0, 2};
+}
+
+// LP64-LABEL: define dso_local void @f_float_uint8_s_arg
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_float_uint8_s_arg
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]], i8 [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_float_uint8_s_arg(struct float_uint8_s a) {}
+
+// LP64-LABEL: define dso_local i64 @f_ret_float_uint8_s
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local { float, i8 } @f_ret_float_uint8_s
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+struct float_uint8_s f_ret_float_uint8_s(void) {
+  return (struct float_uint8_s){1.0, 2};
+}
+
+// LP64-LABEL: define dso_local void @f_float_int32_s_arg
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_float_int32_s_arg
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]], i32 [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_float_int32_s_arg(struct float_int32_s a) {}
+
+// LP64-LABEL: define dso_local i64 @f_ret_float_int32_s
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local { float, i32 } @f_ret_float_int32_s
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+struct float_int32_s f_ret_float_int32_s(void) {
+  return (struct float_int32_s){1.0, 2};
+}
+
+// LP64-LABEL: define dso_local void @f_float_int64_s_arg
+// LP64-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_float_int64_s_arg
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]], i64 [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_float_int64_s_arg(struct float_int64_s a) {}
+
+// LP64-LABEL: define dso_local [2 x i64] @f_ret_float_int64_s
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local { float, i64 } @f_ret_float_int64_s
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+struct float_int64_s f_ret_float_int64_s(void) {
+  return (struct float_int64_s){1.0, 2};
+}
+
+// LP64-LABEL: define dso_local void @f_float_int128bf_s_arg
+// LP64-SAME: (i128 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_float_int128bf_s_arg
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]], i64 [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_float_int128bf_s_arg(struct float_int128bf_s a) {}
+
+// LP64-LABEL: define dso_local i128 @f_ret_float_int128bf_s
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local <{ float, i64 }> @f_ret_float_int128bf_s
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+struct float_int128bf_s f_ret_float_int128bf_s(void) {
+  return (struct float_int128bf_s){1.0, 2};
+}
+
+// The zero-width bitfield means the struct can't be passed according to the
+// floating point calling convention.
+
+// LP64-LABEL: define dso_local void @f_float_int8_zbf_s
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_float_int8_zbf_s
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]], i8 [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_float_int8_zbf_s(struct float_int8_zbf_s a) {}
+
+// LP64-LABEL: define dso_local i64 @f_ret_float_int8_zbf_s
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local { float, i8 } @f_ret_float_int8_zbf_s
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+struct float_int8_zbf_s f_ret_float_int8_zbf_s(void) {
+  return (struct float_int8_zbf_s){1.0, 2};
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_float_int8_s_arg_insufficient_gprs
+// LP64-LP64F-LP64D-SAME: (i32 noundef signext [[A:%.*]], i32 noundef signext [[B:%.*]], i32 noundef signext [[C:%.*]], i32 noundef signext [[D:%.*]], i32 noundef signext [[E:%.*]], i32 noundef signext [[F:%.*]], i32 noundef signext [[G:%.*]], i32 noundef signext [[H:%.*]], i64 [[I_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_float_int8_s_arg_insufficient_gprs(int a, int b, int c, int d, int e,
+                                          int f, int g, int h, struct float_int8_s i) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_struct_float_int8_insufficient_fprs
+// LP64-LP64F-LP64D-SAME: (float noundef [[A:%.*]], float noundef [[B:%.*]], float noundef [[C:%.*]], float noundef [[D:%.*]], float noundef [[E:%.*]], float noundef [[F:%.*]], float noundef [[G:%.*]], float noundef [[H:%.*]], i64 [[I_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_struct_float_int8_insufficient_fprs(float a, float b, float c, float d,
+                                           float e, float f, float g, float h, struct float_int8_s i) {}
+
+// Complex floating-point values or structs containing a single complex
+// floating-point value should be passed as if it were an fp+fp struct.
+
+// LP64-LABEL: define dso_local void @f_floatcomplex
+// LP64-SAME: (i64 noundef [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_floatcomplex
+// LP64F-LP64D-SAME: (float noundef [[A_COERCE0:%.*]], float noundef [[A_COERCE1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_floatcomplex(float __complex__ a) {}
+
+// LP64-LABEL: define dso_local i64 @f_ret_floatcomplex
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local { float, float } @f_ret_floatcomplex
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+float __complex__ f_ret_floatcomplex(void) {
+  return 1.0;
+}
+
+struct floatcomplex_s { float __complex__ c; };
+
+// LP64-LABEL: define dso_local void @f_floatcomplex_s_arg
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_floatcomplex_s_arg
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_floatcomplex_s_arg(struct floatcomplex_s a) {}
+
+// LP64-LABEL: define dso_local i64 @f_ret_floatcomplex_s
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local { float, float } @f_ret_floatcomplex_s
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+struct floatcomplex_s f_ret_floatcomplex_s(void) {
+  return (struct floatcomplex_s){1.0};
+}
+
+// Complex floating-point values or structs containing a single complex
+// floating-point value should be passed in GPRs if no two FPRs is available.
+
+// LP64-LABEL: define dso_local void @f_floatcomplex_insufficient_fprs1
+// LP64-SAME: (i64 noundef [[A_COERCE:%.*]], i64 noundef [[B_COERCE:%.*]], i64 noundef [[C_COERCE:%.*]], i64 noundef [[D_COERCE:%.*]], i64 noundef [[E_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_floatcomplex_insufficient_fprs1
+// LP64F-LP64D-SAME: (float noundef [[A_COERCE0:%.*]], float noundef [[A_COERCE1:%.*]], float noundef [[B_COERCE0:%.*]], float noundef [[B_COERCE1:%.*]], float noundef [[C_COERCE0:%.*]], float noundef [[C_COERCE1:%.*]], float noundef [[D_COERCE0:%.*]], float noundef [[D_COERCE1:%.*]], i64 noundef [[E_COERCE:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_floatcomplex_insufficient_fprs1(float __complex__ a, float __complex__ b,
+                                       float __complex__ c, float __complex__ d,
+                                       float __complex__ e) {}
+
+
+// LP64-LABEL: define dso_local void @f_floatcomplex_s_arg_insufficient_fprs1
+// LP64-SAME: (i64 [[A_COERCE:%.*]], i64 [[B_COERCE:%.*]], i64 [[C_COERCE:%.*]], i64 [[D_COERCE:%.*]], i64 [[E_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_floatcomplex_s_arg_insufficient_fprs1
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]], float [[TMP1:%.*]], float [[TMP2:%.*]], float [[TMP3:%.*]], float [[TMP4:%.*]], float [[TMP5:%.*]], float [[TMP6:%.*]], float [[TMP7:%.*]], i64 [[E_COERCE:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_floatcomplex_s_arg_insufficient_fprs1(struct floatcomplex_s a,
+                                             struct floatcomplex_s b,
+                                             struct floatcomplex_s c,
+                                             struct floatcomplex_s d,
+                                             struct floatcomplex_s e) {}
+
+// LP64-LABEL: define dso_local void @f_floatcomplex_insufficient_fprs2
+// LP64-SAME: (float noundef [[A:%.*]], i64 noundef [[B_COERCE:%.*]], i64 noundef [[C_COERCE:%.*]], i64 noundef [[D_COERCE:%.*]], i64 noundef [[E_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_floatcomplex_insufficient_fprs2
+// LP64F-LP64D-SAME: (float noundef [[A:%.*]], float noundef [[B_COERCE0:%.*]], float noundef [[B_COERCE1:%.*]], float noundef [[C_COERCE0:%.*]], float noundef [[C_COERCE1:%.*]], float noundef [[D_COERCE0:%.*]], float noundef [[D_COERCE1:%.*]], i64 noundef [[E_COERCE:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_floatcomplex_insufficient_fprs2(float a,
+                                       float __complex__ b, float __complex__ c,
+                                       float __complex__ d, float __complex__ e) {}
+
+
+// LP64-LABEL: define dso_local void @f_floatcomplex_s_arg_insufficient_fprs2
+// LP64-SAME: (float noundef [[A:%.*]], i64 [[B_COERCE:%.*]], i64 [[C_COERCE:%.*]], i64 [[D_COERCE:%.*]], i64 [[E_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_floatcomplex_s_arg_insufficient_fprs2
+// LP64F-LP64D-SAME: (float noundef [[A:%.*]], float [[TMP0:%.*]], float [[TMP1:%.*]], float [[TMP2:%.*]], float [[TMP3:%.*]], float [[TMP4:%.*]], float [[TMP5:%.*]], i64 [[E_COERCE:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_floatcomplex_s_arg_insufficient_fprs2(float a,
+                                             struct floatcomplex_s b,
+                                             struct floatcomplex_s c,
+                                             struct floatcomplex_s d,
+                                             struct floatcomplex_s e) {}
+
+// Test single or two-element structs that need flattening. e.g. those
+// containing nested structs, floats in small arrays, zero-length structs etc.
+
+struct floatarr1_s { float a[1]; };
+
+// LP64-LABEL: define dso_local void @f_floatarr1_s_arg
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_floatarr1_s_arg
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_floatarr1_s_arg(struct floatarr1_s a) {}
+
+// LP64-LABEL: define dso_local i64 @f_ret_floatarr1_s
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local float @f_ret_floatarr1_s
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+struct floatarr1_s f_ret_floatarr1_s(void) {
+  return (struct floatarr1_s){{1.0}};
+}
+
+struct floatarr2_s { float a[2]; };
+
+// LP64-LABEL: define dso_local void @f_floatarr2_s_arg
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_floatarr2_s_arg
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_floatarr2_s_arg(struct floatarr2_s a) {}
+
+// LP64-LABEL: define dso_local i64 @f_ret_floatarr2_s
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local { float, float } @f_ret_floatarr2_s
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+struct floatarr2_s f_ret_floatarr2_s(void) {
+  return (struct floatarr2_s){{1.0, 2.0}};
+}
+
+struct floatarr2_tricky1_s { struct { float f[1]; } g[2]; };
+
+// LP64-LABEL: define dso_local void @f_floatarr2_tricky1_s_arg
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_floatarr2_tricky1_s_arg
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_floatarr2_tricky1_s_arg(struct floatarr2_tricky1_s a) {}
+
+// LP64-LABEL: define dso_local i64 @f_ret_floatarr2_tricky1_s
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local { float, float } @f_ret_floatarr2_tricky1_s
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+struct floatarr2_tricky1_s f_ret_floatarr2_tricky1_s(void) {
+  return (struct floatarr2_tricky1_s){{{{1.0}}, {{2.0}}}};
+}
+
+struct floatarr2_tricky2_s { struct {}; struct { float f[1]; } g[2]; };
+
+// LP64-LABEL: define dso_local void @f_floatarr2_tricky2_s_arg
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_floatarr2_tricky2_s_arg
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_floatarr2_tricky2_s_arg(struct floatarr2_tricky2_s a) {}
+
+// LP64-LABEL: define dso_local i64 @f_ret_floatarr2_tricky2_s
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local { float, float } @f_ret_floatarr2_tricky2_s
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+struct floatarr2_tricky2_s f_ret_floatarr2_tricky2_s(void) {
+  return (struct floatarr2_tricky2_s){{}, {{{1.0}}, {{2.0}}}};
+}
+
+struct floatarr2_tricky3_s { union {}; struct { float f[1]; } g[2]; };
+
+// LP64-LABEL: define dso_local void @f_floatarr2_tricky3_s_arg
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_floatarr2_tricky3_s_arg
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_floatarr2_tricky3_s_arg(struct floatarr2_tricky3_s a) {}
+
+// LP64-LABEL: define dso_local i64 @f_ret_floatarr2_tricky3_s
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local { float, float } @f_ret_floatarr2_tricky3_s
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+struct floatarr2_tricky3_s f_ret_floatarr2_tricky3_s(void) {
+  return (struct floatarr2_tricky3_s){{}, {{{1.0}}, {{2.0}}}};
+}
+
+struct floatarr2_tricky4_s { union {}; struct { struct {}; float f[1]; } g[2]; };
+
+// LP64-LABEL: define dso_local void @f_floatarr2_tricky4_s_arg
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_floatarr2_tricky4_s_arg
+// LP64F-LP64D-SAME: (float [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_floatarr2_tricky4_s_arg(struct floatarr2_tricky4_s a) {}
+
+// LP64-LABEL: define dso_local i64 @f_ret_floatarr2_tricky4_s
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local { float, float } @f_ret_floatarr2_tricky4_s
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+struct floatarr2_tricky4_s f_ret_floatarr2_tricky4_s(void) {
+  return (struct floatarr2_tricky4_s){{}, {{{}, {1.0}}, {{}, {2.0}}}};
+}
+
+// Test structs that should be passed according to the normal integer calling
+// convention.
+
+struct int_float_int_s { int a; float b; int c; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_int_float_int_s_arg
+// LP64-LP64F-LP64D-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_int_float_int_s_arg(struct int_float_int_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local [2 x i64] @f_ret_int_float_int_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct int_float_int_s f_ret_int_float_int_s(void) {
+  return (struct int_float_int_s){1, 2.0, 3};
+}
+
+struct char_char_float_s { char a; char b; float c; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_char_char_float_s_arg
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_char_char_float_s_arg(struct char_char_float_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_char_char_float_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct char_char_float_s f_ret_char_char_float_s(void) {
+  return (struct char_char_float_s){1, 2, 3.0};
+}
+
+// Unions are always passed according to the integer calling convention, even
+// if they can only contain a float.
+
+union float_u { float a; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_float_u_arg
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_float_u_arg(union float_u a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_float_u
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+union float_u f_ret_float_u(void) {
+  return (union float_u){1.0};
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_fpr_tracking2
+// LP64-LP64F-LP64D-SAME: (double noundef [[A:%.*]], double noundef [[B:%.*]], double noundef [[C:%.*]], double noundef [[D:%.*]], double noundef [[E:%.*]], double noundef [[F:%.*]], double noundef [[G:%.*]], double noundef [[H:%.*]], i8 noundef zeroext [[I:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_fpr_tracking2(double a, double b, double c, double d, double e, double f,
+                     double g, double h, uint8_t i) {}
+
+// Check that fp, fp+fp, and int+fp structs are lowered correctly. These will
+// be passed in FPR, FPR+FPR, or GPR+FPR regs if sufficient registers are
+// available the widths are <= XLEN and FLEN, and should be expanded to
+// separate arguments in IR. They are passed by the same rules for returns,
+// but will be lowered to simple two-element structs if necessary (as LLVM IR
+// functions cannot return multiple values).
+
+// A struct containing just one floating-point real is passed as though it
+// were a standalone floating-point real.
+
+struct double_s { double f; };
+
+// LP64-LP64F-LABEL: define dso_local void @f_double_s_arg
+// LP64-LP64F-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_double_s_arg
+// LP64D-SAME: (double [[TMP0:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_double_s_arg(struct double_s a) {}
+
+// LP64-LP64F-LABEL: define dso_local i64 @f_ret_double_s
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local double @f_ret_double_s
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+struct double_s f_ret_double_s(void) {
+  return (struct double_s){1.0};
+}
+
+// A struct containing a double and any number of zero-width bitfields is
+// passed as though it were a standalone floating-point real.
+
+struct zbf_double_s { int : 0; double f; };
+struct zbf_double_zbf_s { int : 0; double f; int : 0; };
+
+// LP64-LP64F-LABEL: define dso_local void @f_zbf_double_s_arg
+// LP64-LP64F-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_zbf_double_s_arg
+// LP64D-SAME: (double [[TMP0:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_zbf_double_s_arg(struct zbf_double_s a) {}
+
+// LP64-LP64F-LABEL: define dso_local i64 @f_ret_zbf_double_s
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local double @f_ret_zbf_double_s
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+struct zbf_double_s f_ret_zbf_double_s(void) {
+  return (struct zbf_double_s){1.0};
+}
+
+// LP64-LP64F-LABEL: define dso_local void @f_zbf_double_zbf_s_arg
+// LP64-LP64F-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_zbf_double_zbf_s_arg
+// LP64D-SAME: (double [[TMP0:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_zbf_double_zbf_s_arg(struct zbf_double_zbf_s a) {}
+
+// LP64-LP64F-LABEL: define dso_local i64 @f_ret_zbf_double_zbf_s
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local double @f_ret_zbf_double_zbf_s
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+struct zbf_double_zbf_s f_ret_zbf_double_zbf_s(void) {
+  return (struct zbf_double_zbf_s){1.0};
+}
+
+// Check that structs containing two floating point values (FLEN <= width) are
+// expanded provided sufficient FPRs are available.
+
+struct double_double_s { double f; double g; };
+struct double_float_s { double f; float g; };
+
+// LP64-LP64F-LABEL: define dso_local void @f_double_double_s_arg
+// LP64-LP64F-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_double_double_s_arg
+// LP64D-SAME: (double [[TMP0:%.*]], double [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_double_double_s_arg(struct double_double_s a) {}
+
+// LP64-LP64F-LABEL: define dso_local [2 x i64] @f_ret_double_double_s
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local { double, double } @f_ret_double_double_s
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+struct double_double_s f_ret_double_double_s(void) {
+  return (struct double_double_s){1.0, 2.0};
+}
+
+// LP64-LP64F-LABEL: define dso_local void @f_double_float_s_arg
+// LP64-LP64F-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_double_float_s_arg
+// LP64D-SAME: (double [[TMP0:%.*]], float [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_double_float_s_arg(struct double_float_s a) {}
+
+// LP64-LP64F-LABEL: define dso_local [2 x i64] @f_ret_double_float_s
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local { double, float } @f_ret_double_float_s
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+struct double_float_s f_ret_double_float_s(void) {
+  return (struct double_float_s){1.0, 2.0};
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_double_double_s_arg_insufficient_fprs
+// LP64-LP64F-LP64D-SAME: (float noundef [[A:%.*]], double noundef [[B:%.*]], double noundef [[C:%.*]], double noundef [[D:%.*]], double noundef [[E:%.*]], double noundef [[F:%.*]], double noundef [[G:%.*]], [2 x i64] [[H_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_double_double_s_arg_insufficient_fprs(float a, double b, double c, double d,
+    double e, double f, double g, struct double_double_s h) {}
+
+// Check that structs containing int+double values are expanded, provided
+// sufficient FPRs and GPRs are available. The integer components are neither
+// sign or zero-extended.
+
+struct double_int8_s { double f; int8_t i; };
+struct double_uint8_s { double f; uint8_t i; };
+struct double_int32_s { double f; int32_t i; };
+struct double_int64_s { double f; int64_t i; };
+struct double_int128bf_s { double f; __int128_t i : 64; };
+struct double_int8_zbf_s { double f; int8_t i; int : 0; };
+
+// LP64-LP64F-LABEL: define dso_local void @f_double_int8_s_arg
+// LP64-LP64F-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_double_int8_s_arg
+// LP64D-SAME: (double [[TMP0:%.*]], i8 [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_double_int8_s_arg(struct double_int8_s a) {}
+
+// LP64-LP64F-LABEL: define dso_local [2 x i64] @f_ret_double_int8_s
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local { double, i8 } @f_ret_double_int8_s
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+struct double_int8_s f_ret_double_int8_s(void) {
+  return (struct double_int8_s){1.0, 2};
+}
+
+// LP64-LP64F-LABEL: define dso_local void @f_double_uint8_s_arg
+// LP64-LP64F-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_double_uint8_s_arg
+// LP64D-SAME: (double [[TMP0:%.*]], i8 [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_double_uint8_s_arg(struct double_uint8_s a) {}
+
+// LP64-LP64F-LABEL: define dso_local [2 x i64] @f_ret_double_uint8_s
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local { double, i8 } @f_ret_double_uint8_s
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+struct double_uint8_s f_ret_double_uint8_s(void) {
+  return (struct double_uint8_s){1.0, 2};
+}
+
+// LP64-LP64F-LABEL: define dso_local void @f_double_int32_s_arg
+// LP64-LP64F-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_double_int32_s_arg
+// LP64D-SAME: (double [[TMP0:%.*]], i32 [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_double_int32_s_arg(struct double_int32_s a) {}
+
+// LP64-LP64F-LABEL: define dso_local [2 x i64] @f_ret_double_int32_s
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local { double, i32 } @f_ret_double_int32_s
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+struct double_int32_s f_ret_double_int32_s(void) {
+  return (struct double_int32_s){1.0, 2};
+}
+
+// LP64-LP64F-LABEL: define dso_local void @f_double_int64_s_arg
+// LP64-LP64F-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_double_int64_s_arg
+// LP64D-SAME: (double [[TMP0:%.*]], i64 [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_double_int64_s_arg(struct double_int64_s a) {}
+
+// LP64-LP64F-LABEL: define dso_local [2 x i64] @f_ret_double_int64_s
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local { double, i64 } @f_ret_double_int64_s
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+struct double_int64_s f_ret_double_int64_s(void) {
+  return (struct double_int64_s){1.0, 2};
+}
+
+// LP64-LP64F-LABEL: define dso_local void @f_double_int128bf_s_arg
+// LP64-LP64F-SAME: (i128 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_double_int128bf_s_arg
+// LP64D-SAME: (double [[TMP0:%.*]], i64 [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_double_int128bf_s_arg(struct double_int128bf_s a) {}
+
+// LP64-LP64F-LABEL: define dso_local i128 @f_ret_double_int128bf_s
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local { double, i64 } @f_ret_double_int128bf_s
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+struct double_int128bf_s f_ret_double_int128bf_s(void) {
+  return (struct double_int128bf_s){1.0, 2};
+}
+
+// The zero-width bitfield means the struct can't be passed according to the
+// floating point calling convention.
+
+// LP64-LP64F-LABEL: define dso_local void @f_double_int8_zbf_s
+// LP64-LP64F-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_double_int8_zbf_s
+// LP64D-SAME: (double [[TMP0:%.*]], i8 [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_double_int8_zbf_s(struct double_int8_zbf_s a) {}
+
+// LP64-LP64F-LABEL: define dso_local [2 x i64] @f_ret_double_int8_zbf_s
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local { double, i8 } @f_ret_double_int8_zbf_s
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+struct double_int8_zbf_s f_ret_double_int8_zbf_s(void) {
+  return (struct double_int8_zbf_s){1.0, 2};
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_double_int8_s_arg_insufficient_gprs
+// LP64-LP64F-LP64D-SAME: (i32 noundef signext [[A:%.*]], i32 noundef signext [[B:%.*]], i32 noundef signext [[C:%.*]], i32 noundef signext [[D:%.*]], i32 noundef signext [[E:%.*]], i32 noundef signext [[F:%.*]], i32 noundef signext [[G:%.*]], i32 noundef signext [[H:%.*]], [2 x i64] [[I_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_double_int8_s_arg_insufficient_gprs(int a, int b, int c, int d, int e,
+                                          int f, int g, int h, struct double_int8_s i) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_struct_double_int8_insufficient_fprs
+// LP64-LP64F-LP64D-SAME: (float noundef [[A:%.*]], double noundef [[B:%.*]], double noundef [[C:%.*]], double noundef [[D:%.*]], double noundef [[E:%.*]], double noundef [[F:%.*]], double noundef [[G:%.*]], double noundef [[H:%.*]], [2 x i64] [[I_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_struct_double_int8_insufficient_fprs(float a, double b, double c, double d,
+                                           double e, double f, double g, double h, struct double_int8_s i) {}
+
+// Complex floating-point values or structs containing a single complex
+// floating-point value should be passed as if it were an fp+fp struct.
+
+// LP64-LP64F-LABEL: define dso_local void @f_doublecomplex
+// LP64-LP64F-SAME: ([2 x i64] noundef [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_doublecomplex
+// LP64D-SAME: (double noundef [[A_COERCE0:%.*]], double noundef [[A_COERCE1:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_doublecomplex(double __complex__ a) {}
+
+// LP64-LP64F-LABEL: define dso_local [2 x i64] @f_ret_doublecomplex
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local { double, double } @f_ret_doublecomplex
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+double __complex__ f_ret_doublecomplex(void) {
+  return 1.0;
+}
+
+struct doublecomplex_s { double __complex__ c; };
+
+// LP64-LP64F-LABEL: define dso_local void @f_doublecomplex_s_arg
+// LP64-LP64F-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_doublecomplex_s_arg
+// LP64D-SAME: (double [[TMP0:%.*]], double [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_doublecomplex_s_arg(struct doublecomplex_s a) {}
+
+// LP64-LP64F-LABEL: define dso_local [2 x i64] @f_ret_doublecomplex_s
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local { double, double } @f_ret_doublecomplex_s
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+struct doublecomplex_s f_ret_doublecomplex_s(void) {
+  return (struct doublecomplex_s){1.0};
+}
+
+// Test single or two-element structs that need flattening. e.g. those
+// containing nested structs, doubles in small arrays, zero-length structs etc.
+
+struct doublearr1_s { double a[1]; };
+
+// LP64-LP64F-LABEL: define dso_local void @f_doublearr1_s_arg
+// LP64-LP64F-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_doublearr1_s_arg
+// LP64D-SAME: (double [[TMP0:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_doublearr1_s_arg(struct doublearr1_s a) {}
+
+// LP64-LP64F-LABEL: define dso_local i64 @f_ret_doublearr1_s
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local double @f_ret_doublearr1_s
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+struct doublearr1_s f_ret_doublearr1_s(void) {
+  return (struct doublearr1_s){{1.0}};
+}
+
+struct doublearr2_s { double a[2]; };
+
+// LP64-LP64F-LABEL: define dso_local void @f_doublearr2_s_arg
+// LP64-LP64F-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_doublearr2_s_arg
+// LP64D-SAME: (double [[TMP0:%.*]], double [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_doublearr2_s_arg(struct doublearr2_s a) {}
+
+// LP64-LP64F-LABEL: define dso_local [2 x i64] @f_ret_doublearr2_s
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local { double, double } @f_ret_doublearr2_s
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+struct doublearr2_s f_ret_doublearr2_s(void) {
+  return (struct doublearr2_s){{1.0, 2.0}};
+}
+
+struct doublearr2_tricky1_s { struct { double f[1]; } g[2]; };
+
+// LP64-LP64F-LABEL: define dso_local void @f_doublearr2_tricky1_s_arg
+// LP64-LP64F-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_doublearr2_tricky1_s_arg
+// LP64D-SAME: (double [[TMP0:%.*]], double [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_doublearr2_tricky1_s_arg(struct doublearr2_tricky1_s a) {}
+
+// LP64-LP64F-LABEL: define dso_local [2 x i64] @f_ret_doublearr2_tricky1_s
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local { double, double } @f_ret_doublearr2_tricky1_s
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+struct doublearr2_tricky1_s f_ret_doublearr2_tricky1_s(void) {
+  return (struct doublearr2_tricky1_s){{{{1.0}}, {{2.0}}}};
+}
+
+struct doublearr2_tricky2_s { struct {}; struct { double f[1]; } g[2]; };
+
+// LP64-LP64F-LABEL: define dso_local void @f_doublearr2_tricky2_s_arg
+// LP64-LP64F-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_doublearr2_tricky2_s_arg
+// LP64D-SAME: (double [[TMP0:%.*]], double [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_doublearr2_tricky2_s_arg(struct doublearr2_tricky2_s a) {}
+
+// LP64-LP64F-LABEL: define dso_local [2 x i64] @f_ret_doublearr2_tricky2_s
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local { double, double } @f_ret_doublearr2_tricky2_s
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+struct doublearr2_tricky2_s f_ret_doublearr2_tricky2_s(void) {
+  return (struct doublearr2_tricky2_s){{}, {{{1.0}}, {{2.0}}}};
+}
+
+struct doublearr2_tricky3_s { union {}; struct { double f[1]; } g[2]; };
+
+// LP64-LP64F-LABEL: define dso_local void @f_doublearr2_tricky3_s_arg
+// LP64-LP64F-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_doublearr2_tricky3_s_arg
+// LP64D-SAME: (double [[TMP0:%.*]], double [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_doublearr2_tricky3_s_arg(struct doublearr2_tricky3_s a) {}
+
+// LP64-LP64F-LABEL: define dso_local [2 x i64] @f_ret_doublearr2_tricky3_s
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local { double, double } @f_ret_doublearr2_tricky3_s
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+struct doublearr2_tricky3_s f_ret_doublearr2_tricky3_s(void) {
+  return (struct doublearr2_tricky3_s){{}, {{{1.0}}, {{2.0}}}};
+}
+
+struct doublearr2_tricky4_s { union {}; struct { struct {}; double f[1]; } g[2]; };
+
+// LP64-LP64F-LABEL: define dso_local void @f_doublearr2_tricky4_s_arg
+// LP64-LP64F-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local void @f_doublearr2_tricky4_s_arg
+// LP64D-SAME: (double [[TMP0:%.*]], double [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64D:  entry:
+//
+void f_doublearr2_tricky4_s_arg(struct doublearr2_tricky4_s a) {}
+
+// LP64-LP64F-LABEL: define dso_local [2 x i64] @f_ret_doublearr2_tricky4_s
+// LP64-LP64F-SAME: () #[[ATTR0]] {
+// LP64-LP64F:  entry:
+//
+// LP64D-LABEL: define dso_local { double, double } @f_ret_doublearr2_tricky4_s
+// LP64D-SAME: () #[[ATTR0]] {
+// LP64D:  entry:
+//
+struct doublearr2_tricky4_s f_ret_doublearr2_tricky4_s(void) {
+  return (struct doublearr2_tricky4_s){{}, {{{}, {1.0}}, {{}, {2.0}}}};
+}
+
+// Test structs that should be passed according to the normal integer calling
+// convention.
+
+struct int_double_int_s { int a; double b; int c; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_int_double_int_s_arg
+// LP64-LP64F-LP64D-SAME: (ptr noundef [[A:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_int_double_int_s_arg(struct int_double_int_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_ret_int_double_int_s
+// LP64-LP64F-LP64D-SAME: (ptr noalias sret([[STRUCT_INT_DOUBLE_INT_S:%.*]]) align 8 [[AGG_RESULT:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct int_double_int_s f_ret_int_double_int_s(void) {
+  return (struct int_double_int_s){1, 2.0, 3};
+}
+
+struct char_char_double_s { char a; char b; double c; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_char_char_double_s_arg
+// LP64-LP64F-LP64D-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_char_char_double_s_arg(struct char_char_double_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local [2 x i64] @f_ret_char_char_double_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct char_char_double_s f_ret_char_char_double_s(void) {
+  return (struct char_char_double_s){1, 2, 3.0};
+}
+
+// Unions are always passed according to the integer calling convention, even
+// if they can only contain a double.
+
+union double_u { double a; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_double_u_arg
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_double_u_arg(union double_u a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_double_u
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+union double_u f_ret_double_u(void) {
+  return (union double_u){1.0};
+}
+
+// Check that fp, fp+fp, and int+fp structs are lowered correctly. These will
+// be passed in FPR, FPR+FPR, or GPR+FPR regs if sufficient registers are
+// available the widths are <= XLEN and FLEN, and should be expanded to
+// separate arguments in IR. They are passed by the same rules for returns,
+// but will be lowered to simple two-element structs if necessary (as LLVM IR
+// functions cannot return multiple values).
+// FIXME: Essentially all test cases below involving _Float16 in structs
+// aren't lowered according to the rules in the FP calling convention (i.e.
+// are incorrect for lp64f/lp64d).
+
+struct float16_s { _Float16 f; };
+
+// A struct containing just one floating-point real is passed as though it
+// were a standalone floating-point real.
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_float16_s_arg
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_float16_s_arg(struct float16_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_float16_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct float16_s f_ret_float16_s(void) {
+  return (struct float16_s){1.0};
+}
+
+// A struct containing a double and any number of zero-width bitfields is
+// passed as though it were a standalone floating-point real.
+
+struct zbf_float16_s { int : 0; _Float16 f; };
+struct zbf_float16_zbf_s { int : 0; _Float16 f; int : 0; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_zbf_float16_s_arg
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_zbf_float16_s_arg(struct zbf_float16_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_zbf_float16_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct zbf_float16_s f_ret_zbf_float16_s(void) {
+  return (struct zbf_float16_s){1.0};
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_zbf_float16_zbf_s_arg
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_zbf_float16_zbf_s_arg(struct zbf_float16_zbf_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_zbf_float16_zbf_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct zbf_float16_zbf_s f_ret_zbf_float16_zbf_s(void) {
+  return (struct zbf_float16_zbf_s){1.0};
+}
+
+// Check that structs containing two floating point values (FLEN <= width) are
+// expanded provided sufficient FPRs are available.
+
+struct double_float16_s { double f; _Float16 g; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_double_float16_s_arg
+// LP64-LP64F-LP64D-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_double_float16_s_arg(struct double_float16_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local [2 x i64] @f_ret_double_float16_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct double_float16_s f_ret_double_float16_s(void) {
+  return (struct double_float16_s){1.0, 2.0};
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_double_float16_s_arg_insufficient_fprs
+// LP64-LP64F-LP64D-SAME: (float noundef [[A:%.*]], double noundef [[B:%.*]], double noundef [[C:%.*]], double noundef [[D:%.*]], double noundef [[E:%.*]], double noundef [[F:%.*]], double noundef [[G:%.*]], [2 x i64] [[H_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_double_float16_s_arg_insufficient_fprs(float a, double b, double c, double d,
+    double e, double f, double g, struct double_float16_s h) {}
+
+// Check that structs containing int+_Float16 values are expanded, provided
+// sufficient FPRs and GPRs are available. The integer components are neither
+// sign or zero-extended.
+
+struct float16_int8_s { _Float16 f; int8_t i; };
+struct float16_uint8_s { _Float16 f; uint8_t i; };
+struct float16_int32_s { _Float16 f; int32_t i; };
+struct float16_int64_s { _Float16 f; int64_t i; };
+struct float16_int64bf_s { _Float16 f; int64_t i : 32; };
+struct float16_int8_zbf_s { _Float16 f; int8_t i; int : 0; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_float16_int8_s_arg
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_float16_int8_s_arg(struct float16_int8_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_float16_int8_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct float16_int8_s f_ret_float16_int8_s(void) {
+  return (struct float16_int8_s){1.0, 2};
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_float16_uint8_s_arg
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_float16_uint8_s_arg(struct float16_uint8_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_float16_uint8_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct float16_uint8_s f_ret_float16_uint8_s(void) {
+  return (struct float16_uint8_s){1.0, 2};
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_float16_int32_s_arg
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_float16_int32_s_arg(struct float16_int32_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_float16_int32_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct float16_int32_s f_ret_float16_int32_s(void) {
+  return (struct float16_int32_s){1.0, 2};
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_float16_int64_s_arg
+// LP64-LP64F-LP64D-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_float16_int64_s_arg(struct float16_int64_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local [2 x i64] @f_ret_float16_int64_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct float16_int64_s f_ret_float16_int64_s(void) {
+  return (struct float16_int64_s){1.0, 2};
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_float16_int64bf_s_arg
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_float16_int64bf_s_arg(struct float16_int64bf_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_float16_int64bf_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct float16_int64bf_s f_ret_float16_int64bf_s(void) {
+  return (struct float16_int64bf_s){1.0, 2};
+}
+
+// The zero-width bitfield means the struct can't be passed according to the
+// floating point calling convention.
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_float16_int8_zbf_s
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_float16_int8_zbf_s(struct float16_int8_zbf_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_float16_int8_zbf_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct float16_int8_zbf_s f_ret_float16_int8_zbf_s(void) {
+  return (struct float16_int8_zbf_s){1.0, 2};
+}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_float16_int8_s_arg_insufficient_gprs
+// LP64-LP64F-LP64D-SAME: (i32 noundef signext [[A:%.*]], i32 noundef signext [[B:%.*]], i32 noundef signext [[C:%.*]], i32 noundef signext [[D:%.*]], i32 noundef signext [[E:%.*]], i32 noundef signext [[F:%.*]], i32 noundef signext [[G:%.*]], i32 noundef signext [[H:%.*]], i64 [[I_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_float16_int8_s_arg_insufficient_gprs(int a, int b, int c, int d, int e,
+                                            int f, int g, int h, struct float16_int8_s i) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_struct_float16_int8_insufficient_fprs
+// LP64-LP64F-LP64D-SAME: (float noundef [[A:%.*]], double noundef [[B:%.*]], double noundef [[C:%.*]], double noundef [[D:%.*]], double noundef [[E:%.*]], double noundef [[F:%.*]], double noundef [[G:%.*]], double noundef [[H:%.*]], i64 [[I_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_struct_float16_int8_insufficient_fprs(float a, double b, double c, double d,
+                                             double e, double f, double g, double h, struct float16_int8_s i) {}
+
+// Complex floating-point values or structs containing a single complex
+// floating-point value should be passed as if it were an fp+fp struct.
+
+// LP64-LABEL: define dso_local void @f_float16complex
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_float16complex
+// LP64F-LP64D-SAME: (half noundef [[A_COERCE0:%.*]], half noundef [[A_COERCE1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_float16complex(_Float16 __complex__ a) {}
+
+// LP64-LABEL: define dso_local i64 @f_ret_float16complex
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local { half, half } @f_ret_float16complex
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+_Float16 __complex__ f_ret_float16complex(void) {
+  return 1.0;
+}
+
+struct float16complex_s { _Float16 __complex__ c; };
+
+// LP64-LABEL: define dso_local void @f_float16complex_s_arg
+// LP64-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local void @f_float16complex_s_arg
+// LP64F-LP64D-SAME: (half [[TMP0:%.*]], half [[TMP1:%.*]]) #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+void f_float16complex_s_arg(struct float16complex_s a) {}
+
+// LP64-LABEL: define dso_local i64 @f_ret_float16complex_s
+// LP64-SAME: () #[[ATTR0]] {
+// LP64:  entry:
+//
+// LP64F-LP64D-LABEL: define dso_local { half, half } @f_ret_float16complex_s
+// LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64F-LP64D:  entry:
+//
+struct float16complex_s f_ret_float16complex_s(void) {
+  return (struct float16complex_s){1.0};
+}
+
+// Test single or two-element structs that need flattening. e.g. those
+// containing nested structs, _Float16 in small arrays, zero-length structs etc.
+
+struct float16arr1_s { _Float16 a[1]; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_float16arr1_s_arg
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_float16arr1_s_arg(struct float16arr1_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_float16arr1_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct float16arr1_s f_ret_float16arr1_s(void) {
+  return (struct float16arr1_s){{1.0}};
+}
+
+struct float16arr2_s { _Float16 a[2]; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_float16arr2_s_arg
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_float16arr2_s_arg(struct float16arr2_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_float16arr2_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct float16arr2_s f_ret_float16arr2_s(void) {
+  return (struct float16arr2_s){{1.0, 2.0}};
+}
+
+struct float16arr2_tricky1_s { struct { _Float16 f[1]; } g[2]; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_float16arr2_tricky1_s_arg
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_float16arr2_tricky1_s_arg(struct float16arr2_tricky1_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_float16arr2_tricky1_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct float16arr2_tricky1_s f_ret_float16arr2_tricky1_s(void) {
+  return (struct float16arr2_tricky1_s){{{{1.0}}, {{2.0}}}};
+}
+
+struct float16arr2_tricky2_s { struct {}; struct { _Float16 f[1]; } g[2]; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_float16arr2_tricky2_s_arg
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_float16arr2_tricky2_s_arg(struct float16arr2_tricky2_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_float16arr2_tricky2_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct float16arr2_tricky2_s f_ret_float16arr2_tricky2_s(void) {
+  return (struct float16arr2_tricky2_s){{}, {{{1.0}}, {{2.0}}}};
+}
+
+struct float16arr2_tricky3_s { union {}; struct { _Float16 f[1]; } g[2]; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_float16arr2_tricky3_s_arg
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_float16arr2_tricky3_s_arg(struct float16arr2_tricky3_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_float16arr2_tricky3_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct float16arr2_tricky3_s f_ret_float16arr2_tricky3_s(void) {
+  return (struct float16arr2_tricky3_s){{}, {{{1.0}}, {{2.0}}}};
+}
+
+struct float16arr2_tricky4_s { union {}; struct { struct {}; _Float16 f[1]; } g[2]; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_float16arr2_tricky4_s_arg
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_float16arr2_tricky4_s_arg(struct float16arr2_tricky4_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_float16arr2_tricky4_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct float16arr2_tricky4_s f_ret_float16arr2_tricky4_s(void) {
+  return (struct float16arr2_tricky4_s){{}, {{{}, {1.0}}, {{}, {2.0}}}};
+}
+
+// Test structs that should be passed according to the normal integer calling
+// convention.
+
+struct int_float16_int_s { int a; _Float16 b; int c; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_int_float16_int_s_arg
+// LP64-LP64F-LP64D-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_int_float16_int_s_arg(struct int_float16_int_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local [2 x i64] @f_ret_int_float16_int_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct int_float16_int_s f_ret_int_float16_int_s(void) {
+  return (struct int_float16_int_s){1, 2.0, 3};
+}
+
+struct int64_float16_s { int64_t a; _Float16 b; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_int64_float16_s_arg
+// LP64-LP64F-LP64D-SAME: ([2 x i64] [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_int64_float16_s_arg(struct int64_float16_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local [2 x i64] @f_ret_int64_float16_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct int64_float16_s f_ret_int64_float16_s(void) {
+  return (struct int64_float16_s){1, 2.0};
+}
+
+struct char_char_float16_s { char a; char b; _Float16 c; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_char_char_float16_s_arg
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_char_char_float16_s_arg(struct char_char_float16_s a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_char_char_float16_s
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+struct char_char_float16_s f_ret_char_char_float16_s(void) {
+  return (struct char_char_float16_s){1, 2, 3.0};
+}
+
+// Unions are always passed according to the integer calling convention, even
+// if they can only contain a double.
+
+union float16_u { _Float16 a; };
+
+// LP64-LP64F-LP64D-LABEL: define dso_local void @f_float16_u_arg
+// LP64-LP64F-LP64D-SAME: (i64 [[A_COERCE:%.*]]) #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+void f_float16_u_arg(union float16_u a) {}
+
+// LP64-LP64F-LP64D-LABEL: define dso_local i64 @f_ret_float16_u
+// LP64-LP64F-LP64D-SAME: () #[[ATTR0]] {
+// LP64-LP64F-LP64D:  entry:
+//
+union float16_u f_ret_float16_u(void) {
+  return (union float16_u){1.0};
+}
+
+//// NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+// LP64F: {{.*}}
diff --git a/clang/test/CodeGen/RISCV/riscv64-lp64-abi.c b/clang/test/CodeGen/RISCV/riscv64-lp64-abi.c
deleted file mode 100644
index d3cd9ef1f0116..0000000000000
--- a/clang/test/CodeGen/RISCV/riscv64-lp64-abi.c
+++ /dev/null
@@ -1,49 +0,0 @@
-// RUN: %clang_cc1 -triple riscv64 -emit-llvm %s -o - | FileCheck %s
-
-// This file contains test cases that will have different output for lp64 vs
-// the other 64-bit ABIs.
-
-#include <stddef.h>
-#include <stdint.h>
-
-struct large {
-  int64_t a, b, c, d;
-};
-
-typedef unsigned char v32i8 __attribute__((vector_size(32)));
-
-// Scalars passed on the stack should have signext/zeroext attributes, just as
-// if they were passed in registers.
-
-// CHECK-LABEL: define{{.*}} signext i32 @f_scalar_stack_1(i32 noundef signext %a, i128 noundef %b, float noundef %c, fp128 noundef %d, ptr noundef %0, i8 noundef zeroext %f, i8 noundef signext %g, i8 noundef zeroext %h)
-int f_scalar_stack_1(int32_t a, __int128_t b, float c, long double d, v32i8 e,
-                     uint8_t f, int8_t g, uint8_t h) {
-  return g + h;
-}
-
-// CHECK-LABEL: define{{.*}} void @f_scalar_stack_2(ptr noalias sret(%struct.large) align 8 %agg.result, double noundef %a, i128 noundef %b, fp128 noundef %c, ptr noundef %0, i8 noundef zeroext %e, i8 noundef signext %f, i8 noundef zeroext %g)
-struct large f_scalar_stack_2(double a, __int128_t b, long double c, v32i8 d,
-                              uint8_t e, int8_t f, uint8_t g) {
-  return (struct large){a, e, f, g};
-}
-
-// Complex floating-point values or structs containing a single complex
-// floating-point value should be passed in a GPR.
-
-// CHECK: define{{.*}} void @f_floatcomplex(i64 noundef %a.coerce)
-void f_floatcomplex(float __complex__ a) {}
-
-// CHECK: define{{.*}} i64 @f_ret_floatcomplex()
-float __complex__ f_ret_floatcomplex(void) {
-  return 1.0;
-}
-
-struct floatcomplex_s { float __complex__ c; };
-
-// CHECK: define{{.*}} void @f_floatcomplex_s_arg(i64 %a.coerce)
-void f_floatcomplex_s_arg(struct floatcomplex_s a) {}
-
-// CHECK: define{{.*}} i64 @f_ret_floatcomplex_s()
-struct floatcomplex_s f_ret_floatcomplex_s(void) {
-  return (struct floatcomplex_s){1.0};
-}
diff --git a/clang/test/CodeGen/RISCV/riscv64-lp64-lp64f-abi.c b/clang/test/CodeGen/RISCV/riscv64-lp64-lp64f-abi.c
deleted file mode 100644
index 542ce57d80a7c..0000000000000
--- a/clang/test/CodeGen/RISCV/riscv64-lp64-lp64f-abi.c
+++ /dev/null
@@ -1,30 +0,0 @@
-// RUN: %clang_cc1 -triple riscv64 -emit-llvm %s -o - | FileCheck %s
-// RUN: %clang_cc1 -triple riscv64 -target-feature +f -target-abi lp64f -emit-llvm %s -o - \
-// RUN:     | FileCheck %s
-
-// This file contains test cases that will have the same output for the lp64
-// and lp64f ABIs.
-
-#include <stddef.h>
-#include <stdint.h>
-
-struct large {
-  int64_t a, b, c, d;
-};
-
-typedef unsigned char v32i8 __attribute__((vector_size(32)));
-
-// Scalars passed on the stack should have signext/zeroext attributes, just as
-// if they were passed in registers.
-
-// CHECK-LABEL: define{{.*}} signext i32 @f_scalar_stack_1(i32 noundef signext %a, i128 noundef %b, double noundef %c, fp128 noundef %d, ptr noundef %0, i8 noundef zeroext %f, i8 noundef signext %g, i8 noundef zeroext %h)
-int f_scalar_stack_1(int32_t a, __int128_t b, double c, long double d, v32i8 e,
-                     uint8_t f, int8_t g, uint8_t h) {
-  return g + h;
-}
-
-// CHECK-LABEL: define{{.*}} void @f_scalar_stack_2(ptr noalias sret(%struct.large) align 8 %agg.result, double noundef %a, i128 noundef %b, fp128 noundef %c, ptr noundef %0, i8 noundef zeroext %e, i8 noundef signext %f, i8 noundef zeroext %g)
-struct large f_scalar_stack_2(double a, __int128_t b, long double c, v32i8 d,
-                              uint8_t e, int8_t f, uint8_t g) {
-  return (struct large){a, e, f, g};
-}
diff --git a/clang/test/CodeGen/RISCV/riscv64-lp64-lp64f-lp64d-abi.c b/clang/test/CodeGen/RISCV/riscv64-lp64-lp64f-lp64d-abi.c
deleted file mode 100644
index 44db0ad2cfe69..0000000000000
--- a/clang/test/CodeGen/RISCV/riscv64-lp64-lp64f-lp64d-abi.c
+++ /dev/null
@@ -1,403 +0,0 @@
-// RUN: %clang_cc1 -triple riscv64 -emit-llvm %s -o - | FileCheck %s
-// RUN: %clang_cc1 -triple riscv64 -target-feature +f -target-abi lp64f -emit-llvm %s -o - \
-// RUN:     | FileCheck %s
-// RUN: %clang_cc1 -triple riscv64 -target-feature +d -target-feature +f -target-abi lp64d -emit-llvm %s -o - \
-// RUN:     | FileCheck %s
-
-// This file contains test cases that will have the same output for the lp64,
-// lp64f, and lp64d ABIs.
-
-#include <stddef.h>
-#include <stdint.h>
-
-// CHECK-LABEL: define{{.*}} void @f_void()
-void f_void(void) {}
-
-// Scalar arguments and return values smaller than the word size are extended
-// according to the sign of their type, up to 32 bits
-
-// CHECK-LABEL: define{{.*}} zeroext i1 @f_scalar_0(i1 noundef zeroext %x)
-_Bool f_scalar_0(_Bool x) { return x; }
-
-// CHECK-LABEL: define{{.*}} signext i8 @f_scalar_1(i8 noundef signext %x)
-int8_t f_scalar_1(int8_t x) { return x; }
-
-// CHECK-LABEL: define{{.*}} zeroext i8 @f_scalar_2(i8 noundef zeroext %x)
-uint8_t f_scalar_2(uint8_t x) { return x; }
-
-// CHECK-LABEL: define{{.*}} signext i32 @f_scalar_3(i32 noundef signext %x)
-uint32_t f_scalar_3(int32_t x) { return x; }
-
-// CHECK-LABEL: define{{.*}} i64 @f_scalar_4(i64 noundef %x)
-int64_t f_scalar_4(int64_t x) { return x; }
-
-// CHECK-LABEL: define{{.*}} float @f_fp_scalar_1(float noundef %x)
-float f_fp_scalar_1(float x) { return x; }
-
-// CHECK-LABEL: define{{.*}} double @f_fp_scalar_2(double noundef %x)
-double f_fp_scalar_2(double x) { return x; }
-
-// CHECK-LABEL: define{{.*}} fp128 @f_fp_scalar_3(fp128 noundef %x)
-long double f_fp_scalar_3(long double x) { return x; }
-
-// Empty structs or unions are ignored.
-
-struct empty_s {};
-
-// CHECK-LABEL: define{{.*}} void @f_agg_empty_struct()
-struct empty_s f_agg_empty_struct(struct empty_s x) {
-  return x;
-}
-
-union empty_u {};
-
-// CHECK-LABEL: define{{.*}} void @f_agg_empty_union()
-union empty_u f_agg_empty_union(union empty_u x) {
-  return x;
-}
-
-// Aggregates <= 2*xlen may be passed in registers, so will be coerced to
-// integer arguments. The rules for return are the same.
-
-struct tiny {
-  uint16_t a, b, c, d;
-};
-
-// CHECK-LABEL: define{{.*}} void @f_agg_tiny(i64 %x.coerce)
-void f_agg_tiny(struct tiny x) {
-  x.a += x.b;
-  x.c += x.d;
-}
-
-// CHECK-LABEL: define{{.*}} i64 @f_agg_tiny_ret()
-struct tiny f_agg_tiny_ret(void) {
-  return (struct tiny){1, 2, 3, 4};
-}
-
-typedef uint16_t v4i16 __attribute__((vector_size(8)));
-typedef int64_t v1i64 __attribute__((vector_size(8)));
-
-// CHECK-LABEL: define{{.*}} void @f_vec_tiny_v4i16(i64 noundef %x.coerce)
-void f_vec_tiny_v4i16(v4i16 x) {
-  x[0] = x[1];
-  x[2] = x[3];
-}
-
-// CHECK-LABEL: define{{.*}} i64 @f_vec_tiny_v4i16_ret()
-v4i16 f_vec_tiny_v4i16_ret(void) {
-  return (v4i16){1, 2, 3, 4};
-}
-
-// CHECK-LABEL: define{{.*}} void @f_vec_tiny_v1i64(i64 noundef %x.coerce)
-void f_vec_tiny_v1i64(v1i64 x) {
-  x[0] = 114;
-}
-
-// CHECK-LABEL: define{{.*}} i64 @f_vec_tiny_v1i64_ret()
-v1i64 f_vec_tiny_v1i64_ret(void) {
-  return (v1i64){1};
-}
-
-struct small {
-  int64_t a, *b;
-};
-
-// CHECK-LABEL: define{{.*}} void @f_agg_small([2 x i64] %x.coerce)
-void f_agg_small(struct small x) {
-  x.a += *x.b;
-  x.b = &x.a;
-}
-
-// CHECK-LABEL: define{{.*}} [2 x i64] @f_agg_small_ret()
-struct small f_agg_small_ret(void) {
-  return (struct small){1, 0};
-}
-
-typedef uint16_t v8i16 __attribute__((vector_size(16)));
-typedef __int128_t v1i128 __attribute__((vector_size(16)));
-
-// CHECK-LABEL: define{{.*}} void @f_vec_small_v8i16(i128 noundef %x.coerce)
-void f_vec_small_v8i16(v8i16 x) {
-  x[0] = x[7];
-}
-
-// CHECK-LABEL: define{{.*}} i128 @f_vec_small_v8i16_ret()
-v8i16 f_vec_small_v8i16_ret(void) {
-  return (v8i16){1, 2, 3, 4, 5, 6, 7, 8};
-}
-
-// CHECK-LABEL: define{{.*}} void @f_vec_small_v1i128(i128 noundef %x.coerce)
-void f_vec_small_v1i128(v1i128 x) {
-  x[0] = 114;
-}
-
-// CHECK-LABEL: define{{.*}} i128 @f_vec_small_v1i128_ret()
-v1i128 f_vec_small_v1i128_ret(void) {
-  return (v1i128){1};
-}
-
-// Aggregates of 2*xlen size and 2*xlen alignment should be coerced to a
-// single 2*xlen-sized argument, to ensure that alignment can be maintained if
-// passed on the stack.
-
-struct small_aligned {
-  __int128_t a;
-};
-
-// CHECK-LABEL: define{{.*}} void @f_agg_small_aligned(i128 %x.coerce)
-void f_agg_small_aligned(struct small_aligned x) {
-  x.a += x.a;
-}
-
-// CHECK-LABEL: define{{.*}} i128 @f_agg_small_aligned_ret(i128 %x.coerce)
-struct small_aligned f_agg_small_aligned_ret(struct small_aligned x) {
-  return (struct small_aligned){10};
-}
-
-// Aggregates greater > 2*xlen will be passed and returned indirectly
-struct large {
-  int64_t a, b, c, d;
-};
-
-// CHECK-LABEL: define{{.*}} void @f_agg_large(ptr noundef %x)
-void f_agg_large(struct large x) {
-  x.a = x.b + x.c + x.d;
-}
-
-// The address where the struct should be written to will be the first
-// argument
-// CHECK-LABEL: define{{.*}} void @f_agg_large_ret(ptr noalias sret(%struct.large) align 8 %agg.result, i32 noundef signext %i, i8 noundef signext %j)
-struct large f_agg_large_ret(int32_t i, int8_t j) {
-  return (struct large){1, 2, 3, 4};
-}
-
-typedef unsigned char v32i8 __attribute__((vector_size(32)));
-
-// CHECK-LABEL: define{{.*}} void @f_vec_large_v32i8(ptr noundef %0)
-void f_vec_large_v32i8(v32i8 x) {
-  x[0] = x[7];
-}
-
-// CHECK-LABEL: define{{.*}} void @f_vec_large_v32i8_ret(ptr noalias sret(<32 x i8>) align 32 %agg.result)
-v32i8 f_vec_large_v32i8_ret(void) {
-  return (v32i8){1, 2, 3, 4, 5, 6, 7, 8};
-}
-
-// Scalars passed on the stack should have signext/zeroext attributes, just as
-// if they were passed in registers.
-
-// CHECK-LABEL: define{{.*}} signext i32 @f_scalar_stack_1(i64 %a.coerce, [2 x i64] %b.coerce, i128 %c.coerce, ptr noundef %d, i8 noundef zeroext %e, i8 noundef signext %f, i8 noundef zeroext %g, i8 noundef signext %h)
-int f_scalar_stack_1(struct tiny a, struct small b, struct small_aligned c,
-                     struct large d, uint8_t e, int8_t f, uint8_t g, int8_t h) {
-  return g + h;
-}
-
-// CHECK-LABEL: define{{.*}} signext i32 @f_scalar_stack_2(i32 noundef signext %a, i128 noundef %b, i64 noundef %c, fp128 noundef %d, ptr noundef %0, i8 noundef zeroext %f, i8 noundef signext %g, i8 noundef zeroext %h)
-int f_scalar_stack_2(int32_t a, __int128_t b, int64_t c, long double d, v32i8 e,
-                     uint8_t f, int8_t g, uint8_t h) {
-  return g + h;
-}
-
-// Ensure that scalars passed on the stack are still determined correctly in
-// the presence of large return values that consume a register due to the need
-// to pass a pointer.
-
-// CHECK-LABEL: define{{.*}} void @f_scalar_stack_3(ptr noalias sret(%struct.large) align 8 %agg.result, i32 noundef signext %a, i128 noundef %b, fp128 noundef %c, ptr noundef %0, i8 noundef zeroext %e, i8 noundef signext %f, i8 noundef zeroext %g)
-struct large f_scalar_stack_3(uint32_t a, __int128_t b, long double c, v32i8 d,
-                              uint8_t e, int8_t f, uint8_t g) {
-  return (struct large){a, e, f, g};
-}
-
-// Ensure that ABI lowering happens as expected for vararg calls.
-
-int f_va_callee(int, ...);
-
-// CHECK-LABEL: define{{.*}} void @f_va_caller()
-void f_va_caller(void) {
-  // CHECK: call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i64 noundef 3, double noundef 4.000000e+00, double noundef 5.000000e+00, i64 {{%.*}}, [2 x i64] {{%.*}}, i128 {{%.*}}, ptr noundef {{%.*}})
-  f_va_callee(1, 2, 3LL, 4.0f, 5.0, (struct tiny){6, 7, 8, 9},
-              (struct small){10, NULL}, (struct small_aligned){11},
-              (struct large){12, 13, 14, 15});
-  // CHECK: call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i32 noundef signext 3, i32 noundef signext 4, fp128 noundef 0xL00000000000000004001400000000000, i32 noundef signext 6, i32 noundef signext 7, i32 noundef signext 8, i32 noundef signext 9)
-  f_va_callee(1, 2, 3, 4, 5.0L, 6, 7, 8, 9);
-  // CHECK: call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i32 noundef signext 3, i32 noundef signext 4, i128 {{%.*}}, i32 noundef signext 6, i32 noundef signext 7, i32 noundef signext 8, i32 noundef signext 9)
-  f_va_callee(1, 2, 3, 4, (struct small_aligned){5}, 6, 7, 8, 9);
-  // CHECK: call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i32 noundef signext 3, i32 noundef signext 4, [2 x i64] {{%.*}}, i32 noundef signext 6, i32 noundef signext 7, i32 noundef signext 8, i32 noundef signext 9)
-  f_va_callee(1, 2, 3, 4, (struct small){5, NULL}, 6, 7, 8, 9);
-  // CHECK: call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i32 noundef signext 3, i32 noundef signext 4, i32 noundef signext 5, fp128 noundef 0xL00000000000000004001800000000000, i32 noundef signext 7, i32 noundef signext 8, i32 noundef signext 9)
-  f_va_callee(1, 2, 3, 4, 5, 6.0L, 7, 8, 9);
-  // CHECK: call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i32 noundef signext 3, i32 noundef signext 4, i32 noundef signext 5, i128 {{%.*}}, i32 noundef signext 7, i32 noundef signext 8, i32 noundef signext 9)
-  f_va_callee(1, 2, 3, 4, 5, (struct small_aligned){6}, 7, 8, 9);
-  // CHECK: call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i32 noundef signext 3, i32 noundef signext 4, i32 noundef signext 5, [2 x i64] {{%.*}}, i32 noundef signext 7, i32 noundef signext 8, i32 noundef signext 9)
-  f_va_callee(1, 2, 3, 4, 5, (struct small){6, NULL}, 7, 8, 9);
-  // CHECK: call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i32 noundef signext 3, i32 noundef signext 4, i32 noundef signext 5, i32 noundef signext 6, fp128 noundef 0xL00000000000000004001C00000000000, i32 noundef signext 8, i32 noundef signext 9)
-  f_va_callee(1, 2, 3, 4, 5, 6, 7.0L, 8, 9);
-  // CHECK: call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i32 noundef signext 3, i32 noundef signext 4, i32 noundef signext 5, i32 noundef signext 6, i128 {{%.*}}, i32 noundef signext 8, i32 noundef signext 9)
-  f_va_callee(1, 2, 3, 4, 5, 6, (struct small_aligned){7}, 8, 9);
-  // CHECK: call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i32 noundef signext 3, i32 noundef signext 4, i32 noundef signext 5, i32 noundef signext 6, [2 x i64] {{.*}}, i32 noundef signext 8, i32 noundef signext 9)
-  f_va_callee(1, 2, 3, 4, 5, 6, (struct small){7, NULL}, 8, 9);
-}
-
-// CHECK-LABEL: define{{.*}} signext i32 @f_va_1(ptr noundef %fmt, ...) {{.*}} {
-// CHECK:   [[FMT_ADDR:%.*]] = alloca ptr, align 8
-// CHECK:   [[VA:%.*]] = alloca ptr, align 8
-// CHECK:   [[V:%.*]] = alloca i32, align 4
-// CHECK:   store ptr %fmt, ptr [[FMT_ADDR]], align 8
-// CHECK:   call void @llvm.va_start(ptr [[VA]])
-// CHECK:   [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 8
-// CHECK:   [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR]], i64 8
-// CHECK:   store ptr [[ARGP_NEXT]], ptr [[VA]], align 8
-// CHECK:   [[TMP1:%.*]] = load i32, ptr [[ARGP_CUR]], align 8
-// CHECK:   store i32 [[TMP1]], ptr [[V]], align 4
-// CHECK:   call void @llvm.va_end(ptr [[VA]])
-// CHECK:   [[TMP2:%.*]] = load i32, ptr [[V]], align 4
-// CHECK:   ret i32 [[TMP2]]
-// CHECK: }
-int f_va_1(char *fmt, ...) {
-  __builtin_va_list va;
-
-  __builtin_va_start(va, fmt);
-  int v = __builtin_va_arg(va, int);
-  __builtin_va_end(va);
-
-  return v;
-}
-
-// An "aligned" register pair (where the first register is even-numbered) is
-// used to pass varargs with 2x xlen alignment and 2x xlen size. Ensure the
-// correct offsets are used.
-
-// CHECK-LABEL: @f_va_2(
-// CHECK:         [[FMT_ADDR:%.*]] = alloca ptr, align 8
-// CHECK-NEXT:    [[VA:%.*]] = alloca ptr, align 8
-// CHECK-NEXT:    [[V:%.*]] = alloca fp128, align 16
-// CHECK-NEXT:    store ptr [[FMT:%.*]], ptr [[FMT_ADDR]], align 8
-// CHECK-NEXT:    call void @llvm.va_start(ptr [[VA]])
-// CHECK-NEXT:    [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 8
-// CHECK-NEXT:    [[TMP0:%.*]] = ptrtoint ptr [[ARGP_CUR]] to i64
-// CHECK-NEXT:    [[TMP1:%.*]] = add i64 [[TMP0]], 15
-// CHECK-NEXT:    [[TMP2:%.*]] = and i64 [[TMP1]], -16
-// CHECK-NEXT:    [[ARGP_CUR_ALIGNED:%.*]] = inttoptr i64 [[TMP2]] to ptr
-// CHECK-NEXT:    [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR_ALIGNED]], i64 16
-// CHECK-NEXT:    store ptr [[ARGP_NEXT]], ptr [[VA]], align 8
-// CHECK-NEXT:    [[TMP4:%.*]] = load fp128, ptr [[ARGP_CUR_ALIGNED]], align 16
-// CHECK-NEXT:    store fp128 [[TMP4]], ptr [[V]], align 16
-// CHECK-NEXT:    call void @llvm.va_end(ptr [[VA]])
-// CHECK-NEXT:    [[TMP5:%.*]] = load fp128, ptr [[V]], align 16
-// CHECK-NEXT:    ret fp128 [[TMP5]]
-long double f_va_2(char *fmt, ...) {
-  __builtin_va_list va;
-
-  __builtin_va_start(va, fmt);
-  long double v = __builtin_va_arg(va, long double);
-  __builtin_va_end(va);
-
-  return v;
-}
-
-// Two "aligned" register pairs.
-
-// CHECK-LABEL: @f_va_3(
-// CHECK:         [[FMT_ADDR:%.*]] = alloca ptr, align 8
-// CHECK-NEXT:    [[VA:%.*]] = alloca ptr, align 8
-// CHECK-NEXT:    [[V:%.*]] = alloca fp128, align 16
-// CHECK-NEXT:    [[W:%.*]] = alloca i32, align 4
-// CHECK-NEXT:    [[X:%.*]] = alloca fp128, align 16
-// CHECK-NEXT:    store ptr [[FMT:%.*]], ptr [[FMT_ADDR]], align 8
-// CHECK-NEXT:    call void @llvm.va_start(ptr [[VA]])
-// CHECK-NEXT:    [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 8
-// CHECK-NEXT:    [[TMP0:%.*]] = ptrtoint ptr [[ARGP_CUR]] to i64
-// CHECK-NEXT:    [[TMP1:%.*]] = add i64 [[TMP0]], 15
-// CHECK-NEXT:    [[TMP2:%.*]] = and i64 [[TMP1]], -16
-// CHECK-NEXT:    [[ARGP_CUR_ALIGNED:%.*]] = inttoptr i64 [[TMP2]] to ptr
-// CHECK-NEXT:    [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR_ALIGNED]], i64 16
-// CHECK-NEXT:    store ptr [[ARGP_NEXT]], ptr [[VA]], align 8
-// CHECK-NEXT:    [[TMP4:%.*]] = load fp128, ptr [[ARGP_CUR_ALIGNED]], align 16
-// CHECK-NEXT:    store fp128 [[TMP4]], ptr [[V]], align 16
-// CHECK-NEXT:    [[ARGP_CUR2:%.*]] = load ptr, ptr [[VA]], align 8
-// CHECK-NEXT:    [[ARGP_NEXT3:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR2]], i64 8
-// CHECK-NEXT:    store ptr [[ARGP_NEXT3]], ptr [[VA]], align 8
-// CHECK-NEXT:    [[TMP6:%.*]] = load i32, ptr [[ARGP_CUR2]], align 8
-// CHECK-NEXT:    store i32 [[TMP6]], ptr [[W]], align 4
-// CHECK-NEXT:    [[ARGP_CUR4:%.*]] = load ptr, ptr [[VA]], align 8
-// CHECK-NEXT:    [[TMP7:%.*]] = ptrtoint ptr [[ARGP_CUR4]] to i64
-// CHECK-NEXT:    [[TMP8:%.*]] = add i64 [[TMP7]], 15
-// CHECK-NEXT:    [[TMP9:%.*]] = and i64 [[TMP8]], -16
-// CHECK-NEXT:    [[ARGP_CUR4_ALIGNED:%.*]] = inttoptr i64 [[TMP9]] to ptr
-// CHECK-NEXT:    [[ARGP_NEXT5:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR4_ALIGNED]], i64 16
-// CHECK-NEXT:    store ptr [[ARGP_NEXT5]], ptr [[VA]], align 8
-// CHECK-NEXT:    [[TMP11:%.*]] = load fp128, ptr [[ARGP_CUR4_ALIGNED]], align 16
-// CHECK-NEXT:    store fp128 [[TMP11]], ptr [[X]], align 16
-// CHECK-NEXT:    call void @llvm.va_end(ptr [[VA]])
-// CHECK-NEXT:    [[TMP12:%.*]] = load fp128, ptr [[V]], align 16
-// CHECK-NEXT:    [[TMP13:%.*]] = load fp128, ptr [[X]], align 16
-// CHECK-NEXT:    [[ADD:%.*]] = fadd fp128 [[TMP12]], [[TMP13]]
-// CHECK-NEXT:    ret fp128 [[ADD]]
-long double f_va_3(char *fmt, ...) {
-  __builtin_va_list va;
-
-  __builtin_va_start(va, fmt);
-  long double v = __builtin_va_arg(va, long double);
-  int w = __builtin_va_arg(va, int);
-  long double x = __builtin_va_arg(va, long double);
-  __builtin_va_end(va);
-
-  return v + x;
-}
-
-// CHECK-LABEL: @f_va_4(
-// CHECK:         [[FMT_ADDR:%.*]] = alloca ptr, align 8
-// CHECK-NEXT:    [[VA:%.*]] = alloca ptr, align 8
-// CHECK-NEXT:    [[V:%.*]] = alloca i32, align 4
-// CHECK-NEXT:    [[TS:%.*]] = alloca [[STRUCT_TINY:%.*]], align 2
-// CHECK-NEXT:    [[SS:%.*]] = alloca [[STRUCT_SMALL:%.*]], align 8
-// CHECK-NEXT:    [[LS:%.*]] = alloca [[STRUCT_LARGE:%.*]], align 8
-// CHECK-NEXT:    [[RET:%.*]] = alloca i32, align 4
-// CHECK-NEXT:    store ptr [[FMT:%.*]], ptr [[FMT_ADDR]], align 8
-// CHECK-NEXT:    call void @llvm.va_start(ptr [[VA]])
-// CHECK-NEXT:    [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 8
-// CHECK-NEXT:    [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR]], i64 8
-// CHECK-NEXT:    store ptr [[ARGP_NEXT]], ptr [[VA]], align 8
-// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[ARGP_CUR]], align 8
-// CHECK-NEXT:    store i32 [[TMP1]], ptr [[V]], align 4
-// CHECK-NEXT:    [[ARGP_CUR2:%.*]] = load ptr, ptr [[VA]], align 8
-// CHECK-NEXT:    [[ARGP_NEXT3:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR2]], i64 8
-// CHECK-NEXT:    store ptr [[ARGP_NEXT3]], ptr [[VA]], align 8
-// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 2 [[TS]], ptr align 8 [[ARGP_CUR2]], i64 8, i1 false)
-// CHECK-NEXT:    [[ARGP_CUR4:%.*]] = load ptr, ptr [[VA]], align 8
-// CHECK-NEXT:    [[ARGP_NEXT5:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR4]], i64 16
-// CHECK-NEXT:    store ptr [[ARGP_NEXT5]], ptr [[VA]], align 8
-// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[SS]], ptr align 8 [[ARGP_CUR4]], i64 16, i1 false)
-// CHECK-NEXT:    [[ARGP_CUR6:%.*]] = load ptr, ptr [[VA]], align 8
-// CHECK-NEXT:    [[ARGP_NEXT7:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR6]], i64 8
-// CHECK-NEXT:    store ptr [[ARGP_NEXT7]], ptr [[VA]], align 8
-// CHECK-NEXT:    [[TMP9:%.*]] = load ptr, ptr [[ARGP_CUR6]], align 8
-// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[LS]], ptr align 8 [[TMP9]], i64 32, i1 false)
-// CHECK-NEXT:    call void @llvm.va_end(ptr [[VA]])
-// CHECK-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_TINY]], ptr [[TS]], i32 0, i32 0
-// CHECK-NEXT:    [[TMP12:%.*]] = load i16, ptr [[A]], align 2
-// CHECK-NEXT:    [[CONV:%.*]] = zext i16 [[TMP12]] to i64
-// CHECK-NEXT:    [[A9:%.*]] = getelementptr inbounds [[STRUCT_SMALL]], ptr [[SS]], i32 0, i32 0
-// CHECK-NEXT:    [[TMP13:%.*]] = load i64, ptr [[A9]], align 8
-// CHECK-NEXT:    [[ADD:%.*]] = add nsw i64 [[CONV]], [[TMP13]]
-// CHECK-NEXT:    [[C:%.*]] = getelementptr inbounds [[STRUCT_LARGE]], ptr [[LS]], i32 0, i32 2
-// CHECK-NEXT:    [[TMP14:%.*]] = load i64, ptr [[C]], align 8
-// CHECK-NEXT:    [[ADD10:%.*]] = add nsw i64 [[ADD]], [[TMP14]]
-// CHECK-NEXT:    [[CONV11:%.*]] = trunc i64 [[ADD10]] to i32
-// CHECK-NEXT:    store i32 [[CONV11]], ptr [[RET]], align 4
-// CHECK-NEXT:    [[TMP15:%.*]] = load i32, ptr [[RET]], align 4
-// CHECK-NEXT:    ret i32 [[TMP15]]
-int f_va_4(char *fmt, ...) {
-  __builtin_va_list va;
-
-  __builtin_va_start(va, fmt);
-  int v = __builtin_va_arg(va, int);
-  struct tiny ts = __builtin_va_arg(va, struct tiny);
-  struct small ss = __builtin_va_arg(va, struct small);
-  struct large ls = __builtin_va_arg(va, struct large);
-  __builtin_va_end(va);
-
-  int ret = ts.a + ss.a + ls.c;
-
-  return ret;
-}
diff --git a/clang/test/CodeGen/RISCV/riscv64-lp64d-abi.c b/clang/test/CodeGen/RISCV/riscv64-lp64d-abi.c
deleted file mode 100644
index 11d610a751f18..0000000000000
--- a/clang/test/CodeGen/RISCV/riscv64-lp64d-abi.c
+++ /dev/null
@@ -1,272 +0,0 @@
-// RUN: %clang_cc1 -triple riscv64 -target-feature +d -target-feature +f -target-abi lp64d -emit-llvm %s -o - \
-// RUN:     | FileCheck %s
-
-#include <stdint.h>
-
-// Verify that the tracking of used GPRs and FPRs works correctly by checking
-// that small integers are sign/zero extended when passed in registers.
-
-// Doubles are passed in FPRs, so argument 'i' will be passed zero-extended
-// because it will be passed in a GPR.
-
-// CHECK: define{{.*}} void @f_fpr_tracking(double noundef %a, double noundef %b, double noundef %c, double noundef %d, double noundef %e, double noundef %f, double noundef %g, double noundef %h, i8 noundef zeroext %i)
-void f_fpr_tracking(double a, double b, double c, double d, double e, double f,
-                    double g, double h, uint8_t i) {}
-
-// Check that fp, fp+fp, and int+fp structs are lowered correctly. These will
-// be passed in FPR, FPR+FPR, or GPR+FPR regs if sufficient registers are
-// available the widths are <= XLEN and FLEN, and should be expanded to
-// separate arguments in IR. They are passed by the same rules for returns,
-// but will be lowered to simple two-element structs if necessary (as LLVM IR
-// functions cannot return multiple values).
-
-// A struct containing just one floating-point real is passed as though it
-// were a standalone floating-point real.
-
-struct double_s { double f; };
-
-// CHECK: define{{.*}} void @f_double_s_arg(double %0)
-void f_double_s_arg(struct double_s a) {}
-
-// CHECK: define{{.*}} double @f_ret_double_s()
-struct double_s f_ret_double_s(void) {
-  return (struct double_s){1.0};
-}
-
-// A struct containing a double and any number of zero-width bitfields is
-// passed as though it were a standalone floating-point real.
-
-struct zbf_double_s { int : 0; double f; };
-struct zbf_double_zbf_s { int : 0; double f; int : 0; };
-
-// CHECK: define{{.*}} void @f_zbf_double_s_arg(double %0)
-void f_zbf_double_s_arg(struct zbf_double_s a) {}
-
-// CHECK: define{{.*}} double @f_ret_zbf_double_s()
-struct zbf_double_s f_ret_zbf_double_s(void) {
-  return (struct zbf_double_s){1.0};
-}
-
-// CHECK: define{{.*}} void @f_zbf_double_zbf_s_arg(double %0)
-void f_zbf_double_zbf_s_arg(struct zbf_double_zbf_s a) {}
-
-// CHECK: define{{.*}} double @f_ret_zbf_double_zbf_s()
-struct zbf_double_zbf_s f_ret_zbf_double_zbf_s(void) {
-  return (struct zbf_double_zbf_s){1.0};
-}
-
-// Check that structs containing two floating point values (FLEN <= width) are
-// expanded provided sufficient FPRs are available.
-
-struct double_double_s { double f; double g; };
-struct double_float_s { double f; float g; };
-
-// CHECK: define{{.*}} void @f_double_double_s_arg(double %0, double %1)
-void f_double_double_s_arg(struct double_double_s a) {}
-
-// CHECK: define{{.*}} { double, double } @f_ret_double_double_s()
-struct double_double_s f_ret_double_double_s(void) {
-  return (struct double_double_s){1.0, 2.0};
-}
-
-// CHECK: define{{.*}} void @f_double_float_s_arg(double %0, float %1)
-void f_double_float_s_arg(struct double_float_s a) {}
-
-// CHECK: define{{.*}} { double, float } @f_ret_double_float_s()
-struct double_float_s f_ret_double_float_s(void) {
-  return (struct double_float_s){1.0, 2.0};
-}
-
-// CHECK: define{{.*}} void @f_double_double_s_arg_insufficient_fprs(float noundef %a, double noundef %b, double noundef %c, double noundef %d, double noundef %e, double noundef %f, double noundef %g, [2 x i64] %h.coerce)
-void f_double_double_s_arg_insufficient_fprs(float a, double b, double c, double d,
-    double e, double f, double g, struct double_double_s h) {}
-
-// Check that structs containing int+double values are expanded, provided
-// sufficient FPRs and GPRs are available. The integer components are neither
-// sign or zero-extended.
-
-struct double_int8_s { double f; int8_t i; };
-struct double_uint8_s { double f; uint8_t i; };
-struct double_int32_s { double f; int32_t i; };
-struct double_int64_s { double f; int64_t i; };
-struct double_int128bf_s { double f; __int128_t i : 64; };
-struct double_int8_zbf_s { double f; int8_t i; int : 0; };
-
-// CHECK: define{{.*}} void @f_double_int8_s_arg(double %0, i8 %1)
-void f_double_int8_s_arg(struct double_int8_s a) {}
-
-// CHECK: define{{.*}} { double, i8 } @f_ret_double_int8_s()
-struct double_int8_s f_ret_double_int8_s(void) {
-  return (struct double_int8_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_double_uint8_s_arg(double %0, i8 %1)
-void f_double_uint8_s_arg(struct double_uint8_s a) {}
-
-// CHECK: define{{.*}} { double, i8 } @f_ret_double_uint8_s()
-struct double_uint8_s f_ret_double_uint8_s(void) {
-  return (struct double_uint8_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_double_int32_s_arg(double %0, i32 %1)
-void f_double_int32_s_arg(struct double_int32_s a) {}
-
-// CHECK: define{{.*}} { double, i32 } @f_ret_double_int32_s()
-struct double_int32_s f_ret_double_int32_s(void) {
-  return (struct double_int32_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_double_int64_s_arg(double %0, i64 %1)
-void f_double_int64_s_arg(struct double_int64_s a) {}
-
-// CHECK: define{{.*}} { double, i64 } @f_ret_double_int64_s()
-struct double_int64_s f_ret_double_int64_s(void) {
-  return (struct double_int64_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_double_int128bf_s_arg(double %0, i64 %1)
-void f_double_int128bf_s_arg(struct double_int128bf_s a) {}
-
-// CHECK: define{{.*}} { double, i64 } @f_ret_double_int128bf_s()
-struct double_int128bf_s f_ret_double_int128bf_s(void) {
-  return (struct double_int128bf_s){1.0, 2};
-}
-
-// The zero-width bitfield means the struct can't be passed according to the
-// floating point calling convention.
-
-// CHECK: define{{.*}} void @f_double_int8_zbf_s(double %0, i8 %1)
-void f_double_int8_zbf_s(struct double_int8_zbf_s a) {}
-
-// CHECK: define{{.*}} { double, i8 } @f_ret_double_int8_zbf_s()
-struct double_int8_zbf_s f_ret_double_int8_zbf_s(void) {
-  return (struct double_int8_zbf_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_double_int8_s_arg_insufficient_gprs(i32 noundef signext %a, i32 noundef signext %b, i32 noundef signext %c, i32 noundef signext %d, i32 noundef signext %e, i32 noundef signext %f, i32 noundef signext %g, i32 noundef signext %h, [2 x i64] %i.coerce)
-void f_double_int8_s_arg_insufficient_gprs(int a, int b, int c, int d, int e,
-                                          int f, int g, int h, struct double_int8_s i) {}
-
-// CHECK: define{{.*}} void @f_struct_double_int8_insufficient_fprs(float noundef %a, double noundef %b, double noundef %c, double noundef %d, double noundef %e, double noundef %f, double noundef %g, double noundef %h, [2 x i64] %i.coerce)
-void f_struct_double_int8_insufficient_fprs(float a, double b, double c, double d,
-                                           double e, double f, double g, double h, struct double_int8_s i) {}
-
-// Complex floating-point values or structs containing a single complex
-// floating-point value should be passed as if it were an fp+fp struct.
-
-// CHECK: define{{.*}} void @f_doublecomplex(double noundef %a.coerce0, double noundef %a.coerce1)
-void f_doublecomplex(double __complex__ a) {}
-
-// CHECK: define{{.*}} { double, double } @f_ret_doublecomplex()
-double __complex__ f_ret_doublecomplex(void) {
-  return 1.0;
-}
-
-struct doublecomplex_s { double __complex__ c; };
-
-// CHECK: define{{.*}} void @f_doublecomplex_s_arg(double %0, double %1)
-void f_doublecomplex_s_arg(struct doublecomplex_s a) {}
-
-// CHECK: define{{.*}} { double, double } @f_ret_doublecomplex_s()
-struct doublecomplex_s f_ret_doublecomplex_s(void) {
-  return (struct doublecomplex_s){1.0};
-}
-
-// Test single or two-element structs that need flattening. e.g. those
-// containing nested structs, doubles in small arrays, zero-length structs etc.
-
-struct doublearr1_s { double a[1]; };
-
-// CHECK: define{{.*}} void @f_doublearr1_s_arg(double %0)
-void f_doublearr1_s_arg(struct doublearr1_s a) {}
-
-// CHECK: define{{.*}} double @f_ret_doublearr1_s()
-struct doublearr1_s f_ret_doublearr1_s(void) {
-  return (struct doublearr1_s){{1.0}};
-}
-
-struct doublearr2_s { double a[2]; };
-
-// CHECK: define{{.*}} void @f_doublearr2_s_arg(double %0, double %1)
-void f_doublearr2_s_arg(struct doublearr2_s a) {}
-
-// CHECK: define{{.*}} { double, double } @f_ret_doublearr2_s()
-struct doublearr2_s f_ret_doublearr2_s(void) {
-  return (struct doublearr2_s){{1.0, 2.0}};
-}
-
-struct doublearr2_tricky1_s { struct { double f[1]; } g[2]; };
-
-// CHECK: define{{.*}} void @f_doublearr2_tricky1_s_arg(double %0, double %1)
-void f_doublearr2_tricky1_s_arg(struct doublearr2_tricky1_s a) {}
-
-// CHECK: define{{.*}} { double, double } @f_ret_doublearr2_tricky1_s()
-struct doublearr2_tricky1_s f_ret_doublearr2_tricky1_s(void) {
-  return (struct doublearr2_tricky1_s){{{{1.0}}, {{2.0}}}};
-}
-
-struct doublearr2_tricky2_s { struct {}; struct { double f[1]; } g[2]; };
-
-// CHECK: define{{.*}} void @f_doublearr2_tricky2_s_arg(double %0, double %1)
-void f_doublearr2_tricky2_s_arg(struct doublearr2_tricky2_s a) {}
-
-// CHECK: define{{.*}} { double, double } @f_ret_doublearr2_tricky2_s()
-struct doublearr2_tricky2_s f_ret_doublearr2_tricky2_s(void) {
-  return (struct doublearr2_tricky2_s){{}, {{{1.0}}, {{2.0}}}};
-}
-
-struct doublearr2_tricky3_s { union {}; struct { double f[1]; } g[2]; };
-
-// CHECK: define{{.*}} void @f_doublearr2_tricky3_s_arg(double %0, double %1)
-void f_doublearr2_tricky3_s_arg(struct doublearr2_tricky3_s a) {}
-
-// CHECK: define{{.*}} { double, double } @f_ret_doublearr2_tricky3_s()
-struct doublearr2_tricky3_s f_ret_doublearr2_tricky3_s(void) {
-  return (struct doublearr2_tricky3_s){{}, {{{1.0}}, {{2.0}}}};
-}
-
-struct doublearr2_tricky4_s { union {}; struct { struct {}; double f[1]; } g[2]; };
-
-// CHECK: define{{.*}} void @f_doublearr2_tricky4_s_arg(double %0, double %1)
-void f_doublearr2_tricky4_s_arg(struct doublearr2_tricky4_s a) {}
-
-// CHECK: define{{.*}} { double, double } @f_ret_doublearr2_tricky4_s()
-struct doublearr2_tricky4_s f_ret_doublearr2_tricky4_s(void) {
-  return (struct doublearr2_tricky4_s){{}, {{{}, {1.0}}, {{}, {2.0}}}};
-}
-
-// Test structs that should be passed according to the normal integer calling
-// convention.
-
-struct int_double_int_s { int a; double b; int c; };
-
-// CHECK: define{{.*}} void @f_int_double_int_s_arg(ptr noundef %a)
-void f_int_double_int_s_arg(struct int_double_int_s a) {}
-
-// CHECK: define{{.*}} void @f_ret_int_double_int_s(ptr noalias sret(%struct.int_double_int_s) align 8 %agg.result)
-struct int_double_int_s f_ret_int_double_int_s(void) {
-  return (struct int_double_int_s){1, 2.0, 3};
-}
-
-struct char_char_double_s { char a; char b; double c; };
-
-// CHECK-LABEL: define{{.*}} void @f_char_char_double_s_arg([2 x i64] %a.coerce)
-void f_char_char_double_s_arg(struct char_char_double_s a) {}
-
-// CHECK: define{{.*}} [2 x i64] @f_ret_char_char_double_s()
-struct char_char_double_s f_ret_char_char_double_s(void) {
-  return (struct char_char_double_s){1, 2, 3.0};
-}
-
-// Unions are always passed according to the integer calling convention, even
-// if they can only contain a double.
-
-union double_u { double a; };
-
-// CHECK: define{{.*}} void @f_double_u_arg(i64 %a.coerce)
-void f_double_u_arg(union double_u a) {}
-
-// CHECK: define{{.*}} i64 @f_ret_double_u()
-union double_u f_ret_double_u(void) {
-  return (union double_u){1.0};
-}
diff --git a/clang/test/CodeGen/RISCV/riscv64-lp64f-lp64d-abi.c b/clang/test/CodeGen/RISCV/riscv64-lp64f-lp64d-abi.c
deleted file mode 100644
index e5c85615785cb..0000000000000
--- a/clang/test/CodeGen/RISCV/riscv64-lp64f-lp64d-abi.c
+++ /dev/null
@@ -1,294 +0,0 @@
-// RUN: %clang_cc1 -triple riscv64 -target-feature +f -target-abi lp64f -emit-llvm %s -o - \
-// RUN:     | FileCheck %s
-// RUN: %clang_cc1 -triple riscv64 -target-feature +d -target-feature +f -target-abi lp64d -emit-llvm %s -o - \
-// RUN:     | FileCheck %s
-
-#include <stdint.h>
-
-// Verify that the tracking of used GPRs and FPRs works correctly by checking
-// that small integers are sign/zero extended when passed in registers.
-
-// Floats are passed in FPRs, so argument 'i' will be passed zero-extended
-// because it will be passed in a GPR.
-
-// CHECK: define{{.*}} void @f_fpr_tracking(float noundef %a, float noundef %b, float noundef %c, float noundef %d, float noundef %e, float noundef %f, float noundef %g, float noundef %h, i8 noundef zeroext %i)
-void f_fpr_tracking(float a, float b, float c, float d, float e, float f,
-                    float g, float h, uint8_t i) {}
-
-// Check that fp, fp+fp, and int+fp structs are lowered correctly. These will
-// be passed in FPR, FPR+FPR, or GPR+FPR regs if sufficient registers are
-// available the widths are <= XLEN and FLEN, and should be expanded to
-// separate arguments in IR. They are passed by the same rules for returns,
-// but will be lowered to simple two-element structs if necessary (as LLVM IR
-// functions cannot return multiple values).
-
-// A struct containing just one floating-point real is passed as though it
-// were a standalone floating-point real.
-
-struct float_s { float f; };
-
-// CHECK: define{{.*}} void @f_float_s_arg(float %0)
-void f_float_s_arg(struct float_s a) {}
-
-// CHECK: define{{.*}} float @f_ret_float_s()
-struct float_s f_ret_float_s(void) {
-  return (struct float_s){1.0};
-}
-
-// A struct containing a float and any number of zero-width bitfields is
-// passed as though it were a standalone floating-point real.
-
-struct zbf_float_s { int : 0; float f; };
-struct zbf_float_zbf_s { int : 0; float f; int : 0; };
-
-// CHECK: define{{.*}} void @f_zbf_float_s_arg(float %0)
-void f_zbf_float_s_arg(struct zbf_float_s a) {}
-
-// CHECK: define{{.*}} float @f_ret_zbf_float_s()
-struct zbf_float_s f_ret_zbf_float_s(void) {
-  return (struct zbf_float_s){1.0};
-}
-
-// CHECK: define{{.*}} void @f_zbf_float_zbf_s_arg(float %0)
-void f_zbf_float_zbf_s_arg(struct zbf_float_zbf_s a) {}
-
-// CHECK: define{{.*}} float @f_ret_zbf_float_zbf_s()
-struct zbf_float_zbf_s f_ret_zbf_float_zbf_s(void) {
-  return (struct zbf_float_zbf_s){1.0};
-}
-
-// Check that structs containing two float values (FLEN <= width) are expanded
-// provided sufficient FPRs are available.
-
-struct float_float_s { float f; float g; };
-
-// CHECK: define{{.*}} void @f_float_float_s_arg(float %0, float %1)
-void f_float_float_s_arg(struct float_float_s a) {}
-
-// CHECK: define{{.*}} { float, float } @f_ret_float_float_s()
-struct float_float_s f_ret_float_float_s(void) {
-  return (struct float_float_s){1.0, 2.0};
-}
-
-// CHECK: define{{.*}} void @f_float_float_s_arg_insufficient_fprs(float noundef %a, float noundef %b, float noundef %c, float noundef %d, float noundef %e, float noundef %f, float noundef %g, i64 %h.coerce)
-void f_float_float_s_arg_insufficient_fprs(float a, float b, float c, float d,
-    float e, float f, float g, struct float_float_s h) {}
-
-// Check that structs containing int+float values are expanded, provided
-// sufficient FPRs and GPRs are available. The integer components are neither
-// sign or zero-extended.
-
-struct float_int8_s { float f; int8_t i; };
-struct float_uint8_s { float f; uint8_t i; };
-struct float_int32_s { float f; int32_t i; };
-struct float_int64_s { float f; int64_t i; };
-struct float_int128bf_s { float f; __int128_t i : 64; };
-struct float_int8_zbf_s { float f; int8_t i; int : 0; };
-
-// CHECK: define{{.*}} void @f_float_int8_s_arg(float %0, i8 %1)
-void f_float_int8_s_arg(struct float_int8_s a) {}
-
-// CHECK: define{{.*}} { float, i8 } @f_ret_float_int8_s()
-struct float_int8_s f_ret_float_int8_s(void) {
-  return (struct float_int8_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_float_uint8_s_arg(float %0, i8 %1)
-void f_float_uint8_s_arg(struct float_uint8_s a) {}
-
-// CHECK: define{{.*}} { float, i8 } @f_ret_float_uint8_s()
-struct float_uint8_s f_ret_float_uint8_s(void) {
-  return (struct float_uint8_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_float_int32_s_arg(float %0, i32 %1)
-void f_float_int32_s_arg(struct float_int32_s a) {}
-
-// CHECK: define{{.*}} { float, i32 } @f_ret_float_int32_s()
-struct float_int32_s f_ret_float_int32_s(void) {
-  return (struct float_int32_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_float_int64_s_arg(float %0, i64 %1)
-void f_float_int64_s_arg(struct float_int64_s a) {}
-
-// CHECK: define{{.*}} { float, i64 } @f_ret_float_int64_s()
-struct float_int64_s f_ret_float_int64_s(void) {
-  return (struct float_int64_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_float_int128bf_s_arg(float %0, i64 %1)
-void f_float_int128bf_s_arg(struct float_int128bf_s a) {}
-
-// CHECK: define{{.*}} <{ float, i64 }> @f_ret_float_int128bf_s()
-struct float_int128bf_s f_ret_float_int128bf_s(void) {
-  return (struct float_int128bf_s){1.0, 2};
-}
-
-// The zero-width bitfield means the struct can't be passed according to the
-// floating point calling convention.
-
-// CHECK: define{{.*}} void @f_float_int8_zbf_s(float %0, i8 %1)
-void f_float_int8_zbf_s(struct float_int8_zbf_s a) {}
-
-// CHECK: define{{.*}} { float, i8 } @f_ret_float_int8_zbf_s()
-struct float_int8_zbf_s f_ret_float_int8_zbf_s(void) {
-  return (struct float_int8_zbf_s){1.0, 2};
-}
-
-// CHECK: define{{.*}} void @f_float_int8_s_arg_insufficient_gprs(i32 noundef signext %a, i32 noundef signext %b, i32 noundef signext %c, i32 noundef signext %d, i32 noundef signext %e, i32 noundef signext %f, i32 noundef signext %g, i32 noundef signext %h, i64 %i.coerce)
-void f_float_int8_s_arg_insufficient_gprs(int a, int b, int c, int d, int e,
-                                          int f, int g, int h, struct float_int8_s i) {}
-
-// CHECK: define{{.*}} void @f_struct_float_int8_insufficient_fprs(float noundef %a, float noundef %b, float noundef %c, float noundef %d, float noundef %e, float noundef %f, float noundef %g, float noundef %h, i64 %i.coerce)
-void f_struct_float_int8_insufficient_fprs(float a, float b, float c, float d,
-                                           float e, float f, float g, float h, struct float_int8_s i) {}
-
-// Complex floating-point values or structs containing a single complex
-// floating-point value should be passed as if it were an fp+fp struct.
-
-// CHECK: define{{.*}} void @f_floatcomplex(float noundef %a.coerce0, float noundef %a.coerce1)
-void f_floatcomplex(float __complex__ a) {}
-
-// CHECK: define{{.*}} { float, float } @f_ret_floatcomplex()
-float __complex__ f_ret_floatcomplex(void) {
-  return 1.0;
-}
-
-struct floatcomplex_s { float __complex__ c; };
-
-// CHECK: define{{.*}} void @f_floatcomplex_s_arg(float %0, float %1)
-void f_floatcomplex_s_arg(struct floatcomplex_s a) {}
-
-// CHECK: define{{.*}} { float, float } @f_ret_floatcomplex_s()
-struct floatcomplex_s f_ret_floatcomplex_s(void) {
-  return (struct floatcomplex_s){1.0};
-}
-
-// Complex floating-point values or structs containing a single complex
-// floating-point value should be passed in GPRs if no two FPRs is available.
-
-// CHECK: define{{.*}} void @f_floatcomplex_insufficient_fprs1(float noundef %a.coerce0, float noundef %a.coerce1, float noundef %b.coerce0, float noundef %b.coerce1, float noundef %c.coerce0, float noundef %c.coerce1, float noundef %d.coerce0, float noundef %d.coerce1, i64 noundef %e.coerce)
-void f_floatcomplex_insufficient_fprs1(float __complex__ a, float __complex__ b,
-                                       float __complex__ c, float __complex__ d,
-                                       float __complex__ e) {}
-
-
-// CHECK: define{{.*}} void @f_floatcomplex_s_arg_insufficient_fprs1(float %0, float %1, float %2, float %3, float %4, float %5, float %6, float %7, i64 %e.coerce)
-void f_floatcomplex_s_arg_insufficient_fprs1(struct floatcomplex_s a,
-                                             struct floatcomplex_s b,
-                                             struct floatcomplex_s c,
-                                             struct floatcomplex_s d,
-                                             struct floatcomplex_s e) {}
-
-// CHECK: define{{.*}} void @f_floatcomplex_insufficient_fprs2(float noundef %a, float noundef %b.coerce0, float noundef %b.coerce1, float noundef %c.coerce0, float noundef %c.coerce1, float noundef %d.coerce0, float noundef %d.coerce1, i64 noundef %e.coerce)
-void f_floatcomplex_insufficient_fprs2(float a,
-                                       float __complex__ b, float __complex__ c,
-                                       float __complex__ d, float __complex__ e) {}
-
-
-// CHECK: define{{.*}} void @f_floatcomplex_s_arg_insufficient_fprs2(float noundef %a, float %0, float %1, float %2, float %3, float %4, float %5, i64 %e.coerce)
-void f_floatcomplex_s_arg_insufficient_fprs2(float a,
-                                             struct floatcomplex_s b,
-                                             struct floatcomplex_s c,
-                                             struct floatcomplex_s d,
-                                             struct floatcomplex_s e) {}
-
-// Test single or two-element structs that need flattening. e.g. those
-// containing nested structs, floats in small arrays, zero-length structs etc.
-
-struct floatarr1_s { float a[1]; };
-
-// CHECK: define{{.*}} void @f_floatarr1_s_arg(float %0)
-void f_floatarr1_s_arg(struct floatarr1_s a) {}
-
-// CHECK: define{{.*}} float @f_ret_floatarr1_s()
-struct floatarr1_s f_ret_floatarr1_s(void) {
-  return (struct floatarr1_s){{1.0}};
-}
-
-struct floatarr2_s { float a[2]; };
-
-// CHECK: define{{.*}} void @f_floatarr2_s_arg(float %0, float %1)
-void f_floatarr2_s_arg(struct floatarr2_s a) {}
-
-// CHECK: define{{.*}} { float, float } @f_ret_floatarr2_s()
-struct floatarr2_s f_ret_floatarr2_s(void) {
-  return (struct floatarr2_s){{1.0, 2.0}};
-}
-
-struct floatarr2_tricky1_s { struct { float f[1]; } g[2]; };
-
-// CHECK: define{{.*}} void @f_floatarr2_tricky1_s_arg(float %0, float %1)
-void f_floatarr2_tricky1_s_arg(struct floatarr2_tricky1_s a) {}
-
-// CHECK: define{{.*}} { float, float } @f_ret_floatarr2_tricky1_s()
-struct floatarr2_tricky1_s f_ret_floatarr2_tricky1_s(void) {
-  return (struct floatarr2_tricky1_s){{{{1.0}}, {{2.0}}}};
-}
-
-struct floatarr2_tricky2_s { struct {}; struct { float f[1]; } g[2]; };
-
-// CHECK: define{{.*}} void @f_floatarr2_tricky2_s_arg(float %0, float %1)
-void f_floatarr2_tricky2_s_arg(struct floatarr2_tricky2_s a) {}
-
-// CHECK: define{{.*}} { float, float } @f_ret_floatarr2_tricky2_s()
-struct floatarr2_tricky2_s f_ret_floatarr2_tricky2_s(void) {
-  return (struct floatarr2_tricky2_s){{}, {{{1.0}}, {{2.0}}}};
-}
-
-struct floatarr2_tricky3_s { union {}; struct { float f[1]; } g[2]; };
-
-// CHECK: define{{.*}} void @f_floatarr2_tricky3_s_arg(float %0, float %1)
-void f_floatarr2_tricky3_s_arg(struct floatarr2_tricky3_s a) {}
-
-// CHECK: define{{.*}} { float, float } @f_ret_floatarr2_tricky3_s()
-struct floatarr2_tricky3_s f_ret_floatarr2_tricky3_s(void) {
-  return (struct floatarr2_tricky3_s){{}, {{{1.0}}, {{2.0}}}};
-}
-
-struct floatarr2_tricky4_s { union {}; struct { struct {}; float f[1]; } g[2]; };
-
-// CHECK: define{{.*}} void @f_floatarr2_tricky4_s_arg(float %0, float %1)
-void f_floatarr2_tricky4_s_arg(struct floatarr2_tricky4_s a) {}
-
-// CHECK: define{{.*}} { float, float } @f_ret_floatarr2_tricky4_s()
-struct floatarr2_tricky4_s f_ret_floatarr2_tricky4_s(void) {
-  return (struct floatarr2_tricky4_s){{}, {{{}, {1.0}}, {{}, {2.0}}}};
-}
-
-// Test structs that should be passed according to the normal integer calling
-// convention.
-
-struct int_float_int_s { int a; float b; int c; };
-
-// CHECK: define{{.*}} void @f_int_float_int_s_arg([2 x i64] %a.coerce)
-void f_int_float_int_s_arg(struct int_float_int_s a) {}
-
-// CHECK: define{{.*}} [2 x i64] @f_ret_int_float_int_s()
-struct int_float_int_s f_ret_int_float_int_s(void) {
-  return (struct int_float_int_s){1, 2.0, 3};
-}
-
-struct char_char_float_s { char a; char b; float c; };
-
-// CHECK-LABEL: define{{.*}} void @f_char_char_float_s_arg(i64 %a.coerce)
-void f_char_char_float_s_arg(struct char_char_float_s a) {}
-
-// CHECK: define{{.*}} i64 @f_ret_char_char_float_s()
-struct char_char_float_s f_ret_char_char_float_s(void) {
-  return (struct char_char_float_s){1, 2, 3.0};
-}
-
-// Unions are always passed according to the integer calling convention, even
-// if they can only contain a float.
-
-union float_u { float a; };
-
-// CHECK: define{{.*}} void @f_float_u_arg(i64 %a.coerce)
-void f_float_u_arg(union float_u a) {}
-
-// CHECK: define{{.*}} i64 @f_ret_float_u()
-union float_u f_ret_float_u(void) {
-  return (union float_u){1.0};
-}
diff --git a/clang/test/CodeGen/RISCV/riscv64-vararg.c b/clang/test/CodeGen/RISCV/riscv64-vararg.c
new file mode 100644
index 0000000000000..26261f0095ac9
--- /dev/null
+++ b/clang/test/CodeGen/RISCV/riscv64-vararg.c
@@ -0,0 +1,296 @@
+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --version 2
+// RUN: %clang_cc1 -triple riscv64 -emit-llvm %s -o - | FileCheck %s
+// RUN: %clang_cc1 -triple riscv64 -target-feature +f -target-abi lp64f -emit-llvm %s -o - \
+// RUN:     | FileCheck %s
+// RUN: %clang_cc1 -triple riscv64 -target-feature +d -target-feature +f -target-abi lp64d -emit-llvm %s -o - \
+// RUN:     | FileCheck %s
+
+#include <stddef.h>
+#include <stdint.h>
+
+struct tiny {
+  uint16_t a, b, c, d;
+};
+
+struct small {
+  int64_t a, *b;
+};
+
+struct small_aligned {
+  __int128_t a;
+};
+
+struct large {
+  int64_t a, b, c, d;
+};
+
+// Ensure that ABI lowering happens as expected for vararg calls.
+// Specifically, ensure that signext is emitted for varargs that will be
+// passed in registers but not on the stack. Ensure this takes into account
+// the use of "aligned" register pairs for varargs with 2*xlen alignment.
+
+int f_va_callee(int, ...);
+
+// CHECK-LABEL: define dso_local void @f_va_caller
+// CHECK-SAME: () #[[ATTR0:[0-9]+]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DOTCOMPOUNDLITERAL:%.*]] = alloca [[STRUCT_TINY:%.*]], align 2
+// CHECK-NEXT:    [[DOTCOMPOUNDLITERAL1:%.*]] = alloca [[STRUCT_SMALL:%.*]], align 8
+// CHECK-NEXT:    [[DOTCOMPOUNDLITERAL4:%.*]] = alloca [[STRUCT_SMALL_ALIGNED:%.*]], align 16
+// CHECK-NEXT:    [[DOTCOMPOUNDLITERAL6:%.*]] = alloca [[STRUCT_LARGE:%.*]], align 8
+// CHECK-NEXT:    [[BYVAL_TEMP:%.*]] = alloca [[STRUCT_LARGE]], align 8
+// CHECK-NEXT:    [[DOTCOMPOUNDLITERAL12:%.*]] = alloca [[STRUCT_SMALL_ALIGNED]], align 16
+// CHECK-NEXT:    [[DOTCOMPOUNDLITERAL16:%.*]] = alloca [[STRUCT_SMALL]], align 8
+// CHECK-NEXT:    [[DOTCOMPOUNDLITERAL21:%.*]] = alloca [[STRUCT_SMALL_ALIGNED]], align 16
+// CHECK-NEXT:    [[DOTCOMPOUNDLITERAL25:%.*]] = alloca [[STRUCT_SMALL]], align 8
+// CHECK-NEXT:    [[DOTCOMPOUNDLITERAL30:%.*]] = alloca [[STRUCT_SMALL_ALIGNED]], align 16
+// CHECK-NEXT:    [[DOTCOMPOUNDLITERAL34:%.*]] = alloca [[STRUCT_SMALL]], align 8
+// CHECK-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_TINY]], ptr [[DOTCOMPOUNDLITERAL]], i32 0, i32 0
+// CHECK-NEXT:    store i16 6, ptr [[A]], align 2
+// CHECK-NEXT:    [[B:%.*]] = getelementptr inbounds [[STRUCT_TINY]], ptr [[DOTCOMPOUNDLITERAL]], i32 0, i32 1
+// CHECK-NEXT:    store i16 7, ptr [[B]], align 2
+// CHECK-NEXT:    [[C:%.*]] = getelementptr inbounds [[STRUCT_TINY]], ptr [[DOTCOMPOUNDLITERAL]], i32 0, i32 2
+// CHECK-NEXT:    store i16 8, ptr [[C]], align 2
+// CHECK-NEXT:    [[D:%.*]] = getelementptr inbounds [[STRUCT_TINY]], ptr [[DOTCOMPOUNDLITERAL]], i32 0, i32 3
+// CHECK-NEXT:    store i16 9, ptr [[D]], align 2
+// CHECK-NEXT:    [[A2:%.*]] = getelementptr inbounds [[STRUCT_SMALL]], ptr [[DOTCOMPOUNDLITERAL1]], i32 0, i32 0
+// CHECK-NEXT:    store i64 10, ptr [[A2]], align 8
+// CHECK-NEXT:    [[B3:%.*]] = getelementptr inbounds [[STRUCT_SMALL]], ptr [[DOTCOMPOUNDLITERAL1]], i32 0, i32 1
+// CHECK-NEXT:    store ptr null, ptr [[B3]], align 8
+// CHECK-NEXT:    [[A5:%.*]] = getelementptr inbounds [[STRUCT_SMALL_ALIGNED]], ptr [[DOTCOMPOUNDLITERAL4]], i32 0, i32 0
+// CHECK-NEXT:    store i128 11, ptr [[A5]], align 16
+// CHECK-NEXT:    [[A7:%.*]] = getelementptr inbounds [[STRUCT_LARGE]], ptr [[DOTCOMPOUNDLITERAL6]], i32 0, i32 0
+// CHECK-NEXT:    store i64 12, ptr [[A7]], align 8
+// CHECK-NEXT:    [[B8:%.*]] = getelementptr inbounds [[STRUCT_LARGE]], ptr [[DOTCOMPOUNDLITERAL6]], i32 0, i32 1
+// CHECK-NEXT:    store i64 13, ptr [[B8]], align 8
+// CHECK-NEXT:    [[C9:%.*]] = getelementptr inbounds [[STRUCT_LARGE]], ptr [[DOTCOMPOUNDLITERAL6]], i32 0, i32 2
+// CHECK-NEXT:    store i64 14, ptr [[C9]], align 8
+// CHECK-NEXT:    [[D10:%.*]] = getelementptr inbounds [[STRUCT_LARGE]], ptr [[DOTCOMPOUNDLITERAL6]], i32 0, i32 3
+// CHECK-NEXT:    store i64 15, ptr [[D10]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = load i64, ptr [[DOTCOMPOUNDLITERAL]], align 2
+// CHECK-NEXT:    [[TMP1:%.*]] = load [2 x i64], ptr [[DOTCOMPOUNDLITERAL1]], align 8
+// CHECK-NEXT:    [[COERCE_DIVE:%.*]] = getelementptr inbounds [[STRUCT_SMALL_ALIGNED]], ptr [[DOTCOMPOUNDLITERAL4]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP2:%.*]] = load i128, ptr [[COERCE_DIVE]], align 16
+// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[BYVAL_TEMP]], ptr align 8 [[DOTCOMPOUNDLITERAL6]], i64 32, i1 false)
+// CHECK-NEXT:    [[CALL:%.*]] = call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i64 noundef 3, double noundef 4.000000e+00, double noundef 5.000000e+00, i64 [[TMP0]], [2 x i64] [[TMP1]], i128 [[TMP2]], ptr noundef [[BYVAL_TEMP]])
+// CHECK-NEXT:    [[CALL11:%.*]] = call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i32 noundef signext 3, i32 noundef signext 4, fp128 noundef 0xL00000000000000004001400000000000, i32 noundef signext 6, i32 noundef signext 7, i32 noundef signext 8, i32 noundef signext 9)
+// CHECK-NEXT:    [[A13:%.*]] = getelementptr inbounds [[STRUCT_SMALL_ALIGNED]], ptr [[DOTCOMPOUNDLITERAL12]], i32 0, i32 0
+// CHECK-NEXT:    store i128 5, ptr [[A13]], align 16
+// CHECK-NEXT:    [[COERCE_DIVE14:%.*]] = getelementptr inbounds [[STRUCT_SMALL_ALIGNED]], ptr [[DOTCOMPOUNDLITERAL12]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP3:%.*]] = load i128, ptr [[COERCE_DIVE14]], align 16
+// CHECK-NEXT:    [[CALL15:%.*]] = call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i32 noundef signext 3, i32 noundef signext 4, i128 [[TMP3]], i32 noundef signext 6, i32 noundef signext 7, i32 noundef signext 8, i32 noundef signext 9)
+// CHECK-NEXT:    [[A17:%.*]] = getelementptr inbounds [[STRUCT_SMALL]], ptr [[DOTCOMPOUNDLITERAL16]], i32 0, i32 0
+// CHECK-NEXT:    store i64 5, ptr [[A17]], align 8
+// CHECK-NEXT:    [[B18:%.*]] = getelementptr inbounds [[STRUCT_SMALL]], ptr [[DOTCOMPOUNDLITERAL16]], i32 0, i32 1
+// CHECK-NEXT:    store ptr null, ptr [[B18]], align 8
+// CHECK-NEXT:    [[TMP4:%.*]] = load [2 x i64], ptr [[DOTCOMPOUNDLITERAL16]], align 8
+// CHECK-NEXT:    [[CALL19:%.*]] = call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i32 noundef signext 3, i32 noundef signext 4, [2 x i64] [[TMP4]], i32 noundef signext 6, i32 noundef signext 7, i32 noundef signext 8, i32 noundef signext 9)
+// CHECK-NEXT:    [[CALL20:%.*]] = call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i32 noundef signext 3, i32 noundef signext 4, i32 noundef signext 5, fp128 noundef 0xL00000000000000004001800000000000, i32 noundef signext 7, i32 noundef signext 8, i32 noundef signext 9)
+// CHECK-NEXT:    [[A22:%.*]] = getelementptr inbounds [[STRUCT_SMALL_ALIGNED]], ptr [[DOTCOMPOUNDLITERAL21]], i32 0, i32 0
+// CHECK-NEXT:    store i128 6, ptr [[A22]], align 16
+// CHECK-NEXT:    [[COERCE_DIVE23:%.*]] = getelementptr inbounds [[STRUCT_SMALL_ALIGNED]], ptr [[DOTCOMPOUNDLITERAL21]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP5:%.*]] = load i128, ptr [[COERCE_DIVE23]], align 16
+// CHECK-NEXT:    [[CALL24:%.*]] = call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i32 noundef signext 3, i32 noundef signext 4, i32 noundef signext 5, i128 [[TMP5]], i32 noundef signext 7, i32 noundef signext 8, i32 noundef signext 9)
+// CHECK-NEXT:    [[A26:%.*]] = getelementptr inbounds [[STRUCT_SMALL]], ptr [[DOTCOMPOUNDLITERAL25]], i32 0, i32 0
+// CHECK-NEXT:    store i64 6, ptr [[A26]], align 8
+// CHECK-NEXT:    [[B27:%.*]] = getelementptr inbounds [[STRUCT_SMALL]], ptr [[DOTCOMPOUNDLITERAL25]], i32 0, i32 1
+// CHECK-NEXT:    store ptr null, ptr [[B27]], align 8
+// CHECK-NEXT:    [[TMP6:%.*]] = load [2 x i64], ptr [[DOTCOMPOUNDLITERAL25]], align 8
+// CHECK-NEXT:    [[CALL28:%.*]] = call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i32 noundef signext 3, i32 noundef signext 4, i32 noundef signext 5, [2 x i64] [[TMP6]], i32 noundef signext 7, i32 noundef signext 8, i32 noundef signext 9)
+// CHECK-NEXT:    [[CALL29:%.*]] = call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i32 noundef signext 3, i32 noundef signext 4, i32 noundef signext 5, i32 noundef signext 6, fp128 noundef 0xL00000000000000004001C00000000000, i32 noundef signext 8, i32 noundef signext 9)
+// CHECK-NEXT:    [[A31:%.*]] = getelementptr inbounds [[STRUCT_SMALL_ALIGNED]], ptr [[DOTCOMPOUNDLITERAL30]], i32 0, i32 0
+// CHECK-NEXT:    store i128 7, ptr [[A31]], align 16
+// CHECK-NEXT:    [[COERCE_DIVE32:%.*]] = getelementptr inbounds [[STRUCT_SMALL_ALIGNED]], ptr [[DOTCOMPOUNDLITERAL30]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP7:%.*]] = load i128, ptr [[COERCE_DIVE32]], align 16
+// CHECK-NEXT:    [[CALL33:%.*]] = call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i32 noundef signext 3, i32 noundef signext 4, i32 noundef signext 5, i32 noundef signext 6, i128 [[TMP7]], i32 noundef signext 8, i32 noundef signext 9)
+// CHECK-NEXT:    [[A35:%.*]] = getelementptr inbounds [[STRUCT_SMALL]], ptr [[DOTCOMPOUNDLITERAL34]], i32 0, i32 0
+// CHECK-NEXT:    store i64 7, ptr [[A35]], align 8
+// CHECK-NEXT:    [[B36:%.*]] = getelementptr inbounds [[STRUCT_SMALL]], ptr [[DOTCOMPOUNDLITERAL34]], i32 0, i32 1
+// CHECK-NEXT:    store ptr null, ptr [[B36]], align 8
+// CHECK-NEXT:    [[TMP8:%.*]] = load [2 x i64], ptr [[DOTCOMPOUNDLITERAL34]], align 8
+// CHECK-NEXT:    [[CALL37:%.*]] = call signext i32 (i32, ...) @f_va_callee(i32 noundef signext 1, i32 noundef signext 2, i32 noundef signext 3, i32 noundef signext 4, i32 noundef signext 5, i32 noundef signext 6, [2 x i64] [[TMP8]], i32 noundef signext 8, i32 noundef signext 9)
+// CHECK-NEXT:    ret void
+//
+void f_va_caller(void) {
+  f_va_callee(1, 2, 3LL, 4.0f, 5.0, (struct tiny){6, 7, 8, 9},
+              (struct small){10, NULL}, (struct small_aligned){11},
+              (struct large){12, 13, 14, 15});
+  f_va_callee(1, 2, 3, 4, 5.0L, 6, 7, 8, 9);
+  f_va_callee(1, 2, 3, 4, (struct small_aligned){5}, 6, 7, 8, 9);
+  f_va_callee(1, 2, 3, 4, (struct small){5, NULL}, 6, 7, 8, 9);
+  f_va_callee(1, 2, 3, 4, 5, 6.0L, 7, 8, 9);
+  f_va_callee(1, 2, 3, 4, 5, (struct small_aligned){6}, 7, 8, 9);
+  f_va_callee(1, 2, 3, 4, 5, (struct small){6, NULL}, 7, 8, 9);
+  f_va_callee(1, 2, 3, 4, 5, 6, 7.0L, 8, 9);
+  f_va_callee(1, 2, 3, 4, 5, 6, (struct small_aligned){7}, 8, 9);
+  f_va_callee(1, 2, 3, 4, 5, 6, (struct small){7, NULL}, 8, 9);
+}
+
+// CHECK-LABEL: define dso_local signext i32 @f_va_1
+// CHECK-SAME: (ptr noundef [[FMT:%.*]], ...) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[FMT_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[VA:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[V:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    store ptr [[FMT]], ptr [[FMT_ADDR]], align 8
+// CHECK-NEXT:    call void @llvm.va_start(ptr [[VA]])
+// CHECK-NEXT:    [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 8
+// CHECK-NEXT:    [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR]], i64 8
+// CHECK-NEXT:    store ptr [[ARGP_NEXT]], ptr [[VA]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[ARGP_CUR]], align 8
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[V]], align 4
+// CHECK-NEXT:    call void @llvm.va_end(ptr [[VA]])
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[V]], align 4
+// CHECK-NEXT:    ret i32 [[TMP1]]
+//
+int f_va_1(char *fmt, ...) {
+  __builtin_va_list va;
+
+  __builtin_va_start(va, fmt);
+  int v = __builtin_va_arg(va, int);
+  __builtin_va_end(va);
+
+  return v;
+}
+
+// An "aligned" register pair (where the first register is even-numbered) is
+// used to pass varargs with 2x xlen alignment and 2x xlen size. Ensure the
+// correct offsets are used.
+
+// CHECK-LABEL: define dso_local fp128 @f_va_2
+// CHECK-SAME: (ptr noundef [[FMT:%.*]], ...) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[FMT_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[VA:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[V:%.*]] = alloca fp128, align 16
+// CHECK-NEXT:    store ptr [[FMT]], ptr [[FMT_ADDR]], align 8
+// CHECK-NEXT:    call void @llvm.va_start(ptr [[VA]])
+// CHECK-NEXT:    [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR]], i32 15
+// CHECK-NEXT:    [[ARGP_CUR_ALIGNED:%.*]] = call ptr @llvm.ptrmask.p0.i64(ptr [[TMP0]], i64 -16)
+// CHECK-NEXT:    [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR_ALIGNED]], i64 16
+// CHECK-NEXT:    store ptr [[ARGP_NEXT]], ptr [[VA]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = load fp128, ptr [[ARGP_CUR_ALIGNED]], align 16
+// CHECK-NEXT:    store fp128 [[TMP1]], ptr [[V]], align 16
+// CHECK-NEXT:    call void @llvm.va_end(ptr [[VA]])
+// CHECK-NEXT:    [[TMP2:%.*]] = load fp128, ptr [[V]], align 16
+// CHECK-NEXT:    ret fp128 [[TMP2]]
+//
+long double f_va_2(char *fmt, ...) {
+  __builtin_va_list va;
+
+  __builtin_va_start(va, fmt);
+  long double v = __builtin_va_arg(va, long double);
+  __builtin_va_end(va);
+
+  return v;
+}
+
+// Two "aligned" register pairs.
+
+// CHECK-LABEL: define dso_local fp128 @f_va_3
+// CHECK-SAME: (ptr noundef [[FMT:%.*]], ...) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[FMT_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[VA:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[V:%.*]] = alloca fp128, align 16
+// CHECK-NEXT:    [[W:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[X:%.*]] = alloca fp128, align 16
+// CHECK-NEXT:    store ptr [[FMT]], ptr [[FMT_ADDR]], align 8
+// CHECK-NEXT:    call void @llvm.va_start(ptr [[VA]])
+// CHECK-NEXT:    [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR]], i32 15
+// CHECK-NEXT:    [[ARGP_CUR_ALIGNED:%.*]] = call ptr @llvm.ptrmask.p0.i64(ptr [[TMP0]], i64 -16)
+// CHECK-NEXT:    [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR_ALIGNED]], i64 16
+// CHECK-NEXT:    store ptr [[ARGP_NEXT]], ptr [[VA]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = load fp128, ptr [[ARGP_CUR_ALIGNED]], align 16
+// CHECK-NEXT:    store fp128 [[TMP1]], ptr [[V]], align 16
+// CHECK-NEXT:    [[ARGP_CUR1:%.*]] = load ptr, ptr [[VA]], align 8
+// CHECK-NEXT:    [[ARGP_NEXT2:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR1]], i64 8
+// CHECK-NEXT:    store ptr [[ARGP_NEXT2]], ptr [[VA]], align 8
+// CHECK-NEXT:    [[TMP2:%.*]] = load i32, ptr [[ARGP_CUR1]], align 8
+// CHECK-NEXT:    store i32 [[TMP2]], ptr [[W]], align 4
+// CHECK-NEXT:    [[ARGP_CUR3:%.*]] = load ptr, ptr [[VA]], align 8
+// CHECK-NEXT:    [[TMP3:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR3]], i32 15
+// CHECK-NEXT:    [[ARGP_CUR3_ALIGNED:%.*]] = call ptr @llvm.ptrmask.p0.i64(ptr [[TMP3]], i64 -16)
+// CHECK-NEXT:    [[ARGP_NEXT4:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR3_ALIGNED]], i64 16
+// CHECK-NEXT:    store ptr [[ARGP_NEXT4]], ptr [[VA]], align 8
+// CHECK-NEXT:    [[TMP4:%.*]] = load fp128, ptr [[ARGP_CUR3_ALIGNED]], align 16
+// CHECK-NEXT:    store fp128 [[TMP4]], ptr [[X]], align 16
+// CHECK-NEXT:    call void @llvm.va_end(ptr [[VA]])
+// CHECK-NEXT:    [[TMP5:%.*]] = load fp128, ptr [[V]], align 16
+// CHECK-NEXT:    [[TMP6:%.*]] = load fp128, ptr [[X]], align 16
+// CHECK-NEXT:    [[ADD:%.*]] = fadd fp128 [[TMP5]], [[TMP6]]
+// CHECK-NEXT:    ret fp128 [[ADD]]
+//
+long double f_va_3(char *fmt, ...) {
+  __builtin_va_list va;
+
+  __builtin_va_start(va, fmt);
+  long double v = __builtin_va_arg(va, long double);
+  int w = __builtin_va_arg(va, int);
+  long double x = __builtin_va_arg(va, long double);
+  __builtin_va_end(va);
+
+  return v + x;
+}
+
+// CHECK-LABEL: define dso_local signext i32 @f_va_4
+// CHECK-SAME: (ptr noundef [[FMT:%.*]], ...) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[FMT_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[VA:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[V:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[TS:%.*]] = alloca [[STRUCT_TINY:%.*]], align 2
+// CHECK-NEXT:    [[SS:%.*]] = alloca [[STRUCT_SMALL:%.*]], align 8
+// CHECK-NEXT:    [[LS:%.*]] = alloca [[STRUCT_LARGE:%.*]], align 8
+// CHECK-NEXT:    [[RET:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    store ptr [[FMT]], ptr [[FMT_ADDR]], align 8
+// CHECK-NEXT:    call void @llvm.va_start(ptr [[VA]])
+// CHECK-NEXT:    [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 8
+// CHECK-NEXT:    [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR]], i64 8
+// CHECK-NEXT:    store ptr [[ARGP_NEXT]], ptr [[VA]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[ARGP_CUR]], align 8
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[V]], align 4
+// CHECK-NEXT:    [[ARGP_CUR1:%.*]] = load ptr, ptr [[VA]], align 8
+// CHECK-NEXT:    [[ARGP_NEXT2:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR1]], i64 8
+// CHECK-NEXT:    store ptr [[ARGP_NEXT2]], ptr [[VA]], align 8
+// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 2 [[TS]], ptr align 8 [[ARGP_CUR1]], i64 8, i1 false)
+// CHECK-NEXT:    [[ARGP_CUR3:%.*]] = load ptr, ptr [[VA]], align 8
+// CHECK-NEXT:    [[ARGP_NEXT4:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR3]], i64 16
+// CHECK-NEXT:    store ptr [[ARGP_NEXT4]], ptr [[VA]], align 8
+// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[SS]], ptr align 8 [[ARGP_CUR3]], i64 16, i1 false)
+// CHECK-NEXT:    [[ARGP_CUR5:%.*]] = load ptr, ptr [[VA]], align 8
+// CHECK-NEXT:    [[ARGP_NEXT6:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR5]], i64 8
+// CHECK-NEXT:    store ptr [[ARGP_NEXT6]], ptr [[VA]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = load ptr, ptr [[ARGP_CUR5]], align 8
+// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[LS]], ptr align 8 [[TMP1]], i64 32, i1 false)
+// CHECK-NEXT:    call void @llvm.va_end(ptr [[VA]])
+// CHECK-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_TINY]], ptr [[TS]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP2:%.*]] = load i16, ptr [[A]], align 2
+// CHECK-NEXT:    [[CONV:%.*]] = zext i16 [[TMP2]] to i64
+// CHECK-NEXT:    [[A7:%.*]] = getelementptr inbounds [[STRUCT_SMALL]], ptr [[SS]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP3:%.*]] = load i64, ptr [[A7]], align 8
+// CHECK-NEXT:    [[ADD:%.*]] = add nsw i64 [[CONV]], [[TMP3]]
+// CHECK-NEXT:    [[C:%.*]] = getelementptr inbounds [[STRUCT_LARGE]], ptr [[LS]], i32 0, i32 2
+// CHECK-NEXT:    [[TMP4:%.*]] = load i64, ptr [[C]], align 8
+// CHECK-NEXT:    [[ADD8:%.*]] = add nsw i64 [[ADD]], [[TMP4]]
+// CHECK-NEXT:    [[CONV9:%.*]] = trunc i64 [[ADD8]] to i32
+// CHECK-NEXT:    store i32 [[CONV9]], ptr [[RET]], align 4
+// CHECK-NEXT:    [[TMP5:%.*]] = load i32, ptr [[RET]], align 4
+// CHECK-NEXT:    ret i32 [[TMP5]]
+//
+int f_va_4(char *fmt, ...) {
+  __builtin_va_list va;
+
+  __builtin_va_start(va, fmt);
+  int v = __builtin_va_arg(va, int);
+  struct tiny ts = __builtin_va_arg(va, struct tiny);
+  struct small ss = __builtin_va_arg(va, struct small);
+  struct large ls = __builtin_va_arg(va, struct large);
+  __builtin_va_end(va);
+
+  int ret = ts.a + ss.a + ls.c;
+
+  return ret;
+}
diff --git a/clang/test/CodeGen/SystemZ/sync-builtins-i128-16Al.c b/clang/test/CodeGen/SystemZ/sync-builtins-i128-16Al.c
new file mode 100644
index 0000000000000..1adc17a476c25
--- /dev/null
+++ b/clang/test/CodeGen/SystemZ/sync-builtins-i128-16Al.c
@@ -0,0 +1,206 @@
+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
+// RUN: %clang_cc1 -triple s390x-linux-gnu -O1 -emit-llvm %s -o - \
+// RUN:   | FileCheck %s
+//
+// Test __sync_ builtins for __int128 aligned to 16 bytes.
+
+#include <stdint.h>
+
+__int128 Ptr __attribute__((aligned(16)));
+__int128 Val __attribute__((aligned(16)));
+__int128 OldVal __attribute__((aligned(16)));
+
+// CHECK-LABEL: @f1(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load i128, ptr @Val, align 16, !tbaa [[TBAA2:![0-9]+]]
+// CHECK-NEXT:    [[TMP1:%.*]] = atomicrmw add ptr @Ptr, i128 [[TMP0]] seq_cst, align 16
+// CHECK-NEXT:    store i128 [[TMP1]], ptr [[AGG_RESULT:%.*]], align 8, !tbaa [[TBAA2]]
+// CHECK-NEXT:    ret void
+//
+__int128 f1() {
+  return __sync_fetch_and_add(&Ptr, Val);
+}
+
+// CHECK-LABEL: @f2(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load i128, ptr @Val, align 16, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[TMP1:%.*]] = atomicrmw sub ptr @Ptr, i128 [[TMP0]] seq_cst, align 16
+// CHECK-NEXT:    store i128 [[TMP1]], ptr [[AGG_RESULT:%.*]], align 8, !tbaa [[TBAA2]]
+// CHECK-NEXT:    ret void
+//
+__int128 f2() {
+  return __sync_fetch_and_sub(&Ptr, Val);
+}
+
+// CHECK-LABEL: @f3(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load i128, ptr @Val, align 16, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[TMP1:%.*]] = atomicrmw or ptr @Ptr, i128 [[TMP0]] seq_cst, align 16
+// CHECK-NEXT:    store i128 [[TMP1]], ptr [[AGG_RESULT:%.*]], align 8, !tbaa [[TBAA2]]
+// CHECK-NEXT:    ret void
+//
+__int128 f3() {
+  return __sync_fetch_and_or(&Ptr, Val);
+}
+
+// CHECK-LABEL: @f4(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load i128, ptr @Val, align 16, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[TMP1:%.*]] = atomicrmw and ptr @Ptr, i128 [[TMP0]] seq_cst, align 16
+// CHECK-NEXT:    store i128 [[TMP1]], ptr [[AGG_RESULT:%.*]], align 8, !tbaa [[TBAA2]]
+// CHECK-NEXT:    ret void
+//
+__int128 f4() {
+  return __sync_fetch_and_and(&Ptr, Val);
+}
+
+// CHECK-LABEL: @f5(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load i128, ptr @Val, align 16, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[TMP1:%.*]] = atomicrmw xor ptr @Ptr, i128 [[TMP0]] seq_cst, align 16
+// CHECK-NEXT:    store i128 [[TMP1]], ptr [[AGG_RESULT:%.*]], align 8, !tbaa [[TBAA2]]
+// CHECK-NEXT:    ret void
+//
+__int128 f5() {
+  return __sync_fetch_and_xor(&Ptr, Val);
+}
+
+// CHECK-LABEL: @f6(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load i128, ptr @Val, align 16, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[TMP1:%.*]] = atomicrmw nand ptr @Ptr, i128 [[TMP0]] seq_cst, align 16
+// CHECK-NEXT:    store i128 [[TMP1]], ptr [[AGG_RESULT:%.*]], align 8, !tbaa [[TBAA2]]
+// CHECK-NEXT:    ret void
+//
+__int128 f6() {
+  return __sync_fetch_and_nand(&Ptr, Val);
+}
+
+// CHECK-LABEL: @f7(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load i128, ptr @Val, align 16, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[TMP1:%.*]] = atomicrmw add ptr @Ptr, i128 [[TMP0]] seq_cst, align 16
+// CHECK-NEXT:    [[TMP2:%.*]] = add i128 [[TMP1]], [[TMP0]]
+// CHECK-NEXT:    store i128 [[TMP2]], ptr [[AGG_RESULT:%.*]], align 8, !tbaa [[TBAA2]]
+// CHECK-NEXT:    ret void
+//
+__int128 f7() {
+  return __sync_add_and_fetch(&Ptr, Val);
+}
+
+// CHECK-LABEL: @f8(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load i128, ptr @Val, align 16, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[TMP1:%.*]] = atomicrmw sub ptr @Ptr, i128 [[TMP0]] seq_cst, align 16
+// CHECK-NEXT:    [[TMP2:%.*]] = sub i128 [[TMP1]], [[TMP0]]
+// CHECK-NEXT:    store i128 [[TMP2]], ptr [[AGG_RESULT:%.*]], align 8, !tbaa [[TBAA2]]
+// CHECK-NEXT:    ret void
+//
+__int128 f8() {
+  return __sync_sub_and_fetch(&Ptr, Val);
+}
+
+// CHECK-LABEL: @f9(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load i128, ptr @Val, align 16, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[TMP1:%.*]] = atomicrmw or ptr @Ptr, i128 [[TMP0]] seq_cst, align 16
+// CHECK-NEXT:    [[TMP2:%.*]] = or i128 [[TMP1]], [[TMP0]]
+// CHECK-NEXT:    store i128 [[TMP2]], ptr [[AGG_RESULT:%.*]], align 8, !tbaa [[TBAA2]]
+// CHECK-NEXT:    ret void
+//
+__int128 f9() {
+  return __sync_or_and_fetch(&Ptr, Val);
+}
+
+// CHECK-LABEL: @f10(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load i128, ptr @Val, align 16, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[TMP1:%.*]] = atomicrmw and ptr @Ptr, i128 [[TMP0]] seq_cst, align 16
+// CHECK-NEXT:    [[TMP2:%.*]] = and i128 [[TMP1]], [[TMP0]]
+// CHECK-NEXT:    store i128 [[TMP2]], ptr [[AGG_RESULT:%.*]], align 8, !tbaa [[TBAA2]]
+// CHECK-NEXT:    ret void
+//
+__int128 f10() {
+  return __sync_and_and_fetch(&Ptr, Val);
+}
+
+// CHECK-LABEL: @f11(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load i128, ptr @Val, align 16, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[TMP1:%.*]] = atomicrmw xor ptr @Ptr, i128 [[TMP0]] seq_cst, align 16
+// CHECK-NEXT:    [[TMP2:%.*]] = xor i128 [[TMP1]], [[TMP0]]
+// CHECK-NEXT:    store i128 [[TMP2]], ptr [[AGG_RESULT:%.*]], align 8, !tbaa [[TBAA2]]
+// CHECK-NEXT:    ret void
+//
+__int128 f11() {
+  return __sync_xor_and_fetch(&Ptr, Val);
+}
+
+// CHECK-LABEL: @f12(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load i128, ptr @Val, align 16, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[TMP1:%.*]] = atomicrmw nand ptr @Ptr, i128 [[TMP0]] seq_cst, align 16
+// CHECK-NEXT:    [[TMP2:%.*]] = and i128 [[TMP1]], [[TMP0]]
+// CHECK-NEXT:    [[TMP3:%.*]] = xor i128 [[TMP2]], -1
+// CHECK-NEXT:    store i128 [[TMP3]], ptr [[AGG_RESULT:%.*]], align 8, !tbaa [[TBAA2]]
+// CHECK-NEXT:    ret void
+//
+__int128 f12() {
+  return __sync_nand_and_fetch(&Ptr, Val);
+}
+
+// CHECK-LABEL: @f13(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load i128, ptr @OldVal, align 16, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[TMP1:%.*]] = load i128, ptr @Val, align 16, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[TMP2:%.*]] = cmpxchg ptr @Ptr, i128 [[TMP0]], i128 [[TMP1]] seq_cst seq_cst, align 16
+// CHECK-NEXT:    [[TMP3:%.*]] = extractvalue { i128, i1 } [[TMP2]], 1
+// CHECK-NEXT:    ret i1 [[TMP3]]
+//
+_Bool f13() {
+  return __sync_bool_compare_and_swap(&Ptr, OldVal, Val);
+}
+
+// CHECK-LABEL: @f14(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load i128, ptr @OldVal, align 16, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[TMP1:%.*]] = load i128, ptr @Val, align 16, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[TMP2:%.*]] = cmpxchg ptr @Ptr, i128 [[TMP0]], i128 [[TMP1]] seq_cst seq_cst, align 16
+// CHECK-NEXT:    [[TMP3:%.*]] = extractvalue { i128, i1 } [[TMP2]], 0
+// CHECK-NEXT:    store i128 [[TMP3]], ptr [[AGG_RESULT:%.*]], align 8, !tbaa [[TBAA2]]
+// CHECK-NEXT:    ret void
+//
+__int128 f14() {
+  return __sync_val_compare_and_swap(&Ptr, OldVal, Val);
+}
+
+// CHECK-LABEL: @f15(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load i128, ptr @Val, align 16, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[TMP1:%.*]] = atomicrmw xchg ptr @Ptr, i128 [[TMP0]] seq_cst, align 16
+// CHECK-NEXT:    store i128 [[TMP1]], ptr [[AGG_RESULT:%.*]], align 8, !tbaa [[TBAA2]]
+// CHECK-NEXT:    ret void
+//
+__int128 f15() {
+  return __sync_lock_test_and_set(&Ptr, Val);
+}
+
+// CHECK-LABEL: @f16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    store atomic i128 0, ptr @Ptr release, align 16
+// CHECK-NEXT:    ret void
+//
+void f16() {
+  return __sync_lock_release(&Ptr);
+}
+
+// CHECK-LABEL: @f17(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load i128, ptr @Val, align 16, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[TMP1:%.*]] = atomicrmw xchg ptr @Ptr, i128 [[TMP0]] seq_cst, align 16
+// CHECK-NEXT:    store i128 [[TMP1]], ptr [[AGG_RESULT:%.*]], align 8, !tbaa [[TBAA2]]
+// CHECK-NEXT:    ret void
+//
+__int128 f17() {
+  return __sync_swap(&Ptr, Val);
+}
diff --git a/clang/test/CodeGen/SystemZ/sync-builtins-i128-8Al.c b/clang/test/CodeGen/SystemZ/sync-builtins-i128-8Al.c
new file mode 100644
index 0000000000000..76c9c0ebed2bb
--- /dev/null
+++ b/clang/test/CodeGen/SystemZ/sync-builtins-i128-8Al.c
@@ -0,0 +1,27 @@
+// RUN: %clang_cc1 -triple s390x-linux-gnu -O1 -emit-llvm %s -o - 2>&1 | FileCheck %s
+//
+// Test that an underaligned 16 byte __sync gives a warning.
+
+#include <stdint.h>
+
+__int128 Ptr __attribute__((aligned(8)));
+
+__int128 f1() {
+// CHECK: warning: __sync builtin operation MUST have natural alignment (consider using __atomic). [-Wsync-alignment]
+  return __sync_fetch_and_add(&Ptr, 1);
+}
+
+__int128 f2() {
+// CHECK: warning: __sync builtin operation MUST have natural alignment (consider using __atomic). [-Wsync-alignment]
+  return __sync_sub_and_fetch(&Ptr, 1);
+}
+
+__int128 f3() {
+// CHECK: warning: __sync builtin operation MUST have natural alignment (consider using __atomic). [-Wsync-alignment]
+  return __sync_val_compare_and_swap(&Ptr, 0, 1);
+}
+
+void f4() {
+// CHECK: warning: __sync builtin operation MUST have natural alignment (consider using __atomic). [-Wsync-alignment]
+  __sync_lock_release(&Ptr);
+}
diff --git a/clang/test/CodeGen/WebAssembly/wasm-funcref.c b/clang/test/CodeGen/WebAssembly/wasm-funcref.c
new file mode 100644
index 0000000000000..f01af0db321dd
--- /dev/null
+++ b/clang/test/CodeGen/WebAssembly/wasm-funcref.c
@@ -0,0 +1,99 @@
+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
+// RUN: %clang_cc1 -triple wasm32 -target-feature +reference-types -o - -emit-llvm %s | FileCheck %s
+
+typedef void (*__funcref funcref_t)();
+typedef int (*__funcref fn_funcref_t)(int);
+typedef int (*fn_t)(int);
+
+// Null funcref builtin call
+// CHECK-LABEL: @get_null(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = call ptr addrspace(20) @llvm.wasm.ref.null.func()
+// CHECK-NEXT:    ret ptr addrspace(20) [[TMP0]]
+//
+funcref_t get_null() {
+  return __builtin_wasm_ref_null_func();
+}
+
+// Call to null funcref builtin but requires cast since
+// default return value for builtin is a funcref with function type () -> ().
+// CHECK-LABEL: @get_null_ii(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = call ptr addrspace(20) @llvm.wasm.ref.null.func()
+// CHECK-NEXT:    ret ptr addrspace(20) [[TMP0]]
+//
+fn_funcref_t get_null_ii() {
+  return (fn_funcref_t) __builtin_wasm_ref_null_func();
+}
+
+// Identity function for funcref.
+// CHECK-LABEL: @identity(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[FN_ADDR:%.*]] = alloca ptr addrspace(20), align 4
+// CHECK-NEXT:    store ptr addrspace(20) [[FN:%.*]], ptr [[FN_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load ptr addrspace(20), ptr [[FN_ADDR]], align 4
+// CHECK-NEXT:    ret ptr addrspace(20) [[TMP0]]
+//
+funcref_t identity(funcref_t fn) {
+  return fn;
+}
+
+void helper(funcref_t);
+
+// Pass funcref ref as an argument to a helper function.
+// CHECK-LABEL: @handle(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[FN_ADDR:%.*]] = alloca ptr addrspace(20), align 4
+// CHECK-NEXT:    store ptr addrspace(20) [[FN:%.*]], ptr [[FN_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load ptr addrspace(20), ptr [[FN_ADDR]], align 4
+// CHECK-NEXT:    call void @helper(ptr addrspace(20) noundef [[TMP0]])
+// CHECK-NEXT:    ret i32 0
+//
+int handle(funcref_t fn) {
+  helper(fn);
+  return 0;
+}
+
+// Return funcref from function pointer.
+// CHECK-LABEL: @get_ref(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[FNPTR_ADDR:%.*]] = alloca ptr, align 4
+// CHECK-NEXT:    store ptr [[FNPTR:%.*]], ptr [[FNPTR_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[FNPTR_ADDR]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = addrspacecast ptr [[TMP0]] to ptr addrspace(20)
+// CHECK-NEXT:    ret ptr addrspace(20) [[TMP1]]
+//
+fn_funcref_t get_ref(fn_t fnptr) {
+  return (fn_funcref_t) fnptr;
+}
+
+// Call funcref
+// CHECK-LABEL: @call_fn(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[REF_ADDR:%.*]] = alloca ptr addrspace(20), align 4
+// CHECK-NEXT:    [[X_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    store ptr addrspace(20) [[REF:%.*]], ptr [[REF_ADDR]], align 4
+// CHECK-NEXT:    store i32 [[X:%.*]], ptr [[X_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load ptr addrspace(20), ptr [[REF_ADDR]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[X_ADDR]], align 4
+// CHECK-NEXT:    [[CALL:%.*]] = call addrspace(20) i32 [[TMP0]](i32 noundef [[TMP1]])
+// CHECK-NEXT:    ret i32 [[CALL]]
+//
+int call_fn(fn_funcref_t ref, int x) {
+  return ref(x);
+}
+
+typedef fn_funcref_t (*builtin_refnull_t)();
+
+// Calling ref.null through a function pointer.
+// CHECK-LABEL: @get_null_fptr(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[REFNULL_ADDR:%.*]] = alloca ptr, align 4
+// CHECK-NEXT:    store ptr [[REFNULL:%.*]], ptr [[REFNULL_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[REFNULL_ADDR]], align 4
+// CHECK-NEXT:    [[CALL:%.*]] = call ptr addrspace(20) [[TMP0]]()
+// CHECK-NEXT:    ret ptr addrspace(20) [[CALL]]
+//
+fn_funcref_t get_null_fptr(builtin_refnull_t refnull) {
+  return refnull();
+}
diff --git a/clang/test/CodeGen/WebAssembly/wasm-varargs.c b/clang/test/CodeGen/WebAssembly/wasm-varargs.c
index 3c743692e0942..da22ad6a1b2e9 100644
--- a/clang/test/CodeGen/WebAssembly/wasm-varargs.c
+++ b/clang/test/CodeGen/WebAssembly/wasm-varargs.c
@@ -14,11 +14,11 @@
 // CHECK-NEXT:    [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 4
 // CHECK-NEXT:    [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR]], i32 4
 // CHECK-NEXT:    store ptr [[ARGP_NEXT]], ptr [[VA]], align 4
-// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[ARGP_CUR]], align 4
-// CHECK-NEXT:    store i32 [[TMP1]], ptr [[V]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[ARGP_CUR]], align 4
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[V]], align 4
 // CHECK-NEXT:    call void @llvm.va_end(ptr [[VA]])
-// CHECK-NEXT:    [[TMP2:%.*]] = load i32, ptr [[V]], align 4
-// CHECK-NEXT:    ret i32 [[TMP2]]
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[V]], align 4
+// CHECK-NEXT:    ret i32 [[TMP1]]
 //
 int test_i32(char *fmt, ...) {
   va_list va;
@@ -40,17 +40,15 @@ int test_i32(char *fmt, ...) {
 // CHECK-NEXT:    store ptr [[FMT]], ptr [[FMT_ADDR]], align 4
 // CHECK-NEXT:    call void @llvm.va_start(ptr [[VA]])
 // CHECK-NEXT:    [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 4
-// CHECK-NEXT:    [[TMP0:%.*]] = ptrtoint ptr [[ARGP_CUR]] to i32
-// CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[TMP0]], 7
-// CHECK-NEXT:    [[TMP2:%.*]] = and i32 [[TMP1]], -8
-// CHECK-NEXT:    [[ARGP_CUR_ALIGNED:%.*]] = inttoptr i32 [[TMP2]] to ptr
+// CHECK-NEXT:    [[TMP0:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR]], i32 7
+// CHECK-NEXT:    [[ARGP_CUR_ALIGNED:%.*]] = call ptr @llvm.ptrmask.p0.i32(ptr [[TMP0]], i32 -8)
 // CHECK-NEXT:    [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR_ALIGNED]], i32 8
 // CHECK-NEXT:    store ptr [[ARGP_NEXT]], ptr [[VA]], align 4
-// CHECK-NEXT:    [[TMP4:%.*]] = load i64, ptr [[ARGP_CUR_ALIGNED]], align 8
-// CHECK-NEXT:    store i64 [[TMP4]], ptr [[V]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = load i64, ptr [[ARGP_CUR_ALIGNED]], align 8
+// CHECK-NEXT:    store i64 [[TMP1]], ptr [[V]], align 8
 // CHECK-NEXT:    call void @llvm.va_end(ptr [[VA]])
-// CHECK-NEXT:    [[TMP5:%.*]] = load i64, ptr [[V]], align 8
-// CHECK-NEXT:    ret i64 [[TMP5]]
+// CHECK-NEXT:    [[TMP2:%.*]] = load i64, ptr [[V]], align 8
+// CHECK-NEXT:    ret i64 [[TMP2]]
 //
 long long test_i64(char *fmt, ...) {
   va_list va;
@@ -79,8 +77,8 @@ struct S {
 // CHECK-NEXT:    [[ARGP_CUR:%.*]] = load ptr, ptr [[VA]], align 4
 // CHECK-NEXT:    [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR]], i32 4
 // CHECK-NEXT:    store ptr [[ARGP_NEXT]], ptr [[VA]], align 4
-// CHECK-NEXT:    [[TMP1:%.*]] = load ptr, ptr [[ARGP_CUR]], align 4
-// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[AGG_RESULT]], ptr align 4 [[TMP1]], i32 12, i1 false)
+// CHECK-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[ARGP_CUR]], align 4
+// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[AGG_RESULT]], ptr align 4 [[TMP0]], i32 12, i1 false)
 // CHECK-NEXT:    call void @llvm.va_end(ptr [[VA]])
 // CHECK-NEXT:    ret void
 //
@@ -109,11 +107,11 @@ struct Z {};
 // CHECK-NEXT:    [[ARGP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR]], i32 0
 // CHECK-NEXT:    store ptr [[ARGP_NEXT]], ptr [[VA]], align 4
 // CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 1 [[U]], ptr align 4 [[ARGP_CUR]], i32 0, i1 false)
-// CHECK-NEXT:    [[ARGP_CUR2:%.*]] = load ptr, ptr [[VA]], align 4
-// CHECK-NEXT:    [[ARGP_NEXT3:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR2]], i32 4
-// CHECK-NEXT:    store ptr [[ARGP_NEXT3]], ptr [[VA]], align 4
-// CHECK-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[ARGP_CUR2]], align 4
-// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[AGG_RESULT]], ptr align 4 [[TMP4]], i32 12, i1 false)
+// CHECK-NEXT:    [[ARGP_CUR1:%.*]] = load ptr, ptr [[VA]], align 4
+// CHECK-NEXT:    [[ARGP_NEXT2:%.*]] = getelementptr inbounds i8, ptr [[ARGP_CUR1]], i32 4
+// CHECK-NEXT:    store ptr [[ARGP_NEXT2]], ptr [[VA]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[ARGP_CUR1]], align 4
+// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[AGG_RESULT]], ptr align 4 [[TMP0]], i32 12, i1 false)
 // CHECK-NEXT:    call void @llvm.va_end(ptr [[VA]])
 // CHECK-NEXT:    ret void
 //
diff --git a/clang/test/CodeGen/X86/fexcess-precision.c b/clang/test/CodeGen/X86/fexcess-precision.c
index 325706830f894..68c2f76260918 100644
--- a/clang/test/CodeGen/X86/fexcess-precision.c
+++ b/clang/test/CodeGen/X86/fexcess-precision.c
@@ -380,7 +380,7 @@ _Float16 f(_Float16 a, _Float16 b, _Float16 c, _Float16 d) {
 //
 // CHECK-UNSAFE-LABEL: @getFEM(
 // CHECK-UNSAFE-NEXT:  entry:
-// CHECK-UNSAFE-NEXT:    ret i32 -1
+// CHECK-UNSAFE-NEXT:    ret i32 0
 //
 int getFEM() {
   return __FLT_EVAL_METHOD__;
diff --git a/clang/test/CodeGen/X86/x86_32-arguments-darwin.c b/clang/test/CodeGen/X86/x86_32-arguments-darwin.c
index f4db6b27dabbe..ae385c4121dee 100644
--- a/clang/test/CodeGen/X86/x86_32-arguments-darwin.c
+++ b/clang/test/CodeGen/X86/x86_32-arguments-darwin.c
@@ -304,9 +304,8 @@ struct s62 { T62 x; int y; } __attribute((packed, aligned(8)));
 void f62(int x, struct s62 y) {}
 
 // CHECK-LABEL: define{{.*}} i32 @f63
-// CHECK: ptrtoint
-// CHECK: and {{.*}}, -16
-// CHECK: inttoptr
+// CHECK: getelementptr inbounds i8, ptr {{.*}}, i32 15
+// CHECK: call ptr @llvm.ptrmask.p0.i32(ptr {{.*}}, i32 -16)
 typedef int T63 __attribute((vector_size(16)));
 struct s63 { T63 x; int y; };
 int f63(int i, ...) {
@@ -326,9 +325,8 @@ struct s65 { signed char a[0]; float b; };
 struct s65 f65(void) { return (struct s65){{},2}; }
 
 // CHECK-LABEL: define{{.*}} <2 x i64> @f66
-// CHECK: ptrtoint
-// CHECK: and {{.*}}, -16
-// CHECK: inttoptr
+// CHECK: getelementptr inbounds i8, ptr {{.*}}, i32 15
+// CHECK: call ptr @llvm.ptrmask.p0.i32(ptr {{.*}}, i32 -16)
 typedef int T66 __attribute((vector_size(16)));
 T66 f66(int i, ...) {
   __builtin_va_list ap;
diff --git a/clang/test/CodeGen/X86/x86_64-arguments.c b/clang/test/CodeGen/X86/x86_64-arguments.c
index d42a7d7cfdeac..fe31bfa9e778b 100644
--- a/clang/test/CodeGen/X86/x86_64-arguments.c
+++ b/clang/test/CodeGen/X86/x86_64-arguments.c
@@ -335,10 +335,8 @@ void func43(SA s) {
 }
 
 // CHECK-LABEL: define{{.*}} i32 @f44
-// CHECK: ptrtoint
-// CHECK-NEXT: add i64 %{{[0-9]+}}, 31
-// CHECK-NEXT: and i64 %{{[0-9]+}}, -32
-// CHECK-NEXT: inttoptr
+// CHECK: getelementptr inbounds i8, ptr %{{.+}}, i32 31
+// CHECK-NEXT: call ptr @llvm.ptrmask.p0.i64(ptr %{{[0-9]+}}, i64 -32)
 typedef int T44 __attribute((vector_size(32)));
 struct s44 { T44 x; int y; };
 int f44(int i, ...) {
diff --git a/clang/test/CodeGen/aarch64-debug-sve-vector-types.c b/clang/test/CodeGen/aarch64-debug-sve-vector-types.c
index b2c5b78f93949..ca592b1467a5c 100644
--- a/clang/test/CodeGen/aarch64-debug-sve-vector-types.c
+++ b/clang/test/CodeGen/aarch64-debug-sve-vector-types.c
@@ -9,6 +9,12 @@ void test_locals(void) {
   // CHECK-DAG: ![[REALELTS1_64]] = !DISubrange(lowerBound: 0, upperBound: !DIExpression(DW_OP_constu, 1, DW_OP_bregx, 46, 0, DW_OP_mul, DW_OP_constu, 1, DW_OP_minus))
   __SVBool_t b8;
 
+  // CHECK-DAG: name: "__SVCount_t",{{.*}}, baseType: ![[CT1_2:[0-9]+]]
+  // CHECK-DAG: ![[CT1_2]] = !DICompositeType(tag: DW_TAG_array_type, baseType: ![[ELTTYU8]], flags: DIFlagVector, elements: ![[ELTS0_to_1:[0-9]+]])
+  // CHECK-DAG: ![[ELTS0_to_1]] = !{![[REALELTS0_to_1:[0-9]+]]}
+  // CHECK-DAG: ![[REALELTS0_to_1]] = !DISubrange(lowerBound: 0, upperBound: 1)
+  __SVCount_t c8;
+
   // CHECK-DAG: name: "__SVInt8_t",{{.*}}, baseType: ![[CT8:[0-9]+]]
   // CHECK-DAG: ![[CT8]] = !DICompositeType(tag: DW_TAG_array_type, baseType: ![[ELTTYS8:[0-9]+]], flags: DIFlagVector, elements: ![[ELTS8:[0-9]+]])
   // CHECK-DAG: ![[ELTTYS8]] = !DIBasicType(name: "signed char", size: 8, encoding: DW_ATE_signed_char)
diff --git a/clang/test/CodeGen/aarch64-sve-inline-asm-crash.c b/clang/test/CodeGen/aarch64-sve-inline-asm-crash.c
index 3496c7368ad0d..b831570245179 100644
--- a/clang/test/CodeGen/aarch64-sve-inline-asm-crash.c
+++ b/clang/test/CodeGen/aarch64-sve-inline-asm-crash.c
@@ -20,5 +20,17 @@ __SVBool_t funcB1(__SVBool_t in)
   return ret ;
 }
 
+__SVCount_t funcB1(__SVCount_t in)
+{
+  __SVCount_t ret ;
+  asm volatile (
+    "mov %[ret].b, %[in].b \n"
+    : [ret] "=w" (ret)
+    : [in] "w" (in)
+    :);
+
+  return ret ;
+}
+
 // CHECK: funcB1
 // CHECK-ERROR: fatal error: error in backend: Cannot select
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_abd.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_abd.c
index e1fc69b23b574..be9fb775842a5 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_abd.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_abd.c
@@ -1045,13 +1045,13 @@ svfloat64_t test_svabd_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
 // CHECK-LABEL: @test_svabd_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fabd.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fabd.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svabd_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fabd.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fabd.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svabd_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
@@ -1062,13 +1062,13 @@ svfloat16_t test_svabd_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
 // CHECK-LABEL: @test_svabd_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fabd.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fabd.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svabd_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fabd.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fabd.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svabd_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
@@ -1079,13 +1079,13 @@ svfloat32_t test_svabd_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
 // CHECK-LABEL: @test_svabd_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fabd.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fabd.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svabd_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fabd.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fabd.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svabd_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
@@ -1230,7 +1230,7 @@ svfloat64_t test_svabd_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fabd.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fabd.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svabd_n_f16_xu10__SVBool_tu13__SVFloat16_tDh(
@@ -1238,7 +1238,7 @@ svfloat64_t test_svabd_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fabd.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fabd.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svabd_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
@@ -1251,7 +1251,7 @@ svfloat16_t test_svabd_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fabd.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fabd.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svabd_n_f32_xu10__SVBool_tu13__SVFloat32_tf(
@@ -1259,7 +1259,7 @@ svfloat16_t test_svabd_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fabd.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fabd.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svabd_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
@@ -1272,7 +1272,7 @@ svfloat32_t test_svabd_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fabd.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fabd.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svabd_n_f64_xu10__SVBool_tu13__SVFloat64_td(
@@ -1280,7 +1280,7 @@ svfloat32_t test_svabd_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fabd.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fabd.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svabd_n_f64_x(svbool_t pg, svfloat64_t op1, float64_t op2)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_add.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_add.c
index a39b5b26b1ce4..19bdc9d0816bf 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_add.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_add.c
@@ -1045,13 +1045,13 @@ svfloat64_t test_svadd_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
 // CHECK-LABEL: @test_svadd_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fadd.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fadd.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svadd_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fadd.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fadd.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svadd_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
@@ -1062,13 +1062,13 @@ svfloat16_t test_svadd_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
 // CHECK-LABEL: @test_svadd_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fadd.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fadd.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svadd_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fadd.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fadd.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svadd_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
@@ -1079,13 +1079,13 @@ svfloat32_t test_svadd_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
 // CHECK-LABEL: @test_svadd_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fadd.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fadd.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svadd_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fadd.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fadd.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svadd_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
@@ -1230,7 +1230,7 @@ svfloat64_t test_svadd_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fadd.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fadd.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svadd_n_f16_xu10__SVBool_tu13__SVFloat16_tDh(
@@ -1238,7 +1238,7 @@ svfloat64_t test_svadd_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fadd.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fadd.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svadd_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
@@ -1251,7 +1251,7 @@ svfloat16_t test_svadd_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fadd.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fadd.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svadd_n_f32_xu10__SVBool_tu13__SVFloat32_tf(
@@ -1259,7 +1259,7 @@ svfloat16_t test_svadd_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fadd.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fadd.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svadd_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
@@ -1272,7 +1272,7 @@ svfloat32_t test_svadd_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fadd.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fadd.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svadd_n_f64_xu10__SVBool_tu13__SVFloat64_td(
@@ -1280,7 +1280,7 @@ svfloat32_t test_svadd_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fadd.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fadd.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svadd_n_f64_x(svbool_t pg, svfloat64_t op1, float64_t op2)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_div.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_div.c
index 294a5e855fc2e..7774e14c9c69e 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_div.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_div.c
@@ -597,13 +597,13 @@ svfloat64_t test_svdiv_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
 // CHECK-LABEL: @test_svdiv_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fdiv.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fdiv.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svdiv_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fdiv.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fdiv.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svdiv_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
@@ -614,13 +614,13 @@ svfloat16_t test_svdiv_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
 // CHECK-LABEL: @test_svdiv_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fdiv.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fdiv.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svdiv_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fdiv.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fdiv.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svdiv_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
@@ -631,13 +631,13 @@ svfloat32_t test_svdiv_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
 // CHECK-LABEL: @test_svdiv_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fdiv.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fdiv.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svdiv_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fdiv.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fdiv.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svdiv_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
@@ -782,7 +782,7 @@ svfloat64_t test_svdiv_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fdiv.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fdiv.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svdiv_n_f16_xu10__SVBool_tu13__SVFloat16_tDh(
@@ -790,7 +790,7 @@ svfloat64_t test_svdiv_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fdiv.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fdiv.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svdiv_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
@@ -803,7 +803,7 @@ svfloat16_t test_svdiv_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fdiv.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fdiv.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svdiv_n_f32_xu10__SVBool_tu13__SVFloat32_tf(
@@ -811,7 +811,7 @@ svfloat16_t test_svdiv_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fdiv.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fdiv.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svdiv_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
@@ -824,7 +824,7 @@ svfloat32_t test_svdiv_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fdiv.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fdiv.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svdiv_n_f64_xu10__SVBool_tu13__SVFloat64_td(
@@ -832,7 +832,7 @@ svfloat32_t test_svdiv_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fdiv.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fdiv.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svdiv_n_f64_x(svbool_t pg, svfloat64_t op1, float64_t op2)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_divr.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_divr.c
index 31b1938a8d12d..3bcf95fb33f35 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_divr.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_divr.c
@@ -597,13 +597,13 @@ svfloat64_t test_svdivr_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
 // CHECK-LABEL: @test_svdivr_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fdivr.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fdiv.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svdivr_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fdivr.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fdiv.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svdivr_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
@@ -614,13 +614,13 @@ svfloat16_t test_svdivr_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
 // CHECK-LABEL: @test_svdivr_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fdivr.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fdiv.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svdivr_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fdivr.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fdiv.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svdivr_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
@@ -631,13 +631,13 @@ svfloat32_t test_svdivr_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
 // CHECK-LABEL: @test_svdivr_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fdivr.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fdiv.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svdivr_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fdivr.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fdiv.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svdivr_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
@@ -782,7 +782,7 @@ svfloat64_t test_svdivr_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fdivr.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fdiv.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[DOTSPLAT]], <vscale x 8 x half> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svdivr_n_f16_xu10__SVBool_tu13__SVFloat16_tDh(
@@ -790,7 +790,7 @@ svfloat64_t test_svdivr_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fdivr.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fdiv.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[DOTSPLAT]], <vscale x 8 x half> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svdivr_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
@@ -803,7 +803,7 @@ svfloat16_t test_svdivr_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fdivr.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fdiv.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[DOTSPLAT]], <vscale x 4 x float> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svdivr_n_f32_xu10__SVBool_tu13__SVFloat32_tf(
@@ -811,7 +811,7 @@ svfloat16_t test_svdivr_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fdivr.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fdiv.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[DOTSPLAT]], <vscale x 4 x float> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svdivr_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
@@ -824,7 +824,7 @@ svfloat32_t test_svdivr_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fdivr.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fdiv.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[DOTSPLAT]], <vscale x 2 x double> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svdivr_n_f64_xu10__SVBool_tu13__SVFloat64_td(
@@ -832,7 +832,7 @@ svfloat32_t test_svdivr_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fdivr.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fdiv.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[DOTSPLAT]], <vscale x 2 x double> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svdivr_n_f64_x(svbool_t pg, svfloat64_t op1, float64_t op2)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mad.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mad.c
index cfd5b34c9d1c7..97ac1ad20cdaa 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mad.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mad.c
@@ -1045,13 +1045,13 @@ svfloat64_t test_svmad_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, svfl
 // CHECK-LABEL: @test_svmad_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmad.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmla.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP3:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmad_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmad.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmla.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP3:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svmad_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, svfloat16_t op3)
@@ -1062,13 +1062,13 @@ svfloat16_t test_svmad_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, svfl
 // CHECK-LABEL: @test_svmad_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmad.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmla.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP3:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmad_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmad.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmla.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP3:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svmad_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, svfloat32_t op3)
@@ -1079,13 +1079,13 @@ svfloat32_t test_svmad_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, svfl
 // CHECK-LABEL: @test_svmad_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmad.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmla.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP3:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmad_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmad.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmla.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP3:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svmad_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2, svfloat64_t op3)
@@ -1230,7 +1230,7 @@ svfloat64_t test_svmad_n_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, fl
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmad.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmla.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[DOTSPLAT]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmad_n_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_tDh(
@@ -1238,7 +1238,7 @@ svfloat64_t test_svmad_n_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, fl
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmad.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmla.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[DOTSPLAT]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svmad_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, float16_t op3)
@@ -1251,7 +1251,7 @@ svfloat16_t test_svmad_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, fl
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmad.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmla.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[DOTSPLAT]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmad_n_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_tf(
@@ -1259,7 +1259,7 @@ svfloat16_t test_svmad_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, fl
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmad.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmla.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[DOTSPLAT]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svmad_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, float32_t op3)
@@ -1272,7 +1272,7 @@ svfloat32_t test_svmad_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, fl
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmad.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmla.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[DOTSPLAT]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmad_n_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_td(
@@ -1280,7 +1280,7 @@ svfloat32_t test_svmad_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, fl
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmad.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmla.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[DOTSPLAT]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svmad_n_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2, float64_t op3)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_max.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_max.c
index 43668032a1c56..78a331bd04f05 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_max.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_max.c
@@ -1045,13 +1045,13 @@ svfloat64_t test_svmax_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
 // CHECK-LABEL: @test_svmax_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmax.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmax.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmax_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmax.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmax.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svmax_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
@@ -1062,13 +1062,13 @@ svfloat16_t test_svmax_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
 // CHECK-LABEL: @test_svmax_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmax.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmax.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmax_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmax.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmax.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svmax_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
@@ -1079,13 +1079,13 @@ svfloat32_t test_svmax_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
 // CHECK-LABEL: @test_svmax_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmax.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmax.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmax_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmax.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmax.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svmax_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
@@ -1230,7 +1230,7 @@ svfloat64_t test_svmax_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmax.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmax.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmax_n_f16_xu10__SVBool_tu13__SVFloat16_tDh(
@@ -1238,7 +1238,7 @@ svfloat64_t test_svmax_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmax.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmax.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svmax_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
@@ -1251,7 +1251,7 @@ svfloat16_t test_svmax_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmax.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmax.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmax_n_f32_xu10__SVBool_tu13__SVFloat32_tf(
@@ -1259,7 +1259,7 @@ svfloat16_t test_svmax_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmax.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmax.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svmax_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
@@ -1272,7 +1272,7 @@ svfloat32_t test_svmax_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmax.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmax.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmax_n_f64_xu10__SVBool_tu13__SVFloat64_td(
@@ -1280,7 +1280,7 @@ svfloat32_t test_svmax_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmax.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmax.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svmax_n_f64_x(svbool_t pg, svfloat64_t op1, float64_t op2)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_maxnm.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_maxnm.c
index 4f9e81831d9e3..1bd68b08da176 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_maxnm.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_maxnm.c
@@ -125,13 +125,13 @@ svfloat64_t test_svmaxnm_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
 // CHECK-LABEL: @test_svmaxnm_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmaxnm.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmaxnm.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmaxnm_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmaxnm.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmaxnm.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svmaxnm_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
@@ -142,13 +142,13 @@ svfloat16_t test_svmaxnm_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
 // CHECK-LABEL: @test_svmaxnm_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmaxnm.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmaxnm.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmaxnm_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmaxnm.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmaxnm.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svmaxnm_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
@@ -159,13 +159,13 @@ svfloat32_t test_svmaxnm_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
 // CHECK-LABEL: @test_svmaxnm_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmaxnm.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmaxnm.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmaxnm_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmaxnm.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmaxnm.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svmaxnm_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
@@ -310,7 +310,7 @@ svfloat64_t test_svmaxnm_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmaxnm.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmaxnm.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z20test_svmaxnm_n_f16_xu10__SVBool_tu13__SVFloat16_tDh(
@@ -318,7 +318,7 @@ svfloat64_t test_svmaxnm_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmaxnm.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmaxnm.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svmaxnm_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
@@ -331,7 +331,7 @@ svfloat16_t test_svmaxnm_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmaxnm.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmaxnm.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z20test_svmaxnm_n_f32_xu10__SVBool_tu13__SVFloat32_tf(
@@ -339,7 +339,7 @@ svfloat16_t test_svmaxnm_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmaxnm.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmaxnm.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svmaxnm_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
@@ -352,7 +352,7 @@ svfloat32_t test_svmaxnm_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmaxnm.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmaxnm.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z20test_svmaxnm_n_f64_xu10__SVBool_tu13__SVFloat64_td(
@@ -360,7 +360,7 @@ svfloat32_t test_svmaxnm_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmaxnm.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmaxnm.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svmaxnm_n_f64_x(svbool_t pg, svfloat64_t op1, float64_t op2)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_min.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_min.c
index 9765037b2a57b..b2db3b284ec5e 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_min.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_min.c
@@ -1045,13 +1045,13 @@ svfloat64_t test_svmin_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
 // CHECK-LABEL: @test_svmin_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmin.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmin.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmin_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmin.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmin.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svmin_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
@@ -1062,13 +1062,13 @@ svfloat16_t test_svmin_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
 // CHECK-LABEL: @test_svmin_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmin.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmin.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmin_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmin.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmin.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svmin_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
@@ -1079,13 +1079,13 @@ svfloat32_t test_svmin_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
 // CHECK-LABEL: @test_svmin_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmin.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmin.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmin_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmin.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmin.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svmin_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
@@ -1230,7 +1230,7 @@ svfloat64_t test_svmin_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmin.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmin.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmin_n_f16_xu10__SVBool_tu13__SVFloat16_tDh(
@@ -1238,7 +1238,7 @@ svfloat64_t test_svmin_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmin.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmin.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svmin_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
@@ -1251,7 +1251,7 @@ svfloat16_t test_svmin_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmin.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmin.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmin_n_f32_xu10__SVBool_tu13__SVFloat32_tf(
@@ -1259,7 +1259,7 @@ svfloat16_t test_svmin_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmin.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmin.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svmin_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
@@ -1272,7 +1272,7 @@ svfloat32_t test_svmin_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmin.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmin.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmin_n_f64_xu10__SVBool_tu13__SVFloat64_td(
@@ -1280,7 +1280,7 @@ svfloat32_t test_svmin_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmin.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmin.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svmin_n_f64_x(svbool_t pg, svfloat64_t op1, float64_t op2)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_minnm.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_minnm.c
index 1e343b139653f..44717484859a2 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_minnm.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_minnm.c
@@ -125,13 +125,13 @@ svfloat64_t test_svminnm_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
 // CHECK-LABEL: @test_svminnm_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fminnm.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fminnm.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svminnm_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fminnm.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fminnm.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svminnm_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
@@ -142,13 +142,13 @@ svfloat16_t test_svminnm_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
 // CHECK-LABEL: @test_svminnm_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fminnm.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fminnm.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svminnm_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fminnm.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fminnm.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svminnm_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
@@ -159,13 +159,13 @@ svfloat32_t test_svminnm_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
 // CHECK-LABEL: @test_svminnm_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fminnm.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fminnm.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svminnm_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fminnm.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fminnm.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svminnm_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
@@ -310,7 +310,7 @@ svfloat64_t test_svminnm_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fminnm.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fminnm.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z20test_svminnm_n_f16_xu10__SVBool_tu13__SVFloat16_tDh(
@@ -318,7 +318,7 @@ svfloat64_t test_svminnm_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fminnm.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fminnm.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svminnm_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
@@ -331,7 +331,7 @@ svfloat16_t test_svminnm_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fminnm.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fminnm.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z20test_svminnm_n_f32_xu10__SVBool_tu13__SVFloat32_tf(
@@ -339,7 +339,7 @@ svfloat16_t test_svminnm_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fminnm.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fminnm.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svminnm_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
@@ -352,7 +352,7 @@ svfloat32_t test_svminnm_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fminnm.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fminnm.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z20test_svminnm_n_f64_xu10__SVBool_tu13__SVFloat64_td(
@@ -360,7 +360,7 @@ svfloat32_t test_svminnm_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fminnm.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fminnm.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svminnm_n_f64_x(svbool_t pg, svfloat64_t op1, float64_t op2)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mla.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mla.c
index 358c06ad591fd..c09a16372b08f 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mla.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mla.c
@@ -1045,13 +1045,13 @@ svfloat64_t test_svmla_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, svfl
 // CHECK-LABEL: @test_svmla_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmla.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmla.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmla_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmla.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmla.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svmla_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, svfloat16_t op3)
@@ -1062,13 +1062,13 @@ svfloat16_t test_svmla_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, svfl
 // CHECK-LABEL: @test_svmla_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmla.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmla.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmla_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmla.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmla.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svmla_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, svfloat32_t op3)
@@ -1079,13 +1079,13 @@ svfloat32_t test_svmla_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, svfl
 // CHECK-LABEL: @test_svmla_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmla.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmla.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmla_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmla.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmla.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svmla_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2, svfloat64_t op3)
@@ -1230,7 +1230,7 @@ svfloat64_t test_svmla_n_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, fl
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmla.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmla.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmla_n_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_tDh(
@@ -1238,7 +1238,7 @@ svfloat64_t test_svmla_n_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, fl
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmla.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmla.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svmla_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, float16_t op3)
@@ -1251,7 +1251,7 @@ svfloat16_t test_svmla_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, fl
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmla.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmla.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmla_n_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_tf(
@@ -1259,7 +1259,7 @@ svfloat16_t test_svmla_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, fl
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmla.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmla.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svmla_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, float32_t op3)
@@ -1272,7 +1272,7 @@ svfloat32_t test_svmla_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, fl
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmla.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmla.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmla_n_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_td(
@@ -1280,7 +1280,7 @@ svfloat32_t test_svmla_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, fl
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmla.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmla.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svmla_n_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2, float64_t op3)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mls.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mls.c
index 70cd3b8ccb38e..a2ec89a4894ea 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mls.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mls.c
@@ -1045,13 +1045,13 @@ svfloat64_t test_svmls_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, svfl
 // CHECK-LABEL: @test_svmls_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmls.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmls.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmls_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmls.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmls.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svmls_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, svfloat16_t op3)
@@ -1062,13 +1062,13 @@ svfloat16_t test_svmls_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, svfl
 // CHECK-LABEL: @test_svmls_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmls.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmls.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmls_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmls.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmls.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svmls_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, svfloat32_t op3)
@@ -1079,13 +1079,13 @@ svfloat32_t test_svmls_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, svfl
 // CHECK-LABEL: @test_svmls_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmls.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmls.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmls_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmls.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmls.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svmls_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2, svfloat64_t op3)
@@ -1230,7 +1230,7 @@ svfloat64_t test_svmls_n_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, fl
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmls.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmls.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmls_n_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_tDh(
@@ -1238,7 +1238,7 @@ svfloat64_t test_svmls_n_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, fl
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmls.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmls.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svmls_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, float16_t op3)
@@ -1251,7 +1251,7 @@ svfloat16_t test_svmls_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, fl
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmls.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmls.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmls_n_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_tf(
@@ -1259,7 +1259,7 @@ svfloat16_t test_svmls_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, fl
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmls.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmls.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svmls_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, float32_t op3)
@@ -1272,7 +1272,7 @@ svfloat32_t test_svmls_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, fl
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmls.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmls.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmls_n_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_td(
@@ -1280,7 +1280,7 @@ svfloat32_t test_svmls_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, fl
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmls.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmls.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svmls_n_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2, float64_t op3)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_msb.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_msb.c
index 29f6f0c6e90ba..c415a18606ea3 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_msb.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_msb.c
@@ -1045,13 +1045,13 @@ svfloat64_t test_svmsb_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, svfl
 // CHECK-LABEL: @test_svmsb_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmsb.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmls.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP3:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmsb_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmsb.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmls.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP3:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svmsb_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, svfloat16_t op3)
@@ -1062,13 +1062,13 @@ svfloat16_t test_svmsb_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, svfl
 // CHECK-LABEL: @test_svmsb_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmsb.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmls.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP3:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmsb_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmsb.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmls.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP3:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svmsb_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, svfloat32_t op3)
@@ -1079,13 +1079,13 @@ svfloat32_t test_svmsb_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, svfl
 // CHECK-LABEL: @test_svmsb_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmsb.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmls.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP3:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmsb_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmsb.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmls.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP3:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svmsb_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2, svfloat64_t op3)
@@ -1230,7 +1230,7 @@ svfloat64_t test_svmsb_n_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, fl
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmsb.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmls.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[DOTSPLAT]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmsb_n_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_tDh(
@@ -1238,7 +1238,7 @@ svfloat64_t test_svmsb_n_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, fl
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmsb.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmls.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[DOTSPLAT]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svmsb_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, float16_t op3)
@@ -1251,7 +1251,7 @@ svfloat16_t test_svmsb_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, fl
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmsb.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmls.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[DOTSPLAT]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmsb_n_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_tf(
@@ -1259,7 +1259,7 @@ svfloat16_t test_svmsb_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, fl
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmsb.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmls.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[DOTSPLAT]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svmsb_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, float32_t op3)
@@ -1272,7 +1272,7 @@ svfloat32_t test_svmsb_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, fl
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmsb.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmls.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[DOTSPLAT]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmsb_n_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_td(
@@ -1280,7 +1280,7 @@ svfloat32_t test_svmsb_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, fl
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmsb.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmls.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[DOTSPLAT]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svmsb_n_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2, float64_t op3)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mul.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mul.c
index c9382d4df6438..d95a3b7fb1a19 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mul.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mul.c
@@ -1045,13 +1045,13 @@ svfloat64_t test_svmul_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
 // CHECK-LABEL: @test_svmul_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmul.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmul.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmul_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmul.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmul.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svmul_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
@@ -1062,13 +1062,13 @@ svfloat16_t test_svmul_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
 // CHECK-LABEL: @test_svmul_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmul.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmul.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmul_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmul.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmul.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svmul_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
@@ -1079,13 +1079,13 @@ svfloat32_t test_svmul_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
 // CHECK-LABEL: @test_svmul_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmul.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmul.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svmul_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmul.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmul.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svmul_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
@@ -1230,7 +1230,7 @@ svfloat64_t test_svmul_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmul.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmul.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmul_n_f16_xu10__SVBool_tu13__SVFloat16_tDh(
@@ -1238,7 +1238,7 @@ svfloat64_t test_svmul_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmul.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmul.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svmul_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
@@ -1251,7 +1251,7 @@ svfloat16_t test_svmul_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmul.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmul.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmul_n_f32_xu10__SVBool_tu13__SVFloat32_tf(
@@ -1259,7 +1259,7 @@ svfloat16_t test_svmul_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmul.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmul.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svmul_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
@@ -1272,7 +1272,7 @@ svfloat32_t test_svmul_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmul.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmul.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svmul_n_f64_xu10__SVBool_tu13__SVFloat64_td(
@@ -1280,7 +1280,7 @@ svfloat32_t test_svmul_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmul.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmul.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svmul_n_f64_x(svbool_t pg, svfloat64_t op1, float64_t op2)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mulx.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mulx.c
index 1ce0aa80afe4b..fdee3b0b3ad3b 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mulx.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_mulx.c
@@ -125,13 +125,13 @@ svfloat64_t test_svmulx_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
 // CHECK-LABEL: @test_svmulx_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmulx.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmulx.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svmulx_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmulx.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmulx.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svmulx_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
@@ -142,13 +142,13 @@ svfloat16_t test_svmulx_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
 // CHECK-LABEL: @test_svmulx_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmulx.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmulx.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svmulx_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmulx.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmulx.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svmulx_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
@@ -159,13 +159,13 @@ svfloat32_t test_svmulx_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
 // CHECK-LABEL: @test_svmulx_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmulx.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmulx.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svmulx_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmulx.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmulx.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svmulx_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
@@ -310,7 +310,7 @@ svfloat64_t test_svmulx_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmulx.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmulx.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svmulx_n_f16_xu10__SVBool_tu13__SVFloat16_tDh(
@@ -318,7 +318,7 @@ svfloat64_t test_svmulx_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmulx.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fmulx.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svmulx_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
@@ -331,7 +331,7 @@ svfloat16_t test_svmulx_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmulx.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmulx.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svmulx_n_f32_xu10__SVBool_tu13__SVFloat32_tf(
@@ -339,7 +339,7 @@ svfloat16_t test_svmulx_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmulx.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fmulx.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svmulx_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
@@ -352,7 +352,7 @@ svfloat32_t test_svmulx_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmulx.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmulx.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svmulx_n_f64_xu10__SVBool_tu13__SVFloat64_td(
@@ -360,7 +360,7 @@ svfloat32_t test_svmulx_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmulx.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fmulx.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svmulx_n_f64_x(svbool_t pg, svfloat64_t op1, float64_t op2)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_nmad.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_nmad.c
index f8a48392387fb..e0a18e1967494 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_nmad.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_nmad.c
@@ -125,13 +125,13 @@ svfloat64_t test_svnmad_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, svf
 // CHECK-LABEL: @test_svnmad_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmad.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmla.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP3:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svnmad_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmad.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmla.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP3:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svnmad_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, svfloat16_t op3)
@@ -142,13 +142,13 @@ svfloat16_t test_svnmad_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, svf
 // CHECK-LABEL: @test_svnmad_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmad.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmla.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP3:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svnmad_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmad.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmla.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP3:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svnmad_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, svfloat32_t op3)
@@ -159,13 +159,13 @@ svfloat32_t test_svnmad_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, svf
 // CHECK-LABEL: @test_svnmad_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmad.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmla.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP3:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svnmad_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmad.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmla.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP3:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svnmad_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2, svfloat64_t op3)
@@ -310,7 +310,7 @@ svfloat64_t test_svnmad_n_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, f
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmad.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmla.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[DOTSPLAT]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svnmad_n_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_tDh(
@@ -318,7 +318,7 @@ svfloat64_t test_svnmad_n_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, f
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmad.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmla.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[DOTSPLAT]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svnmad_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, float16_t op3)
@@ -331,7 +331,7 @@ svfloat16_t test_svnmad_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, f
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmad.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmla.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[DOTSPLAT]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svnmad_n_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_tf(
@@ -339,7 +339,7 @@ svfloat16_t test_svnmad_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, f
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmad.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmla.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[DOTSPLAT]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svnmad_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, float32_t op3)
@@ -352,7 +352,7 @@ svfloat32_t test_svnmad_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, f
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmad.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmla.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[DOTSPLAT]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svnmad_n_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_td(
@@ -360,7 +360,7 @@ svfloat32_t test_svnmad_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, f
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmad.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmla.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[DOTSPLAT]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svnmad_n_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2, float64_t op3)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_nmla.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_nmla.c
index 83754625313e5..c9577a9aa7577 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_nmla.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_nmla.c
@@ -125,13 +125,13 @@ svfloat64_t test_svnmla_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, svf
 // CHECK-LABEL: @test_svnmla_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmla.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmla.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svnmla_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmla.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmla.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svnmla_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, svfloat16_t op3)
@@ -142,13 +142,13 @@ svfloat16_t test_svnmla_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, svf
 // CHECK-LABEL: @test_svnmla_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmla.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmla.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svnmla_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmla.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmla.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svnmla_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, svfloat32_t op3)
@@ -159,13 +159,13 @@ svfloat32_t test_svnmla_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, svf
 // CHECK-LABEL: @test_svnmla_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmla.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmla.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svnmla_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmla.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmla.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svnmla_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2, svfloat64_t op3)
@@ -310,7 +310,7 @@ svfloat64_t test_svnmla_n_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, f
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmla.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmla.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svnmla_n_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_tDh(
@@ -318,7 +318,7 @@ svfloat64_t test_svnmla_n_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, f
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmla.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmla.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svnmla_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, float16_t op3)
@@ -331,7 +331,7 @@ svfloat16_t test_svnmla_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, f
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmla.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmla.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svnmla_n_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_tf(
@@ -339,7 +339,7 @@ svfloat16_t test_svnmla_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, f
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmla.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmla.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svnmla_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, float32_t op3)
@@ -352,7 +352,7 @@ svfloat32_t test_svnmla_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, f
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmla.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmla.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svnmla_n_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_td(
@@ -360,7 +360,7 @@ svfloat32_t test_svnmla_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, f
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmla.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmla.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svnmla_n_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2, float64_t op3)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_nmls.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_nmls.c
index 6e8adadfa4ff2..dd8281e1672a7 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_nmls.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_nmls.c
@@ -125,13 +125,13 @@ svfloat64_t test_svnmls_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, svf
 // CHECK-LABEL: @test_svnmls_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmls.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmls.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svnmls_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmls.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmls.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svnmls_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, svfloat16_t op3)
@@ -142,13 +142,13 @@ svfloat16_t test_svnmls_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, svf
 // CHECK-LABEL: @test_svnmls_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmls.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmls.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svnmls_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmls.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmls.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svnmls_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, svfloat32_t op3)
@@ -159,13 +159,13 @@ svfloat32_t test_svnmls_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, svf
 // CHECK-LABEL: @test_svnmls_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmls.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmls.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svnmls_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmls.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmls.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svnmls_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2, svfloat64_t op3)
@@ -310,7 +310,7 @@ svfloat64_t test_svnmls_n_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, f
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmls.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmls.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svnmls_n_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_tDh(
@@ -318,7 +318,7 @@ svfloat64_t test_svnmls_n_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, f
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmls.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmls.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svnmls_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, float16_t op3)
@@ -331,7 +331,7 @@ svfloat16_t test_svnmls_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, f
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmls.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmls.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svnmls_n_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_tf(
@@ -339,7 +339,7 @@ svfloat16_t test_svnmls_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, f
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmls.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmls.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svnmls_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, float32_t op3)
@@ -352,7 +352,7 @@ svfloat32_t test_svnmls_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, f
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmls.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmls.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svnmls_n_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_td(
@@ -360,7 +360,7 @@ svfloat32_t test_svnmls_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, f
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmls.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmls.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svnmls_n_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2, float64_t op3)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_nmsb.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_nmsb.c
index fecb7881940c4..7fa52d52d2b65 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_nmsb.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_nmsb.c
@@ -125,13 +125,13 @@ svfloat64_t test_svnmsb_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, svf
 // CHECK-LABEL: @test_svnmsb_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmsb.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmls.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP3:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svnmsb_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmsb.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmls.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP3:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svnmsb_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, svfloat16_t op3)
@@ -142,13 +142,13 @@ svfloat16_t test_svnmsb_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, svf
 // CHECK-LABEL: @test_svnmsb_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmsb.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmls.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP3:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svnmsb_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmsb.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmls.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP3:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svnmsb_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, svfloat32_t op3)
@@ -159,13 +159,13 @@ svfloat32_t test_svnmsb_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, svf
 // CHECK-LABEL: @test_svnmsb_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmsb.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmls.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP3:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svnmsb_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmsb.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP3:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmls.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP3:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svnmsb_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2, svfloat64_t op3)
@@ -310,7 +310,7 @@ svfloat64_t test_svnmsb_n_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, f
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmsb.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmls.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[DOTSPLAT]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svnmsb_n_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_tDh(
@@ -318,7 +318,7 @@ svfloat64_t test_svnmsb_n_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2, f
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmsb.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fnmls.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[DOTSPLAT]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svnmsb_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, float16_t op3)
@@ -331,7 +331,7 @@ svfloat16_t test_svnmsb_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, f
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmsb.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmls.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[DOTSPLAT]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svnmsb_n_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_tf(
@@ -339,7 +339,7 @@ svfloat16_t test_svnmsb_n_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2, f
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmsb.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fnmls.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[DOTSPLAT]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svnmsb_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, float32_t op3)
@@ -352,7 +352,7 @@ svfloat32_t test_svnmsb_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, f
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP3:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmsb.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmls.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[DOTSPLAT]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svnmsb_n_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_td(
@@ -360,7 +360,7 @@ svfloat32_t test_svnmsb_n_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2, f
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP3:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmsb.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fnmls.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[DOTSPLAT]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svnmsb_n_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2, float64_t op3)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_sub.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_sub.c
index 4ea6e8735cdcb..ec189029caecb 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_sub.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_sub.c
@@ -1045,13 +1045,13 @@ svfloat64_t test_svsub_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
 // CHECK-LABEL: @test_svsub_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fsub.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fsub.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svsub_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fsub.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fsub.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svsub_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
@@ -1062,13 +1062,13 @@ svfloat16_t test_svsub_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
 // CHECK-LABEL: @test_svsub_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fsub.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fsub.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svsub_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fsub.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fsub.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svsub_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
@@ -1079,13 +1079,13 @@ svfloat32_t test_svsub_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
 // CHECK-LABEL: @test_svsub_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fsub.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fsub.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z16test_svsub_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fsub.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fsub.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svsub_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
@@ -1230,7 +1230,7 @@ svfloat64_t test_svsub_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fsub.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fsub.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svsub_n_f16_xu10__SVBool_tu13__SVFloat16_tDh(
@@ -1238,7 +1238,7 @@ svfloat64_t test_svsub_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fsub.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fsub.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svsub_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
@@ -1251,7 +1251,7 @@ svfloat16_t test_svsub_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fsub.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fsub.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svsub_n_f32_xu10__SVBool_tu13__SVFloat32_tf(
@@ -1259,7 +1259,7 @@ svfloat16_t test_svsub_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fsub.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fsub.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svsub_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
@@ -1272,7 +1272,7 @@ svfloat32_t test_svsub_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fsub.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fsub.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z18test_svsub_n_f64_xu10__SVBool_tu13__SVFloat64_td(
@@ -1280,7 +1280,7 @@ svfloat32_t test_svsub_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fsub.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fsub.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svsub_n_f64_x(svbool_t pg, svfloat64_t op1, float64_t op2)
diff --git a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_subr.c b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_subr.c
index a91291511c3d9..e8bffc5f97e5a 100644
--- a/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_subr.c
+++ b/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_subr.c
@@ -1045,13 +1045,13 @@ svfloat64_t test_svsubr_f64_m(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
 // CHECK-LABEL: @test_svsubr_f16_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fsubr.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fsub.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svsubr_f16_xu10__SVBool_tu13__SVFloat16_tu13__SVFloat16_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fsubr.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fsub.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP2:%.*]], <vscale x 8 x half> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svsubr_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
@@ -1062,13 +1062,13 @@ svfloat16_t test_svsubr_f16_x(svbool_t pg, svfloat16_t op1, svfloat16_t op2)
 // CHECK-LABEL: @test_svsubr_f32_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fsubr.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fsub.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svsubr_f32_xu10__SVBool_tu13__SVFloat32_tu13__SVFloat32_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fsubr.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fsub.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP2:%.*]], <vscale x 4 x float> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svsubr_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
@@ -1079,13 +1079,13 @@ svfloat32_t test_svsubr_f32_x(svbool_t pg, svfloat32_t op1, svfloat32_t op2)
 // CHECK-LABEL: @test_svsubr_f64_x(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fsubr.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fsub.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z17test_svsubr_f64_xu10__SVBool_tu13__SVFloat64_tu13__SVFloat64_t(
 // CPP-CHECK-NEXT:  entry:
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fsubr.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[OP2:%.*]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fsub.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP2:%.*]], <vscale x 2 x double> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svsubr_f64_x(svbool_t pg, svfloat64_t op1, svfloat64_t op2)
@@ -1230,7 +1230,7 @@ svfloat64_t test_svsubr_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fsubr.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fsub.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[DOTSPLAT]], <vscale x 8 x half> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svsubr_n_f16_xu10__SVBool_tu13__SVFloat16_tDh(
@@ -1238,7 +1238,7 @@ svfloat64_t test_svsubr_n_f64_m(svbool_t pg, svfloat64_t op1, float64_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 8 x half> poison, half [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 8 x half> [[DOTSPLATINSERT]], <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fsubr.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[OP1:%.*]], <vscale x 8 x half> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x half> @llvm.aarch64.sve.fsub.u.nxv8f16(<vscale x 8 x i1> [[TMP0]], <vscale x 8 x half> [[DOTSPLAT]], <vscale x 8 x half> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 8 x half> [[TMP1]]
 //
 svfloat16_t test_svsubr_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
@@ -1251,7 +1251,7 @@ svfloat16_t test_svsubr_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fsubr.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fsub.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[DOTSPLAT]], <vscale x 4 x float> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svsubr_n_f32_xu10__SVBool_tu13__SVFloat32_tf(
@@ -1259,7 +1259,7 @@ svfloat16_t test_svsubr_n_f16_x(svbool_t pg, svfloat16_t op1, float16_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x float> poison, float [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x float> [[DOTSPLATINSERT]], <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fsubr.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[OP1:%.*]], <vscale x 4 x float> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 4 x float> @llvm.aarch64.sve.fsub.u.nxv4f32(<vscale x 4 x i1> [[TMP0]], <vscale x 4 x float> [[DOTSPLAT]], <vscale x 4 x float> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
 //
 svfloat32_t test_svsubr_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
@@ -1272,7 +1272,7 @@ svfloat32_t test_svsubr_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fsubr.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fsub.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[DOTSPLAT]], <vscale x 2 x double> [[OP1:%.*]])
 // CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 // CPP-CHECK-LABEL: @_Z19test_svsubr_n_f64_xu10__SVBool_tu13__SVFloat64_td(
@@ -1280,7 +1280,7 @@ svfloat32_t test_svsubr_n_f32_x(svbool_t pg, svfloat32_t op1, float32_t op2)
 // CPP-CHECK-NEXT:    [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
 // CPP-CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x double> poison, double [[OP2:%.*]], i64 0
 // CPP-CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x double> [[DOTSPLATINSERT]], <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
-// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fsubr.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[OP1:%.*]], <vscale x 2 x double> [[DOTSPLAT]])
+// CPP-CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 2 x double> @llvm.aarch64.sve.fsub.u.nxv2f64(<vscale x 2 x i1> [[TMP0]], <vscale x 2 x double> [[DOTSPLAT]], <vscale x 2 x double> [[OP1:%.*]])
 // CPP-CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
 //
 svfloat64_t test_svsubr_n_f64_x(svbool_t pg, svfloat64_t op1, float64_t op2)
diff --git a/clang/test/CodeGen/alias.cpp b/clang/test/CodeGen/alias.cpp
new file mode 100644
index 0000000000000..17c1e1ae32f03
--- /dev/null
+++ b/clang/test/CodeGen/alias.cpp
@@ -0,0 +1,27 @@
+// RUN: %clang_cc1 -triple x86_64-linux -verify -emit-llvm-only %s
+// RUN: not %clang_cc1 -triple x86_64-linux -emit-llvm-only -fdiagnostics-parseable-fixits %s 2>&1 | FileCheck %s
+
+void *f1_ifunc(void) { return nullptr; }
+void f1(void) __attribute__((alias("f1_ifunc")));
+// expected-error@-1 {{alias must point to a defined variable or function}}
+// expected-note@-2 {{must refer to its mangled name}}
+// expected-note@-3 {{function by that name is mangled as}}
+// CHECK: fix-it:"{{.*}}":{[[@LINE-4]]:30-[[@LINE-4]]:47}:"alias(\"_Z8f1_ifuncv\")"
+
+void *f6_resolver_resolver(void) { return 0; }
+void *f6_resolver(void) __attribute__((alias("f6_resolver_resolver")));
+// expected-error@-1 {{alias must point to a defined variable or function}}
+// expected-note@-2 {{must refer to its mangled name}}
+// expected-note@-3 {{function by that name is mangled as}}
+// CHECK: fix-it:"{{.*}}":{[[@LINE-4]]:40-[[@LINE-4]]:69}:"alias(\"_Z20f6_resolver_resolverv\")"
+void f6(void) __attribute__((alias("f6_resolver")));
+// expected-error@-1 {{alias must point to a defined variable or function}}
+// expected-note@-2 {{must refer to its mangled name}}
+// expected-note@-3 {{function by that name is mangled as}}
+// CHECK: fix-it:"{{.*}}":{[[@LINE-4]]:30-[[@LINE-4]]:50}:"alias(\"_Z11f6_resolverv\")"
+
+__attribute__((unused, alias("resolver"), deprecated("hahahaha, isn't C great?")))
+void func();
+// expected-error@-2 {{alias must point to a defined variable or function}}
+// expected-note@-3 {{must refer to its mangled name}}
+
diff --git a/clang/test/CodeGen/arm-abi-vector.c b/clang/test/CodeGen/arm-abi-vector.c
index 43620f8550033..7f0cc4bcb0cd8 100644
--- a/clang/test/CodeGen/arm-abi-vector.c
+++ b/clang/test/CodeGen/arm-abi-vector.c
@@ -16,8 +16,7 @@ typedef __attribute__(( ext_vector_type(5) ))  short __short5;
 double varargs_vec_2i(int fixed, ...) {
 // CHECK: varargs_vec_2i
 // CHECK: [[VAR:%.*]] = alloca <2 x i32>, align 8
-// CHECK: [[ALIGN:%.*]] = and i32 {{%.*}}, -8
-// CHECK: [[AP_ALIGN:%.*]] = inttoptr i32 [[ALIGN]] to ptr
+// CHECK: [[AP_ALIGN:%.*]] = call ptr @llvm.ptrmask.p0.i32(ptr {{%.*}}, i32 -8)
 // CHECK: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_ALIGN]], i32 8
 // CHECK: [[VEC:%.*]] = load <2 x i32>, ptr [[AP_ALIGN]], align 8
 // CHECK: store <2 x i32> [[VEC]], ptr [[VAR]], align 8
@@ -29,8 +28,7 @@ double varargs_vec_2i(int fixed, ...) {
 // APCS-GNU: store <2 x i32> [[VEC]], ptr [[VAR]], align 8
 // ANDROID: varargs_vec_2i
 // ANDROID: [[VAR:%.*]] = alloca <2 x i32>, align 8
-// ANDROID: [[ALIGN:%.*]] = and i32 {{%.*}}, -8
-// ANDROID: [[AP_ALIGN:%.*]] = inttoptr i32 [[ALIGN]] to ptr
+// ANDROID: [[AP_ALIGN:%.*]] = call ptr @llvm.ptrmask.p0.i32(ptr {{%.*}}, i32 -8)
 // ANDROID: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_ALIGN]], i32 8
 // ANDROID: [[VEC:%.*]] = load <2 x i32>, ptr [[AP_ALIGN]], align 8
 // ANDROID: store <2 x i32> [[VEC]], ptr [[VAR]], align 8
@@ -85,8 +83,7 @@ double test_3c(__char3 *in) {
 double varargs_vec_5c(int fixed, ...) {
 // CHECK: varargs_vec_5c
 // CHECK: [[VAR:%.*]] = alloca <5 x i8>, align 8
-// CHECK: [[ALIGN:%.*]] = and i32 {{%.*}}, -8
-// CHECK: [[AP_ALIGN:%.*]] = inttoptr i32 [[ALIGN]] to ptr
+// CHECK: [[AP_ALIGN:%.*]] = call ptr @llvm.ptrmask.p0.i32(ptr {{%.*}}, i32 -8)
 // CHECK: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_ALIGN]], i32 8
 // CHECK: [[VEC:%.*]] = load <5 x i8>, ptr [[AP_ALIGN]], align 8
 // CHECK: store <5 x i8> [[VEC]], ptr [[VAR]], align 8
@@ -98,8 +95,7 @@ double varargs_vec_5c(int fixed, ...) {
 // APCS-GNU: store <5 x i8> [[VEC]], ptr [[VAR]], align 8
 // ANDROID: varargs_vec_5c
 // ANDROID: [[VAR:%.*]] = alloca <5 x i8>, align 8
-// ANDROID: [[ALIGN:%.*]] = and i32 {{%.*}}, -8
-// ANDROID: [[AP_ALIGN:%.*]] = inttoptr i32 [[ALIGN]] to ptr
+// ANDROID: [[AP_ALIGN:%.*]] = call ptr @llvm.ptrmask.p0.i32(ptr {{%.*}}, i32 -8)
 // ANDROID: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_ALIGN]], i32 8
 // ANDROID: [[VEC:%.*]] = load <5 x i8>, ptr [[AP_ALIGN]], align 8
 // ANDROID: store <5 x i8> [[VEC]], ptr [[VAR]], align 8
@@ -125,8 +121,7 @@ double test_5c(__char5 *in) {
 double varargs_vec_9c(int fixed, ...) {
 // CHECK: varargs_vec_9c
 // CHECK: [[VAR:%.*]] = alloca <9 x i8>, align 16
-// CHECK: [[ALIGN:%.*]] = and i32 {{%.*}}, -8
-// CHECK: [[AP_ALIGN:%.*]] = inttoptr i32 [[ALIGN]] to ptr
+// CHECK: [[AP_ALIGN:%.*]] = call ptr @llvm.ptrmask.p0.i32(ptr {{%.*}}, i32 -8)
 // CHECK: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_ALIGN]], i32 16
 // CHECK: [[T0:%.*]] = load <9 x i8>, ptr [[AP_ALIGN]], align 8
 // CHECK: store <9 x i8> [[T0]], ptr [[VAR]], align 16
@@ -138,8 +133,7 @@ double varargs_vec_9c(int fixed, ...) {
 // APCS-GNU: store <9 x i8> [[VEC]], ptr [[VAR]], align 16
 // ANDROID: varargs_vec_9c
 // ANDROID: [[VAR:%.*]] = alloca <9 x i8>, align 16
-// ANDROID: [[ALIGN:%.*]] = and i32 {{%.*}}, -8
-// ANDROID: [[AP_ALIGN:%.*]] = inttoptr i32 [[ALIGN]] to ptr
+// ANDROID: [[AP_ALIGN:%.*]] = call ptr @llvm.ptrmask.p0.i32(ptr {{%.*}}, i32 -8)
 // ANDROID: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_ALIGN]], i32 16
 // ANDROID: [[T0:%.*]] = load <9 x i8>, ptr [[AP_ALIGN]], align 8
 // ANDROID: store <9 x i8> [[T0]], ptr [[VAR]], align 16
@@ -194,8 +188,7 @@ double test_19c(__char19 *in) {
 double varargs_vec_3s(int fixed, ...) {
 // CHECK: varargs_vec_3s
 // CHECK: alloca <3 x i16>, align 8
-// CHECK: [[ALIGN:%.*]] = and i32 {{%.*}}, -8
-// CHECK: [[AP_ALIGN:%.*]] = inttoptr i32 [[ALIGN]] to ptr
+// CHECK: [[AP_ALIGN:%.*]] = call ptr @llvm.ptrmask.p0.i32(ptr {{%.*}}, i32 -8)
 // CHECK: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_ALIGN]], i32 8
 // APCS-GNU: varargs_vec_3s
 // APCS-GNU: [[VAR:%.*]] = alloca <3 x i16>, align 8
@@ -204,8 +197,7 @@ double varargs_vec_3s(int fixed, ...) {
 // APCS-GNU: [[VEC:%.*]] = load <3 x i16>, ptr [[AP]], align 4
 // ANDROID: varargs_vec_3s
 // ANDROID: alloca <3 x i16>, align 8
-// ANDROID: [[ALIGN:%.*]] = and i32 {{%.*}}, -8
-// ANDROID: [[AP_ALIGN:%.*]] = inttoptr i32 [[ALIGN]] to ptr
+// ANDROID: [[AP_ALIGN:%.*]] = call ptr @llvm.ptrmask.p0.i32(ptr {{%.*}}, i32 -8)
 // ANDROID: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_ALIGN]], i32 8
   va_list ap;
   double sum = fixed;
@@ -229,8 +221,7 @@ double test_3s(__short3 *in) {
 double varargs_vec_5s(int fixed, ...) {
 // CHECK: varargs_vec_5s
 // CHECK: [[VAR_ALIGN:%.*]] = alloca <5 x i16>, align 16
-// CHECK: [[ALIGN:%.*]] = and i32 {{%.*}}, -8
-// CHECK: [[AP_ALIGN:%.*]] = inttoptr i32 [[ALIGN]] to ptr
+// CHECK: [[AP_ALIGN:%.*]] = call ptr @llvm.ptrmask.p0.i32(ptr {{%.*}}, i32 -8)
 // CHECK: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_ALIGN]], i32 16
 // CHECK: [[VEC:%.*]] = load <5 x i16>, ptr [[AP_ALIGN]], align 8
 // CHECK: store <5 x i16> [[VEC]], ptr [[VAR_ALIGN]], align 16
@@ -241,8 +232,7 @@ double varargs_vec_5s(int fixed, ...) {
 // APCS-GNU: [[VEC:%.*]] = load <5 x i16>, ptr [[AP]], align 4
 // ANDROID: varargs_vec_5s
 // ANDROID: [[VAR_ALIGN:%.*]] = alloca <5 x i16>, align 16
-// ANDROID: [[ALIGN:%.*]] = and i32 {{%.*}}, -8
-// ANDROID: [[AP_ALIGN:%.*]] = inttoptr i32 [[ALIGN]] to ptr
+// ANDROID: [[AP_ALIGN:%.*]] = call ptr @llvm.ptrmask.p0.i32(ptr {{%.*}}, i32 -8)
 // ANDROID: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_ALIGN]], i32 16
 // ANDROID: [[VEC:%.*]] = load <5 x i16>, ptr [[AP_ALIGN]], align 8
 // ANDROID: store <5 x i16> [[VEC]], ptr [[VAR_ALIGN]], align 16
@@ -274,16 +264,14 @@ typedef struct
 
 double varargs_struct(int fixed, ...) {
 // CHECK: varargs_struct
-// CHECK: [[ALIGN:%.*]] = and i32 {{%.*}}, -8
-// CHECK: [[AP_ALIGN:%.*]] = inttoptr i32 [[ALIGN]] to ptr
+// CHECK: [[AP_ALIGN:%.*]] = call ptr @llvm.ptrmask.p0.i32(ptr {{%.*}}, i32 -8)
 // CHECK: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_ALIGN]], i32 16
 // APCS-GNU: varargs_struct
 // APCS-GNU: [[VAR_ALIGN:%.*]] = alloca %struct.StructWithVec
 // APCS-GNU: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr {{%.*}}, i32 16
 // APCS-GNU: call void @llvm.memcpy
 // ANDROID: varargs_struct
-// ANDROID: [[ALIGN:%.*]] = and i32 {{%.*}}, -8
-// ANDROID: [[AP_ALIGN:%.*]] = inttoptr i32 [[ALIGN]] to ptr
+// ANDROID: [[AP_ALIGN:%.*]] = call ptr @llvm.ptrmask.p0.i32(ptr {{%.*}}, i32 -8)
 // ANDROID: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_ALIGN]], i32 16
   va_list ap;
   double sum = fixed;
diff --git a/clang/test/CodeGen/arm-vaarg-align.c b/clang/test/CodeGen/arm-vaarg-align.c
index 2270c8b4f971e..410c1ebd5d146 100644
--- a/clang/test/CodeGen/arm-vaarg-align.c
+++ b/clang/test/CodeGen/arm-vaarg-align.c
@@ -1,7 +1,7 @@
 // REQUIRES: arm-registered-target
 // RUN: %clang_cc1 -triple arm -target-abi aapcs %s -emit-llvm -o - | FileCheck -check-prefix=AAPCS %s
 // RUN: %clang_cc1 -triple arm -target-abi apcs-gnu %s -emit-llvm -o - | FileCheck -check-prefix=APCS-GNU %s
-/* 
+/*
  * Check that va_arg accesses stack according to ABI alignment
  * long long and double require 8-byte alignment under AAPCS
  * however, they only require 4-byte alignment under APCS
@@ -11,10 +11,11 @@ long long t1(int i, ...) {
     // APCS-GNU: t1
     __builtin_va_list ap;
     __builtin_va_start(ap, i);
-    // AAPCS: add i32 %{{.*}} 7
-    // AAPCS: and i32 %{{.*}} -8
-    // APCS-GNU-NOT: add i32 %{{.*}} 7
-    // APCS-GNU-NOT: and i32 %{{.*}} -8
+    // AAPCS: [[ADD:%.*]] = getelementptr inbounds i8, ptr %argp.cur, i32 7
+    // AAPCS = call ptr @llvm.ptrmask.p0.i32(ptr [[ADD]], i32 -8)
+
+    // APCS-GNU-NOT: getelementptr {{.*}}, i32 7
+    // APCS-GNU-NOT: llvm.ptrmask.p0
     long long ll = __builtin_va_arg(ap, long long);
     __builtin_va_end(ap);
     return ll;
@@ -24,10 +25,11 @@ double t2(int i, ...) {
     // APCS-GNU: t2
     __builtin_va_list ap;
     __builtin_va_start(ap, i);
-    // AAPCS: add i32 %{{.*}} 7
-    // AAPCS: and i32 %{{.*}} -8
-    // APCS-GNU-NOT: add i32 %{{.*}} 7
-    // APCS-GNU-NOT: and i32 %{{.*}} -8
+    // AAPCS: [[ADD:%.*]] = getelementptr inbounds i8, ptr %argp.cur, i32 7
+    // AAPCS = call ptr @llvm.ptrmask.p0.i32(ptr [[ADD]], i32 -8)
+
+    // APCS-GNU-NOT: getelementptr {{.*}}, i32 7
+    // APCS-GNU-NOT: llvm.ptrmask.p0
     double ll = __builtin_va_arg(ap, double);
     __builtin_va_end(ap);
     return ll;
diff --git a/clang/test/CodeGen/arm-varargs.c b/clang/test/CodeGen/arm-varargs.c
index c8a8899a967de..17330262e6ad6 100644
--- a/clang/test/CodeGen/arm-varargs.c
+++ b/clang/test/CodeGen/arm-varargs.c
@@ -41,10 +41,8 @@ struct aligned_bigstruct simple_aligned_struct(void) {
 // CHECK-LABEL: define{{.*}} void @simple_aligned_struct(ptr noalias sret(%struct.aligned_bigstruct) align 8 %agg.result)
   return va_arg(the_list, struct aligned_bigstruct);
 // CHECK: [[CUR:%[a-z0-9._]+]] = load ptr, ptr @the_list, align 4
-// CHECK: [[CUR_INT:%[a-z0-9._]+]] = ptrtoint ptr [[CUR]] to i32
-// CHECK: [[CUR_INT_ADD:%[a-z0-9._]+]] = add i32 [[CUR_INT]], 7
-// CHECK: [[CUR_INT_ALIGNED:%[a-z0-9._]+]] = and i32 [[CUR_INT_ADD]], -8
-// CHECK: [[CUR_ALIGNED:%[a-z0-9._]+]] = inttoptr i32 [[CUR_INT_ALIGNED]] to ptr
+// CHECK: [[CUR_INT_ADD:%[a-z0-9._]+]] = getelementptr inbounds i8, ptr [[CUR]], i32 7
+// CHECK: [[CUR_ALIGNED:%[a-z0-9._]+]] = call ptr @llvm.ptrmask.p0.i32(ptr [[CUR_INT_ADD]], i32 -8)
 // CHECK: [[NEXT:%[a-z0-9._]+]] = getelementptr inbounds i8, ptr [[CUR_ALIGNED]], i32 16
 // CHECK: store ptr [[NEXT]], ptr @the_list, align 4
 // CHECK: call void @llvm.memcpy.p0.p0.i32(ptr align 8 %agg.result, ptr align 8 [[CUR_ALIGNED]], i32 16, i1 false)
@@ -55,10 +53,8 @@ double simple_double(void) {
 // CHECK-LABEL: define{{.*}} double @simple_double
   return va_arg(the_list, double);
 // CHECK: [[CUR:%[a-z0-9._]+]] = load ptr, ptr @the_list, align 4
-// CHECK: [[CUR_INT:%[a-z0-9._]+]] = ptrtoint ptr [[CUR]] to i32
-// CHECK: [[CUR_INT_ADD:%[a-z0-9._]+]] = add i32 [[CUR_INT]], 7
-// CHECK: [[CUR_INT_ALIGNED:%[a-z0-9._]+]] = and i32 [[CUR_INT_ADD]], -8
-// CHECK: [[CUR_ALIGNED:%[a-z0-9._]+]] = inttoptr i32 [[CUR_INT_ALIGNED]] to ptr
+// CHECK: [[CUR_INT_ADD:%[a-z0-9._]+]] = getelementptr inbounds i8, ptr [[CUR]], i32 7
+// CHECK: [[CUR_ALIGNED:%[a-z0-9._]+]] = call ptr @llvm.ptrmask.p0.i32(ptr [[CUR_INT_ADD]], i32 -8)
 // CHECK: [[NEXT:%[a-z0-9._]+]] = getelementptr inbounds i8, ptr [[CUR_ALIGNED]], i32 8
 // CHECK: store ptr [[NEXT]], ptr @the_list, align 4
 // CHECK: [[RESULT:%[a-z0-9._]+]] = load double, ptr [[CUR_ALIGNED]]
@@ -110,10 +106,8 @@ underaligned_long_long underaligned_long_long_test(void) {
 // CHECK-LABEL: define{{.*}} i64 @underaligned_long_long_test()
   return va_arg(the_list, underaligned_long_long);
 // CHECK: [[CUR:%[a-z0-9._]+]] = load ptr, ptr @the_list, align 4
-// CHECK: [[CUR_INT:%[a-z0-9._]+]] = ptrtoint ptr [[CUR]] to i32
-// CHECK: [[CUR_INT_ADD:%[a-z0-9._]+]] = add i32 [[CUR_INT]], 7
-// CHECK: [[CUR_INT_ALIGNED:%[a-z0-9._]+]] = and i32 [[CUR_INT_ADD]], -8
-// CHECK: [[CUR_ALIGNED:%[a-z0-9._]+]] = inttoptr i32 [[CUR_INT_ALIGNED]] to ptr
+// CHECK: [[CUR_INT_ADD:%[a-z0-9._]+]] = getelementptr inbounds i8, ptr [[CUR]], i32 7
+// CHECK: [[CUR_ALIGNED:%[a-z0-9._]+]] = call ptr @llvm.ptrmask.p0.i32(ptr [[CUR_INT_ADD]], i32 -8)
 // CHECK: [[NEXT:%[a-z0-9._]+]] = getelementptr inbounds i8, ptr [[CUR_ALIGNED]], i32 8
 // CHECK: store ptr [[NEXT]], ptr @the_list, align 4
 // CHECK: [[RESULT:%[a-z0-9._]+]] = load i64, ptr [[CUR_ALIGNED]]
@@ -125,10 +119,8 @@ overaligned_long_long overaligned_long_long_test(void) {
 // CHECK-LABEL: define{{.*}} i64 @overaligned_long_long_test()
   return va_arg(the_list, overaligned_long_long);
 // CHECK: [[CUR:%[a-z0-9._]+]] = load ptr, ptr @the_list, align 4
-// CHECK: [[CUR_INT:%[a-z0-9._]+]] = ptrtoint ptr [[CUR]] to i32
-// CHECK: [[CUR_INT_ADD:%[a-z0-9._]+]] = add i32 [[CUR_INT]], 7
-// CHECK: [[CUR_INT_ALIGNED:%[a-z0-9._]+]] = and i32 [[CUR_INT_ADD]], -8
-// CHECK: [[CUR_ALIGNED:%[a-z0-9._]+]] = inttoptr i32 [[CUR_INT_ALIGNED]] to ptr
+// CHECK: [[CUR_INT_ADD:%[a-z0-9._]+]] = getelementptr inbounds i8, ptr [[CUR]], i32 7
+// CHECK: [[CUR_ALIGNED:%[a-z0-9._]+]] = call ptr @llvm.ptrmask.p0.i32(ptr [[CUR_INT_ADD]], i32 -8)
 // CHECK: [[NEXT:%[a-z0-9._]+]] = getelementptr inbounds i8, ptr [[CUR_ALIGNED]], i32 8
 // CHECK: store ptr [[NEXT]], ptr @the_list, align 4
 // CHECK: [[RESULT:%[a-z0-9._]+]] = load i64, ptr [[CUR_ALIGNED]]
@@ -196,10 +188,8 @@ overaligned_long_long_struct overaligned_long_long_struct_test(void) {
 // CHECK-LABEL: define{{.*}} void @overaligned_long_long_struct_test(ptr noalias sret(%struct.overaligned_long_long_struct) align 16 %agg.result)
   return va_arg(the_list, overaligned_long_long_struct);
 // CHECK: [[CUR:%[a-z0-9._]+]] = load ptr, ptr @the_list, align 4
-// CHECK: [[CUR_INT:%[a-z0-9._]+]] = ptrtoint ptr [[CUR]] to i32
-// CHECK: [[CUR_INT_ADD:%[a-z0-9._]+]] = add i32 [[CUR_INT]], 7
-// CHECK: [[CUR_INT_ALIGNED:%[a-z0-9._]+]] = and i32 [[CUR_INT_ADD]], -8
-// CHECK: [[CUR_ALIGNED:%[a-z0-9._]+]] = inttoptr i32 [[CUR_INT_ALIGNED]] to ptr
+// CHECK: [[CUR_INT_ADD:%[a-z0-9._]+]] = getelementptr inbounds i8, ptr [[CUR]], i32 7
+// CHECK: [[CUR_ALIGNED:%[a-z0-9._]+]] = call ptr @llvm.ptrmask.p0.i32(ptr [[CUR_INT_ADD]], i32 -8)
 // CHECK: [[NEXT:%[a-z0-9._]+]] = getelementptr inbounds i8, ptr [[CUR_ALIGNED]], i32 16
 // CHECK: store ptr [[NEXT]], ptr @the_list, align 4
 // CHECK: call void @llvm.memcpy.p0.p0.i32(ptr align 16 %agg.result, ptr align 8 [[CUR_ALIGNED]], i32 16, i1 false)
@@ -232,10 +222,8 @@ overaligned_int_struct_member overaligned_int_struct_member_test(void) {
 // CHECK-LABEL: define{{.*}} void @overaligned_int_struct_member_test(ptr noalias sret(%struct.overaligned_int_struct_member) align 16 %agg.result)
   return va_arg(the_list, overaligned_int_struct_member);
 // CHECK: [[CUR:%[a-z0-9._]+]] = load ptr, ptr @the_list, align 4
-// CHECK: [[CUR_INT:%[a-z0-9._]+]] = ptrtoint ptr [[CUR]] to i32
-// CHECK: [[CUR_INT_ADD:%[a-z0-9._]+]] = add i32 [[CUR_INT]], 7
-// CHECK: [[CUR_INT_ALIGNED:%[a-z0-9._]+]] = and i32 [[CUR_INT_ADD]], -8
-// CHECK: [[CUR_ALIGNED:%[a-z0-9._]+]] = inttoptr i32 [[CUR_INT_ALIGNED]] to ptr
+// CHECK: [[CUR_INT_ADD:%[a-z0-9._]+]] = getelementptr inbounds i8, ptr [[CUR]], i32 7
+// CHECK: [[CUR_ALIGNED:%[a-z0-9._]+]] = call ptr @llvm.ptrmask.p0.i32(ptr [[CUR_INT_ADD]], i32 -8)
 // CHECK: [[NEXT:%[a-z0-9._]+]] = getelementptr inbounds i8, ptr [[CUR_ALIGNED]], i32 16
 // CHECK: store ptr [[NEXT]], ptr @the_list, align 4
 // CHECK: call void @llvm.memcpy.p0.p0.i32(ptr align 16 %agg.result, ptr align 8 [[CUR_ALIGNED]], i32 16, i1 false)
@@ -262,10 +250,8 @@ overaligned_long_long_struct_member overaligned_long_long_struct_member_test(voi
 // CHECK-LABEL: define{{.*}} void @overaligned_long_long_struct_member_test(ptr noalias sret(%struct.overaligned_long_long_struct_member) align 16 %agg.result)
   return va_arg(the_list, overaligned_long_long_struct_member);
 // CHECK: [[CUR:%[a-z0-9._]+]] = load ptr, ptr @the_list, align 4
-// CHECK: [[CUR_INT:%[a-z0-9._]+]] = ptrtoint ptr [[CUR]] to i32
-// CHECK: [[CUR_INT_ADD:%[a-z0-9._]+]] = add i32 [[CUR_INT]], 7
-// CHECK: [[CUR_INT_ALIGNED:%[a-z0-9._]+]] = and i32 [[CUR_INT_ADD]], -8
-// CHECK: [[CUR_ALIGNED:%[a-z0-9._]+]] = inttoptr i32 [[CUR_INT_ALIGNED]] to ptr
+// CHECK: [[CUR_INT_ADD:%[a-z0-9._]+]] = getelementptr inbounds i8, ptr [[CUR]], i32 7
+// CHECK: [[CUR_ALIGNED:%[a-z0-9._]+]] = call ptr @llvm.ptrmask.p0.i32(ptr [[CUR_INT_ADD]], i32 -8)
 // CHECK: [[NEXT:%[a-z0-9._]+]] = getelementptr inbounds i8, ptr [[CUR_ALIGNED]], i32 16
 // CHECK: store ptr [[NEXT]], ptr @the_list, align 4
 // CHECK: call void @llvm.memcpy.p0.p0.i32(ptr align 16 %agg.result, ptr align 8 [[CUR_ALIGNED]], i32 16, i1 false)
diff --git a/clang/test/CodeGen/arm64-abi-vector.c b/clang/test/CodeGen/arm64-abi-vector.c
index 2063e46c414a8..81e42315c883b 100644
--- a/clang/test/CodeGen/arm64-abi-vector.c
+++ b/clang/test/CodeGen/arm64-abi-vector.c
@@ -94,8 +94,9 @@ double test_5c(__char5 *in) {
 double varargs_vec_9c(int fixed, ...) {
 // CHECK: varargs_vec_9c
 // CHECK: alloca <9 x i8>, align 16
-// CHECK: [[ALIGN:%.*]] = and i64 {{%.*}}, -16
-// CHECK: [[AP_ALIGN:%.*]] = inttoptr i64 [[ALIGN]] to ptr
+// CHECK: [[AP:%.*]] = load ptr, ptr %ap, align 8
+// CHECK: [[AP_ADD:%.*]] = getelementptr inbounds i8, ptr [[AP]], i32 15
+// CHECK: [[AP_ALIGN:%.*]] = call ptr @llvm.ptrmask.p0.i64(ptr [[AP_ADD]], i64 -16)
 // CHECK: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_ALIGN]], i64 16
   va_list ap;
   double sum = fixed;
@@ -153,8 +154,9 @@ double test_3s(__short3 *in) {
 double varargs_vec_5s(int fixed, ...) {
 // CHECK: varargs_vec_5s
 // CHECK: alloca <5 x i16>, align 16
-// CHECK: [[ALIGN:%.*]] = and i64 {{%.*}}, -16
-// CHECK: [[AP_ALIGN:%.*]] = inttoptr i64 [[ALIGN]] to ptr
+// CHECK: [[AP:%.*]] = load ptr, ptr %ap, align 8
+// CHECK: [[AP_ADD:%.*]] = getelementptr inbounds i8, ptr [[AP]], i32 15
+// CHECK: [[AP_ALIGN:%.*]] = call ptr @llvm.ptrmask.p0.i64(ptr [[AP_ADD]], i64 -16)
 // CHECK: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_ALIGN]], i64 16
   va_list ap;
   double sum = fixed;
@@ -174,8 +176,9 @@ double test_5s(__short5 *in) {
 double varargs_vec_3i(int fixed, ...) {
 // CHECK: varargs_vec_3i
 // CHECK: alloca <3 x i32>, align 16
-// CHECK: [[ALIGN:%.*]] = and i64 {{%.*}}, -16
-// CHECK: [[AP_ALIGN:%.*]] = inttoptr i64 [[ALIGN]] to ptr
+// CHECK: [[AP:%.*]] = load ptr, ptr %ap, align 8
+// CHECK: [[AP_ADD:%.*]] = getelementptr inbounds i8, ptr [[AP]], i32 15
+// CHECK: [[AP_ALIGN:%.*]] = call ptr @llvm.ptrmask.p0.i64(ptr [[AP_ADD]], i64 -16)
 // CHECK: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_ALIGN]], i64 16
   va_list ap;
   double sum = fixed;
@@ -244,8 +247,11 @@ double varargs_vec(int fixed, ...) {
 // CHECK: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_CUR:%.*]], i64 8
   sum = sum + c5.x + c5.y;
   __char9 c9 = va_arg(ap, __char9);
-// CHECK: [[ALIGN:%.*]] = and i64 {{%.*}}, -16
-// CHECK: [[AP_ALIGN:%.*]] = inttoptr i64 [[ALIGN]] to ptr
+
+
+// CHECK: [[AP:%.*]] = load ptr, ptr %ap, align 8
+// CHECK: [[AP_ADD:%.*]] = getelementptr inbounds i8, ptr [[AP]], i32 15
+// CHECK: [[AP_ALIGN:%.*]] = call ptr @llvm.ptrmask.p0.i64(ptr [[AP_ADD]], i64 -16)
 // CHECK: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_ALIGN]], i64 16
   sum = sum + c9.x + c9.y;
   __char19 c19 = va_arg(ap, __char19);
@@ -256,13 +262,17 @@ double varargs_vec(int fixed, ...) {
 // CHECK: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_CUR:%.*]], i64 8
   sum = sum + s3.x + s3.y;
   __short5 s5 = va_arg(ap, __short5);
-// CHECK: [[ALIGN:%.*]] = and i64 {{%.*}}, -16
-// CHECK: [[AP_ALIGN:%.*]] = inttoptr i64 [[ALIGN]] to ptr
+
+// CHECK: [[AP:%.*]] = load ptr, ptr %ap, align 8
+// CHECK: [[AP_ADD:%.*]] = getelementptr inbounds i8, ptr [[AP]], i32 15
+// CHECK: [[AP_ALIGN:%.*]] = call ptr @llvm.ptrmask.p0.i64(ptr [[AP_ADD]], i64 -16)
 // CHECK: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_ALIGN]], i64 16
   sum = sum + s5.x + s5.y;
   __int3 i3 = va_arg(ap, __int3);
-// CHECK: [[ALIGN:%.*]] = and i64 {{%.*}}, -16
-// CHECK: [[AP_ALIGN:%.*]] = inttoptr i64 [[ALIGN]] to ptr
+
+// CHECK: [[AP:%.*]] = load ptr, ptr %ap, align 8
+// CHECK: [[AP_ADD:%.*]] = getelementptr inbounds i8, ptr [[AP]], i32 15
+// CHECK: [[AP_ALIGN:%.*]] = call ptr @llvm.ptrmask.p0.i64(ptr [[AP_ADD]], i64 -16)
 // CHECK: [[AP_NEXT:%.*]] = getelementptr inbounds i8, ptr [[AP_ALIGN]], i64 16
   sum = sum + i3.x + i3.y;
   __int5 i5 = va_arg(ap, __int5);
diff --git a/clang/test/CodeGen/arm64-arguments.c b/clang/test/CodeGen/arm64-arguments.c
index 13728e43dff69..8bc48795c690c 100644
--- a/clang/test/CodeGen/arm64-arguments.c
+++ b/clang/test/CodeGen/arm64-arguments.c
@@ -707,10 +707,8 @@ int32x4_t test_hva(int n, ...) {
 
   // HVA is not indirect, so occupies its full 16 bytes on the stack. but it
   // must be properly aligned.
-// CHECK-LE: [[ALIGN0:%.*]] = ptrtoint ptr [[CURLIST]] to i64
-// CHECK-LE: [[ALIGN1:%.*]] = add i64 [[ALIGN0]], 15
-// CHECK-LE: [[ALIGN2:%.*]] = and i64 [[ALIGN1]], -16
-// CHECK-LE: [[ALIGNED_LIST:%.*]] = inttoptr i64 [[ALIGN2]] to ptr
+// CHECK-LE: [[GEP:%.*]] = getelementptr inbounds i8, ptr [[CURLIST]], i32 15
+// CHECK-LE: [[ALIGNED_LIST:%.*]] = call ptr @llvm.ptrmask.p0.i64(ptr [[GEP]], i64 -16)
 
 // CHECK-LE: [[NEXTLIST:%.*]] = getelementptr inbounds i8, ptr [[ALIGNED_LIST]], i64 32
 // CHECK-LE: store ptr [[NEXTLIST]], ptr [[THELIST]]
@@ -752,11 +750,9 @@ float32x3_t test_hva_v3(int n, ...) {
 
   // HVA is not indirect, so occupies its full 16 bytes on the stack. but it
   // must be properly aligned.
-// CHECK-LE: [[ALIGN0:%.*]] = ptrtoint ptr [[CURLIST]] to i64
-// CHECK-LE: [[ALIGN1:%.*]] = add i64 [[ALIGN0]], 15
-// CHECK-LE: [[ALIGN2:%.*]] = and i64 [[ALIGN1]], -16
-// CHECK-LE: [[ALIGNED_LIST:%.*]] = inttoptr i64 [[ALIGN2]] to ptr
 
+// CHECK-LE: [[GEP:%.*]] = getelementptr inbounds i8, ptr [[CURLIST]], i32 15
+// CHECK-LE: [[ALIGNED_LIST:%.*]] = call ptr @llvm.ptrmask.p0.i64(ptr [[GEP]], i64 -16)
 // CHECK-LE: [[NEXTLIST:%.*]] = getelementptr inbounds i8, ptr [[ALIGNED_LIST]], i64 64
 // CHECK-LE: store ptr [[NEXTLIST]], ptr [[THELIST]]
 
diff --git a/clang/test/CodeGen/arm64_32-vaarg.c b/clang/test/CodeGen/arm64_32-vaarg.c
index ec2fcb128f3a4..9fbcf88ecfdcc 100644
--- a/clang/test/CodeGen/arm64_32-vaarg.c
+++ b/clang/test/CodeGen/arm64_32-vaarg.c
@@ -28,11 +28,8 @@ typedef struct {
 long long test_longlong(OneLongLong input, va_list *mylist) {
   // CHECK-LABEL: define{{.*}} i64 @test_longlong(i64 %input
   // CHECK: [[STARTPTR:%.*]] = load ptr, ptr %mylist
-  // CHECK: [[START:%.*]] = ptrtoint ptr [[STARTPTR]] to i32
-
-  // CHECK: [[ALIGN_TMP:%.*]] = add i32 [[START]], 7
-  // CHECK: [[ALIGNED:%.*]] = and i32 [[ALIGN_TMP]], -8
-  // CHECK: [[ALIGNED_ADDR:%.*]] = inttoptr i32 [[ALIGNED]] to ptr
+  // CHECK: [[ALIGN_TMP:%.+]] = getelementptr inbounds i8, ptr [[STARTPTR]], i32 7
+  // CHECK: [[ALIGNED_ADDR:%.+]] = tail call ptr @llvm.ptrmask.p0.i32(ptr nonnull [[ALIGN_TMP]], i32 -8)
   // CHECK: [[NEXT:%.*]] = getelementptr inbounds i8, ptr [[ALIGNED_ADDR]], i32 8
   // CHECK: store ptr [[NEXT]], ptr %mylist
 
diff --git a/clang/test/CodeGen/armv7k-abi.c b/clang/test/CodeGen/armv7k-abi.c
index 696e9ab994621..e070d5a9c7041 100644
--- a/clang/test/CodeGen/armv7k-abi.c
+++ b/clang/test/CodeGen/armv7k-abi.c
@@ -85,10 +85,11 @@ typedef struct {
 OddlySizedStruct return_oddly_sized_struct() {}
 
 // CHECK: define{{.*}} <4 x float> @test_va_arg_vec(ptr noundef %l)
-// CHECK:   [[ALIGN_TMP:%.*]] = add i32 {{%.*}}, 15
-// CHECK:   [[ALIGNED:%.*]] = and i32 [[ALIGN_TMP]], -16
-// CHECK:   [[ALIGNED_I8:%.*]] = inttoptr i32 [[ALIGNED]] to ptr
-// CHECK:   load <4 x float>, ptr [[ALIGNED_I8]], align 16
+
+
+// CHECK: [[GEP_ALIGN:%.+]] = getelementptr inbounds i8, ptr {{%.*}}, i32 15
+// CHECK: [[ALIGNED:%.*]] = call ptr @llvm.ptrmask.p0.i32(ptr [[GEP_ALIGN]], i32 -16)
+// CHECK:   load <4 x float>, ptr [[ALIGNED]], align 16
 float32x4_t test_va_arg_vec(__builtin_va_list l) {
   return __builtin_va_arg(l, float32x4_t);
 }
diff --git a/clang/test/CodeGen/attr-ifunc.c b/clang/test/CodeGen/attr-ifunc.c
index a0659ae773dd1..4f8fe13530fdb 100644
--- a/clang/test/CodeGen/attr-ifunc.c
+++ b/clang/test/CodeGen/attr-ifunc.c
@@ -12,6 +12,7 @@ void bar(void) __attribute__((ifunc("foo")));
 void *f1_ifunc(void);
 void f1(void) __attribute__((ifunc("f1_ifunc")));
 // expected-error@-1 {{ifunc must point to a defined function}}
+// expected-note@-2 {{must refer to its mangled name}}
 
 void *f2_a(void) __attribute__((alias("f2_b")));
 void *f2_b(void) __attribute__((ifunc("f2_a")));
diff --git a/clang/test/CodeGen/attr-ifunc.cpp b/clang/test/CodeGen/attr-ifunc.cpp
new file mode 100644
index 0000000000000..5b5b2c14b4074
--- /dev/null
+++ b/clang/test/CodeGen/attr-ifunc.cpp
@@ -0,0 +1,27 @@
+// RUN: %clang_cc1 -triple x86_64-linux -fsyntax-only -verify -emit-llvm-only %s
+// RUN: not %clang_cc1 -triple x86_64-linux -emit-llvm-only -fdiagnostics-parseable-fixits %s 2>&1 | FileCheck %s
+
+void *f1_ifunc(void) { return nullptr; }
+void f1(void) __attribute__((ifunc("f1_ifunc")));
+// expected-error@-1 {{ifunc must point to a defined function}}
+// expected-note@-2 {{must refer to its mangled name}}
+// expected-note@-3 {{function by that name is mangled as}}
+// CHECK: fix-it:"{{.*}}":{[[@LINE-4]]:30-[[@LINE-4]]:47}:"ifunc(\"_Z8f1_ifuncv\")"
+
+void *f6_resolver_resolver(void) { return 0; }
+void *f6_resolver(void) __attribute__((ifunc("f6_resolver_resolver")));
+// expected-error@-1 {{ifunc must point to a defined function}}
+// expected-note@-2 {{must refer to its mangled name}}
+// expected-note@-3 {{function by that name is mangled as}}
+// CHECK: fix-it:"{{.*}}":{[[@LINE-4]]:40-[[@LINE-4]]:69}:"ifunc(\"_Z20f6_resolver_resolverv\")"
+void f6(void) __attribute__((ifunc("f6_resolver")));
+// expected-error@-1 {{ifunc must point to a defined function}}
+// expected-note@-2 {{must refer to its mangled name}}
+// expected-note@-3 {{function by that name is mangled as}}
+// CHECK: fix-it:"{{.*}}":{[[@LINE-4]]:30-[[@LINE-4]]:50}:"ifunc(\"_Z11f6_resolverv\")"
+
+__attribute__((unused, ifunc("resolver"), deprecated("hahahaha, isn't C great?")))
+void func();
+// expected-error@-2 {{ifunc must point to a defined function}}
+// expected-note@-3 {{must refer to its mangled name}}
+
diff --git a/clang/test/CodeGen/avr/avr-inline-asm-constraints.c b/clang/test/CodeGen/avr/avr-inline-asm-constraints.c
index c659953ca6247..96774861feb22 100644
--- a/clang/test/CodeGen/avr/avr-inline-asm-constraints.c
+++ b/clang/test/CodeGen/avr/avr-inline-asm-constraints.c
@@ -1,5 +1,10 @@
 // REQUIRES: avr-registered-target
-// RUN: %clang_cc1 -triple avr-unknown-unknown -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -x c -triple avr -target-cpu at90s8515 -emit-llvm -o - %s \
+// RUN:     | FileCheck --check-prefixes=CHECK,AVR25 %s
+// RUN: %clang_cc1 -x c -triple avr -target-cpu atmega328 -emit-llvm -o - %s \
+// RUN:     | FileCheck --check-prefixes=CHECK,AVR51 %s
+// RUN: %clang_cc1 -x c -triple avr -target-cpu atmega2560 -emit-llvm -o - %s \
+// RUN:     | FileCheck --check-prefixes=CHECK,AVR6 %s
 
 int data;
 
@@ -122,3 +127,23 @@ void ora() {
   // CHECK: call addrspace(0) i16 asm "subi r30, $0", "=ra"()
   asm("subi r30, %0" : "=ra"(data));
 }
+
+void escapeChar(void) {
+  asm("_foo:");
+  // AVR25: call addrspace(0) void asm sideeffect "rcall _foo"
+  // AVR51: call addrspace(0) void asm sideeffect "call _foo"
+  // AVR6:  call addrspace(0) void asm sideeffect "call _foo"
+  asm("%~call _foo" ::);
+  // AVR25: call addrspace(0) void asm sideeffect "rjmp _foo"
+  // AVR51: call addrspace(0) void asm sideeffect "jmp _foo"
+  // AVR6:  call addrspace(0) void asm sideeffect "jmp _foo"
+  asm("%~jmp _foo" ::);
+  // AVR25: call addrspace(0) void asm sideeffect "icall"
+  // AVR51: call addrspace(0) void asm sideeffect "icall"
+  // AVR6:  call addrspace(0) void asm sideeffect "eicall"
+  asm("%!icall" ::);
+  // AVR25: call addrspace(0) void asm sideeffect "ijmp"
+  // AVR51: call addrspace(0) void asm sideeffect "ijmp"
+  // AVR6:  call addrspace(0) void asm sideeffect "eijmp"
+  asm("%!ijmp" ::);
+}
diff --git a/clang/test/CodeGen/builtin_set_flt_rounds.c b/clang/test/CodeGen/builtin_set_flt_rounds.c
new file mode 100644
index 0000000000000..fc483fcc232aa
--- /dev/null
+++ b/clang/test/CodeGen/builtin_set_flt_rounds.c
@@ -0,0 +1,8 @@
+// RUN: %clang_cc1 -triple x86_64-gnu-linux %s -emit-llvm -o - | FileCheck %s
+// RUN: %clang_cc1 -triple x86_64-windows-msvc %s -emit-llvm -o - | FileCheck %s
+// RUN: %clang_cc1 -triple aarch64-gnu-linux %s -emit-llvm -o - | FileCheck %s
+// RUN: %clang_cc1 -triple aarch64-windows-msvc %s -emit-llvm -o - | FileCheck %s
+void test_builtin_set_flt_rounds() {
+  __builtin_set_flt_rounds(1);
+  // CHECK: call void @llvm.set.rounding(i32 1)
+}
diff --git a/clang/test/CodeGen/builtins-elementwise-math.c b/clang/test/CodeGen/builtins-elementwise-math.c
index 1b48a12b92056..deb518b1ba597 100644
--- a/clang/test/CodeGen/builtins-elementwise-math.c
+++ b/clang/test/CodeGen/builtins-elementwise-math.c
@@ -355,6 +355,39 @@ void test_builtin_elementwise_cos(float f1, float f2, double d1, double d2,
   vf2 = __builtin_elementwise_cos(vf1);
 }
 
+void test_builtin_elementwise_exp(float f1, float f2, double d1, double d2,
+                                  float4 vf1, float4 vf2) {
+  // CHECK-LABEL: define void @test_builtin_elementwise_exp(
+  // CHECK:      [[F1:%.+]] = load float, ptr %f1.addr, align 4
+  // CHECK-NEXT:  call float @llvm.exp.f32(float [[F1]])
+  f2 = __builtin_elementwise_exp(f1);
+
+  // CHECK:      [[D1:%.+]] = load double, ptr %d1.addr, align 8
+  // CHECK-NEXT: call double @llvm.exp.f64(double [[D1]])
+  d2 = __builtin_elementwise_exp(d1);
+
+  // CHECK:      [[VF1:%.+]] = load <4 x float>, ptr %vf1.addr, align 16
+  // CHECK-NEXT: call <4 x float> @llvm.exp.v4f32(<4 x float> [[VF1]])
+  vf2 = __builtin_elementwise_exp(vf1);
+}
+
+void test_builtin_elementwise_exp2(float f1, float f2, double d1, double d2,
+                                  float4 vf1, float4 vf2) {
+  // CHECK-LABEL: define void @test_builtin_elementwise_exp2(
+  // CHECK:      [[F1:%.+]] = load float, ptr %f1.addr, align 4
+  // CHECK-NEXT:  call float @llvm.exp2.f32(float [[F1]])
+  f2 = __builtin_elementwise_exp2(f1);
+
+  // CHECK:      [[D1:%.+]] = load double, ptr %d1.addr, align 8
+  // CHECK-NEXT: call double @llvm.exp2.f64(double [[D1]])
+  d2 = __builtin_elementwise_exp2(d1);
+
+  // CHECK:      [[VF1:%.+]] = load <4 x float>, ptr %vf1.addr, align 16
+  // CHECK-NEXT: call <4 x float> @llvm.exp2.v4f32(<4 x float> [[VF1]])
+  vf2 = __builtin_elementwise_exp2(vf1);
+}
+
+
 void test_builtin_elementwise_floor(float f1, float f2, double d1, double d2,
                                     float4 vf1, float4 vf2) {
   // CHECK-LABEL: define void @test_builtin_elementwise_floor(
diff --git a/clang/test/CodeGen/builtins-nvptx-native-half-type-err.c b/clang/test/CodeGen/builtins-nvptx-native-half-type-err.c
new file mode 100644
index 0000000000000..5c13c4e9b454c
--- /dev/null
+++ b/clang/test/CodeGen/builtins-nvptx-native-half-type-err.c
@@ -0,0 +1,21 @@
+// REQUIRES: nvptx-registered-target
+//
+// RUN: not %clang_cc1 -fsyntax-only -ffp-contract=off -triple nvptx-unknown-unknown -target-cpu \
+// RUN:   sm_75 -target-feature +ptx70 -fcuda-is-device -x cuda -emit-llvm -o - %s 2>&1 \
+// RUN:   | FileCheck -check-prefix=CHECK-ERROR %s
+
+#define __device__ __attribute__((device))
+typedef __fp16 __fp16v2 __attribute__((ext_vector_type(2)));
+
+__device__ void nvvm_ldg_ldu_native_half_types(const void *p) {
+  __nvvm_ldg_h((const __fp16 *)p);
+  __nvvm_ldg_h2((const __fp16v2 *)p);
+
+  __nvvm_ldu_h((const __fp16 *)p);
+  __nvvm_ldu_h2((const __fp16v2 *)p);
+}
+
+// CHECK-ERROR: error: __nvvm_ldg_h requires native half type support.
+// CHECK-ERROR: error: __nvvm_ldg_h2 requires native half type support.
+// CHECK-ERROR: error: __nvvm_ldu_h requires native half type support.
+// CHECK-ERROR: error: __nvvm_ldu_h2 requires native half type support.
diff --git a/clang/test/CodeGen/builtins-nvptx-native-half-type.c b/clang/test/CodeGen/builtins-nvptx-native-half-type.c
index 9dc61d6014210..670127f6eb61b 100644
--- a/clang/test/CodeGen/builtins-nvptx-native-half-type.c
+++ b/clang/test/CodeGen/builtins-nvptx-native-half-type.c
@@ -173,11 +173,20 @@ __device__ void nvvm_min_max_sm86() {
   // CHECK: ret void
 }
 
+typedef __fp16 __fp16v2 __attribute__((ext_vector_type(2)));
+
 // CHECK-LABEL: nvvm_ldg_native_half_types
 __device__ void nvvm_ldg_native_half_types(const void *p) {
   // CHECK: call half @llvm.nvvm.ldg.global.f.f16.p0
   __nvvm_ldg_h((const __fp16 *)p);
-  typedef __fp16 __fp16v2 __attribute__((ext_vector_type(2)));
   // CHECK: call <2 x half> @llvm.nvvm.ldg.global.f.v2f16.p0
   __nvvm_ldg_h2((const __fp16v2 *)p);
 }
+
+// CHECK-LABEL: nvvm_ldu_native_half_types
+__device__ void nvvm_ldu_native_half_types(const void *p) {
+  // CHECK: call half @llvm.nvvm.ldu.global.f.f16.p0
+  __nvvm_ldu_h((const __fp16 *)p);
+  // CHECK: call <2 x half> @llvm.nvvm.ldu.global.f.v2f16.p0
+  __nvvm_ldu_h2((const __fp16v2 *)p);
+}
diff --git a/clang/test/CodeGen/builtins-nvptx.c b/clang/test/CodeGen/builtins-nvptx.c
index 0851fc829a468..48a2d0241528b 100644
--- a/clang/test/CodeGen/builtins-nvptx.c
+++ b/clang/test/CodeGen/builtins-nvptx.c
@@ -652,6 +652,97 @@ __device__ void nvvm_ldg(const void *p) {
   __nvvm_ldg_d2((const double2 *)p);
 }
 
+// CHECK-LABEL: nvvm_ldu
+__device__ void nvvm_ldu(const void *p) {
+  // CHECK: call i8 @llvm.nvvm.ldu.global.i.i8.p0(ptr {{%[0-9]+}}, i32 1)
+  // CHECK: call i8 @llvm.nvvm.ldu.global.i.i8.p0(ptr {{%[0-9]+}}, i32 1)
+  __nvvm_ldu_c((const char *)p);
+  __nvvm_ldu_uc((const unsigned char *)p);
+
+  // CHECK: call i16 @llvm.nvvm.ldu.global.i.i16.p0(ptr {{%[0-9]+}}, i32 2)
+  // CHECK: call i16 @llvm.nvvm.ldu.global.i.i16.p0(ptr {{%[0-9]+}}, i32 2)
+  __nvvm_ldu_s((const short *)p);
+  __nvvm_ldu_us((const unsigned short *)p);
+
+  // CHECK: call i32 @llvm.nvvm.ldu.global.i.i32.p0(ptr {{%[0-9]+}}, i32 4)
+  // CHECK: call i32 @llvm.nvvm.ldu.global.i.i32.p0(ptr {{%[0-9]+}}, i32 4)
+  __nvvm_ldu_i((const int *)p);
+  __nvvm_ldu_ui((const unsigned int *)p);
+
+  // LP32: call i32 @llvm.nvvm.ldu.global.i.i32.p0(ptr {{%[0-9]+}}, i32 4)
+  // LP32: call i32 @llvm.nvvm.ldu.global.i.i32.p0(ptr {{%[0-9]+}}, i32 4)
+  // LP64: call i64 @llvm.nvvm.ldu.global.i.i64.p0(ptr {{%[0-9]+}}, i32 8)
+  // LP64: call i64 @llvm.nvvm.ldu.global.i.i64.p0(ptr {{%[0-9]+}}, i32 8)
+  __nvvm_ldu_l((const long *)p);
+  __nvvm_ldu_ul((const unsigned long *)p);
+
+  // CHECK: call float @llvm.nvvm.ldu.global.f.f32.p0(ptr {{%[0-9]+}}, i32 4)
+  __nvvm_ldu_f((const float *)p);
+  // CHECK: call double @llvm.nvvm.ldu.global.f.f64.p0(ptr {{%[0-9]+}}, i32 8)
+  __nvvm_ldu_d((const double *)p);
+
+  // CHECK: call <2 x i8> @llvm.nvvm.ldu.global.i.v2i8.p0(ptr {{%[0-9]+}}, i32 2)
+  // CHECK: call <2 x i8> @llvm.nvvm.ldu.global.i.v2i8.p0(ptr {{%[0-9]+}}, i32 2)
+  typedef char char2 __attribute__((ext_vector_type(2)));
+  typedef unsigned char uchar2 __attribute__((ext_vector_type(2)));
+  __nvvm_ldu_c2((const char2 *)p);
+  __nvvm_ldu_uc2((const uchar2 *)p);
+
+  // CHECK: call <4 x i8> @llvm.nvvm.ldu.global.i.v4i8.p0(ptr {{%[0-9]+}}, i32 4)
+  // CHECK: call <4 x i8> @llvm.nvvm.ldu.global.i.v4i8.p0(ptr {{%[0-9]+}}, i32 4)
+  typedef char char4 __attribute__((ext_vector_type(4)));
+  typedef unsigned char uchar4 __attribute__((ext_vector_type(4)));
+  __nvvm_ldu_c4((const char4 *)p);
+  __nvvm_ldu_uc4((const uchar4 *)p);
+
+  // CHECK: call <2 x i16> @llvm.nvvm.ldu.global.i.v2i16.p0(ptr {{%[0-9]+}}, i32 4)
+  // CHECK: call <2 x i16> @llvm.nvvm.ldu.global.i.v2i16.p0(ptr {{%[0-9]+}}, i32 4)
+  typedef short short2 __attribute__((ext_vector_type(2)));
+  typedef unsigned short ushort2 __attribute__((ext_vector_type(2)));
+  __nvvm_ldu_s2((const short2 *)p);
+  __nvvm_ldu_us2((const ushort2 *)p);
+
+  // CHECK: call <4 x i16> @llvm.nvvm.ldu.global.i.v4i16.p0(ptr {{%[0-9]+}}, i32 8)
+  // CHECK: call <4 x i16> @llvm.nvvm.ldu.global.i.v4i16.p0(ptr {{%[0-9]+}}, i32 8)
+  typedef short short4 __attribute__((ext_vector_type(4)));
+  typedef unsigned short ushort4 __attribute__((ext_vector_type(4)));
+  __nvvm_ldu_s4((const short4 *)p);
+  __nvvm_ldu_us4((const ushort4 *)p);
+
+  // CHECK: call <2 x i32> @llvm.nvvm.ldu.global.i.v2i32.p0(ptr {{%[0-9]+}}, i32 8)
+  // CHECK: call <2 x i32> @llvm.nvvm.ldu.global.i.v2i32.p0(ptr {{%[0-9]+}}, i32 8)
+  typedef int int2 __attribute__((ext_vector_type(2)));
+  typedef unsigned int uint2 __attribute__((ext_vector_type(2)));
+  __nvvm_ldu_i2((const int2 *)p);
+  __nvvm_ldu_ui2((const uint2 *)p);
+
+  // CHECK: call <4 x i32> @llvm.nvvm.ldu.global.i.v4i32.p0(ptr {{%[0-9]+}}, i32 16)
+  // CHECK: call <4 x i32> @llvm.nvvm.ldu.global.i.v4i32.p0(ptr {{%[0-9]+}}, i32 16)
+  typedef int int4 __attribute__((ext_vector_type(4)));
+  typedef unsigned int uint4 __attribute__((ext_vector_type(4)));
+  __nvvm_ldu_i4((const int4 *)p);
+  __nvvm_ldu_ui4((const uint4 *)p);
+
+  // CHECK: call <2 x i64> @llvm.nvvm.ldu.global.i.v2i64.p0(ptr {{%[0-9]+}}, i32 16)
+  // CHECK: call <2 x i64> @llvm.nvvm.ldu.global.i.v2i64.p0(ptr {{%[0-9]+}}, i32 16)
+  typedef long long longlong2 __attribute__((ext_vector_type(2)));
+  typedef unsigned long long ulonglong2 __attribute__((ext_vector_type(2)));
+  __nvvm_ldu_ll2((const longlong2 *)p);
+  __nvvm_ldu_ull2((const ulonglong2 *)p);
+
+  // CHECK: call <2 x float> @llvm.nvvm.ldu.global.f.v2f32.p0(ptr {{%[0-9]+}}, i32 8)
+  typedef float float2 __attribute__((ext_vector_type(2)));
+  __nvvm_ldu_f2((const float2 *)p);
+
+  // CHECK: call <4 x float> @llvm.nvvm.ldu.global.f.v4f32.p0(ptr {{%[0-9]+}}, i32 16)
+  typedef float float4 __attribute__((ext_vector_type(4)));
+  __nvvm_ldu_f4((const float4 *)p);
+
+  // CHECK: call <2 x double> @llvm.nvvm.ldu.global.f.v2f64.p0(ptr {{%[0-9]+}}, i32 16)
+  typedef double double2 __attribute__((ext_vector_type(2)));
+  __nvvm_ldu_d2((const double2 *)p);
+}
+
 // CHECK-LABEL: nvvm_shfl
 __device__ void nvvm_shfl(int i, float f, int a, int b) {
   // CHECK: call i32 @llvm.nvvm.shfl.down.i32(i32
diff --git a/clang/test/CodeGen/builtins.c b/clang/test/CodeGen/builtins.c
index 21e14cc9c08c0..1281c5e13ae3e 100644
--- a/clang/test/CodeGen/builtins.c
+++ b/clang/test/CodeGen/builtins.c
@@ -411,6 +411,15 @@ void test_float_builtin_ops(float F, double D, long double LD) {
   resld = __builtin_roundl(LD);
   // CHECK: call x86_fp80 @llvm.round.f80
 
+  resf = __builtin_roundevenf(F);
+  // CHECK: call float @llvm.roundeven.f32
+
+  resd = __builtin_roundeven(D);
+  // CHECK: call double @llvm.roundeven.f64
+
+  resld = __builtin_roundevenl(LD);
+  // CHECK: call x86_fp80 @llvm.roundeven.f80
+  
   resli = __builtin_lroundf (F);
   // CHECK: call i64 @llvm.lround.i64.f32
 
diff --git a/clang/test/CodeGen/catch-undef-behavior.c b/clang/test/CodeGen/catch-undef-behavior.c
index 9f511d54f458c..ca0df0f002f89 100644
--- a/clang/test/CodeGen/catch-undef-behavior.c
+++ b/clang/test/CodeGen/catch-undef-behavior.c
@@ -1,6 +1,6 @@
-// RUN: %clang_cc1 -no-opaque-pointers -fsanitize=alignment,null,object-size,shift-base,shift-exponent,return,signed-integer-overflow,vla-bound,float-cast-overflow,integer-divide-by-zero,bool,returns-nonnull-attribute,nonnull-attribute -fsanitize-recover=alignment,null,object-size,shift-base,shift-exponent,signed-integer-overflow,vla-bound,float-cast-overflow,integer-divide-by-zero,bool,returns-nonnull-attribute,nonnull-attribute -emit-llvm %s -o - -triple x86_64-linux-gnu | FileCheck %s --check-prefix=CHECK-COMMON --check-prefix=CHECK-UBSAN
-// RUN: %clang_cc1 -no-opaque-pointers -fsanitize-trap=alignment,null,object-size,shift-base,shift-exponent,return,signed-integer-overflow,vla-bound,float-cast-overflow,integer-divide-by-zero,bool,returns-nonnull-attribute,nonnull-attribute -fsanitize-recover=alignment,null,object-size,shift-base,shift-exponent,signed-integer-overflow,vla-bound,float-cast-overflow,integer-divide-by-zero,bool,returns-nonnull-attribute,nonnull-attribute -fsanitize=alignment,null,object-size,shift-base,shift-exponent,return,signed-integer-overflow,vla-bound,float-cast-overflow,integer-divide-by-zero,bool,returns-nonnull-attribute,nonnull-attribute -fsanitize-recover=alignment,null,object-size,shift-base,shift-exponent,signed-integer-overflow,vla-bound,float-cast-overflow,integer-divide-by-zero,bool,returns-nonnull-attribute,nonnull-attribute -emit-llvm %s -o - -triple x86_64-linux-gnu | FileCheck %s --check-prefix=CHECK-COMMON --check-prefix=CHECK-TRAP
-// RUN: %clang_cc1 -no-opaque-pointers -fsanitize=signed-integer-overflow -emit-llvm %s -o - -triple x86_64-linux-gnu | FileCheck %s --check-prefix=CHECK-OVERFLOW
+// RUN: %clang_cc1 -fsanitize=alignment,null,object-size,shift-base,shift-exponent,return,signed-integer-overflow,vla-bound,float-cast-overflow,integer-divide-by-zero,bool,returns-nonnull-attribute,nonnull-attribute -fsanitize-recover=alignment,null,object-size,shift-base,shift-exponent,signed-integer-overflow,vla-bound,float-cast-overflow,integer-divide-by-zero,bool,returns-nonnull-attribute,nonnull-attribute -emit-llvm %s -o - -triple x86_64-linux-gnu | FileCheck %s --check-prefix=CHECK-COMMON --check-prefix=CHECK-UBSAN
+// RUN: %clang_cc1 -fsanitize-trap=alignment,null,object-size,shift-base,shift-exponent,return,signed-integer-overflow,vla-bound,float-cast-overflow,integer-divide-by-zero,bool,returns-nonnull-attribute,nonnull-attribute -fsanitize-recover=alignment,null,object-size,shift-base,shift-exponent,signed-integer-overflow,vla-bound,float-cast-overflow,integer-divide-by-zero,bool,returns-nonnull-attribute,nonnull-attribute -fsanitize=alignment,null,object-size,shift-base,shift-exponent,return,signed-integer-overflow,vla-bound,float-cast-overflow,integer-divide-by-zero,bool,returns-nonnull-attribute,nonnull-attribute -fsanitize-recover=alignment,null,object-size,shift-base,shift-exponent,signed-integer-overflow,vla-bound,float-cast-overflow,integer-divide-by-zero,bool,returns-nonnull-attribute,nonnull-attribute -emit-llvm %s -o - -triple x86_64-linux-gnu | FileCheck %s --check-prefix=CHECK-COMMON --check-prefix=CHECK-TRAP
+// RUN: %clang_cc1 -fsanitize=signed-integer-overflow -emit-llvm %s -o - -triple x86_64-linux-gnu | FileCheck %s --check-prefix=CHECK-OVERFLOW
 
 // CHECK-UBSAN: @[[INT:.*]] = private unnamed_addr constant { i16, i16, [6 x i8] } { i16 0, i16 11, [6 x i8] c"'int'\00" }
 
@@ -32,15 +32,14 @@
 void foo(void) {
   union { int i; } u;
 
-  // CHECK-COMMON:      %[[I8PTR:.*]] = bitcast i32* %[[PTR:.*]] to i8*
-  // CHECK-COMMON-NEXT: %[[SIZE:.*]] = call i64 @llvm.objectsize.i64.p0i8(i8* %[[I8PTR]], i1 false, i1 false, i1 false)
+  // CHECK-COMMON: %[[SIZE:.*]] = call i64 @llvm.objectsize.i64.p0(ptr %[[PTR:.*]], i1 false, i1 false, i1 false)
   // CHECK-COMMON-NEXT: %[[OK:.*]] = icmp uge i64 %[[SIZE]], 4
 
   // CHECK-UBSAN: br i1 %[[OK]], {{.*}} !prof ![[WEIGHT_MD:.*]], !nosanitize
   // CHECK-TRAP:  br i1 %[[OK]], {{.*}}
 
   // CHECK-UBSAN:      %[[ARG:.*]] = ptrtoint {{.*}} %[[PTR]] to i64
-  // CHECK-UBSAN-NEXT: call void @__ubsan_handle_type_mismatch_v1(i8* bitcast ({{.*}} @[[LINE_100]] to i8*), i64 %[[ARG]])
+  // CHECK-UBSAN-NEXT: call void @__ubsan_handle_type_mismatch_v1(ptr @[[LINE_100]], i64 %[[ARG]])
 
   // CHECK-TRAP:      call void @llvm.ubsantrap(i8 22) [[NR_NUW:#[0-9]+]]
   // CHECK-TRAP-NEXT: unreachable
@@ -57,7 +56,7 @@ int bar(int *a) {
   // CHECK-COMMON-NEXT: %[[MISALIGN:.*]] = and i64 %[[PTRINT]], 3
   // CHECK-COMMON-NEXT: icmp eq i64 %[[MISALIGN]], 0
 
-  // CHECK-UBSAN: call void @__ubsan_handle_type_mismatch_v1(i8* bitcast ({{.*}} @[[LINE_200]] to i8*), i64 %[[PTRINT]])
+  // CHECK-UBSAN: call void @__ubsan_handle_type_mismatch_v1(ptr @[[LINE_200]], i64 %[[PTRINT]])
 
   // CHECK-TRAP:      call void @llvm.ubsantrap(i8 22) [[NR_NUW]]
   // CHECK-TRAP-NEXT: unreachable
@@ -92,7 +91,7 @@ int lsh_overflow(int a, int b) {
 
   // CHECK-UBSAN:      %[[ARG1:.*]] = zext
   // CHECK-UBSAN-NEXT: %[[ARG2:.*]] = zext
-  // CHECK-UBSAN-NEXT: call void @__ubsan_handle_shift_out_of_bounds(i8* bitcast ({{.*}} @[[LINE_300]] to i8*), i64 %[[ARG1]], i64 %[[ARG2]])
+  // CHECK-UBSAN-NEXT: call void @__ubsan_handle_shift_out_of_bounds(ptr @[[LINE_300]], i64 %[[ARG1]], i64 %[[ARG2]])
   // CHECK-UBSAN-NOT:  call void @__ubsan_handle_shift_out_of_bounds
 
   // CHECK-TRAP:      call void @llvm.ubsantrap(i8 20) [[NR_NUW]]
@@ -112,7 +111,7 @@ int rsh_inbounds(int a, int b) {
 
   // CHECK-UBSAN:      %[[ARG1:.*]] = zext
   // CHECK-UBSAN-NEXT: %[[ARG2:.*]] = zext
-  // CHECK-UBSAN-NEXT: call void @__ubsan_handle_shift_out_of_bounds(i8* bitcast ({{.*}} @[[LINE_400]] to i8*), i64 %[[ARG1]], i64 %[[ARG2]])
+  // CHECK-UBSAN-NEXT: call void @__ubsan_handle_shift_out_of_bounds(ptr @[[LINE_400]], i64 %[[ARG1]], i64 %[[ARG2]])
 
   // CHECK-TRAP:      call void @llvm.ubsantrap(i8 20) [[NR_NUW]]
   // CHECK-TRAP-NEXT: unreachable
@@ -125,7 +124,7 @@ int rsh_inbounds(int a, int b) {
 
 // CHECK-COMMON-LABEL: @load
 int load(int *p) {
-  // CHECK-UBSAN: call void @__ubsan_handle_type_mismatch_v1(i8* bitcast ({{.*}} @[[LINE_500]] to i8*), i64 %{{.*}})
+  // CHECK-UBSAN: call void @__ubsan_handle_type_mismatch_v1(ptr @[[LINE_500]], i64 %{{.*}})
 
   // CHECK-TRAP:      call void @llvm.ubsantrap(i8 22) [[NR_NUW]]
   // CHECK-TRAP-NEXT: unreachable
@@ -135,7 +134,7 @@ int load(int *p) {
 
 // CHECK-COMMON-LABEL: @store
 void store(int *p, int q) {
-  // CHECK-UBSAN: call void @__ubsan_handle_type_mismatch_v1(i8* bitcast ({{.*}} @[[LINE_600]] to i8*), i64 %{{.*}})
+  // CHECK-UBSAN: call void @__ubsan_handle_type_mismatch_v1(ptr @[[LINE_600]], i64 %{{.*}})
 
   // CHECK-TRAP:      call void @llvm.ubsantrap(i8 22) [[NR_NUW]]
   // CHECK-TRAP-NEXT: unreachable
@@ -147,7 +146,7 @@ struct S { int k; };
 
 // CHECK-COMMON-LABEL: @member_access
 int *member_access(struct S *p) {
-  // CHECK-UBSAN: call void @__ubsan_handle_type_mismatch_v1(i8* bitcast ({{.*}} @[[LINE_700]] to i8*), i64 %{{.*}})
+  // CHECK-UBSAN: call void @__ubsan_handle_type_mismatch_v1(ptr @[[LINE_700]], i64 %{{.*}})
 
   // CHECK-TRAP:      call void @llvm.ubsantrap(i8 22) [[NR_NUW]]
   // CHECK-TRAP-NEXT: unreachable
@@ -159,7 +158,7 @@ int *member_access(struct S *p) {
 int signed_overflow(int a, int b) {
   // CHECK-UBSAN:      %[[ARG1:.*]] = zext
   // CHECK-UBSAN-NEXT: %[[ARG2:.*]] = zext
-  // CHECK-UBSAN-NEXT: call void @__ubsan_handle_add_overflow(i8* bitcast ({{.*}} @[[LINE_800]] to i8*), i64 %[[ARG1]], i64 %[[ARG2]])
+  // CHECK-UBSAN-NEXT: call void @__ubsan_handle_add_overflow(ptr @[[LINE_800]], i64 %[[ARG1]], i64 %[[ARG2]])
 
   // CHECK-TRAP:      call void @llvm.ubsantrap(i8 0) [[NR_NUW]]
   // CHECK-TRAP-NEXT: unreachable
@@ -182,7 +181,7 @@ void vla_bound(int n) {
   // CHECK-UBSAN:      icmp sgt i32 %[[PARAM:.*]], 0
   //
   // CHECK-UBSAN:      %[[ARG:.*]] = zext i32 %[[PARAM]] to i64
-  // CHECK-UBSAN-NEXT: call void @__ubsan_handle_vla_bound_not_positive(i8* bitcast ({{.*}} @[[LINE_900]] to i8*), i64 %[[ARG]])
+  // CHECK-UBSAN-NEXT: call void @__ubsan_handle_vla_bound_not_positive(ptr @[[LINE_900]], i64 %[[ARG]])
 #line 900
   int arr[n * 3];
 }
@@ -192,7 +191,7 @@ void vla_bound_unsigned(unsigned int n) {
   // CHECK-UBSAN:      icmp ugt i32 %[[PARAM:.*]], 0
   //
   // CHECK-UBSAN:      %[[ARG:.*]] = zext i32 %[[PARAM]] to i64
-  // CHECK-UBSAN-NEXT: call void @__ubsan_handle_vla_bound_not_positive(i8* bitcast ({{.*}} @[[LINE_1000]] to i8*), i64 %[[ARG]])
+  // CHECK-UBSAN-NEXT: call void @__ubsan_handle_vla_bound_not_positive(ptr @[[LINE_1000]], i64 %[[ARG]])
 #line 1000
   int arr[n * 3];
 }
@@ -227,7 +226,7 @@ int float_int_overflow(float f) {
 
   // CHECK-UBSAN: %[[CAST:.*]] = bitcast float %[[F]] to i32
   // CHECK-UBSAN: %[[ARG:.*]] = zext i32 %[[CAST]] to i64
-  // CHECK-UBSAN: call void @__ubsan_handle_float_cast_overflow(i8* bitcast ({{.*}} @[[LINE_1200]] to i8*), i64 %[[ARG]]
+  // CHECK-UBSAN: call void @__ubsan_handle_float_cast_overflow(ptr @[[LINE_1200]], i64 %[[ARG]]
 
   // CHECK-TRAP:      call void @llvm.ubsantrap(i8 5) [[NR_NUW]]
   // CHECK-TRAP-NEXT: unreachable
@@ -244,9 +243,9 @@ int long_double_int_overflow(long double ld) {
   // CHECK-COMMON: %[[INBOUNDS:.*]] = and i1 %[[GE]], %[[LE]]
   // CHECK-COMMON-NEXT: br i1 %[[INBOUNDS]]
 
-  // CHECK-UBSAN: store x86_fp80 %[[F]], x86_fp80* %[[ALLOCA:.*]], align 16, !nosanitize
-  // CHECK-UBSAN: %[[ARG:.*]] = ptrtoint x86_fp80* %[[ALLOCA]] to i64
-  // CHECK-UBSAN: call void @__ubsan_handle_float_cast_overflow(i8* bitcast ({{.*}} @[[LINE_1300]] to i8*), i64 %[[ARG]]
+  // CHECK-UBSAN: store x86_fp80 %[[F]], ptr %[[ALLOCA:.*]], align 16, !nosanitize
+  // CHECK-UBSAN: %[[ARG:.*]] = ptrtoint ptr %[[ALLOCA]] to i64
+  // CHECK-UBSAN: call void @__ubsan_handle_float_cast_overflow(ptr @[[LINE_1300]], i64 %[[ARG]]
 
   // CHECK-TRAP:      call void @llvm.ubsantrap(i8 5) [[NR_NUW]]
   // CHECK-TRAP-NEXT: unreachable
@@ -261,7 +260,7 @@ unsigned float_uint_overflow(float f) {
   // CHECK-COMMON: %[[INBOUNDS:.*]] = and i1 %[[GE]], %[[LE]]
   // CHECK-COMMON-NEXT: br i1 %[[INBOUNDS]]
 
-  // CHECK-UBSAN: call void @__ubsan_handle_float_cast_overflow(i8* bitcast ({{.*}} @[[LINE_1400]] to i8*),
+  // CHECK-UBSAN: call void @__ubsan_handle_float_cast_overflow(ptr @[[LINE_1400]],
 
   // CHECK-TRAP:      call void @llvm.ubsantrap(i8 5) [[NR_NUW]]
   // CHECK-TRAP-NEXT: unreachable
@@ -276,7 +275,7 @@ signed char fp16_char_overflow(__fp16 *p) {
   // CHECK-COMMON: %[[INBOUNDS:.*]] = and i1 %[[GE]], %[[LE]]
   // CHECK-COMMON-NEXT: br i1 %[[INBOUNDS]]
 
-  // CHECK-UBSAN: call void @__ubsan_handle_float_cast_overflow(i8* bitcast ({{.*}} @[[LINE_1500]] to i8*),
+  // CHECK-UBSAN: call void @__ubsan_handle_float_cast_overflow(ptr @[[LINE_1500]],
 
   // CHECK-TRAP:      call void @llvm.ubsantrap(i8 5) [[NR_NUW]]
   // CHECK-TRAP-NEXT: unreachable
@@ -322,7 +321,7 @@ _Bool sour_bool(_Bool *p) {
   // CHECK-COMMON: %[[OK:.*]] = icmp ule i8 {{.*}}, 1
   // CHECK-COMMON: br i1 %[[OK]]
 
-  // CHECK-UBSAN: call void @__ubsan_handle_load_invalid_value(i8* bitcast ({{.*}}), i64 {{.*}})
+  // CHECK-UBSAN: call void @__ubsan_handle_load_invalid_value(ptr {{.*}}, i64 {{.*}})
 
   // CHECK-TRAP: call void @llvm.ubsantrap(i8 10) [[NR_NUW]]
   // CHECK-TRAP: unreachable
@@ -332,7 +331,7 @@ _Bool sour_bool(_Bool *p) {
 // CHECK-COMMON-LABEL: @ret_nonnull
 __attribute__((returns_nonnull))
 int *ret_nonnull(int *a) {
-  // CHECK-COMMON: [[OK:%.*]] = icmp ne i32* {{.*}}, null
+  // CHECK-COMMON: [[OK:%.*]] = icmp ne ptr {{.*}}, null
   // CHECK-COMMON: br i1 [[OK]]
 
   // CHECK-UBSAN: call void @__ubsan_handle_nonnull_return
@@ -345,7 +344,7 @@ int *ret_nonnull(int *a) {
 // CHECK-COMMON-LABEL: @call_decl_nonnull
 __attribute__((nonnull)) void decl_nonnull(int *a);
 void call_decl_nonnull(int *a) {
-  // CHECK-COMMON: [[OK:%.*]] = icmp ne i32* {{.*}}, null
+  // CHECK-COMMON: [[OK:%.*]] = icmp ne ptr {{.*}}, null
   // CHECK-COMMON: br i1 [[OK]]
 
   // CHECK-UBSAN: call void @__ubsan_handle_nonnull_arg
@@ -359,11 +358,11 @@ extern void *memcpy (void *, const void *, unsigned) __attribute__((nonnull(1, 2
 
 // CHECK-COMMON-LABEL: @call_memcpy_nonnull
 void call_memcpy_nonnull(void *p, void *q, int sz) {
-  // CHECK-COMMON: icmp ne i8* {{.*}}, null
+  // CHECK-COMMON: icmp ne ptr {{.*}}, null
   // CHECK-UBSAN: call void @__ubsan_handle_nonnull_arg
   // CHECK-TRAP: call void @llvm.ubsantrap(i8 16)
 
-  // CHECK-COMMON: icmp ne i8* {{.*}}, null
+  // CHECK-COMMON: icmp ne ptr {{.*}}, null
   // CHECK-UBSAN: call void @__ubsan_handle_nonnull_arg
   // CHECK-TRAP: call void @llvm.ubsantrap(i8 16)
   memcpy(p, q, sz);
@@ -373,11 +372,11 @@ extern void *memmove (void *, const void *, unsigned) __attribute__((nonnull(1,
 
 // CHECK-COMMON-LABEL: @call_memmove_nonnull
 void call_memmove_nonnull(void *p, void *q, int sz) {
-  // CHECK-COMMON: icmp ne i8* {{.*}}, null
+  // CHECK-COMMON: icmp ne ptr {{.*}}, null
   // CHECK-UBSAN: call void @__ubsan_handle_nonnull_arg
   // CHECK-TRAP: call void @llvm.ubsantrap(i8 16)
 
-  // CHECK-COMMON: icmp ne i8* {{.*}}, null
+  // CHECK-COMMON: icmp ne ptr {{.*}}, null
   // CHECK-UBSAN: call void @__ubsan_handle_nonnull_arg
   // CHECK-TRAP: call void @llvm.ubsantrap(i8 16)
   memmove(p, q, sz);
@@ -386,7 +385,7 @@ void call_memmove_nonnull(void *p, void *q, int sz) {
 // CHECK-COMMON-LABEL: @call_nonnull_variadic
 __attribute__((nonnull)) void nonnull_variadic(int a, ...);
 void call_nonnull_variadic(int a, int *b) {
-  // CHECK-COMMON: [[OK:%.*]] = icmp ne i32* {{.*}}, null
+  // CHECK-COMMON: [[OK:%.*]] = icmp ne ptr {{.*}}, null
   // CHECK-COMMON: br i1 [[OK]]
 
   // CHECK-UBSAN: call void @__ubsan_handle_nonnull_arg
diff --git a/clang/test/CodeGen/code-coverage-tsan.c b/clang/test/CodeGen/code-coverage-tsan.c
index 827a746d153fc..c7928fd505e0c 100644
--- a/clang/test/CodeGen/code-coverage-tsan.c
+++ b/clang/test/CodeGen/code-coverage-tsan.c
@@ -1,12 +1,12 @@
 /// -fprofile-update=atomic (implied by -fsanitize=thread) requires the
 /// (potentially concurrent) counter updates to be atomic.
-// RUN: %clang_cc1 -no-opaque-pointers %s -triple x86_64 -emit-llvm -fprofile-update=atomic -ftest-coverage -fprofile-arcs \
+// RUN: %clang_cc1 %s -triple x86_64 -emit-llvm -fprofile-update=atomic -ftest-coverage -fprofile-arcs \
 // RUN:   -coverage-notes-file /dev/null -coverage-data-file /dev/null -o - | FileCheck %s
 
 // CHECK-LABEL: void @foo()
 /// Two counters are incremented by __tsan_atomic64_fetch_add.
-// CHECK:         atomicrmw add i64* {{.*}} @__llvm_gcov_ctr{{.*}} monotonic, align 8
-// CHECK-NEXT:    atomicrmw sub i32*
+// CHECK:         atomicrmw add ptr @__llvm_gcov_ctr{{.*}} monotonic, align 8
+// CHECK-NEXT:    atomicrmw sub ptr
 
 _Atomic(int) cnt;
 void foo(void) { cnt--; }
diff --git a/clang/test/CodeGen/complex-math.c b/clang/test/CodeGen/complex-math.c
index 8f26658cac14d..d5af6db809534 100644
--- a/clang/test/CodeGen/complex-math.c
+++ b/clang/test/CodeGen/complex-math.c
@@ -136,7 +136,7 @@ float _Complex div_float_rc(float a, float _Complex b) {
   // SPIR: call spir_func {{.*}} @__divsc3(
 
   // a / b = (A+iB) / (C+iD) = ((AC+BD)/(CC+DD)) + i((BC-AD)/(CC+DD))
-  // AARCH64-FASTMATH-LABEL: @div_float_rc(float noundef %a, [2 x float] noundef %b.coerce)
+  // AARCH64-FASTMATH-LABEL: @div_float_rc(float noundef nofpclass(nan inf) %a, [2 x float] noundef nofpclass(nan inf) %b.coerce)
   // A = a
   // B = 0
   //
@@ -165,7 +165,7 @@ float _Complex div_float_cc(float _Complex a, float _Complex b) {
   // SPIR: call spir_func {{.*}} @__divsc3(
 
   // a / b = (A+iB) / (C+iD) = ((AC+BD)/(CC+DD)) + i((BC-AD)/(CC+DD))
-  // AARCH64-FASTMATH-LABEL: @div_float_cc([2 x float] noundef %a.coerce, [2 x float] noundef %b.coerce)
+  // AARCH64-FASTMATH-LABEL: @div_float_cc([2 x float] noundef nofpclass(nan inf) %a.coerce, [2 x float] noundef nofpclass(nan inf) %b.coerce)
   //
   // AARCH64-FASTMATH: [[AC:%.*]] = fmul fast float
   // AARCH64-FASTMATH: [[BD:%.*]] = fmul fast float
@@ -313,7 +313,7 @@ double _Complex div_double_rc(double a, double _Complex b) {
   // SPIR: call spir_func {{.*}} @__divdc3(
 
   // a / b = (A+iB) / (C+iD) = ((AC+BD)/(CC+DD)) + i((BC-AD)/(CC+DD))
-  // AARCH64-FASTMATH-LABEL: @div_double_rc(double noundef %a, [2 x double] noundef %b.coerce)
+  // AARCH64-FASTMATH-LABEL: @div_double_rc(double noundef nofpclass(nan inf) %a, [2 x double] noundef nofpclass(nan inf) %b.coerce)
   // A = a
   // B = 0
   //
@@ -342,7 +342,7 @@ double _Complex div_double_cc(double _Complex a, double _Complex b) {
   // SPIR: call spir_func {{.*}} @__divdc3(
 
   // a / b = (A+iB) / (C+iD) = ((AC+BD)/(CC+DD)) + i((BC-AD)/(CC+DD))
-  // AARCH64-FASTMATH-LABEL: @div_double_cc([2 x double] noundef %a.coerce, [2 x double] noundef %b.coerce)
+  // AARCH64-FASTMATH-LABEL: @div_double_cc([2 x double] noundef nofpclass(nan inf) %a.coerce, [2 x double] noundef nofpclass(nan inf) %b.coerce)
   //
   // AARCH64-FASTMATH: [[AC:%.*]] = fmul fast double
   // AARCH64-FASTMATH: [[BD:%.*]] = fmul fast double
@@ -506,7 +506,7 @@ long double _Complex div_long_double_rc(long double a, long double _Complex b) {
   // SPIR: call spir_func {{.*}} @__divdc3(
 
   // a / b = (A+iB) / (C+iD) = ((AC+BD)/(CC+DD)) + i((BC-AD)/(CC+DD))
-  // AARCH64-FASTMATH-LABEL: @div_long_double_rc(fp128 noundef %a, [2 x fp128] noundef %b.coerce)
+  // AARCH64-FASTMATH-LABEL: @div_long_double_rc(fp128 noundef nofpclass(nan inf) %a, [2 x fp128] noundef nofpclass(nan inf) %b.coerce)
   // A = a
   // B = 0
   //
@@ -538,7 +538,7 @@ long double _Complex div_long_double_cc(long double _Complex a, long double _Com
   // SPIR: call spir_func {{.*}} @__divdc3(
 
   // a / b = (A+iB) / (C+iD) = ((AC+BD)/(CC+DD)) + i((BC-AD)/(CC+DD))
-  // AARCH64-FASTMATH-LABEL: @div_long_double_cc([2 x fp128] noundef %a.coerce, [2 x fp128] noundef %b.coerce)
+  // AARCH64-FASTMATH-LABEL: @div_long_double_cc([2 x fp128] noundef nofpclass(nan inf) %a.coerce, [2 x fp128] noundef nofpclass(nan inf) %b.coerce)
   //
   // AARCH64-FASTMATH: [[AC:%.*]] = fmul fast fp128
   // AARCH64-FASTMATH: [[BD:%.*]] = fmul fast fp128
diff --git a/clang/test/CodeGen/compound-literal.c b/clang/test/CodeGen/compound-literal.c
index 967e3fb53dcf9..ecc2c73edfb1e 100644
--- a/clang/test/CodeGen/compound-literal.c
+++ b/clang/test/CodeGen/compound-literal.c
@@ -1,4 +1,4 @@
-// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-apple-darwin -emit-llvm %s -o - | FileCheck %s
+// RUN: %clang_cc1 -triple x86_64-apple-darwin -emit-llvm %s -o - | FileCheck %s
 
 // Capture the type and name so matching later is cleaner.
 struct CompoundTy { int a; };
@@ -28,17 +28,15 @@ void f(void) {
   // CHECK: [[S:%[a-zA-Z0-9.]+]] = alloca [[STRUCT:%[a-zA-Z0-9.]+]],
   struct S s;
   // CHECK-NEXT: [[COMPOUNDLIT:%[a-zA-Z0-9.]+]] = alloca [[STRUCT]]
-  // CHECK-NEXT: [[CX:%[a-zA-Z0-9.]+]] = getelementptr inbounds [[STRUCT]], [[STRUCT]]* [[COMPOUNDLIT]], i32 0, i32 0
-  // CHECK-NEXT: [[SY:%[a-zA-Z0-9.]+]] = getelementptr inbounds [[STRUCT]], [[STRUCT]]* [[S]], i32 0, i32 1
-  // CHECK-NEXT: [[TMP:%[a-zA-Z0-9.]+]] = load i32, i32* [[SY]]
-  // CHECK-NEXT: store i32 [[TMP]], i32* [[CX]]
-  // CHECK-NEXT: [[CY:%[a-zA-Z0-9.]+]] = getelementptr inbounds [[STRUCT]], [[STRUCT]]* [[COMPOUNDLIT]], i32 0, i32 1
-  // CHECK-NEXT: [[SX:%[a-zA-Z0-9.]+]] = getelementptr inbounds [[STRUCT]], [[STRUCT]]* [[S]], i32 0, i32 0
-  // CHECK-NEXT: [[TMP:%[a-zA-Z0-9.]+]] = load i32, i32* [[SX]]
-  // CHECK-NEXT: store i32 [[TMP]], i32* [[CY]]
-  // CHECK-NEXT: [[SI8:%[a-zA-Z0-9.]+]] = bitcast [[STRUCT]]* [[S]] to i8*
-  // CHECK-NEXT: [[COMPOUNDLITI8:%[a-zA-Z0-9.]+]] = bitcast [[STRUCT]]* [[COMPOUNDLIT]] to i8*
-  // CHECK-NEXT: call void @llvm.memcpy{{.*}}(i8* align {{[0-9]+}} [[SI8]], i8* align {{[0-9]+}} [[COMPOUNDLITI8]]
+  // CHECK-NEXT: [[CX:%[a-zA-Z0-9.]+]] = getelementptr inbounds [[STRUCT]], ptr [[COMPOUNDLIT]], i32 0, i32 0
+  // CHECK-NEXT: [[SY:%[a-zA-Z0-9.]+]] = getelementptr inbounds [[STRUCT]], ptr [[S]], i32 0, i32 1
+  // CHECK-NEXT: [[TMP:%[a-zA-Z0-9.]+]] = load i32, ptr [[SY]]
+  // CHECK-NEXT: store i32 [[TMP]], ptr [[CX]]
+  // CHECK-NEXT: [[CY:%[a-zA-Z0-9.]+]] = getelementptr inbounds [[STRUCT]], ptr [[COMPOUNDLIT]], i32 0, i32 1
+  // CHECK-NEXT: [[SX:%[a-zA-Z0-9.]+]] = getelementptr inbounds [[STRUCT]], ptr [[S]], i32 0, i32 0
+  // CHECK-NEXT: [[TMP:%[a-zA-Z0-9.]+]] = load i32, ptr [[SX]]
+  // CHECK-NEXT: store i32 [[TMP]], ptr [[CY]]
+  // CHECK-NEXT: call void @llvm.memcpy{{.*}}(ptr align {{[0-9]+}} [[S]], ptr align {{[0-9]+}} [[COMPOUNDLIT]]
   s = (S){s.y,s.x};
   // CHECK-NEXT: ret void
 }
@@ -56,24 +54,22 @@ struct G g(int x, int y, int z) {
   // CHECK-NEXT: store i32
 
   // Evaluate the compound literal directly in the result value slot.
-  // CHECK-NEXT: [[T0:%.*]] = getelementptr inbounds [[G]], [[G]]* [[RESULT]], i32 0, i32 0
-  // CHECK-NEXT: [[T1:%.*]] = load i32, i32* [[X]], align 4
+  // CHECK-NEXT: [[T0:%.*]] = getelementptr inbounds [[G]], ptr [[RESULT]], i32 0, i32 0
+  // CHECK-NEXT: [[T1:%.*]] = load i32, ptr [[X]], align 4
   // CHECK-NEXT: [[T2:%.*]] = trunc i32 [[T1]] to i16
-  // CHECK-NEXT: store i16 [[T2]], i16* [[T0]], align 2
-  // CHECK-NEXT: [[T0:%.*]] = getelementptr inbounds [[G]], [[G]]* [[RESULT]], i32 0, i32 1
-  // CHECK-NEXT: [[T1:%.*]] = load i32, i32* [[Y]], align 4
+  // CHECK-NEXT: store i16 [[T2]], ptr [[T0]], align 2
+  // CHECK-NEXT: [[T0:%.*]] = getelementptr inbounds [[G]], ptr [[RESULT]], i32 0, i32 1
+  // CHECK-NEXT: [[T1:%.*]] = load i32, ptr [[Y]], align 4
   // CHECK-NEXT: [[T2:%.*]] = trunc i32 [[T1]] to i16
-  // CHECK-NEXT: store i16 [[T2]], i16* [[T0]], align 2
-  // CHECK-NEXT: [[T0:%.*]] = getelementptr inbounds [[G]], [[G]]* [[RESULT]], i32 0, i32 2
-  // CHECK-NEXT: [[T1:%.*]] = load i32, i32* [[Z]], align 4
+  // CHECK-NEXT: store i16 [[T2]], ptr [[T0]], align 2
+  // CHECK-NEXT: [[T0:%.*]] = getelementptr inbounds [[G]], ptr [[RESULT]], i32 0, i32 2
+  // CHECK-NEXT: [[T1:%.*]] = load i32, ptr [[Z]], align 4
   // CHECK-NEXT: [[T2:%.*]] = trunc i32 [[T1]] to i16
-  // CHECK-NEXT: store i16 [[T2]], i16* [[T0]], align 2
+  // CHECK-NEXT: store i16 [[T2]], ptr [[T0]], align 2
   return (struct G) { x, y, z };
 
-  // CHECK-NEXT: [[T0:%.*]] = bitcast i48* [[COERCE_TEMP]] to i8*
-  // CHECK-NEXT: [[T1:%.*]] = bitcast [[G]]* [[RESULT]] to i8*
-  // CHECK-NEXT: call void @llvm.memcpy.p0i8.p0i8.i64(i8* align {{[0-9]+}} [[T0]], i8* align {{[0-9]+}} [[T1]], i64 6
-  // CHECK-NEXT: [[T0:%.*]] = load i48, i48* [[COERCE_TEMP]]
+  // CHECK-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align {{[0-9]+}} [[COERCE_TEMP]], ptr align {{[0-9]+}} [[RESULT]], i64 6
+  // CHECK-NEXT: [[T0:%.*]] = load i48, ptr [[COERCE_TEMP]]
   // CHECK-NEXT: ret i48 [[T0]]
 }
 
@@ -81,9 +77,9 @@ struct G g(int x, int y, int z) {
 // const pointer to a variable initialized by a compound literal.
 // CHECK-LABEL: define{{.*}} i32 @compareMyCLH() #0
 int compareMyCLH(void) {
-  // CHECK: store i8* bitcast ([[MY_CLH]] to i8*)
+  // CHECK: store [[MY_CLH]]
   const void *a = MyCLH;
-  // CHECK: store i8* bitcast ([[MY_CLH]] to i8*)
+  // CHECK: store [[MY_CLH]]
   const void *b = MyCLH;
   return a == b;
 }
@@ -92,7 +88,7 @@ int compareMyCLH(void) {
 // for a local variable.
 // CHECK-LABEL: define{{.*}} i32 @compound_array_fn()
 // CHECK: [[COMPOUND_ARRAY:%.*]] = alloca [8 x i32]
-// CHECK: call void @llvm.memcpy.p0i8.p0i8.i64({{.*}}, i64 32, i1 false)
+// CHECK: call void @llvm.memcpy.p0.p0.i64({{.*}}, i64 32, i1 false)
 int compound_array_fn(void) {
   int compound_array[] = (int[]){1,2,3,4,5,6,7,8};
   return compound_array[0];
diff --git a/clang/test/CodeGen/const-init.c b/clang/test/CodeGen/const-init.c
index 4748d71dca966..7f6035a2178a8 100644
--- a/clang/test/CodeGen/const-init.c
+++ b/clang/test/CodeGen/const-init.c
@@ -1,6 +1,6 @@
 // setting strict FP behaviour in the run line below tests that the compiler
 // does the right thing for global compound literals (compoundliteral test)
-// RUN: %clang_cc1 -no-opaque-pointers -triple i386-pc-linux-gnu -ffreestanding -Wno-pointer-to-int-cast -Wno-int-conversion -ffp-exception-behavior=strict -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple i386-pc-linux-gnu -ffreestanding -Wno-pointer-to-int-cast -Wno-int-conversion -ffp-exception-behavior=strict -emit-llvm -o - %s | FileCheck %s
 
 #include <stdint.h>
 
@@ -61,24 +61,24 @@ struct {
 } __attribute__((__packed__)) gv1  = { .a = 0x0, .b = 7,  };
 
 // PR5118
-// CHECK: @gv2 ={{.*}} global %struct.anon.0 <{ i8 1, i8* null }>, align 1
+// CHECK: @gv2 ={{.*}} global %struct.anon.0 <{ i8 1, ptr null }>, align 1
 struct {
   unsigned char a;
   char *b;
 } __attribute__((__packed__)) gv2 = { 1, (void*)0 };
 
 // Global references
-// CHECK: @g11.l0 = internal global i32 ptrtoint (i32 ()* @g11 to i32)
+// CHECK: @g11.l0 = internal global i32 ptrtoint (ptr @g11 to i32)
 long g11(void) {
   static long l0 = (long) g11;
   return l0;
 }
 
-// CHECK: @g12 ={{.*}} global i32 ptrtoint (i8* @g12_tmp to i32)
+// CHECK: @g12 ={{.*}} global i32 ptrtoint (ptr @g12_tmp to i32)
 static char g12_tmp;
 long g12 = (long) &g12_tmp;
 
-// CHECK: @g13 ={{.*}} global [1 x %struct.g13_s0] [%struct.g13_s0 { i32 ptrtoint (i8* @g12_tmp to i32) }]
+// CHECK: @g13 ={{.*}} global [1 x %struct.g13_s0] [%struct.g13_s0 { i32 ptrtoint (ptr @g12_tmp to i32) }]
 struct g13_s0 {
    long a;
 };
@@ -86,7 +86,7 @@ struct g13_s0 g13[] = {
    { (long) &g12_tmp }
 };
 
-// CHECK: @g14 ={{.*}} global i8* inttoptr (i32 100 to i8*)
+// CHECK: @g14 ={{.*}} global ptr inttoptr (i32 100 to ptr)
 void *g14 = (void*) 100;
 
 // CHECK: @g15 ={{.*}} global i32 -1
@@ -95,16 +95,16 @@ int g15 = (int) (char) ((void*) 0 + 255);
 // CHECK: @g16 ={{.*}} global i64 4294967295
 long long g16 = (long long) ((void*) 0xFFFFFFFF);
 
-// CHECK: @g17 ={{.*}} global i32* @g15
+// CHECK: @g17 ={{.*}} global ptr @g15
 int *g17 = (int *) ((long) &g15);
 
-// CHECK: @g18.p = internal global [1 x i32*] [i32* @g19]
+// CHECK: @g18.p = internal global [1 x ptr] [ptr @g19]
 void g18(void) {
   extern int g19;
   static int *p[] = { &g19 };
 }
 
-// CHECK: @g20.l0 = internal global %struct.g20_s1 { %struct.g20_s0* null, %struct.g20_s0** getelementptr inbounds (%struct.g20_s1, %struct.g20_s1* @g20.l0, i32 0, i32 0) }
+// CHECK: @g20.l0 = internal global %struct.g20_s1 { ptr null, ptr @g20.l0 }
 struct g20_s0;
 struct g20_s1 {
   struct g20_s0 *f0, **f1;
@@ -123,14 +123,14 @@ struct g22 {int x;} __attribute((packed));
 struct g23 {char a; short b; char c; struct g22 d;};
 struct g23 g24 = {1,2,3,4};
 
-// CHECK: @g25.g26 = internal global i8* getelementptr inbounds ([4 x i8], [4 x i8]* @[[FUNC:.*]], i32 0, i32 0)
+// CHECK: @g25.g26 = internal global ptr @[[FUNC:.*]], align 4
 // CHECK: @[[FUNC]] = private unnamed_addr constant [4 x i8] c"g25\00"
 int g25(void) {
   static const char *g26 = __func__;
   return *g26;
 }
 
-// CHECK: @g27.x = internal global i8* bitcast (i8** @g27.x to i8*), align 4
+// CHECK: @g27.x = internal global ptr @g27.x, align 4
 void g27(void) { // PR8073
   static void *x = &x;
 }
@@ -155,7 +155,7 @@ void g29(void) {
       DCC_PASSWD passwd;
   } DCC_SRVR_NM;
   // CHECK: @g29.a = internal global %struct.DCC_SRVR_NM { [2 x i8] c"@\00" }, align 1
-  // CHECK: @g29.b = internal global [1 x i32] [i32 ptrtoint ([5 x i8]* @.str.1 to i32)], align 4
+  // CHECK: @g29.b = internal global [1 x i32] [i32 ptrtoint (ptr @.str.1 to i32)], align 4
   // CHECK: @g29.c = internal global [1 x i32] [i32 97], align 4
   static DCC_SRVR_NM a = { {"@"} };
   static int b[1] = { "asdf" };
diff --git a/clang/test/CodeGen/debug-info-block-decl.c b/clang/test/CodeGen/debug-info-block-decl.c
index 553cb60c72cc6..d94c2e6338c2b 100644
--- a/clang/test/CodeGen/debug-info-block-decl.c
+++ b/clang/test/CodeGen/debug-info-block-decl.c
@@ -1,9 +1,9 @@
-// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-apple-darwin10 -debug-info-kind=limited -fblocks -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple x86_64-apple-darwin10 -debug-info-kind=limited -fblocks -emit-llvm -o - %s | FileCheck %s
 // Assignment and block entry should point to the same line.
 // rdar://problem/14039866
 
 // CHECK: define{{.*}}@main()
-// CHECK: store{{.*}}bitcast{{.*}}, !dbg ![[ASSIGNMENT:[0-9]+]]
+// CHECK: store {{.*}}, !dbg ![[ASSIGNMENT:[0-9]+]]
 // CHECK: define {{.*}} @__main_block_invoke
 // CHECK: , !dbg ![[BLOCK_ENTRY:[0-9]+]]
 
diff --git a/clang/test/CodeGen/debug-info-codeview-buildinfo.c b/clang/test/CodeGen/debug-info-codeview-buildinfo.c
index 4096fac5f7437..4fc55af10a6e2 100644
--- a/clang/test/CodeGen/debug-info-codeview-buildinfo.c
+++ b/clang/test/CodeGen/debug-info-codeview-buildinfo.c
@@ -33,7 +33,7 @@ int main(void) { return 42; }
 // RELATIVE: 0x{{.+}} | LF_BUILDINFO [size = {{.+}}]
 // RELATIVE:          0x{{.+}}: `.`
 
-// DISABLE-NOT: cc1
+// DISABLE-NOT: "-cc1"
 // DISABLE: 0x{{.+}} | LF_BUILDINFO [size = {{.+}}]
 // DISABLE-NEXT:          0x{{.+}}: `{{.*}}`
 // DISABLE-NEXT:          <no type>: ``
diff --git a/clang/test/CodeGen/eval-method-fast-math.cpp b/clang/test/CodeGen/eval-method-fast-math.cpp
deleted file mode 100644
index 0b5b3780efa3a..0000000000000
--- a/clang/test/CodeGen/eval-method-fast-math.cpp
+++ /dev/null
@@ -1,117 +0,0 @@
-// RUN: %clang_cc1 -fexperimental-strict-floating-point  \
-// RUN: -triple x86_64-linux-gnu -emit-llvm -o - %s  \
-// RUN: | FileCheck %s -check-prefixes=CHECK
-
-// RUN: %clang_cc1 -triple i386--linux -emit-llvm -o - %s \
-// RUN: | FileCheck %s -check-prefixes=CHECK-EXT
-
-// RUN: %clang_cc1 -fexperimental-strict-floating-point  \
-// RUN: -mreassociate -freciprocal-math -ffp-contract=fast \
-// RUN: -ffast-math -triple x86_64-linux-gnu \
-// RUN: -emit-llvm -o - %s \
-// RUN: | FileCheck %s -check-prefixes=CHECK-FAST
-
-// RUN: %clang_cc1 -triple i386--linux -mreassociate -freciprocal-math \
-// RUN: -ffp-contract=fast -ffast-math -emit-llvm -o - %s \
-// RUN: | FileCheck %s -check-prefixes=CHECK-FAST
-
-float a = 1.0f, b = 2.0f, c = 3.0f;
-#pragma float_control(precise, off)
-float res2 = a + b + c;
-int val3 = __FLT_EVAL_METHOD__;
-#pragma float_control(precise, on)
-float res3 = a + b + c;
-int val4 = __FLT_EVAL_METHOD__;
-
-// CHECK: @val3 = global i32 -1
-// CHECK: @val4 = global i32 0
-
-// CHECK-EXT: @val3 = global i32 -1
-// CHECK-EXT: @val4 = global i32 2
-
-// CHECK-FAST: @val3 = global i32 -1
-// CHECK-FAST: @val4 = global i32 -1
-
-float res;
-int add(float a, float b, float c) {
-  // CHECK: fadd float
-  // CHECK: load float, ptr
-  // CHECK: fadd float
-  // CHECK: store float
-  // CHECK: ret i32 0
-  res = a + b + c;
-  return __FLT_EVAL_METHOD__;
-}
-
-int add_precise(float a, float b, float c) {
-#pragma float_control(precise, on)
-  // CHECK: fadd float
-  // CHECK: load float, ptr
-  // CHECK: fadd float
-  // CHECK: store float
-  // CHECK: ret i32 0
-  res = a + b + c;
-  return __FLT_EVAL_METHOD__;
-}
-
-#pragma float_control(push)
-#pragma float_control(precise, on)
-int add_precise_1(float a, float b, float c) {
-  // CHECK: fadd float
-  // CHECK: load float, ptr
-  // CHECK: fadd float
-  // CHECK: store float
-  // CHECK: ret i32 0
-  res = a + b + c;
-  return __FLT_EVAL_METHOD__;
-}
-#pragma float_control(pop)
-
-int add_not_precise(float a, float b, float c) {
-  // Fast-math is enabled with this pragma.
-#pragma float_control(precise, off)
-  // CHECK: fadd fast float
-  // CHECK: load float, ptr
-  // CHECK: fadd fast float
-  // CHECK: float {{.*}}, ptr
-  // CHECK: ret i32 -1
-  res = a + b + c;
-  return __FLT_EVAL_METHOD__;
-}
-
-#pragma float_control(push)
-// Fast-math is enabled with this pragma.
-#pragma float_control(precise, off)
-int add_not_precise_1(float a, float b, float c) {
-  // CHECK: fadd fast float
-  // CHECK: load float, ptr
-  // CHECK: fadd fast float
-  // CHECK: float {{.*}}, ptr
-  // CHECK: ret i32 -1
-  res = a + b + c;
-  return __FLT_EVAL_METHOD__;
-}
-#pragma float_control(pop)
-
-int getFPEvalMethod() {
-  // CHECK: ret i32 0
-  return __FLT_EVAL_METHOD__;
-}
-
-float res1;
-int whatever(float a, float b, float c) {
-#pragma float_control(precise, off)
-  // CHECK: load float, ptr
-  // CHECK: fadd fast float
-  // CHECK: store float {{.*}}, ptr
-  // CHECK: store i32 -1
-  // CHECK: store i32 0
-  // CHECK: ret i32 -1
-  res1 = a + b + c;
-  int val1 = __FLT_EVAL_METHOD__;
-  {
-#pragma float_control(precise, on)
-    int val2 = __FLT_EVAL_METHOD__;
-  }
-  return __FLT_EVAL_METHOD__;
-}
diff --git a/clang/test/CodeGen/exceptions-seh-finally.c b/clang/test/CodeGen/exceptions-seh-finally.c
index 6a09a04568f3d..8ef5ca807b1d3 100644
--- a/clang/test/CodeGen/exceptions-seh-finally.c
+++ b/clang/test/CodeGen/exceptions-seh-finally.c
@@ -1,6 +1,6 @@
-// RUN: %clang_cc1 -no-opaque-pointers %s -triple x86_64-pc-win32 -fms-extensions -emit-llvm -O1 -disable-llvm-passes -o - | FileCheck %s
-// RUN: %clang_cc1 -no-opaque-pointers %s -triple i686-pc-win32 -fms-extensions -emit-llvm -O1 -disable-llvm-passes -o - | FileCheck %s
-// RUN: %clang_cc1 -no-opaque-pointers %s -triple aarch64-windows -fms-extensions -emit-llvm -O1 -disable-llvm-passes -o - | FileCheck %s
+// RUN: %clang_cc1 %s -triple x86_64-pc-win32 -fms-extensions -emit-llvm -O1 -disable-llvm-passes -o - | FileCheck %s
+// RUN: %clang_cc1 %s -triple i686-pc-win32 -fms-extensions -emit-llvm -O1 -disable-llvm-passes -o - | FileCheck %s
+// RUN: %clang_cc1 %s -triple aarch64-windows -fms-extensions -emit-llvm -O1 -disable-llvm-passes -o - | FileCheck %s
 // NOTE: we're passing "-O1 -disable-llvm-passes" to avoid adding optnone and noinline everywhere.
 
 void abort(void) __attribute__((noreturn));
@@ -20,14 +20,14 @@ void basic_finally(void) {
 // CHECK:     to label %[[invoke_cont:[^ ]*]] unwind label %[[lpad:[^ ]*]]
 //
 // CHECK: [[invoke_cont]]
-// CHECK: %[[fp:[^ ]*]] = call i8* @llvm.localaddress()
-// CHECK: call void @"?fin$0@0@basic_finally@@"({{i8 noundef( zeroext)?}} 0, i8* noundef %[[fp]])
+// CHECK: %[[fp:[^ ]*]] = call ptr @llvm.localaddress()
+// CHECK: call void @"?fin$0@0@basic_finally@@"({{i8 noundef( zeroext)?}} 0, ptr noundef %[[fp]])
 // CHECK-NEXT: ret void
 //
 // CHECK: [[lpad]]
 // CHECK-NEXT: %[[pad:[^ ]*]] = cleanuppad
-// CHECK: %[[fp:[^ ]*]] = call i8* @llvm.localaddress()
-// CHECK: call void @"?fin$0@0@basic_finally@@"({{i8 noundef( zeroext)?}} 1, i8* noundef %[[fp]])
+// CHECK: %[[fp:[^ ]*]] = call ptr @llvm.localaddress()
+// CHECK: call void @"?fin$0@0@basic_finally@@"({{i8 noundef( zeroext)?}} 1, ptr noundef %[[fp]])
 // CHECK-NEXT: cleanupret from %[[pad]] unwind to caller
 
 // CHECK: define internal void @"?fin$0@0@basic_finally@@"({{.*}})
@@ -60,8 +60,8 @@ void label_in_finally(void) {
 // CHECK:     to label %[[invoke_cont:[^ ]*]] unwind label %[[lpad:[^ ]*]]
 //
 // CHECK: [[invoke_cont]]
-// CHECK: %[[fp:[^ ]*]] = call i8* @llvm.localaddress()
-// CHECK: call void @"?fin$0@0@label_in_finally@@"({{i8 noundef( zeroext)?}} 0, i8* noundef %[[fp]])
+// CHECK: %[[fp:[^ ]*]] = call ptr @llvm.localaddress()
+// CHECK: call void @"?fin$0@0@label_in_finally@@"({{i8 noundef( zeroext)?}} 0, ptr noundef %[[fp]])
 // CHECK: ret void
 
 // CHECK: define internal void @"?fin$0@0@label_in_finally@@"({{.*}})
@@ -88,20 +88,20 @@ void use_abnormal_termination(void) {
 // CHECK:     to label %[[invoke_cont:[^ ]*]] unwind label %[[lpad:[^ ]*]]
 //
 // CHECK: [[invoke_cont]]
-// CHECK: %[[fp:[^ ]*]] = call i8* @llvm.localaddress()
-// CHECK: call void @"?fin$0@0@use_abnormal_termination@@"({{i8 noundef( zeroext)?}} 0, i8* noundef %[[fp]])
+// CHECK: %[[fp:[^ ]*]] = call ptr @llvm.localaddress()
+// CHECK: call void @"?fin$0@0@use_abnormal_termination@@"({{i8 noundef( zeroext)?}} 0, ptr noundef %[[fp]])
 // CHECK: ret void
 //
 // CHECK: [[lpad]]
 // CHECK-NEXT: %[[pad:[^ ]*]] = cleanuppad
-// CHECK: %[[fp:[^ ]*]] = call i8* @llvm.localaddress()
-// CHECK: call void @"?fin$0@0@use_abnormal_termination@@"({{i8 noundef( zeroext)?}} 1, i8* noundef %[[fp]])
+// CHECK: %[[fp:[^ ]*]] = call ptr @llvm.localaddress()
+// CHECK: call void @"?fin$0@0@use_abnormal_termination@@"({{i8 noundef( zeroext)?}} 1, ptr noundef %[[fp]])
 // CHECK-NEXT: cleanupret from %[[pad]] unwind to caller
 
-// CHECK: define internal void @"?fin$0@0@use_abnormal_termination@@"({{i8 noundef( zeroext)?}} %[[abnormal:abnormal_termination]], i8* noundef %frame_pointer)
+// CHECK: define internal void @"?fin$0@0@use_abnormal_termination@@"({{i8 noundef( zeroext)?}} %[[abnormal:abnormal_termination]], ptr noundef %frame_pointer)
 // CHECK-SAME: [[finally_attrs]]
 // CHECK: %[[abnormal_zext:[^ ]*]] = zext i8 %[[abnormal]] to i32
-// CHECK: store i32 %[[abnormal_zext]], i32* @crashed
+// CHECK: store i32 %[[abnormal_zext]], ptr @crashed
 // CHECK-NEXT: ret void
 
 void noreturn_noop_finally(void) {
@@ -280,7 +280,7 @@ void finally_with_func(void) {
 }
 
 // CHECK-LABEL: define internal void @"?fin$0@0@finally_with_func@@"({{[^)]*}})
-// CHECK: call void @cleanup_with_func(i8* noundef getelementptr inbounds ([18 x i8], [18 x i8]* @"??_C@_0BC@COAGBPGM@finally_with_func?$AA@", i{{32|64}} 0, i{{32|64}} 0))
+// CHECK: call void @cleanup_with_func(ptr noundef @"??_C@_0BC@COAGBPGM@finally_with_func?$AA@")
 
 // Look for the absence of noinline.  nounwind is expected; any further
 // attributes should be string attributes.
diff --git a/clang/test/CodeGen/extend-arg-64.c b/clang/test/CodeGen/extend-arg-64.c
index 5f14811db9187..0749523b9ab3d 100644
--- a/clang/test/CodeGen/extend-arg-64.c
+++ b/clang/test/CodeGen/extend-arg-64.c
@@ -1,23 +1,23 @@
-// RUN: %clang_cc1 -no-opaque-pointers -DD128 -triple x86_64-apple-darwin -fextend-arguments=64  \
+// RUN: %clang_cc1 -DD128 -triple x86_64-apple-darwin -fextend-arguments=64  \
 // RUN:            -Wno-strict-prototypes %s -emit-llvm -o - | FileCheck %s -check-prefix=CHECKEXT
 
 // When the option isn't selected, no effect
-// RUN: %clang_cc1 -no-opaque-pointers -DD128 -triple x86_64-apple-darwin  \
+// RUN: %clang_cc1 -DD128 -triple x86_64-apple-darwin  \
 // RUN:            -Wno-strict-prototypes %s -emit-llvm -o - | FileCheck %s \
 // RUN:    --implicit-check-not "ext {{.*}}to i64"
 
 // The option isn't supported on x86, no effect
-// RUN: %clang_cc1 -no-opaque-pointers -triple i386-pc-linux-gnu -fextend-arguments=64 \
+// RUN: %clang_cc1 -triple i386-pc-linux-gnu -fextend-arguments=64 \
 // RUN:            -Wno-strict-prototypes %s -emit-llvm -o - | FileCheck %s \
 // RUN:    --implicit-check-not "ext {{.*}}to i64"
 
 // The option isn't supported on ppc, no effect
-// RUN: %clang_cc1 -no-opaque-pointers -triple ppc64le -fextend-arguments=64 \
+// RUN: %clang_cc1 -triple ppc64le -fextend-arguments=64 \
 // RUN:            -Wno-strict-prototypes %s -emit-llvm -o - | FileCheck %s \
 // RUN:    --implicit-check-not "ext {{.*}}to i64"
 
 // The option isn't supported on ppc, no effect
-// RUN: %clang_cc1 -no-opaque-pointers -DD128 -triple powerpc64-ibm-aix-xcoff -fextend-arguments=64 \
+// RUN: %clang_cc1 -DD128 -triple powerpc64-ibm-aix-xcoff -fextend-arguments=64 \
 // RUN:            -Wno-strict-prototypes %s -emit-llvm -o - | FileCheck %s \
 // RUN:    --implicit-check-not "ext {{.*}}to i64"
 
@@ -42,68 +42,68 @@ __int128 i128;
 int test(void) {
   // CHECK: define{{.*}} i32 @test{{.*}}
 
-  // CHECKEXT:  [[TAG_u32:%.*]] = load i32, i32* @u32{{.*}}
+  // CHECKEXT:  [[TAG_u32:%.*]] = load i32, ptr @u32{{.*}}
   // CHECKEXT: [[CONV_u32:%.*]] = zext i32 [[TAG_u32]] to i64
 
-  // CHECKEXT:  [[TAG_s32:%.*]] = load i32, i32* @s32
+  // CHECKEXT:  [[TAG_s32:%.*]] = load i32, ptr @s32
   // CHECKEXT: [[CONV_s32:%.*]] = sext i32 [[TAG_s32]] to i64
 
-  // CHECKEXT:  [[TAG_u16:%.*]] = load i16, i16* @u16
+  // CHECKEXT:  [[TAG_u16:%.*]] = load i16, ptr @u16
   // CHECKEXT: [[CONV_u16:%.*]] = zext i16 [[TAG_u16]] to i64
 
-  // CHECKEXT:  [[TAG_s16:%.*]] = load i16, i16* @s16
+  // CHECKEXT:  [[TAG_s16:%.*]] = load i16, ptr @s16
   // CHECKEXT: [[CONV_s16:%.*]] = sext i16 [[TAG_s16]] to i64
 
-  // CHECKEXT:  [[TAG_u8:%.*]] = load i8, i8* @u8
+  // CHECKEXT:  [[TAG_u8:%.*]] = load i8, ptr @u8
   // CHECKEXT: [[CONV_u8:%.*]] = zext i8 [[TAG_u8]] to i64
 
-  // CHECKEXT:  [[TAG_s8:%.*]] = load i8, i8* @s8
+  // CHECKEXT:  [[TAG_s8:%.*]] = load i8, ptr @s8
   // CHECKEXT: [[CONV_s8:%.*]] = sext i8 [[TAG_s8]] to i64
   // CHECKEXT: call{{.*}} @vararg(i32 noundef %0, i64 noundef [[CONV_u32]], i64 noundef [[CONV_s32]], i64 noundef [[CONV_u16]], i64 noundef [[CONV_s16]], i64 noundef [[CONV_u8]], i64 noundef [[CONV_s8]]
 
   int sum = 0;
   sum = vararg(sum, u32, s32, u16, s16, u8, s8);
   knr(ll);
-  // CHECKEXT: load i64, i64* @ll
-  // CHECKEXT-NEXT: call void (i64, ...) bitcast {{.*}} @knr
+  // CHECKEXT: load i64, ptr @ll
+  // CHECKEXT-NEXT: call void (i64, ...) @knr
 
   knr(ei23);
-  // CHECKEXT: load i23, i23* @ei23
-  // CHECKEXT-NEXT: call void (i23, ...) bitcast{{.*}} @knr
+  // CHECKEXT: load i23, ptr @ei23
+  // CHECKEXT-NEXT: call void (i23, ...) @knr
 
   knr(ff);
   // CHECKEXT: load float
   // CHECKEXT-NEXT: fpext float {{.*}} to double
-  // CHECKEXT-NEXT: call{{.*}} void (double, ...) bitcast{{.*}} @knr
+  // CHECKEXT-NEXT: call{{.*}} void (double, ...) @knr
 
   knr(dd);
   // CHECKEXT: load double
-  // CHECKEXT-NEXT: call{{.*}} void (double, ...) bitcast{{.*}} @knr
+  // CHECKEXT-NEXT: call{{.*}} void (double, ...) @knr
 
 #ifdef D128
   knr(i128);
   // CHECKEXT: load i128
-  // CHECKEXT: call{{.*}} void (i64, i64, ...) bitcast{{.*}} @knr
+  // CHECKEXT: call{{.*}} void (i64, i64, ...) @knr
 #endif
 
   knr(u32, s32, u16, s16, u8, s8);
-  // CHECKEXT:  [[TAg_u32:%.*]] = load i32, i32* @u32{{.*}}
+  // CHECKEXT:  [[TAg_u32:%.*]] = load i32, ptr @u32{{.*}}
   // CHECKEXT: [[CONv_u32:%.*]] = zext i32 [[TAg_u32]] to i64
 
-  // CHECKEXT:  [[TAg_s32:%.*]] = load i32, i32* @s32
+  // CHECKEXT:  [[TAg_s32:%.*]] = load i32, ptr @s32
   // CHECKEXT: [[CONv_s32:%.*]] = sext i32 [[TAg_s32]] to i64
 
-  // CHECKEXT:  [[TAg_u16:%.*]] = load i16, i16* @u16
+  // CHECKEXT:  [[TAg_u16:%.*]] = load i16, ptr @u16
   // CHECKEXT: [[CONv_u16:%.*]] = zext i16 [[TAg_u16]] to i64
 
-  // CHECKEXT:  [[TAg_s16:%.*]] = load i16, i16* @s16
+  // CHECKEXT:  [[TAg_s16:%.*]] = load i16, ptr @s16
   // CHECKEXT: [[CONv_s16:%.*]] = sext i16 [[TAg_s16]] to i64
 
-  // CHECKEXT:  [[TAg_u8:%.*]] = load i8, i8* @u8
+  // CHECKEXT:  [[TAg_u8:%.*]] = load i8, ptr @u8
   // CHECKEXT: [[CONv_u8:%.*]] = zext i8 [[TAg_u8]] to i64
 
-  // CHECKEXT:  [[TAg_s8:%.*]] = load i8, i8* @s8
+  // CHECKEXT:  [[TAg_s8:%.*]] = load i8, ptr @s8
   // CHECKEXT: [[CONv_s8:%.*]] = sext i8 [[TAg_s8]] to i64
-  // CHECKEXT: call{{.*}} void (i64, i64, i64, i64, i64, i64, ...) bitcast{{.*}} @knr
+  // CHECKEXT: call{{.*}} void (i64, i64, i64, i64, i64, i64, ...) @knr
   return sum;
 }
diff --git a/clang/test/CodeGen/fp-function-attrs.cpp b/clang/test/CodeGen/fp-function-attrs.cpp
index 881d5ae1a2de3..e92c26cdb75f7 100644
--- a/clang/test/CodeGen/fp-function-attrs.cpp
+++ b/clang/test/CodeGen/fp-function-attrs.cpp
@@ -8,7 +8,7 @@ float test_default(float a, float b, float c) {
   return tmp;
 }
 
-// CHECK: define{{.*}} float @_Z12test_defaultfff(float noundef %a, float noundef %b, float noundef %c) [[FAST_ATTRS:#[0-9]+]]
+// CHECK: define{{.*}} float @_Z12test_defaultfff(float noundef nofpclass(nan inf) %a, float noundef nofpclass(nan inf) %b, float noundef nofpclass(nan inf) %c) [[FAST_ATTRS:#[0-9]+]]
 // CHECK: fadd fast float {{%.+}}, {{%.+}}
 // CHECK: fadd fast float {{%.+}}, {{%.+}}
 
@@ -22,7 +22,7 @@ float test_precise_on_pragma(float a, float b, float c) {
   return tmp;
 }
 
-// CHECK: define{{.*}} float @_Z22test_precise_on_pragmafff(float noundef %a, float noundef %b, float noundef %c) [[PRECISE_ATTRS:#[0-9]+]]
+// CHECK: define{{.*}} float @_Z22test_precise_on_pragmafff(float noundef nofpclass(nan inf) %a, float noundef nofpclass(nan inf) %b, float noundef nofpclass(nan inf) %c) [[PRECISE_ATTRS:#[0-9]+]]
 // CHECK: fadd float {{%.+}}, {{%.+}}
 // CHECK: fadd fast float {{%.+}}, {{%.+}}
 
@@ -36,7 +36,7 @@ float test_reassociate_off_pragma(float a, float b, float c) {
   return tmp;
 }
 
-// CHECK: define{{.*}} float @_Z27test_reassociate_off_pragmafff(float noundef %a, float noundef %b, float noundef %c) [[NO_UNSAFE_ATTRS:#[0-9]+]]
+// CHECK: define{{.*}} float @_Z27test_reassociate_off_pragmafff(float noundef nofpclass(nan inf) %a, float noundef nofpclass(nan inf) %b, float noundef nofpclass(nan inf) %c) [[NO_UNSAFE_ATTRS:#[0-9]+]]
 // CHECK: fadd nnan ninf nsz arcp contract afn float {{%.+}}, {{%.+}}
 // CHECK: fadd fast float {{%.+}}, {{%.+}}
 
@@ -49,7 +49,7 @@ float test_contract_on_pragma(float a, float b, float c) {
   return tmp;
 }
 
-// CHECK: define{{.*}} float @_Z23test_contract_on_pragmafff(float noundef %a, float noundef %b, float noundef %c) [[NO_UNSAFE_ATTRS:#[0-9]+]]
+// CHECK: define{{.*}} float @_Z23test_contract_on_pragmafff(float noundef nofpclass(nan inf) %a, float noundef nofpclass(nan inf) %b, float noundef nofpclass(nan inf) %c) [[NO_UNSAFE_ATTRS:#[0-9]+]]
 // CHECK: fmul fast float {{%.+}}, {{%.+}}
 // CHECK: fadd reassoc nnan ninf nsz arcp afn float {{%.+}}, {{%.+}}
 
diff --git a/clang/test/CodeGen/fp-options-to-fast-math-flags.c b/clang/test/CodeGen/fp-options-to-fast-math-flags.c
index d7fcc586fed4b..abdcf8541f225 100644
--- a/clang/test/CodeGen/fp-options-to-fast-math-flags.c
+++ b/clang/test/CodeGen/fp-options-to-fast-math-flags.c
@@ -18,13 +18,13 @@ float test(float a) {
 // CHECK-PRECISE: [[CALL_RES:%.+]] = call float @fn(float noundef {{%.+}})
 // CHECK-PRECISE: {{%.+}} = fadd float {{%.+}}, [[CALL_RES]]
 
-// CHECK-NO-NANS: [[CALL_RES:%.+]] = call nnan float @fn(float noundef {{%.+}})
+// CHECK-NO-NANS: [[CALL_RES:%.+]] = call nnan nofpclass(nan) float @fn(float noundef nofpclass(nan) {{%.+}})
 // CHECK-NO-NANS: {{%.+}} = fadd nnan float {{%.+}}, [[CALL_RES]]
 
-// CHECK-NO-INFS: [[CALL_RES:%.+]] = call ninf float @fn(float noundef {{%.+}})
+// CHECK-NO-INFS: [[CALL_RES:%.+]] = call ninf nofpclass(inf) float @fn(float noundef nofpclass(inf) {{%.+}})
 // CHECK-NO-INFS: {{%.+}} = fadd ninf float {{%.+}}, [[CALL_RES]]
 
-// CHECK-FINITE: [[CALL_RES:%.+]] = call nnan ninf float @fn(float noundef {{%.+}})
+// CHECK-FINITE: [[CALL_RES:%.+]] = call nnan ninf nofpclass(nan inf) float @fn(float noundef nofpclass(nan inf) {{%.+}})
 // CHECK-FINITE: {{%.+}} = fadd nnan ninf float {{%.+}}, [[CALL_RES]]
 
 // CHECK-NO-SIGNED-ZEROS: [[CALL_RES:%.+]] = call nsz float @fn(float noundef {{%.+}})
@@ -42,5 +42,5 @@ float test(float a) {
 // CHECK-UNSAFE: [[CALL_RES:%.+]] = call reassoc nsz arcp afn float @fn(float noundef {{%.+}})
 // CHECK-UNSAFE: {{%.+}} = fadd reassoc nsz arcp afn float {{%.+}}, [[CALL_RES]]
 
-// CHECK-FAST: [[CALL_RES:%.+]] = call reassoc nnan ninf nsz arcp afn float @fn(float noundef {{%.+}})
+// CHECK-FAST: [[CALL_RES:%.+]] = call reassoc nnan ninf nsz arcp afn nofpclass(nan inf) float @fn(float noundef nofpclass(nan inf) {{%.+}})
 // CHECK-FAST: {{%.+}} = fadd reassoc nnan ninf nsz arcp afn float {{%.+}}, [[CALL_RES]]
diff --git a/clang/test/CodeGen/func-attr.c b/clang/test/CodeGen/func-attr.c
index 360cb1282c00a..96c3d91a7a67a 100644
--- a/clang/test/CodeGen/func-attr.c
+++ b/clang/test/CodeGen/func-attr.c
@@ -1,24 +1,26 @@
 // RUN: %clang_cc1 -triple x86_64-linux-gnu -ffast-math \
 // RUN: -ffp-contract=fast -emit-llvm -o - %s | \
-// RUN: FileCheck %s --check-prefixes=CHECK,CHECK-UNSAFE
+// RUN: FileCheck %s --check-prefixes=CHECK,CHECK-UNSAFE,FINITEONLY
 
 // RUN: %clang_cc1 -triple x86_64-linux-gnu -funsafe-math-optimizations \
 // RUN: -ffp-contract=fast -emit-llvm -o - %s | \
-// RUN: FileCheck %s --check-prefixes=CHECK,CHECK-UNSAFE
+// RUN: FileCheck %s --check-prefixes=CHECK,CHECK-UNSAFE,NOFINITEONLY
 
 // RUN: %clang_cc1 -triple x86_64-linux-gnu -funsafe-math-optimizations \
 // RUN: -ffp-contract=on -emit-llvm -o - %s | \
-// RUN: FileCheck %s --check-prefixes=CHECK,CHECK-NOUNSAFE
+// RUN: FileCheck %s --check-prefixes=CHECK,CHECK-NOUNSAFE,NOFINITEONLY
 
 // RUN: %clang_cc1 -triple x86_64-linux-gnu -funsafe-math-optimizations \
 // RUN: -ffp-contract=off -emit-llvm -o - %s | \
-// RUN: FileCheck %s --check-prefixes=CHECK,CHECK-NOUNSAFE
+// RUN: FileCheck %s --check-prefixes=CHECK,CHECK-NOUNSAFE,NOFINITEONLY
 
 float foo(float a, float b) {
   return a+b;
 }
 
-// CHECK:              define{{.*}} float @foo(float noundef %{{.*}}, float noundef %{{.*}}){{.*}} [[ATTRS:#[0-9]+]]
+// FINITEONLY:    define{{.*}} float @foo(float noundef nofpclass(nan inf) %{{.*}}, float noundef nofpclass(nan inf) %{{.*}}){{.*}} [[ATTRS:#[0-9]+]]
+// NOFINITEONLY:    define{{.*}} float @foo(float noundef %{{.*}}, float noundef %{{.*}}){{.*}} [[ATTRS:#[0-9]+]]
+
 // CHECK:              attributes [[ATTRS]] = {
 // CHECK-SAME:           "approx-func-fp-math"="true"
 // CHECK-SAME:           "no-signed-zeros-fp-math"="true"
diff --git a/clang/test/CodeGen/hexagon-linux-vararg.c b/clang/test/CodeGen/hexagon-linux-vararg.c
index ffa949deb6c7b..033e72ab449d3 100644
--- a/clang/test/CodeGen/hexagon-linux-vararg.c
+++ b/clang/test/CodeGen/hexagon-linux-vararg.c
@@ -1,5 +1,5 @@
 // REQUIRES: hexagon-registered-target
-// RUN: %clang_cc1 -no-opaque-pointers -emit-llvm -triple hexagon-unknown-linux-musl %s -o - | FileCheck %s
+// RUN: %clang_cc1 -emit-llvm -triple hexagon-unknown-linux-musl %s -o - | FileCheck %s
 #include <stdarg.h>
 
 struct AAA {
@@ -9,52 +9,50 @@ struct AAA {
   int d;
 };
 
-// CHECK:   call void @llvm.va_start(i8* %arraydecay1)
-// CHECK:   %arraydecay2 = getelementptr inbounds [1 x %struct.__va_list_tag],
-// [1 x %struct.__va_list_tag]* %ap, i32 0, i32 0
+// CHECK:   call void @llvm.va_start(ptr %arraydecay)
+// CHECK:   %arraydecay1 = getelementptr inbounds [1 x %struct.__va_list_tag],
+// ptr %ap, i32 0, i32 0
 // CHECK:   br label %vaarg.maybe_reg
 
 // CHECK: vaarg.maybe_reg:                                  ; preds = %entry
 // CHECK:   %__current_saved_reg_area_pointer_p = getelementptr inbounds
-// %struct.__va_list_tag, %struct.__va_list_tag* %arraydecay2, i32 0, i32 0
-// CHECK:   %__current_saved_reg_area_pointer = load i8*, i8**
+// %struct.__va_list_tag, ptr %arraydecay2, i32 0, i32 0
+// CHECK:   %__current_saved_reg_area_pointer = load ptr, ptr
 // %__current_saved_reg_area_pointer_p
 // CHECK:   %__saved_reg_area_end_pointer_p = getelementptr inbounds
-// %struct.__va_list_tag, %struct.__va_list_tag* %arraydecay2, i32 0, i32 1
-// CHECK:   %__saved_reg_area_end_pointer = load i8*, i8**
+// %struct.__va_list_tag, ptr %arraydecay2, i32 0, i32 1
+// CHECK:   %__saved_reg_area_end_pointer = load ptr, ptr
 // %__saved_reg_area_end_pointer_p
-// CHECK:   %__new_saved_reg_area_pointer = getelementptr i8, i8*
+// CHECK:   %__new_saved_reg_area_pointer = getelementptr i8, ptr
 // %__current_saved_reg_area_pointer, i32 4
-// CHECK:   %0 = icmp sgt i8* %__new_saved_reg_area_pointer,
+// CHECK:   %0 = icmp sgt ptr %__new_saved_reg_area_pointer,
 // %__saved_reg_area_end_pointer
 // CHECK:   br i1 %0, label %vaarg.on_stack, label %vaarg.in_reg
 
 // CHECK: vaarg.in_reg:                                     ; preds =
 // %vaarg.maybe_reg
-// CHECK:   %1 = bitcast i8* %__current_saved_reg_area_pointer to i32*
-// CHECK:   store i8* %__new_saved_reg_area_pointer, i8**
+// CHECK:   store ptr %__new_saved_reg_area_pointer, ptr
 // %__current_saved_reg_area_pointer_p
 // CHECK:   br label %vaarg.end
 
 // CHECK: vaarg.on_stack:                                   ; preds =
 // %vaarg.maybe_reg
 // CHECK:   %__overflow_area_pointer_p = getelementptr inbounds
-// %struct.__va_list_tag, %struct.__va_list_tag* %arraydecay2, i32 0, i32 2
-// CHECK:   %__overflow_area_pointer = load i8*, i8** %__overflow_area_pointer_p
-// CHECK:   %__overflow_area_pointer.next = getelementptr i8, i8*
+// %struct.__va_list_tag, ptr %arraydecay2, i32 0, i32 2
+// CHECK:   %__overflow_area_pointer = load ptr, ptr %__overflow_area_pointer_p
+// CHECK:   %__overflow_area_pointer.next = getelementptr i8, ptr
 // %__overflow_area_pointer, i32 4
-// CHECK:   store i8* %__overflow_area_pointer.next, i8**
+// CHECK:   store ptr %__overflow_area_pointer.next, ptr
 // %__overflow_area_pointer_p
-// CHECK:   store i8* %__overflow_area_pointer.next, i8**
+// CHECK:   store ptr %__overflow_area_pointer.next, ptr
 // %__current_saved_reg_area_pointer_p
-// CHECK:   %2 = bitcast i8* %__overflow_area_pointer to i32*
 // CHECK:   br label %vaarg.end
 
 // CHECK: vaarg.end:                                        ; preds =
 // %vaarg.on_stack, %vaarg.in_reg
-// CHECK:   %vaarg.addr = phi i32* [ %1, %vaarg.in_reg ], [ %2, %vaarg.on_stack
+// CHECK:   %vaarg.addr = phi ptr [ %__current_saved_reg_area_pointer, %vaarg.in_reg ], [ %__overflow_area_pointer, %vaarg.on_stack
 // ]
-// CHECK:   %3 = load i32, i32* %vaarg.addr
+// CHECK:   %1 = load i32, ptr %vaarg.addr
 
 struct AAA aaa = {100, 200, 300, 400};
 
diff --git a/clang/test/CodeGen/ifunc.c b/clang/test/CodeGen/ifunc.c
index 895141914db65..64f7f3d4ec65c 100644
--- a/clang/test/CodeGen/ifunc.c
+++ b/clang/test/CodeGen/ifunc.c
@@ -1,5 +1,5 @@
-// RUN: %clang_cc1 -no-opaque-pointers -triple i386-unknown-linux-gnu -emit-llvm -o - %s | FileCheck %s
-// RUN: %clang_cc1 -no-opaque-pointers -triple i386-unknown-linux-gnu -O2 -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple i386-unknown-linux-gnu -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple i386-unknown-linux-gnu -O2 -emit-llvm -o - %s | FileCheck %s
 
 int foo(int) __attribute__ ((ifunc("foo_ifunc")));
 
@@ -34,8 +34,8 @@ extern void goo(void) __attribute__ ((ifunc("goo_ifunc")));
 void* goo_ifunc(void) {
   return 0;
 }
-// CHECK: @foo = ifunc i32 (i32), i32 (i32)* ()* @foo_ifunc
-// CHECK: @goo = ifunc void (), bitcast (i8* ()* @goo_ifunc to void ()* ()*)
+// CHECK: @foo = ifunc i32 (i32), ptr @foo_ifunc
+// CHECK: @goo = ifunc void (), ptr @goo_ifunc
 
 // CHECK: call i32 @foo(i32
 // CHECK: call void @goo()
diff --git a/clang/test/CodeGen/incomplete-function-type-2.c b/clang/test/CodeGen/incomplete-function-type-2.c
index 24e2c83d085d9..4abed7eb0ab3f 100644
--- a/clang/test/CodeGen/incomplete-function-type-2.c
+++ b/clang/test/CodeGen/incomplete-function-type-2.c
@@ -1,8 +1,8 @@
-// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-apple-darwin -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple x86_64-apple-darwin -emit-llvm -o - %s | FileCheck %s
 
 // PR14355: don't crash
 // Keep this test in its own file because CodeGenTypes has global state.
-// CHECK: define{{.*}} void @test10_foo({}* noundef %p1.coerce) [[NUW:#[0-9]+]] {
+// CHECK: define{{.*}} void @test10_foo(ptr noundef %p1) [[NUW:#[0-9]+]] {
 struct test10_B;
 typedef struct test10_B test10_F3(double);
 void test10_foo(test10_F3 p1);
diff --git a/clang/test/CodeGen/init-memset.c b/clang/test/CodeGen/init-memset.c
index e3e041a449592..e5fd86a74a546 100644
--- a/clang/test/CodeGen/init-memset.c
+++ b/clang/test/CodeGen/init-memset.c
@@ -1,4 +1,4 @@
-// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-unknown-unknown -O0 -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple x86_64-unknown-unknown -O0 -emit-llvm -o - %s | FileCheck %s
 
 void use(void *);
 
@@ -55,12 +55,6 @@ void test_most_a(void) {
 void test_pointers(void) {
   // CHECK-LABEL: define{{.*}} void @test_pointers()
   void *a[] = {&use, &use, &use, &use, &use, &use};
-  // CHECK: call void @llvm.memset.{{.*}}
-  // CHECK: store i8*
-  // CHECK: store i8*
-  // CHECK: store i8*
-  // CHECK: store i8*
-  // CHECK: store i8*
-  // CHECK: store i8*
+  // CHECK: call void @llvm.memcpy.{{.*}}
   use(a);
 }
diff --git a/clang/test/CodeGen/init.c b/clang/test/CodeGen/init.c
index 74e769a40af20..5ebd6853733f8 100644
--- a/clang/test/CodeGen/init.c
+++ b/clang/test/CodeGen/init.c
@@ -1,4 +1,4 @@
-// RUN: %clang_cc1 -no-opaque-pointers -triple i386-unknown-unknown -emit-llvm %s -o - | FileCheck %s
+// RUN: %clang_cc1 -triple i386-unknown-unknown -emit-llvm %s -o - | FileCheck %s
 
 struct I { int k[3]; };
 struct M { struct I o[2]; };
@@ -10,7 +10,7 @@ unsigned v2[2][3] = {[0 ... 1][0 ... 1] = 2222, 3333};
 
 // CHECK-DAG: [1 x %struct.M] [%struct.M { [2 x %struct.I] [%struct.I { [3 x i32] [i32 4, i32 4, i32 0] }, %struct.I { [3 x i32] [i32 4, i32 4, i32 5] }] }],
 // CHECK-DAG: [2 x [3 x i32]] {{[[][[]}}3 x i32] [i32 2222, i32 2222, i32 0], [3 x i32] [i32 2222, i32 2222, i32 3333]],
-// CHECK-DAG: [[INIT14:.*]] = private global [16 x i32] [i32 0, i32 0, i32 0, i32 0, i32 0, i32 17, i32 17, i32 17, i32 17, i32 17, i32 17, i32 17, i32 0, i32 0, i32 0, i32 0], align 4
+// CHECK-DAG: [[INIT14:.*]] = private constant [16 x i32] [i32 0, i32 0, i32 0, i32 0, i32 0, i32 17, i32 17, i32 17, i32 17, i32 17, i32 17, i32 17, i32 0, i32 0, i32 0, i32 0], align 4
 
 void f1(void) {
   // Scalars in braces.
@@ -88,11 +88,11 @@ const char large_array_with_zeroes[1000] = {
 
 char global;
 
-// CHECK-DAG: @large_array_with_zeroes_2 ={{.*}} global <{ [10 x i8*], [90 x i8*] }> <{ [10 x i8*] [i8* null, i8* null, i8* null, i8* null, i8* null, i8* null, i8* null, i8* null, i8* null, i8* @global], [90 x i8*] zeroinitializer }>
+// CHECK-DAG: @large_array_with_zeroes_2 ={{.*}} global <{ [10 x ptr], [90 x ptr] }> <{ [10 x ptr] [ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr @global], [90 x ptr] zeroinitializer }>
 const void *large_array_with_zeroes_2[100] = {
   [9] = &global
 };
-// CHECK-DAG: @large_array_with_zeroes_3 ={{.*}} global <{ [10 x i8*], [990 x i8*] }> <{ [10 x i8*] [i8* null, i8* null, i8* null, i8* null, i8* null, i8* null, i8* null, i8* null, i8* null, i8* @global], [990 x i8*] zeroinitializer }>
+// CHECK-DAG: @large_array_with_zeroes_3 ={{.*}} global <{ [10 x ptr], [990 x ptr] }> <{ [10 x ptr] [ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr @global], [990 x ptr] zeroinitializer }>
 const void *large_array_with_zeroes_3[1000] = {
   [9] = &global
 };
@@ -103,9 +103,9 @@ char test8(int X) {
   return str[X];
   // CHECK-LABEL: @test8(
   // CHECK: call void @llvm.memset
-  // CHECK: store i8 97, i8* %{{[0-9]*}}, align 1
-  // CHECK: store i8 98, i8* %{{[0-9]*}}, align 1
-  // CHECK: store i8 99, i8* %{{[0-9]*}}, align 1
+  // CHECK: store i8 97, ptr %{{[0-9]*}}, align 1
+  // CHECK: store i8 98, ptr %{{[0-9]*}}, align 1
+  // CHECK: store i8 99, ptr %{{[0-9]*}}, align 1
   // CHECK-NOT: getelementptr
   // CHECK: load
 }
@@ -145,7 +145,7 @@ void nonzeroMemseti8(void) {
   // CHECK-LABEL: @nonzeroMemseti8(
   // CHECK-NOT: store
   // CHECK-NOT: memcpy
-  // CHECK: call void @llvm.memset.p0i8.i32(i8* {{.*}}, i8 42, i32 33, i1 false)
+  // CHECK: call void @llvm.memset.p0.i32(ptr {{.*}}, i8 42, i32 33, i1 false)
 }
 
 void nonzeroMemseti16(void) {
@@ -153,7 +153,7 @@ void nonzeroMemseti16(void) {
   // CHECK-LABEL: @nonzeroMemseti16(
   // CHECK-NOT: store
   // CHECK-NOT: memcpy
-  // CHECK: call void @llvm.memset.p0i8.i32(i8* {{.*}}, i8 66, i32 34, i1 false)
+  // CHECK: call void @llvm.memset.p0.i32(ptr {{.*}}, i8 66, i32 34, i1 false)
 }
 
 void nonzeroMemseti32(void) {
@@ -161,7 +161,7 @@ void nonzeroMemseti32(void) {
   // CHECK-LABEL: @nonzeroMemseti32(
   // CHECK-NOT: store
   // CHECK-NOT: memcpy
-  // CHECK: call void @llvm.memset.p0i8.i32(i8* {{.*}}, i8 -16, i32 36, i1 false)
+  // CHECK: call void @llvm.memset.p0.i32(ptr {{.*}}, i8 -16, i32 36, i1 false)
 }
 
 void nonzeroMemseti64(void) {
@@ -169,7 +169,7 @@ void nonzeroMemseti64(void) {
   // CHECK-LABEL: @nonzeroMemseti64(
   // CHECK-NOT: store
   // CHECK-NOT: memcpy
-  // CHECK: call void @llvm.memset.p0i8.i32(i8* {{.*}}, i8 -86, i32 56, i1 false)
+  // CHECK: call void @llvm.memset.p0.i32(ptr {{.*}}, i8 -86, i32 56, i1 false)
 }
 
 void nonzeroMemsetf32(void) {
@@ -177,7 +177,7 @@ void nonzeroMemsetf32(void) {
   // CHECK-LABEL: @nonzeroMemsetf32(
   // CHECK-NOT: store
   // CHECK-NOT: memcpy
-  // CHECK: call void @llvm.memset.p0i8.i32(i8* {{.*}}, i8 101, i32 36, i1 false)
+  // CHECK: call void @llvm.memset.p0.i32(ptr {{.*}}, i8 101, i32 36, i1 false)
 }
 
 void nonzeroMemsetf64(void) {
@@ -185,7 +185,7 @@ void nonzeroMemsetf64(void) {
   // CHECK-LABEL: @nonzeroMemsetf64(
   // CHECK-NOT: store
   // CHECK-NOT: memcpy
-  // CHECK: call void @llvm.memset.p0i8.i32(i8* {{.*}}, i8 68, i32 56, i1 false)
+  // CHECK: call void @llvm.memset.p0.i32(ptr {{.*}}, i8 68, i32 56, i1 false)
 }
 
 void nonzeroPaddedUnionMemset(void) {
@@ -194,7 +194,7 @@ void nonzeroPaddedUnionMemset(void) {
   // CHECK-LABEL: @nonzeroPaddedUnionMemset(
   // CHECK-NOT: store
   // CHECK-NOT: memcpy
-  // CHECK: call void @llvm.memset.p0i8.i32(i8* {{.*}}, i8 -16, i32 36, i1 false)
+  // CHECK: call void @llvm.memset.p0.i32(ptr {{.*}}, i8 -16, i32 36, i1 false)
 }
 
 void nonzeroNestedMemset(void) {
@@ -204,7 +204,7 @@ void nonzeroNestedMemset(void) {
   // CHECK-LABEL: @nonzeroNestedMemset(
   // CHECK-NOT: store
   // CHECK-NOT: memcpy
-  // CHECK: call void @llvm.memset.p0i8.i32(i8* {{.*}}, i8 -16, i32 40, i1 false)
+  // CHECK: call void @llvm.memset.p0.i32(ptr {{.*}}, i8 -16, i32 40, i1 false)
 }
 
 // PR9257
@@ -214,10 +214,10 @@ struct test11S {
 void test11(struct test11S *P) {
   *P = (struct test11S) { .A = { [0 ... 3] = 4 } };
   // CHECK-LABEL: @test11(
-  // CHECK: store i32 4, i32* %{{.*}}, align 4
-  // CHECK: store i32 4, i32* %{{.*}}, align 4
-  // CHECK: store i32 4, i32* %{{.*}}, align 4
-  // CHECK: store i32 4, i32* %{{.*}}, align 4
+  // CHECK: store i32 4, ptr %{{.*}}, align 4
+  // CHECK: store i32 4, ptr %{{.*}}, align 4
+  // CHECK: store i32 4, ptr %{{.*}}, align 4
+  // CHECK: store i32 4, ptr %{{.*}}, align 4
   // CHECK: ret void
 }
 
@@ -238,9 +238,9 @@ void test13(int x) {
 
 // CHECK-LABEL: @PR20473(
 void PR20473(void) {
-  // CHECK: memcpy{{.*}}getelementptr inbounds ([2 x i8], [2 x i8]* @
+  // CHECK: memcpy{{.*}}
   bar((char[2]) {""});
-  // CHECK: memcpy{{.*}}getelementptr inbounds ([3 x i8], [3 x i8]* @
+  // CHECK: memcpy{{.*}}
   bar((char[3]) {""});
 }
 
@@ -250,7 +250,7 @@ struct S14 { int a[16]; };
 
 void test14(struct S14 *s14) {
   // CHECK-LABEL: @test14(
-  // CHECK: call void @llvm.memcpy.p0i8.p0i8.i32(i8* align 4 {{.*}}, i8* align 4 {{.*}} [[INIT14]] {{.*}}, i32 64, i1 false)
+  // CHECK: call void @llvm.memcpy.p0.p0.i32(ptr align 4 {{.*}}, ptr align 4 [[INIT14]], i32 64, i1 false)
   // CHECK-NOT: store
   // CHECK: ret void
   *s14 = (struct S14) { { [5 ... 11] = 17 } };
diff --git a/clang/test/CodeGen/label-array-aggregate-init.c b/clang/test/CodeGen/label-array-aggregate-init.c
index 00a20d5d204aa..4039b925f21b7 100644
--- a/clang/test/CodeGen/label-array-aggregate-init.c
+++ b/clang/test/CodeGen/label-array-aggregate-init.c
@@ -1,6 +1,6 @@
 // RUN: %clang_cc1 -triple x86_64-windows-msvc -emit-llvm %s -o - | FileCheck %s
 
-// CHECK: @constinit = private global [3 x ptr] [ptr blockaddress(@main, %L), ptr null, ptr null]
+// CHECK: @constinit = private constant [3 x ptr] [ptr blockaddress(@main, %L), ptr null, ptr null]
 
 void receivePtrs(void **);
 
diff --git a/clang/test/CodeGen/lto-newpm-pipeline.c b/clang/test/CodeGen/lto-newpm-pipeline.c
index dfeb7bc96411d..1aaa7d46f3ff0 100644
--- a/clang/test/CodeGen/lto-newpm-pipeline.c
+++ b/clang/test/CodeGen/lto-newpm-pipeline.c
@@ -1,28 +1,28 @@
 // REQUIRES: x86-registered-target
 
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-cfg-preserved=0 -fdebug-pass-manager -flto=full -O0 %s 2>&1 | FileCheck %s \
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-analysis-invalidation=0 -fdebug-pass-manager -flto=full -O0 %s 2>&1 | FileCheck %s \
 // RUN:   -check-prefix=CHECK-FULL-O0
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-cfg-preserved=0 -fdebug-pass-manager -flto=thin -O0 %s 2>&1 | FileCheck %s \
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-analysis-invalidation=0 -fdebug-pass-manager -flto=thin -O0 %s 2>&1 | FileCheck %s \
 // RUN:   -check-prefix=CHECK-THIN-O0
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-cfg-preserved=0 -fdebug-pass-manager -flto=full -O1 %s 2>&1 | FileCheck %s \
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-analysis-invalidation=0 -fdebug-pass-manager -flto=full -O1 %s 2>&1 | FileCheck %s \
 // RUN:   -check-prefix=CHECK-FULL-OPTIMIZED
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-cfg-preserved=0 -fdebug-pass-manager -flto=thin -O1 %s 2>&1 | FileCheck %s \
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-analysis-invalidation=0 -fdebug-pass-manager -flto=thin -O1 %s 2>&1 | FileCheck %s \
 // RUN:   -check-prefix=CHECK-THIN-OPTIMIZED
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-cfg-preserved=0 -fdebug-pass-manager -flto=full -O2 %s 2>&1 | FileCheck %s \
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-analysis-invalidation=0 -fdebug-pass-manager -flto=full -O2 %s 2>&1 | FileCheck %s \
 // RUN:   -check-prefix=CHECK-FULL-OPTIMIZED
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-cfg-preserved=0 -fdebug-pass-manager -flto=thin -O2 %s 2>&1 | FileCheck %s \
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-analysis-invalidation=0 -fdebug-pass-manager -flto=thin -O2 %s 2>&1 | FileCheck %s \
 // RUN:   -check-prefix=CHECK-THIN-OPTIMIZED
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-cfg-preserved=0 -fdebug-pass-manager -flto=full -O3 %s 2>&1 | FileCheck %s \
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-analysis-invalidation=0 -fdebug-pass-manager -flto=full -O3 %s 2>&1 | FileCheck %s \
 // RUN:   -check-prefix=CHECK-FULL-OPTIMIZED
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-cfg-preserved=0 -fdebug-pass-manager -flto=thin -O3 %s 2>&1 | FileCheck %s \
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-analysis-invalidation=0 -fdebug-pass-manager -flto=thin -O3 %s 2>&1 | FileCheck %s \
 // RUN:   -check-prefix=CHECK-THIN-OPTIMIZED
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-cfg-preserved=0 -fdebug-pass-manager -flto=full -Os %s 2>&1 | FileCheck %s \
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-analysis-invalidation=0 -fdebug-pass-manager -flto=full -Os %s 2>&1 | FileCheck %s \
 // RUN:   -check-prefix=CHECK-FULL-OPTIMIZED
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-cfg-preserved=0 -fdebug-pass-manager -flto=thin -Os %s 2>&1 | FileCheck %s \
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-analysis-invalidation=0 -fdebug-pass-manager -flto=thin -Os %s 2>&1 | FileCheck %s \
 // RUN:   -check-prefix=CHECK-THIN-OPTIMIZED
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-cfg-preserved=0 -fdebug-pass-manager -flto=full -Oz %s 2>&1 | FileCheck %s \
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-analysis-invalidation=0 -fdebug-pass-manager -flto=full -Oz %s 2>&1 | FileCheck %s \
 // RUN:   -check-prefix=CHECK-FULL-OPTIMIZED
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-cfg-preserved=0 -fdebug-pass-manager -flto=thin -Oz %s 2>&1 | FileCheck %s \
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -mllvm -verify-analysis-invalidation=0 -fdebug-pass-manager -flto=thin -Oz %s 2>&1 | FileCheck %s \
 // RUN:   -check-prefix=CHECK-THIN-OPTIMIZED
 
 // CHECK-FULL-O0: Running pass: AlwaysInlinerPass
diff --git a/clang/test/CodeGen/mangle-blocks.c b/clang/test/CodeGen/mangle-blocks.c
index 2e671dc880325..c87ee3a8ea9d0 100644
--- a/clang/test/CodeGen/mangle-blocks.c
+++ b/clang/test/CodeGen/mangle-blocks.c
@@ -1,4 +1,4 @@
-// RUN: %clang_cc1 -no-opaque-pointers -triple i386-apple-ios -fblocks -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple i386-apple-ios -fblocks -emit-llvm -o - %s | FileCheck %s
 
 void __assert_rtn(const char *, const char *, int, const char *)
     __attribute__ (( noreturn ));
@@ -15,9 +15,9 @@ void (^mangle(void))(void) {
 // CHECK: @.str{{.*}} = private unnamed_addr constant {{.*}}, align 1
 // CHECK: @.str[[STR1:.*]] = private unnamed_addr constant [7 x i8] c"mangle\00", align 1
 
-// CHECK: define internal void @__mangle_block_invoke(i8* noundef %.block_descriptor)
+// CHECK: define internal void @__mangle_block_invoke(ptr noundef %.block_descriptor)
 
-// CHECK: define internal void @__mangle_block_invoke_2(i8* noundef %.block_descriptor){{.*}}{
-// CHECK:   call void @__assert_rtn(i8* noundef getelementptr inbounds ([22 x i8], [22 x i8]* @__func__.__mangle_block_invoke_2, i32 0, i32 0), i8* noundef getelementptr inbounds {{.*}}, i32 noundef 9, i8* noundef getelementptr inbounds ([7 x i8], [7 x i8]* @.str[[STR1]], i32 0, i32 0))
+// CHECK: define internal void @__mangle_block_invoke_2(ptr noundef %.block_descriptor){{.*}}{
+// CHECK:   call void @__assert_rtn(ptr noundef @__func__.__mangle_block_invoke_2, ptr noundef @{{.*}}, i32 noundef 9, ptr noundef @.str[[STR1]])
 // CHECK: }
 
diff --git a/clang/test/CodeGen/matrix-cast.c b/clang/test/CodeGen/matrix-cast.c
index ec0a3f435c9b7..80990b40ea2d0 100644
--- a/clang/test/CodeGen/matrix-cast.c
+++ b/clang/test/CodeGen/matrix-cast.c
@@ -1,4 +1,4 @@
-// RUN: %clang_cc1 -no-opaque-pointers -fenable-matrix -triple x86_64-apple-darwin %s -emit-llvm -disable-llvm-passes -o - | FileCheck %s
+// RUN: %clang_cc1 -fenable-matrix -triple x86_64-apple-darwin %s -emit-llvm -disable-llvm-passes -o - | FileCheck %s
 
 typedef char cx5x5 __attribute__((matrix_type(5, 5)));
 typedef int ix5x5 __attribute__((matrix_type(5, 5)));
@@ -11,9 +11,9 @@ typedef unsigned long unsigned_long_int_5x5 __attribute__((matrix_type(5, 5)));
 
 void cast_char_matrix_to_int(cx5x5 c, ix5x5 i) {
   // CHECK-LABEL: define{{.*}} void @cast_char_matrix_to_int(<25 x i8> noundef %c, <25 x i32> noundef %i)
-  // CHECK:       [[C:%.*]] = load <25 x i8>, <25 x i8>* {{.*}}, align 1
+  // CHECK:       [[C:%.*]] = load <25 x i8>, ptr {{.*}}, align 1
   // CHECK-NEXT:  [[CONV:%.*]] = sext <25 x i8> [[C]] to <25 x i32>
-  // CHECK-NEXT:  store <25 x i32> [[CONV]], <25 x i32>* {{.*}}, align 4
+  // CHECK-NEXT:  store <25 x i32> [[CONV]], ptr {{.*}}, align 4
   // CHECK-NEXT:  ret void
 
   i = (ix5x5)c;
@@ -21,9 +21,9 @@ void cast_char_matrix_to_int(cx5x5 c, ix5x5 i) {
 
 void cast_char_matrix_to_unsigned_int(cx5x5 c, unsigned_int_5x5 u) {
   // CHECK-LABEL: define{{.*}} void @cast_char_matrix_to_unsigned_int(<25 x i8> noundef %c, <25 x i32> noundef %u)
-  // CHECK:       [[C:%.*]] = load <25 x i8>, <25 x i8>* {{.*}}, align 1
+  // CHECK:       [[C:%.*]] = load <25 x i8>, ptr {{.*}}, align 1
   // CHECK-NEXT:  [[CONV:%.*]] = sext <25 x i8> [[C]] to <25 x i32>
-  // CHECK-NEXT:  store <25 x i32> [[CONV]], <25 x i32>* {{.*}}, align 4
+  // CHECK-NEXT:  store <25 x i32> [[CONV]], ptr {{.*}}, align 4
   // CHECK-NEXT:  ret void
 
   u = (unsigned_int_5x5)c;
@@ -31,9 +31,9 @@ void cast_char_matrix_to_unsigned_int(cx5x5 c, unsigned_int_5x5 u) {
 
 void cast_unsigned_long_int_matrix_to_short(unsigned_long_int_5x5 u, sx5x5 s) {
   // CHECK-LABEL: define{{.*}} void @cast_unsigned_long_int_matrix_to_short(<25 x i64> noundef %u, <25 x i16> noundef %s)
-  // CHECK:       [[U:%.*]] = load <25 x i64>, <25 x i64>* {{.*}}, align 8
+  // CHECK:       [[U:%.*]] = load <25 x i64>, ptr {{.*}}, align 8
   // CHECK-NEXT:  [[CONV:%.*]] = trunc <25 x i64> [[U]] to <25 x i16>
-  // CHECK-NEXT:  store <25 x i16> [[CONV]], <25 x i16>* {{.*}}, align 2
+  // CHECK-NEXT:  store <25 x i16> [[CONV]], ptr {{.*}}, align 2
   // CHECK-NEXT:  ret void
 
   s = (sx5x5)u;
@@ -41,9 +41,9 @@ void cast_unsigned_long_int_matrix_to_short(unsigned_long_int_5x5 u, sx5x5 s) {
 
 void cast_int_matrix_to_short(ix5x5 i, sx5x5 s) {
   // CHECK-LABEL: define{{.*}} void @cast_int_matrix_to_short(<25 x i32> noundef %i, <25 x i16> noundef %s)
-  // CHECK:       [[I:%.*]] = load <25 x i32>, <25 x i32>* {{.*}}, align 4
+  // CHECK:       [[I:%.*]] = load <25 x i32>, ptr {{.*}}, align 4
   // CHECK-NEXT:  [[CONV:%.*]] = trunc <25 x i32> [[I]] to <25 x i16>
-  // CHECK-NEXT:  store <25 x i16> [[CONV]], <25 x i16>* {{.*}}, align 2
+  // CHECK-NEXT:  store <25 x i16> [[CONV]], ptr {{.*}}, align 2
   // CHECK-NEXT:  ret void
 
   s = (sx5x5)i;
@@ -51,9 +51,9 @@ void cast_int_matrix_to_short(ix5x5 i, sx5x5 s) {
 
 void cast_int_matrix_to_float(ix5x5 i, fx5x5 f) {
   // CHECK-LABEL: define{{.*}} void @cast_int_matrix_to_float(<25 x i32> noundef %i, <25 x float> noundef %f)
-  // CHECK:       [[I:%.*]] = load <25 x i32>, <25 x i32>* {{.*}}, align 4
+  // CHECK:       [[I:%.*]] = load <25 x i32>, ptr {{.*}}, align 4
   // CHECK-NEXT:  [[CONV:%.*]] = sitofp <25 x i32> [[I]] to <25 x float>
-  // CHECK-NEXT:  store <25 x float> [[CONV]], <25 x float>* {{.*}}, align 4
+  // CHECK-NEXT:  store <25 x float> [[CONV]], ptr {{.*}}, align 4
   // CHECK-NEXT:  ret void
 
   f = (fx5x5)i;
@@ -61,9 +61,9 @@ void cast_int_matrix_to_float(ix5x5 i, fx5x5 f) {
 
 void cast_unsigned_int_matrix_to_float(unsigned_short_int_5x5 u, fx5x5 f) {
   // CHECK-LABEL: define{{.*}} void @cast_unsigned_int_matrix_to_float(<25 x i16> noundef %u, <25 x float> noundef %f)
-  // CHECK:       [[U:%.*]] = load <25 x i16>, <25 x i16>* {{.*}}, align 2
+  // CHECK:       [[U:%.*]] = load <25 x i16>, ptr {{.*}}, align 2
   // CHECK-NEXT:  [[CONV:%.*]] = uitofp <25 x i16> [[U]] to <25 x float>
-  // CHECK-NEXT:  store <25 x float> [[CONV]], <25 x float>* {{.*}}, align 4
+  // CHECK-NEXT:  store <25 x float> [[CONV]], ptr {{.*}}, align 4
   // CHECK-NEXT:  ret void
 
   f = (fx5x5)u;
@@ -71,9 +71,9 @@ void cast_unsigned_int_matrix_to_float(unsigned_short_int_5x5 u, fx5x5 f) {
 
 void cast_double_matrix_to_int(dx5x5 d, ix5x5 i) {
   // CHECK-LABEL: define{{.*}} void @cast_double_matrix_to_int(<25 x double> noundef %d, <25 x i32> noundef %i)
-  // CHECK:       [[D:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8
+  // CHECK:       [[D:%.*]] = load <25 x double>, ptr {{.*}}, align 8
   // CHECK-NEXT:  [[CONV:%.*]] = fptosi <25 x double> [[D]] to <25 x i32>
-  // CHECK-NEXT:  store <25 x i32> [[CONV]], <25 x i32>* {{.*}}, align 4
+  // CHECK-NEXT:  store <25 x i32> [[CONV]], ptr {{.*}}, align 4
   // CHECK-NEXT:  ret void
 
   i = (ix5x5)d;
@@ -81,9 +81,9 @@ void cast_double_matrix_to_int(dx5x5 d, ix5x5 i) {
 
 void cast_float_matrix_to_unsigned_short_int(fx5x5 f, unsigned_short_int_5x5 i) {
   // CHECK-LABEL: define{{.*}} void @cast_float_matrix_to_unsigned_short_int(<25 x float> noundef %f, <25 x i16> noundef %i)
-  // CHECK:       [[F:%.*]] = load <25 x float>, <25 x float>* {{.*}}, align 4
+  // CHECK:       [[F:%.*]] = load <25 x float>, ptr {{.*}}, align 4
   // CHECK-NEXT:  [[CONV:%.*]] = fptoui <25 x float> [[F]] to <25 x i16>
-  // CHECK-NEXT:  store <25 x i16> [[CONV]], <25 x i16>* %1, align 2
+  // CHECK-NEXT:  store <25 x i16> [[CONV]], ptr %i.addr, align 2
   // CHECK-NEXT:  ret void
 
   i = (unsigned_short_int_5x5)f;
@@ -91,9 +91,9 @@ void cast_float_matrix_to_unsigned_short_int(fx5x5 f, unsigned_short_int_5x5 i)
 
 void cast_double_matrix_to_float(dx5x5 d, fx5x5 f) {
   // CHECK-LABEL: define{{.*}} void @cast_double_matrix_to_float(<25 x double> noundef %d, <25 x float> noundef %f)
-  // CHECK:       [[D:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8
+  // CHECK:       [[D:%.*]] = load <25 x double>, ptr {{.*}}, align 8
   // CHECK-NEXT:  [[CONV:%.*]] = fptrunc <25 x double> [[D]] to <25 x float>
-  // CHECK-NEXT:  store <25 x float> [[CONV]], <25 x float>* {{.*}}, align 4
+  // CHECK-NEXT:  store <25 x float> [[CONV]], ptr {{.*}}, align 4
   // CHECK-NEXT:  ret void
 
   f = (fx5x5)d;
@@ -101,9 +101,9 @@ void cast_double_matrix_to_float(dx5x5 d, fx5x5 f) {
 
 void cast_unsigned_short_int_to_unsigned_int(unsigned_short_int_5x5 s, unsigned_int_5x5 i) {
   // CHECK-LABEL: define{{.*}} void @cast_unsigned_short_int_to_unsigned_int(<25 x i16> noundef %s, <25 x i32> noundef %i)
-  // CHECK:       [[S:%.*]] = load <25 x i16>, <25 x i16>* {{.*}}, align 2
+  // CHECK:       [[S:%.*]] = load <25 x i16>, ptr {{.*}}, align 2
   // CHECK-NEXT:  [[CONV:%.*]] = zext <25 x i16> [[S]] to <25 x i32>
-  // CHECK-NEXT:  store <25 x i32> [[CONV]], <25 x i32>* {{.*}}, align 4
+  // CHECK-NEXT:  store <25 x i32> [[CONV]], ptr {{.*}}, align 4
   // CHECK-NEXT:  ret void
 
   i = (unsigned_int_5x5)s;
@@ -111,9 +111,9 @@ void cast_unsigned_short_int_to_unsigned_int(unsigned_short_int_5x5 s, unsigned_
 
 void cast_unsigned_long_int_to_unsigned_short_int(unsigned_long_int_5x5 l, unsigned_short_int_5x5 s) {
   // CHECK-LABEL: define{{.*}} void @cast_unsigned_long_int_to_unsigned_short_int(<25 x i64> noundef %l, <25 x i16> noundef %s)
-  // CHECK:       [[L:%.*]] = load <25 x i64>, <25 x i64>* %0, align 8
+  // CHECK:       [[L:%.*]] = load <25 x i64>, ptr %l.addr, align 8
   // CHECK-NEXT:  [[CONV:%.*]] = trunc <25 x i64> [[L]] to <25 x i16>
-  // CHECK-NEXT:  store <25 x i16> [[CONV]], <25 x i16>* {{.*}}, align 2
+  // CHECK-NEXT:  store <25 x i16> [[CONV]], ptr {{.*}}, align 2
   // CHECK-NEXT:  ret void
 
   s = (unsigned_short_int_5x5)l;
@@ -121,9 +121,9 @@ void cast_unsigned_long_int_to_unsigned_short_int(unsigned_long_int_5x5 l, unsig
 
 void cast_unsigned_short_int_to_int(unsigned_short_int_5x5 u, ix5x5 i) {
   // CHECK-LABEL: define{{.*}} void @cast_unsigned_short_int_to_int(<25 x i16> noundef %u, <25 x i32> noundef %i)
-  // CHECK:       [[U:%.*]] = load <25 x i16>, <25 x i16>* %0, align 2
+  // CHECK:       [[U:%.*]] = load <25 x i16>, ptr %u.addr, align 2
   // CHECK-NEXT:  [[CONV:%.*]] = zext <25 x i16> [[U]] to <25 x i32>
-  // CHECK-NEXT:  store <25 x i32> [[CONV]], <25 x i32>* {{.*}}, align 4
+  // CHECK-NEXT:  store <25 x i32> [[CONV]], ptr {{.*}}, align 4
   // CHECK-NEXT:  ret void
 
   i = (ix5x5)u;
@@ -131,9 +131,9 @@ void cast_unsigned_short_int_to_int(unsigned_short_int_5x5 u, ix5x5 i) {
 
 void cast_int_to_unsigned_long_int(ix5x5 i, unsigned_long_int_5x5 u) {
   // CHECK-LABEL: define{{.*}} void @cast_int_to_unsigned_long_int(<25 x i32> noundef %i, <25 x i64> noundef %u)
-  // CHECK:       [[I:%.*]] = load <25 x i32>, <25 x i32>* %0, align 4
+  // CHECK:       [[I:%.*]] = load <25 x i32>, ptr %i.addr, align 4
   // CHECK-NEXT:  [[CONV:%.*]] = sext <25 x i32> [[I]] to <25 x i64>
-  // CHECK-NEXT:  store <25 x i64> [[CONV]], <25 x i64>* {{.*}}, align 8
+  // CHECK-NEXT:  store <25 x i64> [[CONV]], ptr {{.*}}, align 8
   // CHECK-NEXT:  ret void
 
   u = (unsigned_long_int_5x5)i;
diff --git a/clang/test/CodeGen/matrix-type-operators-fast-math.c b/clang/test/CodeGen/matrix-type-operators-fast-math.c
index ae02553adb90f..724b282830286 100644
--- a/clang/test/CodeGen/matrix-type-operators-fast-math.c
+++ b/clang/test/CodeGen/matrix-type-operators-fast-math.c
@@ -8,7 +8,7 @@ typedef unsigned long long ullx4x2_t __attribute__((matrix_type(4, 2)));
 // Floating point matrix/scalar additions.
 
 void add_matrix_matrix_double(dx5x5_t a, dx5x5_t b, dx5x5_t c) {
-  // CHECK-LABEL: define{{.*}} void @add_matrix_matrix_double(<25 x double> noundef %a, <25 x double> noundef %b, <25 x double> noundef %c)
+  // CHECK-LABEL: define{{.*}} void @add_matrix_matrix_double(<25 x double> noundef nofpclass(nan inf) %a, <25 x double> noundef nofpclass(nan inf) %b, <25 x double> noundef nofpclass(nan inf) %c)
   // CHECK:       [[B:%.*]] = load <25 x double>, ptr {{.*}}, align 8
   // CHECK-NEXT:  [[C:%.*]] = load <25 x double>, ptr {{.*}}, align 8
   // CHECK-NEXT:  [[RES:%.*]] = fadd reassoc nnan ninf nsz arcp afn <25 x double> [[B]], [[C]]
@@ -18,7 +18,7 @@ void add_matrix_matrix_double(dx5x5_t a, dx5x5_t b, dx5x5_t c) {
 }
 
 void add_compound_assign_matrix_double(dx5x5_t a, dx5x5_t b) {
-  // CHECK-LABEL: define{{.*}} void @add_compound_assign_matrix_double(<25 x double> noundef %a, <25 x double> noundef %b)
+  // CHECK-LABEL: define{{.*}} void @add_compound_assign_matrix_double(<25 x double> noundef nofpclass(nan inf) %a, <25 x double> noundef nofpclass(nan inf) %b)
   // CHECK:       [[B:%.*]] = load <25 x double>, ptr {{.*}}, align 8
   // CHECK-NEXT:  [[A:%.*]] = load <25 x double>, ptr {{.*}}, align 8
   // CHECK-NEXT:  [[RES:%.*]] = fadd reassoc nnan ninf nsz arcp afn <25 x double> [[A]], [[B]]
@@ -28,7 +28,7 @@ void add_compound_assign_matrix_double(dx5x5_t a, dx5x5_t b) {
 }
 
 void subtract_compound_assign_matrix_double(dx5x5_t a, dx5x5_t b) {
-  // CHECK-LABEL: define{{.*}} void @subtract_compound_assign_matrix_double(<25 x double> noundef %a, <25 x double> noundef %b)
+  // CHECK-LABEL: define{{.*}} void @subtract_compound_assign_matrix_double(<25 x double> noundef nofpclass(nan inf) %a, <25 x double> noundef nofpclass(nan inf) %b)
   // CHECK:       [[B:%.*]] = load <25 x double>, ptr {{.*}}, align 8
   // CHECK-NEXT:  [[A:%.*]] = load <25 x double>, ptr {{.*}}, align 8
   // CHECK-NEXT:  [[RES:%.*]] = fsub reassoc nnan ninf nsz arcp afn <25 x double> [[A]], [[B]]
@@ -38,7 +38,7 @@ void subtract_compound_assign_matrix_double(dx5x5_t a, dx5x5_t b) {
 }
 
 void add_matrix_scalar_double_float(dx5x5_t a, float vf) {
-  // CHECK-LABEL: define{{.*}} void @add_matrix_scalar_double_float(<25 x double> noundef %a, float noundef %vf)
+  // CHECK-LABEL: define{{.*}} void @add_matrix_scalar_double_float(<25 x double> noundef nofpclass(nan inf) %a, float noundef nofpclass(nan inf) %vf)
   // CHECK:       [[MATRIX:%.*]] = load <25 x double>, ptr {{.*}}, align 8
   // CHECK-NEXT:  [[SCALAR:%.*]] = load float, ptr %vf.addr, align 4
   // CHECK-NEXT:  [[SCALAR_EXT:%.*]] = fpext float [[SCALAR]] to double
@@ -51,7 +51,7 @@ void add_matrix_scalar_double_float(dx5x5_t a, float vf) {
 }
 
 void add_compound_matrix_scalar_double_float(dx5x5_t a, float vf) {
-  // CHECK-LABEL: define{{.*}} void @add_compound_matrix_scalar_double_float(<25 x double> noundef %a, float noundef %vf)
+  // CHECK-LABEL: define{{.*}} void @add_compound_matrix_scalar_double_float(<25 x double> noundef nofpclass(nan inf) %a, float noundef nofpclass(nan inf) %vf)
   // CHECK:  [[SCALAR:%.*]] = load float, ptr %vf.addr, align 4
   // CHECK-NEXT:  [[SCALAR_EXT:%.*]] = fpext float [[SCALAR]] to double
   // CHECK-NEXT:  [[MATRIX:%.*]] = load <25 x double>, ptr {{.*}}, align 8
@@ -64,7 +64,7 @@ void add_compound_matrix_scalar_double_float(dx5x5_t a, float vf) {
 }
 
 void subtract_compound_matrix_scalar_double_float(dx5x5_t a, float vf) {
-  // CHECK-LABEL: define{{.*}} void @subtract_compound_matrix_scalar_double_float(<25 x double> noundef %a, float noundef %vf)
+  // CHECK-LABEL: define{{.*}} void @subtract_compound_matrix_scalar_double_float(<25 x double> noundef nofpclass(nan inf) %a, float noundef nofpclass(nan inf) %vf)
   // CHECK:  [[SCALAR:%.*]] = load float, ptr %vf.addr, align 4
   // CHECK-NEXT:  [[SCALAR_EXT:%.*]] = fpext float [[SCALAR]] to double
   // CHECK-NEXT:  [[MATRIX:%.*]] = load <25 x double>, ptr {{.*}}, align 8
diff --git a/clang/test/CodeGen/matrix-type-operators.c b/clang/test/CodeGen/matrix-type-operators.c
index 62a1838ed3a56..c588685851548 100644
--- a/clang/test/CodeGen/matrix-type-operators.c
+++ b/clang/test/CodeGen/matrix-type-operators.c
@@ -1,5 +1,5 @@
-// RUN: %clang_cc1 -no-opaque-pointers -O0 -fenable-matrix -triple x86_64-apple-darwin %s -emit-llvm -disable-llvm-passes -o - | FileCheck --check-prefixes=CHECK,NOOPT %s
-// RUN: %clang_cc1 -no-opaque-pointers -O1 -fenable-matrix -triple x86_64-apple-darwin %s -emit-llvm -disable-llvm-passes -o - | FileCheck --check-prefixes=CHECK,OPT %s
+// RUN: %clang_cc1 -O0 -fenable-matrix -triple x86_64-apple-darwin %s -emit-llvm -disable-llvm-passes -o - | FileCheck --check-prefixes=CHECK,NOOPT %s
+// RUN: %clang_cc1 -O1 -fenable-matrix -triple x86_64-apple-darwin %s -emit-llvm -disable-llvm-passes -o - | FileCheck --check-prefixes=CHECK,OPT %s
 
 
 typedef double dx5x5_t __attribute__((matrix_type(5, 5)));
@@ -11,241 +11,241 @@ typedef unsigned long long ullx4x2_t __attribute__((matrix_type(4, 2)));
 
 void add_matrix_matrix_double(dx5x5_t a, dx5x5_t b, dx5x5_t c) {
   // CHECK-LABEL: define{{.*}} void @add_matrix_matrix_double(<25 x double> noundef %a, <25 x double> noundef %b, <25 x double> noundef %c)
-  // NOOPT:       [[B:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-  // NOOPT-NEXT:  [[C:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-  // OPT:         [[B:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[C:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[B:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+  // NOOPT-NEXT:  [[C:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+  // OPT:         [[B:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[C:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[RES:%.*]] = fadd <25 x double> [[B]], [[C]]
-  // CHECK-NEXT:  store <25 x double> [[RES]], <25 x double>* {{.*}}, align 8
+  // CHECK-NEXT:  store <25 x double> [[RES]], ptr {{.*}}, align 8
 
   a = b + c;
 }
 
 void add_compound_assign_matrix_double(dx5x5_t a, dx5x5_t b) {
   // CHECK-LABEL: define{{.*}} void @add_compound_assign_matrix_double(<25 x double> noundef %a, <25 x double> noundef %b)
-  // NOOPT:       [[B:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-  // NOOPT-NEXT:  [[A:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-  // OPT:         [[B:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[A:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[B:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+  // NOOPT-NEXT:  [[A:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+  // OPT:         [[B:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[A:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[RES:%.*]] = fadd <25 x double> [[A]], [[B]]
-  // CHECK-NEXT:  store <25 x double> [[RES]], <25 x double>* {{.*}}, align 8
+  // CHECK-NEXT:  store <25 x double> [[RES]], ptr {{.*}}, align 8
 
   a += b;
 }
 
 void subtract_compound_assign_matrix_double(dx5x5_t a, dx5x5_t b) {
   // CHECK-LABEL: define{{.*}} void @subtract_compound_assign_matrix_double(<25 x double> noundef %a, <25 x double> noundef %b)
-  // NOOPT:       [[B:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-  // NOOPT-NEXT:  [[A:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-  // OPT:         [[B:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[A:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[B:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+  // NOOPT-NEXT:  [[A:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+  // OPT:         [[B:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[A:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[RES:%.*]] = fsub <25 x double> [[A]], [[B]]
-  // CHECK-NEXT:  store <25 x double> [[RES]], <25 x double>* {{.*}}, align 8
+  // CHECK-NEXT:  store <25 x double> [[RES]], ptr {{.*}}, align 8
 
   a -= b;
 }
 
 void add_matrix_matrix_float(fx2x3_t a, fx2x3_t b, fx2x3_t c) {
   // CHECK-LABEL: define{{.*}} void @add_matrix_matrix_float(<6 x float> noundef %a, <6 x float> noundef %b, <6 x float> noundef %c)
-  // NOOPT:       [[B:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4{{$}}
-  // NOOPT-NEXT:  [[C:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4{{$}}
-  // OPT:         [[B:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[C:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[B:%.*]] = load <6 x float>, ptr {{.*}}, align 4{{$}}
+  // NOOPT-NEXT:  [[C:%.*]] = load <6 x float>, ptr {{.*}}, align 4{{$}}
+  // OPT:         [[B:%.*]] = load <6 x float>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[C:%.*]] = load <6 x float>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[RES:%.*]] = fadd <6 x float> [[B]], [[C]]
-  // CHECK-NEXT:  store <6 x float> [[RES]], <6 x float>* {{.*}}, align 4
+  // CHECK-NEXT:  store <6 x float> [[RES]], ptr {{.*}}, align 4
 
   a = b + c;
 }
 
 void add_compound_assign_matrix_float(fx2x3_t a, fx2x3_t b) {
   // CHECK-LABEL: define{{.*}} void @add_compound_assign_matrix_float(<6 x float> noundef %a, <6 x float> noundef %b)
-  // NOOPT:       [[B:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4{{$}}
-  // NOOPT-NEXT:  [[A:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4{{$}}
-  // OPT:         [[B:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[A:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[B:%.*]] = load <6 x float>, ptr {{.*}}, align 4{{$}}
+  // NOOPT-NEXT:  [[A:%.*]] = load <6 x float>, ptr {{.*}}, align 4{{$}}
+  // OPT:         [[B:%.*]] = load <6 x float>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[A:%.*]] = load <6 x float>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[RES:%.*]] = fadd <6 x float> [[A]], [[B]]
-  // CHECK-NEXT:  store <6 x float> [[RES]], <6 x float>* {{.*}}, align 4
+  // CHECK-NEXT:  store <6 x float> [[RES]], ptr {{.*}}, align 4
 
   a += b;
 }
 
 void subtract_compound_assign_matrix_float(fx2x3_t a, fx2x3_t b) {
   // CHECK-LABEL: define{{.*}} void @subtract_compound_assign_matrix_float(<6 x float> noundef %a, <6 x float> noundef %b)
-  // NOOPT:       [[B:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4{{$}}
-  // NOOPT-NEXT:  [[A:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4{{$}}
-  // OPT:         [[B:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[A:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[B:%.*]] = load <6 x float>, ptr {{.*}}, align 4{{$}}
+  // NOOPT-NEXT:  [[A:%.*]] = load <6 x float>, ptr {{.*}}, align 4{{$}}
+  // OPT:         [[B:%.*]] = load <6 x float>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[A:%.*]] = load <6 x float>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[RES:%.*]] = fsub <6 x float> [[A]], [[B]]
-  // CHECK-NEXT:  store <6 x float> [[RES]], <6 x float>* {{.*}}, align 4
+  // CHECK-NEXT:  store <6 x float> [[RES]], ptr {{.*}}, align 4
 
   a -= b;
 }
 
 void add_matrix_scalar_double_float(dx5x5_t a, float vf) {
   // CHECK-LABEL: define{{.*}} void @add_matrix_scalar_double_float(<25 x double> noundef %a, float noundef %vf)
-  // NOOPT:       [[MATRIX:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-  // NOOPT-NEXT:  [[SCALAR:%.*]] = load float, float* %vf.addr, align 4{{$}}
-  // OPT:         [[MATRIX:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[SCALAR:%.*]] = load float, float* %vf.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[MATRIX:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+  // NOOPT-NEXT:  [[SCALAR:%.*]] = load float, ptr %vf.addr, align 4{{$}}
+  // OPT:         [[MATRIX:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[SCALAR:%.*]] = load float, ptr %vf.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EXT:%.*]] = fpext float [[SCALAR]] to double
   // CHECK-NEXT:  [[SCALAR_EMBED:%.*]] = insertelement <25 x double> poison, double [[SCALAR_EXT]], i64 0
   // CHECK-NEXT:  [[SCALAR_EMBED1:%.*]] = shufflevector <25 x double> [[SCALAR_EMBED]], <25 x double> poison, <25 x i32> zeroinitializer
   // CHECK-NEXT:  [[RES:%.*]] = fadd <25 x double> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:  store <25 x double> [[RES]], <25 x double>* {{.*}}, align 8
+  // CHECK-NEXT:  store <25 x double> [[RES]], ptr {{.*}}, align 8
 
   a = a + vf;
 }
 
 void add_compound_matrix_scalar_double_float(dx5x5_t a, float vf) {
   // CHECK-LABEL: define{{.*}} void @add_compound_matrix_scalar_double_float(<25 x double> noundef %a, float noundef %vf)
-  // NOOPT:  [[SCALAR:%.*]] = load float, float* %vf.addr, align 4{{$}}
-  // OPT:    [[SCALAR:%.*]] = load float, float* %vf.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:  [[SCALAR:%.*]] = load float, ptr %vf.addr, align 4{{$}}
+  // OPT:    [[SCALAR:%.*]] = load float, ptr %vf.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EXT:%.*]] = fpext float [[SCALAR]] to double
-  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-  // OPT-NEXT:    [[MATRIX:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+  // OPT-NEXT:    [[MATRIX:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EMBED:%.*]] = insertelement <25 x double> poison, double [[SCALAR_EXT]], i64 0
   // CHECK-NEXT:  [[SCALAR_EMBED1:%.*]] = shufflevector <25 x double> [[SCALAR_EMBED]], <25 x double> poison, <25 x i32> zeroinitializer
   // CHECK-NEXT:  [[RES:%.*]] = fadd <25 x double> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:  store <25 x double> [[RES]], <25 x double>* {{.*}}, align 8
+  // CHECK-NEXT:  store <25 x double> [[RES]], ptr {{.*}}, align 8
 
   a += vf;
 }
 
 void subtract_compound_matrix_scalar_double_float(dx5x5_t a, float vf) {
   // CHECK-LABEL: define{{.*}} void @subtract_compound_matrix_scalar_double_float(<25 x double> noundef %a, float noundef %vf)
-  // NOOPT:  [[SCALAR:%.*]] = load float, float* %vf.addr, align 4{{$}}
-  // OPT:    [[SCALAR:%.*]] = load float, float* %vf.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:  [[SCALAR:%.*]] = load float, ptr %vf.addr, align 4{{$}}
+  // OPT:    [[SCALAR:%.*]] = load float, ptr %vf.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EXT:%.*]] = fpext float [[SCALAR]] to double
-  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-  // OPT-NEXT:    [[MATRIX:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+  // OPT-NEXT:    [[MATRIX:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EMBED:%.*]] = insertelement <25 x double> poison, double [[SCALAR_EXT]], i64 0
   // CHECK-NEXT:  [[SCALAR_EMBED1:%.*]] = shufflevector <25 x double> [[SCALAR_EMBED]], <25 x double> poison, <25 x i32> zeroinitializer
   // CHECK-NEXT:  [[RES:%.*]] = fsub <25 x double> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:  store <25 x double> [[RES]], <25 x double>* {{.*}}, align 8
+  // CHECK-NEXT:  store <25 x double> [[RES]], ptr {{.*}}, align 8
 
   a -= vf;
 }
 
 void add_matrix_scalar_double_double(dx5x5_t a, double vd) {
   // CHECK-LABEL: define{{.*}} void @add_matrix_scalar_double_double(<25 x double> noundef %a, double noundef %vd)
-  // NOOPT:       [[MATRIX:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-  // NOOPT-NEXT:  [[SCALAR:%.*]] = load double, double* %vd.addr, align 8{{$}}
-  // OPT:         [[MATRIX:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[SCALAR:%.*]] = load double, double* %vd.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[MATRIX:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+  // NOOPT-NEXT:  [[SCALAR:%.*]] = load double, ptr %vd.addr, align 8{{$}}
+  // OPT:         [[MATRIX:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[SCALAR:%.*]] = load double, ptr %vd.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EMBED:%.*]] = insertelement <25 x double> poison, double [[SCALAR]], i64 0
   // CHECK-NEXT:  [[SCALAR_EMBED1:%.*]] = shufflevector <25 x double> [[SCALAR_EMBED]], <25 x double> poison, <25 x i32> zeroinitializer
   // CHECK-NEXT:  [[RES:%.*]] = fadd <25 x double> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:  store <25 x double> [[RES]], <25 x double>* {{.*}}, align 8
+  // CHECK-NEXT:  store <25 x double> [[RES]], ptr {{.*}}, align 8
 
   a = a + vd;
 }
 
 void add_compound_matrix_scalar_double_double(dx5x5_t a, double vd) {
   // CHECK-LABEL: define{{.*}} void @add_compound_matrix_scalar_double_double(<25 x double> noundef %a, double noundef %vd)
-  // NOOPT:       [[SCALAR:%.*]] = load double, double* %vd.addr, align 8{{$}}
-  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-  // OPT:         [[SCALAR:%.*]] = load double, double* %vd.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[MATRIX:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[SCALAR:%.*]] = load double, ptr %vd.addr, align 8{{$}}
+  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+  // OPT:         [[SCALAR:%.*]] = load double, ptr %vd.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[MATRIX:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EMBED:%.*]] = insertelement <25 x double> poison, double [[SCALAR]], i64 0
   // CHECK-NEXT:  [[SCALAR_EMBED1:%.*]] = shufflevector <25 x double> [[SCALAR_EMBED]], <25 x double> poison, <25 x i32> zeroinitializer
   // CHECK-NEXT:  [[RES:%.*]] = fadd <25 x double> [[MATRIX]], [[SCALAR_EMBED1]]
-  // store <25 x double> [[RES]], <25 x double>* {{.*}}, align 8
+  // store <25 x double> [[RES]], ptr {{.*}}, align 8
   a += vd;
 }
 
 void subtract_compound_matrix_scalar_double_double(dx5x5_t a, double vd) {
   // CHECK-LABEL: define{{.*}} void @subtract_compound_matrix_scalar_double_double(<25 x double> noundef %a, double noundef %vd)
-  // NOOPT:       [[SCALAR:%.*]] = load double, double* %vd.addr, align 8{{$}}
-  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-  // OPT:         [[SCALAR:%.*]] = load double, double* %vd.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[MATRIX:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[SCALAR:%.*]] = load double, ptr %vd.addr, align 8{{$}}
+  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+  // OPT:         [[SCALAR:%.*]] = load double, ptr %vd.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[MATRIX:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EMBED:%.*]] = insertelement <25 x double> poison, double [[SCALAR]], i64 0
   // CHECK-NEXT:  [[SCALAR_EMBED1:%.*]] = shufflevector <25 x double> [[SCALAR_EMBED]], <25 x double> poison, <25 x i32> zeroinitializer
   // CHECK-NEXT:  [[RES:%.*]] = fsub <25 x double> [[MATRIX]], [[SCALAR_EMBED1]]
-  // store <25 x double> [[RES]], <25 x double>* {{.*}}, align 8
+  // store <25 x double> [[RES]], ptr {{.*}}, align 8
   a -= vd;
 }
 
 void add_matrix_scalar_float_float(fx2x3_t b, float vf) {
   // CHECK-LABEL: define{{.*}} void @add_matrix_scalar_float_float(<6 x float> noundef %b, float noundef %vf)
-  // NOOPT:       [[MATRIX:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4{{$}}
-  // NOOPT-NEXT:  [[SCALAR:%.*]] = load float, float* %vf.addr, align 4{{$}}
-  // OPT:         [[MATRIX:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[SCALAR:%.*]] = load float, float* %vf.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[MATRIX:%.*]] = load <6 x float>, ptr {{.*}}, align 4{{$}}
+  // NOOPT-NEXT:  [[SCALAR:%.*]] = load float, ptr %vf.addr, align 4{{$}}
+  // OPT:         [[MATRIX:%.*]] = load <6 x float>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[SCALAR:%.*]] = load float, ptr %vf.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EMBED:%.*]] = insertelement <6 x float> poison, float [[SCALAR]], i64 0
   // CHECK-NEXT:  [[SCALAR_EMBED1:%.*]] = shufflevector <6 x float> [[SCALAR_EMBED]], <6 x float> poison, <6 x i32> zeroinitializer
   // CHECK-NEXT:  [[RES:%.*]] = fadd <6 x float> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:  store <6 x float> [[RES]], <6 x float>* {{.*}}, align 4
+  // CHECK-NEXT:  store <6 x float> [[RES]], ptr {{.*}}, align 4
 
   b = b + vf;
 }
 
 void add_compound_matrix_scalar_float_float(fx2x3_t b, float vf) {
   // CHECK-LABEL: define{{.*}} void @add_compound_matrix_scalar_float_float(<6 x float> noundef %b, float noundef %vf)
-  // NOOPT:       [[SCALAR:%.*]] = load float, float* %vf.addr, align 4{{$}}
-  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <6 x float>, <6 x float>* %0, align 4{{$}}
-  // OPT:         [[SCALAR:%.*]] = load float, float* %vf.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[MATRIX:%.*]] = load <6 x float>, <6 x float>* %0, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[SCALAR:%.*]] = load float, ptr %vf.addr, align 4{{$}}
+  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <6 x float>, ptr %b.addr, align 4{{$}}
+  // OPT:         [[SCALAR:%.*]] = load float, ptr %vf.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[MATRIX:%.*]] = load <6 x float>, ptr %b.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EMBED:%.*]] = insertelement <6 x float> poison, float [[SCALAR]], i64 0
   // CHECK-NEXT:  [[SCALAR_EMBED1:%.*]] = shufflevector <6 x float> [[SCALAR_EMBED]], <6 x float> poison, <6 x i32> zeroinitializer
   // CHECK-NEXT:  [[RES:%.*]] = fadd <6 x float> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:  store <6 x float> [[RES]], <6 x float>* {{.*}}, align 4
+  // CHECK-NEXT:  store <6 x float> [[RES]], ptr {{.*}}, align 4
   b += vf;
 }
 
 void subtract_compound_matrix_scalar_float_float(fx2x3_t b, float vf) {
   // CHECK-LABEL: define{{.*}} void @subtract_compound_matrix_scalar_float_float(<6 x float> noundef %b, float noundef %vf)
-  // NOOPT:       [[SCALAR:%.*]] = load float, float* %vf.addr, align 4{{$}}
-  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <6 x float>, <6 x float>* %0, align 4{{$}}
-  // OPT:         [[SCALAR:%.*]] = load float, float* %vf.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[MATRIX:%.*]] = load <6 x float>, <6 x float>* %0, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[SCALAR:%.*]] = load float, ptr %vf.addr, align 4{{$}}
+  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <6 x float>, ptr %b.addr, align 4{{$}}
+  // OPT:         [[SCALAR:%.*]] = load float, ptr %vf.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[MATRIX:%.*]] = load <6 x float>, ptr %b.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EMBED:%.*]] = insertelement <6 x float> poison, float [[SCALAR]], i64 0
   // CHECK-NEXT:  [[SCALAR_EMBED1:%.*]] = shufflevector <6 x float> [[SCALAR_EMBED]], <6 x float> poison, <6 x i32> zeroinitializer
   // CHECK-NEXT:  [[RES:%.*]] = fsub <6 x float> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:  store <6 x float> [[RES]], <6 x float>* {{.*}}, align 4
+  // CHECK-NEXT:  store <6 x float> [[RES]], ptr {{.*}}, align 4
   b -= vf;
 }
 
 void add_matrix_scalar_float_double(fx2x3_t b, double vd) {
   // CHECK-LABEL: define{{.*}} void @add_matrix_scalar_float_double(<6 x float> noundef %b, double noundef %vd)
-  // NOOPT:       [[MATRIX:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4{{$}}
-  // NOOPT-NEXT:  [[SCALAR:%.*]] = load double, double* %vd.addr, align 8{{$}}
-  // OPT:         [[MATRIX:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[SCALAR:%.*]] = load double, double* %vd.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[MATRIX:%.*]] = load <6 x float>, ptr {{.*}}, align 4{{$}}
+  // NOOPT-NEXT:  [[SCALAR:%.*]] = load double, ptr %vd.addr, align 8{{$}}
+  // OPT:         [[MATRIX:%.*]] = load <6 x float>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[SCALAR:%.*]] = load double, ptr %vd.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_TRUNC:%.*]] = fptrunc double [[SCALAR]] to float
   // CHECK-NEXT:  [[SCALAR_EMBED:%.*]] = insertelement <6 x float> poison, float [[SCALAR_TRUNC]], i64 0
   // CHECK-NEXT:  [[SCALAR_EMBED1:%.*]] = shufflevector <6 x float> [[SCALAR_EMBED]], <6 x float> poison, <6 x i32> zeroinitializer
   // CHECK-NEXT:  [[RES:%.*]] = fadd <6 x float> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:  store <6 x float> [[RES]], <6 x float>* {{.*}}, align 4
+  // CHECK-NEXT:  store <6 x float> [[RES]], ptr {{.*}}, align 4
 
   b = b + vd;
 }
 
 void add_compound_matrix_scalar_float_double(fx2x3_t b, double vd) {
   // CHECK-LABEL: define{{.*}} void @add_compound_matrix_scalar_float_double(<6 x float> noundef %b, double noundef %vd)
-  // NOOPT:       [[SCALAR:%.*]] = load double, double* %vd.addr, align 8{{$}}
-  // OPT:         [[SCALAR:%.*]] = load double, double* %vd.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[SCALAR:%.*]] = load double, ptr %vd.addr, align 8{{$}}
+  // OPT:         [[SCALAR:%.*]] = load double, ptr %vd.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_TRUNC:%.*]] = fptrunc double [[SCALAR]] to float
-  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4{{$}}
-  // OPT-NEXT:    [[MATRIX:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <6 x float>, ptr {{.*}}, align 4{{$}}
+  // OPT-NEXT:    [[MATRIX:%.*]] = load <6 x float>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EMBED:%.*]] = insertelement <6 x float> poison, float [[SCALAR_TRUNC]], i64 0
   // CHECK-NEXT:  [[SCALAR_EMBED1:%.*]] = shufflevector <6 x float> [[SCALAR_EMBED]], <6 x float> poison, <6 x i32> zeroinitializer
   // CHECK-NEXT:  [[RES:%.*]] = fadd <6 x float> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:  store <6 x float> [[RES]], <6 x float>* {{.*}}, align 4
+  // CHECK-NEXT:  store <6 x float> [[RES]], ptr {{.*}}, align 4
   b += vd;
 }
 
 void subtract_compound_matrix_scalar_float_double(fx2x3_t b, double vd) {
   // CHECK-LABEL: define{{.*}} void @subtract_compound_matrix_scalar_float_double(<6 x float> noundef %b, double noundef %vd)
-  // NOOPT:       [[SCALAR:%.*]] = load double, double* %vd.addr, align 8{{$}}
-  // OPT:         [[SCALAR:%.*]] = load double, double* %vd.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[SCALAR:%.*]] = load double, ptr %vd.addr, align 8{{$}}
+  // OPT:         [[SCALAR:%.*]] = load double, ptr %vd.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_TRUNC:%.*]] = fptrunc double [[SCALAR]] to float
-  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4{{$}}
-  // OPT-NEXT:    [[MATRIX:%.*]] = load <6 x float>, <6 x float>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <6 x float>, ptr {{.*}}, align 4{{$}}
+  // OPT-NEXT:    [[MATRIX:%.*]] = load <6 x float>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EMBED:%.*]] = insertelement <6 x float> poison, float [[SCALAR_TRUNC]], i64 0
   // CHECK-NEXT:  [[SCALAR_EMBED1:%.*]] = shufflevector <6 x float> [[SCALAR_EMBED]], <6 x float> poison, <6 x i32> zeroinitializer
   // CHECK-NEXT:  [[RES:%.*]] = fsub <6 x float> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:  store <6 x float> [[RES]], <6 x float>* {{.*}}, align 4
+  // CHECK-NEXT:  store <6 x float> [[RES]], ptr {{.*}}, align 4
   b -= vd;
 }
 
@@ -253,332 +253,332 @@ void subtract_compound_matrix_scalar_float_double(fx2x3_t b, double vd) {
 
 void add_matrix_matrix_int(ix9x3_t a, ix9x3_t b, ix9x3_t c) {
   // CHECK-LABEL: define{{.*}} void @add_matrix_matrix_int(<27 x i32> noundef %a, <27 x i32> noundef %b, <27 x i32> noundef %c)
-  // NOOPT:       [[B:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4{{$}}
-  // NOOPT-NEXT:  [[C:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4{{$}}
-  // OPT:         [[B:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[C:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[B:%.*]] = load <27 x i32>, ptr {{.*}}, align 4{{$}}
+  // NOOPT-NEXT:  [[C:%.*]] = load <27 x i32>, ptr {{.*}}, align 4{{$}}
+  // OPT:         [[B:%.*]] = load <27 x i32>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[C:%.*]] = load <27 x i32>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[RES:%.*]] = add <27 x i32> [[B]], [[C]]
-  // CHECK-NEXT:  store <27 x i32> [[RES]], <27 x i32>* {{.*}}, align 4
+  // CHECK-NEXT:  store <27 x i32> [[RES]], ptr {{.*}}, align 4
   a = b + c;
 }
 
 void add_compound_matrix_matrix_int(ix9x3_t a, ix9x3_t b) {
   // CHECK-LABEL: define{{.*}} void @add_compound_matrix_matrix_int(<27 x i32> noundef %a, <27 x i32> noundef %b)
-  // NOOPT:       [[B:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4{{$}}
-  // NOOPT-NEXT:  [[A:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4{{$}}
-  // OPT:         [[B:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[A:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[B:%.*]] = load <27 x i32>, ptr {{.*}}, align 4{{$}}
+  // NOOPT-NEXT:  [[A:%.*]] = load <27 x i32>, ptr {{.*}}, align 4{{$}}
+  // OPT:         [[B:%.*]] = load <27 x i32>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[A:%.*]] = load <27 x i32>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[RES:%.*]] = add <27 x i32> [[A]], [[B]]
-  // CHECK-NEXT:  store <27 x i32> [[RES]], <27 x i32>* {{.*}}, align 4
+  // CHECK-NEXT:  store <27 x i32> [[RES]], ptr {{.*}}, align 4
   a += b;
 }
 
 void subtract_compound_matrix_matrix_int(ix9x3_t a, ix9x3_t b) {
   // CHECK-LABEL: define{{.*}} void @subtract_compound_matrix_matrix_int(<27 x i32> noundef %a, <27 x i32> noundef %b)
-  // NOOPT:       [[B:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4{{$}}
-  // NOOPT-NEXT:  [[A:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4{{$}}
-  // OPT:         [[B:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[A:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[B:%.*]] = load <27 x i32>, ptr {{.*}}, align 4{{$}}
+  // NOOPT-NEXT:  [[A:%.*]] = load <27 x i32>, ptr {{.*}}, align 4{{$}}
+  // OPT:         [[B:%.*]] = load <27 x i32>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[A:%.*]] = load <27 x i32>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[RES:%.*]] = sub <27 x i32> [[A]], [[B]]
-  // CHECK-NEXT:  store <27 x i32> [[RES]], <27 x i32>* {{.*}}, align 4
+  // CHECK-NEXT:  store <27 x i32> [[RES]], ptr {{.*}}, align 4
   a -= b;
 }
 
 void add_matrix_matrix_unsigned_long_long(ullx4x2_t a, ullx4x2_t b, ullx4x2_t c) {
   // CHECK-LABEL: define{{.*}} void @add_matrix_matrix_unsigned_long_long(<8 x i64> noundef %a, <8 x i64> noundef %b, <8 x i64> noundef %c)
-  // NOOPT:       [[B:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8{{$}}
-  // NOOPT-NEXT:  [[C:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8{{$}}
-  // OPT:         [[B:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[C:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[B:%.*]] = load <8 x i64>, ptr {{.*}}, align 8{{$}}
+  // NOOPT-NEXT:  [[C:%.*]] = load <8 x i64>, ptr {{.*}}, align 8{{$}}
+  // OPT:         [[B:%.*]] = load <8 x i64>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[C:%.*]] = load <8 x i64>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[RES:%.*]] = add <8 x i64> [[B]], [[C]]
-  // CHECK-NEXT:  store <8 x i64> [[RES]], <8 x i64>* {{.*}}, align 8
+  // CHECK-NEXT:  store <8 x i64> [[RES]], ptr {{.*}}, align 8
 
   a = b + c;
 }
 
 void add_compound_matrix_matrix_unsigned_long_long(ullx4x2_t a, ullx4x2_t b) {
   // CHECK-LABEL: define{{.*}} void @add_compound_matrix_matrix_unsigned_long_long(<8 x i64> noundef %a, <8 x i64> noundef %b)
-  // NOOPT:       [[B:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8{{$}}
-  // NOOPT-NEXT:  [[A:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8{{$}}
-  // OPT:         [[B:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:    [[A:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[B:%.*]] = load <8 x i64>, ptr {{.*}}, align 8{{$}}
+  // NOOPT-NEXT:  [[A:%.*]] = load <8 x i64>, ptr {{.*}}, align 8{{$}}
+  // OPT:         [[B:%.*]] = load <8 x i64>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[A:%.*]] = load <8 x i64>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[RES:%.*]] = add <8 x i64> [[A]], [[B]]
-  // CHECK-NEXT:  store <8 x i64> [[RES]], <8 x i64>* {{.*}}, align 8
+  // CHECK-NEXT:  store <8 x i64> [[RES]], ptr {{.*}}, align 8
 
   a += b;
 }
 
 void subtract_compound_matrix_matrix_unsigned_long_long(ullx4x2_t a, ullx4x2_t b) {
   // CHECK-LABEL: define{{.*}} void @subtract_compound_matrix_matrix_unsigned_long_long(<8 x i64> noundef %a, <8 x i64> noundef %b)
-  // NOOPT:       [[B:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8{{$}}
-  // OPT:         [[B:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // NOOPT-NEXT:  [[A:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8{{$}}
-  // OPT-NEXT:    [[A:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[B:%.*]] = load <8 x i64>, ptr {{.*}}, align 8{{$}}
+  // OPT:         [[B:%.*]] = load <8 x i64>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:  [[A:%.*]] = load <8 x i64>, ptr {{.*}}, align 8{{$}}
+  // OPT-NEXT:    [[A:%.*]] = load <8 x i64>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[RES:%.*]] = sub <8 x i64> [[A]], [[B]]
-  // CHECK-NEXT:  store <8 x i64> [[RES]], <8 x i64>* {{.*}}, align 8
+  // CHECK-NEXT:  store <8 x i64> [[RES]], ptr {{.*}}, align 8
 
   a -= b;
 }
 
 void add_matrix_scalar_int_short(ix9x3_t a, short vs) {
   // CHECK-LABEL: define{{.*}} void @add_matrix_scalar_int_short(<27 x i32> noundef %a, i16 noundef signext %vs)
-  // NOOPT:        [[MATRIX:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4{{$}}
-  // NOOPT-NEXT:   [[SCALAR:%.*]] = load i16, i16* %vs.addr, align 2{{$}}
-  // OPT:          [[MATRIX:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:     [[SCALAR:%.*]] = load i16, i16* %vs.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:        [[MATRIX:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4{{$}}
+  // NOOPT-NEXT:   [[SCALAR:%.*]] = load i16, ptr %vs.addr, align 2{{$}}
+  // OPT:          [[MATRIX:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:     [[SCALAR:%.*]] = load i16, ptr %vs.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:   [[SCALAR_EXT:%.*]] = sext i16 [[SCALAR]] to i32
   // CHECK-NEXT:   [[SCALAR_EMBED:%.*]] = insertelement <27 x i32> poison, i32 [[SCALAR_EXT]], i64 0
   // CHECK-NEXT:   [[SCALAR_EMBED1:%.*]] = shufflevector <27 x i32> [[SCALAR_EMBED]], <27 x i32> poison, <27 x i32> zeroinitializer
   // CHECK-NEXT:   [[RES:%.*]] = add <27 x i32> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:   store <27 x i32> [[RES]], <27 x i32>* [[MAT_ADDR]], align 4
+  // CHECK-NEXT:   store <27 x i32> [[RES]], ptr [[MAT_ADDR]], align 4
 
   a = a + vs;
 }
 
 void add_compound_matrix_scalar_int_short(ix9x3_t a, short vs) {
   // CHECK-LABEL: define{{.*}} void @add_compound_matrix_scalar_int_short(<27 x i32> noundef %a, i16 noundef signext %vs)
-  // NOOPT:       [[SCALAR:%.*]] = load i16, i16* %vs.addr, align 2{{$}}
-  // OPT:         [[SCALAR:%.*]] = load i16, i16* %vs.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[SCALAR:%.*]] = load i16, ptr %vs.addr, align 2{{$}}
+  // OPT:         [[SCALAR:%.*]] = load i16, ptr %vs.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EXT:%.*]] = sext i16 [[SCALAR]] to i32
-  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <27 x i32>, <27 x i32>* %0, align 4{{$}}
-  // OPT-NEXT:    [[MATRIX:%.*]] = load <27 x i32>, <27 x i32>* %0, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <27 x i32>, ptr %a.addr, align 4{{$}}
+  // OPT-NEXT:    [[MATRIX:%.*]] = load <27 x i32>, ptr %a.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EMBED:%.*]] = insertelement <27 x i32> poison, i32 [[SCALAR_EXT:%.*]], i64 0
   // CHECK-NEXT:  [[SCALAR_EMBED1:%.*]] = shufflevector <27 x i32> [[SCALAR_EMBED]], <27 x i32> poison, <27 x i32> zeroinitializer
   // CHECK-NEXT:  [[RES:%.*]] = add <27 x i32> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:  store <27 x i32> [[RES]], <27 x i32>* [[MAT_ADDR]], align 4
+  // CHECK-NEXT:  store <27 x i32> [[RES]], ptr [[MAT_ADDR]], align 4
 
   a += vs;
 }
 
 void subtract_compound_matrix_scalar_int_short(ix9x3_t a, short vs) {
   // CHECK-LABEL: define{{.*}} void @subtract_compound_matrix_scalar_int_short(<27 x i32> noundef %a, i16 noundef signext %vs)
-  // NOOPT:       [[SCALAR:%.*]] = load i16, i16* %vs.addr, align 2{{$}}
-  // OPT:         [[SCALAR:%.*]] = load i16, i16* %vs.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[SCALAR:%.*]] = load i16, ptr %vs.addr, align 2{{$}}
+  // OPT:         [[SCALAR:%.*]] = load i16, ptr %vs.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EXT:%.*]] = sext i16 [[SCALAR]] to i32
-  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <27 x i32>, <27 x i32>* %0, align 4{{$}}
-  // OPT-NEXT:    [[MATRIX:%.*]] = load <27 x i32>, <27 x i32>* %0, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <27 x i32>, ptr %a.addr, align 4{{$}}
+  // OPT-NEXT:    [[MATRIX:%.*]] = load <27 x i32>, ptr %a.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EMBED:%.*]] = insertelement <27 x i32> poison, i32 [[SCALAR_EXT:%.*]], i64 0
   // CHECK-NEXT:  [[SCALAR_EMBED1:%.*]] = shufflevector <27 x i32> [[SCALAR_EMBED]], <27 x i32> poison, <27 x i32> zeroinitializer
   // CHECK-NEXT:  [[RES:%.*]] = sub <27 x i32> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:  store <27 x i32> [[RES]], <27 x i32>* [[MAT_ADDR]], align 4
+  // CHECK-NEXT:  store <27 x i32> [[RES]], ptr [[MAT_ADDR]], align 4
 
   a -= vs;
 }
 
 void add_matrix_scalar_int_long_int(ix9x3_t a, long int vli) {
   // CHECK-LABEL: define{{.*}} void @add_matrix_scalar_int_long_int(<27 x i32> noundef %a, i64 noundef %vli)
-  // NOOPT:        [[MATRIX:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4{{$}}
-  // NOOPT-NEXT:   [[SCALAR:%.*]] = load i64, i64* %vli.addr, align 8{{$}}
-  // OPT:          [[MATRIX:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:     [[SCALAR:%.*]] = load i64, i64* %vli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:        [[MATRIX:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4{{$}}
+  // NOOPT-NEXT:   [[SCALAR:%.*]] = load i64, ptr %vli.addr, align 8{{$}}
+  // OPT:          [[MATRIX:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:     [[SCALAR:%.*]] = load i64, ptr %vli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:   [[SCALAR_TRUNC:%.*]] = trunc i64 [[SCALAR]] to i32
   // CHECK-NEXT:   [[SCALAR_EMBED:%.*]] = insertelement <27 x i32> poison, i32 [[SCALAR_TRUNC]], i64 0
   // CHECK-NEXT:   [[SCALAR_EMBED1:%.*]] = shufflevector <27 x i32> [[SCALAR_EMBED]], <27 x i32> poison, <27 x i32> zeroinitializer
   // CHECK-NEXT:   [[RES:%.*]] = add <27 x i32> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:   store <27 x i32> [[RES]], <27 x i32>* [[MAT_ADDR]], align 4
+  // CHECK-NEXT:   store <27 x i32> [[RES]], ptr [[MAT_ADDR]], align 4
 
   a = a + vli;
 }
 
 void add_compound_matrix_scalar_int_long_int(ix9x3_t a, long int vli) {
   // CHECK-LABEL: define{{.*}} void @add_compound_matrix_scalar_int_long_int(<27 x i32> noundef %a, i64 noundef %vli)
-  // NOOPT:       [[SCALAR:%.*]] = load i64, i64* %vli.addr, align 8{{$}}
-  // OPT:         [[SCALAR:%.*]] = load i64, i64* %vli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // CHECK-NEXT:  [[SCALAR_TRUNC:%.*]] = trunc i64 %1 to i32
-  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <27 x i32>, <27 x i32>* %0, align 4{{$}}
-  // OPT-NEXT:    [[MATRIX:%.*]] = load <27 x i32>, <27 x i32>* %0, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[SCALAR:%.*]] = load i64, ptr %vli.addr, align 8{{$}}
+  // OPT:         [[SCALAR:%.*]] = load i64, ptr %vli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // CHECK-NEXT:  [[SCALAR_TRUNC:%.*]] = trunc i64 [[SCALAR]] to i32
+  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <27 x i32>, ptr %a.addr, align 4{{$}}
+  // OPT-NEXT:    [[MATRIX:%.*]] = load <27 x i32>, ptr %a.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EMBED:%.*]] = insertelement <27 x i32> poison, i32 [[SCALAR_TRUNC]], i64 0
   // CHECK-NEXT:  [[SCALAR_EMBED1:%.*]] = shufflevector <27 x i32> [[SCALAR_EMBED]], <27 x i32> poison, <27 x i32> zeroinitializer
   // CHECK-NEXT:  [[RES:%.*]] = add <27 x i32> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:  store <27 x i32> [[RES]], <27 x i32>* [[MAT_ADDR]], align 4
+  // CHECK-NEXT:  store <27 x i32> [[RES]], ptr [[MAT_ADDR]], align 4
 
   a += vli;
 }
 
 void subtract_compound_matrix_scalar_int_long_int(ix9x3_t a, long int vli) {
   // CHECK-LABEL: define{{.*}} void @subtract_compound_matrix_scalar_int_long_int(<27 x i32> noundef %a, i64 noundef %vli)
-  // NOOPT:       [[SCALAR:%.*]] = load i64, i64* %vli.addr, align 8{{$}}
-  // OPT:         [[SCALAR:%.*]] = load i64, i64* %vli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // CHECK-NEXT:  [[SCALAR_TRUNC:%.*]] = trunc i64 %1 to i32
-  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <27 x i32>, <27 x i32>* %0, align 4{{$}}
-  // OPT-NEXT:    [[MATRIX:%.*]] = load <27 x i32>, <27 x i32>* %0, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[SCALAR:%.*]] = load i64, ptr %vli.addr, align 8{{$}}
+  // OPT:         [[SCALAR:%.*]] = load i64, ptr %vli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // CHECK-NEXT:  [[SCALAR_TRUNC:%.*]] = trunc i64 [[SCALAR]] to i32
+  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <27 x i32>, ptr %a.addr, align 4{{$}}
+  // OPT-NEXT:    [[MATRIX:%.*]] = load <27 x i32>, ptr %a.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EMBED:%.*]] = insertelement <27 x i32> poison, i32 [[SCALAR_TRUNC]], i64 0
   // CHECK-NEXT:  [[SCALAR_EMBED1:%.*]] = shufflevector <27 x i32> [[SCALAR_EMBED]], <27 x i32> poison, <27 x i32> zeroinitializer
   // CHECK-NEXT:  [[RES:%.*]] = sub <27 x i32> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:  store <27 x i32> [[RES]], <27 x i32>* [[MAT_ADDR]], align 4
+  // CHECK-NEXT:  store <27 x i32> [[RES]], ptr [[MAT_ADDR]], align 4
 
   a -= vli;
 }
 
 void add_matrix_scalar_int_unsigned_long_long(ix9x3_t a, unsigned long long int vulli) {
   // CHECK-LABEL: define{{.*}} void @add_matrix_scalar_int_unsigned_long_long(<27 x i32> noundef %a, i64 noundef %vulli)
-  // NOOPT:        [[MATRIX:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4{{$}}
-  // NOOPT-NEXT:   [[SCALAR:%.*]] = load i64, i64* %vulli.addr, align 8{{$}}
-  // OPT:          [[MATRIX:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:     [[SCALAR:%.*]] = load i64, i64* %vulli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:        [[MATRIX:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4{{$}}
+  // NOOPT-NEXT:   [[SCALAR:%.*]] = load i64, ptr %vulli.addr, align 8{{$}}
+  // OPT:          [[MATRIX:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:     [[SCALAR:%.*]] = load i64, ptr %vulli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:   [[SCALAR_TRUNC:%.*]] = trunc i64 [[SCALAR]] to i32
   // CHECK-NEXT:   [[SCALAR_EMBED:%.*]] = insertelement <27 x i32> poison, i32 [[SCALAR_TRUNC]], i64 0
   // CHECK-NEXT:   [[SCALAR_EMBED1:%.*]] = shufflevector <27 x i32> [[SCALAR_EMBED]], <27 x i32> poison, <27 x i32> zeroinitializer
   // CHECK-NEXT:   [[RES:%.*]] = add <27 x i32> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:   store <27 x i32> [[RES]], <27 x i32>* [[MAT_ADDR]], align 4
+  // CHECK-NEXT:   store <27 x i32> [[RES]], ptr [[MAT_ADDR]], align 4
 
   a = a + vulli;
 }
 
 void add_compound_matrix_scalar_int_unsigned_long_long(ix9x3_t a, unsigned long long int vulli) {
   // CHECK-LABEL: define{{.*}} void @add_compound_matrix_scalar_int_unsigned_long_long(<27 x i32> noundef %a, i64 noundef %vulli)
-  // NOOPT:        [[SCALAR:%.*]] = load i64, i64* %vulli.addr, align 8{{$}}
-  // OPT:          [[SCALAR:%.*]] = load i64, i64* %vulli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:        [[SCALAR:%.*]] = load i64, ptr %vulli.addr, align 8{{$}}
+  // OPT:          [[SCALAR:%.*]] = load i64, ptr %vulli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:   [[SCALAR_TRUNC:%.*]] = trunc i64 [[SCALAR]] to i32
-  // NOOPT-NEXT:   [[MATRIX:%.*]] = load <27 x i32>, <27 x i32>* [[MATRIX_ADDR:%.*]], align 4{{$}}
-  // OPT-NEXT:     [[MATRIX:%.*]] = load <27 x i32>, <27 x i32>* [[MATRIX_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:   [[MATRIX:%.*]] = load <27 x i32>, ptr [[MATRIX_ADDR:%.*]], align 4{{$}}
+  // OPT-NEXT:     [[MATRIX:%.*]] = load <27 x i32>, ptr [[MATRIX_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:   [[SCALAR_EMBED:%.*]] = insertelement <27 x i32> poison, i32 [[SCALAR_TRUNC]], i64 0
   // CHECK-NEXT:   [[SCALAR_EMBED1:%.*]] = shufflevector <27 x i32> [[SCALAR_EMBED]], <27 x i32> poison, <27 x i32> zeroinitializer
   // CHECK-NEXT:   [[RES:%.*]] = add <27 x i32> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:   store <27 x i32> [[RES]], <27 x i32>* [[MAT_ADDR]], align 4
+  // CHECK-NEXT:   store <27 x i32> [[RES]], ptr [[MAT_ADDR]], align 4
 
   a += vulli;
 }
 
 void subtract_compound_matrix_scalar_int_unsigned_long_long(ix9x3_t a, unsigned long long int vulli) {
   // CHECK-LABEL: define{{.*}} void @subtract_compound_matrix_scalar_int_unsigned_long_long(<27 x i32> noundef %a, i64 noundef %vulli)
-  // NOOPT:        [[SCALAR:%.*]] = load i64, i64* %vulli.addr, align 8{{$}}
-  // OPT:          [[SCALAR:%.*]] = load i64, i64* %vulli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:        [[SCALAR:%.*]] = load i64, ptr %vulli.addr, align 8{{$}}
+  // OPT:          [[SCALAR:%.*]] = load i64, ptr %vulli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:   [[SCALAR_TRUNC:%.*]] = trunc i64 [[SCALAR]] to i32
-  // NOOPT-NEXT:   [[MATRIX:%.*]] = load <27 x i32>, <27 x i32>* [[MATRIX_ADDR:%.*]], align 4{{$}}
-  // OPT-NEXT:     [[MATRIX:%.*]] = load <27 x i32>, <27 x i32>* [[MATRIX_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:   [[MATRIX:%.*]] = load <27 x i32>, ptr [[MATRIX_ADDR:%.*]], align 4{{$}}
+  // OPT-NEXT:     [[MATRIX:%.*]] = load <27 x i32>, ptr [[MATRIX_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:   [[SCALAR_EMBED:%.*]] = insertelement <27 x i32> poison, i32 [[SCALAR_TRUNC]], i64 0
   // CHECK-NEXT:   [[SCALAR_EMBED1:%.*]] = shufflevector <27 x i32> [[SCALAR_EMBED]], <27 x i32> poison, <27 x i32> zeroinitializer
   // CHECK-NEXT:   [[RES:%.*]] = sub <27 x i32> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:   store <27 x i32> [[RES]], <27 x i32>* [[MAT_ADDR]], align 4
+  // CHECK-NEXT:   store <27 x i32> [[RES]], ptr [[MAT_ADDR]], align 4
 
   a -= vulli;
 }
 
 void add_matrix_scalar_long_long_int_short(ullx4x2_t b, short vs) {
   // CHECK-LABEL: define{{.*}} void @add_matrix_scalar_long_long_int_short(<8 x i64> noundef %b, i16 noundef signext %vs)
-  // NOOPT:         [[SCALAR:%.*]] = load i16, i16* %vs.addr, align 2{{$}}
-  // OPT:           [[SCALAR:%.*]] = load i16, i16* %vs.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:         [[SCALAR:%.*]] = load i16, ptr %vs.addr, align 2{{$}}
+  // OPT:           [[SCALAR:%.*]] = load i16, ptr %vs.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[SCALAR_EXT:%.*]] = sext i16 [[SCALAR]] to i64
-  // NOOPT-NEXT:    [[MATRIX:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8{{$}}
-  // OPT-NEXT:      [[MATRIX:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:    [[MATRIX:%.*]] = load <8 x i64>, ptr {{.*}}, align 8{{$}}
+  // OPT-NEXT:      [[MATRIX:%.*]] = load <8 x i64>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[SCALAR_EMBED:%.*]] = insertelement <8 x i64> poison, i64 [[SCALAR_EXT]], i64 0
   // CHECK-NEXT:    [[SCALAR_EMBED1:%.*]] = shufflevector <8 x i64> [[SCALAR_EMBED]], <8 x i64> poison, <8 x i32> zeroinitializer
   // CHECK-NEXT:    [[RES:%.*]] = add <8 x i64> [[SCALAR_EMBED1]], [[MATRIX]]
-  // CHECK-NEXT:    store <8 x i64> [[RES]], <8 x i64>* {{.*}}, align 8
+  // CHECK-NEXT:    store <8 x i64> [[RES]], ptr {{.*}}, align 8
 
   b = vs + b;
 }
 
 void add_compound_matrix_scalar_long_long_int_short(ullx4x2_t b, short vs) {
   // CHECK-LABEL: define{{.*}} void @add_compound_matrix_scalar_long_long_int_short(<8 x i64> noundef %b, i16 noundef signext %vs)
-  // NOOPT:       [[SCALAR:%.*]] = load i16, i16* %vs.addr, align 2{{$}}
-  // OPT:         [[SCALAR:%.*]] = load i16, i16* %vs.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[SCALAR:%.*]] = load i16, ptr %vs.addr, align 2{{$}}
+  // OPT:         [[SCALAR:%.*]] = load i16, ptr %vs.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EXT:%.*]] = sext i16 [[SCALAR]] to i64
-  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <8 x i64>, <8 x i64>* %0, align 8{{$}}
-  // OPT-NEXT:    [[MATRIX:%.*]] = load <8 x i64>, <8 x i64>* %0, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <8 x i64>, ptr %b.addr, align 8{{$}}
+  // OPT-NEXT:    [[MATRIX:%.*]] = load <8 x i64>, ptr %b.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EMBED:%.*]] = insertelement <8 x i64> poison, i64 [[SCALAR_EXT]], i64 0
   // CHECK-NEXT:  [[SCALAR_EMBED1:%.*]] = shufflevector <8 x i64> [[SCALAR_EMBED]], <8 x i64> poison, <8 x i32> zeroinitializer
   // CHECK-NEXT:  [[RES:%.*]] = add <8 x i64> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:  store <8 x i64> [[RES]], <8 x i64>* {{.*}}, align 8
+  // CHECK-NEXT:  store <8 x i64> [[RES]], ptr {{.*}}, align 8
 
   b += vs;
 }
 
 void subtract_compound_matrix_scalar_long_long_int_short(ullx4x2_t b, short vs) {
   // CHECK-LABEL: define{{.*}} void @subtract_compound_matrix_scalar_long_long_int_short(<8 x i64> noundef %b, i16 noundef signext %vs)
-  // NOOPT:       [[SCALAR:%.*]] = load i16, i16* %vs.addr, align 2{{$}}
-  // OPT:         [[SCALAR:%.*]] = load i16, i16* %vs.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[SCALAR:%.*]] = load i16, ptr %vs.addr, align 2{{$}}
+  // OPT:         [[SCALAR:%.*]] = load i16, ptr %vs.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EXT:%.*]] = sext i16 [[SCALAR]] to i64
-  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <8 x i64>, <8 x i64>* %0, align 8{{$}}
-  // OPT-NEXT:    [[MATRIX:%.*]] = load <8 x i64>, <8 x i64>* %0, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:  [[MATRIX:%.*]] = load <8 x i64>, ptr %b.addr, align 8{{$}}
+  // OPT-NEXT:    [[MATRIX:%.*]] = load <8 x i64>, ptr %b.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[SCALAR_EMBED:%.*]] = insertelement <8 x i64> poison, i64 [[SCALAR_EXT]], i64 0
   // CHECK-NEXT:  [[SCALAR_EMBED1:%.*]] = shufflevector <8 x i64> [[SCALAR_EMBED]], <8 x i64> poison, <8 x i32> zeroinitializer
   // CHECK-NEXT:  [[RES:%.*]] = sub <8 x i64> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:  store <8 x i64> [[RES]], <8 x i64>* {{.*}}, align 8
+  // CHECK-NEXT:  store <8 x i64> [[RES]], ptr {{.*}}, align 8
 
   b -= vs;
 }
 
 void add_matrix_scalar_long_long_int_int(ullx4x2_t b, long int vli) {
   // CHECK-LABEL: define{{.*}} void @add_matrix_scalar_long_long_int_int(<8 x i64> noundef %b, i64 noundef %vli)
-  // NOOPT:         [[SCALAR:%.*]] = load i64, i64* %vli.addr, align 8{{$}}
-  // NOOPT-NEXT:    [[MATRIX:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8{{$}}
-  // OPT:           [[SCALAR:%.*]] = load i64, i64* %vli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:      [[MATRIX:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:         [[SCALAR:%.*]] = load i64, ptr %vli.addr, align 8{{$}}
+  // NOOPT-NEXT:    [[MATRIX:%.*]] = load <8 x i64>, ptr {{.*}}, align 8{{$}}
+  // OPT:           [[SCALAR:%.*]] = load i64, ptr %vli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:      [[MATRIX:%.*]] = load <8 x i64>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[SCALAR_EMBED:%.*]] = insertelement <8 x i64> poison, i64 [[SCALAR]], i64 0
   // CHECK-NEXT:    [[SCALAR_EMBED1:%.*]] = shufflevector <8 x i64> [[SCALAR_EMBED]], <8 x i64> poison, <8 x i32> zeroinitializer
   // CHECK-NEXT:    [[RES:%.*]] = add <8 x i64> [[SCALAR_EMBED1]], [[MATRIX]]
-  // CHECK-NEXT:    store <8 x i64> [[RES]], <8 x i64>* {{.*}}, align 8
+  // CHECK-NEXT:    store <8 x i64> [[RES]], ptr {{.*}}, align 8
 
   b = vli + b;
 }
 
 void add_compound_matrix_scalar_long_long_int_int(ullx4x2_t b, long int vli) {
   // CHECK-LABEL: define{{.*}} void @add_compound_matrix_scalar_long_long_int_int(<8 x i64> noundef %b, i64 noundef %vli)
-  // NOOPT:        [[SCALAR:%.*]] = load i64, i64* %vli.addr, align 8{{$}}
-  // NOOPT-NEXT:   [[MATRIX:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8{{$}}
-  // OPT:          [[SCALAR:%.*]] = load i64, i64* %vli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:     [[MATRIX:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:        [[SCALAR:%.*]] = load i64, ptr %vli.addr, align 8{{$}}
+  // NOOPT-NEXT:   [[MATRIX:%.*]] = load <8 x i64>, ptr {{.*}}, align 8{{$}}
+  // OPT:          [[SCALAR:%.*]] = load i64, ptr %vli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:     [[MATRIX:%.*]] = load <8 x i64>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:   [[SCALAR_EMBED:%.*]] = insertelement <8 x i64> poison, i64 [[SCALAR]], i64 0
   // CHECK-NEXT:   [[SCALAR_EMBED1:%.*]] = shufflevector <8 x i64> [[SCALAR_EMBED]], <8 x i64> poison, <8 x i32> zeroinitializer
   // CHECK-NEXT:   [[RES:%.*]] = add <8 x i64> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:   store <8 x i64> [[RES]], <8 x i64>* {{.*}}, align 8
+  // CHECK-NEXT:   store <8 x i64> [[RES]], ptr {{.*}}, align 8
 
   b += vli;
 }
 
 void subtract_compound_matrix_scalar_long_long_int_int(ullx4x2_t b, long int vli) {
   // CHECK-LABEL: define{{.*}} void @subtract_compound_matrix_scalar_long_long_int_int(<8 x i64> noundef %b, i64 noundef %vli)
-  // NOOPT:        [[SCALAR:%.*]] = load i64, i64* %vli.addr, align 8{{$}}
-  // OPT:          [[SCALAR:%.*]] = load i64, i64* %vli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // NOOPT-NEXT:   [[MATRIX:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8{{$}}
-  // OPT-NEXT:     [[MATRIX:%.*]] = load <8 x i64>, <8 x i64>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:        [[SCALAR:%.*]] = load i64, ptr %vli.addr, align 8{{$}}
+  // OPT:          [[SCALAR:%.*]] = load i64, ptr %vli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:   [[MATRIX:%.*]] = load <8 x i64>, ptr {{.*}}, align 8{{$}}
+  // OPT-NEXT:     [[MATRIX:%.*]] = load <8 x i64>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:   [[SCALAR_EMBED:%.*]] = insertelement <8 x i64> poison, i64 [[SCALAR]], i64 0
   // CHECK-NEXT:   [[SCALAR_EMBED1:%.*]] = shufflevector <8 x i64> [[SCALAR_EMBED]], <8 x i64> poison, <8 x i32> zeroinitializer
   // CHECK-NEXT:   [[RES:%.*]] = sub <8 x i64> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:   store <8 x i64> [[RES]], <8 x i64>* {{.*}}, align 8
+  // CHECK-NEXT:   store <8 x i64> [[RES]], ptr {{.*}}, align 8
 
   b -= vli;
 }
 
 void add_matrix_scalar_long_long_int_unsigned_long_long(ullx4x2_t b, unsigned long long int vulli) {
   // CHECK-LABEL: define{{.*}} void @add_matrix_scalar_long_long_int_unsigned_long_long
-  // NOOPT:        [[SCALAR:%.*]] = load i64, i64* %vulli.addr, align 8{{$}}
-  // NOOPT-NEXT:   [[MATRIX:%.*]] = load <8 x i64>, <8 x i64>* %0, align 8{{$}}
-  // OPT:          [[SCALAR:%.*]] = load i64, i64* %vulli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:     [[MATRIX:%.*]] = load <8 x i64>, <8 x i64>* %0, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:        [[SCALAR:%.*]] = load i64, ptr %vulli.addr, align 8{{$}}
+  // NOOPT-NEXT:   [[MATRIX:%.*]] = load <8 x i64>, ptr %b.addr, align 8{{$}}
+  // OPT:          [[SCALAR:%.*]] = load i64, ptr %vulli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:     [[MATRIX:%.*]] = load <8 x i64>, ptr %b.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:   [[SCALAR_EMBED:%.*]] = insertelement <8 x i64> poison, i64 [[SCALAR]], i64 0
   // CHECK-NEXT:   [[SCALAR_EMBED1:%.*]] = shufflevector <8 x i64> [[SCALAR_EMBED]], <8 x i64> poison, <8 x i32> zeroinitializer
   // CHECK-NEXT:   [[RES:%.*]] = add <8 x i64> [[SCALAR_EMBED1]], [[MATRIX]]
-  // CHECK-NEXT:   store <8 x i64> [[RES]], <8 x i64>* {{.*}}, align 8
+  // CHECK-NEXT:   store <8 x i64> [[RES]], ptr {{.*}}, align 8
   b = vulli + b;
 }
 
 void add_compound_matrix_scalar_long_long_int_unsigned_long_long(ullx4x2_t b, unsigned long long int vulli) {
   // CHECK-LABEL: define{{.*}} void @add_compound_matrix_scalar_long_long_int_unsigned_long_long
-  // NOOPT:        [[SCALAR:%.*]] = load i64, i64* %vulli.addr, align 8{{$}}
-  // NOOPT-NEXT:   [[MATRIX:%.*]] = load <8 x i64>, <8 x i64>* %0, align 8{{$}}
-  // OPT:          [[SCALAR:%.*]] = load i64, i64* %vulli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:     [[MATRIX:%.*]] = load <8 x i64>, <8 x i64>* %0, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:        [[SCALAR:%.*]] = load i64, ptr %vulli.addr, align 8{{$}}
+  // NOOPT-NEXT:   [[MATRIX:%.*]] = load <8 x i64>, ptr %b.addr, align 8{{$}}
+  // OPT:          [[SCALAR:%.*]] = load i64, ptr %vulli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:     [[MATRIX:%.*]] = load <8 x i64>, ptr %b.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:   [[SCALAR_EMBED:%.*]] = insertelement <8 x i64> poison, i64 [[SCALAR]], i64 0
   // CHECK-NEXT:   [[SCALAR_EMBED1:%.*]] = shufflevector <8 x i64> [[SCALAR_EMBED]], <8 x i64> poison, <8 x i32> zeroinitializer
   // CHECK-NEXT:   [[RES:%.*]] = add <8 x i64> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:   store <8 x i64> [[RES]], <8 x i64>* {{.*}}, align 8
+  // CHECK-NEXT:   store <8 x i64> [[RES]], ptr {{.*}}, align 8
 
   b += vulli;
 }
 
 void subtract_compound_matrix_scalar_long_long_int_unsigned_long_long(ullx4x2_t b, unsigned long long int vulli) {
   // CHECK-LABEL: define{{.*}} void @subtract_compound_matrix_scalar_long_long_int_unsigned_long_long
-  // NOOPT:        [[SCALAR:%.*]] = load i64, i64* %vulli.addr, align 8{{$}}
-  // NOOPT-NEXT:   [[MATRIX:%.*]] = load <8 x i64>, <8 x i64>* %0, align 8{{$}}
-  // OPT:          [[SCALAR:%.*]] = load i64, i64* %vulli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:     [[MATRIX:%.*]] = load <8 x i64>, <8 x i64>* %0, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:        [[SCALAR:%.*]] = load i64, ptr %vulli.addr, align 8{{$}}
+  // NOOPT-NEXT:   [[MATRIX:%.*]] = load <8 x i64>, ptr %b.addr, align 8{{$}}
+  // OPT:          [[SCALAR:%.*]] = load i64, ptr %vulli.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:     [[MATRIX:%.*]] = load <8 x i64>, ptr %b.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:   [[SCALAR_EMBED:%.*]] = insertelement <8 x i64> poison, i64 [[SCALAR]], i64 0
   // CHECK-NEXT:   [[SCALAR_EMBED1:%.*]] = shufflevector <8 x i64> [[SCALAR_EMBED]], <8 x i64> poison, <8 x i32> zeroinitializer
   // CHECK-NEXT:   [[RES:%.*]] = sub <8 x i64> [[MATRIX]], [[SCALAR_EMBED1]]
-  // CHECK-NEXT:   store <8 x i64> [[RES]], <8 x i64>* {{.*}}, align 8
+  // CHECK-NEXT:   store <8 x i64> [[RES]], ptr {{.*}}, align 8
 
   b -= vulli;
 }
@@ -587,13 +587,12 @@ void subtract_compound_matrix_scalar_long_long_int_unsigned_long_long(ullx4x2_t
 
 void multiply_matrix_matrix_double(dx5x5_t b, dx5x5_t c) {
   // CHECK-LABEL: @multiply_matrix_matrix_double(
-  // NOOPT:         [[B:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-  // NOOPT-NEXT:    [[C:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-  // OPT:           [[B:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:      [[C:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:         [[B:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+  // NOOPT-NEXT:    [[C:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+  // OPT:           [[B:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:      [[C:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[RES:%.*]] = call <25 x double> @llvm.matrix.multiply.v25f64.v25f64.v25f64(<25 x double> [[B]], <25 x double> [[C]], i32 5, i32 5, i32 5)
-  // CHECK-NEXT:    [[A_ADDR:%.*]] = bitcast [25 x double]* %a to <25 x double>*
-  // CHECK-NEXT:    store <25 x double> [[RES]], <25 x double>* [[A_ADDR]], align 8
+  // CHECK-NEXT:    store <25 x double> [[RES]], ptr %a, align 8
   // CHECK:         ret void
   //
 
@@ -603,12 +602,12 @@ void multiply_matrix_matrix_double(dx5x5_t b, dx5x5_t c) {
 
 void multiply_compound_matrix_matrix_double(dx5x5_t b, dx5x5_t c) {
   // CHECK-LABEL: @multiply_compound_matrix_matrix_double(
-  // NOOPT:        [[C:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-  // NOOPT-NEXT:   [[B:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-  // OPT:          [[C:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:     [[B:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:        [[C:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+  // NOOPT-NEXT:   [[B:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+  // OPT:          [[C:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:     [[B:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:   [[RES:%.*]] = call <25 x double> @llvm.matrix.multiply.v25f64.v25f64.v25f64(<25 x double> [[B]], <25 x double> [[C]], i32 5, i32 5, i32 5)
-  // CHECK-NEXT:   store <25 x double> [[RES]], <25 x double>* {{.*}}, align 8
+  // CHECK-NEXT:   store <25 x double> [[RES]], ptr {{.*}}, align 8
   // CHECK-NEXT:   ret void
   b *= c;
 }
@@ -616,13 +615,12 @@ void multiply_compound_matrix_matrix_double(dx5x5_t b, dx5x5_t c) {
 typedef int ix3x9_t __attribute__((matrix_type(3, 9)));
 typedef int ix9x9_t __attribute__((matrix_type(9, 9)));
 // CHECK-LABEL: @multiply_matrix_matrix_int(
-// NOOPT:         [[B:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4{{$}}
-// NOOPT-NEXT:    [[C:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4{{$}}
-// OPT:           [[B:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
-// OPT-NEXT:      [[C:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT:         [[B:%.*]] = load <27 x i32>, ptr {{.*}}, align 4{{$}}
+// NOOPT-NEXT:    [[C:%.*]] = load <27 x i32>, ptr {{.*}}, align 4{{$}}
+// OPT:           [[B:%.*]] = load <27 x i32>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+// OPT-NEXT:      [[C:%.*]] = load <27 x i32>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[RES:%.*]] = call <81 x i32> @llvm.matrix.multiply.v81i32.v27i32.v27i32(<27 x i32> [[B]], <27 x i32> [[C]], i32 9, i32 3, i32 9)
-// CHECK-NEXT:    [[A_ADDR:%.*]] = bitcast [81 x i32]* %a to <81 x i32>*
-// CHECK-NEXT:    store <81 x i32> [[RES]], <81 x i32>* [[A_ADDR]], align 4
+// CHECK-NEXT:    store <81 x i32> [[RES]], ptr %a, align 4
 // CHECK:         ret void
 //
 void multiply_matrix_matrix_int(ix9x3_t b, ix3x9_t c) {
@@ -631,15 +629,15 @@ void multiply_matrix_matrix_int(ix9x3_t b, ix3x9_t c) {
 }
 
 // CHECK-LABEL: @multiply_double_matrix_scalar_float(
-// NOOPT:         [[A:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-// NOOPT-NEXT:    [[S:%.*]] = load float, float* %s.addr, align 4{{$}}
-// OPT:           [[A:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
-// OPT-NEXT:      [[S:%.*]] = load float, float* %s.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT:         [[A:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+// NOOPT-NEXT:    [[S:%.*]] = load float, ptr %s.addr, align 4{{$}}
+// OPT:           [[A:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+// OPT-NEXT:      [[S:%.*]] = load float, ptr %s.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[S_EXT:%.*]] = fpext float [[S]] to double
 // CHECK-NEXT:    [[VECINSERT:%.*]] = insertelement <25 x double> poison, double [[S_EXT]], i64 0
 // CHECK-NEXT:    [[VECSPLAT:%.*]] = shufflevector <25 x double> [[VECINSERT]], <25 x double> poison, <25 x i32> zeroinitializer
 // CHECK-NEXT:    [[RES:%.*]] = fmul <25 x double> [[A]], [[VECSPLAT]]
-// CHECK-NEXT:    store <25 x double> [[RES]], <25 x double>* {{.*}}, align 8
+// CHECK-NEXT:    store <25 x double> [[RES]], ptr {{.*}}, align 8
 // CHECK-NEXT:    ret void
 //
 void multiply_double_matrix_scalar_float(dx5x5_t a, float s) {
@@ -647,15 +645,15 @@ void multiply_double_matrix_scalar_float(dx5x5_t a, float s) {
 }
 
 // CHECK-LABEL: @multiply_compound_double_matrix_scalar_float
-// NOOPT:         [[S:%.*]] = load float, float* %s.addr, align 4{{$}}
-// OPT:           [[S:%.*]] = load float, float* %s.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT:         [[S:%.*]] = load float, ptr %s.addr, align 4{{$}}
+// OPT:           [[S:%.*]] = load float, ptr %s.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[S_EXT:%.*]] = fpext float [[S]] to double
-// NOOPT-NEXT:    [[A:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-// OPT-NEXT:      [[A:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT-NEXT:    [[A:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+// OPT-NEXT:      [[A:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[VECINSERT:%.*]] = insertelement <25 x double> poison, double [[S_EXT]], i64 0
 // CHECK-NEXT:    [[VECSPLAT:%.*]] = shufflevector <25 x double> [[VECINSERT]], <25 x double> poison, <25 x i32> zeroinitializer
 // CHECK-NEXT:    [[RES:%.*]] = fmul <25 x double> [[A]], [[VECSPLAT]]
-// CHECK-NEXT:    store <25 x double> [[RES]], <25 x double>* {{.*}}, align 8
+// CHECK-NEXT:    store <25 x double> [[RES]], ptr {{.*}}, align 8
 // CHECK-NEXT:    ret void
 //
 void multiply_compound_double_matrix_scalar_float(dx5x5_t a, float s) {
@@ -663,14 +661,14 @@ void multiply_compound_double_matrix_scalar_float(dx5x5_t a, float s) {
 }
 
 // CHECK-LABEL: @multiply_double_matrix_scalar_double(
-// NOOPT:         [[A:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-// NOOPT-NEXT:    [[S:%.*]] = load double, double* %s.addr, align 8{{$}}
-// OPT:           [[A:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
-// OPT-NEXT:      [[S:%.*]] = load double, double* %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT:         [[A:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+// NOOPT-NEXT:    [[S:%.*]] = load double, ptr %s.addr, align 8{{$}}
+// OPT:           [[A:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+// OPT-NEXT:      [[S:%.*]] = load double, ptr %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[VECINSERT:%.*]] = insertelement <25 x double> poison, double [[S]], i64 0
 // CHECK-NEXT:    [[VECSPLAT:%.*]] = shufflevector <25 x double> [[VECINSERT]], <25 x double> poison, <25 x i32> zeroinitializer
 // CHECK-NEXT:    [[RES:%.*]] = fmul <25 x double> [[A]], [[VECSPLAT]]
-// CHECK-NEXT:    store <25 x double> [[RES]], <25 x double>* {{.*}}, align 8
+// CHECK-NEXT:    store <25 x double> [[RES]], ptr {{.*}}, align 8
 // CHECK-NEXT:    ret void
 //
 void multiply_double_matrix_scalar_double(dx5x5_t a, double s) {
@@ -678,29 +676,29 @@ void multiply_double_matrix_scalar_double(dx5x5_t a, double s) {
 }
 
 // CHECK-LABEL: @multiply_compound_double_matrix_scalar_double(
-// NOOPT:         [[S:%.*]] = load double, double* %s.addr, align 8{{$}}
-// NOOPT-NEXT:    [[A:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-// OPT:           [[S:%.*]] = load double, double* %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
-// OPT-NEXT:      [[A:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT:         [[S:%.*]] = load double, ptr %s.addr, align 8{{$}}
+// NOOPT-NEXT:    [[A:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+// OPT:           [[S:%.*]] = load double, ptr %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+// OPT-NEXT:      [[A:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[VECINSERT:%.*]] = insertelement <25 x double> poison, double [[S]], i64 0
 // CHECK-NEXT:    [[VECSPLAT:%.*]] = shufflevector <25 x double> [[VECINSERT]], <25 x double> poison, <25 x i32> zeroinitializer
 // CHECK-NEXT:    [[RES:%.*]] = fmul <25 x double> [[A]], [[VECSPLAT]]
-// CHECK-NEXT:    store <25 x double> [[RES]], <25 x double>* {{.*}}, align 8
+// CHECK-NEXT:    store <25 x double> [[RES]], ptr {{.*}}, align 8
 // CHECK-NEXT:    ret void
 void multiply_compound_double_matrix_scalar_double(dx5x5_t a, double s) {
   a *= s;
 }
 
 // CHECK-LABEL: @multiply_float_matrix_scalar_double(
-// NOOPT:         [[S:%.*]] = load double, double* %s.addr, align 8{{$}}
-// OPT:           [[S:%.*]] = load double, double* %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT:         [[S:%.*]] = load double, ptr %s.addr, align 8{{$}}
+// OPT:           [[S:%.*]] = load double, ptr %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[S_TRUNC:%.*]] = fptrunc double [[S]] to float
-// NOOPT-NEXT:    [[MAT:%.*]] = load <6 x float>, <6 x float>* [[MAT_ADDR:%.*]], align 4{{$}}
-// OPT-NEXT:      [[MAT:%.*]] = load <6 x float>, <6 x float>* [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT-NEXT:    [[MAT:%.*]] = load <6 x float>, ptr [[MAT_ADDR:%.*]], align 4{{$}}
+// OPT-NEXT:      [[MAT:%.*]] = load <6 x float>, ptr [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[VECINSERT:%.*]] = insertelement <6 x float> poison, float [[S_TRUNC]], i64 0
 // CHECK-NEXT:    [[VECSPLAT:%.*]] = shufflevector <6 x float> [[VECINSERT]], <6 x float> poison, <6 x i32> zeroinitializer
 // CHECK-NEXT:    [[RES:%.*]] = fmul <6 x float> [[VECSPLAT]], [[MAT]]
-// CHECK-NEXT:    store <6 x float> [[RES]], <6 x float>* [[MAT_ADDR]], align 4
+// CHECK-NEXT:    store <6 x float> [[RES]], ptr [[MAT_ADDR]], align 4
 // CHECK-NEXT:    ret void
 //
 void multiply_float_matrix_scalar_double(fx2x3_t b, double s) {
@@ -708,30 +706,30 @@ void multiply_float_matrix_scalar_double(fx2x3_t b, double s) {
 }
 
 // CHECK-LABEL: @multiply_compound_float_matrix_scalar_double(
-// NOOPT:         [[S:%.*]] = load double, double* %s.addr, align 8{{$}}
-// OPT:           [[S:%.*]] = load double, double* %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT:         [[S:%.*]] = load double, ptr %s.addr, align 8{{$}}
+// OPT:           [[S:%.*]] = load double, ptr %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[S_TRUNC:%.*]] = fptrunc double [[S]] to float
-// NOOPT-NEXT:    [[MAT:%.*]] = load <6 x float>, <6 x float>* [[MAT_ADDR:%.*]], align 4{{$}}
-// OPT-NEXT:      [[MAT:%.*]] = load <6 x float>, <6 x float>* [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT-NEXT:    [[MAT:%.*]] = load <6 x float>, ptr [[MAT_ADDR:%.*]], align 4{{$}}
+// OPT-NEXT:      [[MAT:%.*]] = load <6 x float>, ptr [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[VECINSERT:%.*]] = insertelement <6 x float> poison, float [[S_TRUNC]], i64 0
 // CHECK-NEXT:    [[VECSPLAT:%.*]] = shufflevector <6 x float> [[VECINSERT]], <6 x float> poison, <6 x i32> zeroinitializer
 // CHECK-NEXT:    [[RES:%.*]] = fmul <6 x float> [[MAT]], [[VECSPLAT]]
-// store <6 x float> %3, <6 x float>* %0, align 4
+// store <6 x float> %3, ptr [[MAT_ADDR]], align 4
 // ret void
 void multiply_compound_float_matrix_scalar_double(fx2x3_t b, double s) {
   b *= s;
 }
 
 // CHECK-LABEL: @multiply_int_matrix_scalar_short(
-// NOOPT:         [[S:%.*]] = load i16, i16* %s.addr, align 2{{$}}
-// OPT:           [[S:%.*]] = load i16, i16* %s.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT:         [[S:%.*]] = load i16, ptr %s.addr, align 2{{$}}
+// OPT:           [[S:%.*]] = load i16, ptr %s.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[S_EXT:%.*]] = sext i16 [[S]] to i32
-// NOOPT-NEXT:    [[MAT:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4{{$}}
-// OPT-NEXT:      [[MAT:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT-NEXT:    [[MAT:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4{{$}}
+// OPT-NEXT:      [[MAT:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[VECINSERT:%.*]] = insertelement <27 x i32> poison, i32 [[S_EXT]], i64 0
 // CHECK-NEXT:    [[VECSPLAT:%.*]] = shufflevector <27 x i32> [[VECINSERT]], <27 x i32> poison, <27 x i32> zeroinitializer
 // CHECK-NEXT:    [[RES:%.*]] = mul <27 x i32> [[VECSPLAT]], [[MAT]]
-// CHECK-NEXT:    store <27 x i32> [[RES]], <27 x i32>* [[MAT_ADDR]], align 4
+// CHECK-NEXT:    store <27 x i32> [[RES]], ptr [[MAT_ADDR]], align 4
 // CHECK-NEXT:    ret void
 //
 void multiply_int_matrix_scalar_short(ix9x3_t b, short s) {
@@ -739,15 +737,15 @@ void multiply_int_matrix_scalar_short(ix9x3_t b, short s) {
 }
 
 // CHECK-LABEL: @multiply_compound_int_matrix_scalar_short(
-// NOOPT:        [[S:%.*]] = load i16, i16* %s.addr, align 2{{$}}
-// OPT:          [[S:%.*]] = load i16, i16* %s.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT:        [[S:%.*]] = load i16, ptr %s.addr, align 2{{$}}
+// OPT:          [[S:%.*]] = load i16, ptr %s.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:   [[S_EXT:%.*]] = sext i16 [[S]] to i32
-// NOOPT-NEXT:   [[MAT:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4{{$}}
-// OPT-NEXT:     [[MAT:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT-NEXT:   [[MAT:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4{{$}}
+// OPT-NEXT:     [[MAT:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:   [[VECINSERT:%.*]] = insertelement <27 x i32> poison, i32 [[S_EXT]], i64 0
 // CHECK-NEXT:   [[VECSPLAT:%.*]] = shufflevector <27 x i32> [[VECINSERT]], <27 x i32> poison, <27 x i32> zeroinitializer
 // CHECK-NEXT:   [[RES:%.*]] = mul <27 x i32> [[MAT]], [[VECSPLAT]]
-// CHECK-NEXT:   store <27 x i32> [[RES]], <27 x i32>* [[MAT_ADDR]], align 4
+// CHECK-NEXT:   store <27 x i32> [[RES]], ptr [[MAT_ADDR]], align 4
 // CHECK-NEXT:   ret void
 //
 void multiply_compound_int_matrix_scalar_short(ix9x3_t b, short s) {
@@ -755,15 +753,15 @@ void multiply_compound_int_matrix_scalar_short(ix9x3_t b, short s) {
 }
 
 // CHECK-LABEL: @multiply_int_matrix_scalar_ull(
-// NOOPT:         [[MAT:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4{{$}}
-// OPT:           [[MAT:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
-// NOOPT-NEXT:    [[S:%.*]] = load i64, i64* %s.addr, align 8{{$}}
-// OPT-NEXT:      [[S:%.*]] = load i64, i64* %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT:         [[MAT:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4{{$}}
+// OPT:           [[MAT:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT-NEXT:    [[S:%.*]] = load i64, ptr %s.addr, align 8{{$}}
+// OPT-NEXT:      [[S:%.*]] = load i64, ptr %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[S_TRUNC:%.*]] = trunc i64 [[S]] to i32
 // CHECK-NEXT:    [[VECINSERT:%.*]] = insertelement <27 x i32> poison, i32 [[S_TRUNC]], i64 0
 // CHECK-NEXT:    [[VECSPLAT:%.*]] = shufflevector <27 x i32> [[VECINSERT]], <27 x i32> poison, <27 x i32> zeroinitializer
 // CHECK-NEXT:    [[RES:%.*]] = mul <27 x i32> [[MAT]], [[VECSPLAT]]
-// CHECK-NEXT:    store <27 x i32> [[RES]], <27 x i32>* [[MAT_ADDR]], align 4
+// CHECK-NEXT:    store <27 x i32> [[RES]], ptr [[MAT_ADDR]], align 4
 // CHECK-NEXT:    ret void
 //
 void multiply_int_matrix_scalar_ull(ix9x3_t b, unsigned long long s) {
@@ -772,15 +770,15 @@ void multiply_int_matrix_scalar_ull(ix9x3_t b, unsigned long long s) {
 
 void multiply_compound_int_matrix_scalar_ull(ix9x3_t b, unsigned long long s) {
   // CHECK-LABEL: @multiply_compound_int_matrix_scalar_ull(
-  // NOOPT:         [[S:%.*]] = load i64, i64* %s.addr, align 8{{$}}
-  // OPT:           [[S:%.*]] = load i64, i64* %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:         [[S:%.*]] = load i64, ptr %s.addr, align 8{{$}}
+  // OPT:           [[S:%.*]] = load i64, ptr %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[S_TRUNC:%.*]] = trunc i64 [[S]] to i32
-  // NOOPT-NEXT:    [[MAT:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4{{$}}
-  // OPT-NEXT:      [[MAT:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:    [[MAT:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4{{$}}
+  // OPT-NEXT:      [[MAT:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[VECINSERT:%.*]] = insertelement <27 x i32> poison, i32 [[S_TRUNC]], i64 0
   // CHECK-NEXT:    [[VECSPLAT:%.*]] = shufflevector <27 x i32> [[VECINSERT]], <27 x i32> poison, <27 x i32> zeroinitializer
   // CHECK-NEXT:    [[RES:%.*]] = mul <27 x i32> [[MAT]], [[VECSPLAT]]
-  // CHECK-NEXT:    store <27 x i32> [[RES]], <27 x i32>* [[MAT_ADDR]], align 4
+  // CHECK-NEXT:    store <27 x i32> [[RES]], ptr [[MAT_ADDR]], align 4
   // CHECK-NEXT:    ret void
 
   b *= s;
@@ -789,12 +787,11 @@ void multiply_compound_int_matrix_scalar_ull(ix9x3_t b, unsigned long long s) {
 // CHECK-LABEL: @multiply_float_matrix_constant(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[A_ADDR:%.*]] = alloca [6 x float], align 4
-// CHECK-NEXT:    [[MAT_ADDR:%.*]] = bitcast [6 x float]* [[A_ADDR]] to <6 x float>*
-// CHECK-NEXT:    store <6 x float> [[A:%.*]], <6 x float>* [[MAT_ADDR]], align 4
-// NOOPT-NEXT:    [[MAT:%.*]] = load <6 x float>, <6 x float>* [[MAT_ADDR]], align 4{{$}}
-// OPT-NEXT:      [[MAT:%.*]] = load <6 x float>, <6 x float>* [[MAT_ADDR]], align 4, !tbaa !{{[0-9]+}}{{$}}
+// CHECK-NEXT:    store <6 x float> [[A:%.*]], ptr [[A_ADDR]], align 4
+// NOOPT-NEXT:    [[MAT:%.*]] = load <6 x float>, ptr [[A_ADDR]], align 4{{$}}
+// OPT-NEXT:      [[MAT:%.*]] = load <6 x float>, ptr [[A_ADDR]], align 4, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[RES:%.*]] = fmul <6 x float> [[MAT]], <float 2.500000e+00, float 2.500000e+00, float 2.500000e+00, float 2.500000e+00, float 2.500000e+00, float 2.500000e+00>
-// CHECK-NEXT:    store <6 x float> [[RES]], <6 x float>* [[MAT_ADDR]], align 4
+// CHECK-NEXT:    store <6 x float> [[RES]], ptr [[A_ADDR]], align 4
 // CHECK-NEXT:    ret void
 //
 void multiply_float_matrix_constant(fx2x3_t a) {
@@ -804,12 +801,11 @@ void multiply_float_matrix_constant(fx2x3_t a) {
 // CHECK-LABEL: @multiply_compound_float_matrix_constant(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[A_ADDR:%.*]] = alloca [6 x float], align 4
-// CHECK-NEXT:    [[MAT_ADDR:%.*]] = bitcast [6 x float]* [[A_ADDR]] to <6 x float>*
-// CHECK-NEXT:    store <6 x float> [[A:%.*]], <6 x float>* [[MAT_ADDR]], align 4
-// NOOPT-NEXT:    [[MAT:%.*]] = load <6 x float>, <6 x float>* [[MAT_ADDR]], align 4{{$}}
-// OPT-NEXT:      [[MAT:%.*]] = load <6 x float>, <6 x float>* [[MAT_ADDR]], align 4, !tbaa !{{[0-9]+}}{{$}}
+// CHECK-NEXT:    store <6 x float> [[A:%.*]], ptr [[A_ADDR]], align 4
+// NOOPT-NEXT:    [[MAT:%.*]] = load <6 x float>, ptr [[A_ADDR]], align 4{{$}}
+// OPT-NEXT:      [[MAT:%.*]] = load <6 x float>, ptr [[A_ADDR]], align 4, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[RES:%.*]] = fmul <6 x float> [[MAT]], <float 2.500000e+00, float 2.500000e+00, float 2.500000e+00, float 2.500000e+00, float 2.500000e+00, float 2.500000e+00>
-// CHECK-NEXT:    store <6 x float> [[RES]], <6 x float>* [[MAT_ADDR]], align 4
+// CHECK-NEXT:    store <6 x float> [[RES]], ptr [[A_ADDR]], align 4
 // CHECK-NEXT:    ret void
 void multiply_compound_float_matrix_constant(fx2x3_t a) {
   a *= 2.5;
@@ -818,12 +814,11 @@ void multiply_compound_float_matrix_constant(fx2x3_t a) {
 // CHECK-LABEL: @multiply_int_matrix_constant(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[A_ADDR:%.*]] = alloca [27 x i32], align 4
-// CHECK-NEXT:    [[MAT_ADDR:%.*]] = bitcast [27 x i32]* [[A_ADDR]] to <27 x i32>*
-// CHECK-NEXT:    store <27 x i32> [[A:%.*]], <27 x i32>* [[MAT_ADDR]], align 4
-// NOOPT-NEXT:    [[MAT:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR]], align 4{{$}}
-// OPT-NEXT:      [[MAT:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR]], align 4, !tbaa !{{[0-9]+}}{{$}}
+// CHECK-NEXT:    store <27 x i32> [[A:%.*]], ptr [[A_ADDR]], align 4
+// NOOPT-NEXT:    [[MAT:%.*]] = load <27 x i32>, ptr [[A_ADDR]], align 4{{$}}
+// OPT-NEXT:      [[MAT:%.*]] = load <27 x i32>, ptr [[A_ADDR]], align 4, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[RES:%.*]] = mul <27 x i32> <i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5>, [[MAT]]
-// CHECK-NEXT:    store <27 x i32> [[RES]], <27 x i32>* [[MAT_ADDR]], align 4
+// CHECK-NEXT:    store <27 x i32> [[RES]], ptr [[A_ADDR]], align 4
 // CHECK-NEXT:    ret void
 //
 void multiply_int_matrix_constant(ix9x3_t a) {
@@ -833,12 +828,11 @@ void multiply_int_matrix_constant(ix9x3_t a) {
 // CHECK-LABEL: @multiply_compound_int_matrix_constant(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[A_ADDR:%.*]] = alloca [27 x i32], align 4
-// CHECK-NEXT:    [[MAT_ADDR:%.*]] = bitcast [27 x i32]* [[A_ADDR]] to <27 x i32>*
-// CHECK-NEXT:    store <27 x i32> [[A:%.*]], <27 x i32>* [[MAT_ADDR]], align 4
-// NOOPT-NEXT:    [[MAT:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR]], align 4{{$}}
-// OPT-NEXT:      [[MAT:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR]], align 4, !tbaa !{{[0-9]+}}{{$}}
+// CHECK-NEXT:    store <27 x i32> [[A:%.*]], ptr [[A_ADDR]], align 4
+// NOOPT-NEXT:    [[MAT:%.*]] = load <27 x i32>, ptr [[A_ADDR]], align 4{{$}}
+// OPT-NEXT:      [[MAT:%.*]] = load <27 x i32>, ptr [[A_ADDR]], align 4, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[RES:%.*]] = mul <27 x i32> [[MAT]], <i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5, i32 5>
-// CHECK-NEXT:    store <27 x i32> [[RES]], <27 x i32>* [[MAT_ADDR]], align 4
+// CHECK-NEXT:    store <27 x i32> [[RES]], ptr [[A_ADDR]], align 4
 // CHECK-NEXT:    ret void
 //
 void multiply_compound_int_matrix_constant(ix9x3_t a) {
@@ -846,15 +840,15 @@ void multiply_compound_int_matrix_constant(ix9x3_t a) {
 }
 
 // CHECK-LABEL: @divide_double_matrix_scalar_float(
-// NOOPT:         [[A:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-// NOOPT-NEXT:    [[S:%.*]] = load float, float* %s.addr, align 4{{$}}
-// OPT:           [[A:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
-// OPT-NEXT:      [[S:%.*]] = load float, float* %s.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT:         [[A:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+// NOOPT-NEXT:    [[S:%.*]] = load float, ptr %s.addr, align 4{{$}}
+// OPT:           [[A:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+// OPT-NEXT:      [[S:%.*]] = load float, ptr %s.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[S_EXT:%.*]] = fpext float [[S]] to double
 // CHECK-NEXT:    [[VECINSERT:%.*]] = insertelement <25 x double> poison, double [[S_EXT]], i64 0
 // CHECK-NEXT:    [[VECSPLAT:%.*]] = shufflevector <25 x double> [[VECINSERT]], <25 x double> poison, <25 x i32> zeroinitializer
 // CHECK-NEXT:    [[RES:%.*]] = fdiv <25 x double> [[A]], [[VECSPLAT]]
-// CHECK-NEXT:    store <25 x double> [[RES]], <25 x double>* {{.*}}, align 8
+// CHECK-NEXT:    store <25 x double> [[RES]], ptr {{.*}}, align 8
 // CHECK-NEXT:    ret void
 //
 void divide_double_matrix_scalar_float(dx5x5_t a, float s) {
@@ -862,14 +856,14 @@ void divide_double_matrix_scalar_float(dx5x5_t a, float s) {
 }
 
 // CHECK-LABEL: @divide_double_matrix_scalar_double(
-// NOOPT:         [[A:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-// NOOPT-NEXT:    [[S:%.*]] = load double, double* %s.addr, align 8{{$}}
-// OPT:           [[A:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
-// OPT-NEXT:      [[S:%.*]] = load double, double* %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT:         [[A:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+// NOOPT-NEXT:    [[S:%.*]] = load double, ptr %s.addr, align 8{{$}}
+// OPT:           [[A:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+// OPT-NEXT:      [[S:%.*]] = load double, ptr %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[VECINSERT:%.*]] = insertelement <25 x double> poison, double [[S]], i64 0
 // CHECK-NEXT:    [[VECSPLAT:%.*]] = shufflevector <25 x double> [[VECINSERT]], <25 x double> poison, <25 x i32> zeroinitializer
 // CHECK-NEXT:    [[RES:%.*]] = fdiv <25 x double> [[A]], [[VECSPLAT]]
-// CHECK-NEXT:    store <25 x double> [[RES]], <25 x double>* {{.*}}, align 8
+// CHECK-NEXT:    store <25 x double> [[RES]], ptr {{.*}}, align 8
 // CHECK-NEXT:    ret void
 //
 void divide_double_matrix_scalar_double(dx5x5_t a, double s) {
@@ -877,15 +871,15 @@ void divide_double_matrix_scalar_double(dx5x5_t a, double s) {
 }
 
 // CHECK-LABEL: @divide_float_matrix_scalar_double(
-// NOOPT:         [[MAT:%.*]] = load <6 x float>, <6 x float>* [[MAT_ADDR:%.*]], align 4{{$}}
-// NOOPT-NEXT:    [[S:%.*]] = load double, double* %s.addr, align 8{{$}}
-// OPT:           [[MAT:%.*]] = load <6 x float>, <6 x float>* [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
-// OPT-NEXT:      [[S:%.*]] = load double, double* %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT:         [[MAT:%.*]] = load <6 x float>, ptr [[MAT_ADDR:%.*]], align 4{{$}}
+// NOOPT-NEXT:    [[S:%.*]] = load double, ptr %s.addr, align 8{{$}}
+// OPT:           [[MAT:%.*]] = load <6 x float>, ptr [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
+// OPT-NEXT:      [[S:%.*]] = load double, ptr %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[S_TRUNC:%.*]] = fptrunc double [[S]] to float
 // CHECK-NEXT:    [[VECINSERT:%.*]] = insertelement <6 x float> poison, float [[S_TRUNC]], i64 0
 // CHECK-NEXT:    [[VECSPLAT:%.*]] = shufflevector <6 x float> [[VECINSERT]], <6 x float> poison, <6 x i32> zeroinitializer
 // CHECK-NEXT:    [[RES:%.*]] = fdiv <6 x float> [[MAT]], [[VECSPLAT]]
-// CHECK-NEXT:    store <6 x float> [[RES]], <6 x float>* [[MAT_ADDR]], align 4
+// CHECK-NEXT:    store <6 x float> [[RES]], ptr [[MAT_ADDR]], align 4
 // CHECK-NEXT:    ret void
 //
 void divide_float_matrix_scalar_double(fx2x3_t b, double s) {
@@ -893,15 +887,15 @@ void divide_float_matrix_scalar_double(fx2x3_t b, double s) {
 }
 
 // CHECK-LABEL: @divide_int_matrix_scalar_short(
-// NOOPT:         [[MAT:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4{{$}}
-// NOOPT-NEXT:    [[S:%.*]] = load i16, i16* %s.addr, align 2{{$}}
-// OPT:           [[MAT:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
-// OPT-NEXT:      [[S:%.*]] = load i16, i16* %s.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT:         [[MAT:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4{{$}}
+// NOOPT-NEXT:    [[S:%.*]] = load i16, ptr %s.addr, align 2{{$}}
+// OPT:           [[MAT:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
+// OPT-NEXT:      [[S:%.*]] = load i16, ptr %s.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[S_EXT:%.*]] = sext i16 [[S]] to i32
 // CHECK-NEXT:    [[VECINSERT:%.*]] = insertelement <27 x i32> poison, i32 [[S_EXT]], i64 0
 // CHECK-NEXT:    [[VECSPLAT:%.*]] = shufflevector <27 x i32> [[VECINSERT]], <27 x i32> poison, <27 x i32> zeroinitializer
 // CHECK-NEXT:    [[RES:%.*]] = sdiv <27 x i32> [[MAT]], [[VECSPLAT]]
-// CHECK-NEXT:    store <27 x i32> [[RES]], <27 x i32>* [[MAT_ADDR]], align 4
+// CHECK-NEXT:    store <27 x i32> [[RES]], ptr [[MAT_ADDR]], align 4
 // CHECK-NEXT:    ret void
 //
 void divide_int_matrix_scalar_short(ix9x3_t b, short s) {
@@ -909,15 +903,15 @@ void divide_int_matrix_scalar_short(ix9x3_t b, short s) {
 }
 
 // CHECK-LABEL: @divide_int_matrix_scalar_ull(
-// NOOPT:         [[MAT:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4{{$}}
-// NOOPT-NEXT:    [[S:%.*]] = load i64, i64* %s.addr, align 8{{$}}
-// OPT:           [[MAT:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
-// OPT-NEXT:      [[S:%.*]] = load i64, i64* %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT:         [[MAT:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4{{$}}
+// NOOPT-NEXT:    [[S:%.*]] = load i64, ptr %s.addr, align 8{{$}}
+// OPT:           [[MAT:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
+// OPT-NEXT:      [[S:%.*]] = load i64, ptr %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[S_TRUNC:%.*]] = trunc i64 [[S]] to i32
 // CHECK-NEXT:    [[VECINSERT:%.*]] = insertelement <27 x i32> poison, i32 [[S_TRUNC]], i64 0
 // CHECK-NEXT:    [[VECSPLAT:%.*]] = shufflevector <27 x i32> [[VECINSERT]], <27 x i32> poison, <27 x i32> zeroinitializer
 // CHECK-NEXT:    [[RES:%.*]] = sdiv <27 x i32> [[MAT]], [[VECSPLAT]]
-// CHECK-NEXT:    store <27 x i32> [[RES]], <27 x i32>* [[MAT_ADDR]], align 4
+// CHECK-NEXT:    store <27 x i32> [[RES]], ptr [[MAT_ADDR]], align 4
 // CHECK-NEXT:    ret void
 //
 void divide_int_matrix_scalar_ull(ix9x3_t b, unsigned long long s) {
@@ -925,14 +919,14 @@ void divide_int_matrix_scalar_ull(ix9x3_t b, unsigned long long s) {
 }
 
 // CHECK-LABEL: @divide_ull_matrix_scalar_ull(
-// NOOPT:         [[MAT:%.*]] = load <8 x i64>, <8 x i64>* [[MAT_ADDR:%.*]], align 8{{$}}
-// NOOPT-NEXT:    [[S:%.*]] = load i64, i64* %s.addr, align 8{{$}}
-// OPT:           [[MAT:%.*]] = load <8 x i64>, <8 x i64>* [[MAT_ADDR:%.*]], align 8, !tbaa !{{[0-9]+}}{{$}}
-// OPT-NEXT:      [[S:%.*]] = load i64, i64* %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+// NOOPT:         [[MAT:%.*]] = load <8 x i64>, ptr [[MAT_ADDR:%.*]], align 8{{$}}
+// NOOPT-NEXT:    [[S:%.*]] = load i64, ptr %s.addr, align 8{{$}}
+// OPT:           [[MAT:%.*]] = load <8 x i64>, ptr [[MAT_ADDR:%.*]], align 8, !tbaa !{{[0-9]+}}{{$}}
+// OPT-NEXT:      [[S:%.*]] = load i64, ptr %s.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[VECINSERT:%.*]] = insertelement <8 x i64> poison, i64 [[S]], i64 0
 // CHECK-NEXT:    [[VECSPLAT:%.*]] = shufflevector <8 x i64> [[VECINSERT]], <8 x i64> poison, <8 x i32> zeroinitializer
 // CHECK-NEXT:    [[RES:%.*]] = udiv <8 x i64> [[MAT]], [[VECSPLAT]]
-// CHECK-NEXT:    store <8 x i64> [[RES]], <8 x i64>* [[MAT_ADDR]], align 8
+// CHECK-NEXT:    store <8 x i64> [[RES]], ptr [[MAT_ADDR]], align 8
 // CHECK-NEXT:    ret void
 //
 void divide_ull_matrix_scalar_ull(ullx4x2_t b, unsigned long long s) {
@@ -942,12 +936,11 @@ void divide_ull_matrix_scalar_ull(ullx4x2_t b, unsigned long long s) {
 // CHECK-LABEL: @divide_float_matrix_constant(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[A_ADDR:%.*]] = alloca [6 x float], align 4
-// CHECK-NEXT:    [[MAT_ADDR:%.*]] = bitcast [6 x float]* [[A_ADDR]] to <6 x float>*
-// CHECK-NEXT:    store <6 x float> [[A:%.*]], <6 x float>* [[MAT_ADDR]], align 4
-// NOOPT-NEXT:    [[MAT:%.*]] = load <6 x float>, <6 x float>* [[MAT_ADDR]], align 4{{$}}
-// OPT-NEXT:      [[MAT:%.*]] = load <6 x float>, <6 x float>* [[MAT_ADDR]], align 4, !tbaa !{{[0-9]+}}{{$}}
+// CHECK-NEXT:    store <6 x float> [[A:%.*]], ptr [[A_ADDR]], align 4
+// NOOPT-NEXT:    [[MAT:%.*]] = load <6 x float>, ptr [[A_ADDR]], align 4{{$}}
+// OPT-NEXT:      [[MAT:%.*]] = load <6 x float>, ptr [[A_ADDR]], align 4, !tbaa !{{[0-9]+}}{{$}}
 // CHECK-NEXT:    [[RES:%.*]] = fdiv <6 x float> [[MAT]], <float 2.500000e+00, float 2.500000e+00, float 2.500000e+00, float 2.500000e+00, float 2.500000e+00, float 2.500000e+00>
-// CHECK-NEXT:    store <6 x float> [[RES]], <6 x float>* [[MAT_ADDR]], align 4
+// CHECK-NEXT:    store <6 x float> [[RES]], ptr [[A_ADDR]], align 4
 // CHECK-NEXT:    ret void
 //
 void divide_float_matrix_constant(fx2x3_t a) {
@@ -963,11 +956,11 @@ typedef float fx2x3_t __attribute__((matrix_type(2, 3)));
 // matrixes and indices.
 void insert_double_matrix_const_idx_ll_u_double(dx5x5_t a, double d, fx2x3_t b, float e, int j, unsigned k) {
   // CHECK-LABEL: @insert_double_matrix_const_idx_ll_u_double(
-  // NOOPT:         [[D:%.*]] = load double, double* %d.addr, align 8{{$}}
-  // OPT:           [[D:%.*]] = load double, double* %d.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // CHECK-NEXT:    [[MAT:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
+  // NOOPT:         [[D:%.*]] = load double, ptr %d.addr, align 8{{$}}
+  // OPT:           [[D:%.*]] = load double, ptr %d.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // CHECK-NEXT:    [[MAT:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
   // CHECK-NEXT:    [[MATINS:%.*]] = insertelement <25 x double> [[MAT]], double [[D]], i64 5
-  // CHECK-NEXT:    store <25 x double> [[MATINS]], <25 x double>* {{.*}}, align 8
+  // CHECK-NEXT:    store <25 x double> [[MATINS]], ptr {{.*}}, align 8
   // CHECK-NEXT:    ret void
 
   a[0ll][1u] = d;
@@ -975,11 +968,11 @@ void insert_double_matrix_const_idx_ll_u_double(dx5x5_t a, double d, fx2x3_t b,
 
 void insert_double_matrix_const_idx_i_u_double(dx5x5_t a, double d) {
   // CHECK-LABEL: @insert_double_matrix_const_idx_i_u_double(
-  // NOOPT:         [[D:%.*]] = load double, double* %d.addr, align 8{{$}}
-  // OPT:           [[D:%.*]] = load double, double* %d.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // CHECK-NEXT:    [[MAT:%.*]] = load <25 x double>, <25 x double>* [[MAT_ADDR:%.*]], align 8{{$}}
+  // NOOPT:         [[D:%.*]] = load double, ptr %d.addr, align 8{{$}}
+  // OPT:           [[D:%.*]] = load double, ptr %d.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // CHECK-NEXT:    [[MAT:%.*]] = load <25 x double>, ptr [[MAT_ADDR:%.*]], align 8{{$}}
   // CHECK-NEXT:    [[MATINS:%.*]] = insertelement <25 x double> [[MAT]], double [[D]], i64 21
-  // CHECK-NEXT:    store <25 x double> [[MATINS]], <25 x double>* [[MAT_ADDR]], align 8
+  // CHECK-NEXT:    store <25 x double> [[MATINS]], ptr [[MAT_ADDR]], align 8
   // CHECK-NEXT:    ret void
 
   a[1][4u] = d;
@@ -987,11 +980,11 @@ void insert_double_matrix_const_idx_i_u_double(dx5x5_t a, double d) {
 
 void insert_float_matrix_const_idx_ull_i_float(fx2x3_t b, float e) {
   // CHECK-LABEL: @insert_float_matrix_const_idx_ull_i_float(
-  // NOOPT:         [[E:%.*]] = load float, float* %e.addr, align 4{{$}}
-  // OPT:           [[E:%.*]] = load float, float* %e.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
-  // CHECK-NEXT:    [[MAT:%.*]] = load <6 x float>, <6 x float>* [[MAT_ADDR:%.*]], align 4{{$}}
+  // NOOPT:         [[E:%.*]] = load float, ptr %e.addr, align 4{{$}}
+  // OPT:           [[E:%.*]] = load float, ptr %e.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // CHECK-NEXT:    [[MAT:%.*]] = load <6 x float>, ptr [[MAT_ADDR:%.*]], align 4{{$}}
   // CHECK-NEXT:    [[MATINS:%.*]] = insertelement <6 x float> [[MAT]], float [[E]], i64 3
-  // CHECK-NEXT:    store <6 x float> [[MATINS]], <6 x float>* [[MAT_ADDR]], align 4
+  // CHECK-NEXT:    store <6 x float> [[MATINS]], ptr [[MAT_ADDR]], align 4
   // CHECK-NEXT:    ret void
 
   b[1ull][1] = e;
@@ -999,21 +992,21 @@ void insert_float_matrix_const_idx_ull_i_float(fx2x3_t b, float e) {
 
 void insert_float_matrix_idx_i_u_float(fx2x3_t b, float e, int j, unsigned k) {
   // CHECK-LABEL: @insert_float_matrix_idx_i_u_float(
-  // NOOPT:         [[E:%.*]] = load float, float* %e.addr, align 4{{$}}
-  // NOOPT-NEXT:    [[J:%.*]] = load i32, i32* %j.addr, align 4{{$}}
-  // OPT:           [[E:%.*]] = load float, float* %e.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:      [[J:%.*]] = load i32, i32* %j.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:         [[E:%.*]] = load float, ptr %e.addr, align 4{{$}}
+  // NOOPT-NEXT:    [[J:%.*]] = load i32, ptr %j.addr, align 4{{$}}
+  // OPT:           [[E:%.*]] = load float, ptr %e.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:      [[J:%.*]] = load i32, ptr %j.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[J_EXT:%.*]] = sext i32 [[J]] to i64
-  // NOOPT-NEXT:    [[K:%.*]] = load i32, i32* %k.addr, align 4{{$}}
-  // OPT-NEXT:      [[K:%.*]] = load i32, i32* %k.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:    [[K:%.*]] = load i32, ptr %k.addr, align 4{{$}}
+  // OPT-NEXT:      [[K:%.*]] = load i32, ptr %k.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[K_EXT:%.*]] = zext i32 [[K]] to i64
   // CHECK-NEXT:    [[IDX1:%.*]] = mul i64 [[K_EXT]], 2
   // CHECK-NEXT:    [[IDX2:%.*]] = add i64 [[IDX1]], [[J_EXT]]
   // OPT-NEXT:      [[CMP:%.*]] = icmp ult i64 [[IDX2]], 6
   // OPT-NEXT:      call void @llvm.assume(i1 [[CMP]])
-  // CHECK-NEXT:    [[MAT:%.*]] = load <6 x float>, <6 x float>* [[MAT_ADDR:%.*]], align 4{{$}}
+  // CHECK-NEXT:    [[MAT:%.*]] = load <6 x float>, ptr [[MAT_ADDR:%.*]], align 4{{$}}
   // CHECK-NEXT:    [[MATINS:%.*]] = insertelement <6 x float> [[MAT]], float [[E]], i64 [[IDX2]]
-  // CHECK-NEXT:    store <6 x float> [[MATINS]], <6 x float>* [[MAT_ADDR]], align 4
+  // CHECK-NEXT:    store <6 x float> [[MATINS]], ptr [[MAT_ADDR]], align 4
   // CHECK-NEXT:    ret void
 
   b[j][k] = e;
@@ -1021,20 +1014,20 @@ void insert_float_matrix_idx_i_u_float(fx2x3_t b, float e, int j, unsigned k) {
 
 void insert_float_matrix_idx_s_ull_float(fx2x3_t b, float e, short j, unsigned long long k) {
   // CHECK-LABEL: @insert_float_matrix_idx_s_ull_float(
-  // NOOPT:         [[E:%.*]] = load float, float* %e.addr, align 4{{$}}
-  // NOOPT-NEXT:    [[J:%.*]] = load i16, i16* %j.addr, align 2{{$}}
-  // OPT:           [[E:%.*]] = load float, float* %e.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:      [[J:%.*]] = load i16, i16* %j.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:         [[E:%.*]] = load float, ptr %e.addr, align 4{{$}}
+  // NOOPT-NEXT:    [[J:%.*]] = load i16, ptr %j.addr, align 2{{$}}
+  // OPT:           [[E:%.*]] = load float, ptr %e.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:      [[J:%.*]] = load i16, ptr %j.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[J_EXT:%.*]] = sext i16 [[J]] to i64
-  // NOOPT-NEXT:    [[K:%.*]] = load i64, i64* %k.addr, align 8{{$}}
-  // OPT-NEXT:      [[K:%.*]] = load i64, i64* %k.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:    [[K:%.*]] = load i64, ptr %k.addr, align 8{{$}}
+  // OPT-NEXT:      [[K:%.*]] = load i64, ptr %k.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[IDX1:%.*]] = mul i64 [[K]], 2
   // CHECK-NEXT:    [[IDX2:%.*]] = add i64 [[IDX1]], [[J_EXT]]
   // OPT-NEXT:      [[CMP:%.*]] = icmp ult i64 [[IDX2]], 6
   // OPT-NEXT:      call void @llvm.assume(i1 [[CMP]])
-  // CHECK-NEXT:    [[MAT:%.*]] = load <6 x float>, <6 x float>* [[MAT_ADDR:%.*]], align 4{{$}}
+  // CHECK-NEXT:    [[MAT:%.*]] = load <6 x float>, ptr [[MAT_ADDR:%.*]], align 4{{$}}
   // CHECK-NEXT:    [[MATINS:%.*]] = insertelement <6 x float> [[MAT]], float [[E]], i64 [[IDX2]]
-  // CHECK-NEXT:    store <6 x float> [[MATINS]], <6 x float>* [[MAT_ADDR]], align 4
+  // CHECK-NEXT:    store <6 x float> [[MATINS]], ptr [[MAT_ADDR]], align 4
   // CHECK-NEXT:    ret void
 
   (b)[j][k] = e;
@@ -1044,18 +1037,18 @@ void insert_float_matrix_idx_s_ull_float(fx2x3_t b, float e, short j, unsigned l
 typedef int ix9x3_t __attribute__((matrix_type(9, 3)));
 void insert_int_idx_expr(ix9x3_t a, int i) {
   // CHECK-LABEL: @insert_int_idx_expr(
-  // NOOPT:         [[I1:%.*]] = load i32, i32* %i.addr, align 4{{$}}
-  // NOOPT-NEXT:    [[I2:%.*]] = load i32, i32* %i.addr, align 4{{$}}
-  // OPT:           [[I1:%.*]] = load i32, i32* %i.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:      [[I2:%.*]] = load i32, i32* %i.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:         [[I1:%.*]] = load i32, ptr %i.addr, align 4{{$}}
+  // NOOPT-NEXT:    [[I2:%.*]] = load i32, ptr %i.addr, align 4{{$}}
+  // OPT:           [[I1:%.*]] = load i32, ptr %i.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:      [[I2:%.*]] = load i32, ptr %i.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[I2_ADD:%.*]] = add nsw i32 4, [[I2]]
   // CHECK-NEXT:    [[ADD_EXT:%.*]] = sext i32 [[I2_ADD]] to i64
   // CHECK-NEXT:    [[IDX2:%.*]] = add i64 18, [[ADD_EXT]]
   // OPT-NEXT:      [[CMP:%.*]] = icmp ult i64 [[IDX2]], 27
   // OPT-NEXT:      call void @llvm.assume(i1 [[CMP]])
-  // CHECK-NEXT:    [[MAT:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR:%.*]], align 4{{$}}
+  // CHECK-NEXT:    [[MAT:%.*]] = load <27 x i32>, ptr [[MAT_ADDR:%.*]], align 4{{$}}
   // CHECK-NEXT:    [[MATINS:%.*]] = insertelement <27 x i32> [[MAT]], i32 [[I1]], i64 [[IDX2]]
-  // CHECK-NEXT:    store <27 x i32> [[MATINS]], <27 x i32>* [[MAT_ADDR]], align 4
+  // CHECK-NEXT:    store <27 x i32> [[MATINS]], ptr [[MAT_ADDR]], align 4
   // CHECK-NEXT:    ret void
 
   a[4 + i][1 + 1u] = i;
@@ -1066,14 +1059,13 @@ void insert_int_idx_expr(ix9x3_t a, int i) {
 typedef int ix9x3_t __attribute__((matrix_type(9, 3)));
 void insert_float_into_int_matrix(ix9x3_t *a, int i) {
   // CHECK-LABEL: @insert_float_into_int_matrix(
-  // NOOPT:         [[I:%.*]] = load i32, i32* %i.addr, align 4{{$}}
-  // OPT:           [[I:%.*]] = load i32, i32* %i.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
-  // NOOPT-NEXT:    [[MAT_ADDR1:%.*]] = load [27 x i32]*, [27 x i32]** %a.addr, align 8{{$}}
-  // OPT-NEXT:      [[MAT_ADDR1:%.*]] = load [27 x i32]*, [27 x i32]** %a.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // CHECK-NEXT:    [[MAT_ADDR2:%.*]] = bitcast [27 x i32]* [[MAT_ADDR1]] to <27 x i32>*
-  // CHECK-NEXT:    [[MAT:%.*]] = load <27 x i32>, <27 x i32>* [[MAT_ADDR2]], align 4{{$}}
+  // NOOPT:         [[I:%.*]] = load i32, ptr %i.addr, align 4{{$}}
+  // OPT:           [[I:%.*]] = load i32, ptr %i.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:    [[MAT_ADDR1:%.*]] = load ptr, ptr %a.addr, align 8{{$}}
+  // OPT-NEXT:      [[MAT_ADDR1:%.*]] = load ptr, ptr %a.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // CHECK-NEXT:    [[MAT:%.*]] = load <27 x i32>, ptr [[MAT_ADDR1]], align 4{{$}}
   // CHECK-NEXT:    [[MATINS:%.*]] = insertelement <27 x i32> [[MAT]], i32 [[I]], i64 13
-  // CHECK-NEXT:    store <27 x i32> [[MATINS]], <27 x i32>* [[MAT_ADDR2]], align 4
+  // CHECK-NEXT:    store <27 x i32> [[MATINS]], ptr [[MAT_ADDR1]], align 4
   // CHECK-NEXT:    ret void
 
   (*a)[4][1] = i;
@@ -1085,11 +1077,11 @@ typedef double dx3x3_t __attribute__((matrix_type(3, 3)));
 typedef float fx3x3_t __attribute__((matrix_type(3, 3)));
 void insert_matching_dimensions1(dx3x3_t a, double i) {
   // CHECK-LABEL: @insert_matching_dimensions1(
-  // NOOPT:         [[I:%.*]] = load double, double* %i.addr, align 8{{$}}
-  // OPT:           [[I:%.*]] = load double, double* %i.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // CHECK-NEXT:    [[MAT:%.*]] = load <9 x double>, <9 x double>* [[MAT_ADDR:%.*]], align 8{{$}}
+  // NOOPT:         [[I:%.*]] = load double, ptr %i.addr, align 8{{$}}
+  // OPT:           [[I:%.*]] = load double, ptr %i.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // CHECK-NEXT:    [[MAT:%.*]] = load <9 x double>, ptr [[MAT_ADDR:%.*]], align 8{{$}}
   // CHECK-NEXT:    [[MATINS:%.*]] = insertelement <9 x double> [[MAT]], double [[I]], i64 5
-  // CHECK-NEXT:    store <9 x double> [[MATINS]], <9 x double>* [[MAT_ADDR]], align 8
+  // CHECK-NEXT:    store <9 x double> [[MATINS]], ptr [[MAT_ADDR]], align 8
   // CHECK-NEXT:    ret void
 
   a[2u][1u] = i;
@@ -1097,11 +1089,11 @@ void insert_matching_dimensions1(dx3x3_t a, double i) {
 
 void insert_matching_dimensions(fx3x3_t b, float e) {
   // CHECK-LABEL: @insert_matching_dimensions(
-  // NOOPT:         [[E:%.*]] = load float, float* %e.addr, align 4{{$}}
-  // OPT:           [[E:%.*]] = load float, float* %e.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
-  // CHECK-NEXT:    [[MAT:%.*]] = load <9 x float>, <9 x float>* [[MAT_ADDR:%.*]], align 4{{$}}
+  // NOOPT:         [[E:%.*]] = load float, ptr %e.addr, align 4{{$}}
+  // OPT:           [[E:%.*]] = load float, ptr %e.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // CHECK-NEXT:    [[MAT:%.*]] = load <9 x float>, ptr [[MAT_ADDR:%.*]], align 4{{$}}
   // CHECK-NEXT:    [[MATINS:%.*]] = insertelement <9 x float> [[MAT]], float [[E]], i64 7
-  // CHECK-NEXT:    store <9 x float> [[MATINS]], <9 x float>* [[MAT_ADDR]], align 4
+  // CHECK-NEXT:    store <9 x float> [[MATINS]], ptr [[MAT_ADDR]], align 4
   // CHECK-NEXT:    ret void
 
   b[1u][2u] = e;
@@ -1109,8 +1101,8 @@ void insert_matching_dimensions(fx3x3_t b, float e) {
 
 double extract_double(dx5x5_t a) {
   // CHECK-LABEL: @extract_double(
-  // NOOPT:         [[MAT:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8{{$}}
-  // OPT:           [[MAT:%.*]] = load <25 x double>, <25 x double>* {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:         [[MAT:%.*]] = load <25 x double>, ptr {{.*}}, align 8{{$}}
+  // OPT:           [[MAT:%.*]] = load <25 x double>, ptr {{.*}}, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[MATEXT:%.*]] = extractelement <25 x double> [[MAT]], i64 12
   // CHECK-NEXT:    ret double [[MATEXT]]
 
@@ -1119,8 +1111,8 @@ double extract_double(dx5x5_t a) {
 
 double extract_float(fx3x3_t b) {
   // CHECK-LABEL: @extract_float(
-  // NOOPT:         [[MAT:%.*]] = load <9 x float>, <9 x float>* {{.*}}, align 4{{$}}
-  // OPT:           [[MAT:%.*]] = load <9 x float>, <9 x float>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:         [[MAT:%.*]] = load <9 x float>, ptr {{.*}}, align 4{{$}}
+  // OPT:           [[MAT:%.*]] = load <9 x float>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[MATEXT:%.*]] = extractelement <9 x float> [[MAT]], i64 5
   // CHECK-NEXT:    [[TO_DOUBLE:%.*]] = fpext float [[MATEXT]] to double
   // CHECK-NEXT:    ret double [[TO_DOUBLE]]
@@ -1130,16 +1122,16 @@ double extract_float(fx3x3_t b) {
 
 int extract_int(ix9x3_t c, unsigned long j) {
   // CHECK-LABEL: @extract_int(
-  // NOOPT:         [[J1:%.*]] = load i64, i64* %j.addr, align 8{{$}}
-  // NOOPT-NEXT:    [[J2:%.*]] = load i64, i64* %j.addr, align 8{{$}}
-  // OPT:           [[J1:%.*]] = load i64, i64* %j.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // OPT-NEXT:      [[J2:%.*]] = load i64, i64* %j.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:         [[J1:%.*]] = load i64, ptr %j.addr, align 8{{$}}
+  // NOOPT-NEXT:    [[J2:%.*]] = load i64, ptr %j.addr, align 8{{$}}
+  // OPT:           [[J1:%.*]] = load i64, ptr %j.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:      [[J2:%.*]] = load i64, ptr %j.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[IDX1:%.*]] = mul i64 [[J2]], 9
   // CHECK-NEXT:    [[IDX2:%.*]] = add i64 [[IDX1]], [[J1]]
-  // NOOPT-NEXT:    [[MAT:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4{{$}}
+  // NOOPT-NEXT:    [[MAT:%.*]] = load <27 x i32>, ptr {{.*}}, align 4{{$}}
   // OPT-NEXT:      [[CMP:%.*]] = icmp ult i64 [[IDX2]], 27
   // OPT-NEXT:      call void @llvm.assume(i1 [[CMP]])
-  // OPT-NEXT:      [[MAT:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:      [[MAT:%.*]] = load <27 x i32>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[MATEXT:%.*]] = extractelement <27 x i32> [[MAT]], i64 [[IDX2]]
   // CHECK-NEXT:    ret i32 [[MATEXT]]
 
@@ -1150,21 +1142,20 @@ typedef double dx3x2_t __attribute__((matrix_type(3, 2)));
 
 double test_extract_matrix_pointer1(dx3x2_t **ptr, unsigned j) {
   // CHECK-LABEL: @test_extract_matrix_pointer1(
-  // NOOPT:         [[J:%.*]] = load i32, i32* %j.addr, align 4{{$}}
-  // OPT:           [[J:%.*]] = load i32, i32* %j.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:         [[J:%.*]] = load i32, ptr %j.addr, align 4{{$}}
+  // OPT:           [[J:%.*]] = load i32, ptr %j.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[J_EXT:%.*]] = zext i32 [[J]] to i64
   // CHECK-NEXT:    [[IDX:%.*]] = add i64 3, [[J_EXT]]
-  // NOOPT-NEXT:    [[PTR:%.*]] = load [6 x double]**, [6 x double]*** %ptr.addr, align 8{{$}}
+  // NOOPT-NEXT:    [[PTR:%.*]] = load ptr, ptr %ptr.addr, align 8{{$}}
   // OPT-NEXT:      [[CMP:%.*]] = icmp ult i64 [[IDX]], 6
   // OPT-NEXT:      call void @llvm.assume(i1 [[CMP]])
-  // OPT-NEXT:      [[PTR:%.*]] = load [6 x double]**, [6 x double]*** %ptr.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // CHECK-NEXT:    [[PTR_IDX:%.*]] = getelementptr inbounds [6 x double]*, [6 x double]** [[PTR]], i64 1
-  // NOOPT-NEXT:    [[PTR2:%.*]] = load [6 x double]*, [6 x double]** [[PTR_IDX]], align 8{{$}}
-  // OPT-NEXT:      [[PTR2:%.*]] = load [6 x double]*, [6 x double]** [[PTR_IDX]], align 8, !tbaa !{{[0-9]+}}{{$}}
-  // CHECK-NEXT:    [[PTR2_IDX:%.*]] = getelementptr inbounds [6 x double], [6 x double]* [[PTR2]], i64 2
-  // CHECK-NEXT:    [[MAT_ADDR:%.*]] = bitcast [6 x double]* [[PTR2_IDX]] to <6 x double>*
-  // NOOPT-NEXT:    [[MAT:%.*]] = load <6 x double>, <6 x double>* [[MAT_ADDR]], align 8{{$}}
-  // OPT-NEXT:      [[MAT:%.*]] = load <6 x double>, <6 x double>* [[MAT_ADDR]], align 8, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:      [[PTR:%.*]] = load ptr, ptr %ptr.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // CHECK-NEXT:    [[PTR_IDX:%.*]] = getelementptr inbounds ptr, ptr [[PTR]], i64 1
+  // NOOPT-NEXT:    [[PTR2:%.*]] = load ptr, ptr [[PTR_IDX]], align 8{{$}}
+  // OPT-NEXT:      [[PTR2:%.*]] = load ptr, ptr [[PTR_IDX]], align 8, !tbaa !{{[0-9]+}}{{$}}
+  // CHECK-NEXT:    [[PTR2_IDX:%.*]] = getelementptr inbounds [6 x double], ptr [[PTR2]], i64 2
+  // NOOPT-NEXT:    [[MAT:%.*]] = load <6 x double>, ptr [[PTR2_IDX]], align 8{{$}}
+  // OPT-NEXT:      [[MAT:%.*]] = load <6 x double>, ptr [[PTR2_IDX]], align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[MATEXT:%.*]] = extractelement <6 x double> [[MAT]], i64 [[IDX]]
   // CHECK-NEXT:    ret double [[MATEXT]]
 
@@ -1174,15 +1165,14 @@ double test_extract_matrix_pointer1(dx3x2_t **ptr, unsigned j) {
 double test_extract_matrix_pointer2(dx3x2_t **ptr) {
   // CHECK-LABEL: @test_extract_matrix_pointer2(
   // CHECK-NEXT:  entry:
-  // NOOPT:         [[PTR:%.*]] = load [6 x double]**, [6 x double]*** %ptr.addr, align 8{{$}}
-  // OPT:           [[PTR:%.*]] = load [6 x double]**, [6 x double]*** %ptr.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
-  // CHECK-NEXT:    [[PTR_IDX:%.*]] = getelementptr inbounds [6 x double]*, [6 x double]** [[PTR]], i64 4
-  // NOOPT-NEXT:    [[PTR2:%.*]] = load [6 x double]*, [6 x double]** [[PTR_IDX]], align 8{{$}}
-  // OPT-NEXT:      [[PTR2:%.*]] = load [6 x double]*, [6 x double]** [[PTR_IDX]], align 8, !tbaa !{{[0-9]+}}{{$}}
-  // CHECK-NEXT:    [[PTR2_IDX:%.*]] = getelementptr inbounds [6 x double], [6 x double]* [[PTR2]], i64 6
-  // CHECK-NEXT:    [[MAT_ADDR:%.*]] = bitcast [6 x double]* [[PTR2_IDX]] to <6 x double>*
-  // NOOPT-NEXT:    [[MAT:%.*]] = load <6 x double>, <6 x double>* [[MAT_ADDR]], align 8{{$}}
-  // OPT-NEXT:      [[MAT:%.*]] = load <6 x double>, <6 x double>* [[MAT_ADDR]], align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:         [[PTR:%.*]] = load ptr, ptr %ptr.addr, align 8{{$}}
+  // OPT:           [[PTR:%.*]] = load ptr, ptr %ptr.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // CHECK-NEXT:    [[PTR_IDX:%.*]] = getelementptr inbounds ptr, ptr [[PTR]], i64 4
+  // NOOPT-NEXT:    [[PTR2:%.*]] = load ptr, ptr [[PTR_IDX]], align 8{{$}}
+  // OPT-NEXT:      [[PTR2:%.*]] = load ptr, ptr [[PTR_IDX]], align 8, !tbaa !{{[0-9]+}}{{$}}
+  // CHECK-NEXT:    [[PTR2_IDX:%.*]] = getelementptr inbounds [6 x double], ptr [[PTR2]], i64 6
+  // NOOPT-NEXT:    [[MAT:%.*]] = load <6 x double>, ptr [[PTR2_IDX]], align 8{{$}}
+  // OPT-NEXT:      [[MAT:%.*]] = load <6 x double>, ptr [[PTR2_IDX]], align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[MATEXT:%.*]] = extractelement <6 x double> [[MAT]], i64 5
   // CHECK-NEXT:    ret double [[MATEXT]]
 
@@ -1191,25 +1181,25 @@ double test_extract_matrix_pointer2(dx3x2_t **ptr) {
 
 void insert_extract(dx5x5_t a, fx3x3_t b, unsigned long j, short k) {
   // CHECK-LABEL: @insert_extract(
-  // NOOPT:         [[K:%.*]] = load i16, i16* %k.addr, align 2{{$}}
-  // OPT:           [[K:%.*]] = load i16, i16* %k.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:         [[K:%.*]] = load i16, ptr %k.addr, align 2{{$}}
+  // OPT:           [[K:%.*]] = load i16, ptr %k.addr, align 2, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[K_EXT:%.*]] = sext i16 [[K]] to i64
   // CHECK-NEXT:    [[IDX1:%.*]] = mul i64 [[K_EXT]], 3
   // CHECK-NEXT:    [[IDX2:%.*]] = add i64 [[IDX1]], 0
-  // NOOPT-NEXT:    [[MAT:%.*]] = load <9 x float>, <9 x float>* [[MAT_ADDR:%.*]], align 4{{$}}
+  // NOOPT-NEXT:    [[MAT:%.*]] = load <9 x float>, ptr [[MAT_ADDR:%.*]], align 4{{$}}
   // OPT-NEXT:      [[CMP:%.*]] = icmp ult i64 [[IDX2]], 9
   // OPT-NEXT:      call void @llvm.assume(i1 [[CMP]])
-  // OPT-NEXT:      [[MAT:%.*]] = load <9 x float>, <9 x float>* [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:      [[MAT:%.*]] = load <9 x float>, ptr [[MAT_ADDR:%.*]], align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[MATEXT:%.*]] = extractelement <9 x float> [[MAT]], i64 [[IDX2]]
-  // NOOPT-NEXT:    [[J:%.*]] = load i64, i64* %j.addr, align 8{{$}}
-  // OPT-NEXT:      [[J:%.*]] = load i64, i64* %j.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:    [[J:%.*]] = load i64, ptr %j.addr, align 8{{$}}
+  // OPT-NEXT:      [[J:%.*]] = load i64, ptr %j.addr, align 8, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[IDX3:%.*]] = mul i64 [[J]], 3
   // CHECK-NEXT:    [[IDX4:%.*]] = add i64 [[IDX3]], 2
   // OPT-NEXT:      [[CMP:%.*]] = icmp ult i64 [[IDX4]], 9
   // OPT-NEXT:      call void @llvm.assume(i1 [[CMP]])
-  // CHECK-NEXT:    [[MAT2:%.*]] = load <9 x float>, <9 x float>* [[MAT_ADDR]], align 4{{$}}
+  // CHECK-NEXT:    [[MAT2:%.*]] = load <9 x float>, ptr [[MAT_ADDR]], align 4{{$}}
   // CHECK-NEXT:    [[MATINS:%.*]] = insertelement <9 x float> [[MAT2]], float [[MATEXT]], i64 [[IDX4]]
-  // CHECK-NEXT:    store <9 x float> [[MATINS]], <9 x float>* [[MAT_ADDR]], align 4
+  // CHECK-NEXT:    store <9 x float> [[MATINS]], ptr [[MAT_ADDR]], align 4
   // CHECK-NEXT:    ret void
 
   b[2][j] = b[0][k];
@@ -1217,12 +1207,12 @@ void insert_extract(dx5x5_t a, fx3x3_t b, unsigned long j, short k) {
 
 void insert_compound_stmt(dx5x5_t a) {
   // CHECK-LABEL: define{{.*}} void @insert_compound_stmt(<25 x double> noundef %a)
-  // CHECK:        [[A:%.*]] = load <25 x double>, <25 x double>* [[A_PTR:%.*]], align 8{{$}}
+  // CHECK:        [[A:%.*]] = load <25 x double>, ptr [[A_PTR:%.*]], align 8{{$}}
   // CHECK-NEXT:   [[EXT:%.*]] = extractelement <25 x double> [[A]], i64 17
   // CHECK-NEXT:   [[SUB:%.*]] = fsub double [[EXT]], 1.000000e+00
-  // CHECK-NEXT:   [[A2:%.*]] = load <25 x double>, <25 x double>* [[A_PTR]], align 8{{$}}
+  // CHECK-NEXT:   [[A2:%.*]] = load <25 x double>, ptr [[A_PTR]], align 8{{$}}
   // CHECK-NEXT:   [[INS:%.*]] = insertelement <25 x double> [[A2]], double [[SUB]], i64 17
-  // CHECK-NEXT:   store <25 x double> [[INS]], <25 x double>* [[A_PTR]], align 8
+  // CHECK-NEXT:   store <25 x double> [[INS]], ptr [[A_PTR]], align 8
   // CHECK-NEXT:   ret void
 
   a[2][3] -= 1.0;
@@ -1233,26 +1223,25 @@ struct Foo {
 };
 
 void insert_compound_stmt_field(struct Foo *a, float f, unsigned i, unsigned j) {
-  // CHECK-LABEL: define{{.*}} void @insert_compound_stmt_field(%struct.Foo* noundef %a, float noundef %f, i32 noundef %i, i32 noundef %j)
-  // NOOPT:         [[I:%.*]] = load i32, i32* %i.addr, align 4{{$}}
-  // OPT:           [[I:%.*]] = load i32, i32* %i.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // CHECK-LABEL: define{{.*}} void @insert_compound_stmt_field(ptr noundef %a, float noundef %f, i32 noundef %i, i32 noundef %j)
+  // NOOPT:         [[I:%.*]] = load i32, ptr %i.addr, align 4{{$}}
+  // OPT:           [[I:%.*]] = load i32, ptr %i.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[I_EXT:%.*]] = zext i32 [[I]] to i64
-  // NOOPT-NEXT:    [[J:%.*]] = load i32, i32* %j.addr, align 4{{$}}
-  // OPT-NEXT:      [[J:%.*]] = load i32, i32* %j.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:    [[J:%.*]] = load i32, ptr %j.addr, align 4{{$}}
+  // OPT-NEXT:      [[J:%.*]] = load i32, ptr %j.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:    [[J_EXT:%.*]] = zext i32 [[J]] to i64
   // CHECK-NEXT:    [[IDX1:%.*]] = mul i64 [[J_EXT]], 2
   // CHECK-NEXT:    [[IDX2:%.*]] = add i64 [[IDX1]], [[I_EXT]]
-  // CHECK-NEXT:    [[MAT_PTR:%.*]] = bitcast [6 x float]* %mat to <6 x float>*
   // OPT-NEXT:      [[CMP:%.*]] = icmp ult i64 [[IDX2]], 6
   // OPT-NEXT:      call void @llvm.assume(i1 [[CMP]])
-  // CHECK-NEXT:    [[MAT:%.*]] = load <6 x float>, <6 x float>* [[MAT_PTR]], align 4{{$}}
+  // CHECK-NEXT:    [[MAT:%.*]] = load <6 x float>, ptr %mat, align 4{{$}}
   // CHECK-NEXT:    [[EXT:%.*]] = extractelement <6 x float> [[MAT]], i64 [[IDX2]]
   // CHECK-NEXT:    [[SUM:%.*]] = fadd float [[EXT]], {{.*}}
   // OPT-NEXT:      [[CMP:%.*]] = icmp ult i64 [[IDX2]], 6
   // OPT-NEXT:      call void @llvm.assume(i1 [[CMP]])
-  // CHECK-NEXT:    [[MAT2:%.*]] = load <6 x float>, <6 x float>* [[MAT_PTR]], align 4{{$}}
+  // CHECK-NEXT:    [[MAT2:%.*]] = load <6 x float>, ptr %mat, align 4{{$}}
   // CHECK-NEXT:    [[INS:%.*]] = insertelement <6 x float> [[MAT2]], float [[SUM]], i64 [[IDX2]]
-  // CHECK-NEXT:    store <6 x float> [[INS]], <6 x float>* [[MAT_PTR]], align 4
+  // CHECK-NEXT:    store <6 x float> [[INS]], ptr %mat, align 4
   // CHECK-NEXT:    ret void
 
   a->mat[i][j] += f;
@@ -1260,32 +1249,32 @@ void insert_compound_stmt_field(struct Foo *a, float f, unsigned i, unsigned j)
 
 void matrix_as_idx(ix9x3_t a, int i, int j, dx5x5_t b) {
   // CHECK-LABEL: define{{.*}} void @matrix_as_idx(<27 x i32> noundef %a, i32 noundef %i, i32 noundef %j, <25 x double> noundef %b)
-  // NOOPT:       [[I1:%.*]] = load i32, i32* %i.addr, align 4{{$}}
-  // OPT:         [[I1:%.*]] = load i32, i32* %i.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT:       [[I1:%.*]] = load i32, ptr %i.addr, align 4{{$}}
+  // OPT:         [[I1:%.*]] = load i32, ptr %i.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[I1_EXT:%.*]] = sext i32 [[I1]] to i64
-  // NOOPT-NEXT:  [[J1:%.*]] = load i32, i32* %j.addr, align 4{{$}}
-  // OPT-NEXT:    [[J1:%.*]] = load i32, i32* %j.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:  [[J1:%.*]] = load i32, ptr %j.addr, align 4{{$}}
+  // OPT-NEXT:    [[J1:%.*]] = load i32, ptr %j.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[J1_EXT:%.*]] = sext i32 [[J1]] to i64
   // CHECK-NEXT:  [[IDX1_1:%.*]] = mul i64 [[J1_EXT]], 9
   // CHECK-NEXT:  [[IDX1_2:%.*]] = add i64 [[IDX1_1]], [[I1_EXT]]
-  // NOOPT-NEXT:  [[A:%.*]] = load <27 x i32>, <27 x i32>* %0, align 4{{$}}
+  // NOOPT-NEXT:  [[A:%.*]] = load <27 x i32>, ptr %a.addr, align 4{{$}}
   // OPT-NEXT:    [[CMP:%.*]] = icmp ult i64 [[IDX1_2]], 27
   // OPT-NEXT:    call void @llvm.assume(i1 [[CMP]])
-  // OPT-NEXT:    [[A:%.*]] = load <27 x i32>, <27 x i32>* %0, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[A:%.*]] = load <27 x i32>, ptr %a.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[MI1:%.*]] = extractelement <27 x i32> [[A]], i64 [[IDX1_2]]
   // CHECK-NEXT:  [[MI1_EXT:%.*]] = sext i32 [[MI1]] to i64
-  // NOOPT-NEXT:  [[J2:%.*]] = load i32, i32* %j.addr, align 4{{$}}
-  // OPT-NEXT:    [[J2:%.*]] = load i32, i32* %j.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:  [[J2:%.*]] = load i32, ptr %j.addr, align 4{{$}}
+  // OPT-NEXT:    [[J2:%.*]] = load i32, ptr %j.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[J2_EXT:%.*]] = sext i32 [[J2]] to i64
-  // NOOPT-NEXT:  [[I2:%.*]] = load i32, i32* %i.addr, align 4{{$}}
-  // OPT-NEXT:    [[I2:%.*]] = load i32, i32* %i.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // NOOPT-NEXT:  [[I2:%.*]] = load i32, ptr %i.addr, align 4{{$}}
+  // OPT-NEXT:    [[I2:%.*]] = load i32, ptr %i.addr, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[I2_EXT:%.*]] = sext i32 [[I2]] to i64
   // CHECK-NEXT:  [[IDX2_1:%.*]] = mul i64 [[I2_EXT]], 9
   // CHECK-NEXT:  [[IDX2_2:%.*]] = add i64 [[IDX2_1]], [[J2_EXT]]
-  // NOOPT-NEXT:  [[A2:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4{{$}}
+  // NOOPT-NEXT:  [[A2:%.*]] = load <27 x i32>, ptr {{.*}}, align 4{{$}}
   // OPT-NEXT:    [[CMP:%.*]] = icmp ult i64 [[IDX2_2]], 27
   // OPT-NEXT:    call void @llvm.assume(i1 [[CMP]])
-  // OPT-NEXT:    [[A2:%.*]] = load <27 x i32>, <27 x i32>* {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
+  // OPT-NEXT:    [[A2:%.*]] = load <27 x i32>, ptr {{.*}}, align 4, !tbaa !{{[0-9]+}}{{$}}
   // CHECK-NEXT:  [[MI2:%.*]] = extractelement <27 x i32> [[A2]], i64 [[IDX2_2]]
   // CHECK-NEXT:  [[MI3:%.*]] = add nsw i32 [[MI2]], 2
   // CHECK-NEXT:  [[MI3_EXT:%.*]] = sext i32 [[MI3]] to i64
@@ -1293,8 +1282,8 @@ void matrix_as_idx(ix9x3_t a, int i, int j, dx5x5_t b) {
   // CHECK-NEXT:  [[IDX3_2:%.*]] = add i64 [[IDX3_1]], [[MI1_EXT]]
   // OPT-NEXT:    [[CMP:%.*]] = icmp ult i64 [[IDX3_2]], 25
   // OPT-NEXT:    call void @llvm.assume(i1 [[CMP]])
-  // CHECK-NEXT:  [[B:%.*]] = load <25 x double>, <25 x double>* [[B_PTR:%.*]], align 8{{$}}
+  // CHECK-NEXT:  [[B:%.*]] = load <25 x double>, ptr [[B_PTR:%.*]], align 8{{$}}
   // CHECK-NEXT:  [[INS:%.*]] = insertelement <25 x double> [[B]], double 1.500000e+00, i64 [[IDX3_2]]
-  // CHECK-NEXT:  store <25 x double> [[INS]], <25 x double>* [[B_PTR]], align 8
+  // CHECK-NEXT:  store <25 x double> [[INS]], ptr [[B_PTR]], align 8
   b[a[i][j]][a[j][i] + 2] = 1.5;
 }
diff --git a/clang/test/CodeGen/matrix-type.c b/clang/test/CodeGen/matrix-type.c
index 5025a344a0efd..e52dc20229db9 100644
--- a/clang/test/CodeGen/matrix-type.c
+++ b/clang/test/CodeGen/matrix-type.c
@@ -1,4 +1,4 @@
-// RUN: %clang_cc1 -no-opaque-pointers -fenable-matrix -triple x86_64-apple-darwin %s -emit-llvm -disable-llvm-passes -o - | FileCheck %s
+// RUN: %clang_cc1 -fenable-matrix -triple x86_64-apple-darwin %s -emit-llvm -disable-llvm-passes -o - | FileCheck %s
 
 #if !__has_extension(matrix_types)
 #error Expected extension 'matrix_types' to be enabled
@@ -15,16 +15,14 @@ typedef double dx5x5_t __attribute__((matrix_type(5, 5)));
 void load_store_double(dx5x5_t *a, dx5x5_t *b) {
   // CHECK-LABEL:  define{{.*}} void @load_store_double(
   // CHECK-NEXT:  entry:
-  // CHECK-NEXT:    %a.addr = alloca [25 x double]*, align 8
-  // CHECK-NEXT:    %b.addr = alloca [25 x double]*, align 8
-  // CHECK-NEXT:    store [25 x double]* %a, [25 x double]** %a.addr, align 8
-  // CHECK-NEXT:    store [25 x double]* %b, [25 x double]** %b.addr, align 8
-  // CHECK-NEXT:    %0 = load [25 x double]*, [25 x double]** %b.addr, align 8
-  // CHECK-NEXT:    %1 = bitcast [25 x double]* %0 to <25 x double>*
-  // CHECK-NEXT:    %2 = load <25 x double>, <25 x double>* %1, align 8
-  // CHECK-NEXT:    %3 = load [25 x double]*, [25 x double]** %a.addr, align 8
-  // CHECK-NEXT:    %4 = bitcast [25 x double]* %3 to <25 x double>*
-  // CHECK-NEXT:    store <25 x double> %2, <25 x double>* %4, align 8
+  // CHECK-NEXT:    %a.addr = alloca ptr, align 8
+  // CHECK-NEXT:    %b.addr = alloca ptr, align 8
+  // CHECK-NEXT:    store ptr %a, ptr %a.addr, align 8
+  // CHECK-NEXT:    store ptr %b, ptr %b.addr, align 8
+  // CHECK-NEXT:    %0 = load ptr, ptr %b.addr, align 8
+  // CHECK-NEXT:    %1 = load <25 x double>, ptr %0, align 8
+  // CHECK-NEXT:    %2 = load ptr, ptr %a.addr, align 8
+  // CHECK-NEXT:    store <25 x double> %1, ptr %2, align 8
   // CHECK-NEXT:   ret void
 
   *a = *b;
@@ -34,16 +32,14 @@ typedef float fx3x4_t __attribute__((matrix_type(3, 4)));
 void load_store_float(fx3x4_t *a, fx3x4_t *b) {
   // CHECK-LABEL:  define{{.*}} void @load_store_float(
   // CHECK-NEXT:  entry:
-  // CHECK-NEXT:    %a.addr = alloca [12 x float]*, align 8
-  // CHECK-NEXT:    %b.addr = alloca [12 x float]*, align 8
-  // CHECK-NEXT:    store [12 x float]* %a, [12 x float]** %a.addr, align 8
-  // CHECK-NEXT:    store [12 x float]* %b, [12 x float]** %b.addr, align 8
-  // CHECK-NEXT:    %0 = load [12 x float]*, [12 x float]** %b.addr, align 8
-  // CHECK-NEXT:    %1 = bitcast [12 x float]* %0 to <12 x float>*
-  // CHECK-NEXT:    %2 = load <12 x float>, <12 x float>* %1, align 4
-  // CHECK-NEXT:    %3 = load [12 x float]*, [12 x float]** %a.addr, align 8
-  // CHECK-NEXT:    %4 = bitcast [12 x float]* %3 to <12 x float>*
-  // CHECK-NEXT:    store <12 x float> %2, <12 x float>* %4, align 4
+  // CHECK-NEXT:    %a.addr = alloca ptr, align 8
+  // CHECK-NEXT:    %b.addr = alloca ptr, align 8
+  // CHECK-NEXT:    store ptr %a, ptr %a.addr, align 8
+  // CHECK-NEXT:    store ptr %b, ptr %b.addr, align 8
+  // CHECK-NEXT:    %0 = load ptr, ptr %b.addr, align 8
+  // CHECK-NEXT:    %1 = load <12 x float>, ptr %0, align 4
+  // CHECK-NEXT:    %2 = load ptr, ptr %a.addr, align 8
+  // CHECK-NEXT:    store <12 x float> %1, ptr %2, align 4
   // CHECK-NEXT:   ret void
 
   *a = *b;
@@ -53,16 +49,14 @@ typedef int ix3x4_t __attribute__((matrix_type(4, 3)));
 void load_store_int(ix3x4_t *a, ix3x4_t *b) {
   // CHECK-LABEL:  define{{.*}} void @load_store_int(
   // CHECK-NEXT:  entry:
-  // CHECK-NEXT:    %a.addr = alloca [12 x i32]*, align 8
-  // CHECK-NEXT:    %b.addr = alloca [12 x i32]*, align 8
-  // CHECK-NEXT:    store [12 x i32]* %a, [12 x i32]** %a.addr, align 8
-  // CHECK-NEXT:    store [12 x i32]* %b, [12 x i32]** %b.addr, align 8
-  // CHECK-NEXT:    %0 = load [12 x i32]*, [12 x i32]** %b.addr, align 8
-  // CHECK-NEXT:    %1 = bitcast [12 x i32]* %0 to <12 x i32>*
-  // CHECK-NEXT:    %2 = load <12 x i32>, <12 x i32>* %1, align 4
-  // CHECK-NEXT:    %3 = load [12 x i32]*, [12 x i32]** %a.addr, align 8
-  // CHECK-NEXT:    %4 = bitcast [12 x i32]* %3 to <12 x i32>*
-  // CHECK-NEXT:    store <12 x i32> %2, <12 x i32>* %4, align 4
+  // CHECK-NEXT:    %a.addr = alloca ptr, align 8
+  // CHECK-NEXT:    %b.addr = alloca ptr, align 8
+  // CHECK-NEXT:    store ptr %a, ptr %a.addr, align 8
+  // CHECK-NEXT:    store ptr %b, ptr %b.addr, align 8
+  // CHECK-NEXT:    %0 = load ptr, ptr %b.addr, align 8
+  // CHECK-NEXT:    %1 = load <12 x i32>, ptr %0, align 4
+  // CHECK-NEXT:    %2 = load ptr, ptr %a.addr, align 8
+  // CHECK-NEXT:    store <12 x i32> %1, ptr %2, align 4
   // CHECK-NEXT:   ret void
 
   *a = *b;
@@ -72,16 +66,14 @@ typedef unsigned long long ullx3x4_t __attribute__((matrix_type(4, 3)));
 void load_store_ull(ullx3x4_t *a, ullx3x4_t *b) {
   // CHECK-LABEL:  define{{.*}} void @load_store_ull(
   // CHECK-NEXT:  entry:
-  // CHECK-NEXT:    %a.addr = alloca [12 x i64]*, align 8
-  // CHECK-NEXT:    %b.addr = alloca [12 x i64]*, align 8
-  // CHECK-NEXT:    store [12 x i64]* %a, [12 x i64]** %a.addr, align 8
-  // CHECK-NEXT:    store [12 x i64]* %b, [12 x i64]** %b.addr, align 8
-  // CHECK-NEXT:    %0 = load [12 x i64]*, [12 x i64]** %b.addr, align 8
-  // CHECK-NEXT:    %1 = bitcast [12 x i64]* %0 to <12 x i64>*
-  // CHECK-NEXT:    %2 = load <12 x i64>, <12 x i64>* %1, align 8
-  // CHECK-NEXT:    %3 = load [12 x i64]*, [12 x i64]** %a.addr, align 8
-  // CHECK-NEXT:    %4 = bitcast [12 x i64]* %3 to <12 x i64>*
-  // CHECK-NEXT:    store <12 x i64> %2, <12 x i64>* %4, align 8
+  // CHECK-NEXT:    %a.addr = alloca ptr, align 8
+  // CHECK-NEXT:    %b.addr = alloca ptr, align 8
+  // CHECK-NEXT:    store ptr %a, ptr %a.addr, align 8
+  // CHECK-NEXT:    store ptr %b, ptr %b.addr, align 8
+  // CHECK-NEXT:    %0 = load ptr, ptr %b.addr, align 8
+  // CHECK-NEXT:    %1 = load <12 x i64>, ptr %0, align 8
+  // CHECK-NEXT:    %2 = load ptr, ptr %a.addr, align 8
+  // CHECK-NEXT:    store <12 x i64> %1, ptr %2, align 8
   // CHECK-NEXT:   ret void
 
   *a = *b;
@@ -91,16 +83,14 @@ typedef __fp16 fp16x3x4_t __attribute__((matrix_type(4, 3)));
 void load_store_fp16(fp16x3x4_t *a, fp16x3x4_t *b) {
   // CHECK-LABEL:  define{{.*}} void @load_store_fp16(
   // CHECK-NEXT:  entry:
-  // CHECK-NEXT:    %a.addr = alloca [12 x half]*, align 8
-  // CHECK-NEXT:    %b.addr = alloca [12 x half]*, align 8
-  // CHECK-NEXT:    store [12 x half]* %a, [12 x half]** %a.addr, align 8
-  // CHECK-NEXT:    store [12 x half]* %b, [12 x half]** %b.addr, align 8
-  // CHECK-NEXT:    %0 = load [12 x half]*, [12 x half]** %b.addr, align 8
-  // CHECK-NEXT:    %1 = bitcast [12 x half]* %0 to <12 x half>*
-  // CHECK-NEXT:    %2 = load <12 x half>, <12 x half>* %1, align 2
-  // CHECK-NEXT:    %3 = load [12 x half]*, [12 x half]** %a.addr, align 8
-  // CHECK-NEXT:    %4 = bitcast [12 x half]* %3 to <12 x half>*
-  // CHECK-NEXT:    store <12 x half> %2, <12 x half>* %4, align 2
+  // CHECK-NEXT:    %a.addr = alloca ptr, align 8
+  // CHECK-NEXT:    %b.addr = alloca ptr, align 8
+  // CHECK-NEXT:    store ptr %a, ptr %a.addr, align 8
+  // CHECK-NEXT:    store ptr %b, ptr %b.addr, align 8
+  // CHECK-NEXT:    %0 = load ptr, ptr %b.addr, align 8
+  // CHECK-NEXT:    %1 = load <12 x half>, ptr %0, align 2
+  // CHECK-NEXT:    %2 = load ptr, ptr %a.addr, align 8
+  // CHECK-NEXT:    store <12 x half> %1, ptr %2, align 2
   // CHECK-NEXT:   ret void
 
   *a = *b;
@@ -112,14 +102,12 @@ void parameter_passing(fx3x3_t a, fx3x3_t *b) {
   // CHECK-LABEL: define{{.*}} void @parameter_passing(
   // CHECK-NEXT:  entry:
   // CHECK-NEXT:    %a.addr = alloca [9 x float], align 4
-  // CHECK-NEXT:    %b.addr = alloca [9 x float]*, align 8
-  // CHECK-NEXT:    %0 = bitcast [9 x float]* %a.addr to <9 x float>*
-  // CHECK-NEXT:    store <9 x float> %a, <9 x float>* %0, align 4
-  // CHECK-NEXT:    store [9 x float]* %b, [9 x float]** %b.addr, align 8
-  // CHECK-NEXT:    %1 = load <9 x float>, <9 x float>* %0, align 4
-  // CHECK-NEXT:    %2 = load [9 x float]*, [9 x float]** %b.addr, align 8
-  // CHECK-NEXT:    %3 = bitcast [9 x float]* %2 to <9 x float>*
-  // CHECK-NEXT:    store <9 x float> %1, <9 x float>* %3, align 4
+  // CHECK-NEXT:    %b.addr = alloca ptr, align 8
+  // CHECK-NEXT:    store <9 x float> %a, ptr %a.addr, align 4
+  // CHECK-NEXT:    store ptr %b, ptr %b.addr, align 8
+  // CHECK-NEXT:    %0 = load <9 x float>, ptr %a.addr, align 4
+  // CHECK-NEXT:    %1 = load ptr, ptr %b.addr, align 8
+  // CHECK-NEXT:    store <9 x float> %0, ptr %1, align 4
   // CHECK-NEXT:    ret void
   *b = a;
 }
@@ -127,12 +115,11 @@ void parameter_passing(fx3x3_t a, fx3x3_t *b) {
 fx3x3_t return_matrix(fx3x3_t *a) {
   // CHECK-LABEL: define{{.*}} <9 x float> @return_matrix
   // CHECK-NEXT:  entry:
-  // CHECK-NEXT:    %a.addr = alloca [9 x float]*, align 8
-  // CHECK-NEXT:    store [9 x float]* %a, [9 x float]** %a.addr, align 8
-  // CHECK-NEXT:    %0 = load [9 x float]*, [9 x float]** %a.addr, align 8
-  // CHECK-NEXT:    %1 = bitcast [9 x float]* %0 to <9 x float>*
-  // CHECK-NEXT:    %2 = load <9 x float>, <9 x float>* %1, align 4
-  // CHECK-NEXT:    ret <9 x float> %2
+  // CHECK-NEXT:    %a.addr = alloca ptr, align 8
+  // CHECK-NEXT:    store ptr %a, ptr %a.addr, align 8
+  // CHECK-NEXT:    %0 = load ptr, ptr %a.addr, align 8
+  // CHECK-NEXT:    %1 = load <9 x float>, ptr %0, align 4
+  // CHECK-NEXT:    ret <9 x float> %1
   return *a;
 }
 
@@ -145,18 +132,16 @@ typedef struct {
 void matrix_struct(Matrix *a, Matrix *b) {
   // CHECK-LABEL: define{{.*}} void @matrix_struct(
   // CHECK-NEXT:  entry:
-  // CHECK-NEXT:    %a.addr = alloca %struct.Matrix*, align 8
-  // CHECK-NEXT:    %b.addr = alloca %struct.Matrix*, align 8
-  // CHECK-NEXT:    store %struct.Matrix* %a, %struct.Matrix** %a.addr, align 8
-  // CHECK-NEXT:    store %struct.Matrix* %b, %struct.Matrix** %b.addr, align 8
-  // CHECK-NEXT:    %0 = load %struct.Matrix*, %struct.Matrix** %a.addr, align 8
-  // CHECK-NEXT:    %Data = getelementptr inbounds %struct.Matrix, %struct.Matrix* %0, i32 0, i32 1
-  // CHECK-NEXT:    %1 = bitcast [12 x float]* %Data to <12 x float>*
-  // CHECK-NEXT:    %2 = load <12 x float>, <12 x float>* %1, align 4
-  // CHECK-NEXT:    %3 = load %struct.Matrix*, %struct.Matrix** %b.addr, align 8
-  // CHECK-NEXT:    %Data1 = getelementptr inbounds %struct.Matrix, %struct.Matrix* %3, i32 0, i32 1
-  // CHECK-NEXT:    %4 = bitcast [12 x float]* %Data1 to <12 x float>*
-  // CHECK-NEXT:    store <12 x float> %2, <12 x float>* %4, align 4
+  // CHECK-NEXT:    %a.addr = alloca ptr, align 8
+  // CHECK-NEXT:    %b.addr = alloca ptr, align 8
+  // CHECK-NEXT:    store ptr %a, ptr %a.addr, align 8
+  // CHECK-NEXT:    store ptr %b, ptr %b.addr, align 8
+  // CHECK-NEXT:    %0 = load ptr, ptr %a.addr, align 8
+  // CHECK-NEXT:    %Data = getelementptr inbounds %struct.Matrix, ptr %0, i32 0, i32 1
+  // CHECK-NEXT:    %1 = load <12 x float>, ptr %Data, align 4
+  // CHECK-NEXT:    %2 = load ptr, ptr %b.addr, align 8
+  // CHECK-NEXT:    %Data1 = getelementptr inbounds %struct.Matrix, ptr %2, i32 0, i32 1
+  // CHECK-NEXT:    store <12 x float> %1, ptr %Data1, align 4
   // CHECK-NEXT:    ret void
   b->Data = a->Data;
 }
@@ -166,10 +151,8 @@ void matrix_inline_asm_memory_readwrite(void) {
   // CHECK-LABEL: define{{.*}} void @matrix_inline_asm_memory_readwrite()
   // CHECK-NEXT:  entry:
   // CHECK-NEXT:    [[ALLOCA:%.+]] = alloca [16 x double], align 8
-  // CHECK-NEXT:    [[PTR1:%.+]] = bitcast [16 x double]* [[ALLOCA]] to <16 x double>*
-  // CHECK-NEXT:    [[PTR2:%.+]] = bitcast [16 x double]* [[ALLOCA]] to <16 x double>*
-  // CHECK-NEXT:    [[VAL:%.+]] = load <16 x double>, <16 x double>* [[PTR2]], align 8
-  // CHECK-NEXT:    call void asm sideeffect "", "=*r|m,0,~{memory},~{dirflag},~{fpsr},~{flags}"(<16 x double>* elementtype(<16 x double>) [[PTR1]], <16 x double> [[VAL]])
+  // CHECK-NEXT:    [[VAL:%.+]] = load <16 x double>, ptr [[ALLOCA]], align 8
+  // CHECK-NEXT:    call void asm sideeffect "", "=*r|m,0,~{memory},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(<16 x double>) [[ALLOCA]], <16 x double> [[VAL]])
   // CHECK-NEXT:    ret void
 
   dx4x4_t m;
diff --git a/clang/test/CodeGen/memprof.cpp b/clang/test/CodeGen/memprof.cpp
index b246d1f086942..cb53eaac03e11 100644
--- a/clang/test/CodeGen/memprof.cpp
+++ b/clang/test/CodeGen/memprof.cpp
@@ -11,16 +11,7 @@
 
 // TODO: Use text profile inputs once that is available for memprof.
 //
-// The following commands were used to compile the source to instrumented
-// executables and collect raw binary format profiles:
-//
-// # Collect memory profile:
-// $ clang++ -fuse-ld=lld -no-pie -Wl,--no-rosegment -gmlt \
-//      -fdebug-info-for-profiling -mno-omit-leaf-frame-pointer \
-//      -fno-omit-frame-pointer -fno-optimize-sibling-calls -m64 -Wl,-build-id \
-//      memprof.cpp -o memprof.exe -fmemory-profile
-// $ env MEMPROF_OPTIONS=log_path=stdout ./memprof.exe > memprof.memprofraw
-//
+// To update the inputs below, run Inputs/update_memprof_inputs.sh
 // RUN: llvm-profdata merge %S/Inputs/memprof.memprofraw --profiled-binary %S/Inputs/memprof.exe -o %t.memprofdata
 
 // Profile use:
diff --git a/clang/test/CodeGen/mingw-long-double.c b/clang/test/CodeGen/mingw-long-double.c
index 306024f6ecf26..a50f8f0b3b633 100644
--- a/clang/test/CodeGen/mingw-long-double.c
+++ b/clang/test/CodeGen/mingw-long-double.c
@@ -1,10 +1,10 @@
-// RUN: %clang_cc1 -no-opaque-pointers -triple i686-windows-gnu -emit-llvm -o - %s \
+// RUN: %clang_cc1 -triple i686-windows-gnu -emit-llvm -o - %s \
 // RUN:    | FileCheck %s --check-prefix=GNU32
-// RUN: %clang_cc1 -no-opaque-pointers -triple i686-windows-gnu -emit-llvm -o - %s -mms-bitfields \
+// RUN: %clang_cc1 -triple i686-windows-gnu -emit-llvm -o - %s -mms-bitfields \
 // RUN:    | FileCheck %s --check-prefix=GNU32
-// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-windows-gnu -emit-llvm -o - %s \
+// RUN: %clang_cc1 -triple x86_64-windows-gnu -emit-llvm -o - %s \
 // RUN:    | FileCheck %s --check-prefix=GNU64
-// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-windows-msvc -emit-llvm -o - %s \
+// RUN: %clang_cc1 -triple x86_64-windows-msvc -emit-llvm -o - %s \
 // RUN:    | FileCheck %s --check-prefix=MSC64
 
 struct {
@@ -32,15 +32,15 @@ long double TestLD(long double x) {
   return x * x;
 }
 // GNU32: define dso_local x86_fp80 @TestLD(x86_fp80 noundef %x)
-// GNU64: define dso_local void @TestLD(x86_fp80* noalias sret(x86_fp80) align 16 %agg.result, x86_fp80* noundef %0)
+// GNU64: define dso_local void @TestLD(ptr noalias sret(x86_fp80) align 16 %agg.result, ptr noundef %0)
 // MSC64: define dso_local double @TestLD(double noundef %x)
 
 long double _Complex TestLDC(long double _Complex x) {
   return x * x;
 }
-// GNU32: define dso_local void @TestLDC({ x86_fp80, x86_fp80 }* noalias sret({ x86_fp80, x86_fp80 }) align 4 %agg.result, { x86_fp80, x86_fp80 }* noundef byval({ x86_fp80, x86_fp80 }) align 4 %x)
-// GNU64: define dso_local void @TestLDC({ x86_fp80, x86_fp80 }* noalias sret({ x86_fp80, x86_fp80 }) align 16 %agg.result, { x86_fp80, x86_fp80 }* noundef %x)
-// MSC64: define dso_local void @TestLDC({ double, double }* noalias sret({ double, double }) align 8 %agg.result, { double, double }* noundef %x)
+// GNU32: define dso_local void @TestLDC(ptr noalias sret({ x86_fp80, x86_fp80 }) align 4 %agg.result, ptr noundef byval({ x86_fp80, x86_fp80 }) align 4 %x)
+// GNU64: define dso_local void @TestLDC(ptr noalias sret({ x86_fp80, x86_fp80 }) align 16 %agg.result, ptr noundef %x)
+// MSC64: define dso_local void @TestLDC(ptr noalias sret({ double, double }) align 8 %agg.result, ptr noundef %x)
 
 // GNU32: declare dso_local void @__mulxc3
 // GNU64: declare dso_local void @__mulxc3
@@ -53,11 +53,9 @@ void VarArgLD(int a, ...) {
   __builtin_va_list ap;
   __builtin_va_start(ap, a);
   long double LD = __builtin_va_arg(ap, long double);
-  // GNU32-NOT: load x86_fp80*, x86_fp80**
-  // GNU32: load x86_fp80, x86_fp80*
-  // GNU64: load x86_fp80*, x86_fp80**
-  // GNU64: load x86_fp80, x86_fp80*
-  // MSC64-NOT: load double*, double**
-  // MSC64: load double, double*
+  // GNU32: load x86_fp80, ptr %argp.cur
+  // GNU64: [[P:%.*]] = load ptr, ptr %argp.cur
+  // GNU64: load x86_fp80, ptr [[P]]
+  // MSC64: load double, ptr %argp.cur
   __builtin_va_end(ap);
 }
diff --git a/clang/test/CodeGen/mips-inline-asm-modifiers.c b/clang/test/CodeGen/mips-inline-asm-modifiers.c
index 11c32fcbd7678..c85a585b080bf 100644
--- a/clang/test/CodeGen/mips-inline-asm-modifiers.c
+++ b/clang/test/CodeGen/mips-inline-asm-modifiers.c
@@ -1,4 +1,4 @@
-// RUN: %clang_cc1 -no-opaque-pointers -triple mipsel-unknown-linux -o - -emit-llvm %s \
+// RUN: %clang_cc1 -triple mipsel-unknown-linux -o - -emit-llvm %s \
 // RUN: | FileCheck %s
 
 // This checks that the frontend will accept inline asm operand modifiers
@@ -7,8 +7,8 @@ int printf(const char*, ...);
 
 typedef int v4i32 __attribute__((vector_size(16)));
 
-  // CHECK: %{{[0-9]+}} = call i32 asm ".set noreorder;\0Alw    $0,$1;\0A.set reorder;\0A", "=r,*m,~{$1}"(i32* elementtype(i32) getelementptr inbounds ([8 x i32], [8 x i32]* @b, i32 {{[0-9]+}}, i32 {{[0-9]+}})) #2,
-  // CHECK: %{{[0-9]+}} = call i32 asm "lw    $0,${1:D};\0A", "=r,*m,~{$1}"(i32* elementtype(i32) getelementptr inbounds ([8 x i32], [8 x i32]* @b, i32 {{[0-9]+}}, i32 {{[0-9]+}})) #2,
+  // CHECK: %{{[0-9]+}} = call i32 asm ".set noreorder;\0Alw    $0,$1;\0A.set reorder;\0A", "=r,*m,~{$1}"(ptr elementtype(i32) getelementptr inbounds (i32, ptr @b, i32 {{[0-9]+}})) #2,
+  // CHECK: %{{[0-9]+}} = call i32 asm "lw    $0,${1:D};\0A", "=r,*m,~{$1}"(ptr elementtype(i32) getelementptr inbounds (i32, ptr @b, i32 {{[0-9]+}})) #2,
   // CHECK: %{{[0-9]+}} = call <4 x i32> asm "ldi.w ${0:w},1", "=f,~{$1}"
 int b[8] = {0,1,2,3,4,5,6,7};
 int  main(void)
diff --git a/clang/test/CodeGen/mips-varargs.c b/clang/test/CodeGen/mips-varargs.c
index 54c96c061f6b6..052aedd1cd1e2 100644
--- a/clang/test/CodeGen/mips-varargs.c
+++ b/clang/test/CodeGen/mips-varargs.c
@@ -1,9 +1,9 @@
-// RUN: %clang_cc1 -no-opaque-pointers -triple mips-unknown-linux -o - -emit-llvm %s | FileCheck %s -check-prefixes=ALL,O32 -enable-var-scope
-// RUN: %clang_cc1 -no-opaque-pointers -triple mipsel-unknown-linux -o - -emit-llvm %s | FileCheck %s -check-prefixes=ALL,O32 -enable-var-scope
-// RUN: %clang_cc1 -no-opaque-pointers -triple mips64-unknown-linux -o - -emit-llvm  -target-abi n32 %s | FileCheck %s -check-prefixes=ALL,N32,NEW -enable-var-scope
-// RUN: %clang_cc1 -no-opaque-pointers -triple mips64-unknown-linux -o - -emit-llvm  -target-abi n32 %s | FileCheck %s -check-prefixes=ALL,N32,NEW -enable-var-scope
-// RUN: %clang_cc1 -no-opaque-pointers -triple mips64-unknown-linux -o - -emit-llvm %s | FileCheck %s -check-prefixes=ALL,N64,NEW -enable-var-scope
-// RUN: %clang_cc1 -no-opaque-pointers -triple mips64el-unknown-linux -o - -emit-llvm %s | FileCheck %s -check-prefixes=ALL,N64,NEW -enable-var-scope
+// RUN: %clang_cc1 -triple mips-unknown-linux -o - -emit-llvm %s | FileCheck %s -check-prefixes=ALL,O32 -enable-var-scope
+// RUN: %clang_cc1 -triple mipsel-unknown-linux -o - -emit-llvm %s | FileCheck %s -check-prefixes=ALL,O32 -enable-var-scope
+// RUN: %clang_cc1 -triple mips64-unknown-linux -o - -emit-llvm  -target-abi n32 %s | FileCheck %s -check-prefixes=ALL,N32,NEW -enable-var-scope
+// RUN: %clang_cc1 -triple mips64-unknown-linux -o - -emit-llvm  -target-abi n32 %s | FileCheck %s -check-prefixes=ALL,N32,NEW -enable-var-scope
+// RUN: %clang_cc1 -triple mips64-unknown-linux -o - -emit-llvm %s | FileCheck %s -check-prefixes=ALL,N64,NEW -enable-var-scope
+// RUN: %clang_cc1 -triple mips64el-unknown-linux -o - -emit-llvm %s | FileCheck %s -check-prefixes=ALL,N64,NEW -enable-var-scope
 
 #include <stdarg.h>
 
@@ -19,38 +19,33 @@ int test_i32(char *fmt, ...) {
   return v;
 }
 
-// O32-LABEL: define{{.*}} i32 @test_i32(i8*{{.*}} %fmt, ...)
-// N32-LABEL: define{{.*}} signext i32 @test_i32(i8*{{.*}} %fmt, ...)
-// N64-LABEL: define{{.*}} signext i32 @test_i32(i8*{{.*}} %fmt, ...)
+// O32-LABEL: define{{.*}} i32 @test_i32(ptr{{.*}} %fmt, ...)
+// N32-LABEL: define{{.*}} signext i32 @test_i32(ptr{{.*}} %fmt, ...)
+// N64-LABEL: define{{.*}} signext i32 @test_i32(ptr{{.*}} %fmt, ...)
 //
-// O32:   %va = alloca i8*, align [[$PTRALIGN:4]]
-// N32:   %va = alloca i8*, align [[$PTRALIGN:4]]
-// N64:   %va = alloca i8*, align [[$PTRALIGN:8]]
+// O32:   %va = alloca ptr, align [[$PTRALIGN:4]]
+// N32:   %va = alloca ptr, align [[$PTRALIGN:4]]
+// N64:   %va = alloca ptr, align [[$PTRALIGN:8]]
 // ALL:   [[V:%.*]] = alloca i32, align 4
 // NEW:   [[PROMOTION_TEMP:%.*]] = alloca i32, align 4
 //
-// ALL:   [[VA:%.+]] = bitcast i8** %va to i8*
-// ALL:   call void @llvm.va_start(i8* [[VA]])
-// ALL:   [[AP_CUR:%.+]] = load i8*, i8** %va, align [[$PTRALIGN]]
-// O32:   [[AP_NEXT:%.+]] = getelementptr inbounds i8, i8* [[AP_CUR]], [[$INTPTR_T:i32]] [[$CHUNKSIZE:4]]
-// NEW:   [[AP_NEXT:%.+]] = getelementptr inbounds i8, i8* [[AP_CUR]], [[$INTPTR_T:i32|i64]] [[$CHUNKSIZE:8]]
+// ALL:   call void @llvm.va_start(ptr %va)
+// ALL:   [[AP_CUR:%.+]] = load ptr, ptr %va, align [[$PTRALIGN]]
+// O32:   [[AP_NEXT:%.+]] = getelementptr inbounds i8, ptr [[AP_CUR]], [[$INTPTR_T:i32]] [[$CHUNKSIZE:4]]
+// NEW:   [[AP_NEXT:%.+]] = getelementptr inbounds i8, ptr [[AP_CUR]], [[$INTPTR_T:i32|i64]] [[$CHUNKSIZE:8]]
 //
-// ALL:   store i8* [[AP_NEXT]], i8** %va, align [[$PTRALIGN]]
+// ALL:   store ptr [[AP_NEXT]], ptr %va, align [[$PTRALIGN]]
 //
-// O32:   [[AP_CAST:%.+]] = bitcast i8* [[AP_CUR]] to [[CHUNK_T:i32]]*
-// O32:   [[ARG:%.+]] = load i32, i32* [[AP_CAST]], align [[CHUNKALIGN:4]]
+// O32:   [[ARG:%.+]] = load i32, ptr [[AP_CUR]], align [[CHUNKALIGN:4]]
 //
-// N32:   [[AP_CAST:%.+]] = bitcast i8* [[AP_CUR]] to [[CHUNK_T:i64]]*
-// N32:   [[TMP:%.+]] = load i64, i64* [[AP_CAST]], align [[CHUNKALIGN:8]]
-// N64:   [[AP_CAST:%.+]] = bitcast i8* [[AP_CUR]] to [[CHUNK_T:i64]]*
-// N64:   [[TMP:%.+]] = load i64, i64* [[AP_CAST]], align [[CHUNKALIGN:8]]
+// N32:   [[TMP:%.+]] = load i64, ptr [[AP_CUR]], align [[CHUNKALIGN:8]]
+// N64:   [[TMP:%.+]] = load i64, ptr [[AP_CUR]], align [[CHUNKALIGN:8]]
 // NEW:   [[TMP2:%.+]] = trunc i64 [[TMP]] to i32
-// NEW:   store i32 [[TMP2]], i32* [[PROMOTION_TEMP]], align 4
-// NEW:   [[ARG:%.+]] = load i32, i32* [[PROMOTION_TEMP]], align 4
-// ALL:   store i32 [[ARG]], i32* [[V]], align 4
+// NEW:   store i32 [[TMP2]], ptr [[PROMOTION_TEMP]], align 4
+// NEW:   [[ARG:%.+]] = load i32, ptr [[PROMOTION_TEMP]], align 4
+// ALL:   store i32 [[ARG]], ptr [[V]], align 4
 //
-// ALL:   [[VA1:%.+]] = bitcast i8** %va to i8*
-// ALL:   call void @llvm.va_end(i8* [[VA1]])
+// ALL:   call void @llvm.va_end(ptr %va)
 // ALL: }
 
 long long test_i64(char *fmt, ...) {
@@ -63,28 +58,23 @@ long long test_i64(char *fmt, ...) {
   return v;
 }
 
-// ALL-LABEL: define{{.*}} i64 @test_i64(i8*{{.*}} %fmt, ...)
+// ALL-LABEL: define{{.*}} i64 @test_i64(ptr{{.*}} %fmt, ...)
 //
-// ALL:   %va = alloca i8*, align [[$PTRALIGN]]
-// ALL:   [[VA:%.+]] = bitcast i8** %va to i8*
-// ALL:   call void @llvm.va_start(i8* [[VA]])
-// ALL:   [[AP_CUR:%.+]] = load i8*, i8** %va, align [[$PTRALIGN]]
+// ALL:   %va = alloca ptr, align [[$PTRALIGN]]
+// ALL:   call void @llvm.va_start(ptr %va)
+// ALL:   [[AP_CUR:%.+]] = load ptr, ptr %va, align [[$PTRALIGN]]
 //
 // i64 is 8-byte aligned, while this is within O32's stack alignment there's no
 // guarantee that the offset is still 8-byte aligned after earlier reads.
-// O32:   [[TMP1:%.+]] = ptrtoint i8* [[AP_CUR]] to i32
-// O32:   [[TMP2:%.+]] = add i32 [[TMP1]], 7
-// O32:   [[TMP3:%.+]] = and i32 [[TMP2]], -8
-// O32:   [[AP_CUR:%.+]] = inttoptr i32 [[TMP3]] to i8*
+// O32:   [[TMP1:%.+]] = getelementptr inbounds i8, ptr [[AP_CUR]], i32 7
+// O32:   [[AP_CUR:%.+]] = call ptr @llvm.ptrmask.p0.i32(ptr [[TMP1]], i32 -8)
 //
-// ALL:   [[AP_NEXT:%.+]] = getelementptr inbounds i8, i8* [[AP_CUR]], [[$INTPTR_T]] 8
-// ALL:   store i8* [[AP_NEXT]], i8** %va, align [[$PTRALIGN]]
+// ALL:   [[AP_NEXT:%.+]] = getelementptr inbounds i8, ptr [[AP_CUR]], [[$INTPTR_T]] 8
+// ALL:   store ptr [[AP_NEXT]], ptr %va, align [[$PTRALIGN]]
 //
-// ALL:   [[AP_CAST:%.*]] = bitcast i8* [[AP_CUR]] to i64*
-// ALL:   [[ARG:%.+]] = load i64, i64* [[AP_CAST]], align 8
+// ALL:   [[ARG:%.+]] = load i64, ptr [[AP_CUR]], align 8
 //
-// ALL:   [[VA1:%.+]] = bitcast i8** %va to i8*
-// ALL:   call void @llvm.va_end(i8* [[VA1]])
+// ALL:   call void @llvm.va_end(ptr %va)
 // ALL: }
 
 char *test_ptr(char *fmt, ...) {
@@ -97,32 +87,29 @@ char *test_ptr(char *fmt, ...) {
   return v;
 }
 
-// ALL-LABEL: define{{.*}} i8* @test_ptr(i8*{{.*}} %fmt, ...)
+// ALL-LABEL: define{{.*}} ptr @test_ptr(ptr{{.*}} %fmt, ...)
 //
-// ALL:   %va = alloca i8*, align [[$PTRALIGN]]
-// ALL:   [[V:%.*]] = alloca i8*, align [[$PTRALIGN]]
-// N32:   [[AP_CAST:%.+]] = alloca i8*, align 4
-// ALL:   [[VA:%.+]] = bitcast i8** %va to i8*
-// ALL:   call void @llvm.va_start(i8* [[VA]])
-// ALL:   [[AP_CUR:%.+]] = load i8*, i8** %va, align [[$PTRALIGN]]
-// ALL:   [[AP_NEXT:%.+]] = getelementptr inbounds i8, i8* [[AP_CUR]], [[$INTPTR_T]] [[$CHUNKSIZE]]
-// ALL:   store i8* [[AP_NEXT]], i8** %va, align [[$PTRALIGN]]
+// ALL:   %va = alloca ptr, align [[$PTRALIGN]]
+// ALL:   [[V:%.*]] = alloca ptr, align [[$PTRALIGN]]
+// N32:   [[AP_CAST:%.+]] = alloca ptr, align 4
+// ALL:   call void @llvm.va_start(ptr %va)
+// ALL:   [[AP_CUR:%.+]] = load ptr, ptr %va, align [[$PTRALIGN]]
+// ALL:   [[AP_NEXT:%.+]] = getelementptr inbounds i8, ptr [[AP_CUR]], [[$INTPTR_T]] [[$CHUNKSIZE]]
+// ALL:   store ptr [[AP_NEXT]], ptr %va, align [[$PTRALIGN]]
 //
 // When the chunk size matches the pointer size, this is easy.
-// O32:   [[AP_CAST:%.+]] = bitcast i8* [[AP_CUR]] to i8**
-// N64:   [[AP_CAST:%.+]] = bitcast i8* [[AP_CUR]] to i8**
 // Otherwise we need a promotion temporary.
-// N32:   [[TMP1:%.+]] = bitcast i8* [[AP_CUR]] to i64*
-// N32:   [[TMP2:%.+]] = load i64, i64* [[TMP1]], align 8
+// N32:   [[TMP2:%.+]] = load i64, ptr [[AP_CUR]], align 8
 // N32:   [[TMP3:%.+]] = trunc i64 [[TMP2]] to i32
-// N32:   [[PTR:%.+]] = inttoptr i32 [[TMP3]] to i8*
-// N32:   store i8* [[PTR]], i8** [[AP_CAST]], align 4
+// N32:   [[PTR:%.+]] = inttoptr i32 [[TMP3]] to ptr
+// N32:   store ptr [[PTR]], ptr [[AP_CAST]], align 4
+// N32:   [[ARG:%.+]] = load ptr, ptr [[AP_CAST]], align [[$PTRALIGN]]
 //
-// ALL:   [[ARG:%.+]] = load i8*, i8** [[AP_CAST]], align [[$PTRALIGN]]
-// ALL:   store i8* [[ARG]], i8** [[V]], align [[$PTRALIGN]]
+// O32:   [[ARG:%.+]] = load ptr, ptr [[AP_CUR]], align [[$PTRALIGN]]
+// N64:   [[ARG:%.+]] = load ptr, ptr [[AP_CUR]], align [[$PTRALIGN]]
+// ALL:   store ptr [[ARG]], ptr [[V]], align [[$PTRALIGN]]
 //
-// ALL:   [[VA1:%.+]] = bitcast i8** %va to i8*
-// ALL:   call void @llvm.va_end(i8* [[VA1]])
+// ALL:   call void @llvm.va_end(ptr %va)
 // ALL: }
 
 int test_v4i32(char *fmt, ...) {
@@ -135,39 +122,37 @@ int test_v4i32(char *fmt, ...) {
   return v[0];
 }
 
-// O32-LABEL: define{{.*}} i32 @test_v4i32(i8*{{.*}} %fmt, ...)
-// N32-LABEL: define{{.*}} signext i32 @test_v4i32(i8*{{.*}} %fmt, ...)
-// N64-LABEL: define{{.*}} signext i32 @test_v4i32(i8*{{.*}} %fmt, ...)
+// O32-LABEL: define{{.*}} i32 @test_v4i32(ptr{{.*}} %fmt, ...)
+// N32-LABEL: define{{.*}} signext i32 @test_v4i32(ptr{{.*}} %fmt, ...)
+// N64-LABEL: define{{.*}} signext i32 @test_v4i32(ptr{{.*}} %fmt, ...)
 //
-// ALL:   %va = alloca i8*, align [[$PTRALIGN]]
+// ALL:   %va = alloca ptr, align [[$PTRALIGN]]
 // ALL:   [[V:%.+]] = alloca <4 x i32>, align 16
-// ALL:   [[VA1:%.+]] = bitcast i8** %va to i8*
-// ALL:   call void @llvm.va_start(i8* [[VA1]])
-// ALL:   [[AP_CUR:%.+]] = load i8*, i8** %va, align [[$PTRALIGN]]
+// ALL:   call void @llvm.va_start(ptr %va)
+// ALL:   [[AP_CUR:%.+]] = load ptr, ptr %va, align [[$PTRALIGN]]
 //
 // Vectors are 16-byte aligned, however the O32 ABI has a maximum alignment of
 // 8-bytes since the base of the stack is 8-byte aligned.
-// O32:   [[TMP1:%.+]] = ptrtoint i8* [[AP_CUR]] to i32
-// O32:   [[TMP2:%.+]] = add i32 [[TMP1]], 7
-// O32:   [[TMP3:%.+]] = and i32 [[TMP2]], -8
-// O32:   [[AP_CUR:%.+]] = inttoptr i32 [[TMP3]] to i8*
-//
-// NEW:   [[TMP1:%.+]] = ptrtoint i8* [[AP_CUR]] to [[$INTPTR_T]]
-// NEW:   [[TMP2:%.+]] = add [[$INTPTR_T]] [[TMP1]], 15
-// NEW:   [[TMP3:%.+]] = and [[$INTPTR_T]] [[TMP2]], -16
-// NEW:   [[AP_CUR:%.+]] = inttoptr [[$INTPTR_T]] [[TMP3]] to i8*
-//
-// ALL:   [[AP_NEXT:%.+]] = getelementptr inbounds i8, i8* [[AP_CUR]], [[$INTPTR_T]] 16
-// ALL:   store i8* [[AP_NEXT]], i8** %va, align [[$PTRALIGN]]
-//
-// ALL:   [[AP_CAST:%.+]] = bitcast i8* [[AP_CUR]] to <4 x i32>*
-// O32:   [[ARG:%.+]] = load <4 x i32>, <4 x i32>* [[AP_CAST]], align 8
-// N64:   [[ARG:%.+]] = load <4 x i32>, <4 x i32>* [[AP_CAST]], align 16
-// N32:   [[ARG:%.+]] = load <4 x i32>, <4 x i32>* [[AP_CAST]], align 16
-// ALL:   store <4 x i32> [[ARG]], <4 x i32>* [[V]], align 16
-//
-// ALL:   [[VA1:%.+]] = bitcast i8** %va to i8*
-// ALL:   call void @llvm.va_end(i8* [[VA1]])
+
+// O32:   [[TMP1:%.+]] = getelementptr inbounds i8, ptr [[AP_CUR]], i32 7
+// O32:   [[AP_CUR:%.+]] = call ptr @llvm.ptrmask.p0.i32(ptr [[TMP1]], i32 -8)
+
+// N32:   [[TMP1:%.+]] = getelementptr inbounds i8, ptr [[AP_CUR]], i32 15
+// N32:   [[AP_CUR:%.+]] = call ptr @llvm.ptrmask.p0.i32(ptr [[TMP1]], i32 -16)
+
+// N64:   [[TMP1:%.+]] = getelementptr inbounds i8, ptr [[AP_CUR]], i32 15
+// N64:   [[AP_CUR:%.+]] = call ptr @llvm.ptrmask.p0.i64(ptr [[TMP1]], i64 -16)
+
+//
+// ALL:   [[AP_NEXT:%.+]] = getelementptr inbounds i8, ptr [[AP_CUR]], [[$INTPTR_T]] 16
+// ALL:   store ptr [[AP_NEXT]], ptr %va, align [[$PTRALIGN]]
+//
+// O32:   [[ARG:%.+]] = load <4 x i32>, ptr [[AP_CUR]], align 8
+// N64:   [[ARG:%.+]] = load <4 x i32>, ptr [[AP_CUR]], align 16
+// N32:   [[ARG:%.+]] = load <4 x i32>, ptr [[AP_CUR]], align 16
+// ALL:   store <4 x i32> [[ARG]], ptr [[V]], align 16
+//
+// ALL:   call void @llvm.va_end(ptr %va)
 // ALL:   [[VECEXT:%.+]] = extractelement <4 x i32> {{.*}}, i32 0
 // ALL:   ret i32 [[VECEXT]]
 // ALL: }
diff --git a/clang/test/CodeGen/ms-intrinsics.c b/clang/test/CodeGen/ms-intrinsics.c
index 55e9dc0adf7db..3a9e04433a32f 100644
--- a/clang/test/CodeGen/ms-intrinsics.c
+++ b/clang/test/CodeGen/ms-intrinsics.c
@@ -1,13 +1,13 @@
-// RUN: %clang_cc1 -no-opaque-pointers -ffreestanding -fms-extensions -fms-compatibility -fms-compatibility-version=17.00 \
+// RUN: %clang_cc1 -ffreestanding -fms-extensions -fms-compatibility -fms-compatibility-version=17.00 \
 // RUN:         -triple i686--windows -Oz -emit-llvm %s -o - \
 // RUN:         | FileCheck %s -check-prefixes CHECK,CHECK-I386,CHECK-INTEL
-// RUN: %clang_cc1 -no-opaque-pointers -ffreestanding -fms-extensions -fms-compatibility -fms-compatibility-version=17.00 \
+// RUN: %clang_cc1 -ffreestanding -fms-extensions -fms-compatibility -fms-compatibility-version=17.00 \
 // RUN:         -triple thumbv7--windows -Oz -emit-llvm %s -o - \
 // RUN:         | FileCheck %s --check-prefixes CHECK,CHECK-ARM,CHECK-ARM-ARM64,CHECK-ARM-X64
-// RUN: %clang_cc1 -no-opaque-pointers -ffreestanding -fms-extensions -fms-compatibility -fms-compatibility-version=17.00 \
+// RUN: %clang_cc1 -ffreestanding -fms-extensions -fms-compatibility -fms-compatibility-version=17.00 \
 // RUN:         -triple x86_64--windows -Oz -emit-llvm -target-feature +cx16 %s -o - \
 // RUN:         | FileCheck %s --check-prefixes CHECK,CHECK-X64,CHECK-ARM-X64,CHECK-INTEL,CHECK-64
-// RUN: %clang_cc1 -no-opaque-pointers -ffreestanding -fms-extensions -fms-compatibility -fms-compatibility-version=17.00 \
+// RUN: %clang_cc1 -ffreestanding -fms-extensions -fms-compatibility -fms-compatibility-version=17.00 \
 // RUN:         -triple aarch64-windows -Oz -emit-llvm %s -o - \
 // RUN:         | FileCheck %s --check-prefixes CHECK-ARM-ARM64,CHECK-ARM-X64,CHECK-ARM64,CHECK-64
 
@@ -23,12 +23,12 @@ void test__stosb(unsigned char *Dest, unsigned char Data, size_t Count) {
 }
 
 // CHECK-I386: define{{.*}}void @test__stosb
-// CHECK-I386:   tail call void @llvm.memset.p0i8.i32(i8* align 1 %Dest, i8 %Data, i32 %Count, i1 true)
+// CHECK-I386:   tail call void @llvm.memset.p0.i32(ptr align 1 %Dest, i8 %Data, i32 %Count, i1 true)
 // CHECK-I386:   ret void
 // CHECK-I386: }
 
 // CHECK-X64: define{{.*}}void @test__stosb
-// CHECK-X64:   tail call void @llvm.memset.p0i8.i64(i8* align 1 %Dest, i8 %Data, i64 %Count, i1 true)
+// CHECK-X64:   tail call void @llvm.memset.p0.i64(ptr align 1 %Dest, i8 %Data, i64 %Count, i1 true)
 // CHECK-X64:   ret void
 // CHECK-X64: }
 
@@ -36,12 +36,12 @@ void test__movsb(unsigned char *Dest, unsigned char *Src, size_t Count) {
   return __movsb(Dest, Src, Count);
 }
 // CHECK-I386-LABEL: define{{.*}} void @test__movsb
-// CHECK-I386:   tail call { i8*, i8*, i32 } asm sideeffect "xchg $(%esi, $1$|$1, esi$)\0Arep movsb\0Axchg $(%esi, $1$|$1, esi$)", "={di},=r,={cx},0,1,2,~{memory},~{dirflag},~{fpsr},~{flags}"(i8* %Dest, i8* %Src, i32 %Count)
+// CHECK-I386:   tail call { ptr, ptr, i32 } asm sideeffect "xchg $(%esi, $1$|$1, esi$)\0Arep movsb\0Axchg $(%esi, $1$|$1, esi$)", "={di},=r,={cx},0,1,2,~{memory},~{dirflag},~{fpsr},~{flags}"(ptr %Dest, ptr %Src, i32 %Count)
 // CHECK-I386:   ret void
 // CHECK-I386: }
 
 // CHECK-X64-LABEL: define{{.*}} void @test__movsb
-// CHECK-X64:   call { i8*, i8*, i64 } asm sideeffect "rep movsb", "={di},={si},={cx},0,1,2,~{memory},~{dirflag},~{fpsr},~{flags}"(i8* %Dest, i8* %Src, i64 %Count)
+// CHECK-X64:   call { ptr, ptr, i64 } asm sideeffect "rep movsb", "={di},={si},={cx},0,1,2,~{memory},~{dirflag},~{fpsr},~{flags}"(ptr %Dest, ptr %Src, i64 %Count)
 // CHECK-X64:   ret void
 // CHECK-X64: }
 
@@ -49,12 +49,12 @@ void test__stosw(unsigned short *Dest, unsigned short Data, size_t Count) {
   return __stosw(Dest, Data, Count);
 }
 // CHECK-I386-LABEL: define{{.*}} void @test__stosw
-// CHECK-I386:   call { i16*, i32 } asm sideeffect "rep stosw", "={di},={cx},{ax},0,1,~{memory},~{dirflag},~{fpsr},~{flags}"(i16 %Data, i16* %Dest, i32 %Count)
+// CHECK-I386:   call { ptr, i32 } asm sideeffect "rep stosw", "={di},={cx},{ax},0,1,~{memory},~{dirflag},~{fpsr},~{flags}"(i16 %Data, ptr %Dest, i32 %Count)
 // CHECK-I386:   ret void
 // CHECK-I386: }
 
 // CHECK-X64-LABEL: define{{.*}} void @test__stosw
-// CHECK-X64:   call { i16*, i64 } asm sideeffect "rep stosw", "={di},={cx},{ax},0,1,~{memory},~{dirflag},~{fpsr},~{flags}"(i16 %Data, i16* %Dest, i64 %Count)
+// CHECK-X64:   call { ptr, i64 } asm sideeffect "rep stosw", "={di},={cx},{ax},0,1,~{memory},~{dirflag},~{fpsr},~{flags}"(i16 %Data, ptr %Dest, i64 %Count)
 // CHECK-X64:   ret void
 // CHECK-X64: }
 
@@ -62,12 +62,12 @@ void test__movsw(unsigned short *Dest, unsigned short *Src, size_t Count) {
   return __movsw(Dest, Src, Count);
 }
 // CHECK-I386-LABEL: define{{.*}} void @test__movsw
-// CHECK-I386:   tail call { i16*, i16*, i32 } asm sideeffect "xchg $(%esi, $1$|$1, esi$)\0Arep movsw\0Axchg $(%esi, $1$|$1, esi$)", "={di},=r,={cx},0,1,2,~{memory},~{dirflag},~{fpsr},~{flags}"(i16* %Dest, i16* %Src, i32 %Count)
+// CHECK-I386:   tail call { ptr, ptr, i32 } asm sideeffect "xchg $(%esi, $1$|$1, esi$)\0Arep movsw\0Axchg $(%esi, $1$|$1, esi$)", "={di},=r,={cx},0,1,2,~{memory},~{dirflag},~{fpsr},~{flags}"(ptr %Dest, ptr %Src, i32 %Count)
 // CHECK-I386:   ret void
 // CHECK-I386: }
 
 // CHECK-X64-LABEL: define{{.*}} void @test__movsw
-// CHECK-X64:   call { i16*, i16*, i64 } asm sideeffect "rep movsw", "={di},={si},={cx},0,1,2,~{memory},~{dirflag},~{fpsr},~{flags}"(i16* %Dest, i16* %Src, i64 %Count)
+// CHECK-X64:   call { ptr, ptr, i64 } asm sideeffect "rep movsw", "={di},={si},={cx},0,1,2,~{memory},~{dirflag},~{fpsr},~{flags}"(ptr %Dest, ptr %Src, i64 %Count)
 // CHECK-X64:   ret void
 // CHECK-X64: }
 
@@ -75,12 +75,12 @@ void test__stosd(unsigned long *Dest, unsigned long Data, size_t Count) {
   return __stosd(Dest, Data, Count);
 }
 // CHECK-I386-LABEL: define{{.*}} void @test__stosd
-// CHECK-I386:   call { i32*, i32 } asm sideeffect "rep stos$(l$|d$)", "={di},={cx},{ax},0,1,~{memory},~{dirflag},~{fpsr},~{flags}"(i32 %Data, i32* %Dest, i32 %Count)
+// CHECK-I386:   call { ptr, i32 } asm sideeffect "rep stos$(l$|d$)", "={di},={cx},{ax},0,1,~{memory},~{dirflag},~{fpsr},~{flags}"(i32 %Data, ptr %Dest, i32 %Count)
 // CHECK-I386:   ret void
 // CHECK-I386: }
 
 // CHECK-X64-LABEL: define{{.*}} void @test__stosd
-// CHECK-X64:   call { i32*, i64 } asm sideeffect "rep stos$(l$|d$)", "={di},={cx},{ax},0,1,~{memory},~{dirflag},~{fpsr},~{flags}"(i32 %Data, i32* %Dest, i64 %Count)
+// CHECK-X64:   call { ptr, i64 } asm sideeffect "rep stos$(l$|d$)", "={di},={cx},{ax},0,1,~{memory},~{dirflag},~{fpsr},~{flags}"(i32 %Data, ptr %Dest, i64 %Count)
 // CHECK-X64:   ret void
 // CHECK-X64: }
 
@@ -88,12 +88,12 @@ void test__movsd(unsigned long *Dest, unsigned long *Src, size_t Count) {
   return __movsd(Dest, Src, Count);
 }
 // CHECK-I386-LABEL: define{{.*}} void @test__movsd
-// CHECK-I386:   tail call { i32*, i32*, i32 } asm sideeffect "xchg $(%esi, $1$|$1, esi$)\0Arep movs$(l$|d$)\0Axchg $(%esi, $1$|$1, esi$)", "={di},=r,={cx},0,1,2,~{memory},~{dirflag},~{fpsr},~{flags}"(i32* %Dest, i32* %Src, i32 %Count)
+// CHECK-I386:   tail call { ptr, ptr, i32 } asm sideeffect "xchg $(%esi, $1$|$1, esi$)\0Arep movs$(l$|d$)\0Axchg $(%esi, $1$|$1, esi$)", "={di},=r,={cx},0,1,2,~{memory},~{dirflag},~{fpsr},~{flags}"(ptr %Dest, ptr %Src, i32 %Count)
 // CHECK-I386:   ret void
 // CHECK-I386: }
 
 // CHECK-X64-LABEL: define{{.*}} void @test__movsd
-// CHECK-X64:   call { i32*, i32*, i64 } asm sideeffect "rep movs$(l$|d$)", "={di},={si},={cx},0,1,2,~{memory},~{dirflag},~{fpsr},~{flags}"(i32* %Dest, i32* %Src, i64 %Count)
+// CHECK-X64:   call { ptr, ptr, i64 } asm sideeffect "rep movs$(l$|d$)", "={di},={si},={cx},0,1,2,~{memory},~{dirflag},~{fpsr},~{flags}"(ptr %Dest, ptr %Src, i64 %Count)
 // CHECK-X64:   ret void
 // CHECK-X64: }
 
@@ -102,7 +102,7 @@ void test__stosq(unsigned __int64 *Dest, unsigned __int64 Data, size_t Count) {
   return __stosq(Dest, Data, Count);
 }
 // CHECK-X64-LABEL: define{{.*}} void @test__stosq
-// CHECK-X64:   call { i64*, i64 } asm sideeffect "rep stosq", "={di},={cx},{ax},0,1,~{memory},~{dirflag},~{fpsr},~{flags}"(i64 %Data, i64* %Dest, i64 %Count)
+// CHECK-X64:   call { ptr, i64 } asm sideeffect "rep stosq", "={di},={cx},{ax},0,1,~{memory},~{dirflag},~{fpsr},~{flags}"(i64 %Data, ptr %Dest, i64 %Count)
 // CHECK-X64:   ret void
 // CHECK-X64: }
 
@@ -110,7 +110,7 @@ void test__movsq(unsigned __int64 *Dest, unsigned __int64 *Src, size_t Count) {
   return __movsq(Dest, Src, Count);
 }
 // CHECK-X64-LABEL: define{{.*}} void @test__movsq
-// CHECK-X64:   call { i64*, i64*, i64 } asm sideeffect "rep movsq", "={di},={si},={cx},0,1,2,~{memory},~{dirflag},~{fpsr},~{flags}"(i64* %Dest, i64* %Src, i64 %Count)
+// CHECK-X64:   call { ptr, ptr, i64 } asm sideeffect "rep movsq", "={di},={si},={cx},0,1,2,~{memory},~{dirflag},~{fpsr},~{flags}"(ptr %Dest, ptr %Src, i64 %Count)
 // CHECK-X64:   ret void
 // CHECK-X64: }
 #endif
@@ -133,55 +133,55 @@ void test__int2c(void) {
 void *test_ReturnAddress(void) {
   return _ReturnAddress();
 }
-// CHECK-LABEL: define{{.*}}i8* @test_ReturnAddress()
-// CHECK: = tail call i8* @llvm.returnaddress(i32 0)
-// CHECK: ret i8*
+// CHECK-LABEL: define{{.*}}ptr @test_ReturnAddress()
+// CHECK: = tail call ptr @llvm.returnaddress(i32 0)
+// CHECK: ret ptr
 
 #if defined(__i386__) || defined(__x86_64__) || defined (__aarch64__)
 void *test_AddressOfReturnAddress(void) {
   return _AddressOfReturnAddress();
 }
-// CHECK-INTEL-LABEL: define dso_local i8* @test_AddressOfReturnAddress()
-// CHECK-INTEL: = tail call i8* @llvm.addressofreturnaddress.p0i8()
-// CHECK-INTEL: ret i8*
+// CHECK-INTEL-LABEL: define dso_local ptr @test_AddressOfReturnAddress()
+// CHECK-INTEL: = tail call ptr @llvm.addressofreturnaddress.p0()
+// CHECK-INTEL: ret ptr
 #endif
 
 unsigned char test_BitScanForward(unsigned long *Index, unsigned long Mask) {
   return _BitScanForward(++Index, Mask);
 }
-// CHECK: define{{.*}}i8 @test_BitScanForward(i32* {{[a-z_ ]*}}%Index, i32 {{[a-z_ ]*}}%Mask){{.*}}{
+// CHECK: define{{.*}}i8 @test_BitScanForward(ptr {{[a-z_ ]*}}%Index, i32 {{[a-z_ ]*}}%Mask){{.*}}{
 // CHECK:   [[ISNOTZERO:%[a-z0-9._]+]] = icmp eq i32 %Mask, 0
 // CHECK:   br i1 [[ISNOTZERO]], label %[[END_LABEL:[a-z0-9._]+]], label %[[ISNOTZERO_LABEL:[a-z0-9._]+]]
 // CHECK:   [[END_LABEL]]:
 // CHECK:   [[RESULT:%[a-z0-9._]+]] = phi i8 [ 0, %[[ISZERO_LABEL:[a-z0-9._]+]] ], [ 1, %[[ISNOTZERO_LABEL]] ]
 // CHECK:   ret i8 [[RESULT]]
 // CHECK:   [[ISNOTZERO_LABEL]]:
-// CHECK:   [[IDXGEP:%[a-z0-9._]+]] = getelementptr inbounds i32, i32* %Index, {{i64|i32}} 1
+// CHECK:   [[IDXGEP:%[a-z0-9._]+]] = getelementptr inbounds i32, ptr %Index, {{i64|i32}} 1
 // CHECK:   [[INDEX:%[0-9]+]] = tail call i32 @llvm.cttz.i32(i32 %Mask, i1 true)
-// CHECK:   store i32 [[INDEX]], i32* [[IDXGEP]], align 4
+// CHECK:   store i32 [[INDEX]], ptr [[IDXGEP]], align 4
 // CHECK:   br label %[[END_LABEL]]
 
 unsigned char test_BitScanReverse(unsigned long *Index, unsigned long Mask) {
   return _BitScanReverse(++Index, Mask);
 }
-// CHECK: define{{.*}}i8 @test_BitScanReverse(i32* {{[a-z_ ]*}}%Index, i32 {{[a-z_ ]*}}%Mask){{.*}}{
+// CHECK: define{{.*}}i8 @test_BitScanReverse(ptr {{[a-z_ ]*}}%Index, i32 {{[a-z_ ]*}}%Mask){{.*}}{
 // CHECK:   [[ISNOTZERO:%[0-9]+]] = icmp eq i32 %Mask, 0
 // CHECK:   br i1 [[ISNOTZERO]], label %[[END_LABEL:[a-z0-9._]+]], label %[[ISNOTZERO_LABEL:[a-z0-9._]+]]
 // CHECK:   [[END_LABEL]]:
 // CHECK:   [[RESULT:%[a-z0-9._]+]] = phi i8 [ 0, %[[ISZERO_LABEL:[a-z0-9._]+]] ], [ 1, %[[ISNOTZERO_LABEL]] ]
 // CHECK:   ret i8 [[RESULT]]
 // CHECK:   [[ISNOTZERO_LABEL]]:
-// CHECK:   [[IDXGEP:%[a-z0-9._]+]] = getelementptr inbounds i32, i32* %Index, {{i64|i32}} 1
+// CHECK:   [[IDXGEP:%[a-z0-9._]+]] = getelementptr inbounds i32, ptr %Index, {{i64|i32}} 1
 // CHECK:   [[REVINDEX:%[0-9]+]] = tail call i32 @llvm.ctlz.i32(i32 %Mask, i1 true)
 // CHECK:   [[INDEX:%[0-9]+]] = xor i32 [[REVINDEX]], 31
-// CHECK:   store i32 [[INDEX]], i32* [[IDXGEP]], align 4
+// CHECK:   store i32 [[INDEX]], ptr [[IDXGEP]], align 4
 // CHECK:   br label %[[END_LABEL]]
 
 #if defined(__x86_64__) || defined(__arm__) || defined(__aarch64__)
 unsigned char test_BitScanForward64(unsigned long *Index, unsigned __int64 Mask) {
   return _BitScanForward64(Index, Mask);
 }
-// CHECK-ARM-X64: define{{.*}}i8 @test_BitScanForward64(i32* {{[a-z_ ]*}}%Index, i64 {{[a-z_ ]*}}%Mask){{.*}}{
+// CHECK-ARM-X64: define{{.*}}i8 @test_BitScanForward64(ptr {{[a-z_ ]*}}%Index, i64 {{[a-z_ ]*}}%Mask){{.*}}{
 // CHECK-ARM-X64:   [[ISNOTZERO:%[a-z0-9._]+]] = icmp eq i64 %Mask, 0
 // CHECK-ARM-X64:   br i1 [[ISNOTZERO]], label %[[END_LABEL:[a-z0-9._]+]], label %[[ISNOTZERO_LABEL:[a-z0-9._]+]]
 // CHECK-ARM-X64:   [[END_LABEL]]:
@@ -190,13 +190,13 @@ unsigned char test_BitScanForward64(unsigned long *Index, unsigned __int64 Mask)
 // CHECK-ARM-X64:   [[ISNOTZERO_LABEL]]:
 // CHECK-ARM-X64:   [[INDEX:%[0-9]+]] = tail call i64 @llvm.cttz.i64(i64 %Mask, i1 true)
 // CHECK-ARM-X64:   [[TRUNC_INDEX:%[0-9]+]] = trunc i64 [[INDEX]] to i32
-// CHECK-ARM-X64:   store i32 [[TRUNC_INDEX]], i32* %Index, align 4
+// CHECK-ARM-X64:   store i32 [[TRUNC_INDEX]], ptr %Index, align 4
 // CHECK-ARM-X64:   br label %[[END_LABEL]]
 
 unsigned char test_BitScanReverse64(unsigned long *Index, unsigned __int64 Mask) {
   return _BitScanReverse64(Index, Mask);
 }
-// CHECK-ARM-X64: define{{.*}}i8 @test_BitScanReverse64(i32* {{[a-z_ ]*}}%Index, i64 {{[a-z_ ]*}}%Mask){{.*}}{
+// CHECK-ARM-X64: define{{.*}}i8 @test_BitScanReverse64(ptr {{[a-z_ ]*}}%Index, i64 {{[a-z_ ]*}}%Mask){{.*}}{
 // CHECK-ARM-X64:   [[ISNOTZERO:%[0-9]+]] = icmp eq i64 %Mask, 0
 // CHECK-ARM-X64:   br i1 [[ISNOTZERO]], label %[[END_LABEL:[a-z0-9._]+]], label %[[ISNOTZERO_LABEL:[a-z0-9._]+]]
 // CHECK-ARM-X64:   [[END_LABEL]]:
@@ -206,7 +206,7 @@ unsigned char test_BitScanReverse64(unsigned long *Index, unsigned __int64 Mask)
 // CHECK-ARM-X64:   [[REVINDEX:%[0-9]+]] = tail call i64 @llvm.ctlz.i64(i64 %Mask, i1 true)
 // CHECK-ARM-X64:   [[TRUNC_REVINDEX:%[0-9]+]] = trunc i64 [[REVINDEX]] to i32
 // CHECK-ARM-X64:   [[INDEX:%[0-9]+]] = xor i32 [[TRUNC_REVINDEX]], 63
-// CHECK-ARM-X64:   store i32 [[INDEX]], i32* %Index, align 4
+// CHECK-ARM-X64:   store i32 [[INDEX]], ptr %Index, align 4
 // CHECK-ARM-X64:   br label %[[END_LABEL]]
 #endif
 
@@ -214,12 +214,11 @@ void *test_InterlockedExchangePointer(void * volatile *Target, void *Value) {
   return _InterlockedExchangePointer(Target, Value);
 }
 
-// CHECK: define{{.*}}i8* @test_InterlockedExchangePointer(i8** {{[a-z_ ]*}}%Target, i8* {{[a-z_ ]*}}%Value){{.*}}{
-// CHECK:   %[[TARGET:[0-9]+]] = bitcast i8** %Target to [[iPTR:i[0-9]+]]*
-// CHECK:   %[[VALUE:[0-9]+]] = ptrtoint i8* %Value to [[iPTR]]
-// CHECK:   %[[EXCHANGE:[0-9]+]] = atomicrmw xchg [[iPTR]]* %[[TARGET]], [[iPTR]] %[[VALUE]] seq_cst, align {{4|8}}
-// CHECK:   %[[RESULT:[0-9]+]] = inttoptr [[iPTR]] %[[EXCHANGE]] to i8*
-// CHECK:   ret i8* %[[RESULT]]
+// CHECK: define{{.*}}ptr @test_InterlockedExchangePointer(ptr {{[a-z_ ]*}}%Target, ptr {{[a-z_ ]*}}%Value){{.*}}{
+// CHECK:   %[[VALUE:[0-9]+]] = ptrtoint ptr %Value to [[iPTR:i[0-9]+]]
+// CHECK:   %[[EXCHANGE:[0-9]+]] = atomicrmw xchg ptr %Target, [[iPTR]] %[[VALUE]] seq_cst, align {{4|8}}
+// CHECK:   %[[RESULT:[0-9]+]] = inttoptr [[iPTR]] %[[EXCHANGE]] to ptr
+// CHECK:   ret ptr %[[RESULT]]
 // CHECK: }
 
 void *test_InterlockedCompareExchangePointer(void * volatile *Destination,
@@ -227,14 +226,13 @@ void *test_InterlockedCompareExchangePointer(void * volatile *Destination,
   return _InterlockedCompareExchangePointer(Destination, Exchange, Comparand);
 }
 
-// CHECK: define{{.*}}i8* @test_InterlockedCompareExchangePointer(i8** {{[a-z_ ]*}}%Destination, i8* {{[a-z_ ]*}}%Exchange, i8* {{[a-z_ ]*}}%Comparand){{.*}}{
-// CHECK:   %[[DEST:[0-9]+]] = bitcast i8** %Destination to [[iPTR]]*
-// CHECK:   %[[EXCHANGE:[0-9]+]] = ptrtoint i8* %Exchange to [[iPTR]]
-// CHECK:   %[[COMPARAND:[0-9]+]] = ptrtoint i8* %Comparand to [[iPTR]]
-// CHECK:   %[[XCHG:[0-9]+]] = cmpxchg volatile [[iPTR]]* %[[DEST:[0-9]+]], [[iPTR]] %[[COMPARAND:[0-9]+]], [[iPTR]] %[[EXCHANGE:[0-9]+]] seq_cst seq_cst, align {{4|8}}
+// CHECK: define{{.*}}ptr @test_InterlockedCompareExchangePointer(ptr {{[a-z_ ]*}}%Destination, ptr {{[a-z_ ]*}}%Exchange, ptr {{[a-z_ ]*}}%Comparand){{.*}}{
+// CHECK:   %[[EXCHANGE:[0-9]+]] = ptrtoint ptr %Exchange to [[iPTR]]
+// CHECK:   %[[COMPARAND:[0-9]+]] = ptrtoint ptr %Comparand to [[iPTR]]
+// CHECK:   %[[XCHG:[0-9]+]] = cmpxchg volatile ptr %[[DEST:.+]], [[iPTR]] %[[COMPARAND:[0-9]+]], [[iPTR]] %[[EXCHANGE:[0-9]+]] seq_cst seq_cst, align {{4|8}}
 // CHECK:   %[[EXTRACT:[0-9]+]] = extractvalue { [[iPTR]], i1 } %[[XCHG]], 0
-// CHECK:   %[[RESULT:[0-9]+]] = inttoptr [[iPTR]] %[[EXTRACT]] to i8*
-// CHECK:   ret i8* %[[RESULT:[0-9]+]]
+// CHECK:   %[[RESULT:[0-9]+]] = inttoptr [[iPTR]] %[[EXTRACT]] to ptr
+// CHECK:   ret ptr %[[RESULT:[0-9]+]]
 // CHECK: }
 
 void *test_InterlockedCompareExchangePointer_nf(void * volatile *Destination,
@@ -242,165 +240,164 @@ void *test_InterlockedCompareExchangePointer_nf(void * volatile *Destination,
   return _InterlockedCompareExchangePointer_nf(Destination, Exchange, Comparand);
 }
 
-// CHECK: define{{.*}}i8* @test_InterlockedCompareExchangePointer_nf(i8** {{[a-z_ ]*}}%Destination, i8* {{[a-z_ ]*}}%Exchange, i8* {{[a-z_ ]*}}%Comparand){{.*}}{
-// CHECK:   %[[DEST:[0-9]+]] = bitcast i8** %Destination to [[iPTR]]*
-// CHECK:   %[[EXCHANGE:[0-9]+]] = ptrtoint i8* %Exchange to [[iPTR]]
-// CHECK:   %[[COMPARAND:[0-9]+]] = ptrtoint i8* %Comparand to [[iPTR]]
-// CHECK:   %[[XCHG:[0-9]+]] = cmpxchg volatile [[iPTR]]* %[[DEST:[0-9]+]], [[iPTR]] %[[COMPARAND:[0-9]+]], [[iPTR]] %[[EXCHANGE:[0-9]+]] monotonic monotonic, align {{4|8}}
+// CHECK: define{{.*}}ptr @test_InterlockedCompareExchangePointer_nf(ptr {{[a-z_ ]*}}%Destination, ptr {{[a-z_ ]*}}%Exchange, ptr {{[a-z_ ]*}}%Comparand){{.*}}{
+// CHECK:   %[[EXCHANGE:[0-9]+]] = ptrtoint ptr %Exchange to [[iPTR]]
+// CHECK:   %[[COMPARAND:[0-9]+]] = ptrtoint ptr %Comparand to [[iPTR]]
+// CHECK:   %[[XCHG:[0-9]+]] = cmpxchg volatile ptr %[[DEST:.+]], [[iPTR]] %[[COMPARAND:[0-9]+]], [[iPTR]] %[[EXCHANGE:[0-9]+]] monotonic monotonic, align {{4|8}}
 // CHECK:   %[[EXTRACT:[0-9]+]] = extractvalue { [[iPTR]], i1 } %[[XCHG]], 0
-// CHECK:   %[[RESULT:[0-9]+]] = inttoptr [[iPTR]] %[[EXTRACT]] to i8*
-// CHECK:   ret i8* %[[RESULT:[0-9]+]]
+// CHECK:   %[[RESULT:[0-9]+]] = inttoptr [[iPTR]] %[[EXTRACT]] to ptr
+// CHECK:   ret ptr %[[RESULT:[0-9]+]]
 // CHECK: }
 
 char test_InterlockedExchange8(char volatile *value, char mask) {
   return _InterlockedExchange8(value, mask);
 }
-// CHECK: define{{.*}}i8 @test_InterlockedExchange8(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw xchg i8* %value, i8 %mask seq_cst, align 1
+// CHECK: define{{.*}}i8 @test_InterlockedExchange8(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw xchg ptr %value, i8 %mask seq_cst, align 1
 // CHECK:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 short test_InterlockedExchange16(short volatile *value, short mask) {
   return _InterlockedExchange16(value, mask);
 }
-// CHECK: define{{.*}}i16 @test_InterlockedExchange16(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw xchg i16* %value, i16 %mask seq_cst, align 2
+// CHECK: define{{.*}}i16 @test_InterlockedExchange16(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw xchg ptr %value, i16 %mask seq_cst, align 2
 // CHECK:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 long test_InterlockedExchange(long volatile *value, long mask) {
   return _InterlockedExchange(value, mask);
 }
-// CHECK: define{{.*}}i32 @test_InterlockedExchange(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw xchg i32* %value, i32 %mask seq_cst, align 4
+// CHECK: define{{.*}}i32 @test_InterlockedExchange(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw xchg ptr %value, i32 %mask seq_cst, align 4
 // CHECK:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 char test_InterlockedExchangeAdd8(char volatile *value, char mask) {
   return _InterlockedExchangeAdd8(value, mask);
 }
-// CHECK: define{{.*}}i8 @test_InterlockedExchangeAdd8(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw add i8* %value, i8 %mask seq_cst, align 1
+// CHECK: define{{.*}}i8 @test_InterlockedExchangeAdd8(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw add ptr %value, i8 %mask seq_cst, align 1
 // CHECK:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 short test_InterlockedExchangeAdd16(short volatile *value, short mask) {
   return _InterlockedExchangeAdd16(value, mask);
 }
-// CHECK: define{{.*}}i16 @test_InterlockedExchangeAdd16(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw add i16* %value, i16 %mask seq_cst, align 2
+// CHECK: define{{.*}}i16 @test_InterlockedExchangeAdd16(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw add ptr %value, i16 %mask seq_cst, align 2
 // CHECK:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 long test_InterlockedExchangeAdd(long volatile *value, long mask) {
   return _InterlockedExchangeAdd(value, mask);
 }
-// CHECK: define{{.*}}i32 @test_InterlockedExchangeAdd(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw add i32* %value, i32 %mask seq_cst, align 4
+// CHECK: define{{.*}}i32 @test_InterlockedExchangeAdd(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw add ptr %value, i32 %mask seq_cst, align 4
 // CHECK:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 char test_InterlockedExchangeSub8(char volatile *value, char mask) {
   return _InterlockedExchangeSub8(value, mask);
 }
-// CHECK: define{{.*}}i8 @test_InterlockedExchangeSub8(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw sub i8* %value, i8 %mask seq_cst, align 1
+// CHECK: define{{.*}}i8 @test_InterlockedExchangeSub8(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw sub ptr %value, i8 %mask seq_cst, align 1
 // CHECK:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 short test_InterlockedExchangeSub16(short volatile *value, short mask) {
   return _InterlockedExchangeSub16(value, mask);
 }
-// CHECK: define{{.*}}i16 @test_InterlockedExchangeSub16(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw sub i16* %value, i16 %mask seq_cst, align 2
+// CHECK: define{{.*}}i16 @test_InterlockedExchangeSub16(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw sub ptr %value, i16 %mask seq_cst, align 2
 // CHECK:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 long test_InterlockedExchangeSub(long volatile *value, long mask) {
   return _InterlockedExchangeSub(value, mask);
 }
-// CHECK: define{{.*}}i32 @test_InterlockedExchangeSub(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw sub i32* %value, i32 %mask seq_cst, align 4
+// CHECK: define{{.*}}i32 @test_InterlockedExchangeSub(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw sub ptr %value, i32 %mask seq_cst, align 4
 // CHECK:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 char test_InterlockedOr8(char volatile *value, char mask) {
   return _InterlockedOr8(value, mask);
 }
-// CHECK: define{{.*}}i8 @test_InterlockedOr8(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw or i8* %value, i8 %mask seq_cst, align 1
+// CHECK: define{{.*}}i8 @test_InterlockedOr8(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw or ptr %value, i8 %mask seq_cst, align 1
 // CHECK:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 short test_InterlockedOr16(short volatile *value, short mask) {
   return _InterlockedOr16(value, mask);
 }
-// CHECK: define{{.*}}i16 @test_InterlockedOr16(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw or i16* %value, i16 %mask seq_cst, align 2
+// CHECK: define{{.*}}i16 @test_InterlockedOr16(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw or ptr %value, i16 %mask seq_cst, align 2
 // CHECK:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 long test_InterlockedOr(long volatile *value, long mask) {
   return _InterlockedOr(value, mask);
 }
-// CHECK: define{{.*}}i32 @test_InterlockedOr(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw or i32* %value, i32 %mask seq_cst, align 4
+// CHECK: define{{.*}}i32 @test_InterlockedOr(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw or ptr %value, i32 %mask seq_cst, align 4
 // CHECK:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 char test_InterlockedXor8(char volatile *value, char mask) {
   return _InterlockedXor8(value, mask);
 }
-// CHECK: define{{.*}}i8 @test_InterlockedXor8(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw xor i8* %value, i8 %mask seq_cst, align 1
+// CHECK: define{{.*}}i8 @test_InterlockedXor8(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw xor ptr %value, i8 %mask seq_cst, align 1
 // CHECK:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 short test_InterlockedXor16(short volatile *value, short mask) {
   return _InterlockedXor16(value, mask);
 }
-// CHECK: define{{.*}}i16 @test_InterlockedXor16(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw xor i16* %value, i16 %mask seq_cst, align 2
+// CHECK: define{{.*}}i16 @test_InterlockedXor16(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw xor ptr %value, i16 %mask seq_cst, align 2
 // CHECK:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 long test_InterlockedXor(long volatile *value, long mask) {
   return _InterlockedXor(value, mask);
 }
-// CHECK: define{{.*}}i32 @test_InterlockedXor(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw xor i32* %value, i32 %mask seq_cst, align 4
+// CHECK: define{{.*}}i32 @test_InterlockedXor(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw xor ptr %value, i32 %mask seq_cst, align 4
 // CHECK:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 char test_InterlockedAnd8(char volatile *value, char mask) {
   return _InterlockedAnd8(value, mask);
 }
-// CHECK: define{{.*}}i8 @test_InterlockedAnd8(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw and i8* %value, i8 %mask seq_cst, align 1
+// CHECK: define{{.*}}i8 @test_InterlockedAnd8(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw and ptr %value, i8 %mask seq_cst, align 1
 // CHECK:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 short test_InterlockedAnd16(short volatile *value, short mask) {
   return _InterlockedAnd16(value, mask);
 }
-// CHECK: define{{.*}}i16 @test_InterlockedAnd16(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw and i16* %value, i16 %mask seq_cst, align 2
+// CHECK: define{{.*}}i16 @test_InterlockedAnd16(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw and ptr %value, i16 %mask seq_cst, align 2
 // CHECK:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 long test_InterlockedAnd(long volatile *value, long mask) {
   return _InterlockedAnd(value, mask);
 }
-// CHECK: define{{.*}}i32 @test_InterlockedAnd(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw and i32* %value, i32 %mask seq_cst, align 4
+// CHECK: define{{.*}}i32 @test_InterlockedAnd(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw and ptr %value, i32 %mask seq_cst, align 4
 // CHECK:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 char test_InterlockedCompareExchange8(char volatile *Destination, char Exchange, char Comperand) {
   return _InterlockedCompareExchange8(Destination, Exchange, Comperand);
 }
-// CHECK: define{{.*}}i8 @test_InterlockedCompareExchange8(i8*{{[a-z_ ]*}}%Destination, i8{{[a-z_ ]*}}%Exchange, i8{{[a-z_ ]*}}%Comperand){{.*}}{
-// CHECK: [[TMP:%[0-9]+]] = cmpxchg volatile i8* %Destination, i8 %Comperand, i8 %Exchange seq_cst seq_cst, align 1
+// CHECK: define{{.*}}i8 @test_InterlockedCompareExchange8(ptr{{[a-z_ ]*}}%Destination, i8{{[a-z_ ]*}}%Exchange, i8{{[a-z_ ]*}}%Comperand){{.*}}{
+// CHECK: [[TMP:%[0-9]+]] = cmpxchg volatile ptr %Destination, i8 %Comperand, i8 %Exchange seq_cst seq_cst, align 1
 // CHECK: [[RESULT:%[0-9]+]] = extractvalue { i8, i1 } [[TMP]], 0
 // CHECK: ret i8 [[RESULT]]
 // CHECK: }
@@ -408,8 +405,8 @@ char test_InterlockedCompareExchange8(char volatile *Destination, char Exchange,
 short test_InterlockedCompareExchange16(short volatile *Destination, short Exchange, short Comperand) {
   return _InterlockedCompareExchange16(Destination, Exchange, Comperand);
 }
-// CHECK: define{{.*}}i16 @test_InterlockedCompareExchange16(i16*{{[a-z_ ]*}}%Destination, i16{{[a-z_ ]*}}%Exchange, i16{{[a-z_ ]*}}%Comperand){{.*}}{
-// CHECK: [[TMP:%[0-9]+]] = cmpxchg volatile i16* %Destination, i16 %Comperand, i16 %Exchange seq_cst seq_cst, align 2
+// CHECK: define{{.*}}i16 @test_InterlockedCompareExchange16(ptr{{[a-z_ ]*}}%Destination, i16{{[a-z_ ]*}}%Exchange, i16{{[a-z_ ]*}}%Comperand){{.*}}{
+// CHECK: [[TMP:%[0-9]+]] = cmpxchg volatile ptr %Destination, i16 %Comperand, i16 %Exchange seq_cst seq_cst, align 2
 // CHECK: [[RESULT:%[0-9]+]] = extractvalue { i16, i1 } [[TMP]], 0
 // CHECK: ret i16 [[RESULT]]
 // CHECK: }
@@ -417,8 +414,8 @@ short test_InterlockedCompareExchange16(short volatile *Destination, short Excha
 long test_InterlockedCompareExchange(long volatile *Destination, long Exchange, long Comperand) {
   return _InterlockedCompareExchange(Destination, Exchange, Comperand);
 }
-// CHECK: define{{.*}}i32 @test_InterlockedCompareExchange(i32*{{[a-z_ ]*}}%Destination, i32{{[a-z_ ]*}}%Exchange, i32{{[a-z_ ]*}}%Comperand){{.*}}{
-// CHECK: [[TMP:%[0-9]+]] = cmpxchg volatile i32* %Destination, i32 %Comperand, i32 %Exchange seq_cst seq_cst, align 4
+// CHECK: define{{.*}}i32 @test_InterlockedCompareExchange(ptr{{[a-z_ ]*}}%Destination, i32{{[a-z_ ]*}}%Exchange, i32{{[a-z_ ]*}}%Comperand){{.*}}{
+// CHECK: [[TMP:%[0-9]+]] = cmpxchg volatile ptr %Destination, i32 %Comperand, i32 %Exchange seq_cst seq_cst, align 4
 // CHECK: [[RESULT:%[0-9]+]] = extractvalue { i32, i1 } [[TMP]], 0
 // CHECK: ret i32 [[RESULT]]
 // CHECK: }
@@ -426,8 +423,8 @@ long test_InterlockedCompareExchange(long volatile *Destination, long Exchange,
 __int64 test_InterlockedCompareExchange64(__int64 volatile *Destination, __int64 Exchange, __int64 Comperand) {
   return _InterlockedCompareExchange64(Destination, Exchange, Comperand);
 }
-// CHECK: define{{.*}}i64 @test_InterlockedCompareExchange64(i64*{{[a-z_ ]*}}%Destination, i64{{[a-z_ ]*}}%Exchange, i64{{[a-z_ ]*}}%Comperand){{.*}}{
-// CHECK: [[TMP:%[0-9]+]] = cmpxchg volatile i64* %Destination, i64 %Comperand, i64 %Exchange seq_cst seq_cst, align 8
+// CHECK: define{{.*}}i64 @test_InterlockedCompareExchange64(ptr{{[a-z_ ]*}}%Destination, i64{{[a-z_ ]*}}%Exchange, i64{{[a-z_ ]*}}%Comperand){{.*}}{
+// CHECK: [[TMP:%[0-9]+]] = cmpxchg volatile ptr %Destination, i64 %Comperand, i64 %Exchange seq_cst seq_cst, align 8
 // CHECK: [[RESULT:%[0-9]+]] = extractvalue { i64, i1 } [[TMP]], 0
 // CHECK: ret i64 [[RESULT]]
 // CHECK: }
@@ -439,21 +436,19 @@ unsigned char test_InterlockedCompareExchange128(
   return _InterlockedCompareExchange128(++Destination, ++ExchangeHigh,
                                         ++ExchangeLow, ++ComparandResult);
 }
-// CHECK-64: define{{.*}}i8 @test_InterlockedCompareExchange128(i64*{{[a-z_ ]*}}%Destination, i64{{[a-z_ ]*}}%ExchangeHigh, i64{{[a-z_ ]*}}%ExchangeLow, i64*{{[a-z_ ]*}}%ComparandResult){{.*}}{
-// CHECK-64: %incdec.ptr = getelementptr inbounds i64, i64* %Destination, i64 1
+// CHECK-64: define{{.*}}i8 @test_InterlockedCompareExchange128(ptr{{[a-z_ ]*}}%Destination, i64{{[a-z_ ]*}}%ExchangeHigh, i64{{[a-z_ ]*}}%ExchangeLow, ptr{{[a-z_ ]*}}%ComparandResult){{.*}}{
+// CHECK-64: %incdec.ptr = getelementptr inbounds i64, ptr %Destination, i64 1
 // CHECK-64: %inc = add nsw i64 %ExchangeHigh, 1
 // CHECK-64: %inc1 = add nsw i64 %ExchangeLow, 1
-// CHECK-64: %incdec.ptr2 = getelementptr inbounds i64, i64* %ComparandResult, i64 1
-// CHECK-64: [[DST:%[0-9]+]] = bitcast i64* %incdec.ptr to i128*
-// CHECK-64: [[CNR:%[0-9]+]] = bitcast i64* %incdec.ptr2 to i128*
+// CHECK-64: %incdec.ptr2 = getelementptr inbounds i64, ptr %ComparandResult, i64 1
 // CHECK-64: [[EH:%[0-9]+]] = zext i64 %inc to i128
 // CHECK-64: [[EL:%[0-9]+]] = zext i64 %inc1 to i128
 // CHECK-64: [[EHS:%[0-9]+]] = shl nuw i128 [[EH]], 64
 // CHECK-64: [[EXP:%[0-9]+]] = or i128 [[EHS]], [[EL]]
-// CHECK-64: [[ORG:%[0-9]+]] = load i128, i128* [[CNR]], align 16
-// CHECK-64: [[RES:%[0-9]+]] = cmpxchg volatile i128* [[DST]], i128 [[ORG]], i128 [[EXP]] seq_cst seq_cst, align 16
+// CHECK-64: [[ORG:%[0-9]+]] = load i128, ptr %incdec.ptr2, align 16
+// CHECK-64: [[RES:%[0-9]+]] = cmpxchg volatile ptr %incdec.ptr, i128 [[ORG]], i128 [[EXP]] seq_cst seq_cst, align 16
 // CHECK-64: [[OLD:%[0-9]+]] = extractvalue { i128, i1 } [[RES]], 0
-// CHECK-64: store i128 [[OLD]], i128* [[CNR]], align 16
+// CHECK-64: store i128 [[OLD]], ptr %incdec.ptr2, align 16
 // CHECK-64: [[SUC1:%[0-9]+]] = extractvalue { i128, i1 } [[RES]], 1
 // CHECK-64: [[SUC8:%[0-9]+]] = zext i1 [[SUC1]] to i8
 // CHECK-64: ret i8 [[SUC8]]
@@ -480,19 +475,19 @@ unsigned char test_InterlockedCompareExchange128_rel(
                                             ExchangeLow, ComparandResult);
 }
 // CHECK-ARM64: define{{.*}}i8 @test_InterlockedCompareExchange128_acq({{.*}})
-// CHECK-ARM64: cmpxchg volatile i128* %{{.*}}, i128 %{{.*}}, i128 %{{.*}} acquire acquire, align 16
+// CHECK-ARM64: cmpxchg volatile ptr %{{.*}}, i128 %{{.*}}, i128 %{{.*}} acquire acquire, align 16
 // CHECK-ARM64: define{{.*}}i8 @test_InterlockedCompareExchange128_nf({{.*}})
-// CHECK-ARM64: cmpxchg volatile i128* %{{.*}}, i128 %{{.*}}, i128 %{{.*}} monotonic monotonic, align 16
+// CHECK-ARM64: cmpxchg volatile ptr %{{.*}}, i128 %{{.*}}, i128 %{{.*}} monotonic monotonic, align 16
 // CHECK-ARM64: define{{.*}}i8 @test_InterlockedCompareExchange128_rel({{.*}})
-// CHECK-ARM64: cmpxchg volatile i128* %{{.*}}, i128 %{{.*}}, i128 %{{.*}} release monotonic, align 16
+// CHECK-ARM64: cmpxchg volatile ptr %{{.*}}, i128 %{{.*}}, i128 %{{.*}} release monotonic, align 16
 #endif
 
 short test_InterlockedIncrement16(short volatile *Addend) {
   return _InterlockedIncrement16(++Addend);
 }
-// CHECK: define{{.*}}i16 @test_InterlockedIncrement16(i16*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK: %incdec.ptr = getelementptr inbounds i16, i16* %Addend, {{i64|i32}} 1
-// CHECK: [[TMP:%[0-9]+]] = atomicrmw add i16* %incdec.ptr, i16 1 seq_cst, align 2
+// CHECK: define{{.*}}i16 @test_InterlockedIncrement16(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK: %incdec.ptr = getelementptr inbounds i16, ptr %Addend, {{i64|i32}} 1
+// CHECK: [[TMP:%[0-9]+]] = atomicrmw add ptr %incdec.ptr, i16 1 seq_cst, align 2
 // CHECK: [[RESULT:%[0-9]+]] = add i16 [[TMP]], 1
 // CHECK: ret i16 [[RESULT]]
 // CHECK: }
@@ -500,9 +495,9 @@ short test_InterlockedIncrement16(short volatile *Addend) {
 long test_InterlockedIncrement(long volatile *Addend) {
   return _InterlockedIncrement(++Addend);
 }
-// CHECK: define{{.*}}i32 @test_InterlockedIncrement(i32*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK: %incdec.ptr = getelementptr inbounds i32, i32* %Addend, {{i64|i32}} 1
-// CHECK: [[TMP:%[0-9]+]] = atomicrmw add i32* %incdec.ptr, i32 1 seq_cst, align 4
+// CHECK: define{{.*}}i32 @test_InterlockedIncrement(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK: %incdec.ptr = getelementptr inbounds i32, ptr %Addend, {{i64|i32}} 1
+// CHECK: [[TMP:%[0-9]+]] = atomicrmw add ptr %incdec.ptr, i32 1 seq_cst, align 4
 // CHECK: [[RESULT:%[0-9]+]] = add i32 [[TMP]], 1
 // CHECK: ret i32 [[RESULT]]
 // CHECK: }
@@ -510,8 +505,8 @@ long test_InterlockedIncrement(long volatile *Addend) {
 short test_InterlockedDecrement16(short volatile *Addend) {
   return _InterlockedDecrement16(Addend);
 }
-// CHECK: define{{.*}}i16 @test_InterlockedDecrement16(i16*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK: [[TMP:%[0-9]+]] = atomicrmw sub i16* %Addend, i16 1 seq_cst, align 2
+// CHECK: define{{.*}}i16 @test_InterlockedDecrement16(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK: [[TMP:%[0-9]+]] = atomicrmw sub ptr %Addend, i16 1 seq_cst, align 2
 // CHECK: [[RESULT:%[0-9]+]] = add i16 [[TMP]], -1
 // CHECK: ret i16 [[RESULT]]
 // CHECK: }
@@ -519,8 +514,8 @@ short test_InterlockedDecrement16(short volatile *Addend) {
 long test_InterlockedDecrement(long volatile *Addend) {
   return _InterlockedDecrement(Addend);
 }
-// CHECK: define{{.*}}i32 @test_InterlockedDecrement(i32*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK: [[TMP:%[0-9]+]] = atomicrmw sub i32* %Addend, i32 1 seq_cst, align 4
+// CHECK: define{{.*}}i32 @test_InterlockedDecrement(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK: [[TMP:%[0-9]+]] = atomicrmw sub ptr %Addend, i32 1 seq_cst, align 4
 // CHECK: [[RESULT:%[0-9]+]] = add i32 [[TMP]], -1
 // CHECK: ret i32 [[RESULT]]
 // CHECK: }
@@ -530,84 +525,84 @@ short test_iso_volatile_load16(short volatile *p) { return __iso_volatile_load16
 int test_iso_volatile_load32(int volatile *p) { return __iso_volatile_load32(p); }
 __int64 test_iso_volatile_load64(__int64 volatile *p) { return __iso_volatile_load64(p); }
 
-// CHECK: define{{.*}}i8 @test_iso_volatile_load8(i8*{{[a-z_ ]*}}%p)
-// CHECK: = load volatile i8, i8* %p
-// CHECK: define{{.*}}i16 @test_iso_volatile_load16(i16*{{[a-z_ ]*}}%p)
-// CHECK: = load volatile i16, i16* %p
-// CHECK: define{{.*}}i32 @test_iso_volatile_load32(i32*{{[a-z_ ]*}}%p)
-// CHECK: = load volatile i32, i32* %p
-// CHECK: define{{.*}}i64 @test_iso_volatile_load64(i64*{{[a-z_ ]*}}%p)
-// CHECK: = load volatile i64, i64* %p
+// CHECK: define{{.*}}i8 @test_iso_volatile_load8(ptr{{[a-z_ ]*}}%p)
+// CHECK: = load volatile i8, ptr %p
+// CHECK: define{{.*}}i16 @test_iso_volatile_load16(ptr{{[a-z_ ]*}}%p)
+// CHECK: = load volatile i16, ptr %p
+// CHECK: define{{.*}}i32 @test_iso_volatile_load32(ptr{{[a-z_ ]*}}%p)
+// CHECK: = load volatile i32, ptr %p
+// CHECK: define{{.*}}i64 @test_iso_volatile_load64(ptr{{[a-z_ ]*}}%p)
+// CHECK: = load volatile i64, ptr %p
 
 void test_iso_volatile_store8(char volatile *p, char v) { __iso_volatile_store8(p, v); }
 void test_iso_volatile_store16(short volatile *p, short v) { __iso_volatile_store16(p, v); }
 void test_iso_volatile_store32(int volatile *p, int v) { __iso_volatile_store32(p, v); }
 void test_iso_volatile_store64(__int64 volatile *p, __int64 v) { __iso_volatile_store64(p, v); }
 
-// CHECK: define{{.*}}void @test_iso_volatile_store8(i8*{{[a-z_ ]*}}%p, i8 {{[a-z_ ]*}}%v)
-// CHECK: store volatile i8 %v, i8* %p
-// CHECK: define{{.*}}void @test_iso_volatile_store16(i16*{{[a-z_ ]*}}%p, i16 {{[a-z_ ]*}}%v)
-// CHECK: store volatile i16 %v, i16* %p
-// CHECK: define{{.*}}void @test_iso_volatile_store32(i32*{{[a-z_ ]*}}%p, i32 {{[a-z_ ]*}}%v)
-// CHECK: store volatile i32 %v, i32* %p
-// CHECK: define{{.*}}void @test_iso_volatile_store64(i64*{{[a-z_ ]*}}%p, i64 {{[a-z_ ]*}}%v)
-// CHECK: store volatile i64 %v, i64* %p
+// CHECK: define{{.*}}void @test_iso_volatile_store8(ptr{{[a-z_ ]*}}%p, i8 {{[a-z_ ]*}}%v)
+// CHECK: store volatile i8 %v, ptr %p
+// CHECK: define{{.*}}void @test_iso_volatile_store16(ptr{{[a-z_ ]*}}%p, i16 {{[a-z_ ]*}}%v)
+// CHECK: store volatile i16 %v, ptr %p
+// CHECK: define{{.*}}void @test_iso_volatile_store32(ptr{{[a-z_ ]*}}%p, i32 {{[a-z_ ]*}}%v)
+// CHECK: store volatile i32 %v, ptr %p
+// CHECK: define{{.*}}void @test_iso_volatile_store64(ptr{{[a-z_ ]*}}%p, i64 {{[a-z_ ]*}}%v)
+// CHECK: store volatile i64 %v, ptr %p
 
 
 #if defined(__i386__) || defined(__x86_64__) || defined(__arm__) || defined(__aarch64__)
 __int64 test_InterlockedExchange64(__int64 volatile *value, __int64 mask) {
   return _InterlockedExchange64(value, mask);
 }
-// CHECK: define{{.*}}i64 @test_InterlockedExchange64(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw xchg i64* %value, i64 %mask seq_cst, align 8
+// CHECK: define{{.*}}i64 @test_InterlockedExchange64(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw xchg ptr %value, i64 %mask seq_cst, align 8
 // CHECK:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 __int64 test_InterlockedExchangeAdd64(__int64 volatile *value, __int64 mask) {
   return _InterlockedExchangeAdd64(value, mask);
 }
-// CHECK: define{{.*}}i64 @test_InterlockedExchangeAdd64(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw add i64* %value, i64 %mask seq_cst, align 8
+// CHECK: define{{.*}}i64 @test_InterlockedExchangeAdd64(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw add ptr %value, i64 %mask seq_cst, align 8
 // CHECK:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 __int64 test_InterlockedExchangeSub64(__int64 volatile *value, __int64 mask) {
   return _InterlockedExchangeSub64(value, mask);
 }
-// CHECK: define{{.*}}i64 @test_InterlockedExchangeSub64(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw sub i64* %value, i64 %mask seq_cst, align 8
+// CHECK: define{{.*}}i64 @test_InterlockedExchangeSub64(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw sub ptr %value, i64 %mask seq_cst, align 8
 // CHECK:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 __int64 test_InterlockedOr64(__int64 volatile *value, __int64 mask) {
   return _InterlockedOr64(value, mask);
 }
-// CHECK: define{{.*}}i64 @test_InterlockedOr64(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw or i64* %value, i64 %mask seq_cst, align 8
+// CHECK: define{{.*}}i64 @test_InterlockedOr64(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw or ptr %value, i64 %mask seq_cst, align 8
 // CHECK:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 __int64 test_InterlockedXor64(__int64 volatile *value, __int64 mask) {
   return _InterlockedXor64(value, mask);
 }
-// CHECK: define{{.*}}i64 @test_InterlockedXor64(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw xor i64* %value, i64 %mask seq_cst, align 8
+// CHECK: define{{.*}}i64 @test_InterlockedXor64(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw xor ptr %value, i64 %mask seq_cst, align 8
 // CHECK:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 __int64 test_InterlockedAnd64(__int64 volatile *value, __int64 mask) {
   return _InterlockedAnd64(value, mask);
 }
-// CHECK: define{{.*}}i64 @test_InterlockedAnd64(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw and i64* %value, i64 %mask seq_cst, align 8
+// CHECK: define{{.*}}i64 @test_InterlockedAnd64(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK:   [[RESULT:%[0-9]+]] = atomicrmw and ptr %value, i64 %mask seq_cst, align 8
 // CHECK:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK: }
 
 __int64 test_InterlockedIncrement64(__int64 volatile *Addend) {
   return _InterlockedIncrement64(Addend);
 }
-// CHECK: define{{.*}}i64 @test_InterlockedIncrement64(i64*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK: [[TMP:%[0-9]+]] = atomicrmw add i64* %Addend, i64 1 seq_cst, align 8
+// CHECK: define{{.*}}i64 @test_InterlockedIncrement64(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK: [[TMP:%[0-9]+]] = atomicrmw add ptr %Addend, i64 1 seq_cst, align 8
 // CHECK: [[RESULT:%[0-9]+]] = add i64 [[TMP]], 1
 // CHECK: ret i64 [[RESULT]]
 // CHECK: }
@@ -615,8 +610,8 @@ __int64 test_InterlockedIncrement64(__int64 volatile *Addend) {
 __int64 test_InterlockedDecrement64(__int64 volatile *Addend) {
   return _InterlockedDecrement64(Addend);
 }
-// CHECK: define{{.*}}i64 @test_InterlockedDecrement64(i64*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK: [[TMP:%[0-9]+]] = atomicrmw sub i64* %Addend, i64 1 seq_cst, align 8
+// CHECK: define{{.*}}i64 @test_InterlockedDecrement64(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK: [[TMP:%[0-9]+]] = atomicrmw sub ptr %Addend, i64 1 seq_cst, align 8
 // CHECK: [[RESULT:%[0-9]+]] = add i64 [[TMP]], -1
 // CHECK: ret i64 [[RESULT]]
 // CHECK: }
@@ -625,49 +620,49 @@ __int64 test_InterlockedDecrement64(__int64 volatile *Addend) {
 
 #if defined(__i386__) || defined(__x86_64__)
 long test_InterlockedExchange_HLEAcquire(long volatile *Target, long Value) {
-// CHECK-INTEL: define{{.*}} i32 @test_InterlockedExchange_HLEAcquire(i32*{{[a-z_ ]*}}%Target, i32{{[a-z_ ]*}}%Value)
-// CHECK-INTEL: call i32 asm sideeffect ".byte 0xf2 ; lock ; xchg $($0, $1$|$1, $0$)", "=r,=*m,0,*m,~{memory},~{dirflag},~{fpsr},~{flags}"(i32* elementtype(i32) %Target, i32 %Value, i32* elementtype(i32) %Target)
+// CHECK-INTEL: define{{.*}} i32 @test_InterlockedExchange_HLEAcquire(ptr{{[a-z_ ]*}}%Target, i32{{[a-z_ ]*}}%Value)
+// CHECK-INTEL: call i32 asm sideeffect ".byte 0xf2 ; lock ; xchg $($0, $1$|$1, $0$)", "=r,=*m,0,*m,~{memory},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i32) %Target, i32 %Value, ptr elementtype(i32) %Target)
   return _InterlockedExchange_HLEAcquire(Target, Value);
 }
 long test_InterlockedExchange_HLERelease(long volatile *Target, long Value) {
-// CHECK-INTEL: define{{.*}} i32 @test_InterlockedExchange_HLERelease(i32*{{[a-z_ ]*}}%Target, i32{{[a-z_ ]*}}%Value)
-// CHECK-INTEL: call i32 asm sideeffect ".byte 0xf3 ; lock ; xchg $($0, $1$|$1, $0$)", "=r,=*m,0,*m,~{memory},~{dirflag},~{fpsr},~{flags}"(i32* elementtype(i32) %Target, i32 %Value, i32* elementtype(i32) %Target)
+// CHECK-INTEL: define{{.*}} i32 @test_InterlockedExchange_HLERelease(ptr{{[a-z_ ]*}}%Target, i32{{[a-z_ ]*}}%Value)
+// CHECK-INTEL: call i32 asm sideeffect ".byte 0xf3 ; lock ; xchg $($0, $1$|$1, $0$)", "=r,=*m,0,*m,~{memory},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i32) %Target, i32 %Value, ptr elementtype(i32) %Target)
   return _InterlockedExchange_HLERelease(Target, Value);
 }
 long test_InterlockedCompareExchange_HLEAcquire(long volatile *Destination,
                                                 long Exchange, long Comparand) {
-// CHECK-INTEL: define{{.*}} i32 @test_InterlockedCompareExchange_HLEAcquire(i32*{{[a-z_ ]*}}%Destination, i32{{[a-z_ ]*}}%Exchange, i32{{[a-z_ ]*}}%Comparand)
-// CHECK-INTEL: call i32 asm sideeffect ".byte 0xf2 ; lock ; cmpxchg $($2, $1$|$1, $2$)", "={ax},=*m,r,0,*m,~{memory},~{dirflag},~{fpsr},~{flags}"(i32* elementtype(i32) %Destination, i32 %Exchange, i32 %Comparand, i32* elementtype(i32) %Destination)
+// CHECK-INTEL: define{{.*}} i32 @test_InterlockedCompareExchange_HLEAcquire(ptr{{[a-z_ ]*}}%Destination, i32{{[a-z_ ]*}}%Exchange, i32{{[a-z_ ]*}}%Comparand)
+// CHECK-INTEL: call i32 asm sideeffect ".byte 0xf2 ; lock ; cmpxchg $($2, $1$|$1, $2$)", "={ax},=*m,r,0,*m,~{memory},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i32) %Destination, i32 %Exchange, i32 %Comparand, ptr elementtype(i32) %Destination)
   return _InterlockedCompareExchange_HLEAcquire(Destination, Exchange, Comparand);
 }
 long test_InterlockedCompareExchange_HLERelease(long volatile *Destination,
                                             long Exchange, long Comparand) {
-// CHECK-INTEL: define{{.*}} i32 @test_InterlockedCompareExchange_HLERelease(i32*{{[a-z_ ]*}}%Destination, i32{{[a-z_ ]*}}%Exchange, i32{{[a-z_ ]*}}%Comparand)
-// CHECK-INTEL: call i32 asm sideeffect ".byte 0xf3 ; lock ; cmpxchg $($2, $1$|$1, $2$)", "={ax},=*m,r,0,*m,~{memory},~{dirflag},~{fpsr},~{flags}"(i32* elementtype(i32) %Destination, i32 %Exchange, i32 %Comparand, i32* elementtype(i32) %Destination)
+// CHECK-INTEL: define{{.*}} i32 @test_InterlockedCompareExchange_HLERelease(ptr{{[a-z_ ]*}}%Destination, i32{{[a-z_ ]*}}%Exchange, i32{{[a-z_ ]*}}%Comparand)
+// CHECK-INTEL: call i32 asm sideeffect ".byte 0xf3 ; lock ; cmpxchg $($2, $1$|$1, $2$)", "={ax},=*m,r,0,*m,~{memory},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i32) %Destination, i32 %Exchange, i32 %Comparand, ptr elementtype(i32) %Destination)
   return _InterlockedCompareExchange_HLERelease(Destination, Exchange, Comparand);
 }
 #endif
 #if defined(__x86_64__)
 __int64 test_InterlockedExchange64_HLEAcquire(__int64 volatile *Target, __int64 Value) {
-// CHECK-X64: define{{.*}} i64 @test_InterlockedExchange64_HLEAcquire(i64*{{[a-z_ ]*}}%Target, i64{{[a-z_ ]*}}%Value)
-// CHECK-X64: call i64 asm sideeffect ".byte 0xf2 ; lock ; xchg $($0, $1$|$1, $0$)", "=r,=*m,0,*m,~{memory},~{dirflag},~{fpsr},~{flags}"(i64* elementtype(i64) %Target, i64 %Value, i64* elementtype(i64) %Target)
+// CHECK-X64: define{{.*}} i64 @test_InterlockedExchange64_HLEAcquire(ptr{{[a-z_ ]*}}%Target, i64{{[a-z_ ]*}}%Value)
+// CHECK-X64: call i64 asm sideeffect ".byte 0xf2 ; lock ; xchg $($0, $1$|$1, $0$)", "=r,=*m,0,*m,~{memory},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %Target, i64 %Value, ptr elementtype(i64) %Target)
   return _InterlockedExchange64_HLEAcquire(Target, Value);
 }
 __int64 test_InterlockedExchange64_HLERelease(__int64 volatile *Target, __int64 Value) {
-// CHECK-X64: define{{.*}} i64 @test_InterlockedExchange64_HLERelease(i64*{{[a-z_ ]*}}%Target, i64{{[a-z_ ]*}}%Value)
-// CHECK-X64: call i64 asm sideeffect ".byte 0xf3 ; lock ; xchg $($0, $1$|$1, $0$)", "=r,=*m,0,*m,~{memory},~{dirflag},~{fpsr},~{flags}"(i64* elementtype(i64) %Target, i64 %Value, i64* elementtype(i64) %Target)
+// CHECK-X64: define{{.*}} i64 @test_InterlockedExchange64_HLERelease(ptr{{[a-z_ ]*}}%Target, i64{{[a-z_ ]*}}%Value)
+// CHECK-X64: call i64 asm sideeffect ".byte 0xf3 ; lock ; xchg $($0, $1$|$1, $0$)", "=r,=*m,0,*m,~{memory},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %Target, i64 %Value, ptr elementtype(i64) %Target)
   return _InterlockedExchange64_HLERelease(Target, Value);
 }
 __int64 test_InterlockedCompareExchange64_HLEAcquire(__int64 volatile *Destination,
                                                      __int64 Exchange, __int64 Comparand) {
-// CHECK-X64: define{{.*}} i64 @test_InterlockedCompareExchange64_HLEAcquire(i64*{{[a-z_ ]*}}%Destination, i64{{[a-z_ ]*}}%Exchange, i64{{[a-z_ ]*}}%Comparand)
-// CHECK-X64: call i64 asm sideeffect ".byte 0xf2 ; lock ; cmpxchg $($2, $1$|$1, $2$)", "={ax},=*m,r,0,*m,~{memory},~{dirflag},~{fpsr},~{flags}"(i64* elementtype(i64) %Destination, i64 %Exchange, i64 %Comparand, i64* elementtype(i64) %Destination)
+// CHECK-X64: define{{.*}} i64 @test_InterlockedCompareExchange64_HLEAcquire(ptr{{[a-z_ ]*}}%Destination, i64{{[a-z_ ]*}}%Exchange, i64{{[a-z_ ]*}}%Comparand)
+// CHECK-X64: call i64 asm sideeffect ".byte 0xf2 ; lock ; cmpxchg $($2, $1$|$1, $2$)", "={ax},=*m,r,0,*m,~{memory},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %Destination, i64 %Exchange, i64 %Comparand, ptr elementtype(i64) %Destination)
   return _InterlockedCompareExchange64_HLEAcquire(Destination, Exchange, Comparand);
 }
 __int64 test_InterlockedCompareExchange64_HLERelease(__int64 volatile *Destination,
                                                      __int64 Exchange, __int64 Comparand) {
-// CHECK-X64: define{{.*}} i64 @test_InterlockedCompareExchange64_HLERelease(i64*{{[a-z_ ]*}}%Destination, i64{{[a-z_ ]*}}%Exchange, i64{{[a-z_ ]*}}%Comparand)
-// CHECK-X64: call i64 asm sideeffect ".byte 0xf3 ; lock ; cmpxchg $($2, $1$|$1, $2$)", "={ax},=*m,r,0,*m,~{memory},~{dirflag},~{fpsr},~{flags}"(i64* elementtype(i64) %Destination, i64 %Exchange, i64 %Comparand, i64* elementtype(i64) %Destination)
+// CHECK-X64: define{{.*}} i64 @test_InterlockedCompareExchange64_HLERelease(ptr{{[a-z_ ]*}}%Destination, i64{{[a-z_ ]*}}%Exchange, i64{{[a-z_ ]*}}%Comparand)
+// CHECK-X64: call i64 asm sideeffect ".byte 0xf3 ; lock ; cmpxchg $($2, $1$|$1, $2$)", "={ax},=*m,r,0,*m,~{memory},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %Destination, i64 %Exchange, i64 %Comparand, ptr elementtype(i64) %Destination)
   return _InterlockedCompareExchange64_HLERelease(Destination, Exchange, Comparand);
 }
 #endif
@@ -676,178 +671,178 @@ __int64 test_InterlockedCompareExchange64_HLERelease(__int64 volatile *Destinati
 char test_InterlockedExchangeAdd8_acq(char volatile *value, char mask) {
   return _InterlockedExchangeAdd8_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedExchangeAdd8_acq(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add i8* %value, i8 %mask acquire, align 1
+// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedExchangeAdd8_acq(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add ptr %value, i8 %mask acquire, align 1
 // CHECK-ARM-ARM64:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 char test_InterlockedExchangeAdd8_rel(char volatile *value, char mask) {
   return _InterlockedExchangeAdd8_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedExchangeAdd8_rel(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add i8* %value, i8 %mask release, align 1
+// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedExchangeAdd8_rel(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add ptr %value, i8 %mask release, align 1
 // CHECK-ARM-ARM64:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 char test_InterlockedExchangeAdd8_nf(char volatile *value, char mask) {
   return _InterlockedExchangeAdd8_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedExchangeAdd8_nf(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add i8* %value, i8 %mask monotonic, align 1
+// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedExchangeAdd8_nf(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add ptr %value, i8 %mask monotonic, align 1
 // CHECK-ARM-ARM64:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 short test_InterlockedExchangeAdd16_acq(short volatile *value, short mask) {
   return _InterlockedExchangeAdd16_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedExchangeAdd16_acq(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add i16* %value, i16 %mask acquire, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedExchangeAdd16_acq(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add ptr %value, i16 %mask acquire, align 2
 // CHECK-ARM-ARM64:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 short test_InterlockedExchangeAdd16_rel(short volatile *value, short mask) {
   return _InterlockedExchangeAdd16_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedExchangeAdd16_rel(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add i16* %value, i16 %mask release, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedExchangeAdd16_rel(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add ptr %value, i16 %mask release, align 2
 // CHECK-ARM-ARM64:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 short test_InterlockedExchangeAdd16_nf(short volatile *value, short mask) {
   return _InterlockedExchangeAdd16_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedExchangeAdd16_nf(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add i16* %value, i16 %mask monotonic, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedExchangeAdd16_nf(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add ptr %value, i16 %mask monotonic, align 2
 // CHECK-ARM-ARM64:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 long test_InterlockedExchangeAdd_acq(long volatile *value, long mask) {
   return _InterlockedExchangeAdd_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedExchangeAdd_acq(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add i32* %value, i32 %mask acquire, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedExchangeAdd_acq(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add ptr %value, i32 %mask acquire, align 4
 // CHECK-ARM-ARM64:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 long test_InterlockedExchangeAdd_rel(long volatile *value, long mask) {
   return _InterlockedExchangeAdd_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedExchangeAdd_rel(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add i32* %value, i32 %mask release, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedExchangeAdd_rel(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add ptr %value, i32 %mask release, align 4
 // CHECK-ARM-ARM64:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 long test_InterlockedExchangeAdd_nf(long volatile *value, long mask) {
   return _InterlockedExchangeAdd_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedExchangeAdd_nf(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add i32* %value, i32 %mask monotonic, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedExchangeAdd_nf(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add ptr %value, i32 %mask monotonic, align 4
 // CHECK-ARM-ARM64:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 __int64 test_InterlockedExchangeAdd64_acq(__int64 volatile *value, __int64 mask) {
   return _InterlockedExchangeAdd64_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedExchangeAdd64_acq(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add i64* %value, i64 %mask acquire, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedExchangeAdd64_acq(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add ptr %value, i64 %mask acquire, align 8
 // CHECK-ARM-ARM64:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 __int64 test_InterlockedExchangeAdd64_rel(__int64 volatile *value, __int64 mask) {
   return _InterlockedExchangeAdd64_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedExchangeAdd64_rel(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add i64* %value, i64 %mask release, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedExchangeAdd64_rel(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add ptr %value, i64 %mask release, align 8
 // CHECK-ARM-ARM64:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 __int64 test_InterlockedExchangeAdd64_nf(__int64 volatile *value, __int64 mask) {
   return _InterlockedExchangeAdd64_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedExchangeAdd64_nf(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add i64* %value, i64 %mask monotonic, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedExchangeAdd64_nf(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw add ptr %value, i64 %mask monotonic, align 8
 // CHECK-ARM-ARM64:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 char test_InterlockedExchange8_acq(char volatile *value, char mask) {
   return _InterlockedExchange8_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedExchange8_acq(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg i8* %value, i8 %mask acquire, align 1
+// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedExchange8_acq(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg ptr %value, i8 %mask acquire, align 1
 // CHECK-ARM-ARM64:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 char test_InterlockedExchange8_rel(char volatile *value, char mask) {
   return _InterlockedExchange8_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedExchange8_rel(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg i8* %value, i8 %mask release, align 1
+// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedExchange8_rel(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg ptr %value, i8 %mask release, align 1
 // CHECK-ARM-ARM64:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 char test_InterlockedExchange8_nf(char volatile *value, char mask) {
   return _InterlockedExchange8_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedExchange8_nf(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg i8* %value, i8 %mask monotonic, align 1
+// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedExchange8_nf(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg ptr %value, i8 %mask monotonic, align 1
 // CHECK-ARM-ARM64:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 short test_InterlockedExchange16_acq(short volatile *value, short mask) {
   return _InterlockedExchange16_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedExchange16_acq(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg i16* %value, i16 %mask acquire, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedExchange16_acq(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg ptr %value, i16 %mask acquire, align 2
 // CHECK-ARM-ARM64:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 short test_InterlockedExchange16_rel(short volatile *value, short mask) {
   return _InterlockedExchange16_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedExchange16_rel(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg i16* %value, i16 %mask release, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedExchange16_rel(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg ptr %value, i16 %mask release, align 2
 // CHECK-ARM-ARM64:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 short test_InterlockedExchange16_nf(short volatile *value, short mask) {
   return _InterlockedExchange16_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedExchange16_nf(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg i16* %value, i16 %mask monotonic, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedExchange16_nf(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg ptr %value, i16 %mask monotonic, align 2
 // CHECK-ARM-ARM64:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 long test_InterlockedExchange_acq(long volatile *value, long mask) {
   return _InterlockedExchange_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedExchange_acq(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg i32* %value, i32 %mask acquire, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedExchange_acq(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg ptr %value, i32 %mask acquire, align 4
 // CHECK-ARM-ARM64:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 long test_InterlockedExchange_rel(long volatile *value, long mask) {
   return _InterlockedExchange_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedExchange_rel(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg i32* %value, i32 %mask release, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedExchange_rel(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg ptr %value, i32 %mask release, align 4
 // CHECK-ARM-ARM64:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 long test_InterlockedExchange_nf(long volatile *value, long mask) {
   return _InterlockedExchange_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedExchange_nf(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg i32* %value, i32 %mask monotonic, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedExchange_nf(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg ptr %value, i32 %mask monotonic, align 4
 // CHECK-ARM-ARM64:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 __int64 test_InterlockedExchange64_acq(__int64 volatile *value, __int64 mask) {
   return _InterlockedExchange64_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedExchange64_acq(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg i64* %value, i64 %mask acquire, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedExchange64_acq(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg ptr %value, i64 %mask acquire, align 8
 // CHECK-ARM-ARM64:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 __int64 test_InterlockedExchange64_rel(__int64 volatile *value, __int64 mask) {
   return _InterlockedExchange64_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedExchange64_rel(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg i64* %value, i64 %mask release, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedExchange64_rel(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg ptr %value, i64 %mask release, align 8
 // CHECK-ARM-ARM64:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 __int64 test_InterlockedExchange64_nf(__int64 volatile *value, __int64 mask) {
   return _InterlockedExchange64_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedExchange64_nf(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg i64* %value, i64 %mask monotonic, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedExchange64_nf(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xchg ptr %value, i64 %mask monotonic, align 8
 // CHECK-ARM-ARM64:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 char test_InterlockedCompareExchange8_acq(char volatile *Destination, char Exchange, char Comperand) {
   return _InterlockedCompareExchange8_acq(Destination, Exchange, Comperand);
 }
-// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedCompareExchange8_acq(i8*{{[a-z_ ]*}}%Destination, i8{{[a-z_ ]*}}%Exchange, i8{{[a-z_ ]*}}%Comperand){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile i8* %Destination, i8 %Comperand, i8 %Exchange acquire acquire, align 1
+// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedCompareExchange8_acq(ptr{{[a-z_ ]*}}%Destination, i8{{[a-z_ ]*}}%Exchange, i8{{[a-z_ ]*}}%Comperand){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile ptr %Destination, i8 %Comperand, i8 %Exchange acquire acquire, align 1
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = extractvalue { i8, i1 } [[TMP]], 0
 // CHECK-ARM-ARM64: ret i8 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -855,8 +850,8 @@ char test_InterlockedCompareExchange8_acq(char volatile *Destination, char Excha
 char test_InterlockedCompareExchange8_rel(char volatile *Destination, char Exchange, char Comperand) {
   return _InterlockedCompareExchange8_rel(Destination, Exchange, Comperand);
 }
-// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedCompareExchange8_rel(i8*{{[a-z_ ]*}}%Destination, i8{{[a-z_ ]*}}%Exchange, i8{{[a-z_ ]*}}%Comperand){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile i8* %Destination, i8 %Comperand, i8 %Exchange release monotonic, align 1
+// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedCompareExchange8_rel(ptr{{[a-z_ ]*}}%Destination, i8{{[a-z_ ]*}}%Exchange, i8{{[a-z_ ]*}}%Comperand){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile ptr %Destination, i8 %Comperand, i8 %Exchange release monotonic, align 1
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = extractvalue { i8, i1 } [[TMP]], 0
 // CHECK-ARM-ARM64: ret i8 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -864,8 +859,8 @@ char test_InterlockedCompareExchange8_rel(char volatile *Destination, char Excha
 char test_InterlockedCompareExchange8_nf(char volatile *Destination, char Exchange, char Comperand) {
   return _InterlockedCompareExchange8_nf(Destination, Exchange, Comperand);
 }
-// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedCompareExchange8_nf(i8*{{[a-z_ ]*}}%Destination, i8{{[a-z_ ]*}}%Exchange, i8{{[a-z_ ]*}}%Comperand){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile i8* %Destination, i8 %Comperand, i8 %Exchange monotonic monotonic, align 1
+// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedCompareExchange8_nf(ptr{{[a-z_ ]*}}%Destination, i8{{[a-z_ ]*}}%Exchange, i8{{[a-z_ ]*}}%Comperand){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile ptr %Destination, i8 %Comperand, i8 %Exchange monotonic monotonic, align 1
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = extractvalue { i8, i1 } [[TMP]], 0
 // CHECK-ARM-ARM64: ret i8 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -873,8 +868,8 @@ char test_InterlockedCompareExchange8_nf(char volatile *Destination, char Exchan
 short test_InterlockedCompareExchange16_acq(short volatile *Destination, short Exchange, short Comperand) {
   return _InterlockedCompareExchange16_acq(Destination, Exchange, Comperand);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedCompareExchange16_acq(i16*{{[a-z_ ]*}}%Destination, i16{{[a-z_ ]*}}%Exchange, i16{{[a-z_ ]*}}%Comperand){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile i16* %Destination, i16 %Comperand, i16 %Exchange acquire acquire, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedCompareExchange16_acq(ptr{{[a-z_ ]*}}%Destination, i16{{[a-z_ ]*}}%Exchange, i16{{[a-z_ ]*}}%Comperand){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile ptr %Destination, i16 %Comperand, i16 %Exchange acquire acquire, align 2
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = extractvalue { i16, i1 } [[TMP]], 0
 // CHECK-ARM-ARM64: ret i16 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -882,8 +877,8 @@ short test_InterlockedCompareExchange16_acq(short volatile *Destination, short E
 short test_InterlockedCompareExchange16_rel(short volatile *Destination, short Exchange, short Comperand) {
   return _InterlockedCompareExchange16_rel(Destination, Exchange, Comperand);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedCompareExchange16_rel(i16*{{[a-z_ ]*}}%Destination, i16{{[a-z_ ]*}}%Exchange, i16{{[a-z_ ]*}}%Comperand){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile i16* %Destination, i16 %Comperand, i16 %Exchange release monotonic, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedCompareExchange16_rel(ptr{{[a-z_ ]*}}%Destination, i16{{[a-z_ ]*}}%Exchange, i16{{[a-z_ ]*}}%Comperand){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile ptr %Destination, i16 %Comperand, i16 %Exchange release monotonic, align 2
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = extractvalue { i16, i1 } [[TMP]], 0
 // CHECK-ARM-ARM64: ret i16 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -891,8 +886,8 @@ short test_InterlockedCompareExchange16_rel(short volatile *Destination, short E
 short test_InterlockedCompareExchange16_nf(short volatile *Destination, short Exchange, short Comperand) {
   return _InterlockedCompareExchange16_nf(Destination, Exchange, Comperand);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedCompareExchange16_nf(i16*{{[a-z_ ]*}}%Destination, i16{{[a-z_ ]*}}%Exchange, i16{{[a-z_ ]*}}%Comperand){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile i16* %Destination, i16 %Comperand, i16 %Exchange monotonic monotonic, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedCompareExchange16_nf(ptr{{[a-z_ ]*}}%Destination, i16{{[a-z_ ]*}}%Exchange, i16{{[a-z_ ]*}}%Comperand){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile ptr %Destination, i16 %Comperand, i16 %Exchange monotonic monotonic, align 2
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = extractvalue { i16, i1 } [[TMP]], 0
 // CHECK-ARM-ARM64: ret i16 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -900,8 +895,8 @@ short test_InterlockedCompareExchange16_nf(short volatile *Destination, short Ex
 long test_InterlockedCompareExchange_acq(long volatile *Destination, long Exchange, long Comperand) {
   return _InterlockedCompareExchange_acq(Destination, Exchange, Comperand);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedCompareExchange_acq(i32*{{[a-z_ ]*}}%Destination, i32{{[a-z_ ]*}}%Exchange, i32{{[a-z_ ]*}}%Comperand){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile i32* %Destination, i32 %Comperand, i32 %Exchange acquire acquire, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedCompareExchange_acq(ptr{{[a-z_ ]*}}%Destination, i32{{[a-z_ ]*}}%Exchange, i32{{[a-z_ ]*}}%Comperand){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile ptr %Destination, i32 %Comperand, i32 %Exchange acquire acquire, align 4
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = extractvalue { i32, i1 } [[TMP]], 0
 // CHECK-ARM-ARM64: ret i32 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -909,8 +904,8 @@ long test_InterlockedCompareExchange_acq(long volatile *Destination, long Exchan
 long test_InterlockedCompareExchange_rel(long volatile *Destination, long Exchange, long Comperand) {
   return _InterlockedCompareExchange_rel(Destination, Exchange, Comperand);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedCompareExchange_rel(i32*{{[a-z_ ]*}}%Destination, i32{{[a-z_ ]*}}%Exchange, i32{{[a-z_ ]*}}%Comperand){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile i32* %Destination, i32 %Comperand, i32 %Exchange release monotonic, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedCompareExchange_rel(ptr{{[a-z_ ]*}}%Destination, i32{{[a-z_ ]*}}%Exchange, i32{{[a-z_ ]*}}%Comperand){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile ptr %Destination, i32 %Comperand, i32 %Exchange release monotonic, align 4
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = extractvalue { i32, i1 } [[TMP]], 0
 // CHECK-ARM-ARM64: ret i32 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -918,8 +913,8 @@ long test_InterlockedCompareExchange_rel(long volatile *Destination, long Exchan
 long test_InterlockedCompareExchange_nf(long volatile *Destination, long Exchange, long Comperand) {
   return _InterlockedCompareExchange_nf(Destination, Exchange, Comperand);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedCompareExchange_nf(i32*{{[a-z_ ]*}}%Destination, i32{{[a-z_ ]*}}%Exchange, i32{{[a-z_ ]*}}%Comperand){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile i32* %Destination, i32 %Comperand, i32 %Exchange monotonic monotonic, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedCompareExchange_nf(ptr{{[a-z_ ]*}}%Destination, i32{{[a-z_ ]*}}%Exchange, i32{{[a-z_ ]*}}%Comperand){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile ptr %Destination, i32 %Comperand, i32 %Exchange monotonic monotonic, align 4
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = extractvalue { i32, i1 } [[TMP]], 0
 // CHECK-ARM-ARM64: ret i32 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -927,8 +922,8 @@ long test_InterlockedCompareExchange_nf(long volatile *Destination, long Exchang
 __int64 test_InterlockedCompareExchange64_acq(__int64 volatile *Destination, __int64 Exchange, __int64 Comperand) {
   return _InterlockedCompareExchange64_acq(Destination, Exchange, Comperand);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedCompareExchange64_acq(i64*{{[a-z_ ]*}}%Destination, i64{{[a-z_ ]*}}%Exchange, i64{{[a-z_ ]*}}%Comperand){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile i64* %Destination, i64 %Comperand, i64 %Exchange acquire acquire, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedCompareExchange64_acq(ptr{{[a-z_ ]*}}%Destination, i64{{[a-z_ ]*}}%Exchange, i64{{[a-z_ ]*}}%Comperand){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile ptr %Destination, i64 %Comperand, i64 %Exchange acquire acquire, align 8
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = extractvalue { i64, i1 } [[TMP]], 0
 // CHECK-ARM-ARM64: ret i64 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -936,8 +931,8 @@ __int64 test_InterlockedCompareExchange64_acq(__int64 volatile *Destination, __i
 __int64 test_InterlockedCompareExchange64_rel(__int64 volatile *Destination, __int64 Exchange, __int64 Comperand) {
   return _InterlockedCompareExchange64_rel(Destination, Exchange, Comperand);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedCompareExchange64_rel(i64*{{[a-z_ ]*}}%Destination, i64{{[a-z_ ]*}}%Exchange, i64{{[a-z_ ]*}}%Comperand){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile i64* %Destination, i64 %Comperand, i64 %Exchange release monotonic, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedCompareExchange64_rel(ptr{{[a-z_ ]*}}%Destination, i64{{[a-z_ ]*}}%Exchange, i64{{[a-z_ ]*}}%Comperand){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile ptr %Destination, i64 %Comperand, i64 %Exchange release monotonic, align 8
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = extractvalue { i64, i1 } [[TMP]], 0
 // CHECK-ARM-ARM64: ret i64 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -945,8 +940,8 @@ __int64 test_InterlockedCompareExchange64_rel(__int64 volatile *Destination, __i
 __int64 test_InterlockedCompareExchange64_nf(__int64 volatile *Destination, __int64 Exchange, __int64 Comperand) {
   return _InterlockedCompareExchange64_nf(Destination, Exchange, Comperand);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedCompareExchange64_nf(i64*{{[a-z_ ]*}}%Destination, i64{{[a-z_ ]*}}%Exchange, i64{{[a-z_ ]*}}%Comperand){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile i64* %Destination, i64 %Comperand, i64 %Exchange monotonic monotonic, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedCompareExchange64_nf(ptr{{[a-z_ ]*}}%Destination, i64{{[a-z_ ]*}}%Exchange, i64{{[a-z_ ]*}}%Comperand){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = cmpxchg volatile ptr %Destination, i64 %Comperand, i64 %Exchange monotonic monotonic, align 8
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = extractvalue { i64, i1 } [[TMP]], 0
 // CHECK-ARM-ARM64: ret i64 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -954,296 +949,296 @@ __int64 test_InterlockedCompareExchange64_nf(__int64 volatile *Destination, __in
 char test_InterlockedOr8_acq(char volatile *value, char mask) {
   return _InterlockedOr8_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedOr8_acq(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or i8* %value, i8 %mask acquire, align 1
+// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedOr8_acq(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or ptr %value, i8 %mask acquire, align 1
 // CHECK-ARM-ARM64:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 char test_InterlockedOr8_rel(char volatile *value, char mask) {
   return _InterlockedOr8_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedOr8_rel(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or i8* %value, i8 %mask release, align 1
+// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedOr8_rel(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or ptr %value, i8 %mask release, align 1
 // CHECK-ARM-ARM64:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 char test_InterlockedOr8_nf(char volatile *value, char mask) {
   return _InterlockedOr8_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedOr8_nf(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or i8* %value, i8 %mask monotonic, align 1
+// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedOr8_nf(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or ptr %value, i8 %mask monotonic, align 1
 // CHECK-ARM-ARM64:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 short test_InterlockedOr16_acq(short volatile *value, short mask) {
   return _InterlockedOr16_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedOr16_acq(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or i16* %value, i16 %mask acquire, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedOr16_acq(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or ptr %value, i16 %mask acquire, align 2
 // CHECK-ARM-ARM64:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 short test_InterlockedOr16_rel(short volatile *value, short mask) {
   return _InterlockedOr16_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedOr16_rel(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or i16* %value, i16 %mask release, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedOr16_rel(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or ptr %value, i16 %mask release, align 2
 // CHECK-ARM-ARM64:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 short test_InterlockedOr16_nf(short volatile *value, short mask) {
   return _InterlockedOr16_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedOr16_nf(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or i16* %value, i16 %mask monotonic, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedOr16_nf(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or ptr %value, i16 %mask monotonic, align 2
 // CHECK-ARM-ARM64:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 long test_InterlockedOr_acq(long volatile *value, long mask) {
   return _InterlockedOr_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedOr_acq(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or i32* %value, i32 %mask acquire, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedOr_acq(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or ptr %value, i32 %mask acquire, align 4
 // CHECK-ARM-ARM64:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 long test_InterlockedOr_rel(long volatile *value, long mask) {
   return _InterlockedOr_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedOr_rel(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or i32* %value, i32 %mask release, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedOr_rel(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or ptr %value, i32 %mask release, align 4
 // CHECK-ARM-ARM64:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 long test_InterlockedOr_nf(long volatile *value, long mask) {
   return _InterlockedOr_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedOr_nf(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or i32* %value, i32 %mask monotonic, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedOr_nf(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or ptr %value, i32 %mask monotonic, align 4
 // CHECK-ARM-ARM64:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 __int64 test_InterlockedOr64_acq(__int64 volatile *value, __int64 mask) {
   return _InterlockedOr64_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedOr64_acq(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or i64* %value, i64 %mask acquire, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedOr64_acq(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or ptr %value, i64 %mask acquire, align 8
 // CHECK-ARM-ARM64:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 __int64 test_InterlockedOr64_rel(__int64 volatile *value, __int64 mask) {
   return _InterlockedOr64_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedOr64_rel(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or i64* %value, i64 %mask release, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedOr64_rel(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or ptr %value, i64 %mask release, align 8
 // CHECK-ARM-ARM64:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 __int64 test_InterlockedOr64_nf(__int64 volatile *value, __int64 mask) {
   return _InterlockedOr64_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedOr64_nf(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or i64* %value, i64 %mask monotonic, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedOr64_nf(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw or ptr %value, i64 %mask monotonic, align 8
 // CHECK-ARM-ARM64:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 char test_InterlockedXor8_acq(char volatile *value, char mask) {
   return _InterlockedXor8_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedXor8_acq(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor i8* %value, i8 %mask acquire, align 1
+// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedXor8_acq(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor ptr %value, i8 %mask acquire, align 1
 // CHECK-ARM-ARM64:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 char test_InterlockedXor8_rel(char volatile *value, char mask) {
   return _InterlockedXor8_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedXor8_rel(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor i8* %value, i8 %mask release, align 1
+// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedXor8_rel(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor ptr %value, i8 %mask release, align 1
 // CHECK-ARM-ARM64:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 char test_InterlockedXor8_nf(char volatile *value, char mask) {
   return _InterlockedXor8_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedXor8_nf(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor i8* %value, i8 %mask monotonic, align 1
+// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedXor8_nf(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor ptr %value, i8 %mask monotonic, align 1
 // CHECK-ARM-ARM64:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 short test_InterlockedXor16_acq(short volatile *value, short mask) {
   return _InterlockedXor16_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedXor16_acq(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor i16* %value, i16 %mask acquire, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedXor16_acq(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor ptr %value, i16 %mask acquire, align 2
 // CHECK-ARM-ARM64:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 short test_InterlockedXor16_rel(short volatile *value, short mask) {
   return _InterlockedXor16_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedXor16_rel(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor i16* %value, i16 %mask release, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedXor16_rel(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor ptr %value, i16 %mask release, align 2
 // CHECK-ARM-ARM64:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 short test_InterlockedXor16_nf(short volatile *value, short mask) {
   return _InterlockedXor16_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedXor16_nf(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor i16* %value, i16 %mask monotonic, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedXor16_nf(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor ptr %value, i16 %mask monotonic, align 2
 // CHECK-ARM-ARM64:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 long test_InterlockedXor_acq(long volatile *value, long mask) {
   return _InterlockedXor_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedXor_acq(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor i32* %value, i32 %mask acquire, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedXor_acq(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor ptr %value, i32 %mask acquire, align 4
 // CHECK-ARM-ARM64:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 long test_InterlockedXor_rel(long volatile *value, long mask) {
   return _InterlockedXor_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedXor_rel(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor i32* %value, i32 %mask release, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedXor_rel(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor ptr %value, i32 %mask release, align 4
 // CHECK-ARM-ARM64:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 long test_InterlockedXor_nf(long volatile *value, long mask) {
   return _InterlockedXor_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedXor_nf(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor i32* %value, i32 %mask monotonic, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedXor_nf(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor ptr %value, i32 %mask monotonic, align 4
 // CHECK-ARM-ARM64:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 __int64 test_InterlockedXor64_acq(__int64 volatile *value, __int64 mask) {
   return _InterlockedXor64_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedXor64_acq(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor i64* %value, i64 %mask acquire, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedXor64_acq(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor ptr %value, i64 %mask acquire, align 8
 // CHECK-ARM-ARM64:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 __int64 test_InterlockedXor64_rel(__int64 volatile *value, __int64 mask) {
   return _InterlockedXor64_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedXor64_rel(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor i64* %value, i64 %mask release, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedXor64_rel(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor ptr %value, i64 %mask release, align 8
 // CHECK-ARM-ARM64:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 __int64 test_InterlockedXor64_nf(__int64 volatile *value, __int64 mask) {
   return _InterlockedXor64_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedXor64_nf(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor i64* %value, i64 %mask monotonic, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedXor64_nf(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw xor ptr %value, i64 %mask monotonic, align 8
 // CHECK-ARM-ARM64:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 char test_InterlockedAnd8_acq(char volatile *value, char mask) {
   return _InterlockedAnd8_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedAnd8_acq(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and i8* %value, i8 %mask acquire, align 1
+// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedAnd8_acq(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and ptr %value, i8 %mask acquire, align 1
 // CHECK-ARM-ARM64:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 char test_InterlockedAnd8_rel(char volatile *value, char mask) {
   return _InterlockedAnd8_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedAnd8_rel(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and i8* %value, i8 %mask release, align 1
+// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedAnd8_rel(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and ptr %value, i8 %mask release, align 1
 // CHECK-ARM-ARM64:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 char test_InterlockedAnd8_nf(char volatile *value, char mask) {
   return _InterlockedAnd8_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedAnd8_nf(i8*{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and i8* %value, i8 %mask monotonic, align 1
+// CHECK-ARM-ARM64: define{{.*}}i8 @test_InterlockedAnd8_nf(ptr{{[a-z_ ]*}}%value, i8{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and ptr %value, i8 %mask monotonic, align 1
 // CHECK-ARM-ARM64:   ret i8 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 short test_InterlockedAnd16_acq(short volatile *value, short mask) {
   return _InterlockedAnd16_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedAnd16_acq(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and i16* %value, i16 %mask acquire, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedAnd16_acq(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and ptr %value, i16 %mask acquire, align 2
 // CHECK-ARM-ARM64:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 short test_InterlockedAnd16_rel(short volatile *value, short mask) {
   return _InterlockedAnd16_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedAnd16_rel(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and i16* %value, i16 %mask release, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedAnd16_rel(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and ptr %value, i16 %mask release, align 2
 // CHECK-ARM-ARM64:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 short test_InterlockedAnd16_nf(short volatile *value, short mask) {
   return _InterlockedAnd16_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedAnd16_nf(i16*{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and i16* %value, i16 %mask monotonic, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedAnd16_nf(ptr{{[a-z_ ]*}}%value, i16{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and ptr %value, i16 %mask monotonic, align 2
 // CHECK-ARM-ARM64:   ret i16 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 long test_InterlockedAnd_acq(long volatile *value, long mask) {
   return _InterlockedAnd_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedAnd_acq(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and i32* %value, i32 %mask acquire, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedAnd_acq(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and ptr %value, i32 %mask acquire, align 4
 // CHECK-ARM-ARM64:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 long test_InterlockedAnd_rel(long volatile *value, long mask) {
   return _InterlockedAnd_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedAnd_rel(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and i32* %value, i32 %mask release, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedAnd_rel(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and ptr %value, i32 %mask release, align 4
 // CHECK-ARM-ARM64:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 long test_InterlockedAnd_nf(long volatile *value, long mask) {
   return _InterlockedAnd_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedAnd_nf(i32*{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and i32* %value, i32 %mask monotonic, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedAnd_nf(ptr{{[a-z_ ]*}}%value, i32{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and ptr %value, i32 %mask monotonic, align 4
 // CHECK-ARM-ARM64:   ret i32 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 __int64 test_InterlockedAnd64_acq(__int64 volatile *value, __int64 mask) {
   return _InterlockedAnd64_acq(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedAnd64_acq(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and i64* %value, i64 %mask acquire, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedAnd64_acq(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and ptr %value, i64 %mask acquire, align 8
 // CHECK-ARM-ARM64:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 __int64 test_InterlockedAnd64_rel(__int64 volatile *value, __int64 mask) {
   return _InterlockedAnd64_rel(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedAnd64_rel(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and i64* %value, i64 %mask release, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedAnd64_rel(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and ptr %value, i64 %mask release, align 8
 // CHECK-ARM-ARM64:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 __int64 test_InterlockedAnd64_nf(__int64 volatile *value, __int64 mask) {
   return _InterlockedAnd64_nf(value, mask);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedAnd64_nf(i64*{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
-// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and i64* %value, i64 %mask monotonic, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedAnd64_nf(ptr{{[a-z_ ]*}}%value, i64{{[a-z_ ]*}}%mask){{.*}}{
+// CHECK-ARM-ARM64:   [[RESULT:%[0-9]+]] = atomicrmw and ptr %value, i64 %mask monotonic, align 8
 // CHECK-ARM-ARM64:   ret i64 [[RESULT:%[0-9]+]]
 // CHECK-ARM-ARM64: }
 
 short test_InterlockedIncrement16_acq(short volatile *Addend) {
   return _InterlockedIncrement16_acq(Addend);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedIncrement16_acq(i16*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw add i16* %Addend, i16 1 acquire, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedIncrement16_acq(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw add ptr %Addend, i16 1 acquire, align 2
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = add i16 [[TMP]], 1
 // CHECK-ARM-ARM64: ret i16 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -1251,8 +1246,8 @@ short test_InterlockedIncrement16_acq(short volatile *Addend) {
 short test_InterlockedIncrement16_rel(short volatile *Addend) {
   return _InterlockedIncrement16_rel(Addend);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedIncrement16_rel(i16*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw add i16* %Addend, i16 1 release, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedIncrement16_rel(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw add ptr %Addend, i16 1 release, align 2
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = add i16 [[TMP]], 1
 // CHECK-ARM-ARM64: ret i16 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -1260,8 +1255,8 @@ short test_InterlockedIncrement16_rel(short volatile *Addend) {
 short test_InterlockedIncrement16_nf(short volatile *Addend) {
   return _InterlockedIncrement16_nf(Addend);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedIncrement16_nf(i16*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw add i16* %Addend, i16 1 monotonic, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedIncrement16_nf(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw add ptr %Addend, i16 1 monotonic, align 2
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = add i16 [[TMP]], 1
 // CHECK-ARM-ARM64: ret i16 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -1269,8 +1264,8 @@ short test_InterlockedIncrement16_nf(short volatile *Addend) {
 long test_InterlockedIncrement_acq(long volatile *Addend) {
   return _InterlockedIncrement_acq(Addend);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedIncrement_acq(i32*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw add i32* %Addend, i32 1 acquire, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedIncrement_acq(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw add ptr %Addend, i32 1 acquire, align 4
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = add i32 [[TMP]], 1
 // CHECK-ARM-ARM64: ret i32 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -1278,8 +1273,8 @@ long test_InterlockedIncrement_acq(long volatile *Addend) {
 long test_InterlockedIncrement_rel(long volatile *Addend) {
   return _InterlockedIncrement_rel(Addend);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedIncrement_rel(i32*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw add i32* %Addend, i32 1 release, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedIncrement_rel(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw add ptr %Addend, i32 1 release, align 4
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = add i32 [[TMP]], 1
 // CHECK-ARM-ARM64: ret i32 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -1287,8 +1282,8 @@ long test_InterlockedIncrement_rel(long volatile *Addend) {
 long test_InterlockedIncrement_nf(long volatile *Addend) {
   return _InterlockedIncrement_nf(Addend);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedIncrement_nf(i32*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw add i32* %Addend, i32 1 monotonic, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedIncrement_nf(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw add ptr %Addend, i32 1 monotonic, align 4
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = add i32 [[TMP]], 1
 // CHECK-ARM-ARM64: ret i32 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -1296,8 +1291,8 @@ long test_InterlockedIncrement_nf(long volatile *Addend) {
 __int64 test_InterlockedIncrement64_acq(__int64 volatile *Addend) {
   return _InterlockedIncrement64_acq(Addend);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedIncrement64_acq(i64*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw add i64* %Addend, i64 1 acquire, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedIncrement64_acq(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw add ptr %Addend, i64 1 acquire, align 8
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = add i64 [[TMP]], 1
 // CHECK-ARM-ARM64: ret i64 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -1305,8 +1300,8 @@ __int64 test_InterlockedIncrement64_acq(__int64 volatile *Addend) {
 __int64 test_InterlockedIncrement64_rel(__int64 volatile *Addend) {
   return _InterlockedIncrement64_rel(Addend);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedIncrement64_rel(i64*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw add i64* %Addend, i64 1 release, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedIncrement64_rel(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw add ptr %Addend, i64 1 release, align 8
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = add i64 [[TMP]], 1
 // CHECK-ARM-ARM64: ret i64 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -1314,8 +1309,8 @@ __int64 test_InterlockedIncrement64_rel(__int64 volatile *Addend) {
 __int64 test_InterlockedIncrement64_nf(__int64 volatile *Addend) {
   return _InterlockedIncrement64_nf(Addend);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedIncrement64_nf(i64*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw add i64* %Addend, i64 1 monotonic, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedIncrement64_nf(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw add ptr %Addend, i64 1 monotonic, align 8
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = add i64 [[TMP]], 1
 // CHECK-ARM-ARM64: ret i64 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -1323,8 +1318,8 @@ __int64 test_InterlockedIncrement64_nf(__int64 volatile *Addend) {
 short test_InterlockedDecrement16_acq(short volatile *Addend) {
   return _InterlockedDecrement16_acq(Addend);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedDecrement16_acq(i16*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw sub i16* %Addend, i16 1 acquire, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedDecrement16_acq(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw sub ptr %Addend, i16 1 acquire, align 2
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = add i16 [[TMP]], -1
 // CHECK-ARM-ARM64: ret i16 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -1332,8 +1327,8 @@ short test_InterlockedDecrement16_acq(short volatile *Addend) {
 short test_InterlockedDecrement16_rel(short volatile *Addend) {
   return _InterlockedDecrement16_rel(Addend);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedDecrement16_rel(i16*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw sub i16* %Addend, i16 1 release, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedDecrement16_rel(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw sub ptr %Addend, i16 1 release, align 2
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = add i16 [[TMP]], -1
 // CHECK-ARM-ARM64: ret i16 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -1341,8 +1336,8 @@ short test_InterlockedDecrement16_rel(short volatile *Addend) {
 short test_InterlockedDecrement16_nf(short volatile *Addend) {
   return _InterlockedDecrement16_nf(Addend);
 }
-// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedDecrement16_nf(i16*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw sub i16* %Addend, i16 1 monotonic, align 2
+// CHECK-ARM-ARM64: define{{.*}}i16 @test_InterlockedDecrement16_nf(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw sub ptr %Addend, i16 1 monotonic, align 2
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = add i16 [[TMP]], -1
 // CHECK-ARM-ARM64: ret i16 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -1350,8 +1345,8 @@ short test_InterlockedDecrement16_nf(short volatile *Addend) {
 long test_InterlockedDecrement_acq(long volatile *Addend) {
   return _InterlockedDecrement_acq(Addend);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedDecrement_acq(i32*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw sub i32* %Addend, i32 1 acquire, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedDecrement_acq(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw sub ptr %Addend, i32 1 acquire, align 4
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = add i32 [[TMP]], -1
 // CHECK-ARM-ARM64: ret i32 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -1359,8 +1354,8 @@ long test_InterlockedDecrement_acq(long volatile *Addend) {
 long test_InterlockedDecrement_rel(long volatile *Addend) {
   return _InterlockedDecrement_rel(Addend);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedDecrement_rel(i32*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw sub i32* %Addend, i32 1 release, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedDecrement_rel(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw sub ptr %Addend, i32 1 release, align 4
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = add i32 [[TMP]], -1
 // CHECK-ARM-ARM64: ret i32 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -1368,8 +1363,8 @@ long test_InterlockedDecrement_rel(long volatile *Addend) {
 long test_InterlockedDecrement_nf(long volatile *Addend) {
   return _InterlockedDecrement_nf(Addend);
 }
-// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedDecrement_nf(i32*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw sub i32* %Addend, i32 1 monotonic, align 4
+// CHECK-ARM-ARM64: define{{.*}}i32 @test_InterlockedDecrement_nf(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw sub ptr %Addend, i32 1 monotonic, align 4
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = add i32 [[TMP]], -1
 // CHECK-ARM-ARM64: ret i32 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -1377,8 +1372,8 @@ long test_InterlockedDecrement_nf(long volatile *Addend) {
 __int64 test_InterlockedDecrement64_acq(__int64 volatile *Addend) {
   return _InterlockedDecrement64_acq(Addend);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedDecrement64_acq(i64*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw sub i64* %Addend, i64 1 acquire, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedDecrement64_acq(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw sub ptr %Addend, i64 1 acquire, align 8
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = add i64 [[TMP]], -1
 // CHECK-ARM-ARM64: ret i64 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -1386,8 +1381,8 @@ __int64 test_InterlockedDecrement64_acq(__int64 volatile *Addend) {
 __int64 test_InterlockedDecrement64_rel(__int64 volatile *Addend) {
   return _InterlockedDecrement64_rel(Addend);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedDecrement64_rel(i64*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw sub i64* %Addend, i64 1 release, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedDecrement64_rel(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw sub ptr %Addend, i64 1 release, align 8
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = add i64 [[TMP]], -1
 // CHECK-ARM-ARM64: ret i64 [[RESULT]]
 // CHECK-ARM-ARM64: }
@@ -1395,8 +1390,8 @@ __int64 test_InterlockedDecrement64_rel(__int64 volatile *Addend) {
 __int64 test_InterlockedDecrement64_nf(__int64 volatile *Addend) {
   return _InterlockedDecrement64_nf(Addend);
 }
-// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedDecrement64_nf(i64*{{[a-z_ ]*}}%Addend){{.*}}{
-// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw sub i64* %Addend, i64 1 monotonic, align 8
+// CHECK-ARM-ARM64: define{{.*}}i64 @test_InterlockedDecrement64_nf(ptr{{[a-z_ ]*}}%Addend){{.*}}{
+// CHECK-ARM-ARM64: [[TMP:%[0-9]+]] = atomicrmw sub ptr %Addend, i64 1 monotonic, align 8
 // CHECK-ARM-ARM64: [[RESULT:%[0-9]+]] = add i64 [[TMP]], -1
 // CHECK-ARM-ARM64: ret i64 [[RESULT]]
 // CHECK-ARM-ARM64: }
diff --git a/clang/test/CodeGen/no-bitfield-type-align.c b/clang/test/CodeGen/no-bitfield-type-align.c
index b143f3973c5f9..53ed5e9ad8f85 100644
--- a/clang/test/CodeGen/no-bitfield-type-align.c
+++ b/clang/test/CodeGen/no-bitfield-type-align.c
@@ -1,6 +1,4 @@
-// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-apple-darwin -fno-bitfield-type-align -emit-llvm -o - %s | FileCheck %s
-
-// CHECK: %[[STRUCT_S:.*]] = type { i32 }
+// RUN: %clang_cc1 -triple x86_64-apple-darwin -fno-bitfield-type-align -emit-llvm -o - %s | FileCheck %s
 
 struct S {
   unsigned short:   0;
@@ -9,21 +7,21 @@ struct S {
   unsigned short  f2:15;
 };
 
-// CHECK: define{{.*}} void @test_zero_width_bitfield(%[[STRUCT_S]]* noundef %[[A:.*]])
-// CHECK: %[[BF_LOAD:.*]] = load i32, i32* %[[V1:.*]], align 1
+// CHECK: define{{.*}} void @test_zero_width_bitfield(ptr noundef %[[A:.*]])
+// CHECK: %[[BF_LOAD:.*]] = load i32, ptr %[[V1:.*]], align 1
 // CHECK: %[[BF_CLEAR:.*]] = and i32 %[[BF_LOAD]], 32767
 // CHECK: %[[BF_CAST:.*]] = trunc i32 %[[BF_CLEAR]] to i16
 // CHECK: %[[CONV:.*]] = zext i16 %[[BF_CAST]] to i32
 // CHECK: %[[ADD:.*]] = add nsw i32 %[[CONV]], 1
 // CHECK: %[[CONV1:.*]] = trunc i32 %[[ADD]] to i16
 // CHECK: %[[V2:.*]] = zext i16 %[[CONV1]] to i32
-// CHECK: %[[BF_LOAD2:.*]] = load i32, i32* %[[V1]], align 1
+// CHECK: %[[BF_LOAD2:.*]] = load i32, ptr %[[V1]], align 1
 // CHECK: %[[BF_VALUE:.*]] = and i32 %[[V2]], 32767
 // CHECK: %[[BF_CLEAR3:.*]] = and i32 %[[BF_LOAD2]], -32768
 // CHECK: %[[BF_SET:.*]] = or i32 %[[BF_CLEAR3]], %[[BF_VALUE]]
-// CHECK: store i32 %[[BF_SET]], i32* %[[V1]], align 1
+// CHECK: store i32 %[[BF_SET]], ptr %[[V1]], align 1
 
-// CHECK: %[[BF_LOAD4:.*]] = load i32, i32* %[[V4:.*]], align 1
+// CHECK: %[[BF_LOAD4:.*]] = load i32, ptr %[[V4:.*]], align 1
 // CHECK: %[[BF_LSHR:.*]] = lshr i32 %[[BF_LOAD4]], 15
 // CHECK: %[[BF_CLEAR5:.*]] = and i32 %[[BF_LSHR]], 32767
 // CHECK: %[[BF_CAST6:.*]] = trunc i32 %[[BF_CLEAR5]] to i16
@@ -31,12 +29,12 @@ struct S {
 // CHECK: %[[ADD8:.*]] = add nsw i32 %[[CONV7]], 2
 // CHECK: %[[CONV9:.*]] = trunc i32 %[[ADD8]] to i16
 // CHECK: %[[V5:.*]] = zext i16 %[[CONV9]] to i32
-// CHECK: %[[BF_LOAD10:.*]] = load i32, i32* %[[V4]], align 1
+// CHECK: %[[BF_LOAD10:.*]] = load i32, ptr %[[V4]], align 1
 // CHECK: %[[BF_VALUE11:.*]] = and i32 %[[V5]], 32767
 // CHECK: %[[BF_SHL:.*]] = shl i32 %[[BF_VALUE11]], 15
 // CHECK: %[[BF_CLEAR12:.*]] = and i32 %[[BF_LOAD10]], -1073709057
 // CHECK: %[[BF_SET13:.*]] = or i32 %[[BF_CLEAR12]], %[[BF_SHL]]
-// CHECK: store i32 %[[BF_SET13]], i32* %[[V4]], align 1
+// CHECK: store i32 %[[BF_SET13]], ptr %[[V4]], align 1
 
 void test_zero_width_bitfield(struct S *a) {
   a->f1 += 1;
diff --git a/clang/test/CodeGen/no-builtin.cpp b/clang/test/CodeGen/no-builtin.cpp
index cca13aa486d8a..14bae1fe1a223 100644
--- a/clang/test/CodeGen/no-builtin.cpp
+++ b/clang/test/CodeGen/no-builtin.cpp
@@ -1,4 +1,4 @@
-// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-linux-gnu -S -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -S -emit-llvm -o - %s | FileCheck %s
 
 // CHECK-LABEL: define{{.*}} void @foo_no_mempcy() #0
 extern "C" void foo_no_mempcy() __attribute__((no_builtin("memcpy"))) {}
@@ -28,15 +28,15 @@ struct B : public A {
   virtual ~B();
 };
 
-// CHECK-LABEL: define{{.*}} void @call_a_foo(%struct.A* noundef %a) #3
+// CHECK-LABEL: define{{.*}} void @call_a_foo(ptr noundef %a) #3
 extern "C" void call_a_foo(A *a) {
-  // CHECK: %call = call noundef i32 %2(%struct.A* {{[^,]*}} %0)
+  // CHECK: %call = call noundef i32 %1(ptr {{[^,]*}} %0)
   a->foo(); // virtual call is not annotated
 }
 
-// CHECK-LABEL: define{{.*}} void @call_b_foo(%struct.B* noundef %b) #3
+// CHECK-LABEL: define{{.*}} void @call_b_foo(ptr noundef %b) #3
 extern "C" void call_b_foo(B *b) {
-  // CHECK: %call = call noundef i32 %2(%struct.B* {{[^,]*}} %0)
+  // CHECK: %call = call noundef i32 %1(ptr {{[^,]*}} %0)
   b->foo(); // virtual call is not annotated
 }
 
@@ -49,8 +49,8 @@ extern "C" void call_foo_no_mempcy() {
 A::~A() {} // Anchoring A so A::foo() gets generated
 B::~B() {} // Anchoring B so B::foo() gets generated
 
-// CHECK-LABEL: define linkonce_odr noundef i32 @_ZNK1A3fooEv(%struct.A* noundef{{[^,]*}} %this) unnamed_addr #0 comdat align 2
-// CHECK-LABEL: define linkonce_odr noundef i32 @_ZNK1B3fooEv(%struct.B* noundef{{[^,]*}} %this) unnamed_addr #6 comdat align 2
+// CHECK-LABEL: define linkonce_odr noundef i32 @_ZNK1A3fooEv(ptr noundef{{[^,]*}} %this) unnamed_addr #0 comdat align 2
+// CHECK-LABEL: define linkonce_odr noundef i32 @_ZNK1B3fooEv(ptr noundef{{[^,]*}} %this) unnamed_addr #6 comdat align 2
 
 // CHECK:     attributes #0 = {{{.*}}"no-builtin-memcpy"{{.*}}}
 // CHECK-NOT: attributes #0 = {{{.*}}"no-builtin-memmove"{{.*}}}
diff --git a/clang/test/CodeGen/nofpclass.c b/clang/test/CodeGen/nofpclass.c
new file mode 100644
index 0000000000000..7fea8192cda75
--- /dev/null
+++ b/clang/test/CodeGen/nofpclass.c
@@ -0,0 +1,1437 @@
+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --function-signature --check-attributes --version 2
+// REQUIRES: x86-registered-target
+// RUN: %clang_cc1 -triple x86_64-unknown-unknown -target-feature +avx -fenable-matrix -ffinite-math-only -emit-llvm -o - %s | FileCheck -check-prefixes=CFINITEONLY %s
+// RUN: %clang_cc1 -x cl -triple x86_64-unknown-unknown -target-feature +avx -fenable-matrix -cl-finite-math-only -emit-llvm -o - %s | FileCheck -check-prefixes=CLFINITEONLY %s
+
+// RUN: %clang_cc1 -triple x86_64-unknown-unknown -target-feature +avx -fenable-matrix -menable-no-nans -emit-llvm -o - %s | FileCheck -check-prefixes=NONANS %s
+// RUN: %clang_cc1 -triple x86_64-unknown-unknown -target-feature +avx -fenable-matrix -menable-no-infs -emit-llvm -o - %s | FileCheck -check-prefixes=NOINFS %s
+
+// XUN: %clang_cc1 -triple x86_64-unknown-unknown -target-feature +avx -fenable-matrix -fsignaling-nans -emit-llvm -o - %s | FileCheck -check-prefixes=SNANS %s
+// XUN: %clang_cc1 -triple x86_64-unknown-unknown -target-feature +avx -fenable-matrix -fno-signaling-nans -emit-llvm -o - %s | FileCheck -check-prefixes=NO-SNANS %s
+
+#ifdef __OPENCL_C_VERSION__
+#pragma OPENCL EXTENSION __cl_clang_variadic_functions : enable
+#endif
+
+typedef float float1 __attribute__((ext_vector_type(1)));
+typedef float float2 __attribute__((ext_vector_type(2)));
+typedef _Float16 half2 __attribute__((ext_vector_type(2)));
+typedef double double2 __attribute__((ext_vector_type(2)));
+typedef double dx5x5_t __attribute__((matrix_type(5, 5)));
+
+extern float extern_func(float, double, _Float16);
+extern float2 extern_func_vec(float2, double2, half2);
+extern _Complex float extern_complex(_Complex float, _Complex double, _Complex _Float16);
+extern float variadic(float, ...);
+extern dx5x5_t extern_matrix(dx5x5_t);
+
+
+// CFINITEONLY: Function Attrs: noinline nounwind optnone
+// CFINITEONLY-LABEL: define dso_local nofpclass(nan inf) float @defined_func_f32
+// CFINITEONLY-SAME: (float noundef nofpclass(nan inf) [[A:%.*]], float noundef nofpclass(nan inf) [[B:%.*]], float noundef nofpclass(nan inf) [[C:%.*]]) #[[ATTR0:[0-9]+]] {
+// CFINITEONLY-NEXT:  entry:
+// CFINITEONLY-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// CFINITEONLY-NEXT:    [[B_ADDR:%.*]] = alloca float, align 4
+// CFINITEONLY-NEXT:    [[C_ADDR:%.*]] = alloca float, align 4
+// CFINITEONLY-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// CFINITEONLY-NEXT:    store float [[B]], ptr [[B_ADDR]], align 4
+// CFINITEONLY-NEXT:    store float [[C]], ptr [[C_ADDR]], align 4
+// CFINITEONLY-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// CFINITEONLY-NEXT:    [[TMP1:%.*]] = load float, ptr [[B_ADDR]], align 4
+// CFINITEONLY-NEXT:    [[TMP2:%.*]] = load float, ptr [[C_ADDR]], align 4
+// CFINITEONLY-NEXT:    [[TMP3:%.*]] = call nnan ninf float @llvm.fma.f32(float [[TMP0]], float [[TMP1]], float [[TMP2]])
+// CFINITEONLY-NEXT:    [[ADD:%.*]] = fadd nnan ninf float [[TMP3]], 4.000000e+00
+// CFINITEONLY-NEXT:    ret float [[ADD]]
+//
+// CLFINITEONLY: Function Attrs: mustprogress nofree norecurse nosync nounwind willreturn memory(none)
+// CLFINITEONLY-LABEL: define dso_local nofpclass(nan inf) float @defined_func_f32
+// CLFINITEONLY-SAME: (float noundef nofpclass(nan inf) [[A:%.*]], float noundef nofpclass(nan inf) [[B:%.*]], float noundef nofpclass(nan inf) [[C:%.*]]) local_unnamed_addr #[[ATTR0:[0-9]+]] {
+// CLFINITEONLY-NEXT:  entry:
+// CLFINITEONLY-NEXT:    [[TMP0:%.*]] = tail call nnan ninf float @llvm.fma.f32(float [[A]], float [[B]], float [[C]])
+// CLFINITEONLY-NEXT:    [[ADD:%.*]] = fadd nnan ninf float [[TMP0]], 4.000000e+00
+// CLFINITEONLY-NEXT:    ret float [[ADD]]
+//
+// NONANS: Function Attrs: noinline nounwind optnone
+// NONANS-LABEL: define dso_local nofpclass(nan) float @defined_func_f32
+// NONANS-SAME: (float noundef nofpclass(nan) [[A:%.*]], float noundef nofpclass(nan) [[B:%.*]], float noundef nofpclass(nan) [[C:%.*]]) #[[ATTR0:[0-9]+]] {
+// NONANS-NEXT:  entry:
+// NONANS-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// NONANS-NEXT:    [[B_ADDR:%.*]] = alloca float, align 4
+// NONANS-NEXT:    [[C_ADDR:%.*]] = alloca float, align 4
+// NONANS-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// NONANS-NEXT:    store float [[B]], ptr [[B_ADDR]], align 4
+// NONANS-NEXT:    store float [[C]], ptr [[C_ADDR]], align 4
+// NONANS-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// NONANS-NEXT:    [[TMP1:%.*]] = load float, ptr [[B_ADDR]], align 4
+// NONANS-NEXT:    [[TMP2:%.*]] = load float, ptr [[C_ADDR]], align 4
+// NONANS-NEXT:    [[TMP3:%.*]] = call nnan float @llvm.fma.f32(float [[TMP0]], float [[TMP1]], float [[TMP2]])
+// NONANS-NEXT:    [[ADD:%.*]] = fadd nnan float [[TMP3]], 4.000000e+00
+// NONANS-NEXT:    ret float [[ADD]]
+//
+// NOINFS: Function Attrs: noinline nounwind optnone
+// NOINFS-LABEL: define dso_local nofpclass(inf) float @defined_func_f32
+// NOINFS-SAME: (float noundef nofpclass(inf) [[A:%.*]], float noundef nofpclass(inf) [[B:%.*]], float noundef nofpclass(inf) [[C:%.*]]) #[[ATTR0:[0-9]+]] {
+// NOINFS-NEXT:  entry:
+// NOINFS-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// NOINFS-NEXT:    [[B_ADDR:%.*]] = alloca float, align 4
+// NOINFS-NEXT:    [[C_ADDR:%.*]] = alloca float, align 4
+// NOINFS-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// NOINFS-NEXT:    store float [[B]], ptr [[B_ADDR]], align 4
+// NOINFS-NEXT:    store float [[C]], ptr [[C_ADDR]], align 4
+// NOINFS-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// NOINFS-NEXT:    [[TMP1:%.*]] = load float, ptr [[B_ADDR]], align 4
+// NOINFS-NEXT:    [[TMP2:%.*]] = load float, ptr [[C_ADDR]], align 4
+// NOINFS-NEXT:    [[TMP3:%.*]] = call ninf float @llvm.fma.f32(float [[TMP0]], float [[TMP1]], float [[TMP2]])
+// NOINFS-NEXT:    [[ADD:%.*]] = fadd ninf float [[TMP3]], 4.000000e+00
+// NOINFS-NEXT:    ret float [[ADD]]
+//
+float defined_func_f32(float a, float b, float c) {
+  return __builtin_fmaf(a, b, c) + 4.0f;
+}
+
+// CFINITEONLY: Function Attrs: noinline nounwind optnone
+// CFINITEONLY-LABEL: define dso_local nofpclass(nan inf) <2 x double> @defined_func_v2f64
+// CFINITEONLY-SAME: (<2 x double> noundef nofpclass(nan inf) [[A:%.*]], <2 x double> noundef nofpclass(nan inf) [[B:%.*]], <2 x double> noundef nofpclass(nan inf) [[C:%.*]]) #[[ATTR2:[0-9]+]] {
+// CFINITEONLY-NEXT:  entry:
+// CFINITEONLY-NEXT:    [[A_ADDR:%.*]] = alloca <2 x double>, align 16
+// CFINITEONLY-NEXT:    [[B_ADDR:%.*]] = alloca <2 x double>, align 16
+// CFINITEONLY-NEXT:    [[C_ADDR:%.*]] = alloca <2 x double>, align 16
+// CFINITEONLY-NEXT:    store <2 x double> [[A]], ptr [[A_ADDR]], align 16
+// CFINITEONLY-NEXT:    store <2 x double> [[B]], ptr [[B_ADDR]], align 16
+// CFINITEONLY-NEXT:    store <2 x double> [[C]], ptr [[C_ADDR]], align 16
+// CFINITEONLY-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[A_ADDR]], align 16
+// CFINITEONLY-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[B_ADDR]], align 16
+// CFINITEONLY-NEXT:    [[TMP2:%.*]] = load <2 x double>, ptr [[C_ADDR]], align 16
+// CFINITEONLY-NEXT:    [[TMP3:%.*]] = call nnan ninf <2 x double> @llvm.fma.v2f64(<2 x double> [[TMP0]], <2 x double> [[TMP1]], <2 x double> [[TMP2]])
+// CFINITEONLY-NEXT:    [[ADD:%.*]] = fadd nnan ninf <2 x double> [[TMP3]], <double 4.000000e+00, double 4.000000e+00>
+// CFINITEONLY-NEXT:    ret <2 x double> [[ADD]]
+//
+// CLFINITEONLY: Function Attrs: mustprogress nofree norecurse nosync nounwind willreturn memory(none)
+// CLFINITEONLY-LABEL: define dso_local nofpclass(nan inf) <2 x double> @defined_func_v2f64
+// CLFINITEONLY-SAME: (<2 x double> noundef nofpclass(nan inf) [[A:%.*]], <2 x double> noundef nofpclass(nan inf) [[B:%.*]], <2 x double> noundef nofpclass(nan inf) [[C:%.*]]) local_unnamed_addr #[[ATTR2:[0-9]+]] {
+// CLFINITEONLY-NEXT:  entry:
+// CLFINITEONLY-NEXT:    [[TMP0:%.*]] = tail call nnan ninf <2 x double> @llvm.fma.v2f64(<2 x double> [[A]], <2 x double> [[B]], <2 x double> [[C]])
+// CLFINITEONLY-NEXT:    [[ADD:%.*]] = fadd nnan ninf <2 x double> [[TMP0]], <double 4.000000e+00, double 4.000000e+00>
+// CLFINITEONLY-NEXT:    ret <2 x double> [[ADD]]
+//
+// NONANS: Function Attrs: noinline nounwind optnone
+// NONANS-LABEL: define dso_local nofpclass(nan) <2 x double> @defined_func_v2f64
+// NONANS-SAME: (<2 x double> noundef nofpclass(nan) [[A:%.*]], <2 x double> noundef nofpclass(nan) [[B:%.*]], <2 x double> noundef nofpclass(nan) [[C:%.*]]) #[[ATTR2:[0-9]+]] {
+// NONANS-NEXT:  entry:
+// NONANS-NEXT:    [[A_ADDR:%.*]] = alloca <2 x double>, align 16
+// NONANS-NEXT:    [[B_ADDR:%.*]] = alloca <2 x double>, align 16
+// NONANS-NEXT:    [[C_ADDR:%.*]] = alloca <2 x double>, align 16
+// NONANS-NEXT:    store <2 x double> [[A]], ptr [[A_ADDR]], align 16
+// NONANS-NEXT:    store <2 x double> [[B]], ptr [[B_ADDR]], align 16
+// NONANS-NEXT:    store <2 x double> [[C]], ptr [[C_ADDR]], align 16
+// NONANS-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[A_ADDR]], align 16
+// NONANS-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[B_ADDR]], align 16
+// NONANS-NEXT:    [[TMP2:%.*]] = load <2 x double>, ptr [[C_ADDR]], align 16
+// NONANS-NEXT:    [[TMP3:%.*]] = call nnan <2 x double> @llvm.fma.v2f64(<2 x double> [[TMP0]], <2 x double> [[TMP1]], <2 x double> [[TMP2]])
+// NONANS-NEXT:    [[ADD:%.*]] = fadd nnan <2 x double> [[TMP3]], <double 4.000000e+00, double 4.000000e+00>
+// NONANS-NEXT:    ret <2 x double> [[ADD]]
+//
+// NOINFS: Function Attrs: noinline nounwind optnone
+// NOINFS-LABEL: define dso_local nofpclass(inf) <2 x double> @defined_func_v2f64
+// NOINFS-SAME: (<2 x double> noundef nofpclass(inf) [[A:%.*]], <2 x double> noundef nofpclass(inf) [[B:%.*]], <2 x double> noundef nofpclass(inf) [[C:%.*]]) #[[ATTR2:[0-9]+]] {
+// NOINFS-NEXT:  entry:
+// NOINFS-NEXT:    [[A_ADDR:%.*]] = alloca <2 x double>, align 16
+// NOINFS-NEXT:    [[B_ADDR:%.*]] = alloca <2 x double>, align 16
+// NOINFS-NEXT:    [[C_ADDR:%.*]] = alloca <2 x double>, align 16
+// NOINFS-NEXT:    store <2 x double> [[A]], ptr [[A_ADDR]], align 16
+// NOINFS-NEXT:    store <2 x double> [[B]], ptr [[B_ADDR]], align 16
+// NOINFS-NEXT:    store <2 x double> [[C]], ptr [[C_ADDR]], align 16
+// NOINFS-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[A_ADDR]], align 16
+// NOINFS-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[B_ADDR]], align 16
+// NOINFS-NEXT:    [[TMP2:%.*]] = load <2 x double>, ptr [[C_ADDR]], align 16
+// NOINFS-NEXT:    [[TMP3:%.*]] = call ninf <2 x double> @llvm.fma.v2f64(<2 x double> [[TMP0]], <2 x double> [[TMP1]], <2 x double> [[TMP2]])
+// NOINFS-NEXT:    [[ADD:%.*]] = fadd ninf <2 x double> [[TMP3]], <double 4.000000e+00, double 4.000000e+00>
+// NOINFS-NEXT:    ret <2 x double> [[ADD]]
+//
+double2 defined_func_v2f64(double2 a, double2 b, double2 c) {
+  return __builtin_elementwise_fma(a, b, c) + 4.0;
+}
+
+// CFINITEONLY: Function Attrs: noinline nounwind optnone
+// CFINITEONLY-LABEL: define dso_local nofpclass(nan inf) float @call_extern_func
+// CFINITEONLY-SAME: (float noundef nofpclass(nan inf) [[A:%.*]], double noundef nofpclass(nan inf) [[B:%.*]], half noundef nofpclass(nan inf) [[C:%.*]]) #[[ATTR0]] {
+// CFINITEONLY-NEXT:  entry:
+// CFINITEONLY-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// CFINITEONLY-NEXT:    [[B_ADDR:%.*]] = alloca double, align 8
+// CFINITEONLY-NEXT:    [[C_ADDR:%.*]] = alloca half, align 2
+// CFINITEONLY-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// CFINITEONLY-NEXT:    store double [[B]], ptr [[B_ADDR]], align 8
+// CFINITEONLY-NEXT:    store half [[C]], ptr [[C_ADDR]], align 2
+// CFINITEONLY-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// CFINITEONLY-NEXT:    [[TMP1:%.*]] = load double, ptr [[B_ADDR]], align 8
+// CFINITEONLY-NEXT:    [[TMP2:%.*]] = load half, ptr [[C_ADDR]], align 2
+// CFINITEONLY-NEXT:    [[CALL:%.*]] = call nnan ninf nofpclass(nan inf) float @extern_func(float noundef nofpclass(nan inf) [[TMP0]], double noundef nofpclass(nan inf) [[TMP1]], half noundef nofpclass(nan inf) [[TMP2]])
+// CFINITEONLY-NEXT:    ret float [[CALL]]
+//
+// CLFINITEONLY: Function Attrs: convergent norecurse nounwind
+// CLFINITEONLY-LABEL: define dso_local nofpclass(nan inf) float @call_extern_func
+// CLFINITEONLY-SAME: (float noundef nofpclass(nan inf) [[A:%.*]], double noundef nofpclass(nan inf) [[B:%.*]], half noundef nofpclass(nan inf) [[C:%.*]]) local_unnamed_addr #[[ATTR3:[0-9]+]] {
+// CLFINITEONLY-NEXT:  entry:
+// CLFINITEONLY-NEXT:    [[CALL:%.*]] = tail call nnan ninf nofpclass(nan inf) float @extern_func(float noundef nofpclass(nan inf) [[A]], double noundef nofpclass(nan inf) [[B]], half noundef nofpclass(nan inf) [[C]]) #[[ATTR10:[0-9]+]]
+// CLFINITEONLY-NEXT:    ret float [[CALL]]
+//
+// NONANS: Function Attrs: noinline nounwind optnone
+// NONANS-LABEL: define dso_local nofpclass(nan) float @call_extern_func
+// NONANS-SAME: (float noundef nofpclass(nan) [[A:%.*]], double noundef nofpclass(nan) [[B:%.*]], half noundef nofpclass(nan) [[C:%.*]]) #[[ATTR0]] {
+// NONANS-NEXT:  entry:
+// NONANS-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// NONANS-NEXT:    [[B_ADDR:%.*]] = alloca double, align 8
+// NONANS-NEXT:    [[C_ADDR:%.*]] = alloca half, align 2
+// NONANS-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// NONANS-NEXT:    store double [[B]], ptr [[B_ADDR]], align 8
+// NONANS-NEXT:    store half [[C]], ptr [[C_ADDR]], align 2
+// NONANS-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// NONANS-NEXT:    [[TMP1:%.*]] = load double, ptr [[B_ADDR]], align 8
+// NONANS-NEXT:    [[TMP2:%.*]] = load half, ptr [[C_ADDR]], align 2
+// NONANS-NEXT:    [[CALL:%.*]] = call nnan nofpclass(nan) float @extern_func(float noundef nofpclass(nan) [[TMP0]], double noundef nofpclass(nan) [[TMP1]], half noundef nofpclass(nan) [[TMP2]])
+// NONANS-NEXT:    ret float [[CALL]]
+//
+// NOINFS: Function Attrs: noinline nounwind optnone
+// NOINFS-LABEL: define dso_local nofpclass(inf) float @call_extern_func
+// NOINFS-SAME: (float noundef nofpclass(inf) [[A:%.*]], double noundef nofpclass(inf) [[B:%.*]], half noundef nofpclass(inf) [[C:%.*]]) #[[ATTR0]] {
+// NOINFS-NEXT:  entry:
+// NOINFS-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// NOINFS-NEXT:    [[B_ADDR:%.*]] = alloca double, align 8
+// NOINFS-NEXT:    [[C_ADDR:%.*]] = alloca half, align 2
+// NOINFS-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// NOINFS-NEXT:    store double [[B]], ptr [[B_ADDR]], align 8
+// NOINFS-NEXT:    store half [[C]], ptr [[C_ADDR]], align 2
+// NOINFS-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// NOINFS-NEXT:    [[TMP1:%.*]] = load double, ptr [[B_ADDR]], align 8
+// NOINFS-NEXT:    [[TMP2:%.*]] = load half, ptr [[C_ADDR]], align 2
+// NOINFS-NEXT:    [[CALL:%.*]] = call ninf nofpclass(inf) float @extern_func(float noundef nofpclass(inf) [[TMP0]], double noundef nofpclass(inf) [[TMP1]], half noundef nofpclass(inf) [[TMP2]])
+// NOINFS-NEXT:    ret float [[CALL]]
+//
+float call_extern_func(float a, double b, _Float16 c) {
+  return extern_func(a, b, c);
+}
+
+// CFINITEONLY: Function Attrs: noinline nounwind optnone
+// CFINITEONLY-LABEL: define dso_local nofpclass(nan inf) double @call_extern_func_vec
+// CFINITEONLY-SAME: (double noundef nofpclass(nan inf) [[A_COERCE:%.*]], <2 x double> noundef nofpclass(nan inf) [[B:%.*]], i32 noundef [[C_COERCE:%.*]]) #[[ATTR2]] {
+// CFINITEONLY-NEXT:  entry:
+// CFINITEONLY-NEXT:    [[RETVAL:%.*]] = alloca <2 x float>, align 8
+// CFINITEONLY-NEXT:    [[A:%.*]] = alloca <2 x float>, align 8
+// CFINITEONLY-NEXT:    [[C:%.*]] = alloca <2 x half>, align 4
+// CFINITEONLY-NEXT:    [[A_ADDR:%.*]] = alloca <2 x float>, align 8
+// CFINITEONLY-NEXT:    [[B_ADDR:%.*]] = alloca <2 x double>, align 16
+// CFINITEONLY-NEXT:    [[C_ADDR:%.*]] = alloca <2 x half>, align 4
+// CFINITEONLY-NEXT:    [[COERCE:%.*]] = alloca <2 x float>, align 8
+// CFINITEONLY-NEXT:    [[COERCE3:%.*]] = alloca <2 x half>, align 4
+// CFINITEONLY-NEXT:    [[COERCE4:%.*]] = alloca <2 x float>, align 8
+// CFINITEONLY-NEXT:    store double [[A_COERCE]], ptr [[A]], align 8
+// CFINITEONLY-NEXT:    [[A1:%.*]] = load <2 x float>, ptr [[A]], align 8
+// CFINITEONLY-NEXT:    store i32 [[C_COERCE]], ptr [[C]], align 4
+// CFINITEONLY-NEXT:    [[C2:%.*]] = load <2 x half>, ptr [[C]], align 4
+// CFINITEONLY-NEXT:    store <2 x float> [[A1]], ptr [[A_ADDR]], align 8
+// CFINITEONLY-NEXT:    store <2 x double> [[B]], ptr [[B_ADDR]], align 16
+// CFINITEONLY-NEXT:    store <2 x half> [[C2]], ptr [[C_ADDR]], align 4
+// CFINITEONLY-NEXT:    [[TMP0:%.*]] = load <2 x float>, ptr [[A_ADDR]], align 8
+// CFINITEONLY-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[B_ADDR]], align 16
+// CFINITEONLY-NEXT:    [[TMP2:%.*]] = load <2 x half>, ptr [[C_ADDR]], align 4
+// CFINITEONLY-NEXT:    store <2 x float> [[TMP0]], ptr [[COERCE]], align 8
+// CFINITEONLY-NEXT:    [[TMP3:%.*]] = load double, ptr [[COERCE]], align 8
+// CFINITEONLY-NEXT:    store <2 x half> [[TMP2]], ptr [[COERCE3]], align 4
+// CFINITEONLY-NEXT:    [[TMP4:%.*]] = load i32, ptr [[COERCE3]], align 4
+// CFINITEONLY-NEXT:    [[CALL:%.*]] = call nnan ninf nofpclass(nan inf) double @extern_func_vec(double noundef nofpclass(nan inf) [[TMP3]], <2 x double> noundef nofpclass(nan inf) [[TMP1]], i32 noundef [[TMP4]])
+// CFINITEONLY-NEXT:    store double [[CALL]], ptr [[COERCE4]], align 8
+// CFINITEONLY-NEXT:    [[TMP5:%.*]] = load <2 x float>, ptr [[COERCE4]], align 8
+// CFINITEONLY-NEXT:    store <2 x float> [[TMP5]], ptr [[RETVAL]], align 8
+// CFINITEONLY-NEXT:    [[TMP6:%.*]] = load double, ptr [[RETVAL]], align 8
+// CFINITEONLY-NEXT:    ret double [[TMP6]]
+//
+// CLFINITEONLY: Function Attrs: convergent norecurse nounwind
+// CLFINITEONLY-LABEL: define dso_local nofpclass(nan inf) double @call_extern_func_vec
+// CLFINITEONLY-SAME: (double noundef nofpclass(nan inf) [[A_COERCE:%.*]], <2 x double> noundef nofpclass(nan inf) [[B:%.*]], i32 noundef [[C_COERCE:%.*]]) local_unnamed_addr #[[ATTR5:[0-9]+]] {
+// CLFINITEONLY-NEXT:  entry:
+// CLFINITEONLY-NEXT:    [[CALL:%.*]] = tail call nnan ninf nofpclass(nan inf) double @extern_func_vec(double noundef nofpclass(nan inf) [[A_COERCE]], <2 x double> noundef nofpclass(nan inf) [[B]], i32 noundef [[C_COERCE]]) #[[ATTR10]]
+// CLFINITEONLY-NEXT:    ret double [[CALL]]
+//
+// NONANS: Function Attrs: noinline nounwind optnone
+// NONANS-LABEL: define dso_local nofpclass(nan) double @call_extern_func_vec
+// NONANS-SAME: (double noundef nofpclass(nan) [[A_COERCE:%.*]], <2 x double> noundef nofpclass(nan) [[B:%.*]], i32 noundef [[C_COERCE:%.*]]) #[[ATTR2]] {
+// NONANS-NEXT:  entry:
+// NONANS-NEXT:    [[RETVAL:%.*]] = alloca <2 x float>, align 8
+// NONANS-NEXT:    [[A:%.*]] = alloca <2 x float>, align 8
+// NONANS-NEXT:    [[C:%.*]] = alloca <2 x half>, align 4
+// NONANS-NEXT:    [[A_ADDR:%.*]] = alloca <2 x float>, align 8
+// NONANS-NEXT:    [[B_ADDR:%.*]] = alloca <2 x double>, align 16
+// NONANS-NEXT:    [[C_ADDR:%.*]] = alloca <2 x half>, align 4
+// NONANS-NEXT:    [[COERCE:%.*]] = alloca <2 x float>, align 8
+// NONANS-NEXT:    [[COERCE3:%.*]] = alloca <2 x half>, align 4
+// NONANS-NEXT:    [[COERCE4:%.*]] = alloca <2 x float>, align 8
+// NONANS-NEXT:    store double [[A_COERCE]], ptr [[A]], align 8
+// NONANS-NEXT:    [[A1:%.*]] = load <2 x float>, ptr [[A]], align 8
+// NONANS-NEXT:    store i32 [[C_COERCE]], ptr [[C]], align 4
+// NONANS-NEXT:    [[C2:%.*]] = load <2 x half>, ptr [[C]], align 4
+// NONANS-NEXT:    store <2 x float> [[A1]], ptr [[A_ADDR]], align 8
+// NONANS-NEXT:    store <2 x double> [[B]], ptr [[B_ADDR]], align 16
+// NONANS-NEXT:    store <2 x half> [[C2]], ptr [[C_ADDR]], align 4
+// NONANS-NEXT:    [[TMP0:%.*]] = load <2 x float>, ptr [[A_ADDR]], align 8
+// NONANS-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[B_ADDR]], align 16
+// NONANS-NEXT:    [[TMP2:%.*]] = load <2 x half>, ptr [[C_ADDR]], align 4
+// NONANS-NEXT:    store <2 x float> [[TMP0]], ptr [[COERCE]], align 8
+// NONANS-NEXT:    [[TMP3:%.*]] = load double, ptr [[COERCE]], align 8
+// NONANS-NEXT:    store <2 x half> [[TMP2]], ptr [[COERCE3]], align 4
+// NONANS-NEXT:    [[TMP4:%.*]] = load i32, ptr [[COERCE3]], align 4
+// NONANS-NEXT:    [[CALL:%.*]] = call nnan nofpclass(nan) double @extern_func_vec(double noundef nofpclass(nan) [[TMP3]], <2 x double> noundef nofpclass(nan) [[TMP1]], i32 noundef [[TMP4]])
+// NONANS-NEXT:    store double [[CALL]], ptr [[COERCE4]], align 8
+// NONANS-NEXT:    [[TMP5:%.*]] = load <2 x float>, ptr [[COERCE4]], align 8
+// NONANS-NEXT:    store <2 x float> [[TMP5]], ptr [[RETVAL]], align 8
+// NONANS-NEXT:    [[TMP6:%.*]] = load double, ptr [[RETVAL]], align 8
+// NONANS-NEXT:    ret double [[TMP6]]
+//
+// NOINFS: Function Attrs: noinline nounwind optnone
+// NOINFS-LABEL: define dso_local nofpclass(inf) double @call_extern_func_vec
+// NOINFS-SAME: (double noundef nofpclass(inf) [[A_COERCE:%.*]], <2 x double> noundef nofpclass(inf) [[B:%.*]], i32 noundef [[C_COERCE:%.*]]) #[[ATTR2]] {
+// NOINFS-NEXT:  entry:
+// NOINFS-NEXT:    [[RETVAL:%.*]] = alloca <2 x float>, align 8
+// NOINFS-NEXT:    [[A:%.*]] = alloca <2 x float>, align 8
+// NOINFS-NEXT:    [[C:%.*]] = alloca <2 x half>, align 4
+// NOINFS-NEXT:    [[A_ADDR:%.*]] = alloca <2 x float>, align 8
+// NOINFS-NEXT:    [[B_ADDR:%.*]] = alloca <2 x double>, align 16
+// NOINFS-NEXT:    [[C_ADDR:%.*]] = alloca <2 x half>, align 4
+// NOINFS-NEXT:    [[COERCE:%.*]] = alloca <2 x float>, align 8
+// NOINFS-NEXT:    [[COERCE3:%.*]] = alloca <2 x half>, align 4
+// NOINFS-NEXT:    [[COERCE4:%.*]] = alloca <2 x float>, align 8
+// NOINFS-NEXT:    store double [[A_COERCE]], ptr [[A]], align 8
+// NOINFS-NEXT:    [[A1:%.*]] = load <2 x float>, ptr [[A]], align 8
+// NOINFS-NEXT:    store i32 [[C_COERCE]], ptr [[C]], align 4
+// NOINFS-NEXT:    [[C2:%.*]] = load <2 x half>, ptr [[C]], align 4
+// NOINFS-NEXT:    store <2 x float> [[A1]], ptr [[A_ADDR]], align 8
+// NOINFS-NEXT:    store <2 x double> [[B]], ptr [[B_ADDR]], align 16
+// NOINFS-NEXT:    store <2 x half> [[C2]], ptr [[C_ADDR]], align 4
+// NOINFS-NEXT:    [[TMP0:%.*]] = load <2 x float>, ptr [[A_ADDR]], align 8
+// NOINFS-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[B_ADDR]], align 16
+// NOINFS-NEXT:    [[TMP2:%.*]] = load <2 x half>, ptr [[C_ADDR]], align 4
+// NOINFS-NEXT:    store <2 x float> [[TMP0]], ptr [[COERCE]], align 8
+// NOINFS-NEXT:    [[TMP3:%.*]] = load double, ptr [[COERCE]], align 8
+// NOINFS-NEXT:    store <2 x half> [[TMP2]], ptr [[COERCE3]], align 4
+// NOINFS-NEXT:    [[TMP4:%.*]] = load i32, ptr [[COERCE3]], align 4
+// NOINFS-NEXT:    [[CALL:%.*]] = call ninf nofpclass(inf) double @extern_func_vec(double noundef nofpclass(inf) [[TMP3]], <2 x double> noundef nofpclass(inf) [[TMP1]], i32 noundef [[TMP4]])
+// NOINFS-NEXT:    store double [[CALL]], ptr [[COERCE4]], align 8
+// NOINFS-NEXT:    [[TMP5:%.*]] = load <2 x float>, ptr [[COERCE4]], align 8
+// NOINFS-NEXT:    store <2 x float> [[TMP5]], ptr [[RETVAL]], align 8
+// NOINFS-NEXT:    [[TMP6:%.*]] = load double, ptr [[RETVAL]], align 8
+// NOINFS-NEXT:    ret double [[TMP6]]
+//
+float2 call_extern_func_vec(float2 a, double2 b, half2 c) {
+  return extern_func_vec(a, b, c);
+}
+
+// CFINITEONLY: Function Attrs: noinline nounwind optnone
+// CFINITEONLY-LABEL: define dso_local nofpclass(nan inf) <2 x float> @defined_complex_func
+// CFINITEONLY-SAME: (<2 x float> noundef nofpclass(nan inf) [[A_COERCE:%.*]], double noundef nofpclass(nan inf) [[B_COERCE0:%.*]], double noundef nofpclass(nan inf) [[B_COERCE1:%.*]], <2 x half> noundef nofpclass(nan inf) [[C_COERCE:%.*]]) #[[ATTR4:[0-9]+]] {
+// CFINITEONLY-NEXT:  entry:
+// CFINITEONLY-NEXT:    [[RETVAL:%.*]] = alloca { float, float }, align 4
+// CFINITEONLY-NEXT:    [[A:%.*]] = alloca { float, float }, align 4
+// CFINITEONLY-NEXT:    [[B:%.*]] = alloca { double, double }, align 8
+// CFINITEONLY-NEXT:    [[C:%.*]] = alloca { half, half }, align 2
+// CFINITEONLY-NEXT:    [[COERCE:%.*]] = alloca { float, float }, align 4
+// CFINITEONLY-NEXT:    [[COERCE1:%.*]] = alloca { double, double }, align 8
+// CFINITEONLY-NEXT:    [[COERCE2:%.*]] = alloca { half, half }, align 2
+// CFINITEONLY-NEXT:    [[COERCE3:%.*]] = alloca { float, float }, align 4
+// CFINITEONLY-NEXT:    store <2 x float> [[A_COERCE]], ptr [[A]], align 4
+// CFINITEONLY-NEXT:    [[TMP0:%.*]] = getelementptr inbounds { double, double }, ptr [[B]], i32 0, i32 0
+// CFINITEONLY-NEXT:    store double [[B_COERCE0]], ptr [[TMP0]], align 8
+// CFINITEONLY-NEXT:    [[TMP1:%.*]] = getelementptr inbounds { double, double }, ptr [[B]], i32 0, i32 1
+// CFINITEONLY-NEXT:    store double [[B_COERCE1]], ptr [[TMP1]], align 8
+// CFINITEONLY-NEXT:    store <2 x half> [[C_COERCE]], ptr [[C]], align 2
+// CFINITEONLY-NEXT:    [[A_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[A]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[A_REAL:%.*]] = load float, ptr [[A_REALP]], align 4
+// CFINITEONLY-NEXT:    [[A_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[A]], i32 0, i32 1
+// CFINITEONLY-NEXT:    [[A_IMAG:%.*]] = load float, ptr [[A_IMAGP]], align 4
+// CFINITEONLY-NEXT:    [[B_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[B]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[B_REAL:%.*]] = load double, ptr [[B_REALP]], align 8
+// CFINITEONLY-NEXT:    [[B_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[B]], i32 0, i32 1
+// CFINITEONLY-NEXT:    [[B_IMAG:%.*]] = load double, ptr [[B_IMAGP]], align 8
+// CFINITEONLY-NEXT:    [[C_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[C]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[C_REAL:%.*]] = load half, ptr [[C_REALP]], align 2
+// CFINITEONLY-NEXT:    [[C_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[C]], i32 0, i32 1
+// CFINITEONLY-NEXT:    [[C_IMAG:%.*]] = load half, ptr [[C_IMAGP]], align 2
+// CFINITEONLY-NEXT:    [[COERCE_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[COERCE_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE]], i32 0, i32 1
+// CFINITEONLY-NEXT:    store float [[A_REAL]], ptr [[COERCE_REALP]], align 4
+// CFINITEONLY-NEXT:    store float [[A_IMAG]], ptr [[COERCE_IMAGP]], align 4
+// CFINITEONLY-NEXT:    [[TMP2:%.*]] = load <2 x float>, ptr [[COERCE]], align 4
+// CFINITEONLY-NEXT:    [[COERCE1_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[COERCE1]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[COERCE1_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[COERCE1]], i32 0, i32 1
+// CFINITEONLY-NEXT:    store double [[B_REAL]], ptr [[COERCE1_REALP]], align 8
+// CFINITEONLY-NEXT:    store double [[B_IMAG]], ptr [[COERCE1_IMAGP]], align 8
+// CFINITEONLY-NEXT:    [[TMP3:%.*]] = getelementptr inbounds { double, double }, ptr [[COERCE1]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[TMP4:%.*]] = load double, ptr [[TMP3]], align 8
+// CFINITEONLY-NEXT:    [[TMP5:%.*]] = getelementptr inbounds { double, double }, ptr [[COERCE1]], i32 0, i32 1
+// CFINITEONLY-NEXT:    [[TMP6:%.*]] = load double, ptr [[TMP5]], align 8
+// CFINITEONLY-NEXT:    [[COERCE2_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[COERCE2]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[COERCE2_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[COERCE2]], i32 0, i32 1
+// CFINITEONLY-NEXT:    store half [[C_REAL]], ptr [[COERCE2_REALP]], align 2
+// CFINITEONLY-NEXT:    store half [[C_IMAG]], ptr [[COERCE2_IMAGP]], align 2
+// CFINITEONLY-NEXT:    [[TMP7:%.*]] = load <2 x half>, ptr [[COERCE2]], align 2
+// CFINITEONLY-NEXT:    [[CALL:%.*]] = call nnan ninf nofpclass(nan inf) <2 x float> @extern_complex(<2 x float> noundef nofpclass(nan inf) [[TMP2]], double noundef nofpclass(nan inf) [[TMP4]], double noundef nofpclass(nan inf) [[TMP6]], <2 x half> noundef nofpclass(nan inf) [[TMP7]])
+// CFINITEONLY-NEXT:    store <2 x float> [[CALL]], ptr [[COERCE3]], align 4
+// CFINITEONLY-NEXT:    [[COERCE3_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE3]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[COERCE3_REAL:%.*]] = load float, ptr [[COERCE3_REALP]], align 4
+// CFINITEONLY-NEXT:    [[COERCE3_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE3]], i32 0, i32 1
+// CFINITEONLY-NEXT:    [[COERCE3_IMAG:%.*]] = load float, ptr [[COERCE3_IMAGP]], align 4
+// CFINITEONLY-NEXT:    [[RETVAL_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[RETVAL]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[RETVAL_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[RETVAL]], i32 0, i32 1
+// CFINITEONLY-NEXT:    store float [[COERCE3_REAL]], ptr [[RETVAL_REALP]], align 4
+// CFINITEONLY-NEXT:    store float [[COERCE3_IMAG]], ptr [[RETVAL_IMAGP]], align 4
+// CFINITEONLY-NEXT:    [[TMP8:%.*]] = load <2 x float>, ptr [[RETVAL]], align 4
+// CFINITEONLY-NEXT:    ret <2 x float> [[TMP8]]
+//
+// CLFINITEONLY: Function Attrs: convergent norecurse nounwind
+// CLFINITEONLY-LABEL: define dso_local nofpclass(nan inf) <2 x float> @defined_complex_func
+// CLFINITEONLY-SAME: (<2 x float> noundef nofpclass(nan inf) [[A_COERCE:%.*]], double noundef nofpclass(nan inf) [[B_COERCE0:%.*]], double noundef nofpclass(nan inf) [[B_COERCE1:%.*]], <2 x half> noundef nofpclass(nan inf) [[C_COERCE:%.*]]) local_unnamed_addr #[[ATTR6:[0-9]+]] {
+// CLFINITEONLY-NEXT:  entry:
+// CLFINITEONLY-NEXT:    [[CALL:%.*]] = tail call nnan ninf nofpclass(nan inf) <2 x float> @extern_complex(<2 x float> noundef nofpclass(nan inf) [[A_COERCE]], double noundef nofpclass(nan inf) [[B_COERCE0]], double noundef nofpclass(nan inf) [[B_COERCE1]], <2 x half> noundef nofpclass(nan inf) [[C_COERCE]]) #[[ATTR10]]
+// CLFINITEONLY-NEXT:    ret <2 x float> [[CALL]]
+//
+// NONANS: Function Attrs: noinline nounwind optnone
+// NONANS-LABEL: define dso_local nofpclass(nan) <2 x float> @defined_complex_func
+// NONANS-SAME: (<2 x float> noundef nofpclass(nan) [[A_COERCE:%.*]], double noundef nofpclass(nan) [[B_COERCE0:%.*]], double noundef nofpclass(nan) [[B_COERCE1:%.*]], <2 x half> noundef nofpclass(nan) [[C_COERCE:%.*]]) #[[ATTR4:[0-9]+]] {
+// NONANS-NEXT:  entry:
+// NONANS-NEXT:    [[RETVAL:%.*]] = alloca { float, float }, align 4
+// NONANS-NEXT:    [[A:%.*]] = alloca { float, float }, align 4
+// NONANS-NEXT:    [[B:%.*]] = alloca { double, double }, align 8
+// NONANS-NEXT:    [[C:%.*]] = alloca { half, half }, align 2
+// NONANS-NEXT:    [[COERCE:%.*]] = alloca { float, float }, align 4
+// NONANS-NEXT:    [[COERCE1:%.*]] = alloca { double, double }, align 8
+// NONANS-NEXT:    [[COERCE2:%.*]] = alloca { half, half }, align 2
+// NONANS-NEXT:    [[COERCE3:%.*]] = alloca { float, float }, align 4
+// NONANS-NEXT:    store <2 x float> [[A_COERCE]], ptr [[A]], align 4
+// NONANS-NEXT:    [[TMP0:%.*]] = getelementptr inbounds { double, double }, ptr [[B]], i32 0, i32 0
+// NONANS-NEXT:    store double [[B_COERCE0]], ptr [[TMP0]], align 8
+// NONANS-NEXT:    [[TMP1:%.*]] = getelementptr inbounds { double, double }, ptr [[B]], i32 0, i32 1
+// NONANS-NEXT:    store double [[B_COERCE1]], ptr [[TMP1]], align 8
+// NONANS-NEXT:    store <2 x half> [[C_COERCE]], ptr [[C]], align 2
+// NONANS-NEXT:    [[A_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[A]], i32 0, i32 0
+// NONANS-NEXT:    [[A_REAL:%.*]] = load float, ptr [[A_REALP]], align 4
+// NONANS-NEXT:    [[A_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[A]], i32 0, i32 1
+// NONANS-NEXT:    [[A_IMAG:%.*]] = load float, ptr [[A_IMAGP]], align 4
+// NONANS-NEXT:    [[B_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[B]], i32 0, i32 0
+// NONANS-NEXT:    [[B_REAL:%.*]] = load double, ptr [[B_REALP]], align 8
+// NONANS-NEXT:    [[B_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[B]], i32 0, i32 1
+// NONANS-NEXT:    [[B_IMAG:%.*]] = load double, ptr [[B_IMAGP]], align 8
+// NONANS-NEXT:    [[C_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[C]], i32 0, i32 0
+// NONANS-NEXT:    [[C_REAL:%.*]] = load half, ptr [[C_REALP]], align 2
+// NONANS-NEXT:    [[C_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[C]], i32 0, i32 1
+// NONANS-NEXT:    [[C_IMAG:%.*]] = load half, ptr [[C_IMAGP]], align 2
+// NONANS-NEXT:    [[COERCE_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE]], i32 0, i32 0
+// NONANS-NEXT:    [[COERCE_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE]], i32 0, i32 1
+// NONANS-NEXT:    store float [[A_REAL]], ptr [[COERCE_REALP]], align 4
+// NONANS-NEXT:    store float [[A_IMAG]], ptr [[COERCE_IMAGP]], align 4
+// NONANS-NEXT:    [[TMP2:%.*]] = load <2 x float>, ptr [[COERCE]], align 4
+// NONANS-NEXT:    [[COERCE1_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[COERCE1]], i32 0, i32 0
+// NONANS-NEXT:    [[COERCE1_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[COERCE1]], i32 0, i32 1
+// NONANS-NEXT:    store double [[B_REAL]], ptr [[COERCE1_REALP]], align 8
+// NONANS-NEXT:    store double [[B_IMAG]], ptr [[COERCE1_IMAGP]], align 8
+// NONANS-NEXT:    [[TMP3:%.*]] = getelementptr inbounds { double, double }, ptr [[COERCE1]], i32 0, i32 0
+// NONANS-NEXT:    [[TMP4:%.*]] = load double, ptr [[TMP3]], align 8
+// NONANS-NEXT:    [[TMP5:%.*]] = getelementptr inbounds { double, double }, ptr [[COERCE1]], i32 0, i32 1
+// NONANS-NEXT:    [[TMP6:%.*]] = load double, ptr [[TMP5]], align 8
+// NONANS-NEXT:    [[COERCE2_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[COERCE2]], i32 0, i32 0
+// NONANS-NEXT:    [[COERCE2_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[COERCE2]], i32 0, i32 1
+// NONANS-NEXT:    store half [[C_REAL]], ptr [[COERCE2_REALP]], align 2
+// NONANS-NEXT:    store half [[C_IMAG]], ptr [[COERCE2_IMAGP]], align 2
+// NONANS-NEXT:    [[TMP7:%.*]] = load <2 x half>, ptr [[COERCE2]], align 2
+// NONANS-NEXT:    [[CALL:%.*]] = call nnan nofpclass(nan) <2 x float> @extern_complex(<2 x float> noundef nofpclass(nan) [[TMP2]], double noundef nofpclass(nan) [[TMP4]], double noundef nofpclass(nan) [[TMP6]], <2 x half> noundef nofpclass(nan) [[TMP7]])
+// NONANS-NEXT:    store <2 x float> [[CALL]], ptr [[COERCE3]], align 4
+// NONANS-NEXT:    [[COERCE3_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE3]], i32 0, i32 0
+// NONANS-NEXT:    [[COERCE3_REAL:%.*]] = load float, ptr [[COERCE3_REALP]], align 4
+// NONANS-NEXT:    [[COERCE3_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE3]], i32 0, i32 1
+// NONANS-NEXT:    [[COERCE3_IMAG:%.*]] = load float, ptr [[COERCE3_IMAGP]], align 4
+// NONANS-NEXT:    [[RETVAL_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[RETVAL]], i32 0, i32 0
+// NONANS-NEXT:    [[RETVAL_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[RETVAL]], i32 0, i32 1
+// NONANS-NEXT:    store float [[COERCE3_REAL]], ptr [[RETVAL_REALP]], align 4
+// NONANS-NEXT:    store float [[COERCE3_IMAG]], ptr [[RETVAL_IMAGP]], align 4
+// NONANS-NEXT:    [[TMP8:%.*]] = load <2 x float>, ptr [[RETVAL]], align 4
+// NONANS-NEXT:    ret <2 x float> [[TMP8]]
+//
+// NOINFS: Function Attrs: noinline nounwind optnone
+// NOINFS-LABEL: define dso_local nofpclass(inf) <2 x float> @defined_complex_func
+// NOINFS-SAME: (<2 x float> noundef nofpclass(inf) [[A_COERCE:%.*]], double noundef nofpclass(inf) [[B_COERCE0:%.*]], double noundef nofpclass(inf) [[B_COERCE1:%.*]], <2 x half> noundef nofpclass(inf) [[C_COERCE:%.*]]) #[[ATTR4:[0-9]+]] {
+// NOINFS-NEXT:  entry:
+// NOINFS-NEXT:    [[RETVAL:%.*]] = alloca { float, float }, align 4
+// NOINFS-NEXT:    [[A:%.*]] = alloca { float, float }, align 4
+// NOINFS-NEXT:    [[B:%.*]] = alloca { double, double }, align 8
+// NOINFS-NEXT:    [[C:%.*]] = alloca { half, half }, align 2
+// NOINFS-NEXT:    [[COERCE:%.*]] = alloca { float, float }, align 4
+// NOINFS-NEXT:    [[COERCE1:%.*]] = alloca { double, double }, align 8
+// NOINFS-NEXT:    [[COERCE2:%.*]] = alloca { half, half }, align 2
+// NOINFS-NEXT:    [[COERCE3:%.*]] = alloca { float, float }, align 4
+// NOINFS-NEXT:    store <2 x float> [[A_COERCE]], ptr [[A]], align 4
+// NOINFS-NEXT:    [[TMP0:%.*]] = getelementptr inbounds { double, double }, ptr [[B]], i32 0, i32 0
+// NOINFS-NEXT:    store double [[B_COERCE0]], ptr [[TMP0]], align 8
+// NOINFS-NEXT:    [[TMP1:%.*]] = getelementptr inbounds { double, double }, ptr [[B]], i32 0, i32 1
+// NOINFS-NEXT:    store double [[B_COERCE1]], ptr [[TMP1]], align 8
+// NOINFS-NEXT:    store <2 x half> [[C_COERCE]], ptr [[C]], align 2
+// NOINFS-NEXT:    [[A_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[A]], i32 0, i32 0
+// NOINFS-NEXT:    [[A_REAL:%.*]] = load float, ptr [[A_REALP]], align 4
+// NOINFS-NEXT:    [[A_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[A]], i32 0, i32 1
+// NOINFS-NEXT:    [[A_IMAG:%.*]] = load float, ptr [[A_IMAGP]], align 4
+// NOINFS-NEXT:    [[B_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[B]], i32 0, i32 0
+// NOINFS-NEXT:    [[B_REAL:%.*]] = load double, ptr [[B_REALP]], align 8
+// NOINFS-NEXT:    [[B_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[B]], i32 0, i32 1
+// NOINFS-NEXT:    [[B_IMAG:%.*]] = load double, ptr [[B_IMAGP]], align 8
+// NOINFS-NEXT:    [[C_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[C]], i32 0, i32 0
+// NOINFS-NEXT:    [[C_REAL:%.*]] = load half, ptr [[C_REALP]], align 2
+// NOINFS-NEXT:    [[C_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[C]], i32 0, i32 1
+// NOINFS-NEXT:    [[C_IMAG:%.*]] = load half, ptr [[C_IMAGP]], align 2
+// NOINFS-NEXT:    [[COERCE_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE]], i32 0, i32 0
+// NOINFS-NEXT:    [[COERCE_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE]], i32 0, i32 1
+// NOINFS-NEXT:    store float [[A_REAL]], ptr [[COERCE_REALP]], align 4
+// NOINFS-NEXT:    store float [[A_IMAG]], ptr [[COERCE_IMAGP]], align 4
+// NOINFS-NEXT:    [[TMP2:%.*]] = load <2 x float>, ptr [[COERCE]], align 4
+// NOINFS-NEXT:    [[COERCE1_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[COERCE1]], i32 0, i32 0
+// NOINFS-NEXT:    [[COERCE1_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[COERCE1]], i32 0, i32 1
+// NOINFS-NEXT:    store double [[B_REAL]], ptr [[COERCE1_REALP]], align 8
+// NOINFS-NEXT:    store double [[B_IMAG]], ptr [[COERCE1_IMAGP]], align 8
+// NOINFS-NEXT:    [[TMP3:%.*]] = getelementptr inbounds { double, double }, ptr [[COERCE1]], i32 0, i32 0
+// NOINFS-NEXT:    [[TMP4:%.*]] = load double, ptr [[TMP3]], align 8
+// NOINFS-NEXT:    [[TMP5:%.*]] = getelementptr inbounds { double, double }, ptr [[COERCE1]], i32 0, i32 1
+// NOINFS-NEXT:    [[TMP6:%.*]] = load double, ptr [[TMP5]], align 8
+// NOINFS-NEXT:    [[COERCE2_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[COERCE2]], i32 0, i32 0
+// NOINFS-NEXT:    [[COERCE2_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[COERCE2]], i32 0, i32 1
+// NOINFS-NEXT:    store half [[C_REAL]], ptr [[COERCE2_REALP]], align 2
+// NOINFS-NEXT:    store half [[C_IMAG]], ptr [[COERCE2_IMAGP]], align 2
+// NOINFS-NEXT:    [[TMP7:%.*]] = load <2 x half>, ptr [[COERCE2]], align 2
+// NOINFS-NEXT:    [[CALL:%.*]] = call ninf nofpclass(inf) <2 x float> @extern_complex(<2 x float> noundef nofpclass(inf) [[TMP2]], double noundef nofpclass(inf) [[TMP4]], double noundef nofpclass(inf) [[TMP6]], <2 x half> noundef nofpclass(inf) [[TMP7]])
+// NOINFS-NEXT:    store <2 x float> [[CALL]], ptr [[COERCE3]], align 4
+// NOINFS-NEXT:    [[COERCE3_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE3]], i32 0, i32 0
+// NOINFS-NEXT:    [[COERCE3_REAL:%.*]] = load float, ptr [[COERCE3_REALP]], align 4
+// NOINFS-NEXT:    [[COERCE3_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE3]], i32 0, i32 1
+// NOINFS-NEXT:    [[COERCE3_IMAG:%.*]] = load float, ptr [[COERCE3_IMAGP]], align 4
+// NOINFS-NEXT:    [[RETVAL_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[RETVAL]], i32 0, i32 0
+// NOINFS-NEXT:    [[RETVAL_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[RETVAL]], i32 0, i32 1
+// NOINFS-NEXT:    store float [[COERCE3_REAL]], ptr [[RETVAL_REALP]], align 4
+// NOINFS-NEXT:    store float [[COERCE3_IMAG]], ptr [[RETVAL_IMAGP]], align 4
+// NOINFS-NEXT:    [[TMP8:%.*]] = load <2 x float>, ptr [[RETVAL]], align 4
+// NOINFS-NEXT:    ret <2 x float> [[TMP8]]
+//
+_Complex float defined_complex_func(_Complex float a, _Complex double b, _Complex _Float16 c) {
+  return extern_complex(a, b, c);
+}
+
+// CFINITEONLY: Function Attrs: noinline nounwind optnone
+// CFINITEONLY-LABEL: define dso_local { double, double } @defined_complex_func_f64_ret
+// CFINITEONLY-SAME: (double noundef nofpclass(nan inf) [[C_COERCE0:%.*]], double noundef nofpclass(nan inf) [[C_COERCE1:%.*]]) #[[ATTR0]] {
+// CFINITEONLY-NEXT:  entry:
+// CFINITEONLY-NEXT:    [[RETVAL:%.*]] = alloca { double, double }, align 8
+// CFINITEONLY-NEXT:    [[C:%.*]] = alloca { double, double }, align 8
+// CFINITEONLY-NEXT:    [[TMP0:%.*]] = getelementptr inbounds { double, double }, ptr [[C]], i32 0, i32 0
+// CFINITEONLY-NEXT:    store double [[C_COERCE0]], ptr [[TMP0]], align 8
+// CFINITEONLY-NEXT:    [[TMP1:%.*]] = getelementptr inbounds { double, double }, ptr [[C]], i32 0, i32 1
+// CFINITEONLY-NEXT:    store double [[C_COERCE1]], ptr [[TMP1]], align 8
+// CFINITEONLY-NEXT:    [[C_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[C]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[C_REAL:%.*]] = load double, ptr [[C_REALP]], align 8
+// CFINITEONLY-NEXT:    [[C_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[C]], i32 0, i32 1
+// CFINITEONLY-NEXT:    [[C_IMAG:%.*]] = load double, ptr [[C_IMAGP]], align 8
+// CFINITEONLY-NEXT:    [[C_REALP1:%.*]] = getelementptr inbounds { double, double }, ptr [[C]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[C_REAL2:%.*]] = load double, ptr [[C_REALP1]], align 8
+// CFINITEONLY-NEXT:    [[C_IMAGP3:%.*]] = getelementptr inbounds { double, double }, ptr [[C]], i32 0, i32 1
+// CFINITEONLY-NEXT:    [[C_IMAG4:%.*]] = load double, ptr [[C_IMAGP3]], align 8
+// CFINITEONLY-NEXT:    [[MUL_AC:%.*]] = fmul nnan ninf double [[C_REAL]], [[C_REAL2]]
+// CFINITEONLY-NEXT:    [[MUL_BD:%.*]] = fmul nnan ninf double [[C_IMAG]], [[C_IMAG4]]
+// CFINITEONLY-NEXT:    [[MUL_AD:%.*]] = fmul nnan ninf double [[C_REAL]], [[C_IMAG4]]
+// CFINITEONLY-NEXT:    [[MUL_BC:%.*]] = fmul nnan ninf double [[C_IMAG]], [[C_REAL2]]
+// CFINITEONLY-NEXT:    [[MUL_R:%.*]] = fsub nnan ninf double [[MUL_AC]], [[MUL_BD]]
+// CFINITEONLY-NEXT:    [[MUL_I:%.*]] = fadd nnan ninf double [[MUL_AD]], [[MUL_BC]]
+// CFINITEONLY-NEXT:    [[ISNAN_CMP:%.*]] = fcmp nnan ninf uno double [[MUL_R]], [[MUL_R]]
+// CFINITEONLY-NEXT:    br i1 [[ISNAN_CMP]], label [[COMPLEX_MUL_IMAG_NAN:%.*]], label [[COMPLEX_MUL_CONT:%.*]], !prof [[PROF2:![0-9]+]]
+// CFINITEONLY:       complex_mul_imag_nan:
+// CFINITEONLY-NEXT:    [[ISNAN_CMP5:%.*]] = fcmp nnan ninf uno double [[MUL_I]], [[MUL_I]]
+// CFINITEONLY-NEXT:    br i1 [[ISNAN_CMP5]], label [[COMPLEX_MUL_LIBCALL:%.*]], label [[COMPLEX_MUL_CONT]], !prof [[PROF2]]
+// CFINITEONLY:       complex_mul_libcall:
+// CFINITEONLY-NEXT:    [[CALL:%.*]] = call { double, double } @__muldc3(double noundef nofpclass(nan inf) [[C_REAL]], double noundef nofpclass(nan inf) [[C_IMAG]], double noundef nofpclass(nan inf) [[C_REAL2]], double noundef nofpclass(nan inf) [[C_IMAG4]]) #[[ATTR7:[0-9]+]]
+// CFINITEONLY-NEXT:    [[TMP2:%.*]] = extractvalue { double, double } [[CALL]], 0
+// CFINITEONLY-NEXT:    [[TMP3:%.*]] = extractvalue { double, double } [[CALL]], 1
+// CFINITEONLY-NEXT:    br label [[COMPLEX_MUL_CONT]]
+// CFINITEONLY:       complex_mul_cont:
+// CFINITEONLY-NEXT:    [[REAL_MUL_PHI:%.*]] = phi nnan ninf double [ [[MUL_R]], [[ENTRY:%.*]] ], [ [[MUL_R]], [[COMPLEX_MUL_IMAG_NAN]] ], [ [[TMP2]], [[COMPLEX_MUL_LIBCALL]] ]
+// CFINITEONLY-NEXT:    [[IMAG_MUL_PHI:%.*]] = phi nnan ninf double [ [[MUL_I]], [[ENTRY]] ], [ [[MUL_I]], [[COMPLEX_MUL_IMAG_NAN]] ], [ [[TMP3]], [[COMPLEX_MUL_LIBCALL]] ]
+// CFINITEONLY-NEXT:    [[RETVAL_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[RETVAL]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[RETVAL_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[RETVAL]], i32 0, i32 1
+// CFINITEONLY-NEXT:    store double [[REAL_MUL_PHI]], ptr [[RETVAL_REALP]], align 8
+// CFINITEONLY-NEXT:    store double [[IMAG_MUL_PHI]], ptr [[RETVAL_IMAGP]], align 8
+// CFINITEONLY-NEXT:    [[TMP4:%.*]] = load { double, double }, ptr [[RETVAL]], align 8
+// CFINITEONLY-NEXT:    ret { double, double } [[TMP4]]
+//
+// CLFINITEONLY: Function Attrs: mustprogress nofree norecurse nosync nounwind willreturn memory(none)
+// CLFINITEONLY-LABEL: define dso_local { double, double } @defined_complex_func_f64_ret
+// CLFINITEONLY-SAME: (double noundef nofpclass(nan inf) [[C_COERCE0:%.*]], double noundef nofpclass(nan inf) [[C_COERCE1:%.*]]) local_unnamed_addr #[[ATTR0]] {
+// CLFINITEONLY-NEXT:  entry:
+// CLFINITEONLY-NEXT:    [[MUL_AD:%.*]] = fmul nnan ninf double [[C_COERCE0]], [[C_COERCE1]]
+// CLFINITEONLY-NEXT:    [[MUL_I:%.*]] = fadd nnan ninf double [[MUL_AD]], [[MUL_AD]]
+// CLFINITEONLY-NEXT:    [[MUL_AC:%.*]] = fmul nnan ninf double [[C_COERCE0]], [[C_COERCE0]]
+// CLFINITEONLY-NEXT:    [[MUL_BD:%.*]] = fmul nnan ninf double [[C_COERCE1]], [[C_COERCE1]]
+// CLFINITEONLY-NEXT:    [[MUL_R:%.*]] = fsub nnan ninf double [[MUL_AC]], [[MUL_BD]]
+// CLFINITEONLY-NEXT:    [[DOTFCA_0_INSERT:%.*]] = insertvalue { double, double } poison, double [[MUL_R]], 0
+// CLFINITEONLY-NEXT:    [[DOTFCA_1_INSERT:%.*]] = insertvalue { double, double } [[DOTFCA_0_INSERT]], double [[MUL_I]], 1
+// CLFINITEONLY-NEXT:    ret { double, double } [[DOTFCA_1_INSERT]]
+//
+// NONANS: Function Attrs: noinline nounwind optnone
+// NONANS-LABEL: define dso_local { double, double } @defined_complex_func_f64_ret
+// NONANS-SAME: (double noundef nofpclass(nan) [[C_COERCE0:%.*]], double noundef nofpclass(nan) [[C_COERCE1:%.*]]) #[[ATTR0]] {
+// NONANS-NEXT:  entry:
+// NONANS-NEXT:    [[RETVAL:%.*]] = alloca { double, double }, align 8
+// NONANS-NEXT:    [[C:%.*]] = alloca { double, double }, align 8
+// NONANS-NEXT:    [[TMP0:%.*]] = getelementptr inbounds { double, double }, ptr [[C]], i32 0, i32 0
+// NONANS-NEXT:    store double [[C_COERCE0]], ptr [[TMP0]], align 8
+// NONANS-NEXT:    [[TMP1:%.*]] = getelementptr inbounds { double, double }, ptr [[C]], i32 0, i32 1
+// NONANS-NEXT:    store double [[C_COERCE1]], ptr [[TMP1]], align 8
+// NONANS-NEXT:    [[C_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[C]], i32 0, i32 0
+// NONANS-NEXT:    [[C_REAL:%.*]] = load double, ptr [[C_REALP]], align 8
+// NONANS-NEXT:    [[C_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[C]], i32 0, i32 1
+// NONANS-NEXT:    [[C_IMAG:%.*]] = load double, ptr [[C_IMAGP]], align 8
+// NONANS-NEXT:    [[C_REALP1:%.*]] = getelementptr inbounds { double, double }, ptr [[C]], i32 0, i32 0
+// NONANS-NEXT:    [[C_REAL2:%.*]] = load double, ptr [[C_REALP1]], align 8
+// NONANS-NEXT:    [[C_IMAGP3:%.*]] = getelementptr inbounds { double, double }, ptr [[C]], i32 0, i32 1
+// NONANS-NEXT:    [[C_IMAG4:%.*]] = load double, ptr [[C_IMAGP3]], align 8
+// NONANS-NEXT:    [[MUL_AC:%.*]] = fmul nnan double [[C_REAL]], [[C_REAL2]]
+// NONANS-NEXT:    [[MUL_BD:%.*]] = fmul nnan double [[C_IMAG]], [[C_IMAG4]]
+// NONANS-NEXT:    [[MUL_AD:%.*]] = fmul nnan double [[C_REAL]], [[C_IMAG4]]
+// NONANS-NEXT:    [[MUL_BC:%.*]] = fmul nnan double [[C_IMAG]], [[C_REAL2]]
+// NONANS-NEXT:    [[MUL_R:%.*]] = fsub nnan double [[MUL_AC]], [[MUL_BD]]
+// NONANS-NEXT:    [[MUL_I:%.*]] = fadd nnan double [[MUL_AD]], [[MUL_BC]]
+// NONANS-NEXT:    [[ISNAN_CMP:%.*]] = fcmp nnan uno double [[MUL_R]], [[MUL_R]]
+// NONANS-NEXT:    br i1 [[ISNAN_CMP]], label [[COMPLEX_MUL_IMAG_NAN:%.*]], label [[COMPLEX_MUL_CONT:%.*]], !prof [[PROF2:![0-9]+]]
+// NONANS:       complex_mul_imag_nan:
+// NONANS-NEXT:    [[ISNAN_CMP5:%.*]] = fcmp nnan uno double [[MUL_I]], [[MUL_I]]
+// NONANS-NEXT:    br i1 [[ISNAN_CMP5]], label [[COMPLEX_MUL_LIBCALL:%.*]], label [[COMPLEX_MUL_CONT]], !prof [[PROF2]]
+// NONANS:       complex_mul_libcall:
+// NONANS-NEXT:    [[CALL:%.*]] = call { double, double } @__muldc3(double noundef nofpclass(nan) [[C_REAL]], double noundef nofpclass(nan) [[C_IMAG]], double noundef nofpclass(nan) [[C_REAL2]], double noundef nofpclass(nan) [[C_IMAG4]]) #[[ATTR7:[0-9]+]]
+// NONANS-NEXT:    [[TMP2:%.*]] = extractvalue { double, double } [[CALL]], 0
+// NONANS-NEXT:    [[TMP3:%.*]] = extractvalue { double, double } [[CALL]], 1
+// NONANS-NEXT:    br label [[COMPLEX_MUL_CONT]]
+// NONANS:       complex_mul_cont:
+// NONANS-NEXT:    [[REAL_MUL_PHI:%.*]] = phi nnan double [ [[MUL_R]], [[ENTRY:%.*]] ], [ [[MUL_R]], [[COMPLEX_MUL_IMAG_NAN]] ], [ [[TMP2]], [[COMPLEX_MUL_LIBCALL]] ]
+// NONANS-NEXT:    [[IMAG_MUL_PHI:%.*]] = phi nnan double [ [[MUL_I]], [[ENTRY]] ], [ [[MUL_I]], [[COMPLEX_MUL_IMAG_NAN]] ], [ [[TMP3]], [[COMPLEX_MUL_LIBCALL]] ]
+// NONANS-NEXT:    [[RETVAL_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[RETVAL]], i32 0, i32 0
+// NONANS-NEXT:    [[RETVAL_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[RETVAL]], i32 0, i32 1
+// NONANS-NEXT:    store double [[REAL_MUL_PHI]], ptr [[RETVAL_REALP]], align 8
+// NONANS-NEXT:    store double [[IMAG_MUL_PHI]], ptr [[RETVAL_IMAGP]], align 8
+// NONANS-NEXT:    [[TMP4:%.*]] = load { double, double }, ptr [[RETVAL]], align 8
+// NONANS-NEXT:    ret { double, double } [[TMP4]]
+//
+// NOINFS: Function Attrs: noinline nounwind optnone
+// NOINFS-LABEL: define dso_local { double, double } @defined_complex_func_f64_ret
+// NOINFS-SAME: (double noundef nofpclass(inf) [[C_COERCE0:%.*]], double noundef nofpclass(inf) [[C_COERCE1:%.*]]) #[[ATTR0]] {
+// NOINFS-NEXT:  entry:
+// NOINFS-NEXT:    [[RETVAL:%.*]] = alloca { double, double }, align 8
+// NOINFS-NEXT:    [[C:%.*]] = alloca { double, double }, align 8
+// NOINFS-NEXT:    [[TMP0:%.*]] = getelementptr inbounds { double, double }, ptr [[C]], i32 0, i32 0
+// NOINFS-NEXT:    store double [[C_COERCE0]], ptr [[TMP0]], align 8
+// NOINFS-NEXT:    [[TMP1:%.*]] = getelementptr inbounds { double, double }, ptr [[C]], i32 0, i32 1
+// NOINFS-NEXT:    store double [[C_COERCE1]], ptr [[TMP1]], align 8
+// NOINFS-NEXT:    [[C_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[C]], i32 0, i32 0
+// NOINFS-NEXT:    [[C_REAL:%.*]] = load double, ptr [[C_REALP]], align 8
+// NOINFS-NEXT:    [[C_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[C]], i32 0, i32 1
+// NOINFS-NEXT:    [[C_IMAG:%.*]] = load double, ptr [[C_IMAGP]], align 8
+// NOINFS-NEXT:    [[C_REALP1:%.*]] = getelementptr inbounds { double, double }, ptr [[C]], i32 0, i32 0
+// NOINFS-NEXT:    [[C_REAL2:%.*]] = load double, ptr [[C_REALP1]], align 8
+// NOINFS-NEXT:    [[C_IMAGP3:%.*]] = getelementptr inbounds { double, double }, ptr [[C]], i32 0, i32 1
+// NOINFS-NEXT:    [[C_IMAG4:%.*]] = load double, ptr [[C_IMAGP3]], align 8
+// NOINFS-NEXT:    [[MUL_AC:%.*]] = fmul ninf double [[C_REAL]], [[C_REAL2]]
+// NOINFS-NEXT:    [[MUL_BD:%.*]] = fmul ninf double [[C_IMAG]], [[C_IMAG4]]
+// NOINFS-NEXT:    [[MUL_AD:%.*]] = fmul ninf double [[C_REAL]], [[C_IMAG4]]
+// NOINFS-NEXT:    [[MUL_BC:%.*]] = fmul ninf double [[C_IMAG]], [[C_REAL2]]
+// NOINFS-NEXT:    [[MUL_R:%.*]] = fsub ninf double [[MUL_AC]], [[MUL_BD]]
+// NOINFS-NEXT:    [[MUL_I:%.*]] = fadd ninf double [[MUL_AD]], [[MUL_BC]]
+// NOINFS-NEXT:    [[ISNAN_CMP:%.*]] = fcmp ninf uno double [[MUL_R]], [[MUL_R]]
+// NOINFS-NEXT:    br i1 [[ISNAN_CMP]], label [[COMPLEX_MUL_IMAG_NAN:%.*]], label [[COMPLEX_MUL_CONT:%.*]], !prof [[PROF2:![0-9]+]]
+// NOINFS:       complex_mul_imag_nan:
+// NOINFS-NEXT:    [[ISNAN_CMP5:%.*]] = fcmp ninf uno double [[MUL_I]], [[MUL_I]]
+// NOINFS-NEXT:    br i1 [[ISNAN_CMP5]], label [[COMPLEX_MUL_LIBCALL:%.*]], label [[COMPLEX_MUL_CONT]], !prof [[PROF2]]
+// NOINFS:       complex_mul_libcall:
+// NOINFS-NEXT:    [[CALL:%.*]] = call { double, double } @__muldc3(double noundef nofpclass(inf) [[C_REAL]], double noundef nofpclass(inf) [[C_IMAG]], double noundef nofpclass(inf) [[C_REAL2]], double noundef nofpclass(inf) [[C_IMAG4]]) #[[ATTR7:[0-9]+]]
+// NOINFS-NEXT:    [[TMP2:%.*]] = extractvalue { double, double } [[CALL]], 0
+// NOINFS-NEXT:    [[TMP3:%.*]] = extractvalue { double, double } [[CALL]], 1
+// NOINFS-NEXT:    br label [[COMPLEX_MUL_CONT]]
+// NOINFS:       complex_mul_cont:
+// NOINFS-NEXT:    [[REAL_MUL_PHI:%.*]] = phi ninf double [ [[MUL_R]], [[ENTRY:%.*]] ], [ [[MUL_R]], [[COMPLEX_MUL_IMAG_NAN]] ], [ [[TMP2]], [[COMPLEX_MUL_LIBCALL]] ]
+// NOINFS-NEXT:    [[IMAG_MUL_PHI:%.*]] = phi ninf double [ [[MUL_I]], [[ENTRY]] ], [ [[MUL_I]], [[COMPLEX_MUL_IMAG_NAN]] ], [ [[TMP3]], [[COMPLEX_MUL_LIBCALL]] ]
+// NOINFS-NEXT:    [[RETVAL_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[RETVAL]], i32 0, i32 0
+// NOINFS-NEXT:    [[RETVAL_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[RETVAL]], i32 0, i32 1
+// NOINFS-NEXT:    store double [[REAL_MUL_PHI]], ptr [[RETVAL_REALP]], align 8
+// NOINFS-NEXT:    store double [[IMAG_MUL_PHI]], ptr [[RETVAL_IMAGP]], align 8
+// NOINFS-NEXT:    [[TMP4:%.*]] = load { double, double }, ptr [[RETVAL]], align 8
+// NOINFS-NEXT:    ret { double, double } [[TMP4]]
+//
+_Complex double defined_complex_func_f64_ret(_Complex double c) {
+  return c * c;
+}
+
+// CFINITEONLY: Function Attrs: noinline nounwind optnone
+// CFINITEONLY-LABEL: define dso_local nofpclass(nan inf) <2 x half> @defined_complex_func_f16_ret
+// CFINITEONLY-SAME: (<2 x half> noundef nofpclass(nan inf) [[C_COERCE:%.*]]) #[[ATTR4]] {
+// CFINITEONLY-NEXT:  entry:
+// CFINITEONLY-NEXT:    [[RETVAL:%.*]] = alloca { half, half }, align 2
+// CFINITEONLY-NEXT:    [[C:%.*]] = alloca { half, half }, align 2
+// CFINITEONLY-NEXT:    [[COERCE:%.*]] = alloca { float, float }, align 4
+// CFINITEONLY-NEXT:    store <2 x half> [[C_COERCE]], ptr [[C]], align 2
+// CFINITEONLY-NEXT:    [[C_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[C]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[C_REAL:%.*]] = load half, ptr [[C_REALP]], align 2
+// CFINITEONLY-NEXT:    [[C_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[C]], i32 0, i32 1
+// CFINITEONLY-NEXT:    [[C_IMAG:%.*]] = load half, ptr [[C_IMAGP]], align 2
+// CFINITEONLY-NEXT:    [[EXT:%.*]] = fpext half [[C_REAL]] to float
+// CFINITEONLY-NEXT:    [[EXT1:%.*]] = fpext half [[C_IMAG]] to float
+// CFINITEONLY-NEXT:    [[C_REALP2:%.*]] = getelementptr inbounds { half, half }, ptr [[C]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[C_REAL3:%.*]] = load half, ptr [[C_REALP2]], align 2
+// CFINITEONLY-NEXT:    [[C_IMAGP4:%.*]] = getelementptr inbounds { half, half }, ptr [[C]], i32 0, i32 1
+// CFINITEONLY-NEXT:    [[C_IMAG5:%.*]] = load half, ptr [[C_IMAGP4]], align 2
+// CFINITEONLY-NEXT:    [[EXT6:%.*]] = fpext half [[C_REAL3]] to float
+// CFINITEONLY-NEXT:    [[EXT7:%.*]] = fpext half [[C_IMAG5]] to float
+// CFINITEONLY-NEXT:    [[MUL_AC:%.*]] = fmul nnan ninf float [[EXT]], [[EXT6]]
+// CFINITEONLY-NEXT:    [[MUL_BD:%.*]] = fmul nnan ninf float [[EXT1]], [[EXT7]]
+// CFINITEONLY-NEXT:    [[MUL_AD:%.*]] = fmul nnan ninf float [[EXT]], [[EXT7]]
+// CFINITEONLY-NEXT:    [[MUL_BC:%.*]] = fmul nnan ninf float [[EXT1]], [[EXT6]]
+// CFINITEONLY-NEXT:    [[MUL_R:%.*]] = fsub nnan ninf float [[MUL_AC]], [[MUL_BD]]
+// CFINITEONLY-NEXT:    [[MUL_I:%.*]] = fadd nnan ninf float [[MUL_AD]], [[MUL_BC]]
+// CFINITEONLY-NEXT:    [[ISNAN_CMP:%.*]] = fcmp nnan ninf uno float [[MUL_R]], [[MUL_R]]
+// CFINITEONLY-NEXT:    br i1 [[ISNAN_CMP]], label [[COMPLEX_MUL_IMAG_NAN:%.*]], label [[COMPLEX_MUL_CONT:%.*]], !prof [[PROF2]]
+// CFINITEONLY:       complex_mul_imag_nan:
+// CFINITEONLY-NEXT:    [[ISNAN_CMP8:%.*]] = fcmp nnan ninf uno float [[MUL_I]], [[MUL_I]]
+// CFINITEONLY-NEXT:    br i1 [[ISNAN_CMP8]], label [[COMPLEX_MUL_LIBCALL:%.*]], label [[COMPLEX_MUL_CONT]], !prof [[PROF2]]
+// CFINITEONLY:       complex_mul_libcall:
+// CFINITEONLY-NEXT:    [[CALL:%.*]] = call nnan ninf nofpclass(nan inf) <2 x float> @__mulsc3(float noundef nofpclass(nan inf) [[EXT]], float noundef nofpclass(nan inf) [[EXT1]], float noundef nofpclass(nan inf) [[EXT6]], float noundef nofpclass(nan inf) [[EXT7]]) #[[ATTR7]]
+// CFINITEONLY-NEXT:    store <2 x float> [[CALL]], ptr [[COERCE]], align 4
+// CFINITEONLY-NEXT:    [[COERCE_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[COERCE_REAL:%.*]] = load float, ptr [[COERCE_REALP]], align 4
+// CFINITEONLY-NEXT:    [[COERCE_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE]], i32 0, i32 1
+// CFINITEONLY-NEXT:    [[COERCE_IMAG:%.*]] = load float, ptr [[COERCE_IMAGP]], align 4
+// CFINITEONLY-NEXT:    br label [[COMPLEX_MUL_CONT]]
+// CFINITEONLY:       complex_mul_cont:
+// CFINITEONLY-NEXT:    [[REAL_MUL_PHI:%.*]] = phi nnan ninf float [ [[MUL_R]], [[ENTRY:%.*]] ], [ [[MUL_R]], [[COMPLEX_MUL_IMAG_NAN]] ], [ [[COERCE_REAL]], [[COMPLEX_MUL_LIBCALL]] ]
+// CFINITEONLY-NEXT:    [[IMAG_MUL_PHI:%.*]] = phi nnan ninf float [ [[MUL_I]], [[ENTRY]] ], [ [[MUL_I]], [[COMPLEX_MUL_IMAG_NAN]] ], [ [[COERCE_IMAG]], [[COMPLEX_MUL_LIBCALL]] ]
+// CFINITEONLY-NEXT:    [[UNPROMOTION:%.*]] = fptrunc float [[REAL_MUL_PHI]] to half
+// CFINITEONLY-NEXT:    [[UNPROMOTION9:%.*]] = fptrunc float [[IMAG_MUL_PHI]] to half
+// CFINITEONLY-NEXT:    [[RETVAL_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[RETVAL]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[RETVAL_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[RETVAL]], i32 0, i32 1
+// CFINITEONLY-NEXT:    store half [[UNPROMOTION]], ptr [[RETVAL_REALP]], align 2
+// CFINITEONLY-NEXT:    store half [[UNPROMOTION9]], ptr [[RETVAL_IMAGP]], align 2
+// CFINITEONLY-NEXT:    [[TMP0:%.*]] = load <2 x half>, ptr [[RETVAL]], align 2
+// CFINITEONLY-NEXT:    ret <2 x half> [[TMP0]]
+//
+// CLFINITEONLY: Function Attrs: mustprogress nofree norecurse nosync nounwind willreturn memory(none)
+// CLFINITEONLY-LABEL: define dso_local nofpclass(nan inf) <2 x half> @defined_complex_func_f16_ret
+// CLFINITEONLY-SAME: (<2 x half> noundef nofpclass(nan inf) [[C_COERCE:%.*]]) local_unnamed_addr #[[ATTR7:[0-9]+]] {
+// CLFINITEONLY-NEXT:  entry:
+// CLFINITEONLY-NEXT:    [[C_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <2 x half> [[C_COERCE]], i64 0
+// CLFINITEONLY-NEXT:    [[EXT:%.*]] = fpext half [[C_SROA_0_0_VEC_EXTRACT]] to float
+// CLFINITEONLY-NEXT:    [[C_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <2 x half> [[C_COERCE]], i64 1
+// CLFINITEONLY-NEXT:    [[EXT1:%.*]] = fpext half [[C_SROA_0_2_VEC_EXTRACT]] to float
+// CLFINITEONLY-NEXT:    [[MUL_AD:%.*]] = fmul nnan ninf float [[EXT]], [[EXT1]]
+// CLFINITEONLY-NEXT:    [[MUL_I:%.*]] = fadd nnan ninf float [[MUL_AD]], [[MUL_AD]]
+// CLFINITEONLY-NEXT:    [[MUL_AC:%.*]] = fmul nnan ninf float [[EXT]], [[EXT]]
+// CLFINITEONLY-NEXT:    [[MUL_BD:%.*]] = fmul nnan ninf float [[EXT1]], [[EXT1]]
+// CLFINITEONLY-NEXT:    [[MUL_R:%.*]] = fsub nnan ninf float [[MUL_AC]], [[MUL_BD]]
+// CLFINITEONLY-NEXT:    [[UNPROMOTION:%.*]] = fptrunc float [[MUL_R]] to half
+// CLFINITEONLY-NEXT:    [[UNPROMOTION9:%.*]] = fptrunc float [[MUL_I]] to half
+// CLFINITEONLY-NEXT:    [[RETVAL_SROA_0_0_VEC_INSERT:%.*]] = insertelement <2 x half> undef, half [[UNPROMOTION]], i64 0
+// CLFINITEONLY-NEXT:    [[RETVAL_SROA_0_2_VEC_INSERT:%.*]] = insertelement <2 x half> [[RETVAL_SROA_0_0_VEC_INSERT]], half [[UNPROMOTION9]], i64 1
+// CLFINITEONLY-NEXT:    ret <2 x half> [[RETVAL_SROA_0_2_VEC_INSERT]]
+//
+// NONANS: Function Attrs: noinline nounwind optnone
+// NONANS-LABEL: define dso_local nofpclass(nan) <2 x half> @defined_complex_func_f16_ret
+// NONANS-SAME: (<2 x half> noundef nofpclass(nan) [[C_COERCE:%.*]]) #[[ATTR4]] {
+// NONANS-NEXT:  entry:
+// NONANS-NEXT:    [[RETVAL:%.*]] = alloca { half, half }, align 2
+// NONANS-NEXT:    [[C:%.*]] = alloca { half, half }, align 2
+// NONANS-NEXT:    [[COERCE:%.*]] = alloca { float, float }, align 4
+// NONANS-NEXT:    store <2 x half> [[C_COERCE]], ptr [[C]], align 2
+// NONANS-NEXT:    [[C_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[C]], i32 0, i32 0
+// NONANS-NEXT:    [[C_REAL:%.*]] = load half, ptr [[C_REALP]], align 2
+// NONANS-NEXT:    [[C_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[C]], i32 0, i32 1
+// NONANS-NEXT:    [[C_IMAG:%.*]] = load half, ptr [[C_IMAGP]], align 2
+// NONANS-NEXT:    [[EXT:%.*]] = fpext half [[C_REAL]] to float
+// NONANS-NEXT:    [[EXT1:%.*]] = fpext half [[C_IMAG]] to float
+// NONANS-NEXT:    [[C_REALP2:%.*]] = getelementptr inbounds { half, half }, ptr [[C]], i32 0, i32 0
+// NONANS-NEXT:    [[C_REAL3:%.*]] = load half, ptr [[C_REALP2]], align 2
+// NONANS-NEXT:    [[C_IMAGP4:%.*]] = getelementptr inbounds { half, half }, ptr [[C]], i32 0, i32 1
+// NONANS-NEXT:    [[C_IMAG5:%.*]] = load half, ptr [[C_IMAGP4]], align 2
+// NONANS-NEXT:    [[EXT6:%.*]] = fpext half [[C_REAL3]] to float
+// NONANS-NEXT:    [[EXT7:%.*]] = fpext half [[C_IMAG5]] to float
+// NONANS-NEXT:    [[MUL_AC:%.*]] = fmul nnan float [[EXT]], [[EXT6]]
+// NONANS-NEXT:    [[MUL_BD:%.*]] = fmul nnan float [[EXT1]], [[EXT7]]
+// NONANS-NEXT:    [[MUL_AD:%.*]] = fmul nnan float [[EXT]], [[EXT7]]
+// NONANS-NEXT:    [[MUL_BC:%.*]] = fmul nnan float [[EXT1]], [[EXT6]]
+// NONANS-NEXT:    [[MUL_R:%.*]] = fsub nnan float [[MUL_AC]], [[MUL_BD]]
+// NONANS-NEXT:    [[MUL_I:%.*]] = fadd nnan float [[MUL_AD]], [[MUL_BC]]
+// NONANS-NEXT:    [[ISNAN_CMP:%.*]] = fcmp nnan uno float [[MUL_R]], [[MUL_R]]
+// NONANS-NEXT:    br i1 [[ISNAN_CMP]], label [[COMPLEX_MUL_IMAG_NAN:%.*]], label [[COMPLEX_MUL_CONT:%.*]], !prof [[PROF2]]
+// NONANS:       complex_mul_imag_nan:
+// NONANS-NEXT:    [[ISNAN_CMP8:%.*]] = fcmp nnan uno float [[MUL_I]], [[MUL_I]]
+// NONANS-NEXT:    br i1 [[ISNAN_CMP8]], label [[COMPLEX_MUL_LIBCALL:%.*]], label [[COMPLEX_MUL_CONT]], !prof [[PROF2]]
+// NONANS:       complex_mul_libcall:
+// NONANS-NEXT:    [[CALL:%.*]] = call nnan nofpclass(nan) <2 x float> @__mulsc3(float noundef nofpclass(nan) [[EXT]], float noundef nofpclass(nan) [[EXT1]], float noundef nofpclass(nan) [[EXT6]], float noundef nofpclass(nan) [[EXT7]]) #[[ATTR7]]
+// NONANS-NEXT:    store <2 x float> [[CALL]], ptr [[COERCE]], align 4
+// NONANS-NEXT:    [[COERCE_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE]], i32 0, i32 0
+// NONANS-NEXT:    [[COERCE_REAL:%.*]] = load float, ptr [[COERCE_REALP]], align 4
+// NONANS-NEXT:    [[COERCE_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE]], i32 0, i32 1
+// NONANS-NEXT:    [[COERCE_IMAG:%.*]] = load float, ptr [[COERCE_IMAGP]], align 4
+// NONANS-NEXT:    br label [[COMPLEX_MUL_CONT]]
+// NONANS:       complex_mul_cont:
+// NONANS-NEXT:    [[REAL_MUL_PHI:%.*]] = phi nnan float [ [[MUL_R]], [[ENTRY:%.*]] ], [ [[MUL_R]], [[COMPLEX_MUL_IMAG_NAN]] ], [ [[COERCE_REAL]], [[COMPLEX_MUL_LIBCALL]] ]
+// NONANS-NEXT:    [[IMAG_MUL_PHI:%.*]] = phi nnan float [ [[MUL_I]], [[ENTRY]] ], [ [[MUL_I]], [[COMPLEX_MUL_IMAG_NAN]] ], [ [[COERCE_IMAG]], [[COMPLEX_MUL_LIBCALL]] ]
+// NONANS-NEXT:    [[UNPROMOTION:%.*]] = fptrunc float [[REAL_MUL_PHI]] to half
+// NONANS-NEXT:    [[UNPROMOTION9:%.*]] = fptrunc float [[IMAG_MUL_PHI]] to half
+// NONANS-NEXT:    [[RETVAL_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[RETVAL]], i32 0, i32 0
+// NONANS-NEXT:    [[RETVAL_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[RETVAL]], i32 0, i32 1
+// NONANS-NEXT:    store half [[UNPROMOTION]], ptr [[RETVAL_REALP]], align 2
+// NONANS-NEXT:    store half [[UNPROMOTION9]], ptr [[RETVAL_IMAGP]], align 2
+// NONANS-NEXT:    [[TMP0:%.*]] = load <2 x half>, ptr [[RETVAL]], align 2
+// NONANS-NEXT:    ret <2 x half> [[TMP0]]
+//
+// NOINFS: Function Attrs: noinline nounwind optnone
+// NOINFS-LABEL: define dso_local nofpclass(inf) <2 x half> @defined_complex_func_f16_ret
+// NOINFS-SAME: (<2 x half> noundef nofpclass(inf) [[C_COERCE:%.*]]) #[[ATTR4]] {
+// NOINFS-NEXT:  entry:
+// NOINFS-NEXT:    [[RETVAL:%.*]] = alloca { half, half }, align 2
+// NOINFS-NEXT:    [[C:%.*]] = alloca { half, half }, align 2
+// NOINFS-NEXT:    [[COERCE:%.*]] = alloca { float, float }, align 4
+// NOINFS-NEXT:    store <2 x half> [[C_COERCE]], ptr [[C]], align 2
+// NOINFS-NEXT:    [[C_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[C]], i32 0, i32 0
+// NOINFS-NEXT:    [[C_REAL:%.*]] = load half, ptr [[C_REALP]], align 2
+// NOINFS-NEXT:    [[C_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[C]], i32 0, i32 1
+// NOINFS-NEXT:    [[C_IMAG:%.*]] = load half, ptr [[C_IMAGP]], align 2
+// NOINFS-NEXT:    [[EXT:%.*]] = fpext half [[C_REAL]] to float
+// NOINFS-NEXT:    [[EXT1:%.*]] = fpext half [[C_IMAG]] to float
+// NOINFS-NEXT:    [[C_REALP2:%.*]] = getelementptr inbounds { half, half }, ptr [[C]], i32 0, i32 0
+// NOINFS-NEXT:    [[C_REAL3:%.*]] = load half, ptr [[C_REALP2]], align 2
+// NOINFS-NEXT:    [[C_IMAGP4:%.*]] = getelementptr inbounds { half, half }, ptr [[C]], i32 0, i32 1
+// NOINFS-NEXT:    [[C_IMAG5:%.*]] = load half, ptr [[C_IMAGP4]], align 2
+// NOINFS-NEXT:    [[EXT6:%.*]] = fpext half [[C_REAL3]] to float
+// NOINFS-NEXT:    [[EXT7:%.*]] = fpext half [[C_IMAG5]] to float
+// NOINFS-NEXT:    [[MUL_AC:%.*]] = fmul ninf float [[EXT]], [[EXT6]]
+// NOINFS-NEXT:    [[MUL_BD:%.*]] = fmul ninf float [[EXT1]], [[EXT7]]
+// NOINFS-NEXT:    [[MUL_AD:%.*]] = fmul ninf float [[EXT]], [[EXT7]]
+// NOINFS-NEXT:    [[MUL_BC:%.*]] = fmul ninf float [[EXT1]], [[EXT6]]
+// NOINFS-NEXT:    [[MUL_R:%.*]] = fsub ninf float [[MUL_AC]], [[MUL_BD]]
+// NOINFS-NEXT:    [[MUL_I:%.*]] = fadd ninf float [[MUL_AD]], [[MUL_BC]]
+// NOINFS-NEXT:    [[ISNAN_CMP:%.*]] = fcmp ninf uno float [[MUL_R]], [[MUL_R]]
+// NOINFS-NEXT:    br i1 [[ISNAN_CMP]], label [[COMPLEX_MUL_IMAG_NAN:%.*]], label [[COMPLEX_MUL_CONT:%.*]], !prof [[PROF2]]
+// NOINFS:       complex_mul_imag_nan:
+// NOINFS-NEXT:    [[ISNAN_CMP8:%.*]] = fcmp ninf uno float [[MUL_I]], [[MUL_I]]
+// NOINFS-NEXT:    br i1 [[ISNAN_CMP8]], label [[COMPLEX_MUL_LIBCALL:%.*]], label [[COMPLEX_MUL_CONT]], !prof [[PROF2]]
+// NOINFS:       complex_mul_libcall:
+// NOINFS-NEXT:    [[CALL:%.*]] = call ninf nofpclass(inf) <2 x float> @__mulsc3(float noundef nofpclass(inf) [[EXT]], float noundef nofpclass(inf) [[EXT1]], float noundef nofpclass(inf) [[EXT6]], float noundef nofpclass(inf) [[EXT7]]) #[[ATTR7]]
+// NOINFS-NEXT:    store <2 x float> [[CALL]], ptr [[COERCE]], align 4
+// NOINFS-NEXT:    [[COERCE_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE]], i32 0, i32 0
+// NOINFS-NEXT:    [[COERCE_REAL:%.*]] = load float, ptr [[COERCE_REALP]], align 4
+// NOINFS-NEXT:    [[COERCE_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE]], i32 0, i32 1
+// NOINFS-NEXT:    [[COERCE_IMAG:%.*]] = load float, ptr [[COERCE_IMAGP]], align 4
+// NOINFS-NEXT:    br label [[COMPLEX_MUL_CONT]]
+// NOINFS:       complex_mul_cont:
+// NOINFS-NEXT:    [[REAL_MUL_PHI:%.*]] = phi ninf float [ [[MUL_R]], [[ENTRY:%.*]] ], [ [[MUL_R]], [[COMPLEX_MUL_IMAG_NAN]] ], [ [[COERCE_REAL]], [[COMPLEX_MUL_LIBCALL]] ]
+// NOINFS-NEXT:    [[IMAG_MUL_PHI:%.*]] = phi ninf float [ [[MUL_I]], [[ENTRY]] ], [ [[MUL_I]], [[COMPLEX_MUL_IMAG_NAN]] ], [ [[COERCE_IMAG]], [[COMPLEX_MUL_LIBCALL]] ]
+// NOINFS-NEXT:    [[UNPROMOTION:%.*]] = fptrunc float [[REAL_MUL_PHI]] to half
+// NOINFS-NEXT:    [[UNPROMOTION9:%.*]] = fptrunc float [[IMAG_MUL_PHI]] to half
+// NOINFS-NEXT:    [[RETVAL_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[RETVAL]], i32 0, i32 0
+// NOINFS-NEXT:    [[RETVAL_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[RETVAL]], i32 0, i32 1
+// NOINFS-NEXT:    store half [[UNPROMOTION]], ptr [[RETVAL_REALP]], align 2
+// NOINFS-NEXT:    store half [[UNPROMOTION9]], ptr [[RETVAL_IMAGP]], align 2
+// NOINFS-NEXT:    [[TMP0:%.*]] = load <2 x half>, ptr [[RETVAL]], align 2
+// NOINFS-NEXT:    ret <2 x half> [[TMP0]]
+//
+_Complex _Float16 defined_complex_func_f16_ret(_Complex _Float16 c) {
+  return c * c;
+}
+
+// CFINITEONLY: Function Attrs: noinline nounwind optnone
+// CFINITEONLY-LABEL: define dso_local nofpclass(nan inf) float @call_variadic
+// CFINITEONLY-SAME: (float noundef nofpclass(nan inf) [[F32:%.*]], double noundef nofpclass(nan inf) [[F64:%.*]], half noundef nofpclass(nan inf) [[F16:%.*]], double noundef nofpclass(nan inf) [[V2F32_COERCE:%.*]], <2 x double> noundef nofpclass(nan inf) [[V2F64:%.*]], i32 noundef [[V2F16_COERCE:%.*]], <2 x float> noundef nofpclass(nan inf) [[CF32_COERCE:%.*]], double noundef nofpclass(nan inf) [[CF64_COERCE0:%.*]], double noundef nofpclass(nan inf) [[CF64_COERCE1:%.*]], ptr noundef byval({ half, half }) align 8 [[CF16:%.*]]) #[[ATTR2]] {
+// CFINITEONLY-NEXT:  entry:
+// CFINITEONLY-NEXT:    [[V2F32:%.*]] = alloca <2 x float>, align 8
+// CFINITEONLY-NEXT:    [[V2F16:%.*]] = alloca <2 x half>, align 4
+// CFINITEONLY-NEXT:    [[CF32:%.*]] = alloca { float, float }, align 4
+// CFINITEONLY-NEXT:    [[CF64:%.*]] = alloca { double, double }, align 8
+// CFINITEONLY-NEXT:    [[F32_ADDR:%.*]] = alloca float, align 4
+// CFINITEONLY-NEXT:    [[F64_ADDR:%.*]] = alloca double, align 8
+// CFINITEONLY-NEXT:    [[F16_ADDR:%.*]] = alloca half, align 2
+// CFINITEONLY-NEXT:    [[V2F32_ADDR:%.*]] = alloca <2 x float>, align 8
+// CFINITEONLY-NEXT:    [[V2F64_ADDR:%.*]] = alloca <2 x double>, align 16
+// CFINITEONLY-NEXT:    [[V2F16_ADDR:%.*]] = alloca <2 x half>, align 4
+// CFINITEONLY-NEXT:    [[COERCE:%.*]] = alloca <2 x float>, align 8
+// CFINITEONLY-NEXT:    [[COERCE3:%.*]] = alloca <2 x half>, align 4
+// CFINITEONLY-NEXT:    [[COERCE4:%.*]] = alloca { float, float }, align 4
+// CFINITEONLY-NEXT:    [[INDIRECT_ARG_TEMP:%.*]] = alloca { double, double }, align 8
+// CFINITEONLY-NEXT:    [[COERCE5:%.*]] = alloca { half, half }, align 2
+// CFINITEONLY-NEXT:    store double [[V2F32_COERCE]], ptr [[V2F32]], align 8
+// CFINITEONLY-NEXT:    [[V2F321:%.*]] = load <2 x float>, ptr [[V2F32]], align 8
+// CFINITEONLY-NEXT:    store i32 [[V2F16_COERCE]], ptr [[V2F16]], align 4
+// CFINITEONLY-NEXT:    [[V2F162:%.*]] = load <2 x half>, ptr [[V2F16]], align 4
+// CFINITEONLY-NEXT:    store <2 x float> [[CF32_COERCE]], ptr [[CF32]], align 4
+// CFINITEONLY-NEXT:    [[TMP0:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 0
+// CFINITEONLY-NEXT:    store double [[CF64_COERCE0]], ptr [[TMP0]], align 8
+// CFINITEONLY-NEXT:    [[TMP1:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 1
+// CFINITEONLY-NEXT:    store double [[CF64_COERCE1]], ptr [[TMP1]], align 8
+// CFINITEONLY-NEXT:    store float [[F32]], ptr [[F32_ADDR]], align 4
+// CFINITEONLY-NEXT:    store double [[F64]], ptr [[F64_ADDR]], align 8
+// CFINITEONLY-NEXT:    store half [[F16]], ptr [[F16_ADDR]], align 2
+// CFINITEONLY-NEXT:    store <2 x float> [[V2F321]], ptr [[V2F32_ADDR]], align 8
+// CFINITEONLY-NEXT:    store <2 x double> [[V2F64]], ptr [[V2F64_ADDR]], align 16
+// CFINITEONLY-NEXT:    store <2 x half> [[V2F162]], ptr [[V2F16_ADDR]], align 4
+// CFINITEONLY-NEXT:    [[TMP2:%.*]] = load float, ptr [[F32_ADDR]], align 4
+// CFINITEONLY-NEXT:    [[TMP3:%.*]] = load float, ptr [[F32_ADDR]], align 4
+// CFINITEONLY-NEXT:    [[CONV:%.*]] = fpext float [[TMP3]] to double
+// CFINITEONLY-NEXT:    [[TMP4:%.*]] = load double, ptr [[F64_ADDR]], align 8
+// CFINITEONLY-NEXT:    [[TMP5:%.*]] = load half, ptr [[F16_ADDR]], align 2
+// CFINITEONLY-NEXT:    [[TMP6:%.*]] = load <2 x float>, ptr [[V2F32_ADDR]], align 8
+// CFINITEONLY-NEXT:    [[TMP7:%.*]] = load <2 x double>, ptr [[V2F64_ADDR]], align 16
+// CFINITEONLY-NEXT:    [[TMP8:%.*]] = load <2 x half>, ptr [[V2F16_ADDR]], align 4
+// CFINITEONLY-NEXT:    [[CF32_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[CF32]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[CF32_REAL:%.*]] = load float, ptr [[CF32_REALP]], align 4
+// CFINITEONLY-NEXT:    [[CF32_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[CF32]], i32 0, i32 1
+// CFINITEONLY-NEXT:    [[CF32_IMAG:%.*]] = load float, ptr [[CF32_IMAGP]], align 4
+// CFINITEONLY-NEXT:    [[CF64_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[CF64_REAL:%.*]] = load double, ptr [[CF64_REALP]], align 8
+// CFINITEONLY-NEXT:    [[CF64_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 1
+// CFINITEONLY-NEXT:    [[CF64_IMAG:%.*]] = load double, ptr [[CF64_IMAGP]], align 8
+// CFINITEONLY-NEXT:    [[CF16_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[CF16]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[CF16_REAL:%.*]] = load half, ptr [[CF16_REALP]], align 8
+// CFINITEONLY-NEXT:    [[CF16_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[CF16]], i32 0, i32 1
+// CFINITEONLY-NEXT:    [[CF16_IMAG:%.*]] = load half, ptr [[CF16_IMAGP]], align 2
+// CFINITEONLY-NEXT:    store <2 x float> [[TMP6]], ptr [[COERCE]], align 8
+// CFINITEONLY-NEXT:    [[TMP9:%.*]] = load double, ptr [[COERCE]], align 8
+// CFINITEONLY-NEXT:    store <2 x half> [[TMP8]], ptr [[COERCE3]], align 4
+// CFINITEONLY-NEXT:    [[TMP10:%.*]] = load i32, ptr [[COERCE3]], align 4
+// CFINITEONLY-NEXT:    [[COERCE4_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE4]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[COERCE4_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE4]], i32 0, i32 1
+// CFINITEONLY-NEXT:    store float [[CF32_REAL]], ptr [[COERCE4_REALP]], align 4
+// CFINITEONLY-NEXT:    store float [[CF32_IMAG]], ptr [[COERCE4_IMAGP]], align 4
+// CFINITEONLY-NEXT:    [[TMP11:%.*]] = load <2 x float>, ptr [[COERCE4]], align 4
+// CFINITEONLY-NEXT:    [[INDIRECT_ARG_TEMP_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[INDIRECT_ARG_TEMP]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[INDIRECT_ARG_TEMP_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[INDIRECT_ARG_TEMP]], i32 0, i32 1
+// CFINITEONLY-NEXT:    store double [[CF64_REAL]], ptr [[INDIRECT_ARG_TEMP_REALP]], align 8
+// CFINITEONLY-NEXT:    store double [[CF64_IMAG]], ptr [[INDIRECT_ARG_TEMP_IMAGP]], align 8
+// CFINITEONLY-NEXT:    [[COERCE5_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[COERCE5]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[COERCE5_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[COERCE5]], i32 0, i32 1
+// CFINITEONLY-NEXT:    store half [[CF16_REAL]], ptr [[COERCE5_REALP]], align 2
+// CFINITEONLY-NEXT:    store half [[CF16_IMAG]], ptr [[COERCE5_IMAGP]], align 2
+// CFINITEONLY-NEXT:    [[TMP12:%.*]] = load <2 x half>, ptr [[COERCE5]], align 2
+// CFINITEONLY-NEXT:    [[CALL:%.*]] = call nnan ninf nofpclass(nan inf) float (float, ...) @variadic(float noundef nofpclass(nan inf) [[TMP2]], double noundef nofpclass(nan inf) [[CONV]], double noundef nofpclass(nan inf) [[TMP4]], half noundef nofpclass(nan inf) [[TMP5]], double noundef nofpclass(nan inf) [[TMP9]], <2 x double> noundef nofpclass(nan inf) [[TMP7]], i32 noundef [[TMP10]], <2 x float> noundef nofpclass(nan inf) [[TMP11]], ptr noundef byval({ double, double }) align 8 [[INDIRECT_ARG_TEMP]], <2 x half> noundef nofpclass(nan inf) [[TMP12]])
+// CFINITEONLY-NEXT:    ret float [[CALL]]
+//
+// CLFINITEONLY: Function Attrs: convergent norecurse nounwind
+// CLFINITEONLY-LABEL: define dso_local nofpclass(nan inf) float @call_variadic
+// CLFINITEONLY-SAME: (float noundef nofpclass(nan inf) [[F32:%.*]], double noundef nofpclass(nan inf) [[F64:%.*]], half noundef nofpclass(nan inf) [[F16:%.*]], double noundef nofpclass(nan inf) [[V2F32_COERCE:%.*]], <2 x double> noundef nofpclass(nan inf) [[V2F64:%.*]], i32 noundef [[V2F16_COERCE:%.*]], <2 x float> noundef nofpclass(nan inf) [[CF32_COERCE:%.*]], double noundef nofpclass(nan inf) [[CF64_COERCE0:%.*]], double noundef nofpclass(nan inf) [[CF64_COERCE1:%.*]], ptr nocapture noundef readonly byval({ half, half }) align 8 [[CF16:%.*]]) local_unnamed_addr #[[ATTR5]] {
+// CLFINITEONLY-NEXT:  entry:
+// CLFINITEONLY-NEXT:    [[INDIRECT_ARG_TEMP:%.*]] = alloca { double, double }, align 8
+// CLFINITEONLY-NEXT:    [[CONV:%.*]] = fpext float [[F32]] to double
+// CLFINITEONLY-NEXT:    [[CF16_REAL:%.*]] = load half, ptr [[CF16]], align 8
+// CLFINITEONLY-NEXT:    [[CF16_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[CF16]], i64 0, i32 1
+// CLFINITEONLY-NEXT:    [[CF16_IMAG:%.*]] = load half, ptr [[CF16_IMAGP]], align 2
+// CLFINITEONLY-NEXT:    [[INDIRECT_ARG_TEMP_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[INDIRECT_ARG_TEMP]], i64 0, i32 1
+// CLFINITEONLY-NEXT:    store double [[CF64_COERCE0]], ptr [[INDIRECT_ARG_TEMP]], align 8
+// CLFINITEONLY-NEXT:    store double [[CF64_COERCE1]], ptr [[INDIRECT_ARG_TEMP_IMAGP]], align 8
+// CLFINITEONLY-NEXT:    [[COERCE5_SROA_0_0_VEC_INSERT:%.*]] = insertelement <2 x half> undef, half [[CF16_REAL]], i64 0
+// CLFINITEONLY-NEXT:    [[COERCE5_SROA_0_2_VEC_INSERT:%.*]] = insertelement <2 x half> [[COERCE5_SROA_0_0_VEC_INSERT]], half [[CF16_IMAG]], i64 1
+// CLFINITEONLY-NEXT:    [[CALL:%.*]] = tail call nnan ninf nofpclass(nan inf) float (float, ...) @variadic(float noundef nofpclass(nan inf) [[F32]], double noundef nofpclass(nan inf) [[CONV]], double noundef nofpclass(nan inf) [[F64]], half noundef nofpclass(nan inf) [[F16]], double noundef nofpclass(nan inf) [[V2F32_COERCE]], <2 x double> noundef nofpclass(nan inf) [[V2F64]], i32 noundef [[V2F16_COERCE]], <2 x float> noundef nofpclass(nan inf) [[CF32_COERCE]], ptr noundef nonnull byval({ double, double }) align 8 [[INDIRECT_ARG_TEMP]], <2 x half> noundef nofpclass(nan inf) [[COERCE5_SROA_0_2_VEC_INSERT]]) #[[ATTR10]]
+// CLFINITEONLY-NEXT:    ret float [[CALL]]
+//
+// NONANS: Function Attrs: noinline nounwind optnone
+// NONANS-LABEL: define dso_local nofpclass(nan) float @call_variadic
+// NONANS-SAME: (float noundef nofpclass(nan) [[F32:%.*]], double noundef nofpclass(nan) [[F64:%.*]], half noundef nofpclass(nan) [[F16:%.*]], double noundef nofpclass(nan) [[V2F32_COERCE:%.*]], <2 x double> noundef nofpclass(nan) [[V2F64:%.*]], i32 noundef [[V2F16_COERCE:%.*]], <2 x float> noundef nofpclass(nan) [[CF32_COERCE:%.*]], double noundef nofpclass(nan) [[CF64_COERCE0:%.*]], double noundef nofpclass(nan) [[CF64_COERCE1:%.*]], ptr noundef byval({ half, half }) align 8 [[CF16:%.*]]) #[[ATTR2]] {
+// NONANS-NEXT:  entry:
+// NONANS-NEXT:    [[V2F32:%.*]] = alloca <2 x float>, align 8
+// NONANS-NEXT:    [[V2F16:%.*]] = alloca <2 x half>, align 4
+// NONANS-NEXT:    [[CF32:%.*]] = alloca { float, float }, align 4
+// NONANS-NEXT:    [[CF64:%.*]] = alloca { double, double }, align 8
+// NONANS-NEXT:    [[F32_ADDR:%.*]] = alloca float, align 4
+// NONANS-NEXT:    [[F64_ADDR:%.*]] = alloca double, align 8
+// NONANS-NEXT:    [[F16_ADDR:%.*]] = alloca half, align 2
+// NONANS-NEXT:    [[V2F32_ADDR:%.*]] = alloca <2 x float>, align 8
+// NONANS-NEXT:    [[V2F64_ADDR:%.*]] = alloca <2 x double>, align 16
+// NONANS-NEXT:    [[V2F16_ADDR:%.*]] = alloca <2 x half>, align 4
+// NONANS-NEXT:    [[COERCE:%.*]] = alloca <2 x float>, align 8
+// NONANS-NEXT:    [[COERCE3:%.*]] = alloca <2 x half>, align 4
+// NONANS-NEXT:    [[COERCE4:%.*]] = alloca { float, float }, align 4
+// NONANS-NEXT:    [[INDIRECT_ARG_TEMP:%.*]] = alloca { double, double }, align 8
+// NONANS-NEXT:    [[COERCE5:%.*]] = alloca { half, half }, align 2
+// NONANS-NEXT:    store double [[V2F32_COERCE]], ptr [[V2F32]], align 8
+// NONANS-NEXT:    [[V2F321:%.*]] = load <2 x float>, ptr [[V2F32]], align 8
+// NONANS-NEXT:    store i32 [[V2F16_COERCE]], ptr [[V2F16]], align 4
+// NONANS-NEXT:    [[V2F162:%.*]] = load <2 x half>, ptr [[V2F16]], align 4
+// NONANS-NEXT:    store <2 x float> [[CF32_COERCE]], ptr [[CF32]], align 4
+// NONANS-NEXT:    [[TMP0:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 0
+// NONANS-NEXT:    store double [[CF64_COERCE0]], ptr [[TMP0]], align 8
+// NONANS-NEXT:    [[TMP1:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 1
+// NONANS-NEXT:    store double [[CF64_COERCE1]], ptr [[TMP1]], align 8
+// NONANS-NEXT:    store float [[F32]], ptr [[F32_ADDR]], align 4
+// NONANS-NEXT:    store double [[F64]], ptr [[F64_ADDR]], align 8
+// NONANS-NEXT:    store half [[F16]], ptr [[F16_ADDR]], align 2
+// NONANS-NEXT:    store <2 x float> [[V2F321]], ptr [[V2F32_ADDR]], align 8
+// NONANS-NEXT:    store <2 x double> [[V2F64]], ptr [[V2F64_ADDR]], align 16
+// NONANS-NEXT:    store <2 x half> [[V2F162]], ptr [[V2F16_ADDR]], align 4
+// NONANS-NEXT:    [[TMP2:%.*]] = load float, ptr [[F32_ADDR]], align 4
+// NONANS-NEXT:    [[TMP3:%.*]] = load float, ptr [[F32_ADDR]], align 4
+// NONANS-NEXT:    [[CONV:%.*]] = fpext float [[TMP3]] to double
+// NONANS-NEXT:    [[TMP4:%.*]] = load double, ptr [[F64_ADDR]], align 8
+// NONANS-NEXT:    [[TMP5:%.*]] = load half, ptr [[F16_ADDR]], align 2
+// NONANS-NEXT:    [[TMP6:%.*]] = load <2 x float>, ptr [[V2F32_ADDR]], align 8
+// NONANS-NEXT:    [[TMP7:%.*]] = load <2 x double>, ptr [[V2F64_ADDR]], align 16
+// NONANS-NEXT:    [[TMP8:%.*]] = load <2 x half>, ptr [[V2F16_ADDR]], align 4
+// NONANS-NEXT:    [[CF32_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[CF32]], i32 0, i32 0
+// NONANS-NEXT:    [[CF32_REAL:%.*]] = load float, ptr [[CF32_REALP]], align 4
+// NONANS-NEXT:    [[CF32_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[CF32]], i32 0, i32 1
+// NONANS-NEXT:    [[CF32_IMAG:%.*]] = load float, ptr [[CF32_IMAGP]], align 4
+// NONANS-NEXT:    [[CF64_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 0
+// NONANS-NEXT:    [[CF64_REAL:%.*]] = load double, ptr [[CF64_REALP]], align 8
+// NONANS-NEXT:    [[CF64_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 1
+// NONANS-NEXT:    [[CF64_IMAG:%.*]] = load double, ptr [[CF64_IMAGP]], align 8
+// NONANS-NEXT:    [[CF16_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[CF16]], i32 0, i32 0
+// NONANS-NEXT:    [[CF16_REAL:%.*]] = load half, ptr [[CF16_REALP]], align 8
+// NONANS-NEXT:    [[CF16_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[CF16]], i32 0, i32 1
+// NONANS-NEXT:    [[CF16_IMAG:%.*]] = load half, ptr [[CF16_IMAGP]], align 2
+// NONANS-NEXT:    store <2 x float> [[TMP6]], ptr [[COERCE]], align 8
+// NONANS-NEXT:    [[TMP9:%.*]] = load double, ptr [[COERCE]], align 8
+// NONANS-NEXT:    store <2 x half> [[TMP8]], ptr [[COERCE3]], align 4
+// NONANS-NEXT:    [[TMP10:%.*]] = load i32, ptr [[COERCE3]], align 4
+// NONANS-NEXT:    [[COERCE4_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE4]], i32 0, i32 0
+// NONANS-NEXT:    [[COERCE4_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE4]], i32 0, i32 1
+// NONANS-NEXT:    store float [[CF32_REAL]], ptr [[COERCE4_REALP]], align 4
+// NONANS-NEXT:    store float [[CF32_IMAG]], ptr [[COERCE4_IMAGP]], align 4
+// NONANS-NEXT:    [[TMP11:%.*]] = load <2 x float>, ptr [[COERCE4]], align 4
+// NONANS-NEXT:    [[INDIRECT_ARG_TEMP_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[INDIRECT_ARG_TEMP]], i32 0, i32 0
+// NONANS-NEXT:    [[INDIRECT_ARG_TEMP_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[INDIRECT_ARG_TEMP]], i32 0, i32 1
+// NONANS-NEXT:    store double [[CF64_REAL]], ptr [[INDIRECT_ARG_TEMP_REALP]], align 8
+// NONANS-NEXT:    store double [[CF64_IMAG]], ptr [[INDIRECT_ARG_TEMP_IMAGP]], align 8
+// NONANS-NEXT:    [[COERCE5_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[COERCE5]], i32 0, i32 0
+// NONANS-NEXT:    [[COERCE5_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[COERCE5]], i32 0, i32 1
+// NONANS-NEXT:    store half [[CF16_REAL]], ptr [[COERCE5_REALP]], align 2
+// NONANS-NEXT:    store half [[CF16_IMAG]], ptr [[COERCE5_IMAGP]], align 2
+// NONANS-NEXT:    [[TMP12:%.*]] = load <2 x half>, ptr [[COERCE5]], align 2
+// NONANS-NEXT:    [[CALL:%.*]] = call nnan nofpclass(nan) float (float, ...) @variadic(float noundef nofpclass(nan) [[TMP2]], double noundef nofpclass(nan) [[CONV]], double noundef nofpclass(nan) [[TMP4]], half noundef nofpclass(nan) [[TMP5]], double noundef nofpclass(nan) [[TMP9]], <2 x double> noundef nofpclass(nan) [[TMP7]], i32 noundef [[TMP10]], <2 x float> noundef nofpclass(nan) [[TMP11]], ptr noundef byval({ double, double }) align 8 [[INDIRECT_ARG_TEMP]], <2 x half> noundef nofpclass(nan) [[TMP12]])
+// NONANS-NEXT:    ret float [[CALL]]
+//
+// NOINFS: Function Attrs: noinline nounwind optnone
+// NOINFS-LABEL: define dso_local nofpclass(inf) float @call_variadic
+// NOINFS-SAME: (float noundef nofpclass(inf) [[F32:%.*]], double noundef nofpclass(inf) [[F64:%.*]], half noundef nofpclass(inf) [[F16:%.*]], double noundef nofpclass(inf) [[V2F32_COERCE:%.*]], <2 x double> noundef nofpclass(inf) [[V2F64:%.*]], i32 noundef [[V2F16_COERCE:%.*]], <2 x float> noundef nofpclass(inf) [[CF32_COERCE:%.*]], double noundef nofpclass(inf) [[CF64_COERCE0:%.*]], double noundef nofpclass(inf) [[CF64_COERCE1:%.*]], ptr noundef byval({ half, half }) align 8 [[CF16:%.*]]) #[[ATTR2]] {
+// NOINFS-NEXT:  entry:
+// NOINFS-NEXT:    [[V2F32:%.*]] = alloca <2 x float>, align 8
+// NOINFS-NEXT:    [[V2F16:%.*]] = alloca <2 x half>, align 4
+// NOINFS-NEXT:    [[CF32:%.*]] = alloca { float, float }, align 4
+// NOINFS-NEXT:    [[CF64:%.*]] = alloca { double, double }, align 8
+// NOINFS-NEXT:    [[F32_ADDR:%.*]] = alloca float, align 4
+// NOINFS-NEXT:    [[F64_ADDR:%.*]] = alloca double, align 8
+// NOINFS-NEXT:    [[F16_ADDR:%.*]] = alloca half, align 2
+// NOINFS-NEXT:    [[V2F32_ADDR:%.*]] = alloca <2 x float>, align 8
+// NOINFS-NEXT:    [[V2F64_ADDR:%.*]] = alloca <2 x double>, align 16
+// NOINFS-NEXT:    [[V2F16_ADDR:%.*]] = alloca <2 x half>, align 4
+// NOINFS-NEXT:    [[COERCE:%.*]] = alloca <2 x float>, align 8
+// NOINFS-NEXT:    [[COERCE3:%.*]] = alloca <2 x half>, align 4
+// NOINFS-NEXT:    [[COERCE4:%.*]] = alloca { float, float }, align 4
+// NOINFS-NEXT:    [[INDIRECT_ARG_TEMP:%.*]] = alloca { double, double }, align 8
+// NOINFS-NEXT:    [[COERCE5:%.*]] = alloca { half, half }, align 2
+// NOINFS-NEXT:    store double [[V2F32_COERCE]], ptr [[V2F32]], align 8
+// NOINFS-NEXT:    [[V2F321:%.*]] = load <2 x float>, ptr [[V2F32]], align 8
+// NOINFS-NEXT:    store i32 [[V2F16_COERCE]], ptr [[V2F16]], align 4
+// NOINFS-NEXT:    [[V2F162:%.*]] = load <2 x half>, ptr [[V2F16]], align 4
+// NOINFS-NEXT:    store <2 x float> [[CF32_COERCE]], ptr [[CF32]], align 4
+// NOINFS-NEXT:    [[TMP0:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 0
+// NOINFS-NEXT:    store double [[CF64_COERCE0]], ptr [[TMP0]], align 8
+// NOINFS-NEXT:    [[TMP1:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 1
+// NOINFS-NEXT:    store double [[CF64_COERCE1]], ptr [[TMP1]], align 8
+// NOINFS-NEXT:    store float [[F32]], ptr [[F32_ADDR]], align 4
+// NOINFS-NEXT:    store double [[F64]], ptr [[F64_ADDR]], align 8
+// NOINFS-NEXT:    store half [[F16]], ptr [[F16_ADDR]], align 2
+// NOINFS-NEXT:    store <2 x float> [[V2F321]], ptr [[V2F32_ADDR]], align 8
+// NOINFS-NEXT:    store <2 x double> [[V2F64]], ptr [[V2F64_ADDR]], align 16
+// NOINFS-NEXT:    store <2 x half> [[V2F162]], ptr [[V2F16_ADDR]], align 4
+// NOINFS-NEXT:    [[TMP2:%.*]] = load float, ptr [[F32_ADDR]], align 4
+// NOINFS-NEXT:    [[TMP3:%.*]] = load float, ptr [[F32_ADDR]], align 4
+// NOINFS-NEXT:    [[CONV:%.*]] = fpext float [[TMP3]] to double
+// NOINFS-NEXT:    [[TMP4:%.*]] = load double, ptr [[F64_ADDR]], align 8
+// NOINFS-NEXT:    [[TMP5:%.*]] = load half, ptr [[F16_ADDR]], align 2
+// NOINFS-NEXT:    [[TMP6:%.*]] = load <2 x float>, ptr [[V2F32_ADDR]], align 8
+// NOINFS-NEXT:    [[TMP7:%.*]] = load <2 x double>, ptr [[V2F64_ADDR]], align 16
+// NOINFS-NEXT:    [[TMP8:%.*]] = load <2 x half>, ptr [[V2F16_ADDR]], align 4
+// NOINFS-NEXT:    [[CF32_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[CF32]], i32 0, i32 0
+// NOINFS-NEXT:    [[CF32_REAL:%.*]] = load float, ptr [[CF32_REALP]], align 4
+// NOINFS-NEXT:    [[CF32_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[CF32]], i32 0, i32 1
+// NOINFS-NEXT:    [[CF32_IMAG:%.*]] = load float, ptr [[CF32_IMAGP]], align 4
+// NOINFS-NEXT:    [[CF64_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 0
+// NOINFS-NEXT:    [[CF64_REAL:%.*]] = load double, ptr [[CF64_REALP]], align 8
+// NOINFS-NEXT:    [[CF64_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 1
+// NOINFS-NEXT:    [[CF64_IMAG:%.*]] = load double, ptr [[CF64_IMAGP]], align 8
+// NOINFS-NEXT:    [[CF16_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[CF16]], i32 0, i32 0
+// NOINFS-NEXT:    [[CF16_REAL:%.*]] = load half, ptr [[CF16_REALP]], align 8
+// NOINFS-NEXT:    [[CF16_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[CF16]], i32 0, i32 1
+// NOINFS-NEXT:    [[CF16_IMAG:%.*]] = load half, ptr [[CF16_IMAGP]], align 2
+// NOINFS-NEXT:    store <2 x float> [[TMP6]], ptr [[COERCE]], align 8
+// NOINFS-NEXT:    [[TMP9:%.*]] = load double, ptr [[COERCE]], align 8
+// NOINFS-NEXT:    store <2 x half> [[TMP8]], ptr [[COERCE3]], align 4
+// NOINFS-NEXT:    [[TMP10:%.*]] = load i32, ptr [[COERCE3]], align 4
+// NOINFS-NEXT:    [[COERCE4_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE4]], i32 0, i32 0
+// NOINFS-NEXT:    [[COERCE4_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE4]], i32 0, i32 1
+// NOINFS-NEXT:    store float [[CF32_REAL]], ptr [[COERCE4_REALP]], align 4
+// NOINFS-NEXT:    store float [[CF32_IMAG]], ptr [[COERCE4_IMAGP]], align 4
+// NOINFS-NEXT:    [[TMP11:%.*]] = load <2 x float>, ptr [[COERCE4]], align 4
+// NOINFS-NEXT:    [[INDIRECT_ARG_TEMP_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[INDIRECT_ARG_TEMP]], i32 0, i32 0
+// NOINFS-NEXT:    [[INDIRECT_ARG_TEMP_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[INDIRECT_ARG_TEMP]], i32 0, i32 1
+// NOINFS-NEXT:    store double [[CF64_REAL]], ptr [[INDIRECT_ARG_TEMP_REALP]], align 8
+// NOINFS-NEXT:    store double [[CF64_IMAG]], ptr [[INDIRECT_ARG_TEMP_IMAGP]], align 8
+// NOINFS-NEXT:    [[COERCE5_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[COERCE5]], i32 0, i32 0
+// NOINFS-NEXT:    [[COERCE5_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[COERCE5]], i32 0, i32 1
+// NOINFS-NEXT:    store half [[CF16_REAL]], ptr [[COERCE5_REALP]], align 2
+// NOINFS-NEXT:    store half [[CF16_IMAG]], ptr [[COERCE5_IMAGP]], align 2
+// NOINFS-NEXT:    [[TMP12:%.*]] = load <2 x half>, ptr [[COERCE5]], align 2
+// NOINFS-NEXT:    [[CALL:%.*]] = call ninf nofpclass(inf) float (float, ...) @variadic(float noundef nofpclass(inf) [[TMP2]], double noundef nofpclass(inf) [[CONV]], double noundef nofpclass(inf) [[TMP4]], half noundef nofpclass(inf) [[TMP5]], double noundef nofpclass(inf) [[TMP9]], <2 x double> noundef nofpclass(inf) [[TMP7]], i32 noundef [[TMP10]], <2 x float> noundef nofpclass(inf) [[TMP11]], ptr noundef byval({ double, double }) align 8 [[INDIRECT_ARG_TEMP]], <2 x half> noundef nofpclass(inf) [[TMP12]])
+// NOINFS-NEXT:    ret float [[CALL]]
+//
+float call_variadic(float f32, double f64, _Float16 f16,
+                    float2 v2f32, double2 v2f64, half2 v2f16,
+                    _Complex float cf32, _Complex double cf64, _Complex _Float16 cf16) {
+  return variadic(f32, f32, f64, f16, v2f32, v2f64, v2f16, cf32, cf64, cf16);
+}
+
+// CFINITEONLY: Function Attrs: noinline nounwind optnone
+// CFINITEONLY-LABEL: define dso_local nofpclass(nan inf) float @call_variadic_indirect
+// CFINITEONLY-SAME: (ptr noundef [[FPTR:%.*]], float noundef nofpclass(nan inf) [[F32:%.*]], double noundef nofpclass(nan inf) [[F64:%.*]], half noundef nofpclass(nan inf) [[F16:%.*]], double noundef nofpclass(nan inf) [[V2F32_COERCE:%.*]], <2 x double> noundef nofpclass(nan inf) [[V2F64:%.*]], i32 noundef [[V2F16_COERCE:%.*]], <2 x float> noundef nofpclass(nan inf) [[CF32_COERCE:%.*]], double noundef nofpclass(nan inf) [[CF64_COERCE0:%.*]], double noundef nofpclass(nan inf) [[CF64_COERCE1:%.*]], ptr noundef byval({ half, half }) align 8 [[CF16:%.*]]) #[[ATTR2]] {
+// CFINITEONLY-NEXT:  entry:
+// CFINITEONLY-NEXT:    [[V2F32:%.*]] = alloca <2 x float>, align 8
+// CFINITEONLY-NEXT:    [[V2F16:%.*]] = alloca <2 x half>, align 4
+// CFINITEONLY-NEXT:    [[CF32:%.*]] = alloca { float, float }, align 4
+// CFINITEONLY-NEXT:    [[CF64:%.*]] = alloca { double, double }, align 8
+// CFINITEONLY-NEXT:    [[FPTR_ADDR:%.*]] = alloca ptr, align 8
+// CFINITEONLY-NEXT:    [[F32_ADDR:%.*]] = alloca float, align 4
+// CFINITEONLY-NEXT:    [[F64_ADDR:%.*]] = alloca double, align 8
+// CFINITEONLY-NEXT:    [[F16_ADDR:%.*]] = alloca half, align 2
+// CFINITEONLY-NEXT:    [[V2F32_ADDR:%.*]] = alloca <2 x float>, align 8
+// CFINITEONLY-NEXT:    [[V2F64_ADDR:%.*]] = alloca <2 x double>, align 16
+// CFINITEONLY-NEXT:    [[V2F16_ADDR:%.*]] = alloca <2 x half>, align 4
+// CFINITEONLY-NEXT:    [[COERCE:%.*]] = alloca <2 x float>, align 8
+// CFINITEONLY-NEXT:    [[COERCE3:%.*]] = alloca <2 x half>, align 4
+// CFINITEONLY-NEXT:    [[COERCE4:%.*]] = alloca { float, float }, align 4
+// CFINITEONLY-NEXT:    [[INDIRECT_ARG_TEMP:%.*]] = alloca { double, double }, align 8
+// CFINITEONLY-NEXT:    [[COERCE5:%.*]] = alloca { half, half }, align 2
+// CFINITEONLY-NEXT:    store double [[V2F32_COERCE]], ptr [[V2F32]], align 8
+// CFINITEONLY-NEXT:    [[V2F321:%.*]] = load <2 x float>, ptr [[V2F32]], align 8
+// CFINITEONLY-NEXT:    store i32 [[V2F16_COERCE]], ptr [[V2F16]], align 4
+// CFINITEONLY-NEXT:    [[V2F162:%.*]] = load <2 x half>, ptr [[V2F16]], align 4
+// CFINITEONLY-NEXT:    store <2 x float> [[CF32_COERCE]], ptr [[CF32]], align 4
+// CFINITEONLY-NEXT:    [[TMP0:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 0
+// CFINITEONLY-NEXT:    store double [[CF64_COERCE0]], ptr [[TMP0]], align 8
+// CFINITEONLY-NEXT:    [[TMP1:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 1
+// CFINITEONLY-NEXT:    store double [[CF64_COERCE1]], ptr [[TMP1]], align 8
+// CFINITEONLY-NEXT:    store ptr [[FPTR]], ptr [[FPTR_ADDR]], align 8
+// CFINITEONLY-NEXT:    store float [[F32]], ptr [[F32_ADDR]], align 4
+// CFINITEONLY-NEXT:    store double [[F64]], ptr [[F64_ADDR]], align 8
+// CFINITEONLY-NEXT:    store half [[F16]], ptr [[F16_ADDR]], align 2
+// CFINITEONLY-NEXT:    store <2 x float> [[V2F321]], ptr [[V2F32_ADDR]], align 8
+// CFINITEONLY-NEXT:    store <2 x double> [[V2F64]], ptr [[V2F64_ADDR]], align 16
+// CFINITEONLY-NEXT:    store <2 x half> [[V2F162]], ptr [[V2F16_ADDR]], align 4
+// CFINITEONLY-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[FPTR_ADDR]], align 8
+// CFINITEONLY-NEXT:    [[TMP3:%.*]] = load float, ptr [[F32_ADDR]], align 4
+// CFINITEONLY-NEXT:    [[TMP4:%.*]] = load float, ptr [[F32_ADDR]], align 4
+// CFINITEONLY-NEXT:    [[CONV:%.*]] = fpext float [[TMP4]] to double
+// CFINITEONLY-NEXT:    [[TMP5:%.*]] = load double, ptr [[F64_ADDR]], align 8
+// CFINITEONLY-NEXT:    [[TMP6:%.*]] = load half, ptr [[F16_ADDR]], align 2
+// CFINITEONLY-NEXT:    [[TMP7:%.*]] = load <2 x float>, ptr [[V2F32_ADDR]], align 8
+// CFINITEONLY-NEXT:    [[TMP8:%.*]] = load <2 x double>, ptr [[V2F64_ADDR]], align 16
+// CFINITEONLY-NEXT:    [[TMP9:%.*]] = load <2 x half>, ptr [[V2F16_ADDR]], align 4
+// CFINITEONLY-NEXT:    [[CF32_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[CF32]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[CF32_REAL:%.*]] = load float, ptr [[CF32_REALP]], align 4
+// CFINITEONLY-NEXT:    [[CF32_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[CF32]], i32 0, i32 1
+// CFINITEONLY-NEXT:    [[CF32_IMAG:%.*]] = load float, ptr [[CF32_IMAGP]], align 4
+// CFINITEONLY-NEXT:    [[CF64_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[CF64_REAL:%.*]] = load double, ptr [[CF64_REALP]], align 8
+// CFINITEONLY-NEXT:    [[CF64_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 1
+// CFINITEONLY-NEXT:    [[CF64_IMAG:%.*]] = load double, ptr [[CF64_IMAGP]], align 8
+// CFINITEONLY-NEXT:    [[CF16_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[CF16]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[CF16_REAL:%.*]] = load half, ptr [[CF16_REALP]], align 8
+// CFINITEONLY-NEXT:    [[CF16_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[CF16]], i32 0, i32 1
+// CFINITEONLY-NEXT:    [[CF16_IMAG:%.*]] = load half, ptr [[CF16_IMAGP]], align 2
+// CFINITEONLY-NEXT:    store <2 x float> [[TMP7]], ptr [[COERCE]], align 8
+// CFINITEONLY-NEXT:    [[TMP10:%.*]] = load double, ptr [[COERCE]], align 8
+// CFINITEONLY-NEXT:    store <2 x half> [[TMP9]], ptr [[COERCE3]], align 4
+// CFINITEONLY-NEXT:    [[TMP11:%.*]] = load i32, ptr [[COERCE3]], align 4
+// CFINITEONLY-NEXT:    [[COERCE4_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE4]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[COERCE4_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE4]], i32 0, i32 1
+// CFINITEONLY-NEXT:    store float [[CF32_REAL]], ptr [[COERCE4_REALP]], align 4
+// CFINITEONLY-NEXT:    store float [[CF32_IMAG]], ptr [[COERCE4_IMAGP]], align 4
+// CFINITEONLY-NEXT:    [[TMP12:%.*]] = load <2 x float>, ptr [[COERCE4]], align 4
+// CFINITEONLY-NEXT:    [[INDIRECT_ARG_TEMP_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[INDIRECT_ARG_TEMP]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[INDIRECT_ARG_TEMP_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[INDIRECT_ARG_TEMP]], i32 0, i32 1
+// CFINITEONLY-NEXT:    store double [[CF64_REAL]], ptr [[INDIRECT_ARG_TEMP_REALP]], align 8
+// CFINITEONLY-NEXT:    store double [[CF64_IMAG]], ptr [[INDIRECT_ARG_TEMP_IMAGP]], align 8
+// CFINITEONLY-NEXT:    [[COERCE5_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[COERCE5]], i32 0, i32 0
+// CFINITEONLY-NEXT:    [[COERCE5_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[COERCE5]], i32 0, i32 1
+// CFINITEONLY-NEXT:    store half [[CF16_REAL]], ptr [[COERCE5_REALP]], align 2
+// CFINITEONLY-NEXT:    store half [[CF16_IMAG]], ptr [[COERCE5_IMAGP]], align 2
+// CFINITEONLY-NEXT:    [[TMP13:%.*]] = load <2 x half>, ptr [[COERCE5]], align 2
+// CFINITEONLY-NEXT:    [[CALL:%.*]] = call nnan ninf nofpclass(nan inf) float (float, ...) [[TMP2]](float noundef nofpclass(nan inf) [[TMP3]], double noundef nofpclass(nan inf) [[CONV]], double noundef nofpclass(nan inf) [[TMP5]], half noundef nofpclass(nan inf) [[TMP6]], double noundef nofpclass(nan inf) [[TMP10]], <2 x double> noundef nofpclass(nan inf) [[TMP8]], i32 noundef [[TMP11]], <2 x float> noundef nofpclass(nan inf) [[TMP12]], ptr noundef byval({ double, double }) align 8 [[INDIRECT_ARG_TEMP]], <2 x half> noundef nofpclass(nan inf) [[TMP13]])
+// CFINITEONLY-NEXT:    ret float [[CALL]]
+//
+// CLFINITEONLY: Function Attrs: convergent norecurse nounwind
+// CLFINITEONLY-LABEL: define dso_local nofpclass(nan inf) float @call_variadic_indirect
+// CLFINITEONLY-SAME: (ptr nocapture noundef readonly [[FPTR:%.*]], float noundef nofpclass(nan inf) [[F32:%.*]], double noundef nofpclass(nan inf) [[F64:%.*]], half noundef nofpclass(nan inf) [[F16:%.*]], double noundef nofpclass(nan inf) [[V2F32_COERCE:%.*]], <2 x double> noundef nofpclass(nan inf) [[V2F64:%.*]], i32 noundef [[V2F16_COERCE:%.*]], <2 x float> noundef nofpclass(nan inf) [[CF32_COERCE:%.*]], double noundef nofpclass(nan inf) [[CF64_COERCE0:%.*]], double noundef nofpclass(nan inf) [[CF64_COERCE1:%.*]], ptr nocapture noundef readonly byval({ half, half }) align 8 [[CF16:%.*]]) local_unnamed_addr #[[ATTR5]] {
+// CLFINITEONLY-NEXT:  entry:
+// CLFINITEONLY-NEXT:    [[INDIRECT_ARG_TEMP:%.*]] = alloca { double, double }, align 8
+// CLFINITEONLY-NEXT:    [[CONV:%.*]] = fpext float [[F32]] to double
+// CLFINITEONLY-NEXT:    [[CF16_REAL:%.*]] = load half, ptr [[CF16]], align 8
+// CLFINITEONLY-NEXT:    [[CF16_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[CF16]], i64 0, i32 1
+// CLFINITEONLY-NEXT:    [[CF16_IMAG:%.*]] = load half, ptr [[CF16_IMAGP]], align 2
+// CLFINITEONLY-NEXT:    [[INDIRECT_ARG_TEMP_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[INDIRECT_ARG_TEMP]], i64 0, i32 1
+// CLFINITEONLY-NEXT:    store double [[CF64_COERCE0]], ptr [[INDIRECT_ARG_TEMP]], align 8
+// CLFINITEONLY-NEXT:    store double [[CF64_COERCE1]], ptr [[INDIRECT_ARG_TEMP_IMAGP]], align 8
+// CLFINITEONLY-NEXT:    [[COERCE5_SROA_0_0_VEC_INSERT:%.*]] = insertelement <2 x half> undef, half [[CF16_REAL]], i64 0
+// CLFINITEONLY-NEXT:    [[COERCE5_SROA_0_2_VEC_INSERT:%.*]] = insertelement <2 x half> [[COERCE5_SROA_0_0_VEC_INSERT]], half [[CF16_IMAG]], i64 1
+// CLFINITEONLY-NEXT:    [[CALL:%.*]] = tail call nnan ninf nofpclass(nan inf) float (float, ...) [[FPTR]](float noundef nofpclass(nan inf) [[F32]], double noundef nofpclass(nan inf) [[CONV]], double noundef nofpclass(nan inf) [[F64]], half noundef nofpclass(nan inf) [[F16]], double noundef nofpclass(nan inf) [[V2F32_COERCE]], <2 x double> noundef nofpclass(nan inf) [[V2F64]], i32 noundef [[V2F16_COERCE]], <2 x float> noundef nofpclass(nan inf) [[CF32_COERCE]], ptr noundef nonnull byval({ double, double }) align 8 [[INDIRECT_ARG_TEMP]], <2 x half> noundef nofpclass(nan inf) [[COERCE5_SROA_0_2_VEC_INSERT]]) #[[ATTR10]]
+// CLFINITEONLY-NEXT:    ret float [[CALL]]
+//
+// NONANS: Function Attrs: noinline nounwind optnone
+// NONANS-LABEL: define dso_local nofpclass(nan) float @call_variadic_indirect
+// NONANS-SAME: (ptr noundef [[FPTR:%.*]], float noundef nofpclass(nan) [[F32:%.*]], double noundef nofpclass(nan) [[F64:%.*]], half noundef nofpclass(nan) [[F16:%.*]], double noundef nofpclass(nan) [[V2F32_COERCE:%.*]], <2 x double> noundef nofpclass(nan) [[V2F64:%.*]], i32 noundef [[V2F16_COERCE:%.*]], <2 x float> noundef nofpclass(nan) [[CF32_COERCE:%.*]], double noundef nofpclass(nan) [[CF64_COERCE0:%.*]], double noundef nofpclass(nan) [[CF64_COERCE1:%.*]], ptr noundef byval({ half, half }) align 8 [[CF16:%.*]]) #[[ATTR2]] {
+// NONANS-NEXT:  entry:
+// NONANS-NEXT:    [[V2F32:%.*]] = alloca <2 x float>, align 8
+// NONANS-NEXT:    [[V2F16:%.*]] = alloca <2 x half>, align 4
+// NONANS-NEXT:    [[CF32:%.*]] = alloca { float, float }, align 4
+// NONANS-NEXT:    [[CF64:%.*]] = alloca { double, double }, align 8
+// NONANS-NEXT:    [[FPTR_ADDR:%.*]] = alloca ptr, align 8
+// NONANS-NEXT:    [[F32_ADDR:%.*]] = alloca float, align 4
+// NONANS-NEXT:    [[F64_ADDR:%.*]] = alloca double, align 8
+// NONANS-NEXT:    [[F16_ADDR:%.*]] = alloca half, align 2
+// NONANS-NEXT:    [[V2F32_ADDR:%.*]] = alloca <2 x float>, align 8
+// NONANS-NEXT:    [[V2F64_ADDR:%.*]] = alloca <2 x double>, align 16
+// NONANS-NEXT:    [[V2F16_ADDR:%.*]] = alloca <2 x half>, align 4
+// NONANS-NEXT:    [[COERCE:%.*]] = alloca <2 x float>, align 8
+// NONANS-NEXT:    [[COERCE3:%.*]] = alloca <2 x half>, align 4
+// NONANS-NEXT:    [[COERCE4:%.*]] = alloca { float, float }, align 4
+// NONANS-NEXT:    [[INDIRECT_ARG_TEMP:%.*]] = alloca { double, double }, align 8
+// NONANS-NEXT:    [[COERCE5:%.*]] = alloca { half, half }, align 2
+// NONANS-NEXT:    store double [[V2F32_COERCE]], ptr [[V2F32]], align 8
+// NONANS-NEXT:    [[V2F321:%.*]] = load <2 x float>, ptr [[V2F32]], align 8
+// NONANS-NEXT:    store i32 [[V2F16_COERCE]], ptr [[V2F16]], align 4
+// NONANS-NEXT:    [[V2F162:%.*]] = load <2 x half>, ptr [[V2F16]], align 4
+// NONANS-NEXT:    store <2 x float> [[CF32_COERCE]], ptr [[CF32]], align 4
+// NONANS-NEXT:    [[TMP0:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 0
+// NONANS-NEXT:    store double [[CF64_COERCE0]], ptr [[TMP0]], align 8
+// NONANS-NEXT:    [[TMP1:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 1
+// NONANS-NEXT:    store double [[CF64_COERCE1]], ptr [[TMP1]], align 8
+// NONANS-NEXT:    store ptr [[FPTR]], ptr [[FPTR_ADDR]], align 8
+// NONANS-NEXT:    store float [[F32]], ptr [[F32_ADDR]], align 4
+// NONANS-NEXT:    store double [[F64]], ptr [[F64_ADDR]], align 8
+// NONANS-NEXT:    store half [[F16]], ptr [[F16_ADDR]], align 2
+// NONANS-NEXT:    store <2 x float> [[V2F321]], ptr [[V2F32_ADDR]], align 8
+// NONANS-NEXT:    store <2 x double> [[V2F64]], ptr [[V2F64_ADDR]], align 16
+// NONANS-NEXT:    store <2 x half> [[V2F162]], ptr [[V2F16_ADDR]], align 4
+// NONANS-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[FPTR_ADDR]], align 8
+// NONANS-NEXT:    [[TMP3:%.*]] = load float, ptr [[F32_ADDR]], align 4
+// NONANS-NEXT:    [[TMP4:%.*]] = load float, ptr [[F32_ADDR]], align 4
+// NONANS-NEXT:    [[CONV:%.*]] = fpext float [[TMP4]] to double
+// NONANS-NEXT:    [[TMP5:%.*]] = load double, ptr [[F64_ADDR]], align 8
+// NONANS-NEXT:    [[TMP6:%.*]] = load half, ptr [[F16_ADDR]], align 2
+// NONANS-NEXT:    [[TMP7:%.*]] = load <2 x float>, ptr [[V2F32_ADDR]], align 8
+// NONANS-NEXT:    [[TMP8:%.*]] = load <2 x double>, ptr [[V2F64_ADDR]], align 16
+// NONANS-NEXT:    [[TMP9:%.*]] = load <2 x half>, ptr [[V2F16_ADDR]], align 4
+// NONANS-NEXT:    [[CF32_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[CF32]], i32 0, i32 0
+// NONANS-NEXT:    [[CF32_REAL:%.*]] = load float, ptr [[CF32_REALP]], align 4
+// NONANS-NEXT:    [[CF32_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[CF32]], i32 0, i32 1
+// NONANS-NEXT:    [[CF32_IMAG:%.*]] = load float, ptr [[CF32_IMAGP]], align 4
+// NONANS-NEXT:    [[CF64_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 0
+// NONANS-NEXT:    [[CF64_REAL:%.*]] = load double, ptr [[CF64_REALP]], align 8
+// NONANS-NEXT:    [[CF64_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 1
+// NONANS-NEXT:    [[CF64_IMAG:%.*]] = load double, ptr [[CF64_IMAGP]], align 8
+// NONANS-NEXT:    [[CF16_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[CF16]], i32 0, i32 0
+// NONANS-NEXT:    [[CF16_REAL:%.*]] = load half, ptr [[CF16_REALP]], align 8
+// NONANS-NEXT:    [[CF16_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[CF16]], i32 0, i32 1
+// NONANS-NEXT:    [[CF16_IMAG:%.*]] = load half, ptr [[CF16_IMAGP]], align 2
+// NONANS-NEXT:    store <2 x float> [[TMP7]], ptr [[COERCE]], align 8
+// NONANS-NEXT:    [[TMP10:%.*]] = load double, ptr [[COERCE]], align 8
+// NONANS-NEXT:    store <2 x half> [[TMP9]], ptr [[COERCE3]], align 4
+// NONANS-NEXT:    [[TMP11:%.*]] = load i32, ptr [[COERCE3]], align 4
+// NONANS-NEXT:    [[COERCE4_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE4]], i32 0, i32 0
+// NONANS-NEXT:    [[COERCE4_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE4]], i32 0, i32 1
+// NONANS-NEXT:    store float [[CF32_REAL]], ptr [[COERCE4_REALP]], align 4
+// NONANS-NEXT:    store float [[CF32_IMAG]], ptr [[COERCE4_IMAGP]], align 4
+// NONANS-NEXT:    [[TMP12:%.*]] = load <2 x float>, ptr [[COERCE4]], align 4
+// NONANS-NEXT:    [[INDIRECT_ARG_TEMP_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[INDIRECT_ARG_TEMP]], i32 0, i32 0
+// NONANS-NEXT:    [[INDIRECT_ARG_TEMP_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[INDIRECT_ARG_TEMP]], i32 0, i32 1
+// NONANS-NEXT:    store double [[CF64_REAL]], ptr [[INDIRECT_ARG_TEMP_REALP]], align 8
+// NONANS-NEXT:    store double [[CF64_IMAG]], ptr [[INDIRECT_ARG_TEMP_IMAGP]], align 8
+// NONANS-NEXT:    [[COERCE5_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[COERCE5]], i32 0, i32 0
+// NONANS-NEXT:    [[COERCE5_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[COERCE5]], i32 0, i32 1
+// NONANS-NEXT:    store half [[CF16_REAL]], ptr [[COERCE5_REALP]], align 2
+// NONANS-NEXT:    store half [[CF16_IMAG]], ptr [[COERCE5_IMAGP]], align 2
+// NONANS-NEXT:    [[TMP13:%.*]] = load <2 x half>, ptr [[COERCE5]], align 2
+// NONANS-NEXT:    [[CALL:%.*]] = call nnan nofpclass(nan) float (float, ...) [[TMP2]](float noundef nofpclass(nan) [[TMP3]], double noundef nofpclass(nan) [[CONV]], double noundef nofpclass(nan) [[TMP5]], half noundef nofpclass(nan) [[TMP6]], double noundef nofpclass(nan) [[TMP10]], <2 x double> noundef nofpclass(nan) [[TMP8]], i32 noundef [[TMP11]], <2 x float> noundef nofpclass(nan) [[TMP12]], ptr noundef byval({ double, double }) align 8 [[INDIRECT_ARG_TEMP]], <2 x half> noundef nofpclass(nan) [[TMP13]])
+// NONANS-NEXT:    ret float [[CALL]]
+//
+// NOINFS: Function Attrs: noinline nounwind optnone
+// NOINFS-LABEL: define dso_local nofpclass(inf) float @call_variadic_indirect
+// NOINFS-SAME: (ptr noundef [[FPTR:%.*]], float noundef nofpclass(inf) [[F32:%.*]], double noundef nofpclass(inf) [[F64:%.*]], half noundef nofpclass(inf) [[F16:%.*]], double noundef nofpclass(inf) [[V2F32_COERCE:%.*]], <2 x double> noundef nofpclass(inf) [[V2F64:%.*]], i32 noundef [[V2F16_COERCE:%.*]], <2 x float> noundef nofpclass(inf) [[CF32_COERCE:%.*]], double noundef nofpclass(inf) [[CF64_COERCE0:%.*]], double noundef nofpclass(inf) [[CF64_COERCE1:%.*]], ptr noundef byval({ half, half }) align 8 [[CF16:%.*]]) #[[ATTR2]] {
+// NOINFS-NEXT:  entry:
+// NOINFS-NEXT:    [[V2F32:%.*]] = alloca <2 x float>, align 8
+// NOINFS-NEXT:    [[V2F16:%.*]] = alloca <2 x half>, align 4
+// NOINFS-NEXT:    [[CF32:%.*]] = alloca { float, float }, align 4
+// NOINFS-NEXT:    [[CF64:%.*]] = alloca { double, double }, align 8
+// NOINFS-NEXT:    [[FPTR_ADDR:%.*]] = alloca ptr, align 8
+// NOINFS-NEXT:    [[F32_ADDR:%.*]] = alloca float, align 4
+// NOINFS-NEXT:    [[F64_ADDR:%.*]] = alloca double, align 8
+// NOINFS-NEXT:    [[F16_ADDR:%.*]] = alloca half, align 2
+// NOINFS-NEXT:    [[V2F32_ADDR:%.*]] = alloca <2 x float>, align 8
+// NOINFS-NEXT:    [[V2F64_ADDR:%.*]] = alloca <2 x double>, align 16
+// NOINFS-NEXT:    [[V2F16_ADDR:%.*]] = alloca <2 x half>, align 4
+// NOINFS-NEXT:    [[COERCE:%.*]] = alloca <2 x float>, align 8
+// NOINFS-NEXT:    [[COERCE3:%.*]] = alloca <2 x half>, align 4
+// NOINFS-NEXT:    [[COERCE4:%.*]] = alloca { float, float }, align 4
+// NOINFS-NEXT:    [[INDIRECT_ARG_TEMP:%.*]] = alloca { double, double }, align 8
+// NOINFS-NEXT:    [[COERCE5:%.*]] = alloca { half, half }, align 2
+// NOINFS-NEXT:    store double [[V2F32_COERCE]], ptr [[V2F32]], align 8
+// NOINFS-NEXT:    [[V2F321:%.*]] = load <2 x float>, ptr [[V2F32]], align 8
+// NOINFS-NEXT:    store i32 [[V2F16_COERCE]], ptr [[V2F16]], align 4
+// NOINFS-NEXT:    [[V2F162:%.*]] = load <2 x half>, ptr [[V2F16]], align 4
+// NOINFS-NEXT:    store <2 x float> [[CF32_COERCE]], ptr [[CF32]], align 4
+// NOINFS-NEXT:    [[TMP0:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 0
+// NOINFS-NEXT:    store double [[CF64_COERCE0]], ptr [[TMP0]], align 8
+// NOINFS-NEXT:    [[TMP1:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 1
+// NOINFS-NEXT:    store double [[CF64_COERCE1]], ptr [[TMP1]], align 8
+// NOINFS-NEXT:    store ptr [[FPTR]], ptr [[FPTR_ADDR]], align 8
+// NOINFS-NEXT:    store float [[F32]], ptr [[F32_ADDR]], align 4
+// NOINFS-NEXT:    store double [[F64]], ptr [[F64_ADDR]], align 8
+// NOINFS-NEXT:    store half [[F16]], ptr [[F16_ADDR]], align 2
+// NOINFS-NEXT:    store <2 x float> [[V2F321]], ptr [[V2F32_ADDR]], align 8
+// NOINFS-NEXT:    store <2 x double> [[V2F64]], ptr [[V2F64_ADDR]], align 16
+// NOINFS-NEXT:    store <2 x half> [[V2F162]], ptr [[V2F16_ADDR]], align 4
+// NOINFS-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[FPTR_ADDR]], align 8
+// NOINFS-NEXT:    [[TMP3:%.*]] = load float, ptr [[F32_ADDR]], align 4
+// NOINFS-NEXT:    [[TMP4:%.*]] = load float, ptr [[F32_ADDR]], align 4
+// NOINFS-NEXT:    [[CONV:%.*]] = fpext float [[TMP4]] to double
+// NOINFS-NEXT:    [[TMP5:%.*]] = load double, ptr [[F64_ADDR]], align 8
+// NOINFS-NEXT:    [[TMP6:%.*]] = load half, ptr [[F16_ADDR]], align 2
+// NOINFS-NEXT:    [[TMP7:%.*]] = load <2 x float>, ptr [[V2F32_ADDR]], align 8
+// NOINFS-NEXT:    [[TMP8:%.*]] = load <2 x double>, ptr [[V2F64_ADDR]], align 16
+// NOINFS-NEXT:    [[TMP9:%.*]] = load <2 x half>, ptr [[V2F16_ADDR]], align 4
+// NOINFS-NEXT:    [[CF32_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[CF32]], i32 0, i32 0
+// NOINFS-NEXT:    [[CF32_REAL:%.*]] = load float, ptr [[CF32_REALP]], align 4
+// NOINFS-NEXT:    [[CF32_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[CF32]], i32 0, i32 1
+// NOINFS-NEXT:    [[CF32_IMAG:%.*]] = load float, ptr [[CF32_IMAGP]], align 4
+// NOINFS-NEXT:    [[CF64_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 0
+// NOINFS-NEXT:    [[CF64_REAL:%.*]] = load double, ptr [[CF64_REALP]], align 8
+// NOINFS-NEXT:    [[CF64_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[CF64]], i32 0, i32 1
+// NOINFS-NEXT:    [[CF64_IMAG:%.*]] = load double, ptr [[CF64_IMAGP]], align 8
+// NOINFS-NEXT:    [[CF16_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[CF16]], i32 0, i32 0
+// NOINFS-NEXT:    [[CF16_REAL:%.*]] = load half, ptr [[CF16_REALP]], align 8
+// NOINFS-NEXT:    [[CF16_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[CF16]], i32 0, i32 1
+// NOINFS-NEXT:    [[CF16_IMAG:%.*]] = load half, ptr [[CF16_IMAGP]], align 2
+// NOINFS-NEXT:    store <2 x float> [[TMP7]], ptr [[COERCE]], align 8
+// NOINFS-NEXT:    [[TMP10:%.*]] = load double, ptr [[COERCE]], align 8
+// NOINFS-NEXT:    store <2 x half> [[TMP9]], ptr [[COERCE3]], align 4
+// NOINFS-NEXT:    [[TMP11:%.*]] = load i32, ptr [[COERCE3]], align 4
+// NOINFS-NEXT:    [[COERCE4_REALP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE4]], i32 0, i32 0
+// NOINFS-NEXT:    [[COERCE4_IMAGP:%.*]] = getelementptr inbounds { float, float }, ptr [[COERCE4]], i32 0, i32 1
+// NOINFS-NEXT:    store float [[CF32_REAL]], ptr [[COERCE4_REALP]], align 4
+// NOINFS-NEXT:    store float [[CF32_IMAG]], ptr [[COERCE4_IMAGP]], align 4
+// NOINFS-NEXT:    [[TMP12:%.*]] = load <2 x float>, ptr [[COERCE4]], align 4
+// NOINFS-NEXT:    [[INDIRECT_ARG_TEMP_REALP:%.*]] = getelementptr inbounds { double, double }, ptr [[INDIRECT_ARG_TEMP]], i32 0, i32 0
+// NOINFS-NEXT:    [[INDIRECT_ARG_TEMP_IMAGP:%.*]] = getelementptr inbounds { double, double }, ptr [[INDIRECT_ARG_TEMP]], i32 0, i32 1
+// NOINFS-NEXT:    store double [[CF64_REAL]], ptr [[INDIRECT_ARG_TEMP_REALP]], align 8
+// NOINFS-NEXT:    store double [[CF64_IMAG]], ptr [[INDIRECT_ARG_TEMP_IMAGP]], align 8
+// NOINFS-NEXT:    [[COERCE5_REALP:%.*]] = getelementptr inbounds { half, half }, ptr [[COERCE5]], i32 0, i32 0
+// NOINFS-NEXT:    [[COERCE5_IMAGP:%.*]] = getelementptr inbounds { half, half }, ptr [[COERCE5]], i32 0, i32 1
+// NOINFS-NEXT:    store half [[CF16_REAL]], ptr [[COERCE5_REALP]], align 2
+// NOINFS-NEXT:    store half [[CF16_IMAG]], ptr [[COERCE5_IMAGP]], align 2
+// NOINFS-NEXT:    [[TMP13:%.*]] = load <2 x half>, ptr [[COERCE5]], align 2
+// NOINFS-NEXT:    [[CALL:%.*]] = call ninf nofpclass(inf) float (float, ...) [[TMP2]](float noundef nofpclass(inf) [[TMP3]], double noundef nofpclass(inf) [[CONV]], double noundef nofpclass(inf) [[TMP5]], half noundef nofpclass(inf) [[TMP6]], double noundef nofpclass(inf) [[TMP10]], <2 x double> noundef nofpclass(inf) [[TMP8]], i32 noundef [[TMP11]], <2 x float> noundef nofpclass(inf) [[TMP12]], ptr noundef byval({ double, double }) align 8 [[INDIRECT_ARG_TEMP]], <2 x half> noundef nofpclass(inf) [[TMP13]])
+// NOINFS-NEXT:    ret float [[CALL]]
+//
+float call_variadic_indirect(float fptr(float, ...), float f32, double f64, _Float16 f16,
+                             float2 v2f32, double2 v2f64, half2 v2f16,
+                             _Complex float cf32, _Complex double cf64, _Complex _Float16 cf16) {
+  return (*fptr)(f32, f32, f64, f16, v2f32, v2f64, v2f16, cf32, cf64, cf16);
+}
+
+typedef __attribute__((__vector_size__(4 * sizeof(double)))) double __m256d;
+extern __m256d extern_m256d(__m256d, ...);
+
+// CFINITEONLY: Function Attrs: noinline nounwind optnone
+// CFINITEONLY-LABEL: define dso_local nofpclass(nan inf) <4 x double> @call_m256d
+// CFINITEONLY-SAME: (<4 x double> noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR5:[0-9]+]] {
+// CFINITEONLY-NEXT:  entry:
+// CFINITEONLY-NEXT:    [[X_ADDR:%.*]] = alloca <4 x double>, align 32
+// CFINITEONLY-NEXT:    store <4 x double> [[X]], ptr [[X_ADDR]], align 32
+// CFINITEONLY-NEXT:    [[TMP0:%.*]] = load <4 x double>, ptr [[X_ADDR]], align 32
+// CFINITEONLY-NEXT:    [[TMP1:%.*]] = load <4 x double>, ptr [[X_ADDR]], align 32
+// CFINITEONLY-NEXT:    [[CALL:%.*]] = call nnan ninf nofpclass(nan inf) <4 x double> (<4 x double>, ...) @extern_m256d(<4 x double> noundef nofpclass(nan inf) [[TMP0]], <4 x double> noundef nofpclass(nan inf) [[TMP1]])
+// CFINITEONLY-NEXT:    ret <4 x double> [[CALL]]
+//
+// CLFINITEONLY: Function Attrs: convergent norecurse nounwind
+// CLFINITEONLY-LABEL: define dso_local nofpclass(nan inf) <4 x double> @call_m256d
+// CLFINITEONLY-SAME: (<4 x double> noundef nofpclass(nan inf) [[X:%.*]]) local_unnamed_addr #[[ATTR8:[0-9]+]] {
+// CLFINITEONLY-NEXT:  entry:
+// CLFINITEONLY-NEXT:    [[CALL:%.*]] = tail call nnan ninf nofpclass(nan inf) <4 x double> (<4 x double>, ...) @extern_m256d(<4 x double> noundef nofpclass(nan inf) [[X]], <4 x double> noundef nofpclass(nan inf) [[X]]) #[[ATTR10]]
+// CLFINITEONLY-NEXT:    ret <4 x double> [[CALL]]
+//
+// NONANS: Function Attrs: noinline nounwind optnone
+// NONANS-LABEL: define dso_local nofpclass(nan) <4 x double> @call_m256d
+// NONANS-SAME: (<4 x double> noundef nofpclass(nan) [[X:%.*]]) #[[ATTR5:[0-9]+]] {
+// NONANS-NEXT:  entry:
+// NONANS-NEXT:    [[X_ADDR:%.*]] = alloca <4 x double>, align 32
+// NONANS-NEXT:    store <4 x double> [[X]], ptr [[X_ADDR]], align 32
+// NONANS-NEXT:    [[TMP0:%.*]] = load <4 x double>, ptr [[X_ADDR]], align 32
+// NONANS-NEXT:    [[TMP1:%.*]] = load <4 x double>, ptr [[X_ADDR]], align 32
+// NONANS-NEXT:    [[CALL:%.*]] = call nnan nofpclass(nan) <4 x double> (<4 x double>, ...) @extern_m256d(<4 x double> noundef nofpclass(nan) [[TMP0]], <4 x double> noundef nofpclass(nan) [[TMP1]])
+// NONANS-NEXT:    ret <4 x double> [[CALL]]
+//
+// NOINFS: Function Attrs: noinline nounwind optnone
+// NOINFS-LABEL: define dso_local nofpclass(inf) <4 x double> @call_m256d
+// NOINFS-SAME: (<4 x double> noundef nofpclass(inf) [[X:%.*]]) #[[ATTR5:[0-9]+]] {
+// NOINFS-NEXT:  entry:
+// NOINFS-NEXT:    [[X_ADDR:%.*]] = alloca <4 x double>, align 32
+// NOINFS-NEXT:    store <4 x double> [[X]], ptr [[X_ADDR]], align 32
+// NOINFS-NEXT:    [[TMP0:%.*]] = load <4 x double>, ptr [[X_ADDR]], align 32
+// NOINFS-NEXT:    [[TMP1:%.*]] = load <4 x double>, ptr [[X_ADDR]], align 32
+// NOINFS-NEXT:    [[CALL:%.*]] = call ninf nofpclass(inf) <4 x double> (<4 x double>, ...) @extern_m256d(<4 x double> noundef nofpclass(inf) [[TMP0]], <4 x double> noundef nofpclass(inf) [[TMP1]])
+// NOINFS-NEXT:    ret <4 x double> [[CALL]]
+//
+__m256d call_m256d(__m256d x) {
+  return extern_m256d(x, x);
+}
+
+// CFINITEONLY: Function Attrs: noinline nounwind optnone
+// CFINITEONLY-LABEL: define dso_local nofpclass(nan inf) <25 x double> @call_matrix
+// CFINITEONLY-SAME: (<25 x double> noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR6:[0-9]+]] {
+// CFINITEONLY-NEXT:  entry:
+// CFINITEONLY-NEXT:    [[X_ADDR:%.*]] = alloca [25 x double], align 8
+// CFINITEONLY-NEXT:    store <25 x double> [[X]], ptr [[X_ADDR]], align 8
+// CFINITEONLY-NEXT:    [[TMP0:%.*]] = load <25 x double>, ptr [[X_ADDR]], align 8
+// CFINITEONLY-NEXT:    [[CALL:%.*]] = call nnan ninf nofpclass(nan inf) <25 x double> @extern_matrix(<25 x double> noundef nofpclass(nan inf) [[TMP0]])
+// CFINITEONLY-NEXT:    ret <25 x double> [[CALL]]
+//
+// CLFINITEONLY: Function Attrs: convergent norecurse nounwind
+// CLFINITEONLY-LABEL: define dso_local nofpclass(nan inf) <25 x double> @call_matrix
+// CLFINITEONLY-SAME: (<25 x double> noundef nofpclass(nan inf) [[X:%.*]]) local_unnamed_addr #[[ATTR9:[0-9]+]] {
+// CLFINITEONLY-NEXT:  entry:
+// CLFINITEONLY-NEXT:    [[CALL:%.*]] = tail call nnan ninf nofpclass(nan inf) <25 x double> @extern_matrix(<25 x double> noundef nofpclass(nan inf) [[X]]) #[[ATTR10]]
+// CLFINITEONLY-NEXT:    ret <25 x double> [[CALL]]
+//
+// NONANS: Function Attrs: noinline nounwind optnone
+// NONANS-LABEL: define dso_local nofpclass(nan) <25 x double> @call_matrix
+// NONANS-SAME: (<25 x double> noundef nofpclass(nan) [[X:%.*]]) #[[ATTR6:[0-9]+]] {
+// NONANS-NEXT:  entry:
+// NONANS-NEXT:    [[X_ADDR:%.*]] = alloca [25 x double], align 8
+// NONANS-NEXT:    store <25 x double> [[X]], ptr [[X_ADDR]], align 8
+// NONANS-NEXT:    [[TMP0:%.*]] = load <25 x double>, ptr [[X_ADDR]], align 8
+// NONANS-NEXT:    [[CALL:%.*]] = call nnan nofpclass(nan) <25 x double> @extern_matrix(<25 x double> noundef nofpclass(nan) [[TMP0]])
+// NONANS-NEXT:    ret <25 x double> [[CALL]]
+//
+// NOINFS: Function Attrs: noinline nounwind optnone
+// NOINFS-LABEL: define dso_local nofpclass(inf) <25 x double> @call_matrix
+// NOINFS-SAME: (<25 x double> noundef nofpclass(inf) [[X:%.*]]) #[[ATTR6:[0-9]+]] {
+// NOINFS-NEXT:  entry:
+// NOINFS-NEXT:    [[X_ADDR:%.*]] = alloca [25 x double], align 8
+// NOINFS-NEXT:    store <25 x double> [[X]], ptr [[X_ADDR]], align 8
+// NOINFS-NEXT:    [[TMP0:%.*]] = load <25 x double>, ptr [[X_ADDR]], align 8
+// NOINFS-NEXT:    [[CALL:%.*]] = call ninf nofpclass(inf) <25 x double> @extern_matrix(<25 x double> noundef nofpclass(inf) [[TMP0]])
+// NOINFS-NEXT:    ret <25 x double> [[CALL]]
+//
+dx5x5_t call_matrix(dx5x5_t x) {
+  return extern_matrix(x);
+}
diff --git a/clang/test/CodeGen/packed-nest-unpacked.c b/clang/test/CodeGen/packed-nest-unpacked.c
index 50479efbca106..318345bed90aa 100644
--- a/clang/test/CodeGen/packed-nest-unpacked.c
+++ b/clang/test/CodeGen/packed-nest-unpacked.c
@@ -1,4 +1,4 @@
-// RUN: %clang_cc1 -no-opaque-pointers %s -triple x86_64-apple-macosx10.7.2 -emit-llvm -o - | FileCheck %s
+// RUN: %clang_cc1 %s -triple x86_64-apple-macosx10.7.2 -emit-llvm -o - | FileCheck %s
 
 struct X { int x[6]; };
 struct Y { char x[13]; struct X y; } __attribute((packed));
@@ -9,40 +9,40 @@ struct X foo(void);
 // <rdar://problem/10463337>
 struct X test1(void) {
   // CHECK: @test1
-  // CHECK: call void @llvm.memcpy.p0i8.p0i8.i64(i8* {{.*}}, i8* align 1 bitcast (%struct.X* getelementptr inbounds (%struct.Y, %struct.Y* @g, i32 0, i32 1) to i8*), i64 24, i1 false)
+  // CHECK: call void @llvm.memcpy.p0.p0.i64(ptr {{.*}}, ptr align 1 getelementptr inbounds (%struct.Y, ptr @g, i32 0, i32 1), i64 24, i1 false)
   return g.y;
 }
 struct X test2(void) {
   // CHECK: @test2
-  // CHECK: call void @llvm.memcpy.p0i8.p0i8.i64(i8* {{.*}}, i8* align 1 bitcast (%struct.X* getelementptr inbounds (%struct.Y, %struct.Y* @g, i32 0, i32 1) to i8*), i64 24, i1 false)
+  // CHECK: call void @llvm.memcpy.p0.p0.i64(ptr {{.*}}, ptr align 1 getelementptr inbounds (%struct.Y, ptr @g, i32 0, i32 1), i64 24, i1 false)
   struct X a = g.y;
   return a;
 }
 
 void test3(struct X a) {
   // CHECK: @test3
-  // CHECK: call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 bitcast (%struct.X* getelementptr inbounds (%struct.Y, %struct.Y* @g, i32 0, i32 1) to i8*), i8* {{.*}}, i64 24, i1 false)
+  // CHECK: call void @llvm.memcpy.p0.p0.i64(ptr align 1 getelementptr inbounds (%struct.Y, ptr @g, i32 0, i32 1), ptr {{.*}}, i64 24, i1 false)
   g.y = a;
 }
 
 // <rdar://problem/10530444>
 void test4(void) {
   // CHECK: @test4
-  // CHECK: call void @llvm.memcpy.p0i8.p0i8.i64(i8* {{.*}}, i8* align 1 bitcast (%struct.X* getelementptr inbounds (%struct.Y, %struct.Y* @g, i32 0, i32 1) to i8*), i64 24, i1 false)
+  // CHECK: call void @llvm.memcpy.p0.p0.i64(ptr {{.*}}, ptr align 1 getelementptr inbounds (%struct.Y, ptr @g, i32 0, i32 1), i64 24, i1 false)
   f(g.y);
 }
 
 // PR12395
 int test5(void) {
   // CHECK: @test5
-  // CHECK: load i32, i32* getelementptr inbounds (%struct.Y, %struct.Y* @g, i32 0, i32 1, i32 0, i64 0), align 1
+  // CHECK: load i32, ptr getelementptr inbounds (%struct.Y, ptr @g, i32 0, i32 1), align 1
   return g.y.x[0];
 }
 
 // <rdar://problem/11220251>
 void test6(void) {
   // CHECK: @test6
-  // CHECK: call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 bitcast (%struct.X* getelementptr inbounds (%struct.Y, %struct.Y* @g, i32 0, i32 1) to i8*), i8* align 4 %{{.*}}, i64 24, i1 false)
+  // CHECK: call void @llvm.memcpy.p0.p0.i64(ptr align 1 getelementptr inbounds (%struct.Y, ptr @g, i32 0, i32 1), ptr align 4 %{{.*}}, i64 24, i1 false)
   g.y = foo();
 }
 
@@ -60,14 +60,14 @@ struct YBitfield gbitfield;
 
 unsigned test7(void) {
   // CHECK: @test7
-  // CHECK: load i32, i32* getelementptr inbounds (%struct.YBitfield, %struct.YBitfield* @gbitfield, i32 0, i32 1, i32 0), align 1
+  // CHECK: load i32, ptr getelementptr inbounds (%struct.YBitfield, ptr @gbitfield, i32 0, i32 1), align 1
   return gbitfield.y.b2;
 }
 
 void test8(unsigned x) {
   // CHECK: @test8
-  // CHECK: load i32, i32* getelementptr inbounds (%struct.YBitfield, %struct.YBitfield* @gbitfield, i32 0, i32 1, i32 0), align 1
-  // CHECK: store i32 {{.*}}, i32* getelementptr inbounds (%struct.YBitfield, %struct.YBitfield* @gbitfield, i32 0, i32 1, i32 0), align 1
+  // CHECK: load i32, ptr getelementptr inbounds (%struct.YBitfield, ptr @gbitfield, i32 0, i32 1), align 1
+  // CHECK: store i32 {{.*}}, ptr getelementptr inbounds (%struct.YBitfield, ptr @gbitfield, i32 0, i32 1), align 1
   gbitfield.y.b2 = x;
 }
 
@@ -81,14 +81,14 @@ struct TBitfield tbitfield;
 
 unsigned test9(void) {
   // CHECK: @test9
-  // CHECK: load i16, i16* getelementptr inbounds (%struct.TBitfield, %struct.TBitfield* @tbitfield, i32 0, i32 2), align 1
+  // CHECK: load i16, ptr getelementptr inbounds (%struct.TBitfield, ptr @tbitfield, i32 0, i32 2), align 1
   return tbitfield.c;
 }
 
 void test10(unsigned x) {
   // CHECK: @test10
-  // CHECK: load i16, i16* getelementptr inbounds (%struct.TBitfield, %struct.TBitfield* @tbitfield, i32 0, i32 2), align 1
-  // CHECK: store i16 {{.*}}, i16* getelementptr inbounds (%struct.TBitfield, %struct.TBitfield* @tbitfield, i32 0, i32 2), align 1
+  // CHECK: load i16, ptr getelementptr inbounds (%struct.TBitfield, ptr @tbitfield, i32 0, i32 2), align 1
+  // CHECK: store i16 {{.*}}, ptr getelementptr inbounds (%struct.TBitfield, ptr @tbitfield, i32 0, i32 2), align 1
   tbitfield.c = x;
 }
 
diff --git a/clang/test/CodeGen/partial-reinitialization2.c b/clang/test/CodeGen/partial-reinitialization2.c
index 164a3ca6df7a5..e709c1d4ad1ee 100644
--- a/clang/test/CodeGen/partial-reinitialization2.c
+++ b/clang/test/CodeGen/partial-reinitialization2.c
@@ -1,4 +1,4 @@
-// RUN: %clang_cc1 -no-opaque-pointers %s -triple x86_64-unknown-unknown -emit-llvm -o - | FileCheck %s
+// RUN: %clang_cc1 %s -triple x86_64-unknown-unknown -emit-llvm -o - | FileCheck %s
 
 struct P1 { char x[6]; } g1 = { "foo" };
 struct LP1 { struct P1 p1; };
@@ -15,8 +15,8 @@ union ULP3 { struct LP3 l3; };
 // CHECK-LABEL: test1
 void test1(void)
 {
-  // CHECK: call void @llvm.memcpy{{.*}}%struct.P1, %struct.P1* @g1{{.*}}i64 6, i1 false)
-  // CHECK: store i8 120, i8* %
+  // CHECK: call void @llvm.memcpy{{.*}}ptr align 1 @g1, i64 6, i1 false)
+  // CHECK: store i8 120, ptr %
 
   struct LP1 l = { .p1 = g1, .p1.x[2] = 'x' };
 }
@@ -24,8 +24,8 @@ void test1(void)
 // CHECK-LABEL: test2
 void test2(void)
 {
-  // CHECK: call void @llvm.memcpy{{.*}}%struct.P1, %struct.P1* @g1{{.*}}i64 6, i1 false)
-  // CHECK: store i8 114, i8* %
+  // CHECK: call void @llvm.memcpy{{.*}}ptr align 1 @g1, i64 6, i1 false)
+  // CHECK: store i8 114, ptr %
 
   struct LP1 l = { .p1 = g1, .p1.x[1] = 'r' };
 }
@@ -33,8 +33,8 @@ void test2(void)
 // CHECK-LABEL: test3
 void test3(void)
 {
-  // CHECK: call void @llvm.memcpy{{.*}}%struct.P2* @g2{{.*}}i64 12, i1 false)
-  // CHECK: store i32 10, i32* %
+  // CHECK: call void @llvm.memcpy{{.*}}ptr align 4 @g2, i64 12, i1 false)
+  // CHECK: store i32 10, ptr %
 
   struct LP2 l = { .p2 = g2, .p2.b = 10 };
 }
@@ -64,10 +64,9 @@ union UP2 get123(void)
 void test4(void)
 {
   // CHECK: [[CALL:%[a-z0-9]+]] = call {{.*}}@get123()
-  // CHECK: store{{.*}}[[CALL]], {{.*}}[[TMP0:%[a-z0-9]+]]
-  // CHECK: [[TMP1:%[a-z0-9]+]] = bitcast {{.*}}[[TMP0]]
-  // CHECK: call void @llvm.memcpy{{.*}}[[TMP1]], i64 12, i1 false)
-  // CHECK: store i32 100, i32* %
+  // CHECK: store{{.*}}[[CALL]], {{.*}}[[TMP0:%[a-z0-9.]+]]
+  // CHECK: call void @llvm.memcpy{{.*}}[[TMP0]], i64 12, i1 false)
+  // CHECK: store i32 100, ptr %
 
   struct LUP2 { union UP2 up; } var = { get123(), .up.p2.a = 100 };
 }
@@ -76,15 +75,15 @@ void test4(void)
 void test5(void)
 {
   // .l3 = g3
-  // CHECK: call void @llvm.memcpy{{.*}}%struct.LP3, %struct.LP3* @g3{{.*}}i64 12, i1 false)
+  // CHECK: call void @llvm.memcpy{{.*}}ptr align 1 @g3, i64 12, i1 false)
 
   // .l3.p1 = { [0] = g1 } implicitly sets [1] to zero
-  // CHECK: call void @llvm.memcpy{{.*}}%struct.P1, %struct.P1* @g1{{.*}}i64 6, i1 false)
-  // CHECK: getelementptr{{.*}}%struct.P1, %struct.P1*{{.*}}i64 1
+  // CHECK: call void @llvm.memcpy{{.*}}ptr align 1 @g1, i64 6, i1 false)
+  // CHECK: getelementptr{{.*}}%struct.P1, ptr{{.*}}i64 1
   // CHECK: call void @llvm.memset{{.*}}i8 0, i64 6, i1 false)
 
   // .l3.p1[1].x[1] = 'x'
-  // CHECK: store i8 120, i8* %
+  // CHECK: store i8 120, ptr %
 
   struct LLP3 var = { .l3 = g3, .l3.p1 = { [0] = g1 }, .l3.p1[1].x[1] = 'x' };
 }
@@ -92,15 +91,14 @@ void test5(void)
 // CHECK-LABEL: test6
 void test6(void)
 {
-  // CHECK: [[LP:%[a-z0-9]+]] = getelementptr{{.*}}%struct.LLP2P2, %struct.LLP2P2*{{.*}}, i32 0, i32 0
-  // CHECK: call {{.*}}get456789(%struct.LP2P2* {{.*}}[[LP]])
+  // CHECK: [[LP:%[a-z0-9]+]] = getelementptr{{.*}}%struct.LLP2P2, ptr{{.*}}, i32 0, i32 0
+  // CHECK: call {{.*}}get456789(ptr {{.*}}[[LP]])
 
   // CHECK: [[CALL:%[a-z0-9]+]] = call {{.*}}@get235()
-  // CHECK: store{{.*}}[[CALL]], {{.*}}[[TMP0:%[a-z0-9]+]]
-  // CHECK: [[TMP1:%[a-z0-9]+]] = bitcast {{.*}}[[TMP0]]
-  // CHECK: call void @llvm.memcpy{{.*}}[[TMP1]], i64 12, i1 false)
+  // CHECK: store{{.*}}[[CALL]], {{.*}}[[TMP0:%[a-z0-9.]+]]
+  // CHECK: call void @llvm.memcpy{{.*}}[[TMP0]], i64 12, i1 false)
 
-  // CHECK: store i32 10, i32* %
+  // CHECK: store i32 10, ptr %
 
   struct LLP2P2 { struct LP2P2 lp; } var =  { get456789(),
                                               .lp.p1 = get235(),
diff --git a/clang/test/CodeGen/semantic-interposition.c b/clang/test/CodeGen/semantic-interposition.c
index 7ad650869a08d..243bd41d9dae8 100644
--- a/clang/test/CodeGen/semantic-interposition.c
+++ b/clang/test/CodeGen/semantic-interposition.c
@@ -1,22 +1,22 @@
 /// -fno-semantic-interposition is the default and local aliases (via dso_local) are allowed.
-// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-linux-gnu -emit-llvm -mrelocation-model pic -pic-level 1 %s -o - | FileCheck %s --check-prefixes=CHECK,NOMETADATA
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm -mrelocation-model pic -pic-level 1 %s -o - | FileCheck %s --check-prefixes=CHECK,NOMETADATA
 
 /// -fsemantic-interposition sets a module metadata.
-// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-linux-gnu -emit-llvm -mrelocation-model pic -pic-level 1 -fsemantic-interposition %s -o - | FileCheck %s --check-prefixes=PREEMPT,METADATA
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm -mrelocation-model pic -pic-level 1 -fsemantic-interposition %s -o - | FileCheck %s --check-prefixes=PREEMPT,METADATA
 
 /// Traditional half-baked behavior: interprocedural optimizations are allowed
 /// but local aliases are not used.
-// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-linux-gnu -emit-llvm -mrelocation-model pic -pic-level 1 -fhalf-no-semantic-interposition %s -o - | FileCheck %s --check-prefixes=PREEMPT,NOMETADATA
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm -mrelocation-model pic -pic-level 1 -fhalf-no-semantic-interposition %s -o - | FileCheck %s --check-prefixes=PREEMPT,NOMETADATA
 
 // CHECK: @var = global i32 0, align 4
 // CHECK: @ext_var = external global i32, align 4
-// CHECK: @ifunc = ifunc i32 (), bitcast (i8* ()* @ifunc_resolver to i32 ()* ()*)
+// CHECK: @ifunc = ifunc i32 (), ptr @ifunc_resolver
 // CHECK: define dso_local i32 @func()
 // CHECK: declare i32 @ext()
 
 // PREEMPT: @var = global i32 0, align 4
 // PREEMPT: @ext_var = external global i32, align 4
-// PREEMPT: @ifunc = ifunc i32 (), bitcast (i8* ()* @ifunc_resolver to i32 ()* ()*)
+// PREEMPT: @ifunc = ifunc i32 (), ptr @ifunc_resolver
 // PREEMPT: define i32 @func()
 // PREEMPT: declare i32 @ext()
 
diff --git a/clang/test/CodeGen/sparc-vaarg.c b/clang/test/CodeGen/sparc-vaarg.c
index 537072daf62cc..b78949e09350c 100644
--- a/clang/test/CodeGen/sparc-vaarg.c
+++ b/clang/test/CodeGen/sparc-vaarg.c
@@ -1,10 +1,10 @@
-// RUN: %clang_cc1 -no-opaque-pointers -triple sparc -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple sparc -emit-llvm -o - %s | FileCheck %s
 #include <stdarg.h>
 
 // CHECK-LABEL: define{{.*}} i32 @get_int
 // CHECK: [[RESULT:%[a-z_0-9]+]] = va_arg {{.*}}, i32{{$}}
-// CHECK: store i32 [[RESULT]], i32* [[LOC:%[a-z_0-9]+]]
-// CHECK: [[RESULT2:%[a-z_0-9]+]] = load i32, i32* [[LOC]]
+// CHECK: store i32 [[RESULT]], ptr [[LOC:%[a-z_0-9]+]]
+// CHECK: [[RESULT2:%[a-z_0-9]+]] = load i32, ptr [[LOC]]
 // CHECK: ret i32 [[RESULT2]]
 int get_int(va_list *args) {
   return va_arg(*args, int);
@@ -17,9 +17,8 @@ struct Foo {
 struct Foo dest;
 
 // CHECK-LABEL: define{{.*}} void @get_struct
-// CHECK: [[RESULT:%[a-z_0-9]+]] = va_arg {{.*}}, %struct.Foo*{{$}}
-// CHECK: [[RESULT2:%[a-z_0-9]+]] = bitcast {{.*}} [[RESULT]] to i8*
-// CHECK: call void @llvm.memcpy{{.*}}@dest{{.*}}, i8* align {{[0-9]+}} [[RESULT2]]
+// CHECK: [[RESULT:%[a-z_0-9]+]] = va_arg {{.*}}, ptr{{$}}
+// CHECK: call void @llvm.memcpy{{.*}}@dest{{.*}}, ptr align {{[0-9]+}} [[RESULT]]
 void get_struct(va_list *args) {
  dest = va_arg(*args, struct Foo);
 }
@@ -29,7 +28,7 @@ enum E { Foo_one = 1 };
 enum E enum_dest;
 
 // CHECK-LABEL: define{{.*}} void @get_enum
-// CHECK: va_arg i8** {{.*}}, i32
+// CHECK: va_arg ptr {{.*}}, i32
 void get_enum(va_list *args) {
   enum_dest = va_arg(*args, enum E);
 }
diff --git a/clang/test/CodeGen/sparcv9-dwarf.c b/clang/test/CodeGen/sparcv9-dwarf.c
index 654ce6465517b..b9d64a5adab44 100644
--- a/clang/test/CodeGen/sparcv9-dwarf.c
+++ b/clang/test/CodeGen/sparcv9-dwarf.c
@@ -1,4 +1,4 @@
-// RUN: %clang_cc1 -no-opaque-pointers -triple sparcv9-unknown-unknown -emit-llvm %s -o - | FileCheck %s
+// RUN: %clang_cc1 -triple sparcv9-unknown-unknown -emit-llvm %s -o - | FileCheck %s
 static unsigned char dwarf_reg_size_table[102+1];
 
 int test(void) {
@@ -8,92 +8,92 @@ int test(void) {
 }
 
 // CHECK-LABEL: define{{.*}} signext i32 @test()
-// CHECK:       store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 0)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 1)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 2)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 3)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 4)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 5)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 6)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 7)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 8)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 9)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 10)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 11)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 12)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 13)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 14)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 15)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 16)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 17)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 18)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 19)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 20)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 21)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 22)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 23)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 24)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 25)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 26)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 27)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 28)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 29)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 30)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 31)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 32)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 33)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 34)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 35)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 36)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 37)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 38)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 39)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 40)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 41)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 42)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 43)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 44)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 45)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 46)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 47)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 48)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 49)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 50)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 51)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 52)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 53)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 54)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 55)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 56)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 57)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 58)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 59)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 60)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 61)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 62)
-// CHECK-NEXT:  store i8 4, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 63)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 64)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 65)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 66)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 67)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 68)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 69)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 70)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 71)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 72)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 73)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 74)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 75)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 76)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 77)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 78)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 79)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 80)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 81)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 82)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 83)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 84)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 85)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 86)
-// CHECK-NEXT:  store i8 8, i8* getelementptr inbounds ([103 x i8], [103 x i8]* @dwarf_reg_size_table, i64 0, i64 87)
+// CHECK:       store i8 8, ptr @dwarf_reg_size_table
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 1), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 2), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 3), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 4), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 5), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 6), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 7), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 8), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 9), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 10), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 11), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 12), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 13), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 14), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 15), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 16), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 17), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 18), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 19), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 20), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 21), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 22), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 23), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 24), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 25), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 26), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 27), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 28), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 29), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 30), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 31), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 32), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 33), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 34), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 35), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 36), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 37), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 38), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 39), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 40), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 41), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 42), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 43), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 44), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 45), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 46), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 47), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 48), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 49), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 50), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 51), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 52), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 53), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 54), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 55), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 56), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 57), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 58), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 59), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 60), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 61), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 62), align 1
+// CHECK-NEXT: store i8 4, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 63), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 64), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 65), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 66), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 67), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 68), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 69), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 70), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 71), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 72), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 73), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 74), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 75), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 76), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 77), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 78), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 79), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 80), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 81), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 82), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 83), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 84), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 85), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 86), align 1
+// CHECK-NEXT: store i8 8, ptr getelementptr inbounds (i8, ptr @dwarf_reg_size_table, i32 87), align 1
 // CHECK-NEXT:  ret i32 14
diff --git a/clang/test/CodeGen/staticinit.c b/clang/test/CodeGen/staticinit.c
index 1a876b99f2b84..90b8fa5edb024 100644
--- a/clang/test/CodeGen/staticinit.c
+++ b/clang/test/CodeGen/staticinit.c
@@ -1,5 +1,4 @@
-// RUN: %clang_cc1 -no-opaque-pointers -triple i386-pc-linux-gnu -emit-llvm -o %t %s
-// RUN: grep "g.b = internal global i8. getelementptr" %t
+// RUN: %clang_cc1 -triple i386-pc-linux-gnu -emit-llvm -o - %s | FileCheck %s
 
 struct AStruct { 
   int i;
@@ -15,6 +14,7 @@ void f(void) {
   static struct AStruct myStruct = { 1, "two", 3.0 };
 }
 
+// CHECK: @g.b = internal global ptr @g.a
 void g(void) {
   static char a[10];
   static char *b = a;
@@ -26,7 +26,7 @@ void foo(void) {
   static struct s var = {((void*)&((char*)0)[0])};
 }
 
-// RUN: grep "f1.l0 = internal global i32 ptrtoint (i32 ()\* @f1 to i32)" %t
+// CHECK: @f1.l0 = internal global i32 ptrtoint (ptr @f1 to i32)
 int f1(void) { static int l0 = (unsigned) f1; }
 
 // PR7044
diff --git a/clang/test/CodeGen/svboolx2_t.cpp b/clang/test/CodeGen/svboolx2_t.cpp
new file mode 100644
index 0000000000000..060940d8755f5
--- /dev/null
+++ b/clang/test/CodeGen/svboolx2_t.cpp
@@ -0,0 +1,31 @@
+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -S -emit-llvm -o - %s | FileCheck %s
+
+// CHECK-LABEL: @_Z3foo10svboolx2_t(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[ARG_ADDR:%.*]] = alloca <vscale x 32 x i1>, align 2
+// CHECK-NEXT:    store <vscale x 32 x i1> [[ARG:%.*]], ptr [[ARG_ADDR]], align 2
+// CHECK-NEXT:    [[TMP0:%.*]] = load <vscale x 32 x i1>, ptr [[ARG_ADDR]], align 2
+// CHECK-NEXT:    ret <vscale x 32 x i1> [[TMP0]]
+//
+__clang_svboolx2_t foo(__clang_svboolx2_t arg) { return arg; }
+
+__clang_svboolx2_t bar();
+// CHECK-LABEL: @_Z4foo2v(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[CALL:%.*]] = call <vscale x 32 x i1> @_Z3barv()
+// CHECK-NEXT:    ret <vscale x 32 x i1> [[CALL]]
+//
+__clang_svboolx2_t foo2() { return bar(); }
+
+__clang_svboolx2_t bar2(__clang_svboolx2_t);
+// CHECK-LABEL: @_Z4foo310svboolx2_t(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[ARG_ADDR:%.*]] = alloca <vscale x 32 x i1>, align 2
+// CHECK-NEXT:    store <vscale x 32 x i1> [[ARG:%.*]], ptr [[ARG_ADDR]], align 2
+// CHECK-NEXT:    [[TMP0:%.*]] = load <vscale x 32 x i1>, ptr [[ARG_ADDR]], align 2
+// CHECK-NEXT:    [[CALL:%.*]] = call <vscale x 32 x i1> @_Z4bar210svboolx2_t(<vscale x 32 x i1> [[TMP0]])
+// CHECK-NEXT:    ret <vscale x 32 x i1> [[CALL]]
+//
+__clang_svboolx2_t foo3(__clang_svboolx2_t arg) { return bar2(arg); }
+
diff --git a/clang/test/CodeGen/svboolx4_t.cpp b/clang/test/CodeGen/svboolx4_t.cpp
new file mode 100644
index 0000000000000..8360786c06d70
--- /dev/null
+++ b/clang/test/CodeGen/svboolx4_t.cpp
@@ -0,0 +1,31 @@
+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -S -emit-llvm -o - %s | FileCheck %s
+
+// CHECK-LABEL: @_Z3foo10svboolx4_t(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[ARG_ADDR:%.*]] = alloca <vscale x 64 x i1>, align 2
+// CHECK-NEXT:    store <vscale x 64 x i1> [[ARG:%.*]], ptr [[ARG_ADDR]], align 2
+// CHECK-NEXT:    [[TMP0:%.*]] = load <vscale x 64 x i1>, ptr [[ARG_ADDR]], align 2
+// CHECK-NEXT:    ret <vscale x 64 x i1> [[TMP0]]
+//
+__clang_svboolx4_t foo(__clang_svboolx4_t arg) { return arg; }
+
+__clang_svboolx4_t bar();
+// CHECK-LABEL: @_Z4foo2v(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[CALL:%.*]] = call <vscale x 64 x i1> @_Z3barv()
+// CHECK-NEXT:    ret <vscale x 64 x i1> [[CALL]]
+//
+__clang_svboolx4_t foo2() { return bar(); }
+
+__clang_svboolx4_t bar2(__clang_svboolx4_t);
+// CHECK-LABEL: @_Z4foo310svboolx4_t(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[ARG_ADDR:%.*]] = alloca <vscale x 64 x i1>, align 2
+// CHECK-NEXT:    store <vscale x 64 x i1> [[ARG:%.*]], ptr [[ARG_ADDR]], align 2
+// CHECK-NEXT:    [[TMP0:%.*]] = load <vscale x 64 x i1>, ptr [[ARG_ADDR]], align 2
+// CHECK-NEXT:    [[CALL:%.*]] = call <vscale x 64 x i1> @_Z4bar210svboolx4_t(<vscale x 64 x i1> [[TMP0]])
+// CHECK-NEXT:    ret <vscale x 64 x i1> [[CALL]]
+//
+__clang_svboolx4_t foo3(__clang_svboolx4_t arg) { return bar2(arg); }
+
diff --git a/clang/test/CodeGen/temporary-lifetime-exceptions.cpp b/clang/test/CodeGen/temporary-lifetime-exceptions.cpp
index 4cea0e980750a..50e4a0f56cb2d 100644
--- a/clang/test/CodeGen/temporary-lifetime-exceptions.cpp
+++ b/clang/test/CodeGen/temporary-lifetime-exceptions.cpp
@@ -1,4 +1,4 @@
-// RUN: %clang_cc1 -no-opaque-pointers %s -fexceptions -fcxx-exceptions -std=c++11 -O1 -triple x86_64 -emit-llvm -o - | FileCheck %s
+// RUN: %clang_cc1 %s -fexceptions -fcxx-exceptions -std=c++11 -O1 -triple x86_64 -emit-llvm -o - | FileCheck %s
 
 // lifetime.end should be invoked even if the destructor doesn't run due to an
 // exception thrown from previous ctor call.
@@ -8,17 +8,15 @@ A Baz(const A&);
 
 void Test1() {
   // CHECK-LABEL: @_Z5Test1v(
-  // CHECK: getelementptr
-  // CHECK-NEXT: call void @llvm.lifetime.start.p0i8(i64 1, i8* nonnull [[TMP:[^ ]+]])
-  // CHECK-NEXT: getelementptr
-  // CHECK-NEXT: call void @llvm.lifetime.start.p0i8(i64 1, i8* nonnull [[TMP1:[^ ]+]])
+  // CHECK: call void @llvm.lifetime.start.p0(i64 1, ptr nonnull [[TMP:[^ ]+]])
+  // CHECK: call void @llvm.lifetime.start.p0(i64 1, ptr nonnull [[TMP1:[^ ]+]])
 
   // Normal exit
-  // CHECK: call void @llvm.lifetime.end.p0i8(i64 1, i8* nonnull [[TMP1]])
-  // CHECK-NEXT: call void @llvm.lifetime.end.p0i8(i64 1, i8* nonnull [[TMP]])
+  // CHECK: call void @llvm.lifetime.end.p0(i64 1, ptr nonnull [[TMP1]])
+  // CHECK-NEXT: call void @llvm.lifetime.end.p0(i64 1, ptr nonnull [[TMP]])
 
   // Exception exit
-  // CHECK: call void @llvm.lifetime.end.p0i8(i64 1, i8* nonnull [[TMP1]])
-  // CHECK-NEXT: call void @llvm.lifetime.end.p0i8(i64 1, i8* nonnull [[TMP]])
+  // CHECK: call void @llvm.lifetime.end.p0(i64 1, ptr nonnull [[TMP1]])
+  // CHECK-NEXT: call void @llvm.lifetime.end.p0(i64 1, ptr nonnull [[TMP]])
   Baz(Baz(A()));
 }
diff --git a/clang/test/CodeGen/union-non-trivial-member.cpp b/clang/test/CodeGen/union-non-trivial-member.cpp
new file mode 100644
index 0000000000000..fdc9fd16911e1
--- /dev/null
+++ b/clang/test/CodeGen/union-non-trivial-member.cpp
@@ -0,0 +1,37 @@
+// RUN: %clang_cc1 --std=c++17 -emit-llvm %s -o - -triple x86_64-unknown-linux-gnu | FileCheck %s
+
+struct non_trivial_constructor {
+    constexpr non_trivial_constructor() : x(100) { }
+    int x;
+};
+
+union UnionInt {
+    int a{1000};
+    non_trivial_constructor b;
+};
+
+union UnionNonTrivial {
+    int a;
+    non_trivial_constructor b{};
+};
+
+void f() {
+    UnionInt u1;
+    UnionNonTrivial u2;
+}
+
+// CHECK:      define dso_local void @_Z1fv()
+// CHECK:        call void @_ZN8UnionIntC1Ev
+// CHECK-NEXT:   call void @_ZN15UnionNonTrivialC1Ev
+
+// CHECK:      define {{.*}}void @_ZN8UnionIntC1Ev
+// CHECK:        call void @_ZN8UnionIntC2Ev
+
+// CHECK:      define {{.*}}void @_ZN15UnionNonTrivialC1Ev
+// CHECK:        call void @_ZN15UnionNonTrivialC2Ev
+
+// CHECK:      define {{.*}}void @_ZN8UnionIntC2Ev
+// CHECK:        store i32 1000
+
+// CHECK:      define {{.*}}void @_ZN15UnionNonTrivialC2Ev
+// CHECK:        call void @_ZN23non_trivial_constructorC1Ev
diff --git a/clang/test/CodeGen/union-tbaa1.c b/clang/test/CodeGen/union-tbaa1.c
index 8be8fcea21a99..f6f10b3e5b1e1 100644
--- a/clang/test/CodeGen/union-tbaa1.c
+++ b/clang/test/CodeGen/union-tbaa1.c
@@ -1,4 +1,4 @@
-// RUN: %clang_cc1 -no-opaque-pointers %s -triple hexagon-unknown-elf -O2 -emit-llvm -o - | FileCheck %s
+// RUN: %clang_cc1 %s -triple hexagon-unknown-elf -O2 -emit-llvm -o - | FileCheck %s
 
 typedef union __attribute__((aligned(4))) {
   unsigned short uh[2];
@@ -7,38 +7,43 @@ typedef union __attribute__((aligned(4))) {
 
 void bar(vect32 p[][2]);
 
-// CHECK-LABEL: define{{.*}} void @fred
+// CHECK-LABEL: define dso_local void @fred
+// CHECK-SAME: (i32 noundef [[NUM:%.*]], ptr nocapture noundef writeonly [[VEC:%.*]], ptr nocapture noundef readonly [[INDEX:%.*]], ptr nocapture noundef readonly [[ARR:%.*]]) local_unnamed_addr #[[ATTR0:[0-9]+]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP:%.*]] = alloca [4 x [2 x %union.vect32]], align 8
+// CHECK-NEXT:    call void @llvm.lifetime.start.p0(i64 32, ptr nonnull [[TMP]]) #[[ATTR3:[0-9]+]]
+// CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[INDEX]], align 4, !tbaa [[TBAA2:![0-9]+]]
+// CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [2 x i32], ptr [[ARR]], i32 [[TMP0]]
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[ARRAYIDX]], align 4, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[MUL:%.*]] = mul i32 [[TMP1]], [[NUM]]
+// CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds [4 x [2 x %union.vect32]], ptr [[TMP]], i32 0, i32 [[TMP0]]
+// CHECK-NEXT:    store i32 [[MUL]], ptr [[ARRAYIDX2]], align 8, !tbaa [[TBAA6:![0-9]+]]
+// CHECK-NEXT:    [[ARRAYIDX5:%.*]] = getelementptr inbounds [2 x i32], ptr [[ARR]], i32 [[TMP0]], i32 1
+// CHECK-NEXT:    [[TMP2:%.*]] = load i32, ptr [[ARRAYIDX5]], align 4, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[MUL6:%.*]] = mul i32 [[TMP2]], [[NUM]]
+// CHECK-NEXT:    [[ARRAYIDX8:%.*]] = getelementptr inbounds [4 x [2 x %union.vect32]], ptr [[TMP]], i32 0, i32 [[TMP0]], i32 1
+// CHECK-NEXT:    store i32 [[MUL6]], ptr [[ARRAYIDX8]], align 4, !tbaa [[TBAA6]]
+// CHECK-NEXT:    [[TMP3:%.*]] = lshr i32 [[MUL]], 16
+// CHECK-NEXT:    store i32 [[TMP3]], ptr [[VEC]], align 4, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[TMP4:%.*]] = load i32, ptr [[INDEX]], align 4, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[ARRAYIDX14:%.*]] = getelementptr inbounds [4 x [2 x %union.vect32]], ptr [[TMP]], i32 0, i32 [[TMP4]], i32 1
+// CHECK-NEXT:    [[ARRAYIDX15:%.*]] = getelementptr inbounds [2 x i16], ptr [[ARRAYIDX14]], i32 0, i32 1
+// CHECK-NEXT:    [[TMP5:%.*]] = load i16, ptr [[ARRAYIDX15]], align 2, !tbaa [[TBAA6]]
+// CHECK-NEXT:    [[CONV16:%.*]] = zext i16 [[TMP5]] to i32
+// CHECK-NEXT:    [[ARRAYIDX17:%.*]] = getelementptr inbounds i32, ptr [[VEC]], i32 1
+// CHECK-NEXT:    store i32 [[CONV16]], ptr [[ARRAYIDX17]], align 4, !tbaa [[TBAA2]]
+// CHECK-NEXT:    call void @bar(ptr noundef nonnull [[TMP]]) #[[ATTR3]]
+// CHECK-NEXT:    call void @llvm.lifetime.end.p0(i64 32, ptr nonnull [[TMP]]) #[[ATTR3]]
+// CHECK-NEXT:    ret void
+//
 void fred(unsigned Num, int Vec[2], int *Index, int Arr[4][2]) {
   vect32 Tmp[4][2];
-// Generate tbaa for the load of Index:
-// CHECK: load i32, i32* %Index{{.*}}tbaa
-// But no tbaa for the two stores:
-// CHECK: %uw[[UW1:[0-9]*]] = getelementptr
-// CHECK: store{{.*}}%uw[[UW1]]
-// CHECK: tbaa ![[OCPATH:[0-9]+]]
-// There will be a load after the store, and it will use tbaa. Make sure
-// the check-not above doesn't find it:
-// CHECK: load
   Tmp[*Index][0].uw = Arr[*Index][0] * Num;
-// CHECK: %uw[[UW2:[0-9]*]] = getelementptr
-// CHECK: store{{.*}}%uw[[UW2]]
-// CHECK: tbaa ![[OCPATH]]
   Tmp[*Index][1].uw = Arr[*Index][1] * Num;
-// Same here, don't generate tbaa for the loads:
-// CHECK: %uh[[UH1:[0-9]*]] = bitcast %union.vect32
-// CHECK: %arrayidx[[AX1:[0-9]*]] = getelementptr{{.*}}%uh[[UH1]]
-// CHECK: load i16, i16* %arrayidx[[AX1]]
-// CHECK: tbaa ![[OCPATH]]
-// CHECK: store
   Vec[0] = Tmp[*Index][0].uh[1];
-// CHECK: %uh[[UH2:[0-9]*]] = bitcast %union.vect32
-// CHECK: %arrayidx[[AX2:[0-9]*]] = getelementptr{{.*}}%uh[[UH2]]
-// CHECK: load i16, i16* %arrayidx[[AX2]]
-// CHECK: tbaa ![[OCPATH]]
-// CHECK: store
   Vec[1] = Tmp[*Index][1].uh[1];
   bar(Tmp);
 }
 
-// CHECK-DAG: ![[CHAR:[0-9]+]] = !{!"omnipotent char"
-// CHECK-DAG: ![[OCPATH]] = !{![[CHAR]], ![[CHAR]], i64 0}
+// CHECK-DAG: [[CHAR:![0-9]+]] = !{!"omnipotent char"
+// CHECK-DAG: [[TBAA6]] = !{[[CHAR]], [[CHAR]], i64 0}
diff --git a/clang/test/CodeGen/volatile.c b/clang/test/CodeGen/volatile.c
index 4fa114d375207..14e8f1f498e2c 100644
--- a/clang/test/CodeGen/volatile.c
+++ b/clang/test/CodeGen/volatile.c
@@ -1,6 +1,6 @@
-// RUN: %clang_cc1 -no-opaque-pointers -triple=aarch64-unknown-linux-gnu -emit-llvm < %s | FileCheck %s -check-prefix CHECK -check-prefixes CHECK-IT,CHECK-IT-ARM
-// RUN: %clang_cc1 -no-opaque-pointers -triple=x86_64-unknown-linux-gnu -emit-llvm < %s | FileCheck %s -check-prefix CHECK -check-prefixes CHECK-IT,CHECK-IT-OTHER
-// RUN: %clang_cc1 -no-opaque-pointers -triple=%ms_abi_triple -emit-llvm < %s | FileCheck %s -check-prefix CHECK -check-prefix CHECK-MS
+// RUN: %clang_cc1 -triple=aarch64-unknown-linux-gnu -emit-llvm < %s | FileCheck %s -check-prefix CHECK -check-prefixes CHECK-IT,CHECK-IT-ARM
+// RUN: %clang_cc1 -triple=x86_64-unknown-linux-gnu -emit-llvm < %s | FileCheck %s -check-prefix CHECK -check-prefixes CHECK-IT,CHECK-IT-OTHER
+// RUN: %clang_cc1 -triple=%ms_abi_triple -emit-llvm < %s | FileCheck %s -check-prefix CHECK -check-prefix CHECK-MS
 
 int S;
 volatile int vS;
@@ -42,177 +42,177 @@ int main(void) {
 // CHECK: [[I:%[a-zA-Z0-9_.]+]] = alloca i32
   // load
   i=S;
-// CHECK: load i32, i32* @S
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: load i32, ptr @S
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=vS;
-// CHECK: load volatile i32, i32* @vS
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: load volatile i32, ptr @vS
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=*pS;
-// CHECK: [[PS_VAL:%[a-zA-Z0-9_.]+]] = load i32*, i32** @pS
-// CHECK: load i32, i32* [[PS_VAL]]
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: [[PS_VAL:%[a-zA-Z0-9_.]+]] = load ptr, ptr @pS
+// CHECK: load i32, ptr [[PS_VAL]]
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=*pvS;
-// CHECK: [[PVS_VAL:%[a-zA-Z0-9_.]+]] = load i32*, i32** @pvS
-// CHECK: load volatile i32, i32* [[PVS_VAL]]
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: [[PVS_VAL:%[a-zA-Z0-9_.]+]] = load ptr, ptr @pvS
+// CHECK: load volatile i32, ptr [[PVS_VAL]]
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=A[2];
-// CHECK: load i32, i32* getelementptr {{.*}} @A
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: load i32, ptr getelementptr {{.*}} @A
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=vA[2];
-// CHECK: load volatile i32, i32* getelementptr {{.*}} @vA
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: load volatile i32, ptr getelementptr {{.*}} @vA
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=F.x;
-// CHECK: load i32, i32* getelementptr {{.*}} @F
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: load i32, ptr @F
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=vF.x;
-// CHECK: load volatile i32, i32* getelementptr {{.*}} @vF
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: load volatile i32, ptr @vF
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=F2.x;
-// CHECK: load i32, i32* getelementptr {{.*}} @F2
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: load i32, ptr @F2
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=vF2.x;
-// CHECK: load volatile i32, i32* getelementptr {{.*}} @vF2
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: load volatile i32, ptr @vF2
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=vpF2->x;
-// CHECK: [[VPF2_VAL:%[a-zA-Z0-9_.]+]] = load {{%[a-zA-Z0-9_.]+}}*, {{%[a-zA-Z0-9_.]+}}** @vpF2
+// CHECK: [[VPF2_VAL:%[a-zA-Z0-9_.]+]] = load ptr, ptr @vpF2
 // CHECK: [[ELT:%[a-zA-Z0-9_.]+]] = getelementptr {{.*}} [[VPF2_VAL]]
-// CHECK: load volatile i32, i32* [[ELT]]
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: load volatile i32, ptr [[ELT]]
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=F3.x.y;
-// CHECK: load i32, i32* getelementptr {{.*}} @F3
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: load i32, ptr @F3
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=vF3.x.y;
-// CHECK: load volatile i32, i32* getelementptr {{.*}} @vF3
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: load volatile i32, ptr @vF3
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=BF.x;
-// CHECK-IT: load i8, i8* getelementptr {{.*}} @BF
-// CHECK-MS: load i32, i32* getelementptr {{.*}} @BF
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK-IT: load i8, ptr @BF
+// CHECK-MS: load i32, ptr @BF
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=vBF.x;
-// CHECK-IT-OTHER: load volatile i8, i8* getelementptr {{.*}} @vBF
-// CHECK-IT-ARM: load volatile i32, i32* bitcast {{.*}} @vBF
-// CHECK-MS: load volatile i32, i32* getelementptr {{.*}} @vBF
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK-IT-OTHER: load volatile i8, ptr @vBF
+// CHECK-IT-ARM: load volatile i32, ptr @vBF
+// CHECK-MS: load volatile i32, ptr @vBF
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=V[3];
-// CHECK: load <4 x i32>, <4 x i32>* @V
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: load <4 x i32>, ptr @V
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=vV[3];
-// CHECK: load volatile <4 x i32>, <4 x i32>* @vV
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: load volatile <4 x i32>, ptr @vV
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=VE.yx[1];
-// CHECK: load <4 x i32>, <4 x i32>* @VE
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: load <4 x i32>, ptr @VE
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=vVE.zy[1];
-// CHECK: load volatile <4 x i32>, <4 x i32>* @vVE
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: load volatile <4 x i32>, ptr @vVE
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i = aggFct().x; // Note: not volatile
   // N.b. Aggregate return is extremely target specific, all we can
   // really say here is that there probably shouldn't be a volatile
   // load.
 // CHECK-NOT: load volatile
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i=vtS;
-// CHECK: load volatile i32, i32* @vtS
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: load volatile i32, ptr @vtS
+// CHECK: store i32 {{.*}}, ptr [[I]]
 
 
   // store
   S=i;
-// CHECK: load i32, i32* [[I]]
-// CHECK: store i32 {{.*}}, i32* @S
+// CHECK: load i32, ptr [[I]]
+// CHECK: store i32 {{.*}}, ptr @S
   vS=i;
-// CHECK: load i32, i32* [[I]]
-// CHECK: store volatile i32 {{.*}}, i32* @vS
+// CHECK: load i32, ptr [[I]]
+// CHECK: store volatile i32 {{.*}}, ptr @vS
   *pS=i;
-// CHECK: load i32, i32* [[I]]
-// CHECK: [[PS_VAL:%[a-zA-Z0-9_.]+]] = load i32*, i32** @pS
-// CHECK: store i32 {{.*}}, i32* [[PS_VAL]]
+// CHECK: load i32, ptr [[I]]
+// CHECK: [[PS_VAL:%[a-zA-Z0-9_.]+]] = load ptr, ptr @pS
+// CHECK: store i32 {{.*}}, ptr [[PS_VAL]]
   *pvS=i;
-// CHECK: load i32, i32* [[I]]
-// CHECK: [[PVS_VAL:%[a-zA-Z0-9_.]+]] = load i32*, i32** @pvS
-// CHECK: store volatile i32 {{.*}}, i32* [[PVS_VAL]]
+// CHECK: load i32, ptr [[I]]
+// CHECK: [[PVS_VAL:%[a-zA-Z0-9_.]+]] = load ptr, ptr @pvS
+// CHECK: store volatile i32 {{.*}}, ptr [[PVS_VAL]]
   A[2]=i;
-// CHECK: load i32, i32* [[I]]
-// CHECK: store i32 {{.*}}, i32* getelementptr {{.*}} @A
+// CHECK: load i32, ptr [[I]]
+// CHECK: store i32 {{.*}}, ptr getelementptr {{.*}} @A
   vA[2]=i;
-// CHECK: load i32, i32* [[I]]
-// CHECK: store volatile i32 {{.*}}, i32* getelementptr {{.*}} @vA
+// CHECK: load i32, ptr [[I]]
+// CHECK: store volatile i32 {{.*}}, ptr getelementptr {{.*}} @vA
   F.x=i;
-// CHECK: load i32, i32* [[I]]
-// CHECK: store i32 {{.*}}, i32* getelementptr {{.*}} @F
+// CHECK: load i32, ptr [[I]]
+// CHECK: store i32 {{.*}}, ptr @F
   vF.x=i;
-// CHECK: load i32, i32* [[I]]
-// CHECK: store volatile i32 {{.*}}, i32* getelementptr {{.*}} @vF
+// CHECK: load i32, ptr [[I]]
+// CHECK: store volatile i32 {{.*}}, ptr @vF
   F2.x=i;
-// CHECK: load i32, i32* [[I]]
-// CHECK: store i32 {{.*}}, i32* getelementptr {{.*}} @F2
+// CHECK: load i32, ptr [[I]]
+// CHECK: store i32 {{.*}}, ptr @F2
   vF2.x=i;
-// CHECK: load i32, i32* [[I]]
-// CHECK: store volatile i32 {{.*}}, i32* getelementptr {{.*}} @vF2
+// CHECK: load i32, ptr [[I]]
+// CHECK: store volatile i32 {{.*}}, ptr @vF2
   vpF2->x=i;
-// CHECK: load i32, i32* [[I]]
-// CHECK: [[VPF2_VAL:%[a-zA-Z0-9_.]+]] = load {{%[a-zA-Z0-9._]+}}*, {{%[a-zA-Z0-9._]+}}** @vpF2
+// CHECK: load i32, ptr [[I]]
+// CHECK: [[VPF2_VAL:%[a-zA-Z0-9_.]+]] = load ptr, ptr @vpF2
 // CHECK: [[ELT:%[a-zA-Z0-9_.]+]] = getelementptr {{.*}} [[VPF2_VAL]]
-// CHECK: store volatile i32 {{.*}}, i32* [[ELT]]
+// CHECK: store volatile i32 {{.*}}, ptr [[ELT]]
   vF3.x.y=i;
-// CHECK: load i32, i32* [[I]]
-// CHECK: store volatile i32 {{.*}}, i32* getelementptr {{.*}} @vF3
+// CHECK: load i32, ptr [[I]]
+// CHECK: store volatile i32 {{.*}}, ptr @vF3
   BF.x=i;
-// CHECK: load i32, i32* [[I]]
-// CHECK-IT: load i8, i8* getelementptr {{.*}} @BF
-// CHECK-MS: load i32, i32* getelementptr {{.*}} @BF
-// CHECK-IT: store i8 {{.*}}, i8* getelementptr {{.*}} @BF
-// CHECK-MS: store i32 {{.*}}, i32* getelementptr {{.*}} @BF
+// CHECK: load i32, ptr [[I]]
+// CHECK-IT: load i8, ptr @BF
+// CHECK-MS: load i32, ptr @BF
+// CHECK-IT: store i8 {{.*}}, ptr @BF
+// CHECK-MS: store i32 {{.*}}, ptr @BF
   vBF.x=i;
-// CHECK: load i32, i32* [[I]]
-// CHECK-IT-OTHER: load volatile i8, i8* getelementptr {{.*}} @vBF
-// CHECK-IT-ARM: load volatile i32, i32* bitcast {{.*}} @vBF
-// CHECK-MS: load volatile i32, i32* getelementptr {{.*}} @vBF
-// CHECK-IT-OTHER: store volatile i8 {{.*}}, i8* getelementptr {{.*}} @vBF
-// CHECK-IT-ARM: store volatile i32 {{.*}}, i32* bitcast {{.*}} @vBF
-// CHECK-MS: store volatile i32 {{.*}}, i32* getelementptr {{.*}} @vBF
+// CHECK: load i32, ptr [[I]]
+// CHECK-IT-OTHER: load volatile i8, ptr @vBF
+// CHECK-IT-ARM: load volatile i32, ptr @vBF
+// CHECK-MS: load volatile i32, ptr @vBF
+// CHECK-IT-OTHER: store volatile i8 {{.*}}, ptr @vBF
+// CHECK-IT-ARM: store volatile i32 {{.*}}, ptr @vBF
+// CHECK-MS: store volatile i32 {{.*}}, ptr @vBF
   V[3]=i;
-// CHECK: load i32, i32* [[I]]
-// CHECK: load <4 x i32>, <4 x i32>* @V
-// CHECK: store <4 x i32> {{.*}}, <4 x i32>* @V
+// CHECK: load i32, ptr [[I]]
+// CHECK: load <4 x i32>, ptr @V
+// CHECK: store <4 x i32> {{.*}}, ptr @V
   vV[3]=i;
-// CHECK: load i32, i32* [[I]]
-// CHECK: load volatile <4 x i32>, <4 x i32>* @vV
-// CHECK: store volatile <4 x i32> {{.*}}, <4 x i32>* @vV
+// CHECK: load i32, ptr [[I]]
+// CHECK: load volatile <4 x i32>, ptr @vV
+// CHECK: store volatile <4 x i32> {{.*}}, ptr @vV
   vtS=i;
-// CHECK: load i32, i32* [[I]]
-// CHECK: store volatile i32 {{.*}}, i32* @vtS
+// CHECK: load i32, ptr [[I]]
+// CHECK: store volatile i32 {{.*}}, ptr @vtS
 
   // other ops:
   ++S;
-// CHECK: load i32, i32* @S
-// CHECK: store i32 {{.*}}, i32* @S
+// CHECK: load i32, ptr @S
+// CHECK: store i32 {{.*}}, ptr @S
   ++vS;
-// CHECK: load volatile i32, i32* @vS
-// CHECK: store volatile i32 {{.*}}, i32* @vS
+// CHECK: load volatile i32, ptr @vS
+// CHECK: store volatile i32 {{.*}}, ptr @vS
   i+=S;
-// CHECK: load i32, i32* @S
-// CHECK: load i32, i32* [[I]]
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: load i32, ptr @S
+// CHECK: load i32, ptr [[I]]
+// CHECK: store i32 {{.*}}, ptr [[I]]
   i+=vS;
-// CHECK: load volatile i32, i32* @vS
-// CHECK: load i32, i32* [[I]]
-// CHECK: store i32 {{.*}}, i32* [[I]]
+// CHECK: load volatile i32, ptr @vS
+// CHECK: load i32, ptr [[I]]
+// CHECK: store i32 {{.*}}, ptr [[I]]
   ++vtS;
-// CHECK: load volatile i32, i32* @vtS
-// CHECK: store volatile i32 {{.*}}, i32* @vtS
+// CHECK: load volatile i32, ptr @vtS
+// CHECK: store volatile i32 {{.*}}, ptr @vtS
   (void)vF2;
   // From vF2 to a temporary
-// CHECK: call void @llvm.memcpy.{{.*}}(i8* align {{[0-9]+}} %{{.*}}, i8* {{.*}} @vF2 {{.*}}, i1 true)
+// CHECK: call void @llvm.memcpy.{{.*}}(ptr align {{[0-9]+}} %{{.*}}, ptr {{.*}} @vF2, {{.*}}, i1 true)
   vF2 = vF2;
   // vF2 to itself
-// CHECK: call void @llvm.memcpy.{{.*}}(i8* {{.*@vF2.*}}, i8* {{.*@vF2.*}}, i1 true)
+// CHECK: call void @llvm.memcpy.{{.*}}(ptr {{.*@vF2.*}}, ptr {{.*@vF2.*}}, i1 true)
   vF2 = vF2 = vF2;
   // vF2 to itself twice
-// CHECK: call void @llvm.memcpy.{{.*}}(i8* {{.*@vF2.*}}, i8* {{.*@vF2.*}}, i1 true)
-// CHECK: call void @llvm.memcpy.{{.*}}(i8* {{.*@vF2.*}}, i8* {{.*@vF2.*}}, i1 true)
+// CHECK: call void @llvm.memcpy.{{.*}}(ptr {{.*@vF2.*}}, ptr {{.*@vF2.*}}, i1 true)
+// CHECK: call void @llvm.memcpy.{{.*}}(ptr {{.*@vF2.*}}, ptr {{.*@vF2.*}}, i1 true)
   vF2 = (vF2, vF2);
   // vF2 to a temporary, then vF2 to itself
-// CHECK: call void @llvm.memcpy.{{.*}}(i8* align {{[0-9]+}} %{{.*}}, i8* {{.*@vF2.*}}, i1 true)
-// CHECK: call void @llvm.memcpy.{{.*}}(i8* {{.*@vF2.*}}, i8* {{.*@vF2.*}}, i1 true)
+// CHECK: call void @llvm.memcpy.{{.*}}(ptr align {{[0-9]+}} %{{.*}}, ptr {{.*@vF2.*}}, i1 true)
+// CHECK: call void @llvm.memcpy.{{.*}}(ptr {{.*@vF2.*}}, ptr {{.*@vF2.*}}, i1 true)
 }
diff --git a/clang/test/CodeGen/windows-swiftcall.c b/clang/test/CodeGen/windows-swiftcall.c
index 0aa93fae0581d..3e5c8a4d4b9d7 100644
--- a/clang/test/CodeGen/windows-swiftcall.c
+++ b/clang/test/CodeGen/windows-swiftcall.c
@@ -1,4 +1,4 @@
-// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-unknown-windows -emit-llvm -target-cpu core2 -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple x86_64-unknown-windows -emit-llvm -target-cpu core2 -o - %s | FileCheck %s
 
 #define SWIFTCALL __attribute__((swiftcall))
 #define OUT __attribute__((swift_indirect_result))
@@ -12,29 +12,29 @@
 /*****************************************************************************/
 
 SWIFTCALL void indirect_result_1(OUT int *arg0, OUT float *arg1) {}
-// CHECK-LABEL: define {{.*}} void @indirect_result_1(i32* noalias noundef sret(i32*) align 4 dereferenceable(4){{.*}}, float* noalias noundef align 4 dereferenceable(4){{.*}})
+// CHECK-LABEL: define {{.*}} void @indirect_result_1(ptr noalias noundef sret(ptr) align 4 dereferenceable(4){{.*}}, ptr noalias noundef align 4 dereferenceable(4){{.*}})
 
 // TODO: maybe this shouldn't suppress sret.
 SWIFTCALL int indirect_result_2(OUT int *arg0, OUT float *arg1) {  __builtin_unreachable(); }
-// CHECK-LABEL: define {{.*}} i32 @indirect_result_2(i32* noalias noundef align 4 dereferenceable(4){{.*}}, float* noalias noundef align 4 dereferenceable(4){{.*}})
+// CHECK-LABEL: define {{.*}} i32 @indirect_result_2(ptr noalias noundef align 4 dereferenceable(4){{.*}}, ptr noalias noundef align 4 dereferenceable(4){{.*}})
 
 typedef struct { char array[1024]; } struct_reallybig;
 SWIFTCALL struct_reallybig indirect_result_3(OUT int *arg0, OUT float *arg1) { __builtin_unreachable(); }
-// CHECK-LABEL: define {{.*}} void @indirect_result_3({{.*}}* noalias sret({{.*}}) {{.*}}, i32* noalias noundef align 4 dereferenceable(4){{.*}}, float* noalias noundef align 4 dereferenceable(4){{.*}})
+// CHECK-LABEL: define {{.*}} void @indirect_result_3(ptr noalias sret({{.*}}) {{.*}}, ptr noalias noundef align 4 dereferenceable(4){{.*}}, ptr noalias noundef align 4 dereferenceable(4){{.*}})
 
 SWIFTCALL void context_1(CONTEXT void *self) {}
-// CHECK-LABEL: define {{.*}} void @context_1(i8* noundef swiftself
+// CHECK-LABEL: define {{.*}} void @context_1(ptr noundef swiftself
 
 SWIFTCALL void context_2(void *arg0, CONTEXT void *self) {}
-// CHECK-LABEL: define {{.*}} void @context_2(i8*{{.*}}, i8* noundef swiftself
+// CHECK-LABEL: define {{.*}} void @context_2(ptr{{.*}}, ptr noundef swiftself
 
 SWIFTCALL void context_error_1(CONTEXT int *self, ERROR float **error) {}
-// CHECK-LABEL: define {{.*}} void @context_error_1(i32* noundef swiftself{{.*}}, float** noundef swifterror %0)
-// CHECK:       [[TEMP:%.*]] = alloca float*, align 8
-// CHECK:       [[T0:%.*]] = load float*, float** [[ERRORARG:%.*]], align 8
-// CHECK:       store float* [[T0]], float** [[TEMP]], align 8
-// CHECK:       [[T0:%.*]] = load float*, float** [[TEMP]], align 8
-// CHECK:       store float* [[T0]], float** [[ERRORARG]], align 8
+// CHECK-LABEL: define {{.*}} void @context_error_1(ptr noundef swiftself{{.*}}, ptr noundef swifterror %0)
+// CHECK:       [[TEMP:%.*]] = alloca ptr, align 8
+// CHECK:       [[T0:%.*]] = load ptr, ptr [[ERRORARG:%.*]], align 8
+// CHECK:       store ptr [[T0]], ptr [[TEMP]], align 8
+// CHECK:       [[T0:%.*]] = load ptr, ptr [[TEMP]], align 8
+// CHECK:       store ptr [[T0]], ptr [[ERRORARG]], align 8
 void test_context_error_1(void) {
   int x;
   float *error;
@@ -42,16 +42,16 @@ void test_context_error_1(void) {
 }
 // CHECK-LABEL: define dso_local void @test_context_error_1()
 // CHECK:       [[X:%.*]] = alloca i32, align 4
-// CHECK:       [[ERROR:%.*]] = alloca float*, align 8
-// CHECK:       [[TEMP:%.*]] = alloca swifterror float*, align 8
-// CHECK:       [[T0:%.*]] = load float*, float** [[ERROR]], align 8
-// CHECK:       store float* [[T0]], float** [[TEMP]], align 8
-// CHECK:       call [[SWIFTCC:swiftcc]] void @context_error_1(i32* noundef swiftself [[X]], float** noundef swifterror [[TEMP]])
-// CHECK:       [[T0:%.*]] = load float*, float** [[TEMP]], align 8
-// CHECK:       store float* [[T0]], float** [[ERROR]], align 8
+// CHECK:       [[ERROR:%.*]] = alloca ptr, align 8
+// CHECK:       [[TEMP:%.*]] = alloca swifterror ptr, align 8
+// CHECK:       [[T0:%.*]] = load ptr, ptr [[ERROR]], align 8
+// CHECK:       store ptr [[T0]], ptr [[TEMP]], align 8
+// CHECK:       call [[SWIFTCC:swiftcc]] void @context_error_1(ptr noundef swiftself [[X]], ptr noundef swifterror [[TEMP]])
+// CHECK:       [[T0:%.*]] = load ptr, ptr [[TEMP]], align 8
+// CHECK:       store ptr [[T0]], ptr [[ERROR]], align 8
 
 SWIFTCALL void context_error_2(short s, CONTEXT int *self, ERROR float **error) {}
-// CHECK-LABEL: define {{.*}} void @context_error_2(i16{{.*}}, i32* noundef swiftself{{.*}}, float** noundef swifterror %0)
+// CHECK-LABEL: define {{.*}} void @context_error_2(i16{{.*}}, ptr noundef swiftself{{.*}}, ptr noundef swifterror %0)
 
 /*****************************************************************************/
 /********************************** LOWERING *********************************/
@@ -100,39 +100,35 @@ TEST(struct_1);
 // CHECK-LABEL: define dso_local swiftcc { i64, i64 } @return_struct_1() {{.*}}{
 // CHECK:   [[RET:%.*]] = alloca [[STRUCT1:%.*]], align 4
 // CHECK:   call void @llvm.memset
-// CHECK:   [[CAST:%.*]] = bitcast [[STRUCT1]]* %retval to { i64, i64 }*
-// CHECK:   [[GEP0:%.*]] = getelementptr inbounds { i64, i64 }, { i64, i64 }* [[CAST]], i32 0, i32 0
-// CHECK:   [[T0:%.*]] = load i64, i64* [[GEP0]], align 4
-// CHECK:   [[GEP1:%.*]] = getelementptr inbounds { i64, i64 }, { i64, i64 }* [[CAST]], i32 0, i32 1
-// CHECK:   [[T1:%.*]] = load i64, i64* [[GEP1]], align 4
+// CHECK:   [[GEP0:%.*]] = getelementptr inbounds { i64, i64 }, ptr %retval, i32 0, i32 0
+// CHECK:   [[T0:%.*]] = load i64, ptr [[GEP0]], align 4
+// CHECK:   [[GEP1:%.*]] = getelementptr inbounds { i64, i64 }, ptr %retval, i32 0, i32 1
+// CHECK:   [[T1:%.*]] = load i64, ptr [[GEP1]], align 4
 // CHECK:   [[R0:%.*]] = insertvalue { i64, i64 } poison, i64 [[T0]], 0
 // CHECK:   [[R1:%.*]] = insertvalue { i64, i64 } [[R0]], i64 [[T1]], 1
 // CHECK:   ret { i64, i64 } [[R1]]
 // CHECK: }
 // CHECK-LABEL: define dso_local swiftcc void @take_struct_1(i64 %0, i64 %1) {{.*}}{
 // CHECK:   [[V:%.*]] = alloca [[STRUCT1:%.*]], align 4
-// CHECK:   [[CAST:%.*]] = bitcast [[STRUCT1]]* [[V]] to { i64, i64 }*
-// CHECK:   [[GEP0:%.*]] = getelementptr inbounds { i64, i64 }, { i64, i64 }* [[CAST]], i32 0, i32 0
-// CHECK:   store i64 %0, i64* [[GEP0]], align 4
-// CHECK:   [[GEP1:%.*]] = getelementptr inbounds { i64, i64 }, { i64, i64 }* [[CAST]], i32 0, i32 1
-// CHECK:   store i64 %1, i64* [[GEP1]], align 4
+// CHECK:   [[GEP0:%.*]] = getelementptr inbounds { i64, i64 }, ptr [[V]], i32 0, i32 0
+// CHECK:   store i64 %0, ptr [[GEP0]], align 4
+// CHECK:   [[GEP1:%.*]] = getelementptr inbounds { i64, i64 }, ptr [[V]], i32 0, i32 1
+// CHECK:   store i64 %1, ptr [[GEP1]], align 4
 // CHECK:   ret void
 // CHECK: }
 // CHECK-LABEL: define dso_local void @test_struct_1() {{.*}}{
 // CHECK:   [[AGG:%.*]] = alloca [[STRUCT1:%.*]], align 4
 // CHECK:   [[RET:%.*]] = call swiftcc { i64, i64 } @return_struct_1()
-// CHECK:   [[CAST:%.*]] = bitcast [[STRUCT1]]* [[AGG]] to { i64, i64 }*
-// CHECK:   [[GEP0:%.*]] = getelementptr inbounds { i64, i64 }, { i64, i64 }* [[CAST]], i32 0, i32 0
+// CHECK:   [[GEP0:%.*]] = getelementptr inbounds { i64, i64 }, ptr [[AGG]], i32 0, i32 0
 // CHECK:   [[E0:%.*]] = extractvalue { i64, i64 } [[RET]], 0
-// CHECK:   store i64 [[E0]], i64* [[GEP0]], align 4
-// CHECK:   [[GEP1:%.*]] = getelementptr inbounds { i64, i64 }, { i64, i64 }* [[CAST]], i32 0, i32 1
+// CHECK:   store i64 [[E0]], ptr [[GEP0]], align 4
+// CHECK:   [[GEP1:%.*]] = getelementptr inbounds { i64, i64 }, ptr [[AGG]], i32 0, i32 1
 // CHECK:   [[E1:%.*]] = extractvalue { i64, i64 } [[RET]], 1
-// CHECK:   store i64 [[E1]], i64* [[GEP1]], align 4
-// CHECK:   [[CAST2:%.*]] = bitcast [[STRUCT1]]* [[AGG]] to { i64, i64 }*
-// CHECK:   [[GEP2:%.*]] = getelementptr inbounds { i64, i64 }, { i64, i64 }* [[CAST2]], i32 0, i32 0
-// CHECK:   [[V0:%.*]] = load i64, i64* [[GEP2]], align 4
-// CHECK:   [[GEP3:%.*]] = getelementptr inbounds { i64, i64 }, { i64, i64 }* [[CAST2]], i32 0, i32 1
-// CHECK:   [[V1:%.*]] = load i64, i64* [[GEP3]], align 4
+// CHECK:   store i64 [[E1]], ptr [[GEP1]], align 4
+// CHECK:   [[GEP2:%.*]] = getelementptr inbounds { i64, i64 }, ptr [[AGG]], i32 0, i32 0
+// CHECK:   [[V0:%.*]] = load i64, ptr [[GEP2]], align 4
+// CHECK:   [[GEP3:%.*]] = getelementptr inbounds { i64, i64 }, ptr [[AGG]], i32 0, i32 1
+// CHECK:   [[V1:%.*]] = load i64, ptr [[GEP3]], align 4
 // CHECK:   call swiftcc void @take_struct_1(i64 [[V0]], i64 [[V1]])
 // CHECK:   ret void
 // CHECK: }
@@ -147,41 +143,36 @@ typedef struct {
 TEST(struct_2);
 // CHECK-LABEL: define dso_local swiftcc { i64, i64 } @return_struct_2() {{.*}}{
 // CHECK:   [[RET:%.*]] = alloca [[STRUCT2_TYPE]], align 4
-// CHECK:   [[CASTVAR:%.*]] = bitcast {{.*}} [[RET]]
-// CHECK:   call void @llvm.memcpy{{.*}}({{.*}}[[CASTVAR]], {{.*}}[[STRUCT2_RESULT]]
-// CHECK:   [[CAST:%.*]] = bitcast [[STRUCT2_TYPE]]* [[RET]] to { i64, i64 }*
-// CHECK:   [[GEP0:%.*]] = getelementptr inbounds { i64, i64 }, { i64, i64 }* [[CAST]], i32 0, i32 0
-// CHECK:   [[T0:%.*]] = load i64, i64* [[GEP0]], align 4
-// CHECK:   [[GEP1:%.*]] = getelementptr inbounds { i64, i64 }, { i64, i64 }* [[CAST]], i32 0, i32 1
-// CHECK:   [[T1:%.*]] = load i64, i64* [[GEP1]], align 4
+// CHECK:   call void @llvm.memcpy{{.*}}({{.*}}[[RET]], {{.*}}[[STRUCT2_RESULT]]
+// CHECK:   [[GEP0:%.*]] = getelementptr inbounds { i64, i64 }, ptr [[RET]], i32 0, i32 0
+// CHECK:   [[T0:%.*]] = load i64, ptr [[GEP0]], align 4
+// CHECK:   [[GEP1:%.*]] = getelementptr inbounds { i64, i64 }, ptr [[RET]], i32 0, i32 1
+// CHECK:   [[T1:%.*]] = load i64, ptr [[GEP1]], align 4
 // CHECK:   [[R0:%.*]] = insertvalue { i64, i64 } poison, i64 [[T0]], 0
 // CHECK:   [[R1:%.*]] = insertvalue { i64, i64 } [[R0]], i64 [[T1]], 1
 // CHECK:   ret { i64, i64 } [[R1]]
 // CHECK: }
 // CHECK-LABEL: define dso_local swiftcc void @take_struct_2(i64 %0, i64 %1) {{.*}}{
 // CHECK:   [[V:%.*]] = alloca [[STRUCT:%.*]], align 4
-// CHECK:   [[CAST:%.*]] = bitcast [[STRUCT]]* [[V]] to { i64, i64 }*
-// CHECK:   [[GEP0:%.*]] = getelementptr inbounds { i64, i64 }, { i64, i64 }* [[CAST]], i32 0, i32 0
-// CHECK:   store i64 %0, i64* [[GEP0]], align 4
-// CHECK:   [[GEP1:%.*]] = getelementptr inbounds { i64, i64 }, { i64, i64 }* [[CAST]], i32 0, i32 1
-// CHECK:   store i64 %1, i64* [[GEP1]], align 4
+// CHECK:   [[GEP0:%.*]] = getelementptr inbounds { i64, i64 }, ptr [[V]], i32 0, i32 0
+// CHECK:   store i64 %0, ptr [[GEP0]], align 4
+// CHECK:   [[GEP1:%.*]] = getelementptr inbounds { i64, i64 }, ptr [[V]], i32 0, i32 1
+// CHECK:   store i64 %1, ptr [[GEP1]], align 4
 // CHECK:   ret void
 // CHECK: }
 // CHECK-LABEL: define dso_local void @test_struct_2() {{.*}} {
 // CHECK:   [[TMP:%.*]] = alloca [[STRUCT2_TYPE]], align 4
 // CHECK:   [[CALL:%.*]] = call swiftcc { i64, i64 } @return_struct_2()
-// CHECK:   [[CAST_TMP:%.*]] = bitcast [[STRUCT2_TYPE]]* [[TMP]] to { i64, i64 }*
-// CHECK:   [[GEP:%.*]] = getelementptr inbounds {{.*}} [[CAST_TMP]], i32 0, i32 0
+// CHECK:   [[GEP:%.*]] = getelementptr inbounds {{.*}} [[TMP]], i32 0, i32 0
 // CHECK:   [[T0:%.*]] = extractvalue { i64, i64 } [[CALL]], 0
-// CHECK:   store i64 [[T0]], i64* [[GEP]], align 4
-// CHECK:   [[GEP:%.*]] = getelementptr inbounds {{.*}} [[CAST_TMP]], i32 0, i32 1
+// CHECK:   store i64 [[T0]], ptr [[GEP]], align 4
+// CHECK:   [[GEP:%.*]] = getelementptr inbounds {{.*}} [[TMP]], i32 0, i32 1
 // CHECK:   [[T0:%.*]] = extractvalue { i64, i64 } [[CALL]], 1
-// CHECK:   store i64 [[T0]], i64* [[GEP]], align 4
-// CHECK:   [[CAST:%.*]] = bitcast [[STRUCT2_TYPE]]* [[TMP]] to { i64, i64 }*
-// CHECK:   [[GEP:%.*]] = getelementptr inbounds { i64, i64 }, { i64, i64 }* [[CAST]], i32 0, i32 0
-// CHECK:   [[R0:%.*]] = load i64, i64* [[GEP]], align 4
-// CHECK:   [[GEP:%.*]] = getelementptr inbounds { i64, i64 }, { i64, i64 }* [[CAST]], i32 0, i32 1
-// CHECK:   [[R1:%.*]] = load i64, i64* [[GEP]], align 4
+// CHECK:   store i64 [[T0]], ptr [[GEP]], align 4
+// CHECK:   [[GEP:%.*]] = getelementptr inbounds { i64, i64 }, ptr [[TMP]], i32 0, i32 0
+// CHECK:   [[R0:%.*]] = load i64, ptr [[GEP]], align 4
+// CHECK:   [[GEP:%.*]] = getelementptr inbounds { i64, i64 }, ptr [[TMP]], i32 0, i32 1
+// CHECK:   [[R1:%.*]] = load i64, ptr [[GEP]], align 4
 // CHECK:   call swiftcc void @take_struct_2(i64 [[R0]], i64 [[R1]])
 // CHECK:   ret void
 // CHECK: }
@@ -199,29 +190,24 @@ typedef struct {
 TEST(struct_misaligned_1)
 // CHECK-LABEL: define dso_local swiftcc i64 @return_struct_misaligned_1()
 // CHECK:  [[RET:%.*]] = alloca [[STRUCT:%.*]], align 1
-// CHECK:  [[CAST:%.*]] = bitcast [[STRUCT]]* [[RET]] to i8*
-// CHECK:  call void @llvm.memset{{.*}}(i8* align 1 [[CAST]], i8 0, i64 5
-// CHECK:  [[CAST:%.*]] = bitcast [[STRUCT]]* [[RET]] to { i64 }*
-// CHECK:  [[GEP:%.*]] = getelementptr inbounds { i64 }, { i64 }* [[CAST]], i32 0, i32 0
-// CHECK:  [[R0:%.*]] = load i64, i64* [[GEP]], align 1
+// CHECK:  call void @llvm.memset{{.*}}(ptr align 1 [[RET]], i8 0, i64 5
+// CHECK:  [[GEP:%.*]] = getelementptr inbounds { i64 }, ptr [[RET]], i32 0, i32 0
+// CHECK:  [[R0:%.*]] = load i64, ptr [[GEP]], align 1
 // CHECK:  ret i64 [[R0]]
 // CHECK:}
 // CHECK-LABEL: define dso_local swiftcc void @take_struct_misaligned_1(i64 %0) {{.*}}{
 // CHECK:   [[V:%.*]] = alloca [[STRUCT:%.*]], align 1
-// CHECK:   [[CAST:%.*]] = bitcast [[STRUCT]]* [[V]] to { i64 }*
-// CHECK:   [[GEP:%.*]] = getelementptr inbounds { i64 }, { i64 }* [[CAST]], i32 0, i32 0
-// CHECK:   store i64 %0, i64* [[GEP]], align 1
+// CHECK:   [[GEP:%.*]] = getelementptr inbounds { i64 }, ptr [[V]], i32 0, i32 0
+// CHECK:   store i64 %0, ptr [[GEP]], align 1
 // CHECK:   ret void
 // CHECK: }
 // CHECK: define dso_local void @test_struct_misaligned_1() {{.*}}{
 // CHECK:   [[AGG:%.*]] = alloca [[STRUCT:%.*]], align 1
 // CHECK:   [[CALL:%.*]] = call swiftcc i64 @return_struct_misaligned_1()
-// CHECK:   [[T0:%.*]] = bitcast [[STRUCT]]* [[AGG]] to { i64 }*
-// CHECK:   [[T1:%.*]] = getelementptr inbounds { i64 }, { i64 }* [[T0]], i32 0, i32 0
-// CHECK:   store i64 [[CALL]], i64* [[T1]], align 1
-// CHECK:   [[T0:%.*]] = bitcast [[STRUCT]]* [[AGG]] to { i64 }*
-// CHECK:   [[T1:%.*]] = getelementptr inbounds { i64 }, { i64 }* [[T0]], i32 0, i32 0
-// CHECK:   [[P:%.*]] = load i64, i64* [[T1]], align 1
+// CHECK:   [[T1:%.*]] = getelementptr inbounds { i64 }, ptr [[AGG]], i32 0, i32 0
+// CHECK:   store i64 [[CALL]], ptr [[T1]], align 1
+// CHECK:   [[T1:%.*]] = getelementptr inbounds { i64 }, ptr [[AGG]], i32 0, i32 0
+// CHECK:   [[P:%.*]] = load i64, ptr [[T1]], align 1
 // CHECK:   call swiftcc void @take_struct_misaligned_1(i64 [[P]])
 // CHECK:   ret void
 // CHECK: }
@@ -235,7 +221,7 @@ TEST(struct_big_1)
 // CHECK-LABEL: define {{.*}} void @return_struct_big_1({{.*}} noalias sret
 
 // Should not be byval.
-// CHECK-LABEL: define {{.*}} void @take_struct_big_1({{.*}}* noundef{{( %.*)?}})
+// CHECK-LABEL: define {{.*}} void @take_struct_big_1(ptr noundef{{( %.*)?}})
 
 /*****************************************************************************/
 /********************************* TYPE MERGING ******************************/
@@ -248,28 +234,23 @@ typedef union {
 TEST(union_het_fp)
 // CHECK-LABEL: define dso_local swiftcc i64 @return_union_het_fp()
 // CHECK:  [[RET:%.*]] = alloca [[UNION:%.*]], align 8
-// CHECK:  [[CAST:%.*]] = bitcast [[UNION]]* [[RET]] to i8*
-// CHECK:  call void @llvm.memcpy{{.*}}(i8* align {{[0-9]+}} [[CAST]]
-// CHECK:  [[CAST:%.*]] = bitcast [[UNION]]* [[RET]] to { i64 }*
-// CHECK:  [[GEP:%.*]] = getelementptr inbounds { i64 }, { i64 }* [[CAST]], i32 0, i32 0
-// CHECK:  [[R0:%.*]] = load i64, i64* [[GEP]], align 8
+// CHECK:  call void @llvm.memcpy{{.*}}(ptr align {{[0-9]+}} [[RET]]
+// CHECK:  [[GEP:%.*]] = getelementptr inbounds { i64 }, ptr [[RET]], i32 0, i32 0
+// CHECK:  [[R0:%.*]] = load i64, ptr [[GEP]], align 8
 // CHECK:  ret i64 [[R0]]
 // CHECK-LABEL: define dso_local swiftcc void @take_union_het_fp(i64 %0) {{.*}}{
 // CHECK:   [[V:%.*]] = alloca [[UNION:%.*]], align 8
-// CHECK:   [[CAST:%.*]] = bitcast [[UNION]]* [[V]] to { i64 }*
-// CHECK:   [[GEP:%.*]] = getelementptr inbounds { i64 }, { i64 }* [[CAST]], i32 0, i32 0
-// CHECK:   store i64 %0, i64* [[GEP]], align 8
+// CHECK:   [[GEP:%.*]] = getelementptr inbounds { i64 }, ptr [[V]], i32 0, i32 0
+// CHECK:   store i64 %0, ptr [[GEP]], align 8
 // CHECK:   ret void
 // CHECK: }
 // CHECK-LABEL: define dso_local void @test_union_het_fp() {{.*}}{
 // CHECK:   [[AGG:%.*]] = alloca [[UNION:%.*]], align 8
 // CHECK:   [[CALL:%.*]] = call swiftcc i64 @return_union_het_fp()
-// CHECK:   [[T0:%.*]] = bitcast [[UNION]]* [[AGG]] to { i64 }*
-// CHECK:   [[T1:%.*]] = getelementptr inbounds { i64 }, { i64 }* [[T0]], i32 0, i32 0
-// CHECK:   store i64 [[CALL]], i64* [[T1]], align 8
-// CHECK:   [[T0:%.*]] = bitcast [[UNION]]* [[AGG]] to { i64 }*
-// CHECK:   [[T1:%.*]] = getelementptr inbounds { i64 }, { i64 }* [[T0]], i32 0, i32 0
-// CHECK:   [[V0:%.*]] = load i64, i64* [[T1]], align 8
+// CHECK:   [[T1:%.*]] = getelementptr inbounds { i64 }, ptr [[AGG]], i32 0, i32 0
+// CHECK:   store i64 [[CALL]], ptr [[T1]], align 8
+// CHECK:   [[T1:%.*]] = getelementptr inbounds { i64 }, ptr [[AGG]], i32 0, i32 0
+// CHECK:   [[V0:%.*]] = load i64, ptr [[T1]], align 8
 // CHECK:   call swiftcc void @take_union_het_fp(i64 [[V0]])
 // CHECK:   ret void
 // CHECK: }
@@ -283,12 +264,10 @@ TEST(union_hom_fp)
 // CHECK-LABEL: define dso_local void @test_union_hom_fp()
 // CHECK:   [[TMP:%.*]] = alloca [[REC:%.*]], align 4
 // CHECK:   [[CALL:%.*]] = call [[SWIFTCC]] float @return_union_hom_fp()
-// CHECK:   [[CAST_TMP:%.*]] = bitcast [[REC]]* [[TMP]] to [[AGG:{ float }]]*
-// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], [[AGG]]* [[CAST_TMP]], i32 0, i32 0
-// CHECK:   store float [[CALL]], float* [[T0]], align 4
-// CHECK:   [[CAST_TMP:%.*]] = bitcast [[REC]]* [[TMP]] to [[AGG]]*
-// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], [[AGG]]* [[CAST_TMP]], i32 0, i32 0
-// CHECK:   [[FIRST:%.*]] = load float, float* [[T0]], align 4
+// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG:{ float }]], ptr [[TMP]], i32 0, i32 0
+// CHECK:   store float [[CALL]], ptr [[T0]], align 4
+// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], ptr [[TMP]], i32 0, i32 0
+// CHECK:   [[FIRST:%.*]] = load float, ptr [[T0]], align 4
 // CHECK:   call [[SWIFTCC]] void @take_union_hom_fp(float [[FIRST]])
 // CHECK:   ret void
 
@@ -300,28 +279,26 @@ TEST(union_hom_fp_partial)
 // CHECK: define dso_local void @test_union_hom_fp_partial()
 // CHECK:   [[AGG:%.*]] = alloca [[UNION:%.*]], align 16
 // CHECK:   [[CALL:%.*]] = call swiftcc { float, float, float, float } @return_union_hom_fp_partial()
-// CHECK:   [[CAST:%.*]] = bitcast [[UNION]]* [[AGG]] to { float, float, float, float }*
-// CHECK:   [[T0:%.*]] = getelementptr inbounds { float, float, float, float }, { float, float, float, float }* [[CAST]], i32 0, i32 0
+// CHECK:   [[T0:%.*]] = getelementptr inbounds { float, float, float, float }, ptr [[AGG]], i32 0, i32 0
 // CHECK:   [[T1:%.*]] = extractvalue { float, float, float, float } [[CALL]], 0
-// CHECK:   store float [[T1]], float* [[T0]], align 16
-// CHECK:   [[T0:%.*]] = getelementptr inbounds { float, float, float, float }, { float, float, float, float }* [[CAST]], i32 0, i32 1
+// CHECK:   store float [[T1]], ptr [[T0]], align 16
+// CHECK:   [[T0:%.*]] = getelementptr inbounds { float, float, float, float }, ptr [[AGG]], i32 0, i32 1
 // CHECK:   [[T1:%.*]] = extractvalue { float, float, float, float } [[CALL]], 1
-// CHECK:   store float [[T1]], float* [[T0]], align 4
-// CHECK:   [[T0:%.*]] = getelementptr inbounds { float, float, float, float }, { float, float, float, float }* [[CAST]], i32 0, i32 2
+// CHECK:   store float [[T1]], ptr [[T0]], align 4
+// CHECK:   [[T0:%.*]] = getelementptr inbounds { float, float, float, float }, ptr [[AGG]], i32 0, i32 2
 // CHECK:   [[T1:%.*]] = extractvalue { float, float, float, float } [[CALL]], 2
-// CHECK:   store float [[T1]], float* [[T0]], align 8
-// CHECK:   [[T0:%.*]] = getelementptr inbounds { float, float, float, float }, { float, float, float, float }* [[CAST]], i32 0, i32 3
+// CHECK:   store float [[T1]], ptr [[T0]], align 8
+// CHECK:   [[T0:%.*]] = getelementptr inbounds { float, float, float, float }, ptr [[AGG]], i32 0, i32 3
 // CHECK:   [[T1:%.*]] = extractvalue { float, float, float, float } [[CALL]], 3
-// CHECK:   store float [[T1]], float* [[T0]], align 4
-// CHECK:   [[CAST:%.*]] = bitcast [[UNION]]* [[AGG]] to { float, float, float, float }*
-// CHECK:   [[T0:%.*]] = getelementptr inbounds { float, float, float, float }, { float, float, float, float }* [[CAST]], i32 0, i32 0
-// CHECK:   [[V0:%.*]] = load float, float* [[T0]], align 16
-// CHECK:   [[T0:%.*]] = getelementptr inbounds { float, float, float, float }, { float, float, float, float }* [[CAST]], i32 0, i32 1
-// CHECK:   [[V1:%.*]] = load float, float* [[T0]], align 4
-// CHECK:   [[T0:%.*]] = getelementptr inbounds { float, float, float, float }, { float, float, float, float }* [[CAST]], i32 0, i32 2
-// CHECK:   [[V2:%.*]] = load float, float* [[T0]], align 8
-// CHECK:   [[T0:%.*]] = getelementptr inbounds { float, float, float, float }, { float, float, float, float }* [[CAST]], i32 0, i32 3
-// CHECK:   [[V3:%.*]] = load float, float* [[T0]], align 4
+// CHECK:   store float [[T1]], ptr [[T0]], align 4
+// CHECK:   [[T0:%.*]] = getelementptr inbounds { float, float, float, float }, ptr [[AGG]], i32 0, i32 0
+// CHECK:   [[V0:%.*]] = load float, ptr [[T0]], align 16
+// CHECK:   [[T0:%.*]] = getelementptr inbounds { float, float, float, float }, ptr [[AGG]], i32 0, i32 1
+// CHECK:   [[V1:%.*]] = load float, ptr [[T0]], align 4
+// CHECK:   [[T0:%.*]] = getelementptr inbounds { float, float, float, float }, ptr [[AGG]], i32 0, i32 2
+// CHECK:   [[V2:%.*]] = load float, ptr [[T0]], align 8
+// CHECK:   [[T0:%.*]] = getelementptr inbounds { float, float, float, float }, ptr [[AGG]], i32 0, i32 3
+// CHECK:   [[V3:%.*]] = load float, ptr [[T0]], align 4
 // CHECK:   call swiftcc void @take_union_hom_fp_partial(float [[V0]], float [[V1]], float [[V2]], float [[V3]])
 // CHECK:   ret void
 // CHECK: }
@@ -334,23 +311,21 @@ TEST(union_het_fpv_partial)
 // CHECK-LABEL: define dso_local void @test_union_het_fpv_partial()
 // CHECK:   [[AGG:%.*]] = alloca [[UNION:%.*]], align 16
 // CHECK:   [[CALL:%.*]] = call swiftcc { i64, float, float } @return_union_het_fpv_partial()
-// CHECK:   [[CAST:%.*]] = bitcast [[UNION]]* [[AGG]] to { i64, float, float }*
-// CHECK:   [[T0:%.*]] = getelementptr inbounds { i64, float, float }, { i64, float, float }* [[CAST]], i32 0, i32 0
+// CHECK:   [[T0:%.*]] = getelementptr inbounds { i64, float, float }, ptr [[AGG]], i32 0, i32 0
 // CHECK:   [[T1:%.*]] = extractvalue { i64, float, float } [[CALL]], 0
-// CHECK:   store i64 [[T1]], i64* [[T0]], align 16
-// CHECK:   [[T0:%.*]] = getelementptr inbounds { i64, float, float }, { i64, float, float }* [[CAST]], i32 0, i32 1
+// CHECK:   store i64 [[T1]], ptr [[T0]], align 16
+// CHECK:   [[T0:%.*]] = getelementptr inbounds { i64, float, float }, ptr [[AGG]], i32 0, i32 1
 // CHECK:   [[T1:%.*]] = extractvalue { i64, float, float } [[CALL]], 1
-// CHECK:   store float [[T1]], float* [[T0]], align 8
-// CHECK:   [[T0:%.*]] = getelementptr inbounds { i64, float, float }, { i64, float, float }* [[CAST]], i32 0, i32 2
+// CHECK:   store float [[T1]], ptr [[T0]], align 8
+// CHECK:   [[T0:%.*]] = getelementptr inbounds { i64, float, float }, ptr [[AGG]], i32 0, i32 2
 // CHECK:   [[T1:%.*]] = extractvalue { i64, float, float } [[CALL]], 2
-// CHECK:   store float [[T1]], float* [[T0]], align 4
-// CHECK:   [[CAST:%.*]] = bitcast [[UNION]]* [[AGG]] to { i64, float, float }*
-// CHECK:   [[T0:%.*]] = getelementptr inbounds { i64, float, float }, { i64, float, float }* [[CAST]], i32 0, i32 0
-// CHECK:   [[V0:%.*]] = load i64, i64* [[T0]], align 16
-// CHECK:   [[T0:%.*]] = getelementptr inbounds { i64, float, float }, { i64, float, float }* [[CAST]], i32 0, i32 1
-// CHECK:   [[V1:%.*]] = load float, float* [[T0]], align 8
-// CHECK:   [[T0:%.*]] = getelementptr inbounds { i64, float, float }, { i64, float, float }* [[CAST]], i32 0, i32 2
-// CHECK:   [[V2:%.*]] = load float, float* [[T0]], align 4
+// CHECK:   store float [[T1]], ptr [[T0]], align 4
+// CHECK:   [[T0:%.*]] = getelementptr inbounds { i64, float, float }, ptr [[AGG]], i32 0, i32 0
+// CHECK:   [[V0:%.*]] = load i64, ptr [[T0]], align 16
+// CHECK:   [[T0:%.*]] = getelementptr inbounds { i64, float, float }, ptr [[AGG]], i32 0, i32 1
+// CHECK:   [[V1:%.*]] = load float, ptr [[T0]], align 8
+// CHECK:   [[T0:%.*]] = getelementptr inbounds { i64, float, float }, ptr [[AGG]], i32 0, i32 2
+// CHECK:   [[V2:%.*]] = load float, ptr [[T0]], align 4
 // CHECK:   call swiftcc void @take_union_het_fpv_partial(i64 [[V0]], float [[V1]], float [[V2]])
 // CHECK:   ret void
 // CHECK: }
@@ -370,40 +345,36 @@ TEST(int8)
 // CHECK:   store
 // CHECK:   load
 // CHECK:   store
-// CHECK:   [[CAST_TMP:%.*]] = bitcast [[REC]]* [[RET]] to [[AGG:{ <4 x i32>, <4 x i32> }]]*
-// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], [[AGG]]* [[CAST_TMP]], i32 0, i32 0
-// CHECK:   [[FIRST:%.*]] = load <4 x i32>, <4 x i32>* [[T0]], align
-// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], [[AGG]]* [[CAST_TMP]], i32 0, i32 1
-// CHECK:   [[SECOND:%.*]] = load <4 x i32>, <4 x i32>* [[T0]], align
+// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG:{ <4 x i32>, <4 x i32> }]], ptr [[RET]], i32 0, i32 0
+// CHECK:   [[FIRST:%.*]] = load <4 x i32>, ptr [[T0]], align
+// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], ptr [[RET]], i32 0, i32 1
+// CHECK:   [[SECOND:%.*]] = load <4 x i32>, ptr [[T0]], align
 // CHECK:   [[T0:%.*]] = insertvalue [[UAGG:{ <4 x i32>, <4 x i32> }]] poison, <4 x i32> [[FIRST]], 0
 // CHECK:   [[T1:%.*]] = insertvalue [[UAGG]] [[T0]], <4 x i32> [[SECOND]], 1
 // CHECK:   ret [[UAGG]] [[T1]]
 // CHECK-LABEL: define {{.*}} @take_int8(<4 x i32> noundef %0, <4 x i32> noundef %1)
 // CHECK:   [[V:%.*]] = alloca [[REC]], align
-// CHECK:   [[CAST_TMP:%.*]] = bitcast [[REC]]* [[V]] to [[AGG]]*
-// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], [[AGG]]* [[CAST_TMP]], i32 0, i32 0
-// CHECK:   store <4 x i32> %0, <4 x i32>* [[T0]], align
-// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], [[AGG]]* [[CAST_TMP]], i32 0, i32 1
-// CHECK:   store <4 x i32> %1, <4 x i32>* [[T0]], align
+// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], ptr [[V]], i32 0, i32 0
+// CHECK:   store <4 x i32> %0, ptr [[T0]], align
+// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], ptr [[V]], i32 0, i32 1
+// CHECK:   store <4 x i32> %1, ptr [[T0]], align
 // CHECK:   ret void
 // CHECK-LABEL: define dso_local void @test_int8()
 // CHECK:   [[TMP1:%.*]] = alloca [[REC]], align
 // CHECK:   [[TMP2:%.*]] = alloca [[REC]], align
 // CHECK:   [[CALL:%.*]] = call [[SWIFTCC]] [[UAGG]] @return_int8()
-// CHECK:   [[CAST_TMP:%.*]] = bitcast [[REC]]* [[TMP1]] to [[AGG]]*
-// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], [[AGG]]* [[CAST_TMP]], i32 0, i32 0
+// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], ptr [[TMP1]], i32 0, i32 0
 // CHECK:   [[T1:%.*]] = extractvalue [[UAGG]] [[CALL]], 0
-// CHECK:   store <4 x i32> [[T1]], <4 x i32>* [[T0]], align
-// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], [[AGG]]* [[CAST_TMP]], i32 0, i32 1
+// CHECK:   store <4 x i32> [[T1]], ptr [[T0]], align
+// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], ptr [[TMP1]], i32 0, i32 1
 // CHECK:   [[T1:%.*]] = extractvalue [[UAGG]] [[CALL]], 1
-// CHECK:   store <4 x i32> [[T1]], <4 x i32>* [[T0]], align
-// CHECK:   [[V:%.*]] = load [[REC]], [[REC]]* [[TMP1]], align
-// CHECK:   store [[REC]] [[V]], [[REC]]* [[TMP2]], align
-// CHECK:   [[CAST_TMP:%.*]] = bitcast [[REC]]* [[TMP2]] to [[AGG]]*
-// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], [[AGG]]* [[CAST_TMP]], i32 0, i32 0
-// CHECK:   [[FIRST:%.*]] = load <4 x i32>, <4 x i32>* [[T0]], align
-// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], [[AGG]]* [[CAST_TMP]], i32 0, i32 1
-// CHECK:   [[SECOND:%.*]] = load <4 x i32>, <4 x i32>* [[T0]], align
+// CHECK:   store <4 x i32> [[T1]], ptr [[T0]], align
+// CHECK:   [[V:%.*]] = load [[REC]], ptr [[TMP1]], align
+// CHECK:   store [[REC]] [[V]], ptr [[TMP2]], align
+// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], ptr [[TMP2]], i32 0, i32 0
+// CHECK:   [[FIRST:%.*]] = load <4 x i32>, ptr [[T0]], align
+// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], ptr [[TMP2]], i32 0, i32 1
+// CHECK:   [[SECOND:%.*]] = load <4 x i32>, ptr [[T0]], align
 // CHECK:   call [[SWIFTCC]] void @take_int8(<4 x i32> noundef [[FIRST]], <4 x i32> noundef [[SECOND]])
 // CHECK:   ret void
 
@@ -414,40 +385,36 @@ TEST(int5)
 // CHECK:   store
 // CHECK:   load
 // CHECK:   store
-// CHECK:   [[CAST_TMP:%.*]] = bitcast [[REC]]* [[RET]] to [[AGG:{ <4 x i32>, i32 }]]*
-// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], [[AGG]]* [[CAST_TMP]], i32 0, i32 0
-// CHECK:   [[FIRST:%.*]] = load <4 x i32>, <4 x i32>* [[T0]], align
-// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], [[AGG]]* [[CAST_TMP]], i32 0, i32 1
-// CHECK:   [[SECOND:%.*]] = load i32, i32* [[T0]], align
+// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG:{ <4 x i32>, i32 }]], ptr [[RET]], i32 0, i32 0
+// CHECK:   [[FIRST:%.*]] = load <4 x i32>, ptr [[T0]], align
+// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], ptr [[RET]], i32 0, i32 1
+// CHECK:   [[SECOND:%.*]] = load i32, ptr [[T0]], align
 // CHECK:   [[T0:%.*]] = insertvalue [[UAGG:{ <4 x i32>, i32 }]] poison, <4 x i32> [[FIRST]], 0
 // CHECK:   [[T1:%.*]] = insertvalue [[UAGG]] [[T0]], i32 [[SECOND]], 1
 // CHECK:   ret [[UAGG]] [[T1]]
 // CHECK-LABEL: define {{.*}} @take_int5(<4 x i32> %0, i32 %1)
 // CHECK:   [[V:%.*]] = alloca [[REC]], align
-// CHECK:   [[CAST_TMP:%.*]] = bitcast [[REC]]* [[V]] to [[AGG]]*
-// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], [[AGG]]* [[CAST_TMP]], i32 0, i32 0
-// CHECK:   store <4 x i32> %0, <4 x i32>* [[T0]], align
-// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], [[AGG]]* [[CAST_TMP]], i32 0, i32 1
-// CHECK:   store i32 %1, i32* [[T0]], align
+// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], ptr [[V]], i32 0, i32 0
+// CHECK:   store <4 x i32> %0, ptr [[T0]], align
+// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], ptr [[V]], i32 0, i32 1
+// CHECK:   store i32 %1, ptr [[T0]], align
 // CHECK:   ret void
 // CHECK-LABEL: define dso_local void @test_int5()
 // CHECK:   [[TMP1:%.*]] = alloca [[REC]], align
 // CHECK:   [[TMP2:%.*]] = alloca [[REC]], align
 // CHECK:   [[CALL:%.*]] = call [[SWIFTCC]] [[UAGG]] @return_int5()
-// CHECK:   [[CAST_TMP:%.*]] = bitcast [[REC]]* [[TMP1]] to [[AGG]]*
-// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], [[AGG]]* [[CAST_TMP]], i32 0, i32 0
+// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], ptr [[TMP1]], i32 0, i32 0
 // CHECK:   [[T1:%.*]] = extractvalue [[UAGG]] [[CALL]], 0
-// CHECK:   store <4 x i32> [[T1]], <4 x i32>* [[T0]], align
-// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], [[AGG]]* [[CAST_TMP]], i32 0, i32 1
+// CHECK:   store <4 x i32> [[T1]], ptr [[T0]], align
+// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], ptr [[TMP1]], i32 0, i32 1
 // CHECK:   [[T1:%.*]] = extractvalue [[UAGG]] [[CALL]], 1
-// CHECK:   store i32 [[T1]], i32* [[T0]], align
-// CHECK:   [[V:%.*]] = load [[REC]], [[REC]]* [[TMP1]], align
-// CHECK:   store [[REC]] [[V]], [[REC]]* [[TMP2]], align
-// CHECK:   [[CAST_TMP:%.*]] = bitcast [[REC]]* [[TMP2]] to [[AGG]]*
-// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], [[AGG]]* [[CAST_TMP]], i32 0, i32 0
-// CHECK:   [[FIRST:%.*]] = load <4 x i32>, <4 x i32>* [[T0]], align
-// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], [[AGG]]* [[CAST_TMP]], i32 0, i32 1
-// CHECK:   [[SECOND:%.*]] = load i32, i32* [[T0]], align
+// CHECK:   store i32 [[T1]], ptr [[T0]], align
+// CHECK:   [[V:%.*]] = load [[REC]], ptr [[TMP1]], align
+// CHECK:   store [[REC]] [[V]], ptr [[TMP2]], align
+// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], ptr [[TMP2]], i32 0, i32 0
+// CHECK:   [[FIRST:%.*]] = load <4 x i32>, ptr [[T0]], align
+// CHECK:   [[T0:%.*]] = getelementptr inbounds [[AGG]], ptr [[TMP2]], i32 0, i32 1
+// CHECK:   [[SECOND:%.*]] = load i32, ptr [[T0]], align
 // CHECK:   call [[SWIFTCC]] void @take_int5(<4 x i32> [[FIRST]], i32 [[SECOND]])
 // CHECK:   ret void
 
diff --git a/clang/test/CodeGen/x86_32-align-linux.c b/clang/test/CodeGen/x86_32-align-linux.c
index a58e1ece2e409..d383a2c0a92ff 100644
--- a/clang/test/CodeGen/x86_32-align-linux.c
+++ b/clang/test/CodeGen/x86_32-align-linux.c
@@ -9,10 +9,8 @@
 
 // CHECK-LABEL: define dso_local void @testm128
 // CHECK-LABEL: %argp.cur = load ptr, ptr %args, align 4
-// CHECK-NEXT:  %0 = ptrtoint ptr %argp.cur to i32
-// CHECK-NEXT:  %1 = add i32 %0, 15
-// CHECK-NEXT:  %2 = and i32 %1, -16
-// CHECK-NEXT:  %argp.cur.aligned = inttoptr i32 %2 to ptr
+// CHECK-NEXT: %0 = getelementptr inbounds i8, ptr %argp.cur, i32 15
+// CHECK-NEXT: %argp.cur.aligned = call ptr @llvm.ptrmask.p0.i32(ptr %0, i32 -16)
 void testm128(int argCount, ...) {
   __m128 res;
   __builtin_va_list args;
@@ -23,10 +21,8 @@ void testm128(int argCount, ...) {
 
 // CHECK-LABEL: define dso_local void @testm256
 // CHECK-LABEL: %argp.cur = load ptr, ptr %args, align 4
-// CHECK-NEXT:  %0 = ptrtoint ptr %argp.cur to i32
-// CHECK-NEXT:  %1 = add i32 %0, 31
-// CHECK-NEXT:  %2 = and i32 %1, -32
-// CHECK-NEXT:  %argp.cur.aligned = inttoptr i32 %2 to ptr
+// CHECK-NEXT: %0 = getelementptr inbounds i8, ptr %argp.cur, i32 31
+// CHECK-NEXT: %argp.cur.aligned = call ptr @llvm.ptrmask.p0.i32(ptr %0, i32 -32)
 void testm256(int argCount, ...) {
   __m256 res;
   __builtin_va_list args;
@@ -37,10 +33,8 @@ void testm256(int argCount, ...) {
 
 // CHECK-LABEL: define dso_local void @testm512
 // CHECK-LABEL: %argp.cur = load ptr, ptr %args, align 4
-// CHECK-NEXT:  %0 = ptrtoint ptr %argp.cur to i32
-// CHECK-NEXT:  %1 = add i32 %0, 63
-// CHECK-NEXT:  %2 = and i32 %1, -64
-// CHECK-NEXT:  %argp.cur.aligned = inttoptr i32 %2 to ptr
+// CHECK-NEXT: %0 = getelementptr inbounds i8, ptr %argp.cur, i32 63
+// CHECK-NEXT: %argp.cur.aligned = call ptr @llvm.ptrmask.p0.i32(ptr %0, i32 -64)
 void testm512(int argCount, ...) {
   __m512 res;
   __builtin_va_list args;
diff --git a/clang/test/CodeGenCXX/aarch64-mangle-sve-vectors.cpp b/clang/test/CodeGenCXX/aarch64-mangle-sve-vectors.cpp
index d75efbd25c484..f387824b56661 100644
--- a/clang/test/CodeGenCXX/aarch64-mangle-sve-vectors.cpp
+++ b/clang/test/CodeGenCXX/aarch64-mangle-sve-vectors.cpp
@@ -31,6 +31,8 @@ void f11(S<__SVFloat64_t>) {}
 void f12(S<__SVBFloat16_t>) {}
 // CHECK: _Z3f131SIu10__SVBool_tE
 void f13(S<__SVBool_t>) {}
+// CHECK: _Z3f141SIu11__SVCount_tE
+void f14(S<__SVCount_t>) {}
 
 // The tuple types don't use the internal name for mangling.
 
@@ -106,3 +108,7 @@ void f46(S<__clang_svbfloat16x2_t>) {}
 void f47(S<__clang_svbfloat16x3_t>) {}
 // CHECK: _Z3f481SI14svbfloat16x4_tE
 void f48(S<__clang_svbfloat16x4_t>) {}
+// CHECK: _Z3f491SI10svboolx2_tE
+void f49(S<__clang_svboolx2_t>) {}
+// CHECK: _Z3f501SI10svboolx4_tE
+void f50(S<__clang_svboolx4_t>) {}
diff --git a/clang/test/CodeGenCXX/aarch64-sve-typeinfo.cpp b/clang/test/CodeGenCXX/aarch64-sve-typeinfo.cpp
index 6081ab69a2344..4ee2ed9888438 100644
--- a/clang/test/CodeGenCXX/aarch64-sve-typeinfo.cpp
+++ b/clang/test/CodeGenCXX/aarch64-sve-typeinfo.cpp
@@ -22,6 +22,7 @@ auto &f64 = typeid(__SVFloat64_t);
 auto &bf16 = typeid(__SVBFloat16_t);
 
 auto &b8 = typeid(__SVBool_t);
+auto &c8 = typeid(__SVCount_t);
 
 // CHECK-DAG: @_ZTSu10__SVInt8_t = {{.*}} c"u10__SVInt8_t\00"
 // CHECK-DAG: @_ZTIu10__SVInt8_t = {{.*}} @_ZTVN10__cxxabiv123__fundamental_type_infoE, {{.*}} @_ZTSu10__SVInt8_t
@@ -61,3 +62,6 @@ auto &b8 = typeid(__SVBool_t);
 
 // CHECK-DAG: @_ZTSu10__SVBool_t = {{.*}} c"u10__SVBool_t\00"
 // CHECK-DAG: @_ZTIu10__SVBool_t = {{.*}} @_ZTVN10__cxxabiv123__fundamental_type_infoE, {{.*}} @_ZTSu10__SVBool_t
+
+// CHECK-DAG: @_ZTSu11__SVCount_t = {{.*}} c"u11__SVCount_t\00"
+// CHECK-DAG: @_ZTIu11__SVCount_t = {{.*}} @_ZTVN10__cxxabiv123__fundamental_type_infoE, {{.*}} @_ZTSu11__SVCount_t
diff --git a/clang/test/CodeGenCXX/const-init-cxx11.cpp b/clang/test/CodeGenCXX/const-init-cxx11.cpp
index e6ab50b3b3ef9..dc2c8b9b920fd 100644
--- a/clang/test/CodeGenCXX/const-init-cxx11.cpp
+++ b/clang/test/CodeGenCXX/const-init-cxx11.cpp
@@ -1,4 +1,5 @@
 // RUN: %clang_cc1 -no-opaque-pointers -w -fmerge-all-constants -triple x86_64-elf-gnu -emit-llvm -o - %s -std=c++11 | FileCheck %s
+// RUN: %clang_cc1 -no-opaque-pointers -w -fmerge-all-constants -triple x86_64-elf-gnu -emit-llvm -o - %s -std=c++20 | FileCheck -check-prefix=CHECK20 %s
 
 // FIXME: The padding in all these objects should be zero-initialized.
 namespace StructUnion {
@@ -424,6 +425,7 @@ namespace DR2126 {
 // CHECK: @_ZGRN39ClassTemplateWithHiddenStaticDataMember1SIvE1aE_ = linkonce_odr hidden constant i32 5, comdat
 // CHECK: @_ZN39ClassTemplateWithHiddenStaticDataMember3useE ={{.*}} constant i32* @_ZGRN39ClassTemplateWithHiddenStaticDataMember1SIvE1aE_
 // CHECK: @_ZGRZN20InlineStaticConstRef3funEvE1i_ = linkonce_odr constant i32 10, comdat
+// CHECK20: @_ZZN12LocalVarInit4dtorEvE1a = internal constant {{.*}} i32 103
 
 // Constant initialization tests go before this point,
 // dynamic initialization tests go after.
@@ -483,6 +485,14 @@ namespace LocalVarInit {
   // CHECK-NOT: ret i32 103
   // CHECK: }
   int mutable_() { constexpr Mutable a = { f(103) }; return a.k; }
+
+#if __cplusplus >= 202002L
+  // CHECK20: define {{.*}} @_ZN12LocalVarInit4dtorEv
+  // CHECK20-NOT: call
+  // CHECK20: ret i32 103
+  struct Dtor { constexpr Dtor(int n) : k(n) {} constexpr ~Dtor() {} int k; };
+  int dtor() { constexpr Dtor a = { f(103) }; return a.k; }
+#endif
 }
 
 namespace CrossFuncLabelDiff {
diff --git a/clang/test/CodeGenCXX/const-init-cxx2a.cpp b/clang/test/CodeGenCXX/const-init-cxx2a.cpp
index 3eafef0943872..3c83a9c94adec 100644
--- a/clang/test/CodeGenCXX/const-init-cxx2a.cpp
+++ b/clang/test/CodeGenCXX/const-init-cxx2a.cpp
@@ -11,10 +11,10 @@ struct B {
   constexpr ~B() { n *= 5; }
   int n = 123;
 };
-// CHECK: @b ={{.*}} global {{.*}} i32 123
+// CHECK: @b ={{.*}} constant {{.*}} i32 123
 extern constexpr B b = B();
 
-// CHECK: @_ZL1c = internal global {{.*}} i32 123
+// CHECK: @_ZL1c = internal constant {{.*}} i32 123
 const B c;
 int use_c() { return c.n; }
 
diff --git a/clang/test/CodeGenCXX/debug-info-enum-i128.cpp b/clang/test/CodeGenCXX/debug-info-enum-i128.cpp
index 92bf8eba873c3..22aaf03bd9af5 100644
--- a/clang/test/CodeGenCXX/debug-info-enum-i128.cpp
+++ b/clang/test/CodeGenCXX/debug-info-enum-i128.cpp
@@ -1,9 +1,10 @@
+// FIXME: llvm.org/pr51221, the APSInt leaks
+// RUN: export LSAN_OPTIONS=detect_leaks=0
+
 // RUN: %clang_cc1 %s -triple x86_64-windows-msvc -gcodeview -debug-info-kind=limited -emit-llvm -o - | FileCheck %s
 // RUN: %clang_cc1 %s -triple x86_64-linux-gnu -debug-info-kind=limited -emit-llvm -o - | FileCheck %s
 // RUN: %clang_cc1 %s -triple powerpc64-ibm-aix-xcoff -debug-info-kind=limited -emit-llvm -o - | FileCheck %s
 
-// FIXME: llvm.org/pr51221, the APSInt leaks
-// UNSUPPORTED: asan
 
 enum class uns : __uint128_t { unsval = __uint128_t(1) << 64 };
 uns t1() { return uns::unsval; }
diff --git a/clang/test/CodeGenCXX/decl-ref-inheritance.cpp b/clang/test/CodeGenCXX/decl-ref-inheritance.cpp
new file mode 100644
index 0000000000000..9206ee5ac9369
--- /dev/null
+++ b/clang/test/CodeGenCXX/decl-ref-inheritance.cpp
@@ -0,0 +1,60 @@
+// RUN: %clang_cc1 -triple=x86_64-unknown-linux -emit-llvm %s -o - | FileCheck \
+// RUN: -check-prefix=CHECK-1 %s
+// RUN: %clang_cc1 -triple=x86_64-unknown-linux -emit-llvm %s -o - | FileCheck \
+// RUN: -check-prefix=CHECK-2 %s
+// RUN: %clang_cc1 -triple=x86_64-unknown-linux -emit-llvm %s -o - | FileCheck \
+// RUN: -check-prefix=CHECK-3 %s
+
+// CHECK-1: [[FOO:%.+]] = type { float }
+struct foo {
+  float val;
+};
+
+template <typename T> struct bar : T {
+};
+
+struct baz : bar<foo> {
+  // CHECK-1: define{{.*}} float @_ZN3baz3getEv
+  // CHECK-1: {{%.+}} = getelementptr inbounds [[FOO]], ptr {{%.+}}, i32 0, i32 0
+  float get() {
+    return val;
+  }
+};
+
+int qux() {
+  auto f = baz{};
+  return f.get();
+}
+
+// CHECK-2: [[F:%.+]] = type { ptr }
+struct f {
+  void *g;
+};
+
+template <typename j> struct k : j {
+  // CHECK-2: define{{.*}} void @_ZN1kI1fE1lEv
+  // CHECK-2: {{%.+}} = getelementptr inbounds [[F]], ptr {{%.+}}, i32 0, i32 0
+  virtual void l(){ (void)f::g; }
+};
+
+k<f> q;
+
+// CHECK-3: [[BASE:%.+]] = type { i32 }
+class Base {
+protected:
+  int member;
+};
+
+template <typename Parent>
+struct Subclass : public Parent {
+  // CHECK-3: define{{.*}} i32 @_ZN8SubclassI4BaseE4funcEv
+  // CHECK-3: {{%.+}} = getelementptr inbounds [[BASE]], ptr {{%.+}}, i32 0, i32 0
+  int func() { return Base::member; }
+};
+
+using Impl = Subclass<Base>;
+
+int use() {
+  Impl i;
+  return i.func();
+}
diff --git a/clang/test/CodeGenCXX/dllexport.cpp b/clang/test/CodeGenCXX/dllexport.cpp
index b4e42ed7abab2..c8ac526f4cbe3 100644
--- a/clang/test/CodeGenCXX/dllexport.cpp
+++ b/clang/test/CodeGenCXX/dllexport.cpp
@@ -6,6 +6,8 @@
 
 // RUN: %clang_cc1 -no-enable-noundef-analysis -triple i686-windows-gnu    -emit-llvm -std=c++1y -fno-threadsafe-statics -fms-extensions -O0 -o - %s -w | FileCheck -allow-deprecated-dag-overlap --check-prefix=GNU --check-prefix=G32 %s
 // RUN: %clang_cc1 -no-enable-noundef-analysis -triple x86_64-windows-gnu  -emit-llvm -std=c++1y -fno-threadsafe-statics -fms-extensions -O0 -o - %s -w | FileCheck -allow-deprecated-dag-overlap --check-prefix=GNU %s
+// RUN: %clang_cc1 -no-enable-noundef-analysis -triple x86_64-scei-ps4  -emit-llvm -std=c++1y -fno-threadsafe-statics -fms-extensions -O0 -o - %s -w | FileCheck -allow-deprecated-dag-overlap --check-prefix=PS %s
+// RUN: %clang_cc1 -no-enable-noundef-analysis -triple x86_64-sie-ps5  -emit-llvm -std=c++1y -fno-threadsafe-statics -fms-extensions -O0 -o - %s -w | FileCheck -allow-deprecated-dag-overlap --check-prefix=PS %s
 
 // Helper structs to make templates more expressive.
 struct ImplicitInst_Exported {};
@@ -980,18 +982,21 @@ struct __declspec(dllexport) DerivedFromTemplate : public ClassTemplate<int> {};
 USEMEMFUNC(DerivedFromTemplate, func)
 // M32-DAG: define weak_odr dso_local dllexport x86_thiscallcc void @"?func@?$ClassTemplate@H@@QAEXXZ"
 // G32-DAG: define linkonce_odr dso_local x86_thiscallcc void @_ZN13ClassTemplateIiE4funcEv
+// PS-DAG:  define weak_odr dllexport void @_ZN13ClassTemplateIiE4funcEv
 
 // ExportedTemplate is explicitly exported.
 struct __declspec(dllexport) DerivedFromExportedTemplate : public ExportedClassTemplate<int> {};
 USEMEMFUNC(DerivedFromExportedTemplate, func)
 // M32-DAG: define weak_odr dso_local dllexport x86_thiscallcc void @"?func@?$ExportedClassTemplate@H@@QAEXXZ"
 // G32-DAG: define weak_odr dso_local dllexport x86_thiscallcc void @_ZN21ExportedClassTemplateIiE4funcEv
+// PS-DAG:  define weak_odr dllexport void @_ZN21ExportedClassTemplateIiE4funcEv
 
 // ImportedClassTemplate is explicitly imported.
 struct __declspec(dllexport) DerivedFromImportedTemplate : public ImportedClassTemplate<int> {};
 USEMEMFUNC(DerivedFromImportedTemplate, func)
 // M32-DAG: {{declare|define available_externally}} dllimport x86_thiscallcc void @"?func@?$ImportedClassTemplate@H@@QAEXXZ"
 // G32-DAG: declare dllimport x86_thiscallcc void @_ZN21ImportedClassTemplateIiE4funcEv
+// PS-DAG:  declare dllimport void @_ZN21ImportedClassTemplateIiE4funcEv
 
 // Base class already implicitly instantiated without dll attribute.
 struct DerivedFromTemplateD : public ClassTemplate<double> {};
@@ -999,6 +1004,7 @@ struct __declspec(dllexport) DerivedFromTemplateD2 : public ClassTemplate<double
 USEMEMFUNC(DerivedFromTemplateD2, func)
 // M32-DAG: define weak_odr dso_local dllexport x86_thiscallcc void @"?func@?$ClassTemplate@N@@QAEXXZ"
 // G32-DAG: define linkonce_odr dso_local x86_thiscallcc void @_ZN13ClassTemplateIdE4funcEv
+// PS-DAG:  define weak_odr dllexport void @_ZN13ClassTemplateIdE4funcEv
 
 // MS: Base class already instantiated with different dll attribute.
 struct __declspec(dllimport) DerivedFromTemplateB : public ClassTemplate<bool> {};
@@ -1006,42 +1012,49 @@ struct __declspec(dllexport) DerivedFromTemplateB2 : public ClassTemplate<bool>
 USEMEMFUNC(DerivedFromTemplateB2, func)
 // M32-DAG: {{declare|define available_externally}} dllimport x86_thiscallcc void @"?func@?$ClassTemplate@_N@@QAEXXZ"
 // G32-DAG: define linkonce_odr dso_local x86_thiscallcc void @_ZN13ClassTemplateIbE4funcEv
+// PS-DAG:  declare dllimport void @_ZN13ClassTemplateIbE4funcEv
 
 // Base class already specialized without dll attribute.
 struct __declspec(dllexport) DerivedFromExplicitlySpecializedTemplate : public ExplicitlySpecializedTemplate<int> {};
 USEMEMFUNC(DerivedFromExplicitlySpecializedTemplate, func)
 // M32-DAG: define dso_local x86_thiscallcc void @"?func@?$ExplicitlySpecializedTemplate@H@@QAEXXZ"
 // G32-DAG: define dso_local x86_thiscallcc void @_ZN29ExplicitlySpecializedTemplateIiE4funcEv
+// PS-DAG:  define dso_local void @_ZN29ExplicitlySpecializedTemplateIiE4funcEv
 
 // Base class alredy specialized with export attribute.
 struct __declspec(dllexport) DerivedFromExplicitlyExportSpecializedTemplate : public ExplicitlyExportSpecializedTemplate<int> {};
 USEMEMFUNC(DerivedFromExplicitlyExportSpecializedTemplate, func)
 // M32-DAG: define dso_local dllexport x86_thiscallcc void @"?func@?$ExplicitlyExportSpecializedTemplate@H@@QAEXXZ"
 // G32-DAG: define dso_local dllexport x86_thiscallcc void @_ZN35ExplicitlyExportSpecializedTemplateIiE4funcEv
+// PS-DAG:  define dso_local dllexport void @_ZN35ExplicitlyExportSpecializedTemplateIiE4funcEv
 
 // Base class already specialized with import attribute.
 struct __declspec(dllexport) DerivedFromExplicitlyImportSpecializedTemplate : public ExplicitlyImportSpecializedTemplate<int> {};
 USEMEMFUNC(DerivedFromExplicitlyImportSpecializedTemplate, func)
 // M32-DAG: declare dllimport x86_thiscallcc void @"?func@?$ExplicitlyImportSpecializedTemplate@H@@QAEXXZ"
 // G32-DAG: declare dllimport x86_thiscallcc void @_ZN35ExplicitlyImportSpecializedTemplateIiE4funcEv
+// PS-DAG:  declare dllimport void @_ZN35ExplicitlyImportSpecializedTemplateIiE4funcEv
 
 // Base class already instantiated without dll attribute.
 struct __declspec(dllexport) DerivedFromExplicitlyInstantiatedTemplate : public ExplicitlyInstantiatedTemplate<int> {};
 USEMEMFUNC(DerivedFromExplicitlyInstantiatedTemplate, func)
 // M32-DAG: define weak_odr dso_local x86_thiscallcc void @"?func@?$ExplicitlyInstantiatedTemplate@H@@QAEXXZ"
 // G32-DAG: define weak_odr dso_local x86_thiscallcc void @_ZN30ExplicitlyInstantiatedTemplateIiE4funcEv
+// PS-DAG:  define weak_odr void @_ZN30ExplicitlyInstantiatedTemplateIiE4funcEv
 
 // Base class already instantiated with export attribute.
 struct __declspec(dllexport) DerivedFromExplicitlyExportInstantiatedTemplate : public ExplicitlyExportInstantiatedTemplate<int> {};
 USEMEMFUNC(DerivedFromExplicitlyExportInstantiatedTemplate, func)
 // M32-DAG: define weak_odr dso_local dllexport x86_thiscallcc void @"?func@?$ExplicitlyExportInstantiatedTemplate@H@@QAEXXZ"
 // G32-DAG: define weak_odr dso_local dllexport x86_thiscallcc void @_ZN36ExplicitlyExportInstantiatedTemplateIiE4funcEv
+// PS-DAG:  define weak_odr dllexport void @_ZN36ExplicitlyExportInstantiatedTemplateIiE4funcEv
 
 // Base class already instantiated with import attribute.
 struct __declspec(dllexport) DerivedFromExplicitlyImportInstantiatedTemplate : public ExplicitlyImportInstantiatedTemplate<int> {};
 USEMEMFUNC(DerivedFromExplicitlyImportInstantiatedTemplate, func)
 // M32-DAG: declare dllimport x86_thiscallcc void @"?func@?$ExplicitlyImportInstantiatedTemplate@H@@QAEXXZ"
 // G32-DAG: declare dllimport x86_thiscallcc void @_ZN36ExplicitlyImportInstantiatedTemplateIiE4funcEv
+// PS-DAG:  declare dllimport void @_ZN36ExplicitlyImportInstantiatedTemplateIiE4funcEv
 
 // MS: A dll attribute propagates through multiple levels of instantiation.
 template <typename T> struct TopClass { void func() {} };
@@ -1050,6 +1063,7 @@ struct __declspec(dllexport) BottomClass : public MiddleClass<int> { };
 USEMEMFUNC(BottomClass, func)
 // M32-DAG: define weak_odr dso_local dllexport x86_thiscallcc void @"?func@?$TopClass@H@@QAEXXZ"
 // G32-DAG: define linkonce_odr dso_local x86_thiscallcc void @_ZN8TopClassIiE4funcEv
+// PS-DAG:  define weak_odr dllexport void @_ZN8TopClassIiE4funcEv
 
 template <typename T> struct ExplicitInstantiationDeclTemplateBase { void func() {} };
 extern template struct ExplicitInstantiationDeclTemplateBase<int>;
@@ -1057,6 +1071,7 @@ struct __declspec(dllexport) DerivedFromExplicitInstantiationDeclTemplateBase :
 template struct ExplicitInstantiationDeclTemplateBase<int>;
 // M32-DAG: define weak_odr dso_local dllexport x86_thiscallcc void @"?func@?$ExplicitInstantiationDeclTemplateBase@H@@QAEXXZ"
 // G32-DAG: define weak_odr dso_local x86_thiscallcc void @_ZN37ExplicitInstantiationDeclTemplateBaseIiE4funcEv
+// PS-DAG:  define weak_odr dllexport void @_ZN37ExplicitInstantiationDeclTemplateBaseIiE4funcEv
 
 // PR26076
 struct LayerSelectionBound;
diff --git a/clang/test/CodeGenCXX/dllimport.cpp b/clang/test/CodeGenCXX/dllimport.cpp
index ef4577d4799d7..6fec2f2982d40 100644
--- a/clang/test/CodeGenCXX/dllimport.cpp
+++ b/clang/test/CodeGenCXX/dllimport.cpp
@@ -5,6 +5,8 @@
 // RUN: %clang_cc1 -no-enable-noundef-analysis -triple i686-windows-msvc   -fno-rtti -fno-threadsafe-statics -fms-extensions -fms-compatibility-version=18.00 -emit-llvm -std=c++1y -O1 -disable-llvm-passes -o - %s -DMSABI -w | FileCheck --check-prefix=MO1 --check-prefix=M18 %s
 // RUN: %clang_cc1 -no-enable-noundef-analysis -triple i686-windows-msvc   -fno-rtti -fno-threadsafe-statics -fms-extensions -fms-compatibility-version=19.00 -emit-llvm -std=c++1y -O1 -disable-llvm-passes -o - %s -DMSABI -w | FileCheck --check-prefix=MO1 --check-prefix=M19 %s
 // RUN: %clang_cc1 -no-enable-noundef-analysis -triple i686-windows-gnu    -fno-rtti -fno-threadsafe-statics -fms-extensions                                  -emit-llvm -std=c++1y -O1 -disable-llvm-passes -o - %s         -w | FileCheck --check-prefix=GO1 %s
+// RUN: %clang_cc1 -no-enable-noundef-analysis -triple x86_64-scei-ps4    -fno-rtti -fno-threadsafe-statics -fms-extensions -emit-llvm -std=c++1y -O0 -o - %s         -w | FileCheck --check-prefix=PS %s
+// RUN: %clang_cc1 -no-enable-noundef-analysis -triple x86_64-sie-ps5    -fno-rtti -fno-threadsafe-statics -fms-extensions -emit-llvm -std=c++1y -O0 -o - %s         -w | FileCheck --check-prefix=PS %s
 
 // CHECK-NOT doesn't play nice with CHECK-DAG, so use separate run lines.
 // RUN: %clang_cc1 -no-enable-noundef-analysis -triple i686-windows-msvc   -fno-rtti -fno-threadsafe-statics -fms-extensions -emit-llvm -std=c++1y -O0 -o - %s -DMSABI -w | FileCheck --check-prefix=MSC2 %s
@@ -917,18 +919,21 @@ struct __declspec(dllimport) DerivedFromTemplate : public ClassTemplate<int> {};
 USEMEMFUNC(ClassTemplate<int>, func)
 // M32-DAG: {{declare|define available_externally}} dllimport x86_thiscallcc void @"?func@?$ClassTemplate@H@@QAEXXZ"
 // G32-DAG: define linkonce_odr dso_local x86_thiscallcc void @_ZN13ClassTemplateIiE4funcEv
+// PS-DAG:  declare dllimport void @_ZN13ClassTemplateIiE4funcEv
 
 // ImportedTemplate is explicitly imported.
 struct __declspec(dllimport) DerivedFromImportedTemplate : public ImportedClassTemplate<int> {};
 USEMEMFUNC(ImportedClassTemplate<int>, func)
 // M32-DAG: declare dllimport x86_thiscallcc void @"?func@?$ImportedClassTemplate@H@@QAEXXZ"
 // G32-DAG: declare dllimport x86_thiscallcc void @_ZN21ImportedClassTemplateIiE4funcEv
+// PS-DAG:  declare dllimport void @_ZN21ImportedClassTemplateIiE4funcEv
 
 // ExportedTemplate is explicitly exported.
 struct __declspec(dllimport) DerivedFromExportedTemplate : public ExportedClassTemplate<int> {};
 USEMEMFUNC(ExportedClassTemplate<int>, func)
 // M32-DAG: define weak_odr dso_local dllexport x86_thiscallcc void @"?func@?$ExportedClassTemplate@H@@QAEXXZ"
 // G32-DAG: define weak_odr dso_local dllexport x86_thiscallcc void @_ZN21ExportedClassTemplateIiE4funcEv
+// PS-DAG:  define weak_odr dllexport void @_ZN21ExportedClassTemplateIiE4funcEv
 
 // Base class already implicitly instantiated without attribute.
 struct DerivedFromTemplateD : public ClassTemplate<double> {};
@@ -936,6 +941,7 @@ struct __declspec(dllimport) DerivedFromTemplateD2 : public ClassTemplate<double
 USEMEMFUNC(ClassTemplate<double>, func)
 // M32-DAG: declare dllimport x86_thiscallcc void @"?func@?$ClassTemplate@N@@QAEXXZ"
 // G32-DAG: define linkonce_odr dso_local x86_thiscallcc void @_ZN13ClassTemplateIdE4funcEv
+// PS-DAG:  declare dllimport void @_ZN13ClassTemplateIdE4funcEv
 
 // MS: Base class already instantiated with dfferent attribute.
 struct __declspec(dllexport) DerivedFromTemplateB : public ClassTemplate<bool> {};
@@ -943,42 +949,49 @@ struct __declspec(dllimport) DerivedFromTemplateB2 : public ClassTemplate<bool>
 USEMEMFUNC(ClassTemplate<bool>, func)
 // M32-DAG: define weak_odr dso_local dllexport x86_thiscallcc void @"?func@?$ClassTemplate@_N@@QAEXXZ"
 // G32-DAG: define linkonce_odr dso_local x86_thiscallcc void @_ZN13ClassTemplateIbE4funcEv
+// PS-DAG:  define weak_odr dllexport void @_ZN13ClassTemplateIbE4funcEv
 
 // Base class already specialized without dll attribute.
 struct __declspec(dllimport) DerivedFromExplicitlySpecializedTemplate : public ExplicitlySpecializedTemplate<int> {};
 USEMEMFUNC(ExplicitlySpecializedTemplate<int>, func)
 // M32-DAG: define linkonce_odr dso_local x86_thiscallcc void @"?func@?$ExplicitlySpecializedTemplate@H@@QAEXXZ"
 // G32-DAG: define linkonce_odr dso_local x86_thiscallcc void @_ZN29ExplicitlySpecializedTemplateIiE4funcEv
+// PS-DAG:  define linkonce_odr void @_ZN29ExplicitlySpecializedTemplateIiE4funcEv
 
 // Base class alredy specialized with export attribute.
 struct __declspec(dllimport) DerivedFromExplicitlyExportSpecializedTemplate : public ExplicitlyExportSpecializedTemplate<int> {};
 USEMEMFUNC(ExplicitlyExportSpecializedTemplate<int>, func)
 // M32-DAG: define dso_local dllexport x86_thiscallcc void @"?func@?$ExplicitlyExportSpecializedTemplate@H@@QAEXXZ"
 // G32-DAG: define dso_local dllexport x86_thiscallcc void @_ZN35ExplicitlyExportSpecializedTemplateIiE4funcEv
+// PS-DAG:  define dso_local dllexport void @_ZN35ExplicitlyExportSpecializedTemplateIiE4funcEv
 
 // Base class already specialized with import attribute.
 struct __declspec(dllimport) DerivedFromExplicitlyImportSpecializedTemplate : public ExplicitlyImportSpecializedTemplate<int> {};
 USEMEMFUNC(ExplicitlyImportSpecializedTemplate<int>, func)
 // M32-DAG: declare dllimport x86_thiscallcc void @"?func@?$ExplicitlyImportSpecializedTemplate@H@@QAEXXZ"
 // G32-DAG: declare dllimport x86_thiscallcc void @_ZN35ExplicitlyImportSpecializedTemplateIiE4funcEv
+// PS-DAG:  declare dllimport void @_ZN35ExplicitlyImportSpecializedTemplateIiE4funcEv
 
 // Base class already instantiated without dll attribute.
 struct __declspec(dllimport) DerivedFromExplicitlyInstantiatedTemplate : public ExplicitlyInstantiatedTemplate<int> {};
 USEMEMFUNC(ExplicitlyInstantiatedTemplate<int>, func)
 // M32-DAG: define weak_odr dso_local x86_thiscallcc void @"?func@?$ExplicitlyInstantiatedTemplate@H@@QAEXXZ"
 // G32-DAG: define weak_odr dso_local x86_thiscallcc void @_ZN30ExplicitlyInstantiatedTemplateIiE4funcEv
+// PS-DAG:  define weak_odr void @_ZN30ExplicitlyInstantiatedTemplateIiE4funcEv
 
 // Base class already instantiated with export attribute.
 struct __declspec(dllimport) DerivedFromExplicitlyExportInstantiatedTemplate : public ExplicitlyExportInstantiatedTemplate<int> {};
 USEMEMFUNC(ExplicitlyExportInstantiatedTemplate<int>, func)
 // M32-DAG: define weak_odr dso_local dllexport x86_thiscallcc void @"?func@?$ExplicitlyExportInstantiatedTemplate@H@@QAEXXZ"
 // G32-DAG: define weak_odr dso_local dllexport x86_thiscallcc void @_ZN36ExplicitlyExportInstantiatedTemplateIiE4funcEv
+// PS-DAG:  define weak_odr dllexport void @_ZN36ExplicitlyExportInstantiatedTemplateIiE4funcEv
 
 // Base class already instantiated with import attribute.
 struct __declspec(dllimport) DerivedFromExplicitlyImportInstantiatedTemplate : public ExplicitlyImportInstantiatedTemplate<int> {};
 USEMEMFUNC(ExplicitlyImportInstantiatedTemplate<int>, func)
 // M32-DAG: declare dllimport x86_thiscallcc void @"?func@?$ExplicitlyImportInstantiatedTemplate@H@@QAEXXZ"
 // G32-DAG: declare dllimport x86_thiscallcc void @_ZN36ExplicitlyImportInstantiatedTemplateIiE4funcEv
+// PS-DAG:  declare dllimport void @_ZN36ExplicitlyImportInstantiatedTemplateIiE4funcEv
 
 // MS: A dll attribute propagates through multiple levels of instantiation.
 template <typename T> struct TopClass { void func() {} };
@@ -987,6 +1000,7 @@ struct __declspec(dllimport) BottomClass : public MiddleClass<int> { };
 USEMEMFUNC(TopClass<int>, func)
 // M32-DAG: {{declare|define available_externally}} dllimport x86_thiscallcc void @"?func@?$TopClass@H@@QAEXXZ"
 // G32-DAG: define linkonce_odr dso_local x86_thiscallcc void @_ZN8TopClassIiE4funcEv
+// PS-DAG:  declare dllimport void @_ZN8TopClassIiE4funcEv
 
 template <typename T> struct ExplicitInstantiationDeclTemplateBase { void func() {} };
 extern template struct ExplicitInstantiationDeclTemplateBase<int>;
@@ -995,6 +1009,7 @@ template struct ExplicitInstantiationDeclTemplateBase<int>;
 USEMEMFUNC(ExplicitInstantiationDeclTemplateBase<int>, func)
 // M32-DAG: declare dllimport x86_thiscallcc void @"?func@?$ExplicitInstantiationDeclTemplateBase@H@@QAEXXZ"
 // G32-DAG: define weak_odr dso_local x86_thiscallcc void @_ZN37ExplicitInstantiationDeclTemplateBaseIiE4funcEv
+// PS-DAG:  declare dllimport void @_ZN37ExplicitInstantiationDeclTemplateBaseIiE4funcEv
 
 template <typename T> struct ExplicitInstantiationDeclTemplateBase2 { void func() {} };
 extern template struct ExplicitInstantiationDeclTemplateBase2<int>;
@@ -1003,6 +1018,7 @@ template struct __declspec(dllexport) ExplicitInstantiationDeclTemplateBase2<int
 USEMEMFUNC(ExplicitInstantiationDeclTemplateBase2<int>, func)
 // M32-DAG: declare dllimport x86_thiscallcc void @"?func@?$ExplicitInstantiationDeclTemplateBase2@H@@QAEXXZ"
 // G32-DAG: define weak_odr dso_local x86_thiscallcc void @_ZN38ExplicitInstantiationDeclTemplateBase2IiE4funcEv
+// PS-DAG:  declare dllimport void @_ZN38ExplicitInstantiationDeclTemplateBase2IiE4funcEv
 
 namespace pr39496 {
 // Make sure dll attribute are inherited by static locals also in template
diff --git a/clang/test/CodeGenCXX/ext-int.cpp b/clang/test/CodeGenCXX/ext-int.cpp
index 86c566f903e32..7676dec791f3f 100644
--- a/clang/test/CodeGenCXX/ext-int.cpp
+++ b/clang/test/CodeGenCXX/ext-int.cpp
@@ -200,7 +200,7 @@ void TakesVarargs(int i, ...) {
   // WIN64: store i92 %[[LOADV1]], ptr
 
   // WIN32: %[[CUR1:.+]] = load ptr, ptr %[[ARGS]]
-  // WIN32: %[[NEXT1:.+]] = getelementptr inbounds i8, ptr %[[CUR1]], i32 16 
+  // WIN32: %[[NEXT1:.+]] = getelementptr inbounds i8, ptr %[[CUR1]], i32 16
   // WIN32: store ptr %[[NEXT1]], ptr %[[ARGS]]
   // WIN32: %[[LOADV1:.+]] = load i92, ptr %[[CUR1]]
   // WIN32: store i92 %[[LOADV1]], ptr
@@ -289,18 +289,16 @@ void TakesVarargs(int i, ...) {
   // LIN64: %[[AD5:.+]] = getelementptr inbounds [1 x %struct.__va_list_tag], ptr %[[ARGS]]
   // LIN64: %[[OFAA_P4:.+]] = getelementptr inbounds %struct.__va_list_tag, ptr %[[AD5]], i32 0, i32 2
   // LIN64: %[[OFAA:.+]] = load ptr, ptr %[[OFAA_P4]]
-  // LIN64: %[[TOINT:.+]] = ptrtoint ptr %[[OFAA]] to i64
-  // LIN64: %[[ADD:.+]] = add i64 %[[TOINT]], 31
-  // LIN64: %[[AND:.+]] = and i64 %[[ADD]], -32
-  // LIN64: %[[OFAA_ALIGNED:.+]] = inttoptr i64 %[[AND]] to ptr
+
+  // LIN64: [[OFAA_GEP:%.*]] = getelementptr inbounds i8, ptr %[[OFAA]], i32 31
+  // LIN64: %[[OFAA_ALIGNED:.*]] = call ptr @llvm.ptrmask.p0.i64(ptr [[OFAA_GEP]], i64 -32)
   // LIN64: %[[LOADV5:.+]] = load <8 x i32>, ptr %[[OFAA_ALIGNED]]
   // LIN64: store <8 x i32> %[[LOADV5]], ptr
 
   // LIN32: %[[CUR5:.+]] = load ptr, ptr %[[ARGS]]
-  // LIN32: %[[TOINT:.+]] = ptrtoint ptr %[[CUR5]] to i32
-  // LIN32: %[[ADD:.+]] = add i32 %[[TOINT]], 31
-  // LIN32: %[[AND:.+]] = and i32 %[[ADD]], -32
-  // LIN32: %[[CUR5_ALIGNED:.+]] = inttoptr i32 %[[AND]] to ptr
+
+  // LIN32: [[GEP_CUR5:%.*]] = getelementptr inbounds i8, ptr %[[CUR5]], i32 31
+  // LIN32: %[[CUR5_ALIGNED:.*]] = call ptr @llvm.ptrmask.p0.i32(ptr [[GEP_CUR5]], i32 -32)
   // LIN32: %[[NEXT5:.+]] = getelementptr inbounds i8, ptr %[[CUR5_ALIGNED]], i32 32
   // LIN32: store ptr %[[NEXT5]], ptr %[[ARGS]]
   // LIN32: %[[LOADV5:.+]] = load <8 x i32>, ptr %[[CUR5_ALIGNED]]
diff --git a/clang/test/CodeGenCXX/init-invariant.cpp b/clang/test/CodeGenCXX/init-invariant.cpp
index 3bbea9b32d3e6..fdd5753402ccb 100644
--- a/clang/test/CodeGenCXX/init-invariant.cpp
+++ b/clang/test/CodeGenCXX/init-invariant.cpp
@@ -1,5 +1,5 @@
-// RUN: %clang_cc1 -triple i686-linux-gnu -emit-llvm %s -disable-llvm-passes -o - | FileCheck %s --check-prefix=CHECK-O0
-// RUN: %clang_cc1 -triple i686-linux-gnu -emit-llvm %s -O1 -disable-llvm-passes -o - | FileCheck %s
+// RUN: %clang_cc1 -triple i686-linux-gnu -std=c++20 -emit-llvm %s -disable-llvm-passes -o - | FileCheck %s --check-prefix=CHECK-O0
+// RUN: %clang_cc1 -triple i686-linux-gnu -std=c++20 -emit-llvm %s -O1 -disable-llvm-passes -o - | FileCheck %s
 
 // Check that we add an llvm.invariant.start.p0i8 to mark when a global becomes
 // read-only. If globalopt can fold the initializer, it will then mark the
@@ -16,6 +16,16 @@ struct A {
 // CHECK: @a ={{.*}} global {{.*}} zeroinitializer
 extern const A a = A();
 
+struct A2 {
+  A2();
+  constexpr ~A2() {}
+  int n;
+};
+
+// CHECK: @a2 ={{.*}} global {{.*}} zeroinitializer
+extern const A2 a2 = A2();
+
+
 struct B {
   B();
   mutable int n;
@@ -44,6 +54,9 @@ void e() {
 // CHECK: call void @_ZN1AC1Ev(ptr noundef {{[^,]*}} @a)
 // CHECK: call {{.*}}@llvm.invariant.start.p0(i64 4, ptr @a)
 
+// CHECK: call void @_ZN2A2C1Ev(ptr noundef {{[^,]*}} @a2)
+// CHECK: call {{.*}}@llvm.invariant.start.p0(i64 4, ptr @a2)
+
 // CHECK: call void @_ZN1BC1Ev(ptr noundef {{[^,]*}} @b)
 // CHECK-NOT: call {{.*}}@llvm.invariant.start.p0(i64 noundef 4, ptr @b)
 
diff --git a/clang/test/CodeGenCXX/static-init.cpp b/clang/test/CodeGenCXX/static-init.cpp
index 01c5c5ba9b08d..baf5c12a12aa8 100644
--- a/clang/test/CodeGenCXX/static-init.cpp
+++ b/clang/test/CodeGenCXX/static-init.cpp
@@ -1,5 +1,6 @@
 // RUN: %clang_cc1 %s -triple=x86_64-pc-linuxs -emit-llvm -std=c++98 -o - | FileCheck -check-prefix=CHECK -check-prefix=CHECK98 %s
 // RUN: %clang_cc1 %s -triple=x86_64-pc-linuxs -emit-llvm -std=c++11 -o - | FileCheck -check-prefix=CHECK -check-prefix=CHECK11 %s
+// RUN: %clang_cc1 %s -triple=x86_64-pc-linuxs -emit-llvm -std=c++20 -o - | FileCheck -check-prefix=CHECK -check-prefix=CHECK20 %s
 
 // CHECK: @_ZZ1hvE1i = internal global i32 0, align 4
 // CHECK: @base_req ={{.*}} global [4 x i8] c"foo\00", align 1
@@ -7,6 +8,8 @@
 
 // CHECK: @_ZZN5test31BC1EvE1u = internal global { i8, [3 x i8] } { i8 97, [3 x i8] undef }, align 4
 
+// CHECK20: @_ZZN5test51fEvE1a = internal constant %"struct.test5::A" { i32 42 }
+
 // CHECK: @_ZZ2h2vE1i = linkonce_odr global i32 0, comdat, align 4
 // CHECK: @_ZGVZ2h2vE1i = linkonce_odr global i64 0, comdat, align 8{{$}}
 // CHECK: @_ZZN5test1L6getvarEiE3var = internal constant [4 x i32] [i32 1, i32 0, i32 2, i32 4], align 16
@@ -173,3 +176,18 @@ void useit() {
 // CHECK: define linkonce_odr noundef nonnull align 8 dereferenceable(8) ptr @_ZN5test414useStaticLocalEv()
 // CHECK: ret ptr{{.*}} @_ZZN5test414useStaticLocalEvE3obj
 }
+
+#if __cplusplus >= 202002L
+// A const object with constexpr destructor can be emitted as a constant.
+namespace test5 {
+  struct A {
+    constexpr A(int x) : x_(x) {}
+    constexpr ~A() {}
+    int x_;
+  };
+  const int *f() {
+    static const A a{42};
+    return &a.x_;
+  }
+}
+#endif
diff --git a/clang/test/CodeGenCXX/template-param-objects-linkage.cpp b/clang/test/CodeGenCXX/template-param-objects-linkage.cpp
new file mode 100644
index 0000000000000..63e7d8c646869
--- /dev/null
+++ b/clang/test/CodeGenCXX/template-param-objects-linkage.cpp
@@ -0,0 +1,26 @@
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -std=c++20 %s -emit-llvm -o - | FileCheck %s
+
+struct S { char buf[32]; };
+template<S s> constexpr const char* f() { return s.buf; }
+const char* fbuf = f<S{"a"}>();
+// CHECK: @_ZTAXtl1StlA32_cLc97EEEE = linkonce_odr constant { <{ i8, [31 x i8] }> }
+
+namespace {
+  struct UN { char buf[64]; };
+}
+template <UN un> constexpr const char* g() { return un.buf; }
+const char* gbuf = g<UN{"b"}>();
+// CHECK: @_ZTAXtlN12_GLOBAL__N_12UNEtlA64_cLc98EEEE = internal constant { <{ i8, [63 x i8] }> }
+
+struct Foo { int *i; };
+int m = 0;
+namespace { int n; }
+
+template <Foo foo>
+const int* h() { return foo.i; }
+
+const int* hm = h<Foo{&m}>();
+// CHECK: @_ZTAXtl3FooadL_Z1mEEE = linkonce_odr constant %struct.Foo { ptr @m }
+
+const int* hn = h<Foo{&n}>();
+// CHECK: @_ZTAXtl3FooadL_ZN12_GLOBAL__N_11nEEEE = internal constant %struct.Foo { ptr @_ZN12_GLOBAL__N_11nE }
diff --git a/clang/test/CodeGenCXX/template-param-objects-visibility.cpp b/clang/test/CodeGenCXX/template-param-objects-visibility.cpp
new file mode 100644
index 0000000000000..bf85016b0aff6
--- /dev/null
+++ b/clang/test/CodeGenCXX/template-param-objects-visibility.cpp
@@ -0,0 +1,11 @@
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -std=c++20 %s -emit-llvm -o - | FileCheck %s
+
+struct S { char buf[32]; };
+template<S s> constexpr const char* f() { return s.buf; }
+const char* fbuf = f<S{"a"}>();
+// CHECK: @_ZTAXtl1StlA32_cLc97EEEE = linkonce_odr constant { <{ i8, [31 x i8] }> }
+
+struct __attribute__ ((visibility ("hidden"))) HN { char buf[64]; };
+template <HN hn> constexpr const char* g() { return hn.buf; }
+const char* gbuf = g<HN{"b"}>();
+// CHECK: @_ZTAXtl2HNtlA64_cLc98EEEE = linkonce_odr hidden constant { <{ i8, [63 x i8] }> }
diff --git a/clang/test/CodeGenCXX/template-param-objects.cpp b/clang/test/CodeGenCXX/template-param-objects.cpp
index e36b27185b526..11ebd21521e83 100644
--- a/clang/test/CodeGenCXX/template-param-objects.cpp
+++ b/clang/test/CodeGenCXX/template-param-objects.cpp
@@ -7,7 +7,9 @@ template<S s> constexpr const char *end() { return s.buf + __builtin_strlen(s.bu
 
 // ITANIUM: [[HELLO:@_ZTAXtl1StlA32_cLc104ELc101ELc108ELc108ELc111ELc32ELc119ELc111ELc114ELc108ELc100EEEE]]
 // MSABI: [[HELLO:@"[?][?]__N2US@@3D0GI@@0GF@@0GM@@0GM@@0GP@@0CA@@0HH@@0GP@@0HC@@0GM@@0GE@@0A@@0A@@0A@@0A@@0A@@0A@@0A@@0A@@0A@@0A@@0A@@0A@@0A@@0A@@0A@@0A@@0A@@0A@@0A@@0A@@0A@@@@@"]]
-// CHECK-SAME: = linkonce_odr constant { <{ [11 x i8], [21 x i8] }> } { <{ [11 x i8], [21 x i8] }> <{ [11 x i8] c"hello world", [21 x i8] zeroinitializer }> }, comdat
+// ITANIUM-SAME: = linkonce_odr constant { <{ [11 x i8], [21 x i8] }> } { <{ [11 x i8], [21 x i8] }> <{ [11 x i8] c"hello world", [21 x i8] zeroinitializer }> }, comdat
+// MSABI-SAME: = linkonce_odr dso_local constant { <{ [11 x i8], [21 x i8] }> } { <{ [11 x i8], [21 x i8] }> <{ [11 x i8] c"hello world", [21 x i8] zeroinitializer }> }, comdat
+
 
 // ITANIUM: @p
 // MSABI: @"?p@@3PEBDEB"
diff --git a/clang/test/CodeGenOpenCL/cast_image.cl b/clang/test/CodeGenOpenCL/cast_image.cl
index af68a7e720742..51f46fa7712ed 100644
--- a/clang/test/CodeGenOpenCL/cast_image.cl
+++ b/clang/test/CodeGenOpenCL/cast_image.cl
@@ -1,5 +1,5 @@
 // RUN: %clang_cc1 -no-opaque-pointers -emit-llvm -o - -triple amdgcn--amdhsa %s | FileCheck --check-prefix=AMDGCN %s
-// RUN: %clang_cc1 -no-opaque-pointers -emit-llvm -o - -triple spir-unknown-unknown %s | FileCheck --check-prefix=SPIR %s
+// RUN: %clang_cc1 -no-opaque-pointers -emit-llvm -o - -triple x86_64-unknown-unknown %s | FileCheck --check-prefix=X86 %s
 
 #ifdef __AMDGCN__
 
@@ -11,7 +11,7 @@ constant int* convert(image2d_t img) {
 #else
 
 global int* convert(image2d_t img) {
-  // SPIR: bitcast %opencl.image2d_ro_t addrspace(1)* %img to i32 addrspace(1)*
+  // X86: bitcast %opencl.image2d_ro_t* %img to i32*
   return __builtin_astype(img, global int*);
 }
 
diff --git a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
index a917d128d09d0..88fafbbc98f43 100644
--- a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
+++ b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
@@ -27,7 +27,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // SPIR32-NEXT:    [[A_ADDR:%.*]] = alloca ptr addrspace(1), align 4
 // SPIR32-NEXT:    [[B_ADDR:%.*]] = alloca ptr addrspace(1), align 4
 // SPIR32-NEXT:    [[I_ADDR:%.*]] = alloca i32, align 4
-// SPIR32-NEXT:    [[DEFAULT_QUEUE:%.*]] = alloca ptr, align 4
+// SPIR32-NEXT:    [[DEFAULT_QUEUE:%.*]] = alloca target("spirv.Queue"), align 4
 // SPIR32-NEXT:    [[FLAGS:%.*]] = alloca i32, align 4
 // SPIR32-NEXT:    [[NDRANGE:%.*]] = alloca [[STRUCT_NDRANGE_T:%.*]], align 4
 // SPIR32-NEXT:    [[TMP:%.*]] = alloca [[STRUCT_NDRANGE_T]], align 4
@@ -36,7 +36,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // SPIR32-NEXT:    store ptr addrspace(1) [[B]], ptr [[B_ADDR]], align 4
 // SPIR32-NEXT:    store i32 [[I]], ptr [[I_ADDR]], align 4
 // SPIR32-NEXT:    store i32 0, ptr [[FLAGS]], align 4
-// SPIR32-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[DEFAULT_QUEUE]], align 4
+// SPIR32-NEXT:    [[TMP0:%.*]] = load target("spirv.Queue"), ptr [[DEFAULT_QUEUE]], align 4
 // SPIR32-NEXT:    [[TMP1:%.*]] = load i32, ptr [[FLAGS]], align 4
 // SPIR32-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[TMP]], ptr align 4 [[NDRANGE]], i32 4, i1 false)
 // SPIR32-NEXT:    [[BLOCK_SIZE:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 0
@@ -55,7 +55,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // SPIR32-NEXT:    [[TMP4:%.*]] = load ptr addrspace(1), ptr [[B_ADDR]], align 4
 // SPIR32-NEXT:    store ptr addrspace(1) [[TMP4]], ptr [[BLOCK_CAPTURED2]], align 4
 // SPIR32-NEXT:    [[TMP5:%.*]] = addrspacecast ptr [[BLOCK]] to ptr addrspace(4)
-// SPIR32-NEXT:    [[TMP6:%.*]] = call spir_func i32 @__enqueue_kernel_basic(ptr [[TMP0]], i32 [[TMP1]], ptr byval([[STRUCT_NDRANGE_T]]) [[TMP]], ptr addrspace(4) addrspacecast (ptr @__device_side_enqueue_block_invoke_kernel to ptr addrspace(4)), ptr addrspace(4) [[TMP5]])
+// SPIR32-NEXT:    [[TMP6:%.*]] = call spir_func i32 @__enqueue_kernel_basic(target("spirv.Queue") [[TMP0]], i32 [[TMP1]], ptr byval([[STRUCT_NDRANGE_T]]) [[TMP]], ptr addrspace(4) addrspacecast (ptr @__device_side_enqueue_block_invoke_kernel to ptr addrspace(4)), ptr addrspace(4) [[TMP5]])
 // SPIR32-NEXT:    ret void
 //
 //
@@ -98,7 +98,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP-NEXT:    [[A_ADDR:%.*]] = alloca ptr addrspace(1), align 4
 // STRICTFP-NEXT:    [[B_ADDR:%.*]] = alloca ptr addrspace(1), align 4
 // STRICTFP-NEXT:    [[I_ADDR:%.*]] = alloca i32, align 4
-// STRICTFP-NEXT:    [[DEFAULT_QUEUE:%.*]] = alloca ptr, align 4
+// STRICTFP-NEXT:    [[DEFAULT_QUEUE:%.*]] = alloca target("spirv.Queue"), align 4
 // STRICTFP-NEXT:    [[FLAGS:%.*]] = alloca i32, align 4
 // STRICTFP-NEXT:    [[NDRANGE:%.*]] = alloca [[STRUCT_NDRANGE_T:%.*]], align 4
 // STRICTFP-NEXT:    [[TMP:%.*]] = alloca [[STRUCT_NDRANGE_T]], align 4
@@ -107,7 +107,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP-NEXT:    store ptr addrspace(1) [[B]], ptr [[B_ADDR]], align 4
 // STRICTFP-NEXT:    store i32 [[I]], ptr [[I_ADDR]], align 4
 // STRICTFP-NEXT:    store i32 0, ptr [[FLAGS]], align 4
-// STRICTFP-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[DEFAULT_QUEUE]], align 4
+// STRICTFP-NEXT:    [[TMP0:%.*]] = load target("spirv.Queue"), ptr [[DEFAULT_QUEUE]], align 4
 // STRICTFP-NEXT:    [[TMP1:%.*]] = load i32, ptr [[FLAGS]], align 4
 // STRICTFP-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[TMP]], ptr align 4 [[NDRANGE]], i32 4, i1 false) #[[ATTR5:[0-9]+]]
 // STRICTFP-NEXT:    [[BLOCK_SIZE:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 0
@@ -126,7 +126,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP-NEXT:    [[TMP4:%.*]] = load ptr addrspace(1), ptr [[B_ADDR]], align 4
 // STRICTFP-NEXT:    store ptr addrspace(1) [[TMP4]], ptr [[BLOCK_CAPTURED2]], align 4
 // STRICTFP-NEXT:    [[TMP5:%.*]] = addrspacecast ptr [[BLOCK]] to ptr addrspace(4)
-// STRICTFP-NEXT:    [[TMP6:%.*]] = call spir_func i32 @__enqueue_kernel_basic(ptr [[TMP0]], i32 [[TMP1]], ptr byval([[STRUCT_NDRANGE_T]]) [[TMP]], ptr addrspace(4) addrspacecast (ptr @__device_side_enqueue_block_invoke_kernel to ptr addrspace(4)), ptr addrspace(4) [[TMP5]])
+// STRICTFP-NEXT:    [[TMP6:%.*]] = call spir_func i32 @__enqueue_kernel_basic(target("spirv.Queue") [[TMP0]], i32 [[TMP1]], ptr byval([[STRUCT_NDRANGE_T]]) [[TMP]], ptr addrspace(4) addrspacecast (ptr @__device_side_enqueue_block_invoke_kernel to ptr addrspace(4)), ptr addrspace(4) [[TMP5]])
 // STRICTFP-NEXT:    ret void
 //
 //
diff --git a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue.cl b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue.cl
index 05a15e476f211..f0c164795b764 100644
--- a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue.cl
+++ b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue.cl
@@ -1,9 +1,12 @@
-// RUN: %clang_cc1 -no-enable-noundef-analysis %s -cl-std=CL2.0 -ffake-address-space-map -O0 -emit-llvm -o - -triple "spir-unknown-unknown" | FileCheck %s --check-prefix=COMMON --check-prefix=B32
-// RUN: %clang_cc1 -no-enable-noundef-analysis %s -cl-std=CL2.0 -ffake-address-space-map -O0 -emit-llvm -o - -triple "spir64-unknown-unknown" | FileCheck %s --check-prefix=COMMON --check-prefix=B64
+// RUN: %clang_cc1 -no-enable-noundef-analysis %s -cl-std=CL2.0 -ffake-address-space-map -O0 -emit-llvm -o - -triple "spir-unknown-unknown" | FileCheck %s --check-prefixes=COMMON,B32,SPIR
+// RUN: %clang_cc1 -no-enable-noundef-analysis %s -cl-std=CL2.0 -ffake-address-space-map -O0 -emit-llvm -o - -triple "spir64-unknown-unknown" | FileCheck %s --check-prefixes=COMMON,B64,SPIR
 // RUN: %clang_cc1 -no-enable-noundef-analysis %s -cl-std=CL2.0 -ffake-address-space-map -O1 -emit-llvm -o - -triple "spir64-unknown-unknown" | FileCheck %s --check-prefix=CHECK-LIFETIMES
-// RUN: %clang_cc1 -no-enable-noundef-analysis %s -cl-std=CL3.0 -ffake-address-space-map -O0 -emit-llvm -o - -triple "spir-unknown-unknown" | FileCheck %s --check-prefix=COMMON --check-prefix=B32
-// RUN: %clang_cc1 -no-enable-noundef-analysis %s -cl-std=CL3.0 -ffake-address-space-map -O0 -emit-llvm -o - -triple "spir64-unknown-unknown" | FileCheck %s --check-prefix=COMMON --check-prefix=B64
+// RUN: %clang_cc1 -no-enable-noundef-analysis %s -cl-std=CL3.0 -ffake-address-space-map -O0 -emit-llvm -o - -triple "spir-unknown-unknown" | FileCheck %s --check-prefixes=COMMON,B32,SPIR
+// RUN: %clang_cc1 -no-enable-noundef-analysis %s -cl-std=CL3.0 -ffake-address-space-map -O0 -emit-llvm -o - -triple "spir64-unknown-unknown" | FileCheck %s --check-prefixes=COMMON,B64,SPIR
 // RUN: %clang_cc1 -no-enable-noundef-analysis %s -cl-std=CL3.0 -ffake-address-space-map -O1 -emit-llvm -o - -triple "spir64-unknown-unknown" | FileCheck %s --check-prefix=CHECK-LIFETIMES
+// RUN: %clang_cc1 -no-enable-noundef-analysis %s -cl-std=CL2.0 -ffake-address-space-map -O0 -emit-llvm -o - -triple "x86_64-unknown-linux-gnu" | FileCheck %s --check-prefixes=COMMON,B64,X86
+// RUN: %clang_cc1 -no-enable-noundef-analysis %s -cl-std=CL3.0 -ffake-address-space-map -O0 -emit-llvm -o - -triple "x86_64-unknown-linux-gnu" | FileCheck %s --check-prefixes=COMMON,B64,X86
+// RUN: %clang_cc1 -no-enable-noundef-analysis %s -cl-std=CL3.0 -ffake-address-space-map -O1 -emit-llvm -o - -triple "x86_64-unknown-linux-gnu" | FileCheck %s --check-prefix=CHECK-LIFETIMES
 
 #pragma OPENCL EXTENSION cl_khr_subgroups : enable
 
@@ -29,7 +32,7 @@ typedef struct {int a;} ndrange_t;
 // COMMON: [[BLG11:@__block_literal_global[^ ]*]] = internal addrspace(1) constant { i32, i32, ptr addrspace(4) } { i32 {{[0-9]+}}, i32 {{[0-9]+}}, ptr addrspace(4) addrspacecast (ptr {{@[^ ]+}} to ptr addrspace(4)) }
 
 // Emits block literal [[BL_GLOBAL]], invoke function [[INV_G]] and global block variable @block_G
-// COMMON: define internal spir_func void [[INV_G]](ptr addrspace(4) %{{.*}}, ptr addrspace(3) %{{.*}})
+// COMMON: define internal {{(spir_func )?}}void [[INV_G]](ptr addrspace(4) %{{.*}}, ptr addrspace(3) %{{.*}})
 const bl_t block_G = (bl_t) ^ (local void *a) {};
 
 void callee(int id, __global int *out) {
@@ -38,17 +41,21 @@ void callee(int id, __global int *out) {
 
 // COMMON-LABEL: define{{.*}} spir_kernel void @device_side_enqueue(ptr addrspace(1) align 4 %{{.*}}, ptr addrspace(1) align 4 %b, i32 %i)
 kernel void device_side_enqueue(global int *a, global int *b, int i) {
-  // COMMON: %default_queue = alloca ptr
+  // SPIR: %default_queue = alloca target("spirv.Queue")
+  // X86: %default_queue = alloca ptr
   queue_t default_queue;
   // COMMON: %flags = alloca i32
   unsigned flags = 0;
   // COMMON: %ndrange = alloca %struct.ndrange_t
   ndrange_t ndrange;
-  // COMMON: %clk_event = alloca ptr
+  // SPIR: %clk_event = alloca target("spirv.DeviceEvent")
+  // X86: %clk_event = alloca ptr
   clk_event_t clk_event;
-  // COMMON: %event_wait_list = alloca ptr
+  // SPIR: %event_wait_list = alloca target("spirv.DeviceEvent")
+  // X86: %event_wait_list = alloca ptr
   clk_event_t event_wait_list;
-  // COMMON: %event_wait_list2 = alloca [1 x ptr]
+  // SPIR: %event_wait_list2 = alloca [1 x target("spirv.DeviceEvent")]
+  // X86: %event_wait_list2 = alloca [1 x ptr]
   clk_event_t event_wait_list2[] = {clk_event};
 
   // COMMON: [[NDR:%[a-z0-9]+]] = alloca %struct.ndrange_t, align 4
@@ -76,12 +83,14 @@ kernel void device_side_enqueue(global int *a, global int *b, int i) {
   // CHECK-LIFETIMES: %[[BLOCK_SIZES7:.*]] = alloca [1 x i64]
 
   // Emits block literal on stack and block kernel [[INVLK1]].
-  // COMMON: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
+  // SPIR: [[DEF_Q:%[0-9]+]] = load target("spirv.Queue"), ptr %default_queue
+  // X86: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
   // COMMON: [[FLAGS:%[0-9]+]] = load i32, ptr %flags
   // COMMON: store ptr addrspace(4) addrspacecast (ptr [[INVL1:@__device_side_enqueue_block_invoke[^ ]*]] to ptr addrspace(4)), ptr %block.invoke
   // COMMON: [[BL_I8:%[0-9]+]] ={{.*}} addrspacecast ptr %block to ptr addrspace(4)
-  // COMMON-LABEL: call spir_func i32 @__enqueue_kernel_basic(
-  // COMMON-SAME: ptr [[DEF_Q]], i32 [[FLAGS]], ptr byval(%struct.ndrange_t) [[NDR]]{{([0-9]+)?}},
+  // COMMON-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_basic(
+  // SPIR-SAME: target("spirv.Queue") [[DEF_Q]], i32 [[FLAGS]], ptr byval(%struct.ndrange_t) [[NDR]]{{([0-9]+)?}},
+  // X86-SAME: ptr [[DEF_Q]], i32 [[FLAGS]], ptr byval(%struct.ndrange_t) [[NDR]]{{([0-9]+)?}},
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr [[INVLK1:[^ ]+_kernel]] to ptr addrspace(4)),
   // COMMON-SAME: ptr addrspace(4) [[BL_I8]])
   enqueue_kernel(default_queue, flags, ndrange,
@@ -90,14 +99,16 @@ kernel void device_side_enqueue(global int *a, global int *b, int i) {
                  });
 
   // Emits block literal on stack and block kernel [[INVLK2]].
-  // COMMON: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
+  // SPIR: [[DEF_Q:%[0-9]+]] = load target("spirv.Queue"), ptr %default_queue
+  // X86: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
   // COMMON: [[FLAGS:%[0-9]+]] = load i32, ptr %flags
   // COMMON: [[WAIT_EVNT:%[0-9]+]] ={{.*}} addrspacecast ptr %event_wait_list to ptr addrspace(4)
   // COMMON: [[EVNT:%[0-9]+]] ={{.*}} addrspacecast ptr %clk_event to ptr addrspace(4)
   // COMMON: store ptr addrspace(4) addrspacecast (ptr [[INVL2:@__device_side_enqueue_block_invoke[^ ]*]] to ptr addrspace(4)), ptr %block.invoke
   // COMMON: [[BL_I8:%[0-9]+]] ={{.*}} addrspacecast ptr %block4 to ptr addrspace(4)
-  // COMMON-LABEL: call spir_func i32 @__enqueue_kernel_basic_events
-  // COMMON-SAME: (ptr [[DEF_Q]], i32 [[FLAGS]],  ptr {{.*}}, i32 2, ptr addrspace(4) [[WAIT_EVNT]], ptr addrspace(4) [[EVNT]],
+  // COMMON-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_basic_events
+  // SPIR-SAME: (target("spirv.Queue") [[DEF_Q]], i32 [[FLAGS]],  ptr {{.*}}, i32 2, ptr addrspace(4) [[WAIT_EVNT]], ptr addrspace(4) [[EVNT]],
+  // X86-SAME: (ptr [[DEF_Q]], i32 [[FLAGS]],  ptr {{.*}}, i32 2, ptr addrspace(4) [[WAIT_EVNT]], ptr addrspace(4) [[EVNT]],
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr [[INVLK2:[^ ]+_kernel]] to ptr addrspace(4)),
   // COMMON-SAME: ptr addrspace(4) [[BL_I8]])
   enqueue_kernel(default_queue, flags, ndrange, 2, &event_wait_list, &clk_event,
@@ -105,25 +116,28 @@ kernel void device_side_enqueue(global int *a, global int *b, int i) {
                    a[i] = b[i];
                  });
 
-  // COMMON-LABEL: call spir_func i32 @__enqueue_kernel_basic_events
-  // COMMON-SAME: (ptr {{%[0-9]+}}, i32 {{%[0-9]+}}, ptr {{.*}}, i32 1, ptr addrspace(4) null, ptr addrspace(4) null,
+  // COMMON-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_basic_events
+  // SPIR-SAME: (target("spirv.Queue") {{%[0-9]+}}, i32 {{%[0-9]+}}, ptr {{.*}}, i32 1, ptr addrspace(4) null, ptr addrspace(4) null,
+  // X86-SAME: (ptr {{%[0-9]+}}, i32 {{%[0-9]+}}, ptr {{.*}}, i32 1, ptr addrspace(4) null, ptr addrspace(4) null,
   enqueue_kernel(default_queue, flags, ndrange, 1, 0, 0,
                  ^(void) {
                    return;
                  });
 
   // Emits global block literal [[BLG1]] and block kernel [[INVGK1]].
-  // COMMON: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
+  // SPIR: [[DEF_Q:%[0-9]+]] = load target("spirv.Queue"), ptr %default_queue
+  // X86: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
   // COMMON: [[FLAGS:%[0-9]+]] = load i32, ptr %flags
   // CHECK-LIFETIMES: call void @llvm.lifetime.start.p0(i64 8, ptr nonnull %[[BLOCK_SIZES1]])
-  // CHECK-LIFETIMES-LABEL: call spir_func i32 @__enqueue_kernel_varargs(
+  // CHECK-LIFETIMES-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_varargs(
   // CHECK-LIFETIMES-NEXT: call void @llvm.lifetime.end.p0(i64 8, ptr nonnull %[[BLOCK_SIZES1]])
   // B32: %[[TMP:.*]] = getelementptr [1 x i32], ptr %[[BLOCK_SIZES1]], i32 0, i32 0
   // B32: store i32 256, ptr %[[TMP]], align 4
   // B64: %[[TMP:.*]] = getelementptr [1 x i64], ptr %[[BLOCK_SIZES1]], i32 0, i32 0
   // B64: store i64 256, ptr %[[TMP]], align 8
-  // COMMON-LABEL: call spir_func i32 @__enqueue_kernel_varargs(
-  // COMMON-SAME: ptr [[DEF_Q]], i32 [[FLAGS]], ptr [[NDR]]{{([0-9]+)?}},
+  // COMMON-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_varargs(
+  // SPIR-SAME: target("spirv.Queue") [[DEF_Q]], i32 [[FLAGS]], ptr [[NDR]]{{([0-9]+)?}},
+  // X86-SAME: ptr [[DEF_Q]], i32 [[FLAGS]], ptr [[NDR]]{{([0-9]+)?}},
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr [[INVGK1:[^ ]+_kernel]] to ptr addrspace(4)),
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BLG1]] to ptr addrspace(4)), i32 1,
   // B32-SAME: ptr %[[TMP]])
@@ -136,17 +150,19 @@ kernel void device_side_enqueue(global int *a, global int *b, int i) {
 
   char c;
   // Emits global block literal [[BLG2]] and block kernel [[INVGK2]].
-  // COMMON: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
+  // SPIR: [[DEF_Q:%[0-9]+]] = load target("spirv.Queue"), ptr %default_queue
+  // X86: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
   // COMMON: [[FLAGS:%[0-9]+]] = load i32, ptr %flags
   // CHECK-LIFETIMES-NEXT: call void @llvm.lifetime.start.p0(i64 8, ptr nonnull %[[BLOCK_SIZES2]])
-  // CHECK-LIFETIMES-LABEL: call spir_func i32 @__enqueue_kernel_varargs(
+  // CHECK-LIFETIMES-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_varargs(
   // CHECK-LIFETIMES-NEXT: call void @llvm.lifetime.end.p0(i64 8, ptr nonnull %[[BLOCK_SIZES2]])
   // B32: %[[TMP:.*]] = getelementptr [1 x i32], ptr %[[BLOCK_SIZES2]], i32 0, i32 0
   // B32: store i32 %{{.*}}, ptr %[[TMP]], align 4
   // B64: %[[TMP:.*]] = getelementptr [1 x i64], ptr %[[BLOCK_SIZES2]], i32 0, i32 0
   // B64: store i64 %{{.*}}, ptr %[[TMP]], align 8
-  // COMMON-LABEL: call spir_func i32 @__enqueue_kernel_varargs(
-  // COMMON-SAME: ptr [[DEF_Q]], i32 [[FLAGS]], ptr [[NDR]]{{([0-9]+)?}},
+  // COMMON-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_varargs(
+  // SPIR-SAME: target("spirv.Queue") [[DEF_Q]], i32 [[FLAGS]], ptr [[NDR]]{{([0-9]+)?}},
+  // X86-SAME: ptr [[DEF_Q]], i32 [[FLAGS]], ptr [[NDR]]{{([0-9]+)?}},
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr [[INVGK2:[^ ]+_kernel]] to ptr addrspace(4)),
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BLG2]] to ptr addrspace(4)), i32 1,
   // B32-SAME: ptr %[[TMP]])
@@ -158,20 +174,23 @@ kernel void device_side_enqueue(global int *a, global int *b, int i) {
                  c);
 
   // Emits global block literal [[BLG3]] and block kernel [[INVGK3]].
-  // COMMON: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
+  // SPIR: [[DEF_Q:%[0-9]+]] = load target("spirv.Queue"), ptr %default_queue
+  // X86: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
   // COMMON: [[FLAGS:%[0-9]+]] = load i32, ptr %flags
-  // COMMON: [[AD:%arraydecay[0-9]*]] = getelementptr inbounds [1 x ptr], ptr %event_wait_list2, i{{32|64}} 0, i{{32|64}} 0
+  // SPIR: [[AD:%arraydecay[0-9]*]] = getelementptr inbounds [1 x target("spirv.DeviceEvent")], ptr %event_wait_list2, i{{32|64}} 0, i{{32|64}} 0
+  // X86: [[AD:%arraydecay[0-9]*]] = getelementptr inbounds [1 x ptr], ptr %event_wait_list2, i{{32|64}} 0, i{{32|64}} 0
   // COMMON: [[WAIT_EVNT:%[0-9]+]] ={{.*}} addrspacecast ptr [[AD]] to ptr addrspace(4)
   // COMMON: [[EVNT:%[0-9]+]]  ={{.*}} addrspacecast ptr %clk_event to ptr addrspace(4)
   // CHECK-LIFETIMES: call void @llvm.lifetime.start.p0(i64 8, ptr nonnull %[[BLOCK_SIZES3]])
-  // CHECK-LIFETIMES-LABEL: call spir_func i32 @__enqueue_kernel_events_varargs(
+  // CHECK-LIFETIMES-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_events_varargs(
   // CHECK-LIFETIMES-NEXT: call void @llvm.lifetime.end.p0(i64 8, ptr nonnull %[[BLOCK_SIZES3]])
   // B32: %[[TMP:.*]] = getelementptr [1 x i32], ptr %[[BLOCK_SIZES3]], i32 0, i32 0
   // B32: store i32 256, ptr %[[TMP]], align 4
   // B64: %[[TMP:.*]] = getelementptr [1 x i64], ptr %[[BLOCK_SIZES3]], i32 0, i32 0
   // B64: store i64 256, ptr %[[TMP]], align 8
-  // COMMON-LABEL: call spir_func i32 @__enqueue_kernel_events_varargs
-  // COMMON-SAME: (ptr [[DEF_Q]], i32 [[FLAGS]],  ptr {{.*}}, i32 2, ptr addrspace(4) [[WAIT_EVNT]], ptr addrspace(4) [[EVNT]],
+  // COMMON-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_events_varargs
+  // SPIR-SAME: (target("spirv.Queue") [[DEF_Q]], i32 [[FLAGS]],  ptr {{.*}}, i32 2, ptr addrspace(4) [[WAIT_EVNT]], ptr addrspace(4) [[EVNT]],
+  // X86-SAME: (ptr [[DEF_Q]], i32 [[FLAGS]],  ptr {{.*}}, i32 2, ptr addrspace(4) [[WAIT_EVNT]], ptr addrspace(4) [[EVNT]],
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr [[INVGK3:[^ ]+_kernel]] to ptr addrspace(4)),
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BLG3]] to ptr addrspace(4)), i32 1,
   // B32-SAME: ptr %[[TMP]])
@@ -183,20 +202,23 @@ kernel void device_side_enqueue(global int *a, global int *b, int i) {
                  256);
 
   // Emits global block literal [[BLG4]] and block kernel [[INVGK4]].
-  // COMMON: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
+  // SPIR: [[DEF_Q:%[0-9]+]] = load target("spirv.Queue"), ptr %default_queue
+  // X86: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
   // COMMON: [[FLAGS:%[0-9]+]] = load i32, ptr %flags
-  // COMMON: [[AD:%arraydecay[0-9]*]] = getelementptr inbounds [1 x ptr], ptr %event_wait_list2, i{{32|64}} 0, i{{32|64}} 0
+  // SPIR: [[AD:%arraydecay[0-9]*]] = getelementptr inbounds [1 x target("spirv.DeviceEvent")], ptr %event_wait_list2, i{{32|64}} 0, i{{32|64}} 0
+  // X86: [[AD:%arraydecay[0-9]*]] = getelementptr inbounds [1 x ptr], ptr %event_wait_list2, i{{32|64}} 0, i{{32|64}} 0
   // COMMON: [[WAIT_EVNT:%[0-9]+]] ={{.*}} addrspacecast ptr [[AD]] to ptr addrspace(4)
   // COMMON: [[EVNT:%[0-9]+]]  ={{.*}} addrspacecast ptr %clk_event to ptr addrspace(4)
   // CHECK-LIFETIMES-NEXT: call void @llvm.lifetime.start.p0(i64 8, ptr nonnull %[[BLOCK_SIZES4]])
-  // CHECK-LIFETIMES-LABEL: call spir_func i32 @__enqueue_kernel_events_varargs(
+  // CHECK-LIFETIMES-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_events_varargs(
   // CHECK-LIFETIMES-NEXT: call void @llvm.lifetime.end.p0(i64 8, ptr nonnull %[[BLOCK_SIZES4]])
   // B32: %[[TMP:.*]] = getelementptr [1 x i32], ptr %[[BLOCK_SIZES4]], i32 0, i32 0
   // B32: store i32 %{{.*}}, ptr %[[TMP]], align 4
   // B64: %[[TMP:.*]] = getelementptr [1 x i64], ptr %[[BLOCK_SIZES4]], i32 0, i32 0
   // B64: store i64 %{{.*}}, ptr %[[TMP]], align 8
-  // COMMON-LABEL: call spir_func i32 @__enqueue_kernel_events_varargs
-  // COMMON-SAME: (ptr [[DEF_Q]], i32 [[FLAGS]],  ptr {{.*}}, i32 2, ptr addrspace(4) [[WAIT_EVNT]], ptr addrspace(4) [[EVNT]],
+  // COMMON-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_events_varargs
+  // SPIR-SAME: (target("spirv.Queue") [[DEF_Q]], i32 [[FLAGS]],  ptr {{.*}}, i32 2, ptr addrspace(4) [[WAIT_EVNT]], ptr addrspace(4) [[EVNT]],
+  // X86-SAME: (ptr [[DEF_Q]], i32 [[FLAGS]],  ptr {{.*}}, i32 2, ptr addrspace(4) [[WAIT_EVNT]], ptr addrspace(4) [[EVNT]],
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr [[INVGK4:[^ ]+_kernel]] to ptr addrspace(4)),
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BLG4]] to ptr addrspace(4)), i32 1,
   // B32-SAME: ptr %[[TMP]])
@@ -209,17 +231,19 @@ kernel void device_side_enqueue(global int *a, global int *b, int i) {
 
   long l;
   // Emits global block literal [[BLG5]] and block kernel [[INVGK5]].
-  // COMMON: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
+  // SPIR: [[DEF_Q:%[0-9]+]] = load target("spirv.Queue"), ptr %default_queue
+  // X86: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
   // COMMON: [[FLAGS:%[0-9]+]] = load i32, ptr %flags
   // CHECK-LIFETIMES-NEXT: call void @llvm.lifetime.start.p0(i64 8, ptr nonnull %[[BLOCK_SIZES5]])
-  // CHECK-LIFETIMES-LABEL: call spir_func i32 @__enqueue_kernel_varargs(
+  // CHECK-LIFETIMES-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_varargs(
   // CHECK-LIFETIMES-NEXT: call void @llvm.lifetime.end.p0(i64 8, ptr nonnull %[[BLOCK_SIZES5]])
   // B32: %[[TMP:.*]] = getelementptr [1 x i32], ptr %[[BLOCK_SIZES5]], i32 0, i32 0
   // B32: store i32 %{{.*}}, ptr %[[TMP]], align 4
   // B64: %[[TMP:.*]] = getelementptr [1 x i64], ptr %[[BLOCK_SIZES5]], i32 0, i32 0
   // B64: store i64 %{{.*}}, ptr %[[TMP]], align 8
-  // COMMON-LABEL: call spir_func i32 @__enqueue_kernel_varargs
-  // COMMON-SAME: (ptr [[DEF_Q]], i32 [[FLAGS]], ptr [[NDR]]{{([0-9]+)?}},
+  // COMMON-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_varargs
+  // SPIR-SAME: (target("spirv.Queue") [[DEF_Q]], i32 [[FLAGS]], ptr [[NDR]]{{([0-9]+)?}},
+  // X86-SAME: (ptr [[DEF_Q]], i32 [[FLAGS]], ptr [[NDR]]{{([0-9]+)?}},
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr [[INVGK5:[^ ]+_kernel]] to ptr addrspace(4)),
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BLG5]] to ptr addrspace(4)), i32 1,
   // B32-SAME: ptr %[[TMP]])
@@ -231,10 +255,11 @@ kernel void device_side_enqueue(global int *a, global int *b, int i) {
                  l);
 
   // Emits global block literal [[BLG6]] and block kernel [[INVGK6]].
-  // COMMON: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
+  // SPIR: [[DEF_Q:%[0-9]+]] = load target("spirv.Queue"), ptr %default_queue
+  // X86: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
   // COMMON: [[FLAGS:%[0-9]+]] = load i32, ptr %flags
   // CHECK-LIFETIMES-NEXT: call void @llvm.lifetime.start.p0(i64 24, ptr nonnull %[[BLOCK_SIZES6]])
-  // CHECK-LIFETIMES-LABEL: call spir_func i32 @__enqueue_kernel_varargs(
+  // CHECK-LIFETIMES-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_varargs(
   // CHECK-LIFETIMES-NEXT: call void @llvm.lifetime.end.p0(i64 24, ptr nonnull %[[BLOCK_SIZES6]])
   // B32: %[[TMP:.*]] = getelementptr [3 x i32], ptr %[[BLOCK_SIZES6]], i32 0, i32 0
   // B32: store i32 1, ptr %[[TMP]], align 4
@@ -248,8 +273,9 @@ kernel void device_side_enqueue(global int *a, global int *b, int i) {
   // B64: store i64 2, ptr %[[BLOCK_SIZES62]], align 8
   // B64: %[[BLOCK_SIZES63:.*]] = getelementptr [3 x i64], ptr %[[BLOCK_SIZES6]], i32 0, i32 2
   // B64: store i64 4, ptr %[[BLOCK_SIZES63]], align 8
-  // COMMON-LABEL: call spir_func i32 @__enqueue_kernel_varargs
-  // COMMON-SAME: (ptr [[DEF_Q]], i32 [[FLAGS]], ptr [[NDR]]{{([0-9]+)?}},
+  // COMMON-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_varargs
+  // SPIR-SAME: (target("spirv.Queue") [[DEF_Q]], i32 [[FLAGS]], ptr [[NDR]]{{([0-9]+)?}},
+  // X86-SAME: (ptr [[DEF_Q]], i32 [[FLAGS]], ptr [[NDR]]{{([0-9]+)?}},
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr [[INVGK6:[^ ]+_kernel]] to ptr addrspace(4)),
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BLG6]] to ptr addrspace(4)), i32 3,
   // B32-SAME: ptr %[[TMP]])
@@ -261,17 +287,19 @@ kernel void device_side_enqueue(global int *a, global int *b, int i) {
                  1, 2, 4);
 
   // Emits global block literal [[BLG7]] and block kernel [[INVGK7]].
-  // COMMON: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
+  // SPIR: [[DEF_Q:%[0-9]+]] = load target("spirv.Queue"), ptr %default_queue
+  // X86: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
   // COMMON: [[FLAGS:%[0-9]+]] = load i32, ptr %flags
   // CHECK-LIFETIMES-NEXT: call void @llvm.lifetime.start.p0(i64 8, ptr nonnull %[[BLOCK_SIZES7]])
-  // CHECK-LIFETIMES-LABEL: call spir_func i32 @__enqueue_kernel_varargs(
+  // CHECK-LIFETIMES-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_varargs(
   // CHECK-LIFETIMES-NEXT: call void @llvm.lifetime.end.p0(i64 8, ptr nonnull %[[BLOCK_SIZES7]])
   // B32: %[[TMP:.*]] = getelementptr [1 x i32], ptr %[[BLOCK_SIZES7]], i32 0, i32 0
   // B32: store i32 0, ptr %[[TMP]], align 4
   // B64: %[[TMP:.*]] = getelementptr [1 x i64], ptr %[[BLOCK_SIZES7]], i32 0, i32 0
   // B64: store i64 4294967296, ptr %[[TMP]], align 8
-  // COMMON-LABEL: call spir_func i32 @__enqueue_kernel_varargs
-  // COMMON-SAME: (ptr [[DEF_Q]], i32 [[FLAGS]], ptr [[NDR]]{{([0-9]+)?}},
+  // COMMON-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_varargs
+  // SPIR-SAME: (target("spirv.Queue") [[DEF_Q]], i32 [[FLAGS]], ptr [[NDR]]{{([0-9]+)?}},
+  // X86-SAME: (ptr [[DEF_Q]], i32 [[FLAGS]], ptr [[NDR]]{{([0-9]+)?}},
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr [[INVGK7:[^ ]+_kernel]] to ptr addrspace(4)),
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BLG7]] to ptr addrspace(4)), i32 1,
   // B32-SAME: ptr %[[TMP]])
@@ -297,26 +325,28 @@ kernel void device_side_enqueue(global int *a, global int *b, int i) {
   };
 
   // Uses global block literal [[BLG8]] and invoke function [[INVG8]].
-  // COMMON: call spir_func void @__device_side_enqueue_block_invoke_11(ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BLG8]] to ptr addrspace(4))) [[INVOKE_ATTR:#[0-9]+]]
+  // COMMON: call {{(spir_func )?}}void @__device_side_enqueue_block_invoke_11(ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BLG8]] to ptr addrspace(4))) [[INVOKE_ATTR:#[0-9]+]]
   block_A();
 
   // Emits global block literal [[BLG8]] and block kernel [[INVGK8]]. [[INVGK8]] calls [[INVG8]].
-  // COMMON: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
+  // SPIR: [[DEF_Q:%[0-9]+]] = load target("spirv.Queue"), ptr %default_queue
+  // X86: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
   // COMMON: [[FLAGS:%[0-9]+]] = load i32, ptr %flags
-  // COMMON-LABEL: call spir_func i32 @__enqueue_kernel_basic(
-  // COMMON-SAME: ptr [[DEF_Q]], i32 [[FLAGS]], ptr byval(%struct.ndrange_t) [[NDR]]{{([0-9]+)?}},
+  // COMMON-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_basic(
+  // SPIR-SAME: target("spirv.Queue") [[DEF_Q]], i32 [[FLAGS]], ptr byval(%struct.ndrange_t) [[NDR]]{{([0-9]+)?}},
+  // X86-SAME: ptr [[DEF_Q]], i32 [[FLAGS]], ptr byval(%struct.ndrange_t) [[NDR]]{{([0-9]+)?}},
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr [[INVGK8:[^ ]+_kernel]] to ptr addrspace(4)),
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BLG8]] to ptr addrspace(4)))
   enqueue_kernel(default_queue, flags, ndrange, block_A);
 
   // Uses block kernel [[INVGK8]] and global block literal [[BLG8]].
-  // COMMON: call spir_func i32 @__get_kernel_work_group_size_impl(
+  // COMMON: call {{(spir_func )?}}i32 @__get_kernel_work_group_size_impl(
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr [[INVGK8]] to ptr addrspace(4)),
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BLG8]] to ptr addrspace(4)))
   unsigned size = get_kernel_work_group_size(block_A);
 
   // Uses global block literal [[BLG8]] and invoke function [[INVG8]]. Make sure no redundant block literal and invoke functions are emitted.
-  // COMMON: call spir_func void @__device_side_enqueue_block_invoke_11(ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BLG8]] to ptr addrspace(4)))
+  // COMMON: call {{(spir_func )?}}void @__device_side_enqueue_block_invoke_11(ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BLG8]] to ptr addrspace(4)))
   block_A();
 
   // Make sure that block invoke function is resolved correctly after sequence of assignements.
@@ -332,12 +362,12 @@ kernel void device_side_enqueue(global int *a, global int *b, int i) {
   // COMMON-SAME: to ptr addrspace(4)),
   // COMMON-SAME: ptr %b2,
   bl_t b2 = b1;
-  // COMMON: call spir_func void @block_G_block_invoke(ptr addrspace(4) addrspacecast (ptr addrspace(1)
+  // COMMON: call {{(spir_func )?}}void @block_G_block_invoke(ptr addrspace(4) addrspacecast (ptr addrspace(1)
   // COMMON-SAME: [[BL_GLOBAL]]
   // COOMON-SAME: to ptr addrspace(4)), ptr addrspace(3) null)
   b2(0);
   // Uses global block literal [[BL_GLOBAL]] and block kernel [[INV_G_K]]. [[INV_G_K]] calls [[INV_G]].
-  // COMMON: call spir_func i32 @__get_kernel_preferred_work_group_size_multiple_impl(
+  // COMMON: call {{(spir_func )?}}i32 @__get_kernel_preferred_work_group_size_multiple_impl(
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr [[INV_G_K:[^ ]+_kernel]] to ptr addrspace(4)),
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BL_GLOBAL]] to ptr addrspace(4)))
   size = get_kernel_preferred_work_group_size_multiple(b2);
@@ -347,41 +377,43 @@ kernel void device_side_enqueue(global int *a, global int *b, int i) {
   };
   // Emits block literal on stack and block kernel [[INVLK3]].
   // COMMON: store ptr addrspace(4) addrspacecast (ptr [[INVL3:@__device_side_enqueue_block_invoke[^ ]*]] to ptr addrspace(4)), ptr %block.invoke
-  // COMMON: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
+  // SPIR: [[DEF_Q:%[0-9]+]] = load target("spirv.Queue"), ptr %default_queue
+  // X86: [[DEF_Q:%[0-9]+]] = load ptr, ptr %default_queue
   // COMMON: [[FLAGS:%[0-9]+]] = load i32, ptr %flags
   // COMMON: [[BL_I8:%[0-9]+]] ={{.*}} addrspacecast ptr {{.*}} to ptr addrspace(4)
-  // COMMON-LABEL: call spir_func i32 @__enqueue_kernel_basic(
-  // COMMON-SAME: ptr [[DEF_Q]], i32 [[FLAGS]], ptr byval(%struct.ndrange_t) [[NDR]]{{([0-9]+)?}},
+  // COMMON-LABEL: call {{(spir_func )?}}i32 @__enqueue_kernel_basic(
+  // SPIR-SAME: target("spirv.Queue") [[DEF_Q]], i32 [[FLAGS]], ptr byval(%struct.ndrange_t) [[NDR]]{{([0-9]+)?}},
+  // X86-SAME: ptr [[DEF_Q]], i32 [[FLAGS]], ptr byval(%struct.ndrange_t) [[NDR]]{{([0-9]+)?}},
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr [[INVLK3:[^ ]+_kernel]] to ptr addrspace(4)),
   // COMMON-SAME: ptr addrspace(4) [[BL_I8]])
   enqueue_kernel(default_queue, flags, ndrange, block_C);
 
   // Emits global block literal [[BLG9]] and block kernel [[INVGK9]]. [[INVGK9]] calls [[INV9]].
-  // COMMON: call spir_func i32 @__get_kernel_work_group_size_impl(
+  // COMMON: call {{(spir_func )?}}i32 @__get_kernel_work_group_size_impl(
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr [[INVGK9:[^ ]+_kernel]] to ptr addrspace(4)),
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BLG9]] to ptr addrspace(4)))
   size = get_kernel_work_group_size(block_B);
 
   // Uses global block literal [[BLG8]] and block kernel [[INVGK8]]. Make sure no redundant block literal ind invoke functions are emitted.
-  // COMMON: call spir_func i32 @__get_kernel_preferred_work_group_size_multiple_impl(
+  // COMMON: call {{(spir_func )?}}i32 @__get_kernel_preferred_work_group_size_multiple_impl(
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr [[INVGK8]] to ptr addrspace(4)),
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BLG8]] to ptr addrspace(4)))
   size = get_kernel_preferred_work_group_size_multiple(block_A);
 
   // Uses global block literal [[BL_GLOBAL]] and block kernel [[INV_G_K]]. [[INV_G_K]] calls [[INV_G]].
-  // COMMON: call spir_func i32 @__get_kernel_preferred_work_group_size_multiple_impl(
+  // COMMON: call {{(spir_func )?}}i32 @__get_kernel_preferred_work_group_size_multiple_impl(
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr [[INV_G_K:[^ ]+_kernel]] to ptr addrspace(4)),
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BL_GLOBAL]] to ptr addrspace(4)))
   size = get_kernel_preferred_work_group_size_multiple(block_G);
 
   // Emits global block literal [[BLG10]] and block kernel [[INVGK10]].
-  // COMMON: call spir_func i32 @__get_kernel_max_sub_group_size_for_ndrange_impl(ptr {{[^,]+}},
+  // COMMON: call {{(spir_func )?}}i32 @__get_kernel_max_sub_group_size_for_ndrange_impl(ptr {{[^,]+}},
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr [[INVGK10:[^ ]+_kernel]] to ptr addrspace(4)),
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BLG10]] to ptr addrspace(4)))
   size = get_kernel_max_sub_group_size_for_ndrange(ndrange, ^(){});
 
   // Emits global block literal [[BLG11]] and block kernel [[INVGK11]].
-  // COMMON: call spir_func i32 @__get_kernel_sub_group_count_for_ndrange_impl(ptr {{[^,]+}},
+  // COMMON: call {{(spir_func )?}}i32 @__get_kernel_sub_group_count_for_ndrange_impl(ptr {{[^,]+}},
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr [[INVGK11:[^ ]+_kernel]] to ptr addrspace(4)),
   // COMMON-SAME: ptr addrspace(4) addrspacecast (ptr addrspace(1) [[BLG11]] to ptr addrspace(4)))
   size = get_kernel_sub_group_count_for_ndrange(ndrange, ^(){});
@@ -389,7 +421,7 @@ kernel void device_side_enqueue(global int *a, global int *b, int i) {
 
 // COMMON: define spir_kernel void [[INVLK1]](ptr addrspace(4) %0) #{{[0-9]+}} {
 // COMMON: entry:
-// COMMON:  call spir_func void @__device_side_enqueue_block_invoke(ptr addrspace(4) %0)
+// COMMON:  call {{(spir_func )?}}void @__device_side_enqueue_block_invoke(ptr addrspace(4) %0)
 // COMMON:  ret void
 // COMMON: }
 // COMMON: define spir_kernel void [[INVLK2]](ptr addrspace(4){{.*}})
@@ -400,12 +432,12 @@ kernel void device_side_enqueue(global int *a, global int *b, int i) {
 // COMMON: define spir_kernel void [[INVGK5]](ptr addrspace(4){{.*}}, ptr addrspace(3){{.*}})
 // COMMON: define spir_kernel void [[INVGK6]](ptr addrspace(4) %0, ptr addrspace(3) %1, ptr addrspace(3) %2, ptr addrspace(3) %3) #{{[0-9]+}} {
 // COMMON: entry:
-// COMMON:  call spir_func void @__device_side_enqueue_block_invoke_9(ptr addrspace(4) %0, ptr addrspace(3) %1, ptr addrspace(3) %2, ptr addrspace(3) %3)
+// COMMON:  call {{(spir_func )?}}void @__device_side_enqueue_block_invoke_9(ptr addrspace(4) %0, ptr addrspace(3) %1, ptr addrspace(3) %2, ptr addrspace(3) %3)
 // COMMON:  ret void
 // COMMON: }
 // COMMON: define spir_kernel void [[INVGK7]](ptr addrspace(4){{.*}}, ptr addrspace(3){{.*}})
-// COMMON: define internal spir_func void [[INVG8]](ptr addrspace(4){{.*}}) [[INVG8_INVOKE_FUNC_ATTR:#[0-9]+]]
-// COMMON: define internal spir_func void [[INVG9]](ptr addrspace(4){{.*}}, ptr addrspace(3) %{{.*}})
+// COMMON: define internal {{(spir_func )?}}void [[INVG8]](ptr addrspace(4){{.*}}) [[INVG8_INVOKE_FUNC_ATTR:#[0-9]+]]
+// COMMON: define internal {{(spir_func )?}}void [[INVG9]](ptr addrspace(4){{.*}}, ptr addrspace(3) %{{.*}})
 // COMMON: define spir_kernel void [[INVGK8]](ptr addrspace(4){{.*}})
 // COMMON: define spir_kernel void [[INV_G_K]](ptr addrspace(4){{.*}}, ptr addrspace(3){{.*}})
 // COMMON: define spir_kernel void [[INVLK3]](ptr addrspace(4){{.*}})
@@ -413,5 +445,7 @@ kernel void device_side_enqueue(global int *a, global int *b, int i) {
 // COMMON: define spir_kernel void [[INVGK10]](ptr addrspace(4){{.*}})
 // COMMON: define spir_kernel void [[INVGK11]](ptr addrspace(4){{.*}})
 
-// COMMON: attributes [[INVG8_INVOKE_FUNC_ATTR]] = { convergent noinline nounwind optnone "no-trapping-math"="true" "stack-protector-buffer-size"="8" }
-// COMMON: attributes [[INVOKE_KERNEL_ATTR]] = { convergent nounwind "no-trapping-math"="true" "stack-protector-buffer-size"="8" }
+// SPIR: attributes [[INVG8_INVOKE_FUNC_ATTR]] = { convergent noinline nounwind optnone "no-trapping-math"="true" "stack-protector-buffer-size"="8" }
+// SPIR: attributes [[INVOKE_KERNEL_ATTR]] = { convergent nounwind "no-trapping-math"="true" "stack-protector-buffer-size"="8" }
+// X86: attributes [[INVG8_INVOKE_FUNC_ATTR]] = { convergent noinline nounwind optnone "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-features"="{{[^"]*}}" }
+// X86: attributes [[INVOKE_KERNEL_ATTR]] = { convergent nounwind "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-features"="{{[^"]*}}" }
diff --git a/clang/test/CodeGenOpenCL/intel-subgroups-avc-ext-types.cl b/clang/test/CodeGenOpenCL/intel-subgroups-avc-ext-types.cl
index 3c55276213e62..49e1eaf91fdd2 100644
--- a/clang/test/CodeGenOpenCL/intel-subgroups-avc-ext-types.cl
+++ b/clang/test/CodeGenOpenCL/intel-subgroups-avc-ext-types.cl
@@ -1,45 +1,30 @@
-// RUN: %clang_cc1 -no-opaque-pointers %s -triple spir-unknown-unknown -cl-std=CL1.2 -cl-ext=+cl_intel_device_side_avc_motion_estimation -emit-llvm -o - -O0 | FileCheck %s
+// RUN: %clang_cc1 %s -triple spir-unknown-unknown -cl-std=CL1.2 -cl-ext=+cl_intel_device_side_avc_motion_estimation -emit-llvm -o - -O0 | FileCheck %s
 
-// CHECK: %opencl.intel_sub_group_avc_mce_payload_t = type opaque
-// CHECK: %opencl.intel_sub_group_avc_ime_payload_t = type opaque
-// CHECK: %opencl.intel_sub_group_avc_ref_payload_t = type opaque
-// CHECK: %opencl.intel_sub_group_avc_sic_payload_t = type opaque
+// CHECK: store target("spirv.AvcImePayloadINTEL") zeroinitializer,
+// CHECK: store target("spirv.AvcRefPayloadINTEL") zeroinitializer,
+// CHECK: store target("spirv.AvcSicPayloadINTEL") zeroinitializer,
 
-// CHECK: %opencl.intel_sub_group_avc_mce_result_t = type opaque
-// CHECK: %opencl.intel_sub_group_avc_ime_result_t = type opaque
-// CHECK: %opencl.intel_sub_group_avc_ref_result_t = type opaque
-// CHECK: %opencl.intel_sub_group_avc_sic_result_t = type opaque
+// CHECK: store target("spirv.AvcImeResultINTEL") zeroinitializer,
+// CHECK: store target("spirv.AvcRefResultINTEL") zeroinitializer,
+// CHECK: store target("spirv.AvcSicResultINTEL") zeroinitializer,
 
-// CHECK: %opencl.intel_sub_group_avc_ime_result_single_reference_streamout_t = type opaque
-// CHECK: %opencl.intel_sub_group_avc_ime_result_dual_reference_streamout_t = type opaque
-// CHECK: %opencl.intel_sub_group_avc_ime_single_reference_streamin_t = type opaque
-// CHECK: %opencl.intel_sub_group_avc_ime_dual_reference_streamin_t = type opaque
+// CHECK: store target("spirv.AvcImeResultSingleReferenceStreamoutINTEL") zeroinitializer,
+// CHECK: store target("spirv.AvcImeResultDualReferenceStreamoutINTEL") zeroinitializer,
+// CHECK: store target("spirv.AvcImeSingleReferenceStreaminINTEL") zeroinitializer,
+// CHECK: store target("spirv.AvcImeDualReferenceStreaminINTEL") zeroinitializer,
 
-// CHECK: store %opencl.intel_sub_group_avc_ime_payload_t* null,
-// CHECK: store %opencl.intel_sub_group_avc_ref_payload_t* null,
-// CHECK: store %opencl.intel_sub_group_avc_sic_payload_t* null,
+// CHECK: store target("spirv.AvcImePayloadINTEL") zeroinitializer,
+// CHECK: store target("spirv.AvcRefPayloadINTEL") zeroinitializer,
+// CHECK: store target("spirv.AvcSicPayloadINTEL") zeroinitializer,
 
-// CHECK: store %opencl.intel_sub_group_avc_ime_result_t* null,
-// CHECK: store %opencl.intel_sub_group_avc_ref_result_t* null,
-// CHECK: store %opencl.intel_sub_group_avc_sic_result_t* null,
+// CHECK: store target("spirv.AvcImeResultINTEL") zeroinitializer,
+// CHECK: store target("spirv.AvcRefResultINTEL") zeroinitializer,
+// CHECK: store target("spirv.AvcSicResultINTEL") zeroinitializer,
 
-// CHECK: store %opencl.intel_sub_group_avc_ime_result_single_reference_streamout_t* null,
-// CHECK: store %opencl.intel_sub_group_avc_ime_result_dual_reference_streamout_t* null,
-// CHECK: store %opencl.intel_sub_group_avc_ime_single_reference_streamin_t* null,
-// CHECK: store %opencl.intel_sub_group_avc_ime_dual_reference_streamin_t* null,
-//
-// CHECK: store %opencl.intel_sub_group_avc_ime_payload_t* null,
-// CHECK: store %opencl.intel_sub_group_avc_ref_payload_t* null,
-// CHECK: store %opencl.intel_sub_group_avc_sic_payload_t* null,
-
-// CHECK: store %opencl.intel_sub_group_avc_ime_result_t* null,
-// CHECK: store %opencl.intel_sub_group_avc_ref_result_t* null,
-// CHECK: store %opencl.intel_sub_group_avc_sic_result_t* null,
-
-// CHECK: store %opencl.intel_sub_group_avc_ime_result_single_reference_streamout_t* null,
-// CHECK: store %opencl.intel_sub_group_avc_ime_result_dual_reference_streamout_t* null,
-// CHECK: store %opencl.intel_sub_group_avc_ime_single_reference_streamin_t* null,
-// CHECK: store %opencl.intel_sub_group_avc_ime_dual_reference_streamin_t* null,
+// CHECK: store target("spirv.AvcImeResultSingleReferenceStreamoutINTEL") zeroinitializer,
+// CHECK: store target("spirv.AvcImeResultDualReferenceStreamoutINTEL") zeroinitializer,
+// CHECK: store target("spirv.AvcImeSingleReferenceStreaminINTEL") zeroinitializer,
+// CHECK: store target("spirv.AvcImeDualReferenceStreaminINTEL") zeroinitializer,
 
 #pragma OPENCL EXTENSION cl_intel_device_side_avc_motion_estimation : enable
 
diff --git a/clang/test/CodeGenOpenCL/opencl_types.cl b/clang/test/CodeGenOpenCL/opencl_types.cl
index 17e70a9f774fb..b8ccfa26f16d9 100644
--- a/clang/test/CodeGenOpenCL/opencl_types.cl
+++ b/clang/test/CodeGenOpenCL/opencl_types.cl
@@ -10,65 +10,65 @@ constant sampler_t glb_smp = CLK_ADDRESS_CLAMP_TO_EDGE|CLK_NORMALIZED_COORDS_TRU
 // CHECK-COM-NOT: constant i32
 
 void fnc1(image1d_t img) {}
-// CHECK-SPIR: @fnc1(ptr addrspace(1)
+// CHECK-SPIR: @fnc1(target("spirv.Image", void, 0, 0, 0, 0, 0, 0, 0)
 // CHECK-AMDGCN: @fnc1(ptr addrspace(4)
 
 void fnc1arr(image1d_array_t img) {}
-// CHECK-SPIR: @fnc1arr(ptr addrspace(1)
+// CHECK-SPIR: @fnc1arr(target("spirv.Image", void, 0, 0, 1, 0, 0, 0, 0)
 // CHECK-AMDGCN: @fnc1arr(ptr addrspace(4)
 
 void fnc1buff(image1d_buffer_t img) {}
-// CHECK-SPIR: @fnc1buff(ptr addrspace(1)
+// CHECK-SPIR: @fnc1buff(target("spirv.Image", void, 5, 0, 0, 0, 0, 0, 0)
 // CHECK-AMDGCN: @fnc1buff(ptr addrspace(4)
 
 void fnc2(image2d_t img) {}
-// CHECK-SPIR: @fnc2(ptr addrspace(1)
+// CHECK-SPIR: @fnc2(target("spirv.Image", void, 1, 0, 0, 0, 0, 0, 0)
 // CHECK-AMDGCN: @fnc2(ptr addrspace(4)
 
 void fnc2arr(image2d_array_t img) {}
-// CHECK-SPIR: @fnc2arr(ptr addrspace(1)
+// CHECK-SPIR: @fnc2arr(target("spirv.Image", void, 1, 0, 1, 0, 0, 0, 0)
 // CHECK-AMDGCN: @fnc2arr(ptr addrspace(4)
 
 void fnc3(image3d_t img) {}
-// CHECK-SPIR: @fnc3(ptr addrspace(1)
+// CHECK-SPIR: @fnc3(target("spirv.Image", void, 2, 0, 0, 0, 0, 0, 0)
 // CHECK-AMDGCN: @fnc3(ptr addrspace(4)
 
 void fnc4smp(sampler_t s) {}
-// CHECK-SPIR-LABEL: define {{.*}}void @fnc4smp(ptr addrspace(2)
+// CHECK-SPIR-LABEL: define {{.*}}void @fnc4smp(target("spirv.Sampler")
 // CHECK-AMDGCN-LABEL: define {{.*}}void @fnc4smp(ptr addrspace(4)
 
 kernel void foo(image1d_t img) {
   sampler_t smp = CLK_ADDRESS_CLAMP_TO_EDGE|CLK_NORMALIZED_COORDS_TRUE|CLK_FILTER_LINEAR;
-  // CHECK-SPIR: alloca ptr addrspace(2)
+  // CHECK-SPIR: alloca target("spirv.Sampler")
   // CHECK-AMDGCN: alloca ptr addrspace(4)
   event_t evt;
-  // CHECK-SPIR: alloca ptr
+  // CHECK-SPIR: alloca target("spirv.Event")
   // CHECK-AMDGCN: alloca ptr addrspace(5)
   clk_event_t clk_evt;
-  // CHECK-SPIR: alloca ptr
+  // CHECK-SPIR: alloca target("spirv.DeviceEvent")
   // CHECK-AMDGCN: alloca ptr addrspace(1)
   queue_t queue;
-  // CHECK-SPIR: alloca ptr
+  // CHECK-SPIR: alloca target("spirv.Queue")
   // CHECK-AMDGCN: alloca ptr addrspace(1)
   reserve_id_t rid;
-  // CHECK-SPIR: alloca ptr
+  // CHECK-SPIR: alloca target("spirv.ReserveId")
   // CHECK-AMDGCN: alloca ptr addrspace(1)
-  // CHECK-SPIR: store ptr addrspace(2)
+  // CHECK-SPIR: store target("spirv.Sampler")
   // CHECK-AMDGCN: store ptr addrspace(4)
   fnc4smp(smp);
-  // CHECK-SPIR: call {{.*}}void @fnc4smp(ptr addrspace(2)
+  // CHECK-SPIR: call {{.*}}void @fnc4smp(target("spirv.Sampler")
   // CHECK-AMDGCN: call {{.*}}void @fnc4smp(ptr addrspace(4)
   fnc4smp(glb_smp);
-  // CHECK-SPIR: call {{.*}}void @fnc4smp(ptr addrspace(2)
+  // CHECK-SPIR: call {{.*}}void @fnc4smp(target("spirv.Sampler")
   // CHECK-AMDGCN: call {{.*}}void @fnc4smp(ptr addrspace(4)
 }
 
 kernel void foo_ro_pipe(read_only pipe int p) {}
-// CHECK-SPIR: @foo_ro_pipe(ptr addrspace(1) %p)
+// CHECK-SPIR: @foo_ro_pipe(target("spirv.Pipe", 0) %p)
 // CHECK_AMDGCN: @foo_ro_pipe(ptr addrspace(1) %p)
 
 kernel void foo_wo_pipe(write_only pipe int p) {}
-// CHECK-SPIR: @foo_wo_pipe(ptr addrspace(1) %p)
+// CHECK-SPIR: @foo_wo_pipe(target("spirv.Pipe", 1) %p)
 // CHECK_AMDGCN: @foo_wo_pipe(ptr addrspace(1) %p)
 
 void __attribute__((overloadable)) bad1(image1d_t b, image2d_t c, image2d_t d) {}
diff --git a/clang/test/CodeGenOpenCL/sampler.cl b/clang/test/CodeGenOpenCL/sampler.cl
index 6c367a2327e37..3649afba50fe2 100644
--- a/clang/test/CodeGenOpenCL/sampler.cl
+++ b/clang/test/CodeGenOpenCL/sampler.cl
@@ -1,6 +1,9 @@
-// RUN: %clang_cc1 -no-opaque-pointers %s -emit-llvm -triple spir-unknown-unknown -o - -O0 | FileCheck %s
-// RUN: %clang_cc1 -no-opaque-pointers %s -cl-std=CL2.0 -emit-llvm -triple spir-unknown-unknown -o - -O0 | FileCheck %s
-// RUN: %clang_cc1 -no-opaque-pointers %s -cl-std=clc++ -emit-llvm -triple spir-unknown-unknown -o - -O0 | FileCheck %s
+// RUN: %clang_cc1 %s -emit-llvm -triple spir-unknown-unknown -o - -O0 | FileCheck --check-prefixes=CHECK-COMMON,CHECK-SPIR %s
+// RUN: %clang_cc1 %s -cl-std=CL2.0 -emit-llvm -triple spir-unknown-unknown -o - -O0 | FileCheck --check-prefixes=CHECK-COMMON,CHECK-SPIR %s
+// RUN: %clang_cc1 %s -cl-std=clc++ -emit-llvm -triple spir-unknown-unknown -o - -O0 | FileCheck --check-prefixes=CHECK-COMMON,CHECK-SPIR %s
+// RUN: %clang_cc1 %s -emit-llvm -triple x86_64-unknown-linux-gnu -o - -O0 | FileCheck --check-prefixes=CHECK-COMMON,CHECK-X86 %s
+// RUN: %clang_cc1 %s -cl-std=CL2.0 -emit-llvm -triple x86_64-unknown-linux-gnu -o - -O0 | FileCheck --check-prefixes=CHECK-COMMON,CHECK-X86 %s
+// RUN: %clang_cc1 %s -cl-std=clc++ -emit-llvm -triple x86_64-unknown-linux-gnu -o - -O0 | FileCheck --check-prefixes=CHECK-COMMON,CHECK-X86 %s
 //
 // This test covers 5 cases of sampler initialzation:
 //   1. function argument passing
@@ -17,74 +20,98 @@
 #define CLK_FILTER_NEAREST              0x10
 #define CLK_FILTER_LINEAR               0x20
 
-// CHECK: %opencl.sampler_t = type opaque
-
 // Case 2a
 constant sampler_t glb_smp = CLK_ADDRESS_CLAMP_TO_EDGE | CLK_NORMALIZED_COORDS_TRUE | CLK_FILTER_LINEAR;
-// CHECK-NOT: glb_smp
+// CHECK-COMMON-NOT: glb_smp
 
 // Case 2c
 const sampler_t glb_smp_const = CLK_ADDRESS_CLAMP_TO_EDGE | CLK_NORMALIZED_COORDS_TRUE | CLK_FILTER_LINEAR;
-// CHECK-NOT: glb_smp_const
+// CHECK-COMMON-NOT: glb_smp_const
 
 int get_sampler_initializer(void);
 
 void fnc4smp(sampler_t s) {}
-// CHECK: define{{.*}} spir_func void [[FUNCNAME:@.*fnc4smp.*]](%opencl.sampler_t addrspace(2)* %
+// CHECK-SPIR: define{{.*}} spir_func void [[FUNCNAME:@.*fnc4smp.*]](target("spirv.Sampler") %
+// CHECK-X86: define{{.*}} void [[FUNCNAME:@.*fnc4smp.*]](ptr %
 
 kernel void foo(sampler_t smp_par) {
-  // CHECK-LABEL: define{{.*}} spir_kernel void @foo(%opencl.sampler_t addrspace(2)* %smp_par)
-  // CHECK: [[smp_par_ptr:%[A-Za-z0-9_\.]+]] = alloca %opencl.sampler_t addrspace(2)*
+  // CHECK-SPIR-LABEL: define{{.*}} spir_kernel void @foo(target("spirv.Sampler") %smp_par)
+  // CHECK-SPIR: [[smp_par_ptr:%[A-Za-z0-9_\.]+]] = alloca target("spirv.Sampler")
+  // CHECK-X86-LABEL: define{{.*}} spir_kernel void @foo(ptr %smp_par)
+  // CHECK-X86: [[smp_par_ptr:%[A-Za-z0-9_\.]+]] = alloca ptr
 
   // Case 2b
   sampler_t smp = CLK_ADDRESS_CLAMP_TO_EDGE | CLK_NORMALIZED_COORDS_TRUE | CLK_FILTER_NEAREST;
-  // CHECK: [[smp_ptr:%[A-Za-z0-9_\.]+]] = alloca %opencl.sampler_t addrspace(2)*
-  // CHECK: [[SAMP:%[0-9]+]] = call spir_func %opencl.sampler_t addrspace(2)* @__translate_sampler_initializer(i32 19)
-  // CHECK: store %opencl.sampler_t addrspace(2)* [[SAMP]], %opencl.sampler_t addrspace(2)** [[smp_ptr]]
+  // CHECK-SPIR: [[smp_ptr:%[A-Za-z0-9_\.]+]] = alloca target("spirv.Sampler")
+  // CHECK-SPIR: [[SAMP:%[0-9]+]] = call spir_func target("spirv.Sampler") @__translate_sampler_initializer(i32 19)
+  // CHECK-SPIR: store target("spirv.Sampler") [[SAMP]], ptr [[smp_ptr]]
+  // CHECK-X86: [[smp_ptr:%[A-Za-z0-9_\.]+]] = alloca ptr
+  // CHECK-X86: [[SAMP:%[0-9]+]] = call ptr @__translate_sampler_initializer(i32 19)
+  // CHECK-X86: store ptr [[SAMP]], ptr [[smp_ptr]]
 
   // Case 1b
   fnc4smp(smp);
-  // CHECK-NOT: call %opencl.sampler_t addrspace(2)* @__translate_sampler_initializer(i32 19)
-  // CHECK: [[SAMP:%[0-9]+]] = load %opencl.sampler_t addrspace(2)*, %opencl.sampler_t addrspace(2)** [[smp_ptr]]
-  // CHECK: call spir_func void [[FUNCNAME]](%opencl.sampler_t addrspace(2)* [[SAMP]])
+  // CHECK-SPIR-NOT: call target("spirv.Sampler") @__translate_sampler_initializer(i32 19)
+  // CHECK-SPIR: [[SAMP:%[0-9]+]] = load target("spirv.Sampler"), ptr [[smp_ptr]]
+  // CHECK-SPIR: call spir_func void [[FUNCNAME]](target("spirv.Sampler") [[SAMP]])
+  // CHECK-X86-NOT: call ptr @__translate_sampler_initializer(i32 19)
+  // CHECK-X86: [[SAMP:%[0-9]+]] = load ptr, ptr [[smp_ptr]]
+  // CHECK-X86: call void [[FUNCNAME]](ptr [[SAMP]])
 
   // Case 1b
   fnc4smp(smp);
-  // CHECK-NOT: call %opencl.sampler_t addrspace(2)* @__translate_sampler_initializer(i32 19)
-  // CHECK: [[SAMP:%[0-9]+]] = load %opencl.sampler_t addrspace(2)*, %opencl.sampler_t addrspace(2)** [[smp_ptr]]
-  // CHECK: call spir_func void [[FUNCNAME]](%opencl.sampler_t addrspace(2)* [[SAMP]])
+  // CHECK-SPIR-NOT: call target("spirv.Sampler") @__translate_sampler_initializer(i32 19)
+  // CHECK-SPIR: [[SAMP:%[0-9]+]] = load target("spirv.Sampler"), ptr [[smp_ptr]]
+  // CHECK-SPIR: call spir_func void [[FUNCNAME]](target("spirv.Sampler") [[SAMP]])
+  // CHECK-X86-NOT: call ptr @__translate_sampler_initializer(i32 19)
+  // CHECK-X86: [[SAMP:%[0-9]+]] = load ptr, ptr [[smp_ptr]]
+  // CHECK-X86: call void [[FUNCNAME]](ptr [[SAMP]])
 
   // Case 1a/2a
   fnc4smp(glb_smp);
-  // CHECK: [[SAMP:%[0-9]+]] = call spir_func %opencl.sampler_t addrspace(2)* @__translate_sampler_initializer(i32 35)
-  // CHECK: call spir_func void [[FUNCNAME]](%opencl.sampler_t addrspace(2)* [[SAMP]])
+  // CHECK-SPIR: [[SAMP:%[0-9]+]] = call spir_func target("spirv.Sampler") @__translate_sampler_initializer(i32 35)
+  // CHECK-SPIR: call spir_func void [[FUNCNAME]](target("spirv.Sampler") [[SAMP]])
+  // CHECK-X86: [[SAMP:%[0-9]+]] = call ptr @__translate_sampler_initializer(i32 35)
+  // CHECK-X86: call void [[FUNCNAME]](ptr [[SAMP]])
 
   // Case 1a/2c
   fnc4smp(glb_smp_const);
-  // CHECK: [[SAMP:%[0-9]+]] = call spir_func %opencl.sampler_t addrspace(2)* @__translate_sampler_initializer(i32 35)
-  // CHECK: call spir_func void [[FUNCNAME]](%opencl.sampler_t addrspace(2)* [[SAMP]])
+  // CHECK-SPIR: [[SAMP:%[0-9]+]] = call spir_func target("spirv.Sampler") @__translate_sampler_initializer(i32 35)
+  // CHECK-SPIR: call spir_func void [[FUNCNAME]](target("spirv.Sampler") [[SAMP]])
+  // CHECK-X86: [[SAMP:%[0-9]+]] = call ptr @__translate_sampler_initializer(i32 35)
+  // CHECK-X86: call void [[FUNCNAME]](ptr [[SAMP]])
 
   // Case 1c
   fnc4smp(smp_par);
-  // CHECK: [[SAMP:%[0-9]+]] = load %opencl.sampler_t addrspace(2)*, %opencl.sampler_t addrspace(2)** [[smp_par_ptr]]
-  // CHECK: call spir_func void [[FUNCNAME]](%opencl.sampler_t addrspace(2)* [[SAMP]])
+  // CHECK-SPIR: [[SAMP:%[0-9]+]] = load target("spirv.Sampler"), ptr [[smp_par_ptr]]
+  // CHECK-SPIR: call spir_func void [[FUNCNAME]](target("spirv.Sampler") [[SAMP]])
+  // CHECK-X86: [[SAMP:%[0-9]+]] = load ptr, ptr [[smp_par_ptr]]
+  // CHECK-X86: call void [[FUNCNAME]](ptr [[SAMP]])
 
   fnc4smp(5);
-  // CHECK: [[SAMP:%[0-9]+]] = call spir_func %opencl.sampler_t addrspace(2)* @__translate_sampler_initializer(i32 5)
-  // CHECK: call spir_func void [[FUNCNAME]](%opencl.sampler_t addrspace(2)* [[SAMP]])
+  // CHECK-SPIR: [[SAMP:%[0-9]+]] = call spir_func target("spirv.Sampler") @__translate_sampler_initializer(i32 5)
+  // CHECK-SPIR: call spir_func void [[FUNCNAME]](target("spirv.Sampler") [[SAMP]])
+  // CHECK-X86: [[SAMP:%[0-9]+]] = call ptr @__translate_sampler_initializer(i32 5)
+  // CHECK-X86: call void [[FUNCNAME]](ptr [[SAMP]])
 
   const sampler_t const_smp = CLK_ADDRESS_CLAMP_TO_EDGE | CLK_NORMALIZED_COORDS_TRUE | CLK_FILTER_LINEAR;
   fnc4smp(const_smp);
-   // CHECK: [[CONST_SAMP:%[0-9]+]] = call spir_func %opencl.sampler_t addrspace(2)* @__translate_sampler_initializer(i32 35)
-  // CHECK: store %opencl.sampler_t addrspace(2)* [[CONST_SAMP]], %opencl.sampler_t addrspace(2)** [[CONST_SMP_PTR:%[a-zA-Z0-9]+]]
+  // CHECK-SPIR: [[CONST_SAMP:%[0-9]+]] = call spir_func target("spirv.Sampler") @__translate_sampler_initializer(i32 35)
+  // CHECK-SPIR: store target("spirv.Sampler") [[CONST_SAMP]], ptr [[CONST_SMP_PTR:%[a-zA-Z0-9]+]]
+  // CHECK-X86: [[CONST_SAMP:%[0-9]+]] = call ptr @__translate_sampler_initializer(i32 35)
+  // CHECK-X86: store ptr [[CONST_SAMP]], ptr [[CONST_SMP_PTR:%[a-zA-Z0-9]+]]
   fnc4smp(const_smp);
-  // CHECK: [[SAMP:%[0-9]+]] = load %opencl.sampler_t addrspace(2)*, %opencl.sampler_t addrspace(2)** [[CONST_SMP_PTR]]
-  // CHECK: call spir_func void [[FUNCNAME]](%opencl.sampler_t addrspace(2)* [[SAMP]])
+  // CHECK-SPIR: [[SAMP:%[0-9]+]] = load target("spirv.Sampler"), ptr [[CONST_SMP_PTR]]
+  // CHECK-SPIR: call spir_func void [[FUNCNAME]](target("spirv.Sampler") [[SAMP]])
+  // CHECK-X86: [[SAMP:%[0-9]+]] = load ptr, ptr [[CONST_SMP_PTR]]
+  // CHECK-X86: call void [[FUNCNAME]](ptr [[SAMP]])
 
   constant sampler_t constant_smp = CLK_ADDRESS_CLAMP_TO_EDGE | CLK_NORMALIZED_COORDS_TRUE | CLK_FILTER_LINEAR;
   fnc4smp(constant_smp);
-  // CHECK: [[SAMP:%[0-9]+]] = call spir_func %opencl.sampler_t addrspace(2)* @__translate_sampler_initializer(i32 35)
-  // CHECK: call spir_func void [[FUNCNAME]](%opencl.sampler_t addrspace(2)* [[SAMP]])
+  // CHECK-SPIR: [[SAMP:%[0-9]+]] = call spir_func target("spirv.Sampler") @__translate_sampler_initializer(i32 35)
+  // CHECK-SPIR: call spir_func void [[FUNCNAME]](target("spirv.Sampler") [[SAMP]])
+  // CHECK-X86: [[SAMP:%[0-9]+]] = call ptr @__translate_sampler_initializer(i32 35)
+  // CHECK-X86: call void [[FUNCNAME]](ptr [[SAMP]])
 
   // TODO: enable sampler initialization with non-constant integer.
   //const sampler_t const_smp_func_init = get_sampler_initializer();
diff --git a/clang/test/Driver/aix-maix.c b/clang/test/Driver/aix-maix.c
new file mode 100644
index 0000000000000..2bdcea22dfbdb
--- /dev/null
+++ b/clang/test/Driver/aix-maix.c
@@ -0,0 +1,17 @@
+// Check GCC AIX bitmode compat options.
+
+// RUN: %clang -target powerpc-ibm-aix -maix64 -### -c %s 2>&1 | \
+// RUN:   FileCheck --check-prefix=CHECK64 %s
+
+// RUN: %clang -target powerpc64-ibm-aix -maix32 -### -c %s 2>&1 | \
+// RUN:   FileCheck --check-prefix=CHECK32 %s
+
+// RUN: %clang -target powerpc-unknown-linux -maix64 -### -c %s 2>&1 | \
+// RUN:   FileCheck --check-prefix=ERROR %s
+
+// RUN: %clang -target powerpc64-unknown-linux -maix32 -### -c %s 2>&1 | \
+// RUN:   FileCheck --check-prefix=ERROR %s
+
+// CHECK32: Target: powerpc-ibm-aix
+// CHECK64: Target: powerpc64-ibm-aix
+// ERROR: error: unsupported option '-maix
diff --git a/clang/test/Driver/amdgpu-toolchain-opencl.cl b/clang/test/Driver/amdgpu-toolchain-opencl.cl
index 78c477b5bce9d..c4038e00a0e5d 100644
--- a/clang/test/Driver/amdgpu-toolchain-opencl.cl
+++ b/clang/test/Driver/amdgpu-toolchain-opencl.cl
@@ -25,3 +25,6 @@
 
 // CHK-INC: "-cc1" {{.*}}"-finclude-default-header" "-fdeclare-opencl-builtins" {{.*}}"-x" "cl"
 // CHK-INC-NOT: "-cc1" {{.*}}"-finclude-default-header" "-fdeclare-opencl-builtins" {{.*}}"-x" "cpp-output"
+
+// RUN: %clang -### --target=amdgcn-amd-amdhsa-opencl -x cl -emit-llvm -mcpu=fiji %s 2>&1 | FileCheck -check-prefix=CHK-LINK %s
+// CHK-LINK: ld.lld{{.*}} "--no-undefined" "-shared"
diff --git a/clang/test/Driver/amdgpu-toolchain.c b/clang/test/Driver/amdgpu-toolchain.c
index a00c75b83f3e0..3477d50527c66 100644
--- a/clang/test/Driver/amdgpu-toolchain.c
+++ b/clang/test/Driver/amdgpu-toolchain.c
@@ -6,7 +6,7 @@
 // RUN: %clang -### -g --target=amdgcn-mesa-mesa3d -mcpu=kaveri %s 2>&1 | FileCheck -check-prefix=DWARF_VER %s
 
 // AS_LINK: "-cc1as"
-// AS_LINK: ld.lld{{.*}} "-shared"
+// AS_LINK: ld.lld{{.*}} "--no-undefined" "-shared"
 
 // DWARF_VER: "-dwarf-version=5"
 
diff --git a/clang/test/Driver/arm-abi.c b/clang/test/Driver/arm-abi.c
index ba5c4ba148ce8..548704965b247 100644
--- a/clang/test/Driver/arm-abi.c
+++ b/clang/test/Driver/arm-abi.c
@@ -28,7 +28,9 @@
 // RUN: %clang -target arm--netbsd-eabihf %s -### -o %t.o 2>&1 \
 // RUN:   | FileCheck -check-prefix=CHECK-AAPCS %s
 
-// OpenBSD defaults to aapcs-linux
+// FreeBSD / OpenBSD default to aapcs-linux
+// RUN: %clang -target arm--freebsd- %s -### -o %t.o 2>&1 \
+// RUN:   | FileCheck -check-prefix=CHECK-AAPCS-LINUX %s
 // RUN: %clang -target arm--openbsd- %s -### -o %t.o 2>&1 \
 // RUN:   | FileCheck -check-prefix=CHECK-AAPCS-LINUX %s
 
diff --git a/clang/test/Driver/baremetal.cpp b/clang/test/Driver/baremetal.cpp
index 24890e87c313c..7f2334493c529 100644
--- a/clang/test/Driver/baremetal.cpp
+++ b/clang/test/Driver/baremetal.cpp
@@ -135,7 +135,7 @@
 // CHECK-RV64-SAME: "-Lsome{{[/\\]+}}directory{{[/\\]+}}user{{[/\\]+}}asked{{[/\\]+}}for"
 // CHECK-RV64-SAME: "-L[[SYSROOT:[^"]+]]{{[/\\]+}}lib"
 // CHECK-RV64-SAME: "-L[[RESOURCE_DIR:[^"]+]]{{[/\\]+}}lib{{[/\\]+}}baremetal"
-// CHECK-RV64-SAME: "-lc" "-lm" "-lclang_rt.builtins-riscv64" "-o" "{{.*}}.tmp.out"
+// CHECK-RV64-SAME: "-lc" "-lm" "-lclang_rt.builtins-riscv64" "-X" "-o" "{{.*}}.tmp.out"
 
 // RUN: %clangxx %s -### --target=riscv64-unknown-elf 2>&1 \
 // RUN:     --sysroot=%S/Inputs/basic_riscv64_tree/riscv64-unknown-elf \
@@ -145,7 +145,7 @@
 // CHECK-RV64-DEFAULTCXX-SAME: "-L{{[^"]*}}{{[/\\]+}}Inputs{{[/\\]+}}basic_riscv64_tree{{[/\\]+}}riscv64-unknown-elf{{[/\\]+}}lib"
 // CHECK-RV64-DEFAULTCXX-SAME: "-L[[RESOURCE_DIR]]{{.*}}{{[/\\]+}}lib{{[/\\]+}}baremetal"
 // CHECK-RV64-DEFAULTCXX-SAME: "-lc++" "-lc++abi" "-lunwind"
-// CHECK-RV64-DEFAULTCXX-SAME: "-lc" "-lm" "-lclang_rt.builtins-riscv64" "-o" "a.out"
+// CHECK-RV64-DEFAULTCXX-SAME: "-lc" "-lm" "-lclang_rt.builtins-riscv64" "-X" "-o" "a.out"
 
 // RUN: %clangxx %s -### --target=riscv64-unknown-elf 2>&1 \
 // RUN:     --sysroot=%S/Inputs/basic_riscv64_tree/riscv64-unknown-elf \
@@ -158,7 +158,7 @@
 // CHECK-RV64-LIBCXX-SAME: "-L{{[^"]*}}{{[/\\]+}}Inputs{{[/\\]+}}basic_riscv64_tree{{[/\\]+}}riscv64-unknown-elf{{[/\\]+}}lib"
 // CHECK-RV64-LIBCXX-SAME: "-L[[RESOURCE_DIR]]{{[/\\]+}}lib{{[/\\]+}}baremetal"
 // CHECK-RV64-LIBCXX-SAME: "-lc++" "-lc++abi" "-lunwind"
-// CHECK-RV64-LIBCXX-SAME: "-lc" "-lm" "-lclang_rt.builtins-riscv64" "-o" "a.out"
+// CHECK-RV64-LIBCXX-SAME: "-lc" "-lm" "-lclang_rt.builtins-riscv64" "-X" "-o" "a.out"
 
 // RUN: %clangxx %s -### 2>&1 --target=riscv64-unknown-elf \
 // RUN:     --sysroot=%S/Inputs/basic_riscv64_tree/riscv64-unknown-elf \
@@ -171,7 +171,7 @@
 // CHECK-RV64-LIBSTDCXX-SAME: "-L{{[^"]*}}{{[/\\]+}}Inputs{{[/\\]+}}basic_riscv64_tree{{[/\\]+}}riscv64-unknown-elf{{[/\\]+}}lib"
 // CHECK-RV64-LIBSTDCXX-SAME: "-L[[RESOURCE_DIR]]{{[/\\]+}}lib{{[/\\]+}}baremetal"
 // CHECK-RV64-LIBSTDCXX-SAME: "-lstdc++" "-lsupc++" "-lunwind"
-// CHECK-RV64-LIBSTDCXX-SAME: "-lc" "-lm" "-lclang_rt.builtins-riscv64" "-o" "a.out"
+// CHECK-RV64-LIBSTDCXX-SAME: "-lc" "-lm" "-lclang_rt.builtins-riscv64" "-X" "-o" "a.out"
 
 // RUN: %clang %s -### 2>&1 --target=riscv32-unknown-elf \
 // RUN:     -L some/directory/user/asked/for \
@@ -187,7 +187,7 @@
 // CHECK-RV32-SAME: "-Lsome{{[/\\]+}}directory{{[/\\]+}}user{{[/\\]+}}asked{{[/\\]+}}for"
 // CHECK-RV32-SAME: "-L[[SYSROOT:[^"]+]]{{[/\\]+}}lib"
 // CHECK-RV32-SAME: "-L[[RESOURCE_DIR:[^"]+]]{{[/\\]+}}lib{{[/\\]+}}baremetal"
-// CHECK-RV32-SAME: "-lc" "-lm" "-lclang_rt.builtins-riscv32" "-o" "a.out"
+// CHECK-RV32-SAME: "-lc" "-lm" "-lclang_rt.builtins-riscv32" "-X" "-o" "a.out"
 
 // RUN: %clangxx %s -### 2>&1 --target=riscv32-unknown-elf \
 // RUN:     --sysroot=%S/Inputs/basic_riscv32_tree/riscv32-unknown-elf \
@@ -197,7 +197,7 @@
 // CHECK-RV32-DEFAULTCXX-SAME: "-L{{[^"]*}}{{[/\\]+}}Inputs{{[/\\]+}}basic_riscv32_tree{{[/\\]+}}riscv32-unknown-elf{{[/\\]+}}lib"
 // CHECK-RV32-DEFAULTCXX-SAME: "-L[[RESOURCE_DIR]]{{[/\\]+}}lib{{[/\\]+}}baremetal"
 // CHECK-RV32-DEFAULTCXX-SAME: "-lc++" "-lc++abi" "-lunwind"
-// CHECK-RV32-DEFAULTCXX-SAME: "-lc" "-lm" "-lclang_rt.builtins-riscv32" "-o" "a.out"
+// CHECK-RV32-DEFAULTCXX-SAME: "-lc" "-lm" "-lclang_rt.builtins-riscv32" "-X" "-o" "a.out"
 
 // RUN: %clangxx %s -### 2>&1 --target=riscv32-unknown-elf \
 // RUN:     --sysroot=%S/Inputs/basic_riscv32_tree/riscv32-unknown-elf \
@@ -210,7 +210,7 @@
 // CHECK-RV32-LIBCXX-SAME: "-L{{[^"]*}}{{[/\\]+}}Inputs{{[/\\]+}}basic_riscv32_tree{{[/\\]+}}riscv32-unknown-elf{{[/\\]+}}lib"
 // CHECK-RV32-LIBCXX-SAME: "-L[[RESOURCE_DIR]]{{[/\\]+}}lib{{[/\\]+}}baremetal"
 // CHECK-RV32-LIBCXX-SAME: "-lc++" "-lc++abi" "-lunwind"
-// CHECK-RV32-LIBCXX-SAME: "-lc" "-lm" "-lclang_rt.builtins-riscv32" "-o" "a.out"
+// CHECK-RV32-LIBCXX-SAME: "-lc" "-lm" "-lclang_rt.builtins-riscv32" "-X" "-o" "a.out"
 
 // RUN: %clangxx %s -### 2>&1 --target=riscv32-unknown-elf \
 // RUN:     --sysroot=%S/Inputs/basic_riscv32_tree/riscv32-unknown-elf \
@@ -223,7 +223,7 @@
 // CHECK-RV32-LIBSTDCXX-SAME: "-L{{[^"]*}}{{[/\\]+}}Inputs{{[/\\]+}}basic_riscv32_tree{{[/\\]+}}riscv32-unknown-elf{{[/\\]+}}lib"
 // CHECK-RV32-LIBSTDCXX-SAME: "-L[[RESOURCE_DIR]]{{[/\\]+}}lib{{[/\\]+}}baremetal"
 // CHECK-RV32-LIBSTDCXX-SAME: "-lstdc++" "-lsupc++" "-lunwind"
-// CHECK-RV32-LIBSTDCXX-SAME: "-lc" "-lm" "-lclang_rt.builtins-riscv32" "-o" "a.out"
+// CHECK-RV32-LIBSTDCXX-SAME: "-lc" "-lm" "-lclang_rt.builtins-riscv32" "-X" "-o" "a.out"
 
 // RUN: %clang %s -### 2>&1 --target=riscv64-unknown-elf \
 // RUN:     -nostdlibinc -nobuiltininc \
diff --git a/clang/test/Driver/clang-offload-bundler-asserts-on.c b/clang/test/Driver/clang-offload-bundler-asserts-on.c
index aceda09f7bfa1..db99e31d568b9 100644
--- a/clang/test/Driver/clang-offload-bundler-asserts-on.c
+++ b/clang/test/Driver/clang-offload-bundler-asserts-on.c
@@ -21,12 +21,12 @@
 
 // Tests to check compatibility between Bundle Entry ID formats i.e. between presence/absence of extra hyphen in case of missing environment field
 // RUN: clang-offload-bundler -unbundle -type=a -targets=openmp-amdgcn-amd-amdhsa--gfx906,openmp-amdgcn-amd-amdhsa-gfx908 -input=%t.input-archive.a -output=%t-archive-gfx906-simple.a -output=%t-archive-gfx908-simple.a -debug-only=CodeObjectCompatibility 2>&1 | FileCheck %s -check-prefix=BUNDLECOMPATIBILITY
-// BUNDLECOMPATIBILITY: Compatible: Exact match:        [CodeObject: openmp-amdgcn-amd-amdhsa-gfx906]   :       [Target: openmp-amdgcn-amd-amdhsa--gfx906]
-// BUNDLECOMPATIBILITY: Compatible: Exact match:        [CodeObject: openmp-amdgcn-amd-amdhsa--gfx908]  :       [Target: openmp-amdgcn-amd-amdhsa-gfx908]
+// BUNDLECOMPATIBILITY: Compatible: Exact match:        [CodeObject: openmp-amdgcn-amd-amdhsa--gfx906]  :       [Target: openmp-amdgcn-amd-amdhsa--gfx906]
+// BUNDLECOMPATIBILITY: Compatible: Exact match:        [CodeObject: openmp-amdgcn-amd-amdhsa--gfx908]  :       [Target: openmp-amdgcn-amd-amdhsa--gfx908]
 
 // RUN: clang-offload-bundler -unbundle -type=a -targets=hip-amdgcn-amd-amdhsa--gfx906,hipv4-amdgcn-amd-amdhsa-gfx908 -input=%t.input-archive.a -output=%t-hip-archive-gfx906-simple.a -output=%t-hipv4-archive-gfx908-simple.a -hip-openmp-compatible -debug-only=CodeObjectCompatibility 2>&1 | FileCheck %s -check-prefix=HIPOpenMPCOMPATIBILITY
-// HIPOpenMPCOMPATIBILITY: Compatible: Code Objects are compatible        [CodeObject: openmp-amdgcn-amd-amdhsa-gfx906]   :       [Target: hip-amdgcn-amd-amdhsa--gfx906]
-// HIPOpenMPCOMPATIBILITY: Compatible: Code Objects are compatible        [CodeObject: openmp-amdgcn-amd-amdhsa--gfx908]  :       [Target: hipv4-amdgcn-amd-amdhsa-gfx908]
+// HIPOpenMPCOMPATIBILITY: Compatible: Code Objects are compatible        [CodeObject: openmp-amdgcn-amd-amdhsa--gfx906]  :       [Target: hip-amdgcn-amd-amdhsa--gfx906]
+// HIPOpenMPCOMPATIBILITY: Compatible: Code Objects are compatible        [CodeObject: openmp-amdgcn-amd-amdhsa--gfx908]  :       [Target: hipv4-amdgcn-amd-amdhsa--gfx908]
 
 // Some code so that we can create a binary out of this file.
 int A = 0;
diff --git a/clang/test/Driver/clang-offload-bundler-standardize.c b/clang/test/Driver/clang-offload-bundler-standardize.c
new file mode 100644
index 0000000000000..6a24968c30efd
--- /dev/null
+++ b/clang/test/Driver/clang-offload-bundler-standardize.c
@@ -0,0 +1,37 @@
+// REQUIRES: x86-registered-target
+// REQUIRES: asserts
+// UNSUPPORTED: target={{.*}}-darwin{{.*}}, target={{.*}}-aix{{.*}}
+// REQUIRES: asserts
+
+// Generate the file we can bundle.
+// RUN: %clang -O0 -target %itanium_abi_triple %s -c -o %t.o
+
+//
+// Generate a couple of files to bundle with.
+//
+// RUN: echo 'Content of device file 1' > %t.tgt1
+// RUN: echo 'Content of device file 2' > %t.tgt2
+
+//
+// Check code object compatibility for archive unbundling
+//
+// Create an object bundle with and without env fields
+// RUN: clang-offload-bundler -type=o -targets=host-%itanium_abi_triple,hip-amdgcn-amd-amdhsa-gfx906,hip-amdgcn-amd-amdhsa-gfx908 -input=%t.o -input=%t.tgt1 -input=%t.tgt2 -output=%t.bundle.no.env
+// RUN: clang-offload-bundler -type=o -targets=host-%itanium_abi_triple-,hip-amdgcn-amd-amdhsa--gfx906,hip-amdgcn-amd-amdhsa--gfx908 -input=%t.o -input=%t.tgt1 -input=%t.tgt2 -output=%t.bundle.env
+
+
+// Unbundle bundle.no.env while providing targets with env
+// RUN: clang-offload-bundler -unbundle -type=o -targets=hip-amdgcn-amd-amdhsa--gfx906,hip-amdgcn-amd-amdhsa--gfx908 -input=%t.bundle.no.env -output=%t-hip-amdgcn-amd-amdhsa--gfx906.bc -output=%t-hip-amdgcn-amd-amdhsa--gfx908.bc -debug-only=CodeObjectCompatibility 2>&1 | FileCheck %s -check-prefix=BUNDLE-NO-ENV
+// BUNDLE-NO-ENV: Compatible: Exact match: [CodeObject: hip-amdgcn-amd-amdhsa--gfx906] : [Target: hip-amdgcn-amd-amdhsa--gfx906]
+// BUNDLE-NO-ENV: Compatible: Exact match: [CodeObject: hip-amdgcn-amd-amdhsa--gfx908] : [Target: hip-amdgcn-amd-amdhsa--gfx908]
+
+// Unbundle bundle.env while providing targets with no env
+// RUN: clang-offload-bundler -unbundle -type=o -targets=hip-amdgcn-amd-amdhsa-gfx906,hip-amdgcn-amd-amdhsa-gfx908 -input=%t.bundle.env -output=%t-hip-amdgcn-amd-amdhsa-gfx906.bc -output=%t-hip-amdgcn-amd-amdhsa-gfx908.bc -debug-only=CodeObjectCompatibility 2>&1 | FileCheck %s -check-prefix=BUNDLE-ENV
+// BUNDLE-ENV: Compatible: Exact match: [CodeObject: hip-amdgcn-amd-amdhsa--gfx906] : [Target: hip-amdgcn-amd-amdhsa--gfx906]
+// BUNDLE-ENV: Compatible: Exact match: [CodeObject: hip-amdgcn-amd-amdhsa--gfx908] : [Target: hip-amdgcn-amd-amdhsa--gfx908]
+
+// Some code so that we can create a binary out of this file.
+int A = 0;
+void test_func(void) {
+  ++A;
+}
diff --git a/clang/test/Driver/clang-offload-bundler.c b/clang/test/Driver/clang-offload-bundler.c
index 0d5d26afcb62d..7d0b6b27a60ae 100644
--- a/clang/test/Driver/clang-offload-bundler.c
+++ b/clang/test/Driver/clang-offload-bundler.c
@@ -1,5 +1,4 @@
 // REQUIRES: x86-registered-target
-// REQUIRES: powerpc-registered-target
 // UNSUPPORTED: target={{.*}}-darwin{{.*}}, target={{.*}}-aix{{.*}}
 
 //
@@ -297,11 +296,11 @@
 
 // RUN: clang-offload-bundler -type=o -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -input=%t.o -input=%t.tgt1 -input=%t.tgt2 -output=%t.bundle3.o -### 2>&1 \
 // RUN: | FileCheck %s -DHOST=%itanium_abi_triple -DINOBJ1=%t.o -DINOBJ2=%t.tgt1 -DINOBJ3=%t.tgt2 -DOUTOBJ=%t.bundle3.o --check-prefix CK-OBJ-CMD
-// CK-OBJ-CMD: llvm-objcopy{{(.exe)?}}" "--add-section=__CLANG_OFFLOAD_BUNDLE__host-[[HOST]]={{.*}}" "--set-section-flags=__CLANG_OFFLOAD_BUNDLE__host-[[HOST]]=readonly,exclude" "--add-section=__CLANG_OFFLOAD_BUNDLE__openmp-powerpc64le-ibm-linux-gnu=[[INOBJ2]]" "--set-section-flags=__CLANG_OFFLOAD_BUNDLE__openmp-powerpc64le-ibm-linux-gnu=readonly,exclude" "--add-section=__CLANG_OFFLOAD_BUNDLE__openmp-x86_64-pc-linux-gnu=[[INOBJ3]]" "--set-section-flags=__CLANG_OFFLOAD_BUNDLE__openmp-x86_64-pc-linux-gnu=readonly,exclude" "--" "[[INOBJ1]]" "[[OUTOBJ]]"
+// CK-OBJ-CMD: llvm-objcopy{{(.exe)?}}" "--add-section=__CLANG_OFFLOAD_BUNDLE__host-[[HOST:.+]]={{.*}}" "--set-section-flags=__CLANG_OFFLOAD_BUNDLE__host-[[HOST]]=readonly,exclude" "--add-section=__CLANG_OFFLOAD_BUNDLE__openmp-powerpc64le-ibm-linux-gnu-=[[INOBJ2]]" "--set-section-flags=__CLANG_OFFLOAD_BUNDLE__openmp-powerpc64le-ibm-linux-gnu-=readonly,exclude" "--add-section=__CLANG_OFFLOAD_BUNDLE__openmp-x86_64-pc-linux-gnu-=[[INOBJ3]]" "--set-section-flags=__CLANG_OFFLOAD_BUNDLE__openmp-x86_64-pc-linux-gnu-=readonly,exclude" "--" "[[INOBJ1]]" "[[OUTOBJ]]"
 
 // RUN: clang-offload-bundler -type=o -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -input=%t.o -input=%t.tgt1 -input=%t.tgt2 -output=%t.bundle3.o -### 2>&1 \
 // RUN: | FileCheck %s -DHOST=%itanium_abi_triple -DINOBJ1=%t.o -DINOBJ2=%t.tgt1 -DINOBJ3=%t.tgt2 -DOUTOBJ=%t.bundle3.o --check-prefix CK-OBJ-CMD-INPUTS
-// CK-OBJ-CMD-INPUTS: llvm-objcopy{{(.exe)?}}" "--add-section=__CLANG_OFFLOAD_BUNDLE__host-[[HOST]]={{.*}}" "--set-section-flags=__CLANG_OFFLOAD_BUNDLE__host-[[HOST]]=readonly,exclude" "--add-section=__CLANG_OFFLOAD_BUNDLE__openmp-powerpc64le-ibm-linux-gnu=[[INOBJ2]]" "--set-section-flags=__CLANG_OFFLOAD_BUNDLE__openmp-powerpc64le-ibm-linux-gnu=readonly,exclude" "--add-section=__CLANG_OFFLOAD_BUNDLE__openmp-x86_64-pc-linux-gnu=[[INOBJ3]]" "--set-section-flags=__CLANG_OFFLOAD_BUNDLE__openmp-x86_64-pc-linux-gnu=readonly,exclude" "--" "[[INOBJ1]]" "[[OUTOBJ]]"
+// CK-OBJ-CMD-INPUTS: llvm-objcopy{{(.exe)?}}" "--add-section=__CLANG_OFFLOAD_BUNDLE__host-[[HOST:.+]]={{.*}}" "--set-section-flags=__CLANG_OFFLOAD_BUNDLE__host-[[HOST]]=readonly,exclude" "--add-section=__CLANG_OFFLOAD_BUNDLE__openmp-powerpc64le-ibm-linux-gnu-=[[INOBJ2]]" "--set-section-flags=__CLANG_OFFLOAD_BUNDLE__openmp-powerpc64le-ibm-linux-gnu-=readonly,exclude" "--add-section=__CLANG_OFFLOAD_BUNDLE__openmp-x86_64-pc-linux-gnu-=[[INOBJ3]]" "--set-section-flags=__CLANG_OFFLOAD_BUNDLE__openmp-x86_64-pc-linux-gnu-=readonly,exclude" "--" "[[INOBJ1]]" "[[OUTOBJ]]"
 
 // RUN: clang-offload-bundler -type=o -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -input=%t.o -input=%t.tgt1 -input=%t.tgt2 -output=%t.bundle3.o
 // RUN: clang-offload-bundler -type=o -input=%t.bundle3.o -list | FileCheck -check-prefix=CKLST %s
@@ -482,15 +481,15 @@
 // RUN: llvm-ar t %t-archive-gfx906-simple.a | FileCheck %s -check-prefix=GFX906
 // RUN: clang-offload-bundler -unbundle -type=a -targets=openmp-amdgcn-amd-amdhsa-gfx906:xnack+ -input=%t.input-archive.a -output=%t-archive-gfx906-simple.a
 // RUN: llvm-ar t %t-archive-gfx906-simple.a | FileCheck %s -check-prefix=GFX906
-// GFX906: simple-openmp-amdgcn-amd-amdhsa-gfx906
+// GFX906: simple-openmp-amdgcn-amd-amdhsa--gfx906
 // RUN: llvm-ar t %t-archive-gfx908-simple.a | FileCheck %s -check-prefix=GFX908
 // GFX908-NOT: {{gfx906}}
 // RUN: not clang-offload-bundler -type=o -targets=host-%itanium_abi_triple,openmp-amdgcn-amd-amdhsa-gfx906,openmp-amdgcn-amd-amdhsa-gfx906:sramecc+ -input=%t.o -input=%t.tgt1 -input=%t.tgt2 -output=%t.bad.bundle 2>&1 | FileCheck %s -check-prefix=BADTARGETS
-// BADTARGETS: error: Cannot bundle inputs with conflicting targets: 'openmp-amdgcn-amd-amdhsa-gfx906' and 'openmp-amdgcn-amd-amdhsa-gfx906:sramecc+'
+// BADTARGETS: error: Cannot bundle inputs with conflicting targets: 'openmp-amdgcn-amd-amdhsa--gfx906' and 'openmp-amdgcn-amd-amdhsa--gfx906:sramecc+'
 
 // Check for error if no compatible code object is found in the heterogeneous archive library
 // RUN: not clang-offload-bundler -unbundle -type=a -targets=openmp-amdgcn-amd-amdhsa-gfx803 -input=%t.input-archive.a -output=%t-archive-gfx803-incompatible.a 2>&1 | FileCheck %s -check-prefix=INCOMPATIBLEARCHIVE
-// INCOMPATIBLEARCHIVE: error: no compatible code object found for the target 'openmp-amdgcn-amd-amdhsa-gfx803' in heterogeneous archive library
+// INCOMPATIBLEARCHIVE: error: no compatible code object found for the target 'openmp-amdgcn-amd-amdhsa--gfx803' in heterogeneous archive library
 
 // Check creation of empty archive if allow-missing-bundles is present and no compatible code object is found in the heterogeneous archive library
 // RUN: clang-offload-bundler -unbundle -type=a -targets=openmp-amdgcn-amd-amdhsa-gfx803 -input=%t.input-archive.a -output=%t-archive-gfx803-empty.a -allow-missing-bundles
@@ -500,7 +499,7 @@
 // Check compatibility of OpenMP code objects found in the heterogeneous archive library with HIP code objects of the target
 // RUN: clang-offload-bundler -unbundle -type=a -targets=hip-amdgcn-amd-amdhsa-gfx906,hipv4-amdgcn-amd-amdhsa-gfx908 -input=%t.input-archive.a -output=%t-hip-archive-gfx906-simple.a -output=%t-hipv4-archive-gfx908-simple.a -hip-openmp-compatible
 // RUN: llvm-ar t %t-hip-archive-gfx906-simple.a | FileCheck %s -check-prefix=HIPOPENMPCOMPAT
-// HIPOPENMPCOMPAT: simple-openmp-amdgcn-amd-amdhsa-gfx906
+// HIPOPENMPCOMPAT: simple-openmp-amdgcn-amd-amdhsa--gfx906
 // RUN: llvm-ar t %t-hipv4-archive-gfx908-simple.a | FileCheck %s -check-prefix=HIPv4OPENMPCOMPAT
 // HIPv4OPENMPCOMPAT: simple-openmp-amdgcn-amd-amdhsa--gfx908
 
diff --git a/clang/test/Driver/crash-diagnostics-dir-3.c b/clang/test/Driver/crash-diagnostics-dir-3.c
index 9529e30021045..a91bc48d7e462 100644
--- a/clang/test/Driver/crash-diagnostics-dir-3.c
+++ b/clang/test/Driver/crash-diagnostics-dir-3.c
@@ -1,3 +1,4 @@
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: rm -rf %t
 // RUN: not env CLANG_CRASH_DIAGNOSTICS_DIR=%t %clang -c %s -o - 2>&1 | FileCheck %s
 #pragma clang __debug parser_crash
diff --git a/clang/test/Driver/crash-diagnostics-dir.c b/clang/test/Driver/crash-diagnostics-dir.c
index 44c4af67649a7..16382eff1cde7 100644
--- a/clang/test/Driver/crash-diagnostics-dir.c
+++ b/clang/test/Driver/crash-diagnostics-dir.c
@@ -1,3 +1,4 @@
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: rm -rf %t
 // RUN: not %clang -fcrash-diagnostics-dir=%t -c %s -o - 2>&1 | FileCheck %s
 #pragma clang __debug parser_crash
diff --git a/clang/test/Driver/crash-report-clang-cl.cpp b/clang/test/Driver/crash-report-clang-cl.cpp
index 159792d3eb61a..963c3b6d0ab03 100644
--- a/clang/test/Driver/crash-report-clang-cl.cpp
+++ b/clang/test/Driver/crash-report-clang-cl.cpp
@@ -1,3 +1,4 @@
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: rm -rf %t
 // RUN: mkdir %t
 
diff --git a/clang/test/Driver/crash-report-header.h b/clang/test/Driver/crash-report-header.h
index c1b0ab367683c..04865a0cc300f 100644
--- a/clang/test/Driver/crash-report-header.h
+++ b/clang/test/Driver/crash-report-header.h
@@ -1,3 +1,4 @@
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: rm -rf %t
 // RUN: mkdir %t
 // RUN: env TMPDIR="%t" TEMP="%t" TMP="%t" RC_DEBUG_OPTIONS=1 not %clang -fsyntax-only %s 2>&1 | FileCheck %s
diff --git a/clang/test/Driver/crash-report-spaces.c b/clang/test/Driver/crash-report-spaces.c
index e6da0873673aa..b4d8ac1f57e83 100644
--- a/clang/test/Driver/crash-report-spaces.c
+++ b/clang/test/Driver/crash-report-spaces.c
@@ -1,3 +1,4 @@
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: rm -rf "%t"
 // RUN: mkdir "%t"
 // RUN: cp "%s" "%t/crash report spaces.c"
diff --git a/clang/test/Driver/crash-report-with-asserts.c b/clang/test/Driver/crash-report-with-asserts.c
index 7a614aa8fe025..686c49f339fb7 100644
--- a/clang/test/Driver/crash-report-with-asserts.c
+++ b/clang/test/Driver/crash-report-with-asserts.c
@@ -1,3 +1,4 @@
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: rm -rf %t
 // RUN: mkdir %t
 
diff --git a/clang/test/Driver/crash-report.cpp b/clang/test/Driver/crash-report.cpp
index 455597c9c0fc7..59eee65af57ee 100644
--- a/clang/test/Driver/crash-report.cpp
+++ b/clang/test/Driver/crash-report.cpp
@@ -1,3 +1,4 @@
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: rm -rf %t
 // RUN: mkdir %t
 
diff --git a/clang/test/Driver/emit-reproducer.c b/clang/test/Driver/emit-reproducer.c
index b8d6841077c4d..18e1b4e41b91d 100644
--- a/clang/test/Driver/emit-reproducer.c
+++ b/clang/test/Driver/emit-reproducer.c
@@ -1,3 +1,4 @@
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: rm -rf %t && mkdir %t
 
 // RUN: echo "%s -fcrash-diagnostics-dir=%t -fsyntax-only" | sed -e 's/\\/\\\\/g' > %t.rsp
diff --git a/clang/test/Driver/fpatchable-function-entry.c b/clang/test/Driver/fpatchable-function-entry.c
index da7370a4d87a7..4d0d609584c8d 100644
--- a/clang/test/Driver/fpatchable-function-entry.c
+++ b/clang/test/Driver/fpatchable-function-entry.c
@@ -2,6 +2,8 @@
 // RUN: %clang -target x86_64 %s -fpatchable-function-entry=1 -c -### 2>&1 | FileCheck %s
 // RUN: %clang -target aarch64 %s -fpatchable-function-entry=1 -c -### 2>&1 | FileCheck %s
 // RUN: %clang -target aarch64 %s -fpatchable-function-entry=1,0 -c -### 2>&1 | FileCheck %s
+// RUN: %clang -target loongarch32 %s -fpatchable-function-entry=1,0 -c -### 2>&1 | FileCheck %s
+// RUN: %clang -target loongarch64 %s -fpatchable-function-entry=1,0 -c -### 2>&1 | FileCheck %s
 // RUN: %clang -target riscv32 %s -fpatchable-function-entry=1,0 -c -### 2>&1 | FileCheck %s
 // RUN: %clang -target riscv64 %s -fpatchable-function-entry=1,0 -c -### 2>&1 | FileCheck %s
 // CHECK: "-fpatchable-function-entry=1"
diff --git a/clang/test/Driver/freebsd.c b/clang/test/Driver/freebsd.c
index eedecde5ad5f3..28a888fedf3fe 100644
--- a/clang/test/Driver/freebsd.c
+++ b/clang/test/Driver/freebsd.c
@@ -146,9 +146,10 @@
 
 // RUN: %clang -### %s --target=arm-unknown-freebsd10.0 -no-integrated-as 2>&1 \
 // RUN:   | FileCheck --check-prefix=CHECK-ARM %s
-// CHECK-ARM: "-cc1"{{.*}}" "-exception-model=sjlj"
-// CHECK-ARM: as{{.*}}" "-mfpu=softvfp"{{.*}}"-matpcs"
-// CHECK-ARM-EABI-NOT: as{{.*}}" "-mfpu=vfp"
+// CHECK-ARM-NOT: "-cc1"{{.*}}" "-exception-model=sjlj"
+// CHECK-ARM: as{{.*}}" "-mfpu=softvfp"{{.*}}"-meabi=5"
+// CHECK-ARM-NOT: as{{.*}}" "-mfpu=vfp"
+// CHECK-ARM-NOT: as{{.*}}" "-matpcs"
 
 // RUN: %clang -### %s --target=arm-gnueabi-freebsd10.0 -no-integrated-as 2>&1 \
 // RUN:   | FileCheck --check-prefix=CHECK-ARM-EABI %s
diff --git a/clang/test/Driver/fsanitize.c b/clang/test/Driver/fsanitize.c
index 8ad6dc25b4d17..3b89e132a944f 100644
--- a/clang/test/Driver/fsanitize.c
+++ b/clang/test/Driver/fsanitize.c
@@ -898,6 +898,7 @@
 // RUN: %clang --target=mipsel-unknown-linux-gnu -fsanitize=scudo %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SCUDO
 // RUN: %clang --target=powerpc64-unknown-linux -fsanitize=scudo %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SCUDO
 // RUN: %clang --target=powerpc64le-unknown-linux -fsanitize=scudo %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SCUDO
+// RUN: %clang --target=riscv64-linux-gnu -fsanitize=scudo %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SCUDO
 // CHECK-SCUDO: "-fsanitize=scudo"
 
 // RUN: %clang --target=x86_64-linux-gnu -fsanitize=scudo %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SCUDO-PIE
diff --git a/clang/test/Driver/hip-link-bc-to-bc.hip b/clang/test/Driver/hip-link-bc-to-bc.hip
index 1cfcdad4e4978..6b31456657fff 100644
--- a/clang/test/Driver/hip-link-bc-to-bc.hip
+++ b/clang/test/Driver/hip-link-bc-to-bc.hip
@@ -11,10 +11,10 @@
 // RUN:   2>&1 | FileCheck -check-prefix=BITCODE %s
 
 // BITCODE: "{{.*}}clang-offload-bundler" "-type=bc" "-targets=host-x86_64-unknown-linux-gnu,hip-amdgcn-amd-amdhsa-gfx906" "-input={{.*}}bundle1.bc" "-output=[[B1HOST:.*\.bc]]" "-output=[[B1DEV1:.*\.bc]]" "-unbundle" "-allow-missing-bundles"
-// BITCODE: "{{.*}}clang{{.*}}" "-o" "[[B1DEV2:.*bundle1-gfx906.bc]]" "-x" "ir" "[[B1DEV1]]"
+// BITCODE: "{{.*}}clang{{.*}}" "-o" "[[B1DEV2:.*bundle1-gfx906-.*\.bc]]" "-x" "ir" "[[B1DEV1]]"
 
 // BITCODE: "{{.*}}clang-offload-bundler" "-type=bc" "-targets=host-x86_64-unknown-linux-gnu,hip-amdgcn-amd-amdhsa-gfx906" "-input={{.*}}bundle2.bc" "-output=[[B2HOST:.*\.bc]]" "-output=[[B2DEV1:.*\.bc]]" "-unbundle" "-allow-missing-bundles"
-// BITCODE: "{{.*}}clang{{.*}}" "-o" "[[B2DEV2:.*bundle2-gfx906.bc]]" "-x" "ir" "[[B2DEV1]]"
+// BITCODE: "{{.*}}clang{{.*}}" "-o" "[[B2DEV2:.*bundle2-gfx906-.*\.bc]]" "-x" "ir" "[[B2DEV1]]"
 
 // BITCODE: "{{.*}}llvm-link" "-o" "bundle1-hip-amdgcn-amd-amdhsa-gfx906.bc" "[[B1DEV2]]" "[[B2DEV2]]"
 
diff --git a/clang/test/Driver/hip-runtime-libs-linux.hip b/clang/test/Driver/hip-runtime-libs-linux.hip
index aa71798a80bb4..b0e9f3a460eb0 100644
--- a/clang/test/Driver/hip-runtime-libs-linux.hip
+++ b/clang/test/Driver/hip-runtime-libs-linux.hip
@@ -16,6 +16,10 @@
 // RUN:   --rocm-path=%S/Inputs/rocm %t.o --offload-add-rpath 2>&1 \
 // RUN:   | FileCheck -check-prefixes=ROCM-RPATH %s
 
+// RUN: %clang -### --hip-link --target=x86_64-linux-gnu \
+// RUN:   --rocm-path=%S/Inputs/rocm %t.o -frtlib-add-rpath 2>&1 \
+// RUN:   | FileCheck -check-prefixes=ROCM-RPATH %s
+
 // Test detecting latest /opt/rocm-{release} directory.
 // RUN: rm -rf %t && mkdir -p %t/opt
 // RUN: cp -r %S/Inputs/rocm %t/opt/rocm-3.9.0-1234
diff --git a/clang/test/Driver/hip-temps-linux.hip b/clang/test/Driver/hip-temps-linux.hip
new file mode 100644
index 0000000000000..83a7528dd4560
--- /dev/null
+++ b/clang/test/Driver/hip-temps-linux.hip
@@ -0,0 +1,18 @@
+// REQUIRES: x86-registered-target
+// REQUIRES: amdgpu-registered-target
+// REQUIRES: system-linux
+
+// Check no temporary files or directores are left after compilation.
+// RUN: rm -rf %t/mytmp
+// RUN: mkdir -p %t/mytmp
+// RUN: env TMPDIR="%t/mytmp" %clang --target=x86_64-linux-gnu -nogpulib -nogpuinc \
+// RUN:   --rocm-path=%S/Inputs/rocm -nostdinc -nostdlib -c \
+// RUN:   --offload-arch=gfx1030 -emit-llvm -v %s 2>&1 | \
+// RUN:   FileCheck -check-prefixes=CHECK %s
+// RUN: ls %t/mytmp >%t/mytmp.txt 2>&1
+// RUN: touch %t/empty.txt
+// RUN: diff %t/mytmp.txt %t/empty.txt
+
+// CHECK: -o {{.*}}/mytmp/hip-temps-linux-gfx1030-{{.*}}.bc
+
+int main() {}
diff --git a/clang/test/Driver/hip-temps-windows.hip b/clang/test/Driver/hip-temps-windows.hip
new file mode 100644
index 0000000000000..ac1ac208a65dc
--- /dev/null
+++ b/clang/test/Driver/hip-temps-windows.hip
@@ -0,0 +1,18 @@
+// REQUIRES: x86-registered-target
+// REQUIRES: amdgpu-registered-target
+// REQUIRES: system-windows
+
+// Check no temporary files or directores are left after compilation.
+// RUN: rm -rf %t/mytmp
+// RUN: mkdir -p %t/mytmp
+// RUN: env TMP="%t/mytmp" %clang --target=x86_64-pc-windows-msvc -nogpulib -nogpuinc \
+// RUN:   --rocm-path=%S/Inputs/rocm -nostdinc -nostdlib -c \
+// RUN:   --offload-arch=gfx1030 -emit-llvm -v %s 2>&1 | \
+// RUN:   FileCheck -check-prefixes=CHECK %s
+// RUN: ls %t/mytmp >%t/mytmp.txt 2>&1
+// RUN: touch %t/empty.txt
+// RUN: diff %t/mytmp.txt %t/empty.txt
+
+// CHECK: -o "{{.*}}mytmp{{/|\\\\}}hip-temps-windows-gfx1030-{{.*}}.bc"
+
+int main() {}
diff --git a/clang/test/Driver/ibm-profiling.c b/clang/test/Driver/ibm-profiling.c
new file mode 100644
index 0000000000000..26bc0d7784373
--- /dev/null
+++ b/clang/test/Driver/ibm-profiling.c
@@ -0,0 +1,27 @@
+// Check that -pg throws an error on z/OS.
+// RUN: %clang -### 2>&1 --target=s390x-none-zos -S -pg %s | FileCheck -check-prefix=FAIL-PG-NAME %s
+// FAIL-PG-NAME: error: unsupported option '-pg' for target 's390x-none-zos'
+
+// Check that -p is still used when not linking on AIX.
+// RUN: %clang -### 2>&1 --target=powerpc-ibm-aix7.1.0.0 -S -p -S %s \
+// RUN:   | FileCheck --check-prefix=CHECK %s
+// CHECK-NOT: warning: argument unused during compilation: '-p'
+
+// Check precedence: -pg is unused when passed first on AIX.
+// RUN: %clang -### 2>&1 --target=powerpc-ibm-aix7.1.0.0 --sysroot %S/Inputs/aix_ppc_tree -pg -p %s \
+// RUN:        | FileCheck --check-prefix=CHECK2 %s
+// CHECK2-NOT: warning: argument unused during compilation: '-p' [-Wunused-command-line-argument]
+// CHECK2:     "-isysroot" "[[SYSROOT:[^"]+]]"
+// CHECK2:     "[[SYSROOT]]/usr/lib{{/|\\\\}}mcrt0.o"
+// CHECK2:     "-L[[SYSROOT]]/lib/profiled"
+// CHECK2:     "-L[[SYSROOT]]/usr/lib/profiled"
+
+// Check precedence: -p is unused when passed first on AIX.
+// RUN: %clang -### 2>&1 --target=powerpc-ibm-aix7.1.0.0 --sysroot %S/Inputs/aix_ppc_tree -p -pg %s \
+// RUN:        | FileCheck --check-prefix=CHECK3 %s
+// CHECK3: warning: argument unused during compilation: '-p' [-Wunused-command-line-argument]
+// CHECK3:     "-isysroot" "[[SYSROOT:[^"]+]]"
+// CHECK3:     "[[SYSROOT]]/usr/lib{{/|\\\\}}gcrt0.o"
+// CHECK3:     "-L[[SYSROOT]]/lib/profiled"
+// CHECK3:     "-L[[SYSROOT]]/usr/lib/profiled"
+
diff --git a/clang/test/Driver/linker-wrapper-libs.c b/clang/test/Driver/linker-wrapper-libs.c
index 65d4c2546ba9c..ee05304b7f1a2 100644
--- a/clang/test/Driver/linker-wrapper-libs.c
+++ b/clang/test/Driver/linker-wrapper-libs.c
@@ -2,8 +2,6 @@
 // REQUIRES: nvptx-registered-target
 // REQUIRES: amdgpu-registered-target
 
-// REQUIRES: system-linux
-
 // RUN: %clang -cc1 %s -triple x86_64-unknown-linux-gnu -emit-obj -o %t.elf.o
 
 #if defined(RESOLVES)
diff --git a/clang/test/Driver/output-file-cleanup.c b/clang/test/Driver/output-file-cleanup.c
index a5ca3e546f532..3628df8192652 100644
--- a/clang/test/Driver/output-file-cleanup.c
+++ b/clang/test/Driver/output-file-cleanup.c
@@ -1,3 +1,4 @@
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: rm -f "%t.d" "%t1.s" "%t2.s" "%t3.s" "%t4.s" "%t5.s"
 //
 // RUN: touch %t.s
diff --git a/clang/test/Driver/ppc-cpus.c b/clang/test/Driver/ppc-cpus.c
index 050512f9d87be..4edd83b4d6d28 100644
--- a/clang/test/Driver/ppc-cpus.c
+++ b/clang/test/Driver/ppc-cpus.c
@@ -39,3 +39,6 @@
 // RUN: %clang -### -c -target powerpc64 %s -mcpu=405     2>&1 | FileCheck %s --check-prefix=GENERIC
 //
 // GENERIC: "-target-cpu" "ppc64"
+
+// RUN: %clang -### -c --target=powerpc64 %s -march=generic 2>&1 | FileCheck --check-prefix=MARCH %s
+// MARCH: error: unsupported option '-march=' for target 'powerpc64'
diff --git a/clang/test/Driver/rewrite-map-in-diagnostics.c b/clang/test/Driver/rewrite-map-in-diagnostics.c
index d49d0ec8358fb..5ca8b897e199a 100644
--- a/clang/test/Driver/rewrite-map-in-diagnostics.c
+++ b/clang/test/Driver/rewrite-map-in-diagnostics.c
@@ -1,3 +1,4 @@
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: rm -rf "%t"
 // RUN: mkdir -p "%t"
 // RUN: env TMPDIR="%t" TEMP="%t" TMP="%t" RC_DEBUG_OPTION=1 \
diff --git a/clang/test/Driver/riscv-arch.c b/clang/test/Driver/riscv-arch.c
index 26a58418fe457..610f79d64ada2 100644
--- a/clang/test/Driver/riscv-arch.c
+++ b/clang/test/Driver/riscv-arch.c
@@ -150,7 +150,7 @@
 
 // RUN: %clang --target=riscv32-unknown-elf -march=rv32id -### %s \
 // RUN: -fsyntax-only 2>&1 | FileCheck -check-prefix=RV32ID %s
-// RV32ID: error: invalid arch name 'rv32id'
+// RV32ID: "-target-feature" "+f" "-target-feature" "+d"
 
 // RUN: %clang --target=riscv32-unknown-elf -march=rv32l -### %s \
 // RUN: -fsyntax-only 2>&1 | FileCheck -check-prefix=RV32L %s
@@ -178,7 +178,7 @@
 
 // RUN: %clang --target=riscv64-unknown-elf -march=rv64id -### %s \
 // RUN: -fsyntax-only 2>&1 | FileCheck -check-prefix=RV64ID %s
-// RV64ID: error: invalid arch name 'rv64id'
+// RV64ID: "-target-feature" "+f" "-target-feature" "+d"
 
 // RUN: %clang --target=riscv64-unknown-elf -march=rv64l -### %s \
 // RUN: -fsyntax-only 2>&1 | FileCheck -check-prefix=RV64L %s
@@ -228,25 +228,20 @@
 // RV64-EER: error: invalid arch name 'rv64e',
 // RV64-EER: standard user-level extension 'e' requires 'rv32'
 
-// RUN: %clang --target=riscv32-unknown-elf -march=rv32id -### %s \
-// RUN: -fsyntax-only 2>&1 | FileCheck -check-prefix=RV32-DER %s
-// RV32-DER: error: invalid arch name 'rv32id',
-// RV32-DER: d requires f extension to also be specified
-
 // RUN: %clang --target=riscv32-unknown-elf -march=rv32izve32f -### %s \
 // RUN: -fsyntax-only 2>&1 | FileCheck -check-prefix=RV32-ZVE32F-ER %s
 // RV32-ZVE32F-ER: error: invalid arch name 'rv32izve32f',
-// RV32-ZVE32F-ER: zve32f requires f or zfinx extension to also be specified
+// RV32-ZVE32F-ER: 'zve32f' requires 'f' or 'zfinx' extension to also be specified
 
 // RUN: %clang --target=riscv32-unknown-elf -march=rv32ifzve64d -### %s \
 // RUN: -fsyntax-only 2>&1 | FileCheck -check-prefix=RV32-ZVE64D-ER %s
 // RV32-ZVE64D-ER: error: invalid arch name 'rv32ifzve64d',
-// RV32-ZVE64D-ER: zve64d requires d or zdinx extension to also be specified
+// RV32-ZVE64D-ER: 'zve64d' requires 'd' or 'zdinx' extension to also be specified
 
 // RUN: %clang --target=riscv32-unknown-elf -march=rv32izvl64b -### %s \
 // RUN: -fsyntax-only 2>&1 | FileCheck -check-prefix=RV32-ZVL64B-ER %s
 // RV32-ZVL64B-ER: error: invalid arch name 'rv32izvl64b',
-// RV32-ZVL64B-ER: zvl*b requires v or zve* extension to also be specified
+// RV32-ZVL64B-ER: 'zvl*b' requires 'v' or 'zve*' extension to also be specified
 
 // RUN: %clang --target=riscv32-unknown-elf -march=rv32imw -### %s \
 // RUN: -fsyntax-only 2>&1 | FileCheck -check-prefix=RV32-STD-INVAL %s
@@ -533,7 +528,7 @@
 
 // RUN: %clang --target=riscv32-unknown-elf -march=rv32izve32f -### %s -c 2>&1 | \
 // RUN:   FileCheck -check-prefix=RV32-ZVE32F-REQUIRE-F %s
-// RV32-ZVE32F-REQUIRE-F: error: invalid arch name 'rv32izve32f', zve32f requires f or zfinx extension to also be specified
+// RV32-ZVE32F-REQUIRE-F: error: invalid arch name 'rv32izve32f', 'zve32f' requires 'f' or 'zfinx' extension to also be specified
 
 // RUN: %clang --target=riscv32-unknown-elf -march=rv32ifzve32f -### %s -c 2>&1 | \
 // RUN:   FileCheck -check-prefix=RV32-ZVE32F-GOOD %s
@@ -545,7 +540,7 @@
 
 // RUN: %clang --target=riscv32-unknown-elf -march=rv32izve64f -### %s -c 2>&1 | \
 // RUN:   FileCheck -check-prefix=RV32-ZVE64F-REQUIRE-F %s
-// RV32-ZVE64F-REQUIRE-F: error: invalid arch name 'rv32izve64f', zve32f requires f or zfinx extension to also be specified
+// RV32-ZVE64F-REQUIRE-F: error: invalid arch name 'rv32izve64f', 'zve32f' requires 'f' or 'zfinx' extension to also be specified
 
 // RUN: %clang --target=riscv32-unknown-elf -march=rv32ifzve64f -### %s -c 2>&1 | \
 // RUN:   FileCheck -check-prefix=RV32-ZVE64F-GOOD %s
@@ -553,7 +548,7 @@
 
 // RUN: %clang --target=riscv32-unknown-elf -march=rv32ifzve64d -### %s -c 2>&1 | \
 // RUN:   FileCheck -check-prefix=RV32-ZVE64D-REQUIRE-D %s
-// RV32-ZVE64D-REQUIRE-D: error: invalid arch name 'rv32ifzve64d', zve64d requires d or zdinx extension to also be specified
+// RV32-ZVE64D-REQUIRE-D: error: invalid arch name 'rv32ifzve64d', 'zve64d' requires 'd' or 'zdinx' extension to also be specified
 
 // RUN: %clang --target=riscv32-unknown-elf -march=rv32ifdzve64d -### %s -c 2>&1 | \
 // RUN:   FileCheck -check-prefix=RV32-ZVE64D-GOOD %s
@@ -588,3 +583,13 @@
 // RUN: %clang -target riscv32-unknown-elf -march=rv32i_zmmul1p0 -### %s \
 // RUN: -fsyntax-only 2>&1 | FileCheck -check-prefix=RV32-ZMMUL-GOODVERS %s
 // RV32-ZMMUL-GOODVERS: "-target-feature" "+zmmul"
+
+// RUN: %clang --target=riscv32-unknown-elf -march=rv32ifzfinx -### %s \
+// RUN: -fsyntax-only 2>&1 | FileCheck -check-prefix=RV32-F-ZFINX-ER %s
+// RV32-F-ZFINX-ER: error: invalid arch name 'rv32ifzfinx',
+// RV32-F-ZFINX-ER: 'f' and 'zfinx' extensions are incompatible
+
+// RUN: %clang --target=riscv32-unknown-elf -march=rv32idzdinx -### %s \
+// RUN: -fsyntax-only 2>&1 | FileCheck -check-prefix=RV32-D-ZDINX-ER %s
+// RV32-D-ZDINX-ER: error: invalid arch name 'rv32idzdinx',
+// RV32-D-ZFINX-ER: 'f' and 'zfinx' extensions are incompatible
diff --git a/clang/test/Driver/sanitizer-ld.c b/clang/test/Driver/sanitizer-ld.c
index 0ba209d870c2a..0778a96ae2350 100644
--- a/clang/test/Driver/sanitizer-ld.c
+++ b/clang/test/Driver/sanitizer-ld.c
@@ -457,6 +457,18 @@
 // RUN:   | FileCheck --check-prefix=CHECK-UBSAN-STATIC-DARWIN %s
 // CHECK-UBSAN-STATIC-DARWIN: {{.*}}error: static UndefinedBehaviorSanitizer runtime is not supported on darwin
 
+// RUN: %clang -fsanitize=address -### %s 2>&1 \
+// RUN:     --target=x86_64-apple-darwin -fuse-ld=ld -static-libsan \
+// RUN:     --sysroot=%S/Inputs/basic_linux_tree \
+// RUN:   | FileCheck --check-prefix=CHECK-ASAN-STATIC-DARWIN %s
+// CHECK-ASAN-STATIC-DARWIN: {{.*}}error: static AddressSanitizer runtime is not supported on darwin
+
+// RUN: %clang -fsanitize=thread -### %s 2>&1 \
+// RUN:     --target=x86_64-apple-darwin -fuse-ld=ld -static-libsan \
+// RUN:     --sysroot=%S/Inputs/basic_linux_tree \
+// RUN:   | FileCheck --check-prefix=CHECK-TSAN-STATIC-DARWIN %s
+// CHECK-TSAN-STATIC-DARWIN: {{.*}}error: static ThreadSanitizer runtime is not supported on darwin
+
 // RUN: %clang -fsanitize=address,undefined -### %s 2>&1 \
 // RUN:     --target=i386-unknown-linux -fuse-ld=ld \
 // RUN:     -resource-dir=%S/Inputs/resource_dir \
@@ -731,6 +743,10 @@
 // RUN:   | FileCheck --check-prefix=CHECK-SHADOWCALLSTACK-LINUX-RISCV64 %s
 // CHECK-SHADOWCALLSTACK-LINUX-RISCV64: '-fsanitize=shadow-call-stack' only allowed with '-ffixed-x18'
 
+// RUN: %clang -target riscv64-linux-android -fsanitize=shadow-call-stack %s -### 2>&1 \
+// RUN:   | FileCheck %s --check-prefix=CHECK-SHADOWCALLSTACK-ANDROID-RISCV64
+// CHECK-SHADOWCALLSTACK-ANDROID-RISCV64-NOT: error:
+
 // RUN: %clang -fsanitize=shadow-call-stack -### %s 2>&1 \
 // RUN:     --target=riscv64-unknown-fuchsia -fuse-ld=ld \
 // RUN:   | FileCheck --check-prefix=CHECK-SHADOWCALLSTACK-FUCHSIA-RISCV64 %s
diff --git a/clang/test/Driver/save-stats.c b/clang/test/Driver/save-stats.c
index d70220331ecbc..3c212f4ec4ab5 100644
--- a/clang/test/Driver/save-stats.c
+++ b/clang/test/Driver/save-stats.c
@@ -26,3 +26,14 @@
 
 // RUN: %clang -target x86_64-linux-unknown -save-stats=obj -flto -o obj/dir/save-stats.exe %s -### 2>&1 | FileCheck %s -check-prefix=CHECK-LTO-OBJ
 // CHECK-LTO-OBJ: "-plugin-opt=stats-file=obj/dir{{/|\\\\}}save-stats.stats"
+
+// RUN: env CC_PRINT_INTERNAL_STAT=1 \
+// RUN: %clang -target x86_64-apple-darwin %s -### 2>&1 | FileCheck %s -check-prefix=CHECK-ENV
+// CHECK-ENV: "-stats-file=-"
+// CHECK-ENV-NO: "stats-file-append"
+
+// RUN: env CC_PRINT_INTERNAL_STAT=1 \
+// RUN:     CC_PRINT_INTERNAL_STAT_FILE=/tmp/stats.json \
+// RUN: %clang -target x86_64-apple-darwin %s -### 2>&1 | FileCheck %s -check-prefix=CHECK-ENV-FILE
+// CHECK-ENV-FILE: "-stats-file=/tmp/stats.json"
+// CHECK-ENV-FILE: "-stats-file-append"
diff --git a/clang/test/Driver/zos-profiling-error.c b/clang/test/Driver/zos-profiling-error.c
deleted file mode 100644
index e969dc31a245a..0000000000000
--- a/clang/test/Driver/zos-profiling-error.c
+++ /dev/null
@@ -1,2 +0,0 @@
-// RUN: %clang 2>&1 -### --target=s390x-none-zos -pg -S %s | FileCheck -check-prefix=FAIL-PG-NAME %s
-// FAIL-PG-NAME: error: unsupported option '-pg' for target 's390x-none-zos'
diff --git a/clang/test/ExtractAPI/ignored-symbols-multifile.c b/clang/test/ExtractAPI/ignored-symbols-multifile.c
new file mode 100644
index 0000000000000..d4a4c3dde1252
--- /dev/null
+++ b/clang/test/ExtractAPI/ignored-symbols-multifile.c
@@ -0,0 +1,44 @@
+// RUN: rm -rf %t
+// RUN: split-file %s %t
+// RUN: %clang_cc1 -extract-api -triple arm64-apple-macosx \
+// RUN:   --extract-api-ignores=%t/ignores-list1,%t/ignores-list2,%t/ignores-list3 \
+// RUN:   -x c-header %t/input.h -verify -o - | FileCheck %t/input.h
+
+//--- input.h
+#define IGNORED_6_FILE1 6
+#define IGNORED_2_FILE1 2
+#define IGNORED_5_FILE1 5
+
+#define IGNORED_4_FILE2 4
+#define IGNORED_3_FILE2 3
+
+typedef double IGNORED_1_FILE3;
+typedef int IGNORED_7_FILE3;
+
+typedef float NonIgnored;
+
+// CHECK-NOT: IGNORED_6_FILE1
+// CHECK-NOT: IGNORED_2_FILE1
+// CHECK-NOT: IGNORED_5_FILE1
+
+// CHECK-NOT: IGNORED_4_FILE2
+// CHECK-NOT: IGNORED_3_FILE2
+
+// CHECK-NOT: IGNORED_1_FILE3
+// CHECK-NOT: IGNORED_7_FILE3
+// CHECK: NonIgnored
+
+// expected-no-diagnostics
+
+//--- ignores-list1
+IGNORED_6_FILE1
+IGNORED_2_FILE1
+IGNORED_5_FILE1
+
+//--- ignores-list2
+IGNORED_4_FILE2
+IGNORED_3_FILE2
+
+//--- ignores-list3
+IGNORED_1_FILE3
+IGNORED_7_FILE3
diff --git a/clang/test/Frontend/amdgcn-machine-analysis-remarks.cl b/clang/test/Frontend/amdgcn-machine-analysis-remarks.cl
index cf0c15b6319f1..9403d12afa05a 100644
--- a/clang/test/Frontend/amdgcn-machine-analysis-remarks.cl
+++ b/clang/test/Frontend/amdgcn-machine-analysis-remarks.cl
@@ -2,7 +2,7 @@
 // RUN: %clang_cc1 -triple amdgcn-amd-amdhsa -target-cpu gfx908 -Rpass-analysis=kernel-resource-usage -S -O0 -verify %s -o /dev/null
 
 // expected-remark@+9 {{Function Name: foo}}
-// expected-remark@+8 {{    SGPRs: 9}}
+// expected-remark@+8 {{    SGPRs: 13}}
 // expected-remark@+7 {{    VGPRs: 10}}
 // expected-remark@+6 {{    AGPRs: 12}}
 // expected-remark@+5 {{    ScratchSize [bytes/lane]: 0}}
diff --git a/clang/test/Frontend/darwin-version.c b/clang/test/Frontend/darwin-version.c
index 67f087feafe25..28826014e39f1 100644
--- a/clang/test/Frontend/darwin-version.c
+++ b/clang/test/Frontend/darwin-version.c
@@ -59,6 +59,11 @@
 
 // RUN: %clang_cc1 -triple arm64-apple-ios99.99.99 -dM -E %s | FileCheck --check-prefix=IOS99 %s
 // IOS99: __ENVIRONMENT_IPHONE_OS_VERSION_MIN_REQUIRED__ 999999
+// IOS99-NEXT: __ENVIRONMENT_OS_VERSION_MIN_REQUIRED__ 999999
 
 // RUN: %clang_cc1 -triple arm64-apple-watchos99.9 -dM -E %s | FileCheck --check-prefix=WATCHOS99 %s
-// WATCHOS99: __ENVIRONMENT_WATCH_OS_VERSION_MIN_REQUIRED__ 990900
+// WATCHOS99: __ENVIRONMENT_OS_VERSION_MIN_REQUIRED__ 990900
+// WATCHOS99-NEXT: __ENVIRONMENT_WATCH_OS_VERSION_MIN_REQUIRED__ 990900
+
+// RUN: %clang_cc1 -triple arm64-apple-macos12.4 -dM -E %s | FileCheck --check-prefix=MACOS124 %s
+// MACOS124: __ENVIRONMENT_OS_VERSION_MIN_REQUIRED__ 120400
diff --git a/clang/test/Frontend/plugin-attribute-pp.cpp b/clang/test/Frontend/plugin-attribute-pp.cpp
new file mode 100644
index 0000000000000..94ba14e272eef
--- /dev/null
+++ b/clang/test/Frontend/plugin-attribute-pp.cpp
@@ -0,0 +1,28 @@
+// RUN: %clang -fplugin=%llvmshlibdir/Attribute%pluginext -E %s | FileCheck %s
+// RUN: %clang -fplugin=%llvmshlibdir/Attribute%pluginext -E %s -x c | FileCheck %s
+// REQUIRES: plugins, examples
+
+#ifdef __cplusplus
+# define HAS_ATTR(a) __has_cpp_attribute (a)
+#else
+# define HAS_ATTR(a) __has_c_attribute (a)
+#endif
+
+#if __has_attribute(example)
+// CHECK: has_attribute(example) was true
+has_attribute(example) was true
+#endif
+#if HAS_ATTR(example)
+// CHECK: has_$LANG_attribute(example) was true
+has_$LANG_attribute(example) was true
+#endif
+
+#if __has_attribute(doesnt_exist)
+// CHECK-NOT: has_attribute(doesnt_exist) unexpectedly was true
+has_attribute(doesnt_exist) unexpectedly was true
+#endif
+
+#if HAS_ATTR(doesnt_exist)
+// CHECK-NOT: has_$LANG_attribute(doesnt_exist) unexpectedly was true
+has_$LANG_attribute(doesnt_exist) unexpectedly was true
+#endif
diff --git a/clang/test/Headers/__clang_hip_cmath.hip b/clang/test/Headers/__clang_hip_cmath.hip
index e8d9445204072..03dbc4b675908 100644
--- a/clang/test/Headers/__clang_hip_cmath.hip
+++ b/clang/test/Headers/__clang_hip_cmath.hip
@@ -18,12 +18,12 @@
 
 // DEFAULT-LABEL: @test_fma_f16(
 // DEFAULT-NEXT:  entry:
-// DEFAULT-NEXT:    [[CALL_I:%.*]] = tail call contract half @__ocml_fma_f16(half noundef [[X:%.*]], half noundef [[Y:%.*]], half noundef [[Z:%.*]]) #[[ATTR6:[0-9]+]]
+// DEFAULT-NEXT:    [[CALL_I:%.*]] = tail call contract half @__ocml_fma_f16(half noundef [[X:%.*]], half noundef [[Y:%.*]], half noundef [[Z:%.*]]) #[[ATTR4:[0-9]+]]
 // DEFAULT-NEXT:    ret half [[CALL_I]]
 //
 // FINITEONLY-LABEL: @test_fma_f16(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract half @__ocml_fma_f16(half noundef [[X:%.*]], half noundef [[Y:%.*]], half noundef [[Z:%.*]]) #[[ATTR6:[0-9]+]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) half @__ocml_fma_f16(half noundef nofpclass(nan inf) [[X:%.*]], half noundef nofpclass(nan inf) [[Y:%.*]], half noundef nofpclass(nan inf) [[Z:%.*]]) #[[ATTR4:[0-9]+]]
 // FINITEONLY-NEXT:    ret half [[CALL_I]]
 //
 extern "C" __device__ _Float16 test_fma_f16(_Float16 x, _Float16 y,
@@ -33,12 +33,12 @@ extern "C" __device__ _Float16 test_fma_f16(_Float16 x, _Float16 y,
 
 // DEFAULT-LABEL: @test_pow_f16(
 // DEFAULT-NEXT:  entry:
-// DEFAULT-NEXT:    [[CALL_I:%.*]] = tail call contract half @__ocml_pown_f16(half noundef [[X:%.*]], i32 noundef [[Y:%.*]]) #[[ATTR7:[0-9]+]]
+// DEFAULT-NEXT:    [[CALL_I:%.*]] = tail call contract half @__ocml_pown_f16(half noundef [[X:%.*]], i32 noundef [[Y:%.*]]) #[[ATTR5:[0-9]+]]
 // DEFAULT-NEXT:    ret half [[CALL_I]]
 //
 // FINITEONLY-LABEL: @test_pow_f16(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract half @__ocml_pown_f16(half noundef [[X:%.*]], i32 noundef [[Y:%.*]]) #[[ATTR7:[0-9]+]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) half @__ocml_pown_f16(half noundef nofpclass(nan inf) [[X:%.*]], i32 noundef [[Y:%.*]]) #[[ATTR5:[0-9]+]]
 // FINITEONLY-NEXT:    ret half [[CALL_I]]
 //
 extern "C" __device__ _Float16 test_pow_f16(_Float16 x, int y) {
diff --git a/clang/test/Headers/__clang_hip_math.hip b/clang/test/Headers/__clang_hip_math.hip
index edc9731de9bbd..ec774c43f8124 100644
--- a/clang/test/Headers/__clang_hip_math.hip
+++ b/clang/test/Headers/__clang_hip_math.hip
@@ -255,7 +255,7 @@ extern "C" __device__ long long test_llabs(long x) {
 //
 // FINITEONLY-LABEL: @test_acosf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_acos_f32(float noundef [[X:%.*]]) #[[ATTR13:[0-9]+]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_acos_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13:[0-9]+]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_acosf(float x) {
@@ -269,7 +269,7 @@ extern "C" __device__ float test_acosf(float x) {
 //
 // FINITEONLY-LABEL: @test_acos(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_acos_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_acos_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_acos(double x) {
@@ -283,7 +283,7 @@ extern "C" __device__ double test_acos(double x) {
 //
 // FINITEONLY-LABEL: @test_acoshf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_acosh_f32(float noundef [[X:%.*]]) #[[ATTR14:[0-9]+]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_acosh_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14:[0-9]+]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_acoshf(float x) {
@@ -297,7 +297,7 @@ extern "C" __device__ float test_acoshf(float x) {
 //
 // FINITEONLY-LABEL: @test_acosh(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_acosh_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_acosh_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_acosh(double x) {
@@ -311,7 +311,7 @@ extern "C" __device__ double test_acosh(double x) {
 //
 // FINITEONLY-LABEL: @test_asinf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_asin_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_asin_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_asinf(float x) {
@@ -325,7 +325,7 @@ extern "C" __device__ float test_asinf(float x) {
 //
 // FINITEONLY-LABEL: @test_asin(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_asin_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_asin_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_asin(double x) {
@@ -340,7 +340,7 @@ extern "C" __device__ double test_asin(double x) {
 //
 // FINITEONLY-LABEL: @test_asinhf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_asinh_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_asinh_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_asinhf(float x) {
@@ -354,7 +354,7 @@ extern "C" __device__ float test_asinhf(float x) {
 //
 // FINITEONLY-LABEL: @test_asinh(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_asinh_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_asinh_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_asinh(double x) {
@@ -368,7 +368,7 @@ extern "C" __device__ double test_asinh(double x) {
 //
 // FINITEONLY-LABEL: @test_atan2f(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_atan2_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_atan2_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_atan2f(float x, float y) {
@@ -382,7 +382,7 @@ extern "C" __device__ float test_atan2f(float x, float y) {
 //
 // FINITEONLY-LABEL: @test_atan2(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_atan2_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_atan2_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_atan2(double x, double y) {
@@ -396,7 +396,7 @@ extern "C" __device__ double test_atan2(double x, double y) {
 //
 // FINITEONLY-LABEL: @test_atanf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_atan_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_atan_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_atanf(float x) {
@@ -410,7 +410,7 @@ extern "C" __device__ float test_atanf(float x) {
 //
 // FINITEONLY-LABEL: @test_atan(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_atan_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_atan_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_atan(double x) {
@@ -424,7 +424,7 @@ extern "C" __device__ double test_atan(double x) {
 //
 // FINITEONLY-LABEL: @test_atanhf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_atanh_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_atanh_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_atanhf(float x) {
@@ -438,7 +438,7 @@ extern "C" __device__ float test_atanhf(float x) {
 //
 // FINITEONLY-LABEL: @test_atanh(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_atanh_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_atanh_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_atanh(double x) {
@@ -452,7 +452,7 @@ extern "C" __device__ double test_atanh(double x) {
 //
 // FINITEONLY-LABEL: @test_cbrtf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_cbrt_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_cbrt_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_cbrtf(float x) {
@@ -466,7 +466,7 @@ extern "C" __device__ float test_cbrtf(float x) {
 //
 // FINITEONLY-LABEL: @test_cbrt(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_cbrt_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_cbrt_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_cbrt(double x) {
@@ -480,7 +480,7 @@ extern "C" __device__ double test_cbrt(double x) {
 //
 // FINITEONLY-LABEL: @test_ceilf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_ceil_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_ceil_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_ceilf(float x) {
@@ -494,7 +494,7 @@ extern "C" __device__ float test_ceilf(float x) {
 //
 // FINITEONLY-LABEL: @test_ceil(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_ceil_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_ceil_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_ceil(double x) {
@@ -508,7 +508,7 @@ extern "C" __device__ double test_ceil(double x) {
 //
 // FINITEONLY-LABEL: @test_copysignf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_copysign_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_copysign_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_copysignf(float x, float y) {
@@ -522,7 +522,7 @@ extern "C" __device__ float test_copysignf(float x, float y) {
 //
 // FINITEONLY-LABEL: @test_copysign(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_copysign_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_copysign_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_copysign(double x, double y) {
@@ -536,7 +536,7 @@ extern "C" __device__ double test_copysign(double x, double y) {
 //
 // FINITEONLY-LABEL: @test_cosf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_cos_f32(float noundef [[X:%.*]]) #[[ATTR15:[0-9]+]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_cos_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15:[0-9]+]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_cosf(float x) {
@@ -550,7 +550,7 @@ extern "C" __device__ float test_cosf(float x) {
 //
 // FINITEONLY-LABEL: @test_cos(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_cos_f64(double noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_cos_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_cos(double x) {
@@ -564,7 +564,7 @@ extern "C" __device__ double test_cos(double x) {
 //
 // FINITEONLY-LABEL: @test_coshf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_cosh_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_cosh_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_coshf(float x) {
@@ -578,7 +578,7 @@ extern "C" __device__ float test_coshf(float x) {
 //
 // FINITEONLY-LABEL: @test_cosh(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_cosh_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_cosh_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_cosh(double x) {
@@ -592,7 +592,7 @@ extern "C" __device__ double test_cosh(double x) {
 //
 // FINITEONLY-LABEL: @test_cospif(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_cospi_f32(float noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_cospi_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_cospif(float x) {
@@ -606,7 +606,7 @@ extern "C" __device__ float test_cospif(float x) {
 //
 // FINITEONLY-LABEL: @test_cospi(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_cospi_f64(double noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_cospi_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_cospi(double x) {
@@ -620,7 +620,7 @@ extern "C" __device__ double test_cospi(double x) {
 //
 // FINITEONLY-LABEL: @test_cyl_bessel_i0f(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_i0_f32(float noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_i0_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_cyl_bessel_i0f(float x) {
@@ -634,7 +634,7 @@ extern "C" __device__ float test_cyl_bessel_i0f(float x) {
 //
 // FINITEONLY-LABEL: @test_cyl_bessel_i0(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_i0_f64(double noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_i0_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_cyl_bessel_i0(double x) {
@@ -648,7 +648,7 @@ extern "C" __device__ double test_cyl_bessel_i0(double x) {
 //
 // FINITEONLY-LABEL: @test_cyl_bessel_i1f(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_i1_f32(float noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_i1_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_cyl_bessel_i1f(float x) {
@@ -662,7 +662,7 @@ extern "C" __device__ float test_cyl_bessel_i1f(float x) {
 //
 // FINITEONLY-LABEL: @test_cyl_bessel_i1(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_i1_f64(double noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_i1_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_cyl_bessel_i1(double x) {
@@ -676,7 +676,7 @@ extern "C" __device__ double test_cyl_bessel_i1(double x) {
 //
 // FINITEONLY-LABEL: @test_erfcf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_erfc_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_erfc_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_erfcf(float x) {
@@ -690,7 +690,7 @@ extern "C" __device__ float test_erfcf(float x) {
 //
 // FINITEONLY-LABEL: @test_erfc(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_erfc_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_erfc_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_erfc(double x) {
@@ -704,7 +704,7 @@ extern "C" __device__ double test_erfc(double x) {
 //
 // FINITEONLY-LABEL: @test_erfinvf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_erfinv_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_erfinv_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_erfinvf(float x) {
@@ -718,7 +718,7 @@ extern "C" __device__ float test_erfinvf(float x) {
 //
 // FINITEONLY-LABEL: @test_erfinv(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_erfinv_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_erfinv_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_erfinv(double x) {
@@ -732,7 +732,7 @@ extern "C" __device__ double test_erfinv(double x) {
 //
 // FINITEONLY-LABEL: @test_exp10f(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_exp10_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_exp10_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_exp10f(float x) {
@@ -746,7 +746,7 @@ extern "C" __device__ float test_exp10f(float x) {
 //
 // FINITEONLY-LABEL: @test_exp10(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_exp10_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_exp10_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_exp10(double x) {
@@ -760,14 +760,13 @@ extern "C" __device__ double test_exp10(double x) {
 //
 // FINITEONLY-LABEL: @test_exp2f(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_exp2_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_exp2_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_exp2f(float x) {
   return exp2f(x);
 }
 
-//
 // DEFAULT-LABEL: @test_exp2(
 // DEFAULT-NEXT:  entry:
 // DEFAULT-NEXT:    [[CALL_I:%.*]] = tail call contract double @__ocml_exp2_f64(double noundef [[X:%.*]]) #[[ATTR14]]
@@ -775,7 +774,7 @@ extern "C" __device__ float test_exp2f(float x) {
 //
 // FINITEONLY-LABEL: @test_exp2(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_exp2_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_exp2_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_exp2(double x) {
@@ -789,7 +788,7 @@ extern "C" __device__ double test_exp2(double x) {
 //
 // FINITEONLY-LABEL: @test_expf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_exp_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_exp_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_expf(float x) {
@@ -803,7 +802,7 @@ extern "C" __device__ float test_expf(float x) {
 //
 // FINITEONLY-LABEL: @test_exp(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_exp_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_exp_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_exp(double x) {
@@ -817,7 +816,7 @@ extern "C" __device__ double test_exp(double x) {
 //
 // FINITEONLY-LABEL: @test_expm1f(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_expm1_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_expm1_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_expm1f(float x) {
@@ -831,7 +830,7 @@ extern "C" __device__ float test_expm1f(float x) {
 //
 // FINITEONLY-LABEL: @test_expm1(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_expm1_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_expm1_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_expm1(double x) {
@@ -873,7 +872,7 @@ extern "C" __device__ double test_fabs(double x) {
 //
 // FINITEONLY-LABEL: @test_fdimf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_fdim_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_fdim_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_fdimf(float x, float y) {
@@ -887,7 +886,7 @@ extern "C" __device__ float test_fdimf(float x, float y) {
 //
 // FINITEONLY-LABEL: @test_fdim(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_fdim_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_fdim_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_fdim(double x, double y) {
@@ -915,7 +914,7 @@ extern "C" __device__ float test_fdividef(float x, float y) {
 //
 // FINITEONLY-LABEL: @test_floorf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_floor_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_floor_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_floorf(float x) {
@@ -929,7 +928,7 @@ extern "C" __device__ float test_floorf(float x) {
 //
 // FINITEONLY-LABEL: @test_floor(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_floor_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_floor_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_floor(double x) {
@@ -943,7 +942,7 @@ extern "C" __device__ double test_floor(double x) {
 //
 // FINITEONLY-LABEL: @test_fmaf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_fma_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]], float noundef [[Z:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_fma_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]], float noundef nofpclass(nan inf) [[Z:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_fmaf(float x, float y, float z) {
@@ -957,7 +956,7 @@ extern "C" __device__ float test_fmaf(float x, float y, float z) {
 //
 // FINITEONLY-LABEL: @test_fma(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_fma_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]], double noundef [[Z:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_fma_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]], double noundef nofpclass(nan inf) [[Z:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_fma(double x, double y, double z) {
@@ -971,7 +970,7 @@ extern "C" __device__ double test_fma(double x, double y, double z) {
 //
 // FINITEONLY-LABEL: @test_fma_rn(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_fma_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]], double noundef [[Z:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_fma_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]], double noundef nofpclass(nan inf) [[Z:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_fma_rn(double x, double y, double z) {
@@ -985,7 +984,7 @@ extern "C" __device__ double test_fma_rn(double x, double y, double z) {
 //
 // FINITEONLY-LABEL: @test_fmaxf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_fmax_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_fmax_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_fmaxf(float x, float y) {
@@ -999,7 +998,7 @@ extern "C" __device__ float test_fmaxf(float x, float y) {
 //
 // FINITEONLY-LABEL: @test_fmax(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_fmax_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_fmax_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_fmax(double x, double y) {
@@ -1013,7 +1012,7 @@ extern "C" __device__ double test_fmax(double x, double y) {
 //
 // FINITEONLY-LABEL: @test_fminf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_fmin_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_fmin_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_fminf(float x, float y) {
@@ -1027,7 +1026,7 @@ extern "C" __device__ float test_fminf(float x, float y) {
 //
 // FINITEONLY-LABEL: @test_fmin(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_fmin_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_fmin_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_fmin(double x, double y) {
@@ -1041,7 +1040,7 @@ extern "C" __device__ double test_fmin(double x, double y) {
 //
 // FINITEONLY-LABEL: @test_fmodf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_fmod_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_fmod_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_fmodf(float x, float y) {
@@ -1055,7 +1054,7 @@ extern "C" __device__ float test_fmodf(float x, float y) {
 //
 // FINITEONLY-LABEL: @test_fmod(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_fmod_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_fmod_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_fmod(double x, double y) {
@@ -1076,7 +1075,7 @@ extern "C" __device__ double test_fmod(double x, double y) {
 // FINITEONLY-NEXT:  entry:
 // FINITEONLY-NEXT:    [[__TMP_I:%.*]] = alloca i32, align 4, addrspace(5)
 // FINITEONLY-NEXT:    call void @llvm.lifetime.start.p5(i64 4, ptr addrspace(5) [[__TMP_I]]) #[[ATTR16:[0-9]+]]
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract float @__ocml_frexp_f32(float noundef [[X:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract nofpclass(nan inf) float @__ocml_frexp_f32(float noundef nofpclass(nan inf) [[X:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    [[TMP0:%.*]] = load i32, ptr addrspace(5) [[__TMP_I]], align 4, !tbaa [[TBAA11:![0-9]+]]
 // FINITEONLY-NEXT:    store i32 [[TMP0]], ptr [[Y:%.*]], align 4, !tbaa [[TBAA11]]
 // FINITEONLY-NEXT:    call void @llvm.lifetime.end.p5(i64 4, ptr addrspace(5) [[__TMP_I]]) #[[ATTR16]]
@@ -1100,7 +1099,7 @@ extern "C" __device__ float test_frexpf(float x, int* y) {
 // FINITEONLY-NEXT:  entry:
 // FINITEONLY-NEXT:    [[__TMP_I:%.*]] = alloca i32, align 4, addrspace(5)
 // FINITEONLY-NEXT:    call void @llvm.lifetime.start.p5(i64 4, ptr addrspace(5) [[__TMP_I]]) #[[ATTR16]]
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract double @__ocml_frexp_f64(double noundef [[X:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract nofpclass(nan inf) double @__ocml_frexp_f64(double noundef nofpclass(nan inf) [[X:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    [[TMP0:%.*]] = load i32, ptr addrspace(5) [[__TMP_I]], align 4, !tbaa [[TBAA11]]
 // FINITEONLY-NEXT:    store i32 [[TMP0]], ptr [[Y:%.*]], align 4, !tbaa [[TBAA11]]
 // FINITEONLY-NEXT:    call void @llvm.lifetime.end.p5(i64 4, ptr addrspace(5) [[__TMP_I]]) #[[ATTR16]]
@@ -1117,7 +1116,7 @@ extern "C" __device__ double test_frexp(double x, int* y) {
 //
 // FINITEONLY-LABEL: @test_hypotf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_hypot_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_hypot_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_hypotf(float x, float y) {
@@ -1131,92 +1130,144 @@ extern "C" __device__ float test_hypotf(float x, float y) {
 //
 // FINITEONLY-LABEL: @test_hypot(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_hypot_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_hypot_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_hypot(double x, double y) {
   return hypot(x, y);
 }
 
-// CHECK-LABEL: @test_ilogbf(
-// CHECK-NEXT:  entry:
-// CHECK-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_ilogb_f32(float noundef [[X:%.*]]) #[[ATTR13:[0-9]+]]
-// CHECK-NEXT:    ret i32 [[CALL_I]]
+// DEFAULT-LABEL: @test_ilogbf(
+// DEFAULT-NEXT:  entry:
+// DEFAULT-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_ilogb_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// DEFAULT-NEXT:    ret i32 [[CALL_I]]
+//
+// FINITEONLY-LABEL: @test_ilogbf(
+// FINITEONLY-NEXT:  entry:
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_ilogb_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    ret i32 [[CALL_I]]
 //
 extern "C" __device__ int test_ilogbf(float x) {
   return ilogbf(x);
 }
 
-// CHECK-LABEL: @test_ilogb(
-// CHECK-NEXT:  entry:
-// CHECK-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_ilogb_f64(double noundef [[X:%.*]]) #[[ATTR13]]
-// CHECK-NEXT:    ret i32 [[CALL_I]]
+// DEFAULT-LABEL: @test_ilogb(
+// DEFAULT-NEXT:  entry:
+// DEFAULT-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_ilogb_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// DEFAULT-NEXT:    ret i32 [[CALL_I]]
+//
+// FINITEONLY-LABEL: @test_ilogb(
+// FINITEONLY-NEXT:  entry:
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_ilogb_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    ret i32 [[CALL_I]]
 //
 extern "C" __device__ int test_ilogb(double x) {
   return ilogb(x);
 }
 
-// CHECK-LABEL: @test___finitef(
-// CHECK-NEXT:  entry:
-// CHECK-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_isfinite_f32(float noundef [[X:%.*]]) #[[ATTR13]]
-// CHECK-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
-// CHECK-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
-// CHECK-NEXT:    ret i32 [[CONV]]
+// DEFAULT-LABEL: @test___finitef(
+// DEFAULT-NEXT:  entry:
+// DEFAULT-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_isfinite_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// DEFAULT-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
+// DEFAULT-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
+// DEFAULT-NEXT:    ret i32 [[CONV]]
+//
+// FINITEONLY-LABEL: @test___finitef(
+// FINITEONLY-NEXT:  entry:
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_isfinite_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
+// FINITEONLY-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
+// FINITEONLY-NEXT:    ret i32 [[CONV]]
 //
 extern "C" __device__ BOOL_TYPE test___finitef(float x) {
   return __finitef(x);
 }
 
-// CHECK-LABEL: @test___finite(
-// CHECK-NEXT:  entry:
-// CHECK-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_isfinite_f64(double noundef [[X:%.*]]) #[[ATTR13]]
-// CHECK-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
-// CHECK-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
-// CHECK-NEXT:    ret i32 [[CONV]]
+// DEFAULT-LABEL: @test___finite(
+// DEFAULT-NEXT:  entry:
+// DEFAULT-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_isfinite_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// DEFAULT-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
+// DEFAULT-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
+// DEFAULT-NEXT:    ret i32 [[CONV]]
+//
+// FINITEONLY-LABEL: @test___finite(
+// FINITEONLY-NEXT:  entry:
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_isfinite_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
+// FINITEONLY-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
+// FINITEONLY-NEXT:    ret i32 [[CONV]]
 //
 extern "C" __device__ BOOL_TYPE test___finite(double x) {
   return __finite(x);
 }
 
-// CHECK-LABEL: @test___isinff(
-// CHECK-NEXT:  entry:
-// CHECK-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_isinf_f32(float noundef [[X:%.*]]) #[[ATTR13]]
-// CHECK-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
-// CHECK-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
-// CHECK-NEXT:    ret i32 [[CONV]]
+// DEFAULT-LABEL: @test___isinff(
+// DEFAULT-NEXT:  entry:
+// DEFAULT-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_isinf_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// DEFAULT-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
+// DEFAULT-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
+// DEFAULT-NEXT:    ret i32 [[CONV]]
+//
+// FINITEONLY-LABEL: @test___isinff(
+// FINITEONLY-NEXT:  entry:
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_isinf_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
+// FINITEONLY-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
+// FINITEONLY-NEXT:    ret i32 [[CONV]]
 //
 extern "C" __device__ BOOL_TYPE test___isinff(float x) {
   return __isinff(x);
 }
 
-// CHECK-LABEL: @test___isinf(
-// CHECK-NEXT:  entry:
-// CHECK-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_isinf_f64(double noundef [[X:%.*]]) #[[ATTR13]]
-// CHECK-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
-// CHECK-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
-// CHECK-NEXT:    ret i32 [[CONV]]
+// DEFAULT-LABEL: @test___isinf(
+// DEFAULT-NEXT:  entry:
+// DEFAULT-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_isinf_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// DEFAULT-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
+// DEFAULT-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
+// DEFAULT-NEXT:    ret i32 [[CONV]]
+//
+// FINITEONLY-LABEL: @test___isinf(
+// FINITEONLY-NEXT:  entry:
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_isinf_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
+// FINITEONLY-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
+// FINITEONLY-NEXT:    ret i32 [[CONV]]
 //
 extern "C" __device__ BOOL_TYPE test___isinf(double x) {
   return __isinf(x);
 }
 
-// CHECK-LABEL: @test___isnanf(
-// CHECK-NEXT:  entry:
-// CHECK-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_isnan_f32(float noundef [[X:%.*]]) #[[ATTR13]]
-// CHECK-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
-// CHECK-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
-// CHECK-NEXT:    ret i32 [[CONV]]
+// DEFAULT-LABEL: @test___isnanf(
+// DEFAULT-NEXT:  entry:
+// DEFAULT-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_isnan_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// DEFAULT-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
+// DEFAULT-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
+// DEFAULT-NEXT:    ret i32 [[CONV]]
+//
+// FINITEONLY-LABEL: @test___isnanf(
+// FINITEONLY-NEXT:  entry:
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_isnan_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
+// FINITEONLY-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
+// FINITEONLY-NEXT:    ret i32 [[CONV]]
 //
 extern "C" __device__ BOOL_TYPE test___isnanf(float x) {
   return __isnanf(x);
 }
 
-// CHECK-LABEL: @test___isnan(
-// CHECK-NEXT:  entry:
-// CHECK-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_isnan_f64(double noundef [[X:%.*]]) #[[ATTR13]]
-// CHECK-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
-// CHECK-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
-// CHECK-NEXT:    ret i32 [[CONV]]
+// DEFAULT-LABEL: @test___isnan(
+// DEFAULT-NEXT:  entry:
+// DEFAULT-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_isnan_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// DEFAULT-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
+// DEFAULT-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
+// DEFAULT-NEXT:    ret i32 [[CONV]]
+//
+// FINITEONLY-LABEL: @test___isnan(
+// FINITEONLY-NEXT:  entry:
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_isnan_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
+// FINITEONLY-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
+// FINITEONLY-NEXT:    ret i32 [[CONV]]
 //
 extern "C" __device__ BOOL_TYPE test___isnan(double x) {
   return __isnan(x);
@@ -1229,7 +1280,7 @@ extern "C" __device__ BOOL_TYPE test___isnan(double x) {
 //
 // FINITEONLY-LABEL: @test_j0f(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_j0_f32(float noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_j0_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_j0f(float x) {
@@ -1243,7 +1294,7 @@ extern "C" __device__ float test_j0f(float x) {
 //
 // FINITEONLY-LABEL: @test_j0(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_j0_f64(double noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_j0_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_j0(double x) {
@@ -1257,7 +1308,7 @@ extern "C" __device__ double test_j0(double x) {
 //
 // FINITEONLY-LABEL: @test_j1f(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_j1_f32(float noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_j1_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_j1f(float x) {
@@ -1271,7 +1322,7 @@ extern "C" __device__ float test_j1f(float x) {
 //
 // FINITEONLY-LABEL: @test_j1(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_j1_f64(double noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_j1_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_j1(double x) {
@@ -1318,14 +1369,14 @@ extern "C" __device__ double test_j1(double x) {
 // FINITEONLY-NEXT:    i32 1, label [[IF_THEN2_I:%.*]]
 // FINITEONLY-NEXT:    ]
 // FINITEONLY:       if.then.i:
-// FINITEONLY-NEXT:    [[CALL_I_I:%.*]] = tail call nnan ninf contract float @__ocml_j0_f32(float noundef [[Y:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_j0_f32(float noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    br label [[_ZL3JNFIF_EXIT:%.*]]
 // FINITEONLY:       if.then2.i:
-// FINITEONLY-NEXT:    [[CALL_I20_I:%.*]] = tail call nnan ninf contract float @__ocml_j1_f32(float noundef [[Y]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I20_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_j1_f32(float noundef nofpclass(nan inf) [[Y]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    br label [[_ZL3JNFIF_EXIT]]
 // FINITEONLY:       if.end4.i:
-// FINITEONLY-NEXT:    [[CALL_I21_I:%.*]] = tail call nnan ninf contract float @__ocml_j0_f32(float noundef [[Y]]) #[[ATTR15]]
-// FINITEONLY-NEXT:    [[CALL_I22_I:%.*]] = tail call nnan ninf contract float @__ocml_j1_f32(float noundef [[Y]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I21_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_j0_f32(float noundef nofpclass(nan inf) [[Y]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I22_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_j1_f32(float noundef nofpclass(nan inf) [[Y]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    [[CMP723_I:%.*]] = icmp sgt i32 [[X]], 1
 // FINITEONLY-NEXT:    br i1 [[CMP723_I]], label [[FOR_BODY_I:%.*]], label [[_ZL3JNFIF_EXIT]]
 // FINITEONLY:       for.body.i:
@@ -1388,14 +1439,14 @@ extern "C" __device__ float test_jnf(int x, float y) {
 // FINITEONLY-NEXT:    i32 1, label [[IF_THEN2_I:%.*]]
 // FINITEONLY-NEXT:    ]
 // FINITEONLY:       if.then.i:
-// FINITEONLY-NEXT:    [[CALL_I_I:%.*]] = tail call nnan ninf contract double @__ocml_j0_f64(double noundef [[Y:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_j0_f64(double noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    br label [[_ZL2JNID_EXIT:%.*]]
 // FINITEONLY:       if.then2.i:
-// FINITEONLY-NEXT:    [[CALL_I20_I:%.*]] = tail call nnan ninf contract double @__ocml_j1_f64(double noundef [[Y]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I20_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_j1_f64(double noundef nofpclass(nan inf) [[Y]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    br label [[_ZL2JNID_EXIT]]
 // FINITEONLY:       if.end4.i:
-// FINITEONLY-NEXT:    [[CALL_I21_I:%.*]] = tail call nnan ninf contract double @__ocml_j0_f64(double noundef [[Y]]) #[[ATTR15]]
-// FINITEONLY-NEXT:    [[CALL_I22_I:%.*]] = tail call nnan ninf contract double @__ocml_j1_f64(double noundef [[Y]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I21_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_j0_f64(double noundef nofpclass(nan inf) [[Y]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I22_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_j1_f64(double noundef nofpclass(nan inf) [[Y]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    [[CMP723_I:%.*]] = icmp sgt i32 [[X]], 1
 // FINITEONLY-NEXT:    br i1 [[CMP723_I]], label [[FOR_BODY_I:%.*]], label [[_ZL2JNID_EXIT]]
 // FINITEONLY:       for.body.i:
@@ -1425,7 +1476,7 @@ extern "C" __device__ double test_jn(int x, double y) {
 //
 // FINITEONLY-LABEL: @test_ldexpf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_ldexp_f32(float noundef [[X:%.*]], i32 noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_ldexp_f32(float noundef nofpclass(nan inf) [[X:%.*]], i32 noundef [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_ldexpf(float x, int y) {
@@ -1439,7 +1490,7 @@ extern "C" __device__ float test_ldexpf(float x, int y) {
 //
 // FINITEONLY-LABEL: @test_ldexp(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_ldexp_f64(double noundef [[X:%.*]], i32 noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_ldexp_f64(double noundef nofpclass(nan inf) [[X:%.*]], i32 noundef [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_ldexp(double x, int y) {
@@ -1453,7 +1504,7 @@ extern "C" __device__ double test_ldexp(double x, int y) {
 //
 // FINITEONLY-LABEL: @test_lgammaf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_lgamma_f32(float noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_lgamma_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_lgammaf(float x) {
@@ -1467,7 +1518,7 @@ extern "C" __device__ float test_lgammaf(float x) {
 //
 // FINITEONLY-LABEL: @test_lgamma(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_lgamma_f64(double noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_lgamma_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_lgamma(double x) {
@@ -1482,7 +1533,7 @@ extern "C" __device__ double test_lgamma(double x) {
 //
 // FINITEONLY-LABEL: @test_llrintf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_rint_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_rint_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    [[CONV_I:%.*]] = fptosi float [[CALL_I]] to i64
 // FINITEONLY-NEXT:    ret i64 [[CONV_I]]
 //
@@ -1498,7 +1549,7 @@ extern "C" __device__ long long int test_llrintf(float x) {
 //
 // FINITEONLY-LABEL: @test_llrint(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_rint_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_rint_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    [[CONV_I:%.*]] = fptosi double [[CALL_I]] to i64
 // FINITEONLY-NEXT:    ret i64 [[CONV_I]]
 //
@@ -1514,7 +1565,7 @@ extern "C" __device__ long long int test_llrint(double x) {
 //
 // FINITEONLY-LABEL: @test_llroundf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_round_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_round_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    [[CONV_I:%.*]] = fptosi float [[CALL_I]] to i64
 // FINITEONLY-NEXT:    ret i64 [[CONV_I]]
 //
@@ -1530,7 +1581,7 @@ extern "C" __device__ long long int test_llroundf(float x) {
 //
 // FINITEONLY-LABEL: @test_llround(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_round_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_round_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    [[CONV_I:%.*]] = fptosi double [[CALL_I]] to i64
 // FINITEONLY-NEXT:    ret i64 [[CONV_I]]
 //
@@ -1545,7 +1596,7 @@ extern "C" __device__ long long int test_llround(double x) {
 //
 // FINITEONLY-LABEL: @test_log10f(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_log10_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_log10_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_log10f(float x) {
@@ -1559,7 +1610,7 @@ extern "C" __device__ float test_log10f(float x) {
 //
 // FINITEONLY-LABEL: @test_log10(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_log10_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_log10_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_log10(double x) {
@@ -1573,7 +1624,7 @@ extern "C" __device__ double test_log10(double x) {
 //
 // FINITEONLY-LABEL: @test_log1pf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_log1p_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_log1p_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_log1pf(float x) {
@@ -1587,7 +1638,7 @@ extern "C" __device__ float test_log1pf(float x) {
 //
 // FINITEONLY-LABEL: @test_log1p(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_log1p_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_log1p_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_log1p(double x) {
@@ -1601,7 +1652,7 @@ extern "C" __device__ double test_log1p(double x) {
 //
 // FINITEONLY-LABEL: @test_log2f(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_log2_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_log2_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_log2f(float x) {
@@ -1615,7 +1666,7 @@ extern "C" __device__ float test_log2f(float x) {
 //
 // FINITEONLY-LABEL: @test_log2(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_log2_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_log2_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_log2(double x) {
@@ -1629,7 +1680,7 @@ extern "C" __device__ double test_log2(double x) {
 //
 // FINITEONLY-LABEL: @test_logbf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_logb_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_logb_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_logbf(float x) {
@@ -1643,7 +1694,7 @@ extern "C" __device__ float test_logbf(float x) {
 //
 // FINITEONLY-LABEL: @test_logb(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_logb_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_logb_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_logb(double x) {
@@ -1658,7 +1709,7 @@ extern "C" __device__ double test_logb(double x) {
 //
 // FINITEONLY-LABEL: @test_lrintf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_rint_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_rint_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    [[CONV_I:%.*]] = fptosi float [[CALL_I]] to i64
 // FINITEONLY-NEXT:    ret i64 [[CONV_I]]
 //
@@ -1674,7 +1725,7 @@ extern "C" __device__ long int test_lrintf(float x) {
 //
 // FINITEONLY-LABEL: @test_lrint(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_rint_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_rint_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    [[CONV_I:%.*]] = fptosi double [[CALL_I]] to i64
 // FINITEONLY-NEXT:    ret i64 [[CONV_I]]
 //
@@ -1690,7 +1741,7 @@ extern "C" __device__ long int test_lrint(double x) {
 //
 // FINITEONLY-LABEL: @test_lroundf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_round_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_round_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    [[CONV_I:%.*]] = fptosi float [[CALL_I]] to i64
 // FINITEONLY-NEXT:    ret i64 [[CONV_I]]
 //
@@ -1706,7 +1757,7 @@ extern "C" __device__ long int test_lroundf(float x) {
 //
 // FINITEONLY-LABEL: @test_lround(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_round_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_round_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    [[CONV_I:%.*]] = fptosi double [[CALL_I]] to i64
 // FINITEONLY-NEXT:    ret i64 [[CONV_I]]
 //
@@ -1728,7 +1779,7 @@ extern "C" __device__ long int test_lround(double x) {
 // FINITEONLY-NEXT:  entry:
 // FINITEONLY-NEXT:    [[__TMP_I:%.*]] = alloca float, align 4, addrspace(5)
 // FINITEONLY-NEXT:    call void @llvm.lifetime.start.p5(i64 4, ptr addrspace(5) [[__TMP_I]]) #[[ATTR16]]
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract float @__ocml_modf_f32(float noundef [[X:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract nofpclass(nan inf) float @__ocml_modf_f32(float noundef nofpclass(nan inf) [[X:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    [[TMP0:%.*]] = load float, ptr addrspace(5) [[__TMP_I]], align 4, !tbaa [[TBAA15:![0-9]+]]
 // FINITEONLY-NEXT:    store float [[TMP0]], ptr [[Y:%.*]], align 4, !tbaa [[TBAA15]]
 // FINITEONLY-NEXT:    call void @llvm.lifetime.end.p5(i64 4, ptr addrspace(5) [[__TMP_I]]) #[[ATTR16]]
@@ -1752,7 +1803,7 @@ extern "C" __device__ float test_modff(float x, float* y) {
 // FINITEONLY-NEXT:  entry:
 // FINITEONLY-NEXT:    [[__TMP_I:%.*]] = alloca double, align 8, addrspace(5)
 // FINITEONLY-NEXT:    call void @llvm.lifetime.start.p5(i64 8, ptr addrspace(5) [[__TMP_I]]) #[[ATTR16]]
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract double @__ocml_modf_f64(double noundef [[X:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract nofpclass(nan inf) double @__ocml_modf_f64(double noundef nofpclass(nan inf) [[X:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    [[TMP0:%.*]] = load double, ptr addrspace(5) [[__TMP_I]], align 8, !tbaa [[TBAA17:![0-9]+]]
 // FINITEONLY-NEXT:    store double [[TMP0]], ptr [[Y:%.*]], align 8, !tbaa [[TBAA17]]
 // FINITEONLY-NEXT:    call void @llvm.lifetime.end.p5(i64 8, ptr addrspace(5) [[__TMP_I]]) #[[ATTR16]]
@@ -1998,7 +2049,7 @@ extern "C" __device__ double test_nan_fill() {
 //
 // FINITEONLY-LABEL: @test_nearbyintf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_nearbyint_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_nearbyint_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_nearbyintf(float x) {
@@ -2012,7 +2063,7 @@ extern "C" __device__ float test_nearbyintf(float x) {
 //
 // FINITEONLY-LABEL: @test_nearbyint(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_nearbyint_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_nearbyint_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_nearbyint(double x) {
@@ -2026,7 +2077,7 @@ extern "C" __device__ double test_nearbyint(double x) {
 //
 // FINITEONLY-LABEL: @test_nextafterf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_nextafter_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_nextafter_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_nextafterf(float x, float y) {
@@ -2040,7 +2091,7 @@ extern "C" __device__ float test_nextafterf(float x, float y) {
 //
 // FINITEONLY-LABEL: @test_nextafter(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_nextafter_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_nextafter_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_nextafter(double x, double y) {
@@ -2054,7 +2105,7 @@ extern "C" __device__ double test_nextafter(double x, double y) {
 //
 // FINITEONLY-LABEL: @test_norm3df(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_len3_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]], float noundef [[Z:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_len3_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]], float noundef nofpclass(nan inf) [[Z:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_norm3df(float x, float y, float z) {
@@ -2068,7 +2119,7 @@ extern "C" __device__ float test_norm3df(float x, float y, float z) {
 //
 // FINITEONLY-LABEL: @test_norm3d(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_len3_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]], double noundef [[Z:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_len3_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]], double noundef nofpclass(nan inf) [[Z:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_norm3d(double x, double y, double z) {
@@ -2082,7 +2133,7 @@ extern "C" __device__ double test_norm3d(double x, double y, double z) {
 //
 // FINITEONLY-LABEL: @test_norm4df(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_len4_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]], float noundef [[Z:%.*]], float noundef [[W:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_len4_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]], float noundef nofpclass(nan inf) [[Z:%.*]], float noundef nofpclass(nan inf) [[W:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_norm4df(float x, float y, float z, float w) {
@@ -2096,7 +2147,7 @@ extern "C" __device__ float test_norm4df(float x, float y, float z, float w) {
 //
 // FINITEONLY-LABEL: @test_norm4d(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_len4_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]], double noundef [[Z:%.*]], double noundef [[W:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_len4_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]], double noundef nofpclass(nan inf) [[Z:%.*]], double noundef nofpclass(nan inf) [[W:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_norm4d(double x, double y, double z, double w) {
@@ -2110,7 +2161,7 @@ extern "C" __device__ double test_norm4d(double x, double y, double z, double w)
 //
 // FINITEONLY-LABEL: @test_normcdff(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_ncdf_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_ncdf_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_normcdff(float x) {
@@ -2124,7 +2175,7 @@ extern "C" __device__ float test_normcdff(float x) {
 //
 // FINITEONLY-LABEL: @test_normcdf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_ncdf_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_ncdf_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_normcdf(double x) {
@@ -2138,7 +2189,7 @@ extern "C" __device__ double test_normcdf(double x) {
 //
 // FINITEONLY-LABEL: @test_normcdfinvf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_ncdfinv_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_ncdfinv_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_normcdfinvf(float x) {
@@ -2152,7 +2203,7 @@ extern "C" __device__ float test_normcdfinvf(float x) {
 //
 // FINITEONLY-LABEL: @test_normcdfinv(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_ncdfinv_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_ncdfinv_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_normcdfinv(double x) {
@@ -2196,7 +2247,7 @@ extern "C" __device__ double test_normcdfinv(double x) {
 // FINITEONLY-NEXT:    br i1 [[TOBOOL_NOT_I]], label [[_ZL5NORMFIPKF_EXIT]], label [[WHILE_BODY_I]], !llvm.loop [[LOOP19:![0-9]+]]
 // FINITEONLY:       _ZL5normfiPKf.exit:
 // FINITEONLY-NEXT:    [[__R_0_LCSSA_I:%.*]] = phi float [ 0.000000e+00, [[ENTRY]] ], [ [[ADD_I]], [[WHILE_BODY_I]] ]
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_sqrt_f32(float noundef [[__R_0_LCSSA_I]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_sqrt_f32(float noundef nofpclass(nan inf) [[__R_0_LCSSA_I]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_normf(int x, const float *y) {
@@ -2240,7 +2291,7 @@ extern "C" __device__ float test_normf(int x, const float *y) {
 // FINITEONLY-NEXT:    br i1 [[TOBOOL_NOT_I]], label [[_ZL4NORMIPKD_EXIT]], label [[WHILE_BODY_I]], !llvm.loop [[LOOP20:![0-9]+]]
 // FINITEONLY:       _ZL4normiPKd.exit:
 // FINITEONLY-NEXT:    [[__R_0_LCSSA_I:%.*]] = phi double [ 0.000000e+00, [[ENTRY]] ], [ [[ADD_I]], [[WHILE_BODY_I]] ]
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_sqrt_f64(double noundef [[__R_0_LCSSA_I]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_sqrt_f64(double noundef nofpclass(nan inf) [[__R_0_LCSSA_I]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_norm(int x, const double *y) {
@@ -2254,7 +2305,7 @@ extern "C" __device__ double test_norm(int x, const double *y) {
 //
 // FINITEONLY-LABEL: @test_powf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_pow_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_pow_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_powf(float x, float y) {
@@ -2268,7 +2319,7 @@ extern "C" __device__ float test_powf(float x, float y) {
 //
 // FINITEONLY-LABEL: @test_pow(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_pow_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_pow_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_pow(double x, double y) {
@@ -2282,7 +2333,7 @@ extern "C" __device__ double test_pow(double x, double y) {
 //
 // FINITEONLY-LABEL: @test_powif(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_pown_f32(float noundef [[X:%.*]], i32 noundef [[Y:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_pown_f32(float noundef nofpclass(nan inf) [[X:%.*]], i32 noundef [[Y:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_powif(float x, int y) {
@@ -2296,7 +2347,7 @@ extern "C" __device__ float test_powif(float x, int y) {
 //
 // FINITEONLY-LABEL: @test_powi(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_pown_f64(double noundef [[X:%.*]], i32 noundef [[Y:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_pown_f64(double noundef nofpclass(nan inf) [[X:%.*]], i32 noundef [[Y:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_powi(double x, int y) {
@@ -2310,7 +2361,7 @@ extern "C" __device__ double test_powi(double x, int y) {
 //
 // FINITEONLY-LABEL: @test_rcbrtf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_rcbrt_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_rcbrt_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_rcbrtf(float x) {
@@ -2324,7 +2375,7 @@ extern "C" __device__ float test_rcbrtf(float x) {
 //
 // FINITEONLY-LABEL: @test_rcbrt(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_rcbrt_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_rcbrt_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_rcbrt(double x) {
@@ -2338,7 +2389,7 @@ extern "C" __device__ double test_rcbrt(double x) {
 //
 // FINITEONLY-LABEL: @test_remainderf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_remainder_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_remainder_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_remainderf(float x, float y) {
@@ -2352,7 +2403,7 @@ extern "C" __device__ float test_remainderf(float x, float y) {
 //
 // FINITEONLY-LABEL: @test_remainder(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_remainder_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_remainder_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_remainder(double x, double y) {
@@ -2373,7 +2424,7 @@ extern "C" __device__ double test_remainder(double x, double y) {
 // FINITEONLY-NEXT:  entry:
 // FINITEONLY-NEXT:    [[__TMP_I:%.*]] = alloca i32, align 4, addrspace(5)
 // FINITEONLY-NEXT:    call void @llvm.lifetime.start.p5(i64 4, ptr addrspace(5) [[__TMP_I]]) #[[ATTR16]]
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract float @__ocml_remquo_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract nofpclass(nan inf) float @__ocml_remquo_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    [[TMP0:%.*]] = load i32, ptr addrspace(5) [[__TMP_I]], align 4, !tbaa [[TBAA11]]
 // FINITEONLY-NEXT:    store i32 [[TMP0]], ptr [[Z:%.*]], align 4, !tbaa [[TBAA11]]
 // FINITEONLY-NEXT:    call void @llvm.lifetime.end.p5(i64 4, ptr addrspace(5) [[__TMP_I]]) #[[ATTR16]]
@@ -2397,7 +2448,7 @@ extern "C" __device__ float test_remquof(float x, float y, int* z) {
 // FINITEONLY-NEXT:  entry:
 // FINITEONLY-NEXT:    [[__TMP_I:%.*]] = alloca i32, align 4, addrspace(5)
 // FINITEONLY-NEXT:    call void @llvm.lifetime.start.p5(i64 4, ptr addrspace(5) [[__TMP_I]]) #[[ATTR16]]
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract double @__ocml_remquo_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract nofpclass(nan inf) double @__ocml_remquo_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    [[TMP0:%.*]] = load i32, ptr addrspace(5) [[__TMP_I]], align 4, !tbaa [[TBAA11]]
 // FINITEONLY-NEXT:    store i32 [[TMP0]], ptr [[Z:%.*]], align 4, !tbaa [[TBAA11]]
 // FINITEONLY-NEXT:    call void @llvm.lifetime.end.p5(i64 4, ptr addrspace(5) [[__TMP_I]]) #[[ATTR16]]
@@ -2414,7 +2465,7 @@ extern "C" __device__ double test_remquo(double x, double y, int* z) {
 //
 // FINITEONLY-LABEL: @test_rhypotf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_rhypot_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_rhypot_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_rhypotf(float x, float y) {
@@ -2428,7 +2479,7 @@ extern "C" __device__ float test_rhypotf(float x, float y) {
 //
 // FINITEONLY-LABEL: @test_rhypot(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_rhypot_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_rhypot_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_rhypot(double x, double y) {
@@ -2442,7 +2493,7 @@ extern "C" __device__ double test_rhypot(double x, double y) {
 //
 // FINITEONLY-LABEL: @test_rintf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_rint_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_rint_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_rintf(float x) {
@@ -2456,7 +2507,7 @@ extern "C" __device__ float test_rintf(float x) {
 //
 // FINITEONLY-LABEL: @test_rint(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_rint_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_rint_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_rint(double x) {
@@ -2500,7 +2551,7 @@ extern "C" __device__ double test_rint(double x) {
 // FINITEONLY-NEXT:    br i1 [[TOBOOL_NOT_I]], label [[_ZL6RNORMFIPKF_EXIT]], label [[WHILE_BODY_I]], !llvm.loop [[LOOP21:![0-9]+]]
 // FINITEONLY:       _ZL6rnormfiPKf.exit:
 // FINITEONLY-NEXT:    [[__R_0_LCSSA_I:%.*]] = phi float [ 0.000000e+00, [[ENTRY]] ], [ [[ADD_I]], [[WHILE_BODY_I]] ]
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_rsqrt_f32(float noundef [[__R_0_LCSSA_I]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_rsqrt_f32(float noundef nofpclass(nan inf) [[__R_0_LCSSA_I]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_rnormf(int x, const float* y) {
@@ -2544,7 +2595,7 @@ extern "C" __device__ float test_rnormf(int x, const float* y) {
 // FINITEONLY-NEXT:    br i1 [[TOBOOL_NOT_I]], label [[_ZL5RNORMIPKD_EXIT]], label [[WHILE_BODY_I]], !llvm.loop [[LOOP22:![0-9]+]]
 // FINITEONLY:       _ZL5rnormiPKd.exit:
 // FINITEONLY-NEXT:    [[__R_0_LCSSA_I:%.*]] = phi double [ 0.000000e+00, [[ENTRY]] ], [ [[ADD_I]], [[WHILE_BODY_I]] ]
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_rsqrt_f64(double noundef [[__R_0_LCSSA_I]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_rsqrt_f64(double noundef nofpclass(nan inf) [[__R_0_LCSSA_I]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_rnorm(int x, const double* y) {
@@ -2558,7 +2609,7 @@ extern "C" __device__ double test_rnorm(int x, const double* y) {
 //
 // FINITEONLY-LABEL: @test_rnorm3df(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_rlen3_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]], float noundef [[Z:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_rlen3_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]], float noundef nofpclass(nan inf) [[Z:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_rnorm3df(float x, float y, float z) {
@@ -2572,7 +2623,7 @@ extern "C" __device__ float test_rnorm3df(float x, float y, float z) {
 //
 // FINITEONLY-LABEL: @test_rnorm3d(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_rlen3_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]], double noundef [[Z:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_rlen3_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]], double noundef nofpclass(nan inf) [[Z:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_rnorm3d(double x, double y, double z) {
@@ -2586,7 +2637,7 @@ extern "C" __device__ double test_rnorm3d(double x, double y, double z) {
 //
 // FINITEONLY-LABEL: @test_rnorm4df(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_rlen4_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]], float noundef [[Z:%.*]], float noundef [[W:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_rlen4_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]], float noundef nofpclass(nan inf) [[Z:%.*]], float noundef nofpclass(nan inf) [[W:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_rnorm4df(float x, float y, float z, float w) {
@@ -2600,7 +2651,7 @@ extern "C" __device__ float test_rnorm4df(float x, float y, float z, float w) {
 //
 // FINITEONLY-LABEL: @test_rnorm4d(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_rlen4_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]], double noundef [[Z:%.*]], double noundef [[W:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_rlen4_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]], double noundef nofpclass(nan inf) [[Z:%.*]], double noundef nofpclass(nan inf) [[W:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_rnorm4d(double x, double y, double z, double w) {
@@ -2614,7 +2665,7 @@ extern "C" __device__ double test_rnorm4d(double x, double y, double z, double w
 //
 // FINITEONLY-LABEL: @test_roundf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_round_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_round_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_roundf(float x) {
@@ -2628,7 +2679,7 @@ extern "C" __device__ float test_roundf(float x) {
 //
 // FINITEONLY-LABEL: @test_round(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_round_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_round_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_round(double x) {
@@ -2642,7 +2693,7 @@ extern "C" __device__ double test_round(double x) {
 //
 // FINITEONLY-LABEL: @test_rsqrtf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_rsqrt_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_rsqrt_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_rsqrtf(float x) {
@@ -2656,7 +2707,7 @@ extern "C" __device__ float test_rsqrtf(float x) {
 //
 // FINITEONLY-LABEL: @test_rsqrt(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_rsqrt_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_rsqrt_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_rsqrt(double x) {
@@ -2684,10 +2735,10 @@ extern "C" __device__ double test_rsqrt(double x) {
 // FINITEONLY-NEXT:    br i1 [[CMP_NOT_I]], label [[COND_FALSE_I:%.*]], label [[COND_TRUE_I:%.*]]
 // FINITEONLY:       cond.true.i:
 // FINITEONLY-NEXT:    [[CONV_I:%.*]] = trunc i64 [[Y]] to i32
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_scalbn_f32(float noundef [[X:%.*]], i32 noundef [[CONV_I]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_scalbn_f32(float noundef nofpclass(nan inf) [[X:%.*]], i32 noundef [[CONV_I]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    br label [[_ZL8SCALBLNFFL_EXIT:%.*]]
 // FINITEONLY:       cond.false.i:
-// FINITEONLY-NEXT:    [[CALL2_I:%.*]] = tail call nnan ninf contract float @__ocml_scalb_f32(float noundef [[X]], float noundef 0x43E0000000000000) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL2_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_scalb_f32(float noundef nofpclass(nan inf) [[X]], float noundef nofpclass(nan inf) 0x43E0000000000000) #[[ATTR13]]
 // FINITEONLY-NEXT:    br label [[_ZL8SCALBLNFFL_EXIT]]
 // FINITEONLY:       _ZL8scalblnffl.exit:
 // FINITEONLY-NEXT:    [[COND_I:%.*]] = phi nnan ninf contract float [ [[CALL_I]], [[COND_TRUE_I]] ], [ [[CALL2_I]], [[COND_FALSE_I]] ]
@@ -2718,10 +2769,10 @@ extern "C" __device__ float test_scalblnf(float x, long int y) {
 // FINITEONLY-NEXT:    br i1 [[CMP_NOT_I]], label [[COND_FALSE_I:%.*]], label [[COND_TRUE_I:%.*]]
 // FINITEONLY:       cond.true.i:
 // FINITEONLY-NEXT:    [[CONV_I:%.*]] = trunc i64 [[Y]] to i32
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_scalbn_f64(double noundef [[X:%.*]], i32 noundef [[CONV_I]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_scalbn_f64(double noundef nofpclass(nan inf) [[X:%.*]], i32 noundef [[CONV_I]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    br label [[_ZL7SCALBLNDL_EXIT:%.*]]
 // FINITEONLY:       cond.false.i:
-// FINITEONLY-NEXT:    [[CALL2_I:%.*]] = tail call nnan ninf contract double @__ocml_scalb_f64(double noundef [[X]], double noundef 0x43E0000000000000) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL2_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_scalb_f64(double noundef nofpclass(nan inf) [[X]], double noundef nofpclass(nan inf) 0x43E0000000000000) #[[ATTR13]]
 // FINITEONLY-NEXT:    br label [[_ZL7SCALBLNDL_EXIT]]
 // FINITEONLY:       _ZL7scalblndl.exit:
 // FINITEONLY-NEXT:    [[COND_I:%.*]] = phi nnan ninf contract double [ [[CALL_I]], [[COND_TRUE_I]] ], [ [[CALL2_I]], [[COND_FALSE_I]] ]
@@ -2738,7 +2789,7 @@ extern "C" __device__ double test_scalbln(double x, long int y) {
 //
 // FINITEONLY-LABEL: @test_scalbnf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_scalbn_f32(float noundef [[X:%.*]], i32 noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_scalbn_f32(float noundef nofpclass(nan inf) [[X:%.*]], i32 noundef [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_scalbnf(float x, int y) {
@@ -2752,30 +2803,44 @@ extern "C" __device__ float test_scalbnf(float x, int y) {
 //
 // FINITEONLY-LABEL: @test_scalbn(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_scalbn_f64(double noundef [[X:%.*]], i32 noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_scalbn_f64(double noundef nofpclass(nan inf) [[X:%.*]], i32 noundef [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_scalbn(double x, int y) {
   return scalbn(x, y);
 }
 
-// CHECK-LABEL: @test___signbitf(
-// CHECK-NEXT:  entry:
-// CHECK-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_signbit_f32(float noundef [[X:%.*]]) #[[ATTR13]]
-// CHECK-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
-// CHECK-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
-// CHECK-NEXT:    ret i32 [[CONV]]
+// DEFAULT-LABEL: @test___signbitf(
+// DEFAULT-NEXT:  entry:
+// DEFAULT-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_signbit_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// DEFAULT-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
+// DEFAULT-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
+// DEFAULT-NEXT:    ret i32 [[CONV]]
+//
+// FINITEONLY-LABEL: @test___signbitf(
+// FINITEONLY-NEXT:  entry:
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_signbit_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
+// FINITEONLY-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
+// FINITEONLY-NEXT:    ret i32 [[CONV]]
 //
 extern "C" __device__ BOOL_TYPE test___signbitf(float x) {
   return __signbitf(x);
 }
 
-// CHECK-LABEL: @test___signbit(
-// CHECK-NEXT:  entry:
-// CHECK-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_signbit_f64(double noundef [[X:%.*]]) #[[ATTR13]]
-// CHECK-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
-// CHECK-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
-// CHECK-NEXT:    ret i32 [[CONV]]
+// DEFAULT-LABEL: @test___signbit(
+// DEFAULT-NEXT:  entry:
+// DEFAULT-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_signbit_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// DEFAULT-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
+// DEFAULT-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
+// DEFAULT-NEXT:    ret i32 [[CONV]]
+//
+// FINITEONLY-LABEL: @test___signbit(
+// FINITEONLY-NEXT:  entry:
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call i32 @__ocml_signbit_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[TOBOOL_I:%.*]] = icmp ne i32 [[CALL_I]], 0
+// FINITEONLY-NEXT:    [[CONV:%.*]] = zext i1 [[TOBOOL_I]] to i32
+// FINITEONLY-NEXT:    ret i32 [[CONV]]
 //
 extern "C" __device__ BOOL_TYPE test___signbit(double x) {
   return __signbit(x);
@@ -2796,7 +2861,7 @@ extern "C" __device__ BOOL_TYPE test___signbit(double x) {
 // FINITEONLY-NEXT:  entry:
 // FINITEONLY-NEXT:    [[__TMP_I:%.*]] = alloca float, align 4, addrspace(5)
 // FINITEONLY-NEXT:    call void @llvm.lifetime.start.p5(i64 4, ptr addrspace(5) [[__TMP_I]]) #[[ATTR16]]
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract float @__ocml_sincos_f32(float noundef [[X:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract nofpclass(nan inf) float @__ocml_sincos_f32(float noundef nofpclass(nan inf) [[X:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    store float [[CALL_I]], ptr [[Y:%.*]], align 4, !tbaa [[TBAA15]]
 // FINITEONLY-NEXT:    [[TMP0:%.*]] = load float, ptr addrspace(5) [[__TMP_I]], align 4, !tbaa [[TBAA15]]
 // FINITEONLY-NEXT:    store float [[TMP0]], ptr [[Z:%.*]], align 4, !tbaa [[TBAA15]]
@@ -2822,7 +2887,7 @@ extern "C" __device__ void test_sincosf(float x, float *y, float *z) {
 // FINITEONLY-NEXT:  entry:
 // FINITEONLY-NEXT:    [[__TMP_I:%.*]] = alloca double, align 8, addrspace(5)
 // FINITEONLY-NEXT:    call void @llvm.lifetime.start.p5(i64 8, ptr addrspace(5) [[__TMP_I]]) #[[ATTR16]]
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract double @__ocml_sincos_f64(double noundef [[X:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract nofpclass(nan inf) double @__ocml_sincos_f64(double noundef nofpclass(nan inf) [[X:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    store double [[CALL_I]], ptr [[Y:%.*]], align 8, !tbaa [[TBAA17]]
 // FINITEONLY-NEXT:    [[TMP0:%.*]] = load double, ptr addrspace(5) [[__TMP_I]], align 8, !tbaa [[TBAA17]]
 // FINITEONLY-NEXT:    store double [[TMP0]], ptr [[Z:%.*]], align 8, !tbaa [[TBAA17]]
@@ -2848,7 +2913,7 @@ extern "C" __device__ void test_sincos(double x, double *y, double *z) {
 // FINITEONLY-NEXT:  entry:
 // FINITEONLY-NEXT:    [[__TMP_I:%.*]] = alloca float, align 4, addrspace(5)
 // FINITEONLY-NEXT:    call void @llvm.lifetime.start.p5(i64 4, ptr addrspace(5) [[__TMP_I]]) #[[ATTR16]]
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract float @__ocml_sincospi_f32(float noundef [[X:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract nofpclass(nan inf) float @__ocml_sincospi_f32(float noundef nofpclass(nan inf) [[X:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    store float [[CALL_I]], ptr [[Y:%.*]], align 4, !tbaa [[TBAA15]]
 // FINITEONLY-NEXT:    [[TMP0:%.*]] = load float, ptr addrspace(5) [[__TMP_I]], align 4, !tbaa [[TBAA15]]
 // FINITEONLY-NEXT:    store float [[TMP0]], ptr [[Z:%.*]], align 4, !tbaa [[TBAA15]]
@@ -2874,7 +2939,7 @@ extern "C" __device__ void test_sincospif(float x, float *y, float *z) {
 // FINITEONLY-NEXT:  entry:
 // FINITEONLY-NEXT:    [[__TMP_I:%.*]] = alloca double, align 8, addrspace(5)
 // FINITEONLY-NEXT:    call void @llvm.lifetime.start.p5(i64 8, ptr addrspace(5) [[__TMP_I]]) #[[ATTR16]]
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract double @__ocml_sincospi_f64(double noundef [[X:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = call nnan ninf contract nofpclass(nan inf) double @__ocml_sincospi_f64(double noundef nofpclass(nan inf) [[X:%.*]], ptr addrspace(5) noundef [[__TMP_I]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    store double [[CALL_I]], ptr [[Y:%.*]], align 8, !tbaa [[TBAA17]]
 // FINITEONLY-NEXT:    [[TMP0:%.*]] = load double, ptr addrspace(5) [[__TMP_I]], align 8, !tbaa [[TBAA17]]
 // FINITEONLY-NEXT:    store double [[TMP0]], ptr [[Z:%.*]], align 8, !tbaa [[TBAA17]]
@@ -2892,7 +2957,7 @@ extern "C" __device__ void test_sincospi(double x, double *y, double *z) {
 //
 // FINITEONLY-LABEL: @test_sinf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_sin_f32(float noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_sin_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_sinf(float x) {
@@ -2906,7 +2971,7 @@ extern "C" __device__ float test_sinf(float x) {
 //
 // FINITEONLY-LABEL: @test_sin(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_sin_f64(double noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_sin_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_sin(double x) {
@@ -2920,7 +2985,7 @@ extern "C" __device__ double test_sin(double x) {
 //
 // FINITEONLY-LABEL: @test_sinpif(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_sinpi_f32(float noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_sinpi_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_sinpif(float x) {
@@ -2934,7 +2999,7 @@ extern "C" __device__ float test_sinpif(float x) {
 //
 // FINITEONLY-LABEL: @test_sinpi(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_sinpi_f64(double noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_sinpi_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_sinpi(double x) {
@@ -2948,7 +3013,7 @@ extern "C" __device__ double test_sinpi(double x) {
 //
 // FINITEONLY-LABEL: @test_sqrtf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_sqrt_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_sqrt_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_sqrtf(float x) {
@@ -2962,7 +3027,7 @@ extern "C" __device__ float test_sqrtf(float x) {
 //
 // FINITEONLY-LABEL: @test_sqrt(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_sqrt_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_sqrt_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_sqrt(double x) {
@@ -2976,7 +3041,7 @@ extern "C" __device__ double test_sqrt(double x) {
 //
 // FINITEONLY-LABEL: @test_tanf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_tan_f32(float noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_tan_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_tanf(float x) {
@@ -2990,7 +3055,7 @@ extern "C" __device__ float test_tanf(float x) {
 //
 // FINITEONLY-LABEL: @test_tan(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_tan_f64(double noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_tan_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_tan(double x) {
@@ -3004,7 +3069,7 @@ extern "C" __device__ double test_tan(double x) {
 //
 // FINITEONLY-LABEL: @test_tanhf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_tanh_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_tanh_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_tanhf(float x) {
@@ -3018,7 +3083,7 @@ extern "C" __device__ float test_tanhf(float x) {
 //
 // FINITEONLY-LABEL: @test_tanh(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_tanh_f64(double noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_tanh_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_tanh(double x) {
@@ -3032,7 +3097,7 @@ extern "C" __device__ double test_tanh(double x) {
 //
 // FINITEONLY-LABEL: @test_tgammaf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_tgamma_f32(float noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_tgamma_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_tgammaf(float x) {
@@ -3046,7 +3111,7 @@ extern "C" __device__ float test_tgammaf(float x) {
 //
 // FINITEONLY-LABEL: @test_tgamma(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_tgamma_f64(double noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_tgamma_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_tgamma(double x) {
@@ -3060,7 +3125,7 @@ extern "C" __device__ double test_tgamma(double x) {
 //
 // FINITEONLY-LABEL: @test_truncf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_trunc_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_trunc_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_truncf(float x) {
@@ -3074,7 +3139,7 @@ extern "C" __device__ float test_truncf(float x) {
 //
 // FINITEONLY-LABEL: @test_trunc(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_trunc_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_trunc_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_trunc(double x) {
@@ -3088,7 +3153,7 @@ extern "C" __device__ double test_trunc(double x) {
 //
 // FINITEONLY-LABEL: @test_y0f(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_y0_f32(float noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_y0_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_y0f(float x) {
@@ -3102,7 +3167,7 @@ extern "C" __device__ float test_y0f(float x) {
 //
 // FINITEONLY-LABEL: @test_y0(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_y0_f64(double noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_y0_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_y0(double x) {
@@ -3116,7 +3181,7 @@ extern "C" __device__ double test_y0(double x) {
 //
 // FINITEONLY-LABEL: @test_y1f(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_y1_f32(float noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_y1_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test_y1f(float x) {
@@ -3130,7 +3195,7 @@ extern "C" __device__ float test_y1f(float x) {
 //
 // FINITEONLY-LABEL: @test_y1(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_y1_f64(double noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_y1_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test_y1(double x) {
@@ -3177,14 +3242,14 @@ extern "C" __device__ double test_y1(double x) {
 // FINITEONLY-NEXT:    i32 1, label [[IF_THEN2_I:%.*]]
 // FINITEONLY-NEXT:    ]
 // FINITEONLY:       if.then.i:
-// FINITEONLY-NEXT:    [[CALL_I_I:%.*]] = tail call nnan ninf contract float @__ocml_y0_f32(float noundef [[Y:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_y0_f32(float noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    br label [[_ZL3YNFIF_EXIT:%.*]]
 // FINITEONLY:       if.then2.i:
-// FINITEONLY-NEXT:    [[CALL_I20_I:%.*]] = tail call nnan ninf contract float @__ocml_y1_f32(float noundef [[Y]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I20_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_y1_f32(float noundef nofpclass(nan inf) [[Y]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    br label [[_ZL3YNFIF_EXIT]]
 // FINITEONLY:       if.end4.i:
-// FINITEONLY-NEXT:    [[CALL_I21_I:%.*]] = tail call nnan ninf contract float @__ocml_y0_f32(float noundef [[Y]]) #[[ATTR15]]
-// FINITEONLY-NEXT:    [[CALL_I22_I:%.*]] = tail call nnan ninf contract float @__ocml_y1_f32(float noundef [[Y]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I21_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_y0_f32(float noundef nofpclass(nan inf) [[Y]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I22_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_y1_f32(float noundef nofpclass(nan inf) [[Y]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    [[CMP723_I:%.*]] = icmp sgt i32 [[X]], 1
 // FINITEONLY-NEXT:    br i1 [[CMP723_I]], label [[FOR_BODY_I:%.*]], label [[_ZL3YNFIF_EXIT]]
 // FINITEONLY:       for.body.i:
@@ -3247,14 +3312,14 @@ extern "C" __device__ float test_ynf(int x, float y) {
 // FINITEONLY-NEXT:    i32 1, label [[IF_THEN2_I:%.*]]
 // FINITEONLY-NEXT:    ]
 // FINITEONLY:       if.then.i:
-// FINITEONLY-NEXT:    [[CALL_I_I:%.*]] = tail call nnan ninf contract double @__ocml_y0_f64(double noundef [[Y:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_y0_f64(double noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    br label [[_ZL2YNID_EXIT:%.*]]
 // FINITEONLY:       if.then2.i:
-// FINITEONLY-NEXT:    [[CALL_I20_I:%.*]] = tail call nnan ninf contract double @__ocml_y1_f64(double noundef [[Y]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I20_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_y1_f64(double noundef nofpclass(nan inf) [[Y]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    br label [[_ZL2YNID_EXIT]]
 // FINITEONLY:       if.end4.i:
-// FINITEONLY-NEXT:    [[CALL_I21_I:%.*]] = tail call nnan ninf contract double @__ocml_y0_f64(double noundef [[Y]]) #[[ATTR15]]
-// FINITEONLY-NEXT:    [[CALL_I22_I:%.*]] = tail call nnan ninf contract double @__ocml_y1_f64(double noundef [[Y]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I21_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_y0_f64(double noundef nofpclass(nan inf) [[Y]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I22_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_y1_f64(double noundef nofpclass(nan inf) [[Y]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    [[CMP723_I:%.*]] = icmp sgt i32 [[X]], 1
 // FINITEONLY-NEXT:    br i1 [[CMP723_I]], label [[FOR_BODY_I:%.*]], label [[_ZL2YNID_EXIT]]
 // FINITEONLY:       for.body.i:
@@ -3284,7 +3349,7 @@ extern "C" __device__ double test_yn(int x, double y) {
 //
 // FINITEONLY-LABEL: @test___cosf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_native_cos_f32(float noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_native_cos_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test___cosf(float x) {
@@ -3298,7 +3363,7 @@ extern "C" __device__ float test___cosf(float x) {
 //
 // FINITEONLY-LABEL: @test___exp10f(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_native_exp10_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_native_exp10_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test___exp10f(float x) {
@@ -3312,7 +3377,7 @@ extern "C" __device__ float test___exp10f(float x) {
 //
 // FINITEONLY-LABEL: @test___expf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_native_exp_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_native_exp_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test___expf(float x) {
@@ -3354,7 +3419,7 @@ extern "C" __device__ float test___fdividef(float x, float y) {
 //
 // FINITEONLY-LABEL: @test__fmaf_rn(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_fma_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]], float noundef [[Z:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_fma_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]], float noundef nofpclass(nan inf) [[Z:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test__fmaf_rn(float x, float y, float z) {
@@ -3410,7 +3475,7 @@ extern "C" __device__ float test___frsqrt_rn(float x) {
 //
 // FINITEONLY-LABEL: @test___fsqrt_rn(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_native_sqrt_f32(float noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_native_sqrt_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test___fsqrt_rn(float x) {
@@ -3438,7 +3503,7 @@ extern "C" __device__ float test___fsub_rn(float x, float y) {
 //
 // FINITEONLY-LABEL: @test___log10f(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_native_log10_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_native_log10_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test___log10f(float x) {
@@ -3452,7 +3517,7 @@ extern "C" __device__ float test___log10f(float x) {
 //
 // FINITEONLY-LABEL: @test___log2f(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_native_log2_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_native_log2_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test___log2f(float x) {
@@ -3466,7 +3531,7 @@ extern "C" __device__ float test___log2f(float x) {
 //
 // FINITEONLY-LABEL: @test___logf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_native_log_f32(float noundef [[X:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_native_log_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test___logf(float x) {
@@ -3480,7 +3545,7 @@ extern "C" __device__ float test___logf(float x) {
 //
 // FINITEONLY-LABEL: @test___powf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_pow_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]]) #[[ATTR14]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_pow_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR14]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test___powf(float x, float y) {
@@ -3517,9 +3582,9 @@ extern "C" __device__ float test___saturatef(float x) {
 //
 // FINITEONLY-LABEL: @test___sincosf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_native_sin_f32(float noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_native_sin_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    store float [[CALL_I]], ptr [[Y:%.*]], align 4, !tbaa [[TBAA15]]
-// FINITEONLY-NEXT:    [[CALL1_I:%.*]] = tail call nnan ninf contract float @__ocml_native_cos_f32(float noundef [[X]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL1_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_native_cos_f32(float noundef nofpclass(nan inf) [[X]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    store float [[CALL1_I]], ptr [[Z:%.*]], align 4, !tbaa [[TBAA15]]
 // FINITEONLY-NEXT:    ret void
 //
@@ -3534,7 +3599,7 @@ extern "C" __device__ void test___sincosf(float x, float *y, float *z) {
 //
 // FINITEONLY-LABEL: @test___sinf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_native_sin_f32(float noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_native_sin_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test___sinf(float x) {
@@ -3548,7 +3613,7 @@ extern "C" __device__ float test___sinf(float x) {
 //
 // FINITEONLY-LABEL: @test___tanf(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract float @__ocml_tan_f32(float noundef [[X:%.*]]) #[[ATTR15]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_tan_f32(float noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR15]]
 // FINITEONLY-NEXT:    ret float [[CALL_I]]
 //
 extern "C" __device__ float test___tanf(float x) {
@@ -3618,7 +3683,7 @@ extern "C" __device__ double test___drcp_rn(double x) {
 //
 // FINITEONLY-LABEL: @test___dsqrt_rn(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_sqrt_f64(double noundef [[X:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_sqrt_f64(double noundef nofpclass(nan inf) [[X:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test___dsqrt_rn(double x) {
@@ -3632,7 +3697,7 @@ extern "C" __device__ double test___dsqrt_rn(double x) {
 //
 // FINITEONLY-LABEL: @test__fma_rn(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract double @__ocml_fma_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]], double noundef [[Z:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_fma_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]], double noundef nofpclass(nan inf) [[Z:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I]]
 //
 extern "C" __device__ double test__fma_rn(double x, double y, double z) {
@@ -3646,7 +3711,7 @@ extern "C" __device__ double test__fma_rn(double x, double y, double z) {
 //
 // FINITEONLY-LABEL: @test_float_min(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I_I:%.*]] = tail call nnan ninf contract float @__ocml_fmin_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_fmin_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I_I]]
 //
 extern "C" __device__ float test_float_min(float x, float y) {
@@ -3660,7 +3725,7 @@ extern "C" __device__ float test_float_min(float x, float y) {
 //
 // FINITEONLY-LABEL: @test_float_max(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I_I:%.*]] = tail call nnan ninf contract float @__ocml_fmax_f32(float noundef [[X:%.*]], float noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) float @__ocml_fmax_f32(float noundef nofpclass(nan inf) [[X:%.*]], float noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret float [[CALL_I_I]]
 //
 extern "C" __device__ float test_float_max(float x, float y) {
@@ -3674,7 +3739,7 @@ extern "C" __device__ float test_float_max(float x, float y) {
 //
 // FINITEONLY-LABEL: @test_double_min(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I_I:%.*]] = tail call nnan ninf contract double @__ocml_fmin_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_fmin_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I_I]]
 //
 extern "C" __device__ double test_double_min(double x, double y) {
@@ -3688,7 +3753,7 @@ extern "C" __device__ double test_double_min(double x, double y) {
 //
 // FINITEONLY-LABEL: @test_double_max(
 // FINITEONLY-NEXT:  entry:
-// FINITEONLY-NEXT:    [[CALL_I_I:%.*]] = tail call nnan ninf contract double @__ocml_fmax_f64(double noundef [[X:%.*]], double noundef [[Y:%.*]]) #[[ATTR13]]
+// FINITEONLY-NEXT:    [[CALL_I_I:%.*]] = tail call nnan ninf contract nofpclass(nan inf) double @__ocml_fmax_f64(double noundef nofpclass(nan inf) [[X:%.*]], double noundef nofpclass(nan inf) [[Y:%.*]]) #[[ATTR13]]
 // FINITEONLY-NEXT:    ret double [[CALL_I_I]]
 //
 extern "C" __device__ double test_double_max(double x, double y) {
diff --git a/clang/test/Headers/nvptx_device_math_sin.c b/clang/test/Headers/nvptx_device_math_sin.c
index 92692912789aa..e515db8370138 100644
--- a/clang/test/Headers/nvptx_device_math_sin.c
+++ b/clang/test/Headers/nvptx_device_math_sin.c
@@ -10,10 +10,10 @@
 double math(float f, double d) {
   double r = 0;
 // SLOW:  call float @__nv_sinf(float
-// FAST:  call fast float @__nv_fast_sinf(float
+// FAST:  call fast nofpclass(nan inf) float @__nv_fast_sinf(float
   r += sinf(f);
 // SLOW:  call double @__nv_sin(double
-// FAST:  call fast double @__nv_sin(double
+// FAST:  call fast nofpclass(nan inf) double @__nv_sin(double
   r += sin(d);
   return r;
 }
diff --git a/clang/test/Headers/nvptx_device_math_sin.cpp b/clang/test/Headers/nvptx_device_math_sin.cpp
index 7c6f102cd2501..8e2c50e961813 100644
--- a/clang/test/Headers/nvptx_device_math_sin.cpp
+++ b/clang/test/Headers/nvptx_device_math_sin.cpp
@@ -10,10 +10,10 @@
 double math(float f, double d) {
   double r = 0;
 // SLOW:  call float @__nv_sinf(float
-// FAST:  call fast float @__nv_fast_sinf(float
+// FAST:  call fast nofpclass(nan inf) float @__nv_fast_sinf(float
   r += sin(f);
 // SLOW:  call double @__nv_sin(double
-// FAST:  call fast double @__nv_sin(double
+// FAST:  call fast nofpclass(nan inf) double @__nv_sin(double
   r += sin(d);
   return r;
 }
diff --git a/clang/test/Index/crash-recovery-code-complete.c b/clang/test/Index/crash-recovery-code-complete.c
index dfb47316dc9d7..e7454c6f61022 100644
--- a/clang/test/Index/crash-recovery-code-complete.c
+++ b/clang/test/Index/crash-recovery-code-complete.c
@@ -1,3 +1,4 @@
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: env CINDEXTEST_EDITING=1 CINDEXTEST_PREAMBLE_FILE=%t-preamble.pch \
 // RUN:   not c-index-test -code-completion-at=%s:20:1 \
 // RUN:   "-remap-file=%s,%S/Inputs/crash-recovery-code-complete-remap.c" \
diff --git a/clang/test/Index/crash-recovery-modules.m b/clang/test/Index/crash-recovery-modules.m
index 72e25a0fbf5f2..00020e0e8162c 100644
--- a/clang/test/Index/crash-recovery-modules.m
+++ b/clang/test/Index/crash-recovery-modules.m
@@ -11,7 +11,7 @@
 // RUN: env CINDEXTEST_FAILONERROR=1 c-index-test -test-load-source all -fmodules -fmodules-cache-path=%t -Xclang -fdisable-module-hash -I %S/Inputs/Headers %s > /dev/null
 
 // REQUIRES: crash-recovery
-// UNSUPPORTED: libstdcxx-safe-mode
+// UNSUPPORTED: libstdcxx-safe-mode, hwasan
 
 @import Crash;
 
diff --git a/clang/test/Index/crash-recovery-reparse.c b/clang/test/Index/crash-recovery-reparse.c
index 2e4b51a8ca4b2..813d9d38b9307 100644
--- a/clang/test/Index/crash-recovery-reparse.c
+++ b/clang/test/Index/crash-recovery-reparse.c
@@ -1,3 +1,4 @@
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: env CINDEXTEST_EDITING=1 CINDEXTEST_PREAMBLE_FILE=%t-preamble.pch \
 // RUN:   not c-index-test -test-load-source-reparse 1 local \
 // RUN:   -remap-file="%s,%S/Inputs/crash-recovery-reparse-remap.c" \
diff --git a/clang/test/Index/crash-recovery.c b/clang/test/Index/crash-recovery.c
index bf13c69eb1279..5e1da41076dae 100644
--- a/clang/test/Index/crash-recovery.c
+++ b/clang/test/Index/crash-recovery.c
@@ -1,3 +1,4 @@
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: not c-index-test -test-load-source all %s 2> %t.err
 // RUN: FileCheck < %t.err -check-prefix=CHECK-LOAD-SOURCE-CRASH %s
 // CHECK-LOAD-SOURCE-CRASH: Unable to load translation unit
diff --git a/clang/test/Index/create-libclang-completion-reproducer.c b/clang/test/Index/create-libclang-completion-reproducer.c
index 5683e10075bf5..4eef599ba79dd 100644
--- a/clang/test/Index/create-libclang-completion-reproducer.c
+++ b/clang/test/Index/create-libclang-completion-reproducer.c
@@ -1,3 +1,4 @@
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: rm -rf %t
 // RUN: mkdir %t
 // RUN: env CINDEXTEST_INVOCATION_EMISSION_PATH=%t not c-index-test -code-completion-at=%s:10:1 "-remap-file=%s,%S/Inputs/record-parsing-invocation-remap.c" %s
diff --git a/clang/test/Index/create-libclang-parsing-reproducer.c b/clang/test/Index/create-libclang-parsing-reproducer.c
index 0cfbfb14481dc..1372f3b64cc87 100644
--- a/clang/test/Index/create-libclang-parsing-reproducer.c
+++ b/clang/test/Index/create-libclang-parsing-reproducer.c
@@ -1,3 +1,4 @@
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: rm -rf %t
 // RUN: mkdir %t
 // RUN: env CINDEXTEST_INVOCATION_EMISSION_PATH=%t not c-index-test -test-load-source all %s
diff --git a/clang/test/Index/error-on-deserialized.c b/clang/test/Index/error-on-deserialized.c
index 78b77e5cddaa4..ced34833fd45d 100644
--- a/clang/test/Index/error-on-deserialized.c
+++ b/clang/test/Index/error-on-deserialized.c
@@ -3,6 +3,7 @@
 
 // This tests that we will correctly error out on the deserialized decl.
 
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: c-index-test -write-pch %t.h.pch %S/targeted-top.h
 // RUN: env CINDEXTEST_FAILONERROR=1 not c-index-test -cursor-at=%S/targeted-nested1.h:2:16 %s -include %t.h \
 // RUN:    -Xclang -error-on-deserialized-decl=NestedVar1
diff --git a/clang/test/Index/pipe-size.cl b/clang/test/Index/pipe-size.cl
index 0a384db00ba66..08b936f1a9b07 100644
--- a/clang/test/Index/pipe-size.cl
+++ b/clang/test/Index/pipe-size.cl
@@ -7,9 +7,9 @@ __kernel void testPipe( pipe int test )
     int s = sizeof(test);
     // X86: store ptr %test, ptr %test.addr, align 8
     // X86: store i32 8, ptr %s, align 4
-    // SPIR: store ptr addrspace(1) %test, ptr %test.addr, align 4
+    // SPIR: store target("spirv.Pipe", 0) %test, ptr %test.addr, align 4
     // SPIR: store i32 4, ptr %s, align 4
-    // SPIR64: store ptr addrspace(1) %test, ptr %test.addr, align 8
+    // SPIR64: store target("spirv.Pipe", 0) %test, ptr %test.addr, align 8
     // SPIR64: store i32 8, ptr %s, align 4
     // AMDGCN: store ptr addrspace(1) %test, ptr addrspace(5) %test.addr, align 8
     // AMDGCN: store i32 8, ptr addrspace(5) %s, align 4
diff --git a/clang/test/Index/record-completion-invocation.c b/clang/test/Index/record-completion-invocation.c
index c249565a4b681..4b667134fa2d4 100644
--- a/clang/test/Index/record-completion-invocation.c
+++ b/clang/test/Index/record-completion-invocation.c
@@ -1,3 +1,4 @@
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: rm -rf %t
 // RUN: mkdir %t
 // RUN: env CINDEXTEST_INVOCATION_EMISSION_PATH=%t not c-index-test -code-completion-at=%s:10:1 "-remap-file=%s,%S/Inputs/record-parsing-invocation-remap.c" %s
diff --git a/clang/test/Index/record-parsing-invocation.c b/clang/test/Index/record-parsing-invocation.c
index 3254e58aef048..e0c4cdb05fb00 100644
--- a/clang/test/Index/record-parsing-invocation.c
+++ b/clang/test/Index/record-parsing-invocation.c
@@ -1,3 +1,4 @@
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: rm -rf %t
 // RUN: mkdir %t
 // RUN: env CINDEXTEST_INVOCATION_EMISSION_PATH=%t not c-index-test -test-load-source all %s
diff --git a/clang/test/Lexer/cxx-features.cpp b/clang/test/Lexer/cxx-features.cpp
index 5dee2c8197c89..ca7f987b78ab5 100644
--- a/clang/test/Lexer/cxx-features.cpp
+++ b/clang/test/Lexer/cxx-features.cpp
@@ -356,9 +356,3 @@
 #if defined(NO_EXCEPTIONS) ? check(exceptions, 0, 0, 0, 0, 0, 0) : check(exceptions, 199711, 199711, 199711, 199711, 199711, 199711)
 #error "wrong value for __cpp_exceptions"
 #endif
-
-// --- TS features --
-
-#if check(coroutines, 0, 0, 0, 0, 201703, 201703)
-#error "wrong value for __cpp_coroutines"
-#endif
diff --git a/clang/test/Modules/Inputs/DependsOnModule.framework/Headers/coroutines.h b/clang/test/Modules/Inputs/DependsOnModule.framework/Headers/coroutines.h
index 85281f5a09959..cae4099835d73 100644
--- a/clang/test/Modules/Inputs/DependsOnModule.framework/Headers/coroutines.h
+++ b/clang/test/Modules/Inputs/DependsOnModule.framework/Headers/coroutines.h
@@ -1,3 +1,3 @@
-#ifndef __cpp_coroutines
+#ifndef __cpp_impl_coroutine
 #error coroutines must be enabled
 #endif
diff --git a/clang/test/Modules/Inputs/DependsOnModule.framework/Headers/not_coroutines.h b/clang/test/Modules/Inputs/DependsOnModule.framework/Headers/not_coroutines.h
index 9312b9ad8926e..bf135a2440662 100644
--- a/clang/test/Modules/Inputs/DependsOnModule.framework/Headers/not_coroutines.h
+++ b/clang/test/Modules/Inputs/DependsOnModule.framework/Headers/not_coroutines.h
@@ -1,3 +1,3 @@
-#ifdef __cpp_coroutines
+#ifdef __cpp_impl_coroutine
 #error coroutines must NOT be enabled
 #endif
diff --git a/clang/test/Modules/codegen-opt.test b/clang/test/Modules/codegen-opt.test
index 6536611bd307e..eafd47ae1ee3a 100644
--- a/clang/test/Modules/codegen-opt.test
+++ b/clang/test/Modules/codegen-opt.test
@@ -1,16 +1,16 @@
 RUN: rm -rf %t
 REQUIRES: x86-registered-target
 
-RUN: %clang_cc1 -no-opaque-pointers -triple=x86_64-linux-gnu -fmodules-codegen -x c++ -fmodules -emit-module -fmodule-name=foo %S/Inputs/codegen-opt/foo.modulemap -o %t/foo.pcm
-RUN: %clang_cc1 -no-opaque-pointers -triple=x86_64-linux-gnu -fmodules-codegen -x c++ -fmodules -emit-module -fmodule-name=bar %S/Inputs/codegen-opt/bar.modulemap -o %t/bar.pcm -fmodule-file=%t/foo.pcm
+RUN: %clang_cc1 -triple=x86_64-linux-gnu -fmodules-codegen -x c++ -fmodules -emit-module -fmodule-name=foo %S/Inputs/codegen-opt/foo.modulemap -o %t/foo.pcm
+RUN: %clang_cc1 -triple=x86_64-linux-gnu -fmodules-codegen -x c++ -fmodules -emit-module -fmodule-name=bar %S/Inputs/codegen-opt/bar.modulemap -o %t/bar.pcm -fmodule-file=%t/foo.pcm
 
-RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-linux-gnu -emit-llvm -o - %t/foo.pcm | FileCheck --check-prefix=FOO %s
-RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-linux-gnu -emit-llvm -o - %t/bar.pcm -fmodule-file=%t/foo.pcm | FileCheck --check-prefix=BAR-CMN --check-prefix=BAR %s
-RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-linux-gnu -emit-llvm -o - -fmodules -fmodule-file=%t/foo.pcm -fmodule-file=%t/bar.pcm %S/Inputs/codegen-opt/use.cpp | FileCheck --check-prefix=USE-CMN --check-prefix=USE %s
+RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm -o - %t/foo.pcm | FileCheck --check-prefix=FOO %s
+RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm -o - %t/bar.pcm -fmodule-file=%t/foo.pcm | FileCheck --check-prefix=BAR-CMN --check-prefix=BAR %s
+RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm -o - -fmodules -fmodule-file=%t/foo.pcm -fmodule-file=%t/bar.pcm %S/Inputs/codegen-opt/use.cpp | FileCheck --check-prefix=USE-CMN --check-prefix=USE %s
 
-RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-linux-gnu -emit-llvm -o - -O2 -disable-llvm-passes %t/foo.pcm | FileCheck --check-prefix=FOO %s
-RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-linux-gnu -emit-llvm -o - -O2 -disable-llvm-passes %t/bar.pcm -fmodule-file=%t/foo.pcm | FileCheck --check-prefix=BAR-CMN --check-prefix=BAR-OPT %s
-RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-linux-gnu -emit-llvm -o - -O2 -disable-llvm-passes -fmodules -fmodule-file=%t/foo.pcm -fmodule-file=%t/bar.pcm %S/Inputs/codegen-opt/use.cpp | FileCheck --check-prefix=USE-CMN --check-prefix=USE-OPT %s
+RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm -o - -O2 -disable-llvm-passes %t/foo.pcm | FileCheck --check-prefix=FOO %s
+RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm -o - -O2 -disable-llvm-passes %t/bar.pcm -fmodule-file=%t/foo.pcm | FileCheck --check-prefix=BAR-CMN --check-prefix=BAR-OPT %s
+RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm -o - -O2 -disable-llvm-passes -fmodules -fmodule-file=%t/foo.pcm -fmodule-file=%t/bar.pcm %S/Inputs/codegen-opt/use.cpp | FileCheck --check-prefix=USE-CMN --check-prefix=USE-OPT %s
 
 FOO-NOT: comdat
 FOO: $_Z3foov = comdat any
@@ -22,7 +22,7 @@ FOO: define{{.*}} void @_Z7foo_extv()
 FOO-NOT: {{define|declare}}
 FOO: define weak_odr void @_Z3foov() #{{[0-9]+}} comdat
 FOO-NOT: {{define|declare}}
-FOO: declare void @_Z2f1Ri(i32*
+FOO: declare void @_Z2f1Ri(ptr
 FOO-NOT: {{define|declare}}
 
 Internal functions are not modularly code generated - they are
@@ -49,7 +49,7 @@ BAR: declare void @_Z3foov()
 Include all the available_externally definitions required for bar (foo -> f2)
 BAR-OPT: define available_externally void @_Z3foov()
 BAR-CMN-NOT: {{define|declare}}
-BAR-OPT: declare void @_Z2f1Ri(i32*
+BAR-OPT: declare void @_Z2f1Ri(ptr
 BAR-OPT-NOT: {{define|declare}}
 BAR-OPT: define internal void @_ZL2f2v()
 BAR-OPT-NOT: {{define|declare}}
@@ -65,7 +65,7 @@ USE-OPT: define available_externally void @_Z3barv()
 USE-CMN-NOT: {{define|declare}}
 USE-OPT: define available_externally void @_Z3foov()
 USE-OPT-NOT: {{define|declare}}
-USE-OPT: declare void @_Z2f1Ri(i32*
+USE-OPT: declare void @_Z2f1Ri(ptr
 USE-OPT-NOT: {{define|declare}}
 USE-OPT: define internal void @_ZL2f2v()
 USE-OPT-NOT: {{define|declare}}
diff --git a/clang/test/Modules/codegen.test b/clang/test/Modules/codegen.test
index d9c70e4385f81..77602056defd4 100644
--- a/clang/test/Modules/codegen.test
+++ b/clang/test/Modules/codegen.test
@@ -1,10 +1,10 @@
 RUN: rm -rf %t
 REQUIRES: x86-registered-target
 
-RUN: %clang_cc1 -no-opaque-pointers -triple=x86_64-linux-gnu -fmodules-codegen -fmodules-debuginfo -x c++ -fmodules -emit-module -fmodule-name=foo %S/Inputs/codegen/foo.modulemap -o %t/foo.pcm
+RUN: %clang_cc1 -triple=x86_64-linux-gnu -fmodules-codegen -fmodules-debuginfo -x c++ -fmodules -emit-module -fmodule-name=foo %S/Inputs/codegen/foo.modulemap -o %t/foo.pcm
 
-RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-linux-gnu -emit-llvm -debug-info-kind=limited -o - %t/foo.pcm | FileCheck --check-prefix=FOO --check-prefix=BOTH %s
-RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-linux-gnu -emit-llvm -debug-info-kind=limited -o - -fmodules -disable-llvm-passes -fmodule-file=%t/foo.pcm %S/Inputs/codegen/use.cpp | FileCheck --check-prefix=BOTH --check-prefix=USE %s
+RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm -debug-info-kind=limited -o - %t/foo.pcm | FileCheck --check-prefix=FOO --check-prefix=BOTH %s
+RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm -debug-info-kind=limited -o - -fmodules -disable-llvm-passes -fmodule-file=%t/foo.pcm %S/Inputs/codegen/use.cpp | FileCheck --check-prefix=BOTH --check-prefix=USE %s
 
 
 For want of any better definition, inline asm goes "everywhere" the same as it
@@ -25,8 +25,8 @@ FOO: $_ZN13implicit_dtorD1Ev = comdat any
 USE: $_Z4instIiEvv = comdat any
 USE: $_Z10always_inlv = comdat any
 FOO: $_ZN13implicit_dtorD2Ev = comdat any
-FOO: define weak_odr void @_Z2f1PKcz(i8* noundef %fmt, ...) #{{[0-9]+}} comdat
-FOO:   call void @llvm.va_start(i8* %{{[a-zA-Z0-9]*}})
+FOO: define weak_odr void @_Z2f1PKcz(ptr noundef %fmt, ...) #{{[0-9]+}} comdat
+FOO:   call void @llvm.va_start(ptr %{{[a-zA-Z0-9]*}})
 
 Test that implicit special members are emitted into the FOO module if they're
 ODR used there, otherwise emit them linkonce_odr as usual in the use.
diff --git a/clang/test/Modules/inconsistent-export.cppm b/clang/test/Modules/inconsistent-export.cppm
new file mode 100644
index 0000000000000..5e94d2b37b757
--- /dev/null
+++ b/clang/test/Modules/inconsistent-export.cppm
@@ -0,0 +1,46 @@
+// RUN: rm -fr %t
+// RUN: mkdir %t
+// RUN: split-file %s %t
+//
+// RUN: %clang_cc1 -std=c++20 %t/a.cppm -emit-module-interface -o %t/m-a.pcm
+// RUN: %clang_cc1 -std=c++20 %t/b.cppm -emit-module-interface -o %t/m-b.pcm \
+// RUN:     -fprebuilt-module-path=%t
+// RUN: %clang_cc1 -std=c++20 %t/m.cppm -emit-module-interface -o %t/m.pcm \
+// RUN:     -fprebuilt-module-path=%t
+// RUN: %clang_cc1 -std=c++20 %t/use.cppm -fprebuilt-module-path=%t -emit-obj
+
+//--- a.cppm
+export module m:a;
+namespace n {
+export class a {
+public:
+    virtual ~a() {}
+};
+}
+
+//--- b.cppm
+export module m:b;
+namespace n {
+class a;
+}
+
+//--- m.cppm
+export module m;
+export import :a;
+export import :b;
+
+//--- use.cppm
+// expected-no-diagnostics
+export module u;
+export import m;
+
+struct aa : public n::a {
+    aa() {}
+};
+auto foo(n::a*) {
+    return;
+}
+
+void use() {
+    n::a _;
+}
diff --git a/clang/test/Modules/initializers.cpp b/clang/test/Modules/initializers.cpp
index 435d48d12dfad..dcd9b08ec6f7a 100644
--- a/clang/test/Modules/initializers.cpp
+++ b/clang/test/Modules/initializers.cpp
@@ -1,9 +1,9 @@
-// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-linux-gnu -std=c++17 -emit-llvm -DIMPORT=1 -fmodules %s -o - | FileCheck %s --check-prefixes=CHECK,CHECK-IMPORT,CHECK-NO-NS,CHECK-IMPORT-NO-NS --implicit-check-not=unused
-// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-linux-gnu -std=c++17 -emit-llvm -DIMPORT=1 -DNS -fmodules %s -o - | FileCheck %s --check-prefixes=CHECK,CHECK-IMPORT,CHECK-NS,CHECK-IMPORT-NS --implicit-check-not=unused
-// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-linux-gnu -std=c++17 -emit-llvm -DIMPORT=2 -fmodules %s -o - | FileCheck %s --check-prefixes=CHECK,CHECK-NO-NS --implicit-check-not=unused
-// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-linux-gnu -std=c++17 -emit-llvm -DIMPORT=2 -DNS -fmodules %s -o - | FileCheck %s --check-prefixes=CHECK,CHECK-NS --implicit-check-not=unused
-// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-linux-gnu -std=c++17 -emit-llvm -fmodules %s -o - | FileCheck %s --check-prefixes=CHECK,CHECK-NO-NS --implicit-check-not=unused
-// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-linux-gnu -std=c++17 -emit-llvm -DNS -fmodules %s -o - | FileCheck %s --check-prefixes=CHECK,CHECK-NS --implicit-check-not=unused
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -std=c++17 -emit-llvm -DIMPORT=1 -fmodules %s -o - | FileCheck %s --check-prefixes=CHECK,CHECK-IMPORT,CHECK-NO-NS,CHECK-IMPORT-NO-NS --implicit-check-not=unused
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -std=c++17 -emit-llvm -DIMPORT=1 -DNS -fmodules %s -o - | FileCheck %s --check-prefixes=CHECK,CHECK-IMPORT,CHECK-NS,CHECK-IMPORT-NS --implicit-check-not=unused
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -std=c++17 -emit-llvm -DIMPORT=2 -fmodules %s -o - | FileCheck %s --check-prefixes=CHECK,CHECK-NO-NS --implicit-check-not=unused
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -std=c++17 -emit-llvm -DIMPORT=2 -DNS -fmodules %s -o - | FileCheck %s --check-prefixes=CHECK,CHECK-NS --implicit-check-not=unused
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -std=c++17 -emit-llvm -fmodules %s -o - | FileCheck %s --check-prefixes=CHECK,CHECK-NO-NS --implicit-check-not=unused
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -std=c++17 -emit-llvm -DNS -fmodules %s -o - | FileCheck %s --check-prefixes=CHECK,CHECK-NS --implicit-check-not=unused
 
 // Check that we behave sensibly when importing a header containing strong and
 // weak, ordered and unordered global initializers.
@@ -160,85 +160,85 @@ inline void use(bool b, ...) {
 // CHECK-SAME: @[[XE_INIT:[^,]*]], {{[^@]*}} @[[XE]]
 // CHECK-SAME: @[[XF_INIT:[^,]*]], {{[^@]*}} @[[XF]]
 // CHECK-SAME: @[[XB_INIT:[^,]*]], {{[^@]*}} @[[XB]]
-// CHECK-IMPORT-SAME: @[[TU_INIT:[^,]*]], i8* null }]
+// CHECK-IMPORT-SAME: @[[TU_INIT:[^,]*]], ptr null }]
 
 // FIXME: Should this use __cxa_guard_acquire?
 // CHECK: define {{.*}} @[[E_INIT]]()
-// CHECK: load {{.*}} (i64* @_ZGV
-// CHECK: store {{.*}}, i32* @[[E]],
+// CHECK: load {{.*}} ptr @_ZGV
+// CHECK: store {{.*}}, ptr @[[E]],
 
 // FIXME: Should this use __cxa_guard_acquire?
 // CHECK: define {{.*}} @[[F_INIT]]()
-// CHECK: load {{.*}} (i64* @_ZGV
-// CHECK: store {{.*}}, i32* @[[F]],
+// CHECK: load {{.*}} ptr @_ZGV
+// CHECK: store {{.*}}, ptr @[[F]],
 
 // CHECK: define {{.*}} @[[G_INIT:__cxx_global.*]]()
-// CHECK: load {{.*}} (i64* @_ZGV
-// CHECK: [[G_ADDR:%.+]] = call align 4 i32* @llvm.threadlocal.address.p0i32(i32* align 4 @[[G]])
-// CHECK: store {{.*}}, i32* [[G_ADDR]]
+// CHECK: load {{.*}} ptr @_ZGV
+// CHECK: [[G_ADDR:%.+]] = call align 4 ptr @llvm.threadlocal.address.p0(ptr align 4 @[[G]])
+// CHECK: store {{.*}}, ptr [[G_ADDR]]
 
 // CHECK: define {{.*}} @[[H_INIT:__cxx_global.*]]()
-// CHECK: load {{.*}} (i64* @_ZGV
-// CHECK: [[H_ADDR:%.+]] = call align 4 i32* @llvm.threadlocal.address.p0i32(i32* align 4 @[[H]])
-// CHECK: store {{.*}}, i32* [[H_ADDR]],
+// CHECK: load {{.*}} ptr @_ZGV
+// CHECK: [[H_ADDR:%.+]] = call align 4 ptr @llvm.threadlocal.address.p0(ptr align 4 @[[H]])
+// CHECK: store {{.*}}, ptr [[H_ADDR]],
 
 // FIXME: Should this use __cxa_guard_acquire?
 // CHECK: define {{.*}} @[[XA_INIT]]()
-// CHECK: load {{.*}} (i64* @_ZGV
-// CHECK: store {{.*}}, i32* @[[XA]],
+// CHECK: load {{.*}} ptr @_ZGV
+// CHECK: store {{.*}}, ptr @[[XA]],
 
 // CHECK: define {{.*}} @[[XC_INIT:__cxx_global.*]]()
-// CHECK: load {{.*}} (i64* @_ZGV
-// CHECK: [[XC_ADDR:%.+]] = call align 4 i32* @llvm.threadlocal.address.p0i32(i32* align 4 @[[XC]])
-// CHECK: store {{.*}}, i32* [[XC_ADDR]],
+// CHECK: load {{.*}} ptr @_ZGV
+// CHECK: [[XC_ADDR:%.+]] = call align 4 ptr @llvm.threadlocal.address.p0(ptr align 4 @[[XC]])
+// CHECK: store {{.*}}, ptr [[XC_ADDR]],
 
 // FIXME: Should this use __cxa_guard_acquire?
 // CHECK: define {{.*}} @[[XE_INIT]]()
-// CHECK: load {{.*}} (i64* @_ZGV
-// CHECK: store {{.*}}, i32* @[[XE]],
+// CHECK: load {{.*}} ptr @_ZGV
+// CHECK: store {{.*}}, ptr @[[XE]],
 
 // CHECK: define {{.*}} @[[XG_INIT:__cxx_global.*]]()
-// CHECK: load {{.*}} (i64* @_ZGV
-// CHECK: [[XG_ADDR:%.+]] = call align 4 i32* @llvm.threadlocal.address.p0i32(i32* align 4 @[[XG]])
-// CHECK: store {{.*}}, i32* [[XG_ADDR]],
+// CHECK: load {{.*}} ptr @_ZGV
+// CHECK: [[XG_ADDR:%.+]] = call align 4 ptr @llvm.threadlocal.address.p0(ptr align 4 @[[XG]])
+// CHECK: store {{.*}}, ptr [[XG_ADDR]],
 
 // CHECK: define {{.*}} @[[XH_INIT:__cxx_global.*]]()
-// CHECK: load {{.*}} (i64* @_ZGV
-// CHECK: [[XH_ADDR:%.+]] = call align 4 i32* @llvm.threadlocal.address.p0i32(i32* align 4 @[[XH]])
-// CHECK: store {{.*}}, i32* [[XH_ADDR]],
+// CHECK: load {{.*}} ptr @_ZGV
+// CHECK: [[XH_ADDR:%.+]] = call align 4 ptr @llvm.threadlocal.address.p0(ptr align 4 @[[XH]])
+// CHECK: store {{.*}}, ptr [[XH_ADDR]],
 
 // FIXME: Should this use __cxa_guard_acquire?
 // CHECK: define {{.*}} @[[XF_INIT]]()
-// CHECK: load {{.*}} (i64* @_ZGV
-// CHECK: store {{.*}}, i32* @[[XF]],
+// CHECK: load {{.*}} ptr @_ZGV
+// CHECK: store {{.*}}, ptr @[[XF]],
 
 // CHECK: define {{.*}} @[[XD_INIT:__cxx_global.*]]()
-// CHECK: load {{.*}} (i64* @_ZGV
-// CHECK: [[XD_ADDR:%.+]] = call align 4 i32* @llvm.threadlocal.address.p0i32(i32* align 4 @[[XD]])
-// CHECK: store {{.*}}, i32* [[XD_ADDR]],
+// CHECK: load {{.*}} ptr @_ZGV
+// CHECK: [[XD_ADDR:%.+]] = call align 4 ptr @llvm.threadlocal.address.p0(ptr align 4 @[[XD]])
+// CHECK: store {{.*}}, ptr [[XD_ADDR]],
 
 // FIXME: Should this use __cxa_guard_acquire?
 // CHECK: define {{.*}} @[[XB_INIT]]()
-// CHECK: load {{.*}} (i64* @_ZGV
-// CHECK: store {{.*}}, i32* @[[XB]],
+// CHECK: load {{.*}} ptr @_ZGV
+// CHECK: store {{.*}}, ptr @[[XB]],
 
 // CHECK-IMPORT: define {{.*}} @[[A_INIT:__cxx_global.*]]()
 // CHECK-IMPORT: call noundef i32 @_Z11non_trivialv(
-// CHECK-IMPORT: store {{.*}}, i32* @[[A]],
+// CHECK-IMPORT: store {{.*}}, ptr @[[A]],
 
 // CHECK-IMPORT: define {{.*}} @[[B_INIT:__cxx_global.*]]()
-// CHECK-IMPORT: call i32 @__cxa_guard_acquire(i64* @_ZGV
-// CHECK-IMPORT: store {{.*}}, i32* @[[B]],
+// CHECK-IMPORT: call i32 @__cxa_guard_acquire(ptr @_ZGV
+// CHECK-IMPORT: store {{.*}}, ptr @[[B]],
 
 // CHECK-IMPORT: define {{.*}} @[[C_INIT:__cxx_global.*]]()
 // CHECK-IMPORT: call noundef i32 @_Z11non_trivialv(
-// CHECK-IMPORT: [[C_ADDR:%.+]] = call align 4 i32* @llvm.threadlocal.address.p0i32(i32* align 4 @[[C]])
-// CHECK-IMPORT: store {{.*}}, i32* [[C_ADDR]],
+// CHECK-IMPORT: [[C_ADDR:%.+]] = call align 4 ptr @llvm.threadlocal.address.p0(ptr align 4 @[[C]])
+// CHECK-IMPORT: store {{.*}}, ptr [[C_ADDR]],
 
 // CHECK-IMPORT: define {{.*}} @[[D_INIT:__cxx_global.*]]()
-// CHECK-IMPORT: load {{.*}} (i64* @_ZGV
-// CHECK-IMPORT: [[D_ADDR:%.+]] = call align 4 i32* @llvm.threadlocal.address.p0i32(i32* align 4 @[[D]])
-// CHECK-IMPORT: store {{.*}}, i32* [[D_ADDR]],
+// CHECK-IMPORT: load {{.*}} ptr @_ZGV
+// CHECK-IMPORT: [[D_ADDR:%.+]] = call align 4 ptr @llvm.threadlocal.address.p0(ptr align 4 @[[D]])
+// CHECK-IMPORT: store {{.*}}, ptr [[D_ADDR]],
 
 
 // CHECK-IMPORT: define {{.*}} @[[TU_INIT]]()
diff --git a/clang/test/Modules/no-duplicate-codegen-in-GMF.cppm b/clang/test/Modules/no-duplicate-codegen-in-GMF.cppm
new file mode 100644
index 0000000000000..e4d06415fabd4
--- /dev/null
+++ b/clang/test/Modules/no-duplicate-codegen-in-GMF.cppm
@@ -0,0 +1,40 @@
+// Tests that the declaration won't get emitted after being merged.
+//
+// RUN: rm -rf %t
+// RUN: mkdir -p %t
+// RUN: split-file %s %t
+//
+// RUN: %clang_cc1 -std=c++20 -triple %itanium_abi_triple %t/A.cppm -emit-module-interface -o %t/A.pcm
+// RUN: %clang_cc1 -std=c++20 -triple %itanium_abi_triple %t/B.cppm -emit-module-interface -o %t/B.pcm \
+// RUN:     -fprebuilt-module-path=%t
+// RUN: %clang_cc1 -std=c++20 -triple %itanium_abi_triple %t/B.pcm -S -emit-llvm -o - | FileCheck %t/B.cppm
+
+//--- foo.h
+
+template <class T>
+class foo {
+public:
+    T value;
+    T GetValue() { return value; }
+};
+
+template class foo<int>;
+
+//--- A.cppm
+module;
+#include "foo.h"
+export module A;
+export using ::foo;
+
+//--- B.cppm
+module;
+#include "foo.h"
+export module B;
+import A;
+export using ::foo;
+export int B() {
+    foo<int> f;
+    return f.GetValue();
+}
+
+// CHECK-NOT: define{{.*}}@_ZN3fooIiE8GetValueEv
diff --git a/clang/test/Modules/no-eager-load.cppm b/clang/test/Modules/no-eager-load.cppm
new file mode 100644
index 0000000000000..6632cc60c8eb8
--- /dev/null
+++ b/clang/test/Modules/no-eager-load.cppm
@@ -0,0 +1,110 @@
+// RUN: rm -rf %t
+// RUN: split-file %s %t
+// RUN: cd %t
+//
+// RUN: %clang_cc1 -std=c++20 -emit-module-interface %t/a.cppm -o %t/a.pcm
+// RUN: %clang_cc1 -std=c++20 -emit-module-interface %t/b.cppm -o %t/b.pcm
+// RUN: %clang_cc1 -std=c++20 -fsyntax-only -verify %t/c.cpp \
+// RUN:     -fprebuilt-module-path=%t
+// RUN: %clang_cc1 -std=c++20 -fsyntax-only -verify %t/d.cpp \
+// RUN:     -fprebuilt-module-path=%t
+//
+// RUN: %clang_cc1 -std=c++20 -emit-module-interface %t/e.cppm -o %t/e.pcm
+// RUN: %clang_cc1 -std=c++20 -emit-module-interface %t/f.cppm -o %t/f.pcm
+// RUN: %clang_cc1 -std=c++20 -fsyntax-only -verify %t/g.cpp \
+// RUN:     -fprebuilt-module-path=%t
+//
+// RUN: %clang_cc1 -std=c++20 -emit-module-interface %t/h.cppm \
+// RUN:     -fprebuilt-module-path=%t -o %t/h.pcm
+// RUN: %clang_cc1 -std=c++20 -emit-module-interface %t/i.cppm \
+// RUN:     -fprebuilt-module-path=%t -o %t/i.pcm
+// RUN: %clang_cc1 -std=c++20 -fsyntax-only -verify %t/j.cpp \
+// RUN:     -fprebuilt-module-path=%t
+// RUN: %clang_cc1 -std=c++20 -fsyntax-only -verify %t/k.cpp \
+// RUN:     -fprebuilt-module-path=%t
+
+//--- a.cppm
+export module a;
+export void foo() {
+
+}
+
+//--- b.cppm
+export module b;
+void bar();
+export void foo() {
+    bar();
+}
+
+//--- c.cpp
+// expected-no-diagnostics
+// Since we will load all the declaration lazily, we won't be able to find
+// the ODR violation here.
+import a;
+import b;
+
+//--- d.cpp
+import a;
+import b;
+// Test that we can still check the odr violation if we call the function
+// actually.
+void use() {
+    foo(); // expected-error@* {{'foo' has different definitions in different modules;}}
+           // expected-note@* {{but in 'a' found a different body}}
+}
+
+//--- foo.h
+void foo() {
+
+}
+
+//--- bar.h
+void bar();
+void foo() {
+    bar();
+}
+
+//--- e.cppm
+module;
+#include "foo.h"
+export module e;
+export using ::foo;
+
+//--- f.cppm
+module;
+#include "bar.h"
+export module f;
+export using ::foo;
+
+//--- g.cpp
+import e;
+import f;
+void use() {
+    foo(); // expected-error@* {{'foo' has different definitions in different modules;}}
+           // expected-note@* {{but in 'e.<global>' found a different body}}
+}
+
+//--- h.cppm
+export module h;
+export import a;
+export import b;
+
+//--- i.cppm
+export module i;
+export import e;
+export import f;
+
+//--- j.cpp
+import h;
+void use() {
+    foo(); // expected-error@* {{'foo' has different definitions in different modules;}}
+           // expected-note@* {{but in 'a' found a different body}}
+}
+
+//--- k.cpp
+import i;
+void use() {
+    foo(); // expected-error@* {{'foo' has different definitions in different modules;}}
+           // expected-note@* {{but in 'e.<global>' found a different body}}
+}
+
diff --git a/clang/test/Modules/objc-initializer.m b/clang/test/Modules/objc-initializer.m
index 5736d78823ac4..771b845d831f7 100644
--- a/clang/test/Modules/objc-initializer.m
+++ b/clang/test/Modules/objc-initializer.m
@@ -1,8 +1,8 @@
 // UNSUPPORTED: target={{.*}}-zos{{.*}}, target={{.*}}-aix{{.*}}
 // RUN: rm -rf %t
-// RUN: %clang_cc1 -no-opaque-pointers -fmodules-cache-path=%t -fmodules -fimplicit-module-maps -I %S/Inputs/objc-initializer %s -emit-llvm -o - -fobjc-arc | FileCheck %s
-// RUN: %clang_cc1 -no-opaque-pointers -fmodules-cache-path=%t -fmodules -fimplicit-module-maps -I %S/Inputs/objc-initializer %s -emit-llvm -o - -fobjc-arc -DIMPORT_TOP | FileCheck %s
-// CHECK: kSimDeviceIOGetInterface = internal constant {{.*}} bitcast
+// RUN: %clang_cc1 -fmodules-cache-path=%t -fmodules -fimplicit-module-maps -I %S/Inputs/objc-initializer %s -emit-llvm -o - -fobjc-arc | FileCheck %s
+// RUN: %clang_cc1 -fmodules-cache-path=%t -fmodules -fimplicit-module-maps -I %S/Inputs/objc-initializer %s -emit-llvm -o - -fobjc-arc -DIMPORT_TOP | FileCheck %s
+// CHECK: kSimDeviceIOGetInterface = internal constant ptr
 
 #ifdef IMPORT_TOP
 @import X;
diff --git a/clang/test/Modules/pr38554.cppm b/clang/test/Modules/pr38554.cppm
new file mode 100644
index 0000000000000..0193e5b0e0611
--- /dev/null
+++ b/clang/test/Modules/pr38554.cppm
@@ -0,0 +1,21 @@
+// RUN: rm -rf %t
+// RUN: mkdir -p %t
+// RUN: split-file %s %t
+//
+// RUN: %clang_cc1 -std=c++20 -fsyntax-only -verify %t/c.cppm
+
+//--- a.hpp
+#pragma once
+using a = int;
+
+//--- b.hpp
+#pragma once
+#include "a.hpp"
+a b;
+
+//--- c.cppm
+// expected-no-diagnostics
+module;
+#include "b.hpp"
+export module c;
+export using ::a;
diff --git a/clang/test/Modules/pr58532.cppm b/clang/test/Modules/pr58532.cppm
new file mode 100644
index 0000000000000..cf530b4ac2ccc
--- /dev/null
+++ b/clang/test/Modules/pr58532.cppm
@@ -0,0 +1,34 @@
+// RUN: rm -rf %t
+// RUN: mkdir %t
+// RUN: split-file %s %t
+//
+// RUN: %clang_cc1 -std=c++20 %t/interface.cppm -emit-module-interface \
+// RUN:     -o %t/m.pcm
+// RUN: %clang_cc1 -std=c++20 %t/implementation.cpp -fmodule-file=m=%t/m.pcm \
+// RUN:     -fsyntax-only -verify
+
+//--- invisible.h
+#pragma once // This breaks things.
+const int kInvisibleSymbol = 0;
+struct invisible_struct
+{};
+#define INVISIBLE_DEFINE
+
+//--- visible.h
+#include "invisible.h"
+const int kSadlyUndeclaredSymbol = kInvisibleSymbol;
+using unfortunately_still_invisible_struct = invisible_struct;
+#ifndef INVISIBLE_DEFINE
+#    error "Still not defined."
+#endif
+
+//--- interface.cppm
+module;
+#include "visible.h"
+export module m;
+
+//--- implementation.cpp
+// expected-no-diagnostics
+module;
+#include "visible.h"
+module m;
diff --git a/clang/test/Modules/pr60085.cppm b/clang/test/Modules/pr60085.cppm
new file mode 100644
index 0000000000000..fba6012064047
--- /dev/null
+++ b/clang/test/Modules/pr60085.cppm
@@ -0,0 +1,98 @@
+// RUN: rm -rf %t
+// RUN: mkdir %t
+// RUN: split-file %s %t
+//
+// RUN: %clang_cc1 -std=c++20 -triple %itanium_abi_triple %t/d.cppm \
+// RUN:     -emit-module-interface -o %t/d.pcm
+// RUN: %clang_cc1 -std=c++20 -triple %itanium_abi_triple %t/c.cppm \
+// RUN:     -emit-module-interface -o %t/c.pcm -fmodule-file=%t/d.pcm
+// RUN: %clang_cc1 -std=c++20 -triple %itanium_abi_triple %t/b.cppm \
+// RUN:     -emit-module-interface -o %t/b.pcm -fmodule-file=%t/d.pcm
+// RUN: %clang_cc1 -std=c++20 -triple %itanium_abi_triple %t/a.cppm \
+// RUN:     -emit-module-interface -o %t/a.pcm -fmodule-file=%t/d.pcm \
+// RUN:     -fmodule-file=%t/c.pcm -fmodule-file=%t/b.pcm 
+// RUN: %clang_cc1 -std=c++20 -triple %itanium_abi_triple %t/a.pcm \
+// RUN:     -S -emit-llvm -disable-llvm-passes -o - | FileCheck %t/a.cppm
+//
+// Use -fmodule-file=<module-name>=<BMI-path>
+// RUN: %clang_cc1 -std=c++20 -triple %itanium_abi_triple %t/d.cppm \
+// RUN:     -emit-module-interface -o %t/d.pcm
+// RUN: %clang_cc1 -std=c++20 -triple %itanium_abi_triple %t/c.cppm \
+// RUN:     -emit-module-interface -o %t/c.pcm -fmodule-file=%t/d.pcm
+// RUN: %clang_cc1 -std=c++20 -triple %itanium_abi_triple %t/b.cppm \
+// RUN:     -emit-module-interface -o %t/b.pcm -fmodule-file=%t/d.pcm
+// RUN: %clang_cc1 -std=c++20 -triple %itanium_abi_triple %t/a.cppm \
+// RUN:     -emit-module-interface -o %t/a.pcm -fmodule-file=%t/d.pcm \
+// RUN:     -fmodule-file=%t/c.pcm -fmodule-file=%t/b.pcm 
+// RUN: %clang_cc1 -std=c++20 -triple %itanium_abi_triple %t/a.pcm \
+// RUN:     -S -emit-llvm -disable-llvm-passes -o - | FileCheck %t/a.cppm
+
+//--- d.cppm
+export module d;
+
+export template<typename>
+struct integer {
+	using type = int;
+	
+	static constexpr auto value() {
+		return 0;
+	}
+	
+	friend constexpr void f(integer const x) {
+		x.value();
+	}
+};
+
+export constexpr void ddd(auto const value) {
+	f(value);
+}
+
+
+template<typename T>
+constexpr auto dd = T();
+
+export template<typename T>
+constexpr auto d() {
+	dd<T>;
+}
+
+//--- c.cppm
+export module c;
+
+import d;
+
+template<typename T>
+auto cc = T();
+
+auto c() {
+	cc<integer<int>>;
+	integer<int>().value();
+}
+
+//--- b.cppm
+export module b;
+
+import d;
+
+auto b() {
+	integer<int>::type;
+}
+
+//--- a.cppm
+export module a;
+
+import b;
+import c;
+import d;
+
+constexpr void aa() {
+	d<integer<unsigned>>();
+	ddd(integer<int>());
+}
+
+export extern "C" void a() {
+	aa();
+}
+
+// Checks that we emit the IR successfully.
+// CHECK: define{{.*}}@a(
diff --git a/clang/test/Modules/pr61067.cppm b/clang/test/Modules/pr61067.cppm
new file mode 100644
index 0000000000000..8469a1db1f1c8
--- /dev/null
+++ b/clang/test/Modules/pr61067.cppm
@@ -0,0 +1,40 @@
+// From https://github.com/llvm/llvm-project/issues/61067
+// RUN: rm -rf %t
+// RUN: mkdir -p %t
+// RUN: split-file %s %t
+//
+// RUN: %clang_cc1 -std=c++20 -triple %itanium_abi_triple %t/a.cppm \
+// RUN:     -emit-module-interface -o %t/a.pcm
+// RUN: %clang_cc1 -std=c++20 -triple %itanium_abi_triple %t/b.cppm \
+// RUN:     -emit-module-interface -fmodule-file=a=%t/a.pcm -o %t/b.pcm
+// RUN: %clang_cc1 -std=c++20 -triple %itanium_abi_triple %t/b.pcm -S \
+// RUN:     -emit-llvm -disable-llvm-passes -o - | FileCheck %t/b.cppm
+// RUN: %clang_cc1 -std=c++20 -triple %itanium_abi_triple %t/c.cpp -fmodule-file=a=%t/a.pcm \
+// RUN:     -S -emit-llvm -disable-llvm-passes -o - | FileCheck %t/c.cpp
+
+//--- a.cppm
+export module a;
+
+export struct a {
+	friend bool operator==(a, a) = default;
+};
+
+//--- b.cppm
+export module b;
+
+import a;
+
+void b() {
+	(void)(a() == a());
+}
+
+// CHECK: define{{.*}}linkonce_odr{{.*}}@_ZW1aeqS_1aS0_(
+
+//--- c.cpp
+import a;
+
+int c() {
+    (void)(a() == a());
+}
+
+// CHECK: define{{.*}}linkonce_odr{{.*}}@_ZW1aeqS_1aS0_(
diff --git a/clang/test/Modules/pr61317.cppm b/clang/test/Modules/pr61317.cppm
new file mode 100644
index 0000000000000..4b9c1a3e97eab
--- /dev/null
+++ b/clang/test/Modules/pr61317.cppm
@@ -0,0 +1,49 @@
+// From https://github.com/llvm/llvm-project/issues/61317
+// RUN: rm -rf %t
+// RUN: mkdir -p %t
+// RUN: split-file %s %t
+//
+// RUN: %clang_cc1 -std=c++20 %t/A.cppm -emit-module-interface -o %t/A.pcm
+// RUN: %clang_cc1 -std=c++20 %t/B.cppm -emit-module-interface -o %t/B.pcm \
+// RUN:     -fprebuilt-module-path=%t
+// RUN: %clang_cc1 -std=c++20 %t/Use.cpp -fprebuilt-module-path=%t -fsyntax-only -verify
+
+//--- foo.h
+#ifndef _FOO
+#define _FOO
+
+template <typename T> struct Foo {
+  Foo(T) {}
+};
+
+template <typename T> Foo(T&) -> Foo<T>;
+
+struct Bar {
+  template <typename T>
+    requires requires { Foo{T()}; }
+  void baz() const {}
+};
+
+#endif
+
+//--- A.cppm
+module;
+#include "foo.h"
+export module A;
+export using ::Foo;
+export using ::Bar;
+
+//--- B.cppm
+module;
+#include "foo.h"
+export module B;
+export import A;
+
+//--- Use.cpp
+// expected-no-diagnostics
+import A;
+import B;
+void use() {
+  Bar _; 
+  _.baz<int>();
+}
diff --git a/clang/test/Modules/use-exportas-for-link.m b/clang/test/Modules/use-exportas-for-link.m
index 6f5fd596b4e69..722c65c2973a7 100644
--- a/clang/test/Modules/use-exportas-for-link.m
+++ b/clang/test/Modules/use-exportas-for-link.m
@@ -31,15 +31,17 @@
 #endif
 
 // RUN: %clang_cc1 -emit-llvm -o - -fmodules-cache-path=%t -DE -fmodules -fimplicit-module-maps -F %S/Inputs/exportas-link %s | FileCheck --check-prefix=CHECK_E %s
-// CHECK_E: !llvm.linker.options = !{![[MODULE:[0-9]+]]}
-// CHECK_E: ![[MODULE]] = !{!"-framework", !"SomeKitCore"}
+// CHECK_E: !llvm.linker.options = !{![[MODULE1:[0-9]+]], ![[MODULE2:[0-9]+]]}
+// CHECK_E: ![[MODULE1]] = !{!"-framework", !"OtherKit"}
+// CHECK_E: ![[MODULE2]] = !{!"-framework", !"SomeKitCore"}
 #ifdef E
 @import OtherKit;
 #endif
 
 // RUN: %clang_cc1 -emit-llvm -o - -fmodules-cache-path=%t -DF -fmodules -fimplicit-module-maps -F %S/Inputs/exportas-link %s | FileCheck --check-prefix=CHECK_F %s
-// CHECK_F: !llvm.linker.options = !{![[MODULE:[0-9]+]]}
-// CHECK_F: ![[MODULE]] = !{!"-framework", !"SomeKit"}
+// CHECK_F: !llvm.linker.options = !{![[MODULE1:[0-9]+]], ![[MODULE2:[0-9]+]]}
+// CHECK_F: ![[MODULE1]] = !{!"-framework", !"OtherKit"}
+// CHECK_F: ![[MODULE2]] = !{!"-framework", !"SomeKit"}
 #ifdef F
 @import OtherKit;
 #endif
diff --git a/clang/test/OpenMP/Inputs/libomp.a b/clang/test/OpenMP/Inputs/libomp.a
deleted file mode 100644
index 8b277f0dd5dcd..0000000000000
--- a/clang/test/OpenMP/Inputs/libomp.a
+++ /dev/null
@@ -1 +0,0 @@
-!<arch>
diff --git a/clang/test/OpenMP/align_clause_codegen.cpp b/clang/test/OpenMP/align_clause_codegen.cpp
index 47fed837dfdba..49ece6e3319ec 100644
--- a/clang/test/OpenMP/align_clause_codegen.cpp
+++ b/clang/test/OpenMP/align_clause_codegen.cpp
@@ -99,14 +99,14 @@ int template_test() {
 // CHECK-32-NEXT:    [[DOTFOO10__VOID_ADDR:%.*]] = call ptr @__kmpc_aligned_alloc(i32 [[TMP0]], i32 128, i32 720, ptr null)
 // CHECK-32-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTFOO10__VOID_ADDR]], ptr null)
 // CHECK-32-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTFOO9__VOID_ADDR]], ptr inttoptr (i32 8 to ptr))
-// CHECK-32-NEXT:    [[TMP3:%.*]] = load i32, ptr [[MYALLOC]], align 4
-// CHECK-32-NEXT:    [[CONV:%.*]] = inttoptr i32 [[TMP3]] to ptr
+// CHECK-32-NEXT:    [[TMP1:%.*]] = load i32, ptr [[MYALLOC]], align 4
+// CHECK-32-NEXT:    [[CONV:%.*]] = inttoptr i32 [[TMP1]] to ptr
 // CHECK-32-NEXT:    [[DOTBAR1__VOID_ADDR:%.*]] = call ptr @__kmpc_aligned_alloc(i32 [[TMP0]], i32 4, i32 4, ptr [[CONV]])
-// CHECK-32-NEXT:    [[TMP4:%.*]] = load i32, ptr [[MYALLOC]], align 4
-// CHECK-32-NEXT:    [[CONV1:%.*]] = inttoptr i32 [[TMP4]] to ptr
+// CHECK-32-NEXT:    [[TMP2:%.*]] = load i32, ptr [[MYALLOC]], align 4
+// CHECK-32-NEXT:    [[CONV1:%.*]] = inttoptr i32 [[TMP2]] to ptr
 // CHECK-32-NEXT:    [[DOTBAR2__VOID_ADDR:%.*]] = call ptr @__kmpc_aligned_alloc(i32 [[TMP0]], i32 4, i32 40, ptr [[CONV1]])
-// CHECK-32-NEXT:    [[TMP5:%.*]] = load i32, ptr [[MYALLOC]], align 4
-// CHECK-32-NEXT:    [[CONV2:%.*]] = inttoptr i32 [[TMP5]] to ptr
+// CHECK-32-NEXT:    [[TMP3:%.*]] = load i32, ptr [[MYALLOC]], align 4
+// CHECK-32-NEXT:    [[CONV2:%.*]] = inttoptr i32 [[TMP3]] to ptr
 // CHECK-32-NEXT:    [[DOTBAR3__VOID_ADDR:%.*]] = call ptr @__kmpc_aligned_alloc(i32 [[TMP0]], i32 4, i32 80, ptr [[CONV2]])
 // CHECK-32-NEXT:    [[DOTBAR4__VOID_ADDR:%.*]] = call ptr @__kmpc_aligned_alloc(i32 [[TMP0]], i32 16, i32 4, ptr null)
 // CHECK-32-NEXT:    [[DOTBAR5__VOID_ADDR:%.*]] = call ptr @__kmpc_aligned_alloc(i32 [[TMP0]], i32 16, i32 4, ptr null)
@@ -114,14 +114,14 @@ int template_test() {
 // CHECK-32-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTBAR6__VOID_ADDR]], ptr null)
 // CHECK-32-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTBAR5__VOID_ADDR]], ptr null)
 // CHECK-32-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTBAR4__VOID_ADDR]], ptr null)
-// CHECK-32-NEXT:    [[TMP10:%.*]] = load i32, ptr [[MYALLOC]], align 4
-// CHECK-32-NEXT:    [[CONV3:%.*]] = inttoptr i32 [[TMP10]] to ptr
+// CHECK-32-NEXT:    [[TMP4:%.*]] = load i32, ptr [[MYALLOC]], align 4
+// CHECK-32-NEXT:    [[CONV3:%.*]] = inttoptr i32 [[TMP4]] to ptr
 // CHECK-32-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTBAR3__VOID_ADDR]], ptr [[CONV3]])
-// CHECK-32-NEXT:    [[TMP12:%.*]] = load i32, ptr [[MYALLOC]], align 4
-// CHECK-32-NEXT:    [[CONV4:%.*]] = inttoptr i32 [[TMP12]] to ptr
+// CHECK-32-NEXT:    [[TMP5:%.*]] = load i32, ptr [[MYALLOC]], align 4
+// CHECK-32-NEXT:    [[CONV4:%.*]] = inttoptr i32 [[TMP5]] to ptr
 // CHECK-32-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTBAR2__VOID_ADDR]], ptr [[CONV4]])
-// CHECK-32-NEXT:    [[TMP14:%.*]] = load i32, ptr [[MYALLOC]], align 4
-// CHECK-32-NEXT:    [[CONV5:%.*]] = inttoptr i32 [[TMP14]] to ptr
+// CHECK-32-NEXT:    [[TMP6:%.*]] = load i32, ptr [[MYALLOC]], align 4
+// CHECK-32-NEXT:    [[CONV5:%.*]] = inttoptr i32 [[TMP6]] to ptr
 // CHECK-32-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTBAR1__VOID_ADDR]], ptr [[CONV5]])
 // CHECK-32-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTFOO8__VOID_ADDR]], ptr null)
 // CHECK-32-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTFOO7__VOID_ADDR]], ptr inttoptr (i32 8 to ptr))
@@ -174,14 +174,14 @@ int template_test() {
 // CHECK-NEXT:    [[DOTFOO10__VOID_ADDR:%.*]] = call ptr @__kmpc_aligned_alloc(i32 [[TMP0]], i64 128, i64 720, ptr null)
 // CHECK-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTFOO10__VOID_ADDR]], ptr null)
 // CHECK-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTFOO9__VOID_ADDR]], ptr inttoptr (i64 8 to ptr))
-// CHECK-NEXT:    [[TMP3:%.*]] = load i64, ptr [[MYALLOC]], align 8
-// CHECK-NEXT:    [[CONV:%.*]] = inttoptr i64 [[TMP3]] to ptr
+// CHECK-NEXT:    [[TMP1:%.*]] = load i64, ptr [[MYALLOC]], align 8
+// CHECK-NEXT:    [[CONV:%.*]] = inttoptr i64 [[TMP1]] to ptr
 // CHECK-NEXT:    [[DOTBAR1__VOID_ADDR:%.*]] = call ptr @__kmpc_aligned_alloc(i32 [[TMP0]], i64 4, i64 4, ptr [[CONV]])
-// CHECK-NEXT:    [[TMP4:%.*]] = load i64, ptr [[MYALLOC]], align 8
-// CHECK-NEXT:    [[CONV1:%.*]] = inttoptr i64 [[TMP4]] to ptr
+// CHECK-NEXT:    [[TMP2:%.*]] = load i64, ptr [[MYALLOC]], align 8
+// CHECK-NEXT:    [[CONV1:%.*]] = inttoptr i64 [[TMP2]] to ptr
 // CHECK-NEXT:    [[DOTBAR2__VOID_ADDR:%.*]] = call ptr @__kmpc_aligned_alloc(i32 [[TMP0]], i64 4, i64 48, ptr [[CONV1]])
-// CHECK-NEXT:    [[TMP5:%.*]] = load i64, ptr [[MYALLOC]], align 8
-// CHECK-NEXT:    [[CONV2:%.*]] = inttoptr i64 [[TMP5]] to ptr
+// CHECK-NEXT:    [[TMP3:%.*]] = load i64, ptr [[MYALLOC]], align 8
+// CHECK-NEXT:    [[CONV2:%.*]] = inttoptr i64 [[TMP3]] to ptr
 // CHECK-NEXT:    [[DOTBAR3__VOID_ADDR:%.*]] = call ptr @__kmpc_aligned_alloc(i32 [[TMP0]], i64 4, i64 80, ptr [[CONV2]])
 // CHECK-NEXT:    [[DOTBAR4__VOID_ADDR:%.*]] = call ptr @__kmpc_aligned_alloc(i32 [[TMP0]], i64 16, i64 8, ptr null)
 // CHECK-NEXT:    [[DOTBAR5__VOID_ADDR:%.*]] = call ptr @__kmpc_aligned_alloc(i32 [[TMP0]], i64 16, i64 4, ptr null)
@@ -189,14 +189,14 @@ int template_test() {
 // CHECK-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTBAR6__VOID_ADDR]], ptr null)
 // CHECK-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTBAR5__VOID_ADDR]], ptr null)
 // CHECK-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTBAR4__VOID_ADDR]], ptr null)
-// CHECK-NEXT:    [[TMP10:%.*]] = load i64, ptr [[MYALLOC]], align 8
-// CHECK-NEXT:    [[CONV3:%.*]] = inttoptr i64 [[TMP10]] to ptr
+// CHECK-NEXT:    [[TMP4:%.*]] = load i64, ptr [[MYALLOC]], align 8
+// CHECK-NEXT:    [[CONV3:%.*]] = inttoptr i64 [[TMP4]] to ptr
 // CHECK-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTBAR3__VOID_ADDR]], ptr [[CONV3]])
-// CHECK-NEXT:    [[TMP12:%.*]] = load i64, ptr [[MYALLOC]], align 8
-// CHECK-NEXT:    [[CONV4:%.*]] = inttoptr i64 [[TMP12]] to ptr
+// CHECK-NEXT:    [[TMP5:%.*]] = load i64, ptr [[MYALLOC]], align 8
+// CHECK-NEXT:    [[CONV4:%.*]] = inttoptr i64 [[TMP5]] to ptr
 // CHECK-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTBAR2__VOID_ADDR]], ptr [[CONV4]])
-// CHECK-NEXT:    [[TMP14:%.*]] = load i64, ptr [[MYALLOC]], align 8
-// CHECK-NEXT:    [[CONV5:%.*]] = inttoptr i64 [[TMP14]] to ptr
+// CHECK-NEXT:    [[TMP6:%.*]] = load i64, ptr [[MYALLOC]], align 8
+// CHECK-NEXT:    [[CONV5:%.*]] = inttoptr i64 [[TMP6]] to ptr
 // CHECK-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTBAR1__VOID_ADDR]], ptr [[CONV5]])
 // CHECK-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTFOO8__VOID_ADDR]], ptr null)
 // CHECK-NEXT:    call void @__kmpc_free(i32 [[TMP0]], ptr [[DOTFOO7__VOID_ADDR]], ptr inttoptr (i64 8 to ptr))
diff --git a/clang/test/OpenMP/atomic_compare_codegen.cpp b/clang/test/OpenMP/atomic_compare_codegen.cpp
index 0deb8ac553433..90ea3b2ab92dd 100644
--- a/clang/test/OpenMP/atomic_compare_codegen.cpp
+++ b/clang/test/OpenMP/atomic_compare_codegen.cpp
@@ -6,7 +6,6 @@
 // RUN: %clang_cc1 -verify -triple x86_64-apple-darwin10 -target-cpu core2 -fopenmp-simd -fopenmp-version=51 -x c -emit-llvm %s -o - | FileCheck --check-prefix SIMD-ONLY0 %s
 // RUN: %clang_cc1 -fopenmp-simd -fopenmp-version=51 -x c -triple x86_64-apple-darwin10 -target-cpu core2 -emit-pch -o %t %s
 // RUN: %clang_cc1 -fopenmp-simd -fopenmp-version=51 -x c -triple x86_64-apple-darwin10 -target-cpu core2 -include-pch %t -verify %s -emit-llvm -o - | FileCheck --check-prefix SIMD-ONLY0 %s
-// SIMD-ONLY0-NOT: {{__kmpc|__tgt}}
 // expected-no-diagnostics
 
 #ifndef HEADER
@@ -13808,8 +13807,7 @@ double dxevd() {
 }
 
 #endif
-// CHECK-LABEL: define {{[^@]+}}@foo
-// CHECK-SAME: () #[[ATTR0:[0-9]+]] {
+// CHECK-LABEL: @foo(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[CX:%.*]] = alloca i8, align 1
 // CHECK-NEXT:    [[CE:%.*]] = alloca i8, align 1
@@ -16130,8 +16128,7 @@ double dxevd() {
 // CHECK-NEXT:    ret void
 //
 //
-// CHECK-LABEL: define {{[^@]+}}@bar
-// CHECK-SAME: () #[[ATTR0]] {
+// CHECK-LABEL: @bar(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[CX:%.*]] = alloca i8, align 1
 // CHECK-NEXT:    [[CV:%.*]] = alloca i8, align 1
@@ -25386,8 +25383,7 @@ double dxevd() {
 // CHECK-NEXT:    ret void
 //
 //
-// CHECK-LABEL: define {{[^@]+}}@cxevd
-// CHECK-SAME: () #[[ATTR0]] {
+// CHECK-LABEL: @cxevd(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[CX:%.*]] = alloca i8, align 1
 // CHECK-NEXT:    [[CV:%.*]] = alloca i8, align 1
@@ -25583,8 +25579,7 @@ double dxevd() {
 // CHECK-NEXT:    ret i8 [[TMP132]]
 //
 //
-// CHECK-LABEL: define {{[^@]+}}@ucxevd
-// CHECK-SAME: () #[[ATTR0]] {
+// CHECK-LABEL: @ucxevd(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[UCX:%.*]] = alloca i8, align 1
 // CHECK-NEXT:    [[UCV:%.*]] = alloca i8, align 1
@@ -25780,8 +25775,7 @@ double dxevd() {
 // CHECK-NEXT:    ret i8 [[TMP132]]
 //
 //
-// CHECK-LABEL: define {{[^@]+}}@sxevd
-// CHECK-SAME: () #[[ATTR0]] {
+// CHECK-LABEL: @sxevd(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[SX:%.*]] = alloca i16, align 2
 // CHECK-NEXT:    [[SV:%.*]] = alloca i16, align 2
@@ -25977,8 +25971,7 @@ double dxevd() {
 // CHECK-NEXT:    ret i16 [[TMP132]]
 //
 //
-// CHECK-LABEL: define {{[^@]+}}@usxevd
-// CHECK-SAME: () #[[ATTR0]] {
+// CHECK-LABEL: @usxevd(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[USX:%.*]] = alloca i16, align 2
 // CHECK-NEXT:    [[USV:%.*]] = alloca i16, align 2
@@ -26174,8 +26167,7 @@ double dxevd() {
 // CHECK-NEXT:    ret i16 [[TMP132]]
 //
 //
-// CHECK-LABEL: define {{[^@]+}}@ixevd
-// CHECK-SAME: () #[[ATTR0]] {
+// CHECK-LABEL: @ixevd(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[IX:%.*]] = alloca i32, align 4
 // CHECK-NEXT:    [[IV:%.*]] = alloca i32, align 4
@@ -26371,8 +26363,7 @@ double dxevd() {
 // CHECK-NEXT:    ret i32 [[TMP132]]
 //
 //
-// CHECK-LABEL: define {{[^@]+}}@uixevd
-// CHECK-SAME: () #[[ATTR0]] {
+// CHECK-LABEL: @uixevd(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[UIX:%.*]] = alloca i32, align 4
 // CHECK-NEXT:    [[UIV:%.*]] = alloca i32, align 4
@@ -26568,8 +26559,7 @@ double dxevd() {
 // CHECK-NEXT:    ret i32 [[TMP132]]
 //
 //
-// CHECK-LABEL: define {{[^@]+}}@lxevd
-// CHECK-SAME: () #[[ATTR0]] {
+// CHECK-LABEL: @lxevd(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[LX:%.*]] = alloca i64, align 8
 // CHECK-NEXT:    [[LV:%.*]] = alloca i64, align 8
@@ -26765,8 +26755,7 @@ double dxevd() {
 // CHECK-NEXT:    ret i64 [[TMP132]]
 //
 //
-// CHECK-LABEL: define {{[^@]+}}@ulxevd
-// CHECK-SAME: () #[[ATTR0]] {
+// CHECK-LABEL: @ulxevd(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[ULX:%.*]] = alloca i64, align 8
 // CHECK-NEXT:    [[ULV:%.*]] = alloca i64, align 8
@@ -26962,8 +26951,7 @@ double dxevd() {
 // CHECK-NEXT:    ret i64 [[TMP132]]
 //
 //
-// CHECK-LABEL: define {{[^@]+}}@llxevd
-// CHECK-SAME: () #[[ATTR0]] {
+// CHECK-LABEL: @llxevd(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[LLX:%.*]] = alloca i64, align 8
 // CHECK-NEXT:    [[LLV:%.*]] = alloca i64, align 8
@@ -27159,8 +27147,7 @@ double dxevd() {
 // CHECK-NEXT:    ret i64 [[TMP132]]
 //
 //
-// CHECK-LABEL: define {{[^@]+}}@ullxevd
-// CHECK-SAME: () #[[ATTR0]] {
+// CHECK-LABEL: @ullxevd(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[ULLX:%.*]] = alloca i64, align 8
 // CHECK-NEXT:    [[ULLV:%.*]] = alloca i64, align 8
@@ -27356,8 +27343,7 @@ double dxevd() {
 // CHECK-NEXT:    ret i64 [[TMP132]]
 //
 //
-// CHECK-LABEL: define {{[^@]+}}@fxevd
-// CHECK-SAME: () #[[ATTR0]] {
+// CHECK-LABEL: @fxevd(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[FX:%.*]] = alloca float, align 4
 // CHECK-NEXT:    [[FV:%.*]] = alloca float, align 4
@@ -27589,8 +27575,7 @@ double dxevd() {
 // CHECK-NEXT:    ret float [[TMP168]]
 //
 //
-// CHECK-LABEL: define {{[^@]+}}@dxevd
-// CHECK-SAME: () #[[ATTR0]] {
+// CHECK-LABEL: @dxevd(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[DX:%.*]] = alloca double, align 8
 // CHECK-NEXT:    [[DV:%.*]] = alloca double, align 8
@@ -27820,3 +27805,34164 @@ double dxevd() {
 // CHECK-NEXT:    call void @__kmpc_flush(ptr @[[GLOB1]])
 // CHECK-NEXT:    [[TMP168:%.*]] = load double, ptr [[DV]], align 8
 // CHECK-NEXT:    ret double [[TMP168]]
+//
+//
+// SIMD-ONLY0-LABEL: @foo(
+// SIMD-ONLY0-NEXT:  entry:
+// SIMD-ONLY0-NEXT:    [[CX:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[CE:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[CD:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[UCX:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[UCE:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[UCD:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[SX:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[SE:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[SD:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[USX:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[USE:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[USD:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[IX:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[IE:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[ID:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[UIX:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[UIE:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[UID:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[LX:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LE:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LD:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULX:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULE:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULD:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LLX:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LLE:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LLD:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULLX:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULLE:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULLD:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[FX:%.*]] = alloca float, align 4
+// SIMD-ONLY0-NEXT:    [[FE:%.*]] = alloca float, align 4
+// SIMD-ONLY0-NEXT:    [[FD:%.*]] = alloca float, align 4
+// SIMD-ONLY0-NEXT:    [[DX:%.*]] = alloca double, align 8
+// SIMD-ONLY0-NEXT:    [[DE:%.*]] = alloca double, align 8
+// SIMD-ONLY0-NEXT:    [[DD:%.*]] = alloca double, align 8
+// SIMD-ONLY0-NEXT:    [[TMP0:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV:%.*]] = sext i8 [[TMP0]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1:%.*]] = sext i8 [[TMP1]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[CONV]], [[CONV1]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]]
+// SIMD-ONLY0:       cond.true:
+// SIMD-ONLY0-NEXT:    [[TMP2:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV3:%.*]] = sext i8 [[TMP2]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END:%.*]]
+// SIMD-ONLY0:       cond.false:
+// SIMD-ONLY0-NEXT:    [[TMP3:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV4:%.*]] = sext i8 [[TMP3]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END]]
+// SIMD-ONLY0:       cond.end:
+// SIMD-ONLY0-NEXT:    [[COND:%.*]] = phi i32 [ [[CONV3]], [[COND_TRUE]] ], [ [[CONV4]], [[COND_FALSE]] ]
+// SIMD-ONLY0-NEXT:    [[CONV5:%.*]] = trunc i32 [[COND]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV5]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP4:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV6:%.*]] = sext i8 [[TMP4]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP5:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV7:%.*]] = sext i8 [[TMP5]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP8:%.*]] = icmp slt i32 [[CONV6]], [[CONV7]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP8]], label [[COND_TRUE10:%.*]], label [[COND_FALSE12:%.*]]
+// SIMD-ONLY0:       cond.true10:
+// SIMD-ONLY0-NEXT:    [[TMP6:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV11:%.*]] = sext i8 [[TMP6]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END14:%.*]]
+// SIMD-ONLY0:       cond.false12:
+// SIMD-ONLY0-NEXT:    [[TMP7:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV13:%.*]] = sext i8 [[TMP7]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END14]]
+// SIMD-ONLY0:       cond.end14:
+// SIMD-ONLY0-NEXT:    [[COND15:%.*]] = phi i32 [ [[CONV11]], [[COND_TRUE10]] ], [ [[CONV13]], [[COND_FALSE12]] ]
+// SIMD-ONLY0-NEXT:    [[CONV16:%.*]] = trunc i32 [[COND15]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV16]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP8:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV17:%.*]] = sext i8 [[TMP8]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP9:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV18:%.*]] = sext i8 [[TMP9]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP19:%.*]] = icmp sgt i32 [[CONV17]], [[CONV18]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP19]], label [[COND_TRUE21:%.*]], label [[COND_FALSE23:%.*]]
+// SIMD-ONLY0:       cond.true21:
+// SIMD-ONLY0-NEXT:    [[TMP10:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV22:%.*]] = sext i8 [[TMP10]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END25:%.*]]
+// SIMD-ONLY0:       cond.false23:
+// SIMD-ONLY0-NEXT:    [[TMP11:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV24:%.*]] = sext i8 [[TMP11]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END25]]
+// SIMD-ONLY0:       cond.end25:
+// SIMD-ONLY0-NEXT:    [[COND26:%.*]] = phi i32 [ [[CONV22]], [[COND_TRUE21]] ], [ [[CONV24]], [[COND_FALSE23]] ]
+// SIMD-ONLY0-NEXT:    [[CONV27:%.*]] = trunc i32 [[COND26]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV27]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP12:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV28:%.*]] = sext i8 [[TMP12]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP13:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV29:%.*]] = sext i8 [[TMP13]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP30:%.*]] = icmp slt i32 [[CONV28]], [[CONV29]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP30]], label [[COND_TRUE32:%.*]], label [[COND_FALSE34:%.*]]
+// SIMD-ONLY0:       cond.true32:
+// SIMD-ONLY0-NEXT:    [[TMP14:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV33:%.*]] = sext i8 [[TMP14]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END36:%.*]]
+// SIMD-ONLY0:       cond.false34:
+// SIMD-ONLY0-NEXT:    [[TMP15:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV35:%.*]] = sext i8 [[TMP15]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END36]]
+// SIMD-ONLY0:       cond.end36:
+// SIMD-ONLY0-NEXT:    [[COND37:%.*]] = phi i32 [ [[CONV33]], [[COND_TRUE32]] ], [ [[CONV35]], [[COND_FALSE34]] ]
+// SIMD-ONLY0-NEXT:    [[CONV38:%.*]] = trunc i32 [[COND37]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV38]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP16:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV39:%.*]] = sext i8 [[TMP16]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP17:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV40:%.*]] = sext i8 [[TMP17]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP41:%.*]] = icmp sgt i32 [[CONV39]], [[CONV40]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP41]], label [[IF_THEN:%.*]], label [[IF_END:%.*]]
+// SIMD-ONLY0:       if.then:
+// SIMD-ONLY0-NEXT:    [[TMP18:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP18]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END]]
+// SIMD-ONLY0:       if.end:
+// SIMD-ONLY0-NEXT:    [[TMP19:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV43:%.*]] = sext i8 [[TMP19]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP20:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV44:%.*]] = sext i8 [[TMP20]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP45:%.*]] = icmp slt i32 [[CONV43]], [[CONV44]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP45]], label [[IF_THEN47:%.*]], label [[IF_END48:%.*]]
+// SIMD-ONLY0:       if.then47:
+// SIMD-ONLY0-NEXT:    [[TMP21:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP21]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END48]]
+// SIMD-ONLY0:       if.end48:
+// SIMD-ONLY0-NEXT:    [[TMP22:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV49:%.*]] = sext i8 [[TMP22]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP23:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV50:%.*]] = sext i8 [[TMP23]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP51:%.*]] = icmp sgt i32 [[CONV49]], [[CONV50]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP51]], label [[IF_THEN53:%.*]], label [[IF_END54:%.*]]
+// SIMD-ONLY0:       if.then53:
+// SIMD-ONLY0-NEXT:    [[TMP24:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP24]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END54]]
+// SIMD-ONLY0:       if.end54:
+// SIMD-ONLY0-NEXT:    [[TMP25:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV55:%.*]] = sext i8 [[TMP25]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP26:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV56:%.*]] = sext i8 [[TMP26]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP57:%.*]] = icmp slt i32 [[CONV55]], [[CONV56]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP57]], label [[IF_THEN59:%.*]], label [[IF_END60:%.*]]
+// SIMD-ONLY0:       if.then59:
+// SIMD-ONLY0-NEXT:    [[TMP27:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP27]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END60]]
+// SIMD-ONLY0:       if.end60:
+// SIMD-ONLY0-NEXT:    [[TMP28:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV61:%.*]] = sext i8 [[TMP28]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP29:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV62:%.*]] = sext i8 [[TMP29]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP63:%.*]] = icmp eq i32 [[CONV61]], [[CONV62]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP63]], label [[COND_TRUE65:%.*]], label [[COND_FALSE67:%.*]]
+// SIMD-ONLY0:       cond.true65:
+// SIMD-ONLY0-NEXT:    [[TMP30:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV66:%.*]] = sext i8 [[TMP30]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END69:%.*]]
+// SIMD-ONLY0:       cond.false67:
+// SIMD-ONLY0-NEXT:    [[TMP31:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV68:%.*]] = sext i8 [[TMP31]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END69]]
+// SIMD-ONLY0:       cond.end69:
+// SIMD-ONLY0-NEXT:    [[COND70:%.*]] = phi i32 [ [[CONV66]], [[COND_TRUE65]] ], [ [[CONV68]], [[COND_FALSE67]] ]
+// SIMD-ONLY0-NEXT:    [[CONV71:%.*]] = trunc i32 [[COND70]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV71]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP32:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV72:%.*]] = sext i8 [[TMP32]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP33:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV73:%.*]] = sext i8 [[TMP33]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP74:%.*]] = icmp eq i32 [[CONV72]], [[CONV73]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP74]], label [[COND_TRUE76:%.*]], label [[COND_FALSE78:%.*]]
+// SIMD-ONLY0:       cond.true76:
+// SIMD-ONLY0-NEXT:    [[TMP34:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV77:%.*]] = sext i8 [[TMP34]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END80:%.*]]
+// SIMD-ONLY0:       cond.false78:
+// SIMD-ONLY0-NEXT:    [[TMP35:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV79:%.*]] = sext i8 [[TMP35]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END80]]
+// SIMD-ONLY0:       cond.end80:
+// SIMD-ONLY0-NEXT:    [[COND81:%.*]] = phi i32 [ [[CONV77]], [[COND_TRUE76]] ], [ [[CONV79]], [[COND_FALSE78]] ]
+// SIMD-ONLY0-NEXT:    [[CONV82:%.*]] = trunc i32 [[COND81]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV82]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP36:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV83:%.*]] = sext i8 [[TMP36]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP37:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV84:%.*]] = sext i8 [[TMP37]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP85:%.*]] = icmp eq i32 [[CONV83]], [[CONV84]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP85]], label [[IF_THEN87:%.*]], label [[IF_END88:%.*]]
+// SIMD-ONLY0:       if.then87:
+// SIMD-ONLY0-NEXT:    [[TMP38:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP38]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END88]]
+// SIMD-ONLY0:       if.end88:
+// SIMD-ONLY0-NEXT:    [[TMP39:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV89:%.*]] = sext i8 [[TMP39]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP40:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV90:%.*]] = sext i8 [[TMP40]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP91:%.*]] = icmp eq i32 [[CONV89]], [[CONV90]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP91]], label [[IF_THEN93:%.*]], label [[IF_END94:%.*]]
+// SIMD-ONLY0:       if.then93:
+// SIMD-ONLY0-NEXT:    [[TMP41:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP41]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END94]]
+// SIMD-ONLY0:       if.end94:
+// SIMD-ONLY0-NEXT:    [[TMP42:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV95:%.*]] = zext i8 [[TMP42]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP43:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV96:%.*]] = zext i8 [[TMP43]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP97:%.*]] = icmp sgt i32 [[CONV95]], [[CONV96]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP97]], label [[COND_TRUE99:%.*]], label [[COND_FALSE101:%.*]]
+// SIMD-ONLY0:       cond.true99:
+// SIMD-ONLY0-NEXT:    [[TMP44:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV100:%.*]] = zext i8 [[TMP44]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END103:%.*]]
+// SIMD-ONLY0:       cond.false101:
+// SIMD-ONLY0-NEXT:    [[TMP45:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV102:%.*]] = zext i8 [[TMP45]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END103]]
+// SIMD-ONLY0:       cond.end103:
+// SIMD-ONLY0-NEXT:    [[COND104:%.*]] = phi i32 [ [[CONV100]], [[COND_TRUE99]] ], [ [[CONV102]], [[COND_FALSE101]] ]
+// SIMD-ONLY0-NEXT:    [[CONV105:%.*]] = trunc i32 [[COND104]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV105]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP46:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV106:%.*]] = zext i8 [[TMP46]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP47:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV107:%.*]] = zext i8 [[TMP47]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP108:%.*]] = icmp slt i32 [[CONV106]], [[CONV107]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP108]], label [[COND_TRUE110:%.*]], label [[COND_FALSE112:%.*]]
+// SIMD-ONLY0:       cond.true110:
+// SIMD-ONLY0-NEXT:    [[TMP48:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV111:%.*]] = zext i8 [[TMP48]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END114:%.*]]
+// SIMD-ONLY0:       cond.false112:
+// SIMD-ONLY0-NEXT:    [[TMP49:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV113:%.*]] = zext i8 [[TMP49]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END114]]
+// SIMD-ONLY0:       cond.end114:
+// SIMD-ONLY0-NEXT:    [[COND115:%.*]] = phi i32 [ [[CONV111]], [[COND_TRUE110]] ], [ [[CONV113]], [[COND_FALSE112]] ]
+// SIMD-ONLY0-NEXT:    [[CONV116:%.*]] = trunc i32 [[COND115]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV116]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP50:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV117:%.*]] = zext i8 [[TMP50]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP51:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV118:%.*]] = zext i8 [[TMP51]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP119:%.*]] = icmp sgt i32 [[CONV117]], [[CONV118]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP119]], label [[COND_TRUE121:%.*]], label [[COND_FALSE123:%.*]]
+// SIMD-ONLY0:       cond.true121:
+// SIMD-ONLY0-NEXT:    [[TMP52:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV122:%.*]] = zext i8 [[TMP52]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END125:%.*]]
+// SIMD-ONLY0:       cond.false123:
+// SIMD-ONLY0-NEXT:    [[TMP53:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV124:%.*]] = zext i8 [[TMP53]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END125]]
+// SIMD-ONLY0:       cond.end125:
+// SIMD-ONLY0-NEXT:    [[COND126:%.*]] = phi i32 [ [[CONV122]], [[COND_TRUE121]] ], [ [[CONV124]], [[COND_FALSE123]] ]
+// SIMD-ONLY0-NEXT:    [[CONV127:%.*]] = trunc i32 [[COND126]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV127]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP54:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV128:%.*]] = zext i8 [[TMP54]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP55:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV129:%.*]] = zext i8 [[TMP55]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP130:%.*]] = icmp slt i32 [[CONV128]], [[CONV129]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP130]], label [[COND_TRUE132:%.*]], label [[COND_FALSE134:%.*]]
+// SIMD-ONLY0:       cond.true132:
+// SIMD-ONLY0-NEXT:    [[TMP56:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV133:%.*]] = zext i8 [[TMP56]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END136:%.*]]
+// SIMD-ONLY0:       cond.false134:
+// SIMD-ONLY0-NEXT:    [[TMP57:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV135:%.*]] = zext i8 [[TMP57]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END136]]
+// SIMD-ONLY0:       cond.end136:
+// SIMD-ONLY0-NEXT:    [[COND137:%.*]] = phi i32 [ [[CONV133]], [[COND_TRUE132]] ], [ [[CONV135]], [[COND_FALSE134]] ]
+// SIMD-ONLY0-NEXT:    [[CONV138:%.*]] = trunc i32 [[COND137]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV138]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP58:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV139:%.*]] = zext i8 [[TMP58]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP59:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV140:%.*]] = zext i8 [[TMP59]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP141:%.*]] = icmp sgt i32 [[CONV139]], [[CONV140]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP141]], label [[IF_THEN143:%.*]], label [[IF_END144:%.*]]
+// SIMD-ONLY0:       if.then143:
+// SIMD-ONLY0-NEXT:    [[TMP60:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP60]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END144]]
+// SIMD-ONLY0:       if.end144:
+// SIMD-ONLY0-NEXT:    [[TMP61:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV145:%.*]] = zext i8 [[TMP61]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP62:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV146:%.*]] = zext i8 [[TMP62]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP147:%.*]] = icmp slt i32 [[CONV145]], [[CONV146]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP147]], label [[IF_THEN149:%.*]], label [[IF_END150:%.*]]
+// SIMD-ONLY0:       if.then149:
+// SIMD-ONLY0-NEXT:    [[TMP63:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP63]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END150]]
+// SIMD-ONLY0:       if.end150:
+// SIMD-ONLY0-NEXT:    [[TMP64:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV151:%.*]] = zext i8 [[TMP64]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP65:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV152:%.*]] = zext i8 [[TMP65]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP153:%.*]] = icmp sgt i32 [[CONV151]], [[CONV152]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP153]], label [[IF_THEN155:%.*]], label [[IF_END156:%.*]]
+// SIMD-ONLY0:       if.then155:
+// SIMD-ONLY0-NEXT:    [[TMP66:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP66]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END156]]
+// SIMD-ONLY0:       if.end156:
+// SIMD-ONLY0-NEXT:    [[TMP67:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV157:%.*]] = zext i8 [[TMP67]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP68:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV158:%.*]] = zext i8 [[TMP68]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP159:%.*]] = icmp slt i32 [[CONV157]], [[CONV158]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP159]], label [[IF_THEN161:%.*]], label [[IF_END162:%.*]]
+// SIMD-ONLY0:       if.then161:
+// SIMD-ONLY0-NEXT:    [[TMP69:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP69]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END162]]
+// SIMD-ONLY0:       if.end162:
+// SIMD-ONLY0-NEXT:    [[TMP70:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV163:%.*]] = zext i8 [[TMP70]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP71:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV164:%.*]] = zext i8 [[TMP71]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP165:%.*]] = icmp eq i32 [[CONV163]], [[CONV164]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP165]], label [[COND_TRUE167:%.*]], label [[COND_FALSE169:%.*]]
+// SIMD-ONLY0:       cond.true167:
+// SIMD-ONLY0-NEXT:    [[TMP72:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV168:%.*]] = zext i8 [[TMP72]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END171:%.*]]
+// SIMD-ONLY0:       cond.false169:
+// SIMD-ONLY0-NEXT:    [[TMP73:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV170:%.*]] = zext i8 [[TMP73]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END171]]
+// SIMD-ONLY0:       cond.end171:
+// SIMD-ONLY0-NEXT:    [[COND172:%.*]] = phi i32 [ [[CONV168]], [[COND_TRUE167]] ], [ [[CONV170]], [[COND_FALSE169]] ]
+// SIMD-ONLY0-NEXT:    [[CONV173:%.*]] = trunc i32 [[COND172]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV173]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP74:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV174:%.*]] = zext i8 [[TMP74]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP75:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV175:%.*]] = zext i8 [[TMP75]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP176:%.*]] = icmp eq i32 [[CONV174]], [[CONV175]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP176]], label [[COND_TRUE178:%.*]], label [[COND_FALSE180:%.*]]
+// SIMD-ONLY0:       cond.true178:
+// SIMD-ONLY0-NEXT:    [[TMP76:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV179:%.*]] = zext i8 [[TMP76]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END182:%.*]]
+// SIMD-ONLY0:       cond.false180:
+// SIMD-ONLY0-NEXT:    [[TMP77:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV181:%.*]] = zext i8 [[TMP77]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END182]]
+// SIMD-ONLY0:       cond.end182:
+// SIMD-ONLY0-NEXT:    [[COND183:%.*]] = phi i32 [ [[CONV179]], [[COND_TRUE178]] ], [ [[CONV181]], [[COND_FALSE180]] ]
+// SIMD-ONLY0-NEXT:    [[CONV184:%.*]] = trunc i32 [[COND183]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV184]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP78:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV185:%.*]] = zext i8 [[TMP78]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP79:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV186:%.*]] = zext i8 [[TMP79]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP187:%.*]] = icmp eq i32 [[CONV185]], [[CONV186]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP187]], label [[IF_THEN189:%.*]], label [[IF_END190:%.*]]
+// SIMD-ONLY0:       if.then189:
+// SIMD-ONLY0-NEXT:    [[TMP80:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP80]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END190]]
+// SIMD-ONLY0:       if.end190:
+// SIMD-ONLY0-NEXT:    [[TMP81:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV191:%.*]] = zext i8 [[TMP81]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP82:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV192:%.*]] = zext i8 [[TMP82]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP193:%.*]] = icmp eq i32 [[CONV191]], [[CONV192]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP193]], label [[IF_THEN195:%.*]], label [[IF_END196:%.*]]
+// SIMD-ONLY0:       if.then195:
+// SIMD-ONLY0-NEXT:    [[TMP83:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP83]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END196]]
+// SIMD-ONLY0:       if.end196:
+// SIMD-ONLY0-NEXT:    [[TMP84:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV197:%.*]] = sext i8 [[TMP84]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP85:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV198:%.*]] = sext i8 [[TMP85]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP199:%.*]] = icmp sgt i32 [[CONV197]], [[CONV198]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP199]], label [[COND_TRUE201:%.*]], label [[COND_FALSE203:%.*]]
+// SIMD-ONLY0:       cond.true201:
+// SIMD-ONLY0-NEXT:    [[TMP86:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV202:%.*]] = sext i8 [[TMP86]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END205:%.*]]
+// SIMD-ONLY0:       cond.false203:
+// SIMD-ONLY0-NEXT:    [[TMP87:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV204:%.*]] = sext i8 [[TMP87]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END205]]
+// SIMD-ONLY0:       cond.end205:
+// SIMD-ONLY0-NEXT:    [[COND206:%.*]] = phi i32 [ [[CONV202]], [[COND_TRUE201]] ], [ [[CONV204]], [[COND_FALSE203]] ]
+// SIMD-ONLY0-NEXT:    [[CONV207:%.*]] = trunc i32 [[COND206]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV207]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP88:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV208:%.*]] = sext i8 [[TMP88]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP89:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV209:%.*]] = sext i8 [[TMP89]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP210:%.*]] = icmp slt i32 [[CONV208]], [[CONV209]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP210]], label [[COND_TRUE212:%.*]], label [[COND_FALSE214:%.*]]
+// SIMD-ONLY0:       cond.true212:
+// SIMD-ONLY0-NEXT:    [[TMP90:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV213:%.*]] = sext i8 [[TMP90]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END216:%.*]]
+// SIMD-ONLY0:       cond.false214:
+// SIMD-ONLY0-NEXT:    [[TMP91:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV215:%.*]] = sext i8 [[TMP91]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END216]]
+// SIMD-ONLY0:       cond.end216:
+// SIMD-ONLY0-NEXT:    [[COND217:%.*]] = phi i32 [ [[CONV213]], [[COND_TRUE212]] ], [ [[CONV215]], [[COND_FALSE214]] ]
+// SIMD-ONLY0-NEXT:    [[CONV218:%.*]] = trunc i32 [[COND217]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV218]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP92:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV219:%.*]] = sext i8 [[TMP92]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP93:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV220:%.*]] = sext i8 [[TMP93]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP221:%.*]] = icmp sgt i32 [[CONV219]], [[CONV220]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP221]], label [[COND_TRUE223:%.*]], label [[COND_FALSE225:%.*]]
+// SIMD-ONLY0:       cond.true223:
+// SIMD-ONLY0-NEXT:    [[TMP94:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV224:%.*]] = sext i8 [[TMP94]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END227:%.*]]
+// SIMD-ONLY0:       cond.false225:
+// SIMD-ONLY0-NEXT:    [[TMP95:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV226:%.*]] = sext i8 [[TMP95]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END227]]
+// SIMD-ONLY0:       cond.end227:
+// SIMD-ONLY0-NEXT:    [[COND228:%.*]] = phi i32 [ [[CONV224]], [[COND_TRUE223]] ], [ [[CONV226]], [[COND_FALSE225]] ]
+// SIMD-ONLY0-NEXT:    [[CONV229:%.*]] = trunc i32 [[COND228]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV229]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP96:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV230:%.*]] = sext i8 [[TMP96]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP97:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV231:%.*]] = sext i8 [[TMP97]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP232:%.*]] = icmp slt i32 [[CONV230]], [[CONV231]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP232]], label [[COND_TRUE234:%.*]], label [[COND_FALSE236:%.*]]
+// SIMD-ONLY0:       cond.true234:
+// SIMD-ONLY0-NEXT:    [[TMP98:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV235:%.*]] = sext i8 [[TMP98]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END238:%.*]]
+// SIMD-ONLY0:       cond.false236:
+// SIMD-ONLY0-NEXT:    [[TMP99:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV237:%.*]] = sext i8 [[TMP99]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END238]]
+// SIMD-ONLY0:       cond.end238:
+// SIMD-ONLY0-NEXT:    [[COND239:%.*]] = phi i32 [ [[CONV235]], [[COND_TRUE234]] ], [ [[CONV237]], [[COND_FALSE236]] ]
+// SIMD-ONLY0-NEXT:    [[CONV240:%.*]] = trunc i32 [[COND239]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV240]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP100:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV241:%.*]] = sext i8 [[TMP100]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP101:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV242:%.*]] = sext i8 [[TMP101]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP243:%.*]] = icmp sgt i32 [[CONV241]], [[CONV242]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP243]], label [[IF_THEN245:%.*]], label [[IF_END246:%.*]]
+// SIMD-ONLY0:       if.then245:
+// SIMD-ONLY0-NEXT:    [[TMP102:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP102]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END246]]
+// SIMD-ONLY0:       if.end246:
+// SIMD-ONLY0-NEXT:    [[TMP103:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV247:%.*]] = sext i8 [[TMP103]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP104:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV248:%.*]] = sext i8 [[TMP104]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP249:%.*]] = icmp slt i32 [[CONV247]], [[CONV248]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP249]], label [[IF_THEN251:%.*]], label [[IF_END252:%.*]]
+// SIMD-ONLY0:       if.then251:
+// SIMD-ONLY0-NEXT:    [[TMP105:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP105]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END252]]
+// SIMD-ONLY0:       if.end252:
+// SIMD-ONLY0-NEXT:    [[TMP106:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV253:%.*]] = sext i8 [[TMP106]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP107:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV254:%.*]] = sext i8 [[TMP107]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP255:%.*]] = icmp sgt i32 [[CONV253]], [[CONV254]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP255]], label [[IF_THEN257:%.*]], label [[IF_END258:%.*]]
+// SIMD-ONLY0:       if.then257:
+// SIMD-ONLY0-NEXT:    [[TMP108:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP108]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END258]]
+// SIMD-ONLY0:       if.end258:
+// SIMD-ONLY0-NEXT:    [[TMP109:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV259:%.*]] = sext i8 [[TMP109]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP110:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV260:%.*]] = sext i8 [[TMP110]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP261:%.*]] = icmp slt i32 [[CONV259]], [[CONV260]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP261]], label [[IF_THEN263:%.*]], label [[IF_END264:%.*]]
+// SIMD-ONLY0:       if.then263:
+// SIMD-ONLY0-NEXT:    [[TMP111:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP111]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END264]]
+// SIMD-ONLY0:       if.end264:
+// SIMD-ONLY0-NEXT:    [[TMP112:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV265:%.*]] = sext i8 [[TMP112]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP113:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV266:%.*]] = sext i8 [[TMP113]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP267:%.*]] = icmp eq i32 [[CONV265]], [[CONV266]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP267]], label [[COND_TRUE269:%.*]], label [[COND_FALSE271:%.*]]
+// SIMD-ONLY0:       cond.true269:
+// SIMD-ONLY0-NEXT:    [[TMP114:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV270:%.*]] = sext i8 [[TMP114]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END273:%.*]]
+// SIMD-ONLY0:       cond.false271:
+// SIMD-ONLY0-NEXT:    [[TMP115:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV272:%.*]] = sext i8 [[TMP115]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END273]]
+// SIMD-ONLY0:       cond.end273:
+// SIMD-ONLY0-NEXT:    [[COND274:%.*]] = phi i32 [ [[CONV270]], [[COND_TRUE269]] ], [ [[CONV272]], [[COND_FALSE271]] ]
+// SIMD-ONLY0-NEXT:    [[CONV275:%.*]] = trunc i32 [[COND274]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV275]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP116:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV276:%.*]] = sext i8 [[TMP116]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP117:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV277:%.*]] = sext i8 [[TMP117]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP278:%.*]] = icmp eq i32 [[CONV276]], [[CONV277]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP278]], label [[COND_TRUE280:%.*]], label [[COND_FALSE282:%.*]]
+// SIMD-ONLY0:       cond.true280:
+// SIMD-ONLY0-NEXT:    [[TMP118:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV281:%.*]] = sext i8 [[TMP118]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END284:%.*]]
+// SIMD-ONLY0:       cond.false282:
+// SIMD-ONLY0-NEXT:    [[TMP119:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV283:%.*]] = sext i8 [[TMP119]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END284]]
+// SIMD-ONLY0:       cond.end284:
+// SIMD-ONLY0-NEXT:    [[COND285:%.*]] = phi i32 [ [[CONV281]], [[COND_TRUE280]] ], [ [[CONV283]], [[COND_FALSE282]] ]
+// SIMD-ONLY0-NEXT:    [[CONV286:%.*]] = trunc i32 [[COND285]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV286]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP120:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV287:%.*]] = sext i8 [[TMP120]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP121:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV288:%.*]] = sext i8 [[TMP121]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP289:%.*]] = icmp eq i32 [[CONV287]], [[CONV288]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP289]], label [[IF_THEN291:%.*]], label [[IF_END292:%.*]]
+// SIMD-ONLY0:       if.then291:
+// SIMD-ONLY0-NEXT:    [[TMP122:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP122]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END292]]
+// SIMD-ONLY0:       if.end292:
+// SIMD-ONLY0-NEXT:    [[TMP123:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV293:%.*]] = sext i8 [[TMP123]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP124:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV294:%.*]] = sext i8 [[TMP124]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP295:%.*]] = icmp eq i32 [[CONV293]], [[CONV294]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP295]], label [[IF_THEN297:%.*]], label [[IF_END298:%.*]]
+// SIMD-ONLY0:       if.then297:
+// SIMD-ONLY0-NEXT:    [[TMP125:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP125]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END298]]
+// SIMD-ONLY0:       if.end298:
+// SIMD-ONLY0-NEXT:    [[TMP126:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV299:%.*]] = zext i8 [[TMP126]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP127:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV300:%.*]] = zext i8 [[TMP127]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP301:%.*]] = icmp sgt i32 [[CONV299]], [[CONV300]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP301]], label [[COND_TRUE303:%.*]], label [[COND_FALSE305:%.*]]
+// SIMD-ONLY0:       cond.true303:
+// SIMD-ONLY0-NEXT:    [[TMP128:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV304:%.*]] = zext i8 [[TMP128]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END307:%.*]]
+// SIMD-ONLY0:       cond.false305:
+// SIMD-ONLY0-NEXT:    [[TMP129:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV306:%.*]] = zext i8 [[TMP129]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END307]]
+// SIMD-ONLY0:       cond.end307:
+// SIMD-ONLY0-NEXT:    [[COND308:%.*]] = phi i32 [ [[CONV304]], [[COND_TRUE303]] ], [ [[CONV306]], [[COND_FALSE305]] ]
+// SIMD-ONLY0-NEXT:    [[CONV309:%.*]] = trunc i32 [[COND308]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV309]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP130:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV310:%.*]] = zext i8 [[TMP130]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP131:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV311:%.*]] = zext i8 [[TMP131]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP312:%.*]] = icmp slt i32 [[CONV310]], [[CONV311]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP312]], label [[COND_TRUE314:%.*]], label [[COND_FALSE316:%.*]]
+// SIMD-ONLY0:       cond.true314:
+// SIMD-ONLY0-NEXT:    [[TMP132:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV315:%.*]] = zext i8 [[TMP132]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END318:%.*]]
+// SIMD-ONLY0:       cond.false316:
+// SIMD-ONLY0-NEXT:    [[TMP133:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV317:%.*]] = zext i8 [[TMP133]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END318]]
+// SIMD-ONLY0:       cond.end318:
+// SIMD-ONLY0-NEXT:    [[COND319:%.*]] = phi i32 [ [[CONV315]], [[COND_TRUE314]] ], [ [[CONV317]], [[COND_FALSE316]] ]
+// SIMD-ONLY0-NEXT:    [[CONV320:%.*]] = trunc i32 [[COND319]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV320]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP134:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV321:%.*]] = zext i8 [[TMP134]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP135:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV322:%.*]] = zext i8 [[TMP135]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP323:%.*]] = icmp sgt i32 [[CONV321]], [[CONV322]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP323]], label [[COND_TRUE325:%.*]], label [[COND_FALSE327:%.*]]
+// SIMD-ONLY0:       cond.true325:
+// SIMD-ONLY0-NEXT:    [[TMP136:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV326:%.*]] = zext i8 [[TMP136]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END329:%.*]]
+// SIMD-ONLY0:       cond.false327:
+// SIMD-ONLY0-NEXT:    [[TMP137:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV328:%.*]] = zext i8 [[TMP137]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END329]]
+// SIMD-ONLY0:       cond.end329:
+// SIMD-ONLY0-NEXT:    [[COND330:%.*]] = phi i32 [ [[CONV326]], [[COND_TRUE325]] ], [ [[CONV328]], [[COND_FALSE327]] ]
+// SIMD-ONLY0-NEXT:    [[CONV331:%.*]] = trunc i32 [[COND330]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV331]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP138:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV332:%.*]] = zext i8 [[TMP138]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP139:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV333:%.*]] = zext i8 [[TMP139]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP334:%.*]] = icmp slt i32 [[CONV332]], [[CONV333]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP334]], label [[COND_TRUE336:%.*]], label [[COND_FALSE338:%.*]]
+// SIMD-ONLY0:       cond.true336:
+// SIMD-ONLY0-NEXT:    [[TMP140:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV337:%.*]] = zext i8 [[TMP140]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END340:%.*]]
+// SIMD-ONLY0:       cond.false338:
+// SIMD-ONLY0-NEXT:    [[TMP141:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV339:%.*]] = zext i8 [[TMP141]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END340]]
+// SIMD-ONLY0:       cond.end340:
+// SIMD-ONLY0-NEXT:    [[COND341:%.*]] = phi i32 [ [[CONV337]], [[COND_TRUE336]] ], [ [[CONV339]], [[COND_FALSE338]] ]
+// SIMD-ONLY0-NEXT:    [[CONV342:%.*]] = trunc i32 [[COND341]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV342]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP142:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV343:%.*]] = zext i8 [[TMP142]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP143:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV344:%.*]] = zext i8 [[TMP143]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP345:%.*]] = icmp sgt i32 [[CONV343]], [[CONV344]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP345]], label [[IF_THEN347:%.*]], label [[IF_END348:%.*]]
+// SIMD-ONLY0:       if.then347:
+// SIMD-ONLY0-NEXT:    [[TMP144:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP144]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END348]]
+// SIMD-ONLY0:       if.end348:
+// SIMD-ONLY0-NEXT:    [[TMP145:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV349:%.*]] = zext i8 [[TMP145]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP146:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV350:%.*]] = zext i8 [[TMP146]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP351:%.*]] = icmp slt i32 [[CONV349]], [[CONV350]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP351]], label [[IF_THEN353:%.*]], label [[IF_END354:%.*]]
+// SIMD-ONLY0:       if.then353:
+// SIMD-ONLY0-NEXT:    [[TMP147:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP147]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END354]]
+// SIMD-ONLY0:       if.end354:
+// SIMD-ONLY0-NEXT:    [[TMP148:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV355:%.*]] = zext i8 [[TMP148]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP149:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV356:%.*]] = zext i8 [[TMP149]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP357:%.*]] = icmp sgt i32 [[CONV355]], [[CONV356]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP357]], label [[IF_THEN359:%.*]], label [[IF_END360:%.*]]
+// SIMD-ONLY0:       if.then359:
+// SIMD-ONLY0-NEXT:    [[TMP150:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP150]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END360]]
+// SIMD-ONLY0:       if.end360:
+// SIMD-ONLY0-NEXT:    [[TMP151:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV361:%.*]] = zext i8 [[TMP151]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP152:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV362:%.*]] = zext i8 [[TMP152]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP363:%.*]] = icmp slt i32 [[CONV361]], [[CONV362]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP363]], label [[IF_THEN365:%.*]], label [[IF_END366:%.*]]
+// SIMD-ONLY0:       if.then365:
+// SIMD-ONLY0-NEXT:    [[TMP153:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP153]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END366]]
+// SIMD-ONLY0:       if.end366:
+// SIMD-ONLY0-NEXT:    [[TMP154:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV367:%.*]] = zext i8 [[TMP154]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP155:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV368:%.*]] = zext i8 [[TMP155]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP369:%.*]] = icmp eq i32 [[CONV367]], [[CONV368]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP369]], label [[COND_TRUE371:%.*]], label [[COND_FALSE373:%.*]]
+// SIMD-ONLY0:       cond.true371:
+// SIMD-ONLY0-NEXT:    [[TMP156:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV372:%.*]] = zext i8 [[TMP156]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END375:%.*]]
+// SIMD-ONLY0:       cond.false373:
+// SIMD-ONLY0-NEXT:    [[TMP157:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV374:%.*]] = zext i8 [[TMP157]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END375]]
+// SIMD-ONLY0:       cond.end375:
+// SIMD-ONLY0-NEXT:    [[COND376:%.*]] = phi i32 [ [[CONV372]], [[COND_TRUE371]] ], [ [[CONV374]], [[COND_FALSE373]] ]
+// SIMD-ONLY0-NEXT:    [[CONV377:%.*]] = trunc i32 [[COND376]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV377]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP158:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV378:%.*]] = zext i8 [[TMP158]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP159:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV379:%.*]] = zext i8 [[TMP159]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP380:%.*]] = icmp eq i32 [[CONV378]], [[CONV379]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP380]], label [[COND_TRUE382:%.*]], label [[COND_FALSE384:%.*]]
+// SIMD-ONLY0:       cond.true382:
+// SIMD-ONLY0-NEXT:    [[TMP160:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV383:%.*]] = zext i8 [[TMP160]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END386:%.*]]
+// SIMD-ONLY0:       cond.false384:
+// SIMD-ONLY0-NEXT:    [[TMP161:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV385:%.*]] = zext i8 [[TMP161]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END386]]
+// SIMD-ONLY0:       cond.end386:
+// SIMD-ONLY0-NEXT:    [[COND387:%.*]] = phi i32 [ [[CONV383]], [[COND_TRUE382]] ], [ [[CONV385]], [[COND_FALSE384]] ]
+// SIMD-ONLY0-NEXT:    [[CONV388:%.*]] = trunc i32 [[COND387]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV388]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP162:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV389:%.*]] = zext i8 [[TMP162]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP163:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV390:%.*]] = zext i8 [[TMP163]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP391:%.*]] = icmp eq i32 [[CONV389]], [[CONV390]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP391]], label [[IF_THEN393:%.*]], label [[IF_END394:%.*]]
+// SIMD-ONLY0:       if.then393:
+// SIMD-ONLY0-NEXT:    [[TMP164:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP164]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END394]]
+// SIMD-ONLY0:       if.end394:
+// SIMD-ONLY0-NEXT:    [[TMP165:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV395:%.*]] = zext i8 [[TMP165]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP166:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV396:%.*]] = zext i8 [[TMP166]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP397:%.*]] = icmp eq i32 [[CONV395]], [[CONV396]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP397]], label [[IF_THEN399:%.*]], label [[IF_END400:%.*]]
+// SIMD-ONLY0:       if.then399:
+// SIMD-ONLY0-NEXT:    [[TMP167:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP167]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END400]]
+// SIMD-ONLY0:       if.end400:
+// SIMD-ONLY0-NEXT:    [[TMP168:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV401:%.*]] = sext i8 [[TMP168]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP169:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV402:%.*]] = sext i8 [[TMP169]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP403:%.*]] = icmp sgt i32 [[CONV401]], [[CONV402]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP403]], label [[COND_TRUE405:%.*]], label [[COND_FALSE407:%.*]]
+// SIMD-ONLY0:       cond.true405:
+// SIMD-ONLY0-NEXT:    [[TMP170:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV406:%.*]] = sext i8 [[TMP170]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END409:%.*]]
+// SIMD-ONLY0:       cond.false407:
+// SIMD-ONLY0-NEXT:    [[TMP171:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV408:%.*]] = sext i8 [[TMP171]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END409]]
+// SIMD-ONLY0:       cond.end409:
+// SIMD-ONLY0-NEXT:    [[COND410:%.*]] = phi i32 [ [[CONV406]], [[COND_TRUE405]] ], [ [[CONV408]], [[COND_FALSE407]] ]
+// SIMD-ONLY0-NEXT:    [[CONV411:%.*]] = trunc i32 [[COND410]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV411]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP172:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV412:%.*]] = sext i8 [[TMP172]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP173:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV413:%.*]] = sext i8 [[TMP173]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP414:%.*]] = icmp slt i32 [[CONV412]], [[CONV413]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP414]], label [[COND_TRUE416:%.*]], label [[COND_FALSE418:%.*]]
+// SIMD-ONLY0:       cond.true416:
+// SIMD-ONLY0-NEXT:    [[TMP174:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV417:%.*]] = sext i8 [[TMP174]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END420:%.*]]
+// SIMD-ONLY0:       cond.false418:
+// SIMD-ONLY0-NEXT:    [[TMP175:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV419:%.*]] = sext i8 [[TMP175]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END420]]
+// SIMD-ONLY0:       cond.end420:
+// SIMD-ONLY0-NEXT:    [[COND421:%.*]] = phi i32 [ [[CONV417]], [[COND_TRUE416]] ], [ [[CONV419]], [[COND_FALSE418]] ]
+// SIMD-ONLY0-NEXT:    [[CONV422:%.*]] = trunc i32 [[COND421]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV422]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP176:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV423:%.*]] = sext i8 [[TMP176]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP177:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV424:%.*]] = sext i8 [[TMP177]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP425:%.*]] = icmp sgt i32 [[CONV423]], [[CONV424]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP425]], label [[COND_TRUE427:%.*]], label [[COND_FALSE429:%.*]]
+// SIMD-ONLY0:       cond.true427:
+// SIMD-ONLY0-NEXT:    [[TMP178:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV428:%.*]] = sext i8 [[TMP178]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END431:%.*]]
+// SIMD-ONLY0:       cond.false429:
+// SIMD-ONLY0-NEXT:    [[TMP179:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV430:%.*]] = sext i8 [[TMP179]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END431]]
+// SIMD-ONLY0:       cond.end431:
+// SIMD-ONLY0-NEXT:    [[COND432:%.*]] = phi i32 [ [[CONV428]], [[COND_TRUE427]] ], [ [[CONV430]], [[COND_FALSE429]] ]
+// SIMD-ONLY0-NEXT:    [[CONV433:%.*]] = trunc i32 [[COND432]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV433]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP180:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV434:%.*]] = sext i8 [[TMP180]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP181:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV435:%.*]] = sext i8 [[TMP181]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP436:%.*]] = icmp slt i32 [[CONV434]], [[CONV435]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP436]], label [[COND_TRUE438:%.*]], label [[COND_FALSE440:%.*]]
+// SIMD-ONLY0:       cond.true438:
+// SIMD-ONLY0-NEXT:    [[TMP182:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV439:%.*]] = sext i8 [[TMP182]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END442:%.*]]
+// SIMD-ONLY0:       cond.false440:
+// SIMD-ONLY0-NEXT:    [[TMP183:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV441:%.*]] = sext i8 [[TMP183]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END442]]
+// SIMD-ONLY0:       cond.end442:
+// SIMD-ONLY0-NEXT:    [[COND443:%.*]] = phi i32 [ [[CONV439]], [[COND_TRUE438]] ], [ [[CONV441]], [[COND_FALSE440]] ]
+// SIMD-ONLY0-NEXT:    [[CONV444:%.*]] = trunc i32 [[COND443]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV444]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP184:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV445:%.*]] = sext i8 [[TMP184]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP185:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV446:%.*]] = sext i8 [[TMP185]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP447:%.*]] = icmp sgt i32 [[CONV445]], [[CONV446]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP447]], label [[IF_THEN449:%.*]], label [[IF_END450:%.*]]
+// SIMD-ONLY0:       if.then449:
+// SIMD-ONLY0-NEXT:    [[TMP186:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP186]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END450]]
+// SIMD-ONLY0:       if.end450:
+// SIMD-ONLY0-NEXT:    [[TMP187:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV451:%.*]] = sext i8 [[TMP187]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP188:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV452:%.*]] = sext i8 [[TMP188]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP453:%.*]] = icmp slt i32 [[CONV451]], [[CONV452]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP453]], label [[IF_THEN455:%.*]], label [[IF_END456:%.*]]
+// SIMD-ONLY0:       if.then455:
+// SIMD-ONLY0-NEXT:    [[TMP189:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP189]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END456]]
+// SIMD-ONLY0:       if.end456:
+// SIMD-ONLY0-NEXT:    [[TMP190:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV457:%.*]] = sext i8 [[TMP190]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP191:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV458:%.*]] = sext i8 [[TMP191]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP459:%.*]] = icmp sgt i32 [[CONV457]], [[CONV458]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP459]], label [[IF_THEN461:%.*]], label [[IF_END462:%.*]]
+// SIMD-ONLY0:       if.then461:
+// SIMD-ONLY0-NEXT:    [[TMP192:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP192]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END462]]
+// SIMD-ONLY0:       if.end462:
+// SIMD-ONLY0-NEXT:    [[TMP193:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV463:%.*]] = sext i8 [[TMP193]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP194:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV464:%.*]] = sext i8 [[TMP194]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP465:%.*]] = icmp slt i32 [[CONV463]], [[CONV464]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP465]], label [[IF_THEN467:%.*]], label [[IF_END468:%.*]]
+// SIMD-ONLY0:       if.then467:
+// SIMD-ONLY0-NEXT:    [[TMP195:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP195]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END468]]
+// SIMD-ONLY0:       if.end468:
+// SIMD-ONLY0-NEXT:    [[TMP196:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV469:%.*]] = sext i8 [[TMP196]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP197:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV470:%.*]] = sext i8 [[TMP197]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP471:%.*]] = icmp eq i32 [[CONV469]], [[CONV470]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP471]], label [[COND_TRUE473:%.*]], label [[COND_FALSE475:%.*]]
+// SIMD-ONLY0:       cond.true473:
+// SIMD-ONLY0-NEXT:    [[TMP198:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV474:%.*]] = sext i8 [[TMP198]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END477:%.*]]
+// SIMD-ONLY0:       cond.false475:
+// SIMD-ONLY0-NEXT:    [[TMP199:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV476:%.*]] = sext i8 [[TMP199]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END477]]
+// SIMD-ONLY0:       cond.end477:
+// SIMD-ONLY0-NEXT:    [[COND478:%.*]] = phi i32 [ [[CONV474]], [[COND_TRUE473]] ], [ [[CONV476]], [[COND_FALSE475]] ]
+// SIMD-ONLY0-NEXT:    [[CONV479:%.*]] = trunc i32 [[COND478]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV479]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP200:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV480:%.*]] = sext i8 [[TMP200]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP201:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV481:%.*]] = sext i8 [[TMP201]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP482:%.*]] = icmp eq i32 [[CONV480]], [[CONV481]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP482]], label [[COND_TRUE484:%.*]], label [[COND_FALSE486:%.*]]
+// SIMD-ONLY0:       cond.true484:
+// SIMD-ONLY0-NEXT:    [[TMP202:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV485:%.*]] = sext i8 [[TMP202]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END488:%.*]]
+// SIMD-ONLY0:       cond.false486:
+// SIMD-ONLY0-NEXT:    [[TMP203:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV487:%.*]] = sext i8 [[TMP203]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END488]]
+// SIMD-ONLY0:       cond.end488:
+// SIMD-ONLY0-NEXT:    [[COND489:%.*]] = phi i32 [ [[CONV485]], [[COND_TRUE484]] ], [ [[CONV487]], [[COND_FALSE486]] ]
+// SIMD-ONLY0-NEXT:    [[CONV490:%.*]] = trunc i32 [[COND489]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV490]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP204:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV491:%.*]] = sext i8 [[TMP204]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP205:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV492:%.*]] = sext i8 [[TMP205]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP493:%.*]] = icmp eq i32 [[CONV491]], [[CONV492]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP493]], label [[IF_THEN495:%.*]], label [[IF_END496:%.*]]
+// SIMD-ONLY0:       if.then495:
+// SIMD-ONLY0-NEXT:    [[TMP206:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP206]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END496]]
+// SIMD-ONLY0:       if.end496:
+// SIMD-ONLY0-NEXT:    [[TMP207:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV497:%.*]] = sext i8 [[TMP207]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP208:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV498:%.*]] = sext i8 [[TMP208]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP499:%.*]] = icmp eq i32 [[CONV497]], [[CONV498]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP499]], label [[IF_THEN501:%.*]], label [[IF_END502:%.*]]
+// SIMD-ONLY0:       if.then501:
+// SIMD-ONLY0-NEXT:    [[TMP209:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP209]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END502]]
+// SIMD-ONLY0:       if.end502:
+// SIMD-ONLY0-NEXT:    [[TMP210:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV503:%.*]] = zext i8 [[TMP210]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP211:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV504:%.*]] = zext i8 [[TMP211]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP505:%.*]] = icmp sgt i32 [[CONV503]], [[CONV504]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP505]], label [[COND_TRUE507:%.*]], label [[COND_FALSE509:%.*]]
+// SIMD-ONLY0:       cond.true507:
+// SIMD-ONLY0-NEXT:    [[TMP212:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV508:%.*]] = zext i8 [[TMP212]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END511:%.*]]
+// SIMD-ONLY0:       cond.false509:
+// SIMD-ONLY0-NEXT:    [[TMP213:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV510:%.*]] = zext i8 [[TMP213]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END511]]
+// SIMD-ONLY0:       cond.end511:
+// SIMD-ONLY0-NEXT:    [[COND512:%.*]] = phi i32 [ [[CONV508]], [[COND_TRUE507]] ], [ [[CONV510]], [[COND_FALSE509]] ]
+// SIMD-ONLY0-NEXT:    [[CONV513:%.*]] = trunc i32 [[COND512]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV513]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP214:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV514:%.*]] = zext i8 [[TMP214]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP215:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV515:%.*]] = zext i8 [[TMP215]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP516:%.*]] = icmp slt i32 [[CONV514]], [[CONV515]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP516]], label [[COND_TRUE518:%.*]], label [[COND_FALSE520:%.*]]
+// SIMD-ONLY0:       cond.true518:
+// SIMD-ONLY0-NEXT:    [[TMP216:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV519:%.*]] = zext i8 [[TMP216]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END522:%.*]]
+// SIMD-ONLY0:       cond.false520:
+// SIMD-ONLY0-NEXT:    [[TMP217:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV521:%.*]] = zext i8 [[TMP217]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END522]]
+// SIMD-ONLY0:       cond.end522:
+// SIMD-ONLY0-NEXT:    [[COND523:%.*]] = phi i32 [ [[CONV519]], [[COND_TRUE518]] ], [ [[CONV521]], [[COND_FALSE520]] ]
+// SIMD-ONLY0-NEXT:    [[CONV524:%.*]] = trunc i32 [[COND523]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV524]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP218:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV525:%.*]] = zext i8 [[TMP218]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP219:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV526:%.*]] = zext i8 [[TMP219]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP527:%.*]] = icmp sgt i32 [[CONV525]], [[CONV526]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP527]], label [[COND_TRUE529:%.*]], label [[COND_FALSE531:%.*]]
+// SIMD-ONLY0:       cond.true529:
+// SIMD-ONLY0-NEXT:    [[TMP220:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV530:%.*]] = zext i8 [[TMP220]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END533:%.*]]
+// SIMD-ONLY0:       cond.false531:
+// SIMD-ONLY0-NEXT:    [[TMP221:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV532:%.*]] = zext i8 [[TMP221]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END533]]
+// SIMD-ONLY0:       cond.end533:
+// SIMD-ONLY0-NEXT:    [[COND534:%.*]] = phi i32 [ [[CONV530]], [[COND_TRUE529]] ], [ [[CONV532]], [[COND_FALSE531]] ]
+// SIMD-ONLY0-NEXT:    [[CONV535:%.*]] = trunc i32 [[COND534]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV535]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP222:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV536:%.*]] = zext i8 [[TMP222]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP223:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV537:%.*]] = zext i8 [[TMP223]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP538:%.*]] = icmp slt i32 [[CONV536]], [[CONV537]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP538]], label [[COND_TRUE540:%.*]], label [[COND_FALSE542:%.*]]
+// SIMD-ONLY0:       cond.true540:
+// SIMD-ONLY0-NEXT:    [[TMP224:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV541:%.*]] = zext i8 [[TMP224]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END544:%.*]]
+// SIMD-ONLY0:       cond.false542:
+// SIMD-ONLY0-NEXT:    [[TMP225:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV543:%.*]] = zext i8 [[TMP225]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END544]]
+// SIMD-ONLY0:       cond.end544:
+// SIMD-ONLY0-NEXT:    [[COND545:%.*]] = phi i32 [ [[CONV541]], [[COND_TRUE540]] ], [ [[CONV543]], [[COND_FALSE542]] ]
+// SIMD-ONLY0-NEXT:    [[CONV546:%.*]] = trunc i32 [[COND545]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV546]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP226:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV547:%.*]] = zext i8 [[TMP226]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP227:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV548:%.*]] = zext i8 [[TMP227]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP549:%.*]] = icmp sgt i32 [[CONV547]], [[CONV548]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP549]], label [[IF_THEN551:%.*]], label [[IF_END552:%.*]]
+// SIMD-ONLY0:       if.then551:
+// SIMD-ONLY0-NEXT:    [[TMP228:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP228]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END552]]
+// SIMD-ONLY0:       if.end552:
+// SIMD-ONLY0-NEXT:    [[TMP229:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV553:%.*]] = zext i8 [[TMP229]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP230:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV554:%.*]] = zext i8 [[TMP230]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP555:%.*]] = icmp slt i32 [[CONV553]], [[CONV554]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP555]], label [[IF_THEN557:%.*]], label [[IF_END558:%.*]]
+// SIMD-ONLY0:       if.then557:
+// SIMD-ONLY0-NEXT:    [[TMP231:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP231]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END558]]
+// SIMD-ONLY0:       if.end558:
+// SIMD-ONLY0-NEXT:    [[TMP232:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV559:%.*]] = zext i8 [[TMP232]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP233:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV560:%.*]] = zext i8 [[TMP233]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP561:%.*]] = icmp sgt i32 [[CONV559]], [[CONV560]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP561]], label [[IF_THEN563:%.*]], label [[IF_END564:%.*]]
+// SIMD-ONLY0:       if.then563:
+// SIMD-ONLY0-NEXT:    [[TMP234:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP234]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END564]]
+// SIMD-ONLY0:       if.end564:
+// SIMD-ONLY0-NEXT:    [[TMP235:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV565:%.*]] = zext i8 [[TMP235]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP236:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV566:%.*]] = zext i8 [[TMP236]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP567:%.*]] = icmp slt i32 [[CONV565]], [[CONV566]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP567]], label [[IF_THEN569:%.*]], label [[IF_END570:%.*]]
+// SIMD-ONLY0:       if.then569:
+// SIMD-ONLY0-NEXT:    [[TMP237:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP237]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END570]]
+// SIMD-ONLY0:       if.end570:
+// SIMD-ONLY0-NEXT:    [[TMP238:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV571:%.*]] = zext i8 [[TMP238]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP239:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV572:%.*]] = zext i8 [[TMP239]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP573:%.*]] = icmp eq i32 [[CONV571]], [[CONV572]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP573]], label [[COND_TRUE575:%.*]], label [[COND_FALSE577:%.*]]
+// SIMD-ONLY0:       cond.true575:
+// SIMD-ONLY0-NEXT:    [[TMP240:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV576:%.*]] = zext i8 [[TMP240]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END579:%.*]]
+// SIMD-ONLY0:       cond.false577:
+// SIMD-ONLY0-NEXT:    [[TMP241:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV578:%.*]] = zext i8 [[TMP241]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END579]]
+// SIMD-ONLY0:       cond.end579:
+// SIMD-ONLY0-NEXT:    [[COND580:%.*]] = phi i32 [ [[CONV576]], [[COND_TRUE575]] ], [ [[CONV578]], [[COND_FALSE577]] ]
+// SIMD-ONLY0-NEXT:    [[CONV581:%.*]] = trunc i32 [[COND580]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV581]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP242:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV582:%.*]] = zext i8 [[TMP242]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP243:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV583:%.*]] = zext i8 [[TMP243]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP584:%.*]] = icmp eq i32 [[CONV582]], [[CONV583]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP584]], label [[COND_TRUE586:%.*]], label [[COND_FALSE588:%.*]]
+// SIMD-ONLY0:       cond.true586:
+// SIMD-ONLY0-NEXT:    [[TMP244:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV587:%.*]] = zext i8 [[TMP244]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END590:%.*]]
+// SIMD-ONLY0:       cond.false588:
+// SIMD-ONLY0-NEXT:    [[TMP245:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV589:%.*]] = zext i8 [[TMP245]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END590]]
+// SIMD-ONLY0:       cond.end590:
+// SIMD-ONLY0-NEXT:    [[COND591:%.*]] = phi i32 [ [[CONV587]], [[COND_TRUE586]] ], [ [[CONV589]], [[COND_FALSE588]] ]
+// SIMD-ONLY0-NEXT:    [[CONV592:%.*]] = trunc i32 [[COND591]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV592]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP246:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV593:%.*]] = zext i8 [[TMP246]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP247:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV594:%.*]] = zext i8 [[TMP247]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP595:%.*]] = icmp eq i32 [[CONV593]], [[CONV594]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP595]], label [[IF_THEN597:%.*]], label [[IF_END598:%.*]]
+// SIMD-ONLY0:       if.then597:
+// SIMD-ONLY0-NEXT:    [[TMP248:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP248]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END598]]
+// SIMD-ONLY0:       if.end598:
+// SIMD-ONLY0-NEXT:    [[TMP249:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV599:%.*]] = zext i8 [[TMP249]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP250:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV600:%.*]] = zext i8 [[TMP250]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP601:%.*]] = icmp eq i32 [[CONV599]], [[CONV600]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP601]], label [[IF_THEN603:%.*]], label [[IF_END604:%.*]]
+// SIMD-ONLY0:       if.then603:
+// SIMD-ONLY0-NEXT:    [[TMP251:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP251]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END604]]
+// SIMD-ONLY0:       if.end604:
+// SIMD-ONLY0-NEXT:    [[TMP252:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV605:%.*]] = sext i8 [[TMP252]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP253:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV606:%.*]] = sext i8 [[TMP253]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP607:%.*]] = icmp sgt i32 [[CONV605]], [[CONV606]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP607]], label [[COND_TRUE609:%.*]], label [[COND_FALSE611:%.*]]
+// SIMD-ONLY0:       cond.true609:
+// SIMD-ONLY0-NEXT:    [[TMP254:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV610:%.*]] = sext i8 [[TMP254]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END613:%.*]]
+// SIMD-ONLY0:       cond.false611:
+// SIMD-ONLY0-NEXT:    [[TMP255:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV612:%.*]] = sext i8 [[TMP255]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END613]]
+// SIMD-ONLY0:       cond.end613:
+// SIMD-ONLY0-NEXT:    [[COND614:%.*]] = phi i32 [ [[CONV610]], [[COND_TRUE609]] ], [ [[CONV612]], [[COND_FALSE611]] ]
+// SIMD-ONLY0-NEXT:    [[CONV615:%.*]] = trunc i32 [[COND614]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV615]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP256:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV616:%.*]] = sext i8 [[TMP256]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP257:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV617:%.*]] = sext i8 [[TMP257]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP618:%.*]] = icmp slt i32 [[CONV616]], [[CONV617]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP618]], label [[COND_TRUE620:%.*]], label [[COND_FALSE622:%.*]]
+// SIMD-ONLY0:       cond.true620:
+// SIMD-ONLY0-NEXT:    [[TMP258:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV621:%.*]] = sext i8 [[TMP258]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END624:%.*]]
+// SIMD-ONLY0:       cond.false622:
+// SIMD-ONLY0-NEXT:    [[TMP259:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV623:%.*]] = sext i8 [[TMP259]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END624]]
+// SIMD-ONLY0:       cond.end624:
+// SIMD-ONLY0-NEXT:    [[COND625:%.*]] = phi i32 [ [[CONV621]], [[COND_TRUE620]] ], [ [[CONV623]], [[COND_FALSE622]] ]
+// SIMD-ONLY0-NEXT:    [[CONV626:%.*]] = trunc i32 [[COND625]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV626]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP260:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV627:%.*]] = sext i8 [[TMP260]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP261:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV628:%.*]] = sext i8 [[TMP261]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP629:%.*]] = icmp sgt i32 [[CONV627]], [[CONV628]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP629]], label [[COND_TRUE631:%.*]], label [[COND_FALSE633:%.*]]
+// SIMD-ONLY0:       cond.true631:
+// SIMD-ONLY0-NEXT:    [[TMP262:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV632:%.*]] = sext i8 [[TMP262]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END635:%.*]]
+// SIMD-ONLY0:       cond.false633:
+// SIMD-ONLY0-NEXT:    [[TMP263:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV634:%.*]] = sext i8 [[TMP263]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END635]]
+// SIMD-ONLY0:       cond.end635:
+// SIMD-ONLY0-NEXT:    [[COND636:%.*]] = phi i32 [ [[CONV632]], [[COND_TRUE631]] ], [ [[CONV634]], [[COND_FALSE633]] ]
+// SIMD-ONLY0-NEXT:    [[CONV637:%.*]] = trunc i32 [[COND636]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV637]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP264:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV638:%.*]] = sext i8 [[TMP264]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP265:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV639:%.*]] = sext i8 [[TMP265]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP640:%.*]] = icmp slt i32 [[CONV638]], [[CONV639]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP640]], label [[COND_TRUE642:%.*]], label [[COND_FALSE644:%.*]]
+// SIMD-ONLY0:       cond.true642:
+// SIMD-ONLY0-NEXT:    [[TMP266:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV643:%.*]] = sext i8 [[TMP266]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END646:%.*]]
+// SIMD-ONLY0:       cond.false644:
+// SIMD-ONLY0-NEXT:    [[TMP267:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV645:%.*]] = sext i8 [[TMP267]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END646]]
+// SIMD-ONLY0:       cond.end646:
+// SIMD-ONLY0-NEXT:    [[COND647:%.*]] = phi i32 [ [[CONV643]], [[COND_TRUE642]] ], [ [[CONV645]], [[COND_FALSE644]] ]
+// SIMD-ONLY0-NEXT:    [[CONV648:%.*]] = trunc i32 [[COND647]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV648]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP268:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV649:%.*]] = sext i8 [[TMP268]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP269:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV650:%.*]] = sext i8 [[TMP269]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP651:%.*]] = icmp sgt i32 [[CONV649]], [[CONV650]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP651]], label [[IF_THEN653:%.*]], label [[IF_END654:%.*]]
+// SIMD-ONLY0:       if.then653:
+// SIMD-ONLY0-NEXT:    [[TMP270:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP270]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END654]]
+// SIMD-ONLY0:       if.end654:
+// SIMD-ONLY0-NEXT:    [[TMP271:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV655:%.*]] = sext i8 [[TMP271]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP272:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV656:%.*]] = sext i8 [[TMP272]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP657:%.*]] = icmp slt i32 [[CONV655]], [[CONV656]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP657]], label [[IF_THEN659:%.*]], label [[IF_END660:%.*]]
+// SIMD-ONLY0:       if.then659:
+// SIMD-ONLY0-NEXT:    [[TMP273:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP273]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END660]]
+// SIMD-ONLY0:       if.end660:
+// SIMD-ONLY0-NEXT:    [[TMP274:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV661:%.*]] = sext i8 [[TMP274]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP275:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV662:%.*]] = sext i8 [[TMP275]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP663:%.*]] = icmp sgt i32 [[CONV661]], [[CONV662]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP663]], label [[IF_THEN665:%.*]], label [[IF_END666:%.*]]
+// SIMD-ONLY0:       if.then665:
+// SIMD-ONLY0-NEXT:    [[TMP276:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP276]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END666]]
+// SIMD-ONLY0:       if.end666:
+// SIMD-ONLY0-NEXT:    [[TMP277:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV667:%.*]] = sext i8 [[TMP277]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP278:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV668:%.*]] = sext i8 [[TMP278]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP669:%.*]] = icmp slt i32 [[CONV667]], [[CONV668]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP669]], label [[IF_THEN671:%.*]], label [[IF_END672:%.*]]
+// SIMD-ONLY0:       if.then671:
+// SIMD-ONLY0-NEXT:    [[TMP279:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP279]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END672]]
+// SIMD-ONLY0:       if.end672:
+// SIMD-ONLY0-NEXT:    [[TMP280:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV673:%.*]] = sext i8 [[TMP280]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP281:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV674:%.*]] = sext i8 [[TMP281]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP675:%.*]] = icmp eq i32 [[CONV673]], [[CONV674]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP675]], label [[COND_TRUE677:%.*]], label [[COND_FALSE679:%.*]]
+// SIMD-ONLY0:       cond.true677:
+// SIMD-ONLY0-NEXT:    [[TMP282:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV678:%.*]] = sext i8 [[TMP282]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END681:%.*]]
+// SIMD-ONLY0:       cond.false679:
+// SIMD-ONLY0-NEXT:    [[TMP283:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV680:%.*]] = sext i8 [[TMP283]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END681]]
+// SIMD-ONLY0:       cond.end681:
+// SIMD-ONLY0-NEXT:    [[COND682:%.*]] = phi i32 [ [[CONV678]], [[COND_TRUE677]] ], [ [[CONV680]], [[COND_FALSE679]] ]
+// SIMD-ONLY0-NEXT:    [[CONV683:%.*]] = trunc i32 [[COND682]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV683]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP284:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV684:%.*]] = sext i8 [[TMP284]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP285:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV685:%.*]] = sext i8 [[TMP285]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP686:%.*]] = icmp eq i32 [[CONV684]], [[CONV685]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP686]], label [[COND_TRUE688:%.*]], label [[COND_FALSE690:%.*]]
+// SIMD-ONLY0:       cond.true688:
+// SIMD-ONLY0-NEXT:    [[TMP286:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV689:%.*]] = sext i8 [[TMP286]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END692:%.*]]
+// SIMD-ONLY0:       cond.false690:
+// SIMD-ONLY0-NEXT:    [[TMP287:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV691:%.*]] = sext i8 [[TMP287]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END692]]
+// SIMD-ONLY0:       cond.end692:
+// SIMD-ONLY0-NEXT:    [[COND693:%.*]] = phi i32 [ [[CONV689]], [[COND_TRUE688]] ], [ [[CONV691]], [[COND_FALSE690]] ]
+// SIMD-ONLY0-NEXT:    [[CONV694:%.*]] = trunc i32 [[COND693]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV694]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP288:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV695:%.*]] = sext i8 [[TMP288]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP289:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV696:%.*]] = sext i8 [[TMP289]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP697:%.*]] = icmp eq i32 [[CONV695]], [[CONV696]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP697]], label [[IF_THEN699:%.*]], label [[IF_END700:%.*]]
+// SIMD-ONLY0:       if.then699:
+// SIMD-ONLY0-NEXT:    [[TMP290:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP290]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END700]]
+// SIMD-ONLY0:       if.end700:
+// SIMD-ONLY0-NEXT:    [[TMP291:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV701:%.*]] = sext i8 [[TMP291]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP292:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV702:%.*]] = sext i8 [[TMP292]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP703:%.*]] = icmp eq i32 [[CONV701]], [[CONV702]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP703]], label [[IF_THEN705:%.*]], label [[IF_END706:%.*]]
+// SIMD-ONLY0:       if.then705:
+// SIMD-ONLY0-NEXT:    [[TMP293:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP293]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END706]]
+// SIMD-ONLY0:       if.end706:
+// SIMD-ONLY0-NEXT:    [[TMP294:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV707:%.*]] = zext i8 [[TMP294]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP295:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV708:%.*]] = zext i8 [[TMP295]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP709:%.*]] = icmp sgt i32 [[CONV707]], [[CONV708]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP709]], label [[COND_TRUE711:%.*]], label [[COND_FALSE713:%.*]]
+// SIMD-ONLY0:       cond.true711:
+// SIMD-ONLY0-NEXT:    [[TMP296:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV712:%.*]] = zext i8 [[TMP296]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END715:%.*]]
+// SIMD-ONLY0:       cond.false713:
+// SIMD-ONLY0-NEXT:    [[TMP297:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV714:%.*]] = zext i8 [[TMP297]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END715]]
+// SIMD-ONLY0:       cond.end715:
+// SIMD-ONLY0-NEXT:    [[COND716:%.*]] = phi i32 [ [[CONV712]], [[COND_TRUE711]] ], [ [[CONV714]], [[COND_FALSE713]] ]
+// SIMD-ONLY0-NEXT:    [[CONV717:%.*]] = trunc i32 [[COND716]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV717]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP298:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV718:%.*]] = zext i8 [[TMP298]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP299:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV719:%.*]] = zext i8 [[TMP299]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP720:%.*]] = icmp slt i32 [[CONV718]], [[CONV719]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP720]], label [[COND_TRUE722:%.*]], label [[COND_FALSE724:%.*]]
+// SIMD-ONLY0:       cond.true722:
+// SIMD-ONLY0-NEXT:    [[TMP300:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV723:%.*]] = zext i8 [[TMP300]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END726:%.*]]
+// SIMD-ONLY0:       cond.false724:
+// SIMD-ONLY0-NEXT:    [[TMP301:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV725:%.*]] = zext i8 [[TMP301]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END726]]
+// SIMD-ONLY0:       cond.end726:
+// SIMD-ONLY0-NEXT:    [[COND727:%.*]] = phi i32 [ [[CONV723]], [[COND_TRUE722]] ], [ [[CONV725]], [[COND_FALSE724]] ]
+// SIMD-ONLY0-NEXT:    [[CONV728:%.*]] = trunc i32 [[COND727]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV728]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP302:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV729:%.*]] = zext i8 [[TMP302]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP303:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV730:%.*]] = zext i8 [[TMP303]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP731:%.*]] = icmp sgt i32 [[CONV729]], [[CONV730]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP731]], label [[COND_TRUE733:%.*]], label [[COND_FALSE735:%.*]]
+// SIMD-ONLY0:       cond.true733:
+// SIMD-ONLY0-NEXT:    [[TMP304:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV734:%.*]] = zext i8 [[TMP304]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END737:%.*]]
+// SIMD-ONLY0:       cond.false735:
+// SIMD-ONLY0-NEXT:    [[TMP305:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV736:%.*]] = zext i8 [[TMP305]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END737]]
+// SIMD-ONLY0:       cond.end737:
+// SIMD-ONLY0-NEXT:    [[COND738:%.*]] = phi i32 [ [[CONV734]], [[COND_TRUE733]] ], [ [[CONV736]], [[COND_FALSE735]] ]
+// SIMD-ONLY0-NEXT:    [[CONV739:%.*]] = trunc i32 [[COND738]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV739]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP306:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV740:%.*]] = zext i8 [[TMP306]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP307:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV741:%.*]] = zext i8 [[TMP307]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP742:%.*]] = icmp slt i32 [[CONV740]], [[CONV741]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP742]], label [[COND_TRUE744:%.*]], label [[COND_FALSE746:%.*]]
+// SIMD-ONLY0:       cond.true744:
+// SIMD-ONLY0-NEXT:    [[TMP308:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV745:%.*]] = zext i8 [[TMP308]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END748:%.*]]
+// SIMD-ONLY0:       cond.false746:
+// SIMD-ONLY0-NEXT:    [[TMP309:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV747:%.*]] = zext i8 [[TMP309]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END748]]
+// SIMD-ONLY0:       cond.end748:
+// SIMD-ONLY0-NEXT:    [[COND749:%.*]] = phi i32 [ [[CONV745]], [[COND_TRUE744]] ], [ [[CONV747]], [[COND_FALSE746]] ]
+// SIMD-ONLY0-NEXT:    [[CONV750:%.*]] = trunc i32 [[COND749]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV750]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP310:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV751:%.*]] = zext i8 [[TMP310]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP311:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV752:%.*]] = zext i8 [[TMP311]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP753:%.*]] = icmp sgt i32 [[CONV751]], [[CONV752]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP753]], label [[IF_THEN755:%.*]], label [[IF_END756:%.*]]
+// SIMD-ONLY0:       if.then755:
+// SIMD-ONLY0-NEXT:    [[TMP312:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP312]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END756]]
+// SIMD-ONLY0:       if.end756:
+// SIMD-ONLY0-NEXT:    [[TMP313:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV757:%.*]] = zext i8 [[TMP313]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP314:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV758:%.*]] = zext i8 [[TMP314]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP759:%.*]] = icmp slt i32 [[CONV757]], [[CONV758]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP759]], label [[IF_THEN761:%.*]], label [[IF_END762:%.*]]
+// SIMD-ONLY0:       if.then761:
+// SIMD-ONLY0-NEXT:    [[TMP315:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP315]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END762]]
+// SIMD-ONLY0:       if.end762:
+// SIMD-ONLY0-NEXT:    [[TMP316:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV763:%.*]] = zext i8 [[TMP316]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP317:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV764:%.*]] = zext i8 [[TMP317]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP765:%.*]] = icmp sgt i32 [[CONV763]], [[CONV764]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP765]], label [[IF_THEN767:%.*]], label [[IF_END768:%.*]]
+// SIMD-ONLY0:       if.then767:
+// SIMD-ONLY0-NEXT:    [[TMP318:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP318]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END768]]
+// SIMD-ONLY0:       if.end768:
+// SIMD-ONLY0-NEXT:    [[TMP319:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV769:%.*]] = zext i8 [[TMP319]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP320:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV770:%.*]] = zext i8 [[TMP320]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP771:%.*]] = icmp slt i32 [[CONV769]], [[CONV770]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP771]], label [[IF_THEN773:%.*]], label [[IF_END774:%.*]]
+// SIMD-ONLY0:       if.then773:
+// SIMD-ONLY0-NEXT:    [[TMP321:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP321]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END774]]
+// SIMD-ONLY0:       if.end774:
+// SIMD-ONLY0-NEXT:    [[TMP322:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV775:%.*]] = zext i8 [[TMP322]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP323:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV776:%.*]] = zext i8 [[TMP323]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP777:%.*]] = icmp eq i32 [[CONV775]], [[CONV776]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP777]], label [[COND_TRUE779:%.*]], label [[COND_FALSE781:%.*]]
+// SIMD-ONLY0:       cond.true779:
+// SIMD-ONLY0-NEXT:    [[TMP324:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV780:%.*]] = zext i8 [[TMP324]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END783:%.*]]
+// SIMD-ONLY0:       cond.false781:
+// SIMD-ONLY0-NEXT:    [[TMP325:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV782:%.*]] = zext i8 [[TMP325]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END783]]
+// SIMD-ONLY0:       cond.end783:
+// SIMD-ONLY0-NEXT:    [[COND784:%.*]] = phi i32 [ [[CONV780]], [[COND_TRUE779]] ], [ [[CONV782]], [[COND_FALSE781]] ]
+// SIMD-ONLY0-NEXT:    [[CONV785:%.*]] = trunc i32 [[COND784]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV785]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP326:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV786:%.*]] = zext i8 [[TMP326]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP327:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV787:%.*]] = zext i8 [[TMP327]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP788:%.*]] = icmp eq i32 [[CONV786]], [[CONV787]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP788]], label [[COND_TRUE790:%.*]], label [[COND_FALSE792:%.*]]
+// SIMD-ONLY0:       cond.true790:
+// SIMD-ONLY0-NEXT:    [[TMP328:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV791:%.*]] = zext i8 [[TMP328]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END794:%.*]]
+// SIMD-ONLY0:       cond.false792:
+// SIMD-ONLY0-NEXT:    [[TMP329:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV793:%.*]] = zext i8 [[TMP329]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END794]]
+// SIMD-ONLY0:       cond.end794:
+// SIMD-ONLY0-NEXT:    [[COND795:%.*]] = phi i32 [ [[CONV791]], [[COND_TRUE790]] ], [ [[CONV793]], [[COND_FALSE792]] ]
+// SIMD-ONLY0-NEXT:    [[CONV796:%.*]] = trunc i32 [[COND795]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV796]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP330:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV797:%.*]] = zext i8 [[TMP330]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP331:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV798:%.*]] = zext i8 [[TMP331]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP799:%.*]] = icmp eq i32 [[CONV797]], [[CONV798]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP799]], label [[IF_THEN801:%.*]], label [[IF_END802:%.*]]
+// SIMD-ONLY0:       if.then801:
+// SIMD-ONLY0-NEXT:    [[TMP332:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP332]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END802]]
+// SIMD-ONLY0:       if.end802:
+// SIMD-ONLY0-NEXT:    [[TMP333:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV803:%.*]] = zext i8 [[TMP333]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP334:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV804:%.*]] = zext i8 [[TMP334]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP805:%.*]] = icmp eq i32 [[CONV803]], [[CONV804]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP805]], label [[IF_THEN807:%.*]], label [[IF_END808:%.*]]
+// SIMD-ONLY0:       if.then807:
+// SIMD-ONLY0-NEXT:    [[TMP335:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP335]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END808]]
+// SIMD-ONLY0:       if.end808:
+// SIMD-ONLY0-NEXT:    [[TMP336:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV809:%.*]] = sext i8 [[TMP336]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP337:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV810:%.*]] = sext i8 [[TMP337]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP811:%.*]] = icmp sgt i32 [[CONV809]], [[CONV810]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP811]], label [[COND_TRUE813:%.*]], label [[COND_FALSE815:%.*]]
+// SIMD-ONLY0:       cond.true813:
+// SIMD-ONLY0-NEXT:    [[TMP338:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV814:%.*]] = sext i8 [[TMP338]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END817:%.*]]
+// SIMD-ONLY0:       cond.false815:
+// SIMD-ONLY0-NEXT:    [[TMP339:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV816:%.*]] = sext i8 [[TMP339]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END817]]
+// SIMD-ONLY0:       cond.end817:
+// SIMD-ONLY0-NEXT:    [[COND818:%.*]] = phi i32 [ [[CONV814]], [[COND_TRUE813]] ], [ [[CONV816]], [[COND_FALSE815]] ]
+// SIMD-ONLY0-NEXT:    [[CONV819:%.*]] = trunc i32 [[COND818]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV819]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP340:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV820:%.*]] = sext i8 [[TMP340]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP341:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV821:%.*]] = sext i8 [[TMP341]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP822:%.*]] = icmp slt i32 [[CONV820]], [[CONV821]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP822]], label [[COND_TRUE824:%.*]], label [[COND_FALSE826:%.*]]
+// SIMD-ONLY0:       cond.true824:
+// SIMD-ONLY0-NEXT:    [[TMP342:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV825:%.*]] = sext i8 [[TMP342]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END828:%.*]]
+// SIMD-ONLY0:       cond.false826:
+// SIMD-ONLY0-NEXT:    [[TMP343:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV827:%.*]] = sext i8 [[TMP343]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END828]]
+// SIMD-ONLY0:       cond.end828:
+// SIMD-ONLY0-NEXT:    [[COND829:%.*]] = phi i32 [ [[CONV825]], [[COND_TRUE824]] ], [ [[CONV827]], [[COND_FALSE826]] ]
+// SIMD-ONLY0-NEXT:    [[CONV830:%.*]] = trunc i32 [[COND829]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV830]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP344:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV831:%.*]] = sext i8 [[TMP344]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP345:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV832:%.*]] = sext i8 [[TMP345]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP833:%.*]] = icmp sgt i32 [[CONV831]], [[CONV832]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP833]], label [[COND_TRUE835:%.*]], label [[COND_FALSE837:%.*]]
+// SIMD-ONLY0:       cond.true835:
+// SIMD-ONLY0-NEXT:    [[TMP346:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV836:%.*]] = sext i8 [[TMP346]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END839:%.*]]
+// SIMD-ONLY0:       cond.false837:
+// SIMD-ONLY0-NEXT:    [[TMP347:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV838:%.*]] = sext i8 [[TMP347]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END839]]
+// SIMD-ONLY0:       cond.end839:
+// SIMD-ONLY0-NEXT:    [[COND840:%.*]] = phi i32 [ [[CONV836]], [[COND_TRUE835]] ], [ [[CONV838]], [[COND_FALSE837]] ]
+// SIMD-ONLY0-NEXT:    [[CONV841:%.*]] = trunc i32 [[COND840]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV841]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP348:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV842:%.*]] = sext i8 [[TMP348]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP349:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV843:%.*]] = sext i8 [[TMP349]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP844:%.*]] = icmp slt i32 [[CONV842]], [[CONV843]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP844]], label [[COND_TRUE846:%.*]], label [[COND_FALSE848:%.*]]
+// SIMD-ONLY0:       cond.true846:
+// SIMD-ONLY0-NEXT:    [[TMP350:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV847:%.*]] = sext i8 [[TMP350]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END850:%.*]]
+// SIMD-ONLY0:       cond.false848:
+// SIMD-ONLY0-NEXT:    [[TMP351:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV849:%.*]] = sext i8 [[TMP351]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END850]]
+// SIMD-ONLY0:       cond.end850:
+// SIMD-ONLY0-NEXT:    [[COND851:%.*]] = phi i32 [ [[CONV847]], [[COND_TRUE846]] ], [ [[CONV849]], [[COND_FALSE848]] ]
+// SIMD-ONLY0-NEXT:    [[CONV852:%.*]] = trunc i32 [[COND851]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV852]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP352:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV853:%.*]] = sext i8 [[TMP352]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP353:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV854:%.*]] = sext i8 [[TMP353]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP855:%.*]] = icmp sgt i32 [[CONV853]], [[CONV854]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP855]], label [[IF_THEN857:%.*]], label [[IF_END858:%.*]]
+// SIMD-ONLY0:       if.then857:
+// SIMD-ONLY0-NEXT:    [[TMP354:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP354]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END858]]
+// SIMD-ONLY0:       if.end858:
+// SIMD-ONLY0-NEXT:    [[TMP355:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV859:%.*]] = sext i8 [[TMP355]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP356:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV860:%.*]] = sext i8 [[TMP356]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP861:%.*]] = icmp slt i32 [[CONV859]], [[CONV860]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP861]], label [[IF_THEN863:%.*]], label [[IF_END864:%.*]]
+// SIMD-ONLY0:       if.then863:
+// SIMD-ONLY0-NEXT:    [[TMP357:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP357]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END864]]
+// SIMD-ONLY0:       if.end864:
+// SIMD-ONLY0-NEXT:    [[TMP358:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV865:%.*]] = sext i8 [[TMP358]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP359:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV866:%.*]] = sext i8 [[TMP359]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP867:%.*]] = icmp sgt i32 [[CONV865]], [[CONV866]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP867]], label [[IF_THEN869:%.*]], label [[IF_END870:%.*]]
+// SIMD-ONLY0:       if.then869:
+// SIMD-ONLY0-NEXT:    [[TMP360:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP360]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END870]]
+// SIMD-ONLY0:       if.end870:
+// SIMD-ONLY0-NEXT:    [[TMP361:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV871:%.*]] = sext i8 [[TMP361]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP362:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV872:%.*]] = sext i8 [[TMP362]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP873:%.*]] = icmp slt i32 [[CONV871]], [[CONV872]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP873]], label [[IF_THEN875:%.*]], label [[IF_END876:%.*]]
+// SIMD-ONLY0:       if.then875:
+// SIMD-ONLY0-NEXT:    [[TMP363:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP363]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END876]]
+// SIMD-ONLY0:       if.end876:
+// SIMD-ONLY0-NEXT:    [[TMP364:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV877:%.*]] = sext i8 [[TMP364]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP365:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV878:%.*]] = sext i8 [[TMP365]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP879:%.*]] = icmp eq i32 [[CONV877]], [[CONV878]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP879]], label [[COND_TRUE881:%.*]], label [[COND_FALSE883:%.*]]
+// SIMD-ONLY0:       cond.true881:
+// SIMD-ONLY0-NEXT:    [[TMP366:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV882:%.*]] = sext i8 [[TMP366]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END885:%.*]]
+// SIMD-ONLY0:       cond.false883:
+// SIMD-ONLY0-NEXT:    [[TMP367:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV884:%.*]] = sext i8 [[TMP367]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END885]]
+// SIMD-ONLY0:       cond.end885:
+// SIMD-ONLY0-NEXT:    [[COND886:%.*]] = phi i32 [ [[CONV882]], [[COND_TRUE881]] ], [ [[CONV884]], [[COND_FALSE883]] ]
+// SIMD-ONLY0-NEXT:    [[CONV887:%.*]] = trunc i32 [[COND886]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV887]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP368:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV888:%.*]] = sext i8 [[TMP368]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP369:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV889:%.*]] = sext i8 [[TMP369]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP890:%.*]] = icmp eq i32 [[CONV888]], [[CONV889]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP890]], label [[COND_TRUE892:%.*]], label [[COND_FALSE894:%.*]]
+// SIMD-ONLY0:       cond.true892:
+// SIMD-ONLY0-NEXT:    [[TMP370:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV893:%.*]] = sext i8 [[TMP370]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END896:%.*]]
+// SIMD-ONLY0:       cond.false894:
+// SIMD-ONLY0-NEXT:    [[TMP371:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV895:%.*]] = sext i8 [[TMP371]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END896]]
+// SIMD-ONLY0:       cond.end896:
+// SIMD-ONLY0-NEXT:    [[COND897:%.*]] = phi i32 [ [[CONV893]], [[COND_TRUE892]] ], [ [[CONV895]], [[COND_FALSE894]] ]
+// SIMD-ONLY0-NEXT:    [[CONV898:%.*]] = trunc i32 [[COND897]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV898]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP372:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV899:%.*]] = sext i8 [[TMP372]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP373:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV900:%.*]] = sext i8 [[TMP373]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP901:%.*]] = icmp eq i32 [[CONV899]], [[CONV900]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP901]], label [[IF_THEN903:%.*]], label [[IF_END904:%.*]]
+// SIMD-ONLY0:       if.then903:
+// SIMD-ONLY0-NEXT:    [[TMP374:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP374]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END904]]
+// SIMD-ONLY0:       if.end904:
+// SIMD-ONLY0-NEXT:    [[TMP375:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV905:%.*]] = sext i8 [[TMP375]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP376:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV906:%.*]] = sext i8 [[TMP376]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP907:%.*]] = icmp eq i32 [[CONV905]], [[CONV906]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP907]], label [[IF_THEN909:%.*]], label [[IF_END910:%.*]]
+// SIMD-ONLY0:       if.then909:
+// SIMD-ONLY0-NEXT:    [[TMP377:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP377]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END910]]
+// SIMD-ONLY0:       if.end910:
+// SIMD-ONLY0-NEXT:    [[TMP378:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV911:%.*]] = zext i8 [[TMP378]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP379:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV912:%.*]] = zext i8 [[TMP379]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP913:%.*]] = icmp sgt i32 [[CONV911]], [[CONV912]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP913]], label [[COND_TRUE915:%.*]], label [[COND_FALSE917:%.*]]
+// SIMD-ONLY0:       cond.true915:
+// SIMD-ONLY0-NEXT:    [[TMP380:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV916:%.*]] = zext i8 [[TMP380]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END919:%.*]]
+// SIMD-ONLY0:       cond.false917:
+// SIMD-ONLY0-NEXT:    [[TMP381:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV918:%.*]] = zext i8 [[TMP381]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END919]]
+// SIMD-ONLY0:       cond.end919:
+// SIMD-ONLY0-NEXT:    [[COND920:%.*]] = phi i32 [ [[CONV916]], [[COND_TRUE915]] ], [ [[CONV918]], [[COND_FALSE917]] ]
+// SIMD-ONLY0-NEXT:    [[CONV921:%.*]] = trunc i32 [[COND920]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV921]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP382:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV922:%.*]] = zext i8 [[TMP382]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP383:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV923:%.*]] = zext i8 [[TMP383]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP924:%.*]] = icmp slt i32 [[CONV922]], [[CONV923]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP924]], label [[COND_TRUE926:%.*]], label [[COND_FALSE928:%.*]]
+// SIMD-ONLY0:       cond.true926:
+// SIMD-ONLY0-NEXT:    [[TMP384:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV927:%.*]] = zext i8 [[TMP384]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END930:%.*]]
+// SIMD-ONLY0:       cond.false928:
+// SIMD-ONLY0-NEXT:    [[TMP385:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV929:%.*]] = zext i8 [[TMP385]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END930]]
+// SIMD-ONLY0:       cond.end930:
+// SIMD-ONLY0-NEXT:    [[COND931:%.*]] = phi i32 [ [[CONV927]], [[COND_TRUE926]] ], [ [[CONV929]], [[COND_FALSE928]] ]
+// SIMD-ONLY0-NEXT:    [[CONV932:%.*]] = trunc i32 [[COND931]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV932]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP386:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV933:%.*]] = zext i8 [[TMP386]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP387:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV934:%.*]] = zext i8 [[TMP387]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP935:%.*]] = icmp sgt i32 [[CONV933]], [[CONV934]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP935]], label [[COND_TRUE937:%.*]], label [[COND_FALSE939:%.*]]
+// SIMD-ONLY0:       cond.true937:
+// SIMD-ONLY0-NEXT:    [[TMP388:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV938:%.*]] = zext i8 [[TMP388]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END941:%.*]]
+// SIMD-ONLY0:       cond.false939:
+// SIMD-ONLY0-NEXT:    [[TMP389:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV940:%.*]] = zext i8 [[TMP389]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END941]]
+// SIMD-ONLY0:       cond.end941:
+// SIMD-ONLY0-NEXT:    [[COND942:%.*]] = phi i32 [ [[CONV938]], [[COND_TRUE937]] ], [ [[CONV940]], [[COND_FALSE939]] ]
+// SIMD-ONLY0-NEXT:    [[CONV943:%.*]] = trunc i32 [[COND942]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV943]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP390:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV944:%.*]] = zext i8 [[TMP390]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP391:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV945:%.*]] = zext i8 [[TMP391]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP946:%.*]] = icmp slt i32 [[CONV944]], [[CONV945]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP946]], label [[COND_TRUE948:%.*]], label [[COND_FALSE950:%.*]]
+// SIMD-ONLY0:       cond.true948:
+// SIMD-ONLY0-NEXT:    [[TMP392:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV949:%.*]] = zext i8 [[TMP392]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END952:%.*]]
+// SIMD-ONLY0:       cond.false950:
+// SIMD-ONLY0-NEXT:    [[TMP393:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV951:%.*]] = zext i8 [[TMP393]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END952]]
+// SIMD-ONLY0:       cond.end952:
+// SIMD-ONLY0-NEXT:    [[COND953:%.*]] = phi i32 [ [[CONV949]], [[COND_TRUE948]] ], [ [[CONV951]], [[COND_FALSE950]] ]
+// SIMD-ONLY0-NEXT:    [[CONV954:%.*]] = trunc i32 [[COND953]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV954]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP394:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV955:%.*]] = zext i8 [[TMP394]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP395:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV956:%.*]] = zext i8 [[TMP395]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP957:%.*]] = icmp sgt i32 [[CONV955]], [[CONV956]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP957]], label [[IF_THEN959:%.*]], label [[IF_END960:%.*]]
+// SIMD-ONLY0:       if.then959:
+// SIMD-ONLY0-NEXT:    [[TMP396:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP396]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END960]]
+// SIMD-ONLY0:       if.end960:
+// SIMD-ONLY0-NEXT:    [[TMP397:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV961:%.*]] = zext i8 [[TMP397]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP398:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV962:%.*]] = zext i8 [[TMP398]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP963:%.*]] = icmp slt i32 [[CONV961]], [[CONV962]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP963]], label [[IF_THEN965:%.*]], label [[IF_END966:%.*]]
+// SIMD-ONLY0:       if.then965:
+// SIMD-ONLY0-NEXT:    [[TMP399:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP399]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END966]]
+// SIMD-ONLY0:       if.end966:
+// SIMD-ONLY0-NEXT:    [[TMP400:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV967:%.*]] = zext i8 [[TMP400]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP401:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV968:%.*]] = zext i8 [[TMP401]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP969:%.*]] = icmp sgt i32 [[CONV967]], [[CONV968]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP969]], label [[IF_THEN971:%.*]], label [[IF_END972:%.*]]
+// SIMD-ONLY0:       if.then971:
+// SIMD-ONLY0-NEXT:    [[TMP402:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP402]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END972]]
+// SIMD-ONLY0:       if.end972:
+// SIMD-ONLY0-NEXT:    [[TMP403:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV973:%.*]] = zext i8 [[TMP403]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP404:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV974:%.*]] = zext i8 [[TMP404]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP975:%.*]] = icmp slt i32 [[CONV973]], [[CONV974]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP975]], label [[IF_THEN977:%.*]], label [[IF_END978:%.*]]
+// SIMD-ONLY0:       if.then977:
+// SIMD-ONLY0-NEXT:    [[TMP405:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP405]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END978]]
+// SIMD-ONLY0:       if.end978:
+// SIMD-ONLY0-NEXT:    [[TMP406:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV979:%.*]] = zext i8 [[TMP406]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP407:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV980:%.*]] = zext i8 [[TMP407]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP981:%.*]] = icmp eq i32 [[CONV979]], [[CONV980]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP981]], label [[COND_TRUE983:%.*]], label [[COND_FALSE985:%.*]]
+// SIMD-ONLY0:       cond.true983:
+// SIMD-ONLY0-NEXT:    [[TMP408:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV984:%.*]] = zext i8 [[TMP408]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END987:%.*]]
+// SIMD-ONLY0:       cond.false985:
+// SIMD-ONLY0-NEXT:    [[TMP409:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV986:%.*]] = zext i8 [[TMP409]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END987]]
+// SIMD-ONLY0:       cond.end987:
+// SIMD-ONLY0-NEXT:    [[COND988:%.*]] = phi i32 [ [[CONV984]], [[COND_TRUE983]] ], [ [[CONV986]], [[COND_FALSE985]] ]
+// SIMD-ONLY0-NEXT:    [[CONV989:%.*]] = trunc i32 [[COND988]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV989]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP410:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV990:%.*]] = zext i8 [[TMP410]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP411:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV991:%.*]] = zext i8 [[TMP411]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP992:%.*]] = icmp eq i32 [[CONV990]], [[CONV991]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP992]], label [[COND_TRUE994:%.*]], label [[COND_FALSE996:%.*]]
+// SIMD-ONLY0:       cond.true994:
+// SIMD-ONLY0-NEXT:    [[TMP412:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV995:%.*]] = zext i8 [[TMP412]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END998:%.*]]
+// SIMD-ONLY0:       cond.false996:
+// SIMD-ONLY0-NEXT:    [[TMP413:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV997:%.*]] = zext i8 [[TMP413]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END998]]
+// SIMD-ONLY0:       cond.end998:
+// SIMD-ONLY0-NEXT:    [[COND999:%.*]] = phi i32 [ [[CONV995]], [[COND_TRUE994]] ], [ [[CONV997]], [[COND_FALSE996]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1000:%.*]] = trunc i32 [[COND999]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1000]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP414:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1001:%.*]] = zext i8 [[TMP414]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP415:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1002:%.*]] = zext i8 [[TMP415]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1003:%.*]] = icmp eq i32 [[CONV1001]], [[CONV1002]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1003]], label [[IF_THEN1005:%.*]], label [[IF_END1006:%.*]]
+// SIMD-ONLY0:       if.then1005:
+// SIMD-ONLY0-NEXT:    [[TMP416:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP416]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1006]]
+// SIMD-ONLY0:       if.end1006:
+// SIMD-ONLY0-NEXT:    [[TMP417:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1007:%.*]] = zext i8 [[TMP417]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP418:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1008:%.*]] = zext i8 [[TMP418]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1009:%.*]] = icmp eq i32 [[CONV1007]], [[CONV1008]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1009]], label [[IF_THEN1011:%.*]], label [[IF_END1012:%.*]]
+// SIMD-ONLY0:       if.then1011:
+// SIMD-ONLY0-NEXT:    [[TMP419:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP419]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1012]]
+// SIMD-ONLY0:       if.end1012:
+// SIMD-ONLY0-NEXT:    [[TMP420:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1013:%.*]] = sext i8 [[TMP420]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP421:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1014:%.*]] = sext i8 [[TMP421]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1015:%.*]] = icmp sgt i32 [[CONV1013]], [[CONV1014]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1015]], label [[COND_TRUE1017:%.*]], label [[COND_FALSE1019:%.*]]
+// SIMD-ONLY0:       cond.true1017:
+// SIMD-ONLY0-NEXT:    [[TMP422:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1018:%.*]] = sext i8 [[TMP422]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1021:%.*]]
+// SIMD-ONLY0:       cond.false1019:
+// SIMD-ONLY0-NEXT:    [[TMP423:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1020:%.*]] = sext i8 [[TMP423]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1021]]
+// SIMD-ONLY0:       cond.end1021:
+// SIMD-ONLY0-NEXT:    [[COND1022:%.*]] = phi i32 [ [[CONV1018]], [[COND_TRUE1017]] ], [ [[CONV1020]], [[COND_FALSE1019]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1023:%.*]] = trunc i32 [[COND1022]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1023]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP424:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1024:%.*]] = sext i8 [[TMP424]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP425:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1025:%.*]] = sext i8 [[TMP425]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1026:%.*]] = icmp slt i32 [[CONV1024]], [[CONV1025]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1026]], label [[COND_TRUE1028:%.*]], label [[COND_FALSE1030:%.*]]
+// SIMD-ONLY0:       cond.true1028:
+// SIMD-ONLY0-NEXT:    [[TMP426:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1029:%.*]] = sext i8 [[TMP426]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1032:%.*]]
+// SIMD-ONLY0:       cond.false1030:
+// SIMD-ONLY0-NEXT:    [[TMP427:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1031:%.*]] = sext i8 [[TMP427]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1032]]
+// SIMD-ONLY0:       cond.end1032:
+// SIMD-ONLY0-NEXT:    [[COND1033:%.*]] = phi i32 [ [[CONV1029]], [[COND_TRUE1028]] ], [ [[CONV1031]], [[COND_FALSE1030]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1034:%.*]] = trunc i32 [[COND1033]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1034]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP428:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1035:%.*]] = sext i8 [[TMP428]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP429:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1036:%.*]] = sext i8 [[TMP429]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1037:%.*]] = icmp sgt i32 [[CONV1035]], [[CONV1036]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1037]], label [[COND_TRUE1039:%.*]], label [[COND_FALSE1041:%.*]]
+// SIMD-ONLY0:       cond.true1039:
+// SIMD-ONLY0-NEXT:    [[TMP430:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1040:%.*]] = sext i8 [[TMP430]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1043:%.*]]
+// SIMD-ONLY0:       cond.false1041:
+// SIMD-ONLY0-NEXT:    [[TMP431:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1042:%.*]] = sext i8 [[TMP431]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1043]]
+// SIMD-ONLY0:       cond.end1043:
+// SIMD-ONLY0-NEXT:    [[COND1044:%.*]] = phi i32 [ [[CONV1040]], [[COND_TRUE1039]] ], [ [[CONV1042]], [[COND_FALSE1041]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1045:%.*]] = trunc i32 [[COND1044]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1045]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP432:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1046:%.*]] = sext i8 [[TMP432]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP433:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1047:%.*]] = sext i8 [[TMP433]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1048:%.*]] = icmp slt i32 [[CONV1046]], [[CONV1047]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1048]], label [[COND_TRUE1050:%.*]], label [[COND_FALSE1052:%.*]]
+// SIMD-ONLY0:       cond.true1050:
+// SIMD-ONLY0-NEXT:    [[TMP434:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1051:%.*]] = sext i8 [[TMP434]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1054:%.*]]
+// SIMD-ONLY0:       cond.false1052:
+// SIMD-ONLY0-NEXT:    [[TMP435:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1053:%.*]] = sext i8 [[TMP435]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1054]]
+// SIMD-ONLY0:       cond.end1054:
+// SIMD-ONLY0-NEXT:    [[COND1055:%.*]] = phi i32 [ [[CONV1051]], [[COND_TRUE1050]] ], [ [[CONV1053]], [[COND_FALSE1052]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1056:%.*]] = trunc i32 [[COND1055]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1056]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP436:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1057:%.*]] = sext i8 [[TMP436]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP437:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1058:%.*]] = sext i8 [[TMP437]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1059:%.*]] = icmp sgt i32 [[CONV1057]], [[CONV1058]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1059]], label [[IF_THEN1061:%.*]], label [[IF_END1062:%.*]]
+// SIMD-ONLY0:       if.then1061:
+// SIMD-ONLY0-NEXT:    [[TMP438:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP438]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1062]]
+// SIMD-ONLY0:       if.end1062:
+// SIMD-ONLY0-NEXT:    [[TMP439:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1063:%.*]] = sext i8 [[TMP439]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP440:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1064:%.*]] = sext i8 [[TMP440]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1065:%.*]] = icmp slt i32 [[CONV1063]], [[CONV1064]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1065]], label [[IF_THEN1067:%.*]], label [[IF_END1068:%.*]]
+// SIMD-ONLY0:       if.then1067:
+// SIMD-ONLY0-NEXT:    [[TMP441:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP441]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1068]]
+// SIMD-ONLY0:       if.end1068:
+// SIMD-ONLY0-NEXT:    [[TMP442:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1069:%.*]] = sext i8 [[TMP442]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP443:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1070:%.*]] = sext i8 [[TMP443]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1071:%.*]] = icmp sgt i32 [[CONV1069]], [[CONV1070]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1071]], label [[IF_THEN1073:%.*]], label [[IF_END1074:%.*]]
+// SIMD-ONLY0:       if.then1073:
+// SIMD-ONLY0-NEXT:    [[TMP444:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP444]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1074]]
+// SIMD-ONLY0:       if.end1074:
+// SIMD-ONLY0-NEXT:    [[TMP445:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1075:%.*]] = sext i8 [[TMP445]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP446:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1076:%.*]] = sext i8 [[TMP446]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1077:%.*]] = icmp slt i32 [[CONV1075]], [[CONV1076]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1077]], label [[IF_THEN1079:%.*]], label [[IF_END1080:%.*]]
+// SIMD-ONLY0:       if.then1079:
+// SIMD-ONLY0-NEXT:    [[TMP447:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP447]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1080]]
+// SIMD-ONLY0:       if.end1080:
+// SIMD-ONLY0-NEXT:    [[TMP448:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1081:%.*]] = sext i8 [[TMP448]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP449:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1082:%.*]] = sext i8 [[TMP449]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1083:%.*]] = icmp eq i32 [[CONV1081]], [[CONV1082]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1083]], label [[COND_TRUE1085:%.*]], label [[COND_FALSE1087:%.*]]
+// SIMD-ONLY0:       cond.true1085:
+// SIMD-ONLY0-NEXT:    [[TMP450:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1086:%.*]] = sext i8 [[TMP450]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1089:%.*]]
+// SIMD-ONLY0:       cond.false1087:
+// SIMD-ONLY0-NEXT:    [[TMP451:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1088:%.*]] = sext i8 [[TMP451]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1089]]
+// SIMD-ONLY0:       cond.end1089:
+// SIMD-ONLY0-NEXT:    [[COND1090:%.*]] = phi i32 [ [[CONV1086]], [[COND_TRUE1085]] ], [ [[CONV1088]], [[COND_FALSE1087]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1091:%.*]] = trunc i32 [[COND1090]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1091]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP452:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1092:%.*]] = sext i8 [[TMP452]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP453:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1093:%.*]] = sext i8 [[TMP453]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1094:%.*]] = icmp eq i32 [[CONV1092]], [[CONV1093]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1094]], label [[COND_TRUE1096:%.*]], label [[COND_FALSE1098:%.*]]
+// SIMD-ONLY0:       cond.true1096:
+// SIMD-ONLY0-NEXT:    [[TMP454:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1097:%.*]] = sext i8 [[TMP454]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1100:%.*]]
+// SIMD-ONLY0:       cond.false1098:
+// SIMD-ONLY0-NEXT:    [[TMP455:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1099:%.*]] = sext i8 [[TMP455]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1100]]
+// SIMD-ONLY0:       cond.end1100:
+// SIMD-ONLY0-NEXT:    [[COND1101:%.*]] = phi i32 [ [[CONV1097]], [[COND_TRUE1096]] ], [ [[CONV1099]], [[COND_FALSE1098]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1102:%.*]] = trunc i32 [[COND1101]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1102]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP456:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1103:%.*]] = sext i8 [[TMP456]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP457:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1104:%.*]] = sext i8 [[TMP457]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1105:%.*]] = icmp eq i32 [[CONV1103]], [[CONV1104]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1105]], label [[IF_THEN1107:%.*]], label [[IF_END1108:%.*]]
+// SIMD-ONLY0:       if.then1107:
+// SIMD-ONLY0-NEXT:    [[TMP458:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP458]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1108]]
+// SIMD-ONLY0:       if.end1108:
+// SIMD-ONLY0-NEXT:    [[TMP459:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1109:%.*]] = sext i8 [[TMP459]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP460:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1110:%.*]] = sext i8 [[TMP460]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1111:%.*]] = icmp eq i32 [[CONV1109]], [[CONV1110]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1111]], label [[IF_THEN1113:%.*]], label [[IF_END1114:%.*]]
+// SIMD-ONLY0:       if.then1113:
+// SIMD-ONLY0-NEXT:    [[TMP461:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP461]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1114]]
+// SIMD-ONLY0:       if.end1114:
+// SIMD-ONLY0-NEXT:    [[TMP462:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1115:%.*]] = zext i8 [[TMP462]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP463:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1116:%.*]] = zext i8 [[TMP463]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1117:%.*]] = icmp sgt i32 [[CONV1115]], [[CONV1116]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1117]], label [[COND_TRUE1119:%.*]], label [[COND_FALSE1121:%.*]]
+// SIMD-ONLY0:       cond.true1119:
+// SIMD-ONLY0-NEXT:    [[TMP464:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1120:%.*]] = zext i8 [[TMP464]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1123:%.*]]
+// SIMD-ONLY0:       cond.false1121:
+// SIMD-ONLY0-NEXT:    [[TMP465:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1122:%.*]] = zext i8 [[TMP465]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1123]]
+// SIMD-ONLY0:       cond.end1123:
+// SIMD-ONLY0-NEXT:    [[COND1124:%.*]] = phi i32 [ [[CONV1120]], [[COND_TRUE1119]] ], [ [[CONV1122]], [[COND_FALSE1121]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1125:%.*]] = trunc i32 [[COND1124]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1125]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP466:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1126:%.*]] = zext i8 [[TMP466]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP467:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1127:%.*]] = zext i8 [[TMP467]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1128:%.*]] = icmp slt i32 [[CONV1126]], [[CONV1127]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1128]], label [[COND_TRUE1130:%.*]], label [[COND_FALSE1132:%.*]]
+// SIMD-ONLY0:       cond.true1130:
+// SIMD-ONLY0-NEXT:    [[TMP468:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1131:%.*]] = zext i8 [[TMP468]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1134:%.*]]
+// SIMD-ONLY0:       cond.false1132:
+// SIMD-ONLY0-NEXT:    [[TMP469:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1133:%.*]] = zext i8 [[TMP469]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1134]]
+// SIMD-ONLY0:       cond.end1134:
+// SIMD-ONLY0-NEXT:    [[COND1135:%.*]] = phi i32 [ [[CONV1131]], [[COND_TRUE1130]] ], [ [[CONV1133]], [[COND_FALSE1132]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1136:%.*]] = trunc i32 [[COND1135]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1136]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP470:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1137:%.*]] = zext i8 [[TMP470]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP471:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1138:%.*]] = zext i8 [[TMP471]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1139:%.*]] = icmp sgt i32 [[CONV1137]], [[CONV1138]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1139]], label [[COND_TRUE1141:%.*]], label [[COND_FALSE1143:%.*]]
+// SIMD-ONLY0:       cond.true1141:
+// SIMD-ONLY0-NEXT:    [[TMP472:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1142:%.*]] = zext i8 [[TMP472]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1145:%.*]]
+// SIMD-ONLY0:       cond.false1143:
+// SIMD-ONLY0-NEXT:    [[TMP473:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1144:%.*]] = zext i8 [[TMP473]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1145]]
+// SIMD-ONLY0:       cond.end1145:
+// SIMD-ONLY0-NEXT:    [[COND1146:%.*]] = phi i32 [ [[CONV1142]], [[COND_TRUE1141]] ], [ [[CONV1144]], [[COND_FALSE1143]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1147:%.*]] = trunc i32 [[COND1146]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1147]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP474:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1148:%.*]] = zext i8 [[TMP474]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP475:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1149:%.*]] = zext i8 [[TMP475]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1150:%.*]] = icmp slt i32 [[CONV1148]], [[CONV1149]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1150]], label [[COND_TRUE1152:%.*]], label [[COND_FALSE1154:%.*]]
+// SIMD-ONLY0:       cond.true1152:
+// SIMD-ONLY0-NEXT:    [[TMP476:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1153:%.*]] = zext i8 [[TMP476]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1156:%.*]]
+// SIMD-ONLY0:       cond.false1154:
+// SIMD-ONLY0-NEXT:    [[TMP477:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1155:%.*]] = zext i8 [[TMP477]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1156]]
+// SIMD-ONLY0:       cond.end1156:
+// SIMD-ONLY0-NEXT:    [[COND1157:%.*]] = phi i32 [ [[CONV1153]], [[COND_TRUE1152]] ], [ [[CONV1155]], [[COND_FALSE1154]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1158:%.*]] = trunc i32 [[COND1157]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1158]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP478:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1159:%.*]] = zext i8 [[TMP478]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP479:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1160:%.*]] = zext i8 [[TMP479]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1161:%.*]] = icmp sgt i32 [[CONV1159]], [[CONV1160]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1161]], label [[IF_THEN1163:%.*]], label [[IF_END1164:%.*]]
+// SIMD-ONLY0:       if.then1163:
+// SIMD-ONLY0-NEXT:    [[TMP480:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP480]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1164]]
+// SIMD-ONLY0:       if.end1164:
+// SIMD-ONLY0-NEXT:    [[TMP481:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1165:%.*]] = zext i8 [[TMP481]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP482:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1166:%.*]] = zext i8 [[TMP482]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1167:%.*]] = icmp slt i32 [[CONV1165]], [[CONV1166]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1167]], label [[IF_THEN1169:%.*]], label [[IF_END1170:%.*]]
+// SIMD-ONLY0:       if.then1169:
+// SIMD-ONLY0-NEXT:    [[TMP483:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP483]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1170]]
+// SIMD-ONLY0:       if.end1170:
+// SIMD-ONLY0-NEXT:    [[TMP484:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1171:%.*]] = zext i8 [[TMP484]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP485:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1172:%.*]] = zext i8 [[TMP485]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1173:%.*]] = icmp sgt i32 [[CONV1171]], [[CONV1172]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1173]], label [[IF_THEN1175:%.*]], label [[IF_END1176:%.*]]
+// SIMD-ONLY0:       if.then1175:
+// SIMD-ONLY0-NEXT:    [[TMP486:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP486]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1176]]
+// SIMD-ONLY0:       if.end1176:
+// SIMD-ONLY0-NEXT:    [[TMP487:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1177:%.*]] = zext i8 [[TMP487]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP488:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1178:%.*]] = zext i8 [[TMP488]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1179:%.*]] = icmp slt i32 [[CONV1177]], [[CONV1178]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1179]], label [[IF_THEN1181:%.*]], label [[IF_END1182:%.*]]
+// SIMD-ONLY0:       if.then1181:
+// SIMD-ONLY0-NEXT:    [[TMP489:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP489]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1182]]
+// SIMD-ONLY0:       if.end1182:
+// SIMD-ONLY0-NEXT:    [[TMP490:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1183:%.*]] = zext i8 [[TMP490]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP491:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1184:%.*]] = zext i8 [[TMP491]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1185:%.*]] = icmp eq i32 [[CONV1183]], [[CONV1184]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1185]], label [[COND_TRUE1187:%.*]], label [[COND_FALSE1189:%.*]]
+// SIMD-ONLY0:       cond.true1187:
+// SIMD-ONLY0-NEXT:    [[TMP492:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1188:%.*]] = zext i8 [[TMP492]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1191:%.*]]
+// SIMD-ONLY0:       cond.false1189:
+// SIMD-ONLY0-NEXT:    [[TMP493:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1190:%.*]] = zext i8 [[TMP493]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1191]]
+// SIMD-ONLY0:       cond.end1191:
+// SIMD-ONLY0-NEXT:    [[COND1192:%.*]] = phi i32 [ [[CONV1188]], [[COND_TRUE1187]] ], [ [[CONV1190]], [[COND_FALSE1189]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1193:%.*]] = trunc i32 [[COND1192]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1193]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP494:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1194:%.*]] = zext i8 [[TMP494]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP495:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1195:%.*]] = zext i8 [[TMP495]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1196:%.*]] = icmp eq i32 [[CONV1194]], [[CONV1195]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1196]], label [[COND_TRUE1198:%.*]], label [[COND_FALSE1200:%.*]]
+// SIMD-ONLY0:       cond.true1198:
+// SIMD-ONLY0-NEXT:    [[TMP496:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1199:%.*]] = zext i8 [[TMP496]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1202:%.*]]
+// SIMD-ONLY0:       cond.false1200:
+// SIMD-ONLY0-NEXT:    [[TMP497:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1201:%.*]] = zext i8 [[TMP497]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1202]]
+// SIMD-ONLY0:       cond.end1202:
+// SIMD-ONLY0-NEXT:    [[COND1203:%.*]] = phi i32 [ [[CONV1199]], [[COND_TRUE1198]] ], [ [[CONV1201]], [[COND_FALSE1200]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1204:%.*]] = trunc i32 [[COND1203]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1204]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP498:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1205:%.*]] = zext i8 [[TMP498]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP499:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1206:%.*]] = zext i8 [[TMP499]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1207:%.*]] = icmp eq i32 [[CONV1205]], [[CONV1206]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1207]], label [[IF_THEN1209:%.*]], label [[IF_END1210:%.*]]
+// SIMD-ONLY0:       if.then1209:
+// SIMD-ONLY0-NEXT:    [[TMP500:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP500]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1210]]
+// SIMD-ONLY0:       if.end1210:
+// SIMD-ONLY0-NEXT:    [[TMP501:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1211:%.*]] = zext i8 [[TMP501]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP502:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1212:%.*]] = zext i8 [[TMP502]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1213:%.*]] = icmp eq i32 [[CONV1211]], [[CONV1212]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1213]], label [[IF_THEN1215:%.*]], label [[IF_END1216:%.*]]
+// SIMD-ONLY0:       if.then1215:
+// SIMD-ONLY0-NEXT:    [[TMP503:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP503]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1216]]
+// SIMD-ONLY0:       if.end1216:
+// SIMD-ONLY0-NEXT:    [[TMP504:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1217:%.*]] = sext i16 [[TMP504]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP505:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1218:%.*]] = sext i16 [[TMP505]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1219:%.*]] = icmp sgt i32 [[CONV1217]], [[CONV1218]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1219]], label [[COND_TRUE1221:%.*]], label [[COND_FALSE1223:%.*]]
+// SIMD-ONLY0:       cond.true1221:
+// SIMD-ONLY0-NEXT:    [[TMP506:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1222:%.*]] = sext i16 [[TMP506]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1225:%.*]]
+// SIMD-ONLY0:       cond.false1223:
+// SIMD-ONLY0-NEXT:    [[TMP507:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1224:%.*]] = sext i16 [[TMP507]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1225]]
+// SIMD-ONLY0:       cond.end1225:
+// SIMD-ONLY0-NEXT:    [[COND1226:%.*]] = phi i32 [ [[CONV1222]], [[COND_TRUE1221]] ], [ [[CONV1224]], [[COND_FALSE1223]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1227:%.*]] = trunc i32 [[COND1226]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1227]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP508:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1228:%.*]] = sext i16 [[TMP508]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP509:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1229:%.*]] = sext i16 [[TMP509]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1230:%.*]] = icmp slt i32 [[CONV1228]], [[CONV1229]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1230]], label [[COND_TRUE1232:%.*]], label [[COND_FALSE1234:%.*]]
+// SIMD-ONLY0:       cond.true1232:
+// SIMD-ONLY0-NEXT:    [[TMP510:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1233:%.*]] = sext i16 [[TMP510]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1236:%.*]]
+// SIMD-ONLY0:       cond.false1234:
+// SIMD-ONLY0-NEXT:    [[TMP511:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1235:%.*]] = sext i16 [[TMP511]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1236]]
+// SIMD-ONLY0:       cond.end1236:
+// SIMD-ONLY0-NEXT:    [[COND1237:%.*]] = phi i32 [ [[CONV1233]], [[COND_TRUE1232]] ], [ [[CONV1235]], [[COND_FALSE1234]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1238:%.*]] = trunc i32 [[COND1237]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1238]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP512:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1239:%.*]] = sext i16 [[TMP512]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP513:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1240:%.*]] = sext i16 [[TMP513]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1241:%.*]] = icmp sgt i32 [[CONV1239]], [[CONV1240]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1241]], label [[COND_TRUE1243:%.*]], label [[COND_FALSE1245:%.*]]
+// SIMD-ONLY0:       cond.true1243:
+// SIMD-ONLY0-NEXT:    [[TMP514:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1244:%.*]] = sext i16 [[TMP514]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1247:%.*]]
+// SIMD-ONLY0:       cond.false1245:
+// SIMD-ONLY0-NEXT:    [[TMP515:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1246:%.*]] = sext i16 [[TMP515]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1247]]
+// SIMD-ONLY0:       cond.end1247:
+// SIMD-ONLY0-NEXT:    [[COND1248:%.*]] = phi i32 [ [[CONV1244]], [[COND_TRUE1243]] ], [ [[CONV1246]], [[COND_FALSE1245]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1249:%.*]] = trunc i32 [[COND1248]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1249]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP516:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1250:%.*]] = sext i16 [[TMP516]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP517:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1251:%.*]] = sext i16 [[TMP517]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1252:%.*]] = icmp slt i32 [[CONV1250]], [[CONV1251]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1252]], label [[COND_TRUE1254:%.*]], label [[COND_FALSE1256:%.*]]
+// SIMD-ONLY0:       cond.true1254:
+// SIMD-ONLY0-NEXT:    [[TMP518:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1255:%.*]] = sext i16 [[TMP518]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1258:%.*]]
+// SIMD-ONLY0:       cond.false1256:
+// SIMD-ONLY0-NEXT:    [[TMP519:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1257:%.*]] = sext i16 [[TMP519]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1258]]
+// SIMD-ONLY0:       cond.end1258:
+// SIMD-ONLY0-NEXT:    [[COND1259:%.*]] = phi i32 [ [[CONV1255]], [[COND_TRUE1254]] ], [ [[CONV1257]], [[COND_FALSE1256]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1260:%.*]] = trunc i32 [[COND1259]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1260]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP520:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1261:%.*]] = sext i16 [[TMP520]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP521:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1262:%.*]] = sext i16 [[TMP521]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1263:%.*]] = icmp sgt i32 [[CONV1261]], [[CONV1262]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1263]], label [[IF_THEN1265:%.*]], label [[IF_END1266:%.*]]
+// SIMD-ONLY0:       if.then1265:
+// SIMD-ONLY0-NEXT:    [[TMP522:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP522]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1266]]
+// SIMD-ONLY0:       if.end1266:
+// SIMD-ONLY0-NEXT:    [[TMP523:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1267:%.*]] = sext i16 [[TMP523]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP524:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1268:%.*]] = sext i16 [[TMP524]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1269:%.*]] = icmp slt i32 [[CONV1267]], [[CONV1268]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1269]], label [[IF_THEN1271:%.*]], label [[IF_END1272:%.*]]
+// SIMD-ONLY0:       if.then1271:
+// SIMD-ONLY0-NEXT:    [[TMP525:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP525]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1272]]
+// SIMD-ONLY0:       if.end1272:
+// SIMD-ONLY0-NEXT:    [[TMP526:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1273:%.*]] = sext i16 [[TMP526]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP527:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1274:%.*]] = sext i16 [[TMP527]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1275:%.*]] = icmp sgt i32 [[CONV1273]], [[CONV1274]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1275]], label [[IF_THEN1277:%.*]], label [[IF_END1278:%.*]]
+// SIMD-ONLY0:       if.then1277:
+// SIMD-ONLY0-NEXT:    [[TMP528:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP528]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1278]]
+// SIMD-ONLY0:       if.end1278:
+// SIMD-ONLY0-NEXT:    [[TMP529:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1279:%.*]] = sext i16 [[TMP529]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP530:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1280:%.*]] = sext i16 [[TMP530]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1281:%.*]] = icmp slt i32 [[CONV1279]], [[CONV1280]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1281]], label [[IF_THEN1283:%.*]], label [[IF_END1284:%.*]]
+// SIMD-ONLY0:       if.then1283:
+// SIMD-ONLY0-NEXT:    [[TMP531:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP531]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1284]]
+// SIMD-ONLY0:       if.end1284:
+// SIMD-ONLY0-NEXT:    [[TMP532:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1285:%.*]] = sext i16 [[TMP532]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP533:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1286:%.*]] = sext i16 [[TMP533]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1287:%.*]] = icmp eq i32 [[CONV1285]], [[CONV1286]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1287]], label [[COND_TRUE1289:%.*]], label [[COND_FALSE1291:%.*]]
+// SIMD-ONLY0:       cond.true1289:
+// SIMD-ONLY0-NEXT:    [[TMP534:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1290:%.*]] = sext i16 [[TMP534]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1293:%.*]]
+// SIMD-ONLY0:       cond.false1291:
+// SIMD-ONLY0-NEXT:    [[TMP535:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1292:%.*]] = sext i16 [[TMP535]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1293]]
+// SIMD-ONLY0:       cond.end1293:
+// SIMD-ONLY0-NEXT:    [[COND1294:%.*]] = phi i32 [ [[CONV1290]], [[COND_TRUE1289]] ], [ [[CONV1292]], [[COND_FALSE1291]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1295:%.*]] = trunc i32 [[COND1294]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1295]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP536:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1296:%.*]] = sext i16 [[TMP536]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP537:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1297:%.*]] = sext i16 [[TMP537]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1298:%.*]] = icmp eq i32 [[CONV1296]], [[CONV1297]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1298]], label [[COND_TRUE1300:%.*]], label [[COND_FALSE1302:%.*]]
+// SIMD-ONLY0:       cond.true1300:
+// SIMD-ONLY0-NEXT:    [[TMP538:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1301:%.*]] = sext i16 [[TMP538]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1304:%.*]]
+// SIMD-ONLY0:       cond.false1302:
+// SIMD-ONLY0-NEXT:    [[TMP539:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1303:%.*]] = sext i16 [[TMP539]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1304]]
+// SIMD-ONLY0:       cond.end1304:
+// SIMD-ONLY0-NEXT:    [[COND1305:%.*]] = phi i32 [ [[CONV1301]], [[COND_TRUE1300]] ], [ [[CONV1303]], [[COND_FALSE1302]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1306:%.*]] = trunc i32 [[COND1305]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1306]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP540:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1307:%.*]] = sext i16 [[TMP540]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP541:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1308:%.*]] = sext i16 [[TMP541]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1309:%.*]] = icmp eq i32 [[CONV1307]], [[CONV1308]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1309]], label [[IF_THEN1311:%.*]], label [[IF_END1312:%.*]]
+// SIMD-ONLY0:       if.then1311:
+// SIMD-ONLY0-NEXT:    [[TMP542:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP542]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1312]]
+// SIMD-ONLY0:       if.end1312:
+// SIMD-ONLY0-NEXT:    [[TMP543:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1313:%.*]] = sext i16 [[TMP543]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP544:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1314:%.*]] = sext i16 [[TMP544]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1315:%.*]] = icmp eq i32 [[CONV1313]], [[CONV1314]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1315]], label [[IF_THEN1317:%.*]], label [[IF_END1318:%.*]]
+// SIMD-ONLY0:       if.then1317:
+// SIMD-ONLY0-NEXT:    [[TMP545:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP545]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1318]]
+// SIMD-ONLY0:       if.end1318:
+// SIMD-ONLY0-NEXT:    [[TMP546:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1319:%.*]] = zext i16 [[TMP546]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP547:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1320:%.*]] = zext i16 [[TMP547]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1321:%.*]] = icmp sgt i32 [[CONV1319]], [[CONV1320]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1321]], label [[COND_TRUE1323:%.*]], label [[COND_FALSE1325:%.*]]
+// SIMD-ONLY0:       cond.true1323:
+// SIMD-ONLY0-NEXT:    [[TMP548:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1324:%.*]] = zext i16 [[TMP548]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1327:%.*]]
+// SIMD-ONLY0:       cond.false1325:
+// SIMD-ONLY0-NEXT:    [[TMP549:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1326:%.*]] = zext i16 [[TMP549]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1327]]
+// SIMD-ONLY0:       cond.end1327:
+// SIMD-ONLY0-NEXT:    [[COND1328:%.*]] = phi i32 [ [[CONV1324]], [[COND_TRUE1323]] ], [ [[CONV1326]], [[COND_FALSE1325]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1329:%.*]] = trunc i32 [[COND1328]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1329]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP550:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1330:%.*]] = zext i16 [[TMP550]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP551:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1331:%.*]] = zext i16 [[TMP551]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1332:%.*]] = icmp slt i32 [[CONV1330]], [[CONV1331]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1332]], label [[COND_TRUE1334:%.*]], label [[COND_FALSE1336:%.*]]
+// SIMD-ONLY0:       cond.true1334:
+// SIMD-ONLY0-NEXT:    [[TMP552:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1335:%.*]] = zext i16 [[TMP552]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1338:%.*]]
+// SIMD-ONLY0:       cond.false1336:
+// SIMD-ONLY0-NEXT:    [[TMP553:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1337:%.*]] = zext i16 [[TMP553]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1338]]
+// SIMD-ONLY0:       cond.end1338:
+// SIMD-ONLY0-NEXT:    [[COND1339:%.*]] = phi i32 [ [[CONV1335]], [[COND_TRUE1334]] ], [ [[CONV1337]], [[COND_FALSE1336]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1340:%.*]] = trunc i32 [[COND1339]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1340]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP554:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1341:%.*]] = zext i16 [[TMP554]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP555:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1342:%.*]] = zext i16 [[TMP555]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1343:%.*]] = icmp sgt i32 [[CONV1341]], [[CONV1342]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1343]], label [[COND_TRUE1345:%.*]], label [[COND_FALSE1347:%.*]]
+// SIMD-ONLY0:       cond.true1345:
+// SIMD-ONLY0-NEXT:    [[TMP556:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1346:%.*]] = zext i16 [[TMP556]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1349:%.*]]
+// SIMD-ONLY0:       cond.false1347:
+// SIMD-ONLY0-NEXT:    [[TMP557:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1348:%.*]] = zext i16 [[TMP557]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1349]]
+// SIMD-ONLY0:       cond.end1349:
+// SIMD-ONLY0-NEXT:    [[COND1350:%.*]] = phi i32 [ [[CONV1346]], [[COND_TRUE1345]] ], [ [[CONV1348]], [[COND_FALSE1347]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1351:%.*]] = trunc i32 [[COND1350]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1351]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP558:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1352:%.*]] = zext i16 [[TMP558]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP559:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1353:%.*]] = zext i16 [[TMP559]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1354:%.*]] = icmp slt i32 [[CONV1352]], [[CONV1353]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1354]], label [[COND_TRUE1356:%.*]], label [[COND_FALSE1358:%.*]]
+// SIMD-ONLY0:       cond.true1356:
+// SIMD-ONLY0-NEXT:    [[TMP560:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1357:%.*]] = zext i16 [[TMP560]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1360:%.*]]
+// SIMD-ONLY0:       cond.false1358:
+// SIMD-ONLY0-NEXT:    [[TMP561:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1359:%.*]] = zext i16 [[TMP561]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1360]]
+// SIMD-ONLY0:       cond.end1360:
+// SIMD-ONLY0-NEXT:    [[COND1361:%.*]] = phi i32 [ [[CONV1357]], [[COND_TRUE1356]] ], [ [[CONV1359]], [[COND_FALSE1358]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1362:%.*]] = trunc i32 [[COND1361]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1362]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP562:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1363:%.*]] = zext i16 [[TMP562]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP563:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1364:%.*]] = zext i16 [[TMP563]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1365:%.*]] = icmp sgt i32 [[CONV1363]], [[CONV1364]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1365]], label [[IF_THEN1367:%.*]], label [[IF_END1368:%.*]]
+// SIMD-ONLY0:       if.then1367:
+// SIMD-ONLY0-NEXT:    [[TMP564:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP564]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1368]]
+// SIMD-ONLY0:       if.end1368:
+// SIMD-ONLY0-NEXT:    [[TMP565:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1369:%.*]] = zext i16 [[TMP565]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP566:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1370:%.*]] = zext i16 [[TMP566]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1371:%.*]] = icmp slt i32 [[CONV1369]], [[CONV1370]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1371]], label [[IF_THEN1373:%.*]], label [[IF_END1374:%.*]]
+// SIMD-ONLY0:       if.then1373:
+// SIMD-ONLY0-NEXT:    [[TMP567:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP567]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1374]]
+// SIMD-ONLY0:       if.end1374:
+// SIMD-ONLY0-NEXT:    [[TMP568:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1375:%.*]] = zext i16 [[TMP568]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP569:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1376:%.*]] = zext i16 [[TMP569]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1377:%.*]] = icmp sgt i32 [[CONV1375]], [[CONV1376]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1377]], label [[IF_THEN1379:%.*]], label [[IF_END1380:%.*]]
+// SIMD-ONLY0:       if.then1379:
+// SIMD-ONLY0-NEXT:    [[TMP570:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP570]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1380]]
+// SIMD-ONLY0:       if.end1380:
+// SIMD-ONLY0-NEXT:    [[TMP571:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1381:%.*]] = zext i16 [[TMP571]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP572:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1382:%.*]] = zext i16 [[TMP572]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1383:%.*]] = icmp slt i32 [[CONV1381]], [[CONV1382]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1383]], label [[IF_THEN1385:%.*]], label [[IF_END1386:%.*]]
+// SIMD-ONLY0:       if.then1385:
+// SIMD-ONLY0-NEXT:    [[TMP573:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP573]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1386]]
+// SIMD-ONLY0:       if.end1386:
+// SIMD-ONLY0-NEXT:    [[TMP574:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1387:%.*]] = zext i16 [[TMP574]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP575:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1388:%.*]] = zext i16 [[TMP575]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1389:%.*]] = icmp eq i32 [[CONV1387]], [[CONV1388]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1389]], label [[COND_TRUE1391:%.*]], label [[COND_FALSE1393:%.*]]
+// SIMD-ONLY0:       cond.true1391:
+// SIMD-ONLY0-NEXT:    [[TMP576:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1392:%.*]] = zext i16 [[TMP576]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1395:%.*]]
+// SIMD-ONLY0:       cond.false1393:
+// SIMD-ONLY0-NEXT:    [[TMP577:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1394:%.*]] = zext i16 [[TMP577]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1395]]
+// SIMD-ONLY0:       cond.end1395:
+// SIMD-ONLY0-NEXT:    [[COND1396:%.*]] = phi i32 [ [[CONV1392]], [[COND_TRUE1391]] ], [ [[CONV1394]], [[COND_FALSE1393]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1397:%.*]] = trunc i32 [[COND1396]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1397]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP578:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1398:%.*]] = zext i16 [[TMP578]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP579:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1399:%.*]] = zext i16 [[TMP579]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1400:%.*]] = icmp eq i32 [[CONV1398]], [[CONV1399]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1400]], label [[COND_TRUE1402:%.*]], label [[COND_FALSE1404:%.*]]
+// SIMD-ONLY0:       cond.true1402:
+// SIMD-ONLY0-NEXT:    [[TMP580:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1403:%.*]] = zext i16 [[TMP580]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1406:%.*]]
+// SIMD-ONLY0:       cond.false1404:
+// SIMD-ONLY0-NEXT:    [[TMP581:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1405:%.*]] = zext i16 [[TMP581]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1406]]
+// SIMD-ONLY0:       cond.end1406:
+// SIMD-ONLY0-NEXT:    [[COND1407:%.*]] = phi i32 [ [[CONV1403]], [[COND_TRUE1402]] ], [ [[CONV1405]], [[COND_FALSE1404]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1408:%.*]] = trunc i32 [[COND1407]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1408]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP582:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1409:%.*]] = zext i16 [[TMP582]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP583:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1410:%.*]] = zext i16 [[TMP583]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1411:%.*]] = icmp eq i32 [[CONV1409]], [[CONV1410]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1411]], label [[IF_THEN1413:%.*]], label [[IF_END1414:%.*]]
+// SIMD-ONLY0:       if.then1413:
+// SIMD-ONLY0-NEXT:    [[TMP584:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP584]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1414]]
+// SIMD-ONLY0:       if.end1414:
+// SIMD-ONLY0-NEXT:    [[TMP585:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1415:%.*]] = zext i16 [[TMP585]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP586:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1416:%.*]] = zext i16 [[TMP586]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1417:%.*]] = icmp eq i32 [[CONV1415]], [[CONV1416]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1417]], label [[IF_THEN1419:%.*]], label [[IF_END1420:%.*]]
+// SIMD-ONLY0:       if.then1419:
+// SIMD-ONLY0-NEXT:    [[TMP587:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP587]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1420]]
+// SIMD-ONLY0:       if.end1420:
+// SIMD-ONLY0-NEXT:    [[TMP588:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1421:%.*]] = sext i16 [[TMP588]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP589:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1422:%.*]] = sext i16 [[TMP589]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1423:%.*]] = icmp sgt i32 [[CONV1421]], [[CONV1422]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1423]], label [[COND_TRUE1425:%.*]], label [[COND_FALSE1427:%.*]]
+// SIMD-ONLY0:       cond.true1425:
+// SIMD-ONLY0-NEXT:    [[TMP590:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1426:%.*]] = sext i16 [[TMP590]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1429:%.*]]
+// SIMD-ONLY0:       cond.false1427:
+// SIMD-ONLY0-NEXT:    [[TMP591:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1428:%.*]] = sext i16 [[TMP591]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1429]]
+// SIMD-ONLY0:       cond.end1429:
+// SIMD-ONLY0-NEXT:    [[COND1430:%.*]] = phi i32 [ [[CONV1426]], [[COND_TRUE1425]] ], [ [[CONV1428]], [[COND_FALSE1427]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1431:%.*]] = trunc i32 [[COND1430]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1431]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP592:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1432:%.*]] = sext i16 [[TMP592]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP593:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1433:%.*]] = sext i16 [[TMP593]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1434:%.*]] = icmp slt i32 [[CONV1432]], [[CONV1433]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1434]], label [[COND_TRUE1436:%.*]], label [[COND_FALSE1438:%.*]]
+// SIMD-ONLY0:       cond.true1436:
+// SIMD-ONLY0-NEXT:    [[TMP594:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1437:%.*]] = sext i16 [[TMP594]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1440:%.*]]
+// SIMD-ONLY0:       cond.false1438:
+// SIMD-ONLY0-NEXT:    [[TMP595:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1439:%.*]] = sext i16 [[TMP595]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1440]]
+// SIMD-ONLY0:       cond.end1440:
+// SIMD-ONLY0-NEXT:    [[COND1441:%.*]] = phi i32 [ [[CONV1437]], [[COND_TRUE1436]] ], [ [[CONV1439]], [[COND_FALSE1438]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1442:%.*]] = trunc i32 [[COND1441]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1442]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP596:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1443:%.*]] = sext i16 [[TMP596]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP597:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1444:%.*]] = sext i16 [[TMP597]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1445:%.*]] = icmp sgt i32 [[CONV1443]], [[CONV1444]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1445]], label [[COND_TRUE1447:%.*]], label [[COND_FALSE1449:%.*]]
+// SIMD-ONLY0:       cond.true1447:
+// SIMD-ONLY0-NEXT:    [[TMP598:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1448:%.*]] = sext i16 [[TMP598]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1451:%.*]]
+// SIMD-ONLY0:       cond.false1449:
+// SIMD-ONLY0-NEXT:    [[TMP599:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1450:%.*]] = sext i16 [[TMP599]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1451]]
+// SIMD-ONLY0:       cond.end1451:
+// SIMD-ONLY0-NEXT:    [[COND1452:%.*]] = phi i32 [ [[CONV1448]], [[COND_TRUE1447]] ], [ [[CONV1450]], [[COND_FALSE1449]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1453:%.*]] = trunc i32 [[COND1452]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1453]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP600:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1454:%.*]] = sext i16 [[TMP600]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP601:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1455:%.*]] = sext i16 [[TMP601]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1456:%.*]] = icmp slt i32 [[CONV1454]], [[CONV1455]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1456]], label [[COND_TRUE1458:%.*]], label [[COND_FALSE1460:%.*]]
+// SIMD-ONLY0:       cond.true1458:
+// SIMD-ONLY0-NEXT:    [[TMP602:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1459:%.*]] = sext i16 [[TMP602]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1462:%.*]]
+// SIMD-ONLY0:       cond.false1460:
+// SIMD-ONLY0-NEXT:    [[TMP603:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1461:%.*]] = sext i16 [[TMP603]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1462]]
+// SIMD-ONLY0:       cond.end1462:
+// SIMD-ONLY0-NEXT:    [[COND1463:%.*]] = phi i32 [ [[CONV1459]], [[COND_TRUE1458]] ], [ [[CONV1461]], [[COND_FALSE1460]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1464:%.*]] = trunc i32 [[COND1463]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1464]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP604:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1465:%.*]] = sext i16 [[TMP604]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP605:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1466:%.*]] = sext i16 [[TMP605]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1467:%.*]] = icmp sgt i32 [[CONV1465]], [[CONV1466]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1467]], label [[IF_THEN1469:%.*]], label [[IF_END1470:%.*]]
+// SIMD-ONLY0:       if.then1469:
+// SIMD-ONLY0-NEXT:    [[TMP606:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP606]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1470]]
+// SIMD-ONLY0:       if.end1470:
+// SIMD-ONLY0-NEXT:    [[TMP607:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1471:%.*]] = sext i16 [[TMP607]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP608:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1472:%.*]] = sext i16 [[TMP608]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1473:%.*]] = icmp slt i32 [[CONV1471]], [[CONV1472]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1473]], label [[IF_THEN1475:%.*]], label [[IF_END1476:%.*]]
+// SIMD-ONLY0:       if.then1475:
+// SIMD-ONLY0-NEXT:    [[TMP609:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP609]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1476]]
+// SIMD-ONLY0:       if.end1476:
+// SIMD-ONLY0-NEXT:    [[TMP610:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1477:%.*]] = sext i16 [[TMP610]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP611:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1478:%.*]] = sext i16 [[TMP611]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1479:%.*]] = icmp sgt i32 [[CONV1477]], [[CONV1478]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1479]], label [[IF_THEN1481:%.*]], label [[IF_END1482:%.*]]
+// SIMD-ONLY0:       if.then1481:
+// SIMD-ONLY0-NEXT:    [[TMP612:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP612]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1482]]
+// SIMD-ONLY0:       if.end1482:
+// SIMD-ONLY0-NEXT:    [[TMP613:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1483:%.*]] = sext i16 [[TMP613]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP614:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1484:%.*]] = sext i16 [[TMP614]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1485:%.*]] = icmp slt i32 [[CONV1483]], [[CONV1484]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1485]], label [[IF_THEN1487:%.*]], label [[IF_END1488:%.*]]
+// SIMD-ONLY0:       if.then1487:
+// SIMD-ONLY0-NEXT:    [[TMP615:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP615]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1488]]
+// SIMD-ONLY0:       if.end1488:
+// SIMD-ONLY0-NEXT:    [[TMP616:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1489:%.*]] = sext i16 [[TMP616]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP617:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1490:%.*]] = sext i16 [[TMP617]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1491:%.*]] = icmp eq i32 [[CONV1489]], [[CONV1490]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1491]], label [[COND_TRUE1493:%.*]], label [[COND_FALSE1495:%.*]]
+// SIMD-ONLY0:       cond.true1493:
+// SIMD-ONLY0-NEXT:    [[TMP618:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1494:%.*]] = sext i16 [[TMP618]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1497:%.*]]
+// SIMD-ONLY0:       cond.false1495:
+// SIMD-ONLY0-NEXT:    [[TMP619:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1496:%.*]] = sext i16 [[TMP619]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1497]]
+// SIMD-ONLY0:       cond.end1497:
+// SIMD-ONLY0-NEXT:    [[COND1498:%.*]] = phi i32 [ [[CONV1494]], [[COND_TRUE1493]] ], [ [[CONV1496]], [[COND_FALSE1495]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1499:%.*]] = trunc i32 [[COND1498]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1499]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP620:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1500:%.*]] = sext i16 [[TMP620]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP621:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1501:%.*]] = sext i16 [[TMP621]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1502:%.*]] = icmp eq i32 [[CONV1500]], [[CONV1501]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1502]], label [[COND_TRUE1504:%.*]], label [[COND_FALSE1506:%.*]]
+// SIMD-ONLY0:       cond.true1504:
+// SIMD-ONLY0-NEXT:    [[TMP622:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1505:%.*]] = sext i16 [[TMP622]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1508:%.*]]
+// SIMD-ONLY0:       cond.false1506:
+// SIMD-ONLY0-NEXT:    [[TMP623:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1507:%.*]] = sext i16 [[TMP623]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1508]]
+// SIMD-ONLY0:       cond.end1508:
+// SIMD-ONLY0-NEXT:    [[COND1509:%.*]] = phi i32 [ [[CONV1505]], [[COND_TRUE1504]] ], [ [[CONV1507]], [[COND_FALSE1506]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1510:%.*]] = trunc i32 [[COND1509]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1510]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP624:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1511:%.*]] = sext i16 [[TMP624]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP625:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1512:%.*]] = sext i16 [[TMP625]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1513:%.*]] = icmp eq i32 [[CONV1511]], [[CONV1512]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1513]], label [[IF_THEN1515:%.*]], label [[IF_END1516:%.*]]
+// SIMD-ONLY0:       if.then1515:
+// SIMD-ONLY0-NEXT:    [[TMP626:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP626]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1516]]
+// SIMD-ONLY0:       if.end1516:
+// SIMD-ONLY0-NEXT:    [[TMP627:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1517:%.*]] = sext i16 [[TMP627]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP628:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1518:%.*]] = sext i16 [[TMP628]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1519:%.*]] = icmp eq i32 [[CONV1517]], [[CONV1518]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1519]], label [[IF_THEN1521:%.*]], label [[IF_END1522:%.*]]
+// SIMD-ONLY0:       if.then1521:
+// SIMD-ONLY0-NEXT:    [[TMP629:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP629]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1522]]
+// SIMD-ONLY0:       if.end1522:
+// SIMD-ONLY0-NEXT:    [[TMP630:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1523:%.*]] = zext i16 [[TMP630]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP631:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1524:%.*]] = zext i16 [[TMP631]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1525:%.*]] = icmp sgt i32 [[CONV1523]], [[CONV1524]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1525]], label [[COND_TRUE1527:%.*]], label [[COND_FALSE1529:%.*]]
+// SIMD-ONLY0:       cond.true1527:
+// SIMD-ONLY0-NEXT:    [[TMP632:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1528:%.*]] = zext i16 [[TMP632]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1531:%.*]]
+// SIMD-ONLY0:       cond.false1529:
+// SIMD-ONLY0-NEXT:    [[TMP633:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1530:%.*]] = zext i16 [[TMP633]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1531]]
+// SIMD-ONLY0:       cond.end1531:
+// SIMD-ONLY0-NEXT:    [[COND1532:%.*]] = phi i32 [ [[CONV1528]], [[COND_TRUE1527]] ], [ [[CONV1530]], [[COND_FALSE1529]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1533:%.*]] = trunc i32 [[COND1532]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1533]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP634:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1534:%.*]] = zext i16 [[TMP634]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP635:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1535:%.*]] = zext i16 [[TMP635]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1536:%.*]] = icmp slt i32 [[CONV1534]], [[CONV1535]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1536]], label [[COND_TRUE1538:%.*]], label [[COND_FALSE1540:%.*]]
+// SIMD-ONLY0:       cond.true1538:
+// SIMD-ONLY0-NEXT:    [[TMP636:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1539:%.*]] = zext i16 [[TMP636]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1542:%.*]]
+// SIMD-ONLY0:       cond.false1540:
+// SIMD-ONLY0-NEXT:    [[TMP637:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1541:%.*]] = zext i16 [[TMP637]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1542]]
+// SIMD-ONLY0:       cond.end1542:
+// SIMD-ONLY0-NEXT:    [[COND1543:%.*]] = phi i32 [ [[CONV1539]], [[COND_TRUE1538]] ], [ [[CONV1541]], [[COND_FALSE1540]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1544:%.*]] = trunc i32 [[COND1543]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1544]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP638:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1545:%.*]] = zext i16 [[TMP638]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP639:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1546:%.*]] = zext i16 [[TMP639]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1547:%.*]] = icmp sgt i32 [[CONV1545]], [[CONV1546]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1547]], label [[COND_TRUE1549:%.*]], label [[COND_FALSE1551:%.*]]
+// SIMD-ONLY0:       cond.true1549:
+// SIMD-ONLY0-NEXT:    [[TMP640:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1550:%.*]] = zext i16 [[TMP640]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1553:%.*]]
+// SIMD-ONLY0:       cond.false1551:
+// SIMD-ONLY0-NEXT:    [[TMP641:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1552:%.*]] = zext i16 [[TMP641]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1553]]
+// SIMD-ONLY0:       cond.end1553:
+// SIMD-ONLY0-NEXT:    [[COND1554:%.*]] = phi i32 [ [[CONV1550]], [[COND_TRUE1549]] ], [ [[CONV1552]], [[COND_FALSE1551]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1555:%.*]] = trunc i32 [[COND1554]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1555]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP642:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1556:%.*]] = zext i16 [[TMP642]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP643:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1557:%.*]] = zext i16 [[TMP643]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1558:%.*]] = icmp slt i32 [[CONV1556]], [[CONV1557]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1558]], label [[COND_TRUE1560:%.*]], label [[COND_FALSE1562:%.*]]
+// SIMD-ONLY0:       cond.true1560:
+// SIMD-ONLY0-NEXT:    [[TMP644:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1561:%.*]] = zext i16 [[TMP644]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1564:%.*]]
+// SIMD-ONLY0:       cond.false1562:
+// SIMD-ONLY0-NEXT:    [[TMP645:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1563:%.*]] = zext i16 [[TMP645]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1564]]
+// SIMD-ONLY0:       cond.end1564:
+// SIMD-ONLY0-NEXT:    [[COND1565:%.*]] = phi i32 [ [[CONV1561]], [[COND_TRUE1560]] ], [ [[CONV1563]], [[COND_FALSE1562]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1566:%.*]] = trunc i32 [[COND1565]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1566]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP646:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1567:%.*]] = zext i16 [[TMP646]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP647:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1568:%.*]] = zext i16 [[TMP647]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1569:%.*]] = icmp sgt i32 [[CONV1567]], [[CONV1568]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1569]], label [[IF_THEN1571:%.*]], label [[IF_END1572:%.*]]
+// SIMD-ONLY0:       if.then1571:
+// SIMD-ONLY0-NEXT:    [[TMP648:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP648]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1572]]
+// SIMD-ONLY0:       if.end1572:
+// SIMD-ONLY0-NEXT:    [[TMP649:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1573:%.*]] = zext i16 [[TMP649]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP650:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1574:%.*]] = zext i16 [[TMP650]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1575:%.*]] = icmp slt i32 [[CONV1573]], [[CONV1574]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1575]], label [[IF_THEN1577:%.*]], label [[IF_END1578:%.*]]
+// SIMD-ONLY0:       if.then1577:
+// SIMD-ONLY0-NEXT:    [[TMP651:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP651]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1578]]
+// SIMD-ONLY0:       if.end1578:
+// SIMD-ONLY0-NEXT:    [[TMP652:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1579:%.*]] = zext i16 [[TMP652]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP653:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1580:%.*]] = zext i16 [[TMP653]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1581:%.*]] = icmp sgt i32 [[CONV1579]], [[CONV1580]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1581]], label [[IF_THEN1583:%.*]], label [[IF_END1584:%.*]]
+// SIMD-ONLY0:       if.then1583:
+// SIMD-ONLY0-NEXT:    [[TMP654:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP654]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1584]]
+// SIMD-ONLY0:       if.end1584:
+// SIMD-ONLY0-NEXT:    [[TMP655:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1585:%.*]] = zext i16 [[TMP655]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP656:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1586:%.*]] = zext i16 [[TMP656]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1587:%.*]] = icmp slt i32 [[CONV1585]], [[CONV1586]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1587]], label [[IF_THEN1589:%.*]], label [[IF_END1590:%.*]]
+// SIMD-ONLY0:       if.then1589:
+// SIMD-ONLY0-NEXT:    [[TMP657:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP657]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1590]]
+// SIMD-ONLY0:       if.end1590:
+// SIMD-ONLY0-NEXT:    [[TMP658:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1591:%.*]] = zext i16 [[TMP658]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP659:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1592:%.*]] = zext i16 [[TMP659]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1593:%.*]] = icmp eq i32 [[CONV1591]], [[CONV1592]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1593]], label [[COND_TRUE1595:%.*]], label [[COND_FALSE1597:%.*]]
+// SIMD-ONLY0:       cond.true1595:
+// SIMD-ONLY0-NEXT:    [[TMP660:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1596:%.*]] = zext i16 [[TMP660]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1599:%.*]]
+// SIMD-ONLY0:       cond.false1597:
+// SIMD-ONLY0-NEXT:    [[TMP661:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1598:%.*]] = zext i16 [[TMP661]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1599]]
+// SIMD-ONLY0:       cond.end1599:
+// SIMD-ONLY0-NEXT:    [[COND1600:%.*]] = phi i32 [ [[CONV1596]], [[COND_TRUE1595]] ], [ [[CONV1598]], [[COND_FALSE1597]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1601:%.*]] = trunc i32 [[COND1600]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1601]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP662:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1602:%.*]] = zext i16 [[TMP662]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP663:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1603:%.*]] = zext i16 [[TMP663]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1604:%.*]] = icmp eq i32 [[CONV1602]], [[CONV1603]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1604]], label [[COND_TRUE1606:%.*]], label [[COND_FALSE1608:%.*]]
+// SIMD-ONLY0:       cond.true1606:
+// SIMD-ONLY0-NEXT:    [[TMP664:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1607:%.*]] = zext i16 [[TMP664]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1610:%.*]]
+// SIMD-ONLY0:       cond.false1608:
+// SIMD-ONLY0-NEXT:    [[TMP665:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1609:%.*]] = zext i16 [[TMP665]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1610]]
+// SIMD-ONLY0:       cond.end1610:
+// SIMD-ONLY0-NEXT:    [[COND1611:%.*]] = phi i32 [ [[CONV1607]], [[COND_TRUE1606]] ], [ [[CONV1609]], [[COND_FALSE1608]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1612:%.*]] = trunc i32 [[COND1611]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1612]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP666:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1613:%.*]] = zext i16 [[TMP666]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP667:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1614:%.*]] = zext i16 [[TMP667]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1615:%.*]] = icmp eq i32 [[CONV1613]], [[CONV1614]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1615]], label [[IF_THEN1617:%.*]], label [[IF_END1618:%.*]]
+// SIMD-ONLY0:       if.then1617:
+// SIMD-ONLY0-NEXT:    [[TMP668:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP668]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1618]]
+// SIMD-ONLY0:       if.end1618:
+// SIMD-ONLY0-NEXT:    [[TMP669:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1619:%.*]] = zext i16 [[TMP669]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP670:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1620:%.*]] = zext i16 [[TMP670]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1621:%.*]] = icmp eq i32 [[CONV1619]], [[CONV1620]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1621]], label [[IF_THEN1623:%.*]], label [[IF_END1624:%.*]]
+// SIMD-ONLY0:       if.then1623:
+// SIMD-ONLY0-NEXT:    [[TMP671:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP671]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1624]]
+// SIMD-ONLY0:       if.end1624:
+// SIMD-ONLY0-NEXT:    [[TMP672:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1625:%.*]] = sext i16 [[TMP672]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP673:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1626:%.*]] = sext i16 [[TMP673]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1627:%.*]] = icmp sgt i32 [[CONV1625]], [[CONV1626]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1627]], label [[COND_TRUE1629:%.*]], label [[COND_FALSE1631:%.*]]
+// SIMD-ONLY0:       cond.true1629:
+// SIMD-ONLY0-NEXT:    [[TMP674:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1630:%.*]] = sext i16 [[TMP674]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1633:%.*]]
+// SIMD-ONLY0:       cond.false1631:
+// SIMD-ONLY0-NEXT:    [[TMP675:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1632:%.*]] = sext i16 [[TMP675]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1633]]
+// SIMD-ONLY0:       cond.end1633:
+// SIMD-ONLY0-NEXT:    [[COND1634:%.*]] = phi i32 [ [[CONV1630]], [[COND_TRUE1629]] ], [ [[CONV1632]], [[COND_FALSE1631]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1635:%.*]] = trunc i32 [[COND1634]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1635]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP676:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1636:%.*]] = sext i16 [[TMP676]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP677:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1637:%.*]] = sext i16 [[TMP677]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1638:%.*]] = icmp slt i32 [[CONV1636]], [[CONV1637]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1638]], label [[COND_TRUE1640:%.*]], label [[COND_FALSE1642:%.*]]
+// SIMD-ONLY0:       cond.true1640:
+// SIMD-ONLY0-NEXT:    [[TMP678:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1641:%.*]] = sext i16 [[TMP678]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1644:%.*]]
+// SIMD-ONLY0:       cond.false1642:
+// SIMD-ONLY0-NEXT:    [[TMP679:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1643:%.*]] = sext i16 [[TMP679]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1644]]
+// SIMD-ONLY0:       cond.end1644:
+// SIMD-ONLY0-NEXT:    [[COND1645:%.*]] = phi i32 [ [[CONV1641]], [[COND_TRUE1640]] ], [ [[CONV1643]], [[COND_FALSE1642]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1646:%.*]] = trunc i32 [[COND1645]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1646]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP680:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1647:%.*]] = sext i16 [[TMP680]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP681:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1648:%.*]] = sext i16 [[TMP681]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1649:%.*]] = icmp sgt i32 [[CONV1647]], [[CONV1648]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1649]], label [[COND_TRUE1651:%.*]], label [[COND_FALSE1653:%.*]]
+// SIMD-ONLY0:       cond.true1651:
+// SIMD-ONLY0-NEXT:    [[TMP682:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1652:%.*]] = sext i16 [[TMP682]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1655:%.*]]
+// SIMD-ONLY0:       cond.false1653:
+// SIMD-ONLY0-NEXT:    [[TMP683:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1654:%.*]] = sext i16 [[TMP683]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1655]]
+// SIMD-ONLY0:       cond.end1655:
+// SIMD-ONLY0-NEXT:    [[COND1656:%.*]] = phi i32 [ [[CONV1652]], [[COND_TRUE1651]] ], [ [[CONV1654]], [[COND_FALSE1653]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1657:%.*]] = trunc i32 [[COND1656]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1657]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP684:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1658:%.*]] = sext i16 [[TMP684]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP685:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1659:%.*]] = sext i16 [[TMP685]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1660:%.*]] = icmp slt i32 [[CONV1658]], [[CONV1659]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1660]], label [[COND_TRUE1662:%.*]], label [[COND_FALSE1664:%.*]]
+// SIMD-ONLY0:       cond.true1662:
+// SIMD-ONLY0-NEXT:    [[TMP686:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1663:%.*]] = sext i16 [[TMP686]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1666:%.*]]
+// SIMD-ONLY0:       cond.false1664:
+// SIMD-ONLY0-NEXT:    [[TMP687:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1665:%.*]] = sext i16 [[TMP687]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1666]]
+// SIMD-ONLY0:       cond.end1666:
+// SIMD-ONLY0-NEXT:    [[COND1667:%.*]] = phi i32 [ [[CONV1663]], [[COND_TRUE1662]] ], [ [[CONV1665]], [[COND_FALSE1664]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1668:%.*]] = trunc i32 [[COND1667]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1668]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP688:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1669:%.*]] = sext i16 [[TMP688]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP689:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1670:%.*]] = sext i16 [[TMP689]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1671:%.*]] = icmp sgt i32 [[CONV1669]], [[CONV1670]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1671]], label [[IF_THEN1673:%.*]], label [[IF_END1674:%.*]]
+// SIMD-ONLY0:       if.then1673:
+// SIMD-ONLY0-NEXT:    [[TMP690:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP690]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1674]]
+// SIMD-ONLY0:       if.end1674:
+// SIMD-ONLY0-NEXT:    [[TMP691:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1675:%.*]] = sext i16 [[TMP691]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP692:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1676:%.*]] = sext i16 [[TMP692]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1677:%.*]] = icmp slt i32 [[CONV1675]], [[CONV1676]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1677]], label [[IF_THEN1679:%.*]], label [[IF_END1680:%.*]]
+// SIMD-ONLY0:       if.then1679:
+// SIMD-ONLY0-NEXT:    [[TMP693:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP693]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1680]]
+// SIMD-ONLY0:       if.end1680:
+// SIMD-ONLY0-NEXT:    [[TMP694:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1681:%.*]] = sext i16 [[TMP694]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP695:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1682:%.*]] = sext i16 [[TMP695]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1683:%.*]] = icmp sgt i32 [[CONV1681]], [[CONV1682]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1683]], label [[IF_THEN1685:%.*]], label [[IF_END1686:%.*]]
+// SIMD-ONLY0:       if.then1685:
+// SIMD-ONLY0-NEXT:    [[TMP696:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP696]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1686]]
+// SIMD-ONLY0:       if.end1686:
+// SIMD-ONLY0-NEXT:    [[TMP697:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1687:%.*]] = sext i16 [[TMP697]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP698:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1688:%.*]] = sext i16 [[TMP698]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1689:%.*]] = icmp slt i32 [[CONV1687]], [[CONV1688]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1689]], label [[IF_THEN1691:%.*]], label [[IF_END1692:%.*]]
+// SIMD-ONLY0:       if.then1691:
+// SIMD-ONLY0-NEXT:    [[TMP699:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP699]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1692]]
+// SIMD-ONLY0:       if.end1692:
+// SIMD-ONLY0-NEXT:    [[TMP700:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1693:%.*]] = sext i16 [[TMP700]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP701:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1694:%.*]] = sext i16 [[TMP701]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1695:%.*]] = icmp eq i32 [[CONV1693]], [[CONV1694]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1695]], label [[COND_TRUE1697:%.*]], label [[COND_FALSE1699:%.*]]
+// SIMD-ONLY0:       cond.true1697:
+// SIMD-ONLY0-NEXT:    [[TMP702:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1698:%.*]] = sext i16 [[TMP702]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1701:%.*]]
+// SIMD-ONLY0:       cond.false1699:
+// SIMD-ONLY0-NEXT:    [[TMP703:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1700:%.*]] = sext i16 [[TMP703]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1701]]
+// SIMD-ONLY0:       cond.end1701:
+// SIMD-ONLY0-NEXT:    [[COND1702:%.*]] = phi i32 [ [[CONV1698]], [[COND_TRUE1697]] ], [ [[CONV1700]], [[COND_FALSE1699]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1703:%.*]] = trunc i32 [[COND1702]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1703]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP704:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1704:%.*]] = sext i16 [[TMP704]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP705:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1705:%.*]] = sext i16 [[TMP705]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1706:%.*]] = icmp eq i32 [[CONV1704]], [[CONV1705]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1706]], label [[COND_TRUE1708:%.*]], label [[COND_FALSE1710:%.*]]
+// SIMD-ONLY0:       cond.true1708:
+// SIMD-ONLY0-NEXT:    [[TMP706:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1709:%.*]] = sext i16 [[TMP706]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1712:%.*]]
+// SIMD-ONLY0:       cond.false1710:
+// SIMD-ONLY0-NEXT:    [[TMP707:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1711:%.*]] = sext i16 [[TMP707]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1712]]
+// SIMD-ONLY0:       cond.end1712:
+// SIMD-ONLY0-NEXT:    [[COND1713:%.*]] = phi i32 [ [[CONV1709]], [[COND_TRUE1708]] ], [ [[CONV1711]], [[COND_FALSE1710]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1714:%.*]] = trunc i32 [[COND1713]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1714]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP708:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1715:%.*]] = sext i16 [[TMP708]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP709:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1716:%.*]] = sext i16 [[TMP709]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1717:%.*]] = icmp eq i32 [[CONV1715]], [[CONV1716]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1717]], label [[IF_THEN1719:%.*]], label [[IF_END1720:%.*]]
+// SIMD-ONLY0:       if.then1719:
+// SIMD-ONLY0-NEXT:    [[TMP710:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP710]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1720]]
+// SIMD-ONLY0:       if.end1720:
+// SIMD-ONLY0-NEXT:    [[TMP711:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1721:%.*]] = sext i16 [[TMP711]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP712:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1722:%.*]] = sext i16 [[TMP712]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1723:%.*]] = icmp eq i32 [[CONV1721]], [[CONV1722]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1723]], label [[IF_THEN1725:%.*]], label [[IF_END1726:%.*]]
+// SIMD-ONLY0:       if.then1725:
+// SIMD-ONLY0-NEXT:    [[TMP713:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP713]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1726]]
+// SIMD-ONLY0:       if.end1726:
+// SIMD-ONLY0-NEXT:    [[TMP714:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1727:%.*]] = zext i16 [[TMP714]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP715:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1728:%.*]] = zext i16 [[TMP715]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1729:%.*]] = icmp sgt i32 [[CONV1727]], [[CONV1728]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1729]], label [[COND_TRUE1731:%.*]], label [[COND_FALSE1733:%.*]]
+// SIMD-ONLY0:       cond.true1731:
+// SIMD-ONLY0-NEXT:    [[TMP716:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1732:%.*]] = zext i16 [[TMP716]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1735:%.*]]
+// SIMD-ONLY0:       cond.false1733:
+// SIMD-ONLY0-NEXT:    [[TMP717:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1734:%.*]] = zext i16 [[TMP717]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1735]]
+// SIMD-ONLY0:       cond.end1735:
+// SIMD-ONLY0-NEXT:    [[COND1736:%.*]] = phi i32 [ [[CONV1732]], [[COND_TRUE1731]] ], [ [[CONV1734]], [[COND_FALSE1733]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1737:%.*]] = trunc i32 [[COND1736]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1737]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP718:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1738:%.*]] = zext i16 [[TMP718]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP719:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1739:%.*]] = zext i16 [[TMP719]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1740:%.*]] = icmp slt i32 [[CONV1738]], [[CONV1739]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1740]], label [[COND_TRUE1742:%.*]], label [[COND_FALSE1744:%.*]]
+// SIMD-ONLY0:       cond.true1742:
+// SIMD-ONLY0-NEXT:    [[TMP720:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1743:%.*]] = zext i16 [[TMP720]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1746:%.*]]
+// SIMD-ONLY0:       cond.false1744:
+// SIMD-ONLY0-NEXT:    [[TMP721:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1745:%.*]] = zext i16 [[TMP721]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1746]]
+// SIMD-ONLY0:       cond.end1746:
+// SIMD-ONLY0-NEXT:    [[COND1747:%.*]] = phi i32 [ [[CONV1743]], [[COND_TRUE1742]] ], [ [[CONV1745]], [[COND_FALSE1744]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1748:%.*]] = trunc i32 [[COND1747]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1748]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP722:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1749:%.*]] = zext i16 [[TMP722]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP723:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1750:%.*]] = zext i16 [[TMP723]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1751:%.*]] = icmp sgt i32 [[CONV1749]], [[CONV1750]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1751]], label [[COND_TRUE1753:%.*]], label [[COND_FALSE1755:%.*]]
+// SIMD-ONLY0:       cond.true1753:
+// SIMD-ONLY0-NEXT:    [[TMP724:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1754:%.*]] = zext i16 [[TMP724]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1757:%.*]]
+// SIMD-ONLY0:       cond.false1755:
+// SIMD-ONLY0-NEXT:    [[TMP725:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1756:%.*]] = zext i16 [[TMP725]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1757]]
+// SIMD-ONLY0:       cond.end1757:
+// SIMD-ONLY0-NEXT:    [[COND1758:%.*]] = phi i32 [ [[CONV1754]], [[COND_TRUE1753]] ], [ [[CONV1756]], [[COND_FALSE1755]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1759:%.*]] = trunc i32 [[COND1758]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1759]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP726:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1760:%.*]] = zext i16 [[TMP726]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP727:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1761:%.*]] = zext i16 [[TMP727]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1762:%.*]] = icmp slt i32 [[CONV1760]], [[CONV1761]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1762]], label [[COND_TRUE1764:%.*]], label [[COND_FALSE1766:%.*]]
+// SIMD-ONLY0:       cond.true1764:
+// SIMD-ONLY0-NEXT:    [[TMP728:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1765:%.*]] = zext i16 [[TMP728]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1768:%.*]]
+// SIMD-ONLY0:       cond.false1766:
+// SIMD-ONLY0-NEXT:    [[TMP729:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1767:%.*]] = zext i16 [[TMP729]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1768]]
+// SIMD-ONLY0:       cond.end1768:
+// SIMD-ONLY0-NEXT:    [[COND1769:%.*]] = phi i32 [ [[CONV1765]], [[COND_TRUE1764]] ], [ [[CONV1767]], [[COND_FALSE1766]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1770:%.*]] = trunc i32 [[COND1769]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1770]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP730:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1771:%.*]] = zext i16 [[TMP730]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP731:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1772:%.*]] = zext i16 [[TMP731]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1773:%.*]] = icmp sgt i32 [[CONV1771]], [[CONV1772]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1773]], label [[IF_THEN1775:%.*]], label [[IF_END1776:%.*]]
+// SIMD-ONLY0:       if.then1775:
+// SIMD-ONLY0-NEXT:    [[TMP732:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP732]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1776]]
+// SIMD-ONLY0:       if.end1776:
+// SIMD-ONLY0-NEXT:    [[TMP733:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1777:%.*]] = zext i16 [[TMP733]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP734:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1778:%.*]] = zext i16 [[TMP734]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1779:%.*]] = icmp slt i32 [[CONV1777]], [[CONV1778]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1779]], label [[IF_THEN1781:%.*]], label [[IF_END1782:%.*]]
+// SIMD-ONLY0:       if.then1781:
+// SIMD-ONLY0-NEXT:    [[TMP735:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP735]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1782]]
+// SIMD-ONLY0:       if.end1782:
+// SIMD-ONLY0-NEXT:    [[TMP736:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1783:%.*]] = zext i16 [[TMP736]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP737:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1784:%.*]] = zext i16 [[TMP737]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1785:%.*]] = icmp sgt i32 [[CONV1783]], [[CONV1784]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1785]], label [[IF_THEN1787:%.*]], label [[IF_END1788:%.*]]
+// SIMD-ONLY0:       if.then1787:
+// SIMD-ONLY0-NEXT:    [[TMP738:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP738]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1788]]
+// SIMD-ONLY0:       if.end1788:
+// SIMD-ONLY0-NEXT:    [[TMP739:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1789:%.*]] = zext i16 [[TMP739]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP740:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1790:%.*]] = zext i16 [[TMP740]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1791:%.*]] = icmp slt i32 [[CONV1789]], [[CONV1790]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1791]], label [[IF_THEN1793:%.*]], label [[IF_END1794:%.*]]
+// SIMD-ONLY0:       if.then1793:
+// SIMD-ONLY0-NEXT:    [[TMP741:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP741]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1794]]
+// SIMD-ONLY0:       if.end1794:
+// SIMD-ONLY0-NEXT:    [[TMP742:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1795:%.*]] = zext i16 [[TMP742]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP743:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1796:%.*]] = zext i16 [[TMP743]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1797:%.*]] = icmp eq i32 [[CONV1795]], [[CONV1796]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1797]], label [[COND_TRUE1799:%.*]], label [[COND_FALSE1801:%.*]]
+// SIMD-ONLY0:       cond.true1799:
+// SIMD-ONLY0-NEXT:    [[TMP744:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1800:%.*]] = zext i16 [[TMP744]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1803:%.*]]
+// SIMD-ONLY0:       cond.false1801:
+// SIMD-ONLY0-NEXT:    [[TMP745:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1802:%.*]] = zext i16 [[TMP745]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1803]]
+// SIMD-ONLY0:       cond.end1803:
+// SIMD-ONLY0-NEXT:    [[COND1804:%.*]] = phi i32 [ [[CONV1800]], [[COND_TRUE1799]] ], [ [[CONV1802]], [[COND_FALSE1801]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1805:%.*]] = trunc i32 [[COND1804]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1805]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP746:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1806:%.*]] = zext i16 [[TMP746]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP747:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1807:%.*]] = zext i16 [[TMP747]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1808:%.*]] = icmp eq i32 [[CONV1806]], [[CONV1807]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1808]], label [[COND_TRUE1810:%.*]], label [[COND_FALSE1812:%.*]]
+// SIMD-ONLY0:       cond.true1810:
+// SIMD-ONLY0-NEXT:    [[TMP748:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1811:%.*]] = zext i16 [[TMP748]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1814:%.*]]
+// SIMD-ONLY0:       cond.false1812:
+// SIMD-ONLY0-NEXT:    [[TMP749:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1813:%.*]] = zext i16 [[TMP749]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1814]]
+// SIMD-ONLY0:       cond.end1814:
+// SIMD-ONLY0-NEXT:    [[COND1815:%.*]] = phi i32 [ [[CONV1811]], [[COND_TRUE1810]] ], [ [[CONV1813]], [[COND_FALSE1812]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1816:%.*]] = trunc i32 [[COND1815]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1816]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP750:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1817:%.*]] = zext i16 [[TMP750]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP751:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1818:%.*]] = zext i16 [[TMP751]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1819:%.*]] = icmp eq i32 [[CONV1817]], [[CONV1818]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1819]], label [[IF_THEN1821:%.*]], label [[IF_END1822:%.*]]
+// SIMD-ONLY0:       if.then1821:
+// SIMD-ONLY0-NEXT:    [[TMP752:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP752]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1822]]
+// SIMD-ONLY0:       if.end1822:
+// SIMD-ONLY0-NEXT:    [[TMP753:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1823:%.*]] = zext i16 [[TMP753]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP754:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1824:%.*]] = zext i16 [[TMP754]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1825:%.*]] = icmp eq i32 [[CONV1823]], [[CONV1824]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1825]], label [[IF_THEN1827:%.*]], label [[IF_END1828:%.*]]
+// SIMD-ONLY0:       if.then1827:
+// SIMD-ONLY0-NEXT:    [[TMP755:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP755]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1828]]
+// SIMD-ONLY0:       if.end1828:
+// SIMD-ONLY0-NEXT:    [[TMP756:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1829:%.*]] = sext i16 [[TMP756]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP757:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1830:%.*]] = sext i16 [[TMP757]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1831:%.*]] = icmp sgt i32 [[CONV1829]], [[CONV1830]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1831]], label [[COND_TRUE1833:%.*]], label [[COND_FALSE1835:%.*]]
+// SIMD-ONLY0:       cond.true1833:
+// SIMD-ONLY0-NEXT:    [[TMP758:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1834:%.*]] = sext i16 [[TMP758]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1837:%.*]]
+// SIMD-ONLY0:       cond.false1835:
+// SIMD-ONLY0-NEXT:    [[TMP759:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1836:%.*]] = sext i16 [[TMP759]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1837]]
+// SIMD-ONLY0:       cond.end1837:
+// SIMD-ONLY0-NEXT:    [[COND1838:%.*]] = phi i32 [ [[CONV1834]], [[COND_TRUE1833]] ], [ [[CONV1836]], [[COND_FALSE1835]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1839:%.*]] = trunc i32 [[COND1838]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1839]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP760:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1840:%.*]] = sext i16 [[TMP760]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP761:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1841:%.*]] = sext i16 [[TMP761]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1842:%.*]] = icmp slt i32 [[CONV1840]], [[CONV1841]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1842]], label [[COND_TRUE1844:%.*]], label [[COND_FALSE1846:%.*]]
+// SIMD-ONLY0:       cond.true1844:
+// SIMD-ONLY0-NEXT:    [[TMP762:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1845:%.*]] = sext i16 [[TMP762]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1848:%.*]]
+// SIMD-ONLY0:       cond.false1846:
+// SIMD-ONLY0-NEXT:    [[TMP763:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1847:%.*]] = sext i16 [[TMP763]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1848]]
+// SIMD-ONLY0:       cond.end1848:
+// SIMD-ONLY0-NEXT:    [[COND1849:%.*]] = phi i32 [ [[CONV1845]], [[COND_TRUE1844]] ], [ [[CONV1847]], [[COND_FALSE1846]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1850:%.*]] = trunc i32 [[COND1849]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1850]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP764:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1851:%.*]] = sext i16 [[TMP764]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP765:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1852:%.*]] = sext i16 [[TMP765]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1853:%.*]] = icmp sgt i32 [[CONV1851]], [[CONV1852]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1853]], label [[COND_TRUE1855:%.*]], label [[COND_FALSE1857:%.*]]
+// SIMD-ONLY0:       cond.true1855:
+// SIMD-ONLY0-NEXT:    [[TMP766:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1856:%.*]] = sext i16 [[TMP766]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1859:%.*]]
+// SIMD-ONLY0:       cond.false1857:
+// SIMD-ONLY0-NEXT:    [[TMP767:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1858:%.*]] = sext i16 [[TMP767]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1859]]
+// SIMD-ONLY0:       cond.end1859:
+// SIMD-ONLY0-NEXT:    [[COND1860:%.*]] = phi i32 [ [[CONV1856]], [[COND_TRUE1855]] ], [ [[CONV1858]], [[COND_FALSE1857]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1861:%.*]] = trunc i32 [[COND1860]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1861]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP768:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1862:%.*]] = sext i16 [[TMP768]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP769:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1863:%.*]] = sext i16 [[TMP769]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1864:%.*]] = icmp slt i32 [[CONV1862]], [[CONV1863]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1864]], label [[COND_TRUE1866:%.*]], label [[COND_FALSE1868:%.*]]
+// SIMD-ONLY0:       cond.true1866:
+// SIMD-ONLY0-NEXT:    [[TMP770:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1867:%.*]] = sext i16 [[TMP770]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1870:%.*]]
+// SIMD-ONLY0:       cond.false1868:
+// SIMD-ONLY0-NEXT:    [[TMP771:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1869:%.*]] = sext i16 [[TMP771]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1870]]
+// SIMD-ONLY0:       cond.end1870:
+// SIMD-ONLY0-NEXT:    [[COND1871:%.*]] = phi i32 [ [[CONV1867]], [[COND_TRUE1866]] ], [ [[CONV1869]], [[COND_FALSE1868]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1872:%.*]] = trunc i32 [[COND1871]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1872]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP772:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1873:%.*]] = sext i16 [[TMP772]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP773:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1874:%.*]] = sext i16 [[TMP773]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1875:%.*]] = icmp sgt i32 [[CONV1873]], [[CONV1874]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1875]], label [[IF_THEN1877:%.*]], label [[IF_END1878:%.*]]
+// SIMD-ONLY0:       if.then1877:
+// SIMD-ONLY0-NEXT:    [[TMP774:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP774]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1878]]
+// SIMD-ONLY0:       if.end1878:
+// SIMD-ONLY0-NEXT:    [[TMP775:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1879:%.*]] = sext i16 [[TMP775]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP776:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1880:%.*]] = sext i16 [[TMP776]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1881:%.*]] = icmp slt i32 [[CONV1879]], [[CONV1880]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1881]], label [[IF_THEN1883:%.*]], label [[IF_END1884:%.*]]
+// SIMD-ONLY0:       if.then1883:
+// SIMD-ONLY0-NEXT:    [[TMP777:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP777]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1884]]
+// SIMD-ONLY0:       if.end1884:
+// SIMD-ONLY0-NEXT:    [[TMP778:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1885:%.*]] = sext i16 [[TMP778]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP779:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1886:%.*]] = sext i16 [[TMP779]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1887:%.*]] = icmp sgt i32 [[CONV1885]], [[CONV1886]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1887]], label [[IF_THEN1889:%.*]], label [[IF_END1890:%.*]]
+// SIMD-ONLY0:       if.then1889:
+// SIMD-ONLY0-NEXT:    [[TMP780:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP780]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1890]]
+// SIMD-ONLY0:       if.end1890:
+// SIMD-ONLY0-NEXT:    [[TMP781:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1891:%.*]] = sext i16 [[TMP781]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP782:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1892:%.*]] = sext i16 [[TMP782]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1893:%.*]] = icmp slt i32 [[CONV1891]], [[CONV1892]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1893]], label [[IF_THEN1895:%.*]], label [[IF_END1896:%.*]]
+// SIMD-ONLY0:       if.then1895:
+// SIMD-ONLY0-NEXT:    [[TMP783:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP783]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1896]]
+// SIMD-ONLY0:       if.end1896:
+// SIMD-ONLY0-NEXT:    [[TMP784:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1897:%.*]] = sext i16 [[TMP784]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP785:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1898:%.*]] = sext i16 [[TMP785]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1899:%.*]] = icmp eq i32 [[CONV1897]], [[CONV1898]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1899]], label [[COND_TRUE1901:%.*]], label [[COND_FALSE1903:%.*]]
+// SIMD-ONLY0:       cond.true1901:
+// SIMD-ONLY0-NEXT:    [[TMP786:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1902:%.*]] = sext i16 [[TMP786]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1905:%.*]]
+// SIMD-ONLY0:       cond.false1903:
+// SIMD-ONLY0-NEXT:    [[TMP787:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1904:%.*]] = sext i16 [[TMP787]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1905]]
+// SIMD-ONLY0:       cond.end1905:
+// SIMD-ONLY0-NEXT:    [[COND1906:%.*]] = phi i32 [ [[CONV1902]], [[COND_TRUE1901]] ], [ [[CONV1904]], [[COND_FALSE1903]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1907:%.*]] = trunc i32 [[COND1906]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1907]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP788:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1908:%.*]] = sext i16 [[TMP788]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP789:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1909:%.*]] = sext i16 [[TMP789]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1910:%.*]] = icmp eq i32 [[CONV1908]], [[CONV1909]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1910]], label [[COND_TRUE1912:%.*]], label [[COND_FALSE1914:%.*]]
+// SIMD-ONLY0:       cond.true1912:
+// SIMD-ONLY0-NEXT:    [[TMP790:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1913:%.*]] = sext i16 [[TMP790]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1916:%.*]]
+// SIMD-ONLY0:       cond.false1914:
+// SIMD-ONLY0-NEXT:    [[TMP791:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1915:%.*]] = sext i16 [[TMP791]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1916]]
+// SIMD-ONLY0:       cond.end1916:
+// SIMD-ONLY0-NEXT:    [[COND1917:%.*]] = phi i32 [ [[CONV1913]], [[COND_TRUE1912]] ], [ [[CONV1915]], [[COND_FALSE1914]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1918:%.*]] = trunc i32 [[COND1917]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1918]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP792:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1919:%.*]] = sext i16 [[TMP792]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP793:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1920:%.*]] = sext i16 [[TMP793]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1921:%.*]] = icmp eq i32 [[CONV1919]], [[CONV1920]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1921]], label [[IF_THEN1923:%.*]], label [[IF_END1924:%.*]]
+// SIMD-ONLY0:       if.then1923:
+// SIMD-ONLY0-NEXT:    [[TMP794:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP794]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1924]]
+// SIMD-ONLY0:       if.end1924:
+// SIMD-ONLY0-NEXT:    [[TMP795:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1925:%.*]] = sext i16 [[TMP795]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP796:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1926:%.*]] = sext i16 [[TMP796]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1927:%.*]] = icmp eq i32 [[CONV1925]], [[CONV1926]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1927]], label [[IF_THEN1929:%.*]], label [[IF_END1930:%.*]]
+// SIMD-ONLY0:       if.then1929:
+// SIMD-ONLY0-NEXT:    [[TMP797:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP797]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1930]]
+// SIMD-ONLY0:       if.end1930:
+// SIMD-ONLY0-NEXT:    [[TMP798:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1931:%.*]] = zext i16 [[TMP798]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP799:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1932:%.*]] = zext i16 [[TMP799]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1933:%.*]] = icmp sgt i32 [[CONV1931]], [[CONV1932]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1933]], label [[COND_TRUE1935:%.*]], label [[COND_FALSE1937:%.*]]
+// SIMD-ONLY0:       cond.true1935:
+// SIMD-ONLY0-NEXT:    [[TMP800:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1936:%.*]] = zext i16 [[TMP800]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1939:%.*]]
+// SIMD-ONLY0:       cond.false1937:
+// SIMD-ONLY0-NEXT:    [[TMP801:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1938:%.*]] = zext i16 [[TMP801]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1939]]
+// SIMD-ONLY0:       cond.end1939:
+// SIMD-ONLY0-NEXT:    [[COND1940:%.*]] = phi i32 [ [[CONV1936]], [[COND_TRUE1935]] ], [ [[CONV1938]], [[COND_FALSE1937]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1941:%.*]] = trunc i32 [[COND1940]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1941]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP802:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1942:%.*]] = zext i16 [[TMP802]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP803:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1943:%.*]] = zext i16 [[TMP803]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1944:%.*]] = icmp slt i32 [[CONV1942]], [[CONV1943]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1944]], label [[COND_TRUE1946:%.*]], label [[COND_FALSE1948:%.*]]
+// SIMD-ONLY0:       cond.true1946:
+// SIMD-ONLY0-NEXT:    [[TMP804:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1947:%.*]] = zext i16 [[TMP804]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1950:%.*]]
+// SIMD-ONLY0:       cond.false1948:
+// SIMD-ONLY0-NEXT:    [[TMP805:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1949:%.*]] = zext i16 [[TMP805]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1950]]
+// SIMD-ONLY0:       cond.end1950:
+// SIMD-ONLY0-NEXT:    [[COND1951:%.*]] = phi i32 [ [[CONV1947]], [[COND_TRUE1946]] ], [ [[CONV1949]], [[COND_FALSE1948]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1952:%.*]] = trunc i32 [[COND1951]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1952]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP806:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1953:%.*]] = zext i16 [[TMP806]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP807:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1954:%.*]] = zext i16 [[TMP807]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1955:%.*]] = icmp sgt i32 [[CONV1953]], [[CONV1954]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1955]], label [[COND_TRUE1957:%.*]], label [[COND_FALSE1959:%.*]]
+// SIMD-ONLY0:       cond.true1957:
+// SIMD-ONLY0-NEXT:    [[TMP808:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1958:%.*]] = zext i16 [[TMP808]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1961:%.*]]
+// SIMD-ONLY0:       cond.false1959:
+// SIMD-ONLY0-NEXT:    [[TMP809:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1960:%.*]] = zext i16 [[TMP809]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1961]]
+// SIMD-ONLY0:       cond.end1961:
+// SIMD-ONLY0-NEXT:    [[COND1962:%.*]] = phi i32 [ [[CONV1958]], [[COND_TRUE1957]] ], [ [[CONV1960]], [[COND_FALSE1959]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1963:%.*]] = trunc i32 [[COND1962]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1963]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP810:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1964:%.*]] = zext i16 [[TMP810]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP811:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1965:%.*]] = zext i16 [[TMP811]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1966:%.*]] = icmp slt i32 [[CONV1964]], [[CONV1965]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1966]], label [[COND_TRUE1968:%.*]], label [[COND_FALSE1970:%.*]]
+// SIMD-ONLY0:       cond.true1968:
+// SIMD-ONLY0-NEXT:    [[TMP812:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1969:%.*]] = zext i16 [[TMP812]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1972:%.*]]
+// SIMD-ONLY0:       cond.false1970:
+// SIMD-ONLY0-NEXT:    [[TMP813:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1971:%.*]] = zext i16 [[TMP813]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END1972]]
+// SIMD-ONLY0:       cond.end1972:
+// SIMD-ONLY0-NEXT:    [[COND1973:%.*]] = phi i32 [ [[CONV1969]], [[COND_TRUE1968]] ], [ [[CONV1971]], [[COND_FALSE1970]] ]
+// SIMD-ONLY0-NEXT:    [[CONV1974:%.*]] = trunc i32 [[COND1973]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1974]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP814:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1975:%.*]] = zext i16 [[TMP814]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP815:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1976:%.*]] = zext i16 [[TMP815]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1977:%.*]] = icmp sgt i32 [[CONV1975]], [[CONV1976]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1977]], label [[IF_THEN1979:%.*]], label [[IF_END1980:%.*]]
+// SIMD-ONLY0:       if.then1979:
+// SIMD-ONLY0-NEXT:    [[TMP816:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP816]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1980]]
+// SIMD-ONLY0:       if.end1980:
+// SIMD-ONLY0-NEXT:    [[TMP817:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1981:%.*]] = zext i16 [[TMP817]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP818:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1982:%.*]] = zext i16 [[TMP818]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1983:%.*]] = icmp slt i32 [[CONV1981]], [[CONV1982]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1983]], label [[IF_THEN1985:%.*]], label [[IF_END1986:%.*]]
+// SIMD-ONLY0:       if.then1985:
+// SIMD-ONLY0-NEXT:    [[TMP819:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP819]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1986]]
+// SIMD-ONLY0:       if.end1986:
+// SIMD-ONLY0-NEXT:    [[TMP820:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1987:%.*]] = zext i16 [[TMP820]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP821:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1988:%.*]] = zext i16 [[TMP821]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1989:%.*]] = icmp sgt i32 [[CONV1987]], [[CONV1988]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1989]], label [[IF_THEN1991:%.*]], label [[IF_END1992:%.*]]
+// SIMD-ONLY0:       if.then1991:
+// SIMD-ONLY0-NEXT:    [[TMP822:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP822]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1992]]
+// SIMD-ONLY0:       if.end1992:
+// SIMD-ONLY0-NEXT:    [[TMP823:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1993:%.*]] = zext i16 [[TMP823]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP824:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1994:%.*]] = zext i16 [[TMP824]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1995:%.*]] = icmp slt i32 [[CONV1993]], [[CONV1994]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1995]], label [[IF_THEN1997:%.*]], label [[IF_END1998:%.*]]
+// SIMD-ONLY0:       if.then1997:
+// SIMD-ONLY0-NEXT:    [[TMP825:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP825]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1998]]
+// SIMD-ONLY0:       if.end1998:
+// SIMD-ONLY0-NEXT:    [[TMP826:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1999:%.*]] = zext i16 [[TMP826]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP827:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2000:%.*]] = zext i16 [[TMP827]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2001:%.*]] = icmp eq i32 [[CONV1999]], [[CONV2000]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2001]], label [[COND_TRUE2003:%.*]], label [[COND_FALSE2005:%.*]]
+// SIMD-ONLY0:       cond.true2003:
+// SIMD-ONLY0-NEXT:    [[TMP828:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2004:%.*]] = zext i16 [[TMP828]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2007:%.*]]
+// SIMD-ONLY0:       cond.false2005:
+// SIMD-ONLY0-NEXT:    [[TMP829:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2006:%.*]] = zext i16 [[TMP829]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2007]]
+// SIMD-ONLY0:       cond.end2007:
+// SIMD-ONLY0-NEXT:    [[COND2008:%.*]] = phi i32 [ [[CONV2004]], [[COND_TRUE2003]] ], [ [[CONV2006]], [[COND_FALSE2005]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2009:%.*]] = trunc i32 [[COND2008]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2009]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP830:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2010:%.*]] = zext i16 [[TMP830]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP831:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2011:%.*]] = zext i16 [[TMP831]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2012:%.*]] = icmp eq i32 [[CONV2010]], [[CONV2011]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2012]], label [[COND_TRUE2014:%.*]], label [[COND_FALSE2016:%.*]]
+// SIMD-ONLY0:       cond.true2014:
+// SIMD-ONLY0-NEXT:    [[TMP832:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2015:%.*]] = zext i16 [[TMP832]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2018:%.*]]
+// SIMD-ONLY0:       cond.false2016:
+// SIMD-ONLY0-NEXT:    [[TMP833:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2017:%.*]] = zext i16 [[TMP833]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2018]]
+// SIMD-ONLY0:       cond.end2018:
+// SIMD-ONLY0-NEXT:    [[COND2019:%.*]] = phi i32 [ [[CONV2015]], [[COND_TRUE2014]] ], [ [[CONV2017]], [[COND_FALSE2016]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2020:%.*]] = trunc i32 [[COND2019]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2020]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP834:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2021:%.*]] = zext i16 [[TMP834]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP835:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2022:%.*]] = zext i16 [[TMP835]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2023:%.*]] = icmp eq i32 [[CONV2021]], [[CONV2022]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2023]], label [[IF_THEN2025:%.*]], label [[IF_END2026:%.*]]
+// SIMD-ONLY0:       if.then2025:
+// SIMD-ONLY0-NEXT:    [[TMP836:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP836]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2026]]
+// SIMD-ONLY0:       if.end2026:
+// SIMD-ONLY0-NEXT:    [[TMP837:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2027:%.*]] = zext i16 [[TMP837]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP838:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2028:%.*]] = zext i16 [[TMP838]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2029:%.*]] = icmp eq i32 [[CONV2027]], [[CONV2028]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2029]], label [[IF_THEN2031:%.*]], label [[IF_END2032:%.*]]
+// SIMD-ONLY0:       if.then2031:
+// SIMD-ONLY0-NEXT:    [[TMP839:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP839]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2032]]
+// SIMD-ONLY0:       if.end2032:
+// SIMD-ONLY0-NEXT:    [[TMP840:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2033:%.*]] = sext i16 [[TMP840]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP841:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2034:%.*]] = sext i16 [[TMP841]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2035:%.*]] = icmp sgt i32 [[CONV2033]], [[CONV2034]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2035]], label [[COND_TRUE2037:%.*]], label [[COND_FALSE2039:%.*]]
+// SIMD-ONLY0:       cond.true2037:
+// SIMD-ONLY0-NEXT:    [[TMP842:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2038:%.*]] = sext i16 [[TMP842]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2041:%.*]]
+// SIMD-ONLY0:       cond.false2039:
+// SIMD-ONLY0-NEXT:    [[TMP843:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2040:%.*]] = sext i16 [[TMP843]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2041]]
+// SIMD-ONLY0:       cond.end2041:
+// SIMD-ONLY0-NEXT:    [[COND2042:%.*]] = phi i32 [ [[CONV2038]], [[COND_TRUE2037]] ], [ [[CONV2040]], [[COND_FALSE2039]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2043:%.*]] = trunc i32 [[COND2042]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2043]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP844:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2044:%.*]] = sext i16 [[TMP844]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP845:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2045:%.*]] = sext i16 [[TMP845]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2046:%.*]] = icmp slt i32 [[CONV2044]], [[CONV2045]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2046]], label [[COND_TRUE2048:%.*]], label [[COND_FALSE2050:%.*]]
+// SIMD-ONLY0:       cond.true2048:
+// SIMD-ONLY0-NEXT:    [[TMP846:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2049:%.*]] = sext i16 [[TMP846]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2052:%.*]]
+// SIMD-ONLY0:       cond.false2050:
+// SIMD-ONLY0-NEXT:    [[TMP847:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2051:%.*]] = sext i16 [[TMP847]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2052]]
+// SIMD-ONLY0:       cond.end2052:
+// SIMD-ONLY0-NEXT:    [[COND2053:%.*]] = phi i32 [ [[CONV2049]], [[COND_TRUE2048]] ], [ [[CONV2051]], [[COND_FALSE2050]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2054:%.*]] = trunc i32 [[COND2053]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2054]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP848:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2055:%.*]] = sext i16 [[TMP848]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP849:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2056:%.*]] = sext i16 [[TMP849]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2057:%.*]] = icmp sgt i32 [[CONV2055]], [[CONV2056]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2057]], label [[COND_TRUE2059:%.*]], label [[COND_FALSE2061:%.*]]
+// SIMD-ONLY0:       cond.true2059:
+// SIMD-ONLY0-NEXT:    [[TMP850:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2060:%.*]] = sext i16 [[TMP850]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2063:%.*]]
+// SIMD-ONLY0:       cond.false2061:
+// SIMD-ONLY0-NEXT:    [[TMP851:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2062:%.*]] = sext i16 [[TMP851]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2063]]
+// SIMD-ONLY0:       cond.end2063:
+// SIMD-ONLY0-NEXT:    [[COND2064:%.*]] = phi i32 [ [[CONV2060]], [[COND_TRUE2059]] ], [ [[CONV2062]], [[COND_FALSE2061]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2065:%.*]] = trunc i32 [[COND2064]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2065]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP852:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2066:%.*]] = sext i16 [[TMP852]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP853:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2067:%.*]] = sext i16 [[TMP853]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2068:%.*]] = icmp slt i32 [[CONV2066]], [[CONV2067]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2068]], label [[COND_TRUE2070:%.*]], label [[COND_FALSE2072:%.*]]
+// SIMD-ONLY0:       cond.true2070:
+// SIMD-ONLY0-NEXT:    [[TMP854:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2071:%.*]] = sext i16 [[TMP854]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2074:%.*]]
+// SIMD-ONLY0:       cond.false2072:
+// SIMD-ONLY0-NEXT:    [[TMP855:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2073:%.*]] = sext i16 [[TMP855]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2074]]
+// SIMD-ONLY0:       cond.end2074:
+// SIMD-ONLY0-NEXT:    [[COND2075:%.*]] = phi i32 [ [[CONV2071]], [[COND_TRUE2070]] ], [ [[CONV2073]], [[COND_FALSE2072]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2076:%.*]] = trunc i32 [[COND2075]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2076]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP856:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2077:%.*]] = sext i16 [[TMP856]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP857:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2078:%.*]] = sext i16 [[TMP857]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2079:%.*]] = icmp sgt i32 [[CONV2077]], [[CONV2078]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2079]], label [[IF_THEN2081:%.*]], label [[IF_END2082:%.*]]
+// SIMD-ONLY0:       if.then2081:
+// SIMD-ONLY0-NEXT:    [[TMP858:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP858]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2082]]
+// SIMD-ONLY0:       if.end2082:
+// SIMD-ONLY0-NEXT:    [[TMP859:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2083:%.*]] = sext i16 [[TMP859]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP860:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2084:%.*]] = sext i16 [[TMP860]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2085:%.*]] = icmp slt i32 [[CONV2083]], [[CONV2084]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2085]], label [[IF_THEN2087:%.*]], label [[IF_END2088:%.*]]
+// SIMD-ONLY0:       if.then2087:
+// SIMD-ONLY0-NEXT:    [[TMP861:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP861]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2088]]
+// SIMD-ONLY0:       if.end2088:
+// SIMD-ONLY0-NEXT:    [[TMP862:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2089:%.*]] = sext i16 [[TMP862]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP863:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2090:%.*]] = sext i16 [[TMP863]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2091:%.*]] = icmp sgt i32 [[CONV2089]], [[CONV2090]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2091]], label [[IF_THEN2093:%.*]], label [[IF_END2094:%.*]]
+// SIMD-ONLY0:       if.then2093:
+// SIMD-ONLY0-NEXT:    [[TMP864:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP864]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2094]]
+// SIMD-ONLY0:       if.end2094:
+// SIMD-ONLY0-NEXT:    [[TMP865:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2095:%.*]] = sext i16 [[TMP865]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP866:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2096:%.*]] = sext i16 [[TMP866]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2097:%.*]] = icmp slt i32 [[CONV2095]], [[CONV2096]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2097]], label [[IF_THEN2099:%.*]], label [[IF_END2100:%.*]]
+// SIMD-ONLY0:       if.then2099:
+// SIMD-ONLY0-NEXT:    [[TMP867:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP867]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2100]]
+// SIMD-ONLY0:       if.end2100:
+// SIMD-ONLY0-NEXT:    [[TMP868:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2101:%.*]] = sext i16 [[TMP868]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP869:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2102:%.*]] = sext i16 [[TMP869]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2103:%.*]] = icmp eq i32 [[CONV2101]], [[CONV2102]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2103]], label [[COND_TRUE2105:%.*]], label [[COND_FALSE2107:%.*]]
+// SIMD-ONLY0:       cond.true2105:
+// SIMD-ONLY0-NEXT:    [[TMP870:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2106:%.*]] = sext i16 [[TMP870]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2109:%.*]]
+// SIMD-ONLY0:       cond.false2107:
+// SIMD-ONLY0-NEXT:    [[TMP871:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2108:%.*]] = sext i16 [[TMP871]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2109]]
+// SIMD-ONLY0:       cond.end2109:
+// SIMD-ONLY0-NEXT:    [[COND2110:%.*]] = phi i32 [ [[CONV2106]], [[COND_TRUE2105]] ], [ [[CONV2108]], [[COND_FALSE2107]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2111:%.*]] = trunc i32 [[COND2110]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2111]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP872:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2112:%.*]] = sext i16 [[TMP872]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP873:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2113:%.*]] = sext i16 [[TMP873]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2114:%.*]] = icmp eq i32 [[CONV2112]], [[CONV2113]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2114]], label [[COND_TRUE2116:%.*]], label [[COND_FALSE2118:%.*]]
+// SIMD-ONLY0:       cond.true2116:
+// SIMD-ONLY0-NEXT:    [[TMP874:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2117:%.*]] = sext i16 [[TMP874]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2120:%.*]]
+// SIMD-ONLY0:       cond.false2118:
+// SIMD-ONLY0-NEXT:    [[TMP875:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2119:%.*]] = sext i16 [[TMP875]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2120]]
+// SIMD-ONLY0:       cond.end2120:
+// SIMD-ONLY0-NEXT:    [[COND2121:%.*]] = phi i32 [ [[CONV2117]], [[COND_TRUE2116]] ], [ [[CONV2119]], [[COND_FALSE2118]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2122:%.*]] = trunc i32 [[COND2121]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2122]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP876:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2123:%.*]] = sext i16 [[TMP876]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP877:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2124:%.*]] = sext i16 [[TMP877]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2125:%.*]] = icmp eq i32 [[CONV2123]], [[CONV2124]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2125]], label [[IF_THEN2127:%.*]], label [[IF_END2128:%.*]]
+// SIMD-ONLY0:       if.then2127:
+// SIMD-ONLY0-NEXT:    [[TMP878:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP878]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2128]]
+// SIMD-ONLY0:       if.end2128:
+// SIMD-ONLY0-NEXT:    [[TMP879:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2129:%.*]] = sext i16 [[TMP879]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP880:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2130:%.*]] = sext i16 [[TMP880]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2131:%.*]] = icmp eq i32 [[CONV2129]], [[CONV2130]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2131]], label [[IF_THEN2133:%.*]], label [[IF_END2134:%.*]]
+// SIMD-ONLY0:       if.then2133:
+// SIMD-ONLY0-NEXT:    [[TMP881:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP881]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2134]]
+// SIMD-ONLY0:       if.end2134:
+// SIMD-ONLY0-NEXT:    [[TMP882:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2135:%.*]] = zext i16 [[TMP882]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP883:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2136:%.*]] = zext i16 [[TMP883]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2137:%.*]] = icmp sgt i32 [[CONV2135]], [[CONV2136]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2137]], label [[COND_TRUE2139:%.*]], label [[COND_FALSE2141:%.*]]
+// SIMD-ONLY0:       cond.true2139:
+// SIMD-ONLY0-NEXT:    [[TMP884:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2140:%.*]] = zext i16 [[TMP884]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2143:%.*]]
+// SIMD-ONLY0:       cond.false2141:
+// SIMD-ONLY0-NEXT:    [[TMP885:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2142:%.*]] = zext i16 [[TMP885]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2143]]
+// SIMD-ONLY0:       cond.end2143:
+// SIMD-ONLY0-NEXT:    [[COND2144:%.*]] = phi i32 [ [[CONV2140]], [[COND_TRUE2139]] ], [ [[CONV2142]], [[COND_FALSE2141]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2145:%.*]] = trunc i32 [[COND2144]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2145]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP886:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2146:%.*]] = zext i16 [[TMP886]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP887:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2147:%.*]] = zext i16 [[TMP887]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2148:%.*]] = icmp slt i32 [[CONV2146]], [[CONV2147]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2148]], label [[COND_TRUE2150:%.*]], label [[COND_FALSE2152:%.*]]
+// SIMD-ONLY0:       cond.true2150:
+// SIMD-ONLY0-NEXT:    [[TMP888:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2151:%.*]] = zext i16 [[TMP888]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2154:%.*]]
+// SIMD-ONLY0:       cond.false2152:
+// SIMD-ONLY0-NEXT:    [[TMP889:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2153:%.*]] = zext i16 [[TMP889]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2154]]
+// SIMD-ONLY0:       cond.end2154:
+// SIMD-ONLY0-NEXT:    [[COND2155:%.*]] = phi i32 [ [[CONV2151]], [[COND_TRUE2150]] ], [ [[CONV2153]], [[COND_FALSE2152]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2156:%.*]] = trunc i32 [[COND2155]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2156]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP890:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2157:%.*]] = zext i16 [[TMP890]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP891:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2158:%.*]] = zext i16 [[TMP891]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2159:%.*]] = icmp sgt i32 [[CONV2157]], [[CONV2158]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2159]], label [[COND_TRUE2161:%.*]], label [[COND_FALSE2163:%.*]]
+// SIMD-ONLY0:       cond.true2161:
+// SIMD-ONLY0-NEXT:    [[TMP892:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2162:%.*]] = zext i16 [[TMP892]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2165:%.*]]
+// SIMD-ONLY0:       cond.false2163:
+// SIMD-ONLY0-NEXT:    [[TMP893:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2164:%.*]] = zext i16 [[TMP893]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2165]]
+// SIMD-ONLY0:       cond.end2165:
+// SIMD-ONLY0-NEXT:    [[COND2166:%.*]] = phi i32 [ [[CONV2162]], [[COND_TRUE2161]] ], [ [[CONV2164]], [[COND_FALSE2163]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2167:%.*]] = trunc i32 [[COND2166]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2167]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP894:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2168:%.*]] = zext i16 [[TMP894]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP895:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2169:%.*]] = zext i16 [[TMP895]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2170:%.*]] = icmp slt i32 [[CONV2168]], [[CONV2169]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2170]], label [[COND_TRUE2172:%.*]], label [[COND_FALSE2174:%.*]]
+// SIMD-ONLY0:       cond.true2172:
+// SIMD-ONLY0-NEXT:    [[TMP896:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2173:%.*]] = zext i16 [[TMP896]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2176:%.*]]
+// SIMD-ONLY0:       cond.false2174:
+// SIMD-ONLY0-NEXT:    [[TMP897:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2175:%.*]] = zext i16 [[TMP897]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2176]]
+// SIMD-ONLY0:       cond.end2176:
+// SIMD-ONLY0-NEXT:    [[COND2177:%.*]] = phi i32 [ [[CONV2173]], [[COND_TRUE2172]] ], [ [[CONV2175]], [[COND_FALSE2174]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2178:%.*]] = trunc i32 [[COND2177]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2178]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP898:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2179:%.*]] = zext i16 [[TMP898]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP899:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2180:%.*]] = zext i16 [[TMP899]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2181:%.*]] = icmp sgt i32 [[CONV2179]], [[CONV2180]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2181]], label [[IF_THEN2183:%.*]], label [[IF_END2184:%.*]]
+// SIMD-ONLY0:       if.then2183:
+// SIMD-ONLY0-NEXT:    [[TMP900:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP900]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2184]]
+// SIMD-ONLY0:       if.end2184:
+// SIMD-ONLY0-NEXT:    [[TMP901:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2185:%.*]] = zext i16 [[TMP901]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP902:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2186:%.*]] = zext i16 [[TMP902]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2187:%.*]] = icmp slt i32 [[CONV2185]], [[CONV2186]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2187]], label [[IF_THEN2189:%.*]], label [[IF_END2190:%.*]]
+// SIMD-ONLY0:       if.then2189:
+// SIMD-ONLY0-NEXT:    [[TMP903:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP903]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2190]]
+// SIMD-ONLY0:       if.end2190:
+// SIMD-ONLY0-NEXT:    [[TMP904:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2191:%.*]] = zext i16 [[TMP904]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP905:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2192:%.*]] = zext i16 [[TMP905]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2193:%.*]] = icmp sgt i32 [[CONV2191]], [[CONV2192]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2193]], label [[IF_THEN2195:%.*]], label [[IF_END2196:%.*]]
+// SIMD-ONLY0:       if.then2195:
+// SIMD-ONLY0-NEXT:    [[TMP906:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP906]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2196]]
+// SIMD-ONLY0:       if.end2196:
+// SIMD-ONLY0-NEXT:    [[TMP907:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2197:%.*]] = zext i16 [[TMP907]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP908:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2198:%.*]] = zext i16 [[TMP908]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2199:%.*]] = icmp slt i32 [[CONV2197]], [[CONV2198]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2199]], label [[IF_THEN2201:%.*]], label [[IF_END2202:%.*]]
+// SIMD-ONLY0:       if.then2201:
+// SIMD-ONLY0-NEXT:    [[TMP909:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP909]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2202]]
+// SIMD-ONLY0:       if.end2202:
+// SIMD-ONLY0-NEXT:    [[TMP910:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2203:%.*]] = zext i16 [[TMP910]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP911:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2204:%.*]] = zext i16 [[TMP911]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2205:%.*]] = icmp eq i32 [[CONV2203]], [[CONV2204]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2205]], label [[COND_TRUE2207:%.*]], label [[COND_FALSE2209:%.*]]
+// SIMD-ONLY0:       cond.true2207:
+// SIMD-ONLY0-NEXT:    [[TMP912:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2208:%.*]] = zext i16 [[TMP912]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2211:%.*]]
+// SIMD-ONLY0:       cond.false2209:
+// SIMD-ONLY0-NEXT:    [[TMP913:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2210:%.*]] = zext i16 [[TMP913]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2211]]
+// SIMD-ONLY0:       cond.end2211:
+// SIMD-ONLY0-NEXT:    [[COND2212:%.*]] = phi i32 [ [[CONV2208]], [[COND_TRUE2207]] ], [ [[CONV2210]], [[COND_FALSE2209]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2213:%.*]] = trunc i32 [[COND2212]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2213]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP914:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2214:%.*]] = zext i16 [[TMP914]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP915:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2215:%.*]] = zext i16 [[TMP915]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2216:%.*]] = icmp eq i32 [[CONV2214]], [[CONV2215]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2216]], label [[COND_TRUE2218:%.*]], label [[COND_FALSE2220:%.*]]
+// SIMD-ONLY0:       cond.true2218:
+// SIMD-ONLY0-NEXT:    [[TMP916:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2219:%.*]] = zext i16 [[TMP916]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2222:%.*]]
+// SIMD-ONLY0:       cond.false2220:
+// SIMD-ONLY0-NEXT:    [[TMP917:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2221:%.*]] = zext i16 [[TMP917]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2222]]
+// SIMD-ONLY0:       cond.end2222:
+// SIMD-ONLY0-NEXT:    [[COND2223:%.*]] = phi i32 [ [[CONV2219]], [[COND_TRUE2218]] ], [ [[CONV2221]], [[COND_FALSE2220]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2224:%.*]] = trunc i32 [[COND2223]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2224]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP918:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2225:%.*]] = zext i16 [[TMP918]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP919:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2226:%.*]] = zext i16 [[TMP919]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2227:%.*]] = icmp eq i32 [[CONV2225]], [[CONV2226]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2227]], label [[IF_THEN2229:%.*]], label [[IF_END2230:%.*]]
+// SIMD-ONLY0:       if.then2229:
+// SIMD-ONLY0-NEXT:    [[TMP920:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP920]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2230]]
+// SIMD-ONLY0:       if.end2230:
+// SIMD-ONLY0-NEXT:    [[TMP921:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2231:%.*]] = zext i16 [[TMP921]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP922:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2232:%.*]] = zext i16 [[TMP922]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2233:%.*]] = icmp eq i32 [[CONV2231]], [[CONV2232]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2233]], label [[IF_THEN2235:%.*]], label [[IF_END2236:%.*]]
+// SIMD-ONLY0:       if.then2235:
+// SIMD-ONLY0-NEXT:    [[TMP923:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP923]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2236]]
+// SIMD-ONLY0:       if.end2236:
+// SIMD-ONLY0-NEXT:    [[TMP924:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2237:%.*]] = sext i16 [[TMP924]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP925:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2238:%.*]] = sext i16 [[TMP925]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2239:%.*]] = icmp sgt i32 [[CONV2237]], [[CONV2238]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2239]], label [[COND_TRUE2241:%.*]], label [[COND_FALSE2243:%.*]]
+// SIMD-ONLY0:       cond.true2241:
+// SIMD-ONLY0-NEXT:    [[TMP926:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2242:%.*]] = sext i16 [[TMP926]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2245:%.*]]
+// SIMD-ONLY0:       cond.false2243:
+// SIMD-ONLY0-NEXT:    [[TMP927:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2244:%.*]] = sext i16 [[TMP927]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2245]]
+// SIMD-ONLY0:       cond.end2245:
+// SIMD-ONLY0-NEXT:    [[COND2246:%.*]] = phi i32 [ [[CONV2242]], [[COND_TRUE2241]] ], [ [[CONV2244]], [[COND_FALSE2243]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2247:%.*]] = trunc i32 [[COND2246]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2247]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP928:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2248:%.*]] = sext i16 [[TMP928]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP929:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2249:%.*]] = sext i16 [[TMP929]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2250:%.*]] = icmp slt i32 [[CONV2248]], [[CONV2249]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2250]], label [[COND_TRUE2252:%.*]], label [[COND_FALSE2254:%.*]]
+// SIMD-ONLY0:       cond.true2252:
+// SIMD-ONLY0-NEXT:    [[TMP930:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2253:%.*]] = sext i16 [[TMP930]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2256:%.*]]
+// SIMD-ONLY0:       cond.false2254:
+// SIMD-ONLY0-NEXT:    [[TMP931:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2255:%.*]] = sext i16 [[TMP931]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2256]]
+// SIMD-ONLY0:       cond.end2256:
+// SIMD-ONLY0-NEXT:    [[COND2257:%.*]] = phi i32 [ [[CONV2253]], [[COND_TRUE2252]] ], [ [[CONV2255]], [[COND_FALSE2254]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2258:%.*]] = trunc i32 [[COND2257]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2258]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP932:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2259:%.*]] = sext i16 [[TMP932]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP933:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2260:%.*]] = sext i16 [[TMP933]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2261:%.*]] = icmp sgt i32 [[CONV2259]], [[CONV2260]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2261]], label [[COND_TRUE2263:%.*]], label [[COND_FALSE2265:%.*]]
+// SIMD-ONLY0:       cond.true2263:
+// SIMD-ONLY0-NEXT:    [[TMP934:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2264:%.*]] = sext i16 [[TMP934]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2267:%.*]]
+// SIMD-ONLY0:       cond.false2265:
+// SIMD-ONLY0-NEXT:    [[TMP935:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2266:%.*]] = sext i16 [[TMP935]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2267]]
+// SIMD-ONLY0:       cond.end2267:
+// SIMD-ONLY0-NEXT:    [[COND2268:%.*]] = phi i32 [ [[CONV2264]], [[COND_TRUE2263]] ], [ [[CONV2266]], [[COND_FALSE2265]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2269:%.*]] = trunc i32 [[COND2268]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2269]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP936:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2270:%.*]] = sext i16 [[TMP936]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP937:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2271:%.*]] = sext i16 [[TMP937]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2272:%.*]] = icmp slt i32 [[CONV2270]], [[CONV2271]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2272]], label [[COND_TRUE2274:%.*]], label [[COND_FALSE2276:%.*]]
+// SIMD-ONLY0:       cond.true2274:
+// SIMD-ONLY0-NEXT:    [[TMP938:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2275:%.*]] = sext i16 [[TMP938]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2278:%.*]]
+// SIMD-ONLY0:       cond.false2276:
+// SIMD-ONLY0-NEXT:    [[TMP939:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2277:%.*]] = sext i16 [[TMP939]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2278]]
+// SIMD-ONLY0:       cond.end2278:
+// SIMD-ONLY0-NEXT:    [[COND2279:%.*]] = phi i32 [ [[CONV2275]], [[COND_TRUE2274]] ], [ [[CONV2277]], [[COND_FALSE2276]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2280:%.*]] = trunc i32 [[COND2279]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2280]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP940:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2281:%.*]] = sext i16 [[TMP940]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP941:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2282:%.*]] = sext i16 [[TMP941]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2283:%.*]] = icmp sgt i32 [[CONV2281]], [[CONV2282]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2283]], label [[IF_THEN2285:%.*]], label [[IF_END2286:%.*]]
+// SIMD-ONLY0:       if.then2285:
+// SIMD-ONLY0-NEXT:    [[TMP942:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP942]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2286]]
+// SIMD-ONLY0:       if.end2286:
+// SIMD-ONLY0-NEXT:    [[TMP943:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2287:%.*]] = sext i16 [[TMP943]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP944:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2288:%.*]] = sext i16 [[TMP944]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2289:%.*]] = icmp slt i32 [[CONV2287]], [[CONV2288]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2289]], label [[IF_THEN2291:%.*]], label [[IF_END2292:%.*]]
+// SIMD-ONLY0:       if.then2291:
+// SIMD-ONLY0-NEXT:    [[TMP945:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP945]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2292]]
+// SIMD-ONLY0:       if.end2292:
+// SIMD-ONLY0-NEXT:    [[TMP946:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2293:%.*]] = sext i16 [[TMP946]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP947:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2294:%.*]] = sext i16 [[TMP947]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2295:%.*]] = icmp sgt i32 [[CONV2293]], [[CONV2294]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2295]], label [[IF_THEN2297:%.*]], label [[IF_END2298:%.*]]
+// SIMD-ONLY0:       if.then2297:
+// SIMD-ONLY0-NEXT:    [[TMP948:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP948]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2298]]
+// SIMD-ONLY0:       if.end2298:
+// SIMD-ONLY0-NEXT:    [[TMP949:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2299:%.*]] = sext i16 [[TMP949]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP950:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2300:%.*]] = sext i16 [[TMP950]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2301:%.*]] = icmp slt i32 [[CONV2299]], [[CONV2300]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2301]], label [[IF_THEN2303:%.*]], label [[IF_END2304:%.*]]
+// SIMD-ONLY0:       if.then2303:
+// SIMD-ONLY0-NEXT:    [[TMP951:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP951]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2304]]
+// SIMD-ONLY0:       if.end2304:
+// SIMD-ONLY0-NEXT:    [[TMP952:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2305:%.*]] = sext i16 [[TMP952]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP953:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2306:%.*]] = sext i16 [[TMP953]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2307:%.*]] = icmp eq i32 [[CONV2305]], [[CONV2306]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2307]], label [[COND_TRUE2309:%.*]], label [[COND_FALSE2311:%.*]]
+// SIMD-ONLY0:       cond.true2309:
+// SIMD-ONLY0-NEXT:    [[TMP954:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2310:%.*]] = sext i16 [[TMP954]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2313:%.*]]
+// SIMD-ONLY0:       cond.false2311:
+// SIMD-ONLY0-NEXT:    [[TMP955:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2312:%.*]] = sext i16 [[TMP955]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2313]]
+// SIMD-ONLY0:       cond.end2313:
+// SIMD-ONLY0-NEXT:    [[COND2314:%.*]] = phi i32 [ [[CONV2310]], [[COND_TRUE2309]] ], [ [[CONV2312]], [[COND_FALSE2311]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2315:%.*]] = trunc i32 [[COND2314]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2315]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP956:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2316:%.*]] = sext i16 [[TMP956]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP957:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2317:%.*]] = sext i16 [[TMP957]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2318:%.*]] = icmp eq i32 [[CONV2316]], [[CONV2317]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2318]], label [[COND_TRUE2320:%.*]], label [[COND_FALSE2322:%.*]]
+// SIMD-ONLY0:       cond.true2320:
+// SIMD-ONLY0-NEXT:    [[TMP958:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2321:%.*]] = sext i16 [[TMP958]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2324:%.*]]
+// SIMD-ONLY0:       cond.false2322:
+// SIMD-ONLY0-NEXT:    [[TMP959:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2323:%.*]] = sext i16 [[TMP959]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2324]]
+// SIMD-ONLY0:       cond.end2324:
+// SIMD-ONLY0-NEXT:    [[COND2325:%.*]] = phi i32 [ [[CONV2321]], [[COND_TRUE2320]] ], [ [[CONV2323]], [[COND_FALSE2322]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2326:%.*]] = trunc i32 [[COND2325]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2326]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP960:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2327:%.*]] = sext i16 [[TMP960]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP961:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2328:%.*]] = sext i16 [[TMP961]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2329:%.*]] = icmp eq i32 [[CONV2327]], [[CONV2328]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2329]], label [[IF_THEN2331:%.*]], label [[IF_END2332:%.*]]
+// SIMD-ONLY0:       if.then2331:
+// SIMD-ONLY0-NEXT:    [[TMP962:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP962]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2332]]
+// SIMD-ONLY0:       if.end2332:
+// SIMD-ONLY0-NEXT:    [[TMP963:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2333:%.*]] = sext i16 [[TMP963]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP964:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2334:%.*]] = sext i16 [[TMP964]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2335:%.*]] = icmp eq i32 [[CONV2333]], [[CONV2334]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2335]], label [[IF_THEN2337:%.*]], label [[IF_END2338:%.*]]
+// SIMD-ONLY0:       if.then2337:
+// SIMD-ONLY0-NEXT:    [[TMP965:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP965]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2338]]
+// SIMD-ONLY0:       if.end2338:
+// SIMD-ONLY0-NEXT:    [[TMP966:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2339:%.*]] = zext i16 [[TMP966]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP967:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2340:%.*]] = zext i16 [[TMP967]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2341:%.*]] = icmp sgt i32 [[CONV2339]], [[CONV2340]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2341]], label [[COND_TRUE2343:%.*]], label [[COND_FALSE2345:%.*]]
+// SIMD-ONLY0:       cond.true2343:
+// SIMD-ONLY0-NEXT:    [[TMP968:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2344:%.*]] = zext i16 [[TMP968]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2347:%.*]]
+// SIMD-ONLY0:       cond.false2345:
+// SIMD-ONLY0-NEXT:    [[TMP969:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2346:%.*]] = zext i16 [[TMP969]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2347]]
+// SIMD-ONLY0:       cond.end2347:
+// SIMD-ONLY0-NEXT:    [[COND2348:%.*]] = phi i32 [ [[CONV2344]], [[COND_TRUE2343]] ], [ [[CONV2346]], [[COND_FALSE2345]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2349:%.*]] = trunc i32 [[COND2348]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2349]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP970:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2350:%.*]] = zext i16 [[TMP970]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP971:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2351:%.*]] = zext i16 [[TMP971]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2352:%.*]] = icmp slt i32 [[CONV2350]], [[CONV2351]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2352]], label [[COND_TRUE2354:%.*]], label [[COND_FALSE2356:%.*]]
+// SIMD-ONLY0:       cond.true2354:
+// SIMD-ONLY0-NEXT:    [[TMP972:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2355:%.*]] = zext i16 [[TMP972]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2358:%.*]]
+// SIMD-ONLY0:       cond.false2356:
+// SIMD-ONLY0-NEXT:    [[TMP973:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2357:%.*]] = zext i16 [[TMP973]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2358]]
+// SIMD-ONLY0:       cond.end2358:
+// SIMD-ONLY0-NEXT:    [[COND2359:%.*]] = phi i32 [ [[CONV2355]], [[COND_TRUE2354]] ], [ [[CONV2357]], [[COND_FALSE2356]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2360:%.*]] = trunc i32 [[COND2359]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2360]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP974:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2361:%.*]] = zext i16 [[TMP974]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP975:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2362:%.*]] = zext i16 [[TMP975]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2363:%.*]] = icmp sgt i32 [[CONV2361]], [[CONV2362]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2363]], label [[COND_TRUE2365:%.*]], label [[COND_FALSE2367:%.*]]
+// SIMD-ONLY0:       cond.true2365:
+// SIMD-ONLY0-NEXT:    [[TMP976:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2366:%.*]] = zext i16 [[TMP976]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2369:%.*]]
+// SIMD-ONLY0:       cond.false2367:
+// SIMD-ONLY0-NEXT:    [[TMP977:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2368:%.*]] = zext i16 [[TMP977]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2369]]
+// SIMD-ONLY0:       cond.end2369:
+// SIMD-ONLY0-NEXT:    [[COND2370:%.*]] = phi i32 [ [[CONV2366]], [[COND_TRUE2365]] ], [ [[CONV2368]], [[COND_FALSE2367]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2371:%.*]] = trunc i32 [[COND2370]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2371]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP978:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2372:%.*]] = zext i16 [[TMP978]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP979:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2373:%.*]] = zext i16 [[TMP979]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2374:%.*]] = icmp slt i32 [[CONV2372]], [[CONV2373]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2374]], label [[COND_TRUE2376:%.*]], label [[COND_FALSE2378:%.*]]
+// SIMD-ONLY0:       cond.true2376:
+// SIMD-ONLY0-NEXT:    [[TMP980:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2377:%.*]] = zext i16 [[TMP980]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2380:%.*]]
+// SIMD-ONLY0:       cond.false2378:
+// SIMD-ONLY0-NEXT:    [[TMP981:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2379:%.*]] = zext i16 [[TMP981]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2380]]
+// SIMD-ONLY0:       cond.end2380:
+// SIMD-ONLY0-NEXT:    [[COND2381:%.*]] = phi i32 [ [[CONV2377]], [[COND_TRUE2376]] ], [ [[CONV2379]], [[COND_FALSE2378]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2382:%.*]] = trunc i32 [[COND2381]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2382]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP982:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2383:%.*]] = zext i16 [[TMP982]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP983:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2384:%.*]] = zext i16 [[TMP983]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2385:%.*]] = icmp sgt i32 [[CONV2383]], [[CONV2384]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2385]], label [[IF_THEN2387:%.*]], label [[IF_END2388:%.*]]
+// SIMD-ONLY0:       if.then2387:
+// SIMD-ONLY0-NEXT:    [[TMP984:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP984]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2388]]
+// SIMD-ONLY0:       if.end2388:
+// SIMD-ONLY0-NEXT:    [[TMP985:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2389:%.*]] = zext i16 [[TMP985]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP986:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2390:%.*]] = zext i16 [[TMP986]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2391:%.*]] = icmp slt i32 [[CONV2389]], [[CONV2390]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2391]], label [[IF_THEN2393:%.*]], label [[IF_END2394:%.*]]
+// SIMD-ONLY0:       if.then2393:
+// SIMD-ONLY0-NEXT:    [[TMP987:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP987]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2394]]
+// SIMD-ONLY0:       if.end2394:
+// SIMD-ONLY0-NEXT:    [[TMP988:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2395:%.*]] = zext i16 [[TMP988]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP989:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2396:%.*]] = zext i16 [[TMP989]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2397:%.*]] = icmp sgt i32 [[CONV2395]], [[CONV2396]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2397]], label [[IF_THEN2399:%.*]], label [[IF_END2400:%.*]]
+// SIMD-ONLY0:       if.then2399:
+// SIMD-ONLY0-NEXT:    [[TMP990:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP990]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2400]]
+// SIMD-ONLY0:       if.end2400:
+// SIMD-ONLY0-NEXT:    [[TMP991:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2401:%.*]] = zext i16 [[TMP991]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP992:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2402:%.*]] = zext i16 [[TMP992]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2403:%.*]] = icmp slt i32 [[CONV2401]], [[CONV2402]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2403]], label [[IF_THEN2405:%.*]], label [[IF_END2406:%.*]]
+// SIMD-ONLY0:       if.then2405:
+// SIMD-ONLY0-NEXT:    [[TMP993:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP993]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2406]]
+// SIMD-ONLY0:       if.end2406:
+// SIMD-ONLY0-NEXT:    [[TMP994:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2407:%.*]] = zext i16 [[TMP994]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP995:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2408:%.*]] = zext i16 [[TMP995]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2409:%.*]] = icmp eq i32 [[CONV2407]], [[CONV2408]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2409]], label [[COND_TRUE2411:%.*]], label [[COND_FALSE2413:%.*]]
+// SIMD-ONLY0:       cond.true2411:
+// SIMD-ONLY0-NEXT:    [[TMP996:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2412:%.*]] = zext i16 [[TMP996]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2415:%.*]]
+// SIMD-ONLY0:       cond.false2413:
+// SIMD-ONLY0-NEXT:    [[TMP997:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2414:%.*]] = zext i16 [[TMP997]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2415]]
+// SIMD-ONLY0:       cond.end2415:
+// SIMD-ONLY0-NEXT:    [[COND2416:%.*]] = phi i32 [ [[CONV2412]], [[COND_TRUE2411]] ], [ [[CONV2414]], [[COND_FALSE2413]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2417:%.*]] = trunc i32 [[COND2416]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2417]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP998:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2418:%.*]] = zext i16 [[TMP998]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP999:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2419:%.*]] = zext i16 [[TMP999]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2420:%.*]] = icmp eq i32 [[CONV2418]], [[CONV2419]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2420]], label [[COND_TRUE2422:%.*]], label [[COND_FALSE2424:%.*]]
+// SIMD-ONLY0:       cond.true2422:
+// SIMD-ONLY0-NEXT:    [[TMP1000:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2423:%.*]] = zext i16 [[TMP1000]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2426:%.*]]
+// SIMD-ONLY0:       cond.false2424:
+// SIMD-ONLY0-NEXT:    [[TMP1001:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2425:%.*]] = zext i16 [[TMP1001]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END2426]]
+// SIMD-ONLY0:       cond.end2426:
+// SIMD-ONLY0-NEXT:    [[COND2427:%.*]] = phi i32 [ [[CONV2423]], [[COND_TRUE2422]] ], [ [[CONV2425]], [[COND_FALSE2424]] ]
+// SIMD-ONLY0-NEXT:    [[CONV2428:%.*]] = trunc i32 [[COND2427]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2428]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1002:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2429:%.*]] = zext i16 [[TMP1002]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1003:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2430:%.*]] = zext i16 [[TMP1003]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2431:%.*]] = icmp eq i32 [[CONV2429]], [[CONV2430]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2431]], label [[IF_THEN2433:%.*]], label [[IF_END2434:%.*]]
+// SIMD-ONLY0:       if.then2433:
+// SIMD-ONLY0-NEXT:    [[TMP1004:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1004]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2434]]
+// SIMD-ONLY0:       if.end2434:
+// SIMD-ONLY0-NEXT:    [[TMP1005:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2435:%.*]] = zext i16 [[TMP1005]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1006:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2436:%.*]] = zext i16 [[TMP1006]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2437:%.*]] = icmp eq i32 [[CONV2435]], [[CONV2436]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2437]], label [[IF_THEN2439:%.*]], label [[IF_END2440:%.*]]
+// SIMD-ONLY0:       if.then2439:
+// SIMD-ONLY0-NEXT:    [[TMP1007:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1007]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2440]]
+// SIMD-ONLY0:       if.end2440:
+// SIMD-ONLY0-NEXT:    [[TMP1008:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1009:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2441:%.*]] = icmp sgt i32 [[TMP1008]], [[TMP1009]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2441]], label [[COND_TRUE2443:%.*]], label [[COND_FALSE2444:%.*]]
+// SIMD-ONLY0:       cond.true2443:
+// SIMD-ONLY0-NEXT:    [[TMP1010:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2445:%.*]]
+// SIMD-ONLY0:       cond.false2444:
+// SIMD-ONLY0-NEXT:    [[TMP1011:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2445]]
+// SIMD-ONLY0:       cond.end2445:
+// SIMD-ONLY0-NEXT:    [[COND2446:%.*]] = phi i32 [ [[TMP1010]], [[COND_TRUE2443]] ], [ [[TMP1011]], [[COND_FALSE2444]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2446]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1012:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1013:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2447:%.*]] = icmp slt i32 [[TMP1012]], [[TMP1013]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2447]], label [[COND_TRUE2449:%.*]], label [[COND_FALSE2450:%.*]]
+// SIMD-ONLY0:       cond.true2449:
+// SIMD-ONLY0-NEXT:    [[TMP1014:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2451:%.*]]
+// SIMD-ONLY0:       cond.false2450:
+// SIMD-ONLY0-NEXT:    [[TMP1015:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2451]]
+// SIMD-ONLY0:       cond.end2451:
+// SIMD-ONLY0-NEXT:    [[COND2452:%.*]] = phi i32 [ [[TMP1014]], [[COND_TRUE2449]] ], [ [[TMP1015]], [[COND_FALSE2450]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2452]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1016:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1017:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2453:%.*]] = icmp sgt i32 [[TMP1016]], [[TMP1017]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2453]], label [[COND_TRUE2455:%.*]], label [[COND_FALSE2456:%.*]]
+// SIMD-ONLY0:       cond.true2455:
+// SIMD-ONLY0-NEXT:    [[TMP1018:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2457:%.*]]
+// SIMD-ONLY0:       cond.false2456:
+// SIMD-ONLY0-NEXT:    [[TMP1019:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2457]]
+// SIMD-ONLY0:       cond.end2457:
+// SIMD-ONLY0-NEXT:    [[COND2458:%.*]] = phi i32 [ [[TMP1018]], [[COND_TRUE2455]] ], [ [[TMP1019]], [[COND_FALSE2456]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2458]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1020:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1021:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2459:%.*]] = icmp slt i32 [[TMP1020]], [[TMP1021]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2459]], label [[COND_TRUE2461:%.*]], label [[COND_FALSE2462:%.*]]
+// SIMD-ONLY0:       cond.true2461:
+// SIMD-ONLY0-NEXT:    [[TMP1022:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2463:%.*]]
+// SIMD-ONLY0:       cond.false2462:
+// SIMD-ONLY0-NEXT:    [[TMP1023:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2463]]
+// SIMD-ONLY0:       cond.end2463:
+// SIMD-ONLY0-NEXT:    [[COND2464:%.*]] = phi i32 [ [[TMP1022]], [[COND_TRUE2461]] ], [ [[TMP1023]], [[COND_FALSE2462]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2464]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1024:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1025:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2465:%.*]] = icmp sgt i32 [[TMP1024]], [[TMP1025]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2465]], label [[IF_THEN2467:%.*]], label [[IF_END2468:%.*]]
+// SIMD-ONLY0:       if.then2467:
+// SIMD-ONLY0-NEXT:    [[TMP1026:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1026]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2468]]
+// SIMD-ONLY0:       if.end2468:
+// SIMD-ONLY0-NEXT:    [[TMP1027:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1028:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2469:%.*]] = icmp slt i32 [[TMP1027]], [[TMP1028]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2469]], label [[IF_THEN2471:%.*]], label [[IF_END2472:%.*]]
+// SIMD-ONLY0:       if.then2471:
+// SIMD-ONLY0-NEXT:    [[TMP1029:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1029]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2472]]
+// SIMD-ONLY0:       if.end2472:
+// SIMD-ONLY0-NEXT:    [[TMP1030:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1031:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2473:%.*]] = icmp sgt i32 [[TMP1030]], [[TMP1031]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2473]], label [[IF_THEN2475:%.*]], label [[IF_END2476:%.*]]
+// SIMD-ONLY0:       if.then2475:
+// SIMD-ONLY0-NEXT:    [[TMP1032:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1032]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2476]]
+// SIMD-ONLY0:       if.end2476:
+// SIMD-ONLY0-NEXT:    [[TMP1033:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1034:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2477:%.*]] = icmp slt i32 [[TMP1033]], [[TMP1034]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2477]], label [[IF_THEN2479:%.*]], label [[IF_END2480:%.*]]
+// SIMD-ONLY0:       if.then2479:
+// SIMD-ONLY0-NEXT:    [[TMP1035:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1035]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2480]]
+// SIMD-ONLY0:       if.end2480:
+// SIMD-ONLY0-NEXT:    [[TMP1036:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1037:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2481:%.*]] = icmp eq i32 [[TMP1036]], [[TMP1037]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2481]], label [[COND_TRUE2483:%.*]], label [[COND_FALSE2484:%.*]]
+// SIMD-ONLY0:       cond.true2483:
+// SIMD-ONLY0-NEXT:    [[TMP1038:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2485:%.*]]
+// SIMD-ONLY0:       cond.false2484:
+// SIMD-ONLY0-NEXT:    [[TMP1039:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2485]]
+// SIMD-ONLY0:       cond.end2485:
+// SIMD-ONLY0-NEXT:    [[COND2486:%.*]] = phi i32 [ [[TMP1038]], [[COND_TRUE2483]] ], [ [[TMP1039]], [[COND_FALSE2484]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2486]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1040:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1041:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2487:%.*]] = icmp eq i32 [[TMP1040]], [[TMP1041]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2487]], label [[COND_TRUE2489:%.*]], label [[COND_FALSE2490:%.*]]
+// SIMD-ONLY0:       cond.true2489:
+// SIMD-ONLY0-NEXT:    [[TMP1042:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2491:%.*]]
+// SIMD-ONLY0:       cond.false2490:
+// SIMD-ONLY0-NEXT:    [[TMP1043:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2491]]
+// SIMD-ONLY0:       cond.end2491:
+// SIMD-ONLY0-NEXT:    [[COND2492:%.*]] = phi i32 [ [[TMP1042]], [[COND_TRUE2489]] ], [ [[TMP1043]], [[COND_FALSE2490]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2492]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1044:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1045:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2493:%.*]] = icmp eq i32 [[TMP1044]], [[TMP1045]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2493]], label [[IF_THEN2495:%.*]], label [[IF_END2496:%.*]]
+// SIMD-ONLY0:       if.then2495:
+// SIMD-ONLY0-NEXT:    [[TMP1046:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1046]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2496]]
+// SIMD-ONLY0:       if.end2496:
+// SIMD-ONLY0-NEXT:    [[TMP1047:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1048:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2497:%.*]] = icmp eq i32 [[TMP1047]], [[TMP1048]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2497]], label [[IF_THEN2499:%.*]], label [[IF_END2500:%.*]]
+// SIMD-ONLY0:       if.then2499:
+// SIMD-ONLY0-NEXT:    [[TMP1049:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1049]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2500]]
+// SIMD-ONLY0:       if.end2500:
+// SIMD-ONLY0-NEXT:    [[TMP1050:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1051:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2501:%.*]] = icmp ugt i32 [[TMP1050]], [[TMP1051]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2501]], label [[COND_TRUE2503:%.*]], label [[COND_FALSE2504:%.*]]
+// SIMD-ONLY0:       cond.true2503:
+// SIMD-ONLY0-NEXT:    [[TMP1052:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2505:%.*]]
+// SIMD-ONLY0:       cond.false2504:
+// SIMD-ONLY0-NEXT:    [[TMP1053:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2505]]
+// SIMD-ONLY0:       cond.end2505:
+// SIMD-ONLY0-NEXT:    [[COND2506:%.*]] = phi i32 [ [[TMP1052]], [[COND_TRUE2503]] ], [ [[TMP1053]], [[COND_FALSE2504]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2506]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1054:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1055:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2507:%.*]] = icmp ult i32 [[TMP1054]], [[TMP1055]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2507]], label [[COND_TRUE2509:%.*]], label [[COND_FALSE2510:%.*]]
+// SIMD-ONLY0:       cond.true2509:
+// SIMD-ONLY0-NEXT:    [[TMP1056:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2511:%.*]]
+// SIMD-ONLY0:       cond.false2510:
+// SIMD-ONLY0-NEXT:    [[TMP1057:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2511]]
+// SIMD-ONLY0:       cond.end2511:
+// SIMD-ONLY0-NEXT:    [[COND2512:%.*]] = phi i32 [ [[TMP1056]], [[COND_TRUE2509]] ], [ [[TMP1057]], [[COND_FALSE2510]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2512]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1058:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1059:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2513:%.*]] = icmp ugt i32 [[TMP1058]], [[TMP1059]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2513]], label [[COND_TRUE2515:%.*]], label [[COND_FALSE2516:%.*]]
+// SIMD-ONLY0:       cond.true2515:
+// SIMD-ONLY0-NEXT:    [[TMP1060:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2517:%.*]]
+// SIMD-ONLY0:       cond.false2516:
+// SIMD-ONLY0-NEXT:    [[TMP1061:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2517]]
+// SIMD-ONLY0:       cond.end2517:
+// SIMD-ONLY0-NEXT:    [[COND2518:%.*]] = phi i32 [ [[TMP1060]], [[COND_TRUE2515]] ], [ [[TMP1061]], [[COND_FALSE2516]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2518]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1062:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1063:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2519:%.*]] = icmp ult i32 [[TMP1062]], [[TMP1063]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2519]], label [[COND_TRUE2521:%.*]], label [[COND_FALSE2522:%.*]]
+// SIMD-ONLY0:       cond.true2521:
+// SIMD-ONLY0-NEXT:    [[TMP1064:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2523:%.*]]
+// SIMD-ONLY0:       cond.false2522:
+// SIMD-ONLY0-NEXT:    [[TMP1065:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2523]]
+// SIMD-ONLY0:       cond.end2523:
+// SIMD-ONLY0-NEXT:    [[COND2524:%.*]] = phi i32 [ [[TMP1064]], [[COND_TRUE2521]] ], [ [[TMP1065]], [[COND_FALSE2522]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2524]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1066:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1067:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2525:%.*]] = icmp ugt i32 [[TMP1066]], [[TMP1067]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2525]], label [[IF_THEN2527:%.*]], label [[IF_END2528:%.*]]
+// SIMD-ONLY0:       if.then2527:
+// SIMD-ONLY0-NEXT:    [[TMP1068:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1068]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2528]]
+// SIMD-ONLY0:       if.end2528:
+// SIMD-ONLY0-NEXT:    [[TMP1069:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1070:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2529:%.*]] = icmp ult i32 [[TMP1069]], [[TMP1070]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2529]], label [[IF_THEN2531:%.*]], label [[IF_END2532:%.*]]
+// SIMD-ONLY0:       if.then2531:
+// SIMD-ONLY0-NEXT:    [[TMP1071:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1071]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2532]]
+// SIMD-ONLY0:       if.end2532:
+// SIMD-ONLY0-NEXT:    [[TMP1072:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1073:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2533:%.*]] = icmp ugt i32 [[TMP1072]], [[TMP1073]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2533]], label [[IF_THEN2535:%.*]], label [[IF_END2536:%.*]]
+// SIMD-ONLY0:       if.then2535:
+// SIMD-ONLY0-NEXT:    [[TMP1074:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1074]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2536]]
+// SIMD-ONLY0:       if.end2536:
+// SIMD-ONLY0-NEXT:    [[TMP1075:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1076:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2537:%.*]] = icmp ult i32 [[TMP1075]], [[TMP1076]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2537]], label [[IF_THEN2539:%.*]], label [[IF_END2540:%.*]]
+// SIMD-ONLY0:       if.then2539:
+// SIMD-ONLY0-NEXT:    [[TMP1077:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1077]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2540]]
+// SIMD-ONLY0:       if.end2540:
+// SIMD-ONLY0-NEXT:    [[TMP1078:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1079:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2541:%.*]] = icmp eq i32 [[TMP1078]], [[TMP1079]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2541]], label [[COND_TRUE2543:%.*]], label [[COND_FALSE2544:%.*]]
+// SIMD-ONLY0:       cond.true2543:
+// SIMD-ONLY0-NEXT:    [[TMP1080:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2545:%.*]]
+// SIMD-ONLY0:       cond.false2544:
+// SIMD-ONLY0-NEXT:    [[TMP1081:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2545]]
+// SIMD-ONLY0:       cond.end2545:
+// SIMD-ONLY0-NEXT:    [[COND2546:%.*]] = phi i32 [ [[TMP1080]], [[COND_TRUE2543]] ], [ [[TMP1081]], [[COND_FALSE2544]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2546]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1082:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1083:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2547:%.*]] = icmp eq i32 [[TMP1082]], [[TMP1083]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2547]], label [[COND_TRUE2549:%.*]], label [[COND_FALSE2550:%.*]]
+// SIMD-ONLY0:       cond.true2549:
+// SIMD-ONLY0-NEXT:    [[TMP1084:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2551:%.*]]
+// SIMD-ONLY0:       cond.false2550:
+// SIMD-ONLY0-NEXT:    [[TMP1085:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2551]]
+// SIMD-ONLY0:       cond.end2551:
+// SIMD-ONLY0-NEXT:    [[COND2552:%.*]] = phi i32 [ [[TMP1084]], [[COND_TRUE2549]] ], [ [[TMP1085]], [[COND_FALSE2550]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2552]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1086:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1087:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2553:%.*]] = icmp eq i32 [[TMP1086]], [[TMP1087]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2553]], label [[IF_THEN2555:%.*]], label [[IF_END2556:%.*]]
+// SIMD-ONLY0:       if.then2555:
+// SIMD-ONLY0-NEXT:    [[TMP1088:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1088]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2556]]
+// SIMD-ONLY0:       if.end2556:
+// SIMD-ONLY0-NEXT:    [[TMP1089:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1090:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2557:%.*]] = icmp eq i32 [[TMP1089]], [[TMP1090]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2557]], label [[IF_THEN2559:%.*]], label [[IF_END2560:%.*]]
+// SIMD-ONLY0:       if.then2559:
+// SIMD-ONLY0-NEXT:    [[TMP1091:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1091]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2560]]
+// SIMD-ONLY0:       if.end2560:
+// SIMD-ONLY0-NEXT:    [[TMP1092:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1093:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2561:%.*]] = icmp sgt i32 [[TMP1092]], [[TMP1093]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2561]], label [[COND_TRUE2563:%.*]], label [[COND_FALSE2564:%.*]]
+// SIMD-ONLY0:       cond.true2563:
+// SIMD-ONLY0-NEXT:    [[TMP1094:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2565:%.*]]
+// SIMD-ONLY0:       cond.false2564:
+// SIMD-ONLY0-NEXT:    [[TMP1095:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2565]]
+// SIMD-ONLY0:       cond.end2565:
+// SIMD-ONLY0-NEXT:    [[COND2566:%.*]] = phi i32 [ [[TMP1094]], [[COND_TRUE2563]] ], [ [[TMP1095]], [[COND_FALSE2564]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2566]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1096:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1097:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2567:%.*]] = icmp slt i32 [[TMP1096]], [[TMP1097]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2567]], label [[COND_TRUE2569:%.*]], label [[COND_FALSE2570:%.*]]
+// SIMD-ONLY0:       cond.true2569:
+// SIMD-ONLY0-NEXT:    [[TMP1098:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2571:%.*]]
+// SIMD-ONLY0:       cond.false2570:
+// SIMD-ONLY0-NEXT:    [[TMP1099:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2571]]
+// SIMD-ONLY0:       cond.end2571:
+// SIMD-ONLY0-NEXT:    [[COND2572:%.*]] = phi i32 [ [[TMP1098]], [[COND_TRUE2569]] ], [ [[TMP1099]], [[COND_FALSE2570]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2572]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1100:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1101:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2573:%.*]] = icmp sgt i32 [[TMP1100]], [[TMP1101]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2573]], label [[COND_TRUE2575:%.*]], label [[COND_FALSE2576:%.*]]
+// SIMD-ONLY0:       cond.true2575:
+// SIMD-ONLY0-NEXT:    [[TMP1102:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2577:%.*]]
+// SIMD-ONLY0:       cond.false2576:
+// SIMD-ONLY0-NEXT:    [[TMP1103:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2577]]
+// SIMD-ONLY0:       cond.end2577:
+// SIMD-ONLY0-NEXT:    [[COND2578:%.*]] = phi i32 [ [[TMP1102]], [[COND_TRUE2575]] ], [ [[TMP1103]], [[COND_FALSE2576]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2578]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1104:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1105:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2579:%.*]] = icmp slt i32 [[TMP1104]], [[TMP1105]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2579]], label [[COND_TRUE2581:%.*]], label [[COND_FALSE2582:%.*]]
+// SIMD-ONLY0:       cond.true2581:
+// SIMD-ONLY0-NEXT:    [[TMP1106:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2583:%.*]]
+// SIMD-ONLY0:       cond.false2582:
+// SIMD-ONLY0-NEXT:    [[TMP1107:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2583]]
+// SIMD-ONLY0:       cond.end2583:
+// SIMD-ONLY0-NEXT:    [[COND2584:%.*]] = phi i32 [ [[TMP1106]], [[COND_TRUE2581]] ], [ [[TMP1107]], [[COND_FALSE2582]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2584]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1108:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1109:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2585:%.*]] = icmp sgt i32 [[TMP1108]], [[TMP1109]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2585]], label [[IF_THEN2587:%.*]], label [[IF_END2588:%.*]]
+// SIMD-ONLY0:       if.then2587:
+// SIMD-ONLY0-NEXT:    [[TMP1110:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1110]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2588]]
+// SIMD-ONLY0:       if.end2588:
+// SIMD-ONLY0-NEXT:    [[TMP1111:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1112:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2589:%.*]] = icmp slt i32 [[TMP1111]], [[TMP1112]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2589]], label [[IF_THEN2591:%.*]], label [[IF_END2592:%.*]]
+// SIMD-ONLY0:       if.then2591:
+// SIMD-ONLY0-NEXT:    [[TMP1113:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1113]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2592]]
+// SIMD-ONLY0:       if.end2592:
+// SIMD-ONLY0-NEXT:    [[TMP1114:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1115:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2593:%.*]] = icmp sgt i32 [[TMP1114]], [[TMP1115]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2593]], label [[IF_THEN2595:%.*]], label [[IF_END2596:%.*]]
+// SIMD-ONLY0:       if.then2595:
+// SIMD-ONLY0-NEXT:    [[TMP1116:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1116]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2596]]
+// SIMD-ONLY0:       if.end2596:
+// SIMD-ONLY0-NEXT:    [[TMP1117:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1118:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2597:%.*]] = icmp slt i32 [[TMP1117]], [[TMP1118]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2597]], label [[IF_THEN2599:%.*]], label [[IF_END2600:%.*]]
+// SIMD-ONLY0:       if.then2599:
+// SIMD-ONLY0-NEXT:    [[TMP1119:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1119]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2600]]
+// SIMD-ONLY0:       if.end2600:
+// SIMD-ONLY0-NEXT:    [[TMP1120:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1121:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2601:%.*]] = icmp eq i32 [[TMP1120]], [[TMP1121]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2601]], label [[COND_TRUE2603:%.*]], label [[COND_FALSE2604:%.*]]
+// SIMD-ONLY0:       cond.true2603:
+// SIMD-ONLY0-NEXT:    [[TMP1122:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2605:%.*]]
+// SIMD-ONLY0:       cond.false2604:
+// SIMD-ONLY0-NEXT:    [[TMP1123:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2605]]
+// SIMD-ONLY0:       cond.end2605:
+// SIMD-ONLY0-NEXT:    [[COND2606:%.*]] = phi i32 [ [[TMP1122]], [[COND_TRUE2603]] ], [ [[TMP1123]], [[COND_FALSE2604]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2606]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1124:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1125:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2607:%.*]] = icmp eq i32 [[TMP1124]], [[TMP1125]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2607]], label [[COND_TRUE2609:%.*]], label [[COND_FALSE2610:%.*]]
+// SIMD-ONLY0:       cond.true2609:
+// SIMD-ONLY0-NEXT:    [[TMP1126:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2611:%.*]]
+// SIMD-ONLY0:       cond.false2610:
+// SIMD-ONLY0-NEXT:    [[TMP1127:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2611]]
+// SIMD-ONLY0:       cond.end2611:
+// SIMD-ONLY0-NEXT:    [[COND2612:%.*]] = phi i32 [ [[TMP1126]], [[COND_TRUE2609]] ], [ [[TMP1127]], [[COND_FALSE2610]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2612]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1128:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1129:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2613:%.*]] = icmp eq i32 [[TMP1128]], [[TMP1129]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2613]], label [[IF_THEN2615:%.*]], label [[IF_END2616:%.*]]
+// SIMD-ONLY0:       if.then2615:
+// SIMD-ONLY0-NEXT:    [[TMP1130:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1130]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2616]]
+// SIMD-ONLY0:       if.end2616:
+// SIMD-ONLY0-NEXT:    [[TMP1131:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1132:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2617:%.*]] = icmp eq i32 [[TMP1131]], [[TMP1132]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2617]], label [[IF_THEN2619:%.*]], label [[IF_END2620:%.*]]
+// SIMD-ONLY0:       if.then2619:
+// SIMD-ONLY0-NEXT:    [[TMP1133:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1133]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2620]]
+// SIMD-ONLY0:       if.end2620:
+// SIMD-ONLY0-NEXT:    [[TMP1134:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1135:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2621:%.*]] = icmp ugt i32 [[TMP1134]], [[TMP1135]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2621]], label [[COND_TRUE2623:%.*]], label [[COND_FALSE2624:%.*]]
+// SIMD-ONLY0:       cond.true2623:
+// SIMD-ONLY0-NEXT:    [[TMP1136:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2625:%.*]]
+// SIMD-ONLY0:       cond.false2624:
+// SIMD-ONLY0-NEXT:    [[TMP1137:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2625]]
+// SIMD-ONLY0:       cond.end2625:
+// SIMD-ONLY0-NEXT:    [[COND2626:%.*]] = phi i32 [ [[TMP1136]], [[COND_TRUE2623]] ], [ [[TMP1137]], [[COND_FALSE2624]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2626]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1138:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1139:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2627:%.*]] = icmp ult i32 [[TMP1138]], [[TMP1139]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2627]], label [[COND_TRUE2629:%.*]], label [[COND_FALSE2630:%.*]]
+// SIMD-ONLY0:       cond.true2629:
+// SIMD-ONLY0-NEXT:    [[TMP1140:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2631:%.*]]
+// SIMD-ONLY0:       cond.false2630:
+// SIMD-ONLY0-NEXT:    [[TMP1141:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2631]]
+// SIMD-ONLY0:       cond.end2631:
+// SIMD-ONLY0-NEXT:    [[COND2632:%.*]] = phi i32 [ [[TMP1140]], [[COND_TRUE2629]] ], [ [[TMP1141]], [[COND_FALSE2630]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2632]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1142:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1143:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2633:%.*]] = icmp ugt i32 [[TMP1142]], [[TMP1143]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2633]], label [[COND_TRUE2635:%.*]], label [[COND_FALSE2636:%.*]]
+// SIMD-ONLY0:       cond.true2635:
+// SIMD-ONLY0-NEXT:    [[TMP1144:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2637:%.*]]
+// SIMD-ONLY0:       cond.false2636:
+// SIMD-ONLY0-NEXT:    [[TMP1145:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2637]]
+// SIMD-ONLY0:       cond.end2637:
+// SIMD-ONLY0-NEXT:    [[COND2638:%.*]] = phi i32 [ [[TMP1144]], [[COND_TRUE2635]] ], [ [[TMP1145]], [[COND_FALSE2636]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2638]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1146:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1147:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2639:%.*]] = icmp ult i32 [[TMP1146]], [[TMP1147]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2639]], label [[COND_TRUE2641:%.*]], label [[COND_FALSE2642:%.*]]
+// SIMD-ONLY0:       cond.true2641:
+// SIMD-ONLY0-NEXT:    [[TMP1148:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2643:%.*]]
+// SIMD-ONLY0:       cond.false2642:
+// SIMD-ONLY0-NEXT:    [[TMP1149:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2643]]
+// SIMD-ONLY0:       cond.end2643:
+// SIMD-ONLY0-NEXT:    [[COND2644:%.*]] = phi i32 [ [[TMP1148]], [[COND_TRUE2641]] ], [ [[TMP1149]], [[COND_FALSE2642]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2644]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1150:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1151:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2645:%.*]] = icmp ugt i32 [[TMP1150]], [[TMP1151]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2645]], label [[IF_THEN2647:%.*]], label [[IF_END2648:%.*]]
+// SIMD-ONLY0:       if.then2647:
+// SIMD-ONLY0-NEXT:    [[TMP1152:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1152]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2648]]
+// SIMD-ONLY0:       if.end2648:
+// SIMD-ONLY0-NEXT:    [[TMP1153:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1154:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2649:%.*]] = icmp ult i32 [[TMP1153]], [[TMP1154]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2649]], label [[IF_THEN2651:%.*]], label [[IF_END2652:%.*]]
+// SIMD-ONLY0:       if.then2651:
+// SIMD-ONLY0-NEXT:    [[TMP1155:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1155]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2652]]
+// SIMD-ONLY0:       if.end2652:
+// SIMD-ONLY0-NEXT:    [[TMP1156:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1157:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2653:%.*]] = icmp ugt i32 [[TMP1156]], [[TMP1157]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2653]], label [[IF_THEN2655:%.*]], label [[IF_END2656:%.*]]
+// SIMD-ONLY0:       if.then2655:
+// SIMD-ONLY0-NEXT:    [[TMP1158:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1158]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2656]]
+// SIMD-ONLY0:       if.end2656:
+// SIMD-ONLY0-NEXT:    [[TMP1159:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1160:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2657:%.*]] = icmp ult i32 [[TMP1159]], [[TMP1160]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2657]], label [[IF_THEN2659:%.*]], label [[IF_END2660:%.*]]
+// SIMD-ONLY0:       if.then2659:
+// SIMD-ONLY0-NEXT:    [[TMP1161:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1161]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2660]]
+// SIMD-ONLY0:       if.end2660:
+// SIMD-ONLY0-NEXT:    [[TMP1162:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1163:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2661:%.*]] = icmp eq i32 [[TMP1162]], [[TMP1163]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2661]], label [[COND_TRUE2663:%.*]], label [[COND_FALSE2664:%.*]]
+// SIMD-ONLY0:       cond.true2663:
+// SIMD-ONLY0-NEXT:    [[TMP1164:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2665:%.*]]
+// SIMD-ONLY0:       cond.false2664:
+// SIMD-ONLY0-NEXT:    [[TMP1165:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2665]]
+// SIMD-ONLY0:       cond.end2665:
+// SIMD-ONLY0-NEXT:    [[COND2666:%.*]] = phi i32 [ [[TMP1164]], [[COND_TRUE2663]] ], [ [[TMP1165]], [[COND_FALSE2664]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2666]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1166:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1167:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2667:%.*]] = icmp eq i32 [[TMP1166]], [[TMP1167]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2667]], label [[COND_TRUE2669:%.*]], label [[COND_FALSE2670:%.*]]
+// SIMD-ONLY0:       cond.true2669:
+// SIMD-ONLY0-NEXT:    [[TMP1168:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2671:%.*]]
+// SIMD-ONLY0:       cond.false2670:
+// SIMD-ONLY0-NEXT:    [[TMP1169:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2671]]
+// SIMD-ONLY0:       cond.end2671:
+// SIMD-ONLY0-NEXT:    [[COND2672:%.*]] = phi i32 [ [[TMP1168]], [[COND_TRUE2669]] ], [ [[TMP1169]], [[COND_FALSE2670]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2672]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1170:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1171:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2673:%.*]] = icmp eq i32 [[TMP1170]], [[TMP1171]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2673]], label [[IF_THEN2675:%.*]], label [[IF_END2676:%.*]]
+// SIMD-ONLY0:       if.then2675:
+// SIMD-ONLY0-NEXT:    [[TMP1172:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1172]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2676]]
+// SIMD-ONLY0:       if.end2676:
+// SIMD-ONLY0-NEXT:    [[TMP1173:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1174:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2677:%.*]] = icmp eq i32 [[TMP1173]], [[TMP1174]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2677]], label [[IF_THEN2679:%.*]], label [[IF_END2680:%.*]]
+// SIMD-ONLY0:       if.then2679:
+// SIMD-ONLY0-NEXT:    [[TMP1175:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1175]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2680]]
+// SIMD-ONLY0:       if.end2680:
+// SIMD-ONLY0-NEXT:    [[TMP1176:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1177:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2681:%.*]] = icmp sgt i32 [[TMP1176]], [[TMP1177]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2681]], label [[COND_TRUE2683:%.*]], label [[COND_FALSE2684:%.*]]
+// SIMD-ONLY0:       cond.true2683:
+// SIMD-ONLY0-NEXT:    [[TMP1178:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2685:%.*]]
+// SIMD-ONLY0:       cond.false2684:
+// SIMD-ONLY0-NEXT:    [[TMP1179:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2685]]
+// SIMD-ONLY0:       cond.end2685:
+// SIMD-ONLY0-NEXT:    [[COND2686:%.*]] = phi i32 [ [[TMP1178]], [[COND_TRUE2683]] ], [ [[TMP1179]], [[COND_FALSE2684]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2686]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1180:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1181:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2687:%.*]] = icmp slt i32 [[TMP1180]], [[TMP1181]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2687]], label [[COND_TRUE2689:%.*]], label [[COND_FALSE2690:%.*]]
+// SIMD-ONLY0:       cond.true2689:
+// SIMD-ONLY0-NEXT:    [[TMP1182:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2691:%.*]]
+// SIMD-ONLY0:       cond.false2690:
+// SIMD-ONLY0-NEXT:    [[TMP1183:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2691]]
+// SIMD-ONLY0:       cond.end2691:
+// SIMD-ONLY0-NEXT:    [[COND2692:%.*]] = phi i32 [ [[TMP1182]], [[COND_TRUE2689]] ], [ [[TMP1183]], [[COND_FALSE2690]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2692]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1184:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1185:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2693:%.*]] = icmp sgt i32 [[TMP1184]], [[TMP1185]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2693]], label [[COND_TRUE2695:%.*]], label [[COND_FALSE2696:%.*]]
+// SIMD-ONLY0:       cond.true2695:
+// SIMD-ONLY0-NEXT:    [[TMP1186:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2697:%.*]]
+// SIMD-ONLY0:       cond.false2696:
+// SIMD-ONLY0-NEXT:    [[TMP1187:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2697]]
+// SIMD-ONLY0:       cond.end2697:
+// SIMD-ONLY0-NEXT:    [[COND2698:%.*]] = phi i32 [ [[TMP1186]], [[COND_TRUE2695]] ], [ [[TMP1187]], [[COND_FALSE2696]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2698]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1188:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1189:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2699:%.*]] = icmp slt i32 [[TMP1188]], [[TMP1189]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2699]], label [[COND_TRUE2701:%.*]], label [[COND_FALSE2702:%.*]]
+// SIMD-ONLY0:       cond.true2701:
+// SIMD-ONLY0-NEXT:    [[TMP1190:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2703:%.*]]
+// SIMD-ONLY0:       cond.false2702:
+// SIMD-ONLY0-NEXT:    [[TMP1191:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2703]]
+// SIMD-ONLY0:       cond.end2703:
+// SIMD-ONLY0-NEXT:    [[COND2704:%.*]] = phi i32 [ [[TMP1190]], [[COND_TRUE2701]] ], [ [[TMP1191]], [[COND_FALSE2702]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2704]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1192:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1193:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2705:%.*]] = icmp sgt i32 [[TMP1192]], [[TMP1193]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2705]], label [[IF_THEN2707:%.*]], label [[IF_END2708:%.*]]
+// SIMD-ONLY0:       if.then2707:
+// SIMD-ONLY0-NEXT:    [[TMP1194:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1194]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2708]]
+// SIMD-ONLY0:       if.end2708:
+// SIMD-ONLY0-NEXT:    [[TMP1195:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1196:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2709:%.*]] = icmp slt i32 [[TMP1195]], [[TMP1196]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2709]], label [[IF_THEN2711:%.*]], label [[IF_END2712:%.*]]
+// SIMD-ONLY0:       if.then2711:
+// SIMD-ONLY0-NEXT:    [[TMP1197:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1197]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2712]]
+// SIMD-ONLY0:       if.end2712:
+// SIMD-ONLY0-NEXT:    [[TMP1198:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1199:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2713:%.*]] = icmp sgt i32 [[TMP1198]], [[TMP1199]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2713]], label [[IF_THEN2715:%.*]], label [[IF_END2716:%.*]]
+// SIMD-ONLY0:       if.then2715:
+// SIMD-ONLY0-NEXT:    [[TMP1200:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1200]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2716]]
+// SIMD-ONLY0:       if.end2716:
+// SIMD-ONLY0-NEXT:    [[TMP1201:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1202:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2717:%.*]] = icmp slt i32 [[TMP1201]], [[TMP1202]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2717]], label [[IF_THEN2719:%.*]], label [[IF_END2720:%.*]]
+// SIMD-ONLY0:       if.then2719:
+// SIMD-ONLY0-NEXT:    [[TMP1203:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1203]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2720]]
+// SIMD-ONLY0:       if.end2720:
+// SIMD-ONLY0-NEXT:    [[TMP1204:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1205:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2721:%.*]] = icmp eq i32 [[TMP1204]], [[TMP1205]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2721]], label [[COND_TRUE2723:%.*]], label [[COND_FALSE2724:%.*]]
+// SIMD-ONLY0:       cond.true2723:
+// SIMD-ONLY0-NEXT:    [[TMP1206:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2725:%.*]]
+// SIMD-ONLY0:       cond.false2724:
+// SIMD-ONLY0-NEXT:    [[TMP1207:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2725]]
+// SIMD-ONLY0:       cond.end2725:
+// SIMD-ONLY0-NEXT:    [[COND2726:%.*]] = phi i32 [ [[TMP1206]], [[COND_TRUE2723]] ], [ [[TMP1207]], [[COND_FALSE2724]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2726]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1208:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1209:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2727:%.*]] = icmp eq i32 [[TMP1208]], [[TMP1209]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2727]], label [[COND_TRUE2729:%.*]], label [[COND_FALSE2730:%.*]]
+// SIMD-ONLY0:       cond.true2729:
+// SIMD-ONLY0-NEXT:    [[TMP1210:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2731:%.*]]
+// SIMD-ONLY0:       cond.false2730:
+// SIMD-ONLY0-NEXT:    [[TMP1211:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2731]]
+// SIMD-ONLY0:       cond.end2731:
+// SIMD-ONLY0-NEXT:    [[COND2732:%.*]] = phi i32 [ [[TMP1210]], [[COND_TRUE2729]] ], [ [[TMP1211]], [[COND_FALSE2730]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2732]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1212:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1213:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2733:%.*]] = icmp eq i32 [[TMP1212]], [[TMP1213]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2733]], label [[IF_THEN2735:%.*]], label [[IF_END2736:%.*]]
+// SIMD-ONLY0:       if.then2735:
+// SIMD-ONLY0-NEXT:    [[TMP1214:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1214]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2736]]
+// SIMD-ONLY0:       if.end2736:
+// SIMD-ONLY0-NEXT:    [[TMP1215:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1216:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2737:%.*]] = icmp eq i32 [[TMP1215]], [[TMP1216]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2737]], label [[IF_THEN2739:%.*]], label [[IF_END2740:%.*]]
+// SIMD-ONLY0:       if.then2739:
+// SIMD-ONLY0-NEXT:    [[TMP1217:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1217]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2740]]
+// SIMD-ONLY0:       if.end2740:
+// SIMD-ONLY0-NEXT:    [[TMP1218:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1219:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2741:%.*]] = icmp ugt i32 [[TMP1218]], [[TMP1219]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2741]], label [[COND_TRUE2743:%.*]], label [[COND_FALSE2744:%.*]]
+// SIMD-ONLY0:       cond.true2743:
+// SIMD-ONLY0-NEXT:    [[TMP1220:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2745:%.*]]
+// SIMD-ONLY0:       cond.false2744:
+// SIMD-ONLY0-NEXT:    [[TMP1221:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2745]]
+// SIMD-ONLY0:       cond.end2745:
+// SIMD-ONLY0-NEXT:    [[COND2746:%.*]] = phi i32 [ [[TMP1220]], [[COND_TRUE2743]] ], [ [[TMP1221]], [[COND_FALSE2744]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2746]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1222:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1223:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2747:%.*]] = icmp ult i32 [[TMP1222]], [[TMP1223]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2747]], label [[COND_TRUE2749:%.*]], label [[COND_FALSE2750:%.*]]
+// SIMD-ONLY0:       cond.true2749:
+// SIMD-ONLY0-NEXT:    [[TMP1224:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2751:%.*]]
+// SIMD-ONLY0:       cond.false2750:
+// SIMD-ONLY0-NEXT:    [[TMP1225:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2751]]
+// SIMD-ONLY0:       cond.end2751:
+// SIMD-ONLY0-NEXT:    [[COND2752:%.*]] = phi i32 [ [[TMP1224]], [[COND_TRUE2749]] ], [ [[TMP1225]], [[COND_FALSE2750]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2752]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1226:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1227:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2753:%.*]] = icmp ugt i32 [[TMP1226]], [[TMP1227]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2753]], label [[COND_TRUE2755:%.*]], label [[COND_FALSE2756:%.*]]
+// SIMD-ONLY0:       cond.true2755:
+// SIMD-ONLY0-NEXT:    [[TMP1228:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2757:%.*]]
+// SIMD-ONLY0:       cond.false2756:
+// SIMD-ONLY0-NEXT:    [[TMP1229:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2757]]
+// SIMD-ONLY0:       cond.end2757:
+// SIMD-ONLY0-NEXT:    [[COND2758:%.*]] = phi i32 [ [[TMP1228]], [[COND_TRUE2755]] ], [ [[TMP1229]], [[COND_FALSE2756]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2758]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1230:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1231:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2759:%.*]] = icmp ult i32 [[TMP1230]], [[TMP1231]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2759]], label [[COND_TRUE2761:%.*]], label [[COND_FALSE2762:%.*]]
+// SIMD-ONLY0:       cond.true2761:
+// SIMD-ONLY0-NEXT:    [[TMP1232:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2763:%.*]]
+// SIMD-ONLY0:       cond.false2762:
+// SIMD-ONLY0-NEXT:    [[TMP1233:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2763]]
+// SIMD-ONLY0:       cond.end2763:
+// SIMD-ONLY0-NEXT:    [[COND2764:%.*]] = phi i32 [ [[TMP1232]], [[COND_TRUE2761]] ], [ [[TMP1233]], [[COND_FALSE2762]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2764]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1234:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1235:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2765:%.*]] = icmp ugt i32 [[TMP1234]], [[TMP1235]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2765]], label [[IF_THEN2767:%.*]], label [[IF_END2768:%.*]]
+// SIMD-ONLY0:       if.then2767:
+// SIMD-ONLY0-NEXT:    [[TMP1236:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1236]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2768]]
+// SIMD-ONLY0:       if.end2768:
+// SIMD-ONLY0-NEXT:    [[TMP1237:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1238:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2769:%.*]] = icmp ult i32 [[TMP1237]], [[TMP1238]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2769]], label [[IF_THEN2771:%.*]], label [[IF_END2772:%.*]]
+// SIMD-ONLY0:       if.then2771:
+// SIMD-ONLY0-NEXT:    [[TMP1239:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1239]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2772]]
+// SIMD-ONLY0:       if.end2772:
+// SIMD-ONLY0-NEXT:    [[TMP1240:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1241:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2773:%.*]] = icmp ugt i32 [[TMP1240]], [[TMP1241]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2773]], label [[IF_THEN2775:%.*]], label [[IF_END2776:%.*]]
+// SIMD-ONLY0:       if.then2775:
+// SIMD-ONLY0-NEXT:    [[TMP1242:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1242]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2776]]
+// SIMD-ONLY0:       if.end2776:
+// SIMD-ONLY0-NEXT:    [[TMP1243:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1244:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2777:%.*]] = icmp ult i32 [[TMP1243]], [[TMP1244]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2777]], label [[IF_THEN2779:%.*]], label [[IF_END2780:%.*]]
+// SIMD-ONLY0:       if.then2779:
+// SIMD-ONLY0-NEXT:    [[TMP1245:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1245]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2780]]
+// SIMD-ONLY0:       if.end2780:
+// SIMD-ONLY0-NEXT:    [[TMP1246:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1247:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2781:%.*]] = icmp eq i32 [[TMP1246]], [[TMP1247]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2781]], label [[COND_TRUE2783:%.*]], label [[COND_FALSE2784:%.*]]
+// SIMD-ONLY0:       cond.true2783:
+// SIMD-ONLY0-NEXT:    [[TMP1248:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2785:%.*]]
+// SIMD-ONLY0:       cond.false2784:
+// SIMD-ONLY0-NEXT:    [[TMP1249:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2785]]
+// SIMD-ONLY0:       cond.end2785:
+// SIMD-ONLY0-NEXT:    [[COND2786:%.*]] = phi i32 [ [[TMP1248]], [[COND_TRUE2783]] ], [ [[TMP1249]], [[COND_FALSE2784]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2786]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1250:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1251:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2787:%.*]] = icmp eq i32 [[TMP1250]], [[TMP1251]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2787]], label [[COND_TRUE2789:%.*]], label [[COND_FALSE2790:%.*]]
+// SIMD-ONLY0:       cond.true2789:
+// SIMD-ONLY0-NEXT:    [[TMP1252:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2791:%.*]]
+// SIMD-ONLY0:       cond.false2790:
+// SIMD-ONLY0-NEXT:    [[TMP1253:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2791]]
+// SIMD-ONLY0:       cond.end2791:
+// SIMD-ONLY0-NEXT:    [[COND2792:%.*]] = phi i32 [ [[TMP1252]], [[COND_TRUE2789]] ], [ [[TMP1253]], [[COND_FALSE2790]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2792]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1254:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1255:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2793:%.*]] = icmp eq i32 [[TMP1254]], [[TMP1255]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2793]], label [[IF_THEN2795:%.*]], label [[IF_END2796:%.*]]
+// SIMD-ONLY0:       if.then2795:
+// SIMD-ONLY0-NEXT:    [[TMP1256:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1256]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2796]]
+// SIMD-ONLY0:       if.end2796:
+// SIMD-ONLY0-NEXT:    [[TMP1257:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1258:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2797:%.*]] = icmp eq i32 [[TMP1257]], [[TMP1258]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2797]], label [[IF_THEN2799:%.*]], label [[IF_END2800:%.*]]
+// SIMD-ONLY0:       if.then2799:
+// SIMD-ONLY0-NEXT:    [[TMP1259:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1259]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2800]]
+// SIMD-ONLY0:       if.end2800:
+// SIMD-ONLY0-NEXT:    [[TMP1260:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1261:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2801:%.*]] = icmp sgt i32 [[TMP1260]], [[TMP1261]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2801]], label [[COND_TRUE2803:%.*]], label [[COND_FALSE2804:%.*]]
+// SIMD-ONLY0:       cond.true2803:
+// SIMD-ONLY0-NEXT:    [[TMP1262:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2805:%.*]]
+// SIMD-ONLY0:       cond.false2804:
+// SIMD-ONLY0-NEXT:    [[TMP1263:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2805]]
+// SIMD-ONLY0:       cond.end2805:
+// SIMD-ONLY0-NEXT:    [[COND2806:%.*]] = phi i32 [ [[TMP1262]], [[COND_TRUE2803]] ], [ [[TMP1263]], [[COND_FALSE2804]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2806]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1264:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1265:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2807:%.*]] = icmp slt i32 [[TMP1264]], [[TMP1265]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2807]], label [[COND_TRUE2809:%.*]], label [[COND_FALSE2810:%.*]]
+// SIMD-ONLY0:       cond.true2809:
+// SIMD-ONLY0-NEXT:    [[TMP1266:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2811:%.*]]
+// SIMD-ONLY0:       cond.false2810:
+// SIMD-ONLY0-NEXT:    [[TMP1267:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2811]]
+// SIMD-ONLY0:       cond.end2811:
+// SIMD-ONLY0-NEXT:    [[COND2812:%.*]] = phi i32 [ [[TMP1266]], [[COND_TRUE2809]] ], [ [[TMP1267]], [[COND_FALSE2810]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2812]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1268:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1269:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2813:%.*]] = icmp sgt i32 [[TMP1268]], [[TMP1269]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2813]], label [[COND_TRUE2815:%.*]], label [[COND_FALSE2816:%.*]]
+// SIMD-ONLY0:       cond.true2815:
+// SIMD-ONLY0-NEXT:    [[TMP1270:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2817:%.*]]
+// SIMD-ONLY0:       cond.false2816:
+// SIMD-ONLY0-NEXT:    [[TMP1271:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2817]]
+// SIMD-ONLY0:       cond.end2817:
+// SIMD-ONLY0-NEXT:    [[COND2818:%.*]] = phi i32 [ [[TMP1270]], [[COND_TRUE2815]] ], [ [[TMP1271]], [[COND_FALSE2816]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2818]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1272:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1273:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2819:%.*]] = icmp slt i32 [[TMP1272]], [[TMP1273]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2819]], label [[COND_TRUE2821:%.*]], label [[COND_FALSE2822:%.*]]
+// SIMD-ONLY0:       cond.true2821:
+// SIMD-ONLY0-NEXT:    [[TMP1274:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2823:%.*]]
+// SIMD-ONLY0:       cond.false2822:
+// SIMD-ONLY0-NEXT:    [[TMP1275:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2823]]
+// SIMD-ONLY0:       cond.end2823:
+// SIMD-ONLY0-NEXT:    [[COND2824:%.*]] = phi i32 [ [[TMP1274]], [[COND_TRUE2821]] ], [ [[TMP1275]], [[COND_FALSE2822]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2824]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1276:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1277:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2825:%.*]] = icmp sgt i32 [[TMP1276]], [[TMP1277]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2825]], label [[IF_THEN2827:%.*]], label [[IF_END2828:%.*]]
+// SIMD-ONLY0:       if.then2827:
+// SIMD-ONLY0-NEXT:    [[TMP1278:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1278]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2828]]
+// SIMD-ONLY0:       if.end2828:
+// SIMD-ONLY0-NEXT:    [[TMP1279:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1280:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2829:%.*]] = icmp slt i32 [[TMP1279]], [[TMP1280]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2829]], label [[IF_THEN2831:%.*]], label [[IF_END2832:%.*]]
+// SIMD-ONLY0:       if.then2831:
+// SIMD-ONLY0-NEXT:    [[TMP1281:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1281]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2832]]
+// SIMD-ONLY0:       if.end2832:
+// SIMD-ONLY0-NEXT:    [[TMP1282:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1283:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2833:%.*]] = icmp sgt i32 [[TMP1282]], [[TMP1283]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2833]], label [[IF_THEN2835:%.*]], label [[IF_END2836:%.*]]
+// SIMD-ONLY0:       if.then2835:
+// SIMD-ONLY0-NEXT:    [[TMP1284:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1284]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2836]]
+// SIMD-ONLY0:       if.end2836:
+// SIMD-ONLY0-NEXT:    [[TMP1285:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1286:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2837:%.*]] = icmp slt i32 [[TMP1285]], [[TMP1286]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2837]], label [[IF_THEN2839:%.*]], label [[IF_END2840:%.*]]
+// SIMD-ONLY0:       if.then2839:
+// SIMD-ONLY0-NEXT:    [[TMP1287:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1287]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2840]]
+// SIMD-ONLY0:       if.end2840:
+// SIMD-ONLY0-NEXT:    [[TMP1288:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1289:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2841:%.*]] = icmp eq i32 [[TMP1288]], [[TMP1289]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2841]], label [[COND_TRUE2843:%.*]], label [[COND_FALSE2844:%.*]]
+// SIMD-ONLY0:       cond.true2843:
+// SIMD-ONLY0-NEXT:    [[TMP1290:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2845:%.*]]
+// SIMD-ONLY0:       cond.false2844:
+// SIMD-ONLY0-NEXT:    [[TMP1291:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2845]]
+// SIMD-ONLY0:       cond.end2845:
+// SIMD-ONLY0-NEXT:    [[COND2846:%.*]] = phi i32 [ [[TMP1290]], [[COND_TRUE2843]] ], [ [[TMP1291]], [[COND_FALSE2844]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2846]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1292:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1293:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2847:%.*]] = icmp eq i32 [[TMP1292]], [[TMP1293]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2847]], label [[COND_TRUE2849:%.*]], label [[COND_FALSE2850:%.*]]
+// SIMD-ONLY0:       cond.true2849:
+// SIMD-ONLY0-NEXT:    [[TMP1294:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2851:%.*]]
+// SIMD-ONLY0:       cond.false2850:
+// SIMD-ONLY0-NEXT:    [[TMP1295:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2851]]
+// SIMD-ONLY0:       cond.end2851:
+// SIMD-ONLY0-NEXT:    [[COND2852:%.*]] = phi i32 [ [[TMP1294]], [[COND_TRUE2849]] ], [ [[TMP1295]], [[COND_FALSE2850]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2852]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1296:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1297:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2853:%.*]] = icmp eq i32 [[TMP1296]], [[TMP1297]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2853]], label [[IF_THEN2855:%.*]], label [[IF_END2856:%.*]]
+// SIMD-ONLY0:       if.then2855:
+// SIMD-ONLY0-NEXT:    [[TMP1298:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1298]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2856]]
+// SIMD-ONLY0:       if.end2856:
+// SIMD-ONLY0-NEXT:    [[TMP1299:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1300:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2857:%.*]] = icmp eq i32 [[TMP1299]], [[TMP1300]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2857]], label [[IF_THEN2859:%.*]], label [[IF_END2860:%.*]]
+// SIMD-ONLY0:       if.then2859:
+// SIMD-ONLY0-NEXT:    [[TMP1301:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1301]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2860]]
+// SIMD-ONLY0:       if.end2860:
+// SIMD-ONLY0-NEXT:    [[TMP1302:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1303:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2861:%.*]] = icmp ugt i32 [[TMP1302]], [[TMP1303]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2861]], label [[COND_TRUE2863:%.*]], label [[COND_FALSE2864:%.*]]
+// SIMD-ONLY0:       cond.true2863:
+// SIMD-ONLY0-NEXT:    [[TMP1304:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2865:%.*]]
+// SIMD-ONLY0:       cond.false2864:
+// SIMD-ONLY0-NEXT:    [[TMP1305:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2865]]
+// SIMD-ONLY0:       cond.end2865:
+// SIMD-ONLY0-NEXT:    [[COND2866:%.*]] = phi i32 [ [[TMP1304]], [[COND_TRUE2863]] ], [ [[TMP1305]], [[COND_FALSE2864]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2866]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1306:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1307:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2867:%.*]] = icmp ult i32 [[TMP1306]], [[TMP1307]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2867]], label [[COND_TRUE2869:%.*]], label [[COND_FALSE2870:%.*]]
+// SIMD-ONLY0:       cond.true2869:
+// SIMD-ONLY0-NEXT:    [[TMP1308:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2871:%.*]]
+// SIMD-ONLY0:       cond.false2870:
+// SIMD-ONLY0-NEXT:    [[TMP1309:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2871]]
+// SIMD-ONLY0:       cond.end2871:
+// SIMD-ONLY0-NEXT:    [[COND2872:%.*]] = phi i32 [ [[TMP1308]], [[COND_TRUE2869]] ], [ [[TMP1309]], [[COND_FALSE2870]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2872]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1310:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1311:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2873:%.*]] = icmp ugt i32 [[TMP1310]], [[TMP1311]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2873]], label [[COND_TRUE2875:%.*]], label [[COND_FALSE2876:%.*]]
+// SIMD-ONLY0:       cond.true2875:
+// SIMD-ONLY0-NEXT:    [[TMP1312:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2877:%.*]]
+// SIMD-ONLY0:       cond.false2876:
+// SIMD-ONLY0-NEXT:    [[TMP1313:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2877]]
+// SIMD-ONLY0:       cond.end2877:
+// SIMD-ONLY0-NEXT:    [[COND2878:%.*]] = phi i32 [ [[TMP1312]], [[COND_TRUE2875]] ], [ [[TMP1313]], [[COND_FALSE2876]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2878]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1314:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1315:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2879:%.*]] = icmp ult i32 [[TMP1314]], [[TMP1315]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2879]], label [[COND_TRUE2881:%.*]], label [[COND_FALSE2882:%.*]]
+// SIMD-ONLY0:       cond.true2881:
+// SIMD-ONLY0-NEXT:    [[TMP1316:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2883:%.*]]
+// SIMD-ONLY0:       cond.false2882:
+// SIMD-ONLY0-NEXT:    [[TMP1317:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2883]]
+// SIMD-ONLY0:       cond.end2883:
+// SIMD-ONLY0-NEXT:    [[COND2884:%.*]] = phi i32 [ [[TMP1316]], [[COND_TRUE2881]] ], [ [[TMP1317]], [[COND_FALSE2882]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2884]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1318:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1319:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2885:%.*]] = icmp ugt i32 [[TMP1318]], [[TMP1319]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2885]], label [[IF_THEN2887:%.*]], label [[IF_END2888:%.*]]
+// SIMD-ONLY0:       if.then2887:
+// SIMD-ONLY0-NEXT:    [[TMP1320:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1320]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2888]]
+// SIMD-ONLY0:       if.end2888:
+// SIMD-ONLY0-NEXT:    [[TMP1321:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1322:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2889:%.*]] = icmp ult i32 [[TMP1321]], [[TMP1322]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2889]], label [[IF_THEN2891:%.*]], label [[IF_END2892:%.*]]
+// SIMD-ONLY0:       if.then2891:
+// SIMD-ONLY0-NEXT:    [[TMP1323:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1323]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2892]]
+// SIMD-ONLY0:       if.end2892:
+// SIMD-ONLY0-NEXT:    [[TMP1324:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1325:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2893:%.*]] = icmp ugt i32 [[TMP1324]], [[TMP1325]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2893]], label [[IF_THEN2895:%.*]], label [[IF_END2896:%.*]]
+// SIMD-ONLY0:       if.then2895:
+// SIMD-ONLY0-NEXT:    [[TMP1326:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1326]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2896]]
+// SIMD-ONLY0:       if.end2896:
+// SIMD-ONLY0-NEXT:    [[TMP1327:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1328:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2897:%.*]] = icmp ult i32 [[TMP1327]], [[TMP1328]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2897]], label [[IF_THEN2899:%.*]], label [[IF_END2900:%.*]]
+// SIMD-ONLY0:       if.then2899:
+// SIMD-ONLY0-NEXT:    [[TMP1329:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1329]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2900]]
+// SIMD-ONLY0:       if.end2900:
+// SIMD-ONLY0-NEXT:    [[TMP1330:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1331:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2901:%.*]] = icmp eq i32 [[TMP1330]], [[TMP1331]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2901]], label [[COND_TRUE2903:%.*]], label [[COND_FALSE2904:%.*]]
+// SIMD-ONLY0:       cond.true2903:
+// SIMD-ONLY0-NEXT:    [[TMP1332:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2905:%.*]]
+// SIMD-ONLY0:       cond.false2904:
+// SIMD-ONLY0-NEXT:    [[TMP1333:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2905]]
+// SIMD-ONLY0:       cond.end2905:
+// SIMD-ONLY0-NEXT:    [[COND2906:%.*]] = phi i32 [ [[TMP1332]], [[COND_TRUE2903]] ], [ [[TMP1333]], [[COND_FALSE2904]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2906]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1334:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1335:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2907:%.*]] = icmp eq i32 [[TMP1334]], [[TMP1335]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2907]], label [[COND_TRUE2909:%.*]], label [[COND_FALSE2910:%.*]]
+// SIMD-ONLY0:       cond.true2909:
+// SIMD-ONLY0-NEXT:    [[TMP1336:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2911:%.*]]
+// SIMD-ONLY0:       cond.false2910:
+// SIMD-ONLY0-NEXT:    [[TMP1337:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2911]]
+// SIMD-ONLY0:       cond.end2911:
+// SIMD-ONLY0-NEXT:    [[COND2912:%.*]] = phi i32 [ [[TMP1336]], [[COND_TRUE2909]] ], [ [[TMP1337]], [[COND_FALSE2910]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2912]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1338:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1339:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2913:%.*]] = icmp eq i32 [[TMP1338]], [[TMP1339]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2913]], label [[IF_THEN2915:%.*]], label [[IF_END2916:%.*]]
+// SIMD-ONLY0:       if.then2915:
+// SIMD-ONLY0-NEXT:    [[TMP1340:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1340]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2916]]
+// SIMD-ONLY0:       if.end2916:
+// SIMD-ONLY0-NEXT:    [[TMP1341:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1342:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2917:%.*]] = icmp eq i32 [[TMP1341]], [[TMP1342]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2917]], label [[IF_THEN2919:%.*]], label [[IF_END2920:%.*]]
+// SIMD-ONLY0:       if.then2919:
+// SIMD-ONLY0-NEXT:    [[TMP1343:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1343]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2920]]
+// SIMD-ONLY0:       if.end2920:
+// SIMD-ONLY0-NEXT:    [[TMP1344:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1345:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2921:%.*]] = icmp sgt i32 [[TMP1344]], [[TMP1345]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2921]], label [[COND_TRUE2923:%.*]], label [[COND_FALSE2924:%.*]]
+// SIMD-ONLY0:       cond.true2923:
+// SIMD-ONLY0-NEXT:    [[TMP1346:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2925:%.*]]
+// SIMD-ONLY0:       cond.false2924:
+// SIMD-ONLY0-NEXT:    [[TMP1347:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2925]]
+// SIMD-ONLY0:       cond.end2925:
+// SIMD-ONLY0-NEXT:    [[COND2926:%.*]] = phi i32 [ [[TMP1346]], [[COND_TRUE2923]] ], [ [[TMP1347]], [[COND_FALSE2924]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2926]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1348:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1349:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2927:%.*]] = icmp slt i32 [[TMP1348]], [[TMP1349]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2927]], label [[COND_TRUE2929:%.*]], label [[COND_FALSE2930:%.*]]
+// SIMD-ONLY0:       cond.true2929:
+// SIMD-ONLY0-NEXT:    [[TMP1350:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2931:%.*]]
+// SIMD-ONLY0:       cond.false2930:
+// SIMD-ONLY0-NEXT:    [[TMP1351:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2931]]
+// SIMD-ONLY0:       cond.end2931:
+// SIMD-ONLY0-NEXT:    [[COND2932:%.*]] = phi i32 [ [[TMP1350]], [[COND_TRUE2929]] ], [ [[TMP1351]], [[COND_FALSE2930]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2932]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1352:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1353:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2933:%.*]] = icmp sgt i32 [[TMP1352]], [[TMP1353]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2933]], label [[COND_TRUE2935:%.*]], label [[COND_FALSE2936:%.*]]
+// SIMD-ONLY0:       cond.true2935:
+// SIMD-ONLY0-NEXT:    [[TMP1354:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2937:%.*]]
+// SIMD-ONLY0:       cond.false2936:
+// SIMD-ONLY0-NEXT:    [[TMP1355:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2937]]
+// SIMD-ONLY0:       cond.end2937:
+// SIMD-ONLY0-NEXT:    [[COND2938:%.*]] = phi i32 [ [[TMP1354]], [[COND_TRUE2935]] ], [ [[TMP1355]], [[COND_FALSE2936]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2938]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1356:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1357:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2939:%.*]] = icmp slt i32 [[TMP1356]], [[TMP1357]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2939]], label [[COND_TRUE2941:%.*]], label [[COND_FALSE2942:%.*]]
+// SIMD-ONLY0:       cond.true2941:
+// SIMD-ONLY0-NEXT:    [[TMP1358:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2943:%.*]]
+// SIMD-ONLY0:       cond.false2942:
+// SIMD-ONLY0-NEXT:    [[TMP1359:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2943]]
+// SIMD-ONLY0:       cond.end2943:
+// SIMD-ONLY0-NEXT:    [[COND2944:%.*]] = phi i32 [ [[TMP1358]], [[COND_TRUE2941]] ], [ [[TMP1359]], [[COND_FALSE2942]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2944]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1360:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1361:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2945:%.*]] = icmp sgt i32 [[TMP1360]], [[TMP1361]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2945]], label [[IF_THEN2947:%.*]], label [[IF_END2948:%.*]]
+// SIMD-ONLY0:       if.then2947:
+// SIMD-ONLY0-NEXT:    [[TMP1362:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1362]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2948]]
+// SIMD-ONLY0:       if.end2948:
+// SIMD-ONLY0-NEXT:    [[TMP1363:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1364:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2949:%.*]] = icmp slt i32 [[TMP1363]], [[TMP1364]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2949]], label [[IF_THEN2951:%.*]], label [[IF_END2952:%.*]]
+// SIMD-ONLY0:       if.then2951:
+// SIMD-ONLY0-NEXT:    [[TMP1365:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1365]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2952]]
+// SIMD-ONLY0:       if.end2952:
+// SIMD-ONLY0-NEXT:    [[TMP1366:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1367:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2953:%.*]] = icmp sgt i32 [[TMP1366]], [[TMP1367]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2953]], label [[IF_THEN2955:%.*]], label [[IF_END2956:%.*]]
+// SIMD-ONLY0:       if.then2955:
+// SIMD-ONLY0-NEXT:    [[TMP1368:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1368]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2956]]
+// SIMD-ONLY0:       if.end2956:
+// SIMD-ONLY0-NEXT:    [[TMP1369:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1370:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2957:%.*]] = icmp slt i32 [[TMP1369]], [[TMP1370]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2957]], label [[IF_THEN2959:%.*]], label [[IF_END2960:%.*]]
+// SIMD-ONLY0:       if.then2959:
+// SIMD-ONLY0-NEXT:    [[TMP1371:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1371]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2960]]
+// SIMD-ONLY0:       if.end2960:
+// SIMD-ONLY0-NEXT:    [[TMP1372:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1373:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2961:%.*]] = icmp eq i32 [[TMP1372]], [[TMP1373]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2961]], label [[COND_TRUE2963:%.*]], label [[COND_FALSE2964:%.*]]
+// SIMD-ONLY0:       cond.true2963:
+// SIMD-ONLY0-NEXT:    [[TMP1374:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2965:%.*]]
+// SIMD-ONLY0:       cond.false2964:
+// SIMD-ONLY0-NEXT:    [[TMP1375:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2965]]
+// SIMD-ONLY0:       cond.end2965:
+// SIMD-ONLY0-NEXT:    [[COND2966:%.*]] = phi i32 [ [[TMP1374]], [[COND_TRUE2963]] ], [ [[TMP1375]], [[COND_FALSE2964]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2966]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1376:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1377:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2967:%.*]] = icmp eq i32 [[TMP1376]], [[TMP1377]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2967]], label [[COND_TRUE2969:%.*]], label [[COND_FALSE2970:%.*]]
+// SIMD-ONLY0:       cond.true2969:
+// SIMD-ONLY0-NEXT:    [[TMP1378:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2971:%.*]]
+// SIMD-ONLY0:       cond.false2970:
+// SIMD-ONLY0-NEXT:    [[TMP1379:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2971]]
+// SIMD-ONLY0:       cond.end2971:
+// SIMD-ONLY0-NEXT:    [[COND2972:%.*]] = phi i32 [ [[TMP1378]], [[COND_TRUE2969]] ], [ [[TMP1379]], [[COND_FALSE2970]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2972]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1380:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1381:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2973:%.*]] = icmp eq i32 [[TMP1380]], [[TMP1381]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2973]], label [[IF_THEN2975:%.*]], label [[IF_END2976:%.*]]
+// SIMD-ONLY0:       if.then2975:
+// SIMD-ONLY0-NEXT:    [[TMP1382:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1382]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2976]]
+// SIMD-ONLY0:       if.end2976:
+// SIMD-ONLY0-NEXT:    [[TMP1383:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1384:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2977:%.*]] = icmp eq i32 [[TMP1383]], [[TMP1384]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2977]], label [[IF_THEN2979:%.*]], label [[IF_END2980:%.*]]
+// SIMD-ONLY0:       if.then2979:
+// SIMD-ONLY0-NEXT:    [[TMP1385:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1385]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2980]]
+// SIMD-ONLY0:       if.end2980:
+// SIMD-ONLY0-NEXT:    [[TMP1386:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1387:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2981:%.*]] = icmp ugt i32 [[TMP1386]], [[TMP1387]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2981]], label [[COND_TRUE2983:%.*]], label [[COND_FALSE2984:%.*]]
+// SIMD-ONLY0:       cond.true2983:
+// SIMD-ONLY0-NEXT:    [[TMP1388:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2985:%.*]]
+// SIMD-ONLY0:       cond.false2984:
+// SIMD-ONLY0-NEXT:    [[TMP1389:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2985]]
+// SIMD-ONLY0:       cond.end2985:
+// SIMD-ONLY0-NEXT:    [[COND2986:%.*]] = phi i32 [ [[TMP1388]], [[COND_TRUE2983]] ], [ [[TMP1389]], [[COND_FALSE2984]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2986]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1390:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1391:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2987:%.*]] = icmp ult i32 [[TMP1390]], [[TMP1391]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2987]], label [[COND_TRUE2989:%.*]], label [[COND_FALSE2990:%.*]]
+// SIMD-ONLY0:       cond.true2989:
+// SIMD-ONLY0-NEXT:    [[TMP1392:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2991:%.*]]
+// SIMD-ONLY0:       cond.false2990:
+// SIMD-ONLY0-NEXT:    [[TMP1393:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2991]]
+// SIMD-ONLY0:       cond.end2991:
+// SIMD-ONLY0-NEXT:    [[COND2992:%.*]] = phi i32 [ [[TMP1392]], [[COND_TRUE2989]] ], [ [[TMP1393]], [[COND_FALSE2990]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2992]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1394:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1395:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2993:%.*]] = icmp ugt i32 [[TMP1394]], [[TMP1395]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2993]], label [[COND_TRUE2995:%.*]], label [[COND_FALSE2996:%.*]]
+// SIMD-ONLY0:       cond.true2995:
+// SIMD-ONLY0-NEXT:    [[TMP1396:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2997:%.*]]
+// SIMD-ONLY0:       cond.false2996:
+// SIMD-ONLY0-NEXT:    [[TMP1397:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END2997]]
+// SIMD-ONLY0:       cond.end2997:
+// SIMD-ONLY0-NEXT:    [[COND2998:%.*]] = phi i32 [ [[TMP1396]], [[COND_TRUE2995]] ], [ [[TMP1397]], [[COND_FALSE2996]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND2998]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1398:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1399:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2999:%.*]] = icmp ult i32 [[TMP1398]], [[TMP1399]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2999]], label [[COND_TRUE3001:%.*]], label [[COND_FALSE3002:%.*]]
+// SIMD-ONLY0:       cond.true3001:
+// SIMD-ONLY0-NEXT:    [[TMP1400:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3003:%.*]]
+// SIMD-ONLY0:       cond.false3002:
+// SIMD-ONLY0-NEXT:    [[TMP1401:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3003]]
+// SIMD-ONLY0:       cond.end3003:
+// SIMD-ONLY0-NEXT:    [[COND3004:%.*]] = phi i32 [ [[TMP1400]], [[COND_TRUE3001]] ], [ [[TMP1401]], [[COND_FALSE3002]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND3004]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1402:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1403:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3005:%.*]] = icmp ugt i32 [[TMP1402]], [[TMP1403]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3005]], label [[IF_THEN3007:%.*]], label [[IF_END3008:%.*]]
+// SIMD-ONLY0:       if.then3007:
+// SIMD-ONLY0-NEXT:    [[TMP1404:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1404]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3008]]
+// SIMD-ONLY0:       if.end3008:
+// SIMD-ONLY0-NEXT:    [[TMP1405:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1406:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3009:%.*]] = icmp ult i32 [[TMP1405]], [[TMP1406]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3009]], label [[IF_THEN3011:%.*]], label [[IF_END3012:%.*]]
+// SIMD-ONLY0:       if.then3011:
+// SIMD-ONLY0-NEXT:    [[TMP1407:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1407]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3012]]
+// SIMD-ONLY0:       if.end3012:
+// SIMD-ONLY0-NEXT:    [[TMP1408:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1409:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3013:%.*]] = icmp ugt i32 [[TMP1408]], [[TMP1409]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3013]], label [[IF_THEN3015:%.*]], label [[IF_END3016:%.*]]
+// SIMD-ONLY0:       if.then3015:
+// SIMD-ONLY0-NEXT:    [[TMP1410:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1410]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3016]]
+// SIMD-ONLY0:       if.end3016:
+// SIMD-ONLY0-NEXT:    [[TMP1411:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1412:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3017:%.*]] = icmp ult i32 [[TMP1411]], [[TMP1412]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3017]], label [[IF_THEN3019:%.*]], label [[IF_END3020:%.*]]
+// SIMD-ONLY0:       if.then3019:
+// SIMD-ONLY0-NEXT:    [[TMP1413:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1413]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3020]]
+// SIMD-ONLY0:       if.end3020:
+// SIMD-ONLY0-NEXT:    [[TMP1414:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1415:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3021:%.*]] = icmp eq i32 [[TMP1414]], [[TMP1415]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3021]], label [[COND_TRUE3023:%.*]], label [[COND_FALSE3024:%.*]]
+// SIMD-ONLY0:       cond.true3023:
+// SIMD-ONLY0-NEXT:    [[TMP1416:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3025:%.*]]
+// SIMD-ONLY0:       cond.false3024:
+// SIMD-ONLY0-NEXT:    [[TMP1417:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3025]]
+// SIMD-ONLY0:       cond.end3025:
+// SIMD-ONLY0-NEXT:    [[COND3026:%.*]] = phi i32 [ [[TMP1416]], [[COND_TRUE3023]] ], [ [[TMP1417]], [[COND_FALSE3024]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND3026]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1418:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1419:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3027:%.*]] = icmp eq i32 [[TMP1418]], [[TMP1419]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3027]], label [[COND_TRUE3029:%.*]], label [[COND_FALSE3030:%.*]]
+// SIMD-ONLY0:       cond.true3029:
+// SIMD-ONLY0-NEXT:    [[TMP1420:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3031:%.*]]
+// SIMD-ONLY0:       cond.false3030:
+// SIMD-ONLY0-NEXT:    [[TMP1421:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3031]]
+// SIMD-ONLY0:       cond.end3031:
+// SIMD-ONLY0-NEXT:    [[COND3032:%.*]] = phi i32 [ [[TMP1420]], [[COND_TRUE3029]] ], [ [[TMP1421]], [[COND_FALSE3030]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND3032]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1422:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1423:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3033:%.*]] = icmp eq i32 [[TMP1422]], [[TMP1423]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3033]], label [[IF_THEN3035:%.*]], label [[IF_END3036:%.*]]
+// SIMD-ONLY0:       if.then3035:
+// SIMD-ONLY0-NEXT:    [[TMP1424:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1424]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3036]]
+// SIMD-ONLY0:       if.end3036:
+// SIMD-ONLY0-NEXT:    [[TMP1425:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1426:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3037:%.*]] = icmp eq i32 [[TMP1425]], [[TMP1426]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3037]], label [[IF_THEN3039:%.*]], label [[IF_END3040:%.*]]
+// SIMD-ONLY0:       if.then3039:
+// SIMD-ONLY0-NEXT:    [[TMP1427:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1427]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3040]]
+// SIMD-ONLY0:       if.end3040:
+// SIMD-ONLY0-NEXT:    [[TMP1428:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1429:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3041:%.*]] = icmp sgt i32 [[TMP1428]], [[TMP1429]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3041]], label [[COND_TRUE3043:%.*]], label [[COND_FALSE3044:%.*]]
+// SIMD-ONLY0:       cond.true3043:
+// SIMD-ONLY0-NEXT:    [[TMP1430:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3045:%.*]]
+// SIMD-ONLY0:       cond.false3044:
+// SIMD-ONLY0-NEXT:    [[TMP1431:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3045]]
+// SIMD-ONLY0:       cond.end3045:
+// SIMD-ONLY0-NEXT:    [[COND3046:%.*]] = phi i32 [ [[TMP1430]], [[COND_TRUE3043]] ], [ [[TMP1431]], [[COND_FALSE3044]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND3046]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1432:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1433:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3047:%.*]] = icmp slt i32 [[TMP1432]], [[TMP1433]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3047]], label [[COND_TRUE3049:%.*]], label [[COND_FALSE3050:%.*]]
+// SIMD-ONLY0:       cond.true3049:
+// SIMD-ONLY0-NEXT:    [[TMP1434:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3051:%.*]]
+// SIMD-ONLY0:       cond.false3050:
+// SIMD-ONLY0-NEXT:    [[TMP1435:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3051]]
+// SIMD-ONLY0:       cond.end3051:
+// SIMD-ONLY0-NEXT:    [[COND3052:%.*]] = phi i32 [ [[TMP1434]], [[COND_TRUE3049]] ], [ [[TMP1435]], [[COND_FALSE3050]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND3052]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1436:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1437:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3053:%.*]] = icmp sgt i32 [[TMP1436]], [[TMP1437]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3053]], label [[COND_TRUE3055:%.*]], label [[COND_FALSE3056:%.*]]
+// SIMD-ONLY0:       cond.true3055:
+// SIMD-ONLY0-NEXT:    [[TMP1438:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3057:%.*]]
+// SIMD-ONLY0:       cond.false3056:
+// SIMD-ONLY0-NEXT:    [[TMP1439:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3057]]
+// SIMD-ONLY0:       cond.end3057:
+// SIMD-ONLY0-NEXT:    [[COND3058:%.*]] = phi i32 [ [[TMP1438]], [[COND_TRUE3055]] ], [ [[TMP1439]], [[COND_FALSE3056]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND3058]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1440:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1441:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3059:%.*]] = icmp slt i32 [[TMP1440]], [[TMP1441]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3059]], label [[COND_TRUE3061:%.*]], label [[COND_FALSE3062:%.*]]
+// SIMD-ONLY0:       cond.true3061:
+// SIMD-ONLY0-NEXT:    [[TMP1442:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3063:%.*]]
+// SIMD-ONLY0:       cond.false3062:
+// SIMD-ONLY0-NEXT:    [[TMP1443:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3063]]
+// SIMD-ONLY0:       cond.end3063:
+// SIMD-ONLY0-NEXT:    [[COND3064:%.*]] = phi i32 [ [[TMP1442]], [[COND_TRUE3061]] ], [ [[TMP1443]], [[COND_FALSE3062]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND3064]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1444:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1445:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3065:%.*]] = icmp sgt i32 [[TMP1444]], [[TMP1445]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3065]], label [[IF_THEN3067:%.*]], label [[IF_END3068:%.*]]
+// SIMD-ONLY0:       if.then3067:
+// SIMD-ONLY0-NEXT:    [[TMP1446:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1446]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3068]]
+// SIMD-ONLY0:       if.end3068:
+// SIMD-ONLY0-NEXT:    [[TMP1447:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1448:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3069:%.*]] = icmp slt i32 [[TMP1447]], [[TMP1448]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3069]], label [[IF_THEN3071:%.*]], label [[IF_END3072:%.*]]
+// SIMD-ONLY0:       if.then3071:
+// SIMD-ONLY0-NEXT:    [[TMP1449:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1449]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3072]]
+// SIMD-ONLY0:       if.end3072:
+// SIMD-ONLY0-NEXT:    [[TMP1450:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1451:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3073:%.*]] = icmp sgt i32 [[TMP1450]], [[TMP1451]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3073]], label [[IF_THEN3075:%.*]], label [[IF_END3076:%.*]]
+// SIMD-ONLY0:       if.then3075:
+// SIMD-ONLY0-NEXT:    [[TMP1452:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1452]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3076]]
+// SIMD-ONLY0:       if.end3076:
+// SIMD-ONLY0-NEXT:    [[TMP1453:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1454:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3077:%.*]] = icmp slt i32 [[TMP1453]], [[TMP1454]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3077]], label [[IF_THEN3079:%.*]], label [[IF_END3080:%.*]]
+// SIMD-ONLY0:       if.then3079:
+// SIMD-ONLY0-NEXT:    [[TMP1455:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1455]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3080]]
+// SIMD-ONLY0:       if.end3080:
+// SIMD-ONLY0-NEXT:    [[TMP1456:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1457:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3081:%.*]] = icmp eq i32 [[TMP1456]], [[TMP1457]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3081]], label [[COND_TRUE3083:%.*]], label [[COND_FALSE3084:%.*]]
+// SIMD-ONLY0:       cond.true3083:
+// SIMD-ONLY0-NEXT:    [[TMP1458:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3085:%.*]]
+// SIMD-ONLY0:       cond.false3084:
+// SIMD-ONLY0-NEXT:    [[TMP1459:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3085]]
+// SIMD-ONLY0:       cond.end3085:
+// SIMD-ONLY0-NEXT:    [[COND3086:%.*]] = phi i32 [ [[TMP1458]], [[COND_TRUE3083]] ], [ [[TMP1459]], [[COND_FALSE3084]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND3086]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1460:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1461:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3087:%.*]] = icmp eq i32 [[TMP1460]], [[TMP1461]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3087]], label [[COND_TRUE3089:%.*]], label [[COND_FALSE3090:%.*]]
+// SIMD-ONLY0:       cond.true3089:
+// SIMD-ONLY0-NEXT:    [[TMP1462:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3091:%.*]]
+// SIMD-ONLY0:       cond.false3090:
+// SIMD-ONLY0-NEXT:    [[TMP1463:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3091]]
+// SIMD-ONLY0:       cond.end3091:
+// SIMD-ONLY0-NEXT:    [[COND3092:%.*]] = phi i32 [ [[TMP1462]], [[COND_TRUE3089]] ], [ [[TMP1463]], [[COND_FALSE3090]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND3092]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1464:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1465:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3093:%.*]] = icmp eq i32 [[TMP1464]], [[TMP1465]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3093]], label [[IF_THEN3095:%.*]], label [[IF_END3096:%.*]]
+// SIMD-ONLY0:       if.then3095:
+// SIMD-ONLY0-NEXT:    [[TMP1466:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1466]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3096]]
+// SIMD-ONLY0:       if.end3096:
+// SIMD-ONLY0-NEXT:    [[TMP1467:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1468:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3097:%.*]] = icmp eq i32 [[TMP1467]], [[TMP1468]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3097]], label [[IF_THEN3099:%.*]], label [[IF_END3100:%.*]]
+// SIMD-ONLY0:       if.then3099:
+// SIMD-ONLY0-NEXT:    [[TMP1469:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1469]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3100]]
+// SIMD-ONLY0:       if.end3100:
+// SIMD-ONLY0-NEXT:    [[TMP1470:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1471:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3101:%.*]] = icmp ugt i32 [[TMP1470]], [[TMP1471]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3101]], label [[COND_TRUE3103:%.*]], label [[COND_FALSE3104:%.*]]
+// SIMD-ONLY0:       cond.true3103:
+// SIMD-ONLY0-NEXT:    [[TMP1472:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3105:%.*]]
+// SIMD-ONLY0:       cond.false3104:
+// SIMD-ONLY0-NEXT:    [[TMP1473:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3105]]
+// SIMD-ONLY0:       cond.end3105:
+// SIMD-ONLY0-NEXT:    [[COND3106:%.*]] = phi i32 [ [[TMP1472]], [[COND_TRUE3103]] ], [ [[TMP1473]], [[COND_FALSE3104]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND3106]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1474:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1475:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3107:%.*]] = icmp ult i32 [[TMP1474]], [[TMP1475]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3107]], label [[COND_TRUE3109:%.*]], label [[COND_FALSE3110:%.*]]
+// SIMD-ONLY0:       cond.true3109:
+// SIMD-ONLY0-NEXT:    [[TMP1476:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3111:%.*]]
+// SIMD-ONLY0:       cond.false3110:
+// SIMD-ONLY0-NEXT:    [[TMP1477:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3111]]
+// SIMD-ONLY0:       cond.end3111:
+// SIMD-ONLY0-NEXT:    [[COND3112:%.*]] = phi i32 [ [[TMP1476]], [[COND_TRUE3109]] ], [ [[TMP1477]], [[COND_FALSE3110]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND3112]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1478:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1479:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3113:%.*]] = icmp ugt i32 [[TMP1478]], [[TMP1479]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3113]], label [[COND_TRUE3115:%.*]], label [[COND_FALSE3116:%.*]]
+// SIMD-ONLY0:       cond.true3115:
+// SIMD-ONLY0-NEXT:    [[TMP1480:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3117:%.*]]
+// SIMD-ONLY0:       cond.false3116:
+// SIMD-ONLY0-NEXT:    [[TMP1481:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3117]]
+// SIMD-ONLY0:       cond.end3117:
+// SIMD-ONLY0-NEXT:    [[COND3118:%.*]] = phi i32 [ [[TMP1480]], [[COND_TRUE3115]] ], [ [[TMP1481]], [[COND_FALSE3116]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND3118]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1482:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1483:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3119:%.*]] = icmp ult i32 [[TMP1482]], [[TMP1483]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3119]], label [[COND_TRUE3121:%.*]], label [[COND_FALSE3122:%.*]]
+// SIMD-ONLY0:       cond.true3121:
+// SIMD-ONLY0-NEXT:    [[TMP1484:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3123:%.*]]
+// SIMD-ONLY0:       cond.false3122:
+// SIMD-ONLY0-NEXT:    [[TMP1485:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3123]]
+// SIMD-ONLY0:       cond.end3123:
+// SIMD-ONLY0-NEXT:    [[COND3124:%.*]] = phi i32 [ [[TMP1484]], [[COND_TRUE3121]] ], [ [[TMP1485]], [[COND_FALSE3122]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND3124]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1486:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1487:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3125:%.*]] = icmp ugt i32 [[TMP1486]], [[TMP1487]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3125]], label [[IF_THEN3127:%.*]], label [[IF_END3128:%.*]]
+// SIMD-ONLY0:       if.then3127:
+// SIMD-ONLY0-NEXT:    [[TMP1488:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1488]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3128]]
+// SIMD-ONLY0:       if.end3128:
+// SIMD-ONLY0-NEXT:    [[TMP1489:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1490:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3129:%.*]] = icmp ult i32 [[TMP1489]], [[TMP1490]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3129]], label [[IF_THEN3131:%.*]], label [[IF_END3132:%.*]]
+// SIMD-ONLY0:       if.then3131:
+// SIMD-ONLY0-NEXT:    [[TMP1491:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1491]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3132]]
+// SIMD-ONLY0:       if.end3132:
+// SIMD-ONLY0-NEXT:    [[TMP1492:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1493:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3133:%.*]] = icmp ugt i32 [[TMP1492]], [[TMP1493]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3133]], label [[IF_THEN3135:%.*]], label [[IF_END3136:%.*]]
+// SIMD-ONLY0:       if.then3135:
+// SIMD-ONLY0-NEXT:    [[TMP1494:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1494]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3136]]
+// SIMD-ONLY0:       if.end3136:
+// SIMD-ONLY0-NEXT:    [[TMP1495:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1496:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3137:%.*]] = icmp ult i32 [[TMP1495]], [[TMP1496]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3137]], label [[IF_THEN3139:%.*]], label [[IF_END3140:%.*]]
+// SIMD-ONLY0:       if.then3139:
+// SIMD-ONLY0-NEXT:    [[TMP1497:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1497]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3140]]
+// SIMD-ONLY0:       if.end3140:
+// SIMD-ONLY0-NEXT:    [[TMP1498:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1499:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3141:%.*]] = icmp eq i32 [[TMP1498]], [[TMP1499]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3141]], label [[COND_TRUE3143:%.*]], label [[COND_FALSE3144:%.*]]
+// SIMD-ONLY0:       cond.true3143:
+// SIMD-ONLY0-NEXT:    [[TMP1500:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3145:%.*]]
+// SIMD-ONLY0:       cond.false3144:
+// SIMD-ONLY0-NEXT:    [[TMP1501:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3145]]
+// SIMD-ONLY0:       cond.end3145:
+// SIMD-ONLY0-NEXT:    [[COND3146:%.*]] = phi i32 [ [[TMP1500]], [[COND_TRUE3143]] ], [ [[TMP1501]], [[COND_FALSE3144]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND3146]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1502:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1503:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3147:%.*]] = icmp eq i32 [[TMP1502]], [[TMP1503]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3147]], label [[COND_TRUE3149:%.*]], label [[COND_FALSE3150:%.*]]
+// SIMD-ONLY0:       cond.true3149:
+// SIMD-ONLY0-NEXT:    [[TMP1504:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3151:%.*]]
+// SIMD-ONLY0:       cond.false3150:
+// SIMD-ONLY0-NEXT:    [[TMP1505:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END3151]]
+// SIMD-ONLY0:       cond.end3151:
+// SIMD-ONLY0-NEXT:    [[COND3152:%.*]] = phi i32 [ [[TMP1504]], [[COND_TRUE3149]] ], [ [[TMP1505]], [[COND_FALSE3150]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND3152]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1506:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1507:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3153:%.*]] = icmp eq i32 [[TMP1506]], [[TMP1507]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3153]], label [[IF_THEN3155:%.*]], label [[IF_END3156:%.*]]
+// SIMD-ONLY0:       if.then3155:
+// SIMD-ONLY0-NEXT:    [[TMP1508:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1508]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3156]]
+// SIMD-ONLY0:       if.end3156:
+// SIMD-ONLY0-NEXT:    [[TMP1509:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1510:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3157:%.*]] = icmp eq i32 [[TMP1509]], [[TMP1510]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3157]], label [[IF_THEN3159:%.*]], label [[IF_END3160:%.*]]
+// SIMD-ONLY0:       if.then3159:
+// SIMD-ONLY0-NEXT:    [[TMP1511:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1511]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3160]]
+// SIMD-ONLY0:       if.end3160:
+// SIMD-ONLY0-NEXT:    [[TMP1512:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1513:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3161:%.*]] = icmp sgt i64 [[TMP1512]], [[TMP1513]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3161]], label [[COND_TRUE3163:%.*]], label [[COND_FALSE3164:%.*]]
+// SIMD-ONLY0:       cond.true3163:
+// SIMD-ONLY0-NEXT:    [[TMP1514:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3165:%.*]]
+// SIMD-ONLY0:       cond.false3164:
+// SIMD-ONLY0-NEXT:    [[TMP1515:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3165]]
+// SIMD-ONLY0:       cond.end3165:
+// SIMD-ONLY0-NEXT:    [[COND3166:%.*]] = phi i64 [ [[TMP1514]], [[COND_TRUE3163]] ], [ [[TMP1515]], [[COND_FALSE3164]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3166]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1516:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1517:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3167:%.*]] = icmp slt i64 [[TMP1516]], [[TMP1517]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3167]], label [[COND_TRUE3169:%.*]], label [[COND_FALSE3170:%.*]]
+// SIMD-ONLY0:       cond.true3169:
+// SIMD-ONLY0-NEXT:    [[TMP1518:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3171:%.*]]
+// SIMD-ONLY0:       cond.false3170:
+// SIMD-ONLY0-NEXT:    [[TMP1519:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3171]]
+// SIMD-ONLY0:       cond.end3171:
+// SIMD-ONLY0-NEXT:    [[COND3172:%.*]] = phi i64 [ [[TMP1518]], [[COND_TRUE3169]] ], [ [[TMP1519]], [[COND_FALSE3170]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3172]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1520:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1521:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3173:%.*]] = icmp sgt i64 [[TMP1520]], [[TMP1521]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3173]], label [[COND_TRUE3175:%.*]], label [[COND_FALSE3176:%.*]]
+// SIMD-ONLY0:       cond.true3175:
+// SIMD-ONLY0-NEXT:    [[TMP1522:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3177:%.*]]
+// SIMD-ONLY0:       cond.false3176:
+// SIMD-ONLY0-NEXT:    [[TMP1523:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3177]]
+// SIMD-ONLY0:       cond.end3177:
+// SIMD-ONLY0-NEXT:    [[COND3178:%.*]] = phi i64 [ [[TMP1522]], [[COND_TRUE3175]] ], [ [[TMP1523]], [[COND_FALSE3176]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3178]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1524:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1525:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3179:%.*]] = icmp slt i64 [[TMP1524]], [[TMP1525]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3179]], label [[COND_TRUE3181:%.*]], label [[COND_FALSE3182:%.*]]
+// SIMD-ONLY0:       cond.true3181:
+// SIMD-ONLY0-NEXT:    [[TMP1526:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3183:%.*]]
+// SIMD-ONLY0:       cond.false3182:
+// SIMD-ONLY0-NEXT:    [[TMP1527:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3183]]
+// SIMD-ONLY0:       cond.end3183:
+// SIMD-ONLY0-NEXT:    [[COND3184:%.*]] = phi i64 [ [[TMP1526]], [[COND_TRUE3181]] ], [ [[TMP1527]], [[COND_FALSE3182]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3184]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1528:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1529:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3185:%.*]] = icmp sgt i64 [[TMP1528]], [[TMP1529]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3185]], label [[IF_THEN3187:%.*]], label [[IF_END3188:%.*]]
+// SIMD-ONLY0:       if.then3187:
+// SIMD-ONLY0-NEXT:    [[TMP1530:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1530]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3188]]
+// SIMD-ONLY0:       if.end3188:
+// SIMD-ONLY0-NEXT:    [[TMP1531:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1532:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3189:%.*]] = icmp slt i64 [[TMP1531]], [[TMP1532]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3189]], label [[IF_THEN3191:%.*]], label [[IF_END3192:%.*]]
+// SIMD-ONLY0:       if.then3191:
+// SIMD-ONLY0-NEXT:    [[TMP1533:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1533]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3192]]
+// SIMD-ONLY0:       if.end3192:
+// SIMD-ONLY0-NEXT:    [[TMP1534:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1535:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3193:%.*]] = icmp sgt i64 [[TMP1534]], [[TMP1535]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3193]], label [[IF_THEN3195:%.*]], label [[IF_END3196:%.*]]
+// SIMD-ONLY0:       if.then3195:
+// SIMD-ONLY0-NEXT:    [[TMP1536:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1536]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3196]]
+// SIMD-ONLY0:       if.end3196:
+// SIMD-ONLY0-NEXT:    [[TMP1537:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1538:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3197:%.*]] = icmp slt i64 [[TMP1537]], [[TMP1538]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3197]], label [[IF_THEN3199:%.*]], label [[IF_END3200:%.*]]
+// SIMD-ONLY0:       if.then3199:
+// SIMD-ONLY0-NEXT:    [[TMP1539:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1539]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3200]]
+// SIMD-ONLY0:       if.end3200:
+// SIMD-ONLY0-NEXT:    [[TMP1540:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1541:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3201:%.*]] = icmp eq i64 [[TMP1540]], [[TMP1541]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3201]], label [[COND_TRUE3203:%.*]], label [[COND_FALSE3204:%.*]]
+// SIMD-ONLY0:       cond.true3203:
+// SIMD-ONLY0-NEXT:    [[TMP1542:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3205:%.*]]
+// SIMD-ONLY0:       cond.false3204:
+// SIMD-ONLY0-NEXT:    [[TMP1543:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3205]]
+// SIMD-ONLY0:       cond.end3205:
+// SIMD-ONLY0-NEXT:    [[COND3206:%.*]] = phi i64 [ [[TMP1542]], [[COND_TRUE3203]] ], [ [[TMP1543]], [[COND_FALSE3204]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3206]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1544:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1545:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3207:%.*]] = icmp eq i64 [[TMP1544]], [[TMP1545]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3207]], label [[COND_TRUE3209:%.*]], label [[COND_FALSE3210:%.*]]
+// SIMD-ONLY0:       cond.true3209:
+// SIMD-ONLY0-NEXT:    [[TMP1546:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3211:%.*]]
+// SIMD-ONLY0:       cond.false3210:
+// SIMD-ONLY0-NEXT:    [[TMP1547:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3211]]
+// SIMD-ONLY0:       cond.end3211:
+// SIMD-ONLY0-NEXT:    [[COND3212:%.*]] = phi i64 [ [[TMP1546]], [[COND_TRUE3209]] ], [ [[TMP1547]], [[COND_FALSE3210]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3212]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1548:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1549:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3213:%.*]] = icmp eq i64 [[TMP1548]], [[TMP1549]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3213]], label [[IF_THEN3215:%.*]], label [[IF_END3216:%.*]]
+// SIMD-ONLY0:       if.then3215:
+// SIMD-ONLY0-NEXT:    [[TMP1550:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1550]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3216]]
+// SIMD-ONLY0:       if.end3216:
+// SIMD-ONLY0-NEXT:    [[TMP1551:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1552:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3217:%.*]] = icmp eq i64 [[TMP1551]], [[TMP1552]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3217]], label [[IF_THEN3219:%.*]], label [[IF_END3220:%.*]]
+// SIMD-ONLY0:       if.then3219:
+// SIMD-ONLY0-NEXT:    [[TMP1553:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1553]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3220]]
+// SIMD-ONLY0:       if.end3220:
+// SIMD-ONLY0-NEXT:    [[TMP1554:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1555:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3221:%.*]] = icmp ugt i64 [[TMP1554]], [[TMP1555]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3221]], label [[COND_TRUE3223:%.*]], label [[COND_FALSE3224:%.*]]
+// SIMD-ONLY0:       cond.true3223:
+// SIMD-ONLY0-NEXT:    [[TMP1556:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3225:%.*]]
+// SIMD-ONLY0:       cond.false3224:
+// SIMD-ONLY0-NEXT:    [[TMP1557:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3225]]
+// SIMD-ONLY0:       cond.end3225:
+// SIMD-ONLY0-NEXT:    [[COND3226:%.*]] = phi i64 [ [[TMP1556]], [[COND_TRUE3223]] ], [ [[TMP1557]], [[COND_FALSE3224]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3226]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1558:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1559:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3227:%.*]] = icmp ult i64 [[TMP1558]], [[TMP1559]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3227]], label [[COND_TRUE3229:%.*]], label [[COND_FALSE3230:%.*]]
+// SIMD-ONLY0:       cond.true3229:
+// SIMD-ONLY0-NEXT:    [[TMP1560:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3231:%.*]]
+// SIMD-ONLY0:       cond.false3230:
+// SIMD-ONLY0-NEXT:    [[TMP1561:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3231]]
+// SIMD-ONLY0:       cond.end3231:
+// SIMD-ONLY0-NEXT:    [[COND3232:%.*]] = phi i64 [ [[TMP1560]], [[COND_TRUE3229]] ], [ [[TMP1561]], [[COND_FALSE3230]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3232]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1562:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1563:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3233:%.*]] = icmp ugt i64 [[TMP1562]], [[TMP1563]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3233]], label [[COND_TRUE3235:%.*]], label [[COND_FALSE3236:%.*]]
+// SIMD-ONLY0:       cond.true3235:
+// SIMD-ONLY0-NEXT:    [[TMP1564:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3237:%.*]]
+// SIMD-ONLY0:       cond.false3236:
+// SIMD-ONLY0-NEXT:    [[TMP1565:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3237]]
+// SIMD-ONLY0:       cond.end3237:
+// SIMD-ONLY0-NEXT:    [[COND3238:%.*]] = phi i64 [ [[TMP1564]], [[COND_TRUE3235]] ], [ [[TMP1565]], [[COND_FALSE3236]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3238]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1566:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1567:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3239:%.*]] = icmp ult i64 [[TMP1566]], [[TMP1567]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3239]], label [[COND_TRUE3241:%.*]], label [[COND_FALSE3242:%.*]]
+// SIMD-ONLY0:       cond.true3241:
+// SIMD-ONLY0-NEXT:    [[TMP1568:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3243:%.*]]
+// SIMD-ONLY0:       cond.false3242:
+// SIMD-ONLY0-NEXT:    [[TMP1569:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3243]]
+// SIMD-ONLY0:       cond.end3243:
+// SIMD-ONLY0-NEXT:    [[COND3244:%.*]] = phi i64 [ [[TMP1568]], [[COND_TRUE3241]] ], [ [[TMP1569]], [[COND_FALSE3242]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3244]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1570:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1571:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3245:%.*]] = icmp ugt i64 [[TMP1570]], [[TMP1571]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3245]], label [[IF_THEN3247:%.*]], label [[IF_END3248:%.*]]
+// SIMD-ONLY0:       if.then3247:
+// SIMD-ONLY0-NEXT:    [[TMP1572:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1572]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3248]]
+// SIMD-ONLY0:       if.end3248:
+// SIMD-ONLY0-NEXT:    [[TMP1573:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1574:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3249:%.*]] = icmp ult i64 [[TMP1573]], [[TMP1574]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3249]], label [[IF_THEN3251:%.*]], label [[IF_END3252:%.*]]
+// SIMD-ONLY0:       if.then3251:
+// SIMD-ONLY0-NEXT:    [[TMP1575:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1575]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3252]]
+// SIMD-ONLY0:       if.end3252:
+// SIMD-ONLY0-NEXT:    [[TMP1576:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1577:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3253:%.*]] = icmp ugt i64 [[TMP1576]], [[TMP1577]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3253]], label [[IF_THEN3255:%.*]], label [[IF_END3256:%.*]]
+// SIMD-ONLY0:       if.then3255:
+// SIMD-ONLY0-NEXT:    [[TMP1578:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1578]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3256]]
+// SIMD-ONLY0:       if.end3256:
+// SIMD-ONLY0-NEXT:    [[TMP1579:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1580:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3257:%.*]] = icmp ult i64 [[TMP1579]], [[TMP1580]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3257]], label [[IF_THEN3259:%.*]], label [[IF_END3260:%.*]]
+// SIMD-ONLY0:       if.then3259:
+// SIMD-ONLY0-NEXT:    [[TMP1581:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1581]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3260]]
+// SIMD-ONLY0:       if.end3260:
+// SIMD-ONLY0-NEXT:    [[TMP1582:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1583:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3261:%.*]] = icmp eq i64 [[TMP1582]], [[TMP1583]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3261]], label [[COND_TRUE3263:%.*]], label [[COND_FALSE3264:%.*]]
+// SIMD-ONLY0:       cond.true3263:
+// SIMD-ONLY0-NEXT:    [[TMP1584:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3265:%.*]]
+// SIMD-ONLY0:       cond.false3264:
+// SIMD-ONLY0-NEXT:    [[TMP1585:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3265]]
+// SIMD-ONLY0:       cond.end3265:
+// SIMD-ONLY0-NEXT:    [[COND3266:%.*]] = phi i64 [ [[TMP1584]], [[COND_TRUE3263]] ], [ [[TMP1585]], [[COND_FALSE3264]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3266]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1586:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1587:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3267:%.*]] = icmp eq i64 [[TMP1586]], [[TMP1587]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3267]], label [[COND_TRUE3269:%.*]], label [[COND_FALSE3270:%.*]]
+// SIMD-ONLY0:       cond.true3269:
+// SIMD-ONLY0-NEXT:    [[TMP1588:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3271:%.*]]
+// SIMD-ONLY0:       cond.false3270:
+// SIMD-ONLY0-NEXT:    [[TMP1589:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3271]]
+// SIMD-ONLY0:       cond.end3271:
+// SIMD-ONLY0-NEXT:    [[COND3272:%.*]] = phi i64 [ [[TMP1588]], [[COND_TRUE3269]] ], [ [[TMP1589]], [[COND_FALSE3270]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3272]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1590:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1591:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3273:%.*]] = icmp eq i64 [[TMP1590]], [[TMP1591]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3273]], label [[IF_THEN3275:%.*]], label [[IF_END3276:%.*]]
+// SIMD-ONLY0:       if.then3275:
+// SIMD-ONLY0-NEXT:    [[TMP1592:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1592]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3276]]
+// SIMD-ONLY0:       if.end3276:
+// SIMD-ONLY0-NEXT:    [[TMP1593:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1594:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3277:%.*]] = icmp eq i64 [[TMP1593]], [[TMP1594]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3277]], label [[IF_THEN3279:%.*]], label [[IF_END3280:%.*]]
+// SIMD-ONLY0:       if.then3279:
+// SIMD-ONLY0-NEXT:    [[TMP1595:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1595]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3280]]
+// SIMD-ONLY0:       if.end3280:
+// SIMD-ONLY0-NEXT:    [[TMP1596:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1597:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3281:%.*]] = icmp sgt i64 [[TMP1596]], [[TMP1597]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3281]], label [[COND_TRUE3283:%.*]], label [[COND_FALSE3284:%.*]]
+// SIMD-ONLY0:       cond.true3283:
+// SIMD-ONLY0-NEXT:    [[TMP1598:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3285:%.*]]
+// SIMD-ONLY0:       cond.false3284:
+// SIMD-ONLY0-NEXT:    [[TMP1599:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3285]]
+// SIMD-ONLY0:       cond.end3285:
+// SIMD-ONLY0-NEXT:    [[COND3286:%.*]] = phi i64 [ [[TMP1598]], [[COND_TRUE3283]] ], [ [[TMP1599]], [[COND_FALSE3284]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3286]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1600:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1601:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3287:%.*]] = icmp slt i64 [[TMP1600]], [[TMP1601]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3287]], label [[COND_TRUE3289:%.*]], label [[COND_FALSE3290:%.*]]
+// SIMD-ONLY0:       cond.true3289:
+// SIMD-ONLY0-NEXT:    [[TMP1602:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3291:%.*]]
+// SIMD-ONLY0:       cond.false3290:
+// SIMD-ONLY0-NEXT:    [[TMP1603:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3291]]
+// SIMD-ONLY0:       cond.end3291:
+// SIMD-ONLY0-NEXT:    [[COND3292:%.*]] = phi i64 [ [[TMP1602]], [[COND_TRUE3289]] ], [ [[TMP1603]], [[COND_FALSE3290]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3292]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1604:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1605:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3293:%.*]] = icmp sgt i64 [[TMP1604]], [[TMP1605]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3293]], label [[COND_TRUE3295:%.*]], label [[COND_FALSE3296:%.*]]
+// SIMD-ONLY0:       cond.true3295:
+// SIMD-ONLY0-NEXT:    [[TMP1606:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3297:%.*]]
+// SIMD-ONLY0:       cond.false3296:
+// SIMD-ONLY0-NEXT:    [[TMP1607:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3297]]
+// SIMD-ONLY0:       cond.end3297:
+// SIMD-ONLY0-NEXT:    [[COND3298:%.*]] = phi i64 [ [[TMP1606]], [[COND_TRUE3295]] ], [ [[TMP1607]], [[COND_FALSE3296]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3298]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1608:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1609:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3299:%.*]] = icmp slt i64 [[TMP1608]], [[TMP1609]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3299]], label [[COND_TRUE3301:%.*]], label [[COND_FALSE3302:%.*]]
+// SIMD-ONLY0:       cond.true3301:
+// SIMD-ONLY0-NEXT:    [[TMP1610:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3303:%.*]]
+// SIMD-ONLY0:       cond.false3302:
+// SIMD-ONLY0-NEXT:    [[TMP1611:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3303]]
+// SIMD-ONLY0:       cond.end3303:
+// SIMD-ONLY0-NEXT:    [[COND3304:%.*]] = phi i64 [ [[TMP1610]], [[COND_TRUE3301]] ], [ [[TMP1611]], [[COND_FALSE3302]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3304]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1612:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1613:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3305:%.*]] = icmp sgt i64 [[TMP1612]], [[TMP1613]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3305]], label [[IF_THEN3307:%.*]], label [[IF_END3308:%.*]]
+// SIMD-ONLY0:       if.then3307:
+// SIMD-ONLY0-NEXT:    [[TMP1614:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1614]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3308]]
+// SIMD-ONLY0:       if.end3308:
+// SIMD-ONLY0-NEXT:    [[TMP1615:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1616:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3309:%.*]] = icmp slt i64 [[TMP1615]], [[TMP1616]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3309]], label [[IF_THEN3311:%.*]], label [[IF_END3312:%.*]]
+// SIMD-ONLY0:       if.then3311:
+// SIMD-ONLY0-NEXT:    [[TMP1617:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1617]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3312]]
+// SIMD-ONLY0:       if.end3312:
+// SIMD-ONLY0-NEXT:    [[TMP1618:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1619:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3313:%.*]] = icmp sgt i64 [[TMP1618]], [[TMP1619]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3313]], label [[IF_THEN3315:%.*]], label [[IF_END3316:%.*]]
+// SIMD-ONLY0:       if.then3315:
+// SIMD-ONLY0-NEXT:    [[TMP1620:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1620]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3316]]
+// SIMD-ONLY0:       if.end3316:
+// SIMD-ONLY0-NEXT:    [[TMP1621:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1622:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3317:%.*]] = icmp slt i64 [[TMP1621]], [[TMP1622]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3317]], label [[IF_THEN3319:%.*]], label [[IF_END3320:%.*]]
+// SIMD-ONLY0:       if.then3319:
+// SIMD-ONLY0-NEXT:    [[TMP1623:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1623]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3320]]
+// SIMD-ONLY0:       if.end3320:
+// SIMD-ONLY0-NEXT:    [[TMP1624:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1625:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3321:%.*]] = icmp eq i64 [[TMP1624]], [[TMP1625]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3321]], label [[COND_TRUE3323:%.*]], label [[COND_FALSE3324:%.*]]
+// SIMD-ONLY0:       cond.true3323:
+// SIMD-ONLY0-NEXT:    [[TMP1626:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3325:%.*]]
+// SIMD-ONLY0:       cond.false3324:
+// SIMD-ONLY0-NEXT:    [[TMP1627:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3325]]
+// SIMD-ONLY0:       cond.end3325:
+// SIMD-ONLY0-NEXT:    [[COND3326:%.*]] = phi i64 [ [[TMP1626]], [[COND_TRUE3323]] ], [ [[TMP1627]], [[COND_FALSE3324]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3326]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1628:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1629:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3327:%.*]] = icmp eq i64 [[TMP1628]], [[TMP1629]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3327]], label [[COND_TRUE3329:%.*]], label [[COND_FALSE3330:%.*]]
+// SIMD-ONLY0:       cond.true3329:
+// SIMD-ONLY0-NEXT:    [[TMP1630:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3331:%.*]]
+// SIMD-ONLY0:       cond.false3330:
+// SIMD-ONLY0-NEXT:    [[TMP1631:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3331]]
+// SIMD-ONLY0:       cond.end3331:
+// SIMD-ONLY0-NEXT:    [[COND3332:%.*]] = phi i64 [ [[TMP1630]], [[COND_TRUE3329]] ], [ [[TMP1631]], [[COND_FALSE3330]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3332]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1632:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1633:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3333:%.*]] = icmp eq i64 [[TMP1632]], [[TMP1633]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3333]], label [[IF_THEN3335:%.*]], label [[IF_END3336:%.*]]
+// SIMD-ONLY0:       if.then3335:
+// SIMD-ONLY0-NEXT:    [[TMP1634:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1634]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3336]]
+// SIMD-ONLY0:       if.end3336:
+// SIMD-ONLY0-NEXT:    [[TMP1635:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1636:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3337:%.*]] = icmp eq i64 [[TMP1635]], [[TMP1636]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3337]], label [[IF_THEN3339:%.*]], label [[IF_END3340:%.*]]
+// SIMD-ONLY0:       if.then3339:
+// SIMD-ONLY0-NEXT:    [[TMP1637:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1637]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3340]]
+// SIMD-ONLY0:       if.end3340:
+// SIMD-ONLY0-NEXT:    [[TMP1638:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1639:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3341:%.*]] = icmp ugt i64 [[TMP1638]], [[TMP1639]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3341]], label [[COND_TRUE3343:%.*]], label [[COND_FALSE3344:%.*]]
+// SIMD-ONLY0:       cond.true3343:
+// SIMD-ONLY0-NEXT:    [[TMP1640:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3345:%.*]]
+// SIMD-ONLY0:       cond.false3344:
+// SIMD-ONLY0-NEXT:    [[TMP1641:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3345]]
+// SIMD-ONLY0:       cond.end3345:
+// SIMD-ONLY0-NEXT:    [[COND3346:%.*]] = phi i64 [ [[TMP1640]], [[COND_TRUE3343]] ], [ [[TMP1641]], [[COND_FALSE3344]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3346]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1642:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1643:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3347:%.*]] = icmp ult i64 [[TMP1642]], [[TMP1643]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3347]], label [[COND_TRUE3349:%.*]], label [[COND_FALSE3350:%.*]]
+// SIMD-ONLY0:       cond.true3349:
+// SIMD-ONLY0-NEXT:    [[TMP1644:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3351:%.*]]
+// SIMD-ONLY0:       cond.false3350:
+// SIMD-ONLY0-NEXT:    [[TMP1645:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3351]]
+// SIMD-ONLY0:       cond.end3351:
+// SIMD-ONLY0-NEXT:    [[COND3352:%.*]] = phi i64 [ [[TMP1644]], [[COND_TRUE3349]] ], [ [[TMP1645]], [[COND_FALSE3350]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3352]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1646:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1647:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3353:%.*]] = icmp ugt i64 [[TMP1646]], [[TMP1647]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3353]], label [[COND_TRUE3355:%.*]], label [[COND_FALSE3356:%.*]]
+// SIMD-ONLY0:       cond.true3355:
+// SIMD-ONLY0-NEXT:    [[TMP1648:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3357:%.*]]
+// SIMD-ONLY0:       cond.false3356:
+// SIMD-ONLY0-NEXT:    [[TMP1649:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3357]]
+// SIMD-ONLY0:       cond.end3357:
+// SIMD-ONLY0-NEXT:    [[COND3358:%.*]] = phi i64 [ [[TMP1648]], [[COND_TRUE3355]] ], [ [[TMP1649]], [[COND_FALSE3356]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3358]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1650:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1651:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3359:%.*]] = icmp ult i64 [[TMP1650]], [[TMP1651]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3359]], label [[COND_TRUE3361:%.*]], label [[COND_FALSE3362:%.*]]
+// SIMD-ONLY0:       cond.true3361:
+// SIMD-ONLY0-NEXT:    [[TMP1652:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3363:%.*]]
+// SIMD-ONLY0:       cond.false3362:
+// SIMD-ONLY0-NEXT:    [[TMP1653:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3363]]
+// SIMD-ONLY0:       cond.end3363:
+// SIMD-ONLY0-NEXT:    [[COND3364:%.*]] = phi i64 [ [[TMP1652]], [[COND_TRUE3361]] ], [ [[TMP1653]], [[COND_FALSE3362]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3364]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1654:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1655:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3365:%.*]] = icmp ugt i64 [[TMP1654]], [[TMP1655]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3365]], label [[IF_THEN3367:%.*]], label [[IF_END3368:%.*]]
+// SIMD-ONLY0:       if.then3367:
+// SIMD-ONLY0-NEXT:    [[TMP1656:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1656]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3368]]
+// SIMD-ONLY0:       if.end3368:
+// SIMD-ONLY0-NEXT:    [[TMP1657:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1658:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3369:%.*]] = icmp ult i64 [[TMP1657]], [[TMP1658]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3369]], label [[IF_THEN3371:%.*]], label [[IF_END3372:%.*]]
+// SIMD-ONLY0:       if.then3371:
+// SIMD-ONLY0-NEXT:    [[TMP1659:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1659]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3372]]
+// SIMD-ONLY0:       if.end3372:
+// SIMD-ONLY0-NEXT:    [[TMP1660:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1661:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3373:%.*]] = icmp ugt i64 [[TMP1660]], [[TMP1661]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3373]], label [[IF_THEN3375:%.*]], label [[IF_END3376:%.*]]
+// SIMD-ONLY0:       if.then3375:
+// SIMD-ONLY0-NEXT:    [[TMP1662:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1662]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3376]]
+// SIMD-ONLY0:       if.end3376:
+// SIMD-ONLY0-NEXT:    [[TMP1663:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1664:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3377:%.*]] = icmp ult i64 [[TMP1663]], [[TMP1664]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3377]], label [[IF_THEN3379:%.*]], label [[IF_END3380:%.*]]
+// SIMD-ONLY0:       if.then3379:
+// SIMD-ONLY0-NEXT:    [[TMP1665:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1665]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3380]]
+// SIMD-ONLY0:       if.end3380:
+// SIMD-ONLY0-NEXT:    [[TMP1666:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1667:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3381:%.*]] = icmp eq i64 [[TMP1666]], [[TMP1667]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3381]], label [[COND_TRUE3383:%.*]], label [[COND_FALSE3384:%.*]]
+// SIMD-ONLY0:       cond.true3383:
+// SIMD-ONLY0-NEXT:    [[TMP1668:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3385:%.*]]
+// SIMD-ONLY0:       cond.false3384:
+// SIMD-ONLY0-NEXT:    [[TMP1669:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3385]]
+// SIMD-ONLY0:       cond.end3385:
+// SIMD-ONLY0-NEXT:    [[COND3386:%.*]] = phi i64 [ [[TMP1668]], [[COND_TRUE3383]] ], [ [[TMP1669]], [[COND_FALSE3384]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3386]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1670:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1671:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3387:%.*]] = icmp eq i64 [[TMP1670]], [[TMP1671]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3387]], label [[COND_TRUE3389:%.*]], label [[COND_FALSE3390:%.*]]
+// SIMD-ONLY0:       cond.true3389:
+// SIMD-ONLY0-NEXT:    [[TMP1672:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3391:%.*]]
+// SIMD-ONLY0:       cond.false3390:
+// SIMD-ONLY0-NEXT:    [[TMP1673:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3391]]
+// SIMD-ONLY0:       cond.end3391:
+// SIMD-ONLY0-NEXT:    [[COND3392:%.*]] = phi i64 [ [[TMP1672]], [[COND_TRUE3389]] ], [ [[TMP1673]], [[COND_FALSE3390]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3392]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1674:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1675:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3393:%.*]] = icmp eq i64 [[TMP1674]], [[TMP1675]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3393]], label [[IF_THEN3395:%.*]], label [[IF_END3396:%.*]]
+// SIMD-ONLY0:       if.then3395:
+// SIMD-ONLY0-NEXT:    [[TMP1676:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1676]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3396]]
+// SIMD-ONLY0:       if.end3396:
+// SIMD-ONLY0-NEXT:    [[TMP1677:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1678:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3397:%.*]] = icmp eq i64 [[TMP1677]], [[TMP1678]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3397]], label [[IF_THEN3399:%.*]], label [[IF_END3400:%.*]]
+// SIMD-ONLY0:       if.then3399:
+// SIMD-ONLY0-NEXT:    [[TMP1679:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1679]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3400]]
+// SIMD-ONLY0:       if.end3400:
+// SIMD-ONLY0-NEXT:    [[TMP1680:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1681:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3401:%.*]] = icmp sgt i64 [[TMP1680]], [[TMP1681]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3401]], label [[COND_TRUE3403:%.*]], label [[COND_FALSE3404:%.*]]
+// SIMD-ONLY0:       cond.true3403:
+// SIMD-ONLY0-NEXT:    [[TMP1682:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3405:%.*]]
+// SIMD-ONLY0:       cond.false3404:
+// SIMD-ONLY0-NEXT:    [[TMP1683:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3405]]
+// SIMD-ONLY0:       cond.end3405:
+// SIMD-ONLY0-NEXT:    [[COND3406:%.*]] = phi i64 [ [[TMP1682]], [[COND_TRUE3403]] ], [ [[TMP1683]], [[COND_FALSE3404]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3406]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1684:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1685:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3407:%.*]] = icmp slt i64 [[TMP1684]], [[TMP1685]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3407]], label [[COND_TRUE3409:%.*]], label [[COND_FALSE3410:%.*]]
+// SIMD-ONLY0:       cond.true3409:
+// SIMD-ONLY0-NEXT:    [[TMP1686:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3411:%.*]]
+// SIMD-ONLY0:       cond.false3410:
+// SIMD-ONLY0-NEXT:    [[TMP1687:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3411]]
+// SIMD-ONLY0:       cond.end3411:
+// SIMD-ONLY0-NEXT:    [[COND3412:%.*]] = phi i64 [ [[TMP1686]], [[COND_TRUE3409]] ], [ [[TMP1687]], [[COND_FALSE3410]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3412]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1688:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1689:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3413:%.*]] = icmp sgt i64 [[TMP1688]], [[TMP1689]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3413]], label [[COND_TRUE3415:%.*]], label [[COND_FALSE3416:%.*]]
+// SIMD-ONLY0:       cond.true3415:
+// SIMD-ONLY0-NEXT:    [[TMP1690:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3417:%.*]]
+// SIMD-ONLY0:       cond.false3416:
+// SIMD-ONLY0-NEXT:    [[TMP1691:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3417]]
+// SIMD-ONLY0:       cond.end3417:
+// SIMD-ONLY0-NEXT:    [[COND3418:%.*]] = phi i64 [ [[TMP1690]], [[COND_TRUE3415]] ], [ [[TMP1691]], [[COND_FALSE3416]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3418]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1692:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1693:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3419:%.*]] = icmp slt i64 [[TMP1692]], [[TMP1693]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3419]], label [[COND_TRUE3421:%.*]], label [[COND_FALSE3422:%.*]]
+// SIMD-ONLY0:       cond.true3421:
+// SIMD-ONLY0-NEXT:    [[TMP1694:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3423:%.*]]
+// SIMD-ONLY0:       cond.false3422:
+// SIMD-ONLY0-NEXT:    [[TMP1695:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3423]]
+// SIMD-ONLY0:       cond.end3423:
+// SIMD-ONLY0-NEXT:    [[COND3424:%.*]] = phi i64 [ [[TMP1694]], [[COND_TRUE3421]] ], [ [[TMP1695]], [[COND_FALSE3422]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3424]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1696:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1697:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3425:%.*]] = icmp sgt i64 [[TMP1696]], [[TMP1697]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3425]], label [[IF_THEN3427:%.*]], label [[IF_END3428:%.*]]
+// SIMD-ONLY0:       if.then3427:
+// SIMD-ONLY0-NEXT:    [[TMP1698:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1698]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3428]]
+// SIMD-ONLY0:       if.end3428:
+// SIMD-ONLY0-NEXT:    [[TMP1699:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1700:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3429:%.*]] = icmp slt i64 [[TMP1699]], [[TMP1700]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3429]], label [[IF_THEN3431:%.*]], label [[IF_END3432:%.*]]
+// SIMD-ONLY0:       if.then3431:
+// SIMD-ONLY0-NEXT:    [[TMP1701:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1701]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3432]]
+// SIMD-ONLY0:       if.end3432:
+// SIMD-ONLY0-NEXT:    [[TMP1702:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1703:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3433:%.*]] = icmp sgt i64 [[TMP1702]], [[TMP1703]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3433]], label [[IF_THEN3435:%.*]], label [[IF_END3436:%.*]]
+// SIMD-ONLY0:       if.then3435:
+// SIMD-ONLY0-NEXT:    [[TMP1704:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1704]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3436]]
+// SIMD-ONLY0:       if.end3436:
+// SIMD-ONLY0-NEXT:    [[TMP1705:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1706:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3437:%.*]] = icmp slt i64 [[TMP1705]], [[TMP1706]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3437]], label [[IF_THEN3439:%.*]], label [[IF_END3440:%.*]]
+// SIMD-ONLY0:       if.then3439:
+// SIMD-ONLY0-NEXT:    [[TMP1707:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1707]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3440]]
+// SIMD-ONLY0:       if.end3440:
+// SIMD-ONLY0-NEXT:    [[TMP1708:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1709:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3441:%.*]] = icmp eq i64 [[TMP1708]], [[TMP1709]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3441]], label [[COND_TRUE3443:%.*]], label [[COND_FALSE3444:%.*]]
+// SIMD-ONLY0:       cond.true3443:
+// SIMD-ONLY0-NEXT:    [[TMP1710:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3445:%.*]]
+// SIMD-ONLY0:       cond.false3444:
+// SIMD-ONLY0-NEXT:    [[TMP1711:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3445]]
+// SIMD-ONLY0:       cond.end3445:
+// SIMD-ONLY0-NEXT:    [[COND3446:%.*]] = phi i64 [ [[TMP1710]], [[COND_TRUE3443]] ], [ [[TMP1711]], [[COND_FALSE3444]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3446]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1712:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1713:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3447:%.*]] = icmp eq i64 [[TMP1712]], [[TMP1713]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3447]], label [[COND_TRUE3449:%.*]], label [[COND_FALSE3450:%.*]]
+// SIMD-ONLY0:       cond.true3449:
+// SIMD-ONLY0-NEXT:    [[TMP1714:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3451:%.*]]
+// SIMD-ONLY0:       cond.false3450:
+// SIMD-ONLY0-NEXT:    [[TMP1715:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3451]]
+// SIMD-ONLY0:       cond.end3451:
+// SIMD-ONLY0-NEXT:    [[COND3452:%.*]] = phi i64 [ [[TMP1714]], [[COND_TRUE3449]] ], [ [[TMP1715]], [[COND_FALSE3450]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3452]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1716:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1717:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3453:%.*]] = icmp eq i64 [[TMP1716]], [[TMP1717]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3453]], label [[IF_THEN3455:%.*]], label [[IF_END3456:%.*]]
+// SIMD-ONLY0:       if.then3455:
+// SIMD-ONLY0-NEXT:    [[TMP1718:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1718]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3456]]
+// SIMD-ONLY0:       if.end3456:
+// SIMD-ONLY0-NEXT:    [[TMP1719:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1720:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3457:%.*]] = icmp eq i64 [[TMP1719]], [[TMP1720]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3457]], label [[IF_THEN3459:%.*]], label [[IF_END3460:%.*]]
+// SIMD-ONLY0:       if.then3459:
+// SIMD-ONLY0-NEXT:    [[TMP1721:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1721]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3460]]
+// SIMD-ONLY0:       if.end3460:
+// SIMD-ONLY0-NEXT:    [[TMP1722:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1723:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3461:%.*]] = icmp ugt i64 [[TMP1722]], [[TMP1723]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3461]], label [[COND_TRUE3463:%.*]], label [[COND_FALSE3464:%.*]]
+// SIMD-ONLY0:       cond.true3463:
+// SIMD-ONLY0-NEXT:    [[TMP1724:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3465:%.*]]
+// SIMD-ONLY0:       cond.false3464:
+// SIMD-ONLY0-NEXT:    [[TMP1725:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3465]]
+// SIMD-ONLY0:       cond.end3465:
+// SIMD-ONLY0-NEXT:    [[COND3466:%.*]] = phi i64 [ [[TMP1724]], [[COND_TRUE3463]] ], [ [[TMP1725]], [[COND_FALSE3464]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3466]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1726:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1727:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3467:%.*]] = icmp ult i64 [[TMP1726]], [[TMP1727]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3467]], label [[COND_TRUE3469:%.*]], label [[COND_FALSE3470:%.*]]
+// SIMD-ONLY0:       cond.true3469:
+// SIMD-ONLY0-NEXT:    [[TMP1728:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3471:%.*]]
+// SIMD-ONLY0:       cond.false3470:
+// SIMD-ONLY0-NEXT:    [[TMP1729:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3471]]
+// SIMD-ONLY0:       cond.end3471:
+// SIMD-ONLY0-NEXT:    [[COND3472:%.*]] = phi i64 [ [[TMP1728]], [[COND_TRUE3469]] ], [ [[TMP1729]], [[COND_FALSE3470]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3472]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1730:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1731:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3473:%.*]] = icmp ugt i64 [[TMP1730]], [[TMP1731]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3473]], label [[COND_TRUE3475:%.*]], label [[COND_FALSE3476:%.*]]
+// SIMD-ONLY0:       cond.true3475:
+// SIMD-ONLY0-NEXT:    [[TMP1732:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3477:%.*]]
+// SIMD-ONLY0:       cond.false3476:
+// SIMD-ONLY0-NEXT:    [[TMP1733:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3477]]
+// SIMD-ONLY0:       cond.end3477:
+// SIMD-ONLY0-NEXT:    [[COND3478:%.*]] = phi i64 [ [[TMP1732]], [[COND_TRUE3475]] ], [ [[TMP1733]], [[COND_FALSE3476]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3478]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1734:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1735:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3479:%.*]] = icmp ult i64 [[TMP1734]], [[TMP1735]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3479]], label [[COND_TRUE3481:%.*]], label [[COND_FALSE3482:%.*]]
+// SIMD-ONLY0:       cond.true3481:
+// SIMD-ONLY0-NEXT:    [[TMP1736:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3483:%.*]]
+// SIMD-ONLY0:       cond.false3482:
+// SIMD-ONLY0-NEXT:    [[TMP1737:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3483]]
+// SIMD-ONLY0:       cond.end3483:
+// SIMD-ONLY0-NEXT:    [[COND3484:%.*]] = phi i64 [ [[TMP1736]], [[COND_TRUE3481]] ], [ [[TMP1737]], [[COND_FALSE3482]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3484]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1738:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1739:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3485:%.*]] = icmp ugt i64 [[TMP1738]], [[TMP1739]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3485]], label [[IF_THEN3487:%.*]], label [[IF_END3488:%.*]]
+// SIMD-ONLY0:       if.then3487:
+// SIMD-ONLY0-NEXT:    [[TMP1740:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1740]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3488]]
+// SIMD-ONLY0:       if.end3488:
+// SIMD-ONLY0-NEXT:    [[TMP1741:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1742:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3489:%.*]] = icmp ult i64 [[TMP1741]], [[TMP1742]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3489]], label [[IF_THEN3491:%.*]], label [[IF_END3492:%.*]]
+// SIMD-ONLY0:       if.then3491:
+// SIMD-ONLY0-NEXT:    [[TMP1743:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1743]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3492]]
+// SIMD-ONLY0:       if.end3492:
+// SIMD-ONLY0-NEXT:    [[TMP1744:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1745:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3493:%.*]] = icmp ugt i64 [[TMP1744]], [[TMP1745]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3493]], label [[IF_THEN3495:%.*]], label [[IF_END3496:%.*]]
+// SIMD-ONLY0:       if.then3495:
+// SIMD-ONLY0-NEXT:    [[TMP1746:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1746]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3496]]
+// SIMD-ONLY0:       if.end3496:
+// SIMD-ONLY0-NEXT:    [[TMP1747:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1748:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3497:%.*]] = icmp ult i64 [[TMP1747]], [[TMP1748]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3497]], label [[IF_THEN3499:%.*]], label [[IF_END3500:%.*]]
+// SIMD-ONLY0:       if.then3499:
+// SIMD-ONLY0-NEXT:    [[TMP1749:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1749]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3500]]
+// SIMD-ONLY0:       if.end3500:
+// SIMD-ONLY0-NEXT:    [[TMP1750:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1751:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3501:%.*]] = icmp eq i64 [[TMP1750]], [[TMP1751]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3501]], label [[COND_TRUE3503:%.*]], label [[COND_FALSE3504:%.*]]
+// SIMD-ONLY0:       cond.true3503:
+// SIMD-ONLY0-NEXT:    [[TMP1752:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3505:%.*]]
+// SIMD-ONLY0:       cond.false3504:
+// SIMD-ONLY0-NEXT:    [[TMP1753:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3505]]
+// SIMD-ONLY0:       cond.end3505:
+// SIMD-ONLY0-NEXT:    [[COND3506:%.*]] = phi i64 [ [[TMP1752]], [[COND_TRUE3503]] ], [ [[TMP1753]], [[COND_FALSE3504]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3506]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1754:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1755:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3507:%.*]] = icmp eq i64 [[TMP1754]], [[TMP1755]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3507]], label [[COND_TRUE3509:%.*]], label [[COND_FALSE3510:%.*]]
+// SIMD-ONLY0:       cond.true3509:
+// SIMD-ONLY0-NEXT:    [[TMP1756:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3511:%.*]]
+// SIMD-ONLY0:       cond.false3510:
+// SIMD-ONLY0-NEXT:    [[TMP1757:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3511]]
+// SIMD-ONLY0:       cond.end3511:
+// SIMD-ONLY0-NEXT:    [[COND3512:%.*]] = phi i64 [ [[TMP1756]], [[COND_TRUE3509]] ], [ [[TMP1757]], [[COND_FALSE3510]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3512]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1758:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1759:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3513:%.*]] = icmp eq i64 [[TMP1758]], [[TMP1759]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3513]], label [[IF_THEN3515:%.*]], label [[IF_END3516:%.*]]
+// SIMD-ONLY0:       if.then3515:
+// SIMD-ONLY0-NEXT:    [[TMP1760:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1760]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3516]]
+// SIMD-ONLY0:       if.end3516:
+// SIMD-ONLY0-NEXT:    [[TMP1761:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1762:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3517:%.*]] = icmp eq i64 [[TMP1761]], [[TMP1762]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3517]], label [[IF_THEN3519:%.*]], label [[IF_END3520:%.*]]
+// SIMD-ONLY0:       if.then3519:
+// SIMD-ONLY0-NEXT:    [[TMP1763:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1763]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3520]]
+// SIMD-ONLY0:       if.end3520:
+// SIMD-ONLY0-NEXT:    [[TMP1764:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1765:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3521:%.*]] = icmp sgt i64 [[TMP1764]], [[TMP1765]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3521]], label [[COND_TRUE3523:%.*]], label [[COND_FALSE3524:%.*]]
+// SIMD-ONLY0:       cond.true3523:
+// SIMD-ONLY0-NEXT:    [[TMP1766:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3525:%.*]]
+// SIMD-ONLY0:       cond.false3524:
+// SIMD-ONLY0-NEXT:    [[TMP1767:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3525]]
+// SIMD-ONLY0:       cond.end3525:
+// SIMD-ONLY0-NEXT:    [[COND3526:%.*]] = phi i64 [ [[TMP1766]], [[COND_TRUE3523]] ], [ [[TMP1767]], [[COND_FALSE3524]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3526]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1768:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1769:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3527:%.*]] = icmp slt i64 [[TMP1768]], [[TMP1769]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3527]], label [[COND_TRUE3529:%.*]], label [[COND_FALSE3530:%.*]]
+// SIMD-ONLY0:       cond.true3529:
+// SIMD-ONLY0-NEXT:    [[TMP1770:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3531:%.*]]
+// SIMD-ONLY0:       cond.false3530:
+// SIMD-ONLY0-NEXT:    [[TMP1771:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3531]]
+// SIMD-ONLY0:       cond.end3531:
+// SIMD-ONLY0-NEXT:    [[COND3532:%.*]] = phi i64 [ [[TMP1770]], [[COND_TRUE3529]] ], [ [[TMP1771]], [[COND_FALSE3530]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3532]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1772:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1773:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3533:%.*]] = icmp sgt i64 [[TMP1772]], [[TMP1773]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3533]], label [[COND_TRUE3535:%.*]], label [[COND_FALSE3536:%.*]]
+// SIMD-ONLY0:       cond.true3535:
+// SIMD-ONLY0-NEXT:    [[TMP1774:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3537:%.*]]
+// SIMD-ONLY0:       cond.false3536:
+// SIMD-ONLY0-NEXT:    [[TMP1775:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3537]]
+// SIMD-ONLY0:       cond.end3537:
+// SIMD-ONLY0-NEXT:    [[COND3538:%.*]] = phi i64 [ [[TMP1774]], [[COND_TRUE3535]] ], [ [[TMP1775]], [[COND_FALSE3536]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3538]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1776:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1777:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3539:%.*]] = icmp slt i64 [[TMP1776]], [[TMP1777]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3539]], label [[COND_TRUE3541:%.*]], label [[COND_FALSE3542:%.*]]
+// SIMD-ONLY0:       cond.true3541:
+// SIMD-ONLY0-NEXT:    [[TMP1778:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3543:%.*]]
+// SIMD-ONLY0:       cond.false3542:
+// SIMD-ONLY0-NEXT:    [[TMP1779:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3543]]
+// SIMD-ONLY0:       cond.end3543:
+// SIMD-ONLY0-NEXT:    [[COND3544:%.*]] = phi i64 [ [[TMP1778]], [[COND_TRUE3541]] ], [ [[TMP1779]], [[COND_FALSE3542]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3544]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1780:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1781:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3545:%.*]] = icmp sgt i64 [[TMP1780]], [[TMP1781]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3545]], label [[IF_THEN3547:%.*]], label [[IF_END3548:%.*]]
+// SIMD-ONLY0:       if.then3547:
+// SIMD-ONLY0-NEXT:    [[TMP1782:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1782]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3548]]
+// SIMD-ONLY0:       if.end3548:
+// SIMD-ONLY0-NEXT:    [[TMP1783:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1784:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3549:%.*]] = icmp slt i64 [[TMP1783]], [[TMP1784]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3549]], label [[IF_THEN3551:%.*]], label [[IF_END3552:%.*]]
+// SIMD-ONLY0:       if.then3551:
+// SIMD-ONLY0-NEXT:    [[TMP1785:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1785]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3552]]
+// SIMD-ONLY0:       if.end3552:
+// SIMD-ONLY0-NEXT:    [[TMP1786:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1787:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3553:%.*]] = icmp sgt i64 [[TMP1786]], [[TMP1787]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3553]], label [[IF_THEN3555:%.*]], label [[IF_END3556:%.*]]
+// SIMD-ONLY0:       if.then3555:
+// SIMD-ONLY0-NEXT:    [[TMP1788:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1788]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3556]]
+// SIMD-ONLY0:       if.end3556:
+// SIMD-ONLY0-NEXT:    [[TMP1789:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1790:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3557:%.*]] = icmp slt i64 [[TMP1789]], [[TMP1790]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3557]], label [[IF_THEN3559:%.*]], label [[IF_END3560:%.*]]
+// SIMD-ONLY0:       if.then3559:
+// SIMD-ONLY0-NEXT:    [[TMP1791:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1791]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3560]]
+// SIMD-ONLY0:       if.end3560:
+// SIMD-ONLY0-NEXT:    [[TMP1792:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1793:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3561:%.*]] = icmp eq i64 [[TMP1792]], [[TMP1793]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3561]], label [[COND_TRUE3563:%.*]], label [[COND_FALSE3564:%.*]]
+// SIMD-ONLY0:       cond.true3563:
+// SIMD-ONLY0-NEXT:    [[TMP1794:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3565:%.*]]
+// SIMD-ONLY0:       cond.false3564:
+// SIMD-ONLY0-NEXT:    [[TMP1795:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3565]]
+// SIMD-ONLY0:       cond.end3565:
+// SIMD-ONLY0-NEXT:    [[COND3566:%.*]] = phi i64 [ [[TMP1794]], [[COND_TRUE3563]] ], [ [[TMP1795]], [[COND_FALSE3564]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3566]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1796:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1797:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3567:%.*]] = icmp eq i64 [[TMP1796]], [[TMP1797]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3567]], label [[COND_TRUE3569:%.*]], label [[COND_FALSE3570:%.*]]
+// SIMD-ONLY0:       cond.true3569:
+// SIMD-ONLY0-NEXT:    [[TMP1798:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3571:%.*]]
+// SIMD-ONLY0:       cond.false3570:
+// SIMD-ONLY0-NEXT:    [[TMP1799:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3571]]
+// SIMD-ONLY0:       cond.end3571:
+// SIMD-ONLY0-NEXT:    [[COND3572:%.*]] = phi i64 [ [[TMP1798]], [[COND_TRUE3569]] ], [ [[TMP1799]], [[COND_FALSE3570]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3572]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1800:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1801:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3573:%.*]] = icmp eq i64 [[TMP1800]], [[TMP1801]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3573]], label [[IF_THEN3575:%.*]], label [[IF_END3576:%.*]]
+// SIMD-ONLY0:       if.then3575:
+// SIMD-ONLY0-NEXT:    [[TMP1802:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1802]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3576]]
+// SIMD-ONLY0:       if.end3576:
+// SIMD-ONLY0-NEXT:    [[TMP1803:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1804:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3577:%.*]] = icmp eq i64 [[TMP1803]], [[TMP1804]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3577]], label [[IF_THEN3579:%.*]], label [[IF_END3580:%.*]]
+// SIMD-ONLY0:       if.then3579:
+// SIMD-ONLY0-NEXT:    [[TMP1805:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1805]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3580]]
+// SIMD-ONLY0:       if.end3580:
+// SIMD-ONLY0-NEXT:    [[TMP1806:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1807:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3581:%.*]] = icmp ugt i64 [[TMP1806]], [[TMP1807]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3581]], label [[COND_TRUE3583:%.*]], label [[COND_FALSE3584:%.*]]
+// SIMD-ONLY0:       cond.true3583:
+// SIMD-ONLY0-NEXT:    [[TMP1808:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3585:%.*]]
+// SIMD-ONLY0:       cond.false3584:
+// SIMD-ONLY0-NEXT:    [[TMP1809:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3585]]
+// SIMD-ONLY0:       cond.end3585:
+// SIMD-ONLY0-NEXT:    [[COND3586:%.*]] = phi i64 [ [[TMP1808]], [[COND_TRUE3583]] ], [ [[TMP1809]], [[COND_FALSE3584]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3586]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1810:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1811:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3587:%.*]] = icmp ult i64 [[TMP1810]], [[TMP1811]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3587]], label [[COND_TRUE3589:%.*]], label [[COND_FALSE3590:%.*]]
+// SIMD-ONLY0:       cond.true3589:
+// SIMD-ONLY0-NEXT:    [[TMP1812:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3591:%.*]]
+// SIMD-ONLY0:       cond.false3590:
+// SIMD-ONLY0-NEXT:    [[TMP1813:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3591]]
+// SIMD-ONLY0:       cond.end3591:
+// SIMD-ONLY0-NEXT:    [[COND3592:%.*]] = phi i64 [ [[TMP1812]], [[COND_TRUE3589]] ], [ [[TMP1813]], [[COND_FALSE3590]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3592]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1814:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1815:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3593:%.*]] = icmp ugt i64 [[TMP1814]], [[TMP1815]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3593]], label [[COND_TRUE3595:%.*]], label [[COND_FALSE3596:%.*]]
+// SIMD-ONLY0:       cond.true3595:
+// SIMD-ONLY0-NEXT:    [[TMP1816:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3597:%.*]]
+// SIMD-ONLY0:       cond.false3596:
+// SIMD-ONLY0-NEXT:    [[TMP1817:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3597]]
+// SIMD-ONLY0:       cond.end3597:
+// SIMD-ONLY0-NEXT:    [[COND3598:%.*]] = phi i64 [ [[TMP1816]], [[COND_TRUE3595]] ], [ [[TMP1817]], [[COND_FALSE3596]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3598]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1818:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1819:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3599:%.*]] = icmp ult i64 [[TMP1818]], [[TMP1819]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3599]], label [[COND_TRUE3601:%.*]], label [[COND_FALSE3602:%.*]]
+// SIMD-ONLY0:       cond.true3601:
+// SIMD-ONLY0-NEXT:    [[TMP1820:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3603:%.*]]
+// SIMD-ONLY0:       cond.false3602:
+// SIMD-ONLY0-NEXT:    [[TMP1821:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3603]]
+// SIMD-ONLY0:       cond.end3603:
+// SIMD-ONLY0-NEXT:    [[COND3604:%.*]] = phi i64 [ [[TMP1820]], [[COND_TRUE3601]] ], [ [[TMP1821]], [[COND_FALSE3602]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3604]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1822:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1823:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3605:%.*]] = icmp ugt i64 [[TMP1822]], [[TMP1823]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3605]], label [[IF_THEN3607:%.*]], label [[IF_END3608:%.*]]
+// SIMD-ONLY0:       if.then3607:
+// SIMD-ONLY0-NEXT:    [[TMP1824:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1824]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3608]]
+// SIMD-ONLY0:       if.end3608:
+// SIMD-ONLY0-NEXT:    [[TMP1825:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1826:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3609:%.*]] = icmp ult i64 [[TMP1825]], [[TMP1826]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3609]], label [[IF_THEN3611:%.*]], label [[IF_END3612:%.*]]
+// SIMD-ONLY0:       if.then3611:
+// SIMD-ONLY0-NEXT:    [[TMP1827:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1827]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3612]]
+// SIMD-ONLY0:       if.end3612:
+// SIMD-ONLY0-NEXT:    [[TMP1828:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1829:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3613:%.*]] = icmp ugt i64 [[TMP1828]], [[TMP1829]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3613]], label [[IF_THEN3615:%.*]], label [[IF_END3616:%.*]]
+// SIMD-ONLY0:       if.then3615:
+// SIMD-ONLY0-NEXT:    [[TMP1830:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1830]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3616]]
+// SIMD-ONLY0:       if.end3616:
+// SIMD-ONLY0-NEXT:    [[TMP1831:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1832:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3617:%.*]] = icmp ult i64 [[TMP1831]], [[TMP1832]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3617]], label [[IF_THEN3619:%.*]], label [[IF_END3620:%.*]]
+// SIMD-ONLY0:       if.then3619:
+// SIMD-ONLY0-NEXT:    [[TMP1833:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1833]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3620]]
+// SIMD-ONLY0:       if.end3620:
+// SIMD-ONLY0-NEXT:    [[TMP1834:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1835:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3621:%.*]] = icmp eq i64 [[TMP1834]], [[TMP1835]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3621]], label [[COND_TRUE3623:%.*]], label [[COND_FALSE3624:%.*]]
+// SIMD-ONLY0:       cond.true3623:
+// SIMD-ONLY0-NEXT:    [[TMP1836:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3625:%.*]]
+// SIMD-ONLY0:       cond.false3624:
+// SIMD-ONLY0-NEXT:    [[TMP1837:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3625]]
+// SIMD-ONLY0:       cond.end3625:
+// SIMD-ONLY0-NEXT:    [[COND3626:%.*]] = phi i64 [ [[TMP1836]], [[COND_TRUE3623]] ], [ [[TMP1837]], [[COND_FALSE3624]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3626]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1838:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1839:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3627:%.*]] = icmp eq i64 [[TMP1838]], [[TMP1839]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3627]], label [[COND_TRUE3629:%.*]], label [[COND_FALSE3630:%.*]]
+// SIMD-ONLY0:       cond.true3629:
+// SIMD-ONLY0-NEXT:    [[TMP1840:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3631:%.*]]
+// SIMD-ONLY0:       cond.false3630:
+// SIMD-ONLY0-NEXT:    [[TMP1841:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3631]]
+// SIMD-ONLY0:       cond.end3631:
+// SIMD-ONLY0-NEXT:    [[COND3632:%.*]] = phi i64 [ [[TMP1840]], [[COND_TRUE3629]] ], [ [[TMP1841]], [[COND_FALSE3630]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3632]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1842:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1843:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3633:%.*]] = icmp eq i64 [[TMP1842]], [[TMP1843]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3633]], label [[IF_THEN3635:%.*]], label [[IF_END3636:%.*]]
+// SIMD-ONLY0:       if.then3635:
+// SIMD-ONLY0-NEXT:    [[TMP1844:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1844]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3636]]
+// SIMD-ONLY0:       if.end3636:
+// SIMD-ONLY0-NEXT:    [[TMP1845:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1846:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3637:%.*]] = icmp eq i64 [[TMP1845]], [[TMP1846]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3637]], label [[IF_THEN3639:%.*]], label [[IF_END3640:%.*]]
+// SIMD-ONLY0:       if.then3639:
+// SIMD-ONLY0-NEXT:    [[TMP1847:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1847]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3640]]
+// SIMD-ONLY0:       if.end3640:
+// SIMD-ONLY0-NEXT:    [[TMP1848:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1849:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3641:%.*]] = icmp sgt i64 [[TMP1848]], [[TMP1849]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3641]], label [[COND_TRUE3643:%.*]], label [[COND_FALSE3644:%.*]]
+// SIMD-ONLY0:       cond.true3643:
+// SIMD-ONLY0-NEXT:    [[TMP1850:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3645:%.*]]
+// SIMD-ONLY0:       cond.false3644:
+// SIMD-ONLY0-NEXT:    [[TMP1851:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3645]]
+// SIMD-ONLY0:       cond.end3645:
+// SIMD-ONLY0-NEXT:    [[COND3646:%.*]] = phi i64 [ [[TMP1850]], [[COND_TRUE3643]] ], [ [[TMP1851]], [[COND_FALSE3644]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3646]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1852:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1853:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3647:%.*]] = icmp slt i64 [[TMP1852]], [[TMP1853]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3647]], label [[COND_TRUE3649:%.*]], label [[COND_FALSE3650:%.*]]
+// SIMD-ONLY0:       cond.true3649:
+// SIMD-ONLY0-NEXT:    [[TMP1854:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3651:%.*]]
+// SIMD-ONLY0:       cond.false3650:
+// SIMD-ONLY0-NEXT:    [[TMP1855:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3651]]
+// SIMD-ONLY0:       cond.end3651:
+// SIMD-ONLY0-NEXT:    [[COND3652:%.*]] = phi i64 [ [[TMP1854]], [[COND_TRUE3649]] ], [ [[TMP1855]], [[COND_FALSE3650]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3652]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1856:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1857:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3653:%.*]] = icmp sgt i64 [[TMP1856]], [[TMP1857]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3653]], label [[COND_TRUE3655:%.*]], label [[COND_FALSE3656:%.*]]
+// SIMD-ONLY0:       cond.true3655:
+// SIMD-ONLY0-NEXT:    [[TMP1858:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3657:%.*]]
+// SIMD-ONLY0:       cond.false3656:
+// SIMD-ONLY0-NEXT:    [[TMP1859:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3657]]
+// SIMD-ONLY0:       cond.end3657:
+// SIMD-ONLY0-NEXT:    [[COND3658:%.*]] = phi i64 [ [[TMP1858]], [[COND_TRUE3655]] ], [ [[TMP1859]], [[COND_FALSE3656]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3658]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1860:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1861:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3659:%.*]] = icmp slt i64 [[TMP1860]], [[TMP1861]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3659]], label [[COND_TRUE3661:%.*]], label [[COND_FALSE3662:%.*]]
+// SIMD-ONLY0:       cond.true3661:
+// SIMD-ONLY0-NEXT:    [[TMP1862:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3663:%.*]]
+// SIMD-ONLY0:       cond.false3662:
+// SIMD-ONLY0-NEXT:    [[TMP1863:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3663]]
+// SIMD-ONLY0:       cond.end3663:
+// SIMD-ONLY0-NEXT:    [[COND3664:%.*]] = phi i64 [ [[TMP1862]], [[COND_TRUE3661]] ], [ [[TMP1863]], [[COND_FALSE3662]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3664]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1864:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1865:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3665:%.*]] = icmp sgt i64 [[TMP1864]], [[TMP1865]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3665]], label [[IF_THEN3667:%.*]], label [[IF_END3668:%.*]]
+// SIMD-ONLY0:       if.then3667:
+// SIMD-ONLY0-NEXT:    [[TMP1866:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1866]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3668]]
+// SIMD-ONLY0:       if.end3668:
+// SIMD-ONLY0-NEXT:    [[TMP1867:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1868:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3669:%.*]] = icmp slt i64 [[TMP1867]], [[TMP1868]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3669]], label [[IF_THEN3671:%.*]], label [[IF_END3672:%.*]]
+// SIMD-ONLY0:       if.then3671:
+// SIMD-ONLY0-NEXT:    [[TMP1869:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1869]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3672]]
+// SIMD-ONLY0:       if.end3672:
+// SIMD-ONLY0-NEXT:    [[TMP1870:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1871:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3673:%.*]] = icmp sgt i64 [[TMP1870]], [[TMP1871]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3673]], label [[IF_THEN3675:%.*]], label [[IF_END3676:%.*]]
+// SIMD-ONLY0:       if.then3675:
+// SIMD-ONLY0-NEXT:    [[TMP1872:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1872]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3676]]
+// SIMD-ONLY0:       if.end3676:
+// SIMD-ONLY0-NEXT:    [[TMP1873:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1874:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3677:%.*]] = icmp slt i64 [[TMP1873]], [[TMP1874]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3677]], label [[IF_THEN3679:%.*]], label [[IF_END3680:%.*]]
+// SIMD-ONLY0:       if.then3679:
+// SIMD-ONLY0-NEXT:    [[TMP1875:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1875]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3680]]
+// SIMD-ONLY0:       if.end3680:
+// SIMD-ONLY0-NEXT:    [[TMP1876:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1877:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3681:%.*]] = icmp eq i64 [[TMP1876]], [[TMP1877]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3681]], label [[COND_TRUE3683:%.*]], label [[COND_FALSE3684:%.*]]
+// SIMD-ONLY0:       cond.true3683:
+// SIMD-ONLY0-NEXT:    [[TMP1878:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3685:%.*]]
+// SIMD-ONLY0:       cond.false3684:
+// SIMD-ONLY0-NEXT:    [[TMP1879:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3685]]
+// SIMD-ONLY0:       cond.end3685:
+// SIMD-ONLY0-NEXT:    [[COND3686:%.*]] = phi i64 [ [[TMP1878]], [[COND_TRUE3683]] ], [ [[TMP1879]], [[COND_FALSE3684]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3686]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1880:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1881:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3687:%.*]] = icmp eq i64 [[TMP1880]], [[TMP1881]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3687]], label [[COND_TRUE3689:%.*]], label [[COND_FALSE3690:%.*]]
+// SIMD-ONLY0:       cond.true3689:
+// SIMD-ONLY0-NEXT:    [[TMP1882:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3691:%.*]]
+// SIMD-ONLY0:       cond.false3690:
+// SIMD-ONLY0-NEXT:    [[TMP1883:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3691]]
+// SIMD-ONLY0:       cond.end3691:
+// SIMD-ONLY0-NEXT:    [[COND3692:%.*]] = phi i64 [ [[TMP1882]], [[COND_TRUE3689]] ], [ [[TMP1883]], [[COND_FALSE3690]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3692]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1884:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1885:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3693:%.*]] = icmp eq i64 [[TMP1884]], [[TMP1885]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3693]], label [[IF_THEN3695:%.*]], label [[IF_END3696:%.*]]
+// SIMD-ONLY0:       if.then3695:
+// SIMD-ONLY0-NEXT:    [[TMP1886:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1886]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3696]]
+// SIMD-ONLY0:       if.end3696:
+// SIMD-ONLY0-NEXT:    [[TMP1887:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1888:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3697:%.*]] = icmp eq i64 [[TMP1887]], [[TMP1888]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3697]], label [[IF_THEN3699:%.*]], label [[IF_END3700:%.*]]
+// SIMD-ONLY0:       if.then3699:
+// SIMD-ONLY0-NEXT:    [[TMP1889:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1889]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3700]]
+// SIMD-ONLY0:       if.end3700:
+// SIMD-ONLY0-NEXT:    [[TMP1890:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1891:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3701:%.*]] = icmp ugt i64 [[TMP1890]], [[TMP1891]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3701]], label [[COND_TRUE3703:%.*]], label [[COND_FALSE3704:%.*]]
+// SIMD-ONLY0:       cond.true3703:
+// SIMD-ONLY0-NEXT:    [[TMP1892:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3705:%.*]]
+// SIMD-ONLY0:       cond.false3704:
+// SIMD-ONLY0-NEXT:    [[TMP1893:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3705]]
+// SIMD-ONLY0:       cond.end3705:
+// SIMD-ONLY0-NEXT:    [[COND3706:%.*]] = phi i64 [ [[TMP1892]], [[COND_TRUE3703]] ], [ [[TMP1893]], [[COND_FALSE3704]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3706]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1894:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1895:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3707:%.*]] = icmp ult i64 [[TMP1894]], [[TMP1895]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3707]], label [[COND_TRUE3709:%.*]], label [[COND_FALSE3710:%.*]]
+// SIMD-ONLY0:       cond.true3709:
+// SIMD-ONLY0-NEXT:    [[TMP1896:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3711:%.*]]
+// SIMD-ONLY0:       cond.false3710:
+// SIMD-ONLY0-NEXT:    [[TMP1897:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3711]]
+// SIMD-ONLY0:       cond.end3711:
+// SIMD-ONLY0-NEXT:    [[COND3712:%.*]] = phi i64 [ [[TMP1896]], [[COND_TRUE3709]] ], [ [[TMP1897]], [[COND_FALSE3710]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3712]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1898:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1899:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3713:%.*]] = icmp ugt i64 [[TMP1898]], [[TMP1899]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3713]], label [[COND_TRUE3715:%.*]], label [[COND_FALSE3716:%.*]]
+// SIMD-ONLY0:       cond.true3715:
+// SIMD-ONLY0-NEXT:    [[TMP1900:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3717:%.*]]
+// SIMD-ONLY0:       cond.false3716:
+// SIMD-ONLY0-NEXT:    [[TMP1901:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3717]]
+// SIMD-ONLY0:       cond.end3717:
+// SIMD-ONLY0-NEXT:    [[COND3718:%.*]] = phi i64 [ [[TMP1900]], [[COND_TRUE3715]] ], [ [[TMP1901]], [[COND_FALSE3716]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3718]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1902:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1903:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3719:%.*]] = icmp ult i64 [[TMP1902]], [[TMP1903]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3719]], label [[COND_TRUE3721:%.*]], label [[COND_FALSE3722:%.*]]
+// SIMD-ONLY0:       cond.true3721:
+// SIMD-ONLY0-NEXT:    [[TMP1904:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3723:%.*]]
+// SIMD-ONLY0:       cond.false3722:
+// SIMD-ONLY0-NEXT:    [[TMP1905:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3723]]
+// SIMD-ONLY0:       cond.end3723:
+// SIMD-ONLY0-NEXT:    [[COND3724:%.*]] = phi i64 [ [[TMP1904]], [[COND_TRUE3721]] ], [ [[TMP1905]], [[COND_FALSE3722]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3724]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1906:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1907:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3725:%.*]] = icmp ugt i64 [[TMP1906]], [[TMP1907]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3725]], label [[IF_THEN3727:%.*]], label [[IF_END3728:%.*]]
+// SIMD-ONLY0:       if.then3727:
+// SIMD-ONLY0-NEXT:    [[TMP1908:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1908]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3728]]
+// SIMD-ONLY0:       if.end3728:
+// SIMD-ONLY0-NEXT:    [[TMP1909:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1910:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3729:%.*]] = icmp ult i64 [[TMP1909]], [[TMP1910]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3729]], label [[IF_THEN3731:%.*]], label [[IF_END3732:%.*]]
+// SIMD-ONLY0:       if.then3731:
+// SIMD-ONLY0-NEXT:    [[TMP1911:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1911]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3732]]
+// SIMD-ONLY0:       if.end3732:
+// SIMD-ONLY0-NEXT:    [[TMP1912:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1913:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3733:%.*]] = icmp ugt i64 [[TMP1912]], [[TMP1913]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3733]], label [[IF_THEN3735:%.*]], label [[IF_END3736:%.*]]
+// SIMD-ONLY0:       if.then3735:
+// SIMD-ONLY0-NEXT:    [[TMP1914:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1914]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3736]]
+// SIMD-ONLY0:       if.end3736:
+// SIMD-ONLY0-NEXT:    [[TMP1915:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1916:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3737:%.*]] = icmp ult i64 [[TMP1915]], [[TMP1916]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3737]], label [[IF_THEN3739:%.*]], label [[IF_END3740:%.*]]
+// SIMD-ONLY0:       if.then3739:
+// SIMD-ONLY0-NEXT:    [[TMP1917:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1917]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3740]]
+// SIMD-ONLY0:       if.end3740:
+// SIMD-ONLY0-NEXT:    [[TMP1918:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1919:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3741:%.*]] = icmp eq i64 [[TMP1918]], [[TMP1919]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3741]], label [[COND_TRUE3743:%.*]], label [[COND_FALSE3744:%.*]]
+// SIMD-ONLY0:       cond.true3743:
+// SIMD-ONLY0-NEXT:    [[TMP1920:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3745:%.*]]
+// SIMD-ONLY0:       cond.false3744:
+// SIMD-ONLY0-NEXT:    [[TMP1921:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3745]]
+// SIMD-ONLY0:       cond.end3745:
+// SIMD-ONLY0-NEXT:    [[COND3746:%.*]] = phi i64 [ [[TMP1920]], [[COND_TRUE3743]] ], [ [[TMP1921]], [[COND_FALSE3744]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3746]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1922:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1923:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3747:%.*]] = icmp eq i64 [[TMP1922]], [[TMP1923]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3747]], label [[COND_TRUE3749:%.*]], label [[COND_FALSE3750:%.*]]
+// SIMD-ONLY0:       cond.true3749:
+// SIMD-ONLY0-NEXT:    [[TMP1924:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3751:%.*]]
+// SIMD-ONLY0:       cond.false3750:
+// SIMD-ONLY0-NEXT:    [[TMP1925:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3751]]
+// SIMD-ONLY0:       cond.end3751:
+// SIMD-ONLY0-NEXT:    [[COND3752:%.*]] = phi i64 [ [[TMP1924]], [[COND_TRUE3749]] ], [ [[TMP1925]], [[COND_FALSE3750]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3752]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1926:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1927:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3753:%.*]] = icmp eq i64 [[TMP1926]], [[TMP1927]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3753]], label [[IF_THEN3755:%.*]], label [[IF_END3756:%.*]]
+// SIMD-ONLY0:       if.then3755:
+// SIMD-ONLY0-NEXT:    [[TMP1928:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1928]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3756]]
+// SIMD-ONLY0:       if.end3756:
+// SIMD-ONLY0-NEXT:    [[TMP1929:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1930:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3757:%.*]] = icmp eq i64 [[TMP1929]], [[TMP1930]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3757]], label [[IF_THEN3759:%.*]], label [[IF_END3760:%.*]]
+// SIMD-ONLY0:       if.then3759:
+// SIMD-ONLY0-NEXT:    [[TMP1931:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1931]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3760]]
+// SIMD-ONLY0:       if.end3760:
+// SIMD-ONLY0-NEXT:    [[TMP1932:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1933:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3761:%.*]] = icmp sgt i64 [[TMP1932]], [[TMP1933]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3761]], label [[COND_TRUE3763:%.*]], label [[COND_FALSE3764:%.*]]
+// SIMD-ONLY0:       cond.true3763:
+// SIMD-ONLY0-NEXT:    [[TMP1934:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3765:%.*]]
+// SIMD-ONLY0:       cond.false3764:
+// SIMD-ONLY0-NEXT:    [[TMP1935:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3765]]
+// SIMD-ONLY0:       cond.end3765:
+// SIMD-ONLY0-NEXT:    [[COND3766:%.*]] = phi i64 [ [[TMP1934]], [[COND_TRUE3763]] ], [ [[TMP1935]], [[COND_FALSE3764]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3766]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1936:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1937:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3767:%.*]] = icmp slt i64 [[TMP1936]], [[TMP1937]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3767]], label [[COND_TRUE3769:%.*]], label [[COND_FALSE3770:%.*]]
+// SIMD-ONLY0:       cond.true3769:
+// SIMD-ONLY0-NEXT:    [[TMP1938:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3771:%.*]]
+// SIMD-ONLY0:       cond.false3770:
+// SIMD-ONLY0-NEXT:    [[TMP1939:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3771]]
+// SIMD-ONLY0:       cond.end3771:
+// SIMD-ONLY0-NEXT:    [[COND3772:%.*]] = phi i64 [ [[TMP1938]], [[COND_TRUE3769]] ], [ [[TMP1939]], [[COND_FALSE3770]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3772]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1940:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1941:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3773:%.*]] = icmp sgt i64 [[TMP1940]], [[TMP1941]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3773]], label [[COND_TRUE3775:%.*]], label [[COND_FALSE3776:%.*]]
+// SIMD-ONLY0:       cond.true3775:
+// SIMD-ONLY0-NEXT:    [[TMP1942:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3777:%.*]]
+// SIMD-ONLY0:       cond.false3776:
+// SIMD-ONLY0-NEXT:    [[TMP1943:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3777]]
+// SIMD-ONLY0:       cond.end3777:
+// SIMD-ONLY0-NEXT:    [[COND3778:%.*]] = phi i64 [ [[TMP1942]], [[COND_TRUE3775]] ], [ [[TMP1943]], [[COND_FALSE3776]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3778]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1944:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1945:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3779:%.*]] = icmp slt i64 [[TMP1944]], [[TMP1945]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3779]], label [[COND_TRUE3781:%.*]], label [[COND_FALSE3782:%.*]]
+// SIMD-ONLY0:       cond.true3781:
+// SIMD-ONLY0-NEXT:    [[TMP1946:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3783:%.*]]
+// SIMD-ONLY0:       cond.false3782:
+// SIMD-ONLY0-NEXT:    [[TMP1947:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3783]]
+// SIMD-ONLY0:       cond.end3783:
+// SIMD-ONLY0-NEXT:    [[COND3784:%.*]] = phi i64 [ [[TMP1946]], [[COND_TRUE3781]] ], [ [[TMP1947]], [[COND_FALSE3782]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3784]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1948:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1949:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3785:%.*]] = icmp sgt i64 [[TMP1948]], [[TMP1949]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3785]], label [[IF_THEN3787:%.*]], label [[IF_END3788:%.*]]
+// SIMD-ONLY0:       if.then3787:
+// SIMD-ONLY0-NEXT:    [[TMP1950:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1950]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3788]]
+// SIMD-ONLY0:       if.end3788:
+// SIMD-ONLY0-NEXT:    [[TMP1951:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1952:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3789:%.*]] = icmp slt i64 [[TMP1951]], [[TMP1952]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3789]], label [[IF_THEN3791:%.*]], label [[IF_END3792:%.*]]
+// SIMD-ONLY0:       if.then3791:
+// SIMD-ONLY0-NEXT:    [[TMP1953:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1953]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3792]]
+// SIMD-ONLY0:       if.end3792:
+// SIMD-ONLY0-NEXT:    [[TMP1954:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1955:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3793:%.*]] = icmp sgt i64 [[TMP1954]], [[TMP1955]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3793]], label [[IF_THEN3795:%.*]], label [[IF_END3796:%.*]]
+// SIMD-ONLY0:       if.then3795:
+// SIMD-ONLY0-NEXT:    [[TMP1956:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1956]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3796]]
+// SIMD-ONLY0:       if.end3796:
+// SIMD-ONLY0-NEXT:    [[TMP1957:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1958:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3797:%.*]] = icmp slt i64 [[TMP1957]], [[TMP1958]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3797]], label [[IF_THEN3799:%.*]], label [[IF_END3800:%.*]]
+// SIMD-ONLY0:       if.then3799:
+// SIMD-ONLY0-NEXT:    [[TMP1959:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1959]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3800]]
+// SIMD-ONLY0:       if.end3800:
+// SIMD-ONLY0-NEXT:    [[TMP1960:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1961:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3801:%.*]] = icmp eq i64 [[TMP1960]], [[TMP1961]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3801]], label [[COND_TRUE3803:%.*]], label [[COND_FALSE3804:%.*]]
+// SIMD-ONLY0:       cond.true3803:
+// SIMD-ONLY0-NEXT:    [[TMP1962:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3805:%.*]]
+// SIMD-ONLY0:       cond.false3804:
+// SIMD-ONLY0-NEXT:    [[TMP1963:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3805]]
+// SIMD-ONLY0:       cond.end3805:
+// SIMD-ONLY0-NEXT:    [[COND3806:%.*]] = phi i64 [ [[TMP1962]], [[COND_TRUE3803]] ], [ [[TMP1963]], [[COND_FALSE3804]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3806]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1964:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1965:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3807:%.*]] = icmp eq i64 [[TMP1964]], [[TMP1965]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3807]], label [[COND_TRUE3809:%.*]], label [[COND_FALSE3810:%.*]]
+// SIMD-ONLY0:       cond.true3809:
+// SIMD-ONLY0-NEXT:    [[TMP1966:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3811:%.*]]
+// SIMD-ONLY0:       cond.false3810:
+// SIMD-ONLY0-NEXT:    [[TMP1967:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3811]]
+// SIMD-ONLY0:       cond.end3811:
+// SIMD-ONLY0-NEXT:    [[COND3812:%.*]] = phi i64 [ [[TMP1966]], [[COND_TRUE3809]] ], [ [[TMP1967]], [[COND_FALSE3810]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3812]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1968:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1969:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3813:%.*]] = icmp eq i64 [[TMP1968]], [[TMP1969]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3813]], label [[IF_THEN3815:%.*]], label [[IF_END3816:%.*]]
+// SIMD-ONLY0:       if.then3815:
+// SIMD-ONLY0-NEXT:    [[TMP1970:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1970]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3816]]
+// SIMD-ONLY0:       if.end3816:
+// SIMD-ONLY0-NEXT:    [[TMP1971:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1972:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3817:%.*]] = icmp eq i64 [[TMP1971]], [[TMP1972]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3817]], label [[IF_THEN3819:%.*]], label [[IF_END3820:%.*]]
+// SIMD-ONLY0:       if.then3819:
+// SIMD-ONLY0-NEXT:    [[TMP1973:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1973]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3820]]
+// SIMD-ONLY0:       if.end3820:
+// SIMD-ONLY0-NEXT:    [[TMP1974:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1975:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3821:%.*]] = icmp ugt i64 [[TMP1974]], [[TMP1975]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3821]], label [[COND_TRUE3823:%.*]], label [[COND_FALSE3824:%.*]]
+// SIMD-ONLY0:       cond.true3823:
+// SIMD-ONLY0-NEXT:    [[TMP1976:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3825:%.*]]
+// SIMD-ONLY0:       cond.false3824:
+// SIMD-ONLY0-NEXT:    [[TMP1977:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3825]]
+// SIMD-ONLY0:       cond.end3825:
+// SIMD-ONLY0-NEXT:    [[COND3826:%.*]] = phi i64 [ [[TMP1976]], [[COND_TRUE3823]] ], [ [[TMP1977]], [[COND_FALSE3824]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3826]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1978:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1979:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3827:%.*]] = icmp ult i64 [[TMP1978]], [[TMP1979]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3827]], label [[COND_TRUE3829:%.*]], label [[COND_FALSE3830:%.*]]
+// SIMD-ONLY0:       cond.true3829:
+// SIMD-ONLY0-NEXT:    [[TMP1980:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3831:%.*]]
+// SIMD-ONLY0:       cond.false3830:
+// SIMD-ONLY0-NEXT:    [[TMP1981:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3831]]
+// SIMD-ONLY0:       cond.end3831:
+// SIMD-ONLY0-NEXT:    [[COND3832:%.*]] = phi i64 [ [[TMP1980]], [[COND_TRUE3829]] ], [ [[TMP1981]], [[COND_FALSE3830]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3832]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1982:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1983:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3833:%.*]] = icmp ugt i64 [[TMP1982]], [[TMP1983]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3833]], label [[COND_TRUE3835:%.*]], label [[COND_FALSE3836:%.*]]
+// SIMD-ONLY0:       cond.true3835:
+// SIMD-ONLY0-NEXT:    [[TMP1984:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3837:%.*]]
+// SIMD-ONLY0:       cond.false3836:
+// SIMD-ONLY0-NEXT:    [[TMP1985:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3837]]
+// SIMD-ONLY0:       cond.end3837:
+// SIMD-ONLY0-NEXT:    [[COND3838:%.*]] = phi i64 [ [[TMP1984]], [[COND_TRUE3835]] ], [ [[TMP1985]], [[COND_FALSE3836]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3838]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1986:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1987:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3839:%.*]] = icmp ult i64 [[TMP1986]], [[TMP1987]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3839]], label [[COND_TRUE3841:%.*]], label [[COND_FALSE3842:%.*]]
+// SIMD-ONLY0:       cond.true3841:
+// SIMD-ONLY0-NEXT:    [[TMP1988:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3843:%.*]]
+// SIMD-ONLY0:       cond.false3842:
+// SIMD-ONLY0-NEXT:    [[TMP1989:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3843]]
+// SIMD-ONLY0:       cond.end3843:
+// SIMD-ONLY0-NEXT:    [[COND3844:%.*]] = phi i64 [ [[TMP1988]], [[COND_TRUE3841]] ], [ [[TMP1989]], [[COND_FALSE3842]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3844]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1990:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1991:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3845:%.*]] = icmp ugt i64 [[TMP1990]], [[TMP1991]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3845]], label [[IF_THEN3847:%.*]], label [[IF_END3848:%.*]]
+// SIMD-ONLY0:       if.then3847:
+// SIMD-ONLY0-NEXT:    [[TMP1992:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1992]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3848]]
+// SIMD-ONLY0:       if.end3848:
+// SIMD-ONLY0-NEXT:    [[TMP1993:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1994:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3849:%.*]] = icmp ult i64 [[TMP1993]], [[TMP1994]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3849]], label [[IF_THEN3851:%.*]], label [[IF_END3852:%.*]]
+// SIMD-ONLY0:       if.then3851:
+// SIMD-ONLY0-NEXT:    [[TMP1995:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1995]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3852]]
+// SIMD-ONLY0:       if.end3852:
+// SIMD-ONLY0-NEXT:    [[TMP1996:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1997:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3853:%.*]] = icmp ugt i64 [[TMP1996]], [[TMP1997]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3853]], label [[IF_THEN3855:%.*]], label [[IF_END3856:%.*]]
+// SIMD-ONLY0:       if.then3855:
+// SIMD-ONLY0-NEXT:    [[TMP1998:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP1998]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3856]]
+// SIMD-ONLY0:       if.end3856:
+// SIMD-ONLY0-NEXT:    [[TMP1999:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2000:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3857:%.*]] = icmp ult i64 [[TMP1999]], [[TMP2000]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3857]], label [[IF_THEN3859:%.*]], label [[IF_END3860:%.*]]
+// SIMD-ONLY0:       if.then3859:
+// SIMD-ONLY0-NEXT:    [[TMP2001:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2001]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3860]]
+// SIMD-ONLY0:       if.end3860:
+// SIMD-ONLY0-NEXT:    [[TMP2002:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2003:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3861:%.*]] = icmp eq i64 [[TMP2002]], [[TMP2003]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3861]], label [[COND_TRUE3863:%.*]], label [[COND_FALSE3864:%.*]]
+// SIMD-ONLY0:       cond.true3863:
+// SIMD-ONLY0-NEXT:    [[TMP2004:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3865:%.*]]
+// SIMD-ONLY0:       cond.false3864:
+// SIMD-ONLY0-NEXT:    [[TMP2005:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3865]]
+// SIMD-ONLY0:       cond.end3865:
+// SIMD-ONLY0-NEXT:    [[COND3866:%.*]] = phi i64 [ [[TMP2004]], [[COND_TRUE3863]] ], [ [[TMP2005]], [[COND_FALSE3864]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3866]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2006:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2007:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3867:%.*]] = icmp eq i64 [[TMP2006]], [[TMP2007]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3867]], label [[COND_TRUE3869:%.*]], label [[COND_FALSE3870:%.*]]
+// SIMD-ONLY0:       cond.true3869:
+// SIMD-ONLY0-NEXT:    [[TMP2008:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3871:%.*]]
+// SIMD-ONLY0:       cond.false3870:
+// SIMD-ONLY0-NEXT:    [[TMP2009:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3871]]
+// SIMD-ONLY0:       cond.end3871:
+// SIMD-ONLY0-NEXT:    [[COND3872:%.*]] = phi i64 [ [[TMP2008]], [[COND_TRUE3869]] ], [ [[TMP2009]], [[COND_FALSE3870]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3872]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2010:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2011:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3873:%.*]] = icmp eq i64 [[TMP2010]], [[TMP2011]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3873]], label [[IF_THEN3875:%.*]], label [[IF_END3876:%.*]]
+// SIMD-ONLY0:       if.then3875:
+// SIMD-ONLY0-NEXT:    [[TMP2012:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2012]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3876]]
+// SIMD-ONLY0:       if.end3876:
+// SIMD-ONLY0-NEXT:    [[TMP2013:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2014:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3877:%.*]] = icmp eq i64 [[TMP2013]], [[TMP2014]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3877]], label [[IF_THEN3879:%.*]], label [[IF_END3880:%.*]]
+// SIMD-ONLY0:       if.then3879:
+// SIMD-ONLY0-NEXT:    [[TMP2015:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2015]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3880]]
+// SIMD-ONLY0:       if.end3880:
+// SIMD-ONLY0-NEXT:    [[TMP2016:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2017:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3881:%.*]] = icmp sgt i64 [[TMP2016]], [[TMP2017]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3881]], label [[COND_TRUE3883:%.*]], label [[COND_FALSE3884:%.*]]
+// SIMD-ONLY0:       cond.true3883:
+// SIMD-ONLY0-NEXT:    [[TMP2018:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3885:%.*]]
+// SIMD-ONLY0:       cond.false3884:
+// SIMD-ONLY0-NEXT:    [[TMP2019:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3885]]
+// SIMD-ONLY0:       cond.end3885:
+// SIMD-ONLY0-NEXT:    [[COND3886:%.*]] = phi i64 [ [[TMP2018]], [[COND_TRUE3883]] ], [ [[TMP2019]], [[COND_FALSE3884]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3886]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2020:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2021:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3887:%.*]] = icmp slt i64 [[TMP2020]], [[TMP2021]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3887]], label [[COND_TRUE3889:%.*]], label [[COND_FALSE3890:%.*]]
+// SIMD-ONLY0:       cond.true3889:
+// SIMD-ONLY0-NEXT:    [[TMP2022:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3891:%.*]]
+// SIMD-ONLY0:       cond.false3890:
+// SIMD-ONLY0-NEXT:    [[TMP2023:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3891]]
+// SIMD-ONLY0:       cond.end3891:
+// SIMD-ONLY0-NEXT:    [[COND3892:%.*]] = phi i64 [ [[TMP2022]], [[COND_TRUE3889]] ], [ [[TMP2023]], [[COND_FALSE3890]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3892]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2024:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2025:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3893:%.*]] = icmp sgt i64 [[TMP2024]], [[TMP2025]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3893]], label [[COND_TRUE3895:%.*]], label [[COND_FALSE3896:%.*]]
+// SIMD-ONLY0:       cond.true3895:
+// SIMD-ONLY0-NEXT:    [[TMP2026:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3897:%.*]]
+// SIMD-ONLY0:       cond.false3896:
+// SIMD-ONLY0-NEXT:    [[TMP2027:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3897]]
+// SIMD-ONLY0:       cond.end3897:
+// SIMD-ONLY0-NEXT:    [[COND3898:%.*]] = phi i64 [ [[TMP2026]], [[COND_TRUE3895]] ], [ [[TMP2027]], [[COND_FALSE3896]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3898]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2028:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2029:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3899:%.*]] = icmp slt i64 [[TMP2028]], [[TMP2029]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3899]], label [[COND_TRUE3901:%.*]], label [[COND_FALSE3902:%.*]]
+// SIMD-ONLY0:       cond.true3901:
+// SIMD-ONLY0-NEXT:    [[TMP2030:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3903:%.*]]
+// SIMD-ONLY0:       cond.false3902:
+// SIMD-ONLY0-NEXT:    [[TMP2031:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3903]]
+// SIMD-ONLY0:       cond.end3903:
+// SIMD-ONLY0-NEXT:    [[COND3904:%.*]] = phi i64 [ [[TMP2030]], [[COND_TRUE3901]] ], [ [[TMP2031]], [[COND_FALSE3902]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3904]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2032:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2033:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3905:%.*]] = icmp sgt i64 [[TMP2032]], [[TMP2033]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3905]], label [[IF_THEN3907:%.*]], label [[IF_END3908:%.*]]
+// SIMD-ONLY0:       if.then3907:
+// SIMD-ONLY0-NEXT:    [[TMP2034:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2034]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3908]]
+// SIMD-ONLY0:       if.end3908:
+// SIMD-ONLY0-NEXT:    [[TMP2035:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2036:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3909:%.*]] = icmp slt i64 [[TMP2035]], [[TMP2036]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3909]], label [[IF_THEN3911:%.*]], label [[IF_END3912:%.*]]
+// SIMD-ONLY0:       if.then3911:
+// SIMD-ONLY0-NEXT:    [[TMP2037:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2037]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3912]]
+// SIMD-ONLY0:       if.end3912:
+// SIMD-ONLY0-NEXT:    [[TMP2038:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2039:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3913:%.*]] = icmp sgt i64 [[TMP2038]], [[TMP2039]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3913]], label [[IF_THEN3915:%.*]], label [[IF_END3916:%.*]]
+// SIMD-ONLY0:       if.then3915:
+// SIMD-ONLY0-NEXT:    [[TMP2040:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2040]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3916]]
+// SIMD-ONLY0:       if.end3916:
+// SIMD-ONLY0-NEXT:    [[TMP2041:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2042:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3917:%.*]] = icmp slt i64 [[TMP2041]], [[TMP2042]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3917]], label [[IF_THEN3919:%.*]], label [[IF_END3920:%.*]]
+// SIMD-ONLY0:       if.then3919:
+// SIMD-ONLY0-NEXT:    [[TMP2043:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2043]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3920]]
+// SIMD-ONLY0:       if.end3920:
+// SIMD-ONLY0-NEXT:    [[TMP2044:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2045:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3921:%.*]] = icmp eq i64 [[TMP2044]], [[TMP2045]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3921]], label [[COND_TRUE3923:%.*]], label [[COND_FALSE3924:%.*]]
+// SIMD-ONLY0:       cond.true3923:
+// SIMD-ONLY0-NEXT:    [[TMP2046:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3925:%.*]]
+// SIMD-ONLY0:       cond.false3924:
+// SIMD-ONLY0-NEXT:    [[TMP2047:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3925]]
+// SIMD-ONLY0:       cond.end3925:
+// SIMD-ONLY0-NEXT:    [[COND3926:%.*]] = phi i64 [ [[TMP2046]], [[COND_TRUE3923]] ], [ [[TMP2047]], [[COND_FALSE3924]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3926]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2048:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2049:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3927:%.*]] = icmp eq i64 [[TMP2048]], [[TMP2049]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3927]], label [[COND_TRUE3929:%.*]], label [[COND_FALSE3930:%.*]]
+// SIMD-ONLY0:       cond.true3929:
+// SIMD-ONLY0-NEXT:    [[TMP2050:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3931:%.*]]
+// SIMD-ONLY0:       cond.false3930:
+// SIMD-ONLY0-NEXT:    [[TMP2051:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3931]]
+// SIMD-ONLY0:       cond.end3931:
+// SIMD-ONLY0-NEXT:    [[COND3932:%.*]] = phi i64 [ [[TMP2050]], [[COND_TRUE3929]] ], [ [[TMP2051]], [[COND_FALSE3930]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3932]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2052:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2053:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3933:%.*]] = icmp eq i64 [[TMP2052]], [[TMP2053]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3933]], label [[IF_THEN3935:%.*]], label [[IF_END3936:%.*]]
+// SIMD-ONLY0:       if.then3935:
+// SIMD-ONLY0-NEXT:    [[TMP2054:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2054]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3936]]
+// SIMD-ONLY0:       if.end3936:
+// SIMD-ONLY0-NEXT:    [[TMP2055:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2056:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3937:%.*]] = icmp eq i64 [[TMP2055]], [[TMP2056]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3937]], label [[IF_THEN3939:%.*]], label [[IF_END3940:%.*]]
+// SIMD-ONLY0:       if.then3939:
+// SIMD-ONLY0-NEXT:    [[TMP2057:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2057]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3940]]
+// SIMD-ONLY0:       if.end3940:
+// SIMD-ONLY0-NEXT:    [[TMP2058:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2059:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3941:%.*]] = icmp ugt i64 [[TMP2058]], [[TMP2059]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3941]], label [[COND_TRUE3943:%.*]], label [[COND_FALSE3944:%.*]]
+// SIMD-ONLY0:       cond.true3943:
+// SIMD-ONLY0-NEXT:    [[TMP2060:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3945:%.*]]
+// SIMD-ONLY0:       cond.false3944:
+// SIMD-ONLY0-NEXT:    [[TMP2061:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3945]]
+// SIMD-ONLY0:       cond.end3945:
+// SIMD-ONLY0-NEXT:    [[COND3946:%.*]] = phi i64 [ [[TMP2060]], [[COND_TRUE3943]] ], [ [[TMP2061]], [[COND_FALSE3944]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3946]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2062:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2063:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3947:%.*]] = icmp ult i64 [[TMP2062]], [[TMP2063]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3947]], label [[COND_TRUE3949:%.*]], label [[COND_FALSE3950:%.*]]
+// SIMD-ONLY0:       cond.true3949:
+// SIMD-ONLY0-NEXT:    [[TMP2064:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3951:%.*]]
+// SIMD-ONLY0:       cond.false3950:
+// SIMD-ONLY0-NEXT:    [[TMP2065:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3951]]
+// SIMD-ONLY0:       cond.end3951:
+// SIMD-ONLY0-NEXT:    [[COND3952:%.*]] = phi i64 [ [[TMP2064]], [[COND_TRUE3949]] ], [ [[TMP2065]], [[COND_FALSE3950]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3952]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2066:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2067:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3953:%.*]] = icmp ugt i64 [[TMP2066]], [[TMP2067]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3953]], label [[COND_TRUE3955:%.*]], label [[COND_FALSE3956:%.*]]
+// SIMD-ONLY0:       cond.true3955:
+// SIMD-ONLY0-NEXT:    [[TMP2068:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3957:%.*]]
+// SIMD-ONLY0:       cond.false3956:
+// SIMD-ONLY0-NEXT:    [[TMP2069:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3957]]
+// SIMD-ONLY0:       cond.end3957:
+// SIMD-ONLY0-NEXT:    [[COND3958:%.*]] = phi i64 [ [[TMP2068]], [[COND_TRUE3955]] ], [ [[TMP2069]], [[COND_FALSE3956]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3958]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2070:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2071:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3959:%.*]] = icmp ult i64 [[TMP2070]], [[TMP2071]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3959]], label [[COND_TRUE3961:%.*]], label [[COND_FALSE3962:%.*]]
+// SIMD-ONLY0:       cond.true3961:
+// SIMD-ONLY0-NEXT:    [[TMP2072:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3963:%.*]]
+// SIMD-ONLY0:       cond.false3962:
+// SIMD-ONLY0-NEXT:    [[TMP2073:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3963]]
+// SIMD-ONLY0:       cond.end3963:
+// SIMD-ONLY0-NEXT:    [[COND3964:%.*]] = phi i64 [ [[TMP2072]], [[COND_TRUE3961]] ], [ [[TMP2073]], [[COND_FALSE3962]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3964]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2074:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2075:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3965:%.*]] = icmp ugt i64 [[TMP2074]], [[TMP2075]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3965]], label [[IF_THEN3967:%.*]], label [[IF_END3968:%.*]]
+// SIMD-ONLY0:       if.then3967:
+// SIMD-ONLY0-NEXT:    [[TMP2076:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2076]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3968]]
+// SIMD-ONLY0:       if.end3968:
+// SIMD-ONLY0-NEXT:    [[TMP2077:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2078:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3969:%.*]] = icmp ult i64 [[TMP2077]], [[TMP2078]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3969]], label [[IF_THEN3971:%.*]], label [[IF_END3972:%.*]]
+// SIMD-ONLY0:       if.then3971:
+// SIMD-ONLY0-NEXT:    [[TMP2079:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2079]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3972]]
+// SIMD-ONLY0:       if.end3972:
+// SIMD-ONLY0-NEXT:    [[TMP2080:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2081:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3973:%.*]] = icmp ugt i64 [[TMP2080]], [[TMP2081]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3973]], label [[IF_THEN3975:%.*]], label [[IF_END3976:%.*]]
+// SIMD-ONLY0:       if.then3975:
+// SIMD-ONLY0-NEXT:    [[TMP2082:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2082]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3976]]
+// SIMD-ONLY0:       if.end3976:
+// SIMD-ONLY0-NEXT:    [[TMP2083:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2084:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3977:%.*]] = icmp ult i64 [[TMP2083]], [[TMP2084]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3977]], label [[IF_THEN3979:%.*]], label [[IF_END3980:%.*]]
+// SIMD-ONLY0:       if.then3979:
+// SIMD-ONLY0-NEXT:    [[TMP2085:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2085]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3980]]
+// SIMD-ONLY0:       if.end3980:
+// SIMD-ONLY0-NEXT:    [[TMP2086:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2087:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3981:%.*]] = icmp eq i64 [[TMP2086]], [[TMP2087]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3981]], label [[COND_TRUE3983:%.*]], label [[COND_FALSE3984:%.*]]
+// SIMD-ONLY0:       cond.true3983:
+// SIMD-ONLY0-NEXT:    [[TMP2088:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3985:%.*]]
+// SIMD-ONLY0:       cond.false3984:
+// SIMD-ONLY0-NEXT:    [[TMP2089:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3985]]
+// SIMD-ONLY0:       cond.end3985:
+// SIMD-ONLY0-NEXT:    [[COND3986:%.*]] = phi i64 [ [[TMP2088]], [[COND_TRUE3983]] ], [ [[TMP2089]], [[COND_FALSE3984]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3986]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2090:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2091:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3987:%.*]] = icmp eq i64 [[TMP2090]], [[TMP2091]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3987]], label [[COND_TRUE3989:%.*]], label [[COND_FALSE3990:%.*]]
+// SIMD-ONLY0:       cond.true3989:
+// SIMD-ONLY0-NEXT:    [[TMP2092:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3991:%.*]]
+// SIMD-ONLY0:       cond.false3990:
+// SIMD-ONLY0-NEXT:    [[TMP2093:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END3991]]
+// SIMD-ONLY0:       cond.end3991:
+// SIMD-ONLY0-NEXT:    [[COND3992:%.*]] = phi i64 [ [[TMP2092]], [[COND_TRUE3989]] ], [ [[TMP2093]], [[COND_FALSE3990]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND3992]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2094:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2095:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3993:%.*]] = icmp eq i64 [[TMP2094]], [[TMP2095]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3993]], label [[IF_THEN3995:%.*]], label [[IF_END3996:%.*]]
+// SIMD-ONLY0:       if.then3995:
+// SIMD-ONLY0-NEXT:    [[TMP2096:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2096]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3996]]
+// SIMD-ONLY0:       if.end3996:
+// SIMD-ONLY0-NEXT:    [[TMP2097:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2098:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3997:%.*]] = icmp eq i64 [[TMP2097]], [[TMP2098]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3997]], label [[IF_THEN3999:%.*]], label [[IF_END4000:%.*]]
+// SIMD-ONLY0:       if.then3999:
+// SIMD-ONLY0-NEXT:    [[TMP2099:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2099]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4000]]
+// SIMD-ONLY0:       if.end4000:
+// SIMD-ONLY0-NEXT:    [[TMP2100:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2101:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4001:%.*]] = icmp sgt i64 [[TMP2100]], [[TMP2101]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4001]], label [[COND_TRUE4003:%.*]], label [[COND_FALSE4004:%.*]]
+// SIMD-ONLY0:       cond.true4003:
+// SIMD-ONLY0-NEXT:    [[TMP2102:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4005:%.*]]
+// SIMD-ONLY0:       cond.false4004:
+// SIMD-ONLY0-NEXT:    [[TMP2103:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4005]]
+// SIMD-ONLY0:       cond.end4005:
+// SIMD-ONLY0-NEXT:    [[COND4006:%.*]] = phi i64 [ [[TMP2102]], [[COND_TRUE4003]] ], [ [[TMP2103]], [[COND_FALSE4004]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4006]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2104:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2105:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4007:%.*]] = icmp slt i64 [[TMP2104]], [[TMP2105]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4007]], label [[COND_TRUE4009:%.*]], label [[COND_FALSE4010:%.*]]
+// SIMD-ONLY0:       cond.true4009:
+// SIMD-ONLY0-NEXT:    [[TMP2106:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4011:%.*]]
+// SIMD-ONLY0:       cond.false4010:
+// SIMD-ONLY0-NEXT:    [[TMP2107:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4011]]
+// SIMD-ONLY0:       cond.end4011:
+// SIMD-ONLY0-NEXT:    [[COND4012:%.*]] = phi i64 [ [[TMP2106]], [[COND_TRUE4009]] ], [ [[TMP2107]], [[COND_FALSE4010]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4012]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2108:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2109:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4013:%.*]] = icmp sgt i64 [[TMP2108]], [[TMP2109]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4013]], label [[COND_TRUE4015:%.*]], label [[COND_FALSE4016:%.*]]
+// SIMD-ONLY0:       cond.true4015:
+// SIMD-ONLY0-NEXT:    [[TMP2110:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4017:%.*]]
+// SIMD-ONLY0:       cond.false4016:
+// SIMD-ONLY0-NEXT:    [[TMP2111:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4017]]
+// SIMD-ONLY0:       cond.end4017:
+// SIMD-ONLY0-NEXT:    [[COND4018:%.*]] = phi i64 [ [[TMP2110]], [[COND_TRUE4015]] ], [ [[TMP2111]], [[COND_FALSE4016]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4018]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2112:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2113:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4019:%.*]] = icmp slt i64 [[TMP2112]], [[TMP2113]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4019]], label [[COND_TRUE4021:%.*]], label [[COND_FALSE4022:%.*]]
+// SIMD-ONLY0:       cond.true4021:
+// SIMD-ONLY0-NEXT:    [[TMP2114:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4023:%.*]]
+// SIMD-ONLY0:       cond.false4022:
+// SIMD-ONLY0-NEXT:    [[TMP2115:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4023]]
+// SIMD-ONLY0:       cond.end4023:
+// SIMD-ONLY0-NEXT:    [[COND4024:%.*]] = phi i64 [ [[TMP2114]], [[COND_TRUE4021]] ], [ [[TMP2115]], [[COND_FALSE4022]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4024]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2116:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2117:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4025:%.*]] = icmp sgt i64 [[TMP2116]], [[TMP2117]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4025]], label [[IF_THEN4027:%.*]], label [[IF_END4028:%.*]]
+// SIMD-ONLY0:       if.then4027:
+// SIMD-ONLY0-NEXT:    [[TMP2118:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2118]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4028]]
+// SIMD-ONLY0:       if.end4028:
+// SIMD-ONLY0-NEXT:    [[TMP2119:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2120:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4029:%.*]] = icmp slt i64 [[TMP2119]], [[TMP2120]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4029]], label [[IF_THEN4031:%.*]], label [[IF_END4032:%.*]]
+// SIMD-ONLY0:       if.then4031:
+// SIMD-ONLY0-NEXT:    [[TMP2121:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2121]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4032]]
+// SIMD-ONLY0:       if.end4032:
+// SIMD-ONLY0-NEXT:    [[TMP2122:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2123:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4033:%.*]] = icmp sgt i64 [[TMP2122]], [[TMP2123]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4033]], label [[IF_THEN4035:%.*]], label [[IF_END4036:%.*]]
+// SIMD-ONLY0:       if.then4035:
+// SIMD-ONLY0-NEXT:    [[TMP2124:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2124]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4036]]
+// SIMD-ONLY0:       if.end4036:
+// SIMD-ONLY0-NEXT:    [[TMP2125:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2126:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4037:%.*]] = icmp slt i64 [[TMP2125]], [[TMP2126]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4037]], label [[IF_THEN4039:%.*]], label [[IF_END4040:%.*]]
+// SIMD-ONLY0:       if.then4039:
+// SIMD-ONLY0-NEXT:    [[TMP2127:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2127]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4040]]
+// SIMD-ONLY0:       if.end4040:
+// SIMD-ONLY0-NEXT:    [[TMP2128:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2129:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4041:%.*]] = icmp eq i64 [[TMP2128]], [[TMP2129]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4041]], label [[COND_TRUE4043:%.*]], label [[COND_FALSE4044:%.*]]
+// SIMD-ONLY0:       cond.true4043:
+// SIMD-ONLY0-NEXT:    [[TMP2130:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4045:%.*]]
+// SIMD-ONLY0:       cond.false4044:
+// SIMD-ONLY0-NEXT:    [[TMP2131:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4045]]
+// SIMD-ONLY0:       cond.end4045:
+// SIMD-ONLY0-NEXT:    [[COND4046:%.*]] = phi i64 [ [[TMP2130]], [[COND_TRUE4043]] ], [ [[TMP2131]], [[COND_FALSE4044]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4046]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2132:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2133:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4047:%.*]] = icmp eq i64 [[TMP2132]], [[TMP2133]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4047]], label [[COND_TRUE4049:%.*]], label [[COND_FALSE4050:%.*]]
+// SIMD-ONLY0:       cond.true4049:
+// SIMD-ONLY0-NEXT:    [[TMP2134:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4051:%.*]]
+// SIMD-ONLY0:       cond.false4050:
+// SIMD-ONLY0-NEXT:    [[TMP2135:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4051]]
+// SIMD-ONLY0:       cond.end4051:
+// SIMD-ONLY0-NEXT:    [[COND4052:%.*]] = phi i64 [ [[TMP2134]], [[COND_TRUE4049]] ], [ [[TMP2135]], [[COND_FALSE4050]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4052]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2136:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2137:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4053:%.*]] = icmp eq i64 [[TMP2136]], [[TMP2137]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4053]], label [[IF_THEN4055:%.*]], label [[IF_END4056:%.*]]
+// SIMD-ONLY0:       if.then4055:
+// SIMD-ONLY0-NEXT:    [[TMP2138:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2138]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4056]]
+// SIMD-ONLY0:       if.end4056:
+// SIMD-ONLY0-NEXT:    [[TMP2139:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2140:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4057:%.*]] = icmp eq i64 [[TMP2139]], [[TMP2140]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4057]], label [[IF_THEN4059:%.*]], label [[IF_END4060:%.*]]
+// SIMD-ONLY0:       if.then4059:
+// SIMD-ONLY0-NEXT:    [[TMP2141:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2141]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4060]]
+// SIMD-ONLY0:       if.end4060:
+// SIMD-ONLY0-NEXT:    [[TMP2142:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2143:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4061:%.*]] = icmp ugt i64 [[TMP2142]], [[TMP2143]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4061]], label [[COND_TRUE4063:%.*]], label [[COND_FALSE4064:%.*]]
+// SIMD-ONLY0:       cond.true4063:
+// SIMD-ONLY0-NEXT:    [[TMP2144:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4065:%.*]]
+// SIMD-ONLY0:       cond.false4064:
+// SIMD-ONLY0-NEXT:    [[TMP2145:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4065]]
+// SIMD-ONLY0:       cond.end4065:
+// SIMD-ONLY0-NEXT:    [[COND4066:%.*]] = phi i64 [ [[TMP2144]], [[COND_TRUE4063]] ], [ [[TMP2145]], [[COND_FALSE4064]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4066]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2146:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2147:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4067:%.*]] = icmp ult i64 [[TMP2146]], [[TMP2147]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4067]], label [[COND_TRUE4069:%.*]], label [[COND_FALSE4070:%.*]]
+// SIMD-ONLY0:       cond.true4069:
+// SIMD-ONLY0-NEXT:    [[TMP2148:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4071:%.*]]
+// SIMD-ONLY0:       cond.false4070:
+// SIMD-ONLY0-NEXT:    [[TMP2149:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4071]]
+// SIMD-ONLY0:       cond.end4071:
+// SIMD-ONLY0-NEXT:    [[COND4072:%.*]] = phi i64 [ [[TMP2148]], [[COND_TRUE4069]] ], [ [[TMP2149]], [[COND_FALSE4070]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4072]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2150:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2151:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4073:%.*]] = icmp ugt i64 [[TMP2150]], [[TMP2151]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4073]], label [[COND_TRUE4075:%.*]], label [[COND_FALSE4076:%.*]]
+// SIMD-ONLY0:       cond.true4075:
+// SIMD-ONLY0-NEXT:    [[TMP2152:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4077:%.*]]
+// SIMD-ONLY0:       cond.false4076:
+// SIMD-ONLY0-NEXT:    [[TMP2153:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4077]]
+// SIMD-ONLY0:       cond.end4077:
+// SIMD-ONLY0-NEXT:    [[COND4078:%.*]] = phi i64 [ [[TMP2152]], [[COND_TRUE4075]] ], [ [[TMP2153]], [[COND_FALSE4076]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4078]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2154:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2155:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4079:%.*]] = icmp ult i64 [[TMP2154]], [[TMP2155]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4079]], label [[COND_TRUE4081:%.*]], label [[COND_FALSE4082:%.*]]
+// SIMD-ONLY0:       cond.true4081:
+// SIMD-ONLY0-NEXT:    [[TMP2156:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4083:%.*]]
+// SIMD-ONLY0:       cond.false4082:
+// SIMD-ONLY0-NEXT:    [[TMP2157:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4083]]
+// SIMD-ONLY0:       cond.end4083:
+// SIMD-ONLY0-NEXT:    [[COND4084:%.*]] = phi i64 [ [[TMP2156]], [[COND_TRUE4081]] ], [ [[TMP2157]], [[COND_FALSE4082]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4084]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2158:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2159:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4085:%.*]] = icmp ugt i64 [[TMP2158]], [[TMP2159]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4085]], label [[IF_THEN4087:%.*]], label [[IF_END4088:%.*]]
+// SIMD-ONLY0:       if.then4087:
+// SIMD-ONLY0-NEXT:    [[TMP2160:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2160]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4088]]
+// SIMD-ONLY0:       if.end4088:
+// SIMD-ONLY0-NEXT:    [[TMP2161:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2162:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4089:%.*]] = icmp ult i64 [[TMP2161]], [[TMP2162]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4089]], label [[IF_THEN4091:%.*]], label [[IF_END4092:%.*]]
+// SIMD-ONLY0:       if.then4091:
+// SIMD-ONLY0-NEXT:    [[TMP2163:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2163]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4092]]
+// SIMD-ONLY0:       if.end4092:
+// SIMD-ONLY0-NEXT:    [[TMP2164:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2165:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4093:%.*]] = icmp ugt i64 [[TMP2164]], [[TMP2165]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4093]], label [[IF_THEN4095:%.*]], label [[IF_END4096:%.*]]
+// SIMD-ONLY0:       if.then4095:
+// SIMD-ONLY0-NEXT:    [[TMP2166:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2166]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4096]]
+// SIMD-ONLY0:       if.end4096:
+// SIMD-ONLY0-NEXT:    [[TMP2167:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2168:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4097:%.*]] = icmp ult i64 [[TMP2167]], [[TMP2168]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4097]], label [[IF_THEN4099:%.*]], label [[IF_END4100:%.*]]
+// SIMD-ONLY0:       if.then4099:
+// SIMD-ONLY0-NEXT:    [[TMP2169:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2169]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4100]]
+// SIMD-ONLY0:       if.end4100:
+// SIMD-ONLY0-NEXT:    [[TMP2170:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2171:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4101:%.*]] = icmp eq i64 [[TMP2170]], [[TMP2171]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4101]], label [[COND_TRUE4103:%.*]], label [[COND_FALSE4104:%.*]]
+// SIMD-ONLY0:       cond.true4103:
+// SIMD-ONLY0-NEXT:    [[TMP2172:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4105:%.*]]
+// SIMD-ONLY0:       cond.false4104:
+// SIMD-ONLY0-NEXT:    [[TMP2173:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4105]]
+// SIMD-ONLY0:       cond.end4105:
+// SIMD-ONLY0-NEXT:    [[COND4106:%.*]] = phi i64 [ [[TMP2172]], [[COND_TRUE4103]] ], [ [[TMP2173]], [[COND_FALSE4104]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4106]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2174:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2175:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4107:%.*]] = icmp eq i64 [[TMP2174]], [[TMP2175]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4107]], label [[COND_TRUE4109:%.*]], label [[COND_FALSE4110:%.*]]
+// SIMD-ONLY0:       cond.true4109:
+// SIMD-ONLY0-NEXT:    [[TMP2176:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4111:%.*]]
+// SIMD-ONLY0:       cond.false4110:
+// SIMD-ONLY0-NEXT:    [[TMP2177:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4111]]
+// SIMD-ONLY0:       cond.end4111:
+// SIMD-ONLY0-NEXT:    [[COND4112:%.*]] = phi i64 [ [[TMP2176]], [[COND_TRUE4109]] ], [ [[TMP2177]], [[COND_FALSE4110]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4112]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2178:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2179:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4113:%.*]] = icmp eq i64 [[TMP2178]], [[TMP2179]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4113]], label [[IF_THEN4115:%.*]], label [[IF_END4116:%.*]]
+// SIMD-ONLY0:       if.then4115:
+// SIMD-ONLY0-NEXT:    [[TMP2180:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2180]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4116]]
+// SIMD-ONLY0:       if.end4116:
+// SIMD-ONLY0-NEXT:    [[TMP2181:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2182:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4117:%.*]] = icmp eq i64 [[TMP2181]], [[TMP2182]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4117]], label [[IF_THEN4119:%.*]], label [[IF_END4120:%.*]]
+// SIMD-ONLY0:       if.then4119:
+// SIMD-ONLY0-NEXT:    [[TMP2183:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2183]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4120]]
+// SIMD-ONLY0:       if.end4120:
+// SIMD-ONLY0-NEXT:    [[TMP2184:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2185:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4121:%.*]] = icmp sgt i64 [[TMP2184]], [[TMP2185]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4121]], label [[COND_TRUE4123:%.*]], label [[COND_FALSE4124:%.*]]
+// SIMD-ONLY0:       cond.true4123:
+// SIMD-ONLY0-NEXT:    [[TMP2186:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4125:%.*]]
+// SIMD-ONLY0:       cond.false4124:
+// SIMD-ONLY0-NEXT:    [[TMP2187:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4125]]
+// SIMD-ONLY0:       cond.end4125:
+// SIMD-ONLY0-NEXT:    [[COND4126:%.*]] = phi i64 [ [[TMP2186]], [[COND_TRUE4123]] ], [ [[TMP2187]], [[COND_FALSE4124]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4126]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2188:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2189:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4127:%.*]] = icmp slt i64 [[TMP2188]], [[TMP2189]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4127]], label [[COND_TRUE4129:%.*]], label [[COND_FALSE4130:%.*]]
+// SIMD-ONLY0:       cond.true4129:
+// SIMD-ONLY0-NEXT:    [[TMP2190:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4131:%.*]]
+// SIMD-ONLY0:       cond.false4130:
+// SIMD-ONLY0-NEXT:    [[TMP2191:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4131]]
+// SIMD-ONLY0:       cond.end4131:
+// SIMD-ONLY0-NEXT:    [[COND4132:%.*]] = phi i64 [ [[TMP2190]], [[COND_TRUE4129]] ], [ [[TMP2191]], [[COND_FALSE4130]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4132]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2192:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2193:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4133:%.*]] = icmp sgt i64 [[TMP2192]], [[TMP2193]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4133]], label [[COND_TRUE4135:%.*]], label [[COND_FALSE4136:%.*]]
+// SIMD-ONLY0:       cond.true4135:
+// SIMD-ONLY0-NEXT:    [[TMP2194:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4137:%.*]]
+// SIMD-ONLY0:       cond.false4136:
+// SIMD-ONLY0-NEXT:    [[TMP2195:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4137]]
+// SIMD-ONLY0:       cond.end4137:
+// SIMD-ONLY0-NEXT:    [[COND4138:%.*]] = phi i64 [ [[TMP2194]], [[COND_TRUE4135]] ], [ [[TMP2195]], [[COND_FALSE4136]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4138]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2196:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2197:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4139:%.*]] = icmp slt i64 [[TMP2196]], [[TMP2197]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4139]], label [[COND_TRUE4141:%.*]], label [[COND_FALSE4142:%.*]]
+// SIMD-ONLY0:       cond.true4141:
+// SIMD-ONLY0-NEXT:    [[TMP2198:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4143:%.*]]
+// SIMD-ONLY0:       cond.false4142:
+// SIMD-ONLY0-NEXT:    [[TMP2199:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4143]]
+// SIMD-ONLY0:       cond.end4143:
+// SIMD-ONLY0-NEXT:    [[COND4144:%.*]] = phi i64 [ [[TMP2198]], [[COND_TRUE4141]] ], [ [[TMP2199]], [[COND_FALSE4142]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4144]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2200:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2201:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4145:%.*]] = icmp sgt i64 [[TMP2200]], [[TMP2201]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4145]], label [[IF_THEN4147:%.*]], label [[IF_END4148:%.*]]
+// SIMD-ONLY0:       if.then4147:
+// SIMD-ONLY0-NEXT:    [[TMP2202:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2202]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4148]]
+// SIMD-ONLY0:       if.end4148:
+// SIMD-ONLY0-NEXT:    [[TMP2203:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2204:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4149:%.*]] = icmp slt i64 [[TMP2203]], [[TMP2204]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4149]], label [[IF_THEN4151:%.*]], label [[IF_END4152:%.*]]
+// SIMD-ONLY0:       if.then4151:
+// SIMD-ONLY0-NEXT:    [[TMP2205:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2205]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4152]]
+// SIMD-ONLY0:       if.end4152:
+// SIMD-ONLY0-NEXT:    [[TMP2206:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2207:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4153:%.*]] = icmp sgt i64 [[TMP2206]], [[TMP2207]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4153]], label [[IF_THEN4155:%.*]], label [[IF_END4156:%.*]]
+// SIMD-ONLY0:       if.then4155:
+// SIMD-ONLY0-NEXT:    [[TMP2208:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2208]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4156]]
+// SIMD-ONLY0:       if.end4156:
+// SIMD-ONLY0-NEXT:    [[TMP2209:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2210:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4157:%.*]] = icmp slt i64 [[TMP2209]], [[TMP2210]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4157]], label [[IF_THEN4159:%.*]], label [[IF_END4160:%.*]]
+// SIMD-ONLY0:       if.then4159:
+// SIMD-ONLY0-NEXT:    [[TMP2211:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2211]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4160]]
+// SIMD-ONLY0:       if.end4160:
+// SIMD-ONLY0-NEXT:    [[TMP2212:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2213:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4161:%.*]] = icmp eq i64 [[TMP2212]], [[TMP2213]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4161]], label [[COND_TRUE4163:%.*]], label [[COND_FALSE4164:%.*]]
+// SIMD-ONLY0:       cond.true4163:
+// SIMD-ONLY0-NEXT:    [[TMP2214:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4165:%.*]]
+// SIMD-ONLY0:       cond.false4164:
+// SIMD-ONLY0-NEXT:    [[TMP2215:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4165]]
+// SIMD-ONLY0:       cond.end4165:
+// SIMD-ONLY0-NEXT:    [[COND4166:%.*]] = phi i64 [ [[TMP2214]], [[COND_TRUE4163]] ], [ [[TMP2215]], [[COND_FALSE4164]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4166]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2216:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2217:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4167:%.*]] = icmp eq i64 [[TMP2216]], [[TMP2217]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4167]], label [[COND_TRUE4169:%.*]], label [[COND_FALSE4170:%.*]]
+// SIMD-ONLY0:       cond.true4169:
+// SIMD-ONLY0-NEXT:    [[TMP2218:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4171:%.*]]
+// SIMD-ONLY0:       cond.false4170:
+// SIMD-ONLY0-NEXT:    [[TMP2219:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4171]]
+// SIMD-ONLY0:       cond.end4171:
+// SIMD-ONLY0-NEXT:    [[COND4172:%.*]] = phi i64 [ [[TMP2218]], [[COND_TRUE4169]] ], [ [[TMP2219]], [[COND_FALSE4170]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4172]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2220:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2221:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4173:%.*]] = icmp eq i64 [[TMP2220]], [[TMP2221]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4173]], label [[IF_THEN4175:%.*]], label [[IF_END4176:%.*]]
+// SIMD-ONLY0:       if.then4175:
+// SIMD-ONLY0-NEXT:    [[TMP2222:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2222]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4176]]
+// SIMD-ONLY0:       if.end4176:
+// SIMD-ONLY0-NEXT:    [[TMP2223:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2224:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4177:%.*]] = icmp eq i64 [[TMP2223]], [[TMP2224]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4177]], label [[IF_THEN4179:%.*]], label [[IF_END4180:%.*]]
+// SIMD-ONLY0:       if.then4179:
+// SIMD-ONLY0-NEXT:    [[TMP2225:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2225]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4180]]
+// SIMD-ONLY0:       if.end4180:
+// SIMD-ONLY0-NEXT:    [[TMP2226:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2227:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4181:%.*]] = icmp ugt i64 [[TMP2226]], [[TMP2227]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4181]], label [[COND_TRUE4183:%.*]], label [[COND_FALSE4184:%.*]]
+// SIMD-ONLY0:       cond.true4183:
+// SIMD-ONLY0-NEXT:    [[TMP2228:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4185:%.*]]
+// SIMD-ONLY0:       cond.false4184:
+// SIMD-ONLY0-NEXT:    [[TMP2229:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4185]]
+// SIMD-ONLY0:       cond.end4185:
+// SIMD-ONLY0-NEXT:    [[COND4186:%.*]] = phi i64 [ [[TMP2228]], [[COND_TRUE4183]] ], [ [[TMP2229]], [[COND_FALSE4184]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4186]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2230:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2231:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4187:%.*]] = icmp ult i64 [[TMP2230]], [[TMP2231]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4187]], label [[COND_TRUE4189:%.*]], label [[COND_FALSE4190:%.*]]
+// SIMD-ONLY0:       cond.true4189:
+// SIMD-ONLY0-NEXT:    [[TMP2232:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4191:%.*]]
+// SIMD-ONLY0:       cond.false4190:
+// SIMD-ONLY0-NEXT:    [[TMP2233:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4191]]
+// SIMD-ONLY0:       cond.end4191:
+// SIMD-ONLY0-NEXT:    [[COND4192:%.*]] = phi i64 [ [[TMP2232]], [[COND_TRUE4189]] ], [ [[TMP2233]], [[COND_FALSE4190]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4192]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2234:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2235:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4193:%.*]] = icmp ugt i64 [[TMP2234]], [[TMP2235]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4193]], label [[COND_TRUE4195:%.*]], label [[COND_FALSE4196:%.*]]
+// SIMD-ONLY0:       cond.true4195:
+// SIMD-ONLY0-NEXT:    [[TMP2236:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4197:%.*]]
+// SIMD-ONLY0:       cond.false4196:
+// SIMD-ONLY0-NEXT:    [[TMP2237:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4197]]
+// SIMD-ONLY0:       cond.end4197:
+// SIMD-ONLY0-NEXT:    [[COND4198:%.*]] = phi i64 [ [[TMP2236]], [[COND_TRUE4195]] ], [ [[TMP2237]], [[COND_FALSE4196]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4198]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2238:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2239:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4199:%.*]] = icmp ult i64 [[TMP2238]], [[TMP2239]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4199]], label [[COND_TRUE4201:%.*]], label [[COND_FALSE4202:%.*]]
+// SIMD-ONLY0:       cond.true4201:
+// SIMD-ONLY0-NEXT:    [[TMP2240:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4203:%.*]]
+// SIMD-ONLY0:       cond.false4202:
+// SIMD-ONLY0-NEXT:    [[TMP2241:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4203]]
+// SIMD-ONLY0:       cond.end4203:
+// SIMD-ONLY0-NEXT:    [[COND4204:%.*]] = phi i64 [ [[TMP2240]], [[COND_TRUE4201]] ], [ [[TMP2241]], [[COND_FALSE4202]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4204]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2242:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2243:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4205:%.*]] = icmp ugt i64 [[TMP2242]], [[TMP2243]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4205]], label [[IF_THEN4207:%.*]], label [[IF_END4208:%.*]]
+// SIMD-ONLY0:       if.then4207:
+// SIMD-ONLY0-NEXT:    [[TMP2244:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2244]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4208]]
+// SIMD-ONLY0:       if.end4208:
+// SIMD-ONLY0-NEXT:    [[TMP2245:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2246:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4209:%.*]] = icmp ult i64 [[TMP2245]], [[TMP2246]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4209]], label [[IF_THEN4211:%.*]], label [[IF_END4212:%.*]]
+// SIMD-ONLY0:       if.then4211:
+// SIMD-ONLY0-NEXT:    [[TMP2247:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2247]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4212]]
+// SIMD-ONLY0:       if.end4212:
+// SIMD-ONLY0-NEXT:    [[TMP2248:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2249:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4213:%.*]] = icmp ugt i64 [[TMP2248]], [[TMP2249]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4213]], label [[IF_THEN4215:%.*]], label [[IF_END4216:%.*]]
+// SIMD-ONLY0:       if.then4215:
+// SIMD-ONLY0-NEXT:    [[TMP2250:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2250]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4216]]
+// SIMD-ONLY0:       if.end4216:
+// SIMD-ONLY0-NEXT:    [[TMP2251:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2252:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4217:%.*]] = icmp ult i64 [[TMP2251]], [[TMP2252]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4217]], label [[IF_THEN4219:%.*]], label [[IF_END4220:%.*]]
+// SIMD-ONLY0:       if.then4219:
+// SIMD-ONLY0-NEXT:    [[TMP2253:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2253]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4220]]
+// SIMD-ONLY0:       if.end4220:
+// SIMD-ONLY0-NEXT:    [[TMP2254:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2255:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4221:%.*]] = icmp eq i64 [[TMP2254]], [[TMP2255]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4221]], label [[COND_TRUE4223:%.*]], label [[COND_FALSE4224:%.*]]
+// SIMD-ONLY0:       cond.true4223:
+// SIMD-ONLY0-NEXT:    [[TMP2256:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4225:%.*]]
+// SIMD-ONLY0:       cond.false4224:
+// SIMD-ONLY0-NEXT:    [[TMP2257:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4225]]
+// SIMD-ONLY0:       cond.end4225:
+// SIMD-ONLY0-NEXT:    [[COND4226:%.*]] = phi i64 [ [[TMP2256]], [[COND_TRUE4223]] ], [ [[TMP2257]], [[COND_FALSE4224]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4226]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2258:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2259:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4227:%.*]] = icmp eq i64 [[TMP2258]], [[TMP2259]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4227]], label [[COND_TRUE4229:%.*]], label [[COND_FALSE4230:%.*]]
+// SIMD-ONLY0:       cond.true4229:
+// SIMD-ONLY0-NEXT:    [[TMP2260:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4231:%.*]]
+// SIMD-ONLY0:       cond.false4230:
+// SIMD-ONLY0-NEXT:    [[TMP2261:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4231]]
+// SIMD-ONLY0:       cond.end4231:
+// SIMD-ONLY0-NEXT:    [[COND4232:%.*]] = phi i64 [ [[TMP2260]], [[COND_TRUE4229]] ], [ [[TMP2261]], [[COND_FALSE4230]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4232]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2262:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2263:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4233:%.*]] = icmp eq i64 [[TMP2262]], [[TMP2263]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4233]], label [[IF_THEN4235:%.*]], label [[IF_END4236:%.*]]
+// SIMD-ONLY0:       if.then4235:
+// SIMD-ONLY0-NEXT:    [[TMP2264:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2264]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4236]]
+// SIMD-ONLY0:       if.end4236:
+// SIMD-ONLY0-NEXT:    [[TMP2265:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2266:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4237:%.*]] = icmp eq i64 [[TMP2265]], [[TMP2266]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4237]], label [[IF_THEN4239:%.*]], label [[IF_END4240:%.*]]
+// SIMD-ONLY0:       if.then4239:
+// SIMD-ONLY0-NEXT:    [[TMP2267:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2267]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4240]]
+// SIMD-ONLY0:       if.end4240:
+// SIMD-ONLY0-NEXT:    [[TMP2268:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2269:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4241:%.*]] = icmp sgt i64 [[TMP2268]], [[TMP2269]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4241]], label [[COND_TRUE4243:%.*]], label [[COND_FALSE4244:%.*]]
+// SIMD-ONLY0:       cond.true4243:
+// SIMD-ONLY0-NEXT:    [[TMP2270:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4245:%.*]]
+// SIMD-ONLY0:       cond.false4244:
+// SIMD-ONLY0-NEXT:    [[TMP2271:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4245]]
+// SIMD-ONLY0:       cond.end4245:
+// SIMD-ONLY0-NEXT:    [[COND4246:%.*]] = phi i64 [ [[TMP2270]], [[COND_TRUE4243]] ], [ [[TMP2271]], [[COND_FALSE4244]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4246]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2272:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2273:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4247:%.*]] = icmp slt i64 [[TMP2272]], [[TMP2273]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4247]], label [[COND_TRUE4249:%.*]], label [[COND_FALSE4250:%.*]]
+// SIMD-ONLY0:       cond.true4249:
+// SIMD-ONLY0-NEXT:    [[TMP2274:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4251:%.*]]
+// SIMD-ONLY0:       cond.false4250:
+// SIMD-ONLY0-NEXT:    [[TMP2275:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4251]]
+// SIMD-ONLY0:       cond.end4251:
+// SIMD-ONLY0-NEXT:    [[COND4252:%.*]] = phi i64 [ [[TMP2274]], [[COND_TRUE4249]] ], [ [[TMP2275]], [[COND_FALSE4250]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4252]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2276:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2277:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4253:%.*]] = icmp sgt i64 [[TMP2276]], [[TMP2277]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4253]], label [[COND_TRUE4255:%.*]], label [[COND_FALSE4256:%.*]]
+// SIMD-ONLY0:       cond.true4255:
+// SIMD-ONLY0-NEXT:    [[TMP2278:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4257:%.*]]
+// SIMD-ONLY0:       cond.false4256:
+// SIMD-ONLY0-NEXT:    [[TMP2279:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4257]]
+// SIMD-ONLY0:       cond.end4257:
+// SIMD-ONLY0-NEXT:    [[COND4258:%.*]] = phi i64 [ [[TMP2278]], [[COND_TRUE4255]] ], [ [[TMP2279]], [[COND_FALSE4256]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4258]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2280:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2281:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4259:%.*]] = icmp slt i64 [[TMP2280]], [[TMP2281]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4259]], label [[COND_TRUE4261:%.*]], label [[COND_FALSE4262:%.*]]
+// SIMD-ONLY0:       cond.true4261:
+// SIMD-ONLY0-NEXT:    [[TMP2282:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4263:%.*]]
+// SIMD-ONLY0:       cond.false4262:
+// SIMD-ONLY0-NEXT:    [[TMP2283:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4263]]
+// SIMD-ONLY0:       cond.end4263:
+// SIMD-ONLY0-NEXT:    [[COND4264:%.*]] = phi i64 [ [[TMP2282]], [[COND_TRUE4261]] ], [ [[TMP2283]], [[COND_FALSE4262]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4264]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2284:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2285:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4265:%.*]] = icmp sgt i64 [[TMP2284]], [[TMP2285]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4265]], label [[IF_THEN4267:%.*]], label [[IF_END4268:%.*]]
+// SIMD-ONLY0:       if.then4267:
+// SIMD-ONLY0-NEXT:    [[TMP2286:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2286]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4268]]
+// SIMD-ONLY0:       if.end4268:
+// SIMD-ONLY0-NEXT:    [[TMP2287:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2288:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4269:%.*]] = icmp slt i64 [[TMP2287]], [[TMP2288]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4269]], label [[IF_THEN4271:%.*]], label [[IF_END4272:%.*]]
+// SIMD-ONLY0:       if.then4271:
+// SIMD-ONLY0-NEXT:    [[TMP2289:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2289]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4272]]
+// SIMD-ONLY0:       if.end4272:
+// SIMD-ONLY0-NEXT:    [[TMP2290:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2291:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4273:%.*]] = icmp sgt i64 [[TMP2290]], [[TMP2291]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4273]], label [[IF_THEN4275:%.*]], label [[IF_END4276:%.*]]
+// SIMD-ONLY0:       if.then4275:
+// SIMD-ONLY0-NEXT:    [[TMP2292:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2292]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4276]]
+// SIMD-ONLY0:       if.end4276:
+// SIMD-ONLY0-NEXT:    [[TMP2293:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2294:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4277:%.*]] = icmp slt i64 [[TMP2293]], [[TMP2294]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4277]], label [[IF_THEN4279:%.*]], label [[IF_END4280:%.*]]
+// SIMD-ONLY0:       if.then4279:
+// SIMD-ONLY0-NEXT:    [[TMP2295:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2295]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4280]]
+// SIMD-ONLY0:       if.end4280:
+// SIMD-ONLY0-NEXT:    [[TMP2296:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2297:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4281:%.*]] = icmp eq i64 [[TMP2296]], [[TMP2297]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4281]], label [[COND_TRUE4283:%.*]], label [[COND_FALSE4284:%.*]]
+// SIMD-ONLY0:       cond.true4283:
+// SIMD-ONLY0-NEXT:    [[TMP2298:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4285:%.*]]
+// SIMD-ONLY0:       cond.false4284:
+// SIMD-ONLY0-NEXT:    [[TMP2299:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4285]]
+// SIMD-ONLY0:       cond.end4285:
+// SIMD-ONLY0-NEXT:    [[COND4286:%.*]] = phi i64 [ [[TMP2298]], [[COND_TRUE4283]] ], [ [[TMP2299]], [[COND_FALSE4284]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4286]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2300:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2301:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4287:%.*]] = icmp eq i64 [[TMP2300]], [[TMP2301]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4287]], label [[COND_TRUE4289:%.*]], label [[COND_FALSE4290:%.*]]
+// SIMD-ONLY0:       cond.true4289:
+// SIMD-ONLY0-NEXT:    [[TMP2302:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4291:%.*]]
+// SIMD-ONLY0:       cond.false4290:
+// SIMD-ONLY0-NEXT:    [[TMP2303:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4291]]
+// SIMD-ONLY0:       cond.end4291:
+// SIMD-ONLY0-NEXT:    [[COND4292:%.*]] = phi i64 [ [[TMP2302]], [[COND_TRUE4289]] ], [ [[TMP2303]], [[COND_FALSE4290]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4292]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2304:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2305:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4293:%.*]] = icmp eq i64 [[TMP2304]], [[TMP2305]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4293]], label [[IF_THEN4295:%.*]], label [[IF_END4296:%.*]]
+// SIMD-ONLY0:       if.then4295:
+// SIMD-ONLY0-NEXT:    [[TMP2306:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2306]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4296]]
+// SIMD-ONLY0:       if.end4296:
+// SIMD-ONLY0-NEXT:    [[TMP2307:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2308:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4297:%.*]] = icmp eq i64 [[TMP2307]], [[TMP2308]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4297]], label [[IF_THEN4299:%.*]], label [[IF_END4300:%.*]]
+// SIMD-ONLY0:       if.then4299:
+// SIMD-ONLY0-NEXT:    [[TMP2309:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2309]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4300]]
+// SIMD-ONLY0:       if.end4300:
+// SIMD-ONLY0-NEXT:    [[TMP2310:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2311:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4301:%.*]] = icmp ugt i64 [[TMP2310]], [[TMP2311]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4301]], label [[COND_TRUE4303:%.*]], label [[COND_FALSE4304:%.*]]
+// SIMD-ONLY0:       cond.true4303:
+// SIMD-ONLY0-NEXT:    [[TMP2312:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4305:%.*]]
+// SIMD-ONLY0:       cond.false4304:
+// SIMD-ONLY0-NEXT:    [[TMP2313:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4305]]
+// SIMD-ONLY0:       cond.end4305:
+// SIMD-ONLY0-NEXT:    [[COND4306:%.*]] = phi i64 [ [[TMP2312]], [[COND_TRUE4303]] ], [ [[TMP2313]], [[COND_FALSE4304]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4306]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2314:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2315:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4307:%.*]] = icmp ult i64 [[TMP2314]], [[TMP2315]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4307]], label [[COND_TRUE4309:%.*]], label [[COND_FALSE4310:%.*]]
+// SIMD-ONLY0:       cond.true4309:
+// SIMD-ONLY0-NEXT:    [[TMP2316:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4311:%.*]]
+// SIMD-ONLY0:       cond.false4310:
+// SIMD-ONLY0-NEXT:    [[TMP2317:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4311]]
+// SIMD-ONLY0:       cond.end4311:
+// SIMD-ONLY0-NEXT:    [[COND4312:%.*]] = phi i64 [ [[TMP2316]], [[COND_TRUE4309]] ], [ [[TMP2317]], [[COND_FALSE4310]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4312]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2318:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2319:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4313:%.*]] = icmp ugt i64 [[TMP2318]], [[TMP2319]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4313]], label [[COND_TRUE4315:%.*]], label [[COND_FALSE4316:%.*]]
+// SIMD-ONLY0:       cond.true4315:
+// SIMD-ONLY0-NEXT:    [[TMP2320:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4317:%.*]]
+// SIMD-ONLY0:       cond.false4316:
+// SIMD-ONLY0-NEXT:    [[TMP2321:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4317]]
+// SIMD-ONLY0:       cond.end4317:
+// SIMD-ONLY0-NEXT:    [[COND4318:%.*]] = phi i64 [ [[TMP2320]], [[COND_TRUE4315]] ], [ [[TMP2321]], [[COND_FALSE4316]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4318]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2322:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2323:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4319:%.*]] = icmp ult i64 [[TMP2322]], [[TMP2323]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4319]], label [[COND_TRUE4321:%.*]], label [[COND_FALSE4322:%.*]]
+// SIMD-ONLY0:       cond.true4321:
+// SIMD-ONLY0-NEXT:    [[TMP2324:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4323:%.*]]
+// SIMD-ONLY0:       cond.false4322:
+// SIMD-ONLY0-NEXT:    [[TMP2325:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4323]]
+// SIMD-ONLY0:       cond.end4323:
+// SIMD-ONLY0-NEXT:    [[COND4324:%.*]] = phi i64 [ [[TMP2324]], [[COND_TRUE4321]] ], [ [[TMP2325]], [[COND_FALSE4322]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4324]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2326:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2327:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4325:%.*]] = icmp ugt i64 [[TMP2326]], [[TMP2327]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4325]], label [[IF_THEN4327:%.*]], label [[IF_END4328:%.*]]
+// SIMD-ONLY0:       if.then4327:
+// SIMD-ONLY0-NEXT:    [[TMP2328:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2328]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4328]]
+// SIMD-ONLY0:       if.end4328:
+// SIMD-ONLY0-NEXT:    [[TMP2329:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2330:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4329:%.*]] = icmp ult i64 [[TMP2329]], [[TMP2330]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4329]], label [[IF_THEN4331:%.*]], label [[IF_END4332:%.*]]
+// SIMD-ONLY0:       if.then4331:
+// SIMD-ONLY0-NEXT:    [[TMP2331:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2331]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4332]]
+// SIMD-ONLY0:       if.end4332:
+// SIMD-ONLY0-NEXT:    [[TMP2332:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2333:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4333:%.*]] = icmp ugt i64 [[TMP2332]], [[TMP2333]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4333]], label [[IF_THEN4335:%.*]], label [[IF_END4336:%.*]]
+// SIMD-ONLY0:       if.then4335:
+// SIMD-ONLY0-NEXT:    [[TMP2334:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2334]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4336]]
+// SIMD-ONLY0:       if.end4336:
+// SIMD-ONLY0-NEXT:    [[TMP2335:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2336:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4337:%.*]] = icmp ult i64 [[TMP2335]], [[TMP2336]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4337]], label [[IF_THEN4339:%.*]], label [[IF_END4340:%.*]]
+// SIMD-ONLY0:       if.then4339:
+// SIMD-ONLY0-NEXT:    [[TMP2337:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2337]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4340]]
+// SIMD-ONLY0:       if.end4340:
+// SIMD-ONLY0-NEXT:    [[TMP2338:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2339:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4341:%.*]] = icmp eq i64 [[TMP2338]], [[TMP2339]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4341]], label [[COND_TRUE4343:%.*]], label [[COND_FALSE4344:%.*]]
+// SIMD-ONLY0:       cond.true4343:
+// SIMD-ONLY0-NEXT:    [[TMP2340:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4345:%.*]]
+// SIMD-ONLY0:       cond.false4344:
+// SIMD-ONLY0-NEXT:    [[TMP2341:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4345]]
+// SIMD-ONLY0:       cond.end4345:
+// SIMD-ONLY0-NEXT:    [[COND4346:%.*]] = phi i64 [ [[TMP2340]], [[COND_TRUE4343]] ], [ [[TMP2341]], [[COND_FALSE4344]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4346]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2342:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2343:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4347:%.*]] = icmp eq i64 [[TMP2342]], [[TMP2343]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4347]], label [[COND_TRUE4349:%.*]], label [[COND_FALSE4350:%.*]]
+// SIMD-ONLY0:       cond.true4349:
+// SIMD-ONLY0-NEXT:    [[TMP2344:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4351:%.*]]
+// SIMD-ONLY0:       cond.false4350:
+// SIMD-ONLY0-NEXT:    [[TMP2345:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4351]]
+// SIMD-ONLY0:       cond.end4351:
+// SIMD-ONLY0-NEXT:    [[COND4352:%.*]] = phi i64 [ [[TMP2344]], [[COND_TRUE4349]] ], [ [[TMP2345]], [[COND_FALSE4350]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4352]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2346:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2347:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4353:%.*]] = icmp eq i64 [[TMP2346]], [[TMP2347]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4353]], label [[IF_THEN4355:%.*]], label [[IF_END4356:%.*]]
+// SIMD-ONLY0:       if.then4355:
+// SIMD-ONLY0-NEXT:    [[TMP2348:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2348]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4356]]
+// SIMD-ONLY0:       if.end4356:
+// SIMD-ONLY0-NEXT:    [[TMP2349:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2350:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4357:%.*]] = icmp eq i64 [[TMP2349]], [[TMP2350]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4357]], label [[IF_THEN4359:%.*]], label [[IF_END4360:%.*]]
+// SIMD-ONLY0:       if.then4359:
+// SIMD-ONLY0-NEXT:    [[TMP2351:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2351]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4360]]
+// SIMD-ONLY0:       if.end4360:
+// SIMD-ONLY0-NEXT:    [[TMP2352:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2353:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4361:%.*]] = icmp sgt i64 [[TMP2352]], [[TMP2353]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4361]], label [[COND_TRUE4363:%.*]], label [[COND_FALSE4364:%.*]]
+// SIMD-ONLY0:       cond.true4363:
+// SIMD-ONLY0-NEXT:    [[TMP2354:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4365:%.*]]
+// SIMD-ONLY0:       cond.false4364:
+// SIMD-ONLY0-NEXT:    [[TMP2355:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4365]]
+// SIMD-ONLY0:       cond.end4365:
+// SIMD-ONLY0-NEXT:    [[COND4366:%.*]] = phi i64 [ [[TMP2354]], [[COND_TRUE4363]] ], [ [[TMP2355]], [[COND_FALSE4364]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4366]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2356:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2357:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4367:%.*]] = icmp slt i64 [[TMP2356]], [[TMP2357]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4367]], label [[COND_TRUE4369:%.*]], label [[COND_FALSE4370:%.*]]
+// SIMD-ONLY0:       cond.true4369:
+// SIMD-ONLY0-NEXT:    [[TMP2358:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4371:%.*]]
+// SIMD-ONLY0:       cond.false4370:
+// SIMD-ONLY0-NEXT:    [[TMP2359:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4371]]
+// SIMD-ONLY0:       cond.end4371:
+// SIMD-ONLY0-NEXT:    [[COND4372:%.*]] = phi i64 [ [[TMP2358]], [[COND_TRUE4369]] ], [ [[TMP2359]], [[COND_FALSE4370]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4372]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2360:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2361:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4373:%.*]] = icmp sgt i64 [[TMP2360]], [[TMP2361]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4373]], label [[COND_TRUE4375:%.*]], label [[COND_FALSE4376:%.*]]
+// SIMD-ONLY0:       cond.true4375:
+// SIMD-ONLY0-NEXT:    [[TMP2362:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4377:%.*]]
+// SIMD-ONLY0:       cond.false4376:
+// SIMD-ONLY0-NEXT:    [[TMP2363:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4377]]
+// SIMD-ONLY0:       cond.end4377:
+// SIMD-ONLY0-NEXT:    [[COND4378:%.*]] = phi i64 [ [[TMP2362]], [[COND_TRUE4375]] ], [ [[TMP2363]], [[COND_FALSE4376]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4378]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2364:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2365:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4379:%.*]] = icmp slt i64 [[TMP2364]], [[TMP2365]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4379]], label [[COND_TRUE4381:%.*]], label [[COND_FALSE4382:%.*]]
+// SIMD-ONLY0:       cond.true4381:
+// SIMD-ONLY0-NEXT:    [[TMP2366:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4383:%.*]]
+// SIMD-ONLY0:       cond.false4382:
+// SIMD-ONLY0-NEXT:    [[TMP2367:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4383]]
+// SIMD-ONLY0:       cond.end4383:
+// SIMD-ONLY0-NEXT:    [[COND4384:%.*]] = phi i64 [ [[TMP2366]], [[COND_TRUE4381]] ], [ [[TMP2367]], [[COND_FALSE4382]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4384]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2368:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2369:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4385:%.*]] = icmp sgt i64 [[TMP2368]], [[TMP2369]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4385]], label [[IF_THEN4387:%.*]], label [[IF_END4388:%.*]]
+// SIMD-ONLY0:       if.then4387:
+// SIMD-ONLY0-NEXT:    [[TMP2370:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2370]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4388]]
+// SIMD-ONLY0:       if.end4388:
+// SIMD-ONLY0-NEXT:    [[TMP2371:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2372:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4389:%.*]] = icmp slt i64 [[TMP2371]], [[TMP2372]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4389]], label [[IF_THEN4391:%.*]], label [[IF_END4392:%.*]]
+// SIMD-ONLY0:       if.then4391:
+// SIMD-ONLY0-NEXT:    [[TMP2373:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2373]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4392]]
+// SIMD-ONLY0:       if.end4392:
+// SIMD-ONLY0-NEXT:    [[TMP2374:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2375:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4393:%.*]] = icmp sgt i64 [[TMP2374]], [[TMP2375]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4393]], label [[IF_THEN4395:%.*]], label [[IF_END4396:%.*]]
+// SIMD-ONLY0:       if.then4395:
+// SIMD-ONLY0-NEXT:    [[TMP2376:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2376]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4396]]
+// SIMD-ONLY0:       if.end4396:
+// SIMD-ONLY0-NEXT:    [[TMP2377:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2378:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4397:%.*]] = icmp slt i64 [[TMP2377]], [[TMP2378]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4397]], label [[IF_THEN4399:%.*]], label [[IF_END4400:%.*]]
+// SIMD-ONLY0:       if.then4399:
+// SIMD-ONLY0-NEXT:    [[TMP2379:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2379]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4400]]
+// SIMD-ONLY0:       if.end4400:
+// SIMD-ONLY0-NEXT:    [[TMP2380:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2381:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4401:%.*]] = icmp eq i64 [[TMP2380]], [[TMP2381]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4401]], label [[COND_TRUE4403:%.*]], label [[COND_FALSE4404:%.*]]
+// SIMD-ONLY0:       cond.true4403:
+// SIMD-ONLY0-NEXT:    [[TMP2382:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4405:%.*]]
+// SIMD-ONLY0:       cond.false4404:
+// SIMD-ONLY0-NEXT:    [[TMP2383:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4405]]
+// SIMD-ONLY0:       cond.end4405:
+// SIMD-ONLY0-NEXT:    [[COND4406:%.*]] = phi i64 [ [[TMP2382]], [[COND_TRUE4403]] ], [ [[TMP2383]], [[COND_FALSE4404]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4406]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2384:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2385:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4407:%.*]] = icmp eq i64 [[TMP2384]], [[TMP2385]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4407]], label [[COND_TRUE4409:%.*]], label [[COND_FALSE4410:%.*]]
+// SIMD-ONLY0:       cond.true4409:
+// SIMD-ONLY0-NEXT:    [[TMP2386:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4411:%.*]]
+// SIMD-ONLY0:       cond.false4410:
+// SIMD-ONLY0-NEXT:    [[TMP2387:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4411]]
+// SIMD-ONLY0:       cond.end4411:
+// SIMD-ONLY0-NEXT:    [[COND4412:%.*]] = phi i64 [ [[TMP2386]], [[COND_TRUE4409]] ], [ [[TMP2387]], [[COND_FALSE4410]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4412]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2388:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2389:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4413:%.*]] = icmp eq i64 [[TMP2388]], [[TMP2389]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4413]], label [[IF_THEN4415:%.*]], label [[IF_END4416:%.*]]
+// SIMD-ONLY0:       if.then4415:
+// SIMD-ONLY0-NEXT:    [[TMP2390:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2390]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4416]]
+// SIMD-ONLY0:       if.end4416:
+// SIMD-ONLY0-NEXT:    [[TMP2391:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2392:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4417:%.*]] = icmp eq i64 [[TMP2391]], [[TMP2392]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4417]], label [[IF_THEN4419:%.*]], label [[IF_END4420:%.*]]
+// SIMD-ONLY0:       if.then4419:
+// SIMD-ONLY0-NEXT:    [[TMP2393:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2393]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4420]]
+// SIMD-ONLY0:       if.end4420:
+// SIMD-ONLY0-NEXT:    [[TMP2394:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2395:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4421:%.*]] = icmp ugt i64 [[TMP2394]], [[TMP2395]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4421]], label [[COND_TRUE4423:%.*]], label [[COND_FALSE4424:%.*]]
+// SIMD-ONLY0:       cond.true4423:
+// SIMD-ONLY0-NEXT:    [[TMP2396:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4425:%.*]]
+// SIMD-ONLY0:       cond.false4424:
+// SIMD-ONLY0-NEXT:    [[TMP2397:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4425]]
+// SIMD-ONLY0:       cond.end4425:
+// SIMD-ONLY0-NEXT:    [[COND4426:%.*]] = phi i64 [ [[TMP2396]], [[COND_TRUE4423]] ], [ [[TMP2397]], [[COND_FALSE4424]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4426]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2398:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2399:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4427:%.*]] = icmp ult i64 [[TMP2398]], [[TMP2399]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4427]], label [[COND_TRUE4429:%.*]], label [[COND_FALSE4430:%.*]]
+// SIMD-ONLY0:       cond.true4429:
+// SIMD-ONLY0-NEXT:    [[TMP2400:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4431:%.*]]
+// SIMD-ONLY0:       cond.false4430:
+// SIMD-ONLY0-NEXT:    [[TMP2401:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4431]]
+// SIMD-ONLY0:       cond.end4431:
+// SIMD-ONLY0-NEXT:    [[COND4432:%.*]] = phi i64 [ [[TMP2400]], [[COND_TRUE4429]] ], [ [[TMP2401]], [[COND_FALSE4430]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4432]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2402:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2403:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4433:%.*]] = icmp ugt i64 [[TMP2402]], [[TMP2403]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4433]], label [[COND_TRUE4435:%.*]], label [[COND_FALSE4436:%.*]]
+// SIMD-ONLY0:       cond.true4435:
+// SIMD-ONLY0-NEXT:    [[TMP2404:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4437:%.*]]
+// SIMD-ONLY0:       cond.false4436:
+// SIMD-ONLY0-NEXT:    [[TMP2405:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4437]]
+// SIMD-ONLY0:       cond.end4437:
+// SIMD-ONLY0-NEXT:    [[COND4438:%.*]] = phi i64 [ [[TMP2404]], [[COND_TRUE4435]] ], [ [[TMP2405]], [[COND_FALSE4436]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4438]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2406:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2407:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4439:%.*]] = icmp ult i64 [[TMP2406]], [[TMP2407]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4439]], label [[COND_TRUE4441:%.*]], label [[COND_FALSE4442:%.*]]
+// SIMD-ONLY0:       cond.true4441:
+// SIMD-ONLY0-NEXT:    [[TMP2408:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4443:%.*]]
+// SIMD-ONLY0:       cond.false4442:
+// SIMD-ONLY0-NEXT:    [[TMP2409:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4443]]
+// SIMD-ONLY0:       cond.end4443:
+// SIMD-ONLY0-NEXT:    [[COND4444:%.*]] = phi i64 [ [[TMP2408]], [[COND_TRUE4441]] ], [ [[TMP2409]], [[COND_FALSE4442]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4444]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2410:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2411:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4445:%.*]] = icmp ugt i64 [[TMP2410]], [[TMP2411]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4445]], label [[IF_THEN4447:%.*]], label [[IF_END4448:%.*]]
+// SIMD-ONLY0:       if.then4447:
+// SIMD-ONLY0-NEXT:    [[TMP2412:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2412]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4448]]
+// SIMD-ONLY0:       if.end4448:
+// SIMD-ONLY0-NEXT:    [[TMP2413:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2414:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4449:%.*]] = icmp ult i64 [[TMP2413]], [[TMP2414]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4449]], label [[IF_THEN4451:%.*]], label [[IF_END4452:%.*]]
+// SIMD-ONLY0:       if.then4451:
+// SIMD-ONLY0-NEXT:    [[TMP2415:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2415]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4452]]
+// SIMD-ONLY0:       if.end4452:
+// SIMD-ONLY0-NEXT:    [[TMP2416:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2417:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4453:%.*]] = icmp ugt i64 [[TMP2416]], [[TMP2417]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4453]], label [[IF_THEN4455:%.*]], label [[IF_END4456:%.*]]
+// SIMD-ONLY0:       if.then4455:
+// SIMD-ONLY0-NEXT:    [[TMP2418:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2418]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4456]]
+// SIMD-ONLY0:       if.end4456:
+// SIMD-ONLY0-NEXT:    [[TMP2419:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2420:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4457:%.*]] = icmp ult i64 [[TMP2419]], [[TMP2420]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4457]], label [[IF_THEN4459:%.*]], label [[IF_END4460:%.*]]
+// SIMD-ONLY0:       if.then4459:
+// SIMD-ONLY0-NEXT:    [[TMP2421:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2421]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4460]]
+// SIMD-ONLY0:       if.end4460:
+// SIMD-ONLY0-NEXT:    [[TMP2422:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2423:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4461:%.*]] = icmp eq i64 [[TMP2422]], [[TMP2423]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4461]], label [[COND_TRUE4463:%.*]], label [[COND_FALSE4464:%.*]]
+// SIMD-ONLY0:       cond.true4463:
+// SIMD-ONLY0-NEXT:    [[TMP2424:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4465:%.*]]
+// SIMD-ONLY0:       cond.false4464:
+// SIMD-ONLY0-NEXT:    [[TMP2425:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4465]]
+// SIMD-ONLY0:       cond.end4465:
+// SIMD-ONLY0-NEXT:    [[COND4466:%.*]] = phi i64 [ [[TMP2424]], [[COND_TRUE4463]] ], [ [[TMP2425]], [[COND_FALSE4464]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4466]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2426:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2427:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4467:%.*]] = icmp eq i64 [[TMP2426]], [[TMP2427]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4467]], label [[COND_TRUE4469:%.*]], label [[COND_FALSE4470:%.*]]
+// SIMD-ONLY0:       cond.true4469:
+// SIMD-ONLY0-NEXT:    [[TMP2428:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4471:%.*]]
+// SIMD-ONLY0:       cond.false4470:
+// SIMD-ONLY0-NEXT:    [[TMP2429:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4471]]
+// SIMD-ONLY0:       cond.end4471:
+// SIMD-ONLY0-NEXT:    [[COND4472:%.*]] = phi i64 [ [[TMP2428]], [[COND_TRUE4469]] ], [ [[TMP2429]], [[COND_FALSE4470]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4472]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2430:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2431:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4473:%.*]] = icmp eq i64 [[TMP2430]], [[TMP2431]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4473]], label [[IF_THEN4475:%.*]], label [[IF_END4476:%.*]]
+// SIMD-ONLY0:       if.then4475:
+// SIMD-ONLY0-NEXT:    [[TMP2432:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2432]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4476]]
+// SIMD-ONLY0:       if.end4476:
+// SIMD-ONLY0-NEXT:    [[TMP2433:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2434:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4477:%.*]] = icmp eq i64 [[TMP2433]], [[TMP2434]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4477]], label [[IF_THEN4479:%.*]], label [[IF_END4480:%.*]]
+// SIMD-ONLY0:       if.then4479:
+// SIMD-ONLY0-NEXT:    [[TMP2435:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2435]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4480]]
+// SIMD-ONLY0:       if.end4480:
+// SIMD-ONLY0-NEXT:    [[TMP2436:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2437:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4481:%.*]] = icmp sgt i64 [[TMP2436]], [[TMP2437]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4481]], label [[COND_TRUE4483:%.*]], label [[COND_FALSE4484:%.*]]
+// SIMD-ONLY0:       cond.true4483:
+// SIMD-ONLY0-NEXT:    [[TMP2438:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4485:%.*]]
+// SIMD-ONLY0:       cond.false4484:
+// SIMD-ONLY0-NEXT:    [[TMP2439:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4485]]
+// SIMD-ONLY0:       cond.end4485:
+// SIMD-ONLY0-NEXT:    [[COND4486:%.*]] = phi i64 [ [[TMP2438]], [[COND_TRUE4483]] ], [ [[TMP2439]], [[COND_FALSE4484]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4486]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2440:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2441:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4487:%.*]] = icmp slt i64 [[TMP2440]], [[TMP2441]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4487]], label [[COND_TRUE4489:%.*]], label [[COND_FALSE4490:%.*]]
+// SIMD-ONLY0:       cond.true4489:
+// SIMD-ONLY0-NEXT:    [[TMP2442:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4491:%.*]]
+// SIMD-ONLY0:       cond.false4490:
+// SIMD-ONLY0-NEXT:    [[TMP2443:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4491]]
+// SIMD-ONLY0:       cond.end4491:
+// SIMD-ONLY0-NEXT:    [[COND4492:%.*]] = phi i64 [ [[TMP2442]], [[COND_TRUE4489]] ], [ [[TMP2443]], [[COND_FALSE4490]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4492]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2444:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2445:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4493:%.*]] = icmp sgt i64 [[TMP2444]], [[TMP2445]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4493]], label [[COND_TRUE4495:%.*]], label [[COND_FALSE4496:%.*]]
+// SIMD-ONLY0:       cond.true4495:
+// SIMD-ONLY0-NEXT:    [[TMP2446:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4497:%.*]]
+// SIMD-ONLY0:       cond.false4496:
+// SIMD-ONLY0-NEXT:    [[TMP2447:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4497]]
+// SIMD-ONLY0:       cond.end4497:
+// SIMD-ONLY0-NEXT:    [[COND4498:%.*]] = phi i64 [ [[TMP2446]], [[COND_TRUE4495]] ], [ [[TMP2447]], [[COND_FALSE4496]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4498]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2448:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2449:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4499:%.*]] = icmp slt i64 [[TMP2448]], [[TMP2449]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4499]], label [[COND_TRUE4501:%.*]], label [[COND_FALSE4502:%.*]]
+// SIMD-ONLY0:       cond.true4501:
+// SIMD-ONLY0-NEXT:    [[TMP2450:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4503:%.*]]
+// SIMD-ONLY0:       cond.false4502:
+// SIMD-ONLY0-NEXT:    [[TMP2451:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4503]]
+// SIMD-ONLY0:       cond.end4503:
+// SIMD-ONLY0-NEXT:    [[COND4504:%.*]] = phi i64 [ [[TMP2450]], [[COND_TRUE4501]] ], [ [[TMP2451]], [[COND_FALSE4502]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4504]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2452:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2453:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4505:%.*]] = icmp sgt i64 [[TMP2452]], [[TMP2453]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4505]], label [[IF_THEN4507:%.*]], label [[IF_END4508:%.*]]
+// SIMD-ONLY0:       if.then4507:
+// SIMD-ONLY0-NEXT:    [[TMP2454:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2454]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4508]]
+// SIMD-ONLY0:       if.end4508:
+// SIMD-ONLY0-NEXT:    [[TMP2455:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2456:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4509:%.*]] = icmp slt i64 [[TMP2455]], [[TMP2456]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4509]], label [[IF_THEN4511:%.*]], label [[IF_END4512:%.*]]
+// SIMD-ONLY0:       if.then4511:
+// SIMD-ONLY0-NEXT:    [[TMP2457:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2457]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4512]]
+// SIMD-ONLY0:       if.end4512:
+// SIMD-ONLY0-NEXT:    [[TMP2458:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2459:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4513:%.*]] = icmp sgt i64 [[TMP2458]], [[TMP2459]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4513]], label [[IF_THEN4515:%.*]], label [[IF_END4516:%.*]]
+// SIMD-ONLY0:       if.then4515:
+// SIMD-ONLY0-NEXT:    [[TMP2460:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2460]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4516]]
+// SIMD-ONLY0:       if.end4516:
+// SIMD-ONLY0-NEXT:    [[TMP2461:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2462:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4517:%.*]] = icmp slt i64 [[TMP2461]], [[TMP2462]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4517]], label [[IF_THEN4519:%.*]], label [[IF_END4520:%.*]]
+// SIMD-ONLY0:       if.then4519:
+// SIMD-ONLY0-NEXT:    [[TMP2463:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2463]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4520]]
+// SIMD-ONLY0:       if.end4520:
+// SIMD-ONLY0-NEXT:    [[TMP2464:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2465:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4521:%.*]] = icmp eq i64 [[TMP2464]], [[TMP2465]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4521]], label [[COND_TRUE4523:%.*]], label [[COND_FALSE4524:%.*]]
+// SIMD-ONLY0:       cond.true4523:
+// SIMD-ONLY0-NEXT:    [[TMP2466:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4525:%.*]]
+// SIMD-ONLY0:       cond.false4524:
+// SIMD-ONLY0-NEXT:    [[TMP2467:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4525]]
+// SIMD-ONLY0:       cond.end4525:
+// SIMD-ONLY0-NEXT:    [[COND4526:%.*]] = phi i64 [ [[TMP2466]], [[COND_TRUE4523]] ], [ [[TMP2467]], [[COND_FALSE4524]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4526]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2468:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2469:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4527:%.*]] = icmp eq i64 [[TMP2468]], [[TMP2469]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4527]], label [[COND_TRUE4529:%.*]], label [[COND_FALSE4530:%.*]]
+// SIMD-ONLY0:       cond.true4529:
+// SIMD-ONLY0-NEXT:    [[TMP2470:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4531:%.*]]
+// SIMD-ONLY0:       cond.false4530:
+// SIMD-ONLY0-NEXT:    [[TMP2471:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4531]]
+// SIMD-ONLY0:       cond.end4531:
+// SIMD-ONLY0-NEXT:    [[COND4532:%.*]] = phi i64 [ [[TMP2470]], [[COND_TRUE4529]] ], [ [[TMP2471]], [[COND_FALSE4530]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4532]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2472:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2473:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4533:%.*]] = icmp eq i64 [[TMP2472]], [[TMP2473]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4533]], label [[IF_THEN4535:%.*]], label [[IF_END4536:%.*]]
+// SIMD-ONLY0:       if.then4535:
+// SIMD-ONLY0-NEXT:    [[TMP2474:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2474]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4536]]
+// SIMD-ONLY0:       if.end4536:
+// SIMD-ONLY0-NEXT:    [[TMP2475:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2476:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4537:%.*]] = icmp eq i64 [[TMP2475]], [[TMP2476]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4537]], label [[IF_THEN4539:%.*]], label [[IF_END4540:%.*]]
+// SIMD-ONLY0:       if.then4539:
+// SIMD-ONLY0-NEXT:    [[TMP2477:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2477]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4540]]
+// SIMD-ONLY0:       if.end4540:
+// SIMD-ONLY0-NEXT:    [[TMP2478:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2479:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4541:%.*]] = icmp ugt i64 [[TMP2478]], [[TMP2479]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4541]], label [[COND_TRUE4543:%.*]], label [[COND_FALSE4544:%.*]]
+// SIMD-ONLY0:       cond.true4543:
+// SIMD-ONLY0-NEXT:    [[TMP2480:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4545:%.*]]
+// SIMD-ONLY0:       cond.false4544:
+// SIMD-ONLY0-NEXT:    [[TMP2481:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4545]]
+// SIMD-ONLY0:       cond.end4545:
+// SIMD-ONLY0-NEXT:    [[COND4546:%.*]] = phi i64 [ [[TMP2480]], [[COND_TRUE4543]] ], [ [[TMP2481]], [[COND_FALSE4544]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4546]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2482:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2483:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4547:%.*]] = icmp ult i64 [[TMP2482]], [[TMP2483]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4547]], label [[COND_TRUE4549:%.*]], label [[COND_FALSE4550:%.*]]
+// SIMD-ONLY0:       cond.true4549:
+// SIMD-ONLY0-NEXT:    [[TMP2484:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4551:%.*]]
+// SIMD-ONLY0:       cond.false4550:
+// SIMD-ONLY0-NEXT:    [[TMP2485:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4551]]
+// SIMD-ONLY0:       cond.end4551:
+// SIMD-ONLY0-NEXT:    [[COND4552:%.*]] = phi i64 [ [[TMP2484]], [[COND_TRUE4549]] ], [ [[TMP2485]], [[COND_FALSE4550]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4552]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2486:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2487:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4553:%.*]] = icmp ugt i64 [[TMP2486]], [[TMP2487]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4553]], label [[COND_TRUE4555:%.*]], label [[COND_FALSE4556:%.*]]
+// SIMD-ONLY0:       cond.true4555:
+// SIMD-ONLY0-NEXT:    [[TMP2488:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4557:%.*]]
+// SIMD-ONLY0:       cond.false4556:
+// SIMD-ONLY0-NEXT:    [[TMP2489:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4557]]
+// SIMD-ONLY0:       cond.end4557:
+// SIMD-ONLY0-NEXT:    [[COND4558:%.*]] = phi i64 [ [[TMP2488]], [[COND_TRUE4555]] ], [ [[TMP2489]], [[COND_FALSE4556]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4558]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2490:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2491:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4559:%.*]] = icmp ult i64 [[TMP2490]], [[TMP2491]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4559]], label [[COND_TRUE4561:%.*]], label [[COND_FALSE4562:%.*]]
+// SIMD-ONLY0:       cond.true4561:
+// SIMD-ONLY0-NEXT:    [[TMP2492:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4563:%.*]]
+// SIMD-ONLY0:       cond.false4562:
+// SIMD-ONLY0-NEXT:    [[TMP2493:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4563]]
+// SIMD-ONLY0:       cond.end4563:
+// SIMD-ONLY0-NEXT:    [[COND4564:%.*]] = phi i64 [ [[TMP2492]], [[COND_TRUE4561]] ], [ [[TMP2493]], [[COND_FALSE4562]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4564]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2494:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2495:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4565:%.*]] = icmp ugt i64 [[TMP2494]], [[TMP2495]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4565]], label [[IF_THEN4567:%.*]], label [[IF_END4568:%.*]]
+// SIMD-ONLY0:       if.then4567:
+// SIMD-ONLY0-NEXT:    [[TMP2496:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2496]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4568]]
+// SIMD-ONLY0:       if.end4568:
+// SIMD-ONLY0-NEXT:    [[TMP2497:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2498:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4569:%.*]] = icmp ult i64 [[TMP2497]], [[TMP2498]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4569]], label [[IF_THEN4571:%.*]], label [[IF_END4572:%.*]]
+// SIMD-ONLY0:       if.then4571:
+// SIMD-ONLY0-NEXT:    [[TMP2499:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2499]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4572]]
+// SIMD-ONLY0:       if.end4572:
+// SIMD-ONLY0-NEXT:    [[TMP2500:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2501:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4573:%.*]] = icmp ugt i64 [[TMP2500]], [[TMP2501]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4573]], label [[IF_THEN4575:%.*]], label [[IF_END4576:%.*]]
+// SIMD-ONLY0:       if.then4575:
+// SIMD-ONLY0-NEXT:    [[TMP2502:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2502]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4576]]
+// SIMD-ONLY0:       if.end4576:
+// SIMD-ONLY0-NEXT:    [[TMP2503:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2504:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4577:%.*]] = icmp ult i64 [[TMP2503]], [[TMP2504]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4577]], label [[IF_THEN4579:%.*]], label [[IF_END4580:%.*]]
+// SIMD-ONLY0:       if.then4579:
+// SIMD-ONLY0-NEXT:    [[TMP2505:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2505]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4580]]
+// SIMD-ONLY0:       if.end4580:
+// SIMD-ONLY0-NEXT:    [[TMP2506:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2507:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4581:%.*]] = icmp eq i64 [[TMP2506]], [[TMP2507]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4581]], label [[COND_TRUE4583:%.*]], label [[COND_FALSE4584:%.*]]
+// SIMD-ONLY0:       cond.true4583:
+// SIMD-ONLY0-NEXT:    [[TMP2508:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4585:%.*]]
+// SIMD-ONLY0:       cond.false4584:
+// SIMD-ONLY0-NEXT:    [[TMP2509:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4585]]
+// SIMD-ONLY0:       cond.end4585:
+// SIMD-ONLY0-NEXT:    [[COND4586:%.*]] = phi i64 [ [[TMP2508]], [[COND_TRUE4583]] ], [ [[TMP2509]], [[COND_FALSE4584]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4586]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2510:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2511:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4587:%.*]] = icmp eq i64 [[TMP2510]], [[TMP2511]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4587]], label [[COND_TRUE4589:%.*]], label [[COND_FALSE4590:%.*]]
+// SIMD-ONLY0:       cond.true4589:
+// SIMD-ONLY0-NEXT:    [[TMP2512:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4591:%.*]]
+// SIMD-ONLY0:       cond.false4590:
+// SIMD-ONLY0-NEXT:    [[TMP2513:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4591]]
+// SIMD-ONLY0:       cond.end4591:
+// SIMD-ONLY0-NEXT:    [[COND4592:%.*]] = phi i64 [ [[TMP2512]], [[COND_TRUE4589]] ], [ [[TMP2513]], [[COND_FALSE4590]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND4592]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2514:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2515:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4593:%.*]] = icmp eq i64 [[TMP2514]], [[TMP2515]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4593]], label [[IF_THEN4595:%.*]], label [[IF_END4596:%.*]]
+// SIMD-ONLY0:       if.then4595:
+// SIMD-ONLY0-NEXT:    [[TMP2516:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2516]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4596]]
+// SIMD-ONLY0:       if.end4596:
+// SIMD-ONLY0-NEXT:    [[TMP2517:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2518:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4597:%.*]] = icmp eq i64 [[TMP2517]], [[TMP2518]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4597]], label [[IF_THEN4599:%.*]], label [[IF_END4600:%.*]]
+// SIMD-ONLY0:       if.then4599:
+// SIMD-ONLY0-NEXT:    [[TMP2519:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2519]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4600]]
+// SIMD-ONLY0:       if.end4600:
+// SIMD-ONLY0-NEXT:    [[TMP2520:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2521:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4601:%.*]] = fcmp ogt float [[TMP2520]], [[TMP2521]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4601]], label [[COND_TRUE4603:%.*]], label [[COND_FALSE4604:%.*]]
+// SIMD-ONLY0:       cond.true4603:
+// SIMD-ONLY0-NEXT:    [[TMP2522:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4605:%.*]]
+// SIMD-ONLY0:       cond.false4604:
+// SIMD-ONLY0-NEXT:    [[TMP2523:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4605]]
+// SIMD-ONLY0:       cond.end4605:
+// SIMD-ONLY0-NEXT:    [[COND4606:%.*]] = phi float [ [[TMP2522]], [[COND_TRUE4603]] ], [ [[TMP2523]], [[COND_FALSE4604]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4606]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2524:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2525:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4607:%.*]] = fcmp olt float [[TMP2524]], [[TMP2525]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4607]], label [[COND_TRUE4609:%.*]], label [[COND_FALSE4610:%.*]]
+// SIMD-ONLY0:       cond.true4609:
+// SIMD-ONLY0-NEXT:    [[TMP2526:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4611:%.*]]
+// SIMD-ONLY0:       cond.false4610:
+// SIMD-ONLY0-NEXT:    [[TMP2527:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4611]]
+// SIMD-ONLY0:       cond.end4611:
+// SIMD-ONLY0-NEXT:    [[COND4612:%.*]] = phi float [ [[TMP2526]], [[COND_TRUE4609]] ], [ [[TMP2527]], [[COND_FALSE4610]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4612]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2528:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2529:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4613:%.*]] = fcmp ogt float [[TMP2528]], [[TMP2529]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4613]], label [[COND_TRUE4615:%.*]], label [[COND_FALSE4616:%.*]]
+// SIMD-ONLY0:       cond.true4615:
+// SIMD-ONLY0-NEXT:    [[TMP2530:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4617:%.*]]
+// SIMD-ONLY0:       cond.false4616:
+// SIMD-ONLY0-NEXT:    [[TMP2531:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4617]]
+// SIMD-ONLY0:       cond.end4617:
+// SIMD-ONLY0-NEXT:    [[COND4618:%.*]] = phi float [ [[TMP2530]], [[COND_TRUE4615]] ], [ [[TMP2531]], [[COND_FALSE4616]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4618]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2532:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2533:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4619:%.*]] = fcmp olt float [[TMP2532]], [[TMP2533]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4619]], label [[COND_TRUE4621:%.*]], label [[COND_FALSE4622:%.*]]
+// SIMD-ONLY0:       cond.true4621:
+// SIMD-ONLY0-NEXT:    [[TMP2534:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4623:%.*]]
+// SIMD-ONLY0:       cond.false4622:
+// SIMD-ONLY0-NEXT:    [[TMP2535:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4623]]
+// SIMD-ONLY0:       cond.end4623:
+// SIMD-ONLY0-NEXT:    [[COND4624:%.*]] = phi float [ [[TMP2534]], [[COND_TRUE4621]] ], [ [[TMP2535]], [[COND_FALSE4622]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4624]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2536:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2537:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4625:%.*]] = fcmp ogt float [[TMP2536]], [[TMP2537]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4625]], label [[IF_THEN4627:%.*]], label [[IF_END4628:%.*]]
+// SIMD-ONLY0:       if.then4627:
+// SIMD-ONLY0-NEXT:    [[TMP2538:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2538]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4628]]
+// SIMD-ONLY0:       if.end4628:
+// SIMD-ONLY0-NEXT:    [[TMP2539:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2540:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4629:%.*]] = fcmp olt float [[TMP2539]], [[TMP2540]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4629]], label [[IF_THEN4631:%.*]], label [[IF_END4632:%.*]]
+// SIMD-ONLY0:       if.then4631:
+// SIMD-ONLY0-NEXT:    [[TMP2541:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2541]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4632]]
+// SIMD-ONLY0:       if.end4632:
+// SIMD-ONLY0-NEXT:    [[TMP2542:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2543:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4633:%.*]] = fcmp ogt float [[TMP2542]], [[TMP2543]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4633]], label [[IF_THEN4635:%.*]], label [[IF_END4636:%.*]]
+// SIMD-ONLY0:       if.then4635:
+// SIMD-ONLY0-NEXT:    [[TMP2544:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2544]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4636]]
+// SIMD-ONLY0:       if.end4636:
+// SIMD-ONLY0-NEXT:    [[TMP2545:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2546:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4637:%.*]] = fcmp olt float [[TMP2545]], [[TMP2546]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4637]], label [[IF_THEN4639:%.*]], label [[IF_END4640:%.*]]
+// SIMD-ONLY0:       if.then4639:
+// SIMD-ONLY0-NEXT:    [[TMP2547:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2547]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4640]]
+// SIMD-ONLY0:       if.end4640:
+// SIMD-ONLY0-NEXT:    [[TMP2548:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2549:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4641:%.*]] = fcmp ogt float [[TMP2548]], [[TMP2549]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4641]], label [[COND_TRUE4643:%.*]], label [[COND_FALSE4644:%.*]]
+// SIMD-ONLY0:       cond.true4643:
+// SIMD-ONLY0-NEXT:    [[TMP2550:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4645:%.*]]
+// SIMD-ONLY0:       cond.false4644:
+// SIMD-ONLY0-NEXT:    [[TMP2551:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4645]]
+// SIMD-ONLY0:       cond.end4645:
+// SIMD-ONLY0-NEXT:    [[COND4646:%.*]] = phi float [ [[TMP2550]], [[COND_TRUE4643]] ], [ [[TMP2551]], [[COND_FALSE4644]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4646]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2552:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2553:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4647:%.*]] = fcmp olt float [[TMP2552]], [[TMP2553]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4647]], label [[COND_TRUE4649:%.*]], label [[COND_FALSE4650:%.*]]
+// SIMD-ONLY0:       cond.true4649:
+// SIMD-ONLY0-NEXT:    [[TMP2554:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4651:%.*]]
+// SIMD-ONLY0:       cond.false4650:
+// SIMD-ONLY0-NEXT:    [[TMP2555:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4651]]
+// SIMD-ONLY0:       cond.end4651:
+// SIMD-ONLY0-NEXT:    [[COND4652:%.*]] = phi float [ [[TMP2554]], [[COND_TRUE4649]] ], [ [[TMP2555]], [[COND_FALSE4650]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4652]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2556:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2557:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4653:%.*]] = fcmp ogt float [[TMP2556]], [[TMP2557]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4653]], label [[COND_TRUE4655:%.*]], label [[COND_FALSE4656:%.*]]
+// SIMD-ONLY0:       cond.true4655:
+// SIMD-ONLY0-NEXT:    [[TMP2558:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4657:%.*]]
+// SIMD-ONLY0:       cond.false4656:
+// SIMD-ONLY0-NEXT:    [[TMP2559:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4657]]
+// SIMD-ONLY0:       cond.end4657:
+// SIMD-ONLY0-NEXT:    [[COND4658:%.*]] = phi float [ [[TMP2558]], [[COND_TRUE4655]] ], [ [[TMP2559]], [[COND_FALSE4656]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4658]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2560:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2561:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4659:%.*]] = fcmp olt float [[TMP2560]], [[TMP2561]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4659]], label [[COND_TRUE4661:%.*]], label [[COND_FALSE4662:%.*]]
+// SIMD-ONLY0:       cond.true4661:
+// SIMD-ONLY0-NEXT:    [[TMP2562:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4663:%.*]]
+// SIMD-ONLY0:       cond.false4662:
+// SIMD-ONLY0-NEXT:    [[TMP2563:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4663]]
+// SIMD-ONLY0:       cond.end4663:
+// SIMD-ONLY0-NEXT:    [[COND4664:%.*]] = phi float [ [[TMP2562]], [[COND_TRUE4661]] ], [ [[TMP2563]], [[COND_FALSE4662]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4664]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2564:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2565:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4665:%.*]] = fcmp ogt float [[TMP2564]], [[TMP2565]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4665]], label [[IF_THEN4667:%.*]], label [[IF_END4668:%.*]]
+// SIMD-ONLY0:       if.then4667:
+// SIMD-ONLY0-NEXT:    [[TMP2566:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2566]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4668]]
+// SIMD-ONLY0:       if.end4668:
+// SIMD-ONLY0-NEXT:    [[TMP2567:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2568:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4669:%.*]] = fcmp olt float [[TMP2567]], [[TMP2568]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4669]], label [[IF_THEN4671:%.*]], label [[IF_END4672:%.*]]
+// SIMD-ONLY0:       if.then4671:
+// SIMD-ONLY0-NEXT:    [[TMP2569:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2569]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4672]]
+// SIMD-ONLY0:       if.end4672:
+// SIMD-ONLY0-NEXT:    [[TMP2570:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2571:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4673:%.*]] = fcmp ogt float [[TMP2570]], [[TMP2571]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4673]], label [[IF_THEN4675:%.*]], label [[IF_END4676:%.*]]
+// SIMD-ONLY0:       if.then4675:
+// SIMD-ONLY0-NEXT:    [[TMP2572:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2572]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4676]]
+// SIMD-ONLY0:       if.end4676:
+// SIMD-ONLY0-NEXT:    [[TMP2573:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2574:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4677:%.*]] = fcmp olt float [[TMP2573]], [[TMP2574]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4677]], label [[IF_THEN4679:%.*]], label [[IF_END4680:%.*]]
+// SIMD-ONLY0:       if.then4679:
+// SIMD-ONLY0-NEXT:    [[TMP2575:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2575]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4680]]
+// SIMD-ONLY0:       if.end4680:
+// SIMD-ONLY0-NEXT:    [[TMP2576:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2577:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4681:%.*]] = fcmp ogt float [[TMP2576]], [[TMP2577]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4681]], label [[COND_TRUE4683:%.*]], label [[COND_FALSE4684:%.*]]
+// SIMD-ONLY0:       cond.true4683:
+// SIMD-ONLY0-NEXT:    [[TMP2578:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4685:%.*]]
+// SIMD-ONLY0:       cond.false4684:
+// SIMD-ONLY0-NEXT:    [[TMP2579:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4685]]
+// SIMD-ONLY0:       cond.end4685:
+// SIMD-ONLY0-NEXT:    [[COND4686:%.*]] = phi float [ [[TMP2578]], [[COND_TRUE4683]] ], [ [[TMP2579]], [[COND_FALSE4684]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4686]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2580:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2581:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4687:%.*]] = fcmp olt float [[TMP2580]], [[TMP2581]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4687]], label [[COND_TRUE4689:%.*]], label [[COND_FALSE4690:%.*]]
+// SIMD-ONLY0:       cond.true4689:
+// SIMD-ONLY0-NEXT:    [[TMP2582:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4691:%.*]]
+// SIMD-ONLY0:       cond.false4690:
+// SIMD-ONLY0-NEXT:    [[TMP2583:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4691]]
+// SIMD-ONLY0:       cond.end4691:
+// SIMD-ONLY0-NEXT:    [[COND4692:%.*]] = phi float [ [[TMP2582]], [[COND_TRUE4689]] ], [ [[TMP2583]], [[COND_FALSE4690]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4692]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2584:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2585:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4693:%.*]] = fcmp ogt float [[TMP2584]], [[TMP2585]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4693]], label [[COND_TRUE4695:%.*]], label [[COND_FALSE4696:%.*]]
+// SIMD-ONLY0:       cond.true4695:
+// SIMD-ONLY0-NEXT:    [[TMP2586:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4697:%.*]]
+// SIMD-ONLY0:       cond.false4696:
+// SIMD-ONLY0-NEXT:    [[TMP2587:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4697]]
+// SIMD-ONLY0:       cond.end4697:
+// SIMD-ONLY0-NEXT:    [[COND4698:%.*]] = phi float [ [[TMP2586]], [[COND_TRUE4695]] ], [ [[TMP2587]], [[COND_FALSE4696]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4698]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2588:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2589:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4699:%.*]] = fcmp olt float [[TMP2588]], [[TMP2589]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4699]], label [[COND_TRUE4701:%.*]], label [[COND_FALSE4702:%.*]]
+// SIMD-ONLY0:       cond.true4701:
+// SIMD-ONLY0-NEXT:    [[TMP2590:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4703:%.*]]
+// SIMD-ONLY0:       cond.false4702:
+// SIMD-ONLY0-NEXT:    [[TMP2591:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4703]]
+// SIMD-ONLY0:       cond.end4703:
+// SIMD-ONLY0-NEXT:    [[COND4704:%.*]] = phi float [ [[TMP2590]], [[COND_TRUE4701]] ], [ [[TMP2591]], [[COND_FALSE4702]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4704]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2592:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2593:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4705:%.*]] = fcmp ogt float [[TMP2592]], [[TMP2593]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4705]], label [[IF_THEN4707:%.*]], label [[IF_END4708:%.*]]
+// SIMD-ONLY0:       if.then4707:
+// SIMD-ONLY0-NEXT:    [[TMP2594:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2594]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4708]]
+// SIMD-ONLY0:       if.end4708:
+// SIMD-ONLY0-NEXT:    [[TMP2595:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2596:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4709:%.*]] = fcmp olt float [[TMP2595]], [[TMP2596]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4709]], label [[IF_THEN4711:%.*]], label [[IF_END4712:%.*]]
+// SIMD-ONLY0:       if.then4711:
+// SIMD-ONLY0-NEXT:    [[TMP2597:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2597]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4712]]
+// SIMD-ONLY0:       if.end4712:
+// SIMD-ONLY0-NEXT:    [[TMP2598:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2599:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4713:%.*]] = fcmp ogt float [[TMP2598]], [[TMP2599]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4713]], label [[IF_THEN4715:%.*]], label [[IF_END4716:%.*]]
+// SIMD-ONLY0:       if.then4715:
+// SIMD-ONLY0-NEXT:    [[TMP2600:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2600]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4716]]
+// SIMD-ONLY0:       if.end4716:
+// SIMD-ONLY0-NEXT:    [[TMP2601:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2602:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4717:%.*]] = fcmp olt float [[TMP2601]], [[TMP2602]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4717]], label [[IF_THEN4719:%.*]], label [[IF_END4720:%.*]]
+// SIMD-ONLY0:       if.then4719:
+// SIMD-ONLY0-NEXT:    [[TMP2603:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2603]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4720]]
+// SIMD-ONLY0:       if.end4720:
+// SIMD-ONLY0-NEXT:    [[TMP2604:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2605:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4721:%.*]] = fcmp ogt float [[TMP2604]], [[TMP2605]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4721]], label [[COND_TRUE4723:%.*]], label [[COND_FALSE4724:%.*]]
+// SIMD-ONLY0:       cond.true4723:
+// SIMD-ONLY0-NEXT:    [[TMP2606:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4725:%.*]]
+// SIMD-ONLY0:       cond.false4724:
+// SIMD-ONLY0-NEXT:    [[TMP2607:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4725]]
+// SIMD-ONLY0:       cond.end4725:
+// SIMD-ONLY0-NEXT:    [[COND4726:%.*]] = phi float [ [[TMP2606]], [[COND_TRUE4723]] ], [ [[TMP2607]], [[COND_FALSE4724]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4726]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2608:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2609:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4727:%.*]] = fcmp olt float [[TMP2608]], [[TMP2609]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4727]], label [[COND_TRUE4729:%.*]], label [[COND_FALSE4730:%.*]]
+// SIMD-ONLY0:       cond.true4729:
+// SIMD-ONLY0-NEXT:    [[TMP2610:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4731:%.*]]
+// SIMD-ONLY0:       cond.false4730:
+// SIMD-ONLY0-NEXT:    [[TMP2611:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4731]]
+// SIMD-ONLY0:       cond.end4731:
+// SIMD-ONLY0-NEXT:    [[COND4732:%.*]] = phi float [ [[TMP2610]], [[COND_TRUE4729]] ], [ [[TMP2611]], [[COND_FALSE4730]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4732]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2612:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2613:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4733:%.*]] = fcmp ogt float [[TMP2612]], [[TMP2613]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4733]], label [[COND_TRUE4735:%.*]], label [[COND_FALSE4736:%.*]]
+// SIMD-ONLY0:       cond.true4735:
+// SIMD-ONLY0-NEXT:    [[TMP2614:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4737:%.*]]
+// SIMD-ONLY0:       cond.false4736:
+// SIMD-ONLY0-NEXT:    [[TMP2615:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4737]]
+// SIMD-ONLY0:       cond.end4737:
+// SIMD-ONLY0-NEXT:    [[COND4738:%.*]] = phi float [ [[TMP2614]], [[COND_TRUE4735]] ], [ [[TMP2615]], [[COND_FALSE4736]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4738]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2616:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2617:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4739:%.*]] = fcmp olt float [[TMP2616]], [[TMP2617]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4739]], label [[COND_TRUE4741:%.*]], label [[COND_FALSE4742:%.*]]
+// SIMD-ONLY0:       cond.true4741:
+// SIMD-ONLY0-NEXT:    [[TMP2618:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4743:%.*]]
+// SIMD-ONLY0:       cond.false4742:
+// SIMD-ONLY0-NEXT:    [[TMP2619:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4743]]
+// SIMD-ONLY0:       cond.end4743:
+// SIMD-ONLY0-NEXT:    [[COND4744:%.*]] = phi float [ [[TMP2618]], [[COND_TRUE4741]] ], [ [[TMP2619]], [[COND_FALSE4742]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4744]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2620:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2621:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4745:%.*]] = fcmp ogt float [[TMP2620]], [[TMP2621]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4745]], label [[IF_THEN4747:%.*]], label [[IF_END4748:%.*]]
+// SIMD-ONLY0:       if.then4747:
+// SIMD-ONLY0-NEXT:    [[TMP2622:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2622]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4748]]
+// SIMD-ONLY0:       if.end4748:
+// SIMD-ONLY0-NEXT:    [[TMP2623:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2624:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4749:%.*]] = fcmp olt float [[TMP2623]], [[TMP2624]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4749]], label [[IF_THEN4751:%.*]], label [[IF_END4752:%.*]]
+// SIMD-ONLY0:       if.then4751:
+// SIMD-ONLY0-NEXT:    [[TMP2625:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2625]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4752]]
+// SIMD-ONLY0:       if.end4752:
+// SIMD-ONLY0-NEXT:    [[TMP2626:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2627:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4753:%.*]] = fcmp ogt float [[TMP2626]], [[TMP2627]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4753]], label [[IF_THEN4755:%.*]], label [[IF_END4756:%.*]]
+// SIMD-ONLY0:       if.then4755:
+// SIMD-ONLY0-NEXT:    [[TMP2628:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2628]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4756]]
+// SIMD-ONLY0:       if.end4756:
+// SIMD-ONLY0-NEXT:    [[TMP2629:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2630:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4757:%.*]] = fcmp olt float [[TMP2629]], [[TMP2630]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4757]], label [[IF_THEN4759:%.*]], label [[IF_END4760:%.*]]
+// SIMD-ONLY0:       if.then4759:
+// SIMD-ONLY0-NEXT:    [[TMP2631:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2631]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4760]]
+// SIMD-ONLY0:       if.end4760:
+// SIMD-ONLY0-NEXT:    [[TMP2632:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2633:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4761:%.*]] = fcmp ogt float [[TMP2632]], [[TMP2633]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4761]], label [[COND_TRUE4763:%.*]], label [[COND_FALSE4764:%.*]]
+// SIMD-ONLY0:       cond.true4763:
+// SIMD-ONLY0-NEXT:    [[TMP2634:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4765:%.*]]
+// SIMD-ONLY0:       cond.false4764:
+// SIMD-ONLY0-NEXT:    [[TMP2635:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4765]]
+// SIMD-ONLY0:       cond.end4765:
+// SIMD-ONLY0-NEXT:    [[COND4766:%.*]] = phi float [ [[TMP2634]], [[COND_TRUE4763]] ], [ [[TMP2635]], [[COND_FALSE4764]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4766]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2636:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2637:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4767:%.*]] = fcmp olt float [[TMP2636]], [[TMP2637]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4767]], label [[COND_TRUE4769:%.*]], label [[COND_FALSE4770:%.*]]
+// SIMD-ONLY0:       cond.true4769:
+// SIMD-ONLY0-NEXT:    [[TMP2638:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4771:%.*]]
+// SIMD-ONLY0:       cond.false4770:
+// SIMD-ONLY0-NEXT:    [[TMP2639:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4771]]
+// SIMD-ONLY0:       cond.end4771:
+// SIMD-ONLY0-NEXT:    [[COND4772:%.*]] = phi float [ [[TMP2638]], [[COND_TRUE4769]] ], [ [[TMP2639]], [[COND_FALSE4770]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4772]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2640:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2641:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4773:%.*]] = fcmp ogt float [[TMP2640]], [[TMP2641]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4773]], label [[COND_TRUE4775:%.*]], label [[COND_FALSE4776:%.*]]
+// SIMD-ONLY0:       cond.true4775:
+// SIMD-ONLY0-NEXT:    [[TMP2642:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4777:%.*]]
+// SIMD-ONLY0:       cond.false4776:
+// SIMD-ONLY0-NEXT:    [[TMP2643:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4777]]
+// SIMD-ONLY0:       cond.end4777:
+// SIMD-ONLY0-NEXT:    [[COND4778:%.*]] = phi float [ [[TMP2642]], [[COND_TRUE4775]] ], [ [[TMP2643]], [[COND_FALSE4776]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4778]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2644:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2645:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4779:%.*]] = fcmp olt float [[TMP2644]], [[TMP2645]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4779]], label [[COND_TRUE4781:%.*]], label [[COND_FALSE4782:%.*]]
+// SIMD-ONLY0:       cond.true4781:
+// SIMD-ONLY0-NEXT:    [[TMP2646:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4783:%.*]]
+// SIMD-ONLY0:       cond.false4782:
+// SIMD-ONLY0-NEXT:    [[TMP2647:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4783]]
+// SIMD-ONLY0:       cond.end4783:
+// SIMD-ONLY0-NEXT:    [[COND4784:%.*]] = phi float [ [[TMP2646]], [[COND_TRUE4781]] ], [ [[TMP2647]], [[COND_FALSE4782]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4784]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2648:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2649:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4785:%.*]] = fcmp ogt float [[TMP2648]], [[TMP2649]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4785]], label [[IF_THEN4787:%.*]], label [[IF_END4788:%.*]]
+// SIMD-ONLY0:       if.then4787:
+// SIMD-ONLY0-NEXT:    [[TMP2650:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2650]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4788]]
+// SIMD-ONLY0:       if.end4788:
+// SIMD-ONLY0-NEXT:    [[TMP2651:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2652:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4789:%.*]] = fcmp olt float [[TMP2651]], [[TMP2652]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4789]], label [[IF_THEN4791:%.*]], label [[IF_END4792:%.*]]
+// SIMD-ONLY0:       if.then4791:
+// SIMD-ONLY0-NEXT:    [[TMP2653:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2653]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4792]]
+// SIMD-ONLY0:       if.end4792:
+// SIMD-ONLY0-NEXT:    [[TMP2654:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2655:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4793:%.*]] = fcmp ogt float [[TMP2654]], [[TMP2655]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4793]], label [[IF_THEN4795:%.*]], label [[IF_END4796:%.*]]
+// SIMD-ONLY0:       if.then4795:
+// SIMD-ONLY0-NEXT:    [[TMP2656:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2656]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4796]]
+// SIMD-ONLY0:       if.end4796:
+// SIMD-ONLY0-NEXT:    [[TMP2657:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2658:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4797:%.*]] = fcmp olt float [[TMP2657]], [[TMP2658]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4797]], label [[IF_THEN4799:%.*]], label [[IF_END4800:%.*]]
+// SIMD-ONLY0:       if.then4799:
+// SIMD-ONLY0-NEXT:    [[TMP2659:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2659]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4800]]
+// SIMD-ONLY0:       if.end4800:
+// SIMD-ONLY0-NEXT:    [[TMP2660:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2661:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4801:%.*]] = fcmp ogt float [[TMP2660]], [[TMP2661]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4801]], label [[COND_TRUE4803:%.*]], label [[COND_FALSE4804:%.*]]
+// SIMD-ONLY0:       cond.true4803:
+// SIMD-ONLY0-NEXT:    [[TMP2662:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4805:%.*]]
+// SIMD-ONLY0:       cond.false4804:
+// SIMD-ONLY0-NEXT:    [[TMP2663:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4805]]
+// SIMD-ONLY0:       cond.end4805:
+// SIMD-ONLY0-NEXT:    [[COND4806:%.*]] = phi float [ [[TMP2662]], [[COND_TRUE4803]] ], [ [[TMP2663]], [[COND_FALSE4804]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4806]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2664:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2665:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4807:%.*]] = fcmp olt float [[TMP2664]], [[TMP2665]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4807]], label [[COND_TRUE4809:%.*]], label [[COND_FALSE4810:%.*]]
+// SIMD-ONLY0:       cond.true4809:
+// SIMD-ONLY0-NEXT:    [[TMP2666:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4811:%.*]]
+// SIMD-ONLY0:       cond.false4810:
+// SIMD-ONLY0-NEXT:    [[TMP2667:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4811]]
+// SIMD-ONLY0:       cond.end4811:
+// SIMD-ONLY0-NEXT:    [[COND4812:%.*]] = phi float [ [[TMP2666]], [[COND_TRUE4809]] ], [ [[TMP2667]], [[COND_FALSE4810]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4812]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2668:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2669:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4813:%.*]] = fcmp ogt float [[TMP2668]], [[TMP2669]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4813]], label [[COND_TRUE4815:%.*]], label [[COND_FALSE4816:%.*]]
+// SIMD-ONLY0:       cond.true4815:
+// SIMD-ONLY0-NEXT:    [[TMP2670:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4817:%.*]]
+// SIMD-ONLY0:       cond.false4816:
+// SIMD-ONLY0-NEXT:    [[TMP2671:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4817]]
+// SIMD-ONLY0:       cond.end4817:
+// SIMD-ONLY0-NEXT:    [[COND4818:%.*]] = phi float [ [[TMP2670]], [[COND_TRUE4815]] ], [ [[TMP2671]], [[COND_FALSE4816]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4818]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2672:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2673:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4819:%.*]] = fcmp olt float [[TMP2672]], [[TMP2673]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4819]], label [[COND_TRUE4821:%.*]], label [[COND_FALSE4822:%.*]]
+// SIMD-ONLY0:       cond.true4821:
+// SIMD-ONLY0-NEXT:    [[TMP2674:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4823:%.*]]
+// SIMD-ONLY0:       cond.false4822:
+// SIMD-ONLY0-NEXT:    [[TMP2675:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4823]]
+// SIMD-ONLY0:       cond.end4823:
+// SIMD-ONLY0-NEXT:    [[COND4824:%.*]] = phi float [ [[TMP2674]], [[COND_TRUE4821]] ], [ [[TMP2675]], [[COND_FALSE4822]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND4824]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2676:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2677:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4825:%.*]] = fcmp ogt float [[TMP2676]], [[TMP2677]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4825]], label [[IF_THEN4827:%.*]], label [[IF_END4828:%.*]]
+// SIMD-ONLY0:       if.then4827:
+// SIMD-ONLY0-NEXT:    [[TMP2678:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2678]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4828]]
+// SIMD-ONLY0:       if.end4828:
+// SIMD-ONLY0-NEXT:    [[TMP2679:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2680:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4829:%.*]] = fcmp olt float [[TMP2679]], [[TMP2680]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4829]], label [[IF_THEN4831:%.*]], label [[IF_END4832:%.*]]
+// SIMD-ONLY0:       if.then4831:
+// SIMD-ONLY0-NEXT:    [[TMP2681:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2681]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4832]]
+// SIMD-ONLY0:       if.end4832:
+// SIMD-ONLY0-NEXT:    [[TMP2682:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2683:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4833:%.*]] = fcmp ogt float [[TMP2682]], [[TMP2683]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4833]], label [[IF_THEN4835:%.*]], label [[IF_END4836:%.*]]
+// SIMD-ONLY0:       if.then4835:
+// SIMD-ONLY0-NEXT:    [[TMP2684:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2684]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4836]]
+// SIMD-ONLY0:       if.end4836:
+// SIMD-ONLY0-NEXT:    [[TMP2685:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2686:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP4837:%.*]] = fcmp olt float [[TMP2685]], [[TMP2686]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4837]], label [[IF_THEN4839:%.*]], label [[IF_END4840:%.*]]
+// SIMD-ONLY0:       if.then4839:
+// SIMD-ONLY0-NEXT:    [[TMP2687:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP2687]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END4840]]
+// SIMD-ONLY0:       if.end4840:
+// SIMD-ONLY0-NEXT:    [[TMP2688:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2689:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4841:%.*]] = fcmp ogt double [[TMP2688]], [[TMP2689]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4841]], label [[COND_TRUE4843:%.*]], label [[COND_FALSE4844:%.*]]
+// SIMD-ONLY0:       cond.true4843:
+// SIMD-ONLY0-NEXT:    [[TMP2690:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4845:%.*]]
+// SIMD-ONLY0:       cond.false4844:
+// SIMD-ONLY0-NEXT:    [[TMP2691:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4845]]
+// SIMD-ONLY0:       cond.end4845:
+// SIMD-ONLY0-NEXT:    [[COND4846:%.*]] = phi double [ [[TMP2690]], [[COND_TRUE4843]] ], [ [[TMP2691]], [[COND_FALSE4844]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND4846]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2692:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2693:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4847:%.*]] = fcmp olt double [[TMP2692]], [[TMP2693]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4847]], label [[COND_TRUE4849:%.*]], label [[COND_FALSE4850:%.*]]
+// SIMD-ONLY0:       cond.true4849:
+// SIMD-ONLY0-NEXT:    [[TMP2694:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4851:%.*]]
+// SIMD-ONLY0:       cond.false4850:
+// SIMD-ONLY0-NEXT:    [[TMP2695:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4851]]
+// SIMD-ONLY0:       cond.end4851:
+// SIMD-ONLY0-NEXT:    [[COND4852:%.*]] = phi double [ [[TMP2694]], [[COND_TRUE4849]] ], [ [[TMP2695]], [[COND_FALSE4850]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND4852]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2696:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2697:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4853:%.*]] = fcmp ogt double [[TMP2696]], [[TMP2697]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4853]], label [[COND_TRUE4855:%.*]], label [[COND_FALSE4856:%.*]]
+// SIMD-ONLY0:       cond.true4855:
+// SIMD-ONLY0-NEXT:    [[TMP2698:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4857:%.*]]
+// SIMD-ONLY0:       cond.false4856:
+// SIMD-ONLY0-NEXT:    [[TMP2699:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4857]]
+// SIMD-ONLY0:       cond.end4857:
+// SIMD-ONLY0-NEXT:    [[COND4858:%.*]] = phi double [ [[TMP2698]], [[COND_TRUE4855]] ], [ [[TMP2699]], [[COND_FALSE4856]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND4858]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2700:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2701:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4859:%.*]] = fcmp olt double [[TMP2700]], [[TMP2701]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4859]], label [[COND_TRUE4861:%.*]], label [[COND_FALSE4862:%.*]]
+// SIMD-ONLY0:       cond.true4861:
+// SIMD-ONLY0-NEXT:    [[TMP2702:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4863:%.*]]
+// SIMD-ONLY0:       cond.false4862:
+// SIMD-ONLY0-NEXT:    [[TMP2703:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4863]]
+// SIMD-ONLY0:       cond.end4863:
+// SIMD-ONLY0-NEXT:    [[COND4864:%.*]] = phi double [ [[TMP2702]], [[COND_TRUE4861]] ], [ [[TMP2703]], [[COND_FALSE4862]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND4864]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2704:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2705:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4865:%.*]] = fcmp ogt double [[TMP2704]], [[TMP2705]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4865]], label [[IF_THEN4867:%.*]], label [[IF_END4868:%.*]]
+// SIMD-ONLY0:       if.then4867:
+// SIMD-ONLY0-NEXT:    [[TMP2706:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2706]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4868]]
+// SIMD-ONLY0:       if.end4868:
+// SIMD-ONLY0-NEXT:    [[TMP2707:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2708:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4869:%.*]] = fcmp olt double [[TMP2707]], [[TMP2708]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4869]], label [[IF_THEN4871:%.*]], label [[IF_END4872:%.*]]
+// SIMD-ONLY0:       if.then4871:
+// SIMD-ONLY0-NEXT:    [[TMP2709:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2709]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4872]]
+// SIMD-ONLY0:       if.end4872:
+// SIMD-ONLY0-NEXT:    [[TMP2710:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2711:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4873:%.*]] = fcmp ogt double [[TMP2710]], [[TMP2711]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4873]], label [[IF_THEN4875:%.*]], label [[IF_END4876:%.*]]
+// SIMD-ONLY0:       if.then4875:
+// SIMD-ONLY0-NEXT:    [[TMP2712:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2712]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4876]]
+// SIMD-ONLY0:       if.end4876:
+// SIMD-ONLY0-NEXT:    [[TMP2713:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2714:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4877:%.*]] = fcmp olt double [[TMP2713]], [[TMP2714]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4877]], label [[IF_THEN4879:%.*]], label [[IF_END4880:%.*]]
+// SIMD-ONLY0:       if.then4879:
+// SIMD-ONLY0-NEXT:    [[TMP2715:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2715]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4880]]
+// SIMD-ONLY0:       if.end4880:
+// SIMD-ONLY0-NEXT:    [[TMP2716:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2717:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4881:%.*]] = fcmp ogt double [[TMP2716]], [[TMP2717]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4881]], label [[COND_TRUE4883:%.*]], label [[COND_FALSE4884:%.*]]
+// SIMD-ONLY0:       cond.true4883:
+// SIMD-ONLY0-NEXT:    [[TMP2718:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4885:%.*]]
+// SIMD-ONLY0:       cond.false4884:
+// SIMD-ONLY0-NEXT:    [[TMP2719:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4885]]
+// SIMD-ONLY0:       cond.end4885:
+// SIMD-ONLY0-NEXT:    [[COND4886:%.*]] = phi double [ [[TMP2718]], [[COND_TRUE4883]] ], [ [[TMP2719]], [[COND_FALSE4884]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND4886]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2720:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2721:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4887:%.*]] = fcmp olt double [[TMP2720]], [[TMP2721]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4887]], label [[COND_TRUE4889:%.*]], label [[COND_FALSE4890:%.*]]
+// SIMD-ONLY0:       cond.true4889:
+// SIMD-ONLY0-NEXT:    [[TMP2722:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4891:%.*]]
+// SIMD-ONLY0:       cond.false4890:
+// SIMD-ONLY0-NEXT:    [[TMP2723:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4891]]
+// SIMD-ONLY0:       cond.end4891:
+// SIMD-ONLY0-NEXT:    [[COND4892:%.*]] = phi double [ [[TMP2722]], [[COND_TRUE4889]] ], [ [[TMP2723]], [[COND_FALSE4890]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND4892]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2724:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2725:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4893:%.*]] = fcmp ogt double [[TMP2724]], [[TMP2725]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4893]], label [[COND_TRUE4895:%.*]], label [[COND_FALSE4896:%.*]]
+// SIMD-ONLY0:       cond.true4895:
+// SIMD-ONLY0-NEXT:    [[TMP2726:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4897:%.*]]
+// SIMD-ONLY0:       cond.false4896:
+// SIMD-ONLY0-NEXT:    [[TMP2727:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4897]]
+// SIMD-ONLY0:       cond.end4897:
+// SIMD-ONLY0-NEXT:    [[COND4898:%.*]] = phi double [ [[TMP2726]], [[COND_TRUE4895]] ], [ [[TMP2727]], [[COND_FALSE4896]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND4898]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2728:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2729:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4899:%.*]] = fcmp olt double [[TMP2728]], [[TMP2729]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4899]], label [[COND_TRUE4901:%.*]], label [[COND_FALSE4902:%.*]]
+// SIMD-ONLY0:       cond.true4901:
+// SIMD-ONLY0-NEXT:    [[TMP2730:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4903:%.*]]
+// SIMD-ONLY0:       cond.false4902:
+// SIMD-ONLY0-NEXT:    [[TMP2731:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4903]]
+// SIMD-ONLY0:       cond.end4903:
+// SIMD-ONLY0-NEXT:    [[COND4904:%.*]] = phi double [ [[TMP2730]], [[COND_TRUE4901]] ], [ [[TMP2731]], [[COND_FALSE4902]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND4904]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2732:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2733:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4905:%.*]] = fcmp ogt double [[TMP2732]], [[TMP2733]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4905]], label [[IF_THEN4907:%.*]], label [[IF_END4908:%.*]]
+// SIMD-ONLY0:       if.then4907:
+// SIMD-ONLY0-NEXT:    [[TMP2734:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2734]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4908]]
+// SIMD-ONLY0:       if.end4908:
+// SIMD-ONLY0-NEXT:    [[TMP2735:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2736:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4909:%.*]] = fcmp olt double [[TMP2735]], [[TMP2736]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4909]], label [[IF_THEN4911:%.*]], label [[IF_END4912:%.*]]
+// SIMD-ONLY0:       if.then4911:
+// SIMD-ONLY0-NEXT:    [[TMP2737:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2737]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4912]]
+// SIMD-ONLY0:       if.end4912:
+// SIMD-ONLY0-NEXT:    [[TMP2738:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2739:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4913:%.*]] = fcmp ogt double [[TMP2738]], [[TMP2739]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4913]], label [[IF_THEN4915:%.*]], label [[IF_END4916:%.*]]
+// SIMD-ONLY0:       if.then4915:
+// SIMD-ONLY0-NEXT:    [[TMP2740:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2740]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4916]]
+// SIMD-ONLY0:       if.end4916:
+// SIMD-ONLY0-NEXT:    [[TMP2741:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2742:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4917:%.*]] = fcmp olt double [[TMP2741]], [[TMP2742]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4917]], label [[IF_THEN4919:%.*]], label [[IF_END4920:%.*]]
+// SIMD-ONLY0:       if.then4919:
+// SIMD-ONLY0-NEXT:    [[TMP2743:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2743]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4920]]
+// SIMD-ONLY0:       if.end4920:
+// SIMD-ONLY0-NEXT:    [[TMP2744:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2745:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4921:%.*]] = fcmp ogt double [[TMP2744]], [[TMP2745]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4921]], label [[COND_TRUE4923:%.*]], label [[COND_FALSE4924:%.*]]
+// SIMD-ONLY0:       cond.true4923:
+// SIMD-ONLY0-NEXT:    [[TMP2746:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4925:%.*]]
+// SIMD-ONLY0:       cond.false4924:
+// SIMD-ONLY0-NEXT:    [[TMP2747:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4925]]
+// SIMD-ONLY0:       cond.end4925:
+// SIMD-ONLY0-NEXT:    [[COND4926:%.*]] = phi double [ [[TMP2746]], [[COND_TRUE4923]] ], [ [[TMP2747]], [[COND_FALSE4924]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND4926]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2748:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2749:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4927:%.*]] = fcmp olt double [[TMP2748]], [[TMP2749]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4927]], label [[COND_TRUE4929:%.*]], label [[COND_FALSE4930:%.*]]
+// SIMD-ONLY0:       cond.true4929:
+// SIMD-ONLY0-NEXT:    [[TMP2750:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4931:%.*]]
+// SIMD-ONLY0:       cond.false4930:
+// SIMD-ONLY0-NEXT:    [[TMP2751:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4931]]
+// SIMD-ONLY0:       cond.end4931:
+// SIMD-ONLY0-NEXT:    [[COND4932:%.*]] = phi double [ [[TMP2750]], [[COND_TRUE4929]] ], [ [[TMP2751]], [[COND_FALSE4930]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND4932]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2752:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2753:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4933:%.*]] = fcmp ogt double [[TMP2752]], [[TMP2753]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4933]], label [[COND_TRUE4935:%.*]], label [[COND_FALSE4936:%.*]]
+// SIMD-ONLY0:       cond.true4935:
+// SIMD-ONLY0-NEXT:    [[TMP2754:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4937:%.*]]
+// SIMD-ONLY0:       cond.false4936:
+// SIMD-ONLY0-NEXT:    [[TMP2755:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4937]]
+// SIMD-ONLY0:       cond.end4937:
+// SIMD-ONLY0-NEXT:    [[COND4938:%.*]] = phi double [ [[TMP2754]], [[COND_TRUE4935]] ], [ [[TMP2755]], [[COND_FALSE4936]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND4938]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2756:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2757:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4939:%.*]] = fcmp olt double [[TMP2756]], [[TMP2757]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4939]], label [[COND_TRUE4941:%.*]], label [[COND_FALSE4942:%.*]]
+// SIMD-ONLY0:       cond.true4941:
+// SIMD-ONLY0-NEXT:    [[TMP2758:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4943:%.*]]
+// SIMD-ONLY0:       cond.false4942:
+// SIMD-ONLY0-NEXT:    [[TMP2759:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4943]]
+// SIMD-ONLY0:       cond.end4943:
+// SIMD-ONLY0-NEXT:    [[COND4944:%.*]] = phi double [ [[TMP2758]], [[COND_TRUE4941]] ], [ [[TMP2759]], [[COND_FALSE4942]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND4944]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2760:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2761:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4945:%.*]] = fcmp ogt double [[TMP2760]], [[TMP2761]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4945]], label [[IF_THEN4947:%.*]], label [[IF_END4948:%.*]]
+// SIMD-ONLY0:       if.then4947:
+// SIMD-ONLY0-NEXT:    [[TMP2762:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2762]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4948]]
+// SIMD-ONLY0:       if.end4948:
+// SIMD-ONLY0-NEXT:    [[TMP2763:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2764:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4949:%.*]] = fcmp olt double [[TMP2763]], [[TMP2764]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4949]], label [[IF_THEN4951:%.*]], label [[IF_END4952:%.*]]
+// SIMD-ONLY0:       if.then4951:
+// SIMD-ONLY0-NEXT:    [[TMP2765:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2765]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4952]]
+// SIMD-ONLY0:       if.end4952:
+// SIMD-ONLY0-NEXT:    [[TMP2766:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2767:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4953:%.*]] = fcmp ogt double [[TMP2766]], [[TMP2767]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4953]], label [[IF_THEN4955:%.*]], label [[IF_END4956:%.*]]
+// SIMD-ONLY0:       if.then4955:
+// SIMD-ONLY0-NEXT:    [[TMP2768:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2768]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4956]]
+// SIMD-ONLY0:       if.end4956:
+// SIMD-ONLY0-NEXT:    [[TMP2769:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2770:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4957:%.*]] = fcmp olt double [[TMP2769]], [[TMP2770]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4957]], label [[IF_THEN4959:%.*]], label [[IF_END4960:%.*]]
+// SIMD-ONLY0:       if.then4959:
+// SIMD-ONLY0-NEXT:    [[TMP2771:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2771]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4960]]
+// SIMD-ONLY0:       if.end4960:
+// SIMD-ONLY0-NEXT:    [[TMP2772:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2773:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4961:%.*]] = fcmp ogt double [[TMP2772]], [[TMP2773]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4961]], label [[COND_TRUE4963:%.*]], label [[COND_FALSE4964:%.*]]
+// SIMD-ONLY0:       cond.true4963:
+// SIMD-ONLY0-NEXT:    [[TMP2774:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4965:%.*]]
+// SIMD-ONLY0:       cond.false4964:
+// SIMD-ONLY0-NEXT:    [[TMP2775:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4965]]
+// SIMD-ONLY0:       cond.end4965:
+// SIMD-ONLY0-NEXT:    [[COND4966:%.*]] = phi double [ [[TMP2774]], [[COND_TRUE4963]] ], [ [[TMP2775]], [[COND_FALSE4964]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND4966]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2776:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2777:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4967:%.*]] = fcmp olt double [[TMP2776]], [[TMP2777]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4967]], label [[COND_TRUE4969:%.*]], label [[COND_FALSE4970:%.*]]
+// SIMD-ONLY0:       cond.true4969:
+// SIMD-ONLY0-NEXT:    [[TMP2778:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4971:%.*]]
+// SIMD-ONLY0:       cond.false4970:
+// SIMD-ONLY0-NEXT:    [[TMP2779:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4971]]
+// SIMD-ONLY0:       cond.end4971:
+// SIMD-ONLY0-NEXT:    [[COND4972:%.*]] = phi double [ [[TMP2778]], [[COND_TRUE4969]] ], [ [[TMP2779]], [[COND_FALSE4970]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND4972]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2780:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2781:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4973:%.*]] = fcmp ogt double [[TMP2780]], [[TMP2781]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4973]], label [[COND_TRUE4975:%.*]], label [[COND_FALSE4976:%.*]]
+// SIMD-ONLY0:       cond.true4975:
+// SIMD-ONLY0-NEXT:    [[TMP2782:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4977:%.*]]
+// SIMD-ONLY0:       cond.false4976:
+// SIMD-ONLY0-NEXT:    [[TMP2783:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4977]]
+// SIMD-ONLY0:       cond.end4977:
+// SIMD-ONLY0-NEXT:    [[COND4978:%.*]] = phi double [ [[TMP2782]], [[COND_TRUE4975]] ], [ [[TMP2783]], [[COND_FALSE4976]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND4978]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2784:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2785:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4979:%.*]] = fcmp olt double [[TMP2784]], [[TMP2785]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4979]], label [[COND_TRUE4981:%.*]], label [[COND_FALSE4982:%.*]]
+// SIMD-ONLY0:       cond.true4981:
+// SIMD-ONLY0-NEXT:    [[TMP2786:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4983:%.*]]
+// SIMD-ONLY0:       cond.false4982:
+// SIMD-ONLY0-NEXT:    [[TMP2787:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4983]]
+// SIMD-ONLY0:       cond.end4983:
+// SIMD-ONLY0-NEXT:    [[COND4984:%.*]] = phi double [ [[TMP2786]], [[COND_TRUE4981]] ], [ [[TMP2787]], [[COND_FALSE4982]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND4984]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2788:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2789:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4985:%.*]] = fcmp ogt double [[TMP2788]], [[TMP2789]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4985]], label [[IF_THEN4987:%.*]], label [[IF_END4988:%.*]]
+// SIMD-ONLY0:       if.then4987:
+// SIMD-ONLY0-NEXT:    [[TMP2790:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2790]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4988]]
+// SIMD-ONLY0:       if.end4988:
+// SIMD-ONLY0-NEXT:    [[TMP2791:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2792:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4989:%.*]] = fcmp olt double [[TMP2791]], [[TMP2792]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4989]], label [[IF_THEN4991:%.*]], label [[IF_END4992:%.*]]
+// SIMD-ONLY0:       if.then4991:
+// SIMD-ONLY0-NEXT:    [[TMP2793:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2793]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4992]]
+// SIMD-ONLY0:       if.end4992:
+// SIMD-ONLY0-NEXT:    [[TMP2794:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2795:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4993:%.*]] = fcmp ogt double [[TMP2794]], [[TMP2795]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4993]], label [[IF_THEN4995:%.*]], label [[IF_END4996:%.*]]
+// SIMD-ONLY0:       if.then4995:
+// SIMD-ONLY0-NEXT:    [[TMP2796:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2796]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4996]]
+// SIMD-ONLY0:       if.end4996:
+// SIMD-ONLY0-NEXT:    [[TMP2797:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2798:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4997:%.*]] = fcmp olt double [[TMP2797]], [[TMP2798]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4997]], label [[IF_THEN4999:%.*]], label [[IF_END5000:%.*]]
+// SIMD-ONLY0:       if.then4999:
+// SIMD-ONLY0-NEXT:    [[TMP2799:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2799]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5000]]
+// SIMD-ONLY0:       if.end5000:
+// SIMD-ONLY0-NEXT:    [[TMP2800:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2801:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5001:%.*]] = fcmp ogt double [[TMP2800]], [[TMP2801]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5001]], label [[COND_TRUE5003:%.*]], label [[COND_FALSE5004:%.*]]
+// SIMD-ONLY0:       cond.true5003:
+// SIMD-ONLY0-NEXT:    [[TMP2802:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END5005:%.*]]
+// SIMD-ONLY0:       cond.false5004:
+// SIMD-ONLY0-NEXT:    [[TMP2803:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END5005]]
+// SIMD-ONLY0:       cond.end5005:
+// SIMD-ONLY0-NEXT:    [[COND5006:%.*]] = phi double [ [[TMP2802]], [[COND_TRUE5003]] ], [ [[TMP2803]], [[COND_FALSE5004]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND5006]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2804:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2805:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5007:%.*]] = fcmp olt double [[TMP2804]], [[TMP2805]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5007]], label [[COND_TRUE5009:%.*]], label [[COND_FALSE5010:%.*]]
+// SIMD-ONLY0:       cond.true5009:
+// SIMD-ONLY0-NEXT:    [[TMP2806:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END5011:%.*]]
+// SIMD-ONLY0:       cond.false5010:
+// SIMD-ONLY0-NEXT:    [[TMP2807:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END5011]]
+// SIMD-ONLY0:       cond.end5011:
+// SIMD-ONLY0-NEXT:    [[COND5012:%.*]] = phi double [ [[TMP2806]], [[COND_TRUE5009]] ], [ [[TMP2807]], [[COND_FALSE5010]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND5012]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2808:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2809:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5013:%.*]] = fcmp ogt double [[TMP2808]], [[TMP2809]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5013]], label [[COND_TRUE5015:%.*]], label [[COND_FALSE5016:%.*]]
+// SIMD-ONLY0:       cond.true5015:
+// SIMD-ONLY0-NEXT:    [[TMP2810:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END5017:%.*]]
+// SIMD-ONLY0:       cond.false5016:
+// SIMD-ONLY0-NEXT:    [[TMP2811:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END5017]]
+// SIMD-ONLY0:       cond.end5017:
+// SIMD-ONLY0-NEXT:    [[COND5018:%.*]] = phi double [ [[TMP2810]], [[COND_TRUE5015]] ], [ [[TMP2811]], [[COND_FALSE5016]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND5018]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2812:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2813:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5019:%.*]] = fcmp olt double [[TMP2812]], [[TMP2813]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5019]], label [[COND_TRUE5021:%.*]], label [[COND_FALSE5022:%.*]]
+// SIMD-ONLY0:       cond.true5021:
+// SIMD-ONLY0-NEXT:    [[TMP2814:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END5023:%.*]]
+// SIMD-ONLY0:       cond.false5022:
+// SIMD-ONLY0-NEXT:    [[TMP2815:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END5023]]
+// SIMD-ONLY0:       cond.end5023:
+// SIMD-ONLY0-NEXT:    [[COND5024:%.*]] = phi double [ [[TMP2814]], [[COND_TRUE5021]] ], [ [[TMP2815]], [[COND_FALSE5022]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND5024]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2816:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2817:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5025:%.*]] = fcmp ogt double [[TMP2816]], [[TMP2817]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5025]], label [[IF_THEN5027:%.*]], label [[IF_END5028:%.*]]
+// SIMD-ONLY0:       if.then5027:
+// SIMD-ONLY0-NEXT:    [[TMP2818:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2818]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5028]]
+// SIMD-ONLY0:       if.end5028:
+// SIMD-ONLY0-NEXT:    [[TMP2819:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2820:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5029:%.*]] = fcmp olt double [[TMP2819]], [[TMP2820]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5029]], label [[IF_THEN5031:%.*]], label [[IF_END5032:%.*]]
+// SIMD-ONLY0:       if.then5031:
+// SIMD-ONLY0-NEXT:    [[TMP2821:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2821]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5032]]
+// SIMD-ONLY0:       if.end5032:
+// SIMD-ONLY0-NEXT:    [[TMP2822:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2823:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5033:%.*]] = fcmp ogt double [[TMP2822]], [[TMP2823]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5033]], label [[IF_THEN5035:%.*]], label [[IF_END5036:%.*]]
+// SIMD-ONLY0:       if.then5035:
+// SIMD-ONLY0-NEXT:    [[TMP2824:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2824]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5036]]
+// SIMD-ONLY0:       if.end5036:
+// SIMD-ONLY0-NEXT:    [[TMP2825:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2826:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5037:%.*]] = fcmp olt double [[TMP2825]], [[TMP2826]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5037]], label [[IF_THEN5039:%.*]], label [[IF_END5040:%.*]]
+// SIMD-ONLY0:       if.then5039:
+// SIMD-ONLY0-NEXT:    [[TMP2827:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2827]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5040]]
+// SIMD-ONLY0:       if.end5040:
+// SIMD-ONLY0-NEXT:    [[TMP2828:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2829:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5041:%.*]] = fcmp ogt double [[TMP2828]], [[TMP2829]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5041]], label [[COND_TRUE5043:%.*]], label [[COND_FALSE5044:%.*]]
+// SIMD-ONLY0:       cond.true5043:
+// SIMD-ONLY0-NEXT:    [[TMP2830:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END5045:%.*]]
+// SIMD-ONLY0:       cond.false5044:
+// SIMD-ONLY0-NEXT:    [[TMP2831:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END5045]]
+// SIMD-ONLY0:       cond.end5045:
+// SIMD-ONLY0-NEXT:    [[COND5046:%.*]] = phi double [ [[TMP2830]], [[COND_TRUE5043]] ], [ [[TMP2831]], [[COND_FALSE5044]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND5046]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2832:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2833:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5047:%.*]] = fcmp olt double [[TMP2832]], [[TMP2833]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5047]], label [[COND_TRUE5049:%.*]], label [[COND_FALSE5050:%.*]]
+// SIMD-ONLY0:       cond.true5049:
+// SIMD-ONLY0-NEXT:    [[TMP2834:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END5051:%.*]]
+// SIMD-ONLY0:       cond.false5050:
+// SIMD-ONLY0-NEXT:    [[TMP2835:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END5051]]
+// SIMD-ONLY0:       cond.end5051:
+// SIMD-ONLY0-NEXT:    [[COND5052:%.*]] = phi double [ [[TMP2834]], [[COND_TRUE5049]] ], [ [[TMP2835]], [[COND_FALSE5050]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND5052]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2836:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2837:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5053:%.*]] = fcmp ogt double [[TMP2836]], [[TMP2837]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5053]], label [[COND_TRUE5055:%.*]], label [[COND_FALSE5056:%.*]]
+// SIMD-ONLY0:       cond.true5055:
+// SIMD-ONLY0-NEXT:    [[TMP2838:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END5057:%.*]]
+// SIMD-ONLY0:       cond.false5056:
+// SIMD-ONLY0-NEXT:    [[TMP2839:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END5057]]
+// SIMD-ONLY0:       cond.end5057:
+// SIMD-ONLY0-NEXT:    [[COND5058:%.*]] = phi double [ [[TMP2838]], [[COND_TRUE5055]] ], [ [[TMP2839]], [[COND_FALSE5056]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND5058]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2840:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2841:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5059:%.*]] = fcmp olt double [[TMP2840]], [[TMP2841]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5059]], label [[COND_TRUE5061:%.*]], label [[COND_FALSE5062:%.*]]
+// SIMD-ONLY0:       cond.true5061:
+// SIMD-ONLY0-NEXT:    [[TMP2842:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END5063:%.*]]
+// SIMD-ONLY0:       cond.false5062:
+// SIMD-ONLY0-NEXT:    [[TMP2843:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END5063]]
+// SIMD-ONLY0:       cond.end5063:
+// SIMD-ONLY0-NEXT:    [[COND5064:%.*]] = phi double [ [[TMP2842]], [[COND_TRUE5061]] ], [ [[TMP2843]], [[COND_FALSE5062]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND5064]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2844:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2845:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5065:%.*]] = fcmp ogt double [[TMP2844]], [[TMP2845]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5065]], label [[IF_THEN5067:%.*]], label [[IF_END5068:%.*]]
+// SIMD-ONLY0:       if.then5067:
+// SIMD-ONLY0-NEXT:    [[TMP2846:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2846]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5068]]
+// SIMD-ONLY0:       if.end5068:
+// SIMD-ONLY0-NEXT:    [[TMP2847:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2848:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5069:%.*]] = fcmp olt double [[TMP2847]], [[TMP2848]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5069]], label [[IF_THEN5071:%.*]], label [[IF_END5072:%.*]]
+// SIMD-ONLY0:       if.then5071:
+// SIMD-ONLY0-NEXT:    [[TMP2849:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2849]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5072]]
+// SIMD-ONLY0:       if.end5072:
+// SIMD-ONLY0-NEXT:    [[TMP2850:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2851:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5073:%.*]] = fcmp ogt double [[TMP2850]], [[TMP2851]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5073]], label [[IF_THEN5075:%.*]], label [[IF_END5076:%.*]]
+// SIMD-ONLY0:       if.then5075:
+// SIMD-ONLY0-NEXT:    [[TMP2852:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2852]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5076]]
+// SIMD-ONLY0:       if.end5076:
+// SIMD-ONLY0-NEXT:    [[TMP2853:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2854:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5077:%.*]] = fcmp olt double [[TMP2853]], [[TMP2854]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5077]], label [[IF_THEN5079:%.*]], label [[IF_END5080:%.*]]
+// SIMD-ONLY0:       if.then5079:
+// SIMD-ONLY0-NEXT:    [[TMP2855:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP2855]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5080]]
+// SIMD-ONLY0:       if.end5080:
+// SIMD-ONLY0-NEXT:    ret void
+//
+//
+// SIMD-ONLY0-LABEL: @bar(
+// SIMD-ONLY0-NEXT:  entry:
+// SIMD-ONLY0-NEXT:    [[CX:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[CV:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[CR:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[CE:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[CD:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[UCX:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[UCV:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[UCR:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[UCE:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[UCD:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[SX:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[SV:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[SR:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[SE:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[SD:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[USX:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[USV:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[USR:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[USE:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[USD:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[IX:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[IV:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[IR:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[IE:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[ID:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[UIX:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[UIV:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[UIR:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[UIE:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[UID:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[LX:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LV:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LR:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LE:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LD:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULX:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULV:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULR:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULE:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULD:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LLX:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LLV:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LLR:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LLE:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LLD:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULLX:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULLV:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULLR:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULLE:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULLD:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[FX:%.*]] = alloca float, align 4
+// SIMD-ONLY0-NEXT:    [[FV:%.*]] = alloca float, align 4
+// SIMD-ONLY0-NEXT:    [[FE:%.*]] = alloca float, align 4
+// SIMD-ONLY0-NEXT:    [[FD:%.*]] = alloca float, align 4
+// SIMD-ONLY0-NEXT:    [[DX:%.*]] = alloca double, align 8
+// SIMD-ONLY0-NEXT:    [[DV:%.*]] = alloca double, align 8
+// SIMD-ONLY0-NEXT:    [[DE:%.*]] = alloca double, align 8
+// SIMD-ONLY0-NEXT:    [[DD:%.*]] = alloca double, align 8
+// SIMD-ONLY0-NEXT:    [[TMP0:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP0]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP1:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV:%.*]] = sext i8 [[TMP1]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP2:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1:%.*]] = sext i8 [[TMP2]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[CONV]], [[CONV1]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP]], label [[IF_THEN:%.*]], label [[IF_END:%.*]]
+// SIMD-ONLY0:       if.then:
+// SIMD-ONLY0-NEXT:    [[TMP3:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP3]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END]]
+// SIMD-ONLY0:       if.end:
+// SIMD-ONLY0-NEXT:    [[TMP4:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP4]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP5:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV3:%.*]] = sext i8 [[TMP5]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP6:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV4:%.*]] = sext i8 [[TMP6]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP5:%.*]] = icmp sgt i32 [[CONV3]], [[CONV4]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5]], label [[IF_THEN7:%.*]], label [[IF_END8:%.*]]
+// SIMD-ONLY0:       if.then7:
+// SIMD-ONLY0-NEXT:    [[TMP7:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP7]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END8]]
+// SIMD-ONLY0:       if.end8:
+// SIMD-ONLY0-NEXT:    [[TMP8:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP8]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP9:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV9:%.*]] = sext i8 [[TMP9]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP10:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV10:%.*]] = sext i8 [[TMP10]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP11:%.*]] = icmp slt i32 [[CONV9]], [[CONV10]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP11]], label [[IF_THEN13:%.*]], label [[IF_END14:%.*]]
+// SIMD-ONLY0:       if.then13:
+// SIMD-ONLY0-NEXT:    [[TMP11:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP11]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END14]]
+// SIMD-ONLY0:       if.end14:
+// SIMD-ONLY0-NEXT:    [[TMP12:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP12]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP13:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV15:%.*]] = sext i8 [[TMP13]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP14:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV16:%.*]] = sext i8 [[TMP14]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP17:%.*]] = icmp slt i32 [[CONV15]], [[CONV16]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP17]], label [[IF_THEN19:%.*]], label [[IF_END20:%.*]]
+// SIMD-ONLY0:       if.then19:
+// SIMD-ONLY0-NEXT:    [[TMP15:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP15]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END20]]
+// SIMD-ONLY0:       if.end20:
+// SIMD-ONLY0-NEXT:    [[TMP16:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP16]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP17:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV21:%.*]] = sext i8 [[TMP17]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP18:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV22:%.*]] = sext i8 [[TMP18]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP23:%.*]] = icmp eq i32 [[CONV21]], [[CONV22]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP23]], label [[IF_THEN25:%.*]], label [[IF_END26:%.*]]
+// SIMD-ONLY0:       if.then25:
+// SIMD-ONLY0-NEXT:    [[TMP19:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP19]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END26]]
+// SIMD-ONLY0:       if.end26:
+// SIMD-ONLY0-NEXT:    [[TMP20:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP20]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP21:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV27:%.*]] = sext i8 [[TMP21]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP22:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV28:%.*]] = sext i8 [[TMP22]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP29:%.*]] = icmp eq i32 [[CONV27]], [[CONV28]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP29]], label [[IF_THEN31:%.*]], label [[IF_END32:%.*]]
+// SIMD-ONLY0:       if.then31:
+// SIMD-ONLY0-NEXT:    [[TMP23:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP23]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END32]]
+// SIMD-ONLY0:       if.end32:
+// SIMD-ONLY0-NEXT:    [[TMP24:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV33:%.*]] = sext i8 [[TMP24]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP25:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV34:%.*]] = sext i8 [[TMP25]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP35:%.*]] = icmp sgt i32 [[CONV33]], [[CONV34]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP35]], label [[IF_THEN37:%.*]], label [[IF_END38:%.*]]
+// SIMD-ONLY0:       if.then37:
+// SIMD-ONLY0-NEXT:    [[TMP26:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP26]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END38]]
+// SIMD-ONLY0:       if.end38:
+// SIMD-ONLY0-NEXT:    [[TMP27:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP27]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP28:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV39:%.*]] = sext i8 [[TMP28]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP29:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV40:%.*]] = sext i8 [[TMP29]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP41:%.*]] = icmp sgt i32 [[CONV39]], [[CONV40]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP41]], label [[IF_THEN43:%.*]], label [[IF_END44:%.*]]
+// SIMD-ONLY0:       if.then43:
+// SIMD-ONLY0-NEXT:    [[TMP30:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP30]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END44]]
+// SIMD-ONLY0:       if.end44:
+// SIMD-ONLY0-NEXT:    [[TMP31:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP31]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP32:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV45:%.*]] = sext i8 [[TMP32]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP33:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV46:%.*]] = sext i8 [[TMP33]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP47:%.*]] = icmp slt i32 [[CONV45]], [[CONV46]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP47]], label [[IF_THEN49:%.*]], label [[IF_END50:%.*]]
+// SIMD-ONLY0:       if.then49:
+// SIMD-ONLY0-NEXT:    [[TMP34:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP34]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END50]]
+// SIMD-ONLY0:       if.end50:
+// SIMD-ONLY0-NEXT:    [[TMP35:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP35]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP36:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV51:%.*]] = sext i8 [[TMP36]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP37:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV52:%.*]] = sext i8 [[TMP37]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP53:%.*]] = icmp slt i32 [[CONV51]], [[CONV52]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP53]], label [[IF_THEN55:%.*]], label [[IF_END56:%.*]]
+// SIMD-ONLY0:       if.then55:
+// SIMD-ONLY0-NEXT:    [[TMP38:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP38]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END56]]
+// SIMD-ONLY0:       if.end56:
+// SIMD-ONLY0-NEXT:    [[TMP39:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP39]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP40:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV57:%.*]] = sext i8 [[TMP40]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP41:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV58:%.*]] = sext i8 [[TMP41]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP59:%.*]] = icmp eq i32 [[CONV57]], [[CONV58]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP59]], label [[IF_THEN61:%.*]], label [[IF_END62:%.*]]
+// SIMD-ONLY0:       if.then61:
+// SIMD-ONLY0-NEXT:    [[TMP42:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP42]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END62]]
+// SIMD-ONLY0:       if.end62:
+// SIMD-ONLY0-NEXT:    [[TMP43:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP43]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP44:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV63:%.*]] = sext i8 [[TMP44]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP45:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV64:%.*]] = sext i8 [[TMP45]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP65:%.*]] = icmp eq i32 [[CONV63]], [[CONV64]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP65]], label [[IF_THEN67:%.*]], label [[IF_END68:%.*]]
+// SIMD-ONLY0:       if.then67:
+// SIMD-ONLY0-NEXT:    [[TMP46:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP46]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END68]]
+// SIMD-ONLY0:       if.end68:
+// SIMD-ONLY0-NEXT:    [[TMP47:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP47]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP48:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV69:%.*]] = sext i8 [[TMP48]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP49:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV70:%.*]] = sext i8 [[TMP49]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP71:%.*]] = icmp eq i32 [[CONV69]], [[CONV70]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP71]], label [[IF_THEN73:%.*]], label [[IF_ELSE:%.*]]
+// SIMD-ONLY0:       if.then73:
+// SIMD-ONLY0-NEXT:    [[TMP50:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP50]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END74:%.*]]
+// SIMD-ONLY0:       if.else:
+// SIMD-ONLY0-NEXT:    [[TMP51:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP51]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END74]]
+// SIMD-ONLY0:       if.end74:
+// SIMD-ONLY0-NEXT:    [[TMP52:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV75:%.*]] = sext i8 [[TMP52]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP53:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV76:%.*]] = sext i8 [[TMP53]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP77:%.*]] = icmp eq i32 [[CONV75]], [[CONV76]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP77]], label [[IF_THEN79:%.*]], label [[IF_ELSE80:%.*]]
+// SIMD-ONLY0:       if.then79:
+// SIMD-ONLY0-NEXT:    [[TMP54:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP54]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END81:%.*]]
+// SIMD-ONLY0:       if.else80:
+// SIMD-ONLY0-NEXT:    [[TMP55:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP55]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END81]]
+// SIMD-ONLY0:       if.end81:
+// SIMD-ONLY0-NEXT:    [[TMP56:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV82:%.*]] = sext i8 [[TMP56]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP57:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV83:%.*]] = sext i8 [[TMP57]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP84:%.*]] = icmp eq i32 [[CONV82]], [[CONV83]]
+// SIMD-ONLY0-NEXT:    [[CONV85:%.*]] = zext i1 [[CMP84]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV86:%.*]] = trunc i32 [[CONV85]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV86]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP58:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL:%.*]] = icmp ne i8 [[TMP58]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL]], label [[IF_THEN87:%.*]], label [[IF_END88:%.*]]
+// SIMD-ONLY0:       if.then87:
+// SIMD-ONLY0-NEXT:    [[TMP59:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP59]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END88]]
+// SIMD-ONLY0:       if.end88:
+// SIMD-ONLY0-NEXT:    [[TMP60:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV89:%.*]] = sext i8 [[TMP60]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP61:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV90:%.*]] = sext i8 [[TMP61]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP91:%.*]] = icmp eq i32 [[CONV89]], [[CONV90]]
+// SIMD-ONLY0-NEXT:    [[CONV92:%.*]] = zext i1 [[CMP91]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV93:%.*]] = trunc i32 [[CONV92]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV93]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP62:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL94:%.*]] = icmp ne i8 [[TMP62]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL94]], label [[IF_THEN95:%.*]], label [[IF_END96:%.*]]
+// SIMD-ONLY0:       if.then95:
+// SIMD-ONLY0-NEXT:    [[TMP63:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP63]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END96]]
+// SIMD-ONLY0:       if.end96:
+// SIMD-ONLY0-NEXT:    [[TMP64:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV97:%.*]] = sext i8 [[TMP64]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP65:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV98:%.*]] = sext i8 [[TMP65]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP99:%.*]] = icmp eq i32 [[CONV97]], [[CONV98]]
+// SIMD-ONLY0-NEXT:    [[CONV100:%.*]] = zext i1 [[CMP99]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV101:%.*]] = trunc i32 [[CONV100]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV101]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP66:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL102:%.*]] = icmp ne i8 [[TMP66]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL102]], label [[IF_THEN103:%.*]], label [[IF_ELSE104:%.*]]
+// SIMD-ONLY0:       if.then103:
+// SIMD-ONLY0-NEXT:    [[TMP67:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP67]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END105:%.*]]
+// SIMD-ONLY0:       if.else104:
+// SIMD-ONLY0-NEXT:    [[TMP68:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP68]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END105]]
+// SIMD-ONLY0:       if.end105:
+// SIMD-ONLY0-NEXT:    [[TMP69:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV106:%.*]] = sext i8 [[TMP69]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP70:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV107:%.*]] = sext i8 [[TMP70]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP108:%.*]] = icmp eq i32 [[CONV106]], [[CONV107]]
+// SIMD-ONLY0-NEXT:    [[CONV109:%.*]] = zext i1 [[CMP108]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV110:%.*]] = trunc i32 [[CONV109]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV110]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP71:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL111:%.*]] = icmp ne i8 [[TMP71]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL111]], label [[IF_THEN112:%.*]], label [[IF_ELSE113:%.*]]
+// SIMD-ONLY0:       if.then112:
+// SIMD-ONLY0-NEXT:    [[TMP72:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP72]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END114:%.*]]
+// SIMD-ONLY0:       if.else113:
+// SIMD-ONLY0-NEXT:    [[TMP73:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP73]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END114]]
+// SIMD-ONLY0:       if.end114:
+// SIMD-ONLY0-NEXT:    [[TMP74:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP74]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP75:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV115:%.*]] = sext i8 [[TMP75]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP76:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV116:%.*]] = sext i8 [[TMP76]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP117:%.*]] = icmp sgt i32 [[CONV115]], [[CONV116]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP117]], label [[IF_THEN119:%.*]], label [[IF_END120:%.*]]
+// SIMD-ONLY0:       if.then119:
+// SIMD-ONLY0-NEXT:    [[TMP77:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP77]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END120]]
+// SIMD-ONLY0:       if.end120:
+// SIMD-ONLY0-NEXT:    [[TMP78:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP78]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP79:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV121:%.*]] = sext i8 [[TMP79]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP80:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV122:%.*]] = sext i8 [[TMP80]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP123:%.*]] = icmp sgt i32 [[CONV121]], [[CONV122]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP123]], label [[IF_THEN125:%.*]], label [[IF_END126:%.*]]
+// SIMD-ONLY0:       if.then125:
+// SIMD-ONLY0-NEXT:    [[TMP81:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP81]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END126]]
+// SIMD-ONLY0:       if.end126:
+// SIMD-ONLY0-NEXT:    [[TMP82:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP82]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP83:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV127:%.*]] = sext i8 [[TMP83]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP84:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV128:%.*]] = sext i8 [[TMP84]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP129:%.*]] = icmp slt i32 [[CONV127]], [[CONV128]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP129]], label [[IF_THEN131:%.*]], label [[IF_END132:%.*]]
+// SIMD-ONLY0:       if.then131:
+// SIMD-ONLY0-NEXT:    [[TMP85:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP85]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END132]]
+// SIMD-ONLY0:       if.end132:
+// SIMD-ONLY0-NEXT:    [[TMP86:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP86]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP87:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV133:%.*]] = sext i8 [[TMP87]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP88:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV134:%.*]] = sext i8 [[TMP88]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP135:%.*]] = icmp slt i32 [[CONV133]], [[CONV134]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP135]], label [[IF_THEN137:%.*]], label [[IF_END138:%.*]]
+// SIMD-ONLY0:       if.then137:
+// SIMD-ONLY0-NEXT:    [[TMP89:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP89]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END138]]
+// SIMD-ONLY0:       if.end138:
+// SIMD-ONLY0-NEXT:    [[TMP90:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP90]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP91:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV139:%.*]] = sext i8 [[TMP91]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP92:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV140:%.*]] = sext i8 [[TMP92]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP141:%.*]] = icmp eq i32 [[CONV139]], [[CONV140]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP141]], label [[IF_THEN143:%.*]], label [[IF_END144:%.*]]
+// SIMD-ONLY0:       if.then143:
+// SIMD-ONLY0-NEXT:    [[TMP93:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP93]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END144]]
+// SIMD-ONLY0:       if.end144:
+// SIMD-ONLY0-NEXT:    [[TMP94:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP94]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP95:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV145:%.*]] = sext i8 [[TMP95]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP96:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV146:%.*]] = sext i8 [[TMP96]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP147:%.*]] = icmp eq i32 [[CONV145]], [[CONV146]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP147]], label [[IF_THEN149:%.*]], label [[IF_END150:%.*]]
+// SIMD-ONLY0:       if.then149:
+// SIMD-ONLY0-NEXT:    [[TMP97:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP97]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END150]]
+// SIMD-ONLY0:       if.end150:
+// SIMD-ONLY0-NEXT:    [[TMP98:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV151:%.*]] = sext i8 [[TMP98]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP99:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV152:%.*]] = sext i8 [[TMP99]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP153:%.*]] = icmp sgt i32 [[CONV151]], [[CONV152]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP153]], label [[IF_THEN155:%.*]], label [[IF_END156:%.*]]
+// SIMD-ONLY0:       if.then155:
+// SIMD-ONLY0-NEXT:    [[TMP100:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP100]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END156]]
+// SIMD-ONLY0:       if.end156:
+// SIMD-ONLY0-NEXT:    [[TMP101:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP101]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP102:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV157:%.*]] = sext i8 [[TMP102]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP103:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV158:%.*]] = sext i8 [[TMP103]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP159:%.*]] = icmp sgt i32 [[CONV157]], [[CONV158]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP159]], label [[IF_THEN161:%.*]], label [[IF_END162:%.*]]
+// SIMD-ONLY0:       if.then161:
+// SIMD-ONLY0-NEXT:    [[TMP104:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP104]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END162]]
+// SIMD-ONLY0:       if.end162:
+// SIMD-ONLY0-NEXT:    [[TMP105:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP105]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP106:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV163:%.*]] = sext i8 [[TMP106]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP107:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV164:%.*]] = sext i8 [[TMP107]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP165:%.*]] = icmp slt i32 [[CONV163]], [[CONV164]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP165]], label [[IF_THEN167:%.*]], label [[IF_END168:%.*]]
+// SIMD-ONLY0:       if.then167:
+// SIMD-ONLY0-NEXT:    [[TMP108:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP108]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END168]]
+// SIMD-ONLY0:       if.end168:
+// SIMD-ONLY0-NEXT:    [[TMP109:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP109]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP110:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV169:%.*]] = sext i8 [[TMP110]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP111:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV170:%.*]] = sext i8 [[TMP111]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP171:%.*]] = icmp slt i32 [[CONV169]], [[CONV170]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP171]], label [[IF_THEN173:%.*]], label [[IF_END174:%.*]]
+// SIMD-ONLY0:       if.then173:
+// SIMD-ONLY0-NEXT:    [[TMP112:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP112]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END174]]
+// SIMD-ONLY0:       if.end174:
+// SIMD-ONLY0-NEXT:    [[TMP113:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP113]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP114:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV175:%.*]] = sext i8 [[TMP114]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP115:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV176:%.*]] = sext i8 [[TMP115]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP177:%.*]] = icmp eq i32 [[CONV175]], [[CONV176]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP177]], label [[IF_THEN179:%.*]], label [[IF_END180:%.*]]
+// SIMD-ONLY0:       if.then179:
+// SIMD-ONLY0-NEXT:    [[TMP116:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP116]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END180]]
+// SIMD-ONLY0:       if.end180:
+// SIMD-ONLY0-NEXT:    [[TMP117:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP117]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP118:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV181:%.*]] = sext i8 [[TMP118]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP119:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV182:%.*]] = sext i8 [[TMP119]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP183:%.*]] = icmp eq i32 [[CONV181]], [[CONV182]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP183]], label [[IF_THEN185:%.*]], label [[IF_END186:%.*]]
+// SIMD-ONLY0:       if.then185:
+// SIMD-ONLY0-NEXT:    [[TMP120:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP120]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END186]]
+// SIMD-ONLY0:       if.end186:
+// SIMD-ONLY0-NEXT:    [[TMP121:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP121]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP122:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV187:%.*]] = sext i8 [[TMP122]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP123:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV188:%.*]] = sext i8 [[TMP123]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP189:%.*]] = icmp eq i32 [[CONV187]], [[CONV188]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP189]], label [[IF_THEN191:%.*]], label [[IF_ELSE192:%.*]]
+// SIMD-ONLY0:       if.then191:
+// SIMD-ONLY0-NEXT:    [[TMP124:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP124]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END193:%.*]]
+// SIMD-ONLY0:       if.else192:
+// SIMD-ONLY0-NEXT:    [[TMP125:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP125]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END193]]
+// SIMD-ONLY0:       if.end193:
+// SIMD-ONLY0-NEXT:    [[TMP126:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV194:%.*]] = sext i8 [[TMP126]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP127:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV195:%.*]] = sext i8 [[TMP127]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP196:%.*]] = icmp eq i32 [[CONV194]], [[CONV195]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP196]], label [[IF_THEN198:%.*]], label [[IF_ELSE199:%.*]]
+// SIMD-ONLY0:       if.then198:
+// SIMD-ONLY0-NEXT:    [[TMP128:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP128]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END200:%.*]]
+// SIMD-ONLY0:       if.else199:
+// SIMD-ONLY0-NEXT:    [[TMP129:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP129]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END200]]
+// SIMD-ONLY0:       if.end200:
+// SIMD-ONLY0-NEXT:    [[TMP130:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV201:%.*]] = sext i8 [[TMP130]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP131:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV202:%.*]] = sext i8 [[TMP131]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP203:%.*]] = icmp eq i32 [[CONV201]], [[CONV202]]
+// SIMD-ONLY0-NEXT:    [[CONV204:%.*]] = zext i1 [[CMP203]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV205:%.*]] = trunc i32 [[CONV204]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV205]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP132:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL206:%.*]] = icmp ne i8 [[TMP132]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL206]], label [[IF_THEN207:%.*]], label [[IF_END208:%.*]]
+// SIMD-ONLY0:       if.then207:
+// SIMD-ONLY0-NEXT:    [[TMP133:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP133]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END208]]
+// SIMD-ONLY0:       if.end208:
+// SIMD-ONLY0-NEXT:    [[TMP134:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV209:%.*]] = sext i8 [[TMP134]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP135:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV210:%.*]] = sext i8 [[TMP135]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP211:%.*]] = icmp eq i32 [[CONV209]], [[CONV210]]
+// SIMD-ONLY0-NEXT:    [[CONV212:%.*]] = zext i1 [[CMP211]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV213:%.*]] = trunc i32 [[CONV212]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV213]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP136:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL214:%.*]] = icmp ne i8 [[TMP136]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL214]], label [[IF_THEN215:%.*]], label [[IF_END216:%.*]]
+// SIMD-ONLY0:       if.then215:
+// SIMD-ONLY0-NEXT:    [[TMP137:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP137]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END216]]
+// SIMD-ONLY0:       if.end216:
+// SIMD-ONLY0-NEXT:    [[TMP138:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV217:%.*]] = sext i8 [[TMP138]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP139:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV218:%.*]] = sext i8 [[TMP139]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP219:%.*]] = icmp eq i32 [[CONV217]], [[CONV218]]
+// SIMD-ONLY0-NEXT:    [[CONV220:%.*]] = zext i1 [[CMP219]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV221:%.*]] = trunc i32 [[CONV220]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV221]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP140:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL222:%.*]] = icmp ne i8 [[TMP140]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL222]], label [[IF_THEN223:%.*]], label [[IF_ELSE224:%.*]]
+// SIMD-ONLY0:       if.then223:
+// SIMD-ONLY0-NEXT:    [[TMP141:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP141]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END225:%.*]]
+// SIMD-ONLY0:       if.else224:
+// SIMD-ONLY0-NEXT:    [[TMP142:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP142]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END225]]
+// SIMD-ONLY0:       if.end225:
+// SIMD-ONLY0-NEXT:    [[TMP143:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV226:%.*]] = sext i8 [[TMP143]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP144:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV227:%.*]] = sext i8 [[TMP144]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP228:%.*]] = icmp eq i32 [[CONV226]], [[CONV227]]
+// SIMD-ONLY0-NEXT:    [[CONV229:%.*]] = zext i1 [[CMP228]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV230:%.*]] = trunc i32 [[CONV229]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV230]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP145:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL231:%.*]] = icmp ne i8 [[TMP145]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL231]], label [[IF_THEN232:%.*]], label [[IF_ELSE233:%.*]]
+// SIMD-ONLY0:       if.then232:
+// SIMD-ONLY0-NEXT:    [[TMP146:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP146]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END234:%.*]]
+// SIMD-ONLY0:       if.else233:
+// SIMD-ONLY0-NEXT:    [[TMP147:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP147]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END234]]
+// SIMD-ONLY0:       if.end234:
+// SIMD-ONLY0-NEXT:    [[TMP148:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP148]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP149:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV235:%.*]] = sext i8 [[TMP149]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP150:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV236:%.*]] = sext i8 [[TMP150]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP237:%.*]] = icmp sgt i32 [[CONV235]], [[CONV236]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP237]], label [[IF_THEN239:%.*]], label [[IF_END240:%.*]]
+// SIMD-ONLY0:       if.then239:
+// SIMD-ONLY0-NEXT:    [[TMP151:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP151]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END240]]
+// SIMD-ONLY0:       if.end240:
+// SIMD-ONLY0-NEXT:    [[TMP152:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP152]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP153:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV241:%.*]] = sext i8 [[TMP153]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP154:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV242:%.*]] = sext i8 [[TMP154]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP243:%.*]] = icmp sgt i32 [[CONV241]], [[CONV242]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP243]], label [[IF_THEN245:%.*]], label [[IF_END246:%.*]]
+// SIMD-ONLY0:       if.then245:
+// SIMD-ONLY0-NEXT:    [[TMP155:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP155]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END246]]
+// SIMD-ONLY0:       if.end246:
+// SIMD-ONLY0-NEXT:    [[TMP156:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP156]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP157:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV247:%.*]] = sext i8 [[TMP157]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP158:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV248:%.*]] = sext i8 [[TMP158]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP249:%.*]] = icmp slt i32 [[CONV247]], [[CONV248]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP249]], label [[IF_THEN251:%.*]], label [[IF_END252:%.*]]
+// SIMD-ONLY0:       if.then251:
+// SIMD-ONLY0-NEXT:    [[TMP159:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP159]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END252]]
+// SIMD-ONLY0:       if.end252:
+// SIMD-ONLY0-NEXT:    [[TMP160:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP160]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP161:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV253:%.*]] = sext i8 [[TMP161]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP162:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV254:%.*]] = sext i8 [[TMP162]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP255:%.*]] = icmp slt i32 [[CONV253]], [[CONV254]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP255]], label [[IF_THEN257:%.*]], label [[IF_END258:%.*]]
+// SIMD-ONLY0:       if.then257:
+// SIMD-ONLY0-NEXT:    [[TMP163:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP163]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END258]]
+// SIMD-ONLY0:       if.end258:
+// SIMD-ONLY0-NEXT:    [[TMP164:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP164]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP165:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV259:%.*]] = sext i8 [[TMP165]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP166:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV260:%.*]] = sext i8 [[TMP166]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP261:%.*]] = icmp eq i32 [[CONV259]], [[CONV260]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP261]], label [[IF_THEN263:%.*]], label [[IF_END264:%.*]]
+// SIMD-ONLY0:       if.then263:
+// SIMD-ONLY0-NEXT:    [[TMP167:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP167]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END264]]
+// SIMD-ONLY0:       if.end264:
+// SIMD-ONLY0-NEXT:    [[TMP168:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP168]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP169:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV265:%.*]] = sext i8 [[TMP169]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP170:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV266:%.*]] = sext i8 [[TMP170]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP267:%.*]] = icmp eq i32 [[CONV265]], [[CONV266]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP267]], label [[IF_THEN269:%.*]], label [[IF_END270:%.*]]
+// SIMD-ONLY0:       if.then269:
+// SIMD-ONLY0-NEXT:    [[TMP171:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP171]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END270]]
+// SIMD-ONLY0:       if.end270:
+// SIMD-ONLY0-NEXT:    [[TMP172:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV271:%.*]] = sext i8 [[TMP172]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP173:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV272:%.*]] = sext i8 [[TMP173]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP273:%.*]] = icmp sgt i32 [[CONV271]], [[CONV272]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP273]], label [[IF_THEN275:%.*]], label [[IF_END276:%.*]]
+// SIMD-ONLY0:       if.then275:
+// SIMD-ONLY0-NEXT:    [[TMP174:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP174]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END276]]
+// SIMD-ONLY0:       if.end276:
+// SIMD-ONLY0-NEXT:    [[TMP175:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP175]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP176:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV277:%.*]] = sext i8 [[TMP176]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP177:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV278:%.*]] = sext i8 [[TMP177]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP279:%.*]] = icmp sgt i32 [[CONV277]], [[CONV278]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP279]], label [[IF_THEN281:%.*]], label [[IF_END282:%.*]]
+// SIMD-ONLY0:       if.then281:
+// SIMD-ONLY0-NEXT:    [[TMP178:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP178]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END282]]
+// SIMD-ONLY0:       if.end282:
+// SIMD-ONLY0-NEXT:    [[TMP179:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP179]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP180:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV283:%.*]] = sext i8 [[TMP180]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP181:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV284:%.*]] = sext i8 [[TMP181]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP285:%.*]] = icmp slt i32 [[CONV283]], [[CONV284]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP285]], label [[IF_THEN287:%.*]], label [[IF_END288:%.*]]
+// SIMD-ONLY0:       if.then287:
+// SIMD-ONLY0-NEXT:    [[TMP182:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP182]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END288]]
+// SIMD-ONLY0:       if.end288:
+// SIMD-ONLY0-NEXT:    [[TMP183:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP183]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP184:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV289:%.*]] = sext i8 [[TMP184]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP185:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV290:%.*]] = sext i8 [[TMP185]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP291:%.*]] = icmp slt i32 [[CONV289]], [[CONV290]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP291]], label [[IF_THEN293:%.*]], label [[IF_END294:%.*]]
+// SIMD-ONLY0:       if.then293:
+// SIMD-ONLY0-NEXT:    [[TMP186:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP186]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END294]]
+// SIMD-ONLY0:       if.end294:
+// SIMD-ONLY0-NEXT:    [[TMP187:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP187]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP188:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV295:%.*]] = sext i8 [[TMP188]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP189:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV296:%.*]] = sext i8 [[TMP189]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP297:%.*]] = icmp eq i32 [[CONV295]], [[CONV296]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP297]], label [[IF_THEN299:%.*]], label [[IF_END300:%.*]]
+// SIMD-ONLY0:       if.then299:
+// SIMD-ONLY0-NEXT:    [[TMP190:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP190]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END300]]
+// SIMD-ONLY0:       if.end300:
+// SIMD-ONLY0-NEXT:    [[TMP191:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP191]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP192:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV301:%.*]] = sext i8 [[TMP192]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP193:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV302:%.*]] = sext i8 [[TMP193]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP303:%.*]] = icmp eq i32 [[CONV301]], [[CONV302]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP303]], label [[IF_THEN305:%.*]], label [[IF_END306:%.*]]
+// SIMD-ONLY0:       if.then305:
+// SIMD-ONLY0-NEXT:    [[TMP194:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP194]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END306]]
+// SIMD-ONLY0:       if.end306:
+// SIMD-ONLY0-NEXT:    [[TMP195:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP195]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP196:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV307:%.*]] = sext i8 [[TMP196]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP197:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV308:%.*]] = sext i8 [[TMP197]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP309:%.*]] = icmp eq i32 [[CONV307]], [[CONV308]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP309]], label [[IF_THEN311:%.*]], label [[IF_ELSE312:%.*]]
+// SIMD-ONLY0:       if.then311:
+// SIMD-ONLY0-NEXT:    [[TMP198:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP198]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END313:%.*]]
+// SIMD-ONLY0:       if.else312:
+// SIMD-ONLY0-NEXT:    [[TMP199:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP199]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END313]]
+// SIMD-ONLY0:       if.end313:
+// SIMD-ONLY0-NEXT:    [[TMP200:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV314:%.*]] = sext i8 [[TMP200]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP201:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV315:%.*]] = sext i8 [[TMP201]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP316:%.*]] = icmp eq i32 [[CONV314]], [[CONV315]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP316]], label [[IF_THEN318:%.*]], label [[IF_ELSE319:%.*]]
+// SIMD-ONLY0:       if.then318:
+// SIMD-ONLY0-NEXT:    [[TMP202:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP202]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END320:%.*]]
+// SIMD-ONLY0:       if.else319:
+// SIMD-ONLY0-NEXT:    [[TMP203:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP203]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END320]]
+// SIMD-ONLY0:       if.end320:
+// SIMD-ONLY0-NEXT:    [[TMP204:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV321:%.*]] = sext i8 [[TMP204]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP205:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV322:%.*]] = sext i8 [[TMP205]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP323:%.*]] = icmp eq i32 [[CONV321]], [[CONV322]]
+// SIMD-ONLY0-NEXT:    [[CONV324:%.*]] = zext i1 [[CMP323]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV325:%.*]] = trunc i32 [[CONV324]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV325]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP206:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL326:%.*]] = icmp ne i8 [[TMP206]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL326]], label [[IF_THEN327:%.*]], label [[IF_END328:%.*]]
+// SIMD-ONLY0:       if.then327:
+// SIMD-ONLY0-NEXT:    [[TMP207:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP207]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END328]]
+// SIMD-ONLY0:       if.end328:
+// SIMD-ONLY0-NEXT:    [[TMP208:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV329:%.*]] = sext i8 [[TMP208]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP209:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV330:%.*]] = sext i8 [[TMP209]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP331:%.*]] = icmp eq i32 [[CONV329]], [[CONV330]]
+// SIMD-ONLY0-NEXT:    [[CONV332:%.*]] = zext i1 [[CMP331]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV333:%.*]] = trunc i32 [[CONV332]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV333]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP210:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL334:%.*]] = icmp ne i8 [[TMP210]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL334]], label [[IF_THEN335:%.*]], label [[IF_END336:%.*]]
+// SIMD-ONLY0:       if.then335:
+// SIMD-ONLY0-NEXT:    [[TMP211:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP211]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END336]]
+// SIMD-ONLY0:       if.end336:
+// SIMD-ONLY0-NEXT:    [[TMP212:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV337:%.*]] = sext i8 [[TMP212]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP213:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV338:%.*]] = sext i8 [[TMP213]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP339:%.*]] = icmp eq i32 [[CONV337]], [[CONV338]]
+// SIMD-ONLY0-NEXT:    [[CONV340:%.*]] = zext i1 [[CMP339]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV341:%.*]] = trunc i32 [[CONV340]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV341]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP214:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL342:%.*]] = icmp ne i8 [[TMP214]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL342]], label [[IF_THEN343:%.*]], label [[IF_ELSE344:%.*]]
+// SIMD-ONLY0:       if.then343:
+// SIMD-ONLY0-NEXT:    [[TMP215:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP215]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END345:%.*]]
+// SIMD-ONLY0:       if.else344:
+// SIMD-ONLY0-NEXT:    [[TMP216:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP216]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END345]]
+// SIMD-ONLY0:       if.end345:
+// SIMD-ONLY0-NEXT:    [[TMP217:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV346:%.*]] = sext i8 [[TMP217]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP218:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV347:%.*]] = sext i8 [[TMP218]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP348:%.*]] = icmp eq i32 [[CONV346]], [[CONV347]]
+// SIMD-ONLY0-NEXT:    [[CONV349:%.*]] = zext i1 [[CMP348]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV350:%.*]] = trunc i32 [[CONV349]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV350]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP219:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL351:%.*]] = icmp ne i8 [[TMP219]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL351]], label [[IF_THEN352:%.*]], label [[IF_ELSE353:%.*]]
+// SIMD-ONLY0:       if.then352:
+// SIMD-ONLY0-NEXT:    [[TMP220:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP220]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END354:%.*]]
+// SIMD-ONLY0:       if.else353:
+// SIMD-ONLY0-NEXT:    [[TMP221:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP221]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END354]]
+// SIMD-ONLY0:       if.end354:
+// SIMD-ONLY0-NEXT:    [[TMP222:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP222]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP223:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV355:%.*]] = sext i8 [[TMP223]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP224:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV356:%.*]] = sext i8 [[TMP224]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP357:%.*]] = icmp sgt i32 [[CONV355]], [[CONV356]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP357]], label [[IF_THEN359:%.*]], label [[IF_END360:%.*]]
+// SIMD-ONLY0:       if.then359:
+// SIMD-ONLY0-NEXT:    [[TMP225:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP225]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END360]]
+// SIMD-ONLY0:       if.end360:
+// SIMD-ONLY0-NEXT:    [[TMP226:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP226]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP227:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV361:%.*]] = sext i8 [[TMP227]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP228:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV362:%.*]] = sext i8 [[TMP228]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP363:%.*]] = icmp sgt i32 [[CONV361]], [[CONV362]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP363]], label [[IF_THEN365:%.*]], label [[IF_END366:%.*]]
+// SIMD-ONLY0:       if.then365:
+// SIMD-ONLY0-NEXT:    [[TMP229:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP229]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END366]]
+// SIMD-ONLY0:       if.end366:
+// SIMD-ONLY0-NEXT:    [[TMP230:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP230]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP231:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV367:%.*]] = sext i8 [[TMP231]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP232:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV368:%.*]] = sext i8 [[TMP232]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP369:%.*]] = icmp slt i32 [[CONV367]], [[CONV368]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP369]], label [[IF_THEN371:%.*]], label [[IF_END372:%.*]]
+// SIMD-ONLY0:       if.then371:
+// SIMD-ONLY0-NEXT:    [[TMP233:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP233]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END372]]
+// SIMD-ONLY0:       if.end372:
+// SIMD-ONLY0-NEXT:    [[TMP234:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP234]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP235:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV373:%.*]] = sext i8 [[TMP235]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP236:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV374:%.*]] = sext i8 [[TMP236]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP375:%.*]] = icmp slt i32 [[CONV373]], [[CONV374]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP375]], label [[IF_THEN377:%.*]], label [[IF_END378:%.*]]
+// SIMD-ONLY0:       if.then377:
+// SIMD-ONLY0-NEXT:    [[TMP237:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP237]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END378]]
+// SIMD-ONLY0:       if.end378:
+// SIMD-ONLY0-NEXT:    [[TMP238:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP238]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP239:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV379:%.*]] = sext i8 [[TMP239]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP240:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV380:%.*]] = sext i8 [[TMP240]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP381:%.*]] = icmp eq i32 [[CONV379]], [[CONV380]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP381]], label [[IF_THEN383:%.*]], label [[IF_END384:%.*]]
+// SIMD-ONLY0:       if.then383:
+// SIMD-ONLY0-NEXT:    [[TMP241:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP241]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END384]]
+// SIMD-ONLY0:       if.end384:
+// SIMD-ONLY0-NEXT:    [[TMP242:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP242]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP243:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV385:%.*]] = sext i8 [[TMP243]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP244:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV386:%.*]] = sext i8 [[TMP244]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP387:%.*]] = icmp eq i32 [[CONV385]], [[CONV386]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP387]], label [[IF_THEN389:%.*]], label [[IF_END390:%.*]]
+// SIMD-ONLY0:       if.then389:
+// SIMD-ONLY0-NEXT:    [[TMP245:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP245]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END390]]
+// SIMD-ONLY0:       if.end390:
+// SIMD-ONLY0-NEXT:    [[TMP246:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV391:%.*]] = sext i8 [[TMP246]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP247:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV392:%.*]] = sext i8 [[TMP247]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP393:%.*]] = icmp sgt i32 [[CONV391]], [[CONV392]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP393]], label [[IF_THEN395:%.*]], label [[IF_END396:%.*]]
+// SIMD-ONLY0:       if.then395:
+// SIMD-ONLY0-NEXT:    [[TMP248:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP248]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END396]]
+// SIMD-ONLY0:       if.end396:
+// SIMD-ONLY0-NEXT:    [[TMP249:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP249]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP250:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV397:%.*]] = sext i8 [[TMP250]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP251:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV398:%.*]] = sext i8 [[TMP251]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP399:%.*]] = icmp sgt i32 [[CONV397]], [[CONV398]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP399]], label [[IF_THEN401:%.*]], label [[IF_END402:%.*]]
+// SIMD-ONLY0:       if.then401:
+// SIMD-ONLY0-NEXT:    [[TMP252:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP252]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END402]]
+// SIMD-ONLY0:       if.end402:
+// SIMD-ONLY0-NEXT:    [[TMP253:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP253]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP254:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV403:%.*]] = sext i8 [[TMP254]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP255:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV404:%.*]] = sext i8 [[TMP255]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP405:%.*]] = icmp slt i32 [[CONV403]], [[CONV404]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP405]], label [[IF_THEN407:%.*]], label [[IF_END408:%.*]]
+// SIMD-ONLY0:       if.then407:
+// SIMD-ONLY0-NEXT:    [[TMP256:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP256]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END408]]
+// SIMD-ONLY0:       if.end408:
+// SIMD-ONLY0-NEXT:    [[TMP257:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP257]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP258:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV409:%.*]] = sext i8 [[TMP258]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP259:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV410:%.*]] = sext i8 [[TMP259]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP411:%.*]] = icmp slt i32 [[CONV409]], [[CONV410]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP411]], label [[IF_THEN413:%.*]], label [[IF_END414:%.*]]
+// SIMD-ONLY0:       if.then413:
+// SIMD-ONLY0-NEXT:    [[TMP260:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP260]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END414]]
+// SIMD-ONLY0:       if.end414:
+// SIMD-ONLY0-NEXT:    [[TMP261:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP261]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP262:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV415:%.*]] = sext i8 [[TMP262]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP263:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV416:%.*]] = sext i8 [[TMP263]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP417:%.*]] = icmp eq i32 [[CONV415]], [[CONV416]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP417]], label [[IF_THEN419:%.*]], label [[IF_END420:%.*]]
+// SIMD-ONLY0:       if.then419:
+// SIMD-ONLY0-NEXT:    [[TMP264:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP264]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END420]]
+// SIMD-ONLY0:       if.end420:
+// SIMD-ONLY0-NEXT:    [[TMP265:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP265]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP266:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV421:%.*]] = sext i8 [[TMP266]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP267:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV422:%.*]] = sext i8 [[TMP267]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP423:%.*]] = icmp eq i32 [[CONV421]], [[CONV422]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP423]], label [[IF_THEN425:%.*]], label [[IF_END426:%.*]]
+// SIMD-ONLY0:       if.then425:
+// SIMD-ONLY0-NEXT:    [[TMP268:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP268]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END426]]
+// SIMD-ONLY0:       if.end426:
+// SIMD-ONLY0-NEXT:    [[TMP269:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP269]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP270:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV427:%.*]] = sext i8 [[TMP270]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP271:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV428:%.*]] = sext i8 [[TMP271]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP429:%.*]] = icmp eq i32 [[CONV427]], [[CONV428]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP429]], label [[IF_THEN431:%.*]], label [[IF_ELSE432:%.*]]
+// SIMD-ONLY0:       if.then431:
+// SIMD-ONLY0-NEXT:    [[TMP272:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP272]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END433:%.*]]
+// SIMD-ONLY0:       if.else432:
+// SIMD-ONLY0-NEXT:    [[TMP273:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP273]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END433]]
+// SIMD-ONLY0:       if.end433:
+// SIMD-ONLY0-NEXT:    [[TMP274:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV434:%.*]] = sext i8 [[TMP274]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP275:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV435:%.*]] = sext i8 [[TMP275]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP436:%.*]] = icmp eq i32 [[CONV434]], [[CONV435]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP436]], label [[IF_THEN438:%.*]], label [[IF_ELSE439:%.*]]
+// SIMD-ONLY0:       if.then438:
+// SIMD-ONLY0-NEXT:    [[TMP276:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP276]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END440:%.*]]
+// SIMD-ONLY0:       if.else439:
+// SIMD-ONLY0-NEXT:    [[TMP277:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP277]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END440]]
+// SIMD-ONLY0:       if.end440:
+// SIMD-ONLY0-NEXT:    [[TMP278:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV441:%.*]] = sext i8 [[TMP278]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP279:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV442:%.*]] = sext i8 [[TMP279]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP443:%.*]] = icmp eq i32 [[CONV441]], [[CONV442]]
+// SIMD-ONLY0-NEXT:    [[CONV444:%.*]] = zext i1 [[CMP443]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV445:%.*]] = trunc i32 [[CONV444]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV445]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP280:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL446:%.*]] = icmp ne i8 [[TMP280]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL446]], label [[IF_THEN447:%.*]], label [[IF_END448:%.*]]
+// SIMD-ONLY0:       if.then447:
+// SIMD-ONLY0-NEXT:    [[TMP281:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP281]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END448]]
+// SIMD-ONLY0:       if.end448:
+// SIMD-ONLY0-NEXT:    [[TMP282:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV449:%.*]] = sext i8 [[TMP282]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP283:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV450:%.*]] = sext i8 [[TMP283]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP451:%.*]] = icmp eq i32 [[CONV449]], [[CONV450]]
+// SIMD-ONLY0-NEXT:    [[CONV452:%.*]] = zext i1 [[CMP451]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV453:%.*]] = trunc i32 [[CONV452]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV453]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP284:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL454:%.*]] = icmp ne i8 [[TMP284]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL454]], label [[IF_THEN455:%.*]], label [[IF_END456:%.*]]
+// SIMD-ONLY0:       if.then455:
+// SIMD-ONLY0-NEXT:    [[TMP285:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP285]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END456]]
+// SIMD-ONLY0:       if.end456:
+// SIMD-ONLY0-NEXT:    [[TMP286:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV457:%.*]] = sext i8 [[TMP286]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP287:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV458:%.*]] = sext i8 [[TMP287]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP459:%.*]] = icmp eq i32 [[CONV457]], [[CONV458]]
+// SIMD-ONLY0-NEXT:    [[CONV460:%.*]] = zext i1 [[CMP459]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV461:%.*]] = trunc i32 [[CONV460]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV461]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP288:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL462:%.*]] = icmp ne i8 [[TMP288]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL462]], label [[IF_THEN463:%.*]], label [[IF_ELSE464:%.*]]
+// SIMD-ONLY0:       if.then463:
+// SIMD-ONLY0-NEXT:    [[TMP289:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP289]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END465:%.*]]
+// SIMD-ONLY0:       if.else464:
+// SIMD-ONLY0-NEXT:    [[TMP290:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP290]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END465]]
+// SIMD-ONLY0:       if.end465:
+// SIMD-ONLY0-NEXT:    [[TMP291:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV466:%.*]] = sext i8 [[TMP291]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP292:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV467:%.*]] = sext i8 [[TMP292]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP468:%.*]] = icmp eq i32 [[CONV466]], [[CONV467]]
+// SIMD-ONLY0-NEXT:    [[CONV469:%.*]] = zext i1 [[CMP468]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV470:%.*]] = trunc i32 [[CONV469]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV470]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP293:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL471:%.*]] = icmp ne i8 [[TMP293]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL471]], label [[IF_THEN472:%.*]], label [[IF_ELSE473:%.*]]
+// SIMD-ONLY0:       if.then472:
+// SIMD-ONLY0-NEXT:    [[TMP294:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP294]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END474:%.*]]
+// SIMD-ONLY0:       if.else473:
+// SIMD-ONLY0-NEXT:    [[TMP295:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP295]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END474]]
+// SIMD-ONLY0:       if.end474:
+// SIMD-ONLY0-NEXT:    [[TMP296:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP296]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP297:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV475:%.*]] = sext i8 [[TMP297]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP298:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV476:%.*]] = sext i8 [[TMP298]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP477:%.*]] = icmp sgt i32 [[CONV475]], [[CONV476]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP477]], label [[IF_THEN479:%.*]], label [[IF_END480:%.*]]
+// SIMD-ONLY0:       if.then479:
+// SIMD-ONLY0-NEXT:    [[TMP299:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP299]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END480]]
+// SIMD-ONLY0:       if.end480:
+// SIMD-ONLY0-NEXT:    [[TMP300:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP300]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP301:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV481:%.*]] = sext i8 [[TMP301]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP302:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV482:%.*]] = sext i8 [[TMP302]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP483:%.*]] = icmp sgt i32 [[CONV481]], [[CONV482]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP483]], label [[IF_THEN485:%.*]], label [[IF_END486:%.*]]
+// SIMD-ONLY0:       if.then485:
+// SIMD-ONLY0-NEXT:    [[TMP303:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP303]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END486]]
+// SIMD-ONLY0:       if.end486:
+// SIMD-ONLY0-NEXT:    [[TMP304:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP304]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP305:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV487:%.*]] = sext i8 [[TMP305]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP306:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV488:%.*]] = sext i8 [[TMP306]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP489:%.*]] = icmp slt i32 [[CONV487]], [[CONV488]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP489]], label [[IF_THEN491:%.*]], label [[IF_END492:%.*]]
+// SIMD-ONLY0:       if.then491:
+// SIMD-ONLY0-NEXT:    [[TMP307:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP307]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END492]]
+// SIMD-ONLY0:       if.end492:
+// SIMD-ONLY0-NEXT:    [[TMP308:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP308]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP309:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV493:%.*]] = sext i8 [[TMP309]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP310:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV494:%.*]] = sext i8 [[TMP310]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP495:%.*]] = icmp slt i32 [[CONV493]], [[CONV494]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP495]], label [[IF_THEN497:%.*]], label [[IF_END498:%.*]]
+// SIMD-ONLY0:       if.then497:
+// SIMD-ONLY0-NEXT:    [[TMP311:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP311]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END498]]
+// SIMD-ONLY0:       if.end498:
+// SIMD-ONLY0-NEXT:    [[TMP312:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP312]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP313:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV499:%.*]] = sext i8 [[TMP313]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP314:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV500:%.*]] = sext i8 [[TMP314]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP501:%.*]] = icmp eq i32 [[CONV499]], [[CONV500]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP501]], label [[IF_THEN503:%.*]], label [[IF_END504:%.*]]
+// SIMD-ONLY0:       if.then503:
+// SIMD-ONLY0-NEXT:    [[TMP315:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP315]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END504]]
+// SIMD-ONLY0:       if.end504:
+// SIMD-ONLY0-NEXT:    [[TMP316:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP316]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP317:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV505:%.*]] = sext i8 [[TMP317]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP318:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV506:%.*]] = sext i8 [[TMP318]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP507:%.*]] = icmp eq i32 [[CONV505]], [[CONV506]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP507]], label [[IF_THEN509:%.*]], label [[IF_END510:%.*]]
+// SIMD-ONLY0:       if.then509:
+// SIMD-ONLY0-NEXT:    [[TMP319:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP319]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END510]]
+// SIMD-ONLY0:       if.end510:
+// SIMD-ONLY0-NEXT:    [[TMP320:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV511:%.*]] = sext i8 [[TMP320]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP321:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV512:%.*]] = sext i8 [[TMP321]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP513:%.*]] = icmp sgt i32 [[CONV511]], [[CONV512]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP513]], label [[IF_THEN515:%.*]], label [[IF_END516:%.*]]
+// SIMD-ONLY0:       if.then515:
+// SIMD-ONLY0-NEXT:    [[TMP322:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP322]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END516]]
+// SIMD-ONLY0:       if.end516:
+// SIMD-ONLY0-NEXT:    [[TMP323:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP323]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP324:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV517:%.*]] = sext i8 [[TMP324]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP325:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV518:%.*]] = sext i8 [[TMP325]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP519:%.*]] = icmp sgt i32 [[CONV517]], [[CONV518]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP519]], label [[IF_THEN521:%.*]], label [[IF_END522:%.*]]
+// SIMD-ONLY0:       if.then521:
+// SIMD-ONLY0-NEXT:    [[TMP326:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP326]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END522]]
+// SIMD-ONLY0:       if.end522:
+// SIMD-ONLY0-NEXT:    [[TMP327:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP327]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP328:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV523:%.*]] = sext i8 [[TMP328]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP329:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV524:%.*]] = sext i8 [[TMP329]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP525:%.*]] = icmp slt i32 [[CONV523]], [[CONV524]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP525]], label [[IF_THEN527:%.*]], label [[IF_END528:%.*]]
+// SIMD-ONLY0:       if.then527:
+// SIMD-ONLY0-NEXT:    [[TMP330:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP330]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END528]]
+// SIMD-ONLY0:       if.end528:
+// SIMD-ONLY0-NEXT:    [[TMP331:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP331]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP332:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV529:%.*]] = sext i8 [[TMP332]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP333:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV530:%.*]] = sext i8 [[TMP333]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP531:%.*]] = icmp slt i32 [[CONV529]], [[CONV530]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP531]], label [[IF_THEN533:%.*]], label [[IF_END534:%.*]]
+// SIMD-ONLY0:       if.then533:
+// SIMD-ONLY0-NEXT:    [[TMP334:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP334]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END534]]
+// SIMD-ONLY0:       if.end534:
+// SIMD-ONLY0-NEXT:    [[TMP335:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP335]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP336:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV535:%.*]] = sext i8 [[TMP336]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP337:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV536:%.*]] = sext i8 [[TMP337]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP537:%.*]] = icmp eq i32 [[CONV535]], [[CONV536]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP537]], label [[IF_THEN539:%.*]], label [[IF_END540:%.*]]
+// SIMD-ONLY0:       if.then539:
+// SIMD-ONLY0-NEXT:    [[TMP338:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP338]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END540]]
+// SIMD-ONLY0:       if.end540:
+// SIMD-ONLY0-NEXT:    [[TMP339:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP339]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP340:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV541:%.*]] = sext i8 [[TMP340]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP341:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV542:%.*]] = sext i8 [[TMP341]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP543:%.*]] = icmp eq i32 [[CONV541]], [[CONV542]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP543]], label [[IF_THEN545:%.*]], label [[IF_END546:%.*]]
+// SIMD-ONLY0:       if.then545:
+// SIMD-ONLY0-NEXT:    [[TMP342:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP342]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END546]]
+// SIMD-ONLY0:       if.end546:
+// SIMD-ONLY0-NEXT:    [[TMP343:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP343]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP344:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV547:%.*]] = sext i8 [[TMP344]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP345:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV548:%.*]] = sext i8 [[TMP345]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP549:%.*]] = icmp eq i32 [[CONV547]], [[CONV548]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP549]], label [[IF_THEN551:%.*]], label [[IF_ELSE552:%.*]]
+// SIMD-ONLY0:       if.then551:
+// SIMD-ONLY0-NEXT:    [[TMP346:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP346]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END553:%.*]]
+// SIMD-ONLY0:       if.else552:
+// SIMD-ONLY0-NEXT:    [[TMP347:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP347]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END553]]
+// SIMD-ONLY0:       if.end553:
+// SIMD-ONLY0-NEXT:    [[TMP348:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV554:%.*]] = sext i8 [[TMP348]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP349:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV555:%.*]] = sext i8 [[TMP349]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP556:%.*]] = icmp eq i32 [[CONV554]], [[CONV555]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP556]], label [[IF_THEN558:%.*]], label [[IF_ELSE559:%.*]]
+// SIMD-ONLY0:       if.then558:
+// SIMD-ONLY0-NEXT:    [[TMP350:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP350]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END560:%.*]]
+// SIMD-ONLY0:       if.else559:
+// SIMD-ONLY0-NEXT:    [[TMP351:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP351]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END560]]
+// SIMD-ONLY0:       if.end560:
+// SIMD-ONLY0-NEXT:    [[TMP352:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV561:%.*]] = sext i8 [[TMP352]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP353:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV562:%.*]] = sext i8 [[TMP353]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP563:%.*]] = icmp eq i32 [[CONV561]], [[CONV562]]
+// SIMD-ONLY0-NEXT:    [[CONV564:%.*]] = zext i1 [[CMP563]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV565:%.*]] = trunc i32 [[CONV564]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV565]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP354:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL566:%.*]] = icmp ne i8 [[TMP354]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL566]], label [[IF_THEN567:%.*]], label [[IF_END568:%.*]]
+// SIMD-ONLY0:       if.then567:
+// SIMD-ONLY0-NEXT:    [[TMP355:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP355]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END568]]
+// SIMD-ONLY0:       if.end568:
+// SIMD-ONLY0-NEXT:    [[TMP356:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV569:%.*]] = sext i8 [[TMP356]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP357:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV570:%.*]] = sext i8 [[TMP357]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP571:%.*]] = icmp eq i32 [[CONV569]], [[CONV570]]
+// SIMD-ONLY0-NEXT:    [[CONV572:%.*]] = zext i1 [[CMP571]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV573:%.*]] = trunc i32 [[CONV572]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV573]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP358:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL574:%.*]] = icmp ne i8 [[TMP358]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL574]], label [[IF_THEN575:%.*]], label [[IF_END576:%.*]]
+// SIMD-ONLY0:       if.then575:
+// SIMD-ONLY0-NEXT:    [[TMP359:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP359]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END576]]
+// SIMD-ONLY0:       if.end576:
+// SIMD-ONLY0-NEXT:    [[TMP360:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV577:%.*]] = sext i8 [[TMP360]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP361:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV578:%.*]] = sext i8 [[TMP361]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP579:%.*]] = icmp eq i32 [[CONV577]], [[CONV578]]
+// SIMD-ONLY0-NEXT:    [[CONV580:%.*]] = zext i1 [[CMP579]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV581:%.*]] = trunc i32 [[CONV580]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV581]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP362:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL582:%.*]] = icmp ne i8 [[TMP362]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL582]], label [[IF_THEN583:%.*]], label [[IF_ELSE584:%.*]]
+// SIMD-ONLY0:       if.then583:
+// SIMD-ONLY0-NEXT:    [[TMP363:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP363]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END585:%.*]]
+// SIMD-ONLY0:       if.else584:
+// SIMD-ONLY0-NEXT:    [[TMP364:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP364]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END585]]
+// SIMD-ONLY0:       if.end585:
+// SIMD-ONLY0-NEXT:    [[TMP365:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV586:%.*]] = sext i8 [[TMP365]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP366:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV587:%.*]] = sext i8 [[TMP366]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP588:%.*]] = icmp eq i32 [[CONV586]], [[CONV587]]
+// SIMD-ONLY0-NEXT:    [[CONV589:%.*]] = zext i1 [[CMP588]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV590:%.*]] = trunc i32 [[CONV589]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV590]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP367:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL591:%.*]] = icmp ne i8 [[TMP367]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL591]], label [[IF_THEN592:%.*]], label [[IF_ELSE593:%.*]]
+// SIMD-ONLY0:       if.then592:
+// SIMD-ONLY0-NEXT:    [[TMP368:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP368]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END594:%.*]]
+// SIMD-ONLY0:       if.else593:
+// SIMD-ONLY0-NEXT:    [[TMP369:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP369]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END594]]
+// SIMD-ONLY0:       if.end594:
+// SIMD-ONLY0-NEXT:    [[TMP370:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP370]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP371:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV595:%.*]] = sext i8 [[TMP371]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP372:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV596:%.*]] = sext i8 [[TMP372]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP597:%.*]] = icmp sgt i32 [[CONV595]], [[CONV596]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP597]], label [[IF_THEN599:%.*]], label [[IF_END600:%.*]]
+// SIMD-ONLY0:       if.then599:
+// SIMD-ONLY0-NEXT:    [[TMP373:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP373]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END600]]
+// SIMD-ONLY0:       if.end600:
+// SIMD-ONLY0-NEXT:    [[TMP374:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP374]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP375:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV601:%.*]] = sext i8 [[TMP375]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP376:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV602:%.*]] = sext i8 [[TMP376]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP603:%.*]] = icmp sgt i32 [[CONV601]], [[CONV602]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP603]], label [[IF_THEN605:%.*]], label [[IF_END606:%.*]]
+// SIMD-ONLY0:       if.then605:
+// SIMD-ONLY0-NEXT:    [[TMP377:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP377]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END606]]
+// SIMD-ONLY0:       if.end606:
+// SIMD-ONLY0-NEXT:    [[TMP378:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP378]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP379:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV607:%.*]] = sext i8 [[TMP379]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP380:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV608:%.*]] = sext i8 [[TMP380]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP609:%.*]] = icmp slt i32 [[CONV607]], [[CONV608]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP609]], label [[IF_THEN611:%.*]], label [[IF_END612:%.*]]
+// SIMD-ONLY0:       if.then611:
+// SIMD-ONLY0-NEXT:    [[TMP381:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP381]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END612]]
+// SIMD-ONLY0:       if.end612:
+// SIMD-ONLY0-NEXT:    [[TMP382:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP382]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP383:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV613:%.*]] = sext i8 [[TMP383]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP384:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV614:%.*]] = sext i8 [[TMP384]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP615:%.*]] = icmp slt i32 [[CONV613]], [[CONV614]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP615]], label [[IF_THEN617:%.*]], label [[IF_END618:%.*]]
+// SIMD-ONLY0:       if.then617:
+// SIMD-ONLY0-NEXT:    [[TMP385:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP385]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END618]]
+// SIMD-ONLY0:       if.end618:
+// SIMD-ONLY0-NEXT:    [[TMP386:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP386]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP387:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV619:%.*]] = sext i8 [[TMP387]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP388:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV620:%.*]] = sext i8 [[TMP388]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP621:%.*]] = icmp eq i32 [[CONV619]], [[CONV620]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP621]], label [[IF_THEN623:%.*]], label [[IF_END624:%.*]]
+// SIMD-ONLY0:       if.then623:
+// SIMD-ONLY0-NEXT:    [[TMP389:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP389]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END624]]
+// SIMD-ONLY0:       if.end624:
+// SIMD-ONLY0-NEXT:    [[TMP390:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP390]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP391:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV625:%.*]] = sext i8 [[TMP391]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP392:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV626:%.*]] = sext i8 [[TMP392]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP627:%.*]] = icmp eq i32 [[CONV625]], [[CONV626]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP627]], label [[IF_THEN629:%.*]], label [[IF_END630:%.*]]
+// SIMD-ONLY0:       if.then629:
+// SIMD-ONLY0-NEXT:    [[TMP393:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP393]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END630]]
+// SIMD-ONLY0:       if.end630:
+// SIMD-ONLY0-NEXT:    [[TMP394:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV631:%.*]] = sext i8 [[TMP394]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP395:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV632:%.*]] = sext i8 [[TMP395]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP633:%.*]] = icmp sgt i32 [[CONV631]], [[CONV632]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP633]], label [[IF_THEN635:%.*]], label [[IF_END636:%.*]]
+// SIMD-ONLY0:       if.then635:
+// SIMD-ONLY0-NEXT:    [[TMP396:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP396]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END636]]
+// SIMD-ONLY0:       if.end636:
+// SIMD-ONLY0-NEXT:    [[TMP397:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP397]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP398:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV637:%.*]] = sext i8 [[TMP398]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP399:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV638:%.*]] = sext i8 [[TMP399]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP639:%.*]] = icmp sgt i32 [[CONV637]], [[CONV638]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP639]], label [[IF_THEN641:%.*]], label [[IF_END642:%.*]]
+// SIMD-ONLY0:       if.then641:
+// SIMD-ONLY0-NEXT:    [[TMP400:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP400]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END642]]
+// SIMD-ONLY0:       if.end642:
+// SIMD-ONLY0-NEXT:    [[TMP401:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP401]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP402:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV643:%.*]] = sext i8 [[TMP402]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP403:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV644:%.*]] = sext i8 [[TMP403]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP645:%.*]] = icmp slt i32 [[CONV643]], [[CONV644]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP645]], label [[IF_THEN647:%.*]], label [[IF_END648:%.*]]
+// SIMD-ONLY0:       if.then647:
+// SIMD-ONLY0-NEXT:    [[TMP404:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP404]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END648]]
+// SIMD-ONLY0:       if.end648:
+// SIMD-ONLY0-NEXT:    [[TMP405:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP405]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP406:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV649:%.*]] = sext i8 [[TMP406]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP407:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV650:%.*]] = sext i8 [[TMP407]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP651:%.*]] = icmp slt i32 [[CONV649]], [[CONV650]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP651]], label [[IF_THEN653:%.*]], label [[IF_END654:%.*]]
+// SIMD-ONLY0:       if.then653:
+// SIMD-ONLY0-NEXT:    [[TMP408:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP408]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END654]]
+// SIMD-ONLY0:       if.end654:
+// SIMD-ONLY0-NEXT:    [[TMP409:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP409]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP410:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV655:%.*]] = sext i8 [[TMP410]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP411:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV656:%.*]] = sext i8 [[TMP411]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP657:%.*]] = icmp eq i32 [[CONV655]], [[CONV656]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP657]], label [[IF_THEN659:%.*]], label [[IF_END660:%.*]]
+// SIMD-ONLY0:       if.then659:
+// SIMD-ONLY0-NEXT:    [[TMP412:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP412]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END660]]
+// SIMD-ONLY0:       if.end660:
+// SIMD-ONLY0-NEXT:    [[TMP413:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP413]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP414:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV661:%.*]] = sext i8 [[TMP414]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP415:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV662:%.*]] = sext i8 [[TMP415]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP663:%.*]] = icmp eq i32 [[CONV661]], [[CONV662]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP663]], label [[IF_THEN665:%.*]], label [[IF_END666:%.*]]
+// SIMD-ONLY0:       if.then665:
+// SIMD-ONLY0-NEXT:    [[TMP416:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP416]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END666]]
+// SIMD-ONLY0:       if.end666:
+// SIMD-ONLY0-NEXT:    [[TMP417:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP417]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP418:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV667:%.*]] = sext i8 [[TMP418]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP419:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV668:%.*]] = sext i8 [[TMP419]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP669:%.*]] = icmp eq i32 [[CONV667]], [[CONV668]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP669]], label [[IF_THEN671:%.*]], label [[IF_ELSE672:%.*]]
+// SIMD-ONLY0:       if.then671:
+// SIMD-ONLY0-NEXT:    [[TMP420:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP420]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END673:%.*]]
+// SIMD-ONLY0:       if.else672:
+// SIMD-ONLY0-NEXT:    [[TMP421:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP421]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END673]]
+// SIMD-ONLY0:       if.end673:
+// SIMD-ONLY0-NEXT:    [[TMP422:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV674:%.*]] = sext i8 [[TMP422]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP423:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV675:%.*]] = sext i8 [[TMP423]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP676:%.*]] = icmp eq i32 [[CONV674]], [[CONV675]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP676]], label [[IF_THEN678:%.*]], label [[IF_ELSE679:%.*]]
+// SIMD-ONLY0:       if.then678:
+// SIMD-ONLY0-NEXT:    [[TMP424:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP424]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END680:%.*]]
+// SIMD-ONLY0:       if.else679:
+// SIMD-ONLY0-NEXT:    [[TMP425:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP425]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END680]]
+// SIMD-ONLY0:       if.end680:
+// SIMD-ONLY0-NEXT:    [[TMP426:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV681:%.*]] = sext i8 [[TMP426]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP427:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV682:%.*]] = sext i8 [[TMP427]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP683:%.*]] = icmp eq i32 [[CONV681]], [[CONV682]]
+// SIMD-ONLY0-NEXT:    [[CONV684:%.*]] = zext i1 [[CMP683]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV685:%.*]] = trunc i32 [[CONV684]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV685]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP428:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL686:%.*]] = icmp ne i8 [[TMP428]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL686]], label [[IF_THEN687:%.*]], label [[IF_END688:%.*]]
+// SIMD-ONLY0:       if.then687:
+// SIMD-ONLY0-NEXT:    [[TMP429:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP429]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END688]]
+// SIMD-ONLY0:       if.end688:
+// SIMD-ONLY0-NEXT:    [[TMP430:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV689:%.*]] = sext i8 [[TMP430]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP431:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV690:%.*]] = sext i8 [[TMP431]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP691:%.*]] = icmp eq i32 [[CONV689]], [[CONV690]]
+// SIMD-ONLY0-NEXT:    [[CONV692:%.*]] = zext i1 [[CMP691]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV693:%.*]] = trunc i32 [[CONV692]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV693]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP432:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL694:%.*]] = icmp ne i8 [[TMP432]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL694]], label [[IF_THEN695:%.*]], label [[IF_END696:%.*]]
+// SIMD-ONLY0:       if.then695:
+// SIMD-ONLY0-NEXT:    [[TMP433:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP433]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END696]]
+// SIMD-ONLY0:       if.end696:
+// SIMD-ONLY0-NEXT:    [[TMP434:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV697:%.*]] = sext i8 [[TMP434]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP435:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV698:%.*]] = sext i8 [[TMP435]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP699:%.*]] = icmp eq i32 [[CONV697]], [[CONV698]]
+// SIMD-ONLY0-NEXT:    [[CONV700:%.*]] = zext i1 [[CMP699]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV701:%.*]] = trunc i32 [[CONV700]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV701]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP436:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL702:%.*]] = icmp ne i8 [[TMP436]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL702]], label [[IF_THEN703:%.*]], label [[IF_ELSE704:%.*]]
+// SIMD-ONLY0:       if.then703:
+// SIMD-ONLY0-NEXT:    [[TMP437:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP437]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END705:%.*]]
+// SIMD-ONLY0:       if.else704:
+// SIMD-ONLY0-NEXT:    [[TMP438:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP438]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END705]]
+// SIMD-ONLY0:       if.end705:
+// SIMD-ONLY0-NEXT:    [[TMP439:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV706:%.*]] = sext i8 [[TMP439]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP440:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV707:%.*]] = sext i8 [[TMP440]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP708:%.*]] = icmp eq i32 [[CONV706]], [[CONV707]]
+// SIMD-ONLY0-NEXT:    [[CONV709:%.*]] = zext i1 [[CMP708]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV710:%.*]] = trunc i32 [[CONV709]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV710]], ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP441:%.*]] = load i8, ptr [[CR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL711:%.*]] = icmp ne i8 [[TMP441]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL711]], label [[IF_THEN712:%.*]], label [[IF_ELSE713:%.*]]
+// SIMD-ONLY0:       if.then712:
+// SIMD-ONLY0-NEXT:    [[TMP442:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP442]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END714:%.*]]
+// SIMD-ONLY0:       if.else713:
+// SIMD-ONLY0-NEXT:    [[TMP443:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP443]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END714]]
+// SIMD-ONLY0:       if.end714:
+// SIMD-ONLY0-NEXT:    [[TMP444:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP444]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP445:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV715:%.*]] = zext i8 [[TMP445]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP446:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV716:%.*]] = zext i8 [[TMP446]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP717:%.*]] = icmp sgt i32 [[CONV715]], [[CONV716]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP717]], label [[IF_THEN719:%.*]], label [[IF_END720:%.*]]
+// SIMD-ONLY0:       if.then719:
+// SIMD-ONLY0-NEXT:    [[TMP447:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP447]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END720]]
+// SIMD-ONLY0:       if.end720:
+// SIMD-ONLY0-NEXT:    [[TMP448:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP448]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP449:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV721:%.*]] = zext i8 [[TMP449]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP450:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV722:%.*]] = zext i8 [[TMP450]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP723:%.*]] = icmp sgt i32 [[CONV721]], [[CONV722]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP723]], label [[IF_THEN725:%.*]], label [[IF_END726:%.*]]
+// SIMD-ONLY0:       if.then725:
+// SIMD-ONLY0-NEXT:    [[TMP451:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP451]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END726]]
+// SIMD-ONLY0:       if.end726:
+// SIMD-ONLY0-NEXT:    [[TMP452:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP452]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP453:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV727:%.*]] = zext i8 [[TMP453]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP454:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV728:%.*]] = zext i8 [[TMP454]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP729:%.*]] = icmp slt i32 [[CONV727]], [[CONV728]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP729]], label [[IF_THEN731:%.*]], label [[IF_END732:%.*]]
+// SIMD-ONLY0:       if.then731:
+// SIMD-ONLY0-NEXT:    [[TMP455:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP455]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END732]]
+// SIMD-ONLY0:       if.end732:
+// SIMD-ONLY0-NEXT:    [[TMP456:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP456]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP457:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV733:%.*]] = zext i8 [[TMP457]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP458:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV734:%.*]] = zext i8 [[TMP458]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP735:%.*]] = icmp slt i32 [[CONV733]], [[CONV734]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP735]], label [[IF_THEN737:%.*]], label [[IF_END738:%.*]]
+// SIMD-ONLY0:       if.then737:
+// SIMD-ONLY0-NEXT:    [[TMP459:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP459]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END738]]
+// SIMD-ONLY0:       if.end738:
+// SIMD-ONLY0-NEXT:    [[TMP460:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP460]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP461:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV739:%.*]] = zext i8 [[TMP461]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP462:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV740:%.*]] = zext i8 [[TMP462]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP741:%.*]] = icmp eq i32 [[CONV739]], [[CONV740]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP741]], label [[IF_THEN743:%.*]], label [[IF_END744:%.*]]
+// SIMD-ONLY0:       if.then743:
+// SIMD-ONLY0-NEXT:    [[TMP463:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP463]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END744]]
+// SIMD-ONLY0:       if.end744:
+// SIMD-ONLY0-NEXT:    [[TMP464:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP464]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP465:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV745:%.*]] = zext i8 [[TMP465]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP466:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV746:%.*]] = zext i8 [[TMP466]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP747:%.*]] = icmp eq i32 [[CONV745]], [[CONV746]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP747]], label [[IF_THEN749:%.*]], label [[IF_END750:%.*]]
+// SIMD-ONLY0:       if.then749:
+// SIMD-ONLY0-NEXT:    [[TMP467:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP467]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END750]]
+// SIMD-ONLY0:       if.end750:
+// SIMD-ONLY0-NEXT:    [[TMP468:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV751:%.*]] = zext i8 [[TMP468]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP469:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV752:%.*]] = zext i8 [[TMP469]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP753:%.*]] = icmp sgt i32 [[CONV751]], [[CONV752]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP753]], label [[IF_THEN755:%.*]], label [[IF_END756:%.*]]
+// SIMD-ONLY0:       if.then755:
+// SIMD-ONLY0-NEXT:    [[TMP470:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP470]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END756]]
+// SIMD-ONLY0:       if.end756:
+// SIMD-ONLY0-NEXT:    [[TMP471:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP471]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP472:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV757:%.*]] = zext i8 [[TMP472]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP473:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV758:%.*]] = zext i8 [[TMP473]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP759:%.*]] = icmp sgt i32 [[CONV757]], [[CONV758]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP759]], label [[IF_THEN761:%.*]], label [[IF_END762:%.*]]
+// SIMD-ONLY0:       if.then761:
+// SIMD-ONLY0-NEXT:    [[TMP474:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP474]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END762]]
+// SIMD-ONLY0:       if.end762:
+// SIMD-ONLY0-NEXT:    [[TMP475:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP475]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP476:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV763:%.*]] = zext i8 [[TMP476]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP477:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV764:%.*]] = zext i8 [[TMP477]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP765:%.*]] = icmp slt i32 [[CONV763]], [[CONV764]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP765]], label [[IF_THEN767:%.*]], label [[IF_END768:%.*]]
+// SIMD-ONLY0:       if.then767:
+// SIMD-ONLY0-NEXT:    [[TMP478:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP478]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END768]]
+// SIMD-ONLY0:       if.end768:
+// SIMD-ONLY0-NEXT:    [[TMP479:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP479]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP480:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV769:%.*]] = zext i8 [[TMP480]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP481:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV770:%.*]] = zext i8 [[TMP481]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP771:%.*]] = icmp slt i32 [[CONV769]], [[CONV770]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP771]], label [[IF_THEN773:%.*]], label [[IF_END774:%.*]]
+// SIMD-ONLY0:       if.then773:
+// SIMD-ONLY0-NEXT:    [[TMP482:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP482]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END774]]
+// SIMD-ONLY0:       if.end774:
+// SIMD-ONLY0-NEXT:    [[TMP483:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP483]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP484:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV775:%.*]] = zext i8 [[TMP484]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP485:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV776:%.*]] = zext i8 [[TMP485]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP777:%.*]] = icmp eq i32 [[CONV775]], [[CONV776]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP777]], label [[IF_THEN779:%.*]], label [[IF_END780:%.*]]
+// SIMD-ONLY0:       if.then779:
+// SIMD-ONLY0-NEXT:    [[TMP486:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP486]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END780]]
+// SIMD-ONLY0:       if.end780:
+// SIMD-ONLY0-NEXT:    [[TMP487:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP487]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP488:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV781:%.*]] = zext i8 [[TMP488]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP489:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV782:%.*]] = zext i8 [[TMP489]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP783:%.*]] = icmp eq i32 [[CONV781]], [[CONV782]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP783]], label [[IF_THEN785:%.*]], label [[IF_END786:%.*]]
+// SIMD-ONLY0:       if.then785:
+// SIMD-ONLY0-NEXT:    [[TMP490:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP490]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END786]]
+// SIMD-ONLY0:       if.end786:
+// SIMD-ONLY0-NEXT:    [[TMP491:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP491]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP492:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV787:%.*]] = zext i8 [[TMP492]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP493:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV788:%.*]] = zext i8 [[TMP493]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP789:%.*]] = icmp eq i32 [[CONV787]], [[CONV788]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP789]], label [[IF_THEN791:%.*]], label [[IF_ELSE792:%.*]]
+// SIMD-ONLY0:       if.then791:
+// SIMD-ONLY0-NEXT:    [[TMP494:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP494]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END793:%.*]]
+// SIMD-ONLY0:       if.else792:
+// SIMD-ONLY0-NEXT:    [[TMP495:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP495]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END793]]
+// SIMD-ONLY0:       if.end793:
+// SIMD-ONLY0-NEXT:    [[TMP496:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV794:%.*]] = zext i8 [[TMP496]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP497:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV795:%.*]] = zext i8 [[TMP497]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP796:%.*]] = icmp eq i32 [[CONV794]], [[CONV795]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP796]], label [[IF_THEN798:%.*]], label [[IF_ELSE799:%.*]]
+// SIMD-ONLY0:       if.then798:
+// SIMD-ONLY0-NEXT:    [[TMP498:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP498]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END800:%.*]]
+// SIMD-ONLY0:       if.else799:
+// SIMD-ONLY0-NEXT:    [[TMP499:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP499]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END800]]
+// SIMD-ONLY0:       if.end800:
+// SIMD-ONLY0-NEXT:    [[TMP500:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV801:%.*]] = zext i8 [[TMP500]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP501:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV802:%.*]] = zext i8 [[TMP501]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP803:%.*]] = icmp eq i32 [[CONV801]], [[CONV802]]
+// SIMD-ONLY0-NEXT:    [[CONV804:%.*]] = zext i1 [[CMP803]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV805:%.*]] = trunc i32 [[CONV804]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV805]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP502:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL806:%.*]] = icmp ne i8 [[TMP502]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL806]], label [[IF_THEN807:%.*]], label [[IF_END808:%.*]]
+// SIMD-ONLY0:       if.then807:
+// SIMD-ONLY0-NEXT:    [[TMP503:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP503]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END808]]
+// SIMD-ONLY0:       if.end808:
+// SIMD-ONLY0-NEXT:    [[TMP504:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV809:%.*]] = zext i8 [[TMP504]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP505:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV810:%.*]] = zext i8 [[TMP505]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP811:%.*]] = icmp eq i32 [[CONV809]], [[CONV810]]
+// SIMD-ONLY0-NEXT:    [[CONV812:%.*]] = zext i1 [[CMP811]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV813:%.*]] = trunc i32 [[CONV812]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV813]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP506:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL814:%.*]] = icmp ne i8 [[TMP506]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL814]], label [[IF_THEN815:%.*]], label [[IF_END816:%.*]]
+// SIMD-ONLY0:       if.then815:
+// SIMD-ONLY0-NEXT:    [[TMP507:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP507]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END816]]
+// SIMD-ONLY0:       if.end816:
+// SIMD-ONLY0-NEXT:    [[TMP508:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV817:%.*]] = zext i8 [[TMP508]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP509:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV818:%.*]] = zext i8 [[TMP509]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP819:%.*]] = icmp eq i32 [[CONV817]], [[CONV818]]
+// SIMD-ONLY0-NEXT:    [[CONV820:%.*]] = zext i1 [[CMP819]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV821:%.*]] = trunc i32 [[CONV820]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV821]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP510:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL822:%.*]] = icmp ne i8 [[TMP510]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL822]], label [[IF_THEN823:%.*]], label [[IF_ELSE824:%.*]]
+// SIMD-ONLY0:       if.then823:
+// SIMD-ONLY0-NEXT:    [[TMP511:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP511]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END825:%.*]]
+// SIMD-ONLY0:       if.else824:
+// SIMD-ONLY0-NEXT:    [[TMP512:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP512]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END825]]
+// SIMD-ONLY0:       if.end825:
+// SIMD-ONLY0-NEXT:    [[TMP513:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV826:%.*]] = zext i8 [[TMP513]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP514:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV827:%.*]] = zext i8 [[TMP514]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP828:%.*]] = icmp eq i32 [[CONV826]], [[CONV827]]
+// SIMD-ONLY0-NEXT:    [[CONV829:%.*]] = zext i1 [[CMP828]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV830:%.*]] = trunc i32 [[CONV829]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV830]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP515:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL831:%.*]] = icmp ne i8 [[TMP515]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL831]], label [[IF_THEN832:%.*]], label [[IF_ELSE833:%.*]]
+// SIMD-ONLY0:       if.then832:
+// SIMD-ONLY0-NEXT:    [[TMP516:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP516]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END834:%.*]]
+// SIMD-ONLY0:       if.else833:
+// SIMD-ONLY0-NEXT:    [[TMP517:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP517]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END834]]
+// SIMD-ONLY0:       if.end834:
+// SIMD-ONLY0-NEXT:    [[TMP518:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP518]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP519:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV835:%.*]] = zext i8 [[TMP519]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP520:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV836:%.*]] = zext i8 [[TMP520]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP837:%.*]] = icmp sgt i32 [[CONV835]], [[CONV836]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP837]], label [[IF_THEN839:%.*]], label [[IF_END840:%.*]]
+// SIMD-ONLY0:       if.then839:
+// SIMD-ONLY0-NEXT:    [[TMP521:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP521]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END840]]
+// SIMD-ONLY0:       if.end840:
+// SIMD-ONLY0-NEXT:    [[TMP522:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP522]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP523:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV841:%.*]] = zext i8 [[TMP523]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP524:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV842:%.*]] = zext i8 [[TMP524]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP843:%.*]] = icmp sgt i32 [[CONV841]], [[CONV842]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP843]], label [[IF_THEN845:%.*]], label [[IF_END846:%.*]]
+// SIMD-ONLY0:       if.then845:
+// SIMD-ONLY0-NEXT:    [[TMP525:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP525]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END846]]
+// SIMD-ONLY0:       if.end846:
+// SIMD-ONLY0-NEXT:    [[TMP526:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP526]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP527:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV847:%.*]] = zext i8 [[TMP527]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP528:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV848:%.*]] = zext i8 [[TMP528]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP849:%.*]] = icmp slt i32 [[CONV847]], [[CONV848]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP849]], label [[IF_THEN851:%.*]], label [[IF_END852:%.*]]
+// SIMD-ONLY0:       if.then851:
+// SIMD-ONLY0-NEXT:    [[TMP529:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP529]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END852]]
+// SIMD-ONLY0:       if.end852:
+// SIMD-ONLY0-NEXT:    [[TMP530:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP530]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP531:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV853:%.*]] = zext i8 [[TMP531]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP532:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV854:%.*]] = zext i8 [[TMP532]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP855:%.*]] = icmp slt i32 [[CONV853]], [[CONV854]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP855]], label [[IF_THEN857:%.*]], label [[IF_END858:%.*]]
+// SIMD-ONLY0:       if.then857:
+// SIMD-ONLY0-NEXT:    [[TMP533:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP533]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END858]]
+// SIMD-ONLY0:       if.end858:
+// SIMD-ONLY0-NEXT:    [[TMP534:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP534]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP535:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV859:%.*]] = zext i8 [[TMP535]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP536:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV860:%.*]] = zext i8 [[TMP536]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP861:%.*]] = icmp eq i32 [[CONV859]], [[CONV860]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP861]], label [[IF_THEN863:%.*]], label [[IF_END864:%.*]]
+// SIMD-ONLY0:       if.then863:
+// SIMD-ONLY0-NEXT:    [[TMP537:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP537]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END864]]
+// SIMD-ONLY0:       if.end864:
+// SIMD-ONLY0-NEXT:    [[TMP538:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP538]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP539:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV865:%.*]] = zext i8 [[TMP539]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP540:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV866:%.*]] = zext i8 [[TMP540]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP867:%.*]] = icmp eq i32 [[CONV865]], [[CONV866]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP867]], label [[IF_THEN869:%.*]], label [[IF_END870:%.*]]
+// SIMD-ONLY0:       if.then869:
+// SIMD-ONLY0-NEXT:    [[TMP541:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP541]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END870]]
+// SIMD-ONLY0:       if.end870:
+// SIMD-ONLY0-NEXT:    [[TMP542:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV871:%.*]] = zext i8 [[TMP542]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP543:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV872:%.*]] = zext i8 [[TMP543]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP873:%.*]] = icmp sgt i32 [[CONV871]], [[CONV872]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP873]], label [[IF_THEN875:%.*]], label [[IF_END876:%.*]]
+// SIMD-ONLY0:       if.then875:
+// SIMD-ONLY0-NEXT:    [[TMP544:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP544]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END876]]
+// SIMD-ONLY0:       if.end876:
+// SIMD-ONLY0-NEXT:    [[TMP545:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP545]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP546:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV877:%.*]] = zext i8 [[TMP546]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP547:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV878:%.*]] = zext i8 [[TMP547]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP879:%.*]] = icmp sgt i32 [[CONV877]], [[CONV878]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP879]], label [[IF_THEN881:%.*]], label [[IF_END882:%.*]]
+// SIMD-ONLY0:       if.then881:
+// SIMD-ONLY0-NEXT:    [[TMP548:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP548]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END882]]
+// SIMD-ONLY0:       if.end882:
+// SIMD-ONLY0-NEXT:    [[TMP549:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP549]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP550:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV883:%.*]] = zext i8 [[TMP550]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP551:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV884:%.*]] = zext i8 [[TMP551]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP885:%.*]] = icmp slt i32 [[CONV883]], [[CONV884]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP885]], label [[IF_THEN887:%.*]], label [[IF_END888:%.*]]
+// SIMD-ONLY0:       if.then887:
+// SIMD-ONLY0-NEXT:    [[TMP552:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP552]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END888]]
+// SIMD-ONLY0:       if.end888:
+// SIMD-ONLY0-NEXT:    [[TMP553:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP553]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP554:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV889:%.*]] = zext i8 [[TMP554]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP555:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV890:%.*]] = zext i8 [[TMP555]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP891:%.*]] = icmp slt i32 [[CONV889]], [[CONV890]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP891]], label [[IF_THEN893:%.*]], label [[IF_END894:%.*]]
+// SIMD-ONLY0:       if.then893:
+// SIMD-ONLY0-NEXT:    [[TMP556:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP556]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END894]]
+// SIMD-ONLY0:       if.end894:
+// SIMD-ONLY0-NEXT:    [[TMP557:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP557]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP558:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV895:%.*]] = zext i8 [[TMP558]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP559:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV896:%.*]] = zext i8 [[TMP559]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP897:%.*]] = icmp eq i32 [[CONV895]], [[CONV896]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP897]], label [[IF_THEN899:%.*]], label [[IF_END900:%.*]]
+// SIMD-ONLY0:       if.then899:
+// SIMD-ONLY0-NEXT:    [[TMP560:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP560]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END900]]
+// SIMD-ONLY0:       if.end900:
+// SIMD-ONLY0-NEXT:    [[TMP561:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP561]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP562:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV901:%.*]] = zext i8 [[TMP562]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP563:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV902:%.*]] = zext i8 [[TMP563]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP903:%.*]] = icmp eq i32 [[CONV901]], [[CONV902]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP903]], label [[IF_THEN905:%.*]], label [[IF_END906:%.*]]
+// SIMD-ONLY0:       if.then905:
+// SIMD-ONLY0-NEXT:    [[TMP564:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP564]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END906]]
+// SIMD-ONLY0:       if.end906:
+// SIMD-ONLY0-NEXT:    [[TMP565:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP565]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP566:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV907:%.*]] = zext i8 [[TMP566]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP567:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV908:%.*]] = zext i8 [[TMP567]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP909:%.*]] = icmp eq i32 [[CONV907]], [[CONV908]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP909]], label [[IF_THEN911:%.*]], label [[IF_ELSE912:%.*]]
+// SIMD-ONLY0:       if.then911:
+// SIMD-ONLY0-NEXT:    [[TMP568:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP568]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END913:%.*]]
+// SIMD-ONLY0:       if.else912:
+// SIMD-ONLY0-NEXT:    [[TMP569:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP569]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END913]]
+// SIMD-ONLY0:       if.end913:
+// SIMD-ONLY0-NEXT:    [[TMP570:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV914:%.*]] = zext i8 [[TMP570]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP571:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV915:%.*]] = zext i8 [[TMP571]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP916:%.*]] = icmp eq i32 [[CONV914]], [[CONV915]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP916]], label [[IF_THEN918:%.*]], label [[IF_ELSE919:%.*]]
+// SIMD-ONLY0:       if.then918:
+// SIMD-ONLY0-NEXT:    [[TMP572:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP572]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END920:%.*]]
+// SIMD-ONLY0:       if.else919:
+// SIMD-ONLY0-NEXT:    [[TMP573:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP573]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END920]]
+// SIMD-ONLY0:       if.end920:
+// SIMD-ONLY0-NEXT:    [[TMP574:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV921:%.*]] = zext i8 [[TMP574]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP575:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV922:%.*]] = zext i8 [[TMP575]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP923:%.*]] = icmp eq i32 [[CONV921]], [[CONV922]]
+// SIMD-ONLY0-NEXT:    [[CONV924:%.*]] = zext i1 [[CMP923]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV925:%.*]] = trunc i32 [[CONV924]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV925]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP576:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL926:%.*]] = icmp ne i8 [[TMP576]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL926]], label [[IF_THEN927:%.*]], label [[IF_END928:%.*]]
+// SIMD-ONLY0:       if.then927:
+// SIMD-ONLY0-NEXT:    [[TMP577:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP577]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END928]]
+// SIMD-ONLY0:       if.end928:
+// SIMD-ONLY0-NEXT:    [[TMP578:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV929:%.*]] = zext i8 [[TMP578]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP579:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV930:%.*]] = zext i8 [[TMP579]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP931:%.*]] = icmp eq i32 [[CONV929]], [[CONV930]]
+// SIMD-ONLY0-NEXT:    [[CONV932:%.*]] = zext i1 [[CMP931]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV933:%.*]] = trunc i32 [[CONV932]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV933]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP580:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL934:%.*]] = icmp ne i8 [[TMP580]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL934]], label [[IF_THEN935:%.*]], label [[IF_END936:%.*]]
+// SIMD-ONLY0:       if.then935:
+// SIMD-ONLY0-NEXT:    [[TMP581:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP581]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END936]]
+// SIMD-ONLY0:       if.end936:
+// SIMD-ONLY0-NEXT:    [[TMP582:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV937:%.*]] = zext i8 [[TMP582]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP583:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV938:%.*]] = zext i8 [[TMP583]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP939:%.*]] = icmp eq i32 [[CONV937]], [[CONV938]]
+// SIMD-ONLY0-NEXT:    [[CONV940:%.*]] = zext i1 [[CMP939]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV941:%.*]] = trunc i32 [[CONV940]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV941]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP584:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL942:%.*]] = icmp ne i8 [[TMP584]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL942]], label [[IF_THEN943:%.*]], label [[IF_ELSE944:%.*]]
+// SIMD-ONLY0:       if.then943:
+// SIMD-ONLY0-NEXT:    [[TMP585:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP585]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END945:%.*]]
+// SIMD-ONLY0:       if.else944:
+// SIMD-ONLY0-NEXT:    [[TMP586:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP586]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END945]]
+// SIMD-ONLY0:       if.end945:
+// SIMD-ONLY0-NEXT:    [[TMP587:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV946:%.*]] = zext i8 [[TMP587]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP588:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV947:%.*]] = zext i8 [[TMP588]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP948:%.*]] = icmp eq i32 [[CONV946]], [[CONV947]]
+// SIMD-ONLY0-NEXT:    [[CONV949:%.*]] = zext i1 [[CMP948]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV950:%.*]] = trunc i32 [[CONV949]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV950]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP589:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL951:%.*]] = icmp ne i8 [[TMP589]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL951]], label [[IF_THEN952:%.*]], label [[IF_ELSE953:%.*]]
+// SIMD-ONLY0:       if.then952:
+// SIMD-ONLY0-NEXT:    [[TMP590:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP590]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END954:%.*]]
+// SIMD-ONLY0:       if.else953:
+// SIMD-ONLY0-NEXT:    [[TMP591:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP591]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END954]]
+// SIMD-ONLY0:       if.end954:
+// SIMD-ONLY0-NEXT:    [[TMP592:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP592]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP593:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV955:%.*]] = zext i8 [[TMP593]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP594:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV956:%.*]] = zext i8 [[TMP594]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP957:%.*]] = icmp sgt i32 [[CONV955]], [[CONV956]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP957]], label [[IF_THEN959:%.*]], label [[IF_END960:%.*]]
+// SIMD-ONLY0:       if.then959:
+// SIMD-ONLY0-NEXT:    [[TMP595:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP595]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END960]]
+// SIMD-ONLY0:       if.end960:
+// SIMD-ONLY0-NEXT:    [[TMP596:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP596]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP597:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV961:%.*]] = zext i8 [[TMP597]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP598:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV962:%.*]] = zext i8 [[TMP598]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP963:%.*]] = icmp sgt i32 [[CONV961]], [[CONV962]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP963]], label [[IF_THEN965:%.*]], label [[IF_END966:%.*]]
+// SIMD-ONLY0:       if.then965:
+// SIMD-ONLY0-NEXT:    [[TMP599:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP599]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END966]]
+// SIMD-ONLY0:       if.end966:
+// SIMD-ONLY0-NEXT:    [[TMP600:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP600]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP601:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV967:%.*]] = zext i8 [[TMP601]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP602:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV968:%.*]] = zext i8 [[TMP602]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP969:%.*]] = icmp slt i32 [[CONV967]], [[CONV968]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP969]], label [[IF_THEN971:%.*]], label [[IF_END972:%.*]]
+// SIMD-ONLY0:       if.then971:
+// SIMD-ONLY0-NEXT:    [[TMP603:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP603]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END972]]
+// SIMD-ONLY0:       if.end972:
+// SIMD-ONLY0-NEXT:    [[TMP604:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP604]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP605:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV973:%.*]] = zext i8 [[TMP605]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP606:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV974:%.*]] = zext i8 [[TMP606]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP975:%.*]] = icmp slt i32 [[CONV973]], [[CONV974]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP975]], label [[IF_THEN977:%.*]], label [[IF_END978:%.*]]
+// SIMD-ONLY0:       if.then977:
+// SIMD-ONLY0-NEXT:    [[TMP607:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP607]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END978]]
+// SIMD-ONLY0:       if.end978:
+// SIMD-ONLY0-NEXT:    [[TMP608:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP608]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP609:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV979:%.*]] = zext i8 [[TMP609]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP610:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV980:%.*]] = zext i8 [[TMP610]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP981:%.*]] = icmp eq i32 [[CONV979]], [[CONV980]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP981]], label [[IF_THEN983:%.*]], label [[IF_END984:%.*]]
+// SIMD-ONLY0:       if.then983:
+// SIMD-ONLY0-NEXT:    [[TMP611:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP611]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END984]]
+// SIMD-ONLY0:       if.end984:
+// SIMD-ONLY0-NEXT:    [[TMP612:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP612]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP613:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV985:%.*]] = zext i8 [[TMP613]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP614:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV986:%.*]] = zext i8 [[TMP614]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP987:%.*]] = icmp eq i32 [[CONV985]], [[CONV986]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP987]], label [[IF_THEN989:%.*]], label [[IF_END990:%.*]]
+// SIMD-ONLY0:       if.then989:
+// SIMD-ONLY0-NEXT:    [[TMP615:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP615]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END990]]
+// SIMD-ONLY0:       if.end990:
+// SIMD-ONLY0-NEXT:    [[TMP616:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV991:%.*]] = zext i8 [[TMP616]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP617:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV992:%.*]] = zext i8 [[TMP617]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP993:%.*]] = icmp sgt i32 [[CONV991]], [[CONV992]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP993]], label [[IF_THEN995:%.*]], label [[IF_END996:%.*]]
+// SIMD-ONLY0:       if.then995:
+// SIMD-ONLY0-NEXT:    [[TMP618:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP618]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END996]]
+// SIMD-ONLY0:       if.end996:
+// SIMD-ONLY0-NEXT:    [[TMP619:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP619]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP620:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV997:%.*]] = zext i8 [[TMP620]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP621:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV998:%.*]] = zext i8 [[TMP621]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP999:%.*]] = icmp sgt i32 [[CONV997]], [[CONV998]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP999]], label [[IF_THEN1001:%.*]], label [[IF_END1002:%.*]]
+// SIMD-ONLY0:       if.then1001:
+// SIMD-ONLY0-NEXT:    [[TMP622:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP622]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1002]]
+// SIMD-ONLY0:       if.end1002:
+// SIMD-ONLY0-NEXT:    [[TMP623:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP623]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP624:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1003:%.*]] = zext i8 [[TMP624]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP625:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1004:%.*]] = zext i8 [[TMP625]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1005:%.*]] = icmp slt i32 [[CONV1003]], [[CONV1004]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1005]], label [[IF_THEN1007:%.*]], label [[IF_END1008:%.*]]
+// SIMD-ONLY0:       if.then1007:
+// SIMD-ONLY0-NEXT:    [[TMP626:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP626]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1008]]
+// SIMD-ONLY0:       if.end1008:
+// SIMD-ONLY0-NEXT:    [[TMP627:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP627]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP628:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1009:%.*]] = zext i8 [[TMP628]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP629:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1010:%.*]] = zext i8 [[TMP629]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1011:%.*]] = icmp slt i32 [[CONV1009]], [[CONV1010]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1011]], label [[IF_THEN1013:%.*]], label [[IF_END1014:%.*]]
+// SIMD-ONLY0:       if.then1013:
+// SIMD-ONLY0-NEXT:    [[TMP630:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP630]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1014]]
+// SIMD-ONLY0:       if.end1014:
+// SIMD-ONLY0-NEXT:    [[TMP631:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP631]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP632:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1015:%.*]] = zext i8 [[TMP632]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP633:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1016:%.*]] = zext i8 [[TMP633]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1017:%.*]] = icmp eq i32 [[CONV1015]], [[CONV1016]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1017]], label [[IF_THEN1019:%.*]], label [[IF_END1020:%.*]]
+// SIMD-ONLY0:       if.then1019:
+// SIMD-ONLY0-NEXT:    [[TMP634:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP634]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1020]]
+// SIMD-ONLY0:       if.end1020:
+// SIMD-ONLY0-NEXT:    [[TMP635:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP635]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP636:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1021:%.*]] = zext i8 [[TMP636]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP637:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1022:%.*]] = zext i8 [[TMP637]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1023:%.*]] = icmp eq i32 [[CONV1021]], [[CONV1022]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1023]], label [[IF_THEN1025:%.*]], label [[IF_END1026:%.*]]
+// SIMD-ONLY0:       if.then1025:
+// SIMD-ONLY0-NEXT:    [[TMP638:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP638]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1026]]
+// SIMD-ONLY0:       if.end1026:
+// SIMD-ONLY0-NEXT:    [[TMP639:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP639]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP640:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1027:%.*]] = zext i8 [[TMP640]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP641:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1028:%.*]] = zext i8 [[TMP641]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1029:%.*]] = icmp eq i32 [[CONV1027]], [[CONV1028]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1029]], label [[IF_THEN1031:%.*]], label [[IF_ELSE1032:%.*]]
+// SIMD-ONLY0:       if.then1031:
+// SIMD-ONLY0-NEXT:    [[TMP642:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP642]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1033:%.*]]
+// SIMD-ONLY0:       if.else1032:
+// SIMD-ONLY0-NEXT:    [[TMP643:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP643]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1033]]
+// SIMD-ONLY0:       if.end1033:
+// SIMD-ONLY0-NEXT:    [[TMP644:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1034:%.*]] = zext i8 [[TMP644]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP645:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1035:%.*]] = zext i8 [[TMP645]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1036:%.*]] = icmp eq i32 [[CONV1034]], [[CONV1035]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1036]], label [[IF_THEN1038:%.*]], label [[IF_ELSE1039:%.*]]
+// SIMD-ONLY0:       if.then1038:
+// SIMD-ONLY0-NEXT:    [[TMP646:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP646]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1040:%.*]]
+// SIMD-ONLY0:       if.else1039:
+// SIMD-ONLY0-NEXT:    [[TMP647:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP647]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1040]]
+// SIMD-ONLY0:       if.end1040:
+// SIMD-ONLY0-NEXT:    [[TMP648:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1041:%.*]] = zext i8 [[TMP648]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP649:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1042:%.*]] = zext i8 [[TMP649]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1043:%.*]] = icmp eq i32 [[CONV1041]], [[CONV1042]]
+// SIMD-ONLY0-NEXT:    [[CONV1044:%.*]] = zext i1 [[CMP1043]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1045:%.*]] = trunc i32 [[CONV1044]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1045]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP650:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL1046:%.*]] = icmp ne i8 [[TMP650]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1046]], label [[IF_THEN1047:%.*]], label [[IF_END1048:%.*]]
+// SIMD-ONLY0:       if.then1047:
+// SIMD-ONLY0-NEXT:    [[TMP651:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP651]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1048]]
+// SIMD-ONLY0:       if.end1048:
+// SIMD-ONLY0-NEXT:    [[TMP652:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1049:%.*]] = zext i8 [[TMP652]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP653:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1050:%.*]] = zext i8 [[TMP653]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1051:%.*]] = icmp eq i32 [[CONV1049]], [[CONV1050]]
+// SIMD-ONLY0-NEXT:    [[CONV1052:%.*]] = zext i1 [[CMP1051]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1053:%.*]] = trunc i32 [[CONV1052]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1053]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP654:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL1054:%.*]] = icmp ne i8 [[TMP654]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1054]], label [[IF_THEN1055:%.*]], label [[IF_END1056:%.*]]
+// SIMD-ONLY0:       if.then1055:
+// SIMD-ONLY0-NEXT:    [[TMP655:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP655]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1056]]
+// SIMD-ONLY0:       if.end1056:
+// SIMD-ONLY0-NEXT:    [[TMP656:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1057:%.*]] = zext i8 [[TMP656]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP657:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1058:%.*]] = zext i8 [[TMP657]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1059:%.*]] = icmp eq i32 [[CONV1057]], [[CONV1058]]
+// SIMD-ONLY0-NEXT:    [[CONV1060:%.*]] = zext i1 [[CMP1059]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1061:%.*]] = trunc i32 [[CONV1060]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1061]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP658:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL1062:%.*]] = icmp ne i8 [[TMP658]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1062]], label [[IF_THEN1063:%.*]], label [[IF_ELSE1064:%.*]]
+// SIMD-ONLY0:       if.then1063:
+// SIMD-ONLY0-NEXT:    [[TMP659:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP659]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1065:%.*]]
+// SIMD-ONLY0:       if.else1064:
+// SIMD-ONLY0-NEXT:    [[TMP660:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP660]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1065]]
+// SIMD-ONLY0:       if.end1065:
+// SIMD-ONLY0-NEXT:    [[TMP661:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1066:%.*]] = zext i8 [[TMP661]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP662:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1067:%.*]] = zext i8 [[TMP662]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1068:%.*]] = icmp eq i32 [[CONV1066]], [[CONV1067]]
+// SIMD-ONLY0-NEXT:    [[CONV1069:%.*]] = zext i1 [[CMP1068]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1070:%.*]] = trunc i32 [[CONV1069]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1070]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP663:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL1071:%.*]] = icmp ne i8 [[TMP663]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1071]], label [[IF_THEN1072:%.*]], label [[IF_ELSE1073:%.*]]
+// SIMD-ONLY0:       if.then1072:
+// SIMD-ONLY0-NEXT:    [[TMP664:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP664]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1074:%.*]]
+// SIMD-ONLY0:       if.else1073:
+// SIMD-ONLY0-NEXT:    [[TMP665:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP665]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1074]]
+// SIMD-ONLY0:       if.end1074:
+// SIMD-ONLY0-NEXT:    [[TMP666:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP666]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP667:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1075:%.*]] = zext i8 [[TMP667]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP668:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1076:%.*]] = zext i8 [[TMP668]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1077:%.*]] = icmp sgt i32 [[CONV1075]], [[CONV1076]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1077]], label [[IF_THEN1079:%.*]], label [[IF_END1080:%.*]]
+// SIMD-ONLY0:       if.then1079:
+// SIMD-ONLY0-NEXT:    [[TMP669:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP669]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1080]]
+// SIMD-ONLY0:       if.end1080:
+// SIMD-ONLY0-NEXT:    [[TMP670:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP670]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP671:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1081:%.*]] = zext i8 [[TMP671]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP672:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1082:%.*]] = zext i8 [[TMP672]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1083:%.*]] = icmp sgt i32 [[CONV1081]], [[CONV1082]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1083]], label [[IF_THEN1085:%.*]], label [[IF_END1086:%.*]]
+// SIMD-ONLY0:       if.then1085:
+// SIMD-ONLY0-NEXT:    [[TMP673:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP673]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1086]]
+// SIMD-ONLY0:       if.end1086:
+// SIMD-ONLY0-NEXT:    [[TMP674:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP674]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP675:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1087:%.*]] = zext i8 [[TMP675]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP676:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1088:%.*]] = zext i8 [[TMP676]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1089:%.*]] = icmp slt i32 [[CONV1087]], [[CONV1088]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1089]], label [[IF_THEN1091:%.*]], label [[IF_END1092:%.*]]
+// SIMD-ONLY0:       if.then1091:
+// SIMD-ONLY0-NEXT:    [[TMP677:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP677]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1092]]
+// SIMD-ONLY0:       if.end1092:
+// SIMD-ONLY0-NEXT:    [[TMP678:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP678]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP679:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1093:%.*]] = zext i8 [[TMP679]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP680:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1094:%.*]] = zext i8 [[TMP680]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1095:%.*]] = icmp slt i32 [[CONV1093]], [[CONV1094]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1095]], label [[IF_THEN1097:%.*]], label [[IF_END1098:%.*]]
+// SIMD-ONLY0:       if.then1097:
+// SIMD-ONLY0-NEXT:    [[TMP681:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP681]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1098]]
+// SIMD-ONLY0:       if.end1098:
+// SIMD-ONLY0-NEXT:    [[TMP682:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP682]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP683:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1099:%.*]] = zext i8 [[TMP683]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP684:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1100:%.*]] = zext i8 [[TMP684]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1101:%.*]] = icmp eq i32 [[CONV1099]], [[CONV1100]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1101]], label [[IF_THEN1103:%.*]], label [[IF_END1104:%.*]]
+// SIMD-ONLY0:       if.then1103:
+// SIMD-ONLY0-NEXT:    [[TMP685:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP685]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1104]]
+// SIMD-ONLY0:       if.end1104:
+// SIMD-ONLY0-NEXT:    [[TMP686:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP686]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP687:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1105:%.*]] = zext i8 [[TMP687]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP688:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1106:%.*]] = zext i8 [[TMP688]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1107:%.*]] = icmp eq i32 [[CONV1105]], [[CONV1106]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1107]], label [[IF_THEN1109:%.*]], label [[IF_END1110:%.*]]
+// SIMD-ONLY0:       if.then1109:
+// SIMD-ONLY0-NEXT:    [[TMP689:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP689]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1110]]
+// SIMD-ONLY0:       if.end1110:
+// SIMD-ONLY0-NEXT:    [[TMP690:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1111:%.*]] = zext i8 [[TMP690]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP691:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1112:%.*]] = zext i8 [[TMP691]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1113:%.*]] = icmp sgt i32 [[CONV1111]], [[CONV1112]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1113]], label [[IF_THEN1115:%.*]], label [[IF_END1116:%.*]]
+// SIMD-ONLY0:       if.then1115:
+// SIMD-ONLY0-NEXT:    [[TMP692:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP692]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1116]]
+// SIMD-ONLY0:       if.end1116:
+// SIMD-ONLY0-NEXT:    [[TMP693:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP693]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP694:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1117:%.*]] = zext i8 [[TMP694]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP695:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1118:%.*]] = zext i8 [[TMP695]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1119:%.*]] = icmp sgt i32 [[CONV1117]], [[CONV1118]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1119]], label [[IF_THEN1121:%.*]], label [[IF_END1122:%.*]]
+// SIMD-ONLY0:       if.then1121:
+// SIMD-ONLY0-NEXT:    [[TMP696:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP696]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1122]]
+// SIMD-ONLY0:       if.end1122:
+// SIMD-ONLY0-NEXT:    [[TMP697:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP697]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP698:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1123:%.*]] = zext i8 [[TMP698]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP699:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1124:%.*]] = zext i8 [[TMP699]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1125:%.*]] = icmp slt i32 [[CONV1123]], [[CONV1124]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1125]], label [[IF_THEN1127:%.*]], label [[IF_END1128:%.*]]
+// SIMD-ONLY0:       if.then1127:
+// SIMD-ONLY0-NEXT:    [[TMP700:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP700]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1128]]
+// SIMD-ONLY0:       if.end1128:
+// SIMD-ONLY0-NEXT:    [[TMP701:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP701]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP702:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1129:%.*]] = zext i8 [[TMP702]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP703:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1130:%.*]] = zext i8 [[TMP703]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1131:%.*]] = icmp slt i32 [[CONV1129]], [[CONV1130]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1131]], label [[IF_THEN1133:%.*]], label [[IF_END1134:%.*]]
+// SIMD-ONLY0:       if.then1133:
+// SIMD-ONLY0-NEXT:    [[TMP704:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP704]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1134]]
+// SIMD-ONLY0:       if.end1134:
+// SIMD-ONLY0-NEXT:    [[TMP705:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP705]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP706:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1135:%.*]] = zext i8 [[TMP706]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP707:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1136:%.*]] = zext i8 [[TMP707]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1137:%.*]] = icmp eq i32 [[CONV1135]], [[CONV1136]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1137]], label [[IF_THEN1139:%.*]], label [[IF_END1140:%.*]]
+// SIMD-ONLY0:       if.then1139:
+// SIMD-ONLY0-NEXT:    [[TMP708:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP708]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1140]]
+// SIMD-ONLY0:       if.end1140:
+// SIMD-ONLY0-NEXT:    [[TMP709:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP709]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP710:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1141:%.*]] = zext i8 [[TMP710]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP711:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1142:%.*]] = zext i8 [[TMP711]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1143:%.*]] = icmp eq i32 [[CONV1141]], [[CONV1142]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1143]], label [[IF_THEN1145:%.*]], label [[IF_END1146:%.*]]
+// SIMD-ONLY0:       if.then1145:
+// SIMD-ONLY0-NEXT:    [[TMP712:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP712]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1146]]
+// SIMD-ONLY0:       if.end1146:
+// SIMD-ONLY0-NEXT:    [[TMP713:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP713]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP714:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1147:%.*]] = zext i8 [[TMP714]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP715:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1148:%.*]] = zext i8 [[TMP715]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1149:%.*]] = icmp eq i32 [[CONV1147]], [[CONV1148]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1149]], label [[IF_THEN1151:%.*]], label [[IF_ELSE1152:%.*]]
+// SIMD-ONLY0:       if.then1151:
+// SIMD-ONLY0-NEXT:    [[TMP716:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP716]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1153:%.*]]
+// SIMD-ONLY0:       if.else1152:
+// SIMD-ONLY0-NEXT:    [[TMP717:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP717]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1153]]
+// SIMD-ONLY0:       if.end1153:
+// SIMD-ONLY0-NEXT:    [[TMP718:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1154:%.*]] = zext i8 [[TMP718]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP719:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1155:%.*]] = zext i8 [[TMP719]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1156:%.*]] = icmp eq i32 [[CONV1154]], [[CONV1155]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1156]], label [[IF_THEN1158:%.*]], label [[IF_ELSE1159:%.*]]
+// SIMD-ONLY0:       if.then1158:
+// SIMD-ONLY0-NEXT:    [[TMP720:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP720]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1160:%.*]]
+// SIMD-ONLY0:       if.else1159:
+// SIMD-ONLY0-NEXT:    [[TMP721:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP721]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1160]]
+// SIMD-ONLY0:       if.end1160:
+// SIMD-ONLY0-NEXT:    [[TMP722:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1161:%.*]] = zext i8 [[TMP722]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP723:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1162:%.*]] = zext i8 [[TMP723]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1163:%.*]] = icmp eq i32 [[CONV1161]], [[CONV1162]]
+// SIMD-ONLY0-NEXT:    [[CONV1164:%.*]] = zext i1 [[CMP1163]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1165:%.*]] = trunc i32 [[CONV1164]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1165]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP724:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL1166:%.*]] = icmp ne i8 [[TMP724]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1166]], label [[IF_THEN1167:%.*]], label [[IF_END1168:%.*]]
+// SIMD-ONLY0:       if.then1167:
+// SIMD-ONLY0-NEXT:    [[TMP725:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP725]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1168]]
+// SIMD-ONLY0:       if.end1168:
+// SIMD-ONLY0-NEXT:    [[TMP726:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1169:%.*]] = zext i8 [[TMP726]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP727:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1170:%.*]] = zext i8 [[TMP727]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1171:%.*]] = icmp eq i32 [[CONV1169]], [[CONV1170]]
+// SIMD-ONLY0-NEXT:    [[CONV1172:%.*]] = zext i1 [[CMP1171]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1173:%.*]] = trunc i32 [[CONV1172]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1173]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP728:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL1174:%.*]] = icmp ne i8 [[TMP728]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1174]], label [[IF_THEN1175:%.*]], label [[IF_END1176:%.*]]
+// SIMD-ONLY0:       if.then1175:
+// SIMD-ONLY0-NEXT:    [[TMP729:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP729]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1176]]
+// SIMD-ONLY0:       if.end1176:
+// SIMD-ONLY0-NEXT:    [[TMP730:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1177:%.*]] = zext i8 [[TMP730]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP731:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1178:%.*]] = zext i8 [[TMP731]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1179:%.*]] = icmp eq i32 [[CONV1177]], [[CONV1178]]
+// SIMD-ONLY0-NEXT:    [[CONV1180:%.*]] = zext i1 [[CMP1179]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1181:%.*]] = trunc i32 [[CONV1180]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1181]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP732:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL1182:%.*]] = icmp ne i8 [[TMP732]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1182]], label [[IF_THEN1183:%.*]], label [[IF_ELSE1184:%.*]]
+// SIMD-ONLY0:       if.then1183:
+// SIMD-ONLY0-NEXT:    [[TMP733:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP733]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1185:%.*]]
+// SIMD-ONLY0:       if.else1184:
+// SIMD-ONLY0-NEXT:    [[TMP734:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP734]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1185]]
+// SIMD-ONLY0:       if.end1185:
+// SIMD-ONLY0-NEXT:    [[TMP735:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1186:%.*]] = zext i8 [[TMP735]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP736:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1187:%.*]] = zext i8 [[TMP736]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1188:%.*]] = icmp eq i32 [[CONV1186]], [[CONV1187]]
+// SIMD-ONLY0-NEXT:    [[CONV1189:%.*]] = zext i1 [[CMP1188]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1190:%.*]] = trunc i32 [[CONV1189]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1190]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP737:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL1191:%.*]] = icmp ne i8 [[TMP737]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1191]], label [[IF_THEN1192:%.*]], label [[IF_ELSE1193:%.*]]
+// SIMD-ONLY0:       if.then1192:
+// SIMD-ONLY0-NEXT:    [[TMP738:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP738]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1194:%.*]]
+// SIMD-ONLY0:       if.else1193:
+// SIMD-ONLY0-NEXT:    [[TMP739:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP739]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1194]]
+// SIMD-ONLY0:       if.end1194:
+// SIMD-ONLY0-NEXT:    [[TMP740:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP740]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP741:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1195:%.*]] = zext i8 [[TMP741]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP742:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1196:%.*]] = zext i8 [[TMP742]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1197:%.*]] = icmp sgt i32 [[CONV1195]], [[CONV1196]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1197]], label [[IF_THEN1199:%.*]], label [[IF_END1200:%.*]]
+// SIMD-ONLY0:       if.then1199:
+// SIMD-ONLY0-NEXT:    [[TMP743:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP743]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1200]]
+// SIMD-ONLY0:       if.end1200:
+// SIMD-ONLY0-NEXT:    [[TMP744:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP744]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP745:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1201:%.*]] = zext i8 [[TMP745]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP746:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1202:%.*]] = zext i8 [[TMP746]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1203:%.*]] = icmp sgt i32 [[CONV1201]], [[CONV1202]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1203]], label [[IF_THEN1205:%.*]], label [[IF_END1206:%.*]]
+// SIMD-ONLY0:       if.then1205:
+// SIMD-ONLY0-NEXT:    [[TMP747:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP747]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1206]]
+// SIMD-ONLY0:       if.end1206:
+// SIMD-ONLY0-NEXT:    [[TMP748:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP748]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP749:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1207:%.*]] = zext i8 [[TMP749]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP750:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1208:%.*]] = zext i8 [[TMP750]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1209:%.*]] = icmp slt i32 [[CONV1207]], [[CONV1208]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1209]], label [[IF_THEN1211:%.*]], label [[IF_END1212:%.*]]
+// SIMD-ONLY0:       if.then1211:
+// SIMD-ONLY0-NEXT:    [[TMP751:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP751]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1212]]
+// SIMD-ONLY0:       if.end1212:
+// SIMD-ONLY0-NEXT:    [[TMP752:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP752]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP753:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1213:%.*]] = zext i8 [[TMP753]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP754:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1214:%.*]] = zext i8 [[TMP754]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1215:%.*]] = icmp slt i32 [[CONV1213]], [[CONV1214]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1215]], label [[IF_THEN1217:%.*]], label [[IF_END1218:%.*]]
+// SIMD-ONLY0:       if.then1217:
+// SIMD-ONLY0-NEXT:    [[TMP755:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP755]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1218]]
+// SIMD-ONLY0:       if.end1218:
+// SIMD-ONLY0-NEXT:    [[TMP756:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP756]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP757:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1219:%.*]] = zext i8 [[TMP757]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP758:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1220:%.*]] = zext i8 [[TMP758]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1221:%.*]] = icmp eq i32 [[CONV1219]], [[CONV1220]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1221]], label [[IF_THEN1223:%.*]], label [[IF_END1224:%.*]]
+// SIMD-ONLY0:       if.then1223:
+// SIMD-ONLY0-NEXT:    [[TMP759:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP759]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1224]]
+// SIMD-ONLY0:       if.end1224:
+// SIMD-ONLY0-NEXT:    [[TMP760:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP760]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP761:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1225:%.*]] = zext i8 [[TMP761]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP762:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1226:%.*]] = zext i8 [[TMP762]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1227:%.*]] = icmp eq i32 [[CONV1225]], [[CONV1226]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1227]], label [[IF_THEN1229:%.*]], label [[IF_END1230:%.*]]
+// SIMD-ONLY0:       if.then1229:
+// SIMD-ONLY0-NEXT:    [[TMP763:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP763]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1230]]
+// SIMD-ONLY0:       if.end1230:
+// SIMD-ONLY0-NEXT:    [[TMP764:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1231:%.*]] = zext i8 [[TMP764]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP765:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1232:%.*]] = zext i8 [[TMP765]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1233:%.*]] = icmp sgt i32 [[CONV1231]], [[CONV1232]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1233]], label [[IF_THEN1235:%.*]], label [[IF_END1236:%.*]]
+// SIMD-ONLY0:       if.then1235:
+// SIMD-ONLY0-NEXT:    [[TMP766:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP766]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1236]]
+// SIMD-ONLY0:       if.end1236:
+// SIMD-ONLY0-NEXT:    [[TMP767:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP767]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP768:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1237:%.*]] = zext i8 [[TMP768]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP769:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1238:%.*]] = zext i8 [[TMP769]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1239:%.*]] = icmp sgt i32 [[CONV1237]], [[CONV1238]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1239]], label [[IF_THEN1241:%.*]], label [[IF_END1242:%.*]]
+// SIMD-ONLY0:       if.then1241:
+// SIMD-ONLY0-NEXT:    [[TMP770:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP770]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1242]]
+// SIMD-ONLY0:       if.end1242:
+// SIMD-ONLY0-NEXT:    [[TMP771:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP771]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP772:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1243:%.*]] = zext i8 [[TMP772]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP773:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1244:%.*]] = zext i8 [[TMP773]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1245:%.*]] = icmp slt i32 [[CONV1243]], [[CONV1244]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1245]], label [[IF_THEN1247:%.*]], label [[IF_END1248:%.*]]
+// SIMD-ONLY0:       if.then1247:
+// SIMD-ONLY0-NEXT:    [[TMP774:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP774]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1248]]
+// SIMD-ONLY0:       if.end1248:
+// SIMD-ONLY0-NEXT:    [[TMP775:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP775]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP776:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1249:%.*]] = zext i8 [[TMP776]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP777:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1250:%.*]] = zext i8 [[TMP777]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1251:%.*]] = icmp slt i32 [[CONV1249]], [[CONV1250]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1251]], label [[IF_THEN1253:%.*]], label [[IF_END1254:%.*]]
+// SIMD-ONLY0:       if.then1253:
+// SIMD-ONLY0-NEXT:    [[TMP778:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP778]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1254]]
+// SIMD-ONLY0:       if.end1254:
+// SIMD-ONLY0-NEXT:    [[TMP779:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP779]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP780:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1255:%.*]] = zext i8 [[TMP780]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP781:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1256:%.*]] = zext i8 [[TMP781]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1257:%.*]] = icmp eq i32 [[CONV1255]], [[CONV1256]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1257]], label [[IF_THEN1259:%.*]], label [[IF_END1260:%.*]]
+// SIMD-ONLY0:       if.then1259:
+// SIMD-ONLY0-NEXT:    [[TMP782:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP782]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1260]]
+// SIMD-ONLY0:       if.end1260:
+// SIMD-ONLY0-NEXT:    [[TMP783:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP783]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP784:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1261:%.*]] = zext i8 [[TMP784]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP785:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1262:%.*]] = zext i8 [[TMP785]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1263:%.*]] = icmp eq i32 [[CONV1261]], [[CONV1262]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1263]], label [[IF_THEN1265:%.*]], label [[IF_END1266:%.*]]
+// SIMD-ONLY0:       if.then1265:
+// SIMD-ONLY0-NEXT:    [[TMP786:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP786]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1266]]
+// SIMD-ONLY0:       if.end1266:
+// SIMD-ONLY0-NEXT:    [[TMP787:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP787]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP788:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1267:%.*]] = zext i8 [[TMP788]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP789:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1268:%.*]] = zext i8 [[TMP789]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1269:%.*]] = icmp eq i32 [[CONV1267]], [[CONV1268]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1269]], label [[IF_THEN1271:%.*]], label [[IF_ELSE1272:%.*]]
+// SIMD-ONLY0:       if.then1271:
+// SIMD-ONLY0-NEXT:    [[TMP790:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP790]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1273:%.*]]
+// SIMD-ONLY0:       if.else1272:
+// SIMD-ONLY0-NEXT:    [[TMP791:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP791]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1273]]
+// SIMD-ONLY0:       if.end1273:
+// SIMD-ONLY0-NEXT:    [[TMP792:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1274:%.*]] = zext i8 [[TMP792]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP793:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1275:%.*]] = zext i8 [[TMP793]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1276:%.*]] = icmp eq i32 [[CONV1274]], [[CONV1275]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1276]], label [[IF_THEN1278:%.*]], label [[IF_ELSE1279:%.*]]
+// SIMD-ONLY0:       if.then1278:
+// SIMD-ONLY0-NEXT:    [[TMP794:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP794]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1280:%.*]]
+// SIMD-ONLY0:       if.else1279:
+// SIMD-ONLY0-NEXT:    [[TMP795:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP795]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1280]]
+// SIMD-ONLY0:       if.end1280:
+// SIMD-ONLY0-NEXT:    [[TMP796:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1281:%.*]] = zext i8 [[TMP796]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP797:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1282:%.*]] = zext i8 [[TMP797]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1283:%.*]] = icmp eq i32 [[CONV1281]], [[CONV1282]]
+// SIMD-ONLY0-NEXT:    [[CONV1284:%.*]] = zext i1 [[CMP1283]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1285:%.*]] = trunc i32 [[CONV1284]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1285]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP798:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL1286:%.*]] = icmp ne i8 [[TMP798]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1286]], label [[IF_THEN1287:%.*]], label [[IF_END1288:%.*]]
+// SIMD-ONLY0:       if.then1287:
+// SIMD-ONLY0-NEXT:    [[TMP799:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP799]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1288]]
+// SIMD-ONLY0:       if.end1288:
+// SIMD-ONLY0-NEXT:    [[TMP800:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1289:%.*]] = zext i8 [[TMP800]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP801:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1290:%.*]] = zext i8 [[TMP801]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1291:%.*]] = icmp eq i32 [[CONV1289]], [[CONV1290]]
+// SIMD-ONLY0-NEXT:    [[CONV1292:%.*]] = zext i1 [[CMP1291]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1293:%.*]] = trunc i32 [[CONV1292]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1293]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP802:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL1294:%.*]] = icmp ne i8 [[TMP802]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1294]], label [[IF_THEN1295:%.*]], label [[IF_END1296:%.*]]
+// SIMD-ONLY0:       if.then1295:
+// SIMD-ONLY0-NEXT:    [[TMP803:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP803]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1296]]
+// SIMD-ONLY0:       if.end1296:
+// SIMD-ONLY0-NEXT:    [[TMP804:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1297:%.*]] = zext i8 [[TMP804]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP805:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1298:%.*]] = zext i8 [[TMP805]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1299:%.*]] = icmp eq i32 [[CONV1297]], [[CONV1298]]
+// SIMD-ONLY0-NEXT:    [[CONV1300:%.*]] = zext i1 [[CMP1299]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1301:%.*]] = trunc i32 [[CONV1300]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1301]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP806:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL1302:%.*]] = icmp ne i8 [[TMP806]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1302]], label [[IF_THEN1303:%.*]], label [[IF_ELSE1304:%.*]]
+// SIMD-ONLY0:       if.then1303:
+// SIMD-ONLY0-NEXT:    [[TMP807:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP807]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1305:%.*]]
+// SIMD-ONLY0:       if.else1304:
+// SIMD-ONLY0-NEXT:    [[TMP808:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP808]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1305]]
+// SIMD-ONLY0:       if.end1305:
+// SIMD-ONLY0-NEXT:    [[TMP809:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1306:%.*]] = zext i8 [[TMP809]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP810:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1307:%.*]] = zext i8 [[TMP810]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1308:%.*]] = icmp eq i32 [[CONV1306]], [[CONV1307]]
+// SIMD-ONLY0-NEXT:    [[CONV1309:%.*]] = zext i1 [[CMP1308]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1310:%.*]] = trunc i32 [[CONV1309]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1310]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP811:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL1311:%.*]] = icmp ne i8 [[TMP811]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1311]], label [[IF_THEN1312:%.*]], label [[IF_ELSE1313:%.*]]
+// SIMD-ONLY0:       if.then1312:
+// SIMD-ONLY0-NEXT:    [[TMP812:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP812]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1314:%.*]]
+// SIMD-ONLY0:       if.else1313:
+// SIMD-ONLY0-NEXT:    [[TMP813:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP813]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1314]]
+// SIMD-ONLY0:       if.end1314:
+// SIMD-ONLY0-NEXT:    [[TMP814:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP814]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP815:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1315:%.*]] = zext i8 [[TMP815]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP816:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1316:%.*]] = zext i8 [[TMP816]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1317:%.*]] = icmp sgt i32 [[CONV1315]], [[CONV1316]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1317]], label [[IF_THEN1319:%.*]], label [[IF_END1320:%.*]]
+// SIMD-ONLY0:       if.then1319:
+// SIMD-ONLY0-NEXT:    [[TMP817:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP817]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1320]]
+// SIMD-ONLY0:       if.end1320:
+// SIMD-ONLY0-NEXT:    [[TMP818:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP818]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP819:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1321:%.*]] = zext i8 [[TMP819]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP820:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1322:%.*]] = zext i8 [[TMP820]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1323:%.*]] = icmp sgt i32 [[CONV1321]], [[CONV1322]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1323]], label [[IF_THEN1325:%.*]], label [[IF_END1326:%.*]]
+// SIMD-ONLY0:       if.then1325:
+// SIMD-ONLY0-NEXT:    [[TMP821:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP821]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1326]]
+// SIMD-ONLY0:       if.end1326:
+// SIMD-ONLY0-NEXT:    [[TMP822:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP822]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP823:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1327:%.*]] = zext i8 [[TMP823]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP824:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1328:%.*]] = zext i8 [[TMP824]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1329:%.*]] = icmp slt i32 [[CONV1327]], [[CONV1328]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1329]], label [[IF_THEN1331:%.*]], label [[IF_END1332:%.*]]
+// SIMD-ONLY0:       if.then1331:
+// SIMD-ONLY0-NEXT:    [[TMP825:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP825]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1332]]
+// SIMD-ONLY0:       if.end1332:
+// SIMD-ONLY0-NEXT:    [[TMP826:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP826]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP827:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1333:%.*]] = zext i8 [[TMP827]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP828:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1334:%.*]] = zext i8 [[TMP828]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1335:%.*]] = icmp slt i32 [[CONV1333]], [[CONV1334]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1335]], label [[IF_THEN1337:%.*]], label [[IF_END1338:%.*]]
+// SIMD-ONLY0:       if.then1337:
+// SIMD-ONLY0-NEXT:    [[TMP829:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP829]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1338]]
+// SIMD-ONLY0:       if.end1338:
+// SIMD-ONLY0-NEXT:    [[TMP830:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP830]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP831:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1339:%.*]] = zext i8 [[TMP831]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP832:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1340:%.*]] = zext i8 [[TMP832]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1341:%.*]] = icmp eq i32 [[CONV1339]], [[CONV1340]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1341]], label [[IF_THEN1343:%.*]], label [[IF_END1344:%.*]]
+// SIMD-ONLY0:       if.then1343:
+// SIMD-ONLY0-NEXT:    [[TMP833:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP833]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1344]]
+// SIMD-ONLY0:       if.end1344:
+// SIMD-ONLY0-NEXT:    [[TMP834:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP834]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP835:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1345:%.*]] = zext i8 [[TMP835]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP836:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1346:%.*]] = zext i8 [[TMP836]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1347:%.*]] = icmp eq i32 [[CONV1345]], [[CONV1346]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1347]], label [[IF_THEN1349:%.*]], label [[IF_END1350:%.*]]
+// SIMD-ONLY0:       if.then1349:
+// SIMD-ONLY0-NEXT:    [[TMP837:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP837]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1350]]
+// SIMD-ONLY0:       if.end1350:
+// SIMD-ONLY0-NEXT:    [[TMP838:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1351:%.*]] = zext i8 [[TMP838]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP839:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1352:%.*]] = zext i8 [[TMP839]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1353:%.*]] = icmp sgt i32 [[CONV1351]], [[CONV1352]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1353]], label [[IF_THEN1355:%.*]], label [[IF_END1356:%.*]]
+// SIMD-ONLY0:       if.then1355:
+// SIMD-ONLY0-NEXT:    [[TMP840:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP840]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1356]]
+// SIMD-ONLY0:       if.end1356:
+// SIMD-ONLY0-NEXT:    [[TMP841:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP841]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP842:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1357:%.*]] = zext i8 [[TMP842]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP843:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1358:%.*]] = zext i8 [[TMP843]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1359:%.*]] = icmp sgt i32 [[CONV1357]], [[CONV1358]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1359]], label [[IF_THEN1361:%.*]], label [[IF_END1362:%.*]]
+// SIMD-ONLY0:       if.then1361:
+// SIMD-ONLY0-NEXT:    [[TMP844:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP844]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1362]]
+// SIMD-ONLY0:       if.end1362:
+// SIMD-ONLY0-NEXT:    [[TMP845:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP845]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP846:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1363:%.*]] = zext i8 [[TMP846]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP847:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1364:%.*]] = zext i8 [[TMP847]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1365:%.*]] = icmp slt i32 [[CONV1363]], [[CONV1364]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1365]], label [[IF_THEN1367:%.*]], label [[IF_END1368:%.*]]
+// SIMD-ONLY0:       if.then1367:
+// SIMD-ONLY0-NEXT:    [[TMP848:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP848]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1368]]
+// SIMD-ONLY0:       if.end1368:
+// SIMD-ONLY0-NEXT:    [[TMP849:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP849]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP850:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1369:%.*]] = zext i8 [[TMP850]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP851:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1370:%.*]] = zext i8 [[TMP851]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1371:%.*]] = icmp slt i32 [[CONV1369]], [[CONV1370]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1371]], label [[IF_THEN1373:%.*]], label [[IF_END1374:%.*]]
+// SIMD-ONLY0:       if.then1373:
+// SIMD-ONLY0-NEXT:    [[TMP852:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP852]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1374]]
+// SIMD-ONLY0:       if.end1374:
+// SIMD-ONLY0-NEXT:    [[TMP853:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP853]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP854:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1375:%.*]] = zext i8 [[TMP854]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP855:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1376:%.*]] = zext i8 [[TMP855]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1377:%.*]] = icmp eq i32 [[CONV1375]], [[CONV1376]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1377]], label [[IF_THEN1379:%.*]], label [[IF_END1380:%.*]]
+// SIMD-ONLY0:       if.then1379:
+// SIMD-ONLY0-NEXT:    [[TMP856:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP856]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1380]]
+// SIMD-ONLY0:       if.end1380:
+// SIMD-ONLY0-NEXT:    [[TMP857:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP857]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP858:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1381:%.*]] = zext i8 [[TMP858]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP859:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1382:%.*]] = zext i8 [[TMP859]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1383:%.*]] = icmp eq i32 [[CONV1381]], [[CONV1382]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1383]], label [[IF_THEN1385:%.*]], label [[IF_END1386:%.*]]
+// SIMD-ONLY0:       if.then1385:
+// SIMD-ONLY0-NEXT:    [[TMP860:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP860]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1386]]
+// SIMD-ONLY0:       if.end1386:
+// SIMD-ONLY0-NEXT:    [[TMP861:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP861]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP862:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1387:%.*]] = zext i8 [[TMP862]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP863:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1388:%.*]] = zext i8 [[TMP863]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1389:%.*]] = icmp eq i32 [[CONV1387]], [[CONV1388]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1389]], label [[IF_THEN1391:%.*]], label [[IF_ELSE1392:%.*]]
+// SIMD-ONLY0:       if.then1391:
+// SIMD-ONLY0-NEXT:    [[TMP864:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP864]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1393:%.*]]
+// SIMD-ONLY0:       if.else1392:
+// SIMD-ONLY0-NEXT:    [[TMP865:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP865]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1393]]
+// SIMD-ONLY0:       if.end1393:
+// SIMD-ONLY0-NEXT:    [[TMP866:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1394:%.*]] = zext i8 [[TMP866]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP867:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1395:%.*]] = zext i8 [[TMP867]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1396:%.*]] = icmp eq i32 [[CONV1394]], [[CONV1395]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1396]], label [[IF_THEN1398:%.*]], label [[IF_ELSE1399:%.*]]
+// SIMD-ONLY0:       if.then1398:
+// SIMD-ONLY0-NEXT:    [[TMP868:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP868]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1400:%.*]]
+// SIMD-ONLY0:       if.else1399:
+// SIMD-ONLY0-NEXT:    [[TMP869:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP869]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1400]]
+// SIMD-ONLY0:       if.end1400:
+// SIMD-ONLY0-NEXT:    [[TMP870:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1401:%.*]] = zext i8 [[TMP870]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP871:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1402:%.*]] = zext i8 [[TMP871]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1403:%.*]] = icmp eq i32 [[CONV1401]], [[CONV1402]]
+// SIMD-ONLY0-NEXT:    [[CONV1404:%.*]] = zext i1 [[CMP1403]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1405:%.*]] = trunc i32 [[CONV1404]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1405]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP872:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL1406:%.*]] = icmp ne i8 [[TMP872]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1406]], label [[IF_THEN1407:%.*]], label [[IF_END1408:%.*]]
+// SIMD-ONLY0:       if.then1407:
+// SIMD-ONLY0-NEXT:    [[TMP873:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP873]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1408]]
+// SIMD-ONLY0:       if.end1408:
+// SIMD-ONLY0-NEXT:    [[TMP874:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1409:%.*]] = zext i8 [[TMP874]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP875:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1410:%.*]] = zext i8 [[TMP875]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1411:%.*]] = icmp eq i32 [[CONV1409]], [[CONV1410]]
+// SIMD-ONLY0-NEXT:    [[CONV1412:%.*]] = zext i1 [[CMP1411]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1413:%.*]] = trunc i32 [[CONV1412]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1413]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP876:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL1414:%.*]] = icmp ne i8 [[TMP876]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1414]], label [[IF_THEN1415:%.*]], label [[IF_END1416:%.*]]
+// SIMD-ONLY0:       if.then1415:
+// SIMD-ONLY0-NEXT:    [[TMP877:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP877]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1416]]
+// SIMD-ONLY0:       if.end1416:
+// SIMD-ONLY0-NEXT:    [[TMP878:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1417:%.*]] = zext i8 [[TMP878]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP879:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1418:%.*]] = zext i8 [[TMP879]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1419:%.*]] = icmp eq i32 [[CONV1417]], [[CONV1418]]
+// SIMD-ONLY0-NEXT:    [[CONV1420:%.*]] = zext i1 [[CMP1419]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1421:%.*]] = trunc i32 [[CONV1420]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1421]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP880:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL1422:%.*]] = icmp ne i8 [[TMP880]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1422]], label [[IF_THEN1423:%.*]], label [[IF_ELSE1424:%.*]]
+// SIMD-ONLY0:       if.then1423:
+// SIMD-ONLY0-NEXT:    [[TMP881:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP881]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1425:%.*]]
+// SIMD-ONLY0:       if.else1424:
+// SIMD-ONLY0-NEXT:    [[TMP882:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP882]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1425]]
+// SIMD-ONLY0:       if.end1425:
+// SIMD-ONLY0-NEXT:    [[TMP883:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1426:%.*]] = zext i8 [[TMP883]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP884:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1427:%.*]] = zext i8 [[TMP884]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1428:%.*]] = icmp eq i32 [[CONV1426]], [[CONV1427]]
+// SIMD-ONLY0-NEXT:    [[CONV1429:%.*]] = zext i1 [[CMP1428]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1430:%.*]] = trunc i32 [[CONV1429]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV1430]], ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP885:%.*]] = load i8, ptr [[UCR]], align 1
+// SIMD-ONLY0-NEXT:    [[TOBOOL1431:%.*]] = icmp ne i8 [[TMP885]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1431]], label [[IF_THEN1432:%.*]], label [[IF_ELSE1433:%.*]]
+// SIMD-ONLY0:       if.then1432:
+// SIMD-ONLY0-NEXT:    [[TMP886:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP886]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1434:%.*]]
+// SIMD-ONLY0:       if.else1433:
+// SIMD-ONLY0-NEXT:    [[TMP887:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP887]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    br label [[IF_END1434]]
+// SIMD-ONLY0:       if.end1434:
+// SIMD-ONLY0-NEXT:    [[TMP888:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP888]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP889:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1435:%.*]] = sext i16 [[TMP889]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP890:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1436:%.*]] = sext i16 [[TMP890]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1437:%.*]] = icmp sgt i32 [[CONV1435]], [[CONV1436]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1437]], label [[IF_THEN1439:%.*]], label [[IF_END1440:%.*]]
+// SIMD-ONLY0:       if.then1439:
+// SIMD-ONLY0-NEXT:    [[TMP891:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP891]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1440]]
+// SIMD-ONLY0:       if.end1440:
+// SIMD-ONLY0-NEXT:    [[TMP892:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP892]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP893:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1441:%.*]] = sext i16 [[TMP893]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP894:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1442:%.*]] = sext i16 [[TMP894]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1443:%.*]] = icmp sgt i32 [[CONV1441]], [[CONV1442]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1443]], label [[IF_THEN1445:%.*]], label [[IF_END1446:%.*]]
+// SIMD-ONLY0:       if.then1445:
+// SIMD-ONLY0-NEXT:    [[TMP895:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP895]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1446]]
+// SIMD-ONLY0:       if.end1446:
+// SIMD-ONLY0-NEXT:    [[TMP896:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP896]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP897:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1447:%.*]] = sext i16 [[TMP897]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP898:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1448:%.*]] = sext i16 [[TMP898]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1449:%.*]] = icmp slt i32 [[CONV1447]], [[CONV1448]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1449]], label [[IF_THEN1451:%.*]], label [[IF_END1452:%.*]]
+// SIMD-ONLY0:       if.then1451:
+// SIMD-ONLY0-NEXT:    [[TMP899:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP899]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1452]]
+// SIMD-ONLY0:       if.end1452:
+// SIMD-ONLY0-NEXT:    [[TMP900:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP900]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP901:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1453:%.*]] = sext i16 [[TMP901]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP902:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1454:%.*]] = sext i16 [[TMP902]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1455:%.*]] = icmp slt i32 [[CONV1453]], [[CONV1454]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1455]], label [[IF_THEN1457:%.*]], label [[IF_END1458:%.*]]
+// SIMD-ONLY0:       if.then1457:
+// SIMD-ONLY0-NEXT:    [[TMP903:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP903]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1458]]
+// SIMD-ONLY0:       if.end1458:
+// SIMD-ONLY0-NEXT:    [[TMP904:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP904]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP905:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1459:%.*]] = sext i16 [[TMP905]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP906:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1460:%.*]] = sext i16 [[TMP906]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1461:%.*]] = icmp eq i32 [[CONV1459]], [[CONV1460]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1461]], label [[IF_THEN1463:%.*]], label [[IF_END1464:%.*]]
+// SIMD-ONLY0:       if.then1463:
+// SIMD-ONLY0-NEXT:    [[TMP907:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP907]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1464]]
+// SIMD-ONLY0:       if.end1464:
+// SIMD-ONLY0-NEXT:    [[TMP908:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP908]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP909:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1465:%.*]] = sext i16 [[TMP909]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP910:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1466:%.*]] = sext i16 [[TMP910]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1467:%.*]] = icmp eq i32 [[CONV1465]], [[CONV1466]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1467]], label [[IF_THEN1469:%.*]], label [[IF_END1470:%.*]]
+// SIMD-ONLY0:       if.then1469:
+// SIMD-ONLY0-NEXT:    [[TMP911:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP911]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1470]]
+// SIMD-ONLY0:       if.end1470:
+// SIMD-ONLY0-NEXT:    [[TMP912:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1471:%.*]] = sext i16 [[TMP912]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP913:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1472:%.*]] = sext i16 [[TMP913]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1473:%.*]] = icmp sgt i32 [[CONV1471]], [[CONV1472]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1473]], label [[IF_THEN1475:%.*]], label [[IF_END1476:%.*]]
+// SIMD-ONLY0:       if.then1475:
+// SIMD-ONLY0-NEXT:    [[TMP914:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP914]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1476]]
+// SIMD-ONLY0:       if.end1476:
+// SIMD-ONLY0-NEXT:    [[TMP915:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP915]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP916:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1477:%.*]] = sext i16 [[TMP916]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP917:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1478:%.*]] = sext i16 [[TMP917]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1479:%.*]] = icmp sgt i32 [[CONV1477]], [[CONV1478]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1479]], label [[IF_THEN1481:%.*]], label [[IF_END1482:%.*]]
+// SIMD-ONLY0:       if.then1481:
+// SIMD-ONLY0-NEXT:    [[TMP918:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP918]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1482]]
+// SIMD-ONLY0:       if.end1482:
+// SIMD-ONLY0-NEXT:    [[TMP919:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP919]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP920:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1483:%.*]] = sext i16 [[TMP920]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP921:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1484:%.*]] = sext i16 [[TMP921]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1485:%.*]] = icmp slt i32 [[CONV1483]], [[CONV1484]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1485]], label [[IF_THEN1487:%.*]], label [[IF_END1488:%.*]]
+// SIMD-ONLY0:       if.then1487:
+// SIMD-ONLY0-NEXT:    [[TMP922:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP922]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1488]]
+// SIMD-ONLY0:       if.end1488:
+// SIMD-ONLY0-NEXT:    [[TMP923:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP923]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP924:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1489:%.*]] = sext i16 [[TMP924]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP925:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1490:%.*]] = sext i16 [[TMP925]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1491:%.*]] = icmp slt i32 [[CONV1489]], [[CONV1490]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1491]], label [[IF_THEN1493:%.*]], label [[IF_END1494:%.*]]
+// SIMD-ONLY0:       if.then1493:
+// SIMD-ONLY0-NEXT:    [[TMP926:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP926]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1494]]
+// SIMD-ONLY0:       if.end1494:
+// SIMD-ONLY0-NEXT:    [[TMP927:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP927]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP928:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1495:%.*]] = sext i16 [[TMP928]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP929:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1496:%.*]] = sext i16 [[TMP929]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1497:%.*]] = icmp eq i32 [[CONV1495]], [[CONV1496]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1497]], label [[IF_THEN1499:%.*]], label [[IF_END1500:%.*]]
+// SIMD-ONLY0:       if.then1499:
+// SIMD-ONLY0-NEXT:    [[TMP930:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP930]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1500]]
+// SIMD-ONLY0:       if.end1500:
+// SIMD-ONLY0-NEXT:    [[TMP931:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP931]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP932:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1501:%.*]] = sext i16 [[TMP932]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP933:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1502:%.*]] = sext i16 [[TMP933]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1503:%.*]] = icmp eq i32 [[CONV1501]], [[CONV1502]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1503]], label [[IF_THEN1505:%.*]], label [[IF_END1506:%.*]]
+// SIMD-ONLY0:       if.then1505:
+// SIMD-ONLY0-NEXT:    [[TMP934:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP934]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1506]]
+// SIMD-ONLY0:       if.end1506:
+// SIMD-ONLY0-NEXT:    [[TMP935:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP935]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP936:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1507:%.*]] = sext i16 [[TMP936]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP937:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1508:%.*]] = sext i16 [[TMP937]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1509:%.*]] = icmp eq i32 [[CONV1507]], [[CONV1508]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1509]], label [[IF_THEN1511:%.*]], label [[IF_ELSE1512:%.*]]
+// SIMD-ONLY0:       if.then1511:
+// SIMD-ONLY0-NEXT:    [[TMP938:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP938]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1513:%.*]]
+// SIMD-ONLY0:       if.else1512:
+// SIMD-ONLY0-NEXT:    [[TMP939:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP939]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1513]]
+// SIMD-ONLY0:       if.end1513:
+// SIMD-ONLY0-NEXT:    [[TMP940:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1514:%.*]] = sext i16 [[TMP940]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP941:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1515:%.*]] = sext i16 [[TMP941]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1516:%.*]] = icmp eq i32 [[CONV1514]], [[CONV1515]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1516]], label [[IF_THEN1518:%.*]], label [[IF_ELSE1519:%.*]]
+// SIMD-ONLY0:       if.then1518:
+// SIMD-ONLY0-NEXT:    [[TMP942:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP942]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1520:%.*]]
+// SIMD-ONLY0:       if.else1519:
+// SIMD-ONLY0-NEXT:    [[TMP943:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP943]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1520]]
+// SIMD-ONLY0:       if.end1520:
+// SIMD-ONLY0-NEXT:    [[TMP944:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1521:%.*]] = sext i16 [[TMP944]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP945:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1522:%.*]] = sext i16 [[TMP945]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1523:%.*]] = icmp eq i32 [[CONV1521]], [[CONV1522]]
+// SIMD-ONLY0-NEXT:    [[CONV1524:%.*]] = zext i1 [[CMP1523]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1525:%.*]] = trunc i32 [[CONV1524]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1525]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP946:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL1526:%.*]] = icmp ne i16 [[TMP946]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1526]], label [[IF_THEN1527:%.*]], label [[IF_END1528:%.*]]
+// SIMD-ONLY0:       if.then1527:
+// SIMD-ONLY0-NEXT:    [[TMP947:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP947]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1528]]
+// SIMD-ONLY0:       if.end1528:
+// SIMD-ONLY0-NEXT:    [[TMP948:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1529:%.*]] = sext i16 [[TMP948]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP949:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1530:%.*]] = sext i16 [[TMP949]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1531:%.*]] = icmp eq i32 [[CONV1529]], [[CONV1530]]
+// SIMD-ONLY0-NEXT:    [[CONV1532:%.*]] = zext i1 [[CMP1531]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1533:%.*]] = trunc i32 [[CONV1532]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1533]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP950:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL1534:%.*]] = icmp ne i16 [[TMP950]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1534]], label [[IF_THEN1535:%.*]], label [[IF_END1536:%.*]]
+// SIMD-ONLY0:       if.then1535:
+// SIMD-ONLY0-NEXT:    [[TMP951:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP951]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1536]]
+// SIMD-ONLY0:       if.end1536:
+// SIMD-ONLY0-NEXT:    [[TMP952:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1537:%.*]] = sext i16 [[TMP952]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP953:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1538:%.*]] = sext i16 [[TMP953]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1539:%.*]] = icmp eq i32 [[CONV1537]], [[CONV1538]]
+// SIMD-ONLY0-NEXT:    [[CONV1540:%.*]] = zext i1 [[CMP1539]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1541:%.*]] = trunc i32 [[CONV1540]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1541]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP954:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL1542:%.*]] = icmp ne i16 [[TMP954]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1542]], label [[IF_THEN1543:%.*]], label [[IF_ELSE1544:%.*]]
+// SIMD-ONLY0:       if.then1543:
+// SIMD-ONLY0-NEXT:    [[TMP955:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP955]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1545:%.*]]
+// SIMD-ONLY0:       if.else1544:
+// SIMD-ONLY0-NEXT:    [[TMP956:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP956]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1545]]
+// SIMD-ONLY0:       if.end1545:
+// SIMD-ONLY0-NEXT:    [[TMP957:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1546:%.*]] = sext i16 [[TMP957]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP958:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1547:%.*]] = sext i16 [[TMP958]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1548:%.*]] = icmp eq i32 [[CONV1546]], [[CONV1547]]
+// SIMD-ONLY0-NEXT:    [[CONV1549:%.*]] = zext i1 [[CMP1548]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1550:%.*]] = trunc i32 [[CONV1549]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1550]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP959:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL1551:%.*]] = icmp ne i16 [[TMP959]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1551]], label [[IF_THEN1552:%.*]], label [[IF_ELSE1553:%.*]]
+// SIMD-ONLY0:       if.then1552:
+// SIMD-ONLY0-NEXT:    [[TMP960:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP960]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1554:%.*]]
+// SIMD-ONLY0:       if.else1553:
+// SIMD-ONLY0-NEXT:    [[TMP961:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP961]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1554]]
+// SIMD-ONLY0:       if.end1554:
+// SIMD-ONLY0-NEXT:    [[TMP962:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP962]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP963:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1555:%.*]] = sext i16 [[TMP963]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP964:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1556:%.*]] = sext i16 [[TMP964]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1557:%.*]] = icmp sgt i32 [[CONV1555]], [[CONV1556]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1557]], label [[IF_THEN1559:%.*]], label [[IF_END1560:%.*]]
+// SIMD-ONLY0:       if.then1559:
+// SIMD-ONLY0-NEXT:    [[TMP965:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP965]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1560]]
+// SIMD-ONLY0:       if.end1560:
+// SIMD-ONLY0-NEXT:    [[TMP966:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP966]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP967:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1561:%.*]] = sext i16 [[TMP967]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP968:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1562:%.*]] = sext i16 [[TMP968]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1563:%.*]] = icmp sgt i32 [[CONV1561]], [[CONV1562]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1563]], label [[IF_THEN1565:%.*]], label [[IF_END1566:%.*]]
+// SIMD-ONLY0:       if.then1565:
+// SIMD-ONLY0-NEXT:    [[TMP969:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP969]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1566]]
+// SIMD-ONLY0:       if.end1566:
+// SIMD-ONLY0-NEXT:    [[TMP970:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP970]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP971:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1567:%.*]] = sext i16 [[TMP971]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP972:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1568:%.*]] = sext i16 [[TMP972]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1569:%.*]] = icmp slt i32 [[CONV1567]], [[CONV1568]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1569]], label [[IF_THEN1571:%.*]], label [[IF_END1572:%.*]]
+// SIMD-ONLY0:       if.then1571:
+// SIMD-ONLY0-NEXT:    [[TMP973:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP973]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1572]]
+// SIMD-ONLY0:       if.end1572:
+// SIMD-ONLY0-NEXT:    [[TMP974:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP974]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP975:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1573:%.*]] = sext i16 [[TMP975]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP976:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1574:%.*]] = sext i16 [[TMP976]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1575:%.*]] = icmp slt i32 [[CONV1573]], [[CONV1574]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1575]], label [[IF_THEN1577:%.*]], label [[IF_END1578:%.*]]
+// SIMD-ONLY0:       if.then1577:
+// SIMD-ONLY0-NEXT:    [[TMP977:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP977]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1578]]
+// SIMD-ONLY0:       if.end1578:
+// SIMD-ONLY0-NEXT:    [[TMP978:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP978]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP979:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1579:%.*]] = sext i16 [[TMP979]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP980:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1580:%.*]] = sext i16 [[TMP980]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1581:%.*]] = icmp eq i32 [[CONV1579]], [[CONV1580]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1581]], label [[IF_THEN1583:%.*]], label [[IF_END1584:%.*]]
+// SIMD-ONLY0:       if.then1583:
+// SIMD-ONLY0-NEXT:    [[TMP981:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP981]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1584]]
+// SIMD-ONLY0:       if.end1584:
+// SIMD-ONLY0-NEXT:    [[TMP982:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP982]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP983:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1585:%.*]] = sext i16 [[TMP983]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP984:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1586:%.*]] = sext i16 [[TMP984]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1587:%.*]] = icmp eq i32 [[CONV1585]], [[CONV1586]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1587]], label [[IF_THEN1589:%.*]], label [[IF_END1590:%.*]]
+// SIMD-ONLY0:       if.then1589:
+// SIMD-ONLY0-NEXT:    [[TMP985:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP985]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1590]]
+// SIMD-ONLY0:       if.end1590:
+// SIMD-ONLY0-NEXT:    [[TMP986:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1591:%.*]] = sext i16 [[TMP986]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP987:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1592:%.*]] = sext i16 [[TMP987]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1593:%.*]] = icmp sgt i32 [[CONV1591]], [[CONV1592]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1593]], label [[IF_THEN1595:%.*]], label [[IF_END1596:%.*]]
+// SIMD-ONLY0:       if.then1595:
+// SIMD-ONLY0-NEXT:    [[TMP988:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP988]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1596]]
+// SIMD-ONLY0:       if.end1596:
+// SIMD-ONLY0-NEXT:    [[TMP989:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP989]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP990:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1597:%.*]] = sext i16 [[TMP990]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP991:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1598:%.*]] = sext i16 [[TMP991]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1599:%.*]] = icmp sgt i32 [[CONV1597]], [[CONV1598]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1599]], label [[IF_THEN1601:%.*]], label [[IF_END1602:%.*]]
+// SIMD-ONLY0:       if.then1601:
+// SIMD-ONLY0-NEXT:    [[TMP992:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP992]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1602]]
+// SIMD-ONLY0:       if.end1602:
+// SIMD-ONLY0-NEXT:    [[TMP993:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP993]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP994:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1603:%.*]] = sext i16 [[TMP994]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP995:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1604:%.*]] = sext i16 [[TMP995]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1605:%.*]] = icmp slt i32 [[CONV1603]], [[CONV1604]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1605]], label [[IF_THEN1607:%.*]], label [[IF_END1608:%.*]]
+// SIMD-ONLY0:       if.then1607:
+// SIMD-ONLY0-NEXT:    [[TMP996:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP996]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1608]]
+// SIMD-ONLY0:       if.end1608:
+// SIMD-ONLY0-NEXT:    [[TMP997:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP997]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP998:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1609:%.*]] = sext i16 [[TMP998]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP999:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1610:%.*]] = sext i16 [[TMP999]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1611:%.*]] = icmp slt i32 [[CONV1609]], [[CONV1610]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1611]], label [[IF_THEN1613:%.*]], label [[IF_END1614:%.*]]
+// SIMD-ONLY0:       if.then1613:
+// SIMD-ONLY0-NEXT:    [[TMP1000:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1000]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1614]]
+// SIMD-ONLY0:       if.end1614:
+// SIMD-ONLY0-NEXT:    [[TMP1001:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1001]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1002:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1615:%.*]] = sext i16 [[TMP1002]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1003:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1616:%.*]] = sext i16 [[TMP1003]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1617:%.*]] = icmp eq i32 [[CONV1615]], [[CONV1616]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1617]], label [[IF_THEN1619:%.*]], label [[IF_END1620:%.*]]
+// SIMD-ONLY0:       if.then1619:
+// SIMD-ONLY0-NEXT:    [[TMP1004:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1004]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1620]]
+// SIMD-ONLY0:       if.end1620:
+// SIMD-ONLY0-NEXT:    [[TMP1005:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1005]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1006:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1621:%.*]] = sext i16 [[TMP1006]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1007:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1622:%.*]] = sext i16 [[TMP1007]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1623:%.*]] = icmp eq i32 [[CONV1621]], [[CONV1622]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1623]], label [[IF_THEN1625:%.*]], label [[IF_END1626:%.*]]
+// SIMD-ONLY0:       if.then1625:
+// SIMD-ONLY0-NEXT:    [[TMP1008:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1008]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1626]]
+// SIMD-ONLY0:       if.end1626:
+// SIMD-ONLY0-NEXT:    [[TMP1009:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1009]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1010:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1627:%.*]] = sext i16 [[TMP1010]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1011:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1628:%.*]] = sext i16 [[TMP1011]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1629:%.*]] = icmp eq i32 [[CONV1627]], [[CONV1628]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1629]], label [[IF_THEN1631:%.*]], label [[IF_ELSE1632:%.*]]
+// SIMD-ONLY0:       if.then1631:
+// SIMD-ONLY0-NEXT:    [[TMP1012:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1012]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1633:%.*]]
+// SIMD-ONLY0:       if.else1632:
+// SIMD-ONLY0-NEXT:    [[TMP1013:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1013]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1633]]
+// SIMD-ONLY0:       if.end1633:
+// SIMD-ONLY0-NEXT:    [[TMP1014:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1634:%.*]] = sext i16 [[TMP1014]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1015:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1635:%.*]] = sext i16 [[TMP1015]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1636:%.*]] = icmp eq i32 [[CONV1634]], [[CONV1635]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1636]], label [[IF_THEN1638:%.*]], label [[IF_ELSE1639:%.*]]
+// SIMD-ONLY0:       if.then1638:
+// SIMD-ONLY0-NEXT:    [[TMP1016:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1016]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1640:%.*]]
+// SIMD-ONLY0:       if.else1639:
+// SIMD-ONLY0-NEXT:    [[TMP1017:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1017]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1640]]
+// SIMD-ONLY0:       if.end1640:
+// SIMD-ONLY0-NEXT:    [[TMP1018:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1641:%.*]] = sext i16 [[TMP1018]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1019:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1642:%.*]] = sext i16 [[TMP1019]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1643:%.*]] = icmp eq i32 [[CONV1641]], [[CONV1642]]
+// SIMD-ONLY0-NEXT:    [[CONV1644:%.*]] = zext i1 [[CMP1643]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1645:%.*]] = trunc i32 [[CONV1644]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1645]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1020:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL1646:%.*]] = icmp ne i16 [[TMP1020]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1646]], label [[IF_THEN1647:%.*]], label [[IF_END1648:%.*]]
+// SIMD-ONLY0:       if.then1647:
+// SIMD-ONLY0-NEXT:    [[TMP1021:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1021]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1648]]
+// SIMD-ONLY0:       if.end1648:
+// SIMD-ONLY0-NEXT:    [[TMP1022:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1649:%.*]] = sext i16 [[TMP1022]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1023:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1650:%.*]] = sext i16 [[TMP1023]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1651:%.*]] = icmp eq i32 [[CONV1649]], [[CONV1650]]
+// SIMD-ONLY0-NEXT:    [[CONV1652:%.*]] = zext i1 [[CMP1651]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1653:%.*]] = trunc i32 [[CONV1652]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1653]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1024:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL1654:%.*]] = icmp ne i16 [[TMP1024]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1654]], label [[IF_THEN1655:%.*]], label [[IF_END1656:%.*]]
+// SIMD-ONLY0:       if.then1655:
+// SIMD-ONLY0-NEXT:    [[TMP1025:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1025]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1656]]
+// SIMD-ONLY0:       if.end1656:
+// SIMD-ONLY0-NEXT:    [[TMP1026:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1657:%.*]] = sext i16 [[TMP1026]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1027:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1658:%.*]] = sext i16 [[TMP1027]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1659:%.*]] = icmp eq i32 [[CONV1657]], [[CONV1658]]
+// SIMD-ONLY0-NEXT:    [[CONV1660:%.*]] = zext i1 [[CMP1659]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1661:%.*]] = trunc i32 [[CONV1660]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1661]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1028:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL1662:%.*]] = icmp ne i16 [[TMP1028]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1662]], label [[IF_THEN1663:%.*]], label [[IF_ELSE1664:%.*]]
+// SIMD-ONLY0:       if.then1663:
+// SIMD-ONLY0-NEXT:    [[TMP1029:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1029]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1665:%.*]]
+// SIMD-ONLY0:       if.else1664:
+// SIMD-ONLY0-NEXT:    [[TMP1030:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1030]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1665]]
+// SIMD-ONLY0:       if.end1665:
+// SIMD-ONLY0-NEXT:    [[TMP1031:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1666:%.*]] = sext i16 [[TMP1031]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1032:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1667:%.*]] = sext i16 [[TMP1032]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1668:%.*]] = icmp eq i32 [[CONV1666]], [[CONV1667]]
+// SIMD-ONLY0-NEXT:    [[CONV1669:%.*]] = zext i1 [[CMP1668]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1670:%.*]] = trunc i32 [[CONV1669]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1670]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1033:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL1671:%.*]] = icmp ne i16 [[TMP1033]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1671]], label [[IF_THEN1672:%.*]], label [[IF_ELSE1673:%.*]]
+// SIMD-ONLY0:       if.then1672:
+// SIMD-ONLY0-NEXT:    [[TMP1034:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1034]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1674:%.*]]
+// SIMD-ONLY0:       if.else1673:
+// SIMD-ONLY0-NEXT:    [[TMP1035:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1035]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1674]]
+// SIMD-ONLY0:       if.end1674:
+// SIMD-ONLY0-NEXT:    [[TMP1036:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1036]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1037:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1675:%.*]] = sext i16 [[TMP1037]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1038:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1676:%.*]] = sext i16 [[TMP1038]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1677:%.*]] = icmp sgt i32 [[CONV1675]], [[CONV1676]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1677]], label [[IF_THEN1679:%.*]], label [[IF_END1680:%.*]]
+// SIMD-ONLY0:       if.then1679:
+// SIMD-ONLY0-NEXT:    [[TMP1039:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1039]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1680]]
+// SIMD-ONLY0:       if.end1680:
+// SIMD-ONLY0-NEXT:    [[TMP1040:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1040]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1041:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1681:%.*]] = sext i16 [[TMP1041]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1042:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1682:%.*]] = sext i16 [[TMP1042]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1683:%.*]] = icmp sgt i32 [[CONV1681]], [[CONV1682]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1683]], label [[IF_THEN1685:%.*]], label [[IF_END1686:%.*]]
+// SIMD-ONLY0:       if.then1685:
+// SIMD-ONLY0-NEXT:    [[TMP1043:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1043]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1686]]
+// SIMD-ONLY0:       if.end1686:
+// SIMD-ONLY0-NEXT:    [[TMP1044:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1044]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1045:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1687:%.*]] = sext i16 [[TMP1045]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1046:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1688:%.*]] = sext i16 [[TMP1046]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1689:%.*]] = icmp slt i32 [[CONV1687]], [[CONV1688]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1689]], label [[IF_THEN1691:%.*]], label [[IF_END1692:%.*]]
+// SIMD-ONLY0:       if.then1691:
+// SIMD-ONLY0-NEXT:    [[TMP1047:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1047]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1692]]
+// SIMD-ONLY0:       if.end1692:
+// SIMD-ONLY0-NEXT:    [[TMP1048:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1048]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1049:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1693:%.*]] = sext i16 [[TMP1049]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1050:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1694:%.*]] = sext i16 [[TMP1050]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1695:%.*]] = icmp slt i32 [[CONV1693]], [[CONV1694]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1695]], label [[IF_THEN1697:%.*]], label [[IF_END1698:%.*]]
+// SIMD-ONLY0:       if.then1697:
+// SIMD-ONLY0-NEXT:    [[TMP1051:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1051]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1698]]
+// SIMD-ONLY0:       if.end1698:
+// SIMD-ONLY0-NEXT:    [[TMP1052:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1052]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1053:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1699:%.*]] = sext i16 [[TMP1053]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1054:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1700:%.*]] = sext i16 [[TMP1054]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1701:%.*]] = icmp eq i32 [[CONV1699]], [[CONV1700]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1701]], label [[IF_THEN1703:%.*]], label [[IF_END1704:%.*]]
+// SIMD-ONLY0:       if.then1703:
+// SIMD-ONLY0-NEXT:    [[TMP1055:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1055]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1704]]
+// SIMD-ONLY0:       if.end1704:
+// SIMD-ONLY0-NEXT:    [[TMP1056:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1056]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1057:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1705:%.*]] = sext i16 [[TMP1057]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1058:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1706:%.*]] = sext i16 [[TMP1058]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1707:%.*]] = icmp eq i32 [[CONV1705]], [[CONV1706]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1707]], label [[IF_THEN1709:%.*]], label [[IF_END1710:%.*]]
+// SIMD-ONLY0:       if.then1709:
+// SIMD-ONLY0-NEXT:    [[TMP1059:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1059]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1710]]
+// SIMD-ONLY0:       if.end1710:
+// SIMD-ONLY0-NEXT:    [[TMP1060:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1711:%.*]] = sext i16 [[TMP1060]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1061:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1712:%.*]] = sext i16 [[TMP1061]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1713:%.*]] = icmp sgt i32 [[CONV1711]], [[CONV1712]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1713]], label [[IF_THEN1715:%.*]], label [[IF_END1716:%.*]]
+// SIMD-ONLY0:       if.then1715:
+// SIMD-ONLY0-NEXT:    [[TMP1062:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1062]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1716]]
+// SIMD-ONLY0:       if.end1716:
+// SIMD-ONLY0-NEXT:    [[TMP1063:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1063]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1064:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1717:%.*]] = sext i16 [[TMP1064]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1065:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1718:%.*]] = sext i16 [[TMP1065]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1719:%.*]] = icmp sgt i32 [[CONV1717]], [[CONV1718]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1719]], label [[IF_THEN1721:%.*]], label [[IF_END1722:%.*]]
+// SIMD-ONLY0:       if.then1721:
+// SIMD-ONLY0-NEXT:    [[TMP1066:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1066]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1722]]
+// SIMD-ONLY0:       if.end1722:
+// SIMD-ONLY0-NEXT:    [[TMP1067:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1067]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1068:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1723:%.*]] = sext i16 [[TMP1068]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1069:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1724:%.*]] = sext i16 [[TMP1069]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1725:%.*]] = icmp slt i32 [[CONV1723]], [[CONV1724]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1725]], label [[IF_THEN1727:%.*]], label [[IF_END1728:%.*]]
+// SIMD-ONLY0:       if.then1727:
+// SIMD-ONLY0-NEXT:    [[TMP1070:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1070]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1728]]
+// SIMD-ONLY0:       if.end1728:
+// SIMD-ONLY0-NEXT:    [[TMP1071:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1071]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1072:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1729:%.*]] = sext i16 [[TMP1072]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1073:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1730:%.*]] = sext i16 [[TMP1073]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1731:%.*]] = icmp slt i32 [[CONV1729]], [[CONV1730]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1731]], label [[IF_THEN1733:%.*]], label [[IF_END1734:%.*]]
+// SIMD-ONLY0:       if.then1733:
+// SIMD-ONLY0-NEXT:    [[TMP1074:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1074]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1734]]
+// SIMD-ONLY0:       if.end1734:
+// SIMD-ONLY0-NEXT:    [[TMP1075:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1075]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1076:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1735:%.*]] = sext i16 [[TMP1076]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1077:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1736:%.*]] = sext i16 [[TMP1077]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1737:%.*]] = icmp eq i32 [[CONV1735]], [[CONV1736]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1737]], label [[IF_THEN1739:%.*]], label [[IF_END1740:%.*]]
+// SIMD-ONLY0:       if.then1739:
+// SIMD-ONLY0-NEXT:    [[TMP1078:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1078]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1740]]
+// SIMD-ONLY0:       if.end1740:
+// SIMD-ONLY0-NEXT:    [[TMP1079:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1079]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1080:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1741:%.*]] = sext i16 [[TMP1080]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1081:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1742:%.*]] = sext i16 [[TMP1081]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1743:%.*]] = icmp eq i32 [[CONV1741]], [[CONV1742]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1743]], label [[IF_THEN1745:%.*]], label [[IF_END1746:%.*]]
+// SIMD-ONLY0:       if.then1745:
+// SIMD-ONLY0-NEXT:    [[TMP1082:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1082]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1746]]
+// SIMD-ONLY0:       if.end1746:
+// SIMD-ONLY0-NEXT:    [[TMP1083:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1083]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1084:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1747:%.*]] = sext i16 [[TMP1084]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1085:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1748:%.*]] = sext i16 [[TMP1085]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1749:%.*]] = icmp eq i32 [[CONV1747]], [[CONV1748]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1749]], label [[IF_THEN1751:%.*]], label [[IF_ELSE1752:%.*]]
+// SIMD-ONLY0:       if.then1751:
+// SIMD-ONLY0-NEXT:    [[TMP1086:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1086]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1753:%.*]]
+// SIMD-ONLY0:       if.else1752:
+// SIMD-ONLY0-NEXT:    [[TMP1087:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1087]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1753]]
+// SIMD-ONLY0:       if.end1753:
+// SIMD-ONLY0-NEXT:    [[TMP1088:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1754:%.*]] = sext i16 [[TMP1088]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1089:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1755:%.*]] = sext i16 [[TMP1089]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1756:%.*]] = icmp eq i32 [[CONV1754]], [[CONV1755]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1756]], label [[IF_THEN1758:%.*]], label [[IF_ELSE1759:%.*]]
+// SIMD-ONLY0:       if.then1758:
+// SIMD-ONLY0-NEXT:    [[TMP1090:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1090]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1760:%.*]]
+// SIMD-ONLY0:       if.else1759:
+// SIMD-ONLY0-NEXT:    [[TMP1091:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1091]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1760]]
+// SIMD-ONLY0:       if.end1760:
+// SIMD-ONLY0-NEXT:    [[TMP1092:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1761:%.*]] = sext i16 [[TMP1092]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1093:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1762:%.*]] = sext i16 [[TMP1093]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1763:%.*]] = icmp eq i32 [[CONV1761]], [[CONV1762]]
+// SIMD-ONLY0-NEXT:    [[CONV1764:%.*]] = zext i1 [[CMP1763]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1765:%.*]] = trunc i32 [[CONV1764]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1765]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1094:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL1766:%.*]] = icmp ne i16 [[TMP1094]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1766]], label [[IF_THEN1767:%.*]], label [[IF_END1768:%.*]]
+// SIMD-ONLY0:       if.then1767:
+// SIMD-ONLY0-NEXT:    [[TMP1095:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1095]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1768]]
+// SIMD-ONLY0:       if.end1768:
+// SIMD-ONLY0-NEXT:    [[TMP1096:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1769:%.*]] = sext i16 [[TMP1096]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1097:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1770:%.*]] = sext i16 [[TMP1097]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1771:%.*]] = icmp eq i32 [[CONV1769]], [[CONV1770]]
+// SIMD-ONLY0-NEXT:    [[CONV1772:%.*]] = zext i1 [[CMP1771]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1773:%.*]] = trunc i32 [[CONV1772]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1773]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1098:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL1774:%.*]] = icmp ne i16 [[TMP1098]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1774]], label [[IF_THEN1775:%.*]], label [[IF_END1776:%.*]]
+// SIMD-ONLY0:       if.then1775:
+// SIMD-ONLY0-NEXT:    [[TMP1099:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1099]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1776]]
+// SIMD-ONLY0:       if.end1776:
+// SIMD-ONLY0-NEXT:    [[TMP1100:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1777:%.*]] = sext i16 [[TMP1100]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1101:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1778:%.*]] = sext i16 [[TMP1101]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1779:%.*]] = icmp eq i32 [[CONV1777]], [[CONV1778]]
+// SIMD-ONLY0-NEXT:    [[CONV1780:%.*]] = zext i1 [[CMP1779]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1781:%.*]] = trunc i32 [[CONV1780]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1781]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1102:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL1782:%.*]] = icmp ne i16 [[TMP1102]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1782]], label [[IF_THEN1783:%.*]], label [[IF_ELSE1784:%.*]]
+// SIMD-ONLY0:       if.then1783:
+// SIMD-ONLY0-NEXT:    [[TMP1103:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1103]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1785:%.*]]
+// SIMD-ONLY0:       if.else1784:
+// SIMD-ONLY0-NEXT:    [[TMP1104:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1104]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1785]]
+// SIMD-ONLY0:       if.end1785:
+// SIMD-ONLY0-NEXT:    [[TMP1105:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1786:%.*]] = sext i16 [[TMP1105]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1106:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1787:%.*]] = sext i16 [[TMP1106]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1788:%.*]] = icmp eq i32 [[CONV1786]], [[CONV1787]]
+// SIMD-ONLY0-NEXT:    [[CONV1789:%.*]] = zext i1 [[CMP1788]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1790:%.*]] = trunc i32 [[CONV1789]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1790]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1107:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL1791:%.*]] = icmp ne i16 [[TMP1107]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1791]], label [[IF_THEN1792:%.*]], label [[IF_ELSE1793:%.*]]
+// SIMD-ONLY0:       if.then1792:
+// SIMD-ONLY0-NEXT:    [[TMP1108:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1108]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1794:%.*]]
+// SIMD-ONLY0:       if.else1793:
+// SIMD-ONLY0-NEXT:    [[TMP1109:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1109]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1794]]
+// SIMD-ONLY0:       if.end1794:
+// SIMD-ONLY0-NEXT:    [[TMP1110:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1110]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1111:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1795:%.*]] = sext i16 [[TMP1111]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1112:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1796:%.*]] = sext i16 [[TMP1112]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1797:%.*]] = icmp sgt i32 [[CONV1795]], [[CONV1796]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1797]], label [[IF_THEN1799:%.*]], label [[IF_END1800:%.*]]
+// SIMD-ONLY0:       if.then1799:
+// SIMD-ONLY0-NEXT:    [[TMP1113:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1113]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1800]]
+// SIMD-ONLY0:       if.end1800:
+// SIMD-ONLY0-NEXT:    [[TMP1114:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1114]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1115:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1801:%.*]] = sext i16 [[TMP1115]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1116:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1802:%.*]] = sext i16 [[TMP1116]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1803:%.*]] = icmp sgt i32 [[CONV1801]], [[CONV1802]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1803]], label [[IF_THEN1805:%.*]], label [[IF_END1806:%.*]]
+// SIMD-ONLY0:       if.then1805:
+// SIMD-ONLY0-NEXT:    [[TMP1117:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1117]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1806]]
+// SIMD-ONLY0:       if.end1806:
+// SIMD-ONLY0-NEXT:    [[TMP1118:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1118]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1119:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1807:%.*]] = sext i16 [[TMP1119]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1120:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1808:%.*]] = sext i16 [[TMP1120]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1809:%.*]] = icmp slt i32 [[CONV1807]], [[CONV1808]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1809]], label [[IF_THEN1811:%.*]], label [[IF_END1812:%.*]]
+// SIMD-ONLY0:       if.then1811:
+// SIMD-ONLY0-NEXT:    [[TMP1121:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1121]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1812]]
+// SIMD-ONLY0:       if.end1812:
+// SIMD-ONLY0-NEXT:    [[TMP1122:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1122]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1123:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1813:%.*]] = sext i16 [[TMP1123]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1124:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1814:%.*]] = sext i16 [[TMP1124]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1815:%.*]] = icmp slt i32 [[CONV1813]], [[CONV1814]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1815]], label [[IF_THEN1817:%.*]], label [[IF_END1818:%.*]]
+// SIMD-ONLY0:       if.then1817:
+// SIMD-ONLY0-NEXT:    [[TMP1125:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1125]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1818]]
+// SIMD-ONLY0:       if.end1818:
+// SIMD-ONLY0-NEXT:    [[TMP1126:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1126]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1127:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1819:%.*]] = sext i16 [[TMP1127]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1128:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1820:%.*]] = sext i16 [[TMP1128]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1821:%.*]] = icmp eq i32 [[CONV1819]], [[CONV1820]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1821]], label [[IF_THEN1823:%.*]], label [[IF_END1824:%.*]]
+// SIMD-ONLY0:       if.then1823:
+// SIMD-ONLY0-NEXT:    [[TMP1129:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1129]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1824]]
+// SIMD-ONLY0:       if.end1824:
+// SIMD-ONLY0-NEXT:    [[TMP1130:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1130]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1131:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1825:%.*]] = sext i16 [[TMP1131]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1132:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1826:%.*]] = sext i16 [[TMP1132]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1827:%.*]] = icmp eq i32 [[CONV1825]], [[CONV1826]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1827]], label [[IF_THEN1829:%.*]], label [[IF_END1830:%.*]]
+// SIMD-ONLY0:       if.then1829:
+// SIMD-ONLY0-NEXT:    [[TMP1133:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1133]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1830]]
+// SIMD-ONLY0:       if.end1830:
+// SIMD-ONLY0-NEXT:    [[TMP1134:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1831:%.*]] = sext i16 [[TMP1134]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1135:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1832:%.*]] = sext i16 [[TMP1135]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1833:%.*]] = icmp sgt i32 [[CONV1831]], [[CONV1832]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1833]], label [[IF_THEN1835:%.*]], label [[IF_END1836:%.*]]
+// SIMD-ONLY0:       if.then1835:
+// SIMD-ONLY0-NEXT:    [[TMP1136:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1136]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1836]]
+// SIMD-ONLY0:       if.end1836:
+// SIMD-ONLY0-NEXT:    [[TMP1137:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1137]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1138:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1837:%.*]] = sext i16 [[TMP1138]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1139:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1838:%.*]] = sext i16 [[TMP1139]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1839:%.*]] = icmp sgt i32 [[CONV1837]], [[CONV1838]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1839]], label [[IF_THEN1841:%.*]], label [[IF_END1842:%.*]]
+// SIMD-ONLY0:       if.then1841:
+// SIMD-ONLY0-NEXT:    [[TMP1140:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1140]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1842]]
+// SIMD-ONLY0:       if.end1842:
+// SIMD-ONLY0-NEXT:    [[TMP1141:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1141]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1142:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1843:%.*]] = sext i16 [[TMP1142]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1143:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1844:%.*]] = sext i16 [[TMP1143]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1845:%.*]] = icmp slt i32 [[CONV1843]], [[CONV1844]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1845]], label [[IF_THEN1847:%.*]], label [[IF_END1848:%.*]]
+// SIMD-ONLY0:       if.then1847:
+// SIMD-ONLY0-NEXT:    [[TMP1144:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1144]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1848]]
+// SIMD-ONLY0:       if.end1848:
+// SIMD-ONLY0-NEXT:    [[TMP1145:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1145]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1146:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1849:%.*]] = sext i16 [[TMP1146]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1147:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1850:%.*]] = sext i16 [[TMP1147]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1851:%.*]] = icmp slt i32 [[CONV1849]], [[CONV1850]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1851]], label [[IF_THEN1853:%.*]], label [[IF_END1854:%.*]]
+// SIMD-ONLY0:       if.then1853:
+// SIMD-ONLY0-NEXT:    [[TMP1148:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1148]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1854]]
+// SIMD-ONLY0:       if.end1854:
+// SIMD-ONLY0-NEXT:    [[TMP1149:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1149]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1150:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1855:%.*]] = sext i16 [[TMP1150]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1151:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1856:%.*]] = sext i16 [[TMP1151]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1857:%.*]] = icmp eq i32 [[CONV1855]], [[CONV1856]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1857]], label [[IF_THEN1859:%.*]], label [[IF_END1860:%.*]]
+// SIMD-ONLY0:       if.then1859:
+// SIMD-ONLY0-NEXT:    [[TMP1152:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1152]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1860]]
+// SIMD-ONLY0:       if.end1860:
+// SIMD-ONLY0-NEXT:    [[TMP1153:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1153]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1154:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1861:%.*]] = sext i16 [[TMP1154]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1155:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1862:%.*]] = sext i16 [[TMP1155]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1863:%.*]] = icmp eq i32 [[CONV1861]], [[CONV1862]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1863]], label [[IF_THEN1865:%.*]], label [[IF_END1866:%.*]]
+// SIMD-ONLY0:       if.then1865:
+// SIMD-ONLY0-NEXT:    [[TMP1156:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1156]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1866]]
+// SIMD-ONLY0:       if.end1866:
+// SIMD-ONLY0-NEXT:    [[TMP1157:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1157]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1158:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1867:%.*]] = sext i16 [[TMP1158]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1159:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1868:%.*]] = sext i16 [[TMP1159]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1869:%.*]] = icmp eq i32 [[CONV1867]], [[CONV1868]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1869]], label [[IF_THEN1871:%.*]], label [[IF_ELSE1872:%.*]]
+// SIMD-ONLY0:       if.then1871:
+// SIMD-ONLY0-NEXT:    [[TMP1160:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1160]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1873:%.*]]
+// SIMD-ONLY0:       if.else1872:
+// SIMD-ONLY0-NEXT:    [[TMP1161:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1161]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1873]]
+// SIMD-ONLY0:       if.end1873:
+// SIMD-ONLY0-NEXT:    [[TMP1162:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1874:%.*]] = sext i16 [[TMP1162]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1163:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1875:%.*]] = sext i16 [[TMP1163]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1876:%.*]] = icmp eq i32 [[CONV1874]], [[CONV1875]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1876]], label [[IF_THEN1878:%.*]], label [[IF_ELSE1879:%.*]]
+// SIMD-ONLY0:       if.then1878:
+// SIMD-ONLY0-NEXT:    [[TMP1164:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1164]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1880:%.*]]
+// SIMD-ONLY0:       if.else1879:
+// SIMD-ONLY0-NEXT:    [[TMP1165:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1165]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1880]]
+// SIMD-ONLY0:       if.end1880:
+// SIMD-ONLY0-NEXT:    [[TMP1166:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1881:%.*]] = sext i16 [[TMP1166]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1167:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1882:%.*]] = sext i16 [[TMP1167]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1883:%.*]] = icmp eq i32 [[CONV1881]], [[CONV1882]]
+// SIMD-ONLY0-NEXT:    [[CONV1884:%.*]] = zext i1 [[CMP1883]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1885:%.*]] = trunc i32 [[CONV1884]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1885]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1168:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL1886:%.*]] = icmp ne i16 [[TMP1168]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1886]], label [[IF_THEN1887:%.*]], label [[IF_END1888:%.*]]
+// SIMD-ONLY0:       if.then1887:
+// SIMD-ONLY0-NEXT:    [[TMP1169:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1169]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1888]]
+// SIMD-ONLY0:       if.end1888:
+// SIMD-ONLY0-NEXT:    [[TMP1170:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1889:%.*]] = sext i16 [[TMP1170]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1171:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1890:%.*]] = sext i16 [[TMP1171]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1891:%.*]] = icmp eq i32 [[CONV1889]], [[CONV1890]]
+// SIMD-ONLY0-NEXT:    [[CONV1892:%.*]] = zext i1 [[CMP1891]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1893:%.*]] = trunc i32 [[CONV1892]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1893]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1172:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL1894:%.*]] = icmp ne i16 [[TMP1172]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1894]], label [[IF_THEN1895:%.*]], label [[IF_END1896:%.*]]
+// SIMD-ONLY0:       if.then1895:
+// SIMD-ONLY0-NEXT:    [[TMP1173:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1173]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1896]]
+// SIMD-ONLY0:       if.end1896:
+// SIMD-ONLY0-NEXT:    [[TMP1174:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1897:%.*]] = sext i16 [[TMP1174]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1175:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1898:%.*]] = sext i16 [[TMP1175]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1899:%.*]] = icmp eq i32 [[CONV1897]], [[CONV1898]]
+// SIMD-ONLY0-NEXT:    [[CONV1900:%.*]] = zext i1 [[CMP1899]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1901:%.*]] = trunc i32 [[CONV1900]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1901]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1176:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL1902:%.*]] = icmp ne i16 [[TMP1176]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1902]], label [[IF_THEN1903:%.*]], label [[IF_ELSE1904:%.*]]
+// SIMD-ONLY0:       if.then1903:
+// SIMD-ONLY0-NEXT:    [[TMP1177:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1177]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1905:%.*]]
+// SIMD-ONLY0:       if.else1904:
+// SIMD-ONLY0-NEXT:    [[TMP1178:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1178]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1905]]
+// SIMD-ONLY0:       if.end1905:
+// SIMD-ONLY0-NEXT:    [[TMP1179:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1906:%.*]] = sext i16 [[TMP1179]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1180:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1907:%.*]] = sext i16 [[TMP1180]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1908:%.*]] = icmp eq i32 [[CONV1906]], [[CONV1907]]
+// SIMD-ONLY0-NEXT:    [[CONV1909:%.*]] = zext i1 [[CMP1908]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV1910:%.*]] = trunc i32 [[CONV1909]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV1910]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1181:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL1911:%.*]] = icmp ne i16 [[TMP1181]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL1911]], label [[IF_THEN1912:%.*]], label [[IF_ELSE1913:%.*]]
+// SIMD-ONLY0:       if.then1912:
+// SIMD-ONLY0-NEXT:    [[TMP1182:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1182]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1914:%.*]]
+// SIMD-ONLY0:       if.else1913:
+// SIMD-ONLY0-NEXT:    [[TMP1183:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1183]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1914]]
+// SIMD-ONLY0:       if.end1914:
+// SIMD-ONLY0-NEXT:    [[TMP1184:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1184]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1185:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1915:%.*]] = sext i16 [[TMP1185]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1186:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1916:%.*]] = sext i16 [[TMP1186]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1917:%.*]] = icmp sgt i32 [[CONV1915]], [[CONV1916]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1917]], label [[IF_THEN1919:%.*]], label [[IF_END1920:%.*]]
+// SIMD-ONLY0:       if.then1919:
+// SIMD-ONLY0-NEXT:    [[TMP1187:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1187]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1920]]
+// SIMD-ONLY0:       if.end1920:
+// SIMD-ONLY0-NEXT:    [[TMP1188:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1188]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1189:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1921:%.*]] = sext i16 [[TMP1189]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1190:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1922:%.*]] = sext i16 [[TMP1190]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1923:%.*]] = icmp sgt i32 [[CONV1921]], [[CONV1922]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1923]], label [[IF_THEN1925:%.*]], label [[IF_END1926:%.*]]
+// SIMD-ONLY0:       if.then1925:
+// SIMD-ONLY0-NEXT:    [[TMP1191:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1191]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1926]]
+// SIMD-ONLY0:       if.end1926:
+// SIMD-ONLY0-NEXT:    [[TMP1192:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1192]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1193:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1927:%.*]] = sext i16 [[TMP1193]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1194:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1928:%.*]] = sext i16 [[TMP1194]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1929:%.*]] = icmp slt i32 [[CONV1927]], [[CONV1928]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1929]], label [[IF_THEN1931:%.*]], label [[IF_END1932:%.*]]
+// SIMD-ONLY0:       if.then1931:
+// SIMD-ONLY0-NEXT:    [[TMP1195:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1195]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1932]]
+// SIMD-ONLY0:       if.end1932:
+// SIMD-ONLY0-NEXT:    [[TMP1196:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1196]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1197:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1933:%.*]] = sext i16 [[TMP1197]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1198:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1934:%.*]] = sext i16 [[TMP1198]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1935:%.*]] = icmp slt i32 [[CONV1933]], [[CONV1934]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1935]], label [[IF_THEN1937:%.*]], label [[IF_END1938:%.*]]
+// SIMD-ONLY0:       if.then1937:
+// SIMD-ONLY0-NEXT:    [[TMP1199:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1199]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1938]]
+// SIMD-ONLY0:       if.end1938:
+// SIMD-ONLY0-NEXT:    [[TMP1200:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1200]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1201:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1939:%.*]] = sext i16 [[TMP1201]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1202:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1940:%.*]] = sext i16 [[TMP1202]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1941:%.*]] = icmp eq i32 [[CONV1939]], [[CONV1940]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1941]], label [[IF_THEN1943:%.*]], label [[IF_END1944:%.*]]
+// SIMD-ONLY0:       if.then1943:
+// SIMD-ONLY0-NEXT:    [[TMP1203:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1203]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1944]]
+// SIMD-ONLY0:       if.end1944:
+// SIMD-ONLY0-NEXT:    [[TMP1204:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1204]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1205:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1945:%.*]] = sext i16 [[TMP1205]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1206:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1946:%.*]] = sext i16 [[TMP1206]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1947:%.*]] = icmp eq i32 [[CONV1945]], [[CONV1946]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1947]], label [[IF_THEN1949:%.*]], label [[IF_END1950:%.*]]
+// SIMD-ONLY0:       if.then1949:
+// SIMD-ONLY0-NEXT:    [[TMP1207:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1207]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1950]]
+// SIMD-ONLY0:       if.end1950:
+// SIMD-ONLY0-NEXT:    [[TMP1208:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1951:%.*]] = sext i16 [[TMP1208]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1209:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1952:%.*]] = sext i16 [[TMP1209]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1953:%.*]] = icmp sgt i32 [[CONV1951]], [[CONV1952]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1953]], label [[IF_THEN1955:%.*]], label [[IF_END1956:%.*]]
+// SIMD-ONLY0:       if.then1955:
+// SIMD-ONLY0-NEXT:    [[TMP1210:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1210]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1956]]
+// SIMD-ONLY0:       if.end1956:
+// SIMD-ONLY0-NEXT:    [[TMP1211:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1211]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1212:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1957:%.*]] = sext i16 [[TMP1212]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1213:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1958:%.*]] = sext i16 [[TMP1213]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1959:%.*]] = icmp sgt i32 [[CONV1957]], [[CONV1958]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1959]], label [[IF_THEN1961:%.*]], label [[IF_END1962:%.*]]
+// SIMD-ONLY0:       if.then1961:
+// SIMD-ONLY0-NEXT:    [[TMP1214:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1214]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1962]]
+// SIMD-ONLY0:       if.end1962:
+// SIMD-ONLY0-NEXT:    [[TMP1215:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1215]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1216:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1963:%.*]] = sext i16 [[TMP1216]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1217:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1964:%.*]] = sext i16 [[TMP1217]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1965:%.*]] = icmp slt i32 [[CONV1963]], [[CONV1964]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1965]], label [[IF_THEN1967:%.*]], label [[IF_END1968:%.*]]
+// SIMD-ONLY0:       if.then1967:
+// SIMD-ONLY0-NEXT:    [[TMP1218:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1218]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1968]]
+// SIMD-ONLY0:       if.end1968:
+// SIMD-ONLY0-NEXT:    [[TMP1219:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1219]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1220:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1969:%.*]] = sext i16 [[TMP1220]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1221:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1970:%.*]] = sext i16 [[TMP1221]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1971:%.*]] = icmp slt i32 [[CONV1969]], [[CONV1970]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1971]], label [[IF_THEN1973:%.*]], label [[IF_END1974:%.*]]
+// SIMD-ONLY0:       if.then1973:
+// SIMD-ONLY0-NEXT:    [[TMP1222:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1222]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1974]]
+// SIMD-ONLY0:       if.end1974:
+// SIMD-ONLY0-NEXT:    [[TMP1223:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1223]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1224:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1975:%.*]] = sext i16 [[TMP1224]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1225:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1976:%.*]] = sext i16 [[TMP1225]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1977:%.*]] = icmp eq i32 [[CONV1975]], [[CONV1976]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1977]], label [[IF_THEN1979:%.*]], label [[IF_END1980:%.*]]
+// SIMD-ONLY0:       if.then1979:
+// SIMD-ONLY0-NEXT:    [[TMP1226:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1226]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1980]]
+// SIMD-ONLY0:       if.end1980:
+// SIMD-ONLY0-NEXT:    [[TMP1227:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1227]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1228:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1981:%.*]] = sext i16 [[TMP1228]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1229:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1982:%.*]] = sext i16 [[TMP1229]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1983:%.*]] = icmp eq i32 [[CONV1981]], [[CONV1982]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1983]], label [[IF_THEN1985:%.*]], label [[IF_END1986:%.*]]
+// SIMD-ONLY0:       if.then1985:
+// SIMD-ONLY0-NEXT:    [[TMP1230:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1230]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1986]]
+// SIMD-ONLY0:       if.end1986:
+// SIMD-ONLY0-NEXT:    [[TMP1231:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1231]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1232:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1987:%.*]] = sext i16 [[TMP1232]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1233:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1988:%.*]] = sext i16 [[TMP1233]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1989:%.*]] = icmp eq i32 [[CONV1987]], [[CONV1988]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1989]], label [[IF_THEN1991:%.*]], label [[IF_ELSE1992:%.*]]
+// SIMD-ONLY0:       if.then1991:
+// SIMD-ONLY0-NEXT:    [[TMP1234:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1234]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1993:%.*]]
+// SIMD-ONLY0:       if.else1992:
+// SIMD-ONLY0-NEXT:    [[TMP1235:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1235]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END1993]]
+// SIMD-ONLY0:       if.end1993:
+// SIMD-ONLY0-NEXT:    [[TMP1236:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1994:%.*]] = sext i16 [[TMP1236]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1237:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1995:%.*]] = sext i16 [[TMP1237]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP1996:%.*]] = icmp eq i32 [[CONV1994]], [[CONV1995]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1996]], label [[IF_THEN1998:%.*]], label [[IF_ELSE1999:%.*]]
+// SIMD-ONLY0:       if.then1998:
+// SIMD-ONLY0-NEXT:    [[TMP1238:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1238]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2000:%.*]]
+// SIMD-ONLY0:       if.else1999:
+// SIMD-ONLY0-NEXT:    [[TMP1239:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1239]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2000]]
+// SIMD-ONLY0:       if.end2000:
+// SIMD-ONLY0-NEXT:    [[TMP1240:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2001:%.*]] = sext i16 [[TMP1240]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1241:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2002:%.*]] = sext i16 [[TMP1241]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2003:%.*]] = icmp eq i32 [[CONV2001]], [[CONV2002]]
+// SIMD-ONLY0-NEXT:    [[CONV2004:%.*]] = zext i1 [[CMP2003]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2005:%.*]] = trunc i32 [[CONV2004]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2005]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1242:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2006:%.*]] = icmp ne i16 [[TMP1242]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2006]], label [[IF_THEN2007:%.*]], label [[IF_END2008:%.*]]
+// SIMD-ONLY0:       if.then2007:
+// SIMD-ONLY0-NEXT:    [[TMP1243:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1243]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2008]]
+// SIMD-ONLY0:       if.end2008:
+// SIMD-ONLY0-NEXT:    [[TMP1244:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2009:%.*]] = sext i16 [[TMP1244]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1245:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2010:%.*]] = sext i16 [[TMP1245]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2011:%.*]] = icmp eq i32 [[CONV2009]], [[CONV2010]]
+// SIMD-ONLY0-NEXT:    [[CONV2012:%.*]] = zext i1 [[CMP2011]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2013:%.*]] = trunc i32 [[CONV2012]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2013]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1246:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2014:%.*]] = icmp ne i16 [[TMP1246]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2014]], label [[IF_THEN2015:%.*]], label [[IF_END2016:%.*]]
+// SIMD-ONLY0:       if.then2015:
+// SIMD-ONLY0-NEXT:    [[TMP1247:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1247]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2016]]
+// SIMD-ONLY0:       if.end2016:
+// SIMD-ONLY0-NEXT:    [[TMP1248:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2017:%.*]] = sext i16 [[TMP1248]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1249:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2018:%.*]] = sext i16 [[TMP1249]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2019:%.*]] = icmp eq i32 [[CONV2017]], [[CONV2018]]
+// SIMD-ONLY0-NEXT:    [[CONV2020:%.*]] = zext i1 [[CMP2019]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2021:%.*]] = trunc i32 [[CONV2020]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2021]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1250:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2022:%.*]] = icmp ne i16 [[TMP1250]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2022]], label [[IF_THEN2023:%.*]], label [[IF_ELSE2024:%.*]]
+// SIMD-ONLY0:       if.then2023:
+// SIMD-ONLY0-NEXT:    [[TMP1251:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1251]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2025:%.*]]
+// SIMD-ONLY0:       if.else2024:
+// SIMD-ONLY0-NEXT:    [[TMP1252:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1252]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2025]]
+// SIMD-ONLY0:       if.end2025:
+// SIMD-ONLY0-NEXT:    [[TMP1253:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2026:%.*]] = sext i16 [[TMP1253]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1254:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2027:%.*]] = sext i16 [[TMP1254]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2028:%.*]] = icmp eq i32 [[CONV2026]], [[CONV2027]]
+// SIMD-ONLY0-NEXT:    [[CONV2029:%.*]] = zext i1 [[CMP2028]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2030:%.*]] = trunc i32 [[CONV2029]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2030]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1255:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2031:%.*]] = icmp ne i16 [[TMP1255]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2031]], label [[IF_THEN2032:%.*]], label [[IF_ELSE2033:%.*]]
+// SIMD-ONLY0:       if.then2032:
+// SIMD-ONLY0-NEXT:    [[TMP1256:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1256]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2034:%.*]]
+// SIMD-ONLY0:       if.else2033:
+// SIMD-ONLY0-NEXT:    [[TMP1257:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1257]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2034]]
+// SIMD-ONLY0:       if.end2034:
+// SIMD-ONLY0-NEXT:    [[TMP1258:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1258]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1259:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2035:%.*]] = sext i16 [[TMP1259]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1260:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2036:%.*]] = sext i16 [[TMP1260]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2037:%.*]] = icmp sgt i32 [[CONV2035]], [[CONV2036]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2037]], label [[IF_THEN2039:%.*]], label [[IF_END2040:%.*]]
+// SIMD-ONLY0:       if.then2039:
+// SIMD-ONLY0-NEXT:    [[TMP1261:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1261]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2040]]
+// SIMD-ONLY0:       if.end2040:
+// SIMD-ONLY0-NEXT:    [[TMP1262:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1262]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1263:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2041:%.*]] = sext i16 [[TMP1263]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1264:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2042:%.*]] = sext i16 [[TMP1264]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2043:%.*]] = icmp sgt i32 [[CONV2041]], [[CONV2042]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2043]], label [[IF_THEN2045:%.*]], label [[IF_END2046:%.*]]
+// SIMD-ONLY0:       if.then2045:
+// SIMD-ONLY0-NEXT:    [[TMP1265:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1265]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2046]]
+// SIMD-ONLY0:       if.end2046:
+// SIMD-ONLY0-NEXT:    [[TMP1266:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1266]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1267:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2047:%.*]] = sext i16 [[TMP1267]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1268:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2048:%.*]] = sext i16 [[TMP1268]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2049:%.*]] = icmp slt i32 [[CONV2047]], [[CONV2048]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2049]], label [[IF_THEN2051:%.*]], label [[IF_END2052:%.*]]
+// SIMD-ONLY0:       if.then2051:
+// SIMD-ONLY0-NEXT:    [[TMP1269:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1269]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2052]]
+// SIMD-ONLY0:       if.end2052:
+// SIMD-ONLY0-NEXT:    [[TMP1270:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1270]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1271:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2053:%.*]] = sext i16 [[TMP1271]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1272:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2054:%.*]] = sext i16 [[TMP1272]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2055:%.*]] = icmp slt i32 [[CONV2053]], [[CONV2054]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2055]], label [[IF_THEN2057:%.*]], label [[IF_END2058:%.*]]
+// SIMD-ONLY0:       if.then2057:
+// SIMD-ONLY0-NEXT:    [[TMP1273:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1273]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2058]]
+// SIMD-ONLY0:       if.end2058:
+// SIMD-ONLY0-NEXT:    [[TMP1274:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1274]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1275:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2059:%.*]] = sext i16 [[TMP1275]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1276:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2060:%.*]] = sext i16 [[TMP1276]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2061:%.*]] = icmp eq i32 [[CONV2059]], [[CONV2060]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2061]], label [[IF_THEN2063:%.*]], label [[IF_END2064:%.*]]
+// SIMD-ONLY0:       if.then2063:
+// SIMD-ONLY0-NEXT:    [[TMP1277:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1277]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2064]]
+// SIMD-ONLY0:       if.end2064:
+// SIMD-ONLY0-NEXT:    [[TMP1278:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1278]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1279:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2065:%.*]] = sext i16 [[TMP1279]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1280:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2066:%.*]] = sext i16 [[TMP1280]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2067:%.*]] = icmp eq i32 [[CONV2065]], [[CONV2066]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2067]], label [[IF_THEN2069:%.*]], label [[IF_END2070:%.*]]
+// SIMD-ONLY0:       if.then2069:
+// SIMD-ONLY0-NEXT:    [[TMP1281:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1281]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2070]]
+// SIMD-ONLY0:       if.end2070:
+// SIMD-ONLY0-NEXT:    [[TMP1282:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2071:%.*]] = sext i16 [[TMP1282]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1283:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2072:%.*]] = sext i16 [[TMP1283]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2073:%.*]] = icmp sgt i32 [[CONV2071]], [[CONV2072]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2073]], label [[IF_THEN2075:%.*]], label [[IF_END2076:%.*]]
+// SIMD-ONLY0:       if.then2075:
+// SIMD-ONLY0-NEXT:    [[TMP1284:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1284]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2076]]
+// SIMD-ONLY0:       if.end2076:
+// SIMD-ONLY0-NEXT:    [[TMP1285:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1285]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1286:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2077:%.*]] = sext i16 [[TMP1286]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1287:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2078:%.*]] = sext i16 [[TMP1287]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2079:%.*]] = icmp sgt i32 [[CONV2077]], [[CONV2078]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2079]], label [[IF_THEN2081:%.*]], label [[IF_END2082:%.*]]
+// SIMD-ONLY0:       if.then2081:
+// SIMD-ONLY0-NEXT:    [[TMP1288:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1288]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2082]]
+// SIMD-ONLY0:       if.end2082:
+// SIMD-ONLY0-NEXT:    [[TMP1289:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1289]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1290:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2083:%.*]] = sext i16 [[TMP1290]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1291:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2084:%.*]] = sext i16 [[TMP1291]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2085:%.*]] = icmp slt i32 [[CONV2083]], [[CONV2084]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2085]], label [[IF_THEN2087:%.*]], label [[IF_END2088:%.*]]
+// SIMD-ONLY0:       if.then2087:
+// SIMD-ONLY0-NEXT:    [[TMP1292:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1292]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2088]]
+// SIMD-ONLY0:       if.end2088:
+// SIMD-ONLY0-NEXT:    [[TMP1293:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1293]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1294:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2089:%.*]] = sext i16 [[TMP1294]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1295:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2090:%.*]] = sext i16 [[TMP1295]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2091:%.*]] = icmp slt i32 [[CONV2089]], [[CONV2090]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2091]], label [[IF_THEN2093:%.*]], label [[IF_END2094:%.*]]
+// SIMD-ONLY0:       if.then2093:
+// SIMD-ONLY0-NEXT:    [[TMP1296:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1296]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2094]]
+// SIMD-ONLY0:       if.end2094:
+// SIMD-ONLY0-NEXT:    [[TMP1297:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1297]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1298:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2095:%.*]] = sext i16 [[TMP1298]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1299:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2096:%.*]] = sext i16 [[TMP1299]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2097:%.*]] = icmp eq i32 [[CONV2095]], [[CONV2096]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2097]], label [[IF_THEN2099:%.*]], label [[IF_END2100:%.*]]
+// SIMD-ONLY0:       if.then2099:
+// SIMD-ONLY0-NEXT:    [[TMP1300:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1300]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2100]]
+// SIMD-ONLY0:       if.end2100:
+// SIMD-ONLY0-NEXT:    [[TMP1301:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1301]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1302:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2101:%.*]] = sext i16 [[TMP1302]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1303:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2102:%.*]] = sext i16 [[TMP1303]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2103:%.*]] = icmp eq i32 [[CONV2101]], [[CONV2102]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2103]], label [[IF_THEN2105:%.*]], label [[IF_END2106:%.*]]
+// SIMD-ONLY0:       if.then2105:
+// SIMD-ONLY0-NEXT:    [[TMP1304:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1304]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2106]]
+// SIMD-ONLY0:       if.end2106:
+// SIMD-ONLY0-NEXT:    [[TMP1305:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1305]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1306:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2107:%.*]] = sext i16 [[TMP1306]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1307:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2108:%.*]] = sext i16 [[TMP1307]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2109:%.*]] = icmp eq i32 [[CONV2107]], [[CONV2108]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2109]], label [[IF_THEN2111:%.*]], label [[IF_ELSE2112:%.*]]
+// SIMD-ONLY0:       if.then2111:
+// SIMD-ONLY0-NEXT:    [[TMP1308:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1308]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2113:%.*]]
+// SIMD-ONLY0:       if.else2112:
+// SIMD-ONLY0-NEXT:    [[TMP1309:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1309]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2113]]
+// SIMD-ONLY0:       if.end2113:
+// SIMD-ONLY0-NEXT:    [[TMP1310:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2114:%.*]] = sext i16 [[TMP1310]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1311:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2115:%.*]] = sext i16 [[TMP1311]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2116:%.*]] = icmp eq i32 [[CONV2114]], [[CONV2115]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2116]], label [[IF_THEN2118:%.*]], label [[IF_ELSE2119:%.*]]
+// SIMD-ONLY0:       if.then2118:
+// SIMD-ONLY0-NEXT:    [[TMP1312:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1312]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2120:%.*]]
+// SIMD-ONLY0:       if.else2119:
+// SIMD-ONLY0-NEXT:    [[TMP1313:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1313]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2120]]
+// SIMD-ONLY0:       if.end2120:
+// SIMD-ONLY0-NEXT:    [[TMP1314:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2121:%.*]] = sext i16 [[TMP1314]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1315:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2122:%.*]] = sext i16 [[TMP1315]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2123:%.*]] = icmp eq i32 [[CONV2121]], [[CONV2122]]
+// SIMD-ONLY0-NEXT:    [[CONV2124:%.*]] = zext i1 [[CMP2123]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2125:%.*]] = trunc i32 [[CONV2124]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2125]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1316:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2126:%.*]] = icmp ne i16 [[TMP1316]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2126]], label [[IF_THEN2127:%.*]], label [[IF_END2128:%.*]]
+// SIMD-ONLY0:       if.then2127:
+// SIMD-ONLY0-NEXT:    [[TMP1317:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1317]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2128]]
+// SIMD-ONLY0:       if.end2128:
+// SIMD-ONLY0-NEXT:    [[TMP1318:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2129:%.*]] = sext i16 [[TMP1318]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1319:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2130:%.*]] = sext i16 [[TMP1319]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2131:%.*]] = icmp eq i32 [[CONV2129]], [[CONV2130]]
+// SIMD-ONLY0-NEXT:    [[CONV2132:%.*]] = zext i1 [[CMP2131]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2133:%.*]] = trunc i32 [[CONV2132]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2133]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1320:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2134:%.*]] = icmp ne i16 [[TMP1320]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2134]], label [[IF_THEN2135:%.*]], label [[IF_END2136:%.*]]
+// SIMD-ONLY0:       if.then2135:
+// SIMD-ONLY0-NEXT:    [[TMP1321:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1321]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2136]]
+// SIMD-ONLY0:       if.end2136:
+// SIMD-ONLY0-NEXT:    [[TMP1322:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2137:%.*]] = sext i16 [[TMP1322]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1323:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2138:%.*]] = sext i16 [[TMP1323]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2139:%.*]] = icmp eq i32 [[CONV2137]], [[CONV2138]]
+// SIMD-ONLY0-NEXT:    [[CONV2140:%.*]] = zext i1 [[CMP2139]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2141:%.*]] = trunc i32 [[CONV2140]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2141]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1324:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2142:%.*]] = icmp ne i16 [[TMP1324]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2142]], label [[IF_THEN2143:%.*]], label [[IF_ELSE2144:%.*]]
+// SIMD-ONLY0:       if.then2143:
+// SIMD-ONLY0-NEXT:    [[TMP1325:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1325]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2145:%.*]]
+// SIMD-ONLY0:       if.else2144:
+// SIMD-ONLY0-NEXT:    [[TMP1326:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1326]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2145]]
+// SIMD-ONLY0:       if.end2145:
+// SIMD-ONLY0-NEXT:    [[TMP1327:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2146:%.*]] = sext i16 [[TMP1327]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1328:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2147:%.*]] = sext i16 [[TMP1328]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2148:%.*]] = icmp eq i32 [[CONV2146]], [[CONV2147]]
+// SIMD-ONLY0-NEXT:    [[CONV2149:%.*]] = zext i1 [[CMP2148]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2150:%.*]] = trunc i32 [[CONV2149]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2150]], ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1329:%.*]] = load i16, ptr [[SR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2151:%.*]] = icmp ne i16 [[TMP1329]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2151]], label [[IF_THEN2152:%.*]], label [[IF_ELSE2153:%.*]]
+// SIMD-ONLY0:       if.then2152:
+// SIMD-ONLY0-NEXT:    [[TMP1330:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1330]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2154:%.*]]
+// SIMD-ONLY0:       if.else2153:
+// SIMD-ONLY0-NEXT:    [[TMP1331:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1331]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2154]]
+// SIMD-ONLY0:       if.end2154:
+// SIMD-ONLY0-NEXT:    [[TMP1332:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1332]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1333:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2155:%.*]] = zext i16 [[TMP1333]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1334:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2156:%.*]] = zext i16 [[TMP1334]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2157:%.*]] = icmp sgt i32 [[CONV2155]], [[CONV2156]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2157]], label [[IF_THEN2159:%.*]], label [[IF_END2160:%.*]]
+// SIMD-ONLY0:       if.then2159:
+// SIMD-ONLY0-NEXT:    [[TMP1335:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1335]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2160]]
+// SIMD-ONLY0:       if.end2160:
+// SIMD-ONLY0-NEXT:    [[TMP1336:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1336]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1337:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2161:%.*]] = zext i16 [[TMP1337]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1338:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2162:%.*]] = zext i16 [[TMP1338]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2163:%.*]] = icmp sgt i32 [[CONV2161]], [[CONV2162]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2163]], label [[IF_THEN2165:%.*]], label [[IF_END2166:%.*]]
+// SIMD-ONLY0:       if.then2165:
+// SIMD-ONLY0-NEXT:    [[TMP1339:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1339]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2166]]
+// SIMD-ONLY0:       if.end2166:
+// SIMD-ONLY0-NEXT:    [[TMP1340:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1340]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1341:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2167:%.*]] = zext i16 [[TMP1341]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1342:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2168:%.*]] = zext i16 [[TMP1342]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2169:%.*]] = icmp slt i32 [[CONV2167]], [[CONV2168]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2169]], label [[IF_THEN2171:%.*]], label [[IF_END2172:%.*]]
+// SIMD-ONLY0:       if.then2171:
+// SIMD-ONLY0-NEXT:    [[TMP1343:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1343]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2172]]
+// SIMD-ONLY0:       if.end2172:
+// SIMD-ONLY0-NEXT:    [[TMP1344:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1344]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1345:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2173:%.*]] = zext i16 [[TMP1345]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1346:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2174:%.*]] = zext i16 [[TMP1346]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2175:%.*]] = icmp slt i32 [[CONV2173]], [[CONV2174]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2175]], label [[IF_THEN2177:%.*]], label [[IF_END2178:%.*]]
+// SIMD-ONLY0:       if.then2177:
+// SIMD-ONLY0-NEXT:    [[TMP1347:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1347]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2178]]
+// SIMD-ONLY0:       if.end2178:
+// SIMD-ONLY0-NEXT:    [[TMP1348:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1348]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1349:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2179:%.*]] = zext i16 [[TMP1349]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1350:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2180:%.*]] = zext i16 [[TMP1350]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2181:%.*]] = icmp eq i32 [[CONV2179]], [[CONV2180]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2181]], label [[IF_THEN2183:%.*]], label [[IF_END2184:%.*]]
+// SIMD-ONLY0:       if.then2183:
+// SIMD-ONLY0-NEXT:    [[TMP1351:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1351]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2184]]
+// SIMD-ONLY0:       if.end2184:
+// SIMD-ONLY0-NEXT:    [[TMP1352:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1352]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1353:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2185:%.*]] = zext i16 [[TMP1353]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1354:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2186:%.*]] = zext i16 [[TMP1354]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2187:%.*]] = icmp eq i32 [[CONV2185]], [[CONV2186]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2187]], label [[IF_THEN2189:%.*]], label [[IF_END2190:%.*]]
+// SIMD-ONLY0:       if.then2189:
+// SIMD-ONLY0-NEXT:    [[TMP1355:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1355]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2190]]
+// SIMD-ONLY0:       if.end2190:
+// SIMD-ONLY0-NEXT:    [[TMP1356:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2191:%.*]] = zext i16 [[TMP1356]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1357:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2192:%.*]] = zext i16 [[TMP1357]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2193:%.*]] = icmp sgt i32 [[CONV2191]], [[CONV2192]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2193]], label [[IF_THEN2195:%.*]], label [[IF_END2196:%.*]]
+// SIMD-ONLY0:       if.then2195:
+// SIMD-ONLY0-NEXT:    [[TMP1358:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1358]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2196]]
+// SIMD-ONLY0:       if.end2196:
+// SIMD-ONLY0-NEXT:    [[TMP1359:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1359]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1360:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2197:%.*]] = zext i16 [[TMP1360]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1361:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2198:%.*]] = zext i16 [[TMP1361]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2199:%.*]] = icmp sgt i32 [[CONV2197]], [[CONV2198]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2199]], label [[IF_THEN2201:%.*]], label [[IF_END2202:%.*]]
+// SIMD-ONLY0:       if.then2201:
+// SIMD-ONLY0-NEXT:    [[TMP1362:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1362]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2202]]
+// SIMD-ONLY0:       if.end2202:
+// SIMD-ONLY0-NEXT:    [[TMP1363:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1363]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1364:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2203:%.*]] = zext i16 [[TMP1364]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1365:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2204:%.*]] = zext i16 [[TMP1365]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2205:%.*]] = icmp slt i32 [[CONV2203]], [[CONV2204]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2205]], label [[IF_THEN2207:%.*]], label [[IF_END2208:%.*]]
+// SIMD-ONLY0:       if.then2207:
+// SIMD-ONLY0-NEXT:    [[TMP1366:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1366]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2208]]
+// SIMD-ONLY0:       if.end2208:
+// SIMD-ONLY0-NEXT:    [[TMP1367:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1367]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1368:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2209:%.*]] = zext i16 [[TMP1368]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1369:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2210:%.*]] = zext i16 [[TMP1369]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2211:%.*]] = icmp slt i32 [[CONV2209]], [[CONV2210]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2211]], label [[IF_THEN2213:%.*]], label [[IF_END2214:%.*]]
+// SIMD-ONLY0:       if.then2213:
+// SIMD-ONLY0-NEXT:    [[TMP1370:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1370]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2214]]
+// SIMD-ONLY0:       if.end2214:
+// SIMD-ONLY0-NEXT:    [[TMP1371:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1371]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1372:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2215:%.*]] = zext i16 [[TMP1372]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1373:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2216:%.*]] = zext i16 [[TMP1373]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2217:%.*]] = icmp eq i32 [[CONV2215]], [[CONV2216]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2217]], label [[IF_THEN2219:%.*]], label [[IF_END2220:%.*]]
+// SIMD-ONLY0:       if.then2219:
+// SIMD-ONLY0-NEXT:    [[TMP1374:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1374]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2220]]
+// SIMD-ONLY0:       if.end2220:
+// SIMD-ONLY0-NEXT:    [[TMP1375:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1375]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1376:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2221:%.*]] = zext i16 [[TMP1376]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1377:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2222:%.*]] = zext i16 [[TMP1377]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2223:%.*]] = icmp eq i32 [[CONV2221]], [[CONV2222]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2223]], label [[IF_THEN2225:%.*]], label [[IF_END2226:%.*]]
+// SIMD-ONLY0:       if.then2225:
+// SIMD-ONLY0-NEXT:    [[TMP1378:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1378]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2226]]
+// SIMD-ONLY0:       if.end2226:
+// SIMD-ONLY0-NEXT:    [[TMP1379:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1379]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1380:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2227:%.*]] = zext i16 [[TMP1380]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1381:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2228:%.*]] = zext i16 [[TMP1381]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2229:%.*]] = icmp eq i32 [[CONV2227]], [[CONV2228]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2229]], label [[IF_THEN2231:%.*]], label [[IF_ELSE2232:%.*]]
+// SIMD-ONLY0:       if.then2231:
+// SIMD-ONLY0-NEXT:    [[TMP1382:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1382]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2233:%.*]]
+// SIMD-ONLY0:       if.else2232:
+// SIMD-ONLY0-NEXT:    [[TMP1383:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1383]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2233]]
+// SIMD-ONLY0:       if.end2233:
+// SIMD-ONLY0-NEXT:    [[TMP1384:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2234:%.*]] = zext i16 [[TMP1384]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1385:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2235:%.*]] = zext i16 [[TMP1385]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2236:%.*]] = icmp eq i32 [[CONV2234]], [[CONV2235]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2236]], label [[IF_THEN2238:%.*]], label [[IF_ELSE2239:%.*]]
+// SIMD-ONLY0:       if.then2238:
+// SIMD-ONLY0-NEXT:    [[TMP1386:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1386]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2240:%.*]]
+// SIMD-ONLY0:       if.else2239:
+// SIMD-ONLY0-NEXT:    [[TMP1387:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1387]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2240]]
+// SIMD-ONLY0:       if.end2240:
+// SIMD-ONLY0-NEXT:    [[TMP1388:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2241:%.*]] = zext i16 [[TMP1388]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1389:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2242:%.*]] = zext i16 [[TMP1389]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2243:%.*]] = icmp eq i32 [[CONV2241]], [[CONV2242]]
+// SIMD-ONLY0-NEXT:    [[CONV2244:%.*]] = zext i1 [[CMP2243]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2245:%.*]] = trunc i32 [[CONV2244]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2245]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1390:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2246:%.*]] = icmp ne i16 [[TMP1390]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2246]], label [[IF_THEN2247:%.*]], label [[IF_END2248:%.*]]
+// SIMD-ONLY0:       if.then2247:
+// SIMD-ONLY0-NEXT:    [[TMP1391:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1391]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2248]]
+// SIMD-ONLY0:       if.end2248:
+// SIMD-ONLY0-NEXT:    [[TMP1392:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2249:%.*]] = zext i16 [[TMP1392]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1393:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2250:%.*]] = zext i16 [[TMP1393]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2251:%.*]] = icmp eq i32 [[CONV2249]], [[CONV2250]]
+// SIMD-ONLY0-NEXT:    [[CONV2252:%.*]] = zext i1 [[CMP2251]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2253:%.*]] = trunc i32 [[CONV2252]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2253]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1394:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2254:%.*]] = icmp ne i16 [[TMP1394]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2254]], label [[IF_THEN2255:%.*]], label [[IF_END2256:%.*]]
+// SIMD-ONLY0:       if.then2255:
+// SIMD-ONLY0-NEXT:    [[TMP1395:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1395]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2256]]
+// SIMD-ONLY0:       if.end2256:
+// SIMD-ONLY0-NEXT:    [[TMP1396:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2257:%.*]] = zext i16 [[TMP1396]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1397:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2258:%.*]] = zext i16 [[TMP1397]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2259:%.*]] = icmp eq i32 [[CONV2257]], [[CONV2258]]
+// SIMD-ONLY0-NEXT:    [[CONV2260:%.*]] = zext i1 [[CMP2259]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2261:%.*]] = trunc i32 [[CONV2260]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2261]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1398:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2262:%.*]] = icmp ne i16 [[TMP1398]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2262]], label [[IF_THEN2263:%.*]], label [[IF_ELSE2264:%.*]]
+// SIMD-ONLY0:       if.then2263:
+// SIMD-ONLY0-NEXT:    [[TMP1399:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1399]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2265:%.*]]
+// SIMD-ONLY0:       if.else2264:
+// SIMD-ONLY0-NEXT:    [[TMP1400:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1400]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2265]]
+// SIMD-ONLY0:       if.end2265:
+// SIMD-ONLY0-NEXT:    [[TMP1401:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2266:%.*]] = zext i16 [[TMP1401]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1402:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2267:%.*]] = zext i16 [[TMP1402]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2268:%.*]] = icmp eq i32 [[CONV2266]], [[CONV2267]]
+// SIMD-ONLY0-NEXT:    [[CONV2269:%.*]] = zext i1 [[CMP2268]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2270:%.*]] = trunc i32 [[CONV2269]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2270]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1403:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2271:%.*]] = icmp ne i16 [[TMP1403]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2271]], label [[IF_THEN2272:%.*]], label [[IF_ELSE2273:%.*]]
+// SIMD-ONLY0:       if.then2272:
+// SIMD-ONLY0-NEXT:    [[TMP1404:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1404]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2274:%.*]]
+// SIMD-ONLY0:       if.else2273:
+// SIMD-ONLY0-NEXT:    [[TMP1405:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1405]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2274]]
+// SIMD-ONLY0:       if.end2274:
+// SIMD-ONLY0-NEXT:    [[TMP1406:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1406]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1407:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2275:%.*]] = zext i16 [[TMP1407]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1408:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2276:%.*]] = zext i16 [[TMP1408]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2277:%.*]] = icmp sgt i32 [[CONV2275]], [[CONV2276]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2277]], label [[IF_THEN2279:%.*]], label [[IF_END2280:%.*]]
+// SIMD-ONLY0:       if.then2279:
+// SIMD-ONLY0-NEXT:    [[TMP1409:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1409]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2280]]
+// SIMD-ONLY0:       if.end2280:
+// SIMD-ONLY0-NEXT:    [[TMP1410:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1410]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1411:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2281:%.*]] = zext i16 [[TMP1411]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1412:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2282:%.*]] = zext i16 [[TMP1412]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2283:%.*]] = icmp sgt i32 [[CONV2281]], [[CONV2282]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2283]], label [[IF_THEN2285:%.*]], label [[IF_END2286:%.*]]
+// SIMD-ONLY0:       if.then2285:
+// SIMD-ONLY0-NEXT:    [[TMP1413:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1413]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2286]]
+// SIMD-ONLY0:       if.end2286:
+// SIMD-ONLY0-NEXT:    [[TMP1414:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1414]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1415:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2287:%.*]] = zext i16 [[TMP1415]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1416:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2288:%.*]] = zext i16 [[TMP1416]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2289:%.*]] = icmp slt i32 [[CONV2287]], [[CONV2288]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2289]], label [[IF_THEN2291:%.*]], label [[IF_END2292:%.*]]
+// SIMD-ONLY0:       if.then2291:
+// SIMD-ONLY0-NEXT:    [[TMP1417:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1417]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2292]]
+// SIMD-ONLY0:       if.end2292:
+// SIMD-ONLY0-NEXT:    [[TMP1418:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1418]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1419:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2293:%.*]] = zext i16 [[TMP1419]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1420:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2294:%.*]] = zext i16 [[TMP1420]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2295:%.*]] = icmp slt i32 [[CONV2293]], [[CONV2294]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2295]], label [[IF_THEN2297:%.*]], label [[IF_END2298:%.*]]
+// SIMD-ONLY0:       if.then2297:
+// SIMD-ONLY0-NEXT:    [[TMP1421:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1421]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2298]]
+// SIMD-ONLY0:       if.end2298:
+// SIMD-ONLY0-NEXT:    [[TMP1422:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1422]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1423:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2299:%.*]] = zext i16 [[TMP1423]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1424:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2300:%.*]] = zext i16 [[TMP1424]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2301:%.*]] = icmp eq i32 [[CONV2299]], [[CONV2300]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2301]], label [[IF_THEN2303:%.*]], label [[IF_END2304:%.*]]
+// SIMD-ONLY0:       if.then2303:
+// SIMD-ONLY0-NEXT:    [[TMP1425:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1425]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2304]]
+// SIMD-ONLY0:       if.end2304:
+// SIMD-ONLY0-NEXT:    [[TMP1426:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1426]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1427:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2305:%.*]] = zext i16 [[TMP1427]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1428:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2306:%.*]] = zext i16 [[TMP1428]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2307:%.*]] = icmp eq i32 [[CONV2305]], [[CONV2306]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2307]], label [[IF_THEN2309:%.*]], label [[IF_END2310:%.*]]
+// SIMD-ONLY0:       if.then2309:
+// SIMD-ONLY0-NEXT:    [[TMP1429:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1429]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2310]]
+// SIMD-ONLY0:       if.end2310:
+// SIMD-ONLY0-NEXT:    [[TMP1430:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2311:%.*]] = zext i16 [[TMP1430]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1431:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2312:%.*]] = zext i16 [[TMP1431]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2313:%.*]] = icmp sgt i32 [[CONV2311]], [[CONV2312]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2313]], label [[IF_THEN2315:%.*]], label [[IF_END2316:%.*]]
+// SIMD-ONLY0:       if.then2315:
+// SIMD-ONLY0-NEXT:    [[TMP1432:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1432]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2316]]
+// SIMD-ONLY0:       if.end2316:
+// SIMD-ONLY0-NEXT:    [[TMP1433:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1433]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1434:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2317:%.*]] = zext i16 [[TMP1434]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1435:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2318:%.*]] = zext i16 [[TMP1435]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2319:%.*]] = icmp sgt i32 [[CONV2317]], [[CONV2318]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2319]], label [[IF_THEN2321:%.*]], label [[IF_END2322:%.*]]
+// SIMD-ONLY0:       if.then2321:
+// SIMD-ONLY0-NEXT:    [[TMP1436:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1436]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2322]]
+// SIMD-ONLY0:       if.end2322:
+// SIMD-ONLY0-NEXT:    [[TMP1437:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1437]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1438:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2323:%.*]] = zext i16 [[TMP1438]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1439:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2324:%.*]] = zext i16 [[TMP1439]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2325:%.*]] = icmp slt i32 [[CONV2323]], [[CONV2324]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2325]], label [[IF_THEN2327:%.*]], label [[IF_END2328:%.*]]
+// SIMD-ONLY0:       if.then2327:
+// SIMD-ONLY0-NEXT:    [[TMP1440:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1440]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2328]]
+// SIMD-ONLY0:       if.end2328:
+// SIMD-ONLY0-NEXT:    [[TMP1441:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1441]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1442:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2329:%.*]] = zext i16 [[TMP1442]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1443:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2330:%.*]] = zext i16 [[TMP1443]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2331:%.*]] = icmp slt i32 [[CONV2329]], [[CONV2330]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2331]], label [[IF_THEN2333:%.*]], label [[IF_END2334:%.*]]
+// SIMD-ONLY0:       if.then2333:
+// SIMD-ONLY0-NEXT:    [[TMP1444:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1444]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2334]]
+// SIMD-ONLY0:       if.end2334:
+// SIMD-ONLY0-NEXT:    [[TMP1445:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1445]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1446:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2335:%.*]] = zext i16 [[TMP1446]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1447:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2336:%.*]] = zext i16 [[TMP1447]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2337:%.*]] = icmp eq i32 [[CONV2335]], [[CONV2336]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2337]], label [[IF_THEN2339:%.*]], label [[IF_END2340:%.*]]
+// SIMD-ONLY0:       if.then2339:
+// SIMD-ONLY0-NEXT:    [[TMP1448:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1448]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2340]]
+// SIMD-ONLY0:       if.end2340:
+// SIMD-ONLY0-NEXT:    [[TMP1449:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1449]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1450:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2341:%.*]] = zext i16 [[TMP1450]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1451:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2342:%.*]] = zext i16 [[TMP1451]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2343:%.*]] = icmp eq i32 [[CONV2341]], [[CONV2342]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2343]], label [[IF_THEN2345:%.*]], label [[IF_END2346:%.*]]
+// SIMD-ONLY0:       if.then2345:
+// SIMD-ONLY0-NEXT:    [[TMP1452:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1452]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2346]]
+// SIMD-ONLY0:       if.end2346:
+// SIMD-ONLY0-NEXT:    [[TMP1453:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1453]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1454:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2347:%.*]] = zext i16 [[TMP1454]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1455:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2348:%.*]] = zext i16 [[TMP1455]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2349:%.*]] = icmp eq i32 [[CONV2347]], [[CONV2348]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2349]], label [[IF_THEN2351:%.*]], label [[IF_ELSE2352:%.*]]
+// SIMD-ONLY0:       if.then2351:
+// SIMD-ONLY0-NEXT:    [[TMP1456:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1456]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2353:%.*]]
+// SIMD-ONLY0:       if.else2352:
+// SIMD-ONLY0-NEXT:    [[TMP1457:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1457]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2353]]
+// SIMD-ONLY0:       if.end2353:
+// SIMD-ONLY0-NEXT:    [[TMP1458:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2354:%.*]] = zext i16 [[TMP1458]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1459:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2355:%.*]] = zext i16 [[TMP1459]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2356:%.*]] = icmp eq i32 [[CONV2354]], [[CONV2355]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2356]], label [[IF_THEN2358:%.*]], label [[IF_ELSE2359:%.*]]
+// SIMD-ONLY0:       if.then2358:
+// SIMD-ONLY0-NEXT:    [[TMP1460:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1460]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2360:%.*]]
+// SIMD-ONLY0:       if.else2359:
+// SIMD-ONLY0-NEXT:    [[TMP1461:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1461]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2360]]
+// SIMD-ONLY0:       if.end2360:
+// SIMD-ONLY0-NEXT:    [[TMP1462:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2361:%.*]] = zext i16 [[TMP1462]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1463:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2362:%.*]] = zext i16 [[TMP1463]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2363:%.*]] = icmp eq i32 [[CONV2361]], [[CONV2362]]
+// SIMD-ONLY0-NEXT:    [[CONV2364:%.*]] = zext i1 [[CMP2363]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2365:%.*]] = trunc i32 [[CONV2364]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2365]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1464:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2366:%.*]] = icmp ne i16 [[TMP1464]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2366]], label [[IF_THEN2367:%.*]], label [[IF_END2368:%.*]]
+// SIMD-ONLY0:       if.then2367:
+// SIMD-ONLY0-NEXT:    [[TMP1465:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1465]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2368]]
+// SIMD-ONLY0:       if.end2368:
+// SIMD-ONLY0-NEXT:    [[TMP1466:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2369:%.*]] = zext i16 [[TMP1466]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1467:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2370:%.*]] = zext i16 [[TMP1467]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2371:%.*]] = icmp eq i32 [[CONV2369]], [[CONV2370]]
+// SIMD-ONLY0-NEXT:    [[CONV2372:%.*]] = zext i1 [[CMP2371]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2373:%.*]] = trunc i32 [[CONV2372]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2373]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1468:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2374:%.*]] = icmp ne i16 [[TMP1468]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2374]], label [[IF_THEN2375:%.*]], label [[IF_END2376:%.*]]
+// SIMD-ONLY0:       if.then2375:
+// SIMD-ONLY0-NEXT:    [[TMP1469:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1469]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2376]]
+// SIMD-ONLY0:       if.end2376:
+// SIMD-ONLY0-NEXT:    [[TMP1470:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2377:%.*]] = zext i16 [[TMP1470]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1471:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2378:%.*]] = zext i16 [[TMP1471]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2379:%.*]] = icmp eq i32 [[CONV2377]], [[CONV2378]]
+// SIMD-ONLY0-NEXT:    [[CONV2380:%.*]] = zext i1 [[CMP2379]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2381:%.*]] = trunc i32 [[CONV2380]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2381]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1472:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2382:%.*]] = icmp ne i16 [[TMP1472]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2382]], label [[IF_THEN2383:%.*]], label [[IF_ELSE2384:%.*]]
+// SIMD-ONLY0:       if.then2383:
+// SIMD-ONLY0-NEXT:    [[TMP1473:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1473]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2385:%.*]]
+// SIMD-ONLY0:       if.else2384:
+// SIMD-ONLY0-NEXT:    [[TMP1474:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1474]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2385]]
+// SIMD-ONLY0:       if.end2385:
+// SIMD-ONLY0-NEXT:    [[TMP1475:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2386:%.*]] = zext i16 [[TMP1475]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1476:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2387:%.*]] = zext i16 [[TMP1476]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2388:%.*]] = icmp eq i32 [[CONV2386]], [[CONV2387]]
+// SIMD-ONLY0-NEXT:    [[CONV2389:%.*]] = zext i1 [[CMP2388]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2390:%.*]] = trunc i32 [[CONV2389]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2390]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1477:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2391:%.*]] = icmp ne i16 [[TMP1477]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2391]], label [[IF_THEN2392:%.*]], label [[IF_ELSE2393:%.*]]
+// SIMD-ONLY0:       if.then2392:
+// SIMD-ONLY0-NEXT:    [[TMP1478:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1478]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2394:%.*]]
+// SIMD-ONLY0:       if.else2393:
+// SIMD-ONLY0-NEXT:    [[TMP1479:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1479]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2394]]
+// SIMD-ONLY0:       if.end2394:
+// SIMD-ONLY0-NEXT:    [[TMP1480:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1480]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1481:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2395:%.*]] = zext i16 [[TMP1481]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1482:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2396:%.*]] = zext i16 [[TMP1482]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2397:%.*]] = icmp sgt i32 [[CONV2395]], [[CONV2396]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2397]], label [[IF_THEN2399:%.*]], label [[IF_END2400:%.*]]
+// SIMD-ONLY0:       if.then2399:
+// SIMD-ONLY0-NEXT:    [[TMP1483:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1483]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2400]]
+// SIMD-ONLY0:       if.end2400:
+// SIMD-ONLY0-NEXT:    [[TMP1484:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1484]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1485:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2401:%.*]] = zext i16 [[TMP1485]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1486:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2402:%.*]] = zext i16 [[TMP1486]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2403:%.*]] = icmp sgt i32 [[CONV2401]], [[CONV2402]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2403]], label [[IF_THEN2405:%.*]], label [[IF_END2406:%.*]]
+// SIMD-ONLY0:       if.then2405:
+// SIMD-ONLY0-NEXT:    [[TMP1487:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1487]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2406]]
+// SIMD-ONLY0:       if.end2406:
+// SIMD-ONLY0-NEXT:    [[TMP1488:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1488]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1489:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2407:%.*]] = zext i16 [[TMP1489]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1490:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2408:%.*]] = zext i16 [[TMP1490]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2409:%.*]] = icmp slt i32 [[CONV2407]], [[CONV2408]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2409]], label [[IF_THEN2411:%.*]], label [[IF_END2412:%.*]]
+// SIMD-ONLY0:       if.then2411:
+// SIMD-ONLY0-NEXT:    [[TMP1491:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1491]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2412]]
+// SIMD-ONLY0:       if.end2412:
+// SIMD-ONLY0-NEXT:    [[TMP1492:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1492]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1493:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2413:%.*]] = zext i16 [[TMP1493]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1494:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2414:%.*]] = zext i16 [[TMP1494]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2415:%.*]] = icmp slt i32 [[CONV2413]], [[CONV2414]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2415]], label [[IF_THEN2417:%.*]], label [[IF_END2418:%.*]]
+// SIMD-ONLY0:       if.then2417:
+// SIMD-ONLY0-NEXT:    [[TMP1495:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1495]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2418]]
+// SIMD-ONLY0:       if.end2418:
+// SIMD-ONLY0-NEXT:    [[TMP1496:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1496]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1497:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2419:%.*]] = zext i16 [[TMP1497]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1498:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2420:%.*]] = zext i16 [[TMP1498]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2421:%.*]] = icmp eq i32 [[CONV2419]], [[CONV2420]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2421]], label [[IF_THEN2423:%.*]], label [[IF_END2424:%.*]]
+// SIMD-ONLY0:       if.then2423:
+// SIMD-ONLY0-NEXT:    [[TMP1499:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1499]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2424]]
+// SIMD-ONLY0:       if.end2424:
+// SIMD-ONLY0-NEXT:    [[TMP1500:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1500]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1501:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2425:%.*]] = zext i16 [[TMP1501]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1502:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2426:%.*]] = zext i16 [[TMP1502]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2427:%.*]] = icmp eq i32 [[CONV2425]], [[CONV2426]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2427]], label [[IF_THEN2429:%.*]], label [[IF_END2430:%.*]]
+// SIMD-ONLY0:       if.then2429:
+// SIMD-ONLY0-NEXT:    [[TMP1503:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1503]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2430]]
+// SIMD-ONLY0:       if.end2430:
+// SIMD-ONLY0-NEXT:    [[TMP1504:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2431:%.*]] = zext i16 [[TMP1504]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1505:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2432:%.*]] = zext i16 [[TMP1505]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2433:%.*]] = icmp sgt i32 [[CONV2431]], [[CONV2432]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2433]], label [[IF_THEN2435:%.*]], label [[IF_END2436:%.*]]
+// SIMD-ONLY0:       if.then2435:
+// SIMD-ONLY0-NEXT:    [[TMP1506:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1506]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2436]]
+// SIMD-ONLY0:       if.end2436:
+// SIMD-ONLY0-NEXT:    [[TMP1507:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1507]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1508:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2437:%.*]] = zext i16 [[TMP1508]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1509:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2438:%.*]] = zext i16 [[TMP1509]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2439:%.*]] = icmp sgt i32 [[CONV2437]], [[CONV2438]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2439]], label [[IF_THEN2441:%.*]], label [[IF_END2442:%.*]]
+// SIMD-ONLY0:       if.then2441:
+// SIMD-ONLY0-NEXT:    [[TMP1510:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1510]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2442]]
+// SIMD-ONLY0:       if.end2442:
+// SIMD-ONLY0-NEXT:    [[TMP1511:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1511]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1512:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2443:%.*]] = zext i16 [[TMP1512]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1513:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2444:%.*]] = zext i16 [[TMP1513]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2445:%.*]] = icmp slt i32 [[CONV2443]], [[CONV2444]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2445]], label [[IF_THEN2447:%.*]], label [[IF_END2448:%.*]]
+// SIMD-ONLY0:       if.then2447:
+// SIMD-ONLY0-NEXT:    [[TMP1514:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1514]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2448]]
+// SIMD-ONLY0:       if.end2448:
+// SIMD-ONLY0-NEXT:    [[TMP1515:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1515]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1516:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2449:%.*]] = zext i16 [[TMP1516]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1517:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2450:%.*]] = zext i16 [[TMP1517]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2451:%.*]] = icmp slt i32 [[CONV2449]], [[CONV2450]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2451]], label [[IF_THEN2453:%.*]], label [[IF_END2454:%.*]]
+// SIMD-ONLY0:       if.then2453:
+// SIMD-ONLY0-NEXT:    [[TMP1518:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1518]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2454]]
+// SIMD-ONLY0:       if.end2454:
+// SIMD-ONLY0-NEXT:    [[TMP1519:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1519]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1520:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2455:%.*]] = zext i16 [[TMP1520]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1521:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2456:%.*]] = zext i16 [[TMP1521]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2457:%.*]] = icmp eq i32 [[CONV2455]], [[CONV2456]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2457]], label [[IF_THEN2459:%.*]], label [[IF_END2460:%.*]]
+// SIMD-ONLY0:       if.then2459:
+// SIMD-ONLY0-NEXT:    [[TMP1522:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1522]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2460]]
+// SIMD-ONLY0:       if.end2460:
+// SIMD-ONLY0-NEXT:    [[TMP1523:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1523]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1524:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2461:%.*]] = zext i16 [[TMP1524]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1525:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2462:%.*]] = zext i16 [[TMP1525]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2463:%.*]] = icmp eq i32 [[CONV2461]], [[CONV2462]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2463]], label [[IF_THEN2465:%.*]], label [[IF_END2466:%.*]]
+// SIMD-ONLY0:       if.then2465:
+// SIMD-ONLY0-NEXT:    [[TMP1526:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1526]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2466]]
+// SIMD-ONLY0:       if.end2466:
+// SIMD-ONLY0-NEXT:    [[TMP1527:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1527]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1528:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2467:%.*]] = zext i16 [[TMP1528]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1529:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2468:%.*]] = zext i16 [[TMP1529]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2469:%.*]] = icmp eq i32 [[CONV2467]], [[CONV2468]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2469]], label [[IF_THEN2471:%.*]], label [[IF_ELSE2472:%.*]]
+// SIMD-ONLY0:       if.then2471:
+// SIMD-ONLY0-NEXT:    [[TMP1530:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1530]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2473:%.*]]
+// SIMD-ONLY0:       if.else2472:
+// SIMD-ONLY0-NEXT:    [[TMP1531:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1531]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2473]]
+// SIMD-ONLY0:       if.end2473:
+// SIMD-ONLY0-NEXT:    [[TMP1532:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2474:%.*]] = zext i16 [[TMP1532]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1533:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2475:%.*]] = zext i16 [[TMP1533]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2476:%.*]] = icmp eq i32 [[CONV2474]], [[CONV2475]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2476]], label [[IF_THEN2478:%.*]], label [[IF_ELSE2479:%.*]]
+// SIMD-ONLY0:       if.then2478:
+// SIMD-ONLY0-NEXT:    [[TMP1534:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1534]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2480:%.*]]
+// SIMD-ONLY0:       if.else2479:
+// SIMD-ONLY0-NEXT:    [[TMP1535:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1535]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2480]]
+// SIMD-ONLY0:       if.end2480:
+// SIMD-ONLY0-NEXT:    [[TMP1536:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2481:%.*]] = zext i16 [[TMP1536]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1537:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2482:%.*]] = zext i16 [[TMP1537]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2483:%.*]] = icmp eq i32 [[CONV2481]], [[CONV2482]]
+// SIMD-ONLY0-NEXT:    [[CONV2484:%.*]] = zext i1 [[CMP2483]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2485:%.*]] = trunc i32 [[CONV2484]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2485]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1538:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2486:%.*]] = icmp ne i16 [[TMP1538]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2486]], label [[IF_THEN2487:%.*]], label [[IF_END2488:%.*]]
+// SIMD-ONLY0:       if.then2487:
+// SIMD-ONLY0-NEXT:    [[TMP1539:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1539]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2488]]
+// SIMD-ONLY0:       if.end2488:
+// SIMD-ONLY0-NEXT:    [[TMP1540:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2489:%.*]] = zext i16 [[TMP1540]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1541:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2490:%.*]] = zext i16 [[TMP1541]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2491:%.*]] = icmp eq i32 [[CONV2489]], [[CONV2490]]
+// SIMD-ONLY0-NEXT:    [[CONV2492:%.*]] = zext i1 [[CMP2491]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2493:%.*]] = trunc i32 [[CONV2492]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2493]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1542:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2494:%.*]] = icmp ne i16 [[TMP1542]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2494]], label [[IF_THEN2495:%.*]], label [[IF_END2496:%.*]]
+// SIMD-ONLY0:       if.then2495:
+// SIMD-ONLY0-NEXT:    [[TMP1543:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1543]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2496]]
+// SIMD-ONLY0:       if.end2496:
+// SIMD-ONLY0-NEXT:    [[TMP1544:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2497:%.*]] = zext i16 [[TMP1544]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1545:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2498:%.*]] = zext i16 [[TMP1545]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2499:%.*]] = icmp eq i32 [[CONV2497]], [[CONV2498]]
+// SIMD-ONLY0-NEXT:    [[CONV2500:%.*]] = zext i1 [[CMP2499]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2501:%.*]] = trunc i32 [[CONV2500]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2501]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1546:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2502:%.*]] = icmp ne i16 [[TMP1546]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2502]], label [[IF_THEN2503:%.*]], label [[IF_ELSE2504:%.*]]
+// SIMD-ONLY0:       if.then2503:
+// SIMD-ONLY0-NEXT:    [[TMP1547:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1547]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2505:%.*]]
+// SIMD-ONLY0:       if.else2504:
+// SIMD-ONLY0-NEXT:    [[TMP1548:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1548]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2505]]
+// SIMD-ONLY0:       if.end2505:
+// SIMD-ONLY0-NEXT:    [[TMP1549:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2506:%.*]] = zext i16 [[TMP1549]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1550:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2507:%.*]] = zext i16 [[TMP1550]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2508:%.*]] = icmp eq i32 [[CONV2506]], [[CONV2507]]
+// SIMD-ONLY0-NEXT:    [[CONV2509:%.*]] = zext i1 [[CMP2508]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2510:%.*]] = trunc i32 [[CONV2509]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2510]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1551:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2511:%.*]] = icmp ne i16 [[TMP1551]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2511]], label [[IF_THEN2512:%.*]], label [[IF_ELSE2513:%.*]]
+// SIMD-ONLY0:       if.then2512:
+// SIMD-ONLY0-NEXT:    [[TMP1552:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1552]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2514:%.*]]
+// SIMD-ONLY0:       if.else2513:
+// SIMD-ONLY0-NEXT:    [[TMP1553:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1553]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2514]]
+// SIMD-ONLY0:       if.end2514:
+// SIMD-ONLY0-NEXT:    [[TMP1554:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1554]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1555:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2515:%.*]] = zext i16 [[TMP1555]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1556:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2516:%.*]] = zext i16 [[TMP1556]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2517:%.*]] = icmp sgt i32 [[CONV2515]], [[CONV2516]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2517]], label [[IF_THEN2519:%.*]], label [[IF_END2520:%.*]]
+// SIMD-ONLY0:       if.then2519:
+// SIMD-ONLY0-NEXT:    [[TMP1557:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1557]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2520]]
+// SIMD-ONLY0:       if.end2520:
+// SIMD-ONLY0-NEXT:    [[TMP1558:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1558]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1559:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2521:%.*]] = zext i16 [[TMP1559]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1560:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2522:%.*]] = zext i16 [[TMP1560]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2523:%.*]] = icmp sgt i32 [[CONV2521]], [[CONV2522]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2523]], label [[IF_THEN2525:%.*]], label [[IF_END2526:%.*]]
+// SIMD-ONLY0:       if.then2525:
+// SIMD-ONLY0-NEXT:    [[TMP1561:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1561]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2526]]
+// SIMD-ONLY0:       if.end2526:
+// SIMD-ONLY0-NEXT:    [[TMP1562:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1562]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1563:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2527:%.*]] = zext i16 [[TMP1563]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1564:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2528:%.*]] = zext i16 [[TMP1564]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2529:%.*]] = icmp slt i32 [[CONV2527]], [[CONV2528]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2529]], label [[IF_THEN2531:%.*]], label [[IF_END2532:%.*]]
+// SIMD-ONLY0:       if.then2531:
+// SIMD-ONLY0-NEXT:    [[TMP1565:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1565]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2532]]
+// SIMD-ONLY0:       if.end2532:
+// SIMD-ONLY0-NEXT:    [[TMP1566:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1566]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1567:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2533:%.*]] = zext i16 [[TMP1567]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1568:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2534:%.*]] = zext i16 [[TMP1568]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2535:%.*]] = icmp slt i32 [[CONV2533]], [[CONV2534]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2535]], label [[IF_THEN2537:%.*]], label [[IF_END2538:%.*]]
+// SIMD-ONLY0:       if.then2537:
+// SIMD-ONLY0-NEXT:    [[TMP1569:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1569]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2538]]
+// SIMD-ONLY0:       if.end2538:
+// SIMD-ONLY0-NEXT:    [[TMP1570:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1570]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1571:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2539:%.*]] = zext i16 [[TMP1571]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1572:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2540:%.*]] = zext i16 [[TMP1572]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2541:%.*]] = icmp eq i32 [[CONV2539]], [[CONV2540]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2541]], label [[IF_THEN2543:%.*]], label [[IF_END2544:%.*]]
+// SIMD-ONLY0:       if.then2543:
+// SIMD-ONLY0-NEXT:    [[TMP1573:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1573]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2544]]
+// SIMD-ONLY0:       if.end2544:
+// SIMD-ONLY0-NEXT:    [[TMP1574:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1574]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1575:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2545:%.*]] = zext i16 [[TMP1575]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1576:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2546:%.*]] = zext i16 [[TMP1576]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2547:%.*]] = icmp eq i32 [[CONV2545]], [[CONV2546]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2547]], label [[IF_THEN2549:%.*]], label [[IF_END2550:%.*]]
+// SIMD-ONLY0:       if.then2549:
+// SIMD-ONLY0-NEXT:    [[TMP1577:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1577]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2550]]
+// SIMD-ONLY0:       if.end2550:
+// SIMD-ONLY0-NEXT:    [[TMP1578:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2551:%.*]] = zext i16 [[TMP1578]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1579:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2552:%.*]] = zext i16 [[TMP1579]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2553:%.*]] = icmp sgt i32 [[CONV2551]], [[CONV2552]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2553]], label [[IF_THEN2555:%.*]], label [[IF_END2556:%.*]]
+// SIMD-ONLY0:       if.then2555:
+// SIMD-ONLY0-NEXT:    [[TMP1580:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1580]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2556]]
+// SIMD-ONLY0:       if.end2556:
+// SIMD-ONLY0-NEXT:    [[TMP1581:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1581]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1582:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2557:%.*]] = zext i16 [[TMP1582]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1583:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2558:%.*]] = zext i16 [[TMP1583]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2559:%.*]] = icmp sgt i32 [[CONV2557]], [[CONV2558]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2559]], label [[IF_THEN2561:%.*]], label [[IF_END2562:%.*]]
+// SIMD-ONLY0:       if.then2561:
+// SIMD-ONLY0-NEXT:    [[TMP1584:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1584]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2562]]
+// SIMD-ONLY0:       if.end2562:
+// SIMD-ONLY0-NEXT:    [[TMP1585:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1585]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1586:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2563:%.*]] = zext i16 [[TMP1586]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1587:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2564:%.*]] = zext i16 [[TMP1587]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2565:%.*]] = icmp slt i32 [[CONV2563]], [[CONV2564]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2565]], label [[IF_THEN2567:%.*]], label [[IF_END2568:%.*]]
+// SIMD-ONLY0:       if.then2567:
+// SIMD-ONLY0-NEXT:    [[TMP1588:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1588]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2568]]
+// SIMD-ONLY0:       if.end2568:
+// SIMD-ONLY0-NEXT:    [[TMP1589:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1589]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1590:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2569:%.*]] = zext i16 [[TMP1590]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1591:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2570:%.*]] = zext i16 [[TMP1591]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2571:%.*]] = icmp slt i32 [[CONV2569]], [[CONV2570]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2571]], label [[IF_THEN2573:%.*]], label [[IF_END2574:%.*]]
+// SIMD-ONLY0:       if.then2573:
+// SIMD-ONLY0-NEXT:    [[TMP1592:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1592]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2574]]
+// SIMD-ONLY0:       if.end2574:
+// SIMD-ONLY0-NEXT:    [[TMP1593:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1593]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1594:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2575:%.*]] = zext i16 [[TMP1594]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1595:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2576:%.*]] = zext i16 [[TMP1595]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2577:%.*]] = icmp eq i32 [[CONV2575]], [[CONV2576]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2577]], label [[IF_THEN2579:%.*]], label [[IF_END2580:%.*]]
+// SIMD-ONLY0:       if.then2579:
+// SIMD-ONLY0-NEXT:    [[TMP1596:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1596]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2580]]
+// SIMD-ONLY0:       if.end2580:
+// SIMD-ONLY0-NEXT:    [[TMP1597:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1597]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1598:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2581:%.*]] = zext i16 [[TMP1598]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1599:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2582:%.*]] = zext i16 [[TMP1599]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2583:%.*]] = icmp eq i32 [[CONV2581]], [[CONV2582]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2583]], label [[IF_THEN2585:%.*]], label [[IF_END2586:%.*]]
+// SIMD-ONLY0:       if.then2585:
+// SIMD-ONLY0-NEXT:    [[TMP1600:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1600]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2586]]
+// SIMD-ONLY0:       if.end2586:
+// SIMD-ONLY0-NEXT:    [[TMP1601:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1601]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1602:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2587:%.*]] = zext i16 [[TMP1602]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1603:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2588:%.*]] = zext i16 [[TMP1603]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2589:%.*]] = icmp eq i32 [[CONV2587]], [[CONV2588]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2589]], label [[IF_THEN2591:%.*]], label [[IF_ELSE2592:%.*]]
+// SIMD-ONLY0:       if.then2591:
+// SIMD-ONLY0-NEXT:    [[TMP1604:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1604]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2593:%.*]]
+// SIMD-ONLY0:       if.else2592:
+// SIMD-ONLY0-NEXT:    [[TMP1605:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1605]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2593]]
+// SIMD-ONLY0:       if.end2593:
+// SIMD-ONLY0-NEXT:    [[TMP1606:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2594:%.*]] = zext i16 [[TMP1606]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1607:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2595:%.*]] = zext i16 [[TMP1607]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2596:%.*]] = icmp eq i32 [[CONV2594]], [[CONV2595]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2596]], label [[IF_THEN2598:%.*]], label [[IF_ELSE2599:%.*]]
+// SIMD-ONLY0:       if.then2598:
+// SIMD-ONLY0-NEXT:    [[TMP1608:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1608]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2600:%.*]]
+// SIMD-ONLY0:       if.else2599:
+// SIMD-ONLY0-NEXT:    [[TMP1609:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1609]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2600]]
+// SIMD-ONLY0:       if.end2600:
+// SIMD-ONLY0-NEXT:    [[TMP1610:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2601:%.*]] = zext i16 [[TMP1610]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1611:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2602:%.*]] = zext i16 [[TMP1611]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2603:%.*]] = icmp eq i32 [[CONV2601]], [[CONV2602]]
+// SIMD-ONLY0-NEXT:    [[CONV2604:%.*]] = zext i1 [[CMP2603]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2605:%.*]] = trunc i32 [[CONV2604]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2605]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1612:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2606:%.*]] = icmp ne i16 [[TMP1612]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2606]], label [[IF_THEN2607:%.*]], label [[IF_END2608:%.*]]
+// SIMD-ONLY0:       if.then2607:
+// SIMD-ONLY0-NEXT:    [[TMP1613:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1613]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2608]]
+// SIMD-ONLY0:       if.end2608:
+// SIMD-ONLY0-NEXT:    [[TMP1614:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2609:%.*]] = zext i16 [[TMP1614]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1615:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2610:%.*]] = zext i16 [[TMP1615]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2611:%.*]] = icmp eq i32 [[CONV2609]], [[CONV2610]]
+// SIMD-ONLY0-NEXT:    [[CONV2612:%.*]] = zext i1 [[CMP2611]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2613:%.*]] = trunc i32 [[CONV2612]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2613]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1616:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2614:%.*]] = icmp ne i16 [[TMP1616]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2614]], label [[IF_THEN2615:%.*]], label [[IF_END2616:%.*]]
+// SIMD-ONLY0:       if.then2615:
+// SIMD-ONLY0-NEXT:    [[TMP1617:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1617]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2616]]
+// SIMD-ONLY0:       if.end2616:
+// SIMD-ONLY0-NEXT:    [[TMP1618:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2617:%.*]] = zext i16 [[TMP1618]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1619:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2618:%.*]] = zext i16 [[TMP1619]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2619:%.*]] = icmp eq i32 [[CONV2617]], [[CONV2618]]
+// SIMD-ONLY0-NEXT:    [[CONV2620:%.*]] = zext i1 [[CMP2619]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2621:%.*]] = trunc i32 [[CONV2620]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2621]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1620:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2622:%.*]] = icmp ne i16 [[TMP1620]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2622]], label [[IF_THEN2623:%.*]], label [[IF_ELSE2624:%.*]]
+// SIMD-ONLY0:       if.then2623:
+// SIMD-ONLY0-NEXT:    [[TMP1621:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1621]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2625:%.*]]
+// SIMD-ONLY0:       if.else2624:
+// SIMD-ONLY0-NEXT:    [[TMP1622:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1622]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2625]]
+// SIMD-ONLY0:       if.end2625:
+// SIMD-ONLY0-NEXT:    [[TMP1623:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2626:%.*]] = zext i16 [[TMP1623]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1624:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2627:%.*]] = zext i16 [[TMP1624]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2628:%.*]] = icmp eq i32 [[CONV2626]], [[CONV2627]]
+// SIMD-ONLY0-NEXT:    [[CONV2629:%.*]] = zext i1 [[CMP2628]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2630:%.*]] = trunc i32 [[CONV2629]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2630]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1625:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2631:%.*]] = icmp ne i16 [[TMP1625]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2631]], label [[IF_THEN2632:%.*]], label [[IF_ELSE2633:%.*]]
+// SIMD-ONLY0:       if.then2632:
+// SIMD-ONLY0-NEXT:    [[TMP1626:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1626]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2634:%.*]]
+// SIMD-ONLY0:       if.else2633:
+// SIMD-ONLY0-NEXT:    [[TMP1627:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1627]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2634]]
+// SIMD-ONLY0:       if.end2634:
+// SIMD-ONLY0-NEXT:    [[TMP1628:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1628]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1629:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2635:%.*]] = zext i16 [[TMP1629]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1630:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2636:%.*]] = zext i16 [[TMP1630]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2637:%.*]] = icmp sgt i32 [[CONV2635]], [[CONV2636]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2637]], label [[IF_THEN2639:%.*]], label [[IF_END2640:%.*]]
+// SIMD-ONLY0:       if.then2639:
+// SIMD-ONLY0-NEXT:    [[TMP1631:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1631]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2640]]
+// SIMD-ONLY0:       if.end2640:
+// SIMD-ONLY0-NEXT:    [[TMP1632:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1632]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1633:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2641:%.*]] = zext i16 [[TMP1633]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1634:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2642:%.*]] = zext i16 [[TMP1634]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2643:%.*]] = icmp sgt i32 [[CONV2641]], [[CONV2642]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2643]], label [[IF_THEN2645:%.*]], label [[IF_END2646:%.*]]
+// SIMD-ONLY0:       if.then2645:
+// SIMD-ONLY0-NEXT:    [[TMP1635:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1635]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2646]]
+// SIMD-ONLY0:       if.end2646:
+// SIMD-ONLY0-NEXT:    [[TMP1636:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1636]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1637:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2647:%.*]] = zext i16 [[TMP1637]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1638:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2648:%.*]] = zext i16 [[TMP1638]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2649:%.*]] = icmp slt i32 [[CONV2647]], [[CONV2648]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2649]], label [[IF_THEN2651:%.*]], label [[IF_END2652:%.*]]
+// SIMD-ONLY0:       if.then2651:
+// SIMD-ONLY0-NEXT:    [[TMP1639:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1639]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2652]]
+// SIMD-ONLY0:       if.end2652:
+// SIMD-ONLY0-NEXT:    [[TMP1640:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1640]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1641:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2653:%.*]] = zext i16 [[TMP1641]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1642:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2654:%.*]] = zext i16 [[TMP1642]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2655:%.*]] = icmp slt i32 [[CONV2653]], [[CONV2654]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2655]], label [[IF_THEN2657:%.*]], label [[IF_END2658:%.*]]
+// SIMD-ONLY0:       if.then2657:
+// SIMD-ONLY0-NEXT:    [[TMP1643:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1643]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2658]]
+// SIMD-ONLY0:       if.end2658:
+// SIMD-ONLY0-NEXT:    [[TMP1644:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1644]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1645:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2659:%.*]] = zext i16 [[TMP1645]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1646:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2660:%.*]] = zext i16 [[TMP1646]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2661:%.*]] = icmp eq i32 [[CONV2659]], [[CONV2660]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2661]], label [[IF_THEN2663:%.*]], label [[IF_END2664:%.*]]
+// SIMD-ONLY0:       if.then2663:
+// SIMD-ONLY0-NEXT:    [[TMP1647:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1647]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2664]]
+// SIMD-ONLY0:       if.end2664:
+// SIMD-ONLY0-NEXT:    [[TMP1648:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1648]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1649:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2665:%.*]] = zext i16 [[TMP1649]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1650:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2666:%.*]] = zext i16 [[TMP1650]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2667:%.*]] = icmp eq i32 [[CONV2665]], [[CONV2666]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2667]], label [[IF_THEN2669:%.*]], label [[IF_END2670:%.*]]
+// SIMD-ONLY0:       if.then2669:
+// SIMD-ONLY0-NEXT:    [[TMP1651:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1651]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2670]]
+// SIMD-ONLY0:       if.end2670:
+// SIMD-ONLY0-NEXT:    [[TMP1652:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2671:%.*]] = zext i16 [[TMP1652]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1653:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2672:%.*]] = zext i16 [[TMP1653]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2673:%.*]] = icmp sgt i32 [[CONV2671]], [[CONV2672]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2673]], label [[IF_THEN2675:%.*]], label [[IF_END2676:%.*]]
+// SIMD-ONLY0:       if.then2675:
+// SIMD-ONLY0-NEXT:    [[TMP1654:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1654]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2676]]
+// SIMD-ONLY0:       if.end2676:
+// SIMD-ONLY0-NEXT:    [[TMP1655:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1655]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1656:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2677:%.*]] = zext i16 [[TMP1656]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1657:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2678:%.*]] = zext i16 [[TMP1657]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2679:%.*]] = icmp sgt i32 [[CONV2677]], [[CONV2678]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2679]], label [[IF_THEN2681:%.*]], label [[IF_END2682:%.*]]
+// SIMD-ONLY0:       if.then2681:
+// SIMD-ONLY0-NEXT:    [[TMP1658:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1658]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2682]]
+// SIMD-ONLY0:       if.end2682:
+// SIMD-ONLY0-NEXT:    [[TMP1659:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1659]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1660:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2683:%.*]] = zext i16 [[TMP1660]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1661:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2684:%.*]] = zext i16 [[TMP1661]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2685:%.*]] = icmp slt i32 [[CONV2683]], [[CONV2684]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2685]], label [[IF_THEN2687:%.*]], label [[IF_END2688:%.*]]
+// SIMD-ONLY0:       if.then2687:
+// SIMD-ONLY0-NEXT:    [[TMP1662:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1662]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2688]]
+// SIMD-ONLY0:       if.end2688:
+// SIMD-ONLY0-NEXT:    [[TMP1663:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1663]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1664:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2689:%.*]] = zext i16 [[TMP1664]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1665:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2690:%.*]] = zext i16 [[TMP1665]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2691:%.*]] = icmp slt i32 [[CONV2689]], [[CONV2690]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2691]], label [[IF_THEN2693:%.*]], label [[IF_END2694:%.*]]
+// SIMD-ONLY0:       if.then2693:
+// SIMD-ONLY0-NEXT:    [[TMP1666:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1666]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2694]]
+// SIMD-ONLY0:       if.end2694:
+// SIMD-ONLY0-NEXT:    [[TMP1667:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1667]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1668:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2695:%.*]] = zext i16 [[TMP1668]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1669:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2696:%.*]] = zext i16 [[TMP1669]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2697:%.*]] = icmp eq i32 [[CONV2695]], [[CONV2696]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2697]], label [[IF_THEN2699:%.*]], label [[IF_END2700:%.*]]
+// SIMD-ONLY0:       if.then2699:
+// SIMD-ONLY0-NEXT:    [[TMP1670:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1670]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2700]]
+// SIMD-ONLY0:       if.end2700:
+// SIMD-ONLY0-NEXT:    [[TMP1671:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1671]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1672:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2701:%.*]] = zext i16 [[TMP1672]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1673:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2702:%.*]] = zext i16 [[TMP1673]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2703:%.*]] = icmp eq i32 [[CONV2701]], [[CONV2702]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2703]], label [[IF_THEN2705:%.*]], label [[IF_END2706:%.*]]
+// SIMD-ONLY0:       if.then2705:
+// SIMD-ONLY0-NEXT:    [[TMP1674:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1674]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2706]]
+// SIMD-ONLY0:       if.end2706:
+// SIMD-ONLY0-NEXT:    [[TMP1675:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1675]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1676:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2707:%.*]] = zext i16 [[TMP1676]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1677:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2708:%.*]] = zext i16 [[TMP1677]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2709:%.*]] = icmp eq i32 [[CONV2707]], [[CONV2708]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2709]], label [[IF_THEN2711:%.*]], label [[IF_ELSE2712:%.*]]
+// SIMD-ONLY0:       if.then2711:
+// SIMD-ONLY0-NEXT:    [[TMP1678:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1678]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2713:%.*]]
+// SIMD-ONLY0:       if.else2712:
+// SIMD-ONLY0-NEXT:    [[TMP1679:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1679]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2713]]
+// SIMD-ONLY0:       if.end2713:
+// SIMD-ONLY0-NEXT:    [[TMP1680:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2714:%.*]] = zext i16 [[TMP1680]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1681:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2715:%.*]] = zext i16 [[TMP1681]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2716:%.*]] = icmp eq i32 [[CONV2714]], [[CONV2715]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2716]], label [[IF_THEN2718:%.*]], label [[IF_ELSE2719:%.*]]
+// SIMD-ONLY0:       if.then2718:
+// SIMD-ONLY0-NEXT:    [[TMP1682:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1682]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2720:%.*]]
+// SIMD-ONLY0:       if.else2719:
+// SIMD-ONLY0-NEXT:    [[TMP1683:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1683]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2720]]
+// SIMD-ONLY0:       if.end2720:
+// SIMD-ONLY0-NEXT:    [[TMP1684:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2721:%.*]] = zext i16 [[TMP1684]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1685:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2722:%.*]] = zext i16 [[TMP1685]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2723:%.*]] = icmp eq i32 [[CONV2721]], [[CONV2722]]
+// SIMD-ONLY0-NEXT:    [[CONV2724:%.*]] = zext i1 [[CMP2723]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2725:%.*]] = trunc i32 [[CONV2724]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2725]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1686:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2726:%.*]] = icmp ne i16 [[TMP1686]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2726]], label [[IF_THEN2727:%.*]], label [[IF_END2728:%.*]]
+// SIMD-ONLY0:       if.then2727:
+// SIMD-ONLY0-NEXT:    [[TMP1687:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1687]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2728]]
+// SIMD-ONLY0:       if.end2728:
+// SIMD-ONLY0-NEXT:    [[TMP1688:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2729:%.*]] = zext i16 [[TMP1688]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1689:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2730:%.*]] = zext i16 [[TMP1689]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2731:%.*]] = icmp eq i32 [[CONV2729]], [[CONV2730]]
+// SIMD-ONLY0-NEXT:    [[CONV2732:%.*]] = zext i1 [[CMP2731]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2733:%.*]] = trunc i32 [[CONV2732]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2733]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1690:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2734:%.*]] = icmp ne i16 [[TMP1690]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2734]], label [[IF_THEN2735:%.*]], label [[IF_END2736:%.*]]
+// SIMD-ONLY0:       if.then2735:
+// SIMD-ONLY0-NEXT:    [[TMP1691:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1691]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2736]]
+// SIMD-ONLY0:       if.end2736:
+// SIMD-ONLY0-NEXT:    [[TMP1692:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2737:%.*]] = zext i16 [[TMP1692]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1693:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2738:%.*]] = zext i16 [[TMP1693]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2739:%.*]] = icmp eq i32 [[CONV2737]], [[CONV2738]]
+// SIMD-ONLY0-NEXT:    [[CONV2740:%.*]] = zext i1 [[CMP2739]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2741:%.*]] = trunc i32 [[CONV2740]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2741]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1694:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2742:%.*]] = icmp ne i16 [[TMP1694]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2742]], label [[IF_THEN2743:%.*]], label [[IF_ELSE2744:%.*]]
+// SIMD-ONLY0:       if.then2743:
+// SIMD-ONLY0-NEXT:    [[TMP1695:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1695]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2745:%.*]]
+// SIMD-ONLY0:       if.else2744:
+// SIMD-ONLY0-NEXT:    [[TMP1696:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1696]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2745]]
+// SIMD-ONLY0:       if.end2745:
+// SIMD-ONLY0-NEXT:    [[TMP1697:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2746:%.*]] = zext i16 [[TMP1697]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1698:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2747:%.*]] = zext i16 [[TMP1698]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2748:%.*]] = icmp eq i32 [[CONV2746]], [[CONV2747]]
+// SIMD-ONLY0-NEXT:    [[CONV2749:%.*]] = zext i1 [[CMP2748]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2750:%.*]] = trunc i32 [[CONV2749]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2750]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1699:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2751:%.*]] = icmp ne i16 [[TMP1699]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2751]], label [[IF_THEN2752:%.*]], label [[IF_ELSE2753:%.*]]
+// SIMD-ONLY0:       if.then2752:
+// SIMD-ONLY0-NEXT:    [[TMP1700:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1700]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2754:%.*]]
+// SIMD-ONLY0:       if.else2753:
+// SIMD-ONLY0-NEXT:    [[TMP1701:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1701]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2754]]
+// SIMD-ONLY0:       if.end2754:
+// SIMD-ONLY0-NEXT:    [[TMP1702:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1702]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1703:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2755:%.*]] = zext i16 [[TMP1703]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1704:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2756:%.*]] = zext i16 [[TMP1704]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2757:%.*]] = icmp sgt i32 [[CONV2755]], [[CONV2756]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2757]], label [[IF_THEN2759:%.*]], label [[IF_END2760:%.*]]
+// SIMD-ONLY0:       if.then2759:
+// SIMD-ONLY0-NEXT:    [[TMP1705:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1705]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2760]]
+// SIMD-ONLY0:       if.end2760:
+// SIMD-ONLY0-NEXT:    [[TMP1706:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1706]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1707:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2761:%.*]] = zext i16 [[TMP1707]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1708:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2762:%.*]] = zext i16 [[TMP1708]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2763:%.*]] = icmp sgt i32 [[CONV2761]], [[CONV2762]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2763]], label [[IF_THEN2765:%.*]], label [[IF_END2766:%.*]]
+// SIMD-ONLY0:       if.then2765:
+// SIMD-ONLY0-NEXT:    [[TMP1709:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1709]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2766]]
+// SIMD-ONLY0:       if.end2766:
+// SIMD-ONLY0-NEXT:    [[TMP1710:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1710]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1711:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2767:%.*]] = zext i16 [[TMP1711]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1712:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2768:%.*]] = zext i16 [[TMP1712]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2769:%.*]] = icmp slt i32 [[CONV2767]], [[CONV2768]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2769]], label [[IF_THEN2771:%.*]], label [[IF_END2772:%.*]]
+// SIMD-ONLY0:       if.then2771:
+// SIMD-ONLY0-NEXT:    [[TMP1713:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1713]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2772]]
+// SIMD-ONLY0:       if.end2772:
+// SIMD-ONLY0-NEXT:    [[TMP1714:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1714]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1715:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2773:%.*]] = zext i16 [[TMP1715]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1716:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2774:%.*]] = zext i16 [[TMP1716]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2775:%.*]] = icmp slt i32 [[CONV2773]], [[CONV2774]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2775]], label [[IF_THEN2777:%.*]], label [[IF_END2778:%.*]]
+// SIMD-ONLY0:       if.then2777:
+// SIMD-ONLY0-NEXT:    [[TMP1717:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1717]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2778]]
+// SIMD-ONLY0:       if.end2778:
+// SIMD-ONLY0-NEXT:    [[TMP1718:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1718]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1719:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2779:%.*]] = zext i16 [[TMP1719]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1720:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2780:%.*]] = zext i16 [[TMP1720]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2781:%.*]] = icmp eq i32 [[CONV2779]], [[CONV2780]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2781]], label [[IF_THEN2783:%.*]], label [[IF_END2784:%.*]]
+// SIMD-ONLY0:       if.then2783:
+// SIMD-ONLY0-NEXT:    [[TMP1721:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1721]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2784]]
+// SIMD-ONLY0:       if.end2784:
+// SIMD-ONLY0-NEXT:    [[TMP1722:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1722]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1723:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2785:%.*]] = zext i16 [[TMP1723]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1724:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2786:%.*]] = zext i16 [[TMP1724]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2787:%.*]] = icmp eq i32 [[CONV2785]], [[CONV2786]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2787]], label [[IF_THEN2789:%.*]], label [[IF_END2790:%.*]]
+// SIMD-ONLY0:       if.then2789:
+// SIMD-ONLY0-NEXT:    [[TMP1725:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1725]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2790]]
+// SIMD-ONLY0:       if.end2790:
+// SIMD-ONLY0-NEXT:    [[TMP1726:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2791:%.*]] = zext i16 [[TMP1726]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1727:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2792:%.*]] = zext i16 [[TMP1727]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2793:%.*]] = icmp sgt i32 [[CONV2791]], [[CONV2792]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2793]], label [[IF_THEN2795:%.*]], label [[IF_END2796:%.*]]
+// SIMD-ONLY0:       if.then2795:
+// SIMD-ONLY0-NEXT:    [[TMP1728:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1728]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2796]]
+// SIMD-ONLY0:       if.end2796:
+// SIMD-ONLY0-NEXT:    [[TMP1729:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1729]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1730:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2797:%.*]] = zext i16 [[TMP1730]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1731:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2798:%.*]] = zext i16 [[TMP1731]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2799:%.*]] = icmp sgt i32 [[CONV2797]], [[CONV2798]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2799]], label [[IF_THEN2801:%.*]], label [[IF_END2802:%.*]]
+// SIMD-ONLY0:       if.then2801:
+// SIMD-ONLY0-NEXT:    [[TMP1732:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1732]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2802]]
+// SIMD-ONLY0:       if.end2802:
+// SIMD-ONLY0-NEXT:    [[TMP1733:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1733]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1734:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2803:%.*]] = zext i16 [[TMP1734]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1735:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2804:%.*]] = zext i16 [[TMP1735]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2805:%.*]] = icmp slt i32 [[CONV2803]], [[CONV2804]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2805]], label [[IF_THEN2807:%.*]], label [[IF_END2808:%.*]]
+// SIMD-ONLY0:       if.then2807:
+// SIMD-ONLY0-NEXT:    [[TMP1736:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1736]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2808]]
+// SIMD-ONLY0:       if.end2808:
+// SIMD-ONLY0-NEXT:    [[TMP1737:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1737]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1738:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2809:%.*]] = zext i16 [[TMP1738]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1739:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2810:%.*]] = zext i16 [[TMP1739]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2811:%.*]] = icmp slt i32 [[CONV2809]], [[CONV2810]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2811]], label [[IF_THEN2813:%.*]], label [[IF_END2814:%.*]]
+// SIMD-ONLY0:       if.then2813:
+// SIMD-ONLY0-NEXT:    [[TMP1740:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1740]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2814]]
+// SIMD-ONLY0:       if.end2814:
+// SIMD-ONLY0-NEXT:    [[TMP1741:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1741]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1742:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2815:%.*]] = zext i16 [[TMP1742]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1743:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2816:%.*]] = zext i16 [[TMP1743]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2817:%.*]] = icmp eq i32 [[CONV2815]], [[CONV2816]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2817]], label [[IF_THEN2819:%.*]], label [[IF_END2820:%.*]]
+// SIMD-ONLY0:       if.then2819:
+// SIMD-ONLY0-NEXT:    [[TMP1744:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1744]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2820]]
+// SIMD-ONLY0:       if.end2820:
+// SIMD-ONLY0-NEXT:    [[TMP1745:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1745]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1746:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2821:%.*]] = zext i16 [[TMP1746]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1747:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2822:%.*]] = zext i16 [[TMP1747]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2823:%.*]] = icmp eq i32 [[CONV2821]], [[CONV2822]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2823]], label [[IF_THEN2825:%.*]], label [[IF_END2826:%.*]]
+// SIMD-ONLY0:       if.then2825:
+// SIMD-ONLY0-NEXT:    [[TMP1748:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1748]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2826]]
+// SIMD-ONLY0:       if.end2826:
+// SIMD-ONLY0-NEXT:    [[TMP1749:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1749]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1750:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2827:%.*]] = zext i16 [[TMP1750]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1751:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2828:%.*]] = zext i16 [[TMP1751]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2829:%.*]] = icmp eq i32 [[CONV2827]], [[CONV2828]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2829]], label [[IF_THEN2831:%.*]], label [[IF_ELSE2832:%.*]]
+// SIMD-ONLY0:       if.then2831:
+// SIMD-ONLY0-NEXT:    [[TMP1752:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1752]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2833:%.*]]
+// SIMD-ONLY0:       if.else2832:
+// SIMD-ONLY0-NEXT:    [[TMP1753:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1753]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2833]]
+// SIMD-ONLY0:       if.end2833:
+// SIMD-ONLY0-NEXT:    [[TMP1754:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2834:%.*]] = zext i16 [[TMP1754]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1755:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2835:%.*]] = zext i16 [[TMP1755]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2836:%.*]] = icmp eq i32 [[CONV2834]], [[CONV2835]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2836]], label [[IF_THEN2838:%.*]], label [[IF_ELSE2839:%.*]]
+// SIMD-ONLY0:       if.then2838:
+// SIMD-ONLY0-NEXT:    [[TMP1756:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1756]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2840:%.*]]
+// SIMD-ONLY0:       if.else2839:
+// SIMD-ONLY0-NEXT:    [[TMP1757:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1757]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2840]]
+// SIMD-ONLY0:       if.end2840:
+// SIMD-ONLY0-NEXT:    [[TMP1758:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2841:%.*]] = zext i16 [[TMP1758]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1759:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2842:%.*]] = zext i16 [[TMP1759]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2843:%.*]] = icmp eq i32 [[CONV2841]], [[CONV2842]]
+// SIMD-ONLY0-NEXT:    [[CONV2844:%.*]] = zext i1 [[CMP2843]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2845:%.*]] = trunc i32 [[CONV2844]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2845]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1760:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2846:%.*]] = icmp ne i16 [[TMP1760]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2846]], label [[IF_THEN2847:%.*]], label [[IF_END2848:%.*]]
+// SIMD-ONLY0:       if.then2847:
+// SIMD-ONLY0-NEXT:    [[TMP1761:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1761]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2848]]
+// SIMD-ONLY0:       if.end2848:
+// SIMD-ONLY0-NEXT:    [[TMP1762:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2849:%.*]] = zext i16 [[TMP1762]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1763:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2850:%.*]] = zext i16 [[TMP1763]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2851:%.*]] = icmp eq i32 [[CONV2849]], [[CONV2850]]
+// SIMD-ONLY0-NEXT:    [[CONV2852:%.*]] = zext i1 [[CMP2851]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2853:%.*]] = trunc i32 [[CONV2852]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2853]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1764:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2854:%.*]] = icmp ne i16 [[TMP1764]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2854]], label [[IF_THEN2855:%.*]], label [[IF_END2856:%.*]]
+// SIMD-ONLY0:       if.then2855:
+// SIMD-ONLY0-NEXT:    [[TMP1765:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1765]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2856]]
+// SIMD-ONLY0:       if.end2856:
+// SIMD-ONLY0-NEXT:    [[TMP1766:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2857:%.*]] = zext i16 [[TMP1766]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1767:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2858:%.*]] = zext i16 [[TMP1767]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2859:%.*]] = icmp eq i32 [[CONV2857]], [[CONV2858]]
+// SIMD-ONLY0-NEXT:    [[CONV2860:%.*]] = zext i1 [[CMP2859]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2861:%.*]] = trunc i32 [[CONV2860]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2861]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1768:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2862:%.*]] = icmp ne i16 [[TMP1768]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2862]], label [[IF_THEN2863:%.*]], label [[IF_ELSE2864:%.*]]
+// SIMD-ONLY0:       if.then2863:
+// SIMD-ONLY0-NEXT:    [[TMP1769:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1769]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2865:%.*]]
+// SIMD-ONLY0:       if.else2864:
+// SIMD-ONLY0-NEXT:    [[TMP1770:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1770]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2865]]
+// SIMD-ONLY0:       if.end2865:
+// SIMD-ONLY0-NEXT:    [[TMP1771:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2866:%.*]] = zext i16 [[TMP1771]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP1772:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV2867:%.*]] = zext i16 [[TMP1772]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP2868:%.*]] = icmp eq i32 [[CONV2866]], [[CONV2867]]
+// SIMD-ONLY0-NEXT:    [[CONV2869:%.*]] = zext i1 [[CMP2868]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV2870:%.*]] = trunc i32 [[CONV2869]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV2870]], ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1773:%.*]] = load i16, ptr [[USR]], align 2
+// SIMD-ONLY0-NEXT:    [[TOBOOL2871:%.*]] = icmp ne i16 [[TMP1773]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2871]], label [[IF_THEN2872:%.*]], label [[IF_ELSE2873:%.*]]
+// SIMD-ONLY0:       if.then2872:
+// SIMD-ONLY0-NEXT:    [[TMP1774:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1774]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2874:%.*]]
+// SIMD-ONLY0:       if.else2873:
+// SIMD-ONLY0-NEXT:    [[TMP1775:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP1775]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    br label [[IF_END2874]]
+// SIMD-ONLY0:       if.end2874:
+// SIMD-ONLY0-NEXT:    [[TMP1776:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1776]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1777:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1778:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2875:%.*]] = icmp sgt i32 [[TMP1777]], [[TMP1778]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2875]], label [[IF_THEN2877:%.*]], label [[IF_END2878:%.*]]
+// SIMD-ONLY0:       if.then2877:
+// SIMD-ONLY0-NEXT:    [[TMP1779:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1779]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2878]]
+// SIMD-ONLY0:       if.end2878:
+// SIMD-ONLY0-NEXT:    [[TMP1780:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1780]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1781:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1782:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2879:%.*]] = icmp sgt i32 [[TMP1781]], [[TMP1782]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2879]], label [[IF_THEN2881:%.*]], label [[IF_END2882:%.*]]
+// SIMD-ONLY0:       if.then2881:
+// SIMD-ONLY0-NEXT:    [[TMP1783:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1783]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2882]]
+// SIMD-ONLY0:       if.end2882:
+// SIMD-ONLY0-NEXT:    [[TMP1784:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1784]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1785:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1786:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2883:%.*]] = icmp slt i32 [[TMP1785]], [[TMP1786]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2883]], label [[IF_THEN2885:%.*]], label [[IF_END2886:%.*]]
+// SIMD-ONLY0:       if.then2885:
+// SIMD-ONLY0-NEXT:    [[TMP1787:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1787]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2886]]
+// SIMD-ONLY0:       if.end2886:
+// SIMD-ONLY0-NEXT:    [[TMP1788:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1788]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1789:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1790:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2887:%.*]] = icmp slt i32 [[TMP1789]], [[TMP1790]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2887]], label [[IF_THEN2889:%.*]], label [[IF_END2890:%.*]]
+// SIMD-ONLY0:       if.then2889:
+// SIMD-ONLY0-NEXT:    [[TMP1791:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1791]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2890]]
+// SIMD-ONLY0:       if.end2890:
+// SIMD-ONLY0-NEXT:    [[TMP1792:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1792]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1793:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1794:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2891:%.*]] = icmp eq i32 [[TMP1793]], [[TMP1794]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2891]], label [[IF_THEN2893:%.*]], label [[IF_END2894:%.*]]
+// SIMD-ONLY0:       if.then2893:
+// SIMD-ONLY0-NEXT:    [[TMP1795:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1795]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2894]]
+// SIMD-ONLY0:       if.end2894:
+// SIMD-ONLY0-NEXT:    [[TMP1796:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1796]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1797:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1798:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2895:%.*]] = icmp eq i32 [[TMP1797]], [[TMP1798]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2895]], label [[IF_THEN2897:%.*]], label [[IF_END2898:%.*]]
+// SIMD-ONLY0:       if.then2897:
+// SIMD-ONLY0-NEXT:    [[TMP1799:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1799]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2898]]
+// SIMD-ONLY0:       if.end2898:
+// SIMD-ONLY0-NEXT:    [[TMP1800:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1801:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2899:%.*]] = icmp sgt i32 [[TMP1800]], [[TMP1801]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2899]], label [[IF_THEN2901:%.*]], label [[IF_END2902:%.*]]
+// SIMD-ONLY0:       if.then2901:
+// SIMD-ONLY0-NEXT:    [[TMP1802:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1802]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2902]]
+// SIMD-ONLY0:       if.end2902:
+// SIMD-ONLY0-NEXT:    [[TMP1803:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1803]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1804:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1805:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2903:%.*]] = icmp sgt i32 [[TMP1804]], [[TMP1805]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2903]], label [[IF_THEN2905:%.*]], label [[IF_END2906:%.*]]
+// SIMD-ONLY0:       if.then2905:
+// SIMD-ONLY0-NEXT:    [[TMP1806:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1806]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2906]]
+// SIMD-ONLY0:       if.end2906:
+// SIMD-ONLY0-NEXT:    [[TMP1807:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1807]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1808:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1809:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2907:%.*]] = icmp slt i32 [[TMP1808]], [[TMP1809]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2907]], label [[IF_THEN2909:%.*]], label [[IF_END2910:%.*]]
+// SIMD-ONLY0:       if.then2909:
+// SIMD-ONLY0-NEXT:    [[TMP1810:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1810]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2910]]
+// SIMD-ONLY0:       if.end2910:
+// SIMD-ONLY0-NEXT:    [[TMP1811:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1811]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1812:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1813:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2911:%.*]] = icmp slt i32 [[TMP1812]], [[TMP1813]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2911]], label [[IF_THEN2913:%.*]], label [[IF_END2914:%.*]]
+// SIMD-ONLY0:       if.then2913:
+// SIMD-ONLY0-NEXT:    [[TMP1814:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1814]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2914]]
+// SIMD-ONLY0:       if.end2914:
+// SIMD-ONLY0-NEXT:    [[TMP1815:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1815]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1816:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1817:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2915:%.*]] = icmp eq i32 [[TMP1816]], [[TMP1817]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2915]], label [[IF_THEN2917:%.*]], label [[IF_END2918:%.*]]
+// SIMD-ONLY0:       if.then2917:
+// SIMD-ONLY0-NEXT:    [[TMP1818:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1818]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2918]]
+// SIMD-ONLY0:       if.end2918:
+// SIMD-ONLY0-NEXT:    [[TMP1819:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1819]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1820:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1821:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2919:%.*]] = icmp eq i32 [[TMP1820]], [[TMP1821]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2919]], label [[IF_THEN2921:%.*]], label [[IF_END2922:%.*]]
+// SIMD-ONLY0:       if.then2921:
+// SIMD-ONLY0-NEXT:    [[TMP1822:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1822]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2922]]
+// SIMD-ONLY0:       if.end2922:
+// SIMD-ONLY0-NEXT:    [[TMP1823:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1823]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1824:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1825:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2923:%.*]] = icmp eq i32 [[TMP1824]], [[TMP1825]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2923]], label [[IF_THEN2925:%.*]], label [[IF_ELSE2926:%.*]]
+// SIMD-ONLY0:       if.then2925:
+// SIMD-ONLY0-NEXT:    [[TMP1826:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1826]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2927:%.*]]
+// SIMD-ONLY0:       if.else2926:
+// SIMD-ONLY0-NEXT:    [[TMP1827:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1827]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2927]]
+// SIMD-ONLY0:       if.end2927:
+// SIMD-ONLY0-NEXT:    [[TMP1828:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1829:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2928:%.*]] = icmp eq i32 [[TMP1828]], [[TMP1829]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2928]], label [[IF_THEN2930:%.*]], label [[IF_ELSE2931:%.*]]
+// SIMD-ONLY0:       if.then2930:
+// SIMD-ONLY0-NEXT:    [[TMP1830:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1830]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2932:%.*]]
+// SIMD-ONLY0:       if.else2931:
+// SIMD-ONLY0-NEXT:    [[TMP1831:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1831]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2932]]
+// SIMD-ONLY0:       if.end2932:
+// SIMD-ONLY0-NEXT:    [[TMP1832:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1833:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2933:%.*]] = icmp eq i32 [[TMP1832]], [[TMP1833]]
+// SIMD-ONLY0-NEXT:    [[CONV2934:%.*]] = zext i1 [[CMP2933]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV2934]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1834:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL2935:%.*]] = icmp ne i32 [[TMP1834]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2935]], label [[IF_THEN2936:%.*]], label [[IF_END2937:%.*]]
+// SIMD-ONLY0:       if.then2936:
+// SIMD-ONLY0-NEXT:    [[TMP1835:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1835]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2937]]
+// SIMD-ONLY0:       if.end2937:
+// SIMD-ONLY0-NEXT:    [[TMP1836:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1837:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2938:%.*]] = icmp eq i32 [[TMP1836]], [[TMP1837]]
+// SIMD-ONLY0-NEXT:    [[CONV2939:%.*]] = zext i1 [[CMP2938]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV2939]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1838:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL2940:%.*]] = icmp ne i32 [[TMP1838]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2940]], label [[IF_THEN2941:%.*]], label [[IF_END2942:%.*]]
+// SIMD-ONLY0:       if.then2941:
+// SIMD-ONLY0-NEXT:    [[TMP1839:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1839]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2942]]
+// SIMD-ONLY0:       if.end2942:
+// SIMD-ONLY0-NEXT:    [[TMP1840:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1841:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2943:%.*]] = icmp eq i32 [[TMP1840]], [[TMP1841]]
+// SIMD-ONLY0-NEXT:    [[CONV2944:%.*]] = zext i1 [[CMP2943]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV2944]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1842:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL2945:%.*]] = icmp ne i32 [[TMP1842]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2945]], label [[IF_THEN2946:%.*]], label [[IF_ELSE2947:%.*]]
+// SIMD-ONLY0:       if.then2946:
+// SIMD-ONLY0-NEXT:    [[TMP1843:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1843]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2948:%.*]]
+// SIMD-ONLY0:       if.else2947:
+// SIMD-ONLY0-NEXT:    [[TMP1844:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1844]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2948]]
+// SIMD-ONLY0:       if.end2948:
+// SIMD-ONLY0-NEXT:    [[TMP1845:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1846:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2949:%.*]] = icmp eq i32 [[TMP1845]], [[TMP1846]]
+// SIMD-ONLY0-NEXT:    [[CONV2950:%.*]] = zext i1 [[CMP2949]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV2950]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1847:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL2951:%.*]] = icmp ne i32 [[TMP1847]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL2951]], label [[IF_THEN2952:%.*]], label [[IF_ELSE2953:%.*]]
+// SIMD-ONLY0:       if.then2952:
+// SIMD-ONLY0-NEXT:    [[TMP1848:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1848]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2954:%.*]]
+// SIMD-ONLY0:       if.else2953:
+// SIMD-ONLY0-NEXT:    [[TMP1849:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1849]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2954]]
+// SIMD-ONLY0:       if.end2954:
+// SIMD-ONLY0-NEXT:    [[TMP1850:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1850]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1851:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1852:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2955:%.*]] = icmp sgt i32 [[TMP1851]], [[TMP1852]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2955]], label [[IF_THEN2957:%.*]], label [[IF_END2958:%.*]]
+// SIMD-ONLY0:       if.then2957:
+// SIMD-ONLY0-NEXT:    [[TMP1853:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1853]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2958]]
+// SIMD-ONLY0:       if.end2958:
+// SIMD-ONLY0-NEXT:    [[TMP1854:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1854]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1855:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1856:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2959:%.*]] = icmp sgt i32 [[TMP1855]], [[TMP1856]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2959]], label [[IF_THEN2961:%.*]], label [[IF_END2962:%.*]]
+// SIMD-ONLY0:       if.then2961:
+// SIMD-ONLY0-NEXT:    [[TMP1857:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1857]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2962]]
+// SIMD-ONLY0:       if.end2962:
+// SIMD-ONLY0-NEXT:    [[TMP1858:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1858]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1859:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1860:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2963:%.*]] = icmp slt i32 [[TMP1859]], [[TMP1860]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2963]], label [[IF_THEN2965:%.*]], label [[IF_END2966:%.*]]
+// SIMD-ONLY0:       if.then2965:
+// SIMD-ONLY0-NEXT:    [[TMP1861:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1861]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2966]]
+// SIMD-ONLY0:       if.end2966:
+// SIMD-ONLY0-NEXT:    [[TMP1862:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1862]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1863:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1864:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2967:%.*]] = icmp slt i32 [[TMP1863]], [[TMP1864]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2967]], label [[IF_THEN2969:%.*]], label [[IF_END2970:%.*]]
+// SIMD-ONLY0:       if.then2969:
+// SIMD-ONLY0-NEXT:    [[TMP1865:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1865]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2970]]
+// SIMD-ONLY0:       if.end2970:
+// SIMD-ONLY0-NEXT:    [[TMP1866:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1866]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1867:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1868:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2971:%.*]] = icmp eq i32 [[TMP1867]], [[TMP1868]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2971]], label [[IF_THEN2973:%.*]], label [[IF_END2974:%.*]]
+// SIMD-ONLY0:       if.then2973:
+// SIMD-ONLY0-NEXT:    [[TMP1869:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1869]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2974]]
+// SIMD-ONLY0:       if.end2974:
+// SIMD-ONLY0-NEXT:    [[TMP1870:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1870]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1871:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1872:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2975:%.*]] = icmp eq i32 [[TMP1871]], [[TMP1872]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2975]], label [[IF_THEN2977:%.*]], label [[IF_END2978:%.*]]
+// SIMD-ONLY0:       if.then2977:
+// SIMD-ONLY0-NEXT:    [[TMP1873:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1873]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2978]]
+// SIMD-ONLY0:       if.end2978:
+// SIMD-ONLY0-NEXT:    [[TMP1874:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1875:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2979:%.*]] = icmp sgt i32 [[TMP1874]], [[TMP1875]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2979]], label [[IF_THEN2981:%.*]], label [[IF_END2982:%.*]]
+// SIMD-ONLY0:       if.then2981:
+// SIMD-ONLY0-NEXT:    [[TMP1876:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1876]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2982]]
+// SIMD-ONLY0:       if.end2982:
+// SIMD-ONLY0-NEXT:    [[TMP1877:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1877]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1878:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1879:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2983:%.*]] = icmp sgt i32 [[TMP1878]], [[TMP1879]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2983]], label [[IF_THEN2985:%.*]], label [[IF_END2986:%.*]]
+// SIMD-ONLY0:       if.then2985:
+// SIMD-ONLY0-NEXT:    [[TMP1880:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1880]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2986]]
+// SIMD-ONLY0:       if.end2986:
+// SIMD-ONLY0-NEXT:    [[TMP1881:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1881]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1882:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1883:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2987:%.*]] = icmp slt i32 [[TMP1882]], [[TMP1883]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2987]], label [[IF_THEN2989:%.*]], label [[IF_END2990:%.*]]
+// SIMD-ONLY0:       if.then2989:
+// SIMD-ONLY0-NEXT:    [[TMP1884:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1884]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2990]]
+// SIMD-ONLY0:       if.end2990:
+// SIMD-ONLY0-NEXT:    [[TMP1885:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1885]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1886:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1887:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2991:%.*]] = icmp slt i32 [[TMP1886]], [[TMP1887]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2991]], label [[IF_THEN2993:%.*]], label [[IF_END2994:%.*]]
+// SIMD-ONLY0:       if.then2993:
+// SIMD-ONLY0-NEXT:    [[TMP1888:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1888]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2994]]
+// SIMD-ONLY0:       if.end2994:
+// SIMD-ONLY0-NEXT:    [[TMP1889:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1889]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1890:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1891:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2995:%.*]] = icmp eq i32 [[TMP1890]], [[TMP1891]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2995]], label [[IF_THEN2997:%.*]], label [[IF_END2998:%.*]]
+// SIMD-ONLY0:       if.then2997:
+// SIMD-ONLY0-NEXT:    [[TMP1892:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1892]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END2998]]
+// SIMD-ONLY0:       if.end2998:
+// SIMD-ONLY0-NEXT:    [[TMP1893:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1893]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1894:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1895:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2999:%.*]] = icmp eq i32 [[TMP1894]], [[TMP1895]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2999]], label [[IF_THEN3001:%.*]], label [[IF_END3002:%.*]]
+// SIMD-ONLY0:       if.then3001:
+// SIMD-ONLY0-NEXT:    [[TMP1896:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1896]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3002]]
+// SIMD-ONLY0:       if.end3002:
+// SIMD-ONLY0-NEXT:    [[TMP1897:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1897]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1898:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1899:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3003:%.*]] = icmp eq i32 [[TMP1898]], [[TMP1899]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3003]], label [[IF_THEN3005:%.*]], label [[IF_ELSE3006:%.*]]
+// SIMD-ONLY0:       if.then3005:
+// SIMD-ONLY0-NEXT:    [[TMP1900:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1900]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3007:%.*]]
+// SIMD-ONLY0:       if.else3006:
+// SIMD-ONLY0-NEXT:    [[TMP1901:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1901]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3007]]
+// SIMD-ONLY0:       if.end3007:
+// SIMD-ONLY0-NEXT:    [[TMP1902:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1903:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3008:%.*]] = icmp eq i32 [[TMP1902]], [[TMP1903]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3008]], label [[IF_THEN3010:%.*]], label [[IF_ELSE3011:%.*]]
+// SIMD-ONLY0:       if.then3010:
+// SIMD-ONLY0-NEXT:    [[TMP1904:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1904]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3012:%.*]]
+// SIMD-ONLY0:       if.else3011:
+// SIMD-ONLY0-NEXT:    [[TMP1905:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1905]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3012]]
+// SIMD-ONLY0:       if.end3012:
+// SIMD-ONLY0-NEXT:    [[TMP1906:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1907:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3013:%.*]] = icmp eq i32 [[TMP1906]], [[TMP1907]]
+// SIMD-ONLY0-NEXT:    [[CONV3014:%.*]] = zext i1 [[CMP3013]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3014]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1908:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3015:%.*]] = icmp ne i32 [[TMP1908]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3015]], label [[IF_THEN3016:%.*]], label [[IF_END3017:%.*]]
+// SIMD-ONLY0:       if.then3016:
+// SIMD-ONLY0-NEXT:    [[TMP1909:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1909]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3017]]
+// SIMD-ONLY0:       if.end3017:
+// SIMD-ONLY0-NEXT:    [[TMP1910:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1911:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3018:%.*]] = icmp eq i32 [[TMP1910]], [[TMP1911]]
+// SIMD-ONLY0-NEXT:    [[CONV3019:%.*]] = zext i1 [[CMP3018]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3019]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1912:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3020:%.*]] = icmp ne i32 [[TMP1912]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3020]], label [[IF_THEN3021:%.*]], label [[IF_END3022:%.*]]
+// SIMD-ONLY0:       if.then3021:
+// SIMD-ONLY0-NEXT:    [[TMP1913:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1913]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3022]]
+// SIMD-ONLY0:       if.end3022:
+// SIMD-ONLY0-NEXT:    [[TMP1914:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1915:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3023:%.*]] = icmp eq i32 [[TMP1914]], [[TMP1915]]
+// SIMD-ONLY0-NEXT:    [[CONV3024:%.*]] = zext i1 [[CMP3023]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3024]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1916:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3025:%.*]] = icmp ne i32 [[TMP1916]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3025]], label [[IF_THEN3026:%.*]], label [[IF_ELSE3027:%.*]]
+// SIMD-ONLY0:       if.then3026:
+// SIMD-ONLY0-NEXT:    [[TMP1917:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1917]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3028:%.*]]
+// SIMD-ONLY0:       if.else3027:
+// SIMD-ONLY0-NEXT:    [[TMP1918:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1918]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3028]]
+// SIMD-ONLY0:       if.end3028:
+// SIMD-ONLY0-NEXT:    [[TMP1919:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1920:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3029:%.*]] = icmp eq i32 [[TMP1919]], [[TMP1920]]
+// SIMD-ONLY0-NEXT:    [[CONV3030:%.*]] = zext i1 [[CMP3029]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3030]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1921:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3031:%.*]] = icmp ne i32 [[TMP1921]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3031]], label [[IF_THEN3032:%.*]], label [[IF_ELSE3033:%.*]]
+// SIMD-ONLY0:       if.then3032:
+// SIMD-ONLY0-NEXT:    [[TMP1922:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1922]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3034:%.*]]
+// SIMD-ONLY0:       if.else3033:
+// SIMD-ONLY0-NEXT:    [[TMP1923:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1923]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3034]]
+// SIMD-ONLY0:       if.end3034:
+// SIMD-ONLY0-NEXT:    [[TMP1924:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1924]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1925:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1926:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3035:%.*]] = icmp sgt i32 [[TMP1925]], [[TMP1926]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3035]], label [[IF_THEN3037:%.*]], label [[IF_END3038:%.*]]
+// SIMD-ONLY0:       if.then3037:
+// SIMD-ONLY0-NEXT:    [[TMP1927:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1927]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3038]]
+// SIMD-ONLY0:       if.end3038:
+// SIMD-ONLY0-NEXT:    [[TMP1928:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1928]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1929:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1930:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3039:%.*]] = icmp sgt i32 [[TMP1929]], [[TMP1930]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3039]], label [[IF_THEN3041:%.*]], label [[IF_END3042:%.*]]
+// SIMD-ONLY0:       if.then3041:
+// SIMD-ONLY0-NEXT:    [[TMP1931:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1931]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3042]]
+// SIMD-ONLY0:       if.end3042:
+// SIMD-ONLY0-NEXT:    [[TMP1932:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1932]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1933:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1934:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3043:%.*]] = icmp slt i32 [[TMP1933]], [[TMP1934]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3043]], label [[IF_THEN3045:%.*]], label [[IF_END3046:%.*]]
+// SIMD-ONLY0:       if.then3045:
+// SIMD-ONLY0-NEXT:    [[TMP1935:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1935]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3046]]
+// SIMD-ONLY0:       if.end3046:
+// SIMD-ONLY0-NEXT:    [[TMP1936:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1936]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1937:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1938:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3047:%.*]] = icmp slt i32 [[TMP1937]], [[TMP1938]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3047]], label [[IF_THEN3049:%.*]], label [[IF_END3050:%.*]]
+// SIMD-ONLY0:       if.then3049:
+// SIMD-ONLY0-NEXT:    [[TMP1939:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1939]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3050]]
+// SIMD-ONLY0:       if.end3050:
+// SIMD-ONLY0-NEXT:    [[TMP1940:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1940]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1941:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1942:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3051:%.*]] = icmp eq i32 [[TMP1941]], [[TMP1942]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3051]], label [[IF_THEN3053:%.*]], label [[IF_END3054:%.*]]
+// SIMD-ONLY0:       if.then3053:
+// SIMD-ONLY0-NEXT:    [[TMP1943:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1943]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3054]]
+// SIMD-ONLY0:       if.end3054:
+// SIMD-ONLY0-NEXT:    [[TMP1944:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1944]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1945:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1946:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3055:%.*]] = icmp eq i32 [[TMP1945]], [[TMP1946]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3055]], label [[IF_THEN3057:%.*]], label [[IF_END3058:%.*]]
+// SIMD-ONLY0:       if.then3057:
+// SIMD-ONLY0-NEXT:    [[TMP1947:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1947]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3058]]
+// SIMD-ONLY0:       if.end3058:
+// SIMD-ONLY0-NEXT:    [[TMP1948:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1949:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3059:%.*]] = icmp sgt i32 [[TMP1948]], [[TMP1949]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3059]], label [[IF_THEN3061:%.*]], label [[IF_END3062:%.*]]
+// SIMD-ONLY0:       if.then3061:
+// SIMD-ONLY0-NEXT:    [[TMP1950:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1950]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3062]]
+// SIMD-ONLY0:       if.end3062:
+// SIMD-ONLY0-NEXT:    [[TMP1951:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1951]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1952:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1953:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3063:%.*]] = icmp sgt i32 [[TMP1952]], [[TMP1953]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3063]], label [[IF_THEN3065:%.*]], label [[IF_END3066:%.*]]
+// SIMD-ONLY0:       if.then3065:
+// SIMD-ONLY0-NEXT:    [[TMP1954:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1954]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3066]]
+// SIMD-ONLY0:       if.end3066:
+// SIMD-ONLY0-NEXT:    [[TMP1955:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1955]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1956:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1957:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3067:%.*]] = icmp slt i32 [[TMP1956]], [[TMP1957]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3067]], label [[IF_THEN3069:%.*]], label [[IF_END3070:%.*]]
+// SIMD-ONLY0:       if.then3069:
+// SIMD-ONLY0-NEXT:    [[TMP1958:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1958]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3070]]
+// SIMD-ONLY0:       if.end3070:
+// SIMD-ONLY0-NEXT:    [[TMP1959:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1959]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1960:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1961:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3071:%.*]] = icmp slt i32 [[TMP1960]], [[TMP1961]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3071]], label [[IF_THEN3073:%.*]], label [[IF_END3074:%.*]]
+// SIMD-ONLY0:       if.then3073:
+// SIMD-ONLY0-NEXT:    [[TMP1962:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1962]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3074]]
+// SIMD-ONLY0:       if.end3074:
+// SIMD-ONLY0-NEXT:    [[TMP1963:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1963]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1964:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1965:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3075:%.*]] = icmp eq i32 [[TMP1964]], [[TMP1965]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3075]], label [[IF_THEN3077:%.*]], label [[IF_END3078:%.*]]
+// SIMD-ONLY0:       if.then3077:
+// SIMD-ONLY0-NEXT:    [[TMP1966:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1966]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3078]]
+// SIMD-ONLY0:       if.end3078:
+// SIMD-ONLY0-NEXT:    [[TMP1967:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1967]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1968:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1969:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3079:%.*]] = icmp eq i32 [[TMP1968]], [[TMP1969]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3079]], label [[IF_THEN3081:%.*]], label [[IF_END3082:%.*]]
+// SIMD-ONLY0:       if.then3081:
+// SIMD-ONLY0-NEXT:    [[TMP1970:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1970]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3082]]
+// SIMD-ONLY0:       if.end3082:
+// SIMD-ONLY0-NEXT:    [[TMP1971:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1971]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1972:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1973:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3083:%.*]] = icmp eq i32 [[TMP1972]], [[TMP1973]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3083]], label [[IF_THEN3085:%.*]], label [[IF_ELSE3086:%.*]]
+// SIMD-ONLY0:       if.then3085:
+// SIMD-ONLY0-NEXT:    [[TMP1974:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1974]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3087:%.*]]
+// SIMD-ONLY0:       if.else3086:
+// SIMD-ONLY0-NEXT:    [[TMP1975:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1975]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3087]]
+// SIMD-ONLY0:       if.end3087:
+// SIMD-ONLY0-NEXT:    [[TMP1976:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1977:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3088:%.*]] = icmp eq i32 [[TMP1976]], [[TMP1977]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3088]], label [[IF_THEN3090:%.*]], label [[IF_ELSE3091:%.*]]
+// SIMD-ONLY0:       if.then3090:
+// SIMD-ONLY0-NEXT:    [[TMP1978:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1978]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3092:%.*]]
+// SIMD-ONLY0:       if.else3091:
+// SIMD-ONLY0-NEXT:    [[TMP1979:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1979]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3092]]
+// SIMD-ONLY0:       if.end3092:
+// SIMD-ONLY0-NEXT:    [[TMP1980:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1981:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3093:%.*]] = icmp eq i32 [[TMP1980]], [[TMP1981]]
+// SIMD-ONLY0-NEXT:    [[CONV3094:%.*]] = zext i1 [[CMP3093]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3094]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1982:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3095:%.*]] = icmp ne i32 [[TMP1982]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3095]], label [[IF_THEN3096:%.*]], label [[IF_END3097:%.*]]
+// SIMD-ONLY0:       if.then3096:
+// SIMD-ONLY0-NEXT:    [[TMP1983:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1983]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3097]]
+// SIMD-ONLY0:       if.end3097:
+// SIMD-ONLY0-NEXT:    [[TMP1984:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1985:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3098:%.*]] = icmp eq i32 [[TMP1984]], [[TMP1985]]
+// SIMD-ONLY0-NEXT:    [[CONV3099:%.*]] = zext i1 [[CMP3098]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3099]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1986:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3100:%.*]] = icmp ne i32 [[TMP1986]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3100]], label [[IF_THEN3101:%.*]], label [[IF_END3102:%.*]]
+// SIMD-ONLY0:       if.then3101:
+// SIMD-ONLY0-NEXT:    [[TMP1987:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1987]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3102]]
+// SIMD-ONLY0:       if.end3102:
+// SIMD-ONLY0-NEXT:    [[TMP1988:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1989:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3103:%.*]] = icmp eq i32 [[TMP1988]], [[TMP1989]]
+// SIMD-ONLY0-NEXT:    [[CONV3104:%.*]] = zext i1 [[CMP3103]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3104]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1990:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3105:%.*]] = icmp ne i32 [[TMP1990]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3105]], label [[IF_THEN3106:%.*]], label [[IF_ELSE3107:%.*]]
+// SIMD-ONLY0:       if.then3106:
+// SIMD-ONLY0-NEXT:    [[TMP1991:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1991]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3108:%.*]]
+// SIMD-ONLY0:       if.else3107:
+// SIMD-ONLY0-NEXT:    [[TMP1992:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1992]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3108]]
+// SIMD-ONLY0:       if.end3108:
+// SIMD-ONLY0-NEXT:    [[TMP1993:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1994:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3109:%.*]] = icmp eq i32 [[TMP1993]], [[TMP1994]]
+// SIMD-ONLY0-NEXT:    [[CONV3110:%.*]] = zext i1 [[CMP3109]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3110]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1995:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3111:%.*]] = icmp ne i32 [[TMP1995]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3111]], label [[IF_THEN3112:%.*]], label [[IF_ELSE3113:%.*]]
+// SIMD-ONLY0:       if.then3112:
+// SIMD-ONLY0-NEXT:    [[TMP1996:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1996]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3114:%.*]]
+// SIMD-ONLY0:       if.else3113:
+// SIMD-ONLY0-NEXT:    [[TMP1997:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1997]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3114]]
+// SIMD-ONLY0:       if.end3114:
+// SIMD-ONLY0-NEXT:    [[TMP1998:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP1998]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1999:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2000:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3115:%.*]] = icmp sgt i32 [[TMP1999]], [[TMP2000]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3115]], label [[IF_THEN3117:%.*]], label [[IF_END3118:%.*]]
+// SIMD-ONLY0:       if.then3117:
+// SIMD-ONLY0-NEXT:    [[TMP2001:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2001]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3118]]
+// SIMD-ONLY0:       if.end3118:
+// SIMD-ONLY0-NEXT:    [[TMP2002:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2002]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2003:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2004:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3119:%.*]] = icmp sgt i32 [[TMP2003]], [[TMP2004]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3119]], label [[IF_THEN3121:%.*]], label [[IF_END3122:%.*]]
+// SIMD-ONLY0:       if.then3121:
+// SIMD-ONLY0-NEXT:    [[TMP2005:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2005]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3122]]
+// SIMD-ONLY0:       if.end3122:
+// SIMD-ONLY0-NEXT:    [[TMP2006:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2006]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2007:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2008:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3123:%.*]] = icmp slt i32 [[TMP2007]], [[TMP2008]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3123]], label [[IF_THEN3125:%.*]], label [[IF_END3126:%.*]]
+// SIMD-ONLY0:       if.then3125:
+// SIMD-ONLY0-NEXT:    [[TMP2009:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2009]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3126]]
+// SIMD-ONLY0:       if.end3126:
+// SIMD-ONLY0-NEXT:    [[TMP2010:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2010]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2011:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2012:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3127:%.*]] = icmp slt i32 [[TMP2011]], [[TMP2012]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3127]], label [[IF_THEN3129:%.*]], label [[IF_END3130:%.*]]
+// SIMD-ONLY0:       if.then3129:
+// SIMD-ONLY0-NEXT:    [[TMP2013:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2013]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3130]]
+// SIMD-ONLY0:       if.end3130:
+// SIMD-ONLY0-NEXT:    [[TMP2014:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2014]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2015:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2016:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3131:%.*]] = icmp eq i32 [[TMP2015]], [[TMP2016]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3131]], label [[IF_THEN3133:%.*]], label [[IF_END3134:%.*]]
+// SIMD-ONLY0:       if.then3133:
+// SIMD-ONLY0-NEXT:    [[TMP2017:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2017]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3134]]
+// SIMD-ONLY0:       if.end3134:
+// SIMD-ONLY0-NEXT:    [[TMP2018:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2018]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2019:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2020:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3135:%.*]] = icmp eq i32 [[TMP2019]], [[TMP2020]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3135]], label [[IF_THEN3137:%.*]], label [[IF_END3138:%.*]]
+// SIMD-ONLY0:       if.then3137:
+// SIMD-ONLY0-NEXT:    [[TMP2021:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2021]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3138]]
+// SIMD-ONLY0:       if.end3138:
+// SIMD-ONLY0-NEXT:    [[TMP2022:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2023:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3139:%.*]] = icmp sgt i32 [[TMP2022]], [[TMP2023]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3139]], label [[IF_THEN3141:%.*]], label [[IF_END3142:%.*]]
+// SIMD-ONLY0:       if.then3141:
+// SIMD-ONLY0-NEXT:    [[TMP2024:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2024]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3142]]
+// SIMD-ONLY0:       if.end3142:
+// SIMD-ONLY0-NEXT:    [[TMP2025:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2025]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2026:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2027:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3143:%.*]] = icmp sgt i32 [[TMP2026]], [[TMP2027]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3143]], label [[IF_THEN3145:%.*]], label [[IF_END3146:%.*]]
+// SIMD-ONLY0:       if.then3145:
+// SIMD-ONLY0-NEXT:    [[TMP2028:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2028]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3146]]
+// SIMD-ONLY0:       if.end3146:
+// SIMD-ONLY0-NEXT:    [[TMP2029:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2029]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2030:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2031:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3147:%.*]] = icmp slt i32 [[TMP2030]], [[TMP2031]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3147]], label [[IF_THEN3149:%.*]], label [[IF_END3150:%.*]]
+// SIMD-ONLY0:       if.then3149:
+// SIMD-ONLY0-NEXT:    [[TMP2032:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2032]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3150]]
+// SIMD-ONLY0:       if.end3150:
+// SIMD-ONLY0-NEXT:    [[TMP2033:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2033]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2034:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2035:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3151:%.*]] = icmp slt i32 [[TMP2034]], [[TMP2035]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3151]], label [[IF_THEN3153:%.*]], label [[IF_END3154:%.*]]
+// SIMD-ONLY0:       if.then3153:
+// SIMD-ONLY0-NEXT:    [[TMP2036:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2036]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3154]]
+// SIMD-ONLY0:       if.end3154:
+// SIMD-ONLY0-NEXT:    [[TMP2037:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2037]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2038:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2039:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3155:%.*]] = icmp eq i32 [[TMP2038]], [[TMP2039]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3155]], label [[IF_THEN3157:%.*]], label [[IF_END3158:%.*]]
+// SIMD-ONLY0:       if.then3157:
+// SIMD-ONLY0-NEXT:    [[TMP2040:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2040]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3158]]
+// SIMD-ONLY0:       if.end3158:
+// SIMD-ONLY0-NEXT:    [[TMP2041:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2041]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2042:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2043:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3159:%.*]] = icmp eq i32 [[TMP2042]], [[TMP2043]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3159]], label [[IF_THEN3161:%.*]], label [[IF_END3162:%.*]]
+// SIMD-ONLY0:       if.then3161:
+// SIMD-ONLY0-NEXT:    [[TMP2044:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2044]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3162]]
+// SIMD-ONLY0:       if.end3162:
+// SIMD-ONLY0-NEXT:    [[TMP2045:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2045]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2046:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2047:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3163:%.*]] = icmp eq i32 [[TMP2046]], [[TMP2047]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3163]], label [[IF_THEN3165:%.*]], label [[IF_ELSE3166:%.*]]
+// SIMD-ONLY0:       if.then3165:
+// SIMD-ONLY0-NEXT:    [[TMP2048:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2048]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3167:%.*]]
+// SIMD-ONLY0:       if.else3166:
+// SIMD-ONLY0-NEXT:    [[TMP2049:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2049]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3167]]
+// SIMD-ONLY0:       if.end3167:
+// SIMD-ONLY0-NEXT:    [[TMP2050:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2051:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3168:%.*]] = icmp eq i32 [[TMP2050]], [[TMP2051]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3168]], label [[IF_THEN3170:%.*]], label [[IF_ELSE3171:%.*]]
+// SIMD-ONLY0:       if.then3170:
+// SIMD-ONLY0-NEXT:    [[TMP2052:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2052]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3172:%.*]]
+// SIMD-ONLY0:       if.else3171:
+// SIMD-ONLY0-NEXT:    [[TMP2053:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2053]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3172]]
+// SIMD-ONLY0:       if.end3172:
+// SIMD-ONLY0-NEXT:    [[TMP2054:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2055:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3173:%.*]] = icmp eq i32 [[TMP2054]], [[TMP2055]]
+// SIMD-ONLY0-NEXT:    [[CONV3174:%.*]] = zext i1 [[CMP3173]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3174]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2056:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3175:%.*]] = icmp ne i32 [[TMP2056]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3175]], label [[IF_THEN3176:%.*]], label [[IF_END3177:%.*]]
+// SIMD-ONLY0:       if.then3176:
+// SIMD-ONLY0-NEXT:    [[TMP2057:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2057]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3177]]
+// SIMD-ONLY0:       if.end3177:
+// SIMD-ONLY0-NEXT:    [[TMP2058:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2059:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3178:%.*]] = icmp eq i32 [[TMP2058]], [[TMP2059]]
+// SIMD-ONLY0-NEXT:    [[CONV3179:%.*]] = zext i1 [[CMP3178]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3179]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2060:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3180:%.*]] = icmp ne i32 [[TMP2060]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3180]], label [[IF_THEN3181:%.*]], label [[IF_END3182:%.*]]
+// SIMD-ONLY0:       if.then3181:
+// SIMD-ONLY0-NEXT:    [[TMP2061:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2061]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3182]]
+// SIMD-ONLY0:       if.end3182:
+// SIMD-ONLY0-NEXT:    [[TMP2062:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2063:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3183:%.*]] = icmp eq i32 [[TMP2062]], [[TMP2063]]
+// SIMD-ONLY0-NEXT:    [[CONV3184:%.*]] = zext i1 [[CMP3183]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3184]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2064:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3185:%.*]] = icmp ne i32 [[TMP2064]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3185]], label [[IF_THEN3186:%.*]], label [[IF_ELSE3187:%.*]]
+// SIMD-ONLY0:       if.then3186:
+// SIMD-ONLY0-NEXT:    [[TMP2065:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2065]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3188:%.*]]
+// SIMD-ONLY0:       if.else3187:
+// SIMD-ONLY0-NEXT:    [[TMP2066:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2066]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3188]]
+// SIMD-ONLY0:       if.end3188:
+// SIMD-ONLY0-NEXT:    [[TMP2067:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2068:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3189:%.*]] = icmp eq i32 [[TMP2067]], [[TMP2068]]
+// SIMD-ONLY0-NEXT:    [[CONV3190:%.*]] = zext i1 [[CMP3189]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3190]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2069:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3191:%.*]] = icmp ne i32 [[TMP2069]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3191]], label [[IF_THEN3192:%.*]], label [[IF_ELSE3193:%.*]]
+// SIMD-ONLY0:       if.then3192:
+// SIMD-ONLY0-NEXT:    [[TMP2070:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2070]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3194:%.*]]
+// SIMD-ONLY0:       if.else3193:
+// SIMD-ONLY0-NEXT:    [[TMP2071:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2071]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3194]]
+// SIMD-ONLY0:       if.end3194:
+// SIMD-ONLY0-NEXT:    [[TMP2072:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2072]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2073:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2074:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3195:%.*]] = icmp sgt i32 [[TMP2073]], [[TMP2074]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3195]], label [[IF_THEN3197:%.*]], label [[IF_END3198:%.*]]
+// SIMD-ONLY0:       if.then3197:
+// SIMD-ONLY0-NEXT:    [[TMP2075:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2075]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3198]]
+// SIMD-ONLY0:       if.end3198:
+// SIMD-ONLY0-NEXT:    [[TMP2076:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2076]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2077:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2078:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3199:%.*]] = icmp sgt i32 [[TMP2077]], [[TMP2078]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3199]], label [[IF_THEN3201:%.*]], label [[IF_END3202:%.*]]
+// SIMD-ONLY0:       if.then3201:
+// SIMD-ONLY0-NEXT:    [[TMP2079:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2079]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3202]]
+// SIMD-ONLY0:       if.end3202:
+// SIMD-ONLY0-NEXT:    [[TMP2080:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2080]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2081:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2082:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3203:%.*]] = icmp slt i32 [[TMP2081]], [[TMP2082]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3203]], label [[IF_THEN3205:%.*]], label [[IF_END3206:%.*]]
+// SIMD-ONLY0:       if.then3205:
+// SIMD-ONLY0-NEXT:    [[TMP2083:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2083]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3206]]
+// SIMD-ONLY0:       if.end3206:
+// SIMD-ONLY0-NEXT:    [[TMP2084:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2084]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2085:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2086:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3207:%.*]] = icmp slt i32 [[TMP2085]], [[TMP2086]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3207]], label [[IF_THEN3209:%.*]], label [[IF_END3210:%.*]]
+// SIMD-ONLY0:       if.then3209:
+// SIMD-ONLY0-NEXT:    [[TMP2087:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2087]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3210]]
+// SIMD-ONLY0:       if.end3210:
+// SIMD-ONLY0-NEXT:    [[TMP2088:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2088]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2089:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2090:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3211:%.*]] = icmp eq i32 [[TMP2089]], [[TMP2090]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3211]], label [[IF_THEN3213:%.*]], label [[IF_END3214:%.*]]
+// SIMD-ONLY0:       if.then3213:
+// SIMD-ONLY0-NEXT:    [[TMP2091:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2091]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3214]]
+// SIMD-ONLY0:       if.end3214:
+// SIMD-ONLY0-NEXT:    [[TMP2092:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2092]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2093:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2094:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3215:%.*]] = icmp eq i32 [[TMP2093]], [[TMP2094]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3215]], label [[IF_THEN3217:%.*]], label [[IF_END3218:%.*]]
+// SIMD-ONLY0:       if.then3217:
+// SIMD-ONLY0-NEXT:    [[TMP2095:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2095]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3218]]
+// SIMD-ONLY0:       if.end3218:
+// SIMD-ONLY0-NEXT:    [[TMP2096:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2097:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3219:%.*]] = icmp sgt i32 [[TMP2096]], [[TMP2097]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3219]], label [[IF_THEN3221:%.*]], label [[IF_END3222:%.*]]
+// SIMD-ONLY0:       if.then3221:
+// SIMD-ONLY0-NEXT:    [[TMP2098:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2098]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3222]]
+// SIMD-ONLY0:       if.end3222:
+// SIMD-ONLY0-NEXT:    [[TMP2099:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2099]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2100:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2101:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3223:%.*]] = icmp sgt i32 [[TMP2100]], [[TMP2101]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3223]], label [[IF_THEN3225:%.*]], label [[IF_END3226:%.*]]
+// SIMD-ONLY0:       if.then3225:
+// SIMD-ONLY0-NEXT:    [[TMP2102:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2102]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3226]]
+// SIMD-ONLY0:       if.end3226:
+// SIMD-ONLY0-NEXT:    [[TMP2103:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2103]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2104:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2105:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3227:%.*]] = icmp slt i32 [[TMP2104]], [[TMP2105]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3227]], label [[IF_THEN3229:%.*]], label [[IF_END3230:%.*]]
+// SIMD-ONLY0:       if.then3229:
+// SIMD-ONLY0-NEXT:    [[TMP2106:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2106]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3230]]
+// SIMD-ONLY0:       if.end3230:
+// SIMD-ONLY0-NEXT:    [[TMP2107:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2107]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2108:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2109:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3231:%.*]] = icmp slt i32 [[TMP2108]], [[TMP2109]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3231]], label [[IF_THEN3233:%.*]], label [[IF_END3234:%.*]]
+// SIMD-ONLY0:       if.then3233:
+// SIMD-ONLY0-NEXT:    [[TMP2110:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2110]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3234]]
+// SIMD-ONLY0:       if.end3234:
+// SIMD-ONLY0-NEXT:    [[TMP2111:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2111]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2112:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2113:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3235:%.*]] = icmp eq i32 [[TMP2112]], [[TMP2113]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3235]], label [[IF_THEN3237:%.*]], label [[IF_END3238:%.*]]
+// SIMD-ONLY0:       if.then3237:
+// SIMD-ONLY0-NEXT:    [[TMP2114:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2114]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3238]]
+// SIMD-ONLY0:       if.end3238:
+// SIMD-ONLY0-NEXT:    [[TMP2115:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2115]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2116:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2117:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3239:%.*]] = icmp eq i32 [[TMP2116]], [[TMP2117]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3239]], label [[IF_THEN3241:%.*]], label [[IF_END3242:%.*]]
+// SIMD-ONLY0:       if.then3241:
+// SIMD-ONLY0-NEXT:    [[TMP2118:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2118]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3242]]
+// SIMD-ONLY0:       if.end3242:
+// SIMD-ONLY0-NEXT:    [[TMP2119:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2119]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2120:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2121:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3243:%.*]] = icmp eq i32 [[TMP2120]], [[TMP2121]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3243]], label [[IF_THEN3245:%.*]], label [[IF_ELSE3246:%.*]]
+// SIMD-ONLY0:       if.then3245:
+// SIMD-ONLY0-NEXT:    [[TMP2122:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2122]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3247:%.*]]
+// SIMD-ONLY0:       if.else3246:
+// SIMD-ONLY0-NEXT:    [[TMP2123:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2123]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3247]]
+// SIMD-ONLY0:       if.end3247:
+// SIMD-ONLY0-NEXT:    [[TMP2124:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2125:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3248:%.*]] = icmp eq i32 [[TMP2124]], [[TMP2125]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3248]], label [[IF_THEN3250:%.*]], label [[IF_ELSE3251:%.*]]
+// SIMD-ONLY0:       if.then3250:
+// SIMD-ONLY0-NEXT:    [[TMP2126:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2126]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3252:%.*]]
+// SIMD-ONLY0:       if.else3251:
+// SIMD-ONLY0-NEXT:    [[TMP2127:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2127]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3252]]
+// SIMD-ONLY0:       if.end3252:
+// SIMD-ONLY0-NEXT:    [[TMP2128:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2129:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3253:%.*]] = icmp eq i32 [[TMP2128]], [[TMP2129]]
+// SIMD-ONLY0-NEXT:    [[CONV3254:%.*]] = zext i1 [[CMP3253]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3254]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2130:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3255:%.*]] = icmp ne i32 [[TMP2130]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3255]], label [[IF_THEN3256:%.*]], label [[IF_END3257:%.*]]
+// SIMD-ONLY0:       if.then3256:
+// SIMD-ONLY0-NEXT:    [[TMP2131:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2131]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3257]]
+// SIMD-ONLY0:       if.end3257:
+// SIMD-ONLY0-NEXT:    [[TMP2132:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2133:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3258:%.*]] = icmp eq i32 [[TMP2132]], [[TMP2133]]
+// SIMD-ONLY0-NEXT:    [[CONV3259:%.*]] = zext i1 [[CMP3258]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3259]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2134:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3260:%.*]] = icmp ne i32 [[TMP2134]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3260]], label [[IF_THEN3261:%.*]], label [[IF_END3262:%.*]]
+// SIMD-ONLY0:       if.then3261:
+// SIMD-ONLY0-NEXT:    [[TMP2135:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2135]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3262]]
+// SIMD-ONLY0:       if.end3262:
+// SIMD-ONLY0-NEXT:    [[TMP2136:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2137:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3263:%.*]] = icmp eq i32 [[TMP2136]], [[TMP2137]]
+// SIMD-ONLY0-NEXT:    [[CONV3264:%.*]] = zext i1 [[CMP3263]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3264]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2138:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3265:%.*]] = icmp ne i32 [[TMP2138]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3265]], label [[IF_THEN3266:%.*]], label [[IF_ELSE3267:%.*]]
+// SIMD-ONLY0:       if.then3266:
+// SIMD-ONLY0-NEXT:    [[TMP2139:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2139]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3268:%.*]]
+// SIMD-ONLY0:       if.else3267:
+// SIMD-ONLY0-NEXT:    [[TMP2140:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2140]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3268]]
+// SIMD-ONLY0:       if.end3268:
+// SIMD-ONLY0-NEXT:    [[TMP2141:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2142:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3269:%.*]] = icmp eq i32 [[TMP2141]], [[TMP2142]]
+// SIMD-ONLY0-NEXT:    [[CONV3270:%.*]] = zext i1 [[CMP3269]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3270]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2143:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3271:%.*]] = icmp ne i32 [[TMP2143]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3271]], label [[IF_THEN3272:%.*]], label [[IF_ELSE3273:%.*]]
+// SIMD-ONLY0:       if.then3272:
+// SIMD-ONLY0-NEXT:    [[TMP2144:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2144]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3274:%.*]]
+// SIMD-ONLY0:       if.else3273:
+// SIMD-ONLY0-NEXT:    [[TMP2145:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2145]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3274]]
+// SIMD-ONLY0:       if.end3274:
+// SIMD-ONLY0-NEXT:    [[TMP2146:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2146]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2147:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2148:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3275:%.*]] = icmp sgt i32 [[TMP2147]], [[TMP2148]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3275]], label [[IF_THEN3277:%.*]], label [[IF_END3278:%.*]]
+// SIMD-ONLY0:       if.then3277:
+// SIMD-ONLY0-NEXT:    [[TMP2149:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2149]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3278]]
+// SIMD-ONLY0:       if.end3278:
+// SIMD-ONLY0-NEXT:    [[TMP2150:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2150]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2151:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2152:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3279:%.*]] = icmp sgt i32 [[TMP2151]], [[TMP2152]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3279]], label [[IF_THEN3281:%.*]], label [[IF_END3282:%.*]]
+// SIMD-ONLY0:       if.then3281:
+// SIMD-ONLY0-NEXT:    [[TMP2153:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2153]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3282]]
+// SIMD-ONLY0:       if.end3282:
+// SIMD-ONLY0-NEXT:    [[TMP2154:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2154]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2155:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2156:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3283:%.*]] = icmp slt i32 [[TMP2155]], [[TMP2156]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3283]], label [[IF_THEN3285:%.*]], label [[IF_END3286:%.*]]
+// SIMD-ONLY0:       if.then3285:
+// SIMD-ONLY0-NEXT:    [[TMP2157:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2157]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3286]]
+// SIMD-ONLY0:       if.end3286:
+// SIMD-ONLY0-NEXT:    [[TMP2158:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2158]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2159:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2160:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3287:%.*]] = icmp slt i32 [[TMP2159]], [[TMP2160]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3287]], label [[IF_THEN3289:%.*]], label [[IF_END3290:%.*]]
+// SIMD-ONLY0:       if.then3289:
+// SIMD-ONLY0-NEXT:    [[TMP2161:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2161]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3290]]
+// SIMD-ONLY0:       if.end3290:
+// SIMD-ONLY0-NEXT:    [[TMP2162:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2162]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2163:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2164:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3291:%.*]] = icmp eq i32 [[TMP2163]], [[TMP2164]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3291]], label [[IF_THEN3293:%.*]], label [[IF_END3294:%.*]]
+// SIMD-ONLY0:       if.then3293:
+// SIMD-ONLY0-NEXT:    [[TMP2165:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2165]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3294]]
+// SIMD-ONLY0:       if.end3294:
+// SIMD-ONLY0-NEXT:    [[TMP2166:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2166]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2167:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2168:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3295:%.*]] = icmp eq i32 [[TMP2167]], [[TMP2168]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3295]], label [[IF_THEN3297:%.*]], label [[IF_END3298:%.*]]
+// SIMD-ONLY0:       if.then3297:
+// SIMD-ONLY0-NEXT:    [[TMP2169:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2169]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3298]]
+// SIMD-ONLY0:       if.end3298:
+// SIMD-ONLY0-NEXT:    [[TMP2170:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2171:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3299:%.*]] = icmp sgt i32 [[TMP2170]], [[TMP2171]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3299]], label [[IF_THEN3301:%.*]], label [[IF_END3302:%.*]]
+// SIMD-ONLY0:       if.then3301:
+// SIMD-ONLY0-NEXT:    [[TMP2172:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2172]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3302]]
+// SIMD-ONLY0:       if.end3302:
+// SIMD-ONLY0-NEXT:    [[TMP2173:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2173]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2174:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2175:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3303:%.*]] = icmp sgt i32 [[TMP2174]], [[TMP2175]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3303]], label [[IF_THEN3305:%.*]], label [[IF_END3306:%.*]]
+// SIMD-ONLY0:       if.then3305:
+// SIMD-ONLY0-NEXT:    [[TMP2176:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2176]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3306]]
+// SIMD-ONLY0:       if.end3306:
+// SIMD-ONLY0-NEXT:    [[TMP2177:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2177]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2178:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2179:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3307:%.*]] = icmp slt i32 [[TMP2178]], [[TMP2179]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3307]], label [[IF_THEN3309:%.*]], label [[IF_END3310:%.*]]
+// SIMD-ONLY0:       if.then3309:
+// SIMD-ONLY0-NEXT:    [[TMP2180:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2180]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3310]]
+// SIMD-ONLY0:       if.end3310:
+// SIMD-ONLY0-NEXT:    [[TMP2181:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2181]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2182:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2183:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3311:%.*]] = icmp slt i32 [[TMP2182]], [[TMP2183]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3311]], label [[IF_THEN3313:%.*]], label [[IF_END3314:%.*]]
+// SIMD-ONLY0:       if.then3313:
+// SIMD-ONLY0-NEXT:    [[TMP2184:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2184]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3314]]
+// SIMD-ONLY0:       if.end3314:
+// SIMD-ONLY0-NEXT:    [[TMP2185:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2185]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2186:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2187:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3315:%.*]] = icmp eq i32 [[TMP2186]], [[TMP2187]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3315]], label [[IF_THEN3317:%.*]], label [[IF_END3318:%.*]]
+// SIMD-ONLY0:       if.then3317:
+// SIMD-ONLY0-NEXT:    [[TMP2188:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2188]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3318]]
+// SIMD-ONLY0:       if.end3318:
+// SIMD-ONLY0-NEXT:    [[TMP2189:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2189]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2190:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2191:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3319:%.*]] = icmp eq i32 [[TMP2190]], [[TMP2191]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3319]], label [[IF_THEN3321:%.*]], label [[IF_END3322:%.*]]
+// SIMD-ONLY0:       if.then3321:
+// SIMD-ONLY0-NEXT:    [[TMP2192:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2192]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3322]]
+// SIMD-ONLY0:       if.end3322:
+// SIMD-ONLY0-NEXT:    [[TMP2193:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2193]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2194:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2195:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3323:%.*]] = icmp eq i32 [[TMP2194]], [[TMP2195]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3323]], label [[IF_THEN3325:%.*]], label [[IF_ELSE3326:%.*]]
+// SIMD-ONLY0:       if.then3325:
+// SIMD-ONLY0-NEXT:    [[TMP2196:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2196]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3327:%.*]]
+// SIMD-ONLY0:       if.else3326:
+// SIMD-ONLY0-NEXT:    [[TMP2197:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2197]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3327]]
+// SIMD-ONLY0:       if.end3327:
+// SIMD-ONLY0-NEXT:    [[TMP2198:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2199:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3328:%.*]] = icmp eq i32 [[TMP2198]], [[TMP2199]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3328]], label [[IF_THEN3330:%.*]], label [[IF_ELSE3331:%.*]]
+// SIMD-ONLY0:       if.then3330:
+// SIMD-ONLY0-NEXT:    [[TMP2200:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2200]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3332:%.*]]
+// SIMD-ONLY0:       if.else3331:
+// SIMD-ONLY0-NEXT:    [[TMP2201:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2201]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3332]]
+// SIMD-ONLY0:       if.end3332:
+// SIMD-ONLY0-NEXT:    [[TMP2202:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2203:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3333:%.*]] = icmp eq i32 [[TMP2202]], [[TMP2203]]
+// SIMD-ONLY0-NEXT:    [[CONV3334:%.*]] = zext i1 [[CMP3333]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3334]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2204:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3335:%.*]] = icmp ne i32 [[TMP2204]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3335]], label [[IF_THEN3336:%.*]], label [[IF_END3337:%.*]]
+// SIMD-ONLY0:       if.then3336:
+// SIMD-ONLY0-NEXT:    [[TMP2205:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2205]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3337]]
+// SIMD-ONLY0:       if.end3337:
+// SIMD-ONLY0-NEXT:    [[TMP2206:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2207:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3338:%.*]] = icmp eq i32 [[TMP2206]], [[TMP2207]]
+// SIMD-ONLY0-NEXT:    [[CONV3339:%.*]] = zext i1 [[CMP3338]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3339]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2208:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3340:%.*]] = icmp ne i32 [[TMP2208]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3340]], label [[IF_THEN3341:%.*]], label [[IF_END3342:%.*]]
+// SIMD-ONLY0:       if.then3341:
+// SIMD-ONLY0-NEXT:    [[TMP2209:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2209]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3342]]
+// SIMD-ONLY0:       if.end3342:
+// SIMD-ONLY0-NEXT:    [[TMP2210:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2211:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3343:%.*]] = icmp eq i32 [[TMP2210]], [[TMP2211]]
+// SIMD-ONLY0-NEXT:    [[CONV3344:%.*]] = zext i1 [[CMP3343]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3344]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2212:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3345:%.*]] = icmp ne i32 [[TMP2212]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3345]], label [[IF_THEN3346:%.*]], label [[IF_ELSE3347:%.*]]
+// SIMD-ONLY0:       if.then3346:
+// SIMD-ONLY0-NEXT:    [[TMP2213:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2213]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3348:%.*]]
+// SIMD-ONLY0:       if.else3347:
+// SIMD-ONLY0-NEXT:    [[TMP2214:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2214]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3348]]
+// SIMD-ONLY0:       if.end3348:
+// SIMD-ONLY0-NEXT:    [[TMP2215:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2216:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3349:%.*]] = icmp eq i32 [[TMP2215]], [[TMP2216]]
+// SIMD-ONLY0-NEXT:    [[CONV3350:%.*]] = zext i1 [[CMP3349]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3350]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2217:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3351:%.*]] = icmp ne i32 [[TMP2217]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3351]], label [[IF_THEN3352:%.*]], label [[IF_ELSE3353:%.*]]
+// SIMD-ONLY0:       if.then3352:
+// SIMD-ONLY0-NEXT:    [[TMP2218:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2218]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3354:%.*]]
+// SIMD-ONLY0:       if.else3353:
+// SIMD-ONLY0-NEXT:    [[TMP2219:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2219]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3354]]
+// SIMD-ONLY0:       if.end3354:
+// SIMD-ONLY0-NEXT:    [[TMP2220:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2220]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2221:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2222:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3355:%.*]] = icmp ugt i32 [[TMP2221]], [[TMP2222]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3355]], label [[IF_THEN3357:%.*]], label [[IF_END3358:%.*]]
+// SIMD-ONLY0:       if.then3357:
+// SIMD-ONLY0-NEXT:    [[TMP2223:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2223]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3358]]
+// SIMD-ONLY0:       if.end3358:
+// SIMD-ONLY0-NEXT:    [[TMP2224:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2224]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2225:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2226:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3359:%.*]] = icmp ugt i32 [[TMP2225]], [[TMP2226]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3359]], label [[IF_THEN3361:%.*]], label [[IF_END3362:%.*]]
+// SIMD-ONLY0:       if.then3361:
+// SIMD-ONLY0-NEXT:    [[TMP2227:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2227]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3362]]
+// SIMD-ONLY0:       if.end3362:
+// SIMD-ONLY0-NEXT:    [[TMP2228:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2228]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2229:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2230:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3363:%.*]] = icmp ult i32 [[TMP2229]], [[TMP2230]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3363]], label [[IF_THEN3365:%.*]], label [[IF_END3366:%.*]]
+// SIMD-ONLY0:       if.then3365:
+// SIMD-ONLY0-NEXT:    [[TMP2231:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2231]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3366]]
+// SIMD-ONLY0:       if.end3366:
+// SIMD-ONLY0-NEXT:    [[TMP2232:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2232]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2233:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2234:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3367:%.*]] = icmp ult i32 [[TMP2233]], [[TMP2234]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3367]], label [[IF_THEN3369:%.*]], label [[IF_END3370:%.*]]
+// SIMD-ONLY0:       if.then3369:
+// SIMD-ONLY0-NEXT:    [[TMP2235:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2235]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3370]]
+// SIMD-ONLY0:       if.end3370:
+// SIMD-ONLY0-NEXT:    [[TMP2236:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2236]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2237:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2238:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3371:%.*]] = icmp eq i32 [[TMP2237]], [[TMP2238]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3371]], label [[IF_THEN3373:%.*]], label [[IF_END3374:%.*]]
+// SIMD-ONLY0:       if.then3373:
+// SIMD-ONLY0-NEXT:    [[TMP2239:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2239]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3374]]
+// SIMD-ONLY0:       if.end3374:
+// SIMD-ONLY0-NEXT:    [[TMP2240:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2240]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2241:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2242:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3375:%.*]] = icmp eq i32 [[TMP2241]], [[TMP2242]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3375]], label [[IF_THEN3377:%.*]], label [[IF_END3378:%.*]]
+// SIMD-ONLY0:       if.then3377:
+// SIMD-ONLY0-NEXT:    [[TMP2243:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2243]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3378]]
+// SIMD-ONLY0:       if.end3378:
+// SIMD-ONLY0-NEXT:    [[TMP2244:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2245:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3379:%.*]] = icmp ugt i32 [[TMP2244]], [[TMP2245]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3379]], label [[IF_THEN3381:%.*]], label [[IF_END3382:%.*]]
+// SIMD-ONLY0:       if.then3381:
+// SIMD-ONLY0-NEXT:    [[TMP2246:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2246]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3382]]
+// SIMD-ONLY0:       if.end3382:
+// SIMD-ONLY0-NEXT:    [[TMP2247:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2247]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2248:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2249:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3383:%.*]] = icmp ugt i32 [[TMP2248]], [[TMP2249]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3383]], label [[IF_THEN3385:%.*]], label [[IF_END3386:%.*]]
+// SIMD-ONLY0:       if.then3385:
+// SIMD-ONLY0-NEXT:    [[TMP2250:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2250]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3386]]
+// SIMD-ONLY0:       if.end3386:
+// SIMD-ONLY0-NEXT:    [[TMP2251:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2251]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2252:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2253:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3387:%.*]] = icmp ult i32 [[TMP2252]], [[TMP2253]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3387]], label [[IF_THEN3389:%.*]], label [[IF_END3390:%.*]]
+// SIMD-ONLY0:       if.then3389:
+// SIMD-ONLY0-NEXT:    [[TMP2254:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2254]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3390]]
+// SIMD-ONLY0:       if.end3390:
+// SIMD-ONLY0-NEXT:    [[TMP2255:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2255]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2256:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2257:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3391:%.*]] = icmp ult i32 [[TMP2256]], [[TMP2257]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3391]], label [[IF_THEN3393:%.*]], label [[IF_END3394:%.*]]
+// SIMD-ONLY0:       if.then3393:
+// SIMD-ONLY0-NEXT:    [[TMP2258:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2258]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3394]]
+// SIMD-ONLY0:       if.end3394:
+// SIMD-ONLY0-NEXT:    [[TMP2259:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2259]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2260:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2261:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3395:%.*]] = icmp eq i32 [[TMP2260]], [[TMP2261]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3395]], label [[IF_THEN3397:%.*]], label [[IF_END3398:%.*]]
+// SIMD-ONLY0:       if.then3397:
+// SIMD-ONLY0-NEXT:    [[TMP2262:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2262]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3398]]
+// SIMD-ONLY0:       if.end3398:
+// SIMD-ONLY0-NEXT:    [[TMP2263:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2263]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2264:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2265:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3399:%.*]] = icmp eq i32 [[TMP2264]], [[TMP2265]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3399]], label [[IF_THEN3401:%.*]], label [[IF_END3402:%.*]]
+// SIMD-ONLY0:       if.then3401:
+// SIMD-ONLY0-NEXT:    [[TMP2266:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2266]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3402]]
+// SIMD-ONLY0:       if.end3402:
+// SIMD-ONLY0-NEXT:    [[TMP2267:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2267]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2268:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2269:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3403:%.*]] = icmp eq i32 [[TMP2268]], [[TMP2269]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3403]], label [[IF_THEN3405:%.*]], label [[IF_ELSE3406:%.*]]
+// SIMD-ONLY0:       if.then3405:
+// SIMD-ONLY0-NEXT:    [[TMP2270:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2270]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3407:%.*]]
+// SIMD-ONLY0:       if.else3406:
+// SIMD-ONLY0-NEXT:    [[TMP2271:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2271]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3407]]
+// SIMD-ONLY0:       if.end3407:
+// SIMD-ONLY0-NEXT:    [[TMP2272:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2273:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3408:%.*]] = icmp eq i32 [[TMP2272]], [[TMP2273]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3408]], label [[IF_THEN3410:%.*]], label [[IF_ELSE3411:%.*]]
+// SIMD-ONLY0:       if.then3410:
+// SIMD-ONLY0-NEXT:    [[TMP2274:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2274]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3412:%.*]]
+// SIMD-ONLY0:       if.else3411:
+// SIMD-ONLY0-NEXT:    [[TMP2275:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2275]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3412]]
+// SIMD-ONLY0:       if.end3412:
+// SIMD-ONLY0-NEXT:    [[TMP2276:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2277:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3413:%.*]] = icmp eq i32 [[TMP2276]], [[TMP2277]]
+// SIMD-ONLY0-NEXT:    [[CONV3414:%.*]] = zext i1 [[CMP3413]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3414]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2278:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3415:%.*]] = icmp ne i32 [[TMP2278]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3415]], label [[IF_THEN3416:%.*]], label [[IF_END3417:%.*]]
+// SIMD-ONLY0:       if.then3416:
+// SIMD-ONLY0-NEXT:    [[TMP2279:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2279]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3417]]
+// SIMD-ONLY0:       if.end3417:
+// SIMD-ONLY0-NEXT:    [[TMP2280:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2281:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3418:%.*]] = icmp eq i32 [[TMP2280]], [[TMP2281]]
+// SIMD-ONLY0-NEXT:    [[CONV3419:%.*]] = zext i1 [[CMP3418]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3419]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2282:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3420:%.*]] = icmp ne i32 [[TMP2282]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3420]], label [[IF_THEN3421:%.*]], label [[IF_END3422:%.*]]
+// SIMD-ONLY0:       if.then3421:
+// SIMD-ONLY0-NEXT:    [[TMP2283:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2283]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3422]]
+// SIMD-ONLY0:       if.end3422:
+// SIMD-ONLY0-NEXT:    [[TMP2284:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2285:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3423:%.*]] = icmp eq i32 [[TMP2284]], [[TMP2285]]
+// SIMD-ONLY0-NEXT:    [[CONV3424:%.*]] = zext i1 [[CMP3423]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3424]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2286:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3425:%.*]] = icmp ne i32 [[TMP2286]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3425]], label [[IF_THEN3426:%.*]], label [[IF_ELSE3427:%.*]]
+// SIMD-ONLY0:       if.then3426:
+// SIMD-ONLY0-NEXT:    [[TMP2287:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2287]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3428:%.*]]
+// SIMD-ONLY0:       if.else3427:
+// SIMD-ONLY0-NEXT:    [[TMP2288:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2288]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3428]]
+// SIMD-ONLY0:       if.end3428:
+// SIMD-ONLY0-NEXT:    [[TMP2289:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2290:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3429:%.*]] = icmp eq i32 [[TMP2289]], [[TMP2290]]
+// SIMD-ONLY0-NEXT:    [[CONV3430:%.*]] = zext i1 [[CMP3429]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3430]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2291:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3431:%.*]] = icmp ne i32 [[TMP2291]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3431]], label [[IF_THEN3432:%.*]], label [[IF_ELSE3433:%.*]]
+// SIMD-ONLY0:       if.then3432:
+// SIMD-ONLY0-NEXT:    [[TMP2292:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2292]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3434:%.*]]
+// SIMD-ONLY0:       if.else3433:
+// SIMD-ONLY0-NEXT:    [[TMP2293:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2293]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3434]]
+// SIMD-ONLY0:       if.end3434:
+// SIMD-ONLY0-NEXT:    [[TMP2294:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2294]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2295:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2296:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3435:%.*]] = icmp ugt i32 [[TMP2295]], [[TMP2296]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3435]], label [[IF_THEN3437:%.*]], label [[IF_END3438:%.*]]
+// SIMD-ONLY0:       if.then3437:
+// SIMD-ONLY0-NEXT:    [[TMP2297:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2297]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3438]]
+// SIMD-ONLY0:       if.end3438:
+// SIMD-ONLY0-NEXT:    [[TMP2298:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2298]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2299:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2300:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3439:%.*]] = icmp ugt i32 [[TMP2299]], [[TMP2300]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3439]], label [[IF_THEN3441:%.*]], label [[IF_END3442:%.*]]
+// SIMD-ONLY0:       if.then3441:
+// SIMD-ONLY0-NEXT:    [[TMP2301:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2301]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3442]]
+// SIMD-ONLY0:       if.end3442:
+// SIMD-ONLY0-NEXT:    [[TMP2302:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2302]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2303:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2304:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3443:%.*]] = icmp ult i32 [[TMP2303]], [[TMP2304]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3443]], label [[IF_THEN3445:%.*]], label [[IF_END3446:%.*]]
+// SIMD-ONLY0:       if.then3445:
+// SIMD-ONLY0-NEXT:    [[TMP2305:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2305]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3446]]
+// SIMD-ONLY0:       if.end3446:
+// SIMD-ONLY0-NEXT:    [[TMP2306:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2306]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2307:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2308:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3447:%.*]] = icmp ult i32 [[TMP2307]], [[TMP2308]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3447]], label [[IF_THEN3449:%.*]], label [[IF_END3450:%.*]]
+// SIMD-ONLY0:       if.then3449:
+// SIMD-ONLY0-NEXT:    [[TMP2309:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2309]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3450]]
+// SIMD-ONLY0:       if.end3450:
+// SIMD-ONLY0-NEXT:    [[TMP2310:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2310]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2311:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2312:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3451:%.*]] = icmp eq i32 [[TMP2311]], [[TMP2312]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3451]], label [[IF_THEN3453:%.*]], label [[IF_END3454:%.*]]
+// SIMD-ONLY0:       if.then3453:
+// SIMD-ONLY0-NEXT:    [[TMP2313:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2313]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3454]]
+// SIMD-ONLY0:       if.end3454:
+// SIMD-ONLY0-NEXT:    [[TMP2314:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2314]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2315:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2316:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3455:%.*]] = icmp eq i32 [[TMP2315]], [[TMP2316]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3455]], label [[IF_THEN3457:%.*]], label [[IF_END3458:%.*]]
+// SIMD-ONLY0:       if.then3457:
+// SIMD-ONLY0-NEXT:    [[TMP2317:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2317]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3458]]
+// SIMD-ONLY0:       if.end3458:
+// SIMD-ONLY0-NEXT:    [[TMP2318:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2319:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3459:%.*]] = icmp ugt i32 [[TMP2318]], [[TMP2319]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3459]], label [[IF_THEN3461:%.*]], label [[IF_END3462:%.*]]
+// SIMD-ONLY0:       if.then3461:
+// SIMD-ONLY0-NEXT:    [[TMP2320:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2320]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3462]]
+// SIMD-ONLY0:       if.end3462:
+// SIMD-ONLY0-NEXT:    [[TMP2321:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2321]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2322:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2323:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3463:%.*]] = icmp ugt i32 [[TMP2322]], [[TMP2323]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3463]], label [[IF_THEN3465:%.*]], label [[IF_END3466:%.*]]
+// SIMD-ONLY0:       if.then3465:
+// SIMD-ONLY0-NEXT:    [[TMP2324:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2324]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3466]]
+// SIMD-ONLY0:       if.end3466:
+// SIMD-ONLY0-NEXT:    [[TMP2325:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2325]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2326:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2327:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3467:%.*]] = icmp ult i32 [[TMP2326]], [[TMP2327]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3467]], label [[IF_THEN3469:%.*]], label [[IF_END3470:%.*]]
+// SIMD-ONLY0:       if.then3469:
+// SIMD-ONLY0-NEXT:    [[TMP2328:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2328]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3470]]
+// SIMD-ONLY0:       if.end3470:
+// SIMD-ONLY0-NEXT:    [[TMP2329:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2329]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2330:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2331:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3471:%.*]] = icmp ult i32 [[TMP2330]], [[TMP2331]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3471]], label [[IF_THEN3473:%.*]], label [[IF_END3474:%.*]]
+// SIMD-ONLY0:       if.then3473:
+// SIMD-ONLY0-NEXT:    [[TMP2332:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2332]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3474]]
+// SIMD-ONLY0:       if.end3474:
+// SIMD-ONLY0-NEXT:    [[TMP2333:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2333]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2334:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2335:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3475:%.*]] = icmp eq i32 [[TMP2334]], [[TMP2335]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3475]], label [[IF_THEN3477:%.*]], label [[IF_END3478:%.*]]
+// SIMD-ONLY0:       if.then3477:
+// SIMD-ONLY0-NEXT:    [[TMP2336:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2336]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3478]]
+// SIMD-ONLY0:       if.end3478:
+// SIMD-ONLY0-NEXT:    [[TMP2337:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2337]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2338:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2339:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3479:%.*]] = icmp eq i32 [[TMP2338]], [[TMP2339]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3479]], label [[IF_THEN3481:%.*]], label [[IF_END3482:%.*]]
+// SIMD-ONLY0:       if.then3481:
+// SIMD-ONLY0-NEXT:    [[TMP2340:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2340]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3482]]
+// SIMD-ONLY0:       if.end3482:
+// SIMD-ONLY0-NEXT:    [[TMP2341:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2341]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2342:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2343:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3483:%.*]] = icmp eq i32 [[TMP2342]], [[TMP2343]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3483]], label [[IF_THEN3485:%.*]], label [[IF_ELSE3486:%.*]]
+// SIMD-ONLY0:       if.then3485:
+// SIMD-ONLY0-NEXT:    [[TMP2344:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2344]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3487:%.*]]
+// SIMD-ONLY0:       if.else3486:
+// SIMD-ONLY0-NEXT:    [[TMP2345:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2345]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3487]]
+// SIMD-ONLY0:       if.end3487:
+// SIMD-ONLY0-NEXT:    [[TMP2346:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2347:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3488:%.*]] = icmp eq i32 [[TMP2346]], [[TMP2347]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3488]], label [[IF_THEN3490:%.*]], label [[IF_ELSE3491:%.*]]
+// SIMD-ONLY0:       if.then3490:
+// SIMD-ONLY0-NEXT:    [[TMP2348:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2348]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3492:%.*]]
+// SIMD-ONLY0:       if.else3491:
+// SIMD-ONLY0-NEXT:    [[TMP2349:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2349]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3492]]
+// SIMD-ONLY0:       if.end3492:
+// SIMD-ONLY0-NEXT:    [[TMP2350:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2351:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3493:%.*]] = icmp eq i32 [[TMP2350]], [[TMP2351]]
+// SIMD-ONLY0-NEXT:    [[CONV3494:%.*]] = zext i1 [[CMP3493]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3494]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2352:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3495:%.*]] = icmp ne i32 [[TMP2352]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3495]], label [[IF_THEN3496:%.*]], label [[IF_END3497:%.*]]
+// SIMD-ONLY0:       if.then3496:
+// SIMD-ONLY0-NEXT:    [[TMP2353:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2353]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3497]]
+// SIMD-ONLY0:       if.end3497:
+// SIMD-ONLY0-NEXT:    [[TMP2354:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2355:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3498:%.*]] = icmp eq i32 [[TMP2354]], [[TMP2355]]
+// SIMD-ONLY0-NEXT:    [[CONV3499:%.*]] = zext i1 [[CMP3498]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3499]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2356:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3500:%.*]] = icmp ne i32 [[TMP2356]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3500]], label [[IF_THEN3501:%.*]], label [[IF_END3502:%.*]]
+// SIMD-ONLY0:       if.then3501:
+// SIMD-ONLY0-NEXT:    [[TMP2357:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2357]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3502]]
+// SIMD-ONLY0:       if.end3502:
+// SIMD-ONLY0-NEXT:    [[TMP2358:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2359:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3503:%.*]] = icmp eq i32 [[TMP2358]], [[TMP2359]]
+// SIMD-ONLY0-NEXT:    [[CONV3504:%.*]] = zext i1 [[CMP3503]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3504]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2360:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3505:%.*]] = icmp ne i32 [[TMP2360]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3505]], label [[IF_THEN3506:%.*]], label [[IF_ELSE3507:%.*]]
+// SIMD-ONLY0:       if.then3506:
+// SIMD-ONLY0-NEXT:    [[TMP2361:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2361]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3508:%.*]]
+// SIMD-ONLY0:       if.else3507:
+// SIMD-ONLY0-NEXT:    [[TMP2362:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2362]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3508]]
+// SIMD-ONLY0:       if.end3508:
+// SIMD-ONLY0-NEXT:    [[TMP2363:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2364:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3509:%.*]] = icmp eq i32 [[TMP2363]], [[TMP2364]]
+// SIMD-ONLY0-NEXT:    [[CONV3510:%.*]] = zext i1 [[CMP3509]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3510]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2365:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3511:%.*]] = icmp ne i32 [[TMP2365]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3511]], label [[IF_THEN3512:%.*]], label [[IF_ELSE3513:%.*]]
+// SIMD-ONLY0:       if.then3512:
+// SIMD-ONLY0-NEXT:    [[TMP2366:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2366]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3514:%.*]]
+// SIMD-ONLY0:       if.else3513:
+// SIMD-ONLY0-NEXT:    [[TMP2367:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2367]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3514]]
+// SIMD-ONLY0:       if.end3514:
+// SIMD-ONLY0-NEXT:    [[TMP2368:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2368]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2369:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2370:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3515:%.*]] = icmp ugt i32 [[TMP2369]], [[TMP2370]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3515]], label [[IF_THEN3517:%.*]], label [[IF_END3518:%.*]]
+// SIMD-ONLY0:       if.then3517:
+// SIMD-ONLY0-NEXT:    [[TMP2371:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2371]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3518]]
+// SIMD-ONLY0:       if.end3518:
+// SIMD-ONLY0-NEXT:    [[TMP2372:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2372]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2373:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2374:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3519:%.*]] = icmp ugt i32 [[TMP2373]], [[TMP2374]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3519]], label [[IF_THEN3521:%.*]], label [[IF_END3522:%.*]]
+// SIMD-ONLY0:       if.then3521:
+// SIMD-ONLY0-NEXT:    [[TMP2375:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2375]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3522]]
+// SIMD-ONLY0:       if.end3522:
+// SIMD-ONLY0-NEXT:    [[TMP2376:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2376]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2377:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2378:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3523:%.*]] = icmp ult i32 [[TMP2377]], [[TMP2378]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3523]], label [[IF_THEN3525:%.*]], label [[IF_END3526:%.*]]
+// SIMD-ONLY0:       if.then3525:
+// SIMD-ONLY0-NEXT:    [[TMP2379:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2379]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3526]]
+// SIMD-ONLY0:       if.end3526:
+// SIMD-ONLY0-NEXT:    [[TMP2380:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2380]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2381:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2382:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3527:%.*]] = icmp ult i32 [[TMP2381]], [[TMP2382]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3527]], label [[IF_THEN3529:%.*]], label [[IF_END3530:%.*]]
+// SIMD-ONLY0:       if.then3529:
+// SIMD-ONLY0-NEXT:    [[TMP2383:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2383]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3530]]
+// SIMD-ONLY0:       if.end3530:
+// SIMD-ONLY0-NEXT:    [[TMP2384:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2384]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2385:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2386:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3531:%.*]] = icmp eq i32 [[TMP2385]], [[TMP2386]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3531]], label [[IF_THEN3533:%.*]], label [[IF_END3534:%.*]]
+// SIMD-ONLY0:       if.then3533:
+// SIMD-ONLY0-NEXT:    [[TMP2387:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2387]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3534]]
+// SIMD-ONLY0:       if.end3534:
+// SIMD-ONLY0-NEXT:    [[TMP2388:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2388]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2389:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2390:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3535:%.*]] = icmp eq i32 [[TMP2389]], [[TMP2390]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3535]], label [[IF_THEN3537:%.*]], label [[IF_END3538:%.*]]
+// SIMD-ONLY0:       if.then3537:
+// SIMD-ONLY0-NEXT:    [[TMP2391:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2391]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3538]]
+// SIMD-ONLY0:       if.end3538:
+// SIMD-ONLY0-NEXT:    [[TMP2392:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2393:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3539:%.*]] = icmp ugt i32 [[TMP2392]], [[TMP2393]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3539]], label [[IF_THEN3541:%.*]], label [[IF_END3542:%.*]]
+// SIMD-ONLY0:       if.then3541:
+// SIMD-ONLY0-NEXT:    [[TMP2394:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2394]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3542]]
+// SIMD-ONLY0:       if.end3542:
+// SIMD-ONLY0-NEXT:    [[TMP2395:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2395]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2396:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2397:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3543:%.*]] = icmp ugt i32 [[TMP2396]], [[TMP2397]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3543]], label [[IF_THEN3545:%.*]], label [[IF_END3546:%.*]]
+// SIMD-ONLY0:       if.then3545:
+// SIMD-ONLY0-NEXT:    [[TMP2398:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2398]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3546]]
+// SIMD-ONLY0:       if.end3546:
+// SIMD-ONLY0-NEXT:    [[TMP2399:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2399]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2400:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2401:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3547:%.*]] = icmp ult i32 [[TMP2400]], [[TMP2401]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3547]], label [[IF_THEN3549:%.*]], label [[IF_END3550:%.*]]
+// SIMD-ONLY0:       if.then3549:
+// SIMD-ONLY0-NEXT:    [[TMP2402:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2402]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3550]]
+// SIMD-ONLY0:       if.end3550:
+// SIMD-ONLY0-NEXT:    [[TMP2403:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2403]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2404:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2405:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3551:%.*]] = icmp ult i32 [[TMP2404]], [[TMP2405]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3551]], label [[IF_THEN3553:%.*]], label [[IF_END3554:%.*]]
+// SIMD-ONLY0:       if.then3553:
+// SIMD-ONLY0-NEXT:    [[TMP2406:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2406]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3554]]
+// SIMD-ONLY0:       if.end3554:
+// SIMD-ONLY0-NEXT:    [[TMP2407:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2407]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2408:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2409:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3555:%.*]] = icmp eq i32 [[TMP2408]], [[TMP2409]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3555]], label [[IF_THEN3557:%.*]], label [[IF_END3558:%.*]]
+// SIMD-ONLY0:       if.then3557:
+// SIMD-ONLY0-NEXT:    [[TMP2410:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2410]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3558]]
+// SIMD-ONLY0:       if.end3558:
+// SIMD-ONLY0-NEXT:    [[TMP2411:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2411]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2412:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2413:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3559:%.*]] = icmp eq i32 [[TMP2412]], [[TMP2413]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3559]], label [[IF_THEN3561:%.*]], label [[IF_END3562:%.*]]
+// SIMD-ONLY0:       if.then3561:
+// SIMD-ONLY0-NEXT:    [[TMP2414:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2414]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3562]]
+// SIMD-ONLY0:       if.end3562:
+// SIMD-ONLY0-NEXT:    [[TMP2415:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2415]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2416:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2417:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3563:%.*]] = icmp eq i32 [[TMP2416]], [[TMP2417]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3563]], label [[IF_THEN3565:%.*]], label [[IF_ELSE3566:%.*]]
+// SIMD-ONLY0:       if.then3565:
+// SIMD-ONLY0-NEXT:    [[TMP2418:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2418]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3567:%.*]]
+// SIMD-ONLY0:       if.else3566:
+// SIMD-ONLY0-NEXT:    [[TMP2419:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2419]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3567]]
+// SIMD-ONLY0:       if.end3567:
+// SIMD-ONLY0-NEXT:    [[TMP2420:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2421:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3568:%.*]] = icmp eq i32 [[TMP2420]], [[TMP2421]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3568]], label [[IF_THEN3570:%.*]], label [[IF_ELSE3571:%.*]]
+// SIMD-ONLY0:       if.then3570:
+// SIMD-ONLY0-NEXT:    [[TMP2422:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2422]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3572:%.*]]
+// SIMD-ONLY0:       if.else3571:
+// SIMD-ONLY0-NEXT:    [[TMP2423:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2423]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3572]]
+// SIMD-ONLY0:       if.end3572:
+// SIMD-ONLY0-NEXT:    [[TMP2424:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2425:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3573:%.*]] = icmp eq i32 [[TMP2424]], [[TMP2425]]
+// SIMD-ONLY0-NEXT:    [[CONV3574:%.*]] = zext i1 [[CMP3573]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3574]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2426:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3575:%.*]] = icmp ne i32 [[TMP2426]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3575]], label [[IF_THEN3576:%.*]], label [[IF_END3577:%.*]]
+// SIMD-ONLY0:       if.then3576:
+// SIMD-ONLY0-NEXT:    [[TMP2427:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2427]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3577]]
+// SIMD-ONLY0:       if.end3577:
+// SIMD-ONLY0-NEXT:    [[TMP2428:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2429:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3578:%.*]] = icmp eq i32 [[TMP2428]], [[TMP2429]]
+// SIMD-ONLY0-NEXT:    [[CONV3579:%.*]] = zext i1 [[CMP3578]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3579]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2430:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3580:%.*]] = icmp ne i32 [[TMP2430]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3580]], label [[IF_THEN3581:%.*]], label [[IF_END3582:%.*]]
+// SIMD-ONLY0:       if.then3581:
+// SIMD-ONLY0-NEXT:    [[TMP2431:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2431]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3582]]
+// SIMD-ONLY0:       if.end3582:
+// SIMD-ONLY0-NEXT:    [[TMP2432:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2433:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3583:%.*]] = icmp eq i32 [[TMP2432]], [[TMP2433]]
+// SIMD-ONLY0-NEXT:    [[CONV3584:%.*]] = zext i1 [[CMP3583]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3584]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2434:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3585:%.*]] = icmp ne i32 [[TMP2434]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3585]], label [[IF_THEN3586:%.*]], label [[IF_ELSE3587:%.*]]
+// SIMD-ONLY0:       if.then3586:
+// SIMD-ONLY0-NEXT:    [[TMP2435:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2435]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3588:%.*]]
+// SIMD-ONLY0:       if.else3587:
+// SIMD-ONLY0-NEXT:    [[TMP2436:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2436]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3588]]
+// SIMD-ONLY0:       if.end3588:
+// SIMD-ONLY0-NEXT:    [[TMP2437:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2438:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3589:%.*]] = icmp eq i32 [[TMP2437]], [[TMP2438]]
+// SIMD-ONLY0-NEXT:    [[CONV3590:%.*]] = zext i1 [[CMP3589]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3590]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2439:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3591:%.*]] = icmp ne i32 [[TMP2439]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3591]], label [[IF_THEN3592:%.*]], label [[IF_ELSE3593:%.*]]
+// SIMD-ONLY0:       if.then3592:
+// SIMD-ONLY0-NEXT:    [[TMP2440:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2440]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3594:%.*]]
+// SIMD-ONLY0:       if.else3593:
+// SIMD-ONLY0-NEXT:    [[TMP2441:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2441]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3594]]
+// SIMD-ONLY0:       if.end3594:
+// SIMD-ONLY0-NEXT:    [[TMP2442:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2442]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2443:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2444:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3595:%.*]] = icmp ugt i32 [[TMP2443]], [[TMP2444]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3595]], label [[IF_THEN3597:%.*]], label [[IF_END3598:%.*]]
+// SIMD-ONLY0:       if.then3597:
+// SIMD-ONLY0-NEXT:    [[TMP2445:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2445]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3598]]
+// SIMD-ONLY0:       if.end3598:
+// SIMD-ONLY0-NEXT:    [[TMP2446:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2446]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2447:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2448:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3599:%.*]] = icmp ugt i32 [[TMP2447]], [[TMP2448]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3599]], label [[IF_THEN3601:%.*]], label [[IF_END3602:%.*]]
+// SIMD-ONLY0:       if.then3601:
+// SIMD-ONLY0-NEXT:    [[TMP2449:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2449]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3602]]
+// SIMD-ONLY0:       if.end3602:
+// SIMD-ONLY0-NEXT:    [[TMP2450:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2450]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2451:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2452:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3603:%.*]] = icmp ult i32 [[TMP2451]], [[TMP2452]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3603]], label [[IF_THEN3605:%.*]], label [[IF_END3606:%.*]]
+// SIMD-ONLY0:       if.then3605:
+// SIMD-ONLY0-NEXT:    [[TMP2453:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2453]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3606]]
+// SIMD-ONLY0:       if.end3606:
+// SIMD-ONLY0-NEXT:    [[TMP2454:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2454]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2455:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2456:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3607:%.*]] = icmp ult i32 [[TMP2455]], [[TMP2456]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3607]], label [[IF_THEN3609:%.*]], label [[IF_END3610:%.*]]
+// SIMD-ONLY0:       if.then3609:
+// SIMD-ONLY0-NEXT:    [[TMP2457:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2457]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3610]]
+// SIMD-ONLY0:       if.end3610:
+// SIMD-ONLY0-NEXT:    [[TMP2458:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2458]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2459:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2460:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3611:%.*]] = icmp eq i32 [[TMP2459]], [[TMP2460]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3611]], label [[IF_THEN3613:%.*]], label [[IF_END3614:%.*]]
+// SIMD-ONLY0:       if.then3613:
+// SIMD-ONLY0-NEXT:    [[TMP2461:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2461]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3614]]
+// SIMD-ONLY0:       if.end3614:
+// SIMD-ONLY0-NEXT:    [[TMP2462:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2462]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2463:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2464:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3615:%.*]] = icmp eq i32 [[TMP2463]], [[TMP2464]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3615]], label [[IF_THEN3617:%.*]], label [[IF_END3618:%.*]]
+// SIMD-ONLY0:       if.then3617:
+// SIMD-ONLY0-NEXT:    [[TMP2465:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2465]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3618]]
+// SIMD-ONLY0:       if.end3618:
+// SIMD-ONLY0-NEXT:    [[TMP2466:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2467:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3619:%.*]] = icmp ugt i32 [[TMP2466]], [[TMP2467]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3619]], label [[IF_THEN3621:%.*]], label [[IF_END3622:%.*]]
+// SIMD-ONLY0:       if.then3621:
+// SIMD-ONLY0-NEXT:    [[TMP2468:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2468]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3622]]
+// SIMD-ONLY0:       if.end3622:
+// SIMD-ONLY0-NEXT:    [[TMP2469:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2469]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2470:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2471:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3623:%.*]] = icmp ugt i32 [[TMP2470]], [[TMP2471]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3623]], label [[IF_THEN3625:%.*]], label [[IF_END3626:%.*]]
+// SIMD-ONLY0:       if.then3625:
+// SIMD-ONLY0-NEXT:    [[TMP2472:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2472]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3626]]
+// SIMD-ONLY0:       if.end3626:
+// SIMD-ONLY0-NEXT:    [[TMP2473:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2473]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2474:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2475:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3627:%.*]] = icmp ult i32 [[TMP2474]], [[TMP2475]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3627]], label [[IF_THEN3629:%.*]], label [[IF_END3630:%.*]]
+// SIMD-ONLY0:       if.then3629:
+// SIMD-ONLY0-NEXT:    [[TMP2476:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2476]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3630]]
+// SIMD-ONLY0:       if.end3630:
+// SIMD-ONLY0-NEXT:    [[TMP2477:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2477]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2478:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2479:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3631:%.*]] = icmp ult i32 [[TMP2478]], [[TMP2479]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3631]], label [[IF_THEN3633:%.*]], label [[IF_END3634:%.*]]
+// SIMD-ONLY0:       if.then3633:
+// SIMD-ONLY0-NEXT:    [[TMP2480:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2480]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3634]]
+// SIMD-ONLY0:       if.end3634:
+// SIMD-ONLY0-NEXT:    [[TMP2481:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2481]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2482:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2483:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3635:%.*]] = icmp eq i32 [[TMP2482]], [[TMP2483]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3635]], label [[IF_THEN3637:%.*]], label [[IF_END3638:%.*]]
+// SIMD-ONLY0:       if.then3637:
+// SIMD-ONLY0-NEXT:    [[TMP2484:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2484]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3638]]
+// SIMD-ONLY0:       if.end3638:
+// SIMD-ONLY0-NEXT:    [[TMP2485:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2485]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2486:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2487:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3639:%.*]] = icmp eq i32 [[TMP2486]], [[TMP2487]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3639]], label [[IF_THEN3641:%.*]], label [[IF_END3642:%.*]]
+// SIMD-ONLY0:       if.then3641:
+// SIMD-ONLY0-NEXT:    [[TMP2488:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2488]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3642]]
+// SIMD-ONLY0:       if.end3642:
+// SIMD-ONLY0-NEXT:    [[TMP2489:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2489]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2490:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2491:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3643:%.*]] = icmp eq i32 [[TMP2490]], [[TMP2491]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3643]], label [[IF_THEN3645:%.*]], label [[IF_ELSE3646:%.*]]
+// SIMD-ONLY0:       if.then3645:
+// SIMD-ONLY0-NEXT:    [[TMP2492:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2492]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3647:%.*]]
+// SIMD-ONLY0:       if.else3646:
+// SIMD-ONLY0-NEXT:    [[TMP2493:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2493]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3647]]
+// SIMD-ONLY0:       if.end3647:
+// SIMD-ONLY0-NEXT:    [[TMP2494:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2495:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3648:%.*]] = icmp eq i32 [[TMP2494]], [[TMP2495]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3648]], label [[IF_THEN3650:%.*]], label [[IF_ELSE3651:%.*]]
+// SIMD-ONLY0:       if.then3650:
+// SIMD-ONLY0-NEXT:    [[TMP2496:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2496]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3652:%.*]]
+// SIMD-ONLY0:       if.else3651:
+// SIMD-ONLY0-NEXT:    [[TMP2497:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2497]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3652]]
+// SIMD-ONLY0:       if.end3652:
+// SIMD-ONLY0-NEXT:    [[TMP2498:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2499:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3653:%.*]] = icmp eq i32 [[TMP2498]], [[TMP2499]]
+// SIMD-ONLY0-NEXT:    [[CONV3654:%.*]] = zext i1 [[CMP3653]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3654]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2500:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3655:%.*]] = icmp ne i32 [[TMP2500]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3655]], label [[IF_THEN3656:%.*]], label [[IF_END3657:%.*]]
+// SIMD-ONLY0:       if.then3656:
+// SIMD-ONLY0-NEXT:    [[TMP2501:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2501]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3657]]
+// SIMD-ONLY0:       if.end3657:
+// SIMD-ONLY0-NEXT:    [[TMP2502:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2503:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3658:%.*]] = icmp eq i32 [[TMP2502]], [[TMP2503]]
+// SIMD-ONLY0-NEXT:    [[CONV3659:%.*]] = zext i1 [[CMP3658]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3659]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2504:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3660:%.*]] = icmp ne i32 [[TMP2504]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3660]], label [[IF_THEN3661:%.*]], label [[IF_END3662:%.*]]
+// SIMD-ONLY0:       if.then3661:
+// SIMD-ONLY0-NEXT:    [[TMP2505:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2505]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3662]]
+// SIMD-ONLY0:       if.end3662:
+// SIMD-ONLY0-NEXT:    [[TMP2506:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2507:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3663:%.*]] = icmp eq i32 [[TMP2506]], [[TMP2507]]
+// SIMD-ONLY0-NEXT:    [[CONV3664:%.*]] = zext i1 [[CMP3663]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3664]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2508:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3665:%.*]] = icmp ne i32 [[TMP2508]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3665]], label [[IF_THEN3666:%.*]], label [[IF_ELSE3667:%.*]]
+// SIMD-ONLY0:       if.then3666:
+// SIMD-ONLY0-NEXT:    [[TMP2509:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2509]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3668:%.*]]
+// SIMD-ONLY0:       if.else3667:
+// SIMD-ONLY0-NEXT:    [[TMP2510:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2510]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3668]]
+// SIMD-ONLY0:       if.end3668:
+// SIMD-ONLY0-NEXT:    [[TMP2511:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2512:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3669:%.*]] = icmp eq i32 [[TMP2511]], [[TMP2512]]
+// SIMD-ONLY0-NEXT:    [[CONV3670:%.*]] = zext i1 [[CMP3669]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3670]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2513:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3671:%.*]] = icmp ne i32 [[TMP2513]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3671]], label [[IF_THEN3672:%.*]], label [[IF_ELSE3673:%.*]]
+// SIMD-ONLY0:       if.then3672:
+// SIMD-ONLY0-NEXT:    [[TMP2514:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2514]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3674:%.*]]
+// SIMD-ONLY0:       if.else3673:
+// SIMD-ONLY0-NEXT:    [[TMP2515:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2515]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3674]]
+// SIMD-ONLY0:       if.end3674:
+// SIMD-ONLY0-NEXT:    [[TMP2516:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2516]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2517:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2518:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3675:%.*]] = icmp ugt i32 [[TMP2517]], [[TMP2518]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3675]], label [[IF_THEN3677:%.*]], label [[IF_END3678:%.*]]
+// SIMD-ONLY0:       if.then3677:
+// SIMD-ONLY0-NEXT:    [[TMP2519:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2519]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3678]]
+// SIMD-ONLY0:       if.end3678:
+// SIMD-ONLY0-NEXT:    [[TMP2520:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2520]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2521:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2522:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3679:%.*]] = icmp ugt i32 [[TMP2521]], [[TMP2522]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3679]], label [[IF_THEN3681:%.*]], label [[IF_END3682:%.*]]
+// SIMD-ONLY0:       if.then3681:
+// SIMD-ONLY0-NEXT:    [[TMP2523:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2523]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3682]]
+// SIMD-ONLY0:       if.end3682:
+// SIMD-ONLY0-NEXT:    [[TMP2524:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2524]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2525:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2526:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3683:%.*]] = icmp ult i32 [[TMP2525]], [[TMP2526]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3683]], label [[IF_THEN3685:%.*]], label [[IF_END3686:%.*]]
+// SIMD-ONLY0:       if.then3685:
+// SIMD-ONLY0-NEXT:    [[TMP2527:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2527]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3686]]
+// SIMD-ONLY0:       if.end3686:
+// SIMD-ONLY0-NEXT:    [[TMP2528:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2528]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2529:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2530:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3687:%.*]] = icmp ult i32 [[TMP2529]], [[TMP2530]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3687]], label [[IF_THEN3689:%.*]], label [[IF_END3690:%.*]]
+// SIMD-ONLY0:       if.then3689:
+// SIMD-ONLY0-NEXT:    [[TMP2531:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2531]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3690]]
+// SIMD-ONLY0:       if.end3690:
+// SIMD-ONLY0-NEXT:    [[TMP2532:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2532]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2533:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2534:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3691:%.*]] = icmp eq i32 [[TMP2533]], [[TMP2534]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3691]], label [[IF_THEN3693:%.*]], label [[IF_END3694:%.*]]
+// SIMD-ONLY0:       if.then3693:
+// SIMD-ONLY0-NEXT:    [[TMP2535:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2535]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3694]]
+// SIMD-ONLY0:       if.end3694:
+// SIMD-ONLY0-NEXT:    [[TMP2536:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2536]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2537:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2538:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3695:%.*]] = icmp eq i32 [[TMP2537]], [[TMP2538]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3695]], label [[IF_THEN3697:%.*]], label [[IF_END3698:%.*]]
+// SIMD-ONLY0:       if.then3697:
+// SIMD-ONLY0-NEXT:    [[TMP2539:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2539]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3698]]
+// SIMD-ONLY0:       if.end3698:
+// SIMD-ONLY0-NEXT:    [[TMP2540:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2541:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3699:%.*]] = icmp ugt i32 [[TMP2540]], [[TMP2541]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3699]], label [[IF_THEN3701:%.*]], label [[IF_END3702:%.*]]
+// SIMD-ONLY0:       if.then3701:
+// SIMD-ONLY0-NEXT:    [[TMP2542:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2542]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3702]]
+// SIMD-ONLY0:       if.end3702:
+// SIMD-ONLY0-NEXT:    [[TMP2543:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2543]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2544:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2545:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3703:%.*]] = icmp ugt i32 [[TMP2544]], [[TMP2545]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3703]], label [[IF_THEN3705:%.*]], label [[IF_END3706:%.*]]
+// SIMD-ONLY0:       if.then3705:
+// SIMD-ONLY0-NEXT:    [[TMP2546:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2546]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3706]]
+// SIMD-ONLY0:       if.end3706:
+// SIMD-ONLY0-NEXT:    [[TMP2547:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2547]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2548:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2549:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3707:%.*]] = icmp ult i32 [[TMP2548]], [[TMP2549]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3707]], label [[IF_THEN3709:%.*]], label [[IF_END3710:%.*]]
+// SIMD-ONLY0:       if.then3709:
+// SIMD-ONLY0-NEXT:    [[TMP2550:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2550]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3710]]
+// SIMD-ONLY0:       if.end3710:
+// SIMD-ONLY0-NEXT:    [[TMP2551:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2551]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2552:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2553:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3711:%.*]] = icmp ult i32 [[TMP2552]], [[TMP2553]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3711]], label [[IF_THEN3713:%.*]], label [[IF_END3714:%.*]]
+// SIMD-ONLY0:       if.then3713:
+// SIMD-ONLY0-NEXT:    [[TMP2554:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2554]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3714]]
+// SIMD-ONLY0:       if.end3714:
+// SIMD-ONLY0-NEXT:    [[TMP2555:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2555]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2556:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2557:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3715:%.*]] = icmp eq i32 [[TMP2556]], [[TMP2557]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3715]], label [[IF_THEN3717:%.*]], label [[IF_END3718:%.*]]
+// SIMD-ONLY0:       if.then3717:
+// SIMD-ONLY0-NEXT:    [[TMP2558:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2558]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3718]]
+// SIMD-ONLY0:       if.end3718:
+// SIMD-ONLY0-NEXT:    [[TMP2559:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2559]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2560:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2561:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3719:%.*]] = icmp eq i32 [[TMP2560]], [[TMP2561]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3719]], label [[IF_THEN3721:%.*]], label [[IF_END3722:%.*]]
+// SIMD-ONLY0:       if.then3721:
+// SIMD-ONLY0-NEXT:    [[TMP2562:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2562]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3722]]
+// SIMD-ONLY0:       if.end3722:
+// SIMD-ONLY0-NEXT:    [[TMP2563:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2563]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2564:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2565:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3723:%.*]] = icmp eq i32 [[TMP2564]], [[TMP2565]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3723]], label [[IF_THEN3725:%.*]], label [[IF_ELSE3726:%.*]]
+// SIMD-ONLY0:       if.then3725:
+// SIMD-ONLY0-NEXT:    [[TMP2566:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2566]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3727:%.*]]
+// SIMD-ONLY0:       if.else3726:
+// SIMD-ONLY0-NEXT:    [[TMP2567:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2567]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3727]]
+// SIMD-ONLY0:       if.end3727:
+// SIMD-ONLY0-NEXT:    [[TMP2568:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2569:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3728:%.*]] = icmp eq i32 [[TMP2568]], [[TMP2569]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3728]], label [[IF_THEN3730:%.*]], label [[IF_ELSE3731:%.*]]
+// SIMD-ONLY0:       if.then3730:
+// SIMD-ONLY0-NEXT:    [[TMP2570:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2570]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3732:%.*]]
+// SIMD-ONLY0:       if.else3731:
+// SIMD-ONLY0-NEXT:    [[TMP2571:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2571]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3732]]
+// SIMD-ONLY0:       if.end3732:
+// SIMD-ONLY0-NEXT:    [[TMP2572:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2573:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3733:%.*]] = icmp eq i32 [[TMP2572]], [[TMP2573]]
+// SIMD-ONLY0-NEXT:    [[CONV3734:%.*]] = zext i1 [[CMP3733]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3734]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2574:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3735:%.*]] = icmp ne i32 [[TMP2574]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3735]], label [[IF_THEN3736:%.*]], label [[IF_END3737:%.*]]
+// SIMD-ONLY0:       if.then3736:
+// SIMD-ONLY0-NEXT:    [[TMP2575:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2575]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3737]]
+// SIMD-ONLY0:       if.end3737:
+// SIMD-ONLY0-NEXT:    [[TMP2576:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2577:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3738:%.*]] = icmp eq i32 [[TMP2576]], [[TMP2577]]
+// SIMD-ONLY0-NEXT:    [[CONV3739:%.*]] = zext i1 [[CMP3738]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3739]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2578:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3740:%.*]] = icmp ne i32 [[TMP2578]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3740]], label [[IF_THEN3741:%.*]], label [[IF_END3742:%.*]]
+// SIMD-ONLY0:       if.then3741:
+// SIMD-ONLY0-NEXT:    [[TMP2579:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2579]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3742]]
+// SIMD-ONLY0:       if.end3742:
+// SIMD-ONLY0-NEXT:    [[TMP2580:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2581:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3743:%.*]] = icmp eq i32 [[TMP2580]], [[TMP2581]]
+// SIMD-ONLY0-NEXT:    [[CONV3744:%.*]] = zext i1 [[CMP3743]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3744]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2582:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3745:%.*]] = icmp ne i32 [[TMP2582]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3745]], label [[IF_THEN3746:%.*]], label [[IF_ELSE3747:%.*]]
+// SIMD-ONLY0:       if.then3746:
+// SIMD-ONLY0-NEXT:    [[TMP2583:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2583]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3748:%.*]]
+// SIMD-ONLY0:       if.else3747:
+// SIMD-ONLY0-NEXT:    [[TMP2584:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2584]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3748]]
+// SIMD-ONLY0:       if.end3748:
+// SIMD-ONLY0-NEXT:    [[TMP2585:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2586:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3749:%.*]] = icmp eq i32 [[TMP2585]], [[TMP2586]]
+// SIMD-ONLY0-NEXT:    [[CONV3750:%.*]] = zext i1 [[CMP3749]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3750]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2587:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3751:%.*]] = icmp ne i32 [[TMP2587]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3751]], label [[IF_THEN3752:%.*]], label [[IF_ELSE3753:%.*]]
+// SIMD-ONLY0:       if.then3752:
+// SIMD-ONLY0-NEXT:    [[TMP2588:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2588]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3754:%.*]]
+// SIMD-ONLY0:       if.else3753:
+// SIMD-ONLY0-NEXT:    [[TMP2589:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2589]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3754]]
+// SIMD-ONLY0:       if.end3754:
+// SIMD-ONLY0-NEXT:    [[TMP2590:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2590]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2591:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2592:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3755:%.*]] = icmp ugt i32 [[TMP2591]], [[TMP2592]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3755]], label [[IF_THEN3757:%.*]], label [[IF_END3758:%.*]]
+// SIMD-ONLY0:       if.then3757:
+// SIMD-ONLY0-NEXT:    [[TMP2593:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2593]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3758]]
+// SIMD-ONLY0:       if.end3758:
+// SIMD-ONLY0-NEXT:    [[TMP2594:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2594]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2595:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2596:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3759:%.*]] = icmp ugt i32 [[TMP2595]], [[TMP2596]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3759]], label [[IF_THEN3761:%.*]], label [[IF_END3762:%.*]]
+// SIMD-ONLY0:       if.then3761:
+// SIMD-ONLY0-NEXT:    [[TMP2597:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2597]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3762]]
+// SIMD-ONLY0:       if.end3762:
+// SIMD-ONLY0-NEXT:    [[TMP2598:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2598]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2599:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2600:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3763:%.*]] = icmp ult i32 [[TMP2599]], [[TMP2600]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3763]], label [[IF_THEN3765:%.*]], label [[IF_END3766:%.*]]
+// SIMD-ONLY0:       if.then3765:
+// SIMD-ONLY0-NEXT:    [[TMP2601:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2601]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3766]]
+// SIMD-ONLY0:       if.end3766:
+// SIMD-ONLY0-NEXT:    [[TMP2602:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2602]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2603:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2604:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3767:%.*]] = icmp ult i32 [[TMP2603]], [[TMP2604]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3767]], label [[IF_THEN3769:%.*]], label [[IF_END3770:%.*]]
+// SIMD-ONLY0:       if.then3769:
+// SIMD-ONLY0-NEXT:    [[TMP2605:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2605]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3770]]
+// SIMD-ONLY0:       if.end3770:
+// SIMD-ONLY0-NEXT:    [[TMP2606:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2606]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2607:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2608:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3771:%.*]] = icmp eq i32 [[TMP2607]], [[TMP2608]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3771]], label [[IF_THEN3773:%.*]], label [[IF_END3774:%.*]]
+// SIMD-ONLY0:       if.then3773:
+// SIMD-ONLY0-NEXT:    [[TMP2609:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2609]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3774]]
+// SIMD-ONLY0:       if.end3774:
+// SIMD-ONLY0-NEXT:    [[TMP2610:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2610]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2611:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2612:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3775:%.*]] = icmp eq i32 [[TMP2611]], [[TMP2612]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3775]], label [[IF_THEN3777:%.*]], label [[IF_END3778:%.*]]
+// SIMD-ONLY0:       if.then3777:
+// SIMD-ONLY0-NEXT:    [[TMP2613:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2613]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3778]]
+// SIMD-ONLY0:       if.end3778:
+// SIMD-ONLY0-NEXT:    [[TMP2614:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2615:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3779:%.*]] = icmp ugt i32 [[TMP2614]], [[TMP2615]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3779]], label [[IF_THEN3781:%.*]], label [[IF_END3782:%.*]]
+// SIMD-ONLY0:       if.then3781:
+// SIMD-ONLY0-NEXT:    [[TMP2616:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2616]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3782]]
+// SIMD-ONLY0:       if.end3782:
+// SIMD-ONLY0-NEXT:    [[TMP2617:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2617]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2618:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2619:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3783:%.*]] = icmp ugt i32 [[TMP2618]], [[TMP2619]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3783]], label [[IF_THEN3785:%.*]], label [[IF_END3786:%.*]]
+// SIMD-ONLY0:       if.then3785:
+// SIMD-ONLY0-NEXT:    [[TMP2620:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2620]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3786]]
+// SIMD-ONLY0:       if.end3786:
+// SIMD-ONLY0-NEXT:    [[TMP2621:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2621]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2622:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2623:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3787:%.*]] = icmp ult i32 [[TMP2622]], [[TMP2623]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3787]], label [[IF_THEN3789:%.*]], label [[IF_END3790:%.*]]
+// SIMD-ONLY0:       if.then3789:
+// SIMD-ONLY0-NEXT:    [[TMP2624:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2624]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3790]]
+// SIMD-ONLY0:       if.end3790:
+// SIMD-ONLY0-NEXT:    [[TMP2625:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2625]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2626:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2627:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3791:%.*]] = icmp ult i32 [[TMP2626]], [[TMP2627]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3791]], label [[IF_THEN3793:%.*]], label [[IF_END3794:%.*]]
+// SIMD-ONLY0:       if.then3793:
+// SIMD-ONLY0-NEXT:    [[TMP2628:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2628]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3794]]
+// SIMD-ONLY0:       if.end3794:
+// SIMD-ONLY0-NEXT:    [[TMP2629:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2629]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2630:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2631:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3795:%.*]] = icmp eq i32 [[TMP2630]], [[TMP2631]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3795]], label [[IF_THEN3797:%.*]], label [[IF_END3798:%.*]]
+// SIMD-ONLY0:       if.then3797:
+// SIMD-ONLY0-NEXT:    [[TMP2632:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2632]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3798]]
+// SIMD-ONLY0:       if.end3798:
+// SIMD-ONLY0-NEXT:    [[TMP2633:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2633]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2634:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2635:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3799:%.*]] = icmp eq i32 [[TMP2634]], [[TMP2635]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3799]], label [[IF_THEN3801:%.*]], label [[IF_END3802:%.*]]
+// SIMD-ONLY0:       if.then3801:
+// SIMD-ONLY0-NEXT:    [[TMP2636:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2636]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3802]]
+// SIMD-ONLY0:       if.end3802:
+// SIMD-ONLY0-NEXT:    [[TMP2637:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2637]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2638:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2639:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3803:%.*]] = icmp eq i32 [[TMP2638]], [[TMP2639]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3803]], label [[IF_THEN3805:%.*]], label [[IF_ELSE3806:%.*]]
+// SIMD-ONLY0:       if.then3805:
+// SIMD-ONLY0-NEXT:    [[TMP2640:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2640]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3807:%.*]]
+// SIMD-ONLY0:       if.else3806:
+// SIMD-ONLY0-NEXT:    [[TMP2641:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2641]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3807]]
+// SIMD-ONLY0:       if.end3807:
+// SIMD-ONLY0-NEXT:    [[TMP2642:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2643:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3808:%.*]] = icmp eq i32 [[TMP2642]], [[TMP2643]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3808]], label [[IF_THEN3810:%.*]], label [[IF_ELSE3811:%.*]]
+// SIMD-ONLY0:       if.then3810:
+// SIMD-ONLY0-NEXT:    [[TMP2644:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2644]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3812:%.*]]
+// SIMD-ONLY0:       if.else3811:
+// SIMD-ONLY0-NEXT:    [[TMP2645:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2645]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3812]]
+// SIMD-ONLY0:       if.end3812:
+// SIMD-ONLY0-NEXT:    [[TMP2646:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2647:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3813:%.*]] = icmp eq i32 [[TMP2646]], [[TMP2647]]
+// SIMD-ONLY0-NEXT:    [[CONV3814:%.*]] = zext i1 [[CMP3813]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3814]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2648:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3815:%.*]] = icmp ne i32 [[TMP2648]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3815]], label [[IF_THEN3816:%.*]], label [[IF_END3817:%.*]]
+// SIMD-ONLY0:       if.then3816:
+// SIMD-ONLY0-NEXT:    [[TMP2649:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2649]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3817]]
+// SIMD-ONLY0:       if.end3817:
+// SIMD-ONLY0-NEXT:    [[TMP2650:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2651:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3818:%.*]] = icmp eq i32 [[TMP2650]], [[TMP2651]]
+// SIMD-ONLY0-NEXT:    [[CONV3819:%.*]] = zext i1 [[CMP3818]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3819]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2652:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3820:%.*]] = icmp ne i32 [[TMP2652]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3820]], label [[IF_THEN3821:%.*]], label [[IF_END3822:%.*]]
+// SIMD-ONLY0:       if.then3821:
+// SIMD-ONLY0-NEXT:    [[TMP2653:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2653]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3822]]
+// SIMD-ONLY0:       if.end3822:
+// SIMD-ONLY0-NEXT:    [[TMP2654:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2655:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3823:%.*]] = icmp eq i32 [[TMP2654]], [[TMP2655]]
+// SIMD-ONLY0-NEXT:    [[CONV3824:%.*]] = zext i1 [[CMP3823]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3824]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2656:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3825:%.*]] = icmp ne i32 [[TMP2656]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3825]], label [[IF_THEN3826:%.*]], label [[IF_ELSE3827:%.*]]
+// SIMD-ONLY0:       if.then3826:
+// SIMD-ONLY0-NEXT:    [[TMP2657:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2657]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3828:%.*]]
+// SIMD-ONLY0:       if.else3827:
+// SIMD-ONLY0-NEXT:    [[TMP2658:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2658]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3828]]
+// SIMD-ONLY0:       if.end3828:
+// SIMD-ONLY0-NEXT:    [[TMP2659:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2660:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP3829:%.*]] = icmp eq i32 [[TMP2659]], [[TMP2660]]
+// SIMD-ONLY0-NEXT:    [[CONV3830:%.*]] = zext i1 [[CMP3829]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV3830]], ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2661:%.*]] = load i32, ptr [[UIR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL3831:%.*]] = icmp ne i32 [[TMP2661]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3831]], label [[IF_THEN3832:%.*]], label [[IF_ELSE3833:%.*]]
+// SIMD-ONLY0:       if.then3832:
+// SIMD-ONLY0-NEXT:    [[TMP2662:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2662]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3834:%.*]]
+// SIMD-ONLY0:       if.else3833:
+// SIMD-ONLY0-NEXT:    [[TMP2663:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2663]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END3834]]
+// SIMD-ONLY0:       if.end3834:
+// SIMD-ONLY0-NEXT:    [[TMP2664:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2664]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2665:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2666:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3835:%.*]] = icmp sgt i64 [[TMP2665]], [[TMP2666]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3835]], label [[IF_THEN3837:%.*]], label [[IF_END3838:%.*]]
+// SIMD-ONLY0:       if.then3837:
+// SIMD-ONLY0-NEXT:    [[TMP2667:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2667]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3838]]
+// SIMD-ONLY0:       if.end3838:
+// SIMD-ONLY0-NEXT:    [[TMP2668:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2668]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2669:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2670:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3839:%.*]] = icmp sgt i64 [[TMP2669]], [[TMP2670]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3839]], label [[IF_THEN3841:%.*]], label [[IF_END3842:%.*]]
+// SIMD-ONLY0:       if.then3841:
+// SIMD-ONLY0-NEXT:    [[TMP2671:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2671]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3842]]
+// SIMD-ONLY0:       if.end3842:
+// SIMD-ONLY0-NEXT:    [[TMP2672:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2672]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2673:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2674:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3843:%.*]] = icmp slt i64 [[TMP2673]], [[TMP2674]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3843]], label [[IF_THEN3845:%.*]], label [[IF_END3846:%.*]]
+// SIMD-ONLY0:       if.then3845:
+// SIMD-ONLY0-NEXT:    [[TMP2675:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2675]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3846]]
+// SIMD-ONLY0:       if.end3846:
+// SIMD-ONLY0-NEXT:    [[TMP2676:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2676]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2677:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2678:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3847:%.*]] = icmp slt i64 [[TMP2677]], [[TMP2678]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3847]], label [[IF_THEN3849:%.*]], label [[IF_END3850:%.*]]
+// SIMD-ONLY0:       if.then3849:
+// SIMD-ONLY0-NEXT:    [[TMP2679:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2679]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3850]]
+// SIMD-ONLY0:       if.end3850:
+// SIMD-ONLY0-NEXT:    [[TMP2680:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2680]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2681:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2682:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3851:%.*]] = icmp eq i64 [[TMP2681]], [[TMP2682]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3851]], label [[IF_THEN3853:%.*]], label [[IF_END3854:%.*]]
+// SIMD-ONLY0:       if.then3853:
+// SIMD-ONLY0-NEXT:    [[TMP2683:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2683]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3854]]
+// SIMD-ONLY0:       if.end3854:
+// SIMD-ONLY0-NEXT:    [[TMP2684:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2684]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2685:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2686:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3855:%.*]] = icmp eq i64 [[TMP2685]], [[TMP2686]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3855]], label [[IF_THEN3857:%.*]], label [[IF_END3858:%.*]]
+// SIMD-ONLY0:       if.then3857:
+// SIMD-ONLY0-NEXT:    [[TMP2687:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2687]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3858]]
+// SIMD-ONLY0:       if.end3858:
+// SIMD-ONLY0-NEXT:    [[TMP2688:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2689:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3859:%.*]] = icmp sgt i64 [[TMP2688]], [[TMP2689]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3859]], label [[IF_THEN3861:%.*]], label [[IF_END3862:%.*]]
+// SIMD-ONLY0:       if.then3861:
+// SIMD-ONLY0-NEXT:    [[TMP2690:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2690]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3862]]
+// SIMD-ONLY0:       if.end3862:
+// SIMD-ONLY0-NEXT:    [[TMP2691:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2691]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2692:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2693:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3863:%.*]] = icmp sgt i64 [[TMP2692]], [[TMP2693]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3863]], label [[IF_THEN3865:%.*]], label [[IF_END3866:%.*]]
+// SIMD-ONLY0:       if.then3865:
+// SIMD-ONLY0-NEXT:    [[TMP2694:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2694]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3866]]
+// SIMD-ONLY0:       if.end3866:
+// SIMD-ONLY0-NEXT:    [[TMP2695:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2695]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2696:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2697:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3867:%.*]] = icmp slt i64 [[TMP2696]], [[TMP2697]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3867]], label [[IF_THEN3869:%.*]], label [[IF_END3870:%.*]]
+// SIMD-ONLY0:       if.then3869:
+// SIMD-ONLY0-NEXT:    [[TMP2698:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2698]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3870]]
+// SIMD-ONLY0:       if.end3870:
+// SIMD-ONLY0-NEXT:    [[TMP2699:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2699]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2700:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2701:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3871:%.*]] = icmp slt i64 [[TMP2700]], [[TMP2701]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3871]], label [[IF_THEN3873:%.*]], label [[IF_END3874:%.*]]
+// SIMD-ONLY0:       if.then3873:
+// SIMD-ONLY0-NEXT:    [[TMP2702:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2702]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3874]]
+// SIMD-ONLY0:       if.end3874:
+// SIMD-ONLY0-NEXT:    [[TMP2703:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2703]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2704:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2705:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3875:%.*]] = icmp eq i64 [[TMP2704]], [[TMP2705]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3875]], label [[IF_THEN3877:%.*]], label [[IF_END3878:%.*]]
+// SIMD-ONLY0:       if.then3877:
+// SIMD-ONLY0-NEXT:    [[TMP2706:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2706]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3878]]
+// SIMD-ONLY0:       if.end3878:
+// SIMD-ONLY0-NEXT:    [[TMP2707:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2707]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2708:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2709:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3879:%.*]] = icmp eq i64 [[TMP2708]], [[TMP2709]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3879]], label [[IF_THEN3881:%.*]], label [[IF_END3882:%.*]]
+// SIMD-ONLY0:       if.then3881:
+// SIMD-ONLY0-NEXT:    [[TMP2710:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2710]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3882]]
+// SIMD-ONLY0:       if.end3882:
+// SIMD-ONLY0-NEXT:    [[TMP2711:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2711]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2712:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2713:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3883:%.*]] = icmp eq i64 [[TMP2712]], [[TMP2713]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3883]], label [[IF_THEN3885:%.*]], label [[IF_ELSE3886:%.*]]
+// SIMD-ONLY0:       if.then3885:
+// SIMD-ONLY0-NEXT:    [[TMP2714:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2714]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3887:%.*]]
+// SIMD-ONLY0:       if.else3886:
+// SIMD-ONLY0-NEXT:    [[TMP2715:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2715]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3887]]
+// SIMD-ONLY0:       if.end3887:
+// SIMD-ONLY0-NEXT:    [[TMP2716:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2717:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3888:%.*]] = icmp eq i64 [[TMP2716]], [[TMP2717]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3888]], label [[IF_THEN3890:%.*]], label [[IF_ELSE3891:%.*]]
+// SIMD-ONLY0:       if.then3890:
+// SIMD-ONLY0-NEXT:    [[TMP2718:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2718]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3892:%.*]]
+// SIMD-ONLY0:       if.else3891:
+// SIMD-ONLY0-NEXT:    [[TMP2719:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2719]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3892]]
+// SIMD-ONLY0:       if.end3892:
+// SIMD-ONLY0-NEXT:    [[TMP2720:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2721:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3893:%.*]] = icmp eq i64 [[TMP2720]], [[TMP2721]]
+// SIMD-ONLY0-NEXT:    [[CONV3894:%.*]] = zext i1 [[CMP3893]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV3895:%.*]] = sext i32 [[CONV3894]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV3895]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2722:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL3896:%.*]] = icmp ne i64 [[TMP2722]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3896]], label [[IF_THEN3897:%.*]], label [[IF_END3898:%.*]]
+// SIMD-ONLY0:       if.then3897:
+// SIMD-ONLY0-NEXT:    [[TMP2723:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2723]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3898]]
+// SIMD-ONLY0:       if.end3898:
+// SIMD-ONLY0-NEXT:    [[TMP2724:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2725:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3899:%.*]] = icmp eq i64 [[TMP2724]], [[TMP2725]]
+// SIMD-ONLY0-NEXT:    [[CONV3900:%.*]] = zext i1 [[CMP3899]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV3901:%.*]] = sext i32 [[CONV3900]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV3901]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2726:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL3902:%.*]] = icmp ne i64 [[TMP2726]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3902]], label [[IF_THEN3903:%.*]], label [[IF_END3904:%.*]]
+// SIMD-ONLY0:       if.then3903:
+// SIMD-ONLY0-NEXT:    [[TMP2727:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2727]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3904]]
+// SIMD-ONLY0:       if.end3904:
+// SIMD-ONLY0-NEXT:    [[TMP2728:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2729:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3905:%.*]] = icmp eq i64 [[TMP2728]], [[TMP2729]]
+// SIMD-ONLY0-NEXT:    [[CONV3906:%.*]] = zext i1 [[CMP3905]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV3907:%.*]] = sext i32 [[CONV3906]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV3907]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2730:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL3908:%.*]] = icmp ne i64 [[TMP2730]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3908]], label [[IF_THEN3909:%.*]], label [[IF_ELSE3910:%.*]]
+// SIMD-ONLY0:       if.then3909:
+// SIMD-ONLY0-NEXT:    [[TMP2731:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2731]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3911:%.*]]
+// SIMD-ONLY0:       if.else3910:
+// SIMD-ONLY0-NEXT:    [[TMP2732:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2732]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3911]]
+// SIMD-ONLY0:       if.end3911:
+// SIMD-ONLY0-NEXT:    [[TMP2733:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2734:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3912:%.*]] = icmp eq i64 [[TMP2733]], [[TMP2734]]
+// SIMD-ONLY0-NEXT:    [[CONV3913:%.*]] = zext i1 [[CMP3912]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV3914:%.*]] = sext i32 [[CONV3913]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV3914]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2735:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL3915:%.*]] = icmp ne i64 [[TMP2735]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3915]], label [[IF_THEN3916:%.*]], label [[IF_ELSE3917:%.*]]
+// SIMD-ONLY0:       if.then3916:
+// SIMD-ONLY0-NEXT:    [[TMP2736:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2736]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3918:%.*]]
+// SIMD-ONLY0:       if.else3917:
+// SIMD-ONLY0-NEXT:    [[TMP2737:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2737]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3918]]
+// SIMD-ONLY0:       if.end3918:
+// SIMD-ONLY0-NEXT:    [[TMP2738:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2738]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2739:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2740:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3919:%.*]] = icmp sgt i64 [[TMP2739]], [[TMP2740]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3919]], label [[IF_THEN3921:%.*]], label [[IF_END3922:%.*]]
+// SIMD-ONLY0:       if.then3921:
+// SIMD-ONLY0-NEXT:    [[TMP2741:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2741]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3922]]
+// SIMD-ONLY0:       if.end3922:
+// SIMD-ONLY0-NEXT:    [[TMP2742:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2742]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2743:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2744:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3923:%.*]] = icmp sgt i64 [[TMP2743]], [[TMP2744]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3923]], label [[IF_THEN3925:%.*]], label [[IF_END3926:%.*]]
+// SIMD-ONLY0:       if.then3925:
+// SIMD-ONLY0-NEXT:    [[TMP2745:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2745]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3926]]
+// SIMD-ONLY0:       if.end3926:
+// SIMD-ONLY0-NEXT:    [[TMP2746:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2746]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2747:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2748:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3927:%.*]] = icmp slt i64 [[TMP2747]], [[TMP2748]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3927]], label [[IF_THEN3929:%.*]], label [[IF_END3930:%.*]]
+// SIMD-ONLY0:       if.then3929:
+// SIMD-ONLY0-NEXT:    [[TMP2749:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2749]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3930]]
+// SIMD-ONLY0:       if.end3930:
+// SIMD-ONLY0-NEXT:    [[TMP2750:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2750]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2751:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2752:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3931:%.*]] = icmp slt i64 [[TMP2751]], [[TMP2752]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3931]], label [[IF_THEN3933:%.*]], label [[IF_END3934:%.*]]
+// SIMD-ONLY0:       if.then3933:
+// SIMD-ONLY0-NEXT:    [[TMP2753:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2753]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3934]]
+// SIMD-ONLY0:       if.end3934:
+// SIMD-ONLY0-NEXT:    [[TMP2754:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2754]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2755:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2756:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3935:%.*]] = icmp eq i64 [[TMP2755]], [[TMP2756]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3935]], label [[IF_THEN3937:%.*]], label [[IF_END3938:%.*]]
+// SIMD-ONLY0:       if.then3937:
+// SIMD-ONLY0-NEXT:    [[TMP2757:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2757]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3938]]
+// SIMD-ONLY0:       if.end3938:
+// SIMD-ONLY0-NEXT:    [[TMP2758:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2758]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2759:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2760:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3939:%.*]] = icmp eq i64 [[TMP2759]], [[TMP2760]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3939]], label [[IF_THEN3941:%.*]], label [[IF_END3942:%.*]]
+// SIMD-ONLY0:       if.then3941:
+// SIMD-ONLY0-NEXT:    [[TMP2761:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2761]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3942]]
+// SIMD-ONLY0:       if.end3942:
+// SIMD-ONLY0-NEXT:    [[TMP2762:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2763:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3943:%.*]] = icmp sgt i64 [[TMP2762]], [[TMP2763]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3943]], label [[IF_THEN3945:%.*]], label [[IF_END3946:%.*]]
+// SIMD-ONLY0:       if.then3945:
+// SIMD-ONLY0-NEXT:    [[TMP2764:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2764]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3946]]
+// SIMD-ONLY0:       if.end3946:
+// SIMD-ONLY0-NEXT:    [[TMP2765:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2765]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2766:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2767:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3947:%.*]] = icmp sgt i64 [[TMP2766]], [[TMP2767]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3947]], label [[IF_THEN3949:%.*]], label [[IF_END3950:%.*]]
+// SIMD-ONLY0:       if.then3949:
+// SIMD-ONLY0-NEXT:    [[TMP2768:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2768]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3950]]
+// SIMD-ONLY0:       if.end3950:
+// SIMD-ONLY0-NEXT:    [[TMP2769:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2769]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2770:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2771:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3951:%.*]] = icmp slt i64 [[TMP2770]], [[TMP2771]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3951]], label [[IF_THEN3953:%.*]], label [[IF_END3954:%.*]]
+// SIMD-ONLY0:       if.then3953:
+// SIMD-ONLY0-NEXT:    [[TMP2772:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2772]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3954]]
+// SIMD-ONLY0:       if.end3954:
+// SIMD-ONLY0-NEXT:    [[TMP2773:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2773]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2774:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2775:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3955:%.*]] = icmp slt i64 [[TMP2774]], [[TMP2775]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3955]], label [[IF_THEN3957:%.*]], label [[IF_END3958:%.*]]
+// SIMD-ONLY0:       if.then3957:
+// SIMD-ONLY0-NEXT:    [[TMP2776:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2776]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3958]]
+// SIMD-ONLY0:       if.end3958:
+// SIMD-ONLY0-NEXT:    [[TMP2777:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2777]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2778:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2779:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3959:%.*]] = icmp eq i64 [[TMP2778]], [[TMP2779]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3959]], label [[IF_THEN3961:%.*]], label [[IF_END3962:%.*]]
+// SIMD-ONLY0:       if.then3961:
+// SIMD-ONLY0-NEXT:    [[TMP2780:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2780]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3962]]
+// SIMD-ONLY0:       if.end3962:
+// SIMD-ONLY0-NEXT:    [[TMP2781:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2781]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2782:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2783:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3963:%.*]] = icmp eq i64 [[TMP2782]], [[TMP2783]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3963]], label [[IF_THEN3965:%.*]], label [[IF_END3966:%.*]]
+// SIMD-ONLY0:       if.then3965:
+// SIMD-ONLY0-NEXT:    [[TMP2784:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2784]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3966]]
+// SIMD-ONLY0:       if.end3966:
+// SIMD-ONLY0-NEXT:    [[TMP2785:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2785]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2786:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2787:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3967:%.*]] = icmp eq i64 [[TMP2786]], [[TMP2787]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3967]], label [[IF_THEN3969:%.*]], label [[IF_ELSE3970:%.*]]
+// SIMD-ONLY0:       if.then3969:
+// SIMD-ONLY0-NEXT:    [[TMP2788:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2788]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3971:%.*]]
+// SIMD-ONLY0:       if.else3970:
+// SIMD-ONLY0-NEXT:    [[TMP2789:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2789]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3971]]
+// SIMD-ONLY0:       if.end3971:
+// SIMD-ONLY0-NEXT:    [[TMP2790:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2791:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3972:%.*]] = icmp eq i64 [[TMP2790]], [[TMP2791]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP3972]], label [[IF_THEN3974:%.*]], label [[IF_ELSE3975:%.*]]
+// SIMD-ONLY0:       if.then3974:
+// SIMD-ONLY0-NEXT:    [[TMP2792:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2792]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3976:%.*]]
+// SIMD-ONLY0:       if.else3975:
+// SIMD-ONLY0-NEXT:    [[TMP2793:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2793]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3976]]
+// SIMD-ONLY0:       if.end3976:
+// SIMD-ONLY0-NEXT:    [[TMP2794:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2795:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3977:%.*]] = icmp eq i64 [[TMP2794]], [[TMP2795]]
+// SIMD-ONLY0-NEXT:    [[CONV3978:%.*]] = zext i1 [[CMP3977]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV3979:%.*]] = sext i32 [[CONV3978]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV3979]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2796:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL3980:%.*]] = icmp ne i64 [[TMP2796]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3980]], label [[IF_THEN3981:%.*]], label [[IF_END3982:%.*]]
+// SIMD-ONLY0:       if.then3981:
+// SIMD-ONLY0-NEXT:    [[TMP2797:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2797]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3982]]
+// SIMD-ONLY0:       if.end3982:
+// SIMD-ONLY0-NEXT:    [[TMP2798:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2799:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3983:%.*]] = icmp eq i64 [[TMP2798]], [[TMP2799]]
+// SIMD-ONLY0-NEXT:    [[CONV3984:%.*]] = zext i1 [[CMP3983]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV3985:%.*]] = sext i32 [[CONV3984]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV3985]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2800:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL3986:%.*]] = icmp ne i64 [[TMP2800]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3986]], label [[IF_THEN3987:%.*]], label [[IF_END3988:%.*]]
+// SIMD-ONLY0:       if.then3987:
+// SIMD-ONLY0-NEXT:    [[TMP2801:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2801]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3988]]
+// SIMD-ONLY0:       if.end3988:
+// SIMD-ONLY0-NEXT:    [[TMP2802:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2803:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3989:%.*]] = icmp eq i64 [[TMP2802]], [[TMP2803]]
+// SIMD-ONLY0-NEXT:    [[CONV3990:%.*]] = zext i1 [[CMP3989]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV3991:%.*]] = sext i32 [[CONV3990]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV3991]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2804:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL3992:%.*]] = icmp ne i64 [[TMP2804]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3992]], label [[IF_THEN3993:%.*]], label [[IF_ELSE3994:%.*]]
+// SIMD-ONLY0:       if.then3993:
+// SIMD-ONLY0-NEXT:    [[TMP2805:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2805]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3995:%.*]]
+// SIMD-ONLY0:       if.else3994:
+// SIMD-ONLY0-NEXT:    [[TMP2806:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2806]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END3995]]
+// SIMD-ONLY0:       if.end3995:
+// SIMD-ONLY0-NEXT:    [[TMP2807:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2808:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP3996:%.*]] = icmp eq i64 [[TMP2807]], [[TMP2808]]
+// SIMD-ONLY0-NEXT:    [[CONV3997:%.*]] = zext i1 [[CMP3996]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV3998:%.*]] = sext i32 [[CONV3997]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV3998]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2809:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL3999:%.*]] = icmp ne i64 [[TMP2809]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL3999]], label [[IF_THEN4000:%.*]], label [[IF_ELSE4001:%.*]]
+// SIMD-ONLY0:       if.then4000:
+// SIMD-ONLY0-NEXT:    [[TMP2810:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2810]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4002:%.*]]
+// SIMD-ONLY0:       if.else4001:
+// SIMD-ONLY0-NEXT:    [[TMP2811:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2811]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4002]]
+// SIMD-ONLY0:       if.end4002:
+// SIMD-ONLY0-NEXT:    [[TMP2812:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2812]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2813:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2814:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4003:%.*]] = icmp sgt i64 [[TMP2813]], [[TMP2814]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4003]], label [[IF_THEN4005:%.*]], label [[IF_END4006:%.*]]
+// SIMD-ONLY0:       if.then4005:
+// SIMD-ONLY0-NEXT:    [[TMP2815:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2815]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4006]]
+// SIMD-ONLY0:       if.end4006:
+// SIMD-ONLY0-NEXT:    [[TMP2816:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2816]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2817:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2818:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4007:%.*]] = icmp sgt i64 [[TMP2817]], [[TMP2818]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4007]], label [[IF_THEN4009:%.*]], label [[IF_END4010:%.*]]
+// SIMD-ONLY0:       if.then4009:
+// SIMD-ONLY0-NEXT:    [[TMP2819:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2819]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4010]]
+// SIMD-ONLY0:       if.end4010:
+// SIMD-ONLY0-NEXT:    [[TMP2820:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2820]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2821:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2822:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4011:%.*]] = icmp slt i64 [[TMP2821]], [[TMP2822]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4011]], label [[IF_THEN4013:%.*]], label [[IF_END4014:%.*]]
+// SIMD-ONLY0:       if.then4013:
+// SIMD-ONLY0-NEXT:    [[TMP2823:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2823]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4014]]
+// SIMD-ONLY0:       if.end4014:
+// SIMD-ONLY0-NEXT:    [[TMP2824:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2824]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2825:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2826:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4015:%.*]] = icmp slt i64 [[TMP2825]], [[TMP2826]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4015]], label [[IF_THEN4017:%.*]], label [[IF_END4018:%.*]]
+// SIMD-ONLY0:       if.then4017:
+// SIMD-ONLY0-NEXT:    [[TMP2827:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2827]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4018]]
+// SIMD-ONLY0:       if.end4018:
+// SIMD-ONLY0-NEXT:    [[TMP2828:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2828]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2829:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2830:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4019:%.*]] = icmp eq i64 [[TMP2829]], [[TMP2830]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4019]], label [[IF_THEN4021:%.*]], label [[IF_END4022:%.*]]
+// SIMD-ONLY0:       if.then4021:
+// SIMD-ONLY0-NEXT:    [[TMP2831:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2831]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4022]]
+// SIMD-ONLY0:       if.end4022:
+// SIMD-ONLY0-NEXT:    [[TMP2832:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2832]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2833:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2834:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4023:%.*]] = icmp eq i64 [[TMP2833]], [[TMP2834]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4023]], label [[IF_THEN4025:%.*]], label [[IF_END4026:%.*]]
+// SIMD-ONLY0:       if.then4025:
+// SIMD-ONLY0-NEXT:    [[TMP2835:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2835]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4026]]
+// SIMD-ONLY0:       if.end4026:
+// SIMD-ONLY0-NEXT:    [[TMP2836:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2837:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4027:%.*]] = icmp sgt i64 [[TMP2836]], [[TMP2837]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4027]], label [[IF_THEN4029:%.*]], label [[IF_END4030:%.*]]
+// SIMD-ONLY0:       if.then4029:
+// SIMD-ONLY0-NEXT:    [[TMP2838:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2838]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4030]]
+// SIMD-ONLY0:       if.end4030:
+// SIMD-ONLY0-NEXT:    [[TMP2839:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2839]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2840:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2841:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4031:%.*]] = icmp sgt i64 [[TMP2840]], [[TMP2841]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4031]], label [[IF_THEN4033:%.*]], label [[IF_END4034:%.*]]
+// SIMD-ONLY0:       if.then4033:
+// SIMD-ONLY0-NEXT:    [[TMP2842:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2842]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4034]]
+// SIMD-ONLY0:       if.end4034:
+// SIMD-ONLY0-NEXT:    [[TMP2843:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2843]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2844:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2845:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4035:%.*]] = icmp slt i64 [[TMP2844]], [[TMP2845]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4035]], label [[IF_THEN4037:%.*]], label [[IF_END4038:%.*]]
+// SIMD-ONLY0:       if.then4037:
+// SIMD-ONLY0-NEXT:    [[TMP2846:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2846]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4038]]
+// SIMD-ONLY0:       if.end4038:
+// SIMD-ONLY0-NEXT:    [[TMP2847:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2847]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2848:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2849:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4039:%.*]] = icmp slt i64 [[TMP2848]], [[TMP2849]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4039]], label [[IF_THEN4041:%.*]], label [[IF_END4042:%.*]]
+// SIMD-ONLY0:       if.then4041:
+// SIMD-ONLY0-NEXT:    [[TMP2850:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2850]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4042]]
+// SIMD-ONLY0:       if.end4042:
+// SIMD-ONLY0-NEXT:    [[TMP2851:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2851]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2852:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2853:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4043:%.*]] = icmp eq i64 [[TMP2852]], [[TMP2853]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4043]], label [[IF_THEN4045:%.*]], label [[IF_END4046:%.*]]
+// SIMD-ONLY0:       if.then4045:
+// SIMD-ONLY0-NEXT:    [[TMP2854:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2854]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4046]]
+// SIMD-ONLY0:       if.end4046:
+// SIMD-ONLY0-NEXT:    [[TMP2855:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2855]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2856:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2857:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4047:%.*]] = icmp eq i64 [[TMP2856]], [[TMP2857]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4047]], label [[IF_THEN4049:%.*]], label [[IF_END4050:%.*]]
+// SIMD-ONLY0:       if.then4049:
+// SIMD-ONLY0-NEXT:    [[TMP2858:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2858]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4050]]
+// SIMD-ONLY0:       if.end4050:
+// SIMD-ONLY0-NEXT:    [[TMP2859:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2859]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2860:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2861:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4051:%.*]] = icmp eq i64 [[TMP2860]], [[TMP2861]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4051]], label [[IF_THEN4053:%.*]], label [[IF_ELSE4054:%.*]]
+// SIMD-ONLY0:       if.then4053:
+// SIMD-ONLY0-NEXT:    [[TMP2862:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2862]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4055:%.*]]
+// SIMD-ONLY0:       if.else4054:
+// SIMD-ONLY0-NEXT:    [[TMP2863:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2863]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4055]]
+// SIMD-ONLY0:       if.end4055:
+// SIMD-ONLY0-NEXT:    [[TMP2864:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2865:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4056:%.*]] = icmp eq i64 [[TMP2864]], [[TMP2865]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4056]], label [[IF_THEN4058:%.*]], label [[IF_ELSE4059:%.*]]
+// SIMD-ONLY0:       if.then4058:
+// SIMD-ONLY0-NEXT:    [[TMP2866:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2866]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4060:%.*]]
+// SIMD-ONLY0:       if.else4059:
+// SIMD-ONLY0-NEXT:    [[TMP2867:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2867]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4060]]
+// SIMD-ONLY0:       if.end4060:
+// SIMD-ONLY0-NEXT:    [[TMP2868:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2869:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4061:%.*]] = icmp eq i64 [[TMP2868]], [[TMP2869]]
+// SIMD-ONLY0-NEXT:    [[CONV4062:%.*]] = zext i1 [[CMP4061]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4063:%.*]] = sext i32 [[CONV4062]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4063]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2870:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4064:%.*]] = icmp ne i64 [[TMP2870]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4064]], label [[IF_THEN4065:%.*]], label [[IF_END4066:%.*]]
+// SIMD-ONLY0:       if.then4065:
+// SIMD-ONLY0-NEXT:    [[TMP2871:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2871]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4066]]
+// SIMD-ONLY0:       if.end4066:
+// SIMD-ONLY0-NEXT:    [[TMP2872:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2873:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4067:%.*]] = icmp eq i64 [[TMP2872]], [[TMP2873]]
+// SIMD-ONLY0-NEXT:    [[CONV4068:%.*]] = zext i1 [[CMP4067]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4069:%.*]] = sext i32 [[CONV4068]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4069]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2874:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4070:%.*]] = icmp ne i64 [[TMP2874]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4070]], label [[IF_THEN4071:%.*]], label [[IF_END4072:%.*]]
+// SIMD-ONLY0:       if.then4071:
+// SIMD-ONLY0-NEXT:    [[TMP2875:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2875]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4072]]
+// SIMD-ONLY0:       if.end4072:
+// SIMD-ONLY0-NEXT:    [[TMP2876:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2877:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4073:%.*]] = icmp eq i64 [[TMP2876]], [[TMP2877]]
+// SIMD-ONLY0-NEXT:    [[CONV4074:%.*]] = zext i1 [[CMP4073]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4075:%.*]] = sext i32 [[CONV4074]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4075]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2878:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4076:%.*]] = icmp ne i64 [[TMP2878]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4076]], label [[IF_THEN4077:%.*]], label [[IF_ELSE4078:%.*]]
+// SIMD-ONLY0:       if.then4077:
+// SIMD-ONLY0-NEXT:    [[TMP2879:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2879]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4079:%.*]]
+// SIMD-ONLY0:       if.else4078:
+// SIMD-ONLY0-NEXT:    [[TMP2880:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2880]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4079]]
+// SIMD-ONLY0:       if.end4079:
+// SIMD-ONLY0-NEXT:    [[TMP2881:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2882:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4080:%.*]] = icmp eq i64 [[TMP2881]], [[TMP2882]]
+// SIMD-ONLY0-NEXT:    [[CONV4081:%.*]] = zext i1 [[CMP4080]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4082:%.*]] = sext i32 [[CONV4081]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4082]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2883:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4083:%.*]] = icmp ne i64 [[TMP2883]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4083]], label [[IF_THEN4084:%.*]], label [[IF_ELSE4085:%.*]]
+// SIMD-ONLY0:       if.then4084:
+// SIMD-ONLY0-NEXT:    [[TMP2884:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2884]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4086:%.*]]
+// SIMD-ONLY0:       if.else4085:
+// SIMD-ONLY0-NEXT:    [[TMP2885:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2885]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4086]]
+// SIMD-ONLY0:       if.end4086:
+// SIMD-ONLY0-NEXT:    [[TMP2886:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2886]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2887:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2888:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4087:%.*]] = icmp sgt i64 [[TMP2887]], [[TMP2888]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4087]], label [[IF_THEN4089:%.*]], label [[IF_END4090:%.*]]
+// SIMD-ONLY0:       if.then4089:
+// SIMD-ONLY0-NEXT:    [[TMP2889:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2889]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4090]]
+// SIMD-ONLY0:       if.end4090:
+// SIMD-ONLY0-NEXT:    [[TMP2890:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2890]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2891:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2892:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4091:%.*]] = icmp sgt i64 [[TMP2891]], [[TMP2892]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4091]], label [[IF_THEN4093:%.*]], label [[IF_END4094:%.*]]
+// SIMD-ONLY0:       if.then4093:
+// SIMD-ONLY0-NEXT:    [[TMP2893:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2893]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4094]]
+// SIMD-ONLY0:       if.end4094:
+// SIMD-ONLY0-NEXT:    [[TMP2894:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2894]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2895:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2896:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4095:%.*]] = icmp slt i64 [[TMP2895]], [[TMP2896]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4095]], label [[IF_THEN4097:%.*]], label [[IF_END4098:%.*]]
+// SIMD-ONLY0:       if.then4097:
+// SIMD-ONLY0-NEXT:    [[TMP2897:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2897]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4098]]
+// SIMD-ONLY0:       if.end4098:
+// SIMD-ONLY0-NEXT:    [[TMP2898:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2898]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2899:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2900:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4099:%.*]] = icmp slt i64 [[TMP2899]], [[TMP2900]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4099]], label [[IF_THEN4101:%.*]], label [[IF_END4102:%.*]]
+// SIMD-ONLY0:       if.then4101:
+// SIMD-ONLY0-NEXT:    [[TMP2901:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2901]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4102]]
+// SIMD-ONLY0:       if.end4102:
+// SIMD-ONLY0-NEXT:    [[TMP2902:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2902]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2903:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2904:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4103:%.*]] = icmp eq i64 [[TMP2903]], [[TMP2904]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4103]], label [[IF_THEN4105:%.*]], label [[IF_END4106:%.*]]
+// SIMD-ONLY0:       if.then4105:
+// SIMD-ONLY0-NEXT:    [[TMP2905:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2905]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4106]]
+// SIMD-ONLY0:       if.end4106:
+// SIMD-ONLY0-NEXT:    [[TMP2906:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2906]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2907:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2908:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4107:%.*]] = icmp eq i64 [[TMP2907]], [[TMP2908]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4107]], label [[IF_THEN4109:%.*]], label [[IF_END4110:%.*]]
+// SIMD-ONLY0:       if.then4109:
+// SIMD-ONLY0-NEXT:    [[TMP2909:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2909]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4110]]
+// SIMD-ONLY0:       if.end4110:
+// SIMD-ONLY0-NEXT:    [[TMP2910:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2911:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4111:%.*]] = icmp sgt i64 [[TMP2910]], [[TMP2911]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4111]], label [[IF_THEN4113:%.*]], label [[IF_END4114:%.*]]
+// SIMD-ONLY0:       if.then4113:
+// SIMD-ONLY0-NEXT:    [[TMP2912:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2912]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4114]]
+// SIMD-ONLY0:       if.end4114:
+// SIMD-ONLY0-NEXT:    [[TMP2913:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2913]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2914:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2915:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4115:%.*]] = icmp sgt i64 [[TMP2914]], [[TMP2915]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4115]], label [[IF_THEN4117:%.*]], label [[IF_END4118:%.*]]
+// SIMD-ONLY0:       if.then4117:
+// SIMD-ONLY0-NEXT:    [[TMP2916:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2916]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4118]]
+// SIMD-ONLY0:       if.end4118:
+// SIMD-ONLY0-NEXT:    [[TMP2917:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2917]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2918:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2919:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4119:%.*]] = icmp slt i64 [[TMP2918]], [[TMP2919]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4119]], label [[IF_THEN4121:%.*]], label [[IF_END4122:%.*]]
+// SIMD-ONLY0:       if.then4121:
+// SIMD-ONLY0-NEXT:    [[TMP2920:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2920]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4122]]
+// SIMD-ONLY0:       if.end4122:
+// SIMD-ONLY0-NEXT:    [[TMP2921:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2921]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2922:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2923:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4123:%.*]] = icmp slt i64 [[TMP2922]], [[TMP2923]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4123]], label [[IF_THEN4125:%.*]], label [[IF_END4126:%.*]]
+// SIMD-ONLY0:       if.then4125:
+// SIMD-ONLY0-NEXT:    [[TMP2924:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2924]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4126]]
+// SIMD-ONLY0:       if.end4126:
+// SIMD-ONLY0-NEXT:    [[TMP2925:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2925]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2926:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2927:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4127:%.*]] = icmp eq i64 [[TMP2926]], [[TMP2927]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4127]], label [[IF_THEN4129:%.*]], label [[IF_END4130:%.*]]
+// SIMD-ONLY0:       if.then4129:
+// SIMD-ONLY0-NEXT:    [[TMP2928:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2928]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4130]]
+// SIMD-ONLY0:       if.end4130:
+// SIMD-ONLY0-NEXT:    [[TMP2929:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2929]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2930:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2931:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4131:%.*]] = icmp eq i64 [[TMP2930]], [[TMP2931]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4131]], label [[IF_THEN4133:%.*]], label [[IF_END4134:%.*]]
+// SIMD-ONLY0:       if.then4133:
+// SIMD-ONLY0-NEXT:    [[TMP2932:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2932]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4134]]
+// SIMD-ONLY0:       if.end4134:
+// SIMD-ONLY0-NEXT:    [[TMP2933:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2933]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2934:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2935:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4135:%.*]] = icmp eq i64 [[TMP2934]], [[TMP2935]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4135]], label [[IF_THEN4137:%.*]], label [[IF_ELSE4138:%.*]]
+// SIMD-ONLY0:       if.then4137:
+// SIMD-ONLY0-NEXT:    [[TMP2936:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2936]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4139:%.*]]
+// SIMD-ONLY0:       if.else4138:
+// SIMD-ONLY0-NEXT:    [[TMP2937:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2937]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4139]]
+// SIMD-ONLY0:       if.end4139:
+// SIMD-ONLY0-NEXT:    [[TMP2938:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2939:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4140:%.*]] = icmp eq i64 [[TMP2938]], [[TMP2939]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4140]], label [[IF_THEN4142:%.*]], label [[IF_ELSE4143:%.*]]
+// SIMD-ONLY0:       if.then4142:
+// SIMD-ONLY0-NEXT:    [[TMP2940:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2940]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4144:%.*]]
+// SIMD-ONLY0:       if.else4143:
+// SIMD-ONLY0-NEXT:    [[TMP2941:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2941]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4144]]
+// SIMD-ONLY0:       if.end4144:
+// SIMD-ONLY0-NEXT:    [[TMP2942:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2943:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4145:%.*]] = icmp eq i64 [[TMP2942]], [[TMP2943]]
+// SIMD-ONLY0-NEXT:    [[CONV4146:%.*]] = zext i1 [[CMP4145]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4147:%.*]] = sext i32 [[CONV4146]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4147]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2944:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4148:%.*]] = icmp ne i64 [[TMP2944]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4148]], label [[IF_THEN4149:%.*]], label [[IF_END4150:%.*]]
+// SIMD-ONLY0:       if.then4149:
+// SIMD-ONLY0-NEXT:    [[TMP2945:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2945]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4150]]
+// SIMD-ONLY0:       if.end4150:
+// SIMD-ONLY0-NEXT:    [[TMP2946:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2947:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4151:%.*]] = icmp eq i64 [[TMP2946]], [[TMP2947]]
+// SIMD-ONLY0-NEXT:    [[CONV4152:%.*]] = zext i1 [[CMP4151]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4153:%.*]] = sext i32 [[CONV4152]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4153]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2948:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4154:%.*]] = icmp ne i64 [[TMP2948]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4154]], label [[IF_THEN4155:%.*]], label [[IF_END4156:%.*]]
+// SIMD-ONLY0:       if.then4155:
+// SIMD-ONLY0-NEXT:    [[TMP2949:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2949]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4156]]
+// SIMD-ONLY0:       if.end4156:
+// SIMD-ONLY0-NEXT:    [[TMP2950:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2951:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4157:%.*]] = icmp eq i64 [[TMP2950]], [[TMP2951]]
+// SIMD-ONLY0-NEXT:    [[CONV4158:%.*]] = zext i1 [[CMP4157]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4159:%.*]] = sext i32 [[CONV4158]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4159]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2952:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4160:%.*]] = icmp ne i64 [[TMP2952]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4160]], label [[IF_THEN4161:%.*]], label [[IF_ELSE4162:%.*]]
+// SIMD-ONLY0:       if.then4161:
+// SIMD-ONLY0-NEXT:    [[TMP2953:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2953]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4163:%.*]]
+// SIMD-ONLY0:       if.else4162:
+// SIMD-ONLY0-NEXT:    [[TMP2954:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2954]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4163]]
+// SIMD-ONLY0:       if.end4163:
+// SIMD-ONLY0-NEXT:    [[TMP2955:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2956:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4164:%.*]] = icmp eq i64 [[TMP2955]], [[TMP2956]]
+// SIMD-ONLY0-NEXT:    [[CONV4165:%.*]] = zext i1 [[CMP4164]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4166:%.*]] = sext i32 [[CONV4165]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4166]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2957:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4167:%.*]] = icmp ne i64 [[TMP2957]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4167]], label [[IF_THEN4168:%.*]], label [[IF_ELSE4169:%.*]]
+// SIMD-ONLY0:       if.then4168:
+// SIMD-ONLY0-NEXT:    [[TMP2958:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2958]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4170:%.*]]
+// SIMD-ONLY0:       if.else4169:
+// SIMD-ONLY0-NEXT:    [[TMP2959:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2959]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4170]]
+// SIMD-ONLY0:       if.end4170:
+// SIMD-ONLY0-NEXT:    [[TMP2960:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2960]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2961:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2962:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4171:%.*]] = icmp sgt i64 [[TMP2961]], [[TMP2962]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4171]], label [[IF_THEN4173:%.*]], label [[IF_END4174:%.*]]
+// SIMD-ONLY0:       if.then4173:
+// SIMD-ONLY0-NEXT:    [[TMP2963:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2963]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4174]]
+// SIMD-ONLY0:       if.end4174:
+// SIMD-ONLY0-NEXT:    [[TMP2964:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2964]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2965:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2966:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4175:%.*]] = icmp sgt i64 [[TMP2965]], [[TMP2966]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4175]], label [[IF_THEN4177:%.*]], label [[IF_END4178:%.*]]
+// SIMD-ONLY0:       if.then4177:
+// SIMD-ONLY0-NEXT:    [[TMP2967:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2967]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4178]]
+// SIMD-ONLY0:       if.end4178:
+// SIMD-ONLY0-NEXT:    [[TMP2968:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2968]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2969:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2970:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4179:%.*]] = icmp slt i64 [[TMP2969]], [[TMP2970]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4179]], label [[IF_THEN4181:%.*]], label [[IF_END4182:%.*]]
+// SIMD-ONLY0:       if.then4181:
+// SIMD-ONLY0-NEXT:    [[TMP2971:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2971]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4182]]
+// SIMD-ONLY0:       if.end4182:
+// SIMD-ONLY0-NEXT:    [[TMP2972:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2972]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2973:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2974:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4183:%.*]] = icmp slt i64 [[TMP2973]], [[TMP2974]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4183]], label [[IF_THEN4185:%.*]], label [[IF_END4186:%.*]]
+// SIMD-ONLY0:       if.then4185:
+// SIMD-ONLY0-NEXT:    [[TMP2975:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2975]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4186]]
+// SIMD-ONLY0:       if.end4186:
+// SIMD-ONLY0-NEXT:    [[TMP2976:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2976]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2977:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2978:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4187:%.*]] = icmp eq i64 [[TMP2977]], [[TMP2978]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4187]], label [[IF_THEN4189:%.*]], label [[IF_END4190:%.*]]
+// SIMD-ONLY0:       if.then4189:
+// SIMD-ONLY0-NEXT:    [[TMP2979:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2979]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4190]]
+// SIMD-ONLY0:       if.end4190:
+// SIMD-ONLY0-NEXT:    [[TMP2980:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2980]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2981:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2982:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4191:%.*]] = icmp eq i64 [[TMP2981]], [[TMP2982]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4191]], label [[IF_THEN4193:%.*]], label [[IF_END4194:%.*]]
+// SIMD-ONLY0:       if.then4193:
+// SIMD-ONLY0-NEXT:    [[TMP2983:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2983]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4194]]
+// SIMD-ONLY0:       if.end4194:
+// SIMD-ONLY0-NEXT:    [[TMP2984:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2985:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4195:%.*]] = icmp sgt i64 [[TMP2984]], [[TMP2985]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4195]], label [[IF_THEN4197:%.*]], label [[IF_END4198:%.*]]
+// SIMD-ONLY0:       if.then4197:
+// SIMD-ONLY0-NEXT:    [[TMP2986:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2986]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4198]]
+// SIMD-ONLY0:       if.end4198:
+// SIMD-ONLY0-NEXT:    [[TMP2987:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2987]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2988:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2989:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4199:%.*]] = icmp sgt i64 [[TMP2988]], [[TMP2989]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4199]], label [[IF_THEN4201:%.*]], label [[IF_END4202:%.*]]
+// SIMD-ONLY0:       if.then4201:
+// SIMD-ONLY0-NEXT:    [[TMP2990:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2990]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4202]]
+// SIMD-ONLY0:       if.end4202:
+// SIMD-ONLY0-NEXT:    [[TMP2991:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2991]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2992:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2993:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4203:%.*]] = icmp slt i64 [[TMP2992]], [[TMP2993]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4203]], label [[IF_THEN4205:%.*]], label [[IF_END4206:%.*]]
+// SIMD-ONLY0:       if.then4205:
+// SIMD-ONLY0-NEXT:    [[TMP2994:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2994]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4206]]
+// SIMD-ONLY0:       if.end4206:
+// SIMD-ONLY0-NEXT:    [[TMP2995:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2995]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2996:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2997:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4207:%.*]] = icmp slt i64 [[TMP2996]], [[TMP2997]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4207]], label [[IF_THEN4209:%.*]], label [[IF_END4210:%.*]]
+// SIMD-ONLY0:       if.then4209:
+// SIMD-ONLY0-NEXT:    [[TMP2998:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2998]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4210]]
+// SIMD-ONLY0:       if.end4210:
+// SIMD-ONLY0-NEXT:    [[TMP2999:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP2999]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3000:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3001:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4211:%.*]] = icmp eq i64 [[TMP3000]], [[TMP3001]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4211]], label [[IF_THEN4213:%.*]], label [[IF_END4214:%.*]]
+// SIMD-ONLY0:       if.then4213:
+// SIMD-ONLY0-NEXT:    [[TMP3002:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3002]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4214]]
+// SIMD-ONLY0:       if.end4214:
+// SIMD-ONLY0-NEXT:    [[TMP3003:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3003]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3004:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3005:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4215:%.*]] = icmp eq i64 [[TMP3004]], [[TMP3005]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4215]], label [[IF_THEN4217:%.*]], label [[IF_END4218:%.*]]
+// SIMD-ONLY0:       if.then4217:
+// SIMD-ONLY0-NEXT:    [[TMP3006:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3006]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4218]]
+// SIMD-ONLY0:       if.end4218:
+// SIMD-ONLY0-NEXT:    [[TMP3007:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3007]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3008:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3009:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4219:%.*]] = icmp eq i64 [[TMP3008]], [[TMP3009]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4219]], label [[IF_THEN4221:%.*]], label [[IF_ELSE4222:%.*]]
+// SIMD-ONLY0:       if.then4221:
+// SIMD-ONLY0-NEXT:    [[TMP3010:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3010]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4223:%.*]]
+// SIMD-ONLY0:       if.else4222:
+// SIMD-ONLY0-NEXT:    [[TMP3011:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3011]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4223]]
+// SIMD-ONLY0:       if.end4223:
+// SIMD-ONLY0-NEXT:    [[TMP3012:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3013:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4224:%.*]] = icmp eq i64 [[TMP3012]], [[TMP3013]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4224]], label [[IF_THEN4226:%.*]], label [[IF_ELSE4227:%.*]]
+// SIMD-ONLY0:       if.then4226:
+// SIMD-ONLY0-NEXT:    [[TMP3014:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3014]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4228:%.*]]
+// SIMD-ONLY0:       if.else4227:
+// SIMD-ONLY0-NEXT:    [[TMP3015:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3015]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4228]]
+// SIMD-ONLY0:       if.end4228:
+// SIMD-ONLY0-NEXT:    [[TMP3016:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3017:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4229:%.*]] = icmp eq i64 [[TMP3016]], [[TMP3017]]
+// SIMD-ONLY0-NEXT:    [[CONV4230:%.*]] = zext i1 [[CMP4229]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4231:%.*]] = sext i32 [[CONV4230]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4231]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3018:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4232:%.*]] = icmp ne i64 [[TMP3018]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4232]], label [[IF_THEN4233:%.*]], label [[IF_END4234:%.*]]
+// SIMD-ONLY0:       if.then4233:
+// SIMD-ONLY0-NEXT:    [[TMP3019:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3019]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4234]]
+// SIMD-ONLY0:       if.end4234:
+// SIMD-ONLY0-NEXT:    [[TMP3020:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3021:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4235:%.*]] = icmp eq i64 [[TMP3020]], [[TMP3021]]
+// SIMD-ONLY0-NEXT:    [[CONV4236:%.*]] = zext i1 [[CMP4235]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4237:%.*]] = sext i32 [[CONV4236]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4237]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3022:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4238:%.*]] = icmp ne i64 [[TMP3022]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4238]], label [[IF_THEN4239:%.*]], label [[IF_END4240:%.*]]
+// SIMD-ONLY0:       if.then4239:
+// SIMD-ONLY0-NEXT:    [[TMP3023:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3023]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4240]]
+// SIMD-ONLY0:       if.end4240:
+// SIMD-ONLY0-NEXT:    [[TMP3024:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3025:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4241:%.*]] = icmp eq i64 [[TMP3024]], [[TMP3025]]
+// SIMD-ONLY0-NEXT:    [[CONV4242:%.*]] = zext i1 [[CMP4241]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4243:%.*]] = sext i32 [[CONV4242]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4243]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3026:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4244:%.*]] = icmp ne i64 [[TMP3026]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4244]], label [[IF_THEN4245:%.*]], label [[IF_ELSE4246:%.*]]
+// SIMD-ONLY0:       if.then4245:
+// SIMD-ONLY0-NEXT:    [[TMP3027:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3027]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4247:%.*]]
+// SIMD-ONLY0:       if.else4246:
+// SIMD-ONLY0-NEXT:    [[TMP3028:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3028]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4247]]
+// SIMD-ONLY0:       if.end4247:
+// SIMD-ONLY0-NEXT:    [[TMP3029:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3030:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4248:%.*]] = icmp eq i64 [[TMP3029]], [[TMP3030]]
+// SIMD-ONLY0-NEXT:    [[CONV4249:%.*]] = zext i1 [[CMP4248]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4250:%.*]] = sext i32 [[CONV4249]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4250]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3031:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4251:%.*]] = icmp ne i64 [[TMP3031]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4251]], label [[IF_THEN4252:%.*]], label [[IF_ELSE4253:%.*]]
+// SIMD-ONLY0:       if.then4252:
+// SIMD-ONLY0-NEXT:    [[TMP3032:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3032]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4254:%.*]]
+// SIMD-ONLY0:       if.else4253:
+// SIMD-ONLY0-NEXT:    [[TMP3033:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3033]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4254]]
+// SIMD-ONLY0:       if.end4254:
+// SIMD-ONLY0-NEXT:    [[TMP3034:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3034]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3035:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3036:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4255:%.*]] = icmp sgt i64 [[TMP3035]], [[TMP3036]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4255]], label [[IF_THEN4257:%.*]], label [[IF_END4258:%.*]]
+// SIMD-ONLY0:       if.then4257:
+// SIMD-ONLY0-NEXT:    [[TMP3037:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3037]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4258]]
+// SIMD-ONLY0:       if.end4258:
+// SIMD-ONLY0-NEXT:    [[TMP3038:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3038]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3039:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3040:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4259:%.*]] = icmp sgt i64 [[TMP3039]], [[TMP3040]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4259]], label [[IF_THEN4261:%.*]], label [[IF_END4262:%.*]]
+// SIMD-ONLY0:       if.then4261:
+// SIMD-ONLY0-NEXT:    [[TMP3041:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3041]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4262]]
+// SIMD-ONLY0:       if.end4262:
+// SIMD-ONLY0-NEXT:    [[TMP3042:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3042]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3043:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3044:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4263:%.*]] = icmp slt i64 [[TMP3043]], [[TMP3044]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4263]], label [[IF_THEN4265:%.*]], label [[IF_END4266:%.*]]
+// SIMD-ONLY0:       if.then4265:
+// SIMD-ONLY0-NEXT:    [[TMP3045:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3045]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4266]]
+// SIMD-ONLY0:       if.end4266:
+// SIMD-ONLY0-NEXT:    [[TMP3046:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3046]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3047:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3048:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4267:%.*]] = icmp slt i64 [[TMP3047]], [[TMP3048]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4267]], label [[IF_THEN4269:%.*]], label [[IF_END4270:%.*]]
+// SIMD-ONLY0:       if.then4269:
+// SIMD-ONLY0-NEXT:    [[TMP3049:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3049]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4270]]
+// SIMD-ONLY0:       if.end4270:
+// SIMD-ONLY0-NEXT:    [[TMP3050:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3050]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3051:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3052:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4271:%.*]] = icmp eq i64 [[TMP3051]], [[TMP3052]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4271]], label [[IF_THEN4273:%.*]], label [[IF_END4274:%.*]]
+// SIMD-ONLY0:       if.then4273:
+// SIMD-ONLY0-NEXT:    [[TMP3053:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3053]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4274]]
+// SIMD-ONLY0:       if.end4274:
+// SIMD-ONLY0-NEXT:    [[TMP3054:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3054]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3055:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3056:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4275:%.*]] = icmp eq i64 [[TMP3055]], [[TMP3056]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4275]], label [[IF_THEN4277:%.*]], label [[IF_END4278:%.*]]
+// SIMD-ONLY0:       if.then4277:
+// SIMD-ONLY0-NEXT:    [[TMP3057:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3057]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4278]]
+// SIMD-ONLY0:       if.end4278:
+// SIMD-ONLY0-NEXT:    [[TMP3058:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3059:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4279:%.*]] = icmp sgt i64 [[TMP3058]], [[TMP3059]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4279]], label [[IF_THEN4281:%.*]], label [[IF_END4282:%.*]]
+// SIMD-ONLY0:       if.then4281:
+// SIMD-ONLY0-NEXT:    [[TMP3060:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3060]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4282]]
+// SIMD-ONLY0:       if.end4282:
+// SIMD-ONLY0-NEXT:    [[TMP3061:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3061]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3062:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3063:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4283:%.*]] = icmp sgt i64 [[TMP3062]], [[TMP3063]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4283]], label [[IF_THEN4285:%.*]], label [[IF_END4286:%.*]]
+// SIMD-ONLY0:       if.then4285:
+// SIMD-ONLY0-NEXT:    [[TMP3064:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3064]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4286]]
+// SIMD-ONLY0:       if.end4286:
+// SIMD-ONLY0-NEXT:    [[TMP3065:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3065]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3066:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3067:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4287:%.*]] = icmp slt i64 [[TMP3066]], [[TMP3067]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4287]], label [[IF_THEN4289:%.*]], label [[IF_END4290:%.*]]
+// SIMD-ONLY0:       if.then4289:
+// SIMD-ONLY0-NEXT:    [[TMP3068:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3068]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4290]]
+// SIMD-ONLY0:       if.end4290:
+// SIMD-ONLY0-NEXT:    [[TMP3069:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3069]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3070:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3071:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4291:%.*]] = icmp slt i64 [[TMP3070]], [[TMP3071]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4291]], label [[IF_THEN4293:%.*]], label [[IF_END4294:%.*]]
+// SIMD-ONLY0:       if.then4293:
+// SIMD-ONLY0-NEXT:    [[TMP3072:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3072]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4294]]
+// SIMD-ONLY0:       if.end4294:
+// SIMD-ONLY0-NEXT:    [[TMP3073:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3073]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3074:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3075:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4295:%.*]] = icmp eq i64 [[TMP3074]], [[TMP3075]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4295]], label [[IF_THEN4297:%.*]], label [[IF_END4298:%.*]]
+// SIMD-ONLY0:       if.then4297:
+// SIMD-ONLY0-NEXT:    [[TMP3076:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3076]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4298]]
+// SIMD-ONLY0:       if.end4298:
+// SIMD-ONLY0-NEXT:    [[TMP3077:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3077]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3078:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3079:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4299:%.*]] = icmp eq i64 [[TMP3078]], [[TMP3079]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4299]], label [[IF_THEN4301:%.*]], label [[IF_END4302:%.*]]
+// SIMD-ONLY0:       if.then4301:
+// SIMD-ONLY0-NEXT:    [[TMP3080:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3080]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4302]]
+// SIMD-ONLY0:       if.end4302:
+// SIMD-ONLY0-NEXT:    [[TMP3081:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3081]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3082:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3083:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4303:%.*]] = icmp eq i64 [[TMP3082]], [[TMP3083]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4303]], label [[IF_THEN4305:%.*]], label [[IF_ELSE4306:%.*]]
+// SIMD-ONLY0:       if.then4305:
+// SIMD-ONLY0-NEXT:    [[TMP3084:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3084]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4307:%.*]]
+// SIMD-ONLY0:       if.else4306:
+// SIMD-ONLY0-NEXT:    [[TMP3085:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3085]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4307]]
+// SIMD-ONLY0:       if.end4307:
+// SIMD-ONLY0-NEXT:    [[TMP3086:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3087:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4308:%.*]] = icmp eq i64 [[TMP3086]], [[TMP3087]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4308]], label [[IF_THEN4310:%.*]], label [[IF_ELSE4311:%.*]]
+// SIMD-ONLY0:       if.then4310:
+// SIMD-ONLY0-NEXT:    [[TMP3088:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3088]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4312:%.*]]
+// SIMD-ONLY0:       if.else4311:
+// SIMD-ONLY0-NEXT:    [[TMP3089:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3089]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4312]]
+// SIMD-ONLY0:       if.end4312:
+// SIMD-ONLY0-NEXT:    [[TMP3090:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3091:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4313:%.*]] = icmp eq i64 [[TMP3090]], [[TMP3091]]
+// SIMD-ONLY0-NEXT:    [[CONV4314:%.*]] = zext i1 [[CMP4313]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4315:%.*]] = sext i32 [[CONV4314]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4315]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3092:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4316:%.*]] = icmp ne i64 [[TMP3092]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4316]], label [[IF_THEN4317:%.*]], label [[IF_END4318:%.*]]
+// SIMD-ONLY0:       if.then4317:
+// SIMD-ONLY0-NEXT:    [[TMP3093:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3093]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4318]]
+// SIMD-ONLY0:       if.end4318:
+// SIMD-ONLY0-NEXT:    [[TMP3094:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3095:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4319:%.*]] = icmp eq i64 [[TMP3094]], [[TMP3095]]
+// SIMD-ONLY0-NEXT:    [[CONV4320:%.*]] = zext i1 [[CMP4319]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4321:%.*]] = sext i32 [[CONV4320]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4321]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3096:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4322:%.*]] = icmp ne i64 [[TMP3096]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4322]], label [[IF_THEN4323:%.*]], label [[IF_END4324:%.*]]
+// SIMD-ONLY0:       if.then4323:
+// SIMD-ONLY0-NEXT:    [[TMP3097:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3097]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4324]]
+// SIMD-ONLY0:       if.end4324:
+// SIMD-ONLY0-NEXT:    [[TMP3098:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3099:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4325:%.*]] = icmp eq i64 [[TMP3098]], [[TMP3099]]
+// SIMD-ONLY0-NEXT:    [[CONV4326:%.*]] = zext i1 [[CMP4325]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4327:%.*]] = sext i32 [[CONV4326]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4327]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3100:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4328:%.*]] = icmp ne i64 [[TMP3100]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4328]], label [[IF_THEN4329:%.*]], label [[IF_ELSE4330:%.*]]
+// SIMD-ONLY0:       if.then4329:
+// SIMD-ONLY0-NEXT:    [[TMP3101:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3101]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4331:%.*]]
+// SIMD-ONLY0:       if.else4330:
+// SIMD-ONLY0-NEXT:    [[TMP3102:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3102]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4331]]
+// SIMD-ONLY0:       if.end4331:
+// SIMD-ONLY0-NEXT:    [[TMP3103:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3104:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4332:%.*]] = icmp eq i64 [[TMP3103]], [[TMP3104]]
+// SIMD-ONLY0-NEXT:    [[CONV4333:%.*]] = zext i1 [[CMP4332]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4334:%.*]] = sext i32 [[CONV4333]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4334]], ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3105:%.*]] = load i64, ptr [[LR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4335:%.*]] = icmp ne i64 [[TMP3105]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4335]], label [[IF_THEN4336:%.*]], label [[IF_ELSE4337:%.*]]
+// SIMD-ONLY0:       if.then4336:
+// SIMD-ONLY0-NEXT:    [[TMP3106:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3106]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4338:%.*]]
+// SIMD-ONLY0:       if.else4337:
+// SIMD-ONLY0-NEXT:    [[TMP3107:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3107]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4338]]
+// SIMD-ONLY0:       if.end4338:
+// SIMD-ONLY0-NEXT:    [[TMP3108:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3108]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3109:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3110:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4339:%.*]] = icmp ugt i64 [[TMP3109]], [[TMP3110]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4339]], label [[IF_THEN4341:%.*]], label [[IF_END4342:%.*]]
+// SIMD-ONLY0:       if.then4341:
+// SIMD-ONLY0-NEXT:    [[TMP3111:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3111]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4342]]
+// SIMD-ONLY0:       if.end4342:
+// SIMD-ONLY0-NEXT:    [[TMP3112:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3112]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3113:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3114:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4343:%.*]] = icmp ugt i64 [[TMP3113]], [[TMP3114]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4343]], label [[IF_THEN4345:%.*]], label [[IF_END4346:%.*]]
+// SIMD-ONLY0:       if.then4345:
+// SIMD-ONLY0-NEXT:    [[TMP3115:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3115]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4346]]
+// SIMD-ONLY0:       if.end4346:
+// SIMD-ONLY0-NEXT:    [[TMP3116:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3116]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3117:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3118:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4347:%.*]] = icmp ult i64 [[TMP3117]], [[TMP3118]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4347]], label [[IF_THEN4349:%.*]], label [[IF_END4350:%.*]]
+// SIMD-ONLY0:       if.then4349:
+// SIMD-ONLY0-NEXT:    [[TMP3119:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3119]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4350]]
+// SIMD-ONLY0:       if.end4350:
+// SIMD-ONLY0-NEXT:    [[TMP3120:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3120]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3121:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3122:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4351:%.*]] = icmp ult i64 [[TMP3121]], [[TMP3122]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4351]], label [[IF_THEN4353:%.*]], label [[IF_END4354:%.*]]
+// SIMD-ONLY0:       if.then4353:
+// SIMD-ONLY0-NEXT:    [[TMP3123:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3123]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4354]]
+// SIMD-ONLY0:       if.end4354:
+// SIMD-ONLY0-NEXT:    [[TMP3124:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3124]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3125:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3126:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4355:%.*]] = icmp eq i64 [[TMP3125]], [[TMP3126]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4355]], label [[IF_THEN4357:%.*]], label [[IF_END4358:%.*]]
+// SIMD-ONLY0:       if.then4357:
+// SIMD-ONLY0-NEXT:    [[TMP3127:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3127]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4358]]
+// SIMD-ONLY0:       if.end4358:
+// SIMD-ONLY0-NEXT:    [[TMP3128:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3128]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3129:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3130:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4359:%.*]] = icmp eq i64 [[TMP3129]], [[TMP3130]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4359]], label [[IF_THEN4361:%.*]], label [[IF_END4362:%.*]]
+// SIMD-ONLY0:       if.then4361:
+// SIMD-ONLY0-NEXT:    [[TMP3131:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3131]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4362]]
+// SIMD-ONLY0:       if.end4362:
+// SIMD-ONLY0-NEXT:    [[TMP3132:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3133:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4363:%.*]] = icmp ugt i64 [[TMP3132]], [[TMP3133]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4363]], label [[IF_THEN4365:%.*]], label [[IF_END4366:%.*]]
+// SIMD-ONLY0:       if.then4365:
+// SIMD-ONLY0-NEXT:    [[TMP3134:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3134]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4366]]
+// SIMD-ONLY0:       if.end4366:
+// SIMD-ONLY0-NEXT:    [[TMP3135:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3135]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3136:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3137:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4367:%.*]] = icmp ugt i64 [[TMP3136]], [[TMP3137]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4367]], label [[IF_THEN4369:%.*]], label [[IF_END4370:%.*]]
+// SIMD-ONLY0:       if.then4369:
+// SIMD-ONLY0-NEXT:    [[TMP3138:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3138]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4370]]
+// SIMD-ONLY0:       if.end4370:
+// SIMD-ONLY0-NEXT:    [[TMP3139:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3139]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3140:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3141:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4371:%.*]] = icmp ult i64 [[TMP3140]], [[TMP3141]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4371]], label [[IF_THEN4373:%.*]], label [[IF_END4374:%.*]]
+// SIMD-ONLY0:       if.then4373:
+// SIMD-ONLY0-NEXT:    [[TMP3142:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3142]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4374]]
+// SIMD-ONLY0:       if.end4374:
+// SIMD-ONLY0-NEXT:    [[TMP3143:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3143]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3144:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3145:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4375:%.*]] = icmp ult i64 [[TMP3144]], [[TMP3145]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4375]], label [[IF_THEN4377:%.*]], label [[IF_END4378:%.*]]
+// SIMD-ONLY0:       if.then4377:
+// SIMD-ONLY0-NEXT:    [[TMP3146:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3146]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4378]]
+// SIMD-ONLY0:       if.end4378:
+// SIMD-ONLY0-NEXT:    [[TMP3147:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3147]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3148:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3149:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4379:%.*]] = icmp eq i64 [[TMP3148]], [[TMP3149]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4379]], label [[IF_THEN4381:%.*]], label [[IF_END4382:%.*]]
+// SIMD-ONLY0:       if.then4381:
+// SIMD-ONLY0-NEXT:    [[TMP3150:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3150]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4382]]
+// SIMD-ONLY0:       if.end4382:
+// SIMD-ONLY0-NEXT:    [[TMP3151:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3151]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3152:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3153:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4383:%.*]] = icmp eq i64 [[TMP3152]], [[TMP3153]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4383]], label [[IF_THEN4385:%.*]], label [[IF_END4386:%.*]]
+// SIMD-ONLY0:       if.then4385:
+// SIMD-ONLY0-NEXT:    [[TMP3154:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3154]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4386]]
+// SIMD-ONLY0:       if.end4386:
+// SIMD-ONLY0-NEXT:    [[TMP3155:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3155]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3156:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3157:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4387:%.*]] = icmp eq i64 [[TMP3156]], [[TMP3157]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4387]], label [[IF_THEN4389:%.*]], label [[IF_ELSE4390:%.*]]
+// SIMD-ONLY0:       if.then4389:
+// SIMD-ONLY0-NEXT:    [[TMP3158:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3158]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4391:%.*]]
+// SIMD-ONLY0:       if.else4390:
+// SIMD-ONLY0-NEXT:    [[TMP3159:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3159]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4391]]
+// SIMD-ONLY0:       if.end4391:
+// SIMD-ONLY0-NEXT:    [[TMP3160:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3161:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4392:%.*]] = icmp eq i64 [[TMP3160]], [[TMP3161]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4392]], label [[IF_THEN4394:%.*]], label [[IF_ELSE4395:%.*]]
+// SIMD-ONLY0:       if.then4394:
+// SIMD-ONLY0-NEXT:    [[TMP3162:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3162]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4396:%.*]]
+// SIMD-ONLY0:       if.else4395:
+// SIMD-ONLY0-NEXT:    [[TMP3163:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3163]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4396]]
+// SIMD-ONLY0:       if.end4396:
+// SIMD-ONLY0-NEXT:    [[TMP3164:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3165:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4397:%.*]] = icmp eq i64 [[TMP3164]], [[TMP3165]]
+// SIMD-ONLY0-NEXT:    [[CONV4398:%.*]] = zext i1 [[CMP4397]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4399:%.*]] = sext i32 [[CONV4398]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4399]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3166:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4400:%.*]] = icmp ne i64 [[TMP3166]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4400]], label [[IF_THEN4401:%.*]], label [[IF_END4402:%.*]]
+// SIMD-ONLY0:       if.then4401:
+// SIMD-ONLY0-NEXT:    [[TMP3167:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3167]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4402]]
+// SIMD-ONLY0:       if.end4402:
+// SIMD-ONLY0-NEXT:    [[TMP3168:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3169:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4403:%.*]] = icmp eq i64 [[TMP3168]], [[TMP3169]]
+// SIMD-ONLY0-NEXT:    [[CONV4404:%.*]] = zext i1 [[CMP4403]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4405:%.*]] = sext i32 [[CONV4404]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4405]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3170:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4406:%.*]] = icmp ne i64 [[TMP3170]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4406]], label [[IF_THEN4407:%.*]], label [[IF_END4408:%.*]]
+// SIMD-ONLY0:       if.then4407:
+// SIMD-ONLY0-NEXT:    [[TMP3171:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3171]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4408]]
+// SIMD-ONLY0:       if.end4408:
+// SIMD-ONLY0-NEXT:    [[TMP3172:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3173:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4409:%.*]] = icmp eq i64 [[TMP3172]], [[TMP3173]]
+// SIMD-ONLY0-NEXT:    [[CONV4410:%.*]] = zext i1 [[CMP4409]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4411:%.*]] = sext i32 [[CONV4410]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4411]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3174:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4412:%.*]] = icmp ne i64 [[TMP3174]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4412]], label [[IF_THEN4413:%.*]], label [[IF_ELSE4414:%.*]]
+// SIMD-ONLY0:       if.then4413:
+// SIMD-ONLY0-NEXT:    [[TMP3175:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3175]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4415:%.*]]
+// SIMD-ONLY0:       if.else4414:
+// SIMD-ONLY0-NEXT:    [[TMP3176:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3176]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4415]]
+// SIMD-ONLY0:       if.end4415:
+// SIMD-ONLY0-NEXT:    [[TMP3177:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3178:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4416:%.*]] = icmp eq i64 [[TMP3177]], [[TMP3178]]
+// SIMD-ONLY0-NEXT:    [[CONV4417:%.*]] = zext i1 [[CMP4416]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4418:%.*]] = sext i32 [[CONV4417]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4418]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3179:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4419:%.*]] = icmp ne i64 [[TMP3179]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4419]], label [[IF_THEN4420:%.*]], label [[IF_ELSE4421:%.*]]
+// SIMD-ONLY0:       if.then4420:
+// SIMD-ONLY0-NEXT:    [[TMP3180:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3180]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4422:%.*]]
+// SIMD-ONLY0:       if.else4421:
+// SIMD-ONLY0-NEXT:    [[TMP3181:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3181]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4422]]
+// SIMD-ONLY0:       if.end4422:
+// SIMD-ONLY0-NEXT:    [[TMP3182:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3182]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3183:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3184:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4423:%.*]] = icmp ugt i64 [[TMP3183]], [[TMP3184]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4423]], label [[IF_THEN4425:%.*]], label [[IF_END4426:%.*]]
+// SIMD-ONLY0:       if.then4425:
+// SIMD-ONLY0-NEXT:    [[TMP3185:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3185]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4426]]
+// SIMD-ONLY0:       if.end4426:
+// SIMD-ONLY0-NEXT:    [[TMP3186:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3186]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3187:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3188:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4427:%.*]] = icmp ugt i64 [[TMP3187]], [[TMP3188]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4427]], label [[IF_THEN4429:%.*]], label [[IF_END4430:%.*]]
+// SIMD-ONLY0:       if.then4429:
+// SIMD-ONLY0-NEXT:    [[TMP3189:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3189]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4430]]
+// SIMD-ONLY0:       if.end4430:
+// SIMD-ONLY0-NEXT:    [[TMP3190:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3190]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3191:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3192:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4431:%.*]] = icmp ult i64 [[TMP3191]], [[TMP3192]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4431]], label [[IF_THEN4433:%.*]], label [[IF_END4434:%.*]]
+// SIMD-ONLY0:       if.then4433:
+// SIMD-ONLY0-NEXT:    [[TMP3193:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3193]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4434]]
+// SIMD-ONLY0:       if.end4434:
+// SIMD-ONLY0-NEXT:    [[TMP3194:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3194]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3195:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3196:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4435:%.*]] = icmp ult i64 [[TMP3195]], [[TMP3196]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4435]], label [[IF_THEN4437:%.*]], label [[IF_END4438:%.*]]
+// SIMD-ONLY0:       if.then4437:
+// SIMD-ONLY0-NEXT:    [[TMP3197:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3197]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4438]]
+// SIMD-ONLY0:       if.end4438:
+// SIMD-ONLY0-NEXT:    [[TMP3198:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3198]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3199:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3200:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4439:%.*]] = icmp eq i64 [[TMP3199]], [[TMP3200]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4439]], label [[IF_THEN4441:%.*]], label [[IF_END4442:%.*]]
+// SIMD-ONLY0:       if.then4441:
+// SIMD-ONLY0-NEXT:    [[TMP3201:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3201]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4442]]
+// SIMD-ONLY0:       if.end4442:
+// SIMD-ONLY0-NEXT:    [[TMP3202:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3202]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3203:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3204:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4443:%.*]] = icmp eq i64 [[TMP3203]], [[TMP3204]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4443]], label [[IF_THEN4445:%.*]], label [[IF_END4446:%.*]]
+// SIMD-ONLY0:       if.then4445:
+// SIMD-ONLY0-NEXT:    [[TMP3205:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3205]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4446]]
+// SIMD-ONLY0:       if.end4446:
+// SIMD-ONLY0-NEXT:    [[TMP3206:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3207:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4447:%.*]] = icmp ugt i64 [[TMP3206]], [[TMP3207]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4447]], label [[IF_THEN4449:%.*]], label [[IF_END4450:%.*]]
+// SIMD-ONLY0:       if.then4449:
+// SIMD-ONLY0-NEXT:    [[TMP3208:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3208]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4450]]
+// SIMD-ONLY0:       if.end4450:
+// SIMD-ONLY0-NEXT:    [[TMP3209:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3209]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3210:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3211:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4451:%.*]] = icmp ugt i64 [[TMP3210]], [[TMP3211]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4451]], label [[IF_THEN4453:%.*]], label [[IF_END4454:%.*]]
+// SIMD-ONLY0:       if.then4453:
+// SIMD-ONLY0-NEXT:    [[TMP3212:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3212]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4454]]
+// SIMD-ONLY0:       if.end4454:
+// SIMD-ONLY0-NEXT:    [[TMP3213:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3213]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3214:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3215:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4455:%.*]] = icmp ult i64 [[TMP3214]], [[TMP3215]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4455]], label [[IF_THEN4457:%.*]], label [[IF_END4458:%.*]]
+// SIMD-ONLY0:       if.then4457:
+// SIMD-ONLY0-NEXT:    [[TMP3216:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3216]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4458]]
+// SIMD-ONLY0:       if.end4458:
+// SIMD-ONLY0-NEXT:    [[TMP3217:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3217]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3218:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3219:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4459:%.*]] = icmp ult i64 [[TMP3218]], [[TMP3219]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4459]], label [[IF_THEN4461:%.*]], label [[IF_END4462:%.*]]
+// SIMD-ONLY0:       if.then4461:
+// SIMD-ONLY0-NEXT:    [[TMP3220:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3220]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4462]]
+// SIMD-ONLY0:       if.end4462:
+// SIMD-ONLY0-NEXT:    [[TMP3221:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3221]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3222:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3223:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4463:%.*]] = icmp eq i64 [[TMP3222]], [[TMP3223]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4463]], label [[IF_THEN4465:%.*]], label [[IF_END4466:%.*]]
+// SIMD-ONLY0:       if.then4465:
+// SIMD-ONLY0-NEXT:    [[TMP3224:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3224]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4466]]
+// SIMD-ONLY0:       if.end4466:
+// SIMD-ONLY0-NEXT:    [[TMP3225:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3225]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3226:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3227:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4467:%.*]] = icmp eq i64 [[TMP3226]], [[TMP3227]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4467]], label [[IF_THEN4469:%.*]], label [[IF_END4470:%.*]]
+// SIMD-ONLY0:       if.then4469:
+// SIMD-ONLY0-NEXT:    [[TMP3228:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3228]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4470]]
+// SIMD-ONLY0:       if.end4470:
+// SIMD-ONLY0-NEXT:    [[TMP3229:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3229]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3230:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3231:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4471:%.*]] = icmp eq i64 [[TMP3230]], [[TMP3231]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4471]], label [[IF_THEN4473:%.*]], label [[IF_ELSE4474:%.*]]
+// SIMD-ONLY0:       if.then4473:
+// SIMD-ONLY0-NEXT:    [[TMP3232:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3232]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4475:%.*]]
+// SIMD-ONLY0:       if.else4474:
+// SIMD-ONLY0-NEXT:    [[TMP3233:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3233]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4475]]
+// SIMD-ONLY0:       if.end4475:
+// SIMD-ONLY0-NEXT:    [[TMP3234:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3235:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4476:%.*]] = icmp eq i64 [[TMP3234]], [[TMP3235]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4476]], label [[IF_THEN4478:%.*]], label [[IF_ELSE4479:%.*]]
+// SIMD-ONLY0:       if.then4478:
+// SIMD-ONLY0-NEXT:    [[TMP3236:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3236]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4480:%.*]]
+// SIMD-ONLY0:       if.else4479:
+// SIMD-ONLY0-NEXT:    [[TMP3237:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3237]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4480]]
+// SIMD-ONLY0:       if.end4480:
+// SIMD-ONLY0-NEXT:    [[TMP3238:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3239:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4481:%.*]] = icmp eq i64 [[TMP3238]], [[TMP3239]]
+// SIMD-ONLY0-NEXT:    [[CONV4482:%.*]] = zext i1 [[CMP4481]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4483:%.*]] = sext i32 [[CONV4482]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4483]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3240:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4484:%.*]] = icmp ne i64 [[TMP3240]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4484]], label [[IF_THEN4485:%.*]], label [[IF_END4486:%.*]]
+// SIMD-ONLY0:       if.then4485:
+// SIMD-ONLY0-NEXT:    [[TMP3241:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3241]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4486]]
+// SIMD-ONLY0:       if.end4486:
+// SIMD-ONLY0-NEXT:    [[TMP3242:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3243:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4487:%.*]] = icmp eq i64 [[TMP3242]], [[TMP3243]]
+// SIMD-ONLY0-NEXT:    [[CONV4488:%.*]] = zext i1 [[CMP4487]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4489:%.*]] = sext i32 [[CONV4488]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4489]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3244:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4490:%.*]] = icmp ne i64 [[TMP3244]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4490]], label [[IF_THEN4491:%.*]], label [[IF_END4492:%.*]]
+// SIMD-ONLY0:       if.then4491:
+// SIMD-ONLY0-NEXT:    [[TMP3245:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3245]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4492]]
+// SIMD-ONLY0:       if.end4492:
+// SIMD-ONLY0-NEXT:    [[TMP3246:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3247:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4493:%.*]] = icmp eq i64 [[TMP3246]], [[TMP3247]]
+// SIMD-ONLY0-NEXT:    [[CONV4494:%.*]] = zext i1 [[CMP4493]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4495:%.*]] = sext i32 [[CONV4494]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4495]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3248:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4496:%.*]] = icmp ne i64 [[TMP3248]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4496]], label [[IF_THEN4497:%.*]], label [[IF_ELSE4498:%.*]]
+// SIMD-ONLY0:       if.then4497:
+// SIMD-ONLY0-NEXT:    [[TMP3249:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3249]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4499:%.*]]
+// SIMD-ONLY0:       if.else4498:
+// SIMD-ONLY0-NEXT:    [[TMP3250:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3250]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4499]]
+// SIMD-ONLY0:       if.end4499:
+// SIMD-ONLY0-NEXT:    [[TMP3251:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3252:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4500:%.*]] = icmp eq i64 [[TMP3251]], [[TMP3252]]
+// SIMD-ONLY0-NEXT:    [[CONV4501:%.*]] = zext i1 [[CMP4500]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4502:%.*]] = sext i32 [[CONV4501]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4502]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3253:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4503:%.*]] = icmp ne i64 [[TMP3253]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4503]], label [[IF_THEN4504:%.*]], label [[IF_ELSE4505:%.*]]
+// SIMD-ONLY0:       if.then4504:
+// SIMD-ONLY0-NEXT:    [[TMP3254:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3254]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4506:%.*]]
+// SIMD-ONLY0:       if.else4505:
+// SIMD-ONLY0-NEXT:    [[TMP3255:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3255]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4506]]
+// SIMD-ONLY0:       if.end4506:
+// SIMD-ONLY0-NEXT:    [[TMP3256:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3256]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3257:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3258:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4507:%.*]] = icmp ugt i64 [[TMP3257]], [[TMP3258]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4507]], label [[IF_THEN4509:%.*]], label [[IF_END4510:%.*]]
+// SIMD-ONLY0:       if.then4509:
+// SIMD-ONLY0-NEXT:    [[TMP3259:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3259]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4510]]
+// SIMD-ONLY0:       if.end4510:
+// SIMD-ONLY0-NEXT:    [[TMP3260:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3260]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3261:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3262:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4511:%.*]] = icmp ugt i64 [[TMP3261]], [[TMP3262]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4511]], label [[IF_THEN4513:%.*]], label [[IF_END4514:%.*]]
+// SIMD-ONLY0:       if.then4513:
+// SIMD-ONLY0-NEXT:    [[TMP3263:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3263]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4514]]
+// SIMD-ONLY0:       if.end4514:
+// SIMD-ONLY0-NEXT:    [[TMP3264:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3264]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3265:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3266:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4515:%.*]] = icmp ult i64 [[TMP3265]], [[TMP3266]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4515]], label [[IF_THEN4517:%.*]], label [[IF_END4518:%.*]]
+// SIMD-ONLY0:       if.then4517:
+// SIMD-ONLY0-NEXT:    [[TMP3267:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3267]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4518]]
+// SIMD-ONLY0:       if.end4518:
+// SIMD-ONLY0-NEXT:    [[TMP3268:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3268]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3269:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3270:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4519:%.*]] = icmp ult i64 [[TMP3269]], [[TMP3270]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4519]], label [[IF_THEN4521:%.*]], label [[IF_END4522:%.*]]
+// SIMD-ONLY0:       if.then4521:
+// SIMD-ONLY0-NEXT:    [[TMP3271:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3271]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4522]]
+// SIMD-ONLY0:       if.end4522:
+// SIMD-ONLY0-NEXT:    [[TMP3272:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3272]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3273:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3274:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4523:%.*]] = icmp eq i64 [[TMP3273]], [[TMP3274]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4523]], label [[IF_THEN4525:%.*]], label [[IF_END4526:%.*]]
+// SIMD-ONLY0:       if.then4525:
+// SIMD-ONLY0-NEXT:    [[TMP3275:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3275]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4526]]
+// SIMD-ONLY0:       if.end4526:
+// SIMD-ONLY0-NEXT:    [[TMP3276:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3276]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3277:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3278:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4527:%.*]] = icmp eq i64 [[TMP3277]], [[TMP3278]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4527]], label [[IF_THEN4529:%.*]], label [[IF_END4530:%.*]]
+// SIMD-ONLY0:       if.then4529:
+// SIMD-ONLY0-NEXT:    [[TMP3279:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3279]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4530]]
+// SIMD-ONLY0:       if.end4530:
+// SIMD-ONLY0-NEXT:    [[TMP3280:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3281:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4531:%.*]] = icmp ugt i64 [[TMP3280]], [[TMP3281]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4531]], label [[IF_THEN4533:%.*]], label [[IF_END4534:%.*]]
+// SIMD-ONLY0:       if.then4533:
+// SIMD-ONLY0-NEXT:    [[TMP3282:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3282]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4534]]
+// SIMD-ONLY0:       if.end4534:
+// SIMD-ONLY0-NEXT:    [[TMP3283:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3283]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3284:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3285:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4535:%.*]] = icmp ugt i64 [[TMP3284]], [[TMP3285]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4535]], label [[IF_THEN4537:%.*]], label [[IF_END4538:%.*]]
+// SIMD-ONLY0:       if.then4537:
+// SIMD-ONLY0-NEXT:    [[TMP3286:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3286]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4538]]
+// SIMD-ONLY0:       if.end4538:
+// SIMD-ONLY0-NEXT:    [[TMP3287:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3287]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3288:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3289:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4539:%.*]] = icmp ult i64 [[TMP3288]], [[TMP3289]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4539]], label [[IF_THEN4541:%.*]], label [[IF_END4542:%.*]]
+// SIMD-ONLY0:       if.then4541:
+// SIMD-ONLY0-NEXT:    [[TMP3290:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3290]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4542]]
+// SIMD-ONLY0:       if.end4542:
+// SIMD-ONLY0-NEXT:    [[TMP3291:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3291]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3292:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3293:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4543:%.*]] = icmp ult i64 [[TMP3292]], [[TMP3293]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4543]], label [[IF_THEN4545:%.*]], label [[IF_END4546:%.*]]
+// SIMD-ONLY0:       if.then4545:
+// SIMD-ONLY0-NEXT:    [[TMP3294:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3294]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4546]]
+// SIMD-ONLY0:       if.end4546:
+// SIMD-ONLY0-NEXT:    [[TMP3295:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3295]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3296:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3297:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4547:%.*]] = icmp eq i64 [[TMP3296]], [[TMP3297]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4547]], label [[IF_THEN4549:%.*]], label [[IF_END4550:%.*]]
+// SIMD-ONLY0:       if.then4549:
+// SIMD-ONLY0-NEXT:    [[TMP3298:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3298]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4550]]
+// SIMD-ONLY0:       if.end4550:
+// SIMD-ONLY0-NEXT:    [[TMP3299:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3299]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3300:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3301:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4551:%.*]] = icmp eq i64 [[TMP3300]], [[TMP3301]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4551]], label [[IF_THEN4553:%.*]], label [[IF_END4554:%.*]]
+// SIMD-ONLY0:       if.then4553:
+// SIMD-ONLY0-NEXT:    [[TMP3302:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3302]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4554]]
+// SIMD-ONLY0:       if.end4554:
+// SIMD-ONLY0-NEXT:    [[TMP3303:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3303]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3304:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3305:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4555:%.*]] = icmp eq i64 [[TMP3304]], [[TMP3305]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4555]], label [[IF_THEN4557:%.*]], label [[IF_ELSE4558:%.*]]
+// SIMD-ONLY0:       if.then4557:
+// SIMD-ONLY0-NEXT:    [[TMP3306:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3306]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4559:%.*]]
+// SIMD-ONLY0:       if.else4558:
+// SIMD-ONLY0-NEXT:    [[TMP3307:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3307]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4559]]
+// SIMD-ONLY0:       if.end4559:
+// SIMD-ONLY0-NEXT:    [[TMP3308:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3309:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4560:%.*]] = icmp eq i64 [[TMP3308]], [[TMP3309]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4560]], label [[IF_THEN4562:%.*]], label [[IF_ELSE4563:%.*]]
+// SIMD-ONLY0:       if.then4562:
+// SIMD-ONLY0-NEXT:    [[TMP3310:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3310]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4564:%.*]]
+// SIMD-ONLY0:       if.else4563:
+// SIMD-ONLY0-NEXT:    [[TMP3311:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3311]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4564]]
+// SIMD-ONLY0:       if.end4564:
+// SIMD-ONLY0-NEXT:    [[TMP3312:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3313:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4565:%.*]] = icmp eq i64 [[TMP3312]], [[TMP3313]]
+// SIMD-ONLY0-NEXT:    [[CONV4566:%.*]] = zext i1 [[CMP4565]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4567:%.*]] = sext i32 [[CONV4566]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4567]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3314:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4568:%.*]] = icmp ne i64 [[TMP3314]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4568]], label [[IF_THEN4569:%.*]], label [[IF_END4570:%.*]]
+// SIMD-ONLY0:       if.then4569:
+// SIMD-ONLY0-NEXT:    [[TMP3315:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3315]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4570]]
+// SIMD-ONLY0:       if.end4570:
+// SIMD-ONLY0-NEXT:    [[TMP3316:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3317:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4571:%.*]] = icmp eq i64 [[TMP3316]], [[TMP3317]]
+// SIMD-ONLY0-NEXT:    [[CONV4572:%.*]] = zext i1 [[CMP4571]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4573:%.*]] = sext i32 [[CONV4572]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4573]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3318:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4574:%.*]] = icmp ne i64 [[TMP3318]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4574]], label [[IF_THEN4575:%.*]], label [[IF_END4576:%.*]]
+// SIMD-ONLY0:       if.then4575:
+// SIMD-ONLY0-NEXT:    [[TMP3319:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3319]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4576]]
+// SIMD-ONLY0:       if.end4576:
+// SIMD-ONLY0-NEXT:    [[TMP3320:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3321:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4577:%.*]] = icmp eq i64 [[TMP3320]], [[TMP3321]]
+// SIMD-ONLY0-NEXT:    [[CONV4578:%.*]] = zext i1 [[CMP4577]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4579:%.*]] = sext i32 [[CONV4578]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4579]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3322:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4580:%.*]] = icmp ne i64 [[TMP3322]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4580]], label [[IF_THEN4581:%.*]], label [[IF_ELSE4582:%.*]]
+// SIMD-ONLY0:       if.then4581:
+// SIMD-ONLY0-NEXT:    [[TMP3323:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3323]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4583:%.*]]
+// SIMD-ONLY0:       if.else4582:
+// SIMD-ONLY0-NEXT:    [[TMP3324:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3324]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4583]]
+// SIMD-ONLY0:       if.end4583:
+// SIMD-ONLY0-NEXT:    [[TMP3325:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3326:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4584:%.*]] = icmp eq i64 [[TMP3325]], [[TMP3326]]
+// SIMD-ONLY0-NEXT:    [[CONV4585:%.*]] = zext i1 [[CMP4584]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4586:%.*]] = sext i32 [[CONV4585]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4586]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3327:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4587:%.*]] = icmp ne i64 [[TMP3327]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4587]], label [[IF_THEN4588:%.*]], label [[IF_ELSE4589:%.*]]
+// SIMD-ONLY0:       if.then4588:
+// SIMD-ONLY0-NEXT:    [[TMP3328:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3328]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4590:%.*]]
+// SIMD-ONLY0:       if.else4589:
+// SIMD-ONLY0-NEXT:    [[TMP3329:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3329]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4590]]
+// SIMD-ONLY0:       if.end4590:
+// SIMD-ONLY0-NEXT:    [[TMP3330:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3330]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3331:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3332:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4591:%.*]] = icmp ugt i64 [[TMP3331]], [[TMP3332]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4591]], label [[IF_THEN4593:%.*]], label [[IF_END4594:%.*]]
+// SIMD-ONLY0:       if.then4593:
+// SIMD-ONLY0-NEXT:    [[TMP3333:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3333]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4594]]
+// SIMD-ONLY0:       if.end4594:
+// SIMD-ONLY0-NEXT:    [[TMP3334:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3334]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3335:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3336:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4595:%.*]] = icmp ugt i64 [[TMP3335]], [[TMP3336]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4595]], label [[IF_THEN4597:%.*]], label [[IF_END4598:%.*]]
+// SIMD-ONLY0:       if.then4597:
+// SIMD-ONLY0-NEXT:    [[TMP3337:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3337]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4598]]
+// SIMD-ONLY0:       if.end4598:
+// SIMD-ONLY0-NEXT:    [[TMP3338:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3338]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3339:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3340:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4599:%.*]] = icmp ult i64 [[TMP3339]], [[TMP3340]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4599]], label [[IF_THEN4601:%.*]], label [[IF_END4602:%.*]]
+// SIMD-ONLY0:       if.then4601:
+// SIMD-ONLY0-NEXT:    [[TMP3341:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3341]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4602]]
+// SIMD-ONLY0:       if.end4602:
+// SIMD-ONLY0-NEXT:    [[TMP3342:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3342]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3343:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3344:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4603:%.*]] = icmp ult i64 [[TMP3343]], [[TMP3344]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4603]], label [[IF_THEN4605:%.*]], label [[IF_END4606:%.*]]
+// SIMD-ONLY0:       if.then4605:
+// SIMD-ONLY0-NEXT:    [[TMP3345:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3345]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4606]]
+// SIMD-ONLY0:       if.end4606:
+// SIMD-ONLY0-NEXT:    [[TMP3346:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3346]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3347:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3348:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4607:%.*]] = icmp eq i64 [[TMP3347]], [[TMP3348]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4607]], label [[IF_THEN4609:%.*]], label [[IF_END4610:%.*]]
+// SIMD-ONLY0:       if.then4609:
+// SIMD-ONLY0-NEXT:    [[TMP3349:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3349]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4610]]
+// SIMD-ONLY0:       if.end4610:
+// SIMD-ONLY0-NEXT:    [[TMP3350:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3350]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3351:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3352:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4611:%.*]] = icmp eq i64 [[TMP3351]], [[TMP3352]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4611]], label [[IF_THEN4613:%.*]], label [[IF_END4614:%.*]]
+// SIMD-ONLY0:       if.then4613:
+// SIMD-ONLY0-NEXT:    [[TMP3353:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3353]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4614]]
+// SIMD-ONLY0:       if.end4614:
+// SIMD-ONLY0-NEXT:    [[TMP3354:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3355:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4615:%.*]] = icmp ugt i64 [[TMP3354]], [[TMP3355]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4615]], label [[IF_THEN4617:%.*]], label [[IF_END4618:%.*]]
+// SIMD-ONLY0:       if.then4617:
+// SIMD-ONLY0-NEXT:    [[TMP3356:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3356]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4618]]
+// SIMD-ONLY0:       if.end4618:
+// SIMD-ONLY0-NEXT:    [[TMP3357:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3357]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3358:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3359:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4619:%.*]] = icmp ugt i64 [[TMP3358]], [[TMP3359]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4619]], label [[IF_THEN4621:%.*]], label [[IF_END4622:%.*]]
+// SIMD-ONLY0:       if.then4621:
+// SIMD-ONLY0-NEXT:    [[TMP3360:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3360]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4622]]
+// SIMD-ONLY0:       if.end4622:
+// SIMD-ONLY0-NEXT:    [[TMP3361:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3361]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3362:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3363:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4623:%.*]] = icmp ult i64 [[TMP3362]], [[TMP3363]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4623]], label [[IF_THEN4625:%.*]], label [[IF_END4626:%.*]]
+// SIMD-ONLY0:       if.then4625:
+// SIMD-ONLY0-NEXT:    [[TMP3364:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3364]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4626]]
+// SIMD-ONLY0:       if.end4626:
+// SIMD-ONLY0-NEXT:    [[TMP3365:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3365]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3366:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3367:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4627:%.*]] = icmp ult i64 [[TMP3366]], [[TMP3367]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4627]], label [[IF_THEN4629:%.*]], label [[IF_END4630:%.*]]
+// SIMD-ONLY0:       if.then4629:
+// SIMD-ONLY0-NEXT:    [[TMP3368:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3368]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4630]]
+// SIMD-ONLY0:       if.end4630:
+// SIMD-ONLY0-NEXT:    [[TMP3369:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3369]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3370:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3371:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4631:%.*]] = icmp eq i64 [[TMP3370]], [[TMP3371]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4631]], label [[IF_THEN4633:%.*]], label [[IF_END4634:%.*]]
+// SIMD-ONLY0:       if.then4633:
+// SIMD-ONLY0-NEXT:    [[TMP3372:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3372]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4634]]
+// SIMD-ONLY0:       if.end4634:
+// SIMD-ONLY0-NEXT:    [[TMP3373:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3373]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3374:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3375:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4635:%.*]] = icmp eq i64 [[TMP3374]], [[TMP3375]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4635]], label [[IF_THEN4637:%.*]], label [[IF_END4638:%.*]]
+// SIMD-ONLY0:       if.then4637:
+// SIMD-ONLY0-NEXT:    [[TMP3376:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3376]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4638]]
+// SIMD-ONLY0:       if.end4638:
+// SIMD-ONLY0-NEXT:    [[TMP3377:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3377]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3378:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3379:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4639:%.*]] = icmp eq i64 [[TMP3378]], [[TMP3379]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4639]], label [[IF_THEN4641:%.*]], label [[IF_ELSE4642:%.*]]
+// SIMD-ONLY0:       if.then4641:
+// SIMD-ONLY0-NEXT:    [[TMP3380:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3380]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4643:%.*]]
+// SIMD-ONLY0:       if.else4642:
+// SIMD-ONLY0-NEXT:    [[TMP3381:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3381]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4643]]
+// SIMD-ONLY0:       if.end4643:
+// SIMD-ONLY0-NEXT:    [[TMP3382:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3383:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4644:%.*]] = icmp eq i64 [[TMP3382]], [[TMP3383]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4644]], label [[IF_THEN4646:%.*]], label [[IF_ELSE4647:%.*]]
+// SIMD-ONLY0:       if.then4646:
+// SIMD-ONLY0-NEXT:    [[TMP3384:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3384]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4648:%.*]]
+// SIMD-ONLY0:       if.else4647:
+// SIMD-ONLY0-NEXT:    [[TMP3385:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3385]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4648]]
+// SIMD-ONLY0:       if.end4648:
+// SIMD-ONLY0-NEXT:    [[TMP3386:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3387:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4649:%.*]] = icmp eq i64 [[TMP3386]], [[TMP3387]]
+// SIMD-ONLY0-NEXT:    [[CONV4650:%.*]] = zext i1 [[CMP4649]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4651:%.*]] = sext i32 [[CONV4650]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4651]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3388:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4652:%.*]] = icmp ne i64 [[TMP3388]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4652]], label [[IF_THEN4653:%.*]], label [[IF_END4654:%.*]]
+// SIMD-ONLY0:       if.then4653:
+// SIMD-ONLY0-NEXT:    [[TMP3389:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3389]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4654]]
+// SIMD-ONLY0:       if.end4654:
+// SIMD-ONLY0-NEXT:    [[TMP3390:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3391:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4655:%.*]] = icmp eq i64 [[TMP3390]], [[TMP3391]]
+// SIMD-ONLY0-NEXT:    [[CONV4656:%.*]] = zext i1 [[CMP4655]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4657:%.*]] = sext i32 [[CONV4656]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4657]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3392:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4658:%.*]] = icmp ne i64 [[TMP3392]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4658]], label [[IF_THEN4659:%.*]], label [[IF_END4660:%.*]]
+// SIMD-ONLY0:       if.then4659:
+// SIMD-ONLY0-NEXT:    [[TMP3393:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3393]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4660]]
+// SIMD-ONLY0:       if.end4660:
+// SIMD-ONLY0-NEXT:    [[TMP3394:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3395:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4661:%.*]] = icmp eq i64 [[TMP3394]], [[TMP3395]]
+// SIMD-ONLY0-NEXT:    [[CONV4662:%.*]] = zext i1 [[CMP4661]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4663:%.*]] = sext i32 [[CONV4662]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4663]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3396:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4664:%.*]] = icmp ne i64 [[TMP3396]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4664]], label [[IF_THEN4665:%.*]], label [[IF_ELSE4666:%.*]]
+// SIMD-ONLY0:       if.then4665:
+// SIMD-ONLY0-NEXT:    [[TMP3397:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3397]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4667:%.*]]
+// SIMD-ONLY0:       if.else4666:
+// SIMD-ONLY0-NEXT:    [[TMP3398:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3398]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4667]]
+// SIMD-ONLY0:       if.end4667:
+// SIMD-ONLY0-NEXT:    [[TMP3399:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3400:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4668:%.*]] = icmp eq i64 [[TMP3399]], [[TMP3400]]
+// SIMD-ONLY0-NEXT:    [[CONV4669:%.*]] = zext i1 [[CMP4668]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4670:%.*]] = sext i32 [[CONV4669]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4670]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3401:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4671:%.*]] = icmp ne i64 [[TMP3401]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4671]], label [[IF_THEN4672:%.*]], label [[IF_ELSE4673:%.*]]
+// SIMD-ONLY0:       if.then4672:
+// SIMD-ONLY0-NEXT:    [[TMP3402:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3402]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4674:%.*]]
+// SIMD-ONLY0:       if.else4673:
+// SIMD-ONLY0-NEXT:    [[TMP3403:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3403]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4674]]
+// SIMD-ONLY0:       if.end4674:
+// SIMD-ONLY0-NEXT:    [[TMP3404:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3404]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3405:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3406:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4675:%.*]] = icmp ugt i64 [[TMP3405]], [[TMP3406]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4675]], label [[IF_THEN4677:%.*]], label [[IF_END4678:%.*]]
+// SIMD-ONLY0:       if.then4677:
+// SIMD-ONLY0-NEXT:    [[TMP3407:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3407]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4678]]
+// SIMD-ONLY0:       if.end4678:
+// SIMD-ONLY0-NEXT:    [[TMP3408:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3408]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3409:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3410:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4679:%.*]] = icmp ugt i64 [[TMP3409]], [[TMP3410]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4679]], label [[IF_THEN4681:%.*]], label [[IF_END4682:%.*]]
+// SIMD-ONLY0:       if.then4681:
+// SIMD-ONLY0-NEXT:    [[TMP3411:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3411]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4682]]
+// SIMD-ONLY0:       if.end4682:
+// SIMD-ONLY0-NEXT:    [[TMP3412:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3412]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3413:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3414:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4683:%.*]] = icmp ult i64 [[TMP3413]], [[TMP3414]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4683]], label [[IF_THEN4685:%.*]], label [[IF_END4686:%.*]]
+// SIMD-ONLY0:       if.then4685:
+// SIMD-ONLY0-NEXT:    [[TMP3415:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3415]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4686]]
+// SIMD-ONLY0:       if.end4686:
+// SIMD-ONLY0-NEXT:    [[TMP3416:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3416]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3417:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3418:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4687:%.*]] = icmp ult i64 [[TMP3417]], [[TMP3418]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4687]], label [[IF_THEN4689:%.*]], label [[IF_END4690:%.*]]
+// SIMD-ONLY0:       if.then4689:
+// SIMD-ONLY0-NEXT:    [[TMP3419:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3419]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4690]]
+// SIMD-ONLY0:       if.end4690:
+// SIMD-ONLY0-NEXT:    [[TMP3420:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3420]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3421:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3422:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4691:%.*]] = icmp eq i64 [[TMP3421]], [[TMP3422]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4691]], label [[IF_THEN4693:%.*]], label [[IF_END4694:%.*]]
+// SIMD-ONLY0:       if.then4693:
+// SIMD-ONLY0-NEXT:    [[TMP3423:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3423]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4694]]
+// SIMD-ONLY0:       if.end4694:
+// SIMD-ONLY0-NEXT:    [[TMP3424:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3424]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3425:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3426:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4695:%.*]] = icmp eq i64 [[TMP3425]], [[TMP3426]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4695]], label [[IF_THEN4697:%.*]], label [[IF_END4698:%.*]]
+// SIMD-ONLY0:       if.then4697:
+// SIMD-ONLY0-NEXT:    [[TMP3427:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3427]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4698]]
+// SIMD-ONLY0:       if.end4698:
+// SIMD-ONLY0-NEXT:    [[TMP3428:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3429:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4699:%.*]] = icmp ugt i64 [[TMP3428]], [[TMP3429]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4699]], label [[IF_THEN4701:%.*]], label [[IF_END4702:%.*]]
+// SIMD-ONLY0:       if.then4701:
+// SIMD-ONLY0-NEXT:    [[TMP3430:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3430]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4702]]
+// SIMD-ONLY0:       if.end4702:
+// SIMD-ONLY0-NEXT:    [[TMP3431:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3431]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3432:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3433:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4703:%.*]] = icmp ugt i64 [[TMP3432]], [[TMP3433]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4703]], label [[IF_THEN4705:%.*]], label [[IF_END4706:%.*]]
+// SIMD-ONLY0:       if.then4705:
+// SIMD-ONLY0-NEXT:    [[TMP3434:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3434]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4706]]
+// SIMD-ONLY0:       if.end4706:
+// SIMD-ONLY0-NEXT:    [[TMP3435:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3435]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3436:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3437:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4707:%.*]] = icmp ult i64 [[TMP3436]], [[TMP3437]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4707]], label [[IF_THEN4709:%.*]], label [[IF_END4710:%.*]]
+// SIMD-ONLY0:       if.then4709:
+// SIMD-ONLY0-NEXT:    [[TMP3438:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3438]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4710]]
+// SIMD-ONLY0:       if.end4710:
+// SIMD-ONLY0-NEXT:    [[TMP3439:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3439]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3440:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3441:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4711:%.*]] = icmp ult i64 [[TMP3440]], [[TMP3441]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4711]], label [[IF_THEN4713:%.*]], label [[IF_END4714:%.*]]
+// SIMD-ONLY0:       if.then4713:
+// SIMD-ONLY0-NEXT:    [[TMP3442:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3442]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4714]]
+// SIMD-ONLY0:       if.end4714:
+// SIMD-ONLY0-NEXT:    [[TMP3443:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3443]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3444:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3445:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4715:%.*]] = icmp eq i64 [[TMP3444]], [[TMP3445]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4715]], label [[IF_THEN4717:%.*]], label [[IF_END4718:%.*]]
+// SIMD-ONLY0:       if.then4717:
+// SIMD-ONLY0-NEXT:    [[TMP3446:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3446]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4718]]
+// SIMD-ONLY0:       if.end4718:
+// SIMD-ONLY0-NEXT:    [[TMP3447:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3447]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3448:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3449:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4719:%.*]] = icmp eq i64 [[TMP3448]], [[TMP3449]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4719]], label [[IF_THEN4721:%.*]], label [[IF_END4722:%.*]]
+// SIMD-ONLY0:       if.then4721:
+// SIMD-ONLY0-NEXT:    [[TMP3450:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3450]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4722]]
+// SIMD-ONLY0:       if.end4722:
+// SIMD-ONLY0-NEXT:    [[TMP3451:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3451]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3452:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3453:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4723:%.*]] = icmp eq i64 [[TMP3452]], [[TMP3453]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4723]], label [[IF_THEN4725:%.*]], label [[IF_ELSE4726:%.*]]
+// SIMD-ONLY0:       if.then4725:
+// SIMD-ONLY0-NEXT:    [[TMP3454:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3454]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4727:%.*]]
+// SIMD-ONLY0:       if.else4726:
+// SIMD-ONLY0-NEXT:    [[TMP3455:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3455]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4727]]
+// SIMD-ONLY0:       if.end4727:
+// SIMD-ONLY0-NEXT:    [[TMP3456:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3457:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4728:%.*]] = icmp eq i64 [[TMP3456]], [[TMP3457]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4728]], label [[IF_THEN4730:%.*]], label [[IF_ELSE4731:%.*]]
+// SIMD-ONLY0:       if.then4730:
+// SIMD-ONLY0-NEXT:    [[TMP3458:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3458]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4732:%.*]]
+// SIMD-ONLY0:       if.else4731:
+// SIMD-ONLY0-NEXT:    [[TMP3459:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3459]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4732]]
+// SIMD-ONLY0:       if.end4732:
+// SIMD-ONLY0-NEXT:    [[TMP3460:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3461:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4733:%.*]] = icmp eq i64 [[TMP3460]], [[TMP3461]]
+// SIMD-ONLY0-NEXT:    [[CONV4734:%.*]] = zext i1 [[CMP4733]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4735:%.*]] = sext i32 [[CONV4734]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4735]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3462:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4736:%.*]] = icmp ne i64 [[TMP3462]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4736]], label [[IF_THEN4737:%.*]], label [[IF_END4738:%.*]]
+// SIMD-ONLY0:       if.then4737:
+// SIMD-ONLY0-NEXT:    [[TMP3463:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3463]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4738]]
+// SIMD-ONLY0:       if.end4738:
+// SIMD-ONLY0-NEXT:    [[TMP3464:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3465:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4739:%.*]] = icmp eq i64 [[TMP3464]], [[TMP3465]]
+// SIMD-ONLY0-NEXT:    [[CONV4740:%.*]] = zext i1 [[CMP4739]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4741:%.*]] = sext i32 [[CONV4740]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4741]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3466:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4742:%.*]] = icmp ne i64 [[TMP3466]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4742]], label [[IF_THEN4743:%.*]], label [[IF_END4744:%.*]]
+// SIMD-ONLY0:       if.then4743:
+// SIMD-ONLY0-NEXT:    [[TMP3467:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3467]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4744]]
+// SIMD-ONLY0:       if.end4744:
+// SIMD-ONLY0-NEXT:    [[TMP3468:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3469:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4745:%.*]] = icmp eq i64 [[TMP3468]], [[TMP3469]]
+// SIMD-ONLY0-NEXT:    [[CONV4746:%.*]] = zext i1 [[CMP4745]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4747:%.*]] = sext i32 [[CONV4746]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4747]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3470:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4748:%.*]] = icmp ne i64 [[TMP3470]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4748]], label [[IF_THEN4749:%.*]], label [[IF_ELSE4750:%.*]]
+// SIMD-ONLY0:       if.then4749:
+// SIMD-ONLY0-NEXT:    [[TMP3471:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3471]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4751:%.*]]
+// SIMD-ONLY0:       if.else4750:
+// SIMD-ONLY0-NEXT:    [[TMP3472:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3472]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4751]]
+// SIMD-ONLY0:       if.end4751:
+// SIMD-ONLY0-NEXT:    [[TMP3473:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3474:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4752:%.*]] = icmp eq i64 [[TMP3473]], [[TMP3474]]
+// SIMD-ONLY0-NEXT:    [[CONV4753:%.*]] = zext i1 [[CMP4752]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4754:%.*]] = sext i32 [[CONV4753]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4754]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3475:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4755:%.*]] = icmp ne i64 [[TMP3475]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4755]], label [[IF_THEN4756:%.*]], label [[IF_ELSE4757:%.*]]
+// SIMD-ONLY0:       if.then4756:
+// SIMD-ONLY0-NEXT:    [[TMP3476:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3476]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4758:%.*]]
+// SIMD-ONLY0:       if.else4757:
+// SIMD-ONLY0-NEXT:    [[TMP3477:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3477]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4758]]
+// SIMD-ONLY0:       if.end4758:
+// SIMD-ONLY0-NEXT:    [[TMP3478:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3478]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3479:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3480:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4759:%.*]] = icmp ugt i64 [[TMP3479]], [[TMP3480]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4759]], label [[IF_THEN4761:%.*]], label [[IF_END4762:%.*]]
+// SIMD-ONLY0:       if.then4761:
+// SIMD-ONLY0-NEXT:    [[TMP3481:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3481]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4762]]
+// SIMD-ONLY0:       if.end4762:
+// SIMD-ONLY0-NEXT:    [[TMP3482:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3482]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3483:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3484:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4763:%.*]] = icmp ugt i64 [[TMP3483]], [[TMP3484]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4763]], label [[IF_THEN4765:%.*]], label [[IF_END4766:%.*]]
+// SIMD-ONLY0:       if.then4765:
+// SIMD-ONLY0-NEXT:    [[TMP3485:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3485]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4766]]
+// SIMD-ONLY0:       if.end4766:
+// SIMD-ONLY0-NEXT:    [[TMP3486:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3486]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3487:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3488:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4767:%.*]] = icmp ult i64 [[TMP3487]], [[TMP3488]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4767]], label [[IF_THEN4769:%.*]], label [[IF_END4770:%.*]]
+// SIMD-ONLY0:       if.then4769:
+// SIMD-ONLY0-NEXT:    [[TMP3489:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3489]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4770]]
+// SIMD-ONLY0:       if.end4770:
+// SIMD-ONLY0-NEXT:    [[TMP3490:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3490]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3491:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3492:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4771:%.*]] = icmp ult i64 [[TMP3491]], [[TMP3492]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4771]], label [[IF_THEN4773:%.*]], label [[IF_END4774:%.*]]
+// SIMD-ONLY0:       if.then4773:
+// SIMD-ONLY0-NEXT:    [[TMP3493:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3493]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4774]]
+// SIMD-ONLY0:       if.end4774:
+// SIMD-ONLY0-NEXT:    [[TMP3494:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3494]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3495:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3496:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4775:%.*]] = icmp eq i64 [[TMP3495]], [[TMP3496]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4775]], label [[IF_THEN4777:%.*]], label [[IF_END4778:%.*]]
+// SIMD-ONLY0:       if.then4777:
+// SIMD-ONLY0-NEXT:    [[TMP3497:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3497]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4778]]
+// SIMD-ONLY0:       if.end4778:
+// SIMD-ONLY0-NEXT:    [[TMP3498:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3498]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3499:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3500:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4779:%.*]] = icmp eq i64 [[TMP3499]], [[TMP3500]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4779]], label [[IF_THEN4781:%.*]], label [[IF_END4782:%.*]]
+// SIMD-ONLY0:       if.then4781:
+// SIMD-ONLY0-NEXT:    [[TMP3501:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3501]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4782]]
+// SIMD-ONLY0:       if.end4782:
+// SIMD-ONLY0-NEXT:    [[TMP3502:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3503:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4783:%.*]] = icmp ugt i64 [[TMP3502]], [[TMP3503]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4783]], label [[IF_THEN4785:%.*]], label [[IF_END4786:%.*]]
+// SIMD-ONLY0:       if.then4785:
+// SIMD-ONLY0-NEXT:    [[TMP3504:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3504]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4786]]
+// SIMD-ONLY0:       if.end4786:
+// SIMD-ONLY0-NEXT:    [[TMP3505:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3505]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3506:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3507:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4787:%.*]] = icmp ugt i64 [[TMP3506]], [[TMP3507]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4787]], label [[IF_THEN4789:%.*]], label [[IF_END4790:%.*]]
+// SIMD-ONLY0:       if.then4789:
+// SIMD-ONLY0-NEXT:    [[TMP3508:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3508]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4790]]
+// SIMD-ONLY0:       if.end4790:
+// SIMD-ONLY0-NEXT:    [[TMP3509:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3509]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3510:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3511:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4791:%.*]] = icmp ult i64 [[TMP3510]], [[TMP3511]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4791]], label [[IF_THEN4793:%.*]], label [[IF_END4794:%.*]]
+// SIMD-ONLY0:       if.then4793:
+// SIMD-ONLY0-NEXT:    [[TMP3512:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3512]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4794]]
+// SIMD-ONLY0:       if.end4794:
+// SIMD-ONLY0-NEXT:    [[TMP3513:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3513]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3514:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3515:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4795:%.*]] = icmp ult i64 [[TMP3514]], [[TMP3515]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4795]], label [[IF_THEN4797:%.*]], label [[IF_END4798:%.*]]
+// SIMD-ONLY0:       if.then4797:
+// SIMD-ONLY0-NEXT:    [[TMP3516:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3516]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4798]]
+// SIMD-ONLY0:       if.end4798:
+// SIMD-ONLY0-NEXT:    [[TMP3517:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3517]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3518:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3519:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4799:%.*]] = icmp eq i64 [[TMP3518]], [[TMP3519]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4799]], label [[IF_THEN4801:%.*]], label [[IF_END4802:%.*]]
+// SIMD-ONLY0:       if.then4801:
+// SIMD-ONLY0-NEXT:    [[TMP3520:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3520]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4802]]
+// SIMD-ONLY0:       if.end4802:
+// SIMD-ONLY0-NEXT:    [[TMP3521:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3521]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3522:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3523:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4803:%.*]] = icmp eq i64 [[TMP3522]], [[TMP3523]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4803]], label [[IF_THEN4805:%.*]], label [[IF_END4806:%.*]]
+// SIMD-ONLY0:       if.then4805:
+// SIMD-ONLY0-NEXT:    [[TMP3524:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3524]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4806]]
+// SIMD-ONLY0:       if.end4806:
+// SIMD-ONLY0-NEXT:    [[TMP3525:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3525]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3526:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3527:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4807:%.*]] = icmp eq i64 [[TMP3526]], [[TMP3527]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4807]], label [[IF_THEN4809:%.*]], label [[IF_ELSE4810:%.*]]
+// SIMD-ONLY0:       if.then4809:
+// SIMD-ONLY0-NEXT:    [[TMP3528:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3528]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4811:%.*]]
+// SIMD-ONLY0:       if.else4810:
+// SIMD-ONLY0-NEXT:    [[TMP3529:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3529]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4811]]
+// SIMD-ONLY0:       if.end4811:
+// SIMD-ONLY0-NEXT:    [[TMP3530:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3531:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4812:%.*]] = icmp eq i64 [[TMP3530]], [[TMP3531]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4812]], label [[IF_THEN4814:%.*]], label [[IF_ELSE4815:%.*]]
+// SIMD-ONLY0:       if.then4814:
+// SIMD-ONLY0-NEXT:    [[TMP3532:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3532]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4816:%.*]]
+// SIMD-ONLY0:       if.else4815:
+// SIMD-ONLY0-NEXT:    [[TMP3533:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3533]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4816]]
+// SIMD-ONLY0:       if.end4816:
+// SIMD-ONLY0-NEXT:    [[TMP3534:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3535:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4817:%.*]] = icmp eq i64 [[TMP3534]], [[TMP3535]]
+// SIMD-ONLY0-NEXT:    [[CONV4818:%.*]] = zext i1 [[CMP4817]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4819:%.*]] = sext i32 [[CONV4818]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4819]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3536:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4820:%.*]] = icmp ne i64 [[TMP3536]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4820]], label [[IF_THEN4821:%.*]], label [[IF_END4822:%.*]]
+// SIMD-ONLY0:       if.then4821:
+// SIMD-ONLY0-NEXT:    [[TMP3537:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3537]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4822]]
+// SIMD-ONLY0:       if.end4822:
+// SIMD-ONLY0-NEXT:    [[TMP3538:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3539:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4823:%.*]] = icmp eq i64 [[TMP3538]], [[TMP3539]]
+// SIMD-ONLY0-NEXT:    [[CONV4824:%.*]] = zext i1 [[CMP4823]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4825:%.*]] = sext i32 [[CONV4824]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4825]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3540:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4826:%.*]] = icmp ne i64 [[TMP3540]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4826]], label [[IF_THEN4827:%.*]], label [[IF_END4828:%.*]]
+// SIMD-ONLY0:       if.then4827:
+// SIMD-ONLY0-NEXT:    [[TMP3541:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3541]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4828]]
+// SIMD-ONLY0:       if.end4828:
+// SIMD-ONLY0-NEXT:    [[TMP3542:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3543:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4829:%.*]] = icmp eq i64 [[TMP3542]], [[TMP3543]]
+// SIMD-ONLY0-NEXT:    [[CONV4830:%.*]] = zext i1 [[CMP4829]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4831:%.*]] = sext i32 [[CONV4830]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4831]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3544:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4832:%.*]] = icmp ne i64 [[TMP3544]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4832]], label [[IF_THEN4833:%.*]], label [[IF_ELSE4834:%.*]]
+// SIMD-ONLY0:       if.then4833:
+// SIMD-ONLY0-NEXT:    [[TMP3545:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3545]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4835:%.*]]
+// SIMD-ONLY0:       if.else4834:
+// SIMD-ONLY0-NEXT:    [[TMP3546:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3546]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4835]]
+// SIMD-ONLY0:       if.end4835:
+// SIMD-ONLY0-NEXT:    [[TMP3547:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3548:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4836:%.*]] = icmp eq i64 [[TMP3547]], [[TMP3548]]
+// SIMD-ONLY0-NEXT:    [[CONV4837:%.*]] = zext i1 [[CMP4836]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4838:%.*]] = sext i32 [[CONV4837]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4838]], ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3549:%.*]] = load i64, ptr [[ULR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4839:%.*]] = icmp ne i64 [[TMP3549]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4839]], label [[IF_THEN4840:%.*]], label [[IF_ELSE4841:%.*]]
+// SIMD-ONLY0:       if.then4840:
+// SIMD-ONLY0-NEXT:    [[TMP3550:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3550]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4842:%.*]]
+// SIMD-ONLY0:       if.else4841:
+// SIMD-ONLY0-NEXT:    [[TMP3551:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3551]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4842]]
+// SIMD-ONLY0:       if.end4842:
+// SIMD-ONLY0-NEXT:    [[TMP3552:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3552]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3553:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3554:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4843:%.*]] = icmp sgt i64 [[TMP3553]], [[TMP3554]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4843]], label [[IF_THEN4845:%.*]], label [[IF_END4846:%.*]]
+// SIMD-ONLY0:       if.then4845:
+// SIMD-ONLY0-NEXT:    [[TMP3555:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3555]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4846]]
+// SIMD-ONLY0:       if.end4846:
+// SIMD-ONLY0-NEXT:    [[TMP3556:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3556]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3557:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3558:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4847:%.*]] = icmp sgt i64 [[TMP3557]], [[TMP3558]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4847]], label [[IF_THEN4849:%.*]], label [[IF_END4850:%.*]]
+// SIMD-ONLY0:       if.then4849:
+// SIMD-ONLY0-NEXT:    [[TMP3559:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3559]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4850]]
+// SIMD-ONLY0:       if.end4850:
+// SIMD-ONLY0-NEXT:    [[TMP3560:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3560]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3561:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3562:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4851:%.*]] = icmp slt i64 [[TMP3561]], [[TMP3562]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4851]], label [[IF_THEN4853:%.*]], label [[IF_END4854:%.*]]
+// SIMD-ONLY0:       if.then4853:
+// SIMD-ONLY0-NEXT:    [[TMP3563:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3563]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4854]]
+// SIMD-ONLY0:       if.end4854:
+// SIMD-ONLY0-NEXT:    [[TMP3564:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3564]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3565:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3566:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4855:%.*]] = icmp slt i64 [[TMP3565]], [[TMP3566]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4855]], label [[IF_THEN4857:%.*]], label [[IF_END4858:%.*]]
+// SIMD-ONLY0:       if.then4857:
+// SIMD-ONLY0-NEXT:    [[TMP3567:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3567]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4858]]
+// SIMD-ONLY0:       if.end4858:
+// SIMD-ONLY0-NEXT:    [[TMP3568:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3568]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3569:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3570:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4859:%.*]] = icmp eq i64 [[TMP3569]], [[TMP3570]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4859]], label [[IF_THEN4861:%.*]], label [[IF_END4862:%.*]]
+// SIMD-ONLY0:       if.then4861:
+// SIMD-ONLY0-NEXT:    [[TMP3571:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3571]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4862]]
+// SIMD-ONLY0:       if.end4862:
+// SIMD-ONLY0-NEXT:    [[TMP3572:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3572]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3573:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3574:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4863:%.*]] = icmp eq i64 [[TMP3573]], [[TMP3574]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4863]], label [[IF_THEN4865:%.*]], label [[IF_END4866:%.*]]
+// SIMD-ONLY0:       if.then4865:
+// SIMD-ONLY0-NEXT:    [[TMP3575:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3575]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4866]]
+// SIMD-ONLY0:       if.end4866:
+// SIMD-ONLY0-NEXT:    [[TMP3576:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3577:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4867:%.*]] = icmp sgt i64 [[TMP3576]], [[TMP3577]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4867]], label [[IF_THEN4869:%.*]], label [[IF_END4870:%.*]]
+// SIMD-ONLY0:       if.then4869:
+// SIMD-ONLY0-NEXT:    [[TMP3578:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3578]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4870]]
+// SIMD-ONLY0:       if.end4870:
+// SIMD-ONLY0-NEXT:    [[TMP3579:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3579]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3580:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3581:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4871:%.*]] = icmp sgt i64 [[TMP3580]], [[TMP3581]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4871]], label [[IF_THEN4873:%.*]], label [[IF_END4874:%.*]]
+// SIMD-ONLY0:       if.then4873:
+// SIMD-ONLY0-NEXT:    [[TMP3582:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3582]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4874]]
+// SIMD-ONLY0:       if.end4874:
+// SIMD-ONLY0-NEXT:    [[TMP3583:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3583]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3584:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3585:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4875:%.*]] = icmp slt i64 [[TMP3584]], [[TMP3585]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4875]], label [[IF_THEN4877:%.*]], label [[IF_END4878:%.*]]
+// SIMD-ONLY0:       if.then4877:
+// SIMD-ONLY0-NEXT:    [[TMP3586:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3586]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4878]]
+// SIMD-ONLY0:       if.end4878:
+// SIMD-ONLY0-NEXT:    [[TMP3587:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3587]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3588:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3589:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4879:%.*]] = icmp slt i64 [[TMP3588]], [[TMP3589]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4879]], label [[IF_THEN4881:%.*]], label [[IF_END4882:%.*]]
+// SIMD-ONLY0:       if.then4881:
+// SIMD-ONLY0-NEXT:    [[TMP3590:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3590]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4882]]
+// SIMD-ONLY0:       if.end4882:
+// SIMD-ONLY0-NEXT:    [[TMP3591:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3591]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3592:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3593:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4883:%.*]] = icmp eq i64 [[TMP3592]], [[TMP3593]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4883]], label [[IF_THEN4885:%.*]], label [[IF_END4886:%.*]]
+// SIMD-ONLY0:       if.then4885:
+// SIMD-ONLY0-NEXT:    [[TMP3594:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3594]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4886]]
+// SIMD-ONLY0:       if.end4886:
+// SIMD-ONLY0-NEXT:    [[TMP3595:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3595]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3596:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3597:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4887:%.*]] = icmp eq i64 [[TMP3596]], [[TMP3597]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4887]], label [[IF_THEN4889:%.*]], label [[IF_END4890:%.*]]
+// SIMD-ONLY0:       if.then4889:
+// SIMD-ONLY0-NEXT:    [[TMP3598:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3598]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4890]]
+// SIMD-ONLY0:       if.end4890:
+// SIMD-ONLY0-NEXT:    [[TMP3599:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3599]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3600:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3601:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4891:%.*]] = icmp eq i64 [[TMP3600]], [[TMP3601]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4891]], label [[IF_THEN4893:%.*]], label [[IF_ELSE4894:%.*]]
+// SIMD-ONLY0:       if.then4893:
+// SIMD-ONLY0-NEXT:    [[TMP3602:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3602]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4895:%.*]]
+// SIMD-ONLY0:       if.else4894:
+// SIMD-ONLY0-NEXT:    [[TMP3603:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3603]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4895]]
+// SIMD-ONLY0:       if.end4895:
+// SIMD-ONLY0-NEXT:    [[TMP3604:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3605:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4896:%.*]] = icmp eq i64 [[TMP3604]], [[TMP3605]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4896]], label [[IF_THEN4898:%.*]], label [[IF_ELSE4899:%.*]]
+// SIMD-ONLY0:       if.then4898:
+// SIMD-ONLY0-NEXT:    [[TMP3606:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3606]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4900:%.*]]
+// SIMD-ONLY0:       if.else4899:
+// SIMD-ONLY0-NEXT:    [[TMP3607:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3607]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4900]]
+// SIMD-ONLY0:       if.end4900:
+// SIMD-ONLY0-NEXT:    [[TMP3608:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3609:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4901:%.*]] = icmp eq i64 [[TMP3608]], [[TMP3609]]
+// SIMD-ONLY0-NEXT:    [[CONV4902:%.*]] = zext i1 [[CMP4901]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4903:%.*]] = sext i32 [[CONV4902]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4903]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3610:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4904:%.*]] = icmp ne i64 [[TMP3610]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4904]], label [[IF_THEN4905:%.*]], label [[IF_END4906:%.*]]
+// SIMD-ONLY0:       if.then4905:
+// SIMD-ONLY0-NEXT:    [[TMP3611:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3611]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4906]]
+// SIMD-ONLY0:       if.end4906:
+// SIMD-ONLY0-NEXT:    [[TMP3612:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3613:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4907:%.*]] = icmp eq i64 [[TMP3612]], [[TMP3613]]
+// SIMD-ONLY0-NEXT:    [[CONV4908:%.*]] = zext i1 [[CMP4907]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4909:%.*]] = sext i32 [[CONV4908]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4909]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3614:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4910:%.*]] = icmp ne i64 [[TMP3614]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4910]], label [[IF_THEN4911:%.*]], label [[IF_END4912:%.*]]
+// SIMD-ONLY0:       if.then4911:
+// SIMD-ONLY0-NEXT:    [[TMP3615:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3615]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4912]]
+// SIMD-ONLY0:       if.end4912:
+// SIMD-ONLY0-NEXT:    [[TMP3616:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3617:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4913:%.*]] = icmp eq i64 [[TMP3616]], [[TMP3617]]
+// SIMD-ONLY0-NEXT:    [[CONV4914:%.*]] = zext i1 [[CMP4913]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4915:%.*]] = sext i32 [[CONV4914]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4915]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3618:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4916:%.*]] = icmp ne i64 [[TMP3618]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4916]], label [[IF_THEN4917:%.*]], label [[IF_ELSE4918:%.*]]
+// SIMD-ONLY0:       if.then4917:
+// SIMD-ONLY0-NEXT:    [[TMP3619:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3619]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4919:%.*]]
+// SIMD-ONLY0:       if.else4918:
+// SIMD-ONLY0-NEXT:    [[TMP3620:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3620]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4919]]
+// SIMD-ONLY0:       if.end4919:
+// SIMD-ONLY0-NEXT:    [[TMP3621:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3622:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4920:%.*]] = icmp eq i64 [[TMP3621]], [[TMP3622]]
+// SIMD-ONLY0-NEXT:    [[CONV4921:%.*]] = zext i1 [[CMP4920]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4922:%.*]] = sext i32 [[CONV4921]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4922]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3623:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4923:%.*]] = icmp ne i64 [[TMP3623]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4923]], label [[IF_THEN4924:%.*]], label [[IF_ELSE4925:%.*]]
+// SIMD-ONLY0:       if.then4924:
+// SIMD-ONLY0-NEXT:    [[TMP3624:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3624]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4926:%.*]]
+// SIMD-ONLY0:       if.else4925:
+// SIMD-ONLY0-NEXT:    [[TMP3625:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3625]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4926]]
+// SIMD-ONLY0:       if.end4926:
+// SIMD-ONLY0-NEXT:    [[TMP3626:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3626]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3627:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3628:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4927:%.*]] = icmp sgt i64 [[TMP3627]], [[TMP3628]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4927]], label [[IF_THEN4929:%.*]], label [[IF_END4930:%.*]]
+// SIMD-ONLY0:       if.then4929:
+// SIMD-ONLY0-NEXT:    [[TMP3629:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3629]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4930]]
+// SIMD-ONLY0:       if.end4930:
+// SIMD-ONLY0-NEXT:    [[TMP3630:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3630]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3631:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3632:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4931:%.*]] = icmp sgt i64 [[TMP3631]], [[TMP3632]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4931]], label [[IF_THEN4933:%.*]], label [[IF_END4934:%.*]]
+// SIMD-ONLY0:       if.then4933:
+// SIMD-ONLY0-NEXT:    [[TMP3633:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3633]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4934]]
+// SIMD-ONLY0:       if.end4934:
+// SIMD-ONLY0-NEXT:    [[TMP3634:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3634]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3635:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3636:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4935:%.*]] = icmp slt i64 [[TMP3635]], [[TMP3636]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4935]], label [[IF_THEN4937:%.*]], label [[IF_END4938:%.*]]
+// SIMD-ONLY0:       if.then4937:
+// SIMD-ONLY0-NEXT:    [[TMP3637:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3637]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4938]]
+// SIMD-ONLY0:       if.end4938:
+// SIMD-ONLY0-NEXT:    [[TMP3638:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3638]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3639:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3640:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4939:%.*]] = icmp slt i64 [[TMP3639]], [[TMP3640]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4939]], label [[IF_THEN4941:%.*]], label [[IF_END4942:%.*]]
+// SIMD-ONLY0:       if.then4941:
+// SIMD-ONLY0-NEXT:    [[TMP3641:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3641]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4942]]
+// SIMD-ONLY0:       if.end4942:
+// SIMD-ONLY0-NEXT:    [[TMP3642:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3642]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3643:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3644:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4943:%.*]] = icmp eq i64 [[TMP3643]], [[TMP3644]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4943]], label [[IF_THEN4945:%.*]], label [[IF_END4946:%.*]]
+// SIMD-ONLY0:       if.then4945:
+// SIMD-ONLY0-NEXT:    [[TMP3645:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3645]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4946]]
+// SIMD-ONLY0:       if.end4946:
+// SIMD-ONLY0-NEXT:    [[TMP3646:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3646]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3647:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3648:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4947:%.*]] = icmp eq i64 [[TMP3647]], [[TMP3648]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4947]], label [[IF_THEN4949:%.*]], label [[IF_END4950:%.*]]
+// SIMD-ONLY0:       if.then4949:
+// SIMD-ONLY0-NEXT:    [[TMP3649:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3649]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4950]]
+// SIMD-ONLY0:       if.end4950:
+// SIMD-ONLY0-NEXT:    [[TMP3650:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3651:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4951:%.*]] = icmp sgt i64 [[TMP3650]], [[TMP3651]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4951]], label [[IF_THEN4953:%.*]], label [[IF_END4954:%.*]]
+// SIMD-ONLY0:       if.then4953:
+// SIMD-ONLY0-NEXT:    [[TMP3652:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3652]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4954]]
+// SIMD-ONLY0:       if.end4954:
+// SIMD-ONLY0-NEXT:    [[TMP3653:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3653]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3654:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3655:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4955:%.*]] = icmp sgt i64 [[TMP3654]], [[TMP3655]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4955]], label [[IF_THEN4957:%.*]], label [[IF_END4958:%.*]]
+// SIMD-ONLY0:       if.then4957:
+// SIMD-ONLY0-NEXT:    [[TMP3656:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3656]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4958]]
+// SIMD-ONLY0:       if.end4958:
+// SIMD-ONLY0-NEXT:    [[TMP3657:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3657]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3658:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3659:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4959:%.*]] = icmp slt i64 [[TMP3658]], [[TMP3659]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4959]], label [[IF_THEN4961:%.*]], label [[IF_END4962:%.*]]
+// SIMD-ONLY0:       if.then4961:
+// SIMD-ONLY0-NEXT:    [[TMP3660:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3660]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4962]]
+// SIMD-ONLY0:       if.end4962:
+// SIMD-ONLY0-NEXT:    [[TMP3661:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3661]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3662:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3663:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4963:%.*]] = icmp slt i64 [[TMP3662]], [[TMP3663]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4963]], label [[IF_THEN4965:%.*]], label [[IF_END4966:%.*]]
+// SIMD-ONLY0:       if.then4965:
+// SIMD-ONLY0-NEXT:    [[TMP3664:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3664]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4966]]
+// SIMD-ONLY0:       if.end4966:
+// SIMD-ONLY0-NEXT:    [[TMP3665:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3665]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3666:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3667:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4967:%.*]] = icmp eq i64 [[TMP3666]], [[TMP3667]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4967]], label [[IF_THEN4969:%.*]], label [[IF_END4970:%.*]]
+// SIMD-ONLY0:       if.then4969:
+// SIMD-ONLY0-NEXT:    [[TMP3668:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3668]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4970]]
+// SIMD-ONLY0:       if.end4970:
+// SIMD-ONLY0-NEXT:    [[TMP3669:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3669]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3670:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3671:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4971:%.*]] = icmp eq i64 [[TMP3670]], [[TMP3671]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4971]], label [[IF_THEN4973:%.*]], label [[IF_END4974:%.*]]
+// SIMD-ONLY0:       if.then4973:
+// SIMD-ONLY0-NEXT:    [[TMP3672:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3672]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4974]]
+// SIMD-ONLY0:       if.end4974:
+// SIMD-ONLY0-NEXT:    [[TMP3673:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3673]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3674:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3675:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4975:%.*]] = icmp eq i64 [[TMP3674]], [[TMP3675]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4975]], label [[IF_THEN4977:%.*]], label [[IF_ELSE4978:%.*]]
+// SIMD-ONLY0:       if.then4977:
+// SIMD-ONLY0-NEXT:    [[TMP3676:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3676]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4979:%.*]]
+// SIMD-ONLY0:       if.else4978:
+// SIMD-ONLY0-NEXT:    [[TMP3677:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3677]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4979]]
+// SIMD-ONLY0:       if.end4979:
+// SIMD-ONLY0-NEXT:    [[TMP3678:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3679:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4980:%.*]] = icmp eq i64 [[TMP3678]], [[TMP3679]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP4980]], label [[IF_THEN4982:%.*]], label [[IF_ELSE4983:%.*]]
+// SIMD-ONLY0:       if.then4982:
+// SIMD-ONLY0-NEXT:    [[TMP3680:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3680]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4984:%.*]]
+// SIMD-ONLY0:       if.else4983:
+// SIMD-ONLY0-NEXT:    [[TMP3681:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3681]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4984]]
+// SIMD-ONLY0:       if.end4984:
+// SIMD-ONLY0-NEXT:    [[TMP3682:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3683:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4985:%.*]] = icmp eq i64 [[TMP3682]], [[TMP3683]]
+// SIMD-ONLY0-NEXT:    [[CONV4986:%.*]] = zext i1 [[CMP4985]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4987:%.*]] = sext i32 [[CONV4986]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4987]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3684:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4988:%.*]] = icmp ne i64 [[TMP3684]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4988]], label [[IF_THEN4989:%.*]], label [[IF_END4990:%.*]]
+// SIMD-ONLY0:       if.then4989:
+// SIMD-ONLY0-NEXT:    [[TMP3685:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3685]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4990]]
+// SIMD-ONLY0:       if.end4990:
+// SIMD-ONLY0-NEXT:    [[TMP3686:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3687:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4991:%.*]] = icmp eq i64 [[TMP3686]], [[TMP3687]]
+// SIMD-ONLY0-NEXT:    [[CONV4992:%.*]] = zext i1 [[CMP4991]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4993:%.*]] = sext i32 [[CONV4992]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4993]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3688:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL4994:%.*]] = icmp ne i64 [[TMP3688]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL4994]], label [[IF_THEN4995:%.*]], label [[IF_END4996:%.*]]
+// SIMD-ONLY0:       if.then4995:
+// SIMD-ONLY0-NEXT:    [[TMP3689:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3689]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END4996]]
+// SIMD-ONLY0:       if.end4996:
+// SIMD-ONLY0-NEXT:    [[TMP3690:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3691:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP4997:%.*]] = icmp eq i64 [[TMP3690]], [[TMP3691]]
+// SIMD-ONLY0-NEXT:    [[CONV4998:%.*]] = zext i1 [[CMP4997]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV4999:%.*]] = sext i32 [[CONV4998]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV4999]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3692:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5000:%.*]] = icmp ne i64 [[TMP3692]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5000]], label [[IF_THEN5001:%.*]], label [[IF_ELSE5002:%.*]]
+// SIMD-ONLY0:       if.then5001:
+// SIMD-ONLY0-NEXT:    [[TMP3693:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3693]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5003:%.*]]
+// SIMD-ONLY0:       if.else5002:
+// SIMD-ONLY0-NEXT:    [[TMP3694:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3694]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5003]]
+// SIMD-ONLY0:       if.end5003:
+// SIMD-ONLY0-NEXT:    [[TMP3695:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3696:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5004:%.*]] = icmp eq i64 [[TMP3695]], [[TMP3696]]
+// SIMD-ONLY0-NEXT:    [[CONV5005:%.*]] = zext i1 [[CMP5004]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5006:%.*]] = sext i32 [[CONV5005]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5006]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3697:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5007:%.*]] = icmp ne i64 [[TMP3697]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5007]], label [[IF_THEN5008:%.*]], label [[IF_ELSE5009:%.*]]
+// SIMD-ONLY0:       if.then5008:
+// SIMD-ONLY0-NEXT:    [[TMP3698:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3698]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5010:%.*]]
+// SIMD-ONLY0:       if.else5009:
+// SIMD-ONLY0-NEXT:    [[TMP3699:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3699]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5010]]
+// SIMD-ONLY0:       if.end5010:
+// SIMD-ONLY0-NEXT:    [[TMP3700:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3700]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3701:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3702:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5011:%.*]] = icmp sgt i64 [[TMP3701]], [[TMP3702]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5011]], label [[IF_THEN5013:%.*]], label [[IF_END5014:%.*]]
+// SIMD-ONLY0:       if.then5013:
+// SIMD-ONLY0-NEXT:    [[TMP3703:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3703]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5014]]
+// SIMD-ONLY0:       if.end5014:
+// SIMD-ONLY0-NEXT:    [[TMP3704:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3704]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3705:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3706:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5015:%.*]] = icmp sgt i64 [[TMP3705]], [[TMP3706]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5015]], label [[IF_THEN5017:%.*]], label [[IF_END5018:%.*]]
+// SIMD-ONLY0:       if.then5017:
+// SIMD-ONLY0-NEXT:    [[TMP3707:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3707]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5018]]
+// SIMD-ONLY0:       if.end5018:
+// SIMD-ONLY0-NEXT:    [[TMP3708:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3708]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3709:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3710:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5019:%.*]] = icmp slt i64 [[TMP3709]], [[TMP3710]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5019]], label [[IF_THEN5021:%.*]], label [[IF_END5022:%.*]]
+// SIMD-ONLY0:       if.then5021:
+// SIMD-ONLY0-NEXT:    [[TMP3711:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3711]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5022]]
+// SIMD-ONLY0:       if.end5022:
+// SIMD-ONLY0-NEXT:    [[TMP3712:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3712]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3713:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3714:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5023:%.*]] = icmp slt i64 [[TMP3713]], [[TMP3714]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5023]], label [[IF_THEN5025:%.*]], label [[IF_END5026:%.*]]
+// SIMD-ONLY0:       if.then5025:
+// SIMD-ONLY0-NEXT:    [[TMP3715:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3715]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5026]]
+// SIMD-ONLY0:       if.end5026:
+// SIMD-ONLY0-NEXT:    [[TMP3716:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3716]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3717:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3718:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5027:%.*]] = icmp eq i64 [[TMP3717]], [[TMP3718]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5027]], label [[IF_THEN5029:%.*]], label [[IF_END5030:%.*]]
+// SIMD-ONLY0:       if.then5029:
+// SIMD-ONLY0-NEXT:    [[TMP3719:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3719]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5030]]
+// SIMD-ONLY0:       if.end5030:
+// SIMD-ONLY0-NEXT:    [[TMP3720:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3720]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3721:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3722:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5031:%.*]] = icmp eq i64 [[TMP3721]], [[TMP3722]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5031]], label [[IF_THEN5033:%.*]], label [[IF_END5034:%.*]]
+// SIMD-ONLY0:       if.then5033:
+// SIMD-ONLY0-NEXT:    [[TMP3723:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3723]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5034]]
+// SIMD-ONLY0:       if.end5034:
+// SIMD-ONLY0-NEXT:    [[TMP3724:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3725:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5035:%.*]] = icmp sgt i64 [[TMP3724]], [[TMP3725]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5035]], label [[IF_THEN5037:%.*]], label [[IF_END5038:%.*]]
+// SIMD-ONLY0:       if.then5037:
+// SIMD-ONLY0-NEXT:    [[TMP3726:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3726]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5038]]
+// SIMD-ONLY0:       if.end5038:
+// SIMD-ONLY0-NEXT:    [[TMP3727:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3727]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3728:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3729:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5039:%.*]] = icmp sgt i64 [[TMP3728]], [[TMP3729]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5039]], label [[IF_THEN5041:%.*]], label [[IF_END5042:%.*]]
+// SIMD-ONLY0:       if.then5041:
+// SIMD-ONLY0-NEXT:    [[TMP3730:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3730]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5042]]
+// SIMD-ONLY0:       if.end5042:
+// SIMD-ONLY0-NEXT:    [[TMP3731:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3731]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3732:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3733:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5043:%.*]] = icmp slt i64 [[TMP3732]], [[TMP3733]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5043]], label [[IF_THEN5045:%.*]], label [[IF_END5046:%.*]]
+// SIMD-ONLY0:       if.then5045:
+// SIMD-ONLY0-NEXT:    [[TMP3734:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3734]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5046]]
+// SIMD-ONLY0:       if.end5046:
+// SIMD-ONLY0-NEXT:    [[TMP3735:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3735]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3736:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3737:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5047:%.*]] = icmp slt i64 [[TMP3736]], [[TMP3737]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5047]], label [[IF_THEN5049:%.*]], label [[IF_END5050:%.*]]
+// SIMD-ONLY0:       if.then5049:
+// SIMD-ONLY0-NEXT:    [[TMP3738:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3738]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5050]]
+// SIMD-ONLY0:       if.end5050:
+// SIMD-ONLY0-NEXT:    [[TMP3739:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3739]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3740:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3741:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5051:%.*]] = icmp eq i64 [[TMP3740]], [[TMP3741]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5051]], label [[IF_THEN5053:%.*]], label [[IF_END5054:%.*]]
+// SIMD-ONLY0:       if.then5053:
+// SIMD-ONLY0-NEXT:    [[TMP3742:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3742]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5054]]
+// SIMD-ONLY0:       if.end5054:
+// SIMD-ONLY0-NEXT:    [[TMP3743:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3743]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3744:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3745:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5055:%.*]] = icmp eq i64 [[TMP3744]], [[TMP3745]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5055]], label [[IF_THEN5057:%.*]], label [[IF_END5058:%.*]]
+// SIMD-ONLY0:       if.then5057:
+// SIMD-ONLY0-NEXT:    [[TMP3746:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3746]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5058]]
+// SIMD-ONLY0:       if.end5058:
+// SIMD-ONLY0-NEXT:    [[TMP3747:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3747]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3748:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3749:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5059:%.*]] = icmp eq i64 [[TMP3748]], [[TMP3749]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5059]], label [[IF_THEN5061:%.*]], label [[IF_ELSE5062:%.*]]
+// SIMD-ONLY0:       if.then5061:
+// SIMD-ONLY0-NEXT:    [[TMP3750:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3750]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5063:%.*]]
+// SIMD-ONLY0:       if.else5062:
+// SIMD-ONLY0-NEXT:    [[TMP3751:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3751]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5063]]
+// SIMD-ONLY0:       if.end5063:
+// SIMD-ONLY0-NEXT:    [[TMP3752:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3753:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5064:%.*]] = icmp eq i64 [[TMP3752]], [[TMP3753]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5064]], label [[IF_THEN5066:%.*]], label [[IF_ELSE5067:%.*]]
+// SIMD-ONLY0:       if.then5066:
+// SIMD-ONLY0-NEXT:    [[TMP3754:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3754]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5068:%.*]]
+// SIMD-ONLY0:       if.else5067:
+// SIMD-ONLY0-NEXT:    [[TMP3755:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3755]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5068]]
+// SIMD-ONLY0:       if.end5068:
+// SIMD-ONLY0-NEXT:    [[TMP3756:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3757:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5069:%.*]] = icmp eq i64 [[TMP3756]], [[TMP3757]]
+// SIMD-ONLY0-NEXT:    [[CONV5070:%.*]] = zext i1 [[CMP5069]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5071:%.*]] = sext i32 [[CONV5070]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5071]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3758:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5072:%.*]] = icmp ne i64 [[TMP3758]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5072]], label [[IF_THEN5073:%.*]], label [[IF_END5074:%.*]]
+// SIMD-ONLY0:       if.then5073:
+// SIMD-ONLY0-NEXT:    [[TMP3759:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3759]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5074]]
+// SIMD-ONLY0:       if.end5074:
+// SIMD-ONLY0-NEXT:    [[TMP3760:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3761:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5075:%.*]] = icmp eq i64 [[TMP3760]], [[TMP3761]]
+// SIMD-ONLY0-NEXT:    [[CONV5076:%.*]] = zext i1 [[CMP5075]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5077:%.*]] = sext i32 [[CONV5076]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5077]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3762:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5078:%.*]] = icmp ne i64 [[TMP3762]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5078]], label [[IF_THEN5079:%.*]], label [[IF_END5080:%.*]]
+// SIMD-ONLY0:       if.then5079:
+// SIMD-ONLY0-NEXT:    [[TMP3763:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3763]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5080]]
+// SIMD-ONLY0:       if.end5080:
+// SIMD-ONLY0-NEXT:    [[TMP3764:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3765:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5081:%.*]] = icmp eq i64 [[TMP3764]], [[TMP3765]]
+// SIMD-ONLY0-NEXT:    [[CONV5082:%.*]] = zext i1 [[CMP5081]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5083:%.*]] = sext i32 [[CONV5082]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5083]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3766:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5084:%.*]] = icmp ne i64 [[TMP3766]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5084]], label [[IF_THEN5085:%.*]], label [[IF_ELSE5086:%.*]]
+// SIMD-ONLY0:       if.then5085:
+// SIMD-ONLY0-NEXT:    [[TMP3767:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3767]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5087:%.*]]
+// SIMD-ONLY0:       if.else5086:
+// SIMD-ONLY0-NEXT:    [[TMP3768:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3768]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5087]]
+// SIMD-ONLY0:       if.end5087:
+// SIMD-ONLY0-NEXT:    [[TMP3769:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3770:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5088:%.*]] = icmp eq i64 [[TMP3769]], [[TMP3770]]
+// SIMD-ONLY0-NEXT:    [[CONV5089:%.*]] = zext i1 [[CMP5088]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5090:%.*]] = sext i32 [[CONV5089]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5090]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3771:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5091:%.*]] = icmp ne i64 [[TMP3771]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5091]], label [[IF_THEN5092:%.*]], label [[IF_ELSE5093:%.*]]
+// SIMD-ONLY0:       if.then5092:
+// SIMD-ONLY0-NEXT:    [[TMP3772:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3772]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5094:%.*]]
+// SIMD-ONLY0:       if.else5093:
+// SIMD-ONLY0-NEXT:    [[TMP3773:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3773]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5094]]
+// SIMD-ONLY0:       if.end5094:
+// SIMD-ONLY0-NEXT:    [[TMP3774:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3774]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3775:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3776:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5095:%.*]] = icmp sgt i64 [[TMP3775]], [[TMP3776]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5095]], label [[IF_THEN5097:%.*]], label [[IF_END5098:%.*]]
+// SIMD-ONLY0:       if.then5097:
+// SIMD-ONLY0-NEXT:    [[TMP3777:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3777]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5098]]
+// SIMD-ONLY0:       if.end5098:
+// SIMD-ONLY0-NEXT:    [[TMP3778:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3778]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3779:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3780:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5099:%.*]] = icmp sgt i64 [[TMP3779]], [[TMP3780]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5099]], label [[IF_THEN5101:%.*]], label [[IF_END5102:%.*]]
+// SIMD-ONLY0:       if.then5101:
+// SIMD-ONLY0-NEXT:    [[TMP3781:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3781]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5102]]
+// SIMD-ONLY0:       if.end5102:
+// SIMD-ONLY0-NEXT:    [[TMP3782:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3782]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3783:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3784:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5103:%.*]] = icmp slt i64 [[TMP3783]], [[TMP3784]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5103]], label [[IF_THEN5105:%.*]], label [[IF_END5106:%.*]]
+// SIMD-ONLY0:       if.then5105:
+// SIMD-ONLY0-NEXT:    [[TMP3785:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3785]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5106]]
+// SIMD-ONLY0:       if.end5106:
+// SIMD-ONLY0-NEXT:    [[TMP3786:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3786]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3787:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3788:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5107:%.*]] = icmp slt i64 [[TMP3787]], [[TMP3788]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5107]], label [[IF_THEN5109:%.*]], label [[IF_END5110:%.*]]
+// SIMD-ONLY0:       if.then5109:
+// SIMD-ONLY0-NEXT:    [[TMP3789:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3789]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5110]]
+// SIMD-ONLY0:       if.end5110:
+// SIMD-ONLY0-NEXT:    [[TMP3790:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3790]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3791:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3792:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5111:%.*]] = icmp eq i64 [[TMP3791]], [[TMP3792]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5111]], label [[IF_THEN5113:%.*]], label [[IF_END5114:%.*]]
+// SIMD-ONLY0:       if.then5113:
+// SIMD-ONLY0-NEXT:    [[TMP3793:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3793]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5114]]
+// SIMD-ONLY0:       if.end5114:
+// SIMD-ONLY0-NEXT:    [[TMP3794:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3794]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3795:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3796:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5115:%.*]] = icmp eq i64 [[TMP3795]], [[TMP3796]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5115]], label [[IF_THEN5117:%.*]], label [[IF_END5118:%.*]]
+// SIMD-ONLY0:       if.then5117:
+// SIMD-ONLY0-NEXT:    [[TMP3797:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3797]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5118]]
+// SIMD-ONLY0:       if.end5118:
+// SIMD-ONLY0-NEXT:    [[TMP3798:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3799:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5119:%.*]] = icmp sgt i64 [[TMP3798]], [[TMP3799]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5119]], label [[IF_THEN5121:%.*]], label [[IF_END5122:%.*]]
+// SIMD-ONLY0:       if.then5121:
+// SIMD-ONLY0-NEXT:    [[TMP3800:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3800]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5122]]
+// SIMD-ONLY0:       if.end5122:
+// SIMD-ONLY0-NEXT:    [[TMP3801:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3801]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3802:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3803:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5123:%.*]] = icmp sgt i64 [[TMP3802]], [[TMP3803]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5123]], label [[IF_THEN5125:%.*]], label [[IF_END5126:%.*]]
+// SIMD-ONLY0:       if.then5125:
+// SIMD-ONLY0-NEXT:    [[TMP3804:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3804]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5126]]
+// SIMD-ONLY0:       if.end5126:
+// SIMD-ONLY0-NEXT:    [[TMP3805:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3805]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3806:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3807:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5127:%.*]] = icmp slt i64 [[TMP3806]], [[TMP3807]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5127]], label [[IF_THEN5129:%.*]], label [[IF_END5130:%.*]]
+// SIMD-ONLY0:       if.then5129:
+// SIMD-ONLY0-NEXT:    [[TMP3808:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3808]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5130]]
+// SIMD-ONLY0:       if.end5130:
+// SIMD-ONLY0-NEXT:    [[TMP3809:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3809]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3810:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3811:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5131:%.*]] = icmp slt i64 [[TMP3810]], [[TMP3811]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5131]], label [[IF_THEN5133:%.*]], label [[IF_END5134:%.*]]
+// SIMD-ONLY0:       if.then5133:
+// SIMD-ONLY0-NEXT:    [[TMP3812:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3812]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5134]]
+// SIMD-ONLY0:       if.end5134:
+// SIMD-ONLY0-NEXT:    [[TMP3813:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3813]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3814:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3815:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5135:%.*]] = icmp eq i64 [[TMP3814]], [[TMP3815]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5135]], label [[IF_THEN5137:%.*]], label [[IF_END5138:%.*]]
+// SIMD-ONLY0:       if.then5137:
+// SIMD-ONLY0-NEXT:    [[TMP3816:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3816]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5138]]
+// SIMD-ONLY0:       if.end5138:
+// SIMD-ONLY0-NEXT:    [[TMP3817:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3817]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3818:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3819:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5139:%.*]] = icmp eq i64 [[TMP3818]], [[TMP3819]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5139]], label [[IF_THEN5141:%.*]], label [[IF_END5142:%.*]]
+// SIMD-ONLY0:       if.then5141:
+// SIMD-ONLY0-NEXT:    [[TMP3820:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3820]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5142]]
+// SIMD-ONLY0:       if.end5142:
+// SIMD-ONLY0-NEXT:    [[TMP3821:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3821]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3822:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3823:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5143:%.*]] = icmp eq i64 [[TMP3822]], [[TMP3823]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5143]], label [[IF_THEN5145:%.*]], label [[IF_ELSE5146:%.*]]
+// SIMD-ONLY0:       if.then5145:
+// SIMD-ONLY0-NEXT:    [[TMP3824:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3824]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5147:%.*]]
+// SIMD-ONLY0:       if.else5146:
+// SIMD-ONLY0-NEXT:    [[TMP3825:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3825]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5147]]
+// SIMD-ONLY0:       if.end5147:
+// SIMD-ONLY0-NEXT:    [[TMP3826:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3827:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5148:%.*]] = icmp eq i64 [[TMP3826]], [[TMP3827]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5148]], label [[IF_THEN5150:%.*]], label [[IF_ELSE5151:%.*]]
+// SIMD-ONLY0:       if.then5150:
+// SIMD-ONLY0-NEXT:    [[TMP3828:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3828]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5152:%.*]]
+// SIMD-ONLY0:       if.else5151:
+// SIMD-ONLY0-NEXT:    [[TMP3829:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3829]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5152]]
+// SIMD-ONLY0:       if.end5152:
+// SIMD-ONLY0-NEXT:    [[TMP3830:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3831:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5153:%.*]] = icmp eq i64 [[TMP3830]], [[TMP3831]]
+// SIMD-ONLY0-NEXT:    [[CONV5154:%.*]] = zext i1 [[CMP5153]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5155:%.*]] = sext i32 [[CONV5154]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5155]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3832:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5156:%.*]] = icmp ne i64 [[TMP3832]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5156]], label [[IF_THEN5157:%.*]], label [[IF_END5158:%.*]]
+// SIMD-ONLY0:       if.then5157:
+// SIMD-ONLY0-NEXT:    [[TMP3833:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3833]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5158]]
+// SIMD-ONLY0:       if.end5158:
+// SIMD-ONLY0-NEXT:    [[TMP3834:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3835:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5159:%.*]] = icmp eq i64 [[TMP3834]], [[TMP3835]]
+// SIMD-ONLY0-NEXT:    [[CONV5160:%.*]] = zext i1 [[CMP5159]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5161:%.*]] = sext i32 [[CONV5160]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5161]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3836:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5162:%.*]] = icmp ne i64 [[TMP3836]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5162]], label [[IF_THEN5163:%.*]], label [[IF_END5164:%.*]]
+// SIMD-ONLY0:       if.then5163:
+// SIMD-ONLY0-NEXT:    [[TMP3837:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3837]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5164]]
+// SIMD-ONLY0:       if.end5164:
+// SIMD-ONLY0-NEXT:    [[TMP3838:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3839:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5165:%.*]] = icmp eq i64 [[TMP3838]], [[TMP3839]]
+// SIMD-ONLY0-NEXT:    [[CONV5166:%.*]] = zext i1 [[CMP5165]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5167:%.*]] = sext i32 [[CONV5166]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5167]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3840:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5168:%.*]] = icmp ne i64 [[TMP3840]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5168]], label [[IF_THEN5169:%.*]], label [[IF_ELSE5170:%.*]]
+// SIMD-ONLY0:       if.then5169:
+// SIMD-ONLY0-NEXT:    [[TMP3841:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3841]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5171:%.*]]
+// SIMD-ONLY0:       if.else5170:
+// SIMD-ONLY0-NEXT:    [[TMP3842:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3842]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5171]]
+// SIMD-ONLY0:       if.end5171:
+// SIMD-ONLY0-NEXT:    [[TMP3843:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3844:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5172:%.*]] = icmp eq i64 [[TMP3843]], [[TMP3844]]
+// SIMD-ONLY0-NEXT:    [[CONV5173:%.*]] = zext i1 [[CMP5172]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5174:%.*]] = sext i32 [[CONV5173]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5174]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3845:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5175:%.*]] = icmp ne i64 [[TMP3845]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5175]], label [[IF_THEN5176:%.*]], label [[IF_ELSE5177:%.*]]
+// SIMD-ONLY0:       if.then5176:
+// SIMD-ONLY0-NEXT:    [[TMP3846:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3846]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5178:%.*]]
+// SIMD-ONLY0:       if.else5177:
+// SIMD-ONLY0-NEXT:    [[TMP3847:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3847]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5178]]
+// SIMD-ONLY0:       if.end5178:
+// SIMD-ONLY0-NEXT:    [[TMP3848:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3848]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3849:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3850:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5179:%.*]] = icmp sgt i64 [[TMP3849]], [[TMP3850]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5179]], label [[IF_THEN5181:%.*]], label [[IF_END5182:%.*]]
+// SIMD-ONLY0:       if.then5181:
+// SIMD-ONLY0-NEXT:    [[TMP3851:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3851]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5182]]
+// SIMD-ONLY0:       if.end5182:
+// SIMD-ONLY0-NEXT:    [[TMP3852:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3852]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3853:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3854:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5183:%.*]] = icmp sgt i64 [[TMP3853]], [[TMP3854]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5183]], label [[IF_THEN5185:%.*]], label [[IF_END5186:%.*]]
+// SIMD-ONLY0:       if.then5185:
+// SIMD-ONLY0-NEXT:    [[TMP3855:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3855]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5186]]
+// SIMD-ONLY0:       if.end5186:
+// SIMD-ONLY0-NEXT:    [[TMP3856:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3856]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3857:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3858:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5187:%.*]] = icmp slt i64 [[TMP3857]], [[TMP3858]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5187]], label [[IF_THEN5189:%.*]], label [[IF_END5190:%.*]]
+// SIMD-ONLY0:       if.then5189:
+// SIMD-ONLY0-NEXT:    [[TMP3859:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3859]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5190]]
+// SIMD-ONLY0:       if.end5190:
+// SIMD-ONLY0-NEXT:    [[TMP3860:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3860]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3861:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3862:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5191:%.*]] = icmp slt i64 [[TMP3861]], [[TMP3862]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5191]], label [[IF_THEN5193:%.*]], label [[IF_END5194:%.*]]
+// SIMD-ONLY0:       if.then5193:
+// SIMD-ONLY0-NEXT:    [[TMP3863:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3863]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5194]]
+// SIMD-ONLY0:       if.end5194:
+// SIMD-ONLY0-NEXT:    [[TMP3864:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3864]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3865:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3866:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5195:%.*]] = icmp eq i64 [[TMP3865]], [[TMP3866]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5195]], label [[IF_THEN5197:%.*]], label [[IF_END5198:%.*]]
+// SIMD-ONLY0:       if.then5197:
+// SIMD-ONLY0-NEXT:    [[TMP3867:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3867]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5198]]
+// SIMD-ONLY0:       if.end5198:
+// SIMD-ONLY0-NEXT:    [[TMP3868:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3868]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3869:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3870:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5199:%.*]] = icmp eq i64 [[TMP3869]], [[TMP3870]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5199]], label [[IF_THEN5201:%.*]], label [[IF_END5202:%.*]]
+// SIMD-ONLY0:       if.then5201:
+// SIMD-ONLY0-NEXT:    [[TMP3871:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3871]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5202]]
+// SIMD-ONLY0:       if.end5202:
+// SIMD-ONLY0-NEXT:    [[TMP3872:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3873:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5203:%.*]] = icmp sgt i64 [[TMP3872]], [[TMP3873]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5203]], label [[IF_THEN5205:%.*]], label [[IF_END5206:%.*]]
+// SIMD-ONLY0:       if.then5205:
+// SIMD-ONLY0-NEXT:    [[TMP3874:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3874]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5206]]
+// SIMD-ONLY0:       if.end5206:
+// SIMD-ONLY0-NEXT:    [[TMP3875:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3875]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3876:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3877:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5207:%.*]] = icmp sgt i64 [[TMP3876]], [[TMP3877]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5207]], label [[IF_THEN5209:%.*]], label [[IF_END5210:%.*]]
+// SIMD-ONLY0:       if.then5209:
+// SIMD-ONLY0-NEXT:    [[TMP3878:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3878]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5210]]
+// SIMD-ONLY0:       if.end5210:
+// SIMD-ONLY0-NEXT:    [[TMP3879:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3879]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3880:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3881:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5211:%.*]] = icmp slt i64 [[TMP3880]], [[TMP3881]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5211]], label [[IF_THEN5213:%.*]], label [[IF_END5214:%.*]]
+// SIMD-ONLY0:       if.then5213:
+// SIMD-ONLY0-NEXT:    [[TMP3882:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3882]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5214]]
+// SIMD-ONLY0:       if.end5214:
+// SIMD-ONLY0-NEXT:    [[TMP3883:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3883]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3884:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3885:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5215:%.*]] = icmp slt i64 [[TMP3884]], [[TMP3885]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5215]], label [[IF_THEN5217:%.*]], label [[IF_END5218:%.*]]
+// SIMD-ONLY0:       if.then5217:
+// SIMD-ONLY0-NEXT:    [[TMP3886:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3886]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5218]]
+// SIMD-ONLY0:       if.end5218:
+// SIMD-ONLY0-NEXT:    [[TMP3887:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3887]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3888:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3889:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5219:%.*]] = icmp eq i64 [[TMP3888]], [[TMP3889]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5219]], label [[IF_THEN5221:%.*]], label [[IF_END5222:%.*]]
+// SIMD-ONLY0:       if.then5221:
+// SIMD-ONLY0-NEXT:    [[TMP3890:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3890]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5222]]
+// SIMD-ONLY0:       if.end5222:
+// SIMD-ONLY0-NEXT:    [[TMP3891:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3891]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3892:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3893:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5223:%.*]] = icmp eq i64 [[TMP3892]], [[TMP3893]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5223]], label [[IF_THEN5225:%.*]], label [[IF_END5226:%.*]]
+// SIMD-ONLY0:       if.then5225:
+// SIMD-ONLY0-NEXT:    [[TMP3894:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3894]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5226]]
+// SIMD-ONLY0:       if.end5226:
+// SIMD-ONLY0-NEXT:    [[TMP3895:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3895]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3896:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3897:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5227:%.*]] = icmp eq i64 [[TMP3896]], [[TMP3897]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5227]], label [[IF_THEN5229:%.*]], label [[IF_ELSE5230:%.*]]
+// SIMD-ONLY0:       if.then5229:
+// SIMD-ONLY0-NEXT:    [[TMP3898:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3898]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5231:%.*]]
+// SIMD-ONLY0:       if.else5230:
+// SIMD-ONLY0-NEXT:    [[TMP3899:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3899]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5231]]
+// SIMD-ONLY0:       if.end5231:
+// SIMD-ONLY0-NEXT:    [[TMP3900:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3901:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5232:%.*]] = icmp eq i64 [[TMP3900]], [[TMP3901]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5232]], label [[IF_THEN5234:%.*]], label [[IF_ELSE5235:%.*]]
+// SIMD-ONLY0:       if.then5234:
+// SIMD-ONLY0-NEXT:    [[TMP3902:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3902]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5236:%.*]]
+// SIMD-ONLY0:       if.else5235:
+// SIMD-ONLY0-NEXT:    [[TMP3903:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3903]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5236]]
+// SIMD-ONLY0:       if.end5236:
+// SIMD-ONLY0-NEXT:    [[TMP3904:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3905:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5237:%.*]] = icmp eq i64 [[TMP3904]], [[TMP3905]]
+// SIMD-ONLY0-NEXT:    [[CONV5238:%.*]] = zext i1 [[CMP5237]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5239:%.*]] = sext i32 [[CONV5238]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5239]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3906:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5240:%.*]] = icmp ne i64 [[TMP3906]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5240]], label [[IF_THEN5241:%.*]], label [[IF_END5242:%.*]]
+// SIMD-ONLY0:       if.then5241:
+// SIMD-ONLY0-NEXT:    [[TMP3907:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3907]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5242]]
+// SIMD-ONLY0:       if.end5242:
+// SIMD-ONLY0-NEXT:    [[TMP3908:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3909:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5243:%.*]] = icmp eq i64 [[TMP3908]], [[TMP3909]]
+// SIMD-ONLY0-NEXT:    [[CONV5244:%.*]] = zext i1 [[CMP5243]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5245:%.*]] = sext i32 [[CONV5244]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5245]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3910:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5246:%.*]] = icmp ne i64 [[TMP3910]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5246]], label [[IF_THEN5247:%.*]], label [[IF_END5248:%.*]]
+// SIMD-ONLY0:       if.then5247:
+// SIMD-ONLY0-NEXT:    [[TMP3911:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3911]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5248]]
+// SIMD-ONLY0:       if.end5248:
+// SIMD-ONLY0-NEXT:    [[TMP3912:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3913:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5249:%.*]] = icmp eq i64 [[TMP3912]], [[TMP3913]]
+// SIMD-ONLY0-NEXT:    [[CONV5250:%.*]] = zext i1 [[CMP5249]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5251:%.*]] = sext i32 [[CONV5250]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5251]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3914:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5252:%.*]] = icmp ne i64 [[TMP3914]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5252]], label [[IF_THEN5253:%.*]], label [[IF_ELSE5254:%.*]]
+// SIMD-ONLY0:       if.then5253:
+// SIMD-ONLY0-NEXT:    [[TMP3915:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3915]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5255:%.*]]
+// SIMD-ONLY0:       if.else5254:
+// SIMD-ONLY0-NEXT:    [[TMP3916:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3916]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5255]]
+// SIMD-ONLY0:       if.end5255:
+// SIMD-ONLY0-NEXT:    [[TMP3917:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3918:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5256:%.*]] = icmp eq i64 [[TMP3917]], [[TMP3918]]
+// SIMD-ONLY0-NEXT:    [[CONV5257:%.*]] = zext i1 [[CMP5256]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5258:%.*]] = sext i32 [[CONV5257]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5258]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3919:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5259:%.*]] = icmp ne i64 [[TMP3919]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5259]], label [[IF_THEN5260:%.*]], label [[IF_ELSE5261:%.*]]
+// SIMD-ONLY0:       if.then5260:
+// SIMD-ONLY0-NEXT:    [[TMP3920:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3920]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5262:%.*]]
+// SIMD-ONLY0:       if.else5261:
+// SIMD-ONLY0-NEXT:    [[TMP3921:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3921]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5262]]
+// SIMD-ONLY0:       if.end5262:
+// SIMD-ONLY0-NEXT:    [[TMP3922:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3922]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3923:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3924:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5263:%.*]] = icmp sgt i64 [[TMP3923]], [[TMP3924]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5263]], label [[IF_THEN5265:%.*]], label [[IF_END5266:%.*]]
+// SIMD-ONLY0:       if.then5265:
+// SIMD-ONLY0-NEXT:    [[TMP3925:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3925]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5266]]
+// SIMD-ONLY0:       if.end5266:
+// SIMD-ONLY0-NEXT:    [[TMP3926:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3926]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3927:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3928:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5267:%.*]] = icmp sgt i64 [[TMP3927]], [[TMP3928]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5267]], label [[IF_THEN5269:%.*]], label [[IF_END5270:%.*]]
+// SIMD-ONLY0:       if.then5269:
+// SIMD-ONLY0-NEXT:    [[TMP3929:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3929]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5270]]
+// SIMD-ONLY0:       if.end5270:
+// SIMD-ONLY0-NEXT:    [[TMP3930:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3930]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3931:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3932:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5271:%.*]] = icmp slt i64 [[TMP3931]], [[TMP3932]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5271]], label [[IF_THEN5273:%.*]], label [[IF_END5274:%.*]]
+// SIMD-ONLY0:       if.then5273:
+// SIMD-ONLY0-NEXT:    [[TMP3933:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3933]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5274]]
+// SIMD-ONLY0:       if.end5274:
+// SIMD-ONLY0-NEXT:    [[TMP3934:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3934]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3935:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3936:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5275:%.*]] = icmp slt i64 [[TMP3935]], [[TMP3936]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5275]], label [[IF_THEN5277:%.*]], label [[IF_END5278:%.*]]
+// SIMD-ONLY0:       if.then5277:
+// SIMD-ONLY0-NEXT:    [[TMP3937:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3937]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5278]]
+// SIMD-ONLY0:       if.end5278:
+// SIMD-ONLY0-NEXT:    [[TMP3938:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3938]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3939:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3940:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5279:%.*]] = icmp eq i64 [[TMP3939]], [[TMP3940]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5279]], label [[IF_THEN5281:%.*]], label [[IF_END5282:%.*]]
+// SIMD-ONLY0:       if.then5281:
+// SIMD-ONLY0-NEXT:    [[TMP3941:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3941]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5282]]
+// SIMD-ONLY0:       if.end5282:
+// SIMD-ONLY0-NEXT:    [[TMP3942:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3942]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3943:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3944:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5283:%.*]] = icmp eq i64 [[TMP3943]], [[TMP3944]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5283]], label [[IF_THEN5285:%.*]], label [[IF_END5286:%.*]]
+// SIMD-ONLY0:       if.then5285:
+// SIMD-ONLY0-NEXT:    [[TMP3945:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3945]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5286]]
+// SIMD-ONLY0:       if.end5286:
+// SIMD-ONLY0-NEXT:    [[TMP3946:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3947:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5287:%.*]] = icmp sgt i64 [[TMP3946]], [[TMP3947]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5287]], label [[IF_THEN5289:%.*]], label [[IF_END5290:%.*]]
+// SIMD-ONLY0:       if.then5289:
+// SIMD-ONLY0-NEXT:    [[TMP3948:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3948]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5290]]
+// SIMD-ONLY0:       if.end5290:
+// SIMD-ONLY0-NEXT:    [[TMP3949:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3949]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3950:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3951:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5291:%.*]] = icmp sgt i64 [[TMP3950]], [[TMP3951]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5291]], label [[IF_THEN5293:%.*]], label [[IF_END5294:%.*]]
+// SIMD-ONLY0:       if.then5293:
+// SIMD-ONLY0-NEXT:    [[TMP3952:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3952]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5294]]
+// SIMD-ONLY0:       if.end5294:
+// SIMD-ONLY0-NEXT:    [[TMP3953:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3953]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3954:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3955:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5295:%.*]] = icmp slt i64 [[TMP3954]], [[TMP3955]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5295]], label [[IF_THEN5297:%.*]], label [[IF_END5298:%.*]]
+// SIMD-ONLY0:       if.then5297:
+// SIMD-ONLY0-NEXT:    [[TMP3956:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3956]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5298]]
+// SIMD-ONLY0:       if.end5298:
+// SIMD-ONLY0-NEXT:    [[TMP3957:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3957]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3958:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3959:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5299:%.*]] = icmp slt i64 [[TMP3958]], [[TMP3959]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5299]], label [[IF_THEN5301:%.*]], label [[IF_END5302:%.*]]
+// SIMD-ONLY0:       if.then5301:
+// SIMD-ONLY0-NEXT:    [[TMP3960:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3960]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5302]]
+// SIMD-ONLY0:       if.end5302:
+// SIMD-ONLY0-NEXT:    [[TMP3961:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3961]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3962:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3963:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5303:%.*]] = icmp eq i64 [[TMP3962]], [[TMP3963]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5303]], label [[IF_THEN5305:%.*]], label [[IF_END5306:%.*]]
+// SIMD-ONLY0:       if.then5305:
+// SIMD-ONLY0-NEXT:    [[TMP3964:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3964]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5306]]
+// SIMD-ONLY0:       if.end5306:
+// SIMD-ONLY0-NEXT:    [[TMP3965:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3965]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3966:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3967:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5307:%.*]] = icmp eq i64 [[TMP3966]], [[TMP3967]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5307]], label [[IF_THEN5309:%.*]], label [[IF_END5310:%.*]]
+// SIMD-ONLY0:       if.then5309:
+// SIMD-ONLY0-NEXT:    [[TMP3968:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3968]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5310]]
+// SIMD-ONLY0:       if.end5310:
+// SIMD-ONLY0-NEXT:    [[TMP3969:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3969]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3970:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3971:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5311:%.*]] = icmp eq i64 [[TMP3970]], [[TMP3971]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5311]], label [[IF_THEN5313:%.*]], label [[IF_ELSE5314:%.*]]
+// SIMD-ONLY0:       if.then5313:
+// SIMD-ONLY0-NEXT:    [[TMP3972:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3972]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5315:%.*]]
+// SIMD-ONLY0:       if.else5314:
+// SIMD-ONLY0-NEXT:    [[TMP3973:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3973]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5315]]
+// SIMD-ONLY0:       if.end5315:
+// SIMD-ONLY0-NEXT:    [[TMP3974:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3975:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5316:%.*]] = icmp eq i64 [[TMP3974]], [[TMP3975]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5316]], label [[IF_THEN5318:%.*]], label [[IF_ELSE5319:%.*]]
+// SIMD-ONLY0:       if.then5318:
+// SIMD-ONLY0-NEXT:    [[TMP3976:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3976]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5320:%.*]]
+// SIMD-ONLY0:       if.else5319:
+// SIMD-ONLY0-NEXT:    [[TMP3977:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3977]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5320]]
+// SIMD-ONLY0:       if.end5320:
+// SIMD-ONLY0-NEXT:    [[TMP3978:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3979:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5321:%.*]] = icmp eq i64 [[TMP3978]], [[TMP3979]]
+// SIMD-ONLY0-NEXT:    [[CONV5322:%.*]] = zext i1 [[CMP5321]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5323:%.*]] = sext i32 [[CONV5322]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5323]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3980:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5324:%.*]] = icmp ne i64 [[TMP3980]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5324]], label [[IF_THEN5325:%.*]], label [[IF_END5326:%.*]]
+// SIMD-ONLY0:       if.then5325:
+// SIMD-ONLY0-NEXT:    [[TMP3981:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3981]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5326]]
+// SIMD-ONLY0:       if.end5326:
+// SIMD-ONLY0-NEXT:    [[TMP3982:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3983:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5327:%.*]] = icmp eq i64 [[TMP3982]], [[TMP3983]]
+// SIMD-ONLY0-NEXT:    [[CONV5328:%.*]] = zext i1 [[CMP5327]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5329:%.*]] = sext i32 [[CONV5328]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5329]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3984:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5330:%.*]] = icmp ne i64 [[TMP3984]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5330]], label [[IF_THEN5331:%.*]], label [[IF_END5332:%.*]]
+// SIMD-ONLY0:       if.then5331:
+// SIMD-ONLY0-NEXT:    [[TMP3985:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3985]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5332]]
+// SIMD-ONLY0:       if.end5332:
+// SIMD-ONLY0-NEXT:    [[TMP3986:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3987:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5333:%.*]] = icmp eq i64 [[TMP3986]], [[TMP3987]]
+// SIMD-ONLY0-NEXT:    [[CONV5334:%.*]] = zext i1 [[CMP5333]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5335:%.*]] = sext i32 [[CONV5334]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5335]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3988:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5336:%.*]] = icmp ne i64 [[TMP3988]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5336]], label [[IF_THEN5337:%.*]], label [[IF_ELSE5338:%.*]]
+// SIMD-ONLY0:       if.then5337:
+// SIMD-ONLY0-NEXT:    [[TMP3989:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3989]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5339:%.*]]
+// SIMD-ONLY0:       if.else5338:
+// SIMD-ONLY0-NEXT:    [[TMP3990:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3990]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5339]]
+// SIMD-ONLY0:       if.end5339:
+// SIMD-ONLY0-NEXT:    [[TMP3991:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3992:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5340:%.*]] = icmp eq i64 [[TMP3991]], [[TMP3992]]
+// SIMD-ONLY0-NEXT:    [[CONV5341:%.*]] = zext i1 [[CMP5340]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5342:%.*]] = sext i32 [[CONV5341]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5342]], ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3993:%.*]] = load i64, ptr [[LLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5343:%.*]] = icmp ne i64 [[TMP3993]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5343]], label [[IF_THEN5344:%.*]], label [[IF_ELSE5345:%.*]]
+// SIMD-ONLY0:       if.then5344:
+// SIMD-ONLY0-NEXT:    [[TMP3994:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3994]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5346:%.*]]
+// SIMD-ONLY0:       if.else5345:
+// SIMD-ONLY0-NEXT:    [[TMP3995:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3995]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5346]]
+// SIMD-ONLY0:       if.end5346:
+// SIMD-ONLY0-NEXT:    [[TMP3996:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3996]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3997:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP3998:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5347:%.*]] = icmp ugt i64 [[TMP3997]], [[TMP3998]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5347]], label [[IF_THEN5349:%.*]], label [[IF_END5350:%.*]]
+// SIMD-ONLY0:       if.then5349:
+// SIMD-ONLY0-NEXT:    [[TMP3999:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP3999]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5350]]
+// SIMD-ONLY0:       if.end5350:
+// SIMD-ONLY0-NEXT:    [[TMP4000:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4000]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4001:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4002:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5351:%.*]] = icmp ugt i64 [[TMP4001]], [[TMP4002]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5351]], label [[IF_THEN5353:%.*]], label [[IF_END5354:%.*]]
+// SIMD-ONLY0:       if.then5353:
+// SIMD-ONLY0-NEXT:    [[TMP4003:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4003]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5354]]
+// SIMD-ONLY0:       if.end5354:
+// SIMD-ONLY0-NEXT:    [[TMP4004:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4004]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4005:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4006:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5355:%.*]] = icmp ult i64 [[TMP4005]], [[TMP4006]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5355]], label [[IF_THEN5357:%.*]], label [[IF_END5358:%.*]]
+// SIMD-ONLY0:       if.then5357:
+// SIMD-ONLY0-NEXT:    [[TMP4007:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4007]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5358]]
+// SIMD-ONLY0:       if.end5358:
+// SIMD-ONLY0-NEXT:    [[TMP4008:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4008]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4009:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4010:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5359:%.*]] = icmp ult i64 [[TMP4009]], [[TMP4010]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5359]], label [[IF_THEN5361:%.*]], label [[IF_END5362:%.*]]
+// SIMD-ONLY0:       if.then5361:
+// SIMD-ONLY0-NEXT:    [[TMP4011:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4011]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5362]]
+// SIMD-ONLY0:       if.end5362:
+// SIMD-ONLY0-NEXT:    [[TMP4012:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4012]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4013:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4014:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5363:%.*]] = icmp eq i64 [[TMP4013]], [[TMP4014]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5363]], label [[IF_THEN5365:%.*]], label [[IF_END5366:%.*]]
+// SIMD-ONLY0:       if.then5365:
+// SIMD-ONLY0-NEXT:    [[TMP4015:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4015]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5366]]
+// SIMD-ONLY0:       if.end5366:
+// SIMD-ONLY0-NEXT:    [[TMP4016:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4016]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4017:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4018:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5367:%.*]] = icmp eq i64 [[TMP4017]], [[TMP4018]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5367]], label [[IF_THEN5369:%.*]], label [[IF_END5370:%.*]]
+// SIMD-ONLY0:       if.then5369:
+// SIMD-ONLY0-NEXT:    [[TMP4019:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4019]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5370]]
+// SIMD-ONLY0:       if.end5370:
+// SIMD-ONLY0-NEXT:    [[TMP4020:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4021:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5371:%.*]] = icmp ugt i64 [[TMP4020]], [[TMP4021]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5371]], label [[IF_THEN5373:%.*]], label [[IF_END5374:%.*]]
+// SIMD-ONLY0:       if.then5373:
+// SIMD-ONLY0-NEXT:    [[TMP4022:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4022]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5374]]
+// SIMD-ONLY0:       if.end5374:
+// SIMD-ONLY0-NEXT:    [[TMP4023:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4023]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4024:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4025:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5375:%.*]] = icmp ugt i64 [[TMP4024]], [[TMP4025]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5375]], label [[IF_THEN5377:%.*]], label [[IF_END5378:%.*]]
+// SIMD-ONLY0:       if.then5377:
+// SIMD-ONLY0-NEXT:    [[TMP4026:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4026]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5378]]
+// SIMD-ONLY0:       if.end5378:
+// SIMD-ONLY0-NEXT:    [[TMP4027:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4027]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4028:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4029:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5379:%.*]] = icmp ult i64 [[TMP4028]], [[TMP4029]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5379]], label [[IF_THEN5381:%.*]], label [[IF_END5382:%.*]]
+// SIMD-ONLY0:       if.then5381:
+// SIMD-ONLY0-NEXT:    [[TMP4030:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4030]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5382]]
+// SIMD-ONLY0:       if.end5382:
+// SIMD-ONLY0-NEXT:    [[TMP4031:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4031]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4032:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4033:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5383:%.*]] = icmp ult i64 [[TMP4032]], [[TMP4033]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5383]], label [[IF_THEN5385:%.*]], label [[IF_END5386:%.*]]
+// SIMD-ONLY0:       if.then5385:
+// SIMD-ONLY0-NEXT:    [[TMP4034:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4034]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5386]]
+// SIMD-ONLY0:       if.end5386:
+// SIMD-ONLY0-NEXT:    [[TMP4035:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4035]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4036:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4037:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5387:%.*]] = icmp eq i64 [[TMP4036]], [[TMP4037]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5387]], label [[IF_THEN5389:%.*]], label [[IF_END5390:%.*]]
+// SIMD-ONLY0:       if.then5389:
+// SIMD-ONLY0-NEXT:    [[TMP4038:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4038]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5390]]
+// SIMD-ONLY0:       if.end5390:
+// SIMD-ONLY0-NEXT:    [[TMP4039:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4039]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4040:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4041:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5391:%.*]] = icmp eq i64 [[TMP4040]], [[TMP4041]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5391]], label [[IF_THEN5393:%.*]], label [[IF_END5394:%.*]]
+// SIMD-ONLY0:       if.then5393:
+// SIMD-ONLY0-NEXT:    [[TMP4042:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4042]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5394]]
+// SIMD-ONLY0:       if.end5394:
+// SIMD-ONLY0-NEXT:    [[TMP4043:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4043]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4044:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4045:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5395:%.*]] = icmp eq i64 [[TMP4044]], [[TMP4045]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5395]], label [[IF_THEN5397:%.*]], label [[IF_ELSE5398:%.*]]
+// SIMD-ONLY0:       if.then5397:
+// SIMD-ONLY0-NEXT:    [[TMP4046:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4046]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5399:%.*]]
+// SIMD-ONLY0:       if.else5398:
+// SIMD-ONLY0-NEXT:    [[TMP4047:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4047]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5399]]
+// SIMD-ONLY0:       if.end5399:
+// SIMD-ONLY0-NEXT:    [[TMP4048:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4049:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5400:%.*]] = icmp eq i64 [[TMP4048]], [[TMP4049]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5400]], label [[IF_THEN5402:%.*]], label [[IF_ELSE5403:%.*]]
+// SIMD-ONLY0:       if.then5402:
+// SIMD-ONLY0-NEXT:    [[TMP4050:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4050]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5404:%.*]]
+// SIMD-ONLY0:       if.else5403:
+// SIMD-ONLY0-NEXT:    [[TMP4051:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4051]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5404]]
+// SIMD-ONLY0:       if.end5404:
+// SIMD-ONLY0-NEXT:    [[TMP4052:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4053:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5405:%.*]] = icmp eq i64 [[TMP4052]], [[TMP4053]]
+// SIMD-ONLY0-NEXT:    [[CONV5406:%.*]] = zext i1 [[CMP5405]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5407:%.*]] = sext i32 [[CONV5406]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5407]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4054:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5408:%.*]] = icmp ne i64 [[TMP4054]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5408]], label [[IF_THEN5409:%.*]], label [[IF_END5410:%.*]]
+// SIMD-ONLY0:       if.then5409:
+// SIMD-ONLY0-NEXT:    [[TMP4055:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4055]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5410]]
+// SIMD-ONLY0:       if.end5410:
+// SIMD-ONLY0-NEXT:    [[TMP4056:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4057:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5411:%.*]] = icmp eq i64 [[TMP4056]], [[TMP4057]]
+// SIMD-ONLY0-NEXT:    [[CONV5412:%.*]] = zext i1 [[CMP5411]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5413:%.*]] = sext i32 [[CONV5412]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5413]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4058:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5414:%.*]] = icmp ne i64 [[TMP4058]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5414]], label [[IF_THEN5415:%.*]], label [[IF_END5416:%.*]]
+// SIMD-ONLY0:       if.then5415:
+// SIMD-ONLY0-NEXT:    [[TMP4059:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4059]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5416]]
+// SIMD-ONLY0:       if.end5416:
+// SIMD-ONLY0-NEXT:    [[TMP4060:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4061:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5417:%.*]] = icmp eq i64 [[TMP4060]], [[TMP4061]]
+// SIMD-ONLY0-NEXT:    [[CONV5418:%.*]] = zext i1 [[CMP5417]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5419:%.*]] = sext i32 [[CONV5418]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5419]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4062:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5420:%.*]] = icmp ne i64 [[TMP4062]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5420]], label [[IF_THEN5421:%.*]], label [[IF_ELSE5422:%.*]]
+// SIMD-ONLY0:       if.then5421:
+// SIMD-ONLY0-NEXT:    [[TMP4063:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4063]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5423:%.*]]
+// SIMD-ONLY0:       if.else5422:
+// SIMD-ONLY0-NEXT:    [[TMP4064:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4064]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5423]]
+// SIMD-ONLY0:       if.end5423:
+// SIMD-ONLY0-NEXT:    [[TMP4065:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4066:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5424:%.*]] = icmp eq i64 [[TMP4065]], [[TMP4066]]
+// SIMD-ONLY0-NEXT:    [[CONV5425:%.*]] = zext i1 [[CMP5424]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5426:%.*]] = sext i32 [[CONV5425]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5426]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4067:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5427:%.*]] = icmp ne i64 [[TMP4067]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5427]], label [[IF_THEN5428:%.*]], label [[IF_ELSE5429:%.*]]
+// SIMD-ONLY0:       if.then5428:
+// SIMD-ONLY0-NEXT:    [[TMP4068:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4068]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5430:%.*]]
+// SIMD-ONLY0:       if.else5429:
+// SIMD-ONLY0-NEXT:    [[TMP4069:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4069]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5430]]
+// SIMD-ONLY0:       if.end5430:
+// SIMD-ONLY0-NEXT:    [[TMP4070:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4070]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4071:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4072:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5431:%.*]] = icmp ugt i64 [[TMP4071]], [[TMP4072]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5431]], label [[IF_THEN5433:%.*]], label [[IF_END5434:%.*]]
+// SIMD-ONLY0:       if.then5433:
+// SIMD-ONLY0-NEXT:    [[TMP4073:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4073]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5434]]
+// SIMD-ONLY0:       if.end5434:
+// SIMD-ONLY0-NEXT:    [[TMP4074:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4074]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4075:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4076:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5435:%.*]] = icmp ugt i64 [[TMP4075]], [[TMP4076]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5435]], label [[IF_THEN5437:%.*]], label [[IF_END5438:%.*]]
+// SIMD-ONLY0:       if.then5437:
+// SIMD-ONLY0-NEXT:    [[TMP4077:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4077]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5438]]
+// SIMD-ONLY0:       if.end5438:
+// SIMD-ONLY0-NEXT:    [[TMP4078:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4078]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4079:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4080:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5439:%.*]] = icmp ult i64 [[TMP4079]], [[TMP4080]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5439]], label [[IF_THEN5441:%.*]], label [[IF_END5442:%.*]]
+// SIMD-ONLY0:       if.then5441:
+// SIMD-ONLY0-NEXT:    [[TMP4081:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4081]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5442]]
+// SIMD-ONLY0:       if.end5442:
+// SIMD-ONLY0-NEXT:    [[TMP4082:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4082]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4083:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4084:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5443:%.*]] = icmp ult i64 [[TMP4083]], [[TMP4084]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5443]], label [[IF_THEN5445:%.*]], label [[IF_END5446:%.*]]
+// SIMD-ONLY0:       if.then5445:
+// SIMD-ONLY0-NEXT:    [[TMP4085:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4085]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5446]]
+// SIMD-ONLY0:       if.end5446:
+// SIMD-ONLY0-NEXT:    [[TMP4086:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4086]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4087:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4088:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5447:%.*]] = icmp eq i64 [[TMP4087]], [[TMP4088]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5447]], label [[IF_THEN5449:%.*]], label [[IF_END5450:%.*]]
+// SIMD-ONLY0:       if.then5449:
+// SIMD-ONLY0-NEXT:    [[TMP4089:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4089]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5450]]
+// SIMD-ONLY0:       if.end5450:
+// SIMD-ONLY0-NEXT:    [[TMP4090:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4090]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4091:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4092:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5451:%.*]] = icmp eq i64 [[TMP4091]], [[TMP4092]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5451]], label [[IF_THEN5453:%.*]], label [[IF_END5454:%.*]]
+// SIMD-ONLY0:       if.then5453:
+// SIMD-ONLY0-NEXT:    [[TMP4093:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4093]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5454]]
+// SIMD-ONLY0:       if.end5454:
+// SIMD-ONLY0-NEXT:    [[TMP4094:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4095:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5455:%.*]] = icmp ugt i64 [[TMP4094]], [[TMP4095]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5455]], label [[IF_THEN5457:%.*]], label [[IF_END5458:%.*]]
+// SIMD-ONLY0:       if.then5457:
+// SIMD-ONLY0-NEXT:    [[TMP4096:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4096]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5458]]
+// SIMD-ONLY0:       if.end5458:
+// SIMD-ONLY0-NEXT:    [[TMP4097:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4097]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4098:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4099:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5459:%.*]] = icmp ugt i64 [[TMP4098]], [[TMP4099]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5459]], label [[IF_THEN5461:%.*]], label [[IF_END5462:%.*]]
+// SIMD-ONLY0:       if.then5461:
+// SIMD-ONLY0-NEXT:    [[TMP4100:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4100]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5462]]
+// SIMD-ONLY0:       if.end5462:
+// SIMD-ONLY0-NEXT:    [[TMP4101:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4101]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4102:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4103:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5463:%.*]] = icmp ult i64 [[TMP4102]], [[TMP4103]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5463]], label [[IF_THEN5465:%.*]], label [[IF_END5466:%.*]]
+// SIMD-ONLY0:       if.then5465:
+// SIMD-ONLY0-NEXT:    [[TMP4104:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4104]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5466]]
+// SIMD-ONLY0:       if.end5466:
+// SIMD-ONLY0-NEXT:    [[TMP4105:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4105]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4106:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4107:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5467:%.*]] = icmp ult i64 [[TMP4106]], [[TMP4107]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5467]], label [[IF_THEN5469:%.*]], label [[IF_END5470:%.*]]
+// SIMD-ONLY0:       if.then5469:
+// SIMD-ONLY0-NEXT:    [[TMP4108:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4108]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5470]]
+// SIMD-ONLY0:       if.end5470:
+// SIMD-ONLY0-NEXT:    [[TMP4109:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4109]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4110:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4111:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5471:%.*]] = icmp eq i64 [[TMP4110]], [[TMP4111]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5471]], label [[IF_THEN5473:%.*]], label [[IF_END5474:%.*]]
+// SIMD-ONLY0:       if.then5473:
+// SIMD-ONLY0-NEXT:    [[TMP4112:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4112]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5474]]
+// SIMD-ONLY0:       if.end5474:
+// SIMD-ONLY0-NEXT:    [[TMP4113:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4113]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4114:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4115:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5475:%.*]] = icmp eq i64 [[TMP4114]], [[TMP4115]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5475]], label [[IF_THEN5477:%.*]], label [[IF_END5478:%.*]]
+// SIMD-ONLY0:       if.then5477:
+// SIMD-ONLY0-NEXT:    [[TMP4116:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4116]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5478]]
+// SIMD-ONLY0:       if.end5478:
+// SIMD-ONLY0-NEXT:    [[TMP4117:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4117]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4118:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4119:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5479:%.*]] = icmp eq i64 [[TMP4118]], [[TMP4119]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5479]], label [[IF_THEN5481:%.*]], label [[IF_ELSE5482:%.*]]
+// SIMD-ONLY0:       if.then5481:
+// SIMD-ONLY0-NEXT:    [[TMP4120:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4120]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5483:%.*]]
+// SIMD-ONLY0:       if.else5482:
+// SIMD-ONLY0-NEXT:    [[TMP4121:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4121]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5483]]
+// SIMD-ONLY0:       if.end5483:
+// SIMD-ONLY0-NEXT:    [[TMP4122:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4123:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5484:%.*]] = icmp eq i64 [[TMP4122]], [[TMP4123]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5484]], label [[IF_THEN5486:%.*]], label [[IF_ELSE5487:%.*]]
+// SIMD-ONLY0:       if.then5486:
+// SIMD-ONLY0-NEXT:    [[TMP4124:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4124]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5488:%.*]]
+// SIMD-ONLY0:       if.else5487:
+// SIMD-ONLY0-NEXT:    [[TMP4125:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4125]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5488]]
+// SIMD-ONLY0:       if.end5488:
+// SIMD-ONLY0-NEXT:    [[TMP4126:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4127:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5489:%.*]] = icmp eq i64 [[TMP4126]], [[TMP4127]]
+// SIMD-ONLY0-NEXT:    [[CONV5490:%.*]] = zext i1 [[CMP5489]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5491:%.*]] = sext i32 [[CONV5490]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5491]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4128:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5492:%.*]] = icmp ne i64 [[TMP4128]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5492]], label [[IF_THEN5493:%.*]], label [[IF_END5494:%.*]]
+// SIMD-ONLY0:       if.then5493:
+// SIMD-ONLY0-NEXT:    [[TMP4129:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4129]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5494]]
+// SIMD-ONLY0:       if.end5494:
+// SIMD-ONLY0-NEXT:    [[TMP4130:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4131:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5495:%.*]] = icmp eq i64 [[TMP4130]], [[TMP4131]]
+// SIMD-ONLY0-NEXT:    [[CONV5496:%.*]] = zext i1 [[CMP5495]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5497:%.*]] = sext i32 [[CONV5496]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5497]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4132:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5498:%.*]] = icmp ne i64 [[TMP4132]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5498]], label [[IF_THEN5499:%.*]], label [[IF_END5500:%.*]]
+// SIMD-ONLY0:       if.then5499:
+// SIMD-ONLY0-NEXT:    [[TMP4133:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4133]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5500]]
+// SIMD-ONLY0:       if.end5500:
+// SIMD-ONLY0-NEXT:    [[TMP4134:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4135:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5501:%.*]] = icmp eq i64 [[TMP4134]], [[TMP4135]]
+// SIMD-ONLY0-NEXT:    [[CONV5502:%.*]] = zext i1 [[CMP5501]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5503:%.*]] = sext i32 [[CONV5502]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5503]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4136:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5504:%.*]] = icmp ne i64 [[TMP4136]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5504]], label [[IF_THEN5505:%.*]], label [[IF_ELSE5506:%.*]]
+// SIMD-ONLY0:       if.then5505:
+// SIMD-ONLY0-NEXT:    [[TMP4137:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4137]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5507:%.*]]
+// SIMD-ONLY0:       if.else5506:
+// SIMD-ONLY0-NEXT:    [[TMP4138:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4138]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5507]]
+// SIMD-ONLY0:       if.end5507:
+// SIMD-ONLY0-NEXT:    [[TMP4139:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4140:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5508:%.*]] = icmp eq i64 [[TMP4139]], [[TMP4140]]
+// SIMD-ONLY0-NEXT:    [[CONV5509:%.*]] = zext i1 [[CMP5508]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5510:%.*]] = sext i32 [[CONV5509]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5510]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4141:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5511:%.*]] = icmp ne i64 [[TMP4141]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5511]], label [[IF_THEN5512:%.*]], label [[IF_ELSE5513:%.*]]
+// SIMD-ONLY0:       if.then5512:
+// SIMD-ONLY0-NEXT:    [[TMP4142:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4142]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5514:%.*]]
+// SIMD-ONLY0:       if.else5513:
+// SIMD-ONLY0-NEXT:    [[TMP4143:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4143]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5514]]
+// SIMD-ONLY0:       if.end5514:
+// SIMD-ONLY0-NEXT:    [[TMP4144:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4144]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4145:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4146:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5515:%.*]] = icmp ugt i64 [[TMP4145]], [[TMP4146]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5515]], label [[IF_THEN5517:%.*]], label [[IF_END5518:%.*]]
+// SIMD-ONLY0:       if.then5517:
+// SIMD-ONLY0-NEXT:    [[TMP4147:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4147]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5518]]
+// SIMD-ONLY0:       if.end5518:
+// SIMD-ONLY0-NEXT:    [[TMP4148:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4148]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4149:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4150:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5519:%.*]] = icmp ugt i64 [[TMP4149]], [[TMP4150]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5519]], label [[IF_THEN5521:%.*]], label [[IF_END5522:%.*]]
+// SIMD-ONLY0:       if.then5521:
+// SIMD-ONLY0-NEXT:    [[TMP4151:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4151]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5522]]
+// SIMD-ONLY0:       if.end5522:
+// SIMD-ONLY0-NEXT:    [[TMP4152:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4152]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4153:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4154:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5523:%.*]] = icmp ult i64 [[TMP4153]], [[TMP4154]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5523]], label [[IF_THEN5525:%.*]], label [[IF_END5526:%.*]]
+// SIMD-ONLY0:       if.then5525:
+// SIMD-ONLY0-NEXT:    [[TMP4155:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4155]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5526]]
+// SIMD-ONLY0:       if.end5526:
+// SIMD-ONLY0-NEXT:    [[TMP4156:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4156]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4157:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4158:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5527:%.*]] = icmp ult i64 [[TMP4157]], [[TMP4158]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5527]], label [[IF_THEN5529:%.*]], label [[IF_END5530:%.*]]
+// SIMD-ONLY0:       if.then5529:
+// SIMD-ONLY0-NEXT:    [[TMP4159:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4159]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5530]]
+// SIMD-ONLY0:       if.end5530:
+// SIMD-ONLY0-NEXT:    [[TMP4160:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4160]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4161:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4162:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5531:%.*]] = icmp eq i64 [[TMP4161]], [[TMP4162]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5531]], label [[IF_THEN5533:%.*]], label [[IF_END5534:%.*]]
+// SIMD-ONLY0:       if.then5533:
+// SIMD-ONLY0-NEXT:    [[TMP4163:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4163]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5534]]
+// SIMD-ONLY0:       if.end5534:
+// SIMD-ONLY0-NEXT:    [[TMP4164:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4164]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4165:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4166:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5535:%.*]] = icmp eq i64 [[TMP4165]], [[TMP4166]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5535]], label [[IF_THEN5537:%.*]], label [[IF_END5538:%.*]]
+// SIMD-ONLY0:       if.then5537:
+// SIMD-ONLY0-NEXT:    [[TMP4167:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4167]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5538]]
+// SIMD-ONLY0:       if.end5538:
+// SIMD-ONLY0-NEXT:    [[TMP4168:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4169:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5539:%.*]] = icmp ugt i64 [[TMP4168]], [[TMP4169]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5539]], label [[IF_THEN5541:%.*]], label [[IF_END5542:%.*]]
+// SIMD-ONLY0:       if.then5541:
+// SIMD-ONLY0-NEXT:    [[TMP4170:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4170]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5542]]
+// SIMD-ONLY0:       if.end5542:
+// SIMD-ONLY0-NEXT:    [[TMP4171:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4171]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4172:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4173:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5543:%.*]] = icmp ugt i64 [[TMP4172]], [[TMP4173]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5543]], label [[IF_THEN5545:%.*]], label [[IF_END5546:%.*]]
+// SIMD-ONLY0:       if.then5545:
+// SIMD-ONLY0-NEXT:    [[TMP4174:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4174]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5546]]
+// SIMD-ONLY0:       if.end5546:
+// SIMD-ONLY0-NEXT:    [[TMP4175:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4175]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4176:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4177:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5547:%.*]] = icmp ult i64 [[TMP4176]], [[TMP4177]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5547]], label [[IF_THEN5549:%.*]], label [[IF_END5550:%.*]]
+// SIMD-ONLY0:       if.then5549:
+// SIMD-ONLY0-NEXT:    [[TMP4178:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4178]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5550]]
+// SIMD-ONLY0:       if.end5550:
+// SIMD-ONLY0-NEXT:    [[TMP4179:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4179]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4180:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4181:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5551:%.*]] = icmp ult i64 [[TMP4180]], [[TMP4181]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5551]], label [[IF_THEN5553:%.*]], label [[IF_END5554:%.*]]
+// SIMD-ONLY0:       if.then5553:
+// SIMD-ONLY0-NEXT:    [[TMP4182:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4182]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5554]]
+// SIMD-ONLY0:       if.end5554:
+// SIMD-ONLY0-NEXT:    [[TMP4183:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4183]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4184:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4185:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5555:%.*]] = icmp eq i64 [[TMP4184]], [[TMP4185]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5555]], label [[IF_THEN5557:%.*]], label [[IF_END5558:%.*]]
+// SIMD-ONLY0:       if.then5557:
+// SIMD-ONLY0-NEXT:    [[TMP4186:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4186]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5558]]
+// SIMD-ONLY0:       if.end5558:
+// SIMD-ONLY0-NEXT:    [[TMP4187:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4187]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4188:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4189:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5559:%.*]] = icmp eq i64 [[TMP4188]], [[TMP4189]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5559]], label [[IF_THEN5561:%.*]], label [[IF_END5562:%.*]]
+// SIMD-ONLY0:       if.then5561:
+// SIMD-ONLY0-NEXT:    [[TMP4190:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4190]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5562]]
+// SIMD-ONLY0:       if.end5562:
+// SIMD-ONLY0-NEXT:    [[TMP4191:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4191]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4192:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4193:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5563:%.*]] = icmp eq i64 [[TMP4192]], [[TMP4193]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5563]], label [[IF_THEN5565:%.*]], label [[IF_ELSE5566:%.*]]
+// SIMD-ONLY0:       if.then5565:
+// SIMD-ONLY0-NEXT:    [[TMP4194:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4194]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5567:%.*]]
+// SIMD-ONLY0:       if.else5566:
+// SIMD-ONLY0-NEXT:    [[TMP4195:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4195]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5567]]
+// SIMD-ONLY0:       if.end5567:
+// SIMD-ONLY0-NEXT:    [[TMP4196:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4197:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5568:%.*]] = icmp eq i64 [[TMP4196]], [[TMP4197]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5568]], label [[IF_THEN5570:%.*]], label [[IF_ELSE5571:%.*]]
+// SIMD-ONLY0:       if.then5570:
+// SIMD-ONLY0-NEXT:    [[TMP4198:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4198]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5572:%.*]]
+// SIMD-ONLY0:       if.else5571:
+// SIMD-ONLY0-NEXT:    [[TMP4199:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4199]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5572]]
+// SIMD-ONLY0:       if.end5572:
+// SIMD-ONLY0-NEXT:    [[TMP4200:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4201:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5573:%.*]] = icmp eq i64 [[TMP4200]], [[TMP4201]]
+// SIMD-ONLY0-NEXT:    [[CONV5574:%.*]] = zext i1 [[CMP5573]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5575:%.*]] = sext i32 [[CONV5574]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5575]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4202:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5576:%.*]] = icmp ne i64 [[TMP4202]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5576]], label [[IF_THEN5577:%.*]], label [[IF_END5578:%.*]]
+// SIMD-ONLY0:       if.then5577:
+// SIMD-ONLY0-NEXT:    [[TMP4203:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4203]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5578]]
+// SIMD-ONLY0:       if.end5578:
+// SIMD-ONLY0-NEXT:    [[TMP4204:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4205:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5579:%.*]] = icmp eq i64 [[TMP4204]], [[TMP4205]]
+// SIMD-ONLY0-NEXT:    [[CONV5580:%.*]] = zext i1 [[CMP5579]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5581:%.*]] = sext i32 [[CONV5580]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5581]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4206:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5582:%.*]] = icmp ne i64 [[TMP4206]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5582]], label [[IF_THEN5583:%.*]], label [[IF_END5584:%.*]]
+// SIMD-ONLY0:       if.then5583:
+// SIMD-ONLY0-NEXT:    [[TMP4207:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4207]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5584]]
+// SIMD-ONLY0:       if.end5584:
+// SIMD-ONLY0-NEXT:    [[TMP4208:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4209:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5585:%.*]] = icmp eq i64 [[TMP4208]], [[TMP4209]]
+// SIMD-ONLY0-NEXT:    [[CONV5586:%.*]] = zext i1 [[CMP5585]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5587:%.*]] = sext i32 [[CONV5586]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5587]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4210:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5588:%.*]] = icmp ne i64 [[TMP4210]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5588]], label [[IF_THEN5589:%.*]], label [[IF_ELSE5590:%.*]]
+// SIMD-ONLY0:       if.then5589:
+// SIMD-ONLY0-NEXT:    [[TMP4211:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4211]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5591:%.*]]
+// SIMD-ONLY0:       if.else5590:
+// SIMD-ONLY0-NEXT:    [[TMP4212:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4212]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5591]]
+// SIMD-ONLY0:       if.end5591:
+// SIMD-ONLY0-NEXT:    [[TMP4213:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4214:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5592:%.*]] = icmp eq i64 [[TMP4213]], [[TMP4214]]
+// SIMD-ONLY0-NEXT:    [[CONV5593:%.*]] = zext i1 [[CMP5592]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5594:%.*]] = sext i32 [[CONV5593]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5594]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4215:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5595:%.*]] = icmp ne i64 [[TMP4215]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5595]], label [[IF_THEN5596:%.*]], label [[IF_ELSE5597:%.*]]
+// SIMD-ONLY0:       if.then5596:
+// SIMD-ONLY0-NEXT:    [[TMP4216:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4216]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5598:%.*]]
+// SIMD-ONLY0:       if.else5597:
+// SIMD-ONLY0-NEXT:    [[TMP4217:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4217]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5598]]
+// SIMD-ONLY0:       if.end5598:
+// SIMD-ONLY0-NEXT:    [[TMP4218:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4218]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4219:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4220:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5599:%.*]] = icmp ugt i64 [[TMP4219]], [[TMP4220]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5599]], label [[IF_THEN5601:%.*]], label [[IF_END5602:%.*]]
+// SIMD-ONLY0:       if.then5601:
+// SIMD-ONLY0-NEXT:    [[TMP4221:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4221]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5602]]
+// SIMD-ONLY0:       if.end5602:
+// SIMD-ONLY0-NEXT:    [[TMP4222:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4222]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4223:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4224:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5603:%.*]] = icmp ugt i64 [[TMP4223]], [[TMP4224]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5603]], label [[IF_THEN5605:%.*]], label [[IF_END5606:%.*]]
+// SIMD-ONLY0:       if.then5605:
+// SIMD-ONLY0-NEXT:    [[TMP4225:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4225]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5606]]
+// SIMD-ONLY0:       if.end5606:
+// SIMD-ONLY0-NEXT:    [[TMP4226:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4226]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4227:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4228:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5607:%.*]] = icmp ult i64 [[TMP4227]], [[TMP4228]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5607]], label [[IF_THEN5609:%.*]], label [[IF_END5610:%.*]]
+// SIMD-ONLY0:       if.then5609:
+// SIMD-ONLY0-NEXT:    [[TMP4229:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4229]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5610]]
+// SIMD-ONLY0:       if.end5610:
+// SIMD-ONLY0-NEXT:    [[TMP4230:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4230]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4231:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4232:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5611:%.*]] = icmp ult i64 [[TMP4231]], [[TMP4232]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5611]], label [[IF_THEN5613:%.*]], label [[IF_END5614:%.*]]
+// SIMD-ONLY0:       if.then5613:
+// SIMD-ONLY0-NEXT:    [[TMP4233:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4233]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5614]]
+// SIMD-ONLY0:       if.end5614:
+// SIMD-ONLY0-NEXT:    [[TMP4234:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4234]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4235:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4236:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5615:%.*]] = icmp eq i64 [[TMP4235]], [[TMP4236]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5615]], label [[IF_THEN5617:%.*]], label [[IF_END5618:%.*]]
+// SIMD-ONLY0:       if.then5617:
+// SIMD-ONLY0-NEXT:    [[TMP4237:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4237]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5618]]
+// SIMD-ONLY0:       if.end5618:
+// SIMD-ONLY0-NEXT:    [[TMP4238:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4238]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4239:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4240:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5619:%.*]] = icmp eq i64 [[TMP4239]], [[TMP4240]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5619]], label [[IF_THEN5621:%.*]], label [[IF_END5622:%.*]]
+// SIMD-ONLY0:       if.then5621:
+// SIMD-ONLY0-NEXT:    [[TMP4241:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4241]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5622]]
+// SIMD-ONLY0:       if.end5622:
+// SIMD-ONLY0-NEXT:    [[TMP4242:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4243:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5623:%.*]] = icmp ugt i64 [[TMP4242]], [[TMP4243]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5623]], label [[IF_THEN5625:%.*]], label [[IF_END5626:%.*]]
+// SIMD-ONLY0:       if.then5625:
+// SIMD-ONLY0-NEXT:    [[TMP4244:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4244]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5626]]
+// SIMD-ONLY0:       if.end5626:
+// SIMD-ONLY0-NEXT:    [[TMP4245:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4245]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4246:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4247:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5627:%.*]] = icmp ugt i64 [[TMP4246]], [[TMP4247]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5627]], label [[IF_THEN5629:%.*]], label [[IF_END5630:%.*]]
+// SIMD-ONLY0:       if.then5629:
+// SIMD-ONLY0-NEXT:    [[TMP4248:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4248]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5630]]
+// SIMD-ONLY0:       if.end5630:
+// SIMD-ONLY0-NEXT:    [[TMP4249:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4249]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4250:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4251:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5631:%.*]] = icmp ult i64 [[TMP4250]], [[TMP4251]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5631]], label [[IF_THEN5633:%.*]], label [[IF_END5634:%.*]]
+// SIMD-ONLY0:       if.then5633:
+// SIMD-ONLY0-NEXT:    [[TMP4252:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4252]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5634]]
+// SIMD-ONLY0:       if.end5634:
+// SIMD-ONLY0-NEXT:    [[TMP4253:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4253]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4254:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4255:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5635:%.*]] = icmp ult i64 [[TMP4254]], [[TMP4255]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5635]], label [[IF_THEN5637:%.*]], label [[IF_END5638:%.*]]
+// SIMD-ONLY0:       if.then5637:
+// SIMD-ONLY0-NEXT:    [[TMP4256:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4256]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5638]]
+// SIMD-ONLY0:       if.end5638:
+// SIMD-ONLY0-NEXT:    [[TMP4257:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4257]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4258:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4259:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5639:%.*]] = icmp eq i64 [[TMP4258]], [[TMP4259]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5639]], label [[IF_THEN5641:%.*]], label [[IF_END5642:%.*]]
+// SIMD-ONLY0:       if.then5641:
+// SIMD-ONLY0-NEXT:    [[TMP4260:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4260]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5642]]
+// SIMD-ONLY0:       if.end5642:
+// SIMD-ONLY0-NEXT:    [[TMP4261:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4261]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4262:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4263:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5643:%.*]] = icmp eq i64 [[TMP4262]], [[TMP4263]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5643]], label [[IF_THEN5645:%.*]], label [[IF_END5646:%.*]]
+// SIMD-ONLY0:       if.then5645:
+// SIMD-ONLY0-NEXT:    [[TMP4264:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4264]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5646]]
+// SIMD-ONLY0:       if.end5646:
+// SIMD-ONLY0-NEXT:    [[TMP4265:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4265]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4266:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4267:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5647:%.*]] = icmp eq i64 [[TMP4266]], [[TMP4267]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5647]], label [[IF_THEN5649:%.*]], label [[IF_ELSE5650:%.*]]
+// SIMD-ONLY0:       if.then5649:
+// SIMD-ONLY0-NEXT:    [[TMP4268:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4268]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5651:%.*]]
+// SIMD-ONLY0:       if.else5650:
+// SIMD-ONLY0-NEXT:    [[TMP4269:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4269]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5651]]
+// SIMD-ONLY0:       if.end5651:
+// SIMD-ONLY0-NEXT:    [[TMP4270:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4271:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5652:%.*]] = icmp eq i64 [[TMP4270]], [[TMP4271]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5652]], label [[IF_THEN5654:%.*]], label [[IF_ELSE5655:%.*]]
+// SIMD-ONLY0:       if.then5654:
+// SIMD-ONLY0-NEXT:    [[TMP4272:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4272]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5656:%.*]]
+// SIMD-ONLY0:       if.else5655:
+// SIMD-ONLY0-NEXT:    [[TMP4273:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4273]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5656]]
+// SIMD-ONLY0:       if.end5656:
+// SIMD-ONLY0-NEXT:    [[TMP4274:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4275:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5657:%.*]] = icmp eq i64 [[TMP4274]], [[TMP4275]]
+// SIMD-ONLY0-NEXT:    [[CONV5658:%.*]] = zext i1 [[CMP5657]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5659:%.*]] = sext i32 [[CONV5658]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5659]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4276:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5660:%.*]] = icmp ne i64 [[TMP4276]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5660]], label [[IF_THEN5661:%.*]], label [[IF_END5662:%.*]]
+// SIMD-ONLY0:       if.then5661:
+// SIMD-ONLY0-NEXT:    [[TMP4277:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4277]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5662]]
+// SIMD-ONLY0:       if.end5662:
+// SIMD-ONLY0-NEXT:    [[TMP4278:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4279:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5663:%.*]] = icmp eq i64 [[TMP4278]], [[TMP4279]]
+// SIMD-ONLY0-NEXT:    [[CONV5664:%.*]] = zext i1 [[CMP5663]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5665:%.*]] = sext i32 [[CONV5664]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5665]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4280:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5666:%.*]] = icmp ne i64 [[TMP4280]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5666]], label [[IF_THEN5667:%.*]], label [[IF_END5668:%.*]]
+// SIMD-ONLY0:       if.then5667:
+// SIMD-ONLY0-NEXT:    [[TMP4281:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4281]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5668]]
+// SIMD-ONLY0:       if.end5668:
+// SIMD-ONLY0-NEXT:    [[TMP4282:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4283:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5669:%.*]] = icmp eq i64 [[TMP4282]], [[TMP4283]]
+// SIMD-ONLY0-NEXT:    [[CONV5670:%.*]] = zext i1 [[CMP5669]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5671:%.*]] = sext i32 [[CONV5670]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5671]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4284:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5672:%.*]] = icmp ne i64 [[TMP4284]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5672]], label [[IF_THEN5673:%.*]], label [[IF_ELSE5674:%.*]]
+// SIMD-ONLY0:       if.then5673:
+// SIMD-ONLY0-NEXT:    [[TMP4285:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4285]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5675:%.*]]
+// SIMD-ONLY0:       if.else5674:
+// SIMD-ONLY0-NEXT:    [[TMP4286:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4286]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5675]]
+// SIMD-ONLY0:       if.end5675:
+// SIMD-ONLY0-NEXT:    [[TMP4287:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4288:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5676:%.*]] = icmp eq i64 [[TMP4287]], [[TMP4288]]
+// SIMD-ONLY0-NEXT:    [[CONV5677:%.*]] = zext i1 [[CMP5676]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5678:%.*]] = sext i32 [[CONV5677]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5678]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4289:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5679:%.*]] = icmp ne i64 [[TMP4289]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5679]], label [[IF_THEN5680:%.*]], label [[IF_ELSE5681:%.*]]
+// SIMD-ONLY0:       if.then5680:
+// SIMD-ONLY0-NEXT:    [[TMP4290:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4290]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5682:%.*]]
+// SIMD-ONLY0:       if.else5681:
+// SIMD-ONLY0-NEXT:    [[TMP4291:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4291]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5682]]
+// SIMD-ONLY0:       if.end5682:
+// SIMD-ONLY0-NEXT:    [[TMP4292:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4292]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4293:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4294:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5683:%.*]] = icmp ugt i64 [[TMP4293]], [[TMP4294]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5683]], label [[IF_THEN5685:%.*]], label [[IF_END5686:%.*]]
+// SIMD-ONLY0:       if.then5685:
+// SIMD-ONLY0-NEXT:    [[TMP4295:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4295]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5686]]
+// SIMD-ONLY0:       if.end5686:
+// SIMD-ONLY0-NEXT:    [[TMP4296:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4296]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4297:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4298:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5687:%.*]] = icmp ugt i64 [[TMP4297]], [[TMP4298]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5687]], label [[IF_THEN5689:%.*]], label [[IF_END5690:%.*]]
+// SIMD-ONLY0:       if.then5689:
+// SIMD-ONLY0-NEXT:    [[TMP4299:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4299]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5690]]
+// SIMD-ONLY0:       if.end5690:
+// SIMD-ONLY0-NEXT:    [[TMP4300:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4300]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4301:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4302:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5691:%.*]] = icmp ult i64 [[TMP4301]], [[TMP4302]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5691]], label [[IF_THEN5693:%.*]], label [[IF_END5694:%.*]]
+// SIMD-ONLY0:       if.then5693:
+// SIMD-ONLY0-NEXT:    [[TMP4303:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4303]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5694]]
+// SIMD-ONLY0:       if.end5694:
+// SIMD-ONLY0-NEXT:    [[TMP4304:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4304]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4305:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4306:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5695:%.*]] = icmp ult i64 [[TMP4305]], [[TMP4306]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5695]], label [[IF_THEN5697:%.*]], label [[IF_END5698:%.*]]
+// SIMD-ONLY0:       if.then5697:
+// SIMD-ONLY0-NEXT:    [[TMP4307:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4307]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5698]]
+// SIMD-ONLY0:       if.end5698:
+// SIMD-ONLY0-NEXT:    [[TMP4308:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4308]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4309:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4310:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5699:%.*]] = icmp eq i64 [[TMP4309]], [[TMP4310]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5699]], label [[IF_THEN5701:%.*]], label [[IF_END5702:%.*]]
+// SIMD-ONLY0:       if.then5701:
+// SIMD-ONLY0-NEXT:    [[TMP4311:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4311]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5702]]
+// SIMD-ONLY0:       if.end5702:
+// SIMD-ONLY0-NEXT:    [[TMP4312:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4312]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4313:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4314:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5703:%.*]] = icmp eq i64 [[TMP4313]], [[TMP4314]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5703]], label [[IF_THEN5705:%.*]], label [[IF_END5706:%.*]]
+// SIMD-ONLY0:       if.then5705:
+// SIMD-ONLY0-NEXT:    [[TMP4315:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4315]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5706]]
+// SIMD-ONLY0:       if.end5706:
+// SIMD-ONLY0-NEXT:    [[TMP4316:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4317:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5707:%.*]] = icmp ugt i64 [[TMP4316]], [[TMP4317]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5707]], label [[IF_THEN5709:%.*]], label [[IF_END5710:%.*]]
+// SIMD-ONLY0:       if.then5709:
+// SIMD-ONLY0-NEXT:    [[TMP4318:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4318]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5710]]
+// SIMD-ONLY0:       if.end5710:
+// SIMD-ONLY0-NEXT:    [[TMP4319:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4319]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4320:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4321:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5711:%.*]] = icmp ugt i64 [[TMP4320]], [[TMP4321]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5711]], label [[IF_THEN5713:%.*]], label [[IF_END5714:%.*]]
+// SIMD-ONLY0:       if.then5713:
+// SIMD-ONLY0-NEXT:    [[TMP4322:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4322]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5714]]
+// SIMD-ONLY0:       if.end5714:
+// SIMD-ONLY0-NEXT:    [[TMP4323:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4323]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4324:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4325:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5715:%.*]] = icmp ult i64 [[TMP4324]], [[TMP4325]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5715]], label [[IF_THEN5717:%.*]], label [[IF_END5718:%.*]]
+// SIMD-ONLY0:       if.then5717:
+// SIMD-ONLY0-NEXT:    [[TMP4326:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4326]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5718]]
+// SIMD-ONLY0:       if.end5718:
+// SIMD-ONLY0-NEXT:    [[TMP4327:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4327]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4328:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4329:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5719:%.*]] = icmp ult i64 [[TMP4328]], [[TMP4329]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5719]], label [[IF_THEN5721:%.*]], label [[IF_END5722:%.*]]
+// SIMD-ONLY0:       if.then5721:
+// SIMD-ONLY0-NEXT:    [[TMP4330:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4330]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5722]]
+// SIMD-ONLY0:       if.end5722:
+// SIMD-ONLY0-NEXT:    [[TMP4331:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4331]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4332:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4333:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5723:%.*]] = icmp eq i64 [[TMP4332]], [[TMP4333]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5723]], label [[IF_THEN5725:%.*]], label [[IF_END5726:%.*]]
+// SIMD-ONLY0:       if.then5725:
+// SIMD-ONLY0-NEXT:    [[TMP4334:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4334]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5726]]
+// SIMD-ONLY0:       if.end5726:
+// SIMD-ONLY0-NEXT:    [[TMP4335:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4335]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4336:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4337:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5727:%.*]] = icmp eq i64 [[TMP4336]], [[TMP4337]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5727]], label [[IF_THEN5729:%.*]], label [[IF_END5730:%.*]]
+// SIMD-ONLY0:       if.then5729:
+// SIMD-ONLY0-NEXT:    [[TMP4338:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4338]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5730]]
+// SIMD-ONLY0:       if.end5730:
+// SIMD-ONLY0-NEXT:    [[TMP4339:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4339]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4340:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4341:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5731:%.*]] = icmp eq i64 [[TMP4340]], [[TMP4341]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5731]], label [[IF_THEN5733:%.*]], label [[IF_ELSE5734:%.*]]
+// SIMD-ONLY0:       if.then5733:
+// SIMD-ONLY0-NEXT:    [[TMP4342:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4342]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5735:%.*]]
+// SIMD-ONLY0:       if.else5734:
+// SIMD-ONLY0-NEXT:    [[TMP4343:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4343]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5735]]
+// SIMD-ONLY0:       if.end5735:
+// SIMD-ONLY0-NEXT:    [[TMP4344:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4345:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5736:%.*]] = icmp eq i64 [[TMP4344]], [[TMP4345]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5736]], label [[IF_THEN5738:%.*]], label [[IF_ELSE5739:%.*]]
+// SIMD-ONLY0:       if.then5738:
+// SIMD-ONLY0-NEXT:    [[TMP4346:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4346]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5740:%.*]]
+// SIMD-ONLY0:       if.else5739:
+// SIMD-ONLY0-NEXT:    [[TMP4347:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4347]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5740]]
+// SIMD-ONLY0:       if.end5740:
+// SIMD-ONLY0-NEXT:    [[TMP4348:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4349:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5741:%.*]] = icmp eq i64 [[TMP4348]], [[TMP4349]]
+// SIMD-ONLY0-NEXT:    [[CONV5742:%.*]] = zext i1 [[CMP5741]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5743:%.*]] = sext i32 [[CONV5742]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5743]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4350:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5744:%.*]] = icmp ne i64 [[TMP4350]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5744]], label [[IF_THEN5745:%.*]], label [[IF_END5746:%.*]]
+// SIMD-ONLY0:       if.then5745:
+// SIMD-ONLY0-NEXT:    [[TMP4351:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4351]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5746]]
+// SIMD-ONLY0:       if.end5746:
+// SIMD-ONLY0-NEXT:    [[TMP4352:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4353:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5747:%.*]] = icmp eq i64 [[TMP4352]], [[TMP4353]]
+// SIMD-ONLY0-NEXT:    [[CONV5748:%.*]] = zext i1 [[CMP5747]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5749:%.*]] = sext i32 [[CONV5748]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5749]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4354:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5750:%.*]] = icmp ne i64 [[TMP4354]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5750]], label [[IF_THEN5751:%.*]], label [[IF_END5752:%.*]]
+// SIMD-ONLY0:       if.then5751:
+// SIMD-ONLY0-NEXT:    [[TMP4355:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4355]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5752]]
+// SIMD-ONLY0:       if.end5752:
+// SIMD-ONLY0-NEXT:    [[TMP4356:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4357:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5753:%.*]] = icmp eq i64 [[TMP4356]], [[TMP4357]]
+// SIMD-ONLY0-NEXT:    [[CONV5754:%.*]] = zext i1 [[CMP5753]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5755:%.*]] = sext i32 [[CONV5754]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5755]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4358:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5756:%.*]] = icmp ne i64 [[TMP4358]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5756]], label [[IF_THEN5757:%.*]], label [[IF_ELSE5758:%.*]]
+// SIMD-ONLY0:       if.then5757:
+// SIMD-ONLY0-NEXT:    [[TMP4359:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4359]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5759:%.*]]
+// SIMD-ONLY0:       if.else5758:
+// SIMD-ONLY0-NEXT:    [[TMP4360:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4360]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5759]]
+// SIMD-ONLY0:       if.end5759:
+// SIMD-ONLY0-NEXT:    [[TMP4361:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4362:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5760:%.*]] = icmp eq i64 [[TMP4361]], [[TMP4362]]
+// SIMD-ONLY0-NEXT:    [[CONV5761:%.*]] = zext i1 [[CMP5760]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5762:%.*]] = sext i32 [[CONV5761]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5762]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4363:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5763:%.*]] = icmp ne i64 [[TMP4363]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5763]], label [[IF_THEN5764:%.*]], label [[IF_ELSE5765:%.*]]
+// SIMD-ONLY0:       if.then5764:
+// SIMD-ONLY0-NEXT:    [[TMP4364:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4364]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5766:%.*]]
+// SIMD-ONLY0:       if.else5765:
+// SIMD-ONLY0-NEXT:    [[TMP4365:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4365]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5766]]
+// SIMD-ONLY0:       if.end5766:
+// SIMD-ONLY0-NEXT:    [[TMP4366:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4366]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4367:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4368:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5767:%.*]] = icmp ugt i64 [[TMP4367]], [[TMP4368]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5767]], label [[IF_THEN5769:%.*]], label [[IF_END5770:%.*]]
+// SIMD-ONLY0:       if.then5769:
+// SIMD-ONLY0-NEXT:    [[TMP4369:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4369]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5770]]
+// SIMD-ONLY0:       if.end5770:
+// SIMD-ONLY0-NEXT:    [[TMP4370:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4370]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4371:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4372:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5771:%.*]] = icmp ugt i64 [[TMP4371]], [[TMP4372]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5771]], label [[IF_THEN5773:%.*]], label [[IF_END5774:%.*]]
+// SIMD-ONLY0:       if.then5773:
+// SIMD-ONLY0-NEXT:    [[TMP4373:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4373]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5774]]
+// SIMD-ONLY0:       if.end5774:
+// SIMD-ONLY0-NEXT:    [[TMP4374:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4374]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4375:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4376:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5775:%.*]] = icmp ult i64 [[TMP4375]], [[TMP4376]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5775]], label [[IF_THEN5777:%.*]], label [[IF_END5778:%.*]]
+// SIMD-ONLY0:       if.then5777:
+// SIMD-ONLY0-NEXT:    [[TMP4377:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4377]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5778]]
+// SIMD-ONLY0:       if.end5778:
+// SIMD-ONLY0-NEXT:    [[TMP4378:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4378]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4379:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4380:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5779:%.*]] = icmp ult i64 [[TMP4379]], [[TMP4380]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5779]], label [[IF_THEN5781:%.*]], label [[IF_END5782:%.*]]
+// SIMD-ONLY0:       if.then5781:
+// SIMD-ONLY0-NEXT:    [[TMP4381:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4381]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5782]]
+// SIMD-ONLY0:       if.end5782:
+// SIMD-ONLY0-NEXT:    [[TMP4382:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4382]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4383:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4384:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5783:%.*]] = icmp eq i64 [[TMP4383]], [[TMP4384]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5783]], label [[IF_THEN5785:%.*]], label [[IF_END5786:%.*]]
+// SIMD-ONLY0:       if.then5785:
+// SIMD-ONLY0-NEXT:    [[TMP4385:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4385]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5786]]
+// SIMD-ONLY0:       if.end5786:
+// SIMD-ONLY0-NEXT:    [[TMP4386:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4386]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4387:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4388:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5787:%.*]] = icmp eq i64 [[TMP4387]], [[TMP4388]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5787]], label [[IF_THEN5789:%.*]], label [[IF_END5790:%.*]]
+// SIMD-ONLY0:       if.then5789:
+// SIMD-ONLY0-NEXT:    [[TMP4389:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4389]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5790]]
+// SIMD-ONLY0:       if.end5790:
+// SIMD-ONLY0-NEXT:    [[TMP4390:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4391:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5791:%.*]] = icmp ugt i64 [[TMP4390]], [[TMP4391]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5791]], label [[IF_THEN5793:%.*]], label [[IF_END5794:%.*]]
+// SIMD-ONLY0:       if.then5793:
+// SIMD-ONLY0-NEXT:    [[TMP4392:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4392]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5794]]
+// SIMD-ONLY0:       if.end5794:
+// SIMD-ONLY0-NEXT:    [[TMP4393:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4393]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4394:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4395:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5795:%.*]] = icmp ugt i64 [[TMP4394]], [[TMP4395]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5795]], label [[IF_THEN5797:%.*]], label [[IF_END5798:%.*]]
+// SIMD-ONLY0:       if.then5797:
+// SIMD-ONLY0-NEXT:    [[TMP4396:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4396]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5798]]
+// SIMD-ONLY0:       if.end5798:
+// SIMD-ONLY0-NEXT:    [[TMP4397:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4397]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4398:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4399:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5799:%.*]] = icmp ult i64 [[TMP4398]], [[TMP4399]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5799]], label [[IF_THEN5801:%.*]], label [[IF_END5802:%.*]]
+// SIMD-ONLY0:       if.then5801:
+// SIMD-ONLY0-NEXT:    [[TMP4400:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4400]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5802]]
+// SIMD-ONLY0:       if.end5802:
+// SIMD-ONLY0-NEXT:    [[TMP4401:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4401]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4402:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4403:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5803:%.*]] = icmp ult i64 [[TMP4402]], [[TMP4403]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5803]], label [[IF_THEN5805:%.*]], label [[IF_END5806:%.*]]
+// SIMD-ONLY0:       if.then5805:
+// SIMD-ONLY0-NEXT:    [[TMP4404:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4404]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5806]]
+// SIMD-ONLY0:       if.end5806:
+// SIMD-ONLY0-NEXT:    [[TMP4405:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4405]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4406:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4407:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5807:%.*]] = icmp eq i64 [[TMP4406]], [[TMP4407]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5807]], label [[IF_THEN5809:%.*]], label [[IF_END5810:%.*]]
+// SIMD-ONLY0:       if.then5809:
+// SIMD-ONLY0-NEXT:    [[TMP4408:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4408]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5810]]
+// SIMD-ONLY0:       if.end5810:
+// SIMD-ONLY0-NEXT:    [[TMP4409:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4409]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4410:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4411:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5811:%.*]] = icmp eq i64 [[TMP4410]], [[TMP4411]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5811]], label [[IF_THEN5813:%.*]], label [[IF_END5814:%.*]]
+// SIMD-ONLY0:       if.then5813:
+// SIMD-ONLY0-NEXT:    [[TMP4412:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4412]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5814]]
+// SIMD-ONLY0:       if.end5814:
+// SIMD-ONLY0-NEXT:    [[TMP4413:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4413]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4414:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4415:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5815:%.*]] = icmp eq i64 [[TMP4414]], [[TMP4415]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5815]], label [[IF_THEN5817:%.*]], label [[IF_ELSE5818:%.*]]
+// SIMD-ONLY0:       if.then5817:
+// SIMD-ONLY0-NEXT:    [[TMP4416:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4416]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5819:%.*]]
+// SIMD-ONLY0:       if.else5818:
+// SIMD-ONLY0-NEXT:    [[TMP4417:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4417]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5819]]
+// SIMD-ONLY0:       if.end5819:
+// SIMD-ONLY0-NEXT:    [[TMP4418:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4419:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5820:%.*]] = icmp eq i64 [[TMP4418]], [[TMP4419]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5820]], label [[IF_THEN5822:%.*]], label [[IF_ELSE5823:%.*]]
+// SIMD-ONLY0:       if.then5822:
+// SIMD-ONLY0-NEXT:    [[TMP4420:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4420]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5824:%.*]]
+// SIMD-ONLY0:       if.else5823:
+// SIMD-ONLY0-NEXT:    [[TMP4421:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4421]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5824]]
+// SIMD-ONLY0:       if.end5824:
+// SIMD-ONLY0-NEXT:    [[TMP4422:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4423:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5825:%.*]] = icmp eq i64 [[TMP4422]], [[TMP4423]]
+// SIMD-ONLY0-NEXT:    [[CONV5826:%.*]] = zext i1 [[CMP5825]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5827:%.*]] = sext i32 [[CONV5826]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5827]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4424:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5828:%.*]] = icmp ne i64 [[TMP4424]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5828]], label [[IF_THEN5829:%.*]], label [[IF_END5830:%.*]]
+// SIMD-ONLY0:       if.then5829:
+// SIMD-ONLY0-NEXT:    [[TMP4425:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4425]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5830]]
+// SIMD-ONLY0:       if.end5830:
+// SIMD-ONLY0-NEXT:    [[TMP4426:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4427:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5831:%.*]] = icmp eq i64 [[TMP4426]], [[TMP4427]]
+// SIMD-ONLY0-NEXT:    [[CONV5832:%.*]] = zext i1 [[CMP5831]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5833:%.*]] = sext i32 [[CONV5832]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5833]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4428:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5834:%.*]] = icmp ne i64 [[TMP4428]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5834]], label [[IF_THEN5835:%.*]], label [[IF_END5836:%.*]]
+// SIMD-ONLY0:       if.then5835:
+// SIMD-ONLY0-NEXT:    [[TMP4429:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4429]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5836]]
+// SIMD-ONLY0:       if.end5836:
+// SIMD-ONLY0-NEXT:    [[TMP4430:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4431:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5837:%.*]] = icmp eq i64 [[TMP4430]], [[TMP4431]]
+// SIMD-ONLY0-NEXT:    [[CONV5838:%.*]] = zext i1 [[CMP5837]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5839:%.*]] = sext i32 [[CONV5838]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5839]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4432:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5840:%.*]] = icmp ne i64 [[TMP4432]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5840]], label [[IF_THEN5841:%.*]], label [[IF_ELSE5842:%.*]]
+// SIMD-ONLY0:       if.then5841:
+// SIMD-ONLY0-NEXT:    [[TMP4433:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4433]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5843:%.*]]
+// SIMD-ONLY0:       if.else5842:
+// SIMD-ONLY0-NEXT:    [[TMP4434:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4434]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5843]]
+// SIMD-ONLY0:       if.end5843:
+// SIMD-ONLY0-NEXT:    [[TMP4435:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4436:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP5844:%.*]] = icmp eq i64 [[TMP4435]], [[TMP4436]]
+// SIMD-ONLY0-NEXT:    [[CONV5845:%.*]] = zext i1 [[CMP5844]] to i32
+// SIMD-ONLY0-NEXT:    [[CONV5846:%.*]] = sext i32 [[CONV5845]] to i64
+// SIMD-ONLY0-NEXT:    store i64 [[CONV5846]], ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4437:%.*]] = load i64, ptr [[ULLR]], align 8
+// SIMD-ONLY0-NEXT:    [[TOBOOL5847:%.*]] = icmp ne i64 [[TMP4437]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5847]], label [[IF_THEN5848:%.*]], label [[IF_ELSE5849:%.*]]
+// SIMD-ONLY0:       if.then5848:
+// SIMD-ONLY0-NEXT:    [[TMP4438:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4438]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5850:%.*]]
+// SIMD-ONLY0:       if.else5849:
+// SIMD-ONLY0-NEXT:    [[TMP4439:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP4439]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END5850]]
+// SIMD-ONLY0:       if.end5850:
+// SIMD-ONLY0-NEXT:    [[TMP4440:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4440]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4441:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4442:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5851:%.*]] = fcmp ogt float [[TMP4441]], [[TMP4442]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5851]], label [[IF_THEN5853:%.*]], label [[IF_END5854:%.*]]
+// SIMD-ONLY0:       if.then5853:
+// SIMD-ONLY0-NEXT:    [[TMP4443:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4443]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5854]]
+// SIMD-ONLY0:       if.end5854:
+// SIMD-ONLY0-NEXT:    [[TMP4444:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4444]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4445:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4446:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5855:%.*]] = fcmp ogt float [[TMP4445]], [[TMP4446]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5855]], label [[IF_THEN5857:%.*]], label [[IF_END5858:%.*]]
+// SIMD-ONLY0:       if.then5857:
+// SIMD-ONLY0-NEXT:    [[TMP4447:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4447]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5858]]
+// SIMD-ONLY0:       if.end5858:
+// SIMD-ONLY0-NEXT:    [[TMP4448:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4448]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4449:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4450:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5859:%.*]] = fcmp olt float [[TMP4449]], [[TMP4450]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5859]], label [[IF_THEN5861:%.*]], label [[IF_END5862:%.*]]
+// SIMD-ONLY0:       if.then5861:
+// SIMD-ONLY0-NEXT:    [[TMP4451:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4451]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5862]]
+// SIMD-ONLY0:       if.end5862:
+// SIMD-ONLY0-NEXT:    [[TMP4452:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4452]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4453:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4454:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5863:%.*]] = fcmp olt float [[TMP4453]], [[TMP4454]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5863]], label [[IF_THEN5865:%.*]], label [[IF_END5866:%.*]]
+// SIMD-ONLY0:       if.then5865:
+// SIMD-ONLY0-NEXT:    [[TMP4455:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4455]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5866]]
+// SIMD-ONLY0:       if.end5866:
+// SIMD-ONLY0-NEXT:    [[TMP4456:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4456]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4457:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4458:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5867:%.*]] = fcmp oeq float [[TMP4457]], [[TMP4458]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5867]], label [[IF_THEN5869:%.*]], label [[IF_END5870:%.*]]
+// SIMD-ONLY0:       if.then5869:
+// SIMD-ONLY0-NEXT:    [[TMP4459:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4459]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5870]]
+// SIMD-ONLY0:       if.end5870:
+// SIMD-ONLY0-NEXT:    [[TMP4460:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4460]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4461:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4462:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5871:%.*]] = fcmp oeq float [[TMP4461]], [[TMP4462]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5871]], label [[IF_THEN5873:%.*]], label [[IF_END5874:%.*]]
+// SIMD-ONLY0:       if.then5873:
+// SIMD-ONLY0-NEXT:    [[TMP4463:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4463]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5874]]
+// SIMD-ONLY0:       if.end5874:
+// SIMD-ONLY0-NEXT:    [[TMP4464:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4465:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5875:%.*]] = fcmp ogt float [[TMP4464]], [[TMP4465]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5875]], label [[IF_THEN5877:%.*]], label [[IF_END5878:%.*]]
+// SIMD-ONLY0:       if.then5877:
+// SIMD-ONLY0-NEXT:    [[TMP4466:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4466]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5878]]
+// SIMD-ONLY0:       if.end5878:
+// SIMD-ONLY0-NEXT:    [[TMP4467:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4467]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4468:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4469:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5879:%.*]] = fcmp ogt float [[TMP4468]], [[TMP4469]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5879]], label [[IF_THEN5881:%.*]], label [[IF_END5882:%.*]]
+// SIMD-ONLY0:       if.then5881:
+// SIMD-ONLY0-NEXT:    [[TMP4470:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4470]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5882]]
+// SIMD-ONLY0:       if.end5882:
+// SIMD-ONLY0-NEXT:    [[TMP4471:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4471]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4472:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4473:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5883:%.*]] = fcmp olt float [[TMP4472]], [[TMP4473]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5883]], label [[IF_THEN5885:%.*]], label [[IF_END5886:%.*]]
+// SIMD-ONLY0:       if.then5885:
+// SIMD-ONLY0-NEXT:    [[TMP4474:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4474]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5886]]
+// SIMD-ONLY0:       if.end5886:
+// SIMD-ONLY0-NEXT:    [[TMP4475:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4475]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4476:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4477:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5887:%.*]] = fcmp olt float [[TMP4476]], [[TMP4477]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5887]], label [[IF_THEN5889:%.*]], label [[IF_END5890:%.*]]
+// SIMD-ONLY0:       if.then5889:
+// SIMD-ONLY0-NEXT:    [[TMP4478:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4478]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5890]]
+// SIMD-ONLY0:       if.end5890:
+// SIMD-ONLY0-NEXT:    [[TMP4479:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4479]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4480:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4481:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5891:%.*]] = fcmp oeq float [[TMP4480]], [[TMP4481]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5891]], label [[IF_THEN5893:%.*]], label [[IF_END5894:%.*]]
+// SIMD-ONLY0:       if.then5893:
+// SIMD-ONLY0-NEXT:    [[TMP4482:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4482]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5894]]
+// SIMD-ONLY0:       if.end5894:
+// SIMD-ONLY0-NEXT:    [[TMP4483:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4483]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4484:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4485:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5895:%.*]] = fcmp oeq float [[TMP4484]], [[TMP4485]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5895]], label [[IF_THEN5897:%.*]], label [[IF_END5898:%.*]]
+// SIMD-ONLY0:       if.then5897:
+// SIMD-ONLY0-NEXT:    [[TMP4486:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4486]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5898]]
+// SIMD-ONLY0:       if.end5898:
+// SIMD-ONLY0-NEXT:    [[TMP4487:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4487]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4488:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4489:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5899:%.*]] = fcmp oeq float [[TMP4488]], [[TMP4489]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5899]], label [[IF_THEN5901:%.*]], label [[IF_ELSE5902:%.*]]
+// SIMD-ONLY0:       if.then5901:
+// SIMD-ONLY0-NEXT:    [[TMP4490:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4490]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5903:%.*]]
+// SIMD-ONLY0:       if.else5902:
+// SIMD-ONLY0-NEXT:    [[TMP4491:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4491]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5903]]
+// SIMD-ONLY0:       if.end5903:
+// SIMD-ONLY0-NEXT:    [[TMP4492:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4493:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5904:%.*]] = fcmp oeq float [[TMP4492]], [[TMP4493]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5904]], label [[IF_THEN5906:%.*]], label [[IF_ELSE5907:%.*]]
+// SIMD-ONLY0:       if.then5906:
+// SIMD-ONLY0-NEXT:    [[TMP4494:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4494]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5908:%.*]]
+// SIMD-ONLY0:       if.else5907:
+// SIMD-ONLY0-NEXT:    [[TMP4495:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4495]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5908]]
+// SIMD-ONLY0:       if.end5908:
+// SIMD-ONLY0-NEXT:    [[TMP4496:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4497:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5909:%.*]] = fcmp oeq float [[TMP4496]], [[TMP4497]]
+// SIMD-ONLY0-NEXT:    [[CONV5910:%.*]] = zext i1 [[CMP5909]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV5910]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4498:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL5911:%.*]] = icmp ne i32 [[TMP4498]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5911]], label [[IF_THEN5912:%.*]], label [[IF_END5913:%.*]]
+// SIMD-ONLY0:       if.then5912:
+// SIMD-ONLY0-NEXT:    [[TMP4499:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4499]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5913]]
+// SIMD-ONLY0:       if.end5913:
+// SIMD-ONLY0-NEXT:    [[TMP4500:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4501:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5914:%.*]] = fcmp oeq float [[TMP4500]], [[TMP4501]]
+// SIMD-ONLY0-NEXT:    [[CONV5915:%.*]] = zext i1 [[CMP5914]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV5915]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4502:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL5916:%.*]] = icmp ne i32 [[TMP4502]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5916]], label [[IF_THEN5917:%.*]], label [[IF_END5918:%.*]]
+// SIMD-ONLY0:       if.then5917:
+// SIMD-ONLY0-NEXT:    [[TMP4503:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4503]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5918]]
+// SIMD-ONLY0:       if.end5918:
+// SIMD-ONLY0-NEXT:    [[TMP4504:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4505:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5919:%.*]] = fcmp oeq float [[TMP4504]], [[TMP4505]]
+// SIMD-ONLY0-NEXT:    [[CONV5920:%.*]] = zext i1 [[CMP5919]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV5920]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4506:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL5921:%.*]] = icmp ne i32 [[TMP4506]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5921]], label [[IF_THEN5922:%.*]], label [[IF_ELSE5923:%.*]]
+// SIMD-ONLY0:       if.then5922:
+// SIMD-ONLY0-NEXT:    [[TMP4507:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4507]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5924:%.*]]
+// SIMD-ONLY0:       if.else5923:
+// SIMD-ONLY0-NEXT:    [[TMP4508:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4508]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5924]]
+// SIMD-ONLY0:       if.end5924:
+// SIMD-ONLY0-NEXT:    [[TMP4509:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4510:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5925:%.*]] = fcmp oeq float [[TMP4509]], [[TMP4510]]
+// SIMD-ONLY0-NEXT:    [[CONV5926:%.*]] = zext i1 [[CMP5925]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV5926]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4511:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL5927:%.*]] = icmp ne i32 [[TMP4511]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5927]], label [[IF_THEN5928:%.*]], label [[IF_ELSE5929:%.*]]
+// SIMD-ONLY0:       if.then5928:
+// SIMD-ONLY0-NEXT:    [[TMP4512:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4512]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5930:%.*]]
+// SIMD-ONLY0:       if.else5929:
+// SIMD-ONLY0-NEXT:    [[TMP4513:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4513]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5930]]
+// SIMD-ONLY0:       if.end5930:
+// SIMD-ONLY0-NEXT:    [[TMP4514:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4514]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4515:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4516:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5931:%.*]] = fcmp ogt float [[TMP4515]], [[TMP4516]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5931]], label [[IF_THEN5933:%.*]], label [[IF_END5934:%.*]]
+// SIMD-ONLY0:       if.then5933:
+// SIMD-ONLY0-NEXT:    [[TMP4517:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4517]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5934]]
+// SIMD-ONLY0:       if.end5934:
+// SIMD-ONLY0-NEXT:    [[TMP4518:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4518]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4519:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4520:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5935:%.*]] = fcmp ogt float [[TMP4519]], [[TMP4520]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5935]], label [[IF_THEN5937:%.*]], label [[IF_END5938:%.*]]
+// SIMD-ONLY0:       if.then5937:
+// SIMD-ONLY0-NEXT:    [[TMP4521:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4521]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5938]]
+// SIMD-ONLY0:       if.end5938:
+// SIMD-ONLY0-NEXT:    [[TMP4522:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4522]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4523:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4524:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5939:%.*]] = fcmp olt float [[TMP4523]], [[TMP4524]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5939]], label [[IF_THEN5941:%.*]], label [[IF_END5942:%.*]]
+// SIMD-ONLY0:       if.then5941:
+// SIMD-ONLY0-NEXT:    [[TMP4525:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4525]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5942]]
+// SIMD-ONLY0:       if.end5942:
+// SIMD-ONLY0-NEXT:    [[TMP4526:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4526]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4527:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4528:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5943:%.*]] = fcmp olt float [[TMP4527]], [[TMP4528]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5943]], label [[IF_THEN5945:%.*]], label [[IF_END5946:%.*]]
+// SIMD-ONLY0:       if.then5945:
+// SIMD-ONLY0-NEXT:    [[TMP4529:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4529]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5946]]
+// SIMD-ONLY0:       if.end5946:
+// SIMD-ONLY0-NEXT:    [[TMP4530:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4530]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4531:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4532:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5947:%.*]] = fcmp oeq float [[TMP4531]], [[TMP4532]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5947]], label [[IF_THEN5949:%.*]], label [[IF_END5950:%.*]]
+// SIMD-ONLY0:       if.then5949:
+// SIMD-ONLY0-NEXT:    [[TMP4533:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4533]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5950]]
+// SIMD-ONLY0:       if.end5950:
+// SIMD-ONLY0-NEXT:    [[TMP4534:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4534]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4535:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4536:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5951:%.*]] = fcmp oeq float [[TMP4535]], [[TMP4536]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5951]], label [[IF_THEN5953:%.*]], label [[IF_END5954:%.*]]
+// SIMD-ONLY0:       if.then5953:
+// SIMD-ONLY0-NEXT:    [[TMP4537:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4537]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5954]]
+// SIMD-ONLY0:       if.end5954:
+// SIMD-ONLY0-NEXT:    [[TMP4538:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4539:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5955:%.*]] = fcmp ogt float [[TMP4538]], [[TMP4539]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5955]], label [[IF_THEN5957:%.*]], label [[IF_END5958:%.*]]
+// SIMD-ONLY0:       if.then5957:
+// SIMD-ONLY0-NEXT:    [[TMP4540:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4540]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5958]]
+// SIMD-ONLY0:       if.end5958:
+// SIMD-ONLY0-NEXT:    [[TMP4541:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4541]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4542:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4543:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5959:%.*]] = fcmp ogt float [[TMP4542]], [[TMP4543]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5959]], label [[IF_THEN5961:%.*]], label [[IF_END5962:%.*]]
+// SIMD-ONLY0:       if.then5961:
+// SIMD-ONLY0-NEXT:    [[TMP4544:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4544]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5962]]
+// SIMD-ONLY0:       if.end5962:
+// SIMD-ONLY0-NEXT:    [[TMP4545:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4545]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4546:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4547:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5963:%.*]] = fcmp olt float [[TMP4546]], [[TMP4547]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5963]], label [[IF_THEN5965:%.*]], label [[IF_END5966:%.*]]
+// SIMD-ONLY0:       if.then5965:
+// SIMD-ONLY0-NEXT:    [[TMP4548:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4548]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5966]]
+// SIMD-ONLY0:       if.end5966:
+// SIMD-ONLY0-NEXT:    [[TMP4549:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4549]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4550:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4551:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5967:%.*]] = fcmp olt float [[TMP4550]], [[TMP4551]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5967]], label [[IF_THEN5969:%.*]], label [[IF_END5970:%.*]]
+// SIMD-ONLY0:       if.then5969:
+// SIMD-ONLY0-NEXT:    [[TMP4552:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4552]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5970]]
+// SIMD-ONLY0:       if.end5970:
+// SIMD-ONLY0-NEXT:    [[TMP4553:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4553]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4554:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4555:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5971:%.*]] = fcmp oeq float [[TMP4554]], [[TMP4555]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5971]], label [[IF_THEN5973:%.*]], label [[IF_END5974:%.*]]
+// SIMD-ONLY0:       if.then5973:
+// SIMD-ONLY0-NEXT:    [[TMP4556:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4556]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5974]]
+// SIMD-ONLY0:       if.end5974:
+// SIMD-ONLY0-NEXT:    [[TMP4557:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4557]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4558:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4559:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5975:%.*]] = fcmp oeq float [[TMP4558]], [[TMP4559]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5975]], label [[IF_THEN5977:%.*]], label [[IF_END5978:%.*]]
+// SIMD-ONLY0:       if.then5977:
+// SIMD-ONLY0-NEXT:    [[TMP4560:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4560]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5978]]
+// SIMD-ONLY0:       if.end5978:
+// SIMD-ONLY0-NEXT:    [[TMP4561:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4561]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4562:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4563:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5979:%.*]] = fcmp oeq float [[TMP4562]], [[TMP4563]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5979]], label [[IF_THEN5981:%.*]], label [[IF_ELSE5982:%.*]]
+// SIMD-ONLY0:       if.then5981:
+// SIMD-ONLY0-NEXT:    [[TMP4564:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4564]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5983:%.*]]
+// SIMD-ONLY0:       if.else5982:
+// SIMD-ONLY0-NEXT:    [[TMP4565:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4565]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5983]]
+// SIMD-ONLY0:       if.end5983:
+// SIMD-ONLY0-NEXT:    [[TMP4566:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4567:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5984:%.*]] = fcmp oeq float [[TMP4566]], [[TMP4567]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5984]], label [[IF_THEN5986:%.*]], label [[IF_ELSE5987:%.*]]
+// SIMD-ONLY0:       if.then5986:
+// SIMD-ONLY0-NEXT:    [[TMP4568:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4568]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5988:%.*]]
+// SIMD-ONLY0:       if.else5987:
+// SIMD-ONLY0-NEXT:    [[TMP4569:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4569]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5988]]
+// SIMD-ONLY0:       if.end5988:
+// SIMD-ONLY0-NEXT:    [[TMP4570:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4571:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5989:%.*]] = fcmp oeq float [[TMP4570]], [[TMP4571]]
+// SIMD-ONLY0-NEXT:    [[CONV5990:%.*]] = zext i1 [[CMP5989]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV5990]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4572:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL5991:%.*]] = icmp ne i32 [[TMP4572]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5991]], label [[IF_THEN5992:%.*]], label [[IF_END5993:%.*]]
+// SIMD-ONLY0:       if.then5992:
+// SIMD-ONLY0-NEXT:    [[TMP4573:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4573]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5993]]
+// SIMD-ONLY0:       if.end5993:
+// SIMD-ONLY0-NEXT:    [[TMP4574:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4575:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5994:%.*]] = fcmp oeq float [[TMP4574]], [[TMP4575]]
+// SIMD-ONLY0-NEXT:    [[CONV5995:%.*]] = zext i1 [[CMP5994]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV5995]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4576:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL5996:%.*]] = icmp ne i32 [[TMP4576]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL5996]], label [[IF_THEN5997:%.*]], label [[IF_END5998:%.*]]
+// SIMD-ONLY0:       if.then5997:
+// SIMD-ONLY0-NEXT:    [[TMP4577:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4577]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END5998]]
+// SIMD-ONLY0:       if.end5998:
+// SIMD-ONLY0-NEXT:    [[TMP4578:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4579:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5999:%.*]] = fcmp oeq float [[TMP4578]], [[TMP4579]]
+// SIMD-ONLY0-NEXT:    [[CONV6000:%.*]] = zext i1 [[CMP5999]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6000]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4580:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6001:%.*]] = icmp ne i32 [[TMP4580]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6001]], label [[IF_THEN6002:%.*]], label [[IF_ELSE6003:%.*]]
+// SIMD-ONLY0:       if.then6002:
+// SIMD-ONLY0-NEXT:    [[TMP4581:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4581]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6004:%.*]]
+// SIMD-ONLY0:       if.else6003:
+// SIMD-ONLY0-NEXT:    [[TMP4582:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4582]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6004]]
+// SIMD-ONLY0:       if.end6004:
+// SIMD-ONLY0-NEXT:    [[TMP4583:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4584:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6005:%.*]] = fcmp oeq float [[TMP4583]], [[TMP4584]]
+// SIMD-ONLY0-NEXT:    [[CONV6006:%.*]] = zext i1 [[CMP6005]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6006]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4585:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6007:%.*]] = icmp ne i32 [[TMP4585]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6007]], label [[IF_THEN6008:%.*]], label [[IF_ELSE6009:%.*]]
+// SIMD-ONLY0:       if.then6008:
+// SIMD-ONLY0-NEXT:    [[TMP4586:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4586]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6010:%.*]]
+// SIMD-ONLY0:       if.else6009:
+// SIMD-ONLY0-NEXT:    [[TMP4587:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4587]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6010]]
+// SIMD-ONLY0:       if.end6010:
+// SIMD-ONLY0-NEXT:    [[TMP4588:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4588]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4589:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4590:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6011:%.*]] = fcmp ogt float [[TMP4589]], [[TMP4590]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6011]], label [[IF_THEN6013:%.*]], label [[IF_END6014:%.*]]
+// SIMD-ONLY0:       if.then6013:
+// SIMD-ONLY0-NEXT:    [[TMP4591:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4591]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6014]]
+// SIMD-ONLY0:       if.end6014:
+// SIMD-ONLY0-NEXT:    [[TMP4592:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4592]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4593:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4594:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6015:%.*]] = fcmp ogt float [[TMP4593]], [[TMP4594]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6015]], label [[IF_THEN6017:%.*]], label [[IF_END6018:%.*]]
+// SIMD-ONLY0:       if.then6017:
+// SIMD-ONLY0-NEXT:    [[TMP4595:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4595]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6018]]
+// SIMD-ONLY0:       if.end6018:
+// SIMD-ONLY0-NEXT:    [[TMP4596:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4596]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4597:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4598:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6019:%.*]] = fcmp olt float [[TMP4597]], [[TMP4598]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6019]], label [[IF_THEN6021:%.*]], label [[IF_END6022:%.*]]
+// SIMD-ONLY0:       if.then6021:
+// SIMD-ONLY0-NEXT:    [[TMP4599:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4599]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6022]]
+// SIMD-ONLY0:       if.end6022:
+// SIMD-ONLY0-NEXT:    [[TMP4600:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4600]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4601:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4602:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6023:%.*]] = fcmp olt float [[TMP4601]], [[TMP4602]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6023]], label [[IF_THEN6025:%.*]], label [[IF_END6026:%.*]]
+// SIMD-ONLY0:       if.then6025:
+// SIMD-ONLY0-NEXT:    [[TMP4603:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4603]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6026]]
+// SIMD-ONLY0:       if.end6026:
+// SIMD-ONLY0-NEXT:    [[TMP4604:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4604]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4605:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4606:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6027:%.*]] = fcmp oeq float [[TMP4605]], [[TMP4606]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6027]], label [[IF_THEN6029:%.*]], label [[IF_END6030:%.*]]
+// SIMD-ONLY0:       if.then6029:
+// SIMD-ONLY0-NEXT:    [[TMP4607:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4607]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6030]]
+// SIMD-ONLY0:       if.end6030:
+// SIMD-ONLY0-NEXT:    [[TMP4608:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4608]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4609:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4610:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6031:%.*]] = fcmp oeq float [[TMP4609]], [[TMP4610]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6031]], label [[IF_THEN6033:%.*]], label [[IF_END6034:%.*]]
+// SIMD-ONLY0:       if.then6033:
+// SIMD-ONLY0-NEXT:    [[TMP4611:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4611]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6034]]
+// SIMD-ONLY0:       if.end6034:
+// SIMD-ONLY0-NEXT:    [[TMP4612:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4613:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6035:%.*]] = fcmp ogt float [[TMP4612]], [[TMP4613]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6035]], label [[IF_THEN6037:%.*]], label [[IF_END6038:%.*]]
+// SIMD-ONLY0:       if.then6037:
+// SIMD-ONLY0-NEXT:    [[TMP4614:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4614]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6038]]
+// SIMD-ONLY0:       if.end6038:
+// SIMD-ONLY0-NEXT:    [[TMP4615:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4615]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4616:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4617:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6039:%.*]] = fcmp ogt float [[TMP4616]], [[TMP4617]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6039]], label [[IF_THEN6041:%.*]], label [[IF_END6042:%.*]]
+// SIMD-ONLY0:       if.then6041:
+// SIMD-ONLY0-NEXT:    [[TMP4618:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4618]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6042]]
+// SIMD-ONLY0:       if.end6042:
+// SIMD-ONLY0-NEXT:    [[TMP4619:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4619]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4620:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4621:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6043:%.*]] = fcmp olt float [[TMP4620]], [[TMP4621]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6043]], label [[IF_THEN6045:%.*]], label [[IF_END6046:%.*]]
+// SIMD-ONLY0:       if.then6045:
+// SIMD-ONLY0-NEXT:    [[TMP4622:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4622]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6046]]
+// SIMD-ONLY0:       if.end6046:
+// SIMD-ONLY0-NEXT:    [[TMP4623:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4623]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4624:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4625:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6047:%.*]] = fcmp olt float [[TMP4624]], [[TMP4625]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6047]], label [[IF_THEN6049:%.*]], label [[IF_END6050:%.*]]
+// SIMD-ONLY0:       if.then6049:
+// SIMD-ONLY0-NEXT:    [[TMP4626:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4626]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6050]]
+// SIMD-ONLY0:       if.end6050:
+// SIMD-ONLY0-NEXT:    [[TMP4627:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4627]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4628:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4629:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6051:%.*]] = fcmp oeq float [[TMP4628]], [[TMP4629]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6051]], label [[IF_THEN6053:%.*]], label [[IF_END6054:%.*]]
+// SIMD-ONLY0:       if.then6053:
+// SIMD-ONLY0-NEXT:    [[TMP4630:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4630]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6054]]
+// SIMD-ONLY0:       if.end6054:
+// SIMD-ONLY0-NEXT:    [[TMP4631:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4631]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4632:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4633:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6055:%.*]] = fcmp oeq float [[TMP4632]], [[TMP4633]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6055]], label [[IF_THEN6057:%.*]], label [[IF_END6058:%.*]]
+// SIMD-ONLY0:       if.then6057:
+// SIMD-ONLY0-NEXT:    [[TMP4634:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4634]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6058]]
+// SIMD-ONLY0:       if.end6058:
+// SIMD-ONLY0-NEXT:    [[TMP4635:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4635]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4636:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4637:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6059:%.*]] = fcmp oeq float [[TMP4636]], [[TMP4637]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6059]], label [[IF_THEN6061:%.*]], label [[IF_ELSE6062:%.*]]
+// SIMD-ONLY0:       if.then6061:
+// SIMD-ONLY0-NEXT:    [[TMP4638:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4638]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6063:%.*]]
+// SIMD-ONLY0:       if.else6062:
+// SIMD-ONLY0-NEXT:    [[TMP4639:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4639]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6063]]
+// SIMD-ONLY0:       if.end6063:
+// SIMD-ONLY0-NEXT:    [[TMP4640:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4641:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6064:%.*]] = fcmp oeq float [[TMP4640]], [[TMP4641]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6064]], label [[IF_THEN6066:%.*]], label [[IF_ELSE6067:%.*]]
+// SIMD-ONLY0:       if.then6066:
+// SIMD-ONLY0-NEXT:    [[TMP4642:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4642]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6068:%.*]]
+// SIMD-ONLY0:       if.else6067:
+// SIMD-ONLY0-NEXT:    [[TMP4643:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4643]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6068]]
+// SIMD-ONLY0:       if.end6068:
+// SIMD-ONLY0-NEXT:    [[TMP4644:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4645:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6069:%.*]] = fcmp oeq float [[TMP4644]], [[TMP4645]]
+// SIMD-ONLY0-NEXT:    [[CONV6070:%.*]] = zext i1 [[CMP6069]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6070]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4646:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6071:%.*]] = icmp ne i32 [[TMP4646]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6071]], label [[IF_THEN6072:%.*]], label [[IF_END6073:%.*]]
+// SIMD-ONLY0:       if.then6072:
+// SIMD-ONLY0-NEXT:    [[TMP4647:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4647]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6073]]
+// SIMD-ONLY0:       if.end6073:
+// SIMD-ONLY0-NEXT:    [[TMP4648:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4649:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6074:%.*]] = fcmp oeq float [[TMP4648]], [[TMP4649]]
+// SIMD-ONLY0-NEXT:    [[CONV6075:%.*]] = zext i1 [[CMP6074]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6075]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4650:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6076:%.*]] = icmp ne i32 [[TMP4650]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6076]], label [[IF_THEN6077:%.*]], label [[IF_END6078:%.*]]
+// SIMD-ONLY0:       if.then6077:
+// SIMD-ONLY0-NEXT:    [[TMP4651:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4651]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6078]]
+// SIMD-ONLY0:       if.end6078:
+// SIMD-ONLY0-NEXT:    [[TMP4652:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4653:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6079:%.*]] = fcmp oeq float [[TMP4652]], [[TMP4653]]
+// SIMD-ONLY0-NEXT:    [[CONV6080:%.*]] = zext i1 [[CMP6079]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6080]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4654:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6081:%.*]] = icmp ne i32 [[TMP4654]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6081]], label [[IF_THEN6082:%.*]], label [[IF_ELSE6083:%.*]]
+// SIMD-ONLY0:       if.then6082:
+// SIMD-ONLY0-NEXT:    [[TMP4655:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4655]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6084:%.*]]
+// SIMD-ONLY0:       if.else6083:
+// SIMD-ONLY0-NEXT:    [[TMP4656:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4656]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6084]]
+// SIMD-ONLY0:       if.end6084:
+// SIMD-ONLY0-NEXT:    [[TMP4657:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4658:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6085:%.*]] = fcmp oeq float [[TMP4657]], [[TMP4658]]
+// SIMD-ONLY0-NEXT:    [[CONV6086:%.*]] = zext i1 [[CMP6085]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6086]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4659:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6087:%.*]] = icmp ne i32 [[TMP4659]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6087]], label [[IF_THEN6088:%.*]], label [[IF_ELSE6089:%.*]]
+// SIMD-ONLY0:       if.then6088:
+// SIMD-ONLY0-NEXT:    [[TMP4660:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4660]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6090:%.*]]
+// SIMD-ONLY0:       if.else6089:
+// SIMD-ONLY0-NEXT:    [[TMP4661:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4661]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6090]]
+// SIMD-ONLY0:       if.end6090:
+// SIMD-ONLY0-NEXT:    [[TMP4662:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4662]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4663:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4664:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6091:%.*]] = fcmp ogt float [[TMP4663]], [[TMP4664]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6091]], label [[IF_THEN6093:%.*]], label [[IF_END6094:%.*]]
+// SIMD-ONLY0:       if.then6093:
+// SIMD-ONLY0-NEXT:    [[TMP4665:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4665]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6094]]
+// SIMD-ONLY0:       if.end6094:
+// SIMD-ONLY0-NEXT:    [[TMP4666:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4666]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4667:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4668:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6095:%.*]] = fcmp ogt float [[TMP4667]], [[TMP4668]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6095]], label [[IF_THEN6097:%.*]], label [[IF_END6098:%.*]]
+// SIMD-ONLY0:       if.then6097:
+// SIMD-ONLY0-NEXT:    [[TMP4669:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4669]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6098]]
+// SIMD-ONLY0:       if.end6098:
+// SIMD-ONLY0-NEXT:    [[TMP4670:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4670]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4671:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4672:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6099:%.*]] = fcmp olt float [[TMP4671]], [[TMP4672]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6099]], label [[IF_THEN6101:%.*]], label [[IF_END6102:%.*]]
+// SIMD-ONLY0:       if.then6101:
+// SIMD-ONLY0-NEXT:    [[TMP4673:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4673]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6102]]
+// SIMD-ONLY0:       if.end6102:
+// SIMD-ONLY0-NEXT:    [[TMP4674:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4674]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4675:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4676:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6103:%.*]] = fcmp olt float [[TMP4675]], [[TMP4676]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6103]], label [[IF_THEN6105:%.*]], label [[IF_END6106:%.*]]
+// SIMD-ONLY0:       if.then6105:
+// SIMD-ONLY0-NEXT:    [[TMP4677:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4677]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6106]]
+// SIMD-ONLY0:       if.end6106:
+// SIMD-ONLY0-NEXT:    [[TMP4678:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4678]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4679:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4680:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6107:%.*]] = fcmp oeq float [[TMP4679]], [[TMP4680]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6107]], label [[IF_THEN6109:%.*]], label [[IF_END6110:%.*]]
+// SIMD-ONLY0:       if.then6109:
+// SIMD-ONLY0-NEXT:    [[TMP4681:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4681]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6110]]
+// SIMD-ONLY0:       if.end6110:
+// SIMD-ONLY0-NEXT:    [[TMP4682:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4682]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4683:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4684:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6111:%.*]] = fcmp oeq float [[TMP4683]], [[TMP4684]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6111]], label [[IF_THEN6113:%.*]], label [[IF_END6114:%.*]]
+// SIMD-ONLY0:       if.then6113:
+// SIMD-ONLY0-NEXT:    [[TMP4685:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4685]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6114]]
+// SIMD-ONLY0:       if.end6114:
+// SIMD-ONLY0-NEXT:    [[TMP4686:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4687:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6115:%.*]] = fcmp ogt float [[TMP4686]], [[TMP4687]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6115]], label [[IF_THEN6117:%.*]], label [[IF_END6118:%.*]]
+// SIMD-ONLY0:       if.then6117:
+// SIMD-ONLY0-NEXT:    [[TMP4688:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4688]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6118]]
+// SIMD-ONLY0:       if.end6118:
+// SIMD-ONLY0-NEXT:    [[TMP4689:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4689]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4690:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4691:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6119:%.*]] = fcmp ogt float [[TMP4690]], [[TMP4691]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6119]], label [[IF_THEN6121:%.*]], label [[IF_END6122:%.*]]
+// SIMD-ONLY0:       if.then6121:
+// SIMD-ONLY0-NEXT:    [[TMP4692:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4692]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6122]]
+// SIMD-ONLY0:       if.end6122:
+// SIMD-ONLY0-NEXT:    [[TMP4693:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4693]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4694:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4695:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6123:%.*]] = fcmp olt float [[TMP4694]], [[TMP4695]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6123]], label [[IF_THEN6125:%.*]], label [[IF_END6126:%.*]]
+// SIMD-ONLY0:       if.then6125:
+// SIMD-ONLY0-NEXT:    [[TMP4696:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4696]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6126]]
+// SIMD-ONLY0:       if.end6126:
+// SIMD-ONLY0-NEXT:    [[TMP4697:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4697]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4698:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4699:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6127:%.*]] = fcmp olt float [[TMP4698]], [[TMP4699]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6127]], label [[IF_THEN6129:%.*]], label [[IF_END6130:%.*]]
+// SIMD-ONLY0:       if.then6129:
+// SIMD-ONLY0-NEXT:    [[TMP4700:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4700]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6130]]
+// SIMD-ONLY0:       if.end6130:
+// SIMD-ONLY0-NEXT:    [[TMP4701:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4701]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4702:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4703:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6131:%.*]] = fcmp oeq float [[TMP4702]], [[TMP4703]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6131]], label [[IF_THEN6133:%.*]], label [[IF_END6134:%.*]]
+// SIMD-ONLY0:       if.then6133:
+// SIMD-ONLY0-NEXT:    [[TMP4704:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4704]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6134]]
+// SIMD-ONLY0:       if.end6134:
+// SIMD-ONLY0-NEXT:    [[TMP4705:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4705]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4706:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4707:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6135:%.*]] = fcmp oeq float [[TMP4706]], [[TMP4707]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6135]], label [[IF_THEN6137:%.*]], label [[IF_END6138:%.*]]
+// SIMD-ONLY0:       if.then6137:
+// SIMD-ONLY0-NEXT:    [[TMP4708:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4708]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6138]]
+// SIMD-ONLY0:       if.end6138:
+// SIMD-ONLY0-NEXT:    [[TMP4709:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4709]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4710:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4711:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6139:%.*]] = fcmp oeq float [[TMP4710]], [[TMP4711]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6139]], label [[IF_THEN6141:%.*]], label [[IF_ELSE6142:%.*]]
+// SIMD-ONLY0:       if.then6141:
+// SIMD-ONLY0-NEXT:    [[TMP4712:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4712]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6143:%.*]]
+// SIMD-ONLY0:       if.else6142:
+// SIMD-ONLY0-NEXT:    [[TMP4713:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4713]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6143]]
+// SIMD-ONLY0:       if.end6143:
+// SIMD-ONLY0-NEXT:    [[TMP4714:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4715:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6144:%.*]] = fcmp oeq float [[TMP4714]], [[TMP4715]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6144]], label [[IF_THEN6146:%.*]], label [[IF_ELSE6147:%.*]]
+// SIMD-ONLY0:       if.then6146:
+// SIMD-ONLY0-NEXT:    [[TMP4716:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4716]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6148:%.*]]
+// SIMD-ONLY0:       if.else6147:
+// SIMD-ONLY0-NEXT:    [[TMP4717:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4717]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6148]]
+// SIMD-ONLY0:       if.end6148:
+// SIMD-ONLY0-NEXT:    [[TMP4718:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4719:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6149:%.*]] = fcmp oeq float [[TMP4718]], [[TMP4719]]
+// SIMD-ONLY0-NEXT:    [[CONV6150:%.*]] = zext i1 [[CMP6149]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6150]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4720:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6151:%.*]] = icmp ne i32 [[TMP4720]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6151]], label [[IF_THEN6152:%.*]], label [[IF_END6153:%.*]]
+// SIMD-ONLY0:       if.then6152:
+// SIMD-ONLY0-NEXT:    [[TMP4721:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4721]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6153]]
+// SIMD-ONLY0:       if.end6153:
+// SIMD-ONLY0-NEXT:    [[TMP4722:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4723:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6154:%.*]] = fcmp oeq float [[TMP4722]], [[TMP4723]]
+// SIMD-ONLY0-NEXT:    [[CONV6155:%.*]] = zext i1 [[CMP6154]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6155]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4724:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6156:%.*]] = icmp ne i32 [[TMP4724]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6156]], label [[IF_THEN6157:%.*]], label [[IF_END6158:%.*]]
+// SIMD-ONLY0:       if.then6157:
+// SIMD-ONLY0-NEXT:    [[TMP4725:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4725]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6158]]
+// SIMD-ONLY0:       if.end6158:
+// SIMD-ONLY0-NEXT:    [[TMP4726:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4727:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6159:%.*]] = fcmp oeq float [[TMP4726]], [[TMP4727]]
+// SIMD-ONLY0-NEXT:    [[CONV6160:%.*]] = zext i1 [[CMP6159]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6160]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4728:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6161:%.*]] = icmp ne i32 [[TMP4728]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6161]], label [[IF_THEN6162:%.*]], label [[IF_ELSE6163:%.*]]
+// SIMD-ONLY0:       if.then6162:
+// SIMD-ONLY0-NEXT:    [[TMP4729:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4729]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6164:%.*]]
+// SIMD-ONLY0:       if.else6163:
+// SIMD-ONLY0-NEXT:    [[TMP4730:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4730]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6164]]
+// SIMD-ONLY0:       if.end6164:
+// SIMD-ONLY0-NEXT:    [[TMP4731:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4732:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6165:%.*]] = fcmp oeq float [[TMP4731]], [[TMP4732]]
+// SIMD-ONLY0-NEXT:    [[CONV6166:%.*]] = zext i1 [[CMP6165]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6166]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4733:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6167:%.*]] = icmp ne i32 [[TMP4733]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6167]], label [[IF_THEN6168:%.*]], label [[IF_ELSE6169:%.*]]
+// SIMD-ONLY0:       if.then6168:
+// SIMD-ONLY0-NEXT:    [[TMP4734:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4734]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6170:%.*]]
+// SIMD-ONLY0:       if.else6169:
+// SIMD-ONLY0-NEXT:    [[TMP4735:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4735]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6170]]
+// SIMD-ONLY0:       if.end6170:
+// SIMD-ONLY0-NEXT:    [[TMP4736:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4736]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4737:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4738:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6171:%.*]] = fcmp ogt float [[TMP4737]], [[TMP4738]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6171]], label [[IF_THEN6173:%.*]], label [[IF_END6174:%.*]]
+// SIMD-ONLY0:       if.then6173:
+// SIMD-ONLY0-NEXT:    [[TMP4739:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4739]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6174]]
+// SIMD-ONLY0:       if.end6174:
+// SIMD-ONLY0-NEXT:    [[TMP4740:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4740]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4741:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4742:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6175:%.*]] = fcmp ogt float [[TMP4741]], [[TMP4742]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6175]], label [[IF_THEN6177:%.*]], label [[IF_END6178:%.*]]
+// SIMD-ONLY0:       if.then6177:
+// SIMD-ONLY0-NEXT:    [[TMP4743:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4743]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6178]]
+// SIMD-ONLY0:       if.end6178:
+// SIMD-ONLY0-NEXT:    [[TMP4744:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4744]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4745:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4746:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6179:%.*]] = fcmp olt float [[TMP4745]], [[TMP4746]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6179]], label [[IF_THEN6181:%.*]], label [[IF_END6182:%.*]]
+// SIMD-ONLY0:       if.then6181:
+// SIMD-ONLY0-NEXT:    [[TMP4747:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4747]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6182]]
+// SIMD-ONLY0:       if.end6182:
+// SIMD-ONLY0-NEXT:    [[TMP4748:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4748]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4749:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4750:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6183:%.*]] = fcmp olt float [[TMP4749]], [[TMP4750]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6183]], label [[IF_THEN6185:%.*]], label [[IF_END6186:%.*]]
+// SIMD-ONLY0:       if.then6185:
+// SIMD-ONLY0-NEXT:    [[TMP4751:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4751]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6186]]
+// SIMD-ONLY0:       if.end6186:
+// SIMD-ONLY0-NEXT:    [[TMP4752:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4752]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4753:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4754:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6187:%.*]] = fcmp oeq float [[TMP4753]], [[TMP4754]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6187]], label [[IF_THEN6189:%.*]], label [[IF_END6190:%.*]]
+// SIMD-ONLY0:       if.then6189:
+// SIMD-ONLY0-NEXT:    [[TMP4755:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4755]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6190]]
+// SIMD-ONLY0:       if.end6190:
+// SIMD-ONLY0-NEXT:    [[TMP4756:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4756]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4757:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4758:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6191:%.*]] = fcmp oeq float [[TMP4757]], [[TMP4758]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6191]], label [[IF_THEN6193:%.*]], label [[IF_END6194:%.*]]
+// SIMD-ONLY0:       if.then6193:
+// SIMD-ONLY0-NEXT:    [[TMP4759:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4759]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6194]]
+// SIMD-ONLY0:       if.end6194:
+// SIMD-ONLY0-NEXT:    [[TMP4760:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4761:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6195:%.*]] = fcmp ogt float [[TMP4760]], [[TMP4761]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6195]], label [[IF_THEN6197:%.*]], label [[IF_END6198:%.*]]
+// SIMD-ONLY0:       if.then6197:
+// SIMD-ONLY0-NEXT:    [[TMP4762:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4762]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6198]]
+// SIMD-ONLY0:       if.end6198:
+// SIMD-ONLY0-NEXT:    [[TMP4763:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4763]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4764:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4765:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6199:%.*]] = fcmp ogt float [[TMP4764]], [[TMP4765]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6199]], label [[IF_THEN6201:%.*]], label [[IF_END6202:%.*]]
+// SIMD-ONLY0:       if.then6201:
+// SIMD-ONLY0-NEXT:    [[TMP4766:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4766]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6202]]
+// SIMD-ONLY0:       if.end6202:
+// SIMD-ONLY0-NEXT:    [[TMP4767:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4767]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4768:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4769:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6203:%.*]] = fcmp olt float [[TMP4768]], [[TMP4769]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6203]], label [[IF_THEN6205:%.*]], label [[IF_END6206:%.*]]
+// SIMD-ONLY0:       if.then6205:
+// SIMD-ONLY0-NEXT:    [[TMP4770:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4770]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6206]]
+// SIMD-ONLY0:       if.end6206:
+// SIMD-ONLY0-NEXT:    [[TMP4771:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4771]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4772:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4773:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6207:%.*]] = fcmp olt float [[TMP4772]], [[TMP4773]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6207]], label [[IF_THEN6209:%.*]], label [[IF_END6210:%.*]]
+// SIMD-ONLY0:       if.then6209:
+// SIMD-ONLY0-NEXT:    [[TMP4774:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4774]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6210]]
+// SIMD-ONLY0:       if.end6210:
+// SIMD-ONLY0-NEXT:    [[TMP4775:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4775]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4776:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4777:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6211:%.*]] = fcmp oeq float [[TMP4776]], [[TMP4777]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6211]], label [[IF_THEN6213:%.*]], label [[IF_END6214:%.*]]
+// SIMD-ONLY0:       if.then6213:
+// SIMD-ONLY0-NEXT:    [[TMP4778:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4778]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6214]]
+// SIMD-ONLY0:       if.end6214:
+// SIMD-ONLY0-NEXT:    [[TMP4779:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4779]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4780:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4781:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6215:%.*]] = fcmp oeq float [[TMP4780]], [[TMP4781]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6215]], label [[IF_THEN6217:%.*]], label [[IF_END6218:%.*]]
+// SIMD-ONLY0:       if.then6217:
+// SIMD-ONLY0-NEXT:    [[TMP4782:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4782]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6218]]
+// SIMD-ONLY0:       if.end6218:
+// SIMD-ONLY0-NEXT:    [[TMP4783:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4783]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4784:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4785:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6219:%.*]] = fcmp oeq float [[TMP4784]], [[TMP4785]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6219]], label [[IF_THEN6221:%.*]], label [[IF_ELSE6222:%.*]]
+// SIMD-ONLY0:       if.then6221:
+// SIMD-ONLY0-NEXT:    [[TMP4786:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4786]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6223:%.*]]
+// SIMD-ONLY0:       if.else6222:
+// SIMD-ONLY0-NEXT:    [[TMP4787:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4787]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6223]]
+// SIMD-ONLY0:       if.end6223:
+// SIMD-ONLY0-NEXT:    [[TMP4788:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4789:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6224:%.*]] = fcmp oeq float [[TMP4788]], [[TMP4789]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6224]], label [[IF_THEN6226:%.*]], label [[IF_ELSE6227:%.*]]
+// SIMD-ONLY0:       if.then6226:
+// SIMD-ONLY0-NEXT:    [[TMP4790:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4790]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6228:%.*]]
+// SIMD-ONLY0:       if.else6227:
+// SIMD-ONLY0-NEXT:    [[TMP4791:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4791]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6228]]
+// SIMD-ONLY0:       if.end6228:
+// SIMD-ONLY0-NEXT:    [[TMP4792:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4793:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6229:%.*]] = fcmp oeq float [[TMP4792]], [[TMP4793]]
+// SIMD-ONLY0-NEXT:    [[CONV6230:%.*]] = zext i1 [[CMP6229]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6230]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4794:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6231:%.*]] = icmp ne i32 [[TMP4794]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6231]], label [[IF_THEN6232:%.*]], label [[IF_END6233:%.*]]
+// SIMD-ONLY0:       if.then6232:
+// SIMD-ONLY0-NEXT:    [[TMP4795:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4795]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6233]]
+// SIMD-ONLY0:       if.end6233:
+// SIMD-ONLY0-NEXT:    [[TMP4796:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4797:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6234:%.*]] = fcmp oeq float [[TMP4796]], [[TMP4797]]
+// SIMD-ONLY0-NEXT:    [[CONV6235:%.*]] = zext i1 [[CMP6234]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6235]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4798:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6236:%.*]] = icmp ne i32 [[TMP4798]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6236]], label [[IF_THEN6237:%.*]], label [[IF_END6238:%.*]]
+// SIMD-ONLY0:       if.then6237:
+// SIMD-ONLY0-NEXT:    [[TMP4799:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4799]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6238]]
+// SIMD-ONLY0:       if.end6238:
+// SIMD-ONLY0-NEXT:    [[TMP4800:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4801:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6239:%.*]] = fcmp oeq float [[TMP4800]], [[TMP4801]]
+// SIMD-ONLY0-NEXT:    [[CONV6240:%.*]] = zext i1 [[CMP6239]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6240]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4802:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6241:%.*]] = icmp ne i32 [[TMP4802]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6241]], label [[IF_THEN6242:%.*]], label [[IF_ELSE6243:%.*]]
+// SIMD-ONLY0:       if.then6242:
+// SIMD-ONLY0-NEXT:    [[TMP4803:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4803]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6244:%.*]]
+// SIMD-ONLY0:       if.else6243:
+// SIMD-ONLY0-NEXT:    [[TMP4804:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4804]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6244]]
+// SIMD-ONLY0:       if.end6244:
+// SIMD-ONLY0-NEXT:    [[TMP4805:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4806:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6245:%.*]] = fcmp oeq float [[TMP4805]], [[TMP4806]]
+// SIMD-ONLY0-NEXT:    [[CONV6246:%.*]] = zext i1 [[CMP6245]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6246]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4807:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6247:%.*]] = icmp ne i32 [[TMP4807]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6247]], label [[IF_THEN6248:%.*]], label [[IF_ELSE6249:%.*]]
+// SIMD-ONLY0:       if.then6248:
+// SIMD-ONLY0-NEXT:    [[TMP4808:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4808]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6250:%.*]]
+// SIMD-ONLY0:       if.else6249:
+// SIMD-ONLY0-NEXT:    [[TMP4809:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4809]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6250]]
+// SIMD-ONLY0:       if.end6250:
+// SIMD-ONLY0-NEXT:    [[TMP4810:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4810]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4811:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4812:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6251:%.*]] = fcmp ogt float [[TMP4811]], [[TMP4812]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6251]], label [[IF_THEN6253:%.*]], label [[IF_END6254:%.*]]
+// SIMD-ONLY0:       if.then6253:
+// SIMD-ONLY0-NEXT:    [[TMP4813:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4813]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6254]]
+// SIMD-ONLY0:       if.end6254:
+// SIMD-ONLY0-NEXT:    [[TMP4814:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4814]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4815:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4816:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6255:%.*]] = fcmp ogt float [[TMP4815]], [[TMP4816]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6255]], label [[IF_THEN6257:%.*]], label [[IF_END6258:%.*]]
+// SIMD-ONLY0:       if.then6257:
+// SIMD-ONLY0-NEXT:    [[TMP4817:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4817]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6258]]
+// SIMD-ONLY0:       if.end6258:
+// SIMD-ONLY0-NEXT:    [[TMP4818:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4818]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4819:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4820:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6259:%.*]] = fcmp olt float [[TMP4819]], [[TMP4820]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6259]], label [[IF_THEN6261:%.*]], label [[IF_END6262:%.*]]
+// SIMD-ONLY0:       if.then6261:
+// SIMD-ONLY0-NEXT:    [[TMP4821:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4821]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6262]]
+// SIMD-ONLY0:       if.end6262:
+// SIMD-ONLY0-NEXT:    [[TMP4822:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4822]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4823:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4824:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6263:%.*]] = fcmp olt float [[TMP4823]], [[TMP4824]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6263]], label [[IF_THEN6265:%.*]], label [[IF_END6266:%.*]]
+// SIMD-ONLY0:       if.then6265:
+// SIMD-ONLY0-NEXT:    [[TMP4825:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4825]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6266]]
+// SIMD-ONLY0:       if.end6266:
+// SIMD-ONLY0-NEXT:    [[TMP4826:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4826]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4827:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4828:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6267:%.*]] = fcmp oeq float [[TMP4827]], [[TMP4828]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6267]], label [[IF_THEN6269:%.*]], label [[IF_END6270:%.*]]
+// SIMD-ONLY0:       if.then6269:
+// SIMD-ONLY0-NEXT:    [[TMP4829:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4829]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6270]]
+// SIMD-ONLY0:       if.end6270:
+// SIMD-ONLY0-NEXT:    [[TMP4830:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4830]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4831:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4832:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6271:%.*]] = fcmp oeq float [[TMP4831]], [[TMP4832]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6271]], label [[IF_THEN6273:%.*]], label [[IF_END6274:%.*]]
+// SIMD-ONLY0:       if.then6273:
+// SIMD-ONLY0-NEXT:    [[TMP4833:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4833]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6274]]
+// SIMD-ONLY0:       if.end6274:
+// SIMD-ONLY0-NEXT:    [[TMP4834:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4835:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6275:%.*]] = fcmp ogt float [[TMP4834]], [[TMP4835]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6275]], label [[IF_THEN6277:%.*]], label [[IF_END6278:%.*]]
+// SIMD-ONLY0:       if.then6277:
+// SIMD-ONLY0-NEXT:    [[TMP4836:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4836]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6278]]
+// SIMD-ONLY0:       if.end6278:
+// SIMD-ONLY0-NEXT:    [[TMP4837:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4837]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4838:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4839:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6279:%.*]] = fcmp ogt float [[TMP4838]], [[TMP4839]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6279]], label [[IF_THEN6281:%.*]], label [[IF_END6282:%.*]]
+// SIMD-ONLY0:       if.then6281:
+// SIMD-ONLY0-NEXT:    [[TMP4840:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4840]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6282]]
+// SIMD-ONLY0:       if.end6282:
+// SIMD-ONLY0-NEXT:    [[TMP4841:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4841]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4842:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4843:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6283:%.*]] = fcmp olt float [[TMP4842]], [[TMP4843]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6283]], label [[IF_THEN6285:%.*]], label [[IF_END6286:%.*]]
+// SIMD-ONLY0:       if.then6285:
+// SIMD-ONLY0-NEXT:    [[TMP4844:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4844]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6286]]
+// SIMD-ONLY0:       if.end6286:
+// SIMD-ONLY0-NEXT:    [[TMP4845:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4845]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4846:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4847:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6287:%.*]] = fcmp olt float [[TMP4846]], [[TMP4847]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6287]], label [[IF_THEN6289:%.*]], label [[IF_END6290:%.*]]
+// SIMD-ONLY0:       if.then6289:
+// SIMD-ONLY0-NEXT:    [[TMP4848:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4848]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6290]]
+// SIMD-ONLY0:       if.end6290:
+// SIMD-ONLY0-NEXT:    [[TMP4849:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4849]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4850:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4851:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6291:%.*]] = fcmp oeq float [[TMP4850]], [[TMP4851]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6291]], label [[IF_THEN6293:%.*]], label [[IF_END6294:%.*]]
+// SIMD-ONLY0:       if.then6293:
+// SIMD-ONLY0-NEXT:    [[TMP4852:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4852]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6294]]
+// SIMD-ONLY0:       if.end6294:
+// SIMD-ONLY0-NEXT:    [[TMP4853:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4853]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4854:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4855:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6295:%.*]] = fcmp oeq float [[TMP4854]], [[TMP4855]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6295]], label [[IF_THEN6297:%.*]], label [[IF_END6298:%.*]]
+// SIMD-ONLY0:       if.then6297:
+// SIMD-ONLY0-NEXT:    [[TMP4856:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4856]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6298]]
+// SIMD-ONLY0:       if.end6298:
+// SIMD-ONLY0-NEXT:    [[TMP4857:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4857]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4858:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4859:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6299:%.*]] = fcmp oeq float [[TMP4858]], [[TMP4859]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6299]], label [[IF_THEN6301:%.*]], label [[IF_ELSE6302:%.*]]
+// SIMD-ONLY0:       if.then6301:
+// SIMD-ONLY0-NEXT:    [[TMP4860:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4860]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6303:%.*]]
+// SIMD-ONLY0:       if.else6302:
+// SIMD-ONLY0-NEXT:    [[TMP4861:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4861]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6303]]
+// SIMD-ONLY0:       if.end6303:
+// SIMD-ONLY0-NEXT:    [[TMP4862:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4863:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6304:%.*]] = fcmp oeq float [[TMP4862]], [[TMP4863]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6304]], label [[IF_THEN6306:%.*]], label [[IF_ELSE6307:%.*]]
+// SIMD-ONLY0:       if.then6306:
+// SIMD-ONLY0-NEXT:    [[TMP4864:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4864]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6308:%.*]]
+// SIMD-ONLY0:       if.else6307:
+// SIMD-ONLY0-NEXT:    [[TMP4865:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4865]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6308]]
+// SIMD-ONLY0:       if.end6308:
+// SIMD-ONLY0-NEXT:    [[TMP4866:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4867:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6309:%.*]] = fcmp oeq float [[TMP4866]], [[TMP4867]]
+// SIMD-ONLY0-NEXT:    [[CONV6310:%.*]] = zext i1 [[CMP6309]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6310]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4868:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6311:%.*]] = icmp ne i32 [[TMP4868]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6311]], label [[IF_THEN6312:%.*]], label [[IF_END6313:%.*]]
+// SIMD-ONLY0:       if.then6312:
+// SIMD-ONLY0-NEXT:    [[TMP4869:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4869]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6313]]
+// SIMD-ONLY0:       if.end6313:
+// SIMD-ONLY0-NEXT:    [[TMP4870:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4871:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6314:%.*]] = fcmp oeq float [[TMP4870]], [[TMP4871]]
+// SIMD-ONLY0-NEXT:    [[CONV6315:%.*]] = zext i1 [[CMP6314]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6315]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4872:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6316:%.*]] = icmp ne i32 [[TMP4872]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6316]], label [[IF_THEN6317:%.*]], label [[IF_END6318:%.*]]
+// SIMD-ONLY0:       if.then6317:
+// SIMD-ONLY0-NEXT:    [[TMP4873:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4873]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6318]]
+// SIMD-ONLY0:       if.end6318:
+// SIMD-ONLY0-NEXT:    [[TMP4874:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4875:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6319:%.*]] = fcmp oeq float [[TMP4874]], [[TMP4875]]
+// SIMD-ONLY0-NEXT:    [[CONV6320:%.*]] = zext i1 [[CMP6319]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6320]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4876:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6321:%.*]] = icmp ne i32 [[TMP4876]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6321]], label [[IF_THEN6322:%.*]], label [[IF_ELSE6323:%.*]]
+// SIMD-ONLY0:       if.then6322:
+// SIMD-ONLY0-NEXT:    [[TMP4877:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4877]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6324:%.*]]
+// SIMD-ONLY0:       if.else6323:
+// SIMD-ONLY0-NEXT:    [[TMP4878:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4878]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6324]]
+// SIMD-ONLY0:       if.end6324:
+// SIMD-ONLY0-NEXT:    [[TMP4879:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4880:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6325:%.*]] = fcmp oeq float [[TMP4879]], [[TMP4880]]
+// SIMD-ONLY0-NEXT:    [[CONV6326:%.*]] = zext i1 [[CMP6325]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6326]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4881:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6327:%.*]] = icmp ne i32 [[TMP4881]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6327]], label [[IF_THEN6328:%.*]], label [[IF_ELSE6329:%.*]]
+// SIMD-ONLY0:       if.then6328:
+// SIMD-ONLY0-NEXT:    [[TMP4882:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4882]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6330:%.*]]
+// SIMD-ONLY0:       if.else6329:
+// SIMD-ONLY0-NEXT:    [[TMP4883:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP4883]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    br label [[IF_END6330]]
+// SIMD-ONLY0:       if.end6330:
+// SIMD-ONLY0-NEXT:    [[TMP4884:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4884]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4885:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4886:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6331:%.*]] = fcmp ogt double [[TMP4885]], [[TMP4886]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6331]], label [[IF_THEN6333:%.*]], label [[IF_END6334:%.*]]
+// SIMD-ONLY0:       if.then6333:
+// SIMD-ONLY0-NEXT:    [[TMP4887:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4887]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6334]]
+// SIMD-ONLY0:       if.end6334:
+// SIMD-ONLY0-NEXT:    [[TMP4888:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4888]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4889:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4890:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6335:%.*]] = fcmp ogt double [[TMP4889]], [[TMP4890]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6335]], label [[IF_THEN6337:%.*]], label [[IF_END6338:%.*]]
+// SIMD-ONLY0:       if.then6337:
+// SIMD-ONLY0-NEXT:    [[TMP4891:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4891]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6338]]
+// SIMD-ONLY0:       if.end6338:
+// SIMD-ONLY0-NEXT:    [[TMP4892:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4892]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4893:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4894:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6339:%.*]] = fcmp olt double [[TMP4893]], [[TMP4894]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6339]], label [[IF_THEN6341:%.*]], label [[IF_END6342:%.*]]
+// SIMD-ONLY0:       if.then6341:
+// SIMD-ONLY0-NEXT:    [[TMP4895:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4895]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6342]]
+// SIMD-ONLY0:       if.end6342:
+// SIMD-ONLY0-NEXT:    [[TMP4896:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4896]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4897:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4898:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6343:%.*]] = fcmp olt double [[TMP4897]], [[TMP4898]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6343]], label [[IF_THEN6345:%.*]], label [[IF_END6346:%.*]]
+// SIMD-ONLY0:       if.then6345:
+// SIMD-ONLY0-NEXT:    [[TMP4899:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4899]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6346]]
+// SIMD-ONLY0:       if.end6346:
+// SIMD-ONLY0-NEXT:    [[TMP4900:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4900]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4901:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4902:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6347:%.*]] = fcmp oeq double [[TMP4901]], [[TMP4902]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6347]], label [[IF_THEN6349:%.*]], label [[IF_END6350:%.*]]
+// SIMD-ONLY0:       if.then6349:
+// SIMD-ONLY0-NEXT:    [[TMP4903:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4903]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6350]]
+// SIMD-ONLY0:       if.end6350:
+// SIMD-ONLY0-NEXT:    [[TMP4904:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4904]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4905:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4906:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6351:%.*]] = fcmp oeq double [[TMP4905]], [[TMP4906]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6351]], label [[IF_THEN6353:%.*]], label [[IF_END6354:%.*]]
+// SIMD-ONLY0:       if.then6353:
+// SIMD-ONLY0-NEXT:    [[TMP4907:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4907]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6354]]
+// SIMD-ONLY0:       if.end6354:
+// SIMD-ONLY0-NEXT:    [[TMP4908:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4909:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6355:%.*]] = fcmp ogt double [[TMP4908]], [[TMP4909]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6355]], label [[IF_THEN6357:%.*]], label [[IF_END6358:%.*]]
+// SIMD-ONLY0:       if.then6357:
+// SIMD-ONLY0-NEXT:    [[TMP4910:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4910]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6358]]
+// SIMD-ONLY0:       if.end6358:
+// SIMD-ONLY0-NEXT:    [[TMP4911:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4911]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4912:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4913:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6359:%.*]] = fcmp ogt double [[TMP4912]], [[TMP4913]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6359]], label [[IF_THEN6361:%.*]], label [[IF_END6362:%.*]]
+// SIMD-ONLY0:       if.then6361:
+// SIMD-ONLY0-NEXT:    [[TMP4914:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4914]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6362]]
+// SIMD-ONLY0:       if.end6362:
+// SIMD-ONLY0-NEXT:    [[TMP4915:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4915]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4916:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4917:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6363:%.*]] = fcmp olt double [[TMP4916]], [[TMP4917]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6363]], label [[IF_THEN6365:%.*]], label [[IF_END6366:%.*]]
+// SIMD-ONLY0:       if.then6365:
+// SIMD-ONLY0-NEXT:    [[TMP4918:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4918]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6366]]
+// SIMD-ONLY0:       if.end6366:
+// SIMD-ONLY0-NEXT:    [[TMP4919:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4919]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4920:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4921:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6367:%.*]] = fcmp olt double [[TMP4920]], [[TMP4921]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6367]], label [[IF_THEN6369:%.*]], label [[IF_END6370:%.*]]
+// SIMD-ONLY0:       if.then6369:
+// SIMD-ONLY0-NEXT:    [[TMP4922:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4922]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6370]]
+// SIMD-ONLY0:       if.end6370:
+// SIMD-ONLY0-NEXT:    [[TMP4923:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4923]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4924:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4925:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6371:%.*]] = fcmp oeq double [[TMP4924]], [[TMP4925]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6371]], label [[IF_THEN6373:%.*]], label [[IF_END6374:%.*]]
+// SIMD-ONLY0:       if.then6373:
+// SIMD-ONLY0-NEXT:    [[TMP4926:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4926]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6374]]
+// SIMD-ONLY0:       if.end6374:
+// SIMD-ONLY0-NEXT:    [[TMP4927:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4927]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4928:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4929:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6375:%.*]] = fcmp oeq double [[TMP4928]], [[TMP4929]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6375]], label [[IF_THEN6377:%.*]], label [[IF_END6378:%.*]]
+// SIMD-ONLY0:       if.then6377:
+// SIMD-ONLY0-NEXT:    [[TMP4930:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4930]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6378]]
+// SIMD-ONLY0:       if.end6378:
+// SIMD-ONLY0-NEXT:    [[TMP4931:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4931]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4932:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4933:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6379:%.*]] = fcmp oeq double [[TMP4932]], [[TMP4933]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6379]], label [[IF_THEN6381:%.*]], label [[IF_ELSE6382:%.*]]
+// SIMD-ONLY0:       if.then6381:
+// SIMD-ONLY0-NEXT:    [[TMP4934:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4934]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6383:%.*]]
+// SIMD-ONLY0:       if.else6382:
+// SIMD-ONLY0-NEXT:    [[TMP4935:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4935]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6383]]
+// SIMD-ONLY0:       if.end6383:
+// SIMD-ONLY0-NEXT:    [[TMP4936:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4937:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6384:%.*]] = fcmp oeq double [[TMP4936]], [[TMP4937]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6384]], label [[IF_THEN6386:%.*]], label [[IF_ELSE6387:%.*]]
+// SIMD-ONLY0:       if.then6386:
+// SIMD-ONLY0-NEXT:    [[TMP4938:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4938]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6388:%.*]]
+// SIMD-ONLY0:       if.else6387:
+// SIMD-ONLY0-NEXT:    [[TMP4939:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4939]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6388]]
+// SIMD-ONLY0:       if.end6388:
+// SIMD-ONLY0-NEXT:    [[TMP4940:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4941:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6389:%.*]] = fcmp oeq double [[TMP4940]], [[TMP4941]]
+// SIMD-ONLY0-NEXT:    [[CONV6390:%.*]] = zext i1 [[CMP6389]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6390]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4942:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6391:%.*]] = icmp ne i32 [[TMP4942]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6391]], label [[IF_THEN6392:%.*]], label [[IF_END6393:%.*]]
+// SIMD-ONLY0:       if.then6392:
+// SIMD-ONLY0-NEXT:    [[TMP4943:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4943]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6393]]
+// SIMD-ONLY0:       if.end6393:
+// SIMD-ONLY0-NEXT:    [[TMP4944:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4945:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6394:%.*]] = fcmp oeq double [[TMP4944]], [[TMP4945]]
+// SIMD-ONLY0-NEXT:    [[CONV6395:%.*]] = zext i1 [[CMP6394]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6395]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4946:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6396:%.*]] = icmp ne i32 [[TMP4946]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6396]], label [[IF_THEN6397:%.*]], label [[IF_END6398:%.*]]
+// SIMD-ONLY0:       if.then6397:
+// SIMD-ONLY0-NEXT:    [[TMP4947:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4947]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6398]]
+// SIMD-ONLY0:       if.end6398:
+// SIMD-ONLY0-NEXT:    [[TMP4948:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4949:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6399:%.*]] = fcmp oeq double [[TMP4948]], [[TMP4949]]
+// SIMD-ONLY0-NEXT:    [[CONV6400:%.*]] = zext i1 [[CMP6399]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6400]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4950:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6401:%.*]] = icmp ne i32 [[TMP4950]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6401]], label [[IF_THEN6402:%.*]], label [[IF_ELSE6403:%.*]]
+// SIMD-ONLY0:       if.then6402:
+// SIMD-ONLY0-NEXT:    [[TMP4951:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4951]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6404:%.*]]
+// SIMD-ONLY0:       if.else6403:
+// SIMD-ONLY0-NEXT:    [[TMP4952:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4952]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6404]]
+// SIMD-ONLY0:       if.end6404:
+// SIMD-ONLY0-NEXT:    [[TMP4953:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4954:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6405:%.*]] = fcmp oeq double [[TMP4953]], [[TMP4954]]
+// SIMD-ONLY0-NEXT:    [[CONV6406:%.*]] = zext i1 [[CMP6405]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6406]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4955:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6407:%.*]] = icmp ne i32 [[TMP4955]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6407]], label [[IF_THEN6408:%.*]], label [[IF_ELSE6409:%.*]]
+// SIMD-ONLY0:       if.then6408:
+// SIMD-ONLY0-NEXT:    [[TMP4956:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4956]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6410:%.*]]
+// SIMD-ONLY0:       if.else6409:
+// SIMD-ONLY0-NEXT:    [[TMP4957:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4957]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6410]]
+// SIMD-ONLY0:       if.end6410:
+// SIMD-ONLY0-NEXT:    [[TMP4958:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4958]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4959:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4960:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6411:%.*]] = fcmp ogt double [[TMP4959]], [[TMP4960]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6411]], label [[IF_THEN6413:%.*]], label [[IF_END6414:%.*]]
+// SIMD-ONLY0:       if.then6413:
+// SIMD-ONLY0-NEXT:    [[TMP4961:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4961]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6414]]
+// SIMD-ONLY0:       if.end6414:
+// SIMD-ONLY0-NEXT:    [[TMP4962:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4962]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4963:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4964:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6415:%.*]] = fcmp ogt double [[TMP4963]], [[TMP4964]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6415]], label [[IF_THEN6417:%.*]], label [[IF_END6418:%.*]]
+// SIMD-ONLY0:       if.then6417:
+// SIMD-ONLY0-NEXT:    [[TMP4965:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4965]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6418]]
+// SIMD-ONLY0:       if.end6418:
+// SIMD-ONLY0-NEXT:    [[TMP4966:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4966]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4967:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4968:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6419:%.*]] = fcmp olt double [[TMP4967]], [[TMP4968]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6419]], label [[IF_THEN6421:%.*]], label [[IF_END6422:%.*]]
+// SIMD-ONLY0:       if.then6421:
+// SIMD-ONLY0-NEXT:    [[TMP4969:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4969]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6422]]
+// SIMD-ONLY0:       if.end6422:
+// SIMD-ONLY0-NEXT:    [[TMP4970:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4970]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4971:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4972:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6423:%.*]] = fcmp olt double [[TMP4971]], [[TMP4972]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6423]], label [[IF_THEN6425:%.*]], label [[IF_END6426:%.*]]
+// SIMD-ONLY0:       if.then6425:
+// SIMD-ONLY0-NEXT:    [[TMP4973:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4973]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6426]]
+// SIMD-ONLY0:       if.end6426:
+// SIMD-ONLY0-NEXT:    [[TMP4974:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4974]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4975:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4976:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6427:%.*]] = fcmp oeq double [[TMP4975]], [[TMP4976]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6427]], label [[IF_THEN6429:%.*]], label [[IF_END6430:%.*]]
+// SIMD-ONLY0:       if.then6429:
+// SIMD-ONLY0-NEXT:    [[TMP4977:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4977]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6430]]
+// SIMD-ONLY0:       if.end6430:
+// SIMD-ONLY0-NEXT:    [[TMP4978:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4978]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4979:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4980:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6431:%.*]] = fcmp oeq double [[TMP4979]], [[TMP4980]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6431]], label [[IF_THEN6433:%.*]], label [[IF_END6434:%.*]]
+// SIMD-ONLY0:       if.then6433:
+// SIMD-ONLY0-NEXT:    [[TMP4981:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4981]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6434]]
+// SIMD-ONLY0:       if.end6434:
+// SIMD-ONLY0-NEXT:    [[TMP4982:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4983:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6435:%.*]] = fcmp ogt double [[TMP4982]], [[TMP4983]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6435]], label [[IF_THEN6437:%.*]], label [[IF_END6438:%.*]]
+// SIMD-ONLY0:       if.then6437:
+// SIMD-ONLY0-NEXT:    [[TMP4984:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4984]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6438]]
+// SIMD-ONLY0:       if.end6438:
+// SIMD-ONLY0-NEXT:    [[TMP4985:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4985]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4986:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4987:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6439:%.*]] = fcmp ogt double [[TMP4986]], [[TMP4987]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6439]], label [[IF_THEN6441:%.*]], label [[IF_END6442:%.*]]
+// SIMD-ONLY0:       if.then6441:
+// SIMD-ONLY0-NEXT:    [[TMP4988:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4988]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6442]]
+// SIMD-ONLY0:       if.end6442:
+// SIMD-ONLY0-NEXT:    [[TMP4989:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4989]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4990:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4991:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6443:%.*]] = fcmp olt double [[TMP4990]], [[TMP4991]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6443]], label [[IF_THEN6445:%.*]], label [[IF_END6446:%.*]]
+// SIMD-ONLY0:       if.then6445:
+// SIMD-ONLY0-NEXT:    [[TMP4992:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4992]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6446]]
+// SIMD-ONLY0:       if.end6446:
+// SIMD-ONLY0-NEXT:    [[TMP4993:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4993]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4994:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4995:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6447:%.*]] = fcmp olt double [[TMP4994]], [[TMP4995]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6447]], label [[IF_THEN6449:%.*]], label [[IF_END6450:%.*]]
+// SIMD-ONLY0:       if.then6449:
+// SIMD-ONLY0-NEXT:    [[TMP4996:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4996]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6450]]
+// SIMD-ONLY0:       if.end6450:
+// SIMD-ONLY0-NEXT:    [[TMP4997:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP4997]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4998:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP4999:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6451:%.*]] = fcmp oeq double [[TMP4998]], [[TMP4999]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6451]], label [[IF_THEN6453:%.*]], label [[IF_END6454:%.*]]
+// SIMD-ONLY0:       if.then6453:
+// SIMD-ONLY0-NEXT:    [[TMP5000:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5000]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6454]]
+// SIMD-ONLY0:       if.end6454:
+// SIMD-ONLY0-NEXT:    [[TMP5001:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5001]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5002:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5003:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6455:%.*]] = fcmp oeq double [[TMP5002]], [[TMP5003]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6455]], label [[IF_THEN6457:%.*]], label [[IF_END6458:%.*]]
+// SIMD-ONLY0:       if.then6457:
+// SIMD-ONLY0-NEXT:    [[TMP5004:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5004]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6458]]
+// SIMD-ONLY0:       if.end6458:
+// SIMD-ONLY0-NEXT:    [[TMP5005:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5005]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5006:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5007:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6459:%.*]] = fcmp oeq double [[TMP5006]], [[TMP5007]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6459]], label [[IF_THEN6461:%.*]], label [[IF_ELSE6462:%.*]]
+// SIMD-ONLY0:       if.then6461:
+// SIMD-ONLY0-NEXT:    [[TMP5008:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5008]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6463:%.*]]
+// SIMD-ONLY0:       if.else6462:
+// SIMD-ONLY0-NEXT:    [[TMP5009:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5009]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6463]]
+// SIMD-ONLY0:       if.end6463:
+// SIMD-ONLY0-NEXT:    [[TMP5010:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5011:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6464:%.*]] = fcmp oeq double [[TMP5010]], [[TMP5011]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6464]], label [[IF_THEN6466:%.*]], label [[IF_ELSE6467:%.*]]
+// SIMD-ONLY0:       if.then6466:
+// SIMD-ONLY0-NEXT:    [[TMP5012:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5012]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6468:%.*]]
+// SIMD-ONLY0:       if.else6467:
+// SIMD-ONLY0-NEXT:    [[TMP5013:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5013]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6468]]
+// SIMD-ONLY0:       if.end6468:
+// SIMD-ONLY0-NEXT:    [[TMP5014:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5015:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6469:%.*]] = fcmp oeq double [[TMP5014]], [[TMP5015]]
+// SIMD-ONLY0-NEXT:    [[CONV6470:%.*]] = zext i1 [[CMP6469]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6470]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5016:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6471:%.*]] = icmp ne i32 [[TMP5016]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6471]], label [[IF_THEN6472:%.*]], label [[IF_END6473:%.*]]
+// SIMD-ONLY0:       if.then6472:
+// SIMD-ONLY0-NEXT:    [[TMP5017:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5017]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6473]]
+// SIMD-ONLY0:       if.end6473:
+// SIMD-ONLY0-NEXT:    [[TMP5018:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5019:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6474:%.*]] = fcmp oeq double [[TMP5018]], [[TMP5019]]
+// SIMD-ONLY0-NEXT:    [[CONV6475:%.*]] = zext i1 [[CMP6474]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6475]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5020:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6476:%.*]] = icmp ne i32 [[TMP5020]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6476]], label [[IF_THEN6477:%.*]], label [[IF_END6478:%.*]]
+// SIMD-ONLY0:       if.then6477:
+// SIMD-ONLY0-NEXT:    [[TMP5021:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5021]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6478]]
+// SIMD-ONLY0:       if.end6478:
+// SIMD-ONLY0-NEXT:    [[TMP5022:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5023:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6479:%.*]] = fcmp oeq double [[TMP5022]], [[TMP5023]]
+// SIMD-ONLY0-NEXT:    [[CONV6480:%.*]] = zext i1 [[CMP6479]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6480]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5024:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6481:%.*]] = icmp ne i32 [[TMP5024]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6481]], label [[IF_THEN6482:%.*]], label [[IF_ELSE6483:%.*]]
+// SIMD-ONLY0:       if.then6482:
+// SIMD-ONLY0-NEXT:    [[TMP5025:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5025]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6484:%.*]]
+// SIMD-ONLY0:       if.else6483:
+// SIMD-ONLY0-NEXT:    [[TMP5026:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5026]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6484]]
+// SIMD-ONLY0:       if.end6484:
+// SIMD-ONLY0-NEXT:    [[TMP5027:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5028:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6485:%.*]] = fcmp oeq double [[TMP5027]], [[TMP5028]]
+// SIMD-ONLY0-NEXT:    [[CONV6486:%.*]] = zext i1 [[CMP6485]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6486]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5029:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6487:%.*]] = icmp ne i32 [[TMP5029]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6487]], label [[IF_THEN6488:%.*]], label [[IF_ELSE6489:%.*]]
+// SIMD-ONLY0:       if.then6488:
+// SIMD-ONLY0-NEXT:    [[TMP5030:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5030]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6490:%.*]]
+// SIMD-ONLY0:       if.else6489:
+// SIMD-ONLY0-NEXT:    [[TMP5031:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5031]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6490]]
+// SIMD-ONLY0:       if.end6490:
+// SIMD-ONLY0-NEXT:    [[TMP5032:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5032]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5033:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5034:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6491:%.*]] = fcmp ogt double [[TMP5033]], [[TMP5034]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6491]], label [[IF_THEN6493:%.*]], label [[IF_END6494:%.*]]
+// SIMD-ONLY0:       if.then6493:
+// SIMD-ONLY0-NEXT:    [[TMP5035:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5035]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6494]]
+// SIMD-ONLY0:       if.end6494:
+// SIMD-ONLY0-NEXT:    [[TMP5036:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5036]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5037:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5038:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6495:%.*]] = fcmp ogt double [[TMP5037]], [[TMP5038]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6495]], label [[IF_THEN6497:%.*]], label [[IF_END6498:%.*]]
+// SIMD-ONLY0:       if.then6497:
+// SIMD-ONLY0-NEXT:    [[TMP5039:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5039]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6498]]
+// SIMD-ONLY0:       if.end6498:
+// SIMD-ONLY0-NEXT:    [[TMP5040:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5040]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5041:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5042:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6499:%.*]] = fcmp olt double [[TMP5041]], [[TMP5042]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6499]], label [[IF_THEN6501:%.*]], label [[IF_END6502:%.*]]
+// SIMD-ONLY0:       if.then6501:
+// SIMD-ONLY0-NEXT:    [[TMP5043:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5043]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6502]]
+// SIMD-ONLY0:       if.end6502:
+// SIMD-ONLY0-NEXT:    [[TMP5044:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5044]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5045:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5046:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6503:%.*]] = fcmp olt double [[TMP5045]], [[TMP5046]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6503]], label [[IF_THEN6505:%.*]], label [[IF_END6506:%.*]]
+// SIMD-ONLY0:       if.then6505:
+// SIMD-ONLY0-NEXT:    [[TMP5047:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5047]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6506]]
+// SIMD-ONLY0:       if.end6506:
+// SIMD-ONLY0-NEXT:    [[TMP5048:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5048]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5049:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5050:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6507:%.*]] = fcmp oeq double [[TMP5049]], [[TMP5050]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6507]], label [[IF_THEN6509:%.*]], label [[IF_END6510:%.*]]
+// SIMD-ONLY0:       if.then6509:
+// SIMD-ONLY0-NEXT:    [[TMP5051:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5051]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6510]]
+// SIMD-ONLY0:       if.end6510:
+// SIMD-ONLY0-NEXT:    [[TMP5052:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5052]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5053:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5054:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6511:%.*]] = fcmp oeq double [[TMP5053]], [[TMP5054]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6511]], label [[IF_THEN6513:%.*]], label [[IF_END6514:%.*]]
+// SIMD-ONLY0:       if.then6513:
+// SIMD-ONLY0-NEXT:    [[TMP5055:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5055]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6514]]
+// SIMD-ONLY0:       if.end6514:
+// SIMD-ONLY0-NEXT:    [[TMP5056:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5057:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6515:%.*]] = fcmp ogt double [[TMP5056]], [[TMP5057]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6515]], label [[IF_THEN6517:%.*]], label [[IF_END6518:%.*]]
+// SIMD-ONLY0:       if.then6517:
+// SIMD-ONLY0-NEXT:    [[TMP5058:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5058]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6518]]
+// SIMD-ONLY0:       if.end6518:
+// SIMD-ONLY0-NEXT:    [[TMP5059:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5059]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5060:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5061:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6519:%.*]] = fcmp ogt double [[TMP5060]], [[TMP5061]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6519]], label [[IF_THEN6521:%.*]], label [[IF_END6522:%.*]]
+// SIMD-ONLY0:       if.then6521:
+// SIMD-ONLY0-NEXT:    [[TMP5062:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5062]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6522]]
+// SIMD-ONLY0:       if.end6522:
+// SIMD-ONLY0-NEXT:    [[TMP5063:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5063]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5064:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5065:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6523:%.*]] = fcmp olt double [[TMP5064]], [[TMP5065]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6523]], label [[IF_THEN6525:%.*]], label [[IF_END6526:%.*]]
+// SIMD-ONLY0:       if.then6525:
+// SIMD-ONLY0-NEXT:    [[TMP5066:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5066]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6526]]
+// SIMD-ONLY0:       if.end6526:
+// SIMD-ONLY0-NEXT:    [[TMP5067:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5067]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5068:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5069:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6527:%.*]] = fcmp olt double [[TMP5068]], [[TMP5069]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6527]], label [[IF_THEN6529:%.*]], label [[IF_END6530:%.*]]
+// SIMD-ONLY0:       if.then6529:
+// SIMD-ONLY0-NEXT:    [[TMP5070:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5070]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6530]]
+// SIMD-ONLY0:       if.end6530:
+// SIMD-ONLY0-NEXT:    [[TMP5071:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5071]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5072:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5073:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6531:%.*]] = fcmp oeq double [[TMP5072]], [[TMP5073]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6531]], label [[IF_THEN6533:%.*]], label [[IF_END6534:%.*]]
+// SIMD-ONLY0:       if.then6533:
+// SIMD-ONLY0-NEXT:    [[TMP5074:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5074]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6534]]
+// SIMD-ONLY0:       if.end6534:
+// SIMD-ONLY0-NEXT:    [[TMP5075:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5075]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5076:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5077:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6535:%.*]] = fcmp oeq double [[TMP5076]], [[TMP5077]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6535]], label [[IF_THEN6537:%.*]], label [[IF_END6538:%.*]]
+// SIMD-ONLY0:       if.then6537:
+// SIMD-ONLY0-NEXT:    [[TMP5078:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5078]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6538]]
+// SIMD-ONLY0:       if.end6538:
+// SIMD-ONLY0-NEXT:    [[TMP5079:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5079]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5080:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5081:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6539:%.*]] = fcmp oeq double [[TMP5080]], [[TMP5081]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6539]], label [[IF_THEN6541:%.*]], label [[IF_ELSE6542:%.*]]
+// SIMD-ONLY0:       if.then6541:
+// SIMD-ONLY0-NEXT:    [[TMP5082:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5082]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6543:%.*]]
+// SIMD-ONLY0:       if.else6542:
+// SIMD-ONLY0-NEXT:    [[TMP5083:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5083]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6543]]
+// SIMD-ONLY0:       if.end6543:
+// SIMD-ONLY0-NEXT:    [[TMP5084:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5085:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6544:%.*]] = fcmp oeq double [[TMP5084]], [[TMP5085]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6544]], label [[IF_THEN6546:%.*]], label [[IF_ELSE6547:%.*]]
+// SIMD-ONLY0:       if.then6546:
+// SIMD-ONLY0-NEXT:    [[TMP5086:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5086]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6548:%.*]]
+// SIMD-ONLY0:       if.else6547:
+// SIMD-ONLY0-NEXT:    [[TMP5087:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5087]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6548]]
+// SIMD-ONLY0:       if.end6548:
+// SIMD-ONLY0-NEXT:    [[TMP5088:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5089:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6549:%.*]] = fcmp oeq double [[TMP5088]], [[TMP5089]]
+// SIMD-ONLY0-NEXT:    [[CONV6550:%.*]] = zext i1 [[CMP6549]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6550]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5090:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6551:%.*]] = icmp ne i32 [[TMP5090]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6551]], label [[IF_THEN6552:%.*]], label [[IF_END6553:%.*]]
+// SIMD-ONLY0:       if.then6552:
+// SIMD-ONLY0-NEXT:    [[TMP5091:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5091]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6553]]
+// SIMD-ONLY0:       if.end6553:
+// SIMD-ONLY0-NEXT:    [[TMP5092:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5093:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6554:%.*]] = fcmp oeq double [[TMP5092]], [[TMP5093]]
+// SIMD-ONLY0-NEXT:    [[CONV6555:%.*]] = zext i1 [[CMP6554]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6555]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5094:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6556:%.*]] = icmp ne i32 [[TMP5094]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6556]], label [[IF_THEN6557:%.*]], label [[IF_END6558:%.*]]
+// SIMD-ONLY0:       if.then6557:
+// SIMD-ONLY0-NEXT:    [[TMP5095:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5095]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6558]]
+// SIMD-ONLY0:       if.end6558:
+// SIMD-ONLY0-NEXT:    [[TMP5096:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5097:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6559:%.*]] = fcmp oeq double [[TMP5096]], [[TMP5097]]
+// SIMD-ONLY0-NEXT:    [[CONV6560:%.*]] = zext i1 [[CMP6559]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6560]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5098:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6561:%.*]] = icmp ne i32 [[TMP5098]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6561]], label [[IF_THEN6562:%.*]], label [[IF_ELSE6563:%.*]]
+// SIMD-ONLY0:       if.then6562:
+// SIMD-ONLY0-NEXT:    [[TMP5099:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5099]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6564:%.*]]
+// SIMD-ONLY0:       if.else6563:
+// SIMD-ONLY0-NEXT:    [[TMP5100:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5100]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6564]]
+// SIMD-ONLY0:       if.end6564:
+// SIMD-ONLY0-NEXT:    [[TMP5101:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5102:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6565:%.*]] = fcmp oeq double [[TMP5101]], [[TMP5102]]
+// SIMD-ONLY0-NEXT:    [[CONV6566:%.*]] = zext i1 [[CMP6565]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6566]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5103:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6567:%.*]] = icmp ne i32 [[TMP5103]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6567]], label [[IF_THEN6568:%.*]], label [[IF_ELSE6569:%.*]]
+// SIMD-ONLY0:       if.then6568:
+// SIMD-ONLY0-NEXT:    [[TMP5104:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5104]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6570:%.*]]
+// SIMD-ONLY0:       if.else6569:
+// SIMD-ONLY0-NEXT:    [[TMP5105:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5105]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6570]]
+// SIMD-ONLY0:       if.end6570:
+// SIMD-ONLY0-NEXT:    [[TMP5106:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5106]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5107:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5108:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6571:%.*]] = fcmp ogt double [[TMP5107]], [[TMP5108]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6571]], label [[IF_THEN6573:%.*]], label [[IF_END6574:%.*]]
+// SIMD-ONLY0:       if.then6573:
+// SIMD-ONLY0-NEXT:    [[TMP5109:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5109]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6574]]
+// SIMD-ONLY0:       if.end6574:
+// SIMD-ONLY0-NEXT:    [[TMP5110:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5110]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5111:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5112:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6575:%.*]] = fcmp ogt double [[TMP5111]], [[TMP5112]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6575]], label [[IF_THEN6577:%.*]], label [[IF_END6578:%.*]]
+// SIMD-ONLY0:       if.then6577:
+// SIMD-ONLY0-NEXT:    [[TMP5113:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5113]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6578]]
+// SIMD-ONLY0:       if.end6578:
+// SIMD-ONLY0-NEXT:    [[TMP5114:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5114]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5115:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5116:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6579:%.*]] = fcmp olt double [[TMP5115]], [[TMP5116]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6579]], label [[IF_THEN6581:%.*]], label [[IF_END6582:%.*]]
+// SIMD-ONLY0:       if.then6581:
+// SIMD-ONLY0-NEXT:    [[TMP5117:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5117]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6582]]
+// SIMD-ONLY0:       if.end6582:
+// SIMD-ONLY0-NEXT:    [[TMP5118:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5118]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5119:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5120:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6583:%.*]] = fcmp olt double [[TMP5119]], [[TMP5120]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6583]], label [[IF_THEN6585:%.*]], label [[IF_END6586:%.*]]
+// SIMD-ONLY0:       if.then6585:
+// SIMD-ONLY0-NEXT:    [[TMP5121:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5121]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6586]]
+// SIMD-ONLY0:       if.end6586:
+// SIMD-ONLY0-NEXT:    [[TMP5122:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5122]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5123:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5124:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6587:%.*]] = fcmp oeq double [[TMP5123]], [[TMP5124]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6587]], label [[IF_THEN6589:%.*]], label [[IF_END6590:%.*]]
+// SIMD-ONLY0:       if.then6589:
+// SIMD-ONLY0-NEXT:    [[TMP5125:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5125]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6590]]
+// SIMD-ONLY0:       if.end6590:
+// SIMD-ONLY0-NEXT:    [[TMP5126:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5126]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5127:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5128:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6591:%.*]] = fcmp oeq double [[TMP5127]], [[TMP5128]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6591]], label [[IF_THEN6593:%.*]], label [[IF_END6594:%.*]]
+// SIMD-ONLY0:       if.then6593:
+// SIMD-ONLY0-NEXT:    [[TMP5129:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5129]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6594]]
+// SIMD-ONLY0:       if.end6594:
+// SIMD-ONLY0-NEXT:    [[TMP5130:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5131:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6595:%.*]] = fcmp ogt double [[TMP5130]], [[TMP5131]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6595]], label [[IF_THEN6597:%.*]], label [[IF_END6598:%.*]]
+// SIMD-ONLY0:       if.then6597:
+// SIMD-ONLY0-NEXT:    [[TMP5132:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5132]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6598]]
+// SIMD-ONLY0:       if.end6598:
+// SIMD-ONLY0-NEXT:    [[TMP5133:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5133]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5134:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5135:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6599:%.*]] = fcmp ogt double [[TMP5134]], [[TMP5135]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6599]], label [[IF_THEN6601:%.*]], label [[IF_END6602:%.*]]
+// SIMD-ONLY0:       if.then6601:
+// SIMD-ONLY0-NEXT:    [[TMP5136:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5136]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6602]]
+// SIMD-ONLY0:       if.end6602:
+// SIMD-ONLY0-NEXT:    [[TMP5137:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5137]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5138:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5139:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6603:%.*]] = fcmp olt double [[TMP5138]], [[TMP5139]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6603]], label [[IF_THEN6605:%.*]], label [[IF_END6606:%.*]]
+// SIMD-ONLY0:       if.then6605:
+// SIMD-ONLY0-NEXT:    [[TMP5140:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5140]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6606]]
+// SIMD-ONLY0:       if.end6606:
+// SIMD-ONLY0-NEXT:    [[TMP5141:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5141]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5142:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5143:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6607:%.*]] = fcmp olt double [[TMP5142]], [[TMP5143]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6607]], label [[IF_THEN6609:%.*]], label [[IF_END6610:%.*]]
+// SIMD-ONLY0:       if.then6609:
+// SIMD-ONLY0-NEXT:    [[TMP5144:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5144]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6610]]
+// SIMD-ONLY0:       if.end6610:
+// SIMD-ONLY0-NEXT:    [[TMP5145:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5145]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5146:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5147:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6611:%.*]] = fcmp oeq double [[TMP5146]], [[TMP5147]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6611]], label [[IF_THEN6613:%.*]], label [[IF_END6614:%.*]]
+// SIMD-ONLY0:       if.then6613:
+// SIMD-ONLY0-NEXT:    [[TMP5148:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5148]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6614]]
+// SIMD-ONLY0:       if.end6614:
+// SIMD-ONLY0-NEXT:    [[TMP5149:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5149]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5150:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5151:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6615:%.*]] = fcmp oeq double [[TMP5150]], [[TMP5151]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6615]], label [[IF_THEN6617:%.*]], label [[IF_END6618:%.*]]
+// SIMD-ONLY0:       if.then6617:
+// SIMD-ONLY0-NEXT:    [[TMP5152:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5152]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6618]]
+// SIMD-ONLY0:       if.end6618:
+// SIMD-ONLY0-NEXT:    [[TMP5153:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5153]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5154:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5155:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6619:%.*]] = fcmp oeq double [[TMP5154]], [[TMP5155]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6619]], label [[IF_THEN6621:%.*]], label [[IF_ELSE6622:%.*]]
+// SIMD-ONLY0:       if.then6621:
+// SIMD-ONLY0-NEXT:    [[TMP5156:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5156]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6623:%.*]]
+// SIMD-ONLY0:       if.else6622:
+// SIMD-ONLY0-NEXT:    [[TMP5157:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5157]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6623]]
+// SIMD-ONLY0:       if.end6623:
+// SIMD-ONLY0-NEXT:    [[TMP5158:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5159:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6624:%.*]] = fcmp oeq double [[TMP5158]], [[TMP5159]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6624]], label [[IF_THEN6626:%.*]], label [[IF_ELSE6627:%.*]]
+// SIMD-ONLY0:       if.then6626:
+// SIMD-ONLY0-NEXT:    [[TMP5160:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5160]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6628:%.*]]
+// SIMD-ONLY0:       if.else6627:
+// SIMD-ONLY0-NEXT:    [[TMP5161:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5161]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6628]]
+// SIMD-ONLY0:       if.end6628:
+// SIMD-ONLY0-NEXT:    [[TMP5162:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5163:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6629:%.*]] = fcmp oeq double [[TMP5162]], [[TMP5163]]
+// SIMD-ONLY0-NEXT:    [[CONV6630:%.*]] = zext i1 [[CMP6629]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6630]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5164:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6631:%.*]] = icmp ne i32 [[TMP5164]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6631]], label [[IF_THEN6632:%.*]], label [[IF_END6633:%.*]]
+// SIMD-ONLY0:       if.then6632:
+// SIMD-ONLY0-NEXT:    [[TMP5165:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5165]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6633]]
+// SIMD-ONLY0:       if.end6633:
+// SIMD-ONLY0-NEXT:    [[TMP5166:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5167:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6634:%.*]] = fcmp oeq double [[TMP5166]], [[TMP5167]]
+// SIMD-ONLY0-NEXT:    [[CONV6635:%.*]] = zext i1 [[CMP6634]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6635]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5168:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6636:%.*]] = icmp ne i32 [[TMP5168]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6636]], label [[IF_THEN6637:%.*]], label [[IF_END6638:%.*]]
+// SIMD-ONLY0:       if.then6637:
+// SIMD-ONLY0-NEXT:    [[TMP5169:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5169]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6638]]
+// SIMD-ONLY0:       if.end6638:
+// SIMD-ONLY0-NEXT:    [[TMP5170:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5171:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6639:%.*]] = fcmp oeq double [[TMP5170]], [[TMP5171]]
+// SIMD-ONLY0-NEXT:    [[CONV6640:%.*]] = zext i1 [[CMP6639]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6640]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5172:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6641:%.*]] = icmp ne i32 [[TMP5172]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6641]], label [[IF_THEN6642:%.*]], label [[IF_ELSE6643:%.*]]
+// SIMD-ONLY0:       if.then6642:
+// SIMD-ONLY0-NEXT:    [[TMP5173:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5173]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6644:%.*]]
+// SIMD-ONLY0:       if.else6643:
+// SIMD-ONLY0-NEXT:    [[TMP5174:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5174]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6644]]
+// SIMD-ONLY0:       if.end6644:
+// SIMD-ONLY0-NEXT:    [[TMP5175:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5176:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6645:%.*]] = fcmp oeq double [[TMP5175]], [[TMP5176]]
+// SIMD-ONLY0-NEXT:    [[CONV6646:%.*]] = zext i1 [[CMP6645]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6646]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5177:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6647:%.*]] = icmp ne i32 [[TMP5177]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6647]], label [[IF_THEN6648:%.*]], label [[IF_ELSE6649:%.*]]
+// SIMD-ONLY0:       if.then6648:
+// SIMD-ONLY0-NEXT:    [[TMP5178:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5178]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6650:%.*]]
+// SIMD-ONLY0:       if.else6649:
+// SIMD-ONLY0-NEXT:    [[TMP5179:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5179]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6650]]
+// SIMD-ONLY0:       if.end6650:
+// SIMD-ONLY0-NEXT:    [[TMP5180:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5180]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5181:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5182:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6651:%.*]] = fcmp ogt double [[TMP5181]], [[TMP5182]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6651]], label [[IF_THEN6653:%.*]], label [[IF_END6654:%.*]]
+// SIMD-ONLY0:       if.then6653:
+// SIMD-ONLY0-NEXT:    [[TMP5183:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5183]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6654]]
+// SIMD-ONLY0:       if.end6654:
+// SIMD-ONLY0-NEXT:    [[TMP5184:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5184]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5185:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5186:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6655:%.*]] = fcmp ogt double [[TMP5185]], [[TMP5186]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6655]], label [[IF_THEN6657:%.*]], label [[IF_END6658:%.*]]
+// SIMD-ONLY0:       if.then6657:
+// SIMD-ONLY0-NEXT:    [[TMP5187:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5187]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6658]]
+// SIMD-ONLY0:       if.end6658:
+// SIMD-ONLY0-NEXT:    [[TMP5188:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5188]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5189:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5190:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6659:%.*]] = fcmp olt double [[TMP5189]], [[TMP5190]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6659]], label [[IF_THEN6661:%.*]], label [[IF_END6662:%.*]]
+// SIMD-ONLY0:       if.then6661:
+// SIMD-ONLY0-NEXT:    [[TMP5191:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5191]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6662]]
+// SIMD-ONLY0:       if.end6662:
+// SIMD-ONLY0-NEXT:    [[TMP5192:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5192]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5193:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5194:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6663:%.*]] = fcmp olt double [[TMP5193]], [[TMP5194]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6663]], label [[IF_THEN6665:%.*]], label [[IF_END6666:%.*]]
+// SIMD-ONLY0:       if.then6665:
+// SIMD-ONLY0-NEXT:    [[TMP5195:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5195]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6666]]
+// SIMD-ONLY0:       if.end6666:
+// SIMD-ONLY0-NEXT:    [[TMP5196:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5196]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5197:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5198:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6667:%.*]] = fcmp oeq double [[TMP5197]], [[TMP5198]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6667]], label [[IF_THEN6669:%.*]], label [[IF_END6670:%.*]]
+// SIMD-ONLY0:       if.then6669:
+// SIMD-ONLY0-NEXT:    [[TMP5199:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5199]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6670]]
+// SIMD-ONLY0:       if.end6670:
+// SIMD-ONLY0-NEXT:    [[TMP5200:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5200]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5201:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5202:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6671:%.*]] = fcmp oeq double [[TMP5201]], [[TMP5202]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6671]], label [[IF_THEN6673:%.*]], label [[IF_END6674:%.*]]
+// SIMD-ONLY0:       if.then6673:
+// SIMD-ONLY0-NEXT:    [[TMP5203:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5203]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6674]]
+// SIMD-ONLY0:       if.end6674:
+// SIMD-ONLY0-NEXT:    [[TMP5204:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5205:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6675:%.*]] = fcmp ogt double [[TMP5204]], [[TMP5205]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6675]], label [[IF_THEN6677:%.*]], label [[IF_END6678:%.*]]
+// SIMD-ONLY0:       if.then6677:
+// SIMD-ONLY0-NEXT:    [[TMP5206:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5206]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6678]]
+// SIMD-ONLY0:       if.end6678:
+// SIMD-ONLY0-NEXT:    [[TMP5207:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5207]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5208:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5209:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6679:%.*]] = fcmp ogt double [[TMP5208]], [[TMP5209]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6679]], label [[IF_THEN6681:%.*]], label [[IF_END6682:%.*]]
+// SIMD-ONLY0:       if.then6681:
+// SIMD-ONLY0-NEXT:    [[TMP5210:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5210]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6682]]
+// SIMD-ONLY0:       if.end6682:
+// SIMD-ONLY0-NEXT:    [[TMP5211:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5211]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5212:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5213:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6683:%.*]] = fcmp olt double [[TMP5212]], [[TMP5213]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6683]], label [[IF_THEN6685:%.*]], label [[IF_END6686:%.*]]
+// SIMD-ONLY0:       if.then6685:
+// SIMD-ONLY0-NEXT:    [[TMP5214:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5214]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6686]]
+// SIMD-ONLY0:       if.end6686:
+// SIMD-ONLY0-NEXT:    [[TMP5215:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5215]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5216:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5217:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6687:%.*]] = fcmp olt double [[TMP5216]], [[TMP5217]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6687]], label [[IF_THEN6689:%.*]], label [[IF_END6690:%.*]]
+// SIMD-ONLY0:       if.then6689:
+// SIMD-ONLY0-NEXT:    [[TMP5218:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5218]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6690]]
+// SIMD-ONLY0:       if.end6690:
+// SIMD-ONLY0-NEXT:    [[TMP5219:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5219]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5220:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5221:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6691:%.*]] = fcmp oeq double [[TMP5220]], [[TMP5221]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6691]], label [[IF_THEN6693:%.*]], label [[IF_END6694:%.*]]
+// SIMD-ONLY0:       if.then6693:
+// SIMD-ONLY0-NEXT:    [[TMP5222:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5222]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6694]]
+// SIMD-ONLY0:       if.end6694:
+// SIMD-ONLY0-NEXT:    [[TMP5223:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5223]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5224:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5225:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6695:%.*]] = fcmp oeq double [[TMP5224]], [[TMP5225]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6695]], label [[IF_THEN6697:%.*]], label [[IF_END6698:%.*]]
+// SIMD-ONLY0:       if.then6697:
+// SIMD-ONLY0-NEXT:    [[TMP5226:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5226]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6698]]
+// SIMD-ONLY0:       if.end6698:
+// SIMD-ONLY0-NEXT:    [[TMP5227:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5227]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5228:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5229:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6699:%.*]] = fcmp oeq double [[TMP5228]], [[TMP5229]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6699]], label [[IF_THEN6701:%.*]], label [[IF_ELSE6702:%.*]]
+// SIMD-ONLY0:       if.then6701:
+// SIMD-ONLY0-NEXT:    [[TMP5230:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5230]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6703:%.*]]
+// SIMD-ONLY0:       if.else6702:
+// SIMD-ONLY0-NEXT:    [[TMP5231:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5231]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6703]]
+// SIMD-ONLY0:       if.end6703:
+// SIMD-ONLY0-NEXT:    [[TMP5232:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5233:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6704:%.*]] = fcmp oeq double [[TMP5232]], [[TMP5233]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6704]], label [[IF_THEN6706:%.*]], label [[IF_ELSE6707:%.*]]
+// SIMD-ONLY0:       if.then6706:
+// SIMD-ONLY0-NEXT:    [[TMP5234:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5234]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6708:%.*]]
+// SIMD-ONLY0:       if.else6707:
+// SIMD-ONLY0-NEXT:    [[TMP5235:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5235]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6708]]
+// SIMD-ONLY0:       if.end6708:
+// SIMD-ONLY0-NEXT:    [[TMP5236:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5237:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6709:%.*]] = fcmp oeq double [[TMP5236]], [[TMP5237]]
+// SIMD-ONLY0-NEXT:    [[CONV6710:%.*]] = zext i1 [[CMP6709]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6710]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5238:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6711:%.*]] = icmp ne i32 [[TMP5238]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6711]], label [[IF_THEN6712:%.*]], label [[IF_END6713:%.*]]
+// SIMD-ONLY0:       if.then6712:
+// SIMD-ONLY0-NEXT:    [[TMP5239:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5239]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6713]]
+// SIMD-ONLY0:       if.end6713:
+// SIMD-ONLY0-NEXT:    [[TMP5240:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5241:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6714:%.*]] = fcmp oeq double [[TMP5240]], [[TMP5241]]
+// SIMD-ONLY0-NEXT:    [[CONV6715:%.*]] = zext i1 [[CMP6714]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6715]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5242:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6716:%.*]] = icmp ne i32 [[TMP5242]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6716]], label [[IF_THEN6717:%.*]], label [[IF_END6718:%.*]]
+// SIMD-ONLY0:       if.then6717:
+// SIMD-ONLY0-NEXT:    [[TMP5243:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5243]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6718]]
+// SIMD-ONLY0:       if.end6718:
+// SIMD-ONLY0-NEXT:    [[TMP5244:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5245:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6719:%.*]] = fcmp oeq double [[TMP5244]], [[TMP5245]]
+// SIMD-ONLY0-NEXT:    [[CONV6720:%.*]] = zext i1 [[CMP6719]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6720]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5246:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6721:%.*]] = icmp ne i32 [[TMP5246]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6721]], label [[IF_THEN6722:%.*]], label [[IF_ELSE6723:%.*]]
+// SIMD-ONLY0:       if.then6722:
+// SIMD-ONLY0-NEXT:    [[TMP5247:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5247]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6724:%.*]]
+// SIMD-ONLY0:       if.else6723:
+// SIMD-ONLY0-NEXT:    [[TMP5248:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5248]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6724]]
+// SIMD-ONLY0:       if.end6724:
+// SIMD-ONLY0-NEXT:    [[TMP5249:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5250:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6725:%.*]] = fcmp oeq double [[TMP5249]], [[TMP5250]]
+// SIMD-ONLY0-NEXT:    [[CONV6726:%.*]] = zext i1 [[CMP6725]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6726]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5251:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6727:%.*]] = icmp ne i32 [[TMP5251]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6727]], label [[IF_THEN6728:%.*]], label [[IF_ELSE6729:%.*]]
+// SIMD-ONLY0:       if.then6728:
+// SIMD-ONLY0-NEXT:    [[TMP5252:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5252]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6730:%.*]]
+// SIMD-ONLY0:       if.else6729:
+// SIMD-ONLY0-NEXT:    [[TMP5253:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5253]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6730]]
+// SIMD-ONLY0:       if.end6730:
+// SIMD-ONLY0-NEXT:    [[TMP5254:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5254]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5255:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5256:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6731:%.*]] = fcmp ogt double [[TMP5255]], [[TMP5256]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6731]], label [[IF_THEN6733:%.*]], label [[IF_END6734:%.*]]
+// SIMD-ONLY0:       if.then6733:
+// SIMD-ONLY0-NEXT:    [[TMP5257:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5257]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6734]]
+// SIMD-ONLY0:       if.end6734:
+// SIMD-ONLY0-NEXT:    [[TMP5258:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5258]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5259:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5260:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6735:%.*]] = fcmp ogt double [[TMP5259]], [[TMP5260]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6735]], label [[IF_THEN6737:%.*]], label [[IF_END6738:%.*]]
+// SIMD-ONLY0:       if.then6737:
+// SIMD-ONLY0-NEXT:    [[TMP5261:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5261]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6738]]
+// SIMD-ONLY0:       if.end6738:
+// SIMD-ONLY0-NEXT:    [[TMP5262:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5262]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5263:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5264:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6739:%.*]] = fcmp olt double [[TMP5263]], [[TMP5264]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6739]], label [[IF_THEN6741:%.*]], label [[IF_END6742:%.*]]
+// SIMD-ONLY0:       if.then6741:
+// SIMD-ONLY0-NEXT:    [[TMP5265:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5265]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6742]]
+// SIMD-ONLY0:       if.end6742:
+// SIMD-ONLY0-NEXT:    [[TMP5266:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5266]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5267:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5268:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6743:%.*]] = fcmp olt double [[TMP5267]], [[TMP5268]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6743]], label [[IF_THEN6745:%.*]], label [[IF_END6746:%.*]]
+// SIMD-ONLY0:       if.then6745:
+// SIMD-ONLY0-NEXT:    [[TMP5269:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5269]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6746]]
+// SIMD-ONLY0:       if.end6746:
+// SIMD-ONLY0-NEXT:    [[TMP5270:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5270]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5271:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5272:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6747:%.*]] = fcmp oeq double [[TMP5271]], [[TMP5272]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6747]], label [[IF_THEN6749:%.*]], label [[IF_END6750:%.*]]
+// SIMD-ONLY0:       if.then6749:
+// SIMD-ONLY0-NEXT:    [[TMP5273:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5273]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6750]]
+// SIMD-ONLY0:       if.end6750:
+// SIMD-ONLY0-NEXT:    [[TMP5274:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5274]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5275:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5276:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6751:%.*]] = fcmp oeq double [[TMP5275]], [[TMP5276]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6751]], label [[IF_THEN6753:%.*]], label [[IF_END6754:%.*]]
+// SIMD-ONLY0:       if.then6753:
+// SIMD-ONLY0-NEXT:    [[TMP5277:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5277]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6754]]
+// SIMD-ONLY0:       if.end6754:
+// SIMD-ONLY0-NEXT:    [[TMP5278:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5279:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6755:%.*]] = fcmp ogt double [[TMP5278]], [[TMP5279]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6755]], label [[IF_THEN6757:%.*]], label [[IF_END6758:%.*]]
+// SIMD-ONLY0:       if.then6757:
+// SIMD-ONLY0-NEXT:    [[TMP5280:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5280]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6758]]
+// SIMD-ONLY0:       if.end6758:
+// SIMD-ONLY0-NEXT:    [[TMP5281:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5281]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5282:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5283:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6759:%.*]] = fcmp ogt double [[TMP5282]], [[TMP5283]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6759]], label [[IF_THEN6761:%.*]], label [[IF_END6762:%.*]]
+// SIMD-ONLY0:       if.then6761:
+// SIMD-ONLY0-NEXT:    [[TMP5284:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5284]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6762]]
+// SIMD-ONLY0:       if.end6762:
+// SIMD-ONLY0-NEXT:    [[TMP5285:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5285]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5286:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5287:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6763:%.*]] = fcmp olt double [[TMP5286]], [[TMP5287]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6763]], label [[IF_THEN6765:%.*]], label [[IF_END6766:%.*]]
+// SIMD-ONLY0:       if.then6765:
+// SIMD-ONLY0-NEXT:    [[TMP5288:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5288]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6766]]
+// SIMD-ONLY0:       if.end6766:
+// SIMD-ONLY0-NEXT:    [[TMP5289:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5289]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5290:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5291:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6767:%.*]] = fcmp olt double [[TMP5290]], [[TMP5291]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6767]], label [[IF_THEN6769:%.*]], label [[IF_END6770:%.*]]
+// SIMD-ONLY0:       if.then6769:
+// SIMD-ONLY0-NEXT:    [[TMP5292:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5292]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6770]]
+// SIMD-ONLY0:       if.end6770:
+// SIMD-ONLY0-NEXT:    [[TMP5293:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5293]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5294:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5295:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6771:%.*]] = fcmp oeq double [[TMP5294]], [[TMP5295]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6771]], label [[IF_THEN6773:%.*]], label [[IF_END6774:%.*]]
+// SIMD-ONLY0:       if.then6773:
+// SIMD-ONLY0-NEXT:    [[TMP5296:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5296]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6774]]
+// SIMD-ONLY0:       if.end6774:
+// SIMD-ONLY0-NEXT:    [[TMP5297:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5297]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5298:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5299:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6775:%.*]] = fcmp oeq double [[TMP5298]], [[TMP5299]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6775]], label [[IF_THEN6777:%.*]], label [[IF_END6778:%.*]]
+// SIMD-ONLY0:       if.then6777:
+// SIMD-ONLY0-NEXT:    [[TMP5300:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5300]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6778]]
+// SIMD-ONLY0:       if.end6778:
+// SIMD-ONLY0-NEXT:    [[TMP5301:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5301]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5302:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5303:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6779:%.*]] = fcmp oeq double [[TMP5302]], [[TMP5303]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6779]], label [[IF_THEN6781:%.*]], label [[IF_ELSE6782:%.*]]
+// SIMD-ONLY0:       if.then6781:
+// SIMD-ONLY0-NEXT:    [[TMP5304:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5304]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6783:%.*]]
+// SIMD-ONLY0:       if.else6782:
+// SIMD-ONLY0-NEXT:    [[TMP5305:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5305]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6783]]
+// SIMD-ONLY0:       if.end6783:
+// SIMD-ONLY0-NEXT:    [[TMP5306:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5307:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6784:%.*]] = fcmp oeq double [[TMP5306]], [[TMP5307]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6784]], label [[IF_THEN6786:%.*]], label [[IF_ELSE6787:%.*]]
+// SIMD-ONLY0:       if.then6786:
+// SIMD-ONLY0-NEXT:    [[TMP5308:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5308]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6788:%.*]]
+// SIMD-ONLY0:       if.else6787:
+// SIMD-ONLY0-NEXT:    [[TMP5309:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5309]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6788]]
+// SIMD-ONLY0:       if.end6788:
+// SIMD-ONLY0-NEXT:    [[TMP5310:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5311:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6789:%.*]] = fcmp oeq double [[TMP5310]], [[TMP5311]]
+// SIMD-ONLY0-NEXT:    [[CONV6790:%.*]] = zext i1 [[CMP6789]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6790]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5312:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6791:%.*]] = icmp ne i32 [[TMP5312]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6791]], label [[IF_THEN6792:%.*]], label [[IF_END6793:%.*]]
+// SIMD-ONLY0:       if.then6792:
+// SIMD-ONLY0-NEXT:    [[TMP5313:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5313]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6793]]
+// SIMD-ONLY0:       if.end6793:
+// SIMD-ONLY0-NEXT:    [[TMP5314:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5315:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6794:%.*]] = fcmp oeq double [[TMP5314]], [[TMP5315]]
+// SIMD-ONLY0-NEXT:    [[CONV6795:%.*]] = zext i1 [[CMP6794]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6795]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5316:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6796:%.*]] = icmp ne i32 [[TMP5316]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6796]], label [[IF_THEN6797:%.*]], label [[IF_END6798:%.*]]
+// SIMD-ONLY0:       if.then6797:
+// SIMD-ONLY0-NEXT:    [[TMP5317:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5317]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6798]]
+// SIMD-ONLY0:       if.end6798:
+// SIMD-ONLY0-NEXT:    [[TMP5318:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5319:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6799:%.*]] = fcmp oeq double [[TMP5318]], [[TMP5319]]
+// SIMD-ONLY0-NEXT:    [[CONV6800:%.*]] = zext i1 [[CMP6799]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6800]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5320:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6801:%.*]] = icmp ne i32 [[TMP5320]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6801]], label [[IF_THEN6802:%.*]], label [[IF_ELSE6803:%.*]]
+// SIMD-ONLY0:       if.then6802:
+// SIMD-ONLY0-NEXT:    [[TMP5321:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5321]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6804:%.*]]
+// SIMD-ONLY0:       if.else6803:
+// SIMD-ONLY0-NEXT:    [[TMP5322:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5322]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6804]]
+// SIMD-ONLY0:       if.end6804:
+// SIMD-ONLY0-NEXT:    [[TMP5323:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5324:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6805:%.*]] = fcmp oeq double [[TMP5323]], [[TMP5324]]
+// SIMD-ONLY0-NEXT:    [[CONV6806:%.*]] = zext i1 [[CMP6805]] to i32
+// SIMD-ONLY0-NEXT:    store i32 [[CONV6806]], ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5325:%.*]] = load i32, ptr [[IR]], align 4
+// SIMD-ONLY0-NEXT:    [[TOBOOL6807:%.*]] = icmp ne i32 [[TMP5325]], 0
+// SIMD-ONLY0-NEXT:    br i1 [[TOBOOL6807]], label [[IF_THEN6808:%.*]], label [[IF_ELSE6809:%.*]]
+// SIMD-ONLY0:       if.then6808:
+// SIMD-ONLY0-NEXT:    [[TMP5326:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5326]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6810:%.*]]
+// SIMD-ONLY0:       if.else6809:
+// SIMD-ONLY0-NEXT:    [[TMP5327:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5327]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    br label [[IF_END6810]]
+// SIMD-ONLY0:       if.end6810:
+// SIMD-ONLY0-NEXT:    ret void
+//
+//
+// SIMD-ONLY0-LABEL: @cxevd(
+// SIMD-ONLY0-NEXT:  entry:
+// SIMD-ONLY0-NEXT:    [[CX:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[CV:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[CE:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[CD:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[TMP0:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP0]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP1:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV:%.*]] = sext i8 [[TMP1]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP2:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1:%.*]] = sext i8 [[TMP2]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[CONV]], [[CONV1]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]]
+// SIMD-ONLY0:       cond.true:
+// SIMD-ONLY0-NEXT:    [[TMP3:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV3:%.*]] = sext i8 [[TMP3]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END:%.*]]
+// SIMD-ONLY0:       cond.false:
+// SIMD-ONLY0-NEXT:    [[TMP4:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV4:%.*]] = sext i8 [[TMP4]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END]]
+// SIMD-ONLY0:       cond.end:
+// SIMD-ONLY0-NEXT:    [[COND:%.*]] = phi i32 [ [[CONV3]], [[COND_TRUE]] ], [ [[CONV4]], [[COND_FALSE]] ]
+// SIMD-ONLY0-NEXT:    [[CONV5:%.*]] = trunc i32 [[COND]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV5]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP5:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP5]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP6:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV6:%.*]] = sext i8 [[TMP6]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP7:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV7:%.*]] = sext i8 [[TMP7]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP8:%.*]] = icmp slt i32 [[CONV6]], [[CONV7]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP8]], label [[COND_TRUE10:%.*]], label [[COND_FALSE12:%.*]]
+// SIMD-ONLY0:       cond.true10:
+// SIMD-ONLY0-NEXT:    [[TMP8:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV11:%.*]] = sext i8 [[TMP8]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END14:%.*]]
+// SIMD-ONLY0:       cond.false12:
+// SIMD-ONLY0-NEXT:    [[TMP9:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV13:%.*]] = sext i8 [[TMP9]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END14]]
+// SIMD-ONLY0:       cond.end14:
+// SIMD-ONLY0-NEXT:    [[COND15:%.*]] = phi i32 [ [[CONV11]], [[COND_TRUE10]] ], [ [[CONV13]], [[COND_FALSE12]] ]
+// SIMD-ONLY0-NEXT:    [[CONV16:%.*]] = trunc i32 [[COND15]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV16]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP10:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP10]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP11:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV17:%.*]] = sext i8 [[TMP11]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP12:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV18:%.*]] = sext i8 [[TMP12]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP19:%.*]] = icmp eq i32 [[CONV17]], [[CONV18]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP19]], label [[COND_TRUE21:%.*]], label [[COND_FALSE23:%.*]]
+// SIMD-ONLY0:       cond.true21:
+// SIMD-ONLY0-NEXT:    [[TMP13:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV22:%.*]] = sext i8 [[TMP13]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END25:%.*]]
+// SIMD-ONLY0:       cond.false23:
+// SIMD-ONLY0-NEXT:    [[TMP14:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV24:%.*]] = sext i8 [[TMP14]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END25]]
+// SIMD-ONLY0:       cond.end25:
+// SIMD-ONLY0-NEXT:    [[COND26:%.*]] = phi i32 [ [[CONV22]], [[COND_TRUE21]] ], [ [[CONV24]], [[COND_FALSE23]] ]
+// SIMD-ONLY0-NEXT:    [[CONV27:%.*]] = trunc i32 [[COND26]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV27]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP15:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV28:%.*]] = sext i8 [[TMP15]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP16:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV29:%.*]] = sext i8 [[TMP16]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP30:%.*]] = icmp sgt i32 [[CONV28]], [[CONV29]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP30]], label [[COND_TRUE32:%.*]], label [[COND_FALSE34:%.*]]
+// SIMD-ONLY0:       cond.true32:
+// SIMD-ONLY0-NEXT:    [[TMP17:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV33:%.*]] = sext i8 [[TMP17]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END36:%.*]]
+// SIMD-ONLY0:       cond.false34:
+// SIMD-ONLY0-NEXT:    [[TMP18:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV35:%.*]] = sext i8 [[TMP18]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END36]]
+// SIMD-ONLY0:       cond.end36:
+// SIMD-ONLY0-NEXT:    [[COND37:%.*]] = phi i32 [ [[CONV33]], [[COND_TRUE32]] ], [ [[CONV35]], [[COND_FALSE34]] ]
+// SIMD-ONLY0-NEXT:    [[CONV38:%.*]] = trunc i32 [[COND37]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV38]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP19:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP19]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP20:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV39:%.*]] = sext i8 [[TMP20]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP21:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV40:%.*]] = sext i8 [[TMP21]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP41:%.*]] = icmp slt i32 [[CONV39]], [[CONV40]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP41]], label [[COND_TRUE43:%.*]], label [[COND_FALSE45:%.*]]
+// SIMD-ONLY0:       cond.true43:
+// SIMD-ONLY0-NEXT:    [[TMP22:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV44:%.*]] = sext i8 [[TMP22]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END47:%.*]]
+// SIMD-ONLY0:       cond.false45:
+// SIMD-ONLY0-NEXT:    [[TMP23:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV46:%.*]] = sext i8 [[TMP23]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END47]]
+// SIMD-ONLY0:       cond.end47:
+// SIMD-ONLY0-NEXT:    [[COND48:%.*]] = phi i32 [ [[CONV44]], [[COND_TRUE43]] ], [ [[CONV46]], [[COND_FALSE45]] ]
+// SIMD-ONLY0-NEXT:    [[CONV49:%.*]] = trunc i32 [[COND48]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV49]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP24:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP24]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP25:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV50:%.*]] = sext i8 [[TMP25]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP26:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV51:%.*]] = sext i8 [[TMP26]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP52:%.*]] = icmp eq i32 [[CONV50]], [[CONV51]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP52]], label [[COND_TRUE54:%.*]], label [[COND_FALSE56:%.*]]
+// SIMD-ONLY0:       cond.true54:
+// SIMD-ONLY0-NEXT:    [[TMP27:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV55:%.*]] = sext i8 [[TMP27]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END58:%.*]]
+// SIMD-ONLY0:       cond.false56:
+// SIMD-ONLY0-NEXT:    [[TMP28:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV57:%.*]] = sext i8 [[TMP28]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END58]]
+// SIMD-ONLY0:       cond.end58:
+// SIMD-ONLY0-NEXT:    [[COND59:%.*]] = phi i32 [ [[CONV55]], [[COND_TRUE54]] ], [ [[CONV57]], [[COND_FALSE56]] ]
+// SIMD-ONLY0-NEXT:    [[CONV60:%.*]] = trunc i32 [[COND59]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV60]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP29:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP29]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP30:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP30]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP31:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV61:%.*]] = sext i8 [[TMP31]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP32:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV62:%.*]] = sext i8 [[TMP32]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP63:%.*]] = icmp sgt i32 [[CONV61]], [[CONV62]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP63]], label [[COND_TRUE65:%.*]], label [[COND_FALSE67:%.*]]
+// SIMD-ONLY0:       cond.true65:
+// SIMD-ONLY0-NEXT:    [[TMP33:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV66:%.*]] = sext i8 [[TMP33]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END69:%.*]]
+// SIMD-ONLY0:       cond.false67:
+// SIMD-ONLY0-NEXT:    [[TMP34:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV68:%.*]] = sext i8 [[TMP34]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END69]]
+// SIMD-ONLY0:       cond.end69:
+// SIMD-ONLY0-NEXT:    [[COND70:%.*]] = phi i32 [ [[CONV66]], [[COND_TRUE65]] ], [ [[CONV68]], [[COND_FALSE67]] ]
+// SIMD-ONLY0-NEXT:    [[CONV71:%.*]] = trunc i32 [[COND70]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV71]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP35:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP35]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP36:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV72:%.*]] = sext i8 [[TMP36]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP37:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV73:%.*]] = sext i8 [[TMP37]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP74:%.*]] = icmp slt i32 [[CONV72]], [[CONV73]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP74]], label [[COND_TRUE76:%.*]], label [[COND_FALSE78:%.*]]
+// SIMD-ONLY0:       cond.true76:
+// SIMD-ONLY0-NEXT:    [[TMP38:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV77:%.*]] = sext i8 [[TMP38]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END80:%.*]]
+// SIMD-ONLY0:       cond.false78:
+// SIMD-ONLY0-NEXT:    [[TMP39:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV79:%.*]] = sext i8 [[TMP39]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END80]]
+// SIMD-ONLY0:       cond.end80:
+// SIMD-ONLY0-NEXT:    [[COND81:%.*]] = phi i32 [ [[CONV77]], [[COND_TRUE76]] ], [ [[CONV79]], [[COND_FALSE78]] ]
+// SIMD-ONLY0-NEXT:    [[CONV82:%.*]] = trunc i32 [[COND81]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV82]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP40:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP40]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP41:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV83:%.*]] = sext i8 [[TMP41]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP42:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV84:%.*]] = sext i8 [[TMP42]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP85:%.*]] = icmp eq i32 [[CONV83]], [[CONV84]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP85]], label [[COND_TRUE87:%.*]], label [[COND_FALSE89:%.*]]
+// SIMD-ONLY0:       cond.true87:
+// SIMD-ONLY0-NEXT:    [[TMP43:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV88:%.*]] = sext i8 [[TMP43]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END91:%.*]]
+// SIMD-ONLY0:       cond.false89:
+// SIMD-ONLY0-NEXT:    [[TMP44:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV90:%.*]] = sext i8 [[TMP44]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END91]]
+// SIMD-ONLY0:       cond.end91:
+// SIMD-ONLY0-NEXT:    [[COND92:%.*]] = phi i32 [ [[CONV88]], [[COND_TRUE87]] ], [ [[CONV90]], [[COND_FALSE89]] ]
+// SIMD-ONLY0-NEXT:    [[CONV93:%.*]] = trunc i32 [[COND92]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV93]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP45:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV94:%.*]] = sext i8 [[TMP45]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP46:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV95:%.*]] = sext i8 [[TMP46]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP96:%.*]] = icmp sgt i32 [[CONV94]], [[CONV95]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP96]], label [[COND_TRUE98:%.*]], label [[COND_FALSE100:%.*]]
+// SIMD-ONLY0:       cond.true98:
+// SIMD-ONLY0-NEXT:    [[TMP47:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV99:%.*]] = sext i8 [[TMP47]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END102:%.*]]
+// SIMD-ONLY0:       cond.false100:
+// SIMD-ONLY0-NEXT:    [[TMP48:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV101:%.*]] = sext i8 [[TMP48]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END102]]
+// SIMD-ONLY0:       cond.end102:
+// SIMD-ONLY0-NEXT:    [[COND103:%.*]] = phi i32 [ [[CONV99]], [[COND_TRUE98]] ], [ [[CONV101]], [[COND_FALSE100]] ]
+// SIMD-ONLY0-NEXT:    [[CONV104:%.*]] = trunc i32 [[COND103]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV104]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP49:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP49]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP50:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV105:%.*]] = sext i8 [[TMP50]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP51:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV106:%.*]] = sext i8 [[TMP51]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP107:%.*]] = icmp slt i32 [[CONV105]], [[CONV106]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP107]], label [[COND_TRUE109:%.*]], label [[COND_FALSE111:%.*]]
+// SIMD-ONLY0:       cond.true109:
+// SIMD-ONLY0-NEXT:    [[TMP52:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV110:%.*]] = sext i8 [[TMP52]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END113:%.*]]
+// SIMD-ONLY0:       cond.false111:
+// SIMD-ONLY0-NEXT:    [[TMP53:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV112:%.*]] = sext i8 [[TMP53]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END113]]
+// SIMD-ONLY0:       cond.end113:
+// SIMD-ONLY0-NEXT:    [[COND114:%.*]] = phi i32 [ [[CONV110]], [[COND_TRUE109]] ], [ [[CONV112]], [[COND_FALSE111]] ]
+// SIMD-ONLY0-NEXT:    [[CONV115:%.*]] = trunc i32 [[COND114]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV115]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP54:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP54]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP55:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV116:%.*]] = sext i8 [[TMP55]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP56:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV117:%.*]] = sext i8 [[TMP56]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP118:%.*]] = icmp eq i32 [[CONV116]], [[CONV117]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP118]], label [[COND_TRUE120:%.*]], label [[COND_FALSE122:%.*]]
+// SIMD-ONLY0:       cond.true120:
+// SIMD-ONLY0-NEXT:    [[TMP57:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV121:%.*]] = sext i8 [[TMP57]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END124:%.*]]
+// SIMD-ONLY0:       cond.false122:
+// SIMD-ONLY0-NEXT:    [[TMP58:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV123:%.*]] = sext i8 [[TMP58]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END124]]
+// SIMD-ONLY0:       cond.end124:
+// SIMD-ONLY0-NEXT:    [[COND125:%.*]] = phi i32 [ [[CONV121]], [[COND_TRUE120]] ], [ [[CONV123]], [[COND_FALSE122]] ]
+// SIMD-ONLY0-NEXT:    [[CONV126:%.*]] = trunc i32 [[COND125]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV126]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP59:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP59]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP60:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP60]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP61:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV127:%.*]] = sext i8 [[TMP61]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP62:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV128:%.*]] = sext i8 [[TMP62]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP129:%.*]] = icmp sgt i32 [[CONV127]], [[CONV128]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP129]], label [[COND_TRUE131:%.*]], label [[COND_FALSE133:%.*]]
+// SIMD-ONLY0:       cond.true131:
+// SIMD-ONLY0-NEXT:    [[TMP63:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV132:%.*]] = sext i8 [[TMP63]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END135:%.*]]
+// SIMD-ONLY0:       cond.false133:
+// SIMD-ONLY0-NEXT:    [[TMP64:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV134:%.*]] = sext i8 [[TMP64]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END135]]
+// SIMD-ONLY0:       cond.end135:
+// SIMD-ONLY0-NEXT:    [[COND136:%.*]] = phi i32 [ [[CONV132]], [[COND_TRUE131]] ], [ [[CONV134]], [[COND_FALSE133]] ]
+// SIMD-ONLY0-NEXT:    [[CONV137:%.*]] = trunc i32 [[COND136]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV137]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP65:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP65]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP66:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV138:%.*]] = sext i8 [[TMP66]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP67:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV139:%.*]] = sext i8 [[TMP67]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP140:%.*]] = icmp slt i32 [[CONV138]], [[CONV139]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP140]], label [[COND_TRUE142:%.*]], label [[COND_FALSE144:%.*]]
+// SIMD-ONLY0:       cond.true142:
+// SIMD-ONLY0-NEXT:    [[TMP68:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV143:%.*]] = sext i8 [[TMP68]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END146:%.*]]
+// SIMD-ONLY0:       cond.false144:
+// SIMD-ONLY0-NEXT:    [[TMP69:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV145:%.*]] = sext i8 [[TMP69]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END146]]
+// SIMD-ONLY0:       cond.end146:
+// SIMD-ONLY0-NEXT:    [[COND147:%.*]] = phi i32 [ [[CONV143]], [[COND_TRUE142]] ], [ [[CONV145]], [[COND_FALSE144]] ]
+// SIMD-ONLY0-NEXT:    [[CONV148:%.*]] = trunc i32 [[COND147]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV148]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP70:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP70]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP71:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV149:%.*]] = sext i8 [[TMP71]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP72:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV150:%.*]] = sext i8 [[TMP72]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP151:%.*]] = icmp eq i32 [[CONV149]], [[CONV150]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP151]], label [[COND_TRUE153:%.*]], label [[COND_FALSE155:%.*]]
+// SIMD-ONLY0:       cond.true153:
+// SIMD-ONLY0-NEXT:    [[TMP73:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV154:%.*]] = sext i8 [[TMP73]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END157:%.*]]
+// SIMD-ONLY0:       cond.false155:
+// SIMD-ONLY0-NEXT:    [[TMP74:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV156:%.*]] = sext i8 [[TMP74]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END157]]
+// SIMD-ONLY0:       cond.end157:
+// SIMD-ONLY0-NEXT:    [[COND158:%.*]] = phi i32 [ [[CONV154]], [[COND_TRUE153]] ], [ [[CONV156]], [[COND_FALSE155]] ]
+// SIMD-ONLY0-NEXT:    [[CONV159:%.*]] = trunc i32 [[COND158]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV159]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP75:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV160:%.*]] = sext i8 [[TMP75]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP76:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV161:%.*]] = sext i8 [[TMP76]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP162:%.*]] = icmp sgt i32 [[CONV160]], [[CONV161]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP162]], label [[COND_TRUE164:%.*]], label [[COND_FALSE166:%.*]]
+// SIMD-ONLY0:       cond.true164:
+// SIMD-ONLY0-NEXT:    [[TMP77:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV165:%.*]] = sext i8 [[TMP77]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END168:%.*]]
+// SIMD-ONLY0:       cond.false166:
+// SIMD-ONLY0-NEXT:    [[TMP78:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV167:%.*]] = sext i8 [[TMP78]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END168]]
+// SIMD-ONLY0:       cond.end168:
+// SIMD-ONLY0-NEXT:    [[COND169:%.*]] = phi i32 [ [[CONV165]], [[COND_TRUE164]] ], [ [[CONV167]], [[COND_FALSE166]] ]
+// SIMD-ONLY0-NEXT:    [[CONV170:%.*]] = trunc i32 [[COND169]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV170]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP79:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP79]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP80:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV171:%.*]] = sext i8 [[TMP80]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP81:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV172:%.*]] = sext i8 [[TMP81]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP173:%.*]] = icmp slt i32 [[CONV171]], [[CONV172]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP173]], label [[COND_TRUE175:%.*]], label [[COND_FALSE177:%.*]]
+// SIMD-ONLY0:       cond.true175:
+// SIMD-ONLY0-NEXT:    [[TMP82:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV176:%.*]] = sext i8 [[TMP82]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END179:%.*]]
+// SIMD-ONLY0:       cond.false177:
+// SIMD-ONLY0-NEXT:    [[TMP83:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV178:%.*]] = sext i8 [[TMP83]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END179]]
+// SIMD-ONLY0:       cond.end179:
+// SIMD-ONLY0-NEXT:    [[COND180:%.*]] = phi i32 [ [[CONV176]], [[COND_TRUE175]] ], [ [[CONV178]], [[COND_FALSE177]] ]
+// SIMD-ONLY0-NEXT:    [[CONV181:%.*]] = trunc i32 [[COND180]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV181]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP84:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP84]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP85:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV182:%.*]] = sext i8 [[TMP85]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP86:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV183:%.*]] = sext i8 [[TMP86]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP184:%.*]] = icmp eq i32 [[CONV182]], [[CONV183]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP184]], label [[COND_TRUE186:%.*]], label [[COND_FALSE188:%.*]]
+// SIMD-ONLY0:       cond.true186:
+// SIMD-ONLY0-NEXT:    [[TMP87:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV187:%.*]] = sext i8 [[TMP87]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END190:%.*]]
+// SIMD-ONLY0:       cond.false188:
+// SIMD-ONLY0-NEXT:    [[TMP88:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV189:%.*]] = sext i8 [[TMP88]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END190]]
+// SIMD-ONLY0:       cond.end190:
+// SIMD-ONLY0-NEXT:    [[COND191:%.*]] = phi i32 [ [[CONV187]], [[COND_TRUE186]] ], [ [[CONV189]], [[COND_FALSE188]] ]
+// SIMD-ONLY0-NEXT:    [[CONV192:%.*]] = trunc i32 [[COND191]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV192]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP89:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP89]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP90:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP90]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP91:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV193:%.*]] = sext i8 [[TMP91]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP92:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV194:%.*]] = sext i8 [[TMP92]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP195:%.*]] = icmp sgt i32 [[CONV193]], [[CONV194]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP195]], label [[COND_TRUE197:%.*]], label [[COND_FALSE199:%.*]]
+// SIMD-ONLY0:       cond.true197:
+// SIMD-ONLY0-NEXT:    [[TMP93:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV198:%.*]] = sext i8 [[TMP93]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END201:%.*]]
+// SIMD-ONLY0:       cond.false199:
+// SIMD-ONLY0-NEXT:    [[TMP94:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV200:%.*]] = sext i8 [[TMP94]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END201]]
+// SIMD-ONLY0:       cond.end201:
+// SIMD-ONLY0-NEXT:    [[COND202:%.*]] = phi i32 [ [[CONV198]], [[COND_TRUE197]] ], [ [[CONV200]], [[COND_FALSE199]] ]
+// SIMD-ONLY0-NEXT:    [[CONV203:%.*]] = trunc i32 [[COND202]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV203]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP95:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP95]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP96:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV204:%.*]] = sext i8 [[TMP96]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP97:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV205:%.*]] = sext i8 [[TMP97]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP206:%.*]] = icmp slt i32 [[CONV204]], [[CONV205]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP206]], label [[COND_TRUE208:%.*]], label [[COND_FALSE210:%.*]]
+// SIMD-ONLY0:       cond.true208:
+// SIMD-ONLY0-NEXT:    [[TMP98:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV209:%.*]] = sext i8 [[TMP98]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END212:%.*]]
+// SIMD-ONLY0:       cond.false210:
+// SIMD-ONLY0-NEXT:    [[TMP99:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV211:%.*]] = sext i8 [[TMP99]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END212]]
+// SIMD-ONLY0:       cond.end212:
+// SIMD-ONLY0-NEXT:    [[COND213:%.*]] = phi i32 [ [[CONV209]], [[COND_TRUE208]] ], [ [[CONV211]], [[COND_FALSE210]] ]
+// SIMD-ONLY0-NEXT:    [[CONV214:%.*]] = trunc i32 [[COND213]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV214]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP100:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP100]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP101:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV215:%.*]] = sext i8 [[TMP101]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP102:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV216:%.*]] = sext i8 [[TMP102]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP217:%.*]] = icmp eq i32 [[CONV215]], [[CONV216]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP217]], label [[COND_TRUE219:%.*]], label [[COND_FALSE221:%.*]]
+// SIMD-ONLY0:       cond.true219:
+// SIMD-ONLY0-NEXT:    [[TMP103:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV220:%.*]] = sext i8 [[TMP103]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END223:%.*]]
+// SIMD-ONLY0:       cond.false221:
+// SIMD-ONLY0-NEXT:    [[TMP104:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV222:%.*]] = sext i8 [[TMP104]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END223]]
+// SIMD-ONLY0:       cond.end223:
+// SIMD-ONLY0-NEXT:    [[COND224:%.*]] = phi i32 [ [[CONV220]], [[COND_TRUE219]] ], [ [[CONV222]], [[COND_FALSE221]] ]
+// SIMD-ONLY0-NEXT:    [[CONV225:%.*]] = trunc i32 [[COND224]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV225]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP105:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV226:%.*]] = sext i8 [[TMP105]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP106:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV227:%.*]] = sext i8 [[TMP106]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP228:%.*]] = icmp sgt i32 [[CONV226]], [[CONV227]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP228]], label [[COND_TRUE230:%.*]], label [[COND_FALSE232:%.*]]
+// SIMD-ONLY0:       cond.true230:
+// SIMD-ONLY0-NEXT:    [[TMP107:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV231:%.*]] = sext i8 [[TMP107]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END234:%.*]]
+// SIMD-ONLY0:       cond.false232:
+// SIMD-ONLY0-NEXT:    [[TMP108:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV233:%.*]] = sext i8 [[TMP108]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END234]]
+// SIMD-ONLY0:       cond.end234:
+// SIMD-ONLY0-NEXT:    [[COND235:%.*]] = phi i32 [ [[CONV231]], [[COND_TRUE230]] ], [ [[CONV233]], [[COND_FALSE232]] ]
+// SIMD-ONLY0-NEXT:    [[CONV236:%.*]] = trunc i32 [[COND235]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV236]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP109:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP109]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP110:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV237:%.*]] = sext i8 [[TMP110]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP111:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV238:%.*]] = sext i8 [[TMP111]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP239:%.*]] = icmp slt i32 [[CONV237]], [[CONV238]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP239]], label [[COND_TRUE241:%.*]], label [[COND_FALSE243:%.*]]
+// SIMD-ONLY0:       cond.true241:
+// SIMD-ONLY0-NEXT:    [[TMP112:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV242:%.*]] = sext i8 [[TMP112]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END245:%.*]]
+// SIMD-ONLY0:       cond.false243:
+// SIMD-ONLY0-NEXT:    [[TMP113:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV244:%.*]] = sext i8 [[TMP113]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END245]]
+// SIMD-ONLY0:       cond.end245:
+// SIMD-ONLY0-NEXT:    [[COND246:%.*]] = phi i32 [ [[CONV242]], [[COND_TRUE241]] ], [ [[CONV244]], [[COND_FALSE243]] ]
+// SIMD-ONLY0-NEXT:    [[CONV247:%.*]] = trunc i32 [[COND246]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV247]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP114:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP114]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP115:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV248:%.*]] = sext i8 [[TMP115]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP116:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV249:%.*]] = sext i8 [[TMP116]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP250:%.*]] = icmp eq i32 [[CONV248]], [[CONV249]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP250]], label [[COND_TRUE252:%.*]], label [[COND_FALSE254:%.*]]
+// SIMD-ONLY0:       cond.true252:
+// SIMD-ONLY0-NEXT:    [[TMP117:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV253:%.*]] = sext i8 [[TMP117]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END256:%.*]]
+// SIMD-ONLY0:       cond.false254:
+// SIMD-ONLY0-NEXT:    [[TMP118:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV255:%.*]] = sext i8 [[TMP118]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END256]]
+// SIMD-ONLY0:       cond.end256:
+// SIMD-ONLY0-NEXT:    [[COND257:%.*]] = phi i32 [ [[CONV253]], [[COND_TRUE252]] ], [ [[CONV255]], [[COND_FALSE254]] ]
+// SIMD-ONLY0-NEXT:    [[CONV258:%.*]] = trunc i32 [[COND257]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV258]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP119:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP119]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP120:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP120]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP121:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV259:%.*]] = sext i8 [[TMP121]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP122:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV260:%.*]] = sext i8 [[TMP122]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP261:%.*]] = icmp sgt i32 [[CONV259]], [[CONV260]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP261]], label [[COND_TRUE263:%.*]], label [[COND_FALSE265:%.*]]
+// SIMD-ONLY0:       cond.true263:
+// SIMD-ONLY0-NEXT:    [[TMP123:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV264:%.*]] = sext i8 [[TMP123]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END267:%.*]]
+// SIMD-ONLY0:       cond.false265:
+// SIMD-ONLY0-NEXT:    [[TMP124:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV266:%.*]] = sext i8 [[TMP124]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END267]]
+// SIMD-ONLY0:       cond.end267:
+// SIMD-ONLY0-NEXT:    [[COND268:%.*]] = phi i32 [ [[CONV264]], [[COND_TRUE263]] ], [ [[CONV266]], [[COND_FALSE265]] ]
+// SIMD-ONLY0-NEXT:    [[CONV269:%.*]] = trunc i32 [[COND268]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV269]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP125:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP125]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP126:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV270:%.*]] = sext i8 [[TMP126]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP127:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV271:%.*]] = sext i8 [[TMP127]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP272:%.*]] = icmp slt i32 [[CONV270]], [[CONV271]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP272]], label [[COND_TRUE274:%.*]], label [[COND_FALSE276:%.*]]
+// SIMD-ONLY0:       cond.true274:
+// SIMD-ONLY0-NEXT:    [[TMP128:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV275:%.*]] = sext i8 [[TMP128]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END278:%.*]]
+// SIMD-ONLY0:       cond.false276:
+// SIMD-ONLY0-NEXT:    [[TMP129:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV277:%.*]] = sext i8 [[TMP129]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END278]]
+// SIMD-ONLY0:       cond.end278:
+// SIMD-ONLY0-NEXT:    [[COND279:%.*]] = phi i32 [ [[CONV275]], [[COND_TRUE274]] ], [ [[CONV277]], [[COND_FALSE276]] ]
+// SIMD-ONLY0-NEXT:    [[CONV280:%.*]] = trunc i32 [[COND279]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV280]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP130:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP130]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP131:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV281:%.*]] = sext i8 [[TMP131]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP132:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV282:%.*]] = sext i8 [[TMP132]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP283:%.*]] = icmp eq i32 [[CONV281]], [[CONV282]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP283]], label [[COND_TRUE285:%.*]], label [[COND_FALSE287:%.*]]
+// SIMD-ONLY0:       cond.true285:
+// SIMD-ONLY0-NEXT:    [[TMP133:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV286:%.*]] = sext i8 [[TMP133]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END289:%.*]]
+// SIMD-ONLY0:       cond.false287:
+// SIMD-ONLY0-NEXT:    [[TMP134:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV288:%.*]] = sext i8 [[TMP134]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END289]]
+// SIMD-ONLY0:       cond.end289:
+// SIMD-ONLY0-NEXT:    [[COND290:%.*]] = phi i32 [ [[CONV286]], [[COND_TRUE285]] ], [ [[CONV288]], [[COND_FALSE287]] ]
+// SIMD-ONLY0-NEXT:    [[CONV291:%.*]] = trunc i32 [[COND290]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV291]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP135:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV292:%.*]] = sext i8 [[TMP135]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP136:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV293:%.*]] = sext i8 [[TMP136]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP294:%.*]] = icmp sgt i32 [[CONV292]], [[CONV293]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP294]], label [[COND_TRUE296:%.*]], label [[COND_FALSE298:%.*]]
+// SIMD-ONLY0:       cond.true296:
+// SIMD-ONLY0-NEXT:    [[TMP137:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV297:%.*]] = sext i8 [[TMP137]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END300:%.*]]
+// SIMD-ONLY0:       cond.false298:
+// SIMD-ONLY0-NEXT:    [[TMP138:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV299:%.*]] = sext i8 [[TMP138]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END300]]
+// SIMD-ONLY0:       cond.end300:
+// SIMD-ONLY0-NEXT:    [[COND301:%.*]] = phi i32 [ [[CONV297]], [[COND_TRUE296]] ], [ [[CONV299]], [[COND_FALSE298]] ]
+// SIMD-ONLY0-NEXT:    [[CONV302:%.*]] = trunc i32 [[COND301]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV302]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP139:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP139]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP140:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV303:%.*]] = sext i8 [[TMP140]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP141:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV304:%.*]] = sext i8 [[TMP141]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP305:%.*]] = icmp slt i32 [[CONV303]], [[CONV304]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP305]], label [[COND_TRUE307:%.*]], label [[COND_FALSE309:%.*]]
+// SIMD-ONLY0:       cond.true307:
+// SIMD-ONLY0-NEXT:    [[TMP142:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV308:%.*]] = sext i8 [[TMP142]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END311:%.*]]
+// SIMD-ONLY0:       cond.false309:
+// SIMD-ONLY0-NEXT:    [[TMP143:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV310:%.*]] = sext i8 [[TMP143]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END311]]
+// SIMD-ONLY0:       cond.end311:
+// SIMD-ONLY0-NEXT:    [[COND312:%.*]] = phi i32 [ [[CONV308]], [[COND_TRUE307]] ], [ [[CONV310]], [[COND_FALSE309]] ]
+// SIMD-ONLY0-NEXT:    [[CONV313:%.*]] = trunc i32 [[COND312]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV313]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP144:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP144]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP145:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV314:%.*]] = sext i8 [[TMP145]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP146:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV315:%.*]] = sext i8 [[TMP146]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP316:%.*]] = icmp eq i32 [[CONV314]], [[CONV315]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP316]], label [[COND_TRUE318:%.*]], label [[COND_FALSE320:%.*]]
+// SIMD-ONLY0:       cond.true318:
+// SIMD-ONLY0-NEXT:    [[TMP147:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV319:%.*]] = sext i8 [[TMP147]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END322:%.*]]
+// SIMD-ONLY0:       cond.false320:
+// SIMD-ONLY0-NEXT:    [[TMP148:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV321:%.*]] = sext i8 [[TMP148]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END322]]
+// SIMD-ONLY0:       cond.end322:
+// SIMD-ONLY0-NEXT:    [[COND323:%.*]] = phi i32 [ [[CONV319]], [[COND_TRUE318]] ], [ [[CONV321]], [[COND_FALSE320]] ]
+// SIMD-ONLY0-NEXT:    [[CONV324:%.*]] = trunc i32 [[COND323]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV324]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP149:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP149]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP150:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP150]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP151:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV325:%.*]] = sext i8 [[TMP151]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP152:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV326:%.*]] = sext i8 [[TMP152]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP327:%.*]] = icmp sgt i32 [[CONV325]], [[CONV326]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP327]], label [[COND_TRUE329:%.*]], label [[COND_FALSE331:%.*]]
+// SIMD-ONLY0:       cond.true329:
+// SIMD-ONLY0-NEXT:    [[TMP153:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV330:%.*]] = sext i8 [[TMP153]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END333:%.*]]
+// SIMD-ONLY0:       cond.false331:
+// SIMD-ONLY0-NEXT:    [[TMP154:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV332:%.*]] = sext i8 [[TMP154]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END333]]
+// SIMD-ONLY0:       cond.end333:
+// SIMD-ONLY0-NEXT:    [[COND334:%.*]] = phi i32 [ [[CONV330]], [[COND_TRUE329]] ], [ [[CONV332]], [[COND_FALSE331]] ]
+// SIMD-ONLY0-NEXT:    [[CONV335:%.*]] = trunc i32 [[COND334]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV335]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP155:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP155]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP156:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV336:%.*]] = sext i8 [[TMP156]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP157:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV337:%.*]] = sext i8 [[TMP157]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP338:%.*]] = icmp slt i32 [[CONV336]], [[CONV337]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP338]], label [[COND_TRUE340:%.*]], label [[COND_FALSE342:%.*]]
+// SIMD-ONLY0:       cond.true340:
+// SIMD-ONLY0-NEXT:    [[TMP158:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV341:%.*]] = sext i8 [[TMP158]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END344:%.*]]
+// SIMD-ONLY0:       cond.false342:
+// SIMD-ONLY0-NEXT:    [[TMP159:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV343:%.*]] = sext i8 [[TMP159]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END344]]
+// SIMD-ONLY0:       cond.end344:
+// SIMD-ONLY0-NEXT:    [[COND345:%.*]] = phi i32 [ [[CONV341]], [[COND_TRUE340]] ], [ [[CONV343]], [[COND_FALSE342]] ]
+// SIMD-ONLY0-NEXT:    [[CONV346:%.*]] = trunc i32 [[COND345]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV346]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP160:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP160]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP161:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV347:%.*]] = sext i8 [[TMP161]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP162:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV348:%.*]] = sext i8 [[TMP162]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP349:%.*]] = icmp eq i32 [[CONV347]], [[CONV348]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP349]], label [[COND_TRUE351:%.*]], label [[COND_FALSE353:%.*]]
+// SIMD-ONLY0:       cond.true351:
+// SIMD-ONLY0-NEXT:    [[TMP163:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV352:%.*]] = sext i8 [[TMP163]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END355:%.*]]
+// SIMD-ONLY0:       cond.false353:
+// SIMD-ONLY0-NEXT:    [[TMP164:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV354:%.*]] = sext i8 [[TMP164]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END355]]
+// SIMD-ONLY0:       cond.end355:
+// SIMD-ONLY0-NEXT:    [[COND356:%.*]] = phi i32 [ [[CONV352]], [[COND_TRUE351]] ], [ [[CONV354]], [[COND_FALSE353]] ]
+// SIMD-ONLY0-NEXT:    [[CONV357:%.*]] = trunc i32 [[COND356]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV357]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP165:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV358:%.*]] = sext i8 [[TMP165]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP166:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV359:%.*]] = sext i8 [[TMP166]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP360:%.*]] = icmp sgt i32 [[CONV358]], [[CONV359]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP360]], label [[COND_TRUE362:%.*]], label [[COND_FALSE364:%.*]]
+// SIMD-ONLY0:       cond.true362:
+// SIMD-ONLY0-NEXT:    [[TMP167:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV363:%.*]] = sext i8 [[TMP167]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END366:%.*]]
+// SIMD-ONLY0:       cond.false364:
+// SIMD-ONLY0-NEXT:    [[TMP168:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV365:%.*]] = sext i8 [[TMP168]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END366]]
+// SIMD-ONLY0:       cond.end366:
+// SIMD-ONLY0-NEXT:    [[COND367:%.*]] = phi i32 [ [[CONV363]], [[COND_TRUE362]] ], [ [[CONV365]], [[COND_FALSE364]] ]
+// SIMD-ONLY0-NEXT:    [[CONV368:%.*]] = trunc i32 [[COND367]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV368]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP169:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP169]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP170:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV369:%.*]] = sext i8 [[TMP170]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP171:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV370:%.*]] = sext i8 [[TMP171]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP371:%.*]] = icmp slt i32 [[CONV369]], [[CONV370]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP371]], label [[COND_TRUE373:%.*]], label [[COND_FALSE375:%.*]]
+// SIMD-ONLY0:       cond.true373:
+// SIMD-ONLY0-NEXT:    [[TMP172:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV374:%.*]] = sext i8 [[TMP172]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END377:%.*]]
+// SIMD-ONLY0:       cond.false375:
+// SIMD-ONLY0-NEXT:    [[TMP173:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV376:%.*]] = sext i8 [[TMP173]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END377]]
+// SIMD-ONLY0:       cond.end377:
+// SIMD-ONLY0-NEXT:    [[COND378:%.*]] = phi i32 [ [[CONV374]], [[COND_TRUE373]] ], [ [[CONV376]], [[COND_FALSE375]] ]
+// SIMD-ONLY0-NEXT:    [[CONV379:%.*]] = trunc i32 [[COND378]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV379]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP174:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP174]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP175:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV380:%.*]] = sext i8 [[TMP175]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP176:%.*]] = load i8, ptr [[CE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV381:%.*]] = sext i8 [[TMP176]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP382:%.*]] = icmp eq i32 [[CONV380]], [[CONV381]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP382]], label [[COND_TRUE384:%.*]], label [[COND_FALSE386:%.*]]
+// SIMD-ONLY0:       cond.true384:
+// SIMD-ONLY0-NEXT:    [[TMP177:%.*]] = load i8, ptr [[CD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV385:%.*]] = sext i8 [[TMP177]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END388:%.*]]
+// SIMD-ONLY0:       cond.false386:
+// SIMD-ONLY0-NEXT:    [[TMP178:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV387:%.*]] = sext i8 [[TMP178]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END388]]
+// SIMD-ONLY0:       cond.end388:
+// SIMD-ONLY0-NEXT:    [[COND389:%.*]] = phi i32 [ [[CONV385]], [[COND_TRUE384]] ], [ [[CONV387]], [[COND_FALSE386]] ]
+// SIMD-ONLY0-NEXT:    [[CONV390:%.*]] = trunc i32 [[COND389]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV390]], ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP179:%.*]] = load i8, ptr [[CX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP179]], ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP180:%.*]] = load i8, ptr [[CV]], align 1
+// SIMD-ONLY0-NEXT:    ret i8 [[TMP180]]
+//
+//
+// SIMD-ONLY0-LABEL: @ucxevd(
+// SIMD-ONLY0-NEXT:  entry:
+// SIMD-ONLY0-NEXT:    [[UCX:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[UCV:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[UCE:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[UCD:%.*]] = alloca i8, align 1
+// SIMD-ONLY0-NEXT:    [[TMP0:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP0]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP1:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV:%.*]] = zext i8 [[TMP1]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP2:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV1:%.*]] = zext i8 [[TMP2]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[CONV]], [[CONV1]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]]
+// SIMD-ONLY0:       cond.true:
+// SIMD-ONLY0-NEXT:    [[TMP3:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV3:%.*]] = zext i8 [[TMP3]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END:%.*]]
+// SIMD-ONLY0:       cond.false:
+// SIMD-ONLY0-NEXT:    [[TMP4:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV4:%.*]] = zext i8 [[TMP4]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END]]
+// SIMD-ONLY0:       cond.end:
+// SIMD-ONLY0-NEXT:    [[COND:%.*]] = phi i32 [ [[CONV3]], [[COND_TRUE]] ], [ [[CONV4]], [[COND_FALSE]] ]
+// SIMD-ONLY0-NEXT:    [[CONV5:%.*]] = trunc i32 [[COND]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV5]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP5:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP5]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP6:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV6:%.*]] = zext i8 [[TMP6]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP7:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV7:%.*]] = zext i8 [[TMP7]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP8:%.*]] = icmp slt i32 [[CONV6]], [[CONV7]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP8]], label [[COND_TRUE10:%.*]], label [[COND_FALSE12:%.*]]
+// SIMD-ONLY0:       cond.true10:
+// SIMD-ONLY0-NEXT:    [[TMP8:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV11:%.*]] = zext i8 [[TMP8]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END14:%.*]]
+// SIMD-ONLY0:       cond.false12:
+// SIMD-ONLY0-NEXT:    [[TMP9:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV13:%.*]] = zext i8 [[TMP9]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END14]]
+// SIMD-ONLY0:       cond.end14:
+// SIMD-ONLY0-NEXT:    [[COND15:%.*]] = phi i32 [ [[CONV11]], [[COND_TRUE10]] ], [ [[CONV13]], [[COND_FALSE12]] ]
+// SIMD-ONLY0-NEXT:    [[CONV16:%.*]] = trunc i32 [[COND15]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV16]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP10:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP10]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP11:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV17:%.*]] = zext i8 [[TMP11]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP12:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV18:%.*]] = zext i8 [[TMP12]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP19:%.*]] = icmp eq i32 [[CONV17]], [[CONV18]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP19]], label [[COND_TRUE21:%.*]], label [[COND_FALSE23:%.*]]
+// SIMD-ONLY0:       cond.true21:
+// SIMD-ONLY0-NEXT:    [[TMP13:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV22:%.*]] = zext i8 [[TMP13]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END25:%.*]]
+// SIMD-ONLY0:       cond.false23:
+// SIMD-ONLY0-NEXT:    [[TMP14:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV24:%.*]] = zext i8 [[TMP14]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END25]]
+// SIMD-ONLY0:       cond.end25:
+// SIMD-ONLY0-NEXT:    [[COND26:%.*]] = phi i32 [ [[CONV22]], [[COND_TRUE21]] ], [ [[CONV24]], [[COND_FALSE23]] ]
+// SIMD-ONLY0-NEXT:    [[CONV27:%.*]] = trunc i32 [[COND26]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV27]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP15:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV28:%.*]] = zext i8 [[TMP15]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP16:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV29:%.*]] = zext i8 [[TMP16]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP30:%.*]] = icmp sgt i32 [[CONV28]], [[CONV29]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP30]], label [[COND_TRUE32:%.*]], label [[COND_FALSE34:%.*]]
+// SIMD-ONLY0:       cond.true32:
+// SIMD-ONLY0-NEXT:    [[TMP17:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV33:%.*]] = zext i8 [[TMP17]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END36:%.*]]
+// SIMD-ONLY0:       cond.false34:
+// SIMD-ONLY0-NEXT:    [[TMP18:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV35:%.*]] = zext i8 [[TMP18]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END36]]
+// SIMD-ONLY0:       cond.end36:
+// SIMD-ONLY0-NEXT:    [[COND37:%.*]] = phi i32 [ [[CONV33]], [[COND_TRUE32]] ], [ [[CONV35]], [[COND_FALSE34]] ]
+// SIMD-ONLY0-NEXT:    [[CONV38:%.*]] = trunc i32 [[COND37]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV38]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP19:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP19]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP20:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV39:%.*]] = zext i8 [[TMP20]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP21:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV40:%.*]] = zext i8 [[TMP21]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP41:%.*]] = icmp slt i32 [[CONV39]], [[CONV40]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP41]], label [[COND_TRUE43:%.*]], label [[COND_FALSE45:%.*]]
+// SIMD-ONLY0:       cond.true43:
+// SIMD-ONLY0-NEXT:    [[TMP22:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV44:%.*]] = zext i8 [[TMP22]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END47:%.*]]
+// SIMD-ONLY0:       cond.false45:
+// SIMD-ONLY0-NEXT:    [[TMP23:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV46:%.*]] = zext i8 [[TMP23]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END47]]
+// SIMD-ONLY0:       cond.end47:
+// SIMD-ONLY0-NEXT:    [[COND48:%.*]] = phi i32 [ [[CONV44]], [[COND_TRUE43]] ], [ [[CONV46]], [[COND_FALSE45]] ]
+// SIMD-ONLY0-NEXT:    [[CONV49:%.*]] = trunc i32 [[COND48]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV49]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP24:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP24]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP25:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV50:%.*]] = zext i8 [[TMP25]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP26:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV51:%.*]] = zext i8 [[TMP26]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP52:%.*]] = icmp eq i32 [[CONV50]], [[CONV51]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP52]], label [[COND_TRUE54:%.*]], label [[COND_FALSE56:%.*]]
+// SIMD-ONLY0:       cond.true54:
+// SIMD-ONLY0-NEXT:    [[TMP27:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV55:%.*]] = zext i8 [[TMP27]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END58:%.*]]
+// SIMD-ONLY0:       cond.false56:
+// SIMD-ONLY0-NEXT:    [[TMP28:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV57:%.*]] = zext i8 [[TMP28]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END58]]
+// SIMD-ONLY0:       cond.end58:
+// SIMD-ONLY0-NEXT:    [[COND59:%.*]] = phi i32 [ [[CONV55]], [[COND_TRUE54]] ], [ [[CONV57]], [[COND_FALSE56]] ]
+// SIMD-ONLY0-NEXT:    [[CONV60:%.*]] = trunc i32 [[COND59]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV60]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP29:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP29]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP30:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP30]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP31:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV61:%.*]] = zext i8 [[TMP31]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP32:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV62:%.*]] = zext i8 [[TMP32]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP63:%.*]] = icmp sgt i32 [[CONV61]], [[CONV62]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP63]], label [[COND_TRUE65:%.*]], label [[COND_FALSE67:%.*]]
+// SIMD-ONLY0:       cond.true65:
+// SIMD-ONLY0-NEXT:    [[TMP33:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV66:%.*]] = zext i8 [[TMP33]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END69:%.*]]
+// SIMD-ONLY0:       cond.false67:
+// SIMD-ONLY0-NEXT:    [[TMP34:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV68:%.*]] = zext i8 [[TMP34]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END69]]
+// SIMD-ONLY0:       cond.end69:
+// SIMD-ONLY0-NEXT:    [[COND70:%.*]] = phi i32 [ [[CONV66]], [[COND_TRUE65]] ], [ [[CONV68]], [[COND_FALSE67]] ]
+// SIMD-ONLY0-NEXT:    [[CONV71:%.*]] = trunc i32 [[COND70]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV71]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP35:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP35]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP36:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV72:%.*]] = zext i8 [[TMP36]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP37:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV73:%.*]] = zext i8 [[TMP37]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP74:%.*]] = icmp slt i32 [[CONV72]], [[CONV73]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP74]], label [[COND_TRUE76:%.*]], label [[COND_FALSE78:%.*]]
+// SIMD-ONLY0:       cond.true76:
+// SIMD-ONLY0-NEXT:    [[TMP38:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV77:%.*]] = zext i8 [[TMP38]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END80:%.*]]
+// SIMD-ONLY0:       cond.false78:
+// SIMD-ONLY0-NEXT:    [[TMP39:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV79:%.*]] = zext i8 [[TMP39]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END80]]
+// SIMD-ONLY0:       cond.end80:
+// SIMD-ONLY0-NEXT:    [[COND81:%.*]] = phi i32 [ [[CONV77]], [[COND_TRUE76]] ], [ [[CONV79]], [[COND_FALSE78]] ]
+// SIMD-ONLY0-NEXT:    [[CONV82:%.*]] = trunc i32 [[COND81]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV82]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP40:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP40]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP41:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV83:%.*]] = zext i8 [[TMP41]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP42:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV84:%.*]] = zext i8 [[TMP42]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP85:%.*]] = icmp eq i32 [[CONV83]], [[CONV84]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP85]], label [[COND_TRUE87:%.*]], label [[COND_FALSE89:%.*]]
+// SIMD-ONLY0:       cond.true87:
+// SIMD-ONLY0-NEXT:    [[TMP43:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV88:%.*]] = zext i8 [[TMP43]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END91:%.*]]
+// SIMD-ONLY0:       cond.false89:
+// SIMD-ONLY0-NEXT:    [[TMP44:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV90:%.*]] = zext i8 [[TMP44]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END91]]
+// SIMD-ONLY0:       cond.end91:
+// SIMD-ONLY0-NEXT:    [[COND92:%.*]] = phi i32 [ [[CONV88]], [[COND_TRUE87]] ], [ [[CONV90]], [[COND_FALSE89]] ]
+// SIMD-ONLY0-NEXT:    [[CONV93:%.*]] = trunc i32 [[COND92]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV93]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP45:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV94:%.*]] = zext i8 [[TMP45]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP46:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV95:%.*]] = zext i8 [[TMP46]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP96:%.*]] = icmp sgt i32 [[CONV94]], [[CONV95]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP96]], label [[COND_TRUE98:%.*]], label [[COND_FALSE100:%.*]]
+// SIMD-ONLY0:       cond.true98:
+// SIMD-ONLY0-NEXT:    [[TMP47:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV99:%.*]] = zext i8 [[TMP47]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END102:%.*]]
+// SIMD-ONLY0:       cond.false100:
+// SIMD-ONLY0-NEXT:    [[TMP48:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV101:%.*]] = zext i8 [[TMP48]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END102]]
+// SIMD-ONLY0:       cond.end102:
+// SIMD-ONLY0-NEXT:    [[COND103:%.*]] = phi i32 [ [[CONV99]], [[COND_TRUE98]] ], [ [[CONV101]], [[COND_FALSE100]] ]
+// SIMD-ONLY0-NEXT:    [[CONV104:%.*]] = trunc i32 [[COND103]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV104]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP49:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP49]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP50:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV105:%.*]] = zext i8 [[TMP50]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP51:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV106:%.*]] = zext i8 [[TMP51]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP107:%.*]] = icmp slt i32 [[CONV105]], [[CONV106]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP107]], label [[COND_TRUE109:%.*]], label [[COND_FALSE111:%.*]]
+// SIMD-ONLY0:       cond.true109:
+// SIMD-ONLY0-NEXT:    [[TMP52:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV110:%.*]] = zext i8 [[TMP52]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END113:%.*]]
+// SIMD-ONLY0:       cond.false111:
+// SIMD-ONLY0-NEXT:    [[TMP53:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV112:%.*]] = zext i8 [[TMP53]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END113]]
+// SIMD-ONLY0:       cond.end113:
+// SIMD-ONLY0-NEXT:    [[COND114:%.*]] = phi i32 [ [[CONV110]], [[COND_TRUE109]] ], [ [[CONV112]], [[COND_FALSE111]] ]
+// SIMD-ONLY0-NEXT:    [[CONV115:%.*]] = trunc i32 [[COND114]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV115]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP54:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP54]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP55:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV116:%.*]] = zext i8 [[TMP55]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP56:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV117:%.*]] = zext i8 [[TMP56]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP118:%.*]] = icmp eq i32 [[CONV116]], [[CONV117]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP118]], label [[COND_TRUE120:%.*]], label [[COND_FALSE122:%.*]]
+// SIMD-ONLY0:       cond.true120:
+// SIMD-ONLY0-NEXT:    [[TMP57:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV121:%.*]] = zext i8 [[TMP57]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END124:%.*]]
+// SIMD-ONLY0:       cond.false122:
+// SIMD-ONLY0-NEXT:    [[TMP58:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV123:%.*]] = zext i8 [[TMP58]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END124]]
+// SIMD-ONLY0:       cond.end124:
+// SIMD-ONLY0-NEXT:    [[COND125:%.*]] = phi i32 [ [[CONV121]], [[COND_TRUE120]] ], [ [[CONV123]], [[COND_FALSE122]] ]
+// SIMD-ONLY0-NEXT:    [[CONV126:%.*]] = trunc i32 [[COND125]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV126]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP59:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP59]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP60:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP60]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP61:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV127:%.*]] = zext i8 [[TMP61]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP62:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV128:%.*]] = zext i8 [[TMP62]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP129:%.*]] = icmp sgt i32 [[CONV127]], [[CONV128]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP129]], label [[COND_TRUE131:%.*]], label [[COND_FALSE133:%.*]]
+// SIMD-ONLY0:       cond.true131:
+// SIMD-ONLY0-NEXT:    [[TMP63:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV132:%.*]] = zext i8 [[TMP63]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END135:%.*]]
+// SIMD-ONLY0:       cond.false133:
+// SIMD-ONLY0-NEXT:    [[TMP64:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV134:%.*]] = zext i8 [[TMP64]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END135]]
+// SIMD-ONLY0:       cond.end135:
+// SIMD-ONLY0-NEXT:    [[COND136:%.*]] = phi i32 [ [[CONV132]], [[COND_TRUE131]] ], [ [[CONV134]], [[COND_FALSE133]] ]
+// SIMD-ONLY0-NEXT:    [[CONV137:%.*]] = trunc i32 [[COND136]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV137]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP65:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP65]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP66:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV138:%.*]] = zext i8 [[TMP66]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP67:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV139:%.*]] = zext i8 [[TMP67]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP140:%.*]] = icmp slt i32 [[CONV138]], [[CONV139]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP140]], label [[COND_TRUE142:%.*]], label [[COND_FALSE144:%.*]]
+// SIMD-ONLY0:       cond.true142:
+// SIMD-ONLY0-NEXT:    [[TMP68:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV143:%.*]] = zext i8 [[TMP68]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END146:%.*]]
+// SIMD-ONLY0:       cond.false144:
+// SIMD-ONLY0-NEXT:    [[TMP69:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV145:%.*]] = zext i8 [[TMP69]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END146]]
+// SIMD-ONLY0:       cond.end146:
+// SIMD-ONLY0-NEXT:    [[COND147:%.*]] = phi i32 [ [[CONV143]], [[COND_TRUE142]] ], [ [[CONV145]], [[COND_FALSE144]] ]
+// SIMD-ONLY0-NEXT:    [[CONV148:%.*]] = trunc i32 [[COND147]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV148]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP70:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP70]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP71:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV149:%.*]] = zext i8 [[TMP71]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP72:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV150:%.*]] = zext i8 [[TMP72]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP151:%.*]] = icmp eq i32 [[CONV149]], [[CONV150]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP151]], label [[COND_TRUE153:%.*]], label [[COND_FALSE155:%.*]]
+// SIMD-ONLY0:       cond.true153:
+// SIMD-ONLY0-NEXT:    [[TMP73:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV154:%.*]] = zext i8 [[TMP73]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END157:%.*]]
+// SIMD-ONLY0:       cond.false155:
+// SIMD-ONLY0-NEXT:    [[TMP74:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV156:%.*]] = zext i8 [[TMP74]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END157]]
+// SIMD-ONLY0:       cond.end157:
+// SIMD-ONLY0-NEXT:    [[COND158:%.*]] = phi i32 [ [[CONV154]], [[COND_TRUE153]] ], [ [[CONV156]], [[COND_FALSE155]] ]
+// SIMD-ONLY0-NEXT:    [[CONV159:%.*]] = trunc i32 [[COND158]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV159]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP75:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV160:%.*]] = zext i8 [[TMP75]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP76:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV161:%.*]] = zext i8 [[TMP76]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP162:%.*]] = icmp sgt i32 [[CONV160]], [[CONV161]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP162]], label [[COND_TRUE164:%.*]], label [[COND_FALSE166:%.*]]
+// SIMD-ONLY0:       cond.true164:
+// SIMD-ONLY0-NEXT:    [[TMP77:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV165:%.*]] = zext i8 [[TMP77]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END168:%.*]]
+// SIMD-ONLY0:       cond.false166:
+// SIMD-ONLY0-NEXT:    [[TMP78:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV167:%.*]] = zext i8 [[TMP78]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END168]]
+// SIMD-ONLY0:       cond.end168:
+// SIMD-ONLY0-NEXT:    [[COND169:%.*]] = phi i32 [ [[CONV165]], [[COND_TRUE164]] ], [ [[CONV167]], [[COND_FALSE166]] ]
+// SIMD-ONLY0-NEXT:    [[CONV170:%.*]] = trunc i32 [[COND169]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV170]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP79:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP79]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP80:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV171:%.*]] = zext i8 [[TMP80]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP81:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV172:%.*]] = zext i8 [[TMP81]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP173:%.*]] = icmp slt i32 [[CONV171]], [[CONV172]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP173]], label [[COND_TRUE175:%.*]], label [[COND_FALSE177:%.*]]
+// SIMD-ONLY0:       cond.true175:
+// SIMD-ONLY0-NEXT:    [[TMP82:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV176:%.*]] = zext i8 [[TMP82]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END179:%.*]]
+// SIMD-ONLY0:       cond.false177:
+// SIMD-ONLY0-NEXT:    [[TMP83:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV178:%.*]] = zext i8 [[TMP83]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END179]]
+// SIMD-ONLY0:       cond.end179:
+// SIMD-ONLY0-NEXT:    [[COND180:%.*]] = phi i32 [ [[CONV176]], [[COND_TRUE175]] ], [ [[CONV178]], [[COND_FALSE177]] ]
+// SIMD-ONLY0-NEXT:    [[CONV181:%.*]] = trunc i32 [[COND180]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV181]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP84:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP84]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP85:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV182:%.*]] = zext i8 [[TMP85]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP86:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV183:%.*]] = zext i8 [[TMP86]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP184:%.*]] = icmp eq i32 [[CONV182]], [[CONV183]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP184]], label [[COND_TRUE186:%.*]], label [[COND_FALSE188:%.*]]
+// SIMD-ONLY0:       cond.true186:
+// SIMD-ONLY0-NEXT:    [[TMP87:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV187:%.*]] = zext i8 [[TMP87]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END190:%.*]]
+// SIMD-ONLY0:       cond.false188:
+// SIMD-ONLY0-NEXT:    [[TMP88:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV189:%.*]] = zext i8 [[TMP88]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END190]]
+// SIMD-ONLY0:       cond.end190:
+// SIMD-ONLY0-NEXT:    [[COND191:%.*]] = phi i32 [ [[CONV187]], [[COND_TRUE186]] ], [ [[CONV189]], [[COND_FALSE188]] ]
+// SIMD-ONLY0-NEXT:    [[CONV192:%.*]] = trunc i32 [[COND191]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV192]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP89:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP89]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP90:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP90]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP91:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV193:%.*]] = zext i8 [[TMP91]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP92:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV194:%.*]] = zext i8 [[TMP92]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP195:%.*]] = icmp sgt i32 [[CONV193]], [[CONV194]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP195]], label [[COND_TRUE197:%.*]], label [[COND_FALSE199:%.*]]
+// SIMD-ONLY0:       cond.true197:
+// SIMD-ONLY0-NEXT:    [[TMP93:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV198:%.*]] = zext i8 [[TMP93]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END201:%.*]]
+// SIMD-ONLY0:       cond.false199:
+// SIMD-ONLY0-NEXT:    [[TMP94:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV200:%.*]] = zext i8 [[TMP94]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END201]]
+// SIMD-ONLY0:       cond.end201:
+// SIMD-ONLY0-NEXT:    [[COND202:%.*]] = phi i32 [ [[CONV198]], [[COND_TRUE197]] ], [ [[CONV200]], [[COND_FALSE199]] ]
+// SIMD-ONLY0-NEXT:    [[CONV203:%.*]] = trunc i32 [[COND202]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV203]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP95:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP95]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP96:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV204:%.*]] = zext i8 [[TMP96]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP97:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV205:%.*]] = zext i8 [[TMP97]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP206:%.*]] = icmp slt i32 [[CONV204]], [[CONV205]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP206]], label [[COND_TRUE208:%.*]], label [[COND_FALSE210:%.*]]
+// SIMD-ONLY0:       cond.true208:
+// SIMD-ONLY0-NEXT:    [[TMP98:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV209:%.*]] = zext i8 [[TMP98]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END212:%.*]]
+// SIMD-ONLY0:       cond.false210:
+// SIMD-ONLY0-NEXT:    [[TMP99:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV211:%.*]] = zext i8 [[TMP99]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END212]]
+// SIMD-ONLY0:       cond.end212:
+// SIMD-ONLY0-NEXT:    [[COND213:%.*]] = phi i32 [ [[CONV209]], [[COND_TRUE208]] ], [ [[CONV211]], [[COND_FALSE210]] ]
+// SIMD-ONLY0-NEXT:    [[CONV214:%.*]] = trunc i32 [[COND213]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV214]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP100:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP100]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP101:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV215:%.*]] = zext i8 [[TMP101]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP102:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV216:%.*]] = zext i8 [[TMP102]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP217:%.*]] = icmp eq i32 [[CONV215]], [[CONV216]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP217]], label [[COND_TRUE219:%.*]], label [[COND_FALSE221:%.*]]
+// SIMD-ONLY0:       cond.true219:
+// SIMD-ONLY0-NEXT:    [[TMP103:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV220:%.*]] = zext i8 [[TMP103]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END223:%.*]]
+// SIMD-ONLY0:       cond.false221:
+// SIMD-ONLY0-NEXT:    [[TMP104:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV222:%.*]] = zext i8 [[TMP104]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END223]]
+// SIMD-ONLY0:       cond.end223:
+// SIMD-ONLY0-NEXT:    [[COND224:%.*]] = phi i32 [ [[CONV220]], [[COND_TRUE219]] ], [ [[CONV222]], [[COND_FALSE221]] ]
+// SIMD-ONLY0-NEXT:    [[CONV225:%.*]] = trunc i32 [[COND224]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV225]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP105:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV226:%.*]] = zext i8 [[TMP105]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP106:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV227:%.*]] = zext i8 [[TMP106]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP228:%.*]] = icmp sgt i32 [[CONV226]], [[CONV227]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP228]], label [[COND_TRUE230:%.*]], label [[COND_FALSE232:%.*]]
+// SIMD-ONLY0:       cond.true230:
+// SIMD-ONLY0-NEXT:    [[TMP107:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV231:%.*]] = zext i8 [[TMP107]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END234:%.*]]
+// SIMD-ONLY0:       cond.false232:
+// SIMD-ONLY0-NEXT:    [[TMP108:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV233:%.*]] = zext i8 [[TMP108]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END234]]
+// SIMD-ONLY0:       cond.end234:
+// SIMD-ONLY0-NEXT:    [[COND235:%.*]] = phi i32 [ [[CONV231]], [[COND_TRUE230]] ], [ [[CONV233]], [[COND_FALSE232]] ]
+// SIMD-ONLY0-NEXT:    [[CONV236:%.*]] = trunc i32 [[COND235]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV236]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP109:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP109]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP110:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV237:%.*]] = zext i8 [[TMP110]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP111:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV238:%.*]] = zext i8 [[TMP111]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP239:%.*]] = icmp slt i32 [[CONV237]], [[CONV238]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP239]], label [[COND_TRUE241:%.*]], label [[COND_FALSE243:%.*]]
+// SIMD-ONLY0:       cond.true241:
+// SIMD-ONLY0-NEXT:    [[TMP112:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV242:%.*]] = zext i8 [[TMP112]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END245:%.*]]
+// SIMD-ONLY0:       cond.false243:
+// SIMD-ONLY0-NEXT:    [[TMP113:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV244:%.*]] = zext i8 [[TMP113]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END245]]
+// SIMD-ONLY0:       cond.end245:
+// SIMD-ONLY0-NEXT:    [[COND246:%.*]] = phi i32 [ [[CONV242]], [[COND_TRUE241]] ], [ [[CONV244]], [[COND_FALSE243]] ]
+// SIMD-ONLY0-NEXT:    [[CONV247:%.*]] = trunc i32 [[COND246]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV247]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP114:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP114]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP115:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV248:%.*]] = zext i8 [[TMP115]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP116:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV249:%.*]] = zext i8 [[TMP116]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP250:%.*]] = icmp eq i32 [[CONV248]], [[CONV249]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP250]], label [[COND_TRUE252:%.*]], label [[COND_FALSE254:%.*]]
+// SIMD-ONLY0:       cond.true252:
+// SIMD-ONLY0-NEXT:    [[TMP117:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV253:%.*]] = zext i8 [[TMP117]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END256:%.*]]
+// SIMD-ONLY0:       cond.false254:
+// SIMD-ONLY0-NEXT:    [[TMP118:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV255:%.*]] = zext i8 [[TMP118]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END256]]
+// SIMD-ONLY0:       cond.end256:
+// SIMD-ONLY0-NEXT:    [[COND257:%.*]] = phi i32 [ [[CONV253]], [[COND_TRUE252]] ], [ [[CONV255]], [[COND_FALSE254]] ]
+// SIMD-ONLY0-NEXT:    [[CONV258:%.*]] = trunc i32 [[COND257]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV258]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP119:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP119]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP120:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP120]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP121:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV259:%.*]] = zext i8 [[TMP121]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP122:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV260:%.*]] = zext i8 [[TMP122]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP261:%.*]] = icmp sgt i32 [[CONV259]], [[CONV260]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP261]], label [[COND_TRUE263:%.*]], label [[COND_FALSE265:%.*]]
+// SIMD-ONLY0:       cond.true263:
+// SIMD-ONLY0-NEXT:    [[TMP123:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV264:%.*]] = zext i8 [[TMP123]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END267:%.*]]
+// SIMD-ONLY0:       cond.false265:
+// SIMD-ONLY0-NEXT:    [[TMP124:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV266:%.*]] = zext i8 [[TMP124]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END267]]
+// SIMD-ONLY0:       cond.end267:
+// SIMD-ONLY0-NEXT:    [[COND268:%.*]] = phi i32 [ [[CONV264]], [[COND_TRUE263]] ], [ [[CONV266]], [[COND_FALSE265]] ]
+// SIMD-ONLY0-NEXT:    [[CONV269:%.*]] = trunc i32 [[COND268]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV269]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP125:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP125]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP126:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV270:%.*]] = zext i8 [[TMP126]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP127:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV271:%.*]] = zext i8 [[TMP127]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP272:%.*]] = icmp slt i32 [[CONV270]], [[CONV271]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP272]], label [[COND_TRUE274:%.*]], label [[COND_FALSE276:%.*]]
+// SIMD-ONLY0:       cond.true274:
+// SIMD-ONLY0-NEXT:    [[TMP128:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV275:%.*]] = zext i8 [[TMP128]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END278:%.*]]
+// SIMD-ONLY0:       cond.false276:
+// SIMD-ONLY0-NEXT:    [[TMP129:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV277:%.*]] = zext i8 [[TMP129]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END278]]
+// SIMD-ONLY0:       cond.end278:
+// SIMD-ONLY0-NEXT:    [[COND279:%.*]] = phi i32 [ [[CONV275]], [[COND_TRUE274]] ], [ [[CONV277]], [[COND_FALSE276]] ]
+// SIMD-ONLY0-NEXT:    [[CONV280:%.*]] = trunc i32 [[COND279]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV280]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP130:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP130]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP131:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV281:%.*]] = zext i8 [[TMP131]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP132:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV282:%.*]] = zext i8 [[TMP132]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP283:%.*]] = icmp eq i32 [[CONV281]], [[CONV282]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP283]], label [[COND_TRUE285:%.*]], label [[COND_FALSE287:%.*]]
+// SIMD-ONLY0:       cond.true285:
+// SIMD-ONLY0-NEXT:    [[TMP133:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV286:%.*]] = zext i8 [[TMP133]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END289:%.*]]
+// SIMD-ONLY0:       cond.false287:
+// SIMD-ONLY0-NEXT:    [[TMP134:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV288:%.*]] = zext i8 [[TMP134]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END289]]
+// SIMD-ONLY0:       cond.end289:
+// SIMD-ONLY0-NEXT:    [[COND290:%.*]] = phi i32 [ [[CONV286]], [[COND_TRUE285]] ], [ [[CONV288]], [[COND_FALSE287]] ]
+// SIMD-ONLY0-NEXT:    [[CONV291:%.*]] = trunc i32 [[COND290]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV291]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP135:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV292:%.*]] = zext i8 [[TMP135]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP136:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV293:%.*]] = zext i8 [[TMP136]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP294:%.*]] = icmp sgt i32 [[CONV292]], [[CONV293]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP294]], label [[COND_TRUE296:%.*]], label [[COND_FALSE298:%.*]]
+// SIMD-ONLY0:       cond.true296:
+// SIMD-ONLY0-NEXT:    [[TMP137:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV297:%.*]] = zext i8 [[TMP137]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END300:%.*]]
+// SIMD-ONLY0:       cond.false298:
+// SIMD-ONLY0-NEXT:    [[TMP138:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV299:%.*]] = zext i8 [[TMP138]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END300]]
+// SIMD-ONLY0:       cond.end300:
+// SIMD-ONLY0-NEXT:    [[COND301:%.*]] = phi i32 [ [[CONV297]], [[COND_TRUE296]] ], [ [[CONV299]], [[COND_FALSE298]] ]
+// SIMD-ONLY0-NEXT:    [[CONV302:%.*]] = trunc i32 [[COND301]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV302]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP139:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP139]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP140:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV303:%.*]] = zext i8 [[TMP140]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP141:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV304:%.*]] = zext i8 [[TMP141]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP305:%.*]] = icmp slt i32 [[CONV303]], [[CONV304]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP305]], label [[COND_TRUE307:%.*]], label [[COND_FALSE309:%.*]]
+// SIMD-ONLY0:       cond.true307:
+// SIMD-ONLY0-NEXT:    [[TMP142:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV308:%.*]] = zext i8 [[TMP142]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END311:%.*]]
+// SIMD-ONLY0:       cond.false309:
+// SIMD-ONLY0-NEXT:    [[TMP143:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV310:%.*]] = zext i8 [[TMP143]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END311]]
+// SIMD-ONLY0:       cond.end311:
+// SIMD-ONLY0-NEXT:    [[COND312:%.*]] = phi i32 [ [[CONV308]], [[COND_TRUE307]] ], [ [[CONV310]], [[COND_FALSE309]] ]
+// SIMD-ONLY0-NEXT:    [[CONV313:%.*]] = trunc i32 [[COND312]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV313]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP144:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP144]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP145:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV314:%.*]] = zext i8 [[TMP145]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP146:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV315:%.*]] = zext i8 [[TMP146]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP316:%.*]] = icmp eq i32 [[CONV314]], [[CONV315]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP316]], label [[COND_TRUE318:%.*]], label [[COND_FALSE320:%.*]]
+// SIMD-ONLY0:       cond.true318:
+// SIMD-ONLY0-NEXT:    [[TMP147:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV319:%.*]] = zext i8 [[TMP147]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END322:%.*]]
+// SIMD-ONLY0:       cond.false320:
+// SIMD-ONLY0-NEXT:    [[TMP148:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV321:%.*]] = zext i8 [[TMP148]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END322]]
+// SIMD-ONLY0:       cond.end322:
+// SIMD-ONLY0-NEXT:    [[COND323:%.*]] = phi i32 [ [[CONV319]], [[COND_TRUE318]] ], [ [[CONV321]], [[COND_FALSE320]] ]
+// SIMD-ONLY0-NEXT:    [[CONV324:%.*]] = trunc i32 [[COND323]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV324]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP149:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP149]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP150:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP150]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP151:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV325:%.*]] = zext i8 [[TMP151]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP152:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV326:%.*]] = zext i8 [[TMP152]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP327:%.*]] = icmp sgt i32 [[CONV325]], [[CONV326]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP327]], label [[COND_TRUE329:%.*]], label [[COND_FALSE331:%.*]]
+// SIMD-ONLY0:       cond.true329:
+// SIMD-ONLY0-NEXT:    [[TMP153:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV330:%.*]] = zext i8 [[TMP153]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END333:%.*]]
+// SIMD-ONLY0:       cond.false331:
+// SIMD-ONLY0-NEXT:    [[TMP154:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV332:%.*]] = zext i8 [[TMP154]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END333]]
+// SIMD-ONLY0:       cond.end333:
+// SIMD-ONLY0-NEXT:    [[COND334:%.*]] = phi i32 [ [[CONV330]], [[COND_TRUE329]] ], [ [[CONV332]], [[COND_FALSE331]] ]
+// SIMD-ONLY0-NEXT:    [[CONV335:%.*]] = trunc i32 [[COND334]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV335]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP155:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP155]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP156:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV336:%.*]] = zext i8 [[TMP156]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP157:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV337:%.*]] = zext i8 [[TMP157]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP338:%.*]] = icmp slt i32 [[CONV336]], [[CONV337]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP338]], label [[COND_TRUE340:%.*]], label [[COND_FALSE342:%.*]]
+// SIMD-ONLY0:       cond.true340:
+// SIMD-ONLY0-NEXT:    [[TMP158:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV341:%.*]] = zext i8 [[TMP158]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END344:%.*]]
+// SIMD-ONLY0:       cond.false342:
+// SIMD-ONLY0-NEXT:    [[TMP159:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV343:%.*]] = zext i8 [[TMP159]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END344]]
+// SIMD-ONLY0:       cond.end344:
+// SIMD-ONLY0-NEXT:    [[COND345:%.*]] = phi i32 [ [[CONV341]], [[COND_TRUE340]] ], [ [[CONV343]], [[COND_FALSE342]] ]
+// SIMD-ONLY0-NEXT:    [[CONV346:%.*]] = trunc i32 [[COND345]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV346]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP160:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP160]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP161:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV347:%.*]] = zext i8 [[TMP161]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP162:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV348:%.*]] = zext i8 [[TMP162]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP349:%.*]] = icmp eq i32 [[CONV347]], [[CONV348]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP349]], label [[COND_TRUE351:%.*]], label [[COND_FALSE353:%.*]]
+// SIMD-ONLY0:       cond.true351:
+// SIMD-ONLY0-NEXT:    [[TMP163:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV352:%.*]] = zext i8 [[TMP163]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END355:%.*]]
+// SIMD-ONLY0:       cond.false353:
+// SIMD-ONLY0-NEXT:    [[TMP164:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV354:%.*]] = zext i8 [[TMP164]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END355]]
+// SIMD-ONLY0:       cond.end355:
+// SIMD-ONLY0-NEXT:    [[COND356:%.*]] = phi i32 [ [[CONV352]], [[COND_TRUE351]] ], [ [[CONV354]], [[COND_FALSE353]] ]
+// SIMD-ONLY0-NEXT:    [[CONV357:%.*]] = trunc i32 [[COND356]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV357]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP165:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV358:%.*]] = zext i8 [[TMP165]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP166:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV359:%.*]] = zext i8 [[TMP166]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP360:%.*]] = icmp sgt i32 [[CONV358]], [[CONV359]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP360]], label [[COND_TRUE362:%.*]], label [[COND_FALSE364:%.*]]
+// SIMD-ONLY0:       cond.true362:
+// SIMD-ONLY0-NEXT:    [[TMP167:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV363:%.*]] = zext i8 [[TMP167]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END366:%.*]]
+// SIMD-ONLY0:       cond.false364:
+// SIMD-ONLY0-NEXT:    [[TMP168:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV365:%.*]] = zext i8 [[TMP168]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END366]]
+// SIMD-ONLY0:       cond.end366:
+// SIMD-ONLY0-NEXT:    [[COND367:%.*]] = phi i32 [ [[CONV363]], [[COND_TRUE362]] ], [ [[CONV365]], [[COND_FALSE364]] ]
+// SIMD-ONLY0-NEXT:    [[CONV368:%.*]] = trunc i32 [[COND367]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV368]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP169:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP169]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP170:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV369:%.*]] = zext i8 [[TMP170]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP171:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV370:%.*]] = zext i8 [[TMP171]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP371:%.*]] = icmp slt i32 [[CONV369]], [[CONV370]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP371]], label [[COND_TRUE373:%.*]], label [[COND_FALSE375:%.*]]
+// SIMD-ONLY0:       cond.true373:
+// SIMD-ONLY0-NEXT:    [[TMP172:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV374:%.*]] = zext i8 [[TMP172]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END377:%.*]]
+// SIMD-ONLY0:       cond.false375:
+// SIMD-ONLY0-NEXT:    [[TMP173:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV376:%.*]] = zext i8 [[TMP173]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END377]]
+// SIMD-ONLY0:       cond.end377:
+// SIMD-ONLY0-NEXT:    [[COND378:%.*]] = phi i32 [ [[CONV374]], [[COND_TRUE373]] ], [ [[CONV376]], [[COND_FALSE375]] ]
+// SIMD-ONLY0-NEXT:    [[CONV379:%.*]] = trunc i32 [[COND378]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV379]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP174:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP174]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP175:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV380:%.*]] = zext i8 [[TMP175]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP176:%.*]] = load i8, ptr [[UCE]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV381:%.*]] = zext i8 [[TMP176]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP382:%.*]] = icmp eq i32 [[CONV380]], [[CONV381]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP382]], label [[COND_TRUE384:%.*]], label [[COND_FALSE386:%.*]]
+// SIMD-ONLY0:       cond.true384:
+// SIMD-ONLY0-NEXT:    [[TMP177:%.*]] = load i8, ptr [[UCD]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV385:%.*]] = zext i8 [[TMP177]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END388:%.*]]
+// SIMD-ONLY0:       cond.false386:
+// SIMD-ONLY0-NEXT:    [[TMP178:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[CONV387:%.*]] = zext i8 [[TMP178]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END388]]
+// SIMD-ONLY0:       cond.end388:
+// SIMD-ONLY0-NEXT:    [[COND389:%.*]] = phi i32 [ [[CONV385]], [[COND_TRUE384]] ], [ [[CONV387]], [[COND_FALSE386]] ]
+// SIMD-ONLY0-NEXT:    [[CONV390:%.*]] = trunc i32 [[COND389]] to i8
+// SIMD-ONLY0-NEXT:    store i8 [[CONV390]], ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP179:%.*]] = load i8, ptr [[UCX]], align 1
+// SIMD-ONLY0-NEXT:    store i8 [[TMP179]], ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    [[TMP180:%.*]] = load i8, ptr [[UCV]], align 1
+// SIMD-ONLY0-NEXT:    ret i8 [[TMP180]]
+//
+//
+// SIMD-ONLY0-LABEL: @sxevd(
+// SIMD-ONLY0-NEXT:  entry:
+// SIMD-ONLY0-NEXT:    [[SX:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[SV:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[SE:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[SD:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[TMP0:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP0]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV:%.*]] = sext i16 [[TMP1]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP2:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1:%.*]] = sext i16 [[TMP2]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[CONV]], [[CONV1]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]]
+// SIMD-ONLY0:       cond.true:
+// SIMD-ONLY0-NEXT:    [[TMP3:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV3:%.*]] = sext i16 [[TMP3]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END:%.*]]
+// SIMD-ONLY0:       cond.false:
+// SIMD-ONLY0-NEXT:    [[TMP4:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV4:%.*]] = sext i16 [[TMP4]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END]]
+// SIMD-ONLY0:       cond.end:
+// SIMD-ONLY0-NEXT:    [[COND:%.*]] = phi i32 [ [[CONV3]], [[COND_TRUE]] ], [ [[CONV4]], [[COND_FALSE]] ]
+// SIMD-ONLY0-NEXT:    [[CONV5:%.*]] = trunc i32 [[COND]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV5]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP5:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP5]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP6:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV6:%.*]] = sext i16 [[TMP6]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP7:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV7:%.*]] = sext i16 [[TMP7]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP8:%.*]] = icmp slt i32 [[CONV6]], [[CONV7]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP8]], label [[COND_TRUE10:%.*]], label [[COND_FALSE12:%.*]]
+// SIMD-ONLY0:       cond.true10:
+// SIMD-ONLY0-NEXT:    [[TMP8:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV11:%.*]] = sext i16 [[TMP8]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END14:%.*]]
+// SIMD-ONLY0:       cond.false12:
+// SIMD-ONLY0-NEXT:    [[TMP9:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV13:%.*]] = sext i16 [[TMP9]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END14]]
+// SIMD-ONLY0:       cond.end14:
+// SIMD-ONLY0-NEXT:    [[COND15:%.*]] = phi i32 [ [[CONV11]], [[COND_TRUE10]] ], [ [[CONV13]], [[COND_FALSE12]] ]
+// SIMD-ONLY0-NEXT:    [[CONV16:%.*]] = trunc i32 [[COND15]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV16]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP10:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP10]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP11:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV17:%.*]] = sext i16 [[TMP11]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP12:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV18:%.*]] = sext i16 [[TMP12]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP19:%.*]] = icmp eq i32 [[CONV17]], [[CONV18]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP19]], label [[COND_TRUE21:%.*]], label [[COND_FALSE23:%.*]]
+// SIMD-ONLY0:       cond.true21:
+// SIMD-ONLY0-NEXT:    [[TMP13:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV22:%.*]] = sext i16 [[TMP13]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END25:%.*]]
+// SIMD-ONLY0:       cond.false23:
+// SIMD-ONLY0-NEXT:    [[TMP14:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV24:%.*]] = sext i16 [[TMP14]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END25]]
+// SIMD-ONLY0:       cond.end25:
+// SIMD-ONLY0-NEXT:    [[COND26:%.*]] = phi i32 [ [[CONV22]], [[COND_TRUE21]] ], [ [[CONV24]], [[COND_FALSE23]] ]
+// SIMD-ONLY0-NEXT:    [[CONV27:%.*]] = trunc i32 [[COND26]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV27]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP15:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV28:%.*]] = sext i16 [[TMP15]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP16:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV29:%.*]] = sext i16 [[TMP16]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP30:%.*]] = icmp sgt i32 [[CONV28]], [[CONV29]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP30]], label [[COND_TRUE32:%.*]], label [[COND_FALSE34:%.*]]
+// SIMD-ONLY0:       cond.true32:
+// SIMD-ONLY0-NEXT:    [[TMP17:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV33:%.*]] = sext i16 [[TMP17]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END36:%.*]]
+// SIMD-ONLY0:       cond.false34:
+// SIMD-ONLY0-NEXT:    [[TMP18:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV35:%.*]] = sext i16 [[TMP18]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END36]]
+// SIMD-ONLY0:       cond.end36:
+// SIMD-ONLY0-NEXT:    [[COND37:%.*]] = phi i32 [ [[CONV33]], [[COND_TRUE32]] ], [ [[CONV35]], [[COND_FALSE34]] ]
+// SIMD-ONLY0-NEXT:    [[CONV38:%.*]] = trunc i32 [[COND37]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV38]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP19:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP19]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP20:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV39:%.*]] = sext i16 [[TMP20]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP21:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV40:%.*]] = sext i16 [[TMP21]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP41:%.*]] = icmp slt i32 [[CONV39]], [[CONV40]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP41]], label [[COND_TRUE43:%.*]], label [[COND_FALSE45:%.*]]
+// SIMD-ONLY0:       cond.true43:
+// SIMD-ONLY0-NEXT:    [[TMP22:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV44:%.*]] = sext i16 [[TMP22]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END47:%.*]]
+// SIMD-ONLY0:       cond.false45:
+// SIMD-ONLY0-NEXT:    [[TMP23:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV46:%.*]] = sext i16 [[TMP23]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END47]]
+// SIMD-ONLY0:       cond.end47:
+// SIMD-ONLY0-NEXT:    [[COND48:%.*]] = phi i32 [ [[CONV44]], [[COND_TRUE43]] ], [ [[CONV46]], [[COND_FALSE45]] ]
+// SIMD-ONLY0-NEXT:    [[CONV49:%.*]] = trunc i32 [[COND48]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV49]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP24:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP24]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP25:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV50:%.*]] = sext i16 [[TMP25]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP26:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV51:%.*]] = sext i16 [[TMP26]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP52:%.*]] = icmp eq i32 [[CONV50]], [[CONV51]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP52]], label [[COND_TRUE54:%.*]], label [[COND_FALSE56:%.*]]
+// SIMD-ONLY0:       cond.true54:
+// SIMD-ONLY0-NEXT:    [[TMP27:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV55:%.*]] = sext i16 [[TMP27]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END58:%.*]]
+// SIMD-ONLY0:       cond.false56:
+// SIMD-ONLY0-NEXT:    [[TMP28:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV57:%.*]] = sext i16 [[TMP28]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END58]]
+// SIMD-ONLY0:       cond.end58:
+// SIMD-ONLY0-NEXT:    [[COND59:%.*]] = phi i32 [ [[CONV55]], [[COND_TRUE54]] ], [ [[CONV57]], [[COND_FALSE56]] ]
+// SIMD-ONLY0-NEXT:    [[CONV60:%.*]] = trunc i32 [[COND59]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV60]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP29:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP29]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP30:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP30]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP31:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV61:%.*]] = sext i16 [[TMP31]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP32:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV62:%.*]] = sext i16 [[TMP32]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP63:%.*]] = icmp sgt i32 [[CONV61]], [[CONV62]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP63]], label [[COND_TRUE65:%.*]], label [[COND_FALSE67:%.*]]
+// SIMD-ONLY0:       cond.true65:
+// SIMD-ONLY0-NEXT:    [[TMP33:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV66:%.*]] = sext i16 [[TMP33]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END69:%.*]]
+// SIMD-ONLY0:       cond.false67:
+// SIMD-ONLY0-NEXT:    [[TMP34:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV68:%.*]] = sext i16 [[TMP34]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END69]]
+// SIMD-ONLY0:       cond.end69:
+// SIMD-ONLY0-NEXT:    [[COND70:%.*]] = phi i32 [ [[CONV66]], [[COND_TRUE65]] ], [ [[CONV68]], [[COND_FALSE67]] ]
+// SIMD-ONLY0-NEXT:    [[CONV71:%.*]] = trunc i32 [[COND70]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV71]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP35:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP35]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP36:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV72:%.*]] = sext i16 [[TMP36]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP37:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV73:%.*]] = sext i16 [[TMP37]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP74:%.*]] = icmp slt i32 [[CONV72]], [[CONV73]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP74]], label [[COND_TRUE76:%.*]], label [[COND_FALSE78:%.*]]
+// SIMD-ONLY0:       cond.true76:
+// SIMD-ONLY0-NEXT:    [[TMP38:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV77:%.*]] = sext i16 [[TMP38]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END80:%.*]]
+// SIMD-ONLY0:       cond.false78:
+// SIMD-ONLY0-NEXT:    [[TMP39:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV79:%.*]] = sext i16 [[TMP39]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END80]]
+// SIMD-ONLY0:       cond.end80:
+// SIMD-ONLY0-NEXT:    [[COND81:%.*]] = phi i32 [ [[CONV77]], [[COND_TRUE76]] ], [ [[CONV79]], [[COND_FALSE78]] ]
+// SIMD-ONLY0-NEXT:    [[CONV82:%.*]] = trunc i32 [[COND81]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV82]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP40:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP40]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP41:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV83:%.*]] = sext i16 [[TMP41]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP42:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV84:%.*]] = sext i16 [[TMP42]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP85:%.*]] = icmp eq i32 [[CONV83]], [[CONV84]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP85]], label [[COND_TRUE87:%.*]], label [[COND_FALSE89:%.*]]
+// SIMD-ONLY0:       cond.true87:
+// SIMD-ONLY0-NEXT:    [[TMP43:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV88:%.*]] = sext i16 [[TMP43]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END91:%.*]]
+// SIMD-ONLY0:       cond.false89:
+// SIMD-ONLY0-NEXT:    [[TMP44:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV90:%.*]] = sext i16 [[TMP44]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END91]]
+// SIMD-ONLY0:       cond.end91:
+// SIMD-ONLY0-NEXT:    [[COND92:%.*]] = phi i32 [ [[CONV88]], [[COND_TRUE87]] ], [ [[CONV90]], [[COND_FALSE89]] ]
+// SIMD-ONLY0-NEXT:    [[CONV93:%.*]] = trunc i32 [[COND92]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV93]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP45:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV94:%.*]] = sext i16 [[TMP45]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP46:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV95:%.*]] = sext i16 [[TMP46]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP96:%.*]] = icmp sgt i32 [[CONV94]], [[CONV95]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP96]], label [[COND_TRUE98:%.*]], label [[COND_FALSE100:%.*]]
+// SIMD-ONLY0:       cond.true98:
+// SIMD-ONLY0-NEXT:    [[TMP47:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV99:%.*]] = sext i16 [[TMP47]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END102:%.*]]
+// SIMD-ONLY0:       cond.false100:
+// SIMD-ONLY0-NEXT:    [[TMP48:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV101:%.*]] = sext i16 [[TMP48]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END102]]
+// SIMD-ONLY0:       cond.end102:
+// SIMD-ONLY0-NEXT:    [[COND103:%.*]] = phi i32 [ [[CONV99]], [[COND_TRUE98]] ], [ [[CONV101]], [[COND_FALSE100]] ]
+// SIMD-ONLY0-NEXT:    [[CONV104:%.*]] = trunc i32 [[COND103]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV104]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP49:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP49]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP50:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV105:%.*]] = sext i16 [[TMP50]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP51:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV106:%.*]] = sext i16 [[TMP51]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP107:%.*]] = icmp slt i32 [[CONV105]], [[CONV106]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP107]], label [[COND_TRUE109:%.*]], label [[COND_FALSE111:%.*]]
+// SIMD-ONLY0:       cond.true109:
+// SIMD-ONLY0-NEXT:    [[TMP52:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV110:%.*]] = sext i16 [[TMP52]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END113:%.*]]
+// SIMD-ONLY0:       cond.false111:
+// SIMD-ONLY0-NEXT:    [[TMP53:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV112:%.*]] = sext i16 [[TMP53]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END113]]
+// SIMD-ONLY0:       cond.end113:
+// SIMD-ONLY0-NEXT:    [[COND114:%.*]] = phi i32 [ [[CONV110]], [[COND_TRUE109]] ], [ [[CONV112]], [[COND_FALSE111]] ]
+// SIMD-ONLY0-NEXT:    [[CONV115:%.*]] = trunc i32 [[COND114]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV115]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP54:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP54]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP55:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV116:%.*]] = sext i16 [[TMP55]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP56:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV117:%.*]] = sext i16 [[TMP56]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP118:%.*]] = icmp eq i32 [[CONV116]], [[CONV117]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP118]], label [[COND_TRUE120:%.*]], label [[COND_FALSE122:%.*]]
+// SIMD-ONLY0:       cond.true120:
+// SIMD-ONLY0-NEXT:    [[TMP57:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV121:%.*]] = sext i16 [[TMP57]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END124:%.*]]
+// SIMD-ONLY0:       cond.false122:
+// SIMD-ONLY0-NEXT:    [[TMP58:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV123:%.*]] = sext i16 [[TMP58]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END124]]
+// SIMD-ONLY0:       cond.end124:
+// SIMD-ONLY0-NEXT:    [[COND125:%.*]] = phi i32 [ [[CONV121]], [[COND_TRUE120]] ], [ [[CONV123]], [[COND_FALSE122]] ]
+// SIMD-ONLY0-NEXT:    [[CONV126:%.*]] = trunc i32 [[COND125]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV126]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP59:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP59]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP60:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP60]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP61:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV127:%.*]] = sext i16 [[TMP61]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP62:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV128:%.*]] = sext i16 [[TMP62]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP129:%.*]] = icmp sgt i32 [[CONV127]], [[CONV128]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP129]], label [[COND_TRUE131:%.*]], label [[COND_FALSE133:%.*]]
+// SIMD-ONLY0:       cond.true131:
+// SIMD-ONLY0-NEXT:    [[TMP63:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV132:%.*]] = sext i16 [[TMP63]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END135:%.*]]
+// SIMD-ONLY0:       cond.false133:
+// SIMD-ONLY0-NEXT:    [[TMP64:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV134:%.*]] = sext i16 [[TMP64]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END135]]
+// SIMD-ONLY0:       cond.end135:
+// SIMD-ONLY0-NEXT:    [[COND136:%.*]] = phi i32 [ [[CONV132]], [[COND_TRUE131]] ], [ [[CONV134]], [[COND_FALSE133]] ]
+// SIMD-ONLY0-NEXT:    [[CONV137:%.*]] = trunc i32 [[COND136]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV137]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP65:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP65]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP66:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV138:%.*]] = sext i16 [[TMP66]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP67:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV139:%.*]] = sext i16 [[TMP67]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP140:%.*]] = icmp slt i32 [[CONV138]], [[CONV139]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP140]], label [[COND_TRUE142:%.*]], label [[COND_FALSE144:%.*]]
+// SIMD-ONLY0:       cond.true142:
+// SIMD-ONLY0-NEXT:    [[TMP68:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV143:%.*]] = sext i16 [[TMP68]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END146:%.*]]
+// SIMD-ONLY0:       cond.false144:
+// SIMD-ONLY0-NEXT:    [[TMP69:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV145:%.*]] = sext i16 [[TMP69]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END146]]
+// SIMD-ONLY0:       cond.end146:
+// SIMD-ONLY0-NEXT:    [[COND147:%.*]] = phi i32 [ [[CONV143]], [[COND_TRUE142]] ], [ [[CONV145]], [[COND_FALSE144]] ]
+// SIMD-ONLY0-NEXT:    [[CONV148:%.*]] = trunc i32 [[COND147]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV148]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP70:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP70]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP71:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV149:%.*]] = sext i16 [[TMP71]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP72:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV150:%.*]] = sext i16 [[TMP72]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP151:%.*]] = icmp eq i32 [[CONV149]], [[CONV150]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP151]], label [[COND_TRUE153:%.*]], label [[COND_FALSE155:%.*]]
+// SIMD-ONLY0:       cond.true153:
+// SIMD-ONLY0-NEXT:    [[TMP73:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV154:%.*]] = sext i16 [[TMP73]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END157:%.*]]
+// SIMD-ONLY0:       cond.false155:
+// SIMD-ONLY0-NEXT:    [[TMP74:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV156:%.*]] = sext i16 [[TMP74]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END157]]
+// SIMD-ONLY0:       cond.end157:
+// SIMD-ONLY0-NEXT:    [[COND158:%.*]] = phi i32 [ [[CONV154]], [[COND_TRUE153]] ], [ [[CONV156]], [[COND_FALSE155]] ]
+// SIMD-ONLY0-NEXT:    [[CONV159:%.*]] = trunc i32 [[COND158]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV159]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP75:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV160:%.*]] = sext i16 [[TMP75]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP76:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV161:%.*]] = sext i16 [[TMP76]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP162:%.*]] = icmp sgt i32 [[CONV160]], [[CONV161]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP162]], label [[COND_TRUE164:%.*]], label [[COND_FALSE166:%.*]]
+// SIMD-ONLY0:       cond.true164:
+// SIMD-ONLY0-NEXT:    [[TMP77:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV165:%.*]] = sext i16 [[TMP77]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END168:%.*]]
+// SIMD-ONLY0:       cond.false166:
+// SIMD-ONLY0-NEXT:    [[TMP78:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV167:%.*]] = sext i16 [[TMP78]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END168]]
+// SIMD-ONLY0:       cond.end168:
+// SIMD-ONLY0-NEXT:    [[COND169:%.*]] = phi i32 [ [[CONV165]], [[COND_TRUE164]] ], [ [[CONV167]], [[COND_FALSE166]] ]
+// SIMD-ONLY0-NEXT:    [[CONV170:%.*]] = trunc i32 [[COND169]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV170]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP79:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP79]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP80:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV171:%.*]] = sext i16 [[TMP80]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP81:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV172:%.*]] = sext i16 [[TMP81]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP173:%.*]] = icmp slt i32 [[CONV171]], [[CONV172]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP173]], label [[COND_TRUE175:%.*]], label [[COND_FALSE177:%.*]]
+// SIMD-ONLY0:       cond.true175:
+// SIMD-ONLY0-NEXT:    [[TMP82:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV176:%.*]] = sext i16 [[TMP82]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END179:%.*]]
+// SIMD-ONLY0:       cond.false177:
+// SIMD-ONLY0-NEXT:    [[TMP83:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV178:%.*]] = sext i16 [[TMP83]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END179]]
+// SIMD-ONLY0:       cond.end179:
+// SIMD-ONLY0-NEXT:    [[COND180:%.*]] = phi i32 [ [[CONV176]], [[COND_TRUE175]] ], [ [[CONV178]], [[COND_FALSE177]] ]
+// SIMD-ONLY0-NEXT:    [[CONV181:%.*]] = trunc i32 [[COND180]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV181]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP84:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP84]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP85:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV182:%.*]] = sext i16 [[TMP85]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP86:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV183:%.*]] = sext i16 [[TMP86]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP184:%.*]] = icmp eq i32 [[CONV182]], [[CONV183]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP184]], label [[COND_TRUE186:%.*]], label [[COND_FALSE188:%.*]]
+// SIMD-ONLY0:       cond.true186:
+// SIMD-ONLY0-NEXT:    [[TMP87:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV187:%.*]] = sext i16 [[TMP87]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END190:%.*]]
+// SIMD-ONLY0:       cond.false188:
+// SIMD-ONLY0-NEXT:    [[TMP88:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV189:%.*]] = sext i16 [[TMP88]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END190]]
+// SIMD-ONLY0:       cond.end190:
+// SIMD-ONLY0-NEXT:    [[COND191:%.*]] = phi i32 [ [[CONV187]], [[COND_TRUE186]] ], [ [[CONV189]], [[COND_FALSE188]] ]
+// SIMD-ONLY0-NEXT:    [[CONV192:%.*]] = trunc i32 [[COND191]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV192]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP89:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP89]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP90:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP90]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP91:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV193:%.*]] = sext i16 [[TMP91]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP92:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV194:%.*]] = sext i16 [[TMP92]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP195:%.*]] = icmp sgt i32 [[CONV193]], [[CONV194]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP195]], label [[COND_TRUE197:%.*]], label [[COND_FALSE199:%.*]]
+// SIMD-ONLY0:       cond.true197:
+// SIMD-ONLY0-NEXT:    [[TMP93:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV198:%.*]] = sext i16 [[TMP93]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END201:%.*]]
+// SIMD-ONLY0:       cond.false199:
+// SIMD-ONLY0-NEXT:    [[TMP94:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV200:%.*]] = sext i16 [[TMP94]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END201]]
+// SIMD-ONLY0:       cond.end201:
+// SIMD-ONLY0-NEXT:    [[COND202:%.*]] = phi i32 [ [[CONV198]], [[COND_TRUE197]] ], [ [[CONV200]], [[COND_FALSE199]] ]
+// SIMD-ONLY0-NEXT:    [[CONV203:%.*]] = trunc i32 [[COND202]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV203]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP95:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP95]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP96:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV204:%.*]] = sext i16 [[TMP96]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP97:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV205:%.*]] = sext i16 [[TMP97]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP206:%.*]] = icmp slt i32 [[CONV204]], [[CONV205]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP206]], label [[COND_TRUE208:%.*]], label [[COND_FALSE210:%.*]]
+// SIMD-ONLY0:       cond.true208:
+// SIMD-ONLY0-NEXT:    [[TMP98:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV209:%.*]] = sext i16 [[TMP98]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END212:%.*]]
+// SIMD-ONLY0:       cond.false210:
+// SIMD-ONLY0-NEXT:    [[TMP99:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV211:%.*]] = sext i16 [[TMP99]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END212]]
+// SIMD-ONLY0:       cond.end212:
+// SIMD-ONLY0-NEXT:    [[COND213:%.*]] = phi i32 [ [[CONV209]], [[COND_TRUE208]] ], [ [[CONV211]], [[COND_FALSE210]] ]
+// SIMD-ONLY0-NEXT:    [[CONV214:%.*]] = trunc i32 [[COND213]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV214]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP100:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP100]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP101:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV215:%.*]] = sext i16 [[TMP101]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP102:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV216:%.*]] = sext i16 [[TMP102]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP217:%.*]] = icmp eq i32 [[CONV215]], [[CONV216]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP217]], label [[COND_TRUE219:%.*]], label [[COND_FALSE221:%.*]]
+// SIMD-ONLY0:       cond.true219:
+// SIMD-ONLY0-NEXT:    [[TMP103:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV220:%.*]] = sext i16 [[TMP103]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END223:%.*]]
+// SIMD-ONLY0:       cond.false221:
+// SIMD-ONLY0-NEXT:    [[TMP104:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV222:%.*]] = sext i16 [[TMP104]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END223]]
+// SIMD-ONLY0:       cond.end223:
+// SIMD-ONLY0-NEXT:    [[COND224:%.*]] = phi i32 [ [[CONV220]], [[COND_TRUE219]] ], [ [[CONV222]], [[COND_FALSE221]] ]
+// SIMD-ONLY0-NEXT:    [[CONV225:%.*]] = trunc i32 [[COND224]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV225]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP105:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV226:%.*]] = sext i16 [[TMP105]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP106:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV227:%.*]] = sext i16 [[TMP106]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP228:%.*]] = icmp sgt i32 [[CONV226]], [[CONV227]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP228]], label [[COND_TRUE230:%.*]], label [[COND_FALSE232:%.*]]
+// SIMD-ONLY0:       cond.true230:
+// SIMD-ONLY0-NEXT:    [[TMP107:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV231:%.*]] = sext i16 [[TMP107]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END234:%.*]]
+// SIMD-ONLY0:       cond.false232:
+// SIMD-ONLY0-NEXT:    [[TMP108:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV233:%.*]] = sext i16 [[TMP108]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END234]]
+// SIMD-ONLY0:       cond.end234:
+// SIMD-ONLY0-NEXT:    [[COND235:%.*]] = phi i32 [ [[CONV231]], [[COND_TRUE230]] ], [ [[CONV233]], [[COND_FALSE232]] ]
+// SIMD-ONLY0-NEXT:    [[CONV236:%.*]] = trunc i32 [[COND235]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV236]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP109:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP109]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP110:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV237:%.*]] = sext i16 [[TMP110]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP111:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV238:%.*]] = sext i16 [[TMP111]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP239:%.*]] = icmp slt i32 [[CONV237]], [[CONV238]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP239]], label [[COND_TRUE241:%.*]], label [[COND_FALSE243:%.*]]
+// SIMD-ONLY0:       cond.true241:
+// SIMD-ONLY0-NEXT:    [[TMP112:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV242:%.*]] = sext i16 [[TMP112]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END245:%.*]]
+// SIMD-ONLY0:       cond.false243:
+// SIMD-ONLY0-NEXT:    [[TMP113:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV244:%.*]] = sext i16 [[TMP113]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END245]]
+// SIMD-ONLY0:       cond.end245:
+// SIMD-ONLY0-NEXT:    [[COND246:%.*]] = phi i32 [ [[CONV242]], [[COND_TRUE241]] ], [ [[CONV244]], [[COND_FALSE243]] ]
+// SIMD-ONLY0-NEXT:    [[CONV247:%.*]] = trunc i32 [[COND246]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV247]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP114:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP114]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP115:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV248:%.*]] = sext i16 [[TMP115]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP116:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV249:%.*]] = sext i16 [[TMP116]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP250:%.*]] = icmp eq i32 [[CONV248]], [[CONV249]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP250]], label [[COND_TRUE252:%.*]], label [[COND_FALSE254:%.*]]
+// SIMD-ONLY0:       cond.true252:
+// SIMD-ONLY0-NEXT:    [[TMP117:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV253:%.*]] = sext i16 [[TMP117]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END256:%.*]]
+// SIMD-ONLY0:       cond.false254:
+// SIMD-ONLY0-NEXT:    [[TMP118:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV255:%.*]] = sext i16 [[TMP118]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END256]]
+// SIMD-ONLY0:       cond.end256:
+// SIMD-ONLY0-NEXT:    [[COND257:%.*]] = phi i32 [ [[CONV253]], [[COND_TRUE252]] ], [ [[CONV255]], [[COND_FALSE254]] ]
+// SIMD-ONLY0-NEXT:    [[CONV258:%.*]] = trunc i32 [[COND257]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV258]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP119:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP119]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP120:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP120]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP121:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV259:%.*]] = sext i16 [[TMP121]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP122:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV260:%.*]] = sext i16 [[TMP122]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP261:%.*]] = icmp sgt i32 [[CONV259]], [[CONV260]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP261]], label [[COND_TRUE263:%.*]], label [[COND_FALSE265:%.*]]
+// SIMD-ONLY0:       cond.true263:
+// SIMD-ONLY0-NEXT:    [[TMP123:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV264:%.*]] = sext i16 [[TMP123]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END267:%.*]]
+// SIMD-ONLY0:       cond.false265:
+// SIMD-ONLY0-NEXT:    [[TMP124:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV266:%.*]] = sext i16 [[TMP124]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END267]]
+// SIMD-ONLY0:       cond.end267:
+// SIMD-ONLY0-NEXT:    [[COND268:%.*]] = phi i32 [ [[CONV264]], [[COND_TRUE263]] ], [ [[CONV266]], [[COND_FALSE265]] ]
+// SIMD-ONLY0-NEXT:    [[CONV269:%.*]] = trunc i32 [[COND268]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV269]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP125:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP125]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP126:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV270:%.*]] = sext i16 [[TMP126]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP127:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV271:%.*]] = sext i16 [[TMP127]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP272:%.*]] = icmp slt i32 [[CONV270]], [[CONV271]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP272]], label [[COND_TRUE274:%.*]], label [[COND_FALSE276:%.*]]
+// SIMD-ONLY0:       cond.true274:
+// SIMD-ONLY0-NEXT:    [[TMP128:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV275:%.*]] = sext i16 [[TMP128]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END278:%.*]]
+// SIMD-ONLY0:       cond.false276:
+// SIMD-ONLY0-NEXT:    [[TMP129:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV277:%.*]] = sext i16 [[TMP129]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END278]]
+// SIMD-ONLY0:       cond.end278:
+// SIMD-ONLY0-NEXT:    [[COND279:%.*]] = phi i32 [ [[CONV275]], [[COND_TRUE274]] ], [ [[CONV277]], [[COND_FALSE276]] ]
+// SIMD-ONLY0-NEXT:    [[CONV280:%.*]] = trunc i32 [[COND279]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV280]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP130:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP130]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP131:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV281:%.*]] = sext i16 [[TMP131]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP132:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV282:%.*]] = sext i16 [[TMP132]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP283:%.*]] = icmp eq i32 [[CONV281]], [[CONV282]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP283]], label [[COND_TRUE285:%.*]], label [[COND_FALSE287:%.*]]
+// SIMD-ONLY0:       cond.true285:
+// SIMD-ONLY0-NEXT:    [[TMP133:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV286:%.*]] = sext i16 [[TMP133]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END289:%.*]]
+// SIMD-ONLY0:       cond.false287:
+// SIMD-ONLY0-NEXT:    [[TMP134:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV288:%.*]] = sext i16 [[TMP134]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END289]]
+// SIMD-ONLY0:       cond.end289:
+// SIMD-ONLY0-NEXT:    [[COND290:%.*]] = phi i32 [ [[CONV286]], [[COND_TRUE285]] ], [ [[CONV288]], [[COND_FALSE287]] ]
+// SIMD-ONLY0-NEXT:    [[CONV291:%.*]] = trunc i32 [[COND290]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV291]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP135:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV292:%.*]] = sext i16 [[TMP135]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP136:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV293:%.*]] = sext i16 [[TMP136]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP294:%.*]] = icmp sgt i32 [[CONV292]], [[CONV293]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP294]], label [[COND_TRUE296:%.*]], label [[COND_FALSE298:%.*]]
+// SIMD-ONLY0:       cond.true296:
+// SIMD-ONLY0-NEXT:    [[TMP137:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV297:%.*]] = sext i16 [[TMP137]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END300:%.*]]
+// SIMD-ONLY0:       cond.false298:
+// SIMD-ONLY0-NEXT:    [[TMP138:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV299:%.*]] = sext i16 [[TMP138]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END300]]
+// SIMD-ONLY0:       cond.end300:
+// SIMD-ONLY0-NEXT:    [[COND301:%.*]] = phi i32 [ [[CONV297]], [[COND_TRUE296]] ], [ [[CONV299]], [[COND_FALSE298]] ]
+// SIMD-ONLY0-NEXT:    [[CONV302:%.*]] = trunc i32 [[COND301]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV302]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP139:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP139]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP140:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV303:%.*]] = sext i16 [[TMP140]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP141:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV304:%.*]] = sext i16 [[TMP141]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP305:%.*]] = icmp slt i32 [[CONV303]], [[CONV304]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP305]], label [[COND_TRUE307:%.*]], label [[COND_FALSE309:%.*]]
+// SIMD-ONLY0:       cond.true307:
+// SIMD-ONLY0-NEXT:    [[TMP142:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV308:%.*]] = sext i16 [[TMP142]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END311:%.*]]
+// SIMD-ONLY0:       cond.false309:
+// SIMD-ONLY0-NEXT:    [[TMP143:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV310:%.*]] = sext i16 [[TMP143]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END311]]
+// SIMD-ONLY0:       cond.end311:
+// SIMD-ONLY0-NEXT:    [[COND312:%.*]] = phi i32 [ [[CONV308]], [[COND_TRUE307]] ], [ [[CONV310]], [[COND_FALSE309]] ]
+// SIMD-ONLY0-NEXT:    [[CONV313:%.*]] = trunc i32 [[COND312]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV313]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP144:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP144]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP145:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV314:%.*]] = sext i16 [[TMP145]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP146:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV315:%.*]] = sext i16 [[TMP146]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP316:%.*]] = icmp eq i32 [[CONV314]], [[CONV315]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP316]], label [[COND_TRUE318:%.*]], label [[COND_FALSE320:%.*]]
+// SIMD-ONLY0:       cond.true318:
+// SIMD-ONLY0-NEXT:    [[TMP147:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV319:%.*]] = sext i16 [[TMP147]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END322:%.*]]
+// SIMD-ONLY0:       cond.false320:
+// SIMD-ONLY0-NEXT:    [[TMP148:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV321:%.*]] = sext i16 [[TMP148]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END322]]
+// SIMD-ONLY0:       cond.end322:
+// SIMD-ONLY0-NEXT:    [[COND323:%.*]] = phi i32 [ [[CONV319]], [[COND_TRUE318]] ], [ [[CONV321]], [[COND_FALSE320]] ]
+// SIMD-ONLY0-NEXT:    [[CONV324:%.*]] = trunc i32 [[COND323]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV324]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP149:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP149]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP150:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP150]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP151:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV325:%.*]] = sext i16 [[TMP151]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP152:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV326:%.*]] = sext i16 [[TMP152]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP327:%.*]] = icmp sgt i32 [[CONV325]], [[CONV326]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP327]], label [[COND_TRUE329:%.*]], label [[COND_FALSE331:%.*]]
+// SIMD-ONLY0:       cond.true329:
+// SIMD-ONLY0-NEXT:    [[TMP153:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV330:%.*]] = sext i16 [[TMP153]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END333:%.*]]
+// SIMD-ONLY0:       cond.false331:
+// SIMD-ONLY0-NEXT:    [[TMP154:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV332:%.*]] = sext i16 [[TMP154]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END333]]
+// SIMD-ONLY0:       cond.end333:
+// SIMD-ONLY0-NEXT:    [[COND334:%.*]] = phi i32 [ [[CONV330]], [[COND_TRUE329]] ], [ [[CONV332]], [[COND_FALSE331]] ]
+// SIMD-ONLY0-NEXT:    [[CONV335:%.*]] = trunc i32 [[COND334]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV335]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP155:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP155]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP156:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV336:%.*]] = sext i16 [[TMP156]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP157:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV337:%.*]] = sext i16 [[TMP157]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP338:%.*]] = icmp slt i32 [[CONV336]], [[CONV337]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP338]], label [[COND_TRUE340:%.*]], label [[COND_FALSE342:%.*]]
+// SIMD-ONLY0:       cond.true340:
+// SIMD-ONLY0-NEXT:    [[TMP158:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV341:%.*]] = sext i16 [[TMP158]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END344:%.*]]
+// SIMD-ONLY0:       cond.false342:
+// SIMD-ONLY0-NEXT:    [[TMP159:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV343:%.*]] = sext i16 [[TMP159]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END344]]
+// SIMD-ONLY0:       cond.end344:
+// SIMD-ONLY0-NEXT:    [[COND345:%.*]] = phi i32 [ [[CONV341]], [[COND_TRUE340]] ], [ [[CONV343]], [[COND_FALSE342]] ]
+// SIMD-ONLY0-NEXT:    [[CONV346:%.*]] = trunc i32 [[COND345]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV346]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP160:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP160]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP161:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV347:%.*]] = sext i16 [[TMP161]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP162:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV348:%.*]] = sext i16 [[TMP162]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP349:%.*]] = icmp eq i32 [[CONV347]], [[CONV348]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP349]], label [[COND_TRUE351:%.*]], label [[COND_FALSE353:%.*]]
+// SIMD-ONLY0:       cond.true351:
+// SIMD-ONLY0-NEXT:    [[TMP163:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV352:%.*]] = sext i16 [[TMP163]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END355:%.*]]
+// SIMD-ONLY0:       cond.false353:
+// SIMD-ONLY0-NEXT:    [[TMP164:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV354:%.*]] = sext i16 [[TMP164]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END355]]
+// SIMD-ONLY0:       cond.end355:
+// SIMD-ONLY0-NEXT:    [[COND356:%.*]] = phi i32 [ [[CONV352]], [[COND_TRUE351]] ], [ [[CONV354]], [[COND_FALSE353]] ]
+// SIMD-ONLY0-NEXT:    [[CONV357:%.*]] = trunc i32 [[COND356]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV357]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP165:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV358:%.*]] = sext i16 [[TMP165]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP166:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV359:%.*]] = sext i16 [[TMP166]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP360:%.*]] = icmp sgt i32 [[CONV358]], [[CONV359]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP360]], label [[COND_TRUE362:%.*]], label [[COND_FALSE364:%.*]]
+// SIMD-ONLY0:       cond.true362:
+// SIMD-ONLY0-NEXT:    [[TMP167:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV363:%.*]] = sext i16 [[TMP167]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END366:%.*]]
+// SIMD-ONLY0:       cond.false364:
+// SIMD-ONLY0-NEXT:    [[TMP168:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV365:%.*]] = sext i16 [[TMP168]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END366]]
+// SIMD-ONLY0:       cond.end366:
+// SIMD-ONLY0-NEXT:    [[COND367:%.*]] = phi i32 [ [[CONV363]], [[COND_TRUE362]] ], [ [[CONV365]], [[COND_FALSE364]] ]
+// SIMD-ONLY0-NEXT:    [[CONV368:%.*]] = trunc i32 [[COND367]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV368]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP169:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP169]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP170:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV369:%.*]] = sext i16 [[TMP170]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP171:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV370:%.*]] = sext i16 [[TMP171]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP371:%.*]] = icmp slt i32 [[CONV369]], [[CONV370]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP371]], label [[COND_TRUE373:%.*]], label [[COND_FALSE375:%.*]]
+// SIMD-ONLY0:       cond.true373:
+// SIMD-ONLY0-NEXT:    [[TMP172:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV374:%.*]] = sext i16 [[TMP172]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END377:%.*]]
+// SIMD-ONLY0:       cond.false375:
+// SIMD-ONLY0-NEXT:    [[TMP173:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV376:%.*]] = sext i16 [[TMP173]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END377]]
+// SIMD-ONLY0:       cond.end377:
+// SIMD-ONLY0-NEXT:    [[COND378:%.*]] = phi i32 [ [[CONV374]], [[COND_TRUE373]] ], [ [[CONV376]], [[COND_FALSE375]] ]
+// SIMD-ONLY0-NEXT:    [[CONV379:%.*]] = trunc i32 [[COND378]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV379]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP174:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP174]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP175:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV380:%.*]] = sext i16 [[TMP175]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP176:%.*]] = load i16, ptr [[SE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV381:%.*]] = sext i16 [[TMP176]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP382:%.*]] = icmp eq i32 [[CONV380]], [[CONV381]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP382]], label [[COND_TRUE384:%.*]], label [[COND_FALSE386:%.*]]
+// SIMD-ONLY0:       cond.true384:
+// SIMD-ONLY0-NEXT:    [[TMP177:%.*]] = load i16, ptr [[SD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV385:%.*]] = sext i16 [[TMP177]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END388:%.*]]
+// SIMD-ONLY0:       cond.false386:
+// SIMD-ONLY0-NEXT:    [[TMP178:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV387:%.*]] = sext i16 [[TMP178]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END388]]
+// SIMD-ONLY0:       cond.end388:
+// SIMD-ONLY0-NEXT:    [[COND389:%.*]] = phi i32 [ [[CONV385]], [[COND_TRUE384]] ], [ [[CONV387]], [[COND_FALSE386]] ]
+// SIMD-ONLY0-NEXT:    [[CONV390:%.*]] = trunc i32 [[COND389]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV390]], ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP179:%.*]] = load i16, ptr [[SX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP179]], ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP180:%.*]] = load i16, ptr [[SV]], align 2
+// SIMD-ONLY0-NEXT:    ret i16 [[TMP180]]
+//
+//
+// SIMD-ONLY0-LABEL: @usxevd(
+// SIMD-ONLY0-NEXT:  entry:
+// SIMD-ONLY0-NEXT:    [[USX:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[USV:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[USE:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[USD:%.*]] = alloca i16, align 2
+// SIMD-ONLY0-NEXT:    [[TMP0:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP0]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP1:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV:%.*]] = zext i16 [[TMP1]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP2:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV1:%.*]] = zext i16 [[TMP2]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[CONV]], [[CONV1]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]]
+// SIMD-ONLY0:       cond.true:
+// SIMD-ONLY0-NEXT:    [[TMP3:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV3:%.*]] = zext i16 [[TMP3]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END:%.*]]
+// SIMD-ONLY0:       cond.false:
+// SIMD-ONLY0-NEXT:    [[TMP4:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV4:%.*]] = zext i16 [[TMP4]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END]]
+// SIMD-ONLY0:       cond.end:
+// SIMD-ONLY0-NEXT:    [[COND:%.*]] = phi i32 [ [[CONV3]], [[COND_TRUE]] ], [ [[CONV4]], [[COND_FALSE]] ]
+// SIMD-ONLY0-NEXT:    [[CONV5:%.*]] = trunc i32 [[COND]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV5]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP5:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP5]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP6:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV6:%.*]] = zext i16 [[TMP6]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP7:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV7:%.*]] = zext i16 [[TMP7]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP8:%.*]] = icmp slt i32 [[CONV6]], [[CONV7]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP8]], label [[COND_TRUE10:%.*]], label [[COND_FALSE12:%.*]]
+// SIMD-ONLY0:       cond.true10:
+// SIMD-ONLY0-NEXT:    [[TMP8:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV11:%.*]] = zext i16 [[TMP8]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END14:%.*]]
+// SIMD-ONLY0:       cond.false12:
+// SIMD-ONLY0-NEXT:    [[TMP9:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV13:%.*]] = zext i16 [[TMP9]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END14]]
+// SIMD-ONLY0:       cond.end14:
+// SIMD-ONLY0-NEXT:    [[COND15:%.*]] = phi i32 [ [[CONV11]], [[COND_TRUE10]] ], [ [[CONV13]], [[COND_FALSE12]] ]
+// SIMD-ONLY0-NEXT:    [[CONV16:%.*]] = trunc i32 [[COND15]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV16]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP10:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP10]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP11:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV17:%.*]] = zext i16 [[TMP11]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP12:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV18:%.*]] = zext i16 [[TMP12]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP19:%.*]] = icmp eq i32 [[CONV17]], [[CONV18]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP19]], label [[COND_TRUE21:%.*]], label [[COND_FALSE23:%.*]]
+// SIMD-ONLY0:       cond.true21:
+// SIMD-ONLY0-NEXT:    [[TMP13:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV22:%.*]] = zext i16 [[TMP13]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END25:%.*]]
+// SIMD-ONLY0:       cond.false23:
+// SIMD-ONLY0-NEXT:    [[TMP14:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV24:%.*]] = zext i16 [[TMP14]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END25]]
+// SIMD-ONLY0:       cond.end25:
+// SIMD-ONLY0-NEXT:    [[COND26:%.*]] = phi i32 [ [[CONV22]], [[COND_TRUE21]] ], [ [[CONV24]], [[COND_FALSE23]] ]
+// SIMD-ONLY0-NEXT:    [[CONV27:%.*]] = trunc i32 [[COND26]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV27]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP15:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV28:%.*]] = zext i16 [[TMP15]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP16:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV29:%.*]] = zext i16 [[TMP16]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP30:%.*]] = icmp sgt i32 [[CONV28]], [[CONV29]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP30]], label [[COND_TRUE32:%.*]], label [[COND_FALSE34:%.*]]
+// SIMD-ONLY0:       cond.true32:
+// SIMD-ONLY0-NEXT:    [[TMP17:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV33:%.*]] = zext i16 [[TMP17]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END36:%.*]]
+// SIMD-ONLY0:       cond.false34:
+// SIMD-ONLY0-NEXT:    [[TMP18:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV35:%.*]] = zext i16 [[TMP18]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END36]]
+// SIMD-ONLY0:       cond.end36:
+// SIMD-ONLY0-NEXT:    [[COND37:%.*]] = phi i32 [ [[CONV33]], [[COND_TRUE32]] ], [ [[CONV35]], [[COND_FALSE34]] ]
+// SIMD-ONLY0-NEXT:    [[CONV38:%.*]] = trunc i32 [[COND37]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV38]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP19:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP19]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP20:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV39:%.*]] = zext i16 [[TMP20]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP21:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV40:%.*]] = zext i16 [[TMP21]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP41:%.*]] = icmp slt i32 [[CONV39]], [[CONV40]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP41]], label [[COND_TRUE43:%.*]], label [[COND_FALSE45:%.*]]
+// SIMD-ONLY0:       cond.true43:
+// SIMD-ONLY0-NEXT:    [[TMP22:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV44:%.*]] = zext i16 [[TMP22]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END47:%.*]]
+// SIMD-ONLY0:       cond.false45:
+// SIMD-ONLY0-NEXT:    [[TMP23:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV46:%.*]] = zext i16 [[TMP23]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END47]]
+// SIMD-ONLY0:       cond.end47:
+// SIMD-ONLY0-NEXT:    [[COND48:%.*]] = phi i32 [ [[CONV44]], [[COND_TRUE43]] ], [ [[CONV46]], [[COND_FALSE45]] ]
+// SIMD-ONLY0-NEXT:    [[CONV49:%.*]] = trunc i32 [[COND48]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV49]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP24:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP24]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP25:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV50:%.*]] = zext i16 [[TMP25]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP26:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV51:%.*]] = zext i16 [[TMP26]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP52:%.*]] = icmp eq i32 [[CONV50]], [[CONV51]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP52]], label [[COND_TRUE54:%.*]], label [[COND_FALSE56:%.*]]
+// SIMD-ONLY0:       cond.true54:
+// SIMD-ONLY0-NEXT:    [[TMP27:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV55:%.*]] = zext i16 [[TMP27]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END58:%.*]]
+// SIMD-ONLY0:       cond.false56:
+// SIMD-ONLY0-NEXT:    [[TMP28:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV57:%.*]] = zext i16 [[TMP28]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END58]]
+// SIMD-ONLY0:       cond.end58:
+// SIMD-ONLY0-NEXT:    [[COND59:%.*]] = phi i32 [ [[CONV55]], [[COND_TRUE54]] ], [ [[CONV57]], [[COND_FALSE56]] ]
+// SIMD-ONLY0-NEXT:    [[CONV60:%.*]] = trunc i32 [[COND59]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV60]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP29:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP29]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP30:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP30]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP31:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV61:%.*]] = zext i16 [[TMP31]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP32:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV62:%.*]] = zext i16 [[TMP32]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP63:%.*]] = icmp sgt i32 [[CONV61]], [[CONV62]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP63]], label [[COND_TRUE65:%.*]], label [[COND_FALSE67:%.*]]
+// SIMD-ONLY0:       cond.true65:
+// SIMD-ONLY0-NEXT:    [[TMP33:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV66:%.*]] = zext i16 [[TMP33]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END69:%.*]]
+// SIMD-ONLY0:       cond.false67:
+// SIMD-ONLY0-NEXT:    [[TMP34:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV68:%.*]] = zext i16 [[TMP34]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END69]]
+// SIMD-ONLY0:       cond.end69:
+// SIMD-ONLY0-NEXT:    [[COND70:%.*]] = phi i32 [ [[CONV66]], [[COND_TRUE65]] ], [ [[CONV68]], [[COND_FALSE67]] ]
+// SIMD-ONLY0-NEXT:    [[CONV71:%.*]] = trunc i32 [[COND70]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV71]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP35:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP35]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP36:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV72:%.*]] = zext i16 [[TMP36]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP37:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV73:%.*]] = zext i16 [[TMP37]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP74:%.*]] = icmp slt i32 [[CONV72]], [[CONV73]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP74]], label [[COND_TRUE76:%.*]], label [[COND_FALSE78:%.*]]
+// SIMD-ONLY0:       cond.true76:
+// SIMD-ONLY0-NEXT:    [[TMP38:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV77:%.*]] = zext i16 [[TMP38]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END80:%.*]]
+// SIMD-ONLY0:       cond.false78:
+// SIMD-ONLY0-NEXT:    [[TMP39:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV79:%.*]] = zext i16 [[TMP39]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END80]]
+// SIMD-ONLY0:       cond.end80:
+// SIMD-ONLY0-NEXT:    [[COND81:%.*]] = phi i32 [ [[CONV77]], [[COND_TRUE76]] ], [ [[CONV79]], [[COND_FALSE78]] ]
+// SIMD-ONLY0-NEXT:    [[CONV82:%.*]] = trunc i32 [[COND81]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV82]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP40:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP40]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP41:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV83:%.*]] = zext i16 [[TMP41]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP42:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV84:%.*]] = zext i16 [[TMP42]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP85:%.*]] = icmp eq i32 [[CONV83]], [[CONV84]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP85]], label [[COND_TRUE87:%.*]], label [[COND_FALSE89:%.*]]
+// SIMD-ONLY0:       cond.true87:
+// SIMD-ONLY0-NEXT:    [[TMP43:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV88:%.*]] = zext i16 [[TMP43]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END91:%.*]]
+// SIMD-ONLY0:       cond.false89:
+// SIMD-ONLY0-NEXT:    [[TMP44:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV90:%.*]] = zext i16 [[TMP44]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END91]]
+// SIMD-ONLY0:       cond.end91:
+// SIMD-ONLY0-NEXT:    [[COND92:%.*]] = phi i32 [ [[CONV88]], [[COND_TRUE87]] ], [ [[CONV90]], [[COND_FALSE89]] ]
+// SIMD-ONLY0-NEXT:    [[CONV93:%.*]] = trunc i32 [[COND92]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV93]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP45:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV94:%.*]] = zext i16 [[TMP45]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP46:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV95:%.*]] = zext i16 [[TMP46]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP96:%.*]] = icmp sgt i32 [[CONV94]], [[CONV95]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP96]], label [[COND_TRUE98:%.*]], label [[COND_FALSE100:%.*]]
+// SIMD-ONLY0:       cond.true98:
+// SIMD-ONLY0-NEXT:    [[TMP47:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV99:%.*]] = zext i16 [[TMP47]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END102:%.*]]
+// SIMD-ONLY0:       cond.false100:
+// SIMD-ONLY0-NEXT:    [[TMP48:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV101:%.*]] = zext i16 [[TMP48]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END102]]
+// SIMD-ONLY0:       cond.end102:
+// SIMD-ONLY0-NEXT:    [[COND103:%.*]] = phi i32 [ [[CONV99]], [[COND_TRUE98]] ], [ [[CONV101]], [[COND_FALSE100]] ]
+// SIMD-ONLY0-NEXT:    [[CONV104:%.*]] = trunc i32 [[COND103]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV104]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP49:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP49]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP50:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV105:%.*]] = zext i16 [[TMP50]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP51:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV106:%.*]] = zext i16 [[TMP51]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP107:%.*]] = icmp slt i32 [[CONV105]], [[CONV106]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP107]], label [[COND_TRUE109:%.*]], label [[COND_FALSE111:%.*]]
+// SIMD-ONLY0:       cond.true109:
+// SIMD-ONLY0-NEXT:    [[TMP52:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV110:%.*]] = zext i16 [[TMP52]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END113:%.*]]
+// SIMD-ONLY0:       cond.false111:
+// SIMD-ONLY0-NEXT:    [[TMP53:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV112:%.*]] = zext i16 [[TMP53]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END113]]
+// SIMD-ONLY0:       cond.end113:
+// SIMD-ONLY0-NEXT:    [[COND114:%.*]] = phi i32 [ [[CONV110]], [[COND_TRUE109]] ], [ [[CONV112]], [[COND_FALSE111]] ]
+// SIMD-ONLY0-NEXT:    [[CONV115:%.*]] = trunc i32 [[COND114]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV115]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP54:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP54]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP55:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV116:%.*]] = zext i16 [[TMP55]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP56:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV117:%.*]] = zext i16 [[TMP56]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP118:%.*]] = icmp eq i32 [[CONV116]], [[CONV117]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP118]], label [[COND_TRUE120:%.*]], label [[COND_FALSE122:%.*]]
+// SIMD-ONLY0:       cond.true120:
+// SIMD-ONLY0-NEXT:    [[TMP57:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV121:%.*]] = zext i16 [[TMP57]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END124:%.*]]
+// SIMD-ONLY0:       cond.false122:
+// SIMD-ONLY0-NEXT:    [[TMP58:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV123:%.*]] = zext i16 [[TMP58]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END124]]
+// SIMD-ONLY0:       cond.end124:
+// SIMD-ONLY0-NEXT:    [[COND125:%.*]] = phi i32 [ [[CONV121]], [[COND_TRUE120]] ], [ [[CONV123]], [[COND_FALSE122]] ]
+// SIMD-ONLY0-NEXT:    [[CONV126:%.*]] = trunc i32 [[COND125]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV126]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP59:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP59]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP60:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP60]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP61:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV127:%.*]] = zext i16 [[TMP61]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP62:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV128:%.*]] = zext i16 [[TMP62]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP129:%.*]] = icmp sgt i32 [[CONV127]], [[CONV128]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP129]], label [[COND_TRUE131:%.*]], label [[COND_FALSE133:%.*]]
+// SIMD-ONLY0:       cond.true131:
+// SIMD-ONLY0-NEXT:    [[TMP63:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV132:%.*]] = zext i16 [[TMP63]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END135:%.*]]
+// SIMD-ONLY0:       cond.false133:
+// SIMD-ONLY0-NEXT:    [[TMP64:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV134:%.*]] = zext i16 [[TMP64]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END135]]
+// SIMD-ONLY0:       cond.end135:
+// SIMD-ONLY0-NEXT:    [[COND136:%.*]] = phi i32 [ [[CONV132]], [[COND_TRUE131]] ], [ [[CONV134]], [[COND_FALSE133]] ]
+// SIMD-ONLY0-NEXT:    [[CONV137:%.*]] = trunc i32 [[COND136]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV137]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP65:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP65]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP66:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV138:%.*]] = zext i16 [[TMP66]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP67:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV139:%.*]] = zext i16 [[TMP67]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP140:%.*]] = icmp slt i32 [[CONV138]], [[CONV139]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP140]], label [[COND_TRUE142:%.*]], label [[COND_FALSE144:%.*]]
+// SIMD-ONLY0:       cond.true142:
+// SIMD-ONLY0-NEXT:    [[TMP68:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV143:%.*]] = zext i16 [[TMP68]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END146:%.*]]
+// SIMD-ONLY0:       cond.false144:
+// SIMD-ONLY0-NEXT:    [[TMP69:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV145:%.*]] = zext i16 [[TMP69]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END146]]
+// SIMD-ONLY0:       cond.end146:
+// SIMD-ONLY0-NEXT:    [[COND147:%.*]] = phi i32 [ [[CONV143]], [[COND_TRUE142]] ], [ [[CONV145]], [[COND_FALSE144]] ]
+// SIMD-ONLY0-NEXT:    [[CONV148:%.*]] = trunc i32 [[COND147]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV148]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP70:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP70]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP71:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV149:%.*]] = zext i16 [[TMP71]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP72:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV150:%.*]] = zext i16 [[TMP72]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP151:%.*]] = icmp eq i32 [[CONV149]], [[CONV150]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP151]], label [[COND_TRUE153:%.*]], label [[COND_FALSE155:%.*]]
+// SIMD-ONLY0:       cond.true153:
+// SIMD-ONLY0-NEXT:    [[TMP73:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV154:%.*]] = zext i16 [[TMP73]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END157:%.*]]
+// SIMD-ONLY0:       cond.false155:
+// SIMD-ONLY0-NEXT:    [[TMP74:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV156:%.*]] = zext i16 [[TMP74]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END157]]
+// SIMD-ONLY0:       cond.end157:
+// SIMD-ONLY0-NEXT:    [[COND158:%.*]] = phi i32 [ [[CONV154]], [[COND_TRUE153]] ], [ [[CONV156]], [[COND_FALSE155]] ]
+// SIMD-ONLY0-NEXT:    [[CONV159:%.*]] = trunc i32 [[COND158]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV159]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP75:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV160:%.*]] = zext i16 [[TMP75]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP76:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV161:%.*]] = zext i16 [[TMP76]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP162:%.*]] = icmp sgt i32 [[CONV160]], [[CONV161]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP162]], label [[COND_TRUE164:%.*]], label [[COND_FALSE166:%.*]]
+// SIMD-ONLY0:       cond.true164:
+// SIMD-ONLY0-NEXT:    [[TMP77:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV165:%.*]] = zext i16 [[TMP77]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END168:%.*]]
+// SIMD-ONLY0:       cond.false166:
+// SIMD-ONLY0-NEXT:    [[TMP78:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV167:%.*]] = zext i16 [[TMP78]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END168]]
+// SIMD-ONLY0:       cond.end168:
+// SIMD-ONLY0-NEXT:    [[COND169:%.*]] = phi i32 [ [[CONV165]], [[COND_TRUE164]] ], [ [[CONV167]], [[COND_FALSE166]] ]
+// SIMD-ONLY0-NEXT:    [[CONV170:%.*]] = trunc i32 [[COND169]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV170]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP79:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP79]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP80:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV171:%.*]] = zext i16 [[TMP80]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP81:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV172:%.*]] = zext i16 [[TMP81]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP173:%.*]] = icmp slt i32 [[CONV171]], [[CONV172]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP173]], label [[COND_TRUE175:%.*]], label [[COND_FALSE177:%.*]]
+// SIMD-ONLY0:       cond.true175:
+// SIMD-ONLY0-NEXT:    [[TMP82:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV176:%.*]] = zext i16 [[TMP82]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END179:%.*]]
+// SIMD-ONLY0:       cond.false177:
+// SIMD-ONLY0-NEXT:    [[TMP83:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV178:%.*]] = zext i16 [[TMP83]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END179]]
+// SIMD-ONLY0:       cond.end179:
+// SIMD-ONLY0-NEXT:    [[COND180:%.*]] = phi i32 [ [[CONV176]], [[COND_TRUE175]] ], [ [[CONV178]], [[COND_FALSE177]] ]
+// SIMD-ONLY0-NEXT:    [[CONV181:%.*]] = trunc i32 [[COND180]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV181]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP84:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP84]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP85:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV182:%.*]] = zext i16 [[TMP85]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP86:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV183:%.*]] = zext i16 [[TMP86]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP184:%.*]] = icmp eq i32 [[CONV182]], [[CONV183]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP184]], label [[COND_TRUE186:%.*]], label [[COND_FALSE188:%.*]]
+// SIMD-ONLY0:       cond.true186:
+// SIMD-ONLY0-NEXT:    [[TMP87:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV187:%.*]] = zext i16 [[TMP87]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END190:%.*]]
+// SIMD-ONLY0:       cond.false188:
+// SIMD-ONLY0-NEXT:    [[TMP88:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV189:%.*]] = zext i16 [[TMP88]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END190]]
+// SIMD-ONLY0:       cond.end190:
+// SIMD-ONLY0-NEXT:    [[COND191:%.*]] = phi i32 [ [[CONV187]], [[COND_TRUE186]] ], [ [[CONV189]], [[COND_FALSE188]] ]
+// SIMD-ONLY0-NEXT:    [[CONV192:%.*]] = trunc i32 [[COND191]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV192]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP89:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP89]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP90:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP90]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP91:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV193:%.*]] = zext i16 [[TMP91]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP92:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV194:%.*]] = zext i16 [[TMP92]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP195:%.*]] = icmp sgt i32 [[CONV193]], [[CONV194]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP195]], label [[COND_TRUE197:%.*]], label [[COND_FALSE199:%.*]]
+// SIMD-ONLY0:       cond.true197:
+// SIMD-ONLY0-NEXT:    [[TMP93:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV198:%.*]] = zext i16 [[TMP93]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END201:%.*]]
+// SIMD-ONLY0:       cond.false199:
+// SIMD-ONLY0-NEXT:    [[TMP94:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV200:%.*]] = zext i16 [[TMP94]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END201]]
+// SIMD-ONLY0:       cond.end201:
+// SIMD-ONLY0-NEXT:    [[COND202:%.*]] = phi i32 [ [[CONV198]], [[COND_TRUE197]] ], [ [[CONV200]], [[COND_FALSE199]] ]
+// SIMD-ONLY0-NEXT:    [[CONV203:%.*]] = trunc i32 [[COND202]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV203]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP95:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP95]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP96:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV204:%.*]] = zext i16 [[TMP96]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP97:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV205:%.*]] = zext i16 [[TMP97]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP206:%.*]] = icmp slt i32 [[CONV204]], [[CONV205]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP206]], label [[COND_TRUE208:%.*]], label [[COND_FALSE210:%.*]]
+// SIMD-ONLY0:       cond.true208:
+// SIMD-ONLY0-NEXT:    [[TMP98:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV209:%.*]] = zext i16 [[TMP98]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END212:%.*]]
+// SIMD-ONLY0:       cond.false210:
+// SIMD-ONLY0-NEXT:    [[TMP99:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV211:%.*]] = zext i16 [[TMP99]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END212]]
+// SIMD-ONLY0:       cond.end212:
+// SIMD-ONLY0-NEXT:    [[COND213:%.*]] = phi i32 [ [[CONV209]], [[COND_TRUE208]] ], [ [[CONV211]], [[COND_FALSE210]] ]
+// SIMD-ONLY0-NEXT:    [[CONV214:%.*]] = trunc i32 [[COND213]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV214]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP100:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP100]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP101:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV215:%.*]] = zext i16 [[TMP101]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP102:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV216:%.*]] = zext i16 [[TMP102]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP217:%.*]] = icmp eq i32 [[CONV215]], [[CONV216]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP217]], label [[COND_TRUE219:%.*]], label [[COND_FALSE221:%.*]]
+// SIMD-ONLY0:       cond.true219:
+// SIMD-ONLY0-NEXT:    [[TMP103:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV220:%.*]] = zext i16 [[TMP103]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END223:%.*]]
+// SIMD-ONLY0:       cond.false221:
+// SIMD-ONLY0-NEXT:    [[TMP104:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV222:%.*]] = zext i16 [[TMP104]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END223]]
+// SIMD-ONLY0:       cond.end223:
+// SIMD-ONLY0-NEXT:    [[COND224:%.*]] = phi i32 [ [[CONV220]], [[COND_TRUE219]] ], [ [[CONV222]], [[COND_FALSE221]] ]
+// SIMD-ONLY0-NEXT:    [[CONV225:%.*]] = trunc i32 [[COND224]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV225]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP105:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV226:%.*]] = zext i16 [[TMP105]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP106:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV227:%.*]] = zext i16 [[TMP106]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP228:%.*]] = icmp sgt i32 [[CONV226]], [[CONV227]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP228]], label [[COND_TRUE230:%.*]], label [[COND_FALSE232:%.*]]
+// SIMD-ONLY0:       cond.true230:
+// SIMD-ONLY0-NEXT:    [[TMP107:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV231:%.*]] = zext i16 [[TMP107]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END234:%.*]]
+// SIMD-ONLY0:       cond.false232:
+// SIMD-ONLY0-NEXT:    [[TMP108:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV233:%.*]] = zext i16 [[TMP108]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END234]]
+// SIMD-ONLY0:       cond.end234:
+// SIMD-ONLY0-NEXT:    [[COND235:%.*]] = phi i32 [ [[CONV231]], [[COND_TRUE230]] ], [ [[CONV233]], [[COND_FALSE232]] ]
+// SIMD-ONLY0-NEXT:    [[CONV236:%.*]] = trunc i32 [[COND235]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV236]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP109:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP109]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP110:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV237:%.*]] = zext i16 [[TMP110]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP111:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV238:%.*]] = zext i16 [[TMP111]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP239:%.*]] = icmp slt i32 [[CONV237]], [[CONV238]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP239]], label [[COND_TRUE241:%.*]], label [[COND_FALSE243:%.*]]
+// SIMD-ONLY0:       cond.true241:
+// SIMD-ONLY0-NEXT:    [[TMP112:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV242:%.*]] = zext i16 [[TMP112]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END245:%.*]]
+// SIMD-ONLY0:       cond.false243:
+// SIMD-ONLY0-NEXT:    [[TMP113:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV244:%.*]] = zext i16 [[TMP113]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END245]]
+// SIMD-ONLY0:       cond.end245:
+// SIMD-ONLY0-NEXT:    [[COND246:%.*]] = phi i32 [ [[CONV242]], [[COND_TRUE241]] ], [ [[CONV244]], [[COND_FALSE243]] ]
+// SIMD-ONLY0-NEXT:    [[CONV247:%.*]] = trunc i32 [[COND246]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV247]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP114:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP114]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP115:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV248:%.*]] = zext i16 [[TMP115]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP116:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV249:%.*]] = zext i16 [[TMP116]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP250:%.*]] = icmp eq i32 [[CONV248]], [[CONV249]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP250]], label [[COND_TRUE252:%.*]], label [[COND_FALSE254:%.*]]
+// SIMD-ONLY0:       cond.true252:
+// SIMD-ONLY0-NEXT:    [[TMP117:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV253:%.*]] = zext i16 [[TMP117]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END256:%.*]]
+// SIMD-ONLY0:       cond.false254:
+// SIMD-ONLY0-NEXT:    [[TMP118:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV255:%.*]] = zext i16 [[TMP118]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END256]]
+// SIMD-ONLY0:       cond.end256:
+// SIMD-ONLY0-NEXT:    [[COND257:%.*]] = phi i32 [ [[CONV253]], [[COND_TRUE252]] ], [ [[CONV255]], [[COND_FALSE254]] ]
+// SIMD-ONLY0-NEXT:    [[CONV258:%.*]] = trunc i32 [[COND257]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV258]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP119:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP119]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP120:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP120]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP121:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV259:%.*]] = zext i16 [[TMP121]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP122:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV260:%.*]] = zext i16 [[TMP122]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP261:%.*]] = icmp sgt i32 [[CONV259]], [[CONV260]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP261]], label [[COND_TRUE263:%.*]], label [[COND_FALSE265:%.*]]
+// SIMD-ONLY0:       cond.true263:
+// SIMD-ONLY0-NEXT:    [[TMP123:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV264:%.*]] = zext i16 [[TMP123]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END267:%.*]]
+// SIMD-ONLY0:       cond.false265:
+// SIMD-ONLY0-NEXT:    [[TMP124:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV266:%.*]] = zext i16 [[TMP124]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END267]]
+// SIMD-ONLY0:       cond.end267:
+// SIMD-ONLY0-NEXT:    [[COND268:%.*]] = phi i32 [ [[CONV264]], [[COND_TRUE263]] ], [ [[CONV266]], [[COND_FALSE265]] ]
+// SIMD-ONLY0-NEXT:    [[CONV269:%.*]] = trunc i32 [[COND268]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV269]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP125:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP125]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP126:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV270:%.*]] = zext i16 [[TMP126]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP127:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV271:%.*]] = zext i16 [[TMP127]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP272:%.*]] = icmp slt i32 [[CONV270]], [[CONV271]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP272]], label [[COND_TRUE274:%.*]], label [[COND_FALSE276:%.*]]
+// SIMD-ONLY0:       cond.true274:
+// SIMD-ONLY0-NEXT:    [[TMP128:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV275:%.*]] = zext i16 [[TMP128]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END278:%.*]]
+// SIMD-ONLY0:       cond.false276:
+// SIMD-ONLY0-NEXT:    [[TMP129:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV277:%.*]] = zext i16 [[TMP129]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END278]]
+// SIMD-ONLY0:       cond.end278:
+// SIMD-ONLY0-NEXT:    [[COND279:%.*]] = phi i32 [ [[CONV275]], [[COND_TRUE274]] ], [ [[CONV277]], [[COND_FALSE276]] ]
+// SIMD-ONLY0-NEXT:    [[CONV280:%.*]] = trunc i32 [[COND279]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV280]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP130:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP130]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP131:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV281:%.*]] = zext i16 [[TMP131]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP132:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV282:%.*]] = zext i16 [[TMP132]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP283:%.*]] = icmp eq i32 [[CONV281]], [[CONV282]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP283]], label [[COND_TRUE285:%.*]], label [[COND_FALSE287:%.*]]
+// SIMD-ONLY0:       cond.true285:
+// SIMD-ONLY0-NEXT:    [[TMP133:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV286:%.*]] = zext i16 [[TMP133]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END289:%.*]]
+// SIMD-ONLY0:       cond.false287:
+// SIMD-ONLY0-NEXT:    [[TMP134:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV288:%.*]] = zext i16 [[TMP134]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END289]]
+// SIMD-ONLY0:       cond.end289:
+// SIMD-ONLY0-NEXT:    [[COND290:%.*]] = phi i32 [ [[CONV286]], [[COND_TRUE285]] ], [ [[CONV288]], [[COND_FALSE287]] ]
+// SIMD-ONLY0-NEXT:    [[CONV291:%.*]] = trunc i32 [[COND290]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV291]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP135:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV292:%.*]] = zext i16 [[TMP135]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP136:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV293:%.*]] = zext i16 [[TMP136]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP294:%.*]] = icmp sgt i32 [[CONV292]], [[CONV293]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP294]], label [[COND_TRUE296:%.*]], label [[COND_FALSE298:%.*]]
+// SIMD-ONLY0:       cond.true296:
+// SIMD-ONLY0-NEXT:    [[TMP137:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV297:%.*]] = zext i16 [[TMP137]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END300:%.*]]
+// SIMD-ONLY0:       cond.false298:
+// SIMD-ONLY0-NEXT:    [[TMP138:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV299:%.*]] = zext i16 [[TMP138]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END300]]
+// SIMD-ONLY0:       cond.end300:
+// SIMD-ONLY0-NEXT:    [[COND301:%.*]] = phi i32 [ [[CONV297]], [[COND_TRUE296]] ], [ [[CONV299]], [[COND_FALSE298]] ]
+// SIMD-ONLY0-NEXT:    [[CONV302:%.*]] = trunc i32 [[COND301]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV302]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP139:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP139]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP140:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV303:%.*]] = zext i16 [[TMP140]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP141:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV304:%.*]] = zext i16 [[TMP141]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP305:%.*]] = icmp slt i32 [[CONV303]], [[CONV304]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP305]], label [[COND_TRUE307:%.*]], label [[COND_FALSE309:%.*]]
+// SIMD-ONLY0:       cond.true307:
+// SIMD-ONLY0-NEXT:    [[TMP142:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV308:%.*]] = zext i16 [[TMP142]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END311:%.*]]
+// SIMD-ONLY0:       cond.false309:
+// SIMD-ONLY0-NEXT:    [[TMP143:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV310:%.*]] = zext i16 [[TMP143]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END311]]
+// SIMD-ONLY0:       cond.end311:
+// SIMD-ONLY0-NEXT:    [[COND312:%.*]] = phi i32 [ [[CONV308]], [[COND_TRUE307]] ], [ [[CONV310]], [[COND_FALSE309]] ]
+// SIMD-ONLY0-NEXT:    [[CONV313:%.*]] = trunc i32 [[COND312]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV313]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP144:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP144]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP145:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV314:%.*]] = zext i16 [[TMP145]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP146:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV315:%.*]] = zext i16 [[TMP146]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP316:%.*]] = icmp eq i32 [[CONV314]], [[CONV315]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP316]], label [[COND_TRUE318:%.*]], label [[COND_FALSE320:%.*]]
+// SIMD-ONLY0:       cond.true318:
+// SIMD-ONLY0-NEXT:    [[TMP147:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV319:%.*]] = zext i16 [[TMP147]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END322:%.*]]
+// SIMD-ONLY0:       cond.false320:
+// SIMD-ONLY0-NEXT:    [[TMP148:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV321:%.*]] = zext i16 [[TMP148]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END322]]
+// SIMD-ONLY0:       cond.end322:
+// SIMD-ONLY0-NEXT:    [[COND323:%.*]] = phi i32 [ [[CONV319]], [[COND_TRUE318]] ], [ [[CONV321]], [[COND_FALSE320]] ]
+// SIMD-ONLY0-NEXT:    [[CONV324:%.*]] = trunc i32 [[COND323]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV324]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP149:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP149]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP150:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP150]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP151:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV325:%.*]] = zext i16 [[TMP151]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP152:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV326:%.*]] = zext i16 [[TMP152]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP327:%.*]] = icmp sgt i32 [[CONV325]], [[CONV326]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP327]], label [[COND_TRUE329:%.*]], label [[COND_FALSE331:%.*]]
+// SIMD-ONLY0:       cond.true329:
+// SIMD-ONLY0-NEXT:    [[TMP153:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV330:%.*]] = zext i16 [[TMP153]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END333:%.*]]
+// SIMD-ONLY0:       cond.false331:
+// SIMD-ONLY0-NEXT:    [[TMP154:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV332:%.*]] = zext i16 [[TMP154]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END333]]
+// SIMD-ONLY0:       cond.end333:
+// SIMD-ONLY0-NEXT:    [[COND334:%.*]] = phi i32 [ [[CONV330]], [[COND_TRUE329]] ], [ [[CONV332]], [[COND_FALSE331]] ]
+// SIMD-ONLY0-NEXT:    [[CONV335:%.*]] = trunc i32 [[COND334]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV335]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP155:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP155]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP156:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV336:%.*]] = zext i16 [[TMP156]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP157:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV337:%.*]] = zext i16 [[TMP157]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP338:%.*]] = icmp slt i32 [[CONV336]], [[CONV337]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP338]], label [[COND_TRUE340:%.*]], label [[COND_FALSE342:%.*]]
+// SIMD-ONLY0:       cond.true340:
+// SIMD-ONLY0-NEXT:    [[TMP158:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV341:%.*]] = zext i16 [[TMP158]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END344:%.*]]
+// SIMD-ONLY0:       cond.false342:
+// SIMD-ONLY0-NEXT:    [[TMP159:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV343:%.*]] = zext i16 [[TMP159]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END344]]
+// SIMD-ONLY0:       cond.end344:
+// SIMD-ONLY0-NEXT:    [[COND345:%.*]] = phi i32 [ [[CONV341]], [[COND_TRUE340]] ], [ [[CONV343]], [[COND_FALSE342]] ]
+// SIMD-ONLY0-NEXT:    [[CONV346:%.*]] = trunc i32 [[COND345]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV346]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP160:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP160]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP161:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV347:%.*]] = zext i16 [[TMP161]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP162:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV348:%.*]] = zext i16 [[TMP162]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP349:%.*]] = icmp eq i32 [[CONV347]], [[CONV348]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP349]], label [[COND_TRUE351:%.*]], label [[COND_FALSE353:%.*]]
+// SIMD-ONLY0:       cond.true351:
+// SIMD-ONLY0-NEXT:    [[TMP163:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV352:%.*]] = zext i16 [[TMP163]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END355:%.*]]
+// SIMD-ONLY0:       cond.false353:
+// SIMD-ONLY0-NEXT:    [[TMP164:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV354:%.*]] = zext i16 [[TMP164]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END355]]
+// SIMD-ONLY0:       cond.end355:
+// SIMD-ONLY0-NEXT:    [[COND356:%.*]] = phi i32 [ [[CONV352]], [[COND_TRUE351]] ], [ [[CONV354]], [[COND_FALSE353]] ]
+// SIMD-ONLY0-NEXT:    [[CONV357:%.*]] = trunc i32 [[COND356]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV357]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP165:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV358:%.*]] = zext i16 [[TMP165]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP166:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV359:%.*]] = zext i16 [[TMP166]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP360:%.*]] = icmp sgt i32 [[CONV358]], [[CONV359]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP360]], label [[COND_TRUE362:%.*]], label [[COND_FALSE364:%.*]]
+// SIMD-ONLY0:       cond.true362:
+// SIMD-ONLY0-NEXT:    [[TMP167:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV363:%.*]] = zext i16 [[TMP167]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END366:%.*]]
+// SIMD-ONLY0:       cond.false364:
+// SIMD-ONLY0-NEXT:    [[TMP168:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV365:%.*]] = zext i16 [[TMP168]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END366]]
+// SIMD-ONLY0:       cond.end366:
+// SIMD-ONLY0-NEXT:    [[COND367:%.*]] = phi i32 [ [[CONV363]], [[COND_TRUE362]] ], [ [[CONV365]], [[COND_FALSE364]] ]
+// SIMD-ONLY0-NEXT:    [[CONV368:%.*]] = trunc i32 [[COND367]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV368]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP169:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP169]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP170:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV369:%.*]] = zext i16 [[TMP170]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP171:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV370:%.*]] = zext i16 [[TMP171]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP371:%.*]] = icmp slt i32 [[CONV369]], [[CONV370]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP371]], label [[COND_TRUE373:%.*]], label [[COND_FALSE375:%.*]]
+// SIMD-ONLY0:       cond.true373:
+// SIMD-ONLY0-NEXT:    [[TMP172:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV374:%.*]] = zext i16 [[TMP172]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END377:%.*]]
+// SIMD-ONLY0:       cond.false375:
+// SIMD-ONLY0-NEXT:    [[TMP173:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV376:%.*]] = zext i16 [[TMP173]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END377]]
+// SIMD-ONLY0:       cond.end377:
+// SIMD-ONLY0-NEXT:    [[COND378:%.*]] = phi i32 [ [[CONV374]], [[COND_TRUE373]] ], [ [[CONV376]], [[COND_FALSE375]] ]
+// SIMD-ONLY0-NEXT:    [[CONV379:%.*]] = trunc i32 [[COND378]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV379]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP174:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP174]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP175:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV380:%.*]] = zext i16 [[TMP175]] to i32
+// SIMD-ONLY0-NEXT:    [[TMP176:%.*]] = load i16, ptr [[USE]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV381:%.*]] = zext i16 [[TMP176]] to i32
+// SIMD-ONLY0-NEXT:    [[CMP382:%.*]] = icmp eq i32 [[CONV380]], [[CONV381]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP382]], label [[COND_TRUE384:%.*]], label [[COND_FALSE386:%.*]]
+// SIMD-ONLY0:       cond.true384:
+// SIMD-ONLY0-NEXT:    [[TMP177:%.*]] = load i16, ptr [[USD]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV385:%.*]] = zext i16 [[TMP177]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END388:%.*]]
+// SIMD-ONLY0:       cond.false386:
+// SIMD-ONLY0-NEXT:    [[TMP178:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[CONV387:%.*]] = zext i16 [[TMP178]] to i32
+// SIMD-ONLY0-NEXT:    br label [[COND_END388]]
+// SIMD-ONLY0:       cond.end388:
+// SIMD-ONLY0-NEXT:    [[COND389:%.*]] = phi i32 [ [[CONV385]], [[COND_TRUE384]] ], [ [[CONV387]], [[COND_FALSE386]] ]
+// SIMD-ONLY0-NEXT:    [[CONV390:%.*]] = trunc i32 [[COND389]] to i16
+// SIMD-ONLY0-NEXT:    store i16 [[CONV390]], ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP179:%.*]] = load i16, ptr [[USX]], align 2
+// SIMD-ONLY0-NEXT:    store i16 [[TMP179]], ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    [[TMP180:%.*]] = load i16, ptr [[USV]], align 2
+// SIMD-ONLY0-NEXT:    ret i16 [[TMP180]]
+//
+//
+// SIMD-ONLY0-LABEL: @ixevd(
+// SIMD-ONLY0-NEXT:  entry:
+// SIMD-ONLY0-NEXT:    [[IX:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[IV:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[IE:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[ID:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[TMP0:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP0]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[TMP1]], [[TMP2]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]]
+// SIMD-ONLY0:       cond.true:
+// SIMD-ONLY0-NEXT:    [[TMP3:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END:%.*]]
+// SIMD-ONLY0:       cond.false:
+// SIMD-ONLY0-NEXT:    [[TMP4:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END]]
+// SIMD-ONLY0:       cond.end:
+// SIMD-ONLY0-NEXT:    [[COND:%.*]] = phi i32 [ [[TMP3]], [[COND_TRUE]] ], [ [[TMP4]], [[COND_FALSE]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP5]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP6:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP7:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP1:%.*]] = icmp slt i32 [[TMP6]], [[TMP7]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1]], label [[COND_TRUE2:%.*]], label [[COND_FALSE3:%.*]]
+// SIMD-ONLY0:       cond.true2:
+// SIMD-ONLY0-NEXT:    [[TMP8:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4:%.*]]
+// SIMD-ONLY0:       cond.false3:
+// SIMD-ONLY0-NEXT:    [[TMP9:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4]]
+// SIMD-ONLY0:       cond.end4:
+// SIMD-ONLY0-NEXT:    [[COND5:%.*]] = phi i32 [ [[TMP8]], [[COND_TRUE2]] ], [ [[TMP9]], [[COND_FALSE3]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND5]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP10:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP10]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP11:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP12:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6:%.*]] = icmp eq i32 [[TMP11]], [[TMP12]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6]], label [[COND_TRUE7:%.*]], label [[COND_FALSE8:%.*]]
+// SIMD-ONLY0:       cond.true7:
+// SIMD-ONLY0-NEXT:    [[TMP13:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END9:%.*]]
+// SIMD-ONLY0:       cond.false8:
+// SIMD-ONLY0-NEXT:    [[TMP14:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END9]]
+// SIMD-ONLY0:       cond.end9:
+// SIMD-ONLY0-NEXT:    [[COND10:%.*]] = phi i32 [ [[TMP13]], [[COND_TRUE7]] ], [ [[TMP14]], [[COND_FALSE8]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND10]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP15:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP16:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP11:%.*]] = icmp sgt i32 [[TMP15]], [[TMP16]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP11]], label [[COND_TRUE12:%.*]], label [[COND_FALSE13:%.*]]
+// SIMD-ONLY0:       cond.true12:
+// SIMD-ONLY0-NEXT:    [[TMP17:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END14:%.*]]
+// SIMD-ONLY0:       cond.false13:
+// SIMD-ONLY0-NEXT:    [[TMP18:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END14]]
+// SIMD-ONLY0:       cond.end14:
+// SIMD-ONLY0-NEXT:    [[COND15:%.*]] = phi i32 [ [[TMP17]], [[COND_TRUE12]] ], [ [[TMP18]], [[COND_FALSE13]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND15]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP19:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP19]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP20:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP21:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP16:%.*]] = icmp slt i32 [[TMP20]], [[TMP21]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP16]], label [[COND_TRUE17:%.*]], label [[COND_FALSE18:%.*]]
+// SIMD-ONLY0:       cond.true17:
+// SIMD-ONLY0-NEXT:    [[TMP22:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END19:%.*]]
+// SIMD-ONLY0:       cond.false18:
+// SIMD-ONLY0-NEXT:    [[TMP23:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END19]]
+// SIMD-ONLY0:       cond.end19:
+// SIMD-ONLY0-NEXT:    [[COND20:%.*]] = phi i32 [ [[TMP22]], [[COND_TRUE17]] ], [ [[TMP23]], [[COND_FALSE18]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND20]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP24:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP24]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP25:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP26:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP21:%.*]] = icmp eq i32 [[TMP25]], [[TMP26]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP21]], label [[COND_TRUE22:%.*]], label [[COND_FALSE23:%.*]]
+// SIMD-ONLY0:       cond.true22:
+// SIMD-ONLY0-NEXT:    [[TMP27:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END24:%.*]]
+// SIMD-ONLY0:       cond.false23:
+// SIMD-ONLY0-NEXT:    [[TMP28:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END24]]
+// SIMD-ONLY0:       cond.end24:
+// SIMD-ONLY0-NEXT:    [[COND25:%.*]] = phi i32 [ [[TMP27]], [[COND_TRUE22]] ], [ [[TMP28]], [[COND_FALSE23]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND25]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP29:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP29]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP30:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP30]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP31:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP32:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP26:%.*]] = icmp sgt i32 [[TMP31]], [[TMP32]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP26]], label [[COND_TRUE27:%.*]], label [[COND_FALSE28:%.*]]
+// SIMD-ONLY0:       cond.true27:
+// SIMD-ONLY0-NEXT:    [[TMP33:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END29:%.*]]
+// SIMD-ONLY0:       cond.false28:
+// SIMD-ONLY0-NEXT:    [[TMP34:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END29]]
+// SIMD-ONLY0:       cond.end29:
+// SIMD-ONLY0-NEXT:    [[COND30:%.*]] = phi i32 [ [[TMP33]], [[COND_TRUE27]] ], [ [[TMP34]], [[COND_FALSE28]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND30]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP35:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP35]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP36:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP37:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP31:%.*]] = icmp slt i32 [[TMP36]], [[TMP37]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP31]], label [[COND_TRUE32:%.*]], label [[COND_FALSE33:%.*]]
+// SIMD-ONLY0:       cond.true32:
+// SIMD-ONLY0-NEXT:    [[TMP38:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END34:%.*]]
+// SIMD-ONLY0:       cond.false33:
+// SIMD-ONLY0-NEXT:    [[TMP39:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END34]]
+// SIMD-ONLY0:       cond.end34:
+// SIMD-ONLY0-NEXT:    [[COND35:%.*]] = phi i32 [ [[TMP38]], [[COND_TRUE32]] ], [ [[TMP39]], [[COND_FALSE33]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND35]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP40:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP40]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP41:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP42:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP36:%.*]] = icmp eq i32 [[TMP41]], [[TMP42]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP36]], label [[COND_TRUE37:%.*]], label [[COND_FALSE38:%.*]]
+// SIMD-ONLY0:       cond.true37:
+// SIMD-ONLY0-NEXT:    [[TMP43:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END39:%.*]]
+// SIMD-ONLY0:       cond.false38:
+// SIMD-ONLY0-NEXT:    [[TMP44:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END39]]
+// SIMD-ONLY0:       cond.end39:
+// SIMD-ONLY0-NEXT:    [[COND40:%.*]] = phi i32 [ [[TMP43]], [[COND_TRUE37]] ], [ [[TMP44]], [[COND_FALSE38]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND40]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP45:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP46:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP41:%.*]] = icmp sgt i32 [[TMP45]], [[TMP46]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP41]], label [[COND_TRUE42:%.*]], label [[COND_FALSE43:%.*]]
+// SIMD-ONLY0:       cond.true42:
+// SIMD-ONLY0-NEXT:    [[TMP47:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END44:%.*]]
+// SIMD-ONLY0:       cond.false43:
+// SIMD-ONLY0-NEXT:    [[TMP48:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END44]]
+// SIMD-ONLY0:       cond.end44:
+// SIMD-ONLY0-NEXT:    [[COND45:%.*]] = phi i32 [ [[TMP47]], [[COND_TRUE42]] ], [ [[TMP48]], [[COND_FALSE43]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND45]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP49:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP49]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP50:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP51:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP46:%.*]] = icmp slt i32 [[TMP50]], [[TMP51]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP46]], label [[COND_TRUE47:%.*]], label [[COND_FALSE48:%.*]]
+// SIMD-ONLY0:       cond.true47:
+// SIMD-ONLY0-NEXT:    [[TMP52:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END49:%.*]]
+// SIMD-ONLY0:       cond.false48:
+// SIMD-ONLY0-NEXT:    [[TMP53:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END49]]
+// SIMD-ONLY0:       cond.end49:
+// SIMD-ONLY0-NEXT:    [[COND50:%.*]] = phi i32 [ [[TMP52]], [[COND_TRUE47]] ], [ [[TMP53]], [[COND_FALSE48]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND50]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP54:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP54]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP55:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP56:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP51:%.*]] = icmp eq i32 [[TMP55]], [[TMP56]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP51]], label [[COND_TRUE52:%.*]], label [[COND_FALSE53:%.*]]
+// SIMD-ONLY0:       cond.true52:
+// SIMD-ONLY0-NEXT:    [[TMP57:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END54:%.*]]
+// SIMD-ONLY0:       cond.false53:
+// SIMD-ONLY0-NEXT:    [[TMP58:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END54]]
+// SIMD-ONLY0:       cond.end54:
+// SIMD-ONLY0-NEXT:    [[COND55:%.*]] = phi i32 [ [[TMP57]], [[COND_TRUE52]] ], [ [[TMP58]], [[COND_FALSE53]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND55]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP59:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP59]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP60:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP60]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP61:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP62:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP56:%.*]] = icmp sgt i32 [[TMP61]], [[TMP62]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP56]], label [[COND_TRUE57:%.*]], label [[COND_FALSE58:%.*]]
+// SIMD-ONLY0:       cond.true57:
+// SIMD-ONLY0-NEXT:    [[TMP63:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END59:%.*]]
+// SIMD-ONLY0:       cond.false58:
+// SIMD-ONLY0-NEXT:    [[TMP64:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END59]]
+// SIMD-ONLY0:       cond.end59:
+// SIMD-ONLY0-NEXT:    [[COND60:%.*]] = phi i32 [ [[TMP63]], [[COND_TRUE57]] ], [ [[TMP64]], [[COND_FALSE58]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND60]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP65:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP65]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP66:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP67:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP61:%.*]] = icmp slt i32 [[TMP66]], [[TMP67]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP61]], label [[COND_TRUE62:%.*]], label [[COND_FALSE63:%.*]]
+// SIMD-ONLY0:       cond.true62:
+// SIMD-ONLY0-NEXT:    [[TMP68:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END64:%.*]]
+// SIMD-ONLY0:       cond.false63:
+// SIMD-ONLY0-NEXT:    [[TMP69:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END64]]
+// SIMD-ONLY0:       cond.end64:
+// SIMD-ONLY0-NEXT:    [[COND65:%.*]] = phi i32 [ [[TMP68]], [[COND_TRUE62]] ], [ [[TMP69]], [[COND_FALSE63]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND65]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP70:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP70]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP71:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP72:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP66:%.*]] = icmp eq i32 [[TMP71]], [[TMP72]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP66]], label [[COND_TRUE67:%.*]], label [[COND_FALSE68:%.*]]
+// SIMD-ONLY0:       cond.true67:
+// SIMD-ONLY0-NEXT:    [[TMP73:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END69:%.*]]
+// SIMD-ONLY0:       cond.false68:
+// SIMD-ONLY0-NEXT:    [[TMP74:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END69]]
+// SIMD-ONLY0:       cond.end69:
+// SIMD-ONLY0-NEXT:    [[COND70:%.*]] = phi i32 [ [[TMP73]], [[COND_TRUE67]] ], [ [[TMP74]], [[COND_FALSE68]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND70]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP75:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP76:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP71:%.*]] = icmp sgt i32 [[TMP75]], [[TMP76]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP71]], label [[COND_TRUE72:%.*]], label [[COND_FALSE73:%.*]]
+// SIMD-ONLY0:       cond.true72:
+// SIMD-ONLY0-NEXT:    [[TMP77:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END74:%.*]]
+// SIMD-ONLY0:       cond.false73:
+// SIMD-ONLY0-NEXT:    [[TMP78:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END74]]
+// SIMD-ONLY0:       cond.end74:
+// SIMD-ONLY0-NEXT:    [[COND75:%.*]] = phi i32 [ [[TMP77]], [[COND_TRUE72]] ], [ [[TMP78]], [[COND_FALSE73]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND75]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP79:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP79]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP80:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP81:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP76:%.*]] = icmp slt i32 [[TMP80]], [[TMP81]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP76]], label [[COND_TRUE77:%.*]], label [[COND_FALSE78:%.*]]
+// SIMD-ONLY0:       cond.true77:
+// SIMD-ONLY0-NEXT:    [[TMP82:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END79:%.*]]
+// SIMD-ONLY0:       cond.false78:
+// SIMD-ONLY0-NEXT:    [[TMP83:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END79]]
+// SIMD-ONLY0:       cond.end79:
+// SIMD-ONLY0-NEXT:    [[COND80:%.*]] = phi i32 [ [[TMP82]], [[COND_TRUE77]] ], [ [[TMP83]], [[COND_FALSE78]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND80]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP84:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP84]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP85:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP86:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP81:%.*]] = icmp eq i32 [[TMP85]], [[TMP86]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP81]], label [[COND_TRUE82:%.*]], label [[COND_FALSE83:%.*]]
+// SIMD-ONLY0:       cond.true82:
+// SIMD-ONLY0-NEXT:    [[TMP87:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END84:%.*]]
+// SIMD-ONLY0:       cond.false83:
+// SIMD-ONLY0-NEXT:    [[TMP88:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END84]]
+// SIMD-ONLY0:       cond.end84:
+// SIMD-ONLY0-NEXT:    [[COND85:%.*]] = phi i32 [ [[TMP87]], [[COND_TRUE82]] ], [ [[TMP88]], [[COND_FALSE83]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND85]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP89:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP89]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP90:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP90]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP91:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP92:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP86:%.*]] = icmp sgt i32 [[TMP91]], [[TMP92]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP86]], label [[COND_TRUE87:%.*]], label [[COND_FALSE88:%.*]]
+// SIMD-ONLY0:       cond.true87:
+// SIMD-ONLY0-NEXT:    [[TMP93:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END89:%.*]]
+// SIMD-ONLY0:       cond.false88:
+// SIMD-ONLY0-NEXT:    [[TMP94:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END89]]
+// SIMD-ONLY0:       cond.end89:
+// SIMD-ONLY0-NEXT:    [[COND90:%.*]] = phi i32 [ [[TMP93]], [[COND_TRUE87]] ], [ [[TMP94]], [[COND_FALSE88]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND90]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP95:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP95]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP96:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP97:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP91:%.*]] = icmp slt i32 [[TMP96]], [[TMP97]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP91]], label [[COND_TRUE92:%.*]], label [[COND_FALSE93:%.*]]
+// SIMD-ONLY0:       cond.true92:
+// SIMD-ONLY0-NEXT:    [[TMP98:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END94:%.*]]
+// SIMD-ONLY0:       cond.false93:
+// SIMD-ONLY0-NEXT:    [[TMP99:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END94]]
+// SIMD-ONLY0:       cond.end94:
+// SIMD-ONLY0-NEXT:    [[COND95:%.*]] = phi i32 [ [[TMP98]], [[COND_TRUE92]] ], [ [[TMP99]], [[COND_FALSE93]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND95]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP100:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP100]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP101:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP102:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP96:%.*]] = icmp eq i32 [[TMP101]], [[TMP102]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP96]], label [[COND_TRUE97:%.*]], label [[COND_FALSE98:%.*]]
+// SIMD-ONLY0:       cond.true97:
+// SIMD-ONLY0-NEXT:    [[TMP103:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END99:%.*]]
+// SIMD-ONLY0:       cond.false98:
+// SIMD-ONLY0-NEXT:    [[TMP104:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END99]]
+// SIMD-ONLY0:       cond.end99:
+// SIMD-ONLY0-NEXT:    [[COND100:%.*]] = phi i32 [ [[TMP103]], [[COND_TRUE97]] ], [ [[TMP104]], [[COND_FALSE98]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND100]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP105:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP106:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP101:%.*]] = icmp sgt i32 [[TMP105]], [[TMP106]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP101]], label [[COND_TRUE102:%.*]], label [[COND_FALSE103:%.*]]
+// SIMD-ONLY0:       cond.true102:
+// SIMD-ONLY0-NEXT:    [[TMP107:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END104:%.*]]
+// SIMD-ONLY0:       cond.false103:
+// SIMD-ONLY0-NEXT:    [[TMP108:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END104]]
+// SIMD-ONLY0:       cond.end104:
+// SIMD-ONLY0-NEXT:    [[COND105:%.*]] = phi i32 [ [[TMP107]], [[COND_TRUE102]] ], [ [[TMP108]], [[COND_FALSE103]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND105]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP109:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP109]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP110:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP111:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP106:%.*]] = icmp slt i32 [[TMP110]], [[TMP111]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP106]], label [[COND_TRUE107:%.*]], label [[COND_FALSE108:%.*]]
+// SIMD-ONLY0:       cond.true107:
+// SIMD-ONLY0-NEXT:    [[TMP112:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END109:%.*]]
+// SIMD-ONLY0:       cond.false108:
+// SIMD-ONLY0-NEXT:    [[TMP113:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END109]]
+// SIMD-ONLY0:       cond.end109:
+// SIMD-ONLY0-NEXT:    [[COND110:%.*]] = phi i32 [ [[TMP112]], [[COND_TRUE107]] ], [ [[TMP113]], [[COND_FALSE108]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND110]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP114:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP114]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP115:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP116:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP111:%.*]] = icmp eq i32 [[TMP115]], [[TMP116]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP111]], label [[COND_TRUE112:%.*]], label [[COND_FALSE113:%.*]]
+// SIMD-ONLY0:       cond.true112:
+// SIMD-ONLY0-NEXT:    [[TMP117:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END114:%.*]]
+// SIMD-ONLY0:       cond.false113:
+// SIMD-ONLY0-NEXT:    [[TMP118:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END114]]
+// SIMD-ONLY0:       cond.end114:
+// SIMD-ONLY0-NEXT:    [[COND115:%.*]] = phi i32 [ [[TMP117]], [[COND_TRUE112]] ], [ [[TMP118]], [[COND_FALSE113]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND115]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP119:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP119]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP120:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP120]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP121:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP122:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP116:%.*]] = icmp sgt i32 [[TMP121]], [[TMP122]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP116]], label [[COND_TRUE117:%.*]], label [[COND_FALSE118:%.*]]
+// SIMD-ONLY0:       cond.true117:
+// SIMD-ONLY0-NEXT:    [[TMP123:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END119:%.*]]
+// SIMD-ONLY0:       cond.false118:
+// SIMD-ONLY0-NEXT:    [[TMP124:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END119]]
+// SIMD-ONLY0:       cond.end119:
+// SIMD-ONLY0-NEXT:    [[COND120:%.*]] = phi i32 [ [[TMP123]], [[COND_TRUE117]] ], [ [[TMP124]], [[COND_FALSE118]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND120]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP125:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP125]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP126:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP127:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP121:%.*]] = icmp slt i32 [[TMP126]], [[TMP127]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP121]], label [[COND_TRUE122:%.*]], label [[COND_FALSE123:%.*]]
+// SIMD-ONLY0:       cond.true122:
+// SIMD-ONLY0-NEXT:    [[TMP128:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END124:%.*]]
+// SIMD-ONLY0:       cond.false123:
+// SIMD-ONLY0-NEXT:    [[TMP129:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END124]]
+// SIMD-ONLY0:       cond.end124:
+// SIMD-ONLY0-NEXT:    [[COND125:%.*]] = phi i32 [ [[TMP128]], [[COND_TRUE122]] ], [ [[TMP129]], [[COND_FALSE123]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND125]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP130:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP130]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP131:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP132:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP126:%.*]] = icmp eq i32 [[TMP131]], [[TMP132]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP126]], label [[COND_TRUE127:%.*]], label [[COND_FALSE128:%.*]]
+// SIMD-ONLY0:       cond.true127:
+// SIMD-ONLY0-NEXT:    [[TMP133:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END129:%.*]]
+// SIMD-ONLY0:       cond.false128:
+// SIMD-ONLY0-NEXT:    [[TMP134:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END129]]
+// SIMD-ONLY0:       cond.end129:
+// SIMD-ONLY0-NEXT:    [[COND130:%.*]] = phi i32 [ [[TMP133]], [[COND_TRUE127]] ], [ [[TMP134]], [[COND_FALSE128]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND130]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP135:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP136:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP131:%.*]] = icmp sgt i32 [[TMP135]], [[TMP136]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP131]], label [[COND_TRUE132:%.*]], label [[COND_FALSE133:%.*]]
+// SIMD-ONLY0:       cond.true132:
+// SIMD-ONLY0-NEXT:    [[TMP137:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END134:%.*]]
+// SIMD-ONLY0:       cond.false133:
+// SIMD-ONLY0-NEXT:    [[TMP138:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END134]]
+// SIMD-ONLY0:       cond.end134:
+// SIMD-ONLY0-NEXT:    [[COND135:%.*]] = phi i32 [ [[TMP137]], [[COND_TRUE132]] ], [ [[TMP138]], [[COND_FALSE133]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND135]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP139:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP139]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP140:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP141:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP136:%.*]] = icmp slt i32 [[TMP140]], [[TMP141]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP136]], label [[COND_TRUE137:%.*]], label [[COND_FALSE138:%.*]]
+// SIMD-ONLY0:       cond.true137:
+// SIMD-ONLY0-NEXT:    [[TMP142:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END139:%.*]]
+// SIMD-ONLY0:       cond.false138:
+// SIMD-ONLY0-NEXT:    [[TMP143:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END139]]
+// SIMD-ONLY0:       cond.end139:
+// SIMD-ONLY0-NEXT:    [[COND140:%.*]] = phi i32 [ [[TMP142]], [[COND_TRUE137]] ], [ [[TMP143]], [[COND_FALSE138]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND140]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP144:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP144]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP145:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP146:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP141:%.*]] = icmp eq i32 [[TMP145]], [[TMP146]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP141]], label [[COND_TRUE142:%.*]], label [[COND_FALSE143:%.*]]
+// SIMD-ONLY0:       cond.true142:
+// SIMD-ONLY0-NEXT:    [[TMP147:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END144:%.*]]
+// SIMD-ONLY0:       cond.false143:
+// SIMD-ONLY0-NEXT:    [[TMP148:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END144]]
+// SIMD-ONLY0:       cond.end144:
+// SIMD-ONLY0-NEXT:    [[COND145:%.*]] = phi i32 [ [[TMP147]], [[COND_TRUE142]] ], [ [[TMP148]], [[COND_FALSE143]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND145]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP149:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP149]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP150:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP150]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP151:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP152:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP146:%.*]] = icmp sgt i32 [[TMP151]], [[TMP152]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP146]], label [[COND_TRUE147:%.*]], label [[COND_FALSE148:%.*]]
+// SIMD-ONLY0:       cond.true147:
+// SIMD-ONLY0-NEXT:    [[TMP153:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END149:%.*]]
+// SIMD-ONLY0:       cond.false148:
+// SIMD-ONLY0-NEXT:    [[TMP154:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END149]]
+// SIMD-ONLY0:       cond.end149:
+// SIMD-ONLY0-NEXT:    [[COND150:%.*]] = phi i32 [ [[TMP153]], [[COND_TRUE147]] ], [ [[TMP154]], [[COND_FALSE148]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND150]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP155:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP155]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP156:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP157:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP151:%.*]] = icmp slt i32 [[TMP156]], [[TMP157]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP151]], label [[COND_TRUE152:%.*]], label [[COND_FALSE153:%.*]]
+// SIMD-ONLY0:       cond.true152:
+// SIMD-ONLY0-NEXT:    [[TMP158:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END154:%.*]]
+// SIMD-ONLY0:       cond.false153:
+// SIMD-ONLY0-NEXT:    [[TMP159:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END154]]
+// SIMD-ONLY0:       cond.end154:
+// SIMD-ONLY0-NEXT:    [[COND155:%.*]] = phi i32 [ [[TMP158]], [[COND_TRUE152]] ], [ [[TMP159]], [[COND_FALSE153]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND155]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP160:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP160]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP161:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP162:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP156:%.*]] = icmp eq i32 [[TMP161]], [[TMP162]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP156]], label [[COND_TRUE157:%.*]], label [[COND_FALSE158:%.*]]
+// SIMD-ONLY0:       cond.true157:
+// SIMD-ONLY0-NEXT:    [[TMP163:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END159:%.*]]
+// SIMD-ONLY0:       cond.false158:
+// SIMD-ONLY0-NEXT:    [[TMP164:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END159]]
+// SIMD-ONLY0:       cond.end159:
+// SIMD-ONLY0-NEXT:    [[COND160:%.*]] = phi i32 [ [[TMP163]], [[COND_TRUE157]] ], [ [[TMP164]], [[COND_FALSE158]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND160]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP165:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP166:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP161:%.*]] = icmp sgt i32 [[TMP165]], [[TMP166]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP161]], label [[COND_TRUE162:%.*]], label [[COND_FALSE163:%.*]]
+// SIMD-ONLY0:       cond.true162:
+// SIMD-ONLY0-NEXT:    [[TMP167:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END164:%.*]]
+// SIMD-ONLY0:       cond.false163:
+// SIMD-ONLY0-NEXT:    [[TMP168:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END164]]
+// SIMD-ONLY0:       cond.end164:
+// SIMD-ONLY0-NEXT:    [[COND165:%.*]] = phi i32 [ [[TMP167]], [[COND_TRUE162]] ], [ [[TMP168]], [[COND_FALSE163]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND165]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP169:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP169]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP170:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP171:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP166:%.*]] = icmp slt i32 [[TMP170]], [[TMP171]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP166]], label [[COND_TRUE167:%.*]], label [[COND_FALSE168:%.*]]
+// SIMD-ONLY0:       cond.true167:
+// SIMD-ONLY0-NEXT:    [[TMP172:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END169:%.*]]
+// SIMD-ONLY0:       cond.false168:
+// SIMD-ONLY0-NEXT:    [[TMP173:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END169]]
+// SIMD-ONLY0:       cond.end169:
+// SIMD-ONLY0-NEXT:    [[COND170:%.*]] = phi i32 [ [[TMP172]], [[COND_TRUE167]] ], [ [[TMP173]], [[COND_FALSE168]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND170]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP174:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP174]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP175:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP176:%.*]] = load i32, ptr [[IE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP171:%.*]] = icmp eq i32 [[TMP175]], [[TMP176]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP171]], label [[COND_TRUE172:%.*]], label [[COND_FALSE173:%.*]]
+// SIMD-ONLY0:       cond.true172:
+// SIMD-ONLY0-NEXT:    [[TMP177:%.*]] = load i32, ptr [[ID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END174:%.*]]
+// SIMD-ONLY0:       cond.false173:
+// SIMD-ONLY0-NEXT:    [[TMP178:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END174]]
+// SIMD-ONLY0:       cond.end174:
+// SIMD-ONLY0-NEXT:    [[COND175:%.*]] = phi i32 [ [[TMP177]], [[COND_TRUE172]] ], [ [[TMP178]], [[COND_FALSE173]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND175]], ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP179:%.*]] = load i32, ptr [[IX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP179]], ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP180:%.*]] = load i32, ptr [[IV]], align 4
+// SIMD-ONLY0-NEXT:    ret i32 [[TMP180]]
+//
+//
+// SIMD-ONLY0-LABEL: @uixevd(
+// SIMD-ONLY0-NEXT:  entry:
+// SIMD-ONLY0-NEXT:    [[UIX:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[UIV:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[UIE:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[UID:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[TMP0:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP0]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP:%.*]] = icmp ugt i32 [[TMP1]], [[TMP2]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]]
+// SIMD-ONLY0:       cond.true:
+// SIMD-ONLY0-NEXT:    [[TMP3:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END:%.*]]
+// SIMD-ONLY0:       cond.false:
+// SIMD-ONLY0-NEXT:    [[TMP4:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END]]
+// SIMD-ONLY0:       cond.end:
+// SIMD-ONLY0-NEXT:    [[COND:%.*]] = phi i32 [ [[TMP3]], [[COND_TRUE]] ], [ [[TMP4]], [[COND_FALSE]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP5]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP6:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP7:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP1:%.*]] = icmp ult i32 [[TMP6]], [[TMP7]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1]], label [[COND_TRUE2:%.*]], label [[COND_FALSE3:%.*]]
+// SIMD-ONLY0:       cond.true2:
+// SIMD-ONLY0-NEXT:    [[TMP8:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4:%.*]]
+// SIMD-ONLY0:       cond.false3:
+// SIMD-ONLY0-NEXT:    [[TMP9:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4]]
+// SIMD-ONLY0:       cond.end4:
+// SIMD-ONLY0-NEXT:    [[COND5:%.*]] = phi i32 [ [[TMP8]], [[COND_TRUE2]] ], [ [[TMP9]], [[COND_FALSE3]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND5]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP10:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP10]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP11:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP12:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6:%.*]] = icmp eq i32 [[TMP11]], [[TMP12]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6]], label [[COND_TRUE7:%.*]], label [[COND_FALSE8:%.*]]
+// SIMD-ONLY0:       cond.true7:
+// SIMD-ONLY0-NEXT:    [[TMP13:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END9:%.*]]
+// SIMD-ONLY0:       cond.false8:
+// SIMD-ONLY0-NEXT:    [[TMP14:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END9]]
+// SIMD-ONLY0:       cond.end9:
+// SIMD-ONLY0-NEXT:    [[COND10:%.*]] = phi i32 [ [[TMP13]], [[COND_TRUE7]] ], [ [[TMP14]], [[COND_FALSE8]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND10]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP15:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP16:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP11:%.*]] = icmp ugt i32 [[TMP15]], [[TMP16]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP11]], label [[COND_TRUE12:%.*]], label [[COND_FALSE13:%.*]]
+// SIMD-ONLY0:       cond.true12:
+// SIMD-ONLY0-NEXT:    [[TMP17:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END14:%.*]]
+// SIMD-ONLY0:       cond.false13:
+// SIMD-ONLY0-NEXT:    [[TMP18:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END14]]
+// SIMD-ONLY0:       cond.end14:
+// SIMD-ONLY0-NEXT:    [[COND15:%.*]] = phi i32 [ [[TMP17]], [[COND_TRUE12]] ], [ [[TMP18]], [[COND_FALSE13]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND15]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP19:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP19]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP20:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP21:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP16:%.*]] = icmp ult i32 [[TMP20]], [[TMP21]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP16]], label [[COND_TRUE17:%.*]], label [[COND_FALSE18:%.*]]
+// SIMD-ONLY0:       cond.true17:
+// SIMD-ONLY0-NEXT:    [[TMP22:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END19:%.*]]
+// SIMD-ONLY0:       cond.false18:
+// SIMD-ONLY0-NEXT:    [[TMP23:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END19]]
+// SIMD-ONLY0:       cond.end19:
+// SIMD-ONLY0-NEXT:    [[COND20:%.*]] = phi i32 [ [[TMP22]], [[COND_TRUE17]] ], [ [[TMP23]], [[COND_FALSE18]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND20]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP24:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP24]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP25:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP26:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP21:%.*]] = icmp eq i32 [[TMP25]], [[TMP26]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP21]], label [[COND_TRUE22:%.*]], label [[COND_FALSE23:%.*]]
+// SIMD-ONLY0:       cond.true22:
+// SIMD-ONLY0-NEXT:    [[TMP27:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END24:%.*]]
+// SIMD-ONLY0:       cond.false23:
+// SIMD-ONLY0-NEXT:    [[TMP28:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END24]]
+// SIMD-ONLY0:       cond.end24:
+// SIMD-ONLY0-NEXT:    [[COND25:%.*]] = phi i32 [ [[TMP27]], [[COND_TRUE22]] ], [ [[TMP28]], [[COND_FALSE23]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND25]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP29:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP29]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP30:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP30]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP31:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP32:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP26:%.*]] = icmp ugt i32 [[TMP31]], [[TMP32]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP26]], label [[COND_TRUE27:%.*]], label [[COND_FALSE28:%.*]]
+// SIMD-ONLY0:       cond.true27:
+// SIMD-ONLY0-NEXT:    [[TMP33:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END29:%.*]]
+// SIMD-ONLY0:       cond.false28:
+// SIMD-ONLY0-NEXT:    [[TMP34:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END29]]
+// SIMD-ONLY0:       cond.end29:
+// SIMD-ONLY0-NEXT:    [[COND30:%.*]] = phi i32 [ [[TMP33]], [[COND_TRUE27]] ], [ [[TMP34]], [[COND_FALSE28]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND30]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP35:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP35]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP36:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP37:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP31:%.*]] = icmp ult i32 [[TMP36]], [[TMP37]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP31]], label [[COND_TRUE32:%.*]], label [[COND_FALSE33:%.*]]
+// SIMD-ONLY0:       cond.true32:
+// SIMD-ONLY0-NEXT:    [[TMP38:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END34:%.*]]
+// SIMD-ONLY0:       cond.false33:
+// SIMD-ONLY0-NEXT:    [[TMP39:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END34]]
+// SIMD-ONLY0:       cond.end34:
+// SIMD-ONLY0-NEXT:    [[COND35:%.*]] = phi i32 [ [[TMP38]], [[COND_TRUE32]] ], [ [[TMP39]], [[COND_FALSE33]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND35]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP40:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP40]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP41:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP42:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP36:%.*]] = icmp eq i32 [[TMP41]], [[TMP42]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP36]], label [[COND_TRUE37:%.*]], label [[COND_FALSE38:%.*]]
+// SIMD-ONLY0:       cond.true37:
+// SIMD-ONLY0-NEXT:    [[TMP43:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END39:%.*]]
+// SIMD-ONLY0:       cond.false38:
+// SIMD-ONLY0-NEXT:    [[TMP44:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END39]]
+// SIMD-ONLY0:       cond.end39:
+// SIMD-ONLY0-NEXT:    [[COND40:%.*]] = phi i32 [ [[TMP43]], [[COND_TRUE37]] ], [ [[TMP44]], [[COND_FALSE38]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND40]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP45:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP46:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP41:%.*]] = icmp ugt i32 [[TMP45]], [[TMP46]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP41]], label [[COND_TRUE42:%.*]], label [[COND_FALSE43:%.*]]
+// SIMD-ONLY0:       cond.true42:
+// SIMD-ONLY0-NEXT:    [[TMP47:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END44:%.*]]
+// SIMD-ONLY0:       cond.false43:
+// SIMD-ONLY0-NEXT:    [[TMP48:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END44]]
+// SIMD-ONLY0:       cond.end44:
+// SIMD-ONLY0-NEXT:    [[COND45:%.*]] = phi i32 [ [[TMP47]], [[COND_TRUE42]] ], [ [[TMP48]], [[COND_FALSE43]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND45]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP49:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP49]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP50:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP51:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP46:%.*]] = icmp ult i32 [[TMP50]], [[TMP51]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP46]], label [[COND_TRUE47:%.*]], label [[COND_FALSE48:%.*]]
+// SIMD-ONLY0:       cond.true47:
+// SIMD-ONLY0-NEXT:    [[TMP52:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END49:%.*]]
+// SIMD-ONLY0:       cond.false48:
+// SIMD-ONLY0-NEXT:    [[TMP53:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END49]]
+// SIMD-ONLY0:       cond.end49:
+// SIMD-ONLY0-NEXT:    [[COND50:%.*]] = phi i32 [ [[TMP52]], [[COND_TRUE47]] ], [ [[TMP53]], [[COND_FALSE48]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND50]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP54:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP54]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP55:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP56:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP51:%.*]] = icmp eq i32 [[TMP55]], [[TMP56]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP51]], label [[COND_TRUE52:%.*]], label [[COND_FALSE53:%.*]]
+// SIMD-ONLY0:       cond.true52:
+// SIMD-ONLY0-NEXT:    [[TMP57:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END54:%.*]]
+// SIMD-ONLY0:       cond.false53:
+// SIMD-ONLY0-NEXT:    [[TMP58:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END54]]
+// SIMD-ONLY0:       cond.end54:
+// SIMD-ONLY0-NEXT:    [[COND55:%.*]] = phi i32 [ [[TMP57]], [[COND_TRUE52]] ], [ [[TMP58]], [[COND_FALSE53]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND55]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP59:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP59]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP60:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP60]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP61:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP62:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP56:%.*]] = icmp ugt i32 [[TMP61]], [[TMP62]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP56]], label [[COND_TRUE57:%.*]], label [[COND_FALSE58:%.*]]
+// SIMD-ONLY0:       cond.true57:
+// SIMD-ONLY0-NEXT:    [[TMP63:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END59:%.*]]
+// SIMD-ONLY0:       cond.false58:
+// SIMD-ONLY0-NEXT:    [[TMP64:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END59]]
+// SIMD-ONLY0:       cond.end59:
+// SIMD-ONLY0-NEXT:    [[COND60:%.*]] = phi i32 [ [[TMP63]], [[COND_TRUE57]] ], [ [[TMP64]], [[COND_FALSE58]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND60]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP65:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP65]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP66:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP67:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP61:%.*]] = icmp ult i32 [[TMP66]], [[TMP67]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP61]], label [[COND_TRUE62:%.*]], label [[COND_FALSE63:%.*]]
+// SIMD-ONLY0:       cond.true62:
+// SIMD-ONLY0-NEXT:    [[TMP68:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END64:%.*]]
+// SIMD-ONLY0:       cond.false63:
+// SIMD-ONLY0-NEXT:    [[TMP69:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END64]]
+// SIMD-ONLY0:       cond.end64:
+// SIMD-ONLY0-NEXT:    [[COND65:%.*]] = phi i32 [ [[TMP68]], [[COND_TRUE62]] ], [ [[TMP69]], [[COND_FALSE63]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND65]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP70:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP70]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP71:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP72:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP66:%.*]] = icmp eq i32 [[TMP71]], [[TMP72]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP66]], label [[COND_TRUE67:%.*]], label [[COND_FALSE68:%.*]]
+// SIMD-ONLY0:       cond.true67:
+// SIMD-ONLY0-NEXT:    [[TMP73:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END69:%.*]]
+// SIMD-ONLY0:       cond.false68:
+// SIMD-ONLY0-NEXT:    [[TMP74:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END69]]
+// SIMD-ONLY0:       cond.end69:
+// SIMD-ONLY0-NEXT:    [[COND70:%.*]] = phi i32 [ [[TMP73]], [[COND_TRUE67]] ], [ [[TMP74]], [[COND_FALSE68]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND70]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP75:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP76:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP71:%.*]] = icmp ugt i32 [[TMP75]], [[TMP76]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP71]], label [[COND_TRUE72:%.*]], label [[COND_FALSE73:%.*]]
+// SIMD-ONLY0:       cond.true72:
+// SIMD-ONLY0-NEXT:    [[TMP77:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END74:%.*]]
+// SIMD-ONLY0:       cond.false73:
+// SIMD-ONLY0-NEXT:    [[TMP78:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END74]]
+// SIMD-ONLY0:       cond.end74:
+// SIMD-ONLY0-NEXT:    [[COND75:%.*]] = phi i32 [ [[TMP77]], [[COND_TRUE72]] ], [ [[TMP78]], [[COND_FALSE73]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND75]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP79:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP79]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP80:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP81:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP76:%.*]] = icmp ult i32 [[TMP80]], [[TMP81]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP76]], label [[COND_TRUE77:%.*]], label [[COND_FALSE78:%.*]]
+// SIMD-ONLY0:       cond.true77:
+// SIMD-ONLY0-NEXT:    [[TMP82:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END79:%.*]]
+// SIMD-ONLY0:       cond.false78:
+// SIMD-ONLY0-NEXT:    [[TMP83:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END79]]
+// SIMD-ONLY0:       cond.end79:
+// SIMD-ONLY0-NEXT:    [[COND80:%.*]] = phi i32 [ [[TMP82]], [[COND_TRUE77]] ], [ [[TMP83]], [[COND_FALSE78]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND80]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP84:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP84]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP85:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP86:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP81:%.*]] = icmp eq i32 [[TMP85]], [[TMP86]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP81]], label [[COND_TRUE82:%.*]], label [[COND_FALSE83:%.*]]
+// SIMD-ONLY0:       cond.true82:
+// SIMD-ONLY0-NEXT:    [[TMP87:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END84:%.*]]
+// SIMD-ONLY0:       cond.false83:
+// SIMD-ONLY0-NEXT:    [[TMP88:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END84]]
+// SIMD-ONLY0:       cond.end84:
+// SIMD-ONLY0-NEXT:    [[COND85:%.*]] = phi i32 [ [[TMP87]], [[COND_TRUE82]] ], [ [[TMP88]], [[COND_FALSE83]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND85]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP89:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP89]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP90:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP90]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP91:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP92:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP86:%.*]] = icmp ugt i32 [[TMP91]], [[TMP92]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP86]], label [[COND_TRUE87:%.*]], label [[COND_FALSE88:%.*]]
+// SIMD-ONLY0:       cond.true87:
+// SIMD-ONLY0-NEXT:    [[TMP93:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END89:%.*]]
+// SIMD-ONLY0:       cond.false88:
+// SIMD-ONLY0-NEXT:    [[TMP94:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END89]]
+// SIMD-ONLY0:       cond.end89:
+// SIMD-ONLY0-NEXT:    [[COND90:%.*]] = phi i32 [ [[TMP93]], [[COND_TRUE87]] ], [ [[TMP94]], [[COND_FALSE88]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND90]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP95:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP95]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP96:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP97:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP91:%.*]] = icmp ult i32 [[TMP96]], [[TMP97]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP91]], label [[COND_TRUE92:%.*]], label [[COND_FALSE93:%.*]]
+// SIMD-ONLY0:       cond.true92:
+// SIMD-ONLY0-NEXT:    [[TMP98:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END94:%.*]]
+// SIMD-ONLY0:       cond.false93:
+// SIMD-ONLY0-NEXT:    [[TMP99:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END94]]
+// SIMD-ONLY0:       cond.end94:
+// SIMD-ONLY0-NEXT:    [[COND95:%.*]] = phi i32 [ [[TMP98]], [[COND_TRUE92]] ], [ [[TMP99]], [[COND_FALSE93]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND95]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP100:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP100]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP101:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP102:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP96:%.*]] = icmp eq i32 [[TMP101]], [[TMP102]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP96]], label [[COND_TRUE97:%.*]], label [[COND_FALSE98:%.*]]
+// SIMD-ONLY0:       cond.true97:
+// SIMD-ONLY0-NEXT:    [[TMP103:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END99:%.*]]
+// SIMD-ONLY0:       cond.false98:
+// SIMD-ONLY0-NEXT:    [[TMP104:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END99]]
+// SIMD-ONLY0:       cond.end99:
+// SIMD-ONLY0-NEXT:    [[COND100:%.*]] = phi i32 [ [[TMP103]], [[COND_TRUE97]] ], [ [[TMP104]], [[COND_FALSE98]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND100]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP105:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP106:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP101:%.*]] = icmp ugt i32 [[TMP105]], [[TMP106]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP101]], label [[COND_TRUE102:%.*]], label [[COND_FALSE103:%.*]]
+// SIMD-ONLY0:       cond.true102:
+// SIMD-ONLY0-NEXT:    [[TMP107:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END104:%.*]]
+// SIMD-ONLY0:       cond.false103:
+// SIMD-ONLY0-NEXT:    [[TMP108:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END104]]
+// SIMD-ONLY0:       cond.end104:
+// SIMD-ONLY0-NEXT:    [[COND105:%.*]] = phi i32 [ [[TMP107]], [[COND_TRUE102]] ], [ [[TMP108]], [[COND_FALSE103]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND105]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP109:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP109]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP110:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP111:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP106:%.*]] = icmp ult i32 [[TMP110]], [[TMP111]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP106]], label [[COND_TRUE107:%.*]], label [[COND_FALSE108:%.*]]
+// SIMD-ONLY0:       cond.true107:
+// SIMD-ONLY0-NEXT:    [[TMP112:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END109:%.*]]
+// SIMD-ONLY0:       cond.false108:
+// SIMD-ONLY0-NEXT:    [[TMP113:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END109]]
+// SIMD-ONLY0:       cond.end109:
+// SIMD-ONLY0-NEXT:    [[COND110:%.*]] = phi i32 [ [[TMP112]], [[COND_TRUE107]] ], [ [[TMP113]], [[COND_FALSE108]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND110]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP114:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP114]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP115:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP116:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP111:%.*]] = icmp eq i32 [[TMP115]], [[TMP116]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP111]], label [[COND_TRUE112:%.*]], label [[COND_FALSE113:%.*]]
+// SIMD-ONLY0:       cond.true112:
+// SIMD-ONLY0-NEXT:    [[TMP117:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END114:%.*]]
+// SIMD-ONLY0:       cond.false113:
+// SIMD-ONLY0-NEXT:    [[TMP118:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END114]]
+// SIMD-ONLY0:       cond.end114:
+// SIMD-ONLY0-NEXT:    [[COND115:%.*]] = phi i32 [ [[TMP117]], [[COND_TRUE112]] ], [ [[TMP118]], [[COND_FALSE113]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND115]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP119:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP119]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP120:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP120]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP121:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP122:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP116:%.*]] = icmp ugt i32 [[TMP121]], [[TMP122]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP116]], label [[COND_TRUE117:%.*]], label [[COND_FALSE118:%.*]]
+// SIMD-ONLY0:       cond.true117:
+// SIMD-ONLY0-NEXT:    [[TMP123:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END119:%.*]]
+// SIMD-ONLY0:       cond.false118:
+// SIMD-ONLY0-NEXT:    [[TMP124:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END119]]
+// SIMD-ONLY0:       cond.end119:
+// SIMD-ONLY0-NEXT:    [[COND120:%.*]] = phi i32 [ [[TMP123]], [[COND_TRUE117]] ], [ [[TMP124]], [[COND_FALSE118]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND120]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP125:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP125]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP126:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP127:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP121:%.*]] = icmp ult i32 [[TMP126]], [[TMP127]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP121]], label [[COND_TRUE122:%.*]], label [[COND_FALSE123:%.*]]
+// SIMD-ONLY0:       cond.true122:
+// SIMD-ONLY0-NEXT:    [[TMP128:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END124:%.*]]
+// SIMD-ONLY0:       cond.false123:
+// SIMD-ONLY0-NEXT:    [[TMP129:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END124]]
+// SIMD-ONLY0:       cond.end124:
+// SIMD-ONLY0-NEXT:    [[COND125:%.*]] = phi i32 [ [[TMP128]], [[COND_TRUE122]] ], [ [[TMP129]], [[COND_FALSE123]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND125]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP130:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP130]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP131:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP132:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP126:%.*]] = icmp eq i32 [[TMP131]], [[TMP132]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP126]], label [[COND_TRUE127:%.*]], label [[COND_FALSE128:%.*]]
+// SIMD-ONLY0:       cond.true127:
+// SIMD-ONLY0-NEXT:    [[TMP133:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END129:%.*]]
+// SIMD-ONLY0:       cond.false128:
+// SIMD-ONLY0-NEXT:    [[TMP134:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END129]]
+// SIMD-ONLY0:       cond.end129:
+// SIMD-ONLY0-NEXT:    [[COND130:%.*]] = phi i32 [ [[TMP133]], [[COND_TRUE127]] ], [ [[TMP134]], [[COND_FALSE128]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND130]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP135:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP136:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP131:%.*]] = icmp ugt i32 [[TMP135]], [[TMP136]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP131]], label [[COND_TRUE132:%.*]], label [[COND_FALSE133:%.*]]
+// SIMD-ONLY0:       cond.true132:
+// SIMD-ONLY0-NEXT:    [[TMP137:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END134:%.*]]
+// SIMD-ONLY0:       cond.false133:
+// SIMD-ONLY0-NEXT:    [[TMP138:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END134]]
+// SIMD-ONLY0:       cond.end134:
+// SIMD-ONLY0-NEXT:    [[COND135:%.*]] = phi i32 [ [[TMP137]], [[COND_TRUE132]] ], [ [[TMP138]], [[COND_FALSE133]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND135]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP139:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP139]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP140:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP141:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP136:%.*]] = icmp ult i32 [[TMP140]], [[TMP141]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP136]], label [[COND_TRUE137:%.*]], label [[COND_FALSE138:%.*]]
+// SIMD-ONLY0:       cond.true137:
+// SIMD-ONLY0-NEXT:    [[TMP142:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END139:%.*]]
+// SIMD-ONLY0:       cond.false138:
+// SIMD-ONLY0-NEXT:    [[TMP143:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END139]]
+// SIMD-ONLY0:       cond.end139:
+// SIMD-ONLY0-NEXT:    [[COND140:%.*]] = phi i32 [ [[TMP142]], [[COND_TRUE137]] ], [ [[TMP143]], [[COND_FALSE138]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND140]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP144:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP144]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP145:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP146:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP141:%.*]] = icmp eq i32 [[TMP145]], [[TMP146]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP141]], label [[COND_TRUE142:%.*]], label [[COND_FALSE143:%.*]]
+// SIMD-ONLY0:       cond.true142:
+// SIMD-ONLY0-NEXT:    [[TMP147:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END144:%.*]]
+// SIMD-ONLY0:       cond.false143:
+// SIMD-ONLY0-NEXT:    [[TMP148:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END144]]
+// SIMD-ONLY0:       cond.end144:
+// SIMD-ONLY0-NEXT:    [[COND145:%.*]] = phi i32 [ [[TMP147]], [[COND_TRUE142]] ], [ [[TMP148]], [[COND_FALSE143]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND145]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP149:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP149]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP150:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP150]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP151:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP152:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP146:%.*]] = icmp ugt i32 [[TMP151]], [[TMP152]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP146]], label [[COND_TRUE147:%.*]], label [[COND_FALSE148:%.*]]
+// SIMD-ONLY0:       cond.true147:
+// SIMD-ONLY0-NEXT:    [[TMP153:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END149:%.*]]
+// SIMD-ONLY0:       cond.false148:
+// SIMD-ONLY0-NEXT:    [[TMP154:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END149]]
+// SIMD-ONLY0:       cond.end149:
+// SIMD-ONLY0-NEXT:    [[COND150:%.*]] = phi i32 [ [[TMP153]], [[COND_TRUE147]] ], [ [[TMP154]], [[COND_FALSE148]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND150]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP155:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP155]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP156:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP157:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP151:%.*]] = icmp ult i32 [[TMP156]], [[TMP157]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP151]], label [[COND_TRUE152:%.*]], label [[COND_FALSE153:%.*]]
+// SIMD-ONLY0:       cond.true152:
+// SIMD-ONLY0-NEXT:    [[TMP158:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END154:%.*]]
+// SIMD-ONLY0:       cond.false153:
+// SIMD-ONLY0-NEXT:    [[TMP159:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END154]]
+// SIMD-ONLY0:       cond.end154:
+// SIMD-ONLY0-NEXT:    [[COND155:%.*]] = phi i32 [ [[TMP158]], [[COND_TRUE152]] ], [ [[TMP159]], [[COND_FALSE153]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND155]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP160:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP160]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP161:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP162:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP156:%.*]] = icmp eq i32 [[TMP161]], [[TMP162]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP156]], label [[COND_TRUE157:%.*]], label [[COND_FALSE158:%.*]]
+// SIMD-ONLY0:       cond.true157:
+// SIMD-ONLY0-NEXT:    [[TMP163:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END159:%.*]]
+// SIMD-ONLY0:       cond.false158:
+// SIMD-ONLY0-NEXT:    [[TMP164:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END159]]
+// SIMD-ONLY0:       cond.end159:
+// SIMD-ONLY0-NEXT:    [[COND160:%.*]] = phi i32 [ [[TMP163]], [[COND_TRUE157]] ], [ [[TMP164]], [[COND_FALSE158]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND160]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP165:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP166:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP161:%.*]] = icmp ugt i32 [[TMP165]], [[TMP166]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP161]], label [[COND_TRUE162:%.*]], label [[COND_FALSE163:%.*]]
+// SIMD-ONLY0:       cond.true162:
+// SIMD-ONLY0-NEXT:    [[TMP167:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END164:%.*]]
+// SIMD-ONLY0:       cond.false163:
+// SIMD-ONLY0-NEXT:    [[TMP168:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END164]]
+// SIMD-ONLY0:       cond.end164:
+// SIMD-ONLY0-NEXT:    [[COND165:%.*]] = phi i32 [ [[TMP167]], [[COND_TRUE162]] ], [ [[TMP168]], [[COND_FALSE163]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND165]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP169:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP169]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP170:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP171:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP166:%.*]] = icmp ult i32 [[TMP170]], [[TMP171]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP166]], label [[COND_TRUE167:%.*]], label [[COND_FALSE168:%.*]]
+// SIMD-ONLY0:       cond.true167:
+// SIMD-ONLY0-NEXT:    [[TMP172:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END169:%.*]]
+// SIMD-ONLY0:       cond.false168:
+// SIMD-ONLY0-NEXT:    [[TMP173:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END169]]
+// SIMD-ONLY0:       cond.end169:
+// SIMD-ONLY0-NEXT:    [[COND170:%.*]] = phi i32 [ [[TMP172]], [[COND_TRUE167]] ], [ [[TMP173]], [[COND_FALSE168]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND170]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP174:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP174]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP175:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP176:%.*]] = load i32, ptr [[UIE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP171:%.*]] = icmp eq i32 [[TMP175]], [[TMP176]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP171]], label [[COND_TRUE172:%.*]], label [[COND_FALSE173:%.*]]
+// SIMD-ONLY0:       cond.true172:
+// SIMD-ONLY0-NEXT:    [[TMP177:%.*]] = load i32, ptr [[UID]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END174:%.*]]
+// SIMD-ONLY0:       cond.false173:
+// SIMD-ONLY0-NEXT:    [[TMP178:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END174]]
+// SIMD-ONLY0:       cond.end174:
+// SIMD-ONLY0-NEXT:    [[COND175:%.*]] = phi i32 [ [[TMP177]], [[COND_TRUE172]] ], [ [[TMP178]], [[COND_FALSE173]] ]
+// SIMD-ONLY0-NEXT:    store i32 [[COND175]], ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP179:%.*]] = load i32, ptr [[UIX]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP179]], ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP180:%.*]] = load i32, ptr [[UIV]], align 4
+// SIMD-ONLY0-NEXT:    ret i32 [[TMP180]]
+//
+//
+// SIMD-ONLY0-LABEL: @lxevd(
+// SIMD-ONLY0-NEXT:  entry:
+// SIMD-ONLY0-NEXT:    [[LX:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LV:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LE:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LD:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[TMP0:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP0]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP:%.*]] = icmp sgt i64 [[TMP1]], [[TMP2]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]]
+// SIMD-ONLY0:       cond.true:
+// SIMD-ONLY0-NEXT:    [[TMP3:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END:%.*]]
+// SIMD-ONLY0:       cond.false:
+// SIMD-ONLY0-NEXT:    [[TMP4:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END]]
+// SIMD-ONLY0:       cond.end:
+// SIMD-ONLY0-NEXT:    [[COND:%.*]] = phi i64 [ [[TMP3]], [[COND_TRUE]] ], [ [[TMP4]], [[COND_FALSE]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP5]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP6:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP7:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP1:%.*]] = icmp slt i64 [[TMP6]], [[TMP7]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1]], label [[COND_TRUE2:%.*]], label [[COND_FALSE3:%.*]]
+// SIMD-ONLY0:       cond.true2:
+// SIMD-ONLY0-NEXT:    [[TMP8:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4:%.*]]
+// SIMD-ONLY0:       cond.false3:
+// SIMD-ONLY0-NEXT:    [[TMP9:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4]]
+// SIMD-ONLY0:       cond.end4:
+// SIMD-ONLY0-NEXT:    [[COND5:%.*]] = phi i64 [ [[TMP8]], [[COND_TRUE2]] ], [ [[TMP9]], [[COND_FALSE3]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND5]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP10:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP10]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP11:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP12:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6:%.*]] = icmp eq i64 [[TMP11]], [[TMP12]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6]], label [[COND_TRUE7:%.*]], label [[COND_FALSE8:%.*]]
+// SIMD-ONLY0:       cond.true7:
+// SIMD-ONLY0-NEXT:    [[TMP13:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END9:%.*]]
+// SIMD-ONLY0:       cond.false8:
+// SIMD-ONLY0-NEXT:    [[TMP14:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END9]]
+// SIMD-ONLY0:       cond.end9:
+// SIMD-ONLY0-NEXT:    [[COND10:%.*]] = phi i64 [ [[TMP13]], [[COND_TRUE7]] ], [ [[TMP14]], [[COND_FALSE8]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND10]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP15:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP16:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP11:%.*]] = icmp sgt i64 [[TMP15]], [[TMP16]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP11]], label [[COND_TRUE12:%.*]], label [[COND_FALSE13:%.*]]
+// SIMD-ONLY0:       cond.true12:
+// SIMD-ONLY0-NEXT:    [[TMP17:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END14:%.*]]
+// SIMD-ONLY0:       cond.false13:
+// SIMD-ONLY0-NEXT:    [[TMP18:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END14]]
+// SIMD-ONLY0:       cond.end14:
+// SIMD-ONLY0-NEXT:    [[COND15:%.*]] = phi i64 [ [[TMP17]], [[COND_TRUE12]] ], [ [[TMP18]], [[COND_FALSE13]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND15]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP19:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP19]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP20:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP21:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP16:%.*]] = icmp slt i64 [[TMP20]], [[TMP21]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP16]], label [[COND_TRUE17:%.*]], label [[COND_FALSE18:%.*]]
+// SIMD-ONLY0:       cond.true17:
+// SIMD-ONLY0-NEXT:    [[TMP22:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END19:%.*]]
+// SIMD-ONLY0:       cond.false18:
+// SIMD-ONLY0-NEXT:    [[TMP23:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END19]]
+// SIMD-ONLY0:       cond.end19:
+// SIMD-ONLY0-NEXT:    [[COND20:%.*]] = phi i64 [ [[TMP22]], [[COND_TRUE17]] ], [ [[TMP23]], [[COND_FALSE18]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND20]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP24:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP24]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP25:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP26:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP21:%.*]] = icmp eq i64 [[TMP25]], [[TMP26]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP21]], label [[COND_TRUE22:%.*]], label [[COND_FALSE23:%.*]]
+// SIMD-ONLY0:       cond.true22:
+// SIMD-ONLY0-NEXT:    [[TMP27:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END24:%.*]]
+// SIMD-ONLY0:       cond.false23:
+// SIMD-ONLY0-NEXT:    [[TMP28:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END24]]
+// SIMD-ONLY0:       cond.end24:
+// SIMD-ONLY0-NEXT:    [[COND25:%.*]] = phi i64 [ [[TMP27]], [[COND_TRUE22]] ], [ [[TMP28]], [[COND_FALSE23]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND25]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP29:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP29]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP30:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP30]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP31:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP32:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP26:%.*]] = icmp sgt i64 [[TMP31]], [[TMP32]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP26]], label [[COND_TRUE27:%.*]], label [[COND_FALSE28:%.*]]
+// SIMD-ONLY0:       cond.true27:
+// SIMD-ONLY0-NEXT:    [[TMP33:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END29:%.*]]
+// SIMD-ONLY0:       cond.false28:
+// SIMD-ONLY0-NEXT:    [[TMP34:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END29]]
+// SIMD-ONLY0:       cond.end29:
+// SIMD-ONLY0-NEXT:    [[COND30:%.*]] = phi i64 [ [[TMP33]], [[COND_TRUE27]] ], [ [[TMP34]], [[COND_FALSE28]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND30]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP35:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP35]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP36:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP37:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP31:%.*]] = icmp slt i64 [[TMP36]], [[TMP37]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP31]], label [[COND_TRUE32:%.*]], label [[COND_FALSE33:%.*]]
+// SIMD-ONLY0:       cond.true32:
+// SIMD-ONLY0-NEXT:    [[TMP38:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END34:%.*]]
+// SIMD-ONLY0:       cond.false33:
+// SIMD-ONLY0-NEXT:    [[TMP39:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END34]]
+// SIMD-ONLY0:       cond.end34:
+// SIMD-ONLY0-NEXT:    [[COND35:%.*]] = phi i64 [ [[TMP38]], [[COND_TRUE32]] ], [ [[TMP39]], [[COND_FALSE33]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND35]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP40:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP40]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP41:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP42:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP36:%.*]] = icmp eq i64 [[TMP41]], [[TMP42]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP36]], label [[COND_TRUE37:%.*]], label [[COND_FALSE38:%.*]]
+// SIMD-ONLY0:       cond.true37:
+// SIMD-ONLY0-NEXT:    [[TMP43:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END39:%.*]]
+// SIMD-ONLY0:       cond.false38:
+// SIMD-ONLY0-NEXT:    [[TMP44:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END39]]
+// SIMD-ONLY0:       cond.end39:
+// SIMD-ONLY0-NEXT:    [[COND40:%.*]] = phi i64 [ [[TMP43]], [[COND_TRUE37]] ], [ [[TMP44]], [[COND_FALSE38]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND40]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP45:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP46:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP41:%.*]] = icmp sgt i64 [[TMP45]], [[TMP46]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP41]], label [[COND_TRUE42:%.*]], label [[COND_FALSE43:%.*]]
+// SIMD-ONLY0:       cond.true42:
+// SIMD-ONLY0-NEXT:    [[TMP47:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END44:%.*]]
+// SIMD-ONLY0:       cond.false43:
+// SIMD-ONLY0-NEXT:    [[TMP48:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END44]]
+// SIMD-ONLY0:       cond.end44:
+// SIMD-ONLY0-NEXT:    [[COND45:%.*]] = phi i64 [ [[TMP47]], [[COND_TRUE42]] ], [ [[TMP48]], [[COND_FALSE43]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND45]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP49:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP49]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP50:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP51:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP46:%.*]] = icmp slt i64 [[TMP50]], [[TMP51]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP46]], label [[COND_TRUE47:%.*]], label [[COND_FALSE48:%.*]]
+// SIMD-ONLY0:       cond.true47:
+// SIMD-ONLY0-NEXT:    [[TMP52:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END49:%.*]]
+// SIMD-ONLY0:       cond.false48:
+// SIMD-ONLY0-NEXT:    [[TMP53:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END49]]
+// SIMD-ONLY0:       cond.end49:
+// SIMD-ONLY0-NEXT:    [[COND50:%.*]] = phi i64 [ [[TMP52]], [[COND_TRUE47]] ], [ [[TMP53]], [[COND_FALSE48]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND50]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP54:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP54]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP55:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP56:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP51:%.*]] = icmp eq i64 [[TMP55]], [[TMP56]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP51]], label [[COND_TRUE52:%.*]], label [[COND_FALSE53:%.*]]
+// SIMD-ONLY0:       cond.true52:
+// SIMD-ONLY0-NEXT:    [[TMP57:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END54:%.*]]
+// SIMD-ONLY0:       cond.false53:
+// SIMD-ONLY0-NEXT:    [[TMP58:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END54]]
+// SIMD-ONLY0:       cond.end54:
+// SIMD-ONLY0-NEXT:    [[COND55:%.*]] = phi i64 [ [[TMP57]], [[COND_TRUE52]] ], [ [[TMP58]], [[COND_FALSE53]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND55]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP59:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP59]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP60:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP60]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP61:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP62:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP56:%.*]] = icmp sgt i64 [[TMP61]], [[TMP62]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP56]], label [[COND_TRUE57:%.*]], label [[COND_FALSE58:%.*]]
+// SIMD-ONLY0:       cond.true57:
+// SIMD-ONLY0-NEXT:    [[TMP63:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END59:%.*]]
+// SIMD-ONLY0:       cond.false58:
+// SIMD-ONLY0-NEXT:    [[TMP64:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END59]]
+// SIMD-ONLY0:       cond.end59:
+// SIMD-ONLY0-NEXT:    [[COND60:%.*]] = phi i64 [ [[TMP63]], [[COND_TRUE57]] ], [ [[TMP64]], [[COND_FALSE58]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND60]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP65:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP65]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP66:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP67:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP61:%.*]] = icmp slt i64 [[TMP66]], [[TMP67]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP61]], label [[COND_TRUE62:%.*]], label [[COND_FALSE63:%.*]]
+// SIMD-ONLY0:       cond.true62:
+// SIMD-ONLY0-NEXT:    [[TMP68:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END64:%.*]]
+// SIMD-ONLY0:       cond.false63:
+// SIMD-ONLY0-NEXT:    [[TMP69:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END64]]
+// SIMD-ONLY0:       cond.end64:
+// SIMD-ONLY0-NEXT:    [[COND65:%.*]] = phi i64 [ [[TMP68]], [[COND_TRUE62]] ], [ [[TMP69]], [[COND_FALSE63]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND65]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP70:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP70]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP71:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP72:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP66:%.*]] = icmp eq i64 [[TMP71]], [[TMP72]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP66]], label [[COND_TRUE67:%.*]], label [[COND_FALSE68:%.*]]
+// SIMD-ONLY0:       cond.true67:
+// SIMD-ONLY0-NEXT:    [[TMP73:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END69:%.*]]
+// SIMD-ONLY0:       cond.false68:
+// SIMD-ONLY0-NEXT:    [[TMP74:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END69]]
+// SIMD-ONLY0:       cond.end69:
+// SIMD-ONLY0-NEXT:    [[COND70:%.*]] = phi i64 [ [[TMP73]], [[COND_TRUE67]] ], [ [[TMP74]], [[COND_FALSE68]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND70]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP75:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP76:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP71:%.*]] = icmp sgt i64 [[TMP75]], [[TMP76]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP71]], label [[COND_TRUE72:%.*]], label [[COND_FALSE73:%.*]]
+// SIMD-ONLY0:       cond.true72:
+// SIMD-ONLY0-NEXT:    [[TMP77:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END74:%.*]]
+// SIMD-ONLY0:       cond.false73:
+// SIMD-ONLY0-NEXT:    [[TMP78:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END74]]
+// SIMD-ONLY0:       cond.end74:
+// SIMD-ONLY0-NEXT:    [[COND75:%.*]] = phi i64 [ [[TMP77]], [[COND_TRUE72]] ], [ [[TMP78]], [[COND_FALSE73]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND75]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP79:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP79]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP80:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP81:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP76:%.*]] = icmp slt i64 [[TMP80]], [[TMP81]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP76]], label [[COND_TRUE77:%.*]], label [[COND_FALSE78:%.*]]
+// SIMD-ONLY0:       cond.true77:
+// SIMD-ONLY0-NEXT:    [[TMP82:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END79:%.*]]
+// SIMD-ONLY0:       cond.false78:
+// SIMD-ONLY0-NEXT:    [[TMP83:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END79]]
+// SIMD-ONLY0:       cond.end79:
+// SIMD-ONLY0-NEXT:    [[COND80:%.*]] = phi i64 [ [[TMP82]], [[COND_TRUE77]] ], [ [[TMP83]], [[COND_FALSE78]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND80]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP84:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP84]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP85:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP86:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP81:%.*]] = icmp eq i64 [[TMP85]], [[TMP86]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP81]], label [[COND_TRUE82:%.*]], label [[COND_FALSE83:%.*]]
+// SIMD-ONLY0:       cond.true82:
+// SIMD-ONLY0-NEXT:    [[TMP87:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END84:%.*]]
+// SIMD-ONLY0:       cond.false83:
+// SIMD-ONLY0-NEXT:    [[TMP88:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END84]]
+// SIMD-ONLY0:       cond.end84:
+// SIMD-ONLY0-NEXT:    [[COND85:%.*]] = phi i64 [ [[TMP87]], [[COND_TRUE82]] ], [ [[TMP88]], [[COND_FALSE83]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND85]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP89:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP89]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP90:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP90]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP91:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP92:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP86:%.*]] = icmp sgt i64 [[TMP91]], [[TMP92]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP86]], label [[COND_TRUE87:%.*]], label [[COND_FALSE88:%.*]]
+// SIMD-ONLY0:       cond.true87:
+// SIMD-ONLY0-NEXT:    [[TMP93:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END89:%.*]]
+// SIMD-ONLY0:       cond.false88:
+// SIMD-ONLY0-NEXT:    [[TMP94:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END89]]
+// SIMD-ONLY0:       cond.end89:
+// SIMD-ONLY0-NEXT:    [[COND90:%.*]] = phi i64 [ [[TMP93]], [[COND_TRUE87]] ], [ [[TMP94]], [[COND_FALSE88]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND90]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP95:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP95]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP96:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP97:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP91:%.*]] = icmp slt i64 [[TMP96]], [[TMP97]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP91]], label [[COND_TRUE92:%.*]], label [[COND_FALSE93:%.*]]
+// SIMD-ONLY0:       cond.true92:
+// SIMD-ONLY0-NEXT:    [[TMP98:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END94:%.*]]
+// SIMD-ONLY0:       cond.false93:
+// SIMD-ONLY0-NEXT:    [[TMP99:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END94]]
+// SIMD-ONLY0:       cond.end94:
+// SIMD-ONLY0-NEXT:    [[COND95:%.*]] = phi i64 [ [[TMP98]], [[COND_TRUE92]] ], [ [[TMP99]], [[COND_FALSE93]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND95]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP100:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP100]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP101:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP102:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP96:%.*]] = icmp eq i64 [[TMP101]], [[TMP102]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP96]], label [[COND_TRUE97:%.*]], label [[COND_FALSE98:%.*]]
+// SIMD-ONLY0:       cond.true97:
+// SIMD-ONLY0-NEXT:    [[TMP103:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END99:%.*]]
+// SIMD-ONLY0:       cond.false98:
+// SIMD-ONLY0-NEXT:    [[TMP104:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END99]]
+// SIMD-ONLY0:       cond.end99:
+// SIMD-ONLY0-NEXT:    [[COND100:%.*]] = phi i64 [ [[TMP103]], [[COND_TRUE97]] ], [ [[TMP104]], [[COND_FALSE98]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND100]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP105:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP106:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP101:%.*]] = icmp sgt i64 [[TMP105]], [[TMP106]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP101]], label [[COND_TRUE102:%.*]], label [[COND_FALSE103:%.*]]
+// SIMD-ONLY0:       cond.true102:
+// SIMD-ONLY0-NEXT:    [[TMP107:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END104:%.*]]
+// SIMD-ONLY0:       cond.false103:
+// SIMD-ONLY0-NEXT:    [[TMP108:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END104]]
+// SIMD-ONLY0:       cond.end104:
+// SIMD-ONLY0-NEXT:    [[COND105:%.*]] = phi i64 [ [[TMP107]], [[COND_TRUE102]] ], [ [[TMP108]], [[COND_FALSE103]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND105]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP109:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP109]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP110:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP111:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP106:%.*]] = icmp slt i64 [[TMP110]], [[TMP111]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP106]], label [[COND_TRUE107:%.*]], label [[COND_FALSE108:%.*]]
+// SIMD-ONLY0:       cond.true107:
+// SIMD-ONLY0-NEXT:    [[TMP112:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END109:%.*]]
+// SIMD-ONLY0:       cond.false108:
+// SIMD-ONLY0-NEXT:    [[TMP113:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END109]]
+// SIMD-ONLY0:       cond.end109:
+// SIMD-ONLY0-NEXT:    [[COND110:%.*]] = phi i64 [ [[TMP112]], [[COND_TRUE107]] ], [ [[TMP113]], [[COND_FALSE108]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND110]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP114:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP114]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP115:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP116:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP111:%.*]] = icmp eq i64 [[TMP115]], [[TMP116]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP111]], label [[COND_TRUE112:%.*]], label [[COND_FALSE113:%.*]]
+// SIMD-ONLY0:       cond.true112:
+// SIMD-ONLY0-NEXT:    [[TMP117:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END114:%.*]]
+// SIMD-ONLY0:       cond.false113:
+// SIMD-ONLY0-NEXT:    [[TMP118:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END114]]
+// SIMD-ONLY0:       cond.end114:
+// SIMD-ONLY0-NEXT:    [[COND115:%.*]] = phi i64 [ [[TMP117]], [[COND_TRUE112]] ], [ [[TMP118]], [[COND_FALSE113]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND115]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP119:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP119]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP120:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP120]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP121:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP122:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP116:%.*]] = icmp sgt i64 [[TMP121]], [[TMP122]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP116]], label [[COND_TRUE117:%.*]], label [[COND_FALSE118:%.*]]
+// SIMD-ONLY0:       cond.true117:
+// SIMD-ONLY0-NEXT:    [[TMP123:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END119:%.*]]
+// SIMD-ONLY0:       cond.false118:
+// SIMD-ONLY0-NEXT:    [[TMP124:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END119]]
+// SIMD-ONLY0:       cond.end119:
+// SIMD-ONLY0-NEXT:    [[COND120:%.*]] = phi i64 [ [[TMP123]], [[COND_TRUE117]] ], [ [[TMP124]], [[COND_FALSE118]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND120]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP125:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP125]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP126:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP127:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP121:%.*]] = icmp slt i64 [[TMP126]], [[TMP127]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP121]], label [[COND_TRUE122:%.*]], label [[COND_FALSE123:%.*]]
+// SIMD-ONLY0:       cond.true122:
+// SIMD-ONLY0-NEXT:    [[TMP128:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END124:%.*]]
+// SIMD-ONLY0:       cond.false123:
+// SIMD-ONLY0-NEXT:    [[TMP129:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END124]]
+// SIMD-ONLY0:       cond.end124:
+// SIMD-ONLY0-NEXT:    [[COND125:%.*]] = phi i64 [ [[TMP128]], [[COND_TRUE122]] ], [ [[TMP129]], [[COND_FALSE123]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND125]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP130:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP130]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP131:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP132:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP126:%.*]] = icmp eq i64 [[TMP131]], [[TMP132]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP126]], label [[COND_TRUE127:%.*]], label [[COND_FALSE128:%.*]]
+// SIMD-ONLY0:       cond.true127:
+// SIMD-ONLY0-NEXT:    [[TMP133:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END129:%.*]]
+// SIMD-ONLY0:       cond.false128:
+// SIMD-ONLY0-NEXT:    [[TMP134:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END129]]
+// SIMD-ONLY0:       cond.end129:
+// SIMD-ONLY0-NEXT:    [[COND130:%.*]] = phi i64 [ [[TMP133]], [[COND_TRUE127]] ], [ [[TMP134]], [[COND_FALSE128]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND130]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP135:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP136:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP131:%.*]] = icmp sgt i64 [[TMP135]], [[TMP136]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP131]], label [[COND_TRUE132:%.*]], label [[COND_FALSE133:%.*]]
+// SIMD-ONLY0:       cond.true132:
+// SIMD-ONLY0-NEXT:    [[TMP137:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END134:%.*]]
+// SIMD-ONLY0:       cond.false133:
+// SIMD-ONLY0-NEXT:    [[TMP138:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END134]]
+// SIMD-ONLY0:       cond.end134:
+// SIMD-ONLY0-NEXT:    [[COND135:%.*]] = phi i64 [ [[TMP137]], [[COND_TRUE132]] ], [ [[TMP138]], [[COND_FALSE133]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND135]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP139:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP139]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP140:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP141:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP136:%.*]] = icmp slt i64 [[TMP140]], [[TMP141]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP136]], label [[COND_TRUE137:%.*]], label [[COND_FALSE138:%.*]]
+// SIMD-ONLY0:       cond.true137:
+// SIMD-ONLY0-NEXT:    [[TMP142:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END139:%.*]]
+// SIMD-ONLY0:       cond.false138:
+// SIMD-ONLY0-NEXT:    [[TMP143:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END139]]
+// SIMD-ONLY0:       cond.end139:
+// SIMD-ONLY0-NEXT:    [[COND140:%.*]] = phi i64 [ [[TMP142]], [[COND_TRUE137]] ], [ [[TMP143]], [[COND_FALSE138]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND140]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP144:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP144]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP145:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP146:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP141:%.*]] = icmp eq i64 [[TMP145]], [[TMP146]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP141]], label [[COND_TRUE142:%.*]], label [[COND_FALSE143:%.*]]
+// SIMD-ONLY0:       cond.true142:
+// SIMD-ONLY0-NEXT:    [[TMP147:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END144:%.*]]
+// SIMD-ONLY0:       cond.false143:
+// SIMD-ONLY0-NEXT:    [[TMP148:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END144]]
+// SIMD-ONLY0:       cond.end144:
+// SIMD-ONLY0-NEXT:    [[COND145:%.*]] = phi i64 [ [[TMP147]], [[COND_TRUE142]] ], [ [[TMP148]], [[COND_FALSE143]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND145]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP149:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP149]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP150:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP150]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP151:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP152:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP146:%.*]] = icmp sgt i64 [[TMP151]], [[TMP152]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP146]], label [[COND_TRUE147:%.*]], label [[COND_FALSE148:%.*]]
+// SIMD-ONLY0:       cond.true147:
+// SIMD-ONLY0-NEXT:    [[TMP153:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END149:%.*]]
+// SIMD-ONLY0:       cond.false148:
+// SIMD-ONLY0-NEXT:    [[TMP154:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END149]]
+// SIMD-ONLY0:       cond.end149:
+// SIMD-ONLY0-NEXT:    [[COND150:%.*]] = phi i64 [ [[TMP153]], [[COND_TRUE147]] ], [ [[TMP154]], [[COND_FALSE148]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND150]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP155:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP155]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP156:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP157:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP151:%.*]] = icmp slt i64 [[TMP156]], [[TMP157]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP151]], label [[COND_TRUE152:%.*]], label [[COND_FALSE153:%.*]]
+// SIMD-ONLY0:       cond.true152:
+// SIMD-ONLY0-NEXT:    [[TMP158:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END154:%.*]]
+// SIMD-ONLY0:       cond.false153:
+// SIMD-ONLY0-NEXT:    [[TMP159:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END154]]
+// SIMD-ONLY0:       cond.end154:
+// SIMD-ONLY0-NEXT:    [[COND155:%.*]] = phi i64 [ [[TMP158]], [[COND_TRUE152]] ], [ [[TMP159]], [[COND_FALSE153]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND155]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP160:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP160]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP161:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP162:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP156:%.*]] = icmp eq i64 [[TMP161]], [[TMP162]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP156]], label [[COND_TRUE157:%.*]], label [[COND_FALSE158:%.*]]
+// SIMD-ONLY0:       cond.true157:
+// SIMD-ONLY0-NEXT:    [[TMP163:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END159:%.*]]
+// SIMD-ONLY0:       cond.false158:
+// SIMD-ONLY0-NEXT:    [[TMP164:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END159]]
+// SIMD-ONLY0:       cond.end159:
+// SIMD-ONLY0-NEXT:    [[COND160:%.*]] = phi i64 [ [[TMP163]], [[COND_TRUE157]] ], [ [[TMP164]], [[COND_FALSE158]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND160]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP165:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP166:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP161:%.*]] = icmp sgt i64 [[TMP165]], [[TMP166]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP161]], label [[COND_TRUE162:%.*]], label [[COND_FALSE163:%.*]]
+// SIMD-ONLY0:       cond.true162:
+// SIMD-ONLY0-NEXT:    [[TMP167:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END164:%.*]]
+// SIMD-ONLY0:       cond.false163:
+// SIMD-ONLY0-NEXT:    [[TMP168:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END164]]
+// SIMD-ONLY0:       cond.end164:
+// SIMD-ONLY0-NEXT:    [[COND165:%.*]] = phi i64 [ [[TMP167]], [[COND_TRUE162]] ], [ [[TMP168]], [[COND_FALSE163]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND165]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP169:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP169]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP170:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP171:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP166:%.*]] = icmp slt i64 [[TMP170]], [[TMP171]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP166]], label [[COND_TRUE167:%.*]], label [[COND_FALSE168:%.*]]
+// SIMD-ONLY0:       cond.true167:
+// SIMD-ONLY0-NEXT:    [[TMP172:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END169:%.*]]
+// SIMD-ONLY0:       cond.false168:
+// SIMD-ONLY0-NEXT:    [[TMP173:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END169]]
+// SIMD-ONLY0:       cond.end169:
+// SIMD-ONLY0-NEXT:    [[COND170:%.*]] = phi i64 [ [[TMP172]], [[COND_TRUE167]] ], [ [[TMP173]], [[COND_FALSE168]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND170]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP174:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP174]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP175:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP176:%.*]] = load i64, ptr [[LE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP171:%.*]] = icmp eq i64 [[TMP175]], [[TMP176]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP171]], label [[COND_TRUE172:%.*]], label [[COND_FALSE173:%.*]]
+// SIMD-ONLY0:       cond.true172:
+// SIMD-ONLY0-NEXT:    [[TMP177:%.*]] = load i64, ptr [[LD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END174:%.*]]
+// SIMD-ONLY0:       cond.false173:
+// SIMD-ONLY0-NEXT:    [[TMP178:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END174]]
+// SIMD-ONLY0:       cond.end174:
+// SIMD-ONLY0-NEXT:    [[COND175:%.*]] = phi i64 [ [[TMP177]], [[COND_TRUE172]] ], [ [[TMP178]], [[COND_FALSE173]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND175]], ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP179:%.*]] = load i64, ptr [[LX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP179]], ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP180:%.*]] = load i64, ptr [[LV]], align 8
+// SIMD-ONLY0-NEXT:    ret i64 [[TMP180]]
+//
+//
+// SIMD-ONLY0-LABEL: @ulxevd(
+// SIMD-ONLY0-NEXT:  entry:
+// SIMD-ONLY0-NEXT:    [[ULX:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULV:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULE:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULD:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[TMP0:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP0]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP:%.*]] = icmp ugt i64 [[TMP1]], [[TMP2]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]]
+// SIMD-ONLY0:       cond.true:
+// SIMD-ONLY0-NEXT:    [[TMP3:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END:%.*]]
+// SIMD-ONLY0:       cond.false:
+// SIMD-ONLY0-NEXT:    [[TMP4:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END]]
+// SIMD-ONLY0:       cond.end:
+// SIMD-ONLY0-NEXT:    [[COND:%.*]] = phi i64 [ [[TMP3]], [[COND_TRUE]] ], [ [[TMP4]], [[COND_FALSE]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP5]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP6:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP7:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP1:%.*]] = icmp ult i64 [[TMP6]], [[TMP7]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1]], label [[COND_TRUE2:%.*]], label [[COND_FALSE3:%.*]]
+// SIMD-ONLY0:       cond.true2:
+// SIMD-ONLY0-NEXT:    [[TMP8:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4:%.*]]
+// SIMD-ONLY0:       cond.false3:
+// SIMD-ONLY0-NEXT:    [[TMP9:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4]]
+// SIMD-ONLY0:       cond.end4:
+// SIMD-ONLY0-NEXT:    [[COND5:%.*]] = phi i64 [ [[TMP8]], [[COND_TRUE2]] ], [ [[TMP9]], [[COND_FALSE3]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND5]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP10:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP10]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP11:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP12:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6:%.*]] = icmp eq i64 [[TMP11]], [[TMP12]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6]], label [[COND_TRUE7:%.*]], label [[COND_FALSE8:%.*]]
+// SIMD-ONLY0:       cond.true7:
+// SIMD-ONLY0-NEXT:    [[TMP13:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END9:%.*]]
+// SIMD-ONLY0:       cond.false8:
+// SIMD-ONLY0-NEXT:    [[TMP14:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END9]]
+// SIMD-ONLY0:       cond.end9:
+// SIMD-ONLY0-NEXT:    [[COND10:%.*]] = phi i64 [ [[TMP13]], [[COND_TRUE7]] ], [ [[TMP14]], [[COND_FALSE8]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND10]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP15:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP16:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP11:%.*]] = icmp ugt i64 [[TMP15]], [[TMP16]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP11]], label [[COND_TRUE12:%.*]], label [[COND_FALSE13:%.*]]
+// SIMD-ONLY0:       cond.true12:
+// SIMD-ONLY0-NEXT:    [[TMP17:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END14:%.*]]
+// SIMD-ONLY0:       cond.false13:
+// SIMD-ONLY0-NEXT:    [[TMP18:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END14]]
+// SIMD-ONLY0:       cond.end14:
+// SIMD-ONLY0-NEXT:    [[COND15:%.*]] = phi i64 [ [[TMP17]], [[COND_TRUE12]] ], [ [[TMP18]], [[COND_FALSE13]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND15]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP19:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP19]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP20:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP21:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP16:%.*]] = icmp ult i64 [[TMP20]], [[TMP21]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP16]], label [[COND_TRUE17:%.*]], label [[COND_FALSE18:%.*]]
+// SIMD-ONLY0:       cond.true17:
+// SIMD-ONLY0-NEXT:    [[TMP22:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END19:%.*]]
+// SIMD-ONLY0:       cond.false18:
+// SIMD-ONLY0-NEXT:    [[TMP23:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END19]]
+// SIMD-ONLY0:       cond.end19:
+// SIMD-ONLY0-NEXT:    [[COND20:%.*]] = phi i64 [ [[TMP22]], [[COND_TRUE17]] ], [ [[TMP23]], [[COND_FALSE18]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND20]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP24:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP24]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP25:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP26:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP21:%.*]] = icmp eq i64 [[TMP25]], [[TMP26]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP21]], label [[COND_TRUE22:%.*]], label [[COND_FALSE23:%.*]]
+// SIMD-ONLY0:       cond.true22:
+// SIMD-ONLY0-NEXT:    [[TMP27:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END24:%.*]]
+// SIMD-ONLY0:       cond.false23:
+// SIMD-ONLY0-NEXT:    [[TMP28:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END24]]
+// SIMD-ONLY0:       cond.end24:
+// SIMD-ONLY0-NEXT:    [[COND25:%.*]] = phi i64 [ [[TMP27]], [[COND_TRUE22]] ], [ [[TMP28]], [[COND_FALSE23]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND25]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP29:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP29]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP30:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP30]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP31:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP32:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP26:%.*]] = icmp ugt i64 [[TMP31]], [[TMP32]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP26]], label [[COND_TRUE27:%.*]], label [[COND_FALSE28:%.*]]
+// SIMD-ONLY0:       cond.true27:
+// SIMD-ONLY0-NEXT:    [[TMP33:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END29:%.*]]
+// SIMD-ONLY0:       cond.false28:
+// SIMD-ONLY0-NEXT:    [[TMP34:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END29]]
+// SIMD-ONLY0:       cond.end29:
+// SIMD-ONLY0-NEXT:    [[COND30:%.*]] = phi i64 [ [[TMP33]], [[COND_TRUE27]] ], [ [[TMP34]], [[COND_FALSE28]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND30]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP35:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP35]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP36:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP37:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP31:%.*]] = icmp ult i64 [[TMP36]], [[TMP37]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP31]], label [[COND_TRUE32:%.*]], label [[COND_FALSE33:%.*]]
+// SIMD-ONLY0:       cond.true32:
+// SIMD-ONLY0-NEXT:    [[TMP38:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END34:%.*]]
+// SIMD-ONLY0:       cond.false33:
+// SIMD-ONLY0-NEXT:    [[TMP39:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END34]]
+// SIMD-ONLY0:       cond.end34:
+// SIMD-ONLY0-NEXT:    [[COND35:%.*]] = phi i64 [ [[TMP38]], [[COND_TRUE32]] ], [ [[TMP39]], [[COND_FALSE33]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND35]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP40:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP40]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP41:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP42:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP36:%.*]] = icmp eq i64 [[TMP41]], [[TMP42]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP36]], label [[COND_TRUE37:%.*]], label [[COND_FALSE38:%.*]]
+// SIMD-ONLY0:       cond.true37:
+// SIMD-ONLY0-NEXT:    [[TMP43:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END39:%.*]]
+// SIMD-ONLY0:       cond.false38:
+// SIMD-ONLY0-NEXT:    [[TMP44:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END39]]
+// SIMD-ONLY0:       cond.end39:
+// SIMD-ONLY0-NEXT:    [[COND40:%.*]] = phi i64 [ [[TMP43]], [[COND_TRUE37]] ], [ [[TMP44]], [[COND_FALSE38]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND40]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP45:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP46:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP41:%.*]] = icmp ugt i64 [[TMP45]], [[TMP46]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP41]], label [[COND_TRUE42:%.*]], label [[COND_FALSE43:%.*]]
+// SIMD-ONLY0:       cond.true42:
+// SIMD-ONLY0-NEXT:    [[TMP47:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END44:%.*]]
+// SIMD-ONLY0:       cond.false43:
+// SIMD-ONLY0-NEXT:    [[TMP48:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END44]]
+// SIMD-ONLY0:       cond.end44:
+// SIMD-ONLY0-NEXT:    [[COND45:%.*]] = phi i64 [ [[TMP47]], [[COND_TRUE42]] ], [ [[TMP48]], [[COND_FALSE43]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND45]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP49:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP49]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP50:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP51:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP46:%.*]] = icmp ult i64 [[TMP50]], [[TMP51]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP46]], label [[COND_TRUE47:%.*]], label [[COND_FALSE48:%.*]]
+// SIMD-ONLY0:       cond.true47:
+// SIMD-ONLY0-NEXT:    [[TMP52:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END49:%.*]]
+// SIMD-ONLY0:       cond.false48:
+// SIMD-ONLY0-NEXT:    [[TMP53:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END49]]
+// SIMD-ONLY0:       cond.end49:
+// SIMD-ONLY0-NEXT:    [[COND50:%.*]] = phi i64 [ [[TMP52]], [[COND_TRUE47]] ], [ [[TMP53]], [[COND_FALSE48]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND50]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP54:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP54]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP55:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP56:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP51:%.*]] = icmp eq i64 [[TMP55]], [[TMP56]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP51]], label [[COND_TRUE52:%.*]], label [[COND_FALSE53:%.*]]
+// SIMD-ONLY0:       cond.true52:
+// SIMD-ONLY0-NEXT:    [[TMP57:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END54:%.*]]
+// SIMD-ONLY0:       cond.false53:
+// SIMD-ONLY0-NEXT:    [[TMP58:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END54]]
+// SIMD-ONLY0:       cond.end54:
+// SIMD-ONLY0-NEXT:    [[COND55:%.*]] = phi i64 [ [[TMP57]], [[COND_TRUE52]] ], [ [[TMP58]], [[COND_FALSE53]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND55]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP59:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP59]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP60:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP60]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP61:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP62:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP56:%.*]] = icmp ugt i64 [[TMP61]], [[TMP62]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP56]], label [[COND_TRUE57:%.*]], label [[COND_FALSE58:%.*]]
+// SIMD-ONLY0:       cond.true57:
+// SIMD-ONLY0-NEXT:    [[TMP63:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END59:%.*]]
+// SIMD-ONLY0:       cond.false58:
+// SIMD-ONLY0-NEXT:    [[TMP64:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END59]]
+// SIMD-ONLY0:       cond.end59:
+// SIMD-ONLY0-NEXT:    [[COND60:%.*]] = phi i64 [ [[TMP63]], [[COND_TRUE57]] ], [ [[TMP64]], [[COND_FALSE58]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND60]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP65:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP65]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP66:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP67:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP61:%.*]] = icmp ult i64 [[TMP66]], [[TMP67]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP61]], label [[COND_TRUE62:%.*]], label [[COND_FALSE63:%.*]]
+// SIMD-ONLY0:       cond.true62:
+// SIMD-ONLY0-NEXT:    [[TMP68:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END64:%.*]]
+// SIMD-ONLY0:       cond.false63:
+// SIMD-ONLY0-NEXT:    [[TMP69:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END64]]
+// SIMD-ONLY0:       cond.end64:
+// SIMD-ONLY0-NEXT:    [[COND65:%.*]] = phi i64 [ [[TMP68]], [[COND_TRUE62]] ], [ [[TMP69]], [[COND_FALSE63]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND65]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP70:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP70]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP71:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP72:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP66:%.*]] = icmp eq i64 [[TMP71]], [[TMP72]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP66]], label [[COND_TRUE67:%.*]], label [[COND_FALSE68:%.*]]
+// SIMD-ONLY0:       cond.true67:
+// SIMD-ONLY0-NEXT:    [[TMP73:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END69:%.*]]
+// SIMD-ONLY0:       cond.false68:
+// SIMD-ONLY0-NEXT:    [[TMP74:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END69]]
+// SIMD-ONLY0:       cond.end69:
+// SIMD-ONLY0-NEXT:    [[COND70:%.*]] = phi i64 [ [[TMP73]], [[COND_TRUE67]] ], [ [[TMP74]], [[COND_FALSE68]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND70]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP75:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP76:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP71:%.*]] = icmp ugt i64 [[TMP75]], [[TMP76]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP71]], label [[COND_TRUE72:%.*]], label [[COND_FALSE73:%.*]]
+// SIMD-ONLY0:       cond.true72:
+// SIMD-ONLY0-NEXT:    [[TMP77:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END74:%.*]]
+// SIMD-ONLY0:       cond.false73:
+// SIMD-ONLY0-NEXT:    [[TMP78:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END74]]
+// SIMD-ONLY0:       cond.end74:
+// SIMD-ONLY0-NEXT:    [[COND75:%.*]] = phi i64 [ [[TMP77]], [[COND_TRUE72]] ], [ [[TMP78]], [[COND_FALSE73]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND75]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP79:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP79]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP80:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP81:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP76:%.*]] = icmp ult i64 [[TMP80]], [[TMP81]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP76]], label [[COND_TRUE77:%.*]], label [[COND_FALSE78:%.*]]
+// SIMD-ONLY0:       cond.true77:
+// SIMD-ONLY0-NEXT:    [[TMP82:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END79:%.*]]
+// SIMD-ONLY0:       cond.false78:
+// SIMD-ONLY0-NEXT:    [[TMP83:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END79]]
+// SIMD-ONLY0:       cond.end79:
+// SIMD-ONLY0-NEXT:    [[COND80:%.*]] = phi i64 [ [[TMP82]], [[COND_TRUE77]] ], [ [[TMP83]], [[COND_FALSE78]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND80]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP84:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP84]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP85:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP86:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP81:%.*]] = icmp eq i64 [[TMP85]], [[TMP86]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP81]], label [[COND_TRUE82:%.*]], label [[COND_FALSE83:%.*]]
+// SIMD-ONLY0:       cond.true82:
+// SIMD-ONLY0-NEXT:    [[TMP87:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END84:%.*]]
+// SIMD-ONLY0:       cond.false83:
+// SIMD-ONLY0-NEXT:    [[TMP88:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END84]]
+// SIMD-ONLY0:       cond.end84:
+// SIMD-ONLY0-NEXT:    [[COND85:%.*]] = phi i64 [ [[TMP87]], [[COND_TRUE82]] ], [ [[TMP88]], [[COND_FALSE83]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND85]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP89:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP89]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP90:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP90]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP91:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP92:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP86:%.*]] = icmp ugt i64 [[TMP91]], [[TMP92]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP86]], label [[COND_TRUE87:%.*]], label [[COND_FALSE88:%.*]]
+// SIMD-ONLY0:       cond.true87:
+// SIMD-ONLY0-NEXT:    [[TMP93:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END89:%.*]]
+// SIMD-ONLY0:       cond.false88:
+// SIMD-ONLY0-NEXT:    [[TMP94:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END89]]
+// SIMD-ONLY0:       cond.end89:
+// SIMD-ONLY0-NEXT:    [[COND90:%.*]] = phi i64 [ [[TMP93]], [[COND_TRUE87]] ], [ [[TMP94]], [[COND_FALSE88]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND90]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP95:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP95]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP96:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP97:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP91:%.*]] = icmp ult i64 [[TMP96]], [[TMP97]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP91]], label [[COND_TRUE92:%.*]], label [[COND_FALSE93:%.*]]
+// SIMD-ONLY0:       cond.true92:
+// SIMD-ONLY0-NEXT:    [[TMP98:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END94:%.*]]
+// SIMD-ONLY0:       cond.false93:
+// SIMD-ONLY0-NEXT:    [[TMP99:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END94]]
+// SIMD-ONLY0:       cond.end94:
+// SIMD-ONLY0-NEXT:    [[COND95:%.*]] = phi i64 [ [[TMP98]], [[COND_TRUE92]] ], [ [[TMP99]], [[COND_FALSE93]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND95]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP100:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP100]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP101:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP102:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP96:%.*]] = icmp eq i64 [[TMP101]], [[TMP102]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP96]], label [[COND_TRUE97:%.*]], label [[COND_FALSE98:%.*]]
+// SIMD-ONLY0:       cond.true97:
+// SIMD-ONLY0-NEXT:    [[TMP103:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END99:%.*]]
+// SIMD-ONLY0:       cond.false98:
+// SIMD-ONLY0-NEXT:    [[TMP104:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END99]]
+// SIMD-ONLY0:       cond.end99:
+// SIMD-ONLY0-NEXT:    [[COND100:%.*]] = phi i64 [ [[TMP103]], [[COND_TRUE97]] ], [ [[TMP104]], [[COND_FALSE98]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND100]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP105:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP106:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP101:%.*]] = icmp ugt i64 [[TMP105]], [[TMP106]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP101]], label [[COND_TRUE102:%.*]], label [[COND_FALSE103:%.*]]
+// SIMD-ONLY0:       cond.true102:
+// SIMD-ONLY0-NEXT:    [[TMP107:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END104:%.*]]
+// SIMD-ONLY0:       cond.false103:
+// SIMD-ONLY0-NEXT:    [[TMP108:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END104]]
+// SIMD-ONLY0:       cond.end104:
+// SIMD-ONLY0-NEXT:    [[COND105:%.*]] = phi i64 [ [[TMP107]], [[COND_TRUE102]] ], [ [[TMP108]], [[COND_FALSE103]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND105]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP109:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP109]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP110:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP111:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP106:%.*]] = icmp ult i64 [[TMP110]], [[TMP111]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP106]], label [[COND_TRUE107:%.*]], label [[COND_FALSE108:%.*]]
+// SIMD-ONLY0:       cond.true107:
+// SIMD-ONLY0-NEXT:    [[TMP112:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END109:%.*]]
+// SIMD-ONLY0:       cond.false108:
+// SIMD-ONLY0-NEXT:    [[TMP113:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END109]]
+// SIMD-ONLY0:       cond.end109:
+// SIMD-ONLY0-NEXT:    [[COND110:%.*]] = phi i64 [ [[TMP112]], [[COND_TRUE107]] ], [ [[TMP113]], [[COND_FALSE108]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND110]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP114:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP114]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP115:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP116:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP111:%.*]] = icmp eq i64 [[TMP115]], [[TMP116]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP111]], label [[COND_TRUE112:%.*]], label [[COND_FALSE113:%.*]]
+// SIMD-ONLY0:       cond.true112:
+// SIMD-ONLY0-NEXT:    [[TMP117:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END114:%.*]]
+// SIMD-ONLY0:       cond.false113:
+// SIMD-ONLY0-NEXT:    [[TMP118:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END114]]
+// SIMD-ONLY0:       cond.end114:
+// SIMD-ONLY0-NEXT:    [[COND115:%.*]] = phi i64 [ [[TMP117]], [[COND_TRUE112]] ], [ [[TMP118]], [[COND_FALSE113]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND115]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP119:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP119]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP120:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP120]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP121:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP122:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP116:%.*]] = icmp ugt i64 [[TMP121]], [[TMP122]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP116]], label [[COND_TRUE117:%.*]], label [[COND_FALSE118:%.*]]
+// SIMD-ONLY0:       cond.true117:
+// SIMD-ONLY0-NEXT:    [[TMP123:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END119:%.*]]
+// SIMD-ONLY0:       cond.false118:
+// SIMD-ONLY0-NEXT:    [[TMP124:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END119]]
+// SIMD-ONLY0:       cond.end119:
+// SIMD-ONLY0-NEXT:    [[COND120:%.*]] = phi i64 [ [[TMP123]], [[COND_TRUE117]] ], [ [[TMP124]], [[COND_FALSE118]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND120]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP125:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP125]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP126:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP127:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP121:%.*]] = icmp ult i64 [[TMP126]], [[TMP127]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP121]], label [[COND_TRUE122:%.*]], label [[COND_FALSE123:%.*]]
+// SIMD-ONLY0:       cond.true122:
+// SIMD-ONLY0-NEXT:    [[TMP128:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END124:%.*]]
+// SIMD-ONLY0:       cond.false123:
+// SIMD-ONLY0-NEXT:    [[TMP129:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END124]]
+// SIMD-ONLY0:       cond.end124:
+// SIMD-ONLY0-NEXT:    [[COND125:%.*]] = phi i64 [ [[TMP128]], [[COND_TRUE122]] ], [ [[TMP129]], [[COND_FALSE123]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND125]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP130:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP130]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP131:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP132:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP126:%.*]] = icmp eq i64 [[TMP131]], [[TMP132]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP126]], label [[COND_TRUE127:%.*]], label [[COND_FALSE128:%.*]]
+// SIMD-ONLY0:       cond.true127:
+// SIMD-ONLY0-NEXT:    [[TMP133:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END129:%.*]]
+// SIMD-ONLY0:       cond.false128:
+// SIMD-ONLY0-NEXT:    [[TMP134:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END129]]
+// SIMD-ONLY0:       cond.end129:
+// SIMD-ONLY0-NEXT:    [[COND130:%.*]] = phi i64 [ [[TMP133]], [[COND_TRUE127]] ], [ [[TMP134]], [[COND_FALSE128]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND130]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP135:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP136:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP131:%.*]] = icmp ugt i64 [[TMP135]], [[TMP136]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP131]], label [[COND_TRUE132:%.*]], label [[COND_FALSE133:%.*]]
+// SIMD-ONLY0:       cond.true132:
+// SIMD-ONLY0-NEXT:    [[TMP137:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END134:%.*]]
+// SIMD-ONLY0:       cond.false133:
+// SIMD-ONLY0-NEXT:    [[TMP138:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END134]]
+// SIMD-ONLY0:       cond.end134:
+// SIMD-ONLY0-NEXT:    [[COND135:%.*]] = phi i64 [ [[TMP137]], [[COND_TRUE132]] ], [ [[TMP138]], [[COND_FALSE133]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND135]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP139:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP139]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP140:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP141:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP136:%.*]] = icmp ult i64 [[TMP140]], [[TMP141]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP136]], label [[COND_TRUE137:%.*]], label [[COND_FALSE138:%.*]]
+// SIMD-ONLY0:       cond.true137:
+// SIMD-ONLY0-NEXT:    [[TMP142:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END139:%.*]]
+// SIMD-ONLY0:       cond.false138:
+// SIMD-ONLY0-NEXT:    [[TMP143:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END139]]
+// SIMD-ONLY0:       cond.end139:
+// SIMD-ONLY0-NEXT:    [[COND140:%.*]] = phi i64 [ [[TMP142]], [[COND_TRUE137]] ], [ [[TMP143]], [[COND_FALSE138]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND140]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP144:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP144]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP145:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP146:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP141:%.*]] = icmp eq i64 [[TMP145]], [[TMP146]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP141]], label [[COND_TRUE142:%.*]], label [[COND_FALSE143:%.*]]
+// SIMD-ONLY0:       cond.true142:
+// SIMD-ONLY0-NEXT:    [[TMP147:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END144:%.*]]
+// SIMD-ONLY0:       cond.false143:
+// SIMD-ONLY0-NEXT:    [[TMP148:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END144]]
+// SIMD-ONLY0:       cond.end144:
+// SIMD-ONLY0-NEXT:    [[COND145:%.*]] = phi i64 [ [[TMP147]], [[COND_TRUE142]] ], [ [[TMP148]], [[COND_FALSE143]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND145]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP149:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP149]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP150:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP150]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP151:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP152:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP146:%.*]] = icmp ugt i64 [[TMP151]], [[TMP152]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP146]], label [[COND_TRUE147:%.*]], label [[COND_FALSE148:%.*]]
+// SIMD-ONLY0:       cond.true147:
+// SIMD-ONLY0-NEXT:    [[TMP153:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END149:%.*]]
+// SIMD-ONLY0:       cond.false148:
+// SIMD-ONLY0-NEXT:    [[TMP154:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END149]]
+// SIMD-ONLY0:       cond.end149:
+// SIMD-ONLY0-NEXT:    [[COND150:%.*]] = phi i64 [ [[TMP153]], [[COND_TRUE147]] ], [ [[TMP154]], [[COND_FALSE148]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND150]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP155:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP155]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP156:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP157:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP151:%.*]] = icmp ult i64 [[TMP156]], [[TMP157]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP151]], label [[COND_TRUE152:%.*]], label [[COND_FALSE153:%.*]]
+// SIMD-ONLY0:       cond.true152:
+// SIMD-ONLY0-NEXT:    [[TMP158:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END154:%.*]]
+// SIMD-ONLY0:       cond.false153:
+// SIMD-ONLY0-NEXT:    [[TMP159:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END154]]
+// SIMD-ONLY0:       cond.end154:
+// SIMD-ONLY0-NEXT:    [[COND155:%.*]] = phi i64 [ [[TMP158]], [[COND_TRUE152]] ], [ [[TMP159]], [[COND_FALSE153]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND155]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP160:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP160]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP161:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP162:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP156:%.*]] = icmp eq i64 [[TMP161]], [[TMP162]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP156]], label [[COND_TRUE157:%.*]], label [[COND_FALSE158:%.*]]
+// SIMD-ONLY0:       cond.true157:
+// SIMD-ONLY0-NEXT:    [[TMP163:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END159:%.*]]
+// SIMD-ONLY0:       cond.false158:
+// SIMD-ONLY0-NEXT:    [[TMP164:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END159]]
+// SIMD-ONLY0:       cond.end159:
+// SIMD-ONLY0-NEXT:    [[COND160:%.*]] = phi i64 [ [[TMP163]], [[COND_TRUE157]] ], [ [[TMP164]], [[COND_FALSE158]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND160]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP165:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP166:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP161:%.*]] = icmp ugt i64 [[TMP165]], [[TMP166]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP161]], label [[COND_TRUE162:%.*]], label [[COND_FALSE163:%.*]]
+// SIMD-ONLY0:       cond.true162:
+// SIMD-ONLY0-NEXT:    [[TMP167:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END164:%.*]]
+// SIMD-ONLY0:       cond.false163:
+// SIMD-ONLY0-NEXT:    [[TMP168:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END164]]
+// SIMD-ONLY0:       cond.end164:
+// SIMD-ONLY0-NEXT:    [[COND165:%.*]] = phi i64 [ [[TMP167]], [[COND_TRUE162]] ], [ [[TMP168]], [[COND_FALSE163]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND165]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP169:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP169]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP170:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP171:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP166:%.*]] = icmp ult i64 [[TMP170]], [[TMP171]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP166]], label [[COND_TRUE167:%.*]], label [[COND_FALSE168:%.*]]
+// SIMD-ONLY0:       cond.true167:
+// SIMD-ONLY0-NEXT:    [[TMP172:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END169:%.*]]
+// SIMD-ONLY0:       cond.false168:
+// SIMD-ONLY0-NEXT:    [[TMP173:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END169]]
+// SIMD-ONLY0:       cond.end169:
+// SIMD-ONLY0-NEXT:    [[COND170:%.*]] = phi i64 [ [[TMP172]], [[COND_TRUE167]] ], [ [[TMP173]], [[COND_FALSE168]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND170]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP174:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP174]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP175:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP176:%.*]] = load i64, ptr [[ULE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP171:%.*]] = icmp eq i64 [[TMP175]], [[TMP176]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP171]], label [[COND_TRUE172:%.*]], label [[COND_FALSE173:%.*]]
+// SIMD-ONLY0:       cond.true172:
+// SIMD-ONLY0-NEXT:    [[TMP177:%.*]] = load i64, ptr [[ULD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END174:%.*]]
+// SIMD-ONLY0:       cond.false173:
+// SIMD-ONLY0-NEXT:    [[TMP178:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END174]]
+// SIMD-ONLY0:       cond.end174:
+// SIMD-ONLY0-NEXT:    [[COND175:%.*]] = phi i64 [ [[TMP177]], [[COND_TRUE172]] ], [ [[TMP178]], [[COND_FALSE173]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND175]], ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP179:%.*]] = load i64, ptr [[ULX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP179]], ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP180:%.*]] = load i64, ptr [[ULV]], align 8
+// SIMD-ONLY0-NEXT:    ret i64 [[TMP180]]
+//
+//
+// SIMD-ONLY0-LABEL: @llxevd(
+// SIMD-ONLY0-NEXT:  entry:
+// SIMD-ONLY0-NEXT:    [[LLX:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LLV:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LLE:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[LLD:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[TMP0:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP0]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP:%.*]] = icmp sgt i64 [[TMP1]], [[TMP2]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]]
+// SIMD-ONLY0:       cond.true:
+// SIMD-ONLY0-NEXT:    [[TMP3:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END:%.*]]
+// SIMD-ONLY0:       cond.false:
+// SIMD-ONLY0-NEXT:    [[TMP4:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END]]
+// SIMD-ONLY0:       cond.end:
+// SIMD-ONLY0-NEXT:    [[COND:%.*]] = phi i64 [ [[TMP3]], [[COND_TRUE]] ], [ [[TMP4]], [[COND_FALSE]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP5]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP6:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP7:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP1:%.*]] = icmp slt i64 [[TMP6]], [[TMP7]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1]], label [[COND_TRUE2:%.*]], label [[COND_FALSE3:%.*]]
+// SIMD-ONLY0:       cond.true2:
+// SIMD-ONLY0-NEXT:    [[TMP8:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4:%.*]]
+// SIMD-ONLY0:       cond.false3:
+// SIMD-ONLY0-NEXT:    [[TMP9:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4]]
+// SIMD-ONLY0:       cond.end4:
+// SIMD-ONLY0-NEXT:    [[COND5:%.*]] = phi i64 [ [[TMP8]], [[COND_TRUE2]] ], [ [[TMP9]], [[COND_FALSE3]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND5]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP10:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP10]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP11:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP12:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6:%.*]] = icmp eq i64 [[TMP11]], [[TMP12]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6]], label [[COND_TRUE7:%.*]], label [[COND_FALSE8:%.*]]
+// SIMD-ONLY0:       cond.true7:
+// SIMD-ONLY0-NEXT:    [[TMP13:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END9:%.*]]
+// SIMD-ONLY0:       cond.false8:
+// SIMD-ONLY0-NEXT:    [[TMP14:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END9]]
+// SIMD-ONLY0:       cond.end9:
+// SIMD-ONLY0-NEXT:    [[COND10:%.*]] = phi i64 [ [[TMP13]], [[COND_TRUE7]] ], [ [[TMP14]], [[COND_FALSE8]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND10]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP15:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP16:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP11:%.*]] = icmp sgt i64 [[TMP15]], [[TMP16]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP11]], label [[COND_TRUE12:%.*]], label [[COND_FALSE13:%.*]]
+// SIMD-ONLY0:       cond.true12:
+// SIMD-ONLY0-NEXT:    [[TMP17:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END14:%.*]]
+// SIMD-ONLY0:       cond.false13:
+// SIMD-ONLY0-NEXT:    [[TMP18:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END14]]
+// SIMD-ONLY0:       cond.end14:
+// SIMD-ONLY0-NEXT:    [[COND15:%.*]] = phi i64 [ [[TMP17]], [[COND_TRUE12]] ], [ [[TMP18]], [[COND_FALSE13]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND15]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP19:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP19]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP20:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP21:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP16:%.*]] = icmp slt i64 [[TMP20]], [[TMP21]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP16]], label [[COND_TRUE17:%.*]], label [[COND_FALSE18:%.*]]
+// SIMD-ONLY0:       cond.true17:
+// SIMD-ONLY0-NEXT:    [[TMP22:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END19:%.*]]
+// SIMD-ONLY0:       cond.false18:
+// SIMD-ONLY0-NEXT:    [[TMP23:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END19]]
+// SIMD-ONLY0:       cond.end19:
+// SIMD-ONLY0-NEXT:    [[COND20:%.*]] = phi i64 [ [[TMP22]], [[COND_TRUE17]] ], [ [[TMP23]], [[COND_FALSE18]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND20]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP24:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP24]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP25:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP26:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP21:%.*]] = icmp eq i64 [[TMP25]], [[TMP26]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP21]], label [[COND_TRUE22:%.*]], label [[COND_FALSE23:%.*]]
+// SIMD-ONLY0:       cond.true22:
+// SIMD-ONLY0-NEXT:    [[TMP27:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END24:%.*]]
+// SIMD-ONLY0:       cond.false23:
+// SIMD-ONLY0-NEXT:    [[TMP28:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END24]]
+// SIMD-ONLY0:       cond.end24:
+// SIMD-ONLY0-NEXT:    [[COND25:%.*]] = phi i64 [ [[TMP27]], [[COND_TRUE22]] ], [ [[TMP28]], [[COND_FALSE23]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND25]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP29:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP29]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP30:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP30]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP31:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP32:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP26:%.*]] = icmp sgt i64 [[TMP31]], [[TMP32]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP26]], label [[COND_TRUE27:%.*]], label [[COND_FALSE28:%.*]]
+// SIMD-ONLY0:       cond.true27:
+// SIMD-ONLY0-NEXT:    [[TMP33:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END29:%.*]]
+// SIMD-ONLY0:       cond.false28:
+// SIMD-ONLY0-NEXT:    [[TMP34:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END29]]
+// SIMD-ONLY0:       cond.end29:
+// SIMD-ONLY0-NEXT:    [[COND30:%.*]] = phi i64 [ [[TMP33]], [[COND_TRUE27]] ], [ [[TMP34]], [[COND_FALSE28]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND30]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP35:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP35]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP36:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP37:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP31:%.*]] = icmp slt i64 [[TMP36]], [[TMP37]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP31]], label [[COND_TRUE32:%.*]], label [[COND_FALSE33:%.*]]
+// SIMD-ONLY0:       cond.true32:
+// SIMD-ONLY0-NEXT:    [[TMP38:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END34:%.*]]
+// SIMD-ONLY0:       cond.false33:
+// SIMD-ONLY0-NEXT:    [[TMP39:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END34]]
+// SIMD-ONLY0:       cond.end34:
+// SIMD-ONLY0-NEXT:    [[COND35:%.*]] = phi i64 [ [[TMP38]], [[COND_TRUE32]] ], [ [[TMP39]], [[COND_FALSE33]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND35]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP40:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP40]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP41:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP42:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP36:%.*]] = icmp eq i64 [[TMP41]], [[TMP42]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP36]], label [[COND_TRUE37:%.*]], label [[COND_FALSE38:%.*]]
+// SIMD-ONLY0:       cond.true37:
+// SIMD-ONLY0-NEXT:    [[TMP43:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END39:%.*]]
+// SIMD-ONLY0:       cond.false38:
+// SIMD-ONLY0-NEXT:    [[TMP44:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END39]]
+// SIMD-ONLY0:       cond.end39:
+// SIMD-ONLY0-NEXT:    [[COND40:%.*]] = phi i64 [ [[TMP43]], [[COND_TRUE37]] ], [ [[TMP44]], [[COND_FALSE38]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND40]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP45:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP46:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP41:%.*]] = icmp sgt i64 [[TMP45]], [[TMP46]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP41]], label [[COND_TRUE42:%.*]], label [[COND_FALSE43:%.*]]
+// SIMD-ONLY0:       cond.true42:
+// SIMD-ONLY0-NEXT:    [[TMP47:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END44:%.*]]
+// SIMD-ONLY0:       cond.false43:
+// SIMD-ONLY0-NEXT:    [[TMP48:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END44]]
+// SIMD-ONLY0:       cond.end44:
+// SIMD-ONLY0-NEXT:    [[COND45:%.*]] = phi i64 [ [[TMP47]], [[COND_TRUE42]] ], [ [[TMP48]], [[COND_FALSE43]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND45]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP49:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP49]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP50:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP51:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP46:%.*]] = icmp slt i64 [[TMP50]], [[TMP51]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP46]], label [[COND_TRUE47:%.*]], label [[COND_FALSE48:%.*]]
+// SIMD-ONLY0:       cond.true47:
+// SIMD-ONLY0-NEXT:    [[TMP52:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END49:%.*]]
+// SIMD-ONLY0:       cond.false48:
+// SIMD-ONLY0-NEXT:    [[TMP53:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END49]]
+// SIMD-ONLY0:       cond.end49:
+// SIMD-ONLY0-NEXT:    [[COND50:%.*]] = phi i64 [ [[TMP52]], [[COND_TRUE47]] ], [ [[TMP53]], [[COND_FALSE48]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND50]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP54:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP54]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP55:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP56:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP51:%.*]] = icmp eq i64 [[TMP55]], [[TMP56]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP51]], label [[COND_TRUE52:%.*]], label [[COND_FALSE53:%.*]]
+// SIMD-ONLY0:       cond.true52:
+// SIMD-ONLY0-NEXT:    [[TMP57:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END54:%.*]]
+// SIMD-ONLY0:       cond.false53:
+// SIMD-ONLY0-NEXT:    [[TMP58:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END54]]
+// SIMD-ONLY0:       cond.end54:
+// SIMD-ONLY0-NEXT:    [[COND55:%.*]] = phi i64 [ [[TMP57]], [[COND_TRUE52]] ], [ [[TMP58]], [[COND_FALSE53]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND55]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP59:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP59]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP60:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP60]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP61:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP62:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP56:%.*]] = icmp sgt i64 [[TMP61]], [[TMP62]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP56]], label [[COND_TRUE57:%.*]], label [[COND_FALSE58:%.*]]
+// SIMD-ONLY0:       cond.true57:
+// SIMD-ONLY0-NEXT:    [[TMP63:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END59:%.*]]
+// SIMD-ONLY0:       cond.false58:
+// SIMD-ONLY0-NEXT:    [[TMP64:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END59]]
+// SIMD-ONLY0:       cond.end59:
+// SIMD-ONLY0-NEXT:    [[COND60:%.*]] = phi i64 [ [[TMP63]], [[COND_TRUE57]] ], [ [[TMP64]], [[COND_FALSE58]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND60]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP65:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP65]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP66:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP67:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP61:%.*]] = icmp slt i64 [[TMP66]], [[TMP67]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP61]], label [[COND_TRUE62:%.*]], label [[COND_FALSE63:%.*]]
+// SIMD-ONLY0:       cond.true62:
+// SIMD-ONLY0-NEXT:    [[TMP68:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END64:%.*]]
+// SIMD-ONLY0:       cond.false63:
+// SIMD-ONLY0-NEXT:    [[TMP69:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END64]]
+// SIMD-ONLY0:       cond.end64:
+// SIMD-ONLY0-NEXT:    [[COND65:%.*]] = phi i64 [ [[TMP68]], [[COND_TRUE62]] ], [ [[TMP69]], [[COND_FALSE63]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND65]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP70:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP70]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP71:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP72:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP66:%.*]] = icmp eq i64 [[TMP71]], [[TMP72]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP66]], label [[COND_TRUE67:%.*]], label [[COND_FALSE68:%.*]]
+// SIMD-ONLY0:       cond.true67:
+// SIMD-ONLY0-NEXT:    [[TMP73:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END69:%.*]]
+// SIMD-ONLY0:       cond.false68:
+// SIMD-ONLY0-NEXT:    [[TMP74:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END69]]
+// SIMD-ONLY0:       cond.end69:
+// SIMD-ONLY0-NEXT:    [[COND70:%.*]] = phi i64 [ [[TMP73]], [[COND_TRUE67]] ], [ [[TMP74]], [[COND_FALSE68]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND70]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP75:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP76:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP71:%.*]] = icmp sgt i64 [[TMP75]], [[TMP76]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP71]], label [[COND_TRUE72:%.*]], label [[COND_FALSE73:%.*]]
+// SIMD-ONLY0:       cond.true72:
+// SIMD-ONLY0-NEXT:    [[TMP77:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END74:%.*]]
+// SIMD-ONLY0:       cond.false73:
+// SIMD-ONLY0-NEXT:    [[TMP78:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END74]]
+// SIMD-ONLY0:       cond.end74:
+// SIMD-ONLY0-NEXT:    [[COND75:%.*]] = phi i64 [ [[TMP77]], [[COND_TRUE72]] ], [ [[TMP78]], [[COND_FALSE73]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND75]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP79:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP79]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP80:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP81:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP76:%.*]] = icmp slt i64 [[TMP80]], [[TMP81]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP76]], label [[COND_TRUE77:%.*]], label [[COND_FALSE78:%.*]]
+// SIMD-ONLY0:       cond.true77:
+// SIMD-ONLY0-NEXT:    [[TMP82:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END79:%.*]]
+// SIMD-ONLY0:       cond.false78:
+// SIMD-ONLY0-NEXT:    [[TMP83:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END79]]
+// SIMD-ONLY0:       cond.end79:
+// SIMD-ONLY0-NEXT:    [[COND80:%.*]] = phi i64 [ [[TMP82]], [[COND_TRUE77]] ], [ [[TMP83]], [[COND_FALSE78]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND80]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP84:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP84]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP85:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP86:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP81:%.*]] = icmp eq i64 [[TMP85]], [[TMP86]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP81]], label [[COND_TRUE82:%.*]], label [[COND_FALSE83:%.*]]
+// SIMD-ONLY0:       cond.true82:
+// SIMD-ONLY0-NEXT:    [[TMP87:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END84:%.*]]
+// SIMD-ONLY0:       cond.false83:
+// SIMD-ONLY0-NEXT:    [[TMP88:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END84]]
+// SIMD-ONLY0:       cond.end84:
+// SIMD-ONLY0-NEXT:    [[COND85:%.*]] = phi i64 [ [[TMP87]], [[COND_TRUE82]] ], [ [[TMP88]], [[COND_FALSE83]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND85]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP89:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP89]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP90:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP90]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP91:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP92:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP86:%.*]] = icmp sgt i64 [[TMP91]], [[TMP92]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP86]], label [[COND_TRUE87:%.*]], label [[COND_FALSE88:%.*]]
+// SIMD-ONLY0:       cond.true87:
+// SIMD-ONLY0-NEXT:    [[TMP93:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END89:%.*]]
+// SIMD-ONLY0:       cond.false88:
+// SIMD-ONLY0-NEXT:    [[TMP94:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END89]]
+// SIMD-ONLY0:       cond.end89:
+// SIMD-ONLY0-NEXT:    [[COND90:%.*]] = phi i64 [ [[TMP93]], [[COND_TRUE87]] ], [ [[TMP94]], [[COND_FALSE88]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND90]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP95:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP95]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP96:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP97:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP91:%.*]] = icmp slt i64 [[TMP96]], [[TMP97]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP91]], label [[COND_TRUE92:%.*]], label [[COND_FALSE93:%.*]]
+// SIMD-ONLY0:       cond.true92:
+// SIMD-ONLY0-NEXT:    [[TMP98:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END94:%.*]]
+// SIMD-ONLY0:       cond.false93:
+// SIMD-ONLY0-NEXT:    [[TMP99:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END94]]
+// SIMD-ONLY0:       cond.end94:
+// SIMD-ONLY0-NEXT:    [[COND95:%.*]] = phi i64 [ [[TMP98]], [[COND_TRUE92]] ], [ [[TMP99]], [[COND_FALSE93]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND95]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP100:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP100]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP101:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP102:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP96:%.*]] = icmp eq i64 [[TMP101]], [[TMP102]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP96]], label [[COND_TRUE97:%.*]], label [[COND_FALSE98:%.*]]
+// SIMD-ONLY0:       cond.true97:
+// SIMD-ONLY0-NEXT:    [[TMP103:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END99:%.*]]
+// SIMD-ONLY0:       cond.false98:
+// SIMD-ONLY0-NEXT:    [[TMP104:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END99]]
+// SIMD-ONLY0:       cond.end99:
+// SIMD-ONLY0-NEXT:    [[COND100:%.*]] = phi i64 [ [[TMP103]], [[COND_TRUE97]] ], [ [[TMP104]], [[COND_FALSE98]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND100]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP105:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP106:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP101:%.*]] = icmp sgt i64 [[TMP105]], [[TMP106]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP101]], label [[COND_TRUE102:%.*]], label [[COND_FALSE103:%.*]]
+// SIMD-ONLY0:       cond.true102:
+// SIMD-ONLY0-NEXT:    [[TMP107:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END104:%.*]]
+// SIMD-ONLY0:       cond.false103:
+// SIMD-ONLY0-NEXT:    [[TMP108:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END104]]
+// SIMD-ONLY0:       cond.end104:
+// SIMD-ONLY0-NEXT:    [[COND105:%.*]] = phi i64 [ [[TMP107]], [[COND_TRUE102]] ], [ [[TMP108]], [[COND_FALSE103]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND105]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP109:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP109]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP110:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP111:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP106:%.*]] = icmp slt i64 [[TMP110]], [[TMP111]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP106]], label [[COND_TRUE107:%.*]], label [[COND_FALSE108:%.*]]
+// SIMD-ONLY0:       cond.true107:
+// SIMD-ONLY0-NEXT:    [[TMP112:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END109:%.*]]
+// SIMD-ONLY0:       cond.false108:
+// SIMD-ONLY0-NEXT:    [[TMP113:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END109]]
+// SIMD-ONLY0:       cond.end109:
+// SIMD-ONLY0-NEXT:    [[COND110:%.*]] = phi i64 [ [[TMP112]], [[COND_TRUE107]] ], [ [[TMP113]], [[COND_FALSE108]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND110]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP114:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP114]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP115:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP116:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP111:%.*]] = icmp eq i64 [[TMP115]], [[TMP116]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP111]], label [[COND_TRUE112:%.*]], label [[COND_FALSE113:%.*]]
+// SIMD-ONLY0:       cond.true112:
+// SIMD-ONLY0-NEXT:    [[TMP117:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END114:%.*]]
+// SIMD-ONLY0:       cond.false113:
+// SIMD-ONLY0-NEXT:    [[TMP118:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END114]]
+// SIMD-ONLY0:       cond.end114:
+// SIMD-ONLY0-NEXT:    [[COND115:%.*]] = phi i64 [ [[TMP117]], [[COND_TRUE112]] ], [ [[TMP118]], [[COND_FALSE113]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND115]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP119:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP119]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP120:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP120]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP121:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP122:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP116:%.*]] = icmp sgt i64 [[TMP121]], [[TMP122]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP116]], label [[COND_TRUE117:%.*]], label [[COND_FALSE118:%.*]]
+// SIMD-ONLY0:       cond.true117:
+// SIMD-ONLY0-NEXT:    [[TMP123:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END119:%.*]]
+// SIMD-ONLY0:       cond.false118:
+// SIMD-ONLY0-NEXT:    [[TMP124:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END119]]
+// SIMD-ONLY0:       cond.end119:
+// SIMD-ONLY0-NEXT:    [[COND120:%.*]] = phi i64 [ [[TMP123]], [[COND_TRUE117]] ], [ [[TMP124]], [[COND_FALSE118]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND120]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP125:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP125]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP126:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP127:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP121:%.*]] = icmp slt i64 [[TMP126]], [[TMP127]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP121]], label [[COND_TRUE122:%.*]], label [[COND_FALSE123:%.*]]
+// SIMD-ONLY0:       cond.true122:
+// SIMD-ONLY0-NEXT:    [[TMP128:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END124:%.*]]
+// SIMD-ONLY0:       cond.false123:
+// SIMD-ONLY0-NEXT:    [[TMP129:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END124]]
+// SIMD-ONLY0:       cond.end124:
+// SIMD-ONLY0-NEXT:    [[COND125:%.*]] = phi i64 [ [[TMP128]], [[COND_TRUE122]] ], [ [[TMP129]], [[COND_FALSE123]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND125]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP130:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP130]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP131:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP132:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP126:%.*]] = icmp eq i64 [[TMP131]], [[TMP132]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP126]], label [[COND_TRUE127:%.*]], label [[COND_FALSE128:%.*]]
+// SIMD-ONLY0:       cond.true127:
+// SIMD-ONLY0-NEXT:    [[TMP133:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END129:%.*]]
+// SIMD-ONLY0:       cond.false128:
+// SIMD-ONLY0-NEXT:    [[TMP134:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END129]]
+// SIMD-ONLY0:       cond.end129:
+// SIMD-ONLY0-NEXT:    [[COND130:%.*]] = phi i64 [ [[TMP133]], [[COND_TRUE127]] ], [ [[TMP134]], [[COND_FALSE128]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND130]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP135:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP136:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP131:%.*]] = icmp sgt i64 [[TMP135]], [[TMP136]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP131]], label [[COND_TRUE132:%.*]], label [[COND_FALSE133:%.*]]
+// SIMD-ONLY0:       cond.true132:
+// SIMD-ONLY0-NEXT:    [[TMP137:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END134:%.*]]
+// SIMD-ONLY0:       cond.false133:
+// SIMD-ONLY0-NEXT:    [[TMP138:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END134]]
+// SIMD-ONLY0:       cond.end134:
+// SIMD-ONLY0-NEXT:    [[COND135:%.*]] = phi i64 [ [[TMP137]], [[COND_TRUE132]] ], [ [[TMP138]], [[COND_FALSE133]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND135]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP139:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP139]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP140:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP141:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP136:%.*]] = icmp slt i64 [[TMP140]], [[TMP141]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP136]], label [[COND_TRUE137:%.*]], label [[COND_FALSE138:%.*]]
+// SIMD-ONLY0:       cond.true137:
+// SIMD-ONLY0-NEXT:    [[TMP142:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END139:%.*]]
+// SIMD-ONLY0:       cond.false138:
+// SIMD-ONLY0-NEXT:    [[TMP143:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END139]]
+// SIMD-ONLY0:       cond.end139:
+// SIMD-ONLY0-NEXT:    [[COND140:%.*]] = phi i64 [ [[TMP142]], [[COND_TRUE137]] ], [ [[TMP143]], [[COND_FALSE138]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND140]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP144:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP144]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP145:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP146:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP141:%.*]] = icmp eq i64 [[TMP145]], [[TMP146]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP141]], label [[COND_TRUE142:%.*]], label [[COND_FALSE143:%.*]]
+// SIMD-ONLY0:       cond.true142:
+// SIMD-ONLY0-NEXT:    [[TMP147:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END144:%.*]]
+// SIMD-ONLY0:       cond.false143:
+// SIMD-ONLY0-NEXT:    [[TMP148:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END144]]
+// SIMD-ONLY0:       cond.end144:
+// SIMD-ONLY0-NEXT:    [[COND145:%.*]] = phi i64 [ [[TMP147]], [[COND_TRUE142]] ], [ [[TMP148]], [[COND_FALSE143]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND145]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP149:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP149]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP150:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP150]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP151:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP152:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP146:%.*]] = icmp sgt i64 [[TMP151]], [[TMP152]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP146]], label [[COND_TRUE147:%.*]], label [[COND_FALSE148:%.*]]
+// SIMD-ONLY0:       cond.true147:
+// SIMD-ONLY0-NEXT:    [[TMP153:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END149:%.*]]
+// SIMD-ONLY0:       cond.false148:
+// SIMD-ONLY0-NEXT:    [[TMP154:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END149]]
+// SIMD-ONLY0:       cond.end149:
+// SIMD-ONLY0-NEXT:    [[COND150:%.*]] = phi i64 [ [[TMP153]], [[COND_TRUE147]] ], [ [[TMP154]], [[COND_FALSE148]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND150]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP155:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP155]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP156:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP157:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP151:%.*]] = icmp slt i64 [[TMP156]], [[TMP157]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP151]], label [[COND_TRUE152:%.*]], label [[COND_FALSE153:%.*]]
+// SIMD-ONLY0:       cond.true152:
+// SIMD-ONLY0-NEXT:    [[TMP158:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END154:%.*]]
+// SIMD-ONLY0:       cond.false153:
+// SIMD-ONLY0-NEXT:    [[TMP159:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END154]]
+// SIMD-ONLY0:       cond.end154:
+// SIMD-ONLY0-NEXT:    [[COND155:%.*]] = phi i64 [ [[TMP158]], [[COND_TRUE152]] ], [ [[TMP159]], [[COND_FALSE153]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND155]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP160:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP160]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP161:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP162:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP156:%.*]] = icmp eq i64 [[TMP161]], [[TMP162]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP156]], label [[COND_TRUE157:%.*]], label [[COND_FALSE158:%.*]]
+// SIMD-ONLY0:       cond.true157:
+// SIMD-ONLY0-NEXT:    [[TMP163:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END159:%.*]]
+// SIMD-ONLY0:       cond.false158:
+// SIMD-ONLY0-NEXT:    [[TMP164:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END159]]
+// SIMD-ONLY0:       cond.end159:
+// SIMD-ONLY0-NEXT:    [[COND160:%.*]] = phi i64 [ [[TMP163]], [[COND_TRUE157]] ], [ [[TMP164]], [[COND_FALSE158]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND160]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP165:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP166:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP161:%.*]] = icmp sgt i64 [[TMP165]], [[TMP166]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP161]], label [[COND_TRUE162:%.*]], label [[COND_FALSE163:%.*]]
+// SIMD-ONLY0:       cond.true162:
+// SIMD-ONLY0-NEXT:    [[TMP167:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END164:%.*]]
+// SIMD-ONLY0:       cond.false163:
+// SIMD-ONLY0-NEXT:    [[TMP168:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END164]]
+// SIMD-ONLY0:       cond.end164:
+// SIMD-ONLY0-NEXT:    [[COND165:%.*]] = phi i64 [ [[TMP167]], [[COND_TRUE162]] ], [ [[TMP168]], [[COND_FALSE163]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND165]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP169:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP169]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP170:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP171:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP166:%.*]] = icmp slt i64 [[TMP170]], [[TMP171]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP166]], label [[COND_TRUE167:%.*]], label [[COND_FALSE168:%.*]]
+// SIMD-ONLY0:       cond.true167:
+// SIMD-ONLY0-NEXT:    [[TMP172:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END169:%.*]]
+// SIMD-ONLY0:       cond.false168:
+// SIMD-ONLY0-NEXT:    [[TMP173:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END169]]
+// SIMD-ONLY0:       cond.end169:
+// SIMD-ONLY0-NEXT:    [[COND170:%.*]] = phi i64 [ [[TMP172]], [[COND_TRUE167]] ], [ [[TMP173]], [[COND_FALSE168]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND170]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP174:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP174]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP175:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP176:%.*]] = load i64, ptr [[LLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP171:%.*]] = icmp eq i64 [[TMP175]], [[TMP176]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP171]], label [[COND_TRUE172:%.*]], label [[COND_FALSE173:%.*]]
+// SIMD-ONLY0:       cond.true172:
+// SIMD-ONLY0-NEXT:    [[TMP177:%.*]] = load i64, ptr [[LLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END174:%.*]]
+// SIMD-ONLY0:       cond.false173:
+// SIMD-ONLY0-NEXT:    [[TMP178:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END174]]
+// SIMD-ONLY0:       cond.end174:
+// SIMD-ONLY0-NEXT:    [[COND175:%.*]] = phi i64 [ [[TMP177]], [[COND_TRUE172]] ], [ [[TMP178]], [[COND_FALSE173]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND175]], ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP179:%.*]] = load i64, ptr [[LLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP179]], ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP180:%.*]] = load i64, ptr [[LLV]], align 8
+// SIMD-ONLY0-NEXT:    ret i64 [[TMP180]]
+//
+//
+// SIMD-ONLY0-LABEL: @ullxevd(
+// SIMD-ONLY0-NEXT:  entry:
+// SIMD-ONLY0-NEXT:    [[ULLX:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULLV:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULLE:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[ULLD:%.*]] = alloca i64, align 8
+// SIMD-ONLY0-NEXT:    [[TMP0:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP0]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP:%.*]] = icmp ugt i64 [[TMP1]], [[TMP2]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]]
+// SIMD-ONLY0:       cond.true:
+// SIMD-ONLY0-NEXT:    [[TMP3:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END:%.*]]
+// SIMD-ONLY0:       cond.false:
+// SIMD-ONLY0-NEXT:    [[TMP4:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END]]
+// SIMD-ONLY0:       cond.end:
+// SIMD-ONLY0-NEXT:    [[COND:%.*]] = phi i64 [ [[TMP3]], [[COND_TRUE]] ], [ [[TMP4]], [[COND_FALSE]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP5]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP6:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP7:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP1:%.*]] = icmp ult i64 [[TMP6]], [[TMP7]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1]], label [[COND_TRUE2:%.*]], label [[COND_FALSE3:%.*]]
+// SIMD-ONLY0:       cond.true2:
+// SIMD-ONLY0-NEXT:    [[TMP8:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4:%.*]]
+// SIMD-ONLY0:       cond.false3:
+// SIMD-ONLY0-NEXT:    [[TMP9:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4]]
+// SIMD-ONLY0:       cond.end4:
+// SIMD-ONLY0-NEXT:    [[COND5:%.*]] = phi i64 [ [[TMP8]], [[COND_TRUE2]] ], [ [[TMP9]], [[COND_FALSE3]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND5]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP10:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP10]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP11:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP12:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6:%.*]] = icmp eq i64 [[TMP11]], [[TMP12]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6]], label [[COND_TRUE7:%.*]], label [[COND_FALSE8:%.*]]
+// SIMD-ONLY0:       cond.true7:
+// SIMD-ONLY0-NEXT:    [[TMP13:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END9:%.*]]
+// SIMD-ONLY0:       cond.false8:
+// SIMD-ONLY0-NEXT:    [[TMP14:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END9]]
+// SIMD-ONLY0:       cond.end9:
+// SIMD-ONLY0-NEXT:    [[COND10:%.*]] = phi i64 [ [[TMP13]], [[COND_TRUE7]] ], [ [[TMP14]], [[COND_FALSE8]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND10]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP15:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP16:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP11:%.*]] = icmp ugt i64 [[TMP15]], [[TMP16]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP11]], label [[COND_TRUE12:%.*]], label [[COND_FALSE13:%.*]]
+// SIMD-ONLY0:       cond.true12:
+// SIMD-ONLY0-NEXT:    [[TMP17:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END14:%.*]]
+// SIMD-ONLY0:       cond.false13:
+// SIMD-ONLY0-NEXT:    [[TMP18:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END14]]
+// SIMD-ONLY0:       cond.end14:
+// SIMD-ONLY0-NEXT:    [[COND15:%.*]] = phi i64 [ [[TMP17]], [[COND_TRUE12]] ], [ [[TMP18]], [[COND_FALSE13]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND15]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP19:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP19]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP20:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP21:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP16:%.*]] = icmp ult i64 [[TMP20]], [[TMP21]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP16]], label [[COND_TRUE17:%.*]], label [[COND_FALSE18:%.*]]
+// SIMD-ONLY0:       cond.true17:
+// SIMD-ONLY0-NEXT:    [[TMP22:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END19:%.*]]
+// SIMD-ONLY0:       cond.false18:
+// SIMD-ONLY0-NEXT:    [[TMP23:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END19]]
+// SIMD-ONLY0:       cond.end19:
+// SIMD-ONLY0-NEXT:    [[COND20:%.*]] = phi i64 [ [[TMP22]], [[COND_TRUE17]] ], [ [[TMP23]], [[COND_FALSE18]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND20]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP24:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP24]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP25:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP26:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP21:%.*]] = icmp eq i64 [[TMP25]], [[TMP26]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP21]], label [[COND_TRUE22:%.*]], label [[COND_FALSE23:%.*]]
+// SIMD-ONLY0:       cond.true22:
+// SIMD-ONLY0-NEXT:    [[TMP27:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END24:%.*]]
+// SIMD-ONLY0:       cond.false23:
+// SIMD-ONLY0-NEXT:    [[TMP28:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END24]]
+// SIMD-ONLY0:       cond.end24:
+// SIMD-ONLY0-NEXT:    [[COND25:%.*]] = phi i64 [ [[TMP27]], [[COND_TRUE22]] ], [ [[TMP28]], [[COND_FALSE23]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND25]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP29:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP29]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP30:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP30]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP31:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP32:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP26:%.*]] = icmp ugt i64 [[TMP31]], [[TMP32]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP26]], label [[COND_TRUE27:%.*]], label [[COND_FALSE28:%.*]]
+// SIMD-ONLY0:       cond.true27:
+// SIMD-ONLY0-NEXT:    [[TMP33:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END29:%.*]]
+// SIMD-ONLY0:       cond.false28:
+// SIMD-ONLY0-NEXT:    [[TMP34:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END29]]
+// SIMD-ONLY0:       cond.end29:
+// SIMD-ONLY0-NEXT:    [[COND30:%.*]] = phi i64 [ [[TMP33]], [[COND_TRUE27]] ], [ [[TMP34]], [[COND_FALSE28]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND30]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP35:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP35]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP36:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP37:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP31:%.*]] = icmp ult i64 [[TMP36]], [[TMP37]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP31]], label [[COND_TRUE32:%.*]], label [[COND_FALSE33:%.*]]
+// SIMD-ONLY0:       cond.true32:
+// SIMD-ONLY0-NEXT:    [[TMP38:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END34:%.*]]
+// SIMD-ONLY0:       cond.false33:
+// SIMD-ONLY0-NEXT:    [[TMP39:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END34]]
+// SIMD-ONLY0:       cond.end34:
+// SIMD-ONLY0-NEXT:    [[COND35:%.*]] = phi i64 [ [[TMP38]], [[COND_TRUE32]] ], [ [[TMP39]], [[COND_FALSE33]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND35]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP40:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP40]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP41:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP42:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP36:%.*]] = icmp eq i64 [[TMP41]], [[TMP42]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP36]], label [[COND_TRUE37:%.*]], label [[COND_FALSE38:%.*]]
+// SIMD-ONLY0:       cond.true37:
+// SIMD-ONLY0-NEXT:    [[TMP43:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END39:%.*]]
+// SIMD-ONLY0:       cond.false38:
+// SIMD-ONLY0-NEXT:    [[TMP44:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END39]]
+// SIMD-ONLY0:       cond.end39:
+// SIMD-ONLY0-NEXT:    [[COND40:%.*]] = phi i64 [ [[TMP43]], [[COND_TRUE37]] ], [ [[TMP44]], [[COND_FALSE38]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND40]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP45:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP46:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP41:%.*]] = icmp ugt i64 [[TMP45]], [[TMP46]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP41]], label [[COND_TRUE42:%.*]], label [[COND_FALSE43:%.*]]
+// SIMD-ONLY0:       cond.true42:
+// SIMD-ONLY0-NEXT:    [[TMP47:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END44:%.*]]
+// SIMD-ONLY0:       cond.false43:
+// SIMD-ONLY0-NEXT:    [[TMP48:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END44]]
+// SIMD-ONLY0:       cond.end44:
+// SIMD-ONLY0-NEXT:    [[COND45:%.*]] = phi i64 [ [[TMP47]], [[COND_TRUE42]] ], [ [[TMP48]], [[COND_FALSE43]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND45]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP49:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP49]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP50:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP51:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP46:%.*]] = icmp ult i64 [[TMP50]], [[TMP51]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP46]], label [[COND_TRUE47:%.*]], label [[COND_FALSE48:%.*]]
+// SIMD-ONLY0:       cond.true47:
+// SIMD-ONLY0-NEXT:    [[TMP52:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END49:%.*]]
+// SIMD-ONLY0:       cond.false48:
+// SIMD-ONLY0-NEXT:    [[TMP53:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END49]]
+// SIMD-ONLY0:       cond.end49:
+// SIMD-ONLY0-NEXT:    [[COND50:%.*]] = phi i64 [ [[TMP52]], [[COND_TRUE47]] ], [ [[TMP53]], [[COND_FALSE48]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND50]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP54:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP54]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP55:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP56:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP51:%.*]] = icmp eq i64 [[TMP55]], [[TMP56]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP51]], label [[COND_TRUE52:%.*]], label [[COND_FALSE53:%.*]]
+// SIMD-ONLY0:       cond.true52:
+// SIMD-ONLY0-NEXT:    [[TMP57:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END54:%.*]]
+// SIMD-ONLY0:       cond.false53:
+// SIMD-ONLY0-NEXT:    [[TMP58:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END54]]
+// SIMD-ONLY0:       cond.end54:
+// SIMD-ONLY0-NEXT:    [[COND55:%.*]] = phi i64 [ [[TMP57]], [[COND_TRUE52]] ], [ [[TMP58]], [[COND_FALSE53]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND55]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP59:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP59]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP60:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP60]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP61:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP62:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP56:%.*]] = icmp ugt i64 [[TMP61]], [[TMP62]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP56]], label [[COND_TRUE57:%.*]], label [[COND_FALSE58:%.*]]
+// SIMD-ONLY0:       cond.true57:
+// SIMD-ONLY0-NEXT:    [[TMP63:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END59:%.*]]
+// SIMD-ONLY0:       cond.false58:
+// SIMD-ONLY0-NEXT:    [[TMP64:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END59]]
+// SIMD-ONLY0:       cond.end59:
+// SIMD-ONLY0-NEXT:    [[COND60:%.*]] = phi i64 [ [[TMP63]], [[COND_TRUE57]] ], [ [[TMP64]], [[COND_FALSE58]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND60]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP65:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP65]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP66:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP67:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP61:%.*]] = icmp ult i64 [[TMP66]], [[TMP67]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP61]], label [[COND_TRUE62:%.*]], label [[COND_FALSE63:%.*]]
+// SIMD-ONLY0:       cond.true62:
+// SIMD-ONLY0-NEXT:    [[TMP68:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END64:%.*]]
+// SIMD-ONLY0:       cond.false63:
+// SIMD-ONLY0-NEXT:    [[TMP69:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END64]]
+// SIMD-ONLY0:       cond.end64:
+// SIMD-ONLY0-NEXT:    [[COND65:%.*]] = phi i64 [ [[TMP68]], [[COND_TRUE62]] ], [ [[TMP69]], [[COND_FALSE63]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND65]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP70:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP70]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP71:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP72:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP66:%.*]] = icmp eq i64 [[TMP71]], [[TMP72]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP66]], label [[COND_TRUE67:%.*]], label [[COND_FALSE68:%.*]]
+// SIMD-ONLY0:       cond.true67:
+// SIMD-ONLY0-NEXT:    [[TMP73:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END69:%.*]]
+// SIMD-ONLY0:       cond.false68:
+// SIMD-ONLY0-NEXT:    [[TMP74:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END69]]
+// SIMD-ONLY0:       cond.end69:
+// SIMD-ONLY0-NEXT:    [[COND70:%.*]] = phi i64 [ [[TMP73]], [[COND_TRUE67]] ], [ [[TMP74]], [[COND_FALSE68]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND70]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP75:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP76:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP71:%.*]] = icmp ugt i64 [[TMP75]], [[TMP76]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP71]], label [[COND_TRUE72:%.*]], label [[COND_FALSE73:%.*]]
+// SIMD-ONLY0:       cond.true72:
+// SIMD-ONLY0-NEXT:    [[TMP77:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END74:%.*]]
+// SIMD-ONLY0:       cond.false73:
+// SIMD-ONLY0-NEXT:    [[TMP78:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END74]]
+// SIMD-ONLY0:       cond.end74:
+// SIMD-ONLY0-NEXT:    [[COND75:%.*]] = phi i64 [ [[TMP77]], [[COND_TRUE72]] ], [ [[TMP78]], [[COND_FALSE73]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND75]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP79:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP79]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP80:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP81:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP76:%.*]] = icmp ult i64 [[TMP80]], [[TMP81]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP76]], label [[COND_TRUE77:%.*]], label [[COND_FALSE78:%.*]]
+// SIMD-ONLY0:       cond.true77:
+// SIMD-ONLY0-NEXT:    [[TMP82:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END79:%.*]]
+// SIMD-ONLY0:       cond.false78:
+// SIMD-ONLY0-NEXT:    [[TMP83:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END79]]
+// SIMD-ONLY0:       cond.end79:
+// SIMD-ONLY0-NEXT:    [[COND80:%.*]] = phi i64 [ [[TMP82]], [[COND_TRUE77]] ], [ [[TMP83]], [[COND_FALSE78]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND80]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP84:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP84]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP85:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP86:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP81:%.*]] = icmp eq i64 [[TMP85]], [[TMP86]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP81]], label [[COND_TRUE82:%.*]], label [[COND_FALSE83:%.*]]
+// SIMD-ONLY0:       cond.true82:
+// SIMD-ONLY0-NEXT:    [[TMP87:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END84:%.*]]
+// SIMD-ONLY0:       cond.false83:
+// SIMD-ONLY0-NEXT:    [[TMP88:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END84]]
+// SIMD-ONLY0:       cond.end84:
+// SIMD-ONLY0-NEXT:    [[COND85:%.*]] = phi i64 [ [[TMP87]], [[COND_TRUE82]] ], [ [[TMP88]], [[COND_FALSE83]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND85]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP89:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP89]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP90:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP90]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP91:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP92:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP86:%.*]] = icmp ugt i64 [[TMP91]], [[TMP92]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP86]], label [[COND_TRUE87:%.*]], label [[COND_FALSE88:%.*]]
+// SIMD-ONLY0:       cond.true87:
+// SIMD-ONLY0-NEXT:    [[TMP93:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END89:%.*]]
+// SIMD-ONLY0:       cond.false88:
+// SIMD-ONLY0-NEXT:    [[TMP94:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END89]]
+// SIMD-ONLY0:       cond.end89:
+// SIMD-ONLY0-NEXT:    [[COND90:%.*]] = phi i64 [ [[TMP93]], [[COND_TRUE87]] ], [ [[TMP94]], [[COND_FALSE88]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND90]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP95:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP95]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP96:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP97:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP91:%.*]] = icmp ult i64 [[TMP96]], [[TMP97]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP91]], label [[COND_TRUE92:%.*]], label [[COND_FALSE93:%.*]]
+// SIMD-ONLY0:       cond.true92:
+// SIMD-ONLY0-NEXT:    [[TMP98:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END94:%.*]]
+// SIMD-ONLY0:       cond.false93:
+// SIMD-ONLY0-NEXT:    [[TMP99:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END94]]
+// SIMD-ONLY0:       cond.end94:
+// SIMD-ONLY0-NEXT:    [[COND95:%.*]] = phi i64 [ [[TMP98]], [[COND_TRUE92]] ], [ [[TMP99]], [[COND_FALSE93]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND95]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP100:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP100]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP101:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP102:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP96:%.*]] = icmp eq i64 [[TMP101]], [[TMP102]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP96]], label [[COND_TRUE97:%.*]], label [[COND_FALSE98:%.*]]
+// SIMD-ONLY0:       cond.true97:
+// SIMD-ONLY0-NEXT:    [[TMP103:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END99:%.*]]
+// SIMD-ONLY0:       cond.false98:
+// SIMD-ONLY0-NEXT:    [[TMP104:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END99]]
+// SIMD-ONLY0:       cond.end99:
+// SIMD-ONLY0-NEXT:    [[COND100:%.*]] = phi i64 [ [[TMP103]], [[COND_TRUE97]] ], [ [[TMP104]], [[COND_FALSE98]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND100]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP105:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP106:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP101:%.*]] = icmp ugt i64 [[TMP105]], [[TMP106]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP101]], label [[COND_TRUE102:%.*]], label [[COND_FALSE103:%.*]]
+// SIMD-ONLY0:       cond.true102:
+// SIMD-ONLY0-NEXT:    [[TMP107:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END104:%.*]]
+// SIMD-ONLY0:       cond.false103:
+// SIMD-ONLY0-NEXT:    [[TMP108:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END104]]
+// SIMD-ONLY0:       cond.end104:
+// SIMD-ONLY0-NEXT:    [[COND105:%.*]] = phi i64 [ [[TMP107]], [[COND_TRUE102]] ], [ [[TMP108]], [[COND_FALSE103]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND105]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP109:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP109]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP110:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP111:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP106:%.*]] = icmp ult i64 [[TMP110]], [[TMP111]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP106]], label [[COND_TRUE107:%.*]], label [[COND_FALSE108:%.*]]
+// SIMD-ONLY0:       cond.true107:
+// SIMD-ONLY0-NEXT:    [[TMP112:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END109:%.*]]
+// SIMD-ONLY0:       cond.false108:
+// SIMD-ONLY0-NEXT:    [[TMP113:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END109]]
+// SIMD-ONLY0:       cond.end109:
+// SIMD-ONLY0-NEXT:    [[COND110:%.*]] = phi i64 [ [[TMP112]], [[COND_TRUE107]] ], [ [[TMP113]], [[COND_FALSE108]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND110]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP114:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP114]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP115:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP116:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP111:%.*]] = icmp eq i64 [[TMP115]], [[TMP116]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP111]], label [[COND_TRUE112:%.*]], label [[COND_FALSE113:%.*]]
+// SIMD-ONLY0:       cond.true112:
+// SIMD-ONLY0-NEXT:    [[TMP117:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END114:%.*]]
+// SIMD-ONLY0:       cond.false113:
+// SIMD-ONLY0-NEXT:    [[TMP118:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END114]]
+// SIMD-ONLY0:       cond.end114:
+// SIMD-ONLY0-NEXT:    [[COND115:%.*]] = phi i64 [ [[TMP117]], [[COND_TRUE112]] ], [ [[TMP118]], [[COND_FALSE113]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND115]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP119:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP119]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP120:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP120]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP121:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP122:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP116:%.*]] = icmp ugt i64 [[TMP121]], [[TMP122]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP116]], label [[COND_TRUE117:%.*]], label [[COND_FALSE118:%.*]]
+// SIMD-ONLY0:       cond.true117:
+// SIMD-ONLY0-NEXT:    [[TMP123:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END119:%.*]]
+// SIMD-ONLY0:       cond.false118:
+// SIMD-ONLY0-NEXT:    [[TMP124:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END119]]
+// SIMD-ONLY0:       cond.end119:
+// SIMD-ONLY0-NEXT:    [[COND120:%.*]] = phi i64 [ [[TMP123]], [[COND_TRUE117]] ], [ [[TMP124]], [[COND_FALSE118]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND120]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP125:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP125]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP126:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP127:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP121:%.*]] = icmp ult i64 [[TMP126]], [[TMP127]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP121]], label [[COND_TRUE122:%.*]], label [[COND_FALSE123:%.*]]
+// SIMD-ONLY0:       cond.true122:
+// SIMD-ONLY0-NEXT:    [[TMP128:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END124:%.*]]
+// SIMD-ONLY0:       cond.false123:
+// SIMD-ONLY0-NEXT:    [[TMP129:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END124]]
+// SIMD-ONLY0:       cond.end124:
+// SIMD-ONLY0-NEXT:    [[COND125:%.*]] = phi i64 [ [[TMP128]], [[COND_TRUE122]] ], [ [[TMP129]], [[COND_FALSE123]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND125]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP130:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP130]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP131:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP132:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP126:%.*]] = icmp eq i64 [[TMP131]], [[TMP132]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP126]], label [[COND_TRUE127:%.*]], label [[COND_FALSE128:%.*]]
+// SIMD-ONLY0:       cond.true127:
+// SIMD-ONLY0-NEXT:    [[TMP133:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END129:%.*]]
+// SIMD-ONLY0:       cond.false128:
+// SIMD-ONLY0-NEXT:    [[TMP134:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END129]]
+// SIMD-ONLY0:       cond.end129:
+// SIMD-ONLY0-NEXT:    [[COND130:%.*]] = phi i64 [ [[TMP133]], [[COND_TRUE127]] ], [ [[TMP134]], [[COND_FALSE128]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND130]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP135:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP136:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP131:%.*]] = icmp ugt i64 [[TMP135]], [[TMP136]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP131]], label [[COND_TRUE132:%.*]], label [[COND_FALSE133:%.*]]
+// SIMD-ONLY0:       cond.true132:
+// SIMD-ONLY0-NEXT:    [[TMP137:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END134:%.*]]
+// SIMD-ONLY0:       cond.false133:
+// SIMD-ONLY0-NEXT:    [[TMP138:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END134]]
+// SIMD-ONLY0:       cond.end134:
+// SIMD-ONLY0-NEXT:    [[COND135:%.*]] = phi i64 [ [[TMP137]], [[COND_TRUE132]] ], [ [[TMP138]], [[COND_FALSE133]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND135]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP139:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP139]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP140:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP141:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP136:%.*]] = icmp ult i64 [[TMP140]], [[TMP141]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP136]], label [[COND_TRUE137:%.*]], label [[COND_FALSE138:%.*]]
+// SIMD-ONLY0:       cond.true137:
+// SIMD-ONLY0-NEXT:    [[TMP142:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END139:%.*]]
+// SIMD-ONLY0:       cond.false138:
+// SIMD-ONLY0-NEXT:    [[TMP143:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END139]]
+// SIMD-ONLY0:       cond.end139:
+// SIMD-ONLY0-NEXT:    [[COND140:%.*]] = phi i64 [ [[TMP142]], [[COND_TRUE137]] ], [ [[TMP143]], [[COND_FALSE138]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND140]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP144:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP144]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP145:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP146:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP141:%.*]] = icmp eq i64 [[TMP145]], [[TMP146]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP141]], label [[COND_TRUE142:%.*]], label [[COND_FALSE143:%.*]]
+// SIMD-ONLY0:       cond.true142:
+// SIMD-ONLY0-NEXT:    [[TMP147:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END144:%.*]]
+// SIMD-ONLY0:       cond.false143:
+// SIMD-ONLY0-NEXT:    [[TMP148:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END144]]
+// SIMD-ONLY0:       cond.end144:
+// SIMD-ONLY0-NEXT:    [[COND145:%.*]] = phi i64 [ [[TMP147]], [[COND_TRUE142]] ], [ [[TMP148]], [[COND_FALSE143]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND145]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP149:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP149]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP150:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP150]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP151:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP152:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP146:%.*]] = icmp ugt i64 [[TMP151]], [[TMP152]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP146]], label [[COND_TRUE147:%.*]], label [[COND_FALSE148:%.*]]
+// SIMD-ONLY0:       cond.true147:
+// SIMD-ONLY0-NEXT:    [[TMP153:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END149:%.*]]
+// SIMD-ONLY0:       cond.false148:
+// SIMD-ONLY0-NEXT:    [[TMP154:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END149]]
+// SIMD-ONLY0:       cond.end149:
+// SIMD-ONLY0-NEXT:    [[COND150:%.*]] = phi i64 [ [[TMP153]], [[COND_TRUE147]] ], [ [[TMP154]], [[COND_FALSE148]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND150]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP155:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP155]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP156:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP157:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP151:%.*]] = icmp ult i64 [[TMP156]], [[TMP157]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP151]], label [[COND_TRUE152:%.*]], label [[COND_FALSE153:%.*]]
+// SIMD-ONLY0:       cond.true152:
+// SIMD-ONLY0-NEXT:    [[TMP158:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END154:%.*]]
+// SIMD-ONLY0:       cond.false153:
+// SIMD-ONLY0-NEXT:    [[TMP159:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END154]]
+// SIMD-ONLY0:       cond.end154:
+// SIMD-ONLY0-NEXT:    [[COND155:%.*]] = phi i64 [ [[TMP158]], [[COND_TRUE152]] ], [ [[TMP159]], [[COND_FALSE153]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND155]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP160:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP160]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP161:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP162:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP156:%.*]] = icmp eq i64 [[TMP161]], [[TMP162]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP156]], label [[COND_TRUE157:%.*]], label [[COND_FALSE158:%.*]]
+// SIMD-ONLY0:       cond.true157:
+// SIMD-ONLY0-NEXT:    [[TMP163:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END159:%.*]]
+// SIMD-ONLY0:       cond.false158:
+// SIMD-ONLY0-NEXT:    [[TMP164:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END159]]
+// SIMD-ONLY0:       cond.end159:
+// SIMD-ONLY0-NEXT:    [[COND160:%.*]] = phi i64 [ [[TMP163]], [[COND_TRUE157]] ], [ [[TMP164]], [[COND_FALSE158]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND160]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP165:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP166:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP161:%.*]] = icmp ugt i64 [[TMP165]], [[TMP166]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP161]], label [[COND_TRUE162:%.*]], label [[COND_FALSE163:%.*]]
+// SIMD-ONLY0:       cond.true162:
+// SIMD-ONLY0-NEXT:    [[TMP167:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END164:%.*]]
+// SIMD-ONLY0:       cond.false163:
+// SIMD-ONLY0-NEXT:    [[TMP168:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END164]]
+// SIMD-ONLY0:       cond.end164:
+// SIMD-ONLY0-NEXT:    [[COND165:%.*]] = phi i64 [ [[TMP167]], [[COND_TRUE162]] ], [ [[TMP168]], [[COND_FALSE163]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND165]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP169:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP169]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP170:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP171:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP166:%.*]] = icmp ult i64 [[TMP170]], [[TMP171]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP166]], label [[COND_TRUE167:%.*]], label [[COND_FALSE168:%.*]]
+// SIMD-ONLY0:       cond.true167:
+// SIMD-ONLY0-NEXT:    [[TMP172:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END169:%.*]]
+// SIMD-ONLY0:       cond.false168:
+// SIMD-ONLY0-NEXT:    [[TMP173:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END169]]
+// SIMD-ONLY0:       cond.end169:
+// SIMD-ONLY0-NEXT:    [[COND170:%.*]] = phi i64 [ [[TMP172]], [[COND_TRUE167]] ], [ [[TMP173]], [[COND_FALSE168]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND170]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP174:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP174]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP175:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP176:%.*]] = load i64, ptr [[ULLE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP171:%.*]] = icmp eq i64 [[TMP175]], [[TMP176]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP171]], label [[COND_TRUE172:%.*]], label [[COND_FALSE173:%.*]]
+// SIMD-ONLY0:       cond.true172:
+// SIMD-ONLY0-NEXT:    [[TMP177:%.*]] = load i64, ptr [[ULLD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END174:%.*]]
+// SIMD-ONLY0:       cond.false173:
+// SIMD-ONLY0-NEXT:    [[TMP178:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END174]]
+// SIMD-ONLY0:       cond.end174:
+// SIMD-ONLY0-NEXT:    [[COND175:%.*]] = phi i64 [ [[TMP177]], [[COND_TRUE172]] ], [ [[TMP178]], [[COND_FALSE173]] ]
+// SIMD-ONLY0-NEXT:    store i64 [[COND175]], ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP179:%.*]] = load i64, ptr [[ULLX]], align 8
+// SIMD-ONLY0-NEXT:    store i64 [[TMP179]], ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP180:%.*]] = load i64, ptr [[ULLV]], align 8
+// SIMD-ONLY0-NEXT:    ret i64 [[TMP180]]
+//
+//
+// SIMD-ONLY0-LABEL: @fxevd(
+// SIMD-ONLY0-NEXT:  entry:
+// SIMD-ONLY0-NEXT:    [[FX:%.*]] = alloca float, align 4
+// SIMD-ONLY0-NEXT:    [[FV:%.*]] = alloca float, align 4
+// SIMD-ONLY0-NEXT:    [[FE:%.*]] = alloca float, align 4
+// SIMD-ONLY0-NEXT:    [[FD:%.*]] = alloca float, align 4
+// SIMD-ONLY0-NEXT:    [[TMP0:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP0]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP2:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP:%.*]] = fcmp ogt float [[TMP1]], [[TMP2]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]]
+// SIMD-ONLY0:       cond.true:
+// SIMD-ONLY0-NEXT:    [[TMP3:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END:%.*]]
+// SIMD-ONLY0:       cond.false:
+// SIMD-ONLY0-NEXT:    [[TMP4:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END]]
+// SIMD-ONLY0:       cond.end:
+// SIMD-ONLY0-NEXT:    [[COND:%.*]] = phi float [ [[TMP3]], [[COND_TRUE]] ], [ [[TMP4]], [[COND_FALSE]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP5:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP5]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP6:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP7:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP1:%.*]] = fcmp olt float [[TMP6]], [[TMP7]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1]], label [[COND_TRUE2:%.*]], label [[COND_FALSE3:%.*]]
+// SIMD-ONLY0:       cond.true2:
+// SIMD-ONLY0-NEXT:    [[TMP8:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4:%.*]]
+// SIMD-ONLY0:       cond.false3:
+// SIMD-ONLY0-NEXT:    [[TMP9:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END4]]
+// SIMD-ONLY0:       cond.end4:
+// SIMD-ONLY0-NEXT:    [[COND5:%.*]] = phi float [ [[TMP8]], [[COND_TRUE2]] ], [ [[TMP9]], [[COND_FALSE3]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND5]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP10:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP10]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP11:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP12:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP6:%.*]] = fcmp oeq float [[TMP11]], [[TMP12]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6]], label [[COND_TRUE7:%.*]], label [[COND_FALSE8:%.*]]
+// SIMD-ONLY0:       cond.true7:
+// SIMD-ONLY0-NEXT:    [[TMP13:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END9:%.*]]
+// SIMD-ONLY0:       cond.false8:
+// SIMD-ONLY0-NEXT:    [[TMP14:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END9]]
+// SIMD-ONLY0:       cond.end9:
+// SIMD-ONLY0-NEXT:    [[COND10:%.*]] = phi float [ [[TMP13]], [[COND_TRUE7]] ], [ [[TMP14]], [[COND_FALSE8]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND10]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP15:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP16:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP11:%.*]] = fcmp ogt float [[TMP15]], [[TMP16]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP11]], label [[COND_TRUE12:%.*]], label [[COND_FALSE13:%.*]]
+// SIMD-ONLY0:       cond.true12:
+// SIMD-ONLY0-NEXT:    [[TMP17:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END14:%.*]]
+// SIMD-ONLY0:       cond.false13:
+// SIMD-ONLY0-NEXT:    [[TMP18:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END14]]
+// SIMD-ONLY0:       cond.end14:
+// SIMD-ONLY0-NEXT:    [[COND15:%.*]] = phi float [ [[TMP17]], [[COND_TRUE12]] ], [ [[TMP18]], [[COND_FALSE13]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND15]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP19:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP19]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP20:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP21:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP16:%.*]] = fcmp olt float [[TMP20]], [[TMP21]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP16]], label [[COND_TRUE17:%.*]], label [[COND_FALSE18:%.*]]
+// SIMD-ONLY0:       cond.true17:
+// SIMD-ONLY0-NEXT:    [[TMP22:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END19:%.*]]
+// SIMD-ONLY0:       cond.false18:
+// SIMD-ONLY0-NEXT:    [[TMP23:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END19]]
+// SIMD-ONLY0:       cond.end19:
+// SIMD-ONLY0-NEXT:    [[COND20:%.*]] = phi float [ [[TMP22]], [[COND_TRUE17]] ], [ [[TMP23]], [[COND_FALSE18]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND20]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP24:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP24]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP25:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP26:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP21:%.*]] = fcmp oeq float [[TMP25]], [[TMP26]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP21]], label [[COND_TRUE22:%.*]], label [[COND_FALSE23:%.*]]
+// SIMD-ONLY0:       cond.true22:
+// SIMD-ONLY0-NEXT:    [[TMP27:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END24:%.*]]
+// SIMD-ONLY0:       cond.false23:
+// SIMD-ONLY0-NEXT:    [[TMP28:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END24]]
+// SIMD-ONLY0:       cond.end24:
+// SIMD-ONLY0-NEXT:    [[COND25:%.*]] = phi float [ [[TMP27]], [[COND_TRUE22]] ], [ [[TMP28]], [[COND_FALSE23]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND25]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP29:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP29]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP30:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP30]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP31:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP32:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP26:%.*]] = fcmp ogt float [[TMP31]], [[TMP32]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP26]], label [[COND_TRUE27:%.*]], label [[COND_FALSE28:%.*]]
+// SIMD-ONLY0:       cond.true27:
+// SIMD-ONLY0-NEXT:    [[TMP33:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END29:%.*]]
+// SIMD-ONLY0:       cond.false28:
+// SIMD-ONLY0-NEXT:    [[TMP34:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END29]]
+// SIMD-ONLY0:       cond.end29:
+// SIMD-ONLY0-NEXT:    [[COND30:%.*]] = phi float [ [[TMP33]], [[COND_TRUE27]] ], [ [[TMP34]], [[COND_FALSE28]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND30]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP35:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP35]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP36:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP37:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP31:%.*]] = fcmp olt float [[TMP36]], [[TMP37]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP31]], label [[COND_TRUE32:%.*]], label [[COND_FALSE33:%.*]]
+// SIMD-ONLY0:       cond.true32:
+// SIMD-ONLY0-NEXT:    [[TMP38:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END34:%.*]]
+// SIMD-ONLY0:       cond.false33:
+// SIMD-ONLY0-NEXT:    [[TMP39:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END34]]
+// SIMD-ONLY0:       cond.end34:
+// SIMD-ONLY0-NEXT:    [[COND35:%.*]] = phi float [ [[TMP38]], [[COND_TRUE32]] ], [ [[TMP39]], [[COND_FALSE33]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND35]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP40:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP40]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP41:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP42:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP36:%.*]] = fcmp oeq float [[TMP41]], [[TMP42]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP36]], label [[COND_TRUE37:%.*]], label [[COND_FALSE38:%.*]]
+// SIMD-ONLY0:       cond.true37:
+// SIMD-ONLY0-NEXT:    [[TMP43:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END39:%.*]]
+// SIMD-ONLY0:       cond.false38:
+// SIMD-ONLY0-NEXT:    [[TMP44:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END39]]
+// SIMD-ONLY0:       cond.end39:
+// SIMD-ONLY0-NEXT:    [[COND40:%.*]] = phi float [ [[TMP43]], [[COND_TRUE37]] ], [ [[TMP44]], [[COND_FALSE38]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND40]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP45:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP46:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP41:%.*]] = fcmp ogt float [[TMP45]], [[TMP46]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP41]], label [[COND_TRUE42:%.*]], label [[COND_FALSE43:%.*]]
+// SIMD-ONLY0:       cond.true42:
+// SIMD-ONLY0-NEXT:    [[TMP47:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END44:%.*]]
+// SIMD-ONLY0:       cond.false43:
+// SIMD-ONLY0-NEXT:    [[TMP48:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END44]]
+// SIMD-ONLY0:       cond.end44:
+// SIMD-ONLY0-NEXT:    [[COND45:%.*]] = phi float [ [[TMP47]], [[COND_TRUE42]] ], [ [[TMP48]], [[COND_FALSE43]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND45]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP49:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP49]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP50:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP51:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP46:%.*]] = fcmp olt float [[TMP50]], [[TMP51]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP46]], label [[COND_TRUE47:%.*]], label [[COND_FALSE48:%.*]]
+// SIMD-ONLY0:       cond.true47:
+// SIMD-ONLY0-NEXT:    [[TMP52:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END49:%.*]]
+// SIMD-ONLY0:       cond.false48:
+// SIMD-ONLY0-NEXT:    [[TMP53:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END49]]
+// SIMD-ONLY0:       cond.end49:
+// SIMD-ONLY0-NEXT:    [[COND50:%.*]] = phi float [ [[TMP52]], [[COND_TRUE47]] ], [ [[TMP53]], [[COND_FALSE48]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND50]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP54:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP54]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP55:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP56:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP51:%.*]] = fcmp oeq float [[TMP55]], [[TMP56]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP51]], label [[COND_TRUE52:%.*]], label [[COND_FALSE53:%.*]]
+// SIMD-ONLY0:       cond.true52:
+// SIMD-ONLY0-NEXT:    [[TMP57:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END54:%.*]]
+// SIMD-ONLY0:       cond.false53:
+// SIMD-ONLY0-NEXT:    [[TMP58:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END54]]
+// SIMD-ONLY0:       cond.end54:
+// SIMD-ONLY0-NEXT:    [[COND55:%.*]] = phi float [ [[TMP57]], [[COND_TRUE52]] ], [ [[TMP58]], [[COND_FALSE53]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND55]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP59:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP59]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP60:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP60]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP61:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP62:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP56:%.*]] = fcmp ogt float [[TMP61]], [[TMP62]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP56]], label [[COND_TRUE57:%.*]], label [[COND_FALSE58:%.*]]
+// SIMD-ONLY0:       cond.true57:
+// SIMD-ONLY0-NEXT:    [[TMP63:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END59:%.*]]
+// SIMD-ONLY0:       cond.false58:
+// SIMD-ONLY0-NEXT:    [[TMP64:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END59]]
+// SIMD-ONLY0:       cond.end59:
+// SIMD-ONLY0-NEXT:    [[COND60:%.*]] = phi float [ [[TMP63]], [[COND_TRUE57]] ], [ [[TMP64]], [[COND_FALSE58]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND60]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP65:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP65]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP66:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP67:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP61:%.*]] = fcmp olt float [[TMP66]], [[TMP67]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP61]], label [[COND_TRUE62:%.*]], label [[COND_FALSE63:%.*]]
+// SIMD-ONLY0:       cond.true62:
+// SIMD-ONLY0-NEXT:    [[TMP68:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END64:%.*]]
+// SIMD-ONLY0:       cond.false63:
+// SIMD-ONLY0-NEXT:    [[TMP69:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END64]]
+// SIMD-ONLY0:       cond.end64:
+// SIMD-ONLY0-NEXT:    [[COND65:%.*]] = phi float [ [[TMP68]], [[COND_TRUE62]] ], [ [[TMP69]], [[COND_FALSE63]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND65]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP70:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP70]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP71:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP72:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP66:%.*]] = fcmp oeq float [[TMP71]], [[TMP72]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP66]], label [[COND_TRUE67:%.*]], label [[COND_FALSE68:%.*]]
+// SIMD-ONLY0:       cond.true67:
+// SIMD-ONLY0-NEXT:    [[TMP73:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END69:%.*]]
+// SIMD-ONLY0:       cond.false68:
+// SIMD-ONLY0-NEXT:    [[TMP74:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END69]]
+// SIMD-ONLY0:       cond.end69:
+// SIMD-ONLY0-NEXT:    [[COND70:%.*]] = phi float [ [[TMP73]], [[COND_TRUE67]] ], [ [[TMP74]], [[COND_FALSE68]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND70]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP75:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP76:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP71:%.*]] = fcmp ogt float [[TMP75]], [[TMP76]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP71]], label [[COND_TRUE72:%.*]], label [[COND_FALSE73:%.*]]
+// SIMD-ONLY0:       cond.true72:
+// SIMD-ONLY0-NEXT:    [[TMP77:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END74:%.*]]
+// SIMD-ONLY0:       cond.false73:
+// SIMD-ONLY0-NEXT:    [[TMP78:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END74]]
+// SIMD-ONLY0:       cond.end74:
+// SIMD-ONLY0-NEXT:    [[COND75:%.*]] = phi float [ [[TMP77]], [[COND_TRUE72]] ], [ [[TMP78]], [[COND_FALSE73]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND75]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP79:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP79]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP80:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP81:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP76:%.*]] = fcmp olt float [[TMP80]], [[TMP81]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP76]], label [[COND_TRUE77:%.*]], label [[COND_FALSE78:%.*]]
+// SIMD-ONLY0:       cond.true77:
+// SIMD-ONLY0-NEXT:    [[TMP82:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END79:%.*]]
+// SIMD-ONLY0:       cond.false78:
+// SIMD-ONLY0-NEXT:    [[TMP83:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END79]]
+// SIMD-ONLY0:       cond.end79:
+// SIMD-ONLY0-NEXT:    [[COND80:%.*]] = phi float [ [[TMP82]], [[COND_TRUE77]] ], [ [[TMP83]], [[COND_FALSE78]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND80]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP84:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP84]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP85:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP86:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP81:%.*]] = fcmp oeq float [[TMP85]], [[TMP86]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP81]], label [[COND_TRUE82:%.*]], label [[COND_FALSE83:%.*]]
+// SIMD-ONLY0:       cond.true82:
+// SIMD-ONLY0-NEXT:    [[TMP87:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END84:%.*]]
+// SIMD-ONLY0:       cond.false83:
+// SIMD-ONLY0-NEXT:    [[TMP88:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END84]]
+// SIMD-ONLY0:       cond.end84:
+// SIMD-ONLY0-NEXT:    [[COND85:%.*]] = phi float [ [[TMP87]], [[COND_TRUE82]] ], [ [[TMP88]], [[COND_FALSE83]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND85]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP89:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP89]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP90:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP90]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP91:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP92:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP86:%.*]] = fcmp ogt float [[TMP91]], [[TMP92]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP86]], label [[COND_TRUE87:%.*]], label [[COND_FALSE88:%.*]]
+// SIMD-ONLY0:       cond.true87:
+// SIMD-ONLY0-NEXT:    [[TMP93:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END89:%.*]]
+// SIMD-ONLY0:       cond.false88:
+// SIMD-ONLY0-NEXT:    [[TMP94:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END89]]
+// SIMD-ONLY0:       cond.end89:
+// SIMD-ONLY0-NEXT:    [[COND90:%.*]] = phi float [ [[TMP93]], [[COND_TRUE87]] ], [ [[TMP94]], [[COND_FALSE88]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND90]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP95:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP95]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP96:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP97:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP91:%.*]] = fcmp olt float [[TMP96]], [[TMP97]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP91]], label [[COND_TRUE92:%.*]], label [[COND_FALSE93:%.*]]
+// SIMD-ONLY0:       cond.true92:
+// SIMD-ONLY0-NEXT:    [[TMP98:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END94:%.*]]
+// SIMD-ONLY0:       cond.false93:
+// SIMD-ONLY0-NEXT:    [[TMP99:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END94]]
+// SIMD-ONLY0:       cond.end94:
+// SIMD-ONLY0-NEXT:    [[COND95:%.*]] = phi float [ [[TMP98]], [[COND_TRUE92]] ], [ [[TMP99]], [[COND_FALSE93]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND95]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP100:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP100]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP101:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP102:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP96:%.*]] = fcmp oeq float [[TMP101]], [[TMP102]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP96]], label [[COND_TRUE97:%.*]], label [[COND_FALSE98:%.*]]
+// SIMD-ONLY0:       cond.true97:
+// SIMD-ONLY0-NEXT:    [[TMP103:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END99:%.*]]
+// SIMD-ONLY0:       cond.false98:
+// SIMD-ONLY0-NEXT:    [[TMP104:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END99]]
+// SIMD-ONLY0:       cond.end99:
+// SIMD-ONLY0-NEXT:    [[COND100:%.*]] = phi float [ [[TMP103]], [[COND_TRUE97]] ], [ [[TMP104]], [[COND_FALSE98]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND100]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP105:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP106:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP101:%.*]] = fcmp ogt float [[TMP105]], [[TMP106]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP101]], label [[COND_TRUE102:%.*]], label [[COND_FALSE103:%.*]]
+// SIMD-ONLY0:       cond.true102:
+// SIMD-ONLY0-NEXT:    [[TMP107:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END104:%.*]]
+// SIMD-ONLY0:       cond.false103:
+// SIMD-ONLY0-NEXT:    [[TMP108:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END104]]
+// SIMD-ONLY0:       cond.end104:
+// SIMD-ONLY0-NEXT:    [[COND105:%.*]] = phi float [ [[TMP107]], [[COND_TRUE102]] ], [ [[TMP108]], [[COND_FALSE103]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND105]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP109:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP109]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP110:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP111:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP106:%.*]] = fcmp olt float [[TMP110]], [[TMP111]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP106]], label [[COND_TRUE107:%.*]], label [[COND_FALSE108:%.*]]
+// SIMD-ONLY0:       cond.true107:
+// SIMD-ONLY0-NEXT:    [[TMP112:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END109:%.*]]
+// SIMD-ONLY0:       cond.false108:
+// SIMD-ONLY0-NEXT:    [[TMP113:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END109]]
+// SIMD-ONLY0:       cond.end109:
+// SIMD-ONLY0-NEXT:    [[COND110:%.*]] = phi float [ [[TMP112]], [[COND_TRUE107]] ], [ [[TMP113]], [[COND_FALSE108]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND110]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP114:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP114]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP115:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP116:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP111:%.*]] = fcmp oeq float [[TMP115]], [[TMP116]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP111]], label [[COND_TRUE112:%.*]], label [[COND_FALSE113:%.*]]
+// SIMD-ONLY0:       cond.true112:
+// SIMD-ONLY0-NEXT:    [[TMP117:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END114:%.*]]
+// SIMD-ONLY0:       cond.false113:
+// SIMD-ONLY0-NEXT:    [[TMP118:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END114]]
+// SIMD-ONLY0:       cond.end114:
+// SIMD-ONLY0-NEXT:    [[COND115:%.*]] = phi float [ [[TMP117]], [[COND_TRUE112]] ], [ [[TMP118]], [[COND_FALSE113]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND115]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP119:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP119]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP120:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP120]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP121:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP122:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP116:%.*]] = fcmp ogt float [[TMP121]], [[TMP122]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP116]], label [[COND_TRUE117:%.*]], label [[COND_FALSE118:%.*]]
+// SIMD-ONLY0:       cond.true117:
+// SIMD-ONLY0-NEXT:    [[TMP123:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END119:%.*]]
+// SIMD-ONLY0:       cond.false118:
+// SIMD-ONLY0-NEXT:    [[TMP124:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END119]]
+// SIMD-ONLY0:       cond.end119:
+// SIMD-ONLY0-NEXT:    [[COND120:%.*]] = phi float [ [[TMP123]], [[COND_TRUE117]] ], [ [[TMP124]], [[COND_FALSE118]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND120]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP125:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP125]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP126:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP127:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP121:%.*]] = fcmp olt float [[TMP126]], [[TMP127]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP121]], label [[COND_TRUE122:%.*]], label [[COND_FALSE123:%.*]]
+// SIMD-ONLY0:       cond.true122:
+// SIMD-ONLY0-NEXT:    [[TMP128:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END124:%.*]]
+// SIMD-ONLY0:       cond.false123:
+// SIMD-ONLY0-NEXT:    [[TMP129:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END124]]
+// SIMD-ONLY0:       cond.end124:
+// SIMD-ONLY0-NEXT:    [[COND125:%.*]] = phi float [ [[TMP128]], [[COND_TRUE122]] ], [ [[TMP129]], [[COND_FALSE123]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND125]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP130:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP130]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP131:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP132:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP126:%.*]] = fcmp oeq float [[TMP131]], [[TMP132]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP126]], label [[COND_TRUE127:%.*]], label [[COND_FALSE128:%.*]]
+// SIMD-ONLY0:       cond.true127:
+// SIMD-ONLY0-NEXT:    [[TMP133:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END129:%.*]]
+// SIMD-ONLY0:       cond.false128:
+// SIMD-ONLY0-NEXT:    [[TMP134:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END129]]
+// SIMD-ONLY0:       cond.end129:
+// SIMD-ONLY0-NEXT:    [[COND130:%.*]] = phi float [ [[TMP133]], [[COND_TRUE127]] ], [ [[TMP134]], [[COND_FALSE128]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND130]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP135:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP136:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP131:%.*]] = fcmp ogt float [[TMP135]], [[TMP136]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP131]], label [[COND_TRUE132:%.*]], label [[COND_FALSE133:%.*]]
+// SIMD-ONLY0:       cond.true132:
+// SIMD-ONLY0-NEXT:    [[TMP137:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END134:%.*]]
+// SIMD-ONLY0:       cond.false133:
+// SIMD-ONLY0-NEXT:    [[TMP138:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END134]]
+// SIMD-ONLY0:       cond.end134:
+// SIMD-ONLY0-NEXT:    [[COND135:%.*]] = phi float [ [[TMP137]], [[COND_TRUE132]] ], [ [[TMP138]], [[COND_FALSE133]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND135]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP139:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP139]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP140:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP141:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP136:%.*]] = fcmp olt float [[TMP140]], [[TMP141]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP136]], label [[COND_TRUE137:%.*]], label [[COND_FALSE138:%.*]]
+// SIMD-ONLY0:       cond.true137:
+// SIMD-ONLY0-NEXT:    [[TMP142:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END139:%.*]]
+// SIMD-ONLY0:       cond.false138:
+// SIMD-ONLY0-NEXT:    [[TMP143:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END139]]
+// SIMD-ONLY0:       cond.end139:
+// SIMD-ONLY0-NEXT:    [[COND140:%.*]] = phi float [ [[TMP142]], [[COND_TRUE137]] ], [ [[TMP143]], [[COND_FALSE138]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND140]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP144:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP144]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP145:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP146:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP141:%.*]] = fcmp oeq float [[TMP145]], [[TMP146]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP141]], label [[COND_TRUE142:%.*]], label [[COND_FALSE143:%.*]]
+// SIMD-ONLY0:       cond.true142:
+// SIMD-ONLY0-NEXT:    [[TMP147:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END144:%.*]]
+// SIMD-ONLY0:       cond.false143:
+// SIMD-ONLY0-NEXT:    [[TMP148:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END144]]
+// SIMD-ONLY0:       cond.end144:
+// SIMD-ONLY0-NEXT:    [[COND145:%.*]] = phi float [ [[TMP147]], [[COND_TRUE142]] ], [ [[TMP148]], [[COND_FALSE143]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND145]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP149:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP149]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP150:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP150]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP151:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP152:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP146:%.*]] = fcmp ogt float [[TMP151]], [[TMP152]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP146]], label [[COND_TRUE147:%.*]], label [[COND_FALSE148:%.*]]
+// SIMD-ONLY0:       cond.true147:
+// SIMD-ONLY0-NEXT:    [[TMP153:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END149:%.*]]
+// SIMD-ONLY0:       cond.false148:
+// SIMD-ONLY0-NEXT:    [[TMP154:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END149]]
+// SIMD-ONLY0:       cond.end149:
+// SIMD-ONLY0-NEXT:    [[COND150:%.*]] = phi float [ [[TMP153]], [[COND_TRUE147]] ], [ [[TMP154]], [[COND_FALSE148]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND150]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP155:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP155]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP156:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP157:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP151:%.*]] = fcmp olt float [[TMP156]], [[TMP157]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP151]], label [[COND_TRUE152:%.*]], label [[COND_FALSE153:%.*]]
+// SIMD-ONLY0:       cond.true152:
+// SIMD-ONLY0-NEXT:    [[TMP158:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END154:%.*]]
+// SIMD-ONLY0:       cond.false153:
+// SIMD-ONLY0-NEXT:    [[TMP159:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END154]]
+// SIMD-ONLY0:       cond.end154:
+// SIMD-ONLY0-NEXT:    [[COND155:%.*]] = phi float [ [[TMP158]], [[COND_TRUE152]] ], [ [[TMP159]], [[COND_FALSE153]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND155]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP160:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP160]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP161:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP162:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP156:%.*]] = fcmp oeq float [[TMP161]], [[TMP162]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP156]], label [[COND_TRUE157:%.*]], label [[COND_FALSE158:%.*]]
+// SIMD-ONLY0:       cond.true157:
+// SIMD-ONLY0-NEXT:    [[TMP163:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END159:%.*]]
+// SIMD-ONLY0:       cond.false158:
+// SIMD-ONLY0-NEXT:    [[TMP164:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END159]]
+// SIMD-ONLY0:       cond.end159:
+// SIMD-ONLY0-NEXT:    [[COND160:%.*]] = phi float [ [[TMP163]], [[COND_TRUE157]] ], [ [[TMP164]], [[COND_FALSE158]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND160]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP165:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP166:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP161:%.*]] = fcmp ogt float [[TMP165]], [[TMP166]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP161]], label [[COND_TRUE162:%.*]], label [[COND_FALSE163:%.*]]
+// SIMD-ONLY0:       cond.true162:
+// SIMD-ONLY0-NEXT:    [[TMP167:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END164:%.*]]
+// SIMD-ONLY0:       cond.false163:
+// SIMD-ONLY0-NEXT:    [[TMP168:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END164]]
+// SIMD-ONLY0:       cond.end164:
+// SIMD-ONLY0-NEXT:    [[COND165:%.*]] = phi float [ [[TMP167]], [[COND_TRUE162]] ], [ [[TMP168]], [[COND_FALSE163]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND165]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP169:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP169]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP170:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP171:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP166:%.*]] = fcmp olt float [[TMP170]], [[TMP171]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP166]], label [[COND_TRUE167:%.*]], label [[COND_FALSE168:%.*]]
+// SIMD-ONLY0:       cond.true167:
+// SIMD-ONLY0-NEXT:    [[TMP172:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END169:%.*]]
+// SIMD-ONLY0:       cond.false168:
+// SIMD-ONLY0-NEXT:    [[TMP173:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END169]]
+// SIMD-ONLY0:       cond.end169:
+// SIMD-ONLY0-NEXT:    [[COND170:%.*]] = phi float [ [[TMP172]], [[COND_TRUE167]] ], [ [[TMP173]], [[COND_FALSE168]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND170]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP174:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP174]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP175:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP176:%.*]] = load float, ptr [[FE]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP171:%.*]] = fcmp oeq float [[TMP175]], [[TMP176]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP171]], label [[COND_TRUE172:%.*]], label [[COND_FALSE173:%.*]]
+// SIMD-ONLY0:       cond.true172:
+// SIMD-ONLY0-NEXT:    [[TMP177:%.*]] = load float, ptr [[FD]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END174:%.*]]
+// SIMD-ONLY0:       cond.false173:
+// SIMD-ONLY0-NEXT:    [[TMP178:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    br label [[COND_END174]]
+// SIMD-ONLY0:       cond.end174:
+// SIMD-ONLY0-NEXT:    [[COND175:%.*]] = phi float [ [[TMP177]], [[COND_TRUE172]] ], [ [[TMP178]], [[COND_FALSE173]] ]
+// SIMD-ONLY0-NEXT:    store float [[COND175]], ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP179:%.*]] = load float, ptr [[FX]], align 4
+// SIMD-ONLY0-NEXT:    store float [[TMP179]], ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP180:%.*]] = load float, ptr [[FV]], align 4
+// SIMD-ONLY0-NEXT:    ret float [[TMP180]]
+//
+//
+// SIMD-ONLY0-LABEL: @dxevd(
+// SIMD-ONLY0-NEXT:  entry:
+// SIMD-ONLY0-NEXT:    [[DX:%.*]] = alloca double, align 8
+// SIMD-ONLY0-NEXT:    [[DV:%.*]] = alloca double, align 8
+// SIMD-ONLY0-NEXT:    [[DE:%.*]] = alloca double, align 8
+// SIMD-ONLY0-NEXT:    [[DD:%.*]] = alloca double, align 8
+// SIMD-ONLY0-NEXT:    [[TMP0:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP0]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP1:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP2:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP:%.*]] = fcmp ogt double [[TMP1]], [[TMP2]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]]
+// SIMD-ONLY0:       cond.true:
+// SIMD-ONLY0-NEXT:    [[TMP3:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END:%.*]]
+// SIMD-ONLY0:       cond.false:
+// SIMD-ONLY0-NEXT:    [[TMP4:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END]]
+// SIMD-ONLY0:       cond.end:
+// SIMD-ONLY0-NEXT:    [[COND:%.*]] = phi double [ [[TMP3]], [[COND_TRUE]] ], [ [[TMP4]], [[COND_FALSE]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP5:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP5]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP6:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP7:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP1:%.*]] = fcmp olt double [[TMP6]], [[TMP7]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP1]], label [[COND_TRUE2:%.*]], label [[COND_FALSE3:%.*]]
+// SIMD-ONLY0:       cond.true2:
+// SIMD-ONLY0-NEXT:    [[TMP8:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4:%.*]]
+// SIMD-ONLY0:       cond.false3:
+// SIMD-ONLY0-NEXT:    [[TMP9:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END4]]
+// SIMD-ONLY0:       cond.end4:
+// SIMD-ONLY0-NEXT:    [[COND5:%.*]] = phi double [ [[TMP8]], [[COND_TRUE2]] ], [ [[TMP9]], [[COND_FALSE3]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND5]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP10:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP10]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP11:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP12:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP6:%.*]] = fcmp oeq double [[TMP11]], [[TMP12]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP6]], label [[COND_TRUE7:%.*]], label [[COND_FALSE8:%.*]]
+// SIMD-ONLY0:       cond.true7:
+// SIMD-ONLY0-NEXT:    [[TMP13:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END9:%.*]]
+// SIMD-ONLY0:       cond.false8:
+// SIMD-ONLY0-NEXT:    [[TMP14:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END9]]
+// SIMD-ONLY0:       cond.end9:
+// SIMD-ONLY0-NEXT:    [[COND10:%.*]] = phi double [ [[TMP13]], [[COND_TRUE7]] ], [ [[TMP14]], [[COND_FALSE8]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND10]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP15:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP16:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP11:%.*]] = fcmp ogt double [[TMP15]], [[TMP16]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP11]], label [[COND_TRUE12:%.*]], label [[COND_FALSE13:%.*]]
+// SIMD-ONLY0:       cond.true12:
+// SIMD-ONLY0-NEXT:    [[TMP17:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END14:%.*]]
+// SIMD-ONLY0:       cond.false13:
+// SIMD-ONLY0-NEXT:    [[TMP18:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END14]]
+// SIMD-ONLY0:       cond.end14:
+// SIMD-ONLY0-NEXT:    [[COND15:%.*]] = phi double [ [[TMP17]], [[COND_TRUE12]] ], [ [[TMP18]], [[COND_FALSE13]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND15]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP19:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP19]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP20:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP21:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP16:%.*]] = fcmp olt double [[TMP20]], [[TMP21]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP16]], label [[COND_TRUE17:%.*]], label [[COND_FALSE18:%.*]]
+// SIMD-ONLY0:       cond.true17:
+// SIMD-ONLY0-NEXT:    [[TMP22:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END19:%.*]]
+// SIMD-ONLY0:       cond.false18:
+// SIMD-ONLY0-NEXT:    [[TMP23:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END19]]
+// SIMD-ONLY0:       cond.end19:
+// SIMD-ONLY0-NEXT:    [[COND20:%.*]] = phi double [ [[TMP22]], [[COND_TRUE17]] ], [ [[TMP23]], [[COND_FALSE18]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND20]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP24:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP24]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP25:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP26:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP21:%.*]] = fcmp oeq double [[TMP25]], [[TMP26]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP21]], label [[COND_TRUE22:%.*]], label [[COND_FALSE23:%.*]]
+// SIMD-ONLY0:       cond.true22:
+// SIMD-ONLY0-NEXT:    [[TMP27:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END24:%.*]]
+// SIMD-ONLY0:       cond.false23:
+// SIMD-ONLY0-NEXT:    [[TMP28:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END24]]
+// SIMD-ONLY0:       cond.end24:
+// SIMD-ONLY0-NEXT:    [[COND25:%.*]] = phi double [ [[TMP27]], [[COND_TRUE22]] ], [ [[TMP28]], [[COND_FALSE23]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND25]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP29:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP29]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP30:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP30]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP31:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP32:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP26:%.*]] = fcmp ogt double [[TMP31]], [[TMP32]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP26]], label [[COND_TRUE27:%.*]], label [[COND_FALSE28:%.*]]
+// SIMD-ONLY0:       cond.true27:
+// SIMD-ONLY0-NEXT:    [[TMP33:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END29:%.*]]
+// SIMD-ONLY0:       cond.false28:
+// SIMD-ONLY0-NEXT:    [[TMP34:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END29]]
+// SIMD-ONLY0:       cond.end29:
+// SIMD-ONLY0-NEXT:    [[COND30:%.*]] = phi double [ [[TMP33]], [[COND_TRUE27]] ], [ [[TMP34]], [[COND_FALSE28]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND30]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP35:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP35]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP36:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP37:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP31:%.*]] = fcmp olt double [[TMP36]], [[TMP37]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP31]], label [[COND_TRUE32:%.*]], label [[COND_FALSE33:%.*]]
+// SIMD-ONLY0:       cond.true32:
+// SIMD-ONLY0-NEXT:    [[TMP38:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END34:%.*]]
+// SIMD-ONLY0:       cond.false33:
+// SIMD-ONLY0-NEXT:    [[TMP39:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END34]]
+// SIMD-ONLY0:       cond.end34:
+// SIMD-ONLY0-NEXT:    [[COND35:%.*]] = phi double [ [[TMP38]], [[COND_TRUE32]] ], [ [[TMP39]], [[COND_FALSE33]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND35]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP40:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP40]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP41:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP42:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP36:%.*]] = fcmp oeq double [[TMP41]], [[TMP42]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP36]], label [[COND_TRUE37:%.*]], label [[COND_FALSE38:%.*]]
+// SIMD-ONLY0:       cond.true37:
+// SIMD-ONLY0-NEXT:    [[TMP43:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END39:%.*]]
+// SIMD-ONLY0:       cond.false38:
+// SIMD-ONLY0-NEXT:    [[TMP44:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END39]]
+// SIMD-ONLY0:       cond.end39:
+// SIMD-ONLY0-NEXT:    [[COND40:%.*]] = phi double [ [[TMP43]], [[COND_TRUE37]] ], [ [[TMP44]], [[COND_FALSE38]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND40]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP45:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP46:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP41:%.*]] = fcmp ogt double [[TMP45]], [[TMP46]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP41]], label [[COND_TRUE42:%.*]], label [[COND_FALSE43:%.*]]
+// SIMD-ONLY0:       cond.true42:
+// SIMD-ONLY0-NEXT:    [[TMP47:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END44:%.*]]
+// SIMD-ONLY0:       cond.false43:
+// SIMD-ONLY0-NEXT:    [[TMP48:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END44]]
+// SIMD-ONLY0:       cond.end44:
+// SIMD-ONLY0-NEXT:    [[COND45:%.*]] = phi double [ [[TMP47]], [[COND_TRUE42]] ], [ [[TMP48]], [[COND_FALSE43]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND45]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP49:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP49]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP50:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP51:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP46:%.*]] = fcmp olt double [[TMP50]], [[TMP51]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP46]], label [[COND_TRUE47:%.*]], label [[COND_FALSE48:%.*]]
+// SIMD-ONLY0:       cond.true47:
+// SIMD-ONLY0-NEXT:    [[TMP52:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END49:%.*]]
+// SIMD-ONLY0:       cond.false48:
+// SIMD-ONLY0-NEXT:    [[TMP53:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END49]]
+// SIMD-ONLY0:       cond.end49:
+// SIMD-ONLY0-NEXT:    [[COND50:%.*]] = phi double [ [[TMP52]], [[COND_TRUE47]] ], [ [[TMP53]], [[COND_FALSE48]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND50]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP54:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP54]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP55:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP56:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP51:%.*]] = fcmp oeq double [[TMP55]], [[TMP56]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP51]], label [[COND_TRUE52:%.*]], label [[COND_FALSE53:%.*]]
+// SIMD-ONLY0:       cond.true52:
+// SIMD-ONLY0-NEXT:    [[TMP57:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END54:%.*]]
+// SIMD-ONLY0:       cond.false53:
+// SIMD-ONLY0-NEXT:    [[TMP58:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END54]]
+// SIMD-ONLY0:       cond.end54:
+// SIMD-ONLY0-NEXT:    [[COND55:%.*]] = phi double [ [[TMP57]], [[COND_TRUE52]] ], [ [[TMP58]], [[COND_FALSE53]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND55]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP59:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP59]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP60:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP60]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP61:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP62:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP56:%.*]] = fcmp ogt double [[TMP61]], [[TMP62]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP56]], label [[COND_TRUE57:%.*]], label [[COND_FALSE58:%.*]]
+// SIMD-ONLY0:       cond.true57:
+// SIMD-ONLY0-NEXT:    [[TMP63:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END59:%.*]]
+// SIMD-ONLY0:       cond.false58:
+// SIMD-ONLY0-NEXT:    [[TMP64:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END59]]
+// SIMD-ONLY0:       cond.end59:
+// SIMD-ONLY0-NEXT:    [[COND60:%.*]] = phi double [ [[TMP63]], [[COND_TRUE57]] ], [ [[TMP64]], [[COND_FALSE58]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND60]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP65:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP65]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP66:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP67:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP61:%.*]] = fcmp olt double [[TMP66]], [[TMP67]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP61]], label [[COND_TRUE62:%.*]], label [[COND_FALSE63:%.*]]
+// SIMD-ONLY0:       cond.true62:
+// SIMD-ONLY0-NEXT:    [[TMP68:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END64:%.*]]
+// SIMD-ONLY0:       cond.false63:
+// SIMD-ONLY0-NEXT:    [[TMP69:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END64]]
+// SIMD-ONLY0:       cond.end64:
+// SIMD-ONLY0-NEXT:    [[COND65:%.*]] = phi double [ [[TMP68]], [[COND_TRUE62]] ], [ [[TMP69]], [[COND_FALSE63]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND65]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP70:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP70]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP71:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP72:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP66:%.*]] = fcmp oeq double [[TMP71]], [[TMP72]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP66]], label [[COND_TRUE67:%.*]], label [[COND_FALSE68:%.*]]
+// SIMD-ONLY0:       cond.true67:
+// SIMD-ONLY0-NEXT:    [[TMP73:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END69:%.*]]
+// SIMD-ONLY0:       cond.false68:
+// SIMD-ONLY0-NEXT:    [[TMP74:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END69]]
+// SIMD-ONLY0:       cond.end69:
+// SIMD-ONLY0-NEXT:    [[COND70:%.*]] = phi double [ [[TMP73]], [[COND_TRUE67]] ], [ [[TMP74]], [[COND_FALSE68]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND70]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP75:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP76:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP71:%.*]] = fcmp ogt double [[TMP75]], [[TMP76]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP71]], label [[COND_TRUE72:%.*]], label [[COND_FALSE73:%.*]]
+// SIMD-ONLY0:       cond.true72:
+// SIMD-ONLY0-NEXT:    [[TMP77:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END74:%.*]]
+// SIMD-ONLY0:       cond.false73:
+// SIMD-ONLY0-NEXT:    [[TMP78:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END74]]
+// SIMD-ONLY0:       cond.end74:
+// SIMD-ONLY0-NEXT:    [[COND75:%.*]] = phi double [ [[TMP77]], [[COND_TRUE72]] ], [ [[TMP78]], [[COND_FALSE73]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND75]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP79:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP79]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP80:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP81:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP76:%.*]] = fcmp olt double [[TMP80]], [[TMP81]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP76]], label [[COND_TRUE77:%.*]], label [[COND_FALSE78:%.*]]
+// SIMD-ONLY0:       cond.true77:
+// SIMD-ONLY0-NEXT:    [[TMP82:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END79:%.*]]
+// SIMD-ONLY0:       cond.false78:
+// SIMD-ONLY0-NEXT:    [[TMP83:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END79]]
+// SIMD-ONLY0:       cond.end79:
+// SIMD-ONLY0-NEXT:    [[COND80:%.*]] = phi double [ [[TMP82]], [[COND_TRUE77]] ], [ [[TMP83]], [[COND_FALSE78]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND80]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP84:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP84]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP85:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP86:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP81:%.*]] = fcmp oeq double [[TMP85]], [[TMP86]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP81]], label [[COND_TRUE82:%.*]], label [[COND_FALSE83:%.*]]
+// SIMD-ONLY0:       cond.true82:
+// SIMD-ONLY0-NEXT:    [[TMP87:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END84:%.*]]
+// SIMD-ONLY0:       cond.false83:
+// SIMD-ONLY0-NEXT:    [[TMP88:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END84]]
+// SIMD-ONLY0:       cond.end84:
+// SIMD-ONLY0-NEXT:    [[COND85:%.*]] = phi double [ [[TMP87]], [[COND_TRUE82]] ], [ [[TMP88]], [[COND_FALSE83]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND85]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP89:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP89]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP90:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP90]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP91:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP92:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP86:%.*]] = fcmp ogt double [[TMP91]], [[TMP92]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP86]], label [[COND_TRUE87:%.*]], label [[COND_FALSE88:%.*]]
+// SIMD-ONLY0:       cond.true87:
+// SIMD-ONLY0-NEXT:    [[TMP93:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END89:%.*]]
+// SIMD-ONLY0:       cond.false88:
+// SIMD-ONLY0-NEXT:    [[TMP94:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END89]]
+// SIMD-ONLY0:       cond.end89:
+// SIMD-ONLY0-NEXT:    [[COND90:%.*]] = phi double [ [[TMP93]], [[COND_TRUE87]] ], [ [[TMP94]], [[COND_FALSE88]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND90]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP95:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP95]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP96:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP97:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP91:%.*]] = fcmp olt double [[TMP96]], [[TMP97]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP91]], label [[COND_TRUE92:%.*]], label [[COND_FALSE93:%.*]]
+// SIMD-ONLY0:       cond.true92:
+// SIMD-ONLY0-NEXT:    [[TMP98:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END94:%.*]]
+// SIMD-ONLY0:       cond.false93:
+// SIMD-ONLY0-NEXT:    [[TMP99:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END94]]
+// SIMD-ONLY0:       cond.end94:
+// SIMD-ONLY0-NEXT:    [[COND95:%.*]] = phi double [ [[TMP98]], [[COND_TRUE92]] ], [ [[TMP99]], [[COND_FALSE93]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND95]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP100:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP100]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP101:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP102:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP96:%.*]] = fcmp oeq double [[TMP101]], [[TMP102]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP96]], label [[COND_TRUE97:%.*]], label [[COND_FALSE98:%.*]]
+// SIMD-ONLY0:       cond.true97:
+// SIMD-ONLY0-NEXT:    [[TMP103:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END99:%.*]]
+// SIMD-ONLY0:       cond.false98:
+// SIMD-ONLY0-NEXT:    [[TMP104:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END99]]
+// SIMD-ONLY0:       cond.end99:
+// SIMD-ONLY0-NEXT:    [[COND100:%.*]] = phi double [ [[TMP103]], [[COND_TRUE97]] ], [ [[TMP104]], [[COND_FALSE98]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND100]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP105:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP106:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP101:%.*]] = fcmp ogt double [[TMP105]], [[TMP106]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP101]], label [[COND_TRUE102:%.*]], label [[COND_FALSE103:%.*]]
+// SIMD-ONLY0:       cond.true102:
+// SIMD-ONLY0-NEXT:    [[TMP107:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END104:%.*]]
+// SIMD-ONLY0:       cond.false103:
+// SIMD-ONLY0-NEXT:    [[TMP108:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END104]]
+// SIMD-ONLY0:       cond.end104:
+// SIMD-ONLY0-NEXT:    [[COND105:%.*]] = phi double [ [[TMP107]], [[COND_TRUE102]] ], [ [[TMP108]], [[COND_FALSE103]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND105]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP109:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP109]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP110:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP111:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP106:%.*]] = fcmp olt double [[TMP110]], [[TMP111]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP106]], label [[COND_TRUE107:%.*]], label [[COND_FALSE108:%.*]]
+// SIMD-ONLY0:       cond.true107:
+// SIMD-ONLY0-NEXT:    [[TMP112:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END109:%.*]]
+// SIMD-ONLY0:       cond.false108:
+// SIMD-ONLY0-NEXT:    [[TMP113:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END109]]
+// SIMD-ONLY0:       cond.end109:
+// SIMD-ONLY0-NEXT:    [[COND110:%.*]] = phi double [ [[TMP112]], [[COND_TRUE107]] ], [ [[TMP113]], [[COND_FALSE108]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND110]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP114:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP114]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP115:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP116:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP111:%.*]] = fcmp oeq double [[TMP115]], [[TMP116]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP111]], label [[COND_TRUE112:%.*]], label [[COND_FALSE113:%.*]]
+// SIMD-ONLY0:       cond.true112:
+// SIMD-ONLY0-NEXT:    [[TMP117:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END114:%.*]]
+// SIMD-ONLY0:       cond.false113:
+// SIMD-ONLY0-NEXT:    [[TMP118:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END114]]
+// SIMD-ONLY0:       cond.end114:
+// SIMD-ONLY0-NEXT:    [[COND115:%.*]] = phi double [ [[TMP117]], [[COND_TRUE112]] ], [ [[TMP118]], [[COND_FALSE113]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND115]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP119:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP119]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP120:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP120]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP121:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP122:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP116:%.*]] = fcmp ogt double [[TMP121]], [[TMP122]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP116]], label [[COND_TRUE117:%.*]], label [[COND_FALSE118:%.*]]
+// SIMD-ONLY0:       cond.true117:
+// SIMD-ONLY0-NEXT:    [[TMP123:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END119:%.*]]
+// SIMD-ONLY0:       cond.false118:
+// SIMD-ONLY0-NEXT:    [[TMP124:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END119]]
+// SIMD-ONLY0:       cond.end119:
+// SIMD-ONLY0-NEXT:    [[COND120:%.*]] = phi double [ [[TMP123]], [[COND_TRUE117]] ], [ [[TMP124]], [[COND_FALSE118]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND120]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP125:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP125]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP126:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP127:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP121:%.*]] = fcmp olt double [[TMP126]], [[TMP127]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP121]], label [[COND_TRUE122:%.*]], label [[COND_FALSE123:%.*]]
+// SIMD-ONLY0:       cond.true122:
+// SIMD-ONLY0-NEXT:    [[TMP128:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END124:%.*]]
+// SIMD-ONLY0:       cond.false123:
+// SIMD-ONLY0-NEXT:    [[TMP129:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END124]]
+// SIMD-ONLY0:       cond.end124:
+// SIMD-ONLY0-NEXT:    [[COND125:%.*]] = phi double [ [[TMP128]], [[COND_TRUE122]] ], [ [[TMP129]], [[COND_FALSE123]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND125]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP130:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP130]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP131:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP132:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP126:%.*]] = fcmp oeq double [[TMP131]], [[TMP132]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP126]], label [[COND_TRUE127:%.*]], label [[COND_FALSE128:%.*]]
+// SIMD-ONLY0:       cond.true127:
+// SIMD-ONLY0-NEXT:    [[TMP133:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END129:%.*]]
+// SIMD-ONLY0:       cond.false128:
+// SIMD-ONLY0-NEXT:    [[TMP134:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END129]]
+// SIMD-ONLY0:       cond.end129:
+// SIMD-ONLY0-NEXT:    [[COND130:%.*]] = phi double [ [[TMP133]], [[COND_TRUE127]] ], [ [[TMP134]], [[COND_FALSE128]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND130]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP135:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP136:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP131:%.*]] = fcmp ogt double [[TMP135]], [[TMP136]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP131]], label [[COND_TRUE132:%.*]], label [[COND_FALSE133:%.*]]
+// SIMD-ONLY0:       cond.true132:
+// SIMD-ONLY0-NEXT:    [[TMP137:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END134:%.*]]
+// SIMD-ONLY0:       cond.false133:
+// SIMD-ONLY0-NEXT:    [[TMP138:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END134]]
+// SIMD-ONLY0:       cond.end134:
+// SIMD-ONLY0-NEXT:    [[COND135:%.*]] = phi double [ [[TMP137]], [[COND_TRUE132]] ], [ [[TMP138]], [[COND_FALSE133]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND135]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP139:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP139]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP140:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP141:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP136:%.*]] = fcmp olt double [[TMP140]], [[TMP141]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP136]], label [[COND_TRUE137:%.*]], label [[COND_FALSE138:%.*]]
+// SIMD-ONLY0:       cond.true137:
+// SIMD-ONLY0-NEXT:    [[TMP142:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END139:%.*]]
+// SIMD-ONLY0:       cond.false138:
+// SIMD-ONLY0-NEXT:    [[TMP143:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END139]]
+// SIMD-ONLY0:       cond.end139:
+// SIMD-ONLY0-NEXT:    [[COND140:%.*]] = phi double [ [[TMP142]], [[COND_TRUE137]] ], [ [[TMP143]], [[COND_FALSE138]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND140]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP144:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP144]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP145:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP146:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP141:%.*]] = fcmp oeq double [[TMP145]], [[TMP146]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP141]], label [[COND_TRUE142:%.*]], label [[COND_FALSE143:%.*]]
+// SIMD-ONLY0:       cond.true142:
+// SIMD-ONLY0-NEXT:    [[TMP147:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END144:%.*]]
+// SIMD-ONLY0:       cond.false143:
+// SIMD-ONLY0-NEXT:    [[TMP148:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END144]]
+// SIMD-ONLY0:       cond.end144:
+// SIMD-ONLY0-NEXT:    [[COND145:%.*]] = phi double [ [[TMP147]], [[COND_TRUE142]] ], [ [[TMP148]], [[COND_FALSE143]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND145]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP149:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP149]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP150:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP150]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP151:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP152:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP146:%.*]] = fcmp ogt double [[TMP151]], [[TMP152]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP146]], label [[COND_TRUE147:%.*]], label [[COND_FALSE148:%.*]]
+// SIMD-ONLY0:       cond.true147:
+// SIMD-ONLY0-NEXT:    [[TMP153:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END149:%.*]]
+// SIMD-ONLY0:       cond.false148:
+// SIMD-ONLY0-NEXT:    [[TMP154:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END149]]
+// SIMD-ONLY0:       cond.end149:
+// SIMD-ONLY0-NEXT:    [[COND150:%.*]] = phi double [ [[TMP153]], [[COND_TRUE147]] ], [ [[TMP154]], [[COND_FALSE148]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND150]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP155:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP155]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP156:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP157:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP151:%.*]] = fcmp olt double [[TMP156]], [[TMP157]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP151]], label [[COND_TRUE152:%.*]], label [[COND_FALSE153:%.*]]
+// SIMD-ONLY0:       cond.true152:
+// SIMD-ONLY0-NEXT:    [[TMP158:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END154:%.*]]
+// SIMD-ONLY0:       cond.false153:
+// SIMD-ONLY0-NEXT:    [[TMP159:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END154]]
+// SIMD-ONLY0:       cond.end154:
+// SIMD-ONLY0-NEXT:    [[COND155:%.*]] = phi double [ [[TMP158]], [[COND_TRUE152]] ], [ [[TMP159]], [[COND_FALSE153]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND155]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP160:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP160]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP161:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP162:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP156:%.*]] = fcmp oeq double [[TMP161]], [[TMP162]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP156]], label [[COND_TRUE157:%.*]], label [[COND_FALSE158:%.*]]
+// SIMD-ONLY0:       cond.true157:
+// SIMD-ONLY0-NEXT:    [[TMP163:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END159:%.*]]
+// SIMD-ONLY0:       cond.false158:
+// SIMD-ONLY0-NEXT:    [[TMP164:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END159]]
+// SIMD-ONLY0:       cond.end159:
+// SIMD-ONLY0-NEXT:    [[COND160:%.*]] = phi double [ [[TMP163]], [[COND_TRUE157]] ], [ [[TMP164]], [[COND_FALSE158]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND160]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP165:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP166:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP161:%.*]] = fcmp ogt double [[TMP165]], [[TMP166]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP161]], label [[COND_TRUE162:%.*]], label [[COND_FALSE163:%.*]]
+// SIMD-ONLY0:       cond.true162:
+// SIMD-ONLY0-NEXT:    [[TMP167:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END164:%.*]]
+// SIMD-ONLY0:       cond.false163:
+// SIMD-ONLY0-NEXT:    [[TMP168:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END164]]
+// SIMD-ONLY0:       cond.end164:
+// SIMD-ONLY0-NEXT:    [[COND165:%.*]] = phi double [ [[TMP167]], [[COND_TRUE162]] ], [ [[TMP168]], [[COND_FALSE163]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND165]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP169:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP169]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP170:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP171:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP166:%.*]] = fcmp olt double [[TMP170]], [[TMP171]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP166]], label [[COND_TRUE167:%.*]], label [[COND_FALSE168:%.*]]
+// SIMD-ONLY0:       cond.true167:
+// SIMD-ONLY0-NEXT:    [[TMP172:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END169:%.*]]
+// SIMD-ONLY0:       cond.false168:
+// SIMD-ONLY0-NEXT:    [[TMP173:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END169]]
+// SIMD-ONLY0:       cond.end169:
+// SIMD-ONLY0-NEXT:    [[COND170:%.*]] = phi double [ [[TMP172]], [[COND_TRUE167]] ], [ [[TMP173]], [[COND_FALSE168]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND170]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP174:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP174]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP175:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP176:%.*]] = load double, ptr [[DE]], align 8
+// SIMD-ONLY0-NEXT:    [[CMP171:%.*]] = fcmp oeq double [[TMP175]], [[TMP176]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP171]], label [[COND_TRUE172:%.*]], label [[COND_FALSE173:%.*]]
+// SIMD-ONLY0:       cond.true172:
+// SIMD-ONLY0-NEXT:    [[TMP177:%.*]] = load double, ptr [[DD]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END174:%.*]]
+// SIMD-ONLY0:       cond.false173:
+// SIMD-ONLY0-NEXT:    [[TMP178:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    br label [[COND_END174]]
+// SIMD-ONLY0:       cond.end174:
+// SIMD-ONLY0-NEXT:    [[COND175:%.*]] = phi double [ [[TMP177]], [[COND_TRUE172]] ], [ [[TMP178]], [[COND_FALSE173]] ]
+// SIMD-ONLY0-NEXT:    store double [[COND175]], ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP179:%.*]] = load double, ptr [[DX]], align 8
+// SIMD-ONLY0-NEXT:    store double [[TMP179]], ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    [[TMP180:%.*]] = load double, ptr [[DV]], align 8
+// SIMD-ONLY0-NEXT:    ret double [[TMP180]]
+//
diff --git a/clang/test/OpenMP/bug60602.cpp b/clang/test/OpenMP/bug60602.cpp
new file mode 100644
index 0000000000000..bc5a2f5953142
--- /dev/null
+++ b/clang/test/OpenMP/bug60602.cpp
@@ -0,0 +1,578 @@
+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --include-generated-funcs --replace-value-regex "__omp_offloading_[0-9a-z]+_[0-9a-z]+" "reduction_size[.].+[.]" "pl_cond[.].+[.|,]" --prefix-filecheck-ir-name _ --version 2
+// Test host codegen.
+// RUN: %clang_cc1 -verify -fopenmp -x c++ -triple powerpc64le-unknown-unknown -fopenmp-targets=powerpc64le-ibm-linux-gnu -emit-llvm %s -o - | FileCheck %s
+// expected-no-diagnostics
+
+int kernel_within_loop(int *a, int *b, int N, int num_iters) {
+  int i;
+  for (i = 0; i < num_iters; ++i) {
+#pragma omp target parallel for map(a[0:N]) map(b[0:N])
+    for (int j = 0; j< N; j++)
+      a[j] = b[j];
+
+#pragma omp target teams distribute parallel for map(a[0:N]) map(b[0:N])
+    for (int j = 0; j< N; j+=3)
+      a[j] = b[j] * 2;
+  }
+  return a[N-1];
+}
+// CHECK-LABEL: define dso_local noundef signext i32 @_Z18kernel_within_loopPiS_ii
+// CHECK-SAME: (ptr noundef [[A:%.*]], ptr noundef [[B:%.*]], i32 noundef signext [[N:%.*]], i32 noundef signext [[NUM_ITERS:%.*]]) #[[ATTR0:[0-9]+]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[N_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[NUM_ITERS_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[I:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[N_CASTED:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [3 x i64], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [3 x ptr], align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [3 x ptr], align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [3 x ptr], align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_SIZES11:%.*]] = alloca [3 x i64], align 8
+// CHECK-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTCAPTURE_EXPR_12:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
+// CHECK-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 8
+// CHECK-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
+// CHECK-NEXT:    store i32 [[NUM_ITERS]], ptr [[NUM_ITERS_ADDR]], align 4
+// CHECK-NEXT:    store i32 0, ptr [[I]], align 4
+// CHECK-NEXT:    br label [[FOR_COND:%.*]]
+// CHECK:       for.cond:
+// CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[I]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[NUM_ITERS_ADDR]], align 4
+// CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 [[TMP0]], [[TMP1]]
+// CHECK-NEXT:    br i1 [[CMP]], label [[FOR_BODY:%.*]], label [[FOR_END:%.*]]
+// CHECK:       for.body:
+// CHECK-NEXT:    [[TMP2:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// CHECK-NEXT:    store i32 [[TMP2]], ptr [[N_CASTED]], align 4
+// CHECK-NEXT:    [[TMP3:%.*]] = load i64, ptr [[N_CASTED]], align 8
+// CHECK-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    [[TMP5:%.*]] = load ptr, ptr [[B_ADDR]], align 8
+// CHECK-NEXT:    [[TMP6:%.*]] = load ptr, ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    [[TMP7:%.*]] = load ptr, ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[TMP7]], i64 0
+// CHECK-NEXT:    [[TMP8:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// CHECK-NEXT:    [[CONV:%.*]] = sext i32 [[TMP8]] to i64
+// CHECK-NEXT:    [[TMP9:%.*]] = mul nuw i64 [[CONV]], 4
+// CHECK-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[B_ADDR]], align 8
+// CHECK-NEXT:    [[TMP11:%.*]] = load ptr, ptr [[B_ADDR]], align 8
+// CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds i32, ptr [[TMP11]], i64 0
+// CHECK-NEXT:    [[TMP12:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// CHECK-NEXT:    [[CONV2:%.*]] = sext i32 [[TMP12]] to i64
+// CHECK-NEXT:    [[TMP13:%.*]] = mul nuw i64 [[CONV2]], 4
+// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[DOTOFFLOAD_SIZES]], ptr align 8 @.offload_sizes, i64 24, i1 false)
+// CHECK-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-NEXT:    store i64 [[TMP3]], ptr [[TMP14]], align 8
+// CHECK-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-NEXT:    store i64 [[TMP3]], ptr [[TMP15]], align 8
+// CHECK-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 0
+// CHECK-NEXT:    store ptr null, ptr [[TMP16]], align 8
+// CHECK-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 1
+// CHECK-NEXT:    store ptr [[TMP6]], ptr [[TMP17]], align 8
+// CHECK-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 1
+// CHECK-NEXT:    store ptr [[ARRAYIDX]], ptr [[TMP18]], align 8
+// CHECK-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 1
+// CHECK-NEXT:    store i64 [[TMP9]], ptr [[TMP19]], align 8
+// CHECK-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 1
+// CHECK-NEXT:    store ptr null, ptr [[TMP20]], align 8
+// CHECK-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 2
+// CHECK-NEXT:    store ptr [[TMP10]], ptr [[TMP21]], align 8
+// CHECK-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 2
+// CHECK-NEXT:    store ptr [[ARRAYIDX1]], ptr [[TMP22]], align 8
+// CHECK-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 2
+// CHECK-NEXT:    store i64 [[TMP13]], ptr [[TMP23]], align 8
+// CHECK-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 2
+// CHECK-NEXT:    store ptr null, ptr [[TMP24]], align 8
+// CHECK-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
+// CHECK-NEXT:    store i32 2, ptr [[TMP28]], align 4
+// CHECK-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
+// CHECK-NEXT:    store i32 3, ptr [[TMP29]], align 4
+// CHECK-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 2
+// CHECK-NEXT:    store ptr [[TMP25]], ptr [[TMP30]], align 8
+// CHECK-NEXT:    [[TMP31:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 3
+// CHECK-NEXT:    store ptr [[TMP26]], ptr [[TMP31]], align 8
+// CHECK-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 4
+// CHECK-NEXT:    store ptr [[TMP27]], ptr [[TMP32]], align 8
+// CHECK-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 5
+// CHECK-NEXT:    store ptr @.offload_maptypes, ptr [[TMP33]], align 8
+// CHECK-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 6
+// CHECK-NEXT:    store ptr null, ptr [[TMP34]], align 8
+// CHECK-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 7
+// CHECK-NEXT:    store ptr null, ptr [[TMP35]], align 8
+// CHECK-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 8
+// CHECK-NEXT:    store i64 0, ptr [[TMP36]], align 8
+// CHECK-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 9
+// CHECK-NEXT:    store i64 0, ptr [[TMP37]], align 8
+// CHECK-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 10
+// CHECK-NEXT:    store [3 x i32] [i32 1, i32 0, i32 0], ptr [[TMP38]], align 4
+// CHECK-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 11
+// CHECK-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP39]], align 4
+// CHECK-NEXT:    [[TMP40:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 12
+// CHECK-NEXT:    store i32 0, ptr [[TMP40]], align 4
+// CHECK-NEXT:    [[TMP41:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB2:[0-9]+]], i64 -1, i32 1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z18kernel_within_loopPiS_ii_l9.region_id, ptr [[KERNEL_ARGS]])
+// CHECK-NEXT:    [[TMP42:%.*]] = icmp ne i32 [[TMP41]], 0
+// CHECK-NEXT:    br i1 [[TMP42]], label [[OMP_OFFLOAD_FAILED:%.*]], label [[OMP_OFFLOAD_CONT:%.*]]
+// CHECK:       omp_offload.failed:
+// CHECK-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z18kernel_within_loopPiS_ii_l9(i64 [[TMP3]], ptr [[TMP4]], ptr [[TMP5]]) #[[ATTR3:[0-9]+]]
+// CHECK-NEXT:    br label [[OMP_OFFLOAD_CONT]]
+// CHECK:       omp_offload.cont:
+// CHECK-NEXT:    [[TMP43:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// CHECK-NEXT:    store i32 [[TMP43]], ptr [[N_CASTED3]], align 4
+// CHECK-NEXT:    [[TMP44:%.*]] = load i64, ptr [[N_CASTED3]], align 8
+// CHECK-NEXT:    [[TMP45:%.*]] = load ptr, ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    [[TMP46:%.*]] = load ptr, ptr [[B_ADDR]], align 8
+// CHECK-NEXT:    [[TMP47:%.*]] = load ptr, ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    [[TMP48:%.*]] = load ptr, ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    [[ARRAYIDX4:%.*]] = getelementptr inbounds i32, ptr [[TMP48]], i64 0
+// CHECK-NEXT:    [[TMP49:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// CHECK-NEXT:    [[CONV5:%.*]] = sext i32 [[TMP49]] to i64
+// CHECK-NEXT:    [[TMP50:%.*]] = mul nuw i64 [[CONV5]], 4
+// CHECK-NEXT:    [[TMP51:%.*]] = load ptr, ptr [[B_ADDR]], align 8
+// CHECK-NEXT:    [[TMP52:%.*]] = load ptr, ptr [[B_ADDR]], align 8
+// CHECK-NEXT:    [[ARRAYIDX6:%.*]] = getelementptr inbounds i32, ptr [[TMP52]], i64 0
+// CHECK-NEXT:    [[TMP53:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// CHECK-NEXT:    [[CONV7:%.*]] = sext i32 [[TMP53]] to i64
+// CHECK-NEXT:    [[TMP54:%.*]] = mul nuw i64 [[CONV7]], 4
+// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[DOTOFFLOAD_SIZES11]], ptr align 8 @.offload_sizes.3, i64 24, i1 false)
+// CHECK-NEXT:    [[TMP55:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
+// CHECK-NEXT:    store i64 [[TMP44]], ptr [[TMP55]], align 8
+// CHECK-NEXT:    [[TMP56:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
+// CHECK-NEXT:    store i64 [[TMP44]], ptr [[TMP56]], align 8
+// CHECK-NEXT:    [[TMP57:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS10]], i64 0, i64 0
+// CHECK-NEXT:    store ptr null, ptr [[TMP57]], align 8
+// CHECK-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 1
+// CHECK-NEXT:    store ptr [[TMP47]], ptr [[TMP58]], align 8
+// CHECK-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 1
+// CHECK-NEXT:    store ptr [[ARRAYIDX4]], ptr [[TMP59]], align 8
+// CHECK-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES11]], i32 0, i32 1
+// CHECK-NEXT:    store i64 [[TMP50]], ptr [[TMP60]], align 8
+// CHECK-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS10]], i64 0, i64 1
+// CHECK-NEXT:    store ptr null, ptr [[TMP61]], align 8
+// CHECK-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 2
+// CHECK-NEXT:    store ptr [[TMP51]], ptr [[TMP62]], align 8
+// CHECK-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 2
+// CHECK-NEXT:    store ptr [[ARRAYIDX6]], ptr [[TMP63]], align 8
+// CHECK-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES11]], i32 0, i32 2
+// CHECK-NEXT:    store i64 [[TMP54]], ptr [[TMP64]], align 8
+// CHECK-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS10]], i64 0, i64 2
+// CHECK-NEXT:    store ptr null, ptr [[TMP65]], align 8
+// CHECK-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP67:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES11]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP69:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// CHECK-NEXT:    store i32 [[TMP69]], ptr [[DOTCAPTURE_EXPR_]], align 4
+// CHECK-NEXT:    [[TMP70:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
+// CHECK-NEXT:    [[SUB:%.*]] = sub i32 [[TMP70]], -2
+// CHECK-NEXT:    [[DIV:%.*]] = udiv i32 [[SUB]], 3
+// CHECK-NEXT:    [[SUB13:%.*]] = sub i32 [[DIV]], 1
+// CHECK-NEXT:    store i32 [[SUB13]], ptr [[DOTCAPTURE_EXPR_12]], align 4
+// CHECK-NEXT:    [[TMP71:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_12]], align 4
+// CHECK-NEXT:    [[ADD:%.*]] = add i32 [[TMP71]], 1
+// CHECK-NEXT:    [[TMP72:%.*]] = zext i32 [[ADD]] to i64
+// CHECK-NEXT:    [[TMP73:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 0
+// CHECK-NEXT:    store i32 2, ptr [[TMP73]], align 4
+// CHECK-NEXT:    [[TMP74:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 1
+// CHECK-NEXT:    store i32 3, ptr [[TMP74]], align 4
+// CHECK-NEXT:    [[TMP75:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 2
+// CHECK-NEXT:    store ptr [[TMP66]], ptr [[TMP75]], align 8
+// CHECK-NEXT:    [[TMP76:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 3
+// CHECK-NEXT:    store ptr [[TMP67]], ptr [[TMP76]], align 8
+// CHECK-NEXT:    [[TMP77:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 4
+// CHECK-NEXT:    store ptr [[TMP68]], ptr [[TMP77]], align 8
+// CHECK-NEXT:    [[TMP78:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 5
+// CHECK-NEXT:    store ptr @.offload_maptypes.4, ptr [[TMP78]], align 8
+// CHECK-NEXT:    [[TMP79:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 6
+// CHECK-NEXT:    store ptr null, ptr [[TMP79]], align 8
+// CHECK-NEXT:    [[TMP80:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 7
+// CHECK-NEXT:    store ptr null, ptr [[TMP80]], align 8
+// CHECK-NEXT:    [[TMP81:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 8
+// CHECK-NEXT:    store i64 [[TMP72]], ptr [[TMP81]], align 8
+// CHECK-NEXT:    [[TMP82:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 9
+// CHECK-NEXT:    store i64 0, ptr [[TMP82]], align 8
+// CHECK-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 10
+// CHECK-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP83]], align 4
+// CHECK-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 11
+// CHECK-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP84]], align 4
+// CHECK-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 12
+// CHECK-NEXT:    store i32 0, ptr [[TMP85]], align 4
+// CHECK-NEXT:    [[TMP86:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB2]], i64 -1, i32 0, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z18kernel_within_loopPiS_ii_l13.region_id, ptr [[KERNEL_ARGS14]])
+// CHECK-NEXT:    [[TMP87:%.*]] = icmp ne i32 [[TMP86]], 0
+// CHECK-NEXT:    br i1 [[TMP87]], label [[OMP_OFFLOAD_FAILED15:%.*]], label [[OMP_OFFLOAD_CONT16:%.*]]
+// CHECK:       omp_offload.failed15:
+// CHECK-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z18kernel_within_loopPiS_ii_l13(i64 [[TMP44]], ptr [[TMP45]], ptr [[TMP46]]) #[[ATTR3]]
+// CHECK-NEXT:    br label [[OMP_OFFLOAD_CONT16]]
+// CHECK:       omp_offload.cont16:
+// CHECK-NEXT:    br label [[FOR_INC:%.*]]
+// CHECK:       for.inc:
+// CHECK-NEXT:    [[TMP88:%.*]] = load i32, ptr [[I]], align 4
+// CHECK-NEXT:    [[INC:%.*]] = add nsw i32 [[TMP88]], 1
+// CHECK-NEXT:    store i32 [[INC]], ptr [[I]], align 4
+// CHECK-NEXT:    br label [[FOR_COND]], !llvm.loop [[LOOP5:![0-9]+]]
+// CHECK:       for.end:
+// CHECK-NEXT:    [[TMP89:%.*]] = load ptr, ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    [[TMP90:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// CHECK-NEXT:    [[SUB17:%.*]] = sub nsw i32 [[TMP90]], 1
+// CHECK-NEXT:    [[IDXPROM:%.*]] = sext i32 [[SUB17]] to i64
+// CHECK-NEXT:    [[ARRAYIDX18:%.*]] = getelementptr inbounds i32, ptr [[TMP89]], i64 [[IDXPROM]]
+// CHECK-NEXT:    [[TMP91:%.*]] = load i32, ptr [[ARRAYIDX18]], align 4
+// CHECK-NEXT:    ret i32 [[TMP91]]
+//
+//
+// CHECK-LABEL: define internal void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z18kernel_within_loopPiS_ii_l9
+// CHECK-SAME: (i64 noundef [[N:%.*]], ptr noundef [[A:%.*]], ptr noundef [[B:%.*]]) #[[ATTR1:[0-9]+]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[N_ADDR:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[N_CASTED:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    store i64 [[N]], ptr [[N_ADDR]], align 8
+// CHECK-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[N_CASTED]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = load i64, ptr [[N_CASTED]], align 8
+// CHECK-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[B_ADDR]], align 8
+// CHECK-NEXT:    call void (ptr, i32, ptr, ...) @__kmpc_fork_call(ptr @[[GLOB2]], i32 3, ptr @.omp_outlined., i64 [[TMP1]], ptr [[TMP2]], ptr [[TMP3]])
+// CHECK-NEXT:    ret void
+//
+//
+// CHECK-LABEL: define internal void @.omp_outlined.
+// CHECK-SAME: (ptr noalias noundef [[DOTGLOBAL_TID_:%.*]], ptr noalias noundef [[DOTBOUND_TID_:%.*]], i64 noundef [[N:%.*]], ptr noundef [[A:%.*]], ptr noundef [[B:%.*]]) #[[ATTR2:[0-9]+]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DOTGLOBAL_TID__ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[DOTBOUND_TID__ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[N_ADDR:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[DOTOMP_IV:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[J:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTOMP_LB:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTOMP_UB:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTOMP_STRIDE:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTOMP_IS_LAST:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[J3:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    store ptr [[DOTGLOBAL_TID_]], ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// CHECK-NEXT:    store ptr [[DOTBOUND_TID_]], ptr [[DOTBOUND_TID__ADDR]], align 8
+// CHECK-NEXT:    store i64 [[N]], ptr [[N_ADDR]], align 8
+// CHECK-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[DOTCAPTURE_EXPR_]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
+// CHECK-NEXT:    [[SUB:%.*]] = sub nsw i32 [[TMP1]], 0
+// CHECK-NEXT:    [[DIV:%.*]] = sdiv i32 [[SUB]], 1
+// CHECK-NEXT:    [[SUB2:%.*]] = sub nsw i32 [[DIV]], 1
+// CHECK-NEXT:    store i32 [[SUB2]], ptr [[DOTCAPTURE_EXPR_1]], align 4
+// CHECK-NEXT:    store i32 0, ptr [[J]], align 4
+// CHECK-NEXT:    [[TMP2:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
+// CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 0, [[TMP2]]
+// CHECK-NEXT:    br i1 [[CMP]], label [[OMP_PRECOND_THEN:%.*]], label [[OMP_PRECOND_END:%.*]]
+// CHECK:       omp.precond.then:
+// CHECK-NEXT:    store i32 0, ptr [[DOTOMP_LB]], align 4
+// CHECK-NEXT:    [[TMP3:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
+// CHECK-NEXT:    store i32 [[TMP3]], ptr [[DOTOMP_UB]], align 4
+// CHECK-NEXT:    store i32 1, ptr [[DOTOMP_STRIDE]], align 4
+// CHECK-NEXT:    store i32 0, ptr [[DOTOMP_IS_LAST]], align 4
+// CHECK-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// CHECK-NEXT:    [[TMP5:%.*]] = load i32, ptr [[TMP4]], align 4
+// CHECK-NEXT:    call void @__kmpc_for_static_init_4(ptr @[[GLOB1:[0-9]+]], i32 [[TMP5]], i32 34, ptr [[DOTOMP_IS_LAST]], ptr [[DOTOMP_LB]], ptr [[DOTOMP_UB]], ptr [[DOTOMP_STRIDE]], i32 1, i32 1)
+// CHECK-NEXT:    [[TMP6:%.*]] = load i32, ptr [[DOTOMP_UB]], align 4
+// CHECK-NEXT:    [[TMP7:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
+// CHECK-NEXT:    [[CMP4:%.*]] = icmp sgt i32 [[TMP6]], [[TMP7]]
+// CHECK-NEXT:    br i1 [[CMP4]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]]
+// CHECK:       cond.true:
+// CHECK-NEXT:    [[TMP8:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
+// CHECK-NEXT:    br label [[COND_END:%.*]]
+// CHECK:       cond.false:
+// CHECK-NEXT:    [[TMP9:%.*]] = load i32, ptr [[DOTOMP_UB]], align 4
+// CHECK-NEXT:    br label [[COND_END]]
+// CHECK:       cond.end:
+// CHECK-NEXT:    [[COND:%.*]] = phi i32 [ [[TMP8]], [[COND_TRUE]] ], [ [[TMP9]], [[COND_FALSE]] ]
+// CHECK-NEXT:    store i32 [[COND]], ptr [[DOTOMP_UB]], align 4
+// CHECK-NEXT:    [[TMP10:%.*]] = load i32, ptr [[DOTOMP_LB]], align 4
+// CHECK-NEXT:    store i32 [[TMP10]], ptr [[DOTOMP_IV]], align 4
+// CHECK-NEXT:    br label [[OMP_INNER_FOR_COND:%.*]]
+// CHECK:       omp.inner.for.cond:
+// CHECK-NEXT:    [[TMP11:%.*]] = load i32, ptr [[DOTOMP_IV]], align 4
+// CHECK-NEXT:    [[TMP12:%.*]] = load i32, ptr [[DOTOMP_UB]], align 4
+// CHECK-NEXT:    [[CMP5:%.*]] = icmp sle i32 [[TMP11]], [[TMP12]]
+// CHECK-NEXT:    br i1 [[CMP5]], label [[OMP_INNER_FOR_BODY:%.*]], label [[OMP_INNER_FOR_END:%.*]]
+// CHECK:       omp.inner.for.body:
+// CHECK-NEXT:    [[TMP13:%.*]] = load i32, ptr [[DOTOMP_IV]], align 4
+// CHECK-NEXT:    [[MUL:%.*]] = mul nsw i32 [[TMP13]], 1
+// CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 0, [[MUL]]
+// CHECK-NEXT:    store i32 [[ADD]], ptr [[J3]], align 4
+// CHECK-NEXT:    [[TMP14:%.*]] = load ptr, ptr [[B_ADDR]], align 8
+// CHECK-NEXT:    [[TMP15:%.*]] = load i32, ptr [[J3]], align 4
+// CHECK-NEXT:    [[IDXPROM:%.*]] = sext i32 [[TMP15]] to i64
+// CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[TMP14]], i64 [[IDXPROM]]
+// CHECK-NEXT:    [[TMP16:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
+// CHECK-NEXT:    [[TMP17:%.*]] = load ptr, ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    [[TMP18:%.*]] = load i32, ptr [[J3]], align 4
+// CHECK-NEXT:    [[IDXPROM6:%.*]] = sext i32 [[TMP18]] to i64
+// CHECK-NEXT:    [[ARRAYIDX7:%.*]] = getelementptr inbounds i32, ptr [[TMP17]], i64 [[IDXPROM6]]
+// CHECK-NEXT:    store i32 [[TMP16]], ptr [[ARRAYIDX7]], align 4
+// CHECK-NEXT:    br label [[OMP_BODY_CONTINUE:%.*]]
+// CHECK:       omp.body.continue:
+// CHECK-NEXT:    br label [[OMP_INNER_FOR_INC:%.*]]
+// CHECK:       omp.inner.for.inc:
+// CHECK-NEXT:    [[TMP19:%.*]] = load i32, ptr [[DOTOMP_IV]], align 4
+// CHECK-NEXT:    [[ADD8:%.*]] = add nsw i32 [[TMP19]], 1
+// CHECK-NEXT:    store i32 [[ADD8]], ptr [[DOTOMP_IV]], align 4
+// CHECK-NEXT:    br label [[OMP_INNER_FOR_COND]]
+// CHECK:       omp.inner.for.end:
+// CHECK-NEXT:    br label [[OMP_LOOP_EXIT:%.*]]
+// CHECK:       omp.loop.exit:
+// CHECK-NEXT:    [[TMP20:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// CHECK-NEXT:    [[TMP21:%.*]] = load i32, ptr [[TMP20]], align 4
+// CHECK-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB1]], i32 [[TMP21]])
+// CHECK-NEXT:    br label [[OMP_PRECOND_END]]
+// CHECK:       omp.precond.end:
+// CHECK-NEXT:    ret void
+//
+//
+// CHECK-LABEL: define internal void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z18kernel_within_loopPiS_ii_l13
+// CHECK-SAME: (i64 noundef [[N:%.*]], ptr noundef [[A:%.*]], ptr noundef [[B:%.*]]) #[[ATTR2]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[N_ADDR:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[N_CASTED:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    store i64 [[N]], ptr [[N_ADDR]], align 8
+// CHECK-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[N_CASTED]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = load i64, ptr [[N_CASTED]], align 8
+// CHECK-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[B_ADDR]], align 8
+// CHECK-NEXT:    call void (ptr, i32, ptr, ...) @__kmpc_fork_teams(ptr @[[GLOB2]], i32 3, ptr @.omp_outlined..1, i64 [[TMP1]], ptr [[TMP2]], ptr [[TMP3]])
+// CHECK-NEXT:    ret void
+//
+//
+// CHECK-LABEL: define internal void @.omp_outlined..1
+// CHECK-SAME: (ptr noalias noundef [[DOTGLOBAL_TID_:%.*]], ptr noalias noundef [[DOTBOUND_TID_:%.*]], i64 noundef [[N:%.*]], ptr noundef [[A:%.*]], ptr noundef [[B:%.*]]) #[[ATTR2]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DOTGLOBAL_TID__ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[DOTBOUND_TID__ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[N_ADDR:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[DOTOMP_IV:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[J:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTOMP_COMB_LB:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTOMP_COMB_UB:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTOMP_STRIDE:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTOMP_IS_LAST:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[J3:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[N_CASTED:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    store ptr [[DOTGLOBAL_TID_]], ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// CHECK-NEXT:    store ptr [[DOTBOUND_TID_]], ptr [[DOTBOUND_TID__ADDR]], align 8
+// CHECK-NEXT:    store i64 [[N]], ptr [[N_ADDR]], align 8
+// CHECK-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[DOTCAPTURE_EXPR_]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
+// CHECK-NEXT:    [[SUB:%.*]] = sub i32 [[TMP1]], -2
+// CHECK-NEXT:    [[DIV:%.*]] = udiv i32 [[SUB]], 3
+// CHECK-NEXT:    [[SUB2:%.*]] = sub i32 [[DIV]], 1
+// CHECK-NEXT:    store i32 [[SUB2]], ptr [[DOTCAPTURE_EXPR_1]], align 4
+// CHECK-NEXT:    store i32 0, ptr [[J]], align 4
+// CHECK-NEXT:    [[TMP2:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
+// CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 0, [[TMP2]]
+// CHECK-NEXT:    br i1 [[CMP]], label [[OMP_PRECOND_THEN:%.*]], label [[OMP_PRECOND_END:%.*]]
+// CHECK:       omp.precond.then:
+// CHECK-NEXT:    store i32 0, ptr [[DOTOMP_COMB_LB]], align 4
+// CHECK-NEXT:    [[TMP3:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
+// CHECK-NEXT:    store i32 [[TMP3]], ptr [[DOTOMP_COMB_UB]], align 4
+// CHECK-NEXT:    store i32 1, ptr [[DOTOMP_STRIDE]], align 4
+// CHECK-NEXT:    store i32 0, ptr [[DOTOMP_IS_LAST]], align 4
+// CHECK-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// CHECK-NEXT:    [[TMP5:%.*]] = load i32, ptr [[TMP4]], align 4
+// CHECK-NEXT:    call void @__kmpc_for_static_init_4u(ptr @[[GLOB3:[0-9]+]], i32 [[TMP5]], i32 92, ptr [[DOTOMP_IS_LAST]], ptr [[DOTOMP_COMB_LB]], ptr [[DOTOMP_COMB_UB]], ptr [[DOTOMP_STRIDE]], i32 1, i32 1)
+// CHECK-NEXT:    [[TMP6:%.*]] = load i32, ptr [[DOTOMP_COMB_UB]], align 4
+// CHECK-NEXT:    [[TMP7:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
+// CHECK-NEXT:    [[CMP4:%.*]] = icmp ugt i32 [[TMP6]], [[TMP7]]
+// CHECK-NEXT:    br i1 [[CMP4]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]]
+// CHECK:       cond.true:
+// CHECK-NEXT:    [[TMP8:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
+// CHECK-NEXT:    br label [[COND_END:%.*]]
+// CHECK:       cond.false:
+// CHECK-NEXT:    [[TMP9:%.*]] = load i32, ptr [[DOTOMP_COMB_UB]], align 4
+// CHECK-NEXT:    br label [[COND_END]]
+// CHECK:       cond.end:
+// CHECK-NEXT:    [[COND:%.*]] = phi i32 [ [[TMP8]], [[COND_TRUE]] ], [ [[TMP9]], [[COND_FALSE]] ]
+// CHECK-NEXT:    store i32 [[COND]], ptr [[DOTOMP_COMB_UB]], align 4
+// CHECK-NEXT:    [[TMP10:%.*]] = load i32, ptr [[DOTOMP_COMB_LB]], align 4
+// CHECK-NEXT:    store i32 [[TMP10]], ptr [[DOTOMP_IV]], align 4
+// CHECK-NEXT:    br label [[OMP_INNER_FOR_COND:%.*]]
+// CHECK:       omp.inner.for.cond:
+// CHECK-NEXT:    [[TMP11:%.*]] = load i32, ptr [[DOTOMP_IV]], align 4
+// CHECK-NEXT:    [[TMP12:%.*]] = load i32, ptr [[DOTOMP_COMB_UB]], align 4
+// CHECK-NEXT:    [[ADD:%.*]] = add i32 [[TMP12]], 1
+// CHECK-NEXT:    [[CMP5:%.*]] = icmp ult i32 [[TMP11]], [[ADD]]
+// CHECK-NEXT:    br i1 [[CMP5]], label [[OMP_INNER_FOR_BODY:%.*]], label [[OMP_INNER_FOR_END:%.*]]
+// CHECK:       omp.inner.for.body:
+// CHECK-NEXT:    [[TMP13:%.*]] = load i32, ptr [[DOTOMP_COMB_LB]], align 4
+// CHECK-NEXT:    [[TMP14:%.*]] = zext i32 [[TMP13]] to i64
+// CHECK-NEXT:    [[TMP15:%.*]] = load i32, ptr [[DOTOMP_COMB_UB]], align 4
+// CHECK-NEXT:    [[TMP16:%.*]] = zext i32 [[TMP15]] to i64
+// CHECK-NEXT:    [[TMP17:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// CHECK-NEXT:    store i32 [[TMP17]], ptr [[N_CASTED]], align 4
+// CHECK-NEXT:    [[TMP18:%.*]] = load i64, ptr [[N_CASTED]], align 8
+// CHECK-NEXT:    [[TMP19:%.*]] = load ptr, ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    [[TMP20:%.*]] = load ptr, ptr [[B_ADDR]], align 8
+// CHECK-NEXT:    call void (ptr, i32, ptr, ...) @__kmpc_fork_call(ptr @[[GLOB2]], i32 5, ptr @.omp_outlined..2, i64 [[TMP14]], i64 [[TMP16]], i64 [[TMP18]], ptr [[TMP19]], ptr [[TMP20]])
+// CHECK-NEXT:    br label [[OMP_INNER_FOR_INC:%.*]]
+// CHECK:       omp.inner.for.inc:
+// CHECK-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTOMP_IV]], align 4
+// CHECK-NEXT:    [[TMP22:%.*]] = load i32, ptr [[DOTOMP_STRIDE]], align 4
+// CHECK-NEXT:    [[ADD6:%.*]] = add i32 [[TMP21]], [[TMP22]]
+// CHECK-NEXT:    store i32 [[ADD6]], ptr [[DOTOMP_IV]], align 4
+// CHECK-NEXT:    br label [[OMP_INNER_FOR_COND]]
+// CHECK:       omp.inner.for.end:
+// CHECK-NEXT:    br label [[OMP_LOOP_EXIT:%.*]]
+// CHECK:       omp.loop.exit:
+// CHECK-NEXT:    [[TMP23:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// CHECK-NEXT:    [[TMP24:%.*]] = load i32, ptr [[TMP23]], align 4
+// CHECK-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB3]], i32 [[TMP24]])
+// CHECK-NEXT:    br label [[OMP_PRECOND_END]]
+// CHECK:       omp.precond.end:
+// CHECK-NEXT:    ret void
+//
+//
+// CHECK-LABEL: define internal void @.omp_outlined..2
+// CHECK-SAME: (ptr noalias noundef [[DOTGLOBAL_TID_:%.*]], ptr noalias noundef [[DOTBOUND_TID_:%.*]], i64 noundef [[DOTPREVIOUS_LB_:%.*]], i64 noundef [[DOTPREVIOUS_UB_:%.*]], i64 noundef [[N:%.*]], ptr noundef [[A:%.*]], ptr noundef [[B:%.*]]) #[[ATTR2]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DOTGLOBAL_TID__ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[DOTBOUND_TID__ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[DOTPREVIOUS_LB__ADDR:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[DOTPREVIOUS_UB__ADDR:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[N_ADDR:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[DOTOMP_IV:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[J:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTOMP_LB:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTOMP_UB:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTOMP_STRIDE:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTOMP_IS_LAST:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[J4:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    store ptr [[DOTGLOBAL_TID_]], ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// CHECK-NEXT:    store ptr [[DOTBOUND_TID_]], ptr [[DOTBOUND_TID__ADDR]], align 8
+// CHECK-NEXT:    store i64 [[DOTPREVIOUS_LB_]], ptr [[DOTPREVIOUS_LB__ADDR]], align 8
+// CHECK-NEXT:    store i64 [[DOTPREVIOUS_UB_]], ptr [[DOTPREVIOUS_UB__ADDR]], align 8
+// CHECK-NEXT:    store i64 [[N]], ptr [[N_ADDR]], align 8
+// CHECK-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[DOTCAPTURE_EXPR_]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
+// CHECK-NEXT:    [[SUB:%.*]] = sub i32 [[TMP1]], -2
+// CHECK-NEXT:    [[DIV:%.*]] = udiv i32 [[SUB]], 3
+// CHECK-NEXT:    [[SUB2:%.*]] = sub i32 [[DIV]], 1
+// CHECK-NEXT:    store i32 [[SUB2]], ptr [[DOTCAPTURE_EXPR_1]], align 4
+// CHECK-NEXT:    store i32 0, ptr [[J]], align 4
+// CHECK-NEXT:    [[TMP2:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
+// CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 0, [[TMP2]]
+// CHECK-NEXT:    br i1 [[CMP]], label [[OMP_PRECOND_THEN:%.*]], label [[OMP_PRECOND_END:%.*]]
+// CHECK:       omp.precond.then:
+// CHECK-NEXT:    store i32 0, ptr [[DOTOMP_LB]], align 4
+// CHECK-NEXT:    [[TMP3:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
+// CHECK-NEXT:    store i32 [[TMP3]], ptr [[DOTOMP_UB]], align 4
+// CHECK-NEXT:    [[TMP4:%.*]] = load i64, ptr [[DOTPREVIOUS_LB__ADDR]], align 8
+// CHECK-NEXT:    [[CONV:%.*]] = trunc i64 [[TMP4]] to i32
+// CHECK-NEXT:    [[TMP5:%.*]] = load i64, ptr [[DOTPREVIOUS_UB__ADDR]], align 8
+// CHECK-NEXT:    [[CONV3:%.*]] = trunc i64 [[TMP5]] to i32
+// CHECK-NEXT:    store i32 [[CONV]], ptr [[DOTOMP_LB]], align 4
+// CHECK-NEXT:    store i32 [[CONV3]], ptr [[DOTOMP_UB]], align 4
+// CHECK-NEXT:    store i32 1, ptr [[DOTOMP_STRIDE]], align 4
+// CHECK-NEXT:    store i32 0, ptr [[DOTOMP_IS_LAST]], align 4
+// CHECK-NEXT:    [[TMP6:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// CHECK-NEXT:    [[TMP7:%.*]] = load i32, ptr [[TMP6]], align 4
+// CHECK-NEXT:    call void @__kmpc_for_static_init_4u(ptr @[[GLOB1]], i32 [[TMP7]], i32 34, ptr [[DOTOMP_IS_LAST]], ptr [[DOTOMP_LB]], ptr [[DOTOMP_UB]], ptr [[DOTOMP_STRIDE]], i32 1, i32 1)
+// CHECK-NEXT:    [[TMP8:%.*]] = load i32, ptr [[DOTOMP_UB]], align 4
+// CHECK-NEXT:    [[TMP9:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
+// CHECK-NEXT:    [[CMP5:%.*]] = icmp ugt i32 [[TMP8]], [[TMP9]]
+// CHECK-NEXT:    br i1 [[CMP5]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]]
+// CHECK:       cond.true:
+// CHECK-NEXT:    [[TMP10:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
+// CHECK-NEXT:    br label [[COND_END:%.*]]
+// CHECK:       cond.false:
+// CHECK-NEXT:    [[TMP11:%.*]] = load i32, ptr [[DOTOMP_UB]], align 4
+// CHECK-NEXT:    br label [[COND_END]]
+// CHECK:       cond.end:
+// CHECK-NEXT:    [[COND:%.*]] = phi i32 [ [[TMP10]], [[COND_TRUE]] ], [ [[TMP11]], [[COND_FALSE]] ]
+// CHECK-NEXT:    store i32 [[COND]], ptr [[DOTOMP_UB]], align 4
+// CHECK-NEXT:    [[TMP12:%.*]] = load i32, ptr [[DOTOMP_LB]], align 4
+// CHECK-NEXT:    store i32 [[TMP12]], ptr [[DOTOMP_IV]], align 4
+// CHECK-NEXT:    br label [[OMP_INNER_FOR_COND:%.*]]
+// CHECK:       omp.inner.for.cond:
+// CHECK-NEXT:    [[TMP13:%.*]] = load i32, ptr [[DOTOMP_IV]], align 4
+// CHECK-NEXT:    [[TMP14:%.*]] = load i32, ptr [[DOTOMP_UB]], align 4
+// CHECK-NEXT:    [[ADD:%.*]] = add i32 [[TMP14]], 1
+// CHECK-NEXT:    [[CMP6:%.*]] = icmp ult i32 [[TMP13]], [[ADD]]
+// CHECK-NEXT:    br i1 [[CMP6]], label [[OMP_INNER_FOR_BODY:%.*]], label [[OMP_INNER_FOR_END:%.*]]
+// CHECK:       omp.inner.for.body:
+// CHECK-NEXT:    [[TMP15:%.*]] = load i32, ptr [[DOTOMP_IV]], align 4
+// CHECK-NEXT:    [[MUL:%.*]] = mul i32 [[TMP15]], 3
+// CHECK-NEXT:    [[ADD7:%.*]] = add i32 0, [[MUL]]
+// CHECK-NEXT:    store i32 [[ADD7]], ptr [[J4]], align 4
+// CHECK-NEXT:    [[TMP16:%.*]] = load ptr, ptr [[B_ADDR]], align 8
+// CHECK-NEXT:    [[TMP17:%.*]] = load i32, ptr [[J4]], align 4
+// CHECK-NEXT:    [[IDXPROM:%.*]] = sext i32 [[TMP17]] to i64
+// CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[TMP16]], i64 [[IDXPROM]]
+// CHECK-NEXT:    [[TMP18:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
+// CHECK-NEXT:    [[MUL8:%.*]] = mul nsw i32 [[TMP18]], 2
+// CHECK-NEXT:    [[TMP19:%.*]] = load ptr, ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    [[TMP20:%.*]] = load i32, ptr [[J4]], align 4
+// CHECK-NEXT:    [[IDXPROM9:%.*]] = sext i32 [[TMP20]] to i64
+// CHECK-NEXT:    [[ARRAYIDX10:%.*]] = getelementptr inbounds i32, ptr [[TMP19]], i64 [[IDXPROM9]]
+// CHECK-NEXT:    store i32 [[MUL8]], ptr [[ARRAYIDX10]], align 4
+// CHECK-NEXT:    br label [[OMP_BODY_CONTINUE:%.*]]
+// CHECK:       omp.body.continue:
+// CHECK-NEXT:    br label [[OMP_INNER_FOR_INC:%.*]]
+// CHECK:       omp.inner.for.inc:
+// CHECK-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTOMP_IV]], align 4
+// CHECK-NEXT:    [[ADD11:%.*]] = add i32 [[TMP21]], 1
+// CHECK-NEXT:    store i32 [[ADD11]], ptr [[DOTOMP_IV]], align 4
+// CHECK-NEXT:    br label [[OMP_INNER_FOR_COND]]
+// CHECK:       omp.inner.for.end:
+// CHECK-NEXT:    br label [[OMP_LOOP_EXIT:%.*]]
+// CHECK:       omp.loop.exit:
+// CHECK-NEXT:    [[TMP22:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// CHECK-NEXT:    [[TMP23:%.*]] = load i32, ptr [[TMP22]], align 4
+// CHECK-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB3]], i32 [[TMP23]])
+// CHECK-NEXT:    br label [[OMP_PRECOND_END]]
+// CHECK:       omp.precond.end:
+// CHECK-NEXT:    ret void
+//
+//
+// CHECK-LABEL: define internal void @.omp_offloading.requires_reg
+// CHECK-SAME: () #[[ATTR5:[0-9]+]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    call void @__tgt_register_requires(i64 1)
+// CHECK-NEXT:    ret void
+//
diff --git a/clang/test/OpenMP/distribute_codegen.cpp b/clang/test/OpenMP/distribute_codegen.cpp
index a93d3e1385761..f07ef5336d82b 100644
--- a/clang/test/OpenMP/distribute_codegen.cpp
+++ b/clang/test/OpenMP/distribute_codegen.cpp
@@ -127,6 +127,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[C]], ptr [[C_ADDR]], align 8
@@ -161,7 +162,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    store ptr null, ptr [[TMP15]], align 8
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -318,6 +318,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[C]], ptr [[C_ADDR]], align 8
@@ -352,7 +353,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    store ptr null, ptr [[TMP15]], align 8
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -509,6 +509,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[C]], ptr [[C_ADDR]], align 8
@@ -543,7 +544,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    store ptr null, ptr [[TMP15]], align 8
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -717,6 +717,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i8 0, ptr [[A]], align 1
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i8, ptr [[A]], align 1
 // CHECK1-NEXT:    store i8 [[TMP0]], ptr [[A_CASTED]], align 1
@@ -742,7 +743,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    [[TMP9:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK1-NEXT:    [[ADD4:%.*]] = add nsw i32 [[TMP9]], 1
 // CHECK1-NEXT:    [[TMP10:%.*]] = zext i32 [[ADD4]] to i64
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP11]], align 4
 // CHECK1-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -897,6 +897,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i16 0, ptr [[AA]], align 2
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i16, ptr [[AA]], align 2
 // CHECK1-NEXT:    store i16 [[TMP0]], ptr [[AA_CASTED]], align 2
@@ -909,7 +910,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1055,6 +1055,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[C]], ptr [[C_ADDR]], align 4
@@ -1089,7 +1090,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    store ptr null, ptr [[TMP15]], align 4
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1242,6 +1242,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[C]], ptr [[C_ADDR]], align 4
@@ -1276,7 +1277,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    store ptr null, ptr [[TMP15]], align 4
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1429,6 +1429,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[C]], ptr [[C_ADDR]], align 4
@@ -1463,7 +1464,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    store ptr null, ptr [[TMP15]], align 4
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1633,6 +1633,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i8, align 1
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i8 0, ptr [[A]], align 1
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i8, ptr [[A]], align 1
 // CHECK3-NEXT:    store i8 [[TMP0]], ptr [[A_CASTED]], align 1
@@ -1658,7 +1659,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    [[TMP9:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK3-NEXT:    [[ADD4:%.*]] = add nsw i32 [[TMP9]], 1
 // CHECK3-NEXT:    [[TMP10:%.*]] = zext i32 [[ADD4]] to i64
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP11]], align 4
 // CHECK3-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1813,6 +1813,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i16 0, ptr [[AA]], align 2
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i16, ptr [[AA]], align 2
 // CHECK3-NEXT:    store i16 [[TMP0]], ptr [[AA_CASTED]], align 2
@@ -1825,7 +1826,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    store ptr null, ptr [[TMP4]], align 4
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_firstprivate_codegen.cpp b/clang/test/OpenMP/distribute_firstprivate_codegen.cpp
index c385bc1209357..a1c1fc02a04a0 100644
--- a/clang/test/OpenMP/distribute_firstprivate_codegen.cpp
+++ b/clang/test/OpenMP/distribute_firstprivate_codegen.cpp
@@ -501,6 +501,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store ptr [[G]], ptr [[G1]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIfEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
@@ -554,7 +555,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -808,6 +808,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK9-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -850,7 +851,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP17]], align 8
 // CHECK9-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK9-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1173,6 +1173,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store ptr [[G]], ptr [[G1]], align 4
 // CHECK11-NEXT:    call void @_ZN1SIfEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
@@ -1226,7 +1227,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1478,6 +1478,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK11-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1520,7 +1521,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP17]], align 4
 // CHECK11-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK11-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_lastprivate_codegen.cpp b/clang/test/OpenMP/distribute_lastprivate_codegen.cpp
index 8230ab672185b..417fff2d06fc8 100644
--- a/clang/test/OpenMP/distribute_lastprivate_codegen.cpp
+++ b/clang/test/OpenMP/distribute_lastprivate_codegen.cpp
@@ -485,6 +485,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store ptr [[G]], ptr [[G1]], align 8
@@ -539,7 +540,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -811,6 +811,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK9-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -853,7 +854,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP17]], align 8
 // CHECK9-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK9-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1194,6 +1194,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store ptr [[G]], ptr [[G1]], align 4
@@ -1248,7 +1249,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1518,6 +1518,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK11-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1560,7 +1561,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP17]], align 4
 // CHECK11-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK11-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_parallel_for_codegen.cpp b/clang/test/OpenMP/distribute_parallel_for_codegen.cpp
index f333e83bf627c..c7f80248e5ed2 100644
--- a/clang/test/OpenMP/distribute_parallel_for_codegen.cpp
+++ b/clang/test/OpenMP/distribute_parallel_for_codegen.cpp
@@ -4288,6 +4288,7 @@ int main() {
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [4 x ptr], align 8
@@ -4295,6 +4296,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP7:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_8:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[CH_CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[N_CASTED17:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS18:%.*]] = alloca [5 x ptr], align 8
@@ -4303,6 +4305,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP21:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_22:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_23:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[N_CASTED31:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS32:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS33:%.*]] = alloca [4 x ptr], align 8
@@ -4310,6 +4313,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP35:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_36:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_37:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS42:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[CH_CASTED45:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[N_CASTED46:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS47:%.*]] = alloca [5 x ptr], align 8
@@ -4318,6 +4322,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP50:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_51:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_52:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS57:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[N_CASTED60:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS61:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS62:%.*]] = alloca [4 x ptr], align 8
@@ -4325,6 +4330,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP64:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_65:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_66:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS71:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[CH_CASTED74:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[N_CASTED75:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS76:%.*]] = alloca [5 x ptr], align 8
@@ -4333,6 +4339,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP79:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_80:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_81:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS86:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store i32 10000, ptr [[N]], align 4
 // CHECK9-NEXT:    store i32 100, ptr [[CH]], align 4
@@ -4378,7 +4385,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK9-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4454,7 +4460,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP59:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_9]], align 4
 // CHECK9-NEXT:    [[ADD13:%.*]] = add nsw i32 [[TMP59]], 1
 // CHECK9-NEXT:    [[TMP60:%.*]] = zext i32 [[ADD13]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP61]], align 4
 // CHECK9-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 1
@@ -4539,7 +4544,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP102:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_23]], align 4
 // CHECK9-NEXT:    [[ADD27:%.*]] = add nsw i32 [[TMP102]], 1
 // CHECK9-NEXT:    [[TMP103:%.*]] = zext i32 [[ADD27]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP104:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP104]], align 4
 // CHECK9-NEXT:    [[TMP105:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 1
@@ -4615,7 +4619,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP140:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_37]], align 4
 // CHECK9-NEXT:    [[ADD41:%.*]] = add nsw i32 [[TMP140]], 1
 // CHECK9-NEXT:    [[TMP141:%.*]] = zext i32 [[ADD41]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS42:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP142:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS42]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP142]], align 4
 // CHECK9-NEXT:    [[TMP143:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS42]], i32 0, i32 1
@@ -4700,7 +4703,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP183:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_52]], align 4
 // CHECK9-NEXT:    [[ADD56:%.*]] = add nsw i32 [[TMP183]], 1
 // CHECK9-NEXT:    [[TMP184:%.*]] = zext i32 [[ADD56]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS57:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP185:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS57]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP185]], align 4
 // CHECK9-NEXT:    [[TMP186:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS57]], i32 0, i32 1
@@ -4776,7 +4778,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP221:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_66]], align 4
 // CHECK9-NEXT:    [[ADD70:%.*]] = add nsw i32 [[TMP221]], 1
 // CHECK9-NEXT:    [[TMP222:%.*]] = zext i32 [[ADD70]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS71:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP223:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS71]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP223]], align 4
 // CHECK9-NEXT:    [[TMP224:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS71]], i32 0, i32 1
@@ -4861,7 +4862,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP264:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_81]], align 4
 // CHECK9-NEXT:    [[ADD85:%.*]] = add nsw i32 [[TMP264]], 1
 // CHECK9-NEXT:    [[TMP265:%.*]] = zext i32 [[ADD85]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS86:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP266:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS86]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP266]], align 4
 // CHECK9-NEXT:    [[TMP267:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS86]], i32 0, i32 1
@@ -6588,6 +6588,7 @@ int main() {
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [4 x ptr], align 8
@@ -6595,6 +6596,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP7:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_8:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[CH_CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[N_CASTED17:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS18:%.*]] = alloca [5 x ptr], align 8
@@ -6603,6 +6605,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP21:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_22:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_23:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[N_CASTED31:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS32:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS33:%.*]] = alloca [4 x ptr], align 8
@@ -6610,6 +6613,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP35:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_36:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_37:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS42:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[CH_CASTED45:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[N_CASTED46:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS47:%.*]] = alloca [5 x ptr], align 8
@@ -6618,6 +6622,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP50:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_51:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_52:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS57:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[N_CASTED60:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS61:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS62:%.*]] = alloca [4 x ptr], align 8
@@ -6625,6 +6630,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP64:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_65:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_66:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS71:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[CH_CASTED74:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[N_CASTED75:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS76:%.*]] = alloca [5 x ptr], align 8
@@ -6633,6 +6639,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP79:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_80:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_81:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS86:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 10000, ptr [[N]], align 4
 // CHECK9-NEXT:    store i32 100, ptr [[CH]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N]], align 4
@@ -6677,7 +6684,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK9-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -6753,7 +6759,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP59:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_9]], align 4
 // CHECK9-NEXT:    [[ADD13:%.*]] = add nsw i32 [[TMP59]], 1
 // CHECK9-NEXT:    [[TMP60:%.*]] = zext i32 [[ADD13]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP61]], align 4
 // CHECK9-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 1
@@ -6838,7 +6843,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP102:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_23]], align 4
 // CHECK9-NEXT:    [[ADD27:%.*]] = add nsw i32 [[TMP102]], 1
 // CHECK9-NEXT:    [[TMP103:%.*]] = zext i32 [[ADD27]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP104:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP104]], align 4
 // CHECK9-NEXT:    [[TMP105:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 1
@@ -6914,7 +6918,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP140:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_37]], align 4
 // CHECK9-NEXT:    [[ADD41:%.*]] = add nsw i32 [[TMP140]], 1
 // CHECK9-NEXT:    [[TMP141:%.*]] = zext i32 [[ADD41]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS42:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP142:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS42]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP142]], align 4
 // CHECK9-NEXT:    [[TMP143:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS42]], i32 0, i32 1
@@ -6999,7 +7002,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP183:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_52]], align 4
 // CHECK9-NEXT:    [[ADD56:%.*]] = add nsw i32 [[TMP183]], 1
 // CHECK9-NEXT:    [[TMP184:%.*]] = zext i32 [[ADD56]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS57:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP185:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS57]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP185]], align 4
 // CHECK9-NEXT:    [[TMP186:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS57]], i32 0, i32 1
@@ -7075,7 +7077,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP221:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_66]], align 4
 // CHECK9-NEXT:    [[ADD70:%.*]] = add nsw i32 [[TMP221]], 1
 // CHECK9-NEXT:    [[TMP222:%.*]] = zext i32 [[ADD70]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS71:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP223:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS71]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP223]], align 4
 // CHECK9-NEXT:    [[TMP224:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS71]], i32 0, i32 1
@@ -7160,7 +7161,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP264:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_81]], align 4
 // CHECK9-NEXT:    [[ADD85:%.*]] = add nsw i32 [[TMP264]], 1
 // CHECK9-NEXT:    [[TMP265:%.*]] = zext i32 [[ADD85]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS86:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP266:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS86]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP266]], align 4
 // CHECK9-NEXT:    [[TMP267:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS86]], i32 0, i32 1
@@ -8909,6 +8909,7 @@ int main() {
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [4 x ptr], align 4
@@ -8916,6 +8917,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP7:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_8:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[CH_CASTED:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[N_CASTED17:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS18:%.*]] = alloca [5 x ptr], align 4
@@ -8924,6 +8926,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP21:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_22:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_23:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[N_CASTED31:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS32:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS33:%.*]] = alloca [4 x ptr], align 4
@@ -8931,6 +8934,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP35:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_36:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_37:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS42:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[CH_CASTED45:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[N_CASTED46:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS47:%.*]] = alloca [5 x ptr], align 4
@@ -8939,6 +8943,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP50:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_51:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_52:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS57:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[N_CASTED60:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS61:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS62:%.*]] = alloca [4 x ptr], align 4
@@ -8946,6 +8951,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP64:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_65:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_66:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS71:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[CH_CASTED74:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[N_CASTED75:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS76:%.*]] = alloca [5 x ptr], align 4
@@ -8954,6 +8960,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP79:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_80:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_81:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS86:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store i32 10000, ptr [[N]], align 4
 // CHECK11-NEXT:    store i32 100, ptr [[CH]], align 4
@@ -8999,7 +9006,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK11-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -9075,7 +9081,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP59:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_9]], align 4
 // CHECK11-NEXT:    [[ADD13:%.*]] = add nsw i32 [[TMP59]], 1
 // CHECK11-NEXT:    [[TMP60:%.*]] = zext i32 [[ADD13]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP61]], align 4
 // CHECK11-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 1
@@ -9160,7 +9165,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP102:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_23]], align 4
 // CHECK11-NEXT:    [[ADD27:%.*]] = add nsw i32 [[TMP102]], 1
 // CHECK11-NEXT:    [[TMP103:%.*]] = zext i32 [[ADD27]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP104:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP104]], align 4
 // CHECK11-NEXT:    [[TMP105:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 1
@@ -9236,7 +9240,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP140:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_37]], align 4
 // CHECK11-NEXT:    [[ADD41:%.*]] = add nsw i32 [[TMP140]], 1
 // CHECK11-NEXT:    [[TMP141:%.*]] = zext i32 [[ADD41]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS42:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP142:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS42]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP142]], align 4
 // CHECK11-NEXT:    [[TMP143:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS42]], i32 0, i32 1
@@ -9321,7 +9324,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP183:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_52]], align 4
 // CHECK11-NEXT:    [[ADD56:%.*]] = add nsw i32 [[TMP183]], 1
 // CHECK11-NEXT:    [[TMP184:%.*]] = zext i32 [[ADD56]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS57:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP185:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS57]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP185]], align 4
 // CHECK11-NEXT:    [[TMP186:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS57]], i32 0, i32 1
@@ -9397,7 +9399,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP221:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_66]], align 4
 // CHECK11-NEXT:    [[ADD70:%.*]] = add nsw i32 [[TMP221]], 1
 // CHECK11-NEXT:    [[TMP222:%.*]] = zext i32 [[ADD70]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS71:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP223:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS71]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP223]], align 4
 // CHECK11-NEXT:    [[TMP224:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS71]], i32 0, i32 1
@@ -9482,7 +9483,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP264:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_81]], align 4
 // CHECK11-NEXT:    [[ADD85:%.*]] = add nsw i32 [[TMP264]], 1
 // CHECK11-NEXT:    [[TMP265:%.*]] = zext i32 [[ADD85]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS86:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP266:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS86]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP266]], align 4
 // CHECK11-NEXT:    [[TMP267:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS86]], i32 0, i32 1
@@ -11158,6 +11158,7 @@ int main() {
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [4 x ptr], align 4
@@ -11165,6 +11166,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP7:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_8:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[CH_CASTED:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[N_CASTED17:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS18:%.*]] = alloca [5 x ptr], align 4
@@ -11173,6 +11175,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP21:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_22:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_23:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[N_CASTED31:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS32:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS33:%.*]] = alloca [4 x ptr], align 4
@@ -11180,6 +11183,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP35:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_36:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_37:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS42:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[CH_CASTED45:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[N_CASTED46:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS47:%.*]] = alloca [5 x ptr], align 4
@@ -11188,6 +11192,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP50:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_51:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_52:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS57:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[N_CASTED60:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS61:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS62:%.*]] = alloca [4 x ptr], align 4
@@ -11195,6 +11200,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP64:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_65:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_66:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS71:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[CH_CASTED74:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[N_CASTED75:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS76:%.*]] = alloca [5 x ptr], align 4
@@ -11203,6 +11209,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP79:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_80:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_81:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS86:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 10000, ptr [[N]], align 4
 // CHECK11-NEXT:    store i32 100, ptr [[CH]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N]], align 4
@@ -11247,7 +11254,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK11-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -11323,7 +11329,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP59:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_9]], align 4
 // CHECK11-NEXT:    [[ADD13:%.*]] = add nsw i32 [[TMP59]], 1
 // CHECK11-NEXT:    [[TMP60:%.*]] = zext i32 [[ADD13]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP61]], align 4
 // CHECK11-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 1
@@ -11408,7 +11413,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP102:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_23]], align 4
 // CHECK11-NEXT:    [[ADD27:%.*]] = add nsw i32 [[TMP102]], 1
 // CHECK11-NEXT:    [[TMP103:%.*]] = zext i32 [[ADD27]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP104:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP104]], align 4
 // CHECK11-NEXT:    [[TMP105:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 1
@@ -11484,7 +11488,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP140:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_37]], align 4
 // CHECK11-NEXT:    [[ADD41:%.*]] = add nsw i32 [[TMP140]], 1
 // CHECK11-NEXT:    [[TMP141:%.*]] = zext i32 [[ADD41]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS42:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP142:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS42]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP142]], align 4
 // CHECK11-NEXT:    [[TMP143:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS42]], i32 0, i32 1
@@ -11569,7 +11572,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP183:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_52]], align 4
 // CHECK11-NEXT:    [[ADD56:%.*]] = add nsw i32 [[TMP183]], 1
 // CHECK11-NEXT:    [[TMP184:%.*]] = zext i32 [[ADD56]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS57:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP185:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS57]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP185]], align 4
 // CHECK11-NEXT:    [[TMP186:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS57]], i32 0, i32 1
@@ -11645,7 +11647,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP221:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_66]], align 4
 // CHECK11-NEXT:    [[ADD70:%.*]] = add nsw i32 [[TMP221]], 1
 // CHECK11-NEXT:    [[TMP222:%.*]] = zext i32 [[ADD70]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS71:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP223:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS71]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP223]], align 4
 // CHECK11-NEXT:    [[TMP224:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS71]], i32 0, i32 1
@@ -11730,7 +11731,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP264:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_81]], align 4
 // CHECK11-NEXT:    [[ADD85:%.*]] = add nsw i32 [[TMP264]], 1
 // CHECK11-NEXT:    [[TMP265:%.*]] = zext i32 [[ADD85]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS86:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP266:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS86]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP266]], align 4
 // CHECK11-NEXT:    [[TMP267:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS86]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_parallel_for_firstprivate_codegen.cpp b/clang/test/OpenMP/distribute_parallel_for_firstprivate_codegen.cpp
index dfeb1898823f5..f46a6bf1ef0b1 100644
--- a/clang/test/OpenMP/distribute_parallel_for_firstprivate_codegen.cpp
+++ b/clang/test/OpenMP/distribute_parallel_for_firstprivate_codegen.cpp
@@ -784,6 +784,7 @@ int main() {
 // CHECK8-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK8-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK8-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK8-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK8-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK8-NEXT:    store ptr [[G]], ptr [[G1]], align 8
 // CHECK8-NEXT:    call void @_ZN1SIfEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
@@ -837,7 +838,6 @@ int main() {
 // CHECK8-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK8-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK8-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK8-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK8-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK8-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK8-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1221,6 +1221,7 @@ int main() {
 // CHECK8-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK8-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK8-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK8-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK8-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK8-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK8-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -1263,7 +1264,6 @@ int main() {
 // CHECK8-NEXT:    store ptr null, ptr [[TMP17]], align 8
 // CHECK8-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK8-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK8-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK8-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK8-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK8-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1710,6 +1710,7 @@ int main() {
 // CHECK10-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK10-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK10-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK10-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK10-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK10-NEXT:    store ptr [[G]], ptr [[G1]], align 4
 // CHECK10-NEXT:    call void @_ZN1SIfEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
@@ -1763,7 +1764,6 @@ int main() {
 // CHECK10-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK10-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK10-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK10-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK10-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK10-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK10-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2141,6 +2141,7 @@ int main() {
 // CHECK10-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK10-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK10-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK10-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK10-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK10-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK10-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -2183,7 +2184,6 @@ int main() {
 // CHECK10-NEXT:    store ptr null, ptr [[TMP17]], align 4
 // CHECK10-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK10-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK10-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK10-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK10-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK10-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_parallel_for_if_codegen.cpp b/clang/test/OpenMP/distribute_parallel_for_if_codegen.cpp
index 7ca6b251f3e0a..f8612e1b3ccc4 100644
--- a/clang/test/OpenMP/distribute_parallel_for_if_codegen.cpp
+++ b/clang/test/OpenMP/distribute_parallel_for_if_codegen.cpp
@@ -124,8 +124,9 @@ int main() {
 // CHECK1-SAME: () #[[ATTR0:[0-9]+]] {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -159,7 +160,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9gtid_testv_l47() #[[ATTR2:[0-9]+]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -481,14 +481,16 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP5:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -522,7 +524,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}_main_l85() #[[ATTR2]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -567,7 +568,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP34]], align 8
 // CHECK1-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP37]], align 4
 // CHECK1-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -1049,14 +1049,16 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[ARG_ADDR:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP5:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[ARG]], ptr [[ARG_ADDR]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1090,7 +1092,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIiEiT__l63() #[[ATTR2]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -1135,7 +1136,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP34]], align 8
 // CHECK1-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP37]], align 4
 // CHECK1-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_parallel_for_lastprivate_codegen.cpp b/clang/test/OpenMP/distribute_parallel_for_lastprivate_codegen.cpp
index fdc1c68b5785c..b8c3ee2bd3965 100644
--- a/clang/test/OpenMP/distribute_parallel_for_lastprivate_codegen.cpp
+++ b/clang/test/OpenMP/distribute_parallel_for_lastprivate_codegen.cpp
@@ -759,6 +759,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store ptr [[G]], ptr [[G1]], align 8
@@ -813,7 +814,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1233,6 +1233,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK9-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -1275,7 +1276,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP17]], align 8
 // CHECK9-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK9-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1758,6 +1758,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store ptr [[G]], ptr [[G1]], align 4
@@ -1812,7 +1813,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2226,6 +2226,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK11-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -2268,7 +2269,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP17]], align 4
 // CHECK11-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK11-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_parallel_for_num_threads_codegen.cpp b/clang/test/OpenMP/distribute_parallel_for_num_threads_codegen.cpp
index 9c874b81f2b71..c97b78de3fd61 100644
--- a/clang/test/OpenMP/distribute_parallel_for_num_threads_codegen.cpp
+++ b/clang/test/OpenMP/distribute_parallel_for_num_threads_codegen.cpp
@@ -102,18 +102,19 @@ int main() {
 // CHECK1-NEXT:    [[EXN_SLOT:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[EHSELECTOR_SLOT:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    call void @_ZN1SC1El(ptr noundef nonnull align 8 dereferenceable(24) [[S]], i64 noundef 0)
 // CHECK1-NEXT:    [[CALL:%.*]] = invoke noundef signext i8 @_ZN1ScvcEv(ptr noundef nonnull align 8 dereferenceable(24) [[S]])
 // CHECK1-NEXT:    to label [[INVOKE_CONT:%.*]] unwind label [[LPAD:%.*]]
 // CHECK1:       invoke.cont:
 // CHECK1-NEXT:    store i8 [[CALL]], ptr [[A]], align 1
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -167,7 +168,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK1-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -561,8 +561,9 @@ int main() {
 // CHECK1-SAME: () #[[ATTR7:[0-9]+]] comdat {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -596,7 +597,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIcLi5EEiv_l52() #[[ATTR6]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -637,8 +637,9 @@ int main() {
 // CHECK1-SAME: () #[[ATTR7]] comdat {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -672,7 +673,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainI1SLi1EEiv_l52() #[[ATTR6]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -1361,18 +1361,19 @@ int main() {
 // CHECK5-NEXT:    [[EXN_SLOT:%.*]] = alloca ptr, align 8
 // CHECK5-NEXT:    [[EHSELECTOR_SLOT:%.*]] = alloca i32, align 4
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK5-NEXT:    call void @_ZN1SC1El(ptr noundef nonnull align 8 dereferenceable(24) [[S]], i64 noundef 0)
 // CHECK5-NEXT:    [[CALL:%.*]] = invoke noundef signext i8 @_ZN1ScvcEv(ptr noundef nonnull align 8 dereferenceable(24) [[S]])
 // CHECK5-NEXT:    to label [[INVOKE_CONT:%.*]] unwind label [[LPAD:%.*]]
 // CHECK5:       invoke.cont:
 // CHECK5-NEXT:    store i8 [[CALL]], ptr [[A]], align 1
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK5-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1426,7 +1427,6 @@ int main() {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK5-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK5-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -1820,8 +1820,9 @@ int main() {
 // CHECK5-SAME: () #[[ATTR7:[0-9]+]] comdat {
 // CHECK5-NEXT:  entry:
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK5-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1855,7 +1856,6 @@ int main() {
 // CHECK5-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIcLi5EEiv_l52() #[[ATTR6]]
 // CHECK5-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK5:       omp_offload.cont:
-// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK5-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -1896,8 +1896,9 @@ int main() {
 // CHECK5-SAME: () #[[ATTR7]] comdat {
 // CHECK5-NEXT:  entry:
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK5-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1931,7 +1932,6 @@ int main() {
 // CHECK5-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainI1SLi1EEiv_l52() #[[ATTR6]]
 // CHECK5-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK5:       omp_offload.cont:
-// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK5-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -2620,18 +2620,19 @@ int main() {
 // CHECK9-NEXT:    [[EXN_SLOT:%.*]] = alloca ptr, align 8
 // CHECK9-NEXT:    [[EHSELECTOR_SLOT:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    call void @_ZN1SC1El(ptr noundef nonnull align 8 dereferenceable(24) [[S]], i64 noundef 0)
 // CHECK9-NEXT:    [[CALL:%.*]] = invoke noundef i8 @_ZN1ScvcEv(ptr noundef nonnull align 8 dereferenceable(24) [[S]])
 // CHECK9-NEXT:    to label [[INVOKE_CONT:%.*]] unwind label [[LPAD:%.*]]
 // CHECK9:       invoke.cont:
 // CHECK9-NEXT:    store i8 [[CALL]], ptr [[A]], align 1
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2685,7 +2686,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -3079,8 +3079,9 @@ int main() {
 // CHECK9-SAME: () #[[ATTR7:[0-9]+]] comdat {
 // CHECK9-NEXT:  entry:
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3114,7 +3115,6 @@ int main() {
 // CHECK9-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIcLi5EEiv_l52() #[[ATTR6]]
 // CHECK9-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK9:       omp_offload.cont:
-// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK9-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -3155,8 +3155,9 @@ int main() {
 // CHECK9-SAME: () #[[ATTR7]] comdat {
 // CHECK9-NEXT:  entry:
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3190,7 +3191,6 @@ int main() {
 // CHECK9-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainI1SLi1EEiv_l52() #[[ATTR6]]
 // CHECK9-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK9:       omp_offload.cont:
-// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK9-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -3879,18 +3879,19 @@ int main() {
 // CHECK13-NEXT:    [[EXN_SLOT:%.*]] = alloca ptr, align 8
 // CHECK13-NEXT:    [[EHSELECTOR_SLOT:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK13-NEXT:    call void @_ZN1SC1El(ptr noundef nonnull align 8 dereferenceable(24) [[S]], i64 noundef 0)
 // CHECK13-NEXT:    [[CALL:%.*]] = invoke noundef i8 @_ZN1ScvcEv(ptr noundef nonnull align 8 dereferenceable(24) [[S]])
 // CHECK13-NEXT:    to label [[INVOKE_CONT:%.*]] unwind label [[LPAD:%.*]]
 // CHECK13:       invoke.cont:
 // CHECK13-NEXT:    store i8 [[CALL]], ptr [[A]], align 1
-// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK13-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3944,7 +3945,6 @@ int main() {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK13-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK13-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -4338,8 +4338,9 @@ int main() {
 // CHECK13-SAME: () #[[ATTR7:[0-9]+]] comdat {
 // CHECK13-NEXT:  entry:
 // CHECK13-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK13-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK13-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK13-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4373,7 +4374,6 @@ int main() {
 // CHECK13-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIcLi5EEiv_l52() #[[ATTR6]]
 // CHECK13-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK13:       omp_offload.cont:
-// CHECK13-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK13-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -4414,8 +4414,9 @@ int main() {
 // CHECK13-SAME: () #[[ATTR7]] comdat {
 // CHECK13-NEXT:  entry:
 // CHECK13-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK13-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK13-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK13-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4449,7 +4450,6 @@ int main() {
 // CHECK13-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainI1SLi1EEiv_l52() #[[ATTR6]]
 // CHECK13-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK13:       omp_offload.cont:
-// CHECK13-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK13-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_parallel_for_private_codegen.cpp b/clang/test/OpenMP/distribute_parallel_for_private_codegen.cpp
index 9a606bc2f4581..f62e389a7e298 100644
--- a/clang/test/OpenMP/distribute_parallel_for_private_codegen.cpp
+++ b/clang/test/OpenMP/distribute_parallel_for_private_codegen.cpp
@@ -522,6 +522,7 @@ int main() {
 // CHECK9-NEXT:    [[VAR:%.*]] = alloca ptr, align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store ptr [[G]], ptr [[G1]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIfEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
@@ -533,7 +534,6 @@ int main() {
 // CHECK9-NEXT:    call void @_ZN1SIfEC1Ef(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], float noundef 2.000000e+00)
 // CHECK9-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 8
 // CHECK9-NEXT:    store ptr undef, ptr [[_TMP1]], align 8
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -845,6 +845,7 @@ int main() {
 // CHECK9-NEXT:    [[VAR:%.*]] = alloca ptr, align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK9-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -854,7 +855,6 @@ int main() {
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef signext 2)
 // CHECK9-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 8
 // CHECK9-NEXT:    store ptr undef, ptr [[_TMP1]], align 8
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1240,6 +1240,7 @@ int main() {
 // CHECK11-NEXT:    [[VAR:%.*]] = alloca ptr, align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store ptr [[G]], ptr [[G1]], align 4
 // CHECK11-NEXT:    call void @_ZN1SIfEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
@@ -1251,7 +1252,6 @@ int main() {
 // CHECK11-NEXT:    call void @_ZN1SIfEC1Ef(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], float noundef 2.000000e+00)
 // CHECK11-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 4
 // CHECK11-NEXT:    store ptr undef, ptr [[_TMP1]], align 4
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1557,6 +1557,7 @@ int main() {
 // CHECK11-NEXT:    [[VAR:%.*]] = alloca ptr, align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK11-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1566,7 +1567,6 @@ int main() {
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef 2)
 // CHECK11-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 4
 // CHECK11-NEXT:    store ptr undef, ptr [[_TMP1]], align 4
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_parallel_for_proc_bind_codegen.cpp b/clang/test/OpenMP/distribute_parallel_for_proc_bind_codegen.cpp
index 903dbb6de2880..9e84ed68e0b8d 100644
--- a/clang/test/OpenMP/distribute_parallel_for_proc_bind_codegen.cpp
+++ b/clang/test/OpenMP/distribute_parallel_for_proc_bind_codegen.cpp
@@ -58,9 +58,10 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -94,7 +95,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}_main_l37() #[[ATTR2:[0-9]+]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_parallel_for_simd_codegen.cpp b/clang/test/OpenMP/distribute_parallel_for_simd_codegen.cpp
index a0819493d2cee..529ce78ed6dac 100644
--- a/clang/test/OpenMP/distribute_parallel_for_simd_codegen.cpp
+++ b/clang/test/OpenMP/distribute_parallel_for_simd_codegen.cpp
@@ -4678,6 +4678,7 @@ int main() {
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [4 x ptr], align 8
@@ -4685,6 +4686,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP7:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_8:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[CH_CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[N_CASTED17:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS18:%.*]] = alloca [5 x ptr], align 8
@@ -4693,6 +4695,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP21:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_22:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_23:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[N_CASTED31:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS32:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS33:%.*]] = alloca [4 x ptr], align 8
@@ -4700,6 +4703,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP35:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_36:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_37:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS42:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[CH_CASTED45:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[N_CASTED46:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS47:%.*]] = alloca [5 x ptr], align 8
@@ -4708,6 +4712,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP50:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_51:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_52:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS57:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[N_CASTED60:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS61:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS62:%.*]] = alloca [4 x ptr], align 8
@@ -4715,6 +4720,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP64:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_65:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_66:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS71:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[CH_CASTED74:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[N_CASTED75:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS76:%.*]] = alloca [5 x ptr], align 8
@@ -4723,6 +4729,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP79:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_80:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_81:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS86:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store i32 10000, ptr [[N]], align 4
 // CHECK9-NEXT:    store i32 100, ptr [[CH]], align 4
@@ -4768,7 +4775,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK9-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4844,7 +4850,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP59:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_9]], align 4
 // CHECK9-NEXT:    [[ADD13:%.*]] = add nsw i32 [[TMP59]], 1
 // CHECK9-NEXT:    [[TMP60:%.*]] = zext i32 [[ADD13]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP61]], align 4
 // CHECK9-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 1
@@ -4929,7 +4934,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP102:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_23]], align 4
 // CHECK9-NEXT:    [[ADD27:%.*]] = add nsw i32 [[TMP102]], 1
 // CHECK9-NEXT:    [[TMP103:%.*]] = zext i32 [[ADD27]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP104:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP104]], align 4
 // CHECK9-NEXT:    [[TMP105:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 1
@@ -5005,7 +5009,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP140:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_37]], align 4
 // CHECK9-NEXT:    [[ADD41:%.*]] = add nsw i32 [[TMP140]], 1
 // CHECK9-NEXT:    [[TMP141:%.*]] = zext i32 [[ADD41]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS42:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP142:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS42]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP142]], align 4
 // CHECK9-NEXT:    [[TMP143:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS42]], i32 0, i32 1
@@ -5090,7 +5093,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP183:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_52]], align 4
 // CHECK9-NEXT:    [[ADD56:%.*]] = add nsw i32 [[TMP183]], 1
 // CHECK9-NEXT:    [[TMP184:%.*]] = zext i32 [[ADD56]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS57:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP185:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS57]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP185]], align 4
 // CHECK9-NEXT:    [[TMP186:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS57]], i32 0, i32 1
@@ -5166,7 +5168,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP221:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_66]], align 4
 // CHECK9-NEXT:    [[ADD70:%.*]] = add nsw i32 [[TMP221]], 1
 // CHECK9-NEXT:    [[TMP222:%.*]] = zext i32 [[ADD70]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS71:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP223:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS71]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP223]], align 4
 // CHECK9-NEXT:    [[TMP224:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS71]], i32 0, i32 1
@@ -5251,7 +5252,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP264:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_81]], align 4
 // CHECK9-NEXT:    [[ADD85:%.*]] = add nsw i32 [[TMP264]], 1
 // CHECK9-NEXT:    [[TMP265:%.*]] = zext i32 [[ADD85]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS86:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP266:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS86]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP266]], align 4
 // CHECK9-NEXT:    [[TMP267:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS86]], i32 0, i32 1
@@ -7146,6 +7146,7 @@ int main() {
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [4 x ptr], align 8
@@ -7153,6 +7154,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP7:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_8:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[CH_CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[N_CASTED17:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS18:%.*]] = alloca [5 x ptr], align 8
@@ -7161,6 +7163,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP21:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_22:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_23:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[N_CASTED31:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS32:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS33:%.*]] = alloca [4 x ptr], align 8
@@ -7168,6 +7171,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP35:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_36:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_37:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS42:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[CH_CASTED45:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[N_CASTED46:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS47:%.*]] = alloca [5 x ptr], align 8
@@ -7176,6 +7180,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP50:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_51:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_52:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS57:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[N_CASTED60:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS61:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS62:%.*]] = alloca [4 x ptr], align 8
@@ -7183,6 +7188,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP64:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_65:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_66:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS71:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[CH_CASTED74:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[N_CASTED75:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS76:%.*]] = alloca [5 x ptr], align 8
@@ -7191,6 +7197,7 @@ int main() {
 // CHECK9-NEXT:    [[_TMP79:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_80:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_81:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS86:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 10000, ptr [[N]], align 4
 // CHECK9-NEXT:    store i32 100, ptr [[CH]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N]], align 4
@@ -7235,7 +7242,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK9-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -7311,7 +7317,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP59:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_9]], align 4
 // CHECK9-NEXT:    [[ADD13:%.*]] = add nsw i32 [[TMP59]], 1
 // CHECK9-NEXT:    [[TMP60:%.*]] = zext i32 [[ADD13]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP61]], align 4
 // CHECK9-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 1
@@ -7396,7 +7401,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP102:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_23]], align 4
 // CHECK9-NEXT:    [[ADD27:%.*]] = add nsw i32 [[TMP102]], 1
 // CHECK9-NEXT:    [[TMP103:%.*]] = zext i32 [[ADD27]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP104:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP104]], align 4
 // CHECK9-NEXT:    [[TMP105:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 1
@@ -7472,7 +7476,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP140:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_37]], align 4
 // CHECK9-NEXT:    [[ADD41:%.*]] = add nsw i32 [[TMP140]], 1
 // CHECK9-NEXT:    [[TMP141:%.*]] = zext i32 [[ADD41]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS42:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP142:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS42]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP142]], align 4
 // CHECK9-NEXT:    [[TMP143:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS42]], i32 0, i32 1
@@ -7557,7 +7560,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP183:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_52]], align 4
 // CHECK9-NEXT:    [[ADD56:%.*]] = add nsw i32 [[TMP183]], 1
 // CHECK9-NEXT:    [[TMP184:%.*]] = zext i32 [[ADD56]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS57:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP185:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS57]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP185]], align 4
 // CHECK9-NEXT:    [[TMP186:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS57]], i32 0, i32 1
@@ -7633,7 +7635,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP221:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_66]], align 4
 // CHECK9-NEXT:    [[ADD70:%.*]] = add nsw i32 [[TMP221]], 1
 // CHECK9-NEXT:    [[TMP222:%.*]] = zext i32 [[ADD70]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS71:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP223:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS71]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP223]], align 4
 // CHECK9-NEXT:    [[TMP224:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS71]], i32 0, i32 1
@@ -7718,7 +7719,6 @@ int main() {
 // CHECK9-NEXT:    [[TMP264:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_81]], align 4
 // CHECK9-NEXT:    [[ADD85:%.*]] = add nsw i32 [[TMP264]], 1
 // CHECK9-NEXT:    [[TMP265:%.*]] = zext i32 [[ADD85]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS86:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP266:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS86]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP266]], align 4
 // CHECK9-NEXT:    [[TMP267:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS86]], i32 0, i32 1
@@ -9620,6 +9620,7 @@ int main() {
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [4 x ptr], align 4
@@ -9627,6 +9628,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP7:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_8:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[CH_CASTED:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[N_CASTED17:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS18:%.*]] = alloca [5 x ptr], align 4
@@ -9635,6 +9637,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP21:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_22:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_23:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[N_CASTED31:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS32:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS33:%.*]] = alloca [4 x ptr], align 4
@@ -9642,6 +9645,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP35:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_36:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_37:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS42:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[CH_CASTED45:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[N_CASTED46:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS47:%.*]] = alloca [5 x ptr], align 4
@@ -9650,6 +9654,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP50:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_51:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_52:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS57:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[N_CASTED60:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS61:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS62:%.*]] = alloca [4 x ptr], align 4
@@ -9657,6 +9662,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP64:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_65:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_66:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS71:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[CH_CASTED74:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[N_CASTED75:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS76:%.*]] = alloca [5 x ptr], align 4
@@ -9665,6 +9671,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP79:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_80:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_81:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS86:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store i32 10000, ptr [[N]], align 4
 // CHECK11-NEXT:    store i32 100, ptr [[CH]], align 4
@@ -9710,7 +9717,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK11-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -9786,7 +9792,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP59:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_9]], align 4
 // CHECK11-NEXT:    [[ADD13:%.*]] = add nsw i32 [[TMP59]], 1
 // CHECK11-NEXT:    [[TMP60:%.*]] = zext i32 [[ADD13]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP61]], align 4
 // CHECK11-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 1
@@ -9871,7 +9876,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP102:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_23]], align 4
 // CHECK11-NEXT:    [[ADD27:%.*]] = add nsw i32 [[TMP102]], 1
 // CHECK11-NEXT:    [[TMP103:%.*]] = zext i32 [[ADD27]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP104:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP104]], align 4
 // CHECK11-NEXT:    [[TMP105:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 1
@@ -9947,7 +9951,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP140:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_37]], align 4
 // CHECK11-NEXT:    [[ADD41:%.*]] = add nsw i32 [[TMP140]], 1
 // CHECK11-NEXT:    [[TMP141:%.*]] = zext i32 [[ADD41]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS42:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP142:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS42]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP142]], align 4
 // CHECK11-NEXT:    [[TMP143:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS42]], i32 0, i32 1
@@ -10032,7 +10035,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP183:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_52]], align 4
 // CHECK11-NEXT:    [[ADD56:%.*]] = add nsw i32 [[TMP183]], 1
 // CHECK11-NEXT:    [[TMP184:%.*]] = zext i32 [[ADD56]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS57:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP185:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS57]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP185]], align 4
 // CHECK11-NEXT:    [[TMP186:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS57]], i32 0, i32 1
@@ -10108,7 +10110,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP221:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_66]], align 4
 // CHECK11-NEXT:    [[ADD70:%.*]] = add nsw i32 [[TMP221]], 1
 // CHECK11-NEXT:    [[TMP222:%.*]] = zext i32 [[ADD70]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS71:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP223:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS71]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP223]], align 4
 // CHECK11-NEXT:    [[TMP224:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS71]], i32 0, i32 1
@@ -10193,7 +10194,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP264:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_81]], align 4
 // CHECK11-NEXT:    [[ADD85:%.*]] = add nsw i32 [[TMP264]], 1
 // CHECK11-NEXT:    [[TMP265:%.*]] = zext i32 [[ADD85]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS86:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP266:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS86]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP266]], align 4
 // CHECK11-NEXT:    [[TMP267:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS86]], i32 0, i32 1
@@ -12037,6 +12037,7 @@ int main() {
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [4 x ptr], align 4
@@ -12044,6 +12045,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP7:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_8:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[CH_CASTED:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[N_CASTED17:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS18:%.*]] = alloca [5 x ptr], align 4
@@ -12052,6 +12054,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP21:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_22:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_23:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[N_CASTED31:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS32:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS33:%.*]] = alloca [4 x ptr], align 4
@@ -12059,6 +12062,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP35:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_36:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_37:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS42:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[CH_CASTED45:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[N_CASTED46:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS47:%.*]] = alloca [5 x ptr], align 4
@@ -12067,6 +12071,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP50:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_51:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_52:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS57:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[N_CASTED60:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS61:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS62:%.*]] = alloca [4 x ptr], align 4
@@ -12074,6 +12079,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP64:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_65:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_66:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS71:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[CH_CASTED74:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[N_CASTED75:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS76:%.*]] = alloca [5 x ptr], align 4
@@ -12082,6 +12088,7 @@ int main() {
 // CHECK11-NEXT:    [[_TMP79:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_80:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_81:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS86:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 10000, ptr [[N]], align 4
 // CHECK11-NEXT:    store i32 100, ptr [[CH]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N]], align 4
@@ -12126,7 +12133,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK11-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -12202,7 +12208,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP59:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_9]], align 4
 // CHECK11-NEXT:    [[ADD13:%.*]] = add nsw i32 [[TMP59]], 1
 // CHECK11-NEXT:    [[TMP60:%.*]] = zext i32 [[ADD13]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP61]], align 4
 // CHECK11-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 1
@@ -12287,7 +12292,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP102:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_23]], align 4
 // CHECK11-NEXT:    [[ADD27:%.*]] = add nsw i32 [[TMP102]], 1
 // CHECK11-NEXT:    [[TMP103:%.*]] = zext i32 [[ADD27]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP104:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP104]], align 4
 // CHECK11-NEXT:    [[TMP105:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 1
@@ -12363,7 +12367,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP140:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_37]], align 4
 // CHECK11-NEXT:    [[ADD41:%.*]] = add nsw i32 [[TMP140]], 1
 // CHECK11-NEXT:    [[TMP141:%.*]] = zext i32 [[ADD41]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS42:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP142:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS42]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP142]], align 4
 // CHECK11-NEXT:    [[TMP143:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS42]], i32 0, i32 1
@@ -12448,7 +12451,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP183:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_52]], align 4
 // CHECK11-NEXT:    [[ADD56:%.*]] = add nsw i32 [[TMP183]], 1
 // CHECK11-NEXT:    [[TMP184:%.*]] = zext i32 [[ADD56]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS57:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP185:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS57]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP185]], align 4
 // CHECK11-NEXT:    [[TMP186:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS57]], i32 0, i32 1
@@ -12524,7 +12526,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP221:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_66]], align 4
 // CHECK11-NEXT:    [[ADD70:%.*]] = add nsw i32 [[TMP221]], 1
 // CHECK11-NEXT:    [[TMP222:%.*]] = zext i32 [[ADD70]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS71:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP223:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS71]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP223]], align 4
 // CHECK11-NEXT:    [[TMP224:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS71]], i32 0, i32 1
@@ -12609,7 +12610,6 @@ int main() {
 // CHECK11-NEXT:    [[TMP264:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_81]], align 4
 // CHECK11-NEXT:    [[ADD85:%.*]] = add nsw i32 [[TMP264]], 1
 // CHECK11-NEXT:    [[TMP265:%.*]] = zext i32 [[ADD85]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS86:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP266:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS86]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP266]], align 4
 // CHECK11-NEXT:    [[TMP267:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS86]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_parallel_for_simd_firstprivate_codegen.cpp b/clang/test/OpenMP/distribute_parallel_for_simd_firstprivate_codegen.cpp
index af7cddc3114c1..69890ffd7fec5 100644
--- a/clang/test/OpenMP/distribute_parallel_for_simd_firstprivate_codegen.cpp
+++ b/clang/test/OpenMP/distribute_parallel_for_simd_firstprivate_codegen.cpp
@@ -847,6 +847,7 @@ int main() {
 // CHECK8-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK8-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK8-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK8-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK8-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK8-NEXT:    store ptr [[G]], ptr [[G1]], align 8
 // CHECK8-NEXT:    call void @_ZN1SIfEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
@@ -900,7 +901,6 @@ int main() {
 // CHECK8-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK8-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK8-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK8-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK8-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK8-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK8-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1298,6 +1298,7 @@ int main() {
 // CHECK8-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK8-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK8-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK8-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK8-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK8-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK8-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -1340,7 +1341,6 @@ int main() {
 // CHECK8-NEXT:    store ptr null, ptr [[TMP17]], align 8
 // CHECK8-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK8-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK8-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK8-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK8-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK8-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1801,6 +1801,7 @@ int main() {
 // CHECK10-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK10-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK10-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK10-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK10-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK10-NEXT:    store ptr [[G]], ptr [[G1]], align 4
 // CHECK10-NEXT:    call void @_ZN1SIfEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
@@ -1854,7 +1855,6 @@ int main() {
 // CHECK10-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK10-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK10-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK10-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK10-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK10-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK10-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2246,6 +2246,7 @@ int main() {
 // CHECK10-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK10-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK10-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK10-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK10-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK10-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK10-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -2288,7 +2289,6 @@ int main() {
 // CHECK10-NEXT:    store ptr null, ptr [[TMP17]], align 4
 // CHECK10-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK10-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK10-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK10-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK10-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK10-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_parallel_for_simd_if_codegen.cpp b/clang/test/OpenMP/distribute_parallel_for_simd_if_codegen.cpp
index ad23bf5bdc640..27e0a19095984 100644
--- a/clang/test/OpenMP/distribute_parallel_for_simd_if_codegen.cpp
+++ b/clang/test/OpenMP/distribute_parallel_for_simd_if_codegen.cpp
@@ -121,8 +121,9 @@ int main() {
 // CHECK1-SAME: () #[[ATTR0:[0-9]+]] {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -156,7 +157,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9gtid_testv_l43() #[[ATTR2:[0-9]+]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -506,14 +506,16 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP5:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -547,7 +549,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}_main_l81() #[[ATTR2]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -592,7 +593,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP34]], align 8
 // CHECK1-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP37]], align 4
 // CHECK1-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -1116,14 +1116,16 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[ARG_ADDR:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP5:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[ARG]], ptr [[ARG_ADDR]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1157,7 +1159,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIiEiT__l59() #[[ATTR2]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -1202,7 +1203,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP34]], align 8
 // CHECK1-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP37]], align 4
 // CHECK1-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -1730,8 +1730,9 @@ int main() {
 // CHECK3-SAME: () #[[ATTR0:[0-9]+]] {
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK3-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK3-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1765,7 +1766,6 @@ int main() {
 // CHECK3-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9gtid_testv_l43() #[[ATTR2:[0-9]+]]
 // CHECK3-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK3:       omp_offload.cont:
-// CHECK3-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -2115,14 +2115,16 @@ int main() {
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[_TMP5:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2156,7 +2158,6 @@ int main() {
 // CHECK3-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}_main_l81() #[[ATTR2]]
 // CHECK3-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK3:       omp_offload.cont:
-// CHECK3-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -2201,7 +2202,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP34]], align 8
 // CHECK3-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP37]], align 4
 // CHECK3-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -2960,14 +2960,16 @@ int main() {
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[ARG_ADDR:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[_TMP5:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 [[ARG]], ptr [[ARG_ADDR]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3001,7 +3003,6 @@ int main() {
 // CHECK3-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIiEiT__l59() #[[ATTR2]]
 // CHECK3-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK3:       omp_offload.cont:
-// CHECK3-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -3046,7 +3047,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP34]], align 8
 // CHECK3-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP37]], align 4
 // CHECK3-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -4150,8 +4150,9 @@ int main() {
 // CHECK9-SAME: () #[[ATTR0:[0-9]+]] {
 // CHECK9-NEXT:  entry:
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4185,7 +4186,6 @@ int main() {
 // CHECK9-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9gtid_testv_l43() #[[ATTR2:[0-9]+]]
 // CHECK9-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK9:       omp_offload.cont:
-// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK9-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -4535,14 +4535,16 @@ int main() {
 // CHECK9-NEXT:  entry:
 // CHECK9-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP5:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4576,7 +4578,6 @@ int main() {
 // CHECK9-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}_main_l81() #[[ATTR2]]
 // CHECK9-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK9:       omp_offload.cont:
-// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK9-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -4621,7 +4622,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP34]], align 8
 // CHECK9-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP37]], align 4
 // CHECK9-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -5145,14 +5145,16 @@ int main() {
 // CHECK9-NEXT:  entry:
 // CHECK9-NEXT:    [[ARG_ADDR:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP5:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 [[ARG]], ptr [[ARG_ADDR]], align 4
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5186,7 +5188,6 @@ int main() {
 // CHECK9-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIiEiT__l59() #[[ATTR2]]
 // CHECK9-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK9:       omp_offload.cont:
-// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK9-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -5231,7 +5232,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP34]], align 8
 // CHECK9-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP37]], align 4
 // CHECK9-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -5759,8 +5759,9 @@ int main() {
 // CHECK11-SAME: () #[[ATTR0:[0-9]+]] {
 // CHECK11-NEXT:  entry:
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5794,7 +5795,6 @@ int main() {
 // CHECK11-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9gtid_testv_l43() #[[ATTR2:[0-9]+]]
 // CHECK11-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK11:       omp_offload.cont:
-// CHECK11-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK11-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -6144,14 +6144,16 @@ int main() {
 // CHECK11-NEXT:  entry:
 // CHECK11-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[_TMP5:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -6185,7 +6187,6 @@ int main() {
 // CHECK11-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}_main_l81() #[[ATTR2]]
 // CHECK11-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK11:       omp_offload.cont:
-// CHECK11-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK11-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -6230,7 +6231,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP34]], align 8
 // CHECK11-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP37]], align 4
 // CHECK11-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -6989,14 +6989,16 @@ int main() {
 // CHECK11-NEXT:  entry:
 // CHECK11-NEXT:    [[ARG_ADDR:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[_TMP5:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 [[ARG]], ptr [[ARG_ADDR]], align 4
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -7030,7 +7032,6 @@ int main() {
 // CHECK11-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIiEiT__l59() #[[ATTR2]]
 // CHECK11-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK11:       omp_offload.cont:
-// CHECK11-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK11-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -7075,7 +7076,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP34]], align 8
 // CHECK11-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP37]], align 4
 // CHECK11-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_parallel_for_simd_lastprivate_codegen.cpp b/clang/test/OpenMP/distribute_parallel_for_simd_lastprivate_codegen.cpp
index 78e461c7ce40e..b9ab7ace7968e 100644
--- a/clang/test/OpenMP/distribute_parallel_for_simd_lastprivate_codegen.cpp
+++ b/clang/test/OpenMP/distribute_parallel_for_simd_lastprivate_codegen.cpp
@@ -826,6 +826,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store ptr [[G]], ptr [[G1]], align 8
@@ -880,7 +881,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1314,6 +1314,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK9-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -1356,7 +1357,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP17]], align 8
 // CHECK9-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK9-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1853,6 +1853,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store ptr [[G]], ptr [[G1]], align 4
@@ -1907,7 +1908,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2335,6 +2335,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK11-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -2377,7 +2378,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP17]], align 4
 // CHECK11-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK11-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_parallel_for_simd_num_threads_codegen.cpp b/clang/test/OpenMP/distribute_parallel_for_simd_num_threads_codegen.cpp
index c2d6814d62ee2..de369bf417459 100644
--- a/clang/test/OpenMP/distribute_parallel_for_simd_num_threads_codegen.cpp
+++ b/clang/test/OpenMP/distribute_parallel_for_simd_num_threads_codegen.cpp
@@ -102,18 +102,19 @@ int main() {
 // CHECK1-NEXT:    [[EXN_SLOT:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[EHSELECTOR_SLOT:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    call void @_ZN1SC1El(ptr nonnull align 8 dereferenceable(24) [[S]], i64 0)
 // CHECK1-NEXT:    [[CALL:%.*]] = invoke signext i8 @_ZN1ScvcEv(ptr nonnull align 8 dereferenceable(24) [[S]])
 // CHECK1-NEXT:    to label [[INVOKE_CONT:%.*]] unwind label [[LPAD:%.*]]
 // CHECK1:       invoke.cont:
 // CHECK1-NEXT:    store i8 [[CALL]], ptr [[A]], align 1
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -167,7 +168,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK1-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -589,8 +589,9 @@ int main() {
 // CHECK1-SAME: () #[[ATTR7:[0-9]+]] comdat {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -624,7 +625,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIcLi5EEiv_l52() #[[ATTR6]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -665,8 +665,9 @@ int main() {
 // CHECK1-SAME: () #[[ATTR7]] comdat {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -700,7 +701,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainI1SLi1EEiv_l52() #[[ATTR6]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -1784,18 +1784,19 @@ int main() {
 // CHECK5-NEXT:    [[EXN_SLOT:%.*]] = alloca ptr, align 8
 // CHECK5-NEXT:    [[EHSELECTOR_SLOT:%.*]] = alloca i32, align 4
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK5-NEXT:    call void @_ZN1SC1El(ptr nonnull align 8 dereferenceable(24) [[S]], i64 0)
 // CHECK5-NEXT:    [[CALL:%.*]] = invoke signext i8 @_ZN1ScvcEv(ptr nonnull align 8 dereferenceable(24) [[S]])
 // CHECK5-NEXT:    to label [[INVOKE_CONT:%.*]] unwind label [[LPAD:%.*]]
 // CHECK5:       invoke.cont:
 // CHECK5-NEXT:    store i8 [[CALL]], ptr [[A]], align 1
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK5-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1849,7 +1850,6 @@ int main() {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK5-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK5-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -2271,8 +2271,9 @@ int main() {
 // CHECK5-SAME: () #[[ATTR7:[0-9]+]] comdat {
 // CHECK5-NEXT:  entry:
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK5-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2306,7 +2307,6 @@ int main() {
 // CHECK5-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIcLi5EEiv_l52() #[[ATTR6]]
 // CHECK5-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK5:       omp_offload.cont:
-// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK5-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -2347,8 +2347,9 @@ int main() {
 // CHECK5-SAME: () #[[ATTR7]] comdat {
 // CHECK5-NEXT:  entry:
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK5-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2382,7 +2383,6 @@ int main() {
 // CHECK5-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainI1SLi1EEiv_l52() #[[ATTR6]]
 // CHECK5-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK5:       omp_offload.cont:
-// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK5-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -3127,18 +3127,19 @@ int main() {
 // CHECK9-NEXT:    [[EXN_SLOT:%.*]] = alloca ptr, align 8
 // CHECK9-NEXT:    [[EHSELECTOR_SLOT:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    call void @_ZN1SC1El(ptr nonnull align 8 dereferenceable(24) [[S]], i64 0)
 // CHECK9-NEXT:    [[CALL:%.*]] = invoke i8 @_ZN1ScvcEv(ptr nonnull align 8 dereferenceable(24) [[S]])
 // CHECK9-NEXT:    to label [[INVOKE_CONT:%.*]] unwind label [[LPAD:%.*]]
 // CHECK9:       invoke.cont:
 // CHECK9-NEXT:    store i8 [[CALL]], ptr [[A]], align 1
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3192,7 +3193,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -3614,8 +3614,9 @@ int main() {
 // CHECK9-SAME: () #[[ATTR7:[0-9]+]] comdat {
 // CHECK9-NEXT:  entry:
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3649,7 +3650,6 @@ int main() {
 // CHECK9-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIcLi5EEiv_l52() #[[ATTR6]]
 // CHECK9-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK9:       omp_offload.cont:
-// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK9-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -3690,8 +3690,9 @@ int main() {
 // CHECK9-SAME: () #[[ATTR7]] comdat {
 // CHECK9-NEXT:  entry:
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3725,7 +3726,6 @@ int main() {
 // CHECK9-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainI1SLi1EEiv_l52() #[[ATTR6]]
 // CHECK9-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK9:       omp_offload.cont:
-// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK9-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -4809,18 +4809,19 @@ int main() {
 // CHECK13-NEXT:    [[EXN_SLOT:%.*]] = alloca ptr, align 8
 // CHECK13-NEXT:    [[EHSELECTOR_SLOT:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK13-NEXT:    call void @_ZN1SC1El(ptr nonnull align 8 dereferenceable(24) [[S]], i64 0)
 // CHECK13-NEXT:    [[CALL:%.*]] = invoke i8 @_ZN1ScvcEv(ptr nonnull align 8 dereferenceable(24) [[S]])
 // CHECK13-NEXT:    to label [[INVOKE_CONT:%.*]] unwind label [[LPAD:%.*]]
 // CHECK13:       invoke.cont:
 // CHECK13-NEXT:    store i8 [[CALL]], ptr [[A]], align 1
-// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK13-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4874,7 +4875,6 @@ int main() {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK13-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK13-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -5296,8 +5296,9 @@ int main() {
 // CHECK13-SAME: () #[[ATTR7:[0-9]+]] comdat {
 // CHECK13-NEXT:  entry:
 // CHECK13-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK13-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK13-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK13-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5331,7 +5332,6 @@ int main() {
 // CHECK13-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIcLi5EEiv_l52() #[[ATTR6]]
 // CHECK13-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK13:       omp_offload.cont:
-// CHECK13-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK13-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -5372,8 +5372,9 @@ int main() {
 // CHECK13-SAME: () #[[ATTR7]] comdat {
 // CHECK13-NEXT:  entry:
 // CHECK13-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK13-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK13-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK13-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5407,7 +5408,6 @@ int main() {
 // CHECK13-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainI1SLi1EEiv_l52() #[[ATTR6]]
 // CHECK13-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK13:       omp_offload.cont:
-// CHECK13-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK13-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_parallel_for_simd_private_codegen.cpp b/clang/test/OpenMP/distribute_parallel_for_simd_private_codegen.cpp
index 0816e513772fe..7696d4ab1ff51 100644
--- a/clang/test/OpenMP/distribute_parallel_for_simd_private_codegen.cpp
+++ b/clang/test/OpenMP/distribute_parallel_for_simd_private_codegen.cpp
@@ -576,6 +576,7 @@ int main() {
 // CHECK9-NEXT:    [[VAR:%.*]] = alloca ptr, align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store ptr [[G]], ptr [[G1]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIfEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
@@ -587,7 +588,6 @@ int main() {
 // CHECK9-NEXT:    call void @_ZN1SIfEC1Ef(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], float noundef 2.000000e+00)
 // CHECK9-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 8
 // CHECK9-NEXT:    store ptr undef, ptr [[_TMP1]], align 8
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -913,6 +913,7 @@ int main() {
 // CHECK9-NEXT:    [[VAR:%.*]] = alloca ptr, align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK9-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -922,7 +923,6 @@ int main() {
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef signext 2)
 // CHECK9-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 8
 // CHECK9-NEXT:    store ptr undef, ptr [[_TMP1]], align 8
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1322,6 +1322,7 @@ int main() {
 // CHECK11-NEXT:    [[VAR:%.*]] = alloca ptr, align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store ptr [[G]], ptr [[G1]], align 4
 // CHECK11-NEXT:    call void @_ZN1SIfEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
@@ -1333,7 +1334,6 @@ int main() {
 // CHECK11-NEXT:    call void @_ZN1SIfEC1Ef(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], float noundef 2.000000e+00)
 // CHECK11-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 4
 // CHECK11-NEXT:    store ptr undef, ptr [[_TMP1]], align 4
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1653,6 +1653,7 @@ int main() {
 // CHECK11-NEXT:    [[VAR:%.*]] = alloca ptr, align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK11-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1662,7 +1663,6 @@ int main() {
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef 2)
 // CHECK11-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 4
 // CHECK11-NEXT:    store ptr undef, ptr [[_TMP1]], align 4
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_parallel_for_simd_proc_bind_codegen.cpp b/clang/test/OpenMP/distribute_parallel_for_simd_proc_bind_codegen.cpp
index d992f389100e3..700b3ca0a0494 100644
--- a/clang/test/OpenMP/distribute_parallel_for_simd_proc_bind_codegen.cpp
+++ b/clang/test/OpenMP/distribute_parallel_for_simd_proc_bind_codegen.cpp
@@ -58,9 +58,10 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -94,7 +95,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}_main_l37() #[[ATTR2:[0-9]+]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_private_codegen.cpp b/clang/test/OpenMP/distribute_private_codegen.cpp
index ab654e7ab3fa1..7dd4c5f20b070 100644
--- a/clang/test/OpenMP/distribute_private_codegen.cpp
+++ b/clang/test/OpenMP/distribute_private_codegen.cpp
@@ -349,8 +349,10 @@ int main() {
 // CHECK9-NEXT:    [[VAR:%.*]] = alloca ptr, align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS3:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store ptr [[G]], ptr [[G1]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIfEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
@@ -362,7 +364,6 @@ int main() {
 // CHECK9-NEXT:    call void @_ZN1SIfEC1Ef(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], float noundef 2.000000e+00)
 // CHECK9-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 8
 // CHECK9-NEXT:    store ptr undef, ptr [[_TMP1]], align 8
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -396,7 +397,6 @@ int main() {
 // CHECK9-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}_main_l93() #[[ATTR4:[0-9]+]]
 // CHECK9-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK9:       omp_offload.cont:
-// CHECK9-NEXT:    [[KERNEL_ARGS3:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS3]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK9-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS3]], i32 0, i32 1
@@ -671,6 +671,7 @@ int main() {
 // CHECK9-NEXT:    [[VAR:%.*]] = alloca ptr, align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK9-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -680,7 +681,6 @@ int main() {
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef signext 2)
 // CHECK9-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 8
 // CHECK9-NEXT:    store ptr undef, ptr [[_TMP1]], align 8
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -962,8 +962,10 @@ int main() {
 // CHECK11-NEXT:    [[VAR:%.*]] = alloca ptr, align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS3:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store ptr [[G]], ptr [[G1]], align 4
 // CHECK11-NEXT:    call void @_ZN1SIfEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
@@ -975,7 +977,6 @@ int main() {
 // CHECK11-NEXT:    call void @_ZN1SIfEC1Ef(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], float noundef 2.000000e+00)
 // CHECK11-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 4
 // CHECK11-NEXT:    store ptr undef, ptr [[_TMP1]], align 4
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1009,7 +1010,6 @@ int main() {
 // CHECK11-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}_main_l93() #[[ATTR4:[0-9]+]]
 // CHECK11-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK11:       omp_offload.cont:
-// CHECK11-NEXT:    [[KERNEL_ARGS3:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS3]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK11-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS3]], i32 0, i32 1
@@ -1282,6 +1282,7 @@ int main() {
 // CHECK11-NEXT:    [[VAR:%.*]] = alloca ptr, align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK11-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1291,7 +1292,6 @@ int main() {
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef 2)
 // CHECK11-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 4
 // CHECK11-NEXT:    store ptr undef, ptr [[_TMP1]], align 4
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_simd_codegen.cpp b/clang/test/OpenMP/distribute_simd_codegen.cpp
index bcf49a213f2ab..02d84779f3b85 100644
--- a/clang/test/OpenMP/distribute_simd_codegen.cpp
+++ b/clang/test/OpenMP/distribute_simd_codegen.cpp
@@ -155,6 +155,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[C]], ptr [[C_ADDR]], align 8
@@ -189,7 +190,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    store ptr null, ptr [[TMP15]], align 8
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -355,6 +355,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[C]], ptr [[C_ADDR]], align 8
@@ -389,7 +390,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    store ptr null, ptr [[TMP15]], align 8
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -553,6 +553,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[C]], ptr [[C_ADDR]], align 8
@@ -587,7 +588,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    store ptr null, ptr [[TMP15]], align 8
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -770,6 +770,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i8 0, ptr [[A]], align 1
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i8, ptr [[I]], align 1
 // CHECK1-NEXT:    store i8 [[TMP0]], ptr [[I_CASTED]], align 1
@@ -804,7 +805,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    [[TMP14:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK1-NEXT:    [[ADD4:%.*]] = add nsw i32 [[TMP14]], 1
 // CHECK1-NEXT:    [[TMP15:%.*]] = zext i32 [[ADD4]] to i64
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -982,6 +982,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i16 0, ptr [[AA]], align 2
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i16, ptr [[AA]], align 2
 // CHECK1-NEXT:    store i16 [[TMP0]], ptr [[AA_CASTED]], align 2
@@ -994,7 +995,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK1-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1147,6 +1147,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[C]], ptr [[C_ADDR]], align 4
@@ -1181,7 +1182,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    store ptr null, ptr [[TMP15]], align 4
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1343,6 +1343,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[C]], ptr [[C_ADDR]], align 4
@@ -1377,7 +1378,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    store ptr null, ptr [[TMP15]], align 4
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1537,6 +1537,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[C]], ptr [[C_ADDR]], align 4
@@ -1571,7 +1572,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    store ptr null, ptr [[TMP15]], align 4
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1750,6 +1750,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i8, align 1
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i8 0, ptr [[A]], align 1
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i8, ptr [[I]], align 1
 // CHECK3-NEXT:    store i8 [[TMP0]], ptr [[I_CASTED]], align 1
@@ -1784,7 +1785,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    [[TMP14:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK3-NEXT:    [[ADD4:%.*]] = add nsw i32 [[TMP14]], 1
 // CHECK3-NEXT:    [[TMP15:%.*]] = zext i32 [[ADD4]] to i64
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK3-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1962,6 +1962,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i16 0, ptr [[AA]], align 2
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i16, ptr [[AA]], align 2
 // CHECK3-NEXT:    store i16 [[TMP0]], ptr [[AA_CASTED]], align 2
@@ -1974,7 +1975,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK3-NEXT:    store ptr null, ptr [[TMP4]], align 4
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2127,6 +2127,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 8
 // CHECK5-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 8
 // CHECK5-NEXT:    store ptr [[C]], ptr [[C_ADDR]], align 8
@@ -2161,7 +2162,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK5-NEXT:    store ptr null, ptr [[TMP15]], align 8
 // CHECK5-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK5-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2327,6 +2327,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 8
 // CHECK5-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 8
 // CHECK5-NEXT:    store ptr [[C]], ptr [[C_ADDR]], align 8
@@ -2361,7 +2362,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK5-NEXT:    store ptr null, ptr [[TMP15]], align 8
 // CHECK5-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK5-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2525,6 +2525,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 8
 // CHECK5-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 8
 // CHECK5-NEXT:    store ptr [[C]], ptr [[C_ADDR]], align 8
@@ -2559,7 +2560,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK5-NEXT:    store ptr null, ptr [[TMP15]], align 8
 // CHECK5-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK5-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2742,6 +2742,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i8, align 1
 // CHECK5-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK5-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    store i8 0, ptr [[A]], align 1
 // CHECK5-NEXT:    [[TMP0:%.*]] = load i8, ptr [[I]], align 1
 // CHECK5-NEXT:    store i8 [[TMP0]], ptr [[I_CASTED]], align 1
@@ -2776,7 +2777,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK5-NEXT:    [[TMP14:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK5-NEXT:    [[ADD4:%.*]] = add nsw i32 [[TMP14]], 1
 // CHECK5-NEXT:    [[TMP15:%.*]] = zext i32 [[ADD4]] to i64
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK5-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2985,6 +2985,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    store i16 0, ptr [[AA]], align 2
 // CHECK5-NEXT:    [[TMP0:%.*]] = load i16, ptr [[AA]], align 2
 // CHECK5-NEXT:    store i16 [[TMP0]], ptr [[AA_CASTED]], align 2
@@ -2997,7 +2998,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK5-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK5-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK5-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3150,6 +3150,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK7-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 4
 // CHECK7-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 4
 // CHECK7-NEXT:    store ptr [[C]], ptr [[C_ADDR]], align 4
@@ -3184,7 +3185,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK7-NEXT:    store ptr null, ptr [[TMP15]], align 4
 // CHECK7-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK7-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3346,6 +3346,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK7-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 4
 // CHECK7-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 4
 // CHECK7-NEXT:    store ptr [[C]], ptr [[C_ADDR]], align 4
@@ -3380,7 +3381,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK7-NEXT:    store ptr null, ptr [[TMP15]], align 4
 // CHECK7-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK7-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3540,6 +3540,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK7-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 4
 // CHECK7-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 4
 // CHECK7-NEXT:    store ptr [[C]], ptr [[C_ADDR]], align 4
@@ -3574,7 +3575,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK7-NEXT:    store ptr null, ptr [[TMP15]], align 4
 // CHECK7-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK7-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3753,6 +3753,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK7-NEXT:    [[TMP:%.*]] = alloca i8, align 1
 // CHECK7-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK7-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    store i8 0, ptr [[A]], align 1
 // CHECK7-NEXT:    [[TMP0:%.*]] = load i8, ptr [[I]], align 1
 // CHECK7-NEXT:    store i8 [[TMP0]], ptr [[I_CASTED]], align 1
@@ -3787,7 +3788,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK7-NEXT:    [[TMP14:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK7-NEXT:    [[ADD4:%.*]] = add nsw i32 [[TMP14]], 1
 // CHECK7-NEXT:    [[TMP15:%.*]] = zext i32 [[ADD4]] to i64
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK7-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3996,6 +3996,7 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    store i16 0, ptr [[AA]], align 2
 // CHECK7-NEXT:    [[TMP0:%.*]] = load i16, ptr [[AA]], align 2
 // CHECK7-NEXT:    store i16 [[TMP0]], ptr [[AA_CASTED]], align 2
@@ -4008,7 +4009,6 @@ int fint(void) { return ftemplate<int>(); }
 // CHECK7-NEXT:    store ptr null, ptr [[TMP4]], align 4
 // CHECK7-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK7-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_simd_firstprivate_codegen.cpp b/clang/test/OpenMP/distribute_simd_firstprivate_codegen.cpp
index 60ed8f0e9eb68..8787b131d070e 100644
--- a/clang/test/OpenMP/distribute_simd_firstprivate_codegen.cpp
+++ b/clang/test/OpenMP/distribute_simd_firstprivate_codegen.cpp
@@ -554,6 +554,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store ptr [[G]], ptr [[G1]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIfEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
@@ -607,7 +608,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -868,6 +868,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK9-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -910,7 +911,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP17]], align 8
 // CHECK9-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK9-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1240,6 +1240,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store ptr [[G]], ptr [[G1]], align 4
 // CHECK11-NEXT:    call void @_ZN1SIfEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
@@ -1293,7 +1294,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1552,6 +1552,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK11-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1594,7 +1595,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP17]], align 4
 // CHECK11-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK11-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_simd_lastprivate_codegen.cpp b/clang/test/OpenMP/distribute_simd_lastprivate_codegen.cpp
index d540eda5dbfc5..c26260cb45159 100644
--- a/clang/test/OpenMP/distribute_simd_lastprivate_codegen.cpp
+++ b/clang/test/OpenMP/distribute_simd_lastprivate_codegen.cpp
@@ -540,6 +540,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store ptr [[G]], ptr [[G1]], align 8
@@ -594,7 +595,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -873,6 +873,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK9-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -915,7 +916,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP17]], align 8
 // CHECK9-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK9-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1263,6 +1263,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store ptr [[G]], ptr [[G1]], align 4
@@ -1317,7 +1318,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1594,6 +1594,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK11-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1636,7 +1637,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP17]], align 4
 // CHECK11-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK11-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_simd_private_codegen.cpp b/clang/test/OpenMP/distribute_simd_private_codegen.cpp
index 824a96642f046..181881a43fcbc 100644
--- a/clang/test/OpenMP/distribute_simd_private_codegen.cpp
+++ b/clang/test/OpenMP/distribute_simd_private_codegen.cpp
@@ -390,12 +390,14 @@ int main() {
 // CHECK9-NEXT:    [[VAR:%.*]] = alloca ptr, align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[I_CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS3:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store ptr [[G]], ptr [[G1]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIfEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
@@ -407,7 +409,6 @@ int main() {
 // CHECK9-NEXT:    call void @_ZN1SIfEC1Ef(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], float noundef 2.000000e+00)
 // CHECK9-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 8
 // CHECK9-NEXT:    store ptr undef, ptr [[_TMP1]], align 8
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -452,7 +453,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP19]], align 8
 // CHECK9-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS3:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS3]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP22]], align 4
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS3]], i32 0, i32 1
@@ -745,6 +745,7 @@ int main() {
 // CHECK9-NEXT:    [[VAR:%.*]] = alloca ptr, align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK9-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -754,7 +755,6 @@ int main() {
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef signext 2)
 // CHECK9-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 8
 // CHECK9-NEXT:    store ptr undef, ptr [[_TMP1]], align 8
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1043,12 +1043,14 @@ int main() {
 // CHECK11-NEXT:    [[VAR:%.*]] = alloca ptr, align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[I_CASTED:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS3:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store ptr [[G]], ptr [[G1]], align 4
 // CHECK11-NEXT:    call void @_ZN1SIfEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
@@ -1060,7 +1062,6 @@ int main() {
 // CHECK11-NEXT:    call void @_ZN1SIfEC1Ef(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], float noundef 2.000000e+00)
 // CHECK11-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 4
 // CHECK11-NEXT:    store ptr undef, ptr [[_TMP1]], align 4
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1105,7 +1106,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP19]], align 4
 // CHECK11-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS3:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS3]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS3]], i32 0, i32 1
@@ -1396,6 +1396,7 @@ int main() {
 // CHECK11-NEXT:    [[VAR:%.*]] = alloca ptr, align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK11-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1405,7 +1406,6 @@ int main() {
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef 2)
 // CHECK11-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 4
 // CHECK11-NEXT:    store ptr undef, ptr [[_TMP1]], align 4
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/distribute_simd_reduction_codegen.cpp b/clang/test/OpenMP/distribute_simd_reduction_codegen.cpp
index 72249ee7a719b..979dbdd030cbe 100644
--- a/clang/test/OpenMP/distribute_simd_reduction_codegen.cpp
+++ b/clang/test/OpenMP/distribute_simd_reduction_codegen.cpp
@@ -91,6 +91,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr @_ZZ4mainE5sivar, align 4
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[SIVAR_CASTED]], align 4
@@ -103,7 +104,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -240,6 +240,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr [[T_VAR]], align 4
@@ -253,7 +254,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -395,6 +395,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr @_ZZ4mainE5sivar, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[SIVAR_CASTED]], align 4
@@ -407,7 +408,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP4]], align 4
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -544,6 +544,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr [[T_VAR]], align 4
@@ -557,7 +558,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP4]], align 4
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/for_non_rectangular_codegen.c b/clang/test/OpenMP/for_non_rectangular_codegen.c
new file mode 100644
index 0000000000000..48dd37f6eddc0
--- /dev/null
+++ b/clang/test/OpenMP/for_non_rectangular_codegen.c
@@ -0,0 +1,313 @@
+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --function-signature --include-generated-funcs --replace-value-regex "__omp_offloading_[0-9a-z]+_[0-9a-z]+" "reduction_size[.].+[.]" "pl_cond[.].+[.|,]" --prefix-filecheck-ir-name _
+// RUN: %clang_cc1 -verify -fopenmp -x c -triple x86_64-unknown-unknown -emit-llvm %s -o - | FileCheck %s
+// RUN: %clang_cc1 -fopenmp -x c -triple x86_64-unknown-unknown -emit-pch -o %t %s
+// RUN: %clang_cc1 -fopenmp -x c -triple x86_64-unknown-unknown -include-pch %t -verify %s -emit-llvm -o - | FileCheck %s
+
+// RUN: %clang_cc1 -verify -fopenmp-simd -x c -triple x86_64-unknown-unknown -emit-llvm %s -o - | FileCheck --check-prefix SIMD-ONLY0 %s
+// RUN: %clang_cc1 -fopenmp-simd -x c -triple x86_64-unknown-unknown -emit-pch -o %t %s
+// RUN: %clang_cc1 -fopenmp-simd -x c -triple x86_64-unknown-unknown -include-pch %t -verify %s -emit-llvm -o - | FileCheck --check-prefix SIMD-ONLY0 %s
+// expected-no-diagnostics
+#ifndef HEADER
+#define HEADER
+
+void collapsed(int mp) {
+#pragma omp for collapse(3)
+  for (int j = 0; j < mp; ++j) {
+    for (int i = j; i < mp; ++i) {
+      for (int i0 = 0; i0 < 10; ++i0) {
+        ;
+      }
+    }
+  }
+}
+
+#endif // HEADER
+// CHECK-LABEL: define {{[^@]+}}@collapsed
+// CHECK-SAME: (i32 noundef [[MP:%.*]]) #[[ATTR0:[0-9]+]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[MP_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTOMP_IV:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTLB_MIN:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTLB_MAX:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTMIN_LESS_MAX:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTUPPER:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTLOWER:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[J:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[I:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[I0:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[_TMP15:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTOMP_LB:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[DOTOMP_UB:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[DOTOMP_STRIDE:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[DOTOMP_IS_LAST:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[J19:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[I20:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[I021:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB2:[0-9]+]])
+// CHECK-NEXT:    store i32 [[MP]], ptr [[MP_ADDR]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[MP_ADDR]], align 4
+// CHECK-NEXT:    store i32 [[TMP1]], ptr [[DOTCAPTURE_EXPR_]], align 4
+// CHECK-NEXT:    store i32 0, ptr [[TMP]], align 4
+// CHECK-NEXT:    [[TMP2:%.*]] = load i32, ptr [[TMP]], align 4
+// CHECK-NEXT:    store i32 [[TMP2]], ptr [[DOTLB_MIN]], align 4
+// CHECK-NEXT:    [[TMP3:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
+// CHECK-NEXT:    [[SUB:%.*]] = sub nsw i32 [[TMP3]], 1
+// CHECK-NEXT:    [[DIV:%.*]] = sdiv i32 [[SUB]], 1
+// CHECK-NEXT:    [[MUL:%.*]] = mul nsw i32 [[DIV]], 1
+// CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 0, [[MUL]]
+// CHECK-NEXT:    store i32 [[ADD]], ptr [[TMP]], align 4
+// CHECK-NEXT:    [[TMP4:%.*]] = load i32, ptr [[TMP]], align 4
+// CHECK-NEXT:    store i32 [[TMP4]], ptr [[DOTLB_MAX]], align 4
+// CHECK-NEXT:    [[TMP5:%.*]] = load i32, ptr [[DOTLB_MIN]], align 4
+// CHECK-NEXT:    [[TMP6:%.*]] = load i32, ptr [[DOTLB_MAX]], align 4
+// CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 [[TMP5]], [[TMP6]]
+// CHECK-NEXT:    [[CONV:%.*]] = zext i1 [[CMP]] to i32
+// CHECK-NEXT:    store i32 [[CONV]], ptr [[DOTMIN_LESS_MAX]], align 4
+// CHECK-NEXT:    [[TMP7:%.*]] = load i32, ptr [[MP_ADDR]], align 4
+// CHECK-NEXT:    store i32 [[TMP7]], ptr [[DOTUPPER]], align 4
+// CHECK-NEXT:    [[TMP8:%.*]] = load i32, ptr [[DOTMIN_LESS_MAX]], align 4
+// CHECK-NEXT:    [[TOBOOL:%.*]] = icmp ne i32 [[TMP8]], 0
+// CHECK-NEXT:    br i1 [[TOBOOL]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]]
+// CHECK:       cond.true:
+// CHECK-NEXT:    [[TMP9:%.*]] = load i32, ptr [[DOTLB_MIN]], align 4
+// CHECK-NEXT:    br label [[COND_END:%.*]]
+// CHECK:       cond.false:
+// CHECK-NEXT:    [[TMP10:%.*]] = load i32, ptr [[DOTLB_MAX]], align 4
+// CHECK-NEXT:    br label [[COND_END]]
+// CHECK:       cond.end:
+// CHECK-NEXT:    [[COND:%.*]] = phi i32 [ [[TMP9]], [[COND_TRUE]] ], [ [[TMP10]], [[COND_FALSE]] ]
+// CHECK-NEXT:    store i32 [[COND]], ptr [[TMP]], align 4
+// CHECK-NEXT:    store i32 [[COND]], ptr [[DOTLOWER]], align 4
+// CHECK-NEXT:    [[TMP11:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
+// CHECK-NEXT:    [[SUB4:%.*]] = sub nsw i32 [[TMP11]], 0
+// CHECK-NEXT:    [[DIV5:%.*]] = sdiv i32 [[SUB4]], 1
+// CHECK-NEXT:    [[CONV6:%.*]] = sext i32 [[DIV5]] to i64
+// CHECK-NEXT:    [[TMP12:%.*]] = load i32, ptr [[DOTUPPER]], align 4
+// CHECK-NEXT:    [[TMP13:%.*]] = load i32, ptr [[DOTLOWER]], align 4
+// CHECK-NEXT:    [[SUB7:%.*]] = sub i32 [[TMP12]], [[TMP13]]
+// CHECK-NEXT:    [[SUB8:%.*]] = sub i32 [[SUB7]], 1
+// CHECK-NEXT:    [[ADD9:%.*]] = add i32 [[SUB8]], 1
+// CHECK-NEXT:    [[DIV10:%.*]] = udiv i32 [[ADD9]], 1
+// CHECK-NEXT:    [[CONV11:%.*]] = zext i32 [[DIV10]] to i64
+// CHECK-NEXT:    [[MUL12:%.*]] = mul nsw i64 [[CONV6]], [[CONV11]]
+// CHECK-NEXT:    [[MUL13:%.*]] = mul nsw i64 [[MUL12]], 10
+// CHECK-NEXT:    [[SUB14:%.*]] = sub nsw i64 [[MUL13]], 1
+// CHECK-NEXT:    store i64 [[SUB14]], ptr [[DOTCAPTURE_EXPR_3]], align 8
+// CHECK-NEXT:    store i32 0, ptr [[J]], align 4
+// CHECK-NEXT:    [[TMP14:%.*]] = load i32, ptr [[J]], align 4
+// CHECK-NEXT:    store i32 [[TMP14]], ptr [[I]], align 4
+// CHECK-NEXT:    store i32 0, ptr [[I0]], align 4
+// CHECK-NEXT:    store i32 0, ptr [[_TMP15]], align 4
+// CHECK-NEXT:    [[TMP15:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
+// CHECK-NEXT:    [[CMP16:%.*]] = icmp slt i32 0, [[TMP15]]
+// CHECK-NEXT:    br i1 [[CMP16]], label [[OMP_PRECOND_THEN:%.*]], label [[OMP_PRECOND_END:%.*]]
+// CHECK:       omp.precond.then:
+// CHECK-NEXT:    store i64 0, ptr [[DOTOMP_LB]], align 8
+// CHECK-NEXT:    [[TMP16:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
+// CHECK-NEXT:    store i64 [[TMP16]], ptr [[DOTOMP_UB]], align 8
+// CHECK-NEXT:    store i64 1, ptr [[DOTOMP_STRIDE]], align 8
+// CHECK-NEXT:    store i32 0, ptr [[DOTOMP_IS_LAST]], align 4
+// CHECK-NEXT:    call void @__kmpc_for_static_init_8(ptr @[[GLOB1:[0-9]+]], i32 [[TMP0]], i32 34, ptr [[DOTOMP_IS_LAST]], ptr [[DOTOMP_LB]], ptr [[DOTOMP_UB]], ptr [[DOTOMP_STRIDE]], i64 1, i64 1)
+// CHECK-NEXT:    [[TMP17:%.*]] = load i64, ptr [[DOTOMP_UB]], align 8
+// CHECK-NEXT:    [[TMP18:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
+// CHECK-NEXT:    [[CMP22:%.*]] = icmp sgt i64 [[TMP17]], [[TMP18]]
+// CHECK-NEXT:    br i1 [[CMP22]], label [[COND_TRUE24:%.*]], label [[COND_FALSE25:%.*]]
+// CHECK:       cond.true24:
+// CHECK-NEXT:    [[TMP19:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
+// CHECK-NEXT:    br label [[COND_END26:%.*]]
+// CHECK:       cond.false25:
+// CHECK-NEXT:    [[TMP20:%.*]] = load i64, ptr [[DOTOMP_UB]], align 8
+// CHECK-NEXT:    br label [[COND_END26]]
+// CHECK:       cond.end26:
+// CHECK-NEXT:    [[COND27:%.*]] = phi i64 [ [[TMP19]], [[COND_TRUE24]] ], [ [[TMP20]], [[COND_FALSE25]] ]
+// CHECK-NEXT:    store i64 [[COND27]], ptr [[DOTOMP_UB]], align 8
+// CHECK-NEXT:    [[TMP21:%.*]] = load i64, ptr [[DOTOMP_LB]], align 8
+// CHECK-NEXT:    store i64 [[TMP21]], ptr [[DOTOMP_IV]], align 8
+// CHECK-NEXT:    br label [[OMP_INNER_FOR_COND:%.*]]
+// CHECK:       omp.inner.for.cond:
+// CHECK-NEXT:    [[TMP22:%.*]] = load i64, ptr [[DOTOMP_IV]], align 8
+// CHECK-NEXT:    [[TMP23:%.*]] = load i64, ptr [[DOTOMP_UB]], align 8
+// CHECK-NEXT:    [[CMP28:%.*]] = icmp sle i64 [[TMP22]], [[TMP23]]
+// CHECK-NEXT:    br i1 [[CMP28]], label [[OMP_INNER_FOR_BODY:%.*]], label [[OMP_INNER_FOR_END:%.*]]
+// CHECK:       omp.inner.for.body:
+// CHECK-NEXT:    [[TMP24:%.*]] = load i64, ptr [[DOTOMP_IV]], align 8
+// CHECK-NEXT:    [[TMP25:%.*]] = load i32, ptr [[DOTUPPER]], align 4
+// CHECK-NEXT:    [[TMP26:%.*]] = load i32, ptr [[DOTLOWER]], align 4
+// CHECK-NEXT:    [[SUB30:%.*]] = sub i32 [[TMP25]], [[TMP26]]
+// CHECK-NEXT:    [[SUB31:%.*]] = sub i32 [[SUB30]], 1
+// CHECK-NEXT:    [[ADD32:%.*]] = add i32 [[SUB31]], 1
+// CHECK-NEXT:    [[DIV33:%.*]] = udiv i32 [[ADD32]], 1
+// CHECK-NEXT:    [[MUL34:%.*]] = mul i32 1, [[DIV33]]
+// CHECK-NEXT:    [[MUL35:%.*]] = mul i32 [[MUL34]], 10
+// CHECK-NEXT:    [[CONV36:%.*]] = zext i32 [[MUL35]] to i64
+// CHECK-NEXT:    [[DIV37:%.*]] = sdiv i64 [[TMP24]], [[CONV36]]
+// CHECK-NEXT:    [[MUL38:%.*]] = mul nsw i64 [[DIV37]], 1
+// CHECK-NEXT:    [[ADD39:%.*]] = add nsw i64 0, [[MUL38]]
+// CHECK-NEXT:    [[CONV40:%.*]] = trunc i64 [[ADD39]] to i32
+// CHECK-NEXT:    store i32 [[CONV40]], ptr [[J19]], align 4
+// CHECK-NEXT:    [[TMP27:%.*]] = load i32, ptr [[J19]], align 4
+// CHECK-NEXT:    [[CONV41:%.*]] = sext i32 [[TMP27]] to i64
+// CHECK-NEXT:    [[TMP28:%.*]] = load i64, ptr [[DOTOMP_IV]], align 8
+// CHECK-NEXT:    [[TMP29:%.*]] = load i64, ptr [[DOTOMP_IV]], align 8
+// CHECK-NEXT:    [[TMP30:%.*]] = load i32, ptr [[DOTUPPER]], align 4
+// CHECK-NEXT:    [[TMP31:%.*]] = load i32, ptr [[DOTLOWER]], align 4
+// CHECK-NEXT:    [[SUB42:%.*]] = sub i32 [[TMP30]], [[TMP31]]
+// CHECK-NEXT:    [[SUB43:%.*]] = sub i32 [[SUB42]], 1
+// CHECK-NEXT:    [[ADD44:%.*]] = add i32 [[SUB43]], 1
+// CHECK-NEXT:    [[DIV45:%.*]] = udiv i32 [[ADD44]], 1
+// CHECK-NEXT:    [[MUL46:%.*]] = mul i32 1, [[DIV45]]
+// CHECK-NEXT:    [[MUL47:%.*]] = mul i32 [[MUL46]], 10
+// CHECK-NEXT:    [[CONV48:%.*]] = zext i32 [[MUL47]] to i64
+// CHECK-NEXT:    [[DIV49:%.*]] = sdiv i64 [[TMP29]], [[CONV48]]
+// CHECK-NEXT:    [[TMP32:%.*]] = load i32, ptr [[DOTUPPER]], align 4
+// CHECK-NEXT:    [[TMP33:%.*]] = load i32, ptr [[DOTLOWER]], align 4
+// CHECK-NEXT:    [[SUB50:%.*]] = sub i32 [[TMP32]], [[TMP33]]
+// CHECK-NEXT:    [[SUB51:%.*]] = sub i32 [[SUB50]], 1
+// CHECK-NEXT:    [[ADD52:%.*]] = add i32 [[SUB51]], 1
+// CHECK-NEXT:    [[DIV53:%.*]] = udiv i32 [[ADD52]], 1
+// CHECK-NEXT:    [[MUL54:%.*]] = mul i32 1, [[DIV53]]
+// CHECK-NEXT:    [[MUL55:%.*]] = mul i32 [[MUL54]], 10
+// CHECK-NEXT:    [[CONV56:%.*]] = zext i32 [[MUL55]] to i64
+// CHECK-NEXT:    [[MUL57:%.*]] = mul nsw i64 [[DIV49]], [[CONV56]]
+// CHECK-NEXT:    [[SUB58:%.*]] = sub nsw i64 [[TMP28]], [[MUL57]]
+// CHECK-NEXT:    [[DIV59:%.*]] = sdiv i64 [[SUB58]], 10
+// CHECK-NEXT:    [[MUL60:%.*]] = mul nsw i64 [[DIV59]], 1
+// CHECK-NEXT:    [[ADD61:%.*]] = add nsw i64 [[CONV41]], [[MUL60]]
+// CHECK-NEXT:    [[CONV62:%.*]] = trunc i64 [[ADD61]] to i32
+// CHECK-NEXT:    store i32 [[CONV62]], ptr [[I20]], align 4
+// CHECK-NEXT:    [[TMP34:%.*]] = load i64, ptr [[DOTOMP_IV]], align 8
+// CHECK-NEXT:    [[TMP35:%.*]] = load i64, ptr [[DOTOMP_IV]], align 8
+// CHECK-NEXT:    [[TMP36:%.*]] = load i32, ptr [[DOTUPPER]], align 4
+// CHECK-NEXT:    [[TMP37:%.*]] = load i32, ptr [[DOTLOWER]], align 4
+// CHECK-NEXT:    [[SUB63:%.*]] = sub i32 [[TMP36]], [[TMP37]]
+// CHECK-NEXT:    [[SUB64:%.*]] = sub i32 [[SUB63]], 1
+// CHECK-NEXT:    [[ADD65:%.*]] = add i32 [[SUB64]], 1
+// CHECK-NEXT:    [[DIV66:%.*]] = udiv i32 [[ADD65]], 1
+// CHECK-NEXT:    [[MUL67:%.*]] = mul i32 1, [[DIV66]]
+// CHECK-NEXT:    [[MUL68:%.*]] = mul i32 [[MUL67]], 10
+// CHECK-NEXT:    [[CONV69:%.*]] = zext i32 [[MUL68]] to i64
+// CHECK-NEXT:    [[DIV70:%.*]] = sdiv i64 [[TMP35]], [[CONV69]]
+// CHECK-NEXT:    [[TMP38:%.*]] = load i32, ptr [[DOTUPPER]], align 4
+// CHECK-NEXT:    [[TMP39:%.*]] = load i32, ptr [[DOTLOWER]], align 4
+// CHECK-NEXT:    [[SUB71:%.*]] = sub i32 [[TMP38]], [[TMP39]]
+// CHECK-NEXT:    [[SUB72:%.*]] = sub i32 [[SUB71]], 1
+// CHECK-NEXT:    [[ADD73:%.*]] = add i32 [[SUB72]], 1
+// CHECK-NEXT:    [[DIV74:%.*]] = udiv i32 [[ADD73]], 1
+// CHECK-NEXT:    [[MUL75:%.*]] = mul i32 1, [[DIV74]]
+// CHECK-NEXT:    [[MUL76:%.*]] = mul i32 [[MUL75]], 10
+// CHECK-NEXT:    [[CONV77:%.*]] = zext i32 [[MUL76]] to i64
+// CHECK-NEXT:    [[MUL78:%.*]] = mul nsw i64 [[DIV70]], [[CONV77]]
+// CHECK-NEXT:    [[SUB79:%.*]] = sub nsw i64 [[TMP34]], [[MUL78]]
+// CHECK-NEXT:    [[TMP40:%.*]] = load i64, ptr [[DOTOMP_IV]], align 8
+// CHECK-NEXT:    [[TMP41:%.*]] = load i64, ptr [[DOTOMP_IV]], align 8
+// CHECK-NEXT:    [[TMP42:%.*]] = load i32, ptr [[DOTUPPER]], align 4
+// CHECK-NEXT:    [[TMP43:%.*]] = load i32, ptr [[DOTLOWER]], align 4
+// CHECK-NEXT:    [[SUB80:%.*]] = sub i32 [[TMP42]], [[TMP43]]
+// CHECK-NEXT:    [[SUB81:%.*]] = sub i32 [[SUB80]], 1
+// CHECK-NEXT:    [[ADD82:%.*]] = add i32 [[SUB81]], 1
+// CHECK-NEXT:    [[DIV83:%.*]] = udiv i32 [[ADD82]], 1
+// CHECK-NEXT:    [[MUL84:%.*]] = mul i32 1, [[DIV83]]
+// CHECK-NEXT:    [[MUL85:%.*]] = mul i32 [[MUL84]], 10
+// CHECK-NEXT:    [[CONV86:%.*]] = zext i32 [[MUL85]] to i64
+// CHECK-NEXT:    [[DIV87:%.*]] = sdiv i64 [[TMP41]], [[CONV86]]
+// CHECK-NEXT:    [[TMP44:%.*]] = load i32, ptr [[DOTUPPER]], align 4
+// CHECK-NEXT:    [[TMP45:%.*]] = load i32, ptr [[DOTLOWER]], align 4
+// CHECK-NEXT:    [[SUB88:%.*]] = sub i32 [[TMP44]], [[TMP45]]
+// CHECK-NEXT:    [[SUB89:%.*]] = sub i32 [[SUB88]], 1
+// CHECK-NEXT:    [[ADD90:%.*]] = add i32 [[SUB89]], 1
+// CHECK-NEXT:    [[DIV91:%.*]] = udiv i32 [[ADD90]], 1
+// CHECK-NEXT:    [[MUL92:%.*]] = mul i32 1, [[DIV91]]
+// CHECK-NEXT:    [[MUL93:%.*]] = mul i32 [[MUL92]], 10
+// CHECK-NEXT:    [[CONV94:%.*]] = zext i32 [[MUL93]] to i64
+// CHECK-NEXT:    [[MUL95:%.*]] = mul nsw i64 [[DIV87]], [[CONV94]]
+// CHECK-NEXT:    [[SUB96:%.*]] = sub nsw i64 [[TMP40]], [[MUL95]]
+// CHECK-NEXT:    [[DIV97:%.*]] = sdiv i64 [[SUB96]], 10
+// CHECK-NEXT:    [[MUL98:%.*]] = mul nsw i64 [[DIV97]], 10
+// CHECK-NEXT:    [[SUB99:%.*]] = sub nsw i64 [[SUB79]], [[MUL98]]
+// CHECK-NEXT:    [[MUL100:%.*]] = mul nsw i64 [[SUB99]], 1
+// CHECK-NEXT:    [[ADD101:%.*]] = add nsw i64 0, [[MUL100]]
+// CHECK-NEXT:    [[CONV102:%.*]] = trunc i64 [[ADD101]] to i32
+// CHECK-NEXT:    store i32 [[CONV102]], ptr [[I021]], align 4
+// CHECK-NEXT:    [[TMP46:%.*]] = load i32, ptr [[I20]], align 4
+// CHECK-NEXT:    [[TMP47:%.*]] = load i32, ptr [[MP_ADDR]], align 4
+// CHECK-NEXT:    [[CMP103:%.*]] = icmp slt i32 [[TMP46]], [[TMP47]]
+// CHECK-NEXT:    br i1 [[CMP103]], label [[OMP_BODY_NEXT:%.*]], label [[OMP_BODY_CONTINUE:%.*]]
+// CHECK:       omp.body.next:
+// CHECK-NEXT:    br label [[OMP_BODY_CONTINUE]]
+// CHECK:       omp.body.continue:
+// CHECK-NEXT:    br label [[OMP_INNER_FOR_INC:%.*]]
+// CHECK:       omp.inner.for.inc:
+// CHECK-NEXT:    [[TMP48:%.*]] = load i64, ptr [[DOTOMP_IV]], align 8
+// CHECK-NEXT:    [[ADD105:%.*]] = add nsw i64 [[TMP48]], 1
+// CHECK-NEXT:    store i64 [[ADD105]], ptr [[DOTOMP_IV]], align 8
+// CHECK-NEXT:    br label [[OMP_INNER_FOR_COND]]
+// CHECK:       omp.inner.for.end:
+// CHECK-NEXT:    br label [[OMP_LOOP_EXIT:%.*]]
+// CHECK:       omp.loop.exit:
+// CHECK-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB1]], i32 [[TMP0]])
+// CHECK-NEXT:    br label [[OMP_PRECOND_END]]
+// CHECK:       omp.precond.end:
+// CHECK-NEXT:    call void @__kmpc_barrier(ptr @[[GLOB3:[0-9]+]], i32 [[TMP0]])
+// CHECK-NEXT:    ret void
+//
+//
+// SIMD-ONLY0-LABEL: define {{[^@]+}}@collapsed
+// SIMD-ONLY0-SAME: (i32 noundef [[MP:%.*]]) #[[ATTR0:[0-9]+]] {
+// SIMD-ONLY0-NEXT:  entry:
+// SIMD-ONLY0-NEXT:    [[MP_ADDR:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[J:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[I:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    [[I0:%.*]] = alloca i32, align 4
+// SIMD-ONLY0-NEXT:    store i32 [[MP]], ptr [[MP_ADDR]], align 4
+// SIMD-ONLY0-NEXT:    store i32 0, ptr [[J]], align 4
+// SIMD-ONLY0-NEXT:    br label [[FOR_COND:%.*]]
+// SIMD-ONLY0:       for.cond:
+// SIMD-ONLY0-NEXT:    [[TMP0:%.*]] = load i32, ptr [[J]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP1:%.*]] = load i32, ptr [[MP_ADDR]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP:%.*]] = icmp slt i32 [[TMP0]], [[TMP1]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP]], label [[FOR_BODY:%.*]], label [[FOR_END12:%.*]]
+// SIMD-ONLY0:       for.body:
+// SIMD-ONLY0-NEXT:    [[TMP2:%.*]] = load i32, ptr [[J]], align 4
+// SIMD-ONLY0-NEXT:    store i32 [[TMP2]], ptr [[I]], align 4
+// SIMD-ONLY0-NEXT:    br label [[FOR_COND1:%.*]]
+// SIMD-ONLY0:       for.cond1:
+// SIMD-ONLY0-NEXT:    [[TMP3:%.*]] = load i32, ptr [[I]], align 4
+// SIMD-ONLY0-NEXT:    [[TMP4:%.*]] = load i32, ptr [[MP_ADDR]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP2:%.*]] = icmp slt i32 [[TMP3]], [[TMP4]]
+// SIMD-ONLY0-NEXT:    br i1 [[CMP2]], label [[FOR_BODY3:%.*]], label [[FOR_END9:%.*]]
+// SIMD-ONLY0:       for.body3:
+// SIMD-ONLY0-NEXT:    store i32 0, ptr [[I0]], align 4
+// SIMD-ONLY0-NEXT:    br label [[FOR_COND4:%.*]]
+// SIMD-ONLY0:       for.cond4:
+// SIMD-ONLY0-NEXT:    [[TMP5:%.*]] = load i32, ptr [[I0]], align 4
+// SIMD-ONLY0-NEXT:    [[CMP5:%.*]] = icmp slt i32 [[TMP5]], 10
+// SIMD-ONLY0-NEXT:    br i1 [[CMP5]], label [[FOR_BODY6:%.*]], label [[FOR_END:%.*]]
+// SIMD-ONLY0:       for.body6:
+// SIMD-ONLY0-NEXT:    br label [[FOR_INC:%.*]]
+// SIMD-ONLY0:       for.inc:
+// SIMD-ONLY0-NEXT:    [[TMP6:%.*]] = load i32, ptr [[I0]], align 4
+// SIMD-ONLY0-NEXT:    [[INC:%.*]] = add nsw i32 [[TMP6]], 1
+// SIMD-ONLY0-NEXT:    store i32 [[INC]], ptr [[I0]], align 4
+// SIMD-ONLY0-NEXT:    br label [[FOR_COND4]], !llvm.loop [[LOOP2:![0-9]+]]
+// SIMD-ONLY0:       for.end:
+// SIMD-ONLY0-NEXT:    br label [[FOR_INC7:%.*]]
+// SIMD-ONLY0:       for.inc7:
+// SIMD-ONLY0-NEXT:    [[TMP7:%.*]] = load i32, ptr [[I]], align 4
+// SIMD-ONLY0-NEXT:    [[INC8:%.*]] = add nsw i32 [[TMP7]], 1
+// SIMD-ONLY0-NEXT:    store i32 [[INC8]], ptr [[I]], align 4
+// SIMD-ONLY0-NEXT:    br label [[FOR_COND1]], !llvm.loop [[LOOP4:![0-9]+]]
+// SIMD-ONLY0:       for.end9:
+// SIMD-ONLY0-NEXT:    br label [[FOR_INC10:%.*]]
+// SIMD-ONLY0:       for.inc10:
+// SIMD-ONLY0-NEXT:    [[TMP8:%.*]] = load i32, ptr [[J]], align 4
+// SIMD-ONLY0-NEXT:    [[INC11:%.*]] = add nsw i32 [[TMP8]], 1
+// SIMD-ONLY0-NEXT:    store i32 [[INC11]], ptr [[J]], align 4
+// SIMD-ONLY0-NEXT:    br label [[FOR_COND]], !llvm.loop [[LOOP5:![0-9]+]]
+// SIMD-ONLY0:       for.end12:
+// SIMD-ONLY0-NEXT:    ret void
+//
diff --git a/clang/test/OpenMP/irbuilder_nested_openmp_parallel_empty.c b/clang/test/OpenMP/irbuilder_nested_openmp_parallel_empty.c
index b459cefa686c6..7bdb9749757cc 100644
--- a/clang/test/OpenMP/irbuilder_nested_openmp_parallel_empty.c
+++ b/clang/test/OpenMP/irbuilder_nested_openmp_parallel_empty.c
@@ -122,3 +122,5 @@ void nested_parallel_2(float *r, int a, double b) {
 }
 
 #endif
+//// NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+// IRBUILDER: {{.*}}
diff --git a/clang/test/OpenMP/nested_loop_codegen.cpp b/clang/test/OpenMP/nested_loop_codegen.cpp
index e38d9db29b8ee..f30ca705d79ec 100644
--- a/clang/test/OpenMP/nested_loop_codegen.cpp
+++ b/clang/test/OpenMP/nested_loop_codegen.cpp
@@ -728,91 +728,91 @@ int inline_decl() {
 // CHECK4-NEXT:    [[DOTSTOP:%.*]] = alloca i32, align 4
 // CHECK4-NEXT:    [[DOTSTEP:%.*]] = alloca i32, align 4
 // CHECK4-NEXT:    store ptr [[DISTANCE]], ptr [[DISTANCE_ADDR]], align 8
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[DISTANCE_ADDR]], metadata [[META47:![0-9]+]], metadata !DIExpression()), !dbg [[DBG48:![0-9]+]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[DISTANCE_ADDR]], metadata [[META46:![0-9]+]], metadata !DIExpression()), !dbg [[DBG47:![0-9]+]]
 // CHECK4-NEXT:    store ptr [[__CONTEXT]], ptr [[__CONTEXT_ADDR]], align 8
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[__CONTEXT_ADDR]], metadata [[META49:![0-9]+]], metadata !DIExpression()), !dbg [[DBG48]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[__CONTEXT_ADDR]], metadata [[META48:![0-9]+]], metadata !DIExpression()), !dbg [[DBG47]]
 // CHECK4-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[__CONTEXT_ADDR]], align 8
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[DOTSTART]], metadata [[META50:![0-9]+]], metadata !DIExpression()), !dbg [[DBG52:![0-9]+]]
-// CHECK4-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT_ANON:%.*]], ptr [[TMP0]], i32 0, i32 0, !dbg [[DBG53:![0-9]+]]
-// CHECK4-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[TMP1]], align 8, !dbg [[DBG53]]
-// CHECK4-NEXT:    [[TMP3:%.*]] = load i32, ptr [[TMP2]], align 4, !dbg [[DBG53]]
-// CHECK4-NEXT:    store i32 [[TMP3]], ptr [[DOTSTART]], align 4, !dbg [[DBG52]]
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[DOTSTOP]], metadata [[META55:![0-9]+]], metadata !DIExpression()), !dbg [[DBG56:![0-9]+]]
-// CHECK4-NEXT:    store i32 5, ptr [[DOTSTOP]], align 4, !dbg [[DBG56]]
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[DOTSTEP]], metadata [[META57:![0-9]+]], metadata !DIExpression()), !dbg [[DBG56]]
-// CHECK4-NEXT:    store i32 1, ptr [[DOTSTEP]], align 4, !dbg [[DBG56]]
-// CHECK4-NEXT:    [[TMP4:%.*]] = load i32, ptr [[DOTSTART]], align 4, !dbg [[DBG56]]
-// CHECK4-NEXT:    [[TMP5:%.*]] = load i32, ptr [[DOTSTOP]], align 4, !dbg [[DBG56]]
-// CHECK4-NEXT:    [[CMP:%.*]] = icmp slt i32 [[TMP4]], [[TMP5]], !dbg [[DBG56]]
-// CHECK4-NEXT:    br i1 [[CMP]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]], !dbg [[DBG56]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[DOTSTART]], metadata [[META49:![0-9]+]], metadata !DIExpression()), !dbg [[DBG51:![0-9]+]]
+// CHECK4-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT_ANON:%.*]], ptr [[TMP0]], i32 0, i32 0, !dbg [[DBG52:![0-9]+]]
+// CHECK4-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[TMP1]], align 8, !dbg [[DBG52]]
+// CHECK4-NEXT:    [[TMP3:%.*]] = load i32, ptr [[TMP2]], align 4, !dbg [[DBG52]]
+// CHECK4-NEXT:    store i32 [[TMP3]], ptr [[DOTSTART]], align 4, !dbg [[DBG51]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[DOTSTOP]], metadata [[META54:![0-9]+]], metadata !DIExpression()), !dbg [[DBG55:![0-9]+]]
+// CHECK4-NEXT:    store i32 5, ptr [[DOTSTOP]], align 4, !dbg [[DBG55]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[DOTSTEP]], metadata [[META56:![0-9]+]], metadata !DIExpression()), !dbg [[DBG55]]
+// CHECK4-NEXT:    store i32 1, ptr [[DOTSTEP]], align 4, !dbg [[DBG55]]
+// CHECK4-NEXT:    [[TMP4:%.*]] = load i32, ptr [[DOTSTART]], align 4, !dbg [[DBG55]]
+// CHECK4-NEXT:    [[TMP5:%.*]] = load i32, ptr [[DOTSTOP]], align 4, !dbg [[DBG55]]
+// CHECK4-NEXT:    [[CMP:%.*]] = icmp slt i32 [[TMP4]], [[TMP5]], !dbg [[DBG55]]
+// CHECK4-NEXT:    br i1 [[CMP]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]], !dbg [[DBG55]]
 // CHECK4:       cond.true:
-// CHECK4-NEXT:    [[TMP6:%.*]] = load i32, ptr [[DOTSTOP]], align 4, !dbg [[DBG56]]
-// CHECK4-NEXT:    [[TMP7:%.*]] = load i32, ptr [[DOTSTART]], align 4, !dbg [[DBG56]]
-// CHECK4-NEXT:    [[SUB:%.*]] = sub nsw i32 [[TMP6]], [[TMP7]], !dbg [[DBG56]]
-// CHECK4-NEXT:    [[TMP8:%.*]] = load i32, ptr [[DOTSTEP]], align 4, !dbg [[DBG56]]
-// CHECK4-NEXT:    [[SUB1:%.*]] = sub i32 [[TMP8]], 1, !dbg [[DBG56]]
-// CHECK4-NEXT:    [[ADD:%.*]] = add i32 [[SUB]], [[SUB1]], !dbg [[DBG56]]
-// CHECK4-NEXT:    [[TMP9:%.*]] = load i32, ptr [[DOTSTEP]], align 4, !dbg [[DBG56]]
-// CHECK4-NEXT:    [[DIV:%.*]] = udiv i32 [[ADD]], [[TMP9]], !dbg [[DBG56]]
-// CHECK4-NEXT:    br label [[COND_END:%.*]], !dbg [[DBG56]]
+// CHECK4-NEXT:    [[TMP6:%.*]] = load i32, ptr [[DOTSTOP]], align 4, !dbg [[DBG55]]
+// CHECK4-NEXT:    [[TMP7:%.*]] = load i32, ptr [[DOTSTART]], align 4, !dbg [[DBG55]]
+// CHECK4-NEXT:    [[SUB:%.*]] = sub nsw i32 [[TMP6]], [[TMP7]], !dbg [[DBG55]]
+// CHECK4-NEXT:    [[TMP8:%.*]] = load i32, ptr [[DOTSTEP]], align 4, !dbg [[DBG55]]
+// CHECK4-NEXT:    [[SUB1:%.*]] = sub i32 [[TMP8]], 1, !dbg [[DBG55]]
+// CHECK4-NEXT:    [[ADD:%.*]] = add i32 [[SUB]], [[SUB1]], !dbg [[DBG55]]
+// CHECK4-NEXT:    [[TMP9:%.*]] = load i32, ptr [[DOTSTEP]], align 4, !dbg [[DBG55]]
+// CHECK4-NEXT:    [[DIV:%.*]] = udiv i32 [[ADD]], [[TMP9]], !dbg [[DBG55]]
+// CHECK4-NEXT:    br label [[COND_END:%.*]], !dbg [[DBG55]]
 // CHECK4:       cond.false:
-// CHECK4-NEXT:    br label [[COND_END]], !dbg [[DBG56]]
+// CHECK4-NEXT:    br label [[COND_END]], !dbg [[DBG55]]
 // CHECK4:       cond.end:
-// CHECK4-NEXT:    [[COND:%.*]] = phi i32 [ [[DIV]], [[COND_TRUE]] ], [ 0, [[COND_FALSE]] ], !dbg [[DBG56]]
-// CHECK4-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[DISTANCE_ADDR]], align 8, !dbg [[DBG56]]
-// CHECK4-NEXT:    store i32 [[COND]], ptr [[TMP10]], align 4, !dbg [[DBG56]]
-// CHECK4-NEXT:    ret void, !dbg [[DBG58:![0-9]+]]
+// CHECK4-NEXT:    [[COND:%.*]] = phi i32 [ [[DIV]], [[COND_TRUE]] ], [ 0, [[COND_FALSE]] ], !dbg [[DBG55]]
+// CHECK4-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[DISTANCE_ADDR]], align 8, !dbg [[DBG55]]
+// CHECK4-NEXT:    store i32 [[COND]], ptr [[TMP10]], align 4, !dbg [[DBG55]]
+// CHECK4-NEXT:    ret void, !dbg [[DBG57:![0-9]+]]
 //
 //
 // CHECK4-LABEL: define {{[^@]+}}@__captured_stmt.1
-// CHECK4-SAME: (ptr noundef nonnull align 4 dereferenceable(4) [[LOOPVAR:%.*]], i32 noundef [[LOGICAL:%.*]], ptr noalias noundef [[__CONTEXT:%.*]]) #[[ATTR4]] !dbg [[DBG60:![0-9]+]] {
+// CHECK4-SAME: (ptr noundef nonnull align 4 dereferenceable(4) [[LOOPVAR:%.*]], i32 noundef [[LOGICAL:%.*]], ptr noalias noundef [[__CONTEXT:%.*]]) #[[ATTR4]] !dbg [[DBG59:![0-9]+]] {
 // CHECK4-NEXT:  entry:
 // CHECK4-NEXT:    [[LOOPVAR_ADDR:%.*]] = alloca ptr, align 8
 // CHECK4-NEXT:    [[LOGICAL_ADDR:%.*]] = alloca i32, align 4
 // CHECK4-NEXT:    [[__CONTEXT_ADDR:%.*]] = alloca ptr, align 8
 // CHECK4-NEXT:    store ptr [[LOOPVAR]], ptr [[LOOPVAR_ADDR]], align 8
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[LOOPVAR_ADDR]], metadata [[META68:![0-9]+]], metadata !DIExpression()), !dbg [[DBG69:![0-9]+]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[LOOPVAR_ADDR]], metadata [[META67:![0-9]+]], metadata !DIExpression()), !dbg [[DBG68:![0-9]+]]
 // CHECK4-NEXT:    store i32 [[LOGICAL]], ptr [[LOGICAL_ADDR]], align 4
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[LOGICAL_ADDR]], metadata [[META70:![0-9]+]], metadata !DIExpression()), !dbg [[DBG69]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[LOGICAL_ADDR]], metadata [[META69:![0-9]+]], metadata !DIExpression()), !dbg [[DBG68]]
 // CHECK4-NEXT:    store ptr [[__CONTEXT]], ptr [[__CONTEXT_ADDR]], align 8
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[__CONTEXT_ADDR]], metadata [[META71:![0-9]+]], metadata !DIExpression()), !dbg [[DBG69]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[__CONTEXT_ADDR]], metadata [[META70:![0-9]+]], metadata !DIExpression()), !dbg [[DBG68]]
 // CHECK4-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[__CONTEXT_ADDR]], align 8
-// CHECK4-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT_ANON_0:%.*]], ptr [[TMP0]], i32 0, i32 0, !dbg [[DBG72:![0-9]+]]
-// CHECK4-NEXT:    [[TMP2:%.*]] = load i32, ptr [[TMP1]], align 4, !dbg [[DBG72]]
-// CHECK4-NEXT:    [[TMP3:%.*]] = load i32, ptr [[LOGICAL_ADDR]], align 4, !dbg [[DBG74:![0-9]+]]
-// CHECK4-NEXT:    [[MUL:%.*]] = mul i32 1, [[TMP3]], !dbg [[DBG74]]
-// CHECK4-NEXT:    [[ADD:%.*]] = add i32 [[TMP2]], [[MUL]], !dbg [[DBG74]]
-// CHECK4-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[LOOPVAR_ADDR]], align 8, !dbg [[DBG74]]
-// CHECK4-NEXT:    store i32 [[ADD]], ptr [[TMP4]], align 4, !dbg [[DBG69]]
-// CHECK4-NEXT:    ret void, !dbg [[DBG72]]
+// CHECK4-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT_ANON_0:%.*]], ptr [[TMP0]], i32 0, i32 0, !dbg [[DBG71:![0-9]+]]
+// CHECK4-NEXT:    [[TMP2:%.*]] = load i32, ptr [[TMP1]], align 4, !dbg [[DBG71]]
+// CHECK4-NEXT:    [[TMP3:%.*]] = load i32, ptr [[LOGICAL_ADDR]], align 4, !dbg [[DBG73:![0-9]+]]
+// CHECK4-NEXT:    [[MUL:%.*]] = mul i32 1, [[TMP3]], !dbg [[DBG73]]
+// CHECK4-NEXT:    [[ADD:%.*]] = add i32 [[TMP2]], [[MUL]], !dbg [[DBG73]]
+// CHECK4-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[LOOPVAR_ADDR]], align 8, !dbg [[DBG73]]
+// CHECK4-NEXT:    store i32 [[ADD]], ptr [[TMP4]], align 4, !dbg [[DBG68]]
+// CHECK4-NEXT:    ret void, !dbg [[DBG71]]
 //
 //
 // CHECK4-LABEL: define {{[^@]+}}@_Z11inline_declv
-// CHECK4-SAME: () #[[ATTR0]] !dbg [[DBG77:![0-9]+]] {
+// CHECK4-SAME: () #[[ATTR0]] !dbg [[DBG76:![0-9]+]] {
 // CHECK4-NEXT:  entry:
 // CHECK4-NEXT:    [[STRUCTARG:%.*]] = alloca { ptr, ptr }, align 8
 // CHECK4-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK4-NEXT:    [[RES:%.*]] = alloca i32, align 4
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[I]], metadata [[META78:![0-9]+]], metadata !DIExpression()), !dbg [[DBG79:![0-9]+]]
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[RES]], metadata [[META80:![0-9]+]], metadata !DIExpression()), !dbg [[DBG79]]
-// CHECK4-NEXT:    [[OMP_GLOBAL_THREAD_NUM:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB3:[0-9]+]]), !dbg [[DBG81:![0-9]+]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[I]], metadata [[META77:![0-9]+]], metadata !DIExpression()), !dbg [[DBG78:![0-9]+]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[RES]], metadata [[META79:![0-9]+]], metadata !DIExpression()), !dbg [[DBG78]]
+// CHECK4-NEXT:    [[OMP_GLOBAL_THREAD_NUM:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB3:[0-9]+]]), !dbg [[DBG80:![0-9]+]]
 // CHECK4-NEXT:    br label [[OMP_PARALLEL:%.*]]
 // CHECK4:       omp_parallel:
 // CHECK4-NEXT:    [[GEP_I:%.*]] = getelementptr { ptr, ptr }, ptr [[STRUCTARG]], i32 0, i32 0
 // CHECK4-NEXT:    store ptr [[I]], ptr [[GEP_I]], align 8
 // CHECK4-NEXT:    [[GEP_RES:%.*]] = getelementptr { ptr, ptr }, ptr [[STRUCTARG]], i32 0, i32 1
 // CHECK4-NEXT:    store ptr [[RES]], ptr [[GEP_RES]], align 8
-// CHECK4-NEXT:    call void (ptr, i32, ptr, ...) @__kmpc_fork_call(ptr @[[GLOB3]], i32 1, ptr @_Z11inline_declv..omp_par, ptr [[STRUCTARG]]), !dbg [[DBG82:![0-9]+]]
+// CHECK4-NEXT:    call void (ptr, i32, ptr, ...) @__kmpc_fork_call(ptr @[[GLOB3]], i32 1, ptr @_Z11inline_declv..omp_par, ptr [[STRUCTARG]]), !dbg [[DBG81:![0-9]+]]
 // CHECK4-NEXT:    br label [[OMP_PAR_OUTLINED_EXIT:%.*]]
 // CHECK4:       omp.par.outlined.exit:
 // CHECK4-NEXT:    br label [[OMP_PAR_EXIT_SPLIT:%.*]]
 // CHECK4:       omp.par.exit.split:
-// CHECK4-NEXT:    [[TMP0:%.*]] = load i32, ptr [[RES]], align 4, !dbg [[DBG84:![0-9]+]]
-// CHECK4-NEXT:    ret i32 [[TMP0]], !dbg [[DBG84]]
+// CHECK4-NEXT:    [[TMP0:%.*]] = load i32, ptr [[RES]], align 4, !dbg [[DBG83:![0-9]+]]
+// CHECK4-NEXT:    ret i32 [[TMP0]], !dbg [[DBG83]]
 //
 //
 // CHECK4-LABEL: define {{[^@]+}}@_Z11inline_declv..omp_par
-// CHECK4-SAME: (ptr noalias [[TID_ADDR:%.*]], ptr noalias [[ZERO_ADDR:%.*]], ptr [[TMP0:%.*]]) #[[ATTR1]] !dbg [[DBG85:![0-9]+]] {
+// CHECK4-SAME: (ptr noalias [[TID_ADDR:%.*]], ptr noalias [[ZERO_ADDR:%.*]], ptr [[TMP0:%.*]]) #[[ATTR1]] !dbg [[DBG84:![0-9]+]] {
 // CHECK4-NEXT:  omp.par.entry:
 // CHECK4-NEXT:    [[GEP_I:%.*]] = getelementptr { ptr, ptr }, ptr [[TMP0]], i32 0, i32 0
 // CHECK4-NEXT:    [[LOADGEP_I:%.*]] = load ptr, ptr [[GEP_I]], align 8
@@ -828,61 +828,61 @@ int inline_decl() {
 // CHECK4-NEXT:    [[DOTCOUNT_ADDR:%.*]] = alloca i32, align 4
 // CHECK4-NEXT:    br label [[OMP_PAR_REGION:%.*]]
 // CHECK4:       omp.par.region:
-// CHECK4-NEXT:    store i32 0, ptr [[LOADGEP_I]], align 4, !dbg [[DBG86:![0-9]+]]
-// CHECK4-NEXT:    br label [[FOR_COND:%.*]], !dbg [[DBG86]]
+// CHECK4-NEXT:    store i32 0, ptr [[LOADGEP_I]], align 4, !dbg [[DBG85:![0-9]+]]
+// CHECK4-NEXT:    br label [[FOR_COND:%.*]], !dbg [[DBG85]]
 // CHECK4:       for.cond:
-// CHECK4-NEXT:    [[TMP2:%.*]] = load i32, ptr [[LOADGEP_I]], align 4, !dbg [[DBG88:![0-9]+]]
-// CHECK4-NEXT:    [[CMP:%.*]] = icmp slt i32 [[TMP2]], 10, !dbg [[DBG88]]
-// CHECK4-NEXT:    br i1 [[CMP]], label [[FOR_BODY:%.*]], label [[FOR_END:%.*]], !dbg [[DBG86]]
+// CHECK4-NEXT:    [[TMP2:%.*]] = load i32, ptr [[LOADGEP_I]], align 4, !dbg [[DBG87:![0-9]+]]
+// CHECK4-NEXT:    [[CMP:%.*]] = icmp slt i32 [[TMP2]], 10, !dbg [[DBG87]]
+// CHECK4-NEXT:    br i1 [[CMP]], label [[FOR_BODY:%.*]], label [[FOR_END:%.*]], !dbg [[DBG85]]
 // CHECK4:       for.end:
-// CHECK4-NEXT:    br label [[OMP_PAR_REGION_PARALLEL_AFTER:%.*]], !dbg [[DBG90:![0-9]+]]
+// CHECK4-NEXT:    br label [[OMP_PAR_REGION_PARALLEL_AFTER:%.*]], !dbg [[DBG89:![0-9]+]]
 // CHECK4:       omp.par.region.parallel.after:
 // CHECK4-NEXT:    br label [[OMP_PAR_PRE_FINALIZE:%.*]]
 // CHECK4:       omp.par.pre_finalize:
-// CHECK4-NEXT:    br label [[OMP_PAR_OUTLINED_EXIT_EXITSTUB:%.*]], !dbg [[DBG90]]
+// CHECK4-NEXT:    br label [[OMP_PAR_OUTLINED_EXIT_EXITSTUB:%.*]], !dbg [[DBG89]]
 // CHECK4:       for.body:
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[K]], metadata [[META91:![0-9]+]], metadata !DIExpression()), !dbg [[DBG95:![0-9]+]]
-// CHECK4-NEXT:    store i32 0, ptr [[K]], align 4, !dbg [[DBG95]]
-// CHECK4-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [[STRUCT_ANON_1]], ptr [[AGG_CAPTURED]], i32 0, i32 0, !dbg [[DBG95]]
-// CHECK4-NEXT:    store ptr [[K]], ptr [[TMP3]], align 8, !dbg [[DBG95]]
-// CHECK4-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [[STRUCT_ANON_2]], ptr [[AGG_CAPTURED1]], i32 0, i32 0, !dbg [[DBG95]]
-// CHECK4-NEXT:    [[TMP5:%.*]] = load i32, ptr [[K]], align 4, !dbg [[DBG96:![0-9]+]]
-// CHECK4-NEXT:    store i32 [[TMP5]], ptr [[TMP4]], align 4, !dbg [[DBG95]]
-// CHECK4-NEXT:    call void @__captured_stmt.2(ptr [[DOTCOUNT_ADDR]], ptr [[AGG_CAPTURED]]), !dbg [[DBG95]]
-// CHECK4-NEXT:    [[DOTCOUNT:%.*]] = load i32, ptr [[DOTCOUNT_ADDR]], align 4, !dbg [[DBG95]]
-// CHECK4-NEXT:    br label [[OMP_LOOP_PREHEADER:%.*]], !dbg [[DBG95]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[K]], metadata [[META90:![0-9]+]], metadata !DIExpression()), !dbg [[DBG94:![0-9]+]]
+// CHECK4-NEXT:    store i32 0, ptr [[K]], align 4, !dbg [[DBG94]]
+// CHECK4-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [[STRUCT_ANON_1]], ptr [[AGG_CAPTURED]], i32 0, i32 0, !dbg [[DBG94]]
+// CHECK4-NEXT:    store ptr [[K]], ptr [[TMP3]], align 8, !dbg [[DBG94]]
+// CHECK4-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [[STRUCT_ANON_2]], ptr [[AGG_CAPTURED1]], i32 0, i32 0, !dbg [[DBG94]]
+// CHECK4-NEXT:    [[TMP5:%.*]] = load i32, ptr [[K]], align 4, !dbg [[DBG95:![0-9]+]]
+// CHECK4-NEXT:    store i32 [[TMP5]], ptr [[TMP4]], align 4, !dbg [[DBG94]]
+// CHECK4-NEXT:    call void @__captured_stmt.2(ptr [[DOTCOUNT_ADDR]], ptr [[AGG_CAPTURED]]), !dbg [[DBG94]]
+// CHECK4-NEXT:    [[DOTCOUNT:%.*]] = load i32, ptr [[DOTCOUNT_ADDR]], align 4, !dbg [[DBG94]]
+// CHECK4-NEXT:    br label [[OMP_LOOP_PREHEADER:%.*]], !dbg [[DBG94]]
 // CHECK4:       omp_loop.preheader:
-// CHECK4-NEXT:    br label [[OMP_LOOP_HEADER:%.*]], !dbg [[DBG95]]
+// CHECK4-NEXT:    br label [[OMP_LOOP_HEADER:%.*]], !dbg [[DBG94]]
 // CHECK4:       omp_loop.header:
-// CHECK4-NEXT:    [[OMP_LOOP_IV:%.*]] = phi i32 [ 0, [[OMP_LOOP_PREHEADER]] ], [ [[OMP_LOOP_NEXT:%.*]], [[OMP_LOOP_INC:%.*]] ], !dbg [[DBG95]]
-// CHECK4-NEXT:    br label [[OMP_LOOP_COND:%.*]], !dbg [[DBG95]]
+// CHECK4-NEXT:    [[OMP_LOOP_IV:%.*]] = phi i32 [ 0, [[OMP_LOOP_PREHEADER]] ], [ [[OMP_LOOP_NEXT:%.*]], [[OMP_LOOP_INC:%.*]] ], !dbg [[DBG94]]
+// CHECK4-NEXT:    br label [[OMP_LOOP_COND:%.*]], !dbg [[DBG94]]
 // CHECK4:       omp_loop.cond:
-// CHECK4-NEXT:    [[OMP_LOOP_CMP:%.*]] = icmp ult i32 [[OMP_LOOP_IV]], [[DOTCOUNT]], !dbg [[DBG95]]
-// CHECK4-NEXT:    br i1 [[OMP_LOOP_CMP]], label [[OMP_LOOP_BODY:%.*]], label [[OMP_LOOP_EXIT:%.*]], !dbg [[DBG95]]
+// CHECK4-NEXT:    [[OMP_LOOP_CMP:%.*]] = icmp ult i32 [[OMP_LOOP_IV]], [[DOTCOUNT]], !dbg [[DBG94]]
+// CHECK4-NEXT:    br i1 [[OMP_LOOP_CMP]], label [[OMP_LOOP_BODY:%.*]], label [[OMP_LOOP_EXIT:%.*]], !dbg [[DBG94]]
 // CHECK4:       omp_loop.exit:
-// CHECK4-NEXT:    br label [[OMP_LOOP_AFTER:%.*]], !dbg [[DBG95]]
+// CHECK4-NEXT:    br label [[OMP_LOOP_AFTER:%.*]], !dbg [[DBG94]]
 // CHECK4:       omp_loop.after:
-// CHECK4-NEXT:    br label [[FOR_INC:%.*]], !dbg [[DBG97:![0-9]+]]
+// CHECK4-NEXT:    br label [[FOR_INC:%.*]], !dbg [[DBG96:![0-9]+]]
 // CHECK4:       for.inc:
-// CHECK4-NEXT:    [[TMP6:%.*]] = load i32, ptr [[LOADGEP_I]], align 4, !dbg [[DBG88]]
-// CHECK4-NEXT:    [[INC2:%.*]] = add nsw i32 [[TMP6]], 1, !dbg [[DBG88]]
-// CHECK4-NEXT:    store i32 [[INC2]], ptr [[LOADGEP_I]], align 4, !dbg [[DBG88]]
-// CHECK4-NEXT:    br label [[FOR_COND]], !dbg [[DBG88]], !llvm.loop [[LOOP98:![0-9]+]]
+// CHECK4-NEXT:    [[TMP6:%.*]] = load i32, ptr [[LOADGEP_I]], align 4, !dbg [[DBG87]]
+// CHECK4-NEXT:    [[INC2:%.*]] = add nsw i32 [[TMP6]], 1, !dbg [[DBG87]]
+// CHECK4-NEXT:    store i32 [[INC2]], ptr [[LOADGEP_I]], align 4, !dbg [[DBG87]]
+// CHECK4-NEXT:    br label [[FOR_COND]], !dbg [[DBG87]], !llvm.loop [[LOOP97:![0-9]+]]
 // CHECK4:       omp_loop.body:
-// CHECK4-NEXT:    call void @__captured_stmt.3(ptr [[K]], i32 [[OMP_LOOP_IV]], ptr [[AGG_CAPTURED1]]), !dbg [[DBG95]]
-// CHECK4-NEXT:    [[TMP7:%.*]] = load i32, ptr [[LOADGEP_RES]], align 4, !dbg [[DBG99:![0-9]+]]
-// CHECK4-NEXT:    [[INC:%.*]] = add nsw i32 [[TMP7]], 1, !dbg [[DBG99]]
-// CHECK4-NEXT:    store i32 [[INC]], ptr [[LOADGEP_RES]], align 4, !dbg [[DBG99]]
-// CHECK4-NEXT:    br label [[OMP_LOOP_INC]], !dbg [[DBG95]]
+// CHECK4-NEXT:    call void @__captured_stmt.3(ptr [[K]], i32 [[OMP_LOOP_IV]], ptr [[AGG_CAPTURED1]]), !dbg [[DBG94]]
+// CHECK4-NEXT:    [[TMP7:%.*]] = load i32, ptr [[LOADGEP_RES]], align 4, !dbg [[DBG98:![0-9]+]]
+// CHECK4-NEXT:    [[INC:%.*]] = add nsw i32 [[TMP7]], 1, !dbg [[DBG98]]
+// CHECK4-NEXT:    store i32 [[INC]], ptr [[LOADGEP_RES]], align 4, !dbg [[DBG98]]
+// CHECK4-NEXT:    br label [[OMP_LOOP_INC]], !dbg [[DBG94]]
 // CHECK4:       omp_loop.inc:
-// CHECK4-NEXT:    [[OMP_LOOP_NEXT]] = add nuw i32 [[OMP_LOOP_IV]], 1, !dbg [[DBG95]]
-// CHECK4-NEXT:    br label [[OMP_LOOP_HEADER]], !dbg [[DBG95]]
+// CHECK4-NEXT:    [[OMP_LOOP_NEXT]] = add nuw i32 [[OMP_LOOP_IV]], 1, !dbg [[DBG94]]
+// CHECK4-NEXT:    br label [[OMP_LOOP_HEADER]], !dbg [[DBG94]]
 // CHECK4:       omp.par.outlined.exit.exitStub:
 // CHECK4-NEXT:    ret void
 //
 //
 // CHECK4-LABEL: define {{[^@]+}}@__captured_stmt.2
-// CHECK4-SAME: (ptr noundef nonnull align 4 dereferenceable(4) [[DISTANCE:%.*]], ptr noalias noundef [[__CONTEXT:%.*]]) #[[ATTR4]] !dbg [[DBG101:![0-9]+]] {
+// CHECK4-SAME: (ptr noundef nonnull align 4 dereferenceable(4) [[DISTANCE:%.*]], ptr noalias noundef [[__CONTEXT:%.*]]) #[[ATTR4]] !dbg [[DBG100:![0-9]+]] {
 // CHECK4-NEXT:  entry:
 // CHECK4-NEXT:    [[DISTANCE_ADDR:%.*]] = alloca ptr, align 8
 // CHECK4-NEXT:    [[__CONTEXT_ADDR:%.*]] = alloca ptr, align 8
@@ -890,61 +890,61 @@ int inline_decl() {
 // CHECK4-NEXT:    [[DOTSTOP:%.*]] = alloca i32, align 4
 // CHECK4-NEXT:    [[DOTSTEP:%.*]] = alloca i32, align 4
 // CHECK4-NEXT:    store ptr [[DISTANCE]], ptr [[DISTANCE_ADDR]], align 8
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[DISTANCE_ADDR]], metadata [[META102:![0-9]+]], metadata !DIExpression()), !dbg [[DBG103:![0-9]+]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[DISTANCE_ADDR]], metadata [[META101:![0-9]+]], metadata !DIExpression()), !dbg [[DBG102:![0-9]+]]
 // CHECK4-NEXT:    store ptr [[__CONTEXT]], ptr [[__CONTEXT_ADDR]], align 8
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[__CONTEXT_ADDR]], metadata [[META104:![0-9]+]], metadata !DIExpression()), !dbg [[DBG103]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[__CONTEXT_ADDR]], metadata [[META103:![0-9]+]], metadata !DIExpression()), !dbg [[DBG102]]
 // CHECK4-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[__CONTEXT_ADDR]], align 8
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[DOTSTART]], metadata [[META105:![0-9]+]], metadata !DIExpression()), !dbg [[DBG107:![0-9]+]]
-// CHECK4-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT_ANON_1:%.*]], ptr [[TMP0]], i32 0, i32 0, !dbg [[DBG108:![0-9]+]]
-// CHECK4-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[TMP1]], align 8, !dbg [[DBG108]]
-// CHECK4-NEXT:    [[TMP3:%.*]] = load i32, ptr [[TMP2]], align 4, !dbg [[DBG108]]
-// CHECK4-NEXT:    store i32 [[TMP3]], ptr [[DOTSTART]], align 4, !dbg [[DBG107]]
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[DOTSTOP]], metadata [[META110:![0-9]+]], metadata !DIExpression()), !dbg [[DBG111:![0-9]+]]
-// CHECK4-NEXT:    store i32 5, ptr [[DOTSTOP]], align 4, !dbg [[DBG111]]
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[DOTSTEP]], metadata [[META112:![0-9]+]], metadata !DIExpression()), !dbg [[DBG111]]
-// CHECK4-NEXT:    store i32 1, ptr [[DOTSTEP]], align 4, !dbg [[DBG111]]
-// CHECK4-NEXT:    [[TMP4:%.*]] = load i32, ptr [[DOTSTART]], align 4, !dbg [[DBG111]]
-// CHECK4-NEXT:    [[TMP5:%.*]] = load i32, ptr [[DOTSTOP]], align 4, !dbg [[DBG111]]
-// CHECK4-NEXT:    [[CMP:%.*]] = icmp slt i32 [[TMP4]], [[TMP5]], !dbg [[DBG111]]
-// CHECK4-NEXT:    br i1 [[CMP]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]], !dbg [[DBG111]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[DOTSTART]], metadata [[META104:![0-9]+]], metadata !DIExpression()), !dbg [[DBG106:![0-9]+]]
+// CHECK4-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT_ANON_1:%.*]], ptr [[TMP0]], i32 0, i32 0, !dbg [[DBG107:![0-9]+]]
+// CHECK4-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[TMP1]], align 8, !dbg [[DBG107]]
+// CHECK4-NEXT:    [[TMP3:%.*]] = load i32, ptr [[TMP2]], align 4, !dbg [[DBG107]]
+// CHECK4-NEXT:    store i32 [[TMP3]], ptr [[DOTSTART]], align 4, !dbg [[DBG106]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[DOTSTOP]], metadata [[META109:![0-9]+]], metadata !DIExpression()), !dbg [[DBG110:![0-9]+]]
+// CHECK4-NEXT:    store i32 5, ptr [[DOTSTOP]], align 4, !dbg [[DBG110]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[DOTSTEP]], metadata [[META111:![0-9]+]], metadata !DIExpression()), !dbg [[DBG110]]
+// CHECK4-NEXT:    store i32 1, ptr [[DOTSTEP]], align 4, !dbg [[DBG110]]
+// CHECK4-NEXT:    [[TMP4:%.*]] = load i32, ptr [[DOTSTART]], align 4, !dbg [[DBG110]]
+// CHECK4-NEXT:    [[TMP5:%.*]] = load i32, ptr [[DOTSTOP]], align 4, !dbg [[DBG110]]
+// CHECK4-NEXT:    [[CMP:%.*]] = icmp slt i32 [[TMP4]], [[TMP5]], !dbg [[DBG110]]
+// CHECK4-NEXT:    br i1 [[CMP]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]], !dbg [[DBG110]]
 // CHECK4:       cond.true:
-// CHECK4-NEXT:    [[TMP6:%.*]] = load i32, ptr [[DOTSTOP]], align 4, !dbg [[DBG111]]
-// CHECK4-NEXT:    [[TMP7:%.*]] = load i32, ptr [[DOTSTART]], align 4, !dbg [[DBG111]]
-// CHECK4-NEXT:    [[SUB:%.*]] = sub nsw i32 [[TMP6]], [[TMP7]], !dbg [[DBG111]]
-// CHECK4-NEXT:    [[TMP8:%.*]] = load i32, ptr [[DOTSTEP]], align 4, !dbg [[DBG111]]
-// CHECK4-NEXT:    [[SUB1:%.*]] = sub i32 [[TMP8]], 1, !dbg [[DBG111]]
-// CHECK4-NEXT:    [[ADD:%.*]] = add i32 [[SUB]], [[SUB1]], !dbg [[DBG111]]
-// CHECK4-NEXT:    [[TMP9:%.*]] = load i32, ptr [[DOTSTEP]], align 4, !dbg [[DBG111]]
-// CHECK4-NEXT:    [[DIV:%.*]] = udiv i32 [[ADD]], [[TMP9]], !dbg [[DBG111]]
-// CHECK4-NEXT:    br label [[COND_END:%.*]], !dbg [[DBG111]]
+// CHECK4-NEXT:    [[TMP6:%.*]] = load i32, ptr [[DOTSTOP]], align 4, !dbg [[DBG110]]
+// CHECK4-NEXT:    [[TMP7:%.*]] = load i32, ptr [[DOTSTART]], align 4, !dbg [[DBG110]]
+// CHECK4-NEXT:    [[SUB:%.*]] = sub nsw i32 [[TMP6]], [[TMP7]], !dbg [[DBG110]]
+// CHECK4-NEXT:    [[TMP8:%.*]] = load i32, ptr [[DOTSTEP]], align 4, !dbg [[DBG110]]
+// CHECK4-NEXT:    [[SUB1:%.*]] = sub i32 [[TMP8]], 1, !dbg [[DBG110]]
+// CHECK4-NEXT:    [[ADD:%.*]] = add i32 [[SUB]], [[SUB1]], !dbg [[DBG110]]
+// CHECK4-NEXT:    [[TMP9:%.*]] = load i32, ptr [[DOTSTEP]], align 4, !dbg [[DBG110]]
+// CHECK4-NEXT:    [[DIV:%.*]] = udiv i32 [[ADD]], [[TMP9]], !dbg [[DBG110]]
+// CHECK4-NEXT:    br label [[COND_END:%.*]], !dbg [[DBG110]]
 // CHECK4:       cond.false:
-// CHECK4-NEXT:    br label [[COND_END]], !dbg [[DBG111]]
+// CHECK4-NEXT:    br label [[COND_END]], !dbg [[DBG110]]
 // CHECK4:       cond.end:
-// CHECK4-NEXT:    [[COND:%.*]] = phi i32 [ [[DIV]], [[COND_TRUE]] ], [ 0, [[COND_FALSE]] ], !dbg [[DBG111]]
-// CHECK4-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[DISTANCE_ADDR]], align 8, !dbg [[DBG111]]
-// CHECK4-NEXT:    store i32 [[COND]], ptr [[TMP10]], align 4, !dbg [[DBG111]]
-// CHECK4-NEXT:    ret void, !dbg [[DBG113:![0-9]+]]
+// CHECK4-NEXT:    [[COND:%.*]] = phi i32 [ [[DIV]], [[COND_TRUE]] ], [ 0, [[COND_FALSE]] ], !dbg [[DBG110]]
+// CHECK4-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[DISTANCE_ADDR]], align 8, !dbg [[DBG110]]
+// CHECK4-NEXT:    store i32 [[COND]], ptr [[TMP10]], align 4, !dbg [[DBG110]]
+// CHECK4-NEXT:    ret void, !dbg [[DBG112:![0-9]+]]
 //
 //
 // CHECK4-LABEL: define {{[^@]+}}@__captured_stmt.3
-// CHECK4-SAME: (ptr noundef nonnull align 4 dereferenceable(4) [[LOOPVAR:%.*]], i32 noundef [[LOGICAL:%.*]], ptr noalias noundef [[__CONTEXT:%.*]]) #[[ATTR4]] !dbg [[DBG115:![0-9]+]] {
+// CHECK4-SAME: (ptr noundef nonnull align 4 dereferenceable(4) [[LOOPVAR:%.*]], i32 noundef [[LOGICAL:%.*]], ptr noalias noundef [[__CONTEXT:%.*]]) #[[ATTR4]] !dbg [[DBG114:![0-9]+]] {
 // CHECK4-NEXT:  entry:
 // CHECK4-NEXT:    [[LOOPVAR_ADDR:%.*]] = alloca ptr, align 8
 // CHECK4-NEXT:    [[LOGICAL_ADDR:%.*]] = alloca i32, align 4
 // CHECK4-NEXT:    [[__CONTEXT_ADDR:%.*]] = alloca ptr, align 8
 // CHECK4-NEXT:    store ptr [[LOOPVAR]], ptr [[LOOPVAR_ADDR]], align 8
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[LOOPVAR_ADDR]], metadata [[META116:![0-9]+]], metadata !DIExpression()), !dbg [[DBG117:![0-9]+]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[LOOPVAR_ADDR]], metadata [[META115:![0-9]+]], metadata !DIExpression()), !dbg [[DBG116:![0-9]+]]
 // CHECK4-NEXT:    store i32 [[LOGICAL]], ptr [[LOGICAL_ADDR]], align 4
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[LOGICAL_ADDR]], metadata [[META118:![0-9]+]], metadata !DIExpression()), !dbg [[DBG117]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[LOGICAL_ADDR]], metadata [[META117:![0-9]+]], metadata !DIExpression()), !dbg [[DBG116]]
 // CHECK4-NEXT:    store ptr [[__CONTEXT]], ptr [[__CONTEXT_ADDR]], align 8
-// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[__CONTEXT_ADDR]], metadata [[META119:![0-9]+]], metadata !DIExpression()), !dbg [[DBG117]]
+// CHECK4-NEXT:    call void @llvm.dbg.declare(metadata ptr [[__CONTEXT_ADDR]], metadata [[META118:![0-9]+]], metadata !DIExpression()), !dbg [[DBG116]]
 // CHECK4-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[__CONTEXT_ADDR]], align 8
-// CHECK4-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT_ANON_2:%.*]], ptr [[TMP0]], i32 0, i32 0, !dbg [[DBG120:![0-9]+]]
-// CHECK4-NEXT:    [[TMP2:%.*]] = load i32, ptr [[TMP1]], align 4, !dbg [[DBG120]]
-// CHECK4-NEXT:    [[TMP3:%.*]] = load i32, ptr [[LOGICAL_ADDR]], align 4, !dbg [[DBG122:![0-9]+]]
-// CHECK4-NEXT:    [[MUL:%.*]] = mul i32 1, [[TMP3]], !dbg [[DBG122]]
-// CHECK4-NEXT:    [[ADD:%.*]] = add i32 [[TMP2]], [[MUL]], !dbg [[DBG122]]
-// CHECK4-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[LOOPVAR_ADDR]], align 8, !dbg [[DBG122]]
-// CHECK4-NEXT:    store i32 [[ADD]], ptr [[TMP4]], align 4, !dbg [[DBG117]]
-// CHECK4-NEXT:    ret void, !dbg [[DBG120]]
+// CHECK4-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT_ANON_2:%.*]], ptr [[TMP0]], i32 0, i32 0, !dbg [[DBG119:![0-9]+]]
+// CHECK4-NEXT:    [[TMP2:%.*]] = load i32, ptr [[TMP1]], align 4, !dbg [[DBG119]]
+// CHECK4-NEXT:    [[TMP3:%.*]] = load i32, ptr [[LOGICAL_ADDR]], align 4, !dbg [[DBG121:![0-9]+]]
+// CHECK4-NEXT:    [[MUL:%.*]] = mul i32 1, [[TMP3]], !dbg [[DBG121]]
+// CHECK4-NEXT:    [[ADD:%.*]] = add i32 [[TMP2]], [[MUL]], !dbg [[DBG121]]
+// CHECK4-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[LOOPVAR_ADDR]], align 8, !dbg [[DBG121]]
+// CHECK4-NEXT:    store i32 [[ADD]], ptr [[TMP4]], align 4, !dbg [[DBG116]]
+// CHECK4-NEXT:    ret void, !dbg [[DBG119]]
 //
diff --git a/clang/test/OpenMP/nvptx_lambda_capturing.cpp b/clang/test/OpenMP/nvptx_lambda_capturing.cpp
index 4a6de49e78cf0..9c1317dbc40f3 100644
--- a/clang/test/OpenMP/nvptx_lambda_capturing.cpp
+++ b/clang/test/OpenMP/nvptx_lambda_capturing.cpp
@@ -66,12 +66,14 @@ int main(int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [11 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [11 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [11 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP4:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[_TMP5:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[_TMP6:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS7:%.*]] = alloca [11 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS8:%.*]] = alloca [11 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS9:%.*]] = alloca [11 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS10:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK1-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -188,7 +190,6 @@ int main(int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP63]], align 8
 // CHECK1-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [11 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [11 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP66]], align 4
 // CHECK1-NEXT:    [[TMP67:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -315,7 +316,6 @@ int main(int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP135]], align 8
 // CHECK1-NEXT:    [[TMP136:%.*]] = getelementptr inbounds [11 x ptr], ptr [[DOTOFFLOAD_BASEPTRS7]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP137:%.*]] = getelementptr inbounds [11 x ptr], ptr [[DOTOFFLOAD_PTRS8]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS10:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP138:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS10]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP138]], align 4
 // CHECK1-NEXT:    [[TMP139:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS10]], i32 0, i32 1
@@ -504,10 +504,12 @@ int main(int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP2:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[CLASS_ANON_0]], ptr [[REF_TMP]], i32 0, i32 0
@@ -538,7 +540,6 @@ int main(int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP13]], align 8
 // CHECK1-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -597,7 +598,6 @@ int main(int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP43]], align 8
 // CHECK1-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP46]], align 4
 // CHECK1-NEXT:    [[TMP47:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -722,6 +722,7 @@ int main(int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [2 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[T]], ptr [[T_ADDR]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[T_ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[TMP0]], ptr [[TMP]], align 8
@@ -742,7 +743,6 @@ int main(int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP9]], align 8
 // CHECK1-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP12]], align 4
 // CHECK1-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/nvptx_target_parallel_proc_bind_codegen.cpp b/clang/test/OpenMP/nvptx_target_parallel_proc_bind_codegen.cpp
index b2cf555cafacf..c9639be846503 100644
--- a/clang/test/OpenMP/nvptx_target_parallel_proc_bind_codegen.cpp
+++ b/clang/test/OpenMP/nvptx_target_parallel_proc_bind_codegen.cpp
@@ -52,6 +52,130 @@ int bar(int n){
 }
 
 #endif
+// CHECK-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9ftemplateIiET_i_l27
+// CHECK-SAME: () #[[ATTR0:[0-9]+]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[CAPTURED_VARS_ADDRS:%.*]] = alloca [0 x ptr], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_target_init(ptr @[[GLOB1:[0-9]+]], i8 2, i1 false)
+// CHECK-NEXT:    [[EXEC_USER_CODE:%.*]] = icmp eq i32 [[TMP0]], -1
+// CHECK-NEXT:    br i1 [[EXEC_USER_CODE]], label [[USER_CODE_ENTRY:%.*]], label [[WORKER_EXIT:%.*]]
+// CHECK:       user_code.entry:
+// CHECK-NEXT:    [[TMP1:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB1]])
+// CHECK-NEXT:    call void @__kmpc_parallel_51(ptr @[[GLOB1]], i32 [[TMP1]], i32 1, i32 -1, i32 -1, ptr @__omp_outlined__, ptr null, ptr [[CAPTURED_VARS_ADDRS]], i32 0)
+// CHECK-NEXT:    call void @__kmpc_target_deinit(ptr @[[GLOB1]], i8 2)
+// CHECK-NEXT:    ret void
+// CHECK:       worker.exit:
+// CHECK-NEXT:    ret void
+// CHECK-LABEL: define {{[^@]+}}@__omp_outlined__
+// CHECK-SAME: (ptr noalias noundef [[DOTGLOBAL_TID_:%.*]], ptr noalias noundef [[DOTBOUND_TID_:%.*]]) #[[ATTR1:[0-9]+]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DOTGLOBAL_TID__ADDR:%.*]] = alloca ptr, align 4
+// CHECK-NEXT:    [[DOTBOUND_TID__ADDR:%.*]] = alloca ptr, align 4
+// CHECK-NEXT:    store ptr [[DOTGLOBAL_TID_]], ptr [[DOTGLOBAL_TID__ADDR]], align 4
+// CHECK-NEXT:    store ptr [[DOTBOUND_TID_]], ptr [[DOTBOUND_TID__ADDR]], align 4
+// CHECK-NEXT:    ret void
+// CHECK-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9ftemplateIiET_i_l31
+// CHECK-SAME: (i32 noundef [[AA:%.*]]) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[AA_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[AA_CASTED:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[CAPTURED_VARS_ADDRS:%.*]] = alloca [1 x ptr], align 4
+// CHECK-NEXT:    store i32 [[AA]], ptr [[AA_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_target_init(ptr @[[GLOB1]], i8 2, i1 false)
+// CHECK-NEXT:    [[EXEC_USER_CODE:%.*]] = icmp eq i32 [[TMP0]], -1
+// CHECK-NEXT:    br i1 [[EXEC_USER_CODE]], label [[USER_CODE_ENTRY:%.*]], label [[WORKER_EXIT:%.*]]
+// CHECK:       user_code.entry:
+// CHECK-NEXT:    [[TMP1:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB1]])
+// CHECK-NEXT:    [[TMP2:%.*]] = load i16, ptr [[AA_ADDR]], align 2
+// CHECK-NEXT:    store i16 [[TMP2]], ptr [[AA_CASTED]], align 2
+// CHECK-NEXT:    [[TMP3:%.*]] = load i32, ptr [[AA_CASTED]], align 4
+// CHECK-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[CAPTURED_VARS_ADDRS]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP5:%.*]] = inttoptr i32 [[TMP3]] to ptr
+// CHECK-NEXT:    store ptr [[TMP5]], ptr [[TMP4]], align 4
+// CHECK-NEXT:    call void @__kmpc_parallel_51(ptr @[[GLOB1]], i32 [[TMP1]], i32 1, i32 -1, i32 -1, ptr @__omp_outlined__1, ptr null, ptr [[CAPTURED_VARS_ADDRS]], i32 1)
+// CHECK-NEXT:    call void @__kmpc_target_deinit(ptr @[[GLOB1]], i8 2)
+// CHECK-NEXT:    ret void
+// CHECK:       worker.exit:
+// CHECK-NEXT:    ret void
+// CHECK-LABEL: define {{[^@]+}}@__omp_outlined__1
+// CHECK-SAME: (ptr noalias noundef [[DOTGLOBAL_TID_:%.*]], ptr noalias noundef [[DOTBOUND_TID_:%.*]], i32 noundef [[AA:%.*]]) #[[ATTR1]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DOTGLOBAL_TID__ADDR:%.*]] = alloca ptr, align 4
+// CHECK-NEXT:    [[DOTBOUND_TID__ADDR:%.*]] = alloca ptr, align 4
+// CHECK-NEXT:    [[AA_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    store ptr [[DOTGLOBAL_TID_]], ptr [[DOTGLOBAL_TID__ADDR]], align 4
+// CHECK-NEXT:    store ptr [[DOTBOUND_TID_]], ptr [[DOTBOUND_TID__ADDR]], align 4
+// CHECK-NEXT:    store i32 [[AA]], ptr [[AA_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load i16, ptr [[AA_ADDR]], align 2
+// CHECK-NEXT:    [[CONV:%.*]] = sext i16 [[TMP0]] to i32
+// CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[CONV]], 1
+// CHECK-NEXT:    [[CONV1:%.*]] = trunc i32 [[ADD]] to i16
+// CHECK-NEXT:    store i16 [[CONV1]], ptr [[AA_ADDR]], align 2
+// CHECK-NEXT:    ret void
+// CHECK-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9ftemplateIiET_i_l36
+// CHECK-SAME: (i32 noundef [[A:%.*]], i32 noundef [[AA:%.*]], ptr noundef nonnull align 4 dereferenceable(40) [[B:%.*]]) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[AA_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca ptr, align 4
+// CHECK-NEXT:    [[A_CASTED:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[AA_CASTED:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[CAPTURED_VARS_ADDRS:%.*]] = alloca [3 x ptr], align 4
+// CHECK-NEXT:    store i32 [[A]], ptr [[A_ADDR]], align 4
+// CHECK-NEXT:    store i32 [[AA]], ptr [[AA_ADDR]], align 4
+// CHECK-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[B_ADDR]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = call i32 @__kmpc_target_init(ptr @[[GLOB1]], i8 2, i1 false)
+// CHECK-NEXT:    [[EXEC_USER_CODE:%.*]] = icmp eq i32 [[TMP1]], -1
+// CHECK-NEXT:    br i1 [[EXEC_USER_CODE]], label [[USER_CODE_ENTRY:%.*]], label [[WORKER_EXIT:%.*]]
+// CHECK:       user_code.entry:
+// CHECK-NEXT:    [[TMP2:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB1]])
+// CHECK-NEXT:    [[TMP3:%.*]] = load i32, ptr [[A_ADDR]], align 4
+// CHECK-NEXT:    store i32 [[TMP3]], ptr [[A_CASTED]], align 4
+// CHECK-NEXT:    [[TMP4:%.*]] = load i32, ptr [[A_CASTED]], align 4
+// CHECK-NEXT:    [[TMP5:%.*]] = load i16, ptr [[AA_ADDR]], align 2
+// CHECK-NEXT:    store i16 [[TMP5]], ptr [[AA_CASTED]], align 2
+// CHECK-NEXT:    [[TMP6:%.*]] = load i32, ptr [[AA_CASTED]], align 4
+// CHECK-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [3 x ptr], ptr [[CAPTURED_VARS_ADDRS]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP8:%.*]] = inttoptr i32 [[TMP4]] to ptr
+// CHECK-NEXT:    store ptr [[TMP8]], ptr [[TMP7]], align 4
+// CHECK-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [3 x ptr], ptr [[CAPTURED_VARS_ADDRS]], i32 0, i32 1
+// CHECK-NEXT:    [[TMP10:%.*]] = inttoptr i32 [[TMP6]] to ptr
+// CHECK-NEXT:    store ptr [[TMP10]], ptr [[TMP9]], align 4
+// CHECK-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [3 x ptr], ptr [[CAPTURED_VARS_ADDRS]], i32 0, i32 2
+// CHECK-NEXT:    store ptr [[TMP0]], ptr [[TMP11]], align 4
+// CHECK-NEXT:    call void @__kmpc_parallel_51(ptr @[[GLOB1]], i32 [[TMP2]], i32 1, i32 -1, i32 -1, ptr @__omp_outlined__2, ptr null, ptr [[CAPTURED_VARS_ADDRS]], i32 3)
+// CHECK-NEXT:    call void @__kmpc_target_deinit(ptr @[[GLOB1]], i8 2)
+// CHECK-NEXT:    ret void
+// CHECK:       worker.exit:
+// CHECK-NEXT:    ret void
+// CHECK-LABEL: define {{[^@]+}}@__omp_outlined__2
+// CHECK-SAME: (ptr noalias noundef [[DOTGLOBAL_TID_:%.*]], ptr noalias noundef [[DOTBOUND_TID_:%.*]], i32 noundef [[A:%.*]], i32 noundef [[AA:%.*]], ptr noundef nonnull align 4 dereferenceable(40) [[B:%.*]]) #[[ATTR1]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DOTGLOBAL_TID__ADDR:%.*]] = alloca ptr, align 4
+// CHECK-NEXT:    [[DOTBOUND_TID__ADDR:%.*]] = alloca ptr, align 4
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[AA_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca ptr, align 4
+// CHECK-NEXT:    store ptr [[DOTGLOBAL_TID_]], ptr [[DOTGLOBAL_TID__ADDR]], align 4
+// CHECK-NEXT:    store ptr [[DOTBOUND_TID_]], ptr [[DOTBOUND_TID__ADDR]], align 4
+// CHECK-NEXT:    store i32 [[A]], ptr [[A_ADDR]], align 4
+// CHECK-NEXT:    store i32 [[AA]], ptr [[AA_ADDR]], align 4
+// CHECK-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[B_ADDR]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[A_ADDR]], align 4
+// CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP1]], 1
+// CHECK-NEXT:    store i32 [[ADD]], ptr [[A_ADDR]], align 4
+// CHECK-NEXT:    [[TMP2:%.*]] = load i16, ptr [[AA_ADDR]], align 2
+// CHECK-NEXT:    [[CONV:%.*]] = sext i16 [[TMP2]] to i32
+// CHECK-NEXT:    [[ADD1:%.*]] = add nsw i32 [[CONV]], 1
+// CHECK-NEXT:    [[CONV2:%.*]] = trunc i32 [[ADD1]] to i16
+// CHECK-NEXT:    store i16 [[CONV2]], ptr [[AA_ADDR]], align 2
+// CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [10 x i32], ptr [[TMP0]], i32 0, i32 2
+// CHECK-NEXT:    [[TMP3:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
+// CHECK-NEXT:    [[ADD3:%.*]] = add nsw i32 [[TMP3]], 1
+// CHECK-NEXT:    store i32 [[ADD3]], ptr [[ARRAYIDX]], align 4
+// CHECK-NEXT:    ret void
 // CHECK45-64-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9ftemplateIiET_i_l27
 // CHECK45-64-SAME: () #[[ATTR0:[0-9]+]] {
 // CHECK45-64-NEXT:  entry:
@@ -596,142 +720,6 @@ int bar(int n){
 // CHECK-64-NEXT:    ret void
 //
 //
-// CHECK-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9ftemplateIiET_i_l27
-// CHECK-SAME: () #[[ATTR0:[0-9]+]] {
-// CHECK-NEXT:  entry:
-// CHECK-NEXT:    [[CAPTURED_VARS_ADDRS:%.*]] = alloca [0 x ptr], align 4
-// CHECK-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_target_init(ptr @[[GLOB1:[0-9]+]], i8 2, i1 false)
-// CHECK-NEXT:    [[EXEC_USER_CODE:%.*]] = icmp eq i32 [[TMP0]], -1
-// CHECK-NEXT:    br i1 [[EXEC_USER_CODE]], label [[USER_CODE_ENTRY:%.*]], label [[WORKER_EXIT:%.*]]
-// CHECK:       user_code.entry:
-// CHECK-NEXT:    [[TMP1:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB1]])
-// CHECK-NEXT:    call void @__kmpc_parallel_51(ptr @[[GLOB1]], i32 [[TMP1]], i32 1, i32 -1, i32 -1, ptr @__omp_outlined__, ptr null, ptr [[CAPTURED_VARS_ADDRS]], i32 0)
-// CHECK-NEXT:    call void @__kmpc_target_deinit(ptr @[[GLOB1]], i8 2)
-// CHECK-NEXT:    ret void
-// CHECK:       worker.exit:
-// CHECK-NEXT:    ret void
-//
-//
-// CHECK-LABEL: define {{[^@]+}}@__omp_outlined__
-// CHECK-SAME: (ptr noalias noundef [[DOTGLOBAL_TID_:%.*]], ptr noalias noundef [[DOTBOUND_TID_:%.*]]) #[[ATTR1:[0-9]+]] {
-// CHECK-NEXT:  entry:
-// CHECK-NEXT:    [[DOTGLOBAL_TID__ADDR:%.*]] = alloca ptr, align 4
-// CHECK-NEXT:    [[DOTBOUND_TID__ADDR:%.*]] = alloca ptr, align 4
-// CHECK-NEXT:    store ptr [[DOTGLOBAL_TID_]], ptr [[DOTGLOBAL_TID__ADDR]], align 4
-// CHECK-NEXT:    store ptr [[DOTBOUND_TID_]], ptr [[DOTBOUND_TID__ADDR]], align 4
-// CHECK-NEXT:    ret void
-//
-//
-// CHECK-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9ftemplateIiET_i_l31
-// CHECK-SAME: (i32 noundef [[AA:%.*]]) #[[ATTR0]] {
-// CHECK-NEXT:  entry:
-// CHECK-NEXT:    [[AA_ADDR:%.*]] = alloca i32, align 4
-// CHECK-NEXT:    [[AA_CASTED:%.*]] = alloca i32, align 4
-// CHECK-NEXT:    [[CAPTURED_VARS_ADDRS:%.*]] = alloca [1 x ptr], align 4
-// CHECK-NEXT:    store i32 [[AA]], ptr [[AA_ADDR]], align 4
-// CHECK-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_target_init(ptr @[[GLOB1]], i8 2, i1 false)
-// CHECK-NEXT:    [[EXEC_USER_CODE:%.*]] = icmp eq i32 [[TMP0]], -1
-// CHECK-NEXT:    br i1 [[EXEC_USER_CODE]], label [[USER_CODE_ENTRY:%.*]], label [[WORKER_EXIT:%.*]]
-// CHECK:       user_code.entry:
-// CHECK-NEXT:    [[TMP1:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB1]])
-// CHECK-NEXT:    [[TMP2:%.*]] = load i16, ptr [[AA_ADDR]], align 2
-// CHECK-NEXT:    store i16 [[TMP2]], ptr [[AA_CASTED]], align 2
-// CHECK-NEXT:    [[TMP3:%.*]] = load i32, ptr [[AA_CASTED]], align 4
-// CHECK-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[CAPTURED_VARS_ADDRS]], i32 0, i32 0
-// CHECK-NEXT:    [[TMP5:%.*]] = inttoptr i32 [[TMP3]] to ptr
-// CHECK-NEXT:    store ptr [[TMP5]], ptr [[TMP4]], align 4
-// CHECK-NEXT:    call void @__kmpc_parallel_51(ptr @[[GLOB1]], i32 [[TMP1]], i32 1, i32 -1, i32 -1, ptr @__omp_outlined__1, ptr null, ptr [[CAPTURED_VARS_ADDRS]], i32 1)
-// CHECK-NEXT:    call void @__kmpc_target_deinit(ptr @[[GLOB1]], i8 2)
-// CHECK-NEXT:    ret void
-// CHECK:       worker.exit:
-// CHECK-NEXT:    ret void
-//
-//
-// CHECK-LABEL: define {{[^@]+}}@__omp_outlined__1
-// CHECK-SAME: (ptr noalias noundef [[DOTGLOBAL_TID_:%.*]], ptr noalias noundef [[DOTBOUND_TID_:%.*]], i32 noundef [[AA:%.*]]) #[[ATTR1]] {
-// CHECK-NEXT:  entry:
-// CHECK-NEXT:    [[DOTGLOBAL_TID__ADDR:%.*]] = alloca ptr, align 4
-// CHECK-NEXT:    [[DOTBOUND_TID__ADDR:%.*]] = alloca ptr, align 4
-// CHECK-NEXT:    [[AA_ADDR:%.*]] = alloca i32, align 4
-// CHECK-NEXT:    store ptr [[DOTGLOBAL_TID_]], ptr [[DOTGLOBAL_TID__ADDR]], align 4
-// CHECK-NEXT:    store ptr [[DOTBOUND_TID_]], ptr [[DOTBOUND_TID__ADDR]], align 4
-// CHECK-NEXT:    store i32 [[AA]], ptr [[AA_ADDR]], align 4
-// CHECK-NEXT:    [[TMP0:%.*]] = load i16, ptr [[AA_ADDR]], align 2
-// CHECK-NEXT:    [[CONV:%.*]] = sext i16 [[TMP0]] to i32
-// CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[CONV]], 1
-// CHECK-NEXT:    [[CONV1:%.*]] = trunc i32 [[ADD]] to i16
-// CHECK-NEXT:    store i16 [[CONV1]], ptr [[AA_ADDR]], align 2
-// CHECK-NEXT:    ret void
-//
-//
-// CHECK-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9ftemplateIiET_i_l36
-// CHECK-SAME: (i32 noundef [[A:%.*]], i32 noundef [[AA:%.*]], ptr noundef nonnull align 4 dereferenceable(40) [[B:%.*]]) #[[ATTR0]] {
-// CHECK-NEXT:  entry:
-// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca i32, align 4
-// CHECK-NEXT:    [[AA_ADDR:%.*]] = alloca i32, align 4
-// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca ptr, align 4
-// CHECK-NEXT:    [[A_CASTED:%.*]] = alloca i32, align 4
-// CHECK-NEXT:    [[AA_CASTED:%.*]] = alloca i32, align 4
-// CHECK-NEXT:    [[CAPTURED_VARS_ADDRS:%.*]] = alloca [3 x ptr], align 4
-// CHECK-NEXT:    store i32 [[A]], ptr [[A_ADDR]], align 4
-// CHECK-NEXT:    store i32 [[AA]], ptr [[AA_ADDR]], align 4
-// CHECK-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 4
-// CHECK-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[B_ADDR]], align 4
-// CHECK-NEXT:    [[TMP1:%.*]] = call i32 @__kmpc_target_init(ptr @[[GLOB1]], i8 2, i1 false)
-// CHECK-NEXT:    [[EXEC_USER_CODE:%.*]] = icmp eq i32 [[TMP1]], -1
-// CHECK-NEXT:    br i1 [[EXEC_USER_CODE]], label [[USER_CODE_ENTRY:%.*]], label [[WORKER_EXIT:%.*]]
-// CHECK:       user_code.entry:
-// CHECK-NEXT:    [[TMP2:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB1]])
-// CHECK-NEXT:    [[TMP3:%.*]] = load i32, ptr [[A_ADDR]], align 4
-// CHECK-NEXT:    store i32 [[TMP3]], ptr [[A_CASTED]], align 4
-// CHECK-NEXT:    [[TMP4:%.*]] = load i32, ptr [[A_CASTED]], align 4
-// CHECK-NEXT:    [[TMP5:%.*]] = load i16, ptr [[AA_ADDR]], align 2
-// CHECK-NEXT:    store i16 [[TMP5]], ptr [[AA_CASTED]], align 2
-// CHECK-NEXT:    [[TMP6:%.*]] = load i32, ptr [[AA_CASTED]], align 4
-// CHECK-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [3 x ptr], ptr [[CAPTURED_VARS_ADDRS]], i32 0, i32 0
-// CHECK-NEXT:    [[TMP8:%.*]] = inttoptr i32 [[TMP4]] to ptr
-// CHECK-NEXT:    store ptr [[TMP8]], ptr [[TMP7]], align 4
-// CHECK-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [3 x ptr], ptr [[CAPTURED_VARS_ADDRS]], i32 0, i32 1
-// CHECK-NEXT:    [[TMP10:%.*]] = inttoptr i32 [[TMP6]] to ptr
-// CHECK-NEXT:    store ptr [[TMP10]], ptr [[TMP9]], align 4
-// CHECK-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [3 x ptr], ptr [[CAPTURED_VARS_ADDRS]], i32 0, i32 2
-// CHECK-NEXT:    store ptr [[TMP0]], ptr [[TMP11]], align 4
-// CHECK-NEXT:    call void @__kmpc_parallel_51(ptr @[[GLOB1]], i32 [[TMP2]], i32 1, i32 -1, i32 -1, ptr @__omp_outlined__2, ptr null, ptr [[CAPTURED_VARS_ADDRS]], i32 3)
-// CHECK-NEXT:    call void @__kmpc_target_deinit(ptr @[[GLOB1]], i8 2)
-// CHECK-NEXT:    ret void
-// CHECK:       worker.exit:
-// CHECK-NEXT:    ret void
-//
-//
-// CHECK-LABEL: define {{[^@]+}}@__omp_outlined__2
-// CHECK-SAME: (ptr noalias noundef [[DOTGLOBAL_TID_:%.*]], ptr noalias noundef [[DOTBOUND_TID_:%.*]], i32 noundef [[A:%.*]], i32 noundef [[AA:%.*]], ptr noundef nonnull align 4 dereferenceable(40) [[B:%.*]]) #[[ATTR1]] {
-// CHECK-NEXT:  entry:
-// CHECK-NEXT:    [[DOTGLOBAL_TID__ADDR:%.*]] = alloca ptr, align 4
-// CHECK-NEXT:    [[DOTBOUND_TID__ADDR:%.*]] = alloca ptr, align 4
-// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca i32, align 4
-// CHECK-NEXT:    [[AA_ADDR:%.*]] = alloca i32, align 4
-// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca ptr, align 4
-// CHECK-NEXT:    store ptr [[DOTGLOBAL_TID_]], ptr [[DOTGLOBAL_TID__ADDR]], align 4
-// CHECK-NEXT:    store ptr [[DOTBOUND_TID_]], ptr [[DOTBOUND_TID__ADDR]], align 4
-// CHECK-NEXT:    store i32 [[A]], ptr [[A_ADDR]], align 4
-// CHECK-NEXT:    store i32 [[AA]], ptr [[AA_ADDR]], align 4
-// CHECK-NEXT:    store ptr [[B]], ptr [[B_ADDR]], align 4
-// CHECK-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[B_ADDR]], align 4
-// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[A_ADDR]], align 4
-// CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP1]], 1
-// CHECK-NEXT:    store i32 [[ADD]], ptr [[A_ADDR]], align 4
-// CHECK-NEXT:    [[TMP2:%.*]] = load i16, ptr [[AA_ADDR]], align 2
-// CHECK-NEXT:    [[CONV:%.*]] = sext i16 [[TMP2]] to i32
-// CHECK-NEXT:    [[ADD1:%.*]] = add nsw i32 [[CONV]], 1
-// CHECK-NEXT:    [[CONV2:%.*]] = trunc i32 [[ADD1]] to i16
-// CHECK-NEXT:    store i16 [[CONV2]], ptr [[AA_ADDR]], align 2
-// CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [10 x i32], ptr [[TMP0]], i32 0, i32 2
-// CHECK-NEXT:    [[TMP3:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
-// CHECK-NEXT:    [[ADD3:%.*]] = add nsw i32 [[TMP3]], 1
-// CHECK-NEXT:    store i32 [[ADD3]], ptr [[ARRAYIDX]], align 4
-// CHECK-NEXT:    ret void
-//
-//
 // CHECK-32-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9ftemplateIiET_i_l27
 // CHECK-32-SAME: () #[[ATTR0:[0-9]+]] {
 // CHECK-32-NEXT:  entry:
diff --git a/clang/test/OpenMP/nvptx_target_parallel_reduction_codegen_tbaa_PR46146.cpp b/clang/test/OpenMP/nvptx_target_parallel_reduction_codegen_tbaa_PR46146.cpp
index 17140487a23cd..5b7e31ba5d3b0 100644
--- a/clang/test/OpenMP/nvptx_target_parallel_reduction_codegen_tbaa_PR46146.cpp
+++ b/clang/test/OpenMP/nvptx_target_parallel_reduction_codegen_tbaa_PR46146.cpp
@@ -434,7 +434,7 @@ void test() {
 // CHECK1-NEXT:    [[TMP34:%.*]] = load ptr, ptr [[TMP33]], align 8
 // CHECK1-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP4]], i64 0, i64 0
 // CHECK1-NEXT:    [[TMP36:%.*]] = load ptr, ptr [[TMP35]], align 8
-// CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[TMP36]], ptr align 4 [[TMP34]], i64 8, i1 false), !tbaa.struct !21
+// CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[TMP36]], ptr align 4 [[TMP34]], i64 8, i1 false), !tbaa.struct [[TBAA_STRUCT21:![0-9]+]]
 // CHECK1-NEXT:    br label [[IFCONT6:%.*]]
 // CHECK1:       else5:
 // CHECK1-NEXT:    br label [[IFCONT6]]
@@ -939,7 +939,7 @@ void test() {
 // CHECK1-NEXT:    [[TMP41:%.*]] = load ptr, ptr [[TMP40]], align 8
 // CHECK1-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP4]], i64 0, i64 0
 // CHECK1-NEXT:    [[TMP43:%.*]] = load ptr, ptr [[TMP42]], align 8
-// CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[TMP43]], ptr align 8 [[TMP41]], i64 16, i1 false), !tbaa.struct !27
+// CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[TMP43]], ptr align 8 [[TMP41]], i64 16, i1 false), !tbaa.struct [[TBAA_STRUCT27:![0-9]+]]
 // CHECK1-NEXT:    br label [[IFCONT6:%.*]]
 // CHECK1:       else5:
 // CHECK1-NEXT:    br label [[IFCONT6]]
diff --git a/clang/test/OpenMP/reduction_compound_op.cpp b/clang/test/OpenMP/reduction_compound_op.cpp
index 44dfb6081ecc6..e21944dfd426e 100644
--- a/clang/test/OpenMP/reduction_compound_op.cpp
+++ b/clang/test/OpenMP/reduction_compound_op.cpp
@@ -18,7 +18,6 @@
 //RUN: %clang_cc1 -no-enable-noundef-analysis -triple x86_64-unknown-linux-gnu -fopenmp-simd -DNORM -DCOMP \
 //RUN:  -emit-llvm -o - %s | FileCheck %s --check-prefix SIMD-ONLY
 
-// SIMD-ONLY-NOT: {{__kmpc|__tgt}}
 
 struct Point {
   int x = 0;
@@ -206,10 +205,10 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB1]], i32 [[TMP21]])
 // NORM-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOMP_REDUCTION_RED_LIST]], i64 0, i64 0
 // NORM-NEXT:    store ptr [[RED3]], ptr [[TMP22]], align 8
-// NORM-NEXT:    [[TMP24:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// NORM-NEXT:    [[TMP25:%.*]] = load i32, ptr [[TMP24]], align 4
-// NORM-NEXT:    [[TMP27:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2:[0-9]+]], i32 [[TMP25]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func, ptr @.gomp_critical_user_.reduction.var)
-// NORM-NEXT:    switch i32 [[TMP27]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
+// NORM-NEXT:    [[TMP23:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// NORM-NEXT:    [[TMP24:%.*]] = load i32, ptr [[TMP23]], align 4
+// NORM-NEXT:    [[TMP25:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2:[0-9]+]], i32 [[TMP24]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func, ptr @.gomp_critical_user_.reduction.var)
+// NORM-NEXT:    switch i32 [[TMP25]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
 // NORM-NEXT:    i32 1, label [[DOTOMP_REDUCTION_CASE1:%.*]]
 // NORM-NEXT:    i32 2, label [[DOTOMP_REDUCTION_CASE2:%.*]]
 // NORM-NEXT:    ]
@@ -217,16 +216,16 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointplERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // NORM-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
 // NORM-NEXT:    [[CALL9:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
-// NORM-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], ptr @.gomp_critical_user_.reduction.var)
+// NORM-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], ptr @.gomp_critical_user_.reduction.var)
 // NORM-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // NORM:       .omp.reduction.case2:
-// NORM-NEXT:    [[TMP29:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// NORM-NEXT:    [[TMP30:%.*]] = load i32, ptr [[TMP29]], align 4
-// NORM-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// NORM-NEXT:    [[TMP26:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// NORM-NEXT:    [[TMP27:%.*]] = load i32, ptr [[TMP26]], align 4
+// NORM-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // NORM-NEXT:    [[CALL11:%.*]] = call i64 @_ZNK5PointplERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // NORM-NEXT:    store i64 [[CALL11]], ptr [[REF_TMP10]], align 4
 // NORM-NEXT:    [[CALL12:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP10]])
-// NORM-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// NORM-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // NORM-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // NORM:       .omp.reduction.default:
 // NORM-NEXT:    br label [[OMP_PRECOND_END]]
@@ -243,14 +242,14 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    store ptr [[TMP0]], ptr [[DOTADDR]], align 8
 // NORM-NEXT:    store ptr [[TMP1]], ptr [[DOTADDR1]], align 8
 // NORM-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[DOTADDR]], align 8
-// NORM-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
-// NORM-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP4]], i64 0, i64 0
+// NORM-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
+// NORM-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP3]], i64 0, i64 0
+// NORM-NEXT:    [[TMP5:%.*]] = load ptr, ptr [[TMP4]], align 8
+// NORM-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
 // NORM-NEXT:    [[TMP7:%.*]] = load ptr, ptr [[TMP6]], align 8
-// NORM-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
-// NORM-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[TMP9]], align 8
-// NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointplERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[TMP7]])
+// NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointplERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[TMP5]])
 // NORM-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
-// NORM-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
+// NORM-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
 // NORM-NEXT:    ret void
 //
 //
@@ -351,10 +350,10 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB1]], i32 [[TMP21]])
 // NORM-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOMP_REDUCTION_RED_LIST]], i64 0, i64 0
 // NORM-NEXT:    store ptr [[RED3]], ptr [[TMP22]], align 8
-// NORM-NEXT:    [[TMP24:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// NORM-NEXT:    [[TMP25:%.*]] = load i32, ptr [[TMP24]], align 4
-// NORM-NEXT:    [[TMP27:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.2, ptr @.gomp_critical_user_.reduction.var)
-// NORM-NEXT:    switch i32 [[TMP27]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
+// NORM-NEXT:    [[TMP23:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// NORM-NEXT:    [[TMP24:%.*]] = load i32, ptr [[TMP23]], align 4
+// NORM-NEXT:    [[TMP25:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.2, ptr @.gomp_critical_user_.reduction.var)
+// NORM-NEXT:    switch i32 [[TMP25]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
 // NORM-NEXT:    i32 1, label [[DOTOMP_REDUCTION_CASE1:%.*]]
 // NORM-NEXT:    i32 2, label [[DOTOMP_REDUCTION_CASE2:%.*]]
 // NORM-NEXT:    ]
@@ -362,16 +361,16 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointplERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // NORM-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
 // NORM-NEXT:    [[CALL9:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
-// NORM-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], ptr @.gomp_critical_user_.reduction.var)
+// NORM-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], ptr @.gomp_critical_user_.reduction.var)
 // NORM-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // NORM:       .omp.reduction.case2:
-// NORM-NEXT:    [[TMP29:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// NORM-NEXT:    [[TMP30:%.*]] = load i32, ptr [[TMP29]], align 4
-// NORM-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// NORM-NEXT:    [[TMP26:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// NORM-NEXT:    [[TMP27:%.*]] = load i32, ptr [[TMP26]], align 4
+// NORM-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // NORM-NEXT:    [[CALL11:%.*]] = call i64 @_ZNK5PointplERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // NORM-NEXT:    store i64 [[CALL11]], ptr [[REF_TMP10]], align 4
 // NORM-NEXT:    [[CALL12:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP10]])
-// NORM-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// NORM-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // NORM-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // NORM:       .omp.reduction.default:
 // NORM-NEXT:    br label [[OMP_PRECOND_END]]
@@ -388,14 +387,14 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    store ptr [[TMP0]], ptr [[DOTADDR]], align 8
 // NORM-NEXT:    store ptr [[TMP1]], ptr [[DOTADDR1]], align 8
 // NORM-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[DOTADDR]], align 8
-// NORM-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
-// NORM-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP4]], i64 0, i64 0
+// NORM-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
+// NORM-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP3]], i64 0, i64 0
+// NORM-NEXT:    [[TMP5:%.*]] = load ptr, ptr [[TMP4]], align 8
+// NORM-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
 // NORM-NEXT:    [[TMP7:%.*]] = load ptr, ptr [[TMP6]], align 8
-// NORM-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
-// NORM-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[TMP9]], align 8
-// NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointplERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[TMP7]])
+// NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointplERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[TMP5]])
 // NORM-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
-// NORM-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
+// NORM-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
 // NORM-NEXT:    ret void
 //
 //
@@ -496,10 +495,10 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB1]], i32 [[TMP21]])
 // NORM-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOMP_REDUCTION_RED_LIST]], i64 0, i64 0
 // NORM-NEXT:    store ptr [[RED3]], ptr [[TMP22]], align 8
-// NORM-NEXT:    [[TMP24:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// NORM-NEXT:    [[TMP25:%.*]] = load i32, ptr [[TMP24]], align 4
-// NORM-NEXT:    [[TMP27:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.4, ptr @.gomp_critical_user_.reduction.var)
-// NORM-NEXT:    switch i32 [[TMP27]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
+// NORM-NEXT:    [[TMP23:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// NORM-NEXT:    [[TMP24:%.*]] = load i32, ptr [[TMP23]], align 4
+// NORM-NEXT:    [[TMP25:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.4, ptr @.gomp_critical_user_.reduction.var)
+// NORM-NEXT:    switch i32 [[TMP25]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
 // NORM-NEXT:    i32 1, label [[DOTOMP_REDUCTION_CASE1:%.*]]
 // NORM-NEXT:    i32 2, label [[DOTOMP_REDUCTION_CASE2:%.*]]
 // NORM-NEXT:    ]
@@ -507,16 +506,16 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointmlERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // NORM-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
 // NORM-NEXT:    [[CALL9:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
-// NORM-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], ptr @.gomp_critical_user_.reduction.var)
+// NORM-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], ptr @.gomp_critical_user_.reduction.var)
 // NORM-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // NORM:       .omp.reduction.case2:
-// NORM-NEXT:    [[TMP29:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// NORM-NEXT:    [[TMP30:%.*]] = load i32, ptr [[TMP29]], align 4
-// NORM-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// NORM-NEXT:    [[TMP26:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// NORM-NEXT:    [[TMP27:%.*]] = load i32, ptr [[TMP26]], align 4
+// NORM-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // NORM-NEXT:    [[CALL11:%.*]] = call i64 @_ZNK5PointmlERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // NORM-NEXT:    store i64 [[CALL11]], ptr [[REF_TMP10]], align 4
 // NORM-NEXT:    [[CALL12:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP10]])
-// NORM-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// NORM-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // NORM-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // NORM:       .omp.reduction.default:
 // NORM-NEXT:    br label [[OMP_PRECOND_END]]
@@ -533,14 +532,14 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    store ptr [[TMP0]], ptr [[DOTADDR]], align 8
 // NORM-NEXT:    store ptr [[TMP1]], ptr [[DOTADDR1]], align 8
 // NORM-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[DOTADDR]], align 8
-// NORM-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
-// NORM-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP4]], i64 0, i64 0
+// NORM-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
+// NORM-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP3]], i64 0, i64 0
+// NORM-NEXT:    [[TMP5:%.*]] = load ptr, ptr [[TMP4]], align 8
+// NORM-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
 // NORM-NEXT:    [[TMP7:%.*]] = load ptr, ptr [[TMP6]], align 8
-// NORM-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
-// NORM-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[TMP9]], align 8
-// NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointmlERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[TMP7]])
+// NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointmlERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[TMP5]])
 // NORM-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
-// NORM-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
+// NORM-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
 // NORM-NEXT:    ret void
 //
 //
@@ -641,10 +640,10 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB1]], i32 [[TMP21]])
 // NORM-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOMP_REDUCTION_RED_LIST]], i64 0, i64 0
 // NORM-NEXT:    store ptr [[RED3]], ptr [[TMP22]], align 8
-// NORM-NEXT:    [[TMP24:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// NORM-NEXT:    [[TMP25:%.*]] = load i32, ptr [[TMP24]], align 4
-// NORM-NEXT:    [[TMP27:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.6, ptr @.gomp_critical_user_.reduction.var)
-// NORM-NEXT:    switch i32 [[TMP27]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
+// NORM-NEXT:    [[TMP23:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// NORM-NEXT:    [[TMP24:%.*]] = load i32, ptr [[TMP23]], align 4
+// NORM-NEXT:    [[TMP25:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.6, ptr @.gomp_critical_user_.reduction.var)
+// NORM-NEXT:    switch i32 [[TMP25]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
 // NORM-NEXT:    i32 1, label [[DOTOMP_REDUCTION_CASE1:%.*]]
 // NORM-NEXT:    i32 2, label [[DOTOMP_REDUCTION_CASE2:%.*]]
 // NORM-NEXT:    ]
@@ -652,16 +651,16 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointanERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // NORM-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
 // NORM-NEXT:    [[CALL9:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
-// NORM-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], ptr @.gomp_critical_user_.reduction.var)
+// NORM-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], ptr @.gomp_critical_user_.reduction.var)
 // NORM-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // NORM:       .omp.reduction.case2:
-// NORM-NEXT:    [[TMP29:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// NORM-NEXT:    [[TMP30:%.*]] = load i32, ptr [[TMP29]], align 4
-// NORM-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// NORM-NEXT:    [[TMP26:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// NORM-NEXT:    [[TMP27:%.*]] = load i32, ptr [[TMP26]], align 4
+// NORM-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // NORM-NEXT:    [[CALL11:%.*]] = call i64 @_ZNK5PointanERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // NORM-NEXT:    store i64 [[CALL11]], ptr [[REF_TMP10]], align 4
 // NORM-NEXT:    [[CALL12:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP10]])
-// NORM-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// NORM-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // NORM-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // NORM:       .omp.reduction.default:
 // NORM-NEXT:    br label [[OMP_PRECOND_END]]
@@ -678,14 +677,14 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    store ptr [[TMP0]], ptr [[DOTADDR]], align 8
 // NORM-NEXT:    store ptr [[TMP1]], ptr [[DOTADDR1]], align 8
 // NORM-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[DOTADDR]], align 8
-// NORM-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
-// NORM-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP4]], i64 0, i64 0
+// NORM-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
+// NORM-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP3]], i64 0, i64 0
+// NORM-NEXT:    [[TMP5:%.*]] = load ptr, ptr [[TMP4]], align 8
+// NORM-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
 // NORM-NEXT:    [[TMP7:%.*]] = load ptr, ptr [[TMP6]], align 8
-// NORM-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
-// NORM-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[TMP9]], align 8
-// NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointanERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[TMP7]])
+// NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointanERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[TMP5]])
 // NORM-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
-// NORM-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
+// NORM-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
 // NORM-NEXT:    ret void
 //
 //
@@ -786,10 +785,10 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB1]], i32 [[TMP21]])
 // NORM-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOMP_REDUCTION_RED_LIST]], i64 0, i64 0
 // NORM-NEXT:    store ptr [[RED3]], ptr [[TMP22]], align 8
-// NORM-NEXT:    [[TMP24:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// NORM-NEXT:    [[TMP25:%.*]] = load i32, ptr [[TMP24]], align 4
-// NORM-NEXT:    [[TMP27:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.8, ptr @.gomp_critical_user_.reduction.var)
-// NORM-NEXT:    switch i32 [[TMP27]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
+// NORM-NEXT:    [[TMP23:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// NORM-NEXT:    [[TMP24:%.*]] = load i32, ptr [[TMP23]], align 4
+// NORM-NEXT:    [[TMP25:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.8, ptr @.gomp_critical_user_.reduction.var)
+// NORM-NEXT:    switch i32 [[TMP25]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
 // NORM-NEXT:    i32 1, label [[DOTOMP_REDUCTION_CASE1:%.*]]
 // NORM-NEXT:    i32 2, label [[DOTOMP_REDUCTION_CASE2:%.*]]
 // NORM-NEXT:    ]
@@ -797,16 +796,16 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointorERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // NORM-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
 // NORM-NEXT:    [[CALL9:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
-// NORM-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], ptr @.gomp_critical_user_.reduction.var)
+// NORM-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], ptr @.gomp_critical_user_.reduction.var)
 // NORM-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // NORM:       .omp.reduction.case2:
-// NORM-NEXT:    [[TMP29:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// NORM-NEXT:    [[TMP30:%.*]] = load i32, ptr [[TMP29]], align 4
-// NORM-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// NORM-NEXT:    [[TMP26:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// NORM-NEXT:    [[TMP27:%.*]] = load i32, ptr [[TMP26]], align 4
+// NORM-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // NORM-NEXT:    [[CALL11:%.*]] = call i64 @_ZNK5PointorERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // NORM-NEXT:    store i64 [[CALL11]], ptr [[REF_TMP10]], align 4
 // NORM-NEXT:    [[CALL12:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP10]])
-// NORM-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// NORM-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // NORM-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // NORM:       .omp.reduction.default:
 // NORM-NEXT:    br label [[OMP_PRECOND_END]]
@@ -823,14 +822,14 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    store ptr [[TMP0]], ptr [[DOTADDR]], align 8
 // NORM-NEXT:    store ptr [[TMP1]], ptr [[DOTADDR1]], align 8
 // NORM-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[DOTADDR]], align 8
-// NORM-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
-// NORM-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP4]], i64 0, i64 0
+// NORM-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
+// NORM-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP3]], i64 0, i64 0
+// NORM-NEXT:    [[TMP5:%.*]] = load ptr, ptr [[TMP4]], align 8
+// NORM-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
 // NORM-NEXT:    [[TMP7:%.*]] = load ptr, ptr [[TMP6]], align 8
-// NORM-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
-// NORM-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[TMP9]], align 8
-// NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointorERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[TMP7]])
+// NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointorERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[TMP5]])
 // NORM-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
-// NORM-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
+// NORM-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
 // NORM-NEXT:    ret void
 //
 //
@@ -931,10 +930,10 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB1]], i32 [[TMP21]])
 // NORM-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOMP_REDUCTION_RED_LIST]], i64 0, i64 0
 // NORM-NEXT:    store ptr [[RED3]], ptr [[TMP22]], align 8
-// NORM-NEXT:    [[TMP24:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// NORM-NEXT:    [[TMP25:%.*]] = load i32, ptr [[TMP24]], align 4
-// NORM-NEXT:    [[TMP27:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.10, ptr @.gomp_critical_user_.reduction.var)
-// NORM-NEXT:    switch i32 [[TMP27]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
+// NORM-NEXT:    [[TMP23:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// NORM-NEXT:    [[TMP24:%.*]] = load i32, ptr [[TMP23]], align 4
+// NORM-NEXT:    [[TMP25:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.10, ptr @.gomp_critical_user_.reduction.var)
+// NORM-NEXT:    switch i32 [[TMP25]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
 // NORM-NEXT:    i32 1, label [[DOTOMP_REDUCTION_CASE1:%.*]]
 // NORM-NEXT:    i32 2, label [[DOTOMP_REDUCTION_CASE2:%.*]]
 // NORM-NEXT:    ]
@@ -942,16 +941,16 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointeoERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // NORM-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
 // NORM-NEXT:    [[CALL9:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
-// NORM-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], ptr @.gomp_critical_user_.reduction.var)
+// NORM-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], ptr @.gomp_critical_user_.reduction.var)
 // NORM-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // NORM:       .omp.reduction.case2:
-// NORM-NEXT:    [[TMP29:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// NORM-NEXT:    [[TMP30:%.*]] = load i32, ptr [[TMP29]], align 4
-// NORM-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// NORM-NEXT:    [[TMP26:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// NORM-NEXT:    [[TMP27:%.*]] = load i32, ptr [[TMP26]], align 4
+// NORM-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // NORM-NEXT:    [[CALL11:%.*]] = call i64 @_ZNK5PointeoERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // NORM-NEXT:    store i64 [[CALL11]], ptr [[REF_TMP10]], align 4
 // NORM-NEXT:    [[CALL12:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP10]])
-// NORM-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// NORM-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // NORM-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // NORM:       .omp.reduction.default:
 // NORM-NEXT:    br label [[OMP_PRECOND_END]]
@@ -968,14 +967,14 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    store ptr [[TMP0]], ptr [[DOTADDR]], align 8
 // NORM-NEXT:    store ptr [[TMP1]], ptr [[DOTADDR1]], align 8
 // NORM-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[DOTADDR]], align 8
-// NORM-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
-// NORM-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP4]], i64 0, i64 0
+// NORM-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
+// NORM-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP3]], i64 0, i64 0
+// NORM-NEXT:    [[TMP5:%.*]] = load ptr, ptr [[TMP4]], align 8
+// NORM-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
 // NORM-NEXT:    [[TMP7:%.*]] = load ptr, ptr [[TMP6]], align 8
-// NORM-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
-// NORM-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[TMP9]], align 8
-// NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointeoERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[TMP7]])
+// NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointeoERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[TMP5]])
 // NORM-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
-// NORM-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
+// NORM-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
 // NORM-NEXT:    ret void
 //
 //
@@ -1076,10 +1075,10 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB1]], i32 [[TMP21]])
 // NORM-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOMP_REDUCTION_RED_LIST]], i64 0, i64 0
 // NORM-NEXT:    store ptr [[RED3]], ptr [[TMP22]], align 8
-// NORM-NEXT:    [[TMP24:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// NORM-NEXT:    [[TMP25:%.*]] = load i32, ptr [[TMP24]], align 4
-// NORM-NEXT:    [[TMP27:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.12, ptr @.gomp_critical_user_.reduction.var)
-// NORM-NEXT:    switch i32 [[TMP27]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
+// NORM-NEXT:    [[TMP23:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// NORM-NEXT:    [[TMP24:%.*]] = load i32, ptr [[TMP23]], align 4
+// NORM-NEXT:    [[TMP25:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.12, ptr @.gomp_critical_user_.reduction.var)
+// NORM-NEXT:    switch i32 [[TMP25]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
 // NORM-NEXT:    i32 1, label [[DOTOMP_REDUCTION_CASE1:%.*]]
 // NORM-NEXT:    i32 2, label [[DOTOMP_REDUCTION_CASE2:%.*]]
 // NORM-NEXT:    ]
@@ -1087,16 +1086,16 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointaaERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // NORM-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
 // NORM-NEXT:    [[CALL9:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
-// NORM-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], ptr @.gomp_critical_user_.reduction.var)
+// NORM-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], ptr @.gomp_critical_user_.reduction.var)
 // NORM-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // NORM:       .omp.reduction.case2:
-// NORM-NEXT:    [[TMP29:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// NORM-NEXT:    [[TMP30:%.*]] = load i32, ptr [[TMP29]], align 4
-// NORM-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// NORM-NEXT:    [[TMP26:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// NORM-NEXT:    [[TMP27:%.*]] = load i32, ptr [[TMP26]], align 4
+// NORM-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // NORM-NEXT:    [[CALL11:%.*]] = call i64 @_ZNK5PointaaERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // NORM-NEXT:    store i64 [[CALL11]], ptr [[REF_TMP10]], align 4
 // NORM-NEXT:    [[CALL12:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP10]])
-// NORM-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// NORM-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // NORM-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // NORM:       .omp.reduction.default:
 // NORM-NEXT:    br label [[OMP_PRECOND_END]]
@@ -1113,14 +1112,14 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    store ptr [[TMP0]], ptr [[DOTADDR]], align 8
 // NORM-NEXT:    store ptr [[TMP1]], ptr [[DOTADDR1]], align 8
 // NORM-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[DOTADDR]], align 8
-// NORM-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
-// NORM-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP4]], i64 0, i64 0
+// NORM-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
+// NORM-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP3]], i64 0, i64 0
+// NORM-NEXT:    [[TMP5:%.*]] = load ptr, ptr [[TMP4]], align 8
+// NORM-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
 // NORM-NEXT:    [[TMP7:%.*]] = load ptr, ptr [[TMP6]], align 8
-// NORM-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
-// NORM-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[TMP9]], align 8
-// NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointaaERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[TMP7]])
+// NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointaaERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[TMP5]])
 // NORM-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
-// NORM-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
+// NORM-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
 // NORM-NEXT:    ret void
 //
 //
@@ -1221,10 +1220,10 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB1]], i32 [[TMP21]])
 // NORM-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOMP_REDUCTION_RED_LIST]], i64 0, i64 0
 // NORM-NEXT:    store ptr [[RED3]], ptr [[TMP22]], align 8
-// NORM-NEXT:    [[TMP24:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// NORM-NEXT:    [[TMP25:%.*]] = load i32, ptr [[TMP24]], align 4
-// NORM-NEXT:    [[TMP27:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.14, ptr @.gomp_critical_user_.reduction.var)
-// NORM-NEXT:    switch i32 [[TMP27]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
+// NORM-NEXT:    [[TMP23:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// NORM-NEXT:    [[TMP24:%.*]] = load i32, ptr [[TMP23]], align 4
+// NORM-NEXT:    [[TMP25:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.14, ptr @.gomp_critical_user_.reduction.var)
+// NORM-NEXT:    switch i32 [[TMP25]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
 // NORM-NEXT:    i32 1, label [[DOTOMP_REDUCTION_CASE1:%.*]]
 // NORM-NEXT:    i32 2, label [[DOTOMP_REDUCTION_CASE2:%.*]]
 // NORM-NEXT:    ]
@@ -1232,16 +1231,16 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointooERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // NORM-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
 // NORM-NEXT:    [[CALL9:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
-// NORM-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], ptr @.gomp_critical_user_.reduction.var)
+// NORM-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], ptr @.gomp_critical_user_.reduction.var)
 // NORM-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // NORM:       .omp.reduction.case2:
-// NORM-NEXT:    [[TMP29:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// NORM-NEXT:    [[TMP30:%.*]] = load i32, ptr [[TMP29]], align 4
-// NORM-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// NORM-NEXT:    [[TMP26:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// NORM-NEXT:    [[TMP27:%.*]] = load i32, ptr [[TMP26]], align 4
+// NORM-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // NORM-NEXT:    [[CALL11:%.*]] = call i64 @_ZNK5PointooERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // NORM-NEXT:    store i64 [[CALL11]], ptr [[REF_TMP10]], align 4
 // NORM-NEXT:    [[CALL12:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP10]])
-// NORM-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// NORM-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // NORM-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // NORM:       .omp.reduction.default:
 // NORM-NEXT:    br label [[OMP_PRECOND_END]]
@@ -1258,14 +1257,14 @@ void foo(int N, Point const *Points) {
 // NORM-NEXT:    store ptr [[TMP0]], ptr [[DOTADDR]], align 8
 // NORM-NEXT:    store ptr [[TMP1]], ptr [[DOTADDR1]], align 8
 // NORM-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[DOTADDR]], align 8
-// NORM-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
-// NORM-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP4]], i64 0, i64 0
+// NORM-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
+// NORM-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP3]], i64 0, i64 0
+// NORM-NEXT:    [[TMP5:%.*]] = load ptr, ptr [[TMP4]], align 8
+// NORM-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
 // NORM-NEXT:    [[TMP7:%.*]] = load ptr, ptr [[TMP6]], align 8
-// NORM-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
-// NORM-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[TMP9]], align 8
-// NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointooERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[TMP7]])
+// NORM-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointooERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[TMP5]])
 // NORM-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
-// NORM-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
+// NORM-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
 // NORM-NEXT:    ret void
 //
 //
@@ -1407,23 +1406,23 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB1]], i32 [[TMP21]])
 // COMP-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOMP_REDUCTION_RED_LIST]], i64 0, i64 0
 // COMP-NEXT:    store ptr [[RED3]], ptr [[TMP22]], align 8
-// COMP-NEXT:    [[TMP24:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// COMP-NEXT:    [[TMP25:%.*]] = load i32, ptr [[TMP24]], align 4
-// COMP-NEXT:    [[TMP27:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2:[0-9]+]], i32 [[TMP25]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func, ptr @.gomp_critical_user_.reduction.var)
-// COMP-NEXT:    switch i32 [[TMP27]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
+// COMP-NEXT:    [[TMP23:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// COMP-NEXT:    [[TMP24:%.*]] = load i32, ptr [[TMP23]], align 4
+// COMP-NEXT:    [[TMP25:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2:[0-9]+]], i32 [[TMP24]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func, ptr @.gomp_critical_user_.reduction.var)
+// COMP-NEXT:    switch i32 [[TMP25]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
 // COMP-NEXT:    i32 1, label [[DOTOMP_REDUCTION_CASE1:%.*]]
 // COMP-NEXT:    i32 2, label [[DOTOMP_REDUCTION_CASE2:%.*]]
 // COMP-NEXT:    ]
 // COMP:       .omp.reduction.case1:
 // COMP-NEXT:    [[CALL:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointpLERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
-// COMP-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], ptr @.gomp_critical_user_.reduction.var)
+// COMP-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], ptr @.gomp_critical_user_.reduction.var)
 // COMP-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // COMP:       .omp.reduction.case2:
-// COMP-NEXT:    [[TMP28:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// COMP-NEXT:    [[TMP29:%.*]] = load i32, ptr [[TMP28]], align 4
-// COMP-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP29]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// COMP-NEXT:    [[TMP26:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// COMP-NEXT:    [[TMP27:%.*]] = load i32, ptr [[TMP26]], align 4
+// COMP-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // COMP-NEXT:    [[CALL9:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointpLERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
-// COMP-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP29]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// COMP-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // COMP-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // COMP:       .omp.reduction.default:
 // COMP-NEXT:    br label [[OMP_PRECOND_END]]
@@ -1439,12 +1438,12 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    store ptr [[TMP0]], ptr [[DOTADDR]], align 8
 // COMP-NEXT:    store ptr [[TMP1]], ptr [[DOTADDR1]], align 8
 // COMP-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[DOTADDR]], align 8
-// COMP-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
-// COMP-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP4]], i64 0, i64 0
+// COMP-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
+// COMP-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP3]], i64 0, i64 0
+// COMP-NEXT:    [[TMP5:%.*]] = load ptr, ptr [[TMP4]], align 8
+// COMP-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
 // COMP-NEXT:    [[TMP7:%.*]] = load ptr, ptr [[TMP6]], align 8
-// COMP-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
-// COMP-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[TMP9]], align 8
-// COMP-NEXT:    [[CALL:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointpLERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[TMP7]])
+// COMP-NEXT:    [[CALL:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointpLERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[TMP5]])
 // COMP-NEXT:    ret void
 //
 //
@@ -1543,23 +1542,23 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB1]], i32 [[TMP21]])
 // COMP-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOMP_REDUCTION_RED_LIST]], i64 0, i64 0
 // COMP-NEXT:    store ptr [[RED3]], ptr [[TMP22]], align 8
-// COMP-NEXT:    [[TMP24:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// COMP-NEXT:    [[TMP25:%.*]] = load i32, ptr [[TMP24]], align 4
-// COMP-NEXT:    [[TMP27:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.2, ptr @.gomp_critical_user_.reduction.var)
-// COMP-NEXT:    switch i32 [[TMP27]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
+// COMP-NEXT:    [[TMP23:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// COMP-NEXT:    [[TMP24:%.*]] = load i32, ptr [[TMP23]], align 4
+// COMP-NEXT:    [[TMP25:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.2, ptr @.gomp_critical_user_.reduction.var)
+// COMP-NEXT:    switch i32 [[TMP25]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
 // COMP-NEXT:    i32 1, label [[DOTOMP_REDUCTION_CASE1:%.*]]
 // COMP-NEXT:    i32 2, label [[DOTOMP_REDUCTION_CASE2:%.*]]
 // COMP-NEXT:    ]
 // COMP:       .omp.reduction.case1:
 // COMP-NEXT:    [[CALL:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointpLERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
-// COMP-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], ptr @.gomp_critical_user_.reduction.var)
+// COMP-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], ptr @.gomp_critical_user_.reduction.var)
 // COMP-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // COMP:       .omp.reduction.case2:
-// COMP-NEXT:    [[TMP28:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// COMP-NEXT:    [[TMP29:%.*]] = load i32, ptr [[TMP28]], align 4
-// COMP-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP29]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// COMP-NEXT:    [[TMP26:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// COMP-NEXT:    [[TMP27:%.*]] = load i32, ptr [[TMP26]], align 4
+// COMP-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // COMP-NEXT:    [[CALL9:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointpLERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
-// COMP-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP29]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// COMP-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // COMP-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // COMP:       .omp.reduction.default:
 // COMP-NEXT:    br label [[OMP_PRECOND_END]]
@@ -1575,12 +1574,12 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    store ptr [[TMP0]], ptr [[DOTADDR]], align 8
 // COMP-NEXT:    store ptr [[TMP1]], ptr [[DOTADDR1]], align 8
 // COMP-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[DOTADDR]], align 8
-// COMP-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
-// COMP-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP4]], i64 0, i64 0
+// COMP-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
+// COMP-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP3]], i64 0, i64 0
+// COMP-NEXT:    [[TMP5:%.*]] = load ptr, ptr [[TMP4]], align 8
+// COMP-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
 // COMP-NEXT:    [[TMP7:%.*]] = load ptr, ptr [[TMP6]], align 8
-// COMP-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
-// COMP-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[TMP9]], align 8
-// COMP-NEXT:    [[CALL:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointpLERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[TMP7]])
+// COMP-NEXT:    [[CALL:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointpLERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[TMP5]])
 // COMP-NEXT:    ret void
 //
 //
@@ -1679,23 +1678,23 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB1]], i32 [[TMP21]])
 // COMP-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOMP_REDUCTION_RED_LIST]], i64 0, i64 0
 // COMP-NEXT:    store ptr [[RED3]], ptr [[TMP22]], align 8
-// COMP-NEXT:    [[TMP24:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// COMP-NEXT:    [[TMP25:%.*]] = load i32, ptr [[TMP24]], align 4
-// COMP-NEXT:    [[TMP27:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.4, ptr @.gomp_critical_user_.reduction.var)
-// COMP-NEXT:    switch i32 [[TMP27]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
+// COMP-NEXT:    [[TMP23:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// COMP-NEXT:    [[TMP24:%.*]] = load i32, ptr [[TMP23]], align 4
+// COMP-NEXT:    [[TMP25:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.4, ptr @.gomp_critical_user_.reduction.var)
+// COMP-NEXT:    switch i32 [[TMP25]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
 // COMP-NEXT:    i32 1, label [[DOTOMP_REDUCTION_CASE1:%.*]]
 // COMP-NEXT:    i32 2, label [[DOTOMP_REDUCTION_CASE2:%.*]]
 // COMP-NEXT:    ]
 // COMP:       .omp.reduction.case1:
 // COMP-NEXT:    [[CALL:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointmLERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
-// COMP-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], ptr @.gomp_critical_user_.reduction.var)
+// COMP-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], ptr @.gomp_critical_user_.reduction.var)
 // COMP-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // COMP:       .omp.reduction.case2:
-// COMP-NEXT:    [[TMP28:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// COMP-NEXT:    [[TMP29:%.*]] = load i32, ptr [[TMP28]], align 4
-// COMP-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP29]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// COMP-NEXT:    [[TMP26:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// COMP-NEXT:    [[TMP27:%.*]] = load i32, ptr [[TMP26]], align 4
+// COMP-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // COMP-NEXT:    [[CALL9:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointmLERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
-// COMP-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP29]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// COMP-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // COMP-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // COMP:       .omp.reduction.default:
 // COMP-NEXT:    br label [[OMP_PRECOND_END]]
@@ -1711,12 +1710,12 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    store ptr [[TMP0]], ptr [[DOTADDR]], align 8
 // COMP-NEXT:    store ptr [[TMP1]], ptr [[DOTADDR1]], align 8
 // COMP-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[DOTADDR]], align 8
-// COMP-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
-// COMP-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP4]], i64 0, i64 0
+// COMP-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
+// COMP-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP3]], i64 0, i64 0
+// COMP-NEXT:    [[TMP5:%.*]] = load ptr, ptr [[TMP4]], align 8
+// COMP-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
 // COMP-NEXT:    [[TMP7:%.*]] = load ptr, ptr [[TMP6]], align 8
-// COMP-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
-// COMP-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[TMP9]], align 8
-// COMP-NEXT:    [[CALL:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointmLERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[TMP7]])
+// COMP-NEXT:    [[CALL:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointmLERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[TMP5]])
 // COMP-NEXT:    ret void
 //
 //
@@ -1815,23 +1814,23 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB1]], i32 [[TMP21]])
 // COMP-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOMP_REDUCTION_RED_LIST]], i64 0, i64 0
 // COMP-NEXT:    store ptr [[RED3]], ptr [[TMP22]], align 8
-// COMP-NEXT:    [[TMP24:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// COMP-NEXT:    [[TMP25:%.*]] = load i32, ptr [[TMP24]], align 4
-// COMP-NEXT:    [[TMP27:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.6, ptr @.gomp_critical_user_.reduction.var)
-// COMP-NEXT:    switch i32 [[TMP27]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
+// COMP-NEXT:    [[TMP23:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// COMP-NEXT:    [[TMP24:%.*]] = load i32, ptr [[TMP23]], align 4
+// COMP-NEXT:    [[TMP25:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.6, ptr @.gomp_critical_user_.reduction.var)
+// COMP-NEXT:    switch i32 [[TMP25]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
 // COMP-NEXT:    i32 1, label [[DOTOMP_REDUCTION_CASE1:%.*]]
 // COMP-NEXT:    i32 2, label [[DOTOMP_REDUCTION_CASE2:%.*]]
 // COMP-NEXT:    ]
 // COMP:       .omp.reduction.case1:
 // COMP-NEXT:    [[CALL:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaNERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
-// COMP-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], ptr @.gomp_critical_user_.reduction.var)
+// COMP-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], ptr @.gomp_critical_user_.reduction.var)
 // COMP-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // COMP:       .omp.reduction.case2:
-// COMP-NEXT:    [[TMP28:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// COMP-NEXT:    [[TMP29:%.*]] = load i32, ptr [[TMP28]], align 4
-// COMP-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP29]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// COMP-NEXT:    [[TMP26:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// COMP-NEXT:    [[TMP27:%.*]] = load i32, ptr [[TMP26]], align 4
+// COMP-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // COMP-NEXT:    [[CALL9:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaNERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
-// COMP-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP29]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// COMP-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // COMP-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // COMP:       .omp.reduction.default:
 // COMP-NEXT:    br label [[OMP_PRECOND_END]]
@@ -1847,12 +1846,12 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    store ptr [[TMP0]], ptr [[DOTADDR]], align 8
 // COMP-NEXT:    store ptr [[TMP1]], ptr [[DOTADDR1]], align 8
 // COMP-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[DOTADDR]], align 8
-// COMP-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
-// COMP-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP4]], i64 0, i64 0
+// COMP-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
+// COMP-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP3]], i64 0, i64 0
+// COMP-NEXT:    [[TMP5:%.*]] = load ptr, ptr [[TMP4]], align 8
+// COMP-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
 // COMP-NEXT:    [[TMP7:%.*]] = load ptr, ptr [[TMP6]], align 8
-// COMP-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
-// COMP-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[TMP9]], align 8
-// COMP-NEXT:    [[CALL:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaNERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[TMP7]])
+// COMP-NEXT:    [[CALL:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaNERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[TMP5]])
 // COMP-NEXT:    ret void
 //
 //
@@ -1951,23 +1950,23 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB1]], i32 [[TMP21]])
 // COMP-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOMP_REDUCTION_RED_LIST]], i64 0, i64 0
 // COMP-NEXT:    store ptr [[RED3]], ptr [[TMP22]], align 8
-// COMP-NEXT:    [[TMP24:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// COMP-NEXT:    [[TMP25:%.*]] = load i32, ptr [[TMP24]], align 4
-// COMP-NEXT:    [[TMP27:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.8, ptr @.gomp_critical_user_.reduction.var)
-// COMP-NEXT:    switch i32 [[TMP27]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
+// COMP-NEXT:    [[TMP23:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// COMP-NEXT:    [[TMP24:%.*]] = load i32, ptr [[TMP23]], align 4
+// COMP-NEXT:    [[TMP25:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.8, ptr @.gomp_critical_user_.reduction.var)
+// COMP-NEXT:    switch i32 [[TMP25]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
 // COMP-NEXT:    i32 1, label [[DOTOMP_REDUCTION_CASE1:%.*]]
 // COMP-NEXT:    i32 2, label [[DOTOMP_REDUCTION_CASE2:%.*]]
 // COMP-NEXT:    ]
 // COMP:       .omp.reduction.case1:
 // COMP-NEXT:    [[CALL:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointoRERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
-// COMP-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], ptr @.gomp_critical_user_.reduction.var)
+// COMP-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], ptr @.gomp_critical_user_.reduction.var)
 // COMP-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // COMP:       .omp.reduction.case2:
-// COMP-NEXT:    [[TMP28:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// COMP-NEXT:    [[TMP29:%.*]] = load i32, ptr [[TMP28]], align 4
-// COMP-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP29]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// COMP-NEXT:    [[TMP26:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// COMP-NEXT:    [[TMP27:%.*]] = load i32, ptr [[TMP26]], align 4
+// COMP-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // COMP-NEXT:    [[CALL9:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointoRERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
-// COMP-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP29]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// COMP-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // COMP-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // COMP:       .omp.reduction.default:
 // COMP-NEXT:    br label [[OMP_PRECOND_END]]
@@ -1983,12 +1982,12 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    store ptr [[TMP0]], ptr [[DOTADDR]], align 8
 // COMP-NEXT:    store ptr [[TMP1]], ptr [[DOTADDR1]], align 8
 // COMP-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[DOTADDR]], align 8
-// COMP-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
-// COMP-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP4]], i64 0, i64 0
+// COMP-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
+// COMP-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP3]], i64 0, i64 0
+// COMP-NEXT:    [[TMP5:%.*]] = load ptr, ptr [[TMP4]], align 8
+// COMP-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
 // COMP-NEXT:    [[TMP7:%.*]] = load ptr, ptr [[TMP6]], align 8
-// COMP-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
-// COMP-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[TMP9]], align 8
-// COMP-NEXT:    [[CALL:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointoRERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[TMP7]])
+// COMP-NEXT:    [[CALL:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointoRERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[TMP5]])
 // COMP-NEXT:    ret void
 //
 //
@@ -2087,23 +2086,23 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB1]], i32 [[TMP21]])
 // COMP-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOMP_REDUCTION_RED_LIST]], i64 0, i64 0
 // COMP-NEXT:    store ptr [[RED3]], ptr [[TMP22]], align 8
-// COMP-NEXT:    [[TMP24:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// COMP-NEXT:    [[TMP25:%.*]] = load i32, ptr [[TMP24]], align 4
-// COMP-NEXT:    [[TMP27:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.10, ptr @.gomp_critical_user_.reduction.var)
-// COMP-NEXT:    switch i32 [[TMP27]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
+// COMP-NEXT:    [[TMP23:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// COMP-NEXT:    [[TMP24:%.*]] = load i32, ptr [[TMP23]], align 4
+// COMP-NEXT:    [[TMP25:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.10, ptr @.gomp_critical_user_.reduction.var)
+// COMP-NEXT:    switch i32 [[TMP25]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
 // COMP-NEXT:    i32 1, label [[DOTOMP_REDUCTION_CASE1:%.*]]
 // COMP-NEXT:    i32 2, label [[DOTOMP_REDUCTION_CASE2:%.*]]
 // COMP-NEXT:    ]
 // COMP:       .omp.reduction.case1:
 // COMP-NEXT:    [[CALL:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointeOERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
-// COMP-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], ptr @.gomp_critical_user_.reduction.var)
+// COMP-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], ptr @.gomp_critical_user_.reduction.var)
 // COMP-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // COMP:       .omp.reduction.case2:
-// COMP-NEXT:    [[TMP28:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// COMP-NEXT:    [[TMP29:%.*]] = load i32, ptr [[TMP28]], align 4
-// COMP-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP29]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// COMP-NEXT:    [[TMP26:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// COMP-NEXT:    [[TMP27:%.*]] = load i32, ptr [[TMP26]], align 4
+// COMP-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // COMP-NEXT:    [[CALL9:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointeOERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
-// COMP-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP29]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// COMP-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // COMP-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // COMP:       .omp.reduction.default:
 // COMP-NEXT:    br label [[OMP_PRECOND_END]]
@@ -2119,12 +2118,12 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    store ptr [[TMP0]], ptr [[DOTADDR]], align 8
 // COMP-NEXT:    store ptr [[TMP1]], ptr [[DOTADDR1]], align 8
 // COMP-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[DOTADDR]], align 8
-// COMP-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
-// COMP-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP4]], i64 0, i64 0
+// COMP-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
+// COMP-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP3]], i64 0, i64 0
+// COMP-NEXT:    [[TMP5:%.*]] = load ptr, ptr [[TMP4]], align 8
+// COMP-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
 // COMP-NEXT:    [[TMP7:%.*]] = load ptr, ptr [[TMP6]], align 8
-// COMP-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
-// COMP-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[TMP9]], align 8
-// COMP-NEXT:    [[CALL:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointeOERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[TMP7]])
+// COMP-NEXT:    [[CALL:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointeOERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[TMP5]])
 // COMP-NEXT:    ret void
 //
 //
@@ -2225,10 +2224,10 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB1]], i32 [[TMP21]])
 // COMP-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOMP_REDUCTION_RED_LIST]], i64 0, i64 0
 // COMP-NEXT:    store ptr [[RED3]], ptr [[TMP22]], align 8
-// COMP-NEXT:    [[TMP24:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// COMP-NEXT:    [[TMP25:%.*]] = load i32, ptr [[TMP24]], align 4
-// COMP-NEXT:    [[TMP27:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.12, ptr @.gomp_critical_user_.reduction.var)
-// COMP-NEXT:    switch i32 [[TMP27]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
+// COMP-NEXT:    [[TMP23:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// COMP-NEXT:    [[TMP24:%.*]] = load i32, ptr [[TMP23]], align 4
+// COMP-NEXT:    [[TMP25:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.12, ptr @.gomp_critical_user_.reduction.var)
+// COMP-NEXT:    switch i32 [[TMP25]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
 // COMP-NEXT:    i32 1, label [[DOTOMP_REDUCTION_CASE1:%.*]]
 // COMP-NEXT:    i32 2, label [[DOTOMP_REDUCTION_CASE2:%.*]]
 // COMP-NEXT:    ]
@@ -2236,16 +2235,16 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointaaERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // COMP-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
 // COMP-NEXT:    [[CALL9:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
-// COMP-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], ptr @.gomp_critical_user_.reduction.var)
+// COMP-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], ptr @.gomp_critical_user_.reduction.var)
 // COMP-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // COMP:       .omp.reduction.case2:
-// COMP-NEXT:    [[TMP29:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// COMP-NEXT:    [[TMP30:%.*]] = load i32, ptr [[TMP29]], align 4
-// COMP-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// COMP-NEXT:    [[TMP26:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// COMP-NEXT:    [[TMP27:%.*]] = load i32, ptr [[TMP26]], align 4
+// COMP-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // COMP-NEXT:    [[CALL11:%.*]] = call i64 @_ZNK5PointaaERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // COMP-NEXT:    store i64 [[CALL11]], ptr [[REF_TMP10]], align 4
 // COMP-NEXT:    [[CALL12:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP10]])
-// COMP-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// COMP-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // COMP-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // COMP:       .omp.reduction.default:
 // COMP-NEXT:    br label [[OMP_PRECOND_END]]
@@ -2262,14 +2261,14 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    store ptr [[TMP0]], ptr [[DOTADDR]], align 8
 // COMP-NEXT:    store ptr [[TMP1]], ptr [[DOTADDR1]], align 8
 // COMP-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[DOTADDR]], align 8
-// COMP-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
-// COMP-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP4]], i64 0, i64 0
+// COMP-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
+// COMP-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP3]], i64 0, i64 0
+// COMP-NEXT:    [[TMP5:%.*]] = load ptr, ptr [[TMP4]], align 8
+// COMP-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
 // COMP-NEXT:    [[TMP7:%.*]] = load ptr, ptr [[TMP6]], align 8
-// COMP-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
-// COMP-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[TMP9]], align 8
-// COMP-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointaaERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[TMP7]])
+// COMP-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointaaERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[TMP5]])
 // COMP-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
-// COMP-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
+// COMP-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
 // COMP-NEXT:    ret void
 //
 //
@@ -2370,10 +2369,10 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    call void @__kmpc_for_static_fini(ptr @[[GLOB1]], i32 [[TMP21]])
 // COMP-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOMP_REDUCTION_RED_LIST]], i64 0, i64 0
 // COMP-NEXT:    store ptr [[RED3]], ptr [[TMP22]], align 8
-// COMP-NEXT:    [[TMP24:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// COMP-NEXT:    [[TMP25:%.*]] = load i32, ptr [[TMP24]], align 4
-// COMP-NEXT:    [[TMP27:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.14, ptr @.gomp_critical_user_.reduction.var)
-// COMP-NEXT:    switch i32 [[TMP27]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
+// COMP-NEXT:    [[TMP23:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// COMP-NEXT:    [[TMP24:%.*]] = load i32, ptr [[TMP23]], align 4
+// COMP-NEXT:    [[TMP25:%.*]] = call i32 @__kmpc_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], i32 1, i64 8, ptr [[DOTOMP_REDUCTION_RED_LIST]], ptr @.omp.reduction.reduction_func.14, ptr @.gomp_critical_user_.reduction.var)
+// COMP-NEXT:    switch i32 [[TMP25]], label [[DOTOMP_REDUCTION_DEFAULT:%.*]] [
 // COMP-NEXT:    i32 1, label [[DOTOMP_REDUCTION_CASE1:%.*]]
 // COMP-NEXT:    i32 2, label [[DOTOMP_REDUCTION_CASE2:%.*]]
 // COMP-NEXT:    ]
@@ -2381,16 +2380,16 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointooERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // COMP-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
 // COMP-NEXT:    [[CALL9:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
-// COMP-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP25]], ptr @.gomp_critical_user_.reduction.var)
+// COMP-NEXT:    call void @__kmpc_end_reduce_nowait(ptr @[[GLOB2]], i32 [[TMP24]], ptr @.gomp_critical_user_.reduction.var)
 // COMP-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // COMP:       .omp.reduction.case2:
-// COMP-NEXT:    [[TMP29:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
-// COMP-NEXT:    [[TMP30:%.*]] = load i32, ptr [[TMP29]], align 4
-// COMP-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// COMP-NEXT:    [[TMP26:%.*]] = load ptr, ptr [[DOTGLOBAL_TID__ADDR]], align 8
+// COMP-NEXT:    [[TMP27:%.*]] = load i32, ptr [[TMP26]], align 4
+// COMP-NEXT:    call void @__kmpc_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // COMP-NEXT:    [[CALL11:%.*]] = call i64 @_ZNK5PointooERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[RED3]])
 // COMP-NEXT:    store i64 [[CALL11]], ptr [[REF_TMP10]], align 4
 // COMP-NEXT:    [[CALL12:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP1]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP10]])
-// COMP-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP30]], ptr @.gomp_critical_user_.atomic_reduction.var)
+// COMP-NEXT:    call void @__kmpc_end_critical(ptr @[[GLOB3]], i32 [[TMP27]], ptr @.gomp_critical_user_.atomic_reduction.var)
 // COMP-NEXT:    br label [[DOTOMP_REDUCTION_DEFAULT]]
 // COMP:       .omp.reduction.default:
 // COMP-NEXT:    br label [[OMP_PRECOND_END]]
@@ -2407,14 +2406,14 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    store ptr [[TMP0]], ptr [[DOTADDR]], align 8
 // COMP-NEXT:    store ptr [[TMP1]], ptr [[DOTADDR1]], align 8
 // COMP-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[DOTADDR]], align 8
-// COMP-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
-// COMP-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP4]], i64 0, i64 0
+// COMP-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[DOTADDR1]], align 8
+// COMP-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP3]], i64 0, i64 0
+// COMP-NEXT:    [[TMP5:%.*]] = load ptr, ptr [[TMP4]], align 8
+// COMP-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
 // COMP-NEXT:    [[TMP7:%.*]] = load ptr, ptr [[TMP6]], align 8
-// COMP-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [1 x ptr], ptr [[TMP2]], i64 0, i64 0
-// COMP-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[TMP9]], align 8
-// COMP-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointooERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[TMP7]])
+// COMP-NEXT:    [[CALL:%.*]] = call i64 @_ZNK5PointooERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[TMP5]])
 // COMP-NEXT:    store i64 [[CALL]], ptr [[REF_TMP]], align 4
-// COMP-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP10]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
+// COMP-NEXT:    [[CALL2:%.*]] = call nonnull align 4 dereferenceable(8) ptr @_ZN5PointaSERKS_(ptr nonnull align 4 dereferenceable(8) [[TMP7]], ptr nonnull align 4 dereferenceable(8) [[REF_TMP]])
 // COMP-NEXT:    ret void
 //
 //
@@ -2430,3 +2429,190 @@ void foo(int N, Point const *Points) {
 // COMP-NEXT:    store i32 0, ptr [[Y]], align 4
 // COMP-NEXT:    ret void
 //
+//
+// SIMD-ONLY-LABEL: define {{[^@]+}}@_Z3fooiPK5Point
+// SIMD-ONLY-SAME: (i32 [[N:%.*]], ptr [[POINTS:%.*]]) #[[ATTR0:[0-9]+]] {
+// SIMD-ONLY-NEXT:  entry:
+// SIMD-ONLY-NEXT:    [[N_ADDR:%.*]] = alloca i32, align 4
+// SIMD-ONLY-NEXT:    [[POINTS_ADDR:%.*]] = alloca ptr, align 8
+// SIMD-ONLY-NEXT:    [[RED:%.*]] = alloca [[STRUCT_POINT:%.*]], align 4
+// SIMD-ONLY-NEXT:    [[I:%.*]] = alloca i32, align 4
+// SIMD-ONLY-NEXT:    [[I1:%.*]] = alloca i32, align 4
+// SIMD-ONLY-NEXT:    [[I8:%.*]] = alloca i32, align 4
+// SIMD-ONLY-NEXT:    [[I15:%.*]] = alloca i32, align 4
+// SIMD-ONLY-NEXT:    [[I22:%.*]] = alloca i32, align 4
+// SIMD-ONLY-NEXT:    [[I29:%.*]] = alloca i32, align 4
+// SIMD-ONLY-NEXT:    [[I36:%.*]] = alloca i32, align 4
+// SIMD-ONLY-NEXT:    [[I43:%.*]] = alloca i32, align 4
+// SIMD-ONLY-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
+// SIMD-ONLY-NEXT:    store ptr [[POINTS]], ptr [[POINTS_ADDR]], align 8
+// SIMD-ONLY-NEXT:    call void @_ZN5PointC1Ev(ptr nonnull align 4 dereferenceable(8) [[RED]]) #[[ATTR3:[0-9]+]]
+// SIMD-ONLY-NEXT:    store i32 0, ptr [[I]], align 4
+// SIMD-ONLY-NEXT:    br label [[FOR_COND:%.*]]
+// SIMD-ONLY:       for.cond:
+// SIMD-ONLY-NEXT:    [[TMP0:%.*]] = load i32, ptr [[I]], align 4
+// SIMD-ONLY-NEXT:    [[TMP1:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// SIMD-ONLY-NEXT:    [[CMP:%.*]] = icmp ult i32 [[TMP0]], [[TMP1]]
+// SIMD-ONLY-NEXT:    br i1 [[CMP]], label [[FOR_BODY:%.*]], label [[FOR_END:%.*]]
+// SIMD-ONLY:       for.body:
+// SIMD-ONLY-NEXT:    [[TMP2:%.*]] = load i32, ptr [[I]], align 4
+// SIMD-ONLY-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[POINTS_ADDR]], align 8
+// SIMD-ONLY-NEXT:    call void @_Z4workR5PointiPKS_(ptr nonnull align 4 dereferenceable(8) [[RED]], i32 [[TMP2]], ptr [[TMP3]])
+// SIMD-ONLY-NEXT:    br label [[FOR_INC:%.*]]
+// SIMD-ONLY:       for.inc:
+// SIMD-ONLY-NEXT:    [[TMP4:%.*]] = load i32, ptr [[I]], align 4
+// SIMD-ONLY-NEXT:    [[INC:%.*]] = add i32 [[TMP4]], 1
+// SIMD-ONLY-NEXT:    store i32 [[INC]], ptr [[I]], align 4
+// SIMD-ONLY-NEXT:    br label [[FOR_COND]], !llvm.loop [[LOOP2:![0-9]+]]
+// SIMD-ONLY:       for.end:
+// SIMD-ONLY-NEXT:    store i32 0, ptr [[I1]], align 4
+// SIMD-ONLY-NEXT:    br label [[FOR_COND2:%.*]]
+// SIMD-ONLY:       for.cond2:
+// SIMD-ONLY-NEXT:    [[TMP5:%.*]] = load i32, ptr [[I1]], align 4
+// SIMD-ONLY-NEXT:    [[TMP6:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// SIMD-ONLY-NEXT:    [[CMP3:%.*]] = icmp ult i32 [[TMP5]], [[TMP6]]
+// SIMD-ONLY-NEXT:    br i1 [[CMP3]], label [[FOR_BODY4:%.*]], label [[FOR_END7:%.*]]
+// SIMD-ONLY:       for.body4:
+// SIMD-ONLY-NEXT:    [[TMP7:%.*]] = load i32, ptr [[I1]], align 4
+// SIMD-ONLY-NEXT:    [[TMP8:%.*]] = load ptr, ptr [[POINTS_ADDR]], align 8
+// SIMD-ONLY-NEXT:    call void @_Z4workR5PointiPKS_(ptr nonnull align 4 dereferenceable(8) [[RED]], i32 [[TMP7]], ptr [[TMP8]])
+// SIMD-ONLY-NEXT:    br label [[FOR_INC5:%.*]]
+// SIMD-ONLY:       for.inc5:
+// SIMD-ONLY-NEXT:    [[TMP9:%.*]] = load i32, ptr [[I1]], align 4
+// SIMD-ONLY-NEXT:    [[INC6:%.*]] = add i32 [[TMP9]], 1
+// SIMD-ONLY-NEXT:    store i32 [[INC6]], ptr [[I1]], align 4
+// SIMD-ONLY-NEXT:    br label [[FOR_COND2]], !llvm.loop [[LOOP4:![0-9]+]]
+// SIMD-ONLY:       for.end7:
+// SIMD-ONLY-NEXT:    store i32 0, ptr [[I8]], align 4
+// SIMD-ONLY-NEXT:    br label [[FOR_COND9:%.*]]
+// SIMD-ONLY:       for.cond9:
+// SIMD-ONLY-NEXT:    [[TMP10:%.*]] = load i32, ptr [[I8]], align 4
+// SIMD-ONLY-NEXT:    [[TMP11:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// SIMD-ONLY-NEXT:    [[CMP10:%.*]] = icmp ult i32 [[TMP10]], [[TMP11]]
+// SIMD-ONLY-NEXT:    br i1 [[CMP10]], label [[FOR_BODY11:%.*]], label [[FOR_END14:%.*]]
+// SIMD-ONLY:       for.body11:
+// SIMD-ONLY-NEXT:    [[TMP12:%.*]] = load i32, ptr [[I8]], align 4
+// SIMD-ONLY-NEXT:    [[TMP13:%.*]] = load ptr, ptr [[POINTS_ADDR]], align 8
+// SIMD-ONLY-NEXT:    call void @_Z4workR5PointiPKS_(ptr nonnull align 4 dereferenceable(8) [[RED]], i32 [[TMP12]], ptr [[TMP13]])
+// SIMD-ONLY-NEXT:    br label [[FOR_INC12:%.*]]
+// SIMD-ONLY:       for.inc12:
+// SIMD-ONLY-NEXT:    [[TMP14:%.*]] = load i32, ptr [[I8]], align 4
+// SIMD-ONLY-NEXT:    [[INC13:%.*]] = add i32 [[TMP14]], 1
+// SIMD-ONLY-NEXT:    store i32 [[INC13]], ptr [[I8]], align 4
+// SIMD-ONLY-NEXT:    br label [[FOR_COND9]], !llvm.loop [[LOOP5:![0-9]+]]
+// SIMD-ONLY:       for.end14:
+// SIMD-ONLY-NEXT:    store i32 0, ptr [[I15]], align 4
+// SIMD-ONLY-NEXT:    br label [[FOR_COND16:%.*]]
+// SIMD-ONLY:       for.cond16:
+// SIMD-ONLY-NEXT:    [[TMP15:%.*]] = load i32, ptr [[I15]], align 4
+// SIMD-ONLY-NEXT:    [[TMP16:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// SIMD-ONLY-NEXT:    [[CMP17:%.*]] = icmp ult i32 [[TMP15]], [[TMP16]]
+// SIMD-ONLY-NEXT:    br i1 [[CMP17]], label [[FOR_BODY18:%.*]], label [[FOR_END21:%.*]]
+// SIMD-ONLY:       for.body18:
+// SIMD-ONLY-NEXT:    [[TMP17:%.*]] = load i32, ptr [[I15]], align 4
+// SIMD-ONLY-NEXT:    [[TMP18:%.*]] = load ptr, ptr [[POINTS_ADDR]], align 8
+// SIMD-ONLY-NEXT:    call void @_Z4workR5PointiPKS_(ptr nonnull align 4 dereferenceable(8) [[RED]], i32 [[TMP17]], ptr [[TMP18]])
+// SIMD-ONLY-NEXT:    br label [[FOR_INC19:%.*]]
+// SIMD-ONLY:       for.inc19:
+// SIMD-ONLY-NEXT:    [[TMP19:%.*]] = load i32, ptr [[I15]], align 4
+// SIMD-ONLY-NEXT:    [[INC20:%.*]] = add i32 [[TMP19]], 1
+// SIMD-ONLY-NEXT:    store i32 [[INC20]], ptr [[I15]], align 4
+// SIMD-ONLY-NEXT:    br label [[FOR_COND16]], !llvm.loop [[LOOP6:![0-9]+]]
+// SIMD-ONLY:       for.end21:
+// SIMD-ONLY-NEXT:    store i32 0, ptr [[I22]], align 4
+// SIMD-ONLY-NEXT:    br label [[FOR_COND23:%.*]]
+// SIMD-ONLY:       for.cond23:
+// SIMD-ONLY-NEXT:    [[TMP20:%.*]] = load i32, ptr [[I22]], align 4
+// SIMD-ONLY-NEXT:    [[TMP21:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// SIMD-ONLY-NEXT:    [[CMP24:%.*]] = icmp ult i32 [[TMP20]], [[TMP21]]
+// SIMD-ONLY-NEXT:    br i1 [[CMP24]], label [[FOR_BODY25:%.*]], label [[FOR_END28:%.*]]
+// SIMD-ONLY:       for.body25:
+// SIMD-ONLY-NEXT:    [[TMP22:%.*]] = load i32, ptr [[I22]], align 4
+// SIMD-ONLY-NEXT:    [[TMP23:%.*]] = load ptr, ptr [[POINTS_ADDR]], align 8
+// SIMD-ONLY-NEXT:    call void @_Z4workR5PointiPKS_(ptr nonnull align 4 dereferenceable(8) [[RED]], i32 [[TMP22]], ptr [[TMP23]])
+// SIMD-ONLY-NEXT:    br label [[FOR_INC26:%.*]]
+// SIMD-ONLY:       for.inc26:
+// SIMD-ONLY-NEXT:    [[TMP24:%.*]] = load i32, ptr [[I22]], align 4
+// SIMD-ONLY-NEXT:    [[INC27:%.*]] = add i32 [[TMP24]], 1
+// SIMD-ONLY-NEXT:    store i32 [[INC27]], ptr [[I22]], align 4
+// SIMD-ONLY-NEXT:    br label [[FOR_COND23]], !llvm.loop [[LOOP7:![0-9]+]]
+// SIMD-ONLY:       for.end28:
+// SIMD-ONLY-NEXT:    store i32 0, ptr [[I29]], align 4
+// SIMD-ONLY-NEXT:    br label [[FOR_COND30:%.*]]
+// SIMD-ONLY:       for.cond30:
+// SIMD-ONLY-NEXT:    [[TMP25:%.*]] = load i32, ptr [[I29]], align 4
+// SIMD-ONLY-NEXT:    [[TMP26:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// SIMD-ONLY-NEXT:    [[CMP31:%.*]] = icmp ult i32 [[TMP25]], [[TMP26]]
+// SIMD-ONLY-NEXT:    br i1 [[CMP31]], label [[FOR_BODY32:%.*]], label [[FOR_END35:%.*]]
+// SIMD-ONLY:       for.body32:
+// SIMD-ONLY-NEXT:    [[TMP27:%.*]] = load i32, ptr [[I29]], align 4
+// SIMD-ONLY-NEXT:    [[TMP28:%.*]] = load ptr, ptr [[POINTS_ADDR]], align 8
+// SIMD-ONLY-NEXT:    call void @_Z4workR5PointiPKS_(ptr nonnull align 4 dereferenceable(8) [[RED]], i32 [[TMP27]], ptr [[TMP28]])
+// SIMD-ONLY-NEXT:    br label [[FOR_INC33:%.*]]
+// SIMD-ONLY:       for.inc33:
+// SIMD-ONLY-NEXT:    [[TMP29:%.*]] = load i32, ptr [[I29]], align 4
+// SIMD-ONLY-NEXT:    [[INC34:%.*]] = add i32 [[TMP29]], 1
+// SIMD-ONLY-NEXT:    store i32 [[INC34]], ptr [[I29]], align 4
+// SIMD-ONLY-NEXT:    br label [[FOR_COND30]], !llvm.loop [[LOOP8:![0-9]+]]
+// SIMD-ONLY:       for.end35:
+// SIMD-ONLY-NEXT:    store i32 0, ptr [[I36]], align 4
+// SIMD-ONLY-NEXT:    br label [[FOR_COND37:%.*]]
+// SIMD-ONLY:       for.cond37:
+// SIMD-ONLY-NEXT:    [[TMP30:%.*]] = load i32, ptr [[I36]], align 4
+// SIMD-ONLY-NEXT:    [[TMP31:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// SIMD-ONLY-NEXT:    [[CMP38:%.*]] = icmp ult i32 [[TMP30]], [[TMP31]]
+// SIMD-ONLY-NEXT:    br i1 [[CMP38]], label [[FOR_BODY39:%.*]], label [[FOR_END42:%.*]]
+// SIMD-ONLY:       for.body39:
+// SIMD-ONLY-NEXT:    [[TMP32:%.*]] = load i32, ptr [[I36]], align 4
+// SIMD-ONLY-NEXT:    [[TMP33:%.*]] = load ptr, ptr [[POINTS_ADDR]], align 8
+// SIMD-ONLY-NEXT:    call void @_Z4workR5PointiPKS_(ptr nonnull align 4 dereferenceable(8) [[RED]], i32 [[TMP32]], ptr [[TMP33]])
+// SIMD-ONLY-NEXT:    br label [[FOR_INC40:%.*]]
+// SIMD-ONLY:       for.inc40:
+// SIMD-ONLY-NEXT:    [[TMP34:%.*]] = load i32, ptr [[I36]], align 4
+// SIMD-ONLY-NEXT:    [[INC41:%.*]] = add i32 [[TMP34]], 1
+// SIMD-ONLY-NEXT:    store i32 [[INC41]], ptr [[I36]], align 4
+// SIMD-ONLY-NEXT:    br label [[FOR_COND37]], !llvm.loop [[LOOP9:![0-9]+]]
+// SIMD-ONLY:       for.end42:
+// SIMD-ONLY-NEXT:    store i32 0, ptr [[I43]], align 4
+// SIMD-ONLY-NEXT:    br label [[FOR_COND44:%.*]]
+// SIMD-ONLY:       for.cond44:
+// SIMD-ONLY-NEXT:    [[TMP35:%.*]] = load i32, ptr [[I43]], align 4
+// SIMD-ONLY-NEXT:    [[TMP36:%.*]] = load i32, ptr [[N_ADDR]], align 4
+// SIMD-ONLY-NEXT:    [[CMP45:%.*]] = icmp ult i32 [[TMP35]], [[TMP36]]
+// SIMD-ONLY-NEXT:    br i1 [[CMP45]], label [[FOR_BODY46:%.*]], label [[FOR_END49:%.*]]
+// SIMD-ONLY:       for.body46:
+// SIMD-ONLY-NEXT:    [[TMP37:%.*]] = load i32, ptr [[I43]], align 4
+// SIMD-ONLY-NEXT:    [[TMP38:%.*]] = load ptr, ptr [[POINTS_ADDR]], align 8
+// SIMD-ONLY-NEXT:    call void @_Z4workR5PointiPKS_(ptr nonnull align 4 dereferenceable(8) [[RED]], i32 [[TMP37]], ptr [[TMP38]])
+// SIMD-ONLY-NEXT:    br label [[FOR_INC47:%.*]]
+// SIMD-ONLY:       for.inc47:
+// SIMD-ONLY-NEXT:    [[TMP39:%.*]] = load i32, ptr [[I43]], align 4
+// SIMD-ONLY-NEXT:    [[INC48:%.*]] = add i32 [[TMP39]], 1
+// SIMD-ONLY-NEXT:    store i32 [[INC48]], ptr [[I43]], align 4
+// SIMD-ONLY-NEXT:    br label [[FOR_COND44]], !llvm.loop [[LOOP10:![0-9]+]]
+// SIMD-ONLY:       for.end49:
+// SIMD-ONLY-NEXT:    ret void
+//
+//
+// SIMD-ONLY-LABEL: define {{[^@]+}}@_ZN5PointC1Ev
+// SIMD-ONLY-SAME: (ptr nonnull align 4 dereferenceable(8) [[THIS:%.*]]) unnamed_addr #[[ATTR1:[0-9]+]] comdat align 2 {
+// SIMD-ONLY-NEXT:  entry:
+// SIMD-ONLY-NEXT:    [[THIS_ADDR:%.*]] = alloca ptr, align 8
+// SIMD-ONLY-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
+// SIMD-ONLY-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
+// SIMD-ONLY-NEXT:    call void @_ZN5PointC2Ev(ptr nonnull align 4 dereferenceable(8) [[THIS1]]) #[[ATTR3]]
+// SIMD-ONLY-NEXT:    ret void
+//
+//
+// SIMD-ONLY-LABEL: define {{[^@]+}}@_ZN5PointC2Ev
+// SIMD-ONLY-SAME: (ptr nonnull align 4 dereferenceable(8) [[THIS:%.*]]) unnamed_addr #[[ATTR1]] comdat align 2 {
+// SIMD-ONLY-NEXT:  entry:
+// SIMD-ONLY-NEXT:    [[THIS_ADDR:%.*]] = alloca ptr, align 8
+// SIMD-ONLY-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
+// SIMD-ONLY-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
+// SIMD-ONLY-NEXT:    [[X:%.*]] = getelementptr inbounds [[STRUCT_POINT:%.*]], ptr [[THIS1]], i32 0, i32 0
+// SIMD-ONLY-NEXT:    store i32 0, ptr [[X]], align 4
+// SIMD-ONLY-NEXT:    [[Y:%.*]] = getelementptr inbounds [[STRUCT_POINT]], ptr [[THIS1]], i32 0, i32 1
+// SIMD-ONLY-NEXT:    store i32 0, ptr [[Y]], align 4
+// SIMD-ONLY-NEXT:    ret void
+//
diff --git a/clang/test/OpenMP/reduction_implicit_map.cpp b/clang/test/OpenMP/reduction_implicit_map.cpp
index 610b9e737ca93..d190250b04c73 100644
--- a/clang/test/OpenMP/reduction_implicit_map.cpp
+++ b/clang/test/OpenMP/reduction_implicit_map.cpp
@@ -293,10 +293,12 @@ int main()
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[B:%.*]] = alloca [10 x [10 x [10 x double]]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS4:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[ARRAY_BEGIN:%.*]] = getelementptr inbounds [5 x %class.S2], ptr [[O]], i32 0, i32 0
 // CHECK1-NEXT:    [[ARRAYCTOR_END:%.*]] = getelementptr inbounds [[CLASS_S2:%.*]], ptr [[ARRAY_BEGIN]], i64 5
 // CHECK1-NEXT:    br label [[ARRAYCTOR_LOOP:%.*]]
@@ -316,7 +318,6 @@ int main()
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -358,7 +359,6 @@ int main()
 // CHECK1-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK1-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS4:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS4]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS4]], i32 0, i32 1
@@ -825,6 +825,7 @@ int main()
 // CHECK2-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK2-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK2-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
+// CHECK2-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK2-NEXT:    [[SIZE_CASTED4:%.*]] = alloca i32, align 4
 // CHECK2-NEXT:    [[DOTOFFLOAD_BASEPTRS7:%.*]] = alloca [3 x ptr], align 4
 // CHECK2-NEXT:    [[DOTOFFLOAD_PTRS8:%.*]] = alloca [3 x ptr], align 4
@@ -833,15 +834,18 @@ int main()
 // CHECK2-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
 // CHECK2-NEXT:    [[DOTCAPTURE_EXPR_12:%.*]] = alloca i32, align 4
 // CHECK2-NEXT:    [[DOTCAPTURE_EXPR_13:%.*]] = alloca i32, align 4
+// CHECK2-NEXT:    [[KERNEL_ARGS18:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK2-NEXT:    [[A:%.*]] = alloca [10 x i32], align 4
 // CHECK2-NEXT:    [[SIZE_CASTED21:%.*]] = alloca i32, align 4
 // CHECK2-NEXT:    [[DOTOFFLOAD_BASEPTRS23:%.*]] = alloca [2 x ptr], align 4
 // CHECK2-NEXT:    [[DOTOFFLOAD_PTRS24:%.*]] = alloca [2 x ptr], align 4
 // CHECK2-NEXT:    [[DOTOFFLOAD_MAPPERS25:%.*]] = alloca [2 x ptr], align 4
+// CHECK2-NEXT:    [[KERNEL_ARGS26:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK2-NEXT:    [[SIZE_CASTED29:%.*]] = alloca i32, align 4
 // CHECK2-NEXT:    [[DOTOFFLOAD_BASEPTRS31:%.*]] = alloca [2 x ptr], align 4
 // CHECK2-NEXT:    [[DOTOFFLOAD_PTRS32:%.*]] = alloca [2 x ptr], align 4
 // CHECK2-NEXT:    [[DOTOFFLOAD_MAPPERS33:%.*]] = alloca [2 x ptr], align 4
+// CHECK2-NEXT:    [[KERNEL_ARGS34:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK2-NEXT:    store ptr [[INPUT]], ptr [[INPUT_ADDR]], align 4
 // CHECK2-NEXT:    store i32 [[SIZE]], ptr [[SIZE_ADDR]], align 4
 // CHECK2-NEXT:    store ptr [[OUTPUT]], ptr [[OUTPUT_ADDR]], align 4
@@ -893,7 +897,6 @@ int main()
 // CHECK2-NEXT:    [[TMP26:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_2]], align 4
 // CHECK2-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP26]], 1
 // CHECK2-NEXT:    [[TMP27:%.*]] = zext i32 [[ADD]] to i64
-// CHECK2-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK2-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK2-NEXT:    store i32 2, ptr [[TMP28]], align 4
 // CHECK2-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -975,7 +978,6 @@ int main()
 // CHECK2-NEXT:    [[TMP69:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_13]], align 4
 // CHECK2-NEXT:    [[ADD17:%.*]] = add nsw i32 [[TMP69]], 1
 // CHECK2-NEXT:    [[TMP70:%.*]] = zext i32 [[ADD17]] to i64
-// CHECK2-NEXT:    [[KERNEL_ARGS18:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK2-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS18]], i32 0, i32 0
 // CHECK2-NEXT:    store i32 2, ptr [[TMP71]], align 4
 // CHECK2-NEXT:    [[TMP72:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS18]], i32 0, i32 1
@@ -1027,7 +1029,6 @@ int main()
 // CHECK2-NEXT:    store ptr null, ptr [[TMP93]], align 4
 // CHECK2-NEXT:    [[TMP94:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS23]], i32 0, i32 0
 // CHECK2-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS24]], i32 0, i32 0
-// CHECK2-NEXT:    [[KERNEL_ARGS26:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK2-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS26]], i32 0, i32 0
 // CHECK2-NEXT:    store i32 2, ptr [[TMP96]], align 4
 // CHECK2-NEXT:    [[TMP97:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS26]], i32 0, i32 1
@@ -1079,7 +1080,6 @@ int main()
 // CHECK2-NEXT:    store ptr null, ptr [[TMP118]], align 4
 // CHECK2-NEXT:    [[TMP119:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS31]], i32 0, i32 0
 // CHECK2-NEXT:    [[TMP120:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS32]], i32 0, i32 0
-// CHECK2-NEXT:    [[KERNEL_ARGS34:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK2-NEXT:    [[TMP121:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS34]], i32 0, i32 0
 // CHECK2-NEXT:    store i32 2, ptr [[TMP121]], align 4
 // CHECK2-NEXT:    [[TMP122:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS34]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_codegen_global_capture.cpp b/clang/test/OpenMP/target_codegen_global_capture.cpp
index 4c9556c5aa908..858ac559ccc2e 100644
--- a/clang/test/OpenMP/target_codegen_global_capture.cpp
+++ b/clang/test/OpenMP/target_codegen_global_capture.cpp
@@ -185,6 +185,7 @@ int tbar2(short a, short b, short c, short d){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [9 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [9 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [9 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i16 [[A]], ptr [[A_ADDR]], align 2
 // CHECK1-NEXT:    store i16 [[B]], ptr [[B_ADDR]], align 2
 // CHECK1-NEXT:    store i16 [[C]], ptr [[C_ADDR]], align 2
@@ -286,7 +287,6 @@ int tbar2(short a, short b, short c, short d){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP47]], align 8
 // CHECK1-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP50]], align 4
 // CHECK1-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -506,6 +506,7 @@ int tbar2(short a, short b, short c, short d){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [9 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [9 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [9 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[DOTGLOBAL_TID_]], ptr [[DOTGLOBAL_TID__ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[DOTBOUND_TID_]], ptr [[DOTBOUND_TID__ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 8
@@ -613,7 +614,6 @@ int tbar2(short a, short b, short c, short d){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP51]], align 8
 // CHECK1-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP54:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP54]], align 4
 // CHECK1-NEXT:    [[TMP55:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -829,6 +829,7 @@ int tbar2(short a, short b, short c, short d){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [9 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [9 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [9 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[DOTGLOBAL_TID_]], ptr [[DOTGLOBAL_TID__ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[DOTBOUND_TID_]], ptr [[DOTBOUND_TID__ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 8
@@ -936,7 +937,6 @@ int tbar2(short a, short b, short c, short d){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP51]], align 8
 // CHECK1-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP54:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP54]], align 4
 // CHECK1-NEXT:    [[TMP55:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1097,6 +1097,7 @@ int tbar2(short a, short b, short c, short d){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [9 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [9 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [9 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i16 [[A]], ptr [[A_ADDR]], align 2
 // CHECK3-NEXT:    store i16 [[B]], ptr [[B_ADDR]], align 2
 // CHECK3-NEXT:    store i16 [[C]], ptr [[C_ADDR]], align 2
@@ -1189,7 +1190,6 @@ int tbar2(short a, short b, short c, short d){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP41]], align 4
 // CHECK3-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP44]], align 4
 // CHECK3-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1418,6 +1418,7 @@ int tbar2(short a, short b, short c, short d){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [9 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [9 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [9 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[DOTGLOBAL_TID_]], ptr [[DOTGLOBAL_TID__ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[DOTBOUND_TID_]], ptr [[DOTBOUND_TID__ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 4
@@ -1516,7 +1517,6 @@ int tbar2(short a, short b, short c, short d){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP45]], align 4
 // CHECK3-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP47:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP48]], align 4
 // CHECK3-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1741,6 +1741,7 @@ int tbar2(short a, short b, short c, short d){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [9 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [9 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [9 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[DOTGLOBAL_TID_]], ptr [[DOTGLOBAL_TID__ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[DOTBOUND_TID_]], ptr [[DOTBOUND_TID__ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 4
@@ -1839,7 +1840,6 @@ int tbar2(short a, short b, short c, short d){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP45]], align 4
 // CHECK3-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP47:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP48]], align 4
 // CHECK3-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_has_device_addr_codegen.cpp b/clang/test/OpenMP/target_has_device_addr_codegen.cpp
index c1111b31602e5..ef6f0e79516f3 100644
--- a/clang/test/OpenMP/target_has_device_addr_codegen.cpp
+++ b/clang/test/OpenMP/target_has_device_addr_codegen.cpp
@@ -286,23 +286,29 @@ void use_template() {
 // CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NEXT:    [[TMP:%.*]] = alloca ptr, align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS4:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS7:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS8:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS9:%.*]] = alloca [1 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS10:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[_TMP13:%.*]] = alloca ptr, align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS14:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS15:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS16:%.*]] = alloca [1 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS17:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS20:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS21:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS22:%.*]] = alloca [1 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS26:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS27:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS28:%.*]] = alloca [1 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS29:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -321,7 +327,6 @@ void use_template() {
 // CHECK-NEXT:    store ptr null, ptr [[TMP3]], align 8
 // CHECK-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP6]], align 4
 // CHECK-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -366,7 +371,6 @@ void use_template() {
 // CHECK-NEXT:    store ptr null, ptr [[TMP25]], align 8
 // CHECK-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK-NEXT:    [[KERNEL_ARGS4:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS4]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP28]], align 4
 // CHECK-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS4]], i32 0, i32 1
@@ -408,7 +412,6 @@ void use_template() {
 // CHECK-NEXT:    store ptr null, ptr [[TMP45]], align 8
 // CHECK-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS7]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP47:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS8]], i32 0, i32 0
-// CHECK-NEXT:    [[KERNEL_ARGS10:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS10]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP48]], align 4
 // CHECK-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS10]], i32 0, i32 1
@@ -453,7 +456,6 @@ void use_template() {
 // CHECK-NEXT:    store ptr null, ptr [[TMP67]], align 8
 // CHECK-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS14]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS15]], i32 0, i32 0
-// CHECK-NEXT:    [[KERNEL_ARGS17:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS17]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP70]], align 4
 // CHECK-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS17]], i32 0, i32 1
@@ -495,7 +497,6 @@ void use_template() {
 // CHECK-NEXT:    store ptr null, ptr [[TMP87]], align 8
 // CHECK-NEXT:    [[TMP88:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS20]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP89:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS21]], i32 0, i32 0
-// CHECK-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[TMP90:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP90]], align 4
 // CHECK-NEXT:    [[TMP91:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 1
@@ -537,7 +538,6 @@ void use_template() {
 // CHECK-NEXT:    store ptr null, ptr [[TMP107]], align 8
 // CHECK-NEXT:    [[TMP108:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS26]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP109:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS27]], i32 0, i32 0
-// CHECK-NEXT:    [[KERNEL_ARGS29:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[TMP110:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS29]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP110]], align 4
 // CHECK-NEXT:    [[TMP111:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS29]], i32 0, i32 1
@@ -675,16 +675,20 @@ void use_template() {
 // CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NEXT:    [[TMP:%.*]] = alloca ptr, align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS4:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS7:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS8:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS9:%.*]] = alloca [1 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS10:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS13:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS14:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS15:%.*]] = alloca [1 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS16:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK-NEXT:    call void @llvm.memset.p0.i64(ptr align 4 [[DA]], i8 0, i64 20, i1 false)
 // CHECK-NEXT:    store ptr [[H]], ptr [[RH]], align 8
@@ -700,7 +704,6 @@ void use_template() {
 // CHECK-NEXT:    store ptr null, ptr [[TMP3]], align 8
 // CHECK-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP6]], align 4
 // CHECK-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -745,7 +748,6 @@ void use_template() {
 // CHECK-NEXT:    store ptr null, ptr [[TMP25]], align 8
 // CHECK-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK-NEXT:    [[KERNEL_ARGS4:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS4]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP28]], align 4
 // CHECK-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS4]], i32 0, i32 1
@@ -787,7 +789,6 @@ void use_template() {
 // CHECK-NEXT:    store ptr null, ptr [[TMP45]], align 8
 // CHECK-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS7]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP47:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS8]], i32 0, i32 0
-// CHECK-NEXT:    [[KERNEL_ARGS10:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS10]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP48]], align 4
 // CHECK-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS10]], i32 0, i32 1
@@ -829,7 +830,6 @@ void use_template() {
 // CHECK-NEXT:    store ptr null, ptr [[TMP65]], align 8
 // CHECK-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS13]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP67:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS14]], i32 0, i32 0
-// CHECK-NEXT:    [[KERNEL_ARGS16:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS16]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP68]], align 4
 // CHECK-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS16]], i32 0, i32 1
@@ -881,16 +881,20 @@ void use_template() {
 // CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NEXT:    [[TMP:%.*]] = alloca ptr, align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS4:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS7:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS8:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS9:%.*]] = alloca [1 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS10:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS13:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS14:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS15:%.*]] = alloca [1 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS16:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    store ptr [[ARGC]], ptr [[ARGC_ADDR]], align 8
 // CHECK-NEXT:    call void @llvm.memset.p0.i64(ptr align 8 [[DA]], i8 0, i64 40, i1 false)
 // CHECK-NEXT:    store ptr [[H]], ptr [[RH]], align 8
@@ -906,7 +910,6 @@ void use_template() {
 // CHECK-NEXT:    store ptr null, ptr [[TMP3]], align 8
 // CHECK-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP6]], align 4
 // CHECK-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -951,7 +954,6 @@ void use_template() {
 // CHECK-NEXT:    store ptr null, ptr [[TMP25]], align 8
 // CHECK-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK-NEXT:    [[KERNEL_ARGS4:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS4]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP28]], align 4
 // CHECK-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS4]], i32 0, i32 1
@@ -993,7 +995,6 @@ void use_template() {
 // CHECK-NEXT:    store ptr null, ptr [[TMP45]], align 8
 // CHECK-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS7]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP47:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS8]], i32 0, i32 0
-// CHECK-NEXT:    [[KERNEL_ARGS10:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS10]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP48]], align 4
 // CHECK-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS10]], i32 0, i32 1
@@ -1035,7 +1036,6 @@ void use_template() {
 // CHECK-NEXT:    store ptr null, ptr [[TMP65]], align 8
 // CHECK-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS13]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP67:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS14]], i32 0, i32 0
-// CHECK-NEXT:    [[KERNEL_ARGS16:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS16]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP68]], align 4
 // CHECK-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS16]], i32 0, i32 1
@@ -1199,6 +1199,7 @@ void use_template() {
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [3 x i64], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK-NEXT:    [[TARGETDEV:%.*]] = getelementptr inbounds [[STRUCT_SOMEKERNEL:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -1237,7 +1238,6 @@ void use_template() {
 // CHECK-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP19:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
 // CHECK-NEXT:    [[TMP20:%.*]] = sext i32 [[TMP19]] to i64
-// CHECK-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_has_device_addr_codegen_01.cpp b/clang/test/OpenMP/target_has_device_addr_codegen_01.cpp
index 25c60fa688e74..4a5a385869df4 100644
--- a/clang/test/OpenMP/target_has_device_addr_codegen_01.cpp
+++ b/clang/test/OpenMP/target_has_device_addr_codegen_01.cpp
@@ -45,6 +45,7 @@ int main() {
 // CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [6 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [6 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [6 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK-NEXT:    store float 0.000000e+00, ptr [[A]], align 4
 // CHECK-NEXT:    store ptr [[A]], ptr [[PTR]], align 8
@@ -100,7 +101,6 @@ int main() {
 // CHECK-NEXT:    store ptr null, ptr [[TMP23]], align 8
 // CHECK-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [6 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [6 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP26]], align 4
 // CHECK-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -161,6 +161,7 @@ int main() {
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [5 x i64], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_S:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -209,7 +210,6 @@ int main() {
 // CHECK-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_map_codegen_03.cpp b/clang/test/OpenMP/target_map_codegen_03.cpp
index 552cc794a39bf..03f8175619f63 100644
--- a/clang/test/OpenMP/target_map_codegen_03.cpp
+++ b/clang/test/OpenMP/target_map_codegen_03.cpp
@@ -79,6 +79,7 @@ void implicit_maps_nested_integer (int a){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[DOTGLOBAL_TID_]], ptr [[DOTGLOBAL_TID__ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[DOTBOUND_TID_]], ptr [[DOTBOUND_TID__ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[I]], ptr [[I_ADDR]], align 8
@@ -94,7 +95,6 @@ void implicit_maps_nested_integer (int a){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP5]], align 8
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP8]], align 4
 // CHECK1-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -185,6 +185,7 @@ void implicit_maps_nested_integer (int a){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[DOTGLOBAL_TID_]], ptr [[DOTGLOBAL_TID__ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[DOTBOUND_TID_]], ptr [[DOTBOUND_TID__ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[I]], ptr [[I_ADDR]], align 4
@@ -200,7 +201,6 @@ void implicit_maps_nested_integer (int a){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP8]], align 4
 // CHECK3-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_map_codegen_hold.cpp b/clang/test/OpenMP/target_map_codegen_hold.cpp
index 7008d32d76cad..f6c3db95ec8f3 100644
--- a/clang/test/OpenMP/target_map_codegen_hold.cpp
+++ b/clang/test/OpenMP/target_map_codegen_hold.cpp
@@ -81,35 +81,6 @@
 // MEMBER_OF_1  = 0x1000000000000
 // MEMBER_OF_5  = 0x5000000000000
 
-//.
-// CHECK-USE-PPC64LE: @.offload_sizes = private unnamed_addr constant [7 x i64] [i64 0, i64 4, i64 4, i64 4, i64 0, i64 4, i64 4]
-// CHECK-USE-PPC64LE: @.offload_maptypes = private unnamed_addr constant [7 x i64] [i64 [[#0x2020]], i64 [[#0x1000000002003]], i64 [[#0x1000000002003]], i64 [[#0x2023]], i64 [[#0x2020]], i64 [[#0x5000000002003]], i64 [[#0x5000000002003]]]
-// CHECK-USE-PPC64LE: @.offload_sizes.1 = private unnamed_addr constant [1 x i64] [i64 4]
-// CHECK-USE-PPC64LE: @.offload_maptypes.2 = private unnamed_addr constant [1 x i64] [i64 [[#0x2427]]]
-// CHECK-USE-PPC64LE: @.offload_sizes.3 = private unnamed_addr constant [3 x i64] [i64 0, i64 4, i64 4]
-// CHECK-USE-PPC64LE: @.offload_maptypes.4 = private unnamed_addr constant [3 x i64] [i64 [[#0x2020]], i64 [[#0x1000000002003]], i64 [[#0x1000000002003]]]
-//.
-// CHECK-USE-I386: @.offload_sizes = private unnamed_addr constant [7 x i64] [i64 0, i64 4, i64 4, i64 4, i64 0, i64 4, i64 4]
-// CHECK-USE-I386: @.offload_maptypes = private unnamed_addr constant [7 x i64] [i64 [[#0x2020]], i64 [[#0x1000000002003]], i64 [[#0x1000000002003]], i64 [[#0x2023]], i64 [[#0x2020]], i64 [[#0x5000000002003]], i64 [[#0x5000000002003]]]
-// CHECK-USE-I386: @.offload_sizes.1 = private unnamed_addr constant [1 x i64] [i64 4]
-// CHECK-USE-I386: @.offload_maptypes.2 = private unnamed_addr constant [1 x i64] [i64 [[#0x2427]]]
-// CHECK-USE-I386: @.offload_sizes.3 = private unnamed_addr constant [3 x i64] [i64 0, i64 4, i64 4]
-// CHECK-USE-I386: @.offload_maptypes.4 = private unnamed_addr constant [3 x i64] [i64 [[#0x2020]], i64 [[#0x1000000002003]], i64 [[#0x1000000002003]]]
-//.
-// CHECK-NOUSE-PPC64LE: @.offload_sizes = private unnamed_addr constant [7 x i64] [i64 0, i64 4, i64 4, i64 4, i64 0, i64 4, i64 4]
-// CHECK-NOUSE-PPC64LE: @.offload_maptypes = private unnamed_addr constant [7 x i64] [i64 [[#0x2000]], i64 [[#0x1000000002003]], i64 [[#0x1000000002003]], i64 [[#0x2003]], i64 [[#0x2000]], i64 [[#0x5000000002003]], i64 [[#0x5000000002003]]]
-// CHECK-NOUSE-PPC64LE: @.offload_sizes.1 = private unnamed_addr constant [1 x i64] [i64 4]
-// CHECK-NOUSE-PPC64LE: @.offload_maptypes.2 = private unnamed_addr constant [1 x i64] [i64 [[#0x2407]]]
-// CHECK-NOUSE-PPC64LE: @.offload_sizes.3 = private unnamed_addr constant [3 x i64] [i64 0, i64 4, i64 4]
-// CHECK-NOUSE-PPC64LE: @.offload_maptypes.4 = private unnamed_addr constant [3 x i64] [i64 [[#0x2000]], i64 [[#0x1000000002003]], i64 [[#0x1000000002003]]]
-//.
-// CHECK-NOUSE-I386: @.offload_sizes = private unnamed_addr constant [7 x i64] [i64 0, i64 4, i64 4, i64 4, i64 0, i64 4, i64 4]
-// CHECK-NOUSE-I386: @.offload_maptypes = private unnamed_addr constant [7 x i64] [i64 [[#0x2000]], i64 [[#0x1000000002003]], i64 [[#0x1000000002003]], i64 [[#0x2003]], i64 [[#0x2000]], i64 [[#0x5000000002003]], i64 [[#0x5000000002003]]]
-// CHECK-NOUSE-I386: @.offload_sizes.1 = private unnamed_addr constant [1 x i64] [i64 4]
-// CHECK-NOUSE-I386: @.offload_maptypes.2 = private unnamed_addr constant [1 x i64] [i64 [[#0x2407]]]
-// CHECK-NOUSE-I386: @.offload_sizes.3 = private unnamed_addr constant [3 x i64] [i64 0, i64 4, i64 4]
-// CHECK-NOUSE-I386: @.offload_maptypes.4 = private unnamed_addr constant [3 x i64] [i64 [[#0x2000]], i64 [[#0x1000000002003]], i64 [[#0x1000000002003]]]
-//.
 struct ST {
   int i;
   int j;
@@ -162,6 +133,34 @@ void ST::test_present_members() {
 
 #endif
 //.
+// CHECK-USE-PPC64LE: @.offload_sizes = private unnamed_addr constant [7 x i64] [i64 0, i64 4, i64 4, i64 4, i64 0, i64 4, i64 4]
+// CHECK-USE-PPC64LE: @.offload_maptypes = private unnamed_addr constant [7 x i64] [i64 8224, i64 281474976718851, i64 281474976718851, i64 8227, i64 8224, i64 1407374883561475, i64 1407374883561475]
+// CHECK-USE-PPC64LE: @.offload_sizes.1 = private unnamed_addr constant [1 x i64] [i64 4]
+// CHECK-USE-PPC64LE: @.offload_maptypes.2 = private unnamed_addr constant [1 x i64] [i64 9255]
+// CHECK-USE-PPC64LE: @.offload_sizes.3 = private unnamed_addr constant [3 x i64] [i64 0, i64 4, i64 4]
+// CHECK-USE-PPC64LE: @.offload_maptypes.4 = private unnamed_addr constant [3 x i64] [i64 8224, i64 281474976718851, i64 281474976718851]
+//.
+// CHECK-USE-I386: @.offload_sizes = private unnamed_addr constant [7 x i64] [i64 0, i64 4, i64 4, i64 4, i64 0, i64 4, i64 4]
+// CHECK-USE-I386: @.offload_maptypes = private unnamed_addr constant [7 x i64] [i64 8224, i64 281474976718851, i64 281474976718851, i64 8227, i64 8224, i64 1407374883561475, i64 1407374883561475]
+// CHECK-USE-I386: @.offload_sizes.1 = private unnamed_addr constant [1 x i64] [i64 4]
+// CHECK-USE-I386: @.offload_maptypes.2 = private unnamed_addr constant [1 x i64] [i64 9255]
+// CHECK-USE-I386: @.offload_sizes.3 = private unnamed_addr constant [3 x i64] [i64 0, i64 4, i64 4]
+// CHECK-USE-I386: @.offload_maptypes.4 = private unnamed_addr constant [3 x i64] [i64 8224, i64 281474976718851, i64 281474976718851]
+//.
+// CHECK-NOUSE-PPC64LE: @.offload_sizes = private unnamed_addr constant [7 x i64] [i64 0, i64 4, i64 4, i64 4, i64 0, i64 4, i64 4]
+// CHECK-NOUSE-PPC64LE: @.offload_maptypes = private unnamed_addr constant [7 x i64] [i64 8192, i64 281474976718851, i64 281474976718851, i64 8195, i64 8192, i64 1407374883561475, i64 1407374883561475]
+// CHECK-NOUSE-PPC64LE: @.offload_sizes.1 = private unnamed_addr constant [1 x i64] [i64 4]
+// CHECK-NOUSE-PPC64LE: @.offload_maptypes.2 = private unnamed_addr constant [1 x i64] [i64 9223]
+// CHECK-NOUSE-PPC64LE: @.offload_sizes.3 = private unnamed_addr constant [3 x i64] [i64 0, i64 4, i64 4]
+// CHECK-NOUSE-PPC64LE: @.offload_maptypes.4 = private unnamed_addr constant [3 x i64] [i64 8192, i64 281474976718851, i64 281474976718851]
+//.
+// CHECK-NOUSE-I386: @.offload_sizes = private unnamed_addr constant [7 x i64] [i64 0, i64 4, i64 4, i64 4, i64 0, i64 4, i64 4]
+// CHECK-NOUSE-I386: @.offload_maptypes = private unnamed_addr constant [7 x i64] [i64 8192, i64 281474976718851, i64 281474976718851, i64 8195, i64 8192, i64 1407374883561475, i64 1407374883561475]
+// CHECK-NOUSE-I386: @.offload_sizes.1 = private unnamed_addr constant [1 x i64] [i64 4]
+// CHECK-NOUSE-I386: @.offload_maptypes.2 = private unnamed_addr constant [1 x i64] [i64 9223]
+// CHECK-NOUSE-I386: @.offload_sizes.3 = private unnamed_addr constant [3 x i64] [i64 0, i64 4, i64 4]
+// CHECK-NOUSE-I386: @.offload_maptypes.4 = private unnamed_addr constant [3 x i64] [i64 8192, i64 281474976718851, i64 281474976718851]
+//.
 // CHECK-USE-PPC64LE-LABEL: define {{[^@]+}}@_Z20explicit_maps_singlei
 // CHECK-USE-PPC64LE-SAME: (i32 noundef signext [[II:%.*]]) #[[ATTR0:[0-9]+]] {
 // CHECK-USE-PPC64LE-NEXT:  entry:
@@ -173,156 +172,156 @@ void ST::test_present_members() {
 // CHECK-USE-PPC64LE-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [7 x ptr], align 8
 // CHECK-USE-PPC64LE-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [7 x ptr], align 8
 // CHECK-USE-PPC64LE-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [7 x i64], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-USE-PPC64LE-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK-USE-PPC64LE-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK-USE-PPC64LE-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-USE-PPC64LE-NEXT:    store i32 [[II]], ptr [[II_ADDR]], align 4
 // CHECK-USE-PPC64LE-NEXT:    [[TMP0:%.*]] = load i32, ptr [[II_ADDR]], align 4
 // CHECK-USE-PPC64LE-NEXT:    store i32 [[TMP0]], ptr [[A]], align 4
 // CHECK-USE-PPC64LE-NEXT:    [[I:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[ST1]], i32 0, i32 0
 // CHECK-USE-PPC64LE-NEXT:    [[J:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[ST1]], i32 0, i32 1
 // CHECK-USE-PPC64LE-NEXT:    [[TMP1:%.*]] = getelementptr i32, ptr [[J]], i32 1
-// CHECK-USE-PPC64LE-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[TMP1]] to i64
-// CHECK-USE-PPC64LE-NEXT:    [[TMP5:%.*]] = ptrtoint ptr [[I]] to i64
-// CHECK-USE-PPC64LE-NEXT:    [[TMP6:%.*]] = sub i64 [[TMP4]], [[TMP5]]
-// CHECK-USE-PPC64LE-NEXT:    [[TMP7:%.*]] = sdiv exact i64 [[TMP6]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
+// CHECK-USE-PPC64LE-NEXT:    [[TMP2:%.*]] = ptrtoint ptr [[TMP1]] to i64
+// CHECK-USE-PPC64LE-NEXT:    [[TMP3:%.*]] = ptrtoint ptr [[I]] to i64
+// CHECK-USE-PPC64LE-NEXT:    [[TMP4:%.*]] = sub i64 [[TMP2]], [[TMP3]]
+// CHECK-USE-PPC64LE-NEXT:    [[TMP5:%.*]] = sdiv exact i64 [[TMP4]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
 // CHECK-USE-PPC64LE-NEXT:    [[I1:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[ST2]], i32 0, i32 0
 // CHECK-USE-PPC64LE-NEXT:    [[J2:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[ST2]], i32 0, i32 1
-// CHECK-USE-PPC64LE-NEXT:    [[TMP8:%.*]] = getelementptr i32, ptr [[J2]], i32 1
-// CHECK-USE-PPC64LE-NEXT:    [[TMP11:%.*]] = ptrtoint ptr [[TMP8]] to i64
-// CHECK-USE-PPC64LE-NEXT:    [[TMP12:%.*]] = ptrtoint ptr [[I1]] to i64
-// CHECK-USE-PPC64LE-NEXT:    [[TMP13:%.*]] = sub i64 [[TMP11]], [[TMP12]]
-// CHECK-USE-PPC64LE-NEXT:    [[TMP14:%.*]] = sdiv exact i64 [[TMP13]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
+// CHECK-USE-PPC64LE-NEXT:    [[TMP6:%.*]] = getelementptr i32, ptr [[J2]], i32 1
+// CHECK-USE-PPC64LE-NEXT:    [[TMP7:%.*]] = ptrtoint ptr [[TMP6]] to i64
+// CHECK-USE-PPC64LE-NEXT:    [[TMP8:%.*]] = ptrtoint ptr [[I1]] to i64
+// CHECK-USE-PPC64LE-NEXT:    [[TMP9:%.*]] = sub i64 [[TMP7]], [[TMP8]]
+// CHECK-USE-PPC64LE-NEXT:    [[TMP10:%.*]] = sdiv exact i64 [[TMP9]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
 // CHECK-USE-PPC64LE-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[DOTOFFLOAD_SIZES]], ptr align 8 @.offload_sizes, i64 56, i1 false)
-// CHECK-USE-PPC64LE-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[ST1]], ptr [[TMP16]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[I]], ptr [[TMP18]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK-USE-PPC64LE-NEXT:    store i64 [[TMP7]], ptr [[TMP20]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 0
-// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP21]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 1
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[ST1]], ptr [[TMP22]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 1
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[I]], ptr [[TMP24]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 1
-// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP26]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 2
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[ST1]], ptr [[TMP27]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 2
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[J]], ptr [[TMP29]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP31:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 2
-// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP31]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 3
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[A]], ptr [[TMP32]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 3
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[A]], ptr [[TMP34]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 3
-// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP36]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 4
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[ST2]], ptr [[TMP37]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 4
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[I1]], ptr [[TMP39]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 4
-// CHECK-USE-PPC64LE-NEXT:    store i64 [[TMP14]], ptr [[TMP41]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 4
-// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP42]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 5
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[ST2]], ptr [[TMP43]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 5
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[I1]], ptr [[TMP45]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP47:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 5
-// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP47]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 6
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[ST2]], ptr [[TMP48]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 6
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[J2]], ptr [[TMP50]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 6
-// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP52]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK-USE-PPC64LE-NEXT:    [[TMP54:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK-USE-PPC64LE-NEXT:    [[TMP55:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK-USE-PPC64LE-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP56:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
-// CHECK-USE-PPC64LE-NEXT:    store i32 2, ptr [[TMP56]], align 4
-// CHECK-USE-PPC64LE-NEXT:    [[TMP57:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
-// CHECK-USE-PPC64LE-NEXT:    store i32 7, ptr [[TMP57]], align 4
-// CHECK-USE-PPC64LE-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 2
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[TMP53]], ptr [[TMP58]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 3
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[TMP54]], ptr [[TMP59]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 4
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[TMP55]], ptr [[TMP60]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 5
-// CHECK-USE-PPC64LE-NEXT:    store ptr @.offload_maptypes, ptr [[TMP61]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 6
-// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP62]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 7
-// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP63]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 8
-// CHECK-USE-PPC64LE-NEXT:    store i64 0, ptr [[TMP64]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 9
-// CHECK-USE-PPC64LE-NEXT:    store i64 0, ptr [[TMP65]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 10
-// CHECK-USE-PPC64LE-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP66]], align 4
-// CHECK-USE-PPC64LE-NEXT:    [[TMP67:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 11
-// CHECK-USE-PPC64LE-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP67]], align 4
-// CHECK-USE-PPC64LE-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 12
-// CHECK-USE-PPC64LE-NEXT:    store i32 0, ptr [[TMP68]], align 4
-// CHECK-USE-PPC64LE-NEXT:    [[TMP69:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1:[0-9]+]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}.region_id, ptr [[KERNEL_ARGS]])
-// CHECK-USE-PPC64LE-NEXT:    [[TMP70:%.*]] = icmp ne i32 [[TMP69]], 0
-// CHECK-USE-PPC64LE-NEXT:    br i1 [[TMP70]], label [[OMP_OFFLOAD_FAILED:%.*]], label [[OMP_OFFLOAD_CONT:%.*]]
+// CHECK-USE-PPC64LE-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[ST1]], ptr [[TMP11]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[I]], ptr [[TMP12]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    store i64 [[TMP5]], ptr [[TMP13]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 0
+// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP14]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 1
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[ST1]], ptr [[TMP15]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 1
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[I]], ptr [[TMP16]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 1
+// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP17]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 2
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[ST1]], ptr [[TMP18]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 2
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[J]], ptr [[TMP19]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 2
+// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP20]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 3
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[A]], ptr [[TMP21]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 3
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[A]], ptr [[TMP22]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 3
+// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP23]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 4
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[ST2]], ptr [[TMP24]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 4
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[I1]], ptr [[TMP25]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 4
+// CHECK-USE-PPC64LE-NEXT:    store i64 [[TMP10]], ptr [[TMP26]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 4
+// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP27]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 5
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[ST2]], ptr [[TMP28]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 5
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[I1]], ptr [[TMP29]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 5
+// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP30]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP31:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 6
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[ST2]], ptr [[TMP31]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 6
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[J2]], ptr [[TMP32]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 6
+// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP33]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    store i32 2, ptr [[TMP37]], align 4
+// CHECK-USE-PPC64LE-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
+// CHECK-USE-PPC64LE-NEXT:    store i32 7, ptr [[TMP38]], align 4
+// CHECK-USE-PPC64LE-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 2
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[TMP34]], ptr [[TMP39]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP40:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 3
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[TMP35]], ptr [[TMP40]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 4
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[TMP36]], ptr [[TMP41]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 5
+// CHECK-USE-PPC64LE-NEXT:    store ptr @.offload_maptypes, ptr [[TMP42]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 6
+// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP43]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 7
+// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP44]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 8
+// CHECK-USE-PPC64LE-NEXT:    store i64 0, ptr [[TMP45]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 9
+// CHECK-USE-PPC64LE-NEXT:    store i64 0, ptr [[TMP46]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP47:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 10
+// CHECK-USE-PPC64LE-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP47]], align 4
+// CHECK-USE-PPC64LE-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 11
+// CHECK-USE-PPC64LE-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP48]], align 4
+// CHECK-USE-PPC64LE-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 12
+// CHECK-USE-PPC64LE-NEXT:    store i32 0, ptr [[TMP49]], align 4
+// CHECK-USE-PPC64LE-NEXT:    [[TMP50:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1:[0-9]+]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l100.region_id, ptr [[KERNEL_ARGS]])
+// CHECK-USE-PPC64LE-NEXT:    [[TMP51:%.*]] = icmp ne i32 [[TMP50]], 0
+// CHECK-USE-PPC64LE-NEXT:    br i1 [[TMP51]], label [[OMP_OFFLOAD_FAILED:%.*]], label [[OMP_OFFLOAD_CONT:%.*]]
 // CHECK-USE-PPC64LE:       omp_offload.failed:
-// CHECK-USE-PPC64LE-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}(ptr [[ST1]], ptr [[A]], ptr [[ST2]]) #[[ATTR3:[0-9]+]]
+// CHECK-USE-PPC64LE-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l100(ptr [[ST1]], ptr [[A]], ptr [[ST2]]) #[[ATTR3:[0-9]+]]
 // CHECK-USE-PPC64LE-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK-USE-PPC64LE:       omp_offload.cont:
-// CHECK-USE-PPC64LE-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[A]], ptr [[TMP71]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP73:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[A]], ptr [[TMP73]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP75:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_MAPPERS5]], i64 0, i64 0
-// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP75]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP76:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
-// CHECK-USE-PPC64LE-NEXT:    [[TMP77:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK-USE-PPC64LE-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP78:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
-// CHECK-USE-PPC64LE-NEXT:    store i32 2, ptr [[TMP78]], align 4
-// CHECK-USE-PPC64LE-NEXT:    [[TMP79:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
-// CHECK-USE-PPC64LE-NEXT:    store i32 1, ptr [[TMP79]], align 4
-// CHECK-USE-PPC64LE-NEXT:    [[TMP80:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 2
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[TMP76]], ptr [[TMP80]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP81:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 3
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[TMP77]], ptr [[TMP81]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP82:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 4
-// CHECK-USE-PPC64LE-NEXT:    store ptr @.offload_sizes.1, ptr [[TMP82]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 5
-// CHECK-USE-PPC64LE-NEXT:    store ptr @.offload_maptypes.2, ptr [[TMP83]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 6
-// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP84]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 7
-// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP85]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 8
-// CHECK-USE-PPC64LE-NEXT:    store i64 0, ptr [[TMP86]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP87:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 9
-// CHECK-USE-PPC64LE-NEXT:    store i64 0, ptr [[TMP87]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP88:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 10
-// CHECK-USE-PPC64LE-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP88]], align 4
-// CHECK-USE-PPC64LE-NEXT:    [[TMP89:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 11
-// CHECK-USE-PPC64LE-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP89]], align 4
-// CHECK-USE-PPC64LE-NEXT:    [[TMP90:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 12
-// CHECK-USE-PPC64LE-NEXT:    store i32 0, ptr [[TMP90]], align 4
-// CHECK-USE-PPC64LE-NEXT:    [[TMP91:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}.region_id, ptr [[KERNEL_ARGS6]])
-// CHECK-USE-PPC64LE-NEXT:    [[TMP92:%.*]] = icmp ne i32 [[TMP91]], 0
-// CHECK-USE-PPC64LE-NEXT:    br i1 [[TMP92]], label [[OMP_OFFLOAD_FAILED7:%.*]], label [[OMP_OFFLOAD_CONT8:%.*]]
+// CHECK-USE-PPC64LE-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[A]], ptr [[TMP52]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[A]], ptr [[TMP53]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP54:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_MAPPERS5]], i64 0, i64 0
+// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP54]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP55:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    [[TMP56:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    [[TMP57:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    store i32 2, ptr [[TMP57]], align 4
+// CHECK-USE-PPC64LE-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
+// CHECK-USE-PPC64LE-NEXT:    store i32 1, ptr [[TMP58]], align 4
+// CHECK-USE-PPC64LE-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 2
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[TMP55]], ptr [[TMP59]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 3
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[TMP56]], ptr [[TMP60]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 4
+// CHECK-USE-PPC64LE-NEXT:    store ptr @.offload_sizes.1, ptr [[TMP61]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 5
+// CHECK-USE-PPC64LE-NEXT:    store ptr @.offload_maptypes.2, ptr [[TMP62]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 6
+// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP63]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 7
+// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP64]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 8
+// CHECK-USE-PPC64LE-NEXT:    store i64 0, ptr [[TMP65]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 9
+// CHECK-USE-PPC64LE-NEXT:    store i64 0, ptr [[TMP66]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP67:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 10
+// CHECK-USE-PPC64LE-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP67]], align 4
+// CHECK-USE-PPC64LE-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 11
+// CHECK-USE-PPC64LE-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP68]], align 4
+// CHECK-USE-PPC64LE-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 12
+// CHECK-USE-PPC64LE-NEXT:    store i32 0, ptr [[TMP69]], align 4
+// CHECK-USE-PPC64LE-NEXT:    [[TMP70:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l114.region_id, ptr [[KERNEL_ARGS6]])
+// CHECK-USE-PPC64LE-NEXT:    [[TMP71:%.*]] = icmp ne i32 [[TMP70]], 0
+// CHECK-USE-PPC64LE-NEXT:    br i1 [[TMP71]], label [[OMP_OFFLOAD_FAILED7:%.*]], label [[OMP_OFFLOAD_CONT8:%.*]]
 // CHECK-USE-PPC64LE:       omp_offload.failed7:
-// CHECK-USE-PPC64LE-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}(ptr [[A]]) #[[ATTR3]]
+// CHECK-USE-PPC64LE-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l114(ptr [[A]]) #[[ATTR3]]
 // CHECK-USE-PPC64LE-NEXT:    br label [[OMP_OFFLOAD_CONT8]]
 // CHECK-USE-PPC64LE:       omp_offload.cont8:
 // CHECK-USE-PPC64LE-NEXT:    ret void
 //
 //
-// CHECK-USE-PPC64LE-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}
+// CHECK-USE-PPC64LE-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l100
 // CHECK-USE-PPC64LE-SAME: (ptr noundef nonnull align 4 dereferenceable(8) [[ST1:%.*]], ptr noundef nonnull align 4 dereferenceable(4) [[A:%.*]], ptr noundef nonnull align 4 dereferenceable(8) [[ST2:%.*]]) #[[ATTR1:[0-9]+]] {
 // CHECK-USE-PPC64LE-NEXT:  entry:
 // CHECK-USE-PPC64LE-NEXT:    [[ST1_ADDR:%.*]] = alloca ptr, align 8
@@ -356,7 +355,7 @@ void ST::test_present_members() {
 // CHECK-USE-PPC64LE-NEXT:    ret void
 //
 //
-// CHECK-USE-PPC64LE-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}
+// CHECK-USE-PPC64LE-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l114
 // CHECK-USE-PPC64LE-SAME: (ptr noundef nonnull align 4 dereferenceable(4) [[A:%.*]]) #[[ATTR1]] {
 // CHECK-USE-PPC64LE-NEXT:  entry:
 // CHECK-USE-PPC64LE-NEXT:    [[A_ADDR:%.*]] = alloca ptr, align 8
@@ -376,77 +375,77 @@ void ST::test_present_members() {
 // CHECK-USE-PPC64LE-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK-USE-PPC64LE-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
 // CHECK-USE-PPC64LE-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [3 x i64], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-USE-PPC64LE-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK-USE-PPC64LE-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK-USE-PPC64LE-NEXT:    [[I:%.*]] = getelementptr inbounds [[STRUCT_ST:%.*]], ptr [[THIS1]], i32 0, i32 0
 // CHECK-USE-PPC64LE-NEXT:    [[J:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[THIS1]], i32 0, i32 1
 // CHECK-USE-PPC64LE-NEXT:    [[TMP0:%.*]] = getelementptr i32, ptr [[J]], i32 1
-// CHECK-USE-PPC64LE-NEXT:    [[TMP3:%.*]] = ptrtoint ptr [[TMP0]] to i64
-// CHECK-USE-PPC64LE-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[I]] to i64
-// CHECK-USE-PPC64LE-NEXT:    [[TMP5:%.*]] = sub i64 [[TMP3]], [[TMP4]]
-// CHECK-USE-PPC64LE-NEXT:    [[TMP6:%.*]] = sdiv exact i64 [[TMP5]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
+// CHECK-USE-PPC64LE-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[TMP0]] to i64
+// CHECK-USE-PPC64LE-NEXT:    [[TMP2:%.*]] = ptrtoint ptr [[I]] to i64
+// CHECK-USE-PPC64LE-NEXT:    [[TMP3:%.*]] = sub i64 [[TMP1]], [[TMP2]]
+// CHECK-USE-PPC64LE-NEXT:    [[TMP4:%.*]] = sdiv exact i64 [[TMP3]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
 // CHECK-USE-PPC64LE-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[DOTOFFLOAD_SIZES]], ptr align 8 @.offload_sizes.3, i64 24, i1 false)
-// CHECK-USE-PPC64LE-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[THIS1]], ptr [[TMP8]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[THIS1]], ptr [[TMP5]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[I]], ptr [[TMP6]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    store i64 [[TMP4]], ptr [[TMP7]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 0
+// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP8]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 1
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[THIS1]], ptr [[TMP9]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 1
 // CHECK-USE-PPC64LE-NEXT:    store ptr [[I]], ptr [[TMP10]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK-USE-PPC64LE-NEXT:    store i64 [[TMP6]], ptr [[TMP12]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 0
-// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP13]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 1
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[THIS1]], ptr [[TMP14]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 1
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[I]], ptr [[TMP16]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 1
-// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP18]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 2
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[THIS1]], ptr [[TMP19]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 2
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[J]], ptr [[TMP21]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 2
-// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP23]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK-USE-PPC64LE-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK-USE-PPC64LE-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK-USE-PPC64LE-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
-// CHECK-USE-PPC64LE-NEXT:    store i32 2, ptr [[TMP27]], align 4
-// CHECK-USE-PPC64LE-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
-// CHECK-USE-PPC64LE-NEXT:    store i32 3, ptr [[TMP28]], align 4
-// CHECK-USE-PPC64LE-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 2
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[TMP24]], ptr [[TMP29]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 3
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[TMP25]], ptr [[TMP30]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP31:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 4
-// CHECK-USE-PPC64LE-NEXT:    store ptr [[TMP26]], ptr [[TMP31]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 5
-// CHECK-USE-PPC64LE-NEXT:    store ptr @.offload_maptypes.4, ptr [[TMP32]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 6
-// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP33]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 7
-// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP34]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 8
-// CHECK-USE-PPC64LE-NEXT:    store i64 0, ptr [[TMP35]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 9
-// CHECK-USE-PPC64LE-NEXT:    store i64 0, ptr [[TMP36]], align 8
-// CHECK-USE-PPC64LE-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 10
-// CHECK-USE-PPC64LE-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP37]], align 4
-// CHECK-USE-PPC64LE-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 11
-// CHECK-USE-PPC64LE-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP38]], align 4
-// CHECK-USE-PPC64LE-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 12
-// CHECK-USE-PPC64LE-NEXT:    store i32 0, ptr [[TMP39]], align 4
-// CHECK-USE-PPC64LE-NEXT:    [[TMP40:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l{{[0-9]*}}.region_id, ptr [[KERNEL_ARGS]])
-// CHECK-USE-PPC64LE-NEXT:    [[TMP41:%.*]] = icmp ne i32 [[TMP40]], 0
-// CHECK-USE-PPC64LE-NEXT:    br i1 [[TMP41]], label [[OMP_OFFLOAD_FAILED:%.*]], label [[OMP_OFFLOAD_CONT:%.*]]
+// CHECK-USE-PPC64LE-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 1
+// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP11]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 2
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[THIS1]], ptr [[TMP12]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 2
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[J]], ptr [[TMP13]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 2
+// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP14]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
+// CHECK-USE-PPC64LE-NEXT:    store i32 2, ptr [[TMP18]], align 4
+// CHECK-USE-PPC64LE-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
+// CHECK-USE-PPC64LE-NEXT:    store i32 3, ptr [[TMP19]], align 4
+// CHECK-USE-PPC64LE-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 2
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[TMP15]], ptr [[TMP20]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 3
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[TMP16]], ptr [[TMP21]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 4
+// CHECK-USE-PPC64LE-NEXT:    store ptr [[TMP17]], ptr [[TMP22]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 5
+// CHECK-USE-PPC64LE-NEXT:    store ptr @.offload_maptypes.4, ptr [[TMP23]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 6
+// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP24]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 7
+// CHECK-USE-PPC64LE-NEXT:    store ptr null, ptr [[TMP25]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 8
+// CHECK-USE-PPC64LE-NEXT:    store i64 0, ptr [[TMP26]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 9
+// CHECK-USE-PPC64LE-NEXT:    store i64 0, ptr [[TMP27]], align 8
+// CHECK-USE-PPC64LE-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 10
+// CHECK-USE-PPC64LE-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP28]], align 4
+// CHECK-USE-PPC64LE-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 11
+// CHECK-USE-PPC64LE-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP29]], align 4
+// CHECK-USE-PPC64LE-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 12
+// CHECK-USE-PPC64LE-NEXT:    store i32 0, ptr [[TMP30]], align 4
+// CHECK-USE-PPC64LE-NEXT:    [[TMP31:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l125.region_id, ptr [[KERNEL_ARGS]])
+// CHECK-USE-PPC64LE-NEXT:    [[TMP32:%.*]] = icmp ne i32 [[TMP31]], 0
+// CHECK-USE-PPC64LE-NEXT:    br i1 [[TMP32]], label [[OMP_OFFLOAD_FAILED:%.*]], label [[OMP_OFFLOAD_CONT:%.*]]
 // CHECK-USE-PPC64LE:       omp_offload.failed:
-// CHECK-USE-PPC64LE-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l{{[0-9]*}}(ptr [[THIS1]]) #[[ATTR3]]
+// CHECK-USE-PPC64LE-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l125(ptr [[THIS1]]) #[[ATTR3]]
 // CHECK-USE-PPC64LE-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK-USE-PPC64LE:       omp_offload.cont:
 // CHECK-USE-PPC64LE-NEXT:    ret void
 //
 //
-// CHECK-USE-PPC64LE-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l{{[0-9]*}}
+// CHECK-USE-PPC64LE-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l125
 // CHECK-USE-PPC64LE-SAME: (ptr noundef [[THIS:%.*]]) #[[ATTR1]] {
 // CHECK-USE-PPC64LE-NEXT:  entry:
 // CHECK-USE-PPC64LE-NEXT:    [[THIS_ADDR:%.*]] = alloca ptr, align 8
@@ -481,156 +480,156 @@ void ST::test_present_members() {
 // CHECK-USE-I386-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [7 x ptr], align 4
 // CHECK-USE-I386-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [7 x ptr], align 4
 // CHECK-USE-I386-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [7 x i64], align 4
+// CHECK-USE-I386-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-USE-I386-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK-USE-I386-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK-USE-I386-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
+// CHECK-USE-I386-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-USE-I386-NEXT:    store i32 [[II]], ptr [[II_ADDR]], align 4
 // CHECK-USE-I386-NEXT:    [[TMP0:%.*]] = load i32, ptr [[II_ADDR]], align 4
 // CHECK-USE-I386-NEXT:    store i32 [[TMP0]], ptr [[A]], align 4
 // CHECK-USE-I386-NEXT:    [[I:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[ST1]], i32 0, i32 0
 // CHECK-USE-I386-NEXT:    [[J:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[ST1]], i32 0, i32 1
 // CHECK-USE-I386-NEXT:    [[TMP1:%.*]] = getelementptr i32, ptr [[J]], i32 1
-// CHECK-USE-I386-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[TMP1]] to i64
-// CHECK-USE-I386-NEXT:    [[TMP5:%.*]] = ptrtoint ptr [[I]] to i64
-// CHECK-USE-I386-NEXT:    [[TMP6:%.*]] = sub i64 [[TMP4]], [[TMP5]]
-// CHECK-USE-I386-NEXT:    [[TMP7:%.*]] = sdiv exact i64 [[TMP6]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
+// CHECK-USE-I386-NEXT:    [[TMP2:%.*]] = ptrtoint ptr [[TMP1]] to i64
+// CHECK-USE-I386-NEXT:    [[TMP3:%.*]] = ptrtoint ptr [[I]] to i64
+// CHECK-USE-I386-NEXT:    [[TMP4:%.*]] = sub i64 [[TMP2]], [[TMP3]]
+// CHECK-USE-I386-NEXT:    [[TMP5:%.*]] = sdiv exact i64 [[TMP4]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
 // CHECK-USE-I386-NEXT:    [[I1:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[ST2]], i32 0, i32 0
 // CHECK-USE-I386-NEXT:    [[J2:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[ST2]], i32 0, i32 1
-// CHECK-USE-I386-NEXT:    [[TMP8:%.*]] = getelementptr i32, ptr [[J2]], i32 1
-// CHECK-USE-I386-NEXT:    [[TMP11:%.*]] = ptrtoint ptr [[TMP8]] to i64
-// CHECK-USE-I386-NEXT:    [[TMP12:%.*]] = ptrtoint ptr [[I1]] to i64
-// CHECK-USE-I386-NEXT:    [[TMP13:%.*]] = sub i64 [[TMP11]], [[TMP12]]
-// CHECK-USE-I386-NEXT:    [[TMP14:%.*]] = sdiv exact i64 [[TMP13]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
+// CHECK-USE-I386-NEXT:    [[TMP6:%.*]] = getelementptr i32, ptr [[J2]], i32 1
+// CHECK-USE-I386-NEXT:    [[TMP7:%.*]] = ptrtoint ptr [[TMP6]] to i64
+// CHECK-USE-I386-NEXT:    [[TMP8:%.*]] = ptrtoint ptr [[I1]] to i64
+// CHECK-USE-I386-NEXT:    [[TMP9:%.*]] = sub i64 [[TMP7]], [[TMP8]]
+// CHECK-USE-I386-NEXT:    [[TMP10:%.*]] = sdiv exact i64 [[TMP9]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
 // CHECK-USE-I386-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[DOTOFFLOAD_SIZES]], ptr align 4 @.offload_sizes, i32 56, i1 false)
-// CHECK-USE-I386-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    store ptr [[ST1]], ptr [[TMP16]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    store ptr [[I]], ptr [[TMP18]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    store i64 [[TMP7]], ptr [[TMP20]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP21]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 1
-// CHECK-USE-I386-NEXT:    store ptr [[ST1]], ptr [[TMP22]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 1
-// CHECK-USE-I386-NEXT:    store ptr [[I]], ptr [[TMP24]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 1
-// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP26]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 2
-// CHECK-USE-I386-NEXT:    store ptr [[ST1]], ptr [[TMP27]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 2
-// CHECK-USE-I386-NEXT:    store ptr [[J]], ptr [[TMP29]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP31:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 2
-// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP31]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 3
-// CHECK-USE-I386-NEXT:    store ptr [[A]], ptr [[TMP32]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 3
-// CHECK-USE-I386-NEXT:    store ptr [[A]], ptr [[TMP34]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 3
-// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP36]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 4
-// CHECK-USE-I386-NEXT:    store ptr [[ST2]], ptr [[TMP37]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 4
-// CHECK-USE-I386-NEXT:    store ptr [[I1]], ptr [[TMP39]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 4
-// CHECK-USE-I386-NEXT:    store i64 [[TMP14]], ptr [[TMP41]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 4
-// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP42]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 5
-// CHECK-USE-I386-NEXT:    store ptr [[ST2]], ptr [[TMP43]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 5
-// CHECK-USE-I386-NEXT:    store ptr [[I1]], ptr [[TMP45]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP47:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 5
-// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP47]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 6
-// CHECK-USE-I386-NEXT:    store ptr [[ST2]], ptr [[TMP48]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 6
-// CHECK-USE-I386-NEXT:    store ptr [[J2]], ptr [[TMP50]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 6
-// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP52]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    [[TMP54:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    [[TMP55:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
-// CHECK-USE-I386-NEXT:    [[TMP56:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    store i32 2, ptr [[TMP56]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP57:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
-// CHECK-USE-I386-NEXT:    store i32 7, ptr [[TMP57]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 2
-// CHECK-USE-I386-NEXT:    store ptr [[TMP53]], ptr [[TMP58]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 3
-// CHECK-USE-I386-NEXT:    store ptr [[TMP54]], ptr [[TMP59]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 4
-// CHECK-USE-I386-NEXT:    store ptr [[TMP55]], ptr [[TMP60]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 5
-// CHECK-USE-I386-NEXT:    store ptr @.offload_maptypes, ptr [[TMP61]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 6
-// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP62]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 7
-// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP63]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 8
-// CHECK-USE-I386-NEXT:    store i64 0, ptr [[TMP64]], align 8
-// CHECK-USE-I386-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 9
-// CHECK-USE-I386-NEXT:    store i64 0, ptr [[TMP65]], align 8
-// CHECK-USE-I386-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 10
-// CHECK-USE-I386-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP66]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP67:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 11
-// CHECK-USE-I386-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP67]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 12
-// CHECK-USE-I386-NEXT:    store i32 0, ptr [[TMP68]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP69:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1:[0-9]+]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}.region_id, ptr [[KERNEL_ARGS]])
-// CHECK-USE-I386-NEXT:    [[TMP70:%.*]] = icmp ne i32 [[TMP69]], 0
-// CHECK-USE-I386-NEXT:    br i1 [[TMP70]], label [[OMP_OFFLOAD_FAILED:%.*]], label [[OMP_OFFLOAD_CONT:%.*]]
+// CHECK-USE-I386-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    store ptr [[ST1]], ptr [[TMP11]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    store ptr [[I]], ptr [[TMP12]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    store i64 [[TMP5]], ptr [[TMP13]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP14]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 1
+// CHECK-USE-I386-NEXT:    store ptr [[ST1]], ptr [[TMP15]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 1
+// CHECK-USE-I386-NEXT:    store ptr [[I]], ptr [[TMP16]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 1
+// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP17]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 2
+// CHECK-USE-I386-NEXT:    store ptr [[ST1]], ptr [[TMP18]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 2
+// CHECK-USE-I386-NEXT:    store ptr [[J]], ptr [[TMP19]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 2
+// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP20]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 3
+// CHECK-USE-I386-NEXT:    store ptr [[A]], ptr [[TMP21]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 3
+// CHECK-USE-I386-NEXT:    store ptr [[A]], ptr [[TMP22]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 3
+// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP23]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 4
+// CHECK-USE-I386-NEXT:    store ptr [[ST2]], ptr [[TMP24]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 4
+// CHECK-USE-I386-NEXT:    store ptr [[I1]], ptr [[TMP25]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 4
+// CHECK-USE-I386-NEXT:    store i64 [[TMP10]], ptr [[TMP26]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 4
+// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP27]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 5
+// CHECK-USE-I386-NEXT:    store ptr [[ST2]], ptr [[TMP28]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 5
+// CHECK-USE-I386-NEXT:    store ptr [[I1]], ptr [[TMP29]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 5
+// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP30]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP31:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 6
+// CHECK-USE-I386-NEXT:    store ptr [[ST2]], ptr [[TMP31]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 6
+// CHECK-USE-I386-NEXT:    store ptr [[J2]], ptr [[TMP32]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 6
+// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP33]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    store i32 2, ptr [[TMP37]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
+// CHECK-USE-I386-NEXT:    store i32 7, ptr [[TMP38]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 2
+// CHECK-USE-I386-NEXT:    store ptr [[TMP34]], ptr [[TMP39]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP40:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 3
+// CHECK-USE-I386-NEXT:    store ptr [[TMP35]], ptr [[TMP40]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 4
+// CHECK-USE-I386-NEXT:    store ptr [[TMP36]], ptr [[TMP41]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 5
+// CHECK-USE-I386-NEXT:    store ptr @.offload_maptypes, ptr [[TMP42]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 6
+// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP43]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 7
+// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP44]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 8
+// CHECK-USE-I386-NEXT:    store i64 0, ptr [[TMP45]], align 8
+// CHECK-USE-I386-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 9
+// CHECK-USE-I386-NEXT:    store i64 0, ptr [[TMP46]], align 8
+// CHECK-USE-I386-NEXT:    [[TMP47:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 10
+// CHECK-USE-I386-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP47]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 11
+// CHECK-USE-I386-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP48]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 12
+// CHECK-USE-I386-NEXT:    store i32 0, ptr [[TMP49]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP50:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1:[0-9]+]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l100.region_id, ptr [[KERNEL_ARGS]])
+// CHECK-USE-I386-NEXT:    [[TMP51:%.*]] = icmp ne i32 [[TMP50]], 0
+// CHECK-USE-I386-NEXT:    br i1 [[TMP51]], label [[OMP_OFFLOAD_FAILED:%.*]], label [[OMP_OFFLOAD_CONT:%.*]]
 // CHECK-USE-I386:       omp_offload.failed:
-// CHECK-USE-I386-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}(ptr [[ST1]], ptr [[A]], ptr [[ST2]]) #[[ATTR3:[0-9]+]]
+// CHECK-USE-I386-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l100(ptr [[ST1]], ptr [[A]], ptr [[ST2]]) #[[ATTR3:[0-9]+]]
 // CHECK-USE-I386-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK-USE-I386:       omp_offload.cont:
-// CHECK-USE-I386-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    store ptr [[A]], ptr [[TMP71]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP73:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    store ptr [[A]], ptr [[TMP73]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP75:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_MAPPERS5]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP75]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP76:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    [[TMP77:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
-// CHECK-USE-I386-NEXT:    [[TMP78:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    store i32 2, ptr [[TMP78]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP79:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
-// CHECK-USE-I386-NEXT:    store i32 1, ptr [[TMP79]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP80:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 2
-// CHECK-USE-I386-NEXT:    store ptr [[TMP76]], ptr [[TMP80]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP81:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 3
-// CHECK-USE-I386-NEXT:    store ptr [[TMP77]], ptr [[TMP81]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP82:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 4
-// CHECK-USE-I386-NEXT:    store ptr @.offload_sizes.1, ptr [[TMP82]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 5
-// CHECK-USE-I386-NEXT:    store ptr @.offload_maptypes.2, ptr [[TMP83]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 6
-// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP84]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 7
-// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP85]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 8
-// CHECK-USE-I386-NEXT:    store i64 0, ptr [[TMP86]], align 8
-// CHECK-USE-I386-NEXT:    [[TMP87:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 9
-// CHECK-USE-I386-NEXT:    store i64 0, ptr [[TMP87]], align 8
-// CHECK-USE-I386-NEXT:    [[TMP88:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 10
-// CHECK-USE-I386-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP88]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP89:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 11
-// CHECK-USE-I386-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP89]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP90:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 12
-// CHECK-USE-I386-NEXT:    store i32 0, ptr [[TMP90]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP91:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}.region_id, ptr [[KERNEL_ARGS6]])
-// CHECK-USE-I386-NEXT:    [[TMP92:%.*]] = icmp ne i32 [[TMP91]], 0
-// CHECK-USE-I386-NEXT:    br i1 [[TMP92]], label [[OMP_OFFLOAD_FAILED7:%.*]], label [[OMP_OFFLOAD_CONT8:%.*]]
+// CHECK-USE-I386-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    store ptr [[A]], ptr [[TMP52]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    store ptr [[A]], ptr [[TMP53]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP54:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_MAPPERS5]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP54]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP55:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    [[TMP56:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    [[TMP57:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    store i32 2, ptr [[TMP57]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
+// CHECK-USE-I386-NEXT:    store i32 1, ptr [[TMP58]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 2
+// CHECK-USE-I386-NEXT:    store ptr [[TMP55]], ptr [[TMP59]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 3
+// CHECK-USE-I386-NEXT:    store ptr [[TMP56]], ptr [[TMP60]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 4
+// CHECK-USE-I386-NEXT:    store ptr @.offload_sizes.1, ptr [[TMP61]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 5
+// CHECK-USE-I386-NEXT:    store ptr @.offload_maptypes.2, ptr [[TMP62]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 6
+// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP63]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 7
+// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP64]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 8
+// CHECK-USE-I386-NEXT:    store i64 0, ptr [[TMP65]], align 8
+// CHECK-USE-I386-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 9
+// CHECK-USE-I386-NEXT:    store i64 0, ptr [[TMP66]], align 8
+// CHECK-USE-I386-NEXT:    [[TMP67:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 10
+// CHECK-USE-I386-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP67]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 11
+// CHECK-USE-I386-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP68]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 12
+// CHECK-USE-I386-NEXT:    store i32 0, ptr [[TMP69]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP70:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l114.region_id, ptr [[KERNEL_ARGS6]])
+// CHECK-USE-I386-NEXT:    [[TMP71:%.*]] = icmp ne i32 [[TMP70]], 0
+// CHECK-USE-I386-NEXT:    br i1 [[TMP71]], label [[OMP_OFFLOAD_FAILED7:%.*]], label [[OMP_OFFLOAD_CONT8:%.*]]
 // CHECK-USE-I386:       omp_offload.failed7:
-// CHECK-USE-I386-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}(ptr [[A]]) #[[ATTR3]]
+// CHECK-USE-I386-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l114(ptr [[A]]) #[[ATTR3]]
 // CHECK-USE-I386-NEXT:    br label [[OMP_OFFLOAD_CONT8]]
 // CHECK-USE-I386:       omp_offload.cont8:
 // CHECK-USE-I386-NEXT:    ret void
 //
 //
-// CHECK-USE-I386-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}
+// CHECK-USE-I386-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l100
 // CHECK-USE-I386-SAME: (ptr noundef nonnull align 4 dereferenceable(8) [[ST1:%.*]], ptr noundef nonnull align 4 dereferenceable(4) [[A:%.*]], ptr noundef nonnull align 4 dereferenceable(8) [[ST2:%.*]]) #[[ATTR1:[0-9]+]] {
 // CHECK-USE-I386-NEXT:  entry:
 // CHECK-USE-I386-NEXT:    [[ST1_ADDR:%.*]] = alloca ptr, align 4
@@ -664,7 +663,7 @@ void ST::test_present_members() {
 // CHECK-USE-I386-NEXT:    ret void
 //
 //
-// CHECK-USE-I386-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}
+// CHECK-USE-I386-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l114
 // CHECK-USE-I386-SAME: (ptr noundef nonnull align 4 dereferenceable(4) [[A:%.*]]) #[[ATTR1]] {
 // CHECK-USE-I386-NEXT:  entry:
 // CHECK-USE-I386-NEXT:    [[A_ADDR:%.*]] = alloca ptr, align 4
@@ -684,77 +683,77 @@ void ST::test_present_members() {
 // CHECK-USE-I386-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK-USE-I386-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
 // CHECK-USE-I386-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [3 x i64], align 4
+// CHECK-USE-I386-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-USE-I386-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK-USE-I386-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK-USE-I386-NEXT:    [[I:%.*]] = getelementptr inbounds [[STRUCT_ST:%.*]], ptr [[THIS1]], i32 0, i32 0
 // CHECK-USE-I386-NEXT:    [[J:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[THIS1]], i32 0, i32 1
 // CHECK-USE-I386-NEXT:    [[TMP0:%.*]] = getelementptr i32, ptr [[J]], i32 1
-// CHECK-USE-I386-NEXT:    [[TMP3:%.*]] = ptrtoint ptr [[TMP0]] to i64
-// CHECK-USE-I386-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[I]] to i64
-// CHECK-USE-I386-NEXT:    [[TMP5:%.*]] = sub i64 [[TMP3]], [[TMP4]]
-// CHECK-USE-I386-NEXT:    [[TMP6:%.*]] = sdiv exact i64 [[TMP5]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
+// CHECK-USE-I386-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[TMP0]] to i64
+// CHECK-USE-I386-NEXT:    [[TMP2:%.*]] = ptrtoint ptr [[I]] to i64
+// CHECK-USE-I386-NEXT:    [[TMP3:%.*]] = sub i64 [[TMP1]], [[TMP2]]
+// CHECK-USE-I386-NEXT:    [[TMP4:%.*]] = sdiv exact i64 [[TMP3]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
 // CHECK-USE-I386-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[DOTOFFLOAD_SIZES]], ptr align 4 @.offload_sizes.3, i32 24, i1 false)
-// CHECK-USE-I386-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    store ptr [[THIS1]], ptr [[TMP8]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    store ptr [[THIS1]], ptr [[TMP5]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    store ptr [[I]], ptr [[TMP6]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    store i64 [[TMP4]], ptr [[TMP7]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP8]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 1
+// CHECK-USE-I386-NEXT:    store ptr [[THIS1]], ptr [[TMP9]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 1
 // CHECK-USE-I386-NEXT:    store ptr [[I]], ptr [[TMP10]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    store i64 [[TMP6]], ptr [[TMP12]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP13]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 1
-// CHECK-USE-I386-NEXT:    store ptr [[THIS1]], ptr [[TMP14]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 1
-// CHECK-USE-I386-NEXT:    store ptr [[I]], ptr [[TMP16]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 1
-// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP18]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 2
-// CHECK-USE-I386-NEXT:    store ptr [[THIS1]], ptr [[TMP19]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 2
-// CHECK-USE-I386-NEXT:    store ptr [[J]], ptr [[TMP21]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 2
-// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP23]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
-// CHECK-USE-I386-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
-// CHECK-USE-I386-NEXT:    store i32 2, ptr [[TMP27]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
-// CHECK-USE-I386-NEXT:    store i32 3, ptr [[TMP28]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 2
-// CHECK-USE-I386-NEXT:    store ptr [[TMP24]], ptr [[TMP29]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 3
-// CHECK-USE-I386-NEXT:    store ptr [[TMP25]], ptr [[TMP30]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP31:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 4
-// CHECK-USE-I386-NEXT:    store ptr [[TMP26]], ptr [[TMP31]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 5
-// CHECK-USE-I386-NEXT:    store ptr @.offload_maptypes.4, ptr [[TMP32]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 6
-// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP33]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 7
-// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP34]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 8
-// CHECK-USE-I386-NEXT:    store i64 0, ptr [[TMP35]], align 8
-// CHECK-USE-I386-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 9
-// CHECK-USE-I386-NEXT:    store i64 0, ptr [[TMP36]], align 8
-// CHECK-USE-I386-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 10
-// CHECK-USE-I386-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP37]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 11
-// CHECK-USE-I386-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP38]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 12
-// CHECK-USE-I386-NEXT:    store i32 0, ptr [[TMP39]], align 4
-// CHECK-USE-I386-NEXT:    [[TMP40:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l{{[0-9]*}}.region_id, ptr [[KERNEL_ARGS]])
-// CHECK-USE-I386-NEXT:    [[TMP41:%.*]] = icmp ne i32 [[TMP40]], 0
-// CHECK-USE-I386-NEXT:    br i1 [[TMP41]], label [[OMP_OFFLOAD_FAILED:%.*]], label [[OMP_OFFLOAD_CONT:%.*]]
+// CHECK-USE-I386-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 1
+// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP11]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 2
+// CHECK-USE-I386-NEXT:    store ptr [[THIS1]], ptr [[TMP12]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 2
+// CHECK-USE-I386-NEXT:    store ptr [[J]], ptr [[TMP13]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 2
+// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP14]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
+// CHECK-USE-I386-NEXT:    store i32 2, ptr [[TMP18]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
+// CHECK-USE-I386-NEXT:    store i32 3, ptr [[TMP19]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 2
+// CHECK-USE-I386-NEXT:    store ptr [[TMP15]], ptr [[TMP20]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 3
+// CHECK-USE-I386-NEXT:    store ptr [[TMP16]], ptr [[TMP21]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 4
+// CHECK-USE-I386-NEXT:    store ptr [[TMP17]], ptr [[TMP22]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 5
+// CHECK-USE-I386-NEXT:    store ptr @.offload_maptypes.4, ptr [[TMP23]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 6
+// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP24]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 7
+// CHECK-USE-I386-NEXT:    store ptr null, ptr [[TMP25]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 8
+// CHECK-USE-I386-NEXT:    store i64 0, ptr [[TMP26]], align 8
+// CHECK-USE-I386-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 9
+// CHECK-USE-I386-NEXT:    store i64 0, ptr [[TMP27]], align 8
+// CHECK-USE-I386-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 10
+// CHECK-USE-I386-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP28]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 11
+// CHECK-USE-I386-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP29]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 12
+// CHECK-USE-I386-NEXT:    store i32 0, ptr [[TMP30]], align 4
+// CHECK-USE-I386-NEXT:    [[TMP31:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l125.region_id, ptr [[KERNEL_ARGS]])
+// CHECK-USE-I386-NEXT:    [[TMP32:%.*]] = icmp ne i32 [[TMP31]], 0
+// CHECK-USE-I386-NEXT:    br i1 [[TMP32]], label [[OMP_OFFLOAD_FAILED:%.*]], label [[OMP_OFFLOAD_CONT:%.*]]
 // CHECK-USE-I386:       omp_offload.failed:
-// CHECK-USE-I386-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l{{[0-9]*}}(ptr [[THIS1]]) #[[ATTR3]]
+// CHECK-USE-I386-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l125(ptr [[THIS1]]) #[[ATTR3]]
 // CHECK-USE-I386-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK-USE-I386:       omp_offload.cont:
 // CHECK-USE-I386-NEXT:    ret void
 //
 //
-// CHECK-USE-I386-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l{{[0-9]*}}
+// CHECK-USE-I386-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l125
 // CHECK-USE-I386-SAME: (ptr noundef [[THIS:%.*]]) #[[ATTR1]] {
 // CHECK-USE-I386-NEXT:  entry:
 // CHECK-USE-I386-NEXT:    [[THIS_ADDR:%.*]] = alloca ptr, align 4
@@ -789,162 +788,162 @@ void ST::test_present_members() {
 // CHECK-NOUSE-PPC64LE-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [7 x ptr], align 8
 // CHECK-NOUSE-PPC64LE-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [7 x ptr], align 8
 // CHECK-NOUSE-PPC64LE-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [7 x i64], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NOUSE-PPC64LE-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NOUSE-PPC64LE-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK-NOUSE-PPC64LE-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NOUSE-PPC64LE-NEXT:    store i32 [[II]], ptr [[II_ADDR]], align 4
 // CHECK-NOUSE-PPC64LE-NEXT:    [[TMP0:%.*]] = load i32, ptr [[II_ADDR]], align 4
 // CHECK-NOUSE-PPC64LE-NEXT:    store i32 [[TMP0]], ptr [[A]], align 4
 // CHECK-NOUSE-PPC64LE-NEXT:    [[I:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[ST1]], i32 0, i32 0
 // CHECK-NOUSE-PPC64LE-NEXT:    [[J:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[ST1]], i32 0, i32 1
 // CHECK-NOUSE-PPC64LE-NEXT:    [[TMP1:%.*]] = getelementptr i32, ptr [[J]], i32 1
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[TMP1]] to i64
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP5:%.*]] = ptrtoint ptr [[I]] to i64
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP6:%.*]] = sub i64 [[TMP4]], [[TMP5]]
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP7:%.*]] = sdiv exact i64 [[TMP6]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP2:%.*]] = ptrtoint ptr [[TMP1]] to i64
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP3:%.*]] = ptrtoint ptr [[I]] to i64
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP4:%.*]] = sub i64 [[TMP2]], [[TMP3]]
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP5:%.*]] = sdiv exact i64 [[TMP4]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
 // CHECK-NOUSE-PPC64LE-NEXT:    [[I1:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[ST2]], i32 0, i32 0
 // CHECK-NOUSE-PPC64LE-NEXT:    [[J2:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[ST2]], i32 0, i32 1
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP8:%.*]] = getelementptr i32, ptr [[J2]], i32 1
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP11:%.*]] = ptrtoint ptr [[TMP8]] to i64
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP12:%.*]] = ptrtoint ptr [[I1]] to i64
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP13:%.*]] = sub i64 [[TMP11]], [[TMP12]]
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP14:%.*]] = sdiv exact i64 [[TMP13]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP6:%.*]] = getelementptr i32, ptr [[J2]], i32 1
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP7:%.*]] = ptrtoint ptr [[TMP6]] to i64
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP8:%.*]] = ptrtoint ptr [[I1]] to i64
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP9:%.*]] = sub i64 [[TMP7]], [[TMP8]]
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP10:%.*]] = sdiv exact i64 [[TMP9]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
 // CHECK-NOUSE-PPC64LE-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[DOTOFFLOAD_SIZES]], ptr align 8 @.offload_sizes, i64 56, i1 false)
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[ST1]], ptr [[TMP16]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[I]], ptr [[TMP18]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK-NOUSE-PPC64LE-NEXT:    store i64 [[TMP7]], ptr [[TMP20]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 0
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP21]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 1
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[ST1]], ptr [[TMP22]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 1
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[I]], ptr [[TMP24]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 1
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP26]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 2
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[ST1]], ptr [[TMP27]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 2
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[J]], ptr [[TMP29]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP31:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 2
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP31]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 3
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[A]], ptr [[TMP32]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 3
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[A]], ptr [[TMP34]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 3
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP36]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 4
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[ST2]], ptr [[TMP37]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 4
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[I1]], ptr [[TMP39]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 4
-// CHECK-NOUSE-PPC64LE-NEXT:    store i64 [[TMP14]], ptr [[TMP41]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 4
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP42]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 5
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[ST2]], ptr [[TMP43]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 5
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[I1]], ptr [[TMP45]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP47:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 5
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP47]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 6
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[ST2]], ptr [[TMP48]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 6
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[J2]], ptr [[TMP50]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 6
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP52]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP54:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP55:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK-NOUSE-PPC64LE-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP56:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
-// CHECK-NOUSE-PPC64LE-NEXT:    store i32 2, ptr [[TMP56]], align 4
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP57:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
-// CHECK-NOUSE-PPC64LE-NEXT:    store i32 7, ptr [[TMP57]], align 4
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 2
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[TMP53]], ptr [[TMP58]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 3
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[TMP54]], ptr [[TMP59]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 4
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[TMP55]], ptr [[TMP60]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 5
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr @.offload_maptypes, ptr [[TMP61]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 6
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP62]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 7
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP63]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 8
-// CHECK-NOUSE-PPC64LE-NEXT:    store i64 0, ptr [[TMP64]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 9
-// CHECK-NOUSE-PPC64LE-NEXT:    store i64 0, ptr [[TMP65]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 10
-// CHECK-NOUSE-PPC64LE-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP66]], align 4
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP67:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 11
-// CHECK-NOUSE-PPC64LE-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP67]], align 4
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 12
-// CHECK-NOUSE-PPC64LE-NEXT:    store i32 0, ptr [[TMP68]], align 4
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP69:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1:[0-9]+]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}.region_id, ptr [[KERNEL_ARGS]])
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP70:%.*]] = icmp ne i32 [[TMP69]], 0
-// CHECK-NOUSE-PPC64LE-NEXT:    br i1 [[TMP70]], label [[OMP_OFFLOAD_FAILED:%.*]], label [[OMP_OFFLOAD_CONT:%.*]]
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[ST1]], ptr [[TMP11]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[I]], ptr [[TMP12]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    store i64 [[TMP5]], ptr [[TMP13]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 0
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP14]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 1
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[ST1]], ptr [[TMP15]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 1
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[I]], ptr [[TMP16]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 1
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP17]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 2
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[ST1]], ptr [[TMP18]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 2
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[J]], ptr [[TMP19]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 2
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP20]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 3
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[A]], ptr [[TMP21]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 3
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[A]], ptr [[TMP22]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 3
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP23]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 4
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[ST2]], ptr [[TMP24]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 4
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[I1]], ptr [[TMP25]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 4
+// CHECK-NOUSE-PPC64LE-NEXT:    store i64 [[TMP10]], ptr [[TMP26]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 4
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP27]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 5
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[ST2]], ptr [[TMP28]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 5
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[I1]], ptr [[TMP29]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 5
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP30]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP31:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 6
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[ST2]], ptr [[TMP31]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 6
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[J2]], ptr [[TMP32]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 6
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP33]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    store i32 2, ptr [[TMP37]], align 4
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
+// CHECK-NOUSE-PPC64LE-NEXT:    store i32 7, ptr [[TMP38]], align 4
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 2
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[TMP34]], ptr [[TMP39]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP40:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 3
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[TMP35]], ptr [[TMP40]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 4
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[TMP36]], ptr [[TMP41]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 5
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr @.offload_maptypes, ptr [[TMP42]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 6
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP43]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 7
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP44]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 8
+// CHECK-NOUSE-PPC64LE-NEXT:    store i64 0, ptr [[TMP45]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 9
+// CHECK-NOUSE-PPC64LE-NEXT:    store i64 0, ptr [[TMP46]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP47:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 10
+// CHECK-NOUSE-PPC64LE-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP47]], align 4
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 11
+// CHECK-NOUSE-PPC64LE-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP48]], align 4
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 12
+// CHECK-NOUSE-PPC64LE-NEXT:    store i32 0, ptr [[TMP49]], align 4
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP50:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1:[0-9]+]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l100.region_id, ptr [[KERNEL_ARGS]])
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP51:%.*]] = icmp ne i32 [[TMP50]], 0
+// CHECK-NOUSE-PPC64LE-NEXT:    br i1 [[TMP51]], label [[OMP_OFFLOAD_FAILED:%.*]], label [[OMP_OFFLOAD_CONT:%.*]]
 // CHECK-NOUSE-PPC64LE:       omp_offload.failed:
-// CHECK-NOUSE-PPC64LE-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}() #[[ATTR3:[0-9]+]]
+// CHECK-NOUSE-PPC64LE-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l100() #[[ATTR3:[0-9]+]]
 // CHECK-NOUSE-PPC64LE-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK-NOUSE-PPC64LE:       omp_offload.cont:
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[A]], ptr [[TMP71]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP73:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[A]], ptr [[TMP73]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP75:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_MAPPERS5]], i64 0, i64 0
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP75]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP76:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP77:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK-NOUSE-PPC64LE-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP78:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
-// CHECK-NOUSE-PPC64LE-NEXT:    store i32 2, ptr [[TMP78]], align 4
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP79:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
-// CHECK-NOUSE-PPC64LE-NEXT:    store i32 1, ptr [[TMP79]], align 4
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP80:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 2
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[TMP76]], ptr [[TMP80]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP81:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 3
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[TMP77]], ptr [[TMP81]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP82:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 4
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr @.offload_sizes.1, ptr [[TMP82]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 5
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr @.offload_maptypes.2, ptr [[TMP83]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 6
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP84]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 7
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP85]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 8
-// CHECK-NOUSE-PPC64LE-NEXT:    store i64 0, ptr [[TMP86]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP87:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 9
-// CHECK-NOUSE-PPC64LE-NEXT:    store i64 0, ptr [[TMP87]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP88:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 10
-// CHECK-NOUSE-PPC64LE-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP88]], align 4
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP89:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 11
-// CHECK-NOUSE-PPC64LE-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP89]], align 4
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP90:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 12
-// CHECK-NOUSE-PPC64LE-NEXT:    store i32 0, ptr [[TMP90]], align 4
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP91:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}.region_id, ptr [[KERNEL_ARGS6]])
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP92:%.*]] = icmp ne i32 [[TMP91]], 0
-// CHECK-NOUSE-PPC64LE-NEXT:    br i1 [[TMP92]], label [[OMP_OFFLOAD_FAILED7:%.*]], label [[OMP_OFFLOAD_CONT8:%.*]]
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[A]], ptr [[TMP52]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[A]], ptr [[TMP53]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP54:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_MAPPERS5]], i64 0, i64 0
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP54]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP55:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP56:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP57:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    store i32 2, ptr [[TMP57]], align 4
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
+// CHECK-NOUSE-PPC64LE-NEXT:    store i32 1, ptr [[TMP58]], align 4
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 2
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[TMP55]], ptr [[TMP59]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 3
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[TMP56]], ptr [[TMP60]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 4
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr @.offload_sizes.1, ptr [[TMP61]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 5
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr @.offload_maptypes.2, ptr [[TMP62]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 6
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP63]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 7
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP64]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 8
+// CHECK-NOUSE-PPC64LE-NEXT:    store i64 0, ptr [[TMP65]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 9
+// CHECK-NOUSE-PPC64LE-NEXT:    store i64 0, ptr [[TMP66]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP67:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 10
+// CHECK-NOUSE-PPC64LE-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP67]], align 4
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 11
+// CHECK-NOUSE-PPC64LE-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP68]], align 4
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 12
+// CHECK-NOUSE-PPC64LE-NEXT:    store i32 0, ptr [[TMP69]], align 4
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP70:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l114.region_id, ptr [[KERNEL_ARGS6]])
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP71:%.*]] = icmp ne i32 [[TMP70]], 0
+// CHECK-NOUSE-PPC64LE-NEXT:    br i1 [[TMP71]], label [[OMP_OFFLOAD_FAILED7:%.*]], label [[OMP_OFFLOAD_CONT8:%.*]]
 // CHECK-NOUSE-PPC64LE:       omp_offload.failed7:
-// CHECK-NOUSE-PPC64LE-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}() #[[ATTR3]]
+// CHECK-NOUSE-PPC64LE-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l114() #[[ATTR3]]
 // CHECK-NOUSE-PPC64LE-NEXT:    br label [[OMP_OFFLOAD_CONT8]]
 // CHECK-NOUSE-PPC64LE:       omp_offload.cont8:
 // CHECK-NOUSE-PPC64LE-NEXT:    ret void
 //
 //
-// CHECK-NOUSE-PPC64LE-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}
+// CHECK-NOUSE-PPC64LE-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l100
 // CHECK-NOUSE-PPC64LE-SAME: () #[[ATTR1:[0-9]+]] {
 // CHECK-NOUSE-PPC64LE-NEXT:  entry:
 // CHECK-NOUSE-PPC64LE-NEXT:    ret void
 //
 //
-// CHECK-NOUSE-PPC64LE-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}
+// CHECK-NOUSE-PPC64LE-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l114
 // CHECK-NOUSE-PPC64LE-SAME: () #[[ATTR1]] {
 // CHECK-NOUSE-PPC64LE-NEXT:  entry:
 // CHECK-NOUSE-PPC64LE-NEXT:    ret void
@@ -958,77 +957,77 @@ void ST::test_present_members() {
 // CHECK-NOUSE-PPC64LE-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK-NOUSE-PPC64LE-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
 // CHECK-NOUSE-PPC64LE-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [3 x i64], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK-NOUSE-PPC64LE-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK-NOUSE-PPC64LE-NEXT:    [[I:%.*]] = getelementptr inbounds [[STRUCT_ST:%.*]], ptr [[THIS1]], i32 0, i32 0
 // CHECK-NOUSE-PPC64LE-NEXT:    [[J:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[THIS1]], i32 0, i32 1
 // CHECK-NOUSE-PPC64LE-NEXT:    [[TMP0:%.*]] = getelementptr i32, ptr [[J]], i32 1
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP3:%.*]] = ptrtoint ptr [[TMP0]] to i64
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[I]] to i64
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP5:%.*]] = sub i64 [[TMP3]], [[TMP4]]
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP6:%.*]] = sdiv exact i64 [[TMP5]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[TMP0]] to i64
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP2:%.*]] = ptrtoint ptr [[I]] to i64
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP3:%.*]] = sub i64 [[TMP1]], [[TMP2]]
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP4:%.*]] = sdiv exact i64 [[TMP3]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
 // CHECK-NOUSE-PPC64LE-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[DOTOFFLOAD_SIZES]], ptr align 8 @.offload_sizes.3, i64 24, i1 false)
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[THIS1]], ptr [[TMP8]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[THIS1]], ptr [[TMP5]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[I]], ptr [[TMP6]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    store i64 [[TMP4]], ptr [[TMP7]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 0
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP8]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 1
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[THIS1]], ptr [[TMP9]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 1
 // CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[I]], ptr [[TMP10]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK-NOUSE-PPC64LE-NEXT:    store i64 [[TMP6]], ptr [[TMP12]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 0
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP13]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 1
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[THIS1]], ptr [[TMP14]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 1
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[I]], ptr [[TMP16]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 1
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP18]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 2
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[THIS1]], ptr [[TMP19]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 2
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[J]], ptr [[TMP21]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 2
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP23]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK-NOUSE-PPC64LE-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
-// CHECK-NOUSE-PPC64LE-NEXT:    store i32 2, ptr [[TMP27]], align 4
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
-// CHECK-NOUSE-PPC64LE-NEXT:    store i32 3, ptr [[TMP28]], align 4
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 2
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[TMP24]], ptr [[TMP29]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 3
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[TMP25]], ptr [[TMP30]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP31:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 4
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[TMP26]], ptr [[TMP31]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 5
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr @.offload_maptypes.4, ptr [[TMP32]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 6
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP33]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 7
-// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP34]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 8
-// CHECK-NOUSE-PPC64LE-NEXT:    store i64 0, ptr [[TMP35]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 9
-// CHECK-NOUSE-PPC64LE-NEXT:    store i64 0, ptr [[TMP36]], align 8
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 10
-// CHECK-NOUSE-PPC64LE-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP37]], align 4
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 11
-// CHECK-NOUSE-PPC64LE-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP38]], align 4
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 12
-// CHECK-NOUSE-PPC64LE-NEXT:    store i32 0, ptr [[TMP39]], align 4
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP40:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l{{[0-9]*}}.region_id, ptr [[KERNEL_ARGS]])
-// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP41:%.*]] = icmp ne i32 [[TMP40]], 0
-// CHECK-NOUSE-PPC64LE-NEXT:    br i1 [[TMP41]], label [[OMP_OFFLOAD_FAILED:%.*]], label [[OMP_OFFLOAD_CONT:%.*]]
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 1
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP11]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 2
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[THIS1]], ptr [[TMP12]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 2
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[J]], ptr [[TMP13]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 2
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP14]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
+// CHECK-NOUSE-PPC64LE-NEXT:    store i32 2, ptr [[TMP18]], align 4
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
+// CHECK-NOUSE-PPC64LE-NEXT:    store i32 3, ptr [[TMP19]], align 4
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 2
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[TMP15]], ptr [[TMP20]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 3
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[TMP16]], ptr [[TMP21]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 4
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr [[TMP17]], ptr [[TMP22]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 5
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr @.offload_maptypes.4, ptr [[TMP23]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 6
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP24]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 7
+// CHECK-NOUSE-PPC64LE-NEXT:    store ptr null, ptr [[TMP25]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 8
+// CHECK-NOUSE-PPC64LE-NEXT:    store i64 0, ptr [[TMP26]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 9
+// CHECK-NOUSE-PPC64LE-NEXT:    store i64 0, ptr [[TMP27]], align 8
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 10
+// CHECK-NOUSE-PPC64LE-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP28]], align 4
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 11
+// CHECK-NOUSE-PPC64LE-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP29]], align 4
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 12
+// CHECK-NOUSE-PPC64LE-NEXT:    store i32 0, ptr [[TMP30]], align 4
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP31:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l125.region_id, ptr [[KERNEL_ARGS]])
+// CHECK-NOUSE-PPC64LE-NEXT:    [[TMP32:%.*]] = icmp ne i32 [[TMP31]], 0
+// CHECK-NOUSE-PPC64LE-NEXT:    br i1 [[TMP32]], label [[OMP_OFFLOAD_FAILED:%.*]], label [[OMP_OFFLOAD_CONT:%.*]]
 // CHECK-NOUSE-PPC64LE:       omp_offload.failed:
-// CHECK-NOUSE-PPC64LE-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l{{[0-9]*}}() #[[ATTR3]]
+// CHECK-NOUSE-PPC64LE-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l125() #[[ATTR3]]
 // CHECK-NOUSE-PPC64LE-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK-NOUSE-PPC64LE:       omp_offload.cont:
 // CHECK-NOUSE-PPC64LE-NEXT:    ret void
 //
 //
-// CHECK-NOUSE-PPC64LE-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l{{[0-9]*}}
+// CHECK-NOUSE-PPC64LE-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l125
 // CHECK-NOUSE-PPC64LE-SAME: () #[[ATTR1]] {
 // CHECK-NOUSE-PPC64LE-NEXT:  entry:
 // CHECK-NOUSE-PPC64LE-NEXT:    ret void
@@ -1052,162 +1051,162 @@ void ST::test_present_members() {
 // CHECK-NOUSE-I386-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [7 x ptr], align 4
 // CHECK-NOUSE-I386-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [7 x ptr], align 4
 // CHECK-NOUSE-I386-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [7 x i64], align 4
+// CHECK-NOUSE-I386-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NOUSE-I386-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK-NOUSE-I386-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK-NOUSE-I386-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
+// CHECK-NOUSE-I386-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NOUSE-I386-NEXT:    store i32 [[II]], ptr [[II_ADDR]], align 4
 // CHECK-NOUSE-I386-NEXT:    [[TMP0:%.*]] = load i32, ptr [[II_ADDR]], align 4
 // CHECK-NOUSE-I386-NEXT:    store i32 [[TMP0]], ptr [[A]], align 4
 // CHECK-NOUSE-I386-NEXT:    [[I:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[ST1]], i32 0, i32 0
 // CHECK-NOUSE-I386-NEXT:    [[J:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[ST1]], i32 0, i32 1
 // CHECK-NOUSE-I386-NEXT:    [[TMP1:%.*]] = getelementptr i32, ptr [[J]], i32 1
-// CHECK-NOUSE-I386-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[TMP1]] to i64
-// CHECK-NOUSE-I386-NEXT:    [[TMP5:%.*]] = ptrtoint ptr [[I]] to i64
-// CHECK-NOUSE-I386-NEXT:    [[TMP6:%.*]] = sub i64 [[TMP4]], [[TMP5]]
-// CHECK-NOUSE-I386-NEXT:    [[TMP7:%.*]] = sdiv exact i64 [[TMP6]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
+// CHECK-NOUSE-I386-NEXT:    [[TMP2:%.*]] = ptrtoint ptr [[TMP1]] to i64
+// CHECK-NOUSE-I386-NEXT:    [[TMP3:%.*]] = ptrtoint ptr [[I]] to i64
+// CHECK-NOUSE-I386-NEXT:    [[TMP4:%.*]] = sub i64 [[TMP2]], [[TMP3]]
+// CHECK-NOUSE-I386-NEXT:    [[TMP5:%.*]] = sdiv exact i64 [[TMP4]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
 // CHECK-NOUSE-I386-NEXT:    [[I1:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[ST2]], i32 0, i32 0
 // CHECK-NOUSE-I386-NEXT:    [[J2:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[ST2]], i32 0, i32 1
-// CHECK-NOUSE-I386-NEXT:    [[TMP8:%.*]] = getelementptr i32, ptr [[J2]], i32 1
-// CHECK-NOUSE-I386-NEXT:    [[TMP11:%.*]] = ptrtoint ptr [[TMP8]] to i64
-// CHECK-NOUSE-I386-NEXT:    [[TMP12:%.*]] = ptrtoint ptr [[I1]] to i64
-// CHECK-NOUSE-I386-NEXT:    [[TMP13:%.*]] = sub i64 [[TMP11]], [[TMP12]]
-// CHECK-NOUSE-I386-NEXT:    [[TMP14:%.*]] = sdiv exact i64 [[TMP13]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
+// CHECK-NOUSE-I386-NEXT:    [[TMP6:%.*]] = getelementptr i32, ptr [[J2]], i32 1
+// CHECK-NOUSE-I386-NEXT:    [[TMP7:%.*]] = ptrtoint ptr [[TMP6]] to i64
+// CHECK-NOUSE-I386-NEXT:    [[TMP8:%.*]] = ptrtoint ptr [[I1]] to i64
+// CHECK-NOUSE-I386-NEXT:    [[TMP9:%.*]] = sub i64 [[TMP7]], [[TMP8]]
+// CHECK-NOUSE-I386-NEXT:    [[TMP10:%.*]] = sdiv exact i64 [[TMP9]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
 // CHECK-NOUSE-I386-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[DOTOFFLOAD_SIZES]], ptr align 4 @.offload_sizes, i32 56, i1 false)
-// CHECK-NOUSE-I386-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    store ptr [[ST1]], ptr [[TMP16]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    store ptr [[I]], ptr [[TMP18]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    store i64 [[TMP7]], ptr [[TMP20]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP21]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 1
-// CHECK-NOUSE-I386-NEXT:    store ptr [[ST1]], ptr [[TMP22]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 1
-// CHECK-NOUSE-I386-NEXT:    store ptr [[I]], ptr [[TMP24]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 1
-// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP26]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 2
-// CHECK-NOUSE-I386-NEXT:    store ptr [[ST1]], ptr [[TMP27]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 2
-// CHECK-NOUSE-I386-NEXT:    store ptr [[J]], ptr [[TMP29]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP31:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 2
-// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP31]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 3
-// CHECK-NOUSE-I386-NEXT:    store ptr [[A]], ptr [[TMP32]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 3
-// CHECK-NOUSE-I386-NEXT:    store ptr [[A]], ptr [[TMP34]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 3
-// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP36]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 4
-// CHECK-NOUSE-I386-NEXT:    store ptr [[ST2]], ptr [[TMP37]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 4
-// CHECK-NOUSE-I386-NEXT:    store ptr [[I1]], ptr [[TMP39]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 4
-// CHECK-NOUSE-I386-NEXT:    store i64 [[TMP14]], ptr [[TMP41]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 4
-// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP42]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 5
-// CHECK-NOUSE-I386-NEXT:    store ptr [[ST2]], ptr [[TMP43]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 5
-// CHECK-NOUSE-I386-NEXT:    store ptr [[I1]], ptr [[TMP45]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP47:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 5
-// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP47]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 6
-// CHECK-NOUSE-I386-NEXT:    store ptr [[ST2]], ptr [[TMP48]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 6
-// CHECK-NOUSE-I386-NEXT:    store ptr [[J2]], ptr [[TMP50]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 6
-// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP52]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    [[TMP54:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    [[TMP55:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
-// CHECK-NOUSE-I386-NEXT:    [[TMP56:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    store i32 2, ptr [[TMP56]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP57:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
-// CHECK-NOUSE-I386-NEXT:    store i32 7, ptr [[TMP57]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 2
-// CHECK-NOUSE-I386-NEXT:    store ptr [[TMP53]], ptr [[TMP58]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 3
-// CHECK-NOUSE-I386-NEXT:    store ptr [[TMP54]], ptr [[TMP59]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 4
-// CHECK-NOUSE-I386-NEXT:    store ptr [[TMP55]], ptr [[TMP60]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 5
-// CHECK-NOUSE-I386-NEXT:    store ptr @.offload_maptypes, ptr [[TMP61]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 6
-// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP62]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 7
-// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP63]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 8
-// CHECK-NOUSE-I386-NEXT:    store i64 0, ptr [[TMP64]], align 8
-// CHECK-NOUSE-I386-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 9
-// CHECK-NOUSE-I386-NEXT:    store i64 0, ptr [[TMP65]], align 8
-// CHECK-NOUSE-I386-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 10
-// CHECK-NOUSE-I386-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP66]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP67:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 11
-// CHECK-NOUSE-I386-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP67]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 12
-// CHECK-NOUSE-I386-NEXT:    store i32 0, ptr [[TMP68]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP69:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1:[0-9]+]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}.region_id, ptr [[KERNEL_ARGS]])
-// CHECK-NOUSE-I386-NEXT:    [[TMP70:%.*]] = icmp ne i32 [[TMP69]], 0
-// CHECK-NOUSE-I386-NEXT:    br i1 [[TMP70]], label [[OMP_OFFLOAD_FAILED:%.*]], label [[OMP_OFFLOAD_CONT:%.*]]
+// CHECK-NOUSE-I386-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    store ptr [[ST1]], ptr [[TMP11]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    store ptr [[I]], ptr [[TMP12]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    store i64 [[TMP5]], ptr [[TMP13]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP14]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 1
+// CHECK-NOUSE-I386-NEXT:    store ptr [[ST1]], ptr [[TMP15]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 1
+// CHECK-NOUSE-I386-NEXT:    store ptr [[I]], ptr [[TMP16]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 1
+// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP17]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 2
+// CHECK-NOUSE-I386-NEXT:    store ptr [[ST1]], ptr [[TMP18]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 2
+// CHECK-NOUSE-I386-NEXT:    store ptr [[J]], ptr [[TMP19]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 2
+// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP20]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 3
+// CHECK-NOUSE-I386-NEXT:    store ptr [[A]], ptr [[TMP21]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 3
+// CHECK-NOUSE-I386-NEXT:    store ptr [[A]], ptr [[TMP22]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 3
+// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP23]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 4
+// CHECK-NOUSE-I386-NEXT:    store ptr [[ST2]], ptr [[TMP24]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 4
+// CHECK-NOUSE-I386-NEXT:    store ptr [[I1]], ptr [[TMP25]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 4
+// CHECK-NOUSE-I386-NEXT:    store i64 [[TMP10]], ptr [[TMP26]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 4
+// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP27]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 5
+// CHECK-NOUSE-I386-NEXT:    store ptr [[ST2]], ptr [[TMP28]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 5
+// CHECK-NOUSE-I386-NEXT:    store ptr [[I1]], ptr [[TMP29]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 5
+// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP30]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP31:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 6
+// CHECK-NOUSE-I386-NEXT:    store ptr [[ST2]], ptr [[TMP31]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 6
+// CHECK-NOUSE-I386-NEXT:    store ptr [[J2]], ptr [[TMP32]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 6
+// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP33]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [7 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [7 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    store i32 2, ptr [[TMP37]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
+// CHECK-NOUSE-I386-NEXT:    store i32 7, ptr [[TMP38]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 2
+// CHECK-NOUSE-I386-NEXT:    store ptr [[TMP34]], ptr [[TMP39]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP40:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 3
+// CHECK-NOUSE-I386-NEXT:    store ptr [[TMP35]], ptr [[TMP40]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 4
+// CHECK-NOUSE-I386-NEXT:    store ptr [[TMP36]], ptr [[TMP41]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 5
+// CHECK-NOUSE-I386-NEXT:    store ptr @.offload_maptypes, ptr [[TMP42]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 6
+// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP43]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 7
+// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP44]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 8
+// CHECK-NOUSE-I386-NEXT:    store i64 0, ptr [[TMP45]], align 8
+// CHECK-NOUSE-I386-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 9
+// CHECK-NOUSE-I386-NEXT:    store i64 0, ptr [[TMP46]], align 8
+// CHECK-NOUSE-I386-NEXT:    [[TMP47:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 10
+// CHECK-NOUSE-I386-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP47]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 11
+// CHECK-NOUSE-I386-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP48]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 12
+// CHECK-NOUSE-I386-NEXT:    store i32 0, ptr [[TMP49]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP50:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1:[0-9]+]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l100.region_id, ptr [[KERNEL_ARGS]])
+// CHECK-NOUSE-I386-NEXT:    [[TMP51:%.*]] = icmp ne i32 [[TMP50]], 0
+// CHECK-NOUSE-I386-NEXT:    br i1 [[TMP51]], label [[OMP_OFFLOAD_FAILED:%.*]], label [[OMP_OFFLOAD_CONT:%.*]]
 // CHECK-NOUSE-I386:       omp_offload.failed:
-// CHECK-NOUSE-I386-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}() #[[ATTR3:[0-9]+]]
+// CHECK-NOUSE-I386-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l100() #[[ATTR3:[0-9]+]]
 // CHECK-NOUSE-I386-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK-NOUSE-I386:       omp_offload.cont:
-// CHECK-NOUSE-I386-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    store ptr [[A]], ptr [[TMP71]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP73:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    store ptr [[A]], ptr [[TMP73]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP75:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_MAPPERS5]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP75]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP76:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    [[TMP77:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
-// CHECK-NOUSE-I386-NEXT:    [[TMP78:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    store i32 2, ptr [[TMP78]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP79:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
-// CHECK-NOUSE-I386-NEXT:    store i32 1, ptr [[TMP79]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP80:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 2
-// CHECK-NOUSE-I386-NEXT:    store ptr [[TMP76]], ptr [[TMP80]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP81:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 3
-// CHECK-NOUSE-I386-NEXT:    store ptr [[TMP77]], ptr [[TMP81]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP82:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 4
-// CHECK-NOUSE-I386-NEXT:    store ptr @.offload_sizes.1, ptr [[TMP82]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 5
-// CHECK-NOUSE-I386-NEXT:    store ptr @.offload_maptypes.2, ptr [[TMP83]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 6
-// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP84]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 7
-// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP85]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 8
-// CHECK-NOUSE-I386-NEXT:    store i64 0, ptr [[TMP86]], align 8
-// CHECK-NOUSE-I386-NEXT:    [[TMP87:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 9
-// CHECK-NOUSE-I386-NEXT:    store i64 0, ptr [[TMP87]], align 8
-// CHECK-NOUSE-I386-NEXT:    [[TMP88:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 10
-// CHECK-NOUSE-I386-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP88]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP89:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 11
-// CHECK-NOUSE-I386-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP89]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP90:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 12
-// CHECK-NOUSE-I386-NEXT:    store i32 0, ptr [[TMP90]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP91:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}.region_id, ptr [[KERNEL_ARGS6]])
-// CHECK-NOUSE-I386-NEXT:    [[TMP92:%.*]] = icmp ne i32 [[TMP91]], 0
-// CHECK-NOUSE-I386-NEXT:    br i1 [[TMP92]], label [[OMP_OFFLOAD_FAILED7:%.*]], label [[OMP_OFFLOAD_CONT8:%.*]]
+// CHECK-NOUSE-I386-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    store ptr [[A]], ptr [[TMP52]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    store ptr [[A]], ptr [[TMP53]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP54:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_MAPPERS5]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP54]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP55:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    [[TMP56:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    [[TMP57:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    store i32 2, ptr [[TMP57]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
+// CHECK-NOUSE-I386-NEXT:    store i32 1, ptr [[TMP58]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 2
+// CHECK-NOUSE-I386-NEXT:    store ptr [[TMP55]], ptr [[TMP59]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 3
+// CHECK-NOUSE-I386-NEXT:    store ptr [[TMP56]], ptr [[TMP60]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 4
+// CHECK-NOUSE-I386-NEXT:    store ptr @.offload_sizes.1, ptr [[TMP61]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 5
+// CHECK-NOUSE-I386-NEXT:    store ptr @.offload_maptypes.2, ptr [[TMP62]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 6
+// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP63]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 7
+// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP64]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 8
+// CHECK-NOUSE-I386-NEXT:    store i64 0, ptr [[TMP65]], align 8
+// CHECK-NOUSE-I386-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 9
+// CHECK-NOUSE-I386-NEXT:    store i64 0, ptr [[TMP66]], align 8
+// CHECK-NOUSE-I386-NEXT:    [[TMP67:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 10
+// CHECK-NOUSE-I386-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP67]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 11
+// CHECK-NOUSE-I386-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP68]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 12
+// CHECK-NOUSE-I386-NEXT:    store i32 0, ptr [[TMP69]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP70:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l114.region_id, ptr [[KERNEL_ARGS6]])
+// CHECK-NOUSE-I386-NEXT:    [[TMP71:%.*]] = icmp ne i32 [[TMP70]], 0
+// CHECK-NOUSE-I386-NEXT:    br i1 [[TMP71]], label [[OMP_OFFLOAD_FAILED7:%.*]], label [[OMP_OFFLOAD_CONT8:%.*]]
 // CHECK-NOUSE-I386:       omp_offload.failed7:
-// CHECK-NOUSE-I386-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}() #[[ATTR3]]
+// CHECK-NOUSE-I386-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l114() #[[ATTR3]]
 // CHECK-NOUSE-I386-NEXT:    br label [[OMP_OFFLOAD_CONT8]]
 // CHECK-NOUSE-I386:       omp_offload.cont8:
 // CHECK-NOUSE-I386-NEXT:    ret void
 //
 //
-// CHECK-NOUSE-I386-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}
+// CHECK-NOUSE-I386-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l100
 // CHECK-NOUSE-I386-SAME: () #[[ATTR1:[0-9]+]] {
 // CHECK-NOUSE-I386-NEXT:  entry:
 // CHECK-NOUSE-I386-NEXT:    ret void
 //
 //
-// CHECK-NOUSE-I386-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l{{[0-9]*}}
+// CHECK-NOUSE-I386-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z20explicit_maps_singlei_l114
 // CHECK-NOUSE-I386-SAME: () #[[ATTR1]] {
 // CHECK-NOUSE-I386-NEXT:  entry:
 // CHECK-NOUSE-I386-NEXT:    ret void
@@ -1221,77 +1220,77 @@ void ST::test_present_members() {
 // CHECK-NOUSE-I386-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK-NOUSE-I386-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
 // CHECK-NOUSE-I386-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [3 x i64], align 4
+// CHECK-NOUSE-I386-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NOUSE-I386-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK-NOUSE-I386-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK-NOUSE-I386-NEXT:    [[I:%.*]] = getelementptr inbounds [[STRUCT_ST:%.*]], ptr [[THIS1]], i32 0, i32 0
 // CHECK-NOUSE-I386-NEXT:    [[J:%.*]] = getelementptr inbounds [[STRUCT_ST]], ptr [[THIS1]], i32 0, i32 1
 // CHECK-NOUSE-I386-NEXT:    [[TMP0:%.*]] = getelementptr i32, ptr [[J]], i32 1
-// CHECK-NOUSE-I386-NEXT:    [[TMP3:%.*]] = ptrtoint ptr [[TMP0]] to i64
-// CHECK-NOUSE-I386-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[I]] to i64
-// CHECK-NOUSE-I386-NEXT:    [[TMP5:%.*]] = sub i64 [[TMP3]], [[TMP4]]
-// CHECK-NOUSE-I386-NEXT:    [[TMP6:%.*]] = sdiv exact i64 [[TMP5]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
+// CHECK-NOUSE-I386-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[TMP0]] to i64
+// CHECK-NOUSE-I386-NEXT:    [[TMP2:%.*]] = ptrtoint ptr [[I]] to i64
+// CHECK-NOUSE-I386-NEXT:    [[TMP3:%.*]] = sub i64 [[TMP1]], [[TMP2]]
+// CHECK-NOUSE-I386-NEXT:    [[TMP4:%.*]] = sdiv exact i64 [[TMP3]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
 // CHECK-NOUSE-I386-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[DOTOFFLOAD_SIZES]], ptr align 4 @.offload_sizes.3, i32 24, i1 false)
-// CHECK-NOUSE-I386-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    store ptr [[THIS1]], ptr [[TMP8]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    store ptr [[THIS1]], ptr [[TMP5]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    store ptr [[I]], ptr [[TMP6]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    store i64 [[TMP4]], ptr [[TMP7]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP8]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 1
+// CHECK-NOUSE-I386-NEXT:    store ptr [[THIS1]], ptr [[TMP9]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 1
 // CHECK-NOUSE-I386-NEXT:    store ptr [[I]], ptr [[TMP10]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    store i64 [[TMP6]], ptr [[TMP12]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP13]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 1
-// CHECK-NOUSE-I386-NEXT:    store ptr [[THIS1]], ptr [[TMP14]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 1
-// CHECK-NOUSE-I386-NEXT:    store ptr [[I]], ptr [[TMP16]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 1
-// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP18]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 2
-// CHECK-NOUSE-I386-NEXT:    store ptr [[THIS1]], ptr [[TMP19]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 2
-// CHECK-NOUSE-I386-NEXT:    store ptr [[J]], ptr [[TMP21]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 2
-// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP23]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
-// CHECK-NOUSE-I386-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
-// CHECK-NOUSE-I386-NEXT:    store i32 2, ptr [[TMP27]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
-// CHECK-NOUSE-I386-NEXT:    store i32 3, ptr [[TMP28]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 2
-// CHECK-NOUSE-I386-NEXT:    store ptr [[TMP24]], ptr [[TMP29]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 3
-// CHECK-NOUSE-I386-NEXT:    store ptr [[TMP25]], ptr [[TMP30]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP31:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 4
-// CHECK-NOUSE-I386-NEXT:    store ptr [[TMP26]], ptr [[TMP31]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 5
-// CHECK-NOUSE-I386-NEXT:    store ptr @.offload_maptypes.4, ptr [[TMP32]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 6
-// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP33]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 7
-// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP34]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 8
-// CHECK-NOUSE-I386-NEXT:    store i64 0, ptr [[TMP35]], align 8
-// CHECK-NOUSE-I386-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 9
-// CHECK-NOUSE-I386-NEXT:    store i64 0, ptr [[TMP36]], align 8
-// CHECK-NOUSE-I386-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 10
-// CHECK-NOUSE-I386-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP37]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 11
-// CHECK-NOUSE-I386-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP38]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 12
-// CHECK-NOUSE-I386-NEXT:    store i32 0, ptr [[TMP39]], align 4
-// CHECK-NOUSE-I386-NEXT:    [[TMP40:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l{{[0-9]*}}.region_id, ptr [[KERNEL_ARGS]])
-// CHECK-NOUSE-I386-NEXT:    [[TMP41:%.*]] = icmp ne i32 [[TMP40]], 0
-// CHECK-NOUSE-I386-NEXT:    br i1 [[TMP41]], label [[OMP_OFFLOAD_FAILED:%.*]], label [[OMP_OFFLOAD_CONT:%.*]]
+// CHECK-NOUSE-I386-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 1
+// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP11]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 2
+// CHECK-NOUSE-I386-NEXT:    store ptr [[THIS1]], ptr [[TMP12]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 2
+// CHECK-NOUSE-I386-NEXT:    store ptr [[J]], ptr [[TMP13]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i32 0, i32 2
+// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP14]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
+// CHECK-NOUSE-I386-NEXT:    store i32 2, ptr [[TMP18]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
+// CHECK-NOUSE-I386-NEXT:    store i32 3, ptr [[TMP19]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 2
+// CHECK-NOUSE-I386-NEXT:    store ptr [[TMP15]], ptr [[TMP20]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 3
+// CHECK-NOUSE-I386-NEXT:    store ptr [[TMP16]], ptr [[TMP21]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 4
+// CHECK-NOUSE-I386-NEXT:    store ptr [[TMP17]], ptr [[TMP22]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 5
+// CHECK-NOUSE-I386-NEXT:    store ptr @.offload_maptypes.4, ptr [[TMP23]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 6
+// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP24]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 7
+// CHECK-NOUSE-I386-NEXT:    store ptr null, ptr [[TMP25]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 8
+// CHECK-NOUSE-I386-NEXT:    store i64 0, ptr [[TMP26]], align 8
+// CHECK-NOUSE-I386-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 9
+// CHECK-NOUSE-I386-NEXT:    store i64 0, ptr [[TMP27]], align 8
+// CHECK-NOUSE-I386-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 10
+// CHECK-NOUSE-I386-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP28]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 11
+// CHECK-NOUSE-I386-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP29]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 12
+// CHECK-NOUSE-I386-NEXT:    store i32 0, ptr [[TMP30]], align 4
+// CHECK-NOUSE-I386-NEXT:    [[TMP31:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l125.region_id, ptr [[KERNEL_ARGS]])
+// CHECK-NOUSE-I386-NEXT:    [[TMP32:%.*]] = icmp ne i32 [[TMP31]], 0
+// CHECK-NOUSE-I386-NEXT:    br i1 [[TMP32]], label [[OMP_OFFLOAD_FAILED:%.*]], label [[OMP_OFFLOAD_CONT:%.*]]
 // CHECK-NOUSE-I386:       omp_offload.failed:
-// CHECK-NOUSE-I386-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l{{[0-9]*}}() #[[ATTR3]]
+// CHECK-NOUSE-I386-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l125() #[[ATTR3]]
 // CHECK-NOUSE-I386-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK-NOUSE-I386:       omp_offload.cont:
 // CHECK-NOUSE-I386-NEXT:    ret void
 //
 //
-// CHECK-NOUSE-I386-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l{{[0-9]*}}
+// CHECK-NOUSE-I386-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN2ST20test_present_membersEv_l125
 // CHECK-NOUSE-I386-SAME: () #[[ATTR1]] {
 // CHECK-NOUSE-I386-NEXT:  entry:
 // CHECK-NOUSE-I386-NEXT:    ret void
diff --git a/clang/test/OpenMP/target_map_deref_array_codegen.cpp b/clang/test/OpenMP/target_map_deref_array_codegen.cpp
new file mode 100644
index 0000000000000..11ff8125a0a99
--- /dev/null
+++ b/clang/test/OpenMP/target_map_deref_array_codegen.cpp
@@ -0,0 +1,312 @@
+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --function-signature --include-generated-funcs --replace-value-regex "__omp_offloading_[0-9a-z]+_[0-9a-z]+" "reduction_size[.].+[.]" "pl_cond[.].+[.|,]" --prefix-filecheck-ir-name _
+// RUN: %clang_cc1 -verify -fopenmp -fopenmp-targets=powerpc64le-ibm-linux-gnu -x c++ -triple powerpc64le-unknown-unknown -emit-llvm %s -o - | FileCheck %s
+// RUN: %clang_cc1 -fopenmp -fopenmp-targets=powerpc64le-ibm-linux-gnu -x c++ -std=c++11 -triple powerpc64le-unknown-unknown -emit-pch -o %t %s
+// RUN: %clang_cc1 -fopenmp -fopenmp-targets=powerpc64le-ibm-linux-gnu -x c++ -triple powerpc64le-unknown-unknown -std=c++11 -include-pch %t -verify %s -emit-llvm -o - | FileCheck %s
+
+// expected-no-diagnostics
+#ifndef HEADER
+#define HEADER
+
+extern void *malloc (int __size) throw () __attribute__ ((__malloc__));
+
+void foo(int **t1d)
+{
+  *t1d = (int *) malloc(3 * sizeof(int));
+  for (int j=0; j < 3; j++)
+    (*t1d)[j] = 1;
+  #pragma omp target map(to: (*t1d)[0:3])
+    (*t1d)[2] = 2;
+  #pragma omp target map(tofrom : (**t1d))
+    (*t1d)[0] = 3;
+  int a = 0, b = 0;
+  #pragma omp target map(tofrom : (*(*(t1d+a)+b)))
+    *(*(t1d+a)+b) = 4;
+}
+
+#endif
+
+// CHECK-LABEL: define {{[^@]+}}@_Z3fooPPi
+// CHECK-SAME: (ptr noundef [[T1D:%.*]]) #[[ATTR0:[0-9]+]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[T1D_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[J:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [2 x ptr], align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [2 x ptr], align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [2 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS2:%.*]] = alloca [2 x ptr], align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_PTRS3:%.*]] = alloca [2 x ptr], align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS4:%.*]] = alloca [2 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
+// CHECK-NEXT:    [[A:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[B:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[B_CASTED:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS12:%.*]] = alloca [4 x ptr], align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_PTRS13:%.*]] = alloca [4 x ptr], align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS14:%.*]] = alloca [4 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
+// CHECK-NEXT:    store ptr [[T1D]], ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[CALL:%.*]] = call noalias noundef ptr @_Z6malloci(i32 noundef signext 12) #[[ATTR3:[0-9]+]]
+// CHECK-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    store ptr [[CALL]], ptr [[TMP0]], align 8
+// CHECK-NEXT:    store i32 0, ptr [[J]], align 4
+// CHECK-NEXT:    br label [[FOR_COND:%.*]]
+// CHECK:       for.cond:
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[J]], align 4
+// CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 [[TMP1]], 3
+// CHECK-NEXT:    br i1 [[CMP]], label [[FOR_BODY:%.*]], label [[FOR_END:%.*]]
+// CHECK:       for.body:
+// CHECK-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[TMP2]], align 8
+// CHECK-NEXT:    [[TMP4:%.*]] = load i32, ptr [[J]], align 4
+// CHECK-NEXT:    [[IDXPROM:%.*]] = sext i32 [[TMP4]] to i64
+// CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[TMP3]], i64 [[IDXPROM]]
+// CHECK-NEXT:    store i32 1, ptr [[ARRAYIDX]], align 4
+// CHECK-NEXT:    br label [[FOR_INC:%.*]]
+// CHECK:       for.inc:
+// CHECK-NEXT:    [[TMP5:%.*]] = load i32, ptr [[J]], align 4
+// CHECK-NEXT:    [[INC:%.*]] = add nsw i32 [[TMP5]], 1
+// CHECK-NEXT:    store i32 [[INC]], ptr [[J]], align 4
+// CHECK-NEXT:    br label [[FOR_COND]], !llvm.loop [[LOOP6:![0-9]+]]
+// CHECK:       for.end:
+// CHECK-NEXT:    [[TMP6:%.*]] = load ptr, ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP7:%.*]] = load ptr, ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP8:%.*]] = load ptr, ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP9:%.*]] = load ptr, ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP10:%.*]] = load ptr, ptr [[TMP9]], align 8
+// CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds i32, ptr [[TMP10]], i64 0
+// CHECK-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-NEXT:    store ptr [[TMP7]], ptr [[TMP11]], align 8
+// CHECK-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-NEXT:    store ptr [[TMP8]], ptr [[TMP12]], align 8
+// CHECK-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 0
+// CHECK-NEXT:    store ptr null, ptr [[TMP13]], align 8
+// CHECK-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 1
+// CHECK-NEXT:    store ptr [[TMP8]], ptr [[TMP14]], align 8
+// CHECK-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 1
+// CHECK-NEXT:    store ptr [[ARRAYIDX1]], ptr [[TMP15]], align 8
+// CHECK-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_MAPPERS]], i64 0, i64 1
+// CHECK-NEXT:    store ptr null, ptr [[TMP16]], align 8
+// CHECK-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
+// CHECK-NEXT:    store i32 2, ptr [[TMP19]], align 4
+// CHECK-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
+// CHECK-NEXT:    store i32 2, ptr [[TMP20]], align 4
+// CHECK-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 2
+// CHECK-NEXT:    store ptr [[TMP17]], ptr [[TMP21]], align 8
+// CHECK-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 3
+// CHECK-NEXT:    store ptr [[TMP18]], ptr [[TMP22]], align 8
+// CHECK-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 4
+// CHECK-NEXT:    store ptr @.offload_sizes, ptr [[TMP23]], align 8
+// CHECK-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 5
+// CHECK-NEXT:    store ptr @.offload_maptypes, ptr [[TMP24]], align 8
+// CHECK-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 6
+// CHECK-NEXT:    store ptr null, ptr [[TMP25]], align 8
+// CHECK-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 7
+// CHECK-NEXT:    store ptr null, ptr [[TMP26]], align 8
+// CHECK-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 8
+// CHECK-NEXT:    store i64 0, ptr [[TMP27]], align 8
+// CHECK-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 9
+// CHECK-NEXT:    store i64 0, ptr [[TMP28]], align 8
+// CHECK-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 10
+// CHECK-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP29]], align 4
+// CHECK-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 11
+// CHECK-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP30]], align 4
+// CHECK-NEXT:    [[TMP31:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 12
+// CHECK-NEXT:    store i32 0, ptr [[TMP31]], align 4
+// CHECK-NEXT:    [[TMP32:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1:[0-9]+]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z3fooPPi_l17.region_id, ptr [[KERNEL_ARGS]])
+// CHECK-NEXT:    [[TMP33:%.*]] = icmp ne i32 [[TMP32]], 0
+// CHECK-NEXT:    br i1 [[TMP33]], label [[OMP_OFFLOAD_FAILED:%.*]], label [[OMP_OFFLOAD_CONT:%.*]]
+// CHECK:       omp_offload.failed:
+// CHECK-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z3fooPPi_l17(ptr [[TMP6]]) #[[ATTR3]]
+// CHECK-NEXT:    br label [[OMP_OFFLOAD_CONT]]
+// CHECK:       omp_offload.cont:
+// CHECK-NEXT:    [[TMP34:%.*]] = load ptr, ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP35:%.*]] = load ptr, ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP36:%.*]] = load ptr, ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP37:%.*]] = load ptr, ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP38:%.*]] = load ptr, ptr [[TMP37]], align 8
+// CHECK-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS2]], i32 0, i32 0
+// CHECK-NEXT:    store ptr [[TMP35]], ptr [[TMP39]], align 8
+// CHECK-NEXT:    [[TMP40:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS3]], i32 0, i32 0
+// CHECK-NEXT:    store ptr [[TMP36]], ptr [[TMP40]], align 8
+// CHECK-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_MAPPERS4]], i64 0, i64 0
+// CHECK-NEXT:    store ptr null, ptr [[TMP41]], align 8
+// CHECK-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS2]], i32 0, i32 1
+// CHECK-NEXT:    store ptr [[TMP36]], ptr [[TMP42]], align 8
+// CHECK-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS3]], i32 0, i32 1
+// CHECK-NEXT:    store ptr [[TMP38]], ptr [[TMP43]], align 8
+// CHECK-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_MAPPERS4]], i64 0, i64 1
+// CHECK-NEXT:    store ptr null, ptr [[TMP44]], align 8
+// CHECK-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS2]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS3]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP47:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
+// CHECK-NEXT:    store i32 2, ptr [[TMP47]], align 4
+// CHECK-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
+// CHECK-NEXT:    store i32 2, ptr [[TMP48]], align 4
+// CHECK-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 2
+// CHECK-NEXT:    store ptr [[TMP45]], ptr [[TMP49]], align 8
+// CHECK-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 3
+// CHECK-NEXT:    store ptr [[TMP46]], ptr [[TMP50]], align 8
+// CHECK-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 4
+// CHECK-NEXT:    store ptr @.offload_sizes.1, ptr [[TMP51]], align 8
+// CHECK-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 5
+// CHECK-NEXT:    store ptr @.offload_maptypes.2, ptr [[TMP52]], align 8
+// CHECK-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 6
+// CHECK-NEXT:    store ptr null, ptr [[TMP53]], align 8
+// CHECK-NEXT:    [[TMP54:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 7
+// CHECK-NEXT:    store ptr null, ptr [[TMP54]], align 8
+// CHECK-NEXT:    [[TMP55:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 8
+// CHECK-NEXT:    store i64 0, ptr [[TMP55]], align 8
+// CHECK-NEXT:    [[TMP56:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 9
+// CHECK-NEXT:    store i64 0, ptr [[TMP56]], align 8
+// CHECK-NEXT:    [[TMP57:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 10
+// CHECK-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP57]], align 4
+// CHECK-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 11
+// CHECK-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP58]], align 4
+// CHECK-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 12
+// CHECK-NEXT:    store i32 0, ptr [[TMP59]], align 4
+// CHECK-NEXT:    [[TMP60:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z3fooPPi_l19.region_id, ptr [[KERNEL_ARGS5]])
+// CHECK-NEXT:    [[TMP61:%.*]] = icmp ne i32 [[TMP60]], 0
+// CHECK-NEXT:    br i1 [[TMP61]], label [[OMP_OFFLOAD_FAILED6:%.*]], label [[OMP_OFFLOAD_CONT7:%.*]]
+// CHECK:       omp_offload.failed6:
+// CHECK-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z3fooPPi_l19(ptr [[TMP34]]) #[[ATTR3]]
+// CHECK-NEXT:    br label [[OMP_OFFLOAD_CONT7]]
+// CHECK:       omp_offload.cont7:
+// CHECK-NEXT:    store i32 0, ptr [[A]], align 4
+// CHECK-NEXT:    store i32 0, ptr [[B]], align 4
+// CHECK-NEXT:    [[TMP62:%.*]] = load ptr, ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP63:%.*]] = load i32, ptr [[A]], align 4
+// CHECK-NEXT:    store i32 [[TMP63]], ptr [[A_CASTED]], align 4
+// CHECK-NEXT:    [[TMP64:%.*]] = load i64, ptr [[A_CASTED]], align 8
+// CHECK-NEXT:    [[TMP65:%.*]] = load i32, ptr [[B]], align 4
+// CHECK-NEXT:    store i32 [[TMP65]], ptr [[B_CASTED]], align 4
+// CHECK-NEXT:    [[TMP66:%.*]] = load i64, ptr [[B_CASTED]], align 8
+// CHECK-NEXT:    [[TMP67:%.*]] = load ptr, ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP68:%.*]] = load ptr, ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP69:%.*]] = load i32, ptr [[A]], align 4
+// CHECK-NEXT:    [[IDX_EXT:%.*]] = sext i32 [[TMP69]] to i64
+// CHECK-NEXT:    [[ADD_PTR:%.*]] = getelementptr inbounds ptr, ptr [[TMP68]], i64 [[IDX_EXT]]
+// CHECK-NEXT:    [[TMP70:%.*]] = load ptr, ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP71:%.*]] = load i32, ptr [[A]], align 4
+// CHECK-NEXT:    [[IDX_EXT8:%.*]] = sext i32 [[TMP71]] to i64
+// CHECK-NEXT:    [[ADD_PTR9:%.*]] = getelementptr inbounds ptr, ptr [[TMP70]], i64 [[IDX_EXT8]]
+// CHECK-NEXT:    [[TMP72:%.*]] = load ptr, ptr [[ADD_PTR9]], align 8
+// CHECK-NEXT:    [[TMP73:%.*]] = load i32, ptr [[B]], align 4
+// CHECK-NEXT:    [[IDX_EXT10:%.*]] = sext i32 [[TMP73]] to i64
+// CHECK-NEXT:    [[ADD_PTR11:%.*]] = getelementptr inbounds i32, ptr [[TMP72]], i64 [[IDX_EXT10]]
+// CHECK-NEXT:    [[TMP74:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS12]], i32 0, i32 0
+// CHECK-NEXT:    store ptr [[TMP67]], ptr [[TMP74]], align 8
+// CHECK-NEXT:    [[TMP75:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS13]], i32 0, i32 0
+// CHECK-NEXT:    store ptr [[ADD_PTR]], ptr [[TMP75]], align 8
+// CHECK-NEXT:    [[TMP76:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_MAPPERS14]], i64 0, i64 0
+// CHECK-NEXT:    store ptr null, ptr [[TMP76]], align 8
+// CHECK-NEXT:    [[TMP77:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS12]], i32 0, i32 1
+// CHECK-NEXT:    store ptr [[ADD_PTR]], ptr [[TMP77]], align 8
+// CHECK-NEXT:    [[TMP78:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS13]], i32 0, i32 1
+// CHECK-NEXT:    store ptr [[ADD_PTR11]], ptr [[TMP78]], align 8
+// CHECK-NEXT:    [[TMP79:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_MAPPERS14]], i64 0, i64 1
+// CHECK-NEXT:    store ptr null, ptr [[TMP79]], align 8
+// CHECK-NEXT:    [[TMP80:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS12]], i32 0, i32 2
+// CHECK-NEXT:    store i64 [[TMP64]], ptr [[TMP80]], align 8
+// CHECK-NEXT:    [[TMP81:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS13]], i32 0, i32 2
+// CHECK-NEXT:    store i64 [[TMP64]], ptr [[TMP81]], align 8
+// CHECK-NEXT:    [[TMP82:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_MAPPERS14]], i64 0, i64 2
+// CHECK-NEXT:    store ptr null, ptr [[TMP82]], align 8
+// CHECK-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS12]], i32 0, i32 3
+// CHECK-NEXT:    store i64 [[TMP66]], ptr [[TMP83]], align 8
+// CHECK-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS13]], i32 0, i32 3
+// CHECK-NEXT:    store i64 [[TMP66]], ptr [[TMP84]], align 8
+// CHECK-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_MAPPERS14]], i64 0, i64 3
+// CHECK-NEXT:    store ptr null, ptr [[TMP85]], align 8
+// CHECK-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS12]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP87:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS13]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP88:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
+// CHECK-NEXT:    store i32 2, ptr [[TMP88]], align 4
+// CHECK-NEXT:    [[TMP89:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
+// CHECK-NEXT:    store i32 4, ptr [[TMP89]], align 4
+// CHECK-NEXT:    [[TMP90:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 2
+// CHECK-NEXT:    store ptr [[TMP86]], ptr [[TMP90]], align 8
+// CHECK-NEXT:    [[TMP91:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 3
+// CHECK-NEXT:    store ptr [[TMP87]], ptr [[TMP91]], align 8
+// CHECK-NEXT:    [[TMP92:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 4
+// CHECK-NEXT:    store ptr @.offload_sizes.3, ptr [[TMP92]], align 8
+// CHECK-NEXT:    [[TMP93:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 5
+// CHECK-NEXT:    store ptr @.offload_maptypes.4, ptr [[TMP93]], align 8
+// CHECK-NEXT:    [[TMP94:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 6
+// CHECK-NEXT:    store ptr null, ptr [[TMP94]], align 8
+// CHECK-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 7
+// CHECK-NEXT:    store ptr null, ptr [[TMP95]], align 8
+// CHECK-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 8
+// CHECK-NEXT:    store i64 0, ptr [[TMP96]], align 8
+// CHECK-NEXT:    [[TMP97:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 9
+// CHECK-NEXT:    store i64 0, ptr [[TMP97]], align 8
+// CHECK-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 10
+// CHECK-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP98]], align 4
+// CHECK-NEXT:    [[TMP99:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 11
+// CHECK-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP99]], align 4
+// CHECK-NEXT:    [[TMP100:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 12
+// CHECK-NEXT:    store i32 0, ptr [[TMP100]], align 4
+// CHECK-NEXT:    [[TMP101:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z3fooPPi_l22.region_id, ptr [[KERNEL_ARGS15]])
+// CHECK-NEXT:    [[TMP102:%.*]] = icmp ne i32 [[TMP101]], 0
+// CHECK-NEXT:    br i1 [[TMP102]], label [[OMP_OFFLOAD_FAILED16:%.*]], label [[OMP_OFFLOAD_CONT17:%.*]]
+// CHECK:       omp_offload.failed16:
+// CHECK-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z3fooPPi_l22(ptr [[TMP62]], i64 [[TMP64]], i64 [[TMP66]]) #[[ATTR3]]
+// CHECK-NEXT:    br label [[OMP_OFFLOAD_CONT17]]
+// CHECK:       omp_offload.cont17:
+// CHECK-NEXT:    ret void
+//
+//
+// CHECK-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z3fooPPi_l17
+// CHECK-SAME: (ptr noundef [[T1D:%.*]]) #[[ATTR2:[0-9]+]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[T1D_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    store ptr [[T1D]], ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = load ptr, ptr [[TMP0]], align 8
+// CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[TMP1]], i64 2
+// CHECK-NEXT:    store i32 2, ptr [[ARRAYIDX]], align 4
+// CHECK-NEXT:    ret void
+//
+//
+// CHECK-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z3fooPPi_l19
+// CHECK-SAME: (ptr noundef [[T1D:%.*]]) #[[ATTR2]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[T1D_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    store ptr [[T1D]], ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = load ptr, ptr [[TMP0]], align 8
+// CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[TMP1]], i64 0
+// CHECK-NEXT:    store i32 3, ptr [[ARRAYIDX]], align 4
+// CHECK-NEXT:    ret void
+//
+//
+// CHECK-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z3fooPPi_l22
+// CHECK-SAME: (ptr noundef [[T1D:%.*]], i64 noundef [[A:%.*]], i64 noundef [[B:%.*]]) #[[ATTR2]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[T1D_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    store ptr [[T1D]], ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    store i64 [[A]], ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    store i64 [[B]], ptr [[B_ADDR]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[T1D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[A_ADDR]], align 4
+// CHECK-NEXT:    [[IDX_EXT:%.*]] = sext i32 [[TMP1]] to i64
+// CHECK-NEXT:    [[ADD_PTR:%.*]] = getelementptr inbounds ptr, ptr [[TMP0]], i64 [[IDX_EXT]]
+// CHECK-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[ADD_PTR]], align 8
+// CHECK-NEXT:    [[TMP3:%.*]] = load i32, ptr [[B_ADDR]], align 4
+// CHECK-NEXT:    [[IDX_EXT1:%.*]] = sext i32 [[TMP3]] to i64
+// CHECK-NEXT:    [[ADD_PTR2:%.*]] = getelementptr inbounds i32, ptr [[TMP2]], i64 [[IDX_EXT1]]
+// CHECK-NEXT:    store i32 4, ptr [[ADD_PTR2]], align 4
+// CHECK-NEXT:    ret void
+//
+//
+// CHECK-LABEL: define {{[^@]+}}@.omp_offloading.requires_reg
+// CHECK-SAME: () #[[ATTR4:[0-9]+]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    call void @__tgt_register_requires(i64 1)
+// CHECK-NEXT:    ret void
+//
diff --git a/clang/test/OpenMP/target_map_member_expr_codegen.cpp b/clang/test/OpenMP/target_map_member_expr_codegen.cpp
index fa47fcee41a4c..dc2b73638eda6 100644
--- a/clang/test/OpenMP/target_map_member_expr_codegen.cpp
+++ b/clang/test/OpenMP/target_map_member_expr_codegen.cpp
@@ -56,6 +56,9 @@ class C : public BASE
        for (int i = 0; i < Csize; ++i)
           d.C[i] = 1;
      }
+     #pragma omp target map(from:d.C[0:Csize])
+       for (int i = 0; i < Csize; ++i)
+          d.C[i] = 1;
    }
 };
 
@@ -66,8 +69,6 @@ void foo() {
   descriptor<float> d;
   c.bar(d);
 }
-// CHECK: @.offload_sizes = private unnamed_addr constant [4 x i64] [i64 12, i64 4, i64 4, i64 4]
-// CHECK-NOT: @.offload_sizes = private unnamed_addr constant [4 x i64] [i64 0, i64 4, i64 4, i64 4]
 
 // CHECK-LABEL: define {{[^@]+}}@_Z3foov
 // CHECK-SAME: () #[[ATTR0:[0-9]+]] {
@@ -104,6 +105,7 @@ void foo() {
 // CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK-NEXT:    [[RES:%.*]] = getelementptr inbounds [[CLASS_B:%.*]], ptr [[THIS1]], i32 0, i32 1
@@ -135,7 +137,6 @@ void foo() {
 // CHECK-NEXT:    store ptr null, ptr [[TMP11]], align 8
 // CHECK-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP14]], align 4
 // CHECK-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -189,6 +190,14 @@ void foo() {
 // CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS9:%.*]] = alloca [2 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_PTRS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS11:%.*]] = alloca [2 x ptr], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK-NEXT:    [[_TMP12:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[CSIZE_CASTED13:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_BASEPTRS18:%.*]] = alloca [3 x ptr], align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_PTRS19:%.*]] = alloca [3 x ptr], align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_MAPPERS20:%.*]] = alloca [3 x ptr], align 8
+// CHECK-NEXT:    [[DOTOFFLOAD_SIZES21:%.*]] = alloca [3 x i64], align 8
+// CHECK-NEXT:    [[KERNEL_ARGS22:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK-NEXT:    store ptr [[D]], ptr [[D_ADDR]], align 8
 // CHECK-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
@@ -285,7 +294,6 @@ void foo() {
 // CHECK-NEXT:    store ptr null, ptr [[TMP52]], align 8
 // CHECK-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS9]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP54:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS10]], i32 0, i32 0
-// CHECK-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK-NEXT:    [[TMP55:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK-NEXT:    store i32 2, ptr [[TMP55]], align 4
 // CHECK-NEXT:    [[TMP56:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -323,6 +331,86 @@ void foo() {
 // CHECK-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK-NEXT:    [[TMP72:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
 // CHECK-NEXT:    call void @__tgt_target_data_end_mapper(ptr @[[GLOB1]], i64 -1, i32 3, ptr [[TMP70]], ptr [[TMP71]], ptr [[TMP72]], ptr @.offload_maptypes.1, ptr null, ptr null)
+// CHECK-NEXT:    [[TMP73:%.*]] = load ptr, ptr [[D_ADDR]], align 8
+// CHECK-NEXT:    store ptr [[TMP73]], ptr [[_TMP12]], align 8
+// CHECK-NEXT:    [[TMP74:%.*]] = load i32, ptr [[CSIZE]], align 4
+// CHECK-NEXT:    store i32 [[TMP74]], ptr [[CSIZE_CASTED13]], align 4
+// CHECK-NEXT:    [[TMP75:%.*]] = load i64, ptr [[CSIZE_CASTED13]], align 8
+// CHECK-NEXT:    [[TMP76:%.*]] = load ptr, ptr [[_TMP12]], align 8
+// CHECK-NEXT:    [[TMP77:%.*]] = load ptr, ptr [[_TMP12]], align 8
+// CHECK-NEXT:    [[TMP78:%.*]] = load ptr, ptr [[_TMP12]], align 8
+// CHECK-NEXT:    [[C14:%.*]] = getelementptr inbounds [[STRUCT_DESCRIPTOR]], ptr [[TMP78]], i32 0, i32 1
+// CHECK-NEXT:    [[TMP79:%.*]] = load ptr, ptr [[_TMP12]], align 8
+// CHECK-NEXT:    [[C15:%.*]] = getelementptr inbounds [[STRUCT_DESCRIPTOR]], ptr [[TMP79]], i32 0, i32 1
+// CHECK-NEXT:    [[TMP80:%.*]] = load ptr, ptr [[C15]], align 8
+// CHECK-NEXT:    [[ARRAYIDX16:%.*]] = getelementptr inbounds float, ptr [[TMP80]], i64 0
+// CHECK-NEXT:    [[TMP81:%.*]] = load i32, ptr [[CSIZE]], align 4
+// CHECK-NEXT:    [[CONV17:%.*]] = zext i32 [[TMP81]] to i64
+// CHECK-NEXT:    [[TMP82:%.*]] = mul nuw i64 [[CONV17]], 4
+// CHECK-NEXT:    [[TMP83:%.*]] = getelementptr ptr, ptr [[C14]], i32 1
+// CHECK-NEXT:    [[TMP84:%.*]] = ptrtoint ptr [[TMP83]] to i64
+// CHECK-NEXT:    [[TMP85:%.*]] = ptrtoint ptr [[C14]] to i64
+// CHECK-NEXT:    [[TMP86:%.*]] = sub i64 [[TMP84]], [[TMP85]]
+// CHECK-NEXT:    [[TMP87:%.*]] = sdiv exact i64 [[TMP86]], ptrtoint (ptr getelementptr (i8, ptr null, i32 1) to i64)
+// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[DOTOFFLOAD_SIZES21]], ptr align 8 @.offload_sizes.4, i64 24, i1 false)
+// CHECK-NEXT:    [[TMP88:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS18]], i32 0, i32 0
+// CHECK-NEXT:    store i64 [[TMP75]], ptr [[TMP88]], align 8
+// CHECK-NEXT:    [[TMP89:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS19]], i32 0, i32 0
+// CHECK-NEXT:    store i64 [[TMP75]], ptr [[TMP89]], align 8
+// CHECK-NEXT:    [[TMP90:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS20]], i64 0, i64 0
+// CHECK-NEXT:    store ptr null, ptr [[TMP90]], align 8
+// CHECK-NEXT:    [[TMP91:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS18]], i32 0, i32 1
+// CHECK-NEXT:    store ptr [[TMP77]], ptr [[TMP91]], align 8
+// CHECK-NEXT:    [[TMP92:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS19]], i32 0, i32 1
+// CHECK-NEXT:    store ptr [[C14]], ptr [[TMP92]], align 8
+// CHECK-NEXT:    [[TMP93:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES21]], i32 0, i32 1
+// CHECK-NEXT:    store i64 [[TMP87]], ptr [[TMP93]], align 8
+// CHECK-NEXT:    [[TMP94:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS20]], i64 0, i64 1
+// CHECK-NEXT:    store ptr null, ptr [[TMP94]], align 8
+// CHECK-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS18]], i32 0, i32 2
+// CHECK-NEXT:    store ptr [[C14]], ptr [[TMP95]], align 8
+// CHECK-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS19]], i32 0, i32 2
+// CHECK-NEXT:    store ptr [[ARRAYIDX16]], ptr [[TMP96]], align 8
+// CHECK-NEXT:    [[TMP97:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES21]], i32 0, i32 2
+// CHECK-NEXT:    store i64 [[TMP82]], ptr [[TMP97]], align 8
+// CHECK-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_MAPPERS20]], i64 0, i64 2
+// CHECK-NEXT:    store ptr null, ptr [[TMP98]], align 8
+// CHECK-NEXT:    [[TMP99:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS18]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP100:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS19]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP101:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES21]], i32 0, i32 0
+// CHECK-NEXT:    [[TMP102:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS22]], i32 0, i32 0
+// CHECK-NEXT:    store i32 2, ptr [[TMP102]], align 4
+// CHECK-NEXT:    [[TMP103:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS22]], i32 0, i32 1
+// CHECK-NEXT:    store i32 3, ptr [[TMP103]], align 4
+// CHECK-NEXT:    [[TMP104:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS22]], i32 0, i32 2
+// CHECK-NEXT:    store ptr [[TMP99]], ptr [[TMP104]], align 8
+// CHECK-NEXT:    [[TMP105:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS22]], i32 0, i32 3
+// CHECK-NEXT:    store ptr [[TMP100]], ptr [[TMP105]], align 8
+// CHECK-NEXT:    [[TMP106:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS22]], i32 0, i32 4
+// CHECK-NEXT:    store ptr [[TMP101]], ptr [[TMP106]], align 8
+// CHECK-NEXT:    [[TMP107:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS22]], i32 0, i32 5
+// CHECK-NEXT:    store ptr @.offload_maptypes.5, ptr [[TMP107]], align 8
+// CHECK-NEXT:    [[TMP108:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS22]], i32 0, i32 6
+// CHECK-NEXT:    store ptr null, ptr [[TMP108]], align 8
+// CHECK-NEXT:    [[TMP109:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS22]], i32 0, i32 7
+// CHECK-NEXT:    store ptr null, ptr [[TMP109]], align 8
+// CHECK-NEXT:    [[TMP110:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS22]], i32 0, i32 8
+// CHECK-NEXT:    store i64 0, ptr [[TMP110]], align 8
+// CHECK-NEXT:    [[TMP111:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS22]], i32 0, i32 9
+// CHECK-NEXT:    store i64 0, ptr [[TMP111]], align 8
+// CHECK-NEXT:    [[TMP112:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS22]], i32 0, i32 10
+// CHECK-NEXT:    store [3 x i32] [i32 -1, i32 0, i32 0], ptr [[TMP112]], align 4
+// CHECK-NEXT:    [[TMP113:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS22]], i32 0, i32 11
+// CHECK-NEXT:    store [3 x i32] zeroinitializer, ptr [[TMP113]], align 4
+// CHECK-NEXT:    [[TMP114:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS22]], i32 0, i32 12
+// CHECK-NEXT:    store i32 0, ptr [[TMP114]], align 4
+// CHECK-NEXT:    [[TMP115:%.*]] = call i32 @__tgt_target_kernel(ptr @[[GLOB1]], i64 -1, i32 -1, i32 0, ptr @.{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN1C3barER10descriptorIfE_l59.region_id, ptr [[KERNEL_ARGS22]])
+// CHECK-NEXT:    [[TMP116:%.*]] = icmp ne i32 [[TMP115]], 0
+// CHECK-NEXT:    br i1 [[TMP116]], label [[OMP_OFFLOAD_FAILED23:%.*]], label [[OMP_OFFLOAD_CONT24:%.*]]
+// CHECK:       omp_offload.failed23:
+// CHECK-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN1C3barER10descriptorIfE_l59(i64 [[TMP75]], ptr [[TMP76]]) #[[ATTR3]]
+// CHECK-NEXT:    br label [[OMP_OFFLOAD_CONT24]]
+// CHECK:       omp_offload.cont24:
 // CHECK-NEXT:    ret void
 //
 //
@@ -433,13 +521,49 @@ void foo() {
 // CHECK-NEXT:    [[TMP6:%.*]] = load i32, ptr [[I]], align 4
 // CHECK-NEXT:    [[INC:%.*]] = add nsw i32 [[TMP6]], 1
 // CHECK-NEXT:    store i32 [[INC]], ptr [[I]], align 4
-// CHECK-NEXT:    br label [[FOR_COND]], !llvm.loop [[LOOP5:![0-9]+]]
+// CHECK-NEXT:    br label [[FOR_COND]], !llvm.loop [[LOOP6:![0-9]+]]
+// CHECK:       for.end:
+// CHECK-NEXT:    ret void
+//
+//
+// CHECK-LABEL: define {{[^@]+}}@{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__ZN1C3barER10descriptorIfE_l59
+// CHECK-SAME: (i64 noundef [[CSIZE:%.*]], ptr noundef nonnull align 8 dereferenceable(40) [[D:%.*]]) #[[ATTR2]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[CSIZE_ADDR:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[D_ADDR:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[TMP:%.*]] = alloca ptr, align 8
+// CHECK-NEXT:    [[I:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    store i64 [[CSIZE]], ptr [[CSIZE_ADDR]], align 8
+// CHECK-NEXT:    store ptr [[D]], ptr [[D_ADDR]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[D_ADDR]], align 8
+// CHECK-NEXT:    store ptr [[TMP0]], ptr [[TMP]], align 8
+// CHECK-NEXT:    store i32 0, ptr [[I]], align 4
+// CHECK-NEXT:    br label [[FOR_COND:%.*]]
+// CHECK:       for.cond:
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[I]], align 4
+// CHECK-NEXT:    [[TMP2:%.*]] = load i32, ptr [[CSIZE_ADDR]], align 4
+// CHECK-NEXT:    [[CMP:%.*]] = icmp ult i32 [[TMP1]], [[TMP2]]
+// CHECK-NEXT:    br i1 [[CMP]], label [[FOR_BODY:%.*]], label [[FOR_END:%.*]]
+// CHECK:       for.body:
+// CHECK-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[TMP]], align 8
+// CHECK-NEXT:    [[C:%.*]] = getelementptr inbounds [[STRUCT_DESCRIPTOR:%.*]], ptr [[TMP3]], i32 0, i32 1
+// CHECK-NEXT:    [[TMP4:%.*]] = load ptr, ptr [[C]], align 8
+// CHECK-NEXT:    [[TMP5:%.*]] = load i32, ptr [[I]], align 4
+// CHECK-NEXT:    [[IDXPROM:%.*]] = sext i32 [[TMP5]] to i64
+// CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds float, ptr [[TMP4]], i64 [[IDXPROM]]
+// CHECK-NEXT:    store float 1.000000e+00, ptr [[ARRAYIDX]], align 4
+// CHECK-NEXT:    br label [[FOR_INC:%.*]]
+// CHECK:       for.inc:
+// CHECK-NEXT:    [[TMP6:%.*]] = load i32, ptr [[I]], align 4
+// CHECK-NEXT:    [[INC:%.*]] = add nsw i32 [[TMP6]], 1
+// CHECK-NEXT:    store i32 [[INC]], ptr [[I]], align 4
+// CHECK-NEXT:    br label [[FOR_COND]], !llvm.loop [[LOOP10:![0-9]+]]
 // CHECK:       for.end:
 // CHECK-NEXT:    ret void
 //
 //
 // CHECK-LABEL: define {{[^@]+}}@.omp_offloading.requires_reg
-// CHECK-SAME: () #[[ATTR4:[0-9]+]] section ".text.startup" {
+// CHECK-SAME: () #[[ATTR5:[0-9]+]] section ".text.startup" {
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    call void @__tgt_register_requires(i64 1)
 // CHECK-NEXT:    ret void
diff --git a/clang/test/OpenMP/target_offload_mandatory_codegen.cpp b/clang/test/OpenMP/target_offload_mandatory_codegen.cpp
index 2062e2e390fd4..5803a81c4b1a5 100644
--- a/clang/test/OpenMP/target_offload_mandatory_codegen.cpp
+++ b/clang/test/OpenMP/target_offload_mandatory_codegen.cpp
@@ -71,13 +71,13 @@ void host_dev(int device) {
 // MANDATORY-SAME: (i1 noundef zeroext [[COND:%.*]]) #[[ATTR0]] {
 // MANDATORY-NEXT:  entry:
 // MANDATORY-NEXT:    [[COND_ADDR:%.*]] = alloca i8, align 1
+// MANDATORY-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // MANDATORY-NEXT:    [[FROMBOOL:%.*]] = zext i1 [[COND]] to i8
 // MANDATORY-NEXT:    store i8 [[FROMBOOL]], ptr [[COND_ADDR]], align 1
 // MANDATORY-NEXT:    [[TMP0:%.*]] = load i8, ptr [[COND_ADDR]], align 1
 // MANDATORY-NEXT:    [[TOBOOL:%.*]] = trunc i8 [[TMP0]] to i1
 // MANDATORY-NEXT:    br i1 [[TOBOOL]], label [[OMP_IF_THEN:%.*]], label [[OMP_IF_ELSE:%.*]]
 // MANDATORY:       omp_if.then:
-// MANDATORY-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // MANDATORY-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // MANDATORY-NEXT:    store i32 2, ptr [[TMP1]], align 4
 // MANDATORY-NEXT:    [[TMP2:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -122,12 +122,12 @@ void host_dev(int device) {
 // MANDATORY-NEXT:  entry:
 // MANDATORY-NEXT:    [[DEVICE_ADDR:%.*]] = alloca i32, align 4
 // MANDATORY-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
+// MANDATORY-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // MANDATORY-NEXT:    store i32 [[DEVICE]], ptr [[DEVICE_ADDR]], align 4
 // MANDATORY-NEXT:    [[TMP0:%.*]] = load i32, ptr [[DEVICE_ADDR]], align 4
 // MANDATORY-NEXT:    store i32 [[TMP0]], ptr [[DOTCAPTURE_EXPR_]], align 4
 // MANDATORY-NEXT:    [[TMP1:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
 // MANDATORY-NEXT:    [[TMP2:%.*]] = sext i32 [[TMP1]] to i64
-// MANDATORY-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // MANDATORY-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // MANDATORY-NEXT:    store i32 2, ptr [[TMP3]], align 4
 // MANDATORY-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_ompx_dyn_cgroup_mem_codegen.cpp b/clang/test/OpenMP/target_ompx_dyn_cgroup_mem_codegen.cpp
index a023946f431db..b38a4273d4724 100644
--- a/clang/test/OpenMP/target_ompx_dyn_cgroup_mem_codegen.cpp
+++ b/clang/test/OpenMP/target_ompx_dyn_cgroup_mem_codegen.cpp
@@ -214,9 +214,11 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
@@ -253,7 +255,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP17:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -296,7 +297,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP35]], align 8
 // CHECK1-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP38]], align 4
 // CHECK1-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -351,6 +351,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_4:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_6:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED8:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS9:%.*]] = alloca [1 x ptr], align 8
@@ -406,7 +407,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP24:%.*]] = zext i32 [[ADD]] to i64
 // CHECK1-NEXT:    [[TMP25:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK1-NEXT:    [[TMP26:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP20]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP27]], align 4
 // CHECK1-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -480,6 +480,7 @@ int bar(int n){
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[N_ADDR:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[A:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[B:%.*]] = alloca i16, align 2
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i16, align 2
 // CHECK1-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
@@ -488,9 +489,9 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS1:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -559,7 +560,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP33:%.*]] = load i16, ptr [[DOTCAPTURE_EXPR_]], align 2
 // CHECK1-NEXT:    [[TMP34:%.*]] = sext i16 [[TMP33]] to i32
 // CHECK1-NEXT:    [[TMP35:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP34]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS1:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS1]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP36]], align 4
 // CHECK1-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS1]], i32 0, i32 1
@@ -927,8 +927,8 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR_I:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR1_I:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR2_I:%.*]] = alloca ptr, align 8
-// CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED_I:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED_I:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTADDR:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTADDR1:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[DOTADDR]], align 4
@@ -1103,9 +1103,11 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
@@ -1142,7 +1144,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP17:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1185,7 +1186,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP35]], align 4
 // CHECK3-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP38]], align 4
 // CHECK3-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -1240,6 +1240,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_4:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_6:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR__CASTED8:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS9:%.*]] = alloca [1 x ptr], align 4
@@ -1295,7 +1296,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP24:%.*]] = zext i32 [[ADD]] to i64
 // CHECK3-NEXT:    [[TMP25:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK3-NEXT:    [[TMP26:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP20]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP27]], align 4
 // CHECK3-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1369,6 +1369,7 @@ int bar(int n){
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[N_ADDR:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[A:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[B:%.*]] = alloca i16, align 2
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i16, align 2
 // CHECK3-NEXT:    [[A_CASTED:%.*]] = alloca i32, align 4
@@ -1377,9 +1378,9 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS1:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1448,7 +1449,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP33:%.*]] = load i16, ptr [[DOTCAPTURE_EXPR_]], align 2
 // CHECK3-NEXT:    [[TMP34:%.*]] = sext i16 [[TMP33]] to i32
 // CHECK3-NEXT:    [[TMP35:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP34]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS1:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS1]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP36]], align 4
 // CHECK3-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS1]], i32 0, i32 1
@@ -1812,8 +1812,8 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR_I:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR1_I:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR2_I:%.*]] = alloca ptr, align 4
-// CHECK3-NEXT:    [[DOTCAPTURE_EXPR__CASTED_I:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK3-NEXT:    [[DOTCAPTURE_EXPR__CASTED_I:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTADDR:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTADDR1:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[DOTADDR]], align 4
diff --git a/clang/test/OpenMP/target_parallel_codegen.cpp b/clang/test/OpenMP/target_parallel_codegen.cpp
index 02788b69e2680..a35e64d7c8e77 100644
--- a/clang/test/OpenMP/target_parallel_codegen.cpp
+++ b/clang/test/OpenMP/target_parallel_codegen.cpp
@@ -309,16 +309,19 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[A_CASTED2:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[AA_CASTED3:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS6:%.*]] = alloca [2 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[A_CASTED10:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS13:%.*]] = alloca [9 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS14:%.*]] = alloca [9 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS15:%.*]] = alloca [9 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [9 x i64], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS16:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB1:[0-9]+]])
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
@@ -352,7 +355,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP16]], align 8
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP19]], align 4
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -410,7 +412,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP44]], align 8
 // CHECK1-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS4]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS5]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP47:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP47]], align 4
 // CHECK1-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -521,7 +522,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP97:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_BASEPTRS13]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_PTRS14]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP99:%.*]] = getelementptr inbounds [9 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS16:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP100:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS16]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP100]], align 4
 // CHECK1-NEXT:    [[TMP101:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS16]], i32 0, i32 1
@@ -911,6 +911,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [5 x i64], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
@@ -970,7 +971,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [5 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK1-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1035,6 +1035,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK1-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -1078,7 +1079,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP18]], align 8
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK1-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1133,6 +1133,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK1-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -1166,7 +1167,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP13]], align 8
 // CHECK1-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1412,16 +1412,19 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[A_CASTED2:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[AA_CASTED3:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS6:%.*]] = alloca [2 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[A_CASTED10:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS13:%.*]] = alloca [9 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS14:%.*]] = alloca [9 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS15:%.*]] = alloca [9 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [9 x i64], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS16:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB1:[0-9]+]])
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
@@ -1453,7 +1456,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP14]], align 4
 // CHECK3-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP17]], align 4
 // CHECK3-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1511,7 +1513,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK3-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS4]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS5]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK3-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -1624,7 +1625,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP97:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_BASEPTRS13]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_PTRS14]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP99:%.*]] = getelementptr inbounds [9 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS16:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP100:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS16]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP100]], align 4
 // CHECK3-NEXT:    [[TMP101:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS16]], i32 0, i32 1
@@ -2014,6 +2014,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [5 x i64], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
@@ -2073,7 +2074,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [5 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK3-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2138,6 +2138,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK3-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -2181,7 +2182,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK3-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2236,6 +2236,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK3-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -2269,7 +2270,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP13]], align 4
 // CHECK3-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK3-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_parallel_for_codegen.cpp b/clang/test/OpenMP/target_parallel_for_codegen.cpp
index 2d14c963b0577..405eaadc4f264 100644
--- a/clang/test/OpenMP/target_parallel_for_codegen.cpp
+++ b/clang/test/OpenMP/target_parallel_for_codegen.cpp
@@ -324,6 +324,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[C:%.*]] = alloca [5 x [10 x double]], align 8
 // CHECK1-NEXT:    [[__VLA_EXPR1:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[D:%.*]] = alloca [[STRUCT_TT:%.*]], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[K:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[K_CASTED:%.*]] = alloca i64, align 8
@@ -340,6 +341,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS5:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS7:%.*]] = alloca [2 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[A_CASTED11:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
@@ -347,6 +349,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS15:%.*]] = alloca [10 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS16:%.*]] = alloca [10 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [10 x i64], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS17:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB2:[0-9]+]])
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
@@ -362,7 +365,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP6:%.*]] = mul nuw i64 5, [[TMP5]]
 // CHECK1-NEXT:    [[VLA1:%.*]] = alloca double, i64 [[TMP6]], align 8
 // CHECK1-NEXT:    store i64 [[TMP5]], ptr [[__VLA_EXPR1]], align 8
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -484,7 +486,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP70]], align 8
 // CHECK1-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS5]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP72:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS6]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP73:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP73]], align 4
 // CHECK1-NEXT:    [[TMP74:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 1
@@ -606,7 +607,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP129:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_BASEPTRS14]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP130:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_PTRS15]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP131:%.*]] = getelementptr inbounds [10 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS17:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP132:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS17]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP132]], align 4
 // CHECK1-NEXT:    [[TMP133:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS17]], i32 0, i32 1
@@ -1012,10 +1012,10 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR1_I:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR2_I:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR3_I:%.*]] = alloca ptr, align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[AA_CASTED_I:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[LIN_CASTED_I:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[A_CASTED_I:%.*]] = alloca i64, align 8
-// CHECK1-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTADDR:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTADDR1:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[DOTADDR]], align 4
@@ -1414,6 +1414,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [5 x i64], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
@@ -1473,7 +1474,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [5 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK1-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1538,6 +1538,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK1-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -1581,7 +1582,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP18]], align 8
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK1-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1636,6 +1636,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK1-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -1669,7 +1670,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP13]], align 8
 // CHECK1-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2000,6 +2000,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[C:%.*]] = alloca [5 x [10 x double]], align 8
 // CHECK3-NEXT:    [[__VLA_EXPR1:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[D:%.*]] = alloca [[STRUCT_TT:%.*]], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[K:%.*]] = alloca i64, align 8
 // CHECK3-NEXT:    [[A_CASTED:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[LIN:%.*]] = alloca i32, align 4
@@ -2015,6 +2016,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS5:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS7:%.*]] = alloca [2 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[A_CASTED11:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i32, align 4
@@ -2022,6 +2024,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS15:%.*]] = alloca [10 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS16:%.*]] = alloca [10 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [10 x i64], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS17:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB2:[0-9]+]])
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
@@ -2035,7 +2038,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP4:%.*]] = mul nuw i32 5, [[TMP3]]
 // CHECK3-NEXT:    [[VLA1:%.*]] = alloca double, i32 [[TMP4]], align 8
 // CHECK3-NEXT:    store i32 [[TMP3]], ptr [[__VLA_EXPR1]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2154,7 +2156,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP66]], align 4
 // CHECK3-NEXT:    [[TMP67:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS5]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS6]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP69]], align 4
 // CHECK3-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 1
@@ -2278,7 +2279,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP127:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_BASEPTRS14]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP128:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_PTRS15]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP129:%.*]] = getelementptr inbounds [10 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS17:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP130:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS17]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP130]], align 4
 // CHECK3-NEXT:    [[TMP131:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS17]], i32 0, i32 1
@@ -2682,10 +2682,10 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR1_I:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR2_I:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR3_I:%.*]] = alloca ptr, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[AA_CASTED_I:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[LIN_CASTED_I:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[A_CASTED_I:%.*]] = alloca i32, align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTADDR:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTADDR1:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[DOTADDR]], align 4
@@ -3084,6 +3084,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [5 x i64], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
@@ -3143,7 +3144,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [5 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK3-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3208,6 +3208,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK3-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -3251,7 +3252,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK3-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3306,6 +3306,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK3-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -3339,7 +3340,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP13]], align 4
 // CHECK3-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK3-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5200,6 +5200,7 @@ int bar(int n){
 // CHECK17-NEXT:    [[C:%.*]] = alloca [5 x [10 x double]], align 8
 // CHECK17-NEXT:    [[__VLA_EXPR1:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[D:%.*]] = alloca [[STRUCT_TT:%.*]], align 8
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[K:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[K_CASTED:%.*]] = alloca i64, align 8
@@ -5216,6 +5217,7 @@ int bar(int n){
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS5:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS7:%.*]] = alloca [2 x ptr], align 8
+// CHECK17-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[A_CASTED11:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
@@ -5223,6 +5225,7 @@ int bar(int n){
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS15:%.*]] = alloca [10 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS16:%.*]] = alloca [10 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [10 x i64], align 8
+// CHECK17-NEXT:    [[KERNEL_ARGS17:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB2:[0-9]+]])
 // CHECK17-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK17-NEXT:    store i32 0, ptr [[A]], align 4
@@ -5238,7 +5241,6 @@ int bar(int n){
 // CHECK17-NEXT:    [[TMP6:%.*]] = mul nuw i64 5, [[TMP5]]
 // CHECK17-NEXT:    [[VLA1:%.*]] = alloca double, i64 [[TMP6]], align 8
 // CHECK17-NEXT:    store i64 [[TMP5]], ptr [[__VLA_EXPR1]], align 8
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK17-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5360,7 +5362,6 @@ int bar(int n){
 // CHECK17-NEXT:    store ptr null, ptr [[TMP70]], align 8
 // CHECK17-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS5]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP72:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS6]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP73:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP73]], align 4
 // CHECK17-NEXT:    [[TMP74:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 1
@@ -5482,7 +5483,6 @@ int bar(int n){
 // CHECK17-NEXT:    [[TMP129:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_BASEPTRS14]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP130:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_PTRS15]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP131:%.*]] = getelementptr inbounds [10 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS17:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP132:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS17]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP132]], align 4
 // CHECK17-NEXT:    [[TMP133:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS17]], i32 0, i32 1
@@ -5888,10 +5888,10 @@ int bar(int n){
 // CHECK17-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR1_I:%.*]] = alloca ptr, align 8
 // CHECK17-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR2_I:%.*]] = alloca ptr, align 8
 // CHECK17-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR3_I:%.*]] = alloca ptr, align 8
+// CHECK17-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[AA_CASTED_I:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[LIN_CASTED_I:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[A_CASTED_I:%.*]] = alloca i64, align 8
-// CHECK17-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[DOTADDR:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTADDR1:%.*]] = alloca ptr, align 8
 // CHECK17-NEXT:    store i32 [[TMP0]], ptr [[DOTADDR]], align 4
@@ -6290,6 +6290,7 @@ int bar(int n){
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [5 x i64], align 8
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK17-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK17-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
@@ -6349,7 +6350,6 @@ int bar(int n){
 // CHECK17-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [5 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK17-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -6414,6 +6414,7 @@ int bar(int n){
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK17-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK17-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -6457,7 +6458,6 @@ int bar(int n){
 // CHECK17-NEXT:    store ptr null, ptr [[TMP18]], align 8
 // CHECK17-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK17-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -6512,6 +6512,7 @@ int bar(int n){
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK17-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK17-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -6545,7 +6546,6 @@ int bar(int n){
 // CHECK17-NEXT:    store ptr null, ptr [[TMP13]], align 8
 // CHECK17-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK17-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -6876,6 +6876,7 @@ int bar(int n){
 // CHECK19-NEXT:    [[C:%.*]] = alloca [5 x [10 x double]], align 8
 // CHECK19-NEXT:    [[__VLA_EXPR1:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[D:%.*]] = alloca [[STRUCT_TT:%.*]], align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[K:%.*]] = alloca i64, align 8
 // CHECK19-NEXT:    [[A_CASTED:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[LIN:%.*]] = alloca i32, align 4
@@ -6891,6 +6892,7 @@ int bar(int n){
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS5:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS7:%.*]] = alloca [2 x ptr], align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[A_CASTED11:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i32, align 4
@@ -6898,6 +6900,7 @@ int bar(int n){
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS15:%.*]] = alloca [10 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS16:%.*]] = alloca [10 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [10 x i64], align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS17:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB2:[0-9]+]])
 // CHECK19-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK19-NEXT:    store i32 0, ptr [[A]], align 4
@@ -6911,7 +6914,6 @@ int bar(int n){
 // CHECK19-NEXT:    [[TMP4:%.*]] = mul nuw i32 5, [[TMP3]]
 // CHECK19-NEXT:    [[VLA1:%.*]] = alloca double, i32 [[TMP4]], align 8
 // CHECK19-NEXT:    store i32 [[TMP3]], ptr [[__VLA_EXPR1]], align 4
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK19-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -7030,7 +7032,6 @@ int bar(int n){
 // CHECK19-NEXT:    store ptr null, ptr [[TMP66]], align 4
 // CHECK19-NEXT:    [[TMP67:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS5]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS6]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP69]], align 4
 // CHECK19-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 1
@@ -7154,7 +7155,6 @@ int bar(int n){
 // CHECK19-NEXT:    [[TMP127:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_BASEPTRS14]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP128:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_PTRS15]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP129:%.*]] = getelementptr inbounds [10 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS17:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP130:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS17]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP130]], align 4
 // CHECK19-NEXT:    [[TMP131:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS17]], i32 0, i32 1
@@ -7558,10 +7558,10 @@ int bar(int n){
 // CHECK19-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR1_I:%.*]] = alloca ptr, align 4
 // CHECK19-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR2_I:%.*]] = alloca ptr, align 4
 // CHECK19-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR3_I:%.*]] = alloca ptr, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[AA_CASTED_I:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[LIN_CASTED_I:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[A_CASTED_I:%.*]] = alloca i32, align 4
-// CHECK19-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[DOTADDR:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTADDR1:%.*]] = alloca ptr, align 4
 // CHECK19-NEXT:    store i32 [[TMP0]], ptr [[DOTADDR]], align 4
@@ -7960,6 +7960,7 @@ int bar(int n){
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [5 x i64], align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK19-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK19-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
@@ -8019,7 +8020,6 @@ int bar(int n){
 // CHECK19-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [5 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK19-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -8084,6 +8084,7 @@ int bar(int n){
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK19-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK19-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -8127,7 +8128,6 @@ int bar(int n){
 // CHECK19-NEXT:    store ptr null, ptr [[TMP18]], align 4
 // CHECK19-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK19-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -8182,6 +8182,7 @@ int bar(int n){
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK19-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK19-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -8215,7 +8216,6 @@ int bar(int n){
 // CHECK19-NEXT:    store ptr null, ptr [[TMP13]], align 4
 // CHECK19-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK19-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_parallel_for_simd_codegen.cpp b/clang/test/OpenMP/target_parallel_for_simd_codegen.cpp
index 387c93cd35647..af3a6d8e4db24 100644
--- a/clang/test/OpenMP/target_parallel_for_simd_codegen.cpp
+++ b/clang/test/OpenMP/target_parallel_for_simd_codegen.cpp
@@ -322,11 +322,13 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[A_CASTED3:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[AA_CASTED4:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS5:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS7:%.*]] = alloca [2 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[A_CASTED11:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
@@ -334,6 +336,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS15:%.*]] = alloca [10 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS16:%.*]] = alloca [10 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [10 x i64], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS17:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB2:[0-9]+]])
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
@@ -391,7 +394,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP28]], align 8
 // CHECK1-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP31:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP31]], align 4
 // CHECK1-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -449,7 +451,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP56]], align 8
 // CHECK1-NEXT:    [[TMP57:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS5]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS6]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP59]], align 4
 // CHECK1-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 1
@@ -571,7 +572,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP115:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_BASEPTRS14]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP116:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_PTRS15]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP117:%.*]] = getelementptr inbounds [10 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS17:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP118:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS17]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP118]], align 4
 // CHECK1-NEXT:    [[TMP119:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS17]], i32 0, i32 1
@@ -1342,6 +1342,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [5 x i64], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
@@ -1401,7 +1402,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [5 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK1-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1466,6 +1466,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK1-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -1509,7 +1510,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP18]], align 8
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK1-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1564,6 +1564,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK1-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -1597,7 +1598,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP13]], align 8
 // CHECK1-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1952,11 +1952,13 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[A_CASTED3:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[AA_CASTED4:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS5:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS7:%.*]] = alloca [2 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[A_CASTED11:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i32, align 4
@@ -1964,6 +1966,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS15:%.*]] = alloca [10 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS16:%.*]] = alloca [10 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [10 x i64], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS17:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB2:[0-9]+]])
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
@@ -2016,7 +2019,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP24]], align 4
 // CHECK3-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP27]], align 4
 // CHECK3-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2074,7 +2076,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP52]], align 4
 // CHECK3-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS5]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP54:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS6]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP55:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP55]], align 4
 // CHECK3-NEXT:    [[TMP56:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 1
@@ -2198,7 +2199,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP113:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_BASEPTRS14]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP114:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_PTRS15]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP115:%.*]] = getelementptr inbounds [10 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS17:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP116:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS17]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP116]], align 4
 // CHECK3-NEXT:    [[TMP117:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS17]], i32 0, i32 1
@@ -2967,6 +2967,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [5 x i64], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
@@ -3026,7 +3027,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [5 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK3-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3091,6 +3091,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK3-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -3134,7 +3135,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK3-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3189,6 +3189,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK3-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -3222,7 +3223,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP13]], align 4
 // CHECK3-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK3-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3578,11 +3578,13 @@ int bar(int n){
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[A_CASTED3:%.*]] = alloca i64, align 8
 // CHECK5-NEXT:    [[AA_CASTED4:%.*]] = alloca i64, align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS5:%.*]] = alloca [2 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [2 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS7:%.*]] = alloca [2 x ptr], align 8
+// CHECK5-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK5-NEXT:    [[A_CASTED11:%.*]] = alloca i64, align 8
 // CHECK5-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
@@ -3590,6 +3592,7 @@ int bar(int n){
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS15:%.*]] = alloca [10 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS16:%.*]] = alloca [10 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [10 x i64], align 8
+// CHECK5-NEXT:    [[KERNEL_ARGS17:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB2:[0-9]+]])
 // CHECK5-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK5-NEXT:    store i32 0, ptr [[A]], align 4
@@ -3647,7 +3650,6 @@ int bar(int n){
 // CHECK5-NEXT:    store ptr null, ptr [[TMP28]], align 8
 // CHECK5-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP31:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP31]], align 4
 // CHECK5-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3705,7 +3707,6 @@ int bar(int n){
 // CHECK5-NEXT:    store ptr null, ptr [[TMP56]], align 8
 // CHECK5-NEXT:    [[TMP57:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS5]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS6]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP59]], align 4
 // CHECK5-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 1
@@ -3827,7 +3828,6 @@ int bar(int n){
 // CHECK5-NEXT:    [[TMP115:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_BASEPTRS14]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP116:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_PTRS15]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP117:%.*]] = getelementptr inbounds [10 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS17:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP118:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS17]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP118]], align 4
 // CHECK5-NEXT:    [[TMP119:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS17]], i32 0, i32 1
@@ -4600,6 +4600,7 @@ int bar(int n){
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [6 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [6 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [6 x i64], align 8
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK5-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK5-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
@@ -4678,7 +4679,6 @@ int bar(int n){
 // CHECK5-NEXT:    [[TOBOOL4:%.*]] = trunc i8 [[TMP35]] to i1
 // CHECK5-NEXT:    [[TMP36:%.*]] = select i1 [[TOBOOL4]], i32 0, i32 1
 // CHECK5-NEXT:    [[TMP37:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP36]], 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP38]], align 4
 // CHECK5-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4743,6 +4743,7 @@ int bar(int n){
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK5-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK5-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -4786,7 +4787,6 @@ int bar(int n){
 // CHECK5-NEXT:    store ptr null, ptr [[TMP18]], align 8
 // CHECK5-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK5-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4841,6 +4841,7 @@ int bar(int n){
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK5-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK5-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -4874,7 +4875,6 @@ int bar(int n){
 // CHECK5-NEXT:    store ptr null, ptr [[TMP13]], align 8
 // CHECK5-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK5-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5315,11 +5315,13 @@ int bar(int n){
 // CHECK7-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[A_CASTED3:%.*]] = alloca i32, align 4
 // CHECK7-NEXT:    [[AA_CASTED4:%.*]] = alloca i32, align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_BASEPTRS5:%.*]] = alloca [2 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [2 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS7:%.*]] = alloca [2 x ptr], align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK7-NEXT:    [[A_CASTED11:%.*]] = alloca i32, align 4
 // CHECK7-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i32, align 4
@@ -5327,6 +5329,7 @@ int bar(int n){
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS15:%.*]] = alloca [10 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS16:%.*]] = alloca [10 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [10 x i64], align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS17:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB2:[0-9]+]])
 // CHECK7-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK7-NEXT:    store i32 0, ptr [[A]], align 4
@@ -5379,7 +5382,6 @@ int bar(int n){
 // CHECK7-NEXT:    store ptr null, ptr [[TMP24]], align 4
 // CHECK7-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP27]], align 4
 // CHECK7-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5437,7 +5439,6 @@ int bar(int n){
 // CHECK7-NEXT:    store ptr null, ptr [[TMP52]], align 4
 // CHECK7-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS5]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP54:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS6]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[TMP55:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP55]], align 4
 // CHECK7-NEXT:    [[TMP56:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 1
@@ -5561,7 +5562,6 @@ int bar(int n){
 // CHECK7-NEXT:    [[TMP113:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_BASEPTRS14]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP114:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_PTRS15]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP115:%.*]] = getelementptr inbounds [10 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS17:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[TMP116:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS17]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP116]], align 4
 // CHECK7-NEXT:    [[TMP117:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS17]], i32 0, i32 1
@@ -6332,6 +6332,7 @@ int bar(int n){
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [6 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [6 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [6 x i64], align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK7-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK7-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
@@ -6410,7 +6411,6 @@ int bar(int n){
 // CHECK7-NEXT:    [[TOBOOL4:%.*]] = trunc i8 [[TMP35]] to i1
 // CHECK7-NEXT:    [[TMP36:%.*]] = select i1 [[TOBOOL4]], i32 0, i32 1
 // CHECK7-NEXT:    [[TMP37:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP36]], 0
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP38]], align 4
 // CHECK7-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -6475,6 +6475,7 @@ int bar(int n){
 // CHECK7-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK7-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK7-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -6518,7 +6519,6 @@ int bar(int n){
 // CHECK7-NEXT:    store ptr null, ptr [[TMP18]], align 4
 // CHECK7-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK7-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -6573,6 +6573,7 @@ int bar(int n){
 // CHECK7-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK7-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK7-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -6606,7 +6607,6 @@ int bar(int n){
 // CHECK7-NEXT:    store ptr null, ptr [[TMP13]], align 4
 // CHECK7-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK7-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_parallel_if_codegen.cpp b/clang/test/OpenMP/target_parallel_if_codegen.cpp
index 8796862623c4e..e1fc1d82aff3f 100644
--- a/clang/test/OpenMP/target_parallel_if_codegen.cpp
+++ b/clang/test/OpenMP/target_parallel_if_codegen.cpp
@@ -221,11 +221,13 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_4:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED8:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS12:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS13:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS14:%.*]] = alloca [2 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS16:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
@@ -267,7 +269,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TOBOOL3:%.*]] = trunc i8 [[TMP16]] to i1
 // CHECK1-NEXT:    [[TMP17:%.*]] = select i1 [[TOBOOL3]], i32 0, i32 1
 // CHECK1-NEXT:    [[TMP18:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP17]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP19]], align 4
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -333,7 +334,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TOBOOL15:%.*]] = trunc i8 [[TMP46]] to i1
 // CHECK1-NEXT:    [[TMP47:%.*]] = select i1 [[TOBOOL15]], i32 0, i32 1
 // CHECK1-NEXT:    [[TMP48:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP47]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS16:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS16]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP49]], align 4
 // CHECK1-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS16]], i32 0, i32 1
@@ -387,6 +387,8 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[TMP0]], 1
@@ -413,7 +415,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TOBOOL3:%.*]] = trunc i8 [[TMP9]] to i1
 // CHECK1-NEXT:    [[TMP10:%.*]] = select i1 [[TOBOOL3]], i32 0, i32 1
 // CHECK1-NEXT:    [[TMP11:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP10]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP12]], align 4
 // CHECK1-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -457,7 +458,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[CMP4:%.*]] = icmp sgt i32 [[SUB]], 2
 // CHECK1-NEXT:    br i1 [[CMP4]], label [[OMP_IF_THEN5:%.*]], label [[OMP_IF_ELSE9:%.*]]
 // CHECK1:       omp_if.then5:
-// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP28]], align 4
 // CHECK1-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -510,12 +510,14 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[B:%.*]] = alloca i16, align 2
 // CHECK1-NEXT:    [[A_CASTED1:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[B_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS2:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS3:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS4:%.*]] = alloca [2 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr [[A]], align 4
@@ -529,7 +531,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -584,7 +585,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP31]], align 8
 // CHECK1-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS2]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS3]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP34]], align 4
 // CHECK1-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -890,11 +890,13 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_4:%.*]] = alloca i8, align 1
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR__CASTED8:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS12:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS13:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS14:%.*]] = alloca [2 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS16:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
@@ -936,7 +938,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TOBOOL3:%.*]] = trunc i8 [[TMP16]] to i1
 // CHECK3-NEXT:    [[TMP17:%.*]] = select i1 [[TOBOOL3]], i32 0, i32 1
 // CHECK3-NEXT:    [[TMP18:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP17]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP19]], align 4
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1002,7 +1003,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TOBOOL15:%.*]] = trunc i8 [[TMP46]] to i1
 // CHECK3-NEXT:    [[TMP47:%.*]] = select i1 [[TOBOOL15]], i32 0, i32 1
 // CHECK3-NEXT:    [[TMP48:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP47]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS16:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS16]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP49]], align 4
 // CHECK3-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS16]], i32 0, i32 1
@@ -1056,6 +1056,8 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[TMP0]], 1
@@ -1082,7 +1084,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TOBOOL3:%.*]] = trunc i8 [[TMP9]] to i1
 // CHECK3-NEXT:    [[TMP10:%.*]] = select i1 [[TOBOOL3]], i32 0, i32 1
 // CHECK3-NEXT:    [[TMP11:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP10]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP12]], align 4
 // CHECK3-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1126,7 +1127,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[CMP4:%.*]] = icmp sgt i32 [[SUB]], 2
 // CHECK3-NEXT:    br i1 [[CMP4]], label [[OMP_IF_THEN5:%.*]], label [[OMP_IF_ELSE9:%.*]]
 // CHECK3:       omp_if.then5:
-// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP28]], align 4
 // CHECK3-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -1179,12 +1179,14 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[B:%.*]] = alloca i16, align 2
 // CHECK3-NEXT:    [[A_CASTED1:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[B_CASTED:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS2:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS3:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS4:%.*]] = alloca [2 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr [[A]], align 4
@@ -1198,7 +1200,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP4]], align 4
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1253,7 +1254,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP31]], align 4
 // CHECK3-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS2]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS3]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP34]], align 4
 // CHECK3-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_parallel_num_threads_codegen.cpp b/clang/test/OpenMP/target_parallel_num_threads_codegen.cpp
index f8be5d02d39c8..5d9f0af3dbed1 100644
--- a/clang/test/OpenMP/target_parallel_num_threads_codegen.cpp
+++ b/clang/test/OpenMP/target_parallel_num_threads_codegen.cpp
@@ -228,9 +228,11 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
@@ -268,7 +270,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP17:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
 // CHECK1-NEXT:    [[TMP18:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP17]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP19]], align 4
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -311,7 +312,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP36]], align 8
 // CHECK1-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP39]], align 4
 // CHECK1-NEXT:    [[TMP40:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -360,11 +360,13 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED2:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[DOTCAPTURE_EXPR_]], align 4
@@ -381,7 +383,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP8:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
 // CHECK1-NEXT:    [[TMP9:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP8]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP10]], align 4
 // CHECK1-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -431,7 +432,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP33:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK1-NEXT:    [[TMP34:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP33]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP35]], align 4
 // CHECK1-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -475,6 +475,7 @@ int bar(int n){
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[N_ADDR:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[A:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[B:%.*]] = alloca i16, align 2
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i16, align 2
 // CHECK1-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
@@ -483,9 +484,9 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS1:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -554,7 +555,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP33:%.*]] = load i16, ptr [[DOTCAPTURE_EXPR_]], align 2
 // CHECK1-NEXT:    [[TMP34:%.*]] = zext i16 [[TMP33]] to i32
 // CHECK1-NEXT:    [[TMP35:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP34]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS1:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS1]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP36]], align 4
 // CHECK1-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS1]], i32 0, i32 1
@@ -813,9 +813,11 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
@@ -853,7 +855,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP17:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
 // CHECK3-NEXT:    [[TMP18:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP17]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP19]], align 4
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -896,7 +897,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP36]], align 4
 // CHECK3-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP39]], align 4
 // CHECK3-NEXT:    [[TMP40:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -945,11 +945,13 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR__CASTED2:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[DOTCAPTURE_EXPR_]], align 4
@@ -966,7 +968,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP8:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
 // CHECK3-NEXT:    [[TMP9:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP8]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP10]], align 4
 // CHECK3-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1016,7 +1017,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP33:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK3-NEXT:    [[TMP34:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP33]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP35]], align 4
 // CHECK3-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -1060,6 +1060,7 @@ int bar(int n){
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[N_ADDR:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[A:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[B:%.*]] = alloca i16, align 2
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i16, align 2
 // CHECK3-NEXT:    [[A_CASTED:%.*]] = alloca i32, align 4
@@ -1068,9 +1069,9 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS1:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1139,7 +1140,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP33:%.*]] = load i16, ptr [[DOTCAPTURE_EXPR_]], align 2
 // CHECK3-NEXT:    [[TMP34:%.*]] = zext i16 [[TMP33]] to i32
 // CHECK3-NEXT:    [[TMP35:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP34]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS1:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS1]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP36]], align 4
 // CHECK3-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS1]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_task_affinity_codegen.cpp b/clang/test/OpenMP/target_task_affinity_codegen.cpp
index c0ce4ceacd842..3e5bd7cf803ed 100644
--- a/clang/test/OpenMP/target_task_affinity_codegen.cpp
+++ b/clang/test/OpenMP/target_task_affinity_codegen.cpp
@@ -72,6 +72,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS6:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS7:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS8:%.*]] = alloca [2 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[A]], align 8
 // CHECK1-NEXT:    [[TMP1:%.*]] = load ptr, ptr [[A]], align 8
@@ -116,7 +117,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP23]], align 8
 // CHECK1-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS6]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS7]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP26]], align 4
 // CHECK1-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -302,6 +302,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS6:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS7:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS8:%.*]] = alloca [2 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[A]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = load ptr, ptr [[A]], align 4
@@ -346,7 +347,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP23]], align 4
 // CHECK3-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS6]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS7]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP26]], align 4
 // CHECK3-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_codegen.cpp b/clang/test/OpenMP/target_teams_codegen.cpp
index bad0795e5c965..c18294e1f007d 100644
--- a/clang/test/OpenMP/target_teams_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_codegen.cpp
@@ -328,25 +328,30 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS5:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS7:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[A_CASTED8:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[AA_CASTED9:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS11:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS12:%.*]] = alloca [2 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS13:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[A_CASTED16:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [9 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [9 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS21:%.*]] = alloca [9 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [9 x i64], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS22:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[NN:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[NN_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS27:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS28:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS29:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[NN_CASTED33:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS34:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS35:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS36:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS37:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB1:[0-9]+]])
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
@@ -435,7 +440,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP49]], align 8
 // CHECK1-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS5]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS6]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP52]], align 4
 // CHECK1-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -493,7 +497,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP77]], align 8
 // CHECK1-NEXT:    [[TMP78:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS10]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP79:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS11]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS13:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP80:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS13]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP80]], align 4
 // CHECK1-NEXT:    [[TMP81:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS13]], i32 0, i32 1
@@ -604,7 +607,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP130:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_BASEPTRS19]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP131:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_PTRS20]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP132:%.*]] = getelementptr inbounds [9 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS22:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP133:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS22]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP133]], align 4
 // CHECK1-NEXT:    [[TMP134:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS22]], i32 0, i32 1
@@ -655,7 +657,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP152]], align 8
 // CHECK1-NEXT:    [[TMP153:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS27]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP154:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS28]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP155:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP155]], align 4
 // CHECK1-NEXT:    [[TMP156:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 1
@@ -700,7 +701,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP174]], align 8
 // CHECK1-NEXT:    [[TMP175:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS34]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP176:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS35]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS37:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP177:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS37]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP177]], align 4
 // CHECK1-NEXT:    [[TMP178:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS37]], i32 0, i32 1
@@ -815,10 +815,10 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR1_I:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR2_I:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR3_I:%.*]] = alloca ptr, align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[AA_CASTED_I:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED_I:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED4_I:%.*]] = alloca i64, align 8
-// CHECK1-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTADDR:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTADDR1:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[DOTADDR]], align 4
@@ -1195,6 +1195,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store ptr [[F]], ptr [[F_ADDR]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
@@ -1207,7 +1208,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1313,6 +1313,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [5 x i64], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
@@ -1372,7 +1373,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [5 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK1-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1437,6 +1437,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK1-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -1480,7 +1481,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP18]], align 8
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK1-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1535,6 +1535,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK1-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -1568,7 +1569,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP13]], align 8
 // CHECK1-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1822,25 +1822,30 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS5:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS7:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[A_CASTED8:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[AA_CASTED9:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS10:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS11:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS12:%.*]] = alloca [2 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS13:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[A_CASTED16:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [9 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [9 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS21:%.*]] = alloca [9 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [9 x i64], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS22:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[NN:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[NN_CASTED:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS27:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS28:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS29:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[NN_CASTED33:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS34:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS35:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS36:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS37:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB1:[0-9]+]])
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
@@ -1927,7 +1932,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP47]], align 4
 // CHECK3-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS5]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS6]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP50]], align 4
 // CHECK3-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1985,7 +1989,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP75]], align 4
 // CHECK3-NEXT:    [[TMP76:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS10]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP77:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS11]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS13:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP78:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS13]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP78]], align 4
 // CHECK3-NEXT:    [[TMP79:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS13]], i32 0, i32 1
@@ -2098,7 +2101,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP130:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_BASEPTRS19]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP131:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_PTRS20]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP132:%.*]] = getelementptr inbounds [9 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS22:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP133:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS22]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP133]], align 4
 // CHECK3-NEXT:    [[TMP134:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS22]], i32 0, i32 1
@@ -2149,7 +2151,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP152]], align 4
 // CHECK3-NEXT:    [[TMP153:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS27]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP154:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS28]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP155:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP155]], align 4
 // CHECK3-NEXT:    [[TMP156:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 1
@@ -2194,7 +2195,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP174]], align 4
 // CHECK3-NEXT:    [[TMP175:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS34]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP176:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS35]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS37:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP177:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS37]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP177]], align 4
 // CHECK3-NEXT:    [[TMP178:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS37]], i32 0, i32 1
@@ -2309,10 +2309,10 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR1_I:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR2_I:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR3_I:%.*]] = alloca ptr, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[AA_CASTED_I:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR__CASTED_I:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR__CASTED4_I:%.*]] = alloca i32, align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTADDR:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTADDR1:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[DOTADDR]], align 4
@@ -2689,6 +2689,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[F]], ptr [[F_ADDR]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
@@ -2700,7 +2701,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP3]], align 4
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP6]], align 4
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2806,6 +2806,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [5 x i64], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
@@ -2865,7 +2866,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [5 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK3-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2930,6 +2930,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK3-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -2973,7 +2974,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK3-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3028,6 +3028,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK3-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -3061,7 +3062,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP13]], align 4
 // CHECK3-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK3-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_codegen.cpp
index a37039588afbd..e8ea162bec009 100644
--- a/clang/test/OpenMP/target_teams_distribute_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_codegen.cpp
@@ -321,12 +321,14 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS7:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[A_CASTED8:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[AA_CASTED9:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS11:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS12:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP13:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_17:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[A_CASTED18:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED19:%.*]] = alloca i64, align 8
@@ -335,6 +337,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS24:%.*]] = alloca [10 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [10 x i64], align 8
 // CHECK1-NEXT:    [[_TMP25:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS26:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB2:[0-9]+]])
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
@@ -423,7 +426,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP49]], align 8
 // CHECK1-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS5]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS6]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP52]], align 4
 // CHECK1-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -481,7 +483,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP77]], align 8
 // CHECK1-NEXT:    [[TMP78:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS10]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP79:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS11]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP80:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP80]], align 4
 // CHECK1-NEXT:    [[TMP81:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 1
@@ -603,7 +604,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP136:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_BASEPTRS22]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP137:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_PTRS23]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP138:%.*]] = getelementptr inbounds [10 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS26:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP139:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS26]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP139]], align 4
 // CHECK1-NEXT:    [[TMP140:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS26]], i32 0, i32 1
@@ -773,10 +773,10 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR1_I:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR2_I:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR3_I:%.*]] = alloca ptr, align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[AA_CASTED_I:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED_I:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED4_I:%.*]] = alloca i64, align 8
-// CHECK1-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTADDR:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTADDR1:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[DOTADDR]], align 4
@@ -1336,6 +1336,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [5 x i64], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
@@ -1395,7 +1396,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [5 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK1-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1465,6 +1465,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK1-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -1532,7 +1533,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP30:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_2]], align 4
 // CHECK1-NEXT:    [[ADD5:%.*]] = add i32 [[TMP30]], 1
 // CHECK1-NEXT:    [[TMP31:%.*]] = zext i32 [[ADD5]] to i64
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP32]], align 4
 // CHECK1-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1588,6 +1588,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK1-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -1621,7 +1622,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP13]], align 8
 // CHECK1-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2066,12 +2066,14 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS7:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[A_CASTED8:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[AA_CASTED9:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS10:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS11:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS12:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP13:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_17:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[A_CASTED18:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR__CASTED19:%.*]] = alloca i32, align 4
@@ -2080,6 +2082,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS24:%.*]] = alloca [10 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [10 x i64], align 4
 // CHECK3-NEXT:    [[_TMP25:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS26:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB2:[0-9]+]])
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
@@ -2166,7 +2169,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP47]], align 4
 // CHECK3-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS5]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS6]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP50]], align 4
 // CHECK3-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2224,7 +2226,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP75]], align 4
 // CHECK3-NEXT:    [[TMP76:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS10]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP77:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS11]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP78:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP78]], align 4
 // CHECK3-NEXT:    [[TMP79:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 1
@@ -2348,7 +2349,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP136:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_BASEPTRS22]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP137:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_PTRS23]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP138:%.*]] = getelementptr inbounds [10 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS26:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP139:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS26]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP139]], align 4
 // CHECK3-NEXT:    [[TMP140:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS26]], i32 0, i32 1
@@ -2518,10 +2518,10 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR1_I:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR2_I:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR3_I:%.*]] = alloca ptr, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[AA_CASTED_I:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR__CASTED_I:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR__CASTED4_I:%.*]] = alloca i32, align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTADDR:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTADDR1:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[DOTADDR]], align 4
@@ -3081,6 +3081,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [5 x i64], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
@@ -3140,7 +3141,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [5 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK3-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3210,6 +3210,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK3-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -3277,7 +3278,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP30:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_2]], align 4
 // CHECK3-NEXT:    [[ADD5:%.*]] = add i32 [[TMP30]], 1
 // CHECK3-NEXT:    [[TMP31:%.*]] = zext i32 [[ADD5]] to i64
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP32]], align 4
 // CHECK3-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3333,6 +3333,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK3-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -3366,7 +3367,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP13]], align 4
 // CHECK3-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK3-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_collapse_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_collapse_codegen.cpp
index 67b2acf3fa485..dcc103b9ce3cc 100644
--- a/clang/test/OpenMP/target_teams_distribute_collapse_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_collapse_codegen.cpp
@@ -111,6 +111,7 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -122,7 +123,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -279,6 +279,7 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -290,7 +291,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -450,6 +450,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i64, align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -526,7 +527,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store i64 [[SUB7]], ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK9-NEXT:    [[TMP35:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i64 [[TMP35]], 1
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP36]], align 4
 // CHECK9-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -753,6 +753,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    store ptr [[A]], ptr [[TMP0]], align 8
@@ -762,7 +763,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK9-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -919,6 +919,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i64, align 8
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -994,7 +995,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store i64 [[SUB7]], ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK11-NEXT:    [[TMP34:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i64 [[TMP34]], 1
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP35]], align 4
 // CHECK11-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1219,6 +1219,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    store ptr [[A]], ptr [[TMP0]], align 4
@@ -1228,7 +1229,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK11-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_dist_schedule_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_dist_schedule_codegen.cpp
index 4c4dab839a365..f4d3de7f2b925 100644
--- a/clang/test/OpenMP/target_teams_distribute_dist_schedule_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_dist_schedule_codegen.cpp
@@ -138,14 +138,17 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -157,7 +160,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -200,7 +202,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK1-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -243,7 +244,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK1-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK1-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -557,14 +557,17 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -576,7 +579,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -619,7 +621,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK3-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -662,7 +663,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK3-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK3-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -974,6 +974,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 8
@@ -982,6 +983,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_18:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[N_CASTED19:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
@@ -992,6 +994,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[_TMP24:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_26:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -1040,7 +1043,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK9-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1112,7 +1114,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP56:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK9-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP56]], 1
 // CHECK9-NEXT:    [[TMP57:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK9-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1195,7 +1196,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP97:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_26]], align 4
 // CHECK9-NEXT:    [[ADD30:%.*]] = add nsw i32 [[TMP97]], 1
 // CHECK9-NEXT:    [[TMP98:%.*]] = zext i32 [[ADD30]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP99:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP99]], align 4
 // CHECK9-NEXT:    [[TMP100:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -1612,14 +1612,17 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    store ptr [[A]], ptr [[TMP0]], align 8
@@ -1629,7 +1632,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK9-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1671,7 +1673,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -1713,7 +1714,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK9-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK9-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -2022,6 +2022,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 4
@@ -2030,6 +2031,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_18:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[N_CASTED19:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i32, align 4
@@ -2040,6 +2042,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[_TMP24:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_26:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -2088,7 +2091,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK11-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2161,7 +2163,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP57:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK11-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP57]], 1
 // CHECK11-NEXT:    [[TMP58:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP59]], align 4
 // CHECK11-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -2245,7 +2246,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP99:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_26]], align 4
 // CHECK11-NEXT:    [[ADD30:%.*]] = add nsw i32 [[TMP99]], 1
 // CHECK11-NEXT:    [[TMP100:%.*]] = zext i32 [[ADD30]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP101:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP101]], align 4
 // CHECK11-NEXT:    [[TMP102:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -2659,14 +2659,17 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    store ptr [[A]], ptr [[TMP0]], align 4
@@ -2676,7 +2679,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK11-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2718,7 +2720,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -2760,7 +2761,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK11-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK11-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_firstprivate_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_firstprivate_codegen.cpp
index c19e2d4d35c5c..cf8c4af9c1fce 100644
--- a/clang/test/OpenMP/target_teams_distribute_firstprivate_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_firstprivate_codegen.cpp
@@ -262,6 +262,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr @t_var, align 4
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[T_VAR_CASTED]], align 4
@@ -301,7 +302,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP18]], align 8
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK1-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -542,6 +542,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -582,7 +583,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP15]], align 8
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1071,6 +1071,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr @t_var, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[T_VAR_CASTED]], align 4
@@ -1110,7 +1111,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK3-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1349,6 +1349,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1389,7 +1390,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP15]], align 4
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_lastprivate_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_lastprivate_codegen.cpp
index 03bf0ec637c62..e3bbe2601dd5d 100644
--- a/clang/test/OpenMP/target_teams_distribute_lastprivate_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_lastprivate_codegen.cpp
@@ -482,6 +482,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store ptr [[G]], ptr [[G1]], align 8
@@ -534,7 +535,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP20]], align 8
 // CHECK9-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -809,6 +809,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK9-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -849,7 +850,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP15]], align 8
 // CHECK9-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK9-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1190,6 +1190,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store ptr [[G]], ptr [[G1]], align 4
@@ -1242,7 +1243,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP20]], align 4
 // CHECK11-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1515,6 +1515,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK11-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1555,7 +1556,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP15]], align 4
 // CHECK11-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK11-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_codegen.cpp
index 1a09e0e93be6e..a9a5c43ed208c 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_codegen.cpp
@@ -584,6 +584,7 @@ int target_teams_fun(int *g){
 // CHECK2-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK2-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i32, align 4
 // CHECK2-NEXT:    [[DOTCAPTURE_EXPR_4:%.*]] = alloca i32, align 4
+// CHECK2-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK2-NEXT:    [[N_CASTED7:%.*]] = alloca i64, align 8
 // CHECK2-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [3 x ptr], align 8
 // CHECK2-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [3 x ptr], align 8
@@ -591,6 +592,7 @@ int target_teams_fun(int *g){
 // CHECK2-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
 // CHECK2-NEXT:    [[DOTCAPTURE_EXPR_12:%.*]] = alloca i32, align 4
 // CHECK2-NEXT:    [[DOTCAPTURE_EXPR_13:%.*]] = alloca i32, align 4
+// CHECK2-NEXT:    [[KERNEL_ARGS18:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK2-NEXT:    store ptr [[G]], ptr [[G_ADDR]], align 8
 // CHECK2-NEXT:    store i32 1000, ptr [[N]], align 4
 // CHECK2-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N]], align 4
@@ -650,7 +652,6 @@ int target_teams_fun(int *g){
 // CHECK2-NEXT:    [[TMP28:%.*]] = zext i32 [[ADD]] to i64
 // CHECK2-NEXT:    [[TMP29:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP23]], 0
 // CHECK2-NEXT:    [[TMP30:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP24]], 0
-// CHECK2-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK2-NEXT:    [[TMP31:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK2-NEXT:    store i32 2, ptr [[TMP31]], align 4
 // CHECK2-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -718,7 +719,6 @@ int target_teams_fun(int *g){
 // CHECK2-NEXT:    [[TMP62:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_13]], align 4
 // CHECK2-NEXT:    [[ADD17:%.*]] = add nsw i32 [[TMP62]], 1
 // CHECK2-NEXT:    [[TMP63:%.*]] = zext i32 [[ADD17]] to i64
-// CHECK2-NEXT:    [[KERNEL_ARGS18:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK2-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS18]], i32 0, i32 0
 // CHECK2-NEXT:    store i32 2, ptr [[TMP64]], align 4
 // CHECK2-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS18]], i32 0, i32 1
@@ -1233,6 +1233,7 @@ int target_teams_fun(int *g){
 // CHECK4-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK4-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i32, align 4
 // CHECK4-NEXT:    [[DOTCAPTURE_EXPR_4:%.*]] = alloca i32, align 4
+// CHECK4-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK4-NEXT:    [[N_CASTED7:%.*]] = alloca i32, align 4
 // CHECK4-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [3 x ptr], align 4
 // CHECK4-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [3 x ptr], align 4
@@ -1240,6 +1241,7 @@ int target_teams_fun(int *g){
 // CHECK4-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
 // CHECK4-NEXT:    [[DOTCAPTURE_EXPR_12:%.*]] = alloca i32, align 4
 // CHECK4-NEXT:    [[DOTCAPTURE_EXPR_13:%.*]] = alloca i32, align 4
+// CHECK4-NEXT:    [[KERNEL_ARGS18:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK4-NEXT:    store ptr [[G]], ptr [[G_ADDR]], align 4
 // CHECK4-NEXT:    store i32 1000, ptr [[N]], align 4
 // CHECK4-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N]], align 4
@@ -1299,7 +1301,6 @@ int target_teams_fun(int *g){
 // CHECK4-NEXT:    [[TMP28:%.*]] = zext i32 [[ADD]] to i64
 // CHECK4-NEXT:    [[TMP29:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP23]], 0
 // CHECK4-NEXT:    [[TMP30:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP24]], 0
-// CHECK4-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK4-NEXT:    [[TMP31:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK4-NEXT:    store i32 2, ptr [[TMP31]], align 4
 // CHECK4-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1367,7 +1368,6 @@ int target_teams_fun(int *g){
 // CHECK4-NEXT:    [[TMP62:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_13]], align 4
 // CHECK4-NEXT:    [[ADD17:%.*]] = add nsw i32 [[TMP62]], 1
 // CHECK4-NEXT:    [[TMP63:%.*]] = zext i32 [[ADD17]] to i64
-// CHECK4-NEXT:    [[KERNEL_ARGS18:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK4-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS18]], i32 0, i32 0
 // CHECK4-NEXT:    store i32 2, ptr [[TMP64]], align 4
 // CHECK4-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS18]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_collapse_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_collapse_codegen.cpp
index a6b2cfba59584..09916a9c62186 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_collapse_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_collapse_codegen.cpp
@@ -116,6 +116,7 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -127,7 +128,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -359,6 +359,7 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -370,7 +371,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -601,6 +601,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i64, align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -677,7 +678,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store i64 [[SUB7]], ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK9-NEXT:    [[TMP35:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i64 [[TMP35]], 1
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP36]], align 4
 // CHECK9-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1030,6 +1030,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    store ptr [[A]], ptr [[TMP0]], align 8
@@ -1039,7 +1040,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK9-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1271,6 +1271,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i64, align 8
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -1346,7 +1347,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store i64 [[SUB7]], ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK11-NEXT:    [[TMP34:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i64 [[TMP34]], 1
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP35]], align 4
 // CHECK11-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1701,6 +1701,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    store ptr [[A]], ptr [[TMP0]], align 4
@@ -1710,7 +1711,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK11-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_dist_schedule_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_dist_schedule_codegen.cpp
index a35a2fd44c554..018e78a3960f6 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_dist_schedule_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_dist_schedule_codegen.cpp
@@ -150,14 +150,17 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -169,7 +172,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -212,7 +214,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK1-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -255,7 +256,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK1-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK1-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -791,14 +791,17 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -810,7 +813,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -853,7 +855,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK3-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -896,7 +897,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK3-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK3-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1419,6 +1419,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 8
@@ -1427,6 +1428,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_18:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[N_CASTED19:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
@@ -1437,6 +1439,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[_TMP24:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_26:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -1486,7 +1489,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK9-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1558,7 +1560,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP56:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK9-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP56]], 1
 // CHECK9-NEXT:    [[TMP57:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK9-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1641,7 +1642,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP97:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_26]], align 4
 // CHECK9-NEXT:    [[ADD30:%.*]] = add nsw i32 [[TMP97]], 1
 // CHECK9-NEXT:    [[TMP98:%.*]] = zext i32 [[ADD30]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP99:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP99]], align 4
 // CHECK9-NEXT:    [[TMP100:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -2384,16 +2384,19 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -2404,7 +2407,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK9-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2446,7 +2448,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -2499,7 +2500,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP48]], align 8
 // CHECK9-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP51]], align 4
 // CHECK9-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -3046,6 +3046,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 4
@@ -3054,6 +3055,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_18:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[N_CASTED19:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i32, align 4
@@ -3064,6 +3066,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[_TMP24:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_26:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -3113,7 +3116,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK11-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3186,7 +3188,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP57:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK11-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP57]], 1
 // CHECK11-NEXT:    [[TMP58:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP59]], align 4
 // CHECK11-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -3270,7 +3271,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP99:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_26]], align 4
 // CHECK11-NEXT:    [[ADD30:%.*]] = add nsw i32 [[TMP99]], 1
 // CHECK11-NEXT:    [[TMP100:%.*]] = zext i32 [[ADD30]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP101:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP101]], align 4
 // CHECK11-NEXT:    [[TMP102:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -3998,16 +3998,19 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -4018,7 +4021,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK11-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4060,7 +4062,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -4113,7 +4114,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP48]], align 4
 // CHECK11-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP51]], align 4
 // CHECK11-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_firstprivate_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_firstprivate_codegen.cpp
index c9eee578f579f..79d13e0f38219 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_firstprivate_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_firstprivate_codegen.cpp
@@ -323,6 +323,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr @t_var, align 4
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[T_VAR_CASTED]], align 4
@@ -362,7 +363,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP18]], align 8
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK1-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -733,6 +733,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -773,7 +774,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP15]], align 8
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1392,6 +1392,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr @t_var, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[T_VAR_CASTED]], align 4
@@ -1431,7 +1432,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK3-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1796,6 +1796,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1836,7 +1837,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP15]], align 4
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_if_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_if_codegen.cpp
index 0cdb8bc0f5b19..460edea1e3ea0 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_if_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_if_codegen.cpp
@@ -109,8 +109,9 @@ int main() {
 // CHECK1-SAME: () #[[ATTR0:[0-9]+]] {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -144,7 +145,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9gtid_testv_l48() #[[ATTR2:[0-9]+]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -466,14 +466,15 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP5:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -533,7 +534,6 @@ int main() {
 // CHECK1-NEXT:    [[TOBOOL4:%.*]] = trunc i8 [[TMP24]] to i1
 // CHECK1-NEXT:    [[TMP25:%.*]] = select i1 [[TOBOOL4]], i32 0, i32 1
 // CHECK1-NEXT:    [[TMP26:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP25]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP27]], align 4
 // CHECK1-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -1025,14 +1025,15 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[ARG_ADDR:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[ARG]], ptr [[ARG_ADDR]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1088,7 +1089,6 @@ int main() {
 // CHECK1-NEXT:    [[TOBOOL3:%.*]] = trunc i8 [[TMP23]] to i1
 // CHECK1-NEXT:    [[TMP24:%.*]] = select i1 [[TOBOOL3]], i32 0, i32 1
 // CHECK1-NEXT:    [[TMP25:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP24]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP26]], align 4
 // CHECK1-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_lastprivate_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_lastprivate_codegen.cpp
index cc3d28be30ce7..986659e8b49ad 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_lastprivate_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_lastprivate_codegen.cpp
@@ -743,6 +743,7 @@ int main() {
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK5-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK5-NEXT:    store ptr [[G]], ptr [[G1]], align 8
@@ -795,7 +796,6 @@ int main() {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP20]], align 8
 // CHECK5-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK5-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1224,6 +1224,7 @@ int main() {
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK5-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK5-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -1264,7 +1265,6 @@ int main() {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP15]], align 8
 // CHECK5-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK5-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1750,6 +1750,7 @@ int main() {
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK7-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK7-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK7-NEXT:    store ptr [[G]], ptr [[G1]], align 4
@@ -1802,7 +1803,6 @@ int main() {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP20]], align 4
 // CHECK7-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK7-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2225,6 +2225,7 @@ int main() {
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK7-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK7-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK7-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -2265,7 +2266,6 @@ int main() {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP15]], align 4
 // CHECK7-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK7-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_private_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_private_codegen.cpp
index 375e79d0736a0..6d29d29e626a2 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_private_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_private_codegen.cpp
@@ -298,8 +298,8 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -560,6 +560,7 @@ int main() {
 // CHECK1-NEXT:    [[VAR:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -569,7 +570,6 @@ int main() {
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef signext 2)
 // CHECK1-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 8
 // CHECK1-NEXT:    store ptr undef, ptr [[_TMP1]], align 8
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1038,8 +1038,8 @@ int main() {
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1294,6 +1294,7 @@ int main() {
 // CHECK3-NEXT:    [[VAR:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1303,7 +1304,6 @@ int main() {
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef 2)
 // CHECK3-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 4
 // CHECK3-NEXT:    store ptr undef, ptr [[_TMP1]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_proc_bind_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_proc_bind_codegen.cpp
index f1bc5360d465b..54c7d23a6442d 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_proc_bind_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_proc_bind_codegen.cpp
@@ -53,9 +53,10 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -89,7 +90,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}_main_l36() #[[ATTR2:[0-9]+]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_reduction_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_reduction_codegen.cpp
index 2dc01b18aac20..68b3d7fd524b6 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_reduction_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_reduction_codegen.cpp
@@ -98,6 +98,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    store ptr @_ZZ4mainE5sivar, ptr [[TMP0]], align 8
@@ -107,7 +108,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -388,6 +388,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -398,7 +399,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -684,6 +684,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    store ptr @_ZZ4mainE5sivar, ptr [[TMP0]], align 4
@@ -693,7 +694,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -970,6 +970,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -980,7 +981,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_schedule_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_schedule_codegen.cpp
index 7f663d81ffe59..15fc87d0961d9 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_schedule_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_schedule_codegen.cpp
@@ -210,22 +210,27 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS21:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP22:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS27:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS28:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS29:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP30:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -237,7 +242,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -280,7 +284,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK1-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -323,7 +326,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK1-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK1-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -366,7 +368,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP62]], align 8
 // CHECK1-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS19]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS20]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP65]], align 4
 // CHECK1-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 1
@@ -409,7 +410,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP82]], align 8
 // CHECK1-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS27]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS28]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP85]], align 4
 // CHECK1-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -1244,22 +1244,27 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS21:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP22:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS27:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS28:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS29:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP30:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -1271,7 +1276,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1314,7 +1318,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK3-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -1357,7 +1360,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK3-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK3-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1400,7 +1402,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP62]], align 4
 // CHECK3-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS19]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS20]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP65]], align 4
 // CHECK3-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 1
@@ -1443,7 +1444,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP82]], align 4
 // CHECK3-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS27]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS28]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP85]], align 4
 // CHECK3-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -2251,22 +2251,27 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS21:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP22:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS27:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS28:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS29:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP30:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK5-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK5-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -2278,7 +2283,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK5-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK5-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2321,7 +2325,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK5-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK5-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -2364,7 +2367,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK5-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK5-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -2407,7 +2409,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP62]], align 8
 // CHECK5-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS19]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS20]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP65]], align 4
 // CHECK5-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 1
@@ -2450,7 +2451,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP82]], align 8
 // CHECK5-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS27]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS28]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP85]], align 4
 // CHECK5-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -3285,22 +3285,27 @@ int main (int argc, char **argv) {
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS21:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[_TMP22:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[DOTOFFLOAD_BASEPTRS27:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS28:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS29:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[_TMP30:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK7-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK7-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -3312,7 +3317,6 @@ int main (int argc, char **argv) {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK7-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK7-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3355,7 +3359,6 @@ int main (int argc, char **argv) {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK7-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK7-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -3398,7 +3401,6 @@ int main (int argc, char **argv) {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK7-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK7-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -3441,7 +3443,6 @@ int main (int argc, char **argv) {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP62]], align 4
 // CHECK7-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS19]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS20]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP65]], align 4
 // CHECK7-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 1
@@ -3484,7 +3485,6 @@ int main (int argc, char **argv) {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP82]], align 4
 // CHECK7-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS27]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS28]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP85]], align 4
 // CHECK7-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -4294,6 +4294,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 8
@@ -4302,6 +4303,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_18:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[N_CASTED19:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
@@ -4312,6 +4314,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[_TMP24:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_26:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[N_CASTED34:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS35:%.*]] = alloca [3 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS36:%.*]] = alloca [3 x ptr], align 8
@@ -4320,6 +4323,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[_TMP39:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_40:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_41:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS46:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_49:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[N_CASTED50:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR__CASTED51:%.*]] = alloca i64, align 8
@@ -4330,6 +4334,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[_TMP56:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_57:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_58:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS63:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK13-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK13-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -4379,7 +4384,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK13-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK13-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK13-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4451,7 +4455,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP56:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK13-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP56]], 1
 // CHECK13-NEXT:    [[TMP57:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK13-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -4534,7 +4537,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP97:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_26]], align 4
 // CHECK13-NEXT:    [[ADD30:%.*]] = add nsw i32 [[TMP97]], 1
 // CHECK13-NEXT:    [[TMP98:%.*]] = zext i32 [[ADD30]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP99:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP99]], align 4
 // CHECK13-NEXT:    [[TMP100:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -4606,7 +4608,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP132:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_41]], align 4
 // CHECK13-NEXT:    [[ADD45:%.*]] = add nsw i32 [[TMP132]], 1
 // CHECK13-NEXT:    [[TMP133:%.*]] = zext i32 [[ADD45]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS46:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP134:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS46]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP134]], align 4
 // CHECK13-NEXT:    [[TMP135:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS46]], i32 0, i32 1
@@ -4689,7 +4690,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP173:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_58]], align 4
 // CHECK13-NEXT:    [[ADD62:%.*]] = add nsw i32 [[TMP173]], 1
 // CHECK13-NEXT:    [[TMP174:%.*]] = zext i32 [[ADD62]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS63:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP175:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS63]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP175]], align 4
 // CHECK13-NEXT:    [[TMP176:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS63]], i32 0, i32 1
@@ -5873,26 +5873,31 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_22:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR__CASTED23:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS24:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS25:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS26:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[_TMP27:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK13-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK13-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -5903,7 +5908,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK13-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK13-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5945,7 +5949,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK13-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK13-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -5998,7 +6001,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP48]], align 8
 // CHECK13-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP51]], align 4
 // CHECK13-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -6040,7 +6042,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP68]], align 8
 // CHECK13-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP71]], align 4
 // CHECK13-NEXT:    [[TMP72:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -6093,7 +6094,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP94]], align 8
 // CHECK13-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS24]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS25]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP97:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP97]], align 4
 // CHECK13-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 1
@@ -6952,6 +6952,7 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK15-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 4
@@ -6960,6 +6961,7 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_18:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[N_CASTED19:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i32, align 4
@@ -6970,6 +6972,7 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[_TMP24:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_26:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[N_CASTED34:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS35:%.*]] = alloca [3 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS36:%.*]] = alloca [3 x ptr], align 4
@@ -6978,6 +6981,7 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[_TMP39:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_40:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_41:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS46:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_49:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[N_CASTED50:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR__CASTED51:%.*]] = alloca i32, align 4
@@ -6988,6 +6992,7 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[_TMP56:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_57:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_58:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS63:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK15-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK15-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -7037,7 +7042,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK15-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK15-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK15-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK15-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK15-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -7110,7 +7114,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[TMP57:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK15-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP57]], 1
 // CHECK15-NEXT:    [[TMP58:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK15-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP59]], align 4
 // CHECK15-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -7194,7 +7197,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[TMP99:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_26]], align 4
 // CHECK15-NEXT:    [[ADD30:%.*]] = add nsw i32 [[TMP99]], 1
 // CHECK15-NEXT:    [[TMP100:%.*]] = zext i32 [[ADD30]] to i64
-// CHECK15-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP101:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP101]], align 4
 // CHECK15-NEXT:    [[TMP102:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -7267,7 +7269,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[TMP135:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_41]], align 4
 // CHECK15-NEXT:    [[ADD45:%.*]] = add nsw i32 [[TMP135]], 1
 // CHECK15-NEXT:    [[TMP136:%.*]] = zext i32 [[ADD45]] to i64
-// CHECK15-NEXT:    [[KERNEL_ARGS46:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP137:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS46]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP137]], align 4
 // CHECK15-NEXT:    [[TMP138:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS46]], i32 0, i32 1
@@ -7351,7 +7352,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[TMP177:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_58]], align 4
 // CHECK15-NEXT:    [[ADD62:%.*]] = add nsw i32 [[TMP177]], 1
 // CHECK15-NEXT:    [[TMP178:%.*]] = zext i32 [[ADD62]] to i64
-// CHECK15-NEXT:    [[KERNEL_ARGS63:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP179:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS63]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP179]], align 4
 // CHECK15-NEXT:    [[TMP180:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS63]], i32 0, i32 1
@@ -8510,26 +8510,31 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 4
 // CHECK15-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_22:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR__CASTED23:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS24:%.*]] = alloca [2 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS25:%.*]] = alloca [2 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_MAPPERS26:%.*]] = alloca [2 x ptr], align 4
 // CHECK15-NEXT:    [[_TMP27:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK15-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK15-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -8540,7 +8545,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK15-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK15-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK15-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK15-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK15-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -8582,7 +8586,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK15-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK15-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK15-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK15-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -8635,7 +8638,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    store ptr null, ptr [[TMP48]], align 4
 // CHECK15-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK15-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK15-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP51]], align 4
 // CHECK15-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -8677,7 +8679,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    store ptr null, ptr [[TMP68]], align 4
 // CHECK15-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK15-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK15-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP71]], align 4
 // CHECK15-NEXT:    [[TMP72:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -8730,7 +8731,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    store ptr null, ptr [[TMP94]], align 4
 // CHECK15-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS24]], i32 0, i32 0
 // CHECK15-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS25]], i32 0, i32 0
-// CHECK15-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP97:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP97]], align 4
 // CHECK15-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 1
@@ -9562,6 +9562,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 8
@@ -9570,6 +9571,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_18:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[N_CASTED19:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
@@ -9580,6 +9582,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[_TMP24:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_26:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[N_CASTED34:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS35:%.*]] = alloca [3 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS36:%.*]] = alloca [3 x ptr], align 8
@@ -9588,6 +9591,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[_TMP39:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_40:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_41:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS46:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_49:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[N_CASTED50:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR__CASTED51:%.*]] = alloca i64, align 8
@@ -9598,6 +9602,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[_TMP56:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_57:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_58:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS63:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK17-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK17-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -9647,7 +9652,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK17-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK17-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK17-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -9719,7 +9723,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP56:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK17-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP56]], 1
 // CHECK17-NEXT:    [[TMP57:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK17-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -9802,7 +9805,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP97:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_26]], align 4
 // CHECK17-NEXT:    [[ADD30:%.*]] = add nsw i32 [[TMP97]], 1
 // CHECK17-NEXT:    [[TMP98:%.*]] = zext i32 [[ADD30]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP99:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP99]], align 4
 // CHECK17-NEXT:    [[TMP100:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -9874,7 +9876,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP132:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_41]], align 4
 // CHECK17-NEXT:    [[ADD45:%.*]] = add nsw i32 [[TMP132]], 1
 // CHECK17-NEXT:    [[TMP133:%.*]] = zext i32 [[ADD45]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS46:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP134:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS46]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP134]], align 4
 // CHECK17-NEXT:    [[TMP135:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS46]], i32 0, i32 1
@@ -9957,7 +9958,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP173:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_58]], align 4
 // CHECK17-NEXT:    [[ADD62:%.*]] = add nsw i32 [[TMP173]], 1
 // CHECK17-NEXT:    [[TMP174:%.*]] = zext i32 [[ADD62]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS63:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP175:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS63]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP175]], align 4
 // CHECK17-NEXT:    [[TMP176:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS63]], i32 0, i32 1
@@ -11141,26 +11141,31 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_22:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR__CASTED23:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS24:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS25:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS26:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[_TMP27:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK17-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK17-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -11171,7 +11176,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK17-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK17-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -11213,7 +11217,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK17-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK17-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -11266,7 +11269,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP48]], align 8
 // CHECK17-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP51]], align 4
 // CHECK17-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -11308,7 +11310,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP68]], align 8
 // CHECK17-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP71]], align 4
 // CHECK17-NEXT:    [[TMP72:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -11361,7 +11362,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP94]], align 8
 // CHECK17-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS24]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS25]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP97:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP97]], align 4
 // CHECK17-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 1
@@ -12220,6 +12220,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 4
@@ -12228,6 +12229,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_18:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[N_CASTED19:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i32, align 4
@@ -12238,6 +12240,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[_TMP24:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_26:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[N_CASTED34:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS35:%.*]] = alloca [3 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS36:%.*]] = alloca [3 x ptr], align 4
@@ -12246,6 +12249,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[_TMP39:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_40:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_41:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS46:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_49:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[N_CASTED50:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR__CASTED51:%.*]] = alloca i32, align 4
@@ -12256,6 +12260,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[_TMP56:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_57:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_58:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS63:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK19-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK19-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -12305,7 +12310,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK19-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK19-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK19-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -12378,7 +12382,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP57:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK19-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP57]], 1
 // CHECK19-NEXT:    [[TMP58:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP59]], align 4
 // CHECK19-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -12462,7 +12465,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP99:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_26]], align 4
 // CHECK19-NEXT:    [[ADD30:%.*]] = add nsw i32 [[TMP99]], 1
 // CHECK19-NEXT:    [[TMP100:%.*]] = zext i32 [[ADD30]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP101:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP101]], align 4
 // CHECK19-NEXT:    [[TMP102:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -12535,7 +12537,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP135:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_41]], align 4
 // CHECK19-NEXT:    [[ADD45:%.*]] = add nsw i32 [[TMP135]], 1
 // CHECK19-NEXT:    [[TMP136:%.*]] = zext i32 [[ADD45]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS46:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP137:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS46]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP137]], align 4
 // CHECK19-NEXT:    [[TMP138:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS46]], i32 0, i32 1
@@ -12619,7 +12620,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP177:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_58]], align 4
 // CHECK19-NEXT:    [[ADD62:%.*]] = add nsw i32 [[TMP177]], 1
 // CHECK19-NEXT:    [[TMP178:%.*]] = zext i32 [[ADD62]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS63:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP179:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS63]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP179]], align 4
 // CHECK19-NEXT:    [[TMP180:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS63]], i32 0, i32 1
@@ -13778,26 +13778,31 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_22:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR__CASTED23:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS24:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS25:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS26:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[_TMP27:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK19-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK19-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -13808,7 +13813,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK19-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK19-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -13850,7 +13854,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK19-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK19-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -13903,7 +13906,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP48]], align 4
 // CHECK19-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP51]], align 4
 // CHECK19-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -13945,7 +13947,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP68]], align 4
 // CHECK19-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP71]], align 4
 // CHECK19-NEXT:    [[TMP72:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -13998,7 +13999,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP94]], align 4
 // CHECK19-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS24]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS25]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP97:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP97]], align 4
 // CHECK19-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_codegen.cpp
index 80301d6b2e50a..928af33b10426 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_codegen.cpp
@@ -103,6 +103,7 @@ void test_target_teams_atomic() {
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_4:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[N_CASTED7:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [3 x ptr], align 8
@@ -110,6 +111,7 @@ void test_target_teams_atomic() {
 // CHECK1-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_12:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_13:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS18:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[G]], ptr [[G_ADDR]], align 8
 // CHECK1-NEXT:    store i32 1000, ptr [[N]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N]], align 4
@@ -178,7 +180,6 @@ void test_target_teams_atomic() {
 // CHECK1-NEXT:    [[TMP33:%.*]] = zext i32 [[ADD]] to i64
 // CHECK1-NEXT:    [[TMP34:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP28]], 0
 // CHECK1-NEXT:    [[TMP35:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP29]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP36]], align 4
 // CHECK1-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -246,7 +247,6 @@ void test_target_teams_atomic() {
 // CHECK1-NEXT:    [[TMP67:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_13]], align 4
 // CHECK1-NEXT:    [[ADD17:%.*]] = add nsw i32 [[TMP67]], 1
 // CHECK1-NEXT:    [[TMP68:%.*]] = zext i32 [[ADD17]] to i64
-// CHECK1-NEXT:    [[KERNEL_ARGS18:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS18]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP69]], align 4
 // CHECK1-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS18]], i32 0, i32 1
@@ -810,6 +810,7 @@ void test_target_teams_atomic() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[X]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    store ptr [[X]], ptr [[TMP0]], align 8
@@ -819,7 +820,6 @@ void test_target_teams_atomic() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1045,6 +1045,7 @@ void test_target_teams_atomic() {
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_4:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[N_CASTED7:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [3 x ptr], align 4
@@ -1052,6 +1053,7 @@ void test_target_teams_atomic() {
 // CHECK3-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_12:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_13:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS18:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store ptr [[G]], ptr [[G_ADDR]], align 4
 // CHECK3-NEXT:    store i32 1000, ptr [[N]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N]], align 4
@@ -1120,7 +1122,6 @@ void test_target_teams_atomic() {
 // CHECK3-NEXT:    [[TMP33:%.*]] = zext i32 [[ADD]] to i64
 // CHECK3-NEXT:    [[TMP34:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP28]], 0
 // CHECK3-NEXT:    [[TMP35:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP29]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP36]], align 4
 // CHECK3-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1188,7 +1189,6 @@ void test_target_teams_atomic() {
 // CHECK3-NEXT:    [[TMP67:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_13]], align 4
 // CHECK3-NEXT:    [[ADD17:%.*]] = add nsw i32 [[TMP67]], 1
 // CHECK3-NEXT:    [[TMP68:%.*]] = zext i32 [[ADD17]] to i64
-// CHECK3-NEXT:    [[KERNEL_ARGS18:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS18]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP69]], align 4
 // CHECK3-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS18]], i32 0, i32 1
@@ -1742,6 +1742,7 @@ void test_target_teams_atomic() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[X]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    store ptr [[X]], ptr [[TMP0]], align 4
@@ -1751,7 +1752,6 @@ void test_target_teams_atomic() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_collapse_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_collapse_codegen.cpp
index 6b0847203d835..8ee4e4fc14ca7 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_collapse_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_collapse_codegen.cpp
@@ -116,6 +116,7 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -127,7 +128,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -375,6 +375,7 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -386,7 +387,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -773,6 +773,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i64, align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -849,7 +850,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store i64 [[SUB7]], ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK9-NEXT:    [[TMP35:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i64 [[TMP35]], 1
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP36]], align 4
 // CHECK9-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1238,6 +1238,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    store ptr [[A]], ptr [[TMP0]], align 8
@@ -1247,7 +1248,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK9-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1495,6 +1495,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i64, align 8
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -1570,7 +1571,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store i64 [[SUB7]], ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK11-NEXT:    [[TMP34:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i64 [[TMP34]], 1
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP35]], align 4
 // CHECK11-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1961,6 +1961,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    store ptr [[A]], ptr [[TMP0]], align 4
@@ -1970,7 +1971,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK11-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_dist_schedule_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_dist_schedule_codegen.cpp
index 1ea96dfdce7df..f43962e1192db 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_dist_schedule_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_dist_schedule_codegen.cpp
@@ -150,14 +150,17 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -169,7 +172,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -212,7 +214,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK1-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -255,7 +256,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK1-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK1-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -833,14 +833,17 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -852,7 +855,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -895,7 +897,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK3-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -938,7 +939,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK3-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK3-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1750,6 +1750,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 8
@@ -1758,6 +1759,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_18:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[N_CASTED19:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
@@ -1768,6 +1770,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[_TMP24:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_26:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -1817,7 +1820,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK9-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1889,7 +1891,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP56:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK9-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP56]], 1
 // CHECK9-NEXT:    [[TMP57:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK9-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1972,7 +1973,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP97:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_26]], align 4
 // CHECK9-NEXT:    [[ADD30:%.*]] = add nsw i32 [[TMP97]], 1
 // CHECK9-NEXT:    [[TMP98:%.*]] = zext i32 [[ADD30]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP99:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP99]], align 4
 // CHECK9-NEXT:    [[TMP100:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -2787,16 +2787,19 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -2807,7 +2810,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK9-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2849,7 +2851,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -2902,7 +2903,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP48]], align 8
 // CHECK9-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP51]], align 4
 // CHECK9-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -3491,6 +3491,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 4
@@ -3499,6 +3500,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_18:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[N_CASTED19:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i32, align 4
@@ -3509,6 +3511,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[_TMP24:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_26:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -3558,7 +3561,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK11-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3631,7 +3633,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP57:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK11-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP57]], 1
 // CHECK11-NEXT:    [[TMP58:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP59]], align 4
 // CHECK11-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -3715,7 +3716,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP99:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_26]], align 4
 // CHECK11-NEXT:    [[ADD30:%.*]] = add nsw i32 [[TMP99]], 1
 // CHECK11-NEXT:    [[TMP100:%.*]] = zext i32 [[ADD30]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP101:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP101]], align 4
 // CHECK11-NEXT:    [[TMP102:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -4515,16 +4515,19 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -4535,7 +4538,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK11-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4577,7 +4579,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -4630,7 +4631,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP48]], align 4
 // CHECK11-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP51]], align 4
 // CHECK11-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_firstprivate_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_firstprivate_codegen.cpp
index bc272f8d5e939..ad133966d929d 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_firstprivate_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_firstprivate_codegen.cpp
@@ -321,6 +321,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr @t_var, align 4
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[T_VAR_CASTED]], align 4
@@ -360,7 +361,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP18]], align 8
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK1-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -745,6 +745,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -785,7 +786,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP15]], align 8
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1418,6 +1418,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr @t_var, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[T_VAR_CASTED]], align 4
@@ -1457,7 +1458,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK3-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1836,6 +1836,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1876,7 +1877,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP15]], align 4
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_if_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_if_codegen.cpp
index ee3efd3820572..7cd64ab5a59da 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_if_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_if_codegen.cpp
@@ -116,7 +116,9 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr @Arg, align 4
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[ARG_CASTED]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = load i64, ptr [[ARG_CASTED]], align 8
@@ -128,7 +130,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -162,7 +163,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9gtid_testv_l47(i64 [[TMP1]]) #[[ATTR2:[0-9]+]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP22]], align 4
 // CHECK1-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -527,14 +527,15 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP5:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -594,7 +595,6 @@ int main() {
 // CHECK1-NEXT:    [[TOBOOL4:%.*]] = trunc i8 [[TMP24]] to i1
 // CHECK1-NEXT:    [[TMP25:%.*]] = select i1 [[TOBOOL4]], i32 0, i32 1
 // CHECK1-NEXT:    [[TMP26:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP25]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP27]], align 4
 // CHECK1-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -1128,14 +1128,15 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[ARG_ADDR:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[ARG]], ptr [[ARG_ADDR]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1191,7 +1192,6 @@ int main() {
 // CHECK1-NEXT:    [[TOBOOL3:%.*]] = trunc i8 [[TMP23]] to i1
 // CHECK1-NEXT:    [[TMP24:%.*]] = select i1 [[TOBOOL3]], i32 0, i32 1
 // CHECK1-NEXT:    [[TMP25:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP24]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP26]], align 4
 // CHECK1-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -1728,7 +1728,9 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr @Arg, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[ARG_CASTED]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = load i64, ptr [[ARG_CASTED]], align 8
@@ -1740,7 +1742,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1774,7 +1775,6 @@ int main() {
 // CHECK3-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9gtid_testv_l45(i64 [[TMP1]]) #[[ATTR2:[0-9]+]]
 // CHECK3-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK3:       omp_offload.cont:
-// CHECK3-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP22]], align 4
 // CHECK3-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -2139,14 +2139,15 @@ int main() {
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[_TMP5:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2206,7 +2207,6 @@ int main() {
 // CHECK3-NEXT:    [[TOBOOL4:%.*]] = trunc i8 [[TMP24]] to i1
 // CHECK3-NEXT:    [[TMP25:%.*]] = select i1 [[TOBOOL4]], i32 0, i32 1
 // CHECK3-NEXT:    [[TMP26:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP25]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP27]], align 4
 // CHECK3-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -2970,14 +2970,15 @@ int main() {
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[ARG_ADDR:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 [[ARG]], ptr [[ARG_ADDR]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3033,7 +3034,6 @@ int main() {
 // CHECK3-NEXT:    [[TOBOOL3:%.*]] = trunc i8 [[TMP23]] to i1
 // CHECK3-NEXT:    [[TMP24:%.*]] = select i1 [[TOBOOL3]], i32 0, i32 1
 // CHECK3-NEXT:    [[TMP25:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP24]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP26]], align 4
 // CHECK3-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -4163,7 +4163,9 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = load i32, ptr @Arg, align 4
 // CHECK9-NEXT:    store i32 [[TMP0]], ptr [[ARG_CASTED]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = load i64, ptr [[ARG_CASTED]], align 8
@@ -4175,7 +4177,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK9-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4209,7 +4210,6 @@ int main() {
 // CHECK9-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9gtid_testv_l47(i64 [[TMP1]]) #[[ATTR2:[0-9]+]]
 // CHECK9-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK9:       omp_offload.cont:
-// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP22]], align 4
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -4574,14 +4574,15 @@ int main() {
 // CHECK9-NEXT:  entry:
 // CHECK9-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP5:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4641,7 +4642,6 @@ int main() {
 // CHECK9-NEXT:    [[TOBOOL4:%.*]] = trunc i8 [[TMP24]] to i1
 // CHECK9-NEXT:    [[TMP25:%.*]] = select i1 [[TOBOOL4]], i32 0, i32 1
 // CHECK9-NEXT:    [[TMP26:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP25]], 0
-// CHECK9-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP27]], align 4
 // CHECK9-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -5175,14 +5175,15 @@ int main() {
 // CHECK9-NEXT:  entry:
 // CHECK9-NEXT:    [[ARG_ADDR:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 [[ARG]], ptr [[ARG_ADDR]], align 4
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5238,7 +5239,6 @@ int main() {
 // CHECK9-NEXT:    [[TOBOOL3:%.*]] = trunc i8 [[TMP23]] to i1
 // CHECK9-NEXT:    [[TMP24:%.*]] = select i1 [[TOBOOL3]], i32 0, i32 1
 // CHECK9-NEXT:    [[TMP25:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP24]], 0
-// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP26]], align 4
 // CHECK9-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -5775,7 +5775,9 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = load i32, ptr @Arg, align 4
 // CHECK11-NEXT:    store i32 [[TMP0]], ptr [[ARG_CASTED]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = load i64, ptr [[ARG_CASTED]], align 8
@@ -5787,7 +5789,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK11-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5821,7 +5822,6 @@ int main() {
 // CHECK11-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9gtid_testv_l45(i64 [[TMP1]]) #[[ATTR2:[0-9]+]]
 // CHECK11-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK11:       omp_offload.cont:
-// CHECK11-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -6186,14 +6186,15 @@ int main() {
 // CHECK11-NEXT:  entry:
 // CHECK11-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[_TMP5:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -6253,7 +6254,6 @@ int main() {
 // CHECK11-NEXT:    [[TOBOOL4:%.*]] = trunc i8 [[TMP24]] to i1
 // CHECK11-NEXT:    [[TMP25:%.*]] = select i1 [[TOBOOL4]], i32 0, i32 1
 // CHECK11-NEXT:    [[TMP26:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP25]], 0
-// CHECK11-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP27]], align 4
 // CHECK11-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -7017,14 +7017,15 @@ int main() {
 // CHECK11-NEXT:  entry:
 // CHECK11-NEXT:    [[ARG_ADDR:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 [[ARG]], ptr [[ARG_ADDR]], align 4
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -7080,7 +7081,6 @@ int main() {
 // CHECK11-NEXT:    [[TOBOOL3:%.*]] = trunc i8 [[TMP23]] to i1
 // CHECK11-NEXT:    [[TMP24:%.*]] = select i1 [[TOBOOL3]], i32 0, i32 1
 // CHECK11-NEXT:    [[TMP25:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP24]], 0
-// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP26]], align 4
 // CHECK11-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_lastprivate_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_lastprivate_codegen.cpp
index ea51af89e47f5..7272323ba0a1f 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_lastprivate_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_lastprivate_codegen.cpp
@@ -771,6 +771,7 @@ int main() {
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK5-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK5-NEXT:    store ptr [[G]], ptr [[G1]], align 8
@@ -823,7 +824,6 @@ int main() {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP20]], align 8
 // CHECK5-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK5-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1266,6 +1266,7 @@ int main() {
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK5-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK5-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -1306,7 +1307,6 @@ int main() {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP15]], align 8
 // CHECK5-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK5-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1806,6 +1806,7 @@ int main() {
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK7-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK7-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK7-NEXT:    store ptr [[G]], ptr [[G1]], align 4
@@ -1858,7 +1859,6 @@ int main() {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP20]], align 4
 // CHECK7-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK7-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2295,6 +2295,7 @@ int main() {
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK7-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK7-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK7-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -2335,7 +2336,6 @@ int main() {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP15]], align 4
 // CHECK7-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK7-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_private_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_private_codegen.cpp
index a7d9e847e2f67..5ed05c5d6243a 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_private_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_private_codegen.cpp
@@ -298,8 +298,8 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -574,6 +574,7 @@ int main() {
 // CHECK1-NEXT:    [[VAR:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -583,7 +584,6 @@ int main() {
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef signext 2)
 // CHECK1-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 8
 // CHECK1-NEXT:    store ptr undef, ptr [[_TMP1]], align 8
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1066,8 +1066,8 @@ int main() {
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1336,6 +1336,7 @@ int main() {
 // CHECK3-NEXT:    [[VAR:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1345,7 +1346,6 @@ int main() {
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef 2)
 // CHECK3-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 4
 // CHECK3-NEXT:    store ptr undef, ptr [[_TMP1]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_proc_bind_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_proc_bind_codegen.cpp
index a953d41542673..35787a84ce96b 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_proc_bind_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_proc_bind_codegen.cpp
@@ -53,9 +53,10 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -89,7 +90,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}_main_l36() #[[ATTR2:[0-9]+]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_reduction_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_reduction_codegen.cpp
index 652e0ce08a90b..f6f12b957a8cc 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_reduction_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_reduction_codegen.cpp
@@ -98,6 +98,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    store ptr @_ZZ4mainE5sivar, ptr [[TMP0]], align 8
@@ -107,7 +108,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -402,6 +402,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -412,7 +413,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -712,6 +712,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    store ptr @_ZZ4mainE5sivar, ptr [[TMP0]], align 4
@@ -721,7 +722,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1012,6 +1012,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -1022,7 +1023,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_schedule_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_schedule_codegen.cpp
index c4f504bf066a7..c1ce2cd270f69 100644
--- a/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_schedule_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_parallel_for_simd_schedule_codegen.cpp
@@ -210,22 +210,27 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS21:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP22:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS27:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS28:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS29:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP30:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -237,7 +242,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -280,7 +284,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK1-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -323,7 +326,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK1-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK1-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -366,7 +368,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP62]], align 8
 // CHECK1-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS19]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS20]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP65]], align 4
 // CHECK1-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 1
@@ -409,7 +410,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP82]], align 8
 // CHECK1-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS27]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS28]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP85]], align 4
 // CHECK1-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -1314,22 +1314,27 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS21:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP22:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS27:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS28:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS29:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP30:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -1341,7 +1346,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1384,7 +1388,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK3-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -1427,7 +1430,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK3-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK3-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1470,7 +1472,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP62]], align 4
 // CHECK3-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS19]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS20]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP65]], align 4
 // CHECK3-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 1
@@ -1513,7 +1514,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP82]], align 4
 // CHECK3-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS27]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS28]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP85]], align 4
 // CHECK3-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -2391,22 +2391,27 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS21:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP22:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS27:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS28:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS29:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP30:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK5-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK5-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -2418,7 +2423,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK5-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK5-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2461,7 +2465,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK5-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK5-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -2504,7 +2507,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK5-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK5-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -2547,7 +2549,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP62]], align 8
 // CHECK5-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS19]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS20]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP65]], align 4
 // CHECK5-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 1
@@ -2590,7 +2591,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP82]], align 8
 // CHECK5-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS27]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS28]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP85]], align 4
 // CHECK5-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -3495,22 +3495,27 @@ int main (int argc, char **argv) {
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS21:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[_TMP22:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[DOTOFFLOAD_BASEPTRS27:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS28:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS29:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[_TMP30:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK7-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK7-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -3522,7 +3527,6 @@ int main (int argc, char **argv) {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK7-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK7-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3565,7 +3569,6 @@ int main (int argc, char **argv) {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK7-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK7-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -3608,7 +3611,6 @@ int main (int argc, char **argv) {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK7-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK7-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -3651,7 +3653,6 @@ int main (int argc, char **argv) {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP62]], align 4
 // CHECK7-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS19]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS20]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP65]], align 4
 // CHECK7-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 1
@@ -3694,7 +3695,6 @@ int main (int argc, char **argv) {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP82]], align 4
 // CHECK7-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS27]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS28]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP85]], align 4
 // CHECK7-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -4959,6 +4959,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 8
@@ -4967,6 +4968,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_18:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[N_CASTED19:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
@@ -4977,6 +4979,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[_TMP24:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_26:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[N_CASTED34:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS35:%.*]] = alloca [3 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS36:%.*]] = alloca [3 x ptr], align 8
@@ -4985,6 +4988,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[_TMP39:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_40:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_41:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS46:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_49:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[N_CASTED50:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR__CASTED51:%.*]] = alloca i64, align 8
@@ -4995,6 +4999,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[_TMP56:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_57:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_58:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS63:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK13-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK13-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -5044,7 +5049,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK13-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK13-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK13-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5116,7 +5120,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP56:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK13-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP56]], 1
 // CHECK13-NEXT:    [[TMP57:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK13-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -5199,7 +5202,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP97:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_26]], align 4
 // CHECK13-NEXT:    [[ADD30:%.*]] = add nsw i32 [[TMP97]], 1
 // CHECK13-NEXT:    [[TMP98:%.*]] = zext i32 [[ADD30]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP99:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP99]], align 4
 // CHECK13-NEXT:    [[TMP100:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -5271,7 +5273,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP132:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_41]], align 4
 // CHECK13-NEXT:    [[ADD45:%.*]] = add nsw i32 [[TMP132]], 1
 // CHECK13-NEXT:    [[TMP133:%.*]] = zext i32 [[ADD45]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS46:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP134:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS46]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP134]], align 4
 // CHECK13-NEXT:    [[TMP135:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS46]], i32 0, i32 1
@@ -5354,7 +5355,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP173:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_58]], align 4
 // CHECK13-NEXT:    [[ADD62:%.*]] = add nsw i32 [[TMP173]], 1
 // CHECK13-NEXT:    [[TMP174:%.*]] = zext i32 [[ADD62]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS63:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP175:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS63]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP175]], align 4
 // CHECK13-NEXT:    [[TMP176:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS63]], i32 0, i32 1
@@ -6658,26 +6658,31 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_22:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR__CASTED23:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS24:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS25:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS26:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[_TMP27:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK13-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK13-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -6688,7 +6693,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK13-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK13-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -6730,7 +6734,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK13-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK13-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -6783,7 +6786,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP48]], align 8
 // CHECK13-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP51]], align 4
 // CHECK13-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -6825,7 +6827,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP68]], align 8
 // CHECK13-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP71]], align 4
 // CHECK13-NEXT:    [[TMP72:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -6878,7 +6879,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP94]], align 8
 // CHECK13-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS24]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS25]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP97:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP97]], align 4
 // CHECK13-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 1
@@ -7807,6 +7807,7 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK15-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 4
@@ -7815,6 +7816,7 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_18:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[N_CASTED19:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i32, align 4
@@ -7825,6 +7827,7 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[_TMP24:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_26:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[N_CASTED34:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS35:%.*]] = alloca [3 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS36:%.*]] = alloca [3 x ptr], align 4
@@ -7833,6 +7836,7 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[_TMP39:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_40:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_41:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS46:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_49:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[N_CASTED50:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR__CASTED51:%.*]] = alloca i32, align 4
@@ -7843,6 +7847,7 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[_TMP56:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_57:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_58:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS63:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK15-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK15-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -7892,7 +7897,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK15-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK15-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK15-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK15-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK15-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -7965,7 +7969,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[TMP57:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK15-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP57]], 1
 // CHECK15-NEXT:    [[TMP58:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK15-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP59]], align 4
 // CHECK15-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -8049,7 +8052,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[TMP99:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_26]], align 4
 // CHECK15-NEXT:    [[ADD30:%.*]] = add nsw i32 [[TMP99]], 1
 // CHECK15-NEXT:    [[TMP100:%.*]] = zext i32 [[ADD30]] to i64
-// CHECK15-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP101:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP101]], align 4
 // CHECK15-NEXT:    [[TMP102:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -8122,7 +8124,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[TMP135:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_41]], align 4
 // CHECK15-NEXT:    [[ADD45:%.*]] = add nsw i32 [[TMP135]], 1
 // CHECK15-NEXT:    [[TMP136:%.*]] = zext i32 [[ADD45]] to i64
-// CHECK15-NEXT:    [[KERNEL_ARGS46:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP137:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS46]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP137]], align 4
 // CHECK15-NEXT:    [[TMP138:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS46]], i32 0, i32 1
@@ -8206,7 +8207,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[TMP177:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_58]], align 4
 // CHECK15-NEXT:    [[ADD62:%.*]] = add nsw i32 [[TMP177]], 1
 // CHECK15-NEXT:    [[TMP178:%.*]] = zext i32 [[ADD62]] to i64
-// CHECK15-NEXT:    [[KERNEL_ARGS63:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP179:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS63]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP179]], align 4
 // CHECK15-NEXT:    [[TMP180:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS63]], i32 0, i32 1
@@ -9485,26 +9485,31 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 4
 // CHECK15-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_22:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR__CASTED23:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS24:%.*]] = alloca [2 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS25:%.*]] = alloca [2 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_MAPPERS26:%.*]] = alloca [2 x ptr], align 4
 // CHECK15-NEXT:    [[_TMP27:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK15-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK15-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -9515,7 +9520,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK15-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK15-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK15-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK15-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK15-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -9557,7 +9561,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK15-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK15-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK15-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK15-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -9610,7 +9613,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    store ptr null, ptr [[TMP48]], align 4
 // CHECK15-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK15-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK15-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP51]], align 4
 // CHECK15-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -9652,7 +9654,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    store ptr null, ptr [[TMP68]], align 4
 // CHECK15-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK15-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK15-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP71]], align 4
 // CHECK15-NEXT:    [[TMP72:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -9705,7 +9706,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    store ptr null, ptr [[TMP94]], align 4
 // CHECK15-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS24]], i32 0, i32 0
 // CHECK15-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS25]], i32 0, i32 0
-// CHECK15-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP97:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP97]], align 4
 // CHECK15-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 1
@@ -10607,6 +10607,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 8
@@ -10615,6 +10616,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_18:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[N_CASTED19:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
@@ -10625,6 +10627,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[_TMP24:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_26:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[N_CASTED34:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS35:%.*]] = alloca [3 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS36:%.*]] = alloca [3 x ptr], align 8
@@ -10633,6 +10636,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[_TMP39:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_40:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_41:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS46:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_49:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[N_CASTED50:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR__CASTED51:%.*]] = alloca i64, align 8
@@ -10643,6 +10647,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[_TMP56:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_57:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_58:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS63:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK17-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK17-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -10692,7 +10697,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK17-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK17-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK17-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -10764,7 +10768,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP56:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK17-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP56]], 1
 // CHECK17-NEXT:    [[TMP57:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK17-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -10847,7 +10850,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP97:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_26]], align 4
 // CHECK17-NEXT:    [[ADD30:%.*]] = add nsw i32 [[TMP97]], 1
 // CHECK17-NEXT:    [[TMP98:%.*]] = zext i32 [[ADD30]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP99:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP99]], align 4
 // CHECK17-NEXT:    [[TMP100:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -10919,7 +10921,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP132:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_41]], align 4
 // CHECK17-NEXT:    [[ADD45:%.*]] = add nsw i32 [[TMP132]], 1
 // CHECK17-NEXT:    [[TMP133:%.*]] = zext i32 [[ADD45]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS46:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP134:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS46]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP134]], align 4
 // CHECK17-NEXT:    [[TMP135:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS46]], i32 0, i32 1
@@ -11002,7 +11003,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP173:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_58]], align 4
 // CHECK17-NEXT:    [[ADD62:%.*]] = add nsw i32 [[TMP173]], 1
 // CHECK17-NEXT:    [[TMP174:%.*]] = zext i32 [[ADD62]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS63:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP175:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS63]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP175]], align 4
 // CHECK17-NEXT:    [[TMP176:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS63]], i32 0, i32 1
@@ -12306,26 +12306,31 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_22:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR__CASTED23:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS24:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS25:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS26:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[_TMP27:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK17-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK17-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -12336,7 +12341,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK17-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK17-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -12378,7 +12382,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK17-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK17-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -12431,7 +12434,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP48]], align 8
 // CHECK17-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP51]], align 4
 // CHECK17-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -12473,7 +12475,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP68]], align 8
 // CHECK17-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP71]], align 4
 // CHECK17-NEXT:    [[TMP72:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -12526,7 +12527,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP94]], align 8
 // CHECK17-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS24]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS25]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP97:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP97]], align 4
 // CHECK17-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 1
@@ -13455,6 +13455,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 4
@@ -13463,6 +13464,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_18:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[N_CASTED19:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i32, align 4
@@ -13473,6 +13475,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[_TMP24:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_26:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[N_CASTED34:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS35:%.*]] = alloca [3 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS36:%.*]] = alloca [3 x ptr], align 4
@@ -13481,6 +13484,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[_TMP39:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_40:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_41:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS46:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_49:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[N_CASTED50:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR__CASTED51:%.*]] = alloca i32, align 4
@@ -13491,6 +13495,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[_TMP56:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_57:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_58:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS63:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK19-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK19-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -13540,7 +13545,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK19-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK19-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK19-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -13613,7 +13617,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP57:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK19-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP57]], 1
 // CHECK19-NEXT:    [[TMP58:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP59]], align 4
 // CHECK19-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -13697,7 +13700,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP99:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_26]], align 4
 // CHECK19-NEXT:    [[ADD30:%.*]] = add nsw i32 [[TMP99]], 1
 // CHECK19-NEXT:    [[TMP100:%.*]] = zext i32 [[ADD30]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP101:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP101]], align 4
 // CHECK19-NEXT:    [[TMP102:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -13770,7 +13772,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP135:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_41]], align 4
 // CHECK19-NEXT:    [[ADD45:%.*]] = add nsw i32 [[TMP135]], 1
 // CHECK19-NEXT:    [[TMP136:%.*]] = zext i32 [[ADD45]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS46:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP137:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS46]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP137]], align 4
 // CHECK19-NEXT:    [[TMP138:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS46]], i32 0, i32 1
@@ -13854,7 +13855,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP177:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_58]], align 4
 // CHECK19-NEXT:    [[ADD62:%.*]] = add nsw i32 [[TMP177]], 1
 // CHECK19-NEXT:    [[TMP178:%.*]] = zext i32 [[ADD62]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS63:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP179:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS63]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP179]], align 4
 // CHECK19-NEXT:    [[TMP180:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS63]], i32 0, i32 1
@@ -15133,26 +15133,31 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_22:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR__CASTED23:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS24:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS25:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS26:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[_TMP27:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK19-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK19-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -15163,7 +15168,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK19-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK19-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -15205,7 +15209,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK19-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK19-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -15258,7 +15261,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP48]], align 4
 // CHECK19-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP51]], align 4
 // CHECK19-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -15300,7 +15302,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP68]], align 4
 // CHECK19-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP71]], align 4
 // CHECK19-NEXT:    [[TMP72:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -15353,7 +15354,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP94]], align 4
 // CHECK19-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS24]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS25]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP97:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP97]], align 4
 // CHECK19-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_private_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_private_codegen.cpp
index 9127aec58b9f4..5fb689f0938c7 100644
--- a/clang/test/OpenMP/target_teams_distribute_private_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_private_codegen.cpp
@@ -237,8 +237,8 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -398,6 +398,7 @@ int main() {
 // CHECK1-NEXT:    [[VAR:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -407,7 +408,6 @@ int main() {
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef signext 2)
 // CHECK1-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 8
 // CHECK1-NEXT:    store ptr undef, ptr [[_TMP1]], align 8
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -772,8 +772,8 @@ int main() {
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -931,6 +931,7 @@ int main() {
 // CHECK3-NEXT:    [[VAR:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -940,7 +941,6 @@ int main() {
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef 2)
 // CHECK3-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 4
 // CHECK3-NEXT:    store ptr undef, ptr [[_TMP1]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_reduction_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_reduction_codegen.cpp
index ffc4f6f500da5..ec4e0ac3c21cf 100644
--- a/clang/test/OpenMP/target_teams_distribute_reduction_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_reduction_codegen.cpp
@@ -276,45 +276,55 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[AND_VAR:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[OR_VAR:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS22:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS23:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS24:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP25:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS26:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[BIT_VAR:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS29:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS30:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS31:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP32:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS33:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS36:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS37:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS38:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP39:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS40:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS43:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS44:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS45:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP46:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS47:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS50:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS51:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS52:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP53:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS54:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS57:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS58:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS59:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP60:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    store ptr @_ZZ4mainE5sivar, ptr [[TMP0]], align 8
@@ -324,7 +334,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -366,7 +375,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK1-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -409,7 +417,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK1-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK1-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -452,7 +459,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP62]], align 8
 // CHECK1-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP65]], align 4
 // CHECK1-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -495,7 +501,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP82]], align 8
 // CHECK1-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS22]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS23]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS26:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS26]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP85]], align 4
 // CHECK1-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS26]], i32 0, i32 1
@@ -538,7 +543,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP102]], align 8
 // CHECK1-NEXT:    [[TMP103:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS29]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP104:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS30]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS33:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP105:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS33]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP105]], align 4
 // CHECK1-NEXT:    [[TMP106:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS33]], i32 0, i32 1
@@ -580,7 +584,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP122]], align 8
 // CHECK1-NEXT:    [[TMP123:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS36]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP124:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS37]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS40:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP125:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS40]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP125]], align 4
 // CHECK1-NEXT:    [[TMP126:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS40]], i32 0, i32 1
@@ -622,7 +625,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP142]], align 8
 // CHECK1-NEXT:    [[TMP143:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS43]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP144:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS44]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS47:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP145:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS47]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP145]], align 4
 // CHECK1-NEXT:    [[TMP146:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS47]], i32 0, i32 1
@@ -665,7 +667,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP162]], align 8
 // CHECK1-NEXT:    [[TMP163:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS50]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP164:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS51]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS54:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP165:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS54]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP165]], align 4
 // CHECK1-NEXT:    [[TMP166:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS54]], i32 0, i32 1
@@ -708,7 +709,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP182]], align 8
 // CHECK1-NEXT:    [[TMP183:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS57]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP184:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS58]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP185:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP185]], align 4
 // CHECK1-NEXT:    [[TMP186:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 1
@@ -2119,45 +2119,55 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[AND_VAR:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[OR_VAR:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS22:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS23:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS24:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP25:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS26:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[BIT_VAR:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS29:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS30:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS31:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP32:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS33:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS36:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS37:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS38:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP39:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS40:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS43:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS44:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS45:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP46:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS47:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS50:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS51:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS52:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP53:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS54:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS57:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS58:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS59:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP60:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -2168,7 +2178,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2216,7 +2225,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP25]], align 8
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP28]], align 4
 // CHECK1-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -2265,7 +2273,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP48]], align 8
 // CHECK1-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP51]], align 4
 // CHECK1-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -2314,7 +2321,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP71]], align 8
 // CHECK1-NEXT:    [[TMP72:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP73:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP74:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP74]], align 4
 // CHECK1-NEXT:    [[TMP75:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -2363,7 +2369,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP94]], align 8
 // CHECK1-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS22]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS23]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS26:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP97:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS26]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP97]], align 4
 // CHECK1-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS26]], i32 0, i32 1
@@ -2412,7 +2417,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP117]], align 8
 // CHECK1-NEXT:    [[TMP118:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS29]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP119:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS30]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS33:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP120:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS33]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP120]], align 4
 // CHECK1-NEXT:    [[TMP121:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS33]], i32 0, i32 1
@@ -2460,7 +2464,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP140]], align 8
 // CHECK1-NEXT:    [[TMP141:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS36]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP142:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS37]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS40:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP143:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS40]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP143]], align 4
 // CHECK1-NEXT:    [[TMP144:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS40]], i32 0, i32 1
@@ -2508,7 +2511,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP163]], align 8
 // CHECK1-NEXT:    [[TMP164:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS43]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP165:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS44]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS47:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP166:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS47]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP166]], align 4
 // CHECK1-NEXT:    [[TMP167:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS47]], i32 0, i32 1
@@ -2557,7 +2559,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP186]], align 8
 // CHECK1-NEXT:    [[TMP187:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS50]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP188:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS51]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS54:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP189:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS54]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP189]], align 4
 // CHECK1-NEXT:    [[TMP190:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS54]], i32 0, i32 1
@@ -2606,7 +2607,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP209]], align 8
 // CHECK1-NEXT:    [[TMP210:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS57]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP211:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS58]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP212:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP212]], align 4
 // CHECK1-NEXT:    [[TMP213:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 1
@@ -4111,45 +4111,55 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[AND_VAR:%.*]] = alloca i8, align 1
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[OR_VAR:%.*]] = alloca i8, align 1
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS22:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS23:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS24:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP25:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS26:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[BIT_VAR:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS29:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS30:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS31:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP32:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS33:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS36:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS37:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS38:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP39:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS40:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS43:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS44:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS45:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP46:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS47:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS50:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS51:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS52:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP53:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS54:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS57:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS58:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS59:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP60:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    store ptr @_ZZ4mainE5sivar, ptr [[TMP0]], align 4
@@ -4159,7 +4169,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4201,7 +4210,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK3-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -4244,7 +4252,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK3-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK3-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -4287,7 +4294,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP62]], align 4
 // CHECK3-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP65]], align 4
 // CHECK3-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -4330,7 +4336,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP82]], align 4
 // CHECK3-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS22]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS23]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS26:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS26]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP85]], align 4
 // CHECK3-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS26]], i32 0, i32 1
@@ -4373,7 +4378,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP102]], align 4
 // CHECK3-NEXT:    [[TMP103:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS29]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP104:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS30]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS33:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP105:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS33]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP105]], align 4
 // CHECK3-NEXT:    [[TMP106:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS33]], i32 0, i32 1
@@ -4415,7 +4419,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP122]], align 4
 // CHECK3-NEXT:    [[TMP123:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS36]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP124:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS37]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS40:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP125:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS40]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP125]], align 4
 // CHECK3-NEXT:    [[TMP126:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS40]], i32 0, i32 1
@@ -4457,7 +4460,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP142]], align 4
 // CHECK3-NEXT:    [[TMP143:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS43]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP144:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS44]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS47:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP145:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS47]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP145]], align 4
 // CHECK3-NEXT:    [[TMP146:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS47]], i32 0, i32 1
@@ -4500,7 +4502,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP162]], align 4
 // CHECK3-NEXT:    [[TMP163:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS50]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP164:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS51]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS54:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP165:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS54]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP165]], align 4
 // CHECK3-NEXT:    [[TMP166:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS54]], i32 0, i32 1
@@ -4543,7 +4544,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP182]], align 4
 // CHECK3-NEXT:    [[TMP183:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS57]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP184:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS58]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP185:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP185]], align 4
 // CHECK3-NEXT:    [[TMP186:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 1
@@ -5954,45 +5954,55 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[AND_VAR:%.*]] = alloca i8, align 1
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[OR_VAR:%.*]] = alloca i8, align 1
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS22:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS23:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS24:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP25:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS26:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[BIT_VAR:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS29:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS30:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS31:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP32:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS33:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS36:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS37:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS38:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP39:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS40:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS43:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS44:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS45:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP46:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS47:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS50:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS51:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS52:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP53:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS54:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS57:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS58:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS59:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP60:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -6003,7 +6013,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -6051,7 +6060,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP25]], align 4
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP28]], align 4
 // CHECK3-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -6100,7 +6108,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP48]], align 4
 // CHECK3-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP51]], align 4
 // CHECK3-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -6149,7 +6156,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP71]], align 4
 // CHECK3-NEXT:    [[TMP72:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP73:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP74:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP74]], align 4
 // CHECK3-NEXT:    [[TMP75:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -6198,7 +6204,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP94]], align 4
 // CHECK3-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS22]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS23]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS26:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP97:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS26]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP97]], align 4
 // CHECK3-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS26]], i32 0, i32 1
@@ -6247,7 +6252,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP117]], align 4
 // CHECK3-NEXT:    [[TMP118:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS29]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP119:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS30]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS33:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP120:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS33]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP120]], align 4
 // CHECK3-NEXT:    [[TMP121:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS33]], i32 0, i32 1
@@ -6295,7 +6299,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP140]], align 4
 // CHECK3-NEXT:    [[TMP141:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS36]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP142:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS37]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS40:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP143:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS40]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP143]], align 4
 // CHECK3-NEXT:    [[TMP144:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS40]], i32 0, i32 1
@@ -6343,7 +6346,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP163]], align 4
 // CHECK3-NEXT:    [[TMP164:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS43]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP165:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS44]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS47:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP166:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS47]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP166]], align 4
 // CHECK3-NEXT:    [[TMP167:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS47]], i32 0, i32 1
@@ -6392,7 +6394,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP186]], align 4
 // CHECK3-NEXT:    [[TMP187:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS50]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP188:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS51]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS54:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP189:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS54]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP189]], align 4
 // CHECK3-NEXT:    [[TMP190:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS54]], i32 0, i32 1
@@ -6441,7 +6442,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP209]], align 4
 // CHECK3-NEXT:    [[TMP210:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS57]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP211:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS58]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP212:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP212]], align 4
 // CHECK3-NEXT:    [[TMP213:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_simd_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_simd_codegen.cpp
index 47fd29ebacd75..7cc064f779c5d 100644
--- a/clang/test/OpenMP/target_teams_distribute_simd_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_simd_codegen.cpp
@@ -320,18 +320,21 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS7:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[A_CASTED8:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[AA_CASTED9:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS11:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS12:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP13:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[A_CASTED17:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS20:%.*]] = alloca [9 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS21:%.*]] = alloca [9 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS22:%.*]] = alloca [9 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [9 x i64], align 8
 // CHECK1-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS24:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB2:[0-9]+]])
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
@@ -420,7 +423,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP49]], align 8
 // CHECK1-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS5]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS6]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP52]], align 4
 // CHECK1-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -478,7 +480,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP77]], align 8
 // CHECK1-NEXT:    [[TMP78:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS10]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP79:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS11]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP80:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP80]], align 4
 // CHECK1-NEXT:    [[TMP81:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 1
@@ -589,7 +590,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP130:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_BASEPTRS20]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP131:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_PTRS21]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP132:%.*]] = getelementptr inbounds [9 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS24:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP133:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS24]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP133]], align 4
 // CHECK1-NEXT:    [[TMP134:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS24]], i32 0, i32 1
@@ -766,10 +766,10 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR1_I:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR2_I:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR3_I:%.*]] = alloca ptr, align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[AA_CASTED_I:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED_I:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED4_I:%.*]] = alloca i64, align 8
-// CHECK1-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTADDR:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTADDR1:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[DOTADDR]], align 4
@@ -1331,6 +1331,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [5 x i64], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
@@ -1390,7 +1391,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [5 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK1-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1460,6 +1460,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK1-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -1527,7 +1528,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP30:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_2]], align 4
 // CHECK1-NEXT:    [[ADD5:%.*]] = add i32 [[TMP30]], 1
 // CHECK1-NEXT:    [[TMP31:%.*]] = zext i32 [[ADD5]] to i64
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP32]], align 4
 // CHECK1-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1583,6 +1583,7 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK1-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -1616,7 +1617,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP13]], align 8
 // CHECK1-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2091,18 +2091,21 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS7:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[A_CASTED8:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[AA_CASTED9:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS10:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS11:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS12:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP13:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[A_CASTED17:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS20:%.*]] = alloca [9 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS21:%.*]] = alloca [9 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS22:%.*]] = alloca [9 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [9 x i64], align 4
 // CHECK3-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS24:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB2:[0-9]+]])
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
@@ -2189,7 +2192,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP47]], align 4
 // CHECK3-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS5]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS6]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP50]], align 4
 // CHECK3-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2247,7 +2249,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP75]], align 4
 // CHECK3-NEXT:    [[TMP76:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS10]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP77:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS11]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP78:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP78]], align 4
 // CHECK3-NEXT:    [[TMP79:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 1
@@ -2360,7 +2361,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP130:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_BASEPTRS20]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP131:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_PTRS21]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP132:%.*]] = getelementptr inbounds [9 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS24:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP133:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS24]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP133]], align 4
 // CHECK3-NEXT:    [[TMP134:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS24]], i32 0, i32 1
@@ -2537,10 +2537,10 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR1_I:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR2_I:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR3_I:%.*]] = alloca ptr, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[AA_CASTED_I:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR__CASTED_I:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR__CASTED4_I:%.*]] = alloca i32, align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTADDR:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTADDR1:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[DOTADDR]], align 4
@@ -3102,6 +3102,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [5 x i64], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
@@ -3161,7 +3162,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [5 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK3-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3231,6 +3231,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK3-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -3298,7 +3299,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP30:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_2]], align 4
 // CHECK3-NEXT:    [[ADD5:%.*]] = add i32 [[TMP30]], 1
 // CHECK3-NEXT:    [[TMP31:%.*]] = zext i32 [[ADD5]] to i64
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP32]], align 4
 // CHECK3-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3354,6 +3354,7 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK3-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -3387,7 +3388,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP13]], align 4
 // CHECK3-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK3-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3862,18 +3862,21 @@ int bar(int n){
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS7:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[A_CASTED8:%.*]] = alloca i64, align 8
 // CHECK5-NEXT:    [[AA_CASTED9:%.*]] = alloca i64, align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS11:%.*]] = alloca [2 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS12:%.*]] = alloca [2 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP13:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[A_CASTED17:%.*]] = alloca i64, align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS20:%.*]] = alloca [9 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS21:%.*]] = alloca [9 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS22:%.*]] = alloca [9 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [9 x i64], align 8
 // CHECK5-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS24:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB2:[0-9]+]])
 // CHECK5-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK5-NEXT:    store i32 0, ptr [[A]], align 4
@@ -3962,7 +3965,6 @@ int bar(int n){
 // CHECK5-NEXT:    store ptr null, ptr [[TMP49]], align 8
 // CHECK5-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS5]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS6]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP52]], align 4
 // CHECK5-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4020,7 +4022,6 @@ int bar(int n){
 // CHECK5-NEXT:    store ptr null, ptr [[TMP77]], align 8
 // CHECK5-NEXT:    [[TMP78:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS10]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP79:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS11]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP80:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP80]], align 4
 // CHECK5-NEXT:    [[TMP81:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 1
@@ -4131,7 +4132,6 @@ int bar(int n){
 // CHECK5-NEXT:    [[TMP130:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_BASEPTRS20]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP131:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_PTRS21]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP132:%.*]] = getelementptr inbounds [9 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS24:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP133:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS24]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP133]], align 4
 // CHECK5-NEXT:    [[TMP134:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS24]], i32 0, i32 1
@@ -4308,10 +4308,10 @@ int bar(int n){
 // CHECK5-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR1_I:%.*]] = alloca ptr, align 8
 // CHECK5-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR2_I:%.*]] = alloca ptr, align 8
 // CHECK5-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR3_I:%.*]] = alloca ptr, align 8
+// CHECK5-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[AA_CASTED_I:%.*]] = alloca i64, align 8
 // CHECK5-NEXT:    [[DOTCAPTURE_EXPR__CASTED_I:%.*]] = alloca i64, align 8
 // CHECK5-NEXT:    [[DOTCAPTURE_EXPR__CASTED4_I:%.*]] = alloca i64, align 8
-// CHECK5-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[DOTADDR:%.*]] = alloca i32, align 4
 // CHECK5-NEXT:    [[DOTADDR1:%.*]] = alloca ptr, align 8
 // CHECK5-NEXT:    store i32 [[TMP0]], ptr [[DOTADDR]], align 4
@@ -4875,6 +4875,7 @@ int bar(int n){
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [6 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [6 x i64], align 8
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK5-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK5-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
@@ -4949,7 +4950,6 @@ int bar(int n){
 // CHECK5-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [6 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [6 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [6 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP35]], align 4
 // CHECK5-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5019,6 +5019,7 @@ int bar(int n){
 // CHECK5-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK5-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
 // CHECK5-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK5-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK5-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -5086,7 +5087,6 @@ int bar(int n){
 // CHECK5-NEXT:    [[TMP30:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_2]], align 4
 // CHECK5-NEXT:    [[ADD5:%.*]] = add i32 [[TMP30]], 1
 // CHECK5-NEXT:    [[TMP31:%.*]] = zext i32 [[ADD5]] to i64
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP32]], align 4
 // CHECK5-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5142,6 +5142,7 @@ int bar(int n){
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK5-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK5-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -5175,7 +5176,6 @@ int bar(int n){
 // CHECK5-NEXT:    store ptr null, ptr [[TMP13]], align 8
 // CHECK5-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK5-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5702,18 +5702,21 @@ int bar(int n){
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS7:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[A_CASTED8:%.*]] = alloca i32, align 4
 // CHECK7-NEXT:    [[AA_CASTED9:%.*]] = alloca i32, align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_BASEPTRS10:%.*]] = alloca [2 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS11:%.*]] = alloca [2 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS12:%.*]] = alloca [2 x ptr], align 4
 // CHECK7-NEXT:    [[_TMP13:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[A_CASTED17:%.*]] = alloca i32, align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_BASEPTRS20:%.*]] = alloca [9 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS21:%.*]] = alloca [9 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS22:%.*]] = alloca [9 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [9 x i64], align 4
 // CHECK7-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS24:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[TMP0:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB2:[0-9]+]])
 // CHECK7-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK7-NEXT:    store i32 0, ptr [[A]], align 4
@@ -5800,7 +5803,6 @@ int bar(int n){
 // CHECK7-NEXT:    store ptr null, ptr [[TMP47]], align 4
 // CHECK7-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS5]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS6]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP50]], align 4
 // CHECK7-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5858,7 +5860,6 @@ int bar(int n){
 // CHECK7-NEXT:    store ptr null, ptr [[TMP75]], align 4
 // CHECK7-NEXT:    [[TMP76:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS10]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP77:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS11]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS14:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[TMP78:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP78]], align 4
 // CHECK7-NEXT:    [[TMP79:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS14]], i32 0, i32 1
@@ -5971,7 +5972,6 @@ int bar(int n){
 // CHECK7-NEXT:    [[TMP130:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_BASEPTRS20]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP131:%.*]] = getelementptr inbounds [9 x ptr], ptr [[DOTOFFLOAD_PTRS21]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP132:%.*]] = getelementptr inbounds [9 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS24:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[TMP133:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS24]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP133]], align 4
 // CHECK7-NEXT:    [[TMP134:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS24]], i32 0, i32 1
@@ -6148,10 +6148,10 @@ int bar(int n){
 // CHECK7-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR1_I:%.*]] = alloca ptr, align 4
 // CHECK7-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR2_I:%.*]] = alloca ptr, align 4
 // CHECK7-NEXT:    [[DOTFIRSTPRIV_PTR_ADDR3_I:%.*]] = alloca ptr, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[AA_CASTED_I:%.*]] = alloca i32, align 4
 // CHECK7-NEXT:    [[DOTCAPTURE_EXPR__CASTED_I:%.*]] = alloca i32, align 4
 // CHECK7-NEXT:    [[DOTCAPTURE_EXPR__CASTED4_I:%.*]] = alloca i32, align 4
-// CHECK7-NEXT:    [[KERNEL_ARGS_I:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[DOTADDR:%.*]] = alloca i32, align 4
 // CHECK7-NEXT:    [[DOTADDR1:%.*]] = alloca ptr, align 4
 // CHECK7-NEXT:    store i32 [[TMP0]], ptr [[DOTADDR]], align 4
@@ -6715,6 +6715,7 @@ int bar(int n){
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [6 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [6 x i64], align 4
 // CHECK7-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK7-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK7-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
@@ -6789,7 +6790,6 @@ int bar(int n){
 // CHECK7-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [6 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [6 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [6 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP35]], align 4
 // CHECK7-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -6859,6 +6859,7 @@ int bar(int n){
 // CHECK7-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK7-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
 // CHECK7-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK7-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK7-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -6926,7 +6927,6 @@ int bar(int n){
 // CHECK7-NEXT:    [[TMP30:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_2]], align 4
 // CHECK7-NEXT:    [[ADD5:%.*]] = add i32 [[TMP30]], 1
 // CHECK7-NEXT:    [[TMP31:%.*]] = zext i32 [[ADD5]] to i64
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP32]], align 4
 // CHECK7-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -6982,6 +6982,7 @@ int bar(int n){
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
 // CHECK7-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK7-NEXT:    store i32 0, ptr [[A]], align 4
 // CHECK7-NEXT:    store i16 0, ptr [[AA]], align 2
@@ -7015,7 +7016,6 @@ int bar(int n){
 // CHECK7-NEXT:    store ptr null, ptr [[TMP13]], align 4
 // CHECK7-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP16]], align 4
 // CHECK7-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_simd_collapse_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_simd_collapse_codegen.cpp
index 4aa8008841c04..073204e9fbbf0 100644
--- a/clang/test/OpenMP/target_teams_distribute_simd_collapse_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_simd_collapse_codegen.cpp
@@ -111,6 +111,7 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -122,7 +123,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -287,6 +287,7 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -298,7 +299,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -606,6 +606,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i64, align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -682,7 +683,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store i64 [[SUB7]], ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK9-NEXT:    [[TMP35:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i64 [[TMP35]], 1
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP36]], align 4
 // CHECK9-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -927,6 +927,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    store ptr [[A]], ptr [[TMP0]], align 8
@@ -936,7 +937,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK9-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1101,6 +1101,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i64, align 8
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -1176,7 +1177,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store i64 [[SUB7]], ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK11-NEXT:    [[TMP34:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i64 [[TMP34]], 1
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP35]], align 4
 // CHECK11-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1419,6 +1419,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    store ptr [[A]], ptr [[TMP0]], align 4
@@ -1428,7 +1429,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK11-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_simd_dist_schedule_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_simd_dist_schedule_codegen.cpp
index 5a14f0ab1939e..0d43e6f5b5dfe 100644
--- a/clang/test/OpenMP/target_teams_distribute_simd_dist_schedule_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_simd_dist_schedule_codegen.cpp
@@ -138,14 +138,17 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -157,7 +160,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -200,7 +202,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK1-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -243,7 +244,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK1-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK1-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -578,14 +578,17 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -597,7 +600,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -640,7 +642,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK3-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -683,7 +684,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK3-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK3-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1263,6 +1263,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 8
@@ -1271,6 +1272,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_18:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[N_CASTED19:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i64, align 8
@@ -1281,6 +1283,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[_TMP24:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_26:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -1329,7 +1332,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK9-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1401,7 +1403,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP56:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK9-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP56]], 1
 // CHECK9-NEXT:    [[TMP57:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK9-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1484,7 +1485,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP97:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_26]], align 4
 // CHECK9-NEXT:    [[ADD30:%.*]] = add nsw i32 [[TMP97]], 1
 // CHECK9-NEXT:    [[TMP98:%.*]] = zext i32 [[ADD30]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP99:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP99]], align 4
 // CHECK9-NEXT:    [[TMP100:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -1937,14 +1937,17 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    store ptr [[A]], ptr [[TMP0]], align 8
@@ -1954,7 +1957,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK9-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1996,7 +1998,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -2038,7 +2039,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK9-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK9-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -2368,6 +2368,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 4
@@ -2376,6 +2377,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_18:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[N_CASTED19:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR__CASTED:%.*]] = alloca i32, align 4
@@ -2386,6 +2388,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[_TMP24:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_26:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -2434,7 +2437,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK11-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2507,7 +2509,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP57:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK11-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP57]], 1
 // CHECK11-NEXT:    [[TMP58:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP59]], align 4
 // CHECK11-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -2591,7 +2592,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP99:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_26]], align 4
 // CHECK11-NEXT:    [[ADD30:%.*]] = add nsw i32 [[TMP99]], 1
 // CHECK11-NEXT:    [[TMP100:%.*]] = zext i32 [[ADD30]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP101:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP101]], align 4
 // CHECK11-NEXT:    [[TMP102:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -3041,14 +3041,17 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    store ptr [[A]], ptr [[TMP0]], align 4
@@ -3058,7 +3061,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK11-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3100,7 +3102,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -3142,7 +3143,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK11-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK11-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_simd_firstprivate_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_simd_firstprivate_codegen.cpp
index 20e018504e5c6..fb0824a86d926 100644
--- a/clang/test/OpenMP/target_teams_distribute_simd_firstprivate_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_simd_firstprivate_codegen.cpp
@@ -262,6 +262,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr @t_var, align 4
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[T_VAR_CASTED]], align 4
@@ -301,7 +302,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP18]], align 8
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK1-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -549,6 +549,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -589,7 +590,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP15]], align 8
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1085,6 +1085,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr @t_var, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[T_VAR_CASTED]], align 4
@@ -1124,7 +1125,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK3-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1370,6 +1370,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1410,7 +1411,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP15]], align 4
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_simd_lastprivate_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_simd_lastprivate_codegen.cpp
index f0786333ad125..060b5ba4e6123 100644
--- a/clang/test/OpenMP/target_teams_distribute_simd_lastprivate_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_simd_lastprivate_codegen.cpp
@@ -532,6 +532,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store ptr [[G]], ptr [[G1]], align 8
@@ -584,7 +585,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP20]], align 8
 // CHECK9-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -866,6 +866,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK9-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -906,7 +907,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP15]], align 8
 // CHECK9-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK9-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1254,6 +1254,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store ptr [[G]], ptr [[G1]], align 4
@@ -1306,7 +1307,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP20]], align 4
 // CHECK11-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1586,6 +1586,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK11-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1626,7 +1627,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP15]], align 4
 // CHECK11-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK11-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_simd_private_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_simd_private_codegen.cpp
index e4f62469a8b26..2e7a6d5f92273 100644
--- a/clang/test/OpenMP/target_teams_distribute_simd_private_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_simd_private_codegen.cpp
@@ -237,8 +237,8 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -405,6 +405,7 @@ int main() {
 // CHECK1-NEXT:    [[VAR:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -414,7 +415,6 @@ int main() {
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef signext 2)
 // CHECK1-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 8
 // CHECK1-NEXT:    store ptr undef, ptr [[_TMP1]], align 8
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -786,8 +786,8 @@ int main() {
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -952,6 +952,7 @@ int main() {
 // CHECK3-NEXT:    [[VAR:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -961,7 +962,6 @@ int main() {
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef 2)
 // CHECK3-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 4
 // CHECK3-NEXT:    store ptr undef, ptr [[_TMP1]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_distribute_simd_reduction_codegen.cpp b/clang/test/OpenMP/target_teams_distribute_simd_reduction_codegen.cpp
index 38adc315e8e82..394a44458f587 100644
--- a/clang/test/OpenMP/target_teams_distribute_simd_reduction_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_simd_reduction_codegen.cpp
@@ -86,6 +86,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    store ptr @_ZZ4mainE5sivar, ptr [[TMP0]], align 8
@@ -95,7 +96,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -269,6 +269,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -279,7 +280,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -458,6 +458,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    store ptr @_ZZ4mainE5sivar, ptr [[TMP0]], align 4
@@ -467,7 +468,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -641,6 +641,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -651,7 +652,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_map_codegen.cpp b/clang/test/OpenMP/target_teams_map_codegen.cpp
index 69aa5002ca591..a2b30400287ce 100644
--- a/clang/test/OpenMP/target_teams_map_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_map_codegen.cpp
@@ -84,6 +84,7 @@ void mapInt128() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [2 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    store ptr [[X]], ptr [[TMP0]], align 8
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
@@ -98,7 +99,6 @@ void mapInt128() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP5]], align 8
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP8]], align 4
 // CHECK1-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -162,6 +162,7 @@ void mapInt128() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [2 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    store ptr [[X]], ptr [[TMP0]], align 8
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
@@ -176,7 +177,6 @@ void mapInt128() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP5]], align 8
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP8]], align 4
 // CHECK1-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -256,6 +256,7 @@ void mapInt128() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [2 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    store ptr [[X]], ptr [[TMP0]], align 8
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
@@ -270,7 +271,6 @@ void mapInt128() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP5]], align 8
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP8]], align 4
 // CHECK1-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -406,6 +406,7 @@ void mapInt128() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    store ptr [[X]], ptr [[TMP0]], align 8
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
@@ -414,7 +415,6 @@ void mapInt128() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -484,6 +484,7 @@ void mapInt128() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    store ptr [[X]], ptr [[TMP0]], align 8
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
@@ -492,7 +493,6 @@ void mapInt128() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -562,6 +562,7 @@ void mapInt128() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    store ptr [[X]], ptr [[TMP0]], align 8
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
@@ -570,7 +571,6 @@ void mapInt128() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -642,9 +642,11 @@ void mapInt128() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS4:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    store ptr [[Y]], ptr [[TMP0]], align 8
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
@@ -665,7 +667,6 @@ void mapInt128() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP8]], align 8
 // CHECK1-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP11]], align 4
 // CHECK1-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -719,7 +720,6 @@ void mapInt128() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP34]], align 8
 // CHECK1-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS4:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS4]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP37]], align 4
 // CHECK1-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS4]], i32 0, i32 1
@@ -1005,9 +1005,11 @@ void mapInt128() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS4:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    store ptr [[Y]], ptr [[TMP0]], align 8
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
@@ -1028,7 +1030,6 @@ void mapInt128() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP8]], align 8
 // CHECK1-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP11]], align 4
 // CHECK1-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1082,7 +1083,6 @@ void mapInt128() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP34]], align 8
 // CHECK1-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS4:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS4]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP37]], align 4
 // CHECK1-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS4]], i32 0, i32 1
@@ -1318,6 +1318,7 @@ void mapInt128() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [2 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    store ptr [[X]], ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
@@ -1332,7 +1333,6 @@ void mapInt128() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP8]], align 4
 // CHECK3-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1396,6 +1396,7 @@ void mapInt128() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [2 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    store ptr [[X]], ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
@@ -1410,7 +1411,6 @@ void mapInt128() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP8]], align 4
 // CHECK3-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1490,6 +1490,7 @@ void mapInt128() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [2 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    store ptr [[X]], ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
@@ -1504,7 +1505,6 @@ void mapInt128() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP8]], align 4
 // CHECK3-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1640,6 +1640,7 @@ void mapInt128() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    store ptr [[X]], ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
@@ -1648,7 +1649,6 @@ void mapInt128() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1718,6 +1718,7 @@ void mapInt128() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    store ptr [[X]], ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
@@ -1726,7 +1727,6 @@ void mapInt128() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1796,6 +1796,7 @@ void mapInt128() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    store ptr [[X]], ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
@@ -1804,7 +1805,6 @@ void mapInt128() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1876,9 +1876,11 @@ void mapInt128() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [3 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS4:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    store ptr [[Y]], ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
@@ -1899,7 +1901,6 @@ void mapInt128() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP8]], align 4
 // CHECK3-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP11]], align 4
 // CHECK3-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1953,7 +1954,6 @@ void mapInt128() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP34]], align 4
 // CHECK3-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS4:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS4]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP37]], align 4
 // CHECK3-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS4]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_num_teams_codegen.cpp b/clang/test/OpenMP/target_teams_num_teams_codegen.cpp
index b25d10dac42c2..4aea3fbd932b2 100644
--- a/clang/test/OpenMP/target_teams_num_teams_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_num_teams_codegen.cpp
@@ -213,9 +213,11 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
@@ -253,7 +255,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP17:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
 // CHECK1-NEXT:    [[TMP18:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP17]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP19]], align 4
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -296,7 +297,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP36]], align 8
 // CHECK1-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP39]], align 4
 // CHECK1-NEXT:    [[TMP40:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -345,11 +345,13 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED2:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[DOTCAPTURE_EXPR_]], align 4
@@ -366,7 +368,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP8:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
 // CHECK1-NEXT:    [[TMP9:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP8]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP10]], align 4
 // CHECK1-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -416,7 +417,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP33:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK1-NEXT:    [[TMP34:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP33]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP35]], align 4
 // CHECK1-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -460,6 +460,7 @@ int bar(int n){
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[N_ADDR:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[A:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[B:%.*]] = alloca i16, align 2
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i16, align 2
 // CHECK1-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
@@ -468,9 +469,9 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS1:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -539,7 +540,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP33:%.*]] = load i16, ptr [[DOTCAPTURE_EXPR_]], align 2
 // CHECK1-NEXT:    [[TMP34:%.*]] = sext i16 [[TMP33]] to i32
 // CHECK1-NEXT:    [[TMP35:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP34]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS1:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS1]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP36]], align 4
 // CHECK1-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS1]], i32 0, i32 1
@@ -798,9 +798,11 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
@@ -838,7 +840,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP17:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
 // CHECK3-NEXT:    [[TMP18:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP17]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP19]], align 4
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -881,7 +882,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP36]], align 4
 // CHECK3-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP39]], align 4
 // CHECK3-NEXT:    [[TMP40:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -930,11 +930,13 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR__CASTED2:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[DOTCAPTURE_EXPR_]], align 4
@@ -951,7 +953,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP8:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
 // CHECK3-NEXT:    [[TMP9:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP8]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP10]], align 4
 // CHECK3-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1001,7 +1002,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP33:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK3-NEXT:    [[TMP34:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP33]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP35]], align 4
 // CHECK3-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -1045,6 +1045,7 @@ int bar(int n){
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[N_ADDR:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[A:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[B:%.*]] = alloca i16, align 2
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i16, align 2
 // CHECK3-NEXT:    [[A_CASTED:%.*]] = alloca i32, align 4
@@ -1053,9 +1054,9 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS1:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1124,7 +1125,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP33:%.*]] = load i16, ptr [[DOTCAPTURE_EXPR_]], align 2
 // CHECK3-NEXT:    [[TMP34:%.*]] = sext i16 [[TMP33]] to i32
 // CHECK3-NEXT:    [[TMP35:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP34]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS1:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS1]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP36]], align 4
 // CHECK3-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS1]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_teams_thread_limit_codegen.cpp b/clang/test/OpenMP/target_teams_thread_limit_codegen.cpp
index 6a1dfdfa5fca0..377d983b72786 100644
--- a/clang/test/OpenMP/target_teams_thread_limit_codegen.cpp
+++ b/clang/test/OpenMP/target_teams_thread_limit_codegen.cpp
@@ -213,9 +213,11 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
@@ -253,7 +255,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP17:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
 // CHECK1-NEXT:    [[TMP18:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP17]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP19]], align 4
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -296,7 +297,6 @@ int bar(int n){
 // CHECK1-NEXT:    store ptr null, ptr [[TMP36]], align 8
 // CHECK1-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP39]], align 4
 // CHECK1-NEXT:    [[TMP40:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -347,11 +347,13 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [2 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR__CASTED4:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS5:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS7:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[DOTCAPTURE_EXPR_]], align 4
@@ -382,7 +384,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP15:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP14]], 0
 // CHECK1-NEXT:    [[TMP17:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP15]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -432,7 +433,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP40:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS6]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP41:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_3]], align 4
 // CHECK1-NEXT:    [[TMP42:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP41]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP43]], align 4
 // CHECK1-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 1
@@ -476,6 +476,7 @@ int bar(int n){
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[N_ADDR:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[A:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[B:%.*]] = alloca i16, align 2
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i16, align 2
 // CHECK1-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
@@ -484,9 +485,9 @@ int bar(int n){
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS1:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    store i32 0, ptr [[A]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -555,7 +556,6 @@ int bar(int n){
 // CHECK1-NEXT:    [[TMP33:%.*]] = load i16, ptr [[DOTCAPTURE_EXPR_]], align 2
 // CHECK1-NEXT:    [[TMP34:%.*]] = sext i16 [[TMP33]] to i32
 // CHECK1-NEXT:    [[TMP35:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP34]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS1:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS1]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP36]], align 4
 // CHECK1-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS1]], i32 0, i32 1
@@ -817,9 +817,11 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
@@ -857,7 +859,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP17:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_]], align 4
 // CHECK3-NEXT:    [[TMP18:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP17]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP19]], align 4
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -900,7 +901,6 @@ int bar(int n){
 // CHECK3-NEXT:    store ptr null, ptr [[TMP36]], align 4
 // CHECK3-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS6:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP39]], align 4
 // CHECK3-NEXT:    [[TMP40:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS6]], i32 0, i32 1
@@ -951,11 +951,13 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [2 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR__CASTED4:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS5:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS7:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[DOTCAPTURE_EXPR_]], align 4
@@ -986,7 +988,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP15:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK3-NEXT:    [[TMP16:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP14]], 0
 // CHECK3-NEXT:    [[TMP17:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP15]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1036,7 +1037,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP40:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS6]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP41:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_3]], align 4
 // CHECK3-NEXT:    [[TMP42:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP41]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP43]], align 4
 // CHECK3-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 1
@@ -1080,6 +1080,7 @@ int bar(int n){
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[N_ADDR:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[A:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[B:%.*]] = alloca i16, align 2
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i16, align 2
 // CHECK3-NEXT:    [[A_CASTED:%.*]] = alloca i32, align 4
@@ -1088,9 +1089,9 @@ int bar(int n){
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS1:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    store i32 0, ptr [[A]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1159,7 +1160,6 @@ int bar(int n){
 // CHECK3-NEXT:    [[TMP33:%.*]] = load i16, ptr [[DOTCAPTURE_EXPR_]], align 2
 // CHECK3-NEXT:    [[TMP34:%.*]] = sext i16 [[TMP33]] to i32
 // CHECK3-NEXT:    [[TMP35:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP34]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS1:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS1]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP36]], align 4
 // CHECK3-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS1]], i32 0, i32 1
diff --git a/clang/test/OpenMP/target_update_codegen.cpp b/clang/test/OpenMP/target_update_codegen.cpp
index 0e46715c37641..3d0550c35a182 100644
--- a/clang/test/OpenMP/target_update_codegen.cpp
+++ b/clang/test/OpenMP/target_update_codegen.cpp
@@ -748,18 +748,18 @@ void lvalue(struct S *s, int l, int e) {
 // SIMD-ONLY0-NOT: {{__kmpc|__tgt}}
 #ifdef CK13
 
-// CK13: [[SIZE00:@.+]] = {{.+}}constant [1 x i64] [i64 4]
-// CK13: [[MTYPE00:@.+]] = {{.+}}constant [1 x i64] [i64 1]
+// CK13: [[SIZE00:@.+]] = {{.+}}constant [2 x i64] [i64 {{4|8}}, i64 4]
+// CK13: [[MTYPE00:@.+]] = {{.+}}constant [2 x i64] [i64 1, i64 17]
 
 // CK13-LABEL: lvalue
 void lvalue(int **BB, int a, int b) {
 
-  // CK13-DAG: call void @__tgt_target_data_update_mapper(%struct.ident_t* @{{.+}}, i64 -1, i32 1, i8** [[GEPBP:%.+]], i8** [[GEPP:%.+]], {{.+}}getelementptr {{.+}}[1 x i{{.+}}]* [[SIZE00]], {{.+}}getelementptr {{.+}}[1 x i{{.+}}]* [[MTYPE00]]{{.+}}, i8** null)
+  // CK13-DAG: call void @__tgt_target_data_update_mapper(%struct.ident_t* @{{.+}}, i64 -1, i32 2, i8** [[GEPBP:%.+]], i8** [[GEPP:%.+]], {{.+}}getelementptr {{.+}}[2 x i{{.+}}]* [[SIZE00]], {{.+}}getelementptr {{.+}}[2 x i{{.+}}]* [[MTYPE00]]{{.+}}, i8** null)
   // CK13-DAG: [[GEPBP]] = getelementptr inbounds {{.+}}[[BP:%[^,]+]]
   // CK13-DAG: [[GEPP]] = getelementptr inbounds {{.+}}[[P:%[^,]+]]
 
-  // CK13-DAG: [[BP0:%.+]] = getelementptr inbounds {{.+}}[[BP]], i{{.+}} 0, i{{.+}} 0
-  // CK13-DAG: [[P0:%.+]] = getelementptr inbounds {{.+}}[[P]], i{{.+}} 0, i{{.+}} 0
+  // CK13-DAG: [[BP0:%.+]] = getelementptr inbounds {{.+}}[[BP]], i{{.+}} 0, i{{.+}} 1
+  // CK13-DAG: [[P0:%.+]] = getelementptr inbounds {{.+}}[[P]], i{{.+}} 0, i{{.+}} 1
   // CK13-DAG: [[BPC0:%.+]] = bitcast i8** [[BP0]] to i32***
   // CK13-DAG: [[PC0:%.+]] = bitcast i8** [[P0]] to i32**
   // CK13-DAG: store i32** [[B_VAL1:%.+]], i32*** [[BPC0]]
@@ -772,7 +772,6 @@ void lvalue(int **BB, int a, int b) {
   // CK13-32-DAG: [[ADD_PTR]] = getelementptr inbounds i32*, i32** [[B_VAL:%.+]], i32 [[A_ADDR:%.+]]
   // CK13-64-DAG: [[IDX_EXT]] = sext i32 [[TWO:%.+]] to i64
   // CK13-DAG: [[B_VAL]] = load i32**, i32*** [[BB_ADDR:%.+]]
-  // CK13-DAG: [[B_VAL1]] = load i32**, i32*** [[BB_ADDR]]
   #pragma omp target update to(*(*(BB+a)+b))
   *(*(BB+a)+b) = 1;
   #pragma omp target update from(*(*(BB+a)+b))
@@ -977,8 +976,8 @@ void lvalue_find_base(float *f, int *i) {
 // SIMD-ONLY0-NOT: {{__kmpc|__tgt}}
 #ifdef CK17
 
-// CK17: [[SIZE00:@.+]] = {{.+}}constant [1 x i64] [i64 4]
-// CK17: [[MTYPE00:@.+]] = {{.+}}constant [1 x i64] [i64 1]
+// CK17: [[SIZE00:@.+]] = {{.+}}constant [2 x i64] [i64 {{8|4}}, i64 4]
+// CK17: [[MTYPE00:@.+]] = {{.+}}constant [2 x i64] [i64 1, i64 17]
 
 struct SSA {
   int i;
@@ -988,12 +987,12 @@ struct SSA {
 //CK17-LABEL: lvalue_find_base
 void lvalue_find_base(float **f, SSA *sa) {
 
-  // CK17-DAG: call void @__tgt_target_data_update_mapper(%struct.ident_t* @{{.+}}, i64 -1, i32 1, i8** [[GEPBP:%.+]], i8** [[GEPP:%.+]], {{.+}}getelementptr {{.+}}[1 x i{{.+}}]* [[SIZE00]], {{.+}}getelementptr {{.+}}[1 x i{{.+}}]* [[MTYPE00]]{{.+}}, i8** null)
+  // CK17-DAG: call void @__tgt_target_data_update_mapper(%struct.ident_t* @{{.+}}, i64 -1, i32 2, i8** [[GEPBP:%.+]], i8** [[GEPP:%.+]], {{.+}}getelementptr {{.+}}[2 x i{{.+}}]* [[SIZE00]], {{.+}}getelementptr {{.+}}[2 x i{{.+}}]* [[MTYPE00]]{{.+}}, i8** null)
   // CK17-DAG: [[GEPBP]] = getelementptr inbounds {{.+}}[[BP:%[^,]+]]
   // CK17-DAG: [[GEPP]] = getelementptr inbounds {{.+}}[[P:%[^,]+]]
 
-  // CK17-DAG: [[BP0:%.+]] = getelementptr inbounds {{.+}}[[BP]], i{{.+}} 0, i{{.+}} 0
-  // CK17-DAG: [[P0:%.+]] = getelementptr inbounds {{.+}}[[P]], i{{.+}} 0, i{{.+}} 0
+  // CK17-DAG: [[BP0:%.+]] = getelementptr inbounds {{.+}}[[BP]], i{{.+}} 0, i{{.+}} 1
+  // CK17-DAG: [[P0:%.+]] = getelementptr inbounds {{.+}}[[P]], i{{.+}} 0, i{{.+}} 1
   // CK17-DAG: [[BPC0:%.+]] = bitcast i8** [[BP0]] to float***
   // CK17-DAG: [[PC0:%.+]] = bitcast i8** [[P0]] to float**
   // CK17-DAG: store float** [[F_VAL:%.+]], float*** [[BPC0]],
@@ -1011,7 +1010,6 @@ void lvalue_find_base(float **f, SSA *sa) {
   // CK17-DAG: [[I_2]] = getelementptr inbounds [[SSA:%.+]], [[SSA]]* [[FOUR:%.+]], i32 0, i32 0
   // CK17-DAG: [[FOUR]] = load [[SSA]]*, [[SSA]]** [[SSA_ADDR:%.+]],
   // CK17-DAG: [[F]] = load float**, float*** [[F_ADDR:%.+]],
-  // CK17-DAG: [[F_VAL]] = load float**, float*** [[F_ADDR]],
 
   #pragma omp target update to(*(sa->sa->i+*(1+sa->i+f)))
   *(sa->sa->i+*(1+sa->i+f)) = 1;
diff --git a/clang/test/OpenMP/teams_codegen.cpp b/clang/test/OpenMP/teams_codegen.cpp
index 9d6da1ec7fd4e..3c6bd6dbcf883 100644
--- a/clang/test/OpenMP/teams_codegen.cpp
+++ b/clang/test/OpenMP/teams_codegen.cpp
@@ -309,20 +309,24 @@ void foo() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[COMP_CASTED1:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS4:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[LA_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[COMP_CASTED8:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS9:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS11:%.*]] = alloca [2 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[LA_CASTED15:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[COMP_CASTED16:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS17:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS18:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS19:%.*]] = alloca [2 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS20:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[GBLA_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[GBLB_CASTED:%.*]] = alloca i64, align 8
@@ -331,6 +335,7 @@ void foo() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS24:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS25:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS26:%.*]] = alloca [5 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[GBLC_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[COMP_CASTED31:%.*]] = alloca i64, align 8
@@ -339,6 +344,7 @@ void foo() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS34:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP35:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[_TMP37:%.*]] = alloca ptr, align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS39:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[A]], ptr [[A_ADDR]], align 4
 // CHECK1-NEXT:    store i32 1, ptr [[COMP]], align 4
 // CHECK1-NEXT:    store i32 23, ptr [[LA]], align 4
@@ -354,7 +360,6 @@ void foo() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -399,7 +404,6 @@ void foo() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP26]], align 8
 // CHECK1-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS2]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS3]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK1-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -455,7 +459,6 @@ void foo() {
 // CHECK1-NEXT:    [[TMP55:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS10]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP56:%.*]] = load i32, ptr [[LA]], align 4
 // CHECK1-NEXT:    [[TMP57:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP56]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK1-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -511,7 +514,6 @@ void foo() {
 // CHECK1-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS18]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP85:%.*]] = load i32, ptr [[LA]], align 4
 // CHECK1-NEXT:    [[TMP86:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP85]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS20:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP87:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS20]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP87]], align 4
 // CHECK1-NEXT:    [[TMP88:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS20]], i32 0, i32 1
@@ -602,7 +604,6 @@ void foo() {
 // CHECK1-NEXT:    [[TMP133:%.*]] = trunc i64 [[ADD27]] to i32
 // CHECK1-NEXT:    [[TMP134:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[ADD]], 0
 // CHECK1-NEXT:    [[TMP135:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP133]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP136:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP136]], align 4
 // CHECK1-NEXT:    [[TMP137:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 1
@@ -668,7 +669,6 @@ void foo() {
 // CHECK1-NEXT:    [[ADD38:%.*]] = add nsw i32 [[TMP167]], 2
 // CHECK1-NEXT:    [[TMP168:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[ADD36]], 0
 // CHECK1-NEXT:    [[TMP169:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[ADD38]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS39:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP170:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS39]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP170]], align 4
 // CHECK1-NEXT:    [[TMP171:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS39]], i32 0, i32 1
@@ -919,20 +919,24 @@ void foo() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[COMP_CASTED1:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS4:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[LA_CASTED:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[COMP_CASTED8:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS9:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS10:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS11:%.*]] = alloca [2 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[LA_CASTED15:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[COMP_CASTED16:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS17:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS18:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS19:%.*]] = alloca [2 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS20:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[GBLA_CASTED:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[A_CASTED:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[LC_CASTED:%.*]] = alloca i32, align 4
@@ -940,6 +944,7 @@ void foo() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS24:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS25:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS26:%.*]] = alloca [5 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[GBLC_CASTED:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[COMP_CASTED31:%.*]] = alloca i32, align 4
@@ -948,6 +953,7 @@ void foo() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS34:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP35:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[_TMP37:%.*]] = alloca ptr, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS39:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 [[A]], ptr [[A_ADDR]], align 4
 // CHECK3-NEXT:    store i32 1, ptr [[COMP]], align 4
 // CHECK3-NEXT:    store i32 23, ptr [[LA]], align 4
@@ -963,7 +969,6 @@ void foo() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP4]], align 4
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1008,7 +1013,6 @@ void foo() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP26]], align 4
 // CHECK3-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS2]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS3]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK3-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -1064,7 +1068,6 @@ void foo() {
 // CHECK3-NEXT:    [[TMP55:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS10]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP56:%.*]] = load i32, ptr [[LA]], align 4
 // CHECK3-NEXT:    [[TMP57:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP56]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK3-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -1120,7 +1123,6 @@ void foo() {
 // CHECK3-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS18]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP85:%.*]] = load i32, ptr [[LA]], align 4
 // CHECK3-NEXT:    [[TMP86:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP85]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS20:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP87:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS20]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP87]], align 4
 // CHECK3-NEXT:    [[TMP88:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS20]], i32 0, i32 1
@@ -1208,7 +1210,6 @@ void foo() {
 // CHECK3-NEXT:    [[TMP131:%.*]] = trunc i64 [[ADD27]] to i32
 // CHECK3-NEXT:    [[TMP132:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[ADD]], 0
 // CHECK3-NEXT:    [[TMP133:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP131]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS28:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP134:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP134]], align 4
 // CHECK3-NEXT:    [[TMP135:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS28]], i32 0, i32 1
@@ -1274,7 +1275,6 @@ void foo() {
 // CHECK3-NEXT:    [[ADD38:%.*]] = add nsw i32 [[TMP165]], 2
 // CHECK3-NEXT:    [[TMP166:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[ADD36]], 0
 // CHECK3-NEXT:    [[TMP167:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[ADD38]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS39:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP168:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS39]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP168]], align 4
 // CHECK3-NEXT:    [[TMP169:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS39]], i32 0, i32 1
@@ -1528,10 +1528,12 @@ void foo() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[COMP_CASTED1:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS2:%.*]] = alloca [3 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS3:%.*]] = alloca [3 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS4:%.*]] = alloca [3 x ptr], align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 1, ptr [[COMP]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = load i32, ptr [[COMP]], align 4
 // CHECK9-NEXT:    store i32 [[TMP0]], ptr [[COMP_CASTED]], align 4
@@ -1563,7 +1565,6 @@ void foo() {
 // CHECK9-NEXT:    [[TMP15:%.*]] = trunc i64 [[CONV]] to i32
 // CHECK9-NEXT:    [[TMP16:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP13]], 0
 // CHECK9-NEXT:    [[TMP17:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP15]], 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK9-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1628,7 +1629,6 @@ void foo() {
 // CHECK9-NEXT:    [[TMP49:%.*]] = trunc i64 [[TMP48]] to i32
 // CHECK9-NEXT:    [[TMP50:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP47]], 0
 // CHECK9-NEXT:    [[TMP51:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP49]], 0
-// CHECK9-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP52]], align 4
 // CHECK9-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -1762,10 +1762,12 @@ void foo() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[COMP_CASTED1:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS2:%.*]] = alloca [3 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS3:%.*]] = alloca [3 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS4:%.*]] = alloca [3 x ptr], align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 1, ptr [[COMP]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = load i32, ptr [[COMP]], align 4
 // CHECK11-NEXT:    store i32 [[TMP0]], ptr [[COMP_CASTED]], align 4
@@ -1797,7 +1799,6 @@ void foo() {
 // CHECK11-NEXT:    [[TMP15:%.*]] = trunc i64 [[CONV]] to i32
 // CHECK11-NEXT:    [[TMP16:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP13]], 0
 // CHECK11-NEXT:    [[TMP17:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP15]], 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK11-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1862,7 +1863,6 @@ void foo() {
 // CHECK11-NEXT:    [[TMP49:%.*]] = trunc i64 [[TMP48]] to i32
 // CHECK11-NEXT:    [[TMP50:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP47]], 0
 // CHECK11-NEXT:    [[TMP51:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP49]], 0
-// CHECK11-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP52]], align 4
 // CHECK11-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -2003,10 +2003,12 @@ void foo() {
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [2 x ptr], align 8
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[COMP_CASTED3:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS6:%.*]] = alloca [2 x ptr], align 8
+// CHECK17-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK17-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK17-NEXT:    store i32 1, ptr [[COMP]], align 4
@@ -2031,7 +2033,6 @@ void foo() {
 // CHECK17-NEXT:    [[A2:%.*]] = getelementptr inbounds [[STRUCT_SS]], ptr [[THIS1]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP10:%.*]] = load i32, ptr [[A2]], align 4
 // CHECK17-NEXT:    [[TMP11:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP10]], 0
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP12]], align 4
 // CHECK17-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2088,7 +2089,6 @@ void foo() {
 // CHECK17-NEXT:    [[CONV:%.*]] = fptosi float [[TMP37]] to i32
 // CHECK17-NEXT:    [[ADD:%.*]] = add nsw i32 [[CONV]], 123
 // CHECK17-NEXT:    [[TMP38:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[ADD]], 0
-// CHECK17-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP39]], align 4
 // CHECK17-NEXT:    [[TMP40:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 1
@@ -2216,10 +2216,12 @@ void foo() {
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [2 x ptr], align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[COMP_CASTED3:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS6:%.*]] = alloca [2 x ptr], align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK19-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK19-NEXT:    store i32 1, ptr [[COMP]], align 4
@@ -2244,7 +2246,6 @@ void foo() {
 // CHECK19-NEXT:    [[A2:%.*]] = getelementptr inbounds [[STRUCT_SS]], ptr [[THIS1]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP10:%.*]] = load i32, ptr [[A2]], align 4
 // CHECK19-NEXT:    [[TMP11:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP10]], 0
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP12]], align 4
 // CHECK19-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2301,7 +2302,6 @@ void foo() {
 // CHECK19-NEXT:    [[CONV:%.*]] = fptosi float [[TMP37]] to i32
 // CHECK19-NEXT:    [[ADD:%.*]] = add nsw i32 [[CONV]], 123
 // CHECK19-NEXT:    [[TMP38:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[ADD]], 0
-// CHECK19-NEXT:    [[KERNEL_ARGS8:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP39]], align 4
 // CHECK19-NEXT:    [[TMP40:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS8]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_codegen.cpp b/clang/test/OpenMP/teams_distribute_codegen.cpp
index a11bae68a7e5a..87d6ca297233c 100644
--- a/clang/test/OpenMP/teams_distribute_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_codegen.cpp
@@ -190,6 +190,7 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[N_CASTED4:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS5:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [2 x ptr], align 8
@@ -197,6 +198,7 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[DIV:%.*]] = sdiv i32 [[TMP0]], 128
@@ -251,7 +253,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[TMP26:%.*]] = zext i32 [[ADD]] to i64
 // CHECK1-NEXT:    [[TMP27:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP21]], 0
 // CHECK1-NEXT:    [[TMP28:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP22]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK1-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -312,7 +313,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[TMP56:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK1-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP56]], 1
 // CHECK1-NEXT:    [[TMP57:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK1-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -588,6 +588,7 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[N_CASTED4:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS5:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [2 x ptr], align 4
@@ -595,6 +596,7 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[DIV:%.*]] = sdiv i32 [[TMP0]], 128
@@ -649,7 +651,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[TMP26:%.*]] = zext i32 [[ADD]] to i64
 // CHECK3-NEXT:    [[TMP27:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP21]], 0
 // CHECK3-NEXT:    [[TMP28:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP22]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK3-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -710,7 +711,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[TMP56:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK3-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP56]], 1
 // CHECK3-NEXT:    [[TMP57:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK3-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -983,6 +983,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 100, ptr [[N]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = zext i32 [[TMP0]] to i64
@@ -1028,7 +1029,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK9-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1200,6 +1200,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 100, ptr [[N]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = call ptr @llvm.stacksave()
@@ -1245,7 +1246,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK11-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1418,6 +1418,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK17-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK17-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -1429,7 +1430,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK17-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK17-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1570,6 +1570,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK19-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK19-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -1581,7 +1582,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK19-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK19-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1722,6 +1722,7 @@ int main (int argc, char **argv) {
 // CHECK25-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK25-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK25-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK25-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK25-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK25-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK25-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -1770,7 +1771,6 @@ int main (int argc, char **argv) {
 // CHECK25-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK25-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK25-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK25-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK25-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK25-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK25-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1936,6 +1936,7 @@ int main (int argc, char **argv) {
 // CHECK25-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK25-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
 // CHECK25-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK25-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK25-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK25-NEXT:    store i32 0, ptr [[TE]], align 4
 // CHECK25-NEXT:    store i32 128, ptr [[TH]], align 4
@@ -1969,7 +1970,6 @@ int main (int argc, char **argv) {
 // CHECK25-NEXT:    [[TMP16:%.*]] = load i32, ptr [[TH]], align 4
 // CHECK25-NEXT:    [[TMP17:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP15]], 0
 // CHECK25-NEXT:    [[TMP18:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP16]], 0
-// CHECK25-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK25-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK25-NEXT:    store i32 2, ptr [[TMP19]], align 4
 // CHECK25-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2115,6 +2115,7 @@ int main (int argc, char **argv) {
 // CHECK27-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK27-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK27-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK27-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK27-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK27-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK27-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -2163,7 +2164,6 @@ int main (int argc, char **argv) {
 // CHECK27-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK27-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK27-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK27-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK27-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK27-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK27-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2328,6 +2328,7 @@ int main (int argc, char **argv) {
 // CHECK27-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK27-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
 // CHECK27-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK27-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK27-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK27-NEXT:    store i32 0, ptr [[TE]], align 4
 // CHECK27-NEXT:    store i32 128, ptr [[TH]], align 4
@@ -2361,7 +2362,6 @@ int main (int argc, char **argv) {
 // CHECK27-NEXT:    [[TMP16:%.*]] = load i32, ptr [[TH]], align 4
 // CHECK27-NEXT:    [[TMP17:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP15]], 0
 // CHECK27-NEXT:    [[TMP18:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP16]], 0
-// CHECK27-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK27-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK27-NEXT:    store i32 2, ptr [[TMP19]], align 4
 // CHECK27-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_collapse_codegen.cpp b/clang/test/OpenMP/teams_distribute_collapse_codegen.cpp
index 256c823223aa3..98156e8a99f8b 100644
--- a/clang/test/OpenMP/teams_distribute_collapse_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_collapse_codegen.cpp
@@ -114,6 +114,7 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -125,7 +126,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -282,6 +282,7 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -293,7 +294,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -453,6 +453,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i64, align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -529,7 +530,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store i64 [[SUB7]], ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK9-NEXT:    [[TMP35:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i64 [[TMP35]], 1
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP36]], align 4
 // CHECK9-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -750,6 +750,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    store ptr [[A]], ptr [[TMP0]], align 8
@@ -759,7 +760,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK9-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -916,6 +916,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i64, align 8
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -991,7 +992,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store i64 [[SUB7]], ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK11-NEXT:    [[TMP34:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i64 [[TMP34]], 1
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP35]], align 4
 // CHECK11-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1210,6 +1210,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    store ptr [[A]], ptr [[TMP0]], align 4
@@ -1219,7 +1220,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK11-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_dist_schedule_codegen.cpp b/clang/test/OpenMP/teams_distribute_dist_schedule_codegen.cpp
index 94e47786fd9f7..d0781b8ba88c6 100644
--- a/clang/test/OpenMP/teams_distribute_dist_schedule_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_dist_schedule_codegen.cpp
@@ -147,14 +147,17 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -166,7 +169,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -209,7 +211,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK1-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -252,7 +253,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK1-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK1-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -566,14 +566,17 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -585,7 +588,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -628,7 +630,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK3-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -671,7 +672,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK3-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK3-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -983,6 +983,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 8
@@ -991,6 +992,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[N_CASTED18:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [3 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [3 x ptr], align 8
@@ -999,6 +1001,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_24:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -1047,7 +1050,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK9-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1119,7 +1121,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP56:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK9-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP56]], 1
 // CHECK9-NEXT:    [[TMP57:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK9-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1191,7 +1192,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP91:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_25]], align 4
 // CHECK9-NEXT:    [[ADD29:%.*]] = add nsw i32 [[TMP91]], 1
 // CHECK9-NEXT:    [[TMP92:%.*]] = zext i32 [[ADD29]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP93:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP93]], align 4
 // CHECK9-NEXT:    [[TMP94:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 1
@@ -1600,14 +1600,17 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    store ptr [[A]], ptr [[TMP0]], align 8
@@ -1617,7 +1620,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK9-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1659,7 +1661,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -1701,7 +1702,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK9-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK9-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -2010,6 +2010,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 4
@@ -2018,6 +2019,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[N_CASTED18:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [3 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [3 x ptr], align 4
@@ -2026,6 +2028,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_24:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -2074,7 +2077,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK11-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2147,7 +2149,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP57:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK11-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP57]], 1
 // CHECK11-NEXT:    [[TMP58:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP59]], align 4
 // CHECK11-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -2220,7 +2221,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP93:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_25]], align 4
 // CHECK11-NEXT:    [[ADD29:%.*]] = add nsw i32 [[TMP93]], 1
 // CHECK11-NEXT:    [[TMP94:%.*]] = zext i32 [[ADD29]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP95]], align 4
 // CHECK11-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 1
@@ -2626,14 +2626,17 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    store ptr [[A]], ptr [[TMP0]], align 4
@@ -2643,7 +2646,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK11-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2685,7 +2687,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -2727,7 +2728,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK11-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK11-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_firstprivate_codegen.cpp b/clang/test/OpenMP/teams_distribute_firstprivate_codegen.cpp
index 3f07941cc1b57..571206b301715 100644
--- a/clang/test/OpenMP/teams_distribute_firstprivate_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_firstprivate_codegen.cpp
@@ -265,6 +265,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr @t_var, align 4
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[T_VAR_CASTED]], align 4
@@ -304,7 +305,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP18]], align 8
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK1-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -545,6 +545,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -587,7 +588,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP17]], align 8
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK1-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1076,6 +1076,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr @t_var, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[T_VAR_CASTED]], align 4
@@ -1115,7 +1116,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK3-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1354,6 +1354,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1396,7 +1397,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP17]], align 4
 // CHECK3-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK3-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_lastprivate_codegen.cpp b/clang/test/OpenMP/teams_distribute_lastprivate_codegen.cpp
index a3c4723bad37b..ff70b5fab6114 100644
--- a/clang/test/OpenMP/teams_distribute_lastprivate_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_lastprivate_codegen.cpp
@@ -476,6 +476,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store ptr [[G]], ptr [[G1]], align 8
@@ -530,7 +531,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -799,6 +799,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK9-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -841,7 +842,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP17]], align 8
 // CHECK9-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK9-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1179,6 +1179,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store ptr [[G]], ptr [[G1]], align 4
@@ -1233,7 +1234,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1500,6 +1500,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK11-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1542,7 +1543,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP17]], align 4
 // CHECK11-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK11-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_codegen.cpp
index f7046f352bf6c..00d54c6f8ae2a 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_codegen.cpp
@@ -190,6 +190,7 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[N_CASTED4:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS5:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [2 x ptr], align 8
@@ -197,6 +198,7 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[DIV:%.*]] = sdiv i32 [[TMP0]], 128
@@ -249,7 +251,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP24]], 1
 // CHECK1-NEXT:    [[TMP25:%.*]] = zext i32 [[ADD]] to i64
 // CHECK1-NEXT:    [[TMP26:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP21]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP27]], align 4
 // CHECK1-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -310,7 +311,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[TMP54:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK1-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP54]], 1
 // CHECK1-NEXT:    [[TMP55:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP56:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP56]], align 4
 // CHECK1-NEXT:    [[TMP57:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -797,6 +797,7 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[N_CASTED4:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS5:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [2 x ptr], align 4
@@ -804,6 +805,7 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[DIV:%.*]] = sdiv i32 [[TMP0]], 128
@@ -856,7 +858,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP24]], 1
 // CHECK3-NEXT:    [[TMP25:%.*]] = zext i32 [[ADD]] to i64
 // CHECK3-NEXT:    [[TMP26:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP21]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP27]], align 4
 // CHECK3-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -917,7 +918,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[TMP54:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK3-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP54]], 1
 // CHECK3-NEXT:    [[TMP55:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP56:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP56]], align 4
 // CHECK3-NEXT:    [[TMP57:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1393,6 +1393,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 100, ptr [[N]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = zext i32 [[TMP0]] to i64
@@ -1438,7 +1439,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK9-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1711,6 +1711,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 100, ptr [[N]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = call ptr @llvm.stacksave()
@@ -1756,7 +1757,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK11-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2026,6 +2026,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK17-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK17-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -2037,7 +2038,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK17-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK17-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2251,6 +2251,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK19-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK19-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -2262,7 +2263,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK19-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK19-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2472,6 +2472,7 @@ int main (int argc, char **argv) {
 // CHECK25-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK25-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK25-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK25-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK25-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK25-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK25-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -2520,7 +2521,6 @@ int main (int argc, char **argv) {
 // CHECK25-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK25-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK25-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK25-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK25-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK25-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK25-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2787,6 +2787,7 @@ int main (int argc, char **argv) {
 // CHECK25-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK25-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
 // CHECK25-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK25-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK25-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK25-NEXT:    store i32 0, ptr [[TE]], align 4
 // CHECK25-NEXT:    store i32 128, ptr [[TH]], align 4
@@ -2818,7 +2819,6 @@ int main (int argc, char **argv) {
 // CHECK25-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK25-NEXT:    [[TMP15:%.*]] = load i32, ptr [[TE]], align 4
 // CHECK25-NEXT:    [[TMP16:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP15]], 0
-// CHECK25-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK25-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK25-NEXT:    store i32 2, ptr [[TMP17]], align 4
 // CHECK25-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3037,6 +3037,7 @@ int main (int argc, char **argv) {
 // CHECK27-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK27-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK27-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK27-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK27-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK27-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK27-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -3085,7 +3086,6 @@ int main (int argc, char **argv) {
 // CHECK27-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK27-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK27-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK27-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK27-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK27-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK27-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3347,6 +3347,7 @@ int main (int argc, char **argv) {
 // CHECK27-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK27-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
 // CHECK27-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK27-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK27-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK27-NEXT:    store i32 0, ptr [[TE]], align 4
 // CHECK27-NEXT:    store i32 128, ptr [[TH]], align 4
@@ -3378,7 +3379,6 @@ int main (int argc, char **argv) {
 // CHECK27-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK27-NEXT:    [[TMP15:%.*]] = load i32, ptr [[TE]], align 4
 // CHECK27-NEXT:    [[TMP16:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP15]], 0
-// CHECK27-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK27-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK27-NEXT:    store i32 2, ptr [[TMP17]], align 4
 // CHECK27-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_collapse_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_collapse_codegen.cpp
index e4501d975a531..7841cfd908ba4 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_collapse_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_collapse_codegen.cpp
@@ -119,6 +119,7 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -130,7 +131,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -362,6 +362,7 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -373,7 +374,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -604,6 +604,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i64, align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -680,7 +681,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store i64 [[SUB7]], ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK9-NEXT:    [[TMP35:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i64 [[TMP35]], 1
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP36]], align 4
 // CHECK9-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1021,6 +1021,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    store ptr [[A]], ptr [[TMP0]], align 8
@@ -1030,7 +1031,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK9-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1262,6 +1262,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i64, align 8
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -1337,7 +1338,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store i64 [[SUB7]], ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK11-NEXT:    [[TMP34:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i64 [[TMP34]], 1
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP35]], align 4
 // CHECK11-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1680,6 +1680,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    store ptr [[A]], ptr [[TMP0]], align 4
@@ -1689,7 +1690,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK11-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_copyin_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_copyin_codegen.cpp
index 6df07faa00ee0..229914d64bee8 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_copyin_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_copyin_codegen.cpp
@@ -102,6 +102,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = call align 4 ptr @llvm.threadlocal.address.p0(ptr align 4 @x)
 // CHECK1-NEXT:    [[TMP1:%.*]] = load i32, ptr [[TMP0]], align 4
@@ -121,7 +122,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP8]], align 8
 // CHECK1-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP11]], align 4
 // CHECK1-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -327,6 +327,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = call align 4 ptr @llvm.threadlocal.address.p0(ptr align 4 @x)
 // CHECK1-NEXT:    [[TMP1:%.*]] = load i32, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    store i32 [[TMP1]], ptr [[X_CASTED]], align 4
@@ -345,7 +346,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP8]], align 8
 // CHECK1-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP11]], align 4
 // CHECK1-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -564,6 +564,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = call align 4 ptr @llvm.threadlocal.address.p0(ptr align 4 @x)
 // CHECK3-NEXT:    [[TMP1:%.*]] = load i32, ptr [[TMP0]], align 4
@@ -583,7 +584,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP8]], align 4
 // CHECK3-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP11]], align 4
 // CHECK3-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -784,6 +784,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = call align 4 ptr @llvm.threadlocal.address.p0(ptr align 4 @x)
 // CHECK3-NEXT:    [[TMP1:%.*]] = load i32, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    store i32 [[TMP1]], ptr [[X_CASTED]], align 4
@@ -802,7 +803,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP8]], align 4
 // CHECK3-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP11]], align 4
 // CHECK3-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_dist_schedule_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_dist_schedule_codegen.cpp
index 49667087c8673..30beb98b7b7f3 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_dist_schedule_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_dist_schedule_codegen.cpp
@@ -159,14 +159,17 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -178,7 +181,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -221,7 +223,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK1-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -264,7 +265,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK1-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK1-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -800,14 +800,17 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -819,7 +822,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -862,7 +864,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK3-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -905,7 +906,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK3-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK3-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1428,6 +1428,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 8
@@ -1436,6 +1437,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[M_CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[N_CASTED18:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [4 x ptr], align 8
@@ -1445,6 +1447,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_24:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -1494,7 +1497,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK9-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1566,7 +1568,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP56:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK9-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP56]], 1
 // CHECK9-NEXT:    [[TMP57:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK9-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1647,7 +1648,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP96:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_25]], align 4
 // CHECK9-NEXT:    [[ADD29:%.*]] = add nsw i32 [[TMP96]], 1
 // CHECK9-NEXT:    [[TMP97:%.*]] = zext i32 [[ADD29]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP98]], align 4
 // CHECK9-NEXT:    [[TMP99:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 1
@@ -2375,15 +2375,18 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[M_CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -2394,7 +2397,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK9-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2436,7 +2438,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -2487,7 +2488,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP47]], align 8
 // CHECK9-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP50]], align 4
 // CHECK9-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -3037,6 +3037,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 4
@@ -3045,6 +3046,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[M_CASTED:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[N_CASTED18:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [4 x ptr], align 4
@@ -3054,6 +3056,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_24:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -3103,7 +3106,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK11-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3176,7 +3178,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP57:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK11-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP57]], 1
 // CHECK11-NEXT:    [[TMP58:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP59]], align 4
 // CHECK11-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -3258,7 +3259,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP98:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_25]], align 4
 // CHECK11-NEXT:    [[ADD29:%.*]] = add nsw i32 [[TMP98]], 1
 // CHECK11-NEXT:    [[TMP99:%.*]] = zext i32 [[ADD29]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP100:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP100]], align 4
 // CHECK11-NEXT:    [[TMP101:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 1
@@ -3971,15 +3971,18 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[M_CASTED:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -3990,7 +3993,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK11-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4032,7 +4034,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -4083,7 +4084,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP47]], align 4
 // CHECK11-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP50]], align 4
 // CHECK11-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_firstprivate_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_firstprivate_codegen.cpp
index 1884ea8f6df8f..f97a598783d4e 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_firstprivate_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_firstprivate_codegen.cpp
@@ -299,6 +299,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr @t_var, align 4
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[T_VAR_CASTED]], align 4
@@ -338,7 +339,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP18]], align 8
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK1-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -709,6 +709,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -751,7 +752,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP17]], align 8
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK1-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1370,6 +1370,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr @t_var, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[T_VAR_CASTED]], align 4
@@ -1409,7 +1410,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK3-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1774,6 +1774,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1816,7 +1817,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP17]], align 4
 // CHECK3-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK3-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_if_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_if_codegen.cpp
index 718d4c5812622..22b580988dccc 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_if_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_if_codegen.cpp
@@ -117,8 +117,9 @@ int main() {
 // CHECK1-SAME: () #[[ATTR0:[0-9]+]] {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -152,7 +153,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9gtid_testv_l48() #[[ATTR2:[0-9]+]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -474,15 +474,17 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -516,7 +518,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}_main_l81() #[[ATTR2]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -569,7 +570,6 @@ int main() {
 // CHECK1-NEXT:    [[TOBOOL5:%.*]] = trunc i8 [[TMP38]] to i1
 // CHECK1-NEXT:    [[TMP39:%.*]] = select i1 [[TOBOOL5]], i32 0, i32 1
 // CHECK1-NEXT:    [[TMP40:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP39]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP41]], align 4
 // CHECK1-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -1061,15 +1061,17 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[ARG_ADDR:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[ARG]], ptr [[ARG_ADDR]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1103,7 +1105,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIiEiT__l62() #[[ATTR2]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -1156,7 +1157,6 @@ int main() {
 // CHECK1-NEXT:    [[TOBOOL5:%.*]] = trunc i8 [[TMP38]] to i1
 // CHECK1-NEXT:    [[TMP39:%.*]] = select i1 [[TOBOOL5]], i32 0, i32 1
 // CHECK1-NEXT:    [[TMP40:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP39]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP41]], align 4
 // CHECK1-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_lastprivate_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_lastprivate_codegen.cpp
index 25d1bca100ed0..97060d6d62498 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_lastprivate_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_lastprivate_codegen.cpp
@@ -724,6 +724,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store ptr [[G]], ptr [[G1]], align 8
@@ -778,7 +779,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1195,6 +1195,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK9-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -1237,7 +1238,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP17]], align 8
 // CHECK9-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK9-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1717,6 +1717,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store ptr [[G]], ptr [[G1]], align 4
@@ -1771,7 +1772,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2182,6 +2182,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK11-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -2224,7 +2225,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP17]], align 4
 // CHECK11-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK11-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_num_threads_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_num_threads_codegen.cpp
index a36a096858c56..ca6f26a731e6c 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_num_threads_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_num_threads_codegen.cpp
@@ -82,19 +82,20 @@ int main() {
 // CHECK1-NEXT:    [[EXN_SLOT:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[EHSELECTOR_SLOT:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    call void @_ZN1SC1El(ptr noundef nonnull align 8 dereferenceable(24) [[S]], i64 noundef 0)
 // CHECK1-NEXT:    [[CALL:%.*]] = invoke noundef signext i8 @_ZN1ScvcEv(ptr noundef nonnull align 8 dereferenceable(24) [[S]])
 // CHECK1-NEXT:    to label [[INVOKE_CONT:%.*]] unwind label [[LPAD:%.*]]
 // CHECK1:       invoke.cont:
 // CHECK1-NEXT:    store i8 [[CALL]], ptr [[A]], align 1
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -153,7 +154,6 @@ int main() {
 // CHECK1-NEXT:    [[TMP26:%.*]] = load i8, ptr [[DOTCAPTURE_EXPR_]], align 1
 // CHECK1-NEXT:    [[TMP27:%.*]] = zext i8 [[TMP26]] to i32
 // CHECK1-NEXT:    [[TMP28:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP27]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK1-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -552,8 +552,9 @@ int main() {
 // CHECK1-SAME: () #[[ATTR2]] comdat {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -587,7 +588,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIcLi5EEiv_l36() #[[ATTR6]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -628,10 +628,11 @@ int main() {
 // CHECK1-SAME: () #[[ATTR2]] comdat personality ptr @__gxx_personality_v0 {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[REF_TMP:%.*]] = alloca [[STRUCT_S:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -674,7 +675,6 @@ int main() {
 // CHECK1-NEXT:    [[TMP15:%.*]] = load i8, ptr [[DOTCAPTURE_EXPR_]], align 1
 // CHECK1-NEXT:    [[TMP16:%.*]] = zext i8 [[TMP15]] to i32
 // CHECK1-NEXT:    [[TMP17:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP16]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -1376,19 +1376,20 @@ int main() {
 // CHECK5-NEXT:    [[EXN_SLOT:%.*]] = alloca ptr, align 8
 // CHECK5-NEXT:    [[EHSELECTOR_SLOT:%.*]] = alloca i32, align 4
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK5-NEXT:    call void @_ZN1SC1El(ptr noundef nonnull align 8 dereferenceable(24) [[S]], i64 noundef 0)
 // CHECK5-NEXT:    [[CALL:%.*]] = invoke noundef signext i8 @_ZN1ScvcEv(ptr noundef nonnull align 8 dereferenceable(24) [[S]])
 // CHECK5-NEXT:    to label [[INVOKE_CONT:%.*]] unwind label [[LPAD:%.*]]
 // CHECK5:       invoke.cont:
 // CHECK5-NEXT:    store i8 [[CALL]], ptr [[A]], align 1
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK5-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1447,7 +1448,6 @@ int main() {
 // CHECK5-NEXT:    [[TMP26:%.*]] = load i8, ptr [[DOTCAPTURE_EXPR_]], align 1
 // CHECK5-NEXT:    [[TMP27:%.*]] = zext i8 [[TMP26]] to i32
 // CHECK5-NEXT:    [[TMP28:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP27]], 0
-// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK5-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -1846,8 +1846,9 @@ int main() {
 // CHECK5-SAME: () #[[ATTR2]] comdat {
 // CHECK5-NEXT:  entry:
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK5-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1881,7 +1882,6 @@ int main() {
 // CHECK5-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIcLi5EEiv_l36() #[[ATTR6]]
 // CHECK5-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK5:       omp_offload.cont:
-// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK5-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -1922,10 +1922,11 @@ int main() {
 // CHECK5-SAME: () #[[ATTR2]] comdat personality ptr @__gxx_personality_v0 {
 // CHECK5-NEXT:  entry:
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK5-NEXT:    [[REF_TMP:%.*]] = alloca [[STRUCT_S:%.*]], align 8
 // CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK5-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1968,7 +1969,6 @@ int main() {
 // CHECK5-NEXT:    [[TMP15:%.*]] = load i8, ptr [[DOTCAPTURE_EXPR_]], align 1
 // CHECK5-NEXT:    [[TMP16:%.*]] = zext i8 [[TMP15]] to i32
 // CHECK5-NEXT:    [[TMP17:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP16]], 0
-// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK5-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_private_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_private_codegen.cpp
index c3ba6bc777d8f..0b62934b5b5fa 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_private_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_private_codegen.cpp
@@ -260,8 +260,8 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -522,6 +522,7 @@ int main() {
 // CHECK1-NEXT:    [[VAR:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -531,7 +532,6 @@ int main() {
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef signext 2)
 // CHECK1-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 8
 // CHECK1-NEXT:    store ptr undef, ptr [[_TMP1]], align 8
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1000,8 +1000,8 @@ int main() {
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1256,6 +1256,7 @@ int main() {
 // CHECK3-NEXT:    [[VAR:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1265,7 +1266,6 @@ int main() {
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef 2)
 // CHECK3-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 4
 // CHECK3-NEXT:    store ptr undef, ptr [[_TMP1]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_proc_bind_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_proc_bind_codegen.cpp
index 141adc5c8c9de..85dbe73382c7d 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_proc_bind_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_proc_bind_codegen.cpp
@@ -55,9 +55,10 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -91,7 +92,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}_main_l36() #[[ATTR2:[0-9]+]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_reduction_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_reduction_codegen.cpp
index de787727f3fa9..96fb0fef623a3 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_reduction_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_reduction_codegen.cpp
@@ -102,6 +102,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr @_ZZ4mainE5sivar, align 4
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[SIVAR_CASTED]], align 4
@@ -114,7 +115,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -395,6 +395,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr [[T_VAR]], align 4
@@ -408,7 +409,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -694,6 +694,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr @_ZZ4mainE5sivar, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[SIVAR_CASTED]], align 4
@@ -706,7 +707,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP4]], align 4
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -983,6 +983,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr [[T_VAR]], align 4
@@ -996,7 +997,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP4]], align 4
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_schedule_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_schedule_codegen.cpp
index 0b8915954cff8..a5304673f42e2 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_schedule_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_schedule_codegen.cpp
@@ -224,22 +224,27 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS21:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP22:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS27:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS28:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS29:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP30:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -251,7 +256,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -294,7 +298,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK1-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -337,7 +340,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK1-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK1-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -380,7 +382,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP62]], align 8
 // CHECK1-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS19]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS20]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP65]], align 4
 // CHECK1-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 1
@@ -423,7 +424,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP82]], align 8
 // CHECK1-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS27]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS28]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP85]], align 4
 // CHECK1-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -1258,22 +1258,27 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS21:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP22:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS27:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS28:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS29:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP30:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -1285,7 +1290,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1328,7 +1332,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK3-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -1371,7 +1374,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK3-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK3-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1414,7 +1416,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP62]], align 4
 // CHECK3-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS19]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS20]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP65]], align 4
 // CHECK3-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 1
@@ -1457,7 +1458,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP82]], align 4
 // CHECK3-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS27]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS28]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP85]], align 4
 // CHECK3-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -2265,22 +2265,27 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS21:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP22:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS27:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS28:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS29:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[_TMP30:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK5-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK5-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -2292,7 +2297,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK5-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK5-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2335,7 +2339,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK5-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK5-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -2378,7 +2381,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK5-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK5-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -2421,7 +2423,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP62]], align 8
 // CHECK5-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS19]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS20]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP65]], align 4
 // CHECK5-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 1
@@ -2464,7 +2465,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP82]], align 8
 // CHECK5-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS27]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS28]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP85]], align 4
 // CHECK5-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -3299,22 +3299,27 @@ int main (int argc, char **argv) {
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS21:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[_TMP22:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[DOTOFFLOAD_BASEPTRS27:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_PTRS28:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[DOTOFFLOAD_MAPPERS29:%.*]] = alloca [1 x ptr], align 4
 // CHECK7-NEXT:    [[_TMP30:%.*]] = alloca i32, align 4
+// CHECK7-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK7-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK7-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -3326,7 +3331,6 @@ int main (int argc, char **argv) {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK7-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK7-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK7-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3369,7 +3373,6 @@ int main (int argc, char **argv) {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK7-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK7-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -3412,7 +3415,6 @@ int main (int argc, char **argv) {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK7-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK7-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -3455,7 +3457,6 @@ int main (int argc, char **argv) {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP62]], align 4
 // CHECK7-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS19]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS20]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP65]], align 4
 // CHECK7-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 1
@@ -3498,7 +3499,6 @@ int main (int argc, char **argv) {
 // CHECK7-NEXT:    store ptr null, ptr [[TMP82]], align 4
 // CHECK7-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS27]], i32 0, i32 0
 // CHECK7-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS28]], i32 0, i32 0
-// CHECK7-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK7-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK7-NEXT:    store i32 2, ptr [[TMP85]], align 4
 // CHECK7-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -4308,6 +4308,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 8
@@ -4316,6 +4317,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[M_CASTED:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[N_CASTED18:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [4 x ptr], align 8
@@ -4325,6 +4327,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_24:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[N_CASTED33:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS34:%.*]] = alloca [3 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS35:%.*]] = alloca [3 x ptr], align 8
@@ -4333,6 +4336,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[_TMP38:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_39:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_40:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS45:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[M_CASTED48:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[N_CASTED49:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS50:%.*]] = alloca [4 x ptr], align 8
@@ -4342,6 +4346,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[_TMP54:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_55:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_56:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK13-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK13-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -4391,7 +4396,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK13-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK13-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK13-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4463,7 +4467,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP56:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK13-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP56]], 1
 // CHECK13-NEXT:    [[TMP57:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK13-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -4544,7 +4547,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP96:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_25]], align 4
 // CHECK13-NEXT:    [[ADD29:%.*]] = add nsw i32 [[TMP96]], 1
 // CHECK13-NEXT:    [[TMP97:%.*]] = zext i32 [[ADD29]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP98]], align 4
 // CHECK13-NEXT:    [[TMP99:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 1
@@ -4616,7 +4618,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP131:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_40]], align 4
 // CHECK13-NEXT:    [[ADD44:%.*]] = add nsw i32 [[TMP131]], 1
 // CHECK13-NEXT:    [[TMP132:%.*]] = zext i32 [[ADD44]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS45:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP133:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS45]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP133]], align 4
 // CHECK13-NEXT:    [[TMP134:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS45]], i32 0, i32 1
@@ -4697,7 +4698,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP171:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_56]], align 4
 // CHECK13-NEXT:    [[ADD60:%.*]] = add nsw i32 [[TMP171]], 1
 // CHECK13-NEXT:    [[TMP172:%.*]] = zext i32 [[ADD60]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP173:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP173]], align 4
 // CHECK13-NEXT:    [[TMP174:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 1
@@ -5857,24 +5857,29 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[M_CASTED:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[M_CASTED22:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS23:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS24:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS25:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[_TMP26:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS27:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK13-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK13-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -5885,7 +5890,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK13-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK13-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5927,7 +5931,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK13-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK13-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -5978,7 +5981,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP47]], align 8
 // CHECK13-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP50]], align 4
 // CHECK13-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -6020,7 +6022,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP67]], align 8
 // CHECK13-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP70]], align 4
 // CHECK13-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -6071,7 +6072,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP92]], align 8
 // CHECK13-NEXT:    [[TMP93:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS23]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP94:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS24]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS27:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS27]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP95]], align 4
 // CHECK13-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS27]], i32 0, i32 1
@@ -6936,6 +6936,7 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK15-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 4
@@ -6944,6 +6945,7 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[M_CASTED:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[N_CASTED18:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [4 x ptr], align 4
@@ -6953,6 +6955,7 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_24:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[N_CASTED33:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS34:%.*]] = alloca [3 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS35:%.*]] = alloca [3 x ptr], align 4
@@ -6961,6 +6964,7 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[_TMP38:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_39:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_40:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS45:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[M_CASTED48:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[N_CASTED49:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS50:%.*]] = alloca [4 x ptr], align 4
@@ -6970,6 +6974,7 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[_TMP54:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_55:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTCAPTURE_EXPR_56:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK15-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK15-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -7019,7 +7024,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK15-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK15-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK15-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK15-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK15-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -7092,7 +7096,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[TMP57:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK15-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP57]], 1
 // CHECK15-NEXT:    [[TMP58:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK15-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP59]], align 4
 // CHECK15-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -7174,7 +7177,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[TMP98:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_25]], align 4
 // CHECK15-NEXT:    [[ADD29:%.*]] = add nsw i32 [[TMP98]], 1
 // CHECK15-NEXT:    [[TMP99:%.*]] = zext i32 [[ADD29]] to i64
-// CHECK15-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP100:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP100]], align 4
 // CHECK15-NEXT:    [[TMP101:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 1
@@ -7247,7 +7249,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[TMP134:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_40]], align 4
 // CHECK15-NEXT:    [[ADD44:%.*]] = add nsw i32 [[TMP134]], 1
 // CHECK15-NEXT:    [[TMP135:%.*]] = zext i32 [[ADD44]] to i64
-// CHECK15-NEXT:    [[KERNEL_ARGS45:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP136:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS45]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP136]], align 4
 // CHECK15-NEXT:    [[TMP137:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS45]], i32 0, i32 1
@@ -7329,7 +7330,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[TMP175:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_56]], align 4
 // CHECK15-NEXT:    [[ADD60:%.*]] = add nsw i32 [[TMP175]], 1
 // CHECK15-NEXT:    [[TMP176:%.*]] = zext i32 [[ADD60]] to i64
-// CHECK15-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP177:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP177]], align 4
 // CHECK15-NEXT:    [[TMP178:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 1
@@ -8464,24 +8464,29 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[M_CASTED:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 4
 // CHECK15-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [1 x ptr], align 4
 // CHECK15-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[M_CASTED22:%.*]] = alloca i32, align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_BASEPTRS23:%.*]] = alloca [2 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_PTRS24:%.*]] = alloca [2 x ptr], align 4
 // CHECK15-NEXT:    [[DOTOFFLOAD_MAPPERS25:%.*]] = alloca [2 x ptr], align 4
 // CHECK15-NEXT:    [[_TMP26:%.*]] = alloca i32, align 4
+// CHECK15-NEXT:    [[KERNEL_ARGS27:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK15-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK15-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -8492,7 +8497,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK15-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK15-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK15-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK15-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK15-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -8534,7 +8538,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK15-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK15-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK15-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK15-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -8585,7 +8588,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    store ptr null, ptr [[TMP47]], align 4
 // CHECK15-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK15-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK15-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP50]], align 4
 // CHECK15-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -8627,7 +8629,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    store ptr null, ptr [[TMP67]], align 4
 // CHECK15-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK15-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK15-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP70]], align 4
 // CHECK15-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -8678,7 +8679,6 @@ int main (int argc, char **argv) {
 // CHECK15-NEXT:    store ptr null, ptr [[TMP92]], align 4
 // CHECK15-NEXT:    [[TMP93:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS23]], i32 0, i32 0
 // CHECK15-NEXT:    [[TMP94:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS24]], i32 0, i32 0
-// CHECK15-NEXT:    [[KERNEL_ARGS27:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK15-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS27]], i32 0, i32 0
 // CHECK15-NEXT:    store i32 2, ptr [[TMP95]], align 4
 // CHECK15-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS27]], i32 0, i32 1
@@ -9516,6 +9516,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 8
@@ -9524,6 +9525,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[M_CASTED:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[N_CASTED18:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [4 x ptr], align 8
@@ -9533,6 +9535,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_24:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[N_CASTED33:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS34:%.*]] = alloca [3 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS35:%.*]] = alloca [3 x ptr], align 8
@@ -9541,6 +9544,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[_TMP38:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_39:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_40:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS45:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[M_CASTED48:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[N_CASTED49:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS50:%.*]] = alloca [4 x ptr], align 8
@@ -9550,6 +9554,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[_TMP54:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_55:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_56:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK17-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK17-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -9599,7 +9604,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK17-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK17-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK17-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -9671,7 +9675,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP56:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK17-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP56]], 1
 // CHECK17-NEXT:    [[TMP57:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK17-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -9752,7 +9755,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP96:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_25]], align 4
 // CHECK17-NEXT:    [[ADD29:%.*]] = add nsw i32 [[TMP96]], 1
 // CHECK17-NEXT:    [[TMP97:%.*]] = zext i32 [[ADD29]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP98]], align 4
 // CHECK17-NEXT:    [[TMP99:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 1
@@ -9824,7 +9826,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP131:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_40]], align 4
 // CHECK17-NEXT:    [[ADD44:%.*]] = add nsw i32 [[TMP131]], 1
 // CHECK17-NEXT:    [[TMP132:%.*]] = zext i32 [[ADD44]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS45:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP133:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS45]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP133]], align 4
 // CHECK17-NEXT:    [[TMP134:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS45]], i32 0, i32 1
@@ -9905,7 +9906,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP171:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_56]], align 4
 // CHECK17-NEXT:    [[ADD60:%.*]] = add nsw i32 [[TMP171]], 1
 // CHECK17-NEXT:    [[TMP172:%.*]] = zext i32 [[ADD60]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP173:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP173]], align 4
 // CHECK17-NEXT:    [[TMP174:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 1
@@ -11065,24 +11065,29 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[M_CASTED:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [1 x ptr], align 8
 // CHECK17-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[M_CASTED22:%.*]] = alloca i64, align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS23:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS24:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS25:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[_TMP26:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS27:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK17-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK17-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -11093,7 +11098,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK17-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK17-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -11135,7 +11139,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK17-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK17-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -11186,7 +11189,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP47]], align 8
 // CHECK17-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP50]], align 4
 // CHECK17-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -11228,7 +11230,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP67]], align 8
 // CHECK17-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP70]], align 4
 // CHECK17-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -11279,7 +11280,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP92]], align 8
 // CHECK17-NEXT:    [[TMP93:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS23]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP94:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS24]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS27:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS27]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP95]], align 4
 // CHECK17-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS27]], i32 0, i32 1
@@ -12144,6 +12144,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 4
@@ -12152,6 +12153,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[M_CASTED:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[N_CASTED18:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [4 x ptr], align 4
@@ -12161,6 +12163,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_24:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[N_CASTED33:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS34:%.*]] = alloca [3 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS35:%.*]] = alloca [3 x ptr], align 4
@@ -12169,6 +12172,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[_TMP38:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_39:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_40:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS45:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[M_CASTED48:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[N_CASTED49:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS50:%.*]] = alloca [4 x ptr], align 4
@@ -12178,6 +12182,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[_TMP54:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_55:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_56:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK19-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK19-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -12227,7 +12232,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK19-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK19-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK19-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -12300,7 +12304,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP57:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK19-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP57]], 1
 // CHECK19-NEXT:    [[TMP58:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP59]], align 4
 // CHECK19-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -12382,7 +12385,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP98:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_25]], align 4
 // CHECK19-NEXT:    [[ADD29:%.*]] = add nsw i32 [[TMP98]], 1
 // CHECK19-NEXT:    [[TMP99:%.*]] = zext i32 [[ADD29]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP100:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP100]], align 4
 // CHECK19-NEXT:    [[TMP101:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 1
@@ -12455,7 +12457,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP134:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_40]], align 4
 // CHECK19-NEXT:    [[ADD44:%.*]] = add nsw i32 [[TMP134]], 1
 // CHECK19-NEXT:    [[TMP135:%.*]] = zext i32 [[ADD44]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS45:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP136:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS45]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP136]], align 4
 // CHECK19-NEXT:    [[TMP137:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS45]], i32 0, i32 1
@@ -12537,7 +12538,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP175:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_56]], align 4
 // CHECK19-NEXT:    [[ADD60:%.*]] = add nsw i32 [[TMP175]], 1
 // CHECK19-NEXT:    [[TMP176:%.*]] = zext i32 [[ADD60]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP177:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP177]], align 4
 // CHECK19-NEXT:    [[TMP178:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 1
@@ -13672,24 +13672,29 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[M_CASTED:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[M_CASTED22:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS23:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS24:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS25:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[_TMP26:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS27:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK19-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK19-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -13700,7 +13705,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK19-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK19-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -13742,7 +13746,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK19-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK19-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -13793,7 +13796,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP47]], align 4
 // CHECK19-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP50]], align 4
 // CHECK19-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -13835,7 +13837,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP67]], align 4
 // CHECK19-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP70]], align 4
 // CHECK19-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -13886,7 +13887,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP92]], align 4
 // CHECK19-NEXT:    [[TMP93:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS23]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP94:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS24]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS27:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS27]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP95]], align 4
 // CHECK19-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS27]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_simd_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_simd_codegen.cpp
index c9a831352c50a..c511f9f02aa05 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_simd_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_simd_codegen.cpp
@@ -196,6 +196,7 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[I_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[N_CASTED4:%.*]] = alloca i64, align 8
@@ -205,6 +206,7 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[DIV:%.*]] = sdiv i32 [[TMP0]], 128
@@ -257,7 +259,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP24]], 1
 // CHECK1-NEXT:    [[TMP25:%.*]] = zext i32 [[ADD]] to i64
 // CHECK1-NEXT:    [[TMP26:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP21]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP27]], align 4
 // CHECK1-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -327,7 +328,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[TMP59:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK1-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP59]], 1
 // CHECK1-NEXT:    [[TMP60:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP61]], align 4
 // CHECK1-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -872,6 +872,7 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[I_CASTED:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[N_CASTED4:%.*]] = alloca i32, align 4
@@ -881,6 +882,7 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[DIV:%.*]] = sdiv i32 [[TMP0]], 128
@@ -933,7 +935,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP24]], 1
 // CHECK3-NEXT:    [[TMP25:%.*]] = zext i32 [[ADD]] to i64
 // CHECK3-NEXT:    [[TMP26:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP21]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP27:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP27]], align 4
 // CHECK3-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1003,7 +1004,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[TMP59:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK3-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP59]], 1
 // CHECK3-NEXT:    [[TMP60:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP61]], align 4
 // CHECK3-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1807,6 +1807,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 100, ptr [[N]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = zext i32 [[TMP0]] to i64
@@ -1861,7 +1862,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP26:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP26]], 1
 // CHECK9-NEXT:    [[TMP27:%.*]] = zext i32 [[ADD]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP28]], align 4
 // CHECK9-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2183,6 +2183,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 100, ptr [[N]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = call ptr @llvm.stacksave()
@@ -2237,7 +2238,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP26:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP26]], 1
 // CHECK11-NEXT:    [[TMP27:%.*]] = zext i32 [[ADD]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP28]], align 4
 // CHECK11-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2720,6 +2720,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [2 x ptr], align 8
 // CHECK17-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK17-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK17-NEXT:    [[TMP0:%.*]] = load i32, ptr [[I]], align 4
@@ -2740,7 +2741,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP7]], align 8
 // CHECK17-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP10]], align 4
 // CHECK17-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2995,6 +2995,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK19-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK19-NEXT:    [[TMP0:%.*]] = load i32, ptr [[I]], align 4
@@ -3015,7 +3016,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP7]], align 4
 // CHECK19-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP10:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP10]], align 4
 // CHECK19-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3391,6 +3391,7 @@ int main (int argc, char **argv) {
 // CHECK25-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK25-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK25-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK25-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK25-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK25-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK25-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -3448,7 +3449,6 @@ int main (int argc, char **argv) {
 // CHECK25-NEXT:    [[TMP26:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK25-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP26]], 1
 // CHECK25-NEXT:    [[TMP27:%.*]] = zext i32 [[ADD]] to i64
-// CHECK25-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK25-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK25-NEXT:    store i32 2, ptr [[TMP28]], align 4
 // CHECK25-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3762,6 +3762,7 @@ int main (int argc, char **argv) {
 // CHECK25-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK25-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
 // CHECK25-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK25-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK25-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK25-NEXT:    store i32 0, ptr [[TE]], align 4
 // CHECK25-NEXT:    store i32 128, ptr [[TH]], align 4
@@ -3793,7 +3794,6 @@ int main (int argc, char **argv) {
 // CHECK25-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK25-NEXT:    [[TMP15:%.*]] = load i32, ptr [[TE]], align 4
 // CHECK25-NEXT:    [[TMP16:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP15]], 0
-// CHECK25-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK25-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK25-NEXT:    store i32 2, ptr [[TMP17]], align 4
 // CHECK25-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4028,6 +4028,7 @@ int main (int argc, char **argv) {
 // CHECK27-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK27-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK27-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK27-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK27-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK27-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK27-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -4085,7 +4086,6 @@ int main (int argc, char **argv) {
 // CHECK27-NEXT:    [[TMP26:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK27-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP26]], 1
 // CHECK27-NEXT:    [[TMP27:%.*]] = zext i32 [[ADD]] to i64
-// CHECK27-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK27-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK27-NEXT:    store i32 2, ptr [[TMP28]], align 4
 // CHECK27-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4394,6 +4394,7 @@ int main (int argc, char **argv) {
 // CHECK27-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK27-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
 // CHECK27-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK27-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK27-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK27-NEXT:    store i32 0, ptr [[TE]], align 4
 // CHECK27-NEXT:    store i32 128, ptr [[TH]], align 4
@@ -4425,7 +4426,6 @@ int main (int argc, char **argv) {
 // CHECK27-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK27-NEXT:    [[TMP15:%.*]] = load i32, ptr [[TE]], align 4
 // CHECK27-NEXT:    [[TMP16:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP15]], 0
-// CHECK27-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK27-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK27-NEXT:    store i32 2, ptr [[TMP17]], align 4
 // CHECK27-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_simd_collapse_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_simd_collapse_codegen.cpp
index 3d971c5e92491..ac6bb3a9d835b 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_simd_collapse_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_simd_collapse_codegen.cpp
@@ -124,6 +124,7 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -135,7 +136,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -383,6 +383,7 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -394,7 +395,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -781,6 +781,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i64, align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -857,7 +858,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store i64 [[SUB7]], ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK9-NEXT:    [[TMP35:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i64 [[TMP35]], 1
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP36]], align 4
 // CHECK9-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1234,6 +1234,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    store ptr [[A]], ptr [[TMP0]], align 8
@@ -1243,7 +1244,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK9-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1491,6 +1491,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i64, align 8
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -1566,7 +1567,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store i64 [[SUB7]], ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK11-NEXT:    [[TMP34:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i64 [[TMP34]], 1
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP35]], align 4
 // CHECK11-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1945,6 +1945,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    store ptr [[A]], ptr [[TMP0]], align 4
@@ -1954,7 +1955,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK11-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_simd_dist_schedule_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_simd_dist_schedule_codegen.cpp
index c2579ccf475b2..260b050c69d61 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_simd_dist_schedule_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_simd_dist_schedule_codegen.cpp
@@ -162,14 +162,17 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -181,7 +184,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -224,7 +226,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK1-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -267,7 +268,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK1-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK1-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -845,14 +845,17 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -864,7 +867,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -907,7 +909,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK3-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -950,7 +951,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK3-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK3-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1762,6 +1762,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 8
@@ -1770,6 +1771,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[M_CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[N_CASTED18:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [4 x ptr], align 8
@@ -1779,6 +1781,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_24:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -1828,7 +1831,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK9-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1900,7 +1902,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP56:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK9-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP56]], 1
 // CHECK9-NEXT:    [[TMP57:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK9-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1981,7 +1982,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP96:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_25]], align 4
 // CHECK9-NEXT:    [[ADD29:%.*]] = add nsw i32 [[TMP96]], 1
 // CHECK9-NEXT:    [[TMP97:%.*]] = zext i32 [[ADD29]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP98]], align 4
 // CHECK9-NEXT:    [[TMP99:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 1
@@ -2781,15 +2781,18 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[M_CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -2800,7 +2803,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK9-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2842,7 +2844,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -2893,7 +2894,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP47]], align 8
 // CHECK9-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP50]], align 4
 // CHECK9-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -3485,6 +3485,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 4
@@ -3493,6 +3494,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[M_CASTED:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[N_CASTED18:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [4 x ptr], align 4
@@ -3502,6 +3504,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_24:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -3551,7 +3554,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK11-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3624,7 +3626,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP57:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK11-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP57]], 1
 // CHECK11-NEXT:    [[TMP58:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP59]], align 4
 // CHECK11-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -3706,7 +3707,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP98:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_25]], align 4
 // CHECK11-NEXT:    [[ADD29:%.*]] = add nsw i32 [[TMP98]], 1
 // CHECK11-NEXT:    [[TMP99:%.*]] = zext i32 [[ADD29]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP100:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP100]], align 4
 // CHECK11-NEXT:    [[TMP101:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 1
@@ -4491,15 +4491,18 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[M_CASTED:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -4510,7 +4513,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK11-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4552,7 +4554,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -4603,7 +4604,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP47]], align 4
 // CHECK11-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP50]], align 4
 // CHECK11-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_simd_firstprivate_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_simd_firstprivate_codegen.cpp
index 5dc93d7473d99..c17d3675fd669 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_simd_firstprivate_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_simd_firstprivate_codegen.cpp
@@ -302,6 +302,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr @t_var, align 4
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[T_VAR_CASTED]], align 4
@@ -341,7 +342,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP18]], align 8
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK1-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -726,6 +726,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -768,7 +769,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP17]], align 8
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK1-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1401,6 +1401,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr @t_var, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[T_VAR_CASTED]], align 4
@@ -1440,7 +1441,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK3-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1819,6 +1819,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1861,7 +1862,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP17]], align 4
 // CHECK3-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK3-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_simd_if_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_simd_if_codegen.cpp
index 7d8692db7efb7..e939511a9374e 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_simd_if_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_simd_if_codegen.cpp
@@ -114,8 +114,9 @@ int main() {
 // CHECK1-SAME: () #[[ATTR0:[0-9]+]] {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -149,7 +150,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9gtid_testv_l43() #[[ATTR2:[0-9]+]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -499,15 +499,17 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -541,7 +543,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}_main_l76() #[[ATTR2]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -594,7 +595,6 @@ int main() {
 // CHECK1-NEXT:    [[TOBOOL5:%.*]] = trunc i8 [[TMP38]] to i1
 // CHECK1-NEXT:    [[TMP39:%.*]] = select i1 [[TOBOOL5]], i32 0, i32 1
 // CHECK1-NEXT:    [[TMP40:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP39]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP41]], align 4
 // CHECK1-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -1128,15 +1128,17 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[ARG_ADDR:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[ARG]], ptr [[ARG_ADDR]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1170,7 +1172,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIiEiT__l57() #[[ATTR2]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -1223,7 +1224,6 @@ int main() {
 // CHECK1-NEXT:    [[TOBOOL5:%.*]] = trunc i8 [[TMP38]] to i1
 // CHECK1-NEXT:    [[TMP39:%.*]] = select i1 [[TOBOOL5]], i32 0, i32 1
 // CHECK1-NEXT:    [[TMP40:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP39]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP41]], align 4
 // CHECK1-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -1761,8 +1761,9 @@ int main() {
 // CHECK3-SAME: () #[[ATTR0:[0-9]+]] {
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK3-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK3-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1796,7 +1797,6 @@ int main() {
 // CHECK3-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9gtid_testv_l43() #[[ATTR2:[0-9]+]]
 // CHECK3-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK3:       omp_offload.cont:
-// CHECK3-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -2146,15 +2146,17 @@ int main() {
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK3-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2188,7 +2190,6 @@ int main() {
 // CHECK3-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}_main_l76() #[[ATTR2]]
 // CHECK3-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK3:       omp_offload.cont:
-// CHECK3-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -2241,7 +2242,6 @@ int main() {
 // CHECK3-NEXT:    [[TOBOOL5:%.*]] = trunc i8 [[TMP38]] to i1
 // CHECK3-NEXT:    [[TMP39:%.*]] = select i1 [[TOBOOL5]], i32 0, i32 1
 // CHECK3-NEXT:    [[TMP40:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP39]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP41]], align 4
 // CHECK3-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -3005,15 +3005,17 @@ int main() {
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[ARG_ADDR:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK3-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 [[ARG]], ptr [[ARG_ADDR]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3047,7 +3049,6 @@ int main() {
 // CHECK3-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIiEiT__l57() #[[ATTR2]]
 // CHECK3-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK3:       omp_offload.cont:
-// CHECK3-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK3-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -3100,7 +3101,6 @@ int main() {
 // CHECK3-NEXT:    [[TOBOOL5:%.*]] = trunc i8 [[TMP38]] to i1
 // CHECK3-NEXT:    [[TMP39:%.*]] = select i1 [[TOBOOL5]], i32 0, i32 1
 // CHECK3-NEXT:    [[TMP40:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP39]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP41]], align 4
 // CHECK3-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -4229,8 +4229,9 @@ int main() {
 // CHECK9-SAME: () #[[ATTR0:[0-9]+]] {
 // CHECK9-NEXT:  entry:
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4264,7 +4265,6 @@ int main() {
 // CHECK9-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9gtid_testv_l43() #[[ATTR2:[0-9]+]]
 // CHECK9-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK9:       omp_offload.cont:
-// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK9-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -4614,15 +4614,17 @@ int main() {
 // CHECK9-NEXT:  entry:
 // CHECK9-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK9-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4656,7 +4658,6 @@ int main() {
 // CHECK9-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}_main_l76() #[[ATTR2]]
 // CHECK9-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK9:       omp_offload.cont:
-// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK9-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -4709,7 +4710,6 @@ int main() {
 // CHECK9-NEXT:    [[TOBOOL5:%.*]] = trunc i8 [[TMP38]] to i1
 // CHECK9-NEXT:    [[TMP39:%.*]] = select i1 [[TOBOOL5]], i32 0, i32 1
 // CHECK9-NEXT:    [[TMP40:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP39]], 0
-// CHECK9-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP41]], align 4
 // CHECK9-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -5243,15 +5243,17 @@ int main() {
 // CHECK9-NEXT:  entry:
 // CHECK9-NEXT:    [[ARG_ADDR:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK9-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 [[ARG]], ptr [[ARG_ADDR]], align 4
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5285,7 +5287,6 @@ int main() {
 // CHECK9-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIiEiT__l57() #[[ATTR2]]
 // CHECK9-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK9:       omp_offload.cont:
-// CHECK9-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK9-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -5338,7 +5339,6 @@ int main() {
 // CHECK9-NEXT:    [[TOBOOL5:%.*]] = trunc i8 [[TMP38]] to i1
 // CHECK9-NEXT:    [[TMP39:%.*]] = select i1 [[TOBOOL5]], i32 0, i32 1
 // CHECK9-NEXT:    [[TMP40:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP39]], 0
-// CHECK9-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP41]], align 4
 // CHECK9-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -5876,8 +5876,9 @@ int main() {
 // CHECK11-SAME: () #[[ATTR0:[0-9]+]] {
 // CHECK11-NEXT:  entry:
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5911,7 +5912,6 @@ int main() {
 // CHECK11-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z9gtid_testv_l43() #[[ATTR2:[0-9]+]]
 // CHECK11-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK11:       omp_offload.cont:
-// CHECK11-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK11-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -6261,15 +6261,17 @@ int main() {
 // CHECK11-NEXT:  entry:
 // CHECK11-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK11-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -6303,7 +6305,6 @@ int main() {
 // CHECK11-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}_main_l76() #[[ATTR2]]
 // CHECK11-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK11:       omp_offload.cont:
-// CHECK11-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK11-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -6356,7 +6357,6 @@ int main() {
 // CHECK11-NEXT:    [[TOBOOL5:%.*]] = trunc i8 [[TMP38]] to i1
 // CHECK11-NEXT:    [[TMP39:%.*]] = select i1 [[TOBOOL5]], i32 0, i32 1
 // CHECK11-NEXT:    [[TMP40:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP39]], 0
-// CHECK11-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP41]], align 4
 // CHECK11-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -7120,15 +7120,17 @@ int main() {
 // CHECK11-NEXT:  entry:
 // CHECK11-NEXT:    [[ARG_ADDR:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[ARG_CASTED:%.*]] = alloca i64, align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK11-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 [[ARG]], ptr [[ARG_ADDR]], align 4
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -7162,7 +7164,6 @@ int main() {
 // CHECK11-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIiEiT__l57() #[[ATTR2]]
 // CHECK11-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK11:       omp_offload.cont:
-// CHECK11-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK11-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -7215,7 +7216,6 @@ int main() {
 // CHECK11-NEXT:    [[TOBOOL5:%.*]] = trunc i8 [[TMP38]] to i1
 // CHECK11-NEXT:    [[TMP39:%.*]] = select i1 [[TOBOOL5]], i32 0, i32 1
 // CHECK11-NEXT:    [[TMP40:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP39]], 0
-// CHECK11-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP41]], align 4
 // CHECK11-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_simd_lastprivate_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_simd_lastprivate_codegen.cpp
index 2b5a3d788f2c0..b4362a372d6e9 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_simd_lastprivate_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_simd_lastprivate_codegen.cpp
@@ -797,6 +797,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store ptr [[G]], ptr [[G1]], align 8
@@ -851,7 +852,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1282,6 +1282,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK9-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -1324,7 +1325,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP17]], align 8
 // CHECK9-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK9-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1818,6 +1818,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store ptr [[G]], ptr [[G1]], align 4
@@ -1872,7 +1873,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2297,6 +2297,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK11-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -2339,7 +2340,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP17]], align 4
 // CHECK11-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK11-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_simd_num_threads_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_simd_num_threads_codegen.cpp
index 4792f1279200f..1a1b5d5e9acc6 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_simd_num_threads_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_simd_num_threads_codegen.cpp
@@ -84,19 +84,20 @@ int main() {
 // CHECK1-NEXT:    [[EXN_SLOT:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[EHSELECTOR_SLOT:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    call void @_ZN1SC1El(ptr noundef nonnull align 8 dereferenceable(24) [[S]], i64 noundef 0)
 // CHECK1-NEXT:    [[CALL:%.*]] = invoke noundef signext i8 @_ZN1ScvcEv(ptr noundef nonnull align 8 dereferenceable(24) [[S]])
 // CHECK1-NEXT:    to label [[INVOKE_CONT:%.*]] unwind label [[LPAD:%.*]]
 // CHECK1:       invoke.cont:
 // CHECK1-NEXT:    store i8 [[CALL]], ptr [[A]], align 1
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -155,7 +156,6 @@ int main() {
 // CHECK1-NEXT:    [[TMP26:%.*]] = load i8, ptr [[DOTCAPTURE_EXPR_]], align 1
 // CHECK1-NEXT:    [[TMP27:%.*]] = zext i8 [[TMP26]] to i32
 // CHECK1-NEXT:    [[TMP28:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP27]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK1-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -582,8 +582,9 @@ int main() {
 // CHECK1-SAME: () #[[ATTR2]] comdat {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -617,7 +618,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIcLi5EEiv_l36() #[[ATTR6]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -658,10 +658,11 @@ int main() {
 // CHECK1-SAME: () #[[ATTR2]] comdat personality ptr @__gxx_personality_v0 {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK1-NEXT:    [[REF_TMP:%.*]] = alloca [[STRUCT_S:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -704,7 +705,6 @@ int main() {
 // CHECK1-NEXT:    [[TMP15:%.*]] = load i8, ptr [[DOTCAPTURE_EXPR_]], align 1
 // CHECK1-NEXT:    [[TMP16:%.*]] = zext i8 [[TMP15]] to i32
 // CHECK1-NEXT:    [[TMP17:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP16]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -1811,19 +1811,20 @@ int main() {
 // CHECK5-NEXT:    [[EXN_SLOT:%.*]] = alloca ptr, align 8
 // CHECK5-NEXT:    [[EHSELECTOR_SLOT:%.*]] = alloca i32, align 4
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[A_CASTED:%.*]] = alloca i64, align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK5-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK5-NEXT:    call void @_ZN1SC1El(ptr noundef nonnull align 8 dereferenceable(24) [[S]], i64 noundef 0)
 // CHECK5-NEXT:    [[CALL:%.*]] = invoke noundef signext i8 @_ZN1ScvcEv(ptr noundef nonnull align 8 dereferenceable(24) [[S]])
 // CHECK5-NEXT:    to label [[INVOKE_CONT:%.*]] unwind label [[LPAD:%.*]]
 // CHECK5:       invoke.cont:
 // CHECK5-NEXT:    store i8 [[CALL]], ptr [[A]], align 1
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK5-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1882,7 +1883,6 @@ int main() {
 // CHECK5-NEXT:    [[TMP26:%.*]] = load i8, ptr [[DOTCAPTURE_EXPR_]], align 1
 // CHECK5-NEXT:    [[TMP27:%.*]] = zext i8 [[TMP26]] to i32
 // CHECK5-NEXT:    [[TMP28:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP27]], 0
-// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP29]], align 4
 // CHECK5-NEXT:    [[TMP30:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -2309,8 +2309,9 @@ int main() {
 // CHECK5-SAME: () #[[ATTR2]] comdat {
 // CHECK5-NEXT:  entry:
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
 // CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK5-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2344,7 +2345,6 @@ int main() {
 // CHECK5-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}__Z5tmainIcLi5EEiv_l36() #[[ATTR6]]
 // CHECK5-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK5:       omp_offload.cont:
-// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK5-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
@@ -2385,10 +2385,11 @@ int main() {
 // CHECK5-SAME: () #[[ATTR2]] comdat personality ptr @__gxx_personality_v0 {
 // CHECK5-NEXT:  entry:
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i8, align 1
 // CHECK5-NEXT:    [[REF_TMP:%.*]] = alloca [[STRUCT_S:%.*]], align 8
 // CHECK5-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK5-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2431,7 +2432,6 @@ int main() {
 // CHECK5-NEXT:    [[TMP15:%.*]] = load i8, ptr [[DOTCAPTURE_EXPR_]], align 1
 // CHECK5-NEXT:    [[TMP16:%.*]] = zext i8 [[TMP15]] to i32
 // CHECK5-NEXT:    [[TMP17:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP16]], 0
-// CHECK5-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK5-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_simd_private_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_simd_private_codegen.cpp
index 35698f812ed88..06cb3a6af1c93 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_simd_private_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_simd_private_codegen.cpp
@@ -262,8 +262,8 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -538,6 +538,7 @@ int main() {
 // CHECK1-NEXT:    [[VAR:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -547,7 +548,6 @@ int main() {
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef signext 2)
 // CHECK1-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 8
 // CHECK1-NEXT:    store ptr undef, ptr [[_TMP1]], align 8
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1030,8 +1030,8 @@ int main() {
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1300,6 +1300,7 @@ int main() {
 // CHECK3-NEXT:    [[VAR:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1309,7 +1310,6 @@ int main() {
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef 2)
 // CHECK3-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 4
 // CHECK3-NEXT:    store ptr undef, ptr [[_TMP1]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_simd_proc_bind_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_simd_proc_bind_codegen.cpp
index 590e65c5df49e..ed582616f4ce7 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_simd_proc_bind_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_simd_proc_bind_codegen.cpp
@@ -57,9 +57,10 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -93,7 +94,6 @@ int main() {
 // CHECK1-NEXT:    call void @{{__omp_offloading_[0-9a-z]+_[0-9a-z]+}}_main_l36() #[[ATTR2:[0-9]+]]
 // CHECK1-NEXT:    br label [[OMP_OFFLOAD_CONT]]
 // CHECK1:       omp_offload.cont:
-// CHECK1-NEXT:    [[KERNEL_ARGS2:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP15]], align 4
 // CHECK1-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS2]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_simd_reduction_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_simd_reduction_codegen.cpp
index 1df8bce9953a5..9635e16438832 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_simd_reduction_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_simd_reduction_codegen.cpp
@@ -106,6 +106,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr @_ZZ4mainE5sivar, align 4
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[SIVAR_CASTED]], align 4
@@ -118,7 +119,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -413,6 +413,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr [[T_VAR]], align 4
@@ -426,7 +427,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -726,6 +726,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr @_ZZ4mainE5sivar, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[SIVAR_CASTED]], align 4
@@ -738,7 +739,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP4]], align 4
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1029,6 +1029,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr [[T_VAR]], align 4
@@ -1042,7 +1043,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP4]], align 4
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_simd_schedule_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_simd_schedule_codegen.cpp
index a1254b10fa10c..b11de268e262b 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_simd_schedule_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_simd_schedule_codegen.cpp
@@ -234,22 +234,27 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS21:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP22:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS27:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS28:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS29:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP30:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -261,7 +266,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -304,7 +308,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK1-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -347,7 +350,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK1-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK1-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -390,7 +392,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP62]], align 8
 // CHECK1-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS19]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS20]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP65]], align 4
 // CHECK1-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 1
@@ -433,7 +434,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP82]], align 8
 // CHECK1-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS27]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS28]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP85]], align 4
 // CHECK1-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -1338,22 +1338,27 @@ int main (int argc, char **argv) {
 // CHECK2-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK2-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK2-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK2-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK2-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK2-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK2-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
 // CHECK2-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK2-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK2-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 8
 // CHECK2-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 8
 // CHECK2-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 8
 // CHECK2-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK2-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK2-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [1 x ptr], align 8
 // CHECK2-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [1 x ptr], align 8
 // CHECK2-NEXT:    [[DOTOFFLOAD_MAPPERS21:%.*]] = alloca [1 x ptr], align 8
 // CHECK2-NEXT:    [[_TMP22:%.*]] = alloca i32, align 4
+// CHECK2-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK2-NEXT:    [[DOTOFFLOAD_BASEPTRS27:%.*]] = alloca [1 x ptr], align 8
 // CHECK2-NEXT:    [[DOTOFFLOAD_PTRS28:%.*]] = alloca [1 x ptr], align 8
 // CHECK2-NEXT:    [[DOTOFFLOAD_MAPPERS29:%.*]] = alloca [1 x ptr], align 8
 // CHECK2-NEXT:    [[_TMP30:%.*]] = alloca i32, align 4
+// CHECK2-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK2-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK2-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK2-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -1365,7 +1370,6 @@ int main (int argc, char **argv) {
 // CHECK2-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK2-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK2-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK2-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK2-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK2-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK2-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1408,7 +1412,6 @@ int main (int argc, char **argv) {
 // CHECK2-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK2-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK2-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK2-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK2-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK2-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK2-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -1451,7 +1454,6 @@ int main (int argc, char **argv) {
 // CHECK2-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK2-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK2-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK2-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK2-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK2-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK2-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1494,7 +1496,6 @@ int main (int argc, char **argv) {
 // CHECK2-NEXT:    store ptr null, ptr [[TMP62]], align 8
 // CHECK2-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS19]], i32 0, i32 0
 // CHECK2-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS20]], i32 0, i32 0
-// CHECK2-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK2-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 0
 // CHECK2-NEXT:    store i32 2, ptr [[TMP65]], align 4
 // CHECK2-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 1
@@ -1537,7 +1538,6 @@ int main (int argc, char **argv) {
 // CHECK2-NEXT:    store ptr null, ptr [[TMP82]], align 8
 // CHECK2-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS27]], i32 0, i32 0
 // CHECK2-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS28]], i32 0, i32 0
-// CHECK2-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK2-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK2-NEXT:    store i32 2, ptr [[TMP85]], align 4
 // CHECK2-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -2442,22 +2442,27 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK5-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
 // CHECK5-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 4
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 4
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 4
 // CHECK5-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [1 x ptr], align 4
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [1 x ptr], align 4
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS21:%.*]] = alloca [1 x ptr], align 4
 // CHECK5-NEXT:    [[_TMP22:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[DOTOFFLOAD_BASEPTRS27:%.*]] = alloca [1 x ptr], align 4
 // CHECK5-NEXT:    [[DOTOFFLOAD_PTRS28:%.*]] = alloca [1 x ptr], align 4
 // CHECK5-NEXT:    [[DOTOFFLOAD_MAPPERS29:%.*]] = alloca [1 x ptr], align 4
 // CHECK5-NEXT:    [[_TMP30:%.*]] = alloca i32, align 4
+// CHECK5-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK5-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK5-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -2469,7 +2474,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK5-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK5-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK5-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2512,7 +2516,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK5-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK5-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -2555,7 +2558,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK5-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK5-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -2598,7 +2600,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP62]], align 4
 // CHECK5-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS19]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS20]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP65]], align 4
 // CHECK5-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 1
@@ -2641,7 +2642,6 @@ int main (int argc, char **argv) {
 // CHECK5-NEXT:    store ptr null, ptr [[TMP82]], align 4
 // CHECK5-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS27]], i32 0, i32 0
 // CHECK5-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS28]], i32 0, i32 0
-// CHECK5-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK5-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK5-NEXT:    store i32 2, ptr [[TMP85]], align 4
 // CHECK5-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -3519,22 +3519,27 @@ int main (int argc, char **argv) {
 // CHECK6-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK6-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK6-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK6-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK6-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK6-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK6-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
 // CHECK6-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK6-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK6-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 4
 // CHECK6-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 4
 // CHECK6-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 4
 // CHECK6-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK6-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK6-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [1 x ptr], align 4
 // CHECK6-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [1 x ptr], align 4
 // CHECK6-NEXT:    [[DOTOFFLOAD_MAPPERS21:%.*]] = alloca [1 x ptr], align 4
 // CHECK6-NEXT:    [[_TMP22:%.*]] = alloca i32, align 4
+// CHECK6-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK6-NEXT:    [[DOTOFFLOAD_BASEPTRS27:%.*]] = alloca [1 x ptr], align 4
 // CHECK6-NEXT:    [[DOTOFFLOAD_PTRS28:%.*]] = alloca [1 x ptr], align 4
 // CHECK6-NEXT:    [[DOTOFFLOAD_MAPPERS29:%.*]] = alloca [1 x ptr], align 4
 // CHECK6-NEXT:    [[_TMP30:%.*]] = alloca i32, align 4
+// CHECK6-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK6-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK6-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK6-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -3546,7 +3551,6 @@ int main (int argc, char **argv) {
 // CHECK6-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK6-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK6-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK6-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK6-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK6-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK6-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3589,7 +3593,6 @@ int main (int argc, char **argv) {
 // CHECK6-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK6-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK6-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK6-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK6-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK6-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK6-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -3632,7 +3635,6 @@ int main (int argc, char **argv) {
 // CHECK6-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK6-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK6-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK6-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK6-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK6-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK6-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -3675,7 +3677,6 @@ int main (int argc, char **argv) {
 // CHECK6-NEXT:    store ptr null, ptr [[TMP62]], align 4
 // CHECK6-NEXT:    [[TMP63:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS19]], i32 0, i32 0
 // CHECK6-NEXT:    [[TMP64:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS20]], i32 0, i32 0
-// CHECK6-NEXT:    [[KERNEL_ARGS23:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK6-NEXT:    [[TMP65:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 0
 // CHECK6-NEXT:    store i32 2, ptr [[TMP65]], align 4
 // CHECK6-NEXT:    [[TMP66:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS23]], i32 0, i32 1
@@ -3718,7 +3719,6 @@ int main (int argc, char **argv) {
 // CHECK6-NEXT:    store ptr null, ptr [[TMP82]], align 4
 // CHECK6-NEXT:    [[TMP83:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS27]], i32 0, i32 0
 // CHECK6-NEXT:    [[TMP84:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS28]], i32 0, i32 0
-// CHECK6-NEXT:    [[KERNEL_ARGS31:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK6-NEXT:    [[TMP85:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 0
 // CHECK6-NEXT:    store i32 2, ptr [[TMP85]], align 4
 // CHECK6-NEXT:    [[TMP86:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS31]], i32 0, i32 1
@@ -4983,6 +4983,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 8
@@ -4991,6 +4992,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[M_CASTED:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[N_CASTED18:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [4 x ptr], align 8
@@ -5000,6 +5002,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_24:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[N_CASTED33:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS34:%.*]] = alloca [3 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS35:%.*]] = alloca [3 x ptr], align 8
@@ -5008,6 +5011,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[_TMP38:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_39:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_40:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS45:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[M_CASTED48:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[N_CASTED49:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS50:%.*]] = alloca [4 x ptr], align 8
@@ -5017,6 +5021,7 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[_TMP54:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_55:%.*]] = alloca i32, align 4
 // CHECK13-NEXT:    [[DOTCAPTURE_EXPR_56:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK13-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK13-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -5066,7 +5071,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK13-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK13-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK13-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -5138,7 +5142,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP56:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK13-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP56]], 1
 // CHECK13-NEXT:    [[TMP57:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK13-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -5219,7 +5222,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP96:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_25]], align 4
 // CHECK13-NEXT:    [[ADD29:%.*]] = add nsw i32 [[TMP96]], 1
 // CHECK13-NEXT:    [[TMP97:%.*]] = zext i32 [[ADD29]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP98]], align 4
 // CHECK13-NEXT:    [[TMP99:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 1
@@ -5291,7 +5293,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP131:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_40]], align 4
 // CHECK13-NEXT:    [[ADD44:%.*]] = add nsw i32 [[TMP131]], 1
 // CHECK13-NEXT:    [[TMP132:%.*]] = zext i32 [[ADD44]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS45:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP133:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS45]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP133]], align 4
 // CHECK13-NEXT:    [[TMP134:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS45]], i32 0, i32 1
@@ -5372,7 +5373,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[TMP171:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_56]], align 4
 // CHECK13-NEXT:    [[ADD60:%.*]] = add nsw i32 [[TMP171]], 1
 // CHECK13-NEXT:    [[TMP172:%.*]] = zext i32 [[ADD60]] to i64
-// CHECK13-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP173:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP173]], align 4
 // CHECK13-NEXT:    [[TMP174:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 1
@@ -6652,24 +6652,29 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[M_CASTED:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [1 x ptr], align 8
 // CHECK13-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[M_CASTED22:%.*]] = alloca i64, align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_BASEPTRS23:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_PTRS24:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[DOTOFFLOAD_MAPPERS25:%.*]] = alloca [2 x ptr], align 8
 // CHECK13-NEXT:    [[_TMP26:%.*]] = alloca i32, align 4
+// CHECK13-NEXT:    [[KERNEL_ARGS27:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK13-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK13-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -6680,7 +6685,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK13-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK13-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK13-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -6722,7 +6726,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK13-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK13-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -6773,7 +6776,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP47]], align 8
 // CHECK13-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP50]], align 4
 // CHECK13-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -6815,7 +6817,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP67]], align 8
 // CHECK13-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP70]], align 4
 // CHECK13-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -6866,7 +6867,6 @@ int main (int argc, char **argv) {
 // CHECK13-NEXT:    store ptr null, ptr [[TMP92]], align 8
 // CHECK13-NEXT:    [[TMP93:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS23]], i32 0, i32 0
 // CHECK13-NEXT:    [[TMP94:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS24]], i32 0, i32 0
-// CHECK13-NEXT:    [[KERNEL_ARGS27:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK13-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS27]], i32 0, i32 0
 // CHECK13-NEXT:    store i32 2, ptr [[TMP95]], align 4
 // CHECK13-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS27]], i32 0, i32 1
@@ -7801,6 +7801,7 @@ int main (int argc, char **argv) {
 // CHECK14-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK14-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK14-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK14-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK14-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 8
@@ -7809,6 +7810,7 @@ int main (int argc, char **argv) {
 // CHECK14-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK14-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK14-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK14-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK14-NEXT:    [[M_CASTED:%.*]] = alloca i64, align 8
 // CHECK14-NEXT:    [[N_CASTED18:%.*]] = alloca i64, align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [4 x ptr], align 8
@@ -7818,6 +7820,7 @@ int main (int argc, char **argv) {
 // CHECK14-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
 // CHECK14-NEXT:    [[DOTCAPTURE_EXPR_24:%.*]] = alloca i32, align 4
 // CHECK14-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
+// CHECK14-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK14-NEXT:    [[N_CASTED33:%.*]] = alloca i64, align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_BASEPTRS34:%.*]] = alloca [3 x ptr], align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_PTRS35:%.*]] = alloca [3 x ptr], align 8
@@ -7826,6 +7829,7 @@ int main (int argc, char **argv) {
 // CHECK14-NEXT:    [[_TMP38:%.*]] = alloca i32, align 4
 // CHECK14-NEXT:    [[DOTCAPTURE_EXPR_39:%.*]] = alloca i32, align 4
 // CHECK14-NEXT:    [[DOTCAPTURE_EXPR_40:%.*]] = alloca i32, align 4
+// CHECK14-NEXT:    [[KERNEL_ARGS45:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK14-NEXT:    [[M_CASTED48:%.*]] = alloca i64, align 8
 // CHECK14-NEXT:    [[N_CASTED49:%.*]] = alloca i64, align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_BASEPTRS50:%.*]] = alloca [4 x ptr], align 8
@@ -7835,6 +7839,7 @@ int main (int argc, char **argv) {
 // CHECK14-NEXT:    [[_TMP54:%.*]] = alloca i32, align 4
 // CHECK14-NEXT:    [[DOTCAPTURE_EXPR_55:%.*]] = alloca i32, align 4
 // CHECK14-NEXT:    [[DOTCAPTURE_EXPR_56:%.*]] = alloca i32, align 4
+// CHECK14-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK14-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK14-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK14-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -7884,7 +7889,6 @@ int main (int argc, char **argv) {
 // CHECK14-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK14-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK14-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK14-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK14-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK14-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK14-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -7956,7 +7960,6 @@ int main (int argc, char **argv) {
 // CHECK14-NEXT:    [[TMP56:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK14-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP56]], 1
 // CHECK14-NEXT:    [[TMP57:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK14-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK14-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK14-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK14-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -8037,7 +8040,6 @@ int main (int argc, char **argv) {
 // CHECK14-NEXT:    [[TMP96:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_25]], align 4
 // CHECK14-NEXT:    [[ADD29:%.*]] = add nsw i32 [[TMP96]], 1
 // CHECK14-NEXT:    [[TMP97:%.*]] = zext i32 [[ADD29]] to i64
-// CHECK14-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK14-NEXT:    [[TMP98:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 0
 // CHECK14-NEXT:    store i32 2, ptr [[TMP98]], align 4
 // CHECK14-NEXT:    [[TMP99:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 1
@@ -8109,7 +8111,6 @@ int main (int argc, char **argv) {
 // CHECK14-NEXT:    [[TMP131:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_40]], align 4
 // CHECK14-NEXT:    [[ADD44:%.*]] = add nsw i32 [[TMP131]], 1
 // CHECK14-NEXT:    [[TMP132:%.*]] = zext i32 [[ADD44]] to i64
-// CHECK14-NEXT:    [[KERNEL_ARGS45:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK14-NEXT:    [[TMP133:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS45]], i32 0, i32 0
 // CHECK14-NEXT:    store i32 2, ptr [[TMP133]], align 4
 // CHECK14-NEXT:    [[TMP134:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS45]], i32 0, i32 1
@@ -8190,7 +8191,6 @@ int main (int argc, char **argv) {
 // CHECK14-NEXT:    [[TMP171:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_56]], align 4
 // CHECK14-NEXT:    [[ADD60:%.*]] = add nsw i32 [[TMP171]], 1
 // CHECK14-NEXT:    [[TMP172:%.*]] = zext i32 [[ADD60]] to i64
-// CHECK14-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK14-NEXT:    [[TMP173:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 0
 // CHECK14-NEXT:    store i32 2, ptr [[TMP173]], align 4
 // CHECK14-NEXT:    [[TMP174:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 1
@@ -9470,24 +9470,29 @@ int main (int argc, char **argv) {
 // CHECK14-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK14-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK14-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK14-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK14-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK14-NEXT:    [[M_CASTED:%.*]] = alloca i64, align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 8
 // CHECK14-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK14-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [1 x ptr], align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [1 x ptr], align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [1 x ptr], align 8
 // CHECK14-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK14-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK14-NEXT:    [[M_CASTED22:%.*]] = alloca i64, align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_BASEPTRS23:%.*]] = alloca [2 x ptr], align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_PTRS24:%.*]] = alloca [2 x ptr], align 8
 // CHECK14-NEXT:    [[DOTOFFLOAD_MAPPERS25:%.*]] = alloca [2 x ptr], align 8
 // CHECK14-NEXT:    [[_TMP26:%.*]] = alloca i32, align 4
+// CHECK14-NEXT:    [[KERNEL_ARGS27:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK14-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK14-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK14-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -9498,7 +9503,6 @@ int main (int argc, char **argv) {
 // CHECK14-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK14-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK14-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK14-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK14-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK14-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK14-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -9540,7 +9544,6 @@ int main (int argc, char **argv) {
 // CHECK14-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK14-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK14-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK14-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK14-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK14-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK14-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -9591,7 +9594,6 @@ int main (int argc, char **argv) {
 // CHECK14-NEXT:    store ptr null, ptr [[TMP47]], align 8
 // CHECK14-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK14-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK14-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK14-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK14-NEXT:    store i32 2, ptr [[TMP50]], align 4
 // CHECK14-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -9633,7 +9635,6 @@ int main (int argc, char **argv) {
 // CHECK14-NEXT:    store ptr null, ptr [[TMP67]], align 8
 // CHECK14-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK14-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK14-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK14-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK14-NEXT:    store i32 2, ptr [[TMP70]], align 4
 // CHECK14-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -9684,7 +9685,6 @@ int main (int argc, char **argv) {
 // CHECK14-NEXT:    store ptr null, ptr [[TMP92]], align 8
 // CHECK14-NEXT:    [[TMP93:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS23]], i32 0, i32 0
 // CHECK14-NEXT:    [[TMP94:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS24]], i32 0, i32 0
-// CHECK14-NEXT:    [[KERNEL_ARGS27:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK14-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS27]], i32 0, i32 0
 // CHECK14-NEXT:    store i32 2, ptr [[TMP95]], align 4
 // CHECK14-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS27]], i32 0, i32 1
@@ -10619,6 +10619,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 4
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 4
@@ -10627,6 +10628,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[M_CASTED:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[N_CASTED18:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [4 x ptr], align 4
@@ -10636,6 +10638,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_24:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[N_CASTED33:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS34:%.*]] = alloca [3 x ptr], align 4
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS35:%.*]] = alloca [3 x ptr], align 4
@@ -10644,6 +10647,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[_TMP38:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_39:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_40:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS45:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[M_CASTED48:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[N_CASTED49:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS50:%.*]] = alloca [4 x ptr], align 4
@@ -10653,6 +10657,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[_TMP54:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_55:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTCAPTURE_EXPR_56:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK17-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK17-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -10702,7 +10707,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK17-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK17-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK17-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -10775,7 +10779,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP57:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK17-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP57]], 1
 // CHECK17-NEXT:    [[TMP58:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP59]], align 4
 // CHECK17-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -10857,7 +10860,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP98:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_25]], align 4
 // CHECK17-NEXT:    [[ADD29:%.*]] = add nsw i32 [[TMP98]], 1
 // CHECK17-NEXT:    [[TMP99:%.*]] = zext i32 [[ADD29]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP100:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP100]], align 4
 // CHECK17-NEXT:    [[TMP101:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 1
@@ -10930,7 +10932,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP134:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_40]], align 4
 // CHECK17-NEXT:    [[ADD44:%.*]] = add nsw i32 [[TMP134]], 1
 // CHECK17-NEXT:    [[TMP135:%.*]] = zext i32 [[ADD44]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS45:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP136:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS45]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP136]], align 4
 // CHECK17-NEXT:    [[TMP137:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS45]], i32 0, i32 1
@@ -11012,7 +11013,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP175:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_56]], align 4
 // CHECK17-NEXT:    [[ADD60:%.*]] = add nsw i32 [[TMP175]], 1
 // CHECK17-NEXT:    [[TMP176:%.*]] = zext i32 [[ADD60]] to i64
-// CHECK17-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP177:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP177]], align 4
 // CHECK17-NEXT:    [[TMP178:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 1
@@ -12267,24 +12267,29 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK17-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 4
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK17-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[M_CASTED:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 4
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 4
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 4
 // CHECK17-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [1 x ptr], align 4
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [1 x ptr], align 4
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [1 x ptr], align 4
 // CHECK17-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[M_CASTED22:%.*]] = alloca i32, align 4
 // CHECK17-NEXT:    [[DOTOFFLOAD_BASEPTRS23:%.*]] = alloca [2 x ptr], align 4
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS24:%.*]] = alloca [2 x ptr], align 4
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS25:%.*]] = alloca [2 x ptr], align 4
 // CHECK17-NEXT:    [[_TMP26:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS27:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK17-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK17-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -12295,7 +12300,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK17-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK17-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -12337,7 +12341,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK17-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK17-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -12388,7 +12391,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP47]], align 4
 // CHECK17-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP50]], align 4
 // CHECK17-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -12430,7 +12432,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP67]], align 4
 // CHECK17-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP70]], align 4
 // CHECK17-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -12481,7 +12482,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    store ptr null, ptr [[TMP92]], align 4
 // CHECK17-NEXT:    [[TMP93:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS23]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP94:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS24]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS27:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK17-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS27]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP95]], align 4
 // CHECK17-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS27]], i32 0, i32 1
@@ -13389,6 +13389,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 4
@@ -13397,6 +13398,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[M_CASTED:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[N_CASTED18:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [4 x ptr], align 4
@@ -13406,6 +13408,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_24:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[N_CASTED33:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS34:%.*]] = alloca [3 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS35:%.*]] = alloca [3 x ptr], align 4
@@ -13414,6 +13417,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[_TMP38:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_39:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_40:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS45:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[M_CASTED48:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[N_CASTED49:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS50:%.*]] = alloca [4 x ptr], align 4
@@ -13423,6 +13427,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[_TMP54:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_55:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTCAPTURE_EXPR_56:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK19-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK19-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -13472,7 +13477,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK19-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK19-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK19-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -13545,7 +13549,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP57:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK19-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP57]], 1
 // CHECK19-NEXT:    [[TMP58:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP59]], align 4
 // CHECK19-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -13627,7 +13630,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP98:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_25]], align 4
 // CHECK19-NEXT:    [[ADD29:%.*]] = add nsw i32 [[TMP98]], 1
 // CHECK19-NEXT:    [[TMP99:%.*]] = zext i32 [[ADD29]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP100:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP100]], align 4
 // CHECK19-NEXT:    [[TMP101:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 1
@@ -13700,7 +13702,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP134:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_40]], align 4
 // CHECK19-NEXT:    [[ADD44:%.*]] = add nsw i32 [[TMP134]], 1
 // CHECK19-NEXT:    [[TMP135:%.*]] = zext i32 [[ADD44]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS45:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP136:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS45]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP136]], align 4
 // CHECK19-NEXT:    [[TMP137:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS45]], i32 0, i32 1
@@ -13782,7 +13783,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP175:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_56]], align 4
 // CHECK19-NEXT:    [[ADD60:%.*]] = add nsw i32 [[TMP175]], 1
 // CHECK19-NEXT:    [[TMP176:%.*]] = zext i32 [[ADD60]] to i64
-// CHECK19-NEXT:    [[KERNEL_ARGS61:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP177:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP177]], align 4
 // CHECK19-NEXT:    [[TMP178:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS61]], i32 0, i32 1
@@ -15037,24 +15037,29 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[M_CASTED:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS15:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS16:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS17:%.*]] = alloca [1 x ptr], align 4
 // CHECK19-NEXT:    [[_TMP18:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[M_CASTED22:%.*]] = alloca i32, align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_BASEPTRS23:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS24:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS25:%.*]] = alloca [2 x ptr], align 4
 // CHECK19-NEXT:    [[_TMP26:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS27:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK19-NEXT:    store i32 10, ptr [[M]], align 4
 // CHECK19-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
@@ -15065,7 +15070,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK19-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK19-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -15107,7 +15111,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK19-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK19-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -15158,7 +15161,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP47]], align 4
 // CHECK19-NEXT:    [[TMP48:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP49:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP50]], align 4
 // CHECK19-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -15200,7 +15202,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP67]], align 4
 // CHECK19-NEXT:    [[TMP68:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS15]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP69:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS16]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS19:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP70:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP70]], align 4
 // CHECK19-NEXT:    [[TMP71:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS19]], i32 0, i32 1
@@ -15251,7 +15252,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    store ptr null, ptr [[TMP92]], align 4
 // CHECK19-NEXT:    [[TMP93:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_BASEPTRS23]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP94:%.*]] = getelementptr inbounds [2 x ptr], ptr [[DOTOFFLOAD_PTRS24]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS27:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK19-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS27]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP95]], align 4
 // CHECK19-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS27]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_private_codegen.cpp b/clang/test/OpenMP/teams_distribute_private_codegen.cpp
index e01cf84087b9f..58c2cab50319d 100644
--- a/clang/test/OpenMP/teams_distribute_private_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_private_codegen.cpp
@@ -240,8 +240,8 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -401,6 +401,7 @@ int main() {
 // CHECK1-NEXT:    [[VAR:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -410,7 +411,6 @@ int main() {
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef signext 2)
 // CHECK1-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 8
 // CHECK1-NEXT:    store ptr undef, ptr [[_TMP1]], align 8
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -775,8 +775,8 @@ int main() {
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -934,6 +934,7 @@ int main() {
 // CHECK3-NEXT:    [[VAR:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -943,7 +944,6 @@ int main() {
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef 2)
 // CHECK3-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 4
 // CHECK3-NEXT:    store ptr undef, ptr [[_TMP1]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_reduction_codegen.cpp b/clang/test/OpenMP/teams_distribute_reduction_codegen.cpp
index 9659ee44f480b..a0c18f8d837d1 100644
--- a/clang/test/OpenMP/teams_distribute_reduction_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_reduction_codegen.cpp
@@ -91,6 +91,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr @_ZZ4mainE5sivar, align 4
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[SIVAR_CASTED]], align 4
@@ -103,7 +104,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -270,6 +270,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr [[T_VAR]], align 4
@@ -283,7 +284,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -455,6 +455,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr @_ZZ4mainE5sivar, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[SIVAR_CASTED]], align 4
@@ -467,7 +468,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP4]], align 4
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -634,6 +634,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr [[T_VAR]], align 4
@@ -647,7 +648,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP4]], align 4
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_simd_codegen.cpp b/clang/test/OpenMP/teams_distribute_simd_codegen.cpp
index 249b8922165e3..c7060469881a0 100644
--- a/clang/test/OpenMP/teams_distribute_simd_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_simd_codegen.cpp
@@ -228,6 +228,7 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[N_CASTED4:%.*]] = alloca i64, align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS5:%.*]] = alloca [2 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [2 x ptr], align 8
@@ -235,6 +236,7 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
 // CHECK1-NEXT:    [[DIV:%.*]] = sdiv i32 [[TMP0]], 128
@@ -296,7 +298,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP29]], 1
 // CHECK1-NEXT:    [[TMP30:%.*]] = zext i32 [[ADD]] to i64
 // CHECK1-NEXT:    [[TMP31:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP26]], 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP32]], align 4
 // CHECK1-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -357,7 +358,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[TMP59:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK1-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP59]], 1
 // CHECK1-NEXT:    [[TMP60:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP61]], align 4
 // CHECK1-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -666,6 +666,7 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[N_CASTED4:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS5:%.*]] = alloca [2 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS6:%.*]] = alloca [2 x ptr], align 4
@@ -673,6 +674,7 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N_ADDR]], align 4
 // CHECK3-NEXT:    [[DIV:%.*]] = sdiv i32 [[TMP0]], 128
@@ -734,7 +736,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP29]], 1
 // CHECK3-NEXT:    [[TMP30:%.*]] = zext i32 [[ADD]] to i64
 // CHECK3-NEXT:    [[TMP31:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP26]], 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP32]], align 4
 // CHECK3-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -795,7 +796,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[TMP59:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK3-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP59]], 1
 // CHECK3-NEXT:    [[TMP60:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP61:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP61]], align 4
 // CHECK3-NEXT:    [[TMP62:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1367,6 +1367,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 100, ptr [[N]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = zext i32 [[TMP0]] to i64
@@ -1412,7 +1413,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK9-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1596,6 +1596,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 100, ptr [[N]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = load i32, ptr [[N]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = call ptr @llvm.stacksave()
@@ -1641,7 +1642,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK11-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1979,6 +1979,7 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [3 x i64], align 8
 // CHECK17-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK17-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK17-NEXT:    [[B:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 1
@@ -2012,7 +2013,6 @@ int main (int argc, char **argv) {
 // CHECK17-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK17-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2164,6 +2164,7 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [3 x i64], align 4
 // CHECK19-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK19-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK19-NEXT:    [[B:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 1
@@ -2197,7 +2198,6 @@ int main (int argc, char **argv) {
 // CHECK19-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK19-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2348,6 +2348,7 @@ int main (int argc, char **argv) {
 // CHECK21-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
 // CHECK21-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [3 x i64], align 8
 // CHECK21-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK21-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK21-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK21-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK21-NEXT:    [[B:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 1
@@ -2381,7 +2382,6 @@ int main (int argc, char **argv) {
 // CHECK21-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK21-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK21-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK21-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK21-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK21-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK21-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2583,6 +2583,7 @@ int main (int argc, char **argv) {
 // CHECK23-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
 // CHECK23-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [3 x i64], align 4
 // CHECK23-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK23-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK23-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK23-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK23-NEXT:    [[B:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 1
@@ -2616,7 +2617,6 @@ int main (int argc, char **argv) {
 // CHECK23-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK23-NEXT:    [[TMP16:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK23-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [3 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK23-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK23-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK23-NEXT:    store i32 2, ptr [[TMP18]], align 4
 // CHECK23-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3129,6 +3129,7 @@ int main (int argc, char **argv) {
 // CHECK33-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK33-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK33-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK33-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK33-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK33-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK33-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -3177,7 +3178,6 @@ int main (int argc, char **argv) {
 // CHECK33-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK33-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK33-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK33-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK33-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK33-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK33-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3355,6 +3355,7 @@ int main (int argc, char **argv) {
 // CHECK33-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK33-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
 // CHECK33-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK33-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK33-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK33-NEXT:    store i32 0, ptr [[TE]], align 4
 // CHECK33-NEXT:    store i32 128, ptr [[TH]], align 4
@@ -3386,7 +3387,6 @@ int main (int argc, char **argv) {
 // CHECK33-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK33-NEXT:    [[TMP15:%.*]] = load i32, ptr [[TE]], align 4
 // CHECK33-NEXT:    [[TMP16:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP15]], 0
-// CHECK33-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK33-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK33-NEXT:    store i32 2, ptr [[TMP17]], align 4
 // CHECK33-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3539,6 +3539,7 @@ int main (int argc, char **argv) {
 // CHECK35-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK35-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK35-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK35-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK35-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK35-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK35-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -3587,7 +3588,6 @@ int main (int argc, char **argv) {
 // CHECK35-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK35-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK35-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK35-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK35-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK35-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK35-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3764,6 +3764,7 @@ int main (int argc, char **argv) {
 // CHECK35-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK35-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
 // CHECK35-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK35-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK35-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK35-NEXT:    store i32 0, ptr [[TE]], align 4
 // CHECK35-NEXT:    store i32 128, ptr [[TH]], align 4
@@ -3795,7 +3796,6 @@ int main (int argc, char **argv) {
 // CHECK35-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK35-NEXT:    [[TMP15:%.*]] = load i32, ptr [[TE]], align 4
 // CHECK35-NEXT:    [[TMP16:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP15]], 0
-// CHECK35-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK35-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK35-NEXT:    store i32 2, ptr [[TMP17]], align 4
 // CHECK35-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3948,6 +3948,7 @@ int main (int argc, char **argv) {
 // CHECK37-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK37-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK37-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK37-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK37-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK37-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK37-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -4005,7 +4006,6 @@ int main (int argc, char **argv) {
 // CHECK37-NEXT:    [[TMP26:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK37-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP26]], 1
 // CHECK37-NEXT:    [[TMP27:%.*]] = zext i32 [[ADD]] to i64
-// CHECK37-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK37-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK37-NEXT:    store i32 2, ptr [[TMP28]], align 4
 // CHECK37-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4230,6 +4230,7 @@ int main (int argc, char **argv) {
 // CHECK37-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 8
 // CHECK37-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 8
 // CHECK37-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK37-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK37-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK37-NEXT:    store i32 0, ptr [[TE]], align 4
 // CHECK37-NEXT:    store i32 128, ptr [[TH]], align 4
@@ -4261,7 +4262,6 @@ int main (int argc, char **argv) {
 // CHECK37-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK37-NEXT:    [[TMP15:%.*]] = load i32, ptr [[TE]], align 4
 // CHECK37-NEXT:    [[TMP16:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP15]], 0
-// CHECK37-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK37-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK37-NEXT:    store i32 2, ptr [[TMP17]], align 4
 // CHECK37-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4415,6 +4415,7 @@ int main (int argc, char **argv) {
 // CHECK39-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK39-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK39-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK39-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK39-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK39-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK39-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -4472,7 +4473,6 @@ int main (int argc, char **argv) {
 // CHECK39-NEXT:    [[TMP26:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK39-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP26]], 1
 // CHECK39-NEXT:    [[TMP27:%.*]] = zext i32 [[ADD]] to i64
-// CHECK39-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK39-NEXT:    [[TMP28:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK39-NEXT:    store i32 2, ptr [[TMP28]], align 4
 // CHECK39-NEXT:    [[TMP29:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -4695,6 +4695,7 @@ int main (int argc, char **argv) {
 // CHECK39-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [3 x ptr], align 4
 // CHECK39-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [3 x ptr], align 4
 // CHECK39-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK39-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK39-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK39-NEXT:    store i32 0, ptr [[TE]], align 4
 // CHECK39-NEXT:    store i32 128, ptr [[TH]], align 4
@@ -4726,7 +4727,6 @@ int main (int argc, char **argv) {
 // CHECK39-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [3 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK39-NEXT:    [[TMP15:%.*]] = load i32, ptr [[TE]], align 4
 // CHECK39-NEXT:    [[TMP16:%.*]] = insertvalue [3 x i32] zeroinitializer, i32 [[TMP15]], 0
-// CHECK39-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK39-NEXT:    [[TMP17:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK39-NEXT:    store i32 2, ptr [[TMP17]], align 4
 // CHECK39-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_simd_collapse_codegen.cpp b/clang/test/OpenMP/teams_distribute_simd_collapse_codegen.cpp
index efde7b7425f65..20414448bc831 100644
--- a/clang/test/OpenMP/teams_distribute_simd_collapse_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_simd_collapse_codegen.cpp
@@ -115,6 +115,7 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -126,7 +127,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -291,6 +291,7 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[_TMP2:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -302,7 +303,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -610,6 +610,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i64, align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -686,7 +687,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store i64 [[SUB7]], ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK9-NEXT:    [[TMP35:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i64 [[TMP35]], 1
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP36]], align 4
 // CHECK9-NEXT:    [[TMP37:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -925,6 +925,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    store ptr [[A]], ptr [[TMP0]], align 8
@@ -934,7 +935,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK9-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1099,6 +1099,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_2:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_3:%.*]] = alloca i64, align 8
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -1174,7 +1175,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store i64 [[SUB7]], ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK11-NEXT:    [[TMP34:%.*]] = load i64, ptr [[DOTCAPTURE_EXPR_3]], align 8
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i64 [[TMP34]], 1
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP35]], align 4
 // CHECK11-NEXT:    [[TMP36:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1411,6 +1411,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    store ptr [[A]], ptr [[TMP0]], align 4
@@ -1420,7 +1421,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK11-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_simd_dist_schedule_codegen.cpp b/clang/test/OpenMP/teams_distribute_simd_dist_schedule_codegen.cpp
index a90b9ebf04ffb..96910943e34d3 100644
--- a/clang/test/OpenMP/teams_distribute_simd_dist_schedule_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_simd_dist_schedule_codegen.cpp
@@ -147,14 +147,17 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -166,7 +169,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK1-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -209,7 +211,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK1-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK1-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -252,7 +253,6 @@ int main (int argc, char **argv) {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK1-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK1-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK1-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -587,14 +587,17 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS4:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS5:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP6:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS11:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS12:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS13:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP14:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SS:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -606,7 +609,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK3-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -649,7 +651,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK3-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS3]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS4]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS7:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK3-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS7]], i32 0, i32 1
@@ -692,7 +693,6 @@ int main (int argc, char **argv) {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK3-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS11]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS12]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK3-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK3-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1272,6 +1272,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[N_CASTED3:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 8
@@ -1280,6 +1281,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[N_CASTED18:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [3 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [3 x ptr], align 8
@@ -1288,6 +1290,7 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_24:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 8
@@ -1336,7 +1339,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK9-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK9-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1408,7 +1410,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP56:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK9-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP56]], 1
 // CHECK9-NEXT:    [[TMP57:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP58:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP58]], align 4
 // CHECK9-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -1480,7 +1481,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[TMP91:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_25]], align 4
 // CHECK9-NEXT:    [[ADD29:%.*]] = add nsw i32 [[TMP91]], 1
 // CHECK9-NEXT:    [[TMP92:%.*]] = zext i32 [[ADD29]] to i64
-// CHECK9-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP93:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP93]], align 4
 // CHECK9-NEXT:    [[TMP94:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 1
@@ -1925,14 +1925,17 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    store ptr [[A]], ptr [[TMP0]], align 8
@@ -1942,7 +1945,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK9-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1984,7 +1986,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -2026,7 +2027,6 @@ int main (int argc, char **argv) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP42]], align 8
 // CHECK9-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK9-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
@@ -2356,6 +2356,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[N_CASTED3:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS4:%.*]] = alloca [3 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS5:%.*]] = alloca [3 x ptr], align 4
@@ -2364,6 +2365,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[_TMP8:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_9:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_10:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[N_CASTED18:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS19:%.*]] = alloca [3 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS20:%.*]] = alloca [3 x ptr], align 4
@@ -2372,6 +2374,7 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[_TMP23:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_24:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTCAPTURE_EXPR_25:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    store ptr [[ARGV]], ptr [[ARGV_ADDR]], align 4
@@ -2420,7 +2423,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP21:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_1]], align 4
 // CHECK11-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP21]], 1
 // CHECK11-NEXT:    [[TMP22:%.*]] = zext i32 [[ADD]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP23]], align 4
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2493,7 +2495,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP57:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_10]], align 4
 // CHECK11-NEXT:    [[ADD14:%.*]] = add nsw i32 [[TMP57]], 1
 // CHECK11-NEXT:    [[TMP58:%.*]] = zext i32 [[ADD14]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS15:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP59:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP59]], align 4
 // CHECK11-NEXT:    [[TMP60:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS15]], i32 0, i32 1
@@ -2566,7 +2567,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[TMP93:%.*]] = load i32, ptr [[DOTCAPTURE_EXPR_25]], align 4
 // CHECK11-NEXT:    [[ADD29:%.*]] = add nsw i32 [[TMP93]], 1
 // CHECK11-NEXT:    [[TMP94:%.*]] = zext i32 [[ADD29]] to i64
-// CHECK11-NEXT:    [[KERNEL_ARGS30:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP95:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP95]], align 4
 // CHECK11-NEXT:    [[TMP96:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS30]], i32 0, i32 1
@@ -3008,14 +3008,17 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP4:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS8:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS9:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS10:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP11:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 [[ARGC]], ptr [[ARGC_ADDR]], align 4
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    store ptr [[A]], ptr [[TMP0]], align 4
@@ -3025,7 +3028,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK11-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -3067,7 +3069,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -3109,7 +3110,6 @@ int main (int argc, char **argv) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP42]], align 4
 // CHECK11-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS8]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS9]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS12:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP45:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP45]], align 4
 // CHECK11-NEXT:    [[TMP46:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS12]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_simd_firstprivate_codegen.cpp b/clang/test/OpenMP/teams_distribute_simd_firstprivate_codegen.cpp
index 164a8d91650b8..217c8b1dc9a8f 100644
--- a/clang/test/OpenMP/teams_distribute_simd_firstprivate_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_simd_firstprivate_codegen.cpp
@@ -265,6 +265,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr @t_var, align 4
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[T_VAR_CASTED]], align 4
@@ -304,7 +305,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP18]], align 8
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK1-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -552,6 +552,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -594,7 +595,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP17]], align 8
 // CHECK1-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK1-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1090,6 +1090,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr @t_var, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[T_VAR_CASTED]], align 4
@@ -1129,7 +1130,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP18]], align 4
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK3-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1375,6 +1375,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1417,7 +1418,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP17]], align 4
 // CHECK3-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK3-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_simd_lastprivate_codegen.cpp b/clang/test/OpenMP/teams_distribute_simd_lastprivate_codegen.cpp
index 80a7ccad9bba1..8b2bda6303e72 100644
--- a/clang/test/OpenMP/teams_distribute_simd_lastprivate_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_simd_lastprivate_codegen.cpp
@@ -526,6 +526,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    store ptr [[G]], ptr [[G1]], align 8
@@ -580,7 +581,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP22]], align 8
 // CHECK9-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK9-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -856,6 +856,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK9-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -898,7 +899,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP17]], align 8
 // CHECK9-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK9-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1243,6 +1243,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    store ptr [[G]], ptr [[G1]], align 4
@@ -1297,7 +1298,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP22]], align 4
 // CHECK11-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP24:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP25:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP25]], align 4
 // CHECK11-NEXT:    [[TMP26:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1571,6 +1571,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[_TMP1:%.*]] = alloca i32, align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK11-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1613,7 +1614,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP17]], align 4
 // CHECK11-NEXT:    [[TMP18:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP20]], align 4
 // CHECK11-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_simd_private_codegen.cpp b/clang/test/OpenMP/teams_distribute_simd_private_codegen.cpp
index e4fb4beba475d..f33edf0e50b0e 100644
--- a/clang/test/OpenMP/teams_distribute_simd_private_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_simd_private_codegen.cpp
@@ -241,8 +241,8 @@ int main() {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -409,6 +409,7 @@ int main() {
 // CHECK1-NEXT:    [[VAR:%.*]] = alloca ptr, align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK1-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -418,7 +419,6 @@ int main() {
 // CHECK1-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef signext 2)
 // CHECK1-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 8
 // CHECK1-NEXT:    store ptr undef, ptr [[_TMP1]], align 8
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK1-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -790,8 +790,8 @@ int main() {
 // CHECK3-NEXT:  entry:
 // CHECK3-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
-// CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
+// CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -956,6 +956,7 @@ int main() {
 // CHECK3-NEXT:    [[VAR:%.*]] = alloca ptr, align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
 // CHECK3-NEXT:    [[_TMP1:%.*]] = alloca ptr, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -965,7 +966,6 @@ int main() {
 // CHECK3-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef 2)
 // CHECK3-NEXT:    store ptr [[TEST]], ptr [[VAR]], align 4
 // CHECK3-NEXT:    store ptr undef, ptr [[_TMP1]], align 4
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK3-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_distribute_simd_reduction_codegen.cpp b/clang/test/OpenMP/teams_distribute_simd_reduction_codegen.cpp
index 22912243f2015..3d53139a19d0d 100644
--- a/clang/test/OpenMP/teams_distribute_simd_reduction_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_simd_reduction_codegen.cpp
@@ -91,6 +91,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr @_ZZ4mainE5sivar, align 4
 // CHECK1-NEXT:    store i32 [[TMP0]], ptr [[SIVAR_CASTED]], align 4
@@ -103,7 +104,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -277,6 +277,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK1-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i32, ptr [[T_VAR]], align 4
@@ -290,7 +291,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP4]], align 8
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK1-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -469,6 +469,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr @_ZZ4mainE5sivar, align 4
 // CHECK3-NEXT:    store i32 [[TMP0]], ptr [[SIVAR_CASTED]], align 4
@@ -481,7 +482,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP4]], align 4
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -655,6 +655,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[TMP:%.*]] = alloca i32, align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store i32 0, ptr [[T_VAR]], align 4
 // CHECK3-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[VEC]], ptr align 4 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
 // CHECK3-NEXT:    [[TMP0:%.*]] = load i32, ptr [[T_VAR]], align 4
@@ -668,7 +669,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP4]], align 4
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP7]], align 4
 // CHECK3-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_firstprivate_codegen.cpp b/clang/test/OpenMP/teams_firstprivate_codegen.cpp
index e08d498d874bf..c1617e16b402c 100644
--- a/clang/test/OpenMP/teams_firstprivate_codegen.cpp
+++ b/clang/test/OpenMP/teams_firstprivate_codegen.cpp
@@ -295,10 +295,12 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[T_VAR_CASTED1:%.*]] = alloca i64, align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS3:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS4:%.*]] = alloca [1 x ptr], align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    call void @_ZN1SIfEC1Ev(ptr nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 4
@@ -346,7 +348,6 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP18]], align 8
 // CHECK9-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK9-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -391,7 +392,6 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP40]], align 8
 // CHECK9-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS2]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS3]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP43]], align 4
 // CHECK9-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -640,9 +640,11 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS4:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ev(ptr nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 128
 // CHECK9-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 128 [[VEC]], ptr align 128 @__const._Z5tmainIiET_v.vec, i64 8, i1 false)
@@ -677,7 +679,6 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP11]], align 8
 // CHECK9-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP14]], align 4
 // CHECK9-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -719,7 +720,6 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP31]], align 8
 // CHECK9-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS4:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK9-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS4]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP34]], align 4
 // CHECK9-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS4]], i32 0, i32 1
@@ -1096,10 +1096,12 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [5 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [5 x ptr], align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[T_VAR_CASTED1:%.*]] = alloca i32, align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS3:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS4:%.*]] = alloca [1 x ptr], align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    call void @_ZN1SIfEC1Ev(ptr nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 4
@@ -1147,7 +1149,6 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP18]], align 4
 // CHECK11-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [5 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP21:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP21]], align 4
 // CHECK11-NEXT:    [[TMP22:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1192,7 +1193,6 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP40]], align 4
 // CHECK11-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS2]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS3]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS5:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP43]], align 4
 // CHECK11-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS5]], i32 0, i32 1
@@ -1441,9 +1441,11 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [4 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [4 x ptr], align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS1:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS2:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS3:%.*]] = alloca [1 x ptr], align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS4:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ev(ptr nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 128
 // CHECK11-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 128 [[VEC]], ptr align 128 @__const._Z5tmainIiET_v.vec, i32 8, i1 false)
@@ -1478,7 +1480,6 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP11]], align 4
 // CHECK11-NEXT:    [[TMP12:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP13:%.*]] = getelementptr inbounds [4 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP14]], align 4
 // CHECK11-NEXT:    [[TMP15:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1520,7 +1521,6 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP31]], align 4
 // CHECK11-NEXT:    [[TMP32:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS1]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP33:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS2]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS4:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS]], align 8
 // CHECK11-NEXT:    [[TMP34:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS4]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP34]], align 4
 // CHECK11-NEXT:    [[TMP35:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS4]], i32 0, i32 1
@@ -1898,6 +1898,7 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [8 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [8 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [8 x i64], align 8
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 8
 // CHECK17-NEXT:    store ptr [[S]], ptr [[S_ADDR]], align 8
 // CHECK17-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
@@ -1976,7 +1977,6 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK17-NEXT:    [[TMP40:%.*]] = getelementptr inbounds [8 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [8 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [8 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP43:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP43]], align 4
 // CHECK17-NEXT:    [[TMP44:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2111,6 +2111,7 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK17-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [10 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [10 x ptr], align 8
 // CHECK17-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [10 x i64], align 8
+// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK17-NEXT:    store ptr [[S]], ptr [[S_ADDR]], align 8
 // CHECK17-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
@@ -2214,7 +2215,6 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK17-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK17-NEXT:    [[TMP54:%.*]] = getelementptr inbounds [10 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK17-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK17-NEXT:    [[TMP55:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK17-NEXT:    store i32 2, ptr [[TMP55]], align 4
 // CHECK17-NEXT:    [[TMP56:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2370,6 +2370,7 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [8 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [8 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [8 x i64], align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    store ptr [[A]], ptr [[A_ADDR]], align 4
 // CHECK19-NEXT:    store ptr [[S]], ptr [[S_ADDR]], align 4
 // CHECK19-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
@@ -2446,7 +2447,6 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK19-NEXT:    [[TMP38:%.*]] = getelementptr inbounds [8 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP39:%.*]] = getelementptr inbounds [8 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP40:%.*]] = getelementptr inbounds [8 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP41:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP41]], align 4
 // CHECK19-NEXT:    [[TMP42:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -2581,6 +2581,7 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK19-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [10 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [10 x ptr], align 4
 // CHECK19-NEXT:    [[DOTOFFLOAD_SIZES:%.*]] = alloca [10 x i64], align 4
+// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK19-NEXT:    store ptr [[S]], ptr [[S_ADDR]], align 4
 // CHECK19-NEXT:    store i32 [[N]], ptr [[N_ADDR]], align 4
@@ -2682,7 +2683,6 @@ void array_func(float a[3], St s[2], int n, long double vla1[n]) {
 // CHECK19-NEXT:    [[TMP50:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP51:%.*]] = getelementptr inbounds [10 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
 // CHECK19-NEXT:    [[TMP52:%.*]] = getelementptr inbounds [10 x i64], ptr [[DOTOFFLOAD_SIZES]], i32 0, i32 0
-// CHECK19-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK19-NEXT:    [[TMP53:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK19-NEXT:    store i32 2, ptr [[TMP53]], align 4
 // CHECK19-NEXT:    [[TMP54:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/OpenMP/teams_private_codegen.cpp b/clang/test/OpenMP/teams_private_codegen.cpp
index b8bb9ed3e7484..f6101e4cbf6ef 100644
--- a/clang/test/OpenMP/teams_private_codegen.cpp
+++ b/clang/test/OpenMP/teams_private_codegen.cpp
@@ -196,6 +196,7 @@ int main() {
 // CHECK1-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK1-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
+// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK1-NEXT:    store ptr [[D]], ptr [[D_ADDR]], align 8
 // CHECK1-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
@@ -217,7 +218,6 @@ int main() {
 // CHECK1-NEXT:    store ptr null, ptr [[TMP3]], align 8
 // CHECK1-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK1-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK1-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK1-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK1-NEXT:    store i32 2, ptr [[TMP6]], align 4
 // CHECK1-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -389,6 +389,7 @@ int main() {
 // CHECK3-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK3-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
+// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK3-NEXT:    store ptr [[D]], ptr [[D_ADDR]], align 4
 // CHECK3-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
@@ -410,7 +411,6 @@ int main() {
 // CHECK3-NEXT:    store ptr null, ptr [[TMP3]], align 4
 // CHECK3-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK3-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK3-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK3-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK3-NEXT:    store i32 2, ptr [[TMP6]], align 4
 // CHECK3-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -559,6 +559,7 @@ int main() {
 // CHECK9-NEXT:    [[VEC:%.*]] = alloca [2 x i32], align 4
 // CHECK9-NEXT:    [[S_ARR:%.*]] = alloca [2 x %struct.S], align 4
 // CHECK9-NEXT:    [[VAR:%.*]] = alloca [[STRUCT_S]], align 4
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK9-NEXT:    call void @_ZN2SSC1ERi(ptr noundef nonnull align 8 dereferenceable(16) [[SS]], ptr noundef nonnull align 4 dereferenceable(4) @_ZZ4mainE5sivar)
 // CHECK9-NEXT:    call void @_ZN1SIfEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
@@ -569,7 +570,6 @@ int main() {
 // CHECK9-NEXT:    [[ARRAYINIT_ELEMENT:%.*]] = getelementptr inbounds [[STRUCT_S]], ptr [[ARRAYINIT_BEGIN]], i64 1
 // CHECK9-NEXT:    call void @_ZN1SIfEC1Ef(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], float noundef 2.000000e+00)
 // CHECK9-NEXT:    call void @_ZN1SIfEC1Ef(ptr noundef nonnull align 4 dereferenceable(4) [[VAR]], float noundef 3.000000e+00)
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -727,6 +727,7 @@ int main() {
 // CHECK9-NEXT:    [[VEC:%.*]] = alloca [2 x i32], align 128
 // CHECK9-NEXT:    [[S_ARR:%.*]] = alloca [2 x %struct.S.0], align 128
 // CHECK9-NEXT:    [[VAR:%.*]] = alloca [[STRUCT_S_0]], align 128
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK9-NEXT:    call void @_ZN3SSTIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[SST]])
 // CHECK9-NEXT:    store i32 0, ptr [[T_VAR]], align 128
@@ -736,7 +737,6 @@ int main() {
 // CHECK9-NEXT:    [[ARRAYINIT_ELEMENT:%.*]] = getelementptr inbounds [[STRUCT_S_0]], ptr [[ARRAYINIT_BEGIN]], i64 1
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef signext 2)
 // CHECK9-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[VAR]], i32 noundef signext 3)
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK9-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -795,6 +795,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK9-NEXT:    store ptr [[D]], ptr [[D_ADDR]], align 8
 // CHECK9-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
@@ -816,7 +817,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP3]], align 8
 // CHECK9-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP6]], align 4
 // CHECK9-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1037,6 +1037,7 @@ int main() {
 // CHECK9-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 8
 // CHECK9-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 8
+// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 8
 // CHECK9-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 8
 // CHECK9-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SST:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -1049,7 +1050,6 @@ int main() {
 // CHECK9-NEXT:    store ptr null, ptr [[TMP2]], align 8
 // CHECK9-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK9-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK9-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK9-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK9-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK9-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1156,6 +1156,7 @@ int main() {
 // CHECK11-NEXT:    [[VEC:%.*]] = alloca [2 x i32], align 4
 // CHECK11-NEXT:    [[S_ARR:%.*]] = alloca [2 x %struct.S], align 4
 // CHECK11-NEXT:    [[VAR:%.*]] = alloca [[STRUCT_S]], align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK11-NEXT:    call void @_ZN2SSC1ERi(ptr noundef nonnull align 4 dereferenceable(12) [[SS]], ptr noundef nonnull align 4 dereferenceable(4) @_ZZ4mainE5sivar)
 // CHECK11-NEXT:    call void @_ZN1SIfEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
@@ -1166,7 +1167,6 @@ int main() {
 // CHECK11-NEXT:    [[ARRAYINIT_ELEMENT:%.*]] = getelementptr inbounds [[STRUCT_S]], ptr [[ARRAYINIT_BEGIN]], i32 1
 // CHECK11-NEXT:    call void @_ZN1SIfEC1Ef(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], float noundef 2.000000e+00)
 // CHECK11-NEXT:    call void @_ZN1SIfEC1Ef(ptr noundef nonnull align 4 dereferenceable(4) [[VAR]], float noundef 3.000000e+00)
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1324,6 +1324,7 @@ int main() {
 // CHECK11-NEXT:    [[VEC:%.*]] = alloca [2 x i32], align 128
 // CHECK11-NEXT:    [[S_ARR:%.*]] = alloca [2 x %struct.S.0], align 128
 // CHECK11-NEXT:    [[VAR:%.*]] = alloca [[STRUCT_S_0]], align 128
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[TEST]])
 // CHECK11-NEXT:    call void @_ZN3SSTIiEC1Ev(ptr noundef nonnull align 4 dereferenceable(4) [[SST]])
 // CHECK11-NEXT:    store i32 0, ptr [[T_VAR]], align 128
@@ -1333,7 +1334,6 @@ int main() {
 // CHECK11-NEXT:    [[ARRAYINIT_ELEMENT:%.*]] = getelementptr inbounds [[STRUCT_S_0]], ptr [[ARRAYINIT_BEGIN]], i32 1
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[ARRAYINIT_ELEMENT]], i32 noundef 2)
 // CHECK11-NEXT:    call void @_ZN1SIiEC1Ei(ptr noundef nonnull align 4 dereferenceable(4) [[VAR]], i32 noundef 3)
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP0]], align 4
 // CHECK11-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1392,6 +1392,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK11-NEXT:    store ptr [[D]], ptr [[D_ADDR]], align 4
 // CHECK11-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
@@ -1413,7 +1414,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP3]], align 4
 // CHECK11-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP6]], align 4
 // CHECK11-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
@@ -1634,6 +1634,7 @@ int main() {
 // CHECK11-NEXT:    [[DOTOFFLOAD_BASEPTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_PTRS:%.*]] = alloca [1 x ptr], align 4
 // CHECK11-NEXT:    [[DOTOFFLOAD_MAPPERS:%.*]] = alloca [1 x ptr], align 4
+// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    store ptr [[THIS]], ptr [[THIS_ADDR]], align 4
 // CHECK11-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR]], align 4
 // CHECK11-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_SST:%.*]], ptr [[THIS1]], i32 0, i32 0
@@ -1646,7 +1647,6 @@ int main() {
 // CHECK11-NEXT:    store ptr null, ptr [[TMP2]], align 4
 // CHECK11-NEXT:    [[TMP3:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_BASEPTRS]], i32 0, i32 0
 // CHECK11-NEXT:    [[TMP4:%.*]] = getelementptr inbounds [1 x ptr], ptr [[DOTOFFLOAD_PTRS]], i32 0, i32 0
-// CHECK11-NEXT:    [[KERNEL_ARGS:%.*]] = alloca [[STRUCT___TGT_KERNEL_ARGUMENTS:%.*]], align 8
 // CHECK11-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 0
 // CHECK11-NEXT:    store i32 2, ptr [[TMP5]], align 4
 // CHECK11-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [[STRUCT___TGT_KERNEL_ARGUMENTS]], ptr [[KERNEL_ARGS]], i32 0, i32 1
diff --git a/clang/test/Parser/wasm-funcref.c b/clang/test/Parser/wasm-funcref.c
new file mode 100644
index 0000000000000..5f5b1f60dc46b
--- /dev/null
+++ b/clang/test/Parser/wasm-funcref.c
@@ -0,0 +1,9 @@
+// RUN: %clang_cc1 -triple powerpc-linux-gnu -fsyntax-only -verify %s
+
+// Test that we trigger an error at parse time if using keyword funcref
+// while not using a wasm triple.
+typedef void (*__funcref funcref_t)();     // expected-error {{invalid use of '__funcref' keyword outside the WebAssembly triple}}
+typedef int (*__funcref fn_funcref_t)(int);// expected-error {{invalid use of '__funcref' keyword outside the WebAssembly triple}}
+typedef int (*fn_t)(int);
+
+static fn_funcref_t nullFuncref = 0;
diff --git a/clang/test/Preprocessor/Inputs/header-search-crash/foo.hmap.json b/clang/test/Preprocessor/Inputs/header-search-crash/foo.hmap.json
new file mode 100644
index 0000000000000..ccfd911f0f7fd
--- /dev/null
+++ b/clang/test/Preprocessor/Inputs/header-search-crash/foo.hmap.json
@@ -0,0 +1,6 @@
+{
+  "mappings" :
+    {
+     "Foo.h" : "Foo/Foo.h"
+    }
+}
diff --git a/clang/test/Preprocessor/aarch64-target-features.c b/clang/test/Preprocessor/aarch64-target-features.c
index 2a2f7efe34130..09f464466a56c 100644
--- a/clang/test/Preprocessor/aarch64-target-features.c
+++ b/clang/test/Preprocessor/aarch64-target-features.c
@@ -341,6 +341,10 @@
 // CHECK-MARCH-2: "-cc1"{{.*}} "-triple" "aarch64{{.*}}" "-target-feature" "-fp-armv8" "-target-feature" "-neon" "-target-feature" "-crc" "-target-feature" "-crypto"
 // CHECK-MARCH-3: "-cc1"{{.*}} "-triple" "aarch64{{.*}}" "-target-feature" "-neon"
 
+// While we're checking +nofp, also make sure it stops defining __ARM_FP
+// RUN: %clang -target aarch64-none-linux-gnu -march=armv8-r+nofp -x c -E -dM %s -o - | FileCheck -check-prefix=CHECK-NOFP %s
+// CHECK-NOFP-NOT: #define __ARM_FP{{ }}
+
 // Check +sm4:
 //
 // RUN: %clang -target aarch64 -march=armv8.2a+sm4 -### -c %s 2>&1 | FileCheck -check-prefix=CHECK-SM4 %s
diff --git a/clang/test/Preprocessor/directives_asm.S b/clang/test/Preprocessor/directives_asm.S
new file mode 100644
index 0000000000000..a384b911005f7
--- /dev/null
+++ b/clang/test/Preprocessor/directives_asm.S
@@ -0,0 +1,26 @@
+// REQUIRES: x86-registered-target
+// RUN: %clang --target=i386-pc-linux -c %s -o /dev/null 2>&1 | FileCheck %s
+
+// Check that preprocessor directives are recognised as such, but lines starting
+// with a # that aren't directives are instead treated as comments.
+
+#define MACRO .warning "This is a macro"
+        MACRO
+
+// CHECK: directives_asm.S:8:9: warning: This is a macro
+
+#not a preprocessing directive
+
+// CHECK-NOT: error: invalid preprocessing directive
+
+# 100
+
+        .warning "line number should not change"
+
+// CHECK: directives_asm.S:18:9: warning: line number should not change
+
+#line 100
+
+        .warning "line number should change"
+
+// CHECK: directives_asm.S:101:9: warning: line number should change
diff --git a/clang/test/Preprocessor/feature_tests.c b/clang/test/Preprocessor/feature_tests.c
index 929ff7d3c498b..07ca02279b911 100644
--- a/clang/test/Preprocessor/feature_tests.c
+++ b/clang/test/Preprocessor/feature_tests.c
@@ -24,6 +24,7 @@
 // still return true.
 #if !__has_builtin(__builtin_LINE) || \
     !__has_builtin(__builtin_FILE) || \
+    !__has_builtin(__builtin_FILE_NAME) || \
     !__has_builtin(__builtin_FUNCTION) || \
     !__has_builtin(__builtin_COLUMN) || \
     !__has_builtin(__builtin_types_compatible_p)
diff --git a/clang/test/Preprocessor/feature_tests.cpp b/clang/test/Preprocessor/feature_tests.cpp
index 848f219339724..810cfe88cae5f 100644
--- a/clang/test/Preprocessor/feature_tests.cpp
+++ b/clang/test/Preprocessor/feature_tests.cpp
@@ -23,6 +23,7 @@
 // still return true.
 #if !__has_builtin(__builtin_LINE) || \
     !__has_builtin(__builtin_FILE) || \
+    !__has_builtin(__builtin_FILE_NAME) || \
     !__has_builtin(__builtin_FUNCTION) || \
     !__has_builtin(__builtin_COLUMN) || \
     !__has_builtin(__array_rank) || \
diff --git a/clang/test/Preprocessor/flt_eval_macro.cpp b/clang/test/Preprocessor/flt_eval_macro.cpp
index 37c28f21333f9..fa5d6fcd820b1 100644
--- a/clang/test/Preprocessor/flt_eval_macro.cpp
+++ b/clang/test/Preprocessor/flt_eval_macro.cpp
@@ -17,7 +17,7 @@
 // RUN:   %s -o - | FileCheck %s  -strict-whitespace
 
 // RUN: %clang_cc1 -E -dM -triple=x86_64-apple-macos13.0 -ffast-math \
-// RUN:   %s -o - | FileCheck %s -check-prefix=CHECK-MINUS-ONE -strict-whitespace
+// RUN:   %s -o - | FileCheck %s -check-prefix=CHECK -strict-whitespace
 
 // RUN: %clang_cc1 -E -dM -triple i386-pc-windows -target-cpu pentium4 %s -o - \
 // RUN:   | FileCheck %s  -strict-whitespace
@@ -64,7 +64,6 @@
 
 int foo() {
   // CHECK: #define Name "One"
-  // CHECK-MINUS-ONE: #define Name "MinusOne"
   // EXT: #define Name "Three"
   return Name;
 }
diff --git a/clang/test/Preprocessor/header-search-crash.c b/clang/test/Preprocessor/header-search-crash.c
new file mode 100644
index 0000000000000..8c04216d18ba1
--- /dev/null
+++ b/clang/test/Preprocessor/header-search-crash.c
@@ -0,0 +1,5 @@
+// RUN: rm -rf %t && mkdir %t
+// RUN: %hmaptool write %S/Inputs/header-search-crash/foo.hmap.json %t/foo.hmap
+// RUN: %clang -cc1 -E %s -I %t/foo.hmap -verify
+
+#include "MissingHeader.h" // expected-error {{'MissingHeader.h' file not found}}
diff --git a/clang/test/Preprocessor/macro_redefined.S b/clang/test/Preprocessor/macro_redefined.S
new file mode 100644
index 0000000000000..9e3d5b08cbb2b
--- /dev/null
+++ b/clang/test/Preprocessor/macro_redefined.S
@@ -0,0 +1,10 @@
+// RUN: %clang %s -E -DCLI_MACRO=1 2>&1 | FileCheck %s
+
+#define CLI_MACRO
+// CHECK: macro_redefined.S{{.+}}: warning: 'CLI_MACRO' macro redefined
+// CHECK: <command line>{{.+}}: note: previous definition is here
+
+#define REGULAR_MACRO
+#define REGULAR_MACRO 1
+// CHECK: macro_redefined.S{{.+}}: warning: 'REGULAR_MACRO' macro redefined
+// CHECK: macro_redefined.S{{.+}}: note: previous definition is here
diff --git a/clang/test/Sema/aarch64-sve-vector-exp-ops.c b/clang/test/Sema/aarch64-sve-vector-exp-ops.c
new file mode 100644
index 0000000000000..b64bd9152054d
--- /dev/null
+++ b/clang/test/Sema/aarch64-sve-vector-exp-ops.c
@@ -0,0 +1,18 @@
+// RUN: %clang_cc1 -triple aarch64 -target-feature +f -target-feature +d \
+// RUN:   -target-feature +v -target-feature +zfh  -target-feature +sve -target-feature +experimental-zvfh \
+// RUN:   -disable-O0-optnone -o - -fsyntax-only %s -verify 
+// REQUIRES: aarch64-registered-target
+
+#include <arm_sve.h>
+
+svfloat32_t test_exp_vv_i8mf8(svfloat32_t v) {
+
+  return __builtin_elementwise_exp(v);
+  // expected-error@-1 {{1st argument must be a vector, integer or floating point type}}
+}
+
+svfloat32_t test_exp2_vv_i8mf8(svfloat32_t v) {
+
+  return __builtin_elementwise_exp2(v);
+  // expected-error@-1 {{1st argument must be a vector, integer or floating point type}}
+}
diff --git a/clang/test/Sema/attr-alias-elf.c b/clang/test/Sema/attr-alias-elf.c
index 9fcd872f9ebde..6b629e1b509ae 100644
--- a/clang/test/Sema/attr-alias-elf.c
+++ b/clang/test/Sema/attr-alias-elf.c
@@ -5,9 +5,10 @@ void g1(void) {
 }
 
 void f2(void) __attribute__((alias("g2"))); // expected-error {{alias must point to a defined variable or function}}
-
+// expected-note@-1 {{must refer to its mangled name}}
 
 void f3(void) __attribute__((alias("g3"))); // expected-error {{alias must point to a defined variable or function}}
+// expected-note@-1 {{must refer to its mangled name}}
 void g3(void);
 
 
@@ -46,11 +47,14 @@ extern int a1 __attribute__((alias("b1")));
 int b1 = 42;
 
 extern int a2 __attribute__((alias("b2"))); // expected-error {{alias must point to a defined variable or function}}
+// expected-note@-1 {{must refer to its mangled name}}
 
 extern int a3 __attribute__((alias("b3"))); // expected-error {{alias must point to a defined variable or function}}
+// expected-note@-1 {{must refer to its mangled name}}
 extern int b3;
 
 extern int a4 __attribute__((alias("b4"))); // expected-error {{alias must point to a defined variable or function}}
+// expected-note@-1 {{must refer to its mangled name}}
 typedef int b4;
 
 void test2_bar() {}
diff --git a/clang/test/Sema/attr-alwaysinline.cpp b/clang/test/Sema/attr-alwaysinline.cpp
index 6b8e8f215a4b6..be3f74c8bfa9d 100644
--- a/clang/test/Sema/attr-alwaysinline.cpp
+++ b/clang/test/Sema/attr-alwaysinline.cpp
@@ -25,3 +25,16 @@ void foo() {
 }
 
 [[clang::always_inline]] static int i = bar(); // expected-warning {{'always_inline' attribute only applies to functions and statements}}
+
+// This used to crash the compiler.
+template<int D>
+int foo(int x) {
+  [[clang::always_inline]] return foo<D-1>(x + 1);
+}
+
+// FIXME: This should warn that always_inline statement attribute has higher
+// precedence than the noinline function attribute.
+template<int D> [[gnu::noinline]]
+int bar(int x) {
+  [[clang::always_inline]] return bar<D-1>(x + 1);
+}
diff --git a/clang/test/Sema/attr-noinline.cpp b/clang/test/Sema/attr-noinline.cpp
index d35782f11adbb..ae0f80ca296eb 100644
--- a/clang/test/Sema/attr-noinline.cpp
+++ b/clang/test/Sema/attr-noinline.cpp
@@ -25,3 +25,16 @@ void foo() {
 }
 
 [[clang::noinline]] static int i = bar(); // expected-warning {{'noinline' attribute only applies to functions and statements}}
+
+// This used to crash the compiler.
+template<int D>
+int foo(int x) {
+  [[clang::noinline]] return foo<D-1>(x + 1);
+}
+
+// FIXME: This should warn that noinline statement attribute has higher
+// precedence than the always_inline function attribute.
+template<int D> [[clang::always_inline]]
+int bar(int x) {
+  [[clang::noinline]] return bar<D-1>(x + 1);
+}
diff --git a/clang/test/Sema/builtin_set_flt_rounds.c b/clang/test/Sema/builtin_set_flt_rounds.c
new file mode 100644
index 0000000000000..144e4a2a72dfa
--- /dev/null
+++ b/clang/test/Sema/builtin_set_flt_rounds.c
@@ -0,0 +1,8 @@
+// RUN: %clang_cc1 -triple mipsel-unknown-linux -fsyntax-only %s -verify=expected,unsupported
+// RUN: %clang_cc1 -triple x86_64-gnu-linux -fsyntax-only %s -verify
+struct S {int a;};
+void test_builtin_set_flt_rounds() {
+  __builtin_set_flt_rounds(1); // unsupported-error {{builtin is not supported on this target}}
+  struct S s;
+  __builtin_set_flt_rounds(s); // expected-error {{passing 'struct S' to parameter of incompatible type}}
+}
diff --git a/clang/test/Sema/builtins-elementwise-math.c b/clang/test/Sema/builtins-elementwise-math.c
index c803fcea0d505..35b065437e54f 100644
--- a/clang/test/Sema/builtins-elementwise-math.c
+++ b/clang/test/Sema/builtins-elementwise-math.c
@@ -311,6 +311,49 @@ void test_builtin_elementwise_cos(int i, float f, double d, float4 v, int3 iv, u
   // expected-error@-1 {{1st argument must be a floating point type (was 'unsigned4' (vector of 4 'unsigned int' values))}}
 }
 
+void test_builtin_elementwise_exp(int i, float f, double d, float4 v, int3 iv, unsigned u, unsigned4 uv) {
+
+  struct Foo s = __builtin_elementwise_exp(f);
+  // expected-error@-1 {{initializing 'struct Foo' with an expression of incompatible type 'float'}}
+
+  i = __builtin_elementwise_exp();
+  // expected-error@-1 {{too few arguments to function call, expected 1, have 0}}
+
+  i = __builtin_elementwise_exp(i);
+  // expected-error@-1 {{1st argument must be a floating point type (was 'int')}}
+
+  i = __builtin_elementwise_exp(f, f);
+  // expected-error@-1 {{too many arguments to function call, expected 1, have 2}}
+
+  u = __builtin_elementwise_exp(u);
+  // expected-error@-1 {{1st argument must be a floating point type (was 'unsigned int')}}
+
+  uv = __builtin_elementwise_exp(uv);
+  // expected-error@-1 {{1st argument must be a floating point type (was 'unsigned4' (vector of 4 'unsigned int' values))}}
+}
+
+void test_builtin_elementwise_exp2(int i, float f, double d, float4 v, int3 iv, unsigned u, unsigned4 uv) {
+
+  struct Foo s = __builtin_elementwise_exp2(f);
+  // expected-error@-1 {{initializing 'struct Foo' with an expression of incompatible type 'float'}}
+
+  i = __builtin_elementwise_exp2();
+  // expected-error@-1 {{too few arguments to function call, expected 1, have 0}}
+
+  i = __builtin_elementwise_exp2(i);
+  // expected-error@-1 {{1st argument must be a floating point type (was 'int')}}
+
+  i = __builtin_elementwise_exp2(f, f);
+  // expected-error@-1 {{too many arguments to function call, expected 1, have 2}}
+
+  u = __builtin_elementwise_exp2(u);
+  // expected-error@-1 {{1st argument must be a floating point type (was 'unsigned int')}}
+
+  uv = __builtin_elementwise_exp2(uv);
+  // expected-error@-1 {{1st argument must be a floating point type (was 'unsigned4' (vector of 4 'unsigned int' values))}}
+}
+
+
 void test_builtin_elementwise_floor(int i, float f, double d, float4 v, int3 iv, unsigned u, unsigned4 uv) {
 
   struct Foo s = __builtin_elementwise_floor(f);
diff --git a/clang/test/Sema/inline-asm-validate-m68k.c b/clang/test/Sema/inline-asm-validate-m68k.c
index a0b75304eba00..3abe638324452 100644
--- a/clang/test/Sema/inline-asm-validate-m68k.c
+++ b/clang/test/Sema/inline-asm-validate-m68k.c
@@ -82,5 +82,13 @@ void a(int x) {
 void d(int x) {
   asm ("" :: "d"(x));
 }
+
+// Memory constraints
+void mem() {
+  int x;
+  asm ("" :: "m"(x));
+  asm ("" :: "Q"(x));
+  asm ("" :: "U"(x));
+}
 #endif
 
diff --git a/clang/test/Sema/patchable-function-entry-attr.cpp b/clang/test/Sema/patchable-function-entry-attr.cpp
index 3dd0504987306..9134c851da588 100644
--- a/clang/test/Sema/patchable-function-entry-attr.cpp
+++ b/clang/test/Sema/patchable-function-entry-attr.cpp
@@ -2,6 +2,8 @@
 // RUN: %clang_cc1 -triple aarch64_be -fsyntax-only -verify=silence %s
 // RUN: %clang_cc1 -triple i386 -fsyntax-only -verify=silence %s
 // RUN: %clang_cc1 -triple x86_64 -fsyntax-only -verify=silence %s
+// RUN: %clang_cc1 -triple loongarch32 -fsyntax-only -verify=silence %s
+// RUN: %clang_cc1 -triple loongarch64 -fsyntax-only -verify=silence %s
 // RUN: %clang_cc1 -triple riscv32 -fsyntax-only -verify=silence %s
 // RUN: %clang_cc1 -triple riscv64 -fsyntax-only -verify=silence %s
 // RUN: %clang_cc1 -triple ppc64le -fsyntax-only -verify %s
diff --git a/clang/test/Sema/ppc-attr-target-inline.c b/clang/test/Sema/ppc-attr-target-inline.c
new file mode 100644
index 0000000000000..07ed006822658
--- /dev/null
+++ b/clang/test/Sema/ppc-attr-target-inline.c
@@ -0,0 +1,14 @@
+// REQUIRES: powerpc-registered-target
+// RUN: %clang_cc1 -triple powerpc64le -target-feature +htm -fsyntax-only -emit-llvm %s -verify
+
+__attribute__((always_inline))
+int test1(int *x) {
+  *x = __builtin_ttest();
+  return *x;
+}
+
+__attribute__((target("no-htm")))
+int test2(int *x) {
+  *x = test1(x); // expected-error {{always_inline function 'test1' requires target feature 'htm', but would be inlined into function 'test2' that is compiled without support for 'htm'}}
+  return 0;
+}
diff --git a/clang/test/Sema/riscv-sve-vector-exp-ops.c b/clang/test/Sema/riscv-sve-vector-exp-ops.c
new file mode 100644
index 0000000000000..f76c225403c13
--- /dev/null
+++ b/clang/test/Sema/riscv-sve-vector-exp-ops.c
@@ -0,0 +1,19 @@
+// RUN: %clang_cc1 -triple riscv64 -target-feature +f -target-feature +d \
+// RUN:   -target-feature +v -target-feature +zfh -target-feature +experimental-zvfh \
+// RUN:   -disable-O0-optnone -o - -fsyntax-only %s -verify 
+// REQUIRES: riscv-registered-target
+
+#include <riscv_vector.h>
+
+
+vfloat32mf2_t test_exp_vv_i8mf8(vfloat32mf2_t v) {
+
+  return __builtin_elementwise_exp(v);
+  // expected-error@-1 {{1st argument must be a vector, integer or floating point type}}
+}
+
+vfloat32mf2_t test_exp2_vv_i8mf8(vfloat32mf2_t v) {
+
+  return __builtin_elementwise_exp2(v);
+  // expected-error@-1 {{1st argument must be a vector, integer or floating point type}}
+}
diff --git a/clang/test/Sema/source_location.c b/clang/test/Sema/source_location.c
index 94c09c7952dc6..e4faf042d4876 100644
--- a/clang/test/Sema/source_location.c
+++ b/clang/test/Sema/source_location.c
@@ -17,12 +17,14 @@ char *const NCFUNC = __builtin_FUNCTION();
 
 #ifdef CONST_STRINGS
 _Static_assert(IsEqual(__builtin_FILE(), __FILE__), "");
+_Static_assert(IsEqual(__builtin_FILE_NAME(), __FILE_NAME__), "");
 _Static_assert(__builtin_LINE() == __LINE__, "");
 _Static_assert(IsEqual("", __builtin_FUNCTION()), "");
 
 #line 42 "my_file.c"
 _Static_assert(__builtin_LINE() == 42, "");
 _Static_assert(IsEqual(__builtin_FILE(), "my_file.c"), "");
+_Static_assert(IsEqual(__builtin_FILE_NAME(), "my_file.c"), "");
 
 _Static_assert(__builtin_COLUMN() == __builtin_strlen("_Static_assert(_"), "");
 
diff --git a/clang/test/Sema/warn-unreachable-fallthrough.c b/clang/test/Sema/warn-unreachable-fallthrough.c
new file mode 100644
index 0000000000000..e048a9e7e612f
--- /dev/null
+++ b/clang/test/Sema/warn-unreachable-fallthrough.c
@@ -0,0 +1,13 @@
+// RUN: %clang_cc1 -fsyntax-only -verify -std=c2x -Wunreachable-code-fallthrough %s
+// RUN: %clang_cc1 -fsyntax-only -verify -std=c2x -Wunreachable-code %s
+// RUN: %clang_cc1 -fsyntax-only -verify=code -std=c2x -Wunreachable-code -Wno-unreachable-code-fallthrough %s
+// RUN: %clang_cc1 -fsyntax-only -verify -std=c2x -Wno-unreachable-code -Wunreachable-code-fallthrough %s
+
+int n;
+void f(void){
+     switch (n){
+         [[fallthrough]]; // expected-warning{{fallthrough annotation in unreachable code}}
+                          // code-warning@-1{{never be executed}}
+         case 1:;
+     }
+}
diff --git a/clang/test/Sema/wasm-refs.c b/clang/test/Sema/wasm-refs.c
index 376d539556e6c..10aefc9aa04ee 100644
--- a/clang/test/Sema/wasm-refs.c
+++ b/clang/test/Sema/wasm-refs.c
@@ -46,6 +46,8 @@ __externref_t *illegal_return_1();   // expected-error {{pointer to WebAssembly
 __externref_t ***illegal_return_2(); // expected-error {{pointer to WebAssembly reference type is not allowed}}
 
 void varargs(int, ...);
+typedef void (*__funcref funcref_t)();
+typedef void (*__funcref __funcref funcref_fail_t)(); // expected-warning {{attribute '__funcref' is already applied}}
 
 __externref_t func(__externref_t ref) {
   &ref; // expected-error {{cannot take address of WebAssembly reference}}
@@ -67,5 +69,7 @@ __externref_t func(__externref_t ref) {
   _Alignof(__externref_t ***);   // expected-error {{pointer to WebAssembly reference type is not allowed}};
   varargs(1, ref);               // expected-error {{cannot pass expression of type '__externref_t' to variadic function}}
 
+  funcref_t func = __builtin_wasm_ref_null_func(0); // expected-error {{too many arguments to function call, expected 0, have 1}}
+
   return ref;
 }
diff --git a/clang/test/SemaCXX/Inputs/source-location-file.h b/clang/test/SemaCXX/Inputs/source-location-file.h
index 8a327b661e554..9ca979d40f8ce 100644
--- a/clang/test/SemaCXX/Inputs/source-location-file.h
+++ b/clang/test/SemaCXX/Inputs/source-location-file.h
@@ -4,8 +4,10 @@
 namespace source_location_file {
 
 constexpr const char *FILE = __FILE__;
+constexpr const char *FILE_NAME = __FILE_NAME__;
 
 constexpr SL global_info = SL::current();
+constexpr const char *global_info_filename = __builtin_FILE_NAME();
 
 constexpr SL test_function(SL v = SL::current()) {
   return v;
@@ -15,6 +17,15 @@ constexpr SL test_function_indirect() {
   return test_function();
 }
 
+constexpr const char *test_function_filename(
+                      const char *file_name = __builtin_FILE_NAME()) {
+  return file_name;
+}
+
+constexpr const char *test_function_filename_indirect() {
+  return test_function_filename();
+}
+
 template <class T, class U = SL>
 constexpr U test_function_template(T, U u = U::current()) {
   return u;
@@ -25,13 +36,29 @@ constexpr U test_function_template_indirect(T t) {
   return test_function_template(t);
 }
 
+template <class T, class U = const char *>
+constexpr U test_function_filename_template(T, U u = __builtin_FILE_NAME()) {
+  return u;
+}
+
+template <class T, class U = const char *>
+constexpr U test_function_filename_template_indirect(T t) {
+  return test_function_filename_template(t);
+}
+
 struct TestClass {
   SL info = SL::current();
+  const char *info_file_name = __builtin_FILE_NAME();
   SL ctor_info;
+  const char *ctor_info_file_name = nullptr;
   TestClass() = default;
-  constexpr TestClass(int, SL cinfo = SL::current()) : ctor_info(cinfo) {}
+  constexpr TestClass(int, SL cinfo = SL::current(),
+                      const char *cfile_name = __builtin_FILE_NAME()) :
+                      ctor_info(cinfo), ctor_info_file_name(cfile_name) {}
   template <class T, class U = SL>
-  constexpr TestClass(int, T, U u = U::current()) : ctor_info(u) {}
+  constexpr TestClass(int, T, U u = U::current(),
+                      const char *cfile_name = __builtin_FILE_NAME()) :
+                      ctor_info(u), ctor_info_file_name(cfile_name) {}
 };
 
 template <class T = SL>
@@ -39,6 +66,7 @@ struct AggrClass {
   int x;
   T info;
   T init_info = T::current();
+  const char *init_info_file_name = __builtin_FILE_NAME();
 };
 
 } // namespace source_location_file
diff --git a/clang/test/SemaCXX/builtins-elementwise-math.cpp b/clang/test/SemaCXX/builtins-elementwise-math.cpp
index 8e3e5d6f2d16a..78cfcacb6ab58 100644
--- a/clang/test/SemaCXX/builtins-elementwise-math.cpp
+++ b/clang/test/SemaCXX/builtins-elementwise-math.cpp
@@ -67,6 +67,20 @@ void test_builtin_elementwise_cos() {
   static_assert(!is_const<decltype(__builtin_elementwise_cos(b))>::value);
 }
 
+void test_builtin_elementwise_exp() {
+  const float a = 42.0;
+  float b = 42.3;
+  static_assert(!is_const<decltype(__builtin_elementwise_exp(a))>::value);
+  static_assert(!is_const<decltype(__builtin_elementwise_exp(b))>::value);
+}
+
+void test_builtin_elementwise_exp2() {
+  const float a = 42.0;
+  float b = 42.3;
+  static_assert(!is_const<decltype(__builtin_elementwise_exp2(a))>::value);
+  static_assert(!is_const<decltype(__builtin_elementwise_exp2(b))>::value);
+}
+
 void test_builtin_elementwise_sin() {
   const float a = 42.0;
   float b = 42.3;
diff --git a/clang/test/SemaCXX/constant-expression-cxx11.cpp b/clang/test/SemaCXX/constant-expression-cxx11.cpp
index fea45d56f0fd3..5e7a80c81eaee 100644
--- a/clang/test/SemaCXX/constant-expression-cxx11.cpp
+++ b/clang/test/SemaCXX/constant-expression-cxx11.cpp
@@ -279,7 +279,7 @@ namespace FunctionPointers {
   constexpr auto Select(int n) -> int (*)(int) {
     return n == 2 ? &Double : n == 3 ? &Triple : n == 4 ? &Quadruple : 0;
   }
-  constexpr int Apply(int (*F)(int), int n) { return F(n); } // expected-note {{subexpression}}
+  constexpr int Apply(int (*F)(int), int n) { return F(n); } // expected-note {{'F' evaluates to a null function pointer}}
 
   static_assert(1 + Apply(Select(4), 5) + Apply(Select(3), 7) == 42, "");
 
diff --git a/clang/test/SemaCXX/constexpr-single-element-array.cpp b/clang/test/SemaCXX/constexpr-single-element-array.cpp
new file mode 100644
index 0000000000000..a01b1a1c8f136
--- /dev/null
+++ b/clang/test/SemaCXX/constexpr-single-element-array.cpp
@@ -0,0 +1,19 @@
+// RUN: %clang_cc1 -std=c++20 -verify %s
+
+// This test makes sure that a single element array doesn't produce
+// spurious errors during constexpr evaluation.
+
+// expected-no-diagnostics
+struct Sub { int x; };
+
+struct S {
+  constexpr S() { Arr[0] = Sub{}; }
+  Sub Arr[1];
+};
+
+constexpr bool test() {
+  S s;
+  return true;
+}
+
+static_assert(test());
diff --git a/clang/test/SemaCXX/coroutine-alloc-4.cpp b/clang/test/SemaCXX/coroutine-alloc-4.cpp
index acad2779a2544..262c163fb1789 100644
--- a/clang/test/SemaCXX/coroutine-alloc-4.cpp
+++ b/clang/test/SemaCXX/coroutine-alloc-4.cpp
@@ -1,4 +1,4 @@
-// Tests that we'll find aligned allocation funciton properly.
+// Tests that we'll find aligned allocation function properly.
 // RUN: %clang_cc1 %s -std=c++20 %s -fsyntax-only -verify -fcoro-aligned-allocation
 
 #include "Inputs/std-coroutine.h"
diff --git a/clang/test/SemaCXX/cxx0x-nontrivial-union.cpp b/clang/test/SemaCXX/cxx0x-nontrivial-union.cpp
index ad1f09fee33f0..4a4bf7180a47f 100644
--- a/clang/test/SemaCXX/cxx0x-nontrivial-union.cpp
+++ b/clang/test/SemaCXX/cxx0x-nontrivial-union.cpp
@@ -144,3 +144,47 @@ namespace pr16061 {
 
   Test2<X> t2x;  // expected-error {{call to implicitly-deleted default constructor of 'Test2<X>'}}
 }
+
+namespace GH48416 {
+
+struct non_trivial_constructor {
+    constexpr non_trivial_constructor() : x(100) {}
+    int x;
+};
+
+
+union U1 {
+    int a;
+    non_trivial_constructor b; // expected-note {{has a non-trivial default constructor}}
+};
+
+union U2 {
+    int a{1000};
+    non_trivial_constructor b;
+};
+
+union U3 {
+    int a;
+    non_trivial_constructor b{};
+};
+
+union U4 {
+    int a{}; // expected-note {{previous initialization is here}}
+    non_trivial_constructor b{}; // expected-error {{initializing multiple members of union}}
+};
+
+U1 u1; // expected-error {{call to implicitly-deleted default constructor}}
+U2 u2;
+U3 u3;
+U4 u4;
+
+static_assert(U2().a == 1000, "");
+static_assert(U3().a == 1000, "");
+// expected-error@-1 {{static assertion expression is not an integral constant expression}}
+// expected-note@-2 {{read of member 'a' of union with active member 'b'}}
+static_assert(U2().b.x == 100, "");
+// expected-error@-1 {{static assertion expression is not an integral constant expression}}
+// expected-note@-2 {{read of member 'b' of union with active member 'a'}}
+static_assert(U3().b.x == 100, "");
+
+} // namespace GH48416
diff --git a/clang/test/SemaCXX/cxx1y-generic-lambdas.cpp b/clang/test/SemaCXX/cxx1y-generic-lambdas.cpp
index 463e077ce934c..61dfd654f6d65 100644
--- a/clang/test/SemaCXX/cxx1y-generic-lambdas.cpp
+++ b/clang/test/SemaCXX/cxx1y-generic-lambdas.cpp
@@ -1026,3 +1026,11 @@ namespace PR46637 {
   void *v = x(f); // expected-error {{cannot initialize a variable of type 'void *' with an rvalue of type 'int'}}
   void *w = y(f); // expected-error {{cannot initialize a variable of type 'void *' with an rvalue of type 'int'}}
 }
+
+namespace GH37792 {
+struct A { int x; };
+
+void f() {
+  [](auto t) -> decltype(decltype(t)::x) { return 0; }(A());
+}
+}
diff --git a/clang/test/SemaCXX/decltype.cpp b/clang/test/SemaCXX/decltype.cpp
index 32c61bbccc842..96abb60836e40 100644
--- a/clang/test/SemaCXX/decltype.cpp
+++ b/clang/test/SemaCXX/decltype.cpp
@@ -101,6 +101,44 @@ namespace D5789 {
   template<class T> void foo(decltype(T(LP1{ .p1 = g1, .p1.x[1] = 'x' }))) {}
 }
 
+namespace GH58674 {
+  struct Foo {
+    float value_;
+    struct nested {
+      float value_;
+    };
+  };
+
+  template <typename T>
+  struct TemplateFoo {
+    float value_;
+  };
+
+  float bar;
+
+  template <typename T>
+  struct Animal{};
+
+  template <typename T>
+  class Cat : Animal<T> {
+    using okay = decltype(Foo::value_);
+    using also_okay = decltype(bar);
+    using okay2 = decltype(Foo::nested::value_);
+    using okay3 = decltype(TemplateFoo<T>::value_);
+  public:
+    void meow() {
+      using okay = decltype(Foo::value_);
+      using also_okay = decltype(bar);
+      using okay2 = decltype(Foo::nested::value_);
+      using okay3 = decltype(TemplateFoo<T>::value_);
+    }
+  };
+
+  void baz() {
+      Cat<void>{}.meow();
+  }
+}
+
 template<typename>
 class conditional {
 };
diff --git a/clang/test/SemaCXX/dllexport.cpp b/clang/test/SemaCXX/dllexport.cpp
index bd4b0a7c78df0..8035ea5b61a7f 100644
--- a/clang/test/SemaCXX/dllexport.cpp
+++ b/clang/test/SemaCXX/dllexport.cpp
@@ -4,8 +4,8 @@
 // RUN: %clang_cc1 -triple x86_64-mingw32         -fsyntax-only -fms-extensions -verify -std=c++11 -Wunsupported-dll-base-class-template -DGNU %s
 // RUN: %clang_cc1 -triple i686-windows-itanium   -fsyntax-only -fms-extensions -verify -std=c++11 -Wunsupported-dll-base-class-template -DWI  %s
 // RUN: %clang_cc1 -triple x86_64-windows-itanium -fsyntax-only -fms-extensions -verify -std=c++1y -Wunsupported-dll-base-class-template -DWI  %s
-// RUN: %clang_cc1 -triple x86_64-scei-ps4        -fsyntax-only -fdeclspec      -verify -std=c++11 -Wunsupported-dll-base-class-template -DWI  %s
-// RUN: %clang_cc1 -triple x86_64-sie-ps5         -fsyntax-only -fdeclspec      -verify -std=c++1y -Wunsupported-dll-base-class-template -DWI  %s
+// RUN: %clang_cc1 -triple x86_64-scei-ps4        -fsyntax-only -fdeclspec      -verify -std=c++11 -Wunsupported-dll-base-class-template -DWI -DPS %s
+// RUN: %clang_cc1 -triple x86_64-sie-ps5         -fsyntax-only -fdeclspec      -verify -std=c++1y -Wunsupported-dll-base-class-template -DWI -DPS %s
 
 // Helper structs to make templates more expressive.
 struct ImplicitInst_Exported {};
@@ -390,13 +390,13 @@ template <typename T> struct __declspec(dllexport) ExplicitlyInstantiatedExporte
 template struct ExplicitlyInstantiatedExportedTemplate<IncompleteType3>; // expected-note{{in instantiation of member function 'ExplicitlyInstantiatedExportedTemplate<IncompleteType3>::f' requested here}}
 
 // In MS mode, instantiate members of class templates that are base classes of exported classes.
-#ifdef MS
+#if defined(MS) || defined(PS)
   // expected-note@+3{{forward declaration of 'IncompleteType4'}}
   // expected-note@+3{{in instantiation of member function 'BaseClassTemplateOfExportedClass<IncompleteType4>::f' requested here}}
 #endif
 struct IncompleteType4;
 template <typename T> struct BaseClassTemplateOfExportedClass {
-#ifdef MS
+#if defined(MS) || defined(PS)
   // expected-error@+2{{invalid application of 'sizeof' to an incomplete type 'IncompleteType4'}}
 #endif
   int f() { return sizeof(T); };
@@ -439,7 +439,7 @@ template <typename T> class __declspec(dllexport) ExportedClassTemplate {};
 template <typename T> class __declspec(dllimport) ImportedClassTemplate {};
 
 template <typename T> struct ExplicitlySpecializedTemplate { void func() {} };
-#ifdef MS
+#if defined(MS) || defined(PS)
 // expected-note@+2{{class template 'ExplicitlySpecializedTemplate<int>' was explicitly specialized here}}
 #endif
 template <> struct ExplicitlySpecializedTemplate<int> { void func() {} };
@@ -449,7 +449,7 @@ template <typename T> struct ExplicitlyImportSpecializedTemplate { void func() {
 template <> struct __declspec(dllimport) ExplicitlyImportSpecializedTemplate<int> { void func() {} };
 
 template <typename T> struct ExplicitlyInstantiatedTemplate { void func() {} };
-#ifdef MS
+#if defined(MS) || defined(PS)
 // expected-note@+2{{class template 'ExplicitlyInstantiatedTemplate<int>' was instantiated here}}
 #endif
 template struct ExplicitlyInstantiatedTemplate<int>;
@@ -488,7 +488,7 @@ class __declspec(dllexport) DerivedFromTemplateB : public ClassTemplate<bool> {}
 // The second derived class doesn't change anything, the attribute that was propagated first wins.
 class __declspec(dllimport) DerivedFromTemplateB2 : public ClassTemplate<bool> {};
 
-#ifdef MS
+#if defined(MS) || defined(PS)
 // expected-warning@+3{{propagating dll attribute to explicitly specialized base class template without dll attribute is not supported}}
 // expected-note@+2{{attribute is here}}
 #endif
@@ -500,7 +500,7 @@ struct __declspec(dllexport) DerivedFromExplicitlyExportSpecializedTemplate : pu
 // Base class already specialized with import attribute.
 struct __declspec(dllexport) DerivedFromExplicitlyImportSpecializedTemplate : public ExplicitlyImportSpecializedTemplate<int> {};
 
-#ifdef MS
+#if defined(MS) || defined(PS)
 // expected-warning@+3{{propagating dll attribute to already instantiated base class template without dll attribute is not supported}}
 // expected-note@+2{{attribute is here}}
 #endif
diff --git a/clang/test/SemaCXX/dllimport.cpp b/clang/test/SemaCXX/dllimport.cpp
index 6ab27433d9e93..a48d3786e1d8a 100644
--- a/clang/test/SemaCXX/dllimport.cpp
+++ b/clang/test/SemaCXX/dllimport.cpp
@@ -5,9 +5,9 @@
 // RUN: %clang_cc1 -triple x86_64-mingw32         -fsyntax-only -fms-extensions -verify -std=c++17 -Wunsupported-dll-base-class-template -DGNU %s
 // RUN: %clang_cc1 -triple i686-windows-itanium   -fsyntax-only -fms-extensions -verify -std=c++11 -Wunsupported-dll-base-class-template -DWI %s
 // RUN: %clang_cc1 -triple x86_64-windows-itanium -fsyntax-only -fms-extensions -verify -std=c++17 -Wunsupported-dll-base-class-template -DWI %s
-// RUN: %clang_cc1 -triple x86_64-scei-ps4        -fsyntax-only -fdeclspec      -verify -std=c++11 -Wunsupported-dll-base-class-template -DWI %s
-// RUN: %clang_cc1 -triple x86_64-scei-ps4        -fsyntax-only -fdeclspec      -verify -std=c++17 -Wunsupported-dll-base-class-template -DWI %s
-// RUN: %clang_cc1 -triple x86_64-sie-ps5         -fsyntax-only -fdeclspec      -verify -std=c++17 -Wunsupported-dll-base-class-template -DWI %s
+// RUN: %clang_cc1 -triple x86_64-scei-ps4        -fsyntax-only -fdeclspec      -verify -std=c++11 -Wunsupported-dll-base-class-template -DWI -DPS %s
+// RUN: %clang_cc1 -triple x86_64-scei-ps4        -fsyntax-only -fdeclspec      -verify -std=c++17 -Wunsupported-dll-base-class-template -DWI -DPS %s
+// RUN: %clang_cc1 -triple x86_64-sie-ps5         -fsyntax-only -fdeclspec      -verify -std=c++17 -Wunsupported-dll-base-class-template -DWI -DPS %s
 
 // Helper structs to make templates more expressive.
 struct ImplicitInst_Imported {};
@@ -1557,7 +1557,7 @@ class __declspec(dllimport) DerivedFromTemplateB : public ClassTemplate<bool> {}
 class __declspec(dllexport) DerivedFromTemplateB2 : public ClassTemplate<bool> {};
 
 template <typename T> struct ExplicitlySpecializedTemplate { void func() {} };
-#ifdef MS
+#if defined(MS) || defined(PS)
 // expected-note@+2{{class template 'ExplicitlySpecializedTemplate<int>' was explicitly specialized here}}
 #endif
 template <> struct ExplicitlySpecializedTemplate<int> { void func() {} };
@@ -1567,7 +1567,7 @@ template <typename T> struct ExplicitlyImportSpecializedTemplate { void func() {
 template <> struct __declspec(dllimport) ExplicitlyImportSpecializedTemplate<int> { void func() {} };
 
 template <typename T> struct ExplicitlyInstantiatedTemplate { void func() {} };
-#ifdef MS
+#if defined(MS) || defined(PS)
 // expected-note@+2{{class template 'ExplicitlyInstantiatedTemplate<int>' was instantiated here}}
 #endif
 template struct ExplicitlyInstantiatedTemplate<int>;
@@ -1576,7 +1576,7 @@ template struct __declspec(dllexport) ExplicitlyExportInstantiatedTemplate<int>;
 template <typename T> struct ExplicitlyImportInstantiatedTemplate { void func() {} };
 template struct __declspec(dllimport) ExplicitlyImportInstantiatedTemplate<int>;
 
-#ifdef MS
+#if defined(MS) || defined(PS)
 // expected-warning@+3{{propagating dll attribute to explicitly specialized base class template without dll attribute is not supported}}
 // expected-note@+2{{attribute is here}}
 #endif
@@ -1588,7 +1588,7 @@ struct __declspec(dllimport) DerivedFromExplicitlyExportSpecializedTemplate : pu
 // Base class already specialized with import attribute.
 struct __declspec(dllimport) DerivedFromExplicitlyImportSpecializedTemplate : public ExplicitlyImportSpecializedTemplate<int> {};
 
-#ifdef MS
+#if defined(MS) || defined(PS)
 // expected-warning@+3{{propagating dll attribute to already instantiated base class template without dll attribute is not supported}}
 // expected-note@+2{{attribute is here}}
 #endif
diff --git a/clang/test/SemaCXX/externc-ifunc-resolver.cpp b/clang/test/SemaCXX/externc-ifunc-resolver.cpp
index 70c21b0ac7f9b..aa44525bde2ca 100644
--- a/clang/test/SemaCXX/externc-ifunc-resolver.cpp
+++ b/clang/test/SemaCXX/externc-ifunc-resolver.cpp
@@ -1,4 +1,5 @@
 // RUN: %clang_cc1 -emit-llvm-only -triple x86_64-linux-gnu -verify %s
+// RUN: not %clang_cc1 -triple x86_64-linux -emit-llvm-only -fdiagnostics-parseable-fixits %s 2>&1 | FileCheck %s
 
 extern "C" {
 __attribute__((used)) static void *resolve_foo() { return 0; }
@@ -12,5 +13,10 @@ __attribute__((used)) static void *resolve_foo() { return 0; }
 // some way to improve this diagnostic (likely by diagnosing when we decide
 // this case suppresses alias creation).
 __attribute__((ifunc("resolve_foo"))) void foo(); // expected-error{{ifunc must point to a defined function}}
+// expected-note@-1 {{must refer to its mangled name}}
+// expected-note@-2 {{function by that name is mangled as}}
+// expected-note@-3 {{function by that name is mangled as}}
+// CHECK: fix-it:"{{.*}}":{[[@LINE-4]]:16-[[@LINE-4]]:36}:"ifunc(\"_ZL11resolve_foov\")"
+// CHECK: fix-it:"{{.*}}":{[[@LINE-5]]:16-[[@LINE-5]]:36}:"ifunc(\"_ZN2NSL11resolve_fooEv\")"
 }
 
diff --git a/clang/test/SemaCXX/fold_expr_typo.cpp b/clang/test/SemaCXX/fold_expr_typo.cpp
new file mode 100644
index 0000000000000..0ef9c15b59476
--- /dev/null
+++ b/clang/test/SemaCXX/fold_expr_typo.cpp
@@ -0,0 +1,14 @@
+// RUN: %clang_cc1 -fsyntax-only -verify -std=c++17 %s
+
+template <typename... T>
+void foo(T &&...Params) {
+  foo<T>(Unknown); // expected-error {{expression contains unexpanded parameter pack 'T'}}\
+                      expected-error {{use of undeclared identifier 'Unknown'}}
+  ((foo<T>(Unknown)), ...); // expected-error {{use of undeclared identifier 'Unknown'}}
+}
+
+template <typename... U> struct A {
+  template <typename... T> void foo(T &&...Params) {
+    foo<T>((... + static_cast<U>(1))); // expected-error {{expression contains unexpanded parameter pack 'T'}}
+  }
+};
diff --git a/clang/test/SemaCXX/source_location.cpp b/clang/test/SemaCXX/source_location.cpp
index f7c347ce357aa..06f90f9251643 100644
--- a/clang/test/SemaCXX/source_location.cpp
+++ b/clang/test/SemaCXX/source_location.cpp
@@ -84,6 +84,7 @@ constexpr bool is_same<T, T> = true;
 static_assert(is_same<decltype(__builtin_LINE()), unsigned>);
 static_assert(is_same<decltype(__builtin_COLUMN()), unsigned>);
 static_assert(is_same<decltype(__builtin_FILE()), const char *>);
+static_assert(is_same<decltype(__builtin_FILE_NAME()), const char *>);
 static_assert(is_same<decltype(__builtin_FUNCTION()), const char *>);
 static_assert(is_same<decltype(__builtin_source_location()), const std::source_location::public_impl_alias *>);
 
@@ -91,6 +92,7 @@ static_assert(is_same<decltype(__builtin_source_location()), const std::source_l
 static_assert(noexcept(__builtin_LINE()));
 static_assert(noexcept(__builtin_COLUMN()));
 static_assert(noexcept(__builtin_FILE()));
+static_assert(noexcept(__builtin_FILE_NAME()));
 static_assert(noexcept(__builtin_FUNCTION()));
 static_assert(noexcept(__builtin_source_location()));
 
@@ -346,6 +348,54 @@ void test_aggr_class() {
 
 } // namespace test_file
 
+//===----------------------------------------------------------------------===//
+//                            __builtin_FILE_NAME()
+//===----------------------------------------------------------------------===//
+
+namespace test_file_name {
+constexpr const char *test_file_name_simple(
+  const char *__f = __builtin_FILE_NAME()) {
+  return __f;
+}
+void test_function() {
+#line 900
+  static_assert(is_equal(test_file_name_simple(), __FILE_NAME__));
+  static_assert(is_equal(SLF::test_function_filename(), __FILE_NAME__), "");
+  static_assert(is_equal(SLF::test_function_filename_template(42),
+                         __FILE_NAME__), "");
+
+  static_assert(is_equal(SLF::test_function_filename_indirect(),
+                         SLF::global_info_filename), "");
+  static_assert(is_equal(SLF::test_function_filename_template_indirect(42),
+                         SLF::global_info_filename), "");
+
+  static_assert(test_file_name_simple() != nullptr);
+  static_assert(is_equal(test_file_name_simple(), "source_location.cpp"));
+}
+
+void test_class() {
+#line 315
+  using SLF::TestClass;
+  constexpr TestClass Default;
+  constexpr TestClass InParam{42};
+  constexpr TestClass Template{42, 42};
+  constexpr auto *F = Default.info_file_name;
+  constexpr auto Char = F[0];
+  static_assert(is_equal(Default.info_file_name, SLF::FILE_NAME), "");
+  static_assert(is_equal(InParam.info_file_name, SLF::FILE_NAME), "");
+  static_assert(is_equal(InParam.ctor_info_file_name, __FILE_NAME__), "");
+}
+
+void test_aggr_class() {
+  using Agg = SLF::AggrClass<>;
+  constexpr Agg Default{};
+  constexpr Agg InitOne{42};
+  static_assert(is_equal(Default.init_info_file_name, __FILE_NAME__), "");
+  static_assert(is_equal(InitOne.init_info_file_name, __FILE_NAME__), "");
+}
+
+} // namespace test_file_name
+
 //===----------------------------------------------------------------------===//
 //                            __builtin_FUNCTION()
 //===----------------------------------------------------------------------===//
@@ -487,6 +537,7 @@ static_assert(SL::current().line() == StartLine + 2);
 #line 44 "test_file.c"
 static_assert(is_equal("test_file.c", __FILE__));
 static_assert(is_equal("test_file.c", __builtin_FILE()));
+static_assert(is_equal("test_file.c", __builtin_FILE_NAME()));
 static_assert(is_equal("test_file.c", SL::current().file()));
 static_assert(is_equal("test_file.c", SLF::test_function().file()));
 static_assert(is_equal(SLF::FILE, SLF::test_function_indirect().file()));
diff --git a/clang/test/SemaCXX/statements.cpp b/clang/test/SemaCXX/statements.cpp
index 75ed983ba84b2..48f178dd9a8b3 100644
--- a/clang/test/SemaCXX/statements.cpp
+++ b/clang/test/SemaCXX/statements.cpp
@@ -52,3 +52,14 @@ void test8() {
   int a = test7(1);
   double b = test7(2.0);
 }
+
+namespace GH48405 {
+void foo() {
+  struct S {
+    int i;
+    int j = ({i;}); // expected-error {{invalid use of non-static data member 'i'}}
+                    // expected-error@-1 {{cannot initialize a member subobject of type 'int' with an rvalue of type 'void'}}
+		    // expected-warning@-2 {{use of GNU statement expression extension}}
+  };
+}
+}
diff --git a/clang/test/SemaCXX/unreachable-catch-clauses.cpp b/clang/test/SemaCXX/unreachable-catch-clauses.cpp
index 8dcb47b13c771..5041332779e64 100644
--- a/clang/test/SemaCXX/unreachable-catch-clauses.cpp
+++ b/clang/test/SemaCXX/unreachable-catch-clauses.cpp
@@ -9,5 +9,25 @@ void f();
 void test()
 try {}
 catch (BaseEx &e) { f(); } // expected-note 2{{for type 'BaseEx &'}}
-catch (Ex1 &e) { f(); } // expected-warning {{exception of type 'Ex1 &' will be caught by earlier handler}}
-catch (Ex2 &e) { f(); } // expected-warning {{exception of type 'Ex2 &' (aka 'Ex1 &') will be caught by earlier handler}}
+catch (Ex1 &e) { f(); } // expected-warning {{exception of type 'Ex1 &' will be caught by earlier handler}} \
+                           expected-note {{for type 'Ex1 &'}}
+// FIXME: It would be nicer to only issue one warning on the below line instead
+// of two. We get two diagnostics because the first one is noticing that there
+// is a class hierarchy inversion where the earlier base class handler will
+// catch throwing the derived class and the second one is because Ex2 and Ex1
+// are the same type after canonicalization.
+catch (Ex2 &e) { f(); } // expected-warning 2{{exception of type 'Ex2 &' (aka 'Ex1 &') will be caught by earlier handler}}
+
+namespace GH61177 {
+void func() {
+  const char arr[4] = "abc";
+
+  // We should not issue an "exception will be caught by earlier handler"
+  // diagnostic, as that is a lie.
+  try {
+    throw arr;
+  } catch (char *p) {
+  } catch (const char *p) {
+  }
+}
+} // GH61177
diff --git a/clang/test/SemaCXX/warn-unsafe-buffer-usage-fixits-local-var-span.cpp b/clang/test/SemaCXX/warn-unsafe-buffer-usage-fixits-local-var-span.cpp
index 3cb4f57a6efd6..dacd8673d58bf 100644
--- a/clang/test/SemaCXX/warn-unsafe-buffer-usage-fixits-local-var-span.cpp
+++ b/clang/test/SemaCXX/warn-unsafe-buffer-usage-fixits-local-var-span.cpp
@@ -1,4 +1,3 @@
-// REQUIRES: !system-windows
 // RUN: %clang_cc1 -std=c++20 -Wunsafe-buffer-usage -fdiagnostics-parseable-fixits %s 2>&1 | FileCheck %s
 typedef int * Int_ptr_t;
 typedef int Int_t;
diff --git a/clang/test/SemaCXX/wasm-funcref.cpp b/clang/test/SemaCXX/wasm-funcref.cpp
new file mode 100644
index 0000000000000..364565e9e803c
--- /dev/null
+++ b/clang/test/SemaCXX/wasm-funcref.cpp
@@ -0,0 +1,13 @@
+// RUN: %clang_cc1 -std=c++11 -fcxx-exceptions -fexceptions -fsyntax-only -verify -triple wasm32 -Wno-unused-value -target-feature +reference-types %s
+
+// Testing that funcrefs work on template aliases
+// expected-no-diagnostics
+
+using IntIntFuncref = int(*)(int) __funcref;
+using DoubleQual = IntIntFuncref __funcref;
+
+int get(int);
+
+IntIntFuncref getFuncref() {
+    return get;
+}
diff --git a/clang/test/SemaOpenCL/invalid-kernel-parameters.cl b/clang/test/SemaOpenCL/invalid-kernel-parameters.cl
index de06b381605b4..f0f56515d7731 100644
--- a/clang/test/SemaOpenCL/invalid-kernel-parameters.cl
+++ b/clang/test/SemaOpenCL/invalid-kernel-parameters.cl
@@ -1,4 +1,4 @@
-// RUN: %clang_cc1 -fsyntax-only -verify %s -triple spir-unknown-unknown
+// RUN: %clang_cc1 -fsyntax-only -verify=expected,ocl12 %s -triple spir-unknown-unknown
 // RUN: %clang_cc1 -fsyntax-only -verify %s -triple spir-unknown-unknown -cl-std=CL2.0
 
 kernel void half_arg(half x) { } // expected-error{{declaring function parameter of type '__private half' is not allowed; did you forget * ?}}
@@ -104,6 +104,13 @@ typedef struct Foo // expected-note{{within field of type 'Foo' declared here}}
 
 kernel void pointer_in_struct_arg(Foo arg) { } // expected-error{{struct kernel parameters may not contain pointers}}
 
+typedef struct FooGlobal // ocl12-note{{within field of type 'FooGlobal' declared here}}
+{
+  global int* ptrField; // ocl12-note{{field of illegal pointer type '__global int *' declared here}}
+} FooGlobal;
+
+kernel void global_pointer_in_struct_arg(FooGlobal arg) { } // ocl12-error{{struct kernel parameters may not contain pointers}}
+
 typedef union FooUnion // expected-note{{within field of type 'FooUnion' declared here}}
 {
   int* ptrField; // expected-note-re{{field of illegal pointer type '__{{private|generic}} int *' declared here}}
diff --git a/clang/test/SemaTemplate/address_space-dependent.cpp b/clang/test/SemaTemplate/address_space-dependent.cpp
index 3436cac4d4cb0..c8cc67ef45211 100644
--- a/clang/test/SemaTemplate/address_space-dependent.cpp
+++ b/clang/test/SemaTemplate/address_space-dependent.cpp
@@ -43,7 +43,7 @@ void neg() {
 
 template <long int I>
 void tooBig() {
-  __attribute__((address_space(I))) int *bounds; // expected-error {{address space is larger than the maximum supported (8388587)}}
+  __attribute__((address_space(I))) int *bounds; // expected-error {{address space is larger than the maximum supported (8388586)}}
 }
 
 template <long int I>
@@ -101,7 +101,7 @@ int main() {
   car<1, 2, 3>(); // expected-note {{in instantiation of function template specialization 'car<1, 2, 3>' requested here}}
   HasASTemplateFields<1> HASTF;
   neg<-1>(); // expected-note {{in instantiation of function template specialization 'neg<-1>' requested here}}
-  correct<0x7FFFEA>();
+  correct<0x7FFFE9>();
   tooBig<8388650>(); // expected-note {{in instantiation of function template specialization 'tooBig<8388650L>' requested here}}
 
   __attribute__((address_space(1))) char *x;
diff --git a/clang/test/SemaTemplate/concepts-out-of-line-def.cpp b/clang/test/SemaTemplate/concepts-out-of-line-def.cpp
new file mode 100644
index 0000000000000..222b78e0d22f7
--- /dev/null
+++ b/clang/test/SemaTemplate/concepts-out-of-line-def.cpp
@@ -0,0 +1,129 @@
+// RUN: %clang_cc1 -std=c++20 -verify %s
+// expected-no-diagnostics
+
+static constexpr int PRIMARY = 0;
+static constexpr int SPECIALIZATION_CONCEPT = 1;
+static constexpr int SPECIALIZATION_REQUIRES = 2;
+
+template <class T>
+concept Concept = (sizeof(T) >= 2 * sizeof(int));
+
+struct XY {
+  int x;
+  int y;
+};
+
+namespace members {
+
+template <class T, class U> struct S {
+  static constexpr int primary();
+};
+
+template <class T, class U> constexpr int S<T, U>::primary() {
+  return PRIMARY;
+};
+
+template <Concept C, class U> struct S<C, U> {
+  static constexpr int specialization();
+};
+
+template <class T, class U>
+  requires(sizeof(T) == sizeof(int))
+struct S<T, U> {
+  static constexpr int specialization();
+};
+
+template <Concept C, class U> constexpr int S<C, U>::specialization() {
+  return SPECIALIZATION_CONCEPT;
+}
+
+template <class T, class U>
+  requires(sizeof(T) == sizeof(int))
+constexpr int S<T, U>::specialization() {
+  return SPECIALIZATION_REQUIRES;
+}
+
+static_assert(S<char, double>::primary() == PRIMARY);
+static_assert(S<XY, double>::specialization() == SPECIALIZATION_CONCEPT);
+static_assert(S<int, double>::specialization() == SPECIALIZATION_REQUIRES);
+
+} // namespace members
+
+namespace enumerations {
+
+template <class T, class U> struct S {
+  enum class E : int;
+};
+
+template <class T, class U> enum class S<T, U>::E { Value = PRIMARY };
+
+template <Concept C, class U> struct S<C, U> {
+  enum class E : int;
+};
+
+template <Concept C, class U>
+enum class S<C, U>::E {
+  Value = SPECIALIZATION_CONCEPT
+};
+
+template <class T, class U>
+  requires(sizeof(T) == sizeof(int))
+struct S<T, U> {
+  enum class E : int;
+};
+
+template <class T, class U>
+  requires(sizeof(T) == sizeof(int))
+enum class S<T, U>::E {
+  Value = SPECIALIZATION_REQUIRES
+};
+
+static_assert(static_cast<int>(S<char, double>::E::Value) == PRIMARY);
+static_assert(static_cast<int>(S<XY, double>::E::Value) ==
+              SPECIALIZATION_CONCEPT);
+static_assert(static_cast<int>(S<int, double>::E::Value) ==
+              SPECIALIZATION_REQUIRES);
+
+} // namespace  enumerations
+
+namespace multiple_template_parameter_lists {
+
+template <class Outer>
+struct S {
+  template <class Inner>
+  static constexpr int primary(Inner);
+};
+
+template <class Outer>
+template <class Inner>
+constexpr int S<Outer>::primary(Inner) {
+  return PRIMARY;
+};
+
+template <Concept Outer>
+struct S<Outer> {
+  template <class Inner>
+  static constexpr int specialization(Inner);
+};
+
+template <Concept Outer>
+template <class Inner>
+constexpr int S<Outer>::specialization(Inner) { return SPECIALIZATION_CONCEPT; }
+
+template <class Outer>
+  requires(sizeof(Outer) == sizeof(int))
+struct S<Outer> {
+  template <class Inner>
+  static constexpr int specialization(Inner);
+};
+
+template <class Outer>
+  requires(sizeof(Outer) == sizeof(int))
+template <class Inner>
+constexpr int S<Outer>::specialization(Inner) { return SPECIALIZATION_REQUIRES; }
+
+static_assert(S<char>::primary("str") == PRIMARY);
+static_assert(S<XY>::specialization("str") == SPECIALIZATION_CONCEPT);
+static_assert(S<int>::specialization("str") == SPECIALIZATION_REQUIRES);
+
+} // namespace multiple_template_parameter_lists
diff --git a/clang/test/SemaTemplate/concepts.cpp b/clang/test/SemaTemplate/concepts.cpp
index df4db9b8d4457..bf5896a14991b 100644
--- a/clang/test/SemaTemplate/concepts.cpp
+++ b/clang/test/SemaTemplate/concepts.cpp
@@ -55,10 +55,15 @@ namespace PR45699 {
 }
 
 namespace P0857R0 {
+  template <typename T> static constexpr bool V = true;
+
   void f() {
     auto x = []<bool B> requires B {}; // expected-note {{constraints not satisfied}} expected-note {{false}}
     x.operator()<true>();
     x.operator()<false>(); // expected-error {{no matching member function}}
+
+    auto y = []<typename T> requires V<T> () {};
+    y.operator()<int>(); // OK
   }
 
   template<typename T> concept C = true;
diff --git a/clang/test/SemaTemplate/cxx1z-fold-expressions.cpp b/clang/test/SemaTemplate/cxx1z-fold-expressions.cpp
index 98876baf1f3ec..518eaf0e05239 100644
--- a/clang/test/SemaTemplate/cxx1z-fold-expressions.cpp
+++ b/clang/test/SemaTemplate/cxx1z-fold-expressions.cpp
@@ -97,7 +97,7 @@ namespace PR41845 {
   template <int I> struct Constant {};
 
   template <int... Is> struct Sum {
-    template <int... Js> using type = Constant<((Is + Js) + ... + 0)>; // expected-error {{pack expansion contains parameter packs 'Is' and 'Js' that have different lengths (1 vs. 2)}}
+    template <int... Js> using type = Constant<((Is + Js) + ... + 0)>; // expected-error {{pack expansion contains parameter pack 'Js' that has a different length (1 vs. 2) from outer parameter packs}}
   };
 
   Sum<1>::type<1, 2> x; // expected-note {{instantiation of}}
diff --git a/clang/test/SemaTemplate/pack-deduction.cpp b/clang/test/SemaTemplate/pack-deduction.cpp
index f07bafdf86bca..e42709820e9cf 100644
--- a/clang/test/SemaTemplate/pack-deduction.cpp
+++ b/clang/test/SemaTemplate/pack-deduction.cpp
@@ -134,14 +134,14 @@ namespace partial_full_mix {
   template<typename ...T> struct tuple {};
   template<typename ...T> struct A {
     template<typename ...U> static pair<tuple<T...>, tuple<U...>> f(pair<T, U> ...p);
-    // expected-note@-1 {{[with U = <char, double, long>]: pack expansion contains parameter packs 'T' and 'U' that have different lengths (2 vs. 3)}}
+    // expected-note@-1 {{[with U = <char, double, long>]: pack expansion contains parameter pack 'U' that has a different length (2 vs. 3) from outer parameter packs}}
     // expected-note@-2 {{[with U = <char, double, void>]: pack expansion contains parameter pack 'U' that has a different length (at least 3 vs. 2) from outer parameter packs}}
 
     template<typename ...U> static pair<tuple<T...>, tuple<U...>> g(pair<T, U> ...p, ...);
-    // expected-note@-1 {{[with U = <char, double, long>]: pack expansion contains parameter packs 'T' and 'U' that have different lengths (2 vs. 3)}}
+    // expected-note@-1 {{[with U = <char, double, long>]: pack expansion contains parameter pack 'U' that has a different length (2 vs. 3) from outer parameter packs}}
 
     template<typename ...U> static tuple<U...> h(tuple<pair<T, U>..., pair<int, int>>);
-    // expected-note@-1 {{[with U = <int[2]>]: pack expansion contains parameter packs 'T' and 'U' that have different lengths (2 vs. 1)}}
+    // expected-note@-1 {{[with U = <int[2]>]: pack expansion contains parameter pack 'U' that has a different length (2 vs. 1) from outer parameter packs}}
   };
 
   pair<tuple<int, float>, tuple<char, double>> k1 = A<int, float>().f<char>(pair<int, char>(), pair<float, double>());
diff --git a/clang/test/lit.cfg.py b/clang/test/lit.cfg.py
index 839960e9633df..e9bfaf2e96774 100644
--- a/clang/test/lit.cfg.py
+++ b/clang/test/lit.cfg.py
@@ -282,6 +282,7 @@ def exclude_unsupported_files_for_aix(dirname):
 
 if 'system-aix' in config.available_features:
         config.substitutions.append(('llvm-nm', 'env OBJECT_MODE=any llvm-nm'))
+        config.substitutions.append(('llvm-ar', 'env OBJECT_MODE=any llvm-ar'))
 
 # It is not realistically possible to account for all options that could
 # possibly be present in system and user configuration files, so disable
diff --git a/clang/test/utils/update_cc_test_checks/Inputs/mangled_names.c.funcsig.v2.expected b/clang/test/utils/update_cc_test_checks/Inputs/mangled_names.c.funcsig.v2.expected
index c70179fc21777..636f7af14d060 100644
--- a/clang/test/utils/update_cc_test_checks/Inputs/mangled_names.c.funcsig.v2.expected
+++ b/clang/test/utils/update_cc_test_checks/Inputs/mangled_names.c.funcsig.v2.expected
@@ -1,4 +1,4 @@
-// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --function-signature --version 2
+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --version 2
 // Example input for update_cc_test_checks
 // RUN: %clang_cc1 -triple=x86_64-unknown-linux-gnu -emit-llvm -o - %s | FileCheck %s
 
diff --git a/clang/test/utils/update_cc_test_checks/Inputs/mangled_names.c.v2.expected b/clang/test/utils/update_cc_test_checks/Inputs/mangled_names.c.v2.expected
new file mode 100644
index 0000000000000..636f7af14d060
--- /dev/null
+++ b/clang/test/utils/update_cc_test_checks/Inputs/mangled_names.c.v2.expected
@@ -0,0 +1,43 @@
+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --version 2
+// Example input for update_cc_test_checks
+// RUN: %clang_cc1 -triple=x86_64-unknown-linux-gnu -emit-llvm -o - %s | FileCheck %s
+
+// CHECK-LABEL: define dso_local i64 @test
+// CHECK-SAME: (i64 noundef [[A:%.*]], i32 noundef [[B:%.*]]) #[[ATTR0:[0-9]+]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    store i64 [[A]], ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    store i32 [[B]], ptr [[B_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load i64, ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[B_ADDR]], align 4
+// CHECK-NEXT:    [[CONV:%.*]] = sext i32 [[TMP1]] to i64
+// CHECK-NEXT:    [[ADD:%.*]] = add nsw i64 [[TMP0]], [[CONV]]
+// CHECK-NEXT:    ret i64 [[ADD]]
+//
+long test(long a, int b) {
+  return a + b;
+}
+
+// A function with a mangled name
+// CHECK-LABEL: define dso_local i64 @_Z4testlii
+// CHECK-SAME: (i64 noundef [[A:%.*]], i32 noundef [[B:%.*]], i32 noundef [[C:%.*]]) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[C_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    store i64 [[A]], ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    store i32 [[B]], ptr [[B_ADDR]], align 4
+// CHECK-NEXT:    store i32 [[C]], ptr [[C_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load i64, ptr [[A_ADDR]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[B_ADDR]], align 4
+// CHECK-NEXT:    [[CONV:%.*]] = sext i32 [[TMP1]] to i64
+// CHECK-NEXT:    [[ADD:%.*]] = add nsw i64 [[TMP0]], [[CONV]]
+// CHECK-NEXT:    [[TMP2:%.*]] = load i32, ptr [[C_ADDR]], align 4
+// CHECK-NEXT:    [[CONV1:%.*]] = sext i32 [[TMP2]] to i64
+// CHECK-NEXT:    [[ADD2:%.*]] = add nsw i64 [[ADD]], [[CONV1]]
+// CHECK-NEXT:    ret i64 [[ADD2]]
+//
+__attribute__((overloadable)) long test(long a, int b, int c) {
+  return a + b + c;
+}
diff --git a/clang/test/utils/update_cc_test_checks/mangled_names.test b/clang/test/utils/update_cc_test_checks/mangled_names.test
index 40f8a5d78b521..f0d65f48dfadf 100644
--- a/clang/test/utils/update_cc_test_checks/mangled_names.test
+++ b/clang/test/utils/update_cc_test_checks/mangled_names.test
@@ -19,3 +19,8 @@
 ## Also try --version 2
 # RUN: %update_cc_test_checks %t.c --function-signature --version 2
 # RUN: diff -u %t.c %S/Inputs/mangled_names.c.funcsig.v2.expected
+## Restore the original file without --function-signature and check that
+## --version 2 will implicitly enable it.
+# RUN: cp -f %S/Inputs/mangled_names.c %t.c
+# RUN: %update_cc_test_checks %t.c --version 2
+# RUN: diff -u %t.c %S/Inputs/mangled_names.c.v2.expected
diff --git a/clang/tools/amdgpu-arch/AMDGPUArch.cpp b/clang/tools/amdgpu-arch/AMDGPUArch.cpp
index fbb084a2a1231..b63ff5cfe1759 100644
--- a/clang/tools/amdgpu-arch/AMDGPUArch.cpp
+++ b/clang/tools/amdgpu-arch/AMDGPUArch.cpp
@@ -11,12 +11,25 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "clang/Basic/Version.h"
+#include "llvm/Support/CommandLine.h"
 #include "llvm/Support/DynamicLibrary.h"
 #include "llvm/Support/Error.h"
 #include <memory>
 #include <string>
 #include <vector>
 
+using namespace llvm;
+
+static cl::opt<bool> Help("h", cl::desc("Alias for -help"), cl::Hidden);
+
+// Mark all our options with this category.
+static cl::OptionCategory AMDGPUArchCategory("amdgpu-arch options");
+
+static void PrintVersion(raw_ostream &OS) {
+  OS << clang::getClangToolFullVersion("amdgpu-arch") << '\n';
+}
+
 #if DYNAMIC_HSA
 typedef enum {
   HSA_STATUS_SUCCESS = 0x0,
@@ -102,6 +115,21 @@ static hsa_status_t iterateAgentsCallback(hsa_agent_t Agent, void *Data) {
 }
 
 int main(int argc, char *argv[]) {
+  cl::HideUnrelatedOptions(AMDGPUArchCategory);
+
+  cl::SetVersionPrinter(PrintVersion);
+  cl::ParseCommandLineOptions(
+      argc, argv,
+      "A tool to detect the presence of AMDGPU devices on the system. \n\n"
+      "The tool will output each detected GPU architecture separated by a\n"
+      "newline character. If multiple GPUs of the same architecture are found\n"
+      "a string will be printed for each\n");
+
+  if (Help) {
+    cl::PrintHelpMessage();
+    return 0;
+  }
+
   // Attempt to load the HSA runtime.
   if (llvm::Error Err = loadHSA()) {
     logAllUnhandledErrors(std::move(Err), llvm::errs());
diff --git a/clang/tools/amdgpu-arch/CMakeLists.txt b/clang/tools/amdgpu-arch/CMakeLists.txt
index 5b7ef13924b8e..4349fb9108a71 100644
--- a/clang/tools/amdgpu-arch/CMakeLists.txt
+++ b/clang/tools/amdgpu-arch/CMakeLists.txt
@@ -14,7 +14,8 @@ add_clang_tool(amdgpu-arch AMDGPUArch.cpp)
 find_package(hsa-runtime64 QUIET 1.2.0 HINTS ${CMAKE_INSTALL_PREFIX} PATHS /opt/rocm)
 if(hsa-runtime64_FOUND AND NOT (LLVM_BUILD_32_BITS OR CMAKE_SIZEOF_VOID_P EQUAL 4))
   set_target_properties(amdgpu-arch PROPERTIES INSTALL_RPATH_USE_LINK_PATH ON)
-  target_link_libraries(amdgpu-arch PRIVATE hsa-runtime64::hsa-runtime64)
+  target_link_libraries(amdgpu-arch PRIVATE hsa-runtime64::hsa-runtime64 clangBasic)
 else()
   target_compile_definitions(amdgpu-arch PRIVATE "DYNAMIC_HSA")
+  target_link_libraries(amdgpu-arch PRIVATE clangBasic)
 endif()
diff --git a/clang/tools/c-index-test/c-index-test.c b/clang/tools/c-index-test/c-index-test.c
index 983738dea1291..448435e83027c 100644
--- a/clang/tools/c-index-test/c-index-test.c
+++ b/clang/tools/c-index-test/c-index-test.c
@@ -67,6 +67,27 @@ extern char *basename(const char *);
 extern char *dirname(char *);
 #endif
 
+CXIndex createIndexWithInvocationEmissionPath(int ExcludeDeclarationsFromPCH,
+                                              int DisplayDiagnostics) {
+    CXIndex Idx;
+
+    CXIndexOptions Opts;
+    memset(&Opts, 0, sizeof(Opts));
+    Opts.Size = sizeof(CXIndexOptions);
+    Opts.ExcludeDeclarationsFromPCH = ExcludeDeclarationsFromPCH;
+    Opts.DisplayDiagnostics = DisplayDiagnostics;
+    Opts.InvocationEmissionPath = getenv("CINDEXTEST_INVOCATION_EMISSION_PATH");
+
+    Idx = clang_createIndexWithOptions(&Opts);
+    if (!Idx) {
+        fprintf(stderr,
+                "clang_createIndexWithOptions() failed. "
+                "CINDEX_VERSION_MINOR = %d, sizeof(CXIndexOptions) = %u\n",
+                CINDEX_VERSION_MINOR, Opts.Size);
+    }
+    return Idx;
+}
+
 /** Return the default parsing options. */
 static unsigned getDefaultParsingOptions(void) {
   unsigned options = CXTranslationUnit_DetailedPreprocessingRecord;
@@ -2046,18 +2067,17 @@ int perform_test_load_source(int argc, const char **argv,
   int result;
   unsigned Repeats = 0;
   unsigned I;
-  const char *InvocationPath;
 
-  Idx = clang_createIndex(/* excludeDeclsFromPCH */
-                          (!strcmp(filter, "local") ||
-                           !strcmp(filter, "local-display") ||
-                           !strcmp(filter, "local-pretty"))
-                              ? 1
-                              : 0,
-                          /* displayDiagnostics=*/1);
-  InvocationPath = getenv("CINDEXTEST_INVOCATION_EMISSION_PATH");
-  if (InvocationPath)
-    clang_CXIndex_setInvocationEmissionPathOption(Idx, InvocationPath);
+  Idx =
+      createIndexWithInvocationEmissionPath(/* excludeDeclsFromPCH */
+                                            (!strcmp(filter, "local") ||
+                                             !strcmp(filter, "local-display") ||
+                                             !strcmp(filter, "local-pretty"))
+                                                ? 1
+                                                : 0,
+                                            /* displayDiagnostics=*/1);
+  if (!Idx)
+    return -1;
 
   if ((CommentSchemaFile = parse_comments_schema(argc, argv))) {
     argc--;
@@ -2701,7 +2721,6 @@ int perform_code_completion(int argc, const char **argv, int timing_only) {
   CXTranslationUnit TU;
   unsigned I, Repeats = 1;
   unsigned completionOptions = clang_defaultCodeCompleteOptions();
-  const char *InvocationPath;
 
   if (getenv("CINDEXTEST_CODE_COMPLETE_PATTERNS"))
     completionOptions |= CXCodeComplete_IncludeCodePatterns;
@@ -2724,10 +2743,9 @@ int perform_code_completion(int argc, const char **argv, int timing_only) {
   if (parse_remapped_files(argc, argv, 2, &unsaved_files, &num_unsaved_files))
     return -1;
 
-  CIdx = clang_createIndex(0, 0);
-  InvocationPath = getenv("CINDEXTEST_INVOCATION_EMISSION_PATH");
-  if (InvocationPath)
-    clang_CXIndex_setInvocationEmissionPathOption(CIdx, InvocationPath);
+  CIdx = createIndexWithInvocationEmissionPath(0, 0);
+  if (!CIdx)
+    return -1;
 
   if (getenv("CINDEXTEST_EDITING"))
     Repeats = 5;
@@ -4816,17 +4834,15 @@ static int perform_test_single_symbol_sgf(const char *input, int argc,
   int num_unsaved_files = 0;
   enum CXErrorCode Err;
   int result = 0;
-  const char *InvocationPath;
   CXString SGF;
   const char *usr;
 
   usr = input + strlen("-single-symbol-sgf-for=");
 
-  Idx = clang_createIndex(/* excludeDeclsFromPCH */ 1,
-                          /* displayDiagnostics=*/0);
-  InvocationPath = getenv("CINDEXTEST_INVOCATION_EMISSION_PATH");
-  if (InvocationPath)
-    clang_CXIndex_setInvocationEmissionPathOption(Idx, InvocationPath);
+  Idx = createIndexWithInvocationEmissionPath(/* excludeDeclsFromPCH */ 1,
+                                              /* displayDiagnostics=*/0);
+  if (!Idx)
+    return -1;
 
   if (parse_remapped_files(argc, argv, 0, &unsaved_files, &num_unsaved_files)) {
     result = -1;
diff --git a/clang/tools/clang-format/clang-format-diff.py b/clang/tools/clang-format/clang-format-diff.py
index ce971e414e3d4..1dcc8689d5fef 100755
--- a/clang/tools/clang-format/clang-format-diff.py
+++ b/clang/tools/clang-format/clang-format-diff.py
@@ -34,21 +34,6 @@
 else:
     from io import BytesIO as StringIO
 
-def process_subprocess_result(proc, args):
-  stdout, stderr = proc.communicate()
-  if proc.returncode != 0:
-    sys.exit(proc.returncode)
-  if not args.i:
-    with open(filename) as f:
-      code = f.readlines()
-    formatted_code = StringIO(stdout).readlines()
-    diff = difflib.unified_diff(code, formatted_code,
-                                filename, filename,
-                                '(before formatting)',
-                                '(after formatting)')
-    diff_string = ''.join(diff)
-    if len(diff_string) > 0:
-      sys.stdout.write(diff_string)
 
 def main():
   parser = argparse.ArgumentParser(description=__doc__,
@@ -80,9 +65,6 @@ def main():
                       'file to use.')
   parser.add_argument('-binary', default='clang-format',
                       help='location of binary to use for clang-format')
-  parser.add_argument('-j', default=1, type=int, metavar='N',
-                      help='number of concurrent clang-format processes to spawn in '
-                      'parallel')
   args = parser.parse_args()
 
   # Extract changed lines for each file.
@@ -124,54 +106,46 @@ def main():
           ['-lines', str(start_line) + ':' + str(end_line)])
 
   # Reformat files containing changes in place.
-  lbf = list(lines_by_file.items())
-  procs = [None for i in range(args.j)]
-  while lbf:
-    spawned_one = False
-    for i, proc in enumerate(procs):
-      if not lbf:
-        break
-      if proc is not None and proc.poll() is not None:
-        process_subprocess_result(proc, args)
-        # Set to None to flag the slot as free to start a new process
-        procs[i] = None
-        proc = None
-      if proc is None:
-        filename, lines = lbf.pop()
-        spawned_one = True
-        if args.i and args.verbose:
-          print('Formatting {}'.format(filename))
-        command = [args.binary, filename]
-        if args.i:
-          command.append('-i')
-        if args.sort_includes:
-          command.append('-sort-includes')
-        command.extend(lines)
-        if args.style:
-          command.extend(['-style', args.style])
-        if args.fallback_style:
-          command.extend(['-fallback-style', args.fallback_style])
-        try:
-          procs[i] = subprocess.Popen(command,
-                                      stdout=subprocess.PIPE,
-                                      stderr=None,
-                                      stdin=subprocess.PIPE,
-                                      universal_newlines=True)
-        except OSError as e:
-          # Give the user more context when clang-format isn't
-          # found/isn't executable, etc.
-          raise RuntimeError(
-              'Failed to run "%s" - %s"' % (" ".join(command), e.strerror))
-    # If we didn't spawn a single process after iterating through the whole
-    # list, wait on one of them to finish until we iterate through again, to
-    # prevent spinning in the case where we have a small number of jobs.
-    if not spawned_one:
-      procs[0].wait()
-  # Be sure not to leave any stray processes when exiting.
-  for proc in procs:
-    if proc:
-      proc.wait()
-      process_subprocess_result(proc, args)
+  for filename, lines in lines_by_file.items():
+    if args.i and args.verbose:
+      print('Formatting {}'.format(filename))
+    command = [args.binary, filename]
+    if args.i:
+      command.append('-i')
+    if args.sort_includes:
+      command.append('-sort-includes')
+    command.extend(lines)
+    if args.style:
+      command.extend(['-style', args.style])
+    if args.fallback_style:
+      command.extend(['-fallback-style', args.fallback_style])
+
+    try:
+      p = subprocess.Popen(command,
+                           stdout=subprocess.PIPE,
+                           stderr=None,
+                           stdin=subprocess.PIPE,
+                           universal_newlines=True)
+    except OSError as e:
+      # Give the user more context when clang-format isn't
+      # found/isn't executable, etc.
+      raise RuntimeError(
+        'Failed to run "%s" - %s"' % (" ".join(command), e.strerror))
+
+    stdout, stderr = p.communicate()
+    if p.returncode != 0:
+      sys.exit(p.returncode)
+
+    if not args.i:
+      with open(filename) as f:
+        code = f.readlines()
+      formatted_code = StringIO(stdout).readlines()
+      diff = difflib.unified_diff(code, formatted_code,
+                                  filename, filename,
+                                  '(before formatting)', '(after formatting)')
+      diff_string = ''.join(diff)
+      if len(diff_string) > 0:
+        sys.stdout.write(diff_string)
 
 if __name__ == '__main__':
   main()
diff --git a/clang/tools/clang-fuzzer/handle-llvm/handle_llvm.cpp b/clang/tools/clang-fuzzer/handle-llvm/handle_llvm.cpp
index de47cbf4b1ce3..bd3490e802e44 100644
--- a/clang/tools/clang-fuzzer/handle-llvm/handle_llvm.cpp
+++ b/clang/tools/clang-fuzzer/handle-llvm/handle_llvm.cpp
@@ -103,11 +103,7 @@ static void RunOptimizationPasses(raw_ostream &OS, Module &M,
   PB.registerLoopAnalyses(LAM);
   PB.crossRegisterProxies(LAM, FAM, CGAM, MAM);
 
-  ModulePassManager MPM;
-  if (OL == OptimizationLevel::O0)
-    MPM = PB.buildO0DefaultPipeline(OL);
-  else
-    MPM = PB.buildPerModuleDefaultPipeline(OL);
+  ModulePassManager MPM = PB.buildPerModuleDefaultPipeline(OL);
   MPM.addPass(PrintModulePass(OS));
 
   MPM.run(M, MAM);
diff --git a/clang/tools/clang-linker-wrapper/CMakeLists.txt b/clang/tools/clang-linker-wrapper/CMakeLists.txt
index 8ddfcc6c59d35..198dbff8cf83c 100644
--- a/clang/tools/clang-linker-wrapper/CMakeLists.txt
+++ b/clang/tools/clang-linker-wrapper/CMakeLists.txt
@@ -1,5 +1,3 @@
-include(GNUInstallDirs)
-
 set(LLVM_LINK_COMPONENTS
   ${LLVM_TARGETS_TO_BUILD}
   BitWriter
@@ -27,7 +25,7 @@ if(NOT CLANG_BUILT_STANDALONE)
   set(tablegen_deps intrinsics_gen LinkerWrapperOpts)
 endif()
 
-add_clang_executable(clang-linker-wrapper
+add_clang_tool(clang-linker-wrapper
   ClangLinkerWrapper.cpp
   OffloadWrapper.cpp
 
@@ -39,11 +37,7 @@ set(CLANG_LINKER_WRAPPER_LIB_DEPS
   clangBasic
   )
 
-add_dependencies(clang clang-linker-wrapper)
-
 target_link_libraries(clang-linker-wrapper
   PRIVATE
   ${CLANG_LINKER_WRAPPER_LIB_DEPS}
   )
-
-install(TARGETS clang-linker-wrapper RUNTIME DESTINATION "${CMAKE_INSTALL_BINDIR}")
diff --git a/clang/tools/clang-offload-bundler/CMakeLists.txt b/clang/tools/clang-offload-bundler/CMakeLists.txt
index 7bc22f1479d72..dabd82382cdf0 100644
--- a/clang/tools/clang-offload-bundler/CMakeLists.txt
+++ b/clang/tools/clang-offload-bundler/CMakeLists.txt
@@ -16,8 +16,6 @@ set(CLANG_OFFLOAD_BUNDLER_LIB_DEPS
   clangDriver
   )
 
-add_dependencies(clang clang-offload-bundler)
-
 clang_target_link_libraries(clang-offload-bundler
   PRIVATE
   ${CLANG_OFFLOAD_BUNDLER_LIB_DEPS}
diff --git a/clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp b/clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp
index f8fbcf6f959cf..f20157622d79f 100644
--- a/clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp
+++ b/clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp
@@ -313,6 +313,8 @@ int main(int argc, const char **argv) {
   llvm::DenseSet<StringRef> ParsedTargets;
   // Map {offload-kind}-{triple} to target IDs.
   std::map<std::string, std::set<StringRef>> TargetIDs;
+  // Standardize target names to include env field
+  std::vector<std::string> StandardizedTargetNames;
   for (StringRef Target : TargetNames) {
     if (ParsedTargets.contains(Target)) {
       reportError(createStringError(errc::invalid_argument,
@@ -324,6 +326,8 @@ int main(int argc, const char **argv) {
     bool KindIsValid = OffloadInfo.isOffloadKindValid();
     bool TripleIsValid = OffloadInfo.isTripleValid();
 
+    StandardizedTargetNames.push_back(OffloadInfo.str());
+
     if (!KindIsValid || !TripleIsValid) {
       SmallVector<char, 128u> Buf;
       raw_svector_ostream Msg(Buf);
@@ -348,6 +352,9 @@ int main(int argc, const char **argv) {
 
     ++Index;
   }
+
+  BundlerConfig.TargetNames = StandardizedTargetNames;
+
   for (const auto &TargetID : TargetIDs) {
     if (auto ConflictingTID =
             clang::getConflictTargetIDCombination(TargetID.second)) {
diff --git a/clang/tools/clang-offload-packager/CMakeLists.txt b/clang/tools/clang-offload-packager/CMakeLists.txt
index accc9486f46ab..1c29e376452c1 100644
--- a/clang/tools/clang-offload-packager/CMakeLists.txt
+++ b/clang/tools/clang-offload-packager/CMakeLists.txt
@@ -11,8 +11,6 @@ add_clang_tool(clang-offload-packager
   ${tablegen_deps}
   )
 
-add_dependencies(clang clang-offload-packager)
-
 clang_target_link_libraries(clang-offload-packager
   PRIVATE
   clangBasic
diff --git a/clang/tools/clang-refactor/TestSupport.cpp b/clang/tools/clang-refactor/TestSupport.cpp
index 400313eeab5e5..3fae18c2109a6 100644
--- a/clang/tools/clang-refactor/TestSupport.cpp
+++ b/clang/tools/clang-refactor/TestSupport.cpp
@@ -176,11 +176,11 @@ std::pair<unsigned, unsigned> getLineColumn(StringRef Filename,
 
 bool TestRefactoringResultConsumer::handleAllResults() {
   bool Failed = false;
-  for (auto &Group : llvm::enumerate(Results)) {
+  for (const auto &Group : llvm::enumerate(Results)) {
     // All ranges in the group must produce the same result.
     std::optional<tooling::AtomicChanges> CanonicalResult;
     std::optional<std::string> CanonicalErrorMessage;
-    for (auto &I : llvm::enumerate(Group.value())) {
+    for (const auto &I : llvm::enumerate(Group.value())) {
       Expected<tooling::AtomicChanges> &Result = I.value();
       std::string ErrorMessage;
       bool HasResult = !!Result;
diff --git a/clang/tools/driver/cc1as_main.cpp b/clang/tools/driver/cc1as_main.cpp
index 43d427b1af9b4..ee54a0f7b0eab 100644
--- a/clang/tools/driver/cc1as_main.cpp
+++ b/clang/tools/driver/cc1as_main.cpp
@@ -384,8 +384,8 @@ static bool ExecuteAssemblerImpl(AssemblerInvocation &Opts,
       MemoryBuffer::getFileOrSTDIN(Opts.InputFile, /*IsText=*/true);
 
   if (std::error_code EC = Buffer.getError()) {
-    Error = EC.message();
-    return Diags.Report(diag::err_fe_error_reading) << Opts.InputFile;
+    return Diags.Report(diag::err_fe_error_reading)
+           << Opts.InputFile << EC.message();
   }
 
   SourceMgr SrcMgr;
diff --git a/clang/tools/driver/driver.cpp b/clang/tools/driver/driver.cpp
index 3ec2bcc0bd753..471d0181ff088 100644
--- a/clang/tools/driver/driver.cpp
+++ b/clang/tools/driver/driver.cpp
@@ -307,6 +307,9 @@ static bool SetBackdoorDriverOutputsFromEnvVars(Driver &TheDriver) {
   TheDriver.CCPrintProcessStats =
       checkEnvVar<bool>("CC_PRINT_PROC_STAT", "CC_PRINT_PROC_STAT_FILE",
                         TheDriver.CCPrintStatReportFilename);
+  TheDriver.CCPrintInternalStats =
+      checkEnvVar<bool>("CC_PRINT_INTERNAL_STAT", "CC_PRINT_INTERNAL_STAT_FILE",
+                        TheDriver.CCPrintInternalStatReportFilename);
 
   return true;
 }
@@ -391,55 +394,17 @@ int clang_main(int Argc, char **Argv, const llvm::ToolContext &ToolContext) {
   const char *ProgName =
       ToolContext.NeedsPrependArg ? ToolContext.PrependArg : ToolContext.Path;
 
-  // Parse response files using the GNU syntax, unless we're in CL mode. There
-  // are two ways to put clang in CL compatibility mode: ProgName is either
-  // clang-cl or cl, or --driver-mode=cl is on the command line. The normal
-  // command line parsing can't happen until after response file parsing, so we
-  // have to manually search for a --driver-mode=cl argument the hard way.
-  // Finally, our -cc1 tools don't care which tokenization mode we use because
-  // response files written by clang will tokenize the same way in either mode.
   bool ClangCLMode =
       IsClangCL(getDriverMode(ProgName, llvm::ArrayRef(Args).slice(1)));
-  enum { Default, POSIX, Windows } RSPQuoting = Default;
-  for (const char *F : Args) {
-    if (strcmp(F, "--rsp-quoting=posix") == 0)
-      RSPQuoting = POSIX;
-    else if (strcmp(F, "--rsp-quoting=windows") == 0)
-      RSPQuoting = Windows;
-  }
 
-  // Determines whether we want nullptr markers in Args to indicate response
-  // files end-of-lines. We only use this for the /LINK driver argument with
-  // clang-cl.exe on Windows.
-  bool MarkEOLs = ClangCLMode;
-
-  llvm::cl::TokenizerCallback Tokenizer;
-  if (RSPQuoting == Windows || (RSPQuoting == Default && ClangCLMode))
-    Tokenizer = &llvm::cl::TokenizeWindowsCommandLine;
-  else
-    Tokenizer = &llvm::cl::TokenizeGNUCommandLine;
-
-  if (MarkEOLs && Args.size() > 1 && StringRef(Args[1]).startswith("-cc1"))
-    MarkEOLs = false;
-  llvm::cl::ExpansionContext ECtx(A, Tokenizer);
-  ECtx.setMarkEOLs(MarkEOLs);
-  if (llvm::Error Err = ECtx.expandResponseFiles(Args)) {
+  if (llvm::Error Err = expandResponseFiles(Args, ClangCLMode, A)) {
     llvm::errs() << toString(std::move(Err)) << '\n';
     return 1;
   }
 
-  // Handle -cc1 integrated tools, even if -cc1 was expanded from a response
-  // file.
-  auto FirstArg = llvm::find_if(llvm::drop_begin(Args),
-                                [](const char *A) { return A != nullptr; });
-  if (FirstArg != Args.end() && StringRef(*FirstArg).startswith("-cc1")) {
-    // If -cc1 came from a response file, remove the EOL sentinels.
-    if (MarkEOLs) {
-      auto newEnd = std::remove(Args.begin(), Args.end(), nullptr);
-      Args.resize(newEnd - Args.begin());
-    }
+  // Handle -cc1 integrated tools.
+  if (Args.size() >= 2 && StringRef(Args[1]).startswith("-cc1"))
     return ExecuteCC1Tool(Args, ToolContext);
-  }
 
   // Handle options that need handling before the real command line parsing in
   // Driver::BuildCompilation()
diff --git a/clang/tools/include-mapping/gen_std.py b/clang/tools/include-mapping/gen_std.py
index 291946ef5378b..944fd17263700 100755
--- a/clang/tools/include-mapping/gen_std.py
+++ b/clang/tools/include-mapping/gen_std.py
@@ -245,12 +245,14 @@ def main():
       # We don't index std literal operators (e.g.
       # std::literals::chrono_literals::operator""d), these symbols can't be
       # accessed by std::<symbol_name>.
-      # FIXME: index std::placeholders symbols, placeholders.html page is
-      # different (which contains one entry for _1, _2, ..., _N), we need special
-      # handling.
+      #
+      # std::placeholders symbols are handled manually in StdSpecialSymbolMap.inc
       (symbol_index_root, "chrono.html", "std::chrono::"),
+      (symbol_index_root, "execution.html", "std::execution::"),
+      (symbol_index_root, "numbers.html", "std::numbers::"),
       (symbol_index_root, "filesystem.html", "std::filesystem::"),
       (symbol_index_root, "pmr.html", "std::pmr::"),
+      (symbol_index_root, "ranges.html", "std::ranges::"),
       (symbol_index_root, "regex_constants.html", "std::regex_constants::"),
       (symbol_index_root, "this_thread.html", "std::this_thread::"),
       # Zombie symbols that were available from the Standard Library, but are
diff --git a/clang/tools/libclang/CIndex.cpp b/clang/tools/libclang/CIndex.cpp
index 46c9cfae202e5..30416e46ce173 100644
--- a/clang/tools/libclang/CIndex.cpp
+++ b/clang/tools/libclang/CIndex.cpp
@@ -3648,8 +3648,10 @@ struct RegisterFatalErrorHandler {
 static llvm::ManagedStatic<RegisterFatalErrorHandler>
     RegisterFatalErrorHandlerOnce;
 
-CXIndex clang_createIndex(int excludeDeclarationsFromPCH,
-                          int displayDiagnostics) {
+static CIndexer *clang_createIndex_Impl(
+    int excludeDeclarationsFromPCH, int displayDiagnostics,
+    unsigned char threadBackgroundPriorityForIndexing = CXChoice_Default,
+    unsigned char threadBackgroundPriorityForEditing = CXChoice_Default) {
   // We use crash recovery to make some of our APIs more reliable, implicitly
   // enable it.
   if (!getenv("LIBCLANG_DISABLE_CRASH_RECOVERY"))
@@ -3673,21 +3675,79 @@ CXIndex clang_createIndex(int excludeDeclarationsFromPCH,
   if (displayDiagnostics)
     CIdxr->setDisplayDiagnostics();
 
-  if (getenv("LIBCLANG_BGPRIO_INDEX"))
-    CIdxr->setCXGlobalOptFlags(CIdxr->getCXGlobalOptFlags() |
-                               CXGlobalOpt_ThreadBackgroundPriorityForIndexing);
-  if (getenv("LIBCLANG_BGPRIO_EDIT"))
-    CIdxr->setCXGlobalOptFlags(CIdxr->getCXGlobalOptFlags() |
-                               CXGlobalOpt_ThreadBackgroundPriorityForEditing);
+  unsigned GlobalOptions = CIdxr->getCXGlobalOptFlags();
+  const auto updateGlobalOption =
+      [&GlobalOptions](unsigned char Policy, CXGlobalOptFlags Flag,
+                       const char *EnvironmentVariableName) {
+        switch (Policy) {
+        case CXChoice_Enabled:
+          GlobalOptions |= Flag;
+          break;
+        case CXChoice_Disabled:
+          GlobalOptions &= ~Flag;
+          break;
+        case CXChoice_Default:
+        default: // Fall back to default behavior if Policy is unsupported.
+          if (getenv(EnvironmentVariableName))
+            GlobalOptions |= Flag;
+        }
+      };
+  updateGlobalOption(threadBackgroundPriorityForIndexing,
+                     CXGlobalOpt_ThreadBackgroundPriorityForIndexing,
+                     "LIBCLANG_BGPRIO_INDEX");
+  updateGlobalOption(threadBackgroundPriorityForEditing,
+                     CXGlobalOpt_ThreadBackgroundPriorityForEditing,
+                     "LIBCLANG_BGPRIO_EDIT");
+  CIdxr->setCXGlobalOptFlags(GlobalOptions);
 
   return CIdxr;
 }
 
+CXIndex clang_createIndex(int excludeDeclarationsFromPCH,
+                          int displayDiagnostics) {
+  return clang_createIndex_Impl(excludeDeclarationsFromPCH, displayDiagnostics);
+}
+
 void clang_disposeIndex(CXIndex CIdx) {
   if (CIdx)
     delete static_cast<CIndexer *>(CIdx);
 }
 
+CXIndex clang_createIndexWithOptions(const CXIndexOptions *options) {
+  // Adding new options to struct CXIndexOptions:
+  // 1. If no other new option has been added in the same libclang version,
+  // sizeof(CXIndexOptions) must increase for versioning purposes.
+  // 2. Options should be added at the end of the struct in order to seamlessly
+  // support older struct versions. If options->Size < sizeof(CXIndexOptions),
+  // don't attempt to read the missing options and rely on the default values of
+  // recently added options being reasonable. For example:
+  // if (options->Size >= offsetof(CXIndexOptions, RecentlyAddedMember))
+  //   do_something(options->RecentlyAddedMember);
+
+  // An exception: if a new option is small enough, it can be squeezed into the
+  // /*Reserved*/ bits in CXIndexOptions. Since the default value of each option
+  // is guaranteed to be 0 and the callers are advised to zero out the struct,
+  // programs built against older libclang versions would implicitly set the new
+  // options to default values, which should keep the behavior of previous
+  // libclang versions and thus be backward-compatible.
+
+  // If options->Size > sizeof(CXIndexOptions), the user may have set an option
+  // we can't handle, in which case we return nullptr to report failure.
+  // Replace `!=` with `>` here to support older struct versions. `!=` has the
+  // advantage of catching more usage bugs and no disadvantages while there is a
+  // single supported struct version (the initial version).
+  if (options->Size != sizeof(CXIndexOptions))
+    return nullptr;
+  CIndexer *const CIdxr = clang_createIndex_Impl(
+      options->ExcludeDeclarationsFromPCH, options->DisplayDiagnostics,
+      options->ThreadBackgroundPriorityForIndexing,
+      options->ThreadBackgroundPriorityForEditing);
+  CIdxr->setStorePreamblesInMemory(options->StorePreamblesInMemory);
+  CIdxr->setPreambleStoragePath(options->PreambleStoragePath);
+  CIdxr->setInvocationEmissionPath(options->InvocationEmissionPath);
+  return CIdxr;
+}
+
 void clang_CXIndex_setGlobalOptions(CXIndex CIdx, unsigned options) {
   if (CIdx)
     static_cast<CIndexer *>(CIdx)->setCXGlobalOptFlags(options);
@@ -3897,7 +3957,8 @@ clang_parseTranslationUnit_Impl(CXIndex CIdx, const char *source_filename,
   std::unique_ptr<ASTUnit> Unit(ASTUnit::LoadFromCommandLine(
       Args->data(), Args->data() + Args->size(),
       CXXIdx->getPCHContainerOperations(), Diags,
-      CXXIdx->getClangResourcesPath(), CXXIdx->getOnlyLocalDecls(),
+      CXXIdx->getClangResourcesPath(), CXXIdx->getStorePreamblesInMemory(),
+      CXXIdx->getPreambleStoragePath(), CXXIdx->getOnlyLocalDecls(),
       CaptureDiagnostics, *RemappedFiles.get(),
       /*RemappedFilesKeepOriginalName=*/true, PrecompilePreambleAfterNParses,
       TUKind, CacheCodeCompletionResults, IncludeBriefCommentsInCodeCompletion,
diff --git a/clang/tools/libclang/CIndexer.h b/clang/tools/libclang/CIndexer.h
index de15cc34b5e51..83268a2016c8f 100644
--- a/clang/tools/libclang/CIndexer.h
+++ b/clang/tools/libclang/CIndexer.h
@@ -34,6 +34,7 @@ class IdentifierInfo;
 class CIndexer {
   bool OnlyLocalDecls;
   bool DisplayDiagnostics;
+  bool StorePreamblesInMemory = false;
   unsigned Options; // CXGlobalOptFlags.
 
   std::string ResourcesPath;
@@ -41,6 +42,7 @@ class CIndexer {
 
   std::string ToolchainPath;
 
+  std::string PreambleStoragePath;
   std::string InvocationEmissionPath;
 
 public:
@@ -77,6 +79,17 @@ class CIndexer {
 
   StringRef getClangToolchainPath();
 
+  void setStorePreamblesInMemory(bool StoreInMemory) {
+    StorePreamblesInMemory = StoreInMemory;
+  }
+  bool getStorePreamblesInMemory() const { return StorePreamblesInMemory; }
+
+  void setPreambleStoragePath(StringRef Str) {
+    PreambleStoragePath = Str.str();
+  }
+
+  StringRef getPreambleStoragePath() const { return PreambleStoragePath; }
+
   void setInvocationEmissionPath(StringRef Str) {
     InvocationEmissionPath = std::string(Str);
   }
diff --git a/clang/tools/libclang/CXType.cpp b/clang/tools/libclang/CXType.cpp
index a1d157c63995b..eb8bfc25a7c91 100644
--- a/clang/tools/libclang/CXType.cpp
+++ b/clang/tools/libclang/CXType.cpp
@@ -10,11 +10,11 @@
 //
 //===--------------------------------------------------------------------===//
 
+#include "CXType.h"
 #include "CIndexer.h"
 #include "CXCursor.h"
 #include "CXString.h"
 #include "CXTranslationUnit.h"
-#include "CXType.h"
 #include "clang/AST/Decl.h"
 #include "clang/AST/DeclObjC.h"
 #include "clang/AST/DeclTemplate.h"
@@ -378,7 +378,7 @@ int clang_getFieldDeclBitWidth(CXCursor C) {
     const Decl *D = getCursorDecl(C);
 
     if (const FieldDecl *FD = dyn_cast_or_null<FieldDecl>(D)) {
-      if (FD->isBitField())
+      if (FD->isBitField() && !FD->getBitWidth()->isValueDependent())
         return FD->getBitWidthValue(getCursorContext(C));
     }
   }
diff --git a/clang/tools/libclang/libclang.map b/clang/tools/libclang/libclang.map
index 80b244e0fa907..34b1ef1a54514 100644
--- a/clang/tools/libclang/libclang.map
+++ b/clang/tools/libclang/libclang.map
@@ -421,6 +421,7 @@ LLVM_16 {
 LLVM_17 {
   global:
     clang_CXXMethod_isExplicit;
+    clang_createIndexWithOptions;
 };
 
 # Example of how to add a new symbol version entry.  If you do add a new symbol
diff --git a/clang/tools/nvptx-arch/CMakeLists.txt b/clang/tools/nvptx-arch/CMakeLists.txt
index 1854d34ca075f..3444cf5b70dfd 100644
--- a/clang/tools/nvptx-arch/CMakeLists.txt
+++ b/clang/tools/nvptx-arch/CMakeLists.txt
@@ -13,7 +13,8 @@ find_package(CUDAToolkit QUIET)
 
 # If we found the CUDA library directly we just dynamically link against it.
 if(CUDAToolkit_FOUND AND NOT (LLVM_BUILD_32_BITS OR CMAKE_SIZEOF_VOID_P EQUAL 4))
-  target_link_libraries(nvptx-arch PRIVATE CUDA::cuda_driver)
+  target_link_libraries(nvptx-arch PRIVATE CUDA::cuda_driver clangBasic)
 else()
   target_compile_definitions(nvptx-arch PRIVATE "DYNAMIC_CUDA")
+  target_link_libraries(nvptx-arch PRIVATE clangBasic)
 endif()
diff --git a/clang/tools/nvptx-arch/NVPTXArch.cpp b/clang/tools/nvptx-arch/NVPTXArch.cpp
index 91723324c28e2..4cfc58681cba3 100644
--- a/clang/tools/nvptx-arch/NVPTXArch.cpp
+++ b/clang/tools/nvptx-arch/NVPTXArch.cpp
@@ -11,12 +11,25 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "clang/Basic/Version.h"
+#include "llvm/Support/CommandLine.h"
 #include "llvm/Support/DynamicLibrary.h"
 #include "llvm/Support/Error.h"
 #include <cstdint>
 #include <cstdio>
 #include <memory>
 
+using namespace llvm;
+
+static cl::opt<bool> Help("h", cl::desc("Alias for -help"), cl::Hidden);
+
+static void PrintVersion(raw_ostream &OS) {
+  OS << clang::getClangToolFullVersion("nvptx-arch") << '\n';
+}
+// Mark all our options with this category, everything else (except for -version
+// and -help) will be hidden.
+static cl::OptionCategory NVPTXArchCategory("nvptx-arch options");
+
 #if DYNAMIC_CUDA
 typedef enum cudaError_enum {
   CUDA_SUCCESS = 0,
@@ -79,6 +92,21 @@ static int handleError(CUresult Err) {
 }
 
 int main(int argc, char *argv[]) {
+  cl::HideUnrelatedOptions(NVPTXArchCategory);
+
+  cl::SetVersionPrinter(PrintVersion);
+  cl::ParseCommandLineOptions(
+      argc, argv,
+      "A tool to detect the presence of NVIDIA devices on the system. \n\n"
+      "The tool will output each detected GPU architecture separated by a\n"
+      "newline character. If multiple GPUs of the same architecture are found\n"
+      "a string will be printed for each\n");
+
+  if (Help) {
+    cl::PrintHelpMessage();
+    return 0;
+  }
+
   // Attempt to load the NVPTX driver runtime.
   if (llvm::Error Err = loadCUDA()) {
     logAllUnhandledErrors(std::move(Err), llvm::errs());
diff --git a/clang/unittests/AST/ASTImporterTest.cpp b/clang/unittests/AST/ASTImporterTest.cpp
index fecfac4c5914e..6300551ca4469 100644
--- a/clang/unittests/AST/ASTImporterTest.cpp
+++ b/clang/unittests/AST/ASTImporterTest.cpp
@@ -259,6 +259,10 @@ TEST_P(ImportExpr, ImportSourceLocExpr) {
              Lang_CXX03, Verifier,
              functionDecl(hasDescendant(
                  sourceLocExpr(hasBuiltinStr("__builtin_FILE")))));
+  testImport("void declToImport() { (void)__builtin_FILE_NAME(); }", Lang_CXX03,
+             "", Lang_CXX03, Verifier,
+             functionDecl(hasDescendant(
+                 sourceLocExpr(hasBuiltinStr("__builtin_FILE_NAME")))));
   testImport("void declToImport() { (void)__builtin_COLUMN(); }", Lang_CXX03,
              "", Lang_CXX03, Verifier,
              functionDecl(hasDescendant(
diff --git a/clang/unittests/AST/DeclTest.cpp b/clang/unittests/AST/DeclTest.cpp
index 119551bb44453..a6535119afb53 100644
--- a/clang/unittests/AST/DeclTest.cpp
+++ b/clang/unittests/AST/DeclTest.cpp
@@ -232,16 +232,16 @@ TEST(Decl, ModuleAndInternalLinkage) {
   const auto *f = selectFirst<FunctionDecl>(
       "f", match(functionDecl(hasName("f")).bind("f"), Ctx));
 
-  EXPECT_EQ(a->getLinkageInternal(), InternalLinkage);
-  EXPECT_EQ(f->getLinkageInternal(), InternalLinkage);
+  EXPECT_EQ(a->getFormalLinkage(), InternalLinkage);
+  EXPECT_EQ(f->getFormalLinkage(), InternalLinkage);
 
   const auto *b =
       selectFirst<VarDecl>("b", match(varDecl(hasName("b")).bind("b"), Ctx));
   const auto *g = selectFirst<FunctionDecl>(
       "g", match(functionDecl(hasName("g")).bind("g"), Ctx));
 
-  EXPECT_EQ(b->getLinkageInternal(), ModuleLinkage);
-  EXPECT_EQ(g->getLinkageInternal(), ModuleLinkage);
+  EXPECT_EQ(b->getFormalLinkage(), ModuleLinkage);
+  EXPECT_EQ(g->getFormalLinkage(), ModuleLinkage);
 
   AST = tooling::buildASTFromCodeWithArgs(
       Code.code(), /*Args=*/{"-std=c++20"});
@@ -250,15 +250,15 @@ TEST(Decl, ModuleAndInternalLinkage) {
   f = selectFirst<FunctionDecl>(
       "f", match(functionDecl(hasName("f")).bind("f"), CtxTS));
 
-  EXPECT_EQ(a->getLinkageInternal(), InternalLinkage);
-  EXPECT_EQ(f->getLinkageInternal(), InternalLinkage);
+  EXPECT_EQ(a->getFormalLinkage(), InternalLinkage);
+  EXPECT_EQ(f->getFormalLinkage(), InternalLinkage);
 
   b = selectFirst<VarDecl>("b", match(varDecl(hasName("b")).bind("b"), CtxTS));
   g = selectFirst<FunctionDecl>(
       "g", match(functionDecl(hasName("g")).bind("g"), CtxTS));
 
-  EXPECT_EQ(b->getLinkageInternal(), ModuleLinkage);
-  EXPECT_EQ(g->getLinkageInternal(), ModuleLinkage);
+  EXPECT_EQ(b->getFormalLinkage(), ModuleLinkage);
+  EXPECT_EQ(g->getFormalLinkage(), ModuleLinkage);
 }
 
 TEST(Decl, GetNonTransparentDeclContext) {
diff --git a/clang/unittests/AST/SizelessTypesTest.cpp b/clang/unittests/AST/SizelessTypesTest.cpp
index 9ff3a5b09d89d..0b984b6922934 100644
--- a/clang/unittests/AST/SizelessTypesTest.cpp
+++ b/clang/unittests/AST/SizelessTypesTest.cpp
@@ -45,6 +45,7 @@ TEST_F(SizelessTypeTester, TestSizelessBuiltin) {
   ASSERT_TRUE(Ctx.SveBFloat16Ty->isSizelessBuiltinType());
 
   ASSERT_TRUE(Ctx.SveBoolTy->isSizelessBuiltinType());
+  ASSERT_TRUE(Ctx.SveCountTy->isSizelessBuiltinType());
 
   ASSERT_FALSE(Ctx.VoidTy->isSizelessBuiltinType());
   ASSERT_FALSE(Ctx.PseudoObjectTy->isSizelessBuiltinType());
@@ -55,6 +56,12 @@ TEST_F(SizelessTypeTester, TestSizelessBuiltin) {
       Ctx.getLValueReferenceType(Ctx.SveBoolTy)->isSizelessBuiltinType());
   ASSERT_FALSE(
       Ctx.getRValueReferenceType(Ctx.SveBoolTy)->isSizelessBuiltinType());
+
+  ASSERT_FALSE(Ctx.getPointerType(Ctx.SveCountTy)->isSizelessBuiltinType());
+  ASSERT_FALSE(
+      Ctx.getLValueReferenceType(Ctx.SveCountTy)->isSizelessBuiltinType());
+  ASSERT_FALSE(
+      Ctx.getRValueReferenceType(Ctx.SveCountTy)->isSizelessBuiltinType());
 }
 
 TEST_F(SizelessTypeTester, TestSizeless) {
@@ -75,6 +82,7 @@ TEST_F(SizelessTypeTester, TestSizeless) {
   ASSERT_TRUE(Ctx.SveBFloat16Ty->isSizelessType());
 
   ASSERT_TRUE(Ctx.SveBoolTy->isSizelessType());
+  ASSERT_TRUE(Ctx.SveCountTy->isSizelessType());
 
   ASSERT_FALSE(Ctx.VoidTy->isSizelessType());
   ASSERT_FALSE(Ctx.PseudoObjectTy->isSizelessType());
@@ -83,4 +91,8 @@ TEST_F(SizelessTypeTester, TestSizeless) {
   ASSERT_FALSE(Ctx.getPointerType(Ctx.SveBoolTy)->isSizelessType());
   ASSERT_FALSE(Ctx.getLValueReferenceType(Ctx.SveBoolTy)->isSizelessType());
   ASSERT_FALSE(Ctx.getRValueReferenceType(Ctx.SveBoolTy)->isSizelessType());
+
+  ASSERT_FALSE(Ctx.getPointerType(Ctx.SveCountTy)->isSizelessType());
+  ASSERT_FALSE(Ctx.getLValueReferenceType(Ctx.SveCountTy)->isSizelessType());
+  ASSERT_FALSE(Ctx.getRValueReferenceType(Ctx.SveCountTy)->isSizelessType());
 }
diff --git a/clang/unittests/Format/FormatTest.cpp b/clang/unittests/Format/FormatTest.cpp
index f200c498956e4..efd2942ddf27f 100644
--- a/clang/unittests/Format/FormatTest.cpp
+++ b/clang/unittests/Format/FormatTest.cpp
@@ -10604,14 +10604,6 @@ TEST_F(FormatTest, UnderstandsOverloadedOperators) {
   verifyFormat("foo() { ::operator new(n * sizeof(foo)); }");
 }
 
-TEST_F(FormatTest, SpaceBeforeTemplateCloser) {
-  verifyFormat("C<&operator- > minus;");
-  verifyFormat("C<&operator> > gt;");
-  verifyFormat("C<&operator>= > ge;");
-  verifyFormat("C<&operator<= > le;");
-  verifyFormat("C<&operator< <X>> lt;");
-}
-
 TEST_F(FormatTest, UnderstandsFunctionRefQualification) {
   verifyFormat("void A::b() && {}");
   verifyFormat("void A::b() && noexcept {}");
@@ -24148,6 +24140,14 @@ TEST_F(FormatTest, RequiresClausesPositions) {
                "}",
                Style);
 
+  verifyFormat("template <typename T>\n"
+               "int S::bar(T t) &&\n"
+               "  requires F<T>\n"
+               "{\n"
+               "  return 5;\n"
+               "}",
+               Style);
+
   verifyFormat("template <typename T>\n"
                "int bar(T t)\n"
                "  requires F<T>;",
@@ -24161,6 +24161,14 @@ TEST_F(FormatTest, RequiresClausesPositions) {
                "}",
                Style);
 
+  verifyFormat("template <typename T>\n"
+               "int S::bar(T t) &&\n"
+               "requires F<T>\n"
+               "{\n"
+               "  return 5;\n"
+               "}",
+               Style);
+
   verifyFormat("template <typename T>\n"
                "int bar(T t)\n"
                "requires F<T>\n"
@@ -24174,6 +24182,7 @@ TEST_F(FormatTest, RequiresClausesPositions) {
                "template <typename T> requires Foo<T> void bar() {}\n"
                "template <typename T> void bar() requires Foo<T> {}\n"
                "template <typename T> void bar() requires Foo<T>;\n"
+               "template <typename T> void S::bar() && requires Foo<T> {}\n"
                "template <typename T> requires Foo<T> Bar(T) -> Bar<T>;",
                Style);
 
@@ -24185,6 +24194,8 @@ TEST_F(FormatTest, RequiresClausesPositions) {
                "requires Foo<T> void bar() {}\n"
                "template <typename AAAAAAA>\n"
                "void bar() requires Foo<T> {}\n"
+               "template <typename T>\n"
+               "void S::bar() && requires Foo<T> {}\n"
                "template <typename AAAAAAA>\n"
                "requires Foo<T> Baz(T) -> Baz<T>;",
                ColumnStyle);
@@ -24229,6 +24240,9 @@ TEST_F(FormatTest, RequiresClausesPositions) {
                "void bar()\n"
                "requires Foo<T>;\n"
                "template <typename T>\n"
+               "void S::bar() &&\n"
+               "requires Foo<T> {}\n"
+               "template <typename T>\n"
                "requires Foo<T> Bar(T) -> Bar<T>;",
                Style);
 
@@ -24259,6 +24273,9 @@ TEST_F(FormatTest, RequiresClausesPositions) {
                "void bar()\n"
                "  requires Foo<T> {}\n"
                "template <typename T>\n"
+               "void S::bar() &&\n"
+               "  requires Foo<T> {}\n"
+               "template <typename T>\n"
                "  requires Foo<T> Bar(T) -> Bar<T>;",
                Style);
 
@@ -24289,6 +24306,9 @@ TEST_F(FormatTest, RequiresClausesPositions) {
                "void bar() requires Foo<T>\n"
                "{}\n"
                "template <typename T> void bar() requires Foo<T>;\n"
+               "template <typename T>\n"
+               "void S::bar() && requires Foo<T>\n"
+               "{}\n"
                "template <typename T> requires Foo<T>\n"
                "Bar(T) -> Bar<T>;",
                Style);
@@ -25233,6 +25253,11 @@ TEST_F(FormatTest, RemoveSemicolon) {
                "}",
                Style);
 
+  verifyFormat("class [[deprecated(\"\")]] C {\n"
+               "  int i;\n"
+               "};",
+               Style);
+
   verifyIncompleteFormat("class C final [[deprecated(l]] {});", Style);
 
   // These tests are here to show a problem that may not be easily
diff --git a/clang/unittests/Format/FormatTestVerilog.cpp b/clang/unittests/Format/FormatTestVerilog.cpp
index fc23dd8ea1d0a..a4d6b540bd831 100644
--- a/clang/unittests/Format/FormatTestVerilog.cpp
+++ b/clang/unittests/Format/FormatTestVerilog.cpp
@@ -250,6 +250,45 @@ TEST_F(FormatTestVerilog, Case) {
                "  end\n"
                "endcase",
                Style);
+  // Other colons should not be mistaken as case colons.
+  Style = getLLVMStyle(FormatStyle::LK_Verilog);
+  Style.BitFieldColonSpacing = FormatStyle::BFCS_None;
+  verifyFormat("case (x[1:0])\n"
+               "endcase",
+               Style);
+  verifyFormat("default:\n"
+               "  x[1:0] = x[1:0];",
+               Style);
+  Style.BitFieldColonSpacing = FormatStyle::BFCS_Both;
+  verifyFormat("case (x[1 : 0])\n"
+               "endcase",
+               Style);
+  verifyFormat("default:\n"
+               "  x[1 : 0] = x[1 : 0];",
+               Style);
+  Style = getLLVMStyle(FormatStyle::LK_Verilog);
+  Style.SpacesInContainerLiterals = true;
+  verifyFormat("case ('{x : x, default : 9})\n"
+               "endcase",
+               Style);
+  verifyFormat("x = '{x : x, default : 9};\n", Style);
+  verifyFormat("default:\n"
+               "  x = '{x : x, default : 9};\n",
+               Style);
+  Style.SpacesInContainerLiterals = false;
+  verifyFormat("case ('{x: x, default: 9})\n"
+               "endcase",
+               Style);
+  verifyFormat("x = '{x: x, default: 9};\n", Style);
+  verifyFormat("default:\n"
+               "  x = '{x: x, default: 9};\n",
+               Style);
+}
+
+TEST_F(FormatTestVerilog, Coverage) {
+  verifyFormat("covergroup x\n"
+               "    @@(begin x);\n"
+               "endgroup");
 }
 
 TEST_F(FormatTestVerilog, Declaration) {
@@ -809,5 +848,50 @@ TEST_F(FormatTestVerilog, Primitive) {
                "  endtable\n"
                "endprimitive");
 }
+
+TEST_F(FormatTestVerilog, StructuredProcedure) {
+  // Blocks should be indented correctly.
+  verifyFormat("initial begin\n"
+               "end");
+  verifyFormat("initial begin\n"
+               "  x <= x;\n"
+               "  x <= x;\n"
+               "end");
+  verifyFormat("initial\n"
+               "  x <= x;\n"
+               "x <= x;");
+  verifyFormat("always @(x) begin\n"
+               "end");
+  verifyFormat("always @(x) begin\n"
+               "  x <= x;\n"
+               "  x <= x;\n"
+               "end");
+  verifyFormat("always @(x)\n"
+               "  x <= x;\n"
+               "x <= x;");
+  // Various keywords.
+  verifyFormat("always @(x)\n"
+               "  x <= x;");
+  verifyFormat("always @(posedge x)\n"
+               "  x <= x;");
+  verifyFormat("always\n"
+               "  x <= x;");
+  verifyFormat("always @*\n"
+               "  x <= x;");
+  verifyFormat("always @(*)\n"
+               "  x <= x;");
+  verifyFormat("always_comb\n"
+               "  x <= x;");
+  verifyFormat("always_latch @(x)\n"
+               "  x <= x;");
+  verifyFormat("always_ff @(posedge x)\n"
+               "  x <= x;");
+  verifyFormat("initial\n"
+               "  x <= x;");
+  verifyFormat("final\n"
+               "  x <= x;");
+  verifyFormat("forever\n"
+               "  x <= x;");
+}
 } // namespace format
 } // end namespace clang
diff --git a/clang/unittests/Format/TokenAnnotatorTest.cpp b/clang/unittests/Format/TokenAnnotatorTest.cpp
index b138abc3a9476..9f65709924a36 100644
--- a/clang/unittests/Format/TokenAnnotatorTest.cpp
+++ b/clang/unittests/Format/TokenAnnotatorTest.cpp
@@ -345,6 +345,10 @@ TEST_F(TokenAnnotatorTest, UnderstandsClasses) {
   Tokens = annotate("const class {} c;");
   EXPECT_EQ(Tokens.size(), 7u) << Tokens;
   EXPECT_TOKEN(Tokens[2], tok::l_brace, TT_ClassLBrace);
+
+  Tokens = annotate("class [[deprecated(\"\")]] C { int i; };");
+  EXPECT_EQ(Tokens.size(), 17u) << Tokens;
+  EXPECT_TOKEN(Tokens[10], tok::l_brace, TT_ClassLBrace);
 }
 
 TEST_F(TokenAnnotatorTest, UnderstandsStructs) {
@@ -574,71 +578,6 @@ TEST_F(TokenAnnotatorTest, UnderstandsOverloadedOperators) {
   EXPECT_TOKEN(Tokens[4], tok::l_paren, TT_OverloadedOperatorLParen);
 }
 
-TEST_F(TokenAnnotatorTest, OverloadedOperatorInTemplate) {
-  struct {
-    const char *Text;
-    tok::TokenKind Kind;
-  } Operators[] = {{"+", tok::plus},
-                   {"-", tok::minus},
-                   // FIXME:
-                   // {"*", tok::star},
-                   {"/", tok::slash},
-                   {"%", tok::percent},
-                   {"^", tok::caret},
-                   // FIXME:
-                   // {"&", tok::amp},
-                   {"|", tok::pipe},
-                   {"~", tok::tilde},
-                   {"!", tok::exclaim},
-                   {"=", tok::equal},
-                   // FIXME:
-                   // {"<", tok::less},
-                   {">", tok::greater},
-                   {"+=", tok::plusequal},
-                   {"-=", tok::minusequal},
-                   {"*=", tok::starequal},
-                   {"/=", tok::slashequal},
-                   {"%=", tok::percentequal},
-                   {"^=", tok::caretequal},
-                   {"&=", tok::ampequal},
-                   {"|=", tok::pipeequal},
-                   {"<<", tok::lessless},
-                   {">>", tok::greatergreater},
-                   {">>=", tok::greatergreaterequal},
-                   {"<<=", tok::lesslessequal},
-                   {"==", tok::equalequal},
-                   {"!=", tok::exclaimequal},
-                   {"<=", tok::lessequal},
-                   {">=", tok::greaterequal},
-                   {"<=>", tok::spaceship},
-                   {"&&", tok::ampamp},
-                   {"||", tok::pipepipe},
-                   {"++", tok::plusplus},
-                   {"--", tok::minusminus},
-                   {",", tok::comma},
-                   {"->*", tok::arrowstar},
-                   {"->", tok::arrow}};
-
-  for (const auto &Operator : Operators) {
-    std::string Input("C<&operator");
-    Input += Operator.Text;
-    Input += " > a;";
-    auto Tokens = annotate(std::string(Input));
-    ASSERT_EQ(Tokens.size(), 9u) << Tokens;
-    EXPECT_TOKEN(Tokens[1], tok::less, TT_TemplateOpener);
-    EXPECT_TOKEN(Tokens[4], Operator.Kind, TT_OverloadedOperator);
-    EXPECT_TOKEN(Tokens[5], tok::greater, TT_TemplateCloser);
-  }
-
-  auto Tokens = annotate("C<&operator< <X>> lt;");
-  ASSERT_EQ(Tokens.size(), 12u) << Tokens;
-  EXPECT_TOKEN(Tokens[1], tok::less, TT_TemplateOpener);
-  EXPECT_TOKEN(Tokens[4], tok::less, TT_OverloadedOperator);
-  EXPECT_TOKEN(Tokens[5], tok::less, TT_TemplateOpener);
-  EXPECT_TOKEN(Tokens[7], tok::greater, TT_TemplateCloser);
-  EXPECT_TOKEN(Tokens[8], tok::greater, TT_TemplateCloser);
-}
-
 TEST_F(TokenAnnotatorTest, UnderstandsRequiresClausesAndConcepts) {
   auto Tokens = annotate("template <typename T>\n"
                          "concept C = (Foo && Bar) && (Bar && Baz);");
@@ -740,10 +679,15 @@ TEST_F(TokenAnnotatorTest, UnderstandsRequiresClausesAndConcepts) {
                     "void S::bar() const & requires Baz<T> { }");
   ASSERT_EQ(Tokens.size(), 85u) << Tokens;
   EXPECT_TOKEN(Tokens[13], tok::kw_requires, TT_RequiresClause);
+  EXPECT_TOKEN(Tokens[24], tok::amp, TT_PointerOrReference);
   EXPECT_TOKEN(Tokens[25], tok::kw_requires, TT_RequiresClause);
+  EXPECT_TOKEN(Tokens[35], tok::ampamp, TT_PointerOrReference);
   EXPECT_TOKEN(Tokens[36], tok::kw_requires, TT_RequiresClause);
+  EXPECT_TOKEN(Tokens[47], tok::amp, TT_PointerOrReference);
   EXPECT_TOKEN(Tokens[49], tok::kw_requires, TT_RequiresClause);
+  EXPECT_TOKEN(Tokens[59], tok::ampamp, TT_PointerOrReference);
   EXPECT_TOKEN(Tokens[61], tok::kw_requires, TT_RequiresClause);
+  EXPECT_TOKEN(Tokens[76], tok::amp, TT_PointerOrReference);
   EXPECT_TOKEN(Tokens[77], tok::kw_requires, TT_RequiresClause);
 
   Tokens = annotate("void Class::member() && requires(Constant) {}");
diff --git a/clang/unittests/Frontend/ASTUnitTest.cpp b/clang/unittests/Frontend/ASTUnitTest.cpp
index f3859536f368c..d9d4932b8fee0 100644
--- a/clang/unittests/Frontend/ASTUnitTest.cpp
+++ b/clang/unittests/Frontend/ASTUnitTest.cpp
@@ -167,7 +167,7 @@ TEST_F(ASTUnitTest, LoadFromCommandLineEarlyError) {
   std::unique_ptr<clang::ASTUnit> ErrUnit;
 
   ASTUnit *AST = ASTUnit::LoadFromCommandLine(
-      &Args[0], &Args[4], PCHContainerOps, Diags, "", false,
+      &Args[0], &Args[4], PCHContainerOps, Diags, "", false, "", false,
       CaptureDiagsKind::All, std::nullopt, true, 0, TU_Complete, false, false,
       false, SkipFunctionBodiesScope::None, false, true, false, false,
       std::nullopt, &ErrUnit, nullptr);
diff --git a/clang/unittests/Tooling/ToolingTest.cpp b/clang/unittests/Tooling/ToolingTest.cpp
index 93dbd03719b8b..ebe03fda78f1e 100644
--- a/clang/unittests/Tooling/ToolingTest.cpp
+++ b/clang/unittests/Tooling/ToolingTest.cpp
@@ -17,11 +17,11 @@
 #include "clang/Frontend/FrontendAction.h"
 #include "clang/Frontend/FrontendActions.h"
 #include "clang/Frontend/TextDiagnosticBuffer.h"
+#include "clang/Testing/CommandLineArgs.h"
 #include "clang/Tooling/ArgumentsAdjusters.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringRef.h"
-#include "llvm/MC/TargetRegistry.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/TargetSelect.h"
 #include "llvm/TargetParser/Host.h"
@@ -871,28 +871,10 @@ TEST(ClangToolTest, StripPluginsAdjuster) {
   EXPECT_FALSE(HasFlag("-random-plugin"));
 }
 
-namespace {
-/// Find a target name such that looking for it in TargetRegistry by that name
-/// returns the same target. We expect that there is at least one target
-/// configured with this property.
-std::string getAnyTarget() {
+TEST(addTargetAndModeForProgramName, AddsTargetAndMode) {
   llvm::InitializeAllTargets();
-  for (const auto &Target : llvm::TargetRegistry::targets()) {
-    std::string Error;
-    StringRef TargetName(Target.getName());
-    if (TargetName == "x86-64")
-      TargetName = "x86_64";
-    if (llvm::TargetRegistry::lookupTarget(std::string(TargetName), Error) ==
-        &Target) {
-      return std::string(TargetName);
-    }
-  }
-  return "";
-}
-}
 
-TEST(addTargetAndModeForProgramName, AddsTargetAndMode) {
-  std::string Target = getAnyTarget();
+  std::string Target = getAnyTargetForTesting();
   ASSERT_FALSE(Target.empty());
 
   std::vector<std::string> Args = {"clang", "-foo"};
@@ -905,7 +887,8 @@ TEST(addTargetAndModeForProgramName, AddsTargetAndMode) {
 }
 
 TEST(addTargetAndModeForProgramName, PathIgnored) {
-  std::string Target = getAnyTarget();
+  llvm::InitializeAllTargets();
+  std::string Target = getAnyTargetForTesting();
   ASSERT_FALSE(Target.empty());
 
   SmallString<32> ToolPath;
@@ -919,7 +902,8 @@ TEST(addTargetAndModeForProgramName, PathIgnored) {
 }
 
 TEST(addTargetAndModeForProgramName, IgnoresExistingTarget) {
-  std::string Target = getAnyTarget();
+  llvm::InitializeAllTargets();
+  std::string Target = getAnyTargetForTesting();
   ASSERT_FALSE(Target.empty());
 
   std::vector<std::string> Args = {"clang", "-foo", "-target", "something"};
@@ -936,7 +920,8 @@ TEST(addTargetAndModeForProgramName, IgnoresExistingTarget) {
 }
 
 TEST(addTargetAndModeForProgramName, IgnoresExistingMode) {
-  std::string Target = getAnyTarget();
+  llvm::InitializeAllTargets();
+  std::string Target = getAnyTargetForTesting();
   ASSERT_FALSE(Target.empty());
 
   std::vector<std::string> Args = {"clang", "-foo", "--driver-mode=abc"};
diff --git a/clang/unittests/libclang/LibclangTest.cpp b/clang/unittests/libclang/LibclangTest.cpp
index 11a729fc040d2..ca762ebb7492b 100644
--- a/clang/unittests/libclang/LibclangTest.cpp
+++ b/clang/unittests/libclang/LibclangTest.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "TestUtils.h"
 #include "clang-c/Index.h"
 #include "clang-c/Rewrite.h"
 #include "llvm/ADT/StringRef.h"
@@ -14,7 +15,7 @@
 #include "llvm/Support/Path.h"
 #include "llvm/Support/raw_ostream.h"
 #include "gtest/gtest.h"
-#include "TestUtils.h"
+#include <cstring>
 #include <fstream>
 #include <functional>
 #include <map>
@@ -355,6 +356,194 @@ TEST(libclang, ModuleMapDescriptor) {
   clang_ModuleMapDescriptor_dispose(MMD);
 }
 
+TEST_F(LibclangParseTest, GlobalOptions) {
+  EXPECT_EQ(clang_CXIndex_getGlobalOptions(Index), CXGlobalOpt_None);
+}
+
+class LibclangIndexOptionsTest : public LibclangParseTest {
+  virtual void AdjustOptions(CXIndexOptions &Opts) {}
+
+protected:
+  void CreateIndex() override {
+    CXIndexOptions Opts;
+    memset(&Opts, 0, sizeof(Opts));
+    Opts.Size = sizeof(CXIndexOptions);
+    AdjustOptions(Opts);
+    Index = clang_createIndexWithOptions(&Opts);
+    ASSERT_TRUE(Index);
+  }
+};
+
+TEST_F(LibclangIndexOptionsTest, GlobalOptions) {
+  EXPECT_EQ(clang_CXIndex_getGlobalOptions(Index), CXGlobalOpt_None);
+}
+
+class LibclangIndexingEnabledIndexOptionsTest
+    : public LibclangIndexOptionsTest {
+  void AdjustOptions(CXIndexOptions &Opts) override {
+    Opts.ThreadBackgroundPriorityForIndexing = CXChoice_Enabled;
+  }
+};
+
+TEST_F(LibclangIndexingEnabledIndexOptionsTest, GlobalOptions) {
+  EXPECT_EQ(clang_CXIndex_getGlobalOptions(Index),
+            CXGlobalOpt_ThreadBackgroundPriorityForIndexing);
+}
+
+class LibclangIndexingDisabledEditingEnabledIndexOptionsTest
+    : public LibclangIndexOptionsTest {
+  void AdjustOptions(CXIndexOptions &Opts) override {
+    Opts.ThreadBackgroundPriorityForIndexing = CXChoice_Disabled;
+    Opts.ThreadBackgroundPriorityForEditing = CXChoice_Enabled;
+  }
+};
+
+TEST_F(LibclangIndexingDisabledEditingEnabledIndexOptionsTest, GlobalOptions) {
+  EXPECT_EQ(clang_CXIndex_getGlobalOptions(Index),
+            CXGlobalOpt_ThreadBackgroundPriorityForEditing);
+}
+
+class LibclangBothEnabledIndexOptionsTest : public LibclangIndexOptionsTest {
+  void AdjustOptions(CXIndexOptions &Opts) override {
+    Opts.ThreadBackgroundPriorityForIndexing = CXChoice_Enabled;
+    Opts.ThreadBackgroundPriorityForEditing = CXChoice_Enabled;
+  }
+};
+
+TEST_F(LibclangBothEnabledIndexOptionsTest, GlobalOptions) {
+  EXPECT_EQ(clang_CXIndex_getGlobalOptions(Index),
+            CXGlobalOpt_ThreadBackgroundPriorityForAll);
+}
+
+class LibclangPreambleStorageTest : public LibclangParseTest {
+  std::string Main = "main.cpp";
+
+protected:
+  std::string PreambleDir;
+  void InitializePreambleDir() {
+    llvm::SmallString<128> PathBuffer(TestDir);
+    llvm::sys::path::append(PathBuffer, "preambles");
+    namespace fs = llvm::sys::fs;
+    ASSERT_FALSE(fs::create_directory(PathBuffer, false, fs::perms::owner_all));
+
+    PreambleDir = static_cast<std::string>(PathBuffer);
+    FilesAndDirsToRemove.insert(PreambleDir);
+  }
+
+public:
+  void CountPreamblesInPreambleDir(int PreambleCount) {
+    // For some reason, the preamble is not created without '\n' before `int`.
+    WriteFile(Main, "\nint main() {}");
+
+    TUFlags |= CXTranslationUnit_CreatePreambleOnFirstParse;
+    ClangTU = clang_parseTranslationUnit(Index, Main.c_str(), nullptr, 0,
+                                         nullptr, 0, TUFlags);
+
+    int FileCount = 0;
+
+    namespace fs = llvm::sys::fs;
+    std::error_code EC;
+    for (fs::directory_iterator File(PreambleDir, EC), FileEnd;
+         File != FileEnd && !EC; File.increment(EC)) {
+      ++FileCount;
+
+      EXPECT_EQ(File->type(), fs::file_type::regular_file);
+
+      const auto Filename = llvm::sys::path::filename(File->path());
+      EXPECT_EQ(Filename.size(), std::strlen("preamble-%%%%%%.pch"));
+      EXPECT_TRUE(Filename.startswith("preamble-"));
+      EXPECT_TRUE(Filename.endswith(".pch"));
+
+      const auto Status = File->status();
+      ASSERT_TRUE(Status);
+      if (false) {
+        // The permissions assertion below fails, because the .pch.tmp file is
+        // created with default permissions and replaces the .pch file along
+        // with its permissions. Therefore the permissions set in
+        // TempPCHFile::create() don't matter in the end.
+        EXPECT_EQ(Status->permissions(), fs::owner_read | fs::owner_write);
+      }
+    }
+
+    EXPECT_EQ(FileCount, PreambleCount);
+  }
+};
+
+class LibclangNotOverriddenPreambleStoragePathTest
+    : public LibclangPreambleStorageTest {
+protected:
+  void CreateIndex() override {
+    InitializePreambleDir();
+    LibclangPreambleStorageTest::CreateIndex();
+  }
+};
+
+class LibclangSetPreambleStoragePathTest : public LibclangPreambleStorageTest {
+  virtual bool StorePreamblesInMemory() { return false; }
+  virtual const char *PreambleStoragePath() = 0;
+
+protected:
+  void CreateIndex() override {
+    InitializePreambleDir();
+
+    CXIndexOptions Opts{};
+    Opts.Size = sizeof(CXIndexOptions);
+    Opts.StorePreamblesInMemory = StorePreamblesInMemory();
+    Opts.PreambleStoragePath = PreambleStoragePath();
+    Index = clang_createIndexWithOptions(&Opts);
+    ASSERT_TRUE(Index);
+  }
+};
+
+class LibclangNullPreambleStoragePathTest
+    : public LibclangSetPreambleStoragePathTest {
+  const char *PreambleStoragePath() override { return nullptr; }
+};
+class LibclangEmptyPreambleStoragePathTest
+    : public LibclangSetPreambleStoragePathTest {
+  const char *PreambleStoragePath() override { return ""; }
+};
+class LibclangPreambleDirPreambleStoragePathTest
+    : public LibclangSetPreambleStoragePathTest {
+  const char *PreambleStoragePath() override { return PreambleDir.c_str(); }
+};
+
+class LibclangStoreInMemoryNullPreambleStoragePathTest
+    : public LibclangNullPreambleStoragePathTest {
+  bool StorePreamblesInMemory() override { return true; }
+};
+class LibclangStoreInMemoryEmptyPreambleStoragePathTest
+    : public LibclangEmptyPreambleStoragePathTest {
+  bool StorePreamblesInMemory() override { return true; }
+};
+class LibclangStoreInMemoryPreambleDirPreambleStoragePathTest
+    : public LibclangPreambleDirPreambleStoragePathTest {
+  bool StorePreamblesInMemory() override { return true; }
+};
+
+TEST_F(LibclangNotOverriddenPreambleStoragePathTest, CountPreambles) {
+  CountPreamblesInPreambleDir(0);
+}
+TEST_F(LibclangNullPreambleStoragePathTest, CountPreambles) {
+  CountPreamblesInPreambleDir(0);
+}
+TEST_F(LibclangEmptyPreambleStoragePathTest, CountPreambles) {
+  CountPreamblesInPreambleDir(0);
+}
+TEST_F(LibclangPreambleDirPreambleStoragePathTest, CountPreambles) {
+  CountPreamblesInPreambleDir(1);
+}
+TEST_F(LibclangStoreInMemoryNullPreambleStoragePathTest, CountPreambles) {
+  CountPreamblesInPreambleDir(0);
+}
+TEST_F(LibclangStoreInMemoryEmptyPreambleStoragePathTest, CountPreambles) {
+  CountPreamblesInPreambleDir(0);
+}
+TEST_F(LibclangStoreInMemoryPreambleDirPreambleStoragePathTest,
+       CountPreambles) {
+  CountPreamblesInPreambleDir(0);
+}
+
 TEST_F(LibclangParseTest, AllSkippedRanges) {
   std::string Header = "header.h", Main = "main.cpp";
   WriteFile(Header,
diff --git a/clang/unittests/libclang/TestUtils.h b/clang/unittests/libclang/TestUtils.h
index c69746c4408a9..c78351d4afd20 100644
--- a/clang/unittests/libclang/TestUtils.h
+++ b/clang/unittests/libclang/TestUtils.h
@@ -22,11 +22,11 @@
 #include <vector>
 
 class LibclangParseTest : public ::testing::Test {
-  // std::greater<> to remove files before their parent dirs in TearDown().
-  std::set<std::string, std::greater<>> Files;
   typedef std::unique_ptr<std::string> fixed_addr_string;
   std::map<fixed_addr_string, fixed_addr_string> UnsavedFileContents;
 public:
+  // std::greater<> to remove files before their parent dirs in TearDown().
+  std::set<std::string, std::greater<>> FilesAndDirsToRemove;
   std::string TestDir;
   bool RemoveTestDirRecursivelyDuringTeardown = false;
   CXIndex Index;
@@ -40,7 +40,7 @@ class LibclangParseTest : public ::testing::Test {
     TestDir = std::string(Dir.str());
     TUFlags = CXTranslationUnit_DetailedPreprocessingRecord |
       clang_defaultEditingTranslationUnitOptions();
-    Index = clang_createIndex(0, 0);
+    CreateIndex();
     ClangTU = nullptr;
   }
   void TearDown() override {
@@ -48,7 +48,7 @@ class LibclangParseTest : public ::testing::Test {
     clang_disposeIndex(Index);
 
     namespace fs = llvm::sys::fs;
-    for (const std::string &Path : Files)
+    for (const std::string &Path : FilesAndDirsToRemove)
       EXPECT_FALSE(fs::remove(Path, /*IgnoreNonExisting=*/false));
     if (RemoveTestDirRecursivelyDuringTeardown)
       EXPECT_FALSE(fs::remove_directories(TestDir, /*IgnoreErrors=*/false));
@@ -63,7 +63,7 @@ class LibclangParseTest : public ::testing::Test {
            FileI != FileEnd; ++FileI) {
         ASSERT_NE(*FileI, ".");
         path::append(Path, *FileI);
-        Files.emplace(Path.str());
+        FilesAndDirsToRemove.emplace(Path.str());
       }
       Filename = std::string(Path.str());
     }
@@ -101,6 +101,9 @@ class LibclangParseTest : public ::testing::Test {
     return string;
   };
 
+protected:
+  virtual void CreateIndex() { Index = clang_createIndex(0, 0); }
+
 private:
   template<typename TState>
   static CXChildVisitResult TraverseStateless(CXCursor cx, CXCursor parent,
diff --git a/clang/utils/TableGen/ClangAttrEmitter.cpp b/clang/utils/TableGen/ClangAttrEmitter.cpp
index 063b104984725..d8a651606e9bd 100644
--- a/clang/utils/TableGen/ClangAttrEmitter.cpp
+++ b/clang/utils/TableGen/ClangAttrEmitter.cpp
@@ -2053,7 +2053,7 @@ bool PragmaClangAttributeSupport::isAttributedSupported(
   for (const auto *Subject : Subjects) {
     if (!isSupportedPragmaClangAttributeSubject(*Subject))
       continue;
-    if (SubjectsToRules.find(Subject) == SubjectsToRules.end())
+    if (!SubjectsToRules.contains(Subject))
       return false;
     HasAtLeastOneValidSubject = true;
   }
diff --git a/clang/utils/TableGen/ClangOpenCLBuiltinEmitter.cpp b/clang/utils/TableGen/ClangOpenCLBuiltinEmitter.cpp
index da5d1fdc2eae3..79022ce67232a 100644
--- a/clang/utils/TableGen/ClangOpenCLBuiltinEmitter.cpp
+++ b/clang/utils/TableGen/ClangOpenCLBuiltinEmitter.cpp
@@ -324,6 +324,18 @@ class OpenCLBuiltinTestEmitter : public OpenCLBuiltinFileEmitterBase {
   void emit() override;
 };
 
+// OpenCL builtin header generator.  This class processes the same TableGen
+// input as BuiltinNameEmitter, but generates a .h file that contains a
+// prototype for each builtin function described in the .td input.
+class OpenCLBuiltinHeaderEmitter : public OpenCLBuiltinFileEmitterBase {
+public:
+  OpenCLBuiltinHeaderEmitter(RecordKeeper &Records, raw_ostream &OS)
+      : OpenCLBuiltinFileEmitterBase(Records, OS) {}
+
+  // Entrypoint to generate the header.
+  void emit() override;
+};
+
 } // namespace
 
 void BuiltinNameEmitter::Emit() {
@@ -357,7 +369,7 @@ void BuiltinNameEmitter::ExtractEnumTypes(std::vector<Record *> &Types,
   raw_string_ostream SS(Output);
 
   for (const auto *T : Types) {
-    if (TypesSeen.find(T->getValueAsString("Name")) == TypesSeen.end()) {
+    if (!TypesSeen.contains(T->getValueAsString("Name"))) {
       SS << "  OCLT_" + T->getValueAsString("Name") << ",\n";
       // Save the type names in the same order as their enum value. Note that
       // the Record can be a VectorType or something else, only the name is
@@ -498,7 +510,7 @@ void BuiltinNameEmitter::GetOverloads() {
   std::vector<Record *> Builtins = Records.getAllDerivedDefinitions("Builtin");
   for (const auto *B : Builtins) {
     StringRef BName = B->getValueAsString("Name");
-    if (FctOverloadMap.find(BName) == FctOverloadMap.end()) {
+    if (!FctOverloadMap.contains(BName)) {
       FctOverloadMap.insert(std::make_pair(
           BName, std::vector<std::pair<const Record *, unsigned>>{}));
     }
@@ -895,10 +907,10 @@ static void OCL2Qual(Sema &S, const OpenCLTypeStruct &Ty,
 
   for (const auto *T : Types) {
     // Check this is not an image type
-    if (ImageTypesMap.find(T->getValueAsString("Name")) != ImageTypesMap.end())
+    if (ImageTypesMap.contains(T->getValueAsString("Name")))
       continue;
     // Check we have not seen this Type
-    if (TypesSeen.find(T->getValueAsString("Name")) != TypesSeen.end())
+    if (TypesSeen.contains(T->getValueAsString("Name")))
       continue;
     TypesSeen.insert(std::make_pair(T->getValueAsString("Name"), true));
 
@@ -1083,7 +1095,7 @@ void OpenCLBuiltinFileEmitterBase::expandTypesInSignature(
         // the full type name to the extension.
         StringRef Ext =
             Type->getValueAsDef("Extension")->getValueAsString("ExtName");
-        if (!Ext.empty() && TypeExtMap.find(FullType) == TypeExtMap.end()) {
+        if (!Ext.empty() && !TypeExtMap.contains(FullType)) {
           TypeExtMap.insert({FullType, Ext});
         }
       }
@@ -1260,11 +1272,76 @@ void OpenCLBuiltinTestEmitter::emit() {
   }
 }
 
+void OpenCLBuiltinHeaderEmitter::emit() {
+  emitSourceFileHeader("OpenCL Builtin declarations", OS);
+
+  emitExtensionSetup();
+
+  OS << R"(
+#define __ovld __attribute__((overloadable))
+#define __conv __attribute__((convergent))
+#define __purefn __attribute__((pure))
+#define __cnfn __attribute__((const))
+
+)";
+
+  // Iterate over all builtins; sort to follow order of definition in .td file.
+  std::vector<Record *> Builtins = Records.getAllDerivedDefinitions("Builtin");
+  llvm::sort(Builtins, LessRecord());
+
+  for (const auto *B : Builtins) {
+    StringRef Name = B->getValueAsString("Name");
+
+    std::string OptionalExtensionEndif = emitExtensionGuard(B);
+    std::string OptionalVersionEndif = emitVersionGuard(B);
+
+    SmallVector<SmallVector<std::string, 2>, 4> FTypes;
+    expandTypesInSignature(B->getValueAsListOfDefs("Signature"), FTypes);
+
+    for (const auto &Signature : FTypes) {
+      StringRef OptionalTypeExtEndif = emitTypeExtensionGuards(Signature);
+
+      // Emit function declaration.
+      OS << Signature[0] << " __ovld ";
+      if (B->getValueAsBit("IsConst"))
+        OS << "__cnfn ";
+      if (B->getValueAsBit("IsPure"))
+        OS << "__purefn ";
+      if (B->getValueAsBit("IsConv"))
+        OS << "__conv ";
+
+      OS << Name << "(";
+      if (Signature.size() > 1) {
+        for (unsigned I = 1; I < Signature.size(); I++) {
+          if (I != 1)
+            OS << ", ";
+          OS << Signature[I];
+        }
+      }
+      OS << ");\n";
+
+      OS << OptionalTypeExtEndif;
+    }
+
+    OS << OptionalVersionEndif;
+    OS << OptionalExtensionEndif;
+  }
+
+  OS << "\n// Disable any extensions we may have enabled previously.\n"
+        "#pragma OPENCL EXTENSION all : disable";
+}
+
 void clang::EmitClangOpenCLBuiltins(RecordKeeper &Records, raw_ostream &OS) {
   BuiltinNameEmitter NameChecker(Records, OS);
   NameChecker.Emit();
 }
 
+void clang::EmitClangOpenCLBuiltinHeader(RecordKeeper &Records,
+                                         raw_ostream &OS) {
+  OpenCLBuiltinHeaderEmitter HeaderFileGenerator(Records, OS);
+  HeaderFileGenerator.emit();
+}
+
 void clang::EmitClangOpenCLBuiltinTests(RecordKeeper &Records,
                                         raw_ostream &OS) {
   OpenCLBuiltinTestEmitter TestFileGenerator(Records, OS);
diff --git a/clang/utils/TableGen/SveEmitter.cpp b/clang/utils/TableGen/SveEmitter.cpp
index 1a90767cf0d2b..bc50cbad4b541 100644
--- a/clang/utils/TableGen/SveEmitter.cpp
+++ b/clang/utils/TableGen/SveEmitter.cpp
@@ -1140,7 +1140,9 @@ void SVEEmitter::createHeader(raw_ostream &OS) {
   OS << "typedef __clang_svfloat16x4_t svfloat16x4_t;\n";
   OS << "typedef __clang_svfloat32x4_t svfloat32x4_t;\n";
   OS << "typedef __clang_svfloat64x4_t svfloat64x4_t;\n";
-  OS << "typedef __SVBool_t  svbool_t;\n\n";
+  OS << "typedef __SVBool_t  svbool_t;\n";
+  OS << "typedef __clang_svboolx2_t  svboolx2_t;\n";
+  OS << "typedef __clang_svboolx4_t  svboolx4_t;\n\n";
 
   OS << "typedef __clang_svbfloat16x2_t svbfloat16x2_t;\n";
   OS << "typedef __clang_svbfloat16x3_t svbfloat16x3_t;\n";
diff --git a/clang/utils/TableGen/TableGen.cpp b/clang/utils/TableGen/TableGen.cpp
index 6864ba2040ef9..a67e1d1af5d75 100644
--- a/clang/utils/TableGen/TableGen.cpp
+++ b/clang/utils/TableGen/TableGen.cpp
@@ -65,6 +65,7 @@ enum ActionType {
   GenClangCommentCommandInfo,
   GenClangCommentCommandList,
   GenClangOpenCLBuiltins,
+  GenClangOpenCLBuiltinHeader,
   GenClangOpenCLBuiltinTests,
   GenArmNeon,
   GenArmFP16,
@@ -200,6 +201,9 @@ cl::opt<ActionType> Action(
                    "documentation comments"),
         clEnumValN(GenClangOpenCLBuiltins, "gen-clang-opencl-builtins",
                    "Generate OpenCL builtin declaration handlers"),
+        clEnumValN(GenClangOpenCLBuiltinHeader,
+                   "gen-clang-opencl-builtin-header",
+                   "Generate OpenCL builtin header"),
         clEnumValN(GenClangOpenCLBuiltinTests, "gen-clang-opencl-builtin-tests",
                    "Generate OpenCL builtin declaration tests"),
         clEnumValN(GenArmNeon, "gen-arm-neon", "Generate arm_neon.h for clang"),
@@ -384,6 +388,9 @@ bool ClangTableGenMain(raw_ostream &OS, RecordKeeper &Records) {
   case GenClangOpenCLBuiltins:
     EmitClangOpenCLBuiltins(Records, OS);
     break;
+  case GenClangOpenCLBuiltinHeader:
+    EmitClangOpenCLBuiltinHeader(Records, OS);
+    break;
   case GenClangOpenCLBuiltinTests:
     EmitClangOpenCLBuiltinTests(Records, OS);
     break;
diff --git a/clang/utils/TableGen/TableGenBackends.h b/clang/utils/TableGen/TableGenBackends.h
index 2ba857f66f50b..f7df3dc0ebb67 100644
--- a/clang/utils/TableGen/TableGenBackends.h
+++ b/clang/utils/TableGen/TableGenBackends.h
@@ -124,6 +124,8 @@ void EmitClangOptDocs(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 
 void EmitClangOpenCLBuiltins(llvm::RecordKeeper &Records,
                              llvm::raw_ostream &OS);
+void EmitClangOpenCLBuiltinHeader(llvm::RecordKeeper &Records,
+                                  llvm::raw_ostream &OS);
 void EmitClangOpenCLBuiltinTests(llvm::RecordKeeper &Records,
                                  llvm::raw_ostream &OS);
 
diff --git a/clang/www/c_status.html b/clang/www/c_status.html
index 20eec37fce3de..764c7c58c7a51 100644
--- a/clang/www/c_status.html
+++ b/clang/www/c_status.html
@@ -829,7 +829,15 @@ <h2 id="c2x">C2x implementation status</h2>
     <tr>
       <td>Compatibility of Pointers to Arrays with Qualifiers</td>
       <td><a href="https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2607.pdf">N2607</a></td>
-      <td class="partial" align="center">Partial</td>
+      <td class="partial" align="center">
+        <details><summary>Partial</summary>
+          Much of the proposal is implemented, but Clang lacks pedantic diagnostics
+          in C17 and earlier regarding use of incompatible pointer types as an
+          extension. Further, Clang does not properly compute the correct result
+          type for the <code>?:</code> operator when the result type should be a
+          qualified array type.
+        </details>
+      </td>
     </tr>
     <tr>
       <td>Unclear type relationship between a format specifier and its argument</td>
@@ -1048,7 +1056,7 @@ <h2 id="c2x">C2x implementation status</h2>
     <tr>
       <td>Remove default argument promotions for _FloatN types</td>
       <td><a href="https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2844.pdf">N2844</a></td>
-      <td class="unknown" align="center">Unknown</td>
+      <td class="none" align="center">No</td>
     </tr>
     <tr>
       <td>Revised Suggestions of Change for Numerically Equal/Equivalent</td>
diff --git a/clang/www/cxx_status.html b/clang/www/cxx_status.html
index 3353bd6a9fc28..0c742b358bb6c 100755
--- a/clang/www/cxx_status.html
+++ b/clang/www/cxx_status.html
@@ -1163,7 +1163,7 @@ <h2 id="cxx20">C++20 implementation status</h2>
     <tr>
       <td rowspan="2">Parenthesized initialization of aggregates</td>
       <td><a href="https://wg21.link/p0960r3">P0960R3</a></td>
-      <td rowspan="2" class="unreleased" align="center">Clang 16</td>
+      <td rowspan="2" class="none" align="center">No</td>
     </tr>
       <tr> <!-- from Belfast -->
         <td><a href="https://wg21.link/p1975r0">P1975R0</a></td>
diff --git a/compiler-rt/cmake/Modules/AllSupportedArchDefs.cmake b/compiler-rt/cmake/Modules/AllSupportedArchDefs.cmake
index cc929c521f85f..e3fe5570de26a 100644
--- a/compiler-rt/cmake/Modules/AllSupportedArchDefs.cmake
+++ b/compiler-rt/cmake/Modules/AllSupportedArchDefs.cmake
@@ -70,7 +70,7 @@ set(ALL_SAFESTACK_SUPPORTED_ARCH ${X86} ${X86_64} ${ARM64} ${MIPS32} ${MIPS64}
 set(ALL_CFI_SUPPORTED_ARCH ${X86} ${X86_64} ${ARM32} ${ARM64} ${MIPS64}
     ${HEXAGON})
 set(ALL_SCUDO_STANDALONE_SUPPORTED_ARCH ${X86} ${X86_64} ${ARM32} ${ARM64}
-    ${MIPS32} ${MIPS64} ${PPC64} ${HEXAGON} ${LOONGARCH64})
+    ${MIPS32} ${MIPS64} ${PPC64} ${HEXAGON} ${LOONGARCH64} ${RISCV64})
 if(APPLE)
 set(ALL_XRAY_SUPPORTED_ARCH ${X86_64})
 else()
diff --git a/compiler-rt/cmake/builtin-config-ix.cmake b/compiler-rt/cmake/builtin-config-ix.cmake
index 9f818499a076f..3638cff56eb08 100644
--- a/compiler-rt/cmake/builtin-config-ix.cmake
+++ b/compiler-rt/cmake/builtin-config-ix.cmake
@@ -22,22 +22,6 @@ int foo(int x, int y) {
 }
 ")
 
-builtin_check_c_compiler_source(COMPILER_RT_HAS_FLOAT16
-"
-_Float16 foo(_Float16 x) {
- return x;
-}
-"
-)
-
-builtin_check_c_compiler_source(COMPILER_RT_HAS_BFLOAT16
-"
-__bf16 foo(__bf16 x) {
- return x;
-}
-"
-)
-
 builtin_check_c_compiler_source(COMPILER_RT_HAS_ASM_LSE
 "
 asm(\".arch armv8-a+lse\");
@@ -45,7 +29,7 @@ asm(\"cas w0, w1, [x2]\");
 ")
 
 set(ARM64 aarch64)
-set(ARM32 arm armhf armv4t armv5te armv6 armv6m armv7m armv7em armv7 armv7s armv7k armv8m.main armv8.1m.main)
+set(ARM32 arm armhf armv4t armv5te armv6 armv6m armv7m armv7em armv7 armv7s armv7k armv8m.base armv8m.main armv8.1m.main)
 set(AVR avr)
 set(HEXAGON hexagon)
 set(X86 i386)
diff --git a/compiler-rt/include/profile/MemProfData.inc b/compiler-rt/include/profile/MemProfData.inc
index c533073da751f..b82a4baf6dd74 100644
--- a/compiler-rt/include/profile/MemProfData.inc
+++ b/compiler-rt/include/profile/MemProfData.inc
@@ -19,6 +19,7 @@
  * synced up.
  *
 \*===----------------------------------------------------------------------===*/
+#include <string.h>
 
 #ifdef _MSC_VER
 #define PACKED(...) __pragma(pack(push,1)) __VA_ARGS__ __pragma(pack(pop))
@@ -32,7 +33,9 @@
    (uint64_t)'o' << 24 | (uint64_t)'f' << 16 | (uint64_t)'r' << 8 | (uint64_t)129)
 
 // The version number of the raw binary format.
-#define MEMPROF_RAW_VERSION 2ULL
+#define MEMPROF_RAW_VERSION 3ULL
+
+#define MEMPROF_BUILDID_MAX_SIZE 32ULL
 
 namespace llvm {
 namespace memprof {
@@ -46,37 +49,40 @@ PACKED(struct Header {
   uint64_t StackOffset;
 });
 
-
 // A struct describing the information necessary to describe a /proc/maps
 // segment entry for a particular binary/library identified by its build id.
 PACKED(struct SegmentEntry {
   uint64_t Start;
   uint64_t End;
   uint64_t Offset;
-  // This field is unused until sanitizer procmaps support for build ids for
-  // Linux-Elf is implemented.
-  uint8_t BuildId[32] = {0};
+  uint64_t BuildIdSize;
+  uint8_t BuildId[MEMPROF_BUILDID_MAX_SIZE] = {0};
 
-  SegmentEntry(uint64_t S, uint64_t E, uint64_t O) :
-    Start(S), End(E), Offset(O) {}
+  // This constructor is only used in tests so don't set the BuildId.
+  SegmentEntry(uint64_t S, uint64_t E, uint64_t O)
+      : Start(S), End(E), Offset(O), BuildIdSize(0) {}
 
   SegmentEntry(const SegmentEntry& S) {
     Start = S.Start;
     End = S.End;
     Offset = S.Offset;
+    BuildIdSize = S.BuildIdSize;
+    memcpy(BuildId, S.BuildId, S.BuildIdSize);
   }
 
   SegmentEntry& operator=(const SegmentEntry& S) {
     Start = S.Start;
     End = S.End;
     Offset = S.Offset;
+    BuildIdSize = S.BuildIdSize;
+    memcpy(BuildId, S.BuildId, S.BuildIdSize);
     return *this;
   }
 
   bool operator==(const SegmentEntry& S) const {
-    return Start == S.Start &&
-           End == S.End &&
-           Offset == S.Offset;
+    return Start == S.Start && End == S.End && Offset == S.Offset &&
+           BuildIdSize == S.BuildIdSize &&
+           memcmp(BuildId, S.BuildId, S.BuildIdSize) == 0;
   }
 });
 
diff --git a/compiler-rt/lib/asan/asan_thread.cpp b/compiler-rt/lib/asan/asan_thread.cpp
index 003cd2b9eee8d..56636b51618ae 100644
--- a/compiler-rt/lib/asan/asan_thread.cpp
+++ b/compiler-rt/lib/asan/asan_thread.cpp
@@ -306,6 +306,7 @@ void AsanThread::SetThreadStackAndTls(const InitOptions *options) {
   GetThreadStackAndTls(tid() == kMainTid, &stack_bottom_, &stack_size,
                        &tls_begin_, &tls_size);
   stack_top_ = RoundDownTo(stack_bottom_ + stack_size, ASAN_SHADOW_GRANULARITY);
+  stack_bottom_ = RoundDownTo(stack_bottom_, ASAN_SHADOW_GRANULARITY);
   tls_end_ = tls_begin_ + tls_size;
   dtls_ = DTLS_Get();
 
diff --git a/compiler-rt/lib/builtins/CMakeLists.txt b/compiler-rt/lib/builtins/CMakeLists.txt
index 891a92f5e19b8..0487126a8a54b 100644
--- a/compiler-rt/lib/builtins/CMakeLists.txt
+++ b/compiler-rt/lib/builtins/CMakeLists.txt
@@ -58,6 +58,7 @@ if (COMPILER_RT_STANDALONE_BUILD)
 endif()
 
 include(builtin-config-ix)
+include(CMakePushCheckState)
 
 if(${CMAKE_SYSTEM_NAME} MATCHES "AIX")
   include(CompilerRTAIXUtils)
@@ -190,14 +191,11 @@ set(GENERIC_SOURCES
   umodti3.c
 )
 
-# Build BF16 files only when "__bf16" is available.
-if(COMPILER_RT_HAS_BFLOAT16 AND NOT APPLE)
-  set(GENERIC_SOURCES
-    ${GENERIC_SOURCES}
+# We only build BF16 files when "__bf16" is available.
+set(BF16_SOURCES
     truncdfbf2.c
     truncsfbf2.c
-  )
-endif()
+)
 
 # TODO: Several "tf" files (and divtc3.c, but not multc3.c) are in
 # GENERIC_SOURCES instead of here.
@@ -607,6 +605,7 @@ set(arm64_32_SOURCES ${aarch64_SOURCES})
 set(armv6m_SOURCES ${thumb1_SOURCES})
 set(armv7m_SOURCES ${arm_SOURCES})
 set(armv7em_SOURCES ${arm_SOURCES})
+set(armv8m.base_SOURCES ${thumb1_SOURCES})
 set(armv8m.main_SOURCES ${arm_SOURCES})
 set(armv8.1m.main_SOURCES ${arm_SOURCES})
 
@@ -751,8 +750,6 @@ else ()
     append_list_if(COMPILER_RT_ENABLE_CET -fcf-protection=full BUILTIN_CFLAGS)
   endif()
 
-  append_list_if(COMPILER_RT_HAS_FLOAT16 -DCOMPILER_RT_HAS_FLOAT16 BUILTIN_CFLAGS)
-
   append_list_if(COMPILER_RT_HAS_STD_C11_FLAG -std=c11 BUILTIN_CFLAGS)
 
   # These flags would normally be added to CMAKE_C_FLAGS by the llvm
@@ -784,7 +781,11 @@ else ()
 
   foreach (arch ${BUILTIN_SUPPORTED_ARCH})
     if (CAN_TARGET_${arch})
+      cmake_push_check_state()
+      # TODO: we should probably make most of the checks in builtin-config depend on the target flags.
+      message(STATUS "Performing additional configure checks with target flags: ${TARGET_${arch}_CFLAGS}")
       set(BUILTIN_CFLAGS_${arch} ${BUILTIN_CFLAGS})
+      list(APPEND CMAKE_REQUIRED_FLAGS ${TARGET_${arch}_CFLAGS} ${BUILTIN_CFLAGS_${arch}})
       # For ARM archs, exclude any VFP builtins if VFP is not supported
       if (${arch} MATCHES "^(arm|armhf|armv7|armv7s|armv7k|armv7m|armv7em|armv8m.main|armv8.1m.main)$")
         string(REPLACE ";" " " _TARGET_${arch}_CFLAGS "${TARGET_${arch}_CFLAGS}")
@@ -803,6 +804,15 @@ else ()
           endif()
         endif()
       endif()
+      check_c_source_compiles("_Float16 foo(_Float16 x) { return x; }"
+                              COMPILER_RT_HAS_${arch}_FLOAT16)
+      append_list_if(COMPILER_RT_HAS_${arch}_FLOAT16 -DCOMPILER_RT_HAS_FLOAT16 BUILTIN_CFLAGS_${arch})
+      check_c_source_compiles("__bf16 foo(__bf16 x) { return x; }"
+                              COMPILER_RT_HAS_${arch}_BFLOAT16)
+      # Build BF16 files only when "__bf16" is available.
+      if(COMPILER_RT_HAS_${arch}_BFLOAT16)
+        list(APPEND ${arch}_SOURCES ${BF16_SOURCES})
+      endif()
 
       # Remove a generic C builtin when an arch-specific builtin is specified.
       filter_builtin_sources(${arch}_SOURCES ${arch})
@@ -837,6 +847,7 @@ else ()
                               DEFS ${BUILTIN_DEFS}
                               CFLAGS ${BUILTIN_CFLAGS_${arch}}
                               PARENT_TARGET builtins)
+      cmake_pop_check_state()
     endif ()
   endforeach ()
 endif ()
diff --git a/compiler-rt/lib/builtins/arm/divsi3.S b/compiler-rt/lib/builtins/arm/divsi3.S
index 761bf49d3ed06..faf9af917ab67 100644
--- a/compiler-rt/lib/builtins/arm/divsi3.S
+++ b/compiler-rt/lib/builtins/arm/divsi3.S
@@ -37,7 +37,8 @@ DEFINE_COMPILERRT_FUNCTION(__divsi3)
    sdiv    r0, r0, r1
    bx      lr
 LOCAL_LABEL(divzero):
-   mov     r0,#0
+   // Use movs for compatibility with v8-m.base.
+   movs    r0,#0
    bx      lr
 #else
 ESTABLISH_FRAME
diff --git a/compiler-rt/lib/builtins/arm/udivsi3.S b/compiler-rt/lib/builtins/arm/udivsi3.S
index 9b1b035b33d6c..0567ab4ab7656 100644
--- a/compiler-rt/lib/builtins/arm/udivsi3.S
+++ b/compiler-rt/lib/builtins/arm/udivsi3.S
@@ -32,7 +32,8 @@ DEFINE_COMPILERRT_FUNCTION(__udivsi3)
 	bx  	lr
 
 LOCAL_LABEL(divby0):
-	mov     r0, #0
+	// Use movs for compatibility with v8-m.base.
+	movs    r0, #0
 #  ifdef __ARM_EABI__
 	b       __aeabi_idiv0
 #  else
diff --git a/compiler-rt/lib/builtins/cpu_model.c b/compiler-rt/lib/builtins/cpu_model.c
index 4e3ce7af9926b..01bab46aebac7 100644
--- a/compiler-rt/lib/builtins/cpu_model.c
+++ b/compiler-rt/lib/builtins/cpu_model.c
@@ -924,6 +924,9 @@ int CONSTRUCTOR_ATTRIBUTE __cpu_indicator_init(void) {
 #define HWCAP_SB (1 << 29)
 #endif
 
+#ifndef AT_HWCAP2
+#define AT_HWCAP2 26
+#endif
 #ifndef HWCAP2_DCPODP
 #define HWCAP2_DCPODP (1 << 0)
 #endif
diff --git a/compiler-rt/lib/builtins/floatdidf.c b/compiler-rt/lib/builtins/floatdidf.c
index d37c43b1f2f92..c994aad3f079e 100644
--- a/compiler-rt/lib/builtins/floatdidf.c
+++ b/compiler-rt/lib/builtins/floatdidf.c
@@ -50,7 +50,7 @@ COMPILER_RT_ABI double __floatdidf(di_int a) {
     return 0.0;
   const unsigned N = sizeof(di_int) * CHAR_BIT;
   const di_int s = a >> (N - 1);
-  a = (a ^ s) - s;
+  a = (du_int)(a ^ s) - s;
   int sd = N - __builtin_clzll(a); // number of significant digits
   int e = sd - 1;                  // exponent
   if (sd > DBL_MANT_DIG) {
diff --git a/compiler-rt/lib/builtins/floatdisf.c b/compiler-rt/lib/builtins/floatdisf.c
index 5c6316431e394..0b62ed8689bc6 100644
--- a/compiler-rt/lib/builtins/floatdisf.c
+++ b/compiler-rt/lib/builtins/floatdisf.c
@@ -24,7 +24,7 @@ COMPILER_RT_ABI float __floatdisf(di_int a) {
     return 0.0F;
   const unsigned N = sizeof(di_int) * CHAR_BIT;
   const di_int s = a >> (N - 1);
-  a = (a ^ s) - s;
+  a = (du_int)(a ^ s) - s;
   int sd = N - __builtin_clzll(a); // number of significant digits
   si_int e = sd - 1;               // exponent
   if (sd > FLT_MANT_DIG) {
diff --git a/compiler-rt/lib/builtins/int_mulo_impl.inc b/compiler-rt/lib/builtins/int_mulo_impl.inc
index 567d8b9e6e603..592b7893edbdc 100644
--- a/compiler-rt/lib/builtins/int_mulo_impl.inc
+++ b/compiler-rt/lib/builtins/int_mulo_impl.inc
@@ -18,7 +18,7 @@
 
 static __inline fixint_t __muloXi4(fixint_t a, fixint_t b, int *overflow) {
   const int N = (int)(sizeof(fixint_t) * CHAR_BIT);
-  const fixint_t MIN = (fixint_t)1 << (N - 1);
+  const fixint_t MIN = (fixint_t)((fixuint_t)1 << (N - 1));
   const fixint_t MAX = ~MIN;
   *overflow = 0;
   fixint_t result = a * b;
diff --git a/compiler-rt/lib/builtins/int_mulv_impl.inc b/compiler-rt/lib/builtins/int_mulv_impl.inc
index 1e920716ec499..06559cf302ea6 100644
--- a/compiler-rt/lib/builtins/int_mulv_impl.inc
+++ b/compiler-rt/lib/builtins/int_mulv_impl.inc
@@ -18,7 +18,7 @@
 
 static __inline fixint_t __mulvXi3(fixint_t a, fixint_t b) {
   const int N = (int)(sizeof(fixint_t) * CHAR_BIT);
-  const fixint_t MIN = (fixint_t)1 << (N - 1);
+  const fixint_t MIN = (fixint_t)((fixuint_t)1 << (N - 1));
   const fixint_t MAX = ~MIN;
   if (a == MIN) {
     if (b == 0 || b == 1)
diff --git a/compiler-rt/lib/builtins/mulodi4.c b/compiler-rt/lib/builtins/mulodi4.c
index 7209676a327e4..6ecf92664fb5c 100644
--- a/compiler-rt/lib/builtins/mulodi4.c
+++ b/compiler-rt/lib/builtins/mulodi4.c
@@ -11,6 +11,7 @@
 //===----------------------------------------------------------------------===//
 
 #define fixint_t di_int
+#define fixuint_t du_int
 #include "int_mulo_impl.inc"
 
 // Returns: a * b
diff --git a/compiler-rt/lib/builtins/mulosi4.c b/compiler-rt/lib/builtins/mulosi4.c
index 4e03c24455d67..3fd18a122a463 100644
--- a/compiler-rt/lib/builtins/mulosi4.c
+++ b/compiler-rt/lib/builtins/mulosi4.c
@@ -11,6 +11,7 @@
 //===----------------------------------------------------------------------===//
 
 #define fixint_t si_int
+#define fixuint_t su_int
 #include "int_mulo_impl.inc"
 
 // Returns: a * b
diff --git a/compiler-rt/lib/builtins/muloti4.c b/compiler-rt/lib/builtins/muloti4.c
index 9a7aa85b022bf..9aab6fc3efb33 100644
--- a/compiler-rt/lib/builtins/muloti4.c
+++ b/compiler-rt/lib/builtins/muloti4.c
@@ -19,6 +19,7 @@
 // Effects: sets *overflow to 1  if a * b overflows
 
 #define fixint_t ti_int
+#define fixuint_t tu_int
 #include "int_mulo_impl.inc"
 
 COMPILER_RT_ABI ti_int __muloti4(ti_int a, ti_int b, int *overflow) {
diff --git a/compiler-rt/lib/builtins/mulvdi3.c b/compiler-rt/lib/builtins/mulvdi3.c
index 1d672c6dc155d..d787d297d564e 100644
--- a/compiler-rt/lib/builtins/mulvdi3.c
+++ b/compiler-rt/lib/builtins/mulvdi3.c
@@ -11,6 +11,7 @@
 //===----------------------------------------------------------------------===//
 
 #define fixint_t di_int
+#define fixuint_t du_int
 #include "int_mulv_impl.inc"
 
 // Returns: a * b
diff --git a/compiler-rt/lib/builtins/mulvsi3.c b/compiler-rt/lib/builtins/mulvsi3.c
index 00b2e50eeca91..2571881195fcc 100644
--- a/compiler-rt/lib/builtins/mulvsi3.c
+++ b/compiler-rt/lib/builtins/mulvsi3.c
@@ -11,6 +11,7 @@
 //===----------------------------------------------------------------------===//
 
 #define fixint_t si_int
+#define fixuint_t su_int
 #include "int_mulv_impl.inc"
 
 // Returns: a * b
diff --git a/compiler-rt/lib/builtins/mulvti3.c b/compiler-rt/lib/builtins/mulvti3.c
index ba355149f9a76..fad9b2ae2765b 100644
--- a/compiler-rt/lib/builtins/mulvti3.c
+++ b/compiler-rt/lib/builtins/mulvti3.c
@@ -19,6 +19,7 @@
 // Effects: aborts if a * b overflows
 
 #define fixint_t ti_int
+#define fixuint_t tu_int
 #include "int_mulv_impl.inc"
 
 COMPILER_RT_ABI ti_int __mulvti3(ti_int a, ti_int b) { return __mulvXi3(a, b); }
diff --git a/compiler-rt/lib/fuzzer/FuzzerCommand.h b/compiler-rt/lib/fuzzer/FuzzerCommand.h
index f653fe3587681..eb68be9a65b69 100644
--- a/compiler-rt/lib/fuzzer/FuzzerCommand.h
+++ b/compiler-rt/lib/fuzzer/FuzzerCommand.h
@@ -139,7 +139,7 @@ class Command final {
   // be the equivalent command line.
   std::string toString() const {
     std::stringstream SS;
-    for (auto arg : getArguments())
+    for (const auto &arg : getArguments())
       SS << arg << " ";
     if (hasOutputFile())
       SS << ">" << getOutputFile() << " ";
diff --git a/compiler-rt/lib/fuzzer/FuzzerDataFlowTrace.cpp b/compiler-rt/lib/fuzzer/FuzzerDataFlowTrace.cpp
index 2f9a4d2d7adcb..93bf817a857b4 100644
--- a/compiler-rt/lib/fuzzer/FuzzerDataFlowTrace.cpp
+++ b/compiler-rt/lib/fuzzer/FuzzerDataFlowTrace.cpp
@@ -88,7 +88,7 @@ bool BlockCoverage::AppendCoverage(std::istream &IN) {
 //   * a function with a less frequently executed code gets bigger weight.
 std::vector<double> BlockCoverage::FunctionWeights(size_t NumFunctions) const {
   std::vector<double> Res(NumFunctions);
-  for (auto It : Functions) {
+  for (const auto &It : Functions) {
     auto FunctionID = It.first;
     auto Counters = It.second;
     assert(FunctionID < NumFunctions);
diff --git a/compiler-rt/lib/fuzzer/FuzzerDriver.cpp b/compiler-rt/lib/fuzzer/FuzzerDriver.cpp
index 6b007f2ad45c6..8775de819d5a6 100644
--- a/compiler-rt/lib/fuzzer/FuzzerDriver.cpp
+++ b/compiler-rt/lib/fuzzer/FuzzerDriver.cpp
@@ -463,7 +463,7 @@ int MinimizeCrashInput(const std::vector<std::string> &Args,
         CurrentFilePath = Flags.exact_artifact_path;
         WriteToFile(U, CurrentFilePath);
       }
-      Printf("CRASH_MIN: failed to minimize beyond %s (%d bytes), exiting\n",
+      Printf("CRASH_MIN: failed to minimize beyond %s (%zu bytes), exiting\n",
              CurrentFilePath.c_str(), U.size());
       break;
     }
@@ -535,7 +535,7 @@ void Merge(Fuzzer *F, FuzzingOptions &Options,
 
 int AnalyzeDictionary(Fuzzer *F, const std::vector<Unit> &Dict,
                       UnitVector &Corpus) {
-  Printf("Started dictionary minimization (up to %d tests)\n",
+  Printf("Started dictionary minimization (up to %zu tests)\n",
          Dict.size() * Corpus.size() * 2);
 
   // Scores and usage count for each dictionary unit.
@@ -779,7 +779,7 @@ int FuzzerDriver(int *argc, char ***argv, UserCallback Callback) {
   if (!Options.FocusFunction.empty())
     Options.Entropic = false; // FocusFunction overrides entropic scheduling.
   if (Options.Entropic)
-    Printf("INFO: Running with entropic power schedule (0x%X, %d).\n",
+    Printf("INFO: Running with entropic power schedule (0x%zX, %zu).\n",
            Options.EntropicFeatureFrequencyThreshold,
            Options.EntropicNumberOfRarestFeatures);
   struct EntropicOptions Entropic;
@@ -797,7 +797,7 @@ int FuzzerDriver(int *argc, char ***argv, UserCallback Callback) {
   if (Flags.verbosity)
     Printf("INFO: Seed: %u\n", Seed);
 
-  if (Flags.collect_data_flow && !Flags.fork &&
+  if (Flags.collect_data_flow && Flags.data_flow_trace && !Flags.fork &&
       !(Flags.merge || Flags.set_cover_merge)) {
     if (RunIndividualFiles)
       return CollectDataFlow(Flags.collect_data_flow, Flags.data_flow_trace,
@@ -860,7 +860,7 @@ int FuzzerDriver(int *argc, char ***argv, UserCallback Callback) {
         RunOneTest(F, Path.c_str(), Options.MaxLen);
       auto StopTime = system_clock::now();
       auto MS = duration_cast<milliseconds>(StopTime - StartTime).count();
-      Printf("Executed %s in %zd ms\n", Path.c_str(), (long)MS);
+      Printf("Executed %s in %ld ms\n", Path.c_str(), (long)MS);
     }
     Printf("***\n"
            "*** NOTE: fuzzing was not performed, you have only\n"
diff --git a/compiler-rt/lib/fuzzer/FuzzerInternal.h b/compiler-rt/lib/fuzzer/FuzzerInternal.h
index a732ca87b0f31..88504705137a8 100644
--- a/compiler-rt/lib/fuzzer/FuzzerInternal.h
+++ b/compiler-rt/lib/fuzzer/FuzzerInternal.h
@@ -29,12 +29,11 @@ namespace fuzzer {
 
 using namespace std::chrono;
 
-class Fuzzer {
+class Fuzzer final {
 public:
-
   Fuzzer(UserCallback CB, InputCorpus &Corpus, MutationDispatcher &MD,
-         FuzzingOptions Options);
-  ~Fuzzer();
+         const FuzzingOptions &Options);
+  ~Fuzzer() = delete;
   void Loop(std::vector<SizedFile> &CorporaFiles);
   void ReadAndExecuteSeedCorpora(std::vector<SizedFile> &CorporaFiles);
   void MinimizeCrashLoop(const Unit &U);
diff --git a/compiler-rt/lib/fuzzer/FuzzerLoop.cpp b/compiler-rt/lib/fuzzer/FuzzerLoop.cpp
index 00f5ed7743b61..8b430c5428d88 100644
--- a/compiler-rt/lib/fuzzer/FuzzerLoop.cpp
+++ b/compiler-rt/lib/fuzzer/FuzzerLoop.cpp
@@ -136,7 +136,7 @@ void Fuzzer::HandleMalloc(size_t Size) {
 }
 
 Fuzzer::Fuzzer(UserCallback CB, InputCorpus &Corpus, MutationDispatcher &MD,
-               FuzzingOptions Options)
+               const FuzzingOptions &Options)
     : CB(CB), Corpus(Corpus), MD(MD), Options(Options) {
   if (EF->__sanitizer_set_death_callback)
     EF->__sanitizer_set_death_callback(StaticDeathCallback);
@@ -160,8 +160,6 @@ Fuzzer::Fuzzer(UserCallback CB, InputCorpus &Corpus, MutationDispatcher &MD,
   memset(BaseSha1, 0, sizeof(BaseSha1));
 }
 
-Fuzzer::~Fuzzer() {}
-
 void Fuzzer::AllocateCurrentUnitData() {
   if (CurrentUnitData || MaxInputLen == 0)
     return;
@@ -301,7 +299,7 @@ void Fuzzer::AlarmCallback() {
     Printf("       and the timeout value is %d (use -timeout=N to change)\n",
            Options.UnitTimeoutSec);
     DumpCurrentUnit("timeout-");
-    Printf("==%lu== ERROR: libFuzzer: timeout after %d seconds\n", GetPid(),
+    Printf("==%lu== ERROR: libFuzzer: timeout after %zu seconds\n", GetPid(),
            Seconds);
     PrintStackTrace();
     Printf("SUMMARY: libFuzzer: timeout\n");
@@ -314,9 +312,8 @@ void Fuzzer::RssLimitCallback() {
   if (EF->__sanitizer_acquire_crash_state &&
       !EF->__sanitizer_acquire_crash_state())
     return;
-  Printf(
-      "==%lu== ERROR: libFuzzer: out-of-memory (used: %zdMb; limit: %zdMb)\n",
-      GetPid(), GetPeakRSSMb(), Options.RssLimitMb);
+  Printf("==%lu== ERROR: libFuzzer: out-of-memory (used: %zdMb; limit: %dMb)\n",
+         GetPid(), GetPeakRSSMb(), Options.RssLimitMb);
   Printf("   To change the out-of-memory limit use -rss_limit_mb=<N>\n\n");
   PrintMemoryProfile();
   DumpCurrentUnit("oom-");
@@ -371,7 +368,7 @@ void Fuzzer::PrintFinalStats() {
   Printf("stat::number_of_executed_units: %zd\n", TotalNumberOfRuns);
   Printf("stat::average_exec_per_sec:     %zd\n", ExecPerSec);
   Printf("stat::new_units_added:          %zd\n", NumberOfNewUnitsAdded);
-  Printf("stat::slowest_unit_time_sec:    %zd\n", TimeOfLongestUnitInSeconds);
+  Printf("stat::slowest_unit_time_sec:    %ld\n", TimeOfLongestUnitInSeconds);
   Printf("stat::peak_rss_mb:              %zd\n", GetPeakRSSMb());
 }
 
@@ -455,7 +452,7 @@ void Fuzzer::PrintPulseAndReportSlowInput(const uint8_t *Data, size_t Size) {
       static_cast<long>(static_cast<double>(TimeOfLongestUnitInSeconds) * 1.1);
   if (TimeOfUnit > Threshhold && TimeOfUnit >= Options.ReportSlowUnits) {
     TimeOfLongestUnitInSeconds = TimeOfUnit;
-    Printf("Slowest unit: %zd s:\n", TimeOfLongestUnitInSeconds);
+    Printf("Slowest unit: %ld s:\n", TimeOfLongestUnitInSeconds);
     WriteUnitToFileWithPrefix({Data, Data + Size}, "slow-unit-");
   }
 }
diff --git a/compiler-rt/lib/fuzzer/FuzzerMutate.cpp b/compiler-rt/lib/fuzzer/FuzzerMutate.cpp
index d663900fdc3a1..1abce16d70d94 100644
--- a/compiler-rt/lib/fuzzer/FuzzerMutate.cpp
+++ b/compiler-rt/lib/fuzzer/FuzzerMutate.cpp
@@ -521,7 +521,7 @@ void MutationDispatcher::PrintMutationSequence(bool Verbose) {
 
 std::string MutationDispatcher::MutationSequence() {
   std::string MS;
-  for (auto M : CurrentMutatorSequence) {
+  for (const auto &M : CurrentMutatorSequence) {
     MS += M.Name;
     MS += "-";
   }
diff --git a/compiler-rt/lib/hwasan/hwasan_allocation_functions.cpp b/compiler-rt/lib/hwasan/hwasan_allocation_functions.cpp
index 9cd82dbabd19b..ad359b12904b3 100644
--- a/compiler-rt/lib/hwasan/hwasan_allocation_functions.cpp
+++ b/compiler-rt/lib/hwasan/hwasan_allocation_functions.cpp
@@ -24,6 +24,18 @@ using namespace __hwasan;
 
 struct DlsymAlloc : public DlSymAllocator<DlsymAlloc> {
   static bool UseImpl() { return !hwasan_inited; }
+  static void OnAllocate(const void *ptr, uptr size) {
+#  if CAN_SANITIZE_LEAKS
+    // Suppress leaks from dlerror(). Previously dlsym hack on global array was
+    // used by leak sanitizer as a root region.
+    __lsan_register_root_region(ptr, size);
+#  endif
+  }
+  static void OnFree(const void *ptr, uptr size) {
+#  if CAN_SANITIZE_LEAKS
+    __lsan_unregister_root_region(ptr, size);
+#  endif
+  }
 };
 
 extern "C" {
diff --git a/compiler-rt/lib/hwasan/hwasan_allocator.h b/compiler-rt/lib/hwasan/hwasan_allocator.h
index 67982cad25438..2093079413014 100644
--- a/compiler-rt/lib/hwasan/hwasan_allocator.h
+++ b/compiler-rt/lib/hwasan/hwasan_allocator.h
@@ -68,15 +68,21 @@ struct AP64 {
 
 #if defined(HWASAN_ALIASING_MODE)
   static const uptr kSpaceSize = 1ULL << kAddressTagShift;
+  typedef __sanitizer::DefaultSizeClassMap SizeClassMap;
+#elif SANITIZER_LINUX && !SANITIZER_ANDROID
+  static const uptr kSpaceSize = 0x40000000000ULL;  // 4T.
+  typedef __sanitizer::DefaultSizeClassMap SizeClassMap;
 #else
-  static const uptr kSpaceSize = 0x2000000000ULL;
+  static const uptr kSpaceSize = 0x2000000000ULL;  // 128G.
+  typedef __sanitizer::VeryDenseSizeClassMap SizeClassMap;
 #endif
+
   static const uptr kMetadataSize = sizeof(Metadata);
-  typedef __sanitizer::VeryDenseSizeClassMap SizeClassMap;
   using AddressSpaceView = LocalAddressSpaceView;
   typedef HwasanMapUnmapCallback MapUnmapCallback;
   static const uptr kFlags = 0;
 };
+
 typedef SizeClassAllocator64<AP64> PrimaryAllocator;
 typedef CombinedAllocator<PrimaryAllocator> Allocator;
 typedef Allocator::AllocatorCache AllocatorCache;
diff --git a/compiler-rt/lib/hwasan/hwasan_new_delete.cpp b/compiler-rt/lib/hwasan/hwasan_new_delete.cpp
index f31ac3a04cced..f0fd3726ef1b1 100644
--- a/compiler-rt/lib/hwasan/hwasan_new_delete.cpp
+++ b/compiler-rt/lib/hwasan/hwasan_new_delete.cpp
@@ -142,6 +142,14 @@ INTERCEPTOR_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE void operator delete[](
     void *ptr, std::align_val_t, std::nothrow_t const &) NOEXCEPT {
   OPERATOR_DELETE_BODY;
 }
+INTERCEPTOR_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE void operator delete(
+    void *ptr, size_t, std::align_val_t) NOEXCEPT {
+  OPERATOR_DELETE_BODY;
+}
+INTERCEPTOR_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE void operator delete[](
+    void *ptr, size_t, std::align_val_t) NOEXCEPT {
+  OPERATOR_DELETE_BODY;
+}
 INTERCEPTOR_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE void operator delete(
     void *ptr, size_t, std::align_val_t, std::nothrow_t const &) NOEXCEPT {
   OPERATOR_DELETE_BODY;
diff --git a/compiler-rt/lib/lsan/lsan_common.cpp b/compiler-rt/lib/lsan/lsan_common.cpp
index 8c7195abc7299..f9a456ba668e9 100644
--- a/compiler-rt/lib/lsan/lsan_common.cpp
+++ b/compiler-rt/lib/lsan/lsan_common.cpp
@@ -948,7 +948,7 @@ void LeakReport::PrintSummary() {
 
 uptr LeakReport::ApplySuppressions() {
   LeakSuppressionContext *suppressions = GetSuppressionContext();
-  uptr new_suppressions = false;
+  uptr new_suppressions = 0;
   for (uptr i = 0; i < leaks_.size(); i++) {
     if (suppressions->Suppress(leaks_[i].stack_trace_id, leaks_[i].hit_count,
                                leaks_[i].total_size)) {
diff --git a/compiler-rt/lib/memprof/memprof_allocator.cpp b/compiler-rt/lib/memprof/memprof_allocator.cpp
index c21e4e8a5694c..51c3a66ebd680 100644
--- a/compiler-rt/lib/memprof/memprof_allocator.cpp
+++ b/compiler-rt/lib/memprof/memprof_allocator.cpp
@@ -23,11 +23,11 @@
 #include "sanitizer_common/sanitizer_allocator_checks.h"
 #include "sanitizer_common/sanitizer_allocator_interface.h"
 #include "sanitizer_common/sanitizer_allocator_report.h"
+#include "sanitizer_common/sanitizer_common.h"
 #include "sanitizer_common/sanitizer_errno.h"
 #include "sanitizer_common/sanitizer_file.h"
 #include "sanitizer_common/sanitizer_flags.h"
 #include "sanitizer_common/sanitizer_internal_defs.h"
-#include "sanitizer_common/sanitizer_procmaps.h"
 #include "sanitizer_common/sanitizer_stackdepot.h"
 
 #include <sched.h>
@@ -75,7 +75,7 @@ static int GetCpuId(void) {
   // _memprof_preinit is called via the preinit_array, which subsequently calls
   // malloc. Since this is before _dl_init calls VDSO_SETUP, sched_getcpu
   // will seg fault as the address of __vdso_getcpu will be null.
-  if (!memprof_init_done)
+  if (!memprof_inited)
     return -1;
   return sched_getcpu();
 }
@@ -295,8 +295,10 @@ struct Allocator {
       // memprof_rawprofile.h.
       char *Buffer = nullptr;
 
-      MemoryMappingLayout Layout(/*cache_enabled=*/true);
-      u64 BytesSerialized = SerializeToRawProfile(MIBMap, Layout, Buffer);
+      __sanitizer::ListOfModules List;
+      List.init();
+      ArrayRef<LoadedModule> Modules(List.begin(), List.end());
+      u64 BytesSerialized = SerializeToRawProfile(MIBMap, Modules, Buffer);
       CHECK(Buffer && BytesSerialized && "could not serialize to buffer");
       report_file.Write(Buffer, BytesSerialized);
     }
@@ -445,8 +447,7 @@ struct Allocator {
 
     u64 user_requested_size =
         atomic_exchange(&m->user_requested_size, 0, memory_order_acquire);
-    if (memprof_inited && memprof_init_done &&
-        atomic_load_relaxed(&constructed) &&
+    if (memprof_inited && atomic_load_relaxed(&constructed) &&
         !atomic_load_relaxed(&destructing)) {
       u64 c = GetShadowCount(p, user_requested_size);
       long curtime = GetTimestamp();
diff --git a/compiler-rt/lib/memprof/memprof_internal.h b/compiler-rt/lib/memprof/memprof_internal.h
index bba465e60d826..990e62ce1a55d 100644
--- a/compiler-rt/lib/memprof/memprof_internal.h
+++ b/compiler-rt/lib/memprof/memprof_internal.h
@@ -76,7 +76,6 @@ void *MemprofDlSymNext(const char *sym);
 
 extern int memprof_inited;
 extern int memprof_timestamp_inited;
-extern int memprof_init_done;
 // Used to avoid infinite recursion in __memprof_init().
 extern bool memprof_init_is_running;
 extern void (*death_callback)(void);
diff --git a/compiler-rt/lib/memprof/memprof_rawprofile.cpp b/compiler-rt/lib/memprof/memprof_rawprofile.cpp
index 88f3c34f85549..1a65fbbf6b1e2 100644
--- a/compiler-rt/lib/memprof/memprof_rawprofile.cpp
+++ b/compiler-rt/lib/memprof/memprof_rawprofile.cpp
@@ -33,12 +33,14 @@ void RecordStackId(const uptr Key, UNUSED LockedMemInfoBlock *const &MIB,
 }
 } // namespace
 
-u64 SegmentSizeBytes(MemoryMappingLayoutBase &Layout) {
+u64 SegmentSizeBytes(ArrayRef<LoadedModule> Modules) {
   u64 NumSegmentsToRecord = 0;
-  MemoryMappedSegment segment;
-  for (Layout.Reset(); Layout.Next(&segment);)
-    if (segment.IsReadable() && segment.IsExecutable())
-      NumSegmentsToRecord++;
+  for (const auto &Module : Modules) {
+    for (const auto &Segment : Module.ranges()) {
+      if (Segment.executable)
+        NumSegmentsToRecord++;
+    }
+  }
 
   return sizeof(u64) // A header which stores the number of records.
          + sizeof(SegmentEntry) * NumSegmentsToRecord;
@@ -51,28 +53,31 @@ u64 SegmentSizeBytes(MemoryMappingLayoutBase &Layout) {
 // Start
 // End
 // Offset
-// BuildID 32B
+// UuidSize
+// Uuid 32B
 // ----------
 // ...
-void SerializeSegmentsToBuffer(MemoryMappingLayoutBase &Layout,
+void SerializeSegmentsToBuffer(ArrayRef<LoadedModule> Modules,
                                const u64 ExpectedNumBytes, char *&Buffer) {
   char *Ptr = Buffer;
   // Reserve space for the final count.
   Ptr += sizeof(u64);
 
   u64 NumSegmentsRecorded = 0;
-  MemoryMappedSegment segment;
-
-  for (Layout.Reset(); Layout.Next(&segment);) {
-    if (segment.IsReadable() && segment.IsExecutable()) {
-      // TODO: Record segment.uuid when it is implemented for Linux-Elf.
-      SegmentEntry Entry(segment.start, segment.end, segment.offset);
-      memcpy(Ptr, &Entry, sizeof(SegmentEntry));
-      Ptr += sizeof(SegmentEntry);
-      NumSegmentsRecorded++;
+
+  for (const auto &Module : Modules) {
+    for (const auto &Segment : Module.ranges()) {
+      if (Segment.executable) {
+        SegmentEntry Entry(Segment.beg, Segment.end, Module.base_address());
+        CHECK(Module.uuid_size() <= MEMPROF_BUILDID_MAX_SIZE);
+        Entry.BuildIdSize = Module.uuid_size();
+        memcpy(Entry.BuildId, Module.uuid(), Module.uuid_size());
+        memcpy(Ptr, &Entry, sizeof(SegmentEntry));
+        Ptr += sizeof(SegmentEntry);
+        NumSegmentsRecorded++;
+      }
     }
   }
-
   // Store the number of segments we recorded in the space we reserved.
   *((u64 *)Buffer) = NumSegmentsRecorded;
   CHECK(ExpectedNumBytes >= static_cast<u64>(Ptr - Buffer) &&
@@ -198,11 +203,11 @@ void SerializeMIBInfoToBuffer(MIBMapTy &MIBMap, const Vector<u64> &StackIds,
 // ----------
 // Optional Padding Bytes
 // ...
-u64 SerializeToRawProfile(MIBMapTy &MIBMap, MemoryMappingLayoutBase &Layout,
+u64 SerializeToRawProfile(MIBMapTy &MIBMap, ArrayRef<LoadedModule> Modules,
                           char *&Buffer) {
   // Each section size is rounded up to 8b since the first entry in each section
   // is a u64 which holds the number of entries in the section by convention.
-  const u64 NumSegmentBytes = RoundUpTo(SegmentSizeBytes(Layout), 8);
+  const u64 NumSegmentBytes = RoundUpTo(SegmentSizeBytes(Modules), 8);
 
   Vector<u64> StackIds;
   MIBMap.ForEach(RecordStackId, reinterpret_cast<void *>(&StackIds));
@@ -232,7 +237,7 @@ u64 SerializeToRawProfile(MIBMapTy &MIBMap, MemoryMappingLayoutBase &Layout,
                 sizeof(Header) + NumSegmentBytes + NumMIBInfoBytes};
   Ptr = WriteBytes(header, Ptr);
 
-  SerializeSegmentsToBuffer(Layout, NumSegmentBytes, Ptr);
+  SerializeSegmentsToBuffer(Modules, NumSegmentBytes, Ptr);
   Ptr += NumSegmentBytes;
 
   SerializeMIBInfoToBuffer(MIBMap, StackIds, NumMIBInfoBytes, Ptr);
diff --git a/compiler-rt/lib/memprof/memprof_rawprofile.h b/compiler-rt/lib/memprof/memprof_rawprofile.h
index 575104e7e34eb..a4e9fd4232b5d 100644
--- a/compiler-rt/lib/memprof/memprof_rawprofile.h
+++ b/compiler-rt/lib/memprof/memprof_rawprofile.h
@@ -2,12 +2,12 @@
 #define MEMPROF_RAWPROFILE_H_
 
 #include "memprof_mibmap.h"
-#include "sanitizer_common/sanitizer_procmaps.h"
+#include "sanitizer_common/sanitizer_common.h"
 
 namespace __memprof {
 // Serialize the in-memory representation of the memprof profile to the raw
 // binary format. The format itself is documented memprof_rawprofile.cpp.
-u64 SerializeToRawProfile(MIBMapTy &BlockCache, MemoryMappingLayoutBase &Layout,
+u64 SerializeToRawProfile(MIBMapTy &BlockCache, ArrayRef<LoadedModule> Modules,
                           char *&Buffer);
 } // namespace __memprof
 
diff --git a/compiler-rt/lib/memprof/memprof_rtl.cpp b/compiler-rt/lib/memprof/memprof_rtl.cpp
index d30b80304f6bc..5e2e7bc2be3f7 100644
--- a/compiler-rt/lib/memprof/memprof_rtl.cpp
+++ b/compiler-rt/lib/memprof/memprof_rtl.cpp
@@ -65,7 +65,6 @@ static void CheckUnwind() {
 
 // -------------------------- Globals --------------------- {{{1
 int memprof_inited;
-int memprof_init_done;
 bool memprof_init_is_running;
 int memprof_timestamp_inited;
 long memprof_init_timestamp_s;
@@ -195,11 +194,6 @@ static void MemprofInitInternal() {
 
   InitializeAllocator();
 
-  // On Linux MemprofThread::ThreadStart() calls malloc() that's why
-  // memprof_inited should be set to 1 prior to initializing the threads.
-  memprof_inited = 1;
-  memprof_init_is_running = false;
-
   if (flags()->atexit)
     Atexit(memprof_atexit);
 
@@ -218,7 +212,8 @@ static void MemprofInitInternal() {
 
   VReport(1, "MemProfiler Init done\n");
 
-  memprof_init_done = 1;
+  memprof_init_is_running = false;
+  memprof_inited = 1;
 }
 
 void MemprofInitTime() {
diff --git a/compiler-rt/lib/memprof/tests/rawprofile.cpp b/compiler-rt/lib/memprof/tests/rawprofile.cpp
index 7f6398d1cf86e..3c7372afaedaa 100644
--- a/compiler-rt/lib/memprof/tests/rawprofile.cpp
+++ b/compiler-rt/lib/memprof/tests/rawprofile.cpp
@@ -15,22 +15,9 @@ namespace {
 
 using ::__memprof::MIBMapTy;
 using ::__memprof::SerializeToRawProfile;
-using ::__sanitizer::MemoryMappedSegment;
-using ::__sanitizer::MemoryMappingLayoutBase;
 using ::__sanitizer::StackDepotPut;
 using ::__sanitizer::StackTrace;
 using ::llvm::memprof::MemInfoBlock;
-using ::testing::_;
-using ::testing::Action;
-using ::testing::DoAll;
-using ::testing::Return;
-using ::testing::SetArgPointee;
-
-class MockMemoryMappingLayout final : public MemoryMappingLayoutBase {
-public:
-  MOCK_METHOD(bool, Next, (MemoryMappedSegment *), (override));
-  MOCK_METHOD(void, Reset, (), (override));
-};
 
 uint64_t PopulateFakeMap(const MemInfoBlock &FakeMIB, uint64_t StackPCBegin,
                          MIBMapTy &FakeMap) {
@@ -56,26 +43,13 @@ template <class T = uint64_t> T Read(char *&Buffer) {
 }
 
 TEST(MemProf, Basic) {
-  MockMemoryMappingLayout Layout;
-  MemoryMappedSegment FakeSegment;
-  memset(&FakeSegment, 0, sizeof(FakeSegment));
-  FakeSegment.start = 0x10;
-  FakeSegment.end = 0x20;
-  FakeSegment.offset = 0x10;
-  uint8_t uuid[__sanitizer::kModuleUUIDSize] = {0xC, 0x0, 0xF, 0xF, 0xE, 0xE};
-  memcpy(FakeSegment.uuid, uuid, __sanitizer::kModuleUUIDSize);
-  FakeSegment.protection =
-      __sanitizer::kProtectionExecute | __sanitizer::kProtectionRead;
-
-  const Action<bool(MemoryMappedSegment *)> SetSegment =
-      DoAll(SetArgPointee<0>(FakeSegment), Return(true));
-  EXPECT_CALL(Layout, Next(_))
-      .WillOnce(SetSegment)
-      .WillOnce(Return(false))
-      .WillOnce(SetSegment)
-      .WillRepeatedly(Return(false));
-
-  EXPECT_CALL(Layout, Reset).Times(2);
+  __sanitizer::LoadedModule FakeModule;
+  FakeModule.addAddressRange(/*begin=*/0x10, /*end=*/0x20, /*executable=*/true,
+                             /*writable=*/false, /*name=*/"");
+  const char uuid[MEMPROF_BUILDID_MAX_SIZE] = {0xC, 0x0, 0xF, 0xF, 0xE, 0xE};
+  FakeModule.setUuid(uuid, MEMPROF_BUILDID_MAX_SIZE);
+  __sanitizer::ArrayRef<__sanitizer::LoadedModule> Modules(&FakeModule,
+                                                           (&FakeModule) + 1);
 
   MIBMapTy FakeMap;
   MemInfoBlock FakeMIB;
@@ -90,7 +64,7 @@ TEST(MemProf, Basic) {
   FakeIds[1] = PopulateFakeMap(FakeMIB, /*StackPCBegin=*/3, FakeMap);
 
   char *Ptr = nullptr;
-  uint64_t NumBytes = SerializeToRawProfile(FakeMap, Layout, Ptr);
+  uint64_t NumBytes = SerializeToRawProfile(FakeMap, Modules, Ptr);
   const char *Buffer = Ptr;
 
   ASSERT_GT(NumBytes, 0ULL);
@@ -111,16 +85,16 @@ TEST(MemProf, Basic) {
   // Should be equal to the size of the raw profile header.
   EXPECT_EQ(SegmentOffset, 48ULL);
 
-  // We expect only 1 segment entry, 8b for the count and 56b for SegmentEntry
+  // We expect only 1 segment entry, 8b for the count and 64b for SegmentEntry
   // in memprof_rawprofile.cpp.
-  EXPECT_EQ(MIBOffset - SegmentOffset, 64ULL);
+  EXPECT_EQ(MIBOffset - SegmentOffset, 72ULL);
 
-  EXPECT_EQ(MIBOffset, 112ULL);
+  EXPECT_EQ(MIBOffset, 120ULL);
   // We expect 2 mib entry, 8b for the count and sizeof(uint64_t) +
   // sizeof(MemInfoBlock) contains stack id + MeminfoBlock.
   EXPECT_EQ(StackOffset - MIBOffset, 8 + 2 * (8 + sizeof(MemInfoBlock)));
 
-  EXPECT_EQ(StackOffset, 336ULL);
+  EXPECT_EQ(StackOffset, 408ULL);
   // We expect 2 stack entries, with 5 frames - 8b for total count,
   // 2 * (8b for id, 8b for frame count and 5*8b for fake frames).
   // Since this is the last section, there may be additional padding at the end
@@ -128,16 +102,15 @@ TEST(MemProf, Basic) {
   EXPECT_GE(TotalSize - StackOffset, 8ULL + 2 * (8 + 8 + 5 * 8));
 
   // ============= Check contents.
-  // The Uuid field is not yet populated on Linux-Elf by the sanitizer procmaps
-  // library, so we expect it to be filled with 0 for now.
-  unsigned char ExpectedSegmentBytes[64] = {
-      0x01, 0, 0, 0, 0, 0, 0, 0, // Number of entries
-      0x10, 0, 0, 0, 0, 0, 0, 0, // Start
-      0x20, 0, 0, 0, 0, 0, 0, 0, // End
-      0x10, 0, 0, 0, 0, 0, 0, 0, // Offset
-      0x0,                       // Uuid
+  unsigned char ExpectedSegmentBytes[72] = {
+      0x01, 0,   0,   0,   0,   0,  0, 0, // Number of entries
+      0x10, 0,   0,   0,   0,   0,  0, 0, // Start
+      0x20, 0,   0,   0,   0,   0,  0, 0, // End
+      0x0,  0,   0,   0,   0,   0,  0, 0, // Offset
+      0x20, 0,   0,   0,   0,   0,  0, 0, // UuidSize
+      0xC,  0x0, 0xF, 0xF, 0xE, 0xE       // Uuid
   };
-  EXPECT_EQ(memcmp(Buffer + SegmentOffset, ExpectedSegmentBytes, 64), 0);
+  EXPECT_EQ(memcmp(Buffer + SegmentOffset, ExpectedSegmentBytes, 72), 0);
 
   // Check that the number of entries is 2.
   EXPECT_EQ(*reinterpret_cast<const uint64_t *>(Buffer + MIBOffset), 2ULL);
@@ -193,5 +166,4 @@ TEST(MemProf, Basic) {
                    sizeof(ExpectedStackBytes[1])),
             0);
 }
-
 } // namespace
diff --git a/compiler-rt/lib/profile/CMakeLists.txt b/compiler-rt/lib/profile/CMakeLists.txt
index 5ddb7bc5add71..45e5164891751 100644
--- a/compiler-rt/lib/profile/CMakeLists.txt
+++ b/compiler-rt/lib/profile/CMakeLists.txt
@@ -60,6 +60,7 @@ set(PROFILE_SOURCES
   InstrProfilingNameVar.c
   InstrProfilingVersionVar.c
   InstrProfilingWriter.c
+  InstrProfilingPlatformAIX.c
   InstrProfilingPlatformDarwin.c
   InstrProfilingPlatformFuchsia.c
   InstrProfilingPlatformLinux.c
diff --git a/compiler-rt/lib/profile/InstrProfilingPlatformAIX.c b/compiler-rt/lib/profile/InstrProfilingPlatformAIX.c
new file mode 100644
index 0000000000000..1035894b8ed1c
--- /dev/null
+++ b/compiler-rt/lib/profile/InstrProfilingPlatformAIX.c
@@ -0,0 +1,48 @@
+/*===- InstrProfilingPlatformAIX.c - Profile data AIX platform ------------===*\
+|*
+|* Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+|* See https://llvm.org/LICENSE.txt for license information.
+|* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+|*
+\*===----------------------------------------------------------------------===*/
+
+#if defined(_AIX)
+
+#include "InstrProfiling.h"
+
+// Empty stubs to allow linking object files using the registration-based scheme
+COMPILER_RT_VISIBILITY
+void __llvm_profile_register_function(void *Data_) {}
+
+COMPILER_RT_VISIBILITY
+void __llvm_profile_register_names_function(void *NamesStart,
+                                            uint64_t NamesSize) {}
+
+// The __start_SECNAME and __stop_SECNAME symbols (for SECNAME \in
+// {"__llvm_prf_cnts", "__llvm_prf_data", "__llvm_prf_name", "__llvm_prf_vnds"})
+// are always live when linking on AIX, regardless if the .o's being linked
+// reference symbols from the profile library (for example when no files were
+// compiled with -fprofile-generate). That's because these symbols are kept
+// alive through references in constructor functions that are always live in the
+// default linking model on AIX (-bcdtors:all). The __start_SECNAME and
+// __stop_SECNAME symbols are only resolved by the linker when the SECNAME
+// section exists. So for the scenario where the user objects have no such
+// section (i.e. when they are compiled with -fno-profile-generate), we always
+// define these zero length variables in each of the above 4 sections.
+static int dummy_cnts[0] COMPILER_RT_SECTION(
+    COMPILER_RT_SEG INSTR_PROF_CNTS_SECT_NAME);
+static int dummy_data[0] COMPILER_RT_SECTION(
+    COMPILER_RT_SEG INSTR_PROF_DATA_SECT_NAME);
+static const int dummy_name[0] COMPILER_RT_SECTION(
+    COMPILER_RT_SEG INSTR_PROF_NAME_SECT_NAME);
+static int dummy_vnds[0] COMPILER_RT_SECTION(
+    COMPILER_RT_SEG INSTR_PROF_VNODES_SECT_NAME);
+
+// To avoid GC'ing of the dummy variables by the linker, reference them in an
+// array and reference the array in the runtime registration code
+// (InstrProfilingRuntime.cpp)
+COMPILER_RT_VISIBILITY
+void *__llvm_profile_keep[] = {(void *)&dummy_cnts, (void *)&dummy_data,
+                               (void *)&dummy_name, (void *)&dummy_vnds};
+
+#endif
diff --git a/compiler-rt/lib/profile/InstrProfilingPlatformLinux.c b/compiler-rt/lib/profile/InstrProfilingPlatformLinux.c
index adf4132c6b4d6..f17e543ddb5d8 100644
--- a/compiler-rt/lib/profile/InstrProfilingPlatformLinux.c
+++ b/compiler-rt/lib/profile/InstrProfilingPlatformLinux.c
@@ -230,41 +230,4 @@ COMPILER_RT_VISIBILITY int __llvm_write_binary_ids(ProfDataWriter *Writer) {
 }
 #endif
 
-#if defined(_AIX)
-// Empty stubs to allow linking object files using the registration-based scheme
-COMPILER_RT_VISIBILITY
-void __llvm_profile_register_function(void *Data_) {}
-
-COMPILER_RT_VISIBILITY
-void __llvm_profile_register_names_function(void *NamesStart,
-                                            uint64_t NamesSize) {}
-
-// The __start_SECNAME and __stop_SECNAME symbols (for SECNAME \in
-// {"__llvm_prf_cnts", "__llvm_prf_data", "__llvm_prf_name", "__llvm_prf_vnds"})
-// are always live when linking on AIX, regardless if the .o's being linked
-// reference symbols from the profile library (for example when no files were
-// compiled with -fprofile-generate). That's because these symbols are kept
-// alive through references in constructor functions that are always live in the
-// default linking model on AIX (-bcdtors:all). The __start_SECNAME and
-// __stop_SECNAME symbols are only resolved by the linker when the SECNAME
-// section exists. So for the scenario where the user objects have no such
-// section (i.e. when they are compiled with -fno-profile-generate), we always
-// define these zero length variables in each of the above 4 sections.
-static int dummy_cnts[0] COMPILER_RT_SECTION(
-    COMPILER_RT_SEG INSTR_PROF_CNTS_SECT_NAME);
-static int dummy_data[0] COMPILER_RT_SECTION(
-    COMPILER_RT_SEG INSTR_PROF_DATA_SECT_NAME);
-static const int dummy_name[0] COMPILER_RT_SECTION(
-    COMPILER_RT_SEG INSTR_PROF_NAME_SECT_NAME);
-static int dummy_vnds[0] COMPILER_RT_SECTION(
-    COMPILER_RT_SEG INSTR_PROF_VNODES_SECT_NAME);
-
-// To avoid GC'ing of the dummy variables by the linker, reference them in an
-// array and reference the array in the runtime registration code
-// (InstrProfilingRuntime.cpp)
-COMPILER_RT_VISIBILITY
-void *__llvm_profile_keep[] = {(void *)&dummy_cnts, (void *)&dummy_data,
-                               (void *)&dummy_name, (void *)&dummy_vnds};
-#endif
-
 #endif
diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_common.cpp b/compiler-rt/lib/sanitizer_common/sanitizer_common.cpp
index 82236453157fa..79b7748b8f6e8 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_common.cpp
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_common.cpp
@@ -61,6 +61,26 @@ void NORETURN ReportMmapFailureAndDie(uptr size, const char *mem_type,
   UNREACHABLE("unable to mmap");
 }
 
+void NORETURN ReportMunmapFailureAndDie(void *addr, uptr size, error_t err,
+                                        bool raw_report) {
+  static int recursion_count;
+  if (raw_report || recursion_count) {
+    // If raw report is requested or we went into recursion just die.  The
+    // Report() and CHECK calls below may call munmap recursively and fail.
+    RawWrite("ERROR: Failed to munmap\n");
+    Die();
+  }
+  recursion_count++;
+  Report(
+      "ERROR: %s failed to deallocate 0x%zx (%zd) bytes at address %p (error "
+      "code: %d)\n",
+      SanitizerToolName, size, size, addr, err);
+#if !SANITIZER_GO
+  DumpProcessMap();
+#endif
+  UNREACHABLE("unable to unmmap");
+}
+
 typedef bool UptrComparisonFunction(const uptr &a, const uptr &b);
 typedef bool U32ComparisonFunction(const u32 &a, const u32 &b);
 
diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_common.h b/compiler-rt/lib/sanitizer_common/sanitizer_common.h
index b462e388c232e..72b37941d9b76 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_common.h
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_common.h
@@ -315,6 +315,8 @@ CheckFailed(const char *file, int line, const char *cond, u64 v1, u64 v2);
 void NORETURN ReportMmapFailureAndDie(uptr size, const char *mem_type,
                                       const char *mmap_type, error_t err,
                                       bool raw_report = false);
+void NORETURN ReportMunmapFailureAndDie(void *ptr, uptr size, error_t err,
+                                        bool raw_report = false);
 
 // Returns true if the platform-specific error reported is an OOM error.
 bool ErrorIsOOM(error_t err);
diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc b/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc
index 9fc56238a64fc..51703a34f0e9b 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc
@@ -10375,6 +10375,25 @@ INTERCEPTOR(void, hexdump, const void *ptr, int length, const char *header, int
 #define INIT_HEXDUMP
 #endif
 
+#if SANITIZER_INTERCEPT_ARGP_PARSE
+INTERCEPTOR(int, argp_parse, const struct argp *argp, int argc, char **argv,
+            unsigned flags, int *arg_index, void *input) {
+  void *ctx;
+  COMMON_INTERCEPTOR_ENTER(ctx, argp_parse, argp, argc, argv, flags, arg_index,
+                           input);
+  for (int i = 0; i < argc; i++)
+    COMMON_INTERCEPTOR_READ_RANGE(ctx, argv[i], internal_strlen(argv[i]) + 1);
+  int res = REAL(argp_parse)(argp, argc, argv, flags, arg_index, input);
+  if (!res && arg_index)
+    COMMON_INTERCEPTOR_WRITE_RANGE(ctx, arg_index, sizeof(int));
+  return res;
+}
+
+#define INIT_ARGP_PARSE COMMON_INTERCEPT_FUNCTION(argp_parse);
+#else
+#define INIT_ARGP_PARSE
+#endif
+
 #include "sanitizer_common_interceptors_netbsd_compat.inc"
 
 static void InitializeCommonInterceptors() {
@@ -10694,6 +10713,7 @@ static void InitializeCommonInterceptors() {
   INIT_UNAME;
   INIT___XUNAME;
   INIT_HEXDUMP;
+  INIT_ARGP_PARSE;
 
   INIT___PRINTF_CHK;
 }
diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp b/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp
index ec860fdc4ff9d..37b2b57c0c844 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp
@@ -583,6 +583,7 @@ static void GetTls(uptr *addr, uptr *size) {
       *addr = (uptr)tcb->tcb_dtv[1];
     }
   }
+#else
 #error "Unknown OS"
 #endif
 }
diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_mac.cpp b/compiler-rt/lib/sanitizer_common/sanitizer_mac.cpp
index 23c4c6619de82..fa04fdad18e35 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_mac.cpp
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_mac.cpp
@@ -989,7 +989,7 @@ static void VerifyInterceptorsWorking() {
   // "wrap_puts" within our own dylib.
   Dl_info info_puts, info_runtime;
   RAW_CHECK(dladdr(dlsym(RTLD_DEFAULT, "puts"), &info_puts));
-  RAW_CHECK(dladdr((void *)__sanitizer_report_error_summary, &info_runtime));
+  RAW_CHECK(dladdr((void *)&VerifyInterceptorsWorking, &info_runtime));
   if (internal_strcmp(info_puts.dli_fname, info_runtime.dli_fname) != 0) {
     Report(
         "ERROR: Interceptors are not working. This may be because %s is "
@@ -1039,7 +1039,7 @@ static void StripEnv() {
     return;
 
   Dl_info info;
-  RAW_CHECK(dladdr((void *)__sanitizer_report_error_summary, &info));
+  RAW_CHECK(dladdr((void *)&StripEnv, &info));
   const char *dylib_name = StripModuleName(info.dli_fname);
   bool lib_is_in_env = internal_strstr(dyld_insert_libraries, dylib_name);
   if (!lib_is_in_env)
diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_platform_interceptors.h b/compiler-rt/lib/sanitizer_common/sanitizer_platform_interceptors.h
index 814ff462d1cf5..eb39fabfd5983 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_platform_interceptors.h
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_platform_interceptors.h
@@ -592,6 +592,7 @@
 #define SANITIZER_INTERCEPT_FLOPEN SI_FREEBSD
 #define SANITIZER_INTERCEPT_PROCCTL SI_FREEBSD
 #define SANITIZER_INTERCEPT_HEXDUMP SI_FREEBSD
+#define SANITIZER_INTERCEPT_ARGP_PARSE SI_GLIBC
 
 // This macro gives a way for downstream users to override the above
 // interceptor macros irrespective of the platform they are on. They have
diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_posix.cpp b/compiler-rt/lib/sanitizer_common/sanitizer_posix.cpp
index 75968ad33ccf5..f6b0bbd3a5db7 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_posix.cpp
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_posix.cpp
@@ -57,11 +57,9 @@ void *MmapOrDie(uptr size, const char *mem_type, bool raw_report) {
 void UnmapOrDie(void *addr, uptr size) {
   if (!addr || !size) return;
   uptr res = internal_munmap(addr, size);
-  if (UNLIKELY(internal_iserror(res))) {
-    Report("ERROR: %s failed to deallocate 0x%zx (%zd) bytes at address %p\n",
-           SanitizerToolName, size, size, addr);
-    CHECK("unable to unmap" && 0);
-  }
+  int reserrno;
+  if (UNLIKELY(internal_iserror(res, &reserrno)))
+    ReportMunmapFailureAndDie(addr, size, reserrno);
   DecreaseTotalMmap(size);
 }
 
diff --git a/compiler-rt/lib/scudo/standalone/combined.h b/compiler-rt/lib/scudo/standalone/combined.h
index 52dbf6b145264..250eba0f4ddad 100644
--- a/compiler-rt/lib/scudo/standalone/combined.h
+++ b/compiler-rt/lib/scudo/standalone/combined.h
@@ -745,9 +745,9 @@ class Allocator {
     Str.output();
   }
 
-  void releaseToOS() {
+  void releaseToOS(ReleaseToOS ReleaseType) {
     initThreadMaybe();
-    Primary.releaseToOS();
+    Primary.releaseToOS(ReleaseType);
     Secondary.releaseToOS();
   }
 
diff --git a/compiler-rt/lib/scudo/standalone/common.h b/compiler-rt/lib/scudo/standalone/common.h
index aa15e9e787e2e..82e6cf4aee618 100644
--- a/compiler-rt/lib/scudo/standalone/common.h
+++ b/compiler-rt/lib/scudo/standalone/common.h
@@ -147,6 +147,9 @@ const char *getEnv(const char *Name);
 uptr GetRSS();
 
 u64 getMonotonicTime();
+// Gets the time faster but with less accuracy. Can call getMonotonicTime
+// if no fast version is available.
+u64 getMonotonicTimeFast();
 
 u32 getThreadID();
 
@@ -212,6 +215,13 @@ enum class Option : u8 {
   MaxTSDsCount,         // Number of usable TSDs for the shared registry.
 };
 
+enum class ReleaseToOS : u8 {
+  Normal, // Follow the normal rules for releasing pages to the OS
+  Force,  // Force release pages to the OS, but avoid cases that take too long.
+  ForceAll, // Force release every page possible regardless of how long it will
+            // take.
+};
+
 constexpr unsigned char PatternFillByte = 0xAB;
 
 enum FillContentsMode {
diff --git a/compiler-rt/lib/scudo/standalone/fuchsia.cpp b/compiler-rt/lib/scudo/standalone/fuchsia.cpp
index da684e7f1de0c..ef93542ef62c3 100644
--- a/compiler-rt/lib/scudo/standalone/fuchsia.cpp
+++ b/compiler-rt/lib/scudo/standalone/fuchsia.cpp
@@ -166,6 +166,8 @@ void setMemoryPermission(UNUSED uptr Addr, UNUSED uptr Size, UNUSED uptr Flags,
 
 void releasePagesToOS(UNUSED uptr BaseAddress, uptr Offset, uptr Size,
                       MapPlatformData *Data) {
+  // TODO: DCHECK the BaseAddress is consistent with the data in
+  // MapPlatformData.
   DCHECK(Data);
   DCHECK_NE(Data->Vmar, ZX_HANDLE_INVALID);
   DCHECK_NE(Data->Vmo, ZX_HANDLE_INVALID);
@@ -198,6 +200,7 @@ void HybridMutex::unlock() __TA_NO_THREAD_SAFETY_ANALYSIS {
 void HybridMutex::assertHeldImpl() __TA_NO_THREAD_SAFETY_ANALYSIS {}
 
 u64 getMonotonicTime() { return _zx_clock_get_monotonic(); }
+u64 getMonotonicTimeFast() { return _zx_clock_get_monotonic(); }
 
 u32 getNumberOfCPUs() { return _zx_system_get_num_cpus(); }
 
diff --git a/compiler-rt/lib/scudo/standalone/include/scudo/interface.h b/compiler-rt/lib/scudo/standalone/include/scudo/interface.h
index 23bcfba3982ac..3c083ed7f9da2 100644
--- a/compiler-rt/lib/scudo/standalone/include/scudo/interface.h
+++ b/compiler-rt/lib/scudo/standalone/include/scudo/interface.h
@@ -118,6 +118,10 @@ size_t __scudo_get_ring_buffer_size(void);
 #define M_PURGE -101
 #endif
 
+#ifndef M_PURGE_ALL
+#define M_PURGE_ALL -104
+#endif
+
 // Tune the allocator's choice of memory tags to make it more likely that
 // a certain class of memory errors will be detected. The value argument should
 // be one of the M_MEMTAG_TUNING_* constants below.
diff --git a/compiler-rt/lib/scudo/standalone/linux.cpp b/compiler-rt/lib/scudo/standalone/linux.cpp
index 33757e292f25b..e285d8a3d2d23 100644
--- a/compiler-rt/lib/scudo/standalone/linux.cpp
+++ b/compiler-rt/lib/scudo/standalone/linux.cpp
@@ -140,6 +140,17 @@ u64 getMonotonicTime() {
          static_cast<u64>(TS.tv_nsec);
 }
 
+u64 getMonotonicTimeFast() {
+#if defined(CLOCK_MONOTONIC_COARSE)
+  timespec TS;
+  clock_gettime(CLOCK_MONOTONIC_COARSE, &TS);
+  return static_cast<u64>(TS.tv_sec) * (1000ULL * 1000 * 1000) +
+         static_cast<u64>(TS.tv_nsec);
+#else
+  return getMonotonicTime();
+#endif
+}
+
 u32 getNumberOfCPUs() {
   cpu_set_t CPUs;
   // sched_getaffinity can fail for a variety of legitimate reasons (lack of
diff --git a/compiler-rt/lib/scudo/standalone/local_cache.h b/compiler-rt/lib/scudo/standalone/local_cache.h
index 6e84158659ae9..321d4f5bf8144 100644
--- a/compiler-rt/lib/scudo/standalone/local_cache.h
+++ b/compiler-rt/lib/scudo/standalone/local_cache.h
@@ -62,8 +62,8 @@ template <class SizeClassAllocator> struct SizeClassAllocatorLocalCache {
   struct BatchGroup {
     // `Next` is used by IntrusiveList.
     BatchGroup *Next;
-    // The identifier of each group
-    uptr GroupId;
+    // The compact base address of each group
+    uptr CompactPtrGroupBase;
     // Cache value of TransferBatch::getMaxCached()
     u16 MaxCachedPerBatch;
     // Number of blocks pushed into this group. This is an increment-only
diff --git a/compiler-rt/lib/scudo/standalone/primary32.h b/compiler-rt/lib/scudo/standalone/primary32.h
index df4053a6dde4e..3be689fb0e608 100644
--- a/compiler-rt/lib/scudo/standalone/primary32.h
+++ b/compiler-rt/lib/scudo/standalone/primary32.h
@@ -73,7 +73,7 @@ template <typename Config> class SizeClassAllocator32 {
     DCHECK(isAligned(reinterpret_cast<uptr>(this), alignof(ThisT)));
     PossibleRegions.init();
     u32 Seed;
-    const u64 Time = getMonotonicTime();
+    const u64 Time = getMonotonicTimeFast();
     if (!getRandom(reinterpret_cast<void *>(&Seed), sizeof(Seed)))
       Seed = static_cast<u32>(
           Time ^ (reinterpret_cast<uptr>(SizeClassInfoArray) >> 6));
@@ -122,11 +122,14 @@ template <typename Config> class SizeClassAllocator32 {
     return reinterpret_cast<void *>(static_cast<uptr>(CompactPtr));
   }
 
-  uptr compactPtrGroup(CompactPtrT CompactPtr) {
-    return CompactPtr >> GroupSizeLog;
+  uptr compactPtrGroupBase(CompactPtrT CompactPtr) {
+    const uptr Mask = (static_cast<uptr>(1) << GroupSizeLog) - 1;
+    return CompactPtr & ~Mask;
   }
 
-  uptr batchGroupBase(uptr GroupId) { return GroupId << GroupSizeLog; }
+  uptr decompactGroupBase(uptr CompactPtrGroupBase) {
+    return CompactPtrGroupBase;
+  }
 
   TransferBatch *popBatch(CacheT *C, uptr ClassId) {
     DCHECK_LT(ClassId, NumClasses);
@@ -172,11 +175,12 @@ template <typename Config> class SizeClassAllocator32 {
     // together.
     bool SameGroup = true;
     for (u32 I = 1; I < Size; ++I) {
-      if (compactPtrGroup(Array[I - 1]) != compactPtrGroup(Array[I]))
+      if (compactPtrGroupBase(Array[I - 1]) != compactPtrGroupBase(Array[I]))
         SameGroup = false;
       CompactPtrT Cur = Array[I];
       u32 J = I;
-      while (J > 0 && compactPtrGroup(Cur) < compactPtrGroup(Array[J - 1])) {
+      while (J > 0 &&
+             compactPtrGroupBase(Cur) < compactPtrGroupBase(Array[J - 1])) {
         Array[J] = Array[J - 1];
         --J;
       }
@@ -277,14 +281,14 @@ template <typename Config> class SizeClassAllocator32 {
     return true;
   }
 
-  uptr releaseToOS() {
+  uptr releaseToOS(ReleaseToOS ReleaseType) {
     uptr TotalReleasedBytes = 0;
     for (uptr I = 0; I < NumClasses; I++) {
       if (I == SizeClassMap::BatchClassId)
         continue;
       SizeClassInfo *Sci = getSizeClassInfo(I);
       ScopedLock L(Sci->Mutex);
-      TotalReleasedBytes += releaseToOSMaybe(Sci, I, /*Force=*/true);
+      TotalReleasedBytes += releaseToOSMaybe(Sci, I, ReleaseType);
     }
     return TotalReleasedBytes;
   }
@@ -363,6 +367,11 @@ template <typename Config> class SizeClassAllocator32 {
     const uptr End = Region + MapSize;
     if (End != MapEnd)
       unmap(reinterpret_cast<void *>(End), MapEnd - End);
+
+    DCHECK_EQ(Region % RegionSize, 0U);
+    static_assert(Config::PrimaryRegionSizeLog == GroupSizeLog,
+                  "Memory group should be the same size as Region");
+
     return Region;
   }
 
@@ -417,7 +426,7 @@ template <typename Config> class SizeClassAllocator32 {
       REQUIRES(Sci->Mutex) {
     DCHECK_GT(Size, 0U);
 
-    auto CreateGroup = [&](uptr GroupId) {
+    auto CreateGroup = [&](uptr CompactPtrGroupBase) {
       BatchGroup *BG = nullptr;
       TransferBatch *TB = nullptr;
       if (ClassId == SizeClassMap::BatchClassId) {
@@ -476,7 +485,7 @@ template <typename Config> class SizeClassAllocator32 {
         TB->clear();
       }
 
-      BG->GroupId = GroupId;
+      BG->CompactPtrGroupBase = CompactPtrGroupBase;
       // TODO(chiahungduan): Avoid the use of push_back() in `Batches`.
       BG->Batches.push_front(TB);
       BG->PushedBlocks = 0;
@@ -518,7 +527,7 @@ template <typename Config> class SizeClassAllocator32 {
     if (ClassId == SizeClassMap::BatchClassId) {
       if (Cur == nullptr) {
         // Don't need to classify BatchClassId.
-        Cur = CreateGroup(/*GroupId=*/0);
+        Cur = CreateGroup(/*CompactPtrGroupBase=*/0);
         Sci->FreeList.push_front(Cur);
       }
       InsertBlocks(Cur, Array, Size);
@@ -529,13 +538,15 @@ template <typename Config> class SizeClassAllocator32 {
     // will be pushed next. `Prev` is the element right before `Cur`.
     BatchGroup *Prev = nullptr;
 
-    while (Cur != nullptr && compactPtrGroup(Array[0]) > Cur->GroupId) {
+    while (Cur != nullptr &&
+           compactPtrGroupBase(Array[0]) > Cur->CompactPtrGroupBase) {
       Prev = Cur;
       Cur = Cur->Next;
     }
 
-    if (Cur == nullptr || compactPtrGroup(Array[0]) != Cur->GroupId) {
-      Cur = CreateGroup(compactPtrGroup(Array[0]));
+    if (Cur == nullptr ||
+        compactPtrGroupBase(Array[0]) != Cur->CompactPtrGroupBase) {
+      Cur = CreateGroup(compactPtrGroupBase(Array[0]));
       if (Prev == nullptr)
         Sci->FreeList.push_front(Cur);
       else
@@ -546,7 +557,7 @@ template <typename Config> class SizeClassAllocator32 {
     // id.
     if (SameGroup) {
       for (u32 I = 0; I < Size; ++I)
-        DCHECK_EQ(compactPtrGroup(Array[I]), Cur->GroupId);
+        DCHECK_EQ(compactPtrGroupBase(Array[I]), Cur->CompactPtrGroupBase);
 
       InsertBlocks(Cur, Array, Size);
       return;
@@ -556,17 +567,19 @@ template <typename Config> class SizeClassAllocator32 {
     // push them to their group together.
     u32 Count = 1;
     for (u32 I = 1; I < Size; ++I) {
-      if (compactPtrGroup(Array[I - 1]) != compactPtrGroup(Array[I])) {
-        DCHECK_EQ(compactPtrGroup(Array[I - 1]), Cur->GroupId);
+      if (compactPtrGroupBase(Array[I - 1]) != compactPtrGroupBase(Array[I])) {
+        DCHECK_EQ(compactPtrGroupBase(Array[I - 1]), Cur->CompactPtrGroupBase);
         InsertBlocks(Cur, Array + I - Count, Count);
 
-        while (Cur != nullptr && compactPtrGroup(Array[I]) > Cur->GroupId) {
+        while (Cur != nullptr &&
+               compactPtrGroupBase(Array[I]) > Cur->CompactPtrGroupBase) {
           Prev = Cur;
           Cur = Cur->Next;
         }
 
-        if (Cur == nullptr || compactPtrGroup(Array[I]) != Cur->GroupId) {
-          Cur = CreateGroup(compactPtrGroup(Array[I]));
+        if (Cur == nullptr ||
+            compactPtrGroupBase(Array[I]) != Cur->CompactPtrGroupBase) {
+          Cur = CreateGroup(compactPtrGroupBase(Array[I]));
           DCHECK_NE(Prev, nullptr);
           Sci->FreeList.insert(Prev, Cur);
         }
@@ -662,14 +675,14 @@ template <typename Config> class SizeClassAllocator32 {
 
     if (ClassId != SizeClassMap::BatchClassId) {
       u32 N = 1;
-      uptr CurGroup = compactPtrGroup(ShuffleArray[0]);
+      uptr CurGroup = compactPtrGroupBase(ShuffleArray[0]);
       for (u32 I = 1; I < NumberOfBlocks; I++) {
-        if (UNLIKELY(compactPtrGroup(ShuffleArray[I]) != CurGroup)) {
+        if (UNLIKELY(compactPtrGroupBase(ShuffleArray[I]) != CurGroup)) {
           shuffle(ShuffleArray + I - N, N, &Sci->RandState);
           pushBlocksImpl(C, ClassId, Sci, ShuffleArray + I - N, N,
                          /*SameGroup=*/true);
           N = 1;
-          CurGroup = compactPtrGroup(ShuffleArray[I]);
+          CurGroup = compactPtrGroupBase(ShuffleArray[I]);
         } else {
           ++N;
         }
@@ -714,7 +727,8 @@ template <typename Config> class SizeClassAllocator32 {
   }
 
   NOINLINE uptr releaseToOSMaybe(SizeClassInfo *Sci, uptr ClassId,
-                                 bool Force = false) REQUIRES(Sci->Mutex) {
+                                 ReleaseToOS ReleaseType = ReleaseToOS::Normal)
+      REQUIRES(Sci->Mutex) {
     const uptr BlockSize = getSizeByClassId(ClassId);
     const uptr PageSize = getPageSizeCached();
 
@@ -730,22 +744,24 @@ template <typename Config> class SizeClassAllocator32 {
     if (BytesPushed < PageSize)
       return 0; // Nothing new to release.
 
-    const bool CheckDensity = BlockSize < PageSize / 16U;
+    const bool CheckDensity =
+        BlockSize < PageSize / 16U && ReleaseType != ReleaseToOS::ForceAll;
     // Releasing smaller blocks is expensive, so we want to make sure that a
     // significant amount of bytes are free, and that there has been a good
     // amount of batches pushed to the freelist before attempting to release.
     if (CheckDensity) {
-      if (!Force && BytesPushed < Sci->AllocatedUser / 16U)
+      if (ReleaseType == ReleaseToOS::Normal &&
+          BytesPushed < Sci->AllocatedUser / 16U)
         return 0;
     }
 
-    if (!Force) {
+    if (ReleaseType == ReleaseToOS::Normal) {
       const s32 IntervalMs = atomic_load_relaxed(&ReleaseToOsIntervalMs);
       if (IntervalMs < 0)
         return 0;
       if (Sci->ReleaseInfo.LastReleaseAtNs +
               static_cast<u64>(IntervalMs) * 1000000 >
-          getMonotonicTime()) {
+          getMonotonicTimeFast()) {
         return 0; // Memory was returned recently.
       }
     }
@@ -758,11 +774,12 @@ template <typename Config> class SizeClassAllocator32 {
     const uptr Base = First * RegionSize;
     const uptr NumberOfRegions = Last - First + 1U;
     const uptr GroupSize = (1U << GroupSizeLog);
-    const uptr CurRegionGroupId =
-        compactPtrGroup(compactPtr(ClassId, Sci->CurrentRegion));
+    const uptr CurGroupBase =
+        compactPtrGroupBase(compactPtr(ClassId, Sci->CurrentRegion));
 
     ReleaseRecorder Recorder(Base);
-    PageReleaseContext Context(BlockSize, RegionSize, NumberOfRegions);
+    PageReleaseContext Context(BlockSize, NumberOfRegions,
+                               /*ReleaseSize=*/RegionSize);
 
     auto DecompactPtr = [](CompactPtrT CompactPtr) {
       return reinterpret_cast<uptr>(CompactPtr);
@@ -773,9 +790,13 @@ template <typename Config> class SizeClassAllocator32 {
       if (PushedBytesDelta * BlockSize < PageSize)
         continue;
 
-      uptr AllocatedGroupSize = BG.GroupId == CurRegionGroupId
+      const uptr GroupBase = decompactGroupBase(BG.CompactPtrGroupBase);
+      // The `GroupSize` may not be divided by `BlockSize`, which means there is
+      // an unused space at the end of Region. Exclude that space to avoid
+      // unused page map entry.
+      uptr AllocatedGroupSize = GroupBase == CurGroupBase
                                     ? Sci->CurrentRegionAllocated
-                                    : GroupSize;
+                                    : roundDownSlow(GroupSize, BlockSize);
       if (AllocatedGroupSize == 0)
         continue;
 
@@ -795,33 +816,25 @@ template <typename Config> class SizeClassAllocator32 {
       BG.PushedBlocksAtLastCheckpoint = BG.PushedBlocks;
 
       const uptr MaxContainedBlocks = AllocatedGroupSize / BlockSize;
-      // The first condition to do range marking is that all the blocks in the
-      // range need to be from the same region. In SizeClassAllocator32, this is
-      // true when GroupSize and RegionSize are the same. Another tricky case,
-      // while range marking, the last block in a region needs the logic to mark
-      // the last page. However, in SizeClassAllocator32, the RegionSize
-      // recorded in PageReleaseContext may be different from
-      // `CurrentRegionAllocated` of the current region. This exception excludes
-      // the chance of doing range marking for the current region.
-      const bool CanDoRangeMark =
-          GroupSize == RegionSize && BG.GroupId != CurRegionGroupId;
-
-      if (CanDoRangeMark && NumBlocks == MaxContainedBlocks) {
+      const uptr RegionIndex = (GroupBase - Base) / RegionSize;
+
+      if (NumBlocks == MaxContainedBlocks) {
         for (const auto &It : BG.Batches)
           for (u16 I = 0; I < It.getCount(); ++I)
-            DCHECK_EQ(compactPtrGroup(It.get(I)), BG.GroupId);
+            DCHECK_EQ(compactPtrGroupBase(It.get(I)), BG.CompactPtrGroupBase);
 
-        const uptr From = batchGroupBase(BG.GroupId);
-        const uptr To = batchGroupBase(BG.GroupId) + AllocatedGroupSize;
-        Context.markRangeAsAllCounted(From, To, Base);
+        const uptr To = GroupBase + AllocatedGroupSize;
+        Context.markRangeAsAllCounted(GroupBase, To, GroupBase, RegionIndex,
+                                      AllocatedGroupSize);
       } else {
-        if (CanDoRangeMark)
-          DCHECK_LT(NumBlocks, MaxContainedBlocks);
+        DCHECK_LT(NumBlocks, MaxContainedBlocks);
 
         // Note that we don't always visit blocks in each BatchGroup so that we
         // may miss the chance of releasing certain pages that cross
         // BatchGroups.
-        Context.markFreeBlocks(BG.Batches, DecompactPtr, Base);
+        Context.markFreeBlocksInRegion(BG.Batches, DecompactPtr, GroupBase,
+                                       RegionIndex, AllocatedGroupSize,
+                                       /*MayContainLastBlockInRegion=*/true);
       }
     }
 
@@ -840,7 +853,7 @@ template <typename Config> class SizeClassAllocator32 {
       Sci->ReleaseInfo.LastReleasedBytes = Recorder.getReleasedBytes();
       TotalReleasedBytes += Sci->ReleaseInfo.LastReleasedBytes;
     }
-    Sci->ReleaseInfo.LastReleaseAtNs = getMonotonicTime();
+    Sci->ReleaseInfo.LastReleaseAtNs = getMonotonicTimeFast();
 
     return TotalReleasedBytes;
   }
diff --git a/compiler-rt/lib/scudo/standalone/primary64.h b/compiler-rt/lib/scudo/standalone/primary64.h
index c0e50218fb1d9..1cb6d02f6cd6a 100644
--- a/compiler-rt/lib/scudo/standalone/primary64.h
+++ b/compiler-rt/lib/scudo/standalone/primary64.h
@@ -47,6 +47,7 @@ template <typename Config> class SizeClassAllocator64 {
   typedef typename Config::PrimaryCompactPtrT CompactPtrT;
   static const uptr CompactPtrScale = Config::PrimaryCompactPtrScale;
   static const uptr GroupSizeLog = Config::PrimaryGroupSizeLog;
+  static const uptr GroupScale = GroupSizeLog - CompactPtrScale;
   typedef typename Config::SizeClassMap SizeClassMap;
   typedef SizeClassAllocator64<Config> ThisT;
   typedef SizeClassAllocatorLocalCache<ThisT> CacheT;
@@ -64,12 +65,13 @@ template <typename Config> class SizeClassAllocator64 {
   void init(s32 ReleaseToOsInterval) NO_THREAD_SAFETY_ANALYSIS {
     DCHECK(isAligned(reinterpret_cast<uptr>(this), alignof(ThisT)));
     DCHECK_EQ(PrimaryBase, 0U);
+
     // Reserve the space required for the Primary.
     PrimaryBase = reinterpret_cast<uptr>(map(
         nullptr, PrimarySize, "scudo:primary_reserve", MAP_NOACCESS, &Data));
 
     u32 Seed;
-    const u64 Time = getMonotonicTime();
+    const u64 Time = getMonotonicTimeFast();
     if (!getRandom(reinterpret_cast<void *>(&Seed), sizeof(Seed)))
       Seed = static_cast<u32>(Time ^ (PrimaryBase >> 12));
     const uptr PageSize = getPageSizeCached();
@@ -77,14 +79,53 @@ template <typename Config> class SizeClassAllocator64 {
       RegionInfo *Region = getRegionInfo(I);
       // The actual start of a region is offset by a random number of pages
       // when PrimaryEnableRandomOffset is set.
-      Region->RegionBeg = getRegionBaseByClassId(I) +
+      Region->RegionBeg = (PrimaryBase + (I << Config::PrimaryRegionSizeLog)) +
                           (Config::PrimaryEnableRandomOffset
                                ? ((getRandomModN(&Seed, 16) + 1) * PageSize)
                                : 0);
       Region->RandState = getRandomU32(&Seed);
       Region->ReleaseInfo.LastReleaseAtNs = Time;
     }
+    shuffle(RegionInfoArray, NumClasses, &Seed);
+
     setOption(Option::ReleaseInterval, static_cast<sptr>(ReleaseToOsInterval));
+
+    const uptr GroupSize = (1U << GroupSizeLog);
+    const uptr PagesInGroup = GroupSize / PageSize;
+    const uptr MinSizeClass = getSizeByClassId(1);
+    // When trying to release pages back to memory, visiting smaller size
+    // classes is expensive. Therefore, we only try to release smaller size
+    // classes when the amount of free blocks goes over a certain threshold (See
+    // the comment in releaseToOSMaybe() for more details). For example, for
+    // size class 32, we only do the release when the size of free blocks is
+    // greater than 97% of pages in a group. However, this may introduce another
+    // issue that if the number of free blocks is bouncing between 97% ~ 100%.
+    // Which means we may try many page releases but only release very few of
+    // them (less than 3% in a group). Even though we have
+    // `&ReleaseToOsIntervalMs` which slightly reduce the frequency of these
+    // calls but it will be better to have another guard to mitigate this issue.
+    //
+    // Here we add another constraint on the minimum size requirement. The
+    // constraint is determined by the size of in-use blocks in the minimal size
+    // class. Take size class 32 as an example,
+    //
+    //   +-     one memory group      -+
+    //   +----------------------+------+
+    //   |  97% of free blocks  |      |
+    //   +----------------------+------+
+    //                           \    /
+    //                      3% in-use blocks
+    //
+    //   * The release size threshold is 97%.
+    //
+    // The 3% size in a group is about 7 pages. For two consecutive
+    // releaseToOSMaybe(), we require the difference between `PushedBlocks`
+    // should be greater than 7 pages. This mitigates the page releasing
+    // thrashing which is caused by memory usage bouncing around the threshold.
+    // The smallest size class takes longest time to do the page release so we
+    // use its size of in-use blocks as a heuristic.
+    SmallerBlockReleasePageDelta =
+        PagesInGroup * (1 + MinSizeClass / 16U) / 100;
   }
 
   void unmapTestOnly() NO_THREAD_SAFETY_ANALYSIS {
@@ -280,14 +321,14 @@ template <typename Config> class SizeClassAllocator64 {
     return true;
   }
 
-  uptr releaseToOS() {
+  uptr releaseToOS(ReleaseToOS ReleaseType) {
     uptr TotalReleasedBytes = 0;
     for (uptr I = 0; I < NumClasses; I++) {
       if (I == SizeClassMap::BatchClassId)
         continue;
       RegionInfo *Region = getRegionInfo(I);
       ScopedLock L(Region->Mutex);
-      TotalReleasedBytes += releaseToOSMaybe(Region, I, /*Force=*/true);
+      TotalReleasedBytes += releaseToOSMaybe(Region, I, ReleaseType);
     }
     return TotalReleasedBytes;
   }
@@ -299,9 +340,6 @@ template <typename Config> class SizeClassAllocator64 {
   static uptr getRegionInfoArraySize() { return sizeof(RegionInfoArray); }
 
   uptr getCompactPtrBaseByClassId(uptr ClassId) {
-    // If we are not compacting pointers, base everything off of 0.
-    if (sizeof(CompactPtrT) == sizeof(uptr) && CompactPtrScale == 0)
-      return 0;
     return getRegionInfo(ClassId)->RegionBeg;
   }
 
@@ -413,6 +451,9 @@ template <typename Config> class SizeClassAllocator64 {
   static_assert(sizeof(RegionInfo) % SCUDO_CACHE_LINE_SIZE == 0, "");
 
   uptr PrimaryBase = 0;
+  // The minimum size of pushed blocks that we will try to release the pages in
+  // that size class.
+  uptr SmallerBlockReleasePageDelta = 0;
   MapPlatformData Data = {};
   atomic_s32 ReleaseToOsIntervalMs = {};
   alignas(SCUDO_CACHE_LINE_SIZE) RegionInfo RegionInfoArray[NumClasses];
@@ -422,8 +463,10 @@ template <typename Config> class SizeClassAllocator64 {
     return &RegionInfoArray[ClassId];
   }
 
-  uptr getRegionBaseByClassId(uptr ClassId) const {
-    return PrimaryBase + (ClassId << Config::PrimaryRegionSizeLog);
+  uptr getRegionBaseByClassId(uptr ClassId) {
+    return roundDown(getRegionInfo(ClassId)->RegionBeg - PrimaryBase,
+                     RegionSize) +
+           PrimaryBase;
   }
 
   static CompactPtrT compactPtrInternal(uptr Base, uptr Ptr) {
@@ -435,10 +478,12 @@ template <typename Config> class SizeClassAllocator64 {
   }
 
   static uptr compactPtrGroup(CompactPtrT CompactPtr) {
-    return static_cast<uptr>(CompactPtr) >> (GroupSizeLog - CompactPtrScale);
+    const uptr Mask = (static_cast<uptr>(1) << GroupScale) - 1;
+    return static_cast<uptr>(CompactPtr) & ~Mask;
   }
-  static uptr batchGroupBase(uptr Base, uptr GroupId) {
-    return (GroupId << GroupSizeLog) + Base;
+  static uptr decompactGroupBase(uptr Base, uptr CompactPtrGroupBase) {
+    DCHECK_EQ(CompactPtrGroupBase % (static_cast<uptr>(1) << (GroupScale)), 0U);
+    return Base + (CompactPtrGroupBase << CompactPtrScale);
   }
 
   // Push the blocks to their batch group. The layout will be like,
@@ -464,7 +509,7 @@ template <typename Config> class SizeClassAllocator64 {
       REQUIRES(Region->Mutex) {
     DCHECK_GT(Size, 0U);
 
-    auto CreateGroup = [&](uptr GroupId) {
+    auto CreateGroup = [&](uptr CompactPtrGroupBase) {
       BatchGroup *BG = nullptr;
       TransferBatch *TB = nullptr;
       if (ClassId == SizeClassMap::BatchClassId) {
@@ -523,7 +568,7 @@ template <typename Config> class SizeClassAllocator64 {
         TB->clear();
       }
 
-      BG->GroupId = GroupId;
+      BG->CompactPtrGroupBase = CompactPtrGroupBase;
       // TODO(chiahungduan): Avoid the use of push_back() in `Batches`.
       BG->Batches.push_front(TB);
       BG->PushedBlocks = 0;
@@ -565,7 +610,7 @@ template <typename Config> class SizeClassAllocator64 {
     if (ClassId == SizeClassMap::BatchClassId) {
       if (Cur == nullptr) {
         // Don't need to classify BatchClassId.
-        Cur = CreateGroup(/*GroupId=*/0);
+        Cur = CreateGroup(/*CompactPtrGroupBase=*/0);
         Region->FreeList.push_front(Cur);
       }
       InsertBlocks(Cur, Array, Size);
@@ -576,12 +621,14 @@ template <typename Config> class SizeClassAllocator64 {
     // will be pushed next. `Prev` is the element right before `Cur`.
     BatchGroup *Prev = nullptr;
 
-    while (Cur != nullptr && compactPtrGroup(Array[0]) > Cur->GroupId) {
+    while (Cur != nullptr &&
+           compactPtrGroup(Array[0]) > Cur->CompactPtrGroupBase) {
       Prev = Cur;
       Cur = Cur->Next;
     }
 
-    if (Cur == nullptr || compactPtrGroup(Array[0]) != Cur->GroupId) {
+    if (Cur == nullptr ||
+        compactPtrGroup(Array[0]) != Cur->CompactPtrGroupBase) {
       Cur = CreateGroup(compactPtrGroup(Array[0]));
       if (Prev == nullptr)
         Region->FreeList.push_front(Cur);
@@ -593,7 +640,7 @@ template <typename Config> class SizeClassAllocator64 {
     // id.
     if (SameGroup) {
       for (u32 I = 0; I < Size; ++I)
-        DCHECK_EQ(compactPtrGroup(Array[I]), Cur->GroupId);
+        DCHECK_EQ(compactPtrGroup(Array[I]), Cur->CompactPtrGroupBase);
 
       InsertBlocks(Cur, Array, Size);
       return;
@@ -604,15 +651,17 @@ template <typename Config> class SizeClassAllocator64 {
     u32 Count = 1;
     for (u32 I = 1; I < Size; ++I) {
       if (compactPtrGroup(Array[I - 1]) != compactPtrGroup(Array[I])) {
-        DCHECK_EQ(compactPtrGroup(Array[I - 1]), Cur->GroupId);
+        DCHECK_EQ(compactPtrGroup(Array[I - 1]), Cur->CompactPtrGroupBase);
         InsertBlocks(Cur, Array + I - Count, Count);
 
-        while (Cur != nullptr && compactPtrGroup(Array[I]) > Cur->GroupId) {
+        while (Cur != nullptr &&
+               compactPtrGroup(Array[I]) > Cur->CompactPtrGroupBase) {
           Prev = Cur;
           Cur = Cur->Next;
         }
 
-        if (Cur == nullptr || compactPtrGroup(Array[I]) != Cur->GroupId) {
+        if (Cur == nullptr ||
+            compactPtrGroup(Array[I]) != Cur->CompactPtrGroupBase) {
           Cur = CreateGroup(compactPtrGroup(Array[I]));
           DCHECK_NE(Prev, nullptr);
           Region->FreeList.insert(Prev, Cur);
@@ -756,7 +805,8 @@ template <typename Config> class SizeClassAllocator64 {
   }
 
   NOINLINE uptr releaseToOSMaybe(RegionInfo *Region, uptr ClassId,
-                                 bool Force = false) REQUIRES(Region->Mutex) {
+                                 ReleaseToOS ReleaseType = ReleaseToOS::Normal)
+      REQUIRES(Region->Mutex) {
     const uptr BlockSize = getSizeByClassId(ClassId);
     const uptr PageSize = getPageSizeCached();
 
@@ -772,97 +822,194 @@ template <typename Config> class SizeClassAllocator64 {
     if (BytesPushed < PageSize)
       return 0; // Nothing new to release.
 
-    bool CheckDensity = BlockSize < PageSize / 16U;
+    const bool CheckDensity =
+        BlockSize < PageSize / 16U && ReleaseType != ReleaseToOS::ForceAll;
     // Releasing smaller blocks is expensive, so we want to make sure that a
     // significant amount of bytes are free, and that there has been a good
     // amount of batches pushed to the freelist before attempting to release.
     if (CheckDensity) {
-      if (!Force && BytesPushed < Region->AllocatedUser / 16U)
+      if (ReleaseType == ReleaseToOS::Normal &&
+          BytesPushed < Region->AllocatedUser / 16U)
         return 0;
     }
 
-    if (!Force) {
+    if (ReleaseType == ReleaseToOS::Normal) {
       const s32 IntervalMs = atomic_load_relaxed(&ReleaseToOsIntervalMs);
       if (IntervalMs < 0)
         return 0;
       if (Region->ReleaseInfo.LastReleaseAtNs +
               static_cast<u64>(IntervalMs) * 1000000 >
-          getMonotonicTime()) {
+          getMonotonicTimeFast()) {
         return 0; // Memory was returned recently.
       }
     }
 
     const uptr GroupSize = (1U << GroupSizeLog);
     const uptr AllocatedUserEnd = Region->AllocatedUser + Region->RegionBeg;
-    ReleaseRecorder Recorder(Region->RegionBeg, &Region->Data);
-    PageReleaseContext Context(BlockSize, Region->AllocatedUser,
-                               /*NumberOfRegions=*/1U);
-
     const uptr CompactPtrBase = getCompactPtrBaseByClassId(ClassId);
     auto DecompactPtr = [CompactPtrBase](CompactPtrT CompactPtr) {
       return decompactPtrInternal(CompactPtrBase, CompactPtr);
     };
-    for (BatchGroup &BG : Region->FreeList) {
+
+    // Instead of always preparing PageMap for the entire region, we only do it
+    // for the range of releasing groups. To do that, the free-block marking
+    // process includes visiting BlockGroups twice.
+
+    // The first visit is to determine the range of BatchGroups we are going to
+    // release. And we will extract those BatchGroups out and push into
+    // `GroupToRelease`.
+    SinglyLinkedList<BatchGroup> GroupToRelease;
+    GroupToRelease.clear();
+
+    // This is only used for debugging to ensure the consistency of the number
+    // of groups.
+    uptr NumberOfBatchGroups = Region->FreeList.size();
+
+    for (BatchGroup *BG = Region->FreeList.front(), *Prev = nullptr;
+         BG != nullptr;) {
       const uptr PushedBytesDelta =
-          BG.PushedBlocks - BG.PushedBlocksAtLastCheckpoint;
-      if (PushedBytesDelta * BlockSize < PageSize)
+          BG->PushedBlocks - BG->PushedBlocksAtLastCheckpoint;
+      if (PushedBytesDelta * BlockSize < PageSize) {
+        Prev = BG;
+        BG = BG->Next;
         continue;
+      }
 
-      // Group boundary does not necessarily have the same alignment as Region.
-      // It may sit across a Region boundary. Which means that we may have the
-      // following two cases,
-      //
-      // 1. Group boundary sits before RegionBeg.
-      //
-      //                (BatchGroupBeg)
-      // batchGroupBase  RegionBeg       BatchGroupEnd
-      //        |            |                |
-      //        v            v                v
-      //        +------------+----------------+
-      //         \                           /
-      //          ------   GroupSize   ------
-      //
-      // 2. Group boundary sits after RegionBeg.
+      // Group boundary is always GroupSize-aligned from CompactPtr base. The
+      // layout of memory groups is like,
       //
-      //               (BatchGroupBeg)
-      //    RegionBeg  batchGroupBase               BatchGroupEnd
-      //        |           |                             |
-      //        v           v                             v
-      //        +-----------+-----------------------------+
-      //                     \                           /
-      //                      ------   GroupSize   ------
+      //     (CompactPtrBase)
+      // #1 CompactPtrGroupBase   #2 CompactPtrGroupBase            ...
+      //           |                       |                       |
+      //           v                       v                       v
+      //           +-----------------------+-----------------------+
+      //            \                     / \                     /
+      //             ---   GroupSize   ---   ---   GroupSize   ---
       //
-      // Note that in the first case, the group range before RegionBeg is never
-      // used. Therefore, while calculating the used group size, we should
-      // exclude that part to get the correct size.
-      const uptr BatchGroupBeg =
-          Max(batchGroupBase(CompactPtrBase, BG.GroupId), Region->RegionBeg);
-      DCHECK_GE(AllocatedUserEnd, BatchGroupBeg);
-      const uptr BatchGroupEnd =
-          batchGroupBase(CompactPtrBase, BG.GroupId) + GroupSize;
+      // After decompacting the CompactPtrGroupBase, we expect the alignment
+      // property is held as well.
+      const uptr BatchGroupBase =
+          decompactGroupBase(CompactPtrBase, BG->CompactPtrGroupBase);
+      DCHECK_LE(Region->RegionBeg, BatchGroupBase);
+      DCHECK_GE(AllocatedUserEnd, BatchGroupBase);
+      DCHECK_EQ((Region->RegionBeg - BatchGroupBase) % GroupSize, 0U);
+      const uptr BatchGroupEnd = BatchGroupBase + GroupSize;
       const uptr AllocatedGroupSize = AllocatedUserEnd >= BatchGroupEnd
-                                          ? BatchGroupEnd - BatchGroupBeg
-                                          : AllocatedUserEnd - BatchGroupBeg;
-      if (AllocatedGroupSize == 0)
+                                          ? GroupSize
+                                          : AllocatedUserEnd - BatchGroupBase;
+      if (AllocatedGroupSize == 0) {
+        Prev = BG;
+        BG = BG->Next;
         continue;
+      }
 
       // TransferBatches are pushed in front of BG.Batches. The first one may
       // not have all caches used.
-      const uptr NumBlocks = (BG.Batches.size() - 1) * BG.MaxCachedPerBatch +
-                             BG.Batches.front()->getCount();
+      const uptr NumBlocks = (BG->Batches.size() - 1) * BG->MaxCachedPerBatch +
+                             BG->Batches.front()->getCount();
       const uptr BytesInBG = NumBlocks * BlockSize;
       // Given the randomness property, we try to release the pages only if the
       // bytes used by free blocks exceed certain proportion of group size. Note
       // that this heuristic only applies when all the spaces in a BatchGroup
       // are allocated.
-      if (CheckDensity && (BytesInBG * 100U) / AllocatedGroupSize <
-                              (100U - 1U - BlockSize / 16U)) {
-        continue;
+      if (CheckDensity) {
+        const bool HighDensity = (BytesInBG * 100U) / AllocatedGroupSize >=
+                                 (100U - 1U - BlockSize / 16U);
+        const bool MayHaveReleasedAll = NumBlocks >= (GroupSize / BlockSize);
+        // If all blocks in the group are released, we will do range marking
+        // which is fast. Otherwise, we will wait until we have accumulated
+        // a certain amount of free memory.
+        const bool ReachReleaseDelta =
+            MayHaveReleasedAll ? true
+                               : PushedBytesDelta * BlockSize >=
+                                     PageSize * SmallerBlockReleasePageDelta;
+
+        if (!HighDensity || !ReachReleaseDelta) {
+          Prev = BG;
+          BG = BG->Next;
+          continue;
+        }
       }
 
+      // If `BG` is the first BatchGroup in the list, we only need to advance
+      // `BG` and call FreeList::pop_front(). No update is needed for `Prev`.
+      //
+      //         (BG)   (BG->Next)
+      // Prev     Cur      BG
+      //   |       |       |
+      //   v       v       v
+      //  nil     +--+    +--+
+      //          |X | -> |  | -> ...
+      //          +--+    +--+
+      //
+      // Otherwise, `Prev` will be used to extract the `Cur` from the
+      // `FreeList`.
+      //
+      //         (BG)   (BG->Next)
+      // Prev     Cur      BG
+      //   |       |       |
+      //   v       v       v
+      //  +--+    +--+    +--+
+      //  |  | -> |X | -> |  | -> ...
+      //  +--+    +--+    +--+
+      //
+      // After FreeList::extract(),
+      //
+      // Prev     Cur       BG
+      //   |       |        |
+      //   v       v        v
+      //  +--+    +--+     +--+
+      //  |  |-+  |X |  +->|  | -> ...
+      //  +--+ |  +--+  |  +--+
+      //       +--------+
+      //
+      // Note that we need to advance before pushing this BatchGroup to
+      // GroupToRelease because it's a destructive operation.
+
+      BatchGroup *Cur = BG;
+      BG = BG->Next;
+
+      if (Prev != nullptr)
+        Region->FreeList.extract(Prev, Cur);
+      else
+        Region->FreeList.pop_front();
+      GroupToRelease.push_back(Cur);
+    }
+
+    if (GroupToRelease.empty())
+      return 0;
+
+    const uptr ReleaseBase = decompactGroupBase(
+        CompactPtrBase, GroupToRelease.front()->CompactPtrGroupBase);
+    const uptr LastGroupEnd =
+        Min(decompactGroupBase(CompactPtrBase,
+                               GroupToRelease.back()->CompactPtrGroupBase) +
+                GroupSize,
+            AllocatedUserEnd);
+    // The last block may straddle the group boundary. Rounding up to BlockSize
+    // to get the exact range.
+    const uptr ReleaseEnd =
+        roundUpSlow(LastGroupEnd - Region->RegionBeg, BlockSize) +
+        Region->RegionBeg;
+    const uptr ReleaseRangeSize = ReleaseEnd - ReleaseBase;
+    const uptr ReleaseOffset = ReleaseBase - Region->RegionBeg;
+
+    ReleaseRecorder Recorder(Region->RegionBeg, ReleaseOffset, &Region->Data);
+    PageReleaseContext Context(BlockSize, /*NumberOfRegions=*/1U,
+                               ReleaseRangeSize, ReleaseOffset);
+
+    for (BatchGroup &BG : GroupToRelease) {
       BG.PushedBlocksAtLastCheckpoint = BG.PushedBlocks;
 
-      const uptr BatchGroupUsedEnd = BatchGroupBeg + AllocatedGroupSize;
+      const uptr BatchGroupBase =
+          decompactGroupBase(CompactPtrBase, BG.CompactPtrGroupBase);
+      const uptr BatchGroupEnd = BatchGroupBase + GroupSize;
+      const uptr AllocatedGroupSize = AllocatedUserEnd >= BatchGroupEnd
+                                          ? GroupSize
+                                          : AllocatedUserEnd - BatchGroupBase;
+      const uptr BatchGroupUsedEnd = BatchGroupBase + AllocatedGroupSize;
+      const bool MayContainLastBlockInRegion =
+          BatchGroupUsedEnd == AllocatedUserEnd;
       const bool BlockAlignedWithUsedEnd =
           (BatchGroupUsedEnd - Region->RegionBeg) % BlockSize == 0;
 
@@ -870,24 +1017,29 @@ template <typename Config> class SizeClassAllocator64 {
       if (!BlockAlignedWithUsedEnd)
         ++MaxContainedBlocks;
 
+      const uptr NumBlocks = (BG.Batches.size() - 1) * BG.MaxCachedPerBatch +
+                             BG.Batches.front()->getCount();
+
       if (NumBlocks == MaxContainedBlocks) {
         for (const auto &It : BG.Batches)
           for (u16 I = 0; I < It.getCount(); ++I)
-            DCHECK_EQ(compactPtrGroup(It.get(I)), BG.GroupId);
+            DCHECK_EQ(compactPtrGroup(It.get(I)), BG.CompactPtrGroupBase);
 
-        Context.markRangeAsAllCounted(BatchGroupBeg, BatchGroupUsedEnd,
-                                      Region->RegionBeg);
+        Context.markRangeAsAllCounted(BatchGroupBase, BatchGroupUsedEnd,
+                                      Region->RegionBeg, /*RegionIndex=*/0,
+                                      Region->AllocatedUser);
       } else {
         DCHECK_LT(NumBlocks, MaxContainedBlocks);
         // Note that we don't always visit blocks in each BatchGroup so that we
         // may miss the chance of releasing certain pages that cross
         // BatchGroups.
-        Context.markFreeBlocks(BG.Batches, DecompactPtr, Region->RegionBeg);
+        Context.markFreeBlocksInRegion(
+            BG.Batches, DecompactPtr, Region->RegionBeg, /*RegionIndex=*/0,
+            Region->AllocatedUser, MayContainLastBlockInRegion);
       }
     }
 
-    if (!Context.hasBlockMarked())
-      return 0;
+    DCHECK(Context.hasBlockMarked());
 
     auto SkipRegion = [](UNUSED uptr RegionIndex) { return false; };
     releaseFreeMemoryToOS(Context, Recorder, SkipRegion);
@@ -898,7 +1050,58 @@ template <typename Config> class SizeClassAllocator64 {
       Region->ReleaseInfo.RangesReleased += Recorder.getReleasedRangesCount();
       Region->ReleaseInfo.LastReleasedBytes = Recorder.getReleasedBytes();
     }
-    Region->ReleaseInfo.LastReleaseAtNs = getMonotonicTime();
+    Region->ReleaseInfo.LastReleaseAtNs = getMonotonicTimeFast();
+
+    // Merge GroupToRelease back to the Region::FreeList. Note that both
+    // `Region->FreeList` and `GroupToRelease` are sorted.
+    for (BatchGroup *BG = Region->FreeList.front(), *Prev = nullptr;;) {
+      if (BG == nullptr || GroupToRelease.empty()) {
+        if (!GroupToRelease.empty())
+          Region->FreeList.append_back(&GroupToRelease);
+        break;
+      }
+
+      DCHECK_NE(BG->CompactPtrGroupBase,
+                GroupToRelease.front()->CompactPtrGroupBase);
+
+      if (BG->CompactPtrGroupBase <
+          GroupToRelease.front()->CompactPtrGroupBase) {
+        Prev = BG;
+        BG = BG->Next;
+        continue;
+      }
+
+      // At here, the `BG` is the first BatchGroup with CompactPtrGroupBase
+      // larger than the first element in `GroupToRelease`. We need to insert
+      // `GroupToRelease::front()` (which is `Cur` below)  before `BG`.
+      //
+      //   1. If `Prev` is nullptr, we simply push `Cur` to the front of
+      //      FreeList.
+      //   2. Otherwise, use `insert()` which inserts an element next to `Prev`.
+      //
+      // Afterwards, we don't need to advance `BG` because the order between
+      // `BG` and the new `GroupToRelease::front()` hasn't been checked.
+      BatchGroup *Cur = GroupToRelease.front();
+      GroupToRelease.pop_front();
+      if (Prev == nullptr)
+        Region->FreeList.push_front(Cur);
+      else
+        Region->FreeList.insert(Prev, Cur);
+      DCHECK_EQ(Cur->Next, BG);
+      Prev = Cur;
+    }
+
+    DCHECK_EQ(Region->FreeList.size(), NumberOfBatchGroups);
+    (void)NumberOfBatchGroups;
+
+    if (SCUDO_DEBUG) {
+      BatchGroup *Prev = Region->FreeList.front();
+      for (BatchGroup *Cur = Prev->Next; Cur != nullptr;
+           Prev = Cur, Cur = Cur->Next) {
+        CHECK_LT(Prev->CompactPtrGroupBase, Cur->CompactPtrGroupBase);
+      }
+    }
+
     return Recorder.getReleasedBytes();
   }
 };
diff --git a/compiler-rt/lib/scudo/standalone/release.cpp b/compiler-rt/lib/scudo/standalone/release.cpp
index 3f40dbec6d7a8..938bb41faf695 100644
--- a/compiler-rt/lib/scudo/standalone/release.cpp
+++ b/compiler-rt/lib/scudo/standalone/release.cpp
@@ -10,7 +10,7 @@
 
 namespace scudo {
 
-HybridMutex RegionPageMap::Mutex = {};
-uptr RegionPageMap::StaticBuffer[RegionPageMap::StaticBufferCount];
+BufferPool<RegionPageMap::StaticBufferCount, RegionPageMap::StaticBufferSize>
+    RegionPageMap::Buffers;
 
 } // namespace scudo
diff --git a/compiler-rt/lib/scudo/standalone/release.h b/compiler-rt/lib/scudo/standalone/release.h
index 733326db7b81f..bb66a5d2c409c 100644
--- a/compiler-rt/lib/scudo/standalone/release.h
+++ b/compiler-rt/lib/scudo/standalone/release.h
@@ -18,8 +18,8 @@ namespace scudo {
 
 class ReleaseRecorder {
 public:
-  ReleaseRecorder(uptr Base, MapPlatformData *Data = nullptr)
-      : Base(Base), Data(Data) {}
+  ReleaseRecorder(uptr Base, uptr Offset = 0, MapPlatformData *Data = nullptr)
+      : Base(Base), Offset(Offset), Data(Data) {}
 
   uptr getReleasedRangesCount() const { return ReleasedRangesCount; }
 
@@ -30,7 +30,7 @@ class ReleaseRecorder {
   // Releases [From, To) range of pages back to OS.
   void releasePageRangeToOS(uptr From, uptr To) {
     const uptr Size = To - From;
-    releasePagesToOS(Base, From, Size, Data);
+    releasePagesToOS(Base, From + Offset, Size, Data);
     ReleasedRangesCount++;
     ReleasedBytes += Size;
   }
@@ -38,10 +38,110 @@ class ReleaseRecorder {
 private:
   uptr ReleasedRangesCount = 0;
   uptr ReleasedBytes = 0;
+  // The starting address to release. Note that we may want to combine (Base +
+  // Offset) as a new Base. However, the Base is retrieved from
+  // `MapPlatformData` on Fuchsia, which means the offset won't be aware.
+  // Therefore, store them separately to make it work on all the platforms.
   uptr Base = 0;
+  // The release offset from Base. This is used when we know a given range after
+  // Base will not be released.
+  uptr Offset = 0;
   MapPlatformData *Data = nullptr;
 };
 
+// A buffer pool which holds a fixed number of static buffers for fast buffer
+// allocation. If the request size is greater than `StaticBufferSize`, it'll
+// delegate the allocation to map().
+template <uptr StaticBufferCount, uptr StaticBufferSize> class BufferPool {
+public:
+  // Preserve 1 bit in the `Mask` so that we don't need to do zero-check while
+  // extracting the least significant bit from the `Mask`.
+  static_assert(StaticBufferCount < SCUDO_WORDSIZE, "");
+  static_assert(isAligned(StaticBufferSize, SCUDO_CACHE_LINE_SIZE), "");
+
+  // Return a buffer which is at least `BufferSize`.
+  uptr *getBuffer(const uptr BufferSize) {
+    if (UNLIKELY(BufferSize > StaticBufferSize))
+      return getDynamicBuffer(BufferSize);
+
+    uptr index;
+    {
+      // TODO: In general, we expect this operation should be fast so the
+      // waiting thread won't be put into sleep. The HybridMutex does implement
+      // the busy-waiting but we may want to review the performance and see if
+      // we need an explict spin lock here.
+      ScopedLock L(Mutex);
+      index = getLeastSignificantSetBitIndex(Mask);
+      if (index < StaticBufferCount)
+        Mask ^= static_cast<uptr>(1) << index;
+    }
+
+    if (index >= StaticBufferCount)
+      return getDynamicBuffer(BufferSize);
+
+    const uptr Offset = index * StaticBufferSize;
+    memset(&RawBuffer[Offset], 0, StaticBufferSize);
+    return &RawBuffer[Offset];
+  }
+
+  void releaseBuffer(uptr *Buffer, const uptr BufferSize) {
+    const uptr index = getStaticBufferIndex(Buffer, BufferSize);
+    if (index < StaticBufferCount) {
+      ScopedLock L(Mutex);
+      DCHECK_EQ((Mask & (static_cast<uptr>(1) << index)), 0U);
+      Mask |= static_cast<uptr>(1) << index;
+    } else {
+      unmap(reinterpret_cast<void *>(Buffer),
+            roundUp(BufferSize, getPageSizeCached()));
+    }
+  }
+
+  bool isStaticBufferTestOnly(uptr *Buffer, uptr BufferSize) {
+    return getStaticBufferIndex(Buffer, BufferSize) < StaticBufferCount;
+  }
+
+private:
+  uptr getStaticBufferIndex(uptr *Buffer, uptr BufferSize) {
+    if (UNLIKELY(BufferSize > StaticBufferSize))
+      return StaticBufferCount;
+
+    const uptr BufferBase = reinterpret_cast<uptr>(Buffer);
+    const uptr RawBufferBase = reinterpret_cast<uptr>(RawBuffer);
+
+    if (BufferBase < RawBufferBase ||
+        BufferBase >= RawBufferBase + sizeof(RawBuffer)) {
+      return StaticBufferCount;
+    }
+
+    DCHECK_LE(BufferSize, StaticBufferSize);
+    DCHECK_LE(BufferBase + BufferSize, RawBufferBase + sizeof(RawBuffer));
+    DCHECK_EQ((BufferBase - RawBufferBase) % StaticBufferSize, 0U);
+
+    const uptr index =
+        (BufferBase - RawBufferBase) / (StaticBufferSize * sizeof(uptr));
+    DCHECK_LT(index, StaticBufferCount);
+    return index;
+  }
+
+  uptr *getDynamicBuffer(const uptr BufferSize) {
+    // When using a heap-based buffer, precommit the pages backing the
+    // Vmar by passing |MAP_PRECOMMIT| flag. This allows an optimization
+    // where page fault exceptions are skipped as the allocated memory
+    // is accessed. So far, this is only enabled on Fuchsia. It hasn't proven a
+    // performance benefit on other platforms.
+    const uptr MmapFlags = MAP_ALLOWNOMEM | (SCUDO_FUCHSIA ? MAP_PRECOMMIT : 0);
+    return reinterpret_cast<uptr *>(
+        map(nullptr, roundUp(BufferSize, getPageSizeCached()), "scudo:counters",
+            MmapFlags, &MapData));
+  }
+
+  HybridMutex Mutex;
+  // '1' means that buffer index is not used. '0' means the buffer is in use.
+  uptr Mask GUARDED_BY(Mutex) = ~static_cast<uptr>(0);
+  uptr RawBuffer[StaticBufferCount * StaticBufferSize] GUARDED_BY(Mutex);
+  [[no_unique_address]] MapPlatformData MapData = {};
+};
+
 // A Region page map is used to record the usage of pages in the regions. It
 // implements a packed array of Counters. Each counter occupies 2^N bits, enough
 // to store counter's MaxValue. Ctor will try to use a static buffer first, and
@@ -69,11 +169,7 @@ class RegionPageMap {
   ~RegionPageMap() {
     if (!isAllocated())
       return;
-    if (Buffer == &StaticBuffer[0])
-      Mutex.unlock();
-    else
-      unmap(reinterpret_cast<void *>(Buffer),
-            roundUp(BufferSize, getPageSizeCached()));
+    Buffers.releaseBuffer(Buffer, BufferSize);
     Buffer = nullptr;
   }
 
@@ -81,8 +177,7 @@ class RegionPageMap {
   // specify the thread-safety attribute properly in current code structure.
   // Besides, it's the only place we may want to check thread safety. Therefore,
   // it's fine to bypass the thread-safety analysis now.
-  void reset(uptr NumberOfRegion, uptr CountersPerRegion,
-             uptr MaxValue) NO_THREAD_SAFETY_ANALYSIS {
+  void reset(uptr NumberOfRegion, uptr CountersPerRegion, uptr MaxValue) {
     DCHECK_GT(NumberOfRegion, 0);
     DCHECK_GT(CountersPerRegion, 0);
     DCHECK_GT(MaxValue, 0);
@@ -108,21 +203,8 @@ class RegionPageMap {
         roundUp(NumCounters, static_cast<uptr>(1U) << PackingRatioLog) >>
         PackingRatioLog;
     BufferSize = SizePerRegion * sizeof(*Buffer) * Regions;
-    if (BufferSize <= (StaticBufferCount * sizeof(Buffer[0])) &&
-        Mutex.tryLock()) {
-      Buffer = &StaticBuffer[0];
-      memset(Buffer, 0, BufferSize);
-    } else {
-      // When using a heap-based buffer, precommit the pages backing the
-      // Vmar by passing |MAP_PRECOMMIT| flag. This allows an optimization
-      // where page fault exceptions are skipped as the allocated memory
-      // is accessed.
-      const uptr MmapFlags =
-          MAP_ALLOWNOMEM | (SCUDO_FUCHSIA ? MAP_PRECOMMIT : 0);
-      Buffer = reinterpret_cast<uptr *>(
-          map(nullptr, roundUp(BufferSize, getPageSizeCached()),
-              "scudo:counters", MmapFlags, &MapData));
-    }
+    Buffer = Buffers.getBuffer(BufferSize);
+    DCHECK_NE(Buffer, nullptr);
   }
 
   bool isAllocated() const { return !!Buffer; }
@@ -199,8 +281,6 @@ class RegionPageMap {
 
   uptr getBufferSize() const { return BufferSize; }
 
-  static const uptr StaticBufferCount = 2048U;
-
 private:
   uptr Regions;
   uptr NumCounters;
@@ -212,10 +292,12 @@ class RegionPageMap {
   uptr SizePerRegion;
   uptr BufferSize;
   uptr *Buffer;
-  [[no_unique_address]] MapPlatformData MapData = {};
 
-  static HybridMutex Mutex;
-  static uptr StaticBuffer[StaticBufferCount] GUARDED_BY(Mutex);
+  // We may consider making this configurable if there are cases which may
+  // benefit from this.
+  static const uptr StaticBufferCount = 2U;
+  static const uptr StaticBufferSize = 512U;
+  static BufferPool<StaticBufferCount, StaticBufferSize> Buffers;
 };
 
 template <class ReleaseRecorderT> class FreePagesRangeTracker {
@@ -259,10 +341,9 @@ template <class ReleaseRecorderT> class FreePagesRangeTracker {
 };
 
 struct PageReleaseContext {
-  PageReleaseContext(uptr BlockSize, uptr RegionSize, uptr NumberOfRegions) :
-      BlockSize(BlockSize),
-      RegionSize(RegionSize),
-      NumberOfRegions(NumberOfRegions) {
+  PageReleaseContext(uptr BlockSize, uptr NumberOfRegions, uptr ReleaseSize,
+                     uptr ReleaseOffset = 0)
+      : BlockSize(BlockSize), NumberOfRegions(NumberOfRegions) {
     PageSize = getPageSizeCached();
     if (BlockSize <= PageSize) {
       if (PageSize % BlockSize == 0) {
@@ -294,10 +375,16 @@ struct PageReleaseContext {
       }
     }
 
-    PagesCount = roundUp(RegionSize, PageSize) / PageSize;
+    // TODO: For multiple regions, it's more complicated to support partial
+    // region marking (which includes the complexity of how to handle the last
+    // block in a region). We may consider this after markFreeBlocks() accepts
+    // only free blocks from the same region.
+    if (NumberOfRegions != 1)
+      DCHECK_EQ(ReleaseOffset, 0U);
+
+    PagesCount = roundUp(ReleaseSize, PageSize) / PageSize;
     PageSizeLog = getLog2(PageSize);
-    RoundedRegionSize = PagesCount << PageSizeLog;
-    RoundedSize = NumberOfRegions * RoundedRegionSize;
+    ReleasePageOffset = ReleaseOffset >> PageSizeLog;
   }
 
   // PageMap is lazily allocated when markFreeBlocks() is invoked.
@@ -316,25 +403,17 @@ struct PageReleaseContext {
   // the blocks, we will just mark the page as all counted. Note the `From` and
   // `To` has to be page aligned but with one exception, if `To` is equal to the
   // RegionSize, it's not necessary to be aligned with page size.
-  void markRangeAsAllCounted(uptr From, uptr To, uptr Base) {
+  void markRangeAsAllCounted(uptr From, uptr To, uptr Base,
+                             const uptr RegionIndex, const uptr RegionSize) {
     DCHECK_LT(From, To);
+    DCHECK_LE(To, Base + RegionSize);
     DCHECK_EQ(From % PageSize, 0U);
+    DCHECK_LE(To - From, RegionSize);
 
     ensurePageMapAllocated();
 
-    const uptr FromOffset = From - Base;
-    const uptr ToOffset = To - Base;
-
-    const uptr RegionIndex =
-        NumberOfRegions == 1U ? 0 : FromOffset / RegionSize;
-    if (SCUDO_DEBUG) {
-      const uptr ToRegionIndex =
-          NumberOfRegions == 1U ? 0 : (ToOffset - 1) / RegionSize;
-      CHECK_EQ(RegionIndex, ToRegionIndex);
-    }
-
-    uptr FromInRegion = FromOffset - RegionIndex * RegionSize;
-    uptr ToInRegion = ToOffset - RegionIndex * RegionSize;
+    uptr FromInRegion = From - Base;
+    uptr ToInRegion = To - Base;
     uptr FirstBlockInRange = roundUpSlow(FromInRegion, BlockSize);
 
     // The straddling block sits across entire range.
@@ -364,7 +443,7 @@ struct PageReleaseContext {
       uptr NumBlocksInFirstPage =
           (FromInRegion + PageSize - FirstBlockInRange + BlockSize - 1) /
           BlockSize;
-      PageMap.incN(RegionIndex, FromInRegion >> PageSizeLog,
+      PageMap.incN(RegionIndex, getPageIndex(FromInRegion),
                    NumBlocksInFirstPage);
       FromInRegion = roundUp(FromInRegion + 1, PageSize);
     }
@@ -392,8 +471,8 @@ struct PageReleaseContext {
       // The last block is not aligned to `To`, we need to increment the
       // counter of `next page` by 1.
       if (LastBlockInRange + BlockSize != ToInRegion) {
-        PageMap.incRange(RegionIndex, ToInRegion >> PageSizeLog,
-                         (LastBlockInRange + BlockSize - 1) >> PageSizeLog);
+        PageMap.incRange(RegionIndex, getPageIndex(ToInRegion),
+                         getPageIndex(LastBlockInRange + BlockSize - 1));
       }
     } else {
       ToInRegion = RegionSize;
@@ -402,67 +481,82 @@ struct PageReleaseContext {
     // After handling the first page and the last block, it's safe to mark any
     // page in between the range [From, To).
     if (FromInRegion < ToInRegion) {
-      PageMap.setAsAllCountedRange(RegionIndex, FromInRegion >> PageSizeLog,
-                                   (ToInRegion - 1) >> PageSizeLog);
+      PageMap.setAsAllCountedRange(RegionIndex, getPageIndex(FromInRegion),
+                                   getPageIndex(ToInRegion - 1));
     }
   }
 
-  template<class TransferBatchT, typename DecompactPtrT>
-  void markFreeBlocks(const IntrusiveList<TransferBatchT> &FreeList,
-                      DecompactPtrT DecompactPtr, uptr Base) {
+  template <class TransferBatchT, typename DecompactPtrT>
+  void markFreeBlocksInRegion(const IntrusiveList<TransferBatchT> &FreeList,
+                              DecompactPtrT DecompactPtr, const uptr Base,
+                              const uptr RegionIndex, const uptr RegionSize,
+                              bool MayContainLastBlockInRegion) {
     ensurePageMapAllocated();
 
+    if (MayContainLastBlockInRegion) {
+      const uptr LastBlockInRegion =
+          ((RegionSize / BlockSize) - 1U) * BlockSize;
+      // The last block in a region may not use the entire page, we mark the
+      // following "pretend" memory block(s) as free in advance.
+      //
+      //     Region Boundary
+      //         v
+      //  -----+-----------------------+
+      //       |      Last Page        | <- Rounded Region Boundary
+      //  -----+-----------------------+
+      //   |-----||- trailing blocks  -|
+      //      ^
+      //   last block
+      const uptr RoundedRegionSize = roundUp(RegionSize, PageSize);
+      const uptr TrailingBlockBase = LastBlockInRegion + BlockSize;
+      // Only the last page touched by the last block needs to mark the trailing
+      // blocks. If the difference between `RoundedRegionSize` and
+      // `TrailingBlockBase` is larger than a page, that implies the reported
+      // `RegionSize` may not be accurate.
+      DCHECK_LT(RoundedRegionSize - TrailingBlockBase, PageSize);
+      uptr NumTrailingBlocks =
+          roundUpSlow(RoundedRegionSize - TrailingBlockBase, BlockSize) /
+          BlockSize;
+      if (NumTrailingBlocks > 0) {
+        PageMap.incN(RegionIndex, getPageIndex(TrailingBlockBase),
+                     NumTrailingBlocks);
+      }
+    }
+
     // Iterate over free chunks and count how many free chunks affect each
     // allocated page.
     if (BlockSize <= PageSize && PageSize % BlockSize == 0) {
       // Each chunk affects one page only.
       for (const auto &It : FreeList) {
         for (u16 I = 0; I < It.getCount(); I++) {
-          const uptr P = DecompactPtr(It.get(I)) - Base;
-          if (P >= RoundedSize)
-            continue;
-          const uptr RegionIndex = NumberOfRegions == 1U ? 0 : P / RegionSize;
-          const uptr PInRegion = P - RegionIndex * RegionSize;
-          PageMap.inc(RegionIndex, PInRegion >> PageSizeLog);
+          const uptr PInRegion = DecompactPtr(It.get(I)) - Base;
+          DCHECK_LT(PInRegion, RegionSize);
+          PageMap.inc(RegionIndex, getPageIndex(PInRegion));
         }
       }
     } else {
       // In all other cases chunks might affect more than one page.
       DCHECK_GE(RegionSize, BlockSize);
-      const uptr LastBlockInRegion =
-          ((RegionSize / BlockSize) - 1U) * BlockSize;
       for (const auto &It : FreeList) {
         for (u16 I = 0; I < It.getCount(); I++) {
-          const uptr P = DecompactPtr(It.get(I)) - Base;
-          if (P >= RoundedSize)
-            continue;
-          const uptr RegionIndex = NumberOfRegions == 1U ? 0 : P / RegionSize;
-          uptr PInRegion = P - RegionIndex * RegionSize;
-          PageMap.incRange(RegionIndex, PInRegion >> PageSizeLog,
-                            (PInRegion + BlockSize - 1) >> PageSizeLog);
-          // The last block in a region might straddle a page, so if it's
-          // free, we mark the following "pretend" memory block(s) as free.
-          if (PInRegion == LastBlockInRegion) {
-            PInRegion += BlockSize;
-            while (PInRegion < RoundedRegionSize) {
-              PageMap.incRange(RegionIndex, PInRegion >> PageSizeLog,
-                                (PInRegion + BlockSize - 1) >> PageSizeLog);
-              PInRegion += BlockSize;
-            }
-          }
+          const uptr PInRegion = DecompactPtr(It.get(I)) - Base;
+          PageMap.incRange(RegionIndex, getPageIndex(PInRegion),
+                           getPageIndex(PInRegion + BlockSize - 1));
         }
       }
     }
   }
 
+  uptr getPageIndex(uptr P) { return (P >> PageSizeLog) - ReleasePageOffset; }
+
   uptr BlockSize;
-  uptr RegionSize;
   uptr NumberOfRegions;
+  // For partial region marking, some pages in front are not needed to be
+  // counted.
+  uptr ReleasePageOffset;
   uptr PageSize;
   uptr PagesCount;
   uptr PageSizeLog;
-  uptr RoundedRegionSize;
-  uptr RoundedSize;
   uptr FullPagesBlockCountMax;
   bool SameBlockCountPerPage;
   RegionPageMap PageMap;
@@ -479,6 +573,7 @@ releaseFreeMemoryToOS(PageReleaseContext &Context,
   const uptr BlockSize = Context.BlockSize;
   const uptr PagesCount = Context.PagesCount;
   const uptr NumberOfRegions = Context.NumberOfRegions;
+  const uptr ReleasePageOffset = Context.ReleasePageOffset;
   const uptr FullPagesBlockCountMax = Context.FullPagesBlockCountMax;
   const bool SameBlockCountPerPage = Context.SameBlockCountPerPage;
   RegionPageMap &PageMap = Context.PageMap;
@@ -516,6 +611,10 @@ releaseFreeMemoryToOS(PageReleaseContext &Context,
       }
       uptr PrevPageBoundary = 0;
       uptr CurrentBoundary = 0;
+      if (ReleasePageOffset > 0) {
+        PrevPageBoundary = ReleasePageOffset * PageSize;
+        CurrentBoundary = roundUpSlow(PrevPageBoundary, BlockSize);
+      }
       for (uptr J = 0; J < PagesCount; J++) {
         const uptr PageBoundary = PrevPageBoundary + PageSize;
         uptr BlocksPerPage = Pn;
@@ -538,20 +637,6 @@ releaseFreeMemoryToOS(PageReleaseContext &Context,
   RangeTracker.finish();
 }
 
-// An overload releaseFreeMemoryToOS which doesn't require the page usage
-// information after releasing.
-template <class TransferBatchT, class ReleaseRecorderT, typename DecompactPtrT,
-          typename SkipRegionT>
-NOINLINE void
-releaseFreeMemoryToOS(const IntrusiveList<TransferBatchT> &FreeList,
-                      uptr RegionSize, uptr NumberOfRegions, uptr BlockSize,
-                      ReleaseRecorderT &Recorder, DecompactPtrT DecompactPtr,
-                      SkipRegionT SkipRegion) {
-  PageReleaseContext Context(BlockSize, RegionSize, NumberOfRegions);
-  Context.markFreeBlocks(FreeList, DecompactPtr, Recorder.getBase());
-  releaseFreeMemoryToOS(Context, Recorder, SkipRegion);
-}
-
 } // namespace scudo
 
 #endif // SCUDO_RELEASE_H_
diff --git a/compiler-rt/lib/scudo/standalone/secondary.h b/compiler-rt/lib/scudo/standalone/secondary.h
index b312887722261..b30f51b9c494d 100644
--- a/compiler-rt/lib/scudo/standalone/secondary.h
+++ b/compiler-rt/lib/scudo/standalone/secondary.h
@@ -256,36 +256,36 @@ template <typename Config> class MapAllocatorCache {
         Found = true;
         Entry = Entries[I];
         Entries[I].CommitBase = 0;
+        EntriesCount--;
         break;
       }
     }
-    if (Found) {
-      *H = reinterpret_cast<LargeBlock::Header *>(
-          LargeBlock::addHeaderTag<Config>(HeaderPos));
-      *Zeroed = Entry.Time == 0;
-      if (useMemoryTagging<Config>(Options))
-        setMemoryPermission(Entry.CommitBase, Entry.CommitSize, 0, &Entry.Data);
-      uptr NewBlockBegin = reinterpret_cast<uptr>(*H + 1);
-      if (useMemoryTagging<Config>(Options)) {
-        if (*Zeroed)
-          storeTags(LargeBlock::addHeaderTag<Config>(Entry.CommitBase),
-                    NewBlockBegin);
-        else if (Entry.BlockBegin < NewBlockBegin)
-          storeTags(Entry.BlockBegin, NewBlockBegin);
-        else
-          storeTags(untagPointer(NewBlockBegin),
-                    untagPointer(Entry.BlockBegin));
-      }
-      (*H)->CommitBase = Entry.CommitBase;
-      (*H)->CommitSize = Entry.CommitSize;
-      (*H)->MapBase = Entry.MapBase;
-      (*H)->MapSize = Entry.MapSize;
-      (*H)->Data = Entry.Data;
+    if (!Found)
+      return false;
 
-      ScopedLock L(Mutex);
-      EntriesCount--;
+    *H = reinterpret_cast<LargeBlock::Header *>(
+        LargeBlock::addHeaderTag<Config>(HeaderPos));
+    *Zeroed = Entry.Time == 0;
+    if (useMemoryTagging<Config>(Options))
+      setMemoryPermission(Entry.CommitBase, Entry.CommitSize, 0, &Entry.Data);
+    uptr NewBlockBegin = reinterpret_cast<uptr>(*H + 1);
+    if (useMemoryTagging<Config>(Options)) {
+      if (*Zeroed) {
+        storeTags(LargeBlock::addHeaderTag<Config>(Entry.CommitBase),
+                  NewBlockBegin);
+      } else if (Entry.BlockBegin < NewBlockBegin) {
+        storeTags(Entry.BlockBegin, NewBlockBegin);
+      } else {
+        storeTags(untagPointer(NewBlockBegin),
+                  untagPointer(Entry.BlockBegin));
+      }
     }
-    return Found;
+    (*H)->CommitBase = Entry.CommitBase;
+    (*H)->CommitSize = Entry.CommitSize;
+    (*H)->MapBase = Entry.MapBase;
+    (*H)->MapSize = Entry.MapSize;
+    (*H)->Data = Entry.Data;
+    return true;
   }
 
   bool canCache(uptr Size) {
diff --git a/compiler-rt/lib/scudo/standalone/tests/combined_test.cpp b/compiler-rt/lib/scudo/standalone/tests/combined_test.cpp
index 6f4fa748ed93a..550fdd3dd7606 100644
--- a/compiler-rt/lib/scudo/standalone/tests/combined_test.cpp
+++ b/compiler-rt/lib/scudo/standalone/tests/combined_test.cpp
@@ -92,7 +92,7 @@ template <class TypeParam> struct ScudoCombinedTest : public Test {
     Allocator = std::make_unique<AllocatorT>();
   }
   ~ScudoCombinedTest() {
-    Allocator->releaseToOS();
+    Allocator->releaseToOS(scudo::ReleaseToOS::Force);
     UseQuarantine = true;
   }
 
@@ -412,7 +412,7 @@ SCUDO_TYPED_TEST(ScudoCombinedDeathTest, DisableMemoryTagging) {
     reinterpret_cast<char *>(P)[2048] = 0xaa;
     Allocator->deallocate(P, Origin);
 
-    Allocator->releaseToOS();
+    Allocator->releaseToOS(scudo::ReleaseToOS::Force);
   }
 }
 
@@ -488,7 +488,7 @@ SCUDO_TYPED_TEST(ScudoCombinedTest, ThreadedCombined) {
   }
   for (auto &T : Threads)
     T.join();
-  Allocator->releaseToOS();
+  Allocator->releaseToOS(scudo::ReleaseToOS::Force);
 }
 
 // Test that multiple instantiations of the allocator have not messed up the
@@ -601,7 +601,7 @@ TEST(ScudoCombinedTest, FullRegion) {
 // operation without issue.
 SCUDO_TYPED_TEST(ScudoCombinedTest, ReleaseToOS) {
   auto *Allocator = this->Allocator.get();
-  Allocator->releaseToOS();
+  Allocator->releaseToOS(scudo::ReleaseToOS::Force);
 }
 
 SCUDO_TYPED_TEST(ScudoCombinedTest, OddEven) {
@@ -740,7 +740,7 @@ TEST(ScudoCombinedTest, BasicTrustyConfig) {
   auto *TSD = Allocator->getTSDRegistry()->getTSDAndLock(&UnlockRequired);
   TSD->getCache().drain();
 
-  Allocator->releaseToOS();
+  Allocator->releaseToOS(scudo::ReleaseToOS::Force);
 }
 
 #endif
diff --git a/compiler-rt/lib/scudo/standalone/tests/common_test.cpp b/compiler-rt/lib/scudo/standalone/tests/common_test.cpp
index a322a01fb9391..a9794f03fd7a5 100644
--- a/compiler-rt/lib/scudo/standalone/tests/common_test.cpp
+++ b/compiler-rt/lib/scudo/standalone/tests/common_test.cpp
@@ -69,9 +69,11 @@ TEST(ScudoCommonTest, Zeros) {
   unmap(P, Size, 0, &Data);
 }
 
-#if SCUDO_LINUX && !defined(__powerpc__)
-// This test fails intermediately on PPC, which is why this test is disabled
-// for now on this platform.
+#if 0
+// This test is temorarily disabled because it may not work as expected. E.g.,
+// it doesn't dirty the pages so the pages may not be commited and it may only
+// work on the single thread environment. As a result, this test is flaky and is
+// impacting many test scenarios.
 TEST(ScudoCommonTest, GetRssFromBuffer) {
   constexpr int64_t AllocSize = 10000000;
   constexpr int64_t Error = 3000000;
@@ -88,6 +90,6 @@ TEST(ScudoCommonTest, GetRssFromBuffer) {
     EXPECT_LE(std::abs(Rss - AllocSize - Prev), Error);
   }
 }
-#endif // SCUDO_LINUX
+#endif
 
 } // namespace scudo
diff --git a/compiler-rt/lib/scudo/standalone/tests/primary_test.cpp b/compiler-rt/lib/scudo/standalone/tests/primary_test.cpp
index c7ebcc3f82fbd..51a7038ac7870 100644
--- a/compiler-rt/lib/scudo/standalone/tests/primary_test.cpp
+++ b/compiler-rt/lib/scudo/standalone/tests/primary_test.cpp
@@ -161,7 +161,7 @@ SCUDO_TYPED_TEST(ScudoPrimaryTest, BasicPrimary) {
       Cache.deallocate(ClassId, Pointers[J]);
   }
   Cache.destroy(nullptr);
-  Allocator->releaseToOS();
+  Allocator->releaseToOS(scudo::ReleaseToOS::Force);
   scudo::ScopedString Str;
   Allocator->getStats(&Str);
   Str.output();
@@ -215,7 +215,7 @@ TEST(ScudoPrimaryTest, Primary64OOM) {
     Cache.deallocate(Primary::SizeClassMap::BatchClassId, B);
   }
   Cache.destroy(nullptr);
-  Allocator.releaseToOS();
+  Allocator.releaseToOS(scudo::ReleaseToOS::Force);
   scudo::ScopedString Str;
   Allocator.getStats(&Str);
   Str.output();
@@ -253,7 +253,7 @@ SCUDO_TYPED_TEST(ScudoPrimaryTest, PrimaryIterate) {
     V.pop_back();
   }
   Cache.destroy(nullptr);
-  Allocator->releaseToOS();
+  Allocator->releaseToOS(scudo::ReleaseToOS::Force);
   scudo::ScopedString Str;
   Allocator->getStats(&Str);
   Str.output();
@@ -300,7 +300,7 @@ SCUDO_TYPED_TEST(ScudoPrimaryTest, PrimaryThreaded) {
   }
   for (auto &T : Threads)
     T.join();
-  Allocator->releaseToOS();
+  Allocator->releaseToOS(scudo::ReleaseToOS::Force);
   scudo::ScopedString Str;
   Allocator->getStats(&Str);
   Str.output();
@@ -322,7 +322,7 @@ SCUDO_TYPED_TEST(ScudoPrimaryTest, ReleaseToOS) {
   EXPECT_NE(P, nullptr);
   Cache.deallocate(ClassId, P);
   Cache.destroy(nullptr);
-  EXPECT_GT(Allocator->releaseToOS(), 0U);
+  EXPECT_GT(Allocator->releaseToOS(scudo::ReleaseToOS::Force), 0U);
 }
 
 SCUDO_TYPED_TEST(ScudoPrimaryTest, MemoryGroup) {
diff --git a/compiler-rt/lib/scudo/standalone/tests/release_test.cpp b/compiler-rt/lib/scudo/standalone/tests/release_test.cpp
index 3cc20c4621d2c..1fc4284ca77fe 100644
--- a/compiler-rt/lib/scudo/standalone/tests/release_test.cpp
+++ b/compiler-rt/lib/scudo/standalone/tests/release_test.cpp
@@ -138,15 +138,18 @@ TEST(ScudoReleaseTest, FreePagesRangeTracker) {
 
 class ReleasedPagesRecorder {
 public:
+  ReleasedPagesRecorder() = default;
+  explicit ReleasedPagesRecorder(scudo::uptr Base) : Base(Base) {}
   std::set<scudo::uptr> ReportedPages;
 
   void releasePageRangeToOS(scudo::uptr From, scudo::uptr To) {
     const scudo::uptr PageSize = scudo::getPageSizeCached();
     for (scudo::uptr I = From; I < To; I += PageSize)
-      ReportedPages.insert(I);
+      ReportedPages.insert(I + getBase());
   }
 
-  scudo::uptr getBase() const { return 0; }
+  scudo::uptr getBase() const { return Base; }
+  scudo::uptr Base = 0;
 };
 
 // Simplified version of a TransferBatch.
@@ -217,11 +220,12 @@ template <class SizeClassMap> void testReleaseFreeMemoryToOS() {
     auto SkipRegion = [](UNUSED scudo::uptr RegionIndex) { return false; };
     auto DecompactPtr = [](scudo::uptr P) { return P; };
     ReleasedPagesRecorder Recorder;
-    scudo::PageReleaseContext Context(BlockSize,
-                                      /*RegionSize=*/MaxBlocks * BlockSize,
-                                      /*NumberOfRegions=*/1U);
+    scudo::PageReleaseContext Context(BlockSize, /*NumberOfRegions=*/1U,
+                                      /*ReleaseSize=*/MaxBlocks * BlockSize);
     ASSERT_FALSE(Context.hasBlockMarked());
-    Context.markFreeBlocks(FreeList, DecompactPtr, Recorder.getBase());
+    Context.markFreeBlocksInRegion(FreeList, DecompactPtr, Recorder.getBase(),
+                                   /*RegionIndex=*/0, MaxBlocks * BlockSize,
+                                   /*MayContainLastBlockInRegion=*/true);
     ASSERT_TRUE(Context.hasBlockMarked());
     releaseFreeMemoryToOS(Context, Recorder, SkipRegion);
     scudo::RegionPageMap &PageMap = Context.PageMap;
@@ -324,9 +328,10 @@ template <class SizeClassMap> void testPageMapMarkRange() {
       const scudo::uptr GroupBeg = GroupId * GroupSize;
       const scudo::uptr GroupEnd = GroupBeg + GroupSize;
 
-      scudo::PageReleaseContext Context(BlockSize, RegionSize,
-                                        /*NumberOfRegions=*/1U);
-      Context.markRangeAsAllCounted(GroupBeg, GroupEnd, /*Base=*/0);
+      scudo::PageReleaseContext Context(BlockSize, /*NumberOfRegions=*/1U,
+                                        /*ReleaseSize=*/RegionSize);
+      Context.markRangeAsAllCounted(GroupBeg, GroupEnd, /*Base=*/0U,
+                                    /*RegionIndex=*/0, RegionSize);
 
       scudo::uptr FirstBlock =
           ((GroupBeg + BlockSize - 1) / BlockSize) * BlockSize;
@@ -393,14 +398,145 @@ template <class SizeClassMap> void testPageMapMarkRange() {
     } // Iterate each Group
 
     // Release the entire region. This is to ensure the last page is counted.
-    scudo::PageReleaseContext Context(BlockSize, RegionSize,
-                                      /*NumberOfRegions=*/1U);
-    Context.markRangeAsAllCounted(/*From=*/0U, /*To=*/RegionSize, /*Base=*/0);
+    scudo::PageReleaseContext Context(BlockSize, /*NumberOfRegions=*/1U,
+                                      /*ReleaseSize=*/RegionSize);
+    Context.markRangeAsAllCounted(/*From=*/0U, /*To=*/RegionSize, /*Base=*/0,
+                                  /*RegionIndex=*/0, RegionSize);
     for (scudo::uptr Page = 0; Page < RoundedRegionSize / PageSize; ++Page)
       EXPECT_TRUE(Context.PageMap.isAllCounted(/*Region=*/0, Page));
   } // Iterate each size class
 }
 
+template <class SizeClassMap> void testReleasePartialRegion() {
+  typedef FreeBatch<SizeClassMap> Batch;
+  const scudo::uptr PageSize = scudo::getPageSizeCached();
+  const scudo::uptr ReleaseBase = PageSize;
+  const scudo::uptr BasePageOffset = ReleaseBase / PageSize;
+
+  for (scudo::uptr I = 1; I <= SizeClassMap::LargestClassId; I++) {
+    // In the following, we want to ensure the region includes at least 2 pages
+    // and we will release all the pages except the first one. The handling of
+    // the last block is tricky, so we always test the case that includes the
+    // last block.
+    const scudo::uptr BlockSize = SizeClassMap::getSizeByClassId(I);
+    const scudo::uptr RegionSize =
+        scudo::roundUpSlow(scudo::roundUp(BlockSize, PageSize) * 2, BlockSize) +
+        BlockSize;
+    const scudo::uptr RoundedRegionSize = scudo::roundUp(RegionSize, PageSize);
+
+    scudo::SinglyLinkedList<Batch> FreeList;
+    FreeList.clear();
+
+    // Skip the blocks in the first page and add the remaining.
+    std::vector<scudo::uptr> Pages(RoundedRegionSize / PageSize, 0);
+    for (scudo::uptr Block = scudo::roundUpSlow(PageSize, BlockSize);
+         Block + BlockSize <= RoundedRegionSize; Block += BlockSize) {
+      for (scudo::uptr Page = Block / PageSize;
+           Page <= (Block + BlockSize - 1) / PageSize; ++Page) {
+        ASSERT_LT(Page, Pages.size());
+        ++Pages[Page];
+      }
+    }
+
+    // This follows the logic how we count the last page. It should be
+    // consistent with how markFreeBlocksInRegion() handles the last block.
+    if (RoundedRegionSize % BlockSize != 0)
+      ++Pages.back();
+
+    Batch *CurrentBatch = nullptr;
+    for (scudo::uptr Block = scudo::roundUpSlow(PageSize, BlockSize);
+         Block < RegionSize; Block += BlockSize) {
+      if (CurrentBatch == nullptr ||
+          CurrentBatch->getCount() == Batch::MaxCount) {
+        CurrentBatch = new Batch;
+        CurrentBatch->clear();
+        FreeList.push_back(CurrentBatch);
+      }
+      CurrentBatch->add(Block);
+    }
+
+    auto VerifyReleaseToOs = [&](scudo::PageReleaseContext &Context) {
+      auto SkipRegion = [](UNUSED scudo::uptr RegionIndex) { return false; };
+      ReleasedPagesRecorder Recorder(ReleaseBase);
+      releaseFreeMemoryToOS(Context, Recorder, SkipRegion);
+      const scudo::uptr FirstBlock = scudo::roundUpSlow(PageSize, BlockSize);
+
+      for (scudo::uptr P = 0; P < RoundedRegionSize; P += PageSize) {
+        if (P < FirstBlock) {
+          // If FirstBlock is not aligned with page boundary, the first touched
+          // page will not be released either.
+          EXPECT_TRUE(Recorder.ReportedPages.find(P) ==
+                      Recorder.ReportedPages.end());
+        } else {
+          EXPECT_TRUE(Recorder.ReportedPages.find(P) !=
+                      Recorder.ReportedPages.end());
+        }
+      }
+    };
+
+    // Test marking by visiting each block.
+    {
+      auto DecompactPtr = [](scudo::uptr P) { return P; };
+      scudo::PageReleaseContext Context(BlockSize, /*NumberOfRegions=*/1U,
+                                        /*ReleaseSize=*/RegionSize - PageSize,
+                                        ReleaseBase);
+      Context.markFreeBlocksInRegion(FreeList, DecompactPtr, /*Base=*/0U,
+                                     /*RegionIndex=*/0, RegionSize,
+                                     /*MayContainLastBlockInRegion=*/true);
+      for (const Batch &It : FreeList) {
+        for (scudo::u16 I = 0; I < It.getCount(); I++) {
+          scudo::uptr Block = It.get(I);
+          for (scudo::uptr Page = Block / PageSize;
+               Page <= (Block + BlockSize - 1) / PageSize; ++Page) {
+            EXPECT_EQ(Pages[Page], Context.PageMap.get(/*Region=*/0U,
+                                                       Page - BasePageOffset));
+          }
+        }
+      }
+
+      VerifyReleaseToOs(Context);
+    }
+
+    // Test range marking.
+    {
+      scudo::PageReleaseContext Context(BlockSize, /*NumberOfRegions=*/1U,
+                                        /*ReleaseSize=*/RegionSize - PageSize,
+                                        ReleaseBase);
+      Context.markRangeAsAllCounted(ReleaseBase, RegionSize, /*Base=*/0U,
+                                    /*RegionIndex=*/0, RegionSize);
+      for (scudo::uptr Page = ReleaseBase / PageSize;
+           Page < RoundedRegionSize / PageSize; ++Page) {
+        if (Context.PageMap.get(/*Region=*/0, Page - BasePageOffset) !=
+            Pages[Page]) {
+          EXPECT_TRUE(Context.PageMap.isAllCounted(/*Region=*/0,
+                                                   Page - BasePageOffset));
+        }
+      }
+
+      VerifyReleaseToOs(Context);
+    }
+
+    // Check the buffer size of PageMap.
+    {
+      scudo::PageReleaseContext Full(BlockSize, /*NumberOfRegions=*/1U,
+                                     /*ReleaseSize=*/RegionSize);
+      Full.ensurePageMapAllocated();
+      scudo::PageReleaseContext Partial(BlockSize, /*NumberOfRegions=*/1U,
+                                        /*ReleaseSize=*/RegionSize - PageSize,
+                                        ReleaseBase);
+      Partial.ensurePageMapAllocated();
+
+      EXPECT_GE(Full.PageMap.getBufferSize(), Partial.PageMap.getBufferSize());
+    }
+
+    while (!FreeList.empty()) {
+      CurrentBatch = FreeList.front();
+      FreeList.pop_front();
+      delete CurrentBatch;
+    }
+  } // Iterate each size class
+}
+
 TEST(ScudoReleaseTest, ReleaseFreeMemoryToOSDefault) {
   testReleaseFreeMemoryToOS<scudo::DefaultSizeClassMap>();
 }
@@ -419,3 +555,36 @@ TEST(ScudoReleaseTest, PageMapMarkRange) {
   testPageMapMarkRange<scudo::FuchsiaSizeClassMap>();
   testPageMapMarkRange<scudo::SvelteSizeClassMap>();
 }
+
+TEST(ScudoReleaseTest, ReleasePartialRegion) {
+  testReleasePartialRegion<scudo::DefaultSizeClassMap>();
+  testReleasePartialRegion<scudo::AndroidSizeClassMap>();
+  testReleasePartialRegion<scudo::FuchsiaSizeClassMap>();
+  testReleasePartialRegion<scudo::SvelteSizeClassMap>();
+}
+
+TEST(ScudoReleaseTest, BufferPool) {
+  constexpr scudo::uptr StaticBufferCount = SCUDO_WORDSIZE - 1;
+  constexpr scudo::uptr StaticBufferSize = 512U;
+
+  // Allocate the buffer pool on the heap because it is quite large (slightly
+  // more than StaticBufferCount * StaticBufferSize * sizeof(uptr)) and it may
+  // not fit in the stack on some platforms.
+  using BufferPool = scudo::BufferPool<StaticBufferCount, StaticBufferSize>;
+  std::unique_ptr<BufferPool> Pool(new BufferPool());
+
+  std::vector<std::pair<scudo::uptr *, scudo::uptr>> Buffers;
+  for (scudo::uptr I = 0; I < StaticBufferCount; ++I) {
+    scudo::uptr *P = Pool->getBuffer(StaticBufferSize);
+    EXPECT_TRUE(Pool->isStaticBufferTestOnly(P, StaticBufferSize));
+    Buffers.emplace_back(P, StaticBufferSize);
+  }
+
+  // The static buffer is supposed to be used up.
+  scudo::uptr *P = Pool->getBuffer(StaticBufferSize);
+  EXPECT_FALSE(Pool->isStaticBufferTestOnly(P, StaticBufferSize));
+
+  Pool->releaseBuffer(P, StaticBufferSize);
+  for (auto &Buffer : Buffers)
+    Pool->releaseBuffer(Buffer.first, Buffer.second);
+}
diff --git a/compiler-rt/lib/scudo/standalone/trusty.cpp b/compiler-rt/lib/scudo/standalone/trusty.cpp
index 592514d4c3a6c..c08a4e6f43319 100644
--- a/compiler-rt/lib/scudo/standalone/trusty.cpp
+++ b/compiler-rt/lib/scudo/standalone/trusty.cpp
@@ -85,6 +85,17 @@ u64 getMonotonicTime() {
          static_cast<u64>(TS.tv_nsec);
 }
 
+u64 getMonotonicTimeFast() {
+#if defined(CLOCK_MONOTONIC_COARSE)
+  timespec TS;
+  clock_gettime(CLOCK_MONOTONIC_COARSE, &TS);
+  return static_cast<u64>(TS.tv_sec) * (1000ULL * 1000 * 1000) +
+         static_cast<u64>(TS.tv_nsec);
+#else
+  return getMonotonicTime();
+#endif
+}
+
 u32 getNumberOfCPUs() { return 0; }
 
 u32 getThreadID() { return 0; }
diff --git a/compiler-rt/lib/scudo/standalone/wrappers_c.inc b/compiler-rt/lib/scudo/standalone/wrappers_c.inc
index 37e336ee09d62..3e495eaa4a39e 100644
--- a/compiler-rt/lib/scudo/standalone/wrappers_c.inc
+++ b/compiler-rt/lib/scudo/standalone/wrappers_c.inc
@@ -54,6 +54,8 @@ INTERFACE WEAK struct SCUDO_MALLINFO SCUDO_PREFIX(mallinfo)(void) {
   return Info;
 }
 
+// On Android, mallinfo2 is an alias of mallinfo, so don't define both.
+#if !SCUDO_ANDROID
 INTERFACE WEAK struct __scudo_mallinfo2 SCUDO_PREFIX(mallinfo2)(void) {
   struct __scudo_mallinfo2 Info = {};
   scudo::StatCounters Stats;
@@ -70,6 +72,7 @@ INTERFACE WEAK struct __scudo_mallinfo2 SCUDO_PREFIX(mallinfo2)(void) {
   Info.fordblks = Info.fsmblks;
   return Info;
 }
+#endif
 
 INTERFACE WEAK void *SCUDO_PREFIX(malloc)(size_t size) {
   return scudo::setErrnoOnNull(SCUDO_ALLOCATOR.allocate(
@@ -188,7 +191,10 @@ INTERFACE WEAK int SCUDO_PREFIX(mallopt)(int param, int value) {
                               static_cast<scudo::sptr>(value));
     return 1;
   } else if (param == M_PURGE) {
-    SCUDO_ALLOCATOR.releaseToOS();
+    SCUDO_ALLOCATOR.releaseToOS(scudo::ReleaseToOS::Force);
+    return 1;
+  } else if (param == M_PURGE_ALL) {
+    SCUDO_ALLOCATOR.releaseToOS(scudo::ReleaseToOS::ForceAll);
     return 1;
   } else {
     scudo::Option option;
diff --git a/compiler-rt/lib/scudo/standalone/wrappers_c_bionic.cpp b/compiler-rt/lib/scudo/standalone/wrappers_c_bionic.cpp
index 18c3bf2c0edfa..1b9fe67d920c3 100644
--- a/compiler-rt/lib/scudo/standalone/wrappers_c_bionic.cpp
+++ b/compiler-rt/lib/scudo/standalone/wrappers_c_bionic.cpp
@@ -32,21 +32,6 @@ static scudo::Allocator<scudo::AndroidConfig, SCUDO_PREFIX(malloc_postinit)>
 #undef SCUDO_ALLOCATOR
 #undef SCUDO_PREFIX
 
-// Svelte MallocDispatch definitions.
-#define SCUDO_PREFIX(name) CONCATENATE(scudo_svelte_, name)
-#define SCUDO_ALLOCATOR SvelteAllocator
-
-extern "C" void SCUDO_PREFIX(malloc_postinit)();
-SCUDO_REQUIRE_CONSTANT_INITIALIZATION
-static scudo::Allocator<scudo::AndroidSvelteConfig,
-                        SCUDO_PREFIX(malloc_postinit)>
-    SCUDO_ALLOCATOR;
-
-#include "wrappers_c.inc"
-
-#undef SCUDO_ALLOCATOR
-#undef SCUDO_PREFIX
-
 // TODO(kostyak): support both allocators.
 INTERFACE void __scudo_print_stats(void) { Allocator.printStats(); }
 
diff --git a/compiler-rt/lib/tsan/rtl/tsan_external.cpp b/compiler-rt/lib/tsan/rtl/tsan_external.cpp
index 19ae174f20a59..463f32d7fdc31 100644
--- a/compiler-rt/lib/tsan/rtl/tsan_external.cpp
+++ b/compiler-rt/lib/tsan/rtl/tsan_external.cpp
@@ -60,14 +60,15 @@ uptr TagFromShadowStackFrame(uptr pc) {
 
 #if !SANITIZER_GO
 
-void ExternalAccess(void *addr, uptr caller_pc, void *tag, AccessType typ) {
+void ExternalAccess(void *addr, uptr caller_pc, uptr tsan_caller_pc, void *tag,
+                    AccessType typ) {
   CHECK_LT(tag, atomic_load(&used_tags, memory_order_relaxed));
   ThreadState *thr = cur_thread();
   if (caller_pc) FuncEntry(thr, caller_pc);
   InsertShadowStackFrameForTag(thr, (uptr)tag);
   bool in_ignored_lib;
   if (!caller_pc || !libignore()->IsIgnored(caller_pc, &in_ignored_lib))
-    MemoryAccess(thr, CALLERPC, (uptr)addr, 1, typ);
+    MemoryAccess(thr, tsan_caller_pc, (uptr)addr, 1, typ);
   FuncExit(thr);
   if (caller_pc) FuncExit(thr);
 }
@@ -112,12 +113,12 @@ void __tsan_external_assign_tag(void *addr, void *tag) {
 
 SANITIZER_INTERFACE_ATTRIBUTE
 void __tsan_external_read(void *addr, void *caller_pc, void *tag) {
-  ExternalAccess(addr, STRIP_PAC_PC(caller_pc), tag, kAccessRead);
+  ExternalAccess(addr, STRIP_PAC_PC(caller_pc), CALLERPC, tag, kAccessRead);
 }
 
 SANITIZER_INTERFACE_ATTRIBUTE
 void __tsan_external_write(void *addr, void *caller_pc, void *tag) {
-  ExternalAccess(addr, STRIP_PAC_PC(caller_pc), tag, kAccessWrite);
+  ExternalAccess(addr, STRIP_PAC_PC(caller_pc), CALLERPC, tag, kAccessWrite);
 }
 }  // extern "C"
 
diff --git a/compiler-rt/test/asan/TestCases/Windows/use_after_return_linkage.cpp b/compiler-rt/test/asan/TestCases/Windows/use_after_return_linkage.cpp
index 48c5065a0fa99..cacf461b4f33a 100644
--- a/compiler-rt/test/asan/TestCases/Windows/use_after_return_linkage.cpp
+++ b/compiler-rt/test/asan/TestCases/Windows/use_after_return_linkage.cpp
@@ -1,5 +1,7 @@
 // Make sure LIBCMT doesn't accidentally get added to the list of DEFAULTLIB
-// directives.  REQUIRES: asan-dynamic-runtime
+// directives.
+
+// REQUIRES: asan-dynamic-runtime
 // RUN: %clang_cl_asan -LD %s | FileCheck %s
 // CHECK: Creating library
 // CHECK-NOT: LIBCMT
diff --git a/compiler-rt/test/asan/TestCases/replaceable_new_delete.cpp b/compiler-rt/test/asan/TestCases/replaceable_new_delete_shared.cpp
similarity index 84%
rename from compiler-rt/test/asan/TestCases/replaceable_new_delete.cpp
rename to compiler-rt/test/asan/TestCases/replaceable_new_delete_shared.cpp
index 677a571bba7d7..6a78ebb3489b9 100644
--- a/compiler-rt/test/asan/TestCases/replaceable_new_delete.cpp
+++ b/compiler-rt/test/asan/TestCases/replaceable_new_delete_shared.cpp
@@ -1,4 +1,4 @@
-// Ensure that operator new/delete are still replaceable.
+// Ensure that operator new/delete are still replaceable using shared-libsan.
 
 // FIXME: Weak symbols aren't supported on Windows, although some code in
 // compiler-rt already exists to solve this problem. We should probably define
@@ -6,7 +6,6 @@
 // UNSUPPORTED: target={{.*windows.*}}
 
 // RUN: %clangxx %s -o %t -fsanitize=address -shared-libsan && not %run %t 2>&1 | FileCheck %s
-// RUN: %clangxx %s -o %t -fsanitize=address -static-libsan && not %run %t 2>&1 | FileCheck %s
 
 #include <cstdio>
 #include <cstdlib>
diff --git a/compiler-rt/test/asan/TestCases/replaceable_new_delete_static.cpp b/compiler-rt/test/asan/TestCases/replaceable_new_delete_static.cpp
new file mode 100644
index 0000000000000..98fe610ae72d5
--- /dev/null
+++ b/compiler-rt/test/asan/TestCases/replaceable_new_delete_static.cpp
@@ -0,0 +1,35 @@
+// Ensure that operator new/delete are still replaceable using static-libsan.
+
+// FIXME: Weak symbols aren't supported on Windows, although some code in
+// compiler-rt already exists to solve this problem. We should probably define
+// the new/delete interceptors as "weak" using those workarounds as well.
+// UNSUPPORTED: target={{.*windows.*}}
+
+// darwin only supports `shared-libsan`.
+// REQUIRES: asan-static-runtime
+
+// RUN: %clangxx %s -o %t -fsanitize=address -static-libsan && not %run %t 2>&1 | FileCheck %s
+
+#include <cstdio>
+#include <cstdlib>
+#include <new>
+
+void *operator new[](size_t size) {
+  fprintf(stderr, "replaced new\n");
+  return malloc(size);
+}
+
+void operator delete[](void *ptr) noexcept {
+  fprintf(stderr, "replaced delete\n");
+  return free(ptr);
+}
+
+int main(int argc, char **argv) {
+  // CHECK: replaced new
+  char *x = new char[5];
+  // CHECK: replaced delete
+  delete[] x;
+  // CHECK: ERROR: AddressSanitizer
+  *x = 13;
+  return 0;
+}
diff --git a/compiler-rt/test/asan/TestCases/report_error_summary.cpp b/compiler-rt/test/asan/TestCases/report_error_summary.cpp
new file mode 100644
index 0000000000000..d565d2add7793
--- /dev/null
+++ b/compiler-rt/test/asan/TestCases/report_error_summary.cpp
@@ -0,0 +1,23 @@
+// RUN: %clangxx_asan -O0 %s -o %t && not %run %t 2>&1 | FileCheck %s
+
+// FIXME: Doesn't work with DLLs
+// XFAIL: win32-dynamic-asan
+
+#include <stdio.h>
+
+// Required for ld64 macOS 12.0+
+__attribute__((weak)) extern "C" void foo() {}
+
+extern "C" void __sanitizer_report_error_summary(const char *summary) {
+  fprintf(stderr, "test_report_error_summary\n");
+  // CHECK: test_report_error_summary
+  fflush(stderr);
+}
+
+char *x;
+
+int main() {
+  x = new char[20];
+  delete[] x;
+  return x[0];
+}
diff --git a/compiler-rt/test/fuzzer/dataflow.test b/compiler-rt/test/fuzzer/dataflow.test
index c7d703dbe82f2..9f35374916798 100644
--- a/compiler-rt/test/fuzzer/dataflow.test
+++ b/compiler-rt/test/fuzzer/dataflow.test
@@ -97,6 +97,11 @@ L20_FUZZM-NEXT: F1 00000000000000001000{{$}}
 L20_FUZZM-NEXT: F2 00000000000000000001{{$}}
 L20_FUZZM-NOT: F
 
+# Don't crash with missing data_flow args.
+RUN: rm -rf %t-DFT
+RUN: %t-ThreeFunctionsTest -collect_data_flow=%t-ThreeFunctionsTestDF
+RUN: %t-ThreeFunctionsTest -data_flow_trace=%t-DFT %t/IN/FUZZMU
+
 # Test libFuzzer's built in DFT collection.
 RUN: rm -rf %t-DFT
 RUN: %t-ThreeFunctionsTest  -collect_data_flow=%t-ThreeFunctionsTestDF -data_flow_trace=%t-DFT %t/IN/FUZZMU
diff --git a/compiler-rt/test/hwasan/TestCases/Linux/decorate-proc-maps.c b/compiler-rt/test/hwasan/TestCases/Linux/decorate-proc-maps.c
index 40e8c0ed8bb75..1c4e2e7c76d3b 100644
--- a/compiler-rt/test/hwasan/TestCases/Linux/decorate-proc-maps.c
+++ b/compiler-rt/test/hwasan/TestCases/Linux/decorate-proc-maps.c
@@ -48,7 +48,7 @@ void *ThreadFn(void *arg) {
 int main(void) {
   pthread_t t;
   void * volatile res = malloc(100);
-  void * volatile res2 = malloc(100000);
+  void * volatile res2 = malloc(1000000);
   pthread_create(&t, 0, ThreadFn, 0);
   pthread_join(t, 0);
   int ret_val = (int)(size_t)res;
diff --git a/compiler-rt/test/hwasan/TestCases/realloc-test.cpp b/compiler-rt/test/hwasan/TestCases/realloc-test.cpp
index 136346f57c2ba..e103b609df116 100644
--- a/compiler-rt/test/hwasan/TestCases/realloc-test.cpp
+++ b/compiler-rt/test/hwasan/TestCases/realloc-test.cpp
@@ -4,12 +4,12 @@
 
 #include <assert.h>
 #include <sanitizer/hwasan_interface.h>
+#include <stdlib.h>
 
 #ifdef REALLOCARRAY
 extern "C" void *reallocarray(void *, size_t nmemb, size_t size);
 #define REALLOC(p, s) reallocarray(p, 1, s)
 #else
-#include <stdlib.h>
 #define REALLOC(p, s) realloc(p, s)
 #endif
 
@@ -41,4 +41,5 @@ int main() {
   assert(x4 != x3);
   assert(x4[0] == 10 && x4[1] == 20 && x4[2] == 30 && x4[3] == 40 &&
          x4[4] == 50);
+  free(x4);
 }
diff --git a/compiler-rt/test/hwasan/TestCases/use-after-free.c b/compiler-rt/test/hwasan/TestCases/use-after-free.c
index bd09bbf889df1..24546374341e3 100644
--- a/compiler-rt/test/hwasan/TestCases/use-after-free.c
+++ b/compiler-rt/test/hwasan/TestCases/use-after-free.c
@@ -27,7 +27,7 @@ int main() {
   // of the fault.
   // CHECK: in main {{.*}}use-after-free.c:[[@LINE-6]]
   // Offset is 5 or 11 depending on left/right alignment.
-  // CHECK: is a small unallocated heap chunk; size: 32 offset: {{5|11}}
+  // CHECK: is a small unallocated heap chunk; size: {{16|32}} offset: {{5|11}}
   // CHECK: Cause: use-after-free
   // CHECK: is located 5 bytes inside a 10-byte region
   //
diff --git a/compiler-rt/test/sanitizer_common/TestCases/Linux/argp_parse.c b/compiler-rt/test/sanitizer_common/TestCases/Linux/argp_parse.c
new file mode 100644
index 0000000000000..ec78f276c9eaf
--- /dev/null
+++ b/compiler-rt/test/sanitizer_common/TestCases/Linux/argp_parse.c
@@ -0,0 +1,60 @@
+// RUN: %clang %s -o %t && %run %t -o baz
+
+// argp_parse is glibc specific.
+// UNSUPPORTED: android, target={{.*(freebsd|netbsd).*}}
+
+#include <argp.h>
+#include <assert.h>
+#include <stdlib.h>
+#include <string.h>
+
+struct test {
+  const char *option_value;
+};
+
+static const struct argp_option options[] = {
+    {"option", 'o', "OPTION", 0, "Option", 0},
+    {NULL, 0, NULL, 0, NULL, 0},
+};
+
+static error_t parser(int key, char *arg, struct argp_state *state) {
+  if (key == 'o') {
+    ((struct test *)(state->input))->option_value = arg;
+    return 0;
+  }
+  return ARGP_ERR_UNKNOWN;
+}
+
+static struct argp argp = {.options = options, .parser = parser};
+
+void test_nulls(char *argv0) {
+  char *argv[] = {argv0, NULL};
+  int res = argp_parse(NULL, 1, argv, 0, NULL, NULL);
+  assert(res == 0);
+}
+
+void test_synthetic(char *argv0) {
+  char *argv[] = {argv0, "-o", "foo", "bar", NULL};
+  struct test t = {NULL};
+  int arg_index;
+  int res = argp_parse(&argp, 4, argv, 0, &arg_index, &t);
+  assert(res == 0);
+  assert(arg_index == 3);
+  assert(strcmp(t.option_value, "foo") == 0);
+}
+
+void test_real(int argc, char **argv) {
+  struct test t = {NULL};
+  int arg_index;
+  int res = argp_parse(&argp, argc, argv, 0, &arg_index, &t);
+  assert(res == 0);
+  assert(arg_index == 3);
+  assert(strcmp(t.option_value, "baz") == 0);
+}
+
+int main(int argc, char **argv) {
+  test_nulls(argv[0]);
+  test_synthetic(argv[0]);
+  test_real(argc, argv);
+  return EXIT_SUCCESS;
+}
diff --git a/compiler-rt/test/sanitizer_common/TestCases/Linux/odd_stack_size.cpp b/compiler-rt/test/sanitizer_common/TestCases/Linux/odd_stack_size.cpp
new file mode 100644
index 0000000000000..fc31212b7f18b
--- /dev/null
+++ b/compiler-rt/test/sanitizer_common/TestCases/Linux/odd_stack_size.cpp
@@ -0,0 +1,35 @@
+// RUN: %clangxx -O1 %s -o %t && %run %t
+// UNSUPPORTED: android
+
+// Fail on powerpc64 bots with:
+// AddressSanitizer: CHECK failed: asan_thread.cpp:315 "((AddrIsInStack((uptr)&local))) != (0)"
+// https://lab.llvm.org/buildbot/#/builders/18/builds/8162
+// UNSUPPORTED: target=powerpc64{{.*}}
+
+#include <assert.h>
+#include <stdlib.h>
+#include <sys/resource.h>
+#include <sys/wait.h>
+#include <unistd.h>
+
+int main(int argc, char **argv) {
+  if (getenv("SANITIZER_TEST_REEXECED"))
+    exit(0);
+  struct rlimit rl;
+  assert(!getrlimit(RLIMIT_STACK, &rl));
+  struct rlimit rl_new = rl;
+  rl_new.rlim_cur = 17351;
+  assert(!setrlimit(RLIMIT_STACK, &rl_new));
+  int pid = fork();
+  assert(pid >= 0);
+  if (pid == 0) {
+    const char *envp[] = {"SANITIZER_TEST_REEXECED=1", nullptr};
+    execve(argv[0], argv, const_cast<char **>(envp));
+    assert(false);
+  }
+  int status;
+  while (waitpid(-1, &status, __WALL) != pid) {
+  }
+  assert(WIFEXITED(status) && WEXITSTATUS(status) == 0);
+  return 0;
+}
diff --git a/compiler-rt/test/tsan/Darwin/external-swift-debugging.cpp b/compiler-rt/test/tsan/Darwin/external-swift-debugging.cpp
index 72335edf17896..64475a3e97373 100644
--- a/compiler-rt/test/tsan/Darwin/external-swift-debugging.cpp
+++ b/compiler-rt/test/tsan/Darwin/external-swift-debugging.cpp
@@ -1,6 +1,7 @@
 // RUN: %clangxx_tsan %s -o %t
 // RUN: %deflake %run %t 2>&1 | FileCheck %s
 
+#include <dlfcn.h>
 #include <thread>
 
 #import "../test.h"
@@ -13,6 +14,9 @@ int __tsan_get_report_data(void *report, const char **description, int *count,
                            int *unique_tid_count, void **sleep_trace,
                            unsigned long trace_size);
 int __tsan_get_report_tag(void *report, unsigned long *tag);
+int __tsan_get_report_mop(void *report, unsigned long idx, int *tid, void **addr,
+                          int *size, int *write, int *atomic, void **trace,
+                          unsigned long trace_size);
 }
 
 __attribute__((no_sanitize("thread"), noinline))
@@ -25,7 +29,10 @@ int main(int argc, char *argv[]) {
   barrier_init(&barrier, 2);
   fprintf(stderr, "Start.\n");
   // CHECK: Start.
-  
+
+  fprintf(stderr, "ExternalWrite function address: %p\n", &ExternalWrite);
+  // CHECK: ExternalWrite function address: [[ExternalWrite_addr:0x[0-9a-z]+]]
+
   void *opaque_object = malloc(16);
   std::thread t1([opaque_object] {
     ExternalWrite(opaque_object);
@@ -37,7 +44,9 @@ int main(int argc, char *argv[]) {
   });
   // CHECK: WARNING: ThreadSanitizer: Swift access race
   // CHECK: Modifying access of Swift variable at {{.*}} by thread {{.*}}
+  // CHECK:   #0 ExternalWrite
   // CHECK: Previous modifying access of Swift variable at {{.*}} by thread {{.*}}
+  // CHECK:   #0 ExternalWrite
   // CHECK: SUMMARY: ThreadSanitizer: Swift access race
   t1.join();
   t2.join();
@@ -55,13 +64,28 @@ __tsan_on_report(void *report) {
   __tsan_get_report_data(report, &description, &count, &stack_count, &mop_count,
                          &loc_count, &mutex_count, &thread_count,
                          &unique_tid_count, sleep_trace, 16);
-  fprintf(stderr, "report type = '%s', count = %d\n", description, count);
-  // CHECK: report type = 'external-race', count = 0
+  fprintf(stderr, "report type = '%s', count = %d, mop_count = %d\n", description, count, mop_count);
+  // CHECK: report type = 'external-race', count = 0, mop_count = 2
 
   unsigned long tag;
   __tsan_get_report_tag(report, &tag);
   fprintf(stderr, "tag = %ld\n", tag);
   // CHECK: tag = 1
+
+  int tid, size, write, atomic;
+  void *addr;
+  void *trace[16] = {0};
+  __tsan_get_report_mop(report, /*idx=*/0, &tid, &addr, &size, &write, &atomic,
+                        trace, 16);
+  fprintf(stderr, "Racy write trace (1 of 2):\n");
+  for (int i = 0; i < 16 && trace[i]; i++) {
+    Dl_info info;
+    dladdr(trace[i], &info);
+    fprintf(stderr, "  %d: frame: %p, function: %p %s\n", i, trace[i],
+            info.dli_saddr, info.dli_sname);
+  }
+  // Ensure ExternalWrite() function is top of trace
+  // CHECK: 0: frame: 0x{{[0-9a-z]+}}, function: [[ExternalWrite_addr]] _Z13ExternalWritePv
 }
 
 // CHECK: Done.
diff --git a/compiler-rt/www/index.html b/compiler-rt/www/index.html
index cc772c804213f..eaeb490c550c6 100644
--- a/compiler-rt/www/index.html
+++ b/compiler-rt/www/index.html
@@ -38,7 +38,7 @@ <h1>"compiler-rt" runtime libraries</h1>
     <ul>
     <li><a href="https://clang.llvm.org/docs/AddressSanitizer.html">AddressSanitizer</a></li>
     <li><a href="https://clang.llvm.org/docs/ThreadSanitizer.html">ThreadSanitizer</a></li>
-    <li><a href="https://clang.llvm.org/docs/UsersManual.html#opt-fsanitize-undefined">UndefinedBehaviorSanitizer</a></li>
+    <li><a href="https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html">UndefinedBehaviorSanitizer</a></li>
     <li><a href="https://clang.llvm.org/docs/MemorySanitizer.html">MemorySanitizer</a></li>
     <li><a href="https://clang.llvm.org/docs/LeakSanitizer.html">LeakSanitizer</a></li>
     <li><a href="https://clang.llvm.org/docs/DataFlowSanitizer.html">DataFlowSanitizer</a></li>
diff --git a/cross-project-tests/debuginfo-tests/dexter/dex/debugger/Debuggers.py b/cross-project-tests/debuginfo-tests/dexter/dex/debugger/Debuggers.py
index fbfa629faf8b2..48cb7e1a07898 100644
--- a/cross-project-tests/debuginfo-tests/dexter/dex/debugger/Debuggers.py
+++ b/cross-project-tests/debuginfo-tests/dexter/dex/debugger/Debuggers.py
@@ -18,7 +18,6 @@
 from dex.utils import get_root_directory, Timer
 from dex.utils.Environment import is_native_windows
 from dex.utils.Exceptions import ToolArgumentError
-from dex.utils.Warning import warn
 from dex.utils.Exceptions import DebuggerException
 
 from dex.debugger.DebuggerControllers.DefaultController import DefaultController
@@ -48,9 +47,10 @@ def _warn_meaningless_option(context, option):
     if hasattr(context.options, 'list_debuggers'):
         return
 
-    warn(context,
-         'option <y>"{}"</> is meaningless with this debugger'.format(option),
-         '--debugger={}'.format(context.options.debugger))
+    context.logger.warning(
+         f'option "{option}" is meaningless with this debugger',
+         enable_prefix=True,
+         flag=f'--debugger={context.options.debugger}')
 
 
 def add_debugger_tool_base_arguments(parser, defaults):
@@ -114,6 +114,14 @@ def add_debugger_tool_arguments(parser, context, defaults):
         action='store_true',
         default=False,
         help='pass the debugger paths relative to --source-root-dir')
+    parser.add_argument(
+        '--target-run-args',
+        type=str,
+        metavar='<flags>',
+        default='',
+        help='command line arguments for the test program, in addition to any '
+             'provided by DexCommandLine')
+
 
 def handle_debugger_tool_base_options(context, defaults):  # noqa
     options = context.options
diff --git a/cross-project-tests/debuginfo-tests/dexter/dex/debugger/dbgeng/dbgeng.py b/cross-project-tests/debuginfo-tests/dexter/dex/debugger/dbgeng/dbgeng.py
index c89920d5cd864..8e015669b732e 100644
--- a/cross-project-tests/debuginfo-tests/dexter/dex/debugger/dbgeng/dbgeng.py
+++ b/cross-project-tests/debuginfo-tests/dexter/dex/debugger/dbgeng/dbgeng.py
@@ -96,7 +96,7 @@ def delete_breakpoints(self, ids):
         raise NotImplementedError('delete_conditional_breakpoint is not yet implemented by dbgeng')
 
     def launch(self, cmdline):
-        assert len(cmdline) == 0, "Command lines unimplemented for dbgeng right now"
+        assert len(cmdline) == 0 and not self.context.options.target_run_args, "Command lines unimplemented for dbgeng right now"
         # We are, by this point, already launched.
         self.step_info = probe_process.probe_state(self.client)
 
diff --git a/cross-project-tests/debuginfo-tests/dexter/dex/debugger/lldb/LLDB.py b/cross-project-tests/debuginfo-tests/dexter/dex/debugger/lldb/LLDB.py
index 703f9c4744c25..4ab693c6d1813 100644
--- a/cross-project-tests/debuginfo-tests/dexter/dex/debugger/lldb/LLDB.py
+++ b/cross-project-tests/debuginfo-tests/dexter/dex/debugger/lldb/LLDB.py
@@ -9,6 +9,7 @@
 
 import imp
 import os
+import shlex
 from subprocess import CalledProcessError, check_output, STDOUT
 import sys
 
@@ -171,6 +172,8 @@ def delete_breakpoints(self, ids):
             self._target.BreakpointDelete(id)
 
     def launch(self, cmdline):
+        if self.context.options.target_run_args:
+          cmdline += shlex.split(self.context.options.target_run_args)
         self._process = self._target.LaunchSimple(cmdline, None, os.getcwd())
         if not self._process or self._process.GetNumThreads() == 0:
             raise DebuggerException('could not launch process')
diff --git a/cross-project-tests/debuginfo-tests/dexter/dex/debugger/visualstudio/VisualStudio.py b/cross-project-tests/debuginfo-tests/dexter/dex/debugger/visualstudio/VisualStudio.py
index 281d5f58ec15f..1db20c7788072 100644
--- a/cross-project-tests/debuginfo-tests/dexter/dex/debugger/visualstudio/VisualStudio.py
+++ b/cross-project-tests/debuginfo-tests/dexter/dex/debugger/visualstudio/VisualStudio.py
@@ -10,7 +10,7 @@
 import imp
 import os
 import sys
-from pathlib import PurePath
+from pathlib import PurePath, Path
 from collections import defaultdict, namedtuple
 
 from dex.command.CommandBase import StepExpectInfo
@@ -249,7 +249,13 @@ def _fetch_property(self, props, name):
         assert False, "Couldn't find property {}".format(name)
 
     def launch(self, cmdline):
+        exe_path = Path(self.context.options.executable)
+        self.context.logger.note(f"VS: Using executable: '{exe_path}'")
         cmdline_str = ' '.join(cmdline)
+        if self.context.options.target_run_args:
+          cmdline_str += f" {self.context.options.target_run_args}"
+        if cmdline_str:
+          self.context.logger.note(f"VS: Using executable args: '{cmdline_str}'")
 
         # In a slightly baroque manner, lookup the VS project that runs when
         # you click "run", and set its command line options to the desired
@@ -259,6 +265,7 @@ def launch(self, cmdline):
         ActiveConfiguration = self._fetch_property(project.Properties, 'ActiveConfiguration').Object
         ActiveConfiguration.DebugSettings.CommandArguments = cmdline_str
 
+        self.context.logger.note("Launching VS debugger...")
         self._fn_go(False)
 
     def step(self):
diff --git a/cross-project-tests/debuginfo-tests/dexter/dex/tools/Main.py b/cross-project-tests/debuginfo-tests/dexter/dex/tools/Main.py
index 78fb4f7b7247a..c69ffab66c4e4 100644
--- a/cross-project-tests/debuginfo-tests/dexter/dex/tools/Main.py
+++ b/cross-project-tests/debuginfo-tests/dexter/dex/tools/Main.py
@@ -18,6 +18,7 @@
 from dex.utils import ExtArgParse as argparse
 from dex.utils import get_root_directory
 from dex.utils.Exceptions import Error, ToolArgumentError
+from dex.utils.Logging import Logger
 from dex.utils.UnitTests import unit_tests_ok
 from dex.utils.Version import version
 from dex.utils import WorkingDirectory
@@ -145,6 +146,11 @@ def tool_main(context, tool, args):
             context.o.green('{}\n'.format(context.version))
             return ReturnCode.OK
 
+        if options.verbose:
+            context.logger.verbosity = 2
+        elif options.no_warnings:
+            context.logger.verbosity = 0
+
         if (options.unittest != 'off' and not unit_tests_ok(context)):
             raise Error('<d>unit test failures</>')
 
@@ -171,6 +177,7 @@ class Context(object):
 
     def __init__(self):
         self.o: PrettyOutput = None
+        self.logger: Logger = None
         self.working_directory: str = None
         self.options: dict = None
         self.version: str = None
@@ -182,6 +189,7 @@ def main() -> ReturnCode:
     context = Context()
 
     with PrettyOutput() as context.o:
+        context.logger = Logger(context.o)
         try:
             context.root_directory = get_root_directory()
             # Flag some strings for auto-highlighting.
@@ -192,8 +200,7 @@ def main() -> ReturnCode:
             module = _import_tool_module(tool_name)
             return tool_main(context, module.Tool(context), args)
         except Error as e:
-            context.o.auto(
-                '\nerror: {}\n'.format(str(e)), stream=PrettyOutput.stderr)
+            context.logger.error(str(e))
             try:
                 if context.options.error_debug:
                     raise
diff --git a/cross-project-tests/debuginfo-tests/dexter/dex/tools/TestToolBase.py b/cross-project-tests/debuginfo-tests/dexter/dex/tools/TestToolBase.py
index 5d976ea98145d..349a6f06b0182 100644
--- a/cross-project-tests/debuginfo-tests/dexter/dex/tools/TestToolBase.py
+++ b/cross-project-tests/debuginfo-tests/dexter/dex/tools/TestToolBase.py
@@ -17,7 +17,7 @@
 from dex.debugger.Debuggers import handle_debugger_tool_options
 from dex.heuristic.Heuristic import add_heuristic_tool_arguments
 from dex.tools.ToolBase import ToolBase
-from dex.utils import get_root_directory, warn
+from dex.utils import get_root_directory
 from dex.utils.Exceptions import Error, ToolArgumentError
 from dex.utils.ReturnCode import ReturnCode
 
@@ -53,8 +53,9 @@ def handle_options(self, defaults):
         options = self.context.options
 
         if not options.builder and (options.cflags or options.ldflags):
-            warn(self.context, '--cflags and --ldflags will be ignored when not'
-                               ' using --builder')
+            self.context.logger.warning(
+                '--cflags and --ldflags will be ignored when not using --builder',
+                enable_prefix=True)
 
         if options.vs_solution:
             options.vs_solution = os.path.abspath(options.vs_solution)
diff --git a/cross-project-tests/debuginfo-tests/dexter/dex/tools/ToolBase.py b/cross-project-tests/debuginfo-tests/dexter/dex/tools/ToolBase.py
index eb6ba94c2de82..53274d3850067 100644
--- a/cross-project-tests/debuginfo-tests/dexter/dex/tools/ToolBase.py
+++ b/cross-project-tests/debuginfo-tests/dexter/dex/tools/ToolBase.py
@@ -60,7 +60,7 @@ class defaults(object):
             '--verbose',
             action='store_true',
             default=False,
-            help='enable verbose output')
+            help='enable verbose output (overrides --no-warnings)')
         self.parser.add_argument(
             '-V',
             '--version',
diff --git a/cross-project-tests/debuginfo-tests/dexter/dex/utils/Logging.py b/cross-project-tests/debuginfo-tests/dexter/dex/utils/Logging.py
new file mode 100644
index 0000000000000..11386b4768844
--- /dev/null
+++ b/cross-project-tests/debuginfo-tests/dexter/dex/utils/Logging.py
@@ -0,0 +1,44 @@
+# DExTer : Debugging Experience Tester
+# ~~~~~~   ~         ~~         ~   ~~
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+"""Utility functions for producing command line warnings."""
+
+from dex.utils import PrettyOutput
+
+class Logger(object):
+    def __init__(self, pretty_output: PrettyOutput):
+        self.o = pretty_output
+        self.error_color = self.o.red
+        self.warning_color = self.o.yellow
+        self.note_color = self.o.default
+        self.verbosity = 1
+
+    def error(self, msg, enable_prefix=True, flag=None):
+        if self.verbosity < 0:
+            return
+        if enable_prefix:
+            msg = f'error: {msg}'
+        if flag:
+           msg = f'{msg} <y>[{flag}]</>'
+        self.error_color('{}\n'.format(msg), stream=PrettyOutput.stderr)
+
+    def warning(self, msg, enable_prefix=True, flag=None):
+        if self.verbosity < 1:
+            return
+        if enable_prefix:
+            msg = f'warning: {msg}'
+        if flag:
+           msg = f'{msg} <y>[{flag}]</>'
+        self.warning_color('{}\n'.format(msg), stream=PrettyOutput.stderr)
+
+    def note(self, msg, enable_prefix=True, flag=None):
+        if self.verbosity < 2:
+            return
+        if enable_prefix:
+            msg = f'note: {msg}'
+        if flag:
+           msg = f'{msg} <y>[{flag}]</>'
+        self.note_color('{}\n'.format(msg), stream=PrettyOutput.stderr)
diff --git a/cross-project-tests/debuginfo-tests/dexter/dex/utils/Warning.py b/cross-project-tests/debuginfo-tests/dexter/dex/utils/Warning.py
deleted file mode 100644
index 402861aaed543..0000000000000
--- a/cross-project-tests/debuginfo-tests/dexter/dex/utils/Warning.py
+++ /dev/null
@@ -1,18 +0,0 @@
-# DExTer : Debugging Experience Tester
-# ~~~~~~   ~         ~~         ~   ~~
-#
-# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-# See https://llvm.org/LICENSE.txt for license information.
-# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-"""Utility functions for producing command line warnings."""
-
-
-def warn(context, msg, flag=None):
-    if context.options.no_warnings:
-        return
-
-    msg = msg.rstrip()
-    if flag:
-        msg = '{} <y>[{}]</>'.format(msg, flag)
-
-    context.o.auto('warning: <d>{}</>\n'.format(msg))
diff --git a/cross-project-tests/debuginfo-tests/dexter/dex/utils/WorkingDirectory.py b/cross-project-tests/debuginfo-tests/dexter/dex/utils/WorkingDirectory.py
index 04f9b6435334c..06d776f0501a9 100644
--- a/cross-project-tests/debuginfo-tests/dexter/dex/utils/WorkingDirectory.py
+++ b/cross-project-tests/debuginfo-tests/dexter/dex/utils/WorkingDirectory.py
@@ -12,7 +12,6 @@
 import time
 
 from dex.utils.Exceptions import Error
-from dex.utils.Warning import warn
 
 class WorkingDirectory(object):
     def __init__(self, context, *args, **kwargs):
@@ -42,5 +41,5 @@ def __exit__(self, *args):
             except OSError:
                 time.sleep(0.1)
 
-        warn(self.context, '"{}" left in place (couldn\'t delete)\n'.format(self.path))
+        self.context.logger.warning(f'"{self.path}" left in place (couldn\'t delete)', enable_prefix=True)
         return
diff --git a/cross-project-tests/debuginfo-tests/dexter/dex/utils/__init__.py b/cross-project-tests/debuginfo-tests/dexter/dex/utils/__init__.py
index ac08139d5680d..ae2dea3f2c5ba 100644
--- a/cross-project-tests/debuginfo-tests/dexter/dex/utils/__init__.py
+++ b/cross-project-tests/debuginfo-tests/dexter/dex/utils/__init__.py
@@ -12,10 +12,11 @@
 from dex.utils.PrettyOutputBase import PreserveAutoColors
 from dex.utils.RootDirectory import get_root_directory
 from dex.utils.Timer import Timer
-from dex.utils.Warning import warn
 from dex.utils.WorkingDirectory import WorkingDirectory
 
 if is_native_windows():
     from dex.utils.windows.PrettyOutput import PrettyOutput
 else:
     from dex.utils.posix.PrettyOutput import PrettyOutput
+
+from dex.utils.Logging import Logger
diff --git a/cross-project-tests/debuginfo-tests/dexter/feature_tests/subtools/test/target_run_args.c b/cross-project-tests/debuginfo-tests/dexter/feature_tests/subtools/test/target_run_args.c
new file mode 100644
index 0000000000000..11004adce9121
--- /dev/null
+++ b/cross-project-tests/debuginfo-tests/dexter/feature_tests/subtools/test/target_run_args.c
@@ -0,0 +1,15 @@
+// The dbgeng driver doesn't support --target-run-args yet.
+// UNSUPPORTED: system-windows
+//
+// RUN: %dexter_regression_test --target-run-args "a b 'c d'" -- %s | FileCheck %s
+// CHECK: target_run_args.c:
+
+int main(int argc, const char **argv) {
+  if (argc == 4)
+    return 0; // DexLabel('retline')
+
+  return 1; // DexUnreachable()
+}
+
+// DexExpectWatchValue('argc', '4', on_line=ref('retline'))
+// DexExpectWatchValue('argv[1][0]', "'a'", on_line=ref('retline'))
diff --git a/cross-project-tests/debuginfo-tests/dexter/feature_tests/subtools/test/target_run_args_with_command.c b/cross-project-tests/debuginfo-tests/dexter/feature_tests/subtools/test/target_run_args_with_command.c
new file mode 100644
index 0000000000000..f329cb9d71c5f
--- /dev/null
+++ b/cross-project-tests/debuginfo-tests/dexter/feature_tests/subtools/test/target_run_args_with_command.c
@@ -0,0 +1,16 @@
+// The dbgeng driver doesn't support --target-run-args yet.
+// UNSUPPORTED: system-windows
+//
+// RUN: %dexter_regression_test --target-run-args "a b 'c d'" -- %s | FileCheck %s
+// CHECK: target_run_args_with_command.c:
+
+int main(int argc, const char **argv) {
+  if (argc == 6)
+    return 0; // DexLabel('retline')
+
+  return 1; // DexUnreachable()
+}
+
+// DexCommandLine(['e', 'f'])
+// DexExpectWatchValue('argc', '6', on_line=ref('retline'))
+// DexExpectWatchValue('argv[1][0]', "'e'", on_line=ref('retline'))
diff --git a/flang/CMakeLists.txt b/flang/CMakeLists.txt
index b048ea220e20c..9be6cd7eb1f59 100644
--- a/flang/CMakeLists.txt
+++ b/flang/CMakeLists.txt
@@ -430,11 +430,9 @@ if (FLANG_BUILD_TOOLS)
 endif()
 add_subdirectory(runtime)
 
-option(FLANG_BUILD_EXAMPLES "Build Flang example programs by default." OFF)
-if (FLANG_BUILD_EXAMPLES AND FLANG_STANDALONE_BUILD)
-  message(FATAL_ERROR "Examples are not supported in out-of-tree builds.")
+if (LLVM_INCLUDE_EXAMPLES)
+  add_subdirectory(examples)
 endif()
-add_subdirectory(examples)
 
 if (FLANG_INCLUDE_TESTS)
   add_subdirectory(test)
diff --git a/flang/cmake/modules/AddFlang.cmake b/flang/cmake/modules/AddFlang.cmake
index c54c30dbf6df5..4332eb83c013e 100644
--- a/flang/cmake/modules/AddFlang.cmake
+++ b/flang/cmake/modules/AddFlang.cmake
@@ -127,4 +127,3 @@ macro(add_flang_symlink name dest)
   # Always generate install targets
   llvm_install_symlink(FLANG ${name} ${dest} ALWAYS_GENERATE)
 endmacro()
-
diff --git a/flang/docs/Extensions.md b/flang/docs/Extensions.md
index 7f685c72eada3..4f68007418c51 100644
--- a/flang/docs/Extensions.md
+++ b/flang/docs/Extensions.md
@@ -199,10 +199,10 @@ end
 * DATA statement initialization is allowed for procedure pointers outside
   structure constructors.
 * Nonstandard intrinsic functions: ISNAN, SIZEOF
-* A forward reference to a default INTEGER scalar dummy argument is
-  permitted to appear in a specification expression, such as an array
-  bound, in a scope with IMPLICIT NONE(TYPE) if the name
-  of the dummy argument would have caused it to be implicitly typed
+* A forward reference to a default INTEGER scalar dummy argument or
+  `COMMON` block variable is permitted to appear in a specification
+  expression, such as an array bound, in a scope with IMPLICIT NONE(TYPE)
+  if the name of the variable would have caused it to be implicitly typed
   as default INTEGER if IMPLICIT NONE(TYPE) were absent.
 * OPEN(ACCESS='APPEND') is interpreted as OPEN(POSITION='APPEND')
   to ease porting from Sun Fortran.
@@ -266,6 +266,9 @@ end
   enforce it and the constraint is not necessary for a correct
   implementation.
 * A label may follow a semicolon in fixed form source.
+* A scalar logical dummy argument to a `BIND(C)` procedure does
+  not have to have `KIND=C_BOOL` since it can be converted to/from
+  `_Bool` without loss of information.
 
 ### Extensions supported when enabled by options
 
@@ -557,6 +560,9 @@ end module
   obsolete module file from a previous compilation and then overwriting
   that file later.
 
+* F18 allows `OPTIONAL` dummy arguments to interoperable procedures
+  unless they are `VALUE` (C865).
+
 ## De Facto Standard Features
 
 * `EXTENDS_TYPE_OF()` returns `.TRUE.` if both of its arguments have the
diff --git a/flang/docs/FlangDriver.md b/flang/docs/FlangDriver.md
index e87df93907ef2..6c2a473820634 100644
--- a/flang/docs/FlangDriver.md
+++ b/flang/docs/FlangDriver.md
@@ -485,7 +485,7 @@ reports an error diagnostic and returns `nullptr`.
 For in-tree plugins, there is the CMake flag `FLANG_PLUGIN_SUPPORT`, enabled by
 default, that controls the exporting of executable symbols from `flang-new`,
 which plugins need access to. Additionally, there is the CMake flag
-`FLANG_BUILD_EXAMPLES`, turned off by default, that is used to control if the
+`LLVM_BUILD_EXAMPLES`, turned off by default, that is used to control if the
 example programs are built. This includes plugins that are in the
 `flang/example` directory and added as a `sub_directory` to the
 `flang/examples/CMakeLists.txt`, for example, the `PrintFlangFunctionNames`
diff --git a/flang/docs/HighLevelFIR.md b/flang/docs/HighLevelFIR.md
index 033ac4102ba07..8c671ae216678 100644
--- a/flang/docs/HighLevelFIR.md
+++ b/flang/docs/HighLevelFIR.md
@@ -652,7 +652,6 @@ Syntax:
 %element = hlfir.apply %array_expr %i, %j: (hlfir.expr<?x?xi32>) -> i32
 ```
 
-
 #### Introducing operations for transformational intrinsic functions
 
 Motivation: Represent transformational intrinsics functions at a high-level so
@@ -701,6 +700,39 @@ call will probably be used since there is little point to keep them high level:
 - selected_char_kind, selected_int_kind, selected_real_kind that returns scalar
   integers
 
+#### Introducing operations for composed intrinsic functions
+
+Motivation: optimize commonly composed intrinsic functions (e.g.
+MATMUL(TRANSPOSE(a), b)). This optimization is implemented in Classic Flang.
+
+An operation and runtime function will be added for each commonly used
+composition of intrinsic functions. The operation will be the canonical way to
+write this chained operation (the MLIR canonicalization pass will rewrite the
+operations for the composed intrinsics into this one operation).
+
+These new operations will be treated as though they were standard
+transformational intrinsic functions.
+
+The composed intrinsic operation will return a hlfir.expr<T>. The arguments
+may be hlfir.expr<T>, boxed arrays, simple scalar types (e.g. i32, f32), or
+variables.
+
+To keep things simple, these operations will only match one form of the composed
+intrinsic functions: therefore there will be no optional arguments.
+
+Syntax:
+```
+%res = hlfir."intrinsic_name" %expr_or_var, ...
+```
+
+The composed intrinsic operation will be lowered to a `fir.call` to the newly
+added runtime implementation of the operation.
+
+These operations should not be added where the only improvement is to avoid
+creating a temporary intermediate buffer which would otherwise be removed by
+intelligent bufferization of a hlfir.expr. Similarly, these should not replace
+profitable uses of hlfir.elemental.
+
 #### Introducing operations for character operations and elemental intrinsic functions
 
 
diff --git a/flang/docs/OptionComparison.md b/flang/docs/OptionComparison.md
index 1b2525b9200cd..3a36f40730920 100644
--- a/flang/docs/OptionComparison.md
+++ b/flang/docs/OptionComparison.md
@@ -846,7 +846,7 @@ fdefault-integer-8
    </td>
   </tr>
   <tr>
-   <td>Enable <a href="https://pubs.cray.com/content/S-3901/8.7/cray-fortran-reference-manual/types">Cray pointers</a>
+   <td>Enable <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00113911en_us&docLocale=en_US&page=Types.html">Cray pointers</a>
    </td>
    <td>default
    </td>
@@ -854,7 +854,7 @@ fdefault-integer-8
    </td>
    <td><a href="https://www-01.ibm.com/support/docview.wss?uid=swg27024776&aid=1#page=432">Default (near equivalent)</a>
    </td>
-   <td><a href="https://software.intel.com/en-us/fortran-compiler-developer-guide-and-reference-pointer-integer">Default (near equivalent)</a>
+   <td><a href="https://www.intel.com/content/www/us/en/develop/documentation/fortran-compiler-oneapi-dev-guide-and-reference/top/language-reference/a-to-z-reference/o-to-p/pointer-integer.html">Default (near equivalent)</a>
    </td>
    <td>Mcray
    </td>
diff --git a/flang/examples/CMakeLists.txt b/flang/examples/CMakeLists.txt
index b0a78eeaa4eaa..23fea3920efb6 100644
--- a/flang/examples/CMakeLists.txt
+++ b/flang/examples/CMakeLists.txt
@@ -1,15 +1,3 @@
-if(NOT FLANG_BUILD_EXAMPLES)
-  set(EXCLUDE_FROM_ALL ON)
-endif()
-
-# This test is not run by default as it requires input.
-add_executable(external-hello-world
-  external-hello.cpp
-)
-
-target_link_libraries(external-hello-world
-  FortranRuntime
-)
-
+add_subdirectory(ExternalHelloWorld)
 add_subdirectory(PrintFlangFunctionNames)
 add_subdirectory(FlangOmpReport)
diff --git a/flang/examples/ExternalHelloWorld/CMakeLists.txt b/flang/examples/ExternalHelloWorld/CMakeLists.txt
new file mode 100644
index 0000000000000..042d4b6238ba4
--- /dev/null
+++ b/flang/examples/ExternalHelloWorld/CMakeLists.txt
@@ -0,0 +1,9 @@
+# This test is not run by default as it requires input.
+add_llvm_example(external-hello-world
+  external-hello.cpp
+)
+
+target_link_libraries(external-hello-world
+  PRIVATE
+  FortranRuntime
+  )
diff --git a/flang/examples/external-hello.cpp b/flang/examples/ExternalHelloWorld/external-hello.cpp
similarity index 100%
rename from flang/examples/external-hello.cpp
rename to flang/examples/ExternalHelloWorld/external-hello.cpp
diff --git a/flang/examples/FlangOmpReport/CMakeLists.txt b/flang/examples/FlangOmpReport/CMakeLists.txt
index ab39b2ca4fb1e..bd5172a9fde59 100644
--- a/flang/examples/FlangOmpReport/CMakeLists.txt
+++ b/flang/examples/FlangOmpReport/CMakeLists.txt
@@ -1,4 +1,4 @@
-add_llvm_library(flangOmpReport
+add_llvm_example_library(flangOmpReport
   MODULE
   FlangOmpReport.cpp
   FlangOmpReportVisitor.cpp
diff --git a/flang/examples/PrintFlangFunctionNames/CMakeLists.txt b/flang/examples/PrintFlangFunctionNames/CMakeLists.txt
index 7ba91fa4ef3c3..db20b5ee415b5 100644
--- a/flang/examples/PrintFlangFunctionNames/CMakeLists.txt
+++ b/flang/examples/PrintFlangFunctionNames/CMakeLists.txt
@@ -1,6 +1,6 @@
 # TODO: Note that this is currently only available on Linux.
 # On Windows, we would also have to specify e.g. `PLUGIN_TOOL`.
-add_llvm_library(flangPrintFunctionNames
+add_llvm_example_library(flangPrintFunctionNames
     MODULE
     PrintFlangFunctionNames.cpp
 
diff --git a/flang/include/flang/Common/Fortran-features.h b/flang/include/flang/Common/Fortran-features.h
index b94cf2c6d21f1..6cf90f518b91e 100644
--- a/flang/include/flang/Common/Fortran-features.h
+++ b/flang/include/flang/Common/Fortran-features.h
@@ -31,7 +31,7 @@ ENUM_CLASS(LanguageFeature, BackslashEscapes, OldDebugLines,
     EquivalenceSameNonSequence, AdditionalIntrinsics, AnonymousParents,
     OldLabelDoEndStatements, LogicalIntegerAssignment, EmptySourceFile,
     ProgramReturn, ImplicitNoneTypeNever, ImplicitNoneTypeAlways,
-    ForwardRefDummyImplicitNone, OpenAccessAppend, BOZAsDefaultInteger,
+    ForwardRefImplicitNone, OpenAccessAppend, BOZAsDefaultInteger,
     DistinguishableSpecifics, DefaultSave, PointerInSeqType, NonCharacterFormat,
     SaveMainProgram, SaveBigMainProgramVariables)
 
diff --git a/flang/include/flang/Evaluate/initial-image.h b/flang/include/flang/Evaluate/initial-image.h
index d5b30f53a7660..7cf01a85ba4b0 100644
--- a/flang/include/flang/Evaluate/initial-image.h
+++ b/flang/include/flang/Evaluate/initial-image.h
@@ -107,7 +107,8 @@ class InitialImage {
 
   // Conversions to constant initializers
   std::optional<Expr<SomeType>> AsConstant(FoldingContext &,
-      const DynamicType &, const ConstantSubscripts &, bool padWithZero = false,
+      const DynamicType &, std::optional<std::int64_t> charLength,
+      const ConstantSubscripts &, bool padWithZero = false,
       ConstantSubscript offset = 0) const;
   std::optional<Expr<SomeType>> AsConstantPointer(
       ConstantSubscript offset = 0) const;
diff --git a/flang/include/flang/Evaluate/tools.h b/flang/include/flang/Evaluate/tools.h
index 448909d365940..fa525197f00e9 100644
--- a/flang/include/flang/Evaluate/tools.h
+++ b/flang/include/flang/Evaluate/tools.h
@@ -1227,11 +1227,6 @@ const Symbol *FindCommonBlockContaining(const Symbol &);
 int CountLenParameters(const DerivedTypeSpec &);
 int CountNonConstantLenParameters(const DerivedTypeSpec &);
 
-// 15.5.2.4(4), type compatibility for dummy and actual arguments.
-// Also used for assignment compatibility checking
-bool AreTypeParamCompatible(
-    const semantics::DerivedTypeSpec &, const semantics::DerivedTypeSpec &);
-
 const Symbol &GetUsedModule(const UseDetails &);
 const Symbol *FindFunctionResult(const Symbol &);
 
diff --git a/flang/include/flang/Evaluate/type.h b/flang/include/flang/Evaluate/type.h
index 0338dcb1f1452..c8baa635ea6ef 100644
--- a/flang/include/flang/Evaluate/type.h
+++ b/flang/include/flang/Evaluate/type.h
@@ -157,8 +157,9 @@ class DynamicType {
   std::optional<Expr<SubscriptInteger>> GetCharLength() const;
 
   std::size_t GetAlignment(const TargetCharacteristics &) const;
-  std::optional<Expr<SubscriptInteger>> MeasureSizeInBytes(
-      FoldingContext &, bool aligned) const;
+  std::optional<Expr<SubscriptInteger>> MeasureSizeInBytes(FoldingContext &,
+      bool aligned,
+      std::optional<std::int64_t> charLength = std::nullopt) const;
 
   std::string AsFortran() const;
   std::string AsFortran(std::string &&charLenExpr) const;
diff --git a/flang/include/flang/Frontend/CodeGenOptions.def b/flang/include/flang/Frontend/CodeGenOptions.def
index 7f50442af6d09..c6bd7a5838c48 100644
--- a/flang/include/flang/Frontend/CodeGenOptions.def
+++ b/flang/include/flang/Frontend/CodeGenOptions.def
@@ -24,8 +24,12 @@ CODEGENOPT(OptimizationLevel, 2, 0) ///< The -O[0-3] option specified.
 CODEGENOPT(DebugPassManager, 1, 0) ///< Prints debug information for the new
                                    ///< pass manager.
 
-CODEGENOPT(PICLevel, 2, 0) ///< PIC level of the LLVM module.
 CODEGENOPT(IsPIE, 1, 0) ///< PIE level is the same as PIC Level.
+CODEGENOPT(PICLevel, 2, 0) ///< PIC level of the LLVM module.
+CODEGENOPT(PrepareForFullLTO , 1, 0) ///< Set when -flto is enabled on the
+                                     ///< compile step.
+CODEGENOPT(PrepareForThinLTO , 1, 0) ///< Set when -flto=thin is enabled on the
+                                     ///< compile step.
 CODEGENOPT(StackArrays, 1, 0) ///< -fstack-arrays (enable the stack-arrays pass)
 
 CODEGENOPT(Underscoring, 1, 1)
diff --git a/flang/include/flang/Frontend/LangOptions.def b/flang/include/flang/Frontend/LangOptions.def
index 9471beda8e05a..3648e09154520 100644
--- a/flang/include/flang/Frontend/LangOptions.def
+++ b/flang/include/flang/Frontend/LangOptions.def
@@ -34,6 +34,8 @@ LANGOPT(NoSignedZeros, 1, false)
 LANGOPT(AssociativeMath, 1, false)
 /// Allow division operations to be reassociated
 LANGOPT(ReciprocalMath, 1, false)
+/// Generate code only for OpenMP target device
+LANGOPT(OpenMPIsDevice, 1, false)
 
 #undef LANGOPT
 #undef ENUM_LANGOPT
diff --git a/flang/include/flang/Lower/Bridge.h b/flang/include/flang/Lower/Bridge.h
index 6766613ebc27d..b4ee77a0b166e 100644
--- a/flang/include/flang/Lower/Bridge.h
+++ b/flang/include/flang/Lower/Bridge.h
@@ -19,7 +19,7 @@
 #include "flang/Lower/LoweringOptions.h"
 #include "flang/Lower/StatementContext.h"
 #include "flang/Optimizer/Builder/FIRBuilder.h"
-#include "flang/Optimizer/Support/KindMapping.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "mlir/IR/BuiltinOps.h"
 
 namespace Fortran {
diff --git a/flang/include/flang/Lower/ConvertCall.h b/flang/include/flang/Lower/ConvertCall.h
index 9336468ec2217..76a03ea319f5b 100644
--- a/flang/include/flang/Lower/ConvertCall.h
+++ b/flang/include/flang/Lower/ConvertCall.h
@@ -40,6 +40,10 @@ fir::ExtendedValue genCallOpAndResult(
 mlir::Value argumentHostAssocs(Fortran::lower::AbstractConverter &converter,
                                mlir::Value arg);
 
+/// Is \p procRef an intrinsic module procedure that should be lowered as
+/// intrinsic procedures (with Optimizer/Builder/IntrinsicCall.h)?
+bool isIntrinsicModuleProcRef(const Fortran::evaluate::ProcedureRef &procRef);
+
 /// Lower a ProcedureRef to HLFIR. If this is a function call, return the
 /// lowered result value. Return nothing otherwise.
 std::optional<hlfir::EntityWithAttributes> convertCallToHLFIR(
diff --git a/flang/include/flang/Lower/ConvertExpr.h b/flang/include/flang/Lower/ConvertExpr.h
index e8b388bd0b7f9..a4a8a524e9ada 100644
--- a/flang/include/flang/Lower/ConvertExpr.h
+++ b/flang/include/flang/Lower/ConvertExpr.h
@@ -79,6 +79,14 @@ fir::MutableBoxValue createMutableBox(mlir::Location loc,
                                       AbstractConverter &converter,
                                       const SomeExpr &expr, SymMap &symMap);
 
+/// Return true iff the expression is pointing to a parent component.
+bool isParentComponent(const SomeExpr &expr);
+
+/// Update the extended value to represent the parent component.
+fir::ExtendedValue updateBoxForParentComponent(AbstractConverter &converter,
+                                               fir::ExtendedValue exv,
+                                               const SomeExpr &expr);
+
 /// Create a fir::BoxValue describing the value of \p expr.
 /// If \p expr is a variable without vector subscripts, the fir::BoxValue
 /// described the variable storage. Otherwise, the created fir::BoxValue
diff --git a/flang/include/flang/Optimizer/Builder/FIRBuilder.h b/flang/include/flang/Optimizer/Builder/FIRBuilder.h
index 6882da731b652..2e1c814403955 100644
--- a/flang/include/flang/Optimizer/Builder/FIRBuilder.h
+++ b/flang/include/flang/Optimizer/Builder/FIRBuilder.h
@@ -20,7 +20,7 @@
 #include "flang/Optimizer/Dialect/FIROps.h"
 #include "flang/Optimizer/Dialect/FIROpsSupport.h"
 #include "flang/Optimizer/Dialect/FIRType.h"
-#include "flang/Optimizer/Support/KindMapping.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/IR/BuiltinOps.h"
 #include "llvm/ADT/DenseMap.h"
@@ -336,7 +336,7 @@ class FirOpBuilder : public mlir::OpBuilder, public mlir::OpBuilder::Listener {
   /// Array entities are boxed with a shape and possibly a shift. Character
   /// entities are boxed with a LEN parameter.
   mlir::Value createBox(mlir::Location loc, const fir::ExtendedValue &exv,
-                        bool isPolymorphic = false);
+                        bool isPolymorphic = false, bool isAssumedType = false);
 
   mlir::Value createBox(mlir::Location loc, mlir::Type boxType,
                         mlir::Value addr, mlir::Value shape, mlir::Value slice,
@@ -426,6 +426,10 @@ class FirOpBuilder : public mlir::OpBuilder, public mlir::OpBuilder::Listener {
                                    mlir::Value ub, mlir::Value step,
                                    mlir::Type type);
 
+  /// Create an AbsentOp of \p argTy type and handle special cases, such as
+  /// Character Procedure Tuple arguments.
+  mlir::Value genAbsentOp(mlir::Location loc, mlir::Type argTy);
+
   /// Set default FastMathFlags value for all operations
   /// supporting mlir::arith::FastMathAttr that will be created
   /// by this builder.
diff --git a/flang/include/flang/Optimizer/Builder/Runtime/Assign.h b/flang/include/flang/Optimizer/Builder/Runtime/Assign.h
index 9010013086623..f33a76a6e8299 100644
--- a/flang/include/flang/Optimizer/Builder/Runtime/Assign.h
+++ b/flang/include/flang/Optimizer/Builder/Runtime/Assign.h
@@ -35,5 +35,18 @@ void genAssign(fir::FirOpBuilder &builder, mlir::Location loc,
 void genAssignPolymorphic(fir::FirOpBuilder &builder, mlir::Location loc,
                           mlir::Value destBox, mlir::Value sourceBox);
 
+/// Generate runtime call to AssignExplicitLengthCharacter to assign
+/// \p sourceBox to \p destBox where \p destBox is a whole allocatable character
+/// with explicit or assumed length. After the assignment, the length of
+/// \p destBox will remain what it was, even if allocation or reallocation
+/// occurred. For assignments to a whole allocatable with deferred length,
+/// genAssign should be used.
+/// \p destBox must be a fir.ref<fir.box<T>> and \p sourceBox a fir.box<T>.
+/// \p destBox Fortran descriptor may be modified if destBox is an allocatable
+/// according to Fortran allocatable assignment rules.
+void genAssignExplicitLengthCharacter(fir::FirOpBuilder &builder,
+                                      mlir::Location loc, mlir::Value destBox,
+                                      mlir::Value sourceBox);
+
 } // namespace fir::runtime
 #endif // FORTRAN_OPTIMIZER_BUILDER_RUNTIME_ASSIGN_H
diff --git a/flang/include/flang/Optimizer/Builder/Runtime/Transformational.h b/flang/include/flang/Optimizer/Builder/Runtime/Transformational.h
index f084e2eb7ae3a..ae0a0979902f5 100644
--- a/flang/include/flang/Optimizer/Builder/Runtime/Transformational.h
+++ b/flang/include/flang/Optimizer/Builder/Runtime/Transformational.h
@@ -55,6 +55,10 @@ void genMatmul(fir::FirOpBuilder &builder, mlir::Location loc,
                mlir::Value matrixABox, mlir::Value matrixBBox,
                mlir::Value resultBox);
 
+void genMatmulTranspose(fir::FirOpBuilder &builder, mlir::Location loc,
+                        mlir::Value matrixABox, mlir::Value matrixBBox,
+                        mlir::Value resultBox);
+
 void genPack(fir::FirOpBuilder &builder, mlir::Location loc,
              mlir::Value resultBox, mlir::Value arrayBox, mlir::Value maskBox,
              mlir::Value vectorBox);
diff --git a/flang/include/flang/Optimizer/Dialect/FIROps.td b/flang/include/flang/Optimizer/Dialect/FIROps.td
index 1fd653b6076b9..2bc4ec0401d52 100644
--- a/flang/include/flang/Optimizer/Dialect/FIROps.td
+++ b/flang/include/flang/Optimizer/Dialect/FIROps.td
@@ -2146,7 +2146,9 @@ def fir_DoLoopOp : region_Op<"do_loop",
   }];
 }
 
-def fir_IfOp : region_Op<"if", [NoRegionArguments]> {
+def fir_IfOp : region_Op<"if", [DeclareOpInterfaceMethods<RegionBranchOpInterface, [
+    "getRegionInvocationBounds"]>, RecursiveMemoryEffects,
+    NoRegionArguments]> {
   let summary = "if-then-else conditional operation";
   let description = [{
     Used to conditionally execute operations. This operation is the FIR
@@ -2168,7 +2170,7 @@ def fir_IfOp : region_Op<"if", [NoRegionArguments]> {
 
   let regions = (region
     SizedRegion<1>:$thenRegion,
-    AnyRegion:$elseRegion
+    MaxSizedRegion<1>:$elseRegion
   );
 
   let skipDefaultBuilders = 1;
diff --git a/flang/include/flang/Optimizer/Dialect/FIRType.h b/flang/include/flang/Optimizer/Dialect/FIRType.h
index 3a0254ab4de8e..942c789bcb899 100644
--- a/flang/include/flang/Optimizer/Dialect/FIRType.h
+++ b/flang/include/flang/Optimizer/Dialect/FIRType.h
@@ -261,6 +261,24 @@ inline fir::SequenceType unwrapUntilSeqType(mlir::Type t) {
   }
 }
 
+/// Unwrap the referential and sequential outer types (if any). Returns the
+/// the element if type is fir::RecordType
+inline fir::RecordType unwrapIfDerived(fir::BaseBoxType boxTy) {
+  return fir::unwrapSequenceType(fir::unwrapRefType(boxTy.getEleTy()))
+      .template dyn_cast<fir::RecordType>();
+}
+
+/// Return true iff `boxTy` wraps a fir::RecordType with length parameters
+inline bool isDerivedTypeWithLenParams(fir::BaseBoxType boxTy) {
+  auto recTy = unwrapIfDerived(boxTy);
+  return recTy && recTy.getNumLenParams() > 0;
+}
+
+/// Return true iff `boxTy` wraps a fir::RecordType
+inline bool isDerivedType(fir::BaseBoxType boxTy) {
+  return static_cast<bool>(unwrapIfDerived(boxTy));
+}
+
 #ifndef NDEBUG
 // !fir.ptr<X> and !fir.heap<X> where X is !fir.ptr, !fir.heap, or !fir.ref
 // is undefined and disallowed.
@@ -300,6 +318,16 @@ bool isPolymorphicType(mlir::Type ty);
 /// value.
 bool isUnlimitedPolymorphicType(mlir::Type ty);
 
+/// Return true iff `ty` is the type of an assumed type.
+bool isAssumedType(mlir::Type ty);
+
+/// Return true iff `boxTy` wraps a record type or an unlimited polymorphic
+/// entity. Polymorphic entities with intrinsic type spec do not have addendum
+inline bool boxHasAddendum(fir::BaseBoxType boxTy) {
+  return static_cast<bool>(unwrapIfDerived(boxTy)) ||
+         fir::isUnlimitedPolymorphicType(boxTy);
+}
+
 /// Return the inner type of the given type.
 mlir::Type unwrapInnerType(mlir::Type ty);
 
diff --git a/flang/include/flang/Optimizer/Support/FIRContext.h b/flang/include/flang/Optimizer/Dialect/Support/FIRContext.h
similarity index 100%
rename from flang/include/flang/Optimizer/Support/FIRContext.h
rename to flang/include/flang/Optimizer/Dialect/Support/FIRContext.h
diff --git a/flang/include/flang/Optimizer/Support/KindMapping.h b/flang/include/flang/Optimizer/Dialect/Support/KindMapping.h
similarity index 100%
rename from flang/include/flang/Optimizer/Support/KindMapping.h
rename to flang/include/flang/Optimizer/Dialect/Support/KindMapping.h
diff --git a/flang/include/flang/Optimizer/HLFIR/HLFIROps.h b/flang/include/flang/Optimizer/HLFIR/HLFIROps.h
index 94ea77260e340..88f50fd110db5 100644
--- a/flang/include/flang/Optimizer/HLFIR/HLFIROps.h
+++ b/flang/include/flang/Optimizer/HLFIR/HLFIROps.h
@@ -15,6 +15,7 @@
 #include "flang/Optimizer/Dialect/FortranVariableInterface.h"
 #include "flang/Optimizer/HLFIR/HLFIRDialect.h"
 #include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/IR/PatternMatch.h"
 #include "mlir/Interfaces/InferTypeOpInterface.h"
 #include "mlir/Interfaces/SideEffectInterfaces.h"
 
diff --git a/flang/include/flang/Optimizer/HLFIR/HLFIROps.td b/flang/include/flang/Optimizer/HLFIR/HLFIROps.td
index b797cd2cd17e1..15c5735c1e8eb 100644
--- a/flang/include/flang/Optimizer/HLFIR/HLFIROps.td
+++ b/flang/include/flang/Optimizer/HLFIR/HLFIROps.td
@@ -382,6 +382,9 @@ def hlfir_MatmulOp : hlfir_Op<"matmul",
     $lhs $rhs attr-dict `:` functional-type(operands, results)
   }];
 
+  // MATMUL(TRANSPOSE(...), ...) => hlfir.matmul_transpose
+  let hasCanonicalizeMethod = 1;
+
   let hasVerifier = 1;
 }
 
@@ -402,6 +405,29 @@ def hlfir_TransposeOp : hlfir_Op<"transpose", []> {
   let hasVerifier = 1;
 }
 
+def hlfir_MatmulTransposeOp : hlfir_Op<"matmul_transpose",
+    [DeclareOpInterfaceMethods<ArithFastMathInterface>]> {
+  let summary = "Optimized MATMUL(TRANSPOSE(...), ...)";
+  let description = [{
+    Matrix multiplication where the left hand side is transposed
+  }];
+
+  let arguments = (ins
+    AnyFortranNumericalOrLogicalArrayObject:$lhs,
+    AnyFortranNumericalOrLogicalArrayObject:$rhs,
+    DefaultValuedAttr<Arith_FastMathAttr,
+                      "::mlir::arith::FastMathFlags::none">:$fastmath
+  );
+
+  let results = (outs hlfir_ExprType);
+
+  let assemblyFormat = [{
+    $lhs $rhs attr-dict `:` functional-type(operands, results)
+  }];
+
+  let hasVerifier = 1;
+}
+
 def hlfir_AssociateOp : hlfir_Op<"associate", [AttrSizedOperandSegments,
     DeclareOpInterfaceMethods<fir_FortranVariableOpInterface>]> {
   let summary = "Create a variable from an expression value";
diff --git a/flang/include/flang/Optimizer/HLFIR/Passes.h b/flang/include/flang/Optimizer/HLFIR/Passes.h
index 53d2186843462..2d9ac0c8e64b6 100644
--- a/flang/include/flang/Optimizer/HLFIR/Passes.h
+++ b/flang/include/flang/Optimizer/HLFIR/Passes.h
@@ -23,6 +23,7 @@ namespace hlfir {
 
 std::unique_ptr<mlir::Pass> createConvertHLFIRtoFIRPass();
 std::unique_ptr<mlir::Pass> createBufferizeHLFIRPass();
+std::unique_ptr<mlir::Pass> createLowerHLFIRIntrinsicsPass();
 
 #define GEN_PASS_REGISTRATION
 #include "flang/Optimizer/HLFIR/Passes.h.inc"
diff --git a/flang/include/flang/Optimizer/HLFIR/Passes.td b/flang/include/flang/Optimizer/HLFIR/Passes.td
index 3fac0b2a1ea87..faf5cd99d8ac3 100644
--- a/flang/include/flang/Optimizer/HLFIR/Passes.td
+++ b/flang/include/flang/Optimizer/HLFIR/Passes.td
@@ -20,4 +20,9 @@ def BufferizeHLFIR : Pass<"bufferize-hlfir", "::mlir::ModuleOp"> {
   let constructor = "hlfir::createBufferizeHLFIRPass()";
 }
 
+def LowerHLFIRIntrinsics : Pass<"lower-hlfir-intrinsics", "::mlir::ModuleOp"> {
+  let summary = "Lower HLFIR transformational intrinsic operations";
+  let constructor = "hlfir::createLowerHLFIRIntrinsicsPass()";
+}
+
 #endif //FORTRAN_DIALECT_HLFIR_PASSES
diff --git a/flang/include/flang/Optimizer/Support/Utils.h b/flang/include/flang/Optimizer/Support/Utils.h
index d25a4005f4017..ebf506543ebf4 100644
--- a/flang/include/flang/Optimizer/Support/Utils.h
+++ b/flang/include/flang/Optimizer/Support/Utils.h
@@ -14,9 +14,14 @@
 #define FORTRAN_OPTIMIZER_SUPPORT_UTILS_H
 
 #include "flang/Common/default-kinds.h"
+#include "flang/Optimizer/Dialect/FIROps.h"
 #include "flang/Optimizer/Dialect/FIRType.h"
+#include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/Func/IR/FuncOps.h"
 #include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/BuiltinOps.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/StringRef.h"
 
 namespace fir {
 /// Return the integer value of a arith::ConstantOp.
@@ -24,6 +29,31 @@ inline std::int64_t toInt(mlir::arith::ConstantOp cop) {
   return cop.getValue().cast<mlir::IntegerAttr>().getValue().getSExtValue();
 }
 
+// Reconstruct binding tables for dynamic dispatch.
+using BindingTable = llvm::DenseMap<llvm::StringRef, unsigned>;
+using BindingTables = llvm::DenseMap<llvm::StringRef, BindingTable>;
+
+inline void buildBindingTables(BindingTables &bindingTables,
+                               mlir::ModuleOp mod) {
+
+  // The binding tables are defined in FIR from lowering as fir.dispatch_table
+  // operation. Go through each binding tables and store the procedure name and
+  // binding index for later use by the fir.dispatch conversion pattern.
+  for (auto dispatchTableOp : mod.getOps<fir::DispatchTableOp>()) {
+    unsigned bindingIdx = 0;
+    BindingTable bindings;
+    if (dispatchTableOp.getRegion().empty()) {
+      bindingTables[dispatchTableOp.getSymName()] = bindings;
+      continue;
+    }
+    for (auto dtEntry : dispatchTableOp.getBlock().getOps<fir::DTEntryOp>()) {
+      bindings[dtEntry.getMethod()] = bindingIdx;
+      ++bindingIdx;
+    }
+    bindingTables[dispatchTableOp.getSymName()] = bindings;
+  }
+}
+
 // Translate front-end KINDs for use in the IR and code gen.
 inline std::vector<fir::KindTy>
 fromDefaultKinds(const Fortran::common::IntrinsicTypeDefaultKinds &defKinds) {
diff --git a/flang/include/flang/Runtime/allocatable.h b/flang/include/flang/Runtime/allocatable.h
index 58bbd27c1b97f..4169483398f6a 100644
--- a/flang/include/flang/Runtime/allocatable.h
+++ b/flang/include/flang/Runtime/allocatable.h
@@ -33,6 +33,17 @@ void RTNAME(AllocatableInitCharacter)(Descriptor &, SubscriptValue length = 0,
 void RTNAME(AllocatableInitDerived)(
     Descriptor &, const typeInfo::DerivedType &, int rank = 0, int corank = 0);
 
+// Initializes the descriptor for an allocatable of intrinsic or derived type.
+// These functions are meant to be used in the allocate statement lowering. If
+// the descriptor is allocated, the initialization is skiped so the error
+// handling can be done by AllocatableAllocate.
+void RTNAME(AllocatableInitIntrinsicForAllocate)(
+    Descriptor &, TypeCategory, int kind, int rank = 0, int corank = 0);
+void RTNAME(AllocatableInitCharacterForAllocate)(Descriptor &,
+    SubscriptValue length = 0, int kind = 1, int rank = 0, int corank = 0);
+void RTNAME(AllocatableInitDerivedForAllocate)(
+    Descriptor &, const typeInfo::DerivedType &, int rank = 0, int corank = 0);
+
 // Checks that an allocatable is not already allocated in statements
 // with STAT=.  Use this on a value descriptor before setting bounds or
 // type parameters.  Not necessary on a freshly initialized descriptor.
diff --git a/flang/include/flang/Runtime/iostat.h b/flang/include/flang/Runtime/iostat.h
index 6d6be7c7b8d57..ddb82ce0ca238 100644
--- a/flang/include/flang/Runtime/iostat.h
+++ b/flang/include/flang/Runtime/iostat.h
@@ -82,6 +82,7 @@ enum Iostat {
   IostatBadBackspaceUnit,
   IostatBadUnitNumber,
   IostatBadFlushUnit,
+  IostatBadOpOnChildUnit,
 };
 
 const char *IostatErrorString(int);
diff --git a/flang/include/flang/Runtime/matmul-transpose.h b/flang/include/flang/Runtime/matmul-transpose.h
new file mode 100644
index 0000000000000..7cfb189863df8
--- /dev/null
+++ b/flang/include/flang/Runtime/matmul-transpose.h
@@ -0,0 +1,30 @@
+//===-- include/flang/Runtime/matmul-transpose.h ----------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// API for optimised MATMUL(TRANSPOSE(a), b)
+
+#ifndef FORTRAN_RUNTIME_MATMUL_TRANSPOSE_H_
+#define FORTRAN_RUNTIME_MATMUL_TRANSPOSE_H_
+#include "flang/Runtime/entry-names.h"
+namespace Fortran::runtime {
+class Descriptor;
+extern "C" {
+
+// The most general MATMUL(TRANSPOSE()).  All type and shape information is
+// taken from the arguments' descriptors, and the result is dynamically
+// allocated.
+void RTNAME(MatmulTranspose)(Descriptor &, const Descriptor &,
+    const Descriptor &, const char *sourceFile = nullptr, int line = 0);
+
+// A non-allocating variant; the result's descriptor must be established
+// and have a valid base address.
+void RTNAME(MatmulTransposeDirect)(const Descriptor &, const Descriptor &,
+    const Descriptor &, const char *sourceFile = nullptr, int line = 0);
+} // extern "C"
+} // namespace Fortran::runtime
+#endif // FORTRAN_RUNTIME_MATMUL_TRANSPOSE_H_
diff --git a/flang/include/flang/Semantics/symbol.h b/flang/include/flang/Semantics/symbol.h
index 6a480b44d5a8a..e2a771251e6b5 100644
--- a/flang/include/flang/Semantics/symbol.h
+++ b/flang/include/flang/Semantics/symbol.h
@@ -337,6 +337,7 @@ class ProcBindingDetails : public WithPassArg {
 public:
   explicit ProcBindingDetails(const Symbol &symbol) : symbol_{symbol} {}
   const Symbol &symbol() const { return symbol_; }
+  void ReplaceSymbol(const Symbol &symbol) { symbol_ = symbol; }
 
 private:
   SymbolRef symbol_; // procedure bound to; may be forward
diff --git a/flang/include/flang/Semantics/tools.h b/flang/include/flang/Semantics/tools.h
index 4c3630c67ebd2..2de6e1f8c9ddd 100644
--- a/flang/include/flang/Semantics/tools.h
+++ b/flang/include/flang/Semantics/tools.h
@@ -108,7 +108,7 @@ bool IsBindCProcedure(const Scope &);
 // Returns a pointer to the function's symbol when true, else null
 const Symbol *IsFunctionResultWithSameNameAsFunction(const Symbol &);
 bool IsOrContainsEventOrLockComponent(const Symbol &);
-bool CanBeTypeBoundProc(const Symbol *);
+bool CanBeTypeBoundProc(const Symbol &);
 // Does a non-PARAMETER symbol have explicit initialization with =value or
 // =>target in its declaration (but not in a DATA statement)? (Being
 // ALLOCATABLE or having a derived type with default component initialization
@@ -169,6 +169,7 @@ inline bool IsProtected(const Symbol &symbol) {
 inline bool IsImpliedDoIndex(const Symbol &symbol) {
   return symbol.owner().kind() == Scope::Kind::ImpliedDos;
 }
+SymbolVector FinalsForDerivedTypeInstantiation(const DerivedTypeSpec &);
 bool IsFinalizable(
     const Symbol &, std::set<const DerivedTypeSpec *> * = nullptr);
 bool IsFinalizable(
@@ -252,7 +253,7 @@ const Symbol *FindExternallyVisibleObject(
       expr.u);
 }
 
-// Apply GetUltimate(), then if the symbol is a generic procedure shadowing a
+// Applies GetUltimate(), then if the symbol is a generic procedure shadowing a
 // specific procedure of the same name, return it instead.
 const Symbol &BypassGeneric(const Symbol &);
 
diff --git a/flang/lib/Evaluate/characteristics.cpp b/flang/lib/Evaluate/characteristics.cpp
index 0fe965aeab124..1906f5a446083 100644
--- a/flang/lib/Evaluate/characteristics.cpp
+++ b/flang/lib/Evaluate/characteristics.cpp
@@ -263,8 +263,22 @@ bool DummyDataObject::operator==(const DummyDataObject &that) const {
 }
 
 static bool AreCompatibleDummyDataObjectShapes(const Shape &x, const Shape &y) {
-  // TODO: Validate more than just compatible ranks
-  return GetRank(x) == GetRank(y);
+  int n{GetRank(x)};
+  if (n != GetRank(y)) {
+    return false;
+  }
+  auto xIter{x.begin()};
+  auto yIter{y.begin()};
+  for (; n-- > 0; ++xIter, ++yIter) {
+    if (auto xVal{ToInt64(*xIter)}) {
+      if (auto yVal{ToInt64(*yIter)}) {
+        if (*xVal != *yVal) {
+          return false;
+        }
+      }
+    }
+  }
+  return true;
 }
 
 bool DummyDataObject::IsCompatibleWith(
diff --git a/flang/lib/Evaluate/fold-logical.cpp b/flang/lib/Evaluate/fold-logical.cpp
index 4c3476a53c1a2..43406497a9e66 100644
--- a/flang/lib/Evaluate/fold-logical.cpp
+++ b/flang/lib/Evaluate/fold-logical.cpp
@@ -174,10 +174,13 @@ Expr<Type<TypeCategory::Logical, KIND>> FoldIntrinsicFunction(
   } else if (name == "__builtin_ieee_is_normal") {
     auto restorer{context.messages().DiscardMessages()};
     using DefaultReal = Type<TypeCategory::Real, 4>;
-    return FoldElementalIntrinsic<T, DefaultReal>(context, std::move(funcRef),
-        ScalarFunc<T, DefaultReal>([](const Scalar<DefaultReal> &x) {
-          return Scalar<T>{x.IsNormal()};
-        }));
+    if (args[0] && args[0]->UnwrapExpr() &&
+        IsActuallyConstant(*args[0]->UnwrapExpr())) {
+      return FoldElementalIntrinsic<T, DefaultReal>(context, std::move(funcRef),
+          ScalarFunc<T, DefaultReal>([](const Scalar<DefaultReal> &x) {
+            return Scalar<T>{x.IsNormal()};
+          }));
+    }
   } else if (name == "is_contiguous") {
     if (args.at(0)) {
       if (auto *expr{args[0]->UnwrapExpr()}) {
diff --git a/flang/lib/Evaluate/fold.cpp b/flang/lib/Evaluate/fold.cpp
index 6f61a68294847..0c5d2c24079c8 100644
--- a/flang/lib/Evaluate/fold.cpp
+++ b/flang/lib/Evaluate/fold.cpp
@@ -235,8 +235,14 @@ std::optional<Expr<SomeType>> FoldTransfer(
     }
   }
   std::optional<DynamicType> moldType;
-  if (arguments[1]) {
+  std::optional<std::int64_t> moldLength;
+  if (arguments[1]) { // MOLD=
     moldType = arguments[1]->GetType();
+    if (moldType && moldType->category() == TypeCategory::Character) {
+      if (const auto *chExpr{UnwrapExpr<Expr<SomeCharacter>>(arguments[1])}) {
+        moldLength = ToInt64(Fold(context, chExpr->LEN()));
+      }
+    }
   }
   std::optional<ConstantSubscripts> extents;
   if (arguments.size() == 2) { // no SIZE=
@@ -260,7 +266,8 @@ std::optional<Expr<SomeType>> FoldTransfer(
       }
     }
   }
-  if (sourceBytes && IsActuallyConstant(*source) && moldType && extents) {
+  if (sourceBytes && IsActuallyConstant(*source) && moldType && extents &&
+      (moldLength || moldType->category() != TypeCategory::Character)) {
     std::size_t elements{
         extents->empty() ? 1 : static_cast<std::size_t>((*extents)[0])};
     std::size_t totalBytes{*sourceBytes * elements};
@@ -272,7 +279,7 @@ std::optional<Expr<SomeType>> FoldTransfer(
           image.Add(0, *sourceBytes, *source, context)};
       CHECK(imageResult == InitialImage::Ok);
       return image.AsConstant(
-          context, *moldType, *extents, true /*pad with 0*/);
+          context, *moldType, moldLength, *extents, true /*pad with 0*/);
     }
   }
   return std::nullopt;
diff --git a/flang/lib/Evaluate/initial-image.cpp b/flang/lib/Evaluate/initial-image.cpp
index 0dee0eae059b0..a5f77a603ddb4 100644
--- a/flang/lib/Evaluate/initial-image.cpp
+++ b/flang/lib/Evaluate/initial-image.cpp
@@ -73,10 +73,11 @@ class AsConstantHelper {
   using Result = std::optional<Expr<SomeType>>;
   using Types = AllTypes;
   AsConstantHelper(FoldingContext &context, const DynamicType &type,
-      const ConstantSubscripts &extents, const InitialImage &image,
-      bool padWithZero = false, ConstantSubscript offset = 0)
-      : context_{context}, type_{type}, image_{image}, extents_{extents},
-        padWithZero_{padWithZero}, offset_{offset} {
+      std::optional<std::int64_t> charLength, const ConstantSubscripts &extents,
+      const InitialImage &image, bool padWithZero = false,
+      ConstantSubscript offset = 0)
+      : context_{context}, type_{type}, charLength_{charLength}, image_{image},
+        extents_{extents}, padWithZero_{padWithZero}, offset_{offset} {
     CHECK(!type.IsPolymorphic());
   }
   template <typename T> Result Test() {
@@ -92,8 +93,8 @@ class AsConstantHelper {
     using Scalar = typename Const::Element;
     std::size_t elements{TotalElementCount(extents_)};
     std::vector<Scalar> typedValue(elements);
-    auto elemBytes{
-        ToInt64(type_.MeasureSizeInBytes(context_, GetRank(extents_) > 0))};
+    auto elemBytes{ToInt64(type_.MeasureSizeInBytes(
+        context_, GetRank(extents_) > 0, charLength_))};
     CHECK(elemBytes && *elemBytes >= 0);
     std::size_t stride{static_cast<std::size_t>(*elemBytes)};
     CHECK(offset_ + elements * stride <= image_.data_.size() || padWithZero_);
@@ -123,7 +124,7 @@ class AsConstantHelper {
             CHECK(componentExtents.has_value());
             for (std::size_t j{0}; j < elements; ++j, at += stride) {
               if (Result value{image_.AsConstant(context_, *componentType,
-                      *componentExtents, padWithZero_, at)}) {
+                      std::nullopt, *componentExtents, padWithZero_, at)}) {
                 typedValue[j].emplace(component, std::move(*value));
               }
             }
@@ -182,6 +183,7 @@ class AsConstantHelper {
 private:
   FoldingContext &context_;
   const DynamicType &type_;
+  std::optional<std::int64_t> charLength_;
   const InitialImage &image_;
   ConstantSubscripts extents_; // a copy
   bool padWithZero_;
@@ -189,10 +191,11 @@ class AsConstantHelper {
 };
 
 std::optional<Expr<SomeType>> InitialImage::AsConstant(FoldingContext &context,
-    const DynamicType &type, const ConstantSubscripts &extents,
-    bool padWithZero, ConstantSubscript offset) const {
-  return common::SearchTypes(
-      AsConstantHelper{context, type, extents, *this, padWithZero, offset});
+    const DynamicType &type, std::optional<std::int64_t> charLength,
+    const ConstantSubscripts &extents, bool padWithZero,
+    ConstantSubscript offset) const {
+  return common::SearchTypes(AsConstantHelper{
+      context, type, charLength, extents, *this, padWithZero, offset});
 }
 
 std::optional<Expr<SomeType>> InitialImage::AsConstantPointer(
diff --git a/flang/lib/Evaluate/intrinsics-library.cpp b/flang/lib/Evaluate/intrinsics-library.cpp
index c333ffeac90c7..3a6d28e4ab040 100644
--- a/flang/lib/Evaluate/intrinsics-library.cpp
+++ b/flang/lib/Evaluate/intrinsics-library.cpp
@@ -295,7 +295,7 @@ struct HostRuntimeLibrary<std::complex<HostT>, LibraryVersion::Libm> {
 /// Define libm extensions
 /// Bessel functions are defined in POSIX.1-2001.
 
-// Remove float bessel funcitons for AIX as they are not supported
+// Remove float bessel functions for AIX as they are not supported
 #ifndef _AIX
 template <> struct HostRuntimeLibrary<float, LibraryVersion::LibmExtensions> {
   using F = FuncPointer<float, float>;
diff --git a/flang/lib/Evaluate/tools.cpp b/flang/lib/Evaluate/tools.cpp
index e5bc9cd953fd0..3f62c2c7e4ef9 100644
--- a/flang/lib/Evaluate/tools.cpp
+++ b/flang/lib/Evaluate/tools.cpp
@@ -1598,31 +1598,6 @@ int CountNonConstantLenParameters(const DerivedTypeSpec &type) {
   });
 }
 
-// Are the type parameters of type1 compile-time compatible with the
-// corresponding kind type parameters of type2?  Return true if all constant
-// valued parameters are equal.
-// Used to check assignment statements and argument passing.  See 15.5.2.4(4)
-bool AreTypeParamCompatible(const semantics::DerivedTypeSpec &type1,
-    const semantics::DerivedTypeSpec &type2) {
-  for (const auto &[name, param1] : type1.parameters()) {
-    if (semantics::MaybeIntExpr paramExpr1{param1.GetExplicit()}) {
-      if (IsConstantExpr(*paramExpr1)) {
-        const semantics::ParamValue *param2{type2.FindParameter(name)};
-        if (param2) {
-          if (semantics::MaybeIntExpr paramExpr2{param2->GetExplicit()}) {
-            if (IsConstantExpr(*paramExpr2)) {
-              if (ToInt64(*paramExpr1) != ToInt64(*paramExpr2)) {
-                return false;
-              }
-            }
-          }
-        }
-      }
-    }
-  }
-  return true;
-}
-
 const Symbol &GetUsedModule(const UseDetails &details) {
   return DEREF(details.symbol().owner().symbol());
 }
diff --git a/flang/lib/Evaluate/type.cpp b/flang/lib/Evaluate/type.cpp
index f5d5d5b0efc39..4cc1cb173f223 100644
--- a/flang/lib/Evaluate/type.cpp
+++ b/flang/lib/Evaluate/type.cpp
@@ -140,7 +140,8 @@ std::size_t DynamicType::GetAlignment(
 }
 
 std::optional<Expr<SubscriptInteger>> DynamicType::MeasureSizeInBytes(
-    FoldingContext &context, bool aligned) const {
+    FoldingContext &context, bool aligned,
+    std::optional<std::int64_t> charLength) const {
   switch (category_) {
   case TypeCategory::Integer:
   case TypeCategory::Real:
@@ -149,7 +150,9 @@ std::optional<Expr<SubscriptInteger>> DynamicType::MeasureSizeInBytes(
     return Expr<SubscriptInteger>{
         context.targetCharacteristics().GetByteSize(category_, kind_)};
   case TypeCategory::Character:
-    if (auto len{GetCharLength()}) {
+    if (auto len{charLength ? Expr<SubscriptInteger>{Constant<SubscriptInteger>{
+                                  *charLength}}
+                            : GetCharLength()}) {
       return Fold(context,
           Expr<SubscriptInteger>{
               context.targetCharacteristics().GetByteSize(category_, kind_)} *
@@ -259,13 +262,130 @@ static bool AreSameComponent(const semantics::Symbol &x,
       y.has<semantics::ObjectEntityDetails>();
 }
 
+// TODO: These utilities were cloned out of Semantics to avoid a cyclic
+// dependency and should be repackaged into then "namespace semantics"
+// part of Evaluate/tools.cpp.
+
+static const semantics::Symbol *GetParentComponent(
+    const semantics::DerivedTypeDetails &details,
+    const semantics::Scope &scope) {
+  if (auto extends{details.GetParentComponentName()}) {
+    if (auto iter{scope.find(*extends)}; iter != scope.cend()) {
+      if (const Symbol & symbol{*iter->second};
+          symbol.test(semantics::Symbol::Flag::ParentComp)) {
+        return &symbol;
+      }
+    }
+  }
+  return nullptr;
+}
+
+static const semantics::Symbol *GetParentComponent(
+    const semantics::Symbol *symbol, const semantics::Scope &scope) {
+  if (symbol) {
+    if (const auto *dtDetails{
+            symbol->detailsIf<semantics::DerivedTypeDetails>()}) {
+      return GetParentComponent(*dtDetails, scope);
+    }
+  }
+  return nullptr;
+}
+
+static const semantics::DerivedTypeSpec *GetParentTypeSpec(
+    const semantics::Symbol *symbol, const semantics::Scope &scope) {
+  if (const Symbol * parentComponent{GetParentComponent(symbol, scope)}) {
+    return &parentComponent->get<semantics::ObjectEntityDetails>()
+                .type()
+                ->derivedTypeSpec();
+  } else {
+    return nullptr;
+  }
+}
+
+static const semantics::Scope *GetDerivedTypeParent(
+    const semantics::Scope *scope) {
+  if (scope) {
+    CHECK(scope->IsDerivedType());
+    if (const auto *parent{GetParentTypeSpec(scope->GetSymbol(), *scope)}) {
+      return parent->scope();
+    }
+  }
+  return nullptr;
+}
+
+static const semantics::Symbol *FindComponent(
+    const semantics::Scope *scope, parser::CharBlock name) {
+  if (!scope) {
+    return nullptr;
+  }
+  CHECK(scope->IsDerivedType());
+  auto found{scope->find(name)};
+  if (found != scope->end()) {
+    return &*found->second;
+  } else {
+    return FindComponent(GetDerivedTypeParent(scope), name);
+  }
+}
+
+static bool AreTypeParamCompatible(const semantics::DerivedTypeSpec &x,
+    const semantics::DerivedTypeSpec &y, bool ignoreLenParameters) {
+  const auto *xScope{x.typeSymbol().scope()};
+  const auto *yScope{y.typeSymbol().scope()};
+  for (const auto &[paramName, value] : x.parameters()) {
+    const auto *yValue{y.FindParameter(paramName)};
+    if (!yValue) {
+      return false;
+    }
+    const auto *xParm{FindComponent(xScope, paramName)};
+    const auto *yParm{FindComponent(yScope, paramName)};
+    if (xParm && yParm) {
+      const auto *xTPD{xParm->detailsIf<semantics::TypeParamDetails>()};
+      const auto *yTPD{yParm->detailsIf<semantics::TypeParamDetails>()};
+      if (xTPD && yTPD) {
+        if (xTPD->attr() != yTPD->attr()) {
+          return false;
+        }
+        if (!ignoreLenParameters ||
+            xTPD->attr() != common::TypeParamAttr::Len) {
+          auto xExpr{value.GetExplicit()};
+          auto yExpr{yValue->GetExplicit()};
+          if (xExpr && yExpr) {
+            auto xVal{ToInt64(*xExpr)};
+            auto yVal{ToInt64(*yExpr)};
+            if (xVal && yVal && *xVal != *yVal) {
+              return false;
+            }
+          }
+        }
+      }
+    }
+  }
+  for (const auto &[paramName, _] : y.parameters()) {
+    if (!x.FindParameter(paramName)) {
+      return false; // y has more parameters than x
+    }
+  }
+  return true;
+}
+
 static bool AreSameDerivedType(const semantics::DerivedTypeSpec &x,
-    const semantics::DerivedTypeSpec &y, SetOfDerivedTypePairs &inProgress) {
+    const semantics::DerivedTypeSpec &y, bool ignoreTypeParameterValues,
+    bool ignoreLenParameters, SetOfDerivedTypePairs &inProgress) {
+  if (&x == &y) {
+    return true;
+  }
+  if (!ignoreTypeParameterValues &&
+      !AreTypeParamCompatible(x, y, ignoreLenParameters)) {
+    return false;
+  }
   const auto &xSymbol{x.typeSymbol()};
   const auto &ySymbol{y.typeSymbol()};
-  if (&x == &y || xSymbol == ySymbol) {
+  if (xSymbol == ySymbol) {
     return true;
   }
+  if (xSymbol.name() != ySymbol.name()) {
+    return false;
+  }
   auto thisQuery{std::make_pair(&x, &y)};
   if (inProgress.find(thisQuery) != inProgress.end()) {
     return true; // recursive use of types in components
@@ -273,9 +393,6 @@ static bool AreSameDerivedType(const semantics::DerivedTypeSpec &x,
   inProgress.insert(thisQuery);
   const auto &xDetails{xSymbol.get<semantics::DerivedTypeDetails>()};
   const auto &yDetails{ySymbol.get<semantics::DerivedTypeDetails>()};
-  if (xSymbol.name() != ySymbol.name()) {
-    return false;
-  }
   if (!(xDetails.sequence() && yDetails.sequence()) &&
       !(xSymbol.attrs().test(semantics::Attr::BIND_C) &&
           ySymbol.attrs().test(semantics::Attr::BIND_C))) {
@@ -307,19 +424,23 @@ static bool AreSameDerivedType(const semantics::DerivedTypeSpec &x,
 bool AreSameDerivedType(
     const semantics::DerivedTypeSpec &x, const semantics::DerivedTypeSpec &y) {
   SetOfDerivedTypePairs inProgress;
-  return AreSameDerivedType(x, y, inProgress);
+  return AreSameDerivedType(x, y, false, false, inProgress);
 }
 
 static bool AreCompatibleDerivedTypes(const semantics::DerivedTypeSpec *x,
-    const semantics::DerivedTypeSpec *y, bool isPolymorphic) {
+    const semantics::DerivedTypeSpec *y, bool isPolymorphic,
+    bool ignoreTypeParameterValues, bool ignoreLenTypeParameters) {
   if (!x || !y) {
     return false;
   } else {
-    if (AreSameDerivedType(*x, *y)) {
+    SetOfDerivedTypePairs inProgress;
+    if (AreSameDerivedType(*x, *y, ignoreTypeParameterValues,
+            ignoreLenTypeParameters, inProgress)) {
       return true;
     } else {
       return isPolymorphic &&
-          AreCompatibleDerivedTypes(x, GetParentTypeSpec(*y), true);
+          AreCompatibleDerivedTypes(x, GetParentTypeSpec(*y), true,
+              ignoreTypeParameterValues, ignoreLenTypeParameters);
     }
   }
 }
@@ -342,9 +463,8 @@ static bool AreCompatibleTypes(const DynamicType &x, const DynamicType &y,
   } else {
     const auto *xdt{GetDerivedTypeSpec(x)};
     const auto *ydt{GetDerivedTypeSpec(y)};
-    return AreCompatibleDerivedTypes(xdt, ydt, x.IsPolymorphic()) &&
-        (ignoreTypeParameterValues ||
-            (xdt && ydt && AreTypeParamCompatible(*xdt, *ydt)));
+    return AreCompatibleDerivedTypes(
+        xdt, ydt, x.IsPolymorphic(), ignoreTypeParameterValues, false);
   }
 }
 
@@ -379,12 +499,13 @@ std::optional<bool> DynamicType::ExtendsTypeOf(const DynamicType &that) const {
   const auto *thatDts{evaluate::GetDerivedTypeSpec(that)};
   if (!thisDts || !thatDts) {
     return std::nullopt;
-  } else if (!AreCompatibleDerivedTypes(thatDts, thisDts, true)) {
+  } else if (!AreCompatibleDerivedTypes(thatDts, thisDts, true, true, true)) {
     // Note that I check *thisDts, not its parent, so that EXTENDS_TYPE_OF()
     // is .true. when they are the same type.  This is technically
     // an implementation-defined case in the standard, but every other
     // compiler works this way.
-    if (IsPolymorphic() && AreCompatibleDerivedTypes(thisDts, thatDts, true)) {
+    if (IsPolymorphic() &&
+        AreCompatibleDerivedTypes(thisDts, thatDts, true, true, true)) {
       // 'that' is *this or an extension of *this, and so runtime *this
       // could be an extension of 'that'
       return std::nullopt;
diff --git a/flang/lib/Frontend/CMakeLists.txt b/flang/lib/Frontend/CMakeLists.txt
index df5e324fdad71..7a04a2e0b4336 100644
--- a/flang/lib/Frontend/CMakeLists.txt
+++ b/flang/lib/Frontend/CMakeLists.txt
@@ -13,9 +13,7 @@ add_flang_library(flangFrontend
   TextDiagnostic.cpp
 
   DEPENDS
-  FIRBuilder
   FIRDialect
-  FIRSupport
   FIROptTransformsPassIncGen
   HLFIRDialect
   MLIRIR
@@ -28,6 +26,7 @@ add_flang_library(flangFrontend
   FortranCommon
   FortranLower
   FIRDialect
+  FIRDialectSupport
   FIRSupport
   FIRBuilder
   FIRCodeGen
diff --git a/flang/lib/Frontend/CompilerInvocation.cpp b/flang/lib/Frontend/CompilerInvocation.cpp
index 4fb1eb8ef902c..6e963e2528101 100644
--- a/flang/lib/Frontend/CompilerInvocation.cpp
+++ b/flang/lib/Frontend/CompilerInvocation.cpp
@@ -139,35 +139,47 @@ static void parseCodeGenArgs(Fortran::frontend::CodeGenOptions &opts,
        args.filtered(clang::driver::options::OPT_fembed_offload_object_EQ))
     opts.OffloadObjects.push_back(a->getValue());
 
+  // -flto=full/thin option.
+  if (const llvm::opt::Arg *a =
+          args.getLastArg(clang::driver::options::OPT_flto_EQ)) {
+    llvm::StringRef s = a->getValue();
+    assert((s == "full" || s == "thin") && "Unknown LTO mode.");
+    if (s == "full")
+      opts.PrepareForFullLTO = true;
+    else
+      opts.PrepareForThinLTO = true;
+  }
+
   // -mrelocation-model option.
-  if (const llvm::opt::Arg *A =
+  if (const llvm::opt::Arg *a =
           args.getLastArg(clang::driver::options::OPT_mrelocation_model)) {
-    llvm::StringRef ModelName = A->getValue();
-    auto RM = llvm::StringSwitch<std::optional<llvm::Reloc::Model>>(ModelName)
-                  .Case("static", llvm::Reloc::Static)
-                  .Case("pic", llvm::Reloc::PIC_)
-                  .Case("dynamic-no-pic", llvm::Reloc::DynamicNoPIC)
-                  .Case("ropi", llvm::Reloc::ROPI)
-                  .Case("rwpi", llvm::Reloc::RWPI)
-                  .Case("ropi-rwpi", llvm::Reloc::ROPI_RWPI)
-                  .Default(std::nullopt);
-    if (RM.has_value())
-      opts.setRelocationModel(*RM);
+    llvm::StringRef modelName = a->getValue();
+    auto relocModel =
+        llvm::StringSwitch<std::optional<llvm::Reloc::Model>>(modelName)
+            .Case("static", llvm::Reloc::Static)
+            .Case("pic", llvm::Reloc::PIC_)
+            .Case("dynamic-no-pic", llvm::Reloc::DynamicNoPIC)
+            .Case("ropi", llvm::Reloc::ROPI)
+            .Case("rwpi", llvm::Reloc::RWPI)
+            .Case("ropi-rwpi", llvm::Reloc::ROPI_RWPI)
+            .Default(std::nullopt);
+    if (relocModel.has_value())
+      opts.setRelocationModel(*relocModel);
     else
       diags.Report(clang::diag::err_drv_invalid_value)
-          << A->getAsString(args) << ModelName;
+          << a->getAsString(args) << modelName;
   }
 
   // -pic-level and -pic-is-pie option.
-  if (int PICLevel = getLastArgIntValue(
+  if (int picLevel = getLastArgIntValue(
           args, clang::driver::options::OPT_pic_level, 0, diags)) {
-    if (PICLevel > 2)
+    if (picLevel > 2)
       diags.Report(clang::diag::err_drv_invalid_value)
           << args.getLastArg(clang::driver::options::OPT_pic_level)
                  ->getAsString(args)
-          << PICLevel;
+          << picLevel;
 
-    opts.PICLevel = PICLevel;
+    opts.PICLevel = picLevel;
     if (args.hasArg(clang::driver::options::OPT_pic_is_pie))
       opts.IsPIE = 1;
   }
@@ -670,6 +682,10 @@ static bool parseDialectArgs(CompilerInvocation &res, llvm::opt::ArgList &args,
   if (args.hasArg(clang::driver::options::OPT_fopenmp)) {
     res.getFrontendOpts().features.Enable(
         Fortran::common::LanguageFeature::OpenMP);
+
+    if (args.hasArg(clang::driver::options::OPT_fopenmp_is_device)) {
+      res.getLangOpts().OpenMPIsDevice = 1;
+    }
   }
 
   // -pedantic
diff --git a/flang/lib/Frontend/FrontendAction.cpp b/flang/lib/Frontend/FrontendAction.cpp
index e6ede550092c8..d9fe1232e29be 100644
--- a/flang/lib/Frontend/FrontendAction.cpp
+++ b/flang/lib/Frontend/FrontendAction.cpp
@@ -48,7 +48,7 @@ bool FrontendAction::beginSourceFile(CompilerInstance &ci,
       unsigned diagID;
       if (llvm::vfs::getRealFileSystem()->exists(input.getFile())) {
         ci.getDiagnostics().Report(clang::diag::err_fe_error_reading)
-            << input.getFile();
+            << input.getFile() << "not a regular file";
         diagID = ci.getDiagnostics().getCustomDiagID(
             clang::DiagnosticsEngine::Error, "%0 is not a regular file");
       } else {
diff --git a/flang/lib/Frontend/FrontendActions.cpp b/flang/lib/Frontend/FrontendActions.cpp
index d2d3ab4cf1e35..64cabd87e6a41 100644
--- a/flang/lib/Frontend/FrontendActions.cpp
+++ b/flang/lib/Frontend/FrontendActions.cpp
@@ -18,9 +18,9 @@
 #include "flang/Lower/Bridge.h"
 #include "flang/Lower/PFTBuilder.h"
 #include "flang/Lower/Support/Verifier.h"
-#include "flang/Optimizer/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "flang/Optimizer/Support/InitFIR.h"
-#include "flang/Optimizer/Support/KindMapping.h"
 #include "flang/Optimizer/Support/Utils.h"
 #include "flang/Parser/dump-parse-tree.h"
 #include "flang/Parser/parsing.h"
@@ -177,6 +177,13 @@ bool CodeGenAction::beginSourceFileAction() {
 
   // Fetch module from lb, so we can set
   mlirModule = std::make_unique<mlir::ModuleOp>(lb.getModule());
+
+  if (ci.getInvocation().getFrontendOpts().features.IsEnabled(
+          Fortran::common::LanguageFeature::OpenMP)) {
+    mlir::omp::OpenMPDialect::setIsDevice(
+        *mlirModule, ci.getInvocation().getLangOpts().OpenMPIsDevice);
+  }
+
   setUpTargetMachine();
   const llvm::DataLayout &dl = tm->createDataLayout();
   setMLIRDataLayout(*mlirModule, dl);
@@ -692,7 +699,7 @@ void CodeGenAction::runOptimizationPipeline(llvm::raw_pwrite_stream &os) {
   std::optional<llvm::PGOOptions> pgoOpt;
   llvm::StandardInstrumentations si(llvmModule->getContext(),
                                     opts.DebugPassManager);
-  si.registerCallbacks(pic, &fam);
+  si.registerCallbacks(pic, &mam);
   llvm::PassBuilder pb(tm.get(), pto, pgoOpt, &pic);
 
   // Attempt to load pass plugins and register their callbacks with PB.
@@ -719,8 +726,10 @@ void CodeGenAction::runOptimizationPipeline(llvm::raw_pwrite_stream &os) {
 
   // Create the pass manager.
   llvm::ModulePassManager mpm;
-  if (opts.OptimizationLevel == 0)
-    mpm = pb.buildO0DefaultPipeline(level, false);
+  if (opts.PrepareForFullLTO)
+    mpm = pb.buildLTOPreLinkDefaultPipeline(level);
+  else if (opts.PrepareForThinLTO)
+    mpm = pb.buildThinLTOPreLinkDefaultPipeline(level);
   else
     mpm = pb.buildPerModuleDefaultPipeline(level);
 
diff --git a/flang/lib/Lower/Allocatable.cpp b/flang/lib/Lower/Allocatable.cpp
index 44956d65fc65c..8dfd3aa5db627 100644
--- a/flang/lib/Lower/Allocatable.cpp
+++ b/flang/lib/Lower/Allocatable.cpp
@@ -146,8 +146,8 @@ static void genRuntimeInitCharacter(fir::FirOpBuilder &builder,
       box.isPointer()
           ? fir::runtime::getRuntimeFunc<mkRTKey(PointerNullifyCharacter)>(
                 loc, builder)
-          : fir::runtime::getRuntimeFunc<mkRTKey(AllocatableInitCharacter)>(
-                loc, builder);
+          : fir::runtime::getRuntimeFunc<mkRTKey(
+                AllocatableInitCharacterForAllocate)>(loc, builder);
   llvm::ArrayRef<mlir::Type> inputTypes = callee.getFunctionType().getInputs();
   if (inputTypes.size() != 5)
     fir::emitFatalError(
@@ -371,13 +371,12 @@ class AllocateStmtHelper {
     fir::MutableBoxValue boxAddr =
         genMutableBoxValue(converter, loc, alloc.getAllocObj());
 
-    if (sourceExpr) {
-      genSourceAllocation(alloc, boxAddr);
-    } else if (moldExpr) {
-      genMoldAllocation(alloc, boxAddr);
-    } else {
+    if (sourceExpr)
+      genSourceMoldAllocation(alloc, boxAddr, /*isSource=*/true);
+    else if (moldExpr)
+      genSourceMoldAllocation(alloc, boxAddr, /*isSource=*/false);
+    else
       genSimpleAllocation(alloc, boxAddr);
-    }
   }
 
   static bool lowerBoundsAreOnes(const Allocation &alloc) {
@@ -557,8 +556,10 @@ class AllocateStmtHelper {
     }
   }
 
-  void genSourceAllocation(const Allocation &alloc,
-                           const fir::MutableBoxValue &box) {
+  void genSourceMoldAllocation(const Allocation &alloc,
+                               const fir::MutableBoxValue &box, bool isSource) {
+    fir::ExtendedValue exv = isSource ? sourceExv : moldExv;
+    ;
     // Generate a sequence of runtime calls.
     errorManager.genStatCheck(builder, loc);
     genAllocateObjectInit(box);
@@ -568,24 +569,17 @@ class AllocateStmtHelper {
     // from source for the deferred length parameter.
     if (lenParams.empty() && box.isCharacter() &&
         !box.hasNonDeferredLenParams())
-      lenParams.push_back(fir::factory::readCharLen(builder, loc, sourceExv));
-    if (alloc.type.IsPolymorphic())
-      genRuntimeAllocateApplyMold(builder, loc, box, sourceExv,
+      lenParams.push_back(fir::factory::readCharLen(builder, loc, exv));
+    if (!isSource || alloc.type.IsPolymorphic())
+      genRuntimeAllocateApplyMold(builder, loc, box, exv,
                                   alloc.getSymbol().Rank());
     genSetDeferredLengthParameters(alloc, box);
     genAllocateObjectBounds(alloc, box);
-    mlir::Value stat =
-        genRuntimeAllocateSource(builder, loc, box, sourceExv, errorManager);
-    fir::factory::syncMutableBoxFromIRBox(builder, loc, box);
-    errorManager.assignStat(builder, loc, stat);
-  }
-
-  void genMoldAllocation(const Allocation &alloc,
-                         const fir::MutableBoxValue &box) {
-    genRuntimeAllocateApplyMold(builder, loc, box, moldExv,
-                                alloc.getSymbol().Rank());
-    errorManager.genStatCheck(builder, loc);
-    mlir::Value stat = genRuntimeAllocate(builder, loc, box, errorManager);
+    mlir::Value stat;
+    if (isSource)
+      stat = genRuntimeAllocateSource(builder, loc, box, exv, errorManager);
+    else
+      stat = genRuntimeAllocate(builder, loc, box, errorManager);
     fir::factory::syncMutableBoxFromIRBox(builder, loc, box);
     errorManager.assignStat(builder, loc, stat);
   }
@@ -598,8 +592,8 @@ class AllocateStmtHelper {
         box.isPointer()
             ? fir::runtime::getRuntimeFunc<mkRTKey(PointerNullifyDerived)>(
                   loc, builder)
-            : fir::runtime::getRuntimeFunc<mkRTKey(AllocatableInitDerived)>(
-                  loc, builder);
+            : fir::runtime::getRuntimeFunc<mkRTKey(
+                  AllocatableInitDerivedForAllocate)>(loc, builder);
 
     llvm::ArrayRef<mlir::Type> inputTypes =
         callee.getFunctionType().getInputs();
@@ -625,8 +619,8 @@ class AllocateStmtHelper {
         box.isPointer()
             ? fir::runtime::getRuntimeFunc<mkRTKey(PointerNullifyIntrinsic)>(
                   loc, builder)
-            : fir::runtime::getRuntimeFunc<mkRTKey(AllocatableInitIntrinsic)>(
-                  loc, builder);
+            : fir::runtime::getRuntimeFunc<mkRTKey(
+                  AllocatableInitIntrinsicForAllocate)>(loc, builder);
 
     llvm::ArrayRef<mlir::Type> inputTypes =
         callee.getFunctionType().getInputs();
diff --git a/flang/lib/Lower/Bridge.cpp b/flang/lib/Lower/Bridge.cpp
index 5d5bb53d209a4..2ed552ed4ddd5 100644
--- a/flang/lib/Lower/Bridge.cpp
+++ b/flang/lib/Lower/Bridge.cpp
@@ -41,8 +41,8 @@
 #include "flang/Optimizer/Dialect/FIRAttr.h"
 #include "flang/Optimizer/Dialect/FIRDialect.h"
 #include "flang/Optimizer/Dialect/FIROps.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
 #include "flang/Optimizer/HLFIR/HLFIROps.h"
-#include "flang/Optimizer/Support/FIRContext.h"
 #include "flang/Optimizer/Support/FatalError.h"
 #include "flang/Optimizer/Support/InternalNames.h"
 #include "flang/Optimizer/Transforms/Passes.h"
@@ -3140,6 +3140,14 @@ class FirConverter : public Fortran::lower::AbstractConverter {
                 fir::factory::CharacterExprHelper{*builder, loc}.createAssign(
                     lhs, rhs);
               } else if (isDerivedCategory(lhsType->category())) {
+                // Handle parent component.
+                if (Fortran::lower::isParentComponent(assign.lhs)) {
+                  if (!fir::getBase(lhs).getType().isa<fir::BaseBoxType>())
+                    lhs = fir::getBase(builder->createBox(loc, lhs));
+                  lhs = Fortran::lower::updateBoxForParentComponent(*this, lhs,
+                                                                    assign.lhs);
+                }
+
                 // Fortran 2018 10.2.1.3 p13 and p14
                 // Recursively gen an assignment on each element pair.
                 fir::factory::genRecordAssignment(*builder, loc, lhs, rhs,
diff --git a/flang/lib/Lower/CMakeLists.txt b/flang/lib/Lower/CMakeLists.txt
index 7d2264c3ed8fb..e3c8504a0d602 100644
--- a/flang/lib/Lower/CMakeLists.txt
+++ b/flang/lib/Lower/CMakeLists.txt
@@ -27,17 +27,16 @@ add_flang_library(FortranLower
   Runtime.cpp
   SymbolMap.cpp
   VectorSubscripts.cpp
-
+  
   DEPENDS
   FIRDialect
-  FIRBuilder
-  FIRSupport
   FIRTransforms
   HLFIRDialect
   ${dialect_libs}
 
   LINK_LIBS
   FIRDialect
+  FIRDialectSupport
   FIRBuilder
   FIRSupport
   FIRTransforms
diff --git a/flang/lib/Lower/CallInterface.cpp b/flang/lib/Lower/CallInterface.cpp
index 9e86541660685..8f4b52ac654ea 100644
--- a/flang/lib/Lower/CallInterface.cpp
+++ b/flang/lib/Lower/CallInterface.cpp
@@ -965,7 +965,14 @@ class Fortran::lower::CallInterfaceImpl {
     mlir::Type boxType = fir::wrapInClassOrBoxType(
         type, obj.type.type().IsPolymorphic(), obj.type.type().IsAssumedType());
 
-    if (obj.attrs.test(Attrs::Allocatable) || obj.attrs.test(Attrs::Pointer)) {
+    if (obj.type.type().IsAssumedType() && isBindC) {
+      mlir::Type voidPtrType = fir::ReferenceType::get(
+          mlir::NoneType::get(&interface.converter.getMLIRContext()));
+      addFirOperand(voidPtrType, nextPassedArgPosition(), Property::BaseAddress,
+                    attrs);
+      addPassedArg(PassEntityBy::BaseAddress, entity, characteristics);
+    } else if (obj.attrs.test(Attrs::Allocatable) ||
+               obj.attrs.test(Attrs::Pointer)) {
       // Pass as fir.ref<fir.box> or fir.ref<fir.class>
       mlir::Type boxRefType = fir::ReferenceType::get(boxType);
       addFirOperand(boxRefType, nextPassedArgPosition(), Property::MutableBox,
diff --git a/flang/lib/Lower/ConvertCall.cpp b/flang/lib/Lower/ConvertCall.cpp
index 0fe48585126d9..dcdc4a55eb03b 100644
--- a/flang/lib/Lower/ConvertCall.cpp
+++ b/flang/lib/Lower/ConvertCall.cpp
@@ -35,7 +35,8 @@
 
 static llvm::cl::opt<bool> useHlfirIntrinsicOps(
     "use-hlfir-intrinsic-ops", llvm::cl::init(true),
-    llvm::cl::desc("Lower via HLFIR transformational intrinsic operations such as hlfir.sum"));
+    llvm::cl::desc("Lower via HLFIR transformational intrinsic operations such "
+                   "as hlfir.sum"));
 
 /// Helper to package a Value and its properties into an ExtendedValue.
 static fir::ExtendedValue toExtendedValue(mlir::Location loc, mlir::Value base,
@@ -298,6 +299,19 @@ fir::ExtendedValue Fortran::lower::genCallOpAndResult(
       funcPointer = builder.create<fir::AddrOfOp>(loc, funcOpType, symbolAttr);
     else
       funcSymbolAttr = symbolAttr;
+
+    // Issue a warning if the procedure name conflicts with
+    // a runtime function name a call to which has been already
+    // lowered (implying that the FuncOp has been created).
+    // The behavior is undefined in this case.
+    if (caller.getFuncOp()->hasAttrOfType<mlir::UnitAttr>(
+            fir::FIROpsDialect::getFirRuntimeAttrName()))
+      LLVM_DEBUG(mlir::emitWarning(
+          loc,
+          llvm::Twine("function name '") +
+              llvm::Twine(symbolAttr.getLeafReference()) +
+              llvm::Twine("' conflicts with a runtime function name used by "
+                          "Flang - this may lead to undefined behavior")));
   }
 
   mlir::FunctionType funcType =
@@ -382,9 +396,7 @@ fir::ExtendedValue Fortran::lower::genCallOpAndResult(
           component->GetFirstSymbol(), &symMap);
       mlir::Value passObject = fir::getBase(pass);
       if (fir::isa_ref_type(passObject.getType()))
-        passObject = builder.create<fir::ConvertOp>(
-            loc, passObject.getType().dyn_cast<fir::ReferenceType>().getEleTy(),
-            passObject);
+        passObject = builder.create<fir::LoadOp>(loc, passObject);
       dispatch = builder.create<fir::DispatchOp>(
           loc, funcType.getResults(), builder.getStringAttr(procName),
           passObject, operands, nullptr);
@@ -408,12 +420,13 @@ fir::ExtendedValue Fortran::lower::genCallOpAndResult(
 
   if (allocatedResult) {
     // 7.5.6.3 point 5. Derived-type finalization for nonpointer function.
-    // Check if the derived-type is finalizable if it is a monorphic
+    // Check if the derived-type is finalizable if it is a monomorphic
     // derived-type.
     // For polymorphic and unlimited polymorphic enities call the runtime
     // in any cases.
     std::optional<Fortran::evaluate::DynamicType> retTy =
         caller.getCallDescription().proc().GetType();
+    bool cleanupWithDestroy = false;
     if (!fir::isPointerType(funcType.getResults()[0]) && retTy &&
         (retTy->category() == Fortran::common::TypeCategory::Derived ||
          retTy->IsPolymorphic() || retTy->IsUnlimitedPolymorphic())) {
@@ -423,6 +436,7 @@ fir::ExtendedValue Fortran::lower::genCallOpAndResult(
           fir::runtime::genDerivedTypeDestroy(*bldr, loc,
                                               fir::getBase(*allocatedResult));
         });
+        cleanupWithDestroy = true;
       } else {
         const Fortran::semantics::DerivedTypeSpec &typeSpec =
             retTy->GetDerivedTypeSpec();
@@ -432,12 +446,13 @@ fir::ExtendedValue Fortran::lower::genCallOpAndResult(
             mlir::Value box = bldr->createBox(loc, *allocatedResult);
             fir::runtime::genDerivedTypeDestroy(*bldr, loc, box);
           });
+          cleanupWithDestroy = true;
         }
       }
     }
     allocatedResult->match(
         [&](const fir::MutableBoxValue &box) {
-          if (box.isAllocatable()) {
+          if (box.isAllocatable() && !cleanupWithDestroy) {
             // 9.7.3.2 point 4. Finalize allocatables.
             fir::FirOpBuilder *bldr = &converter.getFirOpBuilder();
             stmtCtx.attachCleanup([bldr, loc, box]() {
@@ -561,6 +576,8 @@ struct CallContext {
 
   fir::FirOpBuilder &getBuilder() { return converter.getFirOpBuilder(); }
 
+  std::string getProcedureName() const { return procRef.proc().GetName(); }
+
   /// Is this a call to an elemental procedure with at least one array argument?
   bool isElementalProcWithArrayArgs() const {
     if (procRef.IsElemental())
@@ -761,7 +778,7 @@ struct ConditionallyPreparedDummy {
       if (type == i1Type)
         elseResultValues.push_back(builder.createBool(loc, false));
       else
-        elseResultValues.push_back(builder.create<fir::AbsentOp>(loc, type));
+        elseResultValues.push_back(builder.genAbsentOp(loc, type));
     }
     builder.create<fir::ResultOp>(loc, elseResultValues);
   }
@@ -1044,7 +1061,7 @@ genUserCall(PreparedActualArguments &loweredActuals,
     mlir::Type argTy = callSiteType.getInput(arg.firArgument);
     if (!preparedActual) {
       // Optional dummy argument for which there is no actual argument.
-      caller.placeInput(arg, builder.create<fir::AbsentOp>(loc, argTy));
+      caller.placeInput(arg, builder.genAbsentOp(loc, argTy));
       continue;
     }
     const auto *expr = arg.entity->UnwrapExpr();
@@ -1146,7 +1163,7 @@ genUserCall(PreparedActualArguments &loweredActuals,
 /// pre-lowered but have not yet been prepared according to the interface.
 static std::optional<hlfir::EntityWithAttributes>
 genIntrinsicRefCore(PreparedActualArguments &loweredActuals,
-                    const Fortran::evaluate::SpecificIntrinsic &intrinsic,
+                    const Fortran::evaluate::SpecificIntrinsic *intrinsic,
                     const fir::IntrinsicArgumentLoweringRules *argLowering,
                     CallContext &callContext) {
   llvm::SmallVector<fir::ExtendedValue> operands;
@@ -1213,10 +1230,10 @@ genIntrinsicRefCore(PreparedActualArguments &loweredActuals,
   std::optional<mlir::Type> scalarResultType;
   if (callContext.resultType)
     scalarResultType = hlfir::getFortranElementType(*callContext.resultType);
+  const std::string intrinsicName = callContext.getProcedureName();
   // Let the intrinsic library lower the intrinsic procedure call.
-  auto [resultExv, mustBeFreed] =
-      genIntrinsicCall(callContext.getBuilder(), loc, intrinsic.name,
-                       scalarResultType, operands);
+  auto [resultExv, mustBeFreed] = genIntrinsicCall(
+      callContext.getBuilder(), loc, intrinsicName, scalarResultType, operands);
   if (!fir::getBase(resultExv))
     return std::nullopt;
   hlfir::EntityWithAttributes resultEntity = extendedValueToHlfirEntity(
@@ -1229,7 +1246,7 @@ genIntrinsicRefCore(PreparedActualArguments &loweredActuals,
     // (this is the only intrinsic implemented in that way so far). The
     // ownership of this address cannot be taken here since it may not be a
     // temp.
-    if (intrinsic.name == "merge")
+    if (intrinsicName == "merge")
       asExpr = builder.create<hlfir::AsExprOp>(loc, resultEntity);
     else
       asExpr = builder.create<hlfir::AsExprOp>(
@@ -1243,11 +1260,12 @@ genIntrinsicRefCore(PreparedActualArguments &loweredActuals,
 /// pre-lowered but have not yet been prepared according to the interface.
 static std::optional<hlfir::EntityWithAttributes>
 genHLFIRIntrinsicRefCore(PreparedActualArguments &loweredActuals,
-                         const Fortran::evaluate::SpecificIntrinsic &intrinsic,
+                         const Fortran::evaluate::SpecificIntrinsic *intrinsic,
                          const fir::IntrinsicArgumentLoweringRules *argLowering,
                          CallContext &callContext) {
   if (!useHlfirIntrinsicOps)
-    return genIntrinsicRefCore(loweredActuals, intrinsic, argLowering, callContext);
+    return genIntrinsicRefCore(loweredActuals, intrinsic, argLowering,
+                               callContext);
 
   fir::FirOpBuilder &builder = callContext.getBuilder();
   mlir::Location loc = callContext.loc;
@@ -1293,8 +1311,8 @@ genHLFIRIntrinsicRefCore(PreparedActualArguments &loweredActuals,
     return hlfir::ExprType::get(builder.getContext(), resultShape, elementType,
                                 /*polymorphic=*/false);
   };
-
-  if (intrinsic.name == "sum") {
+  const std::string intrinsicName = callContext.getProcedureName();
+  if (intrinsicName == "sum") {
     llvm::SmallVector<mlir::Value> operands = getOperandVector(loweredActuals);
     assert(operands.size() == 3);
     mlir::Value array = operands[0];
@@ -1308,7 +1326,7 @@ genHLFIRIntrinsicRefCore(PreparedActualArguments &loweredActuals,
         builder.create<hlfir::SumOp>(loc, resultTy, array, dim, mask);
     return {hlfir::EntityWithAttributes{sumOp.getResult()}};
   }
-  if (intrinsic.name == "matmul") {
+  if (intrinsicName == "matmul") {
     llvm::SmallVector<mlir::Value> operands = getOperandVector(loweredActuals);
     mlir::Type resultTy =
         computeResultType(operands[0], *callContext.resultType);
@@ -1317,7 +1335,7 @@ genHLFIRIntrinsicRefCore(PreparedActualArguments &loweredActuals,
 
     return {hlfir::EntityWithAttributes{matmulOp.getResult()}};
   }
-  if (intrinsic.name == "transpose") {
+  if (intrinsicName == "transpose") {
     llvm::SmallVector<mlir::Value> operands = getOperandVector(loweredActuals);
     hlfir::ExprType::Shape resultShape;
     mlir::Type normalisedResult =
@@ -1509,7 +1527,7 @@ class ElementalIntrinsicCallBuilder
     : public ElementalCallBuilder<ElementalIntrinsicCallBuilder> {
 public:
   ElementalIntrinsicCallBuilder(
-      const Fortran::evaluate::SpecificIntrinsic &intrinsic,
+      const Fortran::evaluate::SpecificIntrinsic *intrinsic,
       const fir::IntrinsicArgumentLoweringRules *argLowering, bool isFunction)
       : intrinsic{intrinsic}, argLowering{argLowering}, isFunction{isFunction} {
   }
@@ -1530,11 +1548,12 @@ class ElementalIntrinsicCallBuilder
   mlir::Value
   computeDynamicCharacterResultLength(PreparedActualArguments &loweredActuals,
                                       CallContext &callContext) {
-    if (intrinsic.name == "adjustr" || intrinsic.name == "adjustl" ||
-        intrinsic.name == "merge")
-      return hlfir::genCharLength(
-          callContext.loc, callContext.getBuilder(),
-          loweredActuals[0].value().getOriginalActual());
+    if (intrinsic)
+      if (intrinsic->name == "adjustr" || intrinsic->name == "adjustl" ||
+          intrinsic->name == "merge")
+        return hlfir::genCharLength(
+            callContext.loc, callContext.getBuilder(),
+            loweredActuals[0].value().getOriginalActual());
     // Character MIN/MAX is the min/max of the arguments length that are
     // present.
     TODO(callContext.loc,
@@ -1542,7 +1561,7 @@ class ElementalIntrinsicCallBuilder
   }
 
 private:
-  const Fortran::evaluate::SpecificIntrinsic &intrinsic;
+  const Fortran::evaluate::SpecificIntrinsic *intrinsic;
   const fir::IntrinsicArgumentLoweringRules *argLowering;
   const bool isFunction;
 };
@@ -1581,18 +1600,22 @@ genIsPresentIfArgMaybeAbsent(mlir::Location loc, hlfir::Entity actual,
 }
 
 /// Lower an intrinsic procedure reference.
+/// \p intrinsic is null if this is an intrinsic module procedure that must be
+/// lowered as if it were an intrinsic module procedure (like C_LOC which is a
+/// procedure from intrinsic module iso_c_binding). Otherwise, \p intrinsic
+/// must not be null.
 static std::optional<hlfir::EntityWithAttributes>
-genIntrinsicRef(const Fortran::evaluate::SpecificIntrinsic &intrinsic,
+genIntrinsicRef(const Fortran::evaluate::SpecificIntrinsic *intrinsic,
                 CallContext &callContext) {
   mlir::Location loc = callContext.loc;
   auto &converter = callContext.converter;
-  if (Fortran::lower::intrinsicRequiresCustomOptionalHandling(
-          callContext.procRef, intrinsic, converter))
+  if (intrinsic && Fortran::lower::intrinsicRequiresCustomOptionalHandling(
+                       callContext.procRef, *intrinsic, converter))
     TODO(loc, "special cases of intrinsic with optional arguments");
 
   PreparedActualArguments loweredActuals;
   const fir::IntrinsicArgumentLoweringRules *argLowering =
-      fir::getIntrinsicArgumentLowering(intrinsic.name);
+      fir::getIntrinsicArgumentLowering(callContext.getProcedureName());
   for (const auto &arg : llvm::enumerate(callContext.procRef.arguments())) {
     auto *expr =
         Fortran::evaluate::UnwrapExpr<Fortran::lower::SomeExpr>(arg.value());
@@ -1638,7 +1661,9 @@ static std::optional<hlfir::EntityWithAttributes>
 genProcedureRef(CallContext &callContext) {
   mlir::Location loc = callContext.loc;
   if (auto *intrinsic = callContext.procRef.proc().GetSpecificIntrinsic())
-    return genIntrinsicRef(*intrinsic, callContext);
+    return genIntrinsicRef(intrinsic, callContext);
+  if (Fortran::lower::isIntrinsicModuleProcRef(callContext.procRef))
+    return genIntrinsicRef(nullptr, callContext);
 
   if (callContext.isStatementFunctionCall())
     return genStmtFunctionRef(loc, callContext.converter, callContext.symMap,
@@ -1699,6 +1724,17 @@ genProcedureRef(CallContext &callContext) {
   return genUserCall(loweredActuals, caller, callSiteType, callContext);
 }
 
+bool Fortran::lower::isIntrinsicModuleProcRef(
+    const Fortran::evaluate::ProcedureRef &procRef) {
+  const Fortran::semantics::Symbol *symbol = procRef.proc().GetSymbol();
+  if (!symbol)
+    return false;
+  const Fortran::semantics::Symbol *module =
+      symbol->GetUltimate().owner().GetSymbol();
+  return module && module->attrs().test(Fortran::semantics::Attr::INTRINSIC) &&
+         module->name().ToString().find("omp_lib") == std::string::npos;
+}
+
 std::optional<hlfir::EntityWithAttributes> Fortran::lower::convertCallToHLFIR(
     mlir::Location loc, Fortran::lower::AbstractConverter &converter,
     const evaluate::ProcedureRef &procRef, std::optional<mlir::Type> resultType,
diff --git a/flang/lib/Lower/ConvertExpr.cpp b/flang/lib/Lower/ConvertExpr.cpp
index 1407b5ea81f73..03b803e8271b1 100644
--- a/flang/lib/Lower/ConvertExpr.cpp
+++ b/flang/lib/Lower/ConvertExpr.cpp
@@ -543,6 +543,18 @@ createBoxProcCharTuple(Fortran::lower::AbstractConverter &converter,
       argTy.cast<mlir::TupleType>().getType(0).cast<fir::BoxProcType>();
   mlir::Location loc = converter.getCurrentLocation();
   auto &builder = converter.getFirOpBuilder();
+
+  // While character procedure arguments are expected here, Fortran allows
+  // actual arguments of other types to be passed instead.
+  // To support this, we cast any reference to the expected type or extract
+  // procedures from their boxes if needed.
+  mlir::Type fromTy = funcAddr.getType();
+  mlir::Type toTy = boxTy.getEleTy();
+  if (fir::isa_ref_type(fromTy))
+    funcAddr = builder.createConvert(loc, toTy, funcAddr);
+  else if (fromTy.isa<fir::BoxProcType>())
+    funcAddr = builder.create<fir::BoxAddrOp>(loc, toTy, funcAddr);
+
   auto boxProc = [&]() -> mlir::Value {
     if (auto host = Fortran::lower::argumentHostAssocs(converter, funcAddr))
       return builder.create<fir::EmboxProcOp>(
@@ -589,17 +601,6 @@ const Fortran::semantics::Symbol &getLastSym(const A &obj) {
   return obj.GetLastSymbol().GetUltimate();
 }
 
-static bool
-isIntrinsicModuleProcRef(const Fortran::evaluate::ProcedureRef &procRef) {
-  const Fortran::semantics::Symbol *symbol = procRef.proc().GetSymbol();
-  if (!symbol)
-    return false;
-  const Fortran::semantics::Symbol *module =
-      symbol->GetUltimate().owner().GetSymbol();
-  return module && module->attrs().test(Fortran::semantics::Attr::INTRINSIC) &&
-         module->name().ToString().find("omp_lib") == std::string::npos;
-}
-
 // Return true if TRANSPOSE should be lowered without a runtime call.
 static bool
 isTransposeOptEnabled(const Fortran::lower::AbstractConverter &converter) {
@@ -1764,6 +1765,11 @@ class ScalarExprLowering {
                                               fir::getBase(exv));
     }
     mlir::Value box = builder.createBox(loc, exv, exv.isPolymorphic());
+    if (Fortran::lower::isParentComponent(expr)) {
+      fir::ExtendedValue newExv =
+          Fortran::lower::updateBoxForParentComponent(converter, box, expr);
+      box = fir::getBase(newExv);
+    }
     return fir::BoxValue(
         box, fir::factory::getNonDefaultLowerBounds(builder, loc, exv),
         fir::factory::getNonDeferredLenParams(exv));
@@ -2442,7 +2448,7 @@ class ScalarExprLowering {
             procRef.proc().GetSpecificIntrinsic())
       return genIntrinsicRef(procRef, resultType, *intrinsic);
 
-    if (isIntrinsicModuleProcRef(procRef))
+    if (Fortran::lower::isIntrinsicModuleProcRef(procRef))
       return genIntrinsicRef(procRef, resultType);
 
     if (isStatementFunctionCall(procRef))
@@ -2466,7 +2472,7 @@ class ScalarExprLowering {
       mlir::Type argTy = callSiteType.getInput(arg.firArgument);
       if (!actual) {
         // Optional dummy argument for which there is no actual argument.
-        caller.placeInput(arg, builder.create<fir::AbsentOp>(loc, argTy));
+        caller.placeInput(arg, builder.genAbsentOp(loc, argTy));
         continue;
       }
       const auto *expr = actual->UnwrapExpr();
@@ -2560,7 +2566,22 @@ class ScalarExprLowering {
                  arg.passBy == PassBy::CharBoxValueAttribute);
           auto helper = fir::factory::CharacterExprHelper{builder, loc};
           auto boxChar = argAddr.match(
-              [&](const fir::CharBoxValue &x) { return helper.createEmbox(x); },
+              [&](const fir::CharBoxValue &x) -> mlir::Value {
+                // If a character procedure was passed instead, handle the
+                // mismatch.
+                auto funcTy =
+                    x.getAddr().getType().dyn_cast<mlir::FunctionType>();
+                if (funcTy && funcTy.getNumResults() == 1 &&
+                    funcTy.getResult(0).isa<fir::BoxCharType>()) {
+                  auto boxTy = funcTy.getResult(0).cast<fir::BoxCharType>();
+                  mlir::Value ref = builder.createConvert(
+                      loc, builder.getRefType(boxTy.getEleTy()), x.getAddr());
+                  auto len = builder.create<fir::UndefOp>(
+                      loc, builder.getCharacterLengthType());
+                  return builder.create<fir::EmboxCharOp>(loc, boxTy, ref, len);
+                }
+                return helper.createEmbox(x);
+              },
               [&](const fir::CharArrayBoxValue &x) {
                 return helper.createEmbox(x);
               },
@@ -2693,39 +2714,49 @@ class ScalarExprLowering {
             // actually a variable.
             box = Fortran::evaluate::IsVariable(*expr)
                       ? builder.createBox(loc, genBoxArg(*expr),
-                                          fir::isPolymorphicType(argTy))
+                                          fir::isPolymorphicType(argTy),
+                                          fir::isAssumedType(argTy))
                       : builder.createBox(getLoc(), genTempExtAddr(*expr),
-                                          fir::isPolymorphicType(argTy));
-
+                                          fir::isPolymorphicType(argTy),
+                                          fir::isAssumedType(argTy));
             if (box.getType().isa<fir::BoxType>() &&
-                fir::isPolymorphicType(argTy)) {
+                fir::isPolymorphicType(argTy) && !fir::isAssumedType(argTy)) {
+              mlir::Type actualTy = argTy;
+              if (Fortran::lower::isParentComponent(*expr))
+                actualTy = fir::BoxType::get(converter.genType(*expr));
               // Rebox can only be performed on a present argument.
               if (arg.isOptional()) {
                 mlir::Value isPresent =
                     genActualIsPresentTest(builder, loc, box);
                 box = builder
-                          .genIfOp(loc, {argTy}, isPresent,
+                          .genIfOp(loc, {actualTy}, isPresent,
                                    /*withElseRegion=*/true)
                           .genThen([&]() {
-                            auto rebox = builder
-                                             .create<fir::ReboxOp>(
-                                                 loc, argTy, box, mlir::Value{},
-                                                 /*slice=*/mlir::Value{})
-                                             .getResult();
+                            auto rebox =
+                                builder
+                                    .create<fir::ReboxOp>(
+                                        loc, actualTy, box, mlir::Value{},
+                                        /*slice=*/mlir::Value{})
+                                    .getResult();
                             builder.create<fir::ResultOp>(loc, rebox);
                           })
                           .genElse([&]() {
                             auto absent =
-                                builder.create<fir::AbsentOp>(loc, argTy)
+                                builder.create<fir::AbsentOp>(loc, actualTy)
                                     .getResult();
                             builder.create<fir::ResultOp>(loc, absent);
                           })
                           .getResults()[0];
               } else {
-                box =
-                    builder.create<fir::ReboxOp>(loc, argTy, box, mlir::Value{},
-                                                 /*slice=*/mlir::Value{});
+                box = builder.create<fir::ReboxOp>(loc, actualTy, box,
+                                                   mlir::Value{},
+                                                   /*slice=*/mlir::Value{});
               }
+            } else if (Fortran::lower::isParentComponent(*expr)) {
+              fir::ExtendedValue newExv =
+                  Fortran::lower::updateBoxForParentComponent(converter, box,
+                                                              *expr);
+              box = fir::getBase(newExv);
             }
           }
           caller.placeInput(arg, box);
@@ -4795,7 +4826,7 @@ class ArrayExprLowering {
         // The intrinsic procedure is called once per element of the array.
         return genElementalIntrinsicProcRef(procRef, retTy, *intrin);
       }
-      if (isIntrinsicModuleProcRef(procRef))
+      if (Fortran::lower::isIntrinsicModuleProcRef(procRef))
         return genElementalIntrinsicProcRef(procRef, retTy);
       if (ScalarExprLowering::isStatementFunctionCall(procRef))
         fir::emitFatalError(loc, "statement function cannot be elemental");
@@ -7199,7 +7230,7 @@ fir::MutableBoxValue Fortran::lower::createMutableBox(
       .genMutableBoxValue(expr);
 }
 
-bool isParentComponent(const Fortran::lower::SomeExpr &expr) {
+bool Fortran::lower::isParentComponent(const Fortran::lower::SomeExpr &expr) {
   if (const Fortran::semantics::Symbol * symbol{GetLastSymbol(expr)}) {
     if (symbol->test(Fortran::semantics::Symbol::Flag::ParentComp))
       return true;
@@ -7207,24 +7238,6 @@ bool isParentComponent(const Fortran::lower::SomeExpr &expr) {
   return false;
 }
 
-template <typename OP>
-mlir::Value createSliceForParentComp(fir::FirOpBuilder &builder,
-                                     mlir::Location loc, OP boxOp,
-                                     fir::ExtendedValue box, mlir::Value field,
-                                     bool isArray) {
-  if (boxOp.getSlice()) {
-    mlir::Value existingSlice = boxOp.getSlice();
-    fir::SliceOp sliceOp =
-        mlir::dyn_cast<fir::SliceOp>(existingSlice.getDefiningOp());
-    llvm::SmallVector<mlir::Value> fields = sliceOp.getFields();
-    fields.push_back(field);
-    return builder.createSlice(loc, box, sliceOp.getTriples(), fields);
-  }
-  if (isArray)
-    return builder.createSlice(loc, box, {}, {field});
-  return {};
-}
-
 // Handling special case where the last component is referring to the
 // parent component.
 //
@@ -7239,54 +7252,35 @@ mlir::Value createSliceForParentComp(fir::FirOpBuilder &builder,
 // y(:)%t  ! just need to update the box with a slice pointing to the first
 //         ! component of `t`.
 // a%t     ! simple conversion to TYPE(t).
-fir::ExtendedValue
-updateBoxForParentComponent(Fortran::lower::AbstractConverter &converter,
-                            fir::ExtendedValue box,
-                            const Fortran::lower::SomeExpr &expr) {
+fir::ExtendedValue Fortran::lower::updateBoxForParentComponent(
+    Fortran::lower::AbstractConverter &converter, fir::ExtendedValue box,
+    const Fortran::lower::SomeExpr &expr) {
   mlir::Location loc = converter.getCurrentLocation();
   auto &builder = converter.getFirOpBuilder();
   mlir::Value boxBase = fir::getBase(box);
   mlir::Operation *op = boxBase.getDefiningOp();
-  fir::BoxType boxTy = boxBase.getType().dyn_cast<fir::BoxType>();
-  mlir::Type boxEleTy = fir::unwrapAllRefAndSeqType(boxTy.getEleTy());
-  auto originalRecTy = boxEleTy.dyn_cast<fir::RecordType>();
   mlir::Type actualTy = converter.genType(expr);
-  mlir::Type eleTy = fir::unwrapAllRefAndSeqType(actualTy);
-  auto parentCompTy = eleTy.dyn_cast<fir::RecordType>();
-  assert(parentCompTy && "expecting derived-type");
-
-  assert(
-      (mlir::dyn_cast<fir::EmboxOp>(op) || mlir::dyn_cast<fir::ReboxOp>(op)) &&
-      "expecting fir.embox or fir.rebox operation");
-
-  if (parentCompTy.getTypeList().empty())
-    TODO(loc, "parent component with no component");
-
-  // Creating a slice with a path to the first component of the parent component
-  // of the extended type.
-  auto firstComponent = parentCompTy.getTypeList().front();
-  auto fieldTy = fir::FieldType::get(boxTy.getContext());
-  auto field = builder.create<fir::FieldIndexOp>(
-      loc, fieldTy, firstComponent.first, originalRecTy,
-      /*typeParams=*/mlir::ValueRange{});
-
-  if (auto embox = mlir::dyn_cast<fir::EmboxOp>(op)) {
-    mlir::Value slice = createSliceForParentComp(builder, loc, embox, box,
-                                                 field, expr.Rank() > 0);
-    auto newBox = builder.create<fir::EmboxOp>(
-        loc, fir::BoxType::get(actualTy), embox.getMemref(), embox.getShape(),
-        slice, embox.getTypeparams());
-    return fir::substBase(box, fir::getBase(newBox));
-  }
-  if (auto rebox = mlir::dyn_cast<fir::ReboxOp>(op)) {
-    mlir::Value slice = createSliceForParentComp(builder, loc, rebox, box,
-                                                 field, expr.Rank() > 0);
-    auto newBox =
-        builder.create<fir::ReboxOp>(loc, fir::BoxType::get(actualTy),
-                                     rebox.getBox(), rebox.getShape(), slice);
-    return fir::substBase(box, fir::getBase(newBox));
-  }
-  return box;
+
+  if (op) {
+    if (auto embox = mlir::dyn_cast<fir::EmboxOp>(op)) {
+      auto newBox = builder.create<fir::EmboxOp>(
+          loc, fir::BoxType::get(actualTy), embox.getMemref(), embox.getShape(),
+          embox.getSlice(), embox.getTypeparams());
+      return fir::substBase(box, newBox);
+    }
+    if (auto rebox = mlir::dyn_cast<fir::ReboxOp>(op)) {
+      auto newBox = builder.create<fir::ReboxOp>(
+          loc, fir::BoxType::get(actualTy), rebox.getBox(), rebox.getShape(),
+          rebox.getSlice());
+      return fir::substBase(box, newBox);
+    }
+  }
+
+  mlir::Value empty;
+  mlir::ValueRange emptyRange;
+  return builder.create<fir::ReboxOp>(loc, fir::BoxType::get(actualTy), boxBase,
+                                      /*shape=*/empty,
+                                      /*slice=*/empty);
 }
 
 fir::ExtendedValue Fortran::lower::createBoxValue(
diff --git a/flang/lib/Lower/ConvertExprToHLFIR.cpp b/flang/lib/Lower/ConvertExprToHLFIR.cpp
index 68839bacd6ef1..2dd61bb15833d 100644
--- a/flang/lib/Lower/ConvertExprToHLFIR.cpp
+++ b/flang/lib/Lower/ConvertExprToHLFIR.cpp
@@ -88,6 +88,7 @@ class HlfirDesignatorBuilder {
     std::string componentName{};
     mlir::Value componentShape;
     hlfir::DesignateOp::Subscripts subscripts;
+    std::optional<bool> complexPart;
     mlir::Value resultShape;
     llvm::SmallVector<mlir::Value> typeParams;
     llvm::SmallVector<mlir::Value, 2> substring;
@@ -98,8 +99,14 @@ class HlfirDesignatorBuilder {
   // fir.box)...
   template <typename T>
   mlir::Type computeDesignatorType(mlir::Type resultValueType,
-                                   const PartInfo &partInfo,
+                                   PartInfo &partInfo,
                                    const T &designatorNode) {
+    // Get base's shape if its a sequence type with no previously computed
+    // result shape
+    if (partInfo.base && resultValueType.isa<fir::SequenceType>() &&
+        !partInfo.resultShape)
+      partInfo.resultShape =
+          hlfir::genShape(getLoc(), getBuilder(), *partInfo.base);
     // Dynamic type of polymorphic base must be kept if the designator is
     // polymorphic.
     if (isPolymorphic(designatorNode))
@@ -144,11 +151,10 @@ class HlfirDesignatorBuilder {
   fir::FortranVariableOpInterface
   genDesignate(mlir::Type designatorType, PartInfo &partInfo,
                fir::FortranVariableFlagsAttr attributes) {
-    std::optional<bool> complexPart;
     auto designate = getBuilder().create<hlfir::DesignateOp>(
         getLoc(), designatorType, partInfo.base.value().getBase(),
         partInfo.componentName, partInfo.componentShape, partInfo.subscripts,
-        partInfo.substring, complexPart, partInfo.resultShape,
+        partInfo.substring, partInfo.complexPart, partInfo.resultShape,
         partInfo.typeParams, attributes);
     return mlir::cast<fir::FortranVariableOpInterface>(
         designate.getOperation());
@@ -240,7 +246,21 @@ class HlfirDesignatorBuilder {
 
   fir::FortranVariableOpInterface
   gen(const Fortran::evaluate::ComplexPart &complexPart) {
-    TODO(getLoc(), "lowering complex part to HLFIR");
+    PartInfo partInfo;
+    fir::factory::Complex cmplxHelper(getBuilder(), getLoc());
+
+    bool complexBit =
+        complexPart.part() == Fortran::evaluate::ComplexPart::Part::IM;
+    partInfo.complexPart = {complexBit};
+
+    mlir::Type resultType = visit(complexPart.complex(), partInfo);
+
+    // Determine complex part type
+    mlir::Type base = hlfir::getFortranElementType(resultType);
+    mlir::Type cmplxValueType = cmplxHelper.getComplexPartType(base);
+    mlir::Type designatorType = changeElementType(resultType, cmplxValueType);
+
+    return genDesignate(designatorType, partInfo, complexPart);
   }
 
   fir::FortranVariableOpInterface
diff --git a/flang/lib/Lower/ConvertVariable.cpp b/flang/lib/Lower/ConvertVariable.cpp
index 9a19e422ba454..1d91c86c3d2f5 100644
--- a/flang/lib/Lower/ConvertVariable.cpp
+++ b/flang/lib/Lower/ConvertVariable.cpp
@@ -32,8 +32,8 @@
 #include "flang/Optimizer/Dialect/FIRAttr.h"
 #include "flang/Optimizer/Dialect/FIRDialect.h"
 #include "flang/Optimizer/Dialect/FIROps.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
 #include "flang/Optimizer/HLFIR/HLFIROps.h"
-#include "flang/Optimizer/Support/FIRContext.h"
 #include "flang/Optimizer/Support/FatalError.h"
 #include "flang/Optimizer/Support/InternalNames.h"
 #include "flang/Semantics/runtime-type-info.h"
@@ -726,13 +726,7 @@ static void deallocateIntentOut(Fortran::lower::AbstractConverter &converter,
           return;
       mlir::Location loc = converter.getCurrentLocation();
       fir::FirOpBuilder &builder = converter.getFirOpBuilder();
-      if (Fortran::semantics::IsOptional(sym)) {
-        auto isPresent = builder.create<fir::IsPresentOp>(
-            loc, builder.getI1Type(), fir::getBase(extVal));
-        builder.genIfThen(loc, isPresent)
-            .genThen([&]() { genDeallocateBox(converter, *mutBox, loc); })
-            .end();
-      } else {
+      auto genDeallocateWithTypeDesc = [&]() {
         if (mutBox->isDerived() || mutBox->isPolymorphic() ||
             mutBox->isUnlimitedPolymorphic()) {
           mlir::Value isAlloc = fir::factory::genIsAllocatedOrAssociatedTest(
@@ -756,6 +750,16 @@ static void deallocateIntentOut(Fortran::lower::AbstractConverter &converter,
         } else {
           genDeallocateBox(converter, *mutBox, loc);
         }
+      };
+
+      if (Fortran::semantics::IsOptional(sym)) {
+        auto isPresent = builder.create<fir::IsPresentOp>(
+            loc, builder.getI1Type(), fir::getBase(extVal));
+        builder.genIfThen(loc, isPresent)
+            .genThen([&]() { genDeallocateWithTypeDesc(); })
+            .end();
+      } else {
+        genDeallocateWithTypeDesc();
       }
     }
   }
diff --git a/flang/lib/Lower/HostAssociations.cpp b/flang/lib/Lower/HostAssociations.cpp
index 4d8a120172eff..008e4fc5c9a0e 100644
--- a/flang/lib/Lower/HostAssociations.cpp
+++ b/flang/lib/Lower/HostAssociations.cpp
@@ -243,7 +243,9 @@ class CapturedCharacterScalars
                                                args.loc);
     std::pair<mlir::Value, mlir::Value> unboxchar =
         charHelp.createUnboxChar(args.valueInTuple);
-    args.symMap.addCharSymbol(sym, unboxchar.first, unboxchar.second);
+    bindCapturedSymbol(sym,
+                       fir::CharBoxValue{unboxchar.first, unboxchar.second},
+                       converter, args.symMap);
   }
 };
 
diff --git a/flang/lib/Lower/IO.cpp b/flang/lib/Lower/IO.cpp
index 6f30da290a6d5..e5a84b43264f7 100644
--- a/flang/lib/Lower/IO.cpp
+++ b/flang/lib/Lower/IO.cpp
@@ -29,7 +29,7 @@
 #include "flang/Optimizer/Builder/Runtime/RTBuilder.h"
 #include "flang/Optimizer/Builder/Todo.h"
 #include "flang/Optimizer/Dialect/FIRDialect.h"
-#include "flang/Optimizer/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
 #include "flang/Parser/parse-tree.h"
 #include "flang/Runtime/io-api.h"
 #include "flang/Semantics/tools.h"
diff --git a/flang/lib/Lower/OpenMP.cpp b/flang/lib/Lower/OpenMP.cpp
index 8fcf0ba226a50..22e5ff322f6fb 100644
--- a/flang/lib/Lower/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP.cpp
@@ -61,7 +61,7 @@ getOmpObjectSymbol(const Fortran::parser::OmpObject &ompObject) {
   return sym;
 }
 
-class dataSharingProcessor {
+class DataSharingProcessor {
   bool hasLastPrivateOp;
   mlir::OpBuilder::InsertPoint lastPrivIP;
   mlir::OpBuilder::InsertPoint insPt;
@@ -92,7 +92,7 @@ class dataSharingProcessor {
   void insertLastPrivateCompare(mlir::Operation *op);
 
 public:
-  dataSharingProcessor(mlir::Operation *op,
+  DataSharingProcessor(mlir::Operation *op,
                        Fortran::lower::AbstractConverter &converter,
                        const Fortran::parser::OmpClauseList &opClauseList,
                        Fortran::lower::pft::Evaluation &eval)
@@ -102,7 +102,7 @@ class dataSharingProcessor {
   bool process();
 };
 
-bool dataSharingProcessor::process() {
+bool DataSharingProcessor::process() {
   insPt = firOpBuilder.saveInsertionPoint();
   collectSymbolsForPrivatization();
   insertLastPrivateCompare(op);
@@ -118,7 +118,7 @@ bool dataSharingProcessor::process() {
   return hasLastPrivateOp;
 }
 
-void dataSharingProcessor::privatizeSymbol(
+void DataSharingProcessor::privatizeSymbol(
     const Fortran::semantics::Symbol *sym,
     [[maybe_unused]] mlir::OpBuilder::InsertPoint *lastPrivIP) {
   // Privatization for symbols which are pre-determined (like loop index
@@ -144,7 +144,7 @@ void dataSharingProcessor::privatizeSymbol(
     converter.copyHostAssociateVar(*sym, lastPrivIP);
 }
 
-void dataSharingProcessor::collectOmpObjectListSymbol(
+void DataSharingProcessor::collectOmpObjectListSymbol(
     const Fortran::parser::OmpObjectList &ompObjectList,
     llvm::SetVector<const Fortran::semantics::Symbol *> &symbolSet) {
   for (const Fortran::parser::OmpObject &ompObject : ompObjectList.v) {
@@ -153,7 +153,7 @@ void dataSharingProcessor::collectOmpObjectListSymbol(
   }
 }
 
-void dataSharingProcessor::collectSymbolsForPrivatization() {
+void DataSharingProcessor::collectSymbolsForPrivatization() {
   bool hasCollapse = false;
   for (const Fortran::parser::OmpClause &clause : opClauseList.v) {
     if (const auto &privateClause =
@@ -177,7 +177,7 @@ void dataSharingProcessor::collectSymbolsForPrivatization() {
     TODO(converter.getCurrentLocation(), "Collapse clause with lastprivate");
 }
 
-bool dataSharingProcessor ::needBarrier() {
+bool DataSharingProcessor ::needBarrier() {
   for (auto sym : privatizedSymbols) {
     if (sym->test(Fortran::semantics::Symbol::Flag::OmpFirstPrivate) &&
         sym->test(Fortran::semantics::Symbol::Flag::OmpLastPrivate))
@@ -186,7 +186,7 @@ bool dataSharingProcessor ::needBarrier() {
   return false;
 }
 
-void dataSharingProcessor ::insertBarrier() {
+void DataSharingProcessor ::insertBarrier() {
   // Emit implicit barrier to synchronize threads and avoid data races on
   // initialization of firstprivate variables and post-update of lastprivate
   // variables.
@@ -198,7 +198,7 @@ void dataSharingProcessor ::insertBarrier() {
     firOpBuilder.create<mlir::omp::BarrierOp>(converter.getCurrentLocation());
 }
 
-void dataSharingProcessor::insertLastPrivateCompare(mlir::Operation *op) {
+void DataSharingProcessor::insertLastPrivateCompare(mlir::Operation *op) {
   mlir::arith::CmpIOp cmpOp;
   bool cmpCreated = false;
   for (const Fortran::parser::OmpClause &clause : opClauseList.v) {
@@ -302,7 +302,7 @@ void dataSharingProcessor::insertLastPrivateCompare(mlir::Operation *op) {
   }
 }
 
-void dataSharingProcessor::collectSymbols(
+void DataSharingProcessor::collectSymbols(
     Fortran::semantics::Symbol::Flag flag) {
   converter.collectSymbolSet(eval, defaultSymbols, flag,
                              /*collectSymbols=*/true,
@@ -319,7 +319,7 @@ void dataSharingProcessor::collectSymbols(
   }
 }
 
-void dataSharingProcessor::collectDefaultSymbols() {
+void DataSharingProcessor::collectDefaultSymbols() {
   for (const Fortran::parser::OmpClause &clause : opClauseList.v) {
     if (const auto &defaultClause =
             std::get_if<Fortran::parser::OmpClause::Default>(&clause.u)) {
@@ -333,12 +333,12 @@ void dataSharingProcessor::collectDefaultSymbols() {
   }
 }
 
-void dataSharingProcessor::privatize() {
+void DataSharingProcessor::privatize() {
   for (auto sym : privatizedSymbols)
     privatizeSymbol(sym, &lastPrivIP);
 }
 
-void dataSharingProcessor::defaultPrivatize() {
+void DataSharingProcessor::defaultPrivatize() {
   for (auto sym : defaultSymbols)
     if (!symbolsInNestedRegions.contains(sym) &&
         !symbolsInParentRegions.contains(sym) &&
@@ -645,7 +645,7 @@ createBodyOfOp(Op &op, Fortran::lower::AbstractConverter &converter,
 
   // Handle privatization. Do not privatize if this is the outer operation.
   if (clauses && !outerCombined) {
-    dataSharingProcessor dsp(op, converter, *clauses, eval);
+    DataSharingProcessor dsp(op, converter, *clauses, eval);
     bool lastPrivateOp = dsp.process();
     // LastPrivatization, due to introduction of
     // new control flow, changes the insertion point,
@@ -1112,7 +1112,8 @@ static int getOperationIdentity(llvm::StringRef reductionOpName,
                                 mlir::Location loc) {
   if (reductionOpName.contains("add"))
     return 0;
-  if (reductionOpName.contains("multiply") || reductionOpName.contains("and"))
+  if (reductionOpName.contains("multiply") || reductionOpName.contains("and") ||
+      reductionOpName.contains("eqv"))
     return 1;
   TODO(loc, "Reduction of some intrinsic operators is not supported");
 }
@@ -1120,17 +1121,38 @@ static int getOperationIdentity(llvm::StringRef reductionOpName,
 static Value getReductionInitValue(mlir::Location loc, mlir::Type type,
                                    llvm::StringRef reductionOpName,
                                    fir::FirOpBuilder &builder) {
-  assert(type.isIntOrIndexOrFloat() &&
-         "only integer and float types are currently supported");
-  if (type.isa<FloatType>())
+  assert((fir::isa_integer(type) || fir::isa_real(type) ||
+          type.isa<fir::LogicalType>()) &&
+         "only integer, logical and real types are currently supported");
+  if (reductionOpName.contains("max")) {
+    if (auto ty = type.dyn_cast<mlir::FloatType>()) {
+      const llvm::fltSemantics &sem = ty.getFloatSemantics();
+      return builder.createRealConstant(
+          loc, type, llvm::APFloat::getLargest(sem, /*Negative=*/true));
+    }
+    unsigned bits = type.getIntOrFloatBitWidth();
+    int64_t minInt = llvm::APInt::getSignedMinValue(bits).getSExtValue();
+    return builder.createIntegerConstant(loc, type, minInt);
+  } else {
+    if (type.isa<FloatType>())
+      return builder.create<mlir::arith::ConstantOp>(
+          loc, type,
+          builder.getFloatAttr(
+              type, (double)getOperationIdentity(reductionOpName, loc)));
+
+    if (type.isa<fir::LogicalType>()) {
+      Value intConst = builder.create<mlir::arith::ConstantOp>(
+          loc, builder.getI1Type(),
+          builder.getIntegerAttr(builder.getI1Type(),
+                                 getOperationIdentity(reductionOpName, loc)));
+      return builder.createConvert(loc, type, intConst);
+    }
+
     return builder.create<mlir::arith::ConstantOp>(
         loc, type,
-        builder.getFloatAttr(
-            type, (double)getOperationIdentity(reductionOpName, loc)));
-
-  return builder.create<mlir::arith::ConstantOp>(
-      loc, type,
-      builder.getIntegerAttr(type, getOperationIdentity(reductionOpName, loc)));
+        builder.getIntegerAttr(type,
+                               getOperationIdentity(reductionOpName, loc)));
+  }
 }
 
 template <typename FloatOp, typename IntegerOp>
@@ -1144,6 +1166,65 @@ static Value getReductionOperation(fir::FirOpBuilder &builder, mlir::Type type,
   return builder.create<FloatOp>(loc, op1, op2);
 }
 
+static omp::ReductionDeclareOp
+createMinimalReductionDecl(fir::FirOpBuilder &builder,
+                           llvm::StringRef reductionOpName, mlir::Type type,
+                           mlir::Location loc) {
+  mlir::ModuleOp module = builder.getModule();
+  mlir::OpBuilder modBuilder(module.getBodyRegion());
+
+  mlir::omp::ReductionDeclareOp decl =
+      modBuilder.create<omp::ReductionDeclareOp>(loc, reductionOpName, type);
+  builder.createBlock(&decl.getInitializerRegion(),
+                      decl.getInitializerRegion().end(), {type}, {loc});
+  builder.setInsertionPointToEnd(&decl.getInitializerRegion().back());
+  Value init = getReductionInitValue(loc, type, reductionOpName, builder);
+  builder.create<omp::YieldOp>(loc, init);
+
+  builder.createBlock(&decl.getReductionRegion(),
+                      decl.getReductionRegion().end(), {type, type},
+                      {loc, loc});
+
+  return decl;
+}
+
+/// Creates an OpenMP reduction declaration and inserts it into the provided
+/// symbol table. The declaration has a constant initializer with the neutral
+/// value `initValue`, and the reduction combiner carried over from `reduce`.
+/// TODO: Generalize this for non-integer types, add atomic region.
+static omp::ReductionDeclareOp
+createReductionDecl(fir::FirOpBuilder &builder, llvm::StringRef reductionOpName,
+                    const Fortran::parser::ProcedureDesignator &procDesignator,
+                    mlir::Type type, mlir::Location loc) {
+  OpBuilder::InsertionGuard guard(builder);
+  mlir::ModuleOp module = builder.getModule();
+
+  auto decl =
+      module.lookupSymbol<mlir::omp::ReductionDeclareOp>(reductionOpName);
+  if (decl)
+    return decl;
+
+  decl = createMinimalReductionDecl(builder, reductionOpName, type, loc);
+  builder.setInsertionPointToEnd(&decl.getReductionRegion().back());
+  mlir::Value op1 = decl.getReductionRegion().front().getArgument(0);
+  mlir::Value op2 = decl.getReductionRegion().front().getArgument(1);
+
+  Value reductionOp;
+  if (const auto *name{
+          Fortran::parser::Unwrap<Fortran::parser::Name>(procDesignator)}) {
+    if (name->source == "max") {
+      reductionOp =
+          getReductionOperation<mlir::arith::MaxFOp, mlir::arith::MaxSIOp>(
+              builder, type, loc, op1, op2);
+    } else {
+      TODO(loc, "Reduction of some intrinsic operators is not supported");
+    }
+  }
+
+  builder.create<omp::YieldOp>(loc, reductionOp);
+  return decl;
+}
+
 /// Creates an OpenMP reduction declaration and inserts it into the provided
 /// symbol table. The declaration has a constant initializer with the neutral
 /// value `initValue`, and the reduction combiner carried over from `reduce`.
@@ -1154,23 +1235,13 @@ static omp::ReductionDeclareOp createReductionDecl(
     mlir::Type type, mlir::Location loc) {
   OpBuilder::InsertionGuard guard(builder);
   mlir::ModuleOp module = builder.getModule();
-  mlir::OpBuilder modBuilder(module.getBodyRegion());
+
   auto decl =
       module.lookupSymbol<mlir::omp::ReductionDeclareOp>(reductionOpName);
-  if (!decl)
-    decl =
-        modBuilder.create<omp::ReductionDeclareOp>(loc, reductionOpName, type);
-  else
+  if (decl)
     return decl;
-  builder.createBlock(&decl.getInitializerRegion(),
-                      decl.getInitializerRegion().end(), {type}, {loc});
-  builder.setInsertionPointToEnd(&decl.getInitializerRegion().back());
-  Value init = getReductionInitValue(loc, type, reductionOpName, builder);
-  builder.create<omp::YieldOp>(loc, init);
 
-  builder.createBlock(&decl.getReductionRegion(),
-                      decl.getReductionRegion().end(), {type, type},
-                      {loc, loc});
+  decl = createMinimalReductionDecl(builder, reductionOpName, type, loc);
   builder.setInsertionPointToEnd(&decl.getReductionRegion().back());
   mlir::Value op1 = decl.getReductionRegion().front().getArgument(0);
   mlir::Value op2 = decl.getReductionRegion().front().getArgument(1);
@@ -1187,9 +1258,25 @@ static omp::ReductionDeclareOp createReductionDecl(
         getReductionOperation<mlir::arith::MulFOp, mlir::arith::MulIOp>(
             builder, type, loc, op1, op2);
     break;
-  case Fortran::parser::DefinedOperator::IntrinsicOperator::AND:
-    reductionOp = builder.create<mlir::arith::AndIOp>(loc, op1, op2);
+  case Fortran::parser::DefinedOperator::IntrinsicOperator::AND: {
+    Value op1I1 = builder.createConvert(loc, builder.getI1Type(), op1);
+    Value op2I1 = builder.createConvert(loc, builder.getI1Type(), op2);
+
+    Value andiOp = builder.create<mlir::arith::AndIOp>(loc, op1I1, op2I1);
+
+    reductionOp = builder.createConvert(loc, type, andiOp);
     break;
+  }
+  case Fortran::parser::DefinedOperator::IntrinsicOperator::EQV: {
+    Value op1I1 = builder.createConvert(loc, builder.getI1Type(), op1);
+    Value op2I1 = builder.createConvert(loc, builder.getI1Type(), op2);
+
+    Value cmpiOp = builder.create<mlir::arith::CmpIOp>(
+        loc, arith::CmpIPredicate::eq, op1I1, op2I1);
+
+    reductionOp = builder.createConvert(loc, type, cmpiOp);
+    break;
+  }
   default:
     TODO(loc, "Reduction of some intrinsic operators is not supported");
   }
@@ -1262,6 +1349,13 @@ getSIMDModifier(const Fortran::parser::OmpScheduleClause &x) {
   return mlir::omp::ScheduleModifier::none;
 }
 
+static std::string getReductionName(llvm::StringRef name, mlir::Type ty) {
+  return (llvm::Twine(name) +
+          (ty.isIntOrIndex() ? llvm::Twine("_i_") : llvm::Twine("_f_")) +
+          llvm::Twine(ty.getIntOrFloatBitWidth()))
+      .str();
+}
+
 static std::string getReductionName(
     Fortran::parser::DefinedOperator::IntrinsicOperator intrinsicOp,
     mlir::Type ty) {
@@ -1276,15 +1370,14 @@ static std::string getReductionName(
     break;
   case Fortran::parser::DefinedOperator::IntrinsicOperator::AND:
     return "and_reduction";
+  case Fortran::parser::DefinedOperator::IntrinsicOperator::EQV:
+    return "eqv_reduction";
   default:
     reductionName = "other_reduction";
     break;
   }
 
-  return (llvm::Twine(reductionName) +
-          (ty.isIntOrIndex() ? llvm::Twine("_i_") : llvm::Twine("_f_")) +
-          llvm::Twine(ty.getIntOrFloatBitWidth()))
-      .str();
+  return getReductionName(reductionName, ty);
 }
 
 static void genOMP(Fortran::lower::AbstractConverter &converter,
@@ -1385,6 +1478,7 @@ static void genOMP(Fortran::lower::AbstractConverter &converter,
         case Fortran::parser::DefinedOperator::IntrinsicOperator::Add:
         case Fortran::parser::DefinedOperator::IntrinsicOperator::Multiply:
         case Fortran::parser::DefinedOperator::IntrinsicOperator::AND:
+        case Fortran::parser::DefinedOperator::IntrinsicOperator::EQV:
           break;
 
         default:
@@ -1401,8 +1495,11 @@ static void genOMP(Fortran::lower::AbstractConverter &converter,
                   symVal.getType().cast<fir::ReferenceType>().getEleTy();
               reductionVars.push_back(symVal);
               if (redType.isa<fir::LogicalType>())
-                redType = firOpBuilder.getI1Type();
-              if (redType.isIntOrIndexOrFloat()) {
+                decl = createReductionDecl(
+                    firOpBuilder,
+                    getReductionName(intrinsicOp, firOpBuilder.getI1Type()),
+                    intrinsicOp, redType, currentLocation);
+              else if (redType.isIntOrIndexOrFloat()) {
                 decl = createReductionDecl(
                     firOpBuilder, getReductionName(intrinsicOp, redType),
                     intrinsicOp, redType, currentLocation);
@@ -1415,9 +1512,34 @@ static void genOMP(Fortran::lower::AbstractConverter &converter,
             }
           }
         }
-      } else {
-        TODO(currentLocation,
-             "Reduction of intrinsic procedures is not supported");
+      } else if (auto reductionIntrinsic =
+                     std::get_if<Fortran::parser::ProcedureDesignator>(
+                         &redOperator.u)) {
+        if (const auto *name{Fortran::parser::Unwrap<Fortran::parser::Name>(
+                reductionIntrinsic)}) {
+          if (name->source != "max") {
+            TODO(currentLocation,
+                 "Reduction of intrinsic procedures is not supported");
+          }
+          for (const auto &ompObject : objectList.v) {
+            if (const auto *name{Fortran::parser::Unwrap<Fortran::parser::Name>(
+                    ompObject)}) {
+              if (const auto *symbol{name->symbol}) {
+                mlir::Value symVal = converter.getSymbolAddress(*symbol);
+                mlir::Type redType =
+                    symVal.getType().cast<fir::ReferenceType>().getEleTy();
+                reductionVars.push_back(symVal);
+                assert(redType.isIntOrIndexOrFloat() &&
+                       "Unsupported reduction type");
+                decl = createReductionDecl(
+                    firOpBuilder, getReductionName("max", redType),
+                    *reductionIntrinsic, redType, currentLocation);
+                reductionDeclSymbols.push_back(SymbolRefAttr::get(
+                    firOpBuilder.getContext(), decl.getSymName()));
+              }
+            }
+          }
+        }
       }
     } else if (const auto &simdlenClause =
                    std::get_if<Fortran::parser::OmpClause::Simdlen>(
@@ -2076,6 +2198,21 @@ void Fortran::lower::genOpenMPDeclarativeConstruct(
       ompDeclConstruct.u);
 }
 
+static mlir::Operation *getCompareFromReductionOp(mlir::Operation *reductionOp,
+                                                  mlir::Value loadVal) {
+  for (auto reductionOperand : reductionOp->getOperands()) {
+    if (auto compareOp = reductionOperand.getDefiningOp()) {
+      if (compareOp->getOperand(0) == loadVal ||
+          compareOp->getOperand(1) == loadVal)
+        assert((mlir::isa<mlir::arith::CmpIOp>(compareOp) ||
+                mlir::isa<mlir::arith::CmpFOp>(compareOp)) &&
+               "Expected comparison not found in reduction intrinsic");
+      return compareOp;
+    }
+  }
+  return nullptr;
+}
+
 // Generate an OpenMP reduction operation.
 // TODO: Currently assumes it is either an integer addition/multiplication
 // reduction, or a logical and reduction. Generalize this for various reduction
@@ -2105,6 +2242,7 @@ void Fortran::lower::genOpenMPReduction(
         case Fortran::parser::DefinedOperator::IntrinsicOperator::Add:
         case Fortran::parser::DefinedOperator::IntrinsicOperator::Multiply:
         case Fortran::parser::DefinedOperator::IntrinsicOperator::AND:
+        case Fortran::parser::DefinedOperator::IntrinsicOperator::EQV:
           break;
         default:
           continue;
@@ -2116,9 +2254,7 @@ void Fortran::lower::genOpenMPReduction(
               mlir::Value reductionVal = converter.getSymbolAddress(*symbol);
               mlir::Type reductionType =
                   reductionVal.getType().cast<fir::ReferenceType>().getEleTy();
-
-              if (intrinsicOp !=
-                  Fortran::parser::DefinedOperator::IntrinsicOperator::AND) {
+              if (!reductionType.isa<fir::LogicalType>()) {
                 if (!reductionType.isIntOrIndexOrFloat())
                   continue;
               }
@@ -2126,8 +2262,7 @@ void Fortran::lower::genOpenMPReduction(
                 if (auto loadOp = mlir::dyn_cast<fir::LoadOp>(
                         reductionValUse.getOwner())) {
                   mlir::Value loadVal = loadOp.getRes();
-                  if (intrinsicOp == Fortran::parser::DefinedOperator::
-                                         IntrinsicOperator::AND) {
+                  if (reductionType.isa<fir::LogicalType>()) {
                     mlir::Operation *reductionOp = findReductionChain(loadVal);
                     fir::ConvertOp convertOp =
                         getConvertFromReductionOp(reductionOp, loadVal);
@@ -2144,6 +2279,40 @@ void Fortran::lower::genOpenMPReduction(
             }
           }
         }
+      } else if (auto reductionIntrinsic =
+                     std::get_if<Fortran::parser::ProcedureDesignator>(
+                         &redOperator.u)) {
+        if (const auto *name{Fortran::parser::Unwrap<Fortran::parser::Name>(
+                reductionIntrinsic)}) {
+          if (name->source != "max") {
+            continue;
+          }
+          for (const auto &ompObject : objectList.v) {
+            if (const auto *name{Fortran::parser::Unwrap<Fortran::parser::Name>(
+                    ompObject)}) {
+              if (const auto *symbol{name->symbol}) {
+                mlir::Value reductionVal = converter.getSymbolAddress(*symbol);
+                for (mlir::OpOperand &reductionValUse :
+                     reductionVal.getUses()) {
+                  if (auto loadOp = mlir::dyn_cast<fir::LoadOp>(
+                          reductionValUse.getOwner())) {
+                    mlir::Value loadVal = loadOp.getRes();
+                    // Max is lowered as a compare -> select.
+                    // Match the pattern here.
+                    mlir::Operation *reductionOp =
+                        findReductionChain(loadVal, &reductionVal);
+                    assert(mlir::isa<mlir::arith::SelectOp>(reductionOp) &&
+                           "Selection Op not found in reduction intrinsic");
+                    mlir::Operation *compareOp =
+                        getCompareFromReductionOp(reductionOp, loadVal);
+                    updateReduction(compareOp, firOpBuilder, loadVal,
+                                    reductionVal);
+                  }
+                }
+              }
+            }
+          }
+        }
       }
     }
   }
diff --git a/flang/lib/Lower/PFTBuilder.cpp b/flang/lib/Lower/PFTBuilder.cpp
index 98f02620bedde..6261a9093cbc3 100644
--- a/flang/lib/Lower/PFTBuilder.cpp
+++ b/flang/lib/Lower/PFTBuilder.cpp
@@ -1394,7 +1394,7 @@ struct SymbolDependenceAnalysis {
   void analyzeLocalEquivalenceSets(const semantics::Scope &scope) {
     if (scope.equivalenceSets().empty())
       return; // no equivalence sets to analyze
-    if (analyzedScopes.find(&scope) != analyzedScopes.end())
+    if (analyzedScopes.contains(&scope))
       return; // equivalence sets already analyzed
 
     analyzedScopes.insert(&scope);
@@ -1542,7 +1542,7 @@ struct SymbolDependenceAnalysis {
     const semantics::Scope &scope = ultimate.owner();
     // Expect the total number of EQUIVALENCE sets to be small for a typical
     // Fortran program.
-    if (aliasSyms.find(&ultimate) != aliasSyms.end()) {
+    if (aliasSyms.contains(&ultimate)) {
       LLVM_DEBUG(llvm::dbgs() << "found aggregate containing " << &ultimate
                               << " " << ultimate.name() << " in <" << &scope
                               << "> " << scope.GetName() << '\n');
diff --git a/flang/lib/Optimizer/Analysis/CMakeLists.txt b/flang/lib/Optimizer/Analysis/CMakeLists.txt
index 4ebe7d8c78c33..a8a3c899f35de 100644
--- a/flang/lib/Optimizer/Analysis/CMakeLists.txt
+++ b/flang/lib/Optimizer/Analysis/CMakeLists.txt
@@ -2,9 +2,8 @@ add_flang_library(FIRAnalysis
   AliasAnalysis.cpp
 
   DEPENDS
-  FIRBuilder
   FIRDialect
-  FIRSupport
+  MLIRIR
 
   LINK_LIBS
   FIRBuilder
diff --git a/flang/lib/Optimizer/Builder/CMakeLists.txt b/flang/lib/Optimizer/Builder/CMakeLists.txt
index 5457a9e3f636a..c93d4b9465148 100644
--- a/flang/lib/Optimizer/Builder/CMakeLists.txt
+++ b/flang/lib/Optimizer/Builder/CMakeLists.txt
@@ -27,12 +27,12 @@ add_flang_library(FIRBuilder
 
   DEPENDS
   FIRDialect
-  FIRSupport
   HLFIRDialect
   ${dialect_libs}
 
   LINK_LIBS
   FIRDialect
+  FIRDialectSupport
   FIRSupport
   HLFIRDialect
   ${dialect_libs}
diff --git a/flang/lib/Optimizer/Builder/FIRBuilder.cpp b/flang/lib/Optimizer/Builder/FIRBuilder.cpp
index 21f36a7e152cd..244f972ca5a34 100644
--- a/flang/lib/Optimizer/Builder/FIRBuilder.cpp
+++ b/flang/lib/Optimizer/Builder/FIRBuilder.cpp
@@ -352,11 +352,19 @@ fir::FirOpBuilder::convertWithSemantics(mlir::Location loc, mlir::Type toTy,
             "element types expected to match"));
     return create<fir::BoxAddrOp>(loc, toTy, val);
   }
+  if (fir::isa_ref_type(fromTy) && toTy.isa<fir::BoxProcType>()) {
+    // Call is expecting a boxed procedure, not a reference to other data type.
+    // Convert the reference to a procedure and embox it.
+    mlir::Type procTy = toTy.cast<fir::BoxProcType>().getEleTy();
+    mlir::Value proc = createConvert(loc, procTy, val);
+    return create<fir::EmboxProcOp>(loc, toTy, proc);
+  }
 
-  if ((fir::isPolymorphicType(fromTy) &&
-       (fir::isAllocatableType(fromTy) || fir::isPointerType(fromTy)) &&
-       fir::isPolymorphicType(toTy)) ||
-      (fir::isPolymorphicType(fromTy) && toTy.isa<fir::BoxType>()))
+  if (((fir::isPolymorphicType(fromTy) &&
+        (fir::isAllocatableType(fromTy) || fir::isPointerType(fromTy)) &&
+        fir::isPolymorphicType(toTy)) ||
+       (fir::isPolymorphicType(fromTy) && toTy.isa<fir::BoxType>())) &&
+      !(fir::isUnlimitedPolymorphicType(fromTy) && fir::isAssumedType(toTy)))
     return create<fir::ReboxOp>(loc, toTy, val, mlir::Value{},
                                 /*slice=*/mlir::Value{});
 
@@ -499,7 +507,8 @@ mlir::Value fir::FirOpBuilder::createSlice(mlir::Location loc,
 
 mlir::Value fir::FirOpBuilder::createBox(mlir::Location loc,
                                          const fir::ExtendedValue &exv,
-                                         bool isPolymorphic) {
+                                         bool isPolymorphic,
+                                         bool isAssumedType) {
   mlir::Value itemAddr = fir::getBase(exv);
   if (itemAddr.getType().isa<fir::BaseBoxType>())
     return itemAddr;
@@ -509,11 +518,20 @@ mlir::Value fir::FirOpBuilder::createBox(mlir::Location loc,
         << itemAddr.getType();
     llvm_unreachable("not a memory reference type");
   }
-  mlir::Type boxTy = fir::BoxType::get(elementType);
+  mlir::Type boxTy;
   mlir::Value tdesc;
-  if (isPolymorphic) {
-    elementType = fir::updateTypeForUnlimitedPolymorphic(elementType);
-    boxTy = fir::ClassType::get(elementType);
+  // Avoid to wrap a box/class with box/class.
+  if (elementType.isa<fir::BaseBoxType>()) {
+    boxTy = elementType;
+  } else {
+    boxTy = fir::BoxType::get(elementType);
+    if (isPolymorphic) {
+      elementType = fir::updateTypeForUnlimitedPolymorphic(elementType);
+      if (isAssumedType)
+        boxTy = fir::BoxType::get(elementType);
+      else
+        boxTy = fir::ClassType::get(elementType);
+    }
   }
 
   return exv.match(
@@ -614,6 +632,18 @@ mlir::Value fir::FirOpBuilder::genExtentFromTriplet(mlir::Location loc,
   return create<mlir::arith::SelectOp>(loc, cmp, div, zero);
 }
 
+mlir::Value fir::FirOpBuilder::genAbsentOp(mlir::Location loc,
+                                           mlir::Type argTy) {
+  if (!fir::isCharacterProcedureTuple(argTy))
+    return create<fir::AbsentOp>(loc, argTy);
+
+  auto boxProc =
+      create<fir::AbsentOp>(loc, argTy.cast<mlir::TupleType>().getType(0));
+  mlir::Value charLen = create<fir::UndefOp>(loc, getCharacterLengthType());
+  return fir::factory::createCharacterProcedureTuple(*this, loc, argTy, boxProc,
+                                                     charLen);
+}
+
 void fir::FirOpBuilder::setCommonAttributes(mlir::Operation *op) const {
   auto fmi = mlir::dyn_cast<mlir::arith::ArithFastMathInterface>(*op);
   if (!fmi)
diff --git a/flang/lib/Optimizer/Builder/IntrinsicCall.cpp b/flang/lib/Optimizer/Builder/IntrinsicCall.cpp
index 14081b7e9a6c3..b933603484581 100644
--- a/flang/lib/Optimizer/Builder/IntrinsicCall.cpp
+++ b/flang/lib/Optimizer/Builder/IntrinsicCall.cpp
@@ -33,15 +33,17 @@
 #include "flang/Optimizer/Builder/Runtime/Transformational.h"
 #include "flang/Optimizer/Builder/Todo.h"
 #include "flang/Optimizer/Dialect/FIROpsSupport.h"
-#include "flang/Optimizer/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
 #include "flang/Optimizer/Support/FatalError.h"
 #include "flang/Optimizer/Support/Utils.h"
 #include "flang/Runtime/entry-names.h"
+#include "flang/Runtime/iostat.h"
 #include "mlir/Dialect/Complex/IR/Complex.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/Dialect/Math/IR/Math.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
+#include "llvm/Support/MathExtras.h"
 #include <optional>
 
 #define DEBUG_TYPE "flang-lower-intrinsic"
@@ -179,6 +181,7 @@ struct IntrinsicLibrary {
                                   llvm::ArrayRef<fir::ExtendedValue>);
   mlir::Value genAnint(mlir::Type, llvm::ArrayRef<mlir::Value>);
   fir::ExtendedValue genAny(mlir::Type, llvm::ArrayRef<fir::ExtendedValue>);
+  mlir::Value genAtand(mlir::Type, llvm::ArrayRef<mlir::Value>);
   fir::ExtendedValue
       genCommandArgumentCount(mlir::Type, llvm::ArrayRef<fir::ExtendedValue>);
   fir::ExtendedValue genAssociated(mlir::Type,
@@ -241,6 +244,7 @@ struct IntrinsicLibrary {
   fir::ExtendedValue genIchar(mlir::Type, llvm::ArrayRef<fir::ExtendedValue>);
   fir::ExtendedValue genFindloc(mlir::Type, llvm::ArrayRef<fir::ExtendedValue>);
   mlir::Value genIeeeIsFinite(mlir::Type, llvm::ArrayRef<mlir::Value>);
+  mlir::Value genIeeeIsNormal(mlir::Type, llvm::ArrayRef<mlir::Value>);
   template <mlir::arith::CmpIPredicate pred>
   fir::ExtendedValue genIeeeTypeCompare(mlir::Type,
                                         llvm::ArrayRef<fir::ExtendedValue>);
@@ -250,7 +254,11 @@ struct IntrinsicLibrary {
   fir::ExtendedValue genIparity(mlir::Type, llvm::ArrayRef<fir::ExtendedValue>);
   fir::ExtendedValue genIsContiguous(mlir::Type,
                                      llvm::ArrayRef<fir::ExtendedValue>);
+  template <Fortran::runtime::io::Iostat value>
+  mlir::Value genIsIostatValue(mlir::Type, llvm::ArrayRef<mlir::Value>);
   mlir::Value genIsNan(mlir::Type, llvm::ArrayRef<mlir::Value>);
+  mlir::Value genIsFPClass(mlir::Type, llvm::ArrayRef<mlir::Value>,
+                           int fpclass);
   mlir::Value genIshft(mlir::Type, llvm::ArrayRef<mlir::Value>);
   mlir::Value genIshftc(mlir::Type, llvm::ArrayRef<mlir::Value>);
   fir::ExtendedValue genLbound(mlir::Type, llvm::ArrayRef<fir::ExtendedValue>);
@@ -261,6 +269,8 @@ struct IntrinsicLibrary {
   template <typename Shift>
   mlir::Value genMask(mlir::Type, llvm::ArrayRef<mlir::Value>);
   fir::ExtendedValue genMatmul(mlir::Type, llvm::ArrayRef<fir::ExtendedValue>);
+  fir::ExtendedValue genMatmulTranspose(mlir::Type,
+                                        llvm::ArrayRef<fir::ExtendedValue>);
   fir::ExtendedValue genMaxloc(mlir::Type, llvm::ArrayRef<fir::ExtendedValue>);
   fir::ExtendedValue genMaxval(mlir::Type, llvm::ArrayRef<fir::ExtendedValue>);
   fir::ExtendedValue genMerge(mlir::Type, llvm::ArrayRef<fir::ExtendedValue>);
@@ -486,6 +496,7 @@ static constexpr IntrinsicHandler handlers[]{
      &I::genAssociated,
      {{{"pointer", asInquired}, {"target", asInquired}}},
      /*isElemental=*/false},
+    {"atand", &I::genAtand},
     {"bessel_jn",
      &I::genBesselJn,
      {{{"n1", asValue}, {"n2", asValue}, {"x", asValue}}},
@@ -623,6 +634,7 @@ static constexpr IntrinsicHandler handlers[]{
     {"ieee_class_ne", &I::genIeeeTypeCompare<mlir::arith::CmpIPredicate::ne>},
     {"ieee_is_finite", &I::genIeeeIsFinite},
     {"ieee_is_nan", &I::genIsNan},
+    {"ieee_is_normal", &I::genIeeeIsNormal},
     {"ieee_round_eq", &I::genIeeeTypeCompare<mlir::arith::CmpIPredicate::eq>},
     {"ieee_round_ne", &I::genIeeeTypeCompare<mlir::arith::CmpIPredicate::ne>},
     {"ieor", &I::genIeor},
@@ -643,6 +655,8 @@ static constexpr IntrinsicHandler handlers[]{
      &I::genIsContiguous,
      {{{"array", asBox}}},
      /*isElemental=*/false},
+    {"is_iostat_end", &I::genIsIostatValue<Fortran::runtime::io::IostatEnd>},
+    {"is_iostat_eor", &I::genIsIostatValue<Fortran::runtime::io::IostatEor>},
     {"ishft", &I::genIshft},
     {"ishftc", &I::genIshftc},
     {"isnan", &I::genIsNan},
@@ -667,6 +681,10 @@ static constexpr IntrinsicHandler handlers[]{
      &I::genMatmul,
      {{{"matrix_a", asAddr}, {"matrix_b", asAddr}}},
      /*isElemental=*/false},
+    {"matmul_transpose",
+     &I::genMatmulTranspose,
+     {{{"matrix_a", asAddr}, {"matrix_b", asAddr}}},
+     /*isElemental=*/false},
     {"max", &I::genExtremum<Extremum::Max, ExtremumBehavior::MinMaxss>},
     {"maxloc",
      &I::genMaxloc,
@@ -1075,13 +1093,54 @@ static mlir::Value genLibCall(fir::FirOpBuilder &builder, mlir::Location loc,
   LLVM_DEBUG(llvm::dbgs() << "Generating '" << libFuncName
                           << "' call with type ";
              libFuncType.dump(); llvm::dbgs() << "\n");
-  mlir::func::FuncOp funcOp =
-      builder.addNamedFunction(loc, libFuncName, libFuncType);
-  // TODO: ensure 'strictfp' setting on the call for "precise/strict"
-  //       FP mode. Set appropriate Fast-Math Flags otherwise.
-  // TODO: we should also mark as many libm function as possible
-  //       with 'pure' attribute (of course, not in strict FP mode).
-  auto libCall = builder.create<fir::CallOp>(loc, funcOp, args);
+  mlir::func::FuncOp funcOp = builder.getNamedFunction(libFuncName);
+
+  if (!funcOp) {
+    funcOp = builder.addNamedFunction(loc, libFuncName, libFuncType);
+    // C-interoperability rules apply to these library functions.
+    funcOp->setAttr(fir::getSymbolAttrName(),
+                    mlir::StringAttr::get(builder.getContext(), libFuncName));
+    // Set fir.runtime attribute to distinguish the function that
+    // was just created from user functions with the same name.
+    funcOp->setAttr(fir::FIROpsDialect::getFirRuntimeAttrName(),
+                    builder.getUnitAttr());
+    auto libCall = builder.create<fir::CallOp>(loc, funcOp, args);
+    // TODO: ensure 'strictfp' setting on the call for "precise/strict"
+    //       FP mode. Set appropriate Fast-Math Flags otherwise.
+    // TODO: we should also mark as many libm function as possible
+    //       with 'pure' attribute (of course, not in strict FP mode).
+    LLVM_DEBUG(libCall.dump(); llvm::dbgs() << "\n");
+    return libCall.getResult(0);
+  }
+
+  // The function with the same name already exists.
+  fir::CallOp libCall;
+  mlir::Type soughtFuncType = funcOp.getFunctionType();
+
+  if (soughtFuncType == libFuncType) {
+    libCall = builder.create<fir::CallOp>(loc, funcOp, args);
+  } else {
+    // A function with the same name might have been declared
+    // before (e.g. with an explicit interface and a binding label).
+    // It is in general incorrect to use the same definition for the library
+    // call, but we have no other options. Type cast the function to match
+    // the requested signature and generate an indirect call to avoid
+    // later failures caused by the signature mismatch.
+    LLVM_DEBUG(mlir::emitWarning(
+        loc, llvm::Twine("function signature mismatch for '") +
+                 llvm::Twine(libFuncName) +
+                 llvm::Twine("' may lead to undefined behavior.")));
+    mlir::SymbolRefAttr funcSymbolAttr = builder.getSymbolRefAttr(libFuncName);
+    mlir::Value funcPointer =
+        builder.create<fir::AddrOfOp>(loc, soughtFuncType, funcSymbolAttr);
+    funcPointer = builder.createConvert(loc, libFuncType, funcPointer);
+
+    llvm::SmallVector<mlir::Value, 3> operands{funcPointer};
+    operands.append(args.begin(), args.end());
+    libCall = builder.create<fir::CallOp>(loc, libFuncType.getResults(),
+                                          nullptr, operands);
+  }
+
   LLVM_DEBUG(libCall.dump(); llvm::dbgs() << "\n");
   return libCall.getResult(0);
 }
@@ -1344,6 +1403,16 @@ static constexpr MathOperation mathOperations[] = {
 };
 
 static constexpr MathOperation ppcMathOperations[] = {
+    // fcfi is just another name for fcfid, there is no llvm.ppc.fcfi.
+    {"__ppc_fcfi", "llvm.ppc.fcfid", genF64F64FuncType, genLibCall},
+    {"__ppc_fcfid", "llvm.ppc.fcfid", genF64F64FuncType, genLibCall},
+    {"__ppc_fcfud", "llvm.ppc.fcfud", genF64F64FuncType, genLibCall},
+    {"__ppc_fctid", "llvm.ppc.fctid", genF64F64FuncType, genLibCall},
+    {"__ppc_fctidz", "llvm.ppc.fctidz", genF64F64FuncType, genLibCall},
+    {"__ppc_fctiw", "llvm.ppc.fctiw", genF64F64FuncType, genLibCall},
+    {"__ppc_fctiwz", "llvm.ppc.fctiwz", genF64F64FuncType, genLibCall},
+    {"__ppc_fctudz", "llvm.ppc.fctudz", genF64F64FuncType, genLibCall},
+    {"__ppc_fctuwz", "llvm.ppc.fctuwz", genF64F64FuncType, genLibCall},
     {"__ppc_fmadd", "llvm.fma.f32", genF32F32F32F32FuncType,
      genMathOp<mlir::math::FmaOp>},
     {"__ppc_fmadd", "llvm.fma.f64", genF64F64F64F64FuncType,
@@ -2316,6 +2385,20 @@ IntrinsicLibrary::genAny(mlir::Type resultType,
   return readAndAddCleanUp(resultMutableBox, resultType, "ANY");
 }
 
+mlir::Value IntrinsicLibrary::genAtand(mlir::Type resultType,
+                                       llvm::ArrayRef<mlir::Value> args) {
+  assert(args.size() == 1);
+  mlir::MLIRContext *context = builder.getContext();
+  mlir::FunctionType ftype =
+      mlir::FunctionType::get(context, {resultType}, {args[0].getType()});
+  mlir::Value atan = getRuntimeCallGenerator("atan", ftype)(builder, loc, args);
+  llvm::APFloat pi = llvm::APFloat(llvm::numbers::pi);
+  mlir::Value dfactor = builder.createRealConstant(
+      loc, mlir::FloatType::getF64(context), llvm::APFloat(180.0) / pi);
+  mlir::Value factor = builder.createConvert(loc, resultType, dfactor);
+  return builder.create<mlir::arith::MulFOp>(loc, atan, factor);
+}
+
 // ASSOCIATED
 fir::ExtendedValue
 IntrinsicLibrary::genAssociated(mlir::Type resultType,
@@ -3037,8 +3120,8 @@ IntrinsicLibrary::genEoshift(mlir::Type resultType,
 
   // Create mutable fir.box to be passed to the runtime for the result.
   mlir::Type resultArrayType = builder.getVarLenSeqTy(resultType, arrayRank);
-  fir::MutableBoxValue resultMutableBox =
-      fir::factory::createTempMutableBox(builder, loc, resultArrayType, {},
+  fir::MutableBoxValue resultMutableBox = fir::factory::createTempMutableBox(
+      builder, loc, resultArrayType, {},
       fir::isPolymorphicType(array.getType()) ? array : mlir::Value{});
   mlir::Value resultIrBox =
       fir::factory::getMutableIRBox(builder, loc, resultMutableBox);
@@ -3603,6 +3686,13 @@ IntrinsicLibrary::genIeeeIsFinite(mlir::Type resultType,
                                           exponent, maxExponent));
 }
 
+mlir::Value
+IntrinsicLibrary::genIeeeIsNormal(mlir::Type resultType,
+                                  llvm::ArrayRef<mlir::Value> args) {
+  // Check if is positive or negative normal
+  return genIsFPClass(resultType, args, 0b101101000);
+}
+
 // IEOR
 mlir::Value IntrinsicLibrary::genIeor(mlir::Type resultType,
                                       llvm::ArrayRef<mlir::Value> args) {
@@ -3693,18 +3783,35 @@ IntrinsicLibrary::genIsContiguous(mlir::Type resultType,
       fir::runtime::genIsContiguous(builder, loc, fir::getBase(args[0])));
 }
 
-mlir::Value IntrinsicLibrary::genIsNan(mlir::Type resultType,
-                                       llvm::ArrayRef<mlir::Value> args) {
+// IS_IOSTAT_END, IS_IOSTAT_EOR
+template <Fortran::runtime::io::Iostat value>
+mlir::Value
+IntrinsicLibrary::genIsIostatValue(mlir::Type resultType,
+                                   llvm::ArrayRef<mlir::Value> args) {
+  assert(args.size() == 1);
+  return builder.create<mlir::arith::CmpIOp>(
+      loc, mlir::arith::CmpIPredicate::eq, args[0],
+      builder.createIntegerConstant(loc, args[0].getType(), value));
+}
+
+mlir::Value IntrinsicLibrary::genIsFPClass(mlir::Type resultType,
+                                           llvm::ArrayRef<mlir::Value> args,
+                                           int fpclass) {
   assert(args.size() == 1);
   mlir::MLIRContext *context = builder.getContext();
   mlir::IntegerType i1ty = mlir::IntegerType::get(context, 1);
   mlir::IntegerType i32ty = mlir::IntegerType::get(context, 32);
-  // The last two bits indicate we are checking for signalling or quiet nan.
-  mlir::Value nan = builder.createIntegerConstant(loc, i32ty, 0b11);
 
-  mlir::Value isnan =
-      builder.create<mlir::LLVM::IsFPClass>(loc, i1ty, args[0], nan);
-  return builder.createConvert(loc, resultType, isnan);
+  mlir::Value test = builder.createIntegerConstant(loc, i32ty, fpclass);
+  mlir::Value isfpclass =
+      builder.create<mlir::LLVM::IsFPClass>(loc, i1ty, args[0], test);
+  return builder.createConvert(loc, resultType, isfpclass);
+}
+
+mlir::Value IntrinsicLibrary::genIsNan(mlir::Type resultType,
+                                       llvm::ArrayRef<mlir::Value> args) {
+  // Check is signaling or quiet nan
+  return genIsFPClass(resultType, args, 0b11);
 }
 
 // ISHFT
@@ -3914,6 +4021,33 @@ IntrinsicLibrary::genMatmul(mlir::Type resultType,
   return readAndAddCleanUp(resultMutableBox, resultType, "MATMUL");
 }
 
+// MATMUL_TRANSPOSE
+fir::ExtendedValue
+IntrinsicLibrary::genMatmulTranspose(mlir::Type resultType,
+                                     llvm::ArrayRef<fir::ExtendedValue> args) {
+  assert(args.size() == 2);
+
+  // Handle required matmul_transpose arguments
+  fir::BoxValue matrixTmpA = builder.createBox(loc, args[0]);
+  mlir::Value matrixA = fir::getBase(matrixTmpA);
+  fir::BoxValue matrixTmpB = builder.createBox(loc, args[1]);
+  mlir::Value matrixB = fir::getBase(matrixTmpB);
+  unsigned resultRank =
+      (matrixTmpA.rank() == 1 || matrixTmpB.rank() == 1) ? 1 : 2;
+
+  // Create mutable fir.box to be passed to the runtime for the result.
+  mlir::Type resultArrayType = builder.getVarLenSeqTy(resultType, resultRank);
+  fir::MutableBoxValue resultMutableBox =
+      fir::factory::createTempMutableBox(builder, loc, resultArrayType);
+  mlir::Value resultIrBox =
+      fir::factory::getMutableIRBox(builder, loc, resultMutableBox);
+  // Call runtime. The runtime is allocating the result.
+  fir::runtime::genMatmulTranspose(builder, loc, resultIrBox, matrixA, matrixB);
+  // Read result from mutable fir.box and add it to the list of temps to be
+  // finalized by the StatementContext.
+  return readAndAddCleanUp(resultMutableBox, resultType, "MATMUL_TRANSPOSE");
+}
+
 // MERGE
 fir::ExtendedValue
 IntrinsicLibrary::genMerge(mlir::Type,
@@ -3934,10 +4068,18 @@ IntrinsicLibrary::genMerge(mlir::Type,
   mlir::Value fsourceCast = fsource;
   if (fir::isPolymorphicType(tsource.getType()) &&
       !fir::isPolymorphicType(fsource.getType())) {
-    tsourceCast = builder.createConvert(loc, fsource.getType(), tsource);
+    tsourceCast = builder.create<fir::ReboxOp>(loc, fsource.getType(), tsource,
+                                               /*shape*/ mlir::Value{},
+                                               /*slice=*/mlir::Value{});
+
+    // builder.createConvert(loc, fsource.getType(), tsource);
   } else if (!fir::isPolymorphicType(tsource.getType()) &&
              fir::isPolymorphicType(fsource.getType())) {
-    fsourceCast = builder.createConvert(loc, tsource.getType(), fsource);
+    fsourceCast = builder.create<fir::ReboxOp>(loc, tsource.getType(), fsource,
+                                               /*shape*/ mlir::Value{},
+                                               /*slice=*/mlir::Value{});
+
+    // fsourceCast = builder.createConvert(loc, tsource.getType(), fsource);
   } else {
     // FSOURCE and TSOURCE are not polymorphic.
     // FSOURCE has the same type as TSOURCE, but they may not have the same MLIR
@@ -5000,8 +5142,8 @@ IntrinsicLibrary::genTransfer(mlir::Type resultType,
   mlir::Type type = (moldRank == 0 && absentSize)
                         ? resultType
                         : builder.getVarLenSeqTy(resultType, 1);
-  fir::MutableBoxValue resultMutableBox =
-      fir::factory::createTempMutableBox(builder, loc, type, {},
+  fir::MutableBoxValue resultMutableBox = fir::factory::createTempMutableBox(
+      builder, loc, type, {},
       fir::isPolymorphicType(mold.getType()) ? mold : mlir::Value{});
 
   if (moldRank == 0 && absentSize) {
diff --git a/flang/lib/Optimizer/Builder/Runtime/Allocatable.cpp b/flang/lib/Optimizer/Builder/Runtime/Allocatable.cpp
index 221d32b9490f9..c05d66230f401 100644
--- a/flang/lib/Optimizer/Builder/Runtime/Allocatable.cpp
+++ b/flang/lib/Optimizer/Builder/Runtime/Allocatable.cpp
@@ -24,7 +24,8 @@ mlir::Value fir::runtime::genMoveAlloc(fir::FirOpBuilder &builder,
   mlir::Value sourceLine{
       fir::factory::locationToLineNo(builder, loc, fTy.getInput(6))};
   mlir::Value declaredTypeDesc;
-  if (fir::isPolymorphicType(from.getType())) {
+  if (fir::isPolymorphicType(from.getType()) &&
+      !fir::isUnlimitedPolymorphicType(from.getType())) {
     fir::ClassType clTy =
         fir::dyn_cast_ptrEleTy(from.getType()).dyn_cast<fir::ClassType>();
     mlir::Type derivedType = fir::unwrapInnerType(clTy.getEleTy());
diff --git a/flang/lib/Optimizer/Builder/Runtime/Assign.cpp b/flang/lib/Optimizer/Builder/Runtime/Assign.cpp
index 785afc79ee7cd..db4de8ac9f377 100644
--- a/flang/lib/Optimizer/Builder/Runtime/Assign.cpp
+++ b/flang/lib/Optimizer/Builder/Runtime/Assign.cpp
@@ -38,3 +38,19 @@ void fir::runtime::genAssignPolymorphic(fir::FirOpBuilder &builder,
                                             sourceBox, sourceFile, sourceLine);
   builder.create<fir::CallOp>(loc, func, args);
 }
+
+void fir::runtime::genAssignExplicitLengthCharacter(fir::FirOpBuilder &builder,
+                                                    mlir::Location loc,
+                                                    mlir::Value destBox,
+                                                    mlir::Value sourceBox) {
+  auto func =
+      fir::runtime::getRuntimeFunc<mkRTKey(AssignExplicitLengthCharacter)>(
+          loc, builder);
+  auto fTy = func.getFunctionType();
+  auto sourceFile = fir::factory::locationToFilename(builder, loc);
+  auto sourceLine =
+      fir::factory::locationToLineNo(builder, loc, fTy.getInput(3));
+  auto args = fir::runtime::createArguments(builder, loc, fTy, destBox,
+                                            sourceBox, sourceFile, sourceLine);
+  builder.create<fir::CallOp>(loc, func, args);
+}
diff --git a/flang/lib/Optimizer/Builder/Runtime/Transformational.cpp b/flang/lib/Optimizer/Builder/Runtime/Transformational.cpp
index c51f38672d25f..3ae097743e3ad 100644
--- a/flang/lib/Optimizer/Builder/Runtime/Transformational.cpp
+++ b/flang/lib/Optimizer/Builder/Runtime/Transformational.cpp
@@ -13,6 +13,7 @@
 #include "flang/Optimizer/Builder/FIRBuilder.h"
 #include "flang/Optimizer/Builder/Runtime/RTBuilder.h"
 #include "flang/Optimizer/Builder/Todo.h"
+#include "flang/Runtime/matmul-transpose.h"
 #include "flang/Runtime/matmul.h"
 #include "flang/Runtime/transformational.h"
 #include "mlir/Dialect/Func/IR/FuncOps.h"
@@ -351,6 +352,23 @@ void fir::runtime::genMatmul(fir::FirOpBuilder &builder, mlir::Location loc,
   builder.create<fir::CallOp>(loc, func, args);
 }
 
+/// Generate call to MatmulTranspose intrinsic runtime routine.
+void fir::runtime::genMatmulTranspose(fir::FirOpBuilder &builder,
+                                      mlir::Location loc, mlir::Value resultBox,
+                                      mlir::Value matrixABox,
+                                      mlir::Value matrixBBox) {
+  auto func =
+      fir::runtime::getRuntimeFunc<mkRTKey(MatmulTranspose)>(loc, builder);
+  auto fTy = func.getFunctionType();
+  auto sourceFile = fir::factory::locationToFilename(builder, loc);
+  auto sourceLine =
+      fir::factory::locationToLineNo(builder, loc, fTy.getInput(4));
+  auto args =
+      fir::runtime::createArguments(builder, loc, fTy, resultBox, matrixABox,
+                                    matrixBBox, sourceFile, sourceLine);
+  builder.create<fir::CallOp>(loc, func, args);
+}
+
 /// Generate call to Pack intrinsic runtime routine.
 void fir::runtime::genPack(fir::FirOpBuilder &builder, mlir::Location loc,
                            mlir::Value resultBox, mlir::Value arrayBox,
diff --git a/flang/lib/Optimizer/CodeGen/BoxedProcedure.cpp b/flang/lib/Optimizer/CodeGen/BoxedProcedure.cpp
index 4df4dcb2d02b7..1ce23fb44f7ee 100644
--- a/flang/lib/Optimizer/CodeGen/BoxedProcedure.cpp
+++ b/flang/lib/Optimizer/CodeGen/BoxedProcedure.cpp
@@ -13,7 +13,7 @@
 #include "flang/Optimizer/Dialect/FIRDialect.h"
 #include "flang/Optimizer/Dialect/FIROps.h"
 #include "flang/Optimizer/Dialect/FIRType.h"
-#include "flang/Optimizer/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
 #include "flang/Optimizer/Support/FatalError.h"
 #include "flang/Optimizer/Support/InternalNames.h"
 #include "mlir/IR/PatternMatch.h"
diff --git a/flang/lib/Optimizer/CodeGen/CMakeLists.txt b/flang/lib/Optimizer/CodeGen/CMakeLists.txt
index 58c48479ff375..1b8fe47084a6c 100644
--- a/flang/lib/Optimizer/CodeGen/CMakeLists.txt
+++ b/flang/lib/Optimizer/CodeGen/CMakeLists.txt
@@ -8,15 +8,14 @@ add_flang_library(FIRCodeGen
   TargetRewrite.cpp
 
   DEPENDS
-  FIRBuilder
   FIRDialect
-  FIRSupport
   FIROptCodeGenPassIncGen
   CGOpsIncGen
 
   LINK_LIBS
   FIRBuilder
   FIRDialect
+  FIRDialectSupport
   FIRSupport
   MLIRComplexToLLVM
   MLIRComplexToStandard
diff --git a/flang/lib/Optimizer/CodeGen/CodeGen.cpp b/flang/lib/Optimizer/CodeGen/CodeGen.cpp
index 75f95f213c87a..6b4591789c545 100644
--- a/flang/lib/Optimizer/CodeGen/CodeGen.cpp
+++ b/flang/lib/Optimizer/CodeGen/CodeGen.cpp
@@ -16,8 +16,10 @@
 #include "flang/ISO_Fortran_binding.h"
 #include "flang/Optimizer/Dialect/FIRAttr.h"
 #include "flang/Optimizer/Dialect/FIROps.h"
+#include "flang/Optimizer/Dialect/FIRType.h"
 #include "flang/Optimizer/Support/InternalNames.h"
 #include "flang/Optimizer/Support/TypeCode.h"
+#include "flang/Optimizer/Support/Utils.h"
 #include "flang/Semantics/runtime-type-info.h"
 #include "mlir/Conversion/ArithCommon/AttrToLLVMConverter.h"
 #include "mlir/Conversion/ArithToLLVM/ArithToLLVM.h"
@@ -50,9 +52,6 @@ namespace fir {
 // fir::LLVMTypeConverter for converting to LLVM IR dialect types.
 #include "TypeConverter.h"
 
-using BindingTable = llvm::DenseMap<llvm::StringRef, unsigned>;
-using BindingTables = llvm::DenseMap<llvm::StringRef, BindingTable>;
-
 // TODO: This should really be recovered from the specified target.
 static constexpr unsigned defaultAlign = 8;
 
@@ -106,7 +105,7 @@ class FIROpConversion : public mlir::ConvertOpToLLVMPattern<FromOp> {
 public:
   explicit FIROpConversion(fir::LLVMTypeConverter &lowering,
                            const fir::FIRToLLVMPassOptions &options,
-                           const BindingTables &bindingTables)
+                           const fir::BindingTables &bindingTables)
       : mlir::ConvertOpToLLVMPattern<FromOp>(lowering), options(options),
         bindingTables(bindingTables) {}
 
@@ -359,7 +358,7 @@ class FIROpConversion : public mlir::ConvertOpToLLVMPattern<FromOp> {
   }
 
   const fir::FIRToLLVMPassOptions &options;
-  const BindingTables &bindingTables;
+  const fir::BindingTables &bindingTables;
 };
 
 /// FIR conversion pattern template
@@ -938,6 +937,10 @@ struct ConvertOpConversion : public FIROpConversion<fir::ConvertOp> {
           rewriter.replaceOpWithNewOp<mlir::LLVM::TruncOp>(convert, toTy, op0);
           return mlir::success();
         }
+        if (fromFirTy == i1Type) {
+          rewriter.replaceOpWithNewOp<mlir::LLVM::ZExtOp>(convert, toTy, op0);
+          return mlir::success();
+        }
         rewriter.replaceOpWithNewOp<mlir::LLVM::SExtOp>(convert, toTy, op0);
         return mlir::success();
       }
@@ -993,7 +996,7 @@ struct DispatchOpConversion : public FIROpConversion<fir::DispatchOp> {
              << "cannot find binding table for " << recordType.getName();
 
     // Lookup for the binding.
-    const BindingTable &bindingTable = bindingsIter->second;
+    const fir::BindingTable &bindingTable = bindingsIter->second;
     auto bindingIter = bindingTable.find(dispatch.getMethod());
     if (bindingIter == bindingTable.end())
       return emitError(loc)
@@ -1336,22 +1339,6 @@ struct EmboxCommonConversion : public FIROpConversion<OP> {
     return CFI_attribute_other;
   }
 
-  static fir::RecordType unwrapIfDerived(fir::BaseBoxType boxTy) {
-    return fir::unwrapSequenceType(fir::dyn_cast_ptrOrBoxEleTy(boxTy))
-        .template dyn_cast<fir::RecordType>();
-  }
-  static bool isDerivedTypeWithLenParams(fir::BaseBoxType boxTy) {
-    auto recTy = unwrapIfDerived(boxTy);
-    return recTy && recTy.getNumLenParams() > 0;
-  }
-  static bool isDerivedType(fir::BaseBoxType boxTy) {
-    return static_cast<bool>(unwrapIfDerived(boxTy));
-  }
-  static bool hasAddendum(fir::BaseBoxType boxTy) {
-    return static_cast<bool>(unwrapIfDerived(boxTy)) ||
-           fir::isUnlimitedPolymorphicType(boxTy);
-  }
-
   // Get the element size and CFI type code of the boxed value.
   std::tuple<mlir::Value, mlir::Value> getSizeAndTypeCode(
       mlir::Location loc, mlir::ConversionPatternRewriter &rewriter,
@@ -1571,7 +1558,7 @@ struct EmboxCommonConversion : public FIROpConversion<OP> {
     descriptor =
         insertField(rewriter, loc, descriptor, {kAttributePosInBox},
                     this->genI32Constant(loc, rewriter, getCFIAttr(boxTy)));
-    const bool hasAddendum = isDerivedType(boxTy) || isUnlimitedPolymorphic;
+    const bool hasAddendum = fir::boxHasAddendum(boxTy);
     descriptor =
         insertField(rewriter, loc, descriptor, {kF18AddendumPosInBox},
                     this->genI32Constant(loc, rewriter, hasAddendum ? 1 : 0));
@@ -1591,8 +1578,8 @@ struct EmboxCommonConversion : public FIROpConversion<OP> {
                 loc, ::getVoidPtrType(mod.getContext()));
           }
         } else {
-          typeDesc =
-              getTypeDescriptor(mod, rewriter, loc, unwrapIfDerived(boxTy));
+          typeDesc = getTypeDescriptor(mod, rewriter, loc,
+                                       fir::unwrapIfDerived(boxTy));
         }
       }
       if (typeDesc)
@@ -1674,7 +1661,7 @@ struct EmboxCommonConversion : public FIROpConversion<OP> {
       // TODO: For initial box that are unlimited polymorphic entities, this
       // code must be made conditional because unlimited polymorphic entities
       // with intrinsic type spec does not have addendum.
-      if (hasAddendum(inputBoxTy))
+      if (fir::boxHasAddendum(inputBoxTy))
         typeDesc = this->loadTypeDescAddress(loc, box.getBox().getType(),
                                              loweredBox, rewriter);
     }
@@ -1826,7 +1813,7 @@ struct EmboxOpConversion : public EmboxCommonConversion<fir::EmboxOp> {
         /*rank=*/0, /*lenParams=*/operands.drop_front(1), sourceBox,
         sourceBoxType);
     dest = insertBaseAddress(rewriter, embox.getLoc(), dest, operands[0]);
-    if (isDerivedTypeWithLenParams(boxTy)) {
+    if (fir::isDerivedTypeWithLenParams(boxTy)) {
       TODO(embox.getLoc(),
            "fir.embox codegen of derived with length parameters");
       return mlir::failure();
@@ -2010,7 +1997,7 @@ struct XEmboxOpConversion : public EmboxCommonConversion<fir::cg::XEmboxOp> {
                              fieldIndices, substringOffset);
     }
     dest = insertBaseAddress(rewriter, loc, dest, base);
-    if (isDerivedTypeWithLenParams(boxTy))
+    if (fir::isDerivedTypeWithLenParams(boxTy))
       TODO(loc, "fir.embox codegen of derived with length parameters");
 
     mlir::Value result =
@@ -3670,7 +3657,7 @@ template <typename FromOp>
 struct MustBeDeadConversion : public FIROpConversion<FromOp> {
   explicit MustBeDeadConversion(fir::LLVMTypeConverter &lowering,
                                 const fir::FIRToLLVMPassOptions &options,
-                                const BindingTables &bindingTables)
+                                const fir::BindingTables &bindingTables)
       : FIROpConversion<FromOp>(lowering, options, bindingTables) {}
   using OpAdaptor = typename FromOp::Adaptor;
 
@@ -3781,24 +3768,8 @@ class FIRToLLVMLowering
     if (mlir::failed(runPipeline(mathConvertionPM, mod)))
       return signalPassFailure();
 
-    // Reconstruct binding tables for dynamic dispatch. The binding tables
-    // are defined in FIR from lowering as fir.dispatch_table operation.
-    // Go through each binding tables and store the procedure name
-    // and binding index for later use by the fir.dispatch conversion pattern.
-    BindingTables bindingTables;
-    for (auto dispatchTableOp : mod.getOps<fir::DispatchTableOp>()) {
-      unsigned bindingIdx = 0;
-      BindingTable bindings;
-      if (dispatchTableOp.getRegion().empty()) {
-        bindingTables[dispatchTableOp.getSymName()] = bindings;
-        continue;
-      }
-      for (auto dtEntry : dispatchTableOp.getBlock().getOps<fir::DTEntryOp>()) {
-        bindings[dtEntry.getMethod()] = bindingIdx;
-        ++bindingIdx;
-      }
-      bindingTables[dispatchTableOp.getSymName()] = bindings;
-    }
+    fir::BindingTables bindingTables;
+    fir::buildBindingTables(bindingTables, mod);
 
     auto *context = getModule().getContext();
     fir::LLVMTypeConverter typeConverter{getModule(),
diff --git a/flang/lib/Optimizer/CodeGen/PreCGRewrite.cpp b/flang/lib/Optimizer/CodeGen/PreCGRewrite.cpp
index a45fed2a36798..0170b56367cf3 100644
--- a/flang/lib/Optimizer/CodeGen/PreCGRewrite.cpp
+++ b/flang/lib/Optimizer/CodeGen/PreCGRewrite.cpp
@@ -17,7 +17,7 @@
 #include "flang/Optimizer/Dialect/FIRDialect.h"
 #include "flang/Optimizer/Dialect/FIROps.h"
 #include "flang/Optimizer/Dialect/FIRType.h"
-#include "flang/Optimizer/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
 #include "mlir/Transforms/DialectConversion.h"
 #include "mlir/Transforms/RegionUtils.h"
 #include "llvm/ADT/STLExtras.h"
diff --git a/flang/lib/Optimizer/CodeGen/Target.cpp b/flang/lib/Optimizer/CodeGen/Target.cpp
index e2d30b0019d40..50d144909a1c7 100644
--- a/flang/lib/Optimizer/CodeGen/Target.cpp
+++ b/flang/lib/Optimizer/CodeGen/Target.cpp
@@ -13,8 +13,8 @@
 #include "Target.h"
 #include "flang/Optimizer/Builder/Todo.h"
 #include "flang/Optimizer/Dialect/FIRType.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "flang/Optimizer/Support/FatalError.h"
-#include "flang/Optimizer/Support/KindMapping.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/TypeRange.h"
 
diff --git a/flang/lib/Optimizer/CodeGen/Target.h b/flang/lib/Optimizer/CodeGen/Target.h
index 6bc56a9dc4559..acffe6c1cec9c 100644
--- a/flang/lib/Optimizer/CodeGen/Target.h
+++ b/flang/lib/Optimizer/CodeGen/Target.h
@@ -13,7 +13,7 @@
 #ifndef FORTRAN_OPTMIZER_CODEGEN_TARGET_H
 #define FORTRAN_OPTMIZER_CODEGEN_TARGET_H
 
-#include "flang/Optimizer/Support/KindMapping.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "llvm/TargetParser/Triple.h"
 #include <memory>
diff --git a/flang/lib/Optimizer/CodeGen/TargetRewrite.cpp b/flang/lib/Optimizer/CodeGen/TargetRewrite.cpp
index c51bf5f6993b9..0957e399f4e56 100644
--- a/flang/lib/Optimizer/CodeGen/TargetRewrite.cpp
+++ b/flang/lib/Optimizer/CodeGen/TargetRewrite.cpp
@@ -24,7 +24,7 @@
 #include "flang/Optimizer/Dialect/FIROps.h"
 #include "flang/Optimizer/Dialect/FIROpsSupport.h"
 #include "flang/Optimizer/Dialect/FIRType.h"
-#include "flang/Optimizer/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
 #include "mlir/Transforms/DialectConversion.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/TypeSwitch.h"
@@ -468,6 +468,16 @@ class TargetRewrite : public fir::impl::TargetRewritePassBase<TargetRewrite> {
     return mlir::success();
   }
 
+  // Returns true if the function should be interoperable with C.
+  static bool isFuncWithCCallingConvention(mlir::Operation *op) {
+    auto funcOp = mlir::dyn_cast<mlir::func::FuncOp>(op);
+    if (!funcOp)
+      return false;
+    return op->hasAttrOfType<mlir::UnitAttr>(
+               fir::FIROpsDialect::getFirRuntimeAttrName()) ||
+           op->hasAttrOfType<mlir::StringAttr>(fir::getSymbolAttrName());
+  }
+
   /// If the signature does not need any special target-specific conversions,
   /// then it is considered portable for any target, and this function will
   /// return `true`. Otherwise, the signature is not portable and `false` is
@@ -475,12 +485,11 @@ class TargetRewrite : public fir::impl::TargetRewritePassBase<TargetRewrite> {
   bool hasPortableSignature(mlir::Type signature, mlir::Operation *op) {
     assert(signature.isa<mlir::FunctionType>());
     auto func = signature.dyn_cast<mlir::FunctionType>();
-    bool hasFirRuntime = op->hasAttrOfType<mlir::UnitAttr>(
-        fir::FIROpsDialect::getFirRuntimeAttrName());
+    bool hasCCallingConv = isFuncWithCCallingConvention(op);
     for (auto ty : func.getResults())
       if ((ty.isa<fir::BoxCharType>() && !noCharacterConversion) ||
           (fir::isa_complex(ty) && !noComplexConversion) ||
-          (ty.isa<mlir::IntegerType>() && hasFirRuntime)) {
+          (ty.isa<mlir::IntegerType>() && hasCCallingConv)) {
         LLVM_DEBUG(llvm::dbgs() << "rewrite " << signature << " for target\n");
         return false;
       }
@@ -488,7 +497,7 @@ class TargetRewrite : public fir::impl::TargetRewritePassBase<TargetRewrite> {
       if (((ty.isa<fir::BoxCharType>() || fir::isCharacterProcedureTuple(ty)) &&
            !noCharacterConversion) ||
           (fir::isa_complex(ty) && !noComplexConversion) ||
-          (ty.isa<mlir::IntegerType>() && hasFirRuntime)) {
+          (ty.isa<mlir::IntegerType>() && hasCCallingConv)) {
         LLVM_DEBUG(llvm::dbgs() << "rewrite " << signature << " for target\n");
         return false;
       }
@@ -552,9 +561,7 @@ class TargetRewrite : public fir::impl::TargetRewritePassBase<TargetRewrite> {
             std::size_t resId = newResTys.size();
             llvm::StringRef extensionAttrName = attr.getIntExtensionAttrName();
             if (!extensionAttrName.empty() &&
-                // TODO: we have to do the same for BIND(C) routines.
-                func->hasAttrOfType<mlir::UnitAttr>(
-                    fir::FIROpsDialect::getFirRuntimeAttrName()))
+                isFuncWithCCallingConvention(func))
               resultAttrs.emplace_back(
                   resId, rewriter->getNamedAttr(extensionAttrName,
                                                 rewriter->getUnitAttr()));
@@ -631,16 +638,14 @@ class TargetRewrite : public fir::impl::TargetRewritePassBase<TargetRewrite> {
             auto argNo = newInTys.size();
             llvm::StringRef extensionAttrName = attr.getIntExtensionAttrName();
             if (!extensionAttrName.empty() &&
-                // TODO: we have to do the same for BIND(C) routines.
-                func->hasAttrOfType<mlir::UnitAttr>(
-                    fir::FIROpsDialect::getFirRuntimeAttrName())) {
+                isFuncWithCCallingConvention(func))
               fixups.emplace_back(FixupTy::Codes::ArgumentType, argNo,
                                   [=](mlir::func::FuncOp func) {
                                     func.setArgAttr(
                                         argNo, extensionAttrName,
                                         mlir::UnitAttr::get(func.getContext()));
                                   });
-            }
+
             newInTys.push_back(argTy);
           })
           .Default([&](mlir::Type ty) { newInTys.push_back(ty); });
diff --git a/flang/lib/Optimizer/CodeGen/TypeConverter.h b/flang/lib/Optimizer/CodeGen/TypeConverter.h
index 4f472d31c93e6..36c02b8ff71c8 100644
--- a/flang/lib/Optimizer/CodeGen/TypeConverter.h
+++ b/flang/lib/Optimizer/CodeGen/TypeConverter.h
@@ -18,8 +18,8 @@
 #include "Target.h"
 #include "flang/Optimizer/Builder/Todo.h" // remove when TODO's are done
 #include "flang/Optimizer/Dialect/FIRType.h"
-#include "flang/Optimizer/Support/FIRContext.h"
-#include "flang/Optimizer/Support/KindMapping.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "mlir/Conversion/LLVMCommon/TypeConverter.h"
 #include "llvm/Support/Debug.h"
 
@@ -47,7 +47,13 @@ namespace fir {
 class LLVMTypeConverter : public mlir::LLVMTypeConverter {
 public:
   LLVMTypeConverter(mlir::ModuleOp module, bool applyTBAA)
-      : mlir::LLVMTypeConverter(module.getContext()),
+      : mlir::LLVMTypeConverter(module.getContext(),
+                                [&] {
+                                  mlir::LowerToLLVMOptions options(
+                                      module.getContext());
+                                  options.useOpaquePointers = false;
+                                  return options;
+                                }()),
         kindMapping(getKindMapping(module)),
         specifics(CodeGenSpecifics::get(module.getContext(),
                                         getTargetTriple(module),
diff --git a/flang/lib/Optimizer/Dialect/CMakeLists.txt b/flang/lib/Optimizer/Dialect/CMakeLists.txt
index 8f82e9264638e..342a9d141e72a 100644
--- a/flang/lib/Optimizer/Dialect/CMakeLists.txt
+++ b/flang/lib/Optimizer/Dialect/CMakeLists.txt
@@ -1,3 +1,5 @@
+add_subdirectory(Support)
+
 add_flang_library(FIRDialect
   FIRAttr.cpp
   FIRDialect.cpp
@@ -8,11 +10,12 @@ add_flang_library(FIRDialect
 
   DEPENDS
   CanonicalizationPatternsIncGen
-  FIRSupport
+  MLIRIR
   FIROpsIncGen
+  intrinsics_gen
 
   LINK_LIBS
-  FIRSupport
+  FIRDialectSupport
   MLIRArithDialect
   MLIROpenMPToLLVM
   MLIRLLVMToLLVMIRTranslation
diff --git a/flang/lib/Optimizer/Dialect/FIRAttr.cpp b/flang/lib/Optimizer/Dialect/FIRAttr.cpp
index 655a568382a80..487109121db0c 100644
--- a/flang/lib/Optimizer/Dialect/FIRAttr.cpp
+++ b/flang/lib/Optimizer/Dialect/FIRAttr.cpp
@@ -12,7 +12,7 @@
 
 #include "flang/Optimizer/Dialect/FIRAttr.h"
 #include "flang/Optimizer/Dialect/FIRDialect.h"
-#include "flang/Optimizer/Support/KindMapping.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "mlir/IR/AttributeSupport.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/DialectImplementation.h"
diff --git a/flang/lib/Optimizer/Dialect/FIROps.cpp b/flang/lib/Optimizer/Dialect/FIROps.cpp
index 671936f8186fd..5d8be2c5cd8e2 100644
--- a/flang/lib/Optimizer/Dialect/FIROps.cpp
+++ b/flang/lib/Optimizer/Dialect/FIROps.cpp
@@ -14,11 +14,12 @@
 #include "flang/Optimizer/Dialect/FIRAttr.h"
 #include "flang/Optimizer/Dialect/FIROpsSupport.h"
 #include "flang/Optimizer/Dialect/FIRType.h"
-#include "flang/Optimizer/Support/FIRContext.h"
-#include "flang/Optimizer/Support/KindMapping.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "flang/Optimizer/Support/Utils.h"
 #include "mlir/Dialect/CommonFolders.h"
 #include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/IR/Attributes.h"
 #include "mlir/IR/BuiltinAttributes.h"
 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/Diagnostics.h"
@@ -3403,6 +3404,59 @@ void fir::IfOp::build(mlir::OpBuilder &builder, mlir::OperationState &result,
   }
 }
 
+// These 2 functions copied from scf.if implementation.
+
+/// Given the region at `index`, or the parent operation if `index` is None,
+/// return the successor regions. These are the regions that may be selected
+/// during the flow of control. `operands` is a set of optional attributes that
+/// correspond to a constant value for each operand, or null if that operand is
+/// not a constant.
+void fir::IfOp::getSuccessorRegions(
+    std::optional<unsigned> index, llvm::ArrayRef<mlir::Attribute> operands,
+    llvm::SmallVectorImpl<mlir::RegionSuccessor> &regions) {
+  // The `then` and the `else` region branch back to the parent operation.
+  if (index) {
+    regions.push_back(mlir::RegionSuccessor(getResults()));
+    return;
+  }
+
+  // Don't consider the else region if it is empty.
+  mlir::Region *elseRegion = &this->getElseRegion();
+  if (elseRegion->empty())
+    elseRegion = nullptr;
+
+  // Otherwise, the successor is dependent on the condition.
+  bool condition;
+  if (auto condAttr = operands.front().dyn_cast_or_null<mlir::IntegerAttr>()) {
+    condition = condAttr.getValue().isOne();
+  } else {
+    // If the condition isn't constant, both regions may be executed.
+    regions.push_back(mlir::RegionSuccessor(&getThenRegion()));
+    // If the else region does not exist, it is not a viable successor.
+    if (elseRegion)
+      regions.push_back(mlir::RegionSuccessor(elseRegion));
+    return;
+  }
+
+  // Add the successor regions using the condition.
+  regions.push_back(
+      mlir::RegionSuccessor(condition ? &getThenRegion() : elseRegion));
+}
+
+void fir::IfOp::getRegionInvocationBounds(
+    llvm::ArrayRef<mlir::Attribute> operands,
+    llvm::SmallVectorImpl<mlir::InvocationBounds> &invocationBounds) {
+  if (auto cond = operands[0].dyn_cast_or_null<mlir::BoolAttr>()) {
+    // If the condition is known, then one region is known to be executed once
+    // and the other zero times.
+    invocationBounds.emplace_back(0, cond.getValue() ? 1 : 0);
+    invocationBounds.emplace_back(0, cond.getValue() ? 0 : 1);
+  } else {
+    // Non-constant condition. Each region may be executed 0 or 1 times.
+    invocationBounds.assign(2, {0, 1});
+  }
+}
+
 mlir::ParseResult fir::IfOp::parse(mlir::OpAsmParser &parser,
                                    mlir::OperationState &result) {
   result.regions.reserve(2);
diff --git a/flang/lib/Optimizer/Dialect/FIRType.cpp b/flang/lib/Optimizer/Dialect/FIRType.cpp
index decb93f3d55e3..c15244d349906 100644
--- a/flang/lib/Optimizer/Dialect/FIRType.cpp
+++ b/flang/lib/Optimizer/Dialect/FIRType.cpp
@@ -12,7 +12,7 @@
 
 #include "flang/Optimizer/Dialect/FIRType.h"
 #include "flang/Optimizer/Dialect/FIRDialect.h"
-#include "flang/Optimizer/Support/KindMapping.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "flang/Tools/PointerModels.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/IR/BuiltinDialect.h"
@@ -219,12 +219,8 @@ mlir::Type dyn_cast_ptrOrBoxEleTy(mlir::Type t) {
   return llvm::TypeSwitch<mlir::Type, mlir::Type>(t)
       .Case<fir::ReferenceType, fir::PointerType, fir::HeapType,
             fir::LLVMPointerType>([](auto p) { return p.getEleTy(); })
-      .Case<fir::BaseBoxType>([](auto p) {
-        auto eleTy = p.getEleTy();
-        if (auto ty = fir::dyn_cast_ptrEleTy(eleTy))
-          return ty;
-        return eleTy;
-      })
+      .Case<fir::BaseBoxType>(
+          [](auto p) { return unwrapRefType(p.getEleTy()); })
       .Default([](mlir::Type) { return mlir::Type{}; });
 }
 
@@ -304,7 +300,7 @@ bool isScalarBoxedRecordType(mlir::Type ty) {
   return false;
 }
 
-static bool isAssumedType(mlir::Type ty) {
+bool isAssumedType(mlir::Type ty) {
   if (auto boxTy = ty.dyn_cast<fir::BoxType>()) {
     if (boxTy.getEleTy().isa<mlir::NoneType>())
       return true;
@@ -460,6 +456,8 @@ static bool cannotBePointerOrHeapElementType(mlir::Type eleTy) {
 mlir::LogicalResult
 fir::BoxType::verify(llvm::function_ref<mlir::InFlightDiagnostic()> emitError,
                      mlir::Type eleTy) {
+  if (eleTy.isa<fir::BaseBoxType>())
+    return emitError() << "invalid element type\n";
   // TODO
   return mlir::success();
 }
diff --git a/flang/lib/Optimizer/Dialect/Support/CMakeLists.txt b/flang/lib/Optimizer/Dialect/Support/CMakeLists.txt
new file mode 100644
index 0000000000000..c37b0549f7fc1
--- /dev/null
+++ b/flang/lib/Optimizer/Dialect/Support/CMakeLists.txt
@@ -0,0 +1,13 @@
+get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
+
+add_flang_library(FIRDialectSupport
+  KindMapping.cpp
+  FIRContext.cpp
+
+  DEPENDS
+  MLIRIR
+  intrinsics_gen
+
+  LINK_LIBS
+  ${dialect_libs}
+)
diff --git a/flang/lib/Optimizer/Support/FIRContext.cpp b/flang/lib/Optimizer/Dialect/Support/FIRContext.cpp
similarity index 95%
rename from flang/lib/Optimizer/Support/FIRContext.cpp
rename to flang/lib/Optimizer/Dialect/Support/FIRContext.cpp
index 2a91a0a68b6bc..a2b77ebcf52ae 100644
--- a/flang/lib/Optimizer/Support/FIRContext.cpp
+++ b/flang/lib/Optimizer/Dialect/Support/FIRContext.cpp
@@ -10,8 +10,8 @@
 //
 //===----------------------------------------------------------------------===//
 
-#include "flang/Optimizer/Support/FIRContext.h"
-#include "flang/Optimizer/Support/KindMapping.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/IR/BuiltinAttributes.h"
 #include "mlir/IR/BuiltinOps.h"
diff --git a/flang/lib/Optimizer/Support/KindMapping.cpp b/flang/lib/Optimizer/Dialect/Support/KindMapping.cpp
similarity index 99%
rename from flang/lib/Optimizer/Support/KindMapping.cpp
rename to flang/lib/Optimizer/Dialect/Support/KindMapping.cpp
index e5500c14476de..c71e3939dc31d 100644
--- a/flang/lib/Optimizer/Support/KindMapping.cpp
+++ b/flang/lib/Optimizer/Dialect/Support/KindMapping.cpp
@@ -10,7 +10,7 @@
 //
 //===----------------------------------------------------------------------===//
 
-#include "flang/Optimizer/Support/KindMapping.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "llvm/Support/CommandLine.h"
 
diff --git a/flang/lib/Optimizer/HLFIR/IR/HLFIROps.cpp b/flang/lib/Optimizer/HLFIR/IR/HLFIROps.cpp
index 53259fe495baa..ef3b4d57e1f7f 100644
--- a/flang/lib/Optimizer/HLFIR/IR/HLFIROps.cpp
+++ b/flang/lib/Optimizer/HLFIR/IR/HLFIROps.cpp
@@ -20,6 +20,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/OpImplementation.h"
 #include "llvm/ADT/TypeSwitch.h"
+#include <iterator>
 #include <optional>
 #include <tuple>
 
@@ -336,7 +337,9 @@ mlir::LogicalResult hlfir::DesignateOp::verify() {
   // length may differ because of substrings.
   if (resultElementType != outputElementType &&
       !(resultElementType.isa<fir::CharacterType>() &&
-        outputElementType.isa<fir::CharacterType>()))
+        outputElementType.isa<fir::CharacterType>()) &&
+      !(resultElementType.isa<mlir::FloatType>() &&
+        outputElementType.isa<fir::RealType>()))
     return emitOpError(
                "result element type is not consistent with operands, expected ")
            << outputElementType;
@@ -636,6 +639,52 @@ mlir::LogicalResult hlfir::MatmulOp::verify() {
   return mlir::success();
 }
 
+mlir::LogicalResult
+hlfir::MatmulOp::canonicalize(MatmulOp matmulOp,
+                              mlir::PatternRewriter &rewriter) {
+  // the only two uses of the transposed matrix should be for the hlfir.matmul
+  // and hlfir.destory
+  auto isOtherwiseUnused = [&](hlfir::TransposeOp transposeOp) -> bool {
+    std::size_t numUses = 0;
+    for (mlir::Operation *user : transposeOp.getResult().getUsers()) {
+      ++numUses;
+      if (user == matmulOp)
+        continue;
+      if (mlir::dyn_cast_or_null<hlfir::DestroyOp>(user))
+        continue;
+      // some other use!
+      return false;
+    }
+    return numUses <= 2;
+  };
+
+  mlir::Value lhs = matmulOp.getLhs();
+  // Rewrite MATMUL(TRANSPOSE(lhs), rhs) => hlfir.matmul_transpose lhs, rhs
+  if (auto transposeOp = lhs.getDefiningOp<hlfir::TransposeOp>()) {
+    if (isOtherwiseUnused(transposeOp)) {
+      mlir::Location loc = matmulOp.getLoc();
+      mlir::Type resultTy = matmulOp.getResult().getType();
+      auto matmulTransposeOp = rewriter.create<hlfir::MatmulTransposeOp>(
+          loc, resultTy, transposeOp.getArray(), matmulOp.getRhs());
+
+      // we don't need to remove any hlfir.destroy because it will be needed for
+      // the new intrinsic result anyway
+      rewriter.replaceOp(matmulOp, matmulTransposeOp.getResult());
+
+      // but we do need to get rid of the hlfir.destroy for the hlfir.transpose
+      // result (which is entirely removed)
+      for (mlir::Operation *user : transposeOp->getResult(0).getUsers())
+        if (auto destroyOp = mlir::dyn_cast_or_null<hlfir::DestroyOp>(user))
+          rewriter.eraseOp(destroyOp);
+      rewriter.eraseOp(transposeOp);
+
+      return mlir::success();
+    }
+  }
+
+  return mlir::failure();
+}
+
 //===----------------------------------------------------------------------===//
 // TransposeOp
 //===----------------------------------------------------------------------===//
@@ -666,6 +715,71 @@ mlir::LogicalResult hlfir::TransposeOp::verify() {
   return mlir::success();
 }
 
+//===----------------------------------------------------------------------===//
+// MatmulTransposeOp
+//===----------------------------------------------------------------------===//
+
+mlir::LogicalResult hlfir::MatmulTransposeOp::verify() {
+  mlir::Value lhs = getLhs();
+  mlir::Value rhs = getRhs();
+  fir::SequenceType lhsTy =
+      hlfir::getFortranElementOrSequenceType(lhs.getType())
+          .cast<fir::SequenceType>();
+  fir::SequenceType rhsTy =
+      hlfir::getFortranElementOrSequenceType(rhs.getType())
+          .cast<fir::SequenceType>();
+  llvm::ArrayRef<int64_t> lhsShape = lhsTy.getShape();
+  llvm::ArrayRef<int64_t> rhsShape = rhsTy.getShape();
+  std::size_t lhsRank = lhsShape.size();
+  std::size_t rhsRank = rhsShape.size();
+  mlir::Type lhsEleTy = lhsTy.getEleTy();
+  mlir::Type rhsEleTy = rhsTy.getEleTy();
+  hlfir::ExprType resultTy = getResult().getType().cast<hlfir::ExprType>();
+  llvm::ArrayRef<int64_t> resultShape = resultTy.getShape();
+  mlir::Type resultEleTy = resultTy.getEleTy();
+
+  // lhs must have rank 2 for the transpose to be valid
+  if ((lhsRank != 2) || ((rhsRank != 1) && (rhsRank != 2)))
+    return emitOpError("array must have either rank 1 or rank 2");
+
+  if (mlir::isa<fir::LogicalType>(lhsEleTy) !=
+      mlir::isa<fir::LogicalType>(rhsEleTy))
+    return emitOpError("if one array is logical, so should the other be");
+
+  // for matmul we compare the last dimension of lhs with the first dimension of
+  // rhs, but for MatmulTranspose, dimensions of lhs are inverted by the
+  // transpose
+  int64_t firstLhsDim = lhsShape[0];
+  int64_t firstRhsDim = rhsShape[0];
+  constexpr int64_t unknownExtent = fir::SequenceType::getUnknownExtent();
+  if (firstLhsDim != firstRhsDim)
+    if ((firstLhsDim != unknownExtent) && (firstRhsDim != unknownExtent))
+      return emitOpError(
+          "the first dimension of LHS should match the first dimension of RHS");
+
+  if (mlir::isa<fir::LogicalType>(lhsEleTy) !=
+      mlir::isa<fir::LogicalType>(resultEleTy))
+    return emitOpError("the result type should be a logical only if the "
+                       "argument types are logical");
+
+  llvm::SmallVector<int64_t, 2> expectedResultShape;
+  if (rhsRank == 2) {
+    expectedResultShape.push_back(lhsShape[1]);
+    expectedResultShape.push_back(rhsShape[1]);
+  } else {
+    // rhsRank == 1
+    expectedResultShape.push_back(lhsShape[1]);
+  }
+  if (resultShape.size() != expectedResultShape.size())
+    return emitOpError("incorrect result shape");
+  if (resultShape[0] != expectedResultShape[0])
+    return emitOpError("incorrect result shape");
+  if (resultShape.size() == 2 && resultShape[1] != expectedResultShape[1])
+    return emitOpError("incorrect result shape");
+
+  return mlir::success();
+}
+
 //===----------------------------------------------------------------------===//
 // AssociateOp
 //===----------------------------------------------------------------------===//
diff --git a/flang/lib/Optimizer/HLFIR/Transforms/BufferizeHLFIR.cpp b/flang/lib/Optimizer/HLFIR/Transforms/BufferizeHLFIR.cpp
index 349a356f52b4a..44a61f0d9ef7d 100644
--- a/flang/lib/Optimizer/HLFIR/Transforms/BufferizeHLFIR.cpp
+++ b/flang/lib/Optimizer/HLFIR/Transforms/BufferizeHLFIR.cpp
@@ -15,23 +15,20 @@
 #include "flang/Optimizer/Builder/Character.h"
 #include "flang/Optimizer/Builder/FIRBuilder.h"
 #include "flang/Optimizer/Builder/HLFIRTools.h"
-#include "flang/Optimizer/Builder/IntrinsicCall.h"
-#include "flang/Optimizer/Builder/MutableBox.h"
 #include "flang/Optimizer/Builder/Runtime/Assign.h"
 #include "flang/Optimizer/Builder/Todo.h"
 #include "flang/Optimizer/Dialect/FIRDialect.h"
 #include "flang/Optimizer/Dialect/FIROps.h"
 #include "flang/Optimizer/Dialect/FIRType.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
 #include "flang/Optimizer/HLFIR/HLFIRDialect.h"
 #include "flang/Optimizer/HLFIR/HLFIROps.h"
 #include "flang/Optimizer/HLFIR/Passes.h"
-#include "flang/Optimizer/Support/FIRContext.h"
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/Pass/Pass.h"
 #include "mlir/Pass/PassManager.h"
 #include "mlir/Transforms/DialectConversion.h"
 #include <mlir/Support/LogicalResult.h>
-#include <optional>
 
 namespace hlfir {
 #define GEN_PASS_DEF_BUFFERIZEHLFIR
@@ -510,219 +507,6 @@ struct ElementalOpConversion
   }
 };
 
-/// Base class for passes converting transformational intrinsic operations into
-/// runtime calls
-template <class OP>
-class HlfirIntrinsicConversion : public mlir::OpConversionPattern<OP> {
-  using mlir::OpConversionPattern<OP>::OpConversionPattern;
-
-protected:
-  struct IntrinsicArgument {
-    mlir::Value val; // allowed to be null if the argument is absent
-    mlir::Type desiredType;
-  };
-
-  /// Lower the arguments to the intrinsic: adding nesecarry boxing and
-  /// conversion to match the signature of the intrinsic in the runtime library.
-  llvm::SmallVector<fir::ExtendedValue, 3>
-  lowerArguments(mlir::Operation *op,
-                 const llvm::ArrayRef<IntrinsicArgument> &args,
-                 mlir::ConversionPatternRewriter &rewriter,
-                 const fir::IntrinsicArgumentLoweringRules *argLowering) const {
-    mlir::Location loc = op->getLoc();
-    fir::KindMapping kindMapping{rewriter.getContext()};
-    fir::FirOpBuilder builder{rewriter, kindMapping};
-
-    llvm::SmallVector<fir::ExtendedValue, 3> ret;
-
-    for (size_t i = 0; i < args.size(); ++i) {
-      mlir::Value arg = args[i].val;
-      mlir::Type desiredType = args[i].desiredType;
-      if (!arg) {
-        ret.emplace_back(fir::getAbsentIntrinsicArgument());
-        continue;
-      }
-      hlfir::Entity entity{arg};
-
-      fir::ArgLoweringRule argRules =
-          fir::lowerIntrinsicArgumentAs(*argLowering, i);
-      switch (argRules.lowerAs) {
-      case fir::LowerIntrinsicArgAs::Value: {
-        if (args[i].desiredType != arg.getType()) {
-          arg = builder.createConvert(loc, desiredType, arg);
-          entity = hlfir::Entity{arg};
-        }
-        auto [exv, cleanup] = hlfir::convertToValue(loc, builder, entity);
-        if (cleanup)
-          TODO(loc, "extended value cleanup");
-        ret.emplace_back(exv);
-      } break;
-      case fir::LowerIntrinsicArgAs::Addr: {
-        auto [exv, cleanup] =
-            hlfir::convertToAddress(loc, builder, entity, desiredType);
-        if (cleanup)
-          TODO(loc, "extended value cleanup");
-        ret.emplace_back(exv);
-      } break;
-      case fir::LowerIntrinsicArgAs::Box: {
-        auto [box, cleanup] =
-            hlfir::convertToBox(loc, builder, entity, desiredType);
-        if (cleanup)
-          TODO(loc, "extended value cleanup");
-        ret.emplace_back(box);
-      } break;
-      case fir::LowerIntrinsicArgAs::Inquired: {
-        if (args[i].desiredType != arg.getType()) {
-          arg = builder.createConvert(loc, desiredType, arg);
-          entity = hlfir::Entity{arg};
-        }
-        // Place hlfir.expr in memory, and unbox fir.boxchar. Other entities
-        // are translated to fir::ExtendedValue without transofrmation (notably,
-        // pointers/allocatable are not dereferenced).
-        // TODO: once lowering to FIR retires, UBOUND and LBOUND can be
-        // simplified since the fir.box lowered here are now guarenteed to
-        // contain the local lower bounds thanks to the hlfir.declare (the extra
-        // rebox can be removed).
-        auto [exv, cleanup] =
-            hlfir::translateToExtendedValue(loc, builder, entity);
-        if (cleanup)
-          TODO(loc, "extended value cleanup");
-        ret.emplace_back(exv);
-      } break;
-      }
-    }
-
-    return ret;
-  }
-
-  void processReturnValue(mlir::Operation *op,
-                          const fir::ExtendedValue &resultExv, bool mustBeFreed,
-                          fir::FirOpBuilder &builder,
-                          mlir::PatternRewriter &rewriter) const {
-    mlir::Location loc = op->getLoc();
-
-    mlir::Value firBase = fir::getBase(resultExv);
-    mlir::Type firBaseTy = firBase.getType();
-
-    std::optional<hlfir::EntityWithAttributes> resultEntity;
-    if (fir::isa_trivial(firBaseTy)) {
-      resultEntity = hlfir::EntityWithAttributes{firBase};
-    } else {
-      resultEntity =
-          hlfir::genDeclare(loc, builder, resultExv, ".tmp.intrinsic_result",
-                            fir::FortranVariableFlagsAttr{});
-    }
-
-    if (resultEntity->isVariable()) {
-      hlfir::AsExprOp asExpr = builder.create<hlfir::AsExprOp>(
-          loc, *resultEntity, builder.createBool(loc, mustBeFreed));
-      resultEntity = hlfir::EntityWithAttributes{asExpr.getResult()};
-    }
-
-    rewriter.replaceOp(op, resultEntity->getBase());
-  }
-};
-
-struct SumOpConversion : public HlfirIntrinsicConversion<hlfir::SumOp> {
-  using HlfirIntrinsicConversion<hlfir::SumOp>::HlfirIntrinsicConversion;
-
-  mlir::LogicalResult
-  matchAndRewrite(hlfir::SumOp sum, OpAdaptor adaptor,
-                  mlir::ConversionPatternRewriter &rewriter) const override {
-    fir::KindMapping kindMapping{rewriter.getContext()};
-    fir::FirOpBuilder builder{rewriter, kindMapping};
-    const mlir::Location &loc = sum->getLoc();
-    HLFIRListener listener{builder, rewriter};
-    builder.setListener(&listener);
-
-    mlir::Type i32 = builder.getI32Type();
-    mlir::Type logicalType = fir::LogicalType::get(
-        builder.getContext(), builder.getKindMap().defaultLogicalKind());
-
-    llvm::SmallVector<IntrinsicArgument, 3> inArgs;
-    inArgs.push_back({sum.getArray(), sum.getArray().getType()});
-    inArgs.push_back({sum.getDim(), i32});
-    inArgs.push_back({sum.getMask(), logicalType});
-
-    auto *argLowering = fir::getIntrinsicArgumentLowering("sum");
-    llvm::SmallVector<fir::ExtendedValue, 3> args =
-        lowerArguments(sum, inArgs, rewriter, argLowering);
-
-    mlir::Type scalarResultType = hlfir::getFortranElementType(sum.getType());
-
-    auto [resultExv, mustBeFreed] =
-        fir::genIntrinsicCall(builder, loc, "sum", scalarResultType, args);
-
-    processReturnValue(sum, resultExv, mustBeFreed, builder, rewriter);
-    return mlir::success();
-  }
-};
-
-struct MatmulOpConversion : public HlfirIntrinsicConversion<hlfir::MatmulOp> {
-  using HlfirIntrinsicConversion<hlfir::MatmulOp>::HlfirIntrinsicConversion;
-
-  mlir::LogicalResult
-  matchAndRewrite(hlfir::MatmulOp matmul, OpAdaptor adaptor,
-                  mlir::ConversionPatternRewriter &rewriter) const override {
-    fir::KindMapping kindMapping{rewriter.getContext()};
-    fir::FirOpBuilder builder{rewriter, kindMapping};
-    const mlir::Location &loc = matmul->getLoc();
-    HLFIRListener listener{builder, rewriter};
-    builder.setListener(&listener);
-
-    mlir::Value lhs = matmul.getLhs();
-    mlir::Value rhs = matmul.getRhs();
-    llvm::SmallVector<IntrinsicArgument, 2> inArgs;
-    inArgs.push_back({lhs, lhs.getType()});
-    inArgs.push_back({rhs, rhs.getType()});
-
-    auto *argLowering = fir::getIntrinsicArgumentLowering("matmul");
-    llvm::SmallVector<fir::ExtendedValue, 2> args =
-        lowerArguments(matmul, inArgs, rewriter, argLowering);
-
-    mlir::Type scalarResultType =
-        hlfir::getFortranElementType(matmul.getType());
-
-    auto [resultExv, mustBeFreed] =
-        fir::genIntrinsicCall(builder, loc, "matmul", scalarResultType, args);
-
-    processReturnValue(matmul, resultExv, mustBeFreed, builder, rewriter);
-    return mlir::success();
-  }
-};
-
-class TransposeOpConversion
-    : public HlfirIntrinsicConversion<hlfir::TransposeOp> {
-  using HlfirIntrinsicConversion<hlfir::TransposeOp>::HlfirIntrinsicConversion;
-
-  mlir::LogicalResult
-  matchAndRewrite(hlfir::TransposeOp transpose, OpAdaptor adaptor,
-                  mlir::ConversionPatternRewriter &rewriter) const override {
-    fir::KindMapping kindMapping{rewriter.getContext()};
-    fir::FirOpBuilder builder{rewriter, kindMapping};
-    const mlir::Location &loc = transpose->getLoc();
-    HLFIRListener listener{builder, rewriter};
-    builder.setListener(&listener);
-
-    mlir::Value arg = transpose.getArray();
-    llvm::SmallVector<IntrinsicArgument, 1> inArgs;
-    inArgs.push_back({arg, arg.getType()});
-
-    auto *argLowering = fir::getIntrinsicArgumentLowering("transpose");
-    llvm::SmallVector<fir::ExtendedValue, 1> args =
-        lowerArguments(transpose, inArgs, rewriter, argLowering);
-
-    mlir::Type scalarResultType =
-        hlfir::getFortranElementType(transpose.getType());
-
-    auto [resultExv, mustBeFreed] = fir::genIntrinsicCall(
-        builder, loc, "transpose", scalarResultType, args);
-
-    processReturnValue(transpose, resultExv, mustBeFreed, builder, rewriter);
-    return mlir::success();
-  }
-};
-
 class BufferizeHLFIR : public hlfir::impl::BufferizeHLFIRBase<BufferizeHLFIR> {
 public:
   void runOnOperation() override {
@@ -740,9 +524,7 @@ class BufferizeHLFIR : public hlfir::impl::BufferizeHLFIRBase<BufferizeHLFIR> {
         .insert<ApplyOpConversion, AsExprOpConversion, AssignOpConversion,
                 AssociateOpConversion, ConcatOpConversion, DestroyOpConversion,
                 ElementalOpConversion, EndAssociateOpConversion,
-                MatmulOpConversion, NoReassocOpConversion,
-                SetLengthOpConversion, SumOpConversion, TransposeOpConversion>(
-            context);
+                NoReassocOpConversion, SetLengthOpConversion>(context);
     mlir::ConversionTarget target(*context);
     target.addIllegalOp<hlfir::ApplyOp, hlfir::AssociateOp, hlfir::ElementalOp,
                         hlfir::EndAssociateOp, hlfir::SetLengthOp,
diff --git a/flang/lib/Optimizer/HLFIR/Transforms/CMakeLists.txt b/flang/lib/Optimizer/HLFIR/Transforms/CMakeLists.txt
index 983929b82c805..0323dce441640 100644
--- a/flang/lib/Optimizer/HLFIR/Transforms/CMakeLists.txt
+++ b/flang/lib/Optimizer/HLFIR/Transforms/CMakeLists.txt
@@ -3,6 +3,7 @@ get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
 add_flang_library(HLFIRTransforms
   BufferizeHLFIR.cpp
   ConvertToFIR.cpp
+  LowerHLFIRIntrinsics.cpp
 
   DEPENDS
   FIRDialect
@@ -12,6 +13,7 @@ add_flang_library(HLFIRTransforms
   LINK_LIBS
   FIRDialect
   FIRBuilder
+  FIRDialectSupport
   FIRSupport
   HLFIRDialect
   MLIRIR
diff --git a/flang/lib/Optimizer/HLFIR/Transforms/ConvertToFIR.cpp b/flang/lib/Optimizer/HLFIR/Transforms/ConvertToFIR.cpp
index 160a77c7b9464..8af01f0f013e1 100644
--- a/flang/lib/Optimizer/HLFIR/Transforms/ConvertToFIR.cpp
+++ b/flang/lib/Optimizer/HLFIR/Transforms/ConvertToFIR.cpp
@@ -18,9 +18,9 @@
 #include "flang/Optimizer/Dialect/FIRDialect.h"
 #include "flang/Optimizer/Dialect/FIROps.h"
 #include "flang/Optimizer/Dialect/FIRType.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
 #include "flang/Optimizer/HLFIR/HLFIROps.h"
 #include "flang/Optimizer/HLFIR/Passes.h"
-#include "flang/Optimizer/Support/FIRContext.h"
 #include "mlir/Transforms/DialectConversion.h"
 
 namespace hlfir {
@@ -112,11 +112,11 @@ class AssignOpConversion : public mlir::OpRewritePattern<hlfir::AssignOp> {
         // Indicate the runtime that it should not reallocate in case of length
         // mismatch, and that it should use the LHS explicit/assumed length if
         // allocating/reallocation the LHS.
-        TODO(loc, "assignment to explicit length whole allocatable");
+        fir::runtime::genAssignExplicitLengthCharacter(builder, loc, to, from);
       } else if (lhs.isPolymorphic()) {
         // Indicate the runtime that the LHS must have the RHS dynamic type
         // after the assignment.
-        TODO(loc, "assignment to whole polymorphic entity");
+        fir::runtime::genAssignPolymorphic(builder, loc, to, from);
       } else {
         fir::runtime::genAssign(builder, loc, to, from);
       }
diff --git a/flang/lib/Optimizer/HLFIR/Transforms/LowerHLFIRIntrinsics.cpp b/flang/lib/Optimizer/HLFIR/Transforms/LowerHLFIRIntrinsics.cpp
new file mode 100644
index 0000000000000..95192a3e79d49
--- /dev/null
+++ b/flang/lib/Optimizer/HLFIR/Transforms/LowerHLFIRIntrinsics.cpp
@@ -0,0 +1,329 @@
+//===- LowerHLFIRIntrinsics.cpp - Bufferize HLFIR  ------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "flang/Optimizer/Builder/FIRBuilder.h"
+#include "flang/Optimizer/Builder/HLFIRTools.h"
+#include "flang/Optimizer/Builder/IntrinsicCall.h"
+#include "flang/Optimizer/Builder/Todo.h"
+#include "flang/Optimizer/Dialect/FIRDialect.h"
+#include "flang/Optimizer/Dialect/FIROps.h"
+#include "flang/Optimizer/Dialect/FIRType.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
+#include "flang/Optimizer/HLFIR/HLFIRDialect.h"
+#include "flang/Optimizer/HLFIR/HLFIROps.h"
+#include "flang/Optimizer/HLFIR/Passes.h"
+#include "mlir/IR/BuiltinDialect.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Pass/Pass.h"
+#include "mlir/Pass/PassManager.h"
+#include "mlir/Support/LogicalResult.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include <mlir/IR/MLIRContext.h>
+#include <optional>
+
+namespace hlfir {
+#define GEN_PASS_DEF_LOWERHLFIRINTRINSICS
+#include "flang/Optimizer/HLFIR/Passes.h.inc"
+} // namespace hlfir
+
+namespace {
+
+/// Base class for passes converting transformational intrinsic operations into
+/// runtime calls
+template <class OP>
+class HlfirIntrinsicConversion : public mlir::OpRewritePattern<OP> {
+  using mlir::OpRewritePattern<OP>::OpRewritePattern;
+
+protected:
+  struct IntrinsicArgument {
+    mlir::Value val; // allowed to be null if the argument is absent
+    mlir::Type desiredType;
+  };
+
+  /// Lower the arguments to the intrinsic: adding nesecarry boxing and
+  /// conversion to match the signature of the intrinsic in the runtime library.
+  llvm::SmallVector<fir::ExtendedValue, 3>
+  lowerArguments(mlir::Operation *op,
+                 const llvm::ArrayRef<IntrinsicArgument> &args,
+                 mlir::PatternRewriter &rewriter,
+                 const fir::IntrinsicArgumentLoweringRules *argLowering) const {
+    mlir::Location loc = op->getLoc();
+    fir::KindMapping kindMapping{rewriter.getContext()};
+    fir::FirOpBuilder builder{rewriter, kindMapping};
+
+    llvm::SmallVector<fir::ExtendedValue, 3> ret;
+    llvm::SmallVector<std::function<void()>, 2> cleanupFns;
+
+    for (size_t i = 0; i < args.size(); ++i) {
+      mlir::Value arg = args[i].val;
+      mlir::Type desiredType = args[i].desiredType;
+      if (!arg) {
+        ret.emplace_back(fir::getAbsentIntrinsicArgument());
+        continue;
+      }
+      hlfir::Entity entity{arg};
+
+      fir::ArgLoweringRule argRules =
+          fir::lowerIntrinsicArgumentAs(*argLowering, i);
+      switch (argRules.lowerAs) {
+      case fir::LowerIntrinsicArgAs::Value: {
+        if (args[i].desiredType != arg.getType()) {
+          arg = builder.createConvert(loc, desiredType, arg);
+          entity = hlfir::Entity{arg};
+        }
+        auto [exv, cleanup] = hlfir::convertToValue(loc, builder, entity);
+        if (cleanup)
+          cleanupFns.push_back(*cleanup);
+        ret.emplace_back(exv);
+      } break;
+      case fir::LowerIntrinsicArgAs::Addr: {
+        auto [exv, cleanup] =
+            hlfir::convertToAddress(loc, builder, entity, desiredType);
+        if (cleanup)
+          cleanupFns.push_back(*cleanup);
+        ret.emplace_back(exv);
+      } break;
+      case fir::LowerIntrinsicArgAs::Box: {
+        auto [box, cleanup] =
+            hlfir::convertToBox(loc, builder, entity, desiredType);
+        if (cleanup)
+          cleanupFns.push_back(*cleanup);
+        ret.emplace_back(box);
+      } break;
+      case fir::LowerIntrinsicArgAs::Inquired: {
+        if (args[i].desiredType != arg.getType()) {
+          arg = builder.createConvert(loc, desiredType, arg);
+          entity = hlfir::Entity{arg};
+        }
+        // Place hlfir.expr in memory, and unbox fir.boxchar. Other entities
+        // are translated to fir::ExtendedValue without transofrmation (notably,
+        // pointers/allocatable are not dereferenced).
+        // TODO: once lowering to FIR retires, UBOUND and LBOUND can be
+        // simplified since the fir.box lowered here are now guarenteed to
+        // contain the local lower bounds thanks to the hlfir.declare (the extra
+        // rebox can be removed).
+        auto [exv, cleanup] =
+            hlfir::translateToExtendedValue(loc, builder, entity);
+        if (cleanup)
+          cleanupFns.push_back(*cleanup);
+        ret.emplace_back(exv);
+      } break;
+      }
+    }
+
+    if (cleanupFns.size()) {
+      auto oldInsertionPoint = builder.saveInsertionPoint();
+      builder.setInsertionPointAfter(op);
+      for (std::function<void()> cleanup : cleanupFns)
+        cleanup();
+      builder.restoreInsertionPoint(oldInsertionPoint);
+    }
+
+    return ret;
+  }
+
+  void processReturnValue(mlir::Operation *op,
+                          const fir::ExtendedValue &resultExv, bool mustBeFreed,
+                          fir::FirOpBuilder &builder,
+                          mlir::PatternRewriter &rewriter) const {
+    mlir::Location loc = op->getLoc();
+
+    mlir::Value firBase = fir::getBase(resultExv);
+    mlir::Type firBaseTy = firBase.getType();
+
+    std::optional<hlfir::EntityWithAttributes> resultEntity;
+    if (fir::isa_trivial(firBaseTy)) {
+      resultEntity = hlfir::EntityWithAttributes{firBase};
+    } else {
+      resultEntity =
+          hlfir::genDeclare(loc, builder, resultExv, ".tmp.intrinsic_result",
+                            fir::FortranVariableFlagsAttr{});
+    }
+
+    if (resultEntity->isVariable()) {
+      hlfir::AsExprOp asExpr = builder.create<hlfir::AsExprOp>(
+          loc, *resultEntity, builder.createBool(loc, mustBeFreed));
+      resultEntity = hlfir::EntityWithAttributes{asExpr.getResult()};
+    }
+
+    mlir::Value base = resultEntity->getBase();
+    if (!mlir::isa<hlfir::ExprType>(base.getType())) {
+      for (mlir::Operation *use : op->getResult(0).getUsers()) {
+        if (mlir::isa<hlfir::DestroyOp>(use))
+          rewriter.eraseOp(use);
+      }
+    }
+    rewriter.replaceAllUsesWith(op->getResults(), {base});
+    rewriter.replaceOp(op, base);
+  }
+};
+
+struct SumOpConversion : public HlfirIntrinsicConversion<hlfir::SumOp> {
+  using HlfirIntrinsicConversion<hlfir::SumOp>::HlfirIntrinsicConversion;
+
+  mlir::LogicalResult
+  matchAndRewrite(hlfir::SumOp sum,
+                  mlir::PatternRewriter &rewriter) const override {
+    fir::KindMapping kindMapping{rewriter.getContext()};
+    fir::FirOpBuilder builder{rewriter, kindMapping};
+    const mlir::Location &loc = sum->getLoc();
+
+    mlir::Type i32 = builder.getI32Type();
+    mlir::Type logicalType = fir::LogicalType::get(
+        builder.getContext(), builder.getKindMap().defaultLogicalKind());
+
+    llvm::SmallVector<IntrinsicArgument, 3> inArgs;
+    inArgs.push_back({sum.getArray(), sum.getArray().getType()});
+    inArgs.push_back({sum.getDim(), i32});
+    inArgs.push_back({sum.getMask(), logicalType});
+
+    auto *argLowering = fir::getIntrinsicArgumentLowering("sum");
+    llvm::SmallVector<fir::ExtendedValue, 3> args =
+        lowerArguments(sum, inArgs, rewriter, argLowering);
+
+    mlir::Type scalarResultType = hlfir::getFortranElementType(sum.getType());
+
+    auto [resultExv, mustBeFreed] =
+        fir::genIntrinsicCall(builder, loc, "sum", scalarResultType, args);
+
+    processReturnValue(sum, resultExv, mustBeFreed, builder, rewriter);
+    return mlir::success();
+  }
+};
+
+struct MatmulOpConversion : public HlfirIntrinsicConversion<hlfir::MatmulOp> {
+  using HlfirIntrinsicConversion<hlfir::MatmulOp>::HlfirIntrinsicConversion;
+
+  mlir::LogicalResult
+  matchAndRewrite(hlfir::MatmulOp matmul,
+                  mlir::PatternRewriter &rewriter) const override {
+    fir::KindMapping kindMapping{rewriter.getContext()};
+    fir::FirOpBuilder builder{rewriter, kindMapping};
+    const mlir::Location &loc = matmul->getLoc();
+
+    mlir::Value lhs = matmul.getLhs();
+    mlir::Value rhs = matmul.getRhs();
+    llvm::SmallVector<IntrinsicArgument, 2> inArgs;
+    inArgs.push_back({lhs, lhs.getType()});
+    inArgs.push_back({rhs, rhs.getType()});
+
+    auto *argLowering = fir::getIntrinsicArgumentLowering("matmul");
+    llvm::SmallVector<fir::ExtendedValue, 2> args =
+        lowerArguments(matmul, inArgs, rewriter, argLowering);
+
+    mlir::Type scalarResultType =
+        hlfir::getFortranElementType(matmul.getType());
+
+    auto [resultExv, mustBeFreed] =
+        fir::genIntrinsicCall(builder, loc, "matmul", scalarResultType, args);
+
+    processReturnValue(matmul, resultExv, mustBeFreed, builder, rewriter);
+    return mlir::success();
+  }
+};
+
+class TransposeOpConversion
+    : public HlfirIntrinsicConversion<hlfir::TransposeOp> {
+  using HlfirIntrinsicConversion<hlfir::TransposeOp>::HlfirIntrinsicConversion;
+
+  mlir::LogicalResult
+  matchAndRewrite(hlfir::TransposeOp transpose,
+                  mlir::PatternRewriter &rewriter) const override {
+    fir::KindMapping kindMapping{rewriter.getContext()};
+    fir::FirOpBuilder builder{rewriter, kindMapping};
+    const mlir::Location &loc = transpose->getLoc();
+
+    mlir::Value arg = transpose.getArray();
+    llvm::SmallVector<IntrinsicArgument, 1> inArgs;
+    inArgs.push_back({arg, arg.getType()});
+
+    auto *argLowering = fir::getIntrinsicArgumentLowering("transpose");
+    llvm::SmallVector<fir::ExtendedValue, 1> args =
+        lowerArguments(transpose, inArgs, rewriter, argLowering);
+
+    mlir::Type scalarResultType =
+        hlfir::getFortranElementType(transpose.getType());
+
+    auto [resultExv, mustBeFreed] = fir::genIntrinsicCall(
+        builder, loc, "transpose", scalarResultType, args);
+
+    processReturnValue(transpose, resultExv, mustBeFreed, builder, rewriter);
+    return mlir::success();
+  }
+};
+
+struct MatmulTransposeOpConversion
+    : public HlfirIntrinsicConversion<hlfir::MatmulTransposeOp> {
+  using HlfirIntrinsicConversion<
+      hlfir::MatmulTransposeOp>::HlfirIntrinsicConversion;
+
+  mlir::LogicalResult
+  matchAndRewrite(hlfir::MatmulTransposeOp multranspose,
+                  mlir::PatternRewriter &rewriter) const override {
+    fir::KindMapping kindMapping{rewriter.getContext()};
+    fir::FirOpBuilder builder{rewriter, kindMapping};
+    const mlir::Location &loc = multranspose->getLoc();
+
+    mlir::Value lhs = multranspose.getLhs();
+    mlir::Value rhs = multranspose.getRhs();
+    llvm::SmallVector<IntrinsicArgument, 2> inArgs;
+    inArgs.push_back({lhs, lhs.getType()});
+    inArgs.push_back({rhs, rhs.getType()});
+
+    auto *argLowering = fir::getIntrinsicArgumentLowering("matmul");
+    llvm::SmallVector<fir::ExtendedValue, 2> args =
+        lowerArguments(multranspose, inArgs, rewriter, argLowering);
+
+    mlir::Type scalarResultType =
+        hlfir::getFortranElementType(multranspose.getType());
+
+    auto [resultExv, mustBeFreed] = fir::genIntrinsicCall(
+        builder, loc, "matmul_transpose", scalarResultType, args);
+
+    processReturnValue(multranspose, resultExv, mustBeFreed, builder, rewriter);
+    return mlir::success();
+  }
+};
+
+class LowerHLFIRIntrinsics
+    : public hlfir::impl::LowerHLFIRIntrinsicsBase<LowerHLFIRIntrinsics> {
+public:
+  void runOnOperation() override {
+    // TODO: make this a pass operating on FuncOp. The issue is that
+    // FirOpBuilder helpers may generate new FuncOp because of runtime/llvm
+    // intrinsics calls creation. This may create race conflict if the pass is
+    // scheduled on FuncOp. A solution could be to provide an optional mutex
+    // when building a FirOpBuilder and locking around FuncOp and GlobalOp
+    // creation, but this needs a bit more thinking, so at this point the pass
+    // is scheduled on the moduleOp.
+    mlir::ModuleOp module = this->getOperation();
+    mlir::MLIRContext *context = &getContext();
+    mlir::RewritePatternSet patterns(context);
+    patterns.insert<MatmulOpConversion, MatmulTransposeOpConversion,
+                    SumOpConversion, TransposeOpConversion>(context);
+    mlir::ConversionTarget target(*context);
+    target.addLegalDialect<mlir::BuiltinDialect, mlir::arith::ArithDialect,
+                           mlir::func::FuncDialect, fir::FIROpsDialect,
+                           hlfir::hlfirDialect>();
+    target.addIllegalOp<hlfir::MatmulOp, hlfir::MatmulTransposeOp, hlfir::SumOp,
+                        hlfir::TransposeOp>();
+    target.markUnknownOpDynamicallyLegal(
+        [](mlir::Operation *) { return true; });
+    if (mlir::failed(
+            mlir::applyFullConversion(module, target, std::move(patterns)))) {
+      mlir::emitError(mlir::UnknownLoc::get(context),
+                      "failure in HLFIR intrinsic lowering");
+      signalPassFailure();
+    }
+  }
+};
+} // namespace
+
+std::unique_ptr<mlir::Pass> hlfir::createLowerHLFIRIntrinsicsPass() {
+  return std::make_unique<LowerHLFIRIntrinsics>();
+}
diff --git a/flang/lib/Optimizer/Support/CMakeLists.txt b/flang/lib/Optimizer/Support/CMakeLists.txt
index 6a1c004ac88a4..33a9ed4e439d5 100644
--- a/flang/lib/Optimizer/Support/CMakeLists.txt
+++ b/flang/lib/Optimizer/Support/CMakeLists.txt
@@ -1,10 +1,8 @@
 get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
 
 add_flang_library(FIRSupport
-  FIRContext.cpp
   InitFIR.cpp
   InternalNames.cpp
-  KindMapping.cpp
 
   DEPENDS
   FIROpsIncGen
diff --git a/flang/lib/Optimizer/Transforms/AbstractResult.cpp b/flang/lib/Optimizer/Transforms/AbstractResult.cpp
index 5ef76feabf8dc..2c0576eaa5cc4 100644
--- a/flang/lib/Optimizer/Transforms/AbstractResult.cpp
+++ b/flang/lib/Optimizer/Transforms/AbstractResult.cpp
@@ -11,7 +11,7 @@
 #include "flang/Optimizer/Dialect/FIRDialect.h"
 #include "flang/Optimizer/Dialect/FIROps.h"
 #include "flang/Optimizer/Dialect/FIRType.h"
-#include "flang/Optimizer/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
 #include "flang/Optimizer/Transforms/Passes.h"
 #include "mlir/Dialect/Func/IR/FuncOps.h"
 #include "mlir/IR/Diagnostics.h"
diff --git a/flang/lib/Optimizer/Transforms/AddDebugFoundation.cpp b/flang/lib/Optimizer/Transforms/AddDebugFoundation.cpp
index 09c5edc677935..be8f26dc678ca 100644
--- a/flang/lib/Optimizer/Transforms/AddDebugFoundation.cpp
+++ b/flang/lib/Optimizer/Transforms/AddDebugFoundation.cpp
@@ -16,7 +16,7 @@
 #include "flang/Optimizer/Dialect/FIRDialect.h"
 #include "flang/Optimizer/Dialect/FIROps.h"
 #include "flang/Optimizer/Dialect/FIRType.h"
-#include "flang/Optimizer/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
 #include "flang/Optimizer/Transforms/Passes.h"
 #include "mlir/Dialect/Func/IR/FuncOps.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
diff --git a/flang/lib/Optimizer/Transforms/ArrayValueCopy.cpp b/flang/lib/Optimizer/Transforms/ArrayValueCopy.cpp
index c0563588bfc30..f07debfc19eba 100644
--- a/flang/lib/Optimizer/Transforms/ArrayValueCopy.cpp
+++ b/flang/lib/Optimizer/Transforms/ArrayValueCopy.cpp
@@ -14,7 +14,7 @@
 #include "flang/Optimizer/Builder/Todo.h"
 #include "flang/Optimizer/Dialect/FIRDialect.h"
 #include "flang/Optimizer/Dialect/FIROpsSupport.h"
-#include "flang/Optimizer/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
 #include "flang/Optimizer/Transforms/Passes.h"
 #include "mlir/Dialect/ControlFlow/IR/ControlFlowOps.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
diff --git a/flang/lib/Optimizer/Transforms/CMakeLists.txt b/flang/lib/Optimizer/Transforms/CMakeLists.txt
index 2db16de2187a2..ca690c341bb26 100644
--- a/flang/lib/Optimizer/Transforms/CMakeLists.txt
+++ b/flang/lib/Optimizer/Transforms/CMakeLists.txt
@@ -17,14 +17,13 @@ add_flang_library(FIRTransforms
   PolymorphicOpConversion.cpp
 
   DEPENDS
-  FIRBuilder
   FIRDialect
-  FIRSupport
   FIROptTransformsPassIncGen
 
   LINK_LIBS
   FIRBuilder
   FIRDialect
+  FIRDialectSupport
   FIRSupport
   HLFIRDialect
   MLIRAffineUtils
diff --git a/flang/lib/Optimizer/Transforms/CharacterConversion.cpp b/flang/lib/Optimizer/Transforms/CharacterConversion.cpp
index 971fb942e2cbd..4940092871cbb 100644
--- a/flang/lib/Optimizer/Transforms/CharacterConversion.cpp
+++ b/flang/lib/Optimizer/Transforms/CharacterConversion.cpp
@@ -9,8 +9,8 @@
 #include "flang/Optimizer/Dialect/FIRDialect.h"
 #include "flang/Optimizer/Dialect/FIROps.h"
 #include "flang/Optimizer/Dialect/FIRType.h"
-#include "flang/Optimizer/Support/FIRContext.h"
-#include "flang/Optimizer/Support/KindMapping.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "flang/Optimizer/Transforms/Passes.h"
 #include "mlir/Dialect/Affine/IR/AffineOps.h"
 #include "mlir/Dialect/Func/IR/FuncOps.h"
diff --git a/flang/lib/Optimizer/Transforms/ControlFlowConverter.cpp b/flang/lib/Optimizer/Transforms/ControlFlowConverter.cpp
index d13b488cfda5e..631989a05b57c 100644
--- a/flang/lib/Optimizer/Transforms/ControlFlowConverter.cpp
+++ b/flang/lib/Optimizer/Transforms/ControlFlowConverter.cpp
@@ -9,9 +9,9 @@
 #include "flang/Optimizer/Dialect/FIRDialect.h"
 #include "flang/Optimizer/Dialect/FIROps.h"
 #include "flang/Optimizer/Dialect/FIROpsSupport.h"
-#include "flang/Optimizer/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "flang/Optimizer/Support/InternalNames.h"
-#include "flang/Optimizer/Support/KindMapping.h"
 #include "flang/Optimizer/Support/TypeCode.h"
 #include "flang/Optimizer/Transforms/Passes.h"
 #include "flang/Runtime/derived-api.h"
diff --git a/flang/lib/Optimizer/Transforms/PolymorphicOpConversion.cpp b/flang/lib/Optimizer/Transforms/PolymorphicOpConversion.cpp
index 580e3ca2191a6..f7ee2c19d45ac 100644
--- a/flang/lib/Optimizer/Transforms/PolymorphicOpConversion.cpp
+++ b/flang/lib/Optimizer/Transforms/PolymorphicOpConversion.cpp
@@ -9,9 +9,9 @@
 #include "flang/Optimizer/Dialect/FIRDialect.h"
 #include "flang/Optimizer/Dialect/FIROps.h"
 #include "flang/Optimizer/Dialect/FIROpsSupport.h"
-#include "flang/Optimizer/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "flang/Optimizer/Support/InternalNames.h"
-#include "flang/Optimizer/Support/KindMapping.h"
 #include "flang/Optimizer/Support/TypeCode.h"
 #include "flang/Optimizer/Transforms/Passes.h"
 #include "flang/Runtime/derived-api.h"
diff --git a/flang/lib/Optimizer/Transforms/SimplifyIntrinsics.cpp b/flang/lib/Optimizer/Transforms/SimplifyIntrinsics.cpp
index 581ba006fea40..244a462cb7225 100644
--- a/flang/lib/Optimizer/Transforms/SimplifyIntrinsics.cpp
+++ b/flang/lib/Optimizer/Transforms/SimplifyIntrinsics.cpp
@@ -29,8 +29,8 @@
 #include "flang/Optimizer/Builder/Todo.h"
 #include "flang/Optimizer/Dialect/FIROps.h"
 #include "flang/Optimizer/Dialect/FIRType.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
 #include "flang/Optimizer/HLFIR/HLFIRDialect.h"
-#include "flang/Optimizer/Support/FIRContext.h"
 #include "flang/Optimizer/Transforms/Passes.h"
 #include "flang/Runtime/entry-names.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
diff --git a/flang/lib/Optimizer/Transforms/StackArrays.cpp b/flang/lib/Optimizer/Transforms/StackArrays.cpp
index c0a981e5f5ee6..60a30d2d1ef64 100644
--- a/flang/lib/Optimizer/Transforms/StackArrays.cpp
+++ b/flang/lib/Optimizer/Transforms/StackArrays.cpp
@@ -12,7 +12,7 @@
 #include "flang/Optimizer/Dialect/FIRDialect.h"
 #include "flang/Optimizer/Dialect/FIROps.h"
 #include "flang/Optimizer/Dialect/FIRType.h"
-#include "flang/Optimizer/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
 #include "flang/Optimizer/Transforms/Passes.h"
 #include "mlir/Analysis/DataFlow/ConstantPropagationAnalysis.h"
 #include "mlir/Analysis/DataFlow/DeadCodeAnalysis.h"
diff --git a/flang/lib/Semantics/check-call.cpp b/flang/lib/Semantics/check-call.cpp
index 275673e6e5ea8..e4b65fc8adfe4 100644
--- a/flang/lib/Semantics/check-call.cpp
+++ b/flang/lib/Semantics/check-call.cpp
@@ -296,16 +296,14 @@ static void CheckExplicitDataArg(const characteristics::DummyDataObject &dummy,
             "Actual argument associated with TYPE(*) %s may not have type-bound procedure '%s'"_err_en_US,
             dummyName, tbp->name());
       }
-      const auto &finals{
-          derived->typeSymbol().get<DerivedTypeDetails>().finals()};
+      auto finals{FinalsForDerivedTypeInstantiation(*derived)};
       if (!finals.empty()) { // 15.5.2.4(2)
+        SourceName name{finals.front()->name()};
         if (auto *msg{messages.Say(
                 "Actual argument associated with TYPE(*) %s may not have derived type '%s' with FINAL subroutine '%s'"_err_en_US,
-                dummyName, derived->typeSymbol().name(),
-                finals.begin()->first)}) {
-          msg->Attach(finals.begin()->first,
-              "FINAL subroutine '%s' in derived type '%s'"_en_US,
-              finals.begin()->first, derived->typeSymbol().name());
+                dummyName, derived->typeSymbol().name(), name)}) {
+          msg->Attach(name, "FINAL subroutine '%s' in derived type '%s'"_en_US,
+              name, derived->typeSymbol().name());
         }
       }
     }
diff --git a/flang/lib/Semantics/check-declarations.cpp b/flang/lib/Semantics/check-declarations.cpp
index 540f004863f48..b7c82a4942636 100644
--- a/flang/lib/Semantics/check-declarations.cpp
+++ b/flang/lib/Semantics/check-declarations.cpp
@@ -254,7 +254,9 @@ void CheckHelper::Check(const Symbol &symbol) {
   if (symbol.attrs().test(Attr::VOLATILE)) {
     CheckVolatile(symbol, derived);
   }
-  CheckBindC(symbol);
+  if (symbol.attrs().test(Attr::BIND_C)) {
+    CheckBindC(symbol);
+  }
   CheckGlobalName(symbol);
   if (isDone) {
     return; // following checks do not apply
@@ -430,7 +432,9 @@ void CheckHelper::Check(const Symbol &symbol) {
 
 void CheckHelper::CheckCommonBlock(const Symbol &symbol) {
   CheckGlobalName(symbol);
-  CheckBindC(symbol);
+  if (symbol.attrs().test(Attr::BIND_C)) {
+    CheckBindC(symbol);
+  }
 }
 
 void CheckHelper::CheckBindCFunctionResult(const Symbol &symbol) { // C1553
@@ -2218,13 +2222,16 @@ void CheckHelper::CheckGlobalName(const Symbol &symbol) {
 }
 
 void CheckHelper::CheckBindC(const Symbol &symbol) {
-  if (!symbol.attrs().test(Attr::BIND_C)) {
-    return;
+  bool isExplicitBindC{symbol.attrs().test(Attr::BIND_C)};
+  if (isExplicitBindC) {
+    CheckConflicting(symbol, Attr::BIND_C, Attr::PARAMETER);
+    CheckConflicting(symbol, Attr::BIND_C, Attr::ELEMENTAL);
+  } else {
+    // symbol must be interoperable (e.g., dummy argument of interoperable
+    // procedure interface) but is not itself BIND(C).
   }
-  CheckConflicting(symbol, Attr::BIND_C, Attr::PARAMETER);
-  CheckConflicting(symbol, Attr::BIND_C, Attr::ELEMENTAL);
   if (const std::string * bindName{symbol.GetBindName()};
-      bindName) { // BIND(C,NAME=...)
+      bindName) { // has a binding name
     if (!bindName->empty()) {
       bool ok{bindName->front() == '_' || parser::IsLetter(bindName->front())};
       for (char ch : *bindName) {
@@ -2237,7 +2244,7 @@ void CheckHelper::CheckBindC(const Symbol &symbol) {
       }
     }
   }
-  if (symbol.GetIsExplicitBindName()) { // C1552, C1529
+  if (symbol.GetIsExplicitBindName()) { // BIND(C,NAME=...); C1552, C1529
     auto defClass{ClassifyProcedure(symbol)};
     if (IsProcedurePointer(symbol)) {
       messages_.Say(symbol.name(),
@@ -2256,33 +2263,75 @@ void CheckHelper::CheckBindC(const Symbol &symbol) {
       context_.SetError(symbol);
     }
   }
-  if (symbol.detailsIf<ObjectEntityDetails>()) {
-    if (!symbol.owner().IsModule()) {
+  if (const auto *object{symbol.detailsIf<ObjectEntityDetails>()}) {
+    if (isExplicitBindC && !symbol.owner().IsModule()) {
       messages_.Say(symbol.name(),
           "A variable with BIND(C) attribute may only appear in the specification part of a module"_err_en_US);
       context_.SetError(symbol);
     }
-    if (auto extents{evaluate::GetConstantExtents(foldingContext_, symbol)};
-        extents && evaluate::GetSize(*extents) == 0) {
-      SayWithDeclaration(symbol, symbol.name(),
-          "Interoperable array must have at least one element"_err_en_US);
-    }
-    if (const auto *type{symbol.GetType()}) {
-      if (const auto *derived{type->AsDerived()}) {
-        if (!derived->typeSymbol().attrs().test(Attr::BIND_C)) {
-          if (auto *msg{messages_.Say(symbol.name(),
-                  "The derived type of a BIND(C) object must also be BIND(C)"_err_en_US)}) {
-            msg->Attach(
-                derived->typeSymbol().name(), "Non-interoperable type"_en_US);
+    auto shape{evaluate::GetShape(foldingContext_, symbol)};
+    if (shape) {
+      if (evaluate::GetRank(*shape) == 0) { // 18.3.4
+        if (isExplicitBindC && IsAllocatableOrPointer(symbol)) {
+          messages_.Say(symbol.name(),
+              "A scalar interoperable variable may not be ALLOCATABLE or POINTER"_err_en_US);
+          context_.SetError(symbol);
+        }
+      } else { // 18.3.5
+        if (auto extents{
+                evaluate::AsConstantExtents(foldingContext_, *shape)}) {
+          if (evaluate::GetSize(*extents) == 0) {
+            SayWithDeclaration(symbol, symbol.name(),
+                "Interoperable array must have at least one element"_err_en_US);
+            context_.SetError(symbol);
           }
+        } else if ((isExplicitBindC || symbol.attrs().test(Attr::VALUE)) &&
+            !evaluate::IsExplicitShape(symbol) && !object->IsAssumedSize()) {
+          SayWithDeclaration(symbol, symbol.name(),
+              "BIND(C) array must have explicit shape or be assumed-size unless a dummy argument without the VALUE attribute"_err_en_US);
           context_.SetError(symbol);
         }
-      } else if (!IsInteroperableIntrinsicType(*type)) {
+      }
+    }
+    if (const auto *type{symbol.GetType()}) {
+      const auto *derived{type->AsDerived()};
+      if (derived && !derived->typeSymbol().attrs().test(Attr::BIND_C)) {
+        if (auto *msg{messages_.Say(symbol.name(),
+                "The derived type of a BIND(C) object must also be BIND(C)"_err_en_US)}) {
+          msg->Attach(
+              derived->typeSymbol().name(), "Non-interoperable type"_en_US);
+        }
+        context_.SetError(symbol);
+      }
+      if (type->IsAssumedType() || IsAssumedLengthCharacter(symbol)) {
+        // ok
+      } else if (IsAllocatableOrPointer(symbol) &&
+          type->category() == DeclTypeSpec::Character &&
+          type->characterTypeSpec().length().isDeferred()) {
+        // ok; F'2018 18.3.6 p2(6)
+      } else if (derived || IsInteroperableIntrinsicType(*type)) {
+        // F'2018 18.3.6 p2(4,5)
+      } else if (type->category() == DeclTypeSpec::Logical && IsDummy(symbol) &&
+          evaluate::GetRank(*shape) == 0) {
+        // Special exception: LOGICAL scalar dummy arguments can be converted
+        // before a call -- & after if not INTENT(IN) -- without loss of
+        // information, and are accepted by some older compilers.
+        messages_.Say(symbol.name(),
+            "A BIND(C) LOGICAL dummy argument should have the interoperable KIND=C_BOOL"_port_en_US);
+      } else if (symbol.attrs().test(Attr::VALUE)) {
+        messages_.Say(symbol.name(),
+            "A BIND(C) VALUE dummy argument must have an interoperable type"_err_en_US);
+        context_.SetError(symbol);
+      } else {
         messages_.Say(symbol.name(),
             "A BIND(C) object must have an interoperable type"_err_en_US);
         context_.SetError(symbol);
       }
     }
+    if (IsOptional(symbol) && !symbol.attrs().test(Attr::VALUE)) {
+      messages_.Say(symbol.name(),
+          "An interoperable procedure with an OPTIONAL dummy argument might not be portable"_port_en_US);
+    }
   } else if (const auto *proc{symbol.detailsIf<ProcEntityDetails>()}) {
     if (!proc->procInterface() ||
         !proc->procInterface()->attrs().test(Attr::BIND_C)) {
@@ -2290,6 +2339,16 @@ void CheckHelper::CheckBindC(const Symbol &symbol) {
           "An interface name with BIND attribute must be specified if the BIND attribute is specified in a procedure declaration statement"_err_en_US);
       context_.SetError(symbol);
     }
+  } else if (const auto *subp{symbol.detailsIf<SubprogramDetails>()}) {
+    for (const Symbol *dummy : subp->dummyArgs()) {
+      if (dummy) {
+        CheckBindC(*dummy);
+      } else {
+        messages_.Say(symbol.name(),
+            "A subprogram interface with the BIND attribute may not have an alternate return argument"_err_en_US);
+        context_.SetError(symbol);
+      }
+    }
   } else if (const auto *derived{symbol.detailsIf<DerivedTypeDetails>()}) {
     if (derived->sequence()) { // C1801
       messages_.Say(symbol.name(),
diff --git a/flang/lib/Semantics/check-omp-structure.cpp b/flang/lib/Semantics/check-omp-structure.cpp
index b30ea418b1e7e..cffdabf8441d0 100644
--- a/flang/lib/Semantics/check-omp-structure.cpp
+++ b/flang/lib/Semantics/check-omp-structure.cpp
@@ -1981,7 +1981,13 @@ void OmpStructureChecker::CheckReductionTypeList(
   CheckIntentInPointerAndDefinable(
       ompObjectList, llvm::omp::Clause::OMPC_reduction);
   CheckReductionArraySection(ompObjectList);
-  CheckMultipleAppearanceAcrossContext(ompObjectList);
+  // If this is a worksharing construct then ensure the reduction variable
+  // is not private in the parallel region that it binds to.
+  OmpDirectiveSet workshareSet{llvm::omp::Directive::OMPD_do,
+      llvm::omp::Directive::OMPD_sections, llvm::omp::Directive::OMPD_do_simd};
+  if (workshareSet.test(GetContext().directive)) {
+    CheckSharedBindingInOuterContext(ompObjectList);
+  }
 }
 
 void OmpStructureChecker::CheckIntentInPointerAndDefinable(
@@ -2026,7 +2032,7 @@ void OmpStructureChecker::CheckReductionArraySection(
   }
 }
 
-void OmpStructureChecker::CheckMultipleAppearanceAcrossContext(
+void OmpStructureChecker::CheckSharedBindingInOuterContext(
     const parser::OmpObjectList &redObjectList) {
   //  TODO: Verify the assumption here that the immediately enclosing region is
   //  the parallel region to which the worksharing construct having reduction
diff --git a/flang/lib/Semantics/check-omp-structure.h b/flang/lib/Semantics/check-omp-structure.h
index 2751cdd6fbe98..992531a586db8 100644
--- a/flang/lib/Semantics/check-omp-structure.h
+++ b/flang/lib/Semantics/check-omp-structure.h
@@ -259,7 +259,7 @@ class OmpStructureChecker
       const parser::OmpObjectList &, const llvm::omp::Clause);
   void CheckArraySection(const parser::ArrayElement &arrayElement,
       const parser::Name &name, const llvm::omp::Clause clause);
-  void CheckMultipleAppearanceAcrossContext(
+  void CheckSharedBindingInOuterContext(
       const parser::OmpObjectList &ompObjectList);
   const parser::OmpObjectList *GetOmpObjectList(const parser::OmpClause &);
   void CheckPredefinedAllocatorRestriction(const parser::CharBlock &source,
diff --git a/flang/lib/Semantics/check-select-type.cpp b/flang/lib/Semantics/check-select-type.cpp
index d1c867c253448..c67248ba62407 100644
--- a/flang/lib/Semantics/check-select-type.cpp
+++ b/flang/lib/Semantics/check-select-type.cpp
@@ -254,16 +254,16 @@ void SelectTypeChecker::Enter(const parser::SelectTypeConstruct &construct) {
       std::get<parser::Statement<parser::SelectTypeStmt>>(construct.t)};
   const auto &selectType{selectTypeStmt.statement};
   const auto &unResolvedSel{std::get<parser::Selector>(selectType.t)};
-  const auto *selector{GetExprFromSelector(unResolvedSel)};
-
-  if (!selector) {
-    return; // expression semantics failed on Selector
-  }
-  if (auto exprType{selector->GetType()}) {
-    const auto &typeCaseList{
-        std::get<std::list<parser::SelectTypeConstruct::TypeCase>>(
-            construct.t)};
-    TypeCaseValues{context_, *exprType}.Check(typeCaseList);
+  if (const auto *selector{GetExprFromSelector(unResolvedSel)}) {
+    if (IsProcedure(*selector)) {
+      context_.Say(
+          selectTypeStmt.source, "Selector may not be a procedure"_err_en_US);
+    } else if (auto exprType{selector->GetType()}) {
+      const auto &typeCaseList{
+          std::get<std::list<parser::SelectTypeConstruct::TypeCase>>(
+              construct.t)};
+      TypeCaseValues{context_, *exprType}.Check(typeCaseList);
+    }
   }
 }
 
diff --git a/flang/lib/Semantics/compute-offsets.cpp b/flang/lib/Semantics/compute-offsets.cpp
index 8789f212feacf..c44660925622b 100644
--- a/flang/lib/Semantics/compute-offsets.cpp
+++ b/flang/lib/Semantics/compute-offsets.cpp
@@ -321,11 +321,12 @@ auto ComputeOffsetsHelper::GetSizeAndAlignment(
     const Symbol &symbol, bool entire) -> SizeAndAlignment {
   auto &targetCharacteristics{context_.targetCharacteristics()};
   if (IsDescriptor(symbol)) {
-    const auto *derived{
-        evaluate::GetDerivedTypeSpec(evaluate::DynamicType::From(symbol))};
+    auto dyType{evaluate::DynamicType::From(symbol)};
+    const auto *derived{evaluate::GetDerivedTypeSpec(dyType)};
     int lenParams{derived ? CountLenParameters(*derived) : 0};
+    bool needAddendum{derived || (dyType && dyType->IsUnlimitedPolymorphic())};
     std::size_t size{runtime::Descriptor::SizeInBytes(
-        symbol.Rank(), derived != nullptr, lenParams)};
+        symbol.Rank(), needAddendum, lenParams)};
     return {size, targetCharacteristics.descriptorAlignment()};
   }
   if (IsProcedurePointer(symbol)) {
diff --git a/flang/lib/Semantics/data-to-inits.cpp b/flang/lib/Semantics/data-to-inits.cpp
index 52191edbab6e1..959c74b62d7cb 100644
--- a/flang/lib/Semantics/data-to-inits.cpp
+++ b/flang/lib/Semantics/data-to-inits.cpp
@@ -569,7 +569,8 @@ static void PopulateWithComponentDefaults(SymbolDataInitialization &init,
               if (auto extents{evaluate::GetConstantExtents(
                       foldingContext, component)}) {
                 if (auto extant{init.image.AsConstant(foldingContext, *dyType,
-                        *extents, false /*don't pad*/, componentOffset)}) {
+                        std::nullopt, *extents, false /*don't pad*/,
+                        componentOffset)}) {
                   initialized = !(*extant == *object->init());
                 }
               }
@@ -905,8 +906,8 @@ void ConstructInitializer(const Symbol &symbol,
       }
     } else if (auto symbolType{evaluate::DynamicType::From(symbol)}) {
       if (auto extents{evaluate::GetConstantExtents(context, symbol)}) {
-        mutableObject.set_init(
-            initialization.image.AsConstant(context, *symbolType, *extents));
+        mutableObject.set_init(initialization.image.AsConstant(
+            context, *symbolType, std::nullopt, *extents));
       } else {
         exprAnalyzer.Say(symbol.name(),
             "internal: unknown shape for '%s' while constructing initializer from DATA"_err_en_US,
diff --git a/flang/lib/Semantics/definable.cpp b/flang/lib/Semantics/definable.cpp
index aa9246fb223e0..613a62cc4986b 100644
--- a/flang/lib/Semantics/definable.cpp
+++ b/flang/lib/Semantics/definable.cpp
@@ -228,8 +228,7 @@ std::optional<parser::Message> WhyNotDefinable(parser::CharBlock at,
             while (spec) {
               bool anyElemental{false};
               const Symbol *anyRankMatch{nullptr};
-              for (const auto &[_, ref] :
-                  spec->typeSymbol().get<DerivedTypeDetails>().finals()) {
+              for (auto ref : FinalsForDerivedTypeInstantiation(*spec)) {
                 const Symbol &ultimate{ref->GetUltimate()};
                 anyElemental |= ultimate.attrs().test(Attr::ELEMENTAL);
                 if (const auto *subp{ultimate.detailsIf<SubprogramDetails>()}) {
diff --git a/flang/lib/Semantics/expression.cpp b/flang/lib/Semantics/expression.cpp
index b5f3a07679fd2..14f2b0f0f7be9 100644
--- a/flang/lib/Semantics/expression.cpp
+++ b/flang/lib/Semantics/expression.cpp
@@ -596,23 +596,24 @@ int ExpressionAnalyzer::AnalyzeKindParam(
   if (!kindParam) {
     return defaultKind;
   }
-  return common::visit(
+  std::int64_t kind{common::visit(
       common::visitors{
-          [](std::uint64_t k) { return static_cast<int>(k); },
+          [](std::uint64_t k) { return static_cast<std::int64_t>(k); },
           [&](const parser::Scalar<
               parser::Integer<parser::Constant<parser::Name>>> &n) {
             if (MaybeExpr ie{Analyze(n)}) {
-              if (std::optional<std::int64_t> i64{ToInt64(*ie)}) {
-                int iv = *i64;
-                if (iv == *i64) {
-                  return iv;
-                }
-              }
+              return ToInt64(*ie).value_or(defaultKind);
             }
-            return defaultKind;
+            return static_cast<std::int64_t>(defaultKind);
           },
       },
-      kindParam->u);
+      kindParam->u)};
+  if (kind != static_cast<int>(kind)) {
+    Say("Unsupported type kind value (%jd)"_err_en_US,
+        static_cast<std::intmax_t>(kind));
+    kind = defaultKind;
+  }
+  return static_cast<int>(kind);
 }
 
 // Common handling of parser::IntLiteralConstant and SignedIntLiteralConstant
@@ -657,7 +658,7 @@ struct IntTypeVisitor {
   }
   ExpressionAnalyzer &analyzer;
   parser::CharBlock digits;
-  int kind;
+  std::int64_t kind;
   bool isDefaultKind;
   bool isNegated;
 };
diff --git a/flang/lib/Semantics/resolve-names.cpp b/flang/lib/Semantics/resolve-names.cpp
index 637d133798a51..ff5ae4af866dd 100644
--- a/flang/lib/Semantics/resolve-names.cpp
+++ b/flang/lib/Semantics/resolve-names.cpp
@@ -2435,13 +2435,15 @@ void ScopeHandler::ApplyImplicitRules(
 }
 
 // Extension: Allow forward references to scalar integer dummy arguments
-// to appear in specification expressions under IMPLICIT NONE(TYPE) when
-// what would otherwise have been their implicit type is default INTEGER.
+// or variables in COMMON to appear in specification expressions under
+// IMPLICIT NONE(TYPE) when what would otherwise have been their implicit
+// type is default INTEGER.
 bool ScopeHandler::ImplicitlyTypeForwardRef(Symbol &symbol) {
-  if (!inSpecificationPart_ || context().HasError(symbol) || !IsDummy(symbol) ||
+  if (!inSpecificationPart_ || context().HasError(symbol) ||
+      !(IsDummy(symbol) || FindCommonBlockContaining(symbol)) ||
       symbol.Rank() != 0 ||
       !context().languageFeatures().IsEnabled(
-          common::LanguageFeature::ForwardRefDummyImplicitNone)) {
+          common::LanguageFeature::ForwardRefImplicitNone)) {
     return false;
   }
   const DeclTypeSpec *type{
@@ -2456,11 +2458,11 @@ bool ScopeHandler::ImplicitlyTypeForwardRef(Symbol &symbol) {
   if (!ConvertToObjectEntity(symbol)) {
     return false;
   }
-  // TODO: check no INTENT(OUT)?
+  // TODO: check no INTENT(OUT) if dummy?
   if (context().languageFeatures().ShouldWarn(
-          common::LanguageFeature::ForwardRefDummyImplicitNone)) {
+          common::LanguageFeature::ForwardRefImplicitNone)) {
     Say(symbol.name(),
-        "Dummy argument '%s' was used without being explicitly typed"_warn_en_US,
+        "'%s' was used without (or before) being explicitly typed"_warn_en_US,
         symbol.name());
   }
   symbol.set(Symbol::Flag::Implicit);
@@ -2639,13 +2641,13 @@ bool ScopeHandler::CheckPossibleBadForwardRef(const Symbol &symbol) {
       context().SetError(symbol);
       return true;
     }
-    if (IsDummy(symbol) && isImplicitNoneType() &&
-        symbol.test(Symbol::Flag::Implicit) && !context().HasError(symbol)) {
-      // Dummy was implicitly typed despite IMPLICIT NONE(TYPE) in
+    if ((IsDummy(symbol) || FindCommonBlockContaining(symbol)) &&
+        isImplicitNoneType() && symbol.test(Symbol::Flag::Implicit) &&
+        !context().HasError(symbol)) {
+      // Dummy or COMMON was implicitly typed despite IMPLICIT NONE(TYPE) in
       // ApplyImplicitRules() due to use in a specification expression,
       // and no explicit type declaration appeared later.
-      Say(symbol.name(),
-          "No explicit type declared for dummy argument '%s'"_err_en_US);
+      Say(symbol.name(), "No explicit type declared for '%s'"_err_en_US);
       context().SetError(symbol);
       return true;
     }
@@ -3998,9 +4000,8 @@ Symbol &SubprogramVisitor::PushSubprogramScope(const parser::Name &name,
     }
     if (isGeneric()) {
       Symbol &genericSymbol{GetGenericSymbol()};
-      if (genericSymbol.has<GenericDetails>()) {
-        genericSymbol.get<GenericDetails>().AddSpecificProc(
-            *symbol, name.source);
+      if (auto *details{genericSymbol.detailsIf<GenericDetails>()}) {
+        details->AddSpecificProc(*symbol, name.source);
       } else {
         CHECK(context().HasError(genericSymbol));
       }
@@ -5145,8 +5146,8 @@ void DeclarationVisitor::Post(
       procedure = NoteInterfaceName(procedureName);
     }
     if (procedure) {
-      if (auto *s{
-              MakeTypeSymbol(bindingName, ProcBindingDetails{*procedure})}) {
+      const Symbol &bindTo{BypassGeneric(*procedure)};
+      if (auto *s{MakeTypeSymbol(bindingName, ProcBindingDetails{bindTo})}) {
         SetPassNameOn(*s);
         if (GetAttrs().test(Attr::DEFERRED)) {
           context().SetError(*s);
@@ -5163,7 +5164,11 @@ void DeclarationVisitor::CheckBindings(
     auto &bindingName{std::get<parser::Name>(declaration.t)};
     if (Symbol * binding{FindInScope(bindingName)}) {
       if (auto *details{binding->detailsIf<ProcBindingDetails>()}) {
-        const Symbol *procedure{FindSubprogram(details->symbol())};
+        const Symbol &ultimate{details->symbol().GetUltimate()};
+        const Symbol &procedure{BypassGeneric(ultimate)};
+        if (&procedure != &ultimate) {
+          details->ReplaceSymbol(procedure);
+        }
         if (!CanBeTypeBoundProc(procedure)) {
           if (details->symbol().name() != binding->name()) {
             Say(binding->name(),
@@ -7475,16 +7480,17 @@ void ResolveNamesVisitor::CreateGeneric(const parser::GenericSpec &x) {
   if (existing) {
     Symbol &ultimate{existing->GetUltimate()};
     if (auto *existingGeneric{ultimate.detailsIf<GenericDetails>()}) {
-      if (const auto *existingUse{existing->detailsIf<UseDetails>()}) {
-        // Create a local copy of a use associated generic so that
-        // it can be locally extended without corrupting the original.
-        genericDetails.CopyFrom(*existingGeneric);
-        if (existingGeneric->specific()) {
-          genericDetails.set_specific(*existingGeneric->specific());
-        }
-        AddGenericUse(genericDetails, existing->name(), existingUse->symbol());
-      } else if (&existing->owner() == &currScope()) {
-        if (existing == &ultimate) {
+      if (&existing->owner() == &currScope()) {
+        if (const auto *existingUse{existing->detailsIf<UseDetails>()}) {
+          // Create a local copy of a use associated generic so that
+          // it can be locally extended without corrupting the original.
+          genericDetails.CopyFrom(*existingGeneric);
+          if (existingGeneric->specific()) {
+            genericDetails.set_specific(*existingGeneric->specific());
+          }
+          AddGenericUse(
+              genericDetails, existing->name(), existingUse->symbol());
+        } else if (existing == &ultimate) {
           // Extending an extant generic in the same scope
           info.Resolve(existing);
           return;
@@ -7492,6 +7498,8 @@ void ResolveNamesVisitor::CreateGeneric(const parser::GenericSpec &x) {
           // Host association of a generic is handled elsewhere
           CHECK(existing->has<HostAssocDetails>());
         }
+      } else {
+        // Create a new generic for this scope.
       }
     } else if (ultimate.has<SubprogramDetails>() ||
         ultimate.has<SubprogramNameDetails>()) {
diff --git a/flang/lib/Semantics/runtime-type-info.cpp b/flang/lib/Semantics/runtime-type-info.cpp
index 153b9841e796e..29f63524b5c07 100644
--- a/flang/lib/Semantics/runtime-type-info.cpp
+++ b/flang/lib/Semantics/runtime-type-info.cpp
@@ -573,12 +573,11 @@ const Symbol *RuntimeTableBuilder::DescribeType(Scope &dtScope) {
       // do not (the runtime will call all of them).
       std::map<int, evaluate::StructureConstructor> specials{
           DescribeSpecialGenerics(dtScope, dtScope, derivedTypeSpec)};
-      const DerivedTypeDetails &dtDetails{dtSymbol->get<DerivedTypeDetails>()};
-      for (const auto &pair : dtDetails.finals()) {
-        DescribeSpecialProc(specials, *pair.second, false /*!isAssignment*/,
-            true, std::nullopt, nullptr, derivedTypeSpec);
-      }
       if (derivedTypeSpec) {
+        for (auto &ref : FinalsForDerivedTypeInstantiation(*derivedTypeSpec)) {
+          DescribeSpecialProc(specials, *ref, false /*!isAssignment*/, true,
+              std::nullopt, nullptr, derivedTypeSpec);
+        }
         IncorporateDefinedIoGenericInterfaces(specials,
             GenericKind::DefinedIo::ReadFormatted, &scope, derivedTypeSpec);
         IncorporateDefinedIoGenericInterfaces(specials,
@@ -1112,15 +1111,18 @@ void RuntimeTableBuilder::DescribeSpecialProc(
       }
     } else { // user defined derived type I/O
       CHECK(proc->dummyArguments.size() >= 4);
+      const auto *ddo{std::get_if<evaluate::characteristics::DummyDataObject>(
+          &proc->dummyArguments[0].u)};
+      if (!ddo) {
+        return;
+      }
       if (derivedTypeSpec &&
-          !std::get<evaluate::characteristics::DummyDataObject>(
-              proc->dummyArguments[0].u)
-               .type.type()
-               .IsTkCompatibleWith(evaluate::DynamicType{*derivedTypeSpec})) {
+          !ddo->type.type().IsTkCompatibleWith(
+              evaluate::DynamicType{*derivedTypeSpec})) {
         // Defined I/O specific procedure is not for this derived type.
         return;
       }
-      if (binding) {
+      if (ddo->type.type().IsPolymorphic()) {
         isArgDescriptorSet |= 1;
       }
       switch (io.value()) {
diff --git a/flang/lib/Semantics/tools.cpp b/flang/lib/Semantics/tools.cpp
index 41a5ac215826a..b4178975f567b 100644
--- a/flang/lib/Semantics/tools.cpp
+++ b/flang/lib/Semantics/tools.cpp
@@ -611,16 +611,19 @@ bool IsOrContainsEventOrLockComponent(const Symbol &original) {
 }
 
 // Check this symbol suitable as a type-bound procedure - C769
-bool CanBeTypeBoundProc(const Symbol *symbol) {
-  if (!symbol || IsDummy(*symbol) || IsProcedurePointer(*symbol)) {
+bool CanBeTypeBoundProc(const Symbol &symbol) {
+  if (IsDummy(symbol) || IsProcedurePointer(symbol)) {
     return false;
-  } else if (symbol->has<SubprogramNameDetails>()) {
-    return symbol->owner().kind() == Scope::Kind::Module;
-  } else if (auto *details{symbol->detailsIf<SubprogramDetails>()}) {
-    return symbol->owner().kind() == Scope::Kind::Module ||
-        details->isInterface();
-  } else if (const auto *proc{symbol->detailsIf<ProcEntityDetails>()}) {
-    return !symbol->attrs().test(Attr::INTRINSIC) &&
+  } else if (symbol.has<SubprogramNameDetails>()) {
+    return symbol.owner().kind() == Scope::Kind::Module;
+  } else if (auto *details{symbol.detailsIf<SubprogramDetails>()}) {
+    if (details->isInterface()) {
+      return !symbol.attrs().test(Attr::ABSTRACT);
+    } else {
+      return symbol.owner().kind() == Scope::Kind::Module;
+    }
+  } else if (const auto *proc{symbol.detailsIf<ProcEntityDetails>()}) {
+    return !symbol.attrs().test(Attr::INTRINSIC) &&
         proc->HasExplicitInterface();
   } else {
     return false;
@@ -702,6 +705,30 @@ bool IsSeparateModuleProcedureInterface(const Symbol *symbol) {
   return false;
 }
 
+SymbolVector FinalsForDerivedTypeInstantiation(const DerivedTypeSpec &spec) {
+  SymbolVector result;
+  const Symbol &typeSymbol{spec.typeSymbol()};
+  if (const auto *derived{typeSymbol.detailsIf<DerivedTypeDetails>()}) {
+    for (const auto &pair : derived->finals()) {
+      const Symbol &subr{*pair.second};
+      // Errors in FINAL subroutines are caught in CheckFinal
+      // in check-declarations.cpp.
+      if (const auto *subprog{subr.detailsIf<SubprogramDetails>()};
+          subprog && subprog->dummyArgs().size() == 1) {
+        if (const Symbol * arg{subprog->dummyArgs()[0]}) {
+          if (const DeclTypeSpec * type{arg->GetType()}) {
+            if (type->category() == DeclTypeSpec::TypeDerived &&
+                evaluate::AreSameDerivedType(spec, type->derivedTypeSpec())) {
+              result.emplace_back(subr);
+            }
+          }
+        }
+      }
+    }
+  }
+  return result;
+}
+
 bool IsFinalizable(
     const Symbol &symbol, std::set<const DerivedTypeSpec *> *inProgress) {
   if (IsPointer(symbol)) {
@@ -720,7 +747,7 @@ bool IsFinalizable(
 
 bool IsFinalizable(const DerivedTypeSpec &derived,
     std::set<const DerivedTypeSpec *> *inProgress) {
-  if (!derived.typeSymbol().get<DerivedTypeDetails>().finals().empty()) {
+  if (!FinalsForDerivedTypeInstantiation(derived).empty()) {
     return true;
   }
   std::set<const DerivedTypeSpec *> basis;
@@ -742,14 +769,12 @@ bool IsFinalizable(const DerivedTypeSpec &derived,
 }
 
 bool HasImpureFinal(const DerivedTypeSpec &derived) {
-  if (const auto *details{
-          derived.typeSymbol().detailsIf<DerivedTypeDetails>()}) {
-    const auto &finals{details->finals()};
-    return std::any_of(finals.begin(), finals.end(),
-        [](const auto &x) { return !IsPureProcedure(*x.second); });
-  } else {
-    return false;
+  for (auto ref : FinalsForDerivedTypeInstantiation(derived)) {
+    if (!IsPureProcedure(*ref)) {
+      return true;
+    }
   }
+  return false;
 }
 
 bool IsAssumedLengthCharacter(const Symbol &symbol) {
diff --git a/flang/lib/Semantics/type.cpp b/flang/lib/Semantics/type.cpp
index afc2baaaeedef..d895f01dba2ea 100644
--- a/flang/lib/Semantics/type.cpp
+++ b/flang/lib/Semantics/type.cpp
@@ -188,7 +188,7 @@ bool DerivedTypeSpec::HasDefaultInitialization(bool ignoreAllocatable) const {
 }
 
 bool DerivedTypeSpec::HasDestruction() const {
-  if (!typeSymbol().get<DerivedTypeDetails>().finals().empty()) {
+  if (!FinalsForDerivedTypeInstantiation(*this).empty()) {
     return true;
   }
   DirectComponentIterator components{*this};
@@ -366,7 +366,7 @@ void DerivedTypeSpec::Instantiate(Scope &containingScope) {
     }
     newScope.set_instantiationContext(contextMessage);
   }
-  // Instantiate every non-parameter symbol from the original derived
+  // Instantiate nearly every non-parameter symbol from the original derived
   // type's scope into the new instance.
   auto restorer2{foldingContext.messages().SetContext(contextMessage)};
   if (PlumbPDTInstantiationDepth(&containingScope) > 100) {
diff --git a/flang/module/__fortran_ppc_intrinsics.f90 b/flang/module/__fortran_ppc_intrinsics.f90
index 9b83239731020..fff18aa36d2fe 100644
--- a/flang/module/__fortran_ppc_intrinsics.f90
+++ b/flang/module/__fortran_ppc_intrinsics.f90
@@ -52,4 +52,72 @@ end function func_r8r8r8r8
   end interface fnmsub
   public :: fnmsub
 
+! fctid, fctidz, fctiw, fctiwz, fctudz, fctuwz
+  abstract interface
+    elemental real(8) function func_r8r8x(x)
+      real(8), intent(in) :: x
+    end function func_r8r8x
+  end interface
+
+  procedure(func_r8r8x) :: __ppc_fctid
+  interface fctid
+    procedure :: __ppc_fctid
+  end interface fctid
+  public :: fctid
+
+  procedure(func_r8r8x) :: __ppc_fctidz
+  interface fctidz
+    procedure :: __ppc_fctidz
+  end interface fctidz
+  public :: fctidz
+
+  procedure(func_r8r8x) :: __ppc_fctiw
+  interface fctiw
+    procedure :: __ppc_fctiw
+  end interface fctiw
+  public :: fctiw
+
+  procedure(func_r8r8x) :: __ppc_fctiwz
+  interface fctiwz
+    procedure :: __ppc_fctiwz
+  end interface fctiwz
+  public :: fctiwz
+
+  procedure(func_r8r8x) :: __ppc_fctudz
+  interface fctudz
+    procedure :: __ppc_fctudz
+  end interface fctudz
+  public :: fctudz
+
+  procedure(func_r8r8x) :: __ppc_fctuwz
+  interface fctuwz
+    procedure :: __ppc_fctuwz
+  end interface fctuwz
+  public :: fctuwz
+
+! fcfi, fcfid, fcfud
+  abstract interface
+    elemental real(8) function func_r8r8i(i)
+      real(8), intent(in) :: i
+    end function func_r8r8i
+  end interface
+
+  procedure(func_r8r8i) :: __ppc_fcfi
+  interface fcfi
+    procedure :: __ppc_fcfi
+  end interface fcfi
+  public :: fcfi
+
+  procedure(func_r8r8i) :: __ppc_fcfid
+  interface fcfid
+    procedure :: __ppc_fcfid
+  end interface fcfid
+  public :: fcfid
+
+  procedure(func_r8r8i) :: __ppc_fcfud
+  interface fcfud
+    procedure :: __ppc_fcfud
+  end interface fcfud
+  public :: fcfud
+
 end module __Fortran_PPC_intrinsics
diff --git a/flang/module/iso_c_binding.f90 b/flang/module/iso_c_binding.f90
index 0fcebc7bd7514..68035579ec620 100644
--- a/flang/module/iso_c_binding.f90
+++ b/flang/module/iso_c_binding.f90
@@ -64,7 +64,7 @@ module iso_c_binding
     c_double_complex = c_double, &
     c_long_double_complex = c_long_double
 
-  integer, parameter :: c_bool = 1 ! TODO: or default LOGICAL?
+  integer, parameter :: c_bool = 1
   integer, parameter :: c_char = 1
 
   ! C characters with special semantics
diff --git a/flang/module/omp_lib.h b/flang/module/omp_lib.h
index 993b00409af72..d8a9aeb152df5 100644
--- a/flang/module/omp_lib.h
+++ b/flang/module/omp_lib.h
@@ -9,7 +9,7 @@
 !dir$ free
 
   integer, parameter :: omp_integer_kind = selected_int_kind(9) ! 32-bit int
-  integer, parameter :: omp_logical_kind = kind(.true.)
+  integer, parameter :: omp_logical_kind = 1 ! C_BOOL
 
   integer, parameter :: omp_sched_kind = omp_integer_kind
   integer, parameter :: omp_proc_bind_kind = omp_integer_kind
diff --git a/flang/runtime/CMakeLists.txt b/flang/runtime/CMakeLists.txt
index e0c93c220e95a..15da5a20aaec8 100644
--- a/flang/runtime/CMakeLists.txt
+++ b/flang/runtime/CMakeLists.txt
@@ -125,6 +125,7 @@ add_flang_library(FortranRuntime
   io-error.cpp
   io-stmt.cpp
   main.cpp
+  matmul-transpose.cpp
   matmul.cpp
   memory.cpp
   misc-intrinsic.cpp
diff --git a/flang/runtime/allocatable.cpp b/flang/runtime/allocatable.cpp
index 7d4f04cf48106..758c814025b96 100644
--- a/flang/runtime/allocatable.cpp
+++ b/flang/runtime/allocatable.cpp
@@ -41,6 +41,30 @@ void RTNAME(AllocatableInitDerived)(Descriptor &descriptor,
       derivedType, nullptr, rank, nullptr, CFI_attribute_allocatable);
 }
 
+void RTNAME(AllocatableInitIntrinsicForAllocate)(Descriptor &descriptor,
+    TypeCategory category, int kind, int rank, int corank) {
+  if (descriptor.IsAllocated()) {
+    return;
+  }
+  RTNAME(AllocatableInitIntrinsic)(descriptor, category, kind, rank, corank);
+}
+
+void RTNAME(AllocatableInitCharacterForAllocate)(Descriptor &descriptor,
+    SubscriptValue length, int kind, int rank, int corank) {
+  if (descriptor.IsAllocated()) {
+    return;
+  }
+  RTNAME(AllocatableInitCharacter)(descriptor, length, kind, rank, corank);
+}
+
+void RTNAME(AllocatableInitDerivedForAllocate)(Descriptor &descriptor,
+    const typeInfo::DerivedType &derivedType, int rank, int corank) {
+  if (descriptor.IsAllocated()) {
+    return;
+  }
+  RTNAME(AllocatableInitDerived)(descriptor, derivedType, rank, corank);
+}
+
 std::int32_t RTNAME(MoveAlloc)(Descriptor &to, Descriptor &from,
     const typeInfo::DerivedType *derivedType, bool hasStat,
     const Descriptor *errMsg, const char *sourceFile, int sourceLine) {
diff --git a/flang/runtime/assign.cpp b/flang/runtime/assign.cpp
index b737e47f6d372..b4d9061846e9d 100644
--- a/flang/runtime/assign.cpp
+++ b/flang/runtime/assign.cpp
@@ -199,19 +199,17 @@ static void DoScalarDefinedAssignment(const Descriptor &to,
 }
 
 static void DoElementalDefinedAssignment(const Descriptor &to,
-    const Descriptor &from, const typeInfo::SpecialBinding &special) {
+    const Descriptor &from, const typeInfo::DerivedType &derived,
+    const typeInfo::SpecialBinding &special) {
   SubscriptValue toAt[maxRank], fromAt[maxRank];
   to.GetLowerBounds(toAt);
   from.GetLowerBounds(fromAt);
   StaticDescriptor<maxRank, true, 8 /*?*/> statDesc[2];
   Descriptor &toElementDesc{statDesc[0].descriptor()};
   Descriptor &fromElementDesc{statDesc[1].descriptor()};
-  toElementDesc = to;
-  toElementDesc.raw().attribute = CFI_attribute_pointer;
-  toElementDesc.raw().rank = 0;
-  fromElementDesc = from;
-  fromElementDesc.raw().attribute = CFI_attribute_pointer;
-  fromElementDesc.raw().rank = 0;
+  toElementDesc.Establish(derived, nullptr, 0, nullptr, CFI_attribute_pointer);
+  fromElementDesc.Establish(
+      derived, nullptr, 0, nullptr, CFI_attribute_pointer);
   for (std::size_t toElements{to.Elements()}; toElements-- > 0;
        to.IncrementSubscripts(toAt), from.IncrementSubscripts(fromAt)) {
     toElementDesc.set_base_addr(to.Element<char>(toAt));
@@ -268,7 +266,7 @@ static void Assign(
       }
       if (const auto *special{toDerived->FindSpecialBinding(
               typeInfo::SpecialBinding::Which::ElementalAssignment)}) {
-        return DoElementalDefinedAssignment(to, from, *special);
+        return DoElementalDefinedAssignment(to, from, *toDerived, *special);
       }
     }
     if ((flags & NeedFinalization) && toDerived->noFinalizationNeeded()) {
diff --git a/flang/runtime/derived-api.cpp b/flang/runtime/derived-api.cpp
index 806a76ac95753..9b3455d9f0293 100644
--- a/flang/runtime/derived-api.cpp
+++ b/flang/runtime/derived-api.cpp
@@ -87,11 +87,19 @@ static const typeInfo::DerivedType *GetDerivedType(const Descriptor &desc) {
 }
 
 bool RTNAME(SameTypeAs)(const Descriptor &a, const Descriptor &b) {
+  // Unlimited polymorphic with intrinsic dynamic type.
+  if (a.raw().type != CFI_type_struct && a.raw().type != CFI_type_other &&
+      b.raw().type != CFI_type_struct && b.raw().type != CFI_type_other)
+    return a.raw().type == b.raw().type;
+
   const typeInfo::DerivedType *derivedTypeA{GetDerivedType(a)};
   const typeInfo::DerivedType *derivedTypeB{GetDerivedType(b)};
+
+  // No dynamic type in one or both descriptor.
   if (derivedTypeA == nullptr || derivedTypeB == nullptr) {
     return false;
   }
+
   // Exact match of derived type.
   if (derivedTypeA == derivedTypeB) {
     return true;
@@ -102,6 +110,10 @@ bool RTNAME(SameTypeAs)(const Descriptor &a, const Descriptor &b) {
 }
 
 bool RTNAME(ExtendsTypeOf)(const Descriptor &a, const Descriptor &mold) {
+  if (a.raw().type != CFI_type_struct && a.raw().type != CFI_type_other &&
+      mold.raw().type != CFI_type_struct && mold.raw().type != CFI_type_other)
+    return a.raw().type == mold.raw().type;
+
   const typeInfo::DerivedType *derivedTypeA{GetDerivedType(a)};
   const typeInfo::DerivedType *derivedTypeMold{GetDerivedType(mold)};
 
diff --git a/flang/runtime/derived.cpp b/flang/runtime/derived.cpp
index 354de1baddd99..981ddb2a6e9d4 100644
--- a/flang/runtime/derived.cpp
+++ b/flang/runtime/derived.cpp
@@ -20,8 +20,8 @@ int Initialize(const Descriptor &instance, const typeInfo::DerivedType &derived,
   std::size_t elements{instance.Elements()};
   std::size_t byteStride{instance.ElementBytes()};
   int stat{StatOk};
-  // Initialize data components in each element; the per-element iteration
-  // constitutes the inner loops, not outer
+  // Initialize data components in each element; the per-element iterations
+  // constitute the inner loops, not the outer ones
   std::size_t myComponents{componentDesc.Elements()};
   for (std::size_t k{0}; k < myComponents; ++k) {
     const auto &comp{
@@ -36,7 +36,14 @@ int Initialize(const Descriptor &instance, const typeInfo::DerivedType &derived,
         if (comp.genre() == typeInfo::Component::Genre::Automatic) {
           stat = ReturnError(terminator, allocDesc.Allocate(), errMsg, hasStat);
           if (stat == StatOk) {
-            stat = Initialize(allocDesc, derived, terminator, hasStat, errMsg);
+            if (const DescriptorAddendum * addendum{allocDesc.Addendum()}) {
+              if (const auto *derived{addendum->derivedType()}) {
+                if (!derived->noInitializationNeeded()) {
+                  stat = Initialize(
+                      allocDesc, *derived, terminator, hasStat, errMsg);
+                }
+              }
+            }
           }
           if (stat != StatOk) {
             break;
diff --git a/flang/runtime/descriptor-io.cpp b/flang/runtime/descriptor-io.cpp
index c6b57bf5f088d..c51a36b1a7bd1 100644
--- a/flang/runtime/descriptor-io.cpp
+++ b/flang/runtime/descriptor-io.cpp
@@ -7,12 +7,14 @@
 //===----------------------------------------------------------------------===//
 
 #include "descriptor-io.h"
+#include "flang/Common/restorer.h"
 
 namespace Fortran::runtime::io::descr {
 
 // User-defined derived type formatted I/O (maybe)
 std::optional<bool> DefinedFormattedIo(IoStatementState &io,
-    const Descriptor &descriptor, const typeInfo::SpecialBinding &special) {
+    const Descriptor &descriptor, const typeInfo::DerivedType &derived,
+    const typeInfo::SpecialBinding &special) {
   std::optional<DataEdit> peek{io.GetNextDataEdit(0 /*to peek at it*/)};
   if (peek &&
       (peek->descriptor == DataEdit::DefinedDerivedType ||
@@ -32,8 +34,8 @@ std::optional<bool> DefinedFormattedIo(IoStatementState &io,
           ioType, io.mutableModes().inNamelist ? "NAMELIST" : "LISTDIRECTED");
       ioTypeLen = std::strlen(ioType);
     }
-    StaticDescriptor<1, true> statDesc;
-    Descriptor &vListDesc{statDesc.descriptor()};
+    StaticDescriptor<1, true> vListStatDesc;
+    Descriptor &vListDesc{vListStatDesc.descriptor()};
     vListDesc.Establish(TypeCategory::Integer, sizeof(int), nullptr, 1);
     vListDesc.set_base_addr(edit.vList);
     vListDesc.GetDimension(0).SetBounds(1, edit.vListEntries);
@@ -47,6 +49,8 @@ std::optional<bool> DefinedFormattedIo(IoStatementState &io,
       external = &ExternalFileUnit::NewUnit(handler, true);
     }
     ChildIo &child{external->PushChildIo(io)};
+    // Child formatted I/O is nonadvancing by definition (F'2018 12.6.2.4).
+    auto restorer{common::ScopedSet(io.mutableModes().nonAdvancing, true)};
     int unit{external->unitNumber()};
     int ioStat{IostatOk};
     char ioMsg[100];
@@ -57,16 +61,36 @@ std::optional<bool> DefinedFormattedIo(IoStatementState &io,
       // I/O subroutine reads counts towards READ(SIZE=).
       startPos = io.InquirePos();
     }
+    std::size_t numElements{descriptor.Elements()};
+    SubscriptValue subscripts[maxRank];
+    descriptor.GetLowerBounds(subscripts);
     if (special.IsArgDescriptor(0)) {
+      // "dtv" argument is "class(t)", pass a descriptor
       auto *p{special.GetProc<void (*)(const Descriptor &, int &, char *,
           const Descriptor &, int &, char *, std::size_t, std::size_t)>()};
-      p(descriptor, unit, ioType, vListDesc, ioStat, ioMsg, ioTypeLen,
-          sizeof ioMsg);
+      StaticDescriptor<1, true, 10 /*?*/> elementStatDesc;
+      Descriptor &elementDesc{elementStatDesc.descriptor()};
+      elementDesc.Establish(
+          derived, nullptr, 0, nullptr, CFI_attribute_pointer);
+      for (; numElements-- > 0; descriptor.IncrementSubscripts(subscripts)) {
+        elementDesc.set_base_addr(descriptor.Element<char>(subscripts));
+        p(elementDesc, unit, ioType, vListDesc, ioStat, ioMsg, ioTypeLen,
+            sizeof ioMsg);
+        if (ioStat != IostatOk) {
+          break;
+        }
+      }
     } else {
+      // "dtv" argument is "type(t)", pass a raw pointer
       auto *p{special.GetProc<void (*)(const void *, int &, char *,
           const Descriptor &, int &, char *, std::size_t, std::size_t)>()};
-      p(descriptor.raw().base_addr, unit, ioType, vListDesc, ioStat, ioMsg,
-          ioTypeLen, sizeof ioMsg);
+      for (; numElements-- > 0; descriptor.IncrementSubscripts(subscripts)) {
+        p(descriptor.Element<char>(subscripts), unit, ioType, vListDesc, ioStat,
+            ioMsg, ioTypeLen, sizeof ioMsg);
+        if (ioStat != IostatOk) {
+          break;
+        }
+      }
     }
     handler.Forward(ioStat, ioMsg, sizeof ioMsg);
     external->PopChildIo(child);
@@ -90,6 +114,7 @@ std::optional<bool> DefinedFormattedIo(IoStatementState &io,
 
 // User-defined derived type unformatted I/O
 bool DefinedUnformattedIo(IoStatementState &io, const Descriptor &descriptor,
+    const typeInfo::DerivedType &derived,
     const typeInfo::SpecialBinding &special) {
   // Unformatted I/O must have an external unit (or child thereof).
   IoErrorHandler &handler{io.GetIoErrorHandler()};
@@ -99,14 +124,34 @@ bool DefinedUnformattedIo(IoStatementState &io, const Descriptor &descriptor,
   int unit{external->unitNumber()};
   int ioStat{IostatOk};
   char ioMsg[100];
+  std::size_t numElements{descriptor.Elements()};
+  SubscriptValue subscripts[maxRank];
+  descriptor.GetLowerBounds(subscripts);
   if (special.IsArgDescriptor(0)) {
+    // "dtv" argument is "class(t)", pass a descriptor
     auto *p{special.GetProc<void (*)(
         const Descriptor &, int &, int &, char *, std::size_t)>()};
-    p(descriptor, unit, ioStat, ioMsg, sizeof ioMsg);
+    StaticDescriptor<1, true, 10 /*?*/> elementStatDesc;
+    Descriptor &elementDesc{elementStatDesc.descriptor()};
+    elementDesc.Establish(derived, nullptr, 0, nullptr, CFI_attribute_pointer);
+    for (; numElements-- > 0; descriptor.IncrementSubscripts(subscripts)) {
+      elementDesc.set_base_addr(descriptor.Element<char>(subscripts));
+      p(elementDesc, unit, ioStat, ioMsg, sizeof ioMsg);
+      if (ioStat != IostatOk) {
+        break;
+      }
+    }
   } else {
+    // "dtv" argument is "type(t)", pass a raw pointer
     auto *p{special.GetProc<void (*)(
         const void *, int &, int &, char *, std::size_t)>()};
-    p(descriptor.raw().base_addr, unit, ioStat, ioMsg, sizeof ioMsg);
+    for (; numElements-- > 0; descriptor.IncrementSubscripts(subscripts)) {
+      p(descriptor.Element<char>(subscripts), unit, ioStat, ioMsg,
+          sizeof ioMsg);
+      if (ioStat != IostatOk) {
+        break;
+      }
+    }
   }
   handler.Forward(ioStat, ioMsg, sizeof ioMsg);
   external->PopChildIo(child);
diff --git a/flang/runtime/descriptor-io.h b/flang/runtime/descriptor-io.h
index 8b2cf674f6b63..7bf55ded31d39 100644
--- a/flang/runtime/descriptor-io.h
+++ b/flang/runtime/descriptor-io.h
@@ -291,8 +291,8 @@ static bool DefaultComponentwiseIO(IoStatementState &io,
   return true;
 }
 
-std::optional<bool> DefinedFormattedIo(
-    IoStatementState &, const Descriptor &, const typeInfo::SpecialBinding &);
+std::optional<bool> DefinedFormattedIo(IoStatementState &, const Descriptor &,
+    const typeInfo::DerivedType &, const typeInfo::SpecialBinding &);
 
 template <Direction DIR>
 static bool FormattedDerivedTypeIO(
@@ -308,15 +308,15 @@ static bool FormattedDerivedTypeIO(
               ? typeInfo::SpecialBinding::Which::ReadFormatted
               : typeInfo::SpecialBinding::Which::WriteFormatted)}) {
     if (std::optional<bool> wasDefined{
-            DefinedFormattedIo(io, descriptor, *special)}) {
+            DefinedFormattedIo(io, descriptor, *type, *special)}) {
       return *wasDefined; // user-defined I/O was applied
     }
   }
   return DefaultComponentwiseIO<DIR>(io, descriptor, *type);
 }
 
-bool DefinedUnformattedIo(
-    IoStatementState &, const Descriptor &, const typeInfo::SpecialBinding &);
+bool DefinedUnformattedIo(IoStatementState &, const Descriptor &,
+    const typeInfo::DerivedType &, const typeInfo::SpecialBinding &);
 
 // Unformatted I/O
 template <Direction DIR>
@@ -332,7 +332,7 @@ static bool UnformattedDescriptorIO(
                 ? typeInfo::SpecialBinding::Which::ReadUnformatted
                 : typeInfo::SpecialBinding::Which::WriteUnformatted)}) {
       // User-defined derived type unformatted I/O
-      return DefinedUnformattedIo(io, descriptor, *special);
+      return DefinedUnformattedIo(io, descriptor, *type, *special);
     } else {
       // Default derived type unformatted I/O
       // TODO: If no component at any level has user defined READ or WRITE
diff --git a/flang/runtime/edit-output.cpp b/flang/runtime/edit-output.cpp
index 965c7ceb7de4a..bfa8e51b1ec55 100644
--- a/flang/runtime/edit-output.cpp
+++ b/flang/runtime/edit-output.cpp
@@ -14,12 +14,20 @@
 
 namespace Fortran::runtime::io {
 
+// In output statement, add a space between numbers and characters.
+static void addSpaceBeforeCharacter(IoStatementState &io) {
+  if (auto *list{io.get_if<ListDirectedStatementState<Direction::Output>>()}) {
+    list->set_lastWasUndelimitedCharacter(false);
+  }
+}
+
 // B/O/Z output of arbitrarily sized data emits a binary/octal/hexadecimal
 // representation of what is interpreted to be a single unsigned integer value.
 // When used with character data, endianness is exposed.
 template <int LOG2_BASE>
 static bool EditBOZOutput(IoStatementState &io, const DataEdit &edit,
     const unsigned char *data0, std::size_t bytes) {
+  addSpaceBeforeCharacter(io);
   int digits{static_cast<int>((bytes * 8) / LOG2_BASE)};
   int get{static_cast<int>(bytes * 8) - digits * LOG2_BASE};
   if (get > 0) {
@@ -100,6 +108,7 @@ static bool EditBOZOutput(IoStatementState &io, const DataEdit &edit,
 template <int KIND>
 bool EditIntegerOutput(IoStatementState &io, const DataEdit &edit,
     common::HostSignedIntType<8 * KIND> n) {
+  addSpaceBeforeCharacter(io);
   char buffer[130], *end{&buffer[sizeof buffer]}, *p{end};
   bool isNegative{n < 0};
   using Unsigned = common::HostUnsignedIntType<8 * KIND>;
@@ -255,6 +264,7 @@ decimal::ConversionToDecimalResult RealOutputEditing<binaryPrecision>::Convert(
 // 13.7.2.3.3 in F'2018
 template <int binaryPrecision>
 bool RealOutputEditing<binaryPrecision>::EditEorDOutput(const DataEdit &edit) {
+  addSpaceBeforeCharacter(io_);
   int editDigits{edit.digits.value_or(0)}; // 'd' field
   int editWidth{edit.width.value_or(0)}; // 'w' field
   int significantDigits{editDigits};
@@ -381,6 +391,7 @@ bool RealOutputEditing<binaryPrecision>::EditEorDOutput(const DataEdit &edit) {
 // 13.7.2.3.2 in F'2018
 template <int binaryPrecision>
 bool RealOutputEditing<binaryPrecision>::EditFOutput(const DataEdit &edit) {
+  addSpaceBeforeCharacter(io_);
   int fracDigits{edit.digits.value_or(0)}; // 'd' field
   const int editWidth{edit.width.value_or(0)}; // 'w' field
   enum decimal::FortranRounding rounding{edit.modes.round};
diff --git a/flang/runtime/format-implementation.h b/flang/runtime/format-implementation.h
index 275cbab2258b0..89c700eaa4b2d 100644
--- a/flang/runtime/format-implementation.h
+++ b/flang/runtime/format-implementation.h
@@ -64,11 +64,14 @@ FormatControl<CONTEXT>::FormatControl(const Terminator &terminator,
 
 template <typename CONTEXT>
 int FormatControl<CONTEXT>::GetIntField(
-    IoErrorHandler &handler, CharType firstCh) {
+    IoErrorHandler &handler, CharType firstCh, bool *hadError) {
   CharType ch{firstCh ? firstCh : PeekNext()};
   if (ch != '-' && ch != '+' && (ch < '0' || ch > '9')) {
     handler.SignalError(IostatErrorInFormat,
         "Invalid FORMAT: integer expected at '%c'", static_cast<char>(ch));
+    if (hadError) {
+      *hadError = true;
+    }
     return 0;
   }
   int result{0};
@@ -86,6 +89,9 @@ int FormatControl<CONTEXT>::GetIntField(
         std::numeric_limits<int>::max() / 10 - (static_cast<int>(ch) - '0')) {
       handler.SignalError(
           IostatErrorInFormat, "FORMAT integer field out of range");
+      if (hadError) {
+        *hadError = true;
+      }
       return result;
     }
     result = 10 * result + ch - '0';
@@ -99,6 +105,9 @@ int FormatControl<CONTEXT>::GetIntField(
   if (negate && (result *= -1) > 0) {
     handler.SignalError(
         IostatErrorInFormat, "FORMAT integer field out of range");
+    if (hadError) {
+      *hadError = true;
+    }
   }
   return result;
 }
@@ -212,7 +221,7 @@ static void HandleControl(CONTEXT &context, char ch, char next, int n) {
 // format validator gauntlet.
 template <typename CONTEXT>
 int FormatControl<CONTEXT>::CueUpNextDataEdit(Context &context, bool stop) {
-  int unlimitedLoopCheck{-1};
+  bool hitUnlimitedLoopEnd{false};
   // Do repetitions remain on an unparenthesized data edit?
   while (height_ > 1 && format_[stack_[height_ - 1].start] != '(') {
     offset_ = stack_[height_ - 1].start;
@@ -267,7 +276,6 @@ int FormatControl<CONTEXT>::CueUpNextDataEdit(Context &context, bool stop) {
       RUNTIME_CHECK(context, format_[stack_[height_].start] == '(');
       if (unlimited || height_ == 0) {
         stack_[height_].remaining = Iteration::unlimited;
-        unlimitedLoopCheck = offset_ - 1;
       } else if (repeat) {
         if (*repeat <= 0) {
           *repeat = 1; // error recovery
@@ -305,12 +313,13 @@ int FormatControl<CONTEXT>::CueUpNextDataEdit(Context &context, bool stop) {
               restart);
           return 0;
         }
-        if (offset_ == unlimitedLoopCheck) {
+        if (hitUnlimitedLoopEnd) {
           ReportBadFormat(context,
               "Unlimited repetition in FORMAT lacks data edit descriptors",
               restart);
           return 0;
         }
+        hitUnlimitedLoopEnd = true;
         offset_ = restart;
       } else if (stack_[height_ - 1].remaining-- > 0) {
         offset_ = restart;
@@ -401,7 +410,7 @@ int FormatControl<CONTEXT>::CueUpNextDataEdit(Context &context, bool stop) {
 
 // Returns the next data edit descriptor
 template <typename CONTEXT>
-DataEdit FormatControl<CONTEXT>::GetNextDataEdit(
+std::optional<DataEdit> FormatControl<CONTEXT>::GetNextDataEdit(
     Context &context, int maxRepeat) {
   int repeat{CueUpNextDataEdit(context)};
   auto start{offset_};
@@ -420,7 +429,7 @@ DataEdit FormatControl<CONTEXT>::GetNextDataEdit(
     if (auto quote{static_cast<char>(PeekNext())};
         quote == '\'' || quote == '"') {
       // Capture the quoted 'iotype'
-      bool ok{false}, tooLong{false};
+      bool ok{false};
       for (++offset_; offset_ < formatLength_;) {
         auto ch{static_cast<char>(format_[offset_++])};
         if (ch == quote &&
@@ -428,31 +437,36 @@ DataEdit FormatControl<CONTEXT>::GetNextDataEdit(
                 static_cast<char>(format_[offset_]) != quote)) {
           ok = true;
           break; // that was terminating quote
-        } else if (edit.ioTypeChars >= edit.maxIoTypeChars) {
-          tooLong = true;
-        } else {
-          edit.ioType[edit.ioTypeChars++] = ch;
-          if (ch == quote) {
-            ++offset_;
-          }
+        }
+        if (edit.ioTypeChars >= edit.maxIoTypeChars) {
+          ReportBadFormat(context, "Excessive DT'iotype' in FORMAT", start);
+          return std::nullopt;
+        }
+        edit.ioType[edit.ioTypeChars++] = ch;
+        if (ch == quote) {
+          ++offset_;
         }
       }
       if (!ok) {
         ReportBadFormat(context, "Unclosed DT'iotype' in FORMAT", start);
-      } else if (tooLong) {
-        ReportBadFormat(context, "Excessive DT'iotype' in FORMAT", start);
+        return std::nullopt;
       }
     }
     if (PeekNext() == '(') {
       // Capture the v_list arguments
-      bool ok{false}, tooLong{false};
+      bool ok{false};
       for (++offset_; offset_ < formatLength_;) {
-        int n{GetIntField(context)};
+        bool hadError{false};
+        int n{GetIntField(context, '\0', &hadError)};
+        if (hadError) {
+          ok = false;
+          break;
+        }
         if (edit.vListEntries >= edit.maxVListEntries) {
-          tooLong = true;
-        } else {
-          edit.vList[edit.vListEntries++] = n;
+          ReportBadFormat(context, "Excessive DT(v_list) in FORMAT", start);
+          return std::nullopt;
         }
+        edit.vList[edit.vListEntries++] = n;
         auto ch{static_cast<char>(GetNextChar(context))};
         if (ch != ',') {
           ok = ch == ')';
@@ -461,8 +475,7 @@ DataEdit FormatControl<CONTEXT>::GetNextDataEdit(
       }
       if (!ok) {
         ReportBadFormat(context, "Unclosed DT(v_list) in FORMAT", start);
-      } else if (tooLong) {
-        ReportBadFormat(context, "Excessive DT(v_list) in FORMAT", start);
+        return std::nullopt;
       }
     }
   }
diff --git a/flang/runtime/format.h b/flang/runtime/format.h
index 718a2677c14c9..9077a849eaec6 100644
--- a/flang/runtime/format.h
+++ b/flang/runtime/format.h
@@ -102,7 +102,7 @@ template <typename CONTEXT> class FormatControl {
   // Extracts the next data edit descriptor, handling control edit descriptors
   // along the way.  If maxRepeat==0, this is a peek at the next data edit
   // descriptor.
-  DataEdit GetNextDataEdit(Context &, int maxRepeat = 1);
+  std::optional<DataEdit> GetNextDataEdit(Context &, int maxRepeat = 1);
 
   // Emit any remaining character literals after the last data item (on output)
   // and perform remaining record positioning actions.
@@ -142,7 +142,8 @@ template <typename CONTEXT> class FormatControl {
     }
     return format_[offset_++];
   }
-  int GetIntField(IoErrorHandler &, CharType firstCh = '\0');
+  int GetIntField(
+      IoErrorHandler &, CharType firstCh = '\0', bool *hadError = nullptr);
 
   // Advances through the FORMAT until the next data edit
   // descriptor has been found; handles control edit descriptors
diff --git a/flang/runtime/io-api.cpp b/flang/runtime/io-api.cpp
index 197cfcc2f7f3d..faf062cd8f596 100644
--- a/flang/runtime/io-api.cpp
+++ b/flang/runtime/io-api.cpp
@@ -374,8 +374,14 @@ Cookie IONAME(BeginOpenUnit)( // OPEN(without NEWUNIT=)
   if (ExternalFileUnit *
       unit{ExternalFileUnit::LookUpOrCreate(
           unitNumber, terminator, wasExtant)}) {
-    return &unit->BeginIoStatement<OpenStatementState>(
-        terminator, *unit, wasExtant, sourceFile, sourceLine);
+    if (ChildIo * child{unit->GetChildIo()}) {
+      return &child->BeginIoStatement<ErroneousIoStatementState>(
+          IostatBadOpOnChildUnit, nullptr /* no unit */, sourceFile,
+          sourceLine);
+    } else {
+      return &unit->BeginIoStatement<OpenStatementState>(
+          terminator, *unit, wasExtant, sourceFile, sourceLine);
+    }
   } else {
     return NoopUnit(terminator, unitNumber, IostatBadUnitNumber);
   }
@@ -414,6 +420,13 @@ Cookie IONAME(BeginWaitAll)(
 Cookie IONAME(BeginClose)(
     ExternalUnit unitNumber, const char *sourceFile, int sourceLine) {
   Terminator terminator{sourceFile, sourceLine};
+  if (ExternalFileUnit * unit{ExternalFileUnit::LookUp(unitNumber)}) {
+    if (ChildIo * child{unit->GetChildIo()}) {
+      return &child->BeginIoStatement<ErroneousIoStatementState>(
+          IostatBadOpOnChildUnit, nullptr /* no unit */, sourceFile,
+          sourceLine);
+    }
+  }
   if (ExternalFileUnit * unit{ExternalFileUnit::LookUpForClose(unitNumber)}) {
     return &unit->BeginIoStatement<CloseStatementState>(
         terminator, *unit, sourceFile, sourceLine);
@@ -427,8 +440,13 @@ Cookie IONAME(BeginFlush)(
     ExternalUnit unitNumber, const char *sourceFile, int sourceLine) {
   Terminator terminator{sourceFile, sourceLine};
   if (ExternalFileUnit * unit{ExternalFileUnit::LookUp(unitNumber)}) {
-    return &unit->BeginIoStatement<ExternalMiscIoStatementState>(terminator,
-        *unit, ExternalMiscIoStatementState::Flush, sourceFile, sourceLine);
+    if (ChildIo * child{unit->GetChildIo()}) {
+      return &child->BeginIoStatement<ExternalMiscIoStatementState>(
+          *unit, ExternalMiscIoStatementState::Flush, sourceFile, sourceLine);
+    } else {
+      return &unit->BeginIoStatement<ExternalMiscIoStatementState>(terminator,
+          *unit, ExternalMiscIoStatementState::Flush, sourceFile, sourceLine);
+    }
   } else {
     // FLUSH(UNIT=bad unit) is an error; an unconnected unit is a no-op
     return NoopUnit(terminator, unitNumber,
@@ -440,8 +458,15 @@ Cookie IONAME(BeginBackspace)(
     ExternalUnit unitNumber, const char *sourceFile, int sourceLine) {
   Terminator terminator{sourceFile, sourceLine};
   if (ExternalFileUnit * unit{ExternalFileUnit::LookUp(unitNumber)}) {
-    return &unit->BeginIoStatement<ExternalMiscIoStatementState>(terminator,
-        *unit, ExternalMiscIoStatementState::Backspace, sourceFile, sourceLine);
+    if (ChildIo * child{unit->GetChildIo()}) {
+      return &child->BeginIoStatement<ErroneousIoStatementState>(
+          IostatBadOpOnChildUnit, nullptr /* no unit */, sourceFile,
+          sourceLine);
+    } else {
+      return &unit->BeginIoStatement<ExternalMiscIoStatementState>(terminator,
+          *unit, ExternalMiscIoStatementState::Backspace, sourceFile,
+          sourceLine);
+    }
   } else {
     return NoopUnit(terminator, unitNumber, IostatBadBackspaceUnit);
   }
@@ -454,8 +479,14 @@ Cookie IONAME(BeginEndfile)(
   if (ExternalFileUnit *
       unit{GetOrCreateUnit(unitNumber, Direction::Output, std::nullopt,
           terminator, errorCookie)}) {
-    return &unit->BeginIoStatement<ExternalMiscIoStatementState>(terminator,
-        *unit, ExternalMiscIoStatementState::Endfile, sourceFile, sourceLine);
+    if (ChildIo * child{unit->GetChildIo()}) {
+      return &child->BeginIoStatement<ErroneousIoStatementState>(
+          IostatBadOpOnChildUnit, nullptr /* no unit */, sourceFile,
+          sourceLine);
+    } else {
+      return &unit->BeginIoStatement<ExternalMiscIoStatementState>(terminator,
+          *unit, ExternalMiscIoStatementState::Endfile, sourceFile, sourceLine);
+    }
   } else {
     return errorCookie;
   }
@@ -468,8 +499,14 @@ Cookie IONAME(BeginRewind)(
   if (ExternalFileUnit *
       unit{GetOrCreateUnit(unitNumber, Direction::Input, std::nullopt,
           terminator, errorCookie)}) {
-    return &unit->BeginIoStatement<ExternalMiscIoStatementState>(terminator,
-        *unit, ExternalMiscIoStatementState::Rewind, sourceFile, sourceLine);
+    if (ChildIo * child{unit->GetChildIo()}) {
+      return &child->BeginIoStatement<ErroneousIoStatementState>(
+          IostatBadOpOnChildUnit, nullptr /* no unit */, sourceFile,
+          sourceLine);
+    } else {
+      return &unit->BeginIoStatement<ExternalMiscIoStatementState>(terminator,
+          *unit, ExternalMiscIoStatementState::Rewind, sourceFile, sourceLine);
+    }
   } else {
     return errorCookie;
   }
@@ -504,8 +541,13 @@ Cookie IONAME(BeginInquireFile)(const char *path, std::size_t pathLength,
       unit{ExternalFileUnit::LookUp(
           trimmed.get(), std::strlen(trimmed.get()))}) {
     // INQUIRE(FILE=) to a connected unit
-    return &unit->BeginIoStatement<InquireUnitState>(
-        terminator, *unit, sourceFile, sourceLine);
+    if (ChildIo * child{unit->GetChildIo()}) {
+      return &child->BeginIoStatement<InquireUnitState>(
+          *unit, sourceFile, sourceLine);
+    } else {
+      return &unit->BeginIoStatement<InquireUnitState>(
+          terminator, *unit, sourceFile, sourceLine);
+    }
   } else {
     return &New<InquireUnconnectedFileState>{terminator}(
         std::move(trimmed), sourceFile, sourceLine)
@@ -566,7 +608,12 @@ bool IONAME(SetAdvance)(
   if (nonAdvancing && io.GetConnectionState().access == Access::Direct) {
     handler.SignalError("Non-advancing I/O attempted on direct access file");
   } else {
-    io.mutableModes().nonAdvancing = nonAdvancing;
+    auto *unit{io.GetExternalFileUnit()};
+    if (unit && unit->GetChildIo()) {
+      // ADVANCE= is ignored for child I/O (12.6.4.8.3 p3)
+    } else {
+      io.mutableModes().nonAdvancing = nonAdvancing;
+    }
   }
   return !handler.InError();
 }
@@ -648,7 +695,12 @@ bool IONAME(SetRec)(Cookie cookie, std::int64_t rec) {
   IoStatementState &io{*cookie};
   IoErrorHandler &handler{io.GetIoErrorHandler()};
   if (auto *unit{io.GetExternalFileUnit()}) {
-    unit->SetDirectRec(rec, handler);
+    if (unit->GetChildIo()) {
+      handler.SignalError(
+          IostatBadOpOnChildUnit, "REC= specifier on child I/O");
+    } else {
+      unit->SetDirectRec(rec, handler);
+    }
   } else if (!io.get_if<ErroneousIoStatementState>()) {
     handler.Crash("SetRec() called on internal unit");
   }
diff --git a/flang/runtime/io-error.cpp b/flang/runtime/io-error.cpp
index 790c579e1c43c..56e4b24cbe062 100644
--- a/flang/runtime/io-error.cpp
+++ b/flang/runtime/io-error.cpp
@@ -18,16 +18,29 @@
 namespace Fortran::runtime::io {
 
 void IoErrorHandler::SignalError(int iostatOrErrno, const char *msg, ...) {
-  if (iostatOrErrno == IostatEnd && (flags_ & hasEnd)) {
-    if (ioStat_ == IostatOk || ioStat_ < IostatEnd) {
-      ioStat_ = IostatEnd;
+  // Note that IOMSG= alone without IOSTAT=/END=/EOR=/ERR= does not suffice
+  // for error recovery (see F'2018 subclause 12.11).
+  switch (iostatOrErrno) {
+  case IostatOk:
+    return;
+  case IostatEnd:
+    if (flags_ & (hasIoStat | hasEnd)) {
+      if (ioStat_ == IostatOk || ioStat_ < IostatEnd) {
+        ioStat_ = IostatEnd;
+      }
+      return;
     }
-  } else if (iostatOrErrno == IostatEor && (flags_ & hasEor)) {
-    if (ioStat_ == IostatOk || ioStat_ < IostatEor) {
-      ioStat_ = IostatEor; // least priority
+    break;
+  case IostatEor:
+    if (flags_ & (hasIoStat | hasEor)) {
+      if (ioStat_ == IostatOk || ioStat_ < IostatEor) {
+        ioStat_ = IostatEor; // least priority
+      }
+      return;
     }
-  } else if (iostatOrErrno != IostatOk) {
-    if (flags_ & (hasIoStat | hasIoMsg | hasErr)) {
+    break;
+  default:
+    if (flags_ & (hasIoStat | hasErr)) {
       if (ioStat_ <= 0) {
         ioStat_ = iostatOrErrno; // priority over END=/EOR=
         if (msg && (flags_ & hasIoMsg)) {
@@ -39,17 +52,21 @@ void IoErrorHandler::SignalError(int iostatOrErrno, const char *msg, ...) {
           va_end(ap);
         }
       }
-    } else if (msg) {
-      va_list ap;
-      va_start(ap, msg);
-      CrashArgs(msg, ap);
-      va_end(ap);
-    } else if (const char *errstr{IostatErrorString(iostatOrErrno)}) {
-      Crash(errstr);
-    } else {
-      Crash("I/O error (errno=%d): %s", iostatOrErrno,
-          std::strerror(iostatOrErrno));
+      return;
     }
+    break;
+  }
+  // I/O error not caught!
+  if (msg) {
+    va_list ap;
+    va_start(ap, msg);
+    CrashArgs(msg, ap);
+    va_end(ap);
+  } else if (const char *errstr{IostatErrorString(iostatOrErrno)}) {
+    Crash(errstr);
+  } else {
+    Crash("I/O error (errno=%d): %s", iostatOrErrno,
+        std::strerror(iostatOrErrno));
   }
 }
 
@@ -59,13 +76,12 @@ void IoErrorHandler::SignalError(int iostatOrErrno) {
 
 void IoErrorHandler::Forward(
     int ioStatOrErrno, const char *msg, std::size_t length) {
-  if (ioStat_ != IostatOk && msg && (flags_ & hasIoMsg)) {
-    ioMsg_ = SaveDefaultCharacter(msg, length, *this);
-  }
-  if (ioStatOrErrno != IostatOk && msg) {
-    SignalError(ioStatOrErrno, "%.*s", static_cast<int>(length), msg);
-  } else {
-    SignalError(ioStatOrErrno);
+  if (ioStatOrErrno != IostatOk) {
+    if (msg) {
+      SignalError(ioStatOrErrno, "%.*s", static_cast<int>(length), msg);
+    } else {
+      SignalError(ioStatOrErrno);
+    }
   }
 }
 
diff --git a/flang/runtime/iostat.cpp b/flang/runtime/iostat.cpp
index 747a776aae7cc..f34bde28be9d0 100644
--- a/flang/runtime/iostat.cpp
+++ b/flang/runtime/iostat.cpp
@@ -109,6 +109,8 @@ const char *IostatErrorString(int iostat) {
     return "Negative unit number is not allowed";
   case IostatBadFlushUnit:
     return "FLUSH attempted on a bad or unconnected unit number";
+  case IostatBadOpOnChildUnit:
+    return "Impermissible I/O statement on child I/O unit";
   default:
     return nullptr;
   }
diff --git a/flang/runtime/matmul-transpose.cpp b/flang/runtime/matmul-transpose.cpp
new file mode 100644
index 0000000000000..345e3d8b41ac3
--- /dev/null
+++ b/flang/runtime/matmul-transpose.cpp
@@ -0,0 +1,296 @@
+//===-- runtime/matmul-transpose.cpp --------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Implements a fused matmul-transpose operation
+//
+// There are two main entry points; one establishes a descriptor for the
+// result and allocates it, and the other expects a result descriptor that
+// points to existing storage.
+//
+// This implementation must handle all combinations of numeric types and
+// kinds (100 - 165 cases depending on the target), plus all combinations
+// of logical kinds (16).  A single template undergoes many instantiations
+// to cover all of the valid possibilities.
+//
+// The usefulness of this optimization should be reviewed once Matmul is swapped
+// to use the faster BLAS routines.
+
+#include "flang/Runtime/matmul-transpose.h"
+#include "terminator.h"
+#include "tools.h"
+#include "flang/Runtime/c-or-cpp.h"
+#include "flang/Runtime/cpp-type.h"
+#include "flang/Runtime/descriptor.h"
+#include <cstring>
+
+namespace {
+using namespace Fortran::runtime;
+
+// Contiguous numeric TRANSPOSE(matrix)*matrix multiplication
+//   TRANSPOSE(matrix(n, rows)) * matrix(n,cols) ->
+//             matrix(rows, n)  * matrix(n,cols) -> matrix(rows,cols)
+// The transpose is implemented by swapping the indices of accesses into the LHS
+//
+// Straightforward algorithm:
+//   DO 1 I = 1, NROWS
+//    DO 1 J = 1, NCOLS
+//     RES(I,J) = 0
+//     DO 1 K = 1, N
+//   1  RES(I,J) = RES(I,J) + X(K,I)*Y(K,J)
+//
+// With loop distribution and transposition to avoid the inner sum
+// reduction and to avoid non-unit strides:
+//   DO 1 I = 1, NROWS
+//    DO 1 J = 1, NCOLS
+//   1 RES(I,J) = 0
+//   DO 2 J = 1, NCOLS
+//    DO 2 I = 1, NROWS
+//     DO 2 K = 1, N
+//   2  RES(I,J) = RES(I,J) + X(K,I)*Y(K,J) ! loop-invariant last term
+template <TypeCategory RCAT, int RKIND, typename XT, typename YT>
+inline static void MatrixTransposedTimesMatrix(
+    CppTypeFor<RCAT, RKIND> *RESTRICT product, SubscriptValue rows,
+    SubscriptValue cols, const XT *RESTRICT x, const YT *RESTRICT y,
+    SubscriptValue n) {
+  using ResultType = CppTypeFor<RCAT, RKIND>;
+
+  std::memset(product, 0, rows * cols * sizeof *product);
+  for (SubscriptValue j{0}; j < cols; ++j) {
+    for (SubscriptValue i{0}; i < rows; ++i) {
+      for (SubscriptValue k{0}; k < n; ++k) {
+        ResultType x_ki = static_cast<ResultType>(x[i * n + k]);
+        ResultType y_kj = static_cast<ResultType>(y[j * n + k]);
+        product[j * rows + i] += x_ki * y_kj;
+      }
+    }
+  }
+}
+
+// Contiguous numeric matrix*vector multiplication
+//   matrix(rows,n) * column vector(n) -> column vector(rows)
+// Straightforward algorithm:
+//   DO 1 I = 1, NROWS
+//    RES(I) = 0
+//    DO 1 K = 1, N
+//   1 RES(I) = RES(I) + X(K,I)*Y(K)
+// With loop distribution and transposition to avoid the inner
+// sum reduction and to avoid non-unit strides:
+//   DO 1 I = 1, NROWS
+//   1 RES(I) = 0
+//   DO 2 I = 1, NROWS
+//    DO 2 K = 1, N
+//   2 RES(I) = RES(I) + X(K,I)*Y(K)
+template <TypeCategory RCAT, int RKIND, typename XT, typename YT>
+inline static void MatrixTransposedTimesVector(
+    CppTypeFor<RCAT, RKIND> *RESTRICT product, SubscriptValue rows,
+    SubscriptValue n, const XT *RESTRICT x, const YT *RESTRICT y) {
+  using ResultType = CppTypeFor<RCAT, RKIND>;
+  std::memset(product, 0, rows * sizeof *product);
+  for (SubscriptValue i{0}; i < rows; ++i) {
+    for (SubscriptValue k{0}; k < n; ++k) {
+      ResultType x_ki = static_cast<ResultType>(x[i * n + k]);
+      ResultType y_k = static_cast<ResultType>(y[k]);
+      product[i] += x_ki * y_k;
+    }
+  }
+}
+
+// Implements an instance of MATMUL for given argument types.
+template <bool IS_ALLOCATING, TypeCategory RCAT, int RKIND, typename XT,
+    typename YT>
+inline static void DoMatmulTranspose(
+    std::conditional_t<IS_ALLOCATING, Descriptor, const Descriptor> &result,
+    const Descriptor &x, const Descriptor &y, Terminator &terminator) {
+  int xRank{x.rank()};
+  int yRank{y.rank()};
+  int resRank{xRank + yRank - 2};
+  if (xRank * yRank != 2 * resRank) {
+    terminator.Crash("MATMUL: bad argument ranks (%d * %d)", xRank, yRank);
+  }
+  SubscriptValue extent[2]{x.GetDimension(1).Extent(),
+      resRank == 2 ? y.GetDimension(1).Extent() : 0};
+  if constexpr (IS_ALLOCATING) {
+    result.Establish(
+        RCAT, RKIND, nullptr, resRank, extent, CFI_attribute_allocatable);
+    for (int j{0}; j < resRank; ++j) {
+      result.GetDimension(j).SetBounds(1, extent[j]);
+    }
+    if (int stat{result.Allocate()}) {
+      terminator.Crash(
+          "MATMUL: could not allocate memory for result; STAT=%d", stat);
+    }
+  } else {
+    RUNTIME_CHECK(terminator, resRank == result.rank());
+    RUNTIME_CHECK(
+        terminator, result.ElementBytes() == static_cast<std::size_t>(RKIND));
+    RUNTIME_CHECK(terminator, result.GetDimension(0).Extent() == extent[0]);
+    RUNTIME_CHECK(terminator,
+        resRank == 1 || result.GetDimension(1).Extent() == extent[1]);
+  }
+  SubscriptValue n{x.GetDimension(0).Extent()};
+  if (n != y.GetDimension(0).Extent()) {
+    terminator.Crash("MATMUL: unacceptable operand shapes (%jdx%jd, %jdx%jd)",
+        static_cast<std::intmax_t>(x.GetDimension(0).Extent()),
+        static_cast<std::intmax_t>(x.GetDimension(1).Extent()),
+        static_cast<std::intmax_t>(y.GetDimension(0).Extent()),
+        static_cast<std::intmax_t>(y.GetDimension(1).Extent()));
+  }
+  using WriteResult =
+      CppTypeFor<RCAT == TypeCategory::Logical ? TypeCategory::Integer : RCAT,
+          RKIND>;
+  const SubscriptValue rows{extent[0]};
+  const SubscriptValue cols{extent[1]};
+  if constexpr (RCAT != TypeCategory::Logical) {
+    if (x.IsContiguous() && y.IsContiguous() &&
+        (IS_ALLOCATING || result.IsContiguous())) {
+      // Contiguous numeric matrices
+      if (resRank == 2) { // M*M -> M
+        MatrixTransposedTimesMatrix<RCAT, RKIND, XT, YT>(
+            result.template OffsetElement<WriteResult>(), rows, cols,
+            x.OffsetElement<XT>(), y.OffsetElement<YT>(), n);
+        return;
+      }
+      if (xRank == 2) { // M*V -> V
+        MatrixTransposedTimesVector<RCAT, RKIND, XT, YT>(
+            result.template OffsetElement<WriteResult>(), rows, n,
+            x.OffsetElement<XT>(), y.OffsetElement<YT>());
+        return;
+      }
+      // else V*M -> V (not allowed because TRANSPOSE() is only defined for rank
+      // 1 matrices
+      terminator.Crash("MATMUL: unacceptable operand shapes (%jdx%jd, %jdx%jd)",
+          static_cast<std::intmax_t>(x.GetDimension(0).Extent()),
+          static_cast<std::intmax_t>(n),
+          static_cast<std::intmax_t>(y.GetDimension(0).Extent()),
+          static_cast<std::intmax_t>(y.GetDimension(1).Extent()));
+      return;
+    }
+  }
+  // General algorithms for LOGICAL and noncontiguity
+  SubscriptValue xLB[2], yLB[2], resLB[2];
+  x.GetLowerBounds(xLB);
+  y.GetLowerBounds(yLB);
+  result.GetLowerBounds(resLB);
+  using ResultType = CppTypeFor<RCAT, RKIND>;
+  if (resRank == 2) { // M*M -> M
+    for (SubscriptValue i{0}; i < rows; ++i) {
+      for (SubscriptValue j{0}; j < cols; ++j) {
+        ResultType res_ij;
+        if constexpr (RCAT == TypeCategory::Logical) {
+          res_ij = false;
+        } else {
+          res_ij = 0;
+        }
+
+        for (SubscriptValue k{0}; k < n; ++k) {
+          SubscriptValue xAt[2]{k + xLB[0], i + xLB[1]};
+          SubscriptValue yAt[2]{k + yLB[0], j + yLB[1]};
+          if constexpr (RCAT == TypeCategory::Logical) {
+            ResultType x_ki = IsLogicalElementTrue(x, xAt);
+            ResultType y_kj = IsLogicalElementTrue(y, yAt);
+            res_ij = res_ij || (x_ki && y_kj);
+          } else {
+            ResultType x_ki = static_cast<ResultType>(*x.Element<XT>(xAt));
+            ResultType y_kj = static_cast<ResultType>(*y.Element<YT>(yAt));
+            res_ij += x_ki * y_kj;
+          }
+        }
+        SubscriptValue resAt[2]{i + resLB[0], j + resLB[1]};
+        *result.template Element<WriteResult>(resAt) = res_ij;
+      }
+    }
+  } else if (xRank == 2) { // M*V -> V
+    for (SubscriptValue i{0}; i < rows; ++i) {
+      ResultType res_i;
+      if constexpr (RCAT == TypeCategory::Logical) {
+        res_i = false;
+      } else {
+        res_i = 0;
+      }
+
+      for (SubscriptValue k{0}; k < n; ++k) {
+        SubscriptValue xAt[2]{k + xLB[0], i + xLB[1]};
+        SubscriptValue yAt[1]{k + yLB[0]};
+        if constexpr (RCAT == TypeCategory::Logical) {
+          ResultType x_ki = IsLogicalElementTrue(x, xAt);
+          ResultType y_k = IsLogicalElementTrue(y, yAt);
+          res_i = res_i || (x_ki && y_k);
+        } else {
+          ResultType x_ki = static_cast<ResultType>(*x.Element<XT>(xAt));
+          ResultType y_k = static_cast<ResultType>(*y.Element<YT>(yAt));
+          res_i += x_ki * y_k;
+        }
+      }
+      SubscriptValue resAt[1]{i + resLB[0]};
+      *result.template Element<WriteResult>(resAt) = res_i;
+    }
+  } else { // V*M -> V
+    // TRANSPOSE(V) not allowed by fortran standard
+    terminator.Crash("MATMUL: unacceptable operand shapes (%jdx%jd, %jdx%jd)",
+        static_cast<std::intmax_t>(x.GetDimension(0).Extent()),
+        static_cast<std::intmax_t>(n),
+        static_cast<std::intmax_t>(y.GetDimension(0).Extent()),
+        static_cast<std::intmax_t>(y.GetDimension(1).Extent()));
+  }
+}
+
+// Maps the dynamic type information from the arguments' descriptors
+// to the right instantiation of DoMatmul() for valid combinations of
+// types.
+template <bool IS_ALLOCATING> struct MatmulTranspose {
+  using ResultDescriptor =
+      std::conditional_t<IS_ALLOCATING, Descriptor, const Descriptor>;
+  template <TypeCategory XCAT, int XKIND> struct MM1 {
+    template <TypeCategory YCAT, int YKIND> struct MM2 {
+      void operator()(ResultDescriptor &result, const Descriptor &x,
+          const Descriptor &y, Terminator &terminator) const {
+        if constexpr (constexpr auto resultType{
+                          GetResultType(XCAT, XKIND, YCAT, YKIND)}) {
+          if constexpr (Fortran::common::IsNumericTypeCategory(
+                            resultType->first) ||
+              resultType->first == TypeCategory::Logical) {
+            return DoMatmulTranspose<IS_ALLOCATING, resultType->first,
+                resultType->second, CppTypeFor<XCAT, XKIND>,
+                CppTypeFor<YCAT, YKIND>>(result, x, y, terminator);
+          }
+        }
+        terminator.Crash("MATMUL: bad operand types (%d(%d), %d(%d))",
+            static_cast<int>(XCAT), XKIND, static_cast<int>(YCAT), YKIND);
+      }
+    };
+    void operator()(ResultDescriptor &result, const Descriptor &x,
+        const Descriptor &y, Terminator &terminator, TypeCategory yCat,
+        int yKind) const {
+      ApplyType<MM2, void>(yCat, yKind, terminator, result, x, y, terminator);
+    }
+  };
+  void operator()(ResultDescriptor &result, const Descriptor &x,
+      const Descriptor &y, const char *sourceFile, int line) const {
+    Terminator terminator{sourceFile, line};
+    auto xCatKind{x.type().GetCategoryAndKind()};
+    auto yCatKind{y.type().GetCategoryAndKind()};
+    RUNTIME_CHECK(terminator, xCatKind.has_value() && yCatKind.has_value());
+    ApplyType<MM1, void>(xCatKind->first, xCatKind->second, terminator, result,
+        x, y, terminator, yCatKind->first, yCatKind->second);
+  }
+};
+} // namespace
+
+namespace Fortran::runtime {
+extern "C" {
+void RTNAME(MatmulTranspose)(Descriptor &result, const Descriptor &x,
+    const Descriptor &y, const char *sourceFile, int line) {
+  MatmulTranspose<true>{}(result, x, y, sourceFile, line);
+}
+void RTNAME(MatmulTransposeDirect)(const Descriptor &result,
+    const Descriptor &x, const Descriptor &y, const char *sourceFile,
+    int line) {
+  MatmulTranspose<false>{}(result, x, y, sourceFile, line);
+}
+} // extern "C"
+} // namespace Fortran::runtime
diff --git a/flang/runtime/pointer.cpp b/flang/runtime/pointer.cpp
index 7305b1e54223c..216e8c70be029 100644
--- a/flang/runtime/pointer.cpp
+++ b/flang/runtime/pointer.cpp
@@ -206,7 +206,8 @@ bool RTNAME(PointerIsAssociatedWith)(
   if (!target) {
     return pointer.raw().base_addr != nullptr;
   }
-  if (!target->raw().base_addr || target->ElementBytes() == 0) {
+  if (!target->raw().base_addr ||
+      (target->raw().type != CFI_type_struct && target->ElementBytes() == 0)) {
     return false;
   }
   int rank{pointer.rank()};
diff --git a/flang/runtime/unit.h b/flang/runtime/unit.h
index c49f479830338..aad896afce513 100644
--- a/flang/runtime/unit.h
+++ b/flang/runtime/unit.h
@@ -203,7 +203,7 @@ class ChildIo {
       ChildListIoStatementState<Direction::Input>,
       ChildUnformattedIoStatementState<Direction::Output>,
       ChildUnformattedIoStatementState<Direction::Input>, InquireUnitState,
-      ErroneousIoStatementState>
+      ErroneousIoStatementState, ExternalMiscIoStatementState>
       u_;
   std::optional<IoStatementState> io_;
 };
diff --git a/flang/test/CMakeLists.txt b/flang/test/CMakeLists.txt
index 3252ff37bd21e..4de1036dfb52b 100644
--- a/flang/test/CMakeLists.txt
+++ b/flang/test/CMakeLists.txt
@@ -3,8 +3,8 @@
 add_subdirectory(lib)
 
 llvm_canonicalize_cmake_booleans(
-  FLANG_BUILD_EXAMPLES
   FLANG_STANDALONE_BUILD
+  LLVM_BUILD_EXAMPLES
   LLVM_BYE_LINK_INTO_TOOLS
   LLVM_ENABLE_PLUGINS
 )
@@ -59,6 +59,7 @@ set(FLANG_TEST_DEPENDS
   bbc
   llvm-dis
   llvm-objdump
+  llvm-readobj
   split-file
   FortranRuntime
   Fortran_main
@@ -74,7 +75,7 @@ if (FLANG_INCLUDE_TESTS)
   endif()
 endif()
 
-if (FLANG_BUILD_EXAMPLES)
+if (LLVM_BUILD_EXAMPLES)
   list(APPEND FLANG_TEST_DEPENDS
     flangPrintFunctionNames
     flangOmpReport
diff --git a/flang/test/Driver/code-gen-rv64.f90 b/flang/test/Driver/code-gen-rv64.f90
new file mode 100644
index 0000000000000..c29ec9eff0aa2
--- /dev/null
+++ b/flang/test/Driver/code-gen-rv64.f90
@@ -0,0 +1,19 @@
+! Test -emit-obj (RISC-V 64)
+
+! REQUIRES: riscv-registered-target
+
+! RUN: rm -f %t.o
+! RUN: %flang_fc1 -triple riscv64-unknown-linux-gnu \
+! RUN:   -target-feature +d -target-feature +c -emit-obj %s -o %t.o
+! RUN: llvm-readobj -h %t.o | FileCheck %s
+
+! RUN: rm -f %t.o
+! RUN: %flang --target=riscv64-unknown-linux-gnu -c %s -o %t.o
+! RUN: llvm-readobj -h %t.o | FileCheck %s
+
+! If Flang failed to emit target-feature info, then Flags will be 0x0.
+! CHECK: Flags [ (0x5)
+! CHECK-NEXT: EF_RISCV_FLOAT_ABI_DOUBLE (0x4)
+! CHECK-NEXT: EF_RISCV_RVC (0x1)
+! CHECK-NEXT: ]
+end program
diff --git a/flang/test/Driver/default-optimization-pipelines.f90 b/flang/test/Driver/default-optimization-pipelines.f90
index 8a6ea57bdab33..08e407f73da5c 100644
--- a/flang/test/Driver/default-optimization-pipelines.f90
+++ b/flang/test/Driver/default-optimization-pipelines.f90
@@ -1,10 +1,18 @@
 ! Verify that`-O{n}` is indeed taken into account when defining the LLVM optimization/middle-end pass pipeline.
 
 ! RUN: %flang -S -O0 %s -Xflang -fdebug-pass-manager -o /dev/null 2>&1 | FileCheck %s --check-prefix=CHECK-O0
+! RUN: %flang -S -O0 %s -flto -Xflang -fdebug-pass-manager -o /dev/null 2>&1 | FileCheck %s --check-prefix=CHECK-O0-ANYLTO
+! RUN: %flang -S -O0 %s -flto=thin -Xflang -fdebug-pass-manager -o /dev/null 2>&1 | FileCheck %s --check-prefix=CHECK-O0-ANYLTO
 ! RUN: %flang_fc1 -S -O0 %s -fdebug-pass-manager -o /dev/null 2>&1 | FileCheck %s --check-prefix=CHECK-O0
+! RUN: %flang_fc1 -S -O0 %s -flto=full -fdebug-pass-manager -o /dev/null 2>&1 | FileCheck %s --check-prefix=CHECK-O0-ANYLTO
+! RUN: %flang_fc1 -S -O0 %s -flto=thin -fdebug-pass-manager -o /dev/null 2>&1 | FileCheck %s --check-prefix=CHECK-O0-ANYLTO
 
 ! RUN: %flang -S -O2 %s -Xflang -fdebug-pass-manager -o /dev/null 2>&1 | FileCheck %s --check-prefix=CHECK-O2
+! RUN: %flang -S -O2 %s -flto -Xflang -fdebug-pass-manager -o /dev/null 2>&1 | FileCheck %s --check-prefix=CHECK-O2-LTO
+! RUN: %flang -S -O2 %s -flto=thin -Xflang -fdebug-pass-manager -o /dev/null 2>&1 | FileCheck %s --check-prefix=CHECK-O2-THINLTO
 ! RUN: %flang_fc1 -S -O2 %s -fdebug-pass-manager -o /dev/null 2>&1 | FileCheck %s --check-prefix=CHECK-O2
+! RUN: %flang_fc1 -S -O2 %s -flto=full -fdebug-pass-manager -o /dev/null 2>&1 | FileCheck %s --check-prefix=CHECK-O2-LTO
+! RUN: %flang_fc1 -S -O2 %s -flto=thin -fdebug-pass-manager -o /dev/null 2>&1 | FileCheck %s --check-prefix=CHECK-O2-THINLTO
 
 ! Verify that only the left-most `-O{n}` is used
 ! RUN: %flang -S -O2 -O0 %s -Xflang -fdebug-pass-manager -o /dev/null 2>&1 | FileCheck %s --check-prefix=CHECK-O0
@@ -12,9 +20,19 @@
 
 ! CHECK-O0-NOT: Running pass: SimplifyCFGPass on simple_loop_
 ! CHECK-O0: Running analysis: TargetLibraryAnalysis on simple_loop_
+! CHECK-O0-ANYLTO: Running pass: CanonicalizeAliasesPass on [module]
+! CHECK-O0-ANYLTO: Running pass: NameAnonGlobalPass on [module]
 
 ! CHECK-O2: Running pass: SimplifyCFGPass on simple_loop_
 
+! CHECK-O2-LTO-NOT: Running pass: EliminateAvailableExternallyPass
+! CHECK-O2-LTO: Running pass: CanonicalizeAliasesPass on [module]
+! CHECK-O2-LTO: Running pass: NameAnonGlobalPass on [module]
+
+! CHECK-O2-THINLTO-NOT: Running pass: LoopVectorizePass
+! CHECK-O2-THINLTO: Running pass: CanonicalizeAliasesPass on [module]
+! CHECK-O2-THINLTO: Running pass: NameAnonGlobalPass on [module]
+
 subroutine simple_loop
   integer :: i
   do i=1,5
diff --git a/flang/test/Driver/driver-help-hidden.f90 b/flang/test/Driver/driver-help-hidden.f90
index 18666ae2c6214..bcb77c9f8bccb 100644
--- a/flang/test/Driver/driver-help-hidden.f90
+++ b/flang/test/Driver/driver-help-hidden.f90
@@ -43,6 +43,8 @@
 ! CHECK-NEXT:                        Enable support for generating executables (experimental)
 ! CHECK-NEXT: -flarge-sizes          Use INTEGER(KIND=8) for the result type in size-related intrinsics
 ! CHECK-NEXT: -flogical-abbreviations Enable logical abbreviations
+! CHECK-NEXT: -flto=<value> Set LTO mode
+! CHECK-NEXT: -flto Enable LTO in 'full' mode
 ! CHECK-NEXT: -fno-automatic         Implies the SAVE attribute for non-automatic local objects in subprograms unless RECURSIVE
 ! CHECK-NEXT: -fno-color-diagnostics  Disable colors in diagnostics
 ! CHECK-NEXT: -fno-integrated-as     Disable the integrated assembler
diff --git a/flang/test/Driver/driver-help.f90 b/flang/test/Driver/driver-help.f90
index 7fdce71190f61..7bafcc6f5991d 100644
--- a/flang/test/Driver/driver-help.f90
+++ b/flang/test/Driver/driver-help.f90
@@ -41,6 +41,8 @@
 ! HELP-NEXT:                        Specify where to find the compiled intrinsic modules
 ! HELP-NEXT: -flarge-sizes          Use INTEGER(KIND=8) for the result type in size-related intrinsics
 ! HELP-NEXT: -flogical-abbreviations Enable logical abbreviations
+! HELP-NEXT: -flto=<value> Set LTO mode
+! HELP-NEXT: -flto Enable LTO in 'full' mode
 ! HELP-NEXT: -fno-automatic         Implies the SAVE attribute for non-automatic local objects in subprograms unless RECURSIVE
 ! HELP-NEXT: -fno-color-diagnostics  Disable colors in diagnostics
 ! HELP-NEXT: -fno-integrated-as      Disable the integrated assembler
@@ -130,6 +132,8 @@
 ! HELP-FC1-NEXT:                        Specify where to find the compiled intrinsic modules
 ! HELP-FC1-NEXT: -flarge-sizes          Use INTEGER(KIND=8) for the result type in size-related intrinsics
 ! HELP-FC1-NEXT: -flogical-abbreviations Enable logical abbreviations
+! HELP-FC1-NEXT: -flto=<value>           Set LTO mode
+! HELP-FC1-NEXT: -flto                   Enable LTO in 'full' mode
 ! HELP-FC1-NEXT: -fno-analyzed-objects-for-unparse
 ! HELP-FC1-NEXT:                        Do not use the analyzed objects when unparsing
 ! HELP-FC1-NEXT: -fno-automatic         Implies the SAVE attribute for non-automatic local objects in subprograms unless RECURSIVE
@@ -138,6 +142,7 @@
 ! HELP-FC1-NEXT: -fno-signed-zeros      Allow optimizations that ignore the sign of floating point zeros
 ! HELP-FC1-NEXT: -fno-stack-arrays      Allocate array temporaries on the heap (default)
 ! HELP-FC1-NEXT: -fopenacc              Enable OpenACC
+! HELP-FC1-NEXT: -fopenmp-is-device     Generate code only for an OpenMP target device.
 ! HELP-FC1-NEXT: -fopenmp               Parse OpenMP pragmas and generate parallel code.
 ! HELP-FC1-NEXT: -fpass-plugin=<dsopath> Load pass plugin from a dynamic shared object file (only with new pass manager).
 ! HELP-FC1-NEXT: -freciprocal-math      Allow division operations to be reassociated
diff --git a/flang/test/Driver/linker-flags.f90 b/flang/test/Driver/linker-flags.f90
index 90c4a940c6d5f..e1350b74ae30a 100644
--- a/flang/test/Driver/linker-flags.f90
+++ b/flang/test/Driver/linker-flags.f90
@@ -17,20 +17,24 @@
 ! CHECK-SAME:  "-o" "[[object_file:.*\.o]]" {{.*}}Inputs/hello.f90
 
 ! Linker invocation to generate the executable
-! GNU-LABEL:  "{{.*}}ld" 
+! NOTE: Since we are cross-compiling, the host toolchain executables may
+!       run on any other platform, such as Windows that use a .exe
+!       suffix. Clang's driver will try to resolve the path to the ld
+!       executable and may find the GNU linker from MinGW or Cygwin.
+! GNU-LABEL:  "{{.*}}ld{{(\.exe)?}}"
 ! GNU-SAME: "[[object_file]]"
 ! GNU-SAME: -lFortran_main
 ! GNU-SAME: -lFortranRuntime
 ! GNU-SAME: -lFortranDecimal
 ! GNU-SAME: -lm
 
-! DARWIN-LABEL:  "{{.*}}ld" 
+! DARWIN-LABEL:  "{{.*}}ld{{(\.exe)?}}"
 ! DARWIN-SAME: "[[object_file]]"
 ! DARWIN-SAME: -lFortran_main
 ! DARWIN-SAME: -lFortranRuntime
 ! DARWIN-SAME: -lFortranDecimal
 
-! MINGW-LABEL:  "{{.*}}ld" 
+! MINGW-LABEL:  "{{.*}}ld{{(\.exe)?}}"
 ! MINGW-SAME: "[[object_file]]"
 ! MINGW-SAME: -lFortran_main
 ! MINGW-SAME: -lFortranRuntime
diff --git a/flang/test/Driver/lto-bc.f90 b/flang/test/Driver/lto-bc.f90
new file mode 100644
index 0000000000000..5e34cdb87c5b1
--- /dev/null
+++ b/flang/test/Driver/lto-bc.f90
@@ -0,0 +1,21 @@
+! Test that the output is LLVM bitcode for LTO and not a native objectfile by
+! disassembling it to LLVM IR.
+! Right now there is nothing special about it and it is similar to non-lto IR,
+! more work is needed to add things like module summaries.
+
+! RUN: %flang %s -c -o - | not llvm-dis -o %t
+! RUN: %flang_fc1 %s -emit-llvm-bc -o - | llvm-dis -o - | FileCheck %s
+
+! RUN: %flang -flto %s -c -o - | llvm-dis -o - | FileCheck %s
+! RUN: %flang -flto=thin %s -c -o - | llvm-dis -o - | FileCheck %s
+
+! CHECK: define void @_QQmain()
+! CHECK-NEXT:  ret void
+! CHECK-NEXT: }
+
+! CHECK-NOT: ^0 = module:
+! CHECK-NOT: ^1 = gv: (name:
+! CHECK-NOT: ^2 = flags:
+! CHECK-NOT: ^3 = blockcount:
+
+end program
diff --git a/flang/test/Driver/lto-flags.f90 b/flang/test/Driver/lto-flags.f90
new file mode 100644
index 0000000000000..5c69f79d35216
--- /dev/null
+++ b/flang/test/Driver/lto-flags.f90
@@ -0,0 +1,32 @@
+! UNSUPPORTED: system-windows
+! RUN: %flang -### -S %s 2>&1 | FileCheck %s --check-prefix=NO-LTO
+! RUN: %flang -### -S -fno-lto %s 2>&1 | FileCheck %s --check-prefix=NO-LTO
+
+! Full LTO and aliases.
+! RUN: %flang -### -S -flto %s 2>&1 | FileCheck %s --check-prefix=FULL-LTO
+! RUN: %flang -### -S -flto=full %s 2>&1 | FileCheck %s --check-prefix=FULL-LTO
+! RUN: %flang -### -S -flto=auto %s 2>&1 | FileCheck %s --check-prefix=FULL-LTO
+! RUN: %flang -### -S -flto=jobserver %s 2>&1 | FileCheck %s --check-prefix=FULL-LTO
+
+! Also check linker plugin opt for Thin LTO
+! RUN: %flang -### -flto=thin %s 2>&1 | FileCheck %s --check-prefix=THIN-LTO
+
+! RUN: %flang -### -S -flto=somelto %s 2>&1 | FileCheck %s --check-prefix=ERROR
+
+! FC1 tests. Check that it does not crash.
+! RUN: %flang_fc1 -S %s -flto -o /dev/null
+! RUN: %flang_fc1 -S %s -flto=full -o /dev/null
+! RUN: %flang_fc1 -S %s -flto=thin -o /dev/null
+
+! NO-LTO: "-fc1"
+! NO-LTO-NOT: flto
+
+! FULL-LTO: "-fc1"
+! FULL-LTO-SAME: "-flto=full"
+
+! THIN-LTO: flang-new: warning: the option '-flto=thin' is a work in progress
+! THIN-LTO: "-fc1"
+! THIN-LTO-SAME: "-flto=thin"
+! THIN-LTO: "-plugin-opt=thinlto"
+
+! ERROR: error: unsupported argument 'somelto' to option '-flto=
diff --git a/flang/test/Driver/omp-frontend-forwarding.f90 b/flang/test/Driver/omp-frontend-forwarding.f90
new file mode 100644
index 0000000000000..063e20b1bee3c
--- /dev/null
+++ b/flang/test/Driver/omp-frontend-forwarding.f90
@@ -0,0 +1,22 @@
+! Test that flang-new OpenMP and OpenMP offload related 
+! commands forward or expand to the appropriate commands 
+! for flang-new -fc1 as expected.
+
+! Test regular -fopenmp with no offload
+! RUN: %flang -### -fopenmp %s 2>&1 | FileCheck --check-prefixes=CHECK-OPENMP %s
+! CHECK-OPENMP: "{{[^"]*}}flang-new" "-fc1" {{.*}} "-fopenmp" {{.*}}.f90"
+! CHECK-OPENMP-NOT: "{{[^"]*}}flang-new" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-device" {{.*}}.f90"
+
+! Test regular -fopenmp with offload for basic fopenmp-is-device flag addition and correct fopenmp 
+! RUN: %flang -### -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa %s 2>&1 | FileCheck --check-prefixes=CHECK-OPENMP-IS-DEVICE %s
+! CHECK-OPENMP-IS-DEVICE: "{{[^"]*}}flang-new" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-device" {{.*}}.f90"
+
+! Testing fembed-offload-object and fopenmp-is-device
+! RUN: %flang -S -### %s -o %t 2>&1 \
+! RUN: -fopenmp --offload-arch=gfx90a \
+! RUN: --target=aarch64-unknown-linux-gnu \
+! RUN:   | FileCheck %s --check-prefixes=CHECK-OPENMP-EMBED
+! CHECK-OPENMP-EMBED: "{{[^"]*}}flang-new" "-fc1" "-triple" "aarch64-unknown-linux-gnu" {{.*}} "-fopenmp" {{.*}}.f90"
+! CHECK-OPENMP-EMBED-NEXT: "{{[^"]*}}flang-new" "-fc1" "-triple" "amdgcn-amd-amdhsa" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-device" {{.*}}.f90"
+! CHECK-OPENMP-EMBED: "{{[^"]*}}clang-offload-packager" {{.*}} "--image=file={{.*}}.bc,triple=amdgcn-amd-amdhsa,arch=gfx90a,kind=openmp"
+! CHECK-OPENMP-EMBED-NEXT: "{{[^"]*}}flang-new" "-fc1" "-triple" "aarch64-unknown-linux-gnu" {{.*}} "-fopenmp" {{.*}} "-fembed-offload-object={{.*}}.out" {{.*}}.bc"
diff --git a/flang/test/Driver/supported-suffices/f03-suffix.f03 b/flang/test/Driver/supported-suffices/f03-suffix.f03
new file mode 100644
index 0000000000000..6e03f9f43fc60
--- /dev/null
+++ b/flang/test/Driver/supported-suffices/f03-suffix.f03
@@ -0,0 +1,5 @@
+! RUN: %flang -### %s 2>&1 | FileCheck %s
+
+! CHECK: "{{.*}}flang-new" "-fc1" {{.*}} "-o" "{{.*}}.o"
+program f03
+end program f03
diff --git a/flang/test/Driver/supported-suffices/f08-suffix.f08 b/flang/test/Driver/supported-suffices/f08-suffix.f08
new file mode 100644
index 0000000000000..d5bcf4ce1de1c
--- /dev/null
+++ b/flang/test/Driver/supported-suffices/f08-suffix.f08
@@ -0,0 +1,5 @@
+! RUN: %flang -### %s 2>&1 | FileCheck %s
+
+! CHECK: "{{.*}}flang-new" "-fc1" {{.*}} "-o" "{{.*}}.o"
+program f08
+end program f08
diff --git a/flang/test/Driver/target-cpu-features-invalid.f90 b/flang/test/Driver/target-cpu-features-invalid.f90
new file mode 100644
index 0000000000000..7ecbe597637c6
--- /dev/null
+++ b/flang/test/Driver/target-cpu-features-invalid.f90
@@ -0,0 +1,13 @@
+! REQUIRES: aarch64-registered-target
+
+! Test that invalid cpu and features are ignored.
+
+! RUN: %flang_fc1 -triple aarch64-linux-gnu -target-cpu supercpu \
+! RUN:   -o /dev/null -S %s 2>&1 | FileCheck %s -check-prefix=CHECK-INVALID-CPU
+
+! RUN: %flang_fc1 -triple aarch64-linux-gnu -target-feature +superspeed \
+! RUN:   -o /dev/null -S %s 2>&1 | FileCheck %s -check-prefix=CHECK-INVALID-FEATURE
+
+
+! CHECK-INVALID-CPU: 'supercpu' is not a recognized processor for this target (ignoring processor)
+! CHECK-INVALID-FEATURE: '+superspeed' is not a recognized feature for this target (ignoring feature)
diff --git a/flang/test/Driver/target-cpu-features.f90 b/flang/test/Driver/target-cpu-features.f90
index 1ce416f0cf533..ca2ed274a6b6e 100644
--- a/flang/test/Driver/target-cpu-features.f90
+++ b/flang/test/Driver/target-cpu-features.f90
@@ -1,5 +1,3 @@
-! REQUIRES: aarch64-registered-target, x86-registered-target
-
 ! Test that -mcpu/march are used and that the -target-cpu and -target-features
 ! are also added to the fc1 command.
 
@@ -22,14 +20,8 @@
 ! RUN: %flang --target=x86_64h-linux-gnu -c %s -### 2>&1 \
 ! RUN: | FileCheck %s -check-prefix=CHECK-X86_64H
 
-
-! Test that invalid cpu and features are ignored.
-
-! RUN: %flang_fc1 -triple aarch64-linux-gnu -target-cpu supercpu \
-! RUN: -o /dev/null -S %s 2>&1 | FileCheck %s -check-prefix=CHECK-INVALID-CPU
-
-! RUN: %flang_fc1 -triple aarch64-linux-gnu -target-feature +superspeed \
-! RUN: -o /dev/null -S %s 2>&1 | FileCheck %s -check-prefix=CHECK-INVALID-FEATURE
+! RUN: %flang --target=riscv64-linux-gnu -c %s -### 2>&1 \
+! RUN: | FileCheck %s -check-prefix=CHECK-RV64
 
 
 ! CHECK-A57: "-fc1" "-triple" "aarch64-unknown-linux-gnu"
@@ -52,5 +44,5 @@
 ! CHECK-X86_64H: "-fc1" "-triple" "x86_64h-unknown-linux-gnu"
 ! CHECK-X86_64H-SAME: "-target-cpu" "x86-64" "-target-feature" "-rdrnd" "-target-feature" "-aes" "-target-feature" "-pclmul" "-target-feature" "-rtm" "-target-feature" "-fsgsbase"
 
-! CHECK-INVALID-CPU: 'supercpu' is not a recognized processor for this target (ignoring processor)
-! CHECK-INVALID-FEATURE: '+superspeed' is not a recognized feature for this target (ignoring feature)
+! CHECK-RV64: "-fc1" "-triple" "riscv64-unknown-linux-gnu"
+! CHECK-RV64-SAME: "-target-cpu" "generic-rv64" "-target-feature" "+m" "-target-feature" "+a" "-target-feature" "+f" "-target-feature" "+d" "-target-feature" "+c"
diff --git a/flang/test/Evaluate/fold-transfer.f90 b/flang/test/Evaluate/fold-transfer.f90
index 833c828a62966..4210fd6fe295d 100644
--- a/flang/test/Evaluate/fold-transfer.f90
+++ b/flang/test/Evaluate/fold-transfer.f90
@@ -39,4 +39,7 @@ module m
   character*5, parameter :: le1c(*) = transfer(le1, [character(5)::])
   character*5, parameter :: be1c(*) = transfer(be1, [character(5)::])
   logical, parameter :: test_i2c_s = all(le1c == ["abcd"//char(0)]) .or. all(be1c == ["efgh"//char(0)])
+
+  character*4, parameter :: i2ss1 = transfer(int(z'61626364', 4), "12345678"(2:5))
+  logical, parameter :: test_i2ss1 = any(i2ss1 == ["abcd", "dcba"])
 end module
diff --git a/flang/test/Examples/print-fns-calls.f90 b/flang/test/Examples/print-fns-calls.f90
index 426de0049663b..f45a0932ee215 100644
--- a/flang/test/Examples/print-fns-calls.f90
+++ b/flang/test/Examples/print-fns-calls.f90
@@ -1,5 +1,5 @@
 ! Check the Flang Print Function Names example plugin doesn't count/print function/subroutine calls (should only count definitions)
-! This requires that the examples are built (FLANG_BUILD_EXAMPLES=ON) to access flangPrintFunctionNames.so
+! This requires that the examples are built (LLVM_BUILD_EXAMPLES=ON) to access flangPrintFunctionNames.so
 
 ! REQUIRES: plugins, examples, shell
 
diff --git a/flang/test/Examples/print-fns-definitions.f90 b/flang/test/Examples/print-fns-definitions.f90
index 98b8e64e66e7b..a1f342a246248 100644
--- a/flang/test/Examples/print-fns-definitions.f90
+++ b/flang/test/Examples/print-fns-definitions.f90
@@ -1,6 +1,6 @@
 ! Check the Flang Print Function Names example plugin prints and counts function/subroutine definitions
 ! This includes internal and external Function/Subroutines, but not Statement Functions
-! This requires that the examples are built (FLANG_BUILD_EXAMPLES=ON) to access flangPrintFunctionNames.so
+! This requires that the examples are built (LLVM_BUILD_EXAMPLES=ON) to access flangPrintFunctionNames.so
 
 ! REQUIRES: plugins, examples, shell
 
diff --git a/flang/test/Examples/print-fns-interfaces.f90 b/flang/test/Examples/print-fns-interfaces.f90
index 0813fd4bd939f..c1e2f3488e232 100644
--- a/flang/test/Examples/print-fns-interfaces.f90
+++ b/flang/test/Examples/print-fns-interfaces.f90
@@ -1,6 +1,6 @@
 ! Check the Flang Print Function Names example plugin doesn't count/print Functions/Subroutines in interfaces
 ! (It should only count definitions, which will appear elsewhere for interfaced functions/subroutines)
-! This requires that the examples are built (FLANG_BUILD_EXAMPLES=ON) to access flangPrintFunctionNames.so
+! This requires that the examples are built (LLVM_BUILD_EXAMPLES=ON) to access flangPrintFunctionNames.so
 
 ! REQUIRES: plugins, examples, shell
 
diff --git a/flang/test/Fir/convert-to-llvm-openmp-and-fir.fir b/flang/test/Fir/convert-to-llvm-openmp-and-fir.fir
index 3a848d1213966..ee94ac19accd7 100644
--- a/flang/test/Fir/convert-to-llvm-openmp-and-fir.fir
+++ b/flang/test/Fir/convert-to-llvm-openmp-and-fir.fir
@@ -485,3 +485,72 @@ func.func @_QPsb() {
 // CHECK:    }
 // CHECK:    llvm.return
 // CHECK:  }
+
+// -----
+
+// CHECK:  omp.reduction.declare @[[EQV_REDUCTION:.*]] : i32 init {
+// CHECK:  ^bb0(%{{.*}}: i32):
+// CHECK:    %[[TRUE:.*]] = llvm.mlir.constant(true) : i1
+// CHECK:    %[[TRUE_EXT:.*]] = llvm.zext %[[TRUE]] : i1 to i32
+// CHECK:    omp.yield(%[[TRUE_EXT]] : i32)
+// CHECK:  } combiner {
+// CHECK:  ^bb0(%[[ARG_1:.*]]: i32, %[[ARG_2:.*]]: i32):
+// CHECK:    %[[ZERO_1:.*]] = llvm.mlir.constant(0 : i64) : i32
+// CHECK:    %[[ARGVAL_1:.*]] = llvm.icmp "ne" %[[ARG_1]], %[[ZERO_1]] : i32
+// CHECK:    %[[ZERO_2:.*]] = llvm.mlir.constant(0 : i64) : i32
+// CHECK:    %[[ARGVAL_2:.*]] = llvm.icmp "ne" %[[ARG_2]], %[[ZERO_2]] : i32
+// CHECK:    %[[RES:.*]] = llvm.icmp "eq" %[[ARGVAL_1]], %[[ARGVAL_2]] : i1
+// CHECK:    %[[RES_EXT:.*]] = llvm.zext %[[RES]] : i1 to i32
+// CHECK:    omp.yield(%[[RES_EXT]] : i32)
+// CHECK:  } 
+// CHECK-LABEL:  @_QPsimple_reduction
+// CHECK-SAME: %[[ARRAY_REF:.*]]: !llvm.ptr<array<100 x i32>>
+// CHECK:    %[[RED_ACCUMULATOR:.*]] = llvm.alloca %2 x i32 {bindc_name = "x", in_type = !fir.logical<4>, operand_segment_sizes = array<i32: 0, 0>, uniq_name = "_QFsimple_reductionEx"} : (i64) -> !llvm.ptr<i32>
+// CHECK:    omp.parallel   {
+// CHECK:      omp.wsloop   reduction(@[[EQV_REDUCTION]] -> %[[RED_ACCUMULATOR]] : !llvm.ptr<i32>) for
+// CHECK:        %[[ARRAY_ELEM_REF:.*]] = llvm.getelementptr %[[ARRAY_REF]][0, %{{.*}}] : (!llvm.ptr<array<100 x i32>>, i64) -> !llvm.ptr<i32>
+// CHECK:        %[[ARRAY_ELEM:.*]] = llvm.load %[[ARRAY_ELEM_REF]] : !llvm.ptr<i32>
+// CHECK:        omp.reduction %[[ARRAY_ELEM]], %[[RED_ACCUMULATOR]] : i32, !llvm.ptr<i32>
+// CHECK:        omp.yield
+// CHECK:      omp.terminator
+// CHECK:    llvm.return
+
+omp.reduction.declare @eqv_reduction : !fir.logical<4> init {
+^bb0(%arg0: !fir.logical<4>):
+  %true = arith.constant true
+  %0 = fir.convert %true : (i1) -> !fir.logical<4>
+  omp.yield(%0 : !fir.logical<4>)
+} combiner {
+^bb0(%arg0: !fir.logical<4>, %arg1: !fir.logical<4>):
+  %0 = fir.convert %arg0 : (!fir.logical<4>) -> i1
+  %1 = fir.convert %arg1 : (!fir.logical<4>) -> i1
+  %2 = arith.cmpi eq, %0, %1 : i1
+  %3 = fir.convert %2 : (i1) -> !fir.logical<4>
+  omp.yield(%3 : !fir.logical<4>)
+} 
+func.func @_QPsimple_reduction(%arg0: !fir.ref<!fir.array<100x!fir.logical<4>>> {fir.bindc_name = "y"}) {
+  %0 = fir.alloca i32 {bindc_name = "i", uniq_name = "_QFsimple_reductionEi"}
+  %1 = fir.alloca !fir.logical<4> {bindc_name = "x", uniq_name = "_QFsimple_reductionEx"}
+  %true = arith.constant true
+  %2 = fir.convert %true : (i1) -> !fir.logical<4>
+  fir.store %2 to %1 : !fir.ref<!fir.logical<4>>
+  omp.parallel   {
+    %3 = fir.alloca i32 {adapt.valuebyref, pinned}
+    %c1_i32 = arith.constant 1 : i32
+    %c100_i32 = arith.constant 100 : i32
+    %c1_i32_0 = arith.constant 1 : i32
+    omp.wsloop   reduction(@eqv_reduction -> %1 : !fir.ref<!fir.logical<4>>) for  (%arg1) : i32 = (%c1_i32) to (%c100_i32) inclusive step (%c1_i32_0) {
+      fir.store %arg1 to %3 : !fir.ref<i32>
+      %4 = fir.load %3 : !fir.ref<i32>
+      %5 = fir.convert %4 : (i32) -> i64
+      %c1_i64 = arith.constant 1 : i64
+      %6 = arith.subi %5, %c1_i64 : i64
+      %7 = fir.coordinate_of %arg0, %6 : (!fir.ref<!fir.array<100x!fir.logical<4>>>, i64) -> !fir.ref<!fir.logical<4>>
+      %8 = fir.load %7 : !fir.ref<!fir.logical<4>>
+      omp.reduction %8, %1 : !fir.logical<4>, !fir.ref<!fir.logical<4>>
+      omp.yield
+    }
+    omp.terminator
+  }
+  return
+}
diff --git a/flang/test/Fir/convert-to-llvm.fir b/flang/test/Fir/convert-to-llvm.fir
index 85b8f666a18f8..cbfb2283ccbc6 100644
--- a/flang/test/Fir/convert-to-llvm.fir
+++ b/flang/test/Fir/convert-to-llvm.fir
@@ -739,6 +739,16 @@ func.func @convert_from_int(%arg0 : i32) {
 // CHECK:         %{{.*}} = llvm.sext %[[ARG0]] : i32 to i64
 // CHECK:         %{{.*}} = llvm.inttoptr %{{.*}} : i64 to !llvm.ptr<i64>
 
+
+func.func @convert_from_i1(%arg0 : i1) {
+  %0 = fir.convert %arg0 : (i1) -> i32
+  return
+}
+
+// CHECK-LABEL: convert_from_i1(
+// CHECK-SAME: %[[ARG0:.*]]: i1
+// CHECK:        %{{.*}} = llvm.zext %[[ARG0]] : i1 to i32 
+
 // -----
 
 // Test `fir.convert` operation conversion from !fir.ref<> type.
diff --git a/flang/test/Fir/invalid-types.fir b/flang/test/Fir/invalid-types.fir
index 67360df271868..18bc9f7b4896a 100644
--- a/flang/test/Fir/invalid-types.fir
+++ b/flang/test/Fir/invalid-types.fir
@@ -167,3 +167,8 @@ func.func private @oth3() -> !fir.vector<10:>
 
 // expected-error@+1 {{invalid element type}}
 func.func private @upe() -> !fir.class<!fir.box<i32>>
+
+// -----
+
+// expected-error@+1 {{invalid element type}}
+func.func private @upe() -> !fir.box<!fir.class<none>>
diff --git a/flang/test/Fir/target-rewrite-integer.fir b/flang/test/Fir/target-rewrite-integer.fir
index 7bc4e4a31a3aa..ad64ca80b4733 100644
--- a/flang/test/Fir/target-rewrite-integer.fir
+++ b/flang/test/Fir/target-rewrite-integer.fir
@@ -79,3 +79,36 @@ func.func @_QPtest_ui1(%arg0: !fir.ref<!fir.logical<4>> {fir.bindc_name = "x"})
   return
 }
 func.func private @_SomeFunc_ui1(ui1) -> ui1 attributes {fir.runtime}
+
+// -----
+
+// subroutine test(x, y)
+//   use iso_c_binding
+//   interface
+//      integer(c_int8_t) function cfun8(x) bind(C)
+//        integer(c_int8_t),value :: x
+//      end function cfun8
+//      integer(c_int16_t) function cfun16(x) bind(C)
+//        integer(c_int16_t),value :: x
+//      end function cfun16
+//   end interface
+//   integer(c_int8_t) :: x
+//   integer(c_int16_t) :: y
+//   x = cfun8(x)
+//   y = cfun16(y)
+// end subroutine test
+
+// ALL-LABEL: @_QPtest_bindc
+// ALL: func.func private @cfun8(i8 {llvm.signext}) -> (i8 {llvm.signext}) attributes {fir.bindc_name = "cfun8"}
+// ALL: func.func private @cfun16(i16 {llvm.signext}) -> (i16 {llvm.signext}) attributes {fir.bindc_name = "cfun16"}
+func.func @_QPtest_bindc(%arg0: !fir.ref<i8> {fir.bindc_name = "x"}, %arg1: !fir.ref<i16> {fir.bindc_name = "y"}) {
+  %0 = fir.load %arg0 : !fir.ref<i8>
+  %1 = fir.call @cfun8(%0) fastmath<contract> : (i8) -> i8
+  fir.store %1 to %arg0 : !fir.ref<i8>
+  %2 = fir.load %arg1 : !fir.ref<i16>
+  %3 = fir.call @cfun16(%2) fastmath<contract> : (i16) -> i16
+  fir.store %3 to %arg1 : !fir.ref<i16>
+  return
+}
+func.func private @cfun8(i8) -> i8 attributes {fir.bindc_name = "cfun8"}
+func.func private @cfun16(i16) -> i16 attributes {fir.bindc_name = "cfun16"}
diff --git a/flang/test/HLFIR/assign-codegen.fir b/flang/test/HLFIR/assign-codegen.fir
index 06c4ae1ab18c5..6da5acc7dec2d 100644
--- a/flang/test/HLFIR/assign-codegen.fir
+++ b/flang/test/HLFIR/assign-codegen.fir
@@ -183,3 +183,26 @@ func.func @alloc_assign(%arg0: !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>,
 // CHECK:  %[[VAL_2:.*]] = fir.convert %[[VAL_0]] : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>) -> !fir.ref<!fir.box<none>>
 // CHECK:  %[[VAL_3:.*]] = fir.convert %[[VAL_1]] : (!fir.box<!fir.array<?xi32>>) -> !fir.box<none>
 // CHECK:  fir.call @_FortranAAssign(%[[VAL_2]], %[[VAL_3]], %{{.*}}, %{{.*}}) : (!fir.ref<!fir.box<none>>, !fir.box<none>, !fir.ref<i8>, i32) -> none
+
+
+func.func @test_alloc_assign_explicit_length_character(%lhs: !fir.ref<!fir.box<!fir.heap<!fir.array<?x!fir.char<1,10>>>>>, %rhs: !fir.box<!fir.array<?x!fir.char<1,?>>>) {
+  hlfir.assign %rhs to %lhs realloc keep_lhs_len : !fir.box<!fir.array<?x!fir.char<1,?>>>, !fir.ref<!fir.box<!fir.heap<!fir.array<?x!fir.char<1,10>>>>>
+  return
+}
+// CHECK-LABEL:   func.func @test_alloc_assign_explicit_length_character(
+// CHECK-SAME:  %[[VAL_0:.*]]: !fir.ref<!fir.box<!fir.heap<!fir.array<?x!fir.char<1,10>>>>>,
+// CHECK-SAME:  %[[VAL_1:.*]]: !fir.box<!fir.array<?x!fir.char<1,?>>>) {
+// CHECK:  %[[VAL_7:.*]] = fir.convert %[[VAL_0]] : (!fir.ref<!fir.box<!fir.heap<!fir.array<?x!fir.char<1,10>>>>>) -> !fir.ref<!fir.box<none>>
+// CHECK:  %[[VAL_8:.*]] = fir.convert %[[VAL_1]] : (!fir.box<!fir.array<?x!fir.char<1,?>>>) -> !fir.box<none>
+// CHECK:  %[[VAL_10:.*]] = fir.call @_FortranAAssignExplicitLengthCharacter(%[[VAL_7]], %[[VAL_8]], %{{.*}}, %{{.*}}) : (!fir.ref<!fir.box<none>>, !fir.box<none>, !fir.ref<i8>, i32) -> none
+
+func.func @test_alloc_assign_polymorphic(%lhs: !fir.ref<!fir.class<!fir.heap<!fir.array<?x!fir.type<t>>>>>, %rhs: !fir.class<!fir.array<?x!fir.type<t>>>) {
+  hlfir.assign %rhs to %lhs realloc : !fir.class<!fir.array<?x!fir.type<t>>>, !fir.ref<!fir.class<!fir.heap<!fir.array<?x!fir.type<t>>>>>
+  return
+}
+// CHECK-LABEL:   func.func @test_alloc_assign_polymorphic(
+// CHECK-SAME:  %[[VAL_0:.*]]: !fir.ref<!fir.class<!fir.heap<!fir.array<?x!fir.type<t>>>>>,
+// CHECK-SAME:  %[[VAL_1:.*]]: !fir.class<!fir.array<?x!fir.type<t>>>) {
+// CHECK:  %[[VAL_7:.*]] = fir.convert %[[VAL_0]] : (!fir.ref<!fir.class<!fir.heap<!fir.array<?x!fir.type<t>>>>>) -> !fir.ref<!fir.box<none>>
+// CHECK:  %[[VAL_8:.*]] = fir.convert %[[VAL_1]] : (!fir.class<!fir.array<?x!fir.type<t>>>) -> !fir.box<none>
+// CHECK:  %[[VAL_10:.*]] = fir.call @_FortranAAssignPolymorphic(%[[VAL_7]], %[[VAL_8]], %{{.*}}, %{{.*}}) : (!fir.ref<!fir.box<none>>, !fir.box<none>, !fir.ref<i8>, i32) -> none
diff --git a/flang/test/HLFIR/invalid.fir b/flang/test/HLFIR/invalid.fir
index 2ec7c689b5ea1..a8ba337ad8b5a 100644
--- a/flang/test/HLFIR/invalid.fir
+++ b/flang/test/HLFIR/invalid.fir
@@ -397,6 +397,48 @@ func.func @bad_transpose3(%arg0: !hlfir.expr<2x3xi32>) {
   return
 }
 
+// -----
+func.func @bad_matmultranspose1(%arg0: !hlfir.expr<?x?x?xi32>, %arg1: !hlfir.expr<?x?xi32>) {
+  // expected-error@+1 {{'hlfir.matmul_transpose' op array must have either rank 1 or rank 2}}
+  %0 = hlfir.matmul_transpose %arg0 %arg1 : (!hlfir.expr<?x?x?xi32>, !hlfir.expr<?x?xi32>) -> !hlfir.expr<?x?xi32>
+  return
+}
+
+// -----
+func.func @bad_matmultranspose2(%arg0: !hlfir.expr<?xi32>, %arg1: !hlfir.expr<?xi32>) {
+  // expected-error@+1 {{'hlfir.matmul_transpose' op array must have either rank 1 or rank 2}}
+  %0 = hlfir.matmul_transpose %arg0 %arg1 : (!hlfir.expr<?xi32>, !hlfir.expr<?xi32>) -> !hlfir.expr<?x?xi32>
+  return
+}
+
+// -----
+func.func @bad_matmultranspose3(%arg0: !hlfir.expr<?x?x!fir.logical<4>>, %arg1: !hlfir.expr<?x?xi32>) {
+  // expected-error@+1 {{'hlfir.matmul_transpose' op if one array is logical, so should the other be}}
+  %0 = hlfir.matmul_transpose %arg0 %arg1 : (!hlfir.expr<?x?x!fir.logical<4>>, !hlfir.expr<?x?xi32>) -> !hlfir.expr<?x?xi32>
+  return
+}
+
+// -----
+func.func @bad_matmultranspose5(%arg0: !hlfir.expr<?x?xi32>, %arg1: !hlfir.expr<?x?xi32>) {
+  // expected-error@+1 {{'hlfir.matmul_transpose' op the result type should be a logical only if the argument types are logical}}
+  %0 = hlfir.matmul_transpose %arg0 %arg1 : (!hlfir.expr<?x?xi32>, !hlfir.expr<?x?xi32>) -> !hlfir.expr<?x?x!fir.logical<4>>
+  return
+}
+
+// -----
+func.func @bad_matmultranspose6(%arg0: !hlfir.expr<2x1xi32>, %arg1: !hlfir.expr<2x3xi32>) {
+  // expected-error@+1 {{'hlfir.matmul_transpose' op incorrect result shape}}
+  %0 = hlfir.matmul_transpose %arg0 %arg1 : (!hlfir.expr<2x1xi32>, !hlfir.expr<2x3xi32>) -> !hlfir.expr<10x30xi32>
+  return
+}
+
+// -----
+func.func @bad_matmultranspose7(%arg0: !hlfir.expr<2x1xi32>, %arg1: !hlfir.expr<2xi32>) {
+  // expected-error@+1 {{'hlfir.matmul_transpose' op incorrect result shape}}
+  %0 = hlfir.matmul_transpose %arg0 %arg1 : (!hlfir.expr<2x1xi32>, !hlfir.expr<2xi32>) -> !hlfir.expr<1x3xi32>
+  return
+}
+
 // -----
 func.func @bad_assign_1(%arg0: !fir.box<!fir.array<?xi32>>, %arg1: !fir.box<!fir.array<?xi32>>) {
   // expected-error@+1 {{'hlfir.assign' op lhs must be an allocatable when `realloc` is set}}
diff --git a/flang/test/HLFIR/matmul-bufferization.fir b/flang/test/HLFIR/matmul-lowering.fir
similarity index 87%
rename from flang/test/HLFIR/matmul-bufferization.fir
rename to flang/test/HLFIR/matmul-lowering.fir
index 54da40cf303f9..d4819f18c62af 100644
--- a/flang/test/HLFIR/matmul-bufferization.fir
+++ b/flang/test/HLFIR/matmul-lowering.fir
@@ -1,5 +1,5 @@
 // Test hlfir.matmul operation lowering to fir runtime call
-// RUN: fir-opt %s -bufferize-hlfir | FileCheck %s
+// RUN: fir-opt %s -lower-hlfir-intrinsics | FileCheck %s
 
 func.func @_QPmatmul1(%arg0: !fir.box<!fir.array<?x?xi32>> {fir.bindc_name = "lhs"}, %arg1: !fir.box<!fir.array<?x?xi32>> {fir.bindc_name = "rhs"}, %arg2: !fir.box<!fir.array<?x?xi32>> {fir.bindc_name = "res"}) {
   %0:2 = hlfir.declare %arg0 {uniq_name = "_QFmatmul1Elhs"} : (!fir.box<!fir.array<?x?xi32>>) -> (!fir.box<!fir.array<?x?xi32>>, !fir.box<!fir.array<?x?xi32>>)
@@ -36,10 +36,10 @@ func.func @_QPmatmul1(%arg0: !fir.box<!fir.array<?x?xi32>> {fir.bindc_name = "lh
 // CHECK-NEXT:    %[[SHIFT:.*]] = fir.shape_shift %[[BOX_DIMS]]#0, %[[BOX_DIMS]]#1
 // TODO: fix alias analysis in hlfir.assign bufferization
 // CHECK-NEXT:    %[[TMP:.*]]:2 = hlfir.declare %[[ADDR]](%[[SHIFT]]) {uniq_name = ".tmp.intrinsic_result"}
-// CHECK:         %[[TUPLE0:.*]] = fir.undefined tuple<!fir.box<!fir.array<?x?xi32>>, i1>
-// CHECK:         %[[TUPLE1:.*]] = fir.insert_value %[[TUPLE0]], %[[TRUE:.*]], [1 : index]
-// CHECK:         %[[TUPLE2:.*]] = fir.insert_value %[[TUPLE1]], %[[TMP]]#0, [0 : index]
-// CHECK:         hlfir.assign %[[TMP]]#0 to %[[RES_VAR]]#0
-// CHECK:         fir.freemem %[[TMP]]#1
+// TODO: add shape information from original intrinsic op
+// CHECK:         %[[TRUE:.*]] = arith.constant true
+// CHECK:         %[[ASEXPR:.*]] = hlfir.as_expr %[[TMP]]#0 move %[[TRUE]] : (!fir.box<!fir.array<?x?xi32>>, i1) -> !hlfir.expr<?x?xi32>
+// CHECK:         hlfir.assign %[[ASEXPR]] to %[[RES_VAR]]#0
+// CHECK:         hlfir.destroy %[[ASEXPR]]
 // CHECK-NEXT:    return
 // CHECK-NEXT:  }
diff --git a/flang/test/HLFIR/matmul_transpose.fir b/flang/test/HLFIR/matmul_transpose.fir
new file mode 100644
index 0000000000000..967edecb463b2
--- /dev/null
+++ b/flang/test/HLFIR/matmul_transpose.fir
@@ -0,0 +1,87 @@
+// Test hlfir.matmul_transpose operation parse, verify (no errors), and unparse
+
+// RUN: fir-opt %s | fir-opt | FileCheck %s
+
+// arguments are expressions of known shape
+func.func @matmul_transpose0(%arg0: !hlfir.expr<2x2xi32>, %arg1: !hlfir.expr<2x2xi32>) {
+  %res = hlfir.matmul_transpose %arg0 %arg1 : (!hlfir.expr<2x2xi32>, !hlfir.expr<2x2xi32>) -> !hlfir.expr<2x2xi32>
+  return
+}
+// CHECK-LABEL: func.func @matmul_transpose0
+// CHECK:           %[[ARG0:.*]]: !hlfir.expr<2x2xi32>,
+// CHECK:           %[[ARG1:.*]]: !hlfir.expr<2x2xi32>) {
+// CHECK-NEXT:    %[[RES:.*]] = hlfir.matmul_transpose %[[ARG0]] %[[ARG1]] : (!hlfir.expr<2x2xi32>, !hlfir.expr<2x2xi32>) -> !hlfir.expr<2x2xi32>
+// CHECK-NEXT:    return
+// CHECK-NEXT:  }
+
+// arguments are expressions of assumed shape
+func.func @matmul_transpose1(%arg0: !hlfir.expr<?x?xi32>, %arg1: !hlfir.expr<?x?xi32>) {
+  %res = hlfir.matmul_transpose %arg0 %arg1 : (!hlfir.expr<?x?xi32>, !hlfir.expr<?x?xi32>) -> !hlfir.expr<?x?xi32>
+  return
+}
+// CHECK-LABEL: func.func @matmul_transpose1
+// CHECK:           %[[ARG0:.*]]: !hlfir.expr<?x?xi32>,
+// CHECK:           %[[ARG1:.*]]: !hlfir.expr<?x?xi32>) {
+// CHECK-NEXT:    %[[RES:.*]] = hlfir.matmul_transpose %[[ARG0]] %[[ARG1]] : (!hlfir.expr<?x?xi32>, !hlfir.expr<?x?xi32>) -> !hlfir.expr<?x?xi32>
+// CHECK-NEXT:    return
+// CHECK-NEXT:  }
+
+// arguments are expressions where only some dimensions are known #1
+func.func @matmul_transpose2(%arg0: !hlfir.expr<?x2xi32>, %arg1: !hlfir.expr<?x2xi32>) {
+  %res = hlfir.matmul_transpose %arg0 %arg1 : (!hlfir.expr<?x2xi32>, !hlfir.expr<?x2xi32>) -> !hlfir.expr<2x2xi32>
+  return
+}
+// CHECK-LABEL: func.func @matmul_transpose2
+// CHECK:           %[[ARG0:.*]]: !hlfir.expr<?x2xi32>,
+// CHECK:           %[[ARG1:.*]]: !hlfir.expr<?x2xi32>) {
+// CHECK-NEXT:    %[[RES:.*]] = hlfir.matmul_transpose %[[ARG0]] %[[ARG1]] : (!hlfir.expr<?x2xi32>, !hlfir.expr<?x2xi32>) -> !hlfir.expr<2x2xi32>
+// CHECK-NEXT:    return
+// CHECK-NEXT:  }
+
+// arguments are expressions where only some dimensions are known #2
+func.func @matmul_transpose3(%arg0: !hlfir.expr<2x?xi32>, %arg1: !hlfir.expr<2x?xi32>) {
+  %res = hlfir.matmul_transpose %arg0 %arg1 : (!hlfir.expr<2x?xi32>, !hlfir.expr<2x?xi32>) -> !hlfir.expr<?x?xi32>
+  return
+}
+// CHECK-LABEL: func.func @matmul_transpose3
+// CHECK:           %[[ARG0:.*]]: !hlfir.expr<2x?xi32>,
+// CHECK:           %[[ARG1:.*]]: !hlfir.expr<2x?xi32>) {
+// CHECK-NEXT:    %[[RES:.*]] = hlfir.matmul_transpose %[[ARG0]] %[[ARG1]] : (!hlfir.expr<2x?xi32>, !hlfir.expr<2x?xi32>) -> !hlfir.expr<?x?xi32>
+// CHECK-NEXT:    return
+// CHECK-NEXT:  }
+
+// arguments are logicals
+func.func @matmul_transpose4(%arg0: !hlfir.expr<?x?x!fir.logical<4>>, %arg1: !hlfir.expr<?x?x!fir.logical<4>>) {
+  %res = hlfir.matmul_transpose %arg0 %arg1 : (!hlfir.expr<?x?x!fir.logical<4>>, !hlfir.expr<?x?x!fir.logical<4>>) -> !hlfir.expr<?x?x!fir.logical<4>>
+  return
+}
+// CHECK-LABEL: func.func @matmul_transpose4
+// CHECK:           %[[ARG0:.*]]: !hlfir.expr<?x?x!fir.logical<4>>,
+// CHECK:           %[[ARG1:.*]]: !hlfir.expr<?x?x!fir.logical<4>>) {
+// CHECK-NEXT:    %[[RES:.*]] = hlfir.matmul_transpose %[[ARG0]] %[[ARG1]] : (!hlfir.expr<?x?x!fir.logical<4>>, !hlfir.expr<?x?x!fir.logical<4>>) -> !hlfir.expr<?x?x!fir.logical<4>>
+// CHECK-NEXT:    return
+// CHECK-NEXT:  }
+
+// rhs is rank 1
+func.func @matmul_transpose6(%arg0: !hlfir.expr<?x?xi32>, %arg1: !hlfir.expr<?xi32>) {
+  %res = hlfir.matmul_transpose %arg0 %arg1 : (!hlfir.expr<?x?xi32>, !hlfir.expr<?xi32>) -> !hlfir.expr<?xi32>
+  return
+}
+// CHECK-LABEL: func.func @matmul_transpose6
+// CHECK:           %[[ARG0:.*]]: !hlfir.expr<?x?xi32>,
+// CHECK:           %[[ARG1:.*]]: !hlfir.expr<?xi32>) {
+// CHECK-NEXT:    %[[RES:.*]] = hlfir.matmul_transpose %[[ARG0]] %[[ARG1]] : (!hlfir.expr<?x?xi32>, !hlfir.expr<?xi32>) -> !hlfir.expr<?xi32>
+// CHECK-NEXT:    return
+// CHECK-NEXT:  }
+
+// arguments are boxed arrays
+func.func @matmul_transpose7(%arg0: !fir.box<!fir.array<2x2xf32>>, %arg1: !fir.box<!fir.array<2x2xf32>>) {
+  %res = hlfir.matmul_transpose %arg0 %arg1 : (!fir.box<!fir.array<2x2xf32>>, !fir.box<!fir.array<2x2xf32>>) -> !hlfir.expr<2x2xf32>
+  return
+}
+// CHECK-LABEL: func.func @matmul_transpose7
+// CHECK:           %[[ARG0:.*]]: !fir.box<!fir.array<2x2xf32>>,
+// CHECK:           %[[ARG1:.*]]: !fir.box<!fir.array<2x2xf32>>) {
+// CHECK-NEXT:    %[[RES:.*]] = hlfir.matmul_transpose %[[ARG0]] %[[ARG1]] : (!fir.box<!fir.array<2x2xf32>>, !fir.box<!fir.array<2x2xf32>>) -> !hlfir.expr<2x2xf32>
+// CHECK-NEXT:    return
+// CHECK-NEXT:  }
diff --git a/flang/test/HLFIR/mul_transpose.f90 b/flang/test/HLFIR/mul_transpose.f90
new file mode 100644
index 0000000000000..c79c742efc441
--- /dev/null
+++ b/flang/test/HLFIR/mul_transpose.f90
@@ -0,0 +1,103 @@
+! RUN: bbc -emit-fir -hlfir %s -o - | FileCheck --check-prefix CHECK-BASE --check-prefix CHECK-ALL %s
+! RUN: bbc -emit-fir -hlfir %s -o - | fir-opt --canonicalize | FileCheck --check-prefix CHECK-CANONICAL --check-prefix CHECK-ALL %s
+! RUN: bbc -emit-fir -hlfir %s -o - | fir-opt --lower-hlfir-intrinsics | FileCheck --check-prefix CHECK-LOWERING --check-prefix CHECK-ALL %s
+! RUN: bbc -emit-fir -hlfir %s -o - | fir-opt --canonicalize | fir-opt --lower-hlfir-intrinsics | FileCheck --check-prefix CHECK-LOWERING-OPT --check-prefix CHECK-ALL %s
+! RUN: bbc -emit-fir -hlfir %s -o - | fir-opt --lower-hlfir-intrinsics | fir-opt --bufferize-hlfir | FileCheck --check-prefix CHECK-BUFFERING --check-prefix CHECK-ALL %s
+
+! Test passing a hlfir.expr from one intrinsic to another
+subroutine mul_transpose(a, b, res)
+  real a(2,1), b(2,2), res(1,2)
+  res = MATMUL(TRANSPOSE(a), b)
+endsubroutine
+
+! CHECK-ALL-LABEL:    func.func @_QPmul_transpose
+! CHECK-ALL:              %[[A_ARG:.*]]: !fir.ref<!fir.array<2x1xf32>> {fir.bindc_name = "a"}
+! CHECK-ALL:              %[[B_ARG:.*]]: !fir.ref<!fir.array<2x2xf32>> {fir.bindc_name = "b"}
+! CHECK-ALL:              %[[RES_ARG:.*]]: !fir.ref<!fir.array<1x2xf32>> {fir.bindc_name = "res"}
+! CHECK-ALL-DAG:        %[[A_DECL:.*]]:2 = hlfir.declare %[[A_ARG]]
+! CHECK-ALL-DAG:        %[[B_DECL:.*]]:2 = hlfir.declare %[[B_ARG]]
+! CHECK-ALL-DAG:        %[[RES_DECL:.*]]:2 = hlfir.declare %[[RES_ARG]]
+
+! CHECK-BASE:           %[[TRANSPOSE_RES:.*]] = hlfir.transpose %[[A_DECL]]#0 : (!fir.ref<!fir.array<2x1xf32>>) -> !hlfir.expr<1x2xf32>
+! CHECK-BASE-NEXT:      %[[MATMUL_RES:.*]] = hlfir.matmul %[[TRANSPOSE_RES]] %[[B_DECL]]#0 {{.*}}: (!hlfir.expr<1x2xf32>, !fir.ref<!fir.array<2x2xf32>>) -> !hlfir.expr<1x2xf32>
+! CHECK-BASE-NEXT:      hlfir.assign %[[MATMUL_RES]] to %[[RES_DECL]]#0 : !hlfir.expr<1x2xf32>, !fir.ref<!fir.array<1x2xf32>>
+! CHECK-BASE-NEXT:      hlfir.destroy %[[MATMUL_RES]]
+! CHECK-BASE-NEXT:      hlfir.destroy %[[TRANSPOSE_RES]]
+
+! CHECK-CANONICAL-NEXT: %[[CHAIN_RES:.*]] = hlfir.matmul_transpose %[[A_DECL]]#0 %[[B_DECL]]#0 : (!fir.ref<!fir.array<2x1xf32>>, !fir.ref<!fir.array<2x2xf32>>) -> !hlfir.expr<1x2xf32>
+! CHECK-CANONICAL-NEXT: hlfir.assign %[[CHAIN_RES]] to %[[RES_DECL]]#0 : !hlfir.expr<1x2xf32>, !fir.ref<!fir.array<1x2xf32>>
+! CHECK-CANONICAL-NEXT: hlfir.destroy %[[CHAIN_RES]]
+
+! CHECK-LOWERING:       %[[A_BOX:.*]] = fir.embox %[[A_DECL]]#1(%{{.*}})
+! CHECK-LOWERING:       %[[TRANSPOSE_CONV_RES:.*]] = fir.convert %[[TRANSPOSE_RES_BOX:.*]] : (!fir.ref<!fir.box<!fir.heap<!fir.array<?x?xf32>>>>) -> !fir.ref<!fir.box<none>>
+! CHECK-LOWERING:       %[[A_BOX_CONV:.*]] = fir.convert %[[A_BOX]] : (!fir.box<!fir.array<2x1xf32>>) -> !fir.box<none>
+! CHECK-LOWERING:       fir.call @_FortranATranspose(%[[TRANSPOSE_CONV_RES]], %[[A_BOX_CONV]], %[[LOC_STR1:.*]], %[[LOC_N1:.*]])
+! CHECK-LOWERING:       %[[TRANSPOSE_RES_LD:.*]] = fir.load %[[TRANSPOSE_RES_BOX:.*]]
+! CHECK-LOWERING:       %[[TRANSPOSE_RES_ADDR:.*]] = fir.box_addr %[[TRANSPOSE_RES_LD]]
+! CHECK-LOWERING:       %[[TRANSPOSE_RES_VAR:.*]]:2 = hlfir.declare %[[TRANSPOSE_RES_ADDR]]({{.*}}) {uniq_name = ".tmp.intrinsic_result"}
+! CHECK-LOWERING:       %[[TRUE:.*]] = arith.constant true
+! CHECK-LOWERING:       %[[TRANSPOSE_EXPR:.*]] = hlfir.as_expr %[[TRANSPOSE_RES_VAR]]#0 move %[[TRUE]] : (!fir.box<!fir.array<?x?xf32>>, i1) -> !hlfir.expr<?x?xf32>
+! CHECK-LOWERING:       %[[TRANSPOSE_ASSOC:.*]]:3 = hlfir.associate %[[TRANSPOSE_EXPR]]({{.*}}) {uniq_name = "adapt.valuebyref"}
+! CHECK-LOWERING:           (!hlfir.expr<?x?xf32>, !fir.shape<2>) -> (!fir.box<!fir.array<?x?xf32>>, !fir.ref<!fir.array<?x?xf32>>, i1)
+
+! CHECK-LOWERING:       %[[LHS_BOX:.*]] = fir.embox %[[TRANSPOSE_ASSOC]]#1
+! CHECK-LOWERING:       %[[B_BOX:.*]] = fir.embox %[[B_DECL]]#1(%{{.*}})
+! CHECK-LOWERING:       %[[MUL_CONV_RES:.*]] = fir.convert %[[MUL_RES_BOX:.*]] : (!fir.ref<!fir.box<!fir.heap<!fir.array<?x?xf32>>>>) -> !fir.ref<!fir.box<none>>
+! CHECK-LOWERING:       %[[LHS_CONV:.*]] = fir.convert %[[LHS_BOX]] : (!fir.box<!fir.array<?x?xf32>>) -> !fir.box<none>
+! CHECK-LOWERING:       %[[B_BOX_CONV:.*]] = fir.convert %[[B_BOX]] : (!fir.box<!fir.array<2x2xf32>>) -> !fir.box<none>
+! CHECK-LOWERING:       fir.call @_FortranAMatmul(%[[MUL_CONV_RES]], %[[LHS_CONV]], %[[B_BOX_CONV]], %[[LOC_STR2:.*]], %[[LOC_N2:.*]])
+! CHECK-LOWERING:       %[[MUL_RES_LD:.*]] = fir.load %[[MUL_RES_BOX:.*]]
+! CHECK-LOWERING:       %[[MUL_RES_ADDR:.*]] = fir.box_addr %[[MUL_RES_LD]]
+! CHECK-LOWERING:       %[[MUL_RES_VAR:.*]]:2 = hlfir.declare %[[MUL_RES_ADDR]]({{.*}}) {uniq_name = ".tmp.intrinsic_result"}
+! CHECK-LOWERING:       %[[TRUE2:.*]] = arith.constant true
+! CHECK-LOWERING:       %[[MUL_EXPR:.*]] = hlfir.as_expr %[[MUL_RES_VAR]]#0 move %[[TRUE2]] : (!fir.box<!fir.array<?x?xf32>>, i1) -> !hlfir.expr<?x?xf32>
+
+! CHECK-LOWERING:       hlfir.end_associate %[[TRANSPOSE_ASSOC]]#1, %[[TRANSPOSE_ASSOC]]#2 : !fir.ref<!fir.array<?x?xf32>>, i1
+! CHECK-LOWERING-NEXT:  hlfir.assign %[[MUL_EXPR]] to %[[RES_DECL]]#0 : !hlfir.expr<?x?xf32>, !fir.ref<!fir.array<1x2xf32>>
+! CHECK-LOWERING-NEXT:  hlfir.destroy %[[MUL_EXPR]]
+! CHECK-LOWERING-NEXT:  hlfir.destroy %[[TRANSPOSE_EXPR]]
+
+! CHECK-LOWERING-OPT:   %[[LHS_BOX:.*]] = fir.embox %[[A_DECL]]#1(%{{.*}})
+! CHECK-LOWERING-OPT:   %[[B_BOX:.*]] = fir.embox %[[B_DECL]]#1(%{{.*}})
+! CHECK-LOWERING-OPT:   %[[MUL_CONV_RES:.*]] = fir.convert %[[MUL_RES_BOX:.*]] : (!fir.ref<!fir.box<!fir.heap<!fir.array<?x?xf32>>>>) -> !fir.ref<!fir.box<none>>
+! CHECK-LOWERING-OPT:   %[[LHS_CONV:.*]] = fir.convert %[[LHS_BOX]] : (!fir.box<!fir.array<2x1xf32>>) -> !fir.box<none>
+! CHECK-LOWERING-OPT:   %[[B_BOX_CONV:.*]] = fir.convert %[[B_BOX]] : (!fir.box<!fir.array<2x2xf32>>) -> !fir.box<none>
+! CHECK-LOWERING-OPT:   fir.call @_FortranAMatmulTranspose(%[[MUL_CONV_RES]], %[[LHS_CONV]], %[[B_BOX_CONV]], %[[LOC_STR2:.*]], %[[LOC_N2:.*]])
+! CHECK-LOWERING-OPT:   %[[MUL_RES_LD:.*]] = fir.load %[[MUL_RES_BOX:.*]]
+! CHECK-LOWERING-OPT:   %[[MUL_RES_ADDR:.*]] = fir.box_addr %[[MUL_RES_LD]]
+! CHECK-LOWERING-OPT:   %[[MUL_RES_VAR:.*]]:2 = hlfir.declare %[[MUL_RES_ADDR]]({{.*}}) {uniq_name = ".tmp.intrinsic_result"}
+! CHECK-LOWERING-OPT:   %[[TRUE2:.*]] = arith.constant true
+! CHECK-LOWERING-OPT:   %[[MUL_EXPR:.*]] = hlfir.as_expr %[[MUL_RES_VAR]]#0 move %[[TRUE2]] : (!fir.box<!fir.array<?x?xf32>>, i1) -> !hlfir.expr<?x?xf32>
+! CHECK-LOWERING-OPT:   hlfir.assign %[[MUL_EXPR]] to %[[RES_DECL]]#0 : !hlfir.expr<?x?xf32>, !fir.ref<!fir.array<1x2xf32>>
+! CHECK-LOWERING-OPT:   hlfir.destroy %[[MUL_EXPR]]
+
+! [argument handling unchanged]
+! CHECK-BUFFERING:      fir.call @_FortranATranspose(
+! CHECK-BUFFERING:      %[[TRANSPOSE_RES_LD:.*]] = fir.load %[[TRANSPOSE_RES_BOX:.*]]
+! CHECK-BUFFERING:      %[[TRANSPOSE_RES_ADDR:.*]] = fir.box_addr %[[TRANSPOSE_RES_LD]]
+! CHECK-BUFFERING:      %[[TRANSPOSE_RES_VAR:.*]]:2 = hlfir.declare %[[TRANSPOSE_RES_ADDR]]({{.*}}) {uniq_name = ".tmp.intrinsic_result"}
+! CHECK-BUFFERING:      %[[TRUE:.*]] = arith.constant true
+! CHECK-BUFFERING:      %[[TUPLE0:.*]] = fir.undefined tuple<!fir.box<!fir.array<?x?xf32>>, i1>
+! CHECK-BUFFERING:      %[[TUPLE1:.*]] = fir.insert_value %[[TUPLE0]], %[[TRUE]], [1 : index]
+! CHECK-BUFFERING:      %[[TUPLE2:.*]] = fir.insert_value %[[TUPLE1]], %[[TRANSPOSE_RES_VAR]]#0, [0 : index]
+
+! CHECK-BUFFERING:      %[[TRANSPOSE_RES_REF:.*]] = fir.convert %[[TRANSPOSE_RES_VAR]]#1 : (!fir.heap<!fir.array<?x?xf32>>) -> !fir.ref<!fir.array<?x?xf32>>
+! CHECK-BUFFERING:      %[[TRANSPOSE_RES_BOX:.*]] = fir.embox %[[TRANSPOSE_RES_REF]]({{.*}})
+! CHECK-BUFFERING:      %[[LHS_CONV:.*]] = fir.convert %[[TRANSPOSE_RES_BOX]] : (!fir.box<!fir.array<?x?xf32>>) -> !fir.box<none>
+! [argument handling unchanged]
+! CHECK-BUFFERING:      fir.call @_FortranAMatmul(
+! CHECK-BUFFERING:      %[[MUL_RES_LD:.*]] = fir.load %[[MUL_RES_BOX:.*]]
+! CHECK-BUFFERING:      %[[MUL_RES_ADDR:.*]] = fir.box_addr %[[MUL_RES_LD]]
+! CHECK-BUFFERING:      %[[MUL_RES_VAR:.*]]:2 = hlfir.declare %[[MUL_RES_ADDR]]({{.*}}) {uniq_name = ".tmp.intrinsic_result"}
+! CHECK-BUFFERING:      %[[TRUE2:.*]] = arith.constant true
+! CHECK-BUFFERING:      %[[TUPLE3:.*]] = fir.undefined tuple<!fir.box<!fir.array<?x?xf32>>, i1>
+! CHECK-BUFFERING:      %[[TUPLE4:.*]] = fir.insert_value %[[TUPLE3]], %[[TRUE2]], [1 : index]
+! CHECK-BUFFERING:      %[[TUPLE5:.*]] = fir.insert_value %[[TUPLE4]], %[[MUL_RES_VAR]]#0, [0 : index]
+
+! CHECK-BUFFERING:      %[[TRANSPOSE_RES_HEAP:.*]] = fir.convert %[[TRANSPOSE_RES_REF]] : (!fir.ref<!fir.array<?x?xf32>>) -> !fir.heap<!fir.array<?x?xf32>>
+! CHECK-BUFFERING-NEXT: fir.freemem %[[TRANSPOSE_RES_HEAP]]
+! CHECK-BUFFERING-NEXT: hlfir.assign %[[MUL_RES_VAR]]#0 to %[[RES_DECL]]#0 : !fir.box<!fir.array<?x?xf32>>, !fir.ref<!fir.array<1x2xf32>>
+! CHECK-BUFFERING-NEXT: fir.freemem %[[MUL_RES_VAR]]#1
+
+! CHECK-ALL-NEXT:       return
+! CHECK-ALL-NEXT:     }
diff --git a/flang/test/HLFIR/sum-bufferization.fir b/flang/test/HLFIR/sum-lowering.fir
similarity index 96%
rename from flang/test/HLFIR/sum-bufferization.fir
rename to flang/test/HLFIR/sum-lowering.fir
index d1bdd9fbd67a1..c031f6dacf27d 100644
--- a/flang/test/HLFIR/sum-bufferization.fir
+++ b/flang/test/HLFIR/sum-lowering.fir
@@ -1,5 +1,5 @@
 // Test hlfir.sum operation lowering to fir runtime call
-// RUN: fir-opt %s -bufferize-hlfir | FileCheck %s
+// RUN: fir-opt %s -lower-hlfir-intrinsics | FileCheck %s
 
 // simple one argument sum
 func.func @_QPsum1(%arg0: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "a"}, %arg1: !fir.ref<i32> {fir.bindc_name = "s"}) {
@@ -64,11 +64,10 @@ func.func @_QPsum2(%arg0: !fir.box<!fir.array<?x?xi32>> {fir.bindc_name = "a"},
 // CHECK-NEXT:    %[[SHIFT:.*]] = fir.shape_shift %[[BOX_DIMS]]#0, %[[BOX_DIMS]]#1
 // TODO: fix alias analysis in hlfir.assign bufferization
 // CHECK-NEXT:    %[[TMP:.*]]:2 = hlfir.declare %[[ADDR]](%[[SHIFT]]) {uniq_name = ".tmp.intrinsic_result"}
-// CHECK:         %[[TUPLE0:.*]] = fir.undefined tuple<!fir.box<!fir.array<?xi32>>, i1>
-// CHECK:         %[[TUPLE1:.*]] = fir.insert_value %[[TUPLE0]], %[[TRUE:.*]], [1 : index]
-// CHECK:         %[[TUPLE2:.*]] fir.insert_value %[[TUPLE1]], %[[TMP]]#0, [0 : index]
-// CHECK:         hlfir.assign %[[TMP]]#0 to %[[RES]]#0
-// CHECK:         fir.freemem %[[TMP]]#1
+// CHECK:         %[[TRUE:.*]] = arith.constant true
+// CHECK:         %[[ASEXPR:.*]] = hlfir.as_expr %[[TMP]]#0 move %[[TRUE]] : (!fir.box<!fir.array<?xi32>>, i1) -> !hlfir.expr<?xi32>
+// CHECK:         hlfir.assign %[[ASEXPR]] to %[[RES]]#0
+// CHECK:         hlfir.destroy %[[ASEXPR]]
 // CHECK-NEXT:    return
 // CHECK-NEXT:  }
 
diff --git a/flang/test/HLFIR/transpose-bufferization.fir b/flang/test/HLFIR/transpose-lowering.fir
similarity index 86%
rename from flang/test/HLFIR/transpose-bufferization.fir
rename to flang/test/HLFIR/transpose-lowering.fir
index 0db359adebd32..733e6f1a61090 100644
--- a/flang/test/HLFIR/transpose-bufferization.fir
+++ b/flang/test/HLFIR/transpose-lowering.fir
@@ -1,5 +1,5 @@
 // Test hlfir.transpose operation lowering to fir runtime call
-// RUN: fir-opt %s -bufferize-hlfir | FileCheck %s
+// RUN: fir-opt %s -lower-hlfir-intrinsics | FileCheck %s
 
 func.func @_QPtranspose1(%arg0: !fir.ref<!fir.array<1x2xi32>> {fir.bindc_name = "m"}, %arg1: !fir.ref<!fir.array<2x1xi32>> {fir.bindc_name = "res"}) {
   %c1 = arith.constant 1 : index
@@ -40,10 +40,9 @@ func.func @_QPtranspose1(%arg0: !fir.ref<!fir.array<1x2xi32>> {fir.bindc_name =
 // CHECK-NEXT:    %[[SHIFT:.*]] = fir.shape_shift %[[BOX_DIMS]]#0, %[[BOX_DIMS]]#1
 // TODO: fix alias analysis in hlfir.assign bufferization
 // CHECK-NEXT:    %[[TMP:.*]]:2 = hlfir.declare %[[ADDR]](%[[SHIFT]]) {uniq_name = ".tmp.intrinsic_result"}
-// CHECK:         %[[TUPLE0:.*]] = fir.undefined tuple<!fir.box<!fir.array<?x?xi32>>, i1>
-// CHECK:         %[[TUPLE1:.*]] = fir.insert_value %[[TUPLE0]], %[[TRUE:.*]], [1 : index]
-// CHECK:         %[[TUPLE2:.*]] = fir.insert_value %[[TUPLE1]], %[[TMP]]#0, [0 : index]
-// CHECK:         hlfir.assign %[[TMP]]#0 to %[[RES_VAR]]#0
-// CHECK:         fir.freemem %[[TMP]]#1
+// CHECK:         %[[TRUE:.*]] = arith.constant true
+// CHECK:         %[[ASEXPR:.*]] = hlfir.as_expr %[[TMP]]#0 move %[[TRUE]] : (!fir.box<!fir.array<?x?xi32>>, i1) -> !hlfir.expr<?x?xi32>
+// CHECK:         hlfir.assign %[[ASEXPR]] to %[[RES_VAR]]#0
+// CHECK:         hlfir.destroy %[[ASEXPR]]
 // CHECK-NEXT:    return
 // CHECK-NEXT:  }
diff --git a/flang/test/Lower/HLFIR/designators.f90 b/flang/test/Lower/HLFIR/designators.f90
index a2aaad536e19c..777d6ea5182e1 100644
--- a/flang/test/Lower/HLFIR/designators.f90
+++ b/flang/test/Lower/HLFIR/designators.f90
@@ -112,3 +112,58 @@ subroutine char_array_section_cst_len(x, n)
 ! CHECK:  %[[VAL_19:.*]] = arith.select %[[VAL_18]], %[[VAL_17]], %[[VAL_14]] : index
 ! CHECK:  %[[VAL_20:.*]] = fir.shape %[[VAL_19]] : (index) -> !fir.shape<1>
 ! CHECK:  %[[VAL_21:.*]] = hlfir.designate %[[VAL_4]]#0 (%[[VAL_10]]:%[[VAL_12]]#1:%[[VAL_13]])  shape %[[VAL_20]] typeparams %[[VAL_3]] : (!fir.box<!fir.array<?x!fir.char<1,5>>>, index, index, index, !fir.shape<1>, index) -> !fir.box<!fir.array<?x!fir.char<1,5>>>
+
+! Checks related to complex numbers
+
+subroutine complex_imag_ref(x)
+  complex :: x(:)
+  print *, x%im
+end subroutine
+! CHECK-LABEL: func.func @_QPcomplex_imag_ref(
+! CHECK:  %[[VAL_2:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = "_QFcomplex_imag_refEx"} : (!fir.box<!fir.array<?x!fir.complex<4>>>) -> (!fir.box<!fir.array<?x!fir.complex<4>>>, !fir.box<!fir.array<?x!fir.complex<4>>>)
+! CHECK:  %[[VAL_3:.*]] = fir.shape %[[VAL_4:.*]]#1 : (index) -> !fir.shape<1>
+! CHECK:  %[[VAL_5:.*]] = hlfir.designate %[[VAL_2]]#0  imag shape %[[VAL_3]] : (!fir.box<!fir.array<?x!fir.complex<4>>>, !fir.shape<1>) -> !fir.box<!fir.array<?xf32>>
+
+subroutine complex_real_ref(x)
+  complex :: x(:)
+  print *, x%re
+end subroutine
+! CHECK-LABEL: func.func @_QPcomplex_real_ref(
+! CHECK:  %[[VAL_2:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = "_QFcomplex_real_refEx"} : (!fir.box<!fir.array<?x!fir.complex<4>>>) -> (!fir.box<!fir.array<?x!fir.complex<4>>>, !fir.box<!fir.array<?x!fir.complex<4>>>)
+! CHECK:  %[[VAL_3:.*]] = fir.shape %[[VAL_4:.*]]#1 : (index) -> !fir.shape<1>
+! CHECK:  %[[VAL_5:.*]] = hlfir.designate %[[VAL_2]]#0  real shape %[[VAL_3]] : (!fir.box<!fir.array<?x!fir.complex<4>>>, !fir.shape<1>) -> !fir.box<!fir.array<?xf32>>
+
+subroutine complex_individual_ref(x, n)
+  complex :: x(:)
+  integer :: n
+  print *, x(n)%im
+end subroutine
+! CHECK-LABEL: func.func @_QPcomplex_individual_ref(
+! CHECK:  %[[VAL_2:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = "_QFcomplex_individual_refEn"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:  %[[VAL_3:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = "_QFcomplex_individual_refEx"} : (!fir.box<!fir.array<?x!fir.complex<4>>>) -> (!fir.box<!fir.array<?x!fir.complex<4>>>, !fir.box<!fir.array<?x!fir.complex<4>>>)
+! CHECK:  %[[VAL_4:.*]] = fir.load %[[VAL_2]]#0 : !fir.ref<i32>
+! CHECK:  %[[VAL_5:.*]] = fir.convert %[[VAL_4]] : (i32) -> i64
+! CHECK:  %[[VAL_6:.*]] = hlfir.designate %1#0 (%[[VAL_5]]) imag : (!fir.box<!fir.array<?x!fir.complex<4>>>, i64) -> !fir.ref<f32>
+
+subroutine complex_slice_ref(x, start, end)
+  complex :: x(:)
+  integer :: start, end
+  print *, x(start:end)%re
+end subroutine
+! CHECK-LABEL: func.func @_QPcomplex_slice_ref(
+! CHECK:  %[[VAL_2:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = "_QFcomplex_slice_refEend"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:  %[[VAL_3:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = "_QFcomplex_slice_refEstart"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:  %[[VAL_4:.*]]:2 = hlfir.declare %arg0 {uniq_name = "_QFcomplex_slice_refEx"} : (!fir.box<!fir.array<?x!fir.complex<4>>>) -> (!fir.box<!fir.array<?x!fir.complex<4>>>, !fir.box<!fir.array<?x!fir.complex<4>>>)
+! CHECK:  %[[VAL_5:.*]] = fir.load %[[VAL_3]]#0 : !fir.ref<i32>
+! CHECK:  %[[VAL_6:.*]] = fir.convert %[[VAL_5]] : (i32) -> i64
+! CHECK:  %[[VAL_7:.*]] = fir.load %[[VAL_2]]#0 : !fir.ref<i32>
+! CHECK:  %[[VAL_8:.*]] = fir.convert %[[VAL_7]] : (i32) -> i64
+! CHECK:  %[[VAL_9:.*]] = fir.convert %[[VAL_6]] : (i64) -> index
+! CHECK:  %[[VAL_10:.*]] = fir.convert %[[VAL_8]] : (i64) -> index
+! CHECK:  %[[VAL_11:.*]] = arith.subi %[[VAL_10]], %[[VAL_9]] : index
+! CHECK:  %[[VAL_12:.*]] = arith.addi %[[VAL_11]], %{{.*}} : index
+! CHECK:  %[[VAL_13:.*]] = arith.divsi %[[VAL_12]], %{{.*}} : index
+! CHECK:  %[[VAL_14:.*]] = arith.cmpi sgt, %[[VAL_13]], %{{.*}} : index
+! CHECK:  %[[VAL_15:.*]] = arith.select %[[VAL_14]], %[[VAL_13]], %{{.*}} : index
+! CHECK:  %[[VAL_16:.*]] = fir.shape %[[VAL_15]] : (index) -> !fir.shape<1>
+! CHECK:  %[[VAL_17:.*]] = hlfir.designate %[[VAL_4]]#0 (%[[VAL_9]]:%[[VAL_10]]:%{{.*}}) real shape %[[VAL_16]] : (!fir.box<!fir.array<?x!fir.complex<4>>>, index, index, index, !fir.shape<1>) -> !fir.box<!fir.array<?xf32>>
diff --git a/flang/test/Lower/HLFIR/dummy-argument-optional.f90 b/flang/test/Lower/HLFIR/dummy-argument-optional.f90
new file mode 100644
index 0000000000000..32ab0e0f6a117
--- /dev/null
+++ b/flang/test/Lower/HLFIR/dummy-argument-optional.f90
@@ -0,0 +1,60 @@
+! RUN: bbc -emit-fir -hlfir -o - %s | FileCheck %s
+
+! Test OPTIONAL lowering on caller/callee
+module opt
+  implicit none
+contains
+
+! Test optional character function
+! CHECK-LABEL: func @_QMoptPchar_proc(
+! CHECK-SAME: %[[arg0:.*]]: !fir.ref<!fir.char<1,3>>,
+character(len=3) function char_proc(i)
+  integer :: i
+  char_proc = "XYZ"
+end function
+! CHECK-LABEL: func @_QMoptPuse_char_proc(
+! CHECK-SAME: %[[arg0:.*]]: tuple<!fir.boxproc<() -> ()>, i64> {fir.char_proc},
+subroutine use_char_proc(f, c)
+  optional :: f
+  interface
+    character(len=3) function f(i)
+      integer :: i
+    end function
+  end interface
+  character(len=3) :: c
+! CHECK: %[[boxProc:.*]] = fir.extract_value %[[arg0]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>) -> !fir.boxproc<() -> ()>
+! CHECK: %[[boxAddr:.*]] = fir.box_addr %[[boxProc]] : (!fir.boxproc<() -> ()>) -> (() -> ())
+! CHECK: %[[boxProc2:.*]] = fir.emboxproc %[[boxAddr]] : (() -> ()) -> !fir.boxproc<() -> ()>
+! CHECK: %[[tuple:.*]] = fir.undefined tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple2:.*]] = fir.insert_value %[[tuple]], %[[boxProc2]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxproc<() -> ()>) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple3:.*]] = fir.insert_value %[[tuple2]], %{{.*}}, [1 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, i64) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[boxProc3:.*]] = fir.extract_value %[[tuple3]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>) -> !fir.boxproc<() -> ()>
+! CHECK: %{{.*}} = fir.is_present %[[boxProc3]] : (!fir.boxproc<() -> ()>) -> i1
+  if (present(f)) then
+    c = f(0)
+  else
+    c = "ABC"
+  end if
+end subroutine
+! CHECK-LABEL: func @_QMoptPcall_use_char_proc(
+subroutine call_use_char_proc()
+  character(len=3) :: c
+! CHECK: %[[boxProc:.*]] = fir.absent !fir.boxproc<() -> ()>
+! CHECK: %[[undef:.*]] = fir.undefined index
+! CHECK: %[[charLen:.*]] = fir.convert %[[undef]] : (index) -> i64
+! CHECK: %[[tuple:.*]] = fir.undefined tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple2:.*]] = fir.insert_value %[[tuple]], %[[boxProc]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxproc<() -> ()>) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple3:.*]] = fir.insert_value %[[tuple2]], %[[charLen]], [1 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, i64) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: fir.call @_QMoptPuse_char_proc(%[[tuple3]], %{{.*}}){{.*}} : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxchar<1>) -> ()
+  call use_char_proc(c=c)
+! CHECK: %[[funcAddr:.*]] = fir.address_of(@_QMoptPchar_proc) : (!fir.ref<!fir.char<1,3>>, index, {{.*}}) -> !fir.boxchar<1>
+! CHECK: %[[c3:.*]] = arith.constant 3 : i64
+! CHECK: %[[boxProc2:.*]] = fir.emboxproc %[[funcAddr]] : ((!fir.ref<!fir.char<1,3>>, index, {{.*}}) -> !fir.boxchar<1>) -> !fir.boxproc<() -> ()>
+! CHECK: %[[tuple4:.*]] = fir.undefined tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple5:.*]] = fir.insert_value %[[tuple4]], %[[boxProc2]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxproc<() -> ()>) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple6:.*]] = fir.insert_value %[[tuple5]], %[[c3]], [1 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, i64) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: fir.call @_QMoptPuse_char_proc(%[[tuple6]], {{.*}}){{.*}} : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxchar<1>) -> ()
+  call use_char_proc(char_proc, c)
+end subroutine
+
+end module
diff --git a/flang/test/Lower/HLFIR/internal-procedures.f90 b/flang/test/Lower/HLFIR/internal-procedures.f90
index 4ad272ed4900d..ca3ec3e74289b 100644
--- a/flang/test/Lower/HLFIR/internal-procedures.f90
+++ b/flang/test/Lower/HLFIR/internal-procedures.f90
@@ -36,3 +36,19 @@ subroutine internal
 ! CHECK:  %[[VAL_5:.*]]:3 = fir.box_dims %[[VAL_3]], %[[VAL_4]] : (!fir.box<!fir.array<?xf32>>, index) -> (index, index, index)
 ! CHECK:  %[[VAL_6:.*]] = fir.shift %[[VAL_5]]#0 : (index) -> !fir.shift<1>
 ! CHECK:  %[[VAL_7:.*]]:2 = hlfir.declare %[[VAL_3]](%[[VAL_6]]) {uniq_name = "_QFtest_assumed_shapeEx"} : (!fir.box<!fir.array<?xf32>>, !fir.shift<1>) -> (!fir.box<!fir.array<?xf32>>, !fir.box<!fir.array<?xf32>>)
+
+subroutine test_scalar_char(c)
+ character(*) :: c
+contains
+subroutine internal()
+  call bar(c)
+end subroutine
+end subroutine
+! CHECK-LABEL:   func.func @_QFtest_scalar_charPinternal(
+! CHECK-SAME:                               %[[VAL_0:.*]]: !fir.ref<tuple<!fir.boxchar<1>>> {fir.host_assoc}) attributes {fir.internal_proc} {
+! CHECK:  %[[VAL_1:.*]] = arith.constant 0 : i32
+! CHECK:  %[[VAL_2:.*]] = fir.coordinate_of %[[VAL_0]], %[[VAL_1]] : (!fir.ref<tuple<!fir.boxchar<1>>>, i32) -> !fir.ref<!fir.boxchar<1>>
+! CHECK:  %[[VAL_3:.*]] = fir.load %[[VAL_2]] : !fir.ref<!fir.boxchar<1>>
+! CHECK:  %[[VAL_4:.*]]:2 = fir.unboxchar %[[VAL_3]] : (!fir.boxchar<1>) -> (!fir.ref<!fir.char<1,?>>, index)
+! CHECK:  %[[VAL_5:.*]]:2 = hlfir.declare %[[VAL_4]]#0 typeparams %[[VAL_4]]#1 {uniq_name = "_QFtest_scalar_charEc"} : (!fir.ref<!fir.char<1,?>>, index) -> (!fir.boxchar<1>, !fir.ref<!fir.char<1,?>>)
+! CHECK:  fir.call @_QPbar(%[[VAL_5]]#0) {{.*}}: (!fir.boxchar<1>) -> ()
diff --git a/flang/test/Lower/HLFIR/intrinsic-module-procedures.f90 b/flang/test/Lower/HLFIR/intrinsic-module-procedures.f90
new file mode 100644
index 0000000000000..40bb39e967265
--- /dev/null
+++ b/flang/test/Lower/HLFIR/intrinsic-module-procedures.f90
@@ -0,0 +1,23 @@
+! Test lowering of intrinsic module procedures to HLFIR. This
+! test is not meant to test every intrinsic module procedure,
+! it only tests that the HFLIR procedure reference lowering
+! infrastructure properly detects and dispatches intrinsic module
+! procedure calls.
+! RUN: bbc -emit-fir -hlfir -o - %s | FileCheck %s
+
+subroutine foo(cptr, x)
+  use iso_c_binding, only : c_ptr, c_loc
+  type(c_ptr) :: cptr
+  integer :: x
+  cptr = c_loc(x)
+end subroutine
+! CHECK-LABEL: func.func @_QPfoo(
+! CHECK:         %[[VAL_3:.*]]:2 = hlfir.declare {{.*}}Ecptr"
+! CHECK:         %[[VAL_3:.*]]:2 = hlfir.declare {{.*}}Ex"
+! CHECK:         %[[VAL_4:.*]] = fir.embox %[[VAL_3]]#1 : (!fir.ref<i32>) -> !fir.box<i32>
+! CHECK:         %[[VAL_5:.*]] = fir.alloca !fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>
+! CHECK:         %[[VAL_6:.*]] = fir.field_index __address, !fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>
+! CHECK:         %[[VAL_7:.*]] = fir.coordinate_of %[[VAL_5]], %[[VAL_6]] : (!fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>, !fir.field) -> !fir.ref<i64>
+! CHECK:         %[[VAL_8:.*]] = fir.box_addr %[[VAL_4]] : (!fir.box<i32>) -> !fir.ref<i32>
+! CHECK:         %[[VAL_9:.*]] = fir.convert %[[VAL_8]] : (!fir.ref<i32>) -> i64
+! CHECK:         fir.store %[[VAL_9]] to %[[VAL_7]] : !fir.ref<i64>
diff --git a/flang/test/Lower/Intrinsics/atand.f90 b/flang/test/Lower/Intrinsics/atand.f90
new file mode 100644
index 0000000000000..eea9c79773cf0
--- /dev/null
+++ b/flang/test/Lower/Intrinsics/atand.f90
@@ -0,0 +1,26 @@
+! RUN: bbc -emit-fir %s -o - | FileCheck %s --check-prefixes="CHECK,CHECK-FAST"
+! RUN: bbc --math-runtime=precise -emit-fir %s -o - | FileCheck %s --check-prefixes="CHECK,CHECK-PRECISE"
+! RUN: %flang_fc1 -emit-fir %s -o - | FileCheck %s --check-prefixes="CHECK,CHECK-FAST"
+
+function test_real4(x)
+  real :: x, test_real4
+  test_real4 = atand(x)
+end function
+
+! CHECK-LABEL: @_QPtest_real4
+! CHECK-PRECISE: %[[atan:.*]] = fir.call @atanf({{%[A-Za-z0-9._]+}}) fastmath<contract> : (f32) -> f32
+! CHECK-FAST: %[[atan:.*]] = math.atan %{{.*}} : f32
+! CHECK: %[[dfactor:.*]] = arith.constant 57.295779513082323 : f64
+! CHECK: %[[factor:.*]] = fir.convert %[[dfactor]] : (f64) -> f32
+! CHECK: %{{.*}} = arith.mulf %[[atan]], %[[factor]] fastmath<contract> : f32
+
+function test_real8(x)
+  real(8) :: x, test_real8
+  test_real8 = atand(x)
+end function
+
+! CHECK-LABEL: @_QPtest_real8
+! CHECK-PRECISE: %[[atan:.*]] = fir.call @atan({{%[A-Za-z0-9._]+}}) fastmath<contract> : (f64) -> f64
+! CHECK-FAST: %[[atan:.*]] = math.atan %{{.*}} : f64
+! CHECK: %[[factor:.*]] = arith.constant 57.295779513082323 : f64
+! CHECK: %{{.*}} = arith.mulf %[[atan]], %[[factor]] fastmath<contract> : f64
diff --git a/flang/test/Lower/Intrinsics/ieee_is_normal.f90 b/flang/test/Lower/Intrinsics/ieee_is_normal.f90
new file mode 100644
index 0000000000000..84b3d2bf0a6d3
--- /dev/null
+++ b/flang/test/Lower/Intrinsics/ieee_is_normal.f90
@@ -0,0 +1,64 @@
+! RUN: bbc -emit-fir %s -o - | FileCheck %s
+! RUN: flang-new -fc1 -emit-fir %s -o - | FileCheck %s
+
+! CHECK-LABEL: ieee_is_normal_f16
+subroutine ieee_is_normal_f16(r)
+  use ieee_arithmetic
+  real(KIND=2) :: r
+  i = ieee_is_normal(r)
+  ! CHECK: %[[test:.*]] = arith.constant 360 : i32
+  ! CHECK: %[[l:.*]] = "llvm.intr.is.fpclass"(%{{.*}}, %[[test]]) : (f16, i32) -> i1
+  ! CHECK: fir.convert %[[l]] : (i1) -> !fir.logical<4>
+end subroutine ieee_is_normal_f16
+
+! CHECK-LABEL: ieee_is_normal_bf16
+subroutine ieee_is_normal_bf16(r)
+  use ieee_arithmetic
+  real(KIND=3) :: r
+  i = ieee_is_normal(r)
+  ! CHECK: %[[test:.*]] = arith.constant 360 : i32
+  ! CHECK: %[[l:.*]] = "llvm.intr.is.fpclass"(%{{.*}}, %[[test]]) : (bf16, i32) -> i1
+  ! CHECK: fir.convert %[[l]] : (i1) -> !fir.logical<4>
+end subroutine ieee_is_normal_bf16
+
+
+
+! CHECK-LABEL: ieee_is_normal_f32
+subroutine ieee_is_normal_f32(r)
+  use ieee_arithmetic
+  real :: r
+  i = ieee_is_normal(r)
+  ! CHECK: %[[test:.*]] = arith.constant 360 : i32
+  ! CHECK: %[[l:.*]] = "llvm.intr.is.fpclass"(%{{.*}}, %[[test]]) : (f32, i32) -> i1
+  ! CHECK: fir.convert %[[l]] : (i1) -> !fir.logical<4>
+end subroutine ieee_is_normal_f32
+
+! CHECK-LABEL: ieee_is_normal_f64
+subroutine ieee_is_normal_f64(r)
+  use ieee_arithmetic
+  real(KIND=8) :: r
+  i = ieee_is_normal(r)
+  ! CHECK: %[[test:.*]] = arith.constant 360 : i32
+  ! CHECK: %[[l:.*]] = "llvm.intr.is.fpclass"(%{{.*}}, %[[test]]) : (f64, i32) -> i1
+  ! CHECK: fir.convert %[[l]] : (i1) -> !fir.logical<4>
+end subroutine ieee_is_normal_f64
+
+! CHECK-LABEL: ieee_is_normal_f80
+subroutine ieee_is_normal_f80(r)
+  use ieee_arithmetic
+  real(KIND=10) :: r
+  i = ieee_is_normal(r)
+  ! CHECK: %[[test:.*]] = arith.constant 360 : i32
+  ! CHECK: %[[l:.*]] = "llvm.intr.is.fpclass"(%{{.*}}, %[[test]]) : (f80, i32) -> i1
+  ! CHECK: fir.convert %[[l]] : (i1) -> !fir.logical<4>
+end subroutine ieee_is_normal_f80
+
+! CHECK-LABEL: ieee_is_normal_f128
+subroutine ieee_is_normal_f128(r)
+  use ieee_arithmetic
+  real(KIND=16) :: r
+  i = ieee_is_normal(r)
+  ! CHECK: %[[test:.*]] = arith.constant 360 : i32
+  ! CHECK: %[[l:.*]] = "llvm.intr.is.fpclass"(%{{.*}}, %[[test]]) : (f128, i32) -> i1
+  ! CHECK: fir.convert %[[l]] : (i1) -> !fir.logical<4>
+end subroutine ieee_is_normal_f128
diff --git a/flang/test/Lower/Intrinsics/is_iostat_value.f90 b/flang/test/Lower/Intrinsics/is_iostat_value.f90
new file mode 100644
index 0000000000000..933dca1dcf193
--- /dev/null
+++ b/flang/test/Lower/Intrinsics/is_iostat_value.f90
@@ -0,0 +1,23 @@
+! RUN: bbc -emit-fir -o - %s | FileCheck %s
+
+  ! CHECK-LABEL: func @_QQmain
+  ! CHECK: %[[V_0:[0-9]+]] = fir.alloca i32 {bindc_name = "i", uniq_name = "_QFEi"}
+  ! CHECK: fir.do_loop
+  do i=-20,20
+    ! CHECK: %[[V_5:[0-9]+]] = fir.load %[[V_0]] : !fir.ref<i32>
+    ! CHECK: %[[V_6:[0-9]+]] = arith.cmpi eq, %[[V_5]], %c-1{{.*}} : i32
+    ! CHECK: fir.if %[[V_6]] {
+    ! CHECK:   %[[V_20:[0-9]+]] = fir.load %[[V_0]] : !fir.ref<i32>
+    ! CHECK:   %[[V_21:[0-9]+]] = fir.call @_FortranAioOutputInteger32(%{{.*}} %[[V_20]]) fastmath<contract> : (!fir.ref<i8>, i32) -> i1
+    ! CHECK: }
+    if (is_iostat_end(i)) print*, "iostat_end =", i
+    ! CHECK: %[[V_7:[0-9]+]] = fir.load %[[V_0]] : !fir.ref<i32>
+    ! CHECK: %[[V_8:[0-9]+]] = arith.cmpi eq, %[[V_7]], %c-2{{.*}} : i32
+    ! CHECK: fir.if %[[V_8]] {
+    ! CHECK:   %[[V_20:[0-9]+]] = fir.load %[[V_0]] : !fir.ref<i32>
+    ! CHECK:   %[[V_21:[0-9]+]] = fir.call @_FortranAioOutputInteger32(%{{.*}} %[[V_20]]) fastmath<contract> : (!fir.ref<i8>, i32) -> i1
+    ! CHECK: }
+    if (is_iostat_eor(i)) print*, "iostat_eor =", i
+  ! CHECK: }
+  enddo
+end
diff --git a/flang/test/Lower/OpenMP/Todo/reduction-eqv.f90 b/flang/test/Lower/OpenMP/Todo/reduction-eqv.f90
deleted file mode 100644
index 36322e9156c6c..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/reduction-eqv.f90
+++ /dev/null
@@ -1,15 +0,0 @@
-! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -o - %s 2>&1 | FileCheck %s
-! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -o - %s 2>&1 | FileCheck %s
-
-! CHECK: not yet implemented: Reduction of some intrinsic operators is not supported
-subroutine reduction_eqv(y)
-  logical :: x, y(100)
-  !$omp parallel
-  !$omp do reduction(.eqv.:x)
-  do i=1, 100
-    x = x .eqv. y(i)
-  end do
-  !$omp end do
-  !$omp end parallel
-  print *, x
-end subroutine
diff --git a/flang/test/Lower/OpenMP/Todo/reduction-max.f90 b/flang/test/Lower/OpenMP/Todo/reduction-max.f90
deleted file mode 100644
index e965e6860712e..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/reduction-max.f90
+++ /dev/null
@@ -1,16 +0,0 @@
-! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -o - %s 2>&1 | FileCheck %s
-! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -o - %s 2>&1 | FileCheck %s
-
-! CHECK: not yet implemented: Reduction of intrinsic procedures is not supported
-subroutine reduction_max(y)
-  integer :: x, y(:)
-  x = 0
-  !$omp parallel
-  !$omp do reduction(max:x)
-  do i=1, 100
-    x = max(x, y(i))
-  end do
-  !$omp end do
-  !$omp end parallel
-  print *, x
-end subroutine
diff --git a/flang/test/Lower/OpenMP/omp-is-device.f90 b/flang/test/Lower/OpenMP/omp-is-device.f90
new file mode 100644
index 0000000000000..63840aca56c0c
--- /dev/null
+++ b/flang/test/Lower/OpenMP/omp-is-device.f90
@@ -0,0 +1,14 @@
+!RUN: %flang_fc1 -emit-fir -fopenmp -fopenmp-is-device %s -o - | FileCheck %s --check-prefix=DEVICE
+!RUN: %flang_fc1 -emit-fir -fopenmp %s -o - | FileCheck %s --check-prefix=HOST
+!RUN: %flang_fc1 -emit-fir -fopenmp-is-device %s -o - | FileCheck %s --check-prefix=DEVICE-FLAG-ONLY
+!RUN: bbc -fopenmp -fopenmp-is-device -emit-fir -o - %s | FileCheck %s --check-prefix=DEVICE
+!RUN: bbc -fopenmp -emit-fir -o - %s | FileCheck %s --check-prefix=HOST
+!RUN: bbc -fopenmp-is-device -emit-fir -o - %s | FileCheck %s --check-prefix=DEVICE-FLAG-ONLY
+
+!DEVICE: module attributes {{{.*}}, omp.is_device = true{{.*}}}
+!HOST: module attributes {{{.*}}, omp.is_device = false{{.*}}}
+!DEVICE-FLAG-ONLY: module attributes {{{.*}}"
+!DEVICE-FLAG-ONLY-NOT: , omp.is_device = {{.*}}
+!DEVICE-FLAG-ONLY-SAME: }
+subroutine omp_subroutine()
+end subroutine omp_subroutine
diff --git a/flang/test/Lower/OpenMP/wsloop-reduction-logical-and.f90 b/flang/test/Lower/OpenMP/wsloop-reduction-logical-and.f90
index 206d63d79f454..425d37398c571 100644
--- a/flang/test/Lower/OpenMP/wsloop-reduction-logical-and.f90
+++ b/flang/test/Lower/OpenMP/wsloop-reduction-logical-and.f90
@@ -2,14 +2,18 @@
 ! RUN: %flang_fc1 -emit-fir -fopenmp %s -o - | FileCheck %s
 
 !CHECK-LABEL: omp.reduction.declare
-!CHECK-SAME: @[[RED_NAME:.*]] : i1 init {
-!CHECK: ^bb0(%{{.*}}: i1):
+!CHECK-SAME: @[[RED_NAME:.*]] : !fir.logical<4> init {
+!CHECK: ^bb0(%{{.*}}: !fir.logical<4>):
 !CHECK:  %true = arith.constant true
-!CHECK:  omp.yield(%true : i1)
+!CHECK:  %[[true_fir:.*]] = fir.convert %true : (i1) -> !fir.logical<4>
+!CHECK:  omp.yield(%[[true_fir]]  : !fir.logical<4>)
 !CHECK: } combiner {
-!CHECK: ^bb0(%[[ARG0:.*]]: i1, %[[ARG1:.*]]: i1):
-!CHECK:  %[[RES:.*]] = arith.andi %[[ARG0]], %[[ARG1]] : i1
-!CHECK:  omp.yield(%[[RES]] : i1)
+!CHECK: ^bb0(%[[ARG0:.*]]: !fir.logical<4>, %[[ARG1:.*]]: !fir.logical<4>):
+!CHECK:  %[[arg0_i1:.*]] = fir.convert %[[ARG0]] : (!fir.logical<4>) -> i1
+!CHECK:  %[[arg1_i1:.*]] = fir.convert %[[ARG1]] : (!fir.logical<4>) -> i1
+!CHECK:  %[[RES:.*]] = arith.andi %[[arg0_i1]], %[[arg1_i1]] : i1
+!CHECK:  %[[RES_logical:.*]] = fir.convert %[[RES]] : (i1) -> !fir.logical<4>
+!CHECK:  omp.yield(%[[RES_logical]] : !fir.logical<4>)
 !CHECK: }
 
 !CHECK-LABEL: func.func @_QPsimple_reduction(
diff --git a/flang/test/Lower/OpenMP/wsloop-reduction-logical-eqv.f90 b/flang/test/Lower/OpenMP/wsloop-reduction-logical-eqv.f90
new file mode 100644
index 0000000000000..e8cf46f8261c4
--- /dev/null
+++ b/flang/test/Lower/OpenMP/wsloop-reduction-logical-eqv.f90
@@ -0,0 +1,136 @@
+! RUN: bbc -emit-fir -fopenmp %s -o - | FileCheck %s
+! RUN: %flang_fc1 -emit-fir -fopenmp %s -o - | FileCheck %s
+
+!CHECK-LABEL: omp.reduction.declare
+!CHECK-SAME: @[[RED_NAME:.*]] : !fir.logical<4> init {
+!CHECK: ^bb0(%{{.*}}: !fir.logical<4>):
+!CHECK:  %true = arith.constant true
+!CHECK:  %[[true_fir:.*]] = fir.convert %true : (i1) -> !fir.logical<4>
+!CHECK:  omp.yield(%[[true_fir]]  : !fir.logical<4>)
+!CHECK: } combiner {
+!CHECK: ^bb0(%[[ARG0:.*]]: !fir.logical<4>, %[[ARG1:.*]]: !fir.logical<4>):
+!CHECK:  %[[arg0_i1:.*]] = fir.convert %[[ARG0]] : (!fir.logical<4>) -> i1
+!CHECK:  %[[arg1_i1:.*]] = fir.convert %[[ARG1]] : (!fir.logical<4>) -> i1
+!CHECK:  %[[RES:.*]] = arith.cmpi eq, %[[arg0_i1]], %[[arg1_i1]] : i1
+!CHECK:  %[[RES_logical:.*]] = fir.convert %[[RES]] : (i1) -> !fir.logical<4>
+!CHECK:  omp.yield(%[[RES_logical]] : !fir.logical<4>)
+!CHECK: }
+
+!CHECK-LABEL: func.func @_QPsimple_reduction(
+!CHECK-SAME: %[[ARRAY:.*]]: !fir.ref<!fir.array<100x!fir.logical<4>>> {fir.bindc_name = "y"}) {
+!CHECK:  %[[IREF:.*]] = fir.alloca i32 {bindc_name = "i", uniq_name = "_QFsimple_reductionEi"}
+!CHECK:  %[[XREF:.*]] = fir.alloca !fir.logical<4> {bindc_name = "x", uniq_name = "_QFsimple_reductionEx"}
+!CHECK:  omp.parallel
+!CHECK:    %[[I_PVT_REF:.*]] = fir.alloca i32 {adapt.valuebyref, pinned}
+!CHECK:    %[[C1_1:.*]] = arith.constant 1 : i32
+!CHECK:    %[[C100:.*]] = arith.constant 100 : i32
+!CHECK:    %[[C1_2:.*]] = arith.constant 1 : i32
+!CHECK:    omp.wsloop   reduction(@[[RED_NAME]] -> %[[XREF]] : !fir.ref<!fir.logical<4>>) for  (%[[IVAL:.*]]) : i32 = (%[[C1_1]]) to (%[[C100]]) inclusive step (%[[C1_2]]) {
+!CHECK:      fir.store %[[IVAL]] to %[[I_PVT_REF]] : !fir.ref<i32>
+!CHECK:      %[[I_PVT_VAL:.*]] = fir.load %[[I_PVT_REF]] : !fir.ref<i32>
+!CHECK:      %[[CONVI_64:.*]] = fir.convert %[[I_PVT_VAL]] : (i32) -> i64
+!CHECK:      %[[C1_64:.*]] = arith.constant 1 : i64
+!CHECK:      %[[SUBI:.*]] = arith.subi %[[CONVI_64]], %[[C1_64]] : i64
+!CHECK:      %[[Y_PVT_REF:.*]] = fir.coordinate_of %[[ARRAY]], %[[SUBI]] : (!fir.ref<!fir.array<100x!fir.logical<4>>>, i64) -> !fir.ref<!fir.logical<4>>
+!CHECK:      %[[YVAL:.*]] = fir.load %[[Y_PVT_REF]] : !fir.ref<!fir.logical<4>>
+!CHECK:      omp.reduction %[[YVAL]], %[[XREF]] : !fir.logical<4>, !fir.ref<!fir.logical<4>>
+!CHECK:      omp.yield
+!CHECK:    omp.terminator
+!CHECK:  return
+subroutine simple_reduction(y)
+  logical :: x, y(100)
+  x = .true.
+  !$omp parallel
+  !$omp do reduction(.eqv.:x)
+  do i=1, 100
+    x = x .eqv. y(i)
+  end do
+  !$omp end do
+  !$omp end parallel
+end subroutine
+
+!CHECK-LABEL: func.func @_QPsimple_reduction_switch_order(
+!CHECK-SAME: %[[ARRAY:.*]]: !fir.ref<!fir.array<100x!fir.logical<4>>> {fir.bindc_name = "y"}) {
+!CHECK:  %[[IREF:.*]] = fir.alloca i32 {bindc_name = "i", uniq_name = "_QFsimple_reduction_switch_orderEi"}
+!CHECK:  %[[XREF:.*]] = fir.alloca !fir.logical<4> {bindc_name = "x", uniq_name = "_QFsimple_reduction_switch_orderEx"}
+!CHECK:  omp.parallel
+!CHECK:    %[[I_PVT_REF:.*]] = fir.alloca i32 {adapt.valuebyref, pinned}
+!CHECK:    %[[C1_1:.*]] = arith.constant 1 : i32
+!CHECK:    %[[C100:.*]] = arith.constant 100 : i32
+!CHECK:    %[[C1_2:.*]] = arith.constant 1 : i32
+!CHECK:    omp.wsloop   reduction(@[[RED_NAME]] -> %[[XREF]] : !fir.ref<!fir.logical<4>>) for  (%[[IVAL:.*]]) : i32 = (%[[C1_1]]) to (%[[C100]]) inclusive step (%[[C1_2]]) {
+!CHECK:      fir.store %[[IVAL]] to %[[I_PVT_REF]] : !fir.ref<i32>
+!CHECK:      %[[I_PVT_VAL:.*]] = fir.load %[[I_PVT_REF]] : !fir.ref<i32>
+!CHECK:      %[[CONVI_64:.*]] = fir.convert %[[I_PVT_VAL]] : (i32) -> i64
+!CHECK:      %[[C1_64:.*]] = arith.constant 1 : i64
+!CHECK:      %[[SUBI:.*]] = arith.subi %[[CONVI_64]], %[[C1_64]] : i64
+!CHECK:      %[[Y_PVT_REF:.*]] = fir.coordinate_of %[[ARRAY]], %[[SUBI]] : (!fir.ref<!fir.array<100x!fir.logical<4>>>, i64) -> !fir.ref<!fir.logical<4>>
+!CHECK:      %[[YVAL:.*]] = fir.load %[[Y_PVT_REF]] : !fir.ref<!fir.logical<4>>
+!CHECK:      omp.reduction %[[YVAL]], %[[XREF]] : !fir.logical<4>, !fir.ref<!fir.logical<4>>
+!CHECK:      omp.yield
+!CHECK:    omp.terminator
+!CHECK:  return
+subroutine simple_reduction_switch_order(y)
+  logical :: x, y(100)
+  x = .true.
+  !$omp parallel
+  !$omp do reduction(.eqv.:x)
+  do i=1, 100
+  x = y(i) .eqv. x
+  end do
+  !$omp end do
+  !$omp end parallel
+end subroutine
+
+!CHECK-LABEL: func.func @_QPmultiple_reductions
+!CHECK-SAME %[[ARRAY:.*]]: !fir.ref<!fir.array<100x!fir.logical<4>>> {fir.bindc_name = "w"}) {
+!CHECK:  %[[IREF:.*]] = fir.alloca i32 {bindc_name = "i", uniq_name = "_QFmultiple_reductionsEi"}
+!CHECK:  %[[XREF:.*]] = fir.alloca !fir.logical<4> {bindc_name = "x", uniq_name = "_QFmultiple_reductionsEx"}
+!CHECK:  %[[YREF:.*]] = fir.alloca !fir.logical<4> {bindc_name = "y", uniq_name = "_QFmultiple_reductionsEy"}
+!CHECK:  %[[ZREF:.*]] = fir.alloca !fir.logical<4> {bindc_name = "z", uniq_name = "_QFmultiple_reductionsEz"}
+!CHECK:  omp.parallel
+!CHECK:    %[[I_PVT_REF:.*]] = fir.alloca i32 {adapt.valuebyref, pinned}
+!CHECK:    %[[C1_1:.*]] = arith.constant 1 : i32
+!CHECK:    %[[C100:.*]] = arith.constant 100 : i32
+!CHECK:    %[[C1_2:.*]] = arith.constant 1 : i32
+!CHECK:    omp.wsloop   reduction(@[[RED_NAME]] -> %[[XREF]] : !fir.ref<!fir.logical<4>>, @[[RED_NAME]] -> %[[YREF]] : !fir.ref<!fir.logical<4>>, @[[RED_NAME]] -> %[[ZREF]] : !fir.ref<!fir.logical<4>>) for  (%[[IVAL:.*]]) : i32 = (%[[C1_1]]) to (%[[C100]]) inclusive step (%[[C1_2]]) {
+!CHECK:      fir.store %[[IVAL]] to %[[I_PVT_REF]] : !fir.ref<i32>
+!CHECK:      %[[I_PVT_VAL1:.*]] = fir.load %[[I_PVT_REF]] : !fir.ref<i32>
+!CHECK:      %[[CONVI_64_1:.*]] = fir.convert %[[I_PVT_VAL1]] : (i32) -> i64
+!CHECK:      %[[C1_64:.*]] = arith.constant 1 : i64
+!CHECK:      %[[SUBI_1:.*]] = arith.subi %[[CONVI_64_1]], %[[C1_64]] : i64
+!CHECK:      %[[W_PVT_REF_1:.*]] = fir.coordinate_of %[[ARRAY]], %[[SUBI_1]] : (!fir.ref<!fir.array<100x!fir.logical<4>>>, i64) -> !fir.ref<!fir.logical<4>>
+!CHECK:      %[[WVAL:.*]] = fir.load %[[W_PVT_REF_1]] : !fir.ref<!fir.logical<4>>
+!CHECK:      omp.reduction %[[WVAL]], %[[XREF]] : !fir.logical<4>, !fir.ref<!fir.logical<4>>
+!CHECK:      %[[I_PVT_VAL2:.*]] = fir.load %[[I_PVT_REF]] : !fir.ref<i32>
+!CHECK:      %[[CONVI_64_2:.*]] = fir.convert %[[I_PVT_VAL2]] : (i32) -> i64
+!CHECK:      %[[C1_64:.*]] = arith.constant 1 : i64
+!CHECK:      %[[SUBI_2:.*]] = arith.subi %[[CONVI_64_2]], %[[C1_64]] : i64
+!CHECK:      %[[W_PVT_REF_2:.*]] = fir.coordinate_of %[[ARRAY]], %[[SUBI_2]] : (!fir.ref<!fir.array<100x!fir.logical<4>>>, i64) -> !fir.ref<!fir.logical<4>>
+!CHECK:      %[[WVAL:.*]] = fir.load %[[W_PVT_REF_2]] : !fir.ref<!fir.logical<4>>
+!CHECK:      omp.reduction %[[WVAL]], %[[YREF]] : !fir.logical<4>, !fir.ref<!fir.logical<4>>
+!CHECK:      %[[I_PVT_VAL3:.*]] = fir.load %[[I_PVT_REF]] : !fir.ref<i32>
+!CHECK:      %[[CONVI_64_3:.*]] = fir.convert %[[I_PVT_VAL3]] : (i32) -> i64
+!CHECK:      %[[C1_64:.*]] = arith.constant 1 : i64
+!CHECK:      %[[SUBI_3:.*]] = arith.subi %[[CONVI_64_3]], %[[C1_64]] : i64
+!CHECK:      %[[W_PVT_REF_3:.*]] = fir.coordinate_of %[[ARRAY]], %[[SUBI_3]] : (!fir.ref<!fir.array<100x!fir.logical<4>>>, i64) -> !fir.ref<!fir.logical<4>>
+!CHECK:      %[[WVAL:.*]] = fir.load %[[W_PVT_REF_3]] : !fir.ref<!fir.logical<4>>
+!CHECK:      omp.reduction %[[WVAL]], %[[ZREF]] : !fir.logical<4>, !fir.ref<!fir.logical<4>>
+!CHECK:      omp.yield
+!CHECK:    omp.terminator
+!CHECK:  return
+subroutine multiple_reductions(w)
+  logical :: x,y,z,w(100)
+  x = .true.
+  y = .true.
+  z = .true.
+  !$omp parallel
+  !$omp do reduction(.eqv.:x,y,z)
+  do i=1, 100
+  x = x .eqv. w(i)
+  y = y .eqv. w(i)
+  z = z .eqv. w(i)
+  end do
+  !$omp end do
+  !$omp end parallel
+end subroutine
diff --git a/flang/test/Lower/OpenMP/wsloop-reduction-max.f90 b/flang/test/Lower/OpenMP/wsloop-reduction-max.f90
new file mode 100644
index 0000000000000..0db5b7f813d74
--- /dev/null
+++ b/flang/test/Lower/OpenMP/wsloop-reduction-max.f90
@@ -0,0 +1,66 @@
+! RUN: bbc -emit-fir -fopenmp -o - %s 2>&1 | FileCheck %s
+! RUN: %flang_fc1 -emit-fir -fopenmp -o - %s 2>&1 | FileCheck %s
+
+!CHECK: omp.reduction.declare @[[MAX_DECLARE_F:.*]] : f32 init {
+!CHECK:   %[[MINIMUM_VAL_F:.*]] = arith.constant -3.40282347E+38 : f32
+!CHECK:   omp.yield(%[[MINIMUM_VAL_F]] : f32)
+!CHECK: combiner
+!CHECK: ^bb0(%[[ARG0_F:.*]]: f32, %[[ARG1_F:.*]]: f32):
+!CHECK:   %[[COMB_VAL_F:.*]] = arith.maxf %[[ARG0_F]], %[[ARG1_F]] {{.*}}: f32
+!CHECK:   omp.yield(%[[COMB_VAL_F]] : f32)
+
+!CHECK: omp.reduction.declare @[[MAX_DECLARE_I:.*]] : i32 init {
+!CHECK:   %[[MINIMUM_VAL_I:.*]] = arith.constant -2147483648 : i32
+!CHECK:   omp.yield(%[[MINIMUM_VAL_I]] : i32)
+!CHECK: combiner
+!CHECK: ^bb0(%[[ARG0_I:.*]]: i32, %[[ARG1_I:.*]]: i32):
+!CHECK:   %[[COMB_VAL_I:.*]] = arith.maxsi %[[ARG0_I]], %[[ARG1_I]] : i32
+!CHECK:   omp.yield(%[[COMB_VAL_I]] : i32)
+
+!CHECK-LABEL: @_QPreduction_max_int
+!CHECK-SAME: %[[Y_BOX:.*]]: !fir.box<!fir.array<?xi32>>
+!CHECK:   %[[X_REF:.*]] = fir.alloca i32 {bindc_name = "x", uniq_name = "_QFreduction_max_intEx"}
+!CHECK:   omp.parallel
+!CHECK:     omp.wsloop reduction(@[[MAX_DECLARE_I]] -> %[[X_REF]] : !fir.ref<i32>) for
+!CHECK:       %[[Y_I_REF:.*]] = fir.coordinate_of %[[Y_BOX]]
+!CHECK:       %[[Y_I:.*]] = fir.load %[[Y_I_REF]] : !fir.ref<i32>
+!CHECK:       omp.reduction %[[Y_I]], %[[X_REF]] : i32, !fir.ref<i32>
+!CHECK:       omp.yield
+!CHECK:     omp.terminator
+
+!CHECK-LABEL: @_QPreduction_max_real
+!CHECK-SAME: %[[Y_BOX:.*]]: !fir.box<!fir.array<?xf32>>
+!CHECK:   %[[X_REF:.*]] = fir.alloca f32 {bindc_name = "x", uniq_name = "_QFreduction_max_realEx"}
+!CHECK:   omp.parallel
+!CHECK:     omp.wsloop reduction(@[[MAX_DECLARE_F]] -> %[[X_REF]] : !fir.ref<f32>) for
+!CHECK:       %[[Y_I_REF:.*]] = fir.coordinate_of %[[Y_BOX]]
+!CHECK:       %[[Y_I:.*]] = fir.load %[[Y_I_REF]] : !fir.ref<f32>
+!CHECK:       omp.reduction %[[Y_I]], %[[X_REF]] : f32, !fir.ref<f32>
+!CHECK:       omp.yield
+!CHECK:     omp.terminator
+
+subroutine reduction_max_int(y)
+  integer :: x, y(:)
+  x = 0
+  !$omp parallel
+  !$omp do reduction(max:x)
+  do i=1, 100
+    x = max(x, y(i))
+  end do
+  !$omp end do
+  !$omp end parallel
+  print *, x
+end subroutine
+
+subroutine reduction_max_real(y)
+  real :: x, y(:)
+  x = 0.0
+  !$omp parallel
+  !$omp do reduction(max:x)
+  do i=1, 100
+    x = max(y(i), x)
+  end do
+  !$omp end do
+  !$omp end parallel
+  print *, x
+end subroutine
diff --git a/flang/test/Lower/allocatable-polymorphic.f90 b/flang/test/Lower/allocatable-polymorphic.f90
index 674c6b989b0ee..b129c7c8cdbe1 100644
--- a/flang/test/Lower/allocatable-polymorphic.f90
+++ b/flang/test/Lower/allocatable-polymorphic.f90
@@ -103,8 +103,8 @@ subroutine test_pointer()
 ! CHECK: %{{.*}} = fir.call @_FortranAPointerAllocate(%[[P_DESC_CAST]], %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) {{.*}}: (!fir.ref<!fir.box<none>>, i1, !fir.box<none>, !fir.ref<i8>, i32) -> i32
 
 ! call p%proc1()
-! CHECK: %[[P_CAST:.*]] = fir.convert %[[P_DESC:.*]] : (!fir.ref<!fir.class<!fir.ptr<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>) -> !fir.class<!fir.ptr<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>
-! CHECK: fir.dispatch "proc1"(%[[P_CAST]] : !fir.class<!fir.ptr<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>)
+! CHECK: %[[P_LOAD:.*]] = fir.load %[[P_DESC:.*]] : !fir.ref<!fir.class<!fir.ptr<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>
+! CHECK: fir.dispatch "proc1"(%[[P_LOAD]] : !fir.class<!fir.ptr<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>)
 
 ! CHECK: %[[TYPE_DESC_P1:.*]] = fir.type_desc !fir.type<_QMpolyTp1{a:i32,b:i32}>
 ! CHECK: %[[C1_DESC_CAST:.*]] = fir.convert %[[C1_DESC:.*]] : (!fir.ref<!fir.class<!fir.ptr<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>) -> !fir.ref<!fir.box<none>>
@@ -125,12 +125,12 @@ subroutine test_pointer()
 ! CHECK: %{{.*}} = fir.call @_FortranAPointerAllocate(%[[C2_DESC_CAST]], %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) {{.*}}: (!fir.ref<!fir.box<none>>, i1, !fir.box<none>, !fir.ref<i8>, i32) -> i32
 
 ! call c1%proc1()
-! CHECK: %[[C1_DESC_CAST:.*]] = fir.convert %[[C1_DESC]] : (!fir.ref<!fir.class<!fir.ptr<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>) -> !fir.class<!fir.ptr<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>
-! CHECK: fir.dispatch "proc1"(%[[C1_DESC_CAST]] : !fir.class<!fir.ptr<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>)
+! CHECK: %[[C1_DESC_LOAD:.*]] = fir.load %[[C1_DESC]] : !fir.ref<!fir.class<!fir.ptr<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>
+! CHECK: fir.dispatch "proc1"(%[[C1_DESC_LOAD]] : !fir.class<!fir.ptr<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>)
 
 ! call c2%proc1()
-! CHECK: %[[C2_DESC_CAST:.*]] = fir.convert %[[C2_DESC]] : (!fir.ref<!fir.class<!fir.ptr<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>) -> !fir.class<!fir.ptr<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>
-! CHECK: fir.dispatch "proc1"(%[[C2_DESC_CAST]] : !fir.class<!fir.ptr<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>)
+! CHECK: %[[C2_DESC_LOAD:.*]] = fir.load %[[C2_DESC]] : !fir.ref<!fir.class<!fir.ptr<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>
+! CHECK: fir.dispatch "proc1"(%[[C2_DESC_LOAD]] : !fir.class<!fir.ptr<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>)
 
 ! call c1%proc2()
 ! CHECK: %[[C1_LOAD:.*]] = fir.load %[[C1_DESC]] : !fir.ref<!fir.class<!fir.ptr<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>
@@ -258,7 +258,7 @@ subroutine test_allocatable()
 ! CHECK: %[[TYPE_DESC_P1_CAST:.*]] = fir.convert %[[TYPE_DESC_P1]] : (!fir.tdesc<!fir.type<_QMpolyTp1{a:i32,b:i32}>>) -> !fir.ref<none>
 ! CHECK: %[[RANK:.*]] = arith.constant 0 : i32
 ! CHECK: %[[C0:.*]] = arith.constant 0 : i32
-! CHECK: fir.call @_FortranAAllocatableInitDerived(%[[P_CAST]], %[[TYPE_DESC_P1_CAST]], %[[RANK]], %[[C0]]) {{.*}}: (!fir.ref<!fir.box<none>>, !fir.ref<none>, i32, i32) -> none
+! CHECK: fir.call @_FortranAAllocatableInitDerivedForAllocate(%[[P_CAST]], %[[TYPE_DESC_P1_CAST]], %[[RANK]], %[[C0]]) {{.*}}: (!fir.ref<!fir.box<none>>, !fir.ref<none>, i32, i32) -> none
 ! CHECK: %[[P_CAST:.*]] = fir.convert %[[P]] : (!fir.ref<!fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>) -> !fir.ref<!fir.box<none>>
 ! CHECK: %{{.*}} = fir.call @_FortranAAllocatableAllocate(%[[P_CAST]], %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) {{.*}}: (!fir.ref<!fir.box<none>>, i1, !fir.box<none>, !fir.ref<i8>, i32) -> i32
 
@@ -267,7 +267,7 @@ subroutine test_allocatable()
 ! CHECK: %[[TYPE_DESC_P1_CAST:.*]] = fir.convert %[[TYPE_DESC_P1]] : (!fir.tdesc<!fir.type<_QMpolyTp1{a:i32,b:i32}>>) -> !fir.ref<none>
 ! CHECK: %[[RANK:.*]] = arith.constant 0 : i32
 ! CHECK: %[[C0:.*]] = arith.constant 0 : i32
-! CHECK: fir.call @_FortranAAllocatableInitDerived(%[[C1_CAST]], %[[TYPE_DESC_P1_CAST]], %[[RANK]], %[[C0]]) {{.*}}: (!fir.ref<!fir.box<none>>, !fir.ref<none>, i32, i32) -> none
+! CHECK: fir.call @_FortranAAllocatableInitDerivedForAllocate(%[[C1_CAST]], %[[TYPE_DESC_P1_CAST]], %[[RANK]], %[[C0]]) {{.*}}: (!fir.ref<!fir.box<none>>, !fir.ref<none>, i32, i32) -> none
 ! CHECK: %[[C1_CAST:.*]] = fir.convert %[[C1]] : (!fir.ref<!fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>) -> !fir.ref<!fir.box<none>>
 ! CHECK: %{{.*}} = fir.call @_FortranAAllocatableAllocate(%[[C1_CAST]], %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) {{.*}}: (!fir.ref<!fir.box<none>>, i1, !fir.box<none>, !fir.ref<i8>, i32) -> i32
 
@@ -276,7 +276,7 @@ subroutine test_allocatable()
 ! CHECK: %[[TYPE_DESC_P2_CAST:.*]] = fir.convert %[[TYPE_DESC_P2]] : (!fir.tdesc<!fir.type<_QMpolyTp2{a:i32,b:i32,c:i32}>>) -> !fir.ref<none>
 ! CHECK: %[[RANK:.*]] = arith.constant 0 : i32
 ! CHECK: %[[C0:.*]] = arith.constant 0 : i32
-! CHECK: fir.call @_FortranAAllocatableInitDerived(%[[C2_CAST]], %[[TYPE_DESC_P2_CAST]], %[[RANK]], %[[C0]]) {{.*}}: (!fir.ref<!fir.box<none>>, !fir.ref<none>, i32, i32) -> none
+! CHECK: fir.call @_FortranAAllocatableInitDerivedForAllocate(%[[C2_CAST]], %[[TYPE_DESC_P2_CAST]], %[[RANK]], %[[C0]]) {{.*}}: (!fir.ref<!fir.box<none>>, !fir.ref<none>, i32, i32) -> none
 ! CHECK: %[[C2_CAST:.*]] = fir.convert %[[C2]] : (!fir.ref<!fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>) -> !fir.ref<!fir.box<none>>
 ! CHECK: %{{.*}} = fir.call @_FortranAAllocatableAllocate(%[[C2_CAST]], %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) {{.*}}: (!fir.ref<!fir.box<none>>, i1, !fir.box<none>, !fir.ref<i8>, i32) -> i32
 
@@ -285,7 +285,7 @@ subroutine test_allocatable()
 ! CHECK: %[[TYPE_DESC_P1_CAST:.*]] = fir.convert %[[TYPE_DESC_P1]] : (!fir.tdesc<!fir.type<_QMpolyTp1{a:i32,b:i32}>>) -> !fir.ref<none>
 ! CHECK: %[[RANK:.*]] = arith.constant 1 : i32
 ! CHECK: %[[C0:.*]] = arith.constant 0 : i32
-! CHECK: fir.call @_FortranAAllocatableInitDerived(%[[C3_CAST]], %[[TYPE_DESC_P1_CAST]], %[[RANK]], %[[C0]]) {{.*}}: (!fir.ref<!fir.box<none>>, !fir.ref<none>, i32, i32) -> none
+! CHECK: fir.call @_FortranAAllocatableInitDerivedForAllocate(%[[C3_CAST]], %[[TYPE_DESC_P1_CAST]], %[[RANK]], %[[C0]]) {{.*}}: (!fir.ref<!fir.box<none>>, !fir.ref<none>, i32, i32) -> none
 ! CHECK: %[[C10:.*]] = arith.constant 10 : i32
 ! CHECK: %[[C0:.*]] = arith.constant 0 : i32
 ! CHECK: %[[C3_CAST:.*]] = fir.convert %[[C3]] : (!fir.ref<!fir.class<!fir.heap<!fir.array<?x!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>>) -> !fir.ref<!fir.box<none>>
@@ -300,7 +300,7 @@ subroutine test_allocatable()
 ! CHECK: %[[TYPE_DESC_P2_CAST:.*]] = fir.convert %[[TYPE_DESC_P2]] : (!fir.tdesc<!fir.type<_QMpolyTp2{a:i32,b:i32,c:i32}>>) -> !fir.ref<none>
 ! CHECK: %[[RANK:.*]] = arith.constant 1 : i32
 ! CHECK: %[[C0:.*]] = arith.constant 0 : i32
-! CHECK: fir.call @_FortranAAllocatableInitDerived(%[[C4_CAST]], %[[TYPE_DESC_P2_CAST]], %[[RANK]], %[[C0]]) {{.*}}: (!fir.ref<!fir.box<none>>, !fir.ref<none>, i32, i32) -> none
+! CHECK: fir.call @_FortranAAllocatableInitDerivedForAllocate(%[[C4_CAST]], %[[TYPE_DESC_P2_CAST]], %[[RANK]], %[[C0]]) {{.*}}: (!fir.ref<!fir.box<none>>, !fir.ref<none>, i32, i32) -> none
 ! CHECK: %[[CST1:.*]] = arith.constant 1 : index
 ! CHECK: %[[C20:.*]] = arith.constant 20 : i32
 ! CHECK: %[[C0:.*]] = arith.constant 0 : i32
@@ -311,18 +311,18 @@ subroutine test_allocatable()
 ! CHECK: %[[C4_CAST:.*]] = fir.convert %[[C4]] : (!fir.ref<!fir.class<!fir.heap<!fir.array<?x!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>>) -> !fir.ref<!fir.box<none>>
 ! CHECK: %{{.*}} = fir.call @_FortranAAllocatableAllocate(%[[C4_CAST]], %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) {{.*}}: (!fir.ref<!fir.box<none>>, i1, !fir.box<none>, !fir.ref<i8>, i32) -> i32
 
-! CHECK: %[[C1_CAST:.*]] = fir.convert %[[C1_DESC]] : (!fir.ref<!fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>) -> !fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>
-! CHECK: fir.dispatch "proc1"(%[[C1_CAST]] : !fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>)
+! CHECK: %[[C1_LOAD1:.*]] = fir.load %[[C1_DESC]] : !fir.ref<!fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>
+! CHECK: fir.dispatch "proc1"(%[[C1_LOAD1]] : !fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>)
 
-! CHECK: %[[C2_CAST:.*]] = fir.convert %[[C2_DESC]] : (!fir.ref<!fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>) -> !fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>
-! CHECK: fir.dispatch "proc1"(%[[C2_CAST]] : !fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>)
+! CHECK: %[[C2_LOAD1:.*]] = fir.load %[[C2_DESC]] : !fir.ref<!fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>
+! CHECK: fir.dispatch "proc1"(%[[C2_LOAD1]] : !fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>)
 
-! CHECK: %[[C1_LOAD:.*]] = fir.load %[[C1_DESC]] : !fir.ref<!fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>
-! CHECK: %[[C1_REBOX:.*]] = fir.rebox %[[C1_LOAD]] : (!fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>) -> !fir.class<!fir.type<_QMpolyTp1{a:i32,b:i32}>>
+! CHECK: %[[C1_LOAD2:.*]] = fir.load %[[C1_DESC]] : !fir.ref<!fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>
+! CHECK: %[[C1_REBOX:.*]] = fir.rebox %[[C1_LOAD2]] : (!fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>) -> !fir.class<!fir.type<_QMpolyTp1{a:i32,b:i32}>>
 ! CHECK: fir.dispatch "proc2"(%[[C1_REBOX]] : !fir.class<!fir.type<_QMpolyTp1{a:i32,b:i32}>>) (%[[C1_REBOX]] : !fir.class<!fir.type<_QMpolyTp1{a:i32,b:i32}>>) {pass_arg_pos = 0 : i32}
 
-! CHECK: %[[C2_LOAD:.*]] = fir.load %[[C2_DESC]] : !fir.ref<!fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>
-! CHECK: %[[C2_REBOX:.*]] = fir.rebox %[[C2_LOAD]] : (!fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>) -> !fir.class<!fir.type<_QMpolyTp1{a:i32,b:i32}>>
+! CHECK: %[[C2_LOAD2:.*]] = fir.load %[[C2_DESC]] : !fir.ref<!fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>>
+! CHECK: %[[C2_REBOX:.*]] = fir.rebox %[[C2_LOAD2]] : (!fir.class<!fir.heap<!fir.type<_QMpolyTp1{a:i32,b:i32}>>>) -> !fir.class<!fir.type<_QMpolyTp1{a:i32,b:i32}>>
 ! CHECK: fir.dispatch "proc2"(%[[C2_REBOX]] : !fir.class<!fir.type<_QMpolyTp1{a:i32,b:i32}>>) (%[[C2_REBOX]] : !fir.class<!fir.type<_QMpolyTp1{a:i32,b:i32}>>) {pass_arg_pos = 0 : i32}
 
 ! CHECK-LABEL: %{{.*}} = fir.do_loop
@@ -381,7 +381,7 @@ subroutine test_unlimited_polymorphic_with_intrinsic_type_spec()
 ! CHECK: %[[KIND:.*]] = arith.constant 4 : i32
 ! CHECK: %[[RANK:.*]] = arith.constant 0 : i32
 ! CHECK: %[[CORANK:.*]] = arith.constant 0 : i32
-! CHECK: %{{.*}} = fir.call @_FortranAAllocatableInitIntrinsic(%[[BOX_NONE]], %[[CAT]], %[[KIND]], %[[RANK]], %[[CORANK]]) {{.*}} : (!fir.ref<!fir.box<none>>, i32, i32, i32, i32) -> none
+! CHECK: %{{.*}} = fir.call @_FortranAAllocatableInitIntrinsicForAllocate(%[[BOX_NONE]], %[[CAT]], %[[KIND]], %[[RANK]], %[[CORANK]]) {{.*}} : (!fir.ref<!fir.box<none>>, i32, i32, i32, i32) -> none
 ! CHECK: %[[BOX_NONE:.*]] = fir.convert %[[P]] : (!fir.ref<!fir.class<!fir.heap<none>>>) -> !fir.ref<!fir.box<none>>
 ! CHECK: %{{.*}} = fir.call @_FortranAAllocatableAllocate(%[[BOX_NONE]], %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) {{.*}} : (!fir.ref<!fir.box<none>>, i1, !fir.box<none>, !fir.ref<i8>, i32) -> i32
 
@@ -548,7 +548,7 @@ subroutine test_allocatable_up_character()
 ! CHECK: %[[KIND:.*]] = arith.constant 1 : i32
 ! CHECK: %[[RANK:.*]] = arith.constant 0 : i32
 ! CHECK: %[[CORANK:.*]] = arith.constant 0 : i32
-! CHECK: %{{.*}} = fir.call @_FortranAAllocatableInitCharacter(%[[A_NONE]], %[[LEN]], %[[KIND]], %[[RANK]], %[[CORANK]]) {{.*}} : (!fir.ref<!fir.box<none>>, i64, i32, i32, i32) -> none
+! CHECK: %{{.*}} = fir.call @_FortranAAllocatableInitCharacterForAllocate(%[[A_NONE]], %[[LEN]], %[[KIND]], %[[RANK]], %[[CORANK]]) {{.*}} : (!fir.ref<!fir.box<none>>, i64, i32, i32, i32) -> none
 ! CHECK: %[[A_NONE:.*]] = fir.convert %[[A:.*]] : (!fir.ref<!fir.class<!fir.heap<none>>>) -> !fir.ref<!fir.box<none>>
 ! CHECK: %{{.*}} = fir.call @_FortranAAllocatableAllocate(%[[A_NONE]], %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) {{.*}} : (!fir.ref<!fir.box<none>>, i1, !fir.box<none>, !fir.ref<i8>, i32) -> i32
 
@@ -568,16 +568,16 @@ program test_alloc
 
 ! LLVM-LABEL: define void @_QMpolyPtest_allocatable()
 
-! LLVM: %{{.*}} = call {} @_FortranAAllocatableInitDerived(ptr %{{.*}}, ptr @_QMpolyE.dt.p1, i32 0, i32 0)
+! LLVM: %{{.*}} = call {} @_FortranAAllocatableInitDerivedForAllocate(ptr %{{.*}}, ptr @_QMpolyE.dt.p1, i32 0, i32 0)
 ! LLVM: %{{.*}} = call i32 @_FortranAAllocatableAllocate(ptr %{{.*}}, i1 false, ptr null, ptr @_QQcl.{{.*}}, i32 {{.*}})
-! LLVM: %{{.*}} = call {} @_FortranAAllocatableInitDerived(ptr %{{.*}}, ptr @_QMpolyE.dt.p1, i32 0, i32 0)
+! LLVM: %{{.*}} = call {} @_FortranAAllocatableInitDerivedForAllocate(ptr %{{.*}}, ptr @_QMpolyE.dt.p1, i32 0, i32 0)
 ! LLVM: %{{.*}} = call i32 @_FortranAAllocatableAllocate(ptr %{{.*}}, i1 false, ptr null, ptr @_QQcl.{{.*}}, i32 {{.*}})
-! LLVM: %{{.*}} = call {} @_FortranAAllocatableInitDerived(ptr %{{.*}}, ptr @_QMpolyE.dt.p2, i32 0, i32 0)
+! LLVM: %{{.*}} = call {} @_FortranAAllocatableInitDerivedForAllocate(ptr %{{.*}}, ptr @_QMpolyE.dt.p2, i32 0, i32 0)
 ! LLVM: %{{.*}} = call i32 @_FortranAAllocatableAllocate(ptr %{{.*}}, i1 false, ptr null, ptr @_QQcl.{{.*}}, i32 {{.*}})
-! LLVM: %{{.*}} = call {} @_FortranAAllocatableInitDerived(ptr %{{.*}}, ptr @_QMpolyE.dt.p1, i32 1, i32 0)
+! LLVM: %{{.*}} = call {} @_FortranAAllocatableInitDerivedForAllocate(ptr %{{.*}}, ptr @_QMpolyE.dt.p1, i32 1, i32 0)
 ! LLVM: %{{.*}} = call {} @_FortranAAllocatableSetBounds(ptr %{{.*}}, i32 0, i64 1, i64 10)
 ! LLVM: %{{.*}} = call i32 @_FortranAAllocatableAllocate(ptr %{{.*}}, i1 false, ptr null, ptr @_QQcl.{{.*}}, i32 {{.*}})
-! LLVM: %{{.*}} = call {} @_FortranAAllocatableInitDerived(ptr %{{.*}}, ptr @_QMpolyE.dt.p2, i32 1, i32 0)
+! LLVM: %{{.*}} = call {} @_FortranAAllocatableInitDerivedForAllocate(ptr %{{.*}}, ptr @_QMpolyE.dt.p2, i32 1, i32 0)
 ! LLVM: %{{.*}} = call {} @_FortranAAllocatableSetBounds(ptr %{{.*}}, i32 0, i64 1, i64 20)
 ! LLVM: %{{.*}} = call i32 @_FortranAAllocatableAllocate(ptr %{{.*}}, i1 false, ptr null, ptr @_QQcl.{{.*}}, i32 {{.*}})
 ! LLVM-COUNT-2:  call void %{{.*}}()
@@ -661,6 +661,6 @@ program test_alloc
 ! LLVM: store { ptr, i64, i32, i8, i8, i8, i8, ptr, [1 x i64] } { ptr null, i64 ptrtoint (ptr getelementptr (%_QMpolyTp1, ptr null, i32 1) to i64), i32 20180515, i8 0, i8 42, i8 2, i8 1, ptr @_QMpolyE.dt.p1, [1 x i64] undef }, ptr %[[ALLOCA1]]
 ! LLVM: %[[LOAD:.*]] = load { ptr, i64, i32, i8, i8, i8, i8, ptr, [1 x i64] }, ptr %[[ALLOCA1]]
 ! LLVM: store { ptr, i64, i32, i8, i8, i8, i8, ptr, [1 x i64] } %[[LOAD]], ptr %[[ALLOCA2]]
-! LLVM: %{{.*}} = call {} @_FortranAAllocatableInitDerived(ptr %[[ALLOCA2]], ptr @_QMpolyE.dt.p1, i32 0, i32 0)
+! LLVM: %{{.*}} = call {} @_FortranAAllocatableInitDerivedForAllocate(ptr %[[ALLOCA2]], ptr @_QMpolyE.dt.p1, i32 0, i32 0)
 ! LLVM: %{{.*}} = call i32 @_FortranAAllocatableAllocate(ptr %[[ALLOCA2]], i1 false, ptr null, ptr @_QQcl.{{.*}}, i32 {{.*}})
 ! LLVM: %{{.*}} = call i32 @_FortranAAllocatableDeallocatePolymorphic(ptr %[[ALLOCA2]], ptr {{.*}}, i1 false, ptr null, ptr @_QQcl.{{.*}}, i32 {{.*}})
diff --git a/flang/test/Lower/allocatable-runtime.f90 b/flang/test/Lower/allocatable-runtime.f90
index 0195cc349bc85..74c75218e00b0 100644
--- a/flang/test/Lower/allocatable-runtime.f90
+++ b/flang/test/Lower/allocatable-runtime.f90
@@ -77,14 +77,14 @@ subroutine char_deferred(n)
   allocate(character(10):: scalar, array(30))
   ! CHECK-DAG: %[[sBoxCast1:.*]] = fir.convert %[[sBoxAddr]] : (!fir.ref<!fir.box<!fir.heap<!fir.char<1,?>>>>) -> !fir.ref<!fir.box<none>>
   ! CHECK-DAG: %[[ten1:.*]] = fir.convert %c10{{.*}} : (i32) -> i64
-  ! CHECK: fir.call @{{.*}}AllocatableInitCharacter(%[[sBoxCast1]], %[[ten1]], %c1{{.*}}, %c0{{.*}}, %c0{{.*}})
+  ! CHECK: fir.call @{{.*}}AllocatableInitCharacterForAllocate(%[[sBoxCast1]], %[[ten1]], %c1{{.*}}, %c0{{.*}}, %c0{{.*}})
   ! CHECK-NOT: AllocatableSetBounds
   ! CHECK: %[[sBoxCast2:.*]] = fir.convert %[[sBoxAddr]] : (!fir.ref<!fir.box<!fir.heap<!fir.char<1,?>>>>) -> !fir.ref<!fir.box<none>>
   ! CHECK: fir.call @{{.*}}AllocatableAllocate(%[[sBoxCast2]]
 
   ! CHECK-DAG: %[[aBoxCast1:.*]] = fir.convert %[[aBoxAddr]] : (!fir.ref<!fir.box<!fir.heap<!fir.array<?x!fir.char<1,?>>>>>) -> !fir.ref<!fir.box<none>>
   ! CHECK-DAG: %[[ten2:.*]] = fir.convert %c10{{.*}} : (i32) -> i64
-  ! CHECK: fir.call @{{.*}}AllocatableInitCharacter(%[[aBoxCast1]], %[[ten2]], %c1{{.*}}, %c1{{.*}}, %c0{{.*}})
+  ! CHECK: fir.call @{{.*}}AllocatableInitCharacterForAllocate(%[[aBoxCast1]], %[[ten2]], %c1{{.*}}, %c1{{.*}}, %c0{{.*}})
   ! CHECK: %[[aBoxCast2:.*]] = fir.convert %[[aBoxAddr]] : (!fir.ref<!fir.box<!fir.heap<!fir.array<?x!fir.char<1,?>>>>>) -> !fir.ref<!fir.box<none>>
   ! CHECK: fir.call @{{.*}}AllocatableSetBounds(%[[aBoxCast2]]
   ! CHECK: %[[aBoxCast3:.*]] = fir.convert %[[aBoxAddr]] : (!fir.ref<!fir.box<!fir.heap<!fir.array<?x!fir.char<1,?>>>>>) -> !fir.ref<!fir.box<none>>
@@ -101,10 +101,10 @@ subroutine char_deferred(n)
   ! CHECK: %[[n:.*]] = fir.load %arg0 : !fir.ref<i32>
   ! CHECK-DAG: %[[ncast1:.*]] = fir.convert %[[n]] : (i32) -> i64
   ! CHECK-DAG: %[[sBoxCast4:.*]] = fir.convert %[[sBoxAddr]] : (!fir.ref<!fir.box<!fir.heap<!fir.char<1,?>>>>) -> !fir.ref<!fir.box<none>>
-  ! CHECK: fir.call @{{.*}}AllocatableInitCharacter(%[[sBoxCast4]], %[[ncast1]], %c1{{.*}}, %c0{{.*}}, %c0{{.*}})
+  ! CHECK: fir.call @{{.*}}AllocatableInitCharacterForAllocate(%[[sBoxCast4]], %[[ncast1]], %c1{{.*}}, %c0{{.*}}, %c0{{.*}})
   ! CHECK-DAG: %[[ncast2:.*]] = fir.convert %[[n]] : (i32) -> i64
   ! CHECK-DAG: %[[aBoxCast5:.*]] = fir.convert %[[aBoxAddr]] : (!fir.ref<!fir.box<!fir.heap<!fir.array<?x!fir.char<1,?>>>>>) -> !fir.ref<!fir.box<none>>
-  ! CHECK: fir.call @{{.*}}AllocatableInitCharacter(%[[aBoxCast5]], %[[ncast2]], %c1{{.*}}, %c1{{.*}}, %c0{{.*}})
+  ! CHECK: fir.call @{{.*}}AllocatableInitCharacterForAllocate(%[[aBoxCast5]], %[[ncast2]], %c1{{.*}}, %c1{{.*}}, %c0{{.*}})
 end subroutine
 
 ! CHECK-LABEL: _QPchar_explicit_cst(
diff --git a/flang/test/Lower/allocate-mold.f90 b/flang/test/Lower/allocate-mold.f90
index 6a1b0aaf73813..245141d94ca42 100644
--- a/flang/test/Lower/allocate-mold.f90
+++ b/flang/test/Lower/allocate-mold.f90
@@ -17,3 +17,27 @@ subroutine scalar_mold_allocation()
 ! CHECK: %{{.*}} = fir.call @_FortranAAllocatableApplyMold(%[[A_REF_BOX_NONE1]], %{{.*}}, %{{.*}}) {{.*}} : (!fir.ref<!fir.box<none>>, !fir.box<none>, i32) -> none
 ! CHECK: %[[A_REF_BOX_NONE2:.*]] = fir.convert %[[A]] : (!fir.ref<!fir.box<!fir.heap<i32>>>) -> !fir.ref<!fir.box<none>>
 ! CHECK: %{{.*}} = fir.call @_FortranAAllocatableAllocate(%[[A_REF_BOX_NONE2]], %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) {{.*}} : (!fir.ref<!fir.box<none>>, i1, !fir.box<none>, !fir.ref<i8>, i32) -> i32
+
+subroutine array_scalar_mold_allocation()
+  real, allocatable :: a(:)
+
+  allocate (a(10), mold=3.0)
+end subroutine array_scalar_mold_allocation
+
+! CHECK-LABEL: func.func @_QParray_scalar_mold_allocation() {
+! CHECK: %[[A:.*]] = fir.alloca !fir.box<!fir.heap<!fir.array<?xf32>>> {bindc_name = "a", uniq_name = "_QFarray_scalar_mold_allocationEa"}
+! CHECK: %[[HEAP_A:.*]] = fir.alloca !fir.heap<!fir.array<?xf32>> {uniq_name = "_QFarray_scalar_mold_allocationEa.addr"}
+! CHECK: %[[EXT0:.*]] = fir.alloca index {uniq_name = "_QFarray_scalar_mold_allocationEa.ext0"}
+! CHECK: %[[ZERO:.*]] = fir.zero_bits !fir.heap<!fir.array<?xf32>>
+! CHECK: fir.store %[[ZERO]] to %[[HEAP_A]] : !fir.ref<!fir.heap<!fir.array<?xf32>>>
+! CHECK: %[[LOADED_A:.*]] = fir.load %[[HEAP_A]] : !fir.ref<!fir.heap<!fir.array<?xf32>>>
+! CHECK: %[[SHAPESHIFT:.*]] = fir.shape_shift {{.*}}, {{.*}} : (index, index) -> !fir.shapeshift<1>
+! CHECK: %[[BOX_SHAPESHIFT:.*]] = fir.embox %[[LOADED_A]](%[[SHAPESHIFT]]) : (!fir.heap<!fir.array<?xf32>>, !fir.shapeshift<1>) -> !fir.box<!fir.heap<!fir.array<?xf32>>>
+! CHECK: fir.store %[[BOX_SHAPESHIFT]] to %[[A]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>
+! CHECK: %[[REF_BOX_A0:.*]] = fir.convert %1 : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>) -> !fir.ref<!fir.box<none>>
+! CHECK: %{{.*}} = fir.call @_FortranAAllocatableApplyMold(%[[REF_BOX_A0]], {{.*}}, {{.*}}) fastmath<contract> : (!fir.ref<!fir.box<none>>, !fir.box<none>, i32) -> none
+! CHECK: %[[C10:.*]] = arith.constant 10 : i32
+! CHECK: %[[REF_BOX_A1:.*]] = fir.convert %1 : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>) -> !fir.ref<!fir.box<none>>
+! CHECK: %{{.*}} = fir.call @_FortranAAllocatableSetBounds(%[[REF_BOX_A1]], {{.*}},{{.*}}, {{.*}}) fastmath<contract> : (!fir.ref<!fir.box<none>>, i32, i64, i64) -> none
+! CHECK: %[[REF_BOX_A2:.*]] = fir.convert %[[A]] : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>) -> !fir.ref<!fir.box<none>>
+! CHECK: %{{.*}} = fir.call @_FortranAAllocatableAllocate(%[[REF_BOX_A2]], %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) {{.*}} : (!fir.ref<!fir.box<none>>, i1, !fir.box<none>, !fir.ref<i8>, i32) -> i32
diff --git a/flang/test/Lower/allocate-source-allocatables.f90 b/flang/test/Lower/allocate-source-allocatables.f90
index f27e660afe311..00dd3852d58a4 100644
--- a/flang/test/Lower/allocate-source-allocatables.f90
+++ b/flang/test/Lower/allocate-source-allocatables.f90
@@ -311,7 +311,7 @@ subroutine test_allocatable_chararray(n, a)
 ! CHECK:         %[[VAL_18:.*]] = arith.constant 1 : i32
 ! CHECK:         %[[VAL_19:.*]] = arith.constant 0 : i32
 ! CHECK:         %[[VAL_20:.*]] = arith.constant 0 : i32
-! CHECK:         %[[VAL_21:.*]] = fir.call @_FortranAAllocatableInitCharacter(%[[VAL_16]], %[[VAL_17]], %[[VAL_18]], %[[VAL_19]], %[[VAL_20]]) {{.*}}: (!fir.ref<!fir.box<none>>, i64, i32, i32, i32) -> none
+! CHECK:         %[[VAL_21:.*]] = fir.call @_FortranAAllocatableInitCharacterForAllocate(%[[VAL_16]], %[[VAL_17]], %[[VAL_18]], %[[VAL_19]], %[[VAL_20]]) {{.*}}: (!fir.ref<!fir.box<none>>, i64, i32, i32, i32) -> none
 ! CHECK:         %[[VAL_22:.*]] = fir.convert %[[VAL_3]] : (!fir.ref<!fir.box<!fir.heap<!fir.char<1,?>>>>) -> !fir.ref<!fir.box<none>>
 ! CHECK:         %[[VAL_23:.*]] = fir.convert %[[VAL_11]] : (!fir.box<!fir.char<1,?>>) -> !fir.box<none>
 ! CHECK:         %[[VAL_25:.*]] = fir.call @_FortranAAllocatableAllocateSource(%[[VAL_22]], %[[VAL_23]], %[[VAL_7]], %[[VAL_8]], %{{.*}}, %{{.*}}) {{.*}}: (!fir.ref<!fir.box<none>>, !fir.box<none>, i1, !fir.box<none>, !fir.ref<i8>, i32) -> i32
diff --git a/flang/test/Lower/assumed-type.f90 b/flang/test/Lower/assumed-type.f90
new file mode 100644
index 0000000000000..b877a1639b36e
--- /dev/null
+++ b/flang/test/Lower/assumed-type.f90
@@ -0,0 +1,42 @@
+! RUN: bbc -polymorphic-type -emit-fir %s -o - | FileCheck %s
+
+module assumed_type_test
+
+  interface
+    subroutine assumed(a)
+      type(*), intent(in), target :: a
+    end subroutine
+  end interface
+
+  interface
+    subroutine assumed_r(a)
+      type(*), intent(in), target :: a(*)
+    end subroutine
+  end interface
+
+contains
+
+  subroutine call_assumed()
+    integer, target :: i
+    call assumed(i)
+  end subroutine
+
+! CHECK-LABEL: func.func @_QMassumed_type_testPcall_assumed() {
+! CHECK: %[[I:.*]] = fir.alloca i32 {bindc_name = "i", fir.target, uniq_name = "_QMassumed_type_testFcall_assumedEi"}
+! CHECK: %[[BOX_NONE:.*]] = fir.embox %[[I]] : (!fir.ref<i32>) -> !fir.box<none>
+! CHECK: fir.call @_QPassumed(%[[BOX_NONE]]) fastmath<contract> : (!fir.box<none>) -> ()
+
+  subroutine call_assumed_r()
+    integer, target :: i(10)
+    call assumed_r(i)
+  end subroutine
+
+! CHECK-LABEL: func.func @_QMassumed_type_testPcall_assumed_r() {
+! CHECK: %[[C10:.*]] = arith.constant 10 : index
+! CHECK: %[[I:.*]] = fir.alloca !fir.array<10xi32> {bindc_name = "i", fir.target, uniq_name = "_QMassumed_type_testFcall_assumed_rEi"}
+! CHECK: %[[SHAPE:.*]] = fir.shape %[[C10]] : (index) -> !fir.shape<1>
+! CHECK: %[[BOX_NONE:.*]] = fir.embox %[[I]](%[[SHAPE]]) : (!fir.ref<!fir.array<10xi32>>, !fir.shape<1>) -> !fir.box<!fir.array<10xnone>>
+! CHECK: %[[CONV:.*]] = fir.convert %[[BOX_NONE]] : (!fir.box<!fir.array<10xnone>>) -> !fir.box<!fir.array<?xnone>>
+! CHECK: fir.call @_QPassumed_r(%[[CONV]]) {{.*}} : (!fir.box<!fir.array<?xnone>>) -> ()
+
+end module
diff --git a/flang/test/Lower/call-by-value.f90 b/flang/test/Lower/call-by-value.f90
index 909d1cc03a1db..8fa8def0f3d99 100644
--- a/flang/test/Lower/call-by-value.f90
+++ b/flang/test/Lower/call-by-value.f90
@@ -2,21 +2,23 @@
 ! RUN: bbc -emit-fir %s -o - | FileCheck %s
 
 !CHECK-LABEL: func @_QQmain()
-!CHECK: %[[LOGICAL:.*]] = fir.alloca !fir.logical<4>
+!CHECK: %[[LOGICAL:.*]] = fir.alloca !fir.logical<1>
 !CHECK: %false = arith.constant false
-!CHECK: %[[VALUE:.*]] = fir.convert %false : (i1) -> !fir.logical<4>
+!CHECK: %[[VALUE:.*]] = fir.convert %false : (i1) -> !fir.logical<1>
 !CHECK: fir.store %[[VALUE]] to %[[LOGICAL]]
 !CHECK: %[[LOAD:.*]] = fir.load %[[LOGICAL]]
 !CHECK: fir.call @omp_set_nested(%[[LOAD]]) {{.*}}: {{.*}}
 
 program call_by_value
+  use iso_c_binding, only: c_bool
   interface
      subroutine omp_set_nested(enable) bind(c)
-       logical, value :: enable
+       import c_bool
+       logical(c_bool), value :: enable
      end subroutine omp_set_nested
   end interface
 
-  logical do_nested
+  logical(c_bool) do_nested
   do_nested = .FALSE.
   call omp_set_nested(do_nested)
 end program call_by_value
diff --git a/flang/test/Lower/derived-type-finalization.f90 b/flang/test/Lower/derived-type-finalization.f90
index 8de16bc986f0a..a93581ac025eb 100644
--- a/flang/test/Lower/derived-type-finalization.f90
+++ b/flang/test/Lower/derived-type-finalization.f90
@@ -10,6 +10,7 @@ module derived_type_finalization
     integer :: a
   contains
     final :: t1_final
+    final :: t1_final_1r
   end type
 
   type :: t2
@@ -28,6 +29,10 @@ subroutine t1_final(this)
     type(t1) :: this
   end subroutine
 
+  subroutine t1_final_1r(this)
+    type(t1) :: this(:)
+  end subroutine
+
   subroutine t2_final(this)
     type(t2) :: this
   end subroutine
@@ -203,6 +208,25 @@ function no_func_ret_finalize() result(ty)
 ! CHECK: %{{.*}} = fir.call @_FortranADestroy
 ! CHECK: return %{{.*}} : !fir.type<_QMderived_type_finalizationTt1{a:i32}>
 
+  function copy(a) result(ty)
+    class(t1), allocatable :: ty(:)
+    integer, intent(in) :: a
+    allocate(t1::ty(a))
+    ty%a = 1
+  end function
+
+  subroutine test_avoid_double_free()
+    class(*), allocatable :: up(:)
+    allocate(up(10), source=copy(10))
+  end subroutine
+
+! CHECK-LABEL: func.func @_QMderived_type_finalizationPtest_avoid_double_free() {
+! CHECK: %[[RES:.*]] = fir.alloca !fir.class<!fir.heap<!fir.array<?x!fir.type<_QMderived_type_finalizationTt1{a:i32}>>>> {bindc_name = ".result"}
+! CHECK: fir.call @_FortranAAllocatableAllocateSource(
+! CHECK-NOT: fir.freemem %{{.*}} : !fir.heap<!fir.array<?x!fir.type<_QMderived_type_finalizationTt1{a:i32}>>>
+! CHECK: %[[RES_CONV:.*]] = fir.convert %[[RES]] : (!fir.ref<!fir.class<!fir.heap<!fir.array<?x!fir.type<_QMderived_type_finalizationTt1{a:i32}>>>>>) -> !fir.box<none>
+! CHECK: %{{.*}} = fir.call @_FortranADestroy(%[[RES_CONV]]) {{.*}} : (!fir.box<none>) -> none
+
 end module
 
 program p
diff --git a/flang/test/Lower/dummy-argument-optional.f90 b/flang/test/Lower/dummy-argument-optional.f90
index 624ed709a7185..e7dafbaea09c9 100644
--- a/flang/test/Lower/dummy-argument-optional.f90
+++ b/flang/test/Lower/dummy-argument-optional.f90
@@ -1,4 +1,5 @@
 ! RUN: bbc -emit-fir %s -o - | FileCheck %s
+! RUN: flang-new -fc1 -fdefault-integer-8 -emit-fir %s -o - | FileCheck %s
 
 ! Test OPTIONAL lowering on caller/callee and PRESENT intrinsic.
 module opt
@@ -68,6 +69,53 @@ subroutine call_character_scalar()
   call character_scalar()
 end subroutine
 
+! Test optional character function
+! CHECK-LABEL: func @_QMoptPchar_proc(
+! CHECK-SAME: %[[arg0:.*]]: !fir.ref<!fir.char<1,3>>,
+character(len=3) function char_proc(i)
+  integer :: i
+  char_proc = "XYZ"
+end function
+! CHECK-LABEL: func @_QMoptPuse_char_proc(
+! CHECK-SAME: %[[arg0:.*]]: tuple<!fir.boxproc<() -> ()>, i64> {fir.char_proc},
+subroutine use_char_proc(f, c)
+  optional :: f
+  interface
+    character(len=3) function f(i)
+      integer :: i
+    end function
+  end interface
+  character(len=3) :: c
+! CHECK: %[[boxProc:.*]] = fir.extract_value %[[arg0]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>) -> !fir.boxproc<() -> ()>
+! CHECK: %[[procAddr:.*]] = fir.box_addr %[[boxProc]] : (!fir.boxproc<() -> ()>) -> (() -> ())
+! CHECK: %{{.*}} = fir.is_present %[[procAddr]] : (() -> ()) -> i1
+  if (present(f)) then
+    c = f(0)
+  else
+    c = "ABC"
+  end if
+end subroutine
+! CHECK-LABEL: func @_QMoptPcall_use_char_proc(
+subroutine call_use_char_proc()
+  character(len=3) :: c
+! CHECK: %[[boxProc:.*]] = fir.absent !fir.boxproc<() -> ()>
+! CHECK: %[[undef:.*]] = fir.undefined index
+! CHECK: %[[charLen:.*]] = fir.convert %[[undef]] : (index) -> i64
+! CHECK: %[[tuple:.*]] = fir.undefined tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple2:.*]] = fir.insert_value %[[tuple]], %[[boxProc]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxproc<() -> ()>) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple3:.*]] = fir.insert_value %[[tuple2]], %[[charLen]], [1 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, i64) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: fir.call @_QMoptPuse_char_proc(%[[tuple3]], %{{.*}}){{.*}} : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxchar<1>) -> ()
+  call use_char_proc(c=c)
+! CHECK: %[[funcAddr:.*]] = fir.address_of(@_QMoptPchar_proc) : (!fir.ref<!fir.char<1,3>>, index, {{.*}}) -> !fir.boxchar<1>
+! CHECK: %[[c3:.*]] = arith.constant 3 : i64
+! CHECK: %[[boxProc2:.*]] = fir.emboxproc %[[funcAddr]] : ((!fir.ref<!fir.char<1,3>>, index, {{.*}}) -> !fir.boxchar<1>) -> !fir.boxproc<() -> ()>
+! CHECK: %[[tuple4:.*]] = fir.undefined tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple5:.*]] = fir.insert_value %[[tuple4]], %[[boxProc2]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxproc<() -> ()>) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple6:.*]] = fir.insert_value %[[tuple5]], %[[c3]], [1 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, i64) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: fir.call @_QMoptPuse_char_proc(%[[tuple6]], {{.*}}){{.*}} : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxchar<1>) -> ()
+  call use_char_proc(char_proc, c)
+end subroutine
+
 ! Test optional assumed shape
 ! CHECK-LABEL: func @_QMoptPassumed_shape(
 ! CHECK-SAME: %[[arg0:.*]]: !fir.box<!fir.array<?xf32>> {fir.bindc_name = "x", fir.optional}) {
diff --git a/flang/test/Lower/implicit-call-mismatch.f90 b/flang/test/Lower/implicit-call-mismatch.f90
new file mode 100644
index 0000000000000..f87dbbd5b46df
--- /dev/null
+++ b/flang/test/Lower/implicit-call-mismatch.f90
@@ -0,0 +1,364 @@
+! Test questionable but existing abuses of implicit interfaces.
+! Lowering must close the eyes and do as if it did not know
+! about the function definition since semantic lets these
+! programs through with a warning.
+! RUN: bbc -emit-fir %s -o - | FileCheck %s
+
+! Test reference to non-char procedure conversion.
+
+subroutine takes_proc(proc)
+  real(8), external :: proc
+end subroutine
+
+subroutine pass_int_to_proc(a)
+  integer(4) :: a
+  call takes_proc(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPpass_int_to_proc(
+! CHECK-SAME: %[[arg0:.*]]: !fir.ref<i32>
+! CHECK: %[[procAddr:.*]] = fir.convert %[[arg0]] : (!fir.ref<i32>) -> (() -> ())
+! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[procAddr]] : (() -> ()) -> !fir.boxproc<() -> ()>
+! CHECK: fir.call @_QPtakes_proc(%[[boxProc]]) {{.*}}: (!fir.boxproc<() -> ()>) -> ()
+
+subroutine pass_logical_to_proc(a)
+  logical(4) :: a
+  call takes_proc(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPpass_logical_to_proc(
+! CHECK-SAME: %[[arg0:.*]]: !fir.ref<!fir.logical<4>>
+! CHECK: %[[procAddr:.*]] = fir.convert %[[arg0]] : (!fir.ref<!fir.logical<4>>) -> (() -> ())
+! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[procAddr]] : (() -> ()) -> !fir.boxproc<() -> ()>
+! CHECK: fir.call @_QPtakes_proc(%[[boxProc]]) {{.*}}: (!fir.boxproc<() -> ()>) -> ()
+
+subroutine pass_real_to_proc(a)
+  real(8) :: a
+  call takes_proc(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPpass_real_to_proc(
+! CHECK-SAME: %[[arg0:.*]]: !fir.ref<f64>
+! CHECK: %[[procAddr:.*]] = fir.convert %[[arg0]] : (!fir.ref<f64>) -> (() -> ())
+! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[procAddr]] : (() -> ()) -> !fir.boxproc<() -> ()>
+! CHECK: fir.call @_QPtakes_proc(%[[boxProc]]) {{.*}}: (!fir.boxproc<() -> ()>) -> ()
+
+subroutine pass_complex_to_proc(a)
+  complex(4) :: a
+  call takes_proc(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPpass_complex_to_proc(
+! CHECK-SAME: %[[arg0:.*]]: !fir.ref<!fir.complex<4>>
+! CHECK: %[[procAddr:.*]] = fir.convert %[[arg0]] : (!fir.ref<!fir.complex<4>>) -> (() -> ())
+! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[procAddr]] : (() -> ()) -> !fir.boxproc<() -> ()>
+! CHECK: fir.call @_QPtakes_proc(%[[boxProc]]) {{.*}}: (!fir.boxproc<() -> ()>) -> ()
+
+subroutine pass_char_to_proc(a)
+  character(8) :: a
+  call takes_proc(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPpass_char_to_proc(
+! CHECK-SAME: %[[arg0:.*]]: !fir.boxchar<1>
+! CHECK: %[[charAndLen:.*]]:2 = fir.unboxchar %[[arg0]] : (!fir.boxchar<1>) -> (!fir.ref<!fir.char<1,?>>, index)
+! CHECK: %[[procAddr:.*]] = fir.convert %[[charAndLen]]#0 : (!fir.ref<!fir.char<1,?>>) -> (() -> ())
+! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[procAddr]] : (() -> ()) -> !fir.boxproc<() -> ()>
+! CHECK: fir.call @_QPtakes_proc(%[[boxProc]]) {{.*}}: (!fir.boxproc<() -> ()>) -> ()
+
+subroutine pass_dt_to_proc(a)
+  type :: dt
+    integer(4) :: i, j
+  end type
+  type(dt) :: a
+
+  call takes_proc(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPpass_dt_to_proc(
+! CHECK-SAME: %[[arg0:.*]]: !fir.ref<!fir.type<_QFpass_dt_to_procTdt{i:i32,j:i32}>>
+! CHECK: %[[procAddr:.*]] = fir.convert %[[arg0]] : (!fir.ref<!fir.type<_QFpass_dt_to_procTdt{i:i32,j:i32}>>) -> (() -> ())
+! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[procAddr]] : (() -> ()) -> !fir.boxproc<() -> ()>
+! CHECK: fir.call @_QPtakes_proc(%[[boxProc]]) {{.*}}: (!fir.boxproc<() -> ()>) -> ()
+
+subroutine pass_array_to_proc(a)
+  integer(4) :: a(10,10)
+  call takes_proc(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPpass_array_to_proc(
+! CHECK-SAME: %[[arg0:.*]]: !fir.ref<!fir.array<10x10xi32>>
+! CHECK: %[[procAddr:.*]] = fir.convert %[[arg0]] : (!fir.ref<!fir.array<10x10xi32>>) -> (() -> ())
+! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[procAddr]] : (() -> ()) -> !fir.boxproc<() -> ()>
+! CHECK: fir.call @_QPtakes_proc(%[[boxProc]]) {{.*}}: (!fir.boxproc<() -> ()>) -> ()
+
+! Test procedure conversion.
+
+subroutine pass_char_proc_to_proc(a)
+  character(8), external :: a
+  call takes_proc(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPpass_char_proc_to_proc(
+! CHECK-SAME: %[[arg0:.*]]: tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[extract:.*]] = fir.extract_value %[[arg0]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>) -> !fir.boxproc<() -> ()>
+! CHECK: %[[procAddr:.*]] = fir.box_addr %[[extract]] : (!fir.boxproc<() -> ()>) -> (() -> ())
+! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[procAddr]] : (() -> ()) -> !fir.boxproc<() -> ()>
+! CHECK: fir.call @_QPtakes_proc(%[[boxProc]]) {{.*}}: (!fir.boxproc<() -> ()>) -> ()
+
+subroutine pass_proc_to_proc(a)
+  external :: a
+  call takes_proc(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPpass_proc_to_proc(
+! CHECK-SAME: %[[arg0:.*]]: !fir.boxproc<() -> ()>
+! CHECK: fir.call @_QPtakes_proc(%[[arg0]]) {{.*}}: (!fir.boxproc<() -> ()>) -> ()
+
+! Test conversion from procedure to other data types.
+
+! CHECK-LABEL: func.func @_QPtest_conversion_from_proc(
+subroutine test_conversion_from_proc
+  external :: proc
+
+  ! CHECK: %[[proc:.*]] = fir.address_of(@_QPproc) : () -> ()
+  ! CHECK: %[[convert:.*]] = fir.convert %[[proc]] : (() -> ()) -> !fir.ref<i32>
+  ! CHECK: fir.call @_QPpass_int_to_proc(%[[convert]])
+  call pass_int_to_proc(proc)
+
+  ! CHECK: %[[proc:.*]] = fir.address_of(@_QPproc) : () -> ()
+  ! CHECK: %[[convert:.*]] = fir.convert %[[proc]] : (() -> ()) -> !fir.ref<!fir.logical<4>>
+  ! CHECK: fir.call @_QPpass_logical_to_proc(%[[convert]])
+  call pass_logical_to_proc(proc)
+
+  ! CHECK: %[[proc:.*]] = fir.address_of(@_QPproc) : () -> ()
+  ! CHECK: %[[convert:.*]] = fir.convert %[[proc]] : (() -> ()) -> !fir.ref<f64>
+  ! CHECK: fir.call @_QPpass_real_to_proc(%[[convert]])
+  call pass_real_to_proc(proc)
+
+  ! CHECK: %[[proc:.*]] = fir.address_of(@_QPproc) : () -> ()
+  ! CHECK: %[[convert:.*]] = fir.convert %[[proc]] : (() -> ()) -> !fir.ref<!fir.complex<4>>
+  ! CHECK: fir.call @_QPpass_complex_to_proc(%[[convert]])
+  call pass_complex_to_proc(proc)
+
+  ! CHECK: %[[proc:.*]] = fir.address_of(@_QPproc) : () -> ()
+  ! CHECK: %[[convert:.*]] = fir.convert %[[proc]] : (() -> ()) -> !fir.ref<!fir.char<1,?>>
+  ! CHECK: %[[len:.*]] = fir.undefined index
+  ! CHECK: %[[box:.*]] = fir.emboxchar %[[convert]], %[[len]] : (!fir.ref<!fir.char<1,?>>, index) -> !fir.boxchar<1>
+  ! CHECK: fir.call @_QPpass_char_to_proc(%[[box]])
+  call pass_char_to_proc(proc)
+
+  ! CHECK: %[[proc:.*]] = fir.address_of(@_QPproc) : () -> ()
+  ! CHECK: %[[convert:.*]] = fir.convert %[[proc]] : (() -> ()) -> !fir.ref<!fir.type<_QFpass_dt_to_procTdt{i:i32,j:i32}>>
+  ! CHECK: fir.call @_QPpass_dt_to_proc(%[[convert]])
+  call pass_dt_to_proc(proc)
+
+  ! CHECK: %[[proc:.*]] = fir.address_of(@_QPproc) : () -> ()
+  ! CHECK: %[[convert:.*]] = fir.convert %[[proc]] : (() -> ()) -> !fir.ref<!fir.array<10x10xi32>>
+  ! CHECK: fir.call @_QPpass_array_to_proc(%[[convert]])
+  call pass_array_to_proc(proc)
+
+  ! CHECK: %[[proc:.*]] = fir.address_of(@_QPproc) : () -> ()
+  ! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[proc]] : (() -> ()) -> !fir.boxproc<() -> ()>
+  ! CHECK: %[[len:.*]] = fir.undefined i64
+  ! CHECK: %[[tuple:.*]] = fir.undefined tuple<!fir.boxproc<() -> ()>, i64>
+  ! CHECK: %[[tuple2:.*]] = fir.insert_value %[[tuple]], %[[boxProc]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxproc<() -> ()>) -> tuple<!fir.boxproc<() -> ()>, i64>
+  ! CHECK: %[[tuple3:.*]] = fir.insert_value %[[tuple2]], %[[len]], [1 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, i64) -> tuple<!fir.boxproc<() -> ()>, i64>
+  ! CHECK: fir.call @_QPpass_char_proc_to_proc(%[[tuple3]]) {{.*}}: (tuple<!fir.boxproc<() -> ()>, i64>) -> ()
+  call pass_char_proc_to_proc(proc)
+
+  ! CHECK: %[[proc:.*]] = fir.address_of(@_QPproc) : () -> ()
+  ! CHECK: %[[box:.*]] = fir.emboxproc %[[proc]] : (() -> ()) -> !fir.boxproc<() -> ()>
+  ! CHECK: fir.call @_QPpass_proc_to_proc(%[[box]])
+  call pass_proc_to_proc(proc)
+end subroutine
+
+! Test reference to char procedure conversion.
+
+subroutine takes_char_proc(cp)
+  character(8), external :: cp
+end subroutine
+
+subroutine pass_int_to_char_proc(a)
+  integer(4) :: a
+  call takes_char_proc(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPpass_int_to_char_proc(
+! CHECK-SAME: %[[arg0:.*]]: !fir.ref<i32>
+! CHECK: %[[procAddr:.*]] = fir.convert %[[arg0]] : (!fir.ref<i32>) -> (() -> ())
+! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[procAddr]] : (() -> ()) -> !fir.boxproc<() -> ()>
+! CHECK: %[[charLen:.*]] = fir.undefined i64
+! CHECK: %[[tuple:.*]] = fir.undefined tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple2:.*]] = fir.insert_value %[[tuple]], %[[boxProc]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxproc<() -> ()>) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple3:.*]] = fir.insert_value %[[tuple2]], %[[charLen]], [1 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, i64) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: fir.call @_QPtakes_char_proc(%[[tuple3]]) {{.*}}: (tuple<!fir.boxproc<() -> ()>, i64>) -> ()
+
+subroutine pass_logical_to_char_proc(a)
+  logical(4) :: a
+  call takes_char_proc(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPpass_logical_to_char_proc(
+! CHECK-SAME: %[[arg0:.*]]: !fir.ref<!fir.logical<4>>
+! CHECK: %[[procAddr:.*]] = fir.convert %[[arg0]] : (!fir.ref<!fir.logical<4>>) -> (() -> ())
+! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[procAddr]] : (() -> ()) -> !fir.boxproc<() -> ()>
+! CHECK: %[[charLen:.*]] = fir.undefined i64
+! CHECK: %[[tuple:.*]] = fir.undefined tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple2:.*]] = fir.insert_value %[[tuple]], %[[boxProc]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxproc<() -> ()>) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple3:.*]] = fir.insert_value %[[tuple2]], %[[charLen]], [1 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, i64) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: fir.call @_QPtakes_char_proc(%[[tuple3]]) {{.*}}: (tuple<!fir.boxproc<() -> ()>, i64>) -> ()
+
+subroutine pass_real_to_char_proc(a)
+  real(8) :: a
+  call takes_char_proc(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPpass_real_to_char_proc(
+! CHECK-SAME: %[[arg0:.*]]: !fir.ref<f64>
+! CHECK: %[[procAddr:.*]] = fir.convert %[[arg0]] : (!fir.ref<f64>) -> (() -> ())
+! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[procAddr]] : (() -> ()) -> !fir.boxproc<() -> ()>
+! CHECK: %[[charLen:.*]] = fir.undefined i64
+! CHECK: %[[tuple:.*]] = fir.undefined tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple2:.*]] = fir.insert_value %[[tuple]], %[[boxProc]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxproc<() -> ()>) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple3:.*]] = fir.insert_value %[[tuple2]], %[[charLen]], [1 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, i64) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: fir.call @_QPtakes_char_proc(%[[tuple3]]) {{.*}}: (tuple<!fir.boxproc<() -> ()>, i64>) -> ()
+
+subroutine pass_complex_to_char_proc(a)
+  complex(4) :: a
+  call takes_char_proc(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPpass_complex_to_char_proc(
+! CHECK-SAME: %[[arg0:.*]]: !fir.ref<!fir.complex<4>>
+! CHECK: %[[procAddr:.*]] = fir.convert %[[arg0]] : (!fir.ref<!fir.complex<4>>) -> (() -> ())
+! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[procAddr]] : (() -> ()) -> !fir.boxproc<() -> ()>
+! CHECK: %[[charLen:.*]] = fir.undefined i64
+! CHECK: %[[tuple:.*]] = fir.undefined tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple2:.*]] = fir.insert_value %[[tuple]], %[[boxProc]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxproc<() -> ()>) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple3:.*]] = fir.insert_value %[[tuple2]], %[[charLen]], [1 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, i64) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: fir.call @_QPtakes_char_proc(%[[tuple3]]) {{.*}}: (tuple<!fir.boxproc<() -> ()>, i64>) -> ()
+
+subroutine pass_char_to_char_proc(a)
+  character(8) :: a
+  call takes_char_proc(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPpass_char_to_char_proc(
+! CHECK-SAME: %[[arg0:.*]]: !fir.boxchar<1>
+! CHECK: %[[charRefAndLen:.*]]:2 = fir.unboxchar %[[arg0]] : (!fir.boxchar<1>) -> (!fir.ref<!fir.char<1,?>>, index)
+! CHECK: %[[charLen:.*]] = arith.constant 8 : index
+! CHECK: %[[procAddr:.*]] = fir.convert %[[charRefAndLen]]#0 : (!fir.ref<!fir.char<1,?>>) -> (() -> ())
+! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[procAddr]] : (() -> ()) -> !fir.boxproc<() -> ()>
+! CHECK: %[[charLen2:.*]] = fir.convert %[[charLen]] : (index) -> i64
+! CHECK: %[[tuple:.*]] = fir.undefined tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple2:.*]] = fir.insert_value %[[tuple]], %[[boxProc]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxproc<() -> ()>) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple3:.*]] = fir.insert_value %[[tuple2]], %[[charLen2]], [1 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, i64) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: fir.call @_QPtakes_char_proc(%[[tuple3]]) {{.*}}: (tuple<!fir.boxproc<() -> ()>, i64>) -> ()
+
+subroutine pass_dt_to_char_proc(a)
+  type :: dt
+    integer(4) :: i, j
+  end type
+  type(dt) :: a
+
+  call takes_char_proc(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPpass_dt_to_char_proc(
+! CHECK-SAME: %[[arg0:.*]]: !fir.ref<!fir.type<_QFpass_dt_to_char_procTdt{i:i32,j:i32}>>
+! CHECK: %[[procAddr:.*]] = fir.convert %[[arg0]] : (!fir.ref<!fir.type<_QFpass_dt_to_char_procTdt{i:i32,j:i32}>>) -> (() -> ())
+! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[procAddr]] : (() -> ()) -> !fir.boxproc<() -> ()>
+! CHECK: %[[charLen:.*]] = fir.undefined i64
+! CHECK: %[[tuple:.*]] = fir.undefined tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple2:.*]] = fir.insert_value %[[tuple]], %[[boxProc]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxproc<() -> ()>) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple3:.*]] = fir.insert_value %[[tuple2]], %[[charLen]], [1 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, i64) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: fir.call @_QPtakes_char_proc(%[[tuple3]]) {{.*}}: (tuple<!fir.boxproc<() -> ()>, i64>) -> ()
+
+subroutine pass_array_to_char_proc(a)
+  integer(4) :: a(10,10)
+  call takes_char_proc(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPpass_array_to_char_proc(
+! CHECK-SAME: %[[arg0:.*]]: !fir.ref<!fir.array<10x10xi32>>
+! CHECK: %[[procAddr:.*]] = fir.convert %[[arg0]] : (!fir.ref<!fir.array<10x10xi32>>) -> (() -> ())
+! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[procAddr]] : (() -> ()) -> !fir.boxproc<() -> ()>
+! CHECK: %[[charLen:.*]] = fir.undefined i64
+! CHECK: %[[tuple:.*]] = fir.undefined tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple2:.*]] = fir.insert_value %[[tuple]], %[[boxProc]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxproc<() -> ()>) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple3:.*]] = fir.insert_value %[[tuple2]], %[[charLen]], [1 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, i64) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: fir.call @_QPtakes_char_proc(%[[tuple3]]) {{.*}}: (tuple<!fir.boxproc<() -> ()>, i64>) -> ()
+
+! Test procedure conversion.
+
+subroutine pass_proc_to_char_proc(a)
+  external :: a
+  call takes_char_proc(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPpass_proc_to_char_proc(
+! CHECK-SAME: %[[arg0:.*]]: !fir.boxproc<() -> ()>
+! CHECK: %[[procAddr:.*]] = fir.box_addr %[[arg0]] : (!fir.boxproc<() -> ()>) -> (() -> ())
+! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[procAddr]] : (() -> ()) -> !fir.boxproc<() -> ()>
+! CHECK: %[[charLen:.*]] = fir.undefined i64
+! CHECK: %[[tuple:.*]] = fir.undefined tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple2:.*]] = fir.insert_value %[[tuple]], %[[boxProc]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxproc<() -> ()>) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple3:.*]] = fir.insert_value %[[tuple2]], %[[charLen]], [1 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, i64) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: fir.call @_QPtakes_char_proc(%[[tuple3]]) {{.*}}: (tuple<!fir.boxproc<() -> ()>, i64>) -> ()
+
+subroutine pass_char_proc_to_char_proc(a)
+  character(8), external :: a
+  call takes_char_proc(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPpass_char_proc_to_char_proc(
+! CHECK-SAME: %[[arg0:.*]]: tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[boxProc:.*]] = fir.extract_value %arg0, [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>) -> !fir.boxproc<() -> ()>
+! CHECK: %[[procAddr:.*]] = fir.box_addr %[[boxProc]] : (!fir.boxproc<() -> ()>) -> (() -> ())
+! CHECK: %[[charLen:.*]] = arith.constant 8 : i64
+! CHECK: %[[boxProc2:.*]] = fir.emboxproc %[[procAddr]] : (() -> ()) -> !fir.boxproc<() -> ()>
+! CHECK: %[[tuple:.*]] = fir.undefined tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple2:.*]] = fir.insert_value %[[tuple]], %[[boxProc2]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxproc<() -> ()>) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: %[[tuple3:.*]] = fir.insert_value %[[tuple2]], %[[charLen]], [1 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, i64) -> tuple<!fir.boxproc<() -> ()>, i64>
+! CHECK: fir.call @_QPtakes_char_proc(%[[tuple3]]) {{.*}}: (tuple<!fir.boxproc<() -> ()>, i64>) -> ()
+
+! Test conversion from character procedure to other data types.
+
+! CHECK-LABEL: func.func @_QPtest_conversion_from_char_proc(
+subroutine test_conversion_from_char_proc
+  character(8), external :: char_proc
+
+  ! CHECK: %[[proc:.*]] = fir.address_of(@_QPchar_proc) : (!fir.ref<!fir.char<1,8>>, index) -> !fir.boxchar<1>
+  ! CHECK: %[[convert:.*]] = fir.convert %[[proc]] : ((!fir.ref<!fir.char<1,8>>, index) -> !fir.boxchar<1>) -> !fir.ref<i32>
+  ! CHECK: fir.call @_QPpass_int_to_char_proc(%[[convert]])
+  call pass_int_to_char_proc(char_proc)
+
+  ! CHECK: %[[proc:.*]] = fir.address_of(@_QPchar_proc) : (!fir.ref<!fir.char<1,8>>, index) -> !fir.boxchar<1>
+  ! CHECK: %[[convert:.*]] = fir.convert %[[proc]] : ((!fir.ref<!fir.char<1,8>>, index) -> !fir.boxchar<1>) -> !fir.ref<!fir.logical<4>>
+  ! CHECK: fir.call @_QPpass_logical_to_char_proc(%[[convert]])
+  call pass_logical_to_char_proc(char_proc)
+
+  ! CHECK: %[[proc:.*]] = fir.address_of(@_QPchar_proc) : (!fir.ref<!fir.char<1,8>>, index) -> !fir.boxchar<1>
+  ! CHECK: %[[convert:.*]] = fir.convert %[[proc]] : ((!fir.ref<!fir.char<1,8>>, index) -> !fir.boxchar<1>) -> !fir.ref<f64>
+  ! CHECK: fir.call @_QPpass_real_to_char_proc(%[[convert]])
+  call pass_real_to_char_proc(char_proc)
+
+  ! CHECK: %[[proc:.*]] = fir.address_of(@_QPchar_proc) : (!fir.ref<!fir.char<1,8>>, index) -> !fir.boxchar<1>
+  ! CHECK: %[[convert:.*]] = fir.convert %[[proc]] : ((!fir.ref<!fir.char<1,8>>, index) -> !fir.boxchar<1>) -> !fir.ref<!fir.complex<4>>
+  ! CHECK: fir.call @_QPpass_complex_to_char_proc(%[[convert]])
+  call pass_complex_to_char_proc(char_proc)
+
+  ! CHECK: %[[proc:.*]] = fir.address_of(@_QPchar_proc) : (!fir.ref<!fir.char<1,8>>, index) -> !fir.boxchar<1>
+  ! CHECK: %[[convert:.*]] = fir.convert %[[proc]] : ((!fir.ref<!fir.char<1,8>>, index) -> !fir.boxchar<1>) -> !fir.ref<!fir.char<1,?>>
+  ! CHECK: %[[len:.*]] = fir.undefined index
+  ! CHECK: %[[box:.*]] = fir.emboxchar %[[convert]], %[[len]] : (!fir.ref<!fir.char<1,?>>, index) -> !fir.boxchar<1>
+  ! CHECK: fir.call @_QPpass_char_to_char_proc(%[[box]])
+  call pass_char_to_char_proc(char_proc)
+
+  ! CHECK: %[[proc:.*]] = fir.address_of(@_QPchar_proc) : (!fir.ref<!fir.char<1,8>>, index) -> !fir.boxchar<1>
+  ! CHECK: %[[convert:.*]] = fir.convert %[[proc]] : ((!fir.ref<!fir.char<1,8>>, index) -> !fir.boxchar<1>) -> !fir.ref<!fir.type<_QFpass_dt_to_char_procTdt{i:i32,j:i32}>>
+  ! CHECK: fir.call @_QPpass_dt_to_char_proc(%[[convert]])
+  call pass_dt_to_char_proc(char_proc)
+
+  ! CHECK: %[[proc:.*]] = fir.address_of(@_QPchar_proc) : (!fir.ref<!fir.char<1,8>>, index) -> !fir.boxchar<1>
+  ! CHECK: %[[convert:.*]] = fir.convert %[[proc]] : ((!fir.ref<!fir.char<1,8>>, index) -> !fir.boxchar<1>) -> !fir.ref<!fir.array<10x10xi32>>
+  ! CHECK: fir.call @_QPpass_array_to_char_proc(%[[convert]])
+  call pass_array_to_char_proc(char_proc)
+
+  ! CHECK: %[[proc:.*]] = fir.address_of(@_QPchar_proc) : (!fir.ref<!fir.char<1,8>>, index) -> !fir.boxchar<1>
+  ! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[proc]] : ((!fir.ref<!fir.char<1,8>>, index) -> !fir.boxchar<1>) -> !fir.boxproc<() -> ()>
+  ! CHECK: fir.call @_QPpass_proc_to_char_proc(%[[boxProc]])
+  call pass_proc_to_char_proc(char_proc)
+
+  ! CHECK: %[[proc:.*]] = fir.address_of(@_QPchar_proc) : (!fir.ref<!fir.char<1,8>>, index) -> !fir.boxchar<1>
+  ! CHECK: %[[len:.*]] = arith.constant 8 : i64
+  ! CHECK: %[[boxProc:.*]] = fir.emboxproc %[[proc]] : ((!fir.ref<!fir.char<1,8>>, index) -> !fir.boxchar<1>) -> !fir.boxproc<() -> ()>
+  ! CHECK: %[[tuple:.*]] = fir.undefined tuple<!fir.boxproc<() -> ()>, i64>
+  ! CHECK: %[[tuple2:.*]] = fir.insert_value %[[tuple]], %[[boxProc]], [0 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, !fir.boxproc<() -> ()>) -> tuple<!fir.boxproc<() -> ()>, i64>
+  ! CHECK: %[[tuple3:.*]] = fir.insert_value %[[tuple2]], %[[len]], [1 : index] : (tuple<!fir.boxproc<() -> ()>, i64>, i64) -> tuple<!fir.boxproc<() -> ()>, i64>
+  ! CHECK: fir.call @_QPpass_char_proc_to_char_proc(%[[tuple3]])
+  call pass_char_proc_to_char_proc(char_proc)
+end subroutine
diff --git a/flang/test/Lower/intentout-deallocate.f90 b/flang/test/Lower/intentout-deallocate.f90
index f48b202102102..d27d7e2e7a966 100644
--- a/flang/test/Lower/intentout-deallocate.f90
+++ b/flang/test/Lower/intentout-deallocate.f90
@@ -236,6 +236,27 @@ subroutine sub15(p)
 ! CHECK:   %[[NULL_TYPE_DESC:.*]] = fir.zero_bits !fir.ref<none>  
 ! CHECK:   %[[BOX_NONE:.*]] = fir.convert %[[ARG0]] : (!fir.ref<!fir.class<!fir.heap<none>>>) -> !fir.ref<!fir.box<none>>
 ! CHECK:   %{{.*}} = fir.call @_FortranAAllocatableDeallocatePolymorphic(%[[BOX_NONE]], %[[NULL_TYPE_DESC]], %{{.*}}, %{{.*}}, %{{.*}}) {{.*}} : (!fir.ref<!fir.box<none>>, !fir.ref<none>, i1, !fir.box<none>, !fir.ref<i8>, i32) -> i32
+! CHECK: }
+
+  subroutine sub16(p)
+    class(t), optional, intent(out), allocatable :: p
+  end subroutine
+
+! CHECK-LABEL: func.func @_QMmod1Psub16(
+! CHECK-SAME: %[[ARG0:.*]]: !fir.ref<!fir.class<!fir.heap<!fir.type<_QMmod1Tt{a:i32}>>>> {fir.bindc_name = "p", fir.optional}) {
+! CHECK: %[[IS_PRESENT:.*]] = fir.is_present %[[ARG0]] : (!fir.ref<!fir.class<!fir.heap<!fir.type<_QMmod1Tt{a:i32}>>>>) -> i1
+! CHECK: fir.if %[[IS_PRESENT]] {
+! CHECK:   %[[BOX:.*]] = fir.load %[[ARG0]] : !fir.ref<!fir.class<!fir.heap<!fir.type<_QMmod1Tt{a:i32}>>>>
+! CHECK:   %[[BOX_ADDR:.*]] = fir.box_addr %[[BOX]] : (!fir.class<!fir.heap<!fir.type<_QMmod1Tt{a:i32}>>>) -> !fir.heap<!fir.type<_QMmod1Tt{a:i32}>>
+! CHECK:   %[[BOX_ADDR_PTR:.*]] = fir.convert %[[BOX_ADDR]] : (!fir.heap<!fir.type<_QMmod1Tt{a:i32}>>) -> i64
+! CHECK:   %[[C0:.*]] = arith.constant 0 : i64
+! CHECK:   %[[IS_ALLOCATED:.*]] = arith.cmpi ne, %[[BOX_ADDR_PTR]], %[[C0]] : i64
+! CHECK:   fir.if %[[IS_ALLOCATED]] {
+! CHECK:     %[[TYPE_DESC:.*]] = fir.type_desc !fir.type<_QMmod1Tt{a:i32}>
+! CHECK:     %[[BOX_NONE:.*]] = fir.convert %[[ARG0]] : (!fir.ref<!fir.class<!fir.heap<!fir.type<_QMmod1Tt{a:i32}>>>>) -> !fir.ref<!fir.box<none>>
+! CHECK:     %[[TYPE_NONE:.*]] = fir.convert %[[TYPE_DESC]] : (!fir.tdesc<!fir.type<_QMmod1Tt{a:i32}>>) -> !fir.ref<none> 
+! CHECK:     %{{.*}} = fir.call @_FortranAAllocatableDeallocatePolymorphic(%[[BOX_NONE]], %[[TYPE_NONE]], %{{.*}}, %{{.*}}, %{{.*}}) {{.*}} : (!fir.ref<!fir.box<none>>, !fir.ref<none>, i1, !fir.box<none>, !fir.ref<i8>, i32) -> i32
+! CHECK:   }
 ! CHECK: }
 
 end module
diff --git a/flang/test/Lower/math-lowering.f90 b/flang/test/Lower/math-lowering.f90
index b6b720045f24b..b1ee05ac85436 100644
--- a/flang/test/Lower/math-lowering.f90
+++ b/flang/test/Lower/math-lowering.f90
@@ -57,6 +57,12 @@ function test_complex8(c)
 ! RELAXED: {{%[A-Za-z0-9._]+}} = complex.abs {{%[A-Za-z0-9._]+}} : complex<f64>
 ! PRECISE: {{%[A-Za-z0-9._]+}} = fir.call @cabs({{%[A-Za-z0-9._]+}}) {{.*}}: (!fir.complex<8>) -> f64
 
+! PRECISE-DAG: func.func private @fabsf(f32) -> f32 attributes {fir.bindc_name = "fabsf", fir.runtime}
+! PRECISE-DAG: func.func private @fabs(f64) -> f64 attributes {fir.bindc_name = "fabs", fir.runtime}
+! PRECISE-DAG: func.func private @llvm.fabs.f128(f128) -> f128 attributes {fir.bindc_name = "llvm.fabs.f128", fir.runtime}
+! PRECISE-DAG: func.func private @cabsf(!fir.complex<4>) -> f32 attributes {fir.bindc_name = "cabsf", fir.runtime}
+! PRECISE-DAG: func.func private @cabs(!fir.complex<8>) -> f64 attributes {fir.bindc_name = "cabs", fir.runtime}
+
 //--- aint.f90
 ! RUN: bbc -emit-fir %t/aint.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL %t/aint.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/aint.f90 -o - | FileCheck --check-prefixes=ALL %t/aint.f90
@@ -95,6 +101,10 @@ function test_real10(x)
 !  test_real16 = aint(x)
 !end function
 
+! ALL-DAG: func.func private @llvm.trunc.f32(f32) -> f32 attributes {fir.bindc_name = "llvm.trunc.f32", fir.runtime}
+! ALL-DAG: func.func private @llvm.trunc.f64(f64) -> f64 attributes {fir.bindc_name = "llvm.trunc.f64", fir.runtime}
+! ALL-DAG: func.func private @llvm.trunc.f80(f80) -> f80 attributes {fir.bindc_name = "llvm.trunc.f80", fir.runtime}
+
 //--- anint.f90
 ! RUN: bbc -emit-fir %t/anint.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL,FAST %t/anint.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/anint.f90 -o - | FileCheck --check-prefixes=ALL,FAST %t/anint.f90
@@ -139,6 +149,10 @@ function test_real10(x)
 !  test_real16 = anint(x)
 !end function
 
+! PRECISE-DAG: func.func private @llvm.round.f32(f32) -> f32 attributes {fir.bindc_name = "llvm.round.f32", fir.runtime}
+! PRECISE-DAG: func.func private @llvm.round.f64(f64) -> f64 attributes {fir.bindc_name = "llvm.round.f64", fir.runtime}
+! PRECISE-DAG: func.func private @llvm.round.f80(f80) -> f80 attributes {fir.bindc_name = "llvm.round.f80", fir.runtime}
+
 //--- atan.f90
 ! RUN: bbc -emit-fir %t/atan.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL,FAST %t/atan.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/atan.f90 -o - | FileCheck --check-prefixes=ALL,FAST %t/atan.f90
@@ -167,6 +181,9 @@ function test_real8(x)
 ! RELAXED: {{%[A-Za-z0-9._]+}} = math.atan {{%[A-Za-z0-9._]+}} {{.*}}: f64
 ! PRECISE: {{%[A-Za-z0-9._]+}} = fir.call @atan({{%[A-Za-z0-9._]+}}) {{.*}}: (f64) -> f64
 
+! PRECISE-DAG: func.func private @atanf(f32) -> f32 attributes {fir.bindc_name = "atanf", fir.runtime}
+! PRECISE-DAG: func.func private @atan(f64) -> f64 attributes {fir.bindc_name = "atan", fir.runtime}
+
 //--- atan2.f90
 ! RUN: bbc -emit-fir %t/atan2.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL,FAST %t/atan2.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/atan2.f90 -o - | FileCheck --check-prefixes=ALL,FAST %t/atan2.f90
@@ -195,6 +212,9 @@ function test_real8(x, y)
 ! RELAXED: {{%[A-Za-z0-9._]+}} = math.atan2 {{%[A-Za-z0-9._]+}}, {{%[A-Za-z0-9._]+}} {{.*}}: f64
 ! PRECISE: {{%[A-Za-z0-9._]+}} = fir.call @atan2({{%[A-Za-z0-9._]+}}, {{%[A-Za-z0-9._]+}}) {{.*}}: (f64, f64) -> f64
 
+! PRECISE-DAG: func.func private @atan2f(f32, f32) -> f32 attributes {fir.bindc_name = "atan2f", fir.runtime}
+! PRECISE-DAG: func.func private @atan2(f64, f64) -> f64 attributes {fir.bindc_name = "atan2", fir.runtime}
+
 //--- ceiling.f90
 ! RUN: bbc -emit-fir %t/ceiling.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL,FAST %t/ceiling.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/ceiling.f90 -o - | FileCheck --check-prefixes=ALL,FAST %t/ceiling.f90
@@ -223,6 +243,9 @@ function test_real8(x)
 ! RELAXED: {{%[A-Za-z0-9._]+}} = math.ceil {{%[A-Za-z0-9._]+}} {{.*}}: f64
 ! PRECISE: {{%[A-Za-z0-9._]+}} = fir.call @ceil({{%[A-Za-z0-9._]+}}) {{.*}}: (f64) -> f64
 
+! PRECISE-DAG: func.func private @ceilf(f32) -> f32 attributes {fir.bindc_name = "ceilf", fir.runtime}
+! PRECISE-DAG: func.func private @ceil(f64) -> f64 attributes {fir.bindc_name = "ceil", fir.runtime}
+
 //--- cos.f90
 ! RUN: bbc -emit-fir %t/cos.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL,FAST %t/cos.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/cos.f90 -o - | FileCheck --check-prefixes=ALL,FAST %t/cos.f90
@@ -251,6 +274,9 @@ function test_real8(x)
 ! RELAXED: {{%[A-Za-z0-9._]+}} = math.cos {{%[A-Za-z0-9._]+}} {{.*}}: f64
 ! PRECISE: {{%[A-Za-z0-9._]+}} = fir.call @cos({{%[A-Za-z0-9._]+}}) {{.*}}: (f64) -> f64
 
+! PRECISE-DAG: func.func private @cosf(f32) -> f32 attributes {fir.bindc_name = "cosf", fir.runtime}
+! PRECISE-DAG: func.func private @cos(f64) -> f64 attributes {fir.bindc_name = "cos", fir.runtime}
+
 //--- cosh.f90
 ! RUN: bbc -emit-fir %t/cosh.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL %t/cosh.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/cosh.f90 -o - | FileCheck --check-prefixes=ALL %t/cosh.f90
@@ -275,6 +301,9 @@ function test_real8(x)
 ! ALL-LABEL: @_QPtest_real8
 ! ALL: {{%[A-Za-z0-9._]+}} = fir.call @cosh({{%[A-Za-z0-9._]+}}) {{.*}}: (f64) -> f64
 
+! ALL-DAG: func.func private @coshf(f32) -> f32 attributes {fir.bindc_name = "coshf", fir.runtime}
+! ALL-DAG: func.func private @cosh(f64) -> f64 attributes {fir.bindc_name = "cosh", fir.runtime}
+
 //--- erf.f90
 ! RUN: bbc -emit-fir %t/erf.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL,FAST %t/erf.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/erf.f90 -o - | FileCheck --check-prefixes=ALL,FAST %t/erf.f90
@@ -303,6 +332,9 @@ function test_real8(x)
 ! RELAXED: {{%[A-Za-z0-9._]+}} = math.erf {{%[A-Za-z0-9._]+}} {{.*}}: f64
 ! PRECISE: {{%[A-Za-z0-9._]+}} = fir.call @erf({{%[A-Za-z0-9._]+}}) {{.*}}: (f64) -> f64
 
+! PRECISE-DAG: func.func private @erff(f32) -> f32 attributes {fir.bindc_name = "erff", fir.runtime}
+! PRECISE-DAG: func.func private @erf(f64) -> f64 attributes {fir.bindc_name = "erf", fir.runtime}
+
 //--- exp.f90
 ! RUN: bbc -emit-fir %t/exp.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL,FAST %t/exp.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/exp.f90 -o - | FileCheck --check-prefixes=ALL,FAST %t/exp.f90
@@ -331,6 +363,9 @@ function test_real8(x)
 ! RELAXED: {{%[A-Za-z0-9._]+}} = math.exp {{%[A-Za-z0-9._]+}} {{.*}}: f64
 ! PRECISE: {{%[A-Za-z0-9._]+}} = fir.call @exp({{%[A-Za-z0-9._]+}}) {{.*}}: (f64) -> f64
 
+! PRECISE-DAG: func.func private @expf(f32) -> f32 attributes {fir.bindc_name = "expf", fir.runtime}
+! PRECISE-DAG: func.func private @exp(f64) -> f64 attributes {fir.bindc_name = "exp", fir.runtime}
+
 //--- floor.f90
 ! RUN: bbc -emit-fir %t/floor.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL,FAST %t/floor.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/floor.f90 -o - | FileCheck --check-prefixes=ALL,FAST %t/floor.f90
@@ -359,6 +394,9 @@ function test_real8(x)
 ! RELAXED: {{%[A-Za-z0-9._]+}} = math.floor {{%[A-Za-z0-9._]+}} {{.*}}: f64
 ! PRECISE: {{%[A-Za-z0-9._]+}} = fir.call @floor({{%[A-Za-z0-9._]+}}) {{.*}}: (f64) -> f64
 
+! PRECISE-DAG: func.func private @floorf(f32) -> f32 attributes {fir.bindc_name = "floorf", fir.runtime}
+! PRECISE-DAG: func.func private @floor(f64) -> f64 attributes {fir.bindc_name = "floor", fir.runtime}
+
 //--- log.f90
 ! RUN: bbc -emit-fir %t/log.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL,FAST %t/log.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/log.f90 -o - | FileCheck --check-prefixes=ALL,FAST %t/log.f90
@@ -387,6 +425,9 @@ function test_real8(x)
 ! RELAXED: {{%[A-Za-z0-9._]+}} = math.log {{%[A-Za-z0-9._]+}} {{.*}}: f64
 ! PRECISE: {{%[A-Za-z0-9._]+}} = fir.call @log({{%[A-Za-z0-9._]+}}) {{.*}}: (f64) -> f64
 
+! PRECISE-DAG: func.func private @logf(f32) -> f32 attributes {fir.bindc_name = "logf", fir.runtime}
+! PRECISE-DAG: func.func private @log(f64) -> f64 attributes {fir.bindc_name = "log", fir.runtime}
+
 //--- log10.f90
 ! RUN: bbc -emit-fir %t/log10.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL,FAST %t/log10.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/log10.f90 -o - | FileCheck --check-prefixes=ALL,FAST %t/log10.f90
@@ -415,6 +456,9 @@ function test_real8(x)
 ! RELAXED: {{%[A-Za-z0-9._]+}} = math.log10 {{%[A-Za-z0-9._]+}} {{.*}}: f64
 ! PRECISE: {{%[A-Za-z0-9._]+}} = fir.call @log10({{%[A-Za-z0-9._]+}}) {{.*}}: (f64) -> f64
 
+! PRECISE-DAG: func.func private @log10f(f32) -> f32 attributes {fir.bindc_name = "log10f", fir.runtime}
+! PRECISE-DAG: func.func private @log10(f64) -> f64 attributes {fir.bindc_name = "log10", fir.runtime}
+
 //--- nint.f90
 ! RUN: bbc -emit-fir %t/nint.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL %t/nint.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/nint.f90 -o - | FileCheck --check-prefixes=ALL %t/nint.f90
@@ -441,6 +485,11 @@ function test_real8(x)
 ! ALL: {{%[A-Za-z0-9._]+}} = fir.call @llvm.lround.i32.f64({{%[A-Za-z0-9._]+}}) {{.*}}: (f64) -> i32
 ! ALL: {{%[A-Za-z0-9._]+}} = fir.call @llvm.lround.i64.f64({{%[A-Za-z0-9._]+}}) {{.*}}: (f64) -> i64
 
+! ALL-DAG: func.func private @llvm.lround.i32.f32(f32) -> i32 attributes {fir.bindc_name = "llvm.lround.i32.f32", fir.runtime}
+! ALL-DAG: func.func private @llvm.lround.i64.f32(f32) -> i64 attributes {fir.bindc_name = "llvm.lround.i64.f32", fir.runtime}
+! ALL-DAG: func.func private @llvm.lround.i32.f64(f64) -> i32 attributes {fir.bindc_name = "llvm.lround.i32.f64", fir.runtime}
+! ALL-DAG: func.func private @llvm.lround.i64.f64(f64) -> i64 attributes {fir.bindc_name = "llvm.lround.i64.f64", fir.runtime}
+
 //--- exponentiation.f90
 ! RUN: bbc -emit-fir %t/exponentiation.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL,FAST %t/exponentiation.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/exponentiation.f90 -o - | FileCheck --check-prefixes=ALL,FAST %t/exponentiation.f90
@@ -495,6 +544,13 @@ function test_real8(x, y, s, i, k)
 ! RELAXED: {{%[A-Za-z0-9._]+}} = math.fpowi {{%[A-Za-z0-9._]+}}, {{%[A-Za-z0-9._]+}} {{.*}}: f64, i64
 ! PRECISE: {{%[A-Za-z0-9._]+}} = fir.call @_FortranAFPow8k({{%[A-Za-z0-9._]+}}, {{%[A-Za-z0-9._]+}}) {{.*}}: (f64, i64) -> f64
 
+! PRECISE-DAG: func.func private @_FortranAFPow4i(f32, i32) -> f32 attributes {fir.bindc_name = "_FortranAFPow4i", fir.runtime}
+! PRECISE-DAG: func.func private @powf(f32, f32) -> f32 attributes {fir.bindc_name = "powf", fir.runtime}
+! PRECISE-DAG: func.func private @_FortranAFPow4k(f32, i64) -> f32 attributes {fir.bindc_name = "_FortranAFPow4k", fir.runtime}
+! PRECISE-DAG: func.func private @_FortranAFPow8i(f64, i32) -> f64 attributes {fir.bindc_name = "_FortranAFPow8i", fir.runtime}
+! PRECISE-DAG: func.func private @pow(f64, f64) -> f64 attributes {fir.bindc_name = "pow", fir.runtime}
+! PRECISE-DAG: func.func private @_FortranAFPow8k(f64, i64) -> f64 attributes {fir.bindc_name = "_FortranAFPow8k", fir.runtime}
+
 //--- sign.f90
 ! RUN: bbc -emit-fir %t/sign.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL,FAST %t/sign.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/sign.f90 -o - | FileCheck --check-prefixes=ALL,FAST %t/sign.f90
@@ -543,6 +599,11 @@ function test_real16(x, y)
 ! RELAXED: {{%[A-Za-z0-9._]+}} = math.copysign {{%[A-Za-z0-9._]+}}, {{%[A-Za-z0-9._]+}} {{.*}}: f128
 ! PRECISE: {{%[A-Za-z0-9._]+}} = fir.call @llvm.copysign.f128({{%[A-Za-z0-9._]+}}, {{%[A-Za-z0-9._]+}}) {{.*}}: (f128, f128) -> f128
 
+! PRECISE-DAG: func.func private @copysignf(f32, f32) -> f32 attributes {fir.bindc_name = "copysignf", fir.runtime}
+! PRECISE-DAG: func.func private @copysign(f64, f64) -> f64 attributes {fir.bindc_name = "copysign", fir.runtime}
+! PRECISE-DAG: func.func private @copysignl(f80, f80) -> f80 attributes {fir.bindc_name = "copysignl", fir.runtime}
+! PRECISE-DAG: func.func private @llvm.copysign.f128(f128, f128) -> f128 attributes {fir.bindc_name = "llvm.copysign.f128", fir.runtime}
+
 //--- sin.f90
 ! RUN: bbc -emit-fir %t/sin.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL,FAST %t/sin.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/sin.f90 -o - | FileCheck --check-prefixes=ALL,FAST %t/sin.f90
@@ -571,6 +632,9 @@ function test_real8(x)
 ! RELAXED: {{%[A-Za-z0-9._]+}} = math.sin {{%[A-Za-z0-9._]+}} {{.*}}: f64
 ! PRECISE: {{%[A-Za-z0-9._]+}} = fir.call @sin({{%[A-Za-z0-9._]+}}) {{.*}}: (f64) -> f64
 
+! PRECISE-DAG: func.func private @sinf(f32) -> f32 attributes {fir.bindc_name = "sinf", fir.runtime}
+! PRECISE-DAG: func.func private @sin(f64) -> f64 attributes {fir.bindc_name = "sin", fir.runtime}
+
 //--- sinh.f90
 ! RUN: bbc -emit-fir %t/sinh.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL %t/sinh.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/sinh.f90 -o - | FileCheck --check-prefixes=ALL %t/sinh.f90
@@ -595,6 +659,9 @@ function test_real8(x)
 ! ALL-LABEL: @_QPtest_real8
 ! ALL: {{%[A-Za-z0-9._]+}} = fir.call @sinh({{%[A-Za-z0-9._]+}}) {{.*}}: (f64) -> f64
 
+! ALL-DAG: func.func private @sinhf(f32) -> f32 attributes {fir.bindc_name = "sinhf", fir.runtime}
+! ALL-DAG: func.func private @sinh(f64) -> f64 attributes {fir.bindc_name = "sinh", fir.runtime}
+
 //--- tanh.f90
 ! RUN: bbc -emit-fir %t/tanh.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL,FAST %t/tanh.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/tanh.f90 -o - | FileCheck --check-prefixes=ALL,FAST %t/tanh.f90
@@ -623,6 +690,9 @@ function test_real8(x)
 ! RELAXED: {{%[A-Za-z0-9._]+}} = math.tanh {{%[A-Za-z0-9._]+}} {{.*}}: f64
 ! PRECISE: {{%[A-Za-z0-9._]+}} = fir.call @tanh({{%[A-Za-z0-9._]+}}) {{.*}}: (f64) -> f64
 
+! PRECISE-DAG: func.func private @tanhf(f32) -> f32 attributes {fir.bindc_name = "tanhf", fir.runtime}
+! PRECISE-DAG: func.func private @tanh(f64) -> f64 attributes {fir.bindc_name = "tanh", fir.runtime}
+
 //--- tan.f90
 ! RUN: bbc -emit-fir %t/tan.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL,FAST %t/tan.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/tan.f90 -o - | FileCheck --check-prefixes=ALL,FAST %t/tan.f90
@@ -650,3 +720,6 @@ function test_real8(x)
 ! FAST: {{%[A-Za-z0-9._]+}} = math.tan {{%[A-Za-z0-9._]+}} {{.*}}: f64
 ! RELAXED: {{%[A-Za-z0-9._]+}} = math.tan {{%[A-Za-z0-9._]+}} {{.*}}: f64
 ! PRECISE: {{%[A-Za-z0-9._]+}} = fir.call @tan({{%[A-Za-z0-9._]+}}) {{.*}}: (f64) -> f64
+
+! PRECISE-DAG: func.func private @tanf(f32) -> f32 attributes {fir.bindc_name = "tanf", fir.runtime}
+! PRECISE-DAG: func.func private @tan(f64) -> f64 attributes {fir.bindc_name = "tan", fir.runtime}
diff --git a/flang/test/Lower/math-name-conflict.f90 b/flang/test/Lower/math-name-conflict.f90
new file mode 100644
index 0000000000000..0e20f63cf132e
--- /dev/null
+++ b/flang/test/Lower/math-name-conflict.f90
@@ -0,0 +1,50 @@
+! REQUIRES: asserts
+! RUN: bbc -emit-fir %s --math-runtime=precise -o - | FileCheck -check-prefix=CHECK %s
+! RUN: bbc -emit-fir %s --math-runtime=precise -debug-only=flang-lower-intrinsic,flang-lower-expr 2>&1 | FileCheck -check-prefix=CHECK-WARN %s
+
+! CHECK-LABEL: func.func @_QPtest
+! CHECK: fir.call @atanh({{[^,]*}}){{.*}}: (i32) -> i32
+! CHECK-LABEL: func.func @_QPtest2
+! CHECK: %[[ADDR:.*]] = fir.address_of(@atanh) : (i32) -> i32
+! CHECK: %[[CAST:.*]] = fir.convert %[[ADDR]] : ((i32) -> i32) -> ((f64) -> f64)
+! CHECK: fir.call %[[CAST]]({{[^,]*}}){{.*}}: (f64) -> f64
+
+subroutine test(x)
+  interface
+     integer function atanh(x) bind(c)
+       integer,Value :: x
+     end function atanh
+  end interface
+  integer :: x
+  print *,atanh(x)
+end subroutine test
+subroutine test2(x)
+  real(8) :: x
+  print *,atanh(x)
+end subroutine test2
+
+! CHECK-LABEL: func.func @_QPtest3
+! CHECK: fir.call @asinh({{[^,]*}}){{.*}}: (f64) -> f64
+! CHECK-LABEL: func.func @_QPtest4
+! CHECK: %[[ADDR:.*]] = fir.address_of(@asinh) : (f64) -> f64
+! CHECK: %[[CAST:.*]] = fir.convert %[[ADDR]] : ((f64) -> f64) -> ((i32) -> i32)
+! CHECK: fir.call %[[CAST]]({{[^,]*}}){{.*}}: (i32) -> i32
+subroutine test3(x)
+  real(8) :: x
+  print *,asinh(x)
+end subroutine test3
+subroutine test4(x)
+  interface
+     integer function asinh(x) bind(c)
+       integer,Value :: x
+     end function asinh
+  end interface
+  integer :: x
+  print *,asinh(x)
+end subroutine test4
+
+! CHECK-WARN:      warning: loc({{.*}}math-name-conflict.f90{{.*}}): function
+! CHECK-WARN-SAME: signature mismatch for 'atanh' may lead to undefined behavior.
+! CHECK-WARN:      warning: loc({{.*}}math-name-conflict.f90{{.*}}): function
+! CHECK-WARN-SAME: name 'asinh' conflicts with a runtime function
+! CHECK-WARN-SAME: name used by Flang - this may lead to undefined behavior
diff --git a/flang/test/Lower/parent-component.f90 b/flang/test/Lower/parent-component.f90
index 071ed53ae7876..0755b5409a2fd 100644
--- a/flang/test/Lower/parent-component.f90
+++ b/flang/test/Lower/parent-component.f90
@@ -51,8 +51,7 @@ subroutine init_with_slice()
   ! CHECK: %[[ADD:.*]] = arith.addi %[[C1]], %[[C2]] : index
   ! CHECK: %[[UB:.*]] = arith.subi %[[ADD]], %[[C1]] : index
   ! CHECK: %[[SHAPE:.*]] = fir.shape %[[C2]] : (index) -> !fir.shape<1>
-  ! CHECK: %[[FIELD:.*]] = fir.field_index a, !fir.type<_QFTc{a:i32,b:i32}>
-  ! CHECK: %[[SLICE:.*]] = fir.slice %[[C1]], %[[UB]], %[[STRIDE]] path %[[FIELD]] : (index, index, index, !fir.field) -> !fir.slice<1>
+  ! CHECK: %[[SLICE:.*]] = fir.slice %[[C1]], %[[UB]], %[[STRIDE]] : (index, index, index) -> !fir.slice<1>
   ! CHECK: %[[BOX:.*]] = fir.embox %[[Y]](%[[SHAPE]]) [%[[SLICE]]] : (!fir.ref<!fir.array<2x!fir.type<_QFTc{a:i32,b:i32}>>>, !fir.shape<1>, !fir.slice<1>) -> !fir.box<!fir.array<2x!fir.type<_QFTp{a:i32}>>>
   ! CHECK: %[[BOX_NONE:.*]] = fir.convert %[[BOX]] : (!fir.box<!fir.array<2x!fir.type<_QFTp{a:i32}>>>) -> !fir.box<none>
   ! CHECK: %[[IS_CONTIGOUS:.*]] = fir.call @_FortranAIsContiguous(%[[BOX_NONE]]) {{.*}}: (!fir.box<none>) -> i1
@@ -69,8 +68,7 @@ subroutine init_with_slice()
   ! CHECK: %[[ADD:.*]] = arith.addi %[[C1]], %[[C2]] : index
   ! CHECK: %[[UB:.*]] = arith.subi %[[ADD]], %[[C1]] : index
   ! CHECK: %[[SHAPE:.*]] = fir.shape %[[C2]] : (index) -> !fir.shape<1>
-  ! CHECK: %[[FIELD:.*]] = fir.field_index a, !fir.type<_QFTc{a:i32,b:i32}>
-  ! CHECK: %[[SLICE:.*]] = fir.slice %[[C1]], %[[UB]], %[[STRIDE]] path %[[FIELD]] : (index, index, index, !fir.field) -> !fir.slice<1>
+  ! CHECK: %[[SLICE:.*]] = fir.slice %{{.*}}, %{{.*}}, %{{.*}} : (index, index, index) -> !fir.slice<1>
   ! CHECK: %[[BOX:.*]] = fir.embox %[[Y]](%[[SHAPE]]) [%[[SLICE]]] : (!fir.ref<!fir.array<2x!fir.type<_QFTc{a:i32,b:i32}>>>, !fir.shape<1>, !fir.slice<1>) -> !fir.box<!fir.array<2x!fir.type<_QFTp{a:i32}>>>
   ! CHECK: %[[BOX_NONE:.*]] = fir.convert %[[BOX]] : (!fir.box<!fir.array<2x!fir.type<_QFTp{a:i32}>>>) -> !fir.box<none>
   ! CHECK: %{{.*}} = fir.call @_FortranAioOutputDescriptor(%{{.*}}, %[[BOX_NONE]]) {{.*}}: (!fir.ref<i8>, !fir.box<none>) -> i1
@@ -84,12 +82,7 @@ subroutine init_no_slice()
   ! CHECK: %[[Y:.*]] = fir.address_of(@_QFFinit_no_sliceEy) : !fir.ref<!fir.array<2x!fir.type<_QFTc{a:i32,b:i32}>>>
   ! CHECK: %[[C2:.*]] = arith.constant 2 : index
   ! CHECK: %[[SHAPE:.*]] = fir.shape %[[C2]] : (index) -> !fir.shape<1>
-  ! CHECK: %[[FIELD:.*]] = fir.field_index a, !fir.type<_QFTc{a:i32,b:i32}>
-  ! CHECK: %[[C0:.*]] = arith.constant 0 : index
-  ! CHECK: %[[BOX_DIM:.*]]:3 = fir.box_dims %{{.*}}, %[[C0]] : (!fir.box<!fir.array<2x!fir.type<_QFTc{a:i32,b:i32}>>>, index) -> (index, index, index)
-  ! CHECK: %[[C1:.*]] = arith.constant 1 : index
-  ! CHECK: %[[SLICE:.*]] = fir.slice %[[C1]], %[[BOX_DIM]]#1, %[[C1]] path %[[FIELD]] : (index, index, index, !fir.field) -> !fir.slice<1>
-  ! CHECK: %[[BOX:.*]] = fir.embox %[[Y]](%[[SHAPE]]) [%[[SLICE]]] : (!fir.ref<!fir.array<2x!fir.type<_QFTc{a:i32,b:i32}>>>, !fir.shape<1>, !fir.slice<1>) -> !fir.box<!fir.array<2x!fir.type<_QFTp{a:i32}>>>
+  ! CHECK: %[[BOX:.*]] = fir.embox %[[Y]](%[[SHAPE]]) : (!fir.ref<!fir.array<2x!fir.type<_QFTc{a:i32,b:i32}>>>, !fir.shape<1>) -> !fir.box<!fir.array<2x!fir.type<_QFTp{a:i32}>>>
   ! CHECK: %[[BOX_NONE:.*]] = fir.convert %[[BOX]] : (!fir.box<!fir.array<2x!fir.type<_QFTp{a:i32}>>>) -> !fir.box<none>
   ! CHECK: %[[IS_CONTIGOUS:.*]] = fir.call @_FortranAIsContiguous(%[[BOX_NONE]]) {{.*}}: (!fir.box<none>) -> i1
   ! CHECK: %[[TEMP:.*]] = fir.if %[[IS_CONTIGOUS]] -> (!fir.heap<!fir.array<2x!fir.type<_QFTp{a:i32}>>>) {
@@ -100,12 +93,7 @@ subroutine init_no_slice()
 
   ! CHECK-LABEL: %{{.*}} = fir.call @_FortranAioBeginExternalListOutput(%{{.*}}, %{{.*}}, %{{.*}}) {{.*}}: (i32, !fir.ref<i8>, i32) -> !fir.ref<i8>
   ! CHECK: %[[SHAPE:.*]] = fir.shape %[[C2]] : (index) -> !fir.shape<1>
-  ! CHECK: %[[FIELD:.*]] = fir.field_index a, !fir.type<_QFTc{a:i32,b:i32}>
-  ! CHECK: %[[C0:.*]] = arith.constant 0 : index
-  ! CHECK: %[[BOX_DIMS:.*]]:3 = fir.box_dims %{{.*}}, %[[C0]] : (!fir.box<!fir.array<2x!fir.type<_QFTc{a:i32,b:i32}>>>, index) -> (index, index, index)
-  ! CHECK: %[[C1:.*]] = arith.constant 1 : index
-  ! CHECK: %[[SLICE:.*]] = fir.slice %[[C1]], %[[BOX_DIMS]]#1, %[[C1]] path %[[FIELD]] : (index, index, index, !fir.field) -> !fir.slice<1>
-  ! CHECK: %[[BOX:.*]] = fir.embox %[[Y]](%[[SHAPE]]) [%[[SLICE]]] : (!fir.ref<!fir.array<2x!fir.type<_QFTc{a:i32,b:i32}>>>, !fir.shape<1>, !fir.slice<1>) -> !fir.box<!fir.array<2x!fir.type<_QFTp{a:i32}>>>
+  ! CHECK: %[[BOX:.*]] = fir.embox %[[Y]](%[[SHAPE]]) : (!fir.ref<!fir.array<2x!fir.type<_QFTc{a:i32,b:i32}>>>, !fir.shape<1>) -> !fir.box<!fir.array<2x!fir.type<_QFTp{a:i32}>>>
   ! CHECK: %[[BOX_NONE:.*]] = fir.convert %[[BOX]] : (!fir.box<!fir.array<2x!fir.type<_QFTp{a:i32}>>>) -> !fir.box<none>
   ! CHECK: %{{.*}} = fir.call @_FortranAioOutputDescriptor(%{{.*}}, %[[BOX_NONE]]) {{.*}}: (!fir.ref<i8>, !fir.box<none>) -> i1
 
@@ -127,14 +115,7 @@ subroutine init_allocatable()
   ! CHECK: %[[LOAD_EXT0:.*]] = fir.load %[[EXT0]] : !fir.ref<index>
   ! CHECK: %[[MEM:.*]] = fir.load %[[ALLOC]] : !fir.ref<!fir.heap<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>>>
   ! CHECK: %[[SHAPE_SHIFT:.*]] = fir.shape_shift %[[LOAD_LB0]], %[[LOAD_EXT0]] : (index, index) -> !fir.shapeshift<1>
-  ! CHECK: %[[FIELD:.*]] = fir.field_index a, !fir.type<_QFTc{a:i32,b:i32}>
-  ! CHECK: %[[C0:.*]] = arith.constant 0 : index
-  ! CHECK: %[[BOX_DIMS:.*]]:3 = fir.box_dims %{{.*}}, %[[C0]] : (!fir.box<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>>, index) -> (index, index, index)
-  ! CHECK: %[[C1:.*]] = arith.constant 1 : index
-  ! CHECK: %[[BOUND_OFFSET:.*]] = arith.subi %[[LOAD_LB0]], %[[C1]] : index
-  ! CHECK: %[[UB:.*]] = arith.addi %[[BOX_DIMS]]#1, %[[BOUND_OFFSET]] : index
-  ! CHECK: %[[SLICE:.*]] = fir.slice %[[LOAD_LB0]], %[[UB]], %[[C1]] path %[[FIELD]] : (index, index, index, !fir.field) -> !fir.slice<1>
-  ! CHECK: %[[BOX:.*]] = fir.embox %[[MEM]](%[[SHAPE_SHIFT]]) [%[[SLICE]]] : (!fir.heap<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>>, !fir.shapeshift<1>, !fir.slice<1>) -> !fir.box<!fir.array<?x!fir.type<_QFTp{a:i32}>>>
+  ! CHECK: %[[BOX:.*]] = fir.embox %[[MEM]](%[[SHAPE_SHIFT]]) : (!fir.heap<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>>, !fir.shapeshift<1>) -> !fir.box<!fir.array<?x!fir.type<_QFTp{a:i32}>>>
   ! CHECK: %[[BOX_NONE:.*]] = fir.convert %[[BOX]] : (!fir.box<!fir.array<?x!fir.type<_QFTp{a:i32}>>>) -> !fir.box<none> 
   ! CHECK: %[[IS_CONTIGOUS:.*]] = fir.call @_FortranAIsContiguous(%[[BOX_NONE]]) {{.*}}: (!fir.box<none>) -> i1
   ! CHECK: %[[TEMP:.*]] = fir.if %[[IS_CONTIGOUS]] -> (!fir.heap<!fir.array<?x!fir.type<_QFTp{a:i32}>>>) {
@@ -148,14 +129,7 @@ subroutine init_allocatable()
   ! CHECK: %[[LOAD_EXT0:.*]] = fir.load %[[EXT0]] : !fir.ref<index>
   ! CHECK: %[[LOAD_ALLOC:.*]] = fir.load %[[ALLOC]] : !fir.ref<!fir.heap<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>>>
   ! CHECK: %[[SHAPE_SHIFT:.*]] = fir.shape_shift %[[LOAD_LB0]], %[[LOAD_EXT0]] : (index, index) -> !fir.shapeshift<1>
-  ! CHECK: %[[FIELD:.*]] = fir.field_index a, !fir.type<_QFTc{a:i32,b:i32}>
-  ! CHECK: %[[C0:.*]] = arith.constant 0 : index
-  ! CHECK: %[[BOX_DIMS:.*]]:3 = fir.box_dims %{{.*}}, %[[C0]] : (!fir.box<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>>, index) -> (index, index, index)
-  ! CHECK: %[[C1:.*]] = arith.constant 1 : index
-  ! CHECK: %[[BOUND_OFFSET:.*]] = arith.subi %[[LOAD_LB0]], %[[C1]] : index
-  ! CHECK: %[[UB:.*]] = arith.addi %[[BOX_DIMS]]#1, %[[BOUND_OFFSET]] : index
-  ! CHECK: %[[SLICE:.*]] = fir.slice %[[LOAD_LB0]], %[[UB]], %[[C1]] path %[[FIELD]] : (index, index, index, !fir.field) -> !fir.slice<1>
-  ! CHECK: %[[BOX:.*]] = fir.embox %[[LOAD_ALLOC]](%[[SHAPE_SHIFT]]) [%[[SLICE]]] : (!fir.heap<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>>, !fir.shapeshift<1>, !fir.slice<1>) -> !fir.box<!fir.array<?x!fir.type<_QFTp{a:i32}>>>
+  ! CHECK: %[[BOX:.*]] = fir.embox %[[LOAD_ALLOC]](%[[SHAPE_SHIFT]]) : (!fir.heap<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>>, !fir.shapeshift<1>) -> !fir.box<!fir.array<?x!fir.type<_QFTp{a:i32}>>>
   ! CHECK: %[[BOX_NONE:.*]] = fir.convert %[[BOX]] : (!fir.box<!fir.array<?x!fir.type<_QFTp{a:i32}>>>) -> !fir.box<none>
   ! CHECK: %{{.*}} = fir.call @_FortranAioOutputDescriptor(%{{.*}}, %[[BOX_NONE]]) {{.*}}: (!fir.ref<i8>, !fir.box<none>) -> i1
 
@@ -182,21 +156,9 @@ subroutine init_assumed(y)
 
   ! CHECK-LABEL: func.func @_QFPinit_assumed(
   ! CHECK-SAME: %[[ARG0:.*]]: !fir.box<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>
-  ! CHECK: %[[BOX:.*]] = fir.rebox %[[ARG0]] : (!fir.box<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>>) -> !fir.box<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>>
-  ! CHECK: %[[FIELD:.*]] = fir.field_index a, !fir.type<_QFTc{a:i32,b:i32}>
-  ! CHECK: %[[C0:.*]] = arith.constant 0 : index
-  ! CHECK: %[[BOX_DIMS:.*]]:3 = fir.box_dims %[[BOX]], %[[C0]] : (!fir.box<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>>, index) -> (index, index, index)
-  ! CHECK: %[[C1:.*]] = arith.constant 1 : index
-  ! CHECK: %[[SLICE:.*]] = fir.slice %[[C1]], %[[BOX_DIMS]]#1, %[[C1]] path %[[FIELD]] : (index, index, index, !fir.field) -> !fir.slice<1>
-  ! CHECK: %{{.*}} = fir.rebox %[[ARG0]] [%[[SLICE]]] : (!fir.box<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>>, !fir.slice<1>) -> !fir.box<!fir.array<?x!fir.type<_QFTp{a:i32}>>>
+  ! CHECK: %[[BOX:.*]] = fir.rebox %[[ARG0]] : (!fir.box<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>>) -> !fir.box<!fir.array<?x!fir.type<_QFTp{a:i32}>>>
 
-  ! CHECK: %[[BOX:.*]] = fir.rebox %[[ARG0]] : (!fir.box<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>>) -> !fir.box<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>>
-  ! CHECK: %[[FIELD:.*]] = fir.field_index a, !fir.type<_QFTc{a:i32,b:i32}>
-  ! CHECK: %[[C0:.*]] = arith.constant 0 : index
-  ! CHECK: %[[BOX_DIMS:.*]]:3 = fir.box_dims %[[BOX]], %[[C0]] : (!fir.box<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>>, index) -> (index, index, index)
-  ! CHECK: %[[C1:.*]] = arith.constant 1 : index
-  ! CHECK: %[[SLICE:.*]] = fir.slice %[[C1]], %[[BOX_DIMS]]#1, %[[C1]] path %[[FIELD]] : (index, index, index, !fir.field) -> !fir.slice<1>
-  ! CHECK: %[[REBOX:.*]] = fir.rebox %arg0 [%[[SLICE]]] : (!fir.box<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>>, !fir.slice<1>) -> !fir.box<!fir.array<?x!fir.type<_QFTp{a:i32}>>>
+  ! CHECK: %[[REBOX:.*]] = fir.rebox %[[ARG0]]  : (!fir.box<!fir.array<?x!fir.type<_QFTc{a:i32,b:i32}>>>) -> !fir.box<!fir.array<?x!fir.type<_QFTp{a:i32}>>>
   ! CHECK: %[[REBOX_CAST:.*]] = fir.convert %[[REBOX]] : (!fir.box<!fir.array<?x!fir.type<_QFTp{a:i32}>>>) -> !fir.box<none>
   ! CHECK: %{{.*}} = fir.call @_FortranAioOutputDescriptor(%{{.*}}, %[[REBOX_CAST]]) {{.*}}: (!fir.ref<i8>, !fir.box<none>) -> i1
 
@@ -211,7 +173,25 @@ subroutine init_existing_field()
   ! CHECK: %[[FIELD_C:.*]] = fir.field_index c, !fir.type<_QFTz{k:i32,c:!fir.type<_QFTc{a:i32,b:i32}>}>
   ! CHECK: %[[SHAPE:.*]] = fir.shape %[[C2]] : (index) -> !fir.shape<1>
   ! CHECK: %[[C1:.*]] = arith.constant 1 : index
-  ! CHECK: %[[FIELD_A:.*]] = fir.field_index a, !fir.type<_QFTc{a:i32,b:i32}>
-  ! CHECK: %[[SLICE:.*]] = fir.slice %[[C1]], %[[C2]], %[[C1]] path %[[FIELD_C]], %[[FIELD_A]] : (index, index, index, !fir.field, !fir.field) -> !fir.slice<1>
+  ! CHECK: %[[SLICE:.*]] = fir.slice %[[C1]], %[[C2]], %[[C1]] path %[[FIELD_C]] : (index, index, index, !fir.field) -> !fir.slice<1>
   ! CHECK: %{{.*}} = fir.embox %[[ALLOCA]](%[[SHAPE]]) [%[[SLICE]]] : (!fir.ref<!fir.array<2x!fir.type<_QFTz{k:i32,c:!fir.type<_QFTc{a:i32,b:i32}>}>>>, !fir.shape<1>, !fir.slice<1>) -> !fir.box<!fir.array<2x!fir.type<_QFTp{a:i32}>>>
+
+  subroutine parent_comp_lhs()
+    type(c) :: a
+    type(p) :: b
+
+    a%p = B
+  end subroutine
+
+! CHECK-LABEL: func.func @_QFPparent_comp_lhs()
+! CHECK: %[[BOX:.*]] = fir.alloca !fir.box<!fir.type<_QFTp{a:i32}>>
+! CHECK: %[[A:.*]] = fir.alloca !fir.type<_QFTc{a:i32,b:i32}> {bindc_name = "a", uniq_name = "_QFFparent_comp_lhsEa"}
+! CHECK: %[[B:.*]] = fir.alloca !fir.type<_QFTp{a:i32}> {bindc_name = "b", uniq_name = "_QFFparent_comp_lhsEb"}
+! CHECK: %[[EMBOX_A:.*]] = fir.embox %[[A]] : (!fir.ref<!fir.type<_QFTc{a:i32,b:i32}>>) -> !fir.box<!fir.type<_QFTp{a:i32}>>
+! CHECK: %[[EMBOX_B:.*]] = fir.embox %[[B]] : (!fir.ref<!fir.type<_QFTp{a:i32}>>) -> !fir.box<!fir.type<_QFTp{a:i32}>>
+! CHECK: fir.store %[[EMBOX_A]] to %[[BOX]] : !fir.ref<!fir.box<!fir.type<_QFTp{a:i32}>>>
+! CHECK: %[[A_NONE:.*]] = fir.convert %[[BOX]] : (!fir.ref<!fir.box<!fir.type<_QFTp{a:i32}>>>) -> !fir.ref<!fir.box<none>>
+! CHECK: %[[B_NONE:.*]] = fir.convert %[[EMBOX_B]] : (!fir.box<!fir.type<_QFTp{a:i32}>>) -> !fir.box<none>
+! CHECK: %{{.*}} = fir.call @_FortranAAssign(%[[A_NONE]], %[[B_NONE]], %{{.*}}, %{{.*}}) {{.*}} : (!fir.ref<!fir.box<none>>, !fir.box<none>, !fir.ref<i8>, i32) -> none
+
 end
diff --git a/flang/test/Lower/polymorphic-temp.f90 b/flang/test/Lower/polymorphic-temp.f90
index 16b629a5d6d29..7858064108b58 100644
--- a/flang/test/Lower/polymorphic-temp.f90
+++ b/flang/test/Lower/polymorphic-temp.f90
@@ -222,7 +222,7 @@ subroutine test_merge_intrinsic2(a, b, i)
 ! CHECK: %[[LOAD_I:.*]] = fir.load %[[I]] : !fir.ref<i32>
 ! CHECK: %[[C1:.*]] = arith.constant 1 : i32
 ! CHECK: %[[CMPI:.*]] = arith.cmpi eq, %[[LOAD_I]], %[[C1]] : i32
-! CHECK: %[[A_CONV:.*]] = fir.convert %[[LOAD_A]] : (!fir.class<!fir.heap<!fir.type<_QMpoly_tmpTp1{a:i32}>>>) -> !fir.box<!fir.heap<!fir.type<_QMpoly_tmpTp1{a:i32}>>>
-! CHECK: %{{.*}} = arith.select %[[CMPI]], %[[A_CONV]], %[[LOAD_B]] : !fir.box<!fir.heap<!fir.type<_QMpoly_tmpTp1{a:i32}>>>
+! CHECK: %[[A_REBOX:.*]] = fir.rebox %[[LOAD_A]] : (!fir.class<!fir.heap<!fir.type<_QMpoly_tmpTp1{a:i32}>>>) -> !fir.box<!fir.heap<!fir.type<_QMpoly_tmpTp1{a:i32}>>> 
+! CHECK: %{{.*}} = arith.select %[[CMPI]], %[[A_REBOX]], %[[LOAD_B]] : !fir.box<!fir.heap<!fir.type<_QMpoly_tmpTp1{a:i32}>>>
 
 end module
diff --git a/flang/test/Lower/polymorphic-types.f90 b/flang/test/Lower/polymorphic-types.f90
index 62ddff090354d..ebd846d25dec4 100644
--- a/flang/test/Lower/polymorphic-types.f90
+++ b/flang/test/Lower/polymorphic-types.f90
@@ -172,13 +172,13 @@ subroutine assumed_type_dummy(a) bind(c)
   end subroutine assumed_type_dummy
 
   ! CHECK-LABEL: func.func @assumed_type_dummy(
-  ! CHECK-SAME: %{{.*}}: !fir.box<none>
+  ! CHECK-SAME: %{{.*}}: !fir.ref<none>
 
   subroutine assumed_type_dummy_array(a) bind(c)
     type(*) :: a(:)
   end subroutine assumed_type_dummy_array
 
   ! CHECK-LABEL: func.func @assumed_type_dummy_array(
-  ! CHECK-SAME: %{{.*}}: !fir.box<!fir.array<?xnone>>
+  ! CHECK-SAME: %{{.*}}: !fir.ref<none>
 
 end module
diff --git a/flang/test/Lower/polymorphic.f90 b/flang/test/Lower/polymorphic.f90
index 16757ff279a86..dd023b9694ff1 100644
--- a/flang/test/Lower/polymorphic.f90
+++ b/flang/test/Lower/polymorphic.f90
@@ -1001,6 +1001,117 @@ subroutine move_alloc_poly(a, b)
 ! CHECK: %[[TYPE_DESC_CONV:.*]] = fir.convert %[[TYPE_DESC]] : (!fir.tdesc<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>) -> !fir.ref<none>
 ! CHECK: %{{.*}} = fir.call @_FortranAMoveAlloc(%[[B_CONV]], %[[A_CONV]], %[[TYPE_DESC_CONV]], %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) {{.*}} : (!fir.ref<!fir.box<none>>, !fir.ref<!fir.box<none>>, !fir.ref<none>, i1, !fir.box<none>, !fir.ref<i8>, i32) -> i32
 
+  subroutine test_parent_comp_in_select_type(s)
+    class(p1), allocatable :: s
+    class(p1), allocatable :: p
+
+    allocate(p1::p)
+
+    select type(s)
+      type is(p2)
+        s%p1 = p
+    end select
+  end subroutine
+
+! CHECK-LABEL: func.func @_QMpolymorphic_testPtest_parent_comp_in_select_type(
+! CHECK-SAME: %[[S:.*]]: !fir.ref<!fir.class<!fir.heap<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>>> {fir.bindc_name = "s"}) {
+! CHECK:  %[[P:.*]] = fir.alloca !fir.class<!fir.heap<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>> {bindc_name = "p", uniq_name = "_QMpolymorphic_testFtest_parent_comp_in_select_typeEp"}
+! CHECK:  %[[LOAD_S:.*]] = fir.load %[[S]] : !fir.ref<!fir.class<!fir.heap<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>>>
+! CHECK:  fir.select_type %[[LOAD_S]] : !fir.class<!fir.heap<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>> [#fir.type_is<!fir.type<_QMpolymorphic_testTp2{a:i32,b:i32,c:f32}>>, ^bb1, unit, ^bb2]
+! CHECK: ^bb1:
+! CHECK:  %[[CONV_S:.*]] = fir.convert %[[LOAD_S]] : (!fir.class<!fir.heap<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>>) -> !fir.box<!fir.heap<!fir.type<_QMpolymorphic_testTp2{a:i32,b:i32,c:f32}>>>
+! CHECK:  %[[REBOX_P1:.*]] = fir.rebox %[[CONV_S]] : (!fir.box<!fir.heap<!fir.type<_QMpolymorphic_testTp2{a:i32,b:i32,c:f32}>>>) -> !fir.box<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>
+! CHECK:  %[[LOAD_P:.*]] = fir.load %[[P]] : !fir.ref<!fir.class<!fir.heap<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>>>
+! CHECK:  %[[LHS_CONV:.*]] = fir.convert %[[REBOX_P1]] : (!fir.box<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>) -> !fir.ref<!fir.box<none>>
+! CHECK:  %[[RHS_CONV:.*]] = fir.convert %[[LOAD_P]] : (!fir.class<!fir.heap<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>>) -> !fir.box<none>
+! CHECK:  %{{.*}} = fir.call @_FortranAAssign(%[[LHS_CONV]], %[[RHS_CONV]], %{{.*}}, %{{.*}}) {{.*}} : (!fir.ref<!fir.box<none>>, !fir.box<none>, !fir.ref<i8>, i32) -> none
+
+  subroutine move_alloc_unlimited_poly(a, b)
+    class(*), allocatable :: a, b
+
+    call move_alloc(a, b)
+  end subroutine
+
+! CHECK-LABEL: func.func @_QMpolymorphic_testPmove_alloc_unlimited_poly(
+! CHECK-SAME: %[[A:.*]]: !fir.ref<!fir.class<!fir.heap<none>>> {fir.bindc_name = "a"}, %[[B:.*]]: !fir.ref<!fir.class<!fir.heap<none>>> {fir.bindc_name = "b"}) {
+! CHECK: %[[NULL:.*]] = fir.zero_bits !fir.ref<none>
+! CHECK: %[[B_CONV:.*]] = fir.convert %[[B]] : (!fir.ref<!fir.class<!fir.heap<none>>>) -> !fir.ref<!fir.box<none>>
+! CHECK: %[[A_CONV:.*]] = fir.convert %[[A]] : (!fir.ref<!fir.class<!fir.heap<none>>>) -> !fir.ref<!fir.box<none>>
+! CHECK: %{{.*}} = fir.call @_FortranAMoveAlloc(%[[B_CONV]], %[[A_CONV]], %[[NULL]], %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) {{.*}} : (!fir.ref<!fir.box<none>>, !fir.ref<!fir.box<none>>, !fir.ref<none>, i1, !fir.box<none>, !fir.ref<i8>, i32) -> i32
+
+  subroutine test_parent_comp_intrinsic(a, b)
+    class(p1) :: a
+    type(p2), allocatable :: b
+    logical :: c
+
+    c = same_type_as(a, b%p1)
+  end subroutine
+
+! CHECK-LABEL: func.func @_QMpolymorphic_testPtest_parent_comp_intrinsic(
+! CHECK-SAME: %[[ARG0:.*]]: !fir.class<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>> {fir.bindc_name = "a"}, %[[ARG1:.*]]: !fir.ref<!fir.box<!fir.heap<!fir.type<_QMpolymorphic_testTp2{a:i32,b:i32,c:f32}>>>> {fir.bindc_name = "b"}) {
+! CHECK: %[[LOAD_ARG1:.*]] = fir.load %[[ARG1]] : !fir.ref<!fir.box<!fir.heap<!fir.type<_QMpolymorphic_testTp2{a:i32,b:i32,c:f32}>>>>
+! CHECK: %[[REBOX_ARG1:.*]] = fir.rebox %[[LOAD_ARG1]] : (!fir.box<!fir.heap<!fir.type<_QMpolymorphic_testTp2{a:i32,b:i32,c:f32}>>>) -> !fir.box<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>
+! CHECK: %[[BOX_NONE_ARG0:.*]] = fir.convert %[[ARG0]] : (!fir.class<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>) -> !fir.box<none>
+! CHECK: %[[BOX_NONE_ARG1:.*]] = fir.convert %[[REBOX_ARG1]] : (!fir.box<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>) -> !fir.box<none>
+! CHECK: %{{.*}} = fir.call @_FortranASameTypeAs(%[[BOX_NONE_ARG0]], %[[BOX_NONE_ARG1]]) {{.*}} : (!fir.box<none>, !fir.box<none>) -> i1
+
+  subroutine test_parent_comp_normal(a)
+    class(p2) :: a
+
+    call print(a%p1)
+  end subroutine
+
+! CHECK-LABEL: func.func @_QMpolymorphic_testPtest_parent_comp_normal(
+! CHECK-SAME: %[[ARG0:.*]]: !fir.class<!fir.type<_QMpolymorphic_testTp2{a:i32,b:i32,c:f32}>> {fir.bindc_name = "a"}) {
+! CHECK: %[[REBOX:.*]] = fir.rebox %[[ARG0]] : (!fir.class<!fir.type<_QMpolymorphic_testTp2{a:i32,b:i32,c:f32}>>) -> !fir.box<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>
+! CHECK: %[[CONV:.*]] = fir.convert %[[REBOX]] : (!fir.box<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>) -> !fir.class<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>
+! CHECK: fir.call @_QMpolymorphic_testPprint(%[[CONV]]) {{.*}} : (!fir.class<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>) -> ()
+
+  subroutine takes_p1_opt(a)
+    class(p1), optional :: a
+  end subroutine
+
+  subroutine test_parent_comp_opt(p)
+    type(p2), allocatable :: p
+    allocate(p)
+    call takes_p1_opt(p%p1)
+  end subroutine
+
+! CHECK-LABEL: func.func @_QMpolymorphic_testPtest_parent_comp_opt(
+! CHECK-SAME: %[[ARG0:.*]]: !fir.ref<!fir.box<!fir.heap<!fir.type<_QMpolymorphic_testTp2{a:i32,b:i32,c:f32}>>>> {fir.bindc_name = "p"}) {
+! CHECK: %[[LOAD_ARG0:.*]] = fir.load %[[ARG0]] : !fir.ref<!fir.box<!fir.heap<!fir.type<_QMpolymorphic_testTp2{a:i32,b:i32,c:f32}>>>>
+! CHECK: %[[RES:.*]] = fir.if %{{.*}} -> (!fir.box<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>) {
+! CHECK:   %[[REBOX:.*]] = fir.rebox %[[LOAD_ARG0:.*]] : (!fir.box<!fir.heap<!fir.type<_QMpolymorphic_testTp2{a:i32,b:i32,c:f32}>>>) -> !fir.box<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>
+! CHECK:   fir.result %[[REBOX]] : !fir.box<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>
+! CHECK: %[[CONV:.*]] = fir.convert %[[RES]] : (!fir.box<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>) -> !fir.class<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>
+! CHECK: fir.call @_QMpolymorphic_testPtakes_p1_opt(%[[CONV]]) {{.*}} : (!fir.class<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>>) -> ()
+
+  subroutine class_with_entry(a)
+    class(p1) :: a,b
+    select type (a)
+    type is(p2)
+      print*, a%c
+    class default
+      print*, a%a
+    end select
+    return
+  entry d(b)
+    select type(b)
+    type is(p2)
+      print*,b%c
+    class default
+      print*,b%a
+    end select
+  end subroutine
+
+! CHECK-LABEL: func.func @_QMpolymorphic_testPclass_with_entry(
+! CHECK-SAME: %[[A:.*]]: !fir.class<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>> {fir.bindc_name = "a"}) {
+! CHECK: %[[B:.*]] = fir.alloca !fir.class<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>> {bindc_name = "b", uniq_name = "_QMpolymorphic_testFclass_with_entryEb"}
+
+! CHECK-LABEL: func.func @_QMpolymorphic_testPd(
+! CHECK-SAME: %[[B:.*]]: !fir.class<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>> {fir.bindc_name = "b"}) {
+! CHECK: %[[A:.*]] = fir.alloca !fir.class<!fir.type<_QMpolymorphic_testTp1{a:i32,b:i32}>> {bindc_name = "a", uniq_name = "_QMpolymorphic_testFclass_with_entryEa"}
+
 end module
 
 program test
diff --git a/flang/test/Lower/ppc-intrinsics.f90 b/flang/test/Lower/ppc-intrinsics.f90
index 293a35a63b061..8410fbdd86aa1 100644
--- a/flang/test/Lower/ppc-intrinsics.f90
+++ b/flang/test/Lower/ppc-intrinsics.f90
@@ -65,3 +65,75 @@ subroutine fnmsub_testd(a, x, y)
 ! CHECK-FIR: fir.call @fir.__ppc_fnmsub.f64.f64.f64.f64
 ! CHECK-LLVMIR: call contract double @llvm.ppc.fnmsub.f64(double %{{[0-9]}}, double %{{[0-9]}}, double %{{[0-9]}})
 end
+
+! CHECK-LABEL: fctid_test
+subroutine fctid_test(x)
+  real(8) :: x, r
+  r = fctid(x)
+! CHECK-FIR: fir.call @fir.__ppc_fctid.f64.f64
+! CHECK-LLVMIR: call contract double @llvm.ppc.fctid(double %{{[0-9]}})
+end
+
+! CHECK-LABEL: fctidz_test
+subroutine fctidz_test(x)
+  real(8) :: x, r
+  r = fctidz(x)
+! CHECK-FIR: fir.call @fir.__ppc_fctidz.f64.f64
+! CHECK-LLVMIR: call contract double @llvm.ppc.fctidz(double %{{[0-9]}})
+end
+
+! CHECK-LABEL: fctiw_test
+subroutine fctiw_test(x)
+  real(8) :: x, r
+  r = fctiw(x)
+! CHECK-FIR: fir.call @fir.__ppc_fctiw.f64.f64
+! CHECK-LLVMIR: call contract double @llvm.ppc.fctiw(double %{{[0-9]}})
+end
+
+! CHECK-LABEL: fctiwz_test
+subroutine fctiwz_test(x)
+  real(8) :: x, r
+  r = fctiwz(x)
+! CHECK-FIR: fir.call @fir.__ppc_fctiwz.f64.f64
+! CHECK-LLVMIR: call contract double @llvm.ppc.fctiwz(double %{{[0-9]}})
+end
+
+! CHECK-LABEL: fctudz_test
+subroutine fctudz_test(x)
+  real(8) :: x, r
+  r = fctudz(x)
+! CHECK-FIR: fir.call @fir.__ppc_fctudz.f64.f64
+! CHECK-LLVMIR: call contract double @llvm.ppc.fctudz(double %{{[0-9]}})
+end
+
+! CHECK-LABEL: fctuwz_test
+subroutine fctuwz_test(x)
+  real(8) :: x, r
+  r = fctuwz(x)
+! CHECK-FIR: fir.call @fir.__ppc_fctuwz.f64.f64
+! CHECK-LLVMIR: call contract double @llvm.ppc.fctuwz(double %{{[0-9]}})
+end
+
+! CHECK-LABEL: fcfi_test
+subroutine fcfi_test(i)
+  real(8) :: i, r
+  r = fcfi(i)
+! CHECK-FIR: fir.call @fir.__ppc_fcfi.f64.f64
+! CHECK-LLVMIR: call contract double @llvm.ppc.fcfid(double %{{[0-9]}})
+end
+
+! CHECK-LABEL: fcfid_test
+subroutine fcfid_test(i)
+  real(8) :: i, r
+  r = fcfid(i)
+! CHECK-FIR: fir.call @fir.__ppc_fcfid.f64.f64
+! CHECK-LLVMIR: call contract double @llvm.ppc.fcfid(double %{{[0-9]}})
+end
+
+! CHECK-LABEL: fcfud_test
+subroutine fcfud_test(i)
+  real(8) :: i, r
+  r = fcfud(i)
+! CHECK-FIR: fir.call @fir.__ppc_fcfud.f64.f64
+! CHECK-LLVMIR: call contract double @llvm.ppc.fcfud(double %{{[0-9]}})
+end
diff --git a/flang/test/Semantics/OpenMP/reduction07.f90 b/flang/test/Semantics/OpenMP/reduction07.f90
index a3b58765d9a6d..98ed69a8d846d 100644
--- a/flang/test/Semantics/OpenMP/reduction07.f90
+++ b/flang/test/Semantics/OpenMP/reduction07.f90
@@ -78,42 +78,6 @@ program omp_reduction
 !$omp parallel reduction(*:a)
 !$omp end parallel
 
-
-!$omp parallel reduction(+:a)
-!ERROR: REDUCTION clause is not allowed on the WORKSHARE directive
-!ERROR: REDUCTION variable 'a' is REDUCTION in outer context must be shared in the parallel regions to which any of the worksharing regions arising from the worksharing construct bind.
-!$omp workshare reduction(*:a)
-a = a + 10
-!$omp end workshare
-!$omp end parallel
-
-!$omp parallel reduction(*:a)
-!$omp end parallel
-
-
-!$omp parallel reduction(+:a)
-!ERROR: REDUCTION clause is not allowed on the SINGLE directive
-!ERROR: REDUCTION variable 'a' is REDUCTION in outer context must be shared in the parallel regions to which any of the worksharing regions arising from the worksharing construct bind.
-!$omp single reduction(*:a)
-a = a + 10
-!$omp end single
-!$omp end parallel
-
-!$omp parallel reduction(+:a)
-!$omp end parallel
-
-
-!$omp parallel reduction(+:a)
-!ERROR: REDUCTION clause is not allowed on the SINGLE directive
-!ERROR: REDUCTION variable 'a' is REDUCTION in outer context must be shared in the parallel regions to which any of the worksharing regions arising from the worksharing construct bind.
-!$omp single reduction(iand:a)
-a = a + 10
-!$omp end single
-!$omp end parallel
-
-!$omp parallel reduction(iand:a)
-!$omp end parallel
-
 !$omp parallel reduction(ieor:a)
 !ERROR: REDUCTION variable 'a' is REDUCTION in outer context must be shared in the parallel regions to which any of the worksharing regions arising from the worksharing construct bind.
 !$omp sections reduction(+:a)
@@ -121,7 +85,17 @@ program omp_reduction
 !$omp end sections
 !$omp end parallel
 
+!$omp parallel private(a)
 !$omp parallel reduction(ieor:a)
 !$omp end parallel
+!$omp end parallel
+
+!$omp task firstprivate(a)
+!$omp parallel do reduction(+:a)
+do i=1,10
+  a=a+j
+end do
+!$omp end parallel do
+!$omp end task
 
 end program omp_reduction
diff --git a/flang/test/Semantics/argshape01.f90 b/flang/test/Semantics/argshape01.f90
new file mode 100644
index 0000000000000..42ba0fab4a5e5
--- /dev/null
+++ b/flang/test/Semantics/argshape01.f90
@@ -0,0 +1,41 @@
+! RUN: %python %S/test_errors.py %s %flang_fc1
+! Detect incompatible argument shapes
+module m
+ contains
+  subroutine s1(a)
+    real, intent(in) :: a(2,3)
+  end
+  subroutine s2(a)
+    real, intent(in) :: a(3,2)
+  end
+  subroutine s1c(s)
+    procedure(s1) :: s
+  end
+  subroutine s2c(s)
+    procedure(s2) :: s
+  end
+end
+
+program main
+  use m
+  procedure(s1), pointer :: ps1
+  procedure(s2), pointer :: ps2
+  call s1c(s1)
+  !ERROR: Actual procedure argument has interface incompatible with dummy argument 's=': incompatible dummy argument #1: incompatible dummy data object shapes
+  call s1c(s2)
+  !ERROR: Actual procedure argument has interface incompatible with dummy argument 's=': incompatible dummy argument #1: incompatible dummy data object shapes
+  call s2c(s1)
+  call s2c(s2)
+  ps1 => s1
+  !ERROR: Procedure pointer 'ps1' associated with incompatible procedure designator 's2': incompatible dummy argument #1: incompatible dummy data object shapes
+  ps1 => s2
+  !ERROR: Procedure pointer 'ps2' associated with incompatible procedure designator 's1': incompatible dummy argument #1: incompatible dummy data object shapes
+  ps2 => s1
+  ps2 => s2
+  call s1c(ps1)
+  !ERROR: Actual procedure argument has interface incompatible with dummy argument 's=': incompatible dummy argument #1: incompatible dummy data object shapes
+  call s1c(ps2)
+  !ERROR: Actual procedure argument has interface incompatible with dummy argument 's=': incompatible dummy argument #1: incompatible dummy data object shapes
+  call s2c(ps1)
+  call s2c(ps2)
+end
diff --git a/flang/test/Semantics/bind-c11.f90 b/flang/test/Semantics/bind-c11.f90
new file mode 100644
index 0000000000000..29ae09bd39f89
--- /dev/null
+++ b/flang/test/Semantics/bind-c11.f90
@@ -0,0 +1,21 @@
+! RUN: %python %S/test_errors.py %s %flang_fc1
+module m
+  !ERROR: A scalar interoperable variable may not be ALLOCATABLE or POINTER
+  real, allocatable, bind(c) :: x1
+  !ERROR: A scalar interoperable variable may not be ALLOCATABLE or POINTER
+  real, pointer, bind(c) :: x2
+  !ERROR: BIND(C) array must have explicit shape or be assumed-size unless a dummy argument without the VALUE attribute
+  real, allocatable, bind(c) :: x3(:)
+ contains
+  subroutine s1(x) bind(c)
+    !PORTABILITY: A BIND(C) LOGICAL dummy argument should have the interoperable KIND=C_BOOL
+    logical(2), intent(in), value :: x
+  end
+  subroutine s2(x) bind(c)
+    !PORTABILITY: An interoperable procedure with an OPTIONAL dummy argument might not be portable
+    integer, intent(in), optional :: x
+  end
+  !ERROR: A subprogram interface with the BIND attribute may not have an alternate return argument
+  subroutine s3(*) bind(c)
+  end
+end
diff --git a/flang/test/Semantics/call03.f90 b/flang/test/Semantics/call03.f90
index a34394cccb8ae..7a860062262a9 100644
--- a/flang/test/Semantics/call03.f90
+++ b/flang/test/Semantics/call03.f90
@@ -168,12 +168,16 @@ subroutine test06 ! 15.5.2.4(4)
     !WARNING: Actual argument expression length '0' is less than expected length '2'
     call ch2("")
     call pdtdefault(vardefault)
+    !ERROR: Actual argument type 'pdt(n=3_4)' is not compatible with dummy argument type 'pdt'
     call pdtdefault(var3)
+    !ERROR: Actual argument type 'pdt(n=4_4)' is not compatible with dummy argument type 'pdt'
     call pdtdefault(var4) ! error
-    call pdt3(vardefault) ! error
+    !ERROR: Actual argument type 'pdt' is not compatible with dummy argument type 'pdt(n=4_4)'
+    call pdt3(vardefault)
     !ERROR: Actual argument type 'pdt(n=3_4)' is not compatible with dummy argument type 'pdt(n=4_4)'
-    call pdt3(var3) ! error
+    call pdt3(var3)
     call pdt3(var4)
+    !ERROR: Actual argument type 'pdt' is not compatible with dummy argument type 'pdt(n=*)'
     call pdt4(vardefault)
     call pdt4(var3)
     call pdt4(var4)
diff --git a/flang/test/Semantics/final03.f90 b/flang/test/Semantics/final03.f90
new file mode 100644
index 0000000000000..c4013efb424eb
--- /dev/null
+++ b/flang/test/Semantics/final03.f90
@@ -0,0 +1,28 @@
+! RUN: %python %S/test_errors.py %s %flang_fc1
+! PDT sensitivity of FINAL subroutines
+module m
+  type :: pdt(k)
+    integer, kind :: k
+   contains
+    final :: finalArr, finalElem
+  end type
+ contains
+  subroutine finalArr(x)
+    type(pdt(1)), intent(in out) :: x(:)
+  end
+  elemental subroutine finalElem(x)
+    type(pdt(3)), intent(in out) :: x
+  end
+end
+
+program test
+  use m
+  type(pdt(1)) x1(1)
+  type(pdt(2)) x2(1)
+  type(pdt(3)) x3(1)
+  !ERROR: Left-hand side of assignment is not definable
+  !BECAUSE: Variable 'x1([INTEGER(8)::1_8])' has a vector subscript and cannot be finalized by non-elemental subroutine 'finalarr'
+  x1([1]) = pdt(1)()
+  x2([1]) = pdt(2)() ! ok, doesn't match either
+  x3([1]) = pdt(3)() ! ok, calls finalElem
+end
diff --git a/flang/test/Semantics/implicit11.f90 b/flang/test/Semantics/implicit11.f90
index 3c68ab5b848b2..8d5a1bd0be7bc 100644
--- a/flang/test/Semantics/implicit11.f90
+++ b/flang/test/Semantics/implicit11.f90
@@ -40,6 +40,7 @@ subroutine s3()
   subroutine s3a()
     implicit none
     real :: a(m, n)
+    !WARN: '%s' was used without (or before) being explicitly typed
     !ERROR: No explicit type declared for 'n'
     common n
   end
diff --git a/flang/test/Semantics/kinds02.f90 b/flang/test/Semantics/kinds02.f90
index b02659e701f31..13dbb803de8e5 100644
--- a/flang/test/Semantics/kinds02.f90
+++ b/flang/test/Semantics/kinds02.f90
@@ -82,6 +82,8 @@
 character(len=*), parameter :: cvar10 = 4_"abcd"
 !ERROR: CHARACTER(KIND=8) is not a supported type
 character(len=*), parameter :: cvar11 = 8_"abcd"
+!ERROR: Unsupported type kind value (5000000000)
+print *, 123_5000000000
 end program
 
 subroutine s(a, b)
diff --git a/flang/test/Semantics/modfile04.f90 b/flang/test/Semantics/modfile04.f90
index c0829c4417984..2779dd4bdf57d 100644
--- a/flang/test/Semantics/modfile04.f90
+++ b/flang/test/Semantics/modfile04.f90
@@ -7,7 +7,7 @@ module m1
 contains
 
   pure subroutine Ss(x, y) bind(c)
-    logical x
+    logical(1) x
     intent(inout) y
     intent(in) x
   end subroutine
@@ -54,7 +54,7 @@ end module m3
 !end type
 !contains
 !pure subroutine ss(x,y) bind(c)
-!logical(4),intent(in)::x
+!logical(1),intent(in)::x
 !real(4),intent(inout)::y
 !end
 !function f1() result(x)
diff --git a/flang/test/Semantics/resolve103.f90 b/flang/test/Semantics/resolve103.f90
index dbc3bca6cbc48..8f55968f43375 100644
--- a/flang/test/Semantics/resolve103.f90
+++ b/flang/test/Semantics/resolve103.f90
@@ -1,17 +1,17 @@
 ! RUN: not %flang_fc1 -pedantic %s 2>&1 | FileCheck %s
-! Test extension: allow forward references to dummy arguments
+! Test extension: allow forward references to dummy arguments or COMMON
 ! from specification expressions in scopes with IMPLICIT NONE(TYPE),
 ! as long as those symbols are eventually typed later with the
 ! same integer type they would have had without IMPLICIT NONE.
 
-!CHECK: Dummy argument 'n1' was used without being explicitly typed
+!CHECK: warning: 'n1' was used without (or before) being explicitly typed
 !CHECK: error: No explicit type declared for dummy argument 'n1'
 subroutine foo1(a, n1)
   implicit none
   real a(n1)
 end
 
-!CHECK: Dummy argument 'n2' was used without being explicitly typed
+!CHECK: warning: 'n2' was used without (or before) being explicitly typed
 subroutine foo2(a, n2)
   implicit none
   real a(n2)
@@ -19,10 +19,35 @@ subroutine foo2(a, n2)
   double precision n2
 end
 
-!CHECK: Dummy argument 'n3' was used without being explicitly typed
+!CHECK: warning: 'n3' was used without (or before) being explicitly typed
 !CHECK-NOT: error: Dummy argument 'n3'
 subroutine foo3(a, n3)
   implicit none
   real a(n3)
   integer n3
 end
+
+!CHECK: warning: 'n4' was used without (or before) being explicitly typed
+!CHECK: error: No explicit type declared for 'n4'
+subroutine foo4(a)
+  implicit none
+  real a(n4)
+  common /b4/ n4
+end
+
+!CHECK: warning: 'n5' was used without (or before) being explicitly typed
+subroutine foo5(a)
+  implicit none
+  real a(n5)
+  common /b5/ n5
+!CHECK: error: The type of 'n5' has already been implicitly declared
+  double precision n5
+end
+
+!CHECK: warning: 'n6' was used without (or before) being explicitly typed
+subroutine foo6(a)
+  implicit none
+  real a(n6)
+  common /b6/ n6
+  integer n6
+end
diff --git a/flang/test/Semantics/resolve32.f90 b/flang/test/Semantics/resolve32.f90
index 060b29ec40b5a..948493b1615e6 100644
--- a/flang/test/Semantics/resolve32.f90
+++ b/flang/test/Semantics/resolve32.f90
@@ -18,6 +18,10 @@ module m
     subroutine foo
     end subroutine
   end interface
+  abstract interface
+    subroutine absfoo
+    end subroutine
+  end interface
   integer :: i
   type t1
     integer :: c
@@ -34,6 +38,8 @@ subroutine foo
     !ERROR: 's3' must be either an accessible module procedure or an external procedure with an explicit interface
     procedure, nopass :: s3
     procedure, nopass :: foo
+    !ERROR: 'absfoo' must be either an accessible module procedure or an external procedure with an explicit interface
+    procedure, nopass :: absfoo
     !ERROR: 'bar' must be either an accessible module procedure or an external procedure with an explicit interface
     procedure, nopass :: bar
     !ERROR: 'i' must be either an accessible module procedure or an external procedure with an explicit interface
diff --git a/flang/test/Semantics/selecttype01.f90 b/flang/test/Semantics/selecttype01.f90
index f0b3e6ea103e8..4ac7fe6aafc46 100644
--- a/flang/test/Semantics/selecttype01.f90
+++ b/flang/test/Semantics/selecttype01.f90
@@ -277,3 +277,14 @@ subroutine WorkingPolymorphism
       print *, "default"
   end select
 end
+
+subroutine CheckNotProcedure
+  use m1
+  !ERROR: Selector may not be a procedure
+  select type (x=>f)
+  end select
+ contains
+  function f() result(res)
+    class(shape), allocatable :: res
+  end
+end
diff --git a/flang/test/Semantics/symbol25.f90 b/flang/test/Semantics/symbol25.f90
new file mode 100644
index 0000000000000..ac3dd37ef92eb
--- /dev/null
+++ b/flang/test/Semantics/symbol25.f90
@@ -0,0 +1,60 @@
+! RUN: %python %S/test_symbols.py %s %flang_fc1
+! Exercise generic redefinitions in inner procedures with conflicting subprograms.
+!DEF: /m Module
+module m
+ !DEF: /m/generic PUBLIC (Subroutine) Generic
+ interface generic
+  !DEF: /m/specific1 PUBLIC (Subroutine) Subprogram
+  module procedure :: specific1
+ end interface
+contains
+ !REF: /m/specific1
+ subroutine specific1
+  print *, 1
+ end subroutine
+ !DEF: /m/specific2 PUBLIC (Subroutine) Subprogram
+ subroutine specific2
+  print *, 2
+ end subroutine
+ !DEF: /m/test PUBLIC (Subroutine) Subprogram
+ subroutine test
+  !REF: /m/specific1
+  call generic
+ end subroutine
+ !DEF: /m/outer PUBLIC (Subroutine) Subprogram
+ subroutine outer
+  !DEF: /m/outer/inner1 (Subroutine) Subprogram
+  call inner1
+ contains
+  !REF: /m/outer/inner1
+  subroutine inner1
+   !DEF: /m/outer/inner1/generic (Subroutine) Generic
+   interface generic
+    !REF: /m/specific2
+    module procedure :: specific2
+   end interface
+   !REF: /m/specific2
+   call generic
+  end subroutine inner1
+ end subroutine outer
+end module m
+!DEF: /main MainProgram
+program main
+ !REF: /m
+ use :: m
+ !REF: /m/specific1
+ call generic
+ !DEF: /main/inner2 (Subroutine) Subprogram
+ call inner2
+contains
+ !REF: /main/inner2
+ subroutine inner2
+  !DEF: /main/inner2/generic (Subroutine) Generic
+  interface generic
+   !DEF: /main/specific2 (Subroutine) Use
+   module procedure :: specific2
+  end interface
+  !REF: /main/specific2
+  call generic
+ end subroutine inner2
+end program
diff --git a/flang/test/Transforms/stack-arrays.fir b/flang/test/Transforms/stack-arrays.fir
index 7b73bf0930743..d470ea704be48 100644
--- a/flang/test/Transforms/stack-arrays.fir
+++ b/flang/test/Transforms/stack-arrays.fir
@@ -48,7 +48,7 @@ func.func @dfa1(%arg0: !fir.ref<!fir.logical<4>> {fir.bindc_name = "cond"}) {
 // CHECK-NEXT:   return
 // CHECK-NEXT: }
 
-// Check scf.if (fir.if is not considered a branch operation)
+// Check scf.if
 func.func @dfa2(%arg0: i1) {
   %a = fir.allocmem !fir.array<1xi8>
   scf.if %arg0 {
@@ -66,6 +66,24 @@ func.func @dfa2(%arg0: i1) {
 // CHECK-NEXT:  return
 // CHECK-NEXT:  }
 
+// Check freemem in both regions
+func.func @dfa3(%arg0: i1) {
+  %a = fir.allocmem !fir.array<1xi8>
+  fir.if %arg0 {
+    fir.freemem %a : !fir.heap<!fir.array<1xi8>>
+  } else {
+    fir.freemem %a : !fir.heap<!fir.array<1xi8>>
+  }
+  return
+}
+// CHECK:     func.func @dfa3(%arg0: i1) {
+// CHECK-NEXT:  %[[MEM:.*]] = fir.alloca !fir.array<1xi8>
+// CHECK-NEXT:  fir.if %arg0 {
+// CHECK-NEXT:  } else {
+// CHECK-NEXT:  }
+// CHECK-NEXT:  return
+// CHECK-NEXT:  }
+
 // check the alloca is placed after all operands become available
 func.func @placement1() {
   // do some stuff with other ssa values
diff --git a/flang/test/lit.site.cfg.py.in b/flang/test/lit.site.cfg.py.in
index dbd84354e854e..46ba53ef672ef 100644
--- a/flang/test/lit.site.cfg.py.in
+++ b/flang/test/lit.site.cfg.py.in
@@ -16,7 +16,7 @@ config.flang_intrinsic_modules_dir = "@FLANG_INTRINSIC_MODULES_DIR@"
 config.flang_llvm_tools_dir = "@CMAKE_BINARY_DIR@/bin"
 config.flang_lib_dir = "@CMAKE_BINARY_DIR@/lib"
 config.flang_test_triple = "@FLANG_TEST_TARGET_TRIPLE@"
-config.flang_examples = @FLANG_BUILD_EXAMPLES@
+config.flang_examples = @LLVM_BUILD_EXAMPLES@
 config.python_executable = "@PYTHON_EXECUTABLE@"
 config.flang_standalone_build = @FLANG_STANDALONE_BUILD@
 config.has_plugins = @LLVM_ENABLE_PLUGINS@
diff --git a/flang/tools/bbc/CMakeLists.txt b/flang/tools/bbc/CMakeLists.txt
index dba7bee2ebb88..0b8c2eb90e5c8 100644
--- a/flang/tools/bbc/CMakeLists.txt
+++ b/flang/tools/bbc/CMakeLists.txt
@@ -12,6 +12,7 @@ llvm_update_compile_flags(bbc)
 get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
 target_link_libraries(bbc PRIVATE
 FIRDialect
+FIRDialectSupport
 FIRSupport
 FIRTransforms
 FIRBuilder
diff --git a/flang/tools/bbc/bbc.cpp b/flang/tools/bbc/bbc.cpp
index af2e162bf4563..d4692a671a852 100644
--- a/flang/tools/bbc/bbc.cpp
+++ b/flang/tools/bbc/bbc.cpp
@@ -19,10 +19,10 @@
 #include "flang/Lower/Bridge.h"
 #include "flang/Lower/PFTBuilder.h"
 #include "flang/Lower/Support/Verifier.h"
-#include "flang/Optimizer/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "flang/Optimizer/Support/InitFIR.h"
 #include "flang/Optimizer/Support/InternalNames.h"
-#include "flang/Optimizer/Support/KindMapping.h"
 #include "flang/Optimizer/Support/Utils.h"
 #include "flang/Optimizer/Transforms/Passes.h"
 #include "flang/Parser/characters.h"
@@ -38,6 +38,7 @@
 #include "flang/Semantics/semantics.h"
 #include "flang/Semantics/unparse-with-symbols.h"
 #include "flang/Version.inc"
+#include "mlir/Dialect/OpenMP/OpenMPDialect.h"
 #include "mlir/IR/AsmState.h"
 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/MLIRContext.h"
@@ -122,6 +123,11 @@ static llvm::cl::opt<bool> enableOpenMP("fopenmp",
                                         llvm::cl::desc("enable openmp"),
                                         llvm::cl::init(false));
 
+static llvm::cl::opt<bool>
+    enableOpenMPDevice("fopenmp-is-device",
+                       llvm::cl::desc("enable openmp device compilation"),
+                       llvm::cl::init(false));
+
 static llvm::cl::opt<bool> enableOpenACC("fopenacc",
                                          llvm::cl::desc("enable openacc"),
                                          llvm::cl::init(false));
@@ -237,6 +243,8 @@ static mlir::LogicalResult convertFortranSourceToMLIR(
       kindMap, loweringOptions, {});
   burnside.lower(parseTree, semanticsContext);
   mlir::ModuleOp mlirModule = burnside.getModule();
+  if (enableOpenMP)
+    mlir::omp::OpenMPDialect::setIsDevice(mlirModule, enableOpenMPDevice);
   std::error_code ec;
   std::string outputName = outputFilename;
   if (!outputName.size())
diff --git a/flang/tools/tco/CMakeLists.txt b/flang/tools/tco/CMakeLists.txt
index 28cbbb5154ccf..207e7225276df 100644
--- a/flang/tools/tco/CMakeLists.txt
+++ b/flang/tools/tco/CMakeLists.txt
@@ -8,6 +8,7 @@ get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
 target_link_libraries(tco PRIVATE
   FIRCodeGen
   FIRDialect
+  FIRDialectSupport
   FIRSupport
   FIRTransforms
   FIRBuilder
diff --git a/flang/tools/tco/tco.cpp b/flang/tools/tco/tco.cpp
index c68e5456ab31b..e7dca6b06b27d 100644
--- a/flang/tools/tco/tco.cpp
+++ b/flang/tools/tco/tco.cpp
@@ -12,10 +12,10 @@
 //===----------------------------------------------------------------------===//
 
 #include "flang/Optimizer/CodeGen/CodeGen.h"
-#include "flang/Optimizer/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "flang/Optimizer/Support/InitFIR.h"
 #include "flang/Optimizer/Support/InternalNames.h"
-#include "flang/Optimizer/Support/KindMapping.h"
 #include "flang/Optimizer/Transforms/Passes.h"
 #include "mlir/IR/AsmState.h"
 #include "mlir/IR/BuiltinOps.h"
diff --git a/flang/unittests/Optimizer/Builder/CharacterTest.cpp b/flang/unittests/Optimizer/Builder/CharacterTest.cpp
index 21230ae39963c..c6defcd51095b 100644
--- a/flang/unittests/Optimizer/Builder/CharacterTest.cpp
+++ b/flang/unittests/Optimizer/Builder/CharacterTest.cpp
@@ -10,8 +10,8 @@
 #include "gtest/gtest.h"
 #include "flang/Optimizer/Builder/BoxValue.h"
 #include "flang/Optimizer/Builder/FIRBuilder.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "flang/Optimizer/Support/InitFIR.h"
-#include "flang/Optimizer/Support/KindMapping.h"
 
 struct CharacterTest : public testing::Test {
 public:
diff --git a/flang/unittests/Optimizer/Builder/ComplexTest.cpp b/flang/unittests/Optimizer/Builder/ComplexTest.cpp
index 4b895ee7d9e24..5364eec904ff4 100644
--- a/flang/unittests/Optimizer/Builder/ComplexTest.cpp
+++ b/flang/unittests/Optimizer/Builder/ComplexTest.cpp
@@ -9,8 +9,8 @@
 #include "flang/Optimizer/Builder/Complex.h"
 #include "gtest/gtest.h"
 #include "flang/Optimizer/Builder/FIRBuilder.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "flang/Optimizer/Support/InitFIR.h"
-#include "flang/Optimizer/Support/KindMapping.h"
 
 struct ComplexTest : public testing::Test {
 public:
diff --git a/flang/unittests/Optimizer/Builder/DoLoopHelperTest.cpp b/flang/unittests/Optimizer/Builder/DoLoopHelperTest.cpp
index 80cf1655df053..7e7206dbf934d 100644
--- a/flang/unittests/Optimizer/Builder/DoLoopHelperTest.cpp
+++ b/flang/unittests/Optimizer/Builder/DoLoopHelperTest.cpp
@@ -8,8 +8,8 @@
 
 #include "flang/Optimizer/Builder/DoLoopHelper.h"
 #include "gtest/gtest.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "flang/Optimizer/Support/InitFIR.h"
-#include "flang/Optimizer/Support/KindMapping.h"
 #include <string>
 
 using namespace mlir;
diff --git a/flang/unittests/Optimizer/Builder/FIRBuilderTest.cpp b/flang/unittests/Optimizer/Builder/FIRBuilderTest.cpp
index 3dedf3d75c5ce..c3436ccd1c779 100644
--- a/flang/unittests/Optimizer/Builder/FIRBuilderTest.cpp
+++ b/flang/unittests/Optimizer/Builder/FIRBuilderTest.cpp
@@ -9,8 +9,8 @@
 #include "flang/Optimizer/Builder/FIRBuilder.h"
 #include "gtest/gtest.h"
 #include "flang/Optimizer/Builder/BoxValue.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "flang/Optimizer/Support/InitFIR.h"
-#include "flang/Optimizer/Support/KindMapping.h"
 
 using namespace mlir;
 
diff --git a/flang/unittests/Optimizer/Builder/HLFIRToolsTest.cpp b/flang/unittests/Optimizer/Builder/HLFIRToolsTest.cpp
index 75bd4b35ccbcf..1858b276f1fc3 100644
--- a/flang/unittests/Optimizer/Builder/HLFIRToolsTest.cpp
+++ b/flang/unittests/Optimizer/Builder/HLFIRToolsTest.cpp
@@ -10,8 +10,8 @@
 #include "gtest/gtest.h"
 #include "flang/Optimizer/Builder/BoxValue.h"
 #include "flang/Optimizer/Builder/FIRBuilder.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "flang/Optimizer/Support/InitFIR.h"
-#include "flang/Optimizer/Support/KindMapping.h"
 
 struct HLFIRToolsTest : public testing::Test {
 public:
diff --git a/flang/unittests/Optimizer/Builder/Runtime/RuntimeCallTestBase.h b/flang/unittests/Optimizer/Builder/Runtime/RuntimeCallTestBase.h
index 0e1c0f4cb60e2..00c27f4bb2142 100644
--- a/flang/unittests/Optimizer/Builder/Runtime/RuntimeCallTestBase.h
+++ b/flang/unittests/Optimizer/Builder/Runtime/RuntimeCallTestBase.h
@@ -11,8 +11,8 @@
 
 #include "gtest/gtest.h"
 #include "flang/Optimizer/Builder/FIRBuilder.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "flang/Optimizer/Support/InitFIR.h"
-#include "flang/Optimizer/Support/KindMapping.h"
 
 struct RuntimeCallTest : public testing::Test {
 public:
diff --git a/flang/unittests/Optimizer/CMakeLists.txt b/flang/unittests/Optimizer/CMakeLists.txt
index 76555fe7bccbf..d55d69598b153 100644
--- a/flang/unittests/Optimizer/CMakeLists.txt
+++ b/flang/unittests/Optimizer/CMakeLists.txt
@@ -4,6 +4,7 @@ set(LIBS
   FIRBuilder
   FIRCodeGen
   FIRDialect
+  FIRDialectSupport
   FIRSupport
   HLFIRDialect
   ${dialect_libs}
diff --git a/flang/unittests/Optimizer/FIRContextTest.cpp b/flang/unittests/Optimizer/FIRContextTest.cpp
index a4027b9828357..88bb4363b2b45 100644
--- a/flang/unittests/Optimizer/FIRContextTest.cpp
+++ b/flang/unittests/Optimizer/FIRContextTest.cpp
@@ -6,11 +6,11 @@
 //
 //===----------------------------------------------------------------------===//
 
-#include "flang/Optimizer/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
 #include "gtest/gtest.h"
 #include "mlir/IR/BuiltinAttributes.h"
 #include "mlir/IR/BuiltinOps.h"
-#include "flang/Optimizer/Support/KindMapping.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "llvm/TargetParser/Host.h"
 #include <string>
 
diff --git a/flang/unittests/Optimizer/KindMappingTest.cpp b/flang/unittests/Optimizer/KindMappingTest.cpp
index 6a11d61817159..8e0d879755fb1 100644
--- a/flang/unittests/Optimizer/KindMappingTest.cpp
+++ b/flang/unittests/Optimizer/KindMappingTest.cpp
@@ -6,7 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
-#include "flang/Optimizer/Support/KindMapping.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "gtest/gtest.h"
 #include <string>
 
diff --git a/flang/unittests/Runtime/Allocatable.cpp b/flang/unittests/Runtime/Allocatable.cpp
index 8e1ec467b89f1..ed8e919320491 100644
--- a/flang/unittests/Runtime/Allocatable.cpp
+++ b/flang/unittests/Runtime/Allocatable.cpp
@@ -94,3 +94,20 @@ TEST(AllocatableTest, AllocateFromScalarSource) {
   EXPECT_EQ(*a->OffsetElement<float>(), 3.4F);
   a->Destroy();
 }
+
+TEST(AllocatableTest, DoubleAllocation) {
+  // CLASS(*), ALLOCATABLE :: r
+  // ALLOCATE(REAL::r)
+  auto r{createAllocatable(TypeCategory::Real, 4, 0)};
+  EXPECT_FALSE(r->IsAllocated());
+  EXPECT_TRUE(r->IsAllocatable());
+  RTNAME(AllocatableAllocate)(*r);
+  EXPECT_TRUE(r->IsAllocated());
+
+  // Make sure AllocatableInitIntrinsicForAllocate doesn't reset the decsriptor
+  // if it is allocated.
+  // ALLOCATE(INTEGER::r)
+  RTNAME(AllocatableInitIntrinsicForAllocate)
+  (*r, Fortran::common::TypeCategory::Integer, 4);
+  EXPECT_TRUE(r->IsAllocated());
+}
diff --git a/flang/unittests/Runtime/CMakeLists.txt b/flang/unittests/Runtime/CMakeLists.txt
index 40eff5d7d19f8..325eb5433f3a0 100644
--- a/flang/unittests/Runtime/CMakeLists.txt
+++ b/flang/unittests/Runtime/CMakeLists.txt
@@ -6,11 +6,13 @@ add_flang_unittest(FlangRuntimeTests
   CommandTest.cpp
   Complex.cpp
   CrashHandlerFixture.cpp
+  Derived.cpp
   ExternalIOTest.cpp
   Format.cpp
   Inquiry.cpp
   ListInputTest.cpp
   Matmul.cpp
+  MatmulTranspose.cpp
   MiscIntrinsic.cpp
   Namelist.cpp
   Numeric.cpp
diff --git a/flang/unittests/Runtime/Derived.cpp b/flang/unittests/Runtime/Derived.cpp
new file mode 100644
index 0000000000000..019d5e8309e4a
--- /dev/null
+++ b/flang/unittests/Runtime/Derived.cpp
@@ -0,0 +1,66 @@
+//===-- flang/unittests/Runtime/Pointer.cpp--------- -------------*- C++-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "gtest/gtest.h"
+#include "tools.h"
+#include "flang/Runtime/derived-api.h"
+#include "flang/Runtime/descriptor.h"
+
+using namespace Fortran::runtime;
+
+TEST(Derived, SameTypeAs) {
+  // INTEGER, POINTER :: i1
+  auto i1{
+      Descriptor::Create(TypeCode{Fortran::common::TypeCategory::Integer, 4}, 4,
+          nullptr, 0, nullptr, CFI_attribute_pointer)};
+  EXPECT_TRUE(RTNAME(SameTypeAs)(*i1, *i1));
+
+  auto r1{Descriptor::Create(TypeCode{Fortran::common::TypeCategory::Real, 4},
+      4, nullptr, 0, nullptr, CFI_attribute_pointer)};
+  EXPECT_FALSE(RTNAME(SameTypeAs)(*i1, *r1));
+
+  // CLASS(*), ALLOCATABLE :: a1
+  auto a1{Descriptor::Create(TypeCode{Fortran::common::TypeCategory::Real, 4},
+      4, nullptr, 0, nullptr, CFI_attribute_allocatable)};
+  a1->raw().elem_len = 0;
+  a1->raw().type = CFI_type_other;
+
+  EXPECT_FALSE(RTNAME(SameTypeAs)(*i1, *a1));
+  EXPECT_FALSE(RTNAME(SameTypeAs)(*a1, *i1));
+  EXPECT_FALSE(RTNAME(SameTypeAs)(*r1, *a1));
+
+  // CLASS(*), ALLOCATABLE :: a2
+  auto a2{Descriptor::Create(TypeCode{Fortran::common::TypeCategory::Real, 4},
+      4, nullptr, 0, nullptr, CFI_attribute_allocatable)};
+  a2->raw().elem_len = 0;
+  a2->raw().type = CFI_type_other;
+
+  EXPECT_FALSE(RTNAME(SameTypeAs)(*a1, *a2));
+
+  // CLASS(*), POINTER :: p1
+  auto p1{Descriptor::Create(TypeCode{Fortran::common::TypeCategory::Real, 4},
+      4, nullptr, 0, nullptr, CFI_attribute_pointer)};
+  p1->raw().elem_len = 0;
+  p1->raw().type = CFI_type_other;
+
+  EXPECT_FALSE(RTNAME(SameTypeAs)(*i1, *p1));
+  EXPECT_FALSE(RTNAME(SameTypeAs)(*p1, *i1));
+}
+
+TEST(Derived, ExtendsTypeOf) {
+  // CLASS(*), POINTER :: i1 - INTEGER dynamic type
+  auto i1{
+      Descriptor::Create(TypeCode{Fortran::common::TypeCategory::Integer, 4}, 4,
+          nullptr, 0, nullptr, CFI_attribute_pointer)};
+  EXPECT_TRUE(RTNAME(ExtendsTypeOf)(*i1, *i1));
+
+  // CLASS(*), POINTER :: r1 - REAL dynamic type
+  auto r1{Descriptor::Create(TypeCode{Fortran::common::TypeCategory::Real, 4},
+      4, nullptr, 0, nullptr, CFI_attribute_pointer)};
+  EXPECT_FALSE(RTNAME(SameTypeAs)(*i1, *r1));
+}
diff --git a/flang/unittests/Runtime/Format.cpp b/flang/unittests/Runtime/Format.cpp
index 970373043131a..1f55b39d59053 100644
--- a/flang/unittests/Runtime/Format.cpp
+++ b/flang/unittests/Runtime/Format.cpp
@@ -10,6 +10,7 @@
 #include "../runtime/connection.h"
 #include "../runtime/format-implementation.h"
 #include "../runtime/io-error.h"
+#include <optional>
 #include <string>
 #include <tuple>
 #include <vector>
@@ -29,7 +30,7 @@ class TestFormatContext : public IoErrorHandler {
   bool AdvanceRecord(int = 1);
   void HandleRelativePosition(std::int64_t);
   void HandleAbsolutePosition(std::int64_t);
-  void Report(const DataEdit &);
+  void Report(const std::optional<DataEdit> &);
   ResultsTy results;
   MutableModes &mutableModes() { return mutableModes_; }
   ConnectionState &GetConnectionState() { return connectionState_; }
@@ -64,25 +65,29 @@ void TestFormatContext::HandleRelativePosition(std::int64_t n) {
   }
 }
 
-void TestFormatContext::Report(const DataEdit &edit) {
-  std::string str{edit.descriptor};
-  if (edit.repeat != 1) {
-    str = std::to_string(edit.repeat) + '*' + str;
-  }
-  if (edit.variation) {
-    str += edit.variation;
-  }
-  if (edit.width) {
-    str += std::to_string(*edit.width);
-  }
-  if (edit.digits) {
-    str += "."s + std::to_string(*edit.digits);
-  }
-  if (edit.expoDigits) {
-    str += "E"s + std::to_string(*edit.expoDigits);
+void TestFormatContext::Report(const std::optional<DataEdit> &edit) {
+  if (edit) {
+    std::string str{edit->descriptor};
+    if (edit->repeat != 1) {
+      str = std::to_string(edit->repeat) + '*' + str;
+    }
+    if (edit->variation) {
+      str += edit->variation;
+    }
+    if (edit->width) {
+      str += std::to_string(*edit->width);
+    }
+    if (edit->digits) {
+      str += "."s + std::to_string(*edit->digits);
+    }
+    if (edit->expoDigits) {
+      str += "E"s + std::to_string(*edit->expoDigits);
+    }
+    // modes?
+    results.push_back(str);
+  } else {
+    results.push_back("(nullopt)"s);
   }
-  // modes?
-  results.push_back(str);
 }
 
 struct FormatTests : public CrashHandlerFixture {};
diff --git a/flang/unittests/Runtime/MatmulTranspose.cpp b/flang/unittests/Runtime/MatmulTranspose.cpp
new file mode 100644
index 0000000000000..83db1328963a6
--- /dev/null
+++ b/flang/unittests/Runtime/MatmulTranspose.cpp
@@ -0,0 +1,126 @@
+//===-- flang/unittests/Runtime/MatmulTranspose.cpp -------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "gtest/gtest.h"
+#include "tools.h"
+#include "flang/Runtime/allocatable.h"
+#include "flang/Runtime/cpp-type.h"
+#include "flang/Runtime/descriptor.h"
+#include "flang/Runtime/matmul-transpose.h"
+#include "flang/Runtime/type-code.h"
+
+using namespace Fortran::runtime;
+using Fortran::common::TypeCategory;
+
+TEST(MatmulTranspose, Basic) {
+  // X 0 1     Y 6  9     Z 6  7  8    M 0 0 1 1    V -1 -2
+  //   2 3       7 10       9 10 11      0 1 0 1
+  //   4 5       8 11
+
+  auto x{MakeArray<TypeCategory::Integer, 4>(
+      std::vector<int>{3, 2}, std::vector<std::int32_t>{0, 2, 4, 1, 3, 5})};
+  auto y{MakeArray<TypeCategory::Integer, 2>(
+      std::vector<int>{3, 2}, std::vector<std::int16_t>{6, 7, 8, 9, 10, 11})};
+  auto z{MakeArray<TypeCategory::Integer, 2>(
+      std::vector<int>{2, 3}, std::vector<std::int16_t>{6, 9, 7, 10, 8, 11})};
+  auto m{MakeArray<TypeCategory::Integer, 2>(std::vector<int>{2, 4},
+      std::vector<std::int16_t>{0, 0, 0, 1, 1, 0, 1, 1})};
+  auto v{MakeArray<TypeCategory::Integer, 8>(
+      std::vector<int>{2}, std::vector<std::int64_t>{-1, -2})};
+  StaticDescriptor<2, true> statDesc;
+  Descriptor &result{statDesc.descriptor()};
+
+  RTNAME(MatmulTranspose)(result, *x, *y, __FILE__, __LINE__);
+  ASSERT_EQ(result.rank(), 2);
+  EXPECT_EQ(result.GetDimension(0).LowerBound(), 1);
+  EXPECT_EQ(result.GetDimension(0).Extent(), 2);
+  EXPECT_EQ(result.GetDimension(1).LowerBound(), 1);
+  EXPECT_EQ(result.GetDimension(1).Extent(), 2);
+  ASSERT_EQ(result.type(), (TypeCode{TypeCategory::Integer, 4}));
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int32_t>(0), 46);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int32_t>(1), 67);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int32_t>(2), 64);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int32_t>(3), 94);
+
+  std::memset(
+      result.raw().base_addr, 0, result.Elements() * result.ElementBytes());
+  result.GetDimension(0).SetLowerBound(0);
+  result.GetDimension(1).SetLowerBound(2);
+  RTNAME(MatmulTransposeDirect)(result, *x, *y, __FILE__, __LINE__);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int32_t>(0), 46);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int32_t>(1), 67);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int32_t>(2), 64);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int32_t>(3), 94);
+  result.Destroy();
+
+  RTNAME(MatmulTranspose)(result, *z, *v, __FILE__, __LINE__);
+  ASSERT_EQ(result.rank(), 1);
+  EXPECT_EQ(result.GetDimension(0).LowerBound(), 1);
+  EXPECT_EQ(result.GetDimension(0).Extent(), 3);
+  ASSERT_EQ(result.type(), (TypeCode{TypeCategory::Integer, 8}));
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int64_t>(0), -24);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int64_t>(1), -27);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int64_t>(2), -30);
+  result.Destroy();
+
+  RTNAME(MatmulTranspose)(result, *m, *z, __FILE__, __LINE__);
+  ASSERT_EQ(result.rank(), 2);
+  ASSERT_EQ(result.GetDimension(0).LowerBound(), 1);
+  ASSERT_EQ(result.GetDimension(0).UpperBound(), 4);
+  ASSERT_EQ(result.GetDimension(1).LowerBound(), 1);
+  ASSERT_EQ(result.GetDimension(1).UpperBound(), 3);
+  ASSERT_EQ(result.type(), (TypeCode{TypeCategory::Integer, 2}));
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int16_t>(0), 0);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int16_t>(1), 9);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int16_t>(2), 6);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int16_t>(3), 15);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int16_t>(4), 0);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int16_t>(5), 10);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int16_t>(6), 7);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int16_t>(7), 17);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int16_t>(8), 0);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int16_t>(9), 11);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int16_t>(10), 8);
+  EXPECT_EQ(*result.ZeroBasedIndexedElement<std::int16_t>(11), 19);
+  result.Destroy();
+
+  // X F F    Y F T    V T F T
+  //   T F      F T
+  //   T T      F F
+  auto xLog{MakeArray<TypeCategory::Logical, 1>(std::vector<int>{3, 2},
+      std::vector<std::uint8_t>{false, true, true, false, false, true})};
+  auto yLog{MakeArray<TypeCategory::Logical, 2>(std::vector<int>{3, 2},
+      std::vector<std::uint16_t>{false, false, false, true, true, false})};
+  auto vLog{MakeArray<TypeCategory::Logical, 1>(
+      std::vector<int>{3}, std::vector<std::uint16_t>{true, false, true})};
+  RTNAME(MatmulTranspose)(result, *xLog, *yLog, __FILE__, __LINE__);
+  ASSERT_EQ(result.rank(), 2);
+  EXPECT_EQ(result.GetDimension(0).LowerBound(), 1);
+  EXPECT_EQ(result.GetDimension(0).Extent(), 2);
+  EXPECT_EQ(result.GetDimension(1).LowerBound(), 1);
+  EXPECT_EQ(result.GetDimension(1).Extent(), 2);
+  ASSERT_EQ(result.type(), (TypeCode{TypeCategory::Logical, 2}));
+  EXPECT_FALSE(
+      static_cast<bool>(*result.ZeroBasedIndexedElement<std::uint16_t>(0)));
+  EXPECT_FALSE(
+      static_cast<bool>(*result.ZeroBasedIndexedElement<std::uint16_t>(1)));
+  EXPECT_TRUE(
+      static_cast<bool>(*result.ZeroBasedIndexedElement<std::uint16_t>(2)));
+  EXPECT_FALSE(
+      static_cast<bool>(*result.ZeroBasedIndexedElement<std::uint16_t>(3)));
+
+  RTNAME(MatmulTranspose)(result, *yLog, *vLog, __FILE__, __LINE__);
+  ASSERT_EQ(result.rank(), 1);
+  EXPECT_EQ(result.GetDimension(0).LowerBound(), 1);
+  EXPECT_EQ(result.GetDimension(0).Extent(), 2);
+  ASSERT_EQ(result.type(), (TypeCode{TypeCategory::Logical, 2}));
+  EXPECT_FALSE(
+      static_cast<bool>(*result.ZeroBasedIndexedElement<std::uint16_t>(0)));
+  EXPECT_TRUE(
+      static_cast<bool>(*result.ZeroBasedIndexedElement<std::uint16_t>(1)));
+}
diff --git a/libc/.clang-tidy b/libc/.clang-tidy
index 741b63b18d6d5..5adada9a3f59a 100644
--- a/libc/.clang-tidy
+++ b/libc/.clang-tidy
@@ -9,17 +9,21 @@ CheckOptions:
   - key:             readability-identifier-naming.MemberCase
     value:           lower_case
   - key:             readability-identifier-naming.MemberIgnoredRegexp
-    value:           "^_[A-Za-z0-9_]+$"
+    value:           "_?(_[A-Za-z0-9]+)*"
   - key:             readability-identifier-naming.VariableCase
     value:           lower_case
   - key:             readability-identifier-naming.VariableIgnoredRegexp
-    value:           "^_[A-Za-z0-9_]+$"
+    value:           "_?(_[A-Za-z0-9]+)*"
   - key:             readability-identifier-naming.FunctionCase
     value:           lower_case
   - key:             readability-identifier-naming.FunctionIgnoredRegexp
-    value:           "^_[A-Za-z0-9_]+$"
+    value:           "_?(_[A-Za-z0-9]+)*"
   - key:             readability-identifier-naming.GlobalConstantCase
     value:           UPPER_CASE
+  - key:             readability-identifier-naming.LocalConstantIgnoredRegexp
+    value:           "[A-Z]+[A-Z0-9]*(_[A-Z0-9]+)*"
+  - key:             readability-identifier-naming.ClassConstantCase
+    value:           UPPER_CASE
   - key:             readability-identifier-naming.ConstexprVariableCase
     value:           UPPER_CASE
   - key:             readability-identifier-naming.GetConfigPerFile
diff --git a/libc/CMakeLists.txt b/libc/CMakeLists.txt
index 743b241ce26ad..0a9627246d0f5 100644
--- a/libc/CMakeLists.txt
+++ b/libc/CMakeLists.txt
@@ -1,5 +1,12 @@
 cmake_minimum_required(VERSION 3.13.4)
 
+# Include LLVM's cmake policies.
+if(NOT DEFINED LLVM_COMMON_CMAKE_UTILS)
+  set(LLVM_COMMON_CMAKE_UTILS ${CMAKE_CURRENT_SOURCE_DIR}/../cmake)
+endif()
+include(${LLVM_COMMON_CMAKE_UTILS}/Modules/CMakePolicy.cmake
+  NO_POLICY_SCOPE)
+
 # Default to C++17
 set(CMAKE_CXX_STANDARD 17)
 
@@ -190,7 +197,7 @@ endif()
 # of the other directories.
 # TODO: Add testing support for the libc GPU target.
 add_subdirectory(lib)
-if(LLVM_INCLUDE_TESTS AND NOT LIBC_TARGET_ARCHITECTURE_IS_GPU)
+if(LLVM_INCLUDE_TESTS)
   add_subdirectory(test)
   add_subdirectory(fuzzing)
 endif()
diff --git a/libc/cmake/modules/LLVMLibCArchitectures.cmake b/libc/cmake/modules/LLVMLibCArchitectures.cmake
index e2ae2e3af2175..27e17cd6a2aa4 100644
--- a/libc/cmake/modules/LLVMLibCArchitectures.cmake
+++ b/libc/cmake/modules/LLVMLibCArchitectures.cmake
@@ -55,6 +55,8 @@ function(get_arch_and_system_from_triple triple arch_var sys_var)
     set(target_arch "x86_64")
   elseif(target_arch MATCHES "^(powerpc|ppc)")
     set(target_arch "power")
+  elseif(target_arch MATCHES "^riscv64")
+    set(target_arch "riscv64")
   else()
     return()
   endif()
@@ -146,6 +148,8 @@ elseif(LIBC_TARGET_ARCHITECTURE STREQUAL "aarch64")
   set(LIBC_TARGET_ARCHITECTURE_IS_AARCH64 TRUE)
 elseif(LIBC_TARGET_ARCHITECTURE STREQUAL "x86_64")
   set(LIBC_TARGET_ARCHITECTURE_IS_X86 TRUE)
+elseif(LIBC_TARGET_ARCHITECTURE STREQUAL "riscv64")
+  set(LIBC_TARGET_ARCHITECTURE_IS_RISCV64 TRUE)
 else()
   message(FATAL_ERROR
           "Unsupported libc target architecture ${LIBC_TARGET_ARCHITECTURE}")
@@ -174,8 +178,8 @@ if(explicit_target_triple AND
   set(LIBC_CROSSBUILD TRUE)
   if(CMAKE_COMPILER_IS_GNUCXX)
     message(FATAL_ERROR
-            "GCC target triple and the explicity specified target triple do "
-            "not match.")
+            "GCC target triple (${libc_compiler_triple}) and the explicity "
+            "specified target triple (${explicit_target_triple}) do not match.")
   else()
     list(APPEND
          LIBC_COMPILE_OPTIONS_DEFAULT "--target=${explicit_target_triple}")
diff --git a/libc/cmake/modules/LLVMLibCCheckCpuFeatures.cmake b/libc/cmake/modules/LLVMLibCCheckCpuFeatures.cmake
index 9a18ac018d77f..adcec772938ae 100644
--- a/libc/cmake/modules/LLVMLibCCheckCpuFeatures.cmake
+++ b/libc/cmake/modules/LLVMLibCCheckCpuFeatures.cmake
@@ -5,6 +5,10 @@
 # Initialize ALL_CPU_FEATURES as empty list.
 set(ALL_CPU_FEATURES "")
 
+if(LIBC_TARGET_ARCHITECTURE_IS_GPU)
+  return()
+endif()
+
 if(${LIBC_TARGET_ARCHITECTURE_IS_X86})
   set(ALL_CPU_FEATURES SSE2 SSE4_2 AVX2 AVX512F AVX512BW FMA)
   set(LIBC_COMPILE_OPTIONS_NATIVE -march=native)
diff --git a/libc/cmake/modules/LLVMLibCObjectRules.cmake b/libc/cmake/modules/LLVMLibCObjectRules.cmake
index 652e97ae27db9..b3503a986efac 100644
--- a/libc/cmake/modules/LLVMLibCObjectRules.cmake
+++ b/libc/cmake/modules/LLVMLibCObjectRules.cmake
@@ -83,67 +83,76 @@ function(_build_gpu_objects fq_target_name internal_target_name)
   )
 
   set(include_dirs ${LIBC_BUILD_DIR}/include ${LIBC_SOURCE_DIR} ${LIBC_BUILD_DIR})
+  set(common_compile_options ${ADD_GPU_OBJ_COMPILE_OPTIONS})
+
+  foreach(add_gpu_obj_src ${ADD_GPU_OBJ_SRCS})
+    # The packaged version will be built for every target GPU architecture. We do
+    # this so we can support multiple accelerators on the same machine.
+    foreach(gpu_arch ${LIBC_GPU_ARCHITECTURES})
+      get_filename_component(src_name ${add_gpu_obj_src} NAME)
+      set(gpu_target_name ${fq_target_name}.${src_name}.${gpu_arch})
+      set(compile_options ${ADD_GPU_OBJ_COMPILE_OPTIONS})
+      # Derive the triple from the specified architecture.
+      if("${gpu_arch}" IN_LIST all_amdgpu_architectures)
+        set(gpu_target_triple "amdgcn-amd-amdhsa")
+        list(APPEND compile_options "-mcpu=${gpu_arch}")
+      elseif("${gpu_arch}" IN_LIST all_nvptx_architectures)
+        set(gpu_target_triple "nvptx64-nvidia-cuda")
+        list(APPEND compile_options "-march=${gpu_arch}")
+      else()
+        message(FATAL_ERROR "Unknown GPU architecture '${gpu_arch}'")
+      endif()
+      list(APPEND compile_options "--target=${gpu_target_triple}")
+      list(APPEND compile_options "-emit-llvm")
+
+      # Build the library for this target architecture. We always emit LLVM-IR for
+      # packaged GPU binaries.
+      add_library(${gpu_target_name}
+        EXCLUDE_FROM_ALL
+        OBJECT
+        ${add_gpu_obj_src}
+        ${ADD_GPU_OBJ_HDRS}
+      )
 
-  # The packaged version will be built for every target GPU architecture. We do
-  # this so we can support multiple accelerators on the same machine.
-  foreach(gpu_arch ${LIBC_GPU_ARCHITECTURES})
-    set(gpu_target_name ${fq_target_name}.${gpu_arch})
-    set(compile_options ${ADD_GPU_OBJ_COMPILE_OPTIONS})
-    # Derive the triple from the specified architecture.
-    if("${gpu_arch}" IN_LIST all_amdgpu_architectures)
-      set(gpu_target_triple "amdgcn-amd-amdhsa")
-      list(APPEND compile_options "-mcpu=${gpu_arch}")
-    elseif("${gpu_arch}" IN_LIST all_nvptx_architectures)
-      set(gpu_target_triple "nvptx64-nvidia-cuda")
-      list(APPEND compile_options "-march=${gpu_arch}")
-    else()
-      message(FATAL_ERROR "Unknown GPU architecture '${gpu_arch}'")
-    endif()
-    list(APPEND compile_options "--target=${gpu_target_triple}")
-    list(APPEND compile_options "-emit-llvm")
-
-    # Build the library for this target architecture. We always emit LLVM-IR for
-    # packaged GPU binaries.
-    add_library(${gpu_target_name}
-      EXCLUDE_FROM_ALL
-      OBJECT
-      ${ADD_GPU_OBJ_SRCS}
-      ${ADD_GPU_OBJ_HDRS}
-    )
-
-    target_compile_options(${gpu_target_name} PRIVATE ${compile_options})
-    target_include_directories(${gpu_target_name} PRIVATE ${include_dirs})
-    target_compile_definitions(${gpu_target_name} PRIVATE LIBC_COPT_PUBLIC_PACKAGING)
-    if(ADD_GPU_OBJ_DEPENDS)
-      add_dependencies(${gpu_target_name} ${ADD_GPU_OBJ_DEPENDS})
-    endif()
-
-    # Append this target to a list of images to package into a single binary.
-    set(input_file $<TARGET_OBJECTS:${gpu_target_name}>)
-    list(APPEND packager_images
-         --image=file=${input_file},arch=${gpu_arch},triple=${gpu_target_triple})
-    list(APPEND gpu_target_names ${gpu_target_name})
+      target_compile_options(${gpu_target_name} PRIVATE ${compile_options})
+      target_include_directories(${gpu_target_name} PRIVATE ${include_dirs})
+      target_compile_definitions(${gpu_target_name} PRIVATE LIBC_COPT_PUBLIC_PACKAGING)
+      if(ADD_GPU_OBJ_DEPENDS)
+        add_dependencies(${gpu_target_name} ${ADD_GPU_OBJ_DEPENDS})
+      endif()
+
+      # Append this target to a list of images to package into a single binary.
+      set(input_file $<TARGET_OBJECTS:${gpu_target_name}>)
+      list(APPEND packager_images
+           --image=file=${input_file},arch=${gpu_arch},triple=${gpu_target_triple})
+      list(APPEND gpu_target_names ${gpu_target_name})
+    endforeach()
+
+    # After building the target for the desired GPUs we must package the output
+    # into a fatbinary, see https://clang.llvm.org/docs/OffloadingDesign.html for
+    # more information.
+    set(packaged_target_name ${fq_target_name}.${src_name}.__gpu__)
+    set(packaged_output_name ${CMAKE_CURRENT_BINARY_DIR}/${fq_target_name}.${src_name}.gpubin)
+
+    add_custom_command(OUTPUT ${packaged_output_name}
+                       COMMAND ${LIBC_CLANG_OFFLOAD_PACKAGER}
+                               ${packager_images} -o ${packaged_output_name}
+                       DEPENDS ${gpu_target_names} ${add_gpu_obj_src} ${ADD_GPU_OBJ_HDRS}
+                       COMMENT "Packaging LLVM offloading binary")
+    add_custom_target(${packaged_target_name} DEPENDS ${packaged_output_name})
+    list(APPEND packaged_gpu_names ${packaged_target_name})
+    list(APPEND packaged_gpu_binaries ${packaged_output_name})
   endforeach()
 
-  # After building the target for the desired GPUs we must package the output
-  # into a fatbinary, see https://clang.llvm.org/docs/OffloadingDesign.html for
-  # more information.
-  set(packaged_target_name ${fq_target_name}.__gpu__)
-  set(packaged_output_name ${CMAKE_CURRENT_BINARY_DIR}/${fq_target_name}.gpubin)
-
-  add_custom_command(OUTPUT ${packaged_output_name}
-                     COMMAND ${LIBC_CLANG_OFFLOAD_PACKAGER}
-                             ${packager_images} -o ${packaged_output_name}
-                     DEPENDS ${gpu_target_names} ${ADD_GPU_OBJ_SRCS} ${ADD_GPU_OBJ_HDRS}
-                     COMMENT "Packaging LLVM offloading binary")
-  add_custom_target(${packaged_target_name} DEPENDS ${packaged_output_name} ${gpu_target_name})
-
   # We create an empty 'stub' file for the host to contain the embedded device
   # code. This will be packaged into 'libcgpu.a'.
   # TODO: In the future we will want to combine every architecture for a target
   #       into a single bitcode file and use that. For now we simply build for
   #       every single one and let the offloading linker handle it.
-  get_filename_component(stub_filename ${ADD_GPU_OBJ_SRCS} NAME)
+  string(FIND ${fq_target_name} "." last_dot_loc REVERSE)
+  math(EXPR name_loc "${last_dot_loc} + 1")
+  string(SUBSTRING ${fq_target_name} ${name_loc} -1 target_name)
+  set(stub_filename "${target_name}.cpp")
   add_custom_command(
     OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/stubs/${stub_filename}"
     COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_CURRENT_BINARY_DIR}/stubs/
@@ -161,11 +170,15 @@ function(_build_gpu_objects fq_target_name internal_target_name)
     OBJECT
     ${CMAKE_CURRENT_BINARY_DIR}/stubs/${stub_filename}
   )
-  target_compile_options(${fq_target_name} BEFORE PRIVATE ${common_compile_options}
-                         -nostdlib -Xclang -fembed-offload-object=${packaged_output_name})
+  target_compile_options(${fq_target_name} BEFORE PRIVATE
+                         ${common_compile_options} -nostdlib)
+  foreach(packaged_gpu_binary ${packaged_gpu_binaries})
+    target_compile_options(${fq_target_name} PRIVATE
+                           "SHELL:-Xclang -fembed-offload-object=${packaged_gpu_binary}")
+  endforeach()
   target_include_directories(${fq_target_name} PRIVATE ${include_dirs})
-  add_dependencies(${fq_target_name} 
-                   ${full_deps_list} ${packaged_target_name} ${stub_target_name})
+  add_dependencies(${fq_target_name}
+                   ${full_deps_list} ${packaged_gpu_names} ${stub_target_name})
 
   # We only build the internal target for a single supported architecture.
   if(LIBC_GPU_TARGET_ARCHITECTURE_IS_AMDGPU OR
@@ -198,18 +211,39 @@ endfunction()
 #       <target_name>
 #       HDRS <list of header files>
 #       SRCS <list of source files>
-#       DEPENDS <list of dependencies>
+#       [ALIAS] <If this object library is an alias for another object library.>
+#       DEPENDS <list of dependencies; Should be a single item for ALIAS libraries>
 #       COMPILE_OPTIONS <optional list of special compile options for this target>
 #       FLAGS <optional list of flags>
 function(create_object_library fq_target_name)
   cmake_parse_arguments(
     "ADD_OBJECT"
-    "NO_GPU_BUNDLE" # No optional arguments
+    "ALIAS;NO_GPU_BUNDLE" # optional arguments
     "CXX_STANDARD" # Single value arguments
     "SRCS;HDRS;COMPILE_OPTIONS;DEPENDS;FLAGS" # Multivalue arguments
     ${ARGN}
   )
 
+  get_fq_deps_list(fq_deps_list ${ADD_OBJECT_DEPENDS})
+
+  if(ADD_OBJECT_ALIAS)
+    if(ADD_OBJECT_SRCS OR ADD_OBJECT_HDRS)
+      message(FATAL_ERROR
+              "${fq_target_name}: object library alias cannot have SRCS and/or HDRS.")
+    endif()
+    list(LENGTH fq_deps_list depends_size)
+    if(NOT ${depends_size} EQUAL 1)
+      message(FATAL_ERROR
+              "${fq_targe_name}: object library alias should have exactly one DEPENDS.")
+    endif()
+    add_library(
+      ${fq_target_name}
+      ALIAS
+      ${fq_deps_list}
+    )
+    return()
+  endif()
+
   if(NOT ADD_OBJECT_SRCS)
     message(FATAL_ERROR "'add_object_library' rule requires SRCS to be specified.")
   endif()
@@ -221,7 +255,6 @@ function(create_object_library fq_target_name)
     set(internal_target_name ${fq_target_name})
   endif()
 
-  get_fq_deps_list(fq_deps_list ${ADD_OBJECT_DEPENDS})
   _get_common_compile_options(
     compile_options
     "${ADD_OBJECT_FLAGS}"
@@ -237,8 +270,7 @@ function(create_object_library fq_target_name)
       SRCS ${ADD_OBJECT_SRCS}
       HDRS ${ADD_OBJECT_HDRS}
       DEPENDS ${fq_deps_list}
-      COMPILE_OPTIONS ${common_compile_options}
-      FLAGS "${ADD_ENTRYPOINT_OBJ_FLAGS}"
+      COMPILE_OPTIONS ${compile_options}
     )
   else()
     add_library(
diff --git a/libc/cmake/modules/LLVMLibCTestRules.cmake b/libc/cmake/modules/LLVMLibCTestRules.cmake
index 3dafa4eb7a403..f28b345fd45cf 100644
--- a/libc/cmake/modules/LLVMLibCTestRules.cmake
+++ b/libc/cmake/modules/LLVMLibCTestRules.cmake
@@ -150,7 +150,7 @@ function(create_libc_unittest fq_target_name)
   )
   target_compile_options(
     ${fq_build_target_name}
-    PRIVATE ${LIBC_COMPILE_OPTIONS_DEFAULT}
+    PRIVATE -fpie ${LIBC_COMPILE_OPTIONS_DEFAULT}
   )
   if(LIBC_UNITTEST_COMPILE_OPTIONS)
     target_compile_options(
@@ -402,7 +402,6 @@ endfunction(add_libc_fuzzer)
 #     SUITE <the suite to which the test should belong>
 #     SRCS <src1.cpp> [src2.cpp ...]
 #     HDRS [hdr1.cpp ...]
-#     STARTUP <fully qualified startup system target name>
 #     DEPENDS <list of entrypoint or other object targets>
 #     ARGS <list of command line arguments to be passed to the test>
 #     ENV <list of environment variables to set before running the test>
@@ -413,14 +412,15 @@ endfunction(add_libc_fuzzer)
 # targets added with add_entrypoint_object or add_object_library.
 function(add_integration_test test_name)
   get_fq_target_name(${test_name} fq_target_name)
-  if(NOT (${LIBC_TARGET_OS} STREQUAL "linux"))
+  set(supported_targets gpu linux)
+  if(NOT (${LIBC_TARGET_OS} IN_LIST supported_targets))
     message(STATUS "Skipping ${fq_target_name} as it is not available on ${LIBC_TARGET_OS}.")
     return()
   endif()
   cmake_parse_arguments(
     "INTEGRATION_TEST"
     "" # No optional arguments
-    "SUITE;STARTUP" # Single value arguments
+    "SUITE" # Single value arguments
     "SRCS;HDRS;DEPENDS;ARGS;ENV;COMPILE_OPTIONS" # Multi-value arguments
     ${ARGN}
   )
@@ -428,28 +428,23 @@ function(add_integration_test test_name)
   if(NOT INTEGRATION_TEST_SUITE)
     message(FATAL_ERROR "SUITE not specified for ${fq_target_name}")
   endif()
-  if(NOT INTEGRATION_TEST_STARTUP)
-    message(FATAL_ERROR "The STARTUP to link to the integration test is missing.")
-  endif()
   if(NOT INTEGRATION_TEST_SRCS)
     message(FATAL_ERROR "The SRCS list for add_integration_test is missing.")
   endif()
+  if(NOT TARGET libc.startup.${LIBC_TARGET_OS}.crt1)
+    message(FATAL_ERROR "The 'crt1' target for the integration test is missing.")
+  endif()
 
   get_fq_target_name(${test_name}.libc fq_libc_target_name)
 
   get_fq_deps_list(fq_deps_list ${INTEGRATION_TEST_DEPENDS})
   list(APPEND fq_deps_list
-      # All integration tests setup TLS area and the main thread's self object.
-      # So, we need to link in the threads implementation. Likewise, the startup
-      # code also has to run init_array callbacks which potentially register
-      # their own atexit callbacks. So, link in exit and atexit also with all
-      # integration tests.
-      libc.src.__support.threads.thread
-      libc.src.stdlib.atexit
-      libc.src.stdlib.exit
-      libc.src.unistd.environ
-  )
-  list(APPEND memory_functions
+      # All integration tests use the operating system's startup object with the
+      # integration test object and need to inherit the same dependencies.
+      libc.startup.${LIBC_TARGET_OS}.crt1
+      libc.test.IntegrationTest.test
+      # We always add the memory functions objects. This is because the
+      # compiler's codegen can emit calls to the C memory functions.
       libc.src.string.bcmp
       libc.src.string.bzero
       libc.src.string.memcmp
@@ -457,9 +452,6 @@ function(add_integration_test test_name)
       libc.src.string.memmove
       libc.src.string.memset
   )
-  # We remove the memory function deps because we want to explicitly add the
-  # object files which include the public symbols of the memory functions.
-  list(REMOVE_ITEM fq_deps_list ${memory_functions})
   list(REMOVE_DUPLICATES fq_deps_list)
 
   # TODO: Instead of gathering internal object files from entrypoints,
@@ -474,13 +466,6 @@ function(add_integration_test test_name)
     endif()
     return()
   endif()
-  # We add the memory functions objects explicitly. Note that we
-  # are adding objects of the targets which contain the public
-  # C symbols. This is because compiler codegen can emit calls to
-  # the C memory functions.
-  foreach(func IN LISTS memory_functions)
-    list(APPEND link_object_files $<TARGET_OBJECTS:${func}>)
-  endforeach()
   list(REMOVE_DUPLICATES link_object_files)
 
   # Make a library of all deps
@@ -512,18 +497,36 @@ function(add_integration_test test_name)
       ${LIBC_BUILD_DIR}/include
   )
   target_compile_options(${fq_build_target_name}
-                         PRIVATE -ffreestanding ${INTEGRATION_TEST_COMPILE_OPTIONS})
+                         PRIVATE -fpie -ffreestanding ${INTEGRATION_TEST_COMPILE_OPTIONS})
+  # The GPU build requires overriding the default CMake triple and architecture.
+  if(LIBC_GPU_TARGET_ARCHITECTURE_IS_AMDGPU)
+    target_compile_options(${fq_build_target_name} PRIVATE
+                           -mcpu=${LIBC_GPU_TARGET_ARCHITECTURE} -emit-llvm
+                           --target=${LIBC_GPU_TARGET_TRIPLE})
+  elseif(LIBC_GPU_TARGET_ARCHITECTURE_IS_NVPTX)
+    target_compile_options(${fq_build_target_name} PRIVATE
+                           -march=${LIBC_GPU_TARGET_ARCHITECTURE}
+                           --target=${LIBC_GPU_TARGET_TRIPLE})
+  endif()
+
   target_link_options(${fq_build_target_name} PRIVATE -nostdlib -static)
-  target_link_libraries(${fq_build_target_name}
-                        ${INTEGRATION_TEST_STARTUP} ${fq_target_name}.__libc__
+  target_link_libraries(${fq_build_target_name} ${fq_target_name}.__libc__
+                        libc.startup.${LIBC_TARGET_OS}.crt1
                         libc.test.IntegrationTest.test)
   add_dependencies(${fq_build_target_name}
                    libc.test.IntegrationTest.test
                    ${INTEGRATION_TEST_DEPENDS})
 
+  # Tests on the GPU require an external loader utility to launch the kernel.
+  if(TARGET libc.utils.gpu.loader)
+    get_target_property(gpu_loader_exe libc.utils.gpu.loader "EXECUTABLE")
+  endif()
+
   add_custom_target(
     ${fq_target_name}
-    COMMAND ${INTEGRATION_TEST_ENV} $<TARGET_FILE:${fq_build_target_name}> ${INTEGRATION_TEST_ARGS}
+    COMMAND ${INTEGRATION_TEST_ENV}
+            $<$<BOOL:${LIBC_TARGET_ARCHITECTURE_IS_GPU}>:${gpu_loader_exe}>
+            $<TARGET_FILE:${fq_build_target_name}> ${INTEGRATION_TEST_ARGS}
     COMMENT "Running integration test ${fq_target_name}"
   )
   add_dependencies(${INTEGRATION_TEST_SUITE} ${fq_target_name})
diff --git a/libc/config/baremetal/api.td b/libc/config/baremetal/api.td
index 6d5a4a440708c..d6897fbecaac7 100644
--- a/libc/config/baremetal/api.td
+++ b/libc/config/baremetal/api.td
@@ -5,6 +5,10 @@ include "spec/stdc.td"
 def CTypeAPI : PublicAPI<"ctype.h"> {
 }
 
+def FEnvAPI : PublicAPI<"fenv.h"> {
+  let Types = ["fenv_t", "fexcept_t"];
+}
+
 def IntTypesAPI : PublicAPI<"inttypes.h"> {
   let Types = ["imaxdiv_t"];
 }
diff --git a/libc/config/baremetal/entrypoints.txt b/libc/config/baremetal/arm/entrypoints.txt
similarity index 85%
rename from libc/config/baremetal/entrypoints.txt
rename to libc/config/baremetal/arm/entrypoints.txt
index 183cf1b66a88d..8f173d5eb6a16 100644
--- a/libc/config/baremetal/entrypoints.txt
+++ b/libc/config/baremetal/arm/entrypoints.txt
@@ -85,6 +85,22 @@ set(TARGET_LIBC_ENTRYPOINTS
 )
 
 set(TARGET_LIBM_ENTRYPOINTS
+    # fenv.h entrypoints
+    libc.src.fenv.feclearexcept
+    libc.src.fenv.fedisableexcept
+    libc.src.fenv.feenableexcept
+    libc.src.fenv.fegetenv
+    libc.src.fenv.fegetexcept
+    libc.src.fenv.fegetexceptflag
+    libc.src.fenv.fegetround
+    libc.src.fenv.feholdexcept
+    libc.src.fenv.fesetenv
+    libc.src.fenv.fesetexceptflag
+    libc.src.fenv.fesetround
+    libc.src.fenv.feraiseexcept
+    libc.src.fenv.fetestexcept
+    libc.src.fenv.feupdateenv
+
     # math.h entrypoints
     libc.src.math.fabs
     libc.src.math.fabsf
diff --git a/libc/config/baremetal/headers.txt b/libc/config/baremetal/arm/headers.txt
similarity index 78%
rename from libc/config/baremetal/headers.txt
rename to libc/config/baremetal/arm/headers.txt
index cc436c7119f4a..fe7c88e922e07 100644
--- a/libc/config/baremetal/headers.txt
+++ b/libc/config/baremetal/arm/headers.txt
@@ -1,8 +1,10 @@
 set(TARGET_PUBLIC_HEADERS
     libc.include.ctype
+    libc.include.fenv
     libc.include.errno
     libc.include.inttypes
     libc.include.math
     libc.include.stdlib
     libc.include.string
+    libc.include.strings
 )
diff --git a/libc/config/darwin/arm/headers.txt b/libc/config/darwin/arm/headers.txt
index 1cf5be227aa28..7366c530304d4 100644
--- a/libc/config/darwin/arm/headers.txt
+++ b/libc/config/darwin/arm/headers.txt
@@ -6,4 +6,5 @@ set(TARGET_PUBLIC_HEADERS
     libc.include.math
     libc.include.stdlib
     libc.include.string
+    libc.include.strings
 )
diff --git a/libc/config/darwin/x86_64/headers.txt b/libc/config/darwin/x86_64/headers.txt
index 039550e1bc572..b33fa45e42080 100644
--- a/libc/config/darwin/x86_64/headers.txt
+++ b/libc/config/darwin/x86_64/headers.txt
@@ -7,4 +7,5 @@ set(TARGET_PUBLIC_HEADERS
     libc.include.math
     libc.include.stdlib
     libc.include.string
+    libc.include.strings
 )
diff --git a/libc/config/linux/aarch64/headers.txt b/libc/config/linux/aarch64/headers.txt
index 6c146a56d8bcc..bca7ab4ac8743 100644
--- a/libc/config/linux/aarch64/headers.txt
+++ b/libc/config/linux/aarch64/headers.txt
@@ -10,6 +10,7 @@ set(TARGET_PUBLIC_HEADERS
     libc.include.stdio
     libc.include.stdlib
     libc.include.string
+    libc.include.strings
     libc.include.sys_mman
     libc.include.sys_socket
     libc.include.sys_syscall
diff --git a/libc/config/linux/api.td b/libc/config/linux/api.td
index 4f51ac7d19120..253546fe83e0a 100644
--- a/libc/config/linux/api.td
+++ b/libc/config/linux/api.td
@@ -54,74 +54,11 @@ def IntTypesAPI : PublicAPI<"inttypes.h"> {
   let Types = ["imaxdiv_t"];
 }
 
-def MathErrHandlingMacro : MacroDef<"math_errhandling"> {
-  let Defn = [{
-    #ifndef math_errhandling
-    #ifdef __FAST_MATH__
-    #define math_errhandling 0
-    #elif defined __NO_MATH_ERRNO__
-    #define math_errhandling (MATH_ERREXCEPT)
-    #else
-    #define math_errhandling (MATH_ERRNO | MATH_ERREXCEPT)
-    #endif
-    #endif // math_errhandling not defined
-  }];
-}
-
-def IsFiniteMacro : MacroDef<"isfinite"> {
-  let Defn = [{
-    #define isfinite(x) __builtin_isfinite(x)
-  }];
-}
-
-def IsInfMacro : MacroDef<"isinf"> {
-  let Defn = [{
-    #define isinf(x) __builtin_isinf(x)
-  }];
-}
-
-def IsNanMacro : MacroDef<"isnan"> {
-  let Defn = [{
-    #define isnan(x) __builtin_isnan(x)
-  }];
-}
-
 def MathAPI : PublicAPI<"math.h"> {
-  let Macros = [
-    SimpleMacroDef<"MATH_ERRNO", "1">,
-    SimpleMacroDef<"MATH_ERREXCEPT", "2">,
-    MathErrHandlingMacro,
-
-    SimpleMacroDef<"HUGE_VAL", "__builtin_huge_val()">,
-    SimpleMacroDef<"INFINITY", "__builtin_inff()">,
-    SimpleMacroDef<"NAN", "__builtin_nanf(\"\")">,
-
-    SimpleMacroDef<"FP_ILOGB0", "(-__INT_MAX__ - 1)">, // INT_MIN
-    SimpleMacroDef<"FP_ILOGBNAN", "__INT_MAX__">,
-
-    IsFiniteMacro,
-    IsInfMacro,
-    IsNanMacro,
-  ];
   let Types = ["double_t", "float_t"];
 }
 
 def FenvAPI: PublicAPI<"fenv.h"> {
-  let Macros = [
-    SimpleMacroDef<"FE_DIVBYZERO", "1">,
-    SimpleMacroDef<"FE_INEXACT", "2">,
-    SimpleMacroDef<"FE_INVALID", "4">,
-    SimpleMacroDef<"FE_OVERFLOW", "8">,
-    SimpleMacroDef<"FE_UNDERFLOW", "16">,
-    SimpleMacroDef<"FE_ALL_EXCEPT", "(FE_DIVBYZERO|FE_INEXACT|FE_INVALID|FE_OVERFLOW|FE_UNDERFLOW)">,
-
-    SimpleMacroDef<"FE_DOWNWARD", "1">,
-    SimpleMacroDef<"FE_TONEAREST", "2">,
-    SimpleMacroDef<"FE_TOWARDZERO", "4">,
-    SimpleMacroDef<"FE_UPWARD", "8">,
-
-    SimpleMacroDef<"FE_DFL_ENV", "((fenv_t *)-1)">,
-  ];
   let Types = ["fenv_t", "fexcept_t"];
 }
 
@@ -171,9 +108,6 @@ def SchedAPI : PublicAPI<"sched.h"> {
 
 def SysMManAPI : PublicAPI<"sys/mman.h"> {
   let Types = ["off_t", "size_t"];
-  let Macros = [
-  ];
-
 }
 
 def SignalAPI : PublicAPI<"signal.h"> {
@@ -240,7 +174,8 @@ def DirentAPI : PublicAPI<"dirent.h"> {
 }
 
 def UniStdAPI : PublicAPI<"unistd.h"> {
-  let Types = ["__exec_argv_t", "__exec_envp_t", "off_t", "pid_t", "size_t", "ssize_t", "uid_t", "__getoptargv_t"];
+  let Types = ["__exec_argv_t", "__exec_envp_t", "off_t", "pid_t", "size_t",
+               "ssize_t", "uid_t", "__getoptargv_t"];
 }
 
 def SysRandomAPI : PublicAPI<"sys/random.h"> {
@@ -248,7 +183,8 @@ def SysRandomAPI : PublicAPI<"sys/random.h"> {
 }
 
 def SysSelectAPI : PublicAPI<"sys/select.h"> {
-  let Types = ["fd_set", "sigset_t", "suseconds_t", "time_t", "struct timespec", "struct timeval"];
+  let Types = ["fd_set", "sigset_t", "suseconds_t", "time_t", "struct timespec",
+               "struct timeval"];
 }
 
 def SysResourceAPI : PublicAPI<"sys/resource.h"> {
diff --git a/libc/config/linux/app.h b/libc/config/linux/app.h
index 160d44e606143..ffd0e01f7b5a4 100644
--- a/libc/config/linux/app.h
+++ b/libc/config/linux/app.h
@@ -35,7 +35,9 @@ struct TLSImage {
   uintptr_t align;
 };
 
-#if defined(LIBC_TARGET_ARCH_IS_X86_64) || defined(LIBC_TARGET_ARCH_IS_AARCH64)
+#if defined(LIBC_TARGET_ARCH_IS_X86_64) ||                                     \
+    defined(LIBC_TARGET_ARCH_IS_AARCH64) ||                                    \
+    defined(LIBC_TARGET_ARCH_IS_RISCV64)
 // At the language level, argc is an int. But we use uint64_t as the x86_64
 // ABI specifies it as an 8 byte value. Likewise, in the ARM64 ABI, arguments
 // are usually passed in registers.  x0 is a doubleword register, so this is
diff --git a/libc/config/linux/arm/entrypoints.txt b/libc/config/linux/arm/entrypoints.txt
index a3f6dafd8b52f..7bafe54b58363 100644
--- a/libc/config/linux/arm/entrypoints.txt
+++ b/libc/config/linux/arm/entrypoints.txt
@@ -89,6 +89,22 @@ set(TARGET_LIBC_ENTRYPOINTS
 )
 
 set(TARGET_LIBM_ENTRYPOINTS
+    # fenv.h entrypoints
+    libc.src.fenv.feclearexcept
+    libc.src.fenv.fedisableexcept
+    libc.src.fenv.feenableexcept
+    libc.src.fenv.fegetenv
+    libc.src.fenv.fegetexcept
+    libc.src.fenv.fegetexceptflag
+    libc.src.fenv.fegetround
+    libc.src.fenv.feholdexcept
+    libc.src.fenv.fesetenv
+    libc.src.fenv.fesetexceptflag
+    libc.src.fenv.fesetround
+    libc.src.fenv.feraiseexcept
+    libc.src.fenv.fetestexcept
+    libc.src.fenv.feupdateenv
+
     # math.h entrypoints
     libc.src.math.fabs
     libc.src.math.fabsf
diff --git a/libc/config/linux/arm/headers.txt b/libc/config/linux/arm/headers.txt
new file mode 100644
index 0000000000000..fe7c88e922e07
--- /dev/null
+++ b/libc/config/linux/arm/headers.txt
@@ -0,0 +1,10 @@
+set(TARGET_PUBLIC_HEADERS
+    libc.include.ctype
+    libc.include.fenv
+    libc.include.errno
+    libc.include.inttypes
+    libc.include.math
+    libc.include.stdlib
+    libc.include.string
+    libc.include.strings
+)
diff --git a/libc/config/linux/riscv64/entrypoints.txt b/libc/config/linux/riscv64/entrypoints.txt
new file mode 100644
index 0000000000000..95e3c1e5bf509
--- /dev/null
+++ b/libc/config/linux/riscv64/entrypoints.txt
@@ -0,0 +1,496 @@
+set(TARGET_LIBC_ENTRYPOINTS
+    # ctype.h entrypoints
+    libc.src.ctype.isalnum
+    libc.src.ctype.isalpha
+    libc.src.ctype.isascii
+    libc.src.ctype.isblank
+    libc.src.ctype.iscntrl
+    libc.src.ctype.isdigit
+    libc.src.ctype.isgraph
+    libc.src.ctype.islower
+    libc.src.ctype.isprint
+    libc.src.ctype.ispunct
+    libc.src.ctype.isspace
+    libc.src.ctype.isupper
+    libc.src.ctype.isxdigit
+    libc.src.ctype.toascii
+    libc.src.ctype.tolower
+    libc.src.ctype.toupper
+
+    # errno.h entrypoints
+    libc.src.errno.errno
+
+    # fcntl.h entrypoints
+    libc.src.fcntl.creat
+    libc.src.fcntl.open
+    libc.src.fcntl.openat
+
+    # sched.h entrypoints
+    libc.src.sched.sched_getaffinity
+    libc.src.sched.sched_setaffinity
+
+    # string.h entrypoints
+    libc.src.string.bcmp
+    libc.src.string.bcopy
+    libc.src.string.bzero
+    libc.src.string.memccpy
+    libc.src.string.memchr
+    libc.src.string.memcmp
+    libc.src.string.memcpy
+    libc.src.string.memmove
+    libc.src.string.mempcpy
+    libc.src.string.memrchr
+    libc.src.string.memset
+    libc.src.string.stpcpy
+    libc.src.string.stpncpy
+    libc.src.string.strcasecmp
+    libc.src.string.strcasestr
+    libc.src.string.strcat
+    libc.src.string.strchr
+    libc.src.string.strcmp
+    libc.src.string.strcoll
+    libc.src.string.strcpy
+    libc.src.string.strcspn
+    libc.src.string.strdup
+    libc.src.string.strerror
+    libc.src.string.strerror_r
+    libc.src.string.strlcat
+    libc.src.string.strlcpy
+    libc.src.string.strlen
+    libc.src.string.strncasecmp
+    libc.src.string.strncat
+    libc.src.string.strncmp
+    libc.src.string.strncpy
+    libc.src.string.strndup
+    libc.src.string.strnlen
+    libc.src.string.strpbrk
+    libc.src.string.strrchr
+    libc.src.string.strsignal
+    libc.src.string.strspn
+    libc.src.string.strstr
+    libc.src.string.strtok
+    libc.src.string.strtok_r
+
+    # inttypes.h entrypoints
+    libc.src.inttypes.imaxabs
+    libc.src.inttypes.imaxdiv
+    libc.src.inttypes.strtoimax
+    libc.src.inttypes.strtoumax
+
+    # stdlib.h entrypoints
+    libc.src.stdlib.abs
+    libc.src.stdlib.atoi
+    libc.src.stdlib.atof
+    libc.src.stdlib.atol
+    libc.src.stdlib.atoll
+    libc.src.stdlib.bsearch
+    libc.src.stdlib.div
+    libc.src.stdlib.labs
+    libc.src.stdlib.ldiv
+    libc.src.stdlib.llabs
+    libc.src.stdlib.lldiv
+    libc.src.stdlib.qsort
+    libc.src.stdlib.rand
+    libc.src.stdlib.srand
+    libc.src.stdlib.strtod
+    libc.src.stdlib.strtof
+    libc.src.stdlib.strtol
+    libc.src.stdlib.strtold
+    libc.src.stdlib.strtoll
+    libc.src.stdlib.strtoul
+    libc.src.stdlib.strtoull
+
+    # stdlib.h external entrypoints
+    libc.src.stdlib.malloc
+    libc.src.stdlib.calloc
+    libc.src.stdlib.realloc
+    libc.src.stdlib.aligned_alloc
+    libc.src.stdlib.free
+
+    # stdio.h entrypoints
+    libc.src.stdio.remove
+    libc.src.stdio.sprintf
+    libc.src.stdio.snprintf
+
+    # sys/mman.h entrypoints
+    libc.src.sys.mman.madvise
+    libc.src.sys.mman.mmap
+    libc.src.sys.mman.mprotect
+    libc.src.sys.mman.munmap
+    libc.src.sys.mman.posix_madvise
+
+    # sys/random.h entrypoints
+    libc.src.sys.random.getrandom
+
+    # sys/resource.h entrypoints
+    libc.src.sys.resource.getrlimit
+    libc.src.sys.resource.setrlimit
+
+    # sys/sendfile entrypoints
+    libc.src.sys.sendfile.sendfile
+
+    # sys/stat.h entrypoints
+    libc.src.sys.stat.chmod
+    libc.src.sys.stat.fchmod
+    libc.src.sys.stat.fchmodat
+    libc.src.sys.stat.fstat
+    libc.src.sys.stat.lstat
+    libc.src.sys.stat.mkdir
+    libc.src.sys.stat.mkdirat
+    libc.src.sys.stat.stat
+
+    # sys/utsname.h entrypoints
+    libc.src.sys.utsname.uname
+
+    # sys/wait.h entrypoints
+    libc.src.sys.wait.wait
+    libc.src.sys.wait.wait4
+    libc.src.sys.wait.waitpid
+
+    # termios.h entrypoints
+    libc.src.termios.cfgetispeed
+    libc.src.termios.cfgetospeed
+    libc.src.termios.cfsetispeed
+    libc.src.termios.cfsetospeed
+    libc.src.termios.tcgetattr
+    libc.src.termios.tcgetsid
+    libc.src.termios.tcdrain
+    libc.src.termios.tcflow
+    libc.src.termios.tcflush
+    libc.src.termios.tcsendbreak
+    libc.src.termios.tcsetattr
+
+    # unistd.h entrypoints
+    libc.src.unistd.access
+    libc.src.unistd.chdir
+    libc.src.unistd.close
+    libc.src.unistd.dup
+    libc.src.unistd.dup2
+    libc.src.unistd.dup3
+    libc.src.unistd.execve
+    libc.src.unistd.fchdir
+    libc.src.unistd.fsync
+    libc.src.unistd.ftruncate
+    libc.src.unistd.getcwd
+    libc.src.unistd.geteuid
+    libc.src.unistd.getpid
+    libc.src.unistd.getppid
+    libc.src.unistd.getuid
+    libc.src.unistd.isatty
+    libc.src.unistd.link
+    libc.src.unistd.linkat
+    libc.src.unistd.lseek
+    libc.src.unistd.pread
+    libc.src.unistd.pwrite
+    libc.src.unistd.read
+    libc.src.unistd.readlink
+    libc.src.unistd.readlinkat
+    libc.src.unistd.rmdir
+    libc.src.unistd.symlink
+    libc.src.unistd.symlinkat
+    libc.src.unistd.sysconf
+    libc.src.unistd.truncate
+    libc.src.unistd.unlink
+    libc.src.unistd.unlinkat
+    libc.src.unistd.write
+)
+
+set(TARGET_LIBM_ENTRYPOINTS
+    # fenv.h entrypoints
+    libc.src.fenv.feclearexcept
+    libc.src.fenv.fedisableexcept
+    libc.src.fenv.feenableexcept
+    libc.src.fenv.fegetenv
+    libc.src.fenv.fegetexcept
+    libc.src.fenv.fegetexceptflag
+    libc.src.fenv.fegetround
+    libc.src.fenv.feholdexcept
+    libc.src.fenv.fesetenv
+    libc.src.fenv.fesetexceptflag
+    libc.src.fenv.fesetround
+    libc.src.fenv.feraiseexcept
+    libc.src.fenv.fetestexcept
+    libc.src.fenv.feupdateenv
+
+    # math.h entrypoints
+    libc.src.math.acosf
+    libc.src.math.acoshf
+    libc.src.math.asin
+    libc.src.math.asinf
+    libc.src.math.asinhf
+    libc.src.math.atanf
+    libc.src.math.atanhf
+    libc.src.math.copysign
+    libc.src.math.copysignf
+    libc.src.math.copysignl
+    libc.src.math.ceil
+    libc.src.math.ceilf
+    libc.src.math.ceill
+    libc.src.math.coshf
+    libc.src.math.cosf
+    libc.src.math.expf
+    libc.src.math.exp10f
+    libc.src.math.exp2f
+    libc.src.math.expm1f
+    libc.src.math.fabs
+    libc.src.math.fabsf
+    libc.src.math.fabsl
+    libc.src.math.fdim
+    libc.src.math.fdimf
+    libc.src.math.fdiml
+    libc.src.math.floor
+    libc.src.math.floorf
+    libc.src.math.floorl
+    libc.src.math.fma
+    libc.src.math.fmaf
+    libc.src.math.fmin
+    libc.src.math.fminf
+    libc.src.math.fminl
+    libc.src.math.fmax
+    libc.src.math.fmaxf
+    libc.src.math.fmaxl
+    libc.src.math.fmod
+    libc.src.math.fmodf
+    libc.src.math.frexp
+    libc.src.math.frexpf
+    libc.src.math.frexpl
+    libc.src.math.hypot
+    libc.src.math.hypotf
+    libc.src.math.ilogb
+    libc.src.math.ilogbf
+    libc.src.math.ilogbl
+    libc.src.math.ldexp
+    libc.src.math.ldexpf
+    libc.src.math.ldexpl
+    libc.src.math.llrint
+    libc.src.math.llrintf
+    libc.src.math.llrintl
+    libc.src.math.llround
+    libc.src.math.llroundf
+    libc.src.math.llroundl
+    libc.src.math.log10f
+    libc.src.math.log1pf
+    libc.src.math.log2f
+    libc.src.math.logf
+    libc.src.math.logb
+    libc.src.math.logbf
+    libc.src.math.logbl
+    libc.src.math.lrint
+    libc.src.math.lrintf
+    libc.src.math.lrintl
+    libc.src.math.lround
+    libc.src.math.lroundf
+    libc.src.math.lroundl
+    libc.src.math.modf
+    libc.src.math.modff
+    libc.src.math.modfl
+    libc.src.math.nearbyint
+    libc.src.math.nearbyintf
+    libc.src.math.nearbyintl
+    libc.src.math.nextafter
+    libc.src.math.nextafterf
+    libc.src.math.nextafterl
+    libc.src.math.pow
+    libc.src.math.remainderf
+    libc.src.math.remainder
+    libc.src.math.remainderl
+    libc.src.math.remquof
+    libc.src.math.remquo
+    libc.src.math.remquol
+    libc.src.math.rint
+    libc.src.math.rintf
+    libc.src.math.rintl
+    libc.src.math.round
+    libc.src.math.roundf
+    libc.src.math.roundl
+    libc.src.math.scalbn
+    libc.src.math.scalbnf
+    libc.src.math.scalbnl
+    libc.src.math.sincosf
+    libc.src.math.sinhf
+    libc.src.math.sinf
+    libc.src.math.sqrt
+    libc.src.math.sqrtf
+    libc.src.math.sqrtl
+    libc.src.math.tanf
+    libc.src.math.tanhf
+    libc.src.math.trunc
+    libc.src.math.truncf
+    libc.src.math.truncl
+)
+
+if(LLVM_LIBC_FULL_BUILD)
+  list(APPEND TARGET_LIBC_ENTRYPOINTS
+    # assert.h entrypoints
+    libc.src.assert.__assert_fail
+
+    # network.h entrypoints
+    libc.src.network.htonl
+    libc.src.network.htons
+    libc.src.network.ntohl
+    libc.src.network.ntohs
+
+    # pthread.h entrypoints
+    libc.src.pthread.pthread_atfork
+    libc.src.pthread.pthread_attr_destroy
+    libc.src.pthread.pthread_attr_init
+    libc.src.pthread.pthread_attr_getdetachstate
+    libc.src.pthread.pthread_attr_getguardsize
+    libc.src.pthread.pthread_attr_getstack
+    libc.src.pthread.pthread_attr_getstacksize
+    libc.src.pthread.pthread_attr_setdetachstate
+    libc.src.pthread.pthread_attr_setguardsize
+    libc.src.pthread.pthread_attr_setstack
+    libc.src.pthread.pthread_attr_setstacksize
+    libc.src.pthread.pthread_create
+    libc.src.pthread.pthread_detach
+    libc.src.pthread.pthread_equal
+    libc.src.pthread.pthread_exit
+    libc.src.pthread.pthread_getname_np
+    libc.src.pthread.pthread_getspecific
+    libc.src.pthread.pthread_join
+    libc.src.pthread.pthread_key_create
+    libc.src.pthread.pthread_key_delete
+    libc.src.pthread.pthread_self
+    libc.src.pthread.pthread_setname_np
+    libc.src.pthread.pthread_mutex_destroy
+    libc.src.pthread.pthread_mutex_init
+    libc.src.pthread.pthread_mutex_lock
+    libc.src.pthread.pthread_mutex_unlock
+    libc.src.pthread.pthread_mutexattr_destroy
+    libc.src.pthread.pthread_mutexattr_init
+    libc.src.pthread.pthread_mutexattr_getpshared
+    libc.src.pthread.pthread_mutexattr_getrobust
+    libc.src.pthread.pthread_mutexattr_gettype
+    libc.src.pthread.pthread_mutexattr_setpshared
+    libc.src.pthread.pthread_mutexattr_setrobust
+    libc.src.pthread.pthread_mutexattr_settype
+    libc.src.pthread.pthread_once
+    libc.src.pthread.pthread_setspecific
+
+    # sched.h entrypoints
+    libc.src.sched.__sched_getcpucount
+
+    # stdio.h entrypoints
+    libc.src.stdio.clearerr
+    libc.src.stdio.clearerr_unlocked
+    libc.src.stdio.fclose
+    libc.src.stdio.flockfile
+    libc.src.stdio.feof
+    libc.src.stdio.feof_unlocked
+    libc.src.stdio.ferror
+    libc.src.stdio.ferror_unlocked
+    libc.src.stdio.fgetc
+    libc.src.stdio.fgetc_unlocked
+    libc.src.stdio.fgets
+    libc.src.stdio.fflush
+    libc.src.stdio.fopen
+    libc.src.stdio.fputc
+    libc.src.stdio.fputs
+    libc.src.stdio.fopencookie
+    libc.src.stdio.fread
+    libc.src.stdio.fread_unlocked
+    libc.src.stdio.fseek
+    libc.src.stdio.ftell
+    libc.src.stdio.funlockfile
+    libc.src.stdio.fwrite
+    libc.src.stdio.fwrite_unlocked
+    libc.src.stdio.fprintf
+    libc.src.stdio.getc
+    libc.src.stdio.getc_unlocked
+    libc.src.stdio.printf
+    libc.src.stdio.sscanf
+    libc.src.stdio.scanf
+    libc.src.stdio.fscanf
+    libc.src.stdio.putc
+    libc.src.stdio.putchar
+    libc.src.stdio.puts
+    libc.src.stdio.setbuf
+    libc.src.stdio.setvbuf
+    libc.src.stdio.stderr
+    libc.src.stdio.stdin
+    libc.src.stdio.stdout
+    libc.src.stdio.ungetc
+
+    # stdlib.h entrypoints
+    libc.src.stdlib._Exit
+    libc.src.stdlib.abort
+    libc.src.stdlib.atexit
+    libc.src.stdlib.exit
+    libc.src.stdlib.getenv
+
+    # signal.h entrypoints
+    libc.src.signal.raise
+    libc.src.signal.kill
+    libc.src.signal.sigaction
+    libc.src.signal.sigaltstack
+    libc.src.signal.sigdelset
+    libc.src.signal.sigaddset
+    libc.src.signal.sigemptyset
+    libc.src.signal.sigprocmask
+    libc.src.signal.sigfillset
+    libc.src.signal.signal
+
+    # spawn.h entrypoints
+    libc.src.spawn.posix_spawn
+    libc.src.spawn.posix_spawn_file_actions_addclose
+    libc.src.spawn.posix_spawn_file_actions_adddup2
+    libc.src.spawn.posix_spawn_file_actions_addopen
+    libc.src.spawn.posix_spawn_file_actions_destroy
+    libc.src.spawn.posix_spawn_file_actions_init
+
+    # threads.h entrypoints
+    libc.src.threads.call_once
+    libc.src.threads.cnd_broadcast
+    libc.src.threads.cnd_destroy
+    libc.src.threads.cnd_init
+    libc.src.threads.cnd_signal
+    libc.src.threads.cnd_wait
+    libc.src.threads.mtx_destroy
+    libc.src.threads.mtx_init
+    libc.src.threads.mtx_lock
+    libc.src.threads.mtx_unlock
+    libc.src.threads.thrd_create
+    libc.src.threads.thrd_current
+    libc.src.threads.thrd_detach
+    libc.src.threads.thrd_equal
+    libc.src.threads.thrd_exit
+    libc.src.threads.thrd_join
+    libc.src.threads.tss_create
+    libc.src.threads.tss_delete
+    libc.src.threads.tss_get
+    libc.src.threads.tss_set
+
+    # time.h entrypoints
+    libc.src.time.asctime
+    libc.src.time.asctime_r
+    libc.src.time.clock_gettime
+    libc.src.time.clock
+    libc.src.time.difftime
+    libc.src.time.gettimeofday
+    libc.src.time.gmtime
+    libc.src.time.gmtime_r
+    libc.src.time.mktime
+    libc.src.time.nanosleep
+    libc.src.time.time
+
+    # unistd.h entrypoints
+    libc.src.unistd.environ
+    libc.src.unistd.execv
+    libc.src.unistd.fork
+    libc.src.unistd.__llvm_libc_syscall
+    libc.src.unistd.getopt
+    libc.src.unistd.optarg
+    libc.src.unistd.optind
+    libc.src.unistd.optopt
+    libc.src.unistd.opterr
+
+    # sys/select.h entrypoints
+    libc.src.sys.select.select
+  )
+endif()
+
+set(TARGET_LLVMLIBC_ENTRYPOINTS
+  ${TARGET_LIBC_ENTRYPOINTS}
+  ${TARGET_LIBM_ENTRYPOINTS}
+)
diff --git a/libc/config/linux/riscv64/headers.txt b/libc/config/linux/riscv64/headers.txt
new file mode 100644
index 0000000000000..6d7e53790ec12
--- /dev/null
+++ b/libc/config/linux/riscv64/headers.txt
@@ -0,0 +1,31 @@
+set(TARGET_PUBLIC_HEADERS
+    libc.include.assert
+    libc.include.ctype
+    libc.include.errno
+    libc.include.fcntl
+    libc.include.fenv
+    libc.include.inttypes
+    libc.include.math
+    libc.include.pthread
+    libc.include.sched
+    libc.include.signal
+    libc.include.spawn
+    libc.include.stdio
+    libc.include.stdlib
+    libc.include.string
+    libc.include.strings
+    libc.include.termios
+    libc.include.threads
+    libc.include.time
+    libc.include.unistd
+
+    libc.include.sys_mman
+    libc.include.sys_random
+    libc.include.sys_resource
+    libc.include.sys_select
+    libc.include.sys_socket
+    libc.include.sys_stat
+    libc.include.sys_syscall
+    libc.include.sys_utsname
+    libc.include.sys_wait
+)
diff --git a/libc/config/linux/x86_64/headers.txt b/libc/config/linux/x86_64/headers.txt
index b2ff2df4a8736..e285105b1224a 100644
--- a/libc/config/linux/x86_64/headers.txt
+++ b/libc/config/linux/x86_64/headers.txt
@@ -15,6 +15,7 @@ set(TARGET_PUBLIC_HEADERS
     libc.include.stdio
     libc.include.stdlib
     libc.include.string
+    libc.include.strings
     libc.include.termios
     libc.include.threads
     libc.include.time
diff --git a/libc/docs/compiler_support.rst b/libc/docs/compiler_support.rst
index cc671126451bc..bdb5620802a43 100644
--- a/libc/docs/compiler_support.rst
+++ b/libc/docs/compiler_support.rst
@@ -17,8 +17,8 @@ For platforms where only ``GCC`` is natively available but maximum performance
 is required it is possible to bootstrap ``Clang`` with ``GCC`` and then use
 ``Clang`` to build the '`libc``" project.
 
-Minimum version
-===============
+Minimum supported versions
+==========================
 
  - ``Clang 11``
- - ``GCC 12.2```
+ - ``GCC 12.2``
diff --git a/libc/docs/dev/code_style.rst b/libc/docs/dev/code_style.rst
index 48197eee2763e..e29ebf99d03e2 100644
--- a/libc/docs/dev/code_style.rst
+++ b/libc/docs/dev/code_style.rst
@@ -102,3 +102,58 @@ followed:
    entrypoint is the only exceptional entrypoint on which other entrypoints
    should explicitly depend on if they set ``errno`` to indicate error
    conditions.
+
+Assertions in libc runtime code
+===============================
+
+The libc developers should, and are encouraged to, use assertions freely in
+the libc runtime code. However, the assertion should be listed via the macro
+``LIBC_ASSERT`` defined in ``src/__support/libc_assert.h``. This macro can be
+used from anywhere in the libc runtime code. Internally, all it does is to
+print the assertion expression and exit. It does not implement the semantics
+of the standard ``assert`` macro. Hence, it can be used from any where in the
+libc runtime code without causing any recursive calls or chicken-and-egg
+situations.
+
+Allocations in the libc runtime code
+====================================
+
+Some libc functions allocate memory. For example, the ``strdup`` function
+allocates new memory into which the input string is duplicated. Allocations
+are typically done by calling a function from the ``malloc`` family of
+functions. Such functions can fail and return an error value to indicate
+allocation failure. To conform to standards, the libc should handle
+allocation failures gracefully and surface the error conditions to the user
+code as appropriate. Since LLVM's libc is implemented in C++, we want
+allocations and deallocations to employ C++ operators ``new`` and ``delete``
+as they implicitly invoke constructors and destructors respectively. However,
+if we use the default ``new`` and ``delete`` operators, the libc will end up
+depending on the C++ runtime. To avoid such a dependence, and to handle
+allocation failures gracefully, we use special ``new`` and ``delete`` operators
+defined in
+`src/__support/CPP/new.h <https://github.com/llvm/llvm-project/blob/main/libc/src/__support/CPP/new.h>`_.
+Allocations and deallocations using these operators employ a pattern like
+this:
+
+.. code-block:: c++
+
+   #include "src/__support/CPP/new.h"
+
+   ...
+
+     __llvm_libc::AllocChecker ac;
+     auto *obj = new (ac) Type(...);
+     if (!ac) {
+       // handle allocator failure.
+     }
+     ...
+     delete obj;
+
+The only exception to using the above pattern is if allocating using the
+``realloc`` function is of value. In such cases, prefer to use only the
+``malloc`` family of functions for allocations and deallocations. Allocation
+failures will still need to be handled gracefully. Further, keep in mind that
+these functions do not call the constructors and destructors of the
+allocated/deallocated objects. So, use these functions carefully and only
+when it is absolutely clear that constructor and desctructor invocation is
+not required.
diff --git a/libc/docs/full_cross_build.rst b/libc/docs/full_cross_build.rst
index 2f95f5b4e8d31..431edb6e9b112 100644
--- a/libc/docs/full_cross_build.rst
+++ b/libc/docs/full_cross_build.rst
@@ -17,7 +17,8 @@ x86_64 *host*.
 Configure the full cross build of the libc
 ==========================================
 
-Below is a simple recipe to configure the libc for a cross build.
+Below is a simple recipe to configure the libc for a cross build.  In this,
+we've set Ninja as the generator, and are building the full libc.
 
 .. code-block:: sh
 
@@ -25,10 +26,11 @@ Below is a simple recipe to configure the libc for a cross build.
   $> mkdir build
   $> cd build
   $> cmake ../llvm  \
-     -G Ninja \ # Generator
-     -DLLVM_ENABLE_PROJECTS=libc  \ # Enable the libc project
-     -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++  \
-     -DLLVM_LIBC_FULL_BUILD=ON   \ # We are building the full libc
+     -G Ninja \
+     -DLLVM_ENABLE_PROJECTS=libc  \
+     -DCMAKE_C_COMPILER=clang \
+     -DCMAKE_CXX_COMPILER=clang++  \
+     -DLLVM_LIBC_FULL_BUILD=ON \
      -DLIBC_TARGET_TRIPLE=<Your target triple>
 
 We will go over the special options passed to the ``cmake`` command above.
diff --git a/libc/docs/full_host_build.rst b/libc/docs/full_host_build.rst
index 835c009c59585..709ba70b22de4 100644
--- a/libc/docs/full_host_build.rst
+++ b/libc/docs/full_host_build.rst
@@ -30,24 +30,32 @@ Configure the full libc build
 ===============================
 
 Below is the list of commands for a simple recipe to build and install the
-libc components along with other components of an LLVM only toolchain.
+libc components along with other components of an LLVM only toolchain.  In this
+we've set the Ninja generator, enabled a full compiler suite, set the build
+type to "Debug", and enabled the Scudo allocator.  The build also tells clang
+to use the freshly built lld and compiler-rt.
 
 .. code-block:: sh
 
    $> cd llvm-project  # The llvm-project checkout
    $> mkdir build
    $> cd build
+   $> SYSROOT=/path/to/sysroot # Remember to set this!
    $> cmake ../llvm  \
-      -G Ninja  \ # Generator
-      -DLLVM_ENABLE_PROJECTS="clang;libc;lld;compiler-rt"   \ # Enabled projects
-      -DCMAKE_BUILD_TYPE=<Debug|Release>  \ # Select build type
-      -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \
-      -DLLVM_LIBC_FULL_BUILD=ON      \  # We want the full libc
-      -DLLVM_LIBC_INCLUDE_SCUDO=ON   \  # Include Scudo in the libc
+      -G Ninja  \
+      -DLLVM_ENABLE_PROJECTS="clang;libc;lld;compiler-rt"   \
+      -DCMAKE_BUILD_TYPE=Debug  \
+      -DCMAKE_C_COMPILER=clang \
+      -DCMAKE_CXX_COMPILER=clang++ \
+      -DLLVM_LIBC_FULL_BUILD=ON \
+      -DLLVM_LIBC_INCLUDE_SCUDO=ON \
       -DCOMPILER_RT_BUILD_SCUDO_STANDALONE_WITH_LLVM_LIBC=ON \
       -DCOMPILER_RT_BUILD_GWP_ASAN=OFF                       \
       -DCOMPILER_RT_SCUDO_STANDALONE_BUILD_SHARED=OFF        \
-      -DCMAKE_INSTALL_PREFIX=<SYSROOT>  # Specify a sysroot directory
+      -DCLANG_DEFAULT_LINKER=lld \
+      -DCLANG_DEFAULT_RTLIB=compiler-rt \
+      -DDEFAULT_SYSROOT=$SYSROOT \
+      -DCMAKE_INSTALL_PREFIX=$SYSROOT
 
 We will go over some of the special options passed to the ``cmake`` command
 above.
@@ -62,8 +70,8 @@ above.
   So, when building the full libc, we should specify that we want to include
   Scudo in the libc. Since the libc currently only supports static linking, we
   also specify that we do not want to build the Scudo shared library.
-* **The install prefix** - This is the path to the tool chain install directory.
-  This is the directory where you intend to set up the sysroot.
+* **Default sysroot and install prefix** - This is the path to the tool chain
+  install directory.  This is the directory where you intend to set up the sysroot.
 
 Build and install
 =================
@@ -73,11 +81,11 @@ install the libc, clang (and its support libraries and builtins), lld and
 compiler-rt, with the following command:
 
 .. code-block:: sh
-   
+
    $> ninja install-clang install-builtins install-compiler-rt  \
       install-core-resource-headers install-libc install-lld
 
-Once the above command completes successfully, the ``<SYSROOT>`` directory you
+Once the above command completes successfully, the ``$SYSROOT`` directory you
 have specified with the CMake configure step above will contain a full LLVM-only
 toolchain with which you can build practical/real-world C applications. See
 `<https://github.com/llvm/llvm-project/tree/main/libc/examples>`_ for examples
@@ -87,5 +95,29 @@ Linux Headers
 =============
 
 If you are using the full libc on Linux, then you will also need to install
-Linux headers in your sysroot. It is left to the reader to figure out the best
-way to install Linux headers on the system they want to use the full libc on.
+Linux headers in your sysroot.  The way to do this varies per system.
+
+These instructions should work on a Debian-based x86_64 system:
+
+.. code-block:: sh
+
+   $> apt download linux-libc-dev
+   $> dpkg -x linux-libc-dev*deb .
+   $> mv usr/include/* /path/to/sysroot/include
+   $> rm -rf usr linux-libc-dev*deb
+   $> ln -s x86_64-linux-gnu/asm ~/Programming/sysroot/include/asm
+
+Using your newly built libc
+===========================
+
+You can now use your newly built libc nearly like you would use any compiler
+invocation:
+
+.. code-block:: sh
+
+   $> /path/to/sysroot/bin/clang -static main.c
+
+.. warning::
+   Because the libc does not yet support dynamic linking, the -static parameter
+   must be added to all clang invocations.
+
diff --git a/libc/docs/index.rst b/libc/docs/index.rst
index 95ba7d25837d4..90422617403e6 100644
--- a/libc/docs/index.rst
+++ b/libc/docs/index.rst
@@ -59,6 +59,7 @@ stages there is no ABI stability in any form.
    :maxdepth: 1
    :caption: Status
 
+   compiler_support
    date_and_time
    math/index.rst
    strings
@@ -69,7 +70,6 @@ stages there is no ABI stability in any form.
    :maxdepth: 1
    :caption: Development
 
-   compiler_support
    build_and_test
    dev/index.rst
    porting
@@ -83,5 +83,6 @@ stages there is no ABI stability in any form.
    Source Code <https://github.com/llvm/llvm-project/tree/main/libc>
    Bug Reports <https://github.com/llvm/llvm-project/labels/libc>
    Discourse <https://discourse.llvm.org/c/runtimes/libc>
+   Join the Discord <https://discord.gg/xS7Z362>
    Discord Channel <https://discord.com/channels/636084430946959380/636732994891284500>
    Buildbot <https://lab.llvm.org/buildbot/#/builders?tags=libc>
diff --git a/libc/docs/stdio.rst b/libc/docs/stdio.rst
index b0ffd67d69518..fdb796513d8fc 100644
--- a/libc/docs/stdio.rst
+++ b/libc/docs/stdio.rst
@@ -23,10 +23,10 @@ convert either to or from those arguments. These functions are the current focus
 (owner: michaelrj).
 
 =============  =========
-Function_Name  Available
+Function Name  Available
 =============  =========
-\*printf       WIP
-\*scanf
+\*printf       Mostly
+\*scanf        |check|
 =============  =========
 
 ``FILE`` Access
@@ -37,15 +37,15 @@ I/O stream, often used to represent a file on the host's hard drive. Currently
 the ``FILE`` object is only available on linux.
 
 =============  =========
-Function_Name  Available
+Function Name  Available
 =============  =========
 fopen          |check|
 freopen
 fclose         |check|
 fflush         |check|
-setbuf
-setvbuf
-ftell
+setbuf         |check|
+setvbuf        |check|
+ftell          |check|
 fgetpos
 fseek          |check|
 fsetpos
@@ -79,15 +79,15 @@ The ``gets`` function was removed in C11 for having no bounds checking and
 therefor being impossible to use safely.
 
 =============  =========
-Function_Name  Available
+Function Name  Available
 =============  =========
-(f)getc
-fgets
+(f)getc        |check|
+fgets          |check|
 getchar
 fread          |check|
-(f)putc
+(f)putc        |check|
 (f)puts        |check|
-putchar
+putchar        |check|
 fwrite         |check|
-ungetc
+ungetc         |check|
 =============  =========
diff --git a/libc/include/CMakeLists.txt b/libc/include/CMakeLists.txt
index fc8e2bc7b4933..d058cb51db0a1 100644
--- a/libc/include/CMakeLists.txt
+++ b/libc/include/CMakeLists.txt
@@ -24,8 +24,8 @@ add_gen_header(
   GEN_HDR dirent.h
   DEPENDS
     .llvm_libc_common_h
-    .llvm-libc-types.DIR
     .llvm-libc-types.ino_t
+    .llvm-libc-types.DIR
     .llvm-libc-types.struct_dirent
 )
 
@@ -45,6 +45,7 @@ add_gen_header(
   GEN_HDR fenv.h
   DEPENDS
     .llvm_libc_common_h
+    .llvm-libc-macros.fenv_macros
     .llvm-libc-types.fenv_t
     .llvm-libc-types.fexcept_t
 )
@@ -64,6 +65,7 @@ add_gen_header(
   GEN_HDR math.h
   DEPENDS
     .llvm_libc_common_h
+    .llvm-libc-macros.math_macros
     .llvm-libc-types.double_t
     .llvm-libc-types.float_t
 )
@@ -122,12 +124,12 @@ add_gen_header(
   DEPENDS
     .llvm_libc_common_h
     .llvm-libc-macros.time_macros
-    .llvm-libc-types.time_t
     .llvm-libc-types.clock_t
-    .llvm-libc-types.clockid_t
+    .llvm-libc-types.time_t
     .llvm-libc-types.struct_tm
     .llvm-libc-types.struct_timespec
     .llvm-libc-types.struct_timeval
+    .llvm-libc-types.clockid_t
 )
 
 add_gen_header(
@@ -137,11 +139,11 @@ add_gen_header(
   DEPENDS
     .llvm_libc_common_h
     .llvm-libc-types.__call_once_func_t
+    .llvm-libc-types.once_flag
     .llvm-libc-types.cnd_t
     .llvm-libc-types.mtx_t
-    .llvm-libc-types.once_flag
-    .llvm-libc-types.thrd_start_t
     .llvm-libc-types.thrd_t
+    .llvm-libc-types.thrd_start_t
     .llvm-libc-types.tss_t
     .llvm-libc-types.tss_dtor_t
 )
@@ -164,11 +166,12 @@ add_gen_header(
     ../config/${LIBC_TARGET_OS}/signal.h.in
   DEPENDS
     .llvm-libc-macros.signal_macros
-    .llvm-libc-types.__sighandler_t
-    .llvm-libc-types.sigset_t
     .llvm-libc-types.sig_atomic_t
-    .llvm-libc-types.stack_t
+    .llvm-libc-types.sigset_t
     .llvm-libc-types.struct_sigaction
+    .llvm-libc-types.union_sigval
+    .llvm-libc-types.siginfo_t
+    .llvm-libc-types.stack_t
     .llvm-libc-types.pid_t
 )
 
@@ -180,10 +183,9 @@ add_gen_header(
     .llvm_libc_common_h
     .llvm-libc-macros.file_seek_macros
     .llvm-libc-macros.stdio_macros
-    .llvm-libc-types.cookie_io_functions_t
-    .llvm-libc-types.FILE
-    .llvm-libc-types.off_t
     .llvm-libc-types.size_t
+    .llvm-libc-types.FILE
+    .llvm-libc-types.cookie_io_functions_t
 )
 
 add_gen_header(
@@ -193,12 +195,12 @@ add_gen_header(
   DEPENDS
     .llvm_libc_common_h
     .llvm-libc-macros.stdlib_macros
-    .llvm-libc-types.__bsearchcompare_t
-    .llvm-libc-types.__qsortcompare_t
     .llvm-libc-types.div_t
     .llvm-libc-types.ldiv_t
     .llvm-libc-types.lldiv_t
     .llvm-libc-types.size_t
+    .llvm-libc-types.__bsearchcompare_t
+    .llvm-libc-types.__qsortcompare_t
     .llvm-libc-types.__atexithandler_t
 )
 
@@ -227,15 +229,15 @@ add_gen_header(
   DEPENDS
     .llvm_libc_common_h
     .llvm-libc-types.__atfork_callback_t
+    .llvm-libc-types.__pthread_once_func_t
     .llvm-libc-types.__pthread_start_t
     .llvm-libc-types.__pthread_tss_dtor_t
     .llvm-libc-types.pthread_attr_t
-    .llvm-libc-types.pthread_key_t
     .llvm-libc-types.pthread_mutex_t
     .llvm-libc-types.pthread_mutexattr_t
-    .llvm-libc-types.pthread_once_t
-    .llvm-libc-types.__pthread_once_func_t
     .llvm-libc-types.pthread_t
+    .llvm-libc-types.pthread_key_t
+    .llvm-libc-types.pthread_once_t
 )
 
 add_gen_header(
@@ -244,8 +246,10 @@ add_gen_header(
   GEN_HDR sched.h
   DEPENDS
     .llvm_libc_common_h
-    .llvm-libc-types.cpu_set_t
     .llvm-libc-macros.sched_macros
+    .llvm-libc-types.pid_t
+    .llvm-libc-types.size_t
+    .llvm-libc-types.cpu_set_t
 )
 
 add_gen_header(
@@ -289,9 +293,10 @@ add_gen_header(
   GEN_HDR sys/mman.h
   DEPENDS
     .llvm_libc_common_h
+    .llvm-libc-macros.sys_mman_macros
     .llvm-libc-types.off_t
+    .llvm-libc-types.size_t
     .llvm-libc-types.ssize_t
-    .llvm-libc-macros.sys_mman_macros
 )
 
 add_gen_header(
@@ -309,8 +314,8 @@ add_gen_header(
   DEPENDS
     .llvm_libc_common_h
     .llvm-libc-macros.sys_random_macros
-    .llvm-libc-types.ssize_t
     .llvm-libc-types.size_t
+    .llvm-libc-types.ssize_t
 )
 
 add_gen_header(
@@ -332,6 +337,16 @@ add_gen_header(
     .llvm_libc_common_h
     .llvm-libc-macros.sys_stat_macros
     .llvm-libc-types.mode_t
+    .llvm-libc-types.dev_t
+    .llvm-libc-types.ino_t
+    .llvm-libc-types.nlink_t
+    .llvm-libc-types.uid_t
+    .llvm-libc-types.gid_t
+    .llvm-libc-types.off_t
+    .llvm-libc-types.struct_timespec
+    .llvm-libc-types.struct_timeval
+    .llvm-libc-types.blksize_t
+    .llvm-libc-types.blkcnt_t
     .llvm-libc-types.struct_stat
 )
 
@@ -344,11 +359,10 @@ add_gen_header(
     .llvm-libc-macros.sys_select_macros
     .llvm-libc-types.fd_set
     .llvm-libc-types.sigset_t
-    .llvm-libc-types.struct_timespec
-    .llvm-libc-types.struct_timeval
     .llvm-libc-types.suseconds_t
     .llvm-libc-types.time_t
-    .llvm-libc-types.ssize_t
+    .llvm-libc-types.struct_timespec
+    .llvm-libc-types.struct_timeval
 )
 
 add_gen_header(
@@ -358,6 +372,7 @@ add_gen_header(
   DEPENDS
     .llvm_libc_common_h
     .llvm-libc-types.off_t
+    .llvm-libc-types.size_t
     .llvm-libc-types.ssize_t
 )
 
@@ -447,6 +462,7 @@ add_gen_header(
     .llvm_libc_common_h
     .llvm-libc-macros.termios_macros
     .llvm-libc-types.cc_t
+    .llvm-libc-types.pid_t
     .llvm-libc-types.speed_t
     .llvm-libc-types.struct_termios
     .llvm-libc-types.tcflag_t
diff --git a/libc/include/fenv.h.def b/libc/include/fenv.h.def
index 489756cf63259..f131a44914abf 100644
--- a/libc/include/fenv.h.def
+++ b/libc/include/fenv.h.def
@@ -10,6 +10,7 @@
 #define LLVM_LIBC_FENV_H
 
 #include <__llvm-libc-common.h>
+#include <llvm-libc-macros/fenv-macros.h>
 
 %%public_api()
 
diff --git a/libc/include/llvm-libc-macros/CMakeLists.txt b/libc/include/llvm-libc-macros/CMakeLists.txt
index a723cc86d2831..74a5977641f48 100644
--- a/libc/include/llvm-libc-macros/CMakeLists.txt
+++ b/libc/include/llvm-libc-macros/CMakeLists.txt
@@ -20,12 +20,24 @@ add_header(
     .linux.fcntl_macros
 )
 
+add_header(
+  fenv_macros
+  HDR
+    fenv-macros.h
+)
+
 add_header(
   file_seek_macros
   HDR
     file-seek-macros.h
 )
 
+add_header(
+  math_macros
+  HDR
+    math-macros.h
+)
+
 add_header(
   sched_macros
   HDR
diff --git a/libc/include/llvm-libc-macros/fenv-macros.h b/libc/include/llvm-libc-macros/fenv-macros.h
new file mode 100644
index 0000000000000..a73ebf41cbb36
--- /dev/null
+++ b/libc/include/llvm-libc-macros/fenv-macros.h
@@ -0,0 +1,27 @@
+//===-- Definition of macros from fenv.h ----------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef __LLVM_LIBC_MACROS_FENV_MACROS_H
+#define __LLVM_LIBC_MACROS_FENV_MACROS_H
+
+#define FE_DIVBYZERO 1
+#define FE_INEXACT 2
+#define FE_INVALID 4
+#define FE_OVERFLOW 8
+#define FE_UNDERFLOW 16
+#define FE_ALL_EXCEPT                                                          \
+  (FE_DIVBYZERO | FE_INEXACT | FE_INVALID | FE_OVERFLOW | FE_UNDERFLOW)
+
+#define FE_DOWNWARD 1
+#define FE_TONEAREST 2
+#define FE_TOWARDZERO 4
+#define FE_UPWARD 8
+
+#define FE_DFL_ENV ((fenv_t *)-1)
+
+#endif // __LLVM_LIBC_MACROS_FENV_MACROS_H
diff --git a/libc/include/llvm-libc-macros/linux/signal-macros.h b/libc/include/llvm-libc-macros/linux/signal-macros.h
index 069a233477979..4dc39dec47de8 100644
--- a/libc/include/llvm-libc-macros/linux/signal-macros.h
+++ b/libc/include/llvm-libc-macros/linux/signal-macros.h
@@ -82,6 +82,9 @@
 #elif defined(__aarch64__)
 #define MINSIGSTKSZ 5120
 #define SIGSTKSZ 16384
+#elif defined(__riscv)
+#define MINSIGSTKSZ 2048
+#define SIGSTKSZ 8192
 #else
 #error "Signal stack sizes not defined for your platform."
 #endif
diff --git a/libc/include/llvm-libc-macros/math-macros.h b/libc/include/llvm-libc-macros/math-macros.h
new file mode 100644
index 0000000000000..136670e665e6b
--- /dev/null
+++ b/libc/include/llvm-libc-macros/math-macros.h
@@ -0,0 +1,34 @@
+//===-- Definition of macros from math.h ----------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef __LLVM_LIBC_MACROS_MATH_MACROS_H
+#define __LLVM_LIBC_MACROS_MATH_MACROS_H
+
+#define MATH_ERRNO 1
+#define MATH_ERREXCEPT 2
+
+#define HUGE_VAL __builtin_huge_val()
+#define INFINITY __builtin_inf()
+#define NAN __builtin_nanf("")
+
+#define FP_ILOGB0 (-__INT_MAX__ - 1)
+#define FP_ILOGBNAN __INT_MAX__
+
+#define isfinite(x) __builtin_isfinite(x)
+#define isinf(x) __builtin_isinf(x)
+#define isnan(x) __builtin_isnan(x)
+
+#ifdef __FAST_MATH__
+#define math_errhandling 0
+#elif defined __NO_MATH_ERRNO__
+#define math_errhandling (MATH_ERREXCEPT)
+#else
+#define math_errhandling (MATH_ERRNO | MATH_ERREXCEPT)
+#endif
+
+#endif // __LLVM_LIBC_MACROS_MATH_MACROS_H
diff --git a/libc/include/llvm-libc-types/fenv_t.h b/libc/include/llvm-libc-types/fenv_t.h
index a95e08179cceb..6d2bea74560db 100644
--- a/libc/include/llvm-libc-types/fenv_t.h
+++ b/libc/include/llvm-libc-types/fenv_t.h
@@ -14,12 +14,19 @@ typedef struct {
   unsigned char __control_word[4];
   unsigned char __status_word[4];
 } fenv_t;
-#endif
-#ifdef __x86_64__
+#elif defined(__x86_64__)
 typedef struct {
   unsigned char __x86_status[28];
   unsigned char __mxcsr[4];
 } fenv_t;
+#elif defined(__arm__) || defined(_M_ARM)
+typedef struct {
+  unsigned int __fpscr;
+} fenv_t;
+#elif defined(__riscv)
+typedef unsigned int fenv_t;
+#else
+#error "fenv_t not defined for your platform"
 #endif
 
 #endif // __LLVM_LIBC_TYPES_FENV_T_H__
diff --git a/libc/include/math.h.def b/libc/include/math.h.def
index 5b27c33cdf275..813bb72c9b45d 100644
--- a/libc/include/math.h.def
+++ b/libc/include/math.h.def
@@ -10,6 +10,7 @@
 #define LLVM_LIBC_MATH_H
 
 #include <__llvm-libc-common.h>
+#include <llvm-libc-macros/math-macros.h>
 
 %%public_api()
 
diff --git a/libc/src/__support/CMakeLists.txt b/libc/src/__support/CMakeLists.txt
index 8329a80d85fb9..e4eb354aefcd3 100644
--- a/libc/src/__support/CMakeLists.txt
+++ b/libc/src/__support/CMakeLists.txt
@@ -204,6 +204,7 @@ add_header_library(
 add_subdirectory(FPUtil)
 add_subdirectory(OSUtil)
 add_subdirectory(StringUtil)
+add_subdirectory(RPC)
 
 # Thread support is used by other "File". So, we add the "threads"
 # before "File".
diff --git a/libc/src/__support/CPP/atomic.h b/libc/src/__support/CPP/atomic.h
index 5ef7f5e09eac2..b0e90e32dadd2 100644
--- a/libc/src/__support/CPP/atomic.h
+++ b/libc/src/__support/CPP/atomic.h
@@ -9,6 +9,8 @@
 #ifndef LLVM_LIBC_SRC_SUPPORT_CPP_ATOMIC_H
 #define LLVM_LIBC_SRC_SUPPORT_CPP_ATOMIC_H
 
+#include "src/__support/macros/attributes.h"
+
 #include "type_traits.h"
 
 namespace __llvm_libc {
@@ -92,6 +94,11 @@ template <typename T> struct Atomic {
   void set(T rhs) { val = rhs; }
 };
 
+// Issue a thread fence with the given memory ordering.
+LIBC_INLINE void atomic_thread_fence(MemoryOrder mem_ord) {
+  __atomic_thread_fence(int(mem_ord));
+}
+
 } // namespace cpp
 } // namespace __llvm_libc
 
diff --git a/libc/src/__support/FPUtil/CMakeLists.txt b/libc/src/__support/FPUtil/CMakeLists.txt
index 6ccb3ba511c37..b737aad1093d0 100644
--- a/libc/src/__support/FPUtil/CMakeLists.txt
+++ b/libc/src/__support/FPUtil/CMakeLists.txt
@@ -3,7 +3,6 @@ add_header_library(
   HDRS
     FEnvImpl.h
   DEPENDS
-    libc.include.errno
     libc.include.fenv
     libc.include.math
     libc.src.__support.macros.attributes
@@ -52,7 +51,6 @@ add_header_library(
     libc.src.__support.CPP.type_traits
     libc.src.__support.common
     libc.include.math
-    libc.include.errno
     libc.src.errno.errno
 )
 
@@ -80,7 +78,6 @@ add_header_library(
     libc.src.__support.common
     libc.src.__support.macros.optimization
     libc.include.math
-    libc.include.errno
     libc.src.errno.errno
 )
 
diff --git a/libc/src/__support/FPUtil/FEnvImpl.h b/libc/src/__support/FPUtil/FEnvImpl.h
index cfee3ef8a996d..ee2bd16e821d9 100644
--- a/libc/src/__support/FPUtil/FEnvImpl.h
+++ b/libc/src/__support/FPUtil/FEnvImpl.h
@@ -11,8 +11,8 @@
 
 #include "src/__support/macros/attributes.h" // LIBC_INLINE
 #include "src/__support/macros/properties/architectures.h"
+#include "src/errno/libc_errno.h"
 
-#include <errno.h>
 #include <fenv.h>
 #include <math.h>
 
@@ -28,6 +28,10 @@
 // the apple condition here should be removed.
 #elif defined(LIBC_TARGET_ARCH_IS_X86) && !defined(__APPLE__)
 #include "x86_64/FEnvImpl.h"
+#elif defined(LIBC_TARGET_ARCH_IS_ARM)
+#include "arm/FEnvImpl.h"
+#elif defined(LIBC_TARGET_ARCH_IS_RISCV64)
+#include "riscv64/FEnvImpl.h"
 #else
 
 namespace __llvm_libc::fputil {
@@ -42,6 +46,10 @@ LIBC_INLINE int set_except(int) { return 0; }
 
 LIBC_INLINE int raise_except(int) { return 0; }
 
+LIBC_INLINE int enable_except(int) { return 0; }
+
+LIBC_INLINE int disable_except(int) { return 0; }
+
 LIBC_INLINE int get_round() { return FE_TONEAREST; }
 
 LIBC_INLINE int set_round(int) { return 0; }
@@ -69,7 +77,7 @@ LIBC_INLINE int raise_except_if_required(int excepts) {
 
 LIBC_INLINE void set_errno_if_required(int err) {
   if (math_errhandling & MATH_ERRNO)
-    errno = err;
+    libc_errno = err;
 }
 
 } // namespace __llvm_libc::fputil
diff --git a/libc/src/__support/FPUtil/FMA.h b/libc/src/__support/FPUtil/FMA.h
index 05df9cb52668b..79a465a5be45a 100644
--- a/libc/src/__support/FPUtil/FMA.h
+++ b/libc/src/__support/FPUtil/FMA.h
@@ -18,6 +18,8 @@
 #include "x86_64/FMA.h"
 #elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
 #include "aarch64/FMA.h"
+#elif defined(LIBC_TARGET_ARCH_IS_RISCV64)
+#include "riscv64/FMA.h"
 #endif
 
 #else
diff --git a/libc/src/__support/FPUtil/NearestIntegerOperations.h b/libc/src/__support/FPUtil/NearestIntegerOperations.h
index 968e7d499ae27..8265ea1cbb3e5 100644
--- a/libc/src/__support/FPUtil/NearestIntegerOperations.h
+++ b/libc/src/__support/FPUtil/NearestIntegerOperations.h
@@ -15,7 +15,6 @@
 #include "src/__support/CPP/type_traits.h"
 #include "src/__support/common.h"
 
-#include <errno.h>
 #include <math.h>
 
 namespace __llvm_libc {
@@ -238,10 +237,8 @@ LIBC_INLINE I rounded_float_to_signed_integer(F x) {
   constexpr I INTEGER_MAX = -(INTEGER_MIN + 1);
   FPBits<F> bits(x);
   auto set_domain_error_and_raise_invalid = []() {
-    if (math_errhandling & MATH_ERRNO)
-      errno = EDOM;
-    if (math_errhandling & MATH_ERREXCEPT)
-      raise_except(FE_INVALID);
+    set_errno_if_required(EDOM);
+    raise_except_if_required(FE_INVALID);
   };
 
   if (bits.is_inf_or_nan()) {
diff --git a/libc/src/__support/FPUtil/XFloat.h b/libc/src/__support/FPUtil/XFloat.h
index d76fc5b58dcde..b3f8f44badd33 100644
--- a/libc/src/__support/FPUtil/XFloat.h
+++ b/libc/src/__support/FPUtil/XFloat.h
@@ -19,7 +19,7 @@ namespace fputil {
 template <size_t Precision> class XFloat {
   static constexpr uint64_t OneMask = (uint64_t(1) << 63);
   UInt<Precision> man;
-  static constexpr uint64_t WordCount = Precision / 64;
+  static constexpr uint64_t WORDCOUNT = Precision / 64;
   int exp;
 
   size_t bit_width(uint64_t x) {
@@ -35,25 +35,25 @@ template <size_t Precision> class XFloat {
 
 public:
   XFloat() : exp(0) {
-    for (int i = 0; i < WordCount; ++i)
+    for (int i = 0; i < WORDCOUNT; ++i)
       man[i] = 0;
   }
 
   XFloat(const XFloat &other) : exp(other.exp) {
-    for (int i = 0; i < WordCount; ++i)
+    for (int i = 0; i < WORDCOUNT; ++i)
       man[i] = other.man[i];
   }
 
   explicit XFloat(double x) {
     auto xn = NormalFloat<double>(x);
     exp = xn.exponent;
-    man[WordCount - 1] = xn.mantissa << 11;
-    for (int i = 0; i < WordCount - 1; ++i)
+    man[WORDCOUNT - 1] = xn.mantissa << 11;
+    for (int i = 0; i < WORDCOUNT - 1; ++i)
       man[i] = 0;
   }
 
   XFloat(int e, const UInt<Precision> &bits) : exp(e) {
-    for (size_t i = 0; i < WordCount; ++i)
+    for (size_t i = 0; i < WORDCOUNT; ++i)
       man[i] = bits[i];
   }
 
@@ -65,7 +65,7 @@ template <size_t Precision> class XFloat {
     size_t carry_width = bit_width(carry);
     carry_width = (carry_width == 64 ? 64 : 63);
     man.shift_right(carry_width);
-    man[WordCount - 1] = man[WordCount - 1] + (carry << (64 - carry_width));
+    man[WORDCOUNT - 1] = man[WORDCOUNT - 1] + (carry << (64 - carry_width));
     exp += carry_width == 64 ? 1 : 0;
     normalize();
   }
@@ -74,7 +74,7 @@ template <size_t Precision> class XFloat {
     if (exp < 0)
       return;
     if (exp > int(Precision - 1)) {
-      for (size_t i = 0; i < WordCount; ++i)
+      for (size_t i = 0; i < WORDCOUNT; ++i)
         man[i] = 0;
       return;
     }
@@ -86,14 +86,14 @@ template <size_t Precision> class XFloat {
   }
 
   double mul(const XFloat<Precision> &other) {
-    constexpr size_t row_words = 2 * WordCount + 1;
+    constexpr size_t row_words = 2 * WORDCOUNT + 1;
     constexpr size_t row_precision = row_words * 64;
     constexpr size_t result_bits = 2 * Precision;
-    UInt<row_precision> rows[WordCount];
+    UInt<row_precision> rows[WORDCOUNT];
 
-    for (size_t r = 0; r < WordCount; ++r) {
+    for (size_t r = 0; r < WORDCOUNT; ++r) {
       for (size_t i = 0; i < row_words; ++i) {
-        if (i < WordCount)
+        if (i < WORDCOUNT)
           rows[r][i] = man[i];
         else
           rows[r][i] = 0;
@@ -102,7 +102,7 @@ template <size_t Precision> class XFloat {
       rows[r].shift_left(r * 64);
     }
 
-    for (size_t r = 1; r < WordCount; ++r) {
+    for (size_t r = 1; r < WORDCOUNT; ++r) {
       rows[0].add(rows[r]);
     }
     int result_exp = exp + other.exp;
@@ -134,14 +134,14 @@ template <size_t Precision> class XFloat {
 
     constexpr uint64_t one = uint64_t(1) << 10;
     constexpr uint64_t excess_mask = (one << 1) - 1;
-    uint64_t excess = man[WordCount - 1] & excess_mask;
-    uint64_t new_man = man[WordCount - 1] >> 11;
+    uint64_t excess = man[WORDCOUNT - 1] & excess_mask;
+    uint64_t new_man = man[WORDCOUNT - 1] >> 11;
     if (excess > one) {
       // We have to round up.
       ++new_man;
     } else if (excess == one) {
       bool greater_than_one = false;
-      for (size_t i = 0; i < WordCount - 1; ++i) {
+      for (size_t i = 0; i < WORDCOUNT - 1; ++i) {
         greater_than_one = (man[i] != 0);
         if (greater_than_one)
           break;
@@ -163,13 +163,13 @@ template <size_t Precision> class XFloat {
   // Normalizes this number.
   void normalize() {
     uint64_t man_bits = 0;
-    for (size_t i = 0; i < WordCount; ++i)
+    for (size_t i = 0; i < WORDCOUNT; ++i)
       man_bits |= man[i];
 
     if (man_bits == 0)
       return;
 
-    while ((man[WordCount - 1] & OneMask) == 0) {
+    while ((man[WORDCOUNT - 1] & OneMask) == 0) {
       man.shift_left(1);
       --exp;
     }
diff --git a/libc/src/__support/FPUtil/arm/FEnvImpl.h b/libc/src/__support/FPUtil/arm/FEnvImpl.h
new file mode 100644
index 0000000000000..02535ecbca199
--- /dev/null
+++ b/libc/src/__support/FPUtil/arm/FEnvImpl.h
@@ -0,0 +1,265 @@
+//===-- arm floating point env manipulation functions -----------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC_SUPPORT_FPUTIL_ARM_FENVIMPL_H
+#define LLVM_LIBC_SRC_SUPPORT_FPUTIL_ARM_FENVIMPL_H
+
+#include "src/__support/FPUtil/FPBits.h"
+#include "src/__support/macros/config.h" // For LIBC_INLINE
+
+#include <fenv.h>
+#include <stdint.h>
+
+namespace __llvm_libc {
+namespace fputil {
+
+struct FEnv {
+  // Arm floating point state is all stored in a single 32-bit register named
+  // fpscr.
+  uint32_t fpscr;
+  static constexpr uint32_t RoundingControlBitPosition = 22;
+  static constexpr uint32_t ExceptionControlBitPosition = 8;
+
+  static constexpr uint32_t TONEAREST = 0x0;
+  static constexpr uint32_t UPWARD = 0x1;
+  static constexpr uint32_t DOWNWARD = 0x2;
+  static constexpr uint32_t TOWARDZERO = 0x3;
+
+  static constexpr uint32_t INVALID_ENABLE = 0x1;
+  static constexpr uint32_t DIVBYZERO_ENABLE = 0x2;
+  static constexpr uint32_t OVERFLOW_ENABLE = 0x4;
+  static constexpr uint32_t UNDERFLOW_ENABLE = 0x8;
+  static constexpr uint32_t INEXACT_ENABLE = 0x10;
+  static constexpr uint32_t DENORMAL_ENABLE = 0x20;
+
+  static constexpr uint32_t INVALID_STATUS = 0x1;
+  static constexpr uint32_t DIVBYZERO_STATUS = 0x2;
+  static constexpr uint32_t OVERFLOW_STATUS = 0x4;
+  static constexpr uint32_t UNDERFLOW_STATUS = 0x8;
+  static constexpr uint32_t INEXACT_STATUS = 0x10;
+  static constexpr uint32_t DENORMAL_STATUS = 0x80;
+
+  LIBC_INLINE static uint32_t get_fpscr() { return __builtin_arm_get_fpscr(); }
+  LIBC_INLINE static void set_fpscr(uint32_t val) {
+    __builtin_arm_set_fpscr(val);
+  }
+
+  LIBC_INLINE static int exception_enable_bits_to_macro(uint32_t status) {
+    return (status & INVALID_ENABLE ? FE_INVALID : 0) |
+           (status & DIVBYZERO_ENABLE ? FE_DIVBYZERO : 0) |
+           (status & OVERFLOW_ENABLE ? FE_OVERFLOW : 0) |
+           (status & UNDERFLOW_ENABLE ? FE_UNDERFLOW : 0) |
+           (status & INEXACT_ENABLE ? FE_INEXACT : 0);
+  }
+
+  LIBC_INLINE static uint32_t exception_macro_to_enable_bits(int except) {
+    return (except & FE_INVALID ? INVALID_ENABLE : 0) |
+           (except & FE_DIVBYZERO ? DIVBYZERO_ENABLE : 0) |
+           (except & FE_OVERFLOW ? OVERFLOW_ENABLE : 0) |
+           (except & FE_UNDERFLOW ? UNDERFLOW_ENABLE : 0) |
+           (except & FE_INEXACT ? INEXACT_ENABLE : 0);
+  }
+
+  LIBC_INLINE static uint32_t exception_macro_to_status_bits(int except) {
+    return (except & FE_INVALID ? INVALID_STATUS : 0) |
+           (except & FE_DIVBYZERO ? DIVBYZERO_STATUS : 0) |
+           (except & FE_OVERFLOW ? OVERFLOW_STATUS : 0) |
+           (except & FE_UNDERFLOW ? UNDERFLOW_STATUS : 0) |
+           (except & FE_INEXACT ? INEXACT_STATUS : 0);
+  }
+
+  LIBC_INLINE static uint32_t exception_status_bits_to_macro(int status) {
+    return (status & INVALID_STATUS ? FE_INVALID : 0) |
+           (status & DIVBYZERO_STATUS ? FE_DIVBYZERO : 0) |
+           (status & OVERFLOW_STATUS ? FE_OVERFLOW : 0) |
+           (status & UNDERFLOW_STATUS ? FE_UNDERFLOW : 0) |
+           (status & INEXACT_STATUS ? FE_INEXACT : 0);
+  }
+};
+
+// Enables exceptions in |excepts| and returns the previously set exceptions.
+LIBC_INLINE int enable_except(int excepts) {
+  uint32_t new_excepts = FEnv::exception_macro_to_enable_bits(excepts);
+  uint32_t fpscr = FEnv::get_fpscr();
+  int old = (fpscr >> FEnv::ExceptionControlBitPosition) & 0x3F;
+  fpscr |= (new_excepts << FEnv::ExceptionControlBitPosition);
+  FEnv::set_fpscr(fpscr);
+  return FEnv::exception_enable_bits_to_macro(old);
+}
+
+// Disables exceptions in |excepts| and returns the previously set exceptions.
+LIBC_INLINE int disable_except(int excepts) {
+  uint32_t disable_bits = FEnv::exception_macro_to_enable_bits(excepts);
+  uint32_t fpscr = FEnv::get_fpscr();
+  int old = (fpscr >> FEnv::ExceptionControlBitPosition) & 0x3F;
+  fpscr &= ~(disable_bits << FEnv::ExceptionControlBitPosition);
+  FEnv::set_fpscr(fpscr);
+  return FEnv::exception_enable_bits_to_macro(old);
+}
+
+// Returns the currently enabled exceptions.
+LIBC_INLINE int get_except() {
+  uint32_t fpscr = FEnv::get_fpscr();
+  int enabled_excepts = (fpscr >> FEnv::ExceptionControlBitPosition) & 0x3F;
+  return FEnv::exception_enable_bits_to_macro(enabled_excepts);
+}
+
+// Clears the exceptions in |excepts|.
+LIBC_INLINE int clear_except(int excepts) {
+  uint32_t fpscr = FEnv::get_fpscr();
+  uint32_t to_clear = FEnv::exception_macro_to_status_bits(excepts);
+  fpscr &= ~to_clear;
+  FEnv::set_fpscr(fpscr);
+  return 0;
+}
+
+// Returns the set of exceptions which are from the input set |excepts|.
+LIBC_INLINE int test_except(int excepts) {
+  uint32_t to_test = FEnv::exception_macro_to_status_bits(excepts);
+  uint32_t fpscr = FEnv::get_fpscr();
+  return FEnv::exception_status_bits_to_macro(fpscr & 0x9F & to_test);
+}
+
+// Set the exceptions in |excepts|.
+LIBC_INLINE int set_except(int excepts) {
+  uint32_t fpscr = FEnv::get_fpscr();
+  FEnv::set_fpscr(fpscr | FEnv::exception_macro_to_status_bits(excepts));
+  return 0;
+}
+
+LIBC_INLINE int raise_except(int excepts) {
+  float zero = 0.0f;
+  float one = 1.0f;
+  float large_value = float(FPBits<float>(FPBits<float>::MAX_NORMAL));
+  float small_value = float(FPBits<float>(FPBits<float>::MIN_NORMAL));
+  auto divfunc = [](float a, float b) {
+    __asm__ __volatile__("flds  s0, %0\n\t"
+                         "flds  s1, %1\n\t"
+                         "fdivs s0, s0, s1\n\t"
+                         : // No outputs
+                         : "m"(a), "m"(b)
+                         : "s0", "s1" /* s0 and s1 are clobbered */);
+  };
+
+  uint32_t to_raise = FEnv::exception_macro_to_status_bits(excepts);
+  int result = 0;
+
+  if (to_raise & FEnv::INVALID_STATUS) {
+    divfunc(zero, zero);
+    uint32_t fpscr = FEnv::get_fpscr();
+    if (!(fpscr & FEnv::INVALID_STATUS))
+      result = -1;
+  }
+  if (to_raise & FEnv::DIVBYZERO_STATUS) {
+    divfunc(one, zero);
+    uint32_t fpscr = FEnv::get_fpscr();
+    if (!(fpscr & FEnv::DIVBYZERO_STATUS))
+      result = -1;
+  }
+  if (to_raise & FEnv::OVERFLOW_STATUS) {
+    divfunc(large_value, small_value);
+    uint32_t fpscr = FEnv::get_fpscr();
+    if (!(fpscr & FEnv::OVERFLOW_STATUS))
+      result = -1;
+  }
+  if (to_raise & FEnv::UNDERFLOW_STATUS) {
+    divfunc(small_value, large_value);
+    uint32_t fpscr = FEnv::get_fpscr();
+    if (!(fpscr & FEnv::UNDERFLOW_STATUS))
+      result = -1;
+  }
+  if (to_raise & FEnv::INEXACT_STATUS) {
+    float two = 2.0f;
+    float three = 3.0f;
+    // 2.0 / 3.0 cannot be represented exactly in any radix 2 floating point
+    // format.
+    divfunc(two, three);
+    uint32_t fpscr = FEnv::get_fpscr();
+    if (!(fpscr & FEnv::INEXACT_STATUS))
+      result = -1;
+  }
+  return result;
+}
+
+LIBC_INLINE int get_round() {
+  uint32_t mode = (FEnv::get_fpscr() >> FEnv::RoundingControlBitPosition) & 0x3;
+  switch (mode) {
+  case FEnv::TONEAREST:
+    return FE_TONEAREST;
+  case FEnv::DOWNWARD:
+    return FE_DOWNWARD;
+  case FEnv::UPWARD:
+    return FE_UPWARD;
+  case FEnv::TOWARDZERO:
+    return FE_TOWARDZERO;
+  default:
+    return -1; // Error value.
+  }
+  return 0;
+}
+
+LIBC_INLINE int set_round(int mode) {
+  uint16_t bits;
+  switch (mode) {
+  case FE_TONEAREST:
+    bits = FEnv::TONEAREST;
+    break;
+  case FE_DOWNWARD:
+    bits = FEnv::DOWNWARD;
+    break;
+  case FE_UPWARD:
+    bits = FEnv::UPWARD;
+    break;
+  case FE_TOWARDZERO:
+    bits = FEnv::TOWARDZERO;
+    break;
+  default:
+    return 1; // To indicate failure
+  }
+
+  uint32_t fpscr = FEnv::get_fpscr();
+  fpscr &= ~(0x3 << FEnv::RoundingControlBitPosition);
+  fpscr |= (bits << FEnv::RoundingControlBitPosition);
+  FEnv::set_fpscr(fpscr);
+
+  return 0;
+}
+
+LIBC_INLINE int get_env(fenv_t *envp) {
+  FEnv *state = reinterpret_cast<FEnv *>(envp);
+  state->fpscr = FEnv::get_fpscr();
+  return 0;
+}
+
+LIBC_INLINE int set_env(const fenv_t *envp) {
+  if (envp == FE_DFL_ENV) {
+    uint32_t fpscr = FEnv::get_fpscr();
+    // Default status implies:
+    // 1. Round to nearest rounding mode.
+    fpscr &= ~(0x3 << FEnv::RoundingControlBitPosition);
+    fpscr |= (FEnv::TONEAREST << FEnv::RoundingControlBitPosition);
+    // 2. All exceptions are disabled.
+    fpscr &= ~(0x3F << FEnv::ExceptionControlBitPosition);
+    // 3. All exceptions are cleared. There are two reserved bits
+    // at bit 5 and 6 so we just write one full byte (6 bits for
+    // the exceptions, and 2 reserved bits.)
+    fpscr &= ~(static_cast<uint32_t>(0xFF));
+
+    FEnv::set_fpscr(fpscr);
+    return 0;
+  }
+
+  const FEnv *state = reinterpret_cast<const FEnv *>(envp);
+  FEnv::set_fpscr(state->fpscr);
+  return 0;
+}
+
+} // namespace fputil
+} // namespace __llvm_libc
+
+#endif // LLVM_LIBC_SRC_SUPPORT_FPUTIL_ARM_FENVIMPL_H
diff --git a/libc/src/__support/FPUtil/dyadic_float.h b/libc/src/__support/FPUtil/dyadic_float.h
index 86c41df18bc6f..a1dc8a5cc36ab 100644
--- a/libc/src/__support/FPUtil/dyadic_float.h
+++ b/libc/src/__support/FPUtil/dyadic_float.h
@@ -104,11 +104,11 @@ template <size_t Bits> struct DyadicFloat {
         sign, exponent + (Bits - 1) + FloatProperties<T>::EXPONENT_BIAS,
         output_bits_t(m_hi) & FloatProperties<T>::MANTISSA_MASK);
 
-    const MantissaType ROUND_MASK = MantissaType(1) << (Bits - PRECISION - 1);
-    const MantissaType STICKY_MASK = ROUND_MASK - MantissaType(1);
+    const MantissaType round_mask = MantissaType(1) << (Bits - PRECISION - 1);
+    const MantissaType sticky_mask = round_mask - MantissaType(1);
 
-    bool round_bit = !(mantissa & ROUND_MASK).is_zero();
-    bool sticky_bit = !(mantissa & STICKY_MASK).is_zero();
+    bool round_bit = !(mantissa & round_mask).is_zero();
+    bool sticky_bit = !(mantissa & sticky_mask).is_zero();
     int round_and_sticky = int(round_bit) * 2 + int(sticky_bit);
     auto d_lo = FPBits<T>::create_value(sign,
                                         exponent + (Bits - PRECISION - 2) +
@@ -156,7 +156,7 @@ constexpr DyadicFloat<Bits> quick_add(DyadicFloat<Bits> a,
     if (result.mantissa.add(b.mantissa)) {
       // Mantissa addition overflow.
       result.shift_right(1);
-      result.mantissa.val[DyadicFloat<Bits>::MantissaType::WordCount - 1] |=
+      result.mantissa.val[DyadicFloat<Bits>::MantissaType::WORDCOUNT - 1] |=
           (uint64_t(1) << 63);
     }
     // Result is already normalized.
@@ -183,7 +183,7 @@ constexpr DyadicFloat<Bits> quick_add(DyadicFloat<Bits> a,
 //   result.mantissa = quick_mul_hi(a.mantissa + b.mantissa)
 //                   ~ (full product a.mantissa * b.mantissa) >> Bits.
 // The errors compared to the mathematical product is bounded by:
-//   2 * errors of quick_mul_hi = 2 * (UInt<Bits>::WordCount - 1) in ULPs.
+//   2 * errors of quick_mul_hi = 2 * (UInt<Bits>::WORDCOUNT - 1) in ULPs.
 // Assume inputs are normalized (by constructors or other functions) so that we
 // don't need to normalize the inputs again in this function.  If the inputs are
 // not normalized, the results might lose precision significantly.
@@ -198,7 +198,7 @@ constexpr DyadicFloat<Bits> quick_mul(DyadicFloat<Bits> a,
     result.mantissa = a.mantissa.quick_mul_hi(b.mantissa);
     // Check the leading bit directly, should be faster than using clz in
     // normalize().
-    if (result.mantissa.val[DyadicFloat<Bits>::MantissaType::WordCount - 1] >>
+    if (result.mantissa.val[DyadicFloat<Bits>::MantissaType::WORDCOUNT - 1] >>
             63 ==
         0)
       result.shift_left(1);
diff --git a/libc/src/__support/FPUtil/generic/FMod.h b/libc/src/__support/FPUtil/generic/FMod.h
index 1879c0ad2c510..fd40d87132922 100644
--- a/libc/src/__support/FPUtil/generic/FMod.h
+++ b/libc/src/__support/FPUtil/generic/FMod.h
@@ -122,9 +122,9 @@ template <typename T> struct FModExceptionalInputHandler {
   static_assert(cpp::is_floating_point_v<T>,
                 "FModCStandardWrapper instantiated with invalid type.");
 
-  LIBC_INLINE static bool PreCheck(T x, T y, T &out) {
+  LIBC_INLINE static bool pre_check(T x, T y, T &out) {
     using FPB = fputil::FPBits<T>;
-    const T quiet_NaN = FPB::build_quiet_nan(0);
+    const T quiet_nan = FPB::build_quiet_nan(0);
     FPB sx(x), sy(y);
     if (LIBC_LIKELY(!sy.is_zero() && !sy.is_inf_or_nan() &&
                     !sx.is_inf_or_nan())) {
@@ -134,14 +134,15 @@ template <typename T> struct FModExceptionalInputHandler {
     if (sx.is_nan() || sy.is_nan()) {
       if ((sx.is_nan() && !sx.is_quiet_nan()) ||
           (sy.is_nan() && !sy.is_quiet_nan()))
-        fputil::set_except(FE_INVALID);
-      out = quiet_NaN;
+        fputil::raise_except_if_required(FE_INVALID);
+      out = quiet_nan;
       return true;
     }
 
     if (sx.is_inf() || sy.is_zero()) {
-      fputil::set_except(FE_INVALID);
-      out = with_errno(quiet_NaN, EDOM);
+      fputil::raise_except_if_required(FE_INVALID);
+      fputil::set_errno_if_required(EDOM);
+      out = quiet_nan;
       return true;
     }
 
@@ -161,7 +162,7 @@ template <typename T> struct FModFastMathWrapper {
   static_assert(cpp::is_floating_point_v<T>,
                 "FModFastMathWrapper instantiated with invalid type.");
 
-  static bool PreCheck(T, T, T &) { return false; }
+  static bool pre_check(T, T, T &) { return false; }
 };
 
 template <typename T> class FModDivisionSimpleHelper {
@@ -302,7 +303,7 @@ class FMod {
 
 public:
   LIBC_INLINE static T eval(T x, T y) {
-    if (T out; Wrapper::PreCheck(x, y, out))
+    if (T out; Wrapper::pre_check(x, y, out))
       return out;
     FPB sx(x), sy(y);
     bool sign = sx.get_sign();
diff --git a/libc/src/__support/FPUtil/riscv64/FEnvImpl.h b/libc/src/__support/FPUtil/riscv64/FEnvImpl.h
new file mode 100644
index 0000000000000..b4c11443b798a
--- /dev/null
+++ b/libc/src/__support/FPUtil/riscv64/FEnvImpl.h
@@ -0,0 +1,180 @@
+//===-- riscv64 floating point env manipulation functions -------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC_SUPPORT_FPUTIL_RISCV64_FENVIMPL_H
+#define LLVM_LIBC_SRC_SUPPORT_FPUTIL_RISCV64_FENVIMPL_H
+
+#include "src/__support/FPUtil/FPBits.h"
+#include "src/__support/macros/attributes.h" // For LIBC_INLINE_ASM
+#include "src/__support/macros/config.h"     // For LIBC_INLINE
+
+#include <fenv.h>
+#include <stdint.h>
+
+namespace __llvm_libc {
+namespace fputil {
+
+struct FEnv {
+  // We will ignore RMM and DYN rounding modes.
+  static constexpr uint32_t TONEAREST = 0x0;
+  static constexpr uint32_t TOWARDZERO = 0x1;
+  static constexpr uint32_t DOWNWARD = 0x2;
+  static constexpr uint32_t UPWARD = 0x3;
+
+  // These are the bit locations of the corresponding exceptions in fcsr.
+  static constexpr uint32_t INEXACT = 0x1;
+  static constexpr uint32_t UNDERFLOW = 0x2;
+  static constexpr uint32_t OVERFLOW = 0x4;
+  static constexpr uint32_t DIVBYZERO = 0x8;
+  static constexpr uint32_t INVALID = 0x10;
+
+  LIBC_INLINE static uint32_t get_frm() {
+    unsigned int rm;
+    LIBC_INLINE_ASM("frrm %0\n\t" : "=r"(rm));
+    return rm;
+  }
+
+  LIBC_INLINE static void set_frm(uint32_t rm) {
+    LIBC_INLINE_ASM("fsrm %0, %0\n\t" : "+r"(rm));
+  }
+
+  LIBC_INLINE static uint32_t get_fflags() {
+    unsigned int flags;
+    LIBC_INLINE_ASM("frflags %0\n\t" : "=r"(flags));
+    return flags;
+  }
+
+  LIBC_INLINE static void set_fflags(uint32_t flags) {
+    LIBC_INLINE_ASM("fsflags %0, %0\n\t" : "+r"(flags));
+  }
+
+  LIBC_INLINE static uint32_t get_fcsr() {
+    unsigned int fcsr;
+    LIBC_INLINE_ASM("frcsr %0\n\t" : "=r"(fcsr));
+    return fcsr;
+  }
+
+  LIBC_INLINE static void set_fcsr(uint32_t fcsr) {
+    LIBC_INLINE_ASM("fscsr %0, %0\n\t" : "+r"(fcsr));
+  }
+
+  LIBC_INLINE static int exception_bits_to_macro(uint32_t status) {
+    return (status & INVALID ? FE_INVALID : 0) |
+           (status & DIVBYZERO ? FE_DIVBYZERO : 0) |
+           (status & OVERFLOW ? FE_OVERFLOW : 0) |
+           (status & UNDERFLOW ? FE_UNDERFLOW : 0) |
+           (status & INEXACT ? FE_INEXACT : 0);
+  }
+
+  LIBC_INLINE static uint32_t exception_macro_to_bits(int except) {
+    return (except & FE_INVALID ? INVALID : 0) |
+           (except & FE_DIVBYZERO ? DIVBYZERO : 0) |
+           (except & FE_OVERFLOW ? OVERFLOW : 0) |
+           (except & FE_UNDERFLOW ? UNDERFLOW : 0) |
+           (except & FE_INEXACT ? INEXACT : 0);
+  }
+};
+
+// Since RISCV does not have exception enable bits, we will just return
+// the failure indicator.
+LIBC_INLINE int enable_except(int) { return -1; }
+
+// Always succeed.
+LIBC_INLINE int disable_except(int) { return 0; }
+
+// Always return "no exceptions enabled".
+LIBC_INLINE int get_except() { return 0; }
+
+LIBC_INLINE int clear_except(int excepts) {
+  uint32_t flags = FEnv::get_fflags();
+  uint32_t to_clear = FEnv::exception_macro_to_bits(excepts);
+  flags &= ~to_clear;
+  FEnv::set_fflags(flags);
+  return 0;
+}
+
+LIBC_INLINE int test_except(int excepts) {
+  uint32_t to_test = FEnv::exception_macro_to_bits(excepts);
+  uint32_t flags = FEnv::get_fflags();
+  return FEnv::exception_bits_to_macro(flags & to_test);
+}
+
+LIBC_INLINE int set_except(int excepts) {
+  uint32_t flags = FEnv::get_fflags();
+  FEnv::set_fflags(flags | FEnv::exception_macro_to_bits(excepts));
+  return 0;
+}
+
+LIBC_INLINE int raise_except(int excepts) {
+  // Since there are no traps, we just set the exception flags.
+  uint32_t flags = FEnv::get_fflags();
+  FEnv::set_fflags(flags | FEnv::exception_macro_to_bits(excepts));
+  return 0;
+}
+
+LIBC_INLINE int get_round() {
+  uint32_t rm = FEnv::get_frm();
+  switch (rm) {
+  case FEnv::TONEAREST:
+    return FE_TONEAREST;
+  case FEnv::DOWNWARD:
+    return FE_DOWNWARD;
+  case FEnv::UPWARD:
+    return FE_UPWARD;
+  case FEnv::TOWARDZERO:
+    return FE_TOWARDZERO;
+  default:
+    return -1; // Error value.
+  }
+  return 0;
+}
+
+LIBC_INLINE int set_round(int mode) {
+  uint32_t rm;
+  switch (mode) {
+  case FE_TONEAREST:
+    rm = FEnv::TONEAREST;
+    break;
+  case FE_DOWNWARD:
+    rm = FEnv::DOWNWARD;
+    break;
+  case FE_UPWARD:
+    rm = FEnv::UPWARD;
+    break;
+  case FE_TOWARDZERO:
+    rm = FEnv::TOWARDZERO;
+    break;
+  default:
+    return -1; // To indicate failure
+  }
+  FEnv::set_frm(rm);
+  return 0;
+}
+
+LIBC_INLINE int get_env(fenv_t *envp) {
+  uint32_t *state = reinterpret_cast<uint32_t *>(envp);
+  *state = FEnv::get_fcsr();
+  return 0;
+}
+
+LIBC_INLINE int set_env(const fenv_t *envp) {
+  if (envp == FE_DFL_ENV) {
+    FEnv::set_frm(FEnv::TONEAREST);
+    FEnv::set_fflags(0);
+    return 0;
+  }
+  uint32_t status = *reinterpret_cast<const uint32_t *>(envp);
+  // We have to do the masking to preserve the reserved bits.
+  FEnv::set_fcsr((status & 0xFF) | (FEnv::get_fcsr() & 0xFFFFFF00));
+  return 0;
+}
+
+} // namespace fputil
+} // namespace __llvm_libc
+
+#endif // LLVM_LIBC_SRC_SUPPORT_FPUTIL_RISCV64_FENVIMPL_H
diff --git a/libc/src/__support/FPUtil/riscv64/FMA.h b/libc/src/__support/FPUtil/riscv64/FMA.h
new file mode 100644
index 0000000000000..9b003f10adbf2
--- /dev/null
+++ b/libc/src/__support/FPUtil/riscv64/FMA.h
@@ -0,0 +1,49 @@
+//===-- RISCV64 implementations of the fma function -------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC_SUPPORT_FPUTIL_RISCV64_FMA_H
+#define LLVM_LIBC_SRC_SUPPORT_FPUTIL_RISCV64_FMA_H
+
+#include "src/__support/macros/properties/architectures.h"
+#include "src/__support/macros/properties/cpu_features.h" // LIBC_TARGET_CPU_HAS_FMA
+
+#if !defined(LIBC_TARGET_ARCH_IS_RISCV64)
+#error "Invalid include"
+#endif
+
+#if !defined(LIBC_TARGET_CPU_HAS_FMA)
+#error "FMA instructions are not supported"
+#endif
+
+#include "src/__support/CPP/type_traits.h"
+
+namespace __llvm_libc {
+namespace fputil {
+
+template <typename T>
+cpp::enable_if_t<cpp::is_same_v<T, float>, T> fma(T x, T y, T z) {
+  float result;
+  __asm__ __volatile__("fmadd.s %0, %1, %2, %3\n\t"
+                       : "=f"(result)
+                       : "f"(x), "f"(y), "f"(z));
+  return result;
+}
+
+template <typename T>
+cpp::enable_if_t<cpp::is_same_v<T, double>, T> fma(T x, T y, T z) {
+  double result;
+  __asm__ __volatile__("fmadd.d %0, %1, %2, %3\n\t"
+                       : "=f"(result)
+                       : "f"(x), "f"(y), "f"(z));
+  return result;
+}
+
+} // namespace fputil
+} // namespace __llvm_libc
+
+#endif // LLVM_LIBC_SRC_SUPPORT_FPUTIL_RISCV64_FMA_H
diff --git a/libc/src/__support/FPUtil/riscv64/sqrt.h b/libc/src/__support/FPUtil/riscv64/sqrt.h
new file mode 100644
index 0000000000000..887000bda7be6
--- /dev/null
+++ b/libc/src/__support/FPUtil/riscv64/sqrt.h
@@ -0,0 +1,39 @@
+//===-- Square root of IEEE 754 floating point numbers ----------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC_SUPPORT_FPUTIL_RISCV64_SQRT_H
+#define LLVM_LIBC_SRC_SUPPORT_FPUTIL_RISCV64_SQRT_H
+
+#include "src/__support/common.h"
+#include "src/__support/macros/properties/architectures.h"
+
+#if !defined(LIBC_TARGET_ARCH_IS_RISCV64)
+#error "Invalid include"
+#endif
+
+#include "src/__support/FPUtil/generic/sqrt.h"
+
+namespace __llvm_libc {
+namespace fputil {
+
+template <> LIBC_INLINE float sqrt<float>(float x) {
+  float result;
+  __asm__ __volatile__("fsqrt.s %0, %1\n\t" : "=f"(result) : "f"(x));
+  return result;
+}
+
+template <> LIBC_INLINE double sqrt<double>(double x) {
+  double result;
+  __asm__ __volatile__("fsqrt.d %0, %1\n\t" : "=f"(result) : "f"(x));
+  return result;
+}
+
+} // namespace fputil
+} // namespace __llvm_libc
+
+#endif // LLVM_LIBC_SRC_SUPPORT_FPUTIL_RISCV64_SQRT_H
diff --git a/libc/src/__support/FPUtil/sqrt.h b/libc/src/__support/FPUtil/sqrt.h
index d64fd22d3f1f9..132e7237c810c 100644
--- a/libc/src/__support/FPUtil/sqrt.h
+++ b/libc/src/__support/FPUtil/sqrt.h
@@ -15,6 +15,8 @@
 #include "x86_64/sqrt.h"
 #elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
 #include "aarch64/sqrt.h"
+#elif defined(LIBC_TARGET_ARCH_IS_RISCV64)
+#include "riscv64/sqrt.h"
 #else
 #include "generic/sqrt.h"
 
diff --git a/libc/src/__support/OSUtil/CMakeLists.txt b/libc/src/__support/OSUtil/CMakeLists.txt
index 50aad32b7ed31..c19677582643e 100644
--- a/libc/src/__support/OSUtil/CMakeLists.txt
+++ b/libc/src/__support/OSUtil/CMakeLists.txt
@@ -8,12 +8,23 @@ if(NOT TARGET ${target_os_util})
   return()
 endif()
 
-add_header_library(
-  osutil
-  HDRS
-    io.h
-    quick_exit.h
-    syscall.h
-  DEPENDS
-    ${target_os_util}
-)
+# The OSUtil is an object library in GPU mode.
+if(NOT LIBC_TARGET_ARCHITECTURE_IS_GPU)
+  add_header_library(
+    osutil
+    HDRS
+      io.h
+      quick_exit.h
+      syscall.h
+    DEPENDS
+      ${target_os_util}
+  )
+else()
+  add_object_library(
+    osutil
+    ALIAS
+      ${target_os_util}
+    DEPENDS
+      ${target_os_util}
+  )
+endif()
diff --git a/libc/src/__support/OSUtil/gpu/CMakeLists.txt b/libc/src/__support/OSUtil/gpu/CMakeLists.txt
index eb6e86feaad3b..d1aa69604e515 100644
--- a/libc/src/__support/OSUtil/gpu/CMakeLists.txt
+++ b/libc/src/__support/OSUtil/gpu/CMakeLists.txt
@@ -2,8 +2,11 @@ add_object_library(
   gpu_util
   SRCS
     quick_exit.cpp
+    io.cpp
   HDRS
     quick_exit.h
+    io.h
   DEPENDS
     libc.src.__support.common
+    libc.src.__support.RPC.rpc_client
 )
diff --git a/libc/src/__support/OSUtil/gpu/io.cpp b/libc/src/__support/OSUtil/gpu/io.cpp
new file mode 100644
index 0000000000000..75ac83ac6909c
--- /dev/null
+++ b/libc/src/__support/OSUtil/gpu/io.cpp
@@ -0,0 +1,29 @@
+//===-------------- GPU implementation of IO utils --------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "io.h"
+
+#include "src/__support/RPC/rpc_client.h"
+#include "src/string/string_utils.h"
+
+namespace __llvm_libc {
+
+void write_to_stderr(const char *msg) {
+  uint64_t length = internal::string_length(msg) + 1;
+  uint64_t buffer_len = sizeof(rpc::Buffer) - sizeof(uint64_t);
+  for (uint64_t i = 0; i < length; i += buffer_len)
+    rpc::client.run(
+        [&](rpc::Buffer *buffer) {
+          buffer->data[0] = rpc::Opcode::PRINT_TO_STDERR;
+          inline_memcpy(reinterpret_cast<char *>(&buffer->data[1]), &msg[i],
+                        (length > buffer_len ? buffer_len : length));
+        },
+        [](rpc::Buffer *) {});
+}
+
+} // namespace __llvm_libc
diff --git a/libc/src/__support/OSUtil/gpu/io.h b/libc/src/__support/OSUtil/gpu/io.h
new file mode 100644
index 0000000000000..e9a4ebf82a111
--- /dev/null
+++ b/libc/src/__support/OSUtil/gpu/io.h
@@ -0,0 +1,18 @@
+//===-------------- GPU implementation of IO utils --------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC_SUPPORT_OSUTIL_GPU_IO_H
+#define LLVM_LIBC_SRC_SUPPORT_OSUTIL_GPU_IO_H
+
+namespace __llvm_libc {
+
+void write_to_stderr(const char *msg);
+
+} // namespace __llvm_libc
+
+#endif // LLVM_LIBC_SRC_SUPPORT_OSUTIL_LINUX_IO_H
diff --git a/libc/src/__support/OSUtil/gpu/quick_exit.cpp b/libc/src/__support/OSUtil/gpu/quick_exit.cpp
index d8c8f032a9403..9be709552dc5c 100644
--- a/libc/src/__support/OSUtil/gpu/quick_exit.cpp
+++ b/libc/src/__support/OSUtil/gpu/quick_exit.cpp
@@ -11,11 +11,21 @@
 
 #include "quick_exit.h"
 
+#include "src/__support/RPC/rpc_client.h"
 #include "src/__support/macros/properties/architectures.h"
 
 namespace __llvm_libc {
 
 void quick_exit(int status) {
+  // TODO: Support asynchronous calls so we don't wait and exit from the GPU
+  // immediately.
+  rpc::client.run(
+      [&](rpc::Buffer *buffer) {
+        buffer->data[0] = rpc::Opcode::EXIT;
+        buffer->data[1] = status;
+      },
+      [](rpc::Buffer *) {});
+
 #if defined(LIBC_TARGET_ARCH_IS_NVPTX)
   asm("exit" ::: "memory");
 #elif defined(LIBC_TARGET_ARCH_IS_AMDGPU)
diff --git a/libc/src/__support/OSUtil/io.h b/libc/src/__support/OSUtil/io.h
index dbf92dc201c3d..e2eee0826b177 100644
--- a/libc/src/__support/OSUtil/io.h
+++ b/libc/src/__support/OSUtil/io.h
@@ -9,7 +9,11 @@
 #ifndef LLVM_LIBC_SRC_SUPPORT_OSUTIL_IO_H
 #define LLVM_LIBC_SRC_SUPPORT_OSUTIL_IO_H
 
-#ifdef __unix__
+#include "src/__support/macros/properties/architectures.h"
+
+#if defined(LIBC_TARGET_ARCH_IS_GPU)
+#include "gpu/io.h"
+#elif defined(__unix__)
 #include "linux/io.h"
 #endif
 
diff --git a/libc/src/__support/OSUtil/linux/riscv64/CMakeLists.txt b/libc/src/__support/OSUtil/linux/riscv64/CMakeLists.txt
new file mode 100644
index 0000000000000..69540e3b45cbc
--- /dev/null
+++ b/libc/src/__support/OSUtil/linux/riscv64/CMakeLists.txt
@@ -0,0 +1,7 @@
+add_header_library(
+  linux_riscv64_util
+  HDRS
+    syscall.h
+  DEPENDS
+    libc.src.__support.common
+)
diff --git a/libc/src/__support/OSUtil/linux/riscv64/syscall.h b/libc/src/__support/OSUtil/linux/riscv64/syscall.h
new file mode 100644
index 0000000000000..c9d2a39941a64
--- /dev/null
+++ b/libc/src/__support/OSUtil/linux/riscv64/syscall.h
@@ -0,0 +1,110 @@
+//===--------- inline implementation of riscv64 syscalls ----------* C++ *-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC_SUPPORT_OSUTIL_LINUX_RISCV64_SYSCALL_H
+#define LLVM_LIBC_SRC_SUPPORT_OSUTIL_LINUX_RISCV64_SYSCALL_H
+
+#include "src/__support/common.h"
+
+#define REGISTER_DECL_0                                                        \
+  register long a7 __asm__("a7") = number;                                     \
+  register long a0 __asm__("a0");
+#define REGISTER_DECL_1                                                        \
+  register long a7 __asm__("a7") = number;                                     \
+  register long a0 __asm__("a0") = arg1;
+#define REGISTER_DECL_2 REGISTER_DECL_1 register long a1 __asm__("a1") = arg2;
+#define REGISTER_DECL_3                                                        \
+  REGISTER_DECL_2                                                              \
+  register long a2 __asm__("a2") = arg3;
+#define REGISTER_DECL_4                                                        \
+  REGISTER_DECL_3                                                              \
+  register long a3 __asm__("a3") = arg4;
+#define REGISTER_DECL_5                                                        \
+  REGISTER_DECL_4                                                              \
+  register long a4 __asm__("a4") = arg5;
+#define REGISTER_DECL_6                                                        \
+  REGISTER_DECL_5                                                              \
+  register long a5 __asm__("a5") = arg6;
+
+#define REGISTER_CONSTRAINT_0 "r"(a7)
+#define REGISTER_CONSTRAINT_1 REGISTER_CONSTRAINT_0, "r"(a0)
+#define REGISTER_CONSTRAINT_2 REGISTER_CONSTRAINT_1, "r"(a1)
+#define REGISTER_CONSTRAINT_3 REGISTER_CONSTRAINT_2, "r"(a2)
+#define REGISTER_CONSTRAINT_4 REGISTER_CONSTRAINT_3, "r"(a3)
+#define REGISTER_CONSTRAINT_5 REGISTER_CONSTRAINT_4, "r"(a4)
+#define REGISTER_CONSTRAINT_6 REGISTER_CONSTRAINT_5, "r"(a5)
+
+#define SYSCALL_INSTR(input_constraint)                                        \
+  LIBC_INLINE_ASM("ecall\n\t" : "=r"(a0) : input_constraint : "memory")
+
+namespace __llvm_libc {
+
+LIBC_INLINE long syscall_impl(long number) {
+  REGISTER_DECL_0;
+  SYSCALL_INSTR(REGISTER_CONSTRAINT_0);
+  return a0;
+}
+
+LIBC_INLINE long syscall_impl(long number, long arg1) {
+  REGISTER_DECL_1;
+  SYSCALL_INSTR(REGISTER_CONSTRAINT_1);
+  return a0;
+}
+
+LIBC_INLINE long syscall_impl(long number, long arg1, long arg2) {
+  REGISTER_DECL_2;
+  SYSCALL_INSTR(REGISTER_CONSTRAINT_2);
+  return a0;
+}
+
+LIBC_INLINE long syscall_impl(long number, long arg1, long arg2, long arg3) {
+  REGISTER_DECL_3;
+  SYSCALL_INSTR(REGISTER_CONSTRAINT_3);
+  return a0;
+}
+
+LIBC_INLINE long syscall_impl(long number, long arg1, long arg2, long arg3,
+                              long arg4) {
+  REGISTER_DECL_4;
+  SYSCALL_INSTR(REGISTER_CONSTRAINT_4);
+  return a0;
+}
+
+LIBC_INLINE long syscall_impl(long number, long arg1, long arg2, long arg3,
+                              long arg4, long arg5) {
+  REGISTER_DECL_5;
+  SYSCALL_INSTR(REGISTER_CONSTRAINT_5);
+  return a0;
+}
+
+LIBC_INLINE long syscall_impl(long number, long arg1, long arg2, long arg3,
+                              long arg4, long arg5, long arg6) {
+  REGISTER_DECL_6;
+  SYSCALL_INSTR(REGISTER_CONSTRAINT_6);
+  return a0;
+}
+
+} // namespace __llvm_libc
+
+#undef REGISTER_DECL_0
+#undef REGISTER_DECL_1
+#undef REGISTER_DECL_2
+#undef REGISTER_DECL_3
+#undef REGISTER_DECL_4
+#undef REGISTER_DECL_5
+#undef REGISTER_DECL_6
+
+#undef REGISTER_CONSTRAINT_0
+#undef REGISTER_CONSTRAINT_1
+#undef REGISTER_CONSTRAINT_2
+#undef REGISTER_CONSTRAINT_3
+#undef REGISTER_CONSTRAINT_4
+#undef REGISTER_CONSTRAINT_5
+#undef REGISTER_CONSTRAINT_6
+
+#endif // LLVM_LIBC_SRC_SUPPORT_OSUTIL_LINUX_RISCV64_SYSCALL_H
diff --git a/libc/src/__support/OSUtil/linux/syscall.h b/libc/src/__support/OSUtil/linux/syscall.h
index 864fa4b99824a..c0366cfd9e71b 100644
--- a/libc/src/__support/OSUtil/linux/syscall.h
+++ b/libc/src/__support/OSUtil/linux/syscall.h
@@ -18,6 +18,8 @@
 #include "aarch64/syscall.h"
 #elif defined(LIBC_TARGET_ARCH_IS_ARM)
 #include "arm/syscall.h"
+#elif defined(LIBC_TARGET_ARCH_IS_RISCV64)
+#include "riscv64/syscall.h"
 #endif
 
 namespace __llvm_libc {
diff --git a/libc/src/__support/RPC/CMakeLists.txt b/libc/src/__support/RPC/CMakeLists.txt
new file mode 100644
index 0000000000000..f5837628c2050
--- /dev/null
+++ b/libc/src/__support/RPC/CMakeLists.txt
@@ -0,0 +1,18 @@
+add_header_library(
+  rpc
+  HDRS
+    rpc.h
+  DEPENDS
+    libc.src.__support.common
+    libc.src.__support.CPP.atomic
+)
+
+add_object_library(
+  rpc_client
+  SRCS
+    rpc_client.cpp
+  HDRS
+    rpc_client.h
+  DEPENDS
+    .rpc
+)
diff --git a/libc/src/__support/RPC/rpc.h b/libc/src/__support/RPC/rpc.h
new file mode 100644
index 0000000000000..c3df09e3f5db2
--- /dev/null
+++ b/libc/src/__support/RPC/rpc.h
@@ -0,0 +1,140 @@
+//===-- Shared memory RPC client / server interface -------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC_SUPPORT_RPC_RPC_H
+#define LLVM_LIBC_SRC_SUPPORT_RPC_RPC_H
+
+#include "src/__support/CPP/atomic.h"
+
+#include <stdint.h>
+
+namespace __llvm_libc {
+namespace rpc {
+
+/// A list of opcodes that we use to invoke certain actions on the server. We
+/// reserve the first 255 values for internal libc usage.
+enum Opcode : uint64_t {
+  NOOP = 0,
+  PRINT_TO_STDERR = 1,
+  EXIT = 2,
+  LIBC_LAST = (1UL << 8) - 1,
+};
+
+/// A fixed size channel used to communicate between the RPC client and server.
+struct Buffer {
+  uint64_t data[8];
+};
+
+/// A common process used to synchronize communication between a client and a
+/// server. The process contains an inbox and an outbox used for signaling
+/// ownership of the shared buffer.
+struct Process {
+  cpp::Atomic<uint32_t> *inbox;
+  cpp::Atomic<uint32_t> *outbox;
+  Buffer *buffer;
+
+  /// Initialize the communication channels.
+  void reset(void *inbox, void *outbox, void *buffer) {
+    *this = {
+        reinterpret_cast<cpp::Atomic<uint32_t> *>(inbox),
+        reinterpret_cast<cpp::Atomic<uint32_t> *>(outbox),
+        reinterpret_cast<Buffer *>(buffer),
+    };
+  }
+};
+
+/// The RPC client used to make requests to the server.
+struct Client : public Process {
+  template <typename F, typename U> void run(F fill, U use);
+};
+
+/// The RPC server used to respond to the client.
+struct Server : public Process {
+  template <typename W, typename C> bool run(W work, C clean);
+};
+
+/// Run the RPC client protocol to communicate with the server. We perform the
+/// following high level actions to complete a communication:
+///   - Apply \p fill to the shared buffer and write 1 to the outbox.
+///   - Wait until the inbox is 1.
+///   - Apply \p use to the shared buffer and write 0 to the outbox.
+///   - Wait until the inbox is 0.
+template <typename F, typename U> void Client::run(F fill, U use) {
+  bool in = inbox->load(cpp::MemoryOrder::RELAXED);
+  bool out = outbox->load(cpp::MemoryOrder::RELAXED);
+  atomic_thread_fence(cpp::MemoryOrder::ACQUIRE);
+  // Write to buffer then to the outbox.
+  if (!in & !out) {
+    fill(buffer);
+    atomic_thread_fence(cpp::MemoryOrder::RELEASE);
+    outbox->store(1, cpp::MemoryOrder::RELEASE);
+    out = 1;
+  }
+  // Wait for the result from the server.
+  if (!in & out) {
+    while (!in)
+      in = inbox->load(cpp::MemoryOrder::RELAXED);
+    atomic_thread_fence(cpp::MemoryOrder::ACQUIRE);
+  }
+  // Read from the buffer and then write to outbox.
+  if (in & out) {
+    use(buffer);
+    atomic_thread_fence(cpp::MemoryOrder::RELEASE);
+    outbox->store(0, cpp::MemoryOrder::RELEASE);
+    out = 0;
+  }
+  // Wait for server to complete the communication.
+  if (in & !out) {
+    while (in)
+      in = inbox->load(cpp::MemoryOrder::RELAXED);
+    atomic_thread_fence(cpp::MemoryOrder::ACQUIRE);
+  }
+}
+
+/// Run the RPC server protocol to communicate with the client. This is
+/// non-blocking and only checks the server a single time. We perform the
+/// following high level actions to complete a communication:
+///   - Query if the inbox is 1 and exit if there is no work to do.
+///   - Apply \p work to the shared buffer and write 1 to the outbox.
+///   - Wait until the inbox is 0.
+///   - Apply \p clean to the shared buffer and write 0 to the outbox.
+template <typename W, typename C> bool Server::run(W work, C clean) {
+  bool in = inbox->load(cpp::MemoryOrder::RELAXED);
+  bool out = outbox->load(cpp::MemoryOrder::RELAXED);
+  atomic_thread_fence(cpp::MemoryOrder::ACQUIRE);
+  // No work to do, exit.
+  if (!in & !out)
+    return false;
+  // Do work then write to the outbox.
+  if (in & !out) {
+    work(buffer);
+    atomic_thread_fence(cpp::MemoryOrder::RELEASE);
+    outbox->store(1, cpp::MemoryOrder::RELEASE);
+    out = 1;
+  }
+  // Wait for the client to read the result.
+  if (in & out) {
+    while (in)
+      in = inbox->load(cpp::MemoryOrder::RELAXED);
+    atomic_thread_fence(cpp::MemoryOrder::ACQUIRE);
+  }
+  // Clean up the buffer and signal the client.
+  if (!in & out) {
+    clean(buffer);
+    atomic_thread_fence(cpp::MemoryOrder::RELEASE);
+    outbox->store(0, cpp::MemoryOrder::RELEASE);
+    out = 0;
+  }
+
+  return true;
+}
+
+} // namespace rpc
+} // namespace __llvm_libc
+
+#endif
diff --git a/libc/src/__support/RPC/rpc_client.cpp b/libc/src/__support/RPC/rpc_client.cpp
new file mode 100644
index 0000000000000..3e64fe5ec67c8
--- /dev/null
+++ b/libc/src/__support/RPC/rpc_client.cpp
@@ -0,0 +1,27 @@
+//===-- Shared memory RPC client instantiation ------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC_SUPPORT_RPC_RPC_CLIENT_H
+#define LLVM_LIBC_SRC_SUPPORT_RPC_RPC_CLIENT_H
+
+#include "rpc.h"
+
+namespace __llvm_libc {
+namespace rpc {
+
+/// The libc client instance used to communicate with the server.
+Client client;
+
+/// Externally visible symbol to signify the usage of an RPC client to
+/// whomever needs to run the server.
+extern "C" [[gnu::visibility("protected")]] const bool __llvm_libc_rpc = false;
+
+} // namespace rpc
+} // namespace __llvm_libc
+
+#endif
diff --git a/libc/src/__support/RPC/rpc_client.h b/libc/src/__support/RPC/rpc_client.h
new file mode 100644
index 0000000000000..509ec2f4185b9
--- /dev/null
+++ b/libc/src/__support/RPC/rpc_client.h
@@ -0,0 +1,23 @@
+//===-- Shared memory RPC client instantiation ------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC_SUPPORT_RPC_RPC_CLIENT_H
+#define LLVM_LIBC_SRC_SUPPORT_RPC_RPC_CLIENT_H
+
+#include "rpc.h"
+
+namespace __llvm_libc {
+namespace rpc {
+
+/// The libc client instance used to communicate with the server.
+extern Client client;
+
+} // namespace rpc
+} // namespace __llvm_libc
+
+#endif
diff --git a/libc/src/__support/UInt.h b/libc/src/__support/UInt.h
index 85de5b77bb22d..2cbd3dfea6fb8 100644
--- a/libc/src/__support/UInt.h
+++ b/libc/src/__support/UInt.h
@@ -27,8 +27,8 @@ template <size_t Bits> struct UInt {
 
   static_assert(Bits > 0 && Bits % 64 == 0,
                 "Number of bits in UInt should be a multiple of 64.");
-  static constexpr size_t WordCount = Bits / 64;
-  uint64_t val[WordCount];
+  static constexpr size_t WORDCOUNT = Bits / 64;
+  uint64_t val[WORDCOUNT];
 
   static constexpr uint64_t MASK32 = 0xFFFFFFFFu;
 
@@ -38,45 +38,45 @@ template <size_t Bits> struct UInt {
   constexpr UInt() {}
 
   constexpr UInt(const UInt<Bits> &other) {
-    for (size_t i = 0; i < WordCount; ++i)
+    for (size_t i = 0; i < WORDCOUNT; ++i)
       val[i] = other.val[i];
   }
 
   template <size_t OtherBits> constexpr UInt(const UInt<OtherBits> &other) {
     if (OtherBits >= Bits) {
-      for (size_t i = 0; i < WordCount; ++i)
+      for (size_t i = 0; i < WORDCOUNT; ++i)
         val[i] = other[i];
     } else {
       size_t i = 0;
       for (; i < OtherBits / 64; ++i)
         val[i] = other[i];
-      for (; i < WordCount; ++i)
+      for (; i < WORDCOUNT; ++i)
         val[i] = 0;
     }
   }
 
   // Construct a UInt from a C array.
-  template <size_t N, enable_if_t<N <= WordCount, int> = 0>
+  template <size_t N, enable_if_t<N <= WORDCOUNT, int> = 0>
   constexpr UInt(const uint64_t (&nums)[N]) {
-    size_t min_wordcount = N < WordCount ? N : WordCount;
+    size_t min_wordcount = N < WORDCOUNT ? N : WORDCOUNT;
     size_t i = 0;
     for (; i < min_wordcount; ++i)
       val[i] = nums[i];
 
     // If nums doesn't completely fill val, then fill the rest with zeroes.
-    for (; i < WordCount; ++i)
+    for (; i < WORDCOUNT; ++i)
       val[i] = 0;
   }
 
   // Initialize the first word to |v| and the rest to 0.
   constexpr UInt(uint64_t v) {
     val[0] = v;
-    for (size_t i = 1; i < WordCount; ++i) {
+    for (size_t i = 1; i < WORDCOUNT; ++i) {
       val[i] = 0;
     }
   }
-  constexpr explicit UInt(const cpp::array<uint64_t, WordCount> &words) {
-    for (size_t i = 0; i < WordCount; ++i)
+  constexpr explicit UInt(const cpp::array<uint64_t, WORDCOUNT> &words) {
+    for (size_t i = 0; i < WORDCOUNT; ++i)
       val[i] = words[i];
   }
 
@@ -91,13 +91,13 @@ template <size_t Bits> struct UInt {
   }
 
   UInt<Bits> &operator=(const UInt<Bits> &other) {
-    for (size_t i = 0; i < WordCount; ++i)
+    for (size_t i = 0; i < WORDCOUNT; ++i)
       val[i] = other.val[i];
     return *this;
   }
 
   constexpr bool is_zero() const {
-    for (size_t i = 0; i < WordCount; ++i) {
+    for (size_t i = 0; i < WORDCOUNT; ++i) {
       if (val[i] != 0)
         return false;
     }
@@ -108,7 +108,7 @@ template <size_t Bits> struct UInt {
   // Returns the carry value produced by the addition operation.
   constexpr uint64_t add(const UInt<Bits> &x) {
     SumCarry<uint64_t> s{0, 0};
-    for (size_t i = 0; i < WordCount; ++i) {
+    for (size_t i = 0; i < WORDCOUNT; ++i) {
       s = add_with_carry(val[i], x.val[i], s.carry);
       val[i] = s.sum;
     }
@@ -118,7 +118,7 @@ template <size_t Bits> struct UInt {
   constexpr UInt<Bits> operator+(const UInt<Bits> &other) const {
     UInt<Bits> result;
     SumCarry<uint64_t> s{0, 0};
-    for (size_t i = 0; i < WordCount; ++i) {
+    for (size_t i = 0; i < WORDCOUNT; ++i) {
       s = add_with_carry(val[i], other.val[i], s.carry);
       result.val[i] = s.sum;
     }
@@ -134,7 +134,7 @@ template <size_t Bits> struct UInt {
   // Returns the carry value produced by the subtraction operation.
   constexpr uint64_t sub(const UInt<Bits> &x) {
     DiffBorrow<uint64_t> d{0, 0};
-    for (size_t i = 0; i < WordCount; ++i) {
+    for (size_t i = 0; i < WORDCOUNT; ++i) {
       d = sub_with_borrow(val[i], x.val[i], d.borrow);
       val[i] = d.diff;
     }
@@ -144,7 +144,7 @@ template <size_t Bits> struct UInt {
   constexpr UInt<Bits> operator-(const UInt<Bits> &other) const {
     UInt<Bits> result;
     DiffBorrow<uint64_t> d{0, 0};
-    for (size_t i = 0; i < WordCount; ++i) {
+    for (size_t i = 0; i < WORDCOUNT; ++i) {
       d = sub_with_borrow(val[i], other.val[i], d.borrow);
       result.val[i] = d.diff;
     }
@@ -166,7 +166,7 @@ template <size_t Bits> struct UInt {
   constexpr uint64_t mul(uint64_t x) {
     UInt<128> partial_sum(0);
     uint64_t carry = 0;
-    for (size_t i = 0; i < WordCount; ++i) {
+    for (size_t i = 0; i < WORDCOUNT; ++i) {
       NumberPair<uint64_t> prod = full_mul(val[i], x);
       UInt<128> tmp({prod.lo, prod.hi});
       carry += partial_sum.add(tmp);
@@ -179,13 +179,13 @@ template <size_t Bits> struct UInt {
   }
 
   constexpr UInt<Bits> operator*(const UInt<Bits> &other) const {
-    if constexpr (WordCount == 1) {
+    if constexpr (WORDCOUNT == 1) {
       return {val[0] * other.val[0]};
     } else {
       UInt<Bits> result(0);
       UInt<128> partial_sum(0);
       uint64_t carry = 0;
-      for (size_t i = 0; i < WordCount; ++i) {
+      for (size_t i = 0; i < WORDCOUNT; ++i) {
         for (size_t j = 0; j <= i; j++) {
           NumberPair<uint64_t> prod = full_mul(val[j], other.val[i - j]);
           UInt<128> tmp({prod.lo, prod.hi});
@@ -206,10 +206,11 @@ template <size_t Bits> struct UInt {
     UInt<Bits + OtherBits> result(0);
     UInt<128> partial_sum(0);
     uint64_t carry = 0;
-    constexpr size_t OtherWordCount = UInt<OtherBits>::WordCount;
-    for (size_t i = 0; i <= WordCount + OtherWordCount - 2; ++i) {
-      const size_t lower_idx = i < OtherWordCount ? 0 : i - OtherWordCount + 1;
-      const size_t upper_idx = i < WordCount ? i : WordCount - 1;
+    constexpr size_t OTHER_WORDCOUNT = UInt<OtherBits>::WORDCOUNT;
+    for (size_t i = 0; i <= WORDCOUNT + OTHER_WORDCOUNT - 2; ++i) {
+      const size_t lower_idx =
+          i < OTHER_WORDCOUNT ? 0 : i - OTHER_WORDCOUNT + 1;
+      const size_t upper_idx = i < WORDCOUNT ? i : WORDCOUNT - 1;
       for (size_t j = lower_idx; j <= upper_idx; ++j) {
         NumberPair<uint64_t> prod = full_mul(val[j], other.val[i - j]);
         UInt<128> tmp({prod.lo, prod.hi});
@@ -220,7 +221,7 @@ template <size_t Bits> struct UInt {
       partial_sum.val[1] = carry;
       carry = 0;
     }
-    result.val[WordCount + OtherWordCount - 1] = partial_sum.val[0];
+    result.val[WORDCOUNT + OTHER_WORDCOUNT - 1] = partial_sum.val[0];
     return result;
   }
 
@@ -228,7 +229,7 @@ template <size_t Bits> struct UInt {
   // `Bits` least significant bits of the full product, while this function will
   // approximate `Bits` most significant bits of the full product with errors
   // bounded by:
-  //   0 <= (a.full_mul(b) >> Bits) - a.quick_mul_hi(b)) <= WordCount - 1.
+  //   0 <= (a.full_mul(b) >> Bits) - a.quick_mul_hi(b)) <= WORDCOUNT - 1.
   //
   // An example usage of this is to quickly (but less accurately) compute the
   // product of (normalized) mantissas of floating point numbers:
@@ -240,7 +241,7 @@ template <size_t Bits> struct UInt {
   //
   // Performance summary:
   //   Number of 64-bit x 64-bit -> 128-bit multiplications performed.
-  //   Bits  WordCount  ful_mul  quick_mul_hi  Error bound
+  //   Bits  WORDCOUNT  ful_mul  quick_mul_hi  Error bound
   //    128      2         4           3            1
   //    196      3         9           6            2
   //    256      4        16          10            3
@@ -249,26 +250,26 @@ template <size_t Bits> struct UInt {
     UInt<Bits> result(0);
     UInt<128> partial_sum(0);
     uint64_t carry = 0;
-    // First round of accumulation for those at WordCount - 1 in the full
+    // First round of accumulation for those at WORDCOUNT - 1 in the full
     // product.
-    for (size_t i = 0; i < WordCount; ++i) {
+    for (size_t i = 0; i < WORDCOUNT; ++i) {
       NumberPair<uint64_t> prod =
-          full_mul(val[i], other.val[WordCount - 1 - i]);
+          full_mul(val[i], other.val[WORDCOUNT - 1 - i]);
       UInt<128> tmp({prod.lo, prod.hi});
       carry += partial_sum.add(tmp);
     }
-    for (size_t i = WordCount; i < 2 * WordCount - 1; ++i) {
+    for (size_t i = WORDCOUNT; i < 2 * WORDCOUNT - 1; ++i) {
       partial_sum.val[0] = partial_sum.val[1];
       partial_sum.val[1] = carry;
       carry = 0;
-      for (size_t j = i - WordCount + 1; j < WordCount; ++j) {
+      for (size_t j = i - WORDCOUNT + 1; j < WORDCOUNT; ++j) {
         NumberPair<uint64_t> prod = full_mul(val[j], other.val[i - j]);
         UInt<128> tmp({prod.lo, prod.hi});
         carry += partial_sum.add(tmp);
       }
-      result.val[i - WordCount] = partial_sum.val[0];
+      result.val[i - WORDCOUNT] = partial_sum.val[0];
     }
-    result.val[WordCount - 1] = partial_sum.val[1];
+    result.val[WORDCOUNT - 1] = partial_sum.val[1];
     return result;
   }
 
@@ -338,7 +339,7 @@ template <size_t Bits> struct UInt {
 
   constexpr uint64_t clz() {
     uint64_t leading_zeroes = 0;
-    for (size_t i = WordCount; i > 0; --i) {
+    for (size_t i = WORDCOUNT; i > 0; --i) {
       if (val[i - 1] == 0) {
         leading_zeroes += sizeof(uint64_t) * 8;
       } else {
@@ -370,17 +371,17 @@ template <size_t Bits> struct UInt {
 
     const size_t drop = s / 64;  // Number of words to drop
     const size_t shift = s % 64; // Bits to shift in the remaining words.
-    size_t i = WordCount;
+    size_t i = WORDCOUNT;
 
-    if (drop < WordCount) {
-      i = WordCount - 1;
+    if (drop < WORDCOUNT) {
+      i = WORDCOUNT - 1;
       if (shift > 0) {
-        for (size_t j = WordCount - 1 - drop; j > 0; --i, --j) {
+        for (size_t j = WORDCOUNT - 1 - drop; j > 0; --i, --j) {
           val[i] = (val[j] << shift) | (val[j - 1] >> (64 - shift));
         }
         val[i] = val[0] << shift;
       } else {
-        for (size_t j = WordCount - 1 - drop; j > 0; --i, --j) {
+        for (size_t j = WORDCOUNT - 1 - drop; j > 0; --i, --j) {
           val[i] = val[j];
         }
         val[i] = val[0];
@@ -427,21 +428,21 @@ template <size_t Bits> struct UInt {
 
     size_t i = 0;
 
-    if (drop < WordCount) {
+    if (drop < WORDCOUNT) {
       if (shift > 0) {
-        for (size_t j = drop; j < WordCount - 1; ++i, ++j) {
+        for (size_t j = drop; j < WORDCOUNT - 1; ++i, ++j) {
           val[i] = (val[j] >> shift) | (val[j + 1] << (64 - shift));
         }
-        val[i] = val[WordCount - 1] >> shift;
+        val[i] = val[WORDCOUNT - 1] >> shift;
         ++i;
       } else {
-        for (size_t j = drop; j < WordCount; ++i, ++j) {
+        for (size_t j = drop; j < WORDCOUNT; ++i, ++j) {
           val[i] = val[j];
         }
       }
     }
 
-    for (; i < WordCount; ++i) {
+    for (; i < WORDCOUNT; ++i) {
       val[i] = 0;
     }
   }
@@ -459,52 +460,52 @@ template <size_t Bits> struct UInt {
 
   constexpr UInt<Bits> operator&(const UInt<Bits> &other) const {
     UInt<Bits> result;
-    for (size_t i = 0; i < WordCount; ++i)
+    for (size_t i = 0; i < WORDCOUNT; ++i)
       result.val[i] = val[i] & other.val[i];
     return result;
   }
 
   constexpr UInt<Bits> &operator&=(const UInt<Bits> &other) {
-    for (size_t i = 0; i < WordCount; ++i)
+    for (size_t i = 0; i < WORDCOUNT; ++i)
       val[i] &= other.val[i];
     return *this;
   }
 
   constexpr UInt<Bits> operator|(const UInt<Bits> &other) const {
     UInt<Bits> result;
-    for (size_t i = 0; i < WordCount; ++i)
+    for (size_t i = 0; i < WORDCOUNT; ++i)
       result.val[i] = val[i] | other.val[i];
     return result;
   }
 
   constexpr UInt<Bits> &operator|=(const UInt<Bits> &other) {
-    for (size_t i = 0; i < WordCount; ++i)
+    for (size_t i = 0; i < WORDCOUNT; ++i)
       val[i] |= other.val[i];
     return *this;
   }
 
   constexpr UInt<Bits> operator^(const UInt<Bits> &other) const {
     UInt<Bits> result;
-    for (size_t i = 0; i < WordCount; ++i)
+    for (size_t i = 0; i < WORDCOUNT; ++i)
       result.val[i] = val[i] ^ other.val[i];
     return result;
   }
 
   constexpr UInt<Bits> &operator^=(const UInt<Bits> &other) {
-    for (size_t i = 0; i < WordCount; ++i)
+    for (size_t i = 0; i < WORDCOUNT; ++i)
       val[i] ^= other.val[i];
     return *this;
   }
 
   constexpr UInt<Bits> operator~() const {
     UInt<Bits> result;
-    for (size_t i = 0; i < WordCount; ++i)
+    for (size_t i = 0; i < WORDCOUNT; ++i)
       result.val[i] = ~val[i];
     return result;
   }
 
   constexpr bool operator==(const UInt<Bits> &other) const {
-    for (size_t i = 0; i < WordCount; ++i) {
+    for (size_t i = 0; i < WORDCOUNT; ++i) {
       if (val[i] != other.val[i])
         return false;
     }
@@ -512,7 +513,7 @@ template <size_t Bits> struct UInt {
   }
 
   constexpr bool operator!=(const UInt<Bits> &other) const {
-    for (size_t i = 0; i < WordCount; ++i) {
+    for (size_t i = 0; i < WORDCOUNT; ++i) {
       if (val[i] != other.val[i])
         return true;
     }
@@ -520,7 +521,7 @@ template <size_t Bits> struct UInt {
   }
 
   constexpr bool operator>(const UInt<Bits> &other) const {
-    for (size_t i = WordCount; i > 0; --i) {
+    for (size_t i = WORDCOUNT; i > 0; --i) {
       uint64_t word = val[i - 1];
       uint64_t other_word = other.val[i - 1];
       if (word > other_word)
@@ -533,7 +534,7 @@ template <size_t Bits> struct UInt {
   }
 
   constexpr bool operator>=(const UInt<Bits> &other) const {
-    for (size_t i = WordCount; i > 0; --i) {
+    for (size_t i = WORDCOUNT; i > 0; --i) {
       uint64_t word = val[i - 1];
       uint64_t other_word = other.val[i - 1];
       if (word > other_word)
@@ -546,7 +547,7 @@ template <size_t Bits> struct UInt {
   }
 
   constexpr bool operator<(const UInt<Bits> &other) const {
-    for (size_t i = WordCount; i > 0; --i) {
+    for (size_t i = WORDCOUNT; i > 0; --i) {
       uint64_t word = val[i - 1];
       uint64_t other_word = other.val[i - 1];
       if (word > other_word)
@@ -559,7 +560,7 @@ template <size_t Bits> struct UInt {
   }
 
   constexpr bool operator<=(const UInt<Bits> &other) const {
-    for (size_t i = WordCount; i > 0; --i) {
+    for (size_t i = WORDCOUNT; i > 0; --i) {
       uint64_t word = val[i - 1];
       uint64_t other_word = other.val[i - 1];
       if (word > other_word)
diff --git a/libc/src/__support/macros/properties/architectures.h b/libc/src/__support/macros/properties/architectures.h
index 238e5feef192b..1247fd6ef5cbc 100644
--- a/libc/src/__support/macros/properties/architectures.h
+++ b/libc/src/__support/macros/properties/architectures.h
@@ -45,6 +45,10 @@
 #define LIBC_TARGET_ARCH_IS_AARCH64
 #endif
 
+#if defined(__riscv) && (__riscv_xlen == 64)
+#define LIBC_TARGET_ARCH_IS_RISCV64
+#endif
+
 #if (defined(LIBC_TARGET_ARCH_IS_AARCH64) || defined(LIBC_TARGET_ARCH_IS_ARM))
 #define LIBC_TARGET_ARCH_IS_ANY_ARM
 #endif
diff --git a/libc/src/__support/macros/properties/cpu_features.h b/libc/src/__support/macros/properties/cpu_features.h
index b986351ecbec7..9884d561195f4 100644
--- a/libc/src/__support/macros/properties/cpu_features.h
+++ b/libc/src/__support/macros/properties/cpu_features.h
@@ -36,7 +36,8 @@
 #define LIBC_TARGET_CPU_HAS_AVX512BW
 #endif
 
-#if defined(__ARM_FEATURE_FMA) || defined(__AVX2__) || defined(__FMA__)
+#if defined(__ARM_FEATURE_FMA) || defined(__AVX2__) || defined(__FMA__) ||     \
+    defined(__riscv)
 #define LIBC_TARGET_CPU_HAS_FMA
 #endif
 
diff --git a/libc/src/__support/threads/CMakeLists.txt b/libc/src/__support/threads/CMakeLists.txt
index e71f436fe1751..b77eb0de0f8c1 100644
--- a/libc/src/__support/threads/CMakeLists.txt
+++ b/libc/src/__support/threads/CMakeLists.txt
@@ -54,16 +54,11 @@ if(TARGET libc.src.__support.threads.${LIBC_TARGET_OS}.thread)
   )
 endif()
 
-if(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${LIBC_TARGET_OS}/callonce.cpp)
+if(TARGET libc.src.__support.threads.${LIBC_TARGET_OS}.callonce)
   add_object_library(
     callonce
-    SRCS
-      ${LIBC_TARGET_OS}/callonce.cpp
-    HDRS
-      callonce.h
+    ALIAS
     DEPENDS
-      libc.include.sys_syscall
-      libc.src.__support.CPP.atomic
-      libc.src.__support.OSUtil.osutil
+      .${LIBC_TARGET_OS}.callonce
   )
 endif()
diff --git a/libc/src/__support/threads/linux/CMakeLists.txt b/libc/src/__support/threads/linux/CMakeLists.txt
index 62351da5e529a..642eead727726 100644
--- a/libc/src/__support/threads/linux/CMakeLists.txt
+++ b/libc/src/__support/threads/linux/CMakeLists.txt
@@ -40,3 +40,15 @@ add_object_library(
     -fno-omit-frame-pointer # This allows us to sniff out the thread args from
                             # the new thread's stack reliably.
 )
+
+add_object_library(
+  callonce
+  SRCS
+    callonce.cpp
+  HDRS
+    ../callonce.h
+  DEPENDS
+    libc.include.sys_syscall
+    libc.src.__support.CPP.atomic
+    libc.src.__support.OSUtil.osutil
+)
diff --git a/libc/src/__support/threads/linux/thread.cpp b/libc/src/__support/threads/linux/thread.cpp
index f158f16ec2dfd..e9b95eed99a58 100644
--- a/libc/src/__support/threads/linux/thread.cpp
+++ b/libc/src/__support/threads/linux/thread.cpp
@@ -55,6 +55,16 @@ static constexpr unsigned CLONE_SYSCALL_FLAGS =
                            // wake the joining thread.
     | CLONE_SETTLS;        // Setup the thread pointer of the new thread.
 
+#ifdef LIBC_TARGET_ARCH_IS_AARCH64
+#define CLONE_RESULT_REGISTER "x0"
+#elif defined(LIBC_TARGET_ARCH_IS_RISCV64)
+#define CLONE_RESULT_REGISTER "t0"
+#elif defined(LIBC_TARGET_ARCH_IS_X86_64)
+#define CLONE_RESULT_REGISTER "rax"
+#else
+#error "CLONE_RESULT_REGISTER not defined for your target architecture"
+#endif
+
 LIBC_INLINE ErrorOr<void *> alloc_stack(size_t size) {
   long mmap_result =
       __llvm_libc::syscall_impl(MMAP_SYSCALL_NUMBER,
@@ -111,6 +121,10 @@ __attribute__((always_inline)) inline uintptr_t get_start_args_addr() {
   // is set to the stack pointer where start args are stored. So, we fetch
   // from there.
   return reinterpret_cast<uintptr_t>(__builtin_frame_address(1));
+#elif defined(LIBC_TARGET_ARCH_IS_RISCV64)
+  // The current frame pointer is the previous stack pointer where the start
+  // args are stored.
+  return reinterpret_cast<uintptr_t>(__builtin_frame_address(0));
 #endif
 }
 
@@ -190,16 +204,17 @@ int Thread::run(ThreadStyle style, ThreadRunner runner, void *arg, void *stack,
   // Also, we want the result of the syscall to be in a register as the child
   // thread gets a completely different stack after it is created. The stack
   // variables from this function will not be availalbe to the child thread.
-#ifdef LIBC_TARGET_ARCH_IS_X86_64
-  long register clone_result asm("rax");
+#if defined(LIBC_TARGET_ARCH_IS_X86_64)
+  long register clone_result asm(CLONE_RESULT_REGISTER);
   clone_result = __llvm_libc::syscall_impl(
       SYS_clone, CLONE_SYSCALL_FLAGS, adjusted_stack,
       &attrib->tid,    // The address where the child tid is written
       &clear_tid->val, // The futex where the child thread status is signalled
       tls.tp           // The thread pointer value for the new thread.
   );
-#elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
-  long register clone_result asm("x0");
+#elif defined(LIBC_TARGET_ARCH_IS_AARCH64) ||                                  \
+    defined(LIBC_TARGET_ARCH_IS_RISCV64)
+  long register clone_result asm(CLONE_RESULT_REGISTER);
   clone_result = __llvm_libc::syscall_impl(
       SYS_clone, CLONE_SYSCALL_FLAGS, adjusted_stack,
       &attrib->tid,   // The address where the child tid is written
@@ -220,6 +235,8 @@ int Thread::run(ThreadStyle style, ThreadRunner runner, void *arg, void *stack,
 #else
     asm volatile("mov x29, sp");
 #endif
+#elif defined(LIBC_TARGET_ARCH_IS_RISCV64)
+    asm volatile("mv fp, sp");
 #endif
     start_thread();
   } else if (clone_result < 0) {
diff --git a/libc/src/__support/threads/thread.h b/libc/src/__support/threads/thread.h
index 127182db588e7..9197d0f3873a9 100644
--- a/libc/src/__support/threads/thread.h
+++ b/libc/src/__support/threads/thread.h
@@ -37,7 +37,8 @@ union ThreadReturnValue {
 };
 
 #if (defined(LIBC_TARGET_ARCH_IS_AARCH64) ||                                   \
-     defined(LIBC_TARGET_ARCH_IS_X86_64))
+     defined(LIBC_TARGET_ARCH_IS_X86_64) ||                                    \
+     defined(LIBC_TARGET_ARCH_IS_RISCV64))
 constexpr unsigned int STACK_ALIGNMENT = 16;
 #endif
 // TODO: Provide stack alignment requirements for other architectures.
diff --git a/libc/src/dirent/closedir.cpp b/libc/src/dirent/closedir.cpp
index 39dfc0ccc5f0c..2863106b4ce89 100644
--- a/libc/src/dirent/closedir.cpp
+++ b/libc/src/dirent/closedir.cpp
@@ -10,9 +10,9 @@
 
 #include "src/__support/File/dir.h"
 #include "src/__support/common.h"
+#include "src/errno/libc_errno.h"
 
 #include <dirent.h>
-#include <errno.h>
 
 namespace __llvm_libc {
 
@@ -20,7 +20,7 @@ LLVM_LIBC_FUNCTION(int, closedir, (::DIR * dir)) {
   auto *d = reinterpret_cast<__llvm_libc::Dir *>(dir);
   int retval = d->close();
   if (retval != 0) {
-    errno = retval;
+    libc_errno = retval;
     return -1;
   }
   return 0;
diff --git a/libc/src/dirent/opendir.cpp b/libc/src/dirent/opendir.cpp
index cf5ee18d3f050..06749e82c75f1 100644
--- a/libc/src/dirent/opendir.cpp
+++ b/libc/src/dirent/opendir.cpp
@@ -10,16 +10,16 @@
 
 #include "src/__support/File/dir.h"
 #include "src/__support/common.h"
+#include "src/errno/libc_errno.h"
 
 #include <dirent.h>
-#include <errno.h>
 
 namespace __llvm_libc {
 
 LLVM_LIBC_FUNCTION(::DIR *, opendir, (const char *name)) {
   auto dir = Dir::open(name);
   if (!dir) {
-    errno = dir.error();
+    libc_errno = dir.error();
     return nullptr;
   }
   return reinterpret_cast<DIR *>(dir.value());
diff --git a/libc/src/dirent/readdir.cpp b/libc/src/dirent/readdir.cpp
index b9ce30140ee0e..10040aa15b63a 100644
--- a/libc/src/dirent/readdir.cpp
+++ b/libc/src/dirent/readdir.cpp
@@ -10,9 +10,9 @@
 
 #include "src/__support/File/dir.h"
 #include "src/__support/common.h"
+#include "src/errno/libc_errno.h"
 
 #include <dirent.h>
-#include <errno.h>
 
 namespace __llvm_libc {
 
@@ -20,7 +20,7 @@ LLVM_LIBC_FUNCTION(struct ::dirent *, readdir, (::DIR * dir)) {
   auto *d = reinterpret_cast<__llvm_libc::Dir *>(dir);
   auto dirent_val = d->read();
   if (!dirent_val) {
-    errno = dirent_val.error();
+    libc_errno = dirent_val.error();
     return nullptr;
   }
   return dirent_val;
diff --git a/libc/src/errno/CMakeLists.txt b/libc/src/errno/CMakeLists.txt
index 61b56047ce0a4..2fa1381a70cc6 100644
--- a/libc/src/errno/CMakeLists.txt
+++ b/libc/src/errno/CMakeLists.txt
@@ -4,7 +4,6 @@ add_entrypoint_object(
     libc_errno.cpp
   HDRS
     libc_errno.h     # Include this
-    llvmlibc_errno.h # DEPRECATED: Will be removed soon
   DEPENDS
     libc.include.errno
 )
diff --git a/libc/src/errno/libc_errno.cpp b/libc/src/errno/libc_errno.cpp
index 98a76c2cbcf49..b8c276cb853fb 100644
--- a/libc/src/errno/libc_errno.cpp
+++ b/libc/src/errno/libc_errno.cpp
@@ -9,9 +9,9 @@
 namespace __llvm_libc {
 
 extern "C" {
+#ifdef LIBC_COPT_PUBLIC_PACKAGING
 // TODO: Declare __llvmlibc_errno only under LIBC_COPT_PUBLIC_PACKAGING and
 // __llvmlibc_internal_errno otherwise.
-//
 // In overlay mode, this will be an unused thread local variable as libc_errno
 // will resolve to errno from the system libc's errno.h. In full build mode
 // however, libc_errno will resolve to this thread local variable via the errno
@@ -19,7 +19,9 @@ extern "C" {
 // TODO: Use a macro to distinguish full build and overlay build which can be
 //       used to exclude __llvmlibc_errno under overlay build.
 thread_local int __llvmlibc_errno;
+#else
 thread_local int __llvmlibc_internal_errno;
+#endif
 } // extern "C"
 
 } // namespace __llvm_libc
diff --git a/libc/src/errno/libc_errno.h b/libc/src/errno/libc_errno.h
index 586d88feba906..28f8d0d13e736 100644
--- a/libc/src/errno/libc_errno.h
+++ b/libc/src/errno/libc_errno.h
@@ -29,7 +29,7 @@ extern thread_local int __llvmlibc_internal_errno;
 // libc_errno, this header file will be "shipped" via an add_entrypoint_object
 // target. At which point libc_errno, should point to __llvmlibc_internal_errno
 // if LIBC_COPT_PUBLIC_PACKAGING is not defined.
-#define libc_errno errno
+#define libc_errno __llvm_libc::__llvmlibc_internal_errno
 
 } // namespace __llvm_libc
 #endif
diff --git a/libc/src/errno/llvmlibc_errno.h b/libc/src/errno/llvmlibc_errno.h
deleted file mode 100644
index eb1f66c2e4080..0000000000000
--- a/libc/src/errno/llvmlibc_errno.h
+++ /dev/null
@@ -1,21 +0,0 @@
-//===-- Implementation header for errno -------------------------*- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LLVM_LIBC_SRC_ERRNO_LLVMLIBC_ERRNO_H
-#define LLVM_LIBC_SRC_ERRNO_LLVMLIBC_ERRNO_H
-
-#include <errno.h>
-
-// DEPRECATED: Use libc_errno from libc_errno.h instead. This macro is only
-// present to facilitate gradual transition (as in, in multiple simple patches)
-// to libc_errno.
-// TODO: After all of libc/src and libc/test is switched over to use libc_errno,
-// remove this macro and header file.
-#define llvmlibc_errno errno
-
-#endif // LLVM_LIBC_SRC_ERRNO_LLVMLIBC_ERRNO_H
diff --git a/libc/src/fcntl/linux/CMakeLists.txt b/libc/src/fcntl/linux/CMakeLists.txt
index 926292ca93cd2..87b8d4695c4fc 100644
--- a/libc/src/fcntl/linux/CMakeLists.txt
+++ b/libc/src/fcntl/linux/CMakeLists.txt
@@ -5,7 +5,6 @@ add_entrypoint_object(
   HDRS
     ../creat.h
   DEPENDS
-    libc.include.errno
     libc.include.fcntl
     libc.src.__support.OSUtil.osutil
     libc.src.errno.errno
@@ -18,7 +17,6 @@ add_entrypoint_object(
   HDRS
     ../open.h
   DEPENDS
-    libc.include.errno
     libc.include.fcntl
     libc.src.__support.OSUtil.osutil
     libc.src.errno.errno
@@ -31,7 +29,6 @@ add_entrypoint_object(
   HDRS
     ../openat.h
   DEPENDS
-    libc.include.errno
     libc.include.fcntl
     libc.src.__support.OSUtil.osutil
     libc.src.errno.errno
diff --git a/libc/src/fcntl/linux/creat.cpp b/libc/src/fcntl/linux/creat.cpp
index 4cf7b792b4d58..e2b0df7ef5f8b 100644
--- a/libc/src/fcntl/linux/creat.cpp
+++ b/libc/src/fcntl/linux/creat.cpp
@@ -10,8 +10,8 @@
 
 #include "src/__support/OSUtil/syscall.h" // For internal syscall function.
 #include "src/__support/common.h"
+#include "src/errno/libc_errno.h"
 
-#include <errno.h>
 #include <fcntl.h>
 #include <sys/syscall.h> // For syscall numbers.
 
@@ -29,7 +29,7 @@ LLVM_LIBC_FUNCTION(int, creat, (const char *path, int mode_flags)) {
   if (fd > 0)
     return fd;
 
-  errno = -fd;
+  libc_errno = -fd;
   return -1;
 }
 
diff --git a/libc/src/fcntl/linux/open.cpp b/libc/src/fcntl/linux/open.cpp
index 8fe68debcdcfe..e6ce370d890ed 100644
--- a/libc/src/fcntl/linux/open.cpp
+++ b/libc/src/fcntl/linux/open.cpp
@@ -10,8 +10,8 @@
 
 #include "src/__support/OSUtil/syscall.h" // For internal syscall function.
 #include "src/__support/common.h"
+#include "src/errno/libc_errno.h"
 
-#include <errno.h>
 #include <fcntl.h>
 #include <stdarg.h>
 #include <sys/syscall.h> // For syscall numbers.
@@ -38,7 +38,7 @@ LLVM_LIBC_FUNCTION(int, open, (const char *path, int flags, ...)) {
   if (fd > 0)
     return fd;
 
-  errno = -fd;
+  libc_errno = -fd;
   return -1;
 }
 
diff --git a/libc/src/fcntl/linux/openat.cpp b/libc/src/fcntl/linux/openat.cpp
index 23e2c6b89cdcb..d8a1faecd8b73 100644
--- a/libc/src/fcntl/linux/openat.cpp
+++ b/libc/src/fcntl/linux/openat.cpp
@@ -10,8 +10,8 @@
 
 #include "src/__support/OSUtil/syscall.h" // For internal syscall function.
 #include "src/__support/common.h"
+#include "src/errno/libc_errno.h"
 
-#include <errno.h>
 #include <fcntl.h>
 #include <stdarg.h>
 #include <sys/syscall.h> // For syscall numbers.
@@ -33,7 +33,7 @@ LLVM_LIBC_FUNCTION(int, openat, (int dfd, const char *path, int flags, ...)) {
   if (fd > 0)
     return fd;
 
-  errno = -fd;
+  libc_errno = -fd;
   return -1;
 }
 
diff --git a/libc/src/inttypes/CMakeLists.txt b/libc/src/inttypes/CMakeLists.txt
index f26273c3a5738..c3111ed80ff81 100644
--- a/libc/src/inttypes/CMakeLists.txt
+++ b/libc/src/inttypes/CMakeLists.txt
@@ -6,6 +6,7 @@ add_entrypoint_object(
     strtoimax.h
   DEPENDS
     libc.src.__support.str_to_integer
+    libc.src.errno.errno
 )
 
 add_entrypoint_object(
@@ -16,6 +17,7 @@ add_entrypoint_object(
     strtoumax.h
   DEPENDS
     libc.src.__support.str_to_integer
+    libc.src.errno.errno
 )
 
 add_entrypoint_object(
diff --git a/libc/src/inttypes/strtoimax.cpp b/libc/src/inttypes/strtoimax.cpp
index 3fa22662b653e..5a2562380aa6a 100644
--- a/libc/src/inttypes/strtoimax.cpp
+++ b/libc/src/inttypes/strtoimax.cpp
@@ -9,6 +9,7 @@
 #include "src/inttypes/strtoimax.h"
 #include "src/__support/common.h"
 #include "src/__support/str_to_integer.h"
+#include "src/errno/libc_errno.h"
 
 namespace __llvm_libc {
 
@@ -17,7 +18,7 @@ LLVM_LIBC_FUNCTION(intmax_t, strtoimax,
                     int base)) {
   auto result = internal::strtointeger<intmax_t>(str, base);
   if (result.has_error())
-    errno = result.error;
+    libc_errno = result.error;
 
   if (str_end != nullptr)
     *str_end = const_cast<char *>(str + result.parsed_len);
diff --git a/libc/src/inttypes/strtoumax.cpp b/libc/src/inttypes/strtoumax.cpp
index 6c1ecb6d75814..0e33c55ec3e2d 100644
--- a/libc/src/inttypes/strtoumax.cpp
+++ b/libc/src/inttypes/strtoumax.cpp
@@ -9,6 +9,7 @@
 #include "src/inttypes/strtoumax.h"
 #include "src/__support/common.h"
 #include "src/__support/str_to_integer.h"
+#include "src/errno/libc_errno.h"
 
 namespace __llvm_libc {
 
@@ -17,7 +18,7 @@ LLVM_LIBC_FUNCTION(uintmax_t, strtoumax,
                     int base)) {
   auto result = internal::strtointeger<uintmax_t>(str, base);
   if (result.has_error())
-    errno = result.error;
+    libc_errno = result.error;
 
   if (str_end != nullptr)
     *str_end = const_cast<char *>(str + result.parsed_len);
diff --git a/libc/src/math/generic/CMakeLists.txt b/libc/src/math/generic/CMakeLists.txt
index 922e16f4c05e6..09aefc67d1b51 100644
--- a/libc/src/math/generic/CMakeLists.txt
+++ b/libc/src/math/generic/CMakeLists.txt
@@ -774,6 +774,7 @@ add_entrypoint_object(
   HDRS
     ../log10.h
   DEPENDS
+    libc.src.__support.FPUtil.fenv_impl
     libc.src.__support.FPUtil.fp_bits
     libc.src.__support.FPUtil.multiply_add
     libc.src.__support.FPUtil.double_double
diff --git a/libc/src/math/generic/acosf.cpp b/libc/src/math/generic/acosf.cpp
index d75e36209a4a7..5835dfa617056 100644
--- a/libc/src/math/generic/acosf.cpp
+++ b/libc/src/math/generic/acosf.cpp
@@ -79,8 +79,7 @@ LLVM_LIBC_FUNCTION(float, acosf, (float x)) {
       fputil::set_errno_if_required(EDOM);
       fputil::raise_except_if_required(FE_INVALID);
     }
-    return x +
-           FPBits::build_nan(1 << (fputil::MantissaWidth<float>::VALUE - 1));
+    return x + FPBits::build_quiet_nan(0);
   }
 
   // When 0.5 < |x| <= 1, we perform range reduction as follow:
diff --git a/libc/src/math/generic/acoshf.cpp b/libc/src/math/generic/acoshf.cpp
index 12e0796c91301..ac225fe5a808f 100644
--- a/libc/src/math/generic/acoshf.cpp
+++ b/libc/src/math/generic/acoshf.cpp
@@ -25,7 +25,8 @@ LLVM_LIBC_FUNCTION(float, acoshf, (float x)) {
 
   if (LIBC_UNLIKELY(x < 1.0f)) {
     // x < 1.
-    fputil::set_except(FE_INVALID);
+    fputil::set_errno_if_required(EDOM);
+    fputil::raise_except_if_required(FE_INVALID);
     return FPBits_t::build_quiet_nan(0);
   }
 
diff --git a/libc/src/math/generic/atanhf.cpp b/libc/src/math/generic/atanhf.cpp
index 0a077007de784..b0c92fa8de87d 100644
--- a/libc/src/math/generic/atanhf.cpp
+++ b/libc/src/math/generic/atanhf.cpp
@@ -24,15 +24,15 @@ LLVM_LIBC_FUNCTION(float, atanhf, (float x)) {
     if (xbits.is_nan()) {
       return x;
     }
-    // |x| == 0
+    // |x| == 1.0
     if (x_abs == 0x3F80'0000U) {
-      fputil::set_except(FE_DIVBYZERO);
-      return with_errno(FPBits::inf(sign).get_val(), ERANGE);
+      fputil::set_errno_if_required(ERANGE);
+      fputil::raise_except_if_required(FE_DIVBYZERO);
+      return FPBits::inf(sign).get_val();
     } else {
-      fputil::set_except(FE_INVALID);
-      return with_errno(
-          FPBits::build_nan(1 << (fputil::MantissaWidth<float>::VALUE - 1)),
-          EDOM);
+      fputil::set_errno_if_required(EDOM);
+      fputil::raise_except_if_required(FE_INVALID);
+      return FPBits::build_quiet_nan(0);
     }
   }
 
diff --git a/libc/src/math/generic/cosf.cpp b/libc/src/math/generic/cosf.cpp
index 89b7d63b9e8c1..ef94804bda60d 100644
--- a/libc/src/math/generic/cosf.cpp
+++ b/libc/src/math/generic/cosf.cpp
@@ -117,8 +117,7 @@ LLVM_LIBC_FUNCTION(float, cosf, (float x)) {
       fputil::set_errno_if_required(EDOM);
       fputil::raise_except_if_required(FE_INVALID);
     }
-    return x +
-           FPBits::build_nan(1 << (fputil::MantissaWidth<float>::VALUE - 1));
+    return x + FPBits::build_quiet_nan(0);
   }
 
   // Combine the results with the sine of sum formula:
diff --git a/libc/src/math/generic/exp2f.cpp b/libc/src/math/generic/exp2f.cpp
index 116f40fc0a178..3c319d288e1c9 100644
--- a/libc/src/math/generic/exp2f.cpp
+++ b/libc/src/math/generic/exp2f.cpp
@@ -21,9 +21,9 @@
 
 namespace __llvm_libc {
 
-constexpr uint32_t exval1 = 0x3b42'9d37U;
-constexpr uint32_t exval2 = 0xbcf3'a937U;
-constexpr uint32_t exval_mask = exval1 & exval2;
+constexpr uint32_t EXVAL1 = 0x3b42'9d37U;
+constexpr uint32_t EXVAL2 = 0xbcf3'a937U;
+constexpr uint32_t EXVAL_MASK = EXVAL1 & EXVAL2;
 
 LLVM_LIBC_FUNCTION(float, exp2f, (float x)) {
   using FPBits = typename fputil::FPBits<float>;
@@ -73,11 +73,11 @@ LLVM_LIBC_FUNCTION(float, exp2f, (float x)) {
   }
 
   // Check exceptional values.
-  if (LIBC_UNLIKELY((x_u & exval_mask) == exval_mask)) {
-    if (LIBC_UNLIKELY(x_u == exval1)) { // x = 0x1.853a6ep-9f
+  if (LIBC_UNLIKELY((x_u & EXVAL_MASK) == EXVAL_MASK)) {
+    if (LIBC_UNLIKELY(x_u == EXVAL1)) { // x = 0x1.853a6ep-9f
       if (fputil::get_round() == FE_TONEAREST)
         return 0x1.00870ap+0f;
-    } else if (LIBC_UNLIKELY(x_u == exval2)) { // x = -0x1.e7526ep-6f
+    } else if (LIBC_UNLIKELY(x_u == EXVAL2)) { // x = -0x1.e7526ep-6f
       if (fputil::get_round() == FE_TONEAREST)
         return 0x1.f58d62p-1f;
     }
diff --git a/libc/src/math/generic/log10.cpp b/libc/src/math/generic/log10.cpp
index 10d57a8f30861..b2dd29f5f74d1 100644
--- a/libc/src/math/generic/log10.cpp
+++ b/libc/src/math/generic/log10.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/math/log10.h"
+#include "src/__support/FPUtil/FEnvImpl.h"
 #include "src/__support/FPUtil/FPBits.h"
 #include "src/__support/FPUtil/double_double.h"
 #include "src/__support/FPUtil/dyadic_float.h"
@@ -916,7 +917,6 @@ double log10_accurate(int e_x, int index, double m_x) {
 
   // Further range reductions.
   double scale = 0x1.0p+7;
-  const fputil::DyadicFloat<128> NEG_ONE(-1.0);
   for (size_t i = 0; i < R_STEPS; ++i) {
     scale *= 0x1.0p+4;
     int id = static_cast<int>(fputil::multiply_add(mx.hi, scale, 0x1.0p+4));
@@ -953,9 +953,13 @@ LLVM_LIBC_FUNCTION(double, log10, (double x)) {
                     xbits.uintval() > FPBits_t::MAX_NORMAL)) {
     if (xbits.is_zero()) {
       // return -Inf and raise FE_DIVBYZERO.
-      return -1.0 / 0.0;
+      fputil::set_errno_if_required(ERANGE);
+      fputil::raise_except_if_required(FE_DIVBYZERO);
+      return static_cast<double>(FPBits_t::neg_inf());
     }
     if (xbits.get_sign() && !xbits.is_nan()) {
+      fputil::set_errno_if_required(EDOM);
+      fputil::raise_except_if_required(FE_INVALID);
       return FPBits_t::build_quiet_nan(0);
     }
     if (xbits.is_inf_or_nan()) {
diff --git a/libc/src/math/generic/log10f.cpp b/libc/src/math/generic/log10f.cpp
index ba2f01601ec65..bcf17c672a638 100644
--- a/libc/src/math/generic/log10f.cpp
+++ b/libc/src/math/generic/log10f.cpp
@@ -152,12 +152,14 @@ LLVM_LIBC_FUNCTION(float, log10f, (float x)) {
   if (LIBC_UNLIKELY(x_u < FPBits::MIN_NORMAL || x_u > FPBits::MAX_NORMAL)) {
     if (xbits.is_zero()) {
       // Return -inf and raise FE_DIVBYZERO
-      fputil::raise_except(FE_DIVBYZERO);
+      fputil::set_errno_if_required(ERANGE);
+      fputil::raise_except_if_required(FE_DIVBYZERO);
       return static_cast<float>(FPBits::neg_inf());
     }
     if (xbits.get_sign() && !xbits.is_nan()) {
       // Return NaN and raise FE_INVALID
-      fputil::raise_except(FE_INVALID);
+      fputil::set_errno_if_required(EDOM);
+      fputil::raise_except_if_required(FE_INVALID);
       return FPBits::build_quiet_nan(0);
     }
     if (xbits.is_inf_or_nan()) {
diff --git a/libc/src/math/generic/log1pf.cpp b/libc/src/math/generic/log1pf.cpp
index 1a18e0cd6861f..bf6a91a5fc466 100644
--- a/libc/src/math/generic/log1pf.cpp
+++ b/libc/src/math/generic/log1pf.cpp
@@ -45,7 +45,8 @@ LIBC_INLINE float log(double x) {
 
   if (LIBC_UNLIKELY(x_u > FPBits::MAX_NORMAL)) {
     if (xbits.get_sign() && !xbits.is_nan()) {
-      fputil::raise_except(FE_INVALID);
+      fputil::set_errno_if_required(EDOM);
+      fputil::raise_except_if_required(FE_INVALID);
       return fputil::FPBits<float>::build_quiet_nan(0);
     }
     return static_cast<float>(x);
@@ -103,7 +104,8 @@ LLVM_LIBC_FUNCTION(float, log1pf, (float x)) {
     case 0xbd1d20afU: // x = -0x1.3a415ep-5f
       return fputil::round_result_slightly_up(-0x1.407112p-5f);
     case 0xbf800000U: // x = -1.0
-      fputil::raise_except(FE_DIVBYZERO);
+      fputil::set_errno_if_required(ERANGE);
+      fputil::raise_except_if_required(FE_DIVBYZERO);
       return static_cast<float>(fputil::FPBits<float>::neg_inf());
 #ifndef LIBC_TARGET_CPU_HAS_FMA
     case 0x4cc1c80bU: // x = 0x1.839016p+26f
diff --git a/libc/src/math/generic/log2f.cpp b/libc/src/math/generic/log2f.cpp
index 47abcf8d1f772..ae3c67a770aa6 100644
--- a/libc/src/math/generic/log2f.cpp
+++ b/libc/src/math/generic/log2f.cpp
@@ -121,10 +121,12 @@ LLVM_LIBC_FUNCTION(float, log2f, (float x)) {
   // Exceptional inputs.
   if (LIBC_UNLIKELY(x_u < FPBits::MIN_NORMAL || x_u > FPBits::MAX_NORMAL)) {
     if (xbits.is_zero()) {
-      fputil::raise_except(FE_DIVBYZERO);
+      fputil::set_errno_if_required(ERANGE);
+      fputil::raise_except_if_required(FE_DIVBYZERO);
       return static_cast<float>(FPBits::neg_inf());
     }
     if (xbits.get_sign() && !xbits.is_nan()) {
+      fputil::set_errno_if_required(EDOM);
       fputil::raise_except(FE_INVALID);
       return FPBits::build_quiet_nan(0);
     }
diff --git a/libc/src/math/generic/logf.cpp b/libc/src/math/generic/logf.cpp
index b579b5b7e863f..f2c1fa33309bf 100644
--- a/libc/src/math/generic/logf.cpp
+++ b/libc/src/math/generic/logf.cpp
@@ -96,12 +96,14 @@ LLVM_LIBC_FUNCTION(float, logf, (float x)) {
   if (LIBC_UNLIKELY(x_u < FPBits::MIN_NORMAL || x_u > FPBits::MAX_NORMAL)) {
     if (xbits.is_zero()) {
       // Return -inf and raise FE_DIVBYZERO
-      fputil::raise_except(FE_DIVBYZERO);
+      fputil::set_errno_if_required(ERANGE);
+      fputil::raise_except_if_required(FE_DIVBYZERO);
       return static_cast<float>(FPBits::neg_inf());
     }
     if (xbits.get_sign() && !xbits.is_nan()) {
       // Return NaN and raise FE_INVALID
-      fputil::raise_except(FE_INVALID);
+      fputil::set_errno_if_required(EDOM);
+      fputil::raise_except_if_required(FE_INVALID);
       return FPBits::build_quiet_nan(0);
     }
     if (xbits.is_inf_or_nan()) {
diff --git a/libc/src/math/generic/math_utils.h b/libc/src/math/generic/math_utils.h
index 7a43e58806329..0399f7d0adfff 100644
--- a/libc/src/math/generic/math_utils.h
+++ b/libc/src/math/generic/math_utils.h
@@ -12,7 +12,8 @@
 #include "src/__support/CPP/bit.h"
 #include "src/__support/CPP/type_traits.h"
 #include "src/__support/common.h"
-#include <errno.h>
+#include "src/errno/libc_errno.h"
+
 #include <math.h>
 
 #include <stdint.h>
@@ -52,7 +53,7 @@ template <> struct XFlowValues<double> {
 
 template <typename T> LIBC_INLINE T with_errno(T x, int err) {
   if (math_errhandling & MATH_ERRNO)
-    errno = err;
+    libc_errno = err;
   return x;
 }
 
diff --git a/libc/src/math/generic/sinf.cpp b/libc/src/math/generic/sinf.cpp
index 86f47ace2cc8e..1641c44e9fc00 100644
--- a/libc/src/math/generic/sinf.cpp
+++ b/libc/src/math/generic/sinf.cpp
@@ -138,8 +138,7 @@ LLVM_LIBC_FUNCTION(float, sinf, (float x)) {
       fputil::set_errno_if_required(EDOM);
       fputil::raise_except_if_required(FE_INVALID);
     }
-    return x +
-           FPBits::build_nan(1 << (fputil::MantissaWidth<float>::VALUE - 1));
+    return x + FPBits::build_quiet_nan(0);
   }
 
   // Combine the results with the sine of sum formula:
diff --git a/libc/src/math/generic/tanf.cpp b/libc/src/math/generic/tanf.cpp
index e2208203a2e8a..217664f8b2acb 100644
--- a/libc/src/math/generic/tanf.cpp
+++ b/libc/src/math/generic/tanf.cpp
@@ -114,8 +114,7 @@ LLVM_LIBC_FUNCTION(float, tanf, (float x)) {
         fputil::set_errno_if_required(EDOM);
         fputil::raise_except_if_required(FE_INVALID);
       }
-      return x +
-             FPBits::build_nan(1 << (fputil::MantissaWidth<float>::VALUE - 1));
+      return x + FPBits::build_quiet_nan(0);
     }
     // Other large exceptional values
     if (auto r = TANF_EXCEPTS.lookup_odd(x_abs, x_sign);
diff --git a/libc/src/sched/linux/CMakeLists.txt b/libc/src/sched/linux/CMakeLists.txt
index 2246066d7c524..82d3ab5c3a5ca 100644
--- a/libc/src/sched/linux/CMakeLists.txt
+++ b/libc/src/sched/linux/CMakeLists.txt
@@ -5,7 +5,6 @@ add_entrypoint_object(
   HDRS
     ../sched_getaffinity.h
   DEPENDS
-    libc.include.errno
     libc.include.sched
     libc.src.__support.OSUtil.osutil
     libc.src.errno.errno
@@ -18,7 +17,6 @@ add_entrypoint_object(
   HDRS
     ../sched_setaffinity.h
   DEPENDS
-    libc.include.errno
     libc.include.sched
     libc.src.__support.OSUtil.osutil
     libc.src.errno.errno
diff --git a/libc/src/sched/linux/sched_getaffinity.cpp b/libc/src/sched/linux/sched_getaffinity.cpp
index a4c3de9808f0b..f53bf4d99675a 100644
--- a/libc/src/sched/linux/sched_getaffinity.cpp
+++ b/libc/src/sched/linux/sched_getaffinity.cpp
@@ -10,8 +10,8 @@
 
 #include "src/__support/OSUtil/syscall.h" // For internal syscall function.
 #include "src/__support/common.h"
+#include "src/errno/libc_errno.h"
 
-#include <errno.h>
 #include <sched.h>
 #include <stdint.h>
 #include <sys/syscall.h> // For syscall numbers.
@@ -23,7 +23,7 @@ LLVM_LIBC_FUNCTION(int, sched_getaffinity,
   long ret =
       __llvm_libc::syscall_impl(SYS_sched_getaffinity, tid, cpuset_size, mask);
   if (ret < 0) {
-    errno = -ret;
+    libc_errno = -ret;
     return -1;
   }
   if (size_t(ret) < cpuset_size) {
diff --git a/libc/src/sched/linux/sched_setaffinity.cpp b/libc/src/sched/linux/sched_setaffinity.cpp
index 6feb9977912bb..07e51ee362af0 100644
--- a/libc/src/sched/linux/sched_setaffinity.cpp
+++ b/libc/src/sched/linux/sched_setaffinity.cpp
@@ -10,8 +10,8 @@
 
 #include "src/__support/OSUtil/syscall.h" // For internal syscall function.
 #include "src/__support/common.h"
+#include "src/errno/libc_errno.h"
 
-#include <errno.h>
 #include <sched.h>
 #include <sys/syscall.h> // For syscall numbers.
 
@@ -22,7 +22,7 @@ LLVM_LIBC_FUNCTION(int, sched_setaffinity,
   long ret =
       __llvm_libc::syscall_impl(SYS_sched_setaffinity, tid, cpuset_size, mask);
   if (ret < 0) {
-    errno = -ret;
+    libc_errno = -ret;
     return -1;
   }
   return 0;
diff --git a/libc/src/spawn/linux/posix_spawn.cpp b/libc/src/spawn/linux/posix_spawn.cpp
index 2f8287238ec86..504ac5d0a3938 100644
--- a/libc/src/spawn/linux/posix_spawn.cpp
+++ b/libc/src/spawn/linux/posix_spawn.cpp
@@ -14,6 +14,7 @@
 #include "src/spawn/file_actions.h"
 
 #include <fcntl.h>
+#include <signal.h> // For SIGCHLD
 #include <spawn.h>
 #include <sys/syscall.h> // For syscall numbers.
 
@@ -50,8 +51,15 @@ cpp::optional<int> open(const char *path, int oflags, mode_t mode) {
 
 void close(int fd) { __llvm_libc::syscall_impl(SYS_close, fd); }
 
+// We use dup3 if dup2 is not available, similar to our implementation of dup2
 bool dup2(int fd, int newfd) {
+#ifdef SYS_dup2
   long ret = __llvm_libc::syscall_impl(SYS_dup2, fd, newfd);
+#elif defined(SYS_dup3)
+  long ret = __llvm_libc::syscall_impl(SYS_dup3, fd, newfd, 0);
+#else
+#error "SYS_dup2 and SYS_dup3 not available for the target."
+#endif
   return ret < 0 ? false : true;
 }
 
diff --git a/libc/src/stdlib/linux/CMakeLists.txt b/libc/src/stdlib/linux/CMakeLists.txt
index 7d471e5e95fbf..360d3634f361c 100644
--- a/libc/src/stdlib/linux/CMakeLists.txt
+++ b/libc/src/stdlib/linux/CMakeLists.txt
@@ -19,5 +19,5 @@ add_entrypoint_object(
   DEPENDS
     libc.include.stdlib
     libc.src.signal.raise
-    ._Exit
+    libc.src.stdlib._Exit
 )
diff --git a/libc/src/string/memory_utils/CMakeLists.txt b/libc/src/string/memory_utils/CMakeLists.txt
index 5210958fe9baa..30123b09a75be 100644
--- a/libc/src/string/memory_utils/CMakeLists.txt
+++ b/libc/src/string/memory_utils/CMakeLists.txt
@@ -2,12 +2,12 @@
 add_header_library(
   memory_utils
   HDRS
+    aarch64/memcmp_implementations.h
+    aarch64/memcpy_implementations.h
     bcmp_implementations.h
     bzero_implementations.h
     memcmp_implementations.h
     memcpy_implementations.h
-    aarch64/memcpy_implementations.h
-    x86_64/memcpy_implementations.h
     memmove_implementations.h
     memset_implementations.h
     op_aarch64.h
@@ -15,6 +15,8 @@ add_header_library(
     op_generic.h
     op_x86.h
     utils.h
+    x86_64/memcmp_implementations.h
+    x86_64/memcpy_implementations.h
   DEPS
     libc.src.__support.CPP.bit
     libc.src.__support.CPP.cstddef
diff --git a/libc/src/string/memory_utils/aarch64/memcmp_implementations.h b/libc/src/string/memory_utils/aarch64/memcmp_implementations.h
new file mode 100644
index 0000000000000..2c9308de2a95e
--- /dev/null
+++ b/libc/src/string/memory_utils/aarch64/memcmp_implementations.h
@@ -0,0 +1,70 @@
+//===-- Memcmp implementation for aarch64 -----------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+#ifndef LIBC_SRC_STRING_MEMORY_UTILS_X86_64_MEMCMP_IMPLEMENTATIONS_H
+#define LIBC_SRC_STRING_MEMORY_UTILS_X86_64_MEMCMP_IMPLEMENTATIONS_H
+
+#include "src/__support/macros/config.h"       // LIBC_INLINE
+#include "src/__support/macros/optimization.h" // LIBC_UNLIKELY
+#include "src/string/memory_utils/op_aarch64.h"
+#include "src/string/memory_utils/op_generic.h"
+#include "src/string/memory_utils/utils.h" // MemcmpReturnType
+
+namespace __llvm_libc {
+
+[[maybe_unused]] LIBC_INLINE MemcmpReturnType
+inline_memcmp_generic_gt16(CPtr p1, CPtr p2, size_t count) {
+  if (LIBC_UNLIKELY(count >= 384)) {
+    if (auto value = generic::Memcmp<16>::block(p1, p2))
+      return value;
+    align_to_next_boundary<16, Arg::P1>(p1, p2, count);
+  }
+  return generic::Memcmp<16>::loop_and_tail(p1, p2, count);
+}
+
+[[maybe_unused]] LIBC_INLINE MemcmpReturnType
+inline_memcmp_aarch64_neon_gt16(CPtr p1, CPtr p2, size_t count) {
+  if (LIBC_UNLIKELY(count >= 128)) { // [128, ∞]
+    if (auto value = generic::Memcmp<16>::block(p1, p2))
+      return value;
+    align_to_next_boundary<16, Arg::P1>(p1, p2, count);
+    return generic::Memcmp<32>::loop_and_tail(p1, p2, count);
+  }
+  if (generic::Bcmp<16>::block(p1, p2)) // [16, 16]
+    return generic::Memcmp<16>::block(p1, p2);
+  if (count < 32) // [17, 31]
+    return generic::Memcmp<16>::tail(p1, p2, count);
+  if (generic::Bcmp<16>::block(p1 + 16, p2 + 16)) // [32, 32]
+    return generic::Memcmp<16>::block(p1 + 16, p2 + 16);
+  if (count < 64) // [33, 63]
+    return generic::Memcmp<32>::tail(p1, p2, count);
+  // [64, 127]
+  return generic::Memcmp<16>::loop_and_tail(p1 + 32, p2 + 32, count - 32);
+}
+
+LIBC_INLINE MemcmpReturnType inline_memcmp_aarch64(CPtr p1, CPtr p2,
+                                                   size_t count) {
+  if (count == 0)
+    return MemcmpReturnType::ZERO();
+  if (count == 1)
+    return generic::Memcmp<1>::block(p1, p2);
+  if (count == 2)
+    return generic::Memcmp<2>::block(p1, p2);
+  if (count == 3)
+    return generic::Memcmp<3>::block(p1, p2);
+  if (count <= 8)
+    return generic::Memcmp<4>::head_tail(p1, p2, count);
+  if (count <= 16)
+    return generic::Memcmp<8>::head_tail(p1, p2, count);
+  if constexpr (aarch64::kNeon)
+    return inline_memcmp_aarch64_neon_gt16(p1, p2, count);
+  else
+    return inline_memcmp_generic_gt16(p1, p2, count);
+}
+} // namespace __llvm_libc
+
+#endif // LIBC_SRC_STRING_MEMORY_UTILS_X86_64_MEMCMP_IMPLEMENTATIONS_H
diff --git a/libc/src/string/memory_utils/bcmp_implementations.h b/libc/src/string/memory_utils/bcmp_implementations.h
index 483d3c2f98c99..1c238cb88398f 100644
--- a/libc/src/string/memory_utils/bcmp_implementations.h
+++ b/libc/src/string/memory_utils/bcmp_implementations.h
@@ -167,12 +167,8 @@ LIBC_INLINE BcmpReturnType inline_bcmp(CPtr p1, CPtr p2, size_t count) {
   return inline_bcmp_x86(p1, p2, count);
 #elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
   return inline_bcmp_aarch64(p1, p2, count);
-#elif defined(LIBC_TARGET_ARCH_IS_ARM)
-  return inline_bcmp_embedded_tiny(p1, p2, count);
-#elif defined(LIBC_TARGET_ARCH_IS_GPU)
-  return inline_bcmp_embedded_tiny(p1, p2, count);
 #else
-#error "Unsupported platform"
+  return inline_bcmp_embedded_tiny(p1, p2, count);
 #endif
 }
 
diff --git a/libc/src/string/memory_utils/memcmp_implementations.h b/libc/src/string/memory_utils/memcmp_implementations.h
index f99e2e47a8309..9403f3060263b 100644
--- a/libc/src/string/memory_utils/memcmp_implementations.h
+++ b/libc/src/string/memory_utils/memcmp_implementations.h
@@ -12,15 +12,19 @@
 #include "src/__support/common.h"
 #include "src/__support/macros/optimization.h" // LIBC_UNLIKELY LIBC_LOOP_NOUNROLL
 #include "src/__support/macros/properties/architectures.h"
-#include "src/string/memory_utils/op_aarch64.h"
-#include "src/string/memory_utils/op_builtin.h"
 #include "src/string/memory_utils/op_generic.h"
-#include "src/string/memory_utils/op_x86.h"
-#include "src/string/memory_utils/utils.h"
+#include "src/string/memory_utils/utils.h" // CPtr MemcmpReturnType
 
 #include <stddef.h> // size_t
 
+#if defined(LIBC_TARGET_ARCH_IS_X86)
+#include "src/string/memory_utils/x86_64/memcmp_implementations.h"
+#elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
+#include "src/string/memory_utils/aarch64/memcmp_implementations.h"
+#endif
+
 namespace __llvm_libc {
+
 [[maybe_unused]] LIBC_INLINE MemcmpReturnType
 inline_memcmp_embedded_tiny(CPtr p1, CPtr p2, size_t count) {
   LIBC_LOOP_NOUNROLL
@@ -30,121 +34,13 @@ inline_memcmp_embedded_tiny(CPtr p1, CPtr p2, size_t count) {
   return MemcmpReturnType::ZERO();
 }
 
-#if defined(LIBC_TARGET_ARCH_IS_X86) || defined(LIBC_TARGET_ARCH_IS_AARCH64)
-[[maybe_unused]] LIBC_INLINE MemcmpReturnType
-inline_memcmp_generic_gt16(CPtr p1, CPtr p2, size_t count) {
-  if (LIBC_UNLIKELY(count >= 384)) {
-    if (auto value = generic::Memcmp<16>::block(p1, p2))
-      return value;
-    align_to_next_boundary<16, Arg::P1>(p1, p2, count);
-  }
-  return generic::Memcmp<16>::loop_and_tail(p1, p2, count);
-}
-#endif // defined(LIBC_TARGET_ARCH_IS_X86) ||
-       // defined(LIBC_TARGET_ARCH_IS_AARCH64)
-
-#if defined(LIBC_TARGET_ARCH_IS_X86)
-[[maybe_unused]] LIBC_INLINE MemcmpReturnType
-inline_memcmp_x86_sse2_gt16(CPtr p1, CPtr p2, size_t count) {
-  if (LIBC_UNLIKELY(count >= 384)) {
-    if (auto value = x86::sse2::Memcmp<16>::block(p1, p2))
-      return value;
-    align_to_next_boundary<16, Arg::P1>(p1, p2, count);
-  }
-  return x86::sse2::Memcmp<16>::loop_and_tail(p1, p2, count);
-}
-
-[[maybe_unused]] LIBC_INLINE MemcmpReturnType
-inline_memcmp_x86_avx2_gt16(CPtr p1, CPtr p2, size_t count) {
-  if (count <= 32)
-    return x86::sse2::Memcmp<16>::head_tail(p1, p2, count);
-  if (count <= 64)
-    return x86::avx2::Memcmp<32>::head_tail(p1, p2, count);
-  if (count <= 128)
-    return x86::avx2::Memcmp<64>::head_tail(p1, p2, count);
-  if (LIBC_UNLIKELY(count >= 384)) {
-    if (auto value = x86::avx2::Memcmp<32>::block(p1, p2))
-      return value;
-    align_to_next_boundary<32, Arg::P1>(p1, p2, count);
-  }
-  return x86::avx2::Memcmp<32>::loop_and_tail(p1, p2, count);
-}
-
-[[maybe_unused]] LIBC_INLINE MemcmpReturnType
-inline_memcmp_x86_avx512bw_gt16(CPtr p1, CPtr p2, size_t count) {
-  if (count <= 32)
-    return x86::sse2::Memcmp<16>::head_tail(p1, p2, count);
-  if (count <= 64)
-    return x86::avx2::Memcmp<32>::head_tail(p1, p2, count);
-  if (count <= 128)
-    return x86::avx512bw::Memcmp<64>::head_tail(p1, p2, count);
-  if (LIBC_UNLIKELY(count >= 384)) {
-    if (auto value = x86::avx512bw::Memcmp<64>::block(p1, p2))
-      return value;
-    align_to_next_boundary<64, Arg::P1>(p1, p2, count);
-  }
-  return x86::avx512bw::Memcmp<64>::loop_and_tail(p1, p2, count);
-}
-
-#endif // defined(LIBC_TARGET_ARCH_IS_X86)
-
-#if defined(LIBC_TARGET_ARCH_IS_AARCH64)
-[[maybe_unused]] LIBC_INLINE MemcmpReturnType
-inline_memcmp_aarch64_neon_gt16(CPtr p1, CPtr p2, size_t count) {
-  if (LIBC_UNLIKELY(count >= 128)) { // [128, ∞]
-    if (auto value = generic::Memcmp<16>::block(p1, p2))
-      return value;
-    align_to_next_boundary<16, Arg::P1>(p1, p2, count);
-    return generic::Memcmp<32>::loop_and_tail(p1, p2, count);
-  }
-  if (generic::Bcmp<16>::block(p1, p2)) // [16, 16]
-    return generic::Memcmp<16>::block(p1, p2);
-  if (count < 32) // [17, 31]
-    return generic::Memcmp<16>::tail(p1, p2, count);
-  if (generic::Bcmp<16>::block(p1 + 16, p2 + 16)) // [32, 32]
-    return generic::Memcmp<16>::block(p1 + 16, p2 + 16);
-  if (count < 64) // [33, 63]
-    return generic::Memcmp<32>::tail(p1, p2, count);
-  // [64, 127]
-  return generic::Memcmp<16>::loop_and_tail(p1 + 32, p2 + 32, count - 32);
-}
-#endif // defined(LIBC_TARGET_ARCH_IS_AARCH64)
-
 LIBC_INLINE MemcmpReturnType inline_memcmp(CPtr p1, CPtr p2, size_t count) {
-#if defined(LIBC_TARGET_ARCH_IS_X86) || defined(LIBC_TARGET_ARCH_IS_AARCH64)
-  if (count == 0)
-    return MemcmpReturnType::ZERO();
-  if (count == 1)
-    return generic::Memcmp<1>::block(p1, p2);
-  if (count == 2)
-    return generic::Memcmp<2>::block(p1, p2);
-  if (count == 3)
-    return generic::Memcmp<3>::block(p1, p2);
-  if (count <= 8)
-    return generic::Memcmp<4>::head_tail(p1, p2, count);
-  if (count <= 16)
-    return generic::Memcmp<8>::head_tail(p1, p2, count);
 #if defined(LIBC_TARGET_ARCH_IS_X86)
-  if constexpr (x86::kAvx512BW)
-    return inline_memcmp_x86_avx512bw_gt16(p1, p2, count);
-  else if constexpr (x86::kAvx2)
-    return inline_memcmp_x86_avx2_gt16(p1, p2, count);
-  else if constexpr (x86::kSse2)
-    return inline_memcmp_x86_sse2_gt16(p1, p2, count);
-  else
-    return inline_memcmp_generic_gt16(p1, p2, count);
+  return inline_memcmp_x86(p1, p2, count);
 #elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
-  if constexpr (aarch64::kNeon)
-    return inline_memcmp_aarch64_neon_gt16(p1, p2, count);
-  else
-    return inline_memcmp_generic_gt16(p1, p2, count);
-#endif
-#elif defined(LIBC_TARGET_ARCH_IS_ARM)
-  return inline_memcmp_embedded_tiny(p1, p2, count);
-#elif defined(LIBC_TARGET_ARCH_IS_GPU)
-  return inline_memcmp_embedded_tiny(p1, p2, count);
+  return inline_memcmp_aarch64(p1, p2, count);
 #else
-#error "Unsupported platform"
+  return inline_memcmp_embedded_tiny(p1, p2, count);
 #endif
 }
 
diff --git a/libc/src/string/memory_utils/memcpy_implementations.h b/libc/src/string/memory_utils/memcpy_implementations.h
index f0344b3319bb6..b1b60ff590d50 100644
--- a/libc/src/string/memory_utils/memcpy_implementations.h
+++ b/libc/src/string/memory_utils/memcpy_implementations.h
@@ -36,15 +36,14 @@ inline_memcpy_embedded_tiny(Ptr __restrict dst, CPtr __restrict src,
 LIBC_INLINE void inline_memcpy(Ptr __restrict dst, CPtr __restrict src,
                                size_t count) {
   using namespace __llvm_libc::builtin;
-#if defined(LIBC_COPT_MEMCPY_USE_EMBEDDED_TINY) ||                             \
-    defined(LIBC_TARGET_ARCH_IS_ARM) || defined(LIBC_TARGET_ARCH_IS_GPU)
+#if defined(LIBC_COPT_MEMCPY_USE_EMBEDDED_TINY)
   return inline_memcpy_embedded_tiny(dst, src, count);
 #elif defined(LIBC_TARGET_ARCH_IS_X86)
   return inline_memcpy_x86_maybe_interpose_repmovsb(dst, src, count);
 #elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
   return inline_memcpy_aarch64(dst, src, count);
 #else
-#error "Unsupported platform"
+  return inline_memcpy_embedded_tiny(dst, src, count);
 #endif
 }
 
diff --git a/libc/src/string/memory_utils/memmove_implementations.h b/libc/src/string/memory_utils/memmove_implementations.h
index 2d920daf47081..1eb6d4e7e235a 100644
--- a/libc/src/string/memory_utils/memmove_implementations.h
+++ b/libc/src/string/memory_utils/memmove_implementations.h
@@ -102,12 +102,8 @@ LIBC_INLINE void inline_memmove(Ptr dst, CPtr src, size_t count) {
     return generic::Memmove<64, kMaxSize>::loop_and_tail_backward(dst, src,
                                                                   count);
   }
-#elif defined(LIBC_TARGET_ARCH_IS_ARM)
-  return inline_memmove_embedded_tiny(dst, src, count);
-#elif defined(LIBC_TARGET_ARCH_IS_GPU)
-  return inline_memmove_embedded_tiny(dst, src, count);
 #else
-#error "Unsupported platform"
+  return inline_memmove_embedded_tiny(dst, src, count);
 #endif
 }
 
diff --git a/libc/src/string/memory_utils/memset_implementations.h b/libc/src/string/memory_utils/memset_implementations.h
index 089d749961886..16c11470572b5 100644
--- a/libc/src/string/memory_utils/memset_implementations.h
+++ b/libc/src/string/memory_utils/memset_implementations.h
@@ -106,12 +106,8 @@ LIBC_INLINE static void inline_memset(Ptr dst, uint8_t value, size_t count) {
 #elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
   static constexpr size_t kMaxSize = aarch64::kNeon ? 16 : 8;
   return inline_memset_aarch64<kMaxSize>(dst, value, count);
-#elif defined(LIBC_TARGET_ARCH_IS_ARM)
-  return inline_memset_embedded_tiny(dst, value, count);
-#elif defined(LIBC_TARGET_ARCH_IS_GPU)
-  return inline_memset_embedded_tiny(dst, value, count);
 #else
-#error "Unsupported platform"
+  return inline_memset_embedded_tiny(dst, value, count);
 #endif
 }
 
diff --git a/libc/src/string/memory_utils/x86_64/memcmp_implementations.h b/libc/src/string/memory_utils/x86_64/memcmp_implementations.h
new file mode 100644
index 0000000000000..26de1d9b7b971
--- /dev/null
+++ b/libc/src/string/memory_utils/x86_64/memcmp_implementations.h
@@ -0,0 +1,96 @@
+//===-- Memcmp implementation for x86_64 ------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+#ifndef LIBC_SRC_STRING_MEMORY_UTILS_X86_64_MEMCMP_IMPLEMENTATIONS_H
+#define LIBC_SRC_STRING_MEMORY_UTILS_X86_64_MEMCMP_IMPLEMENTATIONS_H
+
+#include "src/__support/macros/config.h"       // LIBC_INLINE
+#include "src/__support/macros/optimization.h" // LIBC_UNLIKELY
+#include "src/string/memory_utils/op_generic.h"
+#include "src/string/memory_utils/op_x86.h"
+#include "src/string/memory_utils/utils.h" // MemcmpReturnType
+
+namespace __llvm_libc {
+
+[[maybe_unused]] LIBC_INLINE MemcmpReturnType
+inline_memcmp_generic_gt16(CPtr p1, CPtr p2, size_t count) {
+  if (LIBC_UNLIKELY(count >= 384)) {
+    if (auto value = generic::Memcmp<16>::block(p1, p2))
+      return value;
+    align_to_next_boundary<16, Arg::P1>(p1, p2, count);
+  }
+  return generic::Memcmp<16>::loop_and_tail(p1, p2, count);
+}
+
+[[maybe_unused]] LIBC_INLINE MemcmpReturnType
+inline_memcmp_x86_sse2_gt16(CPtr p1, CPtr p2, size_t count) {
+  if (LIBC_UNLIKELY(count >= 384)) {
+    if (auto value = x86::sse2::Memcmp<16>::block(p1, p2))
+      return value;
+    align_to_next_boundary<16, Arg::P1>(p1, p2, count);
+  }
+  return x86::sse2::Memcmp<16>::loop_and_tail(p1, p2, count);
+}
+
+[[maybe_unused]] LIBC_INLINE MemcmpReturnType
+inline_memcmp_x86_avx2_gt16(CPtr p1, CPtr p2, size_t count) {
+  if (count <= 32)
+    return x86::sse2::Memcmp<16>::head_tail(p1, p2, count);
+  if (count <= 64)
+    return x86::avx2::Memcmp<32>::head_tail(p1, p2, count);
+  if (count <= 128)
+    return x86::avx2::Memcmp<64>::head_tail(p1, p2, count);
+  if (LIBC_UNLIKELY(count >= 384)) {
+    if (auto value = x86::avx2::Memcmp<32>::block(p1, p2))
+      return value;
+    align_to_next_boundary<32, Arg::P1>(p1, p2, count);
+  }
+  return x86::avx2::Memcmp<32>::loop_and_tail(p1, p2, count);
+}
+
+[[maybe_unused]] LIBC_INLINE MemcmpReturnType
+inline_memcmp_x86_avx512bw_gt16(CPtr p1, CPtr p2, size_t count) {
+  if (count <= 32)
+    return x86::sse2::Memcmp<16>::head_tail(p1, p2, count);
+  if (count <= 64)
+    return x86::avx2::Memcmp<32>::head_tail(p1, p2, count);
+  if (count <= 128)
+    return x86::avx512bw::Memcmp<64>::head_tail(p1, p2, count);
+  if (LIBC_UNLIKELY(count >= 384)) {
+    if (auto value = x86::avx512bw::Memcmp<64>::block(p1, p2))
+      return value;
+    align_to_next_boundary<64, Arg::P1>(p1, p2, count);
+  }
+  return x86::avx512bw::Memcmp<64>::loop_and_tail(p1, p2, count);
+}
+
+LIBC_INLINE MemcmpReturnType inline_memcmp_x86(CPtr p1, CPtr p2, size_t count) {
+
+  if (count == 0)
+    return MemcmpReturnType::ZERO();
+  if (count == 1)
+    return generic::Memcmp<1>::block(p1, p2);
+  if (count == 2)
+    return generic::Memcmp<2>::block(p1, p2);
+  if (count == 3)
+    return generic::Memcmp<3>::block(p1, p2);
+  if (count <= 8)
+    return generic::Memcmp<4>::head_tail(p1, p2, count);
+  if (count <= 16)
+    return generic::Memcmp<8>::head_tail(p1, p2, count);
+  if constexpr (x86::kAvx512BW)
+    return inline_memcmp_x86_avx512bw_gt16(p1, p2, count);
+  else if constexpr (x86::kAvx2)
+    return inline_memcmp_x86_avx2_gt16(p1, p2, count);
+  else if constexpr (x86::kSse2)
+    return inline_memcmp_x86_sse2_gt16(p1, p2, count);
+  else
+    return inline_memcmp_generic_gt16(p1, p2, count);
+}
+} // namespace __llvm_libc
+
+#endif // LIBC_SRC_STRING_MEMORY_UTILS_X86_64_MEMCMP_IMPLEMENTATIONS_H
diff --git a/libc/src/sys/stat/linux/CMakeLists.txt b/libc/src/sys/stat/linux/CMakeLists.txt
index 20deb795fde30..415d2fa5c8771 100644
--- a/libc/src/sys/stat/linux/CMakeLists.txt
+++ b/libc/src/sys/stat/linux/CMakeLists.txt
@@ -74,7 +74,6 @@ add_header_library(
     libc.include.sys_syscall
     libc.src.__support.OSUtil.osutil
     libc.src.__support.common
-    libc.src.errno.errno
 )
 
 add_entrypoint_object(
@@ -87,6 +86,7 @@ add_entrypoint_object(
     .kernel_statx
     libc.include.fcntl
     libc.include.sys_stat
+    libc.src.errno.errno
 )
 
 add_entrypoint_object(
@@ -99,6 +99,7 @@ add_entrypoint_object(
     .kernel_statx
     libc.include.fcntl
     libc.include.sys_stat
+    libc.src.errno.errno
 )
 
 add_entrypoint_object(
@@ -111,4 +112,5 @@ add_entrypoint_object(
     .kernel_statx
     libc.include.fcntl
     libc.include.sys_stat
+    libc.src.errno.errno
 )
diff --git a/libc/src/sys/stat/linux/fstat.cpp b/libc/src/sys/stat/linux/fstat.cpp
index 6baf716157ee4..e187e5cd48eb6 100644
--- a/libc/src/sys/stat/linux/fstat.cpp
+++ b/libc/src/sys/stat/linux/fstat.cpp
@@ -8,6 +8,7 @@
 
 #include "src/sys/stat/fstat.h"
 #include "kernel_statx.h"
+#include "src/errno/libc_errno.h"
 
 #include "src/__support/common.h"
 
@@ -17,7 +18,12 @@
 namespace __llvm_libc {
 
 LLVM_LIBC_FUNCTION(int, fstat, (int fd, struct stat *statbuf)) {
-  return statx(fd, "", AT_EMPTY_PATH, statbuf);
+  int err = statx(fd, "", AT_EMPTY_PATH, statbuf);
+  if (err != 0) {
+    libc_errno = err;
+    return -1;
+  }
+  return 0;
 }
 
 } // namespace __llvm_libc
diff --git a/libc/src/sys/stat/linux/kernel_statx.h b/libc/src/sys/stat/linux/kernel_statx.h
index c3d5777eb9781..cd2119a58e130 100644
--- a/libc/src/sys/stat/linux/kernel_statx.h
+++ b/libc/src/sys/stat/linux/kernel_statx.h
@@ -12,7 +12,6 @@
 #include "src/__support/OSUtil/syscall.h" // For internal syscall function.
 #include "src/__support/common.h"
 
-#include <errno.h>
 #include <stdint.h>
 #include <sys/stat.h>
 #include <sys/syscall.h> // For syscall numbers.
@@ -76,10 +75,8 @@ LIBC_INLINE int statx(int dirfd, const char *__restrict path, int flags,
   ::statx_buf xbuf;
   long ret = __llvm_libc::syscall_impl(SYS_statx, dirfd, path, flags,
                                        ::STATX_BASIC_STATS_MASK, &xbuf);
-  if (ret < 0) {
-    errno = -ret;
-    return -1;
-  }
+  if (ret < 0)
+    return -ret;
 
   statbuf->st_dev = MKDEV(xbuf.stx_dev_major, xbuf.stx_dev_minor);
   statbuf->st_ino = xbuf.stx_ino;
diff --git a/libc/src/sys/stat/linux/lstat.cpp b/libc/src/sys/stat/linux/lstat.cpp
index 816760a8c1452..15a721ba784bc 100644
--- a/libc/src/sys/stat/linux/lstat.cpp
+++ b/libc/src/sys/stat/linux/lstat.cpp
@@ -8,6 +8,7 @@
 
 #include "src/sys/stat/lstat.h"
 #include "kernel_statx.h"
+#include "src/errno/libc_errno.h"
 
 #include "src/__support/OSUtil/syscall.h" // For internal syscall function.
 #include "src/__support/common.h"
@@ -20,7 +21,12 @@ namespace __llvm_libc {
 LLVM_LIBC_FUNCTION(int, lstat,
                    (const char *__restrict path,
                     struct stat *__restrict statbuf)) {
-  return statx(AT_FDCWD, path, AT_SYMLINK_NOFOLLOW, statbuf);
+  int err = statx(AT_FDCWD, path, AT_SYMLINK_NOFOLLOW, statbuf);
+  if (err != 0) {
+    libc_errno = err;
+    return -1;
+  }
+  return 0;
 }
 
 } // namespace __llvm_libc
diff --git a/libc/src/sys/stat/linux/stat.cpp b/libc/src/sys/stat/linux/stat.cpp
index 92e661898aa33..9c3b56bff4746 100644
--- a/libc/src/sys/stat/linux/stat.cpp
+++ b/libc/src/sys/stat/linux/stat.cpp
@@ -8,6 +8,7 @@
 
 #include "src/sys/stat/stat.h"
 #include "kernel_statx.h"
+#include "src/errno/libc_errno.h"
 
 #include "src/__support/common.h"
 
@@ -19,7 +20,12 @@ namespace __llvm_libc {
 LLVM_LIBC_FUNCTION(int, stat,
                    (const char *__restrict path,
                     struct stat *__restrict statbuf)) {
-  return statx(AT_FDCWD, path, 0, statbuf);
+  int err = statx(AT_FDCWD, path, 0, statbuf);
+  if (err != 0) {
+    libc_errno = err;
+    return -1;
+  }
+  return 0;
 }
 
 } // namespace __llvm_libc
diff --git a/libc/src/termios/linux/tcdrain.cpp b/libc/src/termios/linux/tcdrain.cpp
index 910e62bfda9ec..1a51cd30abb98 100644
--- a/libc/src/termios/linux/tcdrain.cpp
+++ b/libc/src/termios/linux/tcdrain.cpp
@@ -21,7 +21,7 @@ namespace __llvm_libc {
 LLVM_LIBC_FUNCTION(int, tcdrain, (int fd)) {
   long ret = __llvm_libc::syscall_impl(SYS_ioctl, fd, TCSBRK, 1);
   if (ret < 0) {
-    errno = -ret;
+    libc_errno = -ret;
     return -1;
   }
   return 0;
diff --git a/libc/src/termios/linux/tcflush.cpp b/libc/src/termios/linux/tcflush.cpp
index fd876afa17d79..8e1f4077d109b 100644
--- a/libc/src/termios/linux/tcflush.cpp
+++ b/libc/src/termios/linux/tcflush.cpp
@@ -21,7 +21,7 @@ namespace __llvm_libc {
 LLVM_LIBC_FUNCTION(int, tcflush, (int fd, int queue_selector)) {
   long ret = __llvm_libc::syscall_impl(SYS_ioctl, fd, TCFLSH, queue_selector);
   if (ret < 0) {
-    errno = -ret;
+    libc_errno = -ret;
     return -1;
   }
   return 0;
diff --git a/libc/src/termios/linux/tcgetsid.cpp b/libc/src/termios/linux/tcgetsid.cpp
index b366fd4e9b711..c0491d9a3c1a5 100644
--- a/libc/src/termios/linux/tcgetsid.cpp
+++ b/libc/src/termios/linux/tcgetsid.cpp
@@ -22,7 +22,7 @@ LLVM_LIBC_FUNCTION(pid_t, tcgetsid, (int fd)) {
   pid_t sid;
   long ret = __llvm_libc::syscall_impl(SYS_ioctl, fd, TIOCGSID, &sid);
   if (ret < 0) {
-    errno = -ret;
+    libc_errno = -ret;
     return -1;
   }
   return sid;
diff --git a/libc/src/threads/tss_get.h b/libc/src/threads/tss_get.h
index 9f1d95ab747ed..4cedce988333f 100644
--- a/libc/src/threads/tss_get.h
+++ b/libc/src/threads/tss_get.h
@@ -9,11 +9,11 @@
 #ifndef LLVM_LIBC_SRC_THREADS_TSS_GET_H
 #define LLVM_LIBC_SRC_THREADS_TSS_GET_H
 
-#include <pthread.h>
+#include <threads.h>
 
 namespace __llvm_libc {
 
-void *tss_get(pthread_key_t);
+void *tss_get(tss_t);
 
 } // namespace __llvm_libc
 
diff --git a/libc/src/time/nanosleep.cpp b/libc/src/time/nanosleep.cpp
index fe042fc0f1c60..ef20f28b639b9 100644
--- a/libc/src/time/nanosleep.cpp
+++ b/libc/src/time/nanosleep.cpp
@@ -20,7 +20,7 @@ LLVM_LIBC_FUNCTION(int, nanosleep,
                    (const struct timespec *req, struct timespec *rem)) {
   int ret = __llvm_libc::syscall_impl(SYS_nanosleep, req, rem);
   if (ret < 0) {
-    errno = -ret;
+    libc_errno = -ret;
     return -1;
   }
   return ret;
diff --git a/libc/src/unistd/linux/fork.cpp b/libc/src/unistd/linux/fork.cpp
index 92d19e253ea42..89d27d3c36f89 100644
--- a/libc/src/unistd/linux/fork.cpp
+++ b/libc/src/unistd/linux/fork.cpp
@@ -14,6 +14,7 @@
 #include "src/__support/threads/thread.h" // For thread self object
 
 #include "src/errno/libc_errno.h"
+#include <signal.h>      // For SIGCHLD
 #include <sys/syscall.h> // For syscall numbers.
 
 namespace __llvm_libc {
diff --git a/libc/startup/gpu/amdgpu/CMakeLists.txt b/libc/startup/gpu/amdgpu/CMakeLists.txt
index be202371fa3a1..891d20993b080 100644
--- a/libc/startup/gpu/amdgpu/CMakeLists.txt
+++ b/libc/startup/gpu/amdgpu/CMakeLists.txt
@@ -2,11 +2,12 @@ add_startup_object(
   crt1
   SRC
     start.cpp
+  DEPENDS
+    libc.src.__support.RPC.rpc_client
   COMPILE_OPTIONS
     -ffreestanding # To avoid compiler warnings about calling the main function.
     -fno-builtin
     -nogpulib # Do not include any GPU vendor libraries.
-    -nostdinc
     -mcpu=${LIBC_GPU_TARGET_ARCHITECTURE}
     -emit-llvm # AMDGPU's intermediate object file format is bitcode.
     --target=${LIBC_GPU_TARGET_TRIPLE}
@@ -15,5 +16,10 @@ add_startup_object(
 get_fq_target_name(crt1 fq_name)
 
 # Ensure that clang uses the correct linker for this object type.
-target_link_libraries(${fq_name} PUBLIC 
-                      "--target=${LIBC_GPU_TARGET_TRIPLE}" "-flto")
+target_link_libraries(
+  ${fq_name}
+  PUBLIC
+  "-mcpu=${LIBC_GPU_TARGET_ARCHITECTURE}"
+  "--target=${LIBC_GPU_TARGET_TRIPLE}"
+  "-flto"
+)
diff --git a/libc/startup/gpu/amdgpu/start.cpp b/libc/startup/gpu/amdgpu/start.cpp
index 3be3745d4c7a0..cc30982e148ff 100644
--- a/libc/startup/gpu/amdgpu/start.cpp
+++ b/libc/startup/gpu/amdgpu/start.cpp
@@ -6,9 +6,13 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/__support/RPC/rpc_client.h"
+
 extern "C" int main(int argc, char **argv);
 
 extern "C" [[gnu::visibility("protected"), clang::amdgpu_kernel]] void
-_start(int argc, char **argv, int *ret) {
+_start(int argc, char **argv, int *ret, void *in, void *out, void *buffer) {
+  __llvm_libc::rpc::client.reset(in, out, buffer);
+
   __atomic_fetch_or(ret, main(argc, argv), __ATOMIC_RELAXED);
 }
diff --git a/libc/startup/linux/aarch64/CMakeLists.txt b/libc/startup/linux/aarch64/CMakeLists.txt
index 14d6409af6cc1..14efe0e58f302 100644
--- a/libc/startup/linux/aarch64/CMakeLists.txt
+++ b/libc/startup/linux/aarch64/CMakeLists.txt
@@ -6,10 +6,12 @@ add_startup_object(
     libc.config.linux.app_h
     libc.include.sys_mman
     libc.include.sys_syscall
+    libc.src.__support.threads.thread
     libc.src.__support.OSUtil.osutil
     libc.src.stdlib.exit
     libc.src.stdlib.atexit
     libc.src.string.memory_utils.memcpy_implementation
+    libc.src.unistd.environ
   COMPILE_OPTIONS
     -fno-omit-frame-pointer
     -ffreestanding # To avoid compiler warnings about calling the main function.
diff --git a/libc/startup/linux/aarch64/start.cpp b/libc/startup/linux/aarch64/start.cpp
index 3896822799566..b02ea7d3a6131 100644
--- a/libc/startup/linux/aarch64/start.cpp
+++ b/libc/startup/linux/aarch64/start.cpp
@@ -23,9 +23,6 @@
 #include <unistd.h>
 
 extern "C" int main(int, char **, char **);
-// The BFD linker requires a reference to __dso_handle to trigger creating
-// a symbol for it when -nostdlib is used..
-extern "C" void *__dso_handle = nullptr;
 
 // Source documentation:
 // https://github.com/ARM-software/abi-aa/tree/main/sysvabi64
diff --git a/libc/startup/linux/riscv64/CMakeLists.txt b/libc/startup/linux/riscv64/CMakeLists.txt
new file mode 100644
index 0000000000000..14efe0e58f302
--- /dev/null
+++ b/libc/startup/linux/riscv64/CMakeLists.txt
@@ -0,0 +1,18 @@
+add_startup_object(
+  crt1
+  SRC
+    start.cpp
+  DEPENDS
+    libc.config.linux.app_h
+    libc.include.sys_mman
+    libc.include.sys_syscall
+    libc.src.__support.threads.thread
+    libc.src.__support.OSUtil.osutil
+    libc.src.stdlib.exit
+    libc.src.stdlib.atexit
+    libc.src.string.memory_utils.memcpy_implementation
+    libc.src.unistd.environ
+  COMPILE_OPTIONS
+    -fno-omit-frame-pointer
+    -ffreestanding # To avoid compiler warnings about calling the main function.
+)
diff --git a/libc/startup/linux/riscv64/start.cpp b/libc/startup/linux/riscv64/start.cpp
new file mode 100644
index 0000000000000..068658c4d0336
--- /dev/null
+++ b/libc/startup/linux/riscv64/start.cpp
@@ -0,0 +1,210 @@
+//===-- Implementation of crt for riscv64 ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "config/linux/app.h"
+#include "src/__support/OSUtil/syscall.h"
+#include "src/__support/threads/thread.h"
+#include "src/stdlib/atexit.h"
+#include "src/stdlib/exit.h"
+#include "src/string/memory_utils/memcpy_implementations.h"
+
+#include <linux/auxvec.h>
+#include <linux/elf.h>
+#include <stdint.h>
+#include <sys/mman.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+extern "C" int main(int, char **, char **);
+
+namespace __llvm_libc {
+
+#ifdef SYS_mmap2
+static constexpr long MMAP_SYSCALL_NUMBER = SYS_mmap2;
+#elif SYS_mmap
+static constexpr long MMAP_SYSCALL_NUMBER = SYS_mmap;
+#else
+#error "Target platform does not have SYS_mmap or SYS_mmap2 defined"
+#endif
+
+AppProperties app;
+
+static ThreadAttributes main_thread_attrib;
+
+void init_tls(TLSDescriptor &tls_descriptor) {
+  if (app.tls.size == 0) {
+    tls_descriptor.size = 0;
+    tls_descriptor.tp = 0;
+    return;
+  }
+
+  // riscv64 follows the variant 1 TLS layout:
+  const uintptr_t size_of_pointers = 2 * sizeof(uintptr_t);
+  uintptr_t padding = 0;
+  const uintptr_t ALIGNMENT_MASK = app.tls.align - 1;
+  uintptr_t diff = size_of_pointers & ALIGNMENT_MASK;
+  if (diff != 0)
+    padding += (ALIGNMENT_MASK - diff) + 1;
+
+  uintptr_t alloc_size = size_of_pointers + padding + app.tls.size;
+
+  // We cannot call the mmap function here as the functions set errno on
+  // failure. Since errno is implemented via a thread local variable, we cannot
+  // use errno before TLS is setup.
+  long mmap_ret_val = __llvm_libc::syscall_impl(
+      MMAP_SYSCALL_NUMBER, nullptr, alloc_size, PROT_READ | PROT_WRITE,
+      MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+  // We cannot check the return value with MAP_FAILED as that is the return
+  // of the mmap function and not the mmap syscall.
+  if (mmap_ret_val < 0 && static_cast<uintptr_t>(mmap_ret_val) > -app.pageSize)
+    __llvm_libc::syscall_impl(SYS_exit, 1);
+  uintptr_t thread_ptr = uintptr_t(reinterpret_cast<uintptr_t *>(mmap_ret_val));
+  uintptr_t tls_addr = thread_ptr + size_of_pointers + padding;
+  __llvm_libc::inline_memcpy(reinterpret_cast<char *>(tls_addr),
+                             reinterpret_cast<const char *>(app.tls.address),
+                             app.tls.init_size);
+  tls_descriptor.size = alloc_size;
+  tls_descriptor.addr = thread_ptr;
+  tls_descriptor.tp = tls_addr;
+}
+
+void cleanup_tls(uintptr_t addr, uintptr_t size) {
+  if (size == 0)
+    return;
+  __llvm_libc::syscall_impl(SYS_munmap, addr, size);
+}
+
+static void set_thread_ptr(uintptr_t val) {
+  LIBC_INLINE_ASM("ld tp, %0\n\t" : : "m"(val));
+}
+
+using InitCallback = void(int, char **, char **);
+using FiniCallback = void(void);
+
+extern "C" {
+// These arrays are present in the .init_array and .fini_array sections.
+// The symbols are inserted by linker when it sees references to them.
+extern uintptr_t __preinit_array_start[];
+extern uintptr_t __preinit_array_end[];
+extern uintptr_t __init_array_start[];
+extern uintptr_t __init_array_end[];
+extern uintptr_t __fini_array_start[];
+extern uintptr_t __fini_array_end[];
+}
+
+static void call_init_array_callbacks(int argc, char **argv, char **env) {
+  size_t preinit_array_size = __preinit_array_end - __preinit_array_start;
+  for (size_t i = 0; i < preinit_array_size; ++i)
+    reinterpret_cast<InitCallback *>(__preinit_array_start[i])(argc, argv, env);
+  size_t init_array_size = __init_array_end - __init_array_start;
+  for (size_t i = 0; i < init_array_size; ++i)
+    reinterpret_cast<InitCallback *>(__init_array_start[i])(argc, argv, env);
+}
+
+static void call_fini_array_callbacks() {
+  size_t fini_array_size = __fini_array_end - __fini_array_start;
+  for (size_t i = fini_array_size; i > 0; --i)
+    reinterpret_cast<FiniCallback *>(__fini_array_start[i - 1])();
+}
+
+} // namespace __llvm_libc
+
+using __llvm_libc::app;
+
+// TODO: Would be nice to use the aux entry structure from elf.h when available.
+struct AuxEntry {
+  uint64_t type;
+  uint64_t value;
+};
+
+__attribute__((noinline)) static void do_start() {
+  auto tid = __llvm_libc::syscall_impl(SYS_gettid);
+  if (tid <= 0)
+    __llvm_libc::syscall_impl(SYS_exit, 1);
+  __llvm_libc::main_thread_attrib.tid = tid;
+
+  // After the argv array, is a 8-byte long NULL value before the array of env
+  // values. The end of the env values is marked by another 8-byte long NULL
+  // value. We step over it (the "+ 1" below) to get to the env values.
+  uint64_t *env_ptr = app.args->argv + app.args->argc + 1;
+  uint64_t *env_end_marker = env_ptr;
+  app.envPtr = env_ptr;
+  while (*env_end_marker)
+    ++env_end_marker;
+
+  // Initialize the POSIX global declared in unistd.h
+  environ = reinterpret_cast<char **>(env_ptr);
+
+  // After the env array, is the aux-vector. The end of the aux-vector is
+  // denoted by an AT_NULL entry.
+  Elf64_Phdr *programHdrTable = nullptr;
+  uintptr_t programHdrCount;
+  for (AuxEntry *aux_entry = reinterpret_cast<AuxEntry *>(env_end_marker + 1);
+       aux_entry->type != AT_NULL; ++aux_entry) {
+    switch (aux_entry->type) {
+    case AT_PHDR:
+      programHdrTable = reinterpret_cast<Elf64_Phdr *>(aux_entry->value);
+      break;
+    case AT_PHNUM:
+      programHdrCount = aux_entry->value;
+      break;
+    case AT_PAGESZ:
+      app.pageSize = aux_entry->value;
+      break;
+    default:
+      break; // TODO: Read other useful entries from the aux vector.
+    }
+  }
+
+  app.tls.size = 0;
+  for (uintptr_t i = 0; i < programHdrCount; ++i) {
+    Elf64_Phdr *phdr = programHdrTable + i;
+    if (phdr->p_type != PT_TLS)
+      continue;
+    // TODO: p_vaddr value has to be adjusted for static-pie executables.
+    app.tls.address = phdr->p_vaddr;
+    app.tls.size = phdr->p_memsz;
+    app.tls.init_size = phdr->p_filesz;
+    app.tls.align = phdr->p_align;
+  }
+
+  __llvm_libc::TLSDescriptor tls;
+  __llvm_libc::init_tls(tls);
+  if (tls.size != 0)
+    __llvm_libc::set_thread_ptr(tls.tp);
+
+  __llvm_libc::self.attrib = &__llvm_libc::main_thread_attrib;
+  __llvm_libc::main_thread_attrib.atexit_callback_mgr =
+      __llvm_libc::internal::get_thread_atexit_callback_mgr();
+
+  // We want the fini array callbacks to be run after other atexit
+  // callbacks are run. So, we register them before running the init
+  // array callbacks as they can potentially register their own atexit
+  // callbacks.
+  __llvm_libc::atexit(&__llvm_libc::call_fini_array_callbacks);
+
+  __llvm_libc::call_init_array_callbacks(
+      app.args->argc, reinterpret_cast<char **>(app.args->argv),
+      reinterpret_cast<char **>(env_ptr));
+
+  int retval = main(app.args->argc, reinterpret_cast<char **>(app.args->argv),
+                    reinterpret_cast<char **>(env_ptr));
+
+  // TODO: TLS cleanup should be done after all other atexit callbacks
+  // are run. So, register a cleanup callback for it with atexit before
+  // everything else.
+  __llvm_libc::cleanup_tls(tls.addr, tls.size);
+  __llvm_libc::exit(retval);
+}
+
+extern "C" void _start() {
+  // Fetch the args using the frame pointer.
+  app.args = reinterpret_cast<__llvm_libc::Args *>(
+      reinterpret_cast<uintptr_t *>(__builtin_frame_address(0)));
+  do_start();
+}
diff --git a/libc/startup/linux/x86_64/start.cpp b/libc/startup/linux/x86_64/start.cpp
index 666cfeb657511..15d5de59d42c1 100644
--- a/libc/startup/linux/x86_64/start.cpp
+++ b/libc/startup/linux/x86_64/start.cpp
@@ -22,9 +22,6 @@
 #include <unistd.h>
 
 extern "C" int main(int, char **, char **);
-// The BFD linker requires a reference to __dso_handle to trigger creating
-// a symbol for it when -nostdlib is used..
-extern "C" void *__dso_handle = nullptr;
 
 namespace __llvm_libc {
 
diff --git a/libc/test/CMakeLists.txt b/libc/test/CMakeLists.txt
index c6c470c6d0411..710f1138a6f2b 100644
--- a/libc/test/CMakeLists.txt
+++ b/libc/test/CMakeLists.txt
@@ -1,32 +1,44 @@
-add_subdirectory(UnitTest)
+add_custom_target(check-libc)
+add_custom_target(libc-unit-tests)
+add_dependencies(check-libc libc-unit-tests)
 
-if(LLVM_LIBC_FULL_BUILD AND NOT
-  (LIBC_TARGET_ARCHITECTURE_IS_GPU OR LIBC_TARGET_OS_IS_BAREMETAL))
-  add_subdirectory(IntegrationTest)
+add_custom_target(exhaustive-check-libc)
+add_custom_target(libc-long-running-tests)
+
+if(NOT LIBC_TARGET_ARCHITECTURE_IS_GPU)
+  add_subdirectory(UnitTest)
 endif()
 
 add_header_library(
   errno_setter_matcher
   HDRS
     ErrnoSetterMatcher.h
+  DEPENDS
+    libc.src.errno.errno
 )
 
-add_custom_target(check-libc)
-add_custom_target(libc-unit-tests)
-add_dependencies(check-libc libc-unit-tests)
+if(LIBC_TARGET_ARCHITECTURE_IS_GPU AND
+   (NOT TARGET libc.utils.gpu.loader OR NOT TARGET libc.startup.gpu.crt1))
+  message(WARNING "Cannot build libc GPU tests, missing loader implementation")
+  return()
+endif()
 
-add_custom_target(exhaustive-check-libc)
-add_custom_target(libc-long-running-tests)
+if(NOT LIBC_TARGET_ARCHITECTURE_IS_GPU)
+  add_subdirectory(src)
+  add_subdirectory(utils)
+endif()
 
-add_subdirectory(src)
-add_subdirectory(utils)
+if(LLVM_LIBC_FULL_BUILD AND NOT LIBC_TARGET_OS_IS_BAREMETAL)
+  add_subdirectory(IntegrationTest)
+endif()
 
 if(NOT LLVM_LIBC_FULL_BUILD)
   return()
 endif()
 
-if(NOT (${LIBC_TARGET_OS} STREQUAL "linux") OR CMAKE_COMPILER_IS_GNUCXX)
-  # Integration tests are currently only available for clang on linux.
+if(NOT ${LIBC_TARGET_OS} STREQUAL "linux" AND
+   NOT ${LIBC_TARGET_OS} STREQUAL "gpu")
+  # Integration tests are currently only available for linux and the GPU.
   return()
 endif()
 add_subdirectory(integration)
diff --git a/libc/test/ErrnoSetterMatcher.h b/libc/test/ErrnoSetterMatcher.h
index d2dab988dc0de..36ba9323b12d1 100644
--- a/libc/test/ErrnoSetterMatcher.h
+++ b/libc/test/ErrnoSetterMatcher.h
@@ -9,9 +9,9 @@
 #ifndef LLVM_LIBC_TEST_ERRNOSETTERMATCHER_H
 #define LLVM_LIBC_TEST_ERRNOSETTERMATCHER_H
 
+#include "src/errno/libc_errno.h"
 #include "test/UnitTest/Test.h"
 
-#include <errno.h>
 #include <string.h>
 
 namespace __llvm_libc {
@@ -42,8 +42,8 @@ template <typename T> class ErrnoSetterMatcher : public Matcher<T> {
 
   bool match(T Got) {
     ActualReturn = Got;
-    ActualErrno = errno;
-    errno = 0;
+    ActualErrno = libc_errno;
+    libc_errno = 0;
     return Got == ExpectedReturn && ActualErrno == ExpectedErrno;
   }
 };
diff --git a/libc/test/IntegrationTest/CMakeLists.txt b/libc/test/IntegrationTest/CMakeLists.txt
index 3d8a3497bbb2f..62bd114645b57 100644
--- a/libc/test/IntegrationTest/CMakeLists.txt
+++ b/libc/test/IntegrationTest/CMakeLists.txt
@@ -1,9 +1,25 @@
+if(LIBC_GPU_TARGET_ARCHITECTURE_IS_AMDGPU)
+  set(TEST_COMPILE_FLAGS
+    -mcpu=${LIBC_GPU_TARGET_ARCHITECTURE}
+    -emit-llvm # AMDGPU's intermediate object file format is bitcode.
+    --target=${LIBC_GPU_TARGET_TRIPLE}
+  )
+elseif(LIBC_GPU_TARGET_ARCHITECTURE_IS_NVPTX)
+  set(TEST_COMPILE_FLAGS
+    -march=${LIBC_GPU_TARGET_ARCHITECTURE}
+    --target=${LIBC_GPU_TARGET_TRIPLE}
+  )
+endif()
+
 add_object_library(
   test
   SRCS
     test.cpp
+  COMPILE_OPTIONS
+    ${TEST_COMPILE_FLAGS}
   HDRS
     test.h
   DEPENDS
     libc.src.__support.OSUtil.osutil
+  NO_GPU_BUNDLE # Compile this file directly without special GPU handling.
 )
diff --git a/libc/test/IntegrationTest/test.cpp b/libc/test/IntegrationTest/test.cpp
index 53f9c95423019..4d2a7f08cc06e 100644
--- a/libc/test/IntegrationTest/test.cpp
+++ b/libc/test/IntegrationTest/test.cpp
@@ -9,6 +9,43 @@
 #include <stddef.h>
 #include <stdint.h>
 
+// Integration tests rely on the following memory functions. This is because the
+// compiler code generation can emit calls to them. We want to map the external
+// entrypoint to the internal implementation of the function used for testing.
+// This is done manually as not all targets support aliases.
+
+namespace __llvm_libc {
+
+int bcmp(const void *lhs, const void *rhs, size_t count);
+void bzero(void *ptr, size_t count);
+int memcmp(const void *lhs, const void *rhs, size_t count);
+void *memcpy(void *__restrict, const void *__restrict, size_t);
+void *memmove(void *dst, const void *src, size_t count);
+void *memset(void *ptr, int value, size_t count);
+
+} // namespace __llvm_libc
+
+extern "C" {
+
+int bcmp(const void *lhs, const void *rhs, size_t count) {
+  return __llvm_libc::bcmp(lhs, rhs, count);
+}
+void bzero(void *ptr, size_t count) { __llvm_libc::bzero(ptr, count); }
+int memcmp(const void *lhs, const void *rhs, size_t count) {
+  return __llvm_libc::memcmp(lhs, rhs, count);
+}
+void *memcpy(void *__restrict dst, const void *__restrict src, size_t count) {
+  return __llvm_libc::memcpy(dst, src, count);
+}
+void *memmove(void *dst, const void *src, size_t count) {
+  return __llvm_libc::memmove(dst, src, count);
+}
+void *memset(void *ptr, int value, size_t count) {
+  return __llvm_libc::memset(ptr, value, count);
+}
+
+} // extern "C"
+
 // Integration tests cannot use the SCUDO standalone allocator as SCUDO pulls
 // various other parts of the libc. Since SCUDO development does not use
 // LLVM libc build rules, it is very hard to keep track or pull all that SCUDO
@@ -34,4 +71,7 @@ void *realloc(void *ptr, size_t s) {
   return malloc(s);
 }
 
+// Integration tests are linked with -nostdlib. BFD linker expects
+// __dso_handle when -nostdlib is used.
+void *__dso_handle = nullptr;
 } // extern "C"
diff --git a/libc/test/UnitTest/FPMatcher.h b/libc/test/UnitTest/FPMatcher.h
index f804b652f2759..89ccf1a874e39 100644
--- a/libc/test/UnitTest/FPMatcher.h
+++ b/libc/test/UnitTest/FPMatcher.h
@@ -14,7 +14,6 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/RoundingModeUtils.h"
 
-#include <errno.h>
 #include <math.h>
 
 namespace __llvm_libc {
@@ -106,8 +105,8 @@ FPMatcher<T, C> getMatcher(T expectedValue) {
 #define EXPECT_MATH_ERRNO(expected)                                            \
   do {                                                                         \
     if (math_errhandling & MATH_ERRNO) {                                       \
-      int actual = errno;                                                      \
-      errno = 0;                                                               \
+      int actual = libc_errno;                                                 \
+      libc_errno = 0;                                                          \
       EXPECT_EQ(actual, expected);                                             \
     }                                                                          \
   } while (0)
@@ -115,8 +114,8 @@ FPMatcher<T, C> getMatcher(T expectedValue) {
 #define ASSERT_MATH_ERRNO(expected)                                            \
   do {                                                                         \
     if (math_errhandling & MATH_ERRNO) {                                       \
-      int actual = errno;                                                      \
-      errno = 0;                                                               \
+      int actual = libc_errno;                                                 \
+      libc_errno = 0;                                                          \
       ASSERT_EQ(actual, expected);                                             \
     }                                                                          \
   } while (0)
diff --git a/libc/test/integration/CMakeLists.txt b/libc/test/integration/CMakeLists.txt
index 058c7e85ecef6..7dbcc9544a0b9 100644
--- a/libc/test/integration/CMakeLists.txt
+++ b/libc/test/integration/CMakeLists.txt
@@ -6,16 +6,6 @@ function(add_libc_integration_test_suite name)
   add_dependencies(libc-integration-tests ${name})
 endfunction()
 
-add_library(
-  libc_integration_test_dummy
-  STATIC
-  dummy.cpp
-)
-set_target_properties(libc_integration_test_dummy
-    PROPERTIES
-      ARCHIVE_OUTPUT_NAME dummy
-      ARCHIVE_OUTPUT_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
-
 add_subdirectory(startup)
 add_subdirectory(scudo)
 add_subdirectory(src)
diff --git a/libc/test/integration/dummy.cpp b/libc/test/integration/dummy.cpp
deleted file mode 100644
index e69de29bb2d1d..0000000000000
diff --git a/libc/test/integration/src/__support/threads/CMakeLists.txt b/libc/test/integration/src/__support/threads/CMakeLists.txt
index 592c0b31261f3..5a12d28ada3fd 100644
--- a/libc/test/integration/src/__support/threads/CMakeLists.txt
+++ b/libc/test/integration/src/__support/threads/CMakeLists.txt
@@ -11,8 +11,6 @@ add_integration_test(
     libc-support-threads-integration-tests
   SRCS
     thread_detach_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.src.__support.threads.mutex
     libc.src.__support.threads.thread
@@ -24,8 +22,6 @@ add_integration_test(
     libc-support-threads-integration-tests
   SRCS
     thread_tls_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.src.__support.threads.thread
 )
diff --git a/libc/test/integration/src/pthread/CMakeLists.txt b/libc/test/integration/src/pthread/CMakeLists.txt
index a6a1690f609ba..106cac8e4d802 100644
--- a/libc/test/integration/src/pthread/CMakeLists.txt
+++ b/libc/test/integration/src/pthread/CMakeLists.txt
@@ -6,8 +6,6 @@ add_integration_test(
     libc-pthread-integration-tests
   SRCS
     pthread_mutex_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.include.pthread
     libc.src.errno.errno
@@ -25,8 +23,6 @@ add_integration_test(
     libc-pthread-integration-tests
   SRCS
     pthread_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.include.pthread
     libc.src.pthread.pthread_create
@@ -39,8 +35,6 @@ add_integration_test(
     libc-pthread-integration-tests
   SRCS
     pthread_equal_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.include.pthread
     libc.src.errno.errno
@@ -60,8 +54,6 @@ add_integration_test(
     libc-pthread-integration-tests
   SRCS
     pthread_name_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.include.errno
     libc.include.pthread
@@ -83,8 +75,6 @@ add_integration_test(
     libc-pthread-integration-tests
   SRCS
     pthread_exit_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.include.pthread
     libc.src.pthread.pthread_create
@@ -98,8 +88,6 @@ add_integration_test(
     libc-pthread-integration-tests
   SRCS
     pthread_tss_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.include.pthread
     libc.src.pthread.pthread_create
@@ -117,8 +105,6 @@ add_integration_test(
     libc-pthread-integration-tests
   SRCS
     pthread_once_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.include.pthread
     libc.src.pthread.pthread_once
diff --git a/libc/test/integration/src/spawn/CMakeLists.txt b/libc/test/integration/src/spawn/CMakeLists.txt
index d6307cb2ad883..caed59fc1b7aa 100644
--- a/libc/test/integration/src/spawn/CMakeLists.txt
+++ b/libc/test/integration/src/spawn/CMakeLists.txt
@@ -26,8 +26,6 @@ add_integration_test(
     spawn-integration-tests
   SRCS
     posix_spawn_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc_posix_spawn_test_binary
     libc.test.integration.src.spawn.test_binary_properties
diff --git a/libc/test/integration/src/stdio/CMakeLists.txt b/libc/test/integration/src/stdio/CMakeLists.txt
index ed9af297257b2..61caa2e5fa17b 100644
--- a/libc/test/integration/src/stdio/CMakeLists.txt
+++ b/libc/test/integration/src/stdio/CMakeLists.txt
@@ -9,8 +9,6 @@ add_integration_test(
     stdio-integration-tests
   SRCS
     sprintf_size_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.src.stdio.sprintf
   ARGS
@@ -26,8 +24,6 @@ add_integration_test(
     stdio-integration-tests
   SRCS
     sprintf_size_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   ARGS
     "%s %c %d"
     "First arg"
diff --git a/libc/test/integration/src/stdlib/CMakeLists.txt b/libc/test/integration/src/stdlib/CMakeLists.txt
index 1f600f8dfa5c8..0985a80ce7a09 100644
--- a/libc/test/integration/src/stdlib/CMakeLists.txt
+++ b/libc/test/integration/src/stdlib/CMakeLists.txt
@@ -7,8 +7,6 @@ add_integration_test(
     stdlib-integration-tests
   SRCS
     getenv_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.src.stdlib.getenv
   ENV
diff --git a/libc/test/integration/src/threads/CMakeLists.txt b/libc/test/integration/src/threads/CMakeLists.txt
index 5f9e49ed5a8d3..d74b923751ad6 100644
--- a/libc/test/integration/src/threads/CMakeLists.txt
+++ b/libc/test/integration/src/threads/CMakeLists.txt
@@ -6,8 +6,6 @@ add_integration_test(
     libc-threads-integration-tests
   SRCS
     mtx_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.include.threads
     libc.src.errno.errno
@@ -25,8 +23,6 @@ add_integration_test(
     libc-threads-integration-tests
   SRCS
     thrd_equal_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.include.threads
     libc.src.threads.mtx_destroy
@@ -45,8 +41,6 @@ add_integration_test(
     libc-threads-integration-tests
   SRCS
     thrd_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.include.threads
     libc.src.errno.errno
@@ -60,8 +54,6 @@ add_integration_test(
     libc-threads-integration-tests
   SRCS
     thrd_exit_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.include.threads
     libc.src.threads.thrd_create
@@ -75,8 +67,6 @@ add_integration_test(
     libc-threads-integration-tests
   SRCS
     tss_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.include.threads
     libc.src.threads.thrd_create
@@ -94,8 +84,6 @@ add_integration_test(
     libc-threads-integration-tests
   SRCS
     call_once_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.include.threads
     libc.src.threads.call_once
@@ -114,8 +102,6 @@ add_integration_test(
     libc-threads-integration-tests
   SRCS
     cnd_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.include.threads
     libc.src.threads.cnd_init
diff --git a/libc/test/integration/src/threads/cnd_test.cpp b/libc/test/integration/src/threads/cnd_test.cpp
index 5869df985ff66..ecc04ffa780eb 100644
--- a/libc/test/integration/src/threads/cnd_test.cpp
+++ b/libc/test/integration/src/threads/cnd_test.cpp
@@ -35,7 +35,7 @@ namespace wait_notify_broadcast_test {
 // |broadcast_count| by 1 before they start waiting on |broadcast_cnd|, and
 // decrement it by 1 after getting signalled on |broadcast_cnd|.
 
-constexpr unsigned int THRD_COUNT = 10000;
+constexpr unsigned int THRD_COUNT = 1000;
 
 static __llvm_libc::cpp::Atomic<unsigned int> broadcast_count(0);
 static cnd_t broadcast_cnd, threads_ready_cnd;
diff --git a/libc/test/integration/src/unistd/CMakeLists.txt b/libc/test/integration/src/unistd/CMakeLists.txt
index 1abe01697ab54..72fdb8fc7e6b0 100644
--- a/libc/test/integration/src/unistd/CMakeLists.txt
+++ b/libc/test/integration/src/unistd/CMakeLists.txt
@@ -7,11 +7,9 @@ add_integration_test(
     unistd-integration-tests
   SRCS
     getcwd_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
-    libc.include.errno
     libc.src.__support.CPP.string_view
+    libc.src.errno.errno
     libc.src.stdlib.getenv
     libc.src.unistd.getcwd
 )
@@ -22,8 +20,6 @@ add_integration_test(
     unistd-integration-tests
   SRCS
     fork_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc.include.errno
     libc.include.signal
@@ -67,8 +63,6 @@ add_integration_test(
     unistd-integration-tests
   SRCS
     execv_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc_execv_test_normal_exit
     libc_execv_test_signal_exit
@@ -86,8 +80,6 @@ add_integration_test(
     unistd-integration-tests
   SRCS
     execve_test.cpp
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     libc_execv_test_normal_exit
     libc_execv_test_signal_exit
diff --git a/libc/test/integration/src/unistd/getcwd_test.cpp b/libc/test/integration/src/unistd/getcwd_test.cpp
index 96af6dc2e3a7c..3214ebf7085cf 100644
--- a/libc/test/integration/src/unistd/getcwd_test.cpp
+++ b/libc/test/integration/src/unistd/getcwd_test.cpp
@@ -7,12 +7,12 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/CPP/string_view.h"
+#include "src/errno/libc_errno.h"
 #include "src/stdlib/getenv.h"
 #include "src/unistd/getcwd.h"
 
 #include "test/IntegrationTest/test.h"
 
-#include <errno.h>
 #include <stdlib.h> // For malloc and free
 
 using __llvm_libc::cpp::string_view;
@@ -30,13 +30,13 @@ TEST_MAIN(int argc, char **argv, char **envp) {
   // Bad size
   cwd = __llvm_libc::getcwd(buffer, 0);
   ASSERT_TRUE(cwd == nullptr);
-  ASSERT_EQ(errno, EINVAL);
-  errno = 0;
+  ASSERT_EQ(libc_errno, EINVAL);
+  libc_errno = 0;
 
   // Insufficient size
   cwd = __llvm_libc::getcwd(buffer, 2);
   ASSERT_TRUE(cwd == nullptr);
-  int err = errno;
+  int err = libc_errno;
   ASSERT_EQ(err, ERANGE);
 
   return 0;
diff --git a/libc/test/integration/startup/gpu/CMakeLists.txt b/libc/test/integration/startup/gpu/CMakeLists.txt
new file mode 100644
index 0000000000000..5451a27c28874
--- /dev/null
+++ b/libc/test/integration/startup/gpu/CMakeLists.txt
@@ -0,0 +1,11 @@
+add_custom_target(libc-startup-tests)
+add_dependencies(libc-integration-tests libc-startup-tests)
+
+add_integration_test(
+  startup_args_test
+  SUITE libc-startup-tests
+  SRCS
+    args_test.cpp
+  ARGS
+    1 2 3
+)
diff --git a/libc/test/integration/startup/gpu/args_test.cpp b/libc/test/integration/startup/gpu/args_test.cpp
new file mode 100644
index 0000000000000..f3a5410691c22
--- /dev/null
+++ b/libc/test/integration/startup/gpu/args_test.cpp
@@ -0,0 +1,27 @@
+//===-- Loader test to check args to main ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "test/IntegrationTest/test.h"
+
+static bool my_streq(const char *lhs, const char *rhs) {
+  const char *l, *r;
+  for (l = lhs, r = rhs; *l != '\0' && *r != '\0'; ++l, ++r)
+    if (*l != *r)
+      return false;
+
+  return *l == '\0' && *r == '\0';
+}
+
+TEST_MAIN(int argc, char **argv) {
+  ASSERT_TRUE(argc == 4);
+  ASSERT_TRUE(my_streq(argv[1], "1"));
+  ASSERT_TRUE(my_streq(argv[2], "2"));
+  ASSERT_TRUE(my_streq(argv[3], "3"));
+
+  return 0;
+}
diff --git a/libc/test/integration/startup/linux/CMakeLists.txt b/libc/test/integration/startup/linux/CMakeLists.txt
index 675e8e3d1540b..6f0517a832d20 100644
--- a/libc/test/integration/startup/linux/CMakeLists.txt
+++ b/libc/test/integration/startup/linux/CMakeLists.txt
@@ -9,8 +9,6 @@ add_dependencies(libc-integration-tests libc-startup-tests)
 add_integration_test(
   startup_args_test
   SUITE libc-startup-tests
-  STARTUP
-    libc.startup.linux.crt1
   SRCS
     args_test.cpp
   ARGS
@@ -23,8 +21,6 @@ add_integration_test(
 add_integration_test(
   startup_no_envp_test
   SUITE libc-startup-tests
-  STARTUP
-    libc.startup.linux.crt1
   SRCS
     main_without_envp.cpp
 )
@@ -32,8 +28,6 @@ add_integration_test(
 add_integration_test(
   startup_no_args_test
   SUITE libc-startup-tests
-  STARTUP
-    libc.startup.linux.crt1
   SRCS
     main_without_args.cpp
 )
@@ -41,8 +35,6 @@ add_integration_test(
 add_integration_test(
   startup_tls_test
   SUITE libc-startup-tests
-  STARTUP
-    libc.startup.linux.crt1
   SRCS
     tls_test.cpp
   DEPENDS
@@ -55,8 +47,6 @@ add_integration_test(
 add_integration_test(
   init_fini_array_test
   SUITE libc-startup-tests
-  STARTUP
-    libc.startup.linux.crt1
   SRCS
     init_fini_array_test.cpp
 )
diff --git a/libc/test/integration/startup/linux/tls_test.cpp b/libc/test/integration/startup/linux/tls_test.cpp
index 76b8af0bb8d7f..f392610545e91 100644
--- a/libc/test/integration/startup/linux/tls_test.cpp
+++ b/libc/test/integration/startup/linux/tls_test.cpp
@@ -9,7 +9,7 @@
 #include "include/errno.h"
 #include "include/sys/mman.h"
 
-#include "src/errno/llvmlibc_errno.h"
+#include "src/errno/libc_errno.h"
 #include "src/sys/mman/mmap.h"
 #include "test/IntegrationTest/test.h"
 
@@ -28,11 +28,11 @@ TEST_MAIN(int argc, char **argv, char **envp) {
   // set in errno. Since errno is implemented using a thread
   // local var, this helps us test setting of errno and
   // reading it back.
-  ASSERT_TRUE(llvmlibc_errno == 0);
+  ASSERT_TRUE(libc_errno == 0);
   void *addr = __llvm_libc::mmap(nullptr, 0, PROT_READ,
                                  MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
   ASSERT_TRUE(addr == MAP_FAILED);
-  ASSERT_TRUE(llvmlibc_errno == EINVAL);
+  ASSERT_TRUE(libc_errno == EINVAL);
 
   return 0;
 }
diff --git a/libc/test/src/CMakeLists.txt b/libc/test/src/CMakeLists.txt
index a9b788ca4e70a..6c3c987e3291d 100644
--- a/libc/test/src/CMakeLists.txt
+++ b/libc/test/src/CMakeLists.txt
@@ -60,14 +60,12 @@ if(${LIBC_TARGET_OS} STREQUAL "linux")
   add_subdirectory(pthread)
 endif()
 
-if(LLVM_RUNTIMES_BUILD OR LIBC_HDRGEN_EXE OR CMAKE_COMPILER_IS_GNUCXX)
+if(LLVM_RUNTIMES_BUILD OR LIBC_HDRGEN_EXE)
   # The public API test below uses tablegen to generate the test
   # source file. Since tablegen is not available during a runtimes
   # build, we will skip the test.
   # If a different libc-hdrgen binary is being used, then also we
   # skip the api-test as we cannot generate the test source file.
-  # Additionally, this is an integration test and those only work with clang
-  # for the moment.
   return()
 endif()
 
@@ -117,8 +115,6 @@ add_integration_test(
     libc-api-test
   SRCS
     ${public_test}
-  STARTUP
-    libc.startup.linux.crt1
   DEPENDS
     ${api-test-entrypoints}
 )
diff --git a/libc/test/src/__support/CMakeLists.txt b/libc/test/src/__support/CMakeLists.txt
index fa35aa45712ff..a8c5517238882 100644
--- a/libc/test/src/__support/CMakeLists.txt
+++ b/libc/test/src/__support/CMakeLists.txt
@@ -40,6 +40,7 @@ add_libc_unittest(
   DEPENDS
     libc.src.__support.str_to_float
     libc.src.__support.uint128
+    libc.src.errno.errno
 )
 
 add_libc_unittest(
diff --git a/libc/test/src/__support/str_to_float_comparison_test.cpp b/libc/test/src/__support/str_to_float_comparison_test.cpp
index 5f5f71cbb1e24..19f3f869f3c4c 100644
--- a/libc/test/src/__support/str_to_float_comparison_test.cpp
+++ b/libc/test/src/__support/str_to_float_comparison_test.cpp
@@ -12,6 +12,7 @@
 
 // #include "src/__support/FPUtil/FPBits.h"
 
+#include <cstdint>
 #include <fstream>
 #include <iostream>
 #include <string>
diff --git a/libc/test/src/__support/str_to_float_test.cpp b/libc/test/src/__support/str_to_float_test.cpp
index b5bb3b931a8af..4ebdb33549ef6 100644
--- a/libc/test/src/__support/str_to_float_test.cpp
+++ b/libc/test/src/__support/str_to_float_test.cpp
@@ -9,6 +9,7 @@
 #include "src/__support/FPUtil/FPBits.h"
 #include "src/__support/UInt128.h"
 #include "src/__support/str_to_float.h"
+#include "src/errno/libc_errno.h"
 
 #include "test/UnitTest/Test.h"
 
@@ -78,7 +79,7 @@ class LlvmLibcStrToFloatTest : public __llvm_libc::testing::Test {
     typename __llvm_libc::fputil::FPBits<T>::UIntType actual_output_mantissa =
         0;
     uint32_t actual_output_exp2 = 0;
-    errno = 0;
+    libc_errno = 0;
 
     auto result = __llvm_libc::internal::simple_decimal_conversion<T>(numStart);
 
@@ -250,7 +251,7 @@ TEST(LlvmLibcStrToFloatTest, SimpleDecimalConversionExtraTypes) {
   uint32_t float_output_mantissa = 0;
   uint32_t output_exp2 = 0;
 
-  errno = 0;
+  libc_errno = 0;
   auto float_result = __llvm_libc::internal::simple_decimal_conversion<float>(
       "123456789012345678900");
   float_output_mantissa = float_result.num.mantissa;
@@ -262,7 +263,7 @@ TEST(LlvmLibcStrToFloatTest, SimpleDecimalConversionExtraTypes) {
   uint64_t double_output_mantissa = 0;
   output_exp2 = 0;
 
-  errno = 0;
+  libc_errno = 0;
   auto double_result = __llvm_libc::internal::simple_decimal_conversion<double>(
       "123456789012345678900");
 
diff --git a/libc/test/src/dirent/CMakeLists.txt b/libc/test/src/dirent/CMakeLists.txt
index cb96dd510265c..014fe37cc09f6 100644
--- a/libc/test/src/dirent/CMakeLists.txt
+++ b/libc/test/src/dirent/CMakeLists.txt
@@ -8,11 +8,11 @@ add_libc_unittest(
   SRCS
     dirent_test.cpp
   DEPENDS
-    libc.include.dirent
     libc.src.__support.CPP.string_view
     libc.src.dirent.closedir
     libc.src.dirent.dirfd
     libc.src.dirent.opendir
     libc.src.dirent.readdir
+    libc.src.errno.errno
 )
 
diff --git a/libc/test/src/dirent/dirent_test.cpp b/libc/test/src/dirent/dirent_test.cpp
index 7c168e78a63a1..72f9f4632c8dc 100644
--- a/libc/test/src/dirent/dirent_test.cpp
+++ b/libc/test/src/dirent/dirent_test.cpp
@@ -11,11 +11,11 @@
 #include "src/dirent/dirfd.h"
 #include "src/dirent/opendir.h"
 #include "src/dirent/readdir.h"
+#include "src/errno/libc_errno.h"
 
 #include "test/UnitTest/Test.h"
 
 #include <dirent.h>
-#include <errno.h>
 
 using string_view = __llvm_libc::cpp::string_view;
 
@@ -44,7 +44,7 @@ TEST(LlvmLibcDirentTest, SimpleOpenAndRead) {
   }
 
   // Verify that we don't break out of the above loop in error.
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
 
   ASSERT_TRUE(file1 != nullptr);
   ASSERT_TRUE(file2 != nullptr);
@@ -55,17 +55,17 @@ TEST(LlvmLibcDirentTest, SimpleOpenAndRead) {
 }
 
 TEST(LlvmLibcDirentTest, OpenNonExistentDir) {
-  errno = 0;
+  libc_errno = 0;
   ::DIR *dir = __llvm_libc::opendir("___xyz123__.non_existent__");
   ASSERT_TRUE(dir == nullptr);
-  ASSERT_EQ(errno, ENOENT);
-  errno = 0;
+  ASSERT_EQ(libc_errno, ENOENT);
+  libc_errno = 0;
 }
 
 TEST(LlvmLibcDirentTest, OpenFile) {
-  errno = 0;
+  libc_errno = 0;
   ::DIR *dir = __llvm_libc::opendir("testdata/file1.txt");
   ASSERT_TRUE(dir == nullptr);
-  ASSERT_EQ(errno, ENOTDIR);
-  errno = 0;
+  ASSERT_EQ(libc_errno, ENOTDIR);
+  libc_errno = 0;
 }
diff --git a/libc/test/src/errno/errno_test.cpp b/libc/test/src/errno/errno_test.cpp
index 86a9a88e86685..33185c2bcf6f5 100644
--- a/libc/test/src/errno/errno_test.cpp
+++ b/libc/test/src/errno/errno_test.cpp
@@ -6,11 +6,11 @@
 //
 //===----------------------------------------------------------------------===//
 
-#include "src/errno/llvmlibc_errno.h"
+#include "src/errno/libc_errno.h"
 #include "test/UnitTest/Test.h"
 
 TEST(LlvmLibcErrnoTest, Basic) {
   int test_val = 123;
-  llvmlibc_errno = test_val;
-  ASSERT_EQ(test_val, llvmlibc_errno);
+  libc_errno = test_val;
+  ASSERT_EQ(test_val, libc_errno);
 }
diff --git a/libc/test/src/fcntl/CMakeLists.txt b/libc/test/src/fcntl/CMakeLists.txt
index 877e00b9cc420..d90b2fab10347 100644
--- a/libc/test/src/fcntl/CMakeLists.txt
+++ b/libc/test/src/fcntl/CMakeLists.txt
@@ -9,8 +9,8 @@ add_libc_unittest(
   SRCS
     creat_test.cpp
   DEPENDS
-    libc.include.errno
     libc.include.fcntl
+    libc.src.errno.errno
     libc.src.fcntl.creat
     libc.src.fcntl.open
     libc.src.unistd.close
@@ -24,8 +24,8 @@ add_libc_unittest(
   SRCS
     openat_test.cpp
   DEPENDS
-    libc.include.errno
     libc.include.fcntl
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.fcntl.openat
     libc.src.unistd.close
diff --git a/libc/test/src/fcntl/creat_test.cpp b/libc/test/src/fcntl/creat_test.cpp
index dfe5600f0f811..b7d577ad461ce 100644
--- a/libc/test/src/fcntl/creat_test.cpp
+++ b/libc/test/src/fcntl/creat_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/creat.h"
 #include "src/fcntl/open.h"
 #include "src/unistd/close.h"
@@ -13,18 +14,16 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
-
 TEST(LlvmLibcCreatTest, CreatAndOpen) {
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
   constexpr const char *TEST_FILE = "testdata/creat.test";
   int fd = __llvm_libc::creat(TEST_FILE, S_IRWXU);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
 
   fd = __llvm_libc::open(TEST_FILE, O_RDONLY);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
 
diff --git a/libc/test/src/fcntl/openat_test.cpp b/libc/test/src/fcntl/openat_test.cpp
index 9716b1f71182d..5310d4647f851 100644
--- a/libc/test/src/fcntl/openat_test.cpp
+++ b/libc/test/src/fcntl/openat_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/fcntl/openat.h"
 #include "src/unistd/close.h"
@@ -14,7 +15,6 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
 #include <fcntl.h>
 
 TEST(LlvmLibcUniStd, OpenAndReadTest) {
@@ -22,13 +22,13 @@ TEST(LlvmLibcUniStd, OpenAndReadTest) {
   constexpr const char *TEST_DIR = "testdata";
   constexpr const char *TEST_FILE = "openat.test";
   int dir_fd = __llvm_libc::open(TEST_DIR, O_DIRECTORY);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(dir_fd, 0);
   constexpr const char TEST_MSG[] = "openat test";
   constexpr int TEST_MSG_SIZE = sizeof(TEST_MSG) - 1;
 
   int read_fd = __llvm_libc::openat(dir_fd, TEST_FILE, O_RDONLY);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(read_fd, 0);
   char read_buf[TEST_MSG_SIZE];
   ASSERT_THAT(__llvm_libc::read(read_fd, read_buf, TEST_MSG_SIZE),
diff --git a/libc/test/src/fenv/enabled_exceptions_test.cpp b/libc/test/src/fenv/enabled_exceptions_test.cpp
index 0952a5345a50f..bb0c572da7541 100644
--- a/libc/test/src/fenv/enabled_exceptions_test.cpp
+++ b/libc/test/src/fenv/enabled_exceptions_test.cpp
@@ -21,8 +21,8 @@
 // This test enables an exception and verifies that raising that exception
 // triggers SIGFPE.
 TEST(LlvmLibcExceptionStatusTest, RaiseAndCrash) {
-#if defined(LIBC_TARGET_ARCH_IS_AARCH64)
-  // Few aarch64 HW implementations do not trap exceptions. We skip this test
+#if defined(LIBC_TARGET_ARCH_IS_ANY_ARM) || defined(LIBC_TARGET_ARCH_IS_RISCV64)
+  // Few Arm HW implementations do not trap exceptions. We skip this test
   // completely on such HW.
   //
   // Whether HW supports trapping exceptions or not is deduced by enabling an
@@ -33,7 +33,7 @@ TEST(LlvmLibcExceptionStatusTest, RaiseAndCrash) {
   __llvm_libc::fputil::enable_except(FE_DIVBYZERO);
   if (__llvm_libc::fputil::get_except() == 0)
     return;
-#endif // defined(LIBC_TARGET_ARCH_IS_AARCH64)
+#endif // Architectures where exception trapping is not supported
 
   // TODO: Install a floating point exception handler and verify that the
   // the expected exception was raised. One will have to longjmp back from
diff --git a/libc/test/src/fenv/feenableexcept_test.cpp b/libc/test/src/fenv/feenableexcept_test.cpp
index 78596aa14e59f..c27a2bff6a038 100644
--- a/libc/test/src/fenv/feenableexcept_test.cpp
+++ b/libc/test/src/fenv/feenableexcept_test.cpp
@@ -16,8 +16,8 @@
 #include <fenv.h>
 
 TEST(LlvmLibcFEnvTest, EnableTest) {
-#if defined(LIBC_TARGET_ARCH_IS_AARCH64)
-  // Few aarch64 HW implementations do not trap exceptions. We skip this test
+#if defined(LIBC_TARGET_ARCH_IS_ANY_ARM) || defined(LIBC_TARGET_ARCH_IS_RISCV64)
+  // Few Arm HW implementations do not trap exceptions. We skip this test
   // completely on such HW.
   //
   // Whether HW supports trapping exceptions or not is deduced by enabling an
@@ -28,7 +28,7 @@ TEST(LlvmLibcFEnvTest, EnableTest) {
   __llvm_libc::feenableexcept(FE_DIVBYZERO);
   if (__llvm_libc::fegetexcept() == 0)
     return;
-#endif // defined(LIBC_TARGET_ARCH_IS_AARCH64)
+#endif // Architectures where exception trapping is not supported
 
   int excepts[] = {FE_DIVBYZERO, FE_INVALID, FE_INEXACT, FE_OVERFLOW,
                    FE_UNDERFLOW};
diff --git a/libc/test/src/fenv/feholdexcept_test.cpp b/libc/test/src/fenv/feholdexcept_test.cpp
index 043f7c49e68fc..40e336ef41af3 100644
--- a/libc/test/src/fenv/feholdexcept_test.cpp
+++ b/libc/test/src/fenv/feholdexcept_test.cpp
@@ -16,8 +16,8 @@
 #include <fenv.h>
 
 TEST(LlvmLibcFEnvTest, RaiseAndCrash) {
-#if defined(LIBC_TARGET_ARCH_IS_AARCH64)
-  // Few aarch64 HW implementations do not trap exceptions. We skip this test
+#if defined(LIBC_TARGET_ARCH_IS_ANY_ARM) || defined(LIBC_TARGET_ARCH_IS_RISCV64)
+  // Few Arm HW implementations do not trap exceptions. We skip this test
   // completely on such HW.
   //
   // Whether HW supports trapping exceptions or not is deduced by enabling an
@@ -28,7 +28,7 @@ TEST(LlvmLibcFEnvTest, RaiseAndCrash) {
   __llvm_libc::fputil::enable_except(FE_DIVBYZERO);
   if (__llvm_libc::fputil::get_except() == 0)
     return;
-#endif // defined(LIBC_TARGET_ARCH_IS_AARCH64)
+#endif // Architectures where exception trapping is not supported
 
   int excepts[] = {FE_DIVBYZERO, FE_INVALID, FE_INEXACT, FE_OVERFLOW,
                    FE_UNDERFLOW};
diff --git a/libc/test/src/math/CMakeLists.txt b/libc/test/src/math/CMakeLists.txt
index 54cabf89c87ab..37e6183593b68 100644
--- a/libc/test/src/math/CMakeLists.txt
+++ b/libc/test/src/math/CMakeLists.txt
@@ -10,7 +10,6 @@ add_fp_unittest(
   HDRS
     sdcomp26094.h
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.src.math.cosf
     libc.src.__support.CPP.array
@@ -39,7 +38,6 @@ add_fp_unittest(
   HDRS
     sdcomp26094.h
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.src.math.sinf
     libc.src.__support.CPP.array
@@ -68,7 +66,6 @@ add_fp_unittest(
   HDRS
     sdcomp26094.h
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.src.math.sincosf
     libc.src.__support.CPP.array
@@ -85,7 +82,6 @@ add_fp_unittest(
   HDRS
     sdcomp26094.h
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.src.math.tanf
     libc.src.__support.CPP.array
@@ -327,7 +323,6 @@ add_fp_unittest(
   HDRS
     RoundToIntegerTest.h
   DEPENDS
-    libc.include.errno
     libc.include.math
     libc.src.errno.errno
     libc.src.fenv.feclearexcept
@@ -347,7 +342,6 @@ add_fp_unittest(
   HDRS
     RoundToIntegerTest.h
   DEPENDS
-    libc.include.errno
     libc.include.math
     libc.src.errno.errno
     libc.src.fenv.feclearexcept
@@ -367,7 +361,6 @@ add_fp_unittest(
   HDRS
     RoundToIntegerTest.h
   DEPENDS
-    libc.include.errno
     libc.include.math
     libc.src.errno.errno
     libc.src.fenv.feclearexcept
@@ -387,7 +380,6 @@ add_fp_unittest(
   HDRS
     RoundToIntegerTest.h
   DEPENDS
-    libc.include.errno
     libc.include.math
     libc.src.errno.errno
     libc.src.fenv.feclearexcept
@@ -407,7 +399,6 @@ add_fp_unittest(
   HDRS
     RoundToIntegerTest.h
   DEPENDS
-    libc.include.errno
     libc.include.math
     libc.src.errno.errno
     libc.src.fenv.feclearexcept
@@ -427,7 +418,6 @@ add_fp_unittest(
   HDRS
     RoundToIntegerTest.h
   DEPENDS
-    libc.include.errno
     libc.include.math
     libc.src.errno.errno
     libc.src.fenv.feclearexcept
@@ -589,7 +579,6 @@ add_fp_unittest(
   SRCS
     expf_test.cpp
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.include.math
     libc.src.math.expf
@@ -604,7 +593,6 @@ add_fp_unittest(
   SRCS
     exp2f_test.cpp
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.include.math
     libc.src.math.exp2f
@@ -619,7 +607,6 @@ add_fp_unittest(
   SRCS
     exp10f_test.cpp
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.include.math
     libc.src.math.exp10f
@@ -1268,7 +1255,6 @@ add_fp_unittest(
   SRCS
     expm1f_test.cpp
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.include.math
     libc.src.math.expm1f
@@ -1295,7 +1281,6 @@ add_fp_unittest(
   SRCS
     logf_test.cpp
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.include.math
     libc.src.math.logf
@@ -1310,7 +1295,6 @@ add_fp_unittest(
   SRCS
     log2f_test.cpp
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.include.math
     libc.src.math.log2f
@@ -1325,7 +1309,6 @@ add_fp_unittest(
  SRCS
    log10_test.cpp
  DEPENDS
-   libc.include.errno
    libc.src.errno.errno
    libc.include.math
    libc.src.math.log10
@@ -1342,7 +1325,6 @@ add_fp_unittest(
   SRCS
     log10f_test.cpp
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.include.math
     libc.src.math.log10f
@@ -1357,7 +1339,6 @@ add_fp_unittest(
   SRCS
     log1pf_test.cpp
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.include.math
     libc.src.math.log1pf
@@ -1374,6 +1355,7 @@ add_fp_unittest(
     FModTest.h
   DEPENDS
     libc.include.math
+    libc.src.errno.errno
     libc.src.math.fmodf
     libc.src.__support.FPUtil.basic_operations
     libc.src.__support.FPUtil.nearest_integer_operations
@@ -1389,6 +1371,7 @@ add_fp_unittest(
     FModTest.h
   DEPENDS
     libc.include.math
+    libc.src.errno.errno
     libc.src.math.fmod
     libc.src.__support.FPUtil.basic_operations
     libc.src.__support.FPUtil.nearest_integer_operations
@@ -1419,7 +1402,6 @@ add_fp_unittest(
   HDRS
     sdcomp26094.h
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.src.math.coshf
     libc.src.__support.CPP.array
@@ -1436,7 +1418,6 @@ add_fp_unittest(
   HDRS
     sdcomp26094.h
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.src.math.sinhf
     libc.src.__support.CPP.array
@@ -1463,7 +1444,6 @@ add_fp_unittest(
   SRCS
     atanhf_test.cpp
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.src.math.atanhf
     libc.src.__support.FPUtil.fp_bits
@@ -1477,7 +1457,6 @@ add_fp_unittest(
   SRCS
     asinhf_test.cpp
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.src.math.asinhf
     libc.src.__support.FPUtil.fp_bits
@@ -1491,7 +1470,6 @@ add_fp_unittest(
   SRCS
     acoshf_test.cpp
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.src.math.acoshf
     libc.src.__support.FPUtil.fp_bits
@@ -1505,7 +1483,6 @@ add_fp_unittest(
   SRCS
     asinf_test.cpp
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.src.math.asinf
     libc.src.__support.FPUtil.fp_bits
@@ -1519,7 +1496,6 @@ add_fp_unittest(
   SRCS
     asin_test.cpp
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.src.math.asin
 )
@@ -1532,7 +1508,6 @@ add_fp_unittest(
   SRCS
     acosf_test.cpp
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.src.math.acosf
     libc.src.__support.FPUtil.fp_bits
@@ -1546,7 +1521,6 @@ add_fp_unittest(
   SRCS
     atanf_test.cpp
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.src.math.atanf
     libc.src.__support.FPUtil.fp_bits
@@ -1574,7 +1548,6 @@ add_fp_unittest(
   SRCS
     pow_test.cpp
   DEPENDS
-    libc.include.errno
     libc.src.errno.errno
     libc.src.math.pow
 )
diff --git a/libc/test/src/math/HypotTest.h b/libc/test/src/math/HypotTest.h
index e8b4cf06ad973..8d3fd88dec31d 100644
--- a/libc/test/src/math/HypotTest.h
+++ b/libc/test/src/math/HypotTest.h
@@ -59,7 +59,7 @@ class HypotTestTemplate : public __llvm_libc::testing::Test {
   }
 
   void test_subnormal_range(Func func) {
-    constexpr UIntType COUNT = 1000001;
+    constexpr UIntType COUNT = 100001;
     for (unsigned scale = 0; scale < 4; ++scale) {
       UIntType max_value = FPBits::MAX_SUBNORMAL << scale;
       UIntType step = (max_value - FPBits::MIN_SUBNORMAL) / COUNT;
@@ -84,7 +84,7 @@ class HypotTestTemplate : public __llvm_libc::testing::Test {
   }
 
   void test_normal_range(Func func) {
-    constexpr UIntType COUNT = 1000001;
+    constexpr UIntType COUNT = 100001;
     constexpr UIntType STEP = (FPBits::MAX_NORMAL - FPBits::MIN_NORMAL) / COUNT;
     for (int signs = 0; signs < 4; ++signs) {
       for (UIntType v = FPBits::MIN_NORMAL, w = FPBits::MAX_NORMAL;
diff --git a/libc/test/src/math/RoundToIntegerTest.h b/libc/test/src/math/RoundToIntegerTest.h
index 823ec8deaf27e..a8be9481d3c6e 100644
--- a/libc/test/src/math/RoundToIntegerTest.h
+++ b/libc/test/src/math/RoundToIntegerTest.h
@@ -42,7 +42,7 @@ class RoundToIntegerTestTemplate : public __llvm_libc::testing::Test {
 
   void test_one_input(RoundToIntegerFunc func, F input, I expected,
                       bool expectError) {
-    errno = 0;
+    libc_errno = 0;
     __llvm_libc::fputil::clear_except(FE_ALL_EXCEPT);
 
     ASSERT_EQ(func(input), expected);
diff --git a/libc/test/src/math/acosf_test.cpp b/libc/test/src/math/acosf_test.cpp
index ea79aff37304e..0d2db1275d123 100644
--- a/libc/test/src/math/acosf_test.cpp
+++ b/libc/test/src/math/acosf_test.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/acosf.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
@@ -23,7 +24,7 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(float)
 
 TEST(LlvmLibcAcosfTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
 
   EXPECT_FP_EQ(aNaN, __llvm_libc::acosf(aNaN));
   EXPECT_MATH_ERRNO(0);
diff --git a/libc/test/src/math/acoshf_test.cpp b/libc/test/src/math/acoshf_test.cpp
index 4ac01b7fcb94c..03f3a30c4ab07 100644
--- a/libc/test/src/math/acoshf_test.cpp
+++ b/libc/test/src/math/acoshf_test.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/acoshf.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
@@ -23,13 +24,13 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(float)
 
 TEST(LlvmLibcAcoshfTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
 
   EXPECT_FP_EQ(aNaN, __llvm_libc::acoshf(aNaN));
   EXPECT_MATH_ERRNO(0);
 
   EXPECT_FP_EQ(aNaN, __llvm_libc::acoshf(0.0f));
-  EXPECT_MATH_ERRNO(0);
+  EXPECT_MATH_ERRNO(EDOM);
 
   EXPECT_FP_EQ(0.0f, __llvm_libc::acoshf(1.0f));
   EXPECT_MATH_ERRNO(0);
@@ -38,7 +39,7 @@ TEST(LlvmLibcAcoshfTest, SpecialNumbers) {
   EXPECT_MATH_ERRNO(0);
 
   EXPECT_FP_EQ(aNaN, __llvm_libc::acoshf(neg_inf));
-  EXPECT_MATH_ERRNO(0);
+  EXPECT_MATH_ERRNO(EDOM);
 }
 
 TEST(LlvmLibcAcoshfTest, InFloatRange) {
diff --git a/libc/test/src/math/asin_test.cpp b/libc/test/src/math/asin_test.cpp
index d1b8483b8824e..2ccbebfdb22bf 100644
--- a/libc/test/src/math/asin_test.cpp
+++ b/libc/test/src/math/asin_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/math/asin.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
@@ -22,7 +23,7 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(double)
 
 TEST(LlvmLibcAsinTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
 
   EXPECT_FP_EQ(aNaN, __llvm_libc::asin(aNaN));
   EXPECT_MATH_ERRNO(0);
diff --git a/libc/test/src/math/asinf_test.cpp b/libc/test/src/math/asinf_test.cpp
index 890f21f51612e..439d23c8d1b58 100644
--- a/libc/test/src/math/asinf_test.cpp
+++ b/libc/test/src/math/asinf_test.cpp
@@ -8,6 +8,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/asinf.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
@@ -24,7 +25,7 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(float)
 
 TEST(LlvmLibcAsinfTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
 
   EXPECT_FP_EQ(aNaN, __llvm_libc::asinf(aNaN));
   EXPECT_MATH_ERRNO(0);
diff --git a/libc/test/src/math/asinhf_test.cpp b/libc/test/src/math/asinhf_test.cpp
index d2c6584454abe..906f834a78b22 100644
--- a/libc/test/src/math/asinhf_test.cpp
+++ b/libc/test/src/math/asinhf_test.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/asinhf.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
@@ -23,7 +24,7 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(float)
 
 TEST(LlvmLibcAsinhfTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
 
   EXPECT_FP_EQ(aNaN, __llvm_libc::asinhf(aNaN));
   EXPECT_MATH_ERRNO(0);
diff --git a/libc/test/src/math/atanf_test.cpp b/libc/test/src/math/atanf_test.cpp
index 7601ffb00dd5e..dbe1d485295ac 100644
--- a/libc/test/src/math/atanf_test.cpp
+++ b/libc/test/src/math/atanf_test.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/atanf.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
@@ -25,7 +26,7 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(float)
 
 TEST(LlvmLibcAtanfTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
   __llvm_libc::fputil::clear_except(FE_ALL_EXCEPT);
   EXPECT_FP_EQ(aNaN, __llvm_libc::atanf(aNaN));
   EXPECT_FP_EXCEPTION(0);
diff --git a/libc/test/src/math/atanhf_test.cpp b/libc/test/src/math/atanhf_test.cpp
index d4b884894a9e4..b11a9dfe1e2b0 100644
--- a/libc/test/src/math/atanhf_test.cpp
+++ b/libc/test/src/math/atanhf_test.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/atanhf.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
@@ -23,7 +24,7 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(float)
 
 TEST(LlvmLibcAtanhfTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
   __llvm_libc::fputil::clear_except(FE_ALL_EXCEPT);
   EXPECT_FP_EQ(aNaN, __llvm_libc::atanhf(aNaN));
   EXPECT_FP_EXCEPTION(0);
diff --git a/libc/test/src/math/cosf_test.cpp b/libc/test/src/math/cosf_test.cpp
index dae3c9336cfe9..721ca32068485 100644
--- a/libc/test/src/math/cosf_test.cpp
+++ b/libc/test/src/math/cosf_test.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/cosf.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
@@ -25,7 +26,7 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(float)
 
 TEST(LlvmLibcCosfTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
 
   EXPECT_FP_EQ(aNaN, __llvm_libc::cosf(aNaN));
   EXPECT_MATH_ERRNO(0);
diff --git a/libc/test/src/math/coshf_test.cpp b/libc/test/src/math/coshf_test.cpp
index c151fbaff4b28..88354a100a533 100644
--- a/libc/test/src/math/coshf_test.cpp
+++ b/libc/test/src/math/coshf_test.cpp
@@ -8,6 +8,7 @@
 
 #include "src/__support/CPP/array.h"
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/coshf.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
@@ -24,7 +25,7 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(float)
 
 TEST(LlvmLibcCoshfTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
 
   EXPECT_FP_EQ(aNaN, __llvm_libc::coshf(aNaN));
   EXPECT_MATH_ERRNO(0);
@@ -43,7 +44,7 @@ TEST(LlvmLibcCoshfTest, SpecialNumbers) {
 }
 
 TEST(LlvmLibcCoshfTest, Overflow) {
-  errno = 0;
+  libc_errno = 0;
   EXPECT_FP_EQ_WITH_EXCEPTION(
       inf, __llvm_libc::coshf(float(FPBits(0x7f7fffffU))), FE_OVERFLOW);
   EXPECT_MATH_ERRNO(ERANGE);
diff --git a/libc/test/src/math/exp10f_test.cpp b/libc/test/src/math/exp10f_test.cpp
index 1ea7b80087b35..2feb064cf565b 100644
--- a/libc/test/src/math/exp10f_test.cpp
+++ b/libc/test/src/math/exp10f_test.cpp
@@ -7,13 +7,13 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/exp10f.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
 #include "utils/MPFRWrapper/MPFRUtils.h"
 #include <math.h>
 
-#include <errno.h>
 #include <stdint.h>
 
 namespace mpfr = __llvm_libc::testing::mpfr;
@@ -21,7 +21,7 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(float)
 
 TEST(LlvmLibcExp10fTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
 
   EXPECT_FP_EQ(aNaN, __llvm_libc::exp10f(aNaN));
   EXPECT_MATH_ERRNO(0);
@@ -40,7 +40,7 @@ TEST(LlvmLibcExp10fTest, SpecialNumbers) {
 }
 
 TEST(LlvmLibcExp10fTest, Overflow) {
-  errno = 0;
+  libc_errno = 0;
   EXPECT_FP_EQ_WITH_EXCEPTION(
       inf, __llvm_libc::exp10f(float(FPBits(0x7f7fffffU))), FE_OVERFLOW);
   EXPECT_MATH_ERRNO(ERANGE);
@@ -55,7 +55,7 @@ TEST(LlvmLibcExp10fTest, Overflow) {
 }
 
 TEST(LlvmLibcExp10fTest, Underflow) {
-  errno = 0;
+  libc_errno = 0;
   EXPECT_FP_EQ_WITH_EXCEPTION(
       0.0f, __llvm_libc::exp10f(float(FPBits(0xff7fffffU))), FE_UNDERFLOW);
   EXPECT_MATH_ERRNO(ERANGE);
@@ -96,7 +96,7 @@ TEST(LlvmLibcExp10fTest, TrickyInputs) {
       0x41200000, // x = 10.0f
   };
   for (int i = 0; i < N; ++i) {
-    errno = 0;
+    libc_errno = 0;
     float x = float(FPBits(INPUTS[i]));
     EXPECT_MPFR_MATCH_ALL_ROUNDING(mpfr::Operation::Exp10, x,
                                    __llvm_libc::exp10f(x), 0.5);
@@ -112,14 +112,14 @@ TEST(LlvmLibcExp10fTest, InFloatRange) {
     float x = float(FPBits(v));
     if (isnan(x) || isinf(x))
       continue;
-    errno = 0;
+    libc_errno = 0;
     float result = __llvm_libc::exp10f(x);
 
     // If the computation resulted in an error or did not produce valid result
     // in the single-precision floating point range, then ignore comparing with
     // MPFR result as MPFR can still produce valid results because of its
     // wider precision.
-    if (isnan(result) || isinf(result) || errno != 0)
+    if (isnan(result) || isinf(result) || libc_errno != 0)
       continue;
     ASSERT_MPFR_MATCH_ALL_ROUNDING(mpfr::Operation::Exp10, x,
                                    __llvm_libc::exp10f(x), 0.5);
diff --git a/libc/test/src/math/exp2f_test.cpp b/libc/test/src/math/exp2f_test.cpp
index b38c4d43ed535..e442a7c57a5e1 100644
--- a/libc/test/src/math/exp2f_test.cpp
+++ b/libc/test/src/math/exp2f_test.cpp
@@ -8,13 +8,13 @@
 
 #include "src/__support/FPUtil/FPBits.h"
 #include "src/__support/macros/properties/cpu_features.h" // LIBC_TARGET_CPU_HAS_FMA
+#include "src/errno/libc_errno.h"
 #include "src/math/exp2f.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
 #include "utils/MPFRWrapper/MPFRUtils.h"
 #include <math.h>
 
-#include <errno.h>
 #include <stdint.h>
 
 namespace mpfr = __llvm_libc::testing::mpfr;
@@ -22,7 +22,7 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(float)
 
 TEST(LlvmLibcExp2fTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
 
   EXPECT_FP_EQ(aNaN, __llvm_libc::exp2f(aNaN));
   EXPECT_MATH_ERRNO(0);
@@ -41,7 +41,7 @@ TEST(LlvmLibcExp2fTest, SpecialNumbers) {
 }
 
 TEST(LlvmLibcExp2fTest, Overflow) {
-  errno = 0;
+  libc_errno = 0;
   EXPECT_FP_EQ_WITH_EXCEPTION(
       inf, __llvm_libc::exp2f(float(FPBits(0x7f7fffffU))), FE_OVERFLOW);
   EXPECT_MATH_ERRNO(ERANGE);
@@ -72,7 +72,7 @@ TEST(LlvmLibcExp2fTest, TrickyInputs) {
       0xc3150000U, /*-0x1.2ap+7f*/
   };
   for (int i = 0; i < N; ++i) {
-    errno = 0;
+    libc_errno = 0;
     float x = float(FPBits(INPUTS[i]));
     EXPECT_MPFR_MATCH_ALL_ROUNDING(mpfr::Operation::Exp2, x,
                                    __llvm_libc::exp2f(x), 0.5);
@@ -81,7 +81,7 @@ TEST(LlvmLibcExp2fTest, TrickyInputs) {
 }
 
 TEST(LlvmLibcExp2fTest, Underflow) {
-  errno = 0;
+  libc_errno = 0;
   EXPECT_FP_EQ_WITH_EXCEPTION(
       0.0f, __llvm_libc::exp2f(float(FPBits(0xff7fffffU))), FE_UNDERFLOW);
   EXPECT_MATH_ERRNO(ERANGE);
@@ -109,14 +109,14 @@ TEST(LlvmLibcExp2fTest, InFloatRange) {
     float x = float(FPBits(v));
     if (isnan(x) || isinf(x))
       continue;
-    errno = 0;
+    libc_errno = 0;
     float result = __llvm_libc::exp2f(x);
 
     // If the computation resulted in an error or did not produce valid result
     // in the single-precision floating point range, then ignore comparing with
     // MPFR result as MPFR can still produce valid results because of its
     // wider precision.
-    if (isnan(result) || isinf(result) || errno != 0)
+    if (isnan(result) || isinf(result) || libc_errno != 0)
       continue;
     ASSERT_MPFR_MATCH_ALL_ROUNDING(mpfr::Operation::Exp2, x,
                                    __llvm_libc::exp2f(x), 0.5);
diff --git a/libc/test/src/math/expf_test.cpp b/libc/test/src/math/expf_test.cpp
index bcd1dcdb8b6e1..1926c9c03da0c 100644
--- a/libc/test/src/math/expf_test.cpp
+++ b/libc/test/src/math/expf_test.cpp
@@ -7,13 +7,13 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/expf.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
 #include "utils/MPFRWrapper/MPFRUtils.h"
 #include <math.h>
 
-#include <errno.h>
 #include <stdint.h>
 
 namespace mpfr = __llvm_libc::testing::mpfr;
@@ -21,7 +21,7 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(float)
 
 TEST(LlvmLibcExpfTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
 
   EXPECT_FP_EQ(aNaN, __llvm_libc::expf(aNaN));
   EXPECT_MATH_ERRNO(0);
@@ -40,7 +40,7 @@ TEST(LlvmLibcExpfTest, SpecialNumbers) {
 }
 
 TEST(LlvmLibcExpfTest, Overflow) {
-  errno = 0;
+  libc_errno = 0;
   EXPECT_FP_EQ_WITH_EXCEPTION(
       inf, __llvm_libc::expf(float(FPBits(0x7f7fffffU))), FE_OVERFLOW);
   EXPECT_MATH_ERRNO(ERANGE);
@@ -55,7 +55,7 @@ TEST(LlvmLibcExpfTest, Overflow) {
 }
 
 TEST(LlvmLibcExpfTest, Underflow) {
-  errno = 0;
+  libc_errno = 0;
   EXPECT_FP_EQ_WITH_EXCEPTION(
       0.0f, __llvm_libc::expf(float(FPBits(0xff7fffffU))), FE_UNDERFLOW);
   EXPECT_MATH_ERRNO(ERANGE);
@@ -76,7 +76,7 @@ TEST(LlvmLibcExpfTest, Underflow) {
 TEST(LlvmLibcExpfTest, Borderline) {
   float x;
 
-  errno = 0;
+  libc_errno = 0;
   x = float(FPBits(0x42affff8U));
   ASSERT_MPFR_MATCH_ALL_ROUNDING(mpfr::Operation::Exp, x, __llvm_libc::expf(x),
                                  0.5);
@@ -110,14 +110,14 @@ TEST(LlvmLibcExpfTest, InFloatRange) {
     float x = float(FPBits(v));
     if (isnan(x) || isinf(x))
       continue;
-    errno = 0;
+    libc_errno = 0;
     float result = __llvm_libc::expf(x);
 
     // If the computation resulted in an error or did not produce valid result
     // in the single-precision floating point range, then ignore comparing with
     // MPFR result as MPFR can still produce valid results because of its
     // wider precision.
-    if (isnan(result) || isinf(result) || errno != 0)
+    if (isnan(result) || isinf(result) || libc_errno != 0)
       continue;
     EXPECT_MPFR_MATCH_ALL_ROUNDING(mpfr::Operation::Exp, x,
                                    __llvm_libc::expf(x), 0.5);
diff --git a/libc/test/src/math/expm1f_test.cpp b/libc/test/src/math/expm1f_test.cpp
index 74df79fe7a390..81bc18a18dccc 100644
--- a/libc/test/src/math/expm1f_test.cpp
+++ b/libc/test/src/math/expm1f_test.cpp
@@ -7,13 +7,13 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/expm1f.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
 #include "utils/MPFRWrapper/MPFRUtils.h"
 #include <math.h>
 
-#include <errno.h>
 #include <stdint.h>
 
 namespace mpfr = __llvm_libc::testing::mpfr;
@@ -21,7 +21,7 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(float)
 
 TEST(LlvmLibcExpm1fTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
 
   EXPECT_FP_EQ(aNaN, __llvm_libc::expm1f(aNaN));
   EXPECT_MATH_ERRNO(0);
@@ -40,7 +40,7 @@ TEST(LlvmLibcExpm1fTest, SpecialNumbers) {
 }
 
 TEST(LlvmLibcExpm1fTest, Overflow) {
-  errno = 0;
+  libc_errno = 0;
   EXPECT_FP_EQ_WITH_EXCEPTION(
       inf, __llvm_libc::expm1f(float(FPBits(0x7f7fffffU))), FE_OVERFLOW);
   EXPECT_MATH_ERRNO(ERANGE);
@@ -55,7 +55,7 @@ TEST(LlvmLibcExpm1fTest, Overflow) {
 }
 
 TEST(LlvmLibcExpm1fTest, Underflow) {
-  errno = 0;
+  libc_errno = 0;
   EXPECT_FP_EQ(-1.0f, __llvm_libc::expm1f(float(FPBits(0xff7fffffU))));
 
   float x = float(FPBits(0xc2cffff8U));
@@ -70,7 +70,7 @@ TEST(LlvmLibcExpm1fTest, Underflow) {
 TEST(LlvmLibcExpm1fTest, Borderline) {
   float x;
 
-  errno = 0;
+  libc_errno = 0;
   x = float(FPBits(0x42affff8U));
   ASSERT_MPFR_MATCH_ALL_ROUNDING(mpfr::Operation::Expm1, x,
                                  __llvm_libc::expm1f(x), 0.5);
@@ -119,14 +119,14 @@ TEST(LlvmLibcExpm1fTest, InFloatRange) {
     float x = float(FPBits(v));
     if (isnan(x) || isinf(x))
       continue;
-    errno = 0;
+    libc_errno = 0;
     float result = __llvm_libc::expm1f(x);
 
     // If the computation resulted in an error or did not produce valid result
     // in the single-precision floating point range, then ignore comparing with
     // MPFR result as MPFR can still produce valid results because of its
     // wider precision.
-    if (isnan(result) || isinf(result) || errno != 0)
+    if (isnan(result) || isinf(result) || libc_errno != 0)
       continue;
     ASSERT_MPFR_MATCH_ALL_ROUNDING(mpfr::Operation::Expm1, x,
                                    __llvm_libc::expm1f(x), 0.5);
diff --git a/libc/test/src/math/log10_test.cpp b/libc/test/src/math/log10_test.cpp
index edefa4cf02a5d..15e5ee6ae918a 100644
--- a/libc/test/src/math/log10_test.cpp
+++ b/libc/test/src/math/log10_test.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/log10.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
@@ -84,7 +85,7 @@ TEST(LlvmLibcLog10Test, InDoubleRange) {
       double x = FPBits(v).get_val();
       if (isnan(x) || isinf(x) || x < 0.0)
         continue;
-      errno = 0;
+      libc_errno = 0;
       double result = __llvm_libc::log10(x);
       ++cc;
       if (isnan(result))
diff --git a/libc/test/src/math/log1pf_test.cpp b/libc/test/src/math/log1pf_test.cpp
index 973c71945055e..e5137e62ce0a1 100644
--- a/libc/test/src/math/log1pf_test.cpp
+++ b/libc/test/src/math/log1pf_test.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/log1pf.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
@@ -75,7 +76,7 @@ TEST(LlvmLibclog1pfTest, InFloatRange) {
     float x = float(FPBits(v));
     if (isnan(x) || isinf(x))
       continue;
-    errno = 0;
+    libc_errno = 0;
     ASSERT_MPFR_MATCH_ALL_ROUNDING(mpfr::Operation::Log1p, x,
                                    __llvm_libc::log1pf(x), 0.5);
   }
diff --git a/libc/test/src/math/log2f_test.cpp b/libc/test/src/math/log2f_test.cpp
index 08619efb5203f..73bc2b02c5dcb 100644
--- a/libc/test/src/math/log2f_test.cpp
+++ b/libc/test/src/math/log2f_test.cpp
@@ -7,13 +7,13 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/log2f.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
 #include "utils/MPFRWrapper/MPFRUtils.h"
 #include <math.h>
 
-#include <errno.h>
 #include <stdint.h>
 
 namespace mpfr = __llvm_libc::testing::mpfr;
@@ -50,13 +50,13 @@ TEST(LlvmLibcLog2fTest, InFloatRange) {
     float x = float(FPBits(v));
     if (isnan(x) || isinf(x))
       continue;
-    errno = 0;
+    libc_errno = 0;
     float result = __llvm_libc::log2f(x);
     // If the computation resulted in an error or did not produce valid result
     // in the single-precision floating point range, then ignore comparing with
     // MPFR result as MPFR can still produce valid results because of its
     // wider precision.
-    if (isnan(result) || isinf(result) || errno != 0)
+    if (isnan(result) || isinf(result) || libc_errno != 0)
       continue;
     ASSERT_MPFR_MATCH_ALL_ROUNDING(mpfr::Operation::Log2, x,
                                    __llvm_libc::log2f(x), 0.5);
diff --git a/libc/test/src/math/logf_test.cpp b/libc/test/src/math/logf_test.cpp
index 4feb1ad99bfca..bf3aad7893992 100644
--- a/libc/test/src/math/logf_test.cpp
+++ b/libc/test/src/math/logf_test.cpp
@@ -13,7 +13,6 @@
 #include "utils/MPFRWrapper/MPFRUtils.h"
 #include <math.h>
 
-#include <errno.h>
 #include <stdint.h>
 
 namespace mpfr = __llvm_libc::testing::mpfr;
diff --git a/libc/test/src/math/pow_test.cpp b/libc/test/src/math/pow_test.cpp
index 2be8b87500a4b..eff578ae9e380 100644
--- a/libc/test/src/math/pow_test.cpp
+++ b/libc/test/src/math/pow_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/math/pow.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
@@ -22,7 +23,7 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(double)
 
 TEST(LlvmLibcAsinTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
 
   EXPECT_FP_EQ(aNaN, __llvm_libc::pow(aNaN, aNaN));
   EXPECT_MATH_ERRNO(0);
diff --git a/libc/test/src/math/sincosf_test.cpp b/libc/test/src/math/sincosf_test.cpp
index 1fd89190c34d2..8f580c9243ab1 100644
--- a/libc/test/src/math/sincosf_test.cpp
+++ b/libc/test/src/math/sincosf_test.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/sincosf.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
@@ -25,7 +26,7 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(float)
 
 TEST(LlvmLibcSinCosfTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
   float sin, cos;
 
   __llvm_libc::sincosf(aNaN, &sin, &cos);
diff --git a/libc/test/src/math/sinf_test.cpp b/libc/test/src/math/sinf_test.cpp
index 3559515c80a55..a4a2eda7fd376 100644
--- a/libc/test/src/math/sinf_test.cpp
+++ b/libc/test/src/math/sinf_test.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/sinf.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
@@ -25,7 +26,7 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(float)
 
 TEST(LlvmLibcSinfTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
 
   EXPECT_FP_EQ(aNaN, __llvm_libc::sinf(aNaN));
   EXPECT_MATH_ERRNO(0);
diff --git a/libc/test/src/math/sinhf_test.cpp b/libc/test/src/math/sinhf_test.cpp
index 6bec6426d1d51..c0a07756f17c1 100644
--- a/libc/test/src/math/sinhf_test.cpp
+++ b/libc/test/src/math/sinhf_test.cpp
@@ -8,6 +8,7 @@
 
 #include "src/__support/CPP/array.h"
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/sinhf.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
@@ -24,7 +25,7 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(float)
 
 TEST(LlvmLibcSinhfTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
 
   EXPECT_FP_EQ(aNaN, __llvm_libc::sinhf(aNaN));
   EXPECT_MATH_ERRNO(0);
@@ -67,7 +68,7 @@ TEST(LlvmLibcSinhfTest, SmallValues) {
 }
 
 TEST(LlvmLibcSinhfTest, Overflow) {
-  errno = 0;
+  libc_errno = 0;
   EXPECT_FP_EQ_WITH_EXCEPTION(
       inf, __llvm_libc::sinhf(float(FPBits(0x7f7fffffU))), FE_OVERFLOW);
   EXPECT_MATH_ERRNO(ERANGE);
diff --git a/libc/test/src/math/tanf_test.cpp b/libc/test/src/math/tanf_test.cpp
index cfe1762ecb58d..a513a036aea16 100644
--- a/libc/test/src/math/tanf_test.cpp
+++ b/libc/test/src/math/tanf_test.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/tanf.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
@@ -25,7 +26,7 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(float)
 
 TEST(LlvmLibcTanfTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
 
   EXPECT_FP_EQ(aNaN, __llvm_libc::tanf(aNaN));
   EXPECT_MATH_ERRNO(0);
diff --git a/libc/test/src/math/tanhf_test.cpp b/libc/test/src/math/tanhf_test.cpp
index dc8af63f2b9e8..6e7a1b472a611 100644
--- a/libc/test/src/math/tanhf_test.cpp
+++ b/libc/test/src/math/tanhf_test.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/FPUtil/FPBits.h"
+#include "src/errno/libc_errno.h"
 #include "src/math/tanhf.h"
 #include "test/UnitTest/FPMatcher.h"
 #include "test/UnitTest/Test.h"
@@ -23,7 +24,7 @@ namespace mpfr = __llvm_libc::testing::mpfr;
 DECLARE_SPECIAL_CONSTANTS(float)
 
 TEST(LlvmLibcTanhfTest, SpecialNumbers) {
-  errno = 0;
+  libc_errno = 0;
 
   EXPECT_FP_EQ(aNaN, __llvm_libc::tanhf(aNaN));
   EXPECT_MATH_ERRNO(0);
diff --git a/libc/test/src/sched/CMakeLists.txt b/libc/test/src/sched/CMakeLists.txt
index cddb574fcbc13..ad4dfdb553e1b 100644
--- a/libc/test/src/sched/CMakeLists.txt
+++ b/libc/test/src/sched/CMakeLists.txt
@@ -7,10 +7,10 @@ add_libc_unittest(
   SRCS
     affinity_test.cpp
   DEPENDS
-    libc.include.errno
     libc.include.sched
     libc.include.sys_syscall
     libc.src.__support.OSUtil.osutil
+    libc.src.errno.errno
     libc.src.sched.sched_getaffinity
     libc.src.sched.sched_setaffinity
     libc.test.errno_setter_matcher
@@ -23,10 +23,10 @@ add_libc_unittest(
   SRCS
     cpu_count_test.cpp
   DEPENDS
-    libc.include.errno
     libc.include.sched
     libc.include.sys_syscall
     libc.src.__support.OSUtil.osutil
+    libc.src.errno.errno
     libc.src.sched.sched_getaffinity
     libc.src.sched.__sched_getcpucount
     libc.test.errno_setter_matcher
diff --git a/libc/test/src/sched/affinity_test.cpp b/libc/test/src/sched/affinity_test.cpp
index 446e85721e91f..b8da2ebf71480 100644
--- a/libc/test/src/sched/affinity_test.cpp
+++ b/libc/test/src/sched/affinity_test.cpp
@@ -7,17 +7,17 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/OSUtil/syscall.h"
+#include "src/errno/libc_errno.h"
 #include "src/sched/sched_getaffinity.h"
 #include "src/sched/sched_setaffinity.h"
 #include "test/ErrnoSetterMatcher.h"
 
-#include <errno.h>
 #include <sched.h>
 #include <sys/syscall.h>
 
 TEST(LlvmLibcSchedAffinityTest, SmokeTest) {
   cpu_set_t mask;
-  errno = 0;
+  libc_errno = 0;
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
   pid_t tid = __llvm_libc::syscall_impl(SYS_gettid);
   ASSERT_GT(tid, pid_t(0));
@@ -32,13 +32,13 @@ TEST(LlvmLibcSchedAffinityTest, BadMask) {
   using __llvm_libc::testing::ErrnoSetterMatcher::Fails;
   pid_t tid = __llvm_libc::syscall_impl(SYS_gettid);
 
-  errno = 0;
+  libc_errno = 0;
   ASSERT_THAT(__llvm_libc::sched_getaffinity(tid, sizeof(cpu_set_t), nullptr),
               Fails(EFAULT));
 
-  errno = 0;
+  libc_errno = 0;
   ASSERT_THAT(__llvm_libc::sched_setaffinity(tid, sizeof(cpu_set_t), nullptr),
               Fails(EFAULT));
 
-  errno = 0;
+  libc_errno = 0;
 }
diff --git a/libc/test/src/sched/cpu_count_test.cpp b/libc/test/src/sched/cpu_count_test.cpp
index 02b3ff2b381e7..1ca643ce74869 100644
--- a/libc/test/src/sched/cpu_count_test.cpp
+++ b/libc/test/src/sched/cpu_count_test.cpp
@@ -7,17 +7,17 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/OSUtil/syscall.h"
+#include "src/errno/libc_errno.h"
 #include "src/sched/sched_getaffinity.h"
 #include "src/sched/sched_getcpucount.h"
 #include "test/ErrnoSetterMatcher.h"
 
-#include <errno.h>
 #include <sched.h>
 #include <sys/syscall.h>
 
 TEST(LlvmLibcSchedAffinityTest, SmokeTest) {
   cpu_set_t mask;
-  errno = 0;
+  libc_errno = 0;
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
   pid_t tid = __llvm_libc::syscall_impl(SYS_gettid);
   ASSERT_GT(tid, pid_t(0));
diff --git a/libc/test/src/stdlib/strtod_test.cpp b/libc/test/src/stdlib/strtod_test.cpp
index 3160ef3210c8a..779c810f72083 100644
--- a/libc/test/src/stdlib/strtod_test.cpp
+++ b/libc/test/src/stdlib/strtod_test.cpp
@@ -191,4 +191,9 @@ TEST_F(LlvmLibcStrToDTest, FuzzFailures) {
            "NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN?",
            0, uint64_t(0));
   run_test("0x.666E40", 9, uint64_t(0x3fd99b9000000000));
+
+  // glibc version 2.36 and higher (not tested with lower versions) disagrees
+  // with this result, but ours is correct for the nearest rounding mode. See
+  // this bug: https://sourceware.org/bugzilla/show_bug.cgi?id=30220
+  run_test("0x30000002222225p-1077", 22, uint64_t(0x0006000000444445), ERANGE);
 }
diff --git a/libc/test/src/string/CMakeLists.txt b/libc/test/src/string/CMakeLists.txt
index c9623a330b1c2..67f49ab3a114f 100644
--- a/libc/test/src/string/CMakeLists.txt
+++ b/libc/test/src/string/CMakeLists.txt
@@ -161,7 +161,6 @@ add_libc_unittest(
   SRCS
     strdup_test.cpp
   DEPENDS
-    libc.include.errno
     libc.include.stdlib
     libc.src.string.strdup
     libc.src.errno.errno
diff --git a/libc/test/src/string/strdup_test.cpp b/libc/test/src/string/strdup_test.cpp
index ffbc55a70fb21..ee46882233dd3 100644
--- a/libc/test/src/string/strdup_test.cpp
+++ b/libc/test/src/string/strdup_test.cpp
@@ -6,18 +6,18 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/string/strdup.h"
 #include "test/UnitTest/Test.h"
 
-#include <errno.h>
 #include <stdlib.h>
 
 TEST(LlvmLibcStrDupTest, EmptyString) {
   const char *empty = "";
 
-  errno = 0;
+  libc_errno = 0;
   char *result = __llvm_libc::strdup(empty);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
 
   ASSERT_NE(result, static_cast<char *>(nullptr));
   ASSERT_NE(empty, const_cast<const char *>(result));
@@ -28,9 +28,9 @@ TEST(LlvmLibcStrDupTest, EmptyString) {
 TEST(LlvmLibcStrDupTest, AnyString) {
   const char *abc = "abc";
 
-  errno = 0;
+  libc_errno = 0;
   char *result = __llvm_libc::strdup(abc);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
 
   ASSERT_NE(result, static_cast<char *>(nullptr));
   ASSERT_NE(abc, const_cast<const char *>(result));
@@ -39,9 +39,9 @@ TEST(LlvmLibcStrDupTest, AnyString) {
 }
 
 TEST(LlvmLibcStrDupTest, NullPtr) {
-  errno = 0;
+  libc_errno = 0;
   char *result = __llvm_libc::strdup(nullptr);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
 
   ASSERT_EQ(result, static_cast<char *>(nullptr));
 }
diff --git a/libc/test/src/string/strerror_test.cpp b/libc/test/src/string/strerror_test.cpp
index 3830e1b6c01d8..20a8fba6f30fb 100644
--- a/libc/test/src/string/strerror_test.cpp
+++ b/libc/test/src/string/strerror_test.cpp
@@ -150,7 +150,7 @@ TEST(LlvmLibcStrErrorTest, KnownErrors) {
   };
 
   for (size_t i = 0; i < (sizeof(message_array) / sizeof(char *)); ++i) {
-    EXPECT_STREQ(__llvm_libc::strerror(i), message_array[i]);
+    EXPECT_STREQ(__llvm_libc::strerror(static_cast<int>(i)), message_array[i]);
   }
 }
 
diff --git a/libc/test/src/sys/mman/linux/CMakeLists.txt b/libc/test/src/sys/mman/linux/CMakeLists.txt
index cbc1accd2dfd3..8b35842cd810a 100644
--- a/libc/test/src/sys/mman/linux/CMakeLists.txt
+++ b/libc/test/src/sys/mman/linux/CMakeLists.txt
@@ -7,7 +7,6 @@ add_libc_unittest(
   SRCS
     mmap_test.cpp
   DEPENDS
-    libc.include.errno
     libc.include.sys_mman
     libc.src.errno.errno
     libc.src.sys.mman.mmap
@@ -23,7 +22,6 @@ if (NOT LLVM_USE_SANITIZER)
     SRCS
       mprotect_test.cpp
     DEPENDS
-      libc.include.errno
       libc.include.sys_mman
       libc.include.signal
       libc.src.errno.errno
@@ -41,7 +39,6 @@ add_libc_unittest(
   SRCS
   madvise_test.cpp
   DEPENDS
-    libc.include.errno
     libc.include.sys_mman
     libc.src.errno.errno
     libc.src.sys.mman.mmap
@@ -58,7 +55,6 @@ add_libc_unittest(
   SRCS
   posix_madvise_test.cpp
   DEPENDS
-    libc.include.errno
     libc.include.sys_mman
     libc.src.errno.errno
     libc.src.sys.mman.mmap
diff --git a/libc/test/src/sys/mman/linux/madvise_test.cpp b/libc/test/src/sys/mman/linux/madvise_test.cpp
index 4f8c24022dbbc..dcc6c25b9f1de 100644
--- a/libc/test/src/sys/mman/linux/madvise_test.cpp
+++ b/libc/test/src/sys/mman/linux/madvise_test.cpp
@@ -6,13 +6,13 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/sys/mman/madvise.h"
 #include "src/sys/mman/mmap.h"
 #include "src/sys/mman/munmap.h"
 #include "test/ErrnoSetterMatcher.h"
 #include "test/UnitTest/Test.h"
 
-#include <errno.h>
 #include <sys/mman.h>
 
 using __llvm_libc::testing::ErrnoSetterMatcher::Fails;
@@ -20,10 +20,10 @@ using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
 
 TEST(LlvmLibcMadviseTest, NoError) {
   size_t alloc_size = 128;
-  errno = 0;
+  libc_errno = 0;
   void *addr = __llvm_libc::mmap(nullptr, alloc_size, PROT_READ,
                                  MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
-  EXPECT_EQ(0, errno);
+  EXPECT_EQ(0, libc_errno);
   EXPECT_NE(addr, MAP_FAILED);
 
   EXPECT_THAT(__llvm_libc::madvise(addr, alloc_size, MADV_RANDOM), Succeeds());
@@ -37,6 +37,6 @@ TEST(LlvmLibcMadviseTest, NoError) {
 }
 
 TEST(LlvmLibcMadviseTest, Error_BadPtr) {
-  errno = 0;
+  libc_errno = 0;
   EXPECT_THAT(__llvm_libc::madvise(nullptr, 8, MADV_SEQUENTIAL), Fails(ENOMEM));
 }
diff --git a/libc/test/src/sys/mman/linux/mmap_test.cpp b/libc/test/src/sys/mman/linux/mmap_test.cpp
index 41fb737b35630..015dad8ad947d 100644
--- a/libc/test/src/sys/mman/linux/mmap_test.cpp
+++ b/libc/test/src/sys/mman/linux/mmap_test.cpp
@@ -6,12 +6,12 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/sys/mman/mmap.h"
 #include "src/sys/mman/munmap.h"
 #include "test/ErrnoSetterMatcher.h"
 #include "test/UnitTest/Test.h"
 
-#include <errno.h>
 #include <sys/mman.h>
 
 using __llvm_libc::testing::ErrnoSetterMatcher::Fails;
@@ -19,10 +19,10 @@ using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
 
 TEST(LlvmLibcMMapTest, NoError) {
   size_t alloc_size = 128;
-  errno = 0;
+  libc_errno = 0;
   void *addr = __llvm_libc::mmap(nullptr, alloc_size, PROT_READ,
                                  MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
-  EXPECT_EQ(0, errno);
+  EXPECT_EQ(0, libc_errno);
   EXPECT_NE(addr, MAP_FAILED);
 
   int *array = reinterpret_cast<int *>(addr);
@@ -34,7 +34,7 @@ TEST(LlvmLibcMMapTest, NoError) {
 }
 
 TEST(LlvmLibcMMapTest, Error_InvalidSize) {
-  errno = 0;
+  libc_errno = 0;
   void *addr = __llvm_libc::mmap(nullptr, 0, PROT_READ,
                                  MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
   EXPECT_THAT(addr, Fails(EINVAL, MAP_FAILED));
diff --git a/libc/test/src/sys/mman/linux/mprotect_test.cpp b/libc/test/src/sys/mman/linux/mprotect_test.cpp
index 66c2e9a031743..ea356149a1763 100644
--- a/libc/test/src/sys/mman/linux/mprotect_test.cpp
+++ b/libc/test/src/sys/mman/linux/mprotect_test.cpp
@@ -6,13 +6,13 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/sys/mman/mmap.h"
 #include "src/sys/mman/mprotect.h"
 #include "src/sys/mman/munmap.h"
 #include "test/ErrnoSetterMatcher.h"
 #include "test/UnitTest/Test.h"
 
-#include <errno.h>
 #include <signal.h>
 #include <sys/mman.h>
 
@@ -21,10 +21,10 @@ using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
 
 TEST(LlvmLibcMProtectTest, NoError) {
   size_t alloc_size = 128;
-  errno = 0;
+  libc_errno = 0;
   void *addr = __llvm_libc::mmap(nullptr, alloc_size, PROT_READ,
                                  MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
-  EXPECT_EQ(0, errno);
+  EXPECT_EQ(0, libc_errno);
   EXPECT_NE(addr, MAP_FAILED);
 
   int *array = reinterpret_cast<int *>(addr);
diff --git a/libc/test/src/sys/mman/linux/posix_madvise_test.cpp b/libc/test/src/sys/mman/linux/posix_madvise_test.cpp
index 151e043351778..cd1dbc0e31ed5 100644
--- a/libc/test/src/sys/mman/linux/posix_madvise_test.cpp
+++ b/libc/test/src/sys/mman/linux/posix_madvise_test.cpp
@@ -6,13 +6,13 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/sys/mman/mmap.h"
 #include "src/sys/mman/munmap.h"
 #include "src/sys/mman/posix_madvise.h"
 #include "test/ErrnoSetterMatcher.h"
 #include "test/UnitTest/Test.h"
 
-#include <errno.h>
 #include <sys/mman.h>
 
 using __llvm_libc::testing::ErrnoSetterMatcher::Fails;
@@ -20,10 +20,10 @@ using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
 
 TEST(LlvmLibcPosixMadviseTest, NoError) {
   size_t alloc_size = 128;
-  errno = 0;
+  libc_errno = 0;
   void *addr = __llvm_libc::mmap(nullptr, alloc_size, PROT_READ,
                                  MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
-  EXPECT_EQ(0, errno);
+  EXPECT_EQ(0, libc_errno);
   EXPECT_NE(addr, MAP_FAILED);
 
   EXPECT_EQ(__llvm_libc::posix_madvise(addr, alloc_size, POSIX_MADV_RANDOM), 0);
@@ -37,7 +37,7 @@ TEST(LlvmLibcPosixMadviseTest, NoError) {
 }
 
 TEST(LlvmLibcPosixMadviseTest, Error_BadPtr) {
-  errno = 0;
+  libc_errno = 0;
   // posix_madvise is a no-op on DONTNEED, so it shouldn't fail even with the
   // nullptr.
   EXPECT_EQ(__llvm_libc::posix_madvise(nullptr, 8, POSIX_MADV_DONTNEED), 0);
diff --git a/libc/test/src/sys/random/linux/CMakeLists.txt b/libc/test/src/sys/random/linux/CMakeLists.txt
index 3dcfd8626c055..6277749087697 100644
--- a/libc/test/src/sys/random/linux/CMakeLists.txt
+++ b/libc/test/src/sys/random/linux/CMakeLists.txt
@@ -7,7 +7,6 @@ add_libc_unittest(
   SRCS
     getrandom_test.cpp
   DEPENDS
-    libc.include.errno
     libc.include.math
     libc.include.sys_random
     libc.src.errno.errno
diff --git a/libc/test/src/sys/random/linux/getrandom_test.cpp b/libc/test/src/sys/random/linux/getrandom_test.cpp
index 3514613020fe4..34b663627fe64 100644
--- a/libc/test/src/sys/random/linux/getrandom_test.cpp
+++ b/libc/test/src/sys/random/linux/getrandom_test.cpp
@@ -1,3 +1,12 @@
+//===-- Unittests for getrandom -------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "src/errno/libc_errno.h"
 #include "src/math/fabs.h"
 #include "src/sys/random/getrandom.h"
 #include "test/ErrnoSetterMatcher.h"
@@ -7,17 +16,17 @@ TEST(LlvmLibcGetRandomTest, InvalidFlag) {
   using __llvm_libc::testing::ErrnoSetterMatcher::Fails;
   static constexpr size_t SIZE = 256;
   char data[SIZE];
-  errno = 0;
+  libc_errno = 0;
   ASSERT_THAT(__llvm_libc::getrandom(data, SIZE, -1), Fails(EINVAL));
-  errno = 0;
+  libc_errno = 0;
 }
 
 TEST(LlvmLibcGetRandomTest, InvalidBuffer) {
   using __llvm_libc::testing::ErrnoSetterMatcher::Fails;
 
-  errno = 0;
+  libc_errno = 0;
   ASSERT_THAT(__llvm_libc::getrandom(nullptr, 65536, 0), Fails(EFAULT));
-  errno = 0;
+  libc_errno = 0;
 }
 
 TEST(LlvmLibcGetRandomTest, PiEstimation) {
diff --git a/libc/test/src/sys/resource/getrlimit_setrlimit_test.cpp b/libc/test/src/sys/resource/getrlimit_setrlimit_test.cpp
index 285624af6ef07..aa7a549876aaa 100644
--- a/libc/test/src/sys/resource/getrlimit_setrlimit_test.cpp
+++ b/libc/test/src/sys/resource/getrlimit_setrlimit_test.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/CPP/string_view.h"
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/sys/resource/getrlimit.h"
 #include "src/sys/resource/setrlimit.h"
@@ -15,7 +16,6 @@
 #include "test/ErrnoSetterMatcher.h"
 #include "test/UnitTest/Test.h"
 
-#include <errno.h>
 #include <sys/resource.h>
 
 TEST(LlvmLibcResourceLimitsTest, SetNoFileLimit) {
@@ -28,14 +28,14 @@ TEST(LlvmLibcResourceLimitsTest, SetNoFileLimit) {
 
   constexpr const char *TEST_FILE1 = "testdata/resource_limits1.test";
   constexpr const char *TEST_FILE2 = "testdata/resource_limits2.test";
-  errno = 0;
+  libc_errno = 0;
 
   int fd1 = __llvm_libc::open(TEST_FILE1, O_CREAT | O_WRONLY, S_IRWXU);
   ASSERT_GT(fd1, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   int fd2 = __llvm_libc::open(TEST_FILE2, O_CREAT | O_WRONLY, S_IRWXU);
   ASSERT_GT(fd2, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
 
   ASSERT_THAT(__llvm_libc::close(fd1), Succeeds(0));
   ASSERT_THAT(__llvm_libc::close(fd2), Succeeds(0));
@@ -48,22 +48,22 @@ TEST(LlvmLibcResourceLimitsTest, SetNoFileLimit) {
   // One can now only open one of the files successfully.
   fd1 = __llvm_libc::open(TEST_FILE1, O_RDONLY);
   ASSERT_GT(fd1, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   fd2 = __llvm_libc::open(TEST_FILE2, O_RDONLY);
   ASSERT_LT(fd2, 0);
-  ASSERT_NE(errno, 0);
+  ASSERT_NE(libc_errno, 0);
 
-  errno = 0;
+  libc_errno = 0;
   ASSERT_THAT(__llvm_libc::close(fd1), Succeeds(0));
 
   fd2 = __llvm_libc::open(TEST_FILE2, O_RDONLY);
   ASSERT_GT(fd2, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   fd1 = __llvm_libc::open(TEST_FILE1, O_RDONLY);
   ASSERT_LT(fd1, 0);
-  ASSERT_NE(errno, 0);
+  ASSERT_NE(libc_errno, 0);
 
-  errno = 0;
+  libc_errno = 0;
   ASSERT_THAT(__llvm_libc::close(fd2), Succeeds(0));
 
   ASSERT_THAT(__llvm_libc::unlink(TEST_FILE1), Succeeds(0));
diff --git a/libc/test/src/sys/select/select_ui_test.cpp b/libc/test/src/sys/select/select_ui_test.cpp
index d290b18654d27..e10d1aa170960 100644
--- a/libc/test/src/sys/select/select_ui_test.cpp
+++ b/libc/test/src/sys/select/select_ui_test.cpp
@@ -6,11 +6,11 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/sys/select/select.h"
 #include "src/unistd/read.h"
 #include "test/UnitTest/Test.h"
 
-#include <errno.h>
 #include <sys/select.h>
 #include <unistd.h>
 
@@ -18,7 +18,7 @@
 // Instead, one has to run it manually and press a key on the keyboard
 // to make the test succeed.
 TEST(LlvmLibcSelectTest, ReadStdinAfterSelect) {
-  errno = 0;
+  libc_errno = 0;
   constexpr int STDIN_FD = 0;
   fd_set set;
   FD_ZERO(&set);
diff --git a/libc/test/src/sys/sendfile/sendfile_test.cpp b/libc/test/src/sys/sendfile/sendfile_test.cpp
index 95103aac7c103..c77965ea96409 100644
--- a/libc/test/src/sys/sendfile/sendfile_test.cpp
+++ b/libc/test/src/sys/sendfile/sendfile_test.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/CPP/string_view.h"
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/sys/sendfile/sendfile.h"
 #include "src/unistd/close.h"
@@ -17,7 +18,6 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
 #include <fcntl.h>
 #include <sys/stat.h>
 
@@ -36,20 +36,20 @@ TEST(LlvmLibcSendfileTest, CreateAndTransfer) {
   constexpr const char *OUT_FILE = "testdata/sendfile_out.test";
   const char IN_DATA[] = "sendfile test";
   constexpr ssize_t IN_SIZE = ssize_t(sizeof(IN_DATA));
-  errno = 0;
+  libc_errno = 0;
 
   int in_fd = __llvm_libc::open(IN_FILE, O_CREAT | O_WRONLY, S_IRWXU);
   ASSERT_GT(in_fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_EQ(__llvm_libc::write(in_fd, IN_DATA, IN_SIZE), IN_SIZE);
   ASSERT_THAT(__llvm_libc::close(in_fd), Succeeds(0));
 
   in_fd = __llvm_libc::open(IN_FILE, O_RDONLY);
   ASSERT_GT(in_fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   int out_fd = __llvm_libc::open(OUT_FILE, O_CREAT | O_WRONLY, S_IRWXU);
   ASSERT_GT(out_fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ssize_t size = __llvm_libc::sendfile(in_fd, out_fd, nullptr, IN_SIZE);
   ASSERT_EQ(size, IN_SIZE);
   ASSERT_THAT(__llvm_libc::close(in_fd), Succeeds(0));
@@ -57,7 +57,7 @@ TEST(LlvmLibcSendfileTest, CreateAndTransfer) {
 
   out_fd = __llvm_libc::open(OUT_FILE, O_RDONLY);
   ASSERT_GT(out_fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   char buf[IN_SIZE];
   ASSERT_EQ(IN_SIZE, __llvm_libc::read(out_fd, buf, IN_SIZE));
   ASSERT_EQ(cpp::string_view(buf), cpp::string_view(IN_DATA));
diff --git a/libc/test/src/sys/stat/CMakeLists.txt b/libc/test/src/sys/stat/CMakeLists.txt
index ef2027dc29168..f21b7a0323a0f 100644
--- a/libc/test/src/sys/stat/CMakeLists.txt
+++ b/libc/test/src/sys/stat/CMakeLists.txt
@@ -9,9 +9,9 @@ add_libc_unittest(
   SRCS
     chmod_test.cpp
   DEPENDS
-    libc.include.errno
     libc.include.fcntl
     libc.include.sys_stat
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.sys.stat.chmod
     libc.src.unistd.close
@@ -25,9 +25,9 @@ add_libc_unittest(
   SRCS
     fchmodat_test.cpp
   DEPENDS
-    libc.include.errno
     libc.include.fcntl
     libc.include.sys_stat
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.sys.stat.fchmodat
     libc.src.unistd.close
@@ -41,9 +41,9 @@ add_libc_unittest(
   SRCS
     fchmod_test.cpp
   DEPENDS
-    libc.include.errno
     libc.include.fcntl
     libc.include.sys_stat
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.sys.stat.fchmod
     libc.src.unistd.close
@@ -57,9 +57,9 @@ add_libc_unittest(
   SRCS
     mkdirat_test.cpp
   DEPENDS
-    libc.include.errno
     libc.include.fcntl
     libc.include.sys_stat
+    libc.src.errno.errno
     libc.src.sys.stat.mkdirat
     libc.src.unistd.rmdir
 )
@@ -71,9 +71,9 @@ add_libc_unittest(
   SRCS
     stat_test.cpp
   DEPENDS
-    libc.include.errno
     libc.include.fcntl
     libc.include.sys_stat
+    libc.src.errno.errno
     libc.src.sys.stat.stat
     libc.src.fcntl.open
     libc.src.unistd.close
@@ -87,9 +87,9 @@ add_libc_unittest(
   SRCS
     lstat_test.cpp
   DEPENDS
-    libc.include.errno
     libc.include.fcntl
     libc.include.sys_stat
+    libc.src.errno.errno
     libc.src.sys.stat.lstat
     libc.src.fcntl.open
     libc.src.unistd.close
@@ -103,9 +103,9 @@ add_libc_unittest(
   SRCS
     fstat_test.cpp
   DEPENDS
-    libc.include.errno
     libc.include.fcntl
     libc.include.sys_stat
+    libc.src.errno.errno
     libc.src.sys.stat.fstat
     libc.src.fcntl.open
     libc.src.unistd.close
diff --git a/libc/test/src/sys/stat/chmod_test.cpp b/libc/test/src/sys/stat/chmod_test.cpp
index 456b867a2164a..d8da9e390301a 100644
--- a/libc/test/src/sys/stat/chmod_test.cpp
+++ b/libc/test/src/sys/stat/chmod_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/sys/stat/chmod.h"
 #include "src/unistd/close.h"
@@ -14,7 +15,6 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
 #include <fcntl.h>
 #include <sys/stat.h>
 
@@ -29,36 +29,36 @@ TEST(LlvmLibcChmodTest, ChangeAndOpen) {
   constexpr const char *TEST_FILE = "testdata/chmod.test";
   const char WRITE_DATA[] = "test data";
   constexpr ssize_t WRITE_SIZE = ssize_t(sizeof(WRITE_DATA));
-  errno = 0;
+  libc_errno = 0;
 
   int fd = __llvm_libc::open(TEST_FILE, O_APPEND | O_WRONLY);
   ASSERT_GT(fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_EQ(__llvm_libc::write(fd, WRITE_DATA, sizeof(WRITE_DATA)), WRITE_SIZE);
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
 
   fd = __llvm_libc::open(TEST_FILE, O_PATH);
   ASSERT_GT(fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
   EXPECT_THAT(__llvm_libc::chmod(TEST_FILE, S_IRUSR), Succeeds(0));
 
   // Opening for writing should fail.
   EXPECT_EQ(__llvm_libc::open(TEST_FILE, O_APPEND | O_WRONLY), -1);
-  EXPECT_NE(errno, 0);
-  errno = 0;
+  EXPECT_NE(libc_errno, 0);
+  libc_errno = 0;
   // But opening for reading should succeed.
   fd = __llvm_libc::open(TEST_FILE, O_APPEND | O_RDONLY);
   EXPECT_GT(fd, 0);
-  EXPECT_EQ(errno, 0);
+  EXPECT_EQ(libc_errno, 0);
 
   EXPECT_THAT(__llvm_libc::close(fd), Succeeds(0));
   EXPECT_THAT(__llvm_libc::chmod(TEST_FILE, S_IRWXU), Succeeds(0));
 }
 
 TEST(LlvmLibcChmodTest, NonExistentFile) {
-  errno = 0;
+  libc_errno = 0;
   using __llvm_libc::testing::ErrnoSetterMatcher::Fails;
   ASSERT_THAT(__llvm_libc::chmod("non-existent-file", S_IRUSR), Fails(ENOENT));
-  errno = 0;
+  libc_errno = 0;
 }
diff --git a/libc/test/src/sys/stat/fchmod_test.cpp b/libc/test/src/sys/stat/fchmod_test.cpp
index 54d945760865f..c75688bac9d3b 100644
--- a/libc/test/src/sys/stat/fchmod_test.cpp
+++ b/libc/test/src/sys/stat/fchmod_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/sys/stat/fchmod.h"
 #include "src/unistd/close.h"
@@ -14,7 +15,6 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
 #include <fcntl.h>
 #include <sys/stat.h>
 
@@ -29,36 +29,36 @@ TEST(LlvmLibcChmodTest, ChangeAndOpen) {
   constexpr const char *TEST_FILE = "testdata/fchmod.test";
   const char WRITE_DATA[] = "test data";
   constexpr ssize_t WRITE_SIZE = ssize_t(sizeof(WRITE_DATA));
-  errno = 0;
+  libc_errno = 0;
 
   int fd = __llvm_libc::open(TEST_FILE, O_APPEND | O_WRONLY);
   ASSERT_GT(fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_EQ(__llvm_libc::write(fd, WRITE_DATA, sizeof(WRITE_DATA)), WRITE_SIZE);
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
 
   fd = __llvm_libc::open(TEST_FILE, O_APPEND | O_WRONLY);
   ASSERT_GT(fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   EXPECT_THAT(__llvm_libc::fchmod(fd, S_IRUSR), Succeeds(0));
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
 
   // Opening for writing should fail.
   EXPECT_EQ(__llvm_libc::open(TEST_FILE, O_APPEND | O_WRONLY), -1);
-  EXPECT_NE(errno, 0);
-  errno = 0;
+  EXPECT_NE(libc_errno, 0);
+  libc_errno = 0;
   // But opening for reading should succeed.
   fd = __llvm_libc::open(TEST_FILE, O_APPEND | O_RDONLY);
   EXPECT_GT(fd, 0);
-  EXPECT_EQ(errno, 0);
+  EXPECT_EQ(libc_errno, 0);
 
   EXPECT_THAT(__llvm_libc::fchmod(fd, S_IRWXU), Succeeds(0));
   EXPECT_THAT(__llvm_libc::close(fd), Succeeds(0));
 }
 
 TEST(LlvmLibcChmodTest, NonExistentFile) {
-  errno = 0;
+  libc_errno = 0;
   ASSERT_EQ(__llvm_libc::fchmod(-1, S_IRUSR), -1);
-  ASSERT_NE(errno, 0);
-  errno = 0;
+  ASSERT_NE(libc_errno, 0);
+  libc_errno = 0;
 }
diff --git a/libc/test/src/sys/stat/fchmodat_test.cpp b/libc/test/src/sys/stat/fchmodat_test.cpp
index 109a162b8466f..fd2a053d12336 100644
--- a/libc/test/src/sys/stat/fchmodat_test.cpp
+++ b/libc/test/src/sys/stat/fchmodat_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/sys/stat/fchmodat.h"
 #include "src/unistd/close.h"
@@ -14,7 +15,6 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
 #include <fcntl.h>
 #include <sys/stat.h>
 
@@ -31,29 +31,29 @@ TEST(LlvmLibcFchmodatTest, ChangeAndOpen) {
   constexpr const char *TEST_FILE_BASENAME = "fchmodat.test";
   const char WRITE_DATA[] = "fchmodat test";
   constexpr ssize_t WRITE_SIZE = ssize_t(sizeof(WRITE_DATA));
-  errno = 0;
+  libc_errno = 0;
 
   int fd = __llvm_libc::open(TEST_FILE, O_CREAT | O_WRONLY, S_IRWXU);
   ASSERT_GT(fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_EQ(__llvm_libc::write(fd, WRITE_DATA, sizeof(WRITE_DATA)), WRITE_SIZE);
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
 
   int dirfd = __llvm_libc::open(TEST_DIR, O_DIRECTORY);
   ASSERT_GT(dirfd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
 
   EXPECT_THAT(__llvm_libc::fchmodat(dirfd, TEST_FILE_BASENAME, S_IRUSR, 0),
               Succeeds(0));
 
   // Opening for writing should fail.
   EXPECT_EQ(__llvm_libc::open(TEST_FILE, O_APPEND | O_WRONLY), -1);
-  EXPECT_NE(errno, 0);
-  errno = 0;
+  EXPECT_NE(libc_errno, 0);
+  libc_errno = 0;
   // But opening for reading should succeed.
   fd = __llvm_libc::open(TEST_FILE, O_APPEND | O_RDONLY);
   EXPECT_GT(fd, 0);
-  EXPECT_EQ(errno, 0);
+  EXPECT_EQ(libc_errno, 0);
 
   EXPECT_THAT(__llvm_libc::close(fd), Succeeds(0));
   EXPECT_THAT(__llvm_libc::fchmodat(dirfd, TEST_FILE_BASENAME, S_IRWXU, 0),
@@ -63,9 +63,9 @@ TEST(LlvmLibcFchmodatTest, ChangeAndOpen) {
 }
 
 TEST(LlvmLibcFchmodatTest, NonExistentFile) {
-  errno = 0;
+  libc_errno = 0;
   using __llvm_libc::testing::ErrnoSetterMatcher::Fails;
   ASSERT_THAT(__llvm_libc::fchmodat(AT_FDCWD, "non-existent-file", S_IRUSR, 0),
               Fails(ENOENT));
-  errno = 0;
+  libc_errno = 0;
 }
diff --git a/libc/test/src/sys/stat/fstat_test.cpp b/libc/test/src/sys/stat/fstat_test.cpp
index 6db0d653fb607..66d9d3c83f908 100644
--- a/libc/test/src/sys/stat/fstat_test.cpp
+++ b/libc/test/src/sys/stat/fstat_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/sys/stat/fstat.h"
 #include "src/unistd/close.h"
@@ -14,7 +15,6 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
 #include <fcntl.h>
 #include <sys/stat.h>
 
@@ -27,11 +27,11 @@ TEST(LlvmLibcFStatTest, CreatAndReadMode) {
   // make it readonly using chmod. We test that chmod actually succeeded by
   // trying to open the file for writing and failing.
   constexpr const char *TEST_FILE = "testdata/fstat.test";
-  errno = 0;
+  libc_errno = 0;
 
   int fd = __llvm_libc::open(TEST_FILE, O_CREAT | O_WRONLY, S_IRWXU);
   ASSERT_GT(fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
 
   struct stat statbuf;
   ASSERT_THAT(__llvm_libc::fstat(fd, &statbuf), Succeeds(0));
@@ -43,9 +43,9 @@ TEST(LlvmLibcFStatTest, CreatAndReadMode) {
 }
 
 TEST(LlvmLibcFStatTest, NonExistentFile) {
-  errno = 0;
+  libc_errno = 0;
   using __llvm_libc::testing::ErrnoSetterMatcher::Fails;
   struct stat statbuf;
   ASSERT_THAT(__llvm_libc::fstat(-1, &statbuf), Fails(EBADF));
-  errno = 0;
+  libc_errno = 0;
 }
diff --git a/libc/test/src/sys/stat/lstat_test.cpp b/libc/test/src/sys/stat/lstat_test.cpp
index b5a06c82d5fc7..6cedb2fc5fcf8 100644
--- a/libc/test/src/sys/stat/lstat_test.cpp
+++ b/libc/test/src/sys/stat/lstat_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/sys/stat/lstat.h"
 #include "src/unistd/close.h"
@@ -14,7 +15,6 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
 #include <fcntl.h>
 #include <sys/stat.h>
 
@@ -27,11 +27,11 @@ TEST(LlvmLibcLStatTest, CreatAndReadMode) {
   // make it readonly using chmod. We test that chmod actually succeeded by
   // trying to open the file for writing and failing.
   constexpr const char *TEST_FILE = "testdata/lstat.test";
-  errno = 0;
+  libc_errno = 0;
 
   int fd = __llvm_libc::open(TEST_FILE, O_CREAT | O_WRONLY, S_IRWXU);
   ASSERT_GT(fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
 
   struct stat statbuf;
@@ -43,9 +43,9 @@ TEST(LlvmLibcLStatTest, CreatAndReadMode) {
 }
 
 TEST(LlvmLibcLStatTest, NonExistentFile) {
-  errno = 0;
+  libc_errno = 0;
   using __llvm_libc::testing::ErrnoSetterMatcher::Fails;
   struct stat statbuf;
   ASSERT_THAT(__llvm_libc::lstat("non-existent-file", &statbuf), Fails(ENOENT));
-  errno = 0;
+  libc_errno = 0;
 }
diff --git a/libc/test/src/sys/stat/mkdirat_test.cpp b/libc/test/src/sys/stat/mkdirat_test.cpp
index 2e742009a0b19..7f302a41b54b2 100644
--- a/libc/test/src/sys/stat/mkdirat_test.cpp
+++ b/libc/test/src/sys/stat/mkdirat_test.cpp
@@ -12,7 +12,6 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
 #include <fcntl.h>
 
 TEST(LlvmLibcMkdiratTest, CreateAndRemove) {
diff --git a/libc/test/src/sys/stat/stat_test.cpp b/libc/test/src/sys/stat/stat_test.cpp
index ba6e302465627..0a3eb3e070a15 100644
--- a/libc/test/src/sys/stat/stat_test.cpp
+++ b/libc/test/src/sys/stat/stat_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/sys/stat/stat.h"
 #include "src/unistd/close.h"
@@ -14,7 +15,6 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
 #include <fcntl.h>
 #include <sys/stat.h>
 
@@ -27,11 +27,11 @@ TEST(LlvmLibcStatTest, CreatAndReadMode) {
   // make it readonly using chmod. We test that chmod actually succeeded by
   // trying to open the file for writing and failing.
   constexpr const char *TEST_FILE = "testdata/stat.test";
-  errno = 0;
+  libc_errno = 0;
 
   int fd = __llvm_libc::open(TEST_FILE, O_CREAT | O_WRONLY, S_IRWXU);
   ASSERT_GT(fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
 
   struct stat statbuf;
@@ -43,9 +43,9 @@ TEST(LlvmLibcStatTest, CreatAndReadMode) {
 }
 
 TEST(LlvmLibcStatTest, NonExistentFile) {
-  errno = 0;
+  libc_errno = 0;
   using __llvm_libc::testing::ErrnoSetterMatcher::Fails;
   struct stat statbuf;
   ASSERT_THAT(__llvm_libc::stat("non-existent-file", &statbuf), Fails(ENOENT));
-  errno = 0;
+  libc_errno = 0;
 }
diff --git a/libc/test/src/unistd/CMakeLists.txt b/libc/test/src/unistd/CMakeLists.txt
index b278c35eaa6cf..ea19aba0943ea 100644
--- a/libc/test/src/unistd/CMakeLists.txt
+++ b/libc/test/src/unistd/CMakeLists.txt
@@ -9,8 +9,8 @@ add_libc_unittest(
   SRCS
     access_test.cpp
   DEPENDS
-    libc.include.errno
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.unistd.access
     libc.src.unistd.close
@@ -24,8 +24,8 @@ add_libc_unittest(
   SRCS
     chdir_test.cpp
   DEPENDS
-    libc.include.errno
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.unistd.chdir
     libc.src.unistd.close
@@ -41,6 +41,7 @@ add_libc_unittest(
   DEPENDS
     libc.include.errno
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.unistd.close
     libc.src.unistd.dup
@@ -59,6 +60,7 @@ add_libc_unittest(
   DEPENDS
     libc.include.errno
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.unistd.close
     libc.src.unistd.dup2
@@ -77,6 +79,7 @@ add_libc_unittest(
   DEPENDS
     libc.include.errno
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.unistd.close
     libc.src.unistd.dup3
@@ -95,6 +98,7 @@ add_libc_unittest(
   DEPENDS
     libc.include.errno
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.unistd.fchdir
     libc.src.unistd.close
@@ -110,6 +114,7 @@ add_libc_unittest(
   DEPENDS
     libc.include.errno
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.unistd.close
     libc.src.unistd.read
@@ -128,6 +133,7 @@ add_libc_unittest(
   DEPENDS
     libc.include.errno
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.unistd.close
     libc.src.unistd.fsync
@@ -147,6 +153,7 @@ add_libc_unittest(
   DEPENDS
     libc.include.errno
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.unistd.close
     libc.src.unistd.fsync
@@ -164,6 +171,7 @@ add_libc_unittest(
   DEPENDS
     libc.include.errno
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.unistd.close
     libc.src.unistd.link
@@ -179,6 +187,7 @@ add_libc_unittest(
   DEPENDS
     libc.include.errno
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.unistd.close
     libc.src.unistd.linkat
@@ -194,6 +203,7 @@ add_libc_unittest(
   DEPENDS
     libc.include.errno
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.unistd.close
     libc.src.unistd.lseek
@@ -210,6 +220,7 @@ add_libc_unittest(
   DEPENDS
     libc.include.errno
     libc.include.fcntl
+    libc.src.errno.errno
     libc.src.sys.stat.mkdir
     libc.src.unistd.rmdir
 )
@@ -223,6 +234,7 @@ add_libc_unittest(
   DEPENDS
     libc.include.errno
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.unistd.readlink
     libc.src.unistd.symlink
     libc.src.unistd.unlink
@@ -239,6 +251,7 @@ add_libc_unittest(
     libc.include.errno
     libc.include.fcntl
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.unistd.readlinkat
     libc.src.unistd.symlink
     libc.src.unistd.unlink
@@ -254,6 +267,7 @@ add_libc_unittest(
   DEPENDS
     libc.include.errno
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.unistd.close
     libc.src.unistd.symlink
@@ -269,6 +283,7 @@ add_libc_unittest(
   DEPENDS
     libc.include.errno
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.unistd.close
     libc.src.unistd.symlinkat
@@ -284,6 +299,7 @@ add_libc_unittest(
   DEPENDS
     libc.include.errno
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.unistd.close
     libc.src.unistd.read
@@ -302,6 +318,7 @@ add_libc_unittest(
   DEPENDS
     libc.include.errno
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.unistd.close
     libc.src.unistd.unlink
@@ -316,6 +333,7 @@ add_libc_unittest(
   DEPENDS
     libc.include.errno
     libc.include.unistd
+    libc.src.errno.errno
     libc.src.fcntl.open
     libc.src.fcntl.openat
     libc.src.unistd.close
@@ -362,7 +380,6 @@ add_libc_unittest(
     libc.src.unistd.isatty
     libc.src.fcntl.open
     libc.src.unistd.close
-    libc.include.errno
     libc.src.errno.errno
 )
 
diff --git a/libc/test/src/unistd/access_test.cpp b/libc/test/src/unistd/access_test.cpp
index 30c3cb4572c19..92e7685567d84 100644
--- a/libc/test/src/unistd/access_test.cpp
+++ b/libc/test/src/unistd/access_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/unistd/access.h"
 #include "src/unistd/close.h"
@@ -14,40 +15,39 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
 #include <unistd.h>
 
 TEST(LlvmLibcAccessTest, CreateAndTest) {
   // The test strategy is to repeatedly create a file in different modes and
   // test that it is accessable in those modes but not in others.
-  errno = 0;
+  libc_errno = 0;
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
   constexpr const char *TEST_FILE = "testdata/access.test";
   int fd = __llvm_libc::open(TEST_FILE, O_WRONLY | O_CREAT, S_IRWXU);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
 
   ASSERT_EQ(__llvm_libc::access(TEST_FILE, F_OK), 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_EQ(__llvm_libc::access(TEST_FILE, X_OK | W_OK | R_OK), 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_THAT(__llvm_libc::unlink(TEST_FILE), Succeeds(0));
 
   fd = __llvm_libc::open(TEST_FILE, O_WRONLY | O_CREAT, S_IXUSR);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
   ASSERT_EQ(__llvm_libc::access(TEST_FILE, F_OK), 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_EQ(__llvm_libc::access(TEST_FILE, X_OK), 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_EQ(__llvm_libc::access(TEST_FILE, R_OK), -1);
-  ASSERT_EQ(errno, EACCES);
-  errno = 0;
+  ASSERT_EQ(libc_errno, EACCES);
+  libc_errno = 0;
   ASSERT_EQ(__llvm_libc::access(TEST_FILE, W_OK), -1);
-  ASSERT_EQ(errno, EACCES);
-  errno = 0;
+  ASSERT_EQ(libc_errno, EACCES);
+  libc_errno = 0;
   ASSERT_THAT(__llvm_libc::unlink(TEST_FILE), Succeeds(0));
 }
 
diff --git a/libc/test/src/unistd/chdir_test.cpp b/libc/test/src/unistd/chdir_test.cpp
index 998d42aeddfda..05e3e5dcbca28 100644
--- a/libc/test/src/unistd/chdir_test.cpp
+++ b/libc/test/src/unistd/chdir_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/unistd/chdir.h"
 #include "src/unistd/close.h"
@@ -13,7 +14,6 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
 #include <fcntl.h>
 
 TEST(LlvmLibcChdirTest, ChangeAndOpen) {
@@ -25,23 +25,23 @@ TEST(LlvmLibcChdirTest, ChangeAndOpen) {
   constexpr const char *TEST_DIR = "testdata";
   constexpr const char *TEST_FILE = "testdata/chdir.test";
   constexpr const char *TEST_FILE_BASE = "chdir.test";
-  errno = 0;
+  libc_errno = 0;
 
   int fd = __llvm_libc::open(TEST_FILE, O_PATH);
   ASSERT_GT(fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
 
   ASSERT_THAT(__llvm_libc::chdir(TEST_DIR), Succeeds(0));
   fd = __llvm_libc::open(TEST_FILE_BASE, O_PATH);
   ASSERT_GT(fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
 }
 
 TEST(LlvmLibcChdirTest, ChangeToNonExistentDir) {
-  errno = 0;
+  libc_errno = 0;
   using __llvm_libc::testing::ErrnoSetterMatcher::Fails;
   ASSERT_THAT(__llvm_libc::chdir("non-existent-dir"), Fails(ENOENT));
-  errno = 0;
+  libc_errno = 0;
 }
diff --git a/libc/test/src/unistd/dup2_test.cpp b/libc/test/src/unistd/dup2_test.cpp
index b595a66a5a109..545ea2eff80be 100644
--- a/libc/test/src/unistd/dup2_test.cpp
+++ b/libc/test/src/unistd/dup2_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/unistd/close.h"
 #include "src/unistd/dup2.h"
@@ -16,18 +17,16 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
-
 TEST(LlvmLibcdupTest, ReadAndWriteViaDup) {
   constexpr int DUPFD = 0xD0;
-  errno = 0;
+  libc_errno = 0;
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
   constexpr const char *TEST_FILE = "testdata/dup2.test";
   int fd = __llvm_libc::open(TEST_FILE, O_WRONLY | O_CREAT, S_IRWXU);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   int dupfd = __llvm_libc::dup2(fd, DUPFD);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_EQ(dupfd, DUPFD);
 
   // Write something via the dup
@@ -39,10 +38,10 @@ TEST(LlvmLibcdupTest, ReadAndWriteViaDup) {
 
   // Reopen the file for reading and create a dup.
   fd = __llvm_libc::open(TEST_FILE, O_RDONLY);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   dupfd = __llvm_libc::dup2(fd, DUPFD);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_EQ(dupfd, DUPFD);
 
   // Read the file content via the dup.
diff --git a/libc/test/src/unistd/dup3_test.cpp b/libc/test/src/unistd/dup3_test.cpp
index 3d4b2e50cbff3..575e1291b9a27 100644
--- a/libc/test/src/unistd/dup3_test.cpp
+++ b/libc/test/src/unistd/dup3_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/unistd/close.h"
 #include "src/unistd/dup3.h"
@@ -16,8 +17,6 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
-
 // The tests here are exactly the same as those of dup2. We only test the
 // plumbing of the dup3 syscall and not the dup3 functionality itself as it is
 // a simple syscall wrapper. Testing dup3 functionality is beyond the scope of
@@ -25,15 +24,15 @@
 
 TEST(LlvmLibcdupTest, ReadAndWriteViaDup) {
   constexpr int DUPFD = 0xD0;
-  errno = 0;
+  libc_errno = 0;
   using __llvm_libc::testing::ErrnoSetterMatcher::Fails;
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
   constexpr const char *TEST_FILE = "testdata/dup3.test";
   int fd = __llvm_libc::open(TEST_FILE, O_WRONLY | O_CREAT, S_IRWXU);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   int dupfd = __llvm_libc::dup3(fd, DUPFD, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_EQ(dupfd, DUPFD);
 
   // Write something via the dup
@@ -45,10 +44,10 @@ TEST(LlvmLibcdupTest, ReadAndWriteViaDup) {
 
   // Reopen the file for reading and create a dup.
   fd = __llvm_libc::open(TEST_FILE, O_RDONLY);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   dupfd = __llvm_libc::dup3(fd, DUPFD, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_EQ(dupfd, DUPFD);
 
   // Read the file content via the dup.
diff --git a/libc/test/src/unistd/dup_test.cpp b/libc/test/src/unistd/dup_test.cpp
index 414176261f437..ff877727b0470 100644
--- a/libc/test/src/unistd/dup_test.cpp
+++ b/libc/test/src/unistd/dup_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/unistd/close.h"
 #include "src/unistd/dup.h"
@@ -16,17 +17,15 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
-
 TEST(LlvmLibcdupTest, ReadAndWriteViaDup) {
-  errno = 0;
+  libc_errno = 0;
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
   constexpr const char *TEST_FILE = "testdata/dup.test";
   int fd = __llvm_libc::open(TEST_FILE, O_WRONLY | O_CREAT, S_IRWXU);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   int dupfd = __llvm_libc::dup(fd);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(dupfd, 0);
 
   // Write something via the dup
@@ -38,10 +37,10 @@ TEST(LlvmLibcdupTest, ReadAndWriteViaDup) {
 
   // Reopen the file for reading and create a dup.
   fd = __llvm_libc::open(TEST_FILE, O_RDONLY);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   dupfd = __llvm_libc::dup(fd);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(dupfd, 0);
 
   // Read the file content via the dup.
diff --git a/libc/test/src/unistd/fchdir_test.cpp b/libc/test/src/unistd/fchdir_test.cpp
index b331f4e193794..35d3cd3b76eab 100644
--- a/libc/test/src/unistd/fchdir_test.cpp
+++ b/libc/test/src/unistd/fchdir_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/unistd/close.h"
 #include "src/unistd/fchdir.h"
@@ -13,7 +14,6 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
 #include <fcntl.h>
 
 TEST(LlvmLibcChdirTest, ChangeAndOpen) {
@@ -25,28 +25,28 @@ TEST(LlvmLibcChdirTest, ChangeAndOpen) {
   constexpr const char *TEST_DIR = "testdata";
   constexpr const char *TEST_FILE = "testdata/fchdir.test";
   constexpr const char *TEST_FILE_BASE = "fchdir.test";
-  errno = 0;
+  libc_errno = 0;
 
   int dir_fd = __llvm_libc::open(TEST_DIR, O_DIRECTORY);
   ASSERT_GT(dir_fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   int file_fd = __llvm_libc::open(TEST_FILE, O_PATH);
   ASSERT_GT(file_fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_THAT(__llvm_libc::close(file_fd), Succeeds(0));
 
   ASSERT_THAT(__llvm_libc::fchdir(dir_fd), Succeeds(0));
   file_fd = __llvm_libc::open(TEST_FILE_BASE, O_PATH);
   ASSERT_GT(file_fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_THAT(__llvm_libc::close(file_fd), Succeeds(0));
   ASSERT_THAT(__llvm_libc::close(dir_fd), Succeeds(0));
 }
 
 TEST(LlvmLibcChdirTest, ChangeToNonExistentDir) {
   using __llvm_libc::testing::ErrnoSetterMatcher::Fails;
-  errno = 0;
+  libc_errno = 0;
   ASSERT_EQ(__llvm_libc::fchdir(0), -1);
-  ASSERT_NE(errno, 0);
-  errno = 0;
+  ASSERT_NE(libc_errno, 0);
+  libc_errno = 0;
 }
diff --git a/libc/test/src/unistd/ftruncate_test.cpp b/libc/test/src/unistd/ftruncate_test.cpp
index 77d3bd7931bf7..0814ef3862618 100644
--- a/libc/test/src/unistd/ftruncate_test.cpp
+++ b/libc/test/src/unistd/ftruncate_test.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/CPP/string_view.h"
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/unistd/close.h"
 #include "src/unistd/ftruncate.h"
@@ -16,8 +17,6 @@
 #include "test/ErrnoSetterMatcher.h"
 #include "test/UnitTest/Test.h"
 
-#include <errno.h>
-
 namespace cpp = __llvm_libc::cpp;
 
 TEST(LlvmLibcFtruncateTest, CreateAndTruncate) {
@@ -32,16 +31,16 @@ TEST(LlvmLibcFtruncateTest, CreateAndTruncate) {
   //   2. Read it to make sure what was written is actually in the file.
   //   3. Truncate to 1 byte.
   //   4. Try to read more than 1 byte and fail.
-  errno = 0;
+  libc_errno = 0;
   int fd = __llvm_libc::open(TEST_FILE, O_WRONLY | O_CREAT, S_IRWXU);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   ASSERT_EQ(ssize_t(WRITE_SIZE),
             __llvm_libc::write(fd, WRITE_DATA, WRITE_SIZE));
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
 
   fd = __llvm_libc::open(TEST_FILE, O_RDONLY);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   ASSERT_EQ(ssize_t(WRITE_SIZE), __llvm_libc::read(fd, buf, WRITE_SIZE));
   ASSERT_EQ(cpp::string_view(buf), cpp::string_view(WRITE_DATA));
@@ -51,12 +50,12 @@ TEST(LlvmLibcFtruncateTest, CreateAndTruncate) {
   // writing.
   fd = __llvm_libc::open(TEST_FILE, O_WRONLY);
   ASSERT_GT(fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_THAT(__llvm_libc::ftruncate(fd, off_t(1)), Succeeds(0));
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
 
   fd = __llvm_libc::open(TEST_FILE, O_RDONLY);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   ASSERT_EQ(ssize_t(1), __llvm_libc::read(fd, buf, WRITE_SIZE));
   ASSERT_EQ(buf[0], WRITE_DATA[0]);
diff --git a/libc/test/src/unistd/isatty_test.cpp b/libc/test/src/unistd/isatty_test.cpp
index 0d949eec88f8c..c910e360192d4 100644
--- a/libc/test/src/unistd/isatty_test.cpp
+++ b/libc/test/src/unistd/isatty_test.cpp
@@ -6,21 +6,20 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/unistd/close.h"
 #include "src/unistd/isatty.h"
 #include "test/ErrnoSetterMatcher.h"
 #include "test/UnitTest/Test.h"
 
-#include <errno.h>
-
 using __llvm_libc::testing::ErrnoSetterMatcher::Fails;
 using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
 
 TEST(LlvmLibcIsATTYTest, StdInOutTests) {
   // If stdin is connected to a terminal, assume that all of the standard i/o
   // fds are.
-  errno = 0;
+  libc_errno = 0;
   if (__llvm_libc::isatty(0)) {
     EXPECT_THAT(__llvm_libc::isatty(0), Succeeds(1)); // stdin
     EXPECT_THAT(__llvm_libc::isatty(1), Succeeds(1)); // stdout
@@ -33,16 +32,16 @@ TEST(LlvmLibcIsATTYTest, StdInOutTests) {
 }
 
 TEST(LlvmLibcIsATTYTest, BadFdTest) {
-  errno = 0;
+  libc_errno = 0;
   EXPECT_THAT(__llvm_libc::isatty(-1), Fails(EBADF, 0)); // invalid fd
 }
 
 TEST(LlvmLibcIsATTYTest, DevTTYTest) {
   constexpr const char *TTY_FILE = "/dev/tty";
-  errno = 0;
+  libc_errno = 0;
   int fd = __llvm_libc::open(TTY_FILE, O_RDONLY);
   if (fd > 0) {
-    ASSERT_EQ(errno, 0);
+    ASSERT_EQ(libc_errno, 0);
     EXPECT_THAT(__llvm_libc::isatty(fd), Succeeds(1));
     ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
   }
@@ -50,9 +49,9 @@ TEST(LlvmLibcIsATTYTest, DevTTYTest) {
 
 TEST(LlvmLibcIsATTYTest, FileTest) {
   constexpr const char *TEST_FILE = "testdata/isatty.test";
-  errno = 0;
+  libc_errno = 0;
   int fd = __llvm_libc::open(TEST_FILE, O_WRONLY | O_CREAT, S_IRWXU);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   EXPECT_THAT(__llvm_libc::isatty(fd), Fails(ENOTTY, 0));
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
diff --git a/libc/test/src/unistd/link_test.cpp b/libc/test/src/unistd/link_test.cpp
index b42336f171fe2..ef98559877e0f 100644
--- a/libc/test/src/unistd/link_test.cpp
+++ b/libc/test/src/unistd/link_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/unistd/close.h"
 #include "src/unistd/link.h"
@@ -13,8 +14,6 @@
 #include "test/ErrnoSetterMatcher.h"
 #include "test/UnitTest/Test.h"
 
-#include <errno.h>
-
 TEST(LlvmLibcLinkTest, CreateAndUnlink) {
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
   constexpr const char *TEST_FILE = "testdata/link.test";
@@ -25,16 +24,16 @@ TEST(LlvmLibcLinkTest, CreateAndUnlink) {
   //   2. Create a link to that file.
   //   3. Open the link to check that the link was created.
   //   4. Cleanup the file and its link.
-  errno = 0;
+  libc_errno = 0;
   int write_fd = __llvm_libc::open(TEST_FILE, O_WRONLY | O_CREAT, S_IRWXU);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(write_fd, 0);
   ASSERT_THAT(__llvm_libc::close(write_fd), Succeeds(0));
   ASSERT_THAT(__llvm_libc::link(TEST_FILE, TEST_FILE_LINK), Succeeds(0));
 
   int link_fd = __llvm_libc::open(TEST_FILE_LINK, O_PATH);
   ASSERT_GT(link_fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_THAT(__llvm_libc::close(link_fd), Succeeds(0));
 
   ASSERT_THAT(__llvm_libc::unlink(TEST_FILE), Succeeds(0));
diff --git a/libc/test/src/unistd/linkat_test.cpp b/libc/test/src/unistd/linkat_test.cpp
index 5a61b4e9301e8..e61c7c17a80f2 100644
--- a/libc/test/src/unistd/linkat_test.cpp
+++ b/libc/test/src/unistd/linkat_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/unistd/close.h"
 #include "src/unistd/linkat.h"
@@ -13,8 +14,6 @@
 #include "test/ErrnoSetterMatcher.h"
 #include "test/UnitTest/Test.h"
 
-#include <errno.h>
-
 TEST(LlvmLibcLinkatTest, CreateAndUnlink) {
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
   constexpr const char *TEST_DIR = "testdata";
@@ -28,9 +27,9 @@ TEST(LlvmLibcLinkatTest, CreateAndUnlink) {
   //   2. Create a link to that file.
   //   3. Open the link to check that the link was created.
   //   4. Cleanup the file and its link.
-  errno = 0;
+  libc_errno = 0;
   int write_fd = __llvm_libc::open(TEST_FILE_PATH, O_WRONLY | O_CREAT, S_IRWXU);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(write_fd, 0);
   ASSERT_THAT(__llvm_libc::close(write_fd), Succeeds(0));
 
@@ -40,7 +39,7 @@ TEST(LlvmLibcLinkatTest, CreateAndUnlink) {
 
   int link_fd = __llvm_libc::open(TEST_FILE_LINK_PATH, O_PATH);
   ASSERT_GT(link_fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_THAT(__llvm_libc::close(link_fd), Succeeds(0));
 
   ASSERT_THAT(__llvm_libc::unlink(TEST_FILE_PATH), Succeeds(0));
diff --git a/libc/test/src/unistd/lseek_test.cpp b/libc/test/src/unistd/lseek_test.cpp
index 065753fe427db..f235f7fcb12eb 100644
--- a/libc/test/src/unistd/lseek_test.cpp
+++ b/libc/test/src/unistd/lseek_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/unistd/close.h"
 #include "src/unistd/lseek.h"
@@ -14,14 +15,13 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
 #include <unistd.h>
 
 TEST(LlvmLibcUniStd, LseekTest) {
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
   constexpr const char *TEST_FILE = "testdata/lseek.test";
   int fd = __llvm_libc::open(TEST_FILE, O_RDONLY);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   constexpr const char LSEEK_TEST[] = "lseek test";
   constexpr int LSEEK_TEST_SIZE = sizeof(LSEEK_TEST) - 1;
@@ -54,7 +54,7 @@ TEST(LlvmLibcUniStd, LseekFailsTest) {
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
   constexpr const char *TEST_FILE = "testdata/lseek.test";
   int fd = __llvm_libc::open(TEST_FILE, O_RDONLY);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   EXPECT_THAT(__llvm_libc::lseek(fd, -1, SEEK_CUR), Fails(EINVAL));
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
diff --git a/libc/test/src/unistd/pread_pwrite_test.cpp b/libc/test/src/unistd/pread_pwrite_test.cpp
index cfc5e37d43c7c..4cfbeaf6c7936 100644
--- a/libc/test/src/unistd/pread_pwrite_test.cpp
+++ b/libc/test/src/unistd/pread_pwrite_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/unistd/close.h"
 #include "src/unistd/fsync.h"
@@ -17,8 +18,6 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
-
 TEST(LlvmLibcUniStd, PWriteAndPReadBackTest) {
   // The strategy here is that we first create a file and write to it. Next,
   // we open that file again and write at an offset. Finally, we open the
@@ -34,14 +33,14 @@ TEST(LlvmLibcUniStd, PWriteAndPReadBackTest) {
 
   constexpr const char *TEST_FILE = "testdata/pread_pwrite.test";
   int fd = __llvm_libc::open(TEST_FILE, O_WRONLY | O_CREAT, S_IRWXU);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   ASSERT_THAT(__llvm_libc::write(fd, HELLO, HELLO_SIZE), Succeeds(HELLO_SIZE));
   ASSERT_THAT(__llvm_libc::fsync(fd), Succeeds(0));
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
 
   fd = __llvm_libc::open(TEST_FILE, O_WRONLY);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   ASSERT_THAT(__llvm_libc::pwrite(fd, HELLO, HELLO_SIZE, OFFSET),
               Succeeds(HELLO_SIZE));
@@ -49,7 +48,7 @@ TEST(LlvmLibcUniStd, PWriteAndPReadBackTest) {
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
 
   fd = __llvm_libc::open(TEST_FILE, O_RDONLY);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   char read_buf[OFFSET_TEXT_SIZE];
   ASSERT_THAT(__llvm_libc::pread(fd, read_buf, HELLO_SIZE, OFFSET),
diff --git a/libc/test/src/unistd/read_write_test.cpp b/libc/test/src/unistd/read_write_test.cpp
index aa402e2977805..68299f639878a 100644
--- a/libc/test/src/unistd/read_write_test.cpp
+++ b/libc/test/src/unistd/read_write_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/unistd/close.h"
 #include "src/unistd/fsync.h"
@@ -15,13 +16,11 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
-
 TEST(LlvmLibcUniStd, WriteAndReadBackTest) {
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
   constexpr const char *TEST_FILE = "__unistd_read_write.test";
   int write_fd = __llvm_libc::open(TEST_FILE, O_WRONLY | O_CREAT, S_IRWXU);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(write_fd, 0);
   constexpr const char HELLO[] = "hello";
   constexpr int HELLO_SIZE = sizeof(HELLO);
@@ -31,7 +30,7 @@ TEST(LlvmLibcUniStd, WriteAndReadBackTest) {
   ASSERT_THAT(__llvm_libc::close(write_fd), Succeeds(0));
 
   int read_fd = __llvm_libc::open(TEST_FILE, O_RDONLY);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(read_fd, 0);
   char read_buf[10];
   ASSERT_THAT(__llvm_libc::read(read_fd, read_buf, HELLO_SIZE),
diff --git a/libc/test/src/unistd/readlink_test.cpp b/libc/test/src/unistd/readlink_test.cpp
index df02a05429a26..ab9223fffcd80 100644
--- a/libc/test/src/unistd/readlink_test.cpp
+++ b/libc/test/src/unistd/readlink_test.cpp
@@ -7,21 +7,20 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/CPP/string_view.h"
+#include "src/errno/libc_errno.h"
 #include "src/unistd/readlink.h"
 #include "src/unistd/symlink.h"
 #include "src/unistd/unlink.h"
 #include "test/ErrnoSetterMatcher.h"
 #include "test/UnitTest/Test.h"
 
-#include <errno.h>
-
 namespace cpp = __llvm_libc::cpp;
 
 TEST(LlvmLibcReadlinkTest, CreateAndUnlink) {
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
   constexpr const char LINK_VAL[] = "readlink_test_value";
   constexpr const char LINK[] = "testdata/readlink.test.link";
-  errno = 0;
+  libc_errno = 0;
 
   // The test strategy is as follows:
   //   1. Create a symlink with value LINK_VAL.
@@ -31,7 +30,7 @@ TEST(LlvmLibcReadlinkTest, CreateAndUnlink) {
 
   char buf[sizeof(LINK_VAL)];
   ssize_t len = __llvm_libc::readlink(LINK, buf, sizeof(buf));
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_EQ(cpp::string_view(buf, len), cpp::string_view(LINK_VAL));
 
   ASSERT_THAT(__llvm_libc::unlink(LINK), Succeeds(0));
diff --git a/libc/test/src/unistd/readlinkat_test.cpp b/libc/test/src/unistd/readlinkat_test.cpp
index e92ded5b7a479..eb23f2ee85b6f 100644
--- a/libc/test/src/unistd/readlinkat_test.cpp
+++ b/libc/test/src/unistd/readlinkat_test.cpp
@@ -7,13 +7,13 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/CPP/string_view.h"
+#include "src/errno/libc_errno.h"
 #include "src/unistd/readlinkat.h"
 #include "src/unistd/symlink.h"
 #include "src/unistd/unlink.h"
 #include "test/ErrnoSetterMatcher.h"
 #include "test/UnitTest/Test.h"
 
-#include <errno.h>
 #include <fcntl.h>
 
 namespace cpp = __llvm_libc::cpp;
@@ -22,7 +22,7 @@ TEST(LlvmLibcReadlinkatTest, CreateAndUnlink) {
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
   constexpr const char LINK_VAL[] = "readlinkat_test_value";
   constexpr const char LINK[] = "testdata/readlinkat.test.link";
-  errno = 0;
+  libc_errno = 0;
 
   // The test strategy is as follows:
   //   1. Create a symlink with value LINK_VAL.
@@ -32,7 +32,7 @@ TEST(LlvmLibcReadlinkatTest, CreateAndUnlink) {
 
   char buf[sizeof(LINK_VAL)];
   ssize_t len = __llvm_libc::readlinkat(AT_FDCWD, LINK, buf, sizeof(buf));
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_EQ(cpp::string_view(buf, len), cpp::string_view(LINK_VAL));
 
   ASSERT_THAT(__llvm_libc::unlink(LINK), Succeeds(0));
diff --git a/libc/test/src/unistd/rmdir_test.cpp b/libc/test/src/unistd/rmdir_test.cpp
index 1580166c90a4f..a5eb8d3a974c5 100644
--- a/libc/test/src/unistd/rmdir_test.cpp
+++ b/libc/test/src/unistd/rmdir_test.cpp
@@ -6,13 +6,13 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/sys/stat/mkdir.h"
 #include "src/unistd/rmdir.h"
 #include "test/ErrnoSetterMatcher.h"
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
 #include <fcntl.h>
 
 TEST(LlvmLibcRmdirTest, CreateAndRemove) {
diff --git a/libc/test/src/unistd/symlink_test.cpp b/libc/test/src/unistd/symlink_test.cpp
index 489ecfd5dda9e..44821e720b875 100644
--- a/libc/test/src/unistd/symlink_test.cpp
+++ b/libc/test/src/unistd/symlink_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/unistd/close.h"
 #include "src/unistd/symlink.h"
@@ -13,8 +14,6 @@
 #include "test/ErrnoSetterMatcher.h"
 #include "test/UnitTest/Test.h"
 
-#include <errno.h>
-
 TEST(LlvmLibcSymlinkTest, CreateAndUnlink) {
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
   constexpr const char *TEST_FILE_BASE = "symlink.test";
@@ -26,9 +25,9 @@ TEST(LlvmLibcSymlinkTest, CreateAndUnlink) {
   //   2. Create a symlink to that file.
   //   3. Open the symlink to check that the symlink was created.
   //   4. Cleanup the file and its symlink.
-  errno = 0;
+  libc_errno = 0;
   int write_fd = __llvm_libc::open(TEST_FILE, O_WRONLY | O_CREAT, S_IRWXU);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(write_fd, 0);
   ASSERT_THAT(__llvm_libc::close(write_fd), Succeeds(0));
 
@@ -37,7 +36,7 @@ TEST(LlvmLibcSymlinkTest, CreateAndUnlink) {
 
   int symlink_fd = __llvm_libc::open(TEST_FILE_LINK, O_PATH);
   ASSERT_GT(symlink_fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_THAT(__llvm_libc::close(symlink_fd), Succeeds(0));
 
   ASSERT_THAT(__llvm_libc::unlink(TEST_FILE), Succeeds(0));
diff --git a/libc/test/src/unistd/symlinkat_test.cpp b/libc/test/src/unistd/symlinkat_test.cpp
index 9f4658d275c0c..847bbbf5d5301 100644
--- a/libc/test/src/unistd/symlinkat_test.cpp
+++ b/libc/test/src/unistd/symlinkat_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/unistd/close.h"
 #include "src/unistd/symlinkat.h"
@@ -13,8 +14,6 @@
 #include "test/ErrnoSetterMatcher.h"
 #include "test/UnitTest/Test.h"
 
-#include <errno.h>
-
 TEST(LlvmLibcSymlinkatTest, CreateAndUnlink) {
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
   constexpr const char *TEST_DIR = "testdata";
@@ -28,9 +27,9 @@ TEST(LlvmLibcSymlinkatTest, CreateAndUnlink) {
   //   2. Create a link to that file.
   //   3. Open the link to check that the link was created.
   //   4. Cleanup the file and its link.
-  errno = 0;
+  libc_errno = 0;
   int write_fd = __llvm_libc::open(TEST_FILE_PATH, O_WRONLY | O_CREAT, S_IRWXU);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(write_fd, 0);
   ASSERT_THAT(__llvm_libc::close(write_fd), Succeeds(0));
 
@@ -40,7 +39,7 @@ TEST(LlvmLibcSymlinkatTest, CreateAndUnlink) {
 
   int link_fd = __llvm_libc::open(TEST_FILE_LINK_PATH, O_PATH);
   ASSERT_GT(link_fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_THAT(__llvm_libc::close(link_fd), Succeeds(0));
 
   ASSERT_THAT(__llvm_libc::close(dir_fd), Succeeds(0));
diff --git a/libc/test/src/unistd/syscall_test.cpp b/libc/test/src/unistd/syscall_test.cpp
index 7d0cc309806bf..9c9e24282a0de 100644
--- a/libc/test/src/unistd/syscall_test.cpp
+++ b/libc/test/src/unistd/syscall_test.cpp
@@ -6,11 +6,11 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/unistd/syscall.h"
 #include "test/ErrnoSetterMatcher.h"
 #include "test/UnitTest/Test.h"
 
-#include <errno.h>
 #include <fcntl.h>
 #include <sys/syscall.h> // For syscall numbers.
 #include <unistd.h>
@@ -26,26 +26,44 @@ using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
 // because the macro generates a call to the actual internal function
 // (__llvm_libc_syscall) which is inside the namespace.
 TEST(LlvmLibcSyscallTest, TrivialCall) {
-  errno = 0;
+  libc_errno = 0;
 
   ASSERT_GE(__llvm_libc::syscall(SYS_gettid), 0l);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
 }
 
 TEST(LlvmLibcSyscallTest, SymlinkCreateDestroy) {
   constexpr const char LINK_VAL[] = "syscall_readlink_test_value";
   constexpr const char LINK[] = "testdata/syscall_readlink.test.link";
 
+#ifdef SYS_symlink
   ASSERT_GE(__llvm_libc::syscall(SYS_symlink, LINK_VAL, LINK), 0l);
-  ASSERT_EQ(errno, 0);
+#elif defined(SYS_symlinkat)
+  ASSERT_GE(__llvm_libc::syscall(SYS_symlinkat, LINK_VAL, AT_FDCWD, LINK), 0l);
+#else
+#error "Symlink syscalls not available."
+#endif
+  ASSERT_EQ(libc_errno, 0);
 
   char buf[sizeof(LINK_VAL)];
 
+#ifdef SYS_readlink
   ASSERT_GE(__llvm_libc::syscall(SYS_readlink, LINK, buf, sizeof(buf)), 0l);
-  ASSERT_EQ(errno, 0);
-
+#elif defined(SYS_readlinkat)
+  ASSERT_GE(
+      __llvm_libc::syscall(SYS_readlinkat, AT_FDCWD, LINK, buf, sizeof(buf)),
+      0l);
+#endif
+  ASSERT_EQ(libc_errno, 0);
+
+#ifdef SYS_unlink
   ASSERT_GE(__llvm_libc::syscall(SYS_unlink, LINK), 0l);
-  ASSERT_EQ(errno, 0);
+#elif defined(SYS_unlinkat)
+  ASSERT_GE(__llvm_libc::syscall(SYS_unlinkat, AT_FDCWD, LINK, 0), 0l);
+#else
+#error "Unlink syscalls not available."
+#endif
+  ASSERT_EQ(libc_errno, 0);
 }
 
 TEST(LlvmLibcSyscallTest, FileReadWrite) {
@@ -54,19 +72,26 @@ TEST(LlvmLibcSyscallTest, FileReadWrite) {
 
   constexpr const char *TEST_FILE = "testdata/syscall_pread_pwrite.test";
 
+#ifdef SYS_open
   int fd =
       __llvm_libc::syscall(SYS_open, TEST_FILE, O_WRONLY | O_CREAT, S_IRWXU);
+#elif defined(SYS_openat)
+  int fd = __llvm_libc::syscall(SYS_openat, AT_FDCWD, TEST_FILE,
+                                O_WRONLY | O_CREAT, S_IRWXU);
+#else
+#error "Open syscalls not available to available."
+#endif
   ASSERT_GT(fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
 
   ASSERT_GE(__llvm_libc::syscall(SYS_pwrite64, fd, HELLO, HELLO_SIZE, 0), 0l);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
 
   ASSERT_GE(__llvm_libc::syscall(SYS_fsync, fd), 0l);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
 
   ASSERT_GE(__llvm_libc::syscall(SYS_close, fd), 0l);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
 }
 
 TEST(LlvmLibcSyscallTest, FileLinkCreateDestroy) {
@@ -83,33 +108,67 @@ TEST(LlvmLibcSyscallTest, FileLinkCreateDestroy) {
   //   3. Open the link to check that the link was created.
   //   4. Cleanup the file and its link.
 
+#ifdef SYS_open
   int write_fd = __llvm_libc::syscall(SYS_open, TEST_FILE_PATH,
                                       O_WRONLY | O_CREAT, S_IRWXU);
+#elif defined(SYS_openat)
+  int write_fd = __llvm_libc::syscall(SYS_openat, AT_FDCWD, TEST_FILE_PATH,
+                                      O_WRONLY | O_CREAT, S_IRWXU);
+#else
+#error "Open syscalls not available to available."
+#endif
   ASSERT_GT(write_fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
 
   ASSERT_GE(__llvm_libc::syscall(SYS_close, write_fd), 0l);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
 
+#ifdef SYS_open
   int dir_fd = __llvm_libc::syscall(SYS_open, TEST_DIR, O_DIRECTORY, 0);
+#elif defined(SYS_openat)
+  int dir_fd =
+      __llvm_libc::syscall(SYS_openat, AT_FDCWD, TEST_DIR, O_DIRECTORY, 0);
+#else
+#error "Open syscalls not available to available."
+#endif
   ASSERT_GT(dir_fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
 
   ASSERT_GE(__llvm_libc::syscall(SYS_linkat, dir_fd, TEST_FILE, dir_fd,
                                  TEST_FILE_LINK, 0),
             0l);
-  ASSERT_EQ(errno, 0);
-
+  ASSERT_EQ(libc_errno, 0);
+#ifdef SYS_open
   int link_fd = __llvm_libc::syscall(SYS_open, TEST_FILE_LINK_PATH, O_PATH, 0);
+#elif defined(SYS_openat)
+  int link_fd = __llvm_libc::syscall(SYS_openat, AT_FDCWD, TEST_FILE_LINK_PATH,
+                                     O_PATH, 0);
+#else
+#error "Open syscalls not available to available."
+#endif
   ASSERT_GT(link_fd, 0);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
 
+#ifdef SYS_unlink
   ASSERT_GE(__llvm_libc::syscall(SYS_unlink, TEST_FILE_PATH), 0l);
-  ASSERT_EQ(errno, 0);
+#elif defined(SYS_unlinkat)
+  ASSERT_GE(__llvm_libc::syscall(SYS_unlinkat, AT_FDCWD, TEST_FILE_PATH, 0),
+            0l);
+#else
+#error "Unlink syscalls not available."
+#endif
+  ASSERT_EQ(libc_errno, 0);
 
+#ifdef SYS_unlink
   ASSERT_GE(__llvm_libc::syscall(SYS_unlink, TEST_FILE_LINK_PATH), 0l);
-  ASSERT_EQ(errno, 0);
+#elif defined(SYS_unlinkat)
+  ASSERT_GE(
+      __llvm_libc::syscall(SYS_unlinkat, AT_FDCWD, TEST_FILE_LINK_PATH, 0), 0l);
+#else
+#error "Unlink syscalls not available."
+#endif
+  ASSERT_EQ(libc_errno, 0);
 
   ASSERT_GE(__llvm_libc::syscall(SYS_close, dir_fd), 0l);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
 }
diff --git a/libc/test/src/unistd/truncate_test.cpp b/libc/test/src/unistd/truncate_test.cpp
index 70d6d8944c403..85047d2899305 100644
--- a/libc/test/src/unistd/truncate_test.cpp
+++ b/libc/test/src/unistd/truncate_test.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "src/__support/CPP/string_view.h"
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/unistd/close.h"
 #include "src/unistd/read.h"
@@ -16,8 +17,6 @@
 #include "test/ErrnoSetterMatcher.h"
 #include "test/UnitTest/Test.h"
 
-#include <errno.h>
-
 namespace cpp = __llvm_libc::cpp;
 
 TEST(LlvmLibcTruncateTest, CreateAndTruncate) {
@@ -32,16 +31,16 @@ TEST(LlvmLibcTruncateTest, CreateAndTruncate) {
   //   2. Read it to make sure what was written is actually in the file.
   //   3. Truncate to 1 byte.
   //   4. Try to read more than 1 byte and fail.
-  errno = 0;
+  libc_errno = 0;
   int fd = __llvm_libc::open(TEST_FILE, O_WRONLY | O_CREAT, S_IRWXU);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   ASSERT_EQ(ssize_t(WRITE_SIZE),
             __llvm_libc::write(fd, WRITE_DATA, WRITE_SIZE));
   ASSERT_THAT(__llvm_libc::close(fd), Succeeds(0));
 
   fd = __llvm_libc::open(TEST_FILE, O_RDONLY);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   ASSERT_EQ(ssize_t(WRITE_SIZE), __llvm_libc::read(fd, buf, WRITE_SIZE));
   ASSERT_EQ(cpp::string_view(buf), cpp::string_view(WRITE_DATA));
@@ -50,7 +49,7 @@ TEST(LlvmLibcTruncateTest, CreateAndTruncate) {
   ASSERT_THAT(__llvm_libc::truncate(TEST_FILE, off_t(1)), Succeeds(0));
 
   fd = __llvm_libc::open(TEST_FILE, O_RDONLY);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(fd, 0);
   ASSERT_EQ(ssize_t(1), __llvm_libc::read(fd, buf, WRITE_SIZE));
   ASSERT_EQ(buf[0], WRITE_DATA[0]);
diff --git a/libc/test/src/unistd/unlink_test.cpp b/libc/test/src/unistd/unlink_test.cpp
index d8588a5a7d744..2f6eeb1cc0c5f 100644
--- a/libc/test/src/unistd/unlink_test.cpp
+++ b/libc/test/src/unistd/unlink_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/unistd/close.h"
 #include "src/unistd/unlink.h"
@@ -13,13 +14,11 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
-
 TEST(LlvmLibcUnlinkTest, CreateAndUnlink) {
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
   constexpr const char *TEST_FILE = "testdata/unlink.test";
   int write_fd = __llvm_libc::open(TEST_FILE, O_WRONLY | O_CREAT, S_IRWXU);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(write_fd, 0);
   ASSERT_THAT(__llvm_libc::close(write_fd), Succeeds(0));
   ASSERT_THAT(__llvm_libc::unlink(TEST_FILE), Succeeds(0));
diff --git a/libc/test/src/unistd/unlinkat_test.cpp b/libc/test/src/unistd/unlinkat_test.cpp
index 32cecc2d669e3..dbec719dc6635 100644
--- a/libc/test/src/unistd/unlinkat_test.cpp
+++ b/libc/test/src/unistd/unlinkat_test.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "src/errno/libc_errno.h"
 #include "src/fcntl/open.h"
 #include "src/fcntl/openat.h"
 #include "src/unistd/close.h"
@@ -14,18 +15,16 @@
 #include "test/UnitTest/Test.h"
 #include "utils/testutils/FDReader.h"
 
-#include <errno.h>
-
 TEST(LlvmLibcUnlinkatTest, CreateAndDeleteTest) {
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
   constexpr const char *TEST_DIR = "testdata";
   constexpr const char *TEST_FILE = "openat.test";
   int dir_fd = __llvm_libc::open(TEST_DIR, O_DIRECTORY);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(dir_fd, 0);
   int write_fd =
       __llvm_libc::openat(dir_fd, TEST_FILE, O_WRONLY | O_CREAT, S_IRWXU);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(write_fd, 0);
   ASSERT_THAT(__llvm_libc::close(write_fd), Succeeds(0));
   ASSERT_THAT(__llvm_libc::unlinkat(dir_fd, TEST_FILE, 0), Succeeds(0));
@@ -35,7 +34,7 @@ TEST(LlvmLibcUnlinkatTest, CreateAndDeleteTest) {
 TEST(LlvmLibcUnlinkatTest, UnlinkatNonExistentFile) {
   constexpr const char *TEST_DIR = "testdata";
   int dir_fd = __llvm_libc::open(TEST_DIR, O_DIRECTORY);
-  ASSERT_EQ(errno, 0);
+  ASSERT_EQ(libc_errno, 0);
   ASSERT_GT(dir_fd, 0);
   using __llvm_libc::testing::ErrnoSetterMatcher::Fails;
   using __llvm_libc::testing::ErrnoSetterMatcher::Succeeds;
diff --git a/libc/utils/HdrGen/CMakeLists.txt b/libc/utils/HdrGen/CMakeLists.txt
index c68314bb41a0c..0b1612e8a886a 100644
--- a/libc/utils/HdrGen/CMakeLists.txt
+++ b/libc/utils/HdrGen/CMakeLists.txt
@@ -14,7 +14,7 @@ add_tablegen(libc-hdrgen LIBC
   PublicAPICommand.h
 )
 
-target_include_directories(libc-hdrgen PRIVATE ${LIBC_SOURCE_DIR} ${LLVM_LIBC_INCLUDE_DIRS})
+target_include_directories(libc-hdrgen PRIVATE ${LIBC_SOURCE_DIR})
 target_link_libraries(libc-hdrgen PRIVATE LibcTableGenUtil)
 
 add_subdirectory(PrototypeTestGen)
diff --git a/libc/utils/HdrGen/PrototypeTestGen/.clang-tidy b/libc/utils/HdrGen/PrototypeTestGen/.clang-tidy
deleted file mode 100644
index d404f83828f7c..0000000000000
--- a/libc/utils/HdrGen/PrototypeTestGen/.clang-tidy
+++ /dev/null
@@ -1,3 +0,0 @@
-CheckOptions:
-  - key:             readability-identifier-naming.VariableCase
-    value:           camelBack
diff --git a/libc/utils/LibcTableGenUtil/CMakeLists.txt b/libc/utils/LibcTableGenUtil/CMakeLists.txt
index 41881e78da23d..222f177ee2f77 100644
--- a/libc/utils/LibcTableGenUtil/CMakeLists.txt
+++ b/libc/utils/LibcTableGenUtil/CMakeLists.txt
@@ -2,6 +2,7 @@ add_llvm_library(
   LibcTableGenUtil
   APIIndexer.cpp
   APIIndexer.h
+  DISABLE_LLVM_LINK_LLVM_DYLIB
   LINK_COMPONENTS Support TableGen
 )
-target_include_directories(LibcTableGenUtil PUBLIC ${LIBC_SOURCE_DIR} ${LLVM_LIBC_INCLUDE_DIRS})
+target_include_directories(LibcTableGenUtil PUBLIC ${LIBC_SOURCE_DIR})
diff --git a/libc/utils/gpu/loader/CMakeLists.txt b/libc/utils/gpu/loader/CMakeLists.txt
index f66a9a2e2aae8..f643bfcd9abbd 100644
--- a/libc/utils/gpu/loader/CMakeLists.txt
+++ b/libc/utils/gpu/loader/CMakeLists.txt
@@ -4,4 +4,17 @@ target_include_directories(gpu_loader PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
 find_package(hsa-runtime64 QUIET 1.2.0 HINTS ${CMAKE_INSTALL_PREFIX} PATHS /opt/rocm)
 if(hsa-runtime64_FOUND)
   add_subdirectory(amdgpu)
+else()
+  message(STATUS "Skipping HSA loader for gpu target, no HSA was detected")
+endif()
+
+# Add a custom target to be used for testing.
+if(TARGET amdhsa_loader AND LIBC_GPU_TARGET_ARCHITECTURE_IS_AMDGPU)
+  add_custom_target(libc.utils.gpu.loader)
+  add_dependencies(libc.utils.gpu.loader amdhsa_loader)
+  set_target_properties(
+    libc.utils.gpu.loader
+    PROPERTIES
+      EXECUTABLE "$<TARGET_FILE:amdhsa_loader>"
+  )
 endif()
diff --git a/libc/utils/gpu/loader/amdgpu/CMakeLists.txt b/libc/utils/gpu/loader/amdgpu/CMakeLists.txt
index 524e808f7f8a4..bef97af0d195f 100644
--- a/libc/utils/gpu/loader/amdgpu/CMakeLists.txt
+++ b/libc/utils/gpu/loader/amdgpu/CMakeLists.txt
@@ -1,4 +1,7 @@
 add_executable(amdhsa_loader Loader.cpp)
+add_dependencies(amdhsa_loader libc.src.__support.RPC.rpc)
+
+target_include_directories(amdhsa_loader PRIVATE ${LIBC_SOURCE_DIR})
 target_link_libraries(amdhsa_loader
   PRIVATE
   hsa-runtime64::hsa-runtime64
diff --git a/libc/utils/gpu/loader/amdgpu/Loader.cpp b/libc/utils/gpu/loader/amdgpu/Loader.cpp
index 5f444d83083a4..3136dc2509790 100644
--- a/libc/utils/gpu/loader/amdgpu/Loader.cpp
+++ b/libc/utils/gpu/loader/amdgpu/Loader.cpp
@@ -15,6 +15,8 @@
 
 #include "Loader.h"
 
+#include "src/__support/RPC/rpc.h"
+
 #include <hsa/hsa.h>
 #include <hsa/hsa_ext_amd.h>
 
@@ -31,8 +33,35 @@ struct kernel_args_t {
   int argc;
   void *argv;
   void *ret;
+  void *inbox;
+  void *outbox;
+  void *buffer;
 };
 
+static __llvm_libc::rpc::Server server;
+
+/// Queries the RPC client at least once and performs server-side work if there
+/// are any active requests.
+void handle_server() {
+  while (server.run(
+      [&](__llvm_libc::rpc::Buffer *buffer) {
+        switch (static_cast<__llvm_libc::rpc::Opcode>(buffer->data[0])) {
+        case __llvm_libc::rpc::Opcode::PRINT_TO_STDERR: {
+          fputs(reinterpret_cast<const char *>(&buffer->data[1]), stderr);
+          break;
+        }
+        case __llvm_libc::rpc::Opcode::EXIT: {
+          exit(buffer->data[1]);
+          break;
+        }
+        default:
+          return;
+        };
+      },
+      [](__llvm_libc::rpc::Buffer *buffer) {}))
+    ;
+}
+
 /// Print the error code and exit if \p code indicates an error.
 static void handle_error(hsa_status_t code) {
   if (code == HSA_STATUS_SUCCESS || code == HSA_STATUS_INFO_BREAK)
@@ -278,6 +307,26 @@ int load(int argc, char **argv, void *image, size_t size) {
     handle_error(err);
   hsa_amd_memory_fill(dev_ret, 0, sizeof(int));
 
+  // Allocate finegrained memory for the RPC server and client to share.
+  void *server_inbox;
+  void *server_outbox;
+  void *buffer;
+  if (hsa_status_t err = hsa_amd_memory_pool_allocate(
+          finegrained_pool, sizeof(__llvm_libc::cpp::Atomic<int>),
+          /*flags=*/0, &server_inbox))
+    handle_error(err);
+  if (hsa_status_t err = hsa_amd_memory_pool_allocate(
+          finegrained_pool, sizeof(__llvm_libc::cpp::Atomic<int>),
+          /*flags=*/0, &server_outbox))
+    handle_error(err);
+  if (hsa_status_t err = hsa_amd_memory_pool_allocate(
+          finegrained_pool, sizeof(__llvm_libc::rpc::Buffer),
+          /*flags=*/0, &buffer))
+    handle_error(err);
+  hsa_amd_agents_allow_access(1, &dev_agent, nullptr, server_inbox);
+  hsa_amd_agents_allow_access(1, &dev_agent, nullptr, server_outbox);
+  hsa_amd_agents_allow_access(1, &dev_agent, nullptr, buffer);
+
   // Initialie all the arguments (explicit and implicit) to zero, then set the
   // explicit arguments to the values created above.
   std::memset(args, 0, args_size);
@@ -285,6 +334,9 @@ int load(int argc, char **argv, void *image, size_t size) {
   kernel_args->argc = argc;
   kernel_args->argv = dev_argv;
   kernel_args->ret = dev_ret;
+  kernel_args->inbox = server_outbox;
+  kernel_args->outbox = server_inbox;
+  kernel_args->buffer = buffer;
 
   // Obtain a packet from the queue.
   uint64_t packet_id = hsa_queue_add_write_index_relaxed(queue, 1);
@@ -316,6 +368,9 @@ int load(int argc, char **argv, void *image, size_t size) {
           hsa_signal_create(1, 0, nullptr, &packet->completion_signal))
     handle_error(err);
 
+  // Initialize the RPC server's buffer for host-device communication.
+  server.reset(server_inbox, server_outbox, buffer);
+
   // Initialize the packet header and set the doorbell signal to begin execution
   // by the HSA runtime.
   uint16_t header =
@@ -326,11 +381,12 @@ int load(int argc, char **argv, void *image, size_t size) {
                    __ATOMIC_RELEASE);
   hsa_signal_store_relaxed(queue->doorbell_signal, packet_id);
 
-  // Wait until the kernel has completed execution on the device.
-  while (hsa_signal_wait_scacquire(packet->completion_signal,
-                                   HSA_SIGNAL_CONDITION_EQ, 0, UINT64_MAX,
-                                   HSA_WAIT_STATE_ACTIVE) != 0)
-    ;
+  // Wait until the kernel has completed execution on the device. Periodically
+  // check the RPC client for work to be performed on the server.
+  while (hsa_signal_wait_scacquire(
+             packet->completion_signal, HSA_SIGNAL_CONDITION_EQ, 0,
+             /*timeout_hint=*/1024, HSA_WAIT_STATE_ACTIVE) != 0)
+    handle_server();
 
   // Create a memory signal and copy the return value back from the device into
   // a new buffer.
diff --git a/libcxx/.clang-format b/libcxx/.clang-format
index 68180bbef4c87..acf987dbe8860 100644
--- a/libcxx/.clang-format
+++ b/libcxx/.clang-format
@@ -37,7 +37,6 @@ AttributeMacros: ['_LIBCPP_HIDE_FROM_ABI',
                   '_LIBCPP_EXCLUDE_FROM_EXPLICIT_INSTANTIATION',
                   '_LIBCPP_HIDE_FROM_ABI_AFTER_V1',
                   '_LIBCPP_INLINE_VISIBILITY',
-                  '_LIBCPP_CONSTEVAL',
                   '_LIBCPP_NOALIAS',
                   '_LIBCPP_USING_IF_EXISTS',
                   '_LIBCPP_DEPRECATED',
diff --git a/libcxx/CMakeLists.txt b/libcxx/CMakeLists.txt
index 0c1b2fe32f7ae..cf572af74bd13 100644
--- a/libcxx/CMakeLists.txt
+++ b/libcxx/CMakeLists.txt
@@ -24,6 +24,7 @@ set(LIBCXX_BINARY_DIR  ${CMAKE_CURRENT_BINARY_DIR})
 set(LIBCXX_BINARY_INCLUDE_DIR "${LIBCXX_BINARY_DIR}/include/c++build")
 
 include(GNUInstallDirs)
+include(WarningFlags)
 
 # Require out of source build.
 include(MacroEnsureOutOfSourceBuild)
@@ -543,6 +544,11 @@ function(cxx_add_basic_build_flags target)
   # library.
   target_compile_definitions(${target} PRIVATE -D_LIBCPP_BUILDING_LIBRARY)
 
+  # Make sure the library can be build without transitive includes. This makes
+  # it easier to upgrade the library to a newer language standard without build
+  # errors.
+  target_compile_definitions(${target} PRIVATE -D_LIBCPP_REMOVE_TRANSITIVE_INCLUDES)
+
   if (NOT LIBCXX_ENABLE_NEW_DELETE_DEFINITIONS)
     target_compile_definitions(${target} PRIVATE -D_LIBCPP_DISABLE_NEW_DELETE_DEFINITIONS)
   endif()
@@ -558,65 +564,6 @@ function(cxx_add_basic_build_flags target)
   target_compile_options(${target} PUBLIC "${LIBCXX_ADDITIONAL_COMPILE_FLAGS}")
 endfunction()
 
-# Warning flags ===============================================================
-function(cxx_add_warning_flags target)
-  target_compile_definitions(${target} PUBLIC -D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
-  if (MSVC)
-    # -W4 is the cl.exe/clang-cl equivalent of -Wall. (In cl.exe and clang-cl,
-    # -Wall is equivalent to -Weverything in GCC style compiler drivers.)
-    target_add_compile_flags_if_supported(${target} PRIVATE -W4)
-  else()
-    target_add_compile_flags_if_supported(${target} PRIVATE -Wall)
-  endif()
-  target_add_compile_flags_if_supported(${target} PRIVATE -Wextra -W -Wwrite-strings
-                                                          -Wno-unused-parameter -Wno-long-long
-                                                          -Werror=return-type -Wextra-semi -Wundef
-                                                          -Wformat-nonliteral)
-  if ("${CMAKE_CXX_COMPILER_ID}" MATCHES "Clang")
-    target_add_compile_flags_if_supported(${target} PRIVATE
-      -Wno-user-defined-literals
-      -Wno-covered-switch-default
-      -Wno-suggest-override
-    )
-    if (LIBCXX_TARGETING_CLANG_CL)
-      target_add_compile_flags_if_supported(${target} PRIVATE
-        -Wno-c++98-compat
-        -Wno-c++98-compat-pedantic
-        -Wno-c++11-compat
-        -Wno-undef
-        -Wno-reserved-id-macro
-        -Wno-gnu-include-next
-        -Wno-gcc-compat # For ignoring "'diagnose_if' is a clang extension" warnings
-        -Wno-zero-as-null-pointer-constant # FIXME: Remove this and fix all occurrences.
-        -Wno-deprecated-dynamic-exception-spec # For auto_ptr
-        -Wno-sign-conversion
-        -Wno-old-style-cast
-        -Wno-deprecated # FIXME: Remove this and fix all occurrences.
-        -Wno-shift-sign-overflow # FIXME: Why do we need this with clang-cl but not clang?
-        -Wno-double-promotion # FIXME: remove me
-      )
-    endif()
-  elseif("${CMAKE_CXX_COMPILER_ID}" MATCHES "GNU")
-    target_add_compile_flags_if_supported(${target} PRIVATE
-      -Wno-attributes
-      -Wno-literal-suffix
-      -Wno-c++14-compat
-      -Wno-noexcept-type
-      -Wno-suggest-override)
-  endif()
-  if (LIBCXX_ENABLE_WERROR)
-    target_add_compile_flags_if_supported(${target} PRIVATE -Werror)
-    target_add_compile_flags_if_supported(${target} PRIVATE -WX)
-  else()
-    # TODO(EricWF) Remove this. We shouldn't be suppressing errors when -Werror is
-    # added elsewhere.
-    target_add_compile_flags_if_supported(${target} PRIVATE -Wno-error)
-  endif()
-  if (LIBCXX_ENABLE_PEDANTIC)
-    target_add_compile_flags_if_supported(${target} PRIVATE -pedantic)
-  endif()
-endfunction()
-
 # Exception flags =============================================================
 function(cxx_add_exception_flags target)
   if (LIBCXX_ENABLE_EXCEPTIONS)
@@ -899,7 +846,7 @@ endif()
 # Setup all common build flags =================================================
 function(cxx_add_common_build_flags target)
   cxx_add_basic_build_flags(${target})
-  cxx_add_warning_flags(${target})
+  cxx_add_warning_flags(${target} ${LIBCXX_ENABLE_WERROR} ${LIBCXX_ENABLE_PEDANTIC})
   cxx_add_windows_flags(${target})
   cxx_add_exception_flags(${target})
   cxx_add_rtti_flags(${target})
diff --git a/libcxx/benchmarks/CMakeLists.txt b/libcxx/benchmarks/CMakeLists.txt
index 572f210d71004..bf7c4b42b0ac6 100644
--- a/libcxx/benchmarks/CMakeLists.txt
+++ b/libcxx/benchmarks/CMakeLists.txt
@@ -162,6 +162,7 @@ set(BENCHMARK_TESTS
     algorithms/lower_bound.bench.cpp
     algorithms/make_heap.bench.cpp
     algorithms/make_heap_then_sort_heap.bench.cpp
+    algorithms/min.bench.cpp
     algorithms/min_max_element.bench.cpp
     algorithms/pop_heap.bench.cpp
     algorithms/push_heap.bench.cpp
@@ -175,6 +176,7 @@ set(BENCHMARK_TESTS
     algorithms/sort.bench.cpp
     algorithms/sort_heap.bench.cpp
     algorithms/stable_sort.bench.cpp
+    libcxxabi/dynamic_cast.bench.cpp
     allocation.bench.cpp
     deque.bench.cpp
     deque_iterator.bench.cpp
diff --git a/libcxx/benchmarks/algorithms/min.bench.cpp b/libcxx/benchmarks/algorithms/min.bench.cpp
new file mode 100644
index 0000000000000..1e1dd4e71c2ef
--- /dev/null
+++ b/libcxx/benchmarks/algorithms/min.bench.cpp
@@ -0,0 +1,70 @@
+#include <algorithm>
+#include <cassert>
+
+#include <benchmark/benchmark.h>
+
+void run_sizes(auto benchmark) {
+  benchmark->Arg(1)
+      ->Arg(2)
+      ->Arg(3)
+      ->Arg(4)
+      ->Arg(5)
+      ->Arg(6)
+      ->Arg(7)
+      ->Arg(8)
+      ->Arg(9)
+      ->Arg(10)
+      ->Arg(11)
+      ->Arg(12)
+      ->Arg(13)
+      ->Arg(14)
+      ->Arg(15)
+      ->Arg(16)
+      ->Arg(17)
+      ->Arg(18)
+      ->Arg(19)
+      ->Arg(20)
+      ->Arg(21)
+      ->Arg(22)
+      ->Arg(23)
+      ->Arg(24)
+      ->Arg(25)
+      ->Arg(26)
+      ->Arg(27)
+      ->Arg(28)
+      ->Arg(29)
+      ->Arg(30)
+      ->Arg(31)
+      ->Arg(32)
+      ->Arg(64)
+      ->Arg(512)
+      ->Arg(1024)
+      ->Arg(4000)
+      ->Arg(4096)
+      ->Arg(5500)
+      ->Arg(64000)
+      ->Arg(65536)
+      ->Arg(70000);
+}
+
+template <class T>
+static void BM_std_min(benchmark::State& state) {
+  std::vector<T> vec(state.range(), 3);
+
+  for (auto _ : state) {
+    benchmark::DoNotOptimize(vec);
+    benchmark::DoNotOptimize(std::ranges::min(vec));
+  }
+}
+BENCHMARK(BM_std_min<char>)->Apply(run_sizes);
+BENCHMARK(BM_std_min<short>)->Apply(run_sizes);
+BENCHMARK(BM_std_min<int>)->Apply(run_sizes);
+BENCHMARK(BM_std_min<long long>)->Apply(run_sizes);
+BENCHMARK(BM_std_min<__int128>)->Apply(run_sizes);
+BENCHMARK(BM_std_min<unsigned char>)->Apply(run_sizes);
+BENCHMARK(BM_std_min<unsigned short>)->Apply(run_sizes);
+BENCHMARK(BM_std_min<unsigned int>)->Apply(run_sizes);
+BENCHMARK(BM_std_min<unsigned long long>)->Apply(run_sizes);
+BENCHMARK(BM_std_min<unsigned __int128>)->Apply(run_sizes);
+
+BENCHMARK_MAIN();
diff --git a/libcxx/benchmarks/libcxxabi/dynamic_cast.bench.cpp b/libcxx/benchmarks/libcxxabi/dynamic_cast.bench.cpp
new file mode 100644
index 0000000000000..439eea81d9a19
--- /dev/null
+++ b/libcxx/benchmarks/libcxxabi/dynamic_cast.bench.cpp
@@ -0,0 +1,172 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include <cstddef>
+
+#include "benchmark/benchmark.h"
+
+template <std::size_t Depth>
+struct Chain : Chain<Depth - 1> {};
+
+template <>
+struct Chain<0> {
+  virtual ~Chain() noexcept = default;
+};
+
+template <std::size_t Index, std::size_t Depth>
+struct Dag : Dag<Index, Depth - 1>, Dag<Index + 1, Depth - 1> {};
+
+template <std::size_t Index>
+struct Dag<Index, 0> {
+  virtual ~Dag() noexcept = default;
+};
+
+template <std::size_t Depth>
+struct VChain : virtual VChain<Depth - 1> {};
+
+template <>
+struct VChain<0> {
+  virtual ~VChain() noexcept = default;
+};
+
+template <std::size_t Index, std::size_t Depth>
+struct VDag : virtual VDag<Index, Depth - 1>, virtual VDag<Index + 1, Depth - 1> {};
+
+template <std::size_t Index>
+struct VDag<Index, 0> {
+  virtual ~VDag() noexcept = default;
+};
+
+template <typename Dyn, typename From, typename To = Dyn>
+static void DynCast(benchmark::State& state) {
+  Dyn obj;
+  From* from_ptr = &obj;
+  for (auto _ : state) {
+    To* to_ptr = dynamic_cast<To*>(from_ptr);
+    benchmark::DoNotOptimize(to_ptr);
+  }
+}
+
+static void StaticCast(benchmark::State& state) {
+  Chain<9> obj;
+  Chain<0>* from_ptr = &obj;
+  for (auto _ : state) {
+    Chain<9>* to_ptr = static_cast<Chain<9>*>(from_ptr);
+    benchmark::DoNotOptimize(to_ptr);
+  }
+}
+
+// Downcast along a chain from base to the most derived type
+BENCHMARK(DynCast<Chain<1>, Chain<0>>)->Name("Chain, 1 level");
+BENCHMARK(DynCast<Chain<2>, Chain<0>>)->Name("Chain, 2 levels");
+BENCHMARK(DynCast<Chain<3>, Chain<0>>)->Name("Chain, 3 levels");
+BENCHMARK(DynCast<Chain<4>, Chain<0>>)->Name("Chain, 4 levels");
+BENCHMARK(DynCast<Chain<5>, Chain<0>>)->Name("Chain, 5 levels");
+BENCHMARK(DynCast<Chain<6>, Chain<0>>)->Name("Chain, 6 levels");
+BENCHMARK(DynCast<Chain<7>, Chain<0>>)->Name("Chain, 7 levels");
+BENCHMARK(DynCast<Chain<8>, Chain<0>>)->Name("Chain, 8 levels");
+BENCHMARK(DynCast<Chain<9>, Chain<0>>)->Name("Chain, 9 levels");
+
+// Downcast along a chain from base to the middle of the chain
+BENCHMARK(DynCast<Chain<2>, Chain<0>, Chain<1>>)->Name("Chain middle, 1 level");
+BENCHMARK(DynCast<Chain<4>, Chain<0>, Chain<2>>)->Name("Chain middle, 2 levels");
+BENCHMARK(DynCast<Chain<6>, Chain<0>, Chain<3>>)->Name("Chain middle, 3 levels");
+BENCHMARK(DynCast<Chain<8>, Chain<0>, Chain<4>>)->Name("Chain middle, 4 levels");
+
+// Downcast along a chain that fails
+BENCHMARK(DynCast<Chain<1>, Chain<0>, Chain<9>>)->Name("Chain fail, 1 level");
+BENCHMARK(DynCast<Chain<2>, Chain<0>, Chain<9>>)->Name("Chain fail, 2 levels");
+BENCHMARK(DynCast<Chain<3>, Chain<0>, Chain<9>>)->Name("Chain fail, 3 levels");
+BENCHMARK(DynCast<Chain<4>, Chain<0>, Chain<9>>)->Name("Chain fail, 4 levels");
+BENCHMARK(DynCast<Chain<5>, Chain<0>, Chain<9>>)->Name("Chain fail, 5 levels");
+BENCHMARK(DynCast<Chain<6>, Chain<0>, Chain<9>>)->Name("Chain fail, 6 levels");
+BENCHMARK(DynCast<Chain<7>, Chain<0>, Chain<9>>)->Name("Chain fail, 7 levels");
+BENCHMARK(DynCast<Chain<8>, Chain<0>, Chain<9>>)->Name("Chain fail, 8 levels");
+
+// Downcast along a virtual inheritance chain from base to the most derived type
+BENCHMARK(DynCast<VChain<1>, VChain<0>>)->Name("VChain, 1 level");
+BENCHMARK(DynCast<VChain<2>, VChain<0>>)->Name("VChain, 2 levels");
+BENCHMARK(DynCast<VChain<3>, VChain<0>>)->Name("VChain, 3 levels");
+BENCHMARK(DynCast<VChain<4>, VChain<0>>)->Name("VChain, 4 levels");
+BENCHMARK(DynCast<VChain<5>, VChain<0>>)->Name("VChain, 5 levels");
+
+// Downcast along a virtual inheritance chain from base to the middle of the chain
+BENCHMARK(DynCast<VChain<2>, VChain<0>, VChain<1>>)->Name("VChain middle, 1 level");
+BENCHMARK(DynCast<VChain<4>, VChain<0>, VChain<2>>)->Name("VChain middle, 2 levels");
+BENCHMARK(DynCast<VChain<6>, VChain<0>, VChain<3>>)->Name("VChain middle, 3 levels");
+BENCHMARK(DynCast<VChain<8>, VChain<0>, VChain<4>>)->Name("VChain middle, 4 levels");
+
+// Downcast along a virtual chain that fails
+BENCHMARK(DynCast<VChain<1>, VChain<0>, VChain<8>>)->Name("VChain fail, 1 level");
+BENCHMARK(DynCast<VChain<2>, VChain<0>, VChain<8>>)->Name("VChain fail, 2 levels");
+BENCHMARK(DynCast<VChain<3>, VChain<0>, VChain<8>>)->Name("VChain fail, 3 levels");
+BENCHMARK(DynCast<VChain<4>, VChain<0>, VChain<8>>)->Name("VChain fail, 4 levels");
+BENCHMARK(DynCast<VChain<5>, VChain<0>, VChain<8>>)->Name("VChain fail, 5 levels");
+
+// Downcast along a DAG from base to the most derived type
+BENCHMARK(DynCast<Dag<0, 3>, Dag<3, 0>>)->Name("DAG rightmost, 3 levels");
+BENCHMARK(DynCast<Dag<0, 4>, Dag<4, 0>>)->Name("DAG rightmost, 4 levels");
+BENCHMARK(DynCast<Dag<0, 5>, Dag<5, 0>>)->Name("DAG rightmost, 5 levels");
+BENCHMARK(DynCast<Dag<0, 3>, Dag<0, 0>>)->Name("DAG leftmost, 3 levels");
+BENCHMARK(DynCast<Dag<0, 4>, Dag<0, 0>>)->Name("DAG leftmost, 4 levels");
+BENCHMARK(DynCast<Dag<0, 5>, Dag<0, 0>>)->Name("DAG leftmost, 5 levels");
+
+// Downcast along a DAG from base to the middle of the DAG
+BENCHMARK(DynCast<Dag<0, 4>, Dag<4, 0>, Dag<3, 1>>)->Name("DAG rightmost middle, 1 level");
+BENCHMARK(DynCast<Dag<0, 4>, Dag<4, 0>, Dag<2, 2>>)->Name("DAG rightmost middle, 2 levels");
+BENCHMARK(DynCast<Dag<0, 4>, Dag<4, 0>, Dag<1, 3>>)->Name("DAG rightmost middle, 3 levels");
+BENCHMARK(DynCast<Dag<0, 4>, Dag<0, 0>, Dag<0, 1>>)->Name("DAG leftmost middle, 1 level");
+BENCHMARK(DynCast<Dag<0, 4>, Dag<0, 0>, Dag<0, 2>>)->Name("DAG leftmost middle, 2 levels");
+BENCHMARK(DynCast<Dag<0, 4>, Dag<0, 0>, Dag<0, 3>>)->Name("DAG leftmost middle, 3 levels");
+
+// Sidecast along a DAG
+BENCHMARK(DynCast<Dag<0, 3>, Dag<3, 0>, Dag<0, 0>>)->Name("DAG sidecast, 3 levels");
+BENCHMARK(DynCast<Dag<0, 3>, Dag<2, 1>, Dag<0, 1>>)->Name("DAG sidecast, 2 levels");
+BENCHMARK(DynCast<Dag<0, 3>, Dag<1, 2>, Dag<0, 2>>)->Name("DAG sidecast, 1 level");
+
+// Sidecast along a DAG that fails
+BENCHMARK(DynCast<Dag<0, 3>, Dag<3, 0>, Dag<0, 4>>)->Name("DAG sidecast fail, 3 levels");
+BENCHMARK(DynCast<Dag<0, 3>, Dag<2, 1>, Dag<0, 4>>)->Name("DAG sidecast fail, 2 levels");
+BENCHMARK(DynCast<Dag<0, 3>, Dag<1, 2>, Dag<0, 4>>)->Name("DAG sidecast fail, 1 level");
+
+// Downcast along a virtual inheritance DAG from base to the most derived type
+BENCHMARK(DynCast<VDag<0, 3>, VDag<3, 0>>)->Name("VDAG rightmost, 3 levels");
+BENCHMARK(DynCast<VDag<0, 4>, VDag<4, 0>>)->Name("VDAG rightmost, 4 levels");
+BENCHMARK(DynCast<VDag<0, 5>, VDag<5, 0>>)->Name("VDAG rightmost, 5 levels");
+BENCHMARK(DynCast<VDag<0, 3>, VDag<0, 0>>)->Name("VDAG leftmost, 3 levels");
+BENCHMARK(DynCast<VDag<0, 4>, VDag<0, 0>>)->Name("VDAG leftmost, 4 levels");
+BENCHMARK(DynCast<VDag<0, 5>, VDag<0, 0>>)->Name("VDAG leftmost, 5 levels");
+
+// Downcast along a virtual inheritance DAG from base to the middle of the DAG
+BENCHMARK(DynCast<VDag<0, 3>, VDag<3, 0>, VDag<2, 1>>)->Name("VDAG rightmost middle, 1 level");
+BENCHMARK(DynCast<VDag<0, 4>, VDag<4, 0>, VDag<2, 2>>)->Name("VDAG rightmost middle, 2 levels");
+BENCHMARK(DynCast<VDag<0, 5>, VDag<5, 0>, VDag<2, 3>>)->Name("VDAG rightmost middle, 3 levels");
+BENCHMARK(DynCast<VDag<0, 3>, VDag<0, 0>, VDag<0, 1>>)->Name("VDAG leftmost middle, 1 level");
+BENCHMARK(DynCast<VDag<0, 4>, VDag<0, 0>, VDag<0, 2>>)->Name("VDAG leftmost middle, 2 levels");
+BENCHMARK(DynCast<VDag<0, 5>, VDag<0, 0>, VDag<0, 3>>)->Name("VDAG leftmost middle, 3 levels");
+
+// Sidecast along a virtual inheritance DAG
+BENCHMARK(DynCast<VDag<0, 3>, VDag<3, 0>, VDag<0, 0>>)->Name("VDAG sidecast, 3 levels");
+BENCHMARK(DynCast<VDag<0, 3>, VDag<2, 1>, VDag<0, 1>>)->Name("VDAG sidecast, 2 levels");
+BENCHMARK(DynCast<VDag<0, 3>, VDag<1, 2>, VDag<0, 2>>)->Name("VDAG sidecast, 1 level");
+
+// Sidecast along a virtual inheritance DAG that fails
+BENCHMARK(DynCast<VDag<0, 3>, VDag<3, 0>, VDag<0, 4>>)->Name("VDAG sidecast fail, 3 levels");
+BENCHMARK(DynCast<VDag<0, 3>, VDag<2, 1>, VDag<0, 4>>)->Name("VDAG sidecast fail, 2 levels");
+BENCHMARK(DynCast<VDag<0, 3>, VDag<1, 2>, VDag<0, 4>>)->Name("VDAG sidecast fail, 1 level");
+
+// Cast to complete object pointer
+BENCHMARK(DynCast<Chain<8>, Chain<0>, void>)->Name("Chain to complete");
+BENCHMARK(DynCast<VChain<5>, VChain<0>, void>)->Name("VChain to complete");
+BENCHMARK(DynCast<Dag<0, 3>, Dag<3, 0>, void>)->Name("DAG to complete");
+BENCHMARK(DynCast<VDag<0, 3>, VDag<3, 0>, void>)->Name("VDAG to complete");
+
+// Static cast as the baseline.
+BENCHMARK(StaticCast)->Name("Static");
+
+BENCHMARK_MAIN();
diff --git a/libcxx/cmake/caches/Apple.cmake b/libcxx/cmake/caches/Apple.cmake
index 5224e04687a14..c9ee3b075bf77 100644
--- a/libcxx/cmake/caches/Apple.cmake
+++ b/libcxx/cmake/caches/Apple.cmake
@@ -17,5 +17,3 @@ set(LIBCXXABI_ENABLE_FORGIVING_DYNAMIC_CAST ON CACHE BOOL "")
 
 set(LIBCXX_TEST_CONFIG "apple-libc++-shared.cfg.in" CACHE STRING "")
 set(LIBCXXABI_TEST_CONFIG "apple-libc++abi-shared.cfg.in" CACHE STRING "")
-set(LIBCXX_TEST_PARAMS "stdlib=apple-libc++" CACHE STRING "")
-set(LIBCXXABI_TEST_PARAMS "${LIBCXX_TEST_PARAMS}" CACHE STRING "")
diff --git a/libcxx/docs/ReleaseNotes.rst b/libcxx/docs/ReleaseNotes.rst
index 549efe727f2ce..aa14e6bfcd97d 100644
--- a/libcxx/docs/ReleaseNotes.rst
+++ b/libcxx/docs/ReleaseNotes.rst
@@ -48,6 +48,9 @@ Improvements and New Features
 - ``std::string_view`` now provides iterators that check for out-of-bounds accesses when the safe
   libc++ mode is enabled.
 
+- The performance of ``dynamic_cast`` on its hot paths is greatly improved and is as efficient as the
+  ``libsupc++`` implementation. Note that the performance improvements are shipped in ``libcxxabi``.
+
 Deprecations and Removals
 -------------------------
 
@@ -59,7 +62,7 @@ Deprecations and Removals
   includes are removed based on the language version used. Incidental transitive
   inclusions of the following headers have been removed:
 
-  - C++2b: ``bit``, ``cstring``, ``type_traits``
+  - C++2b: ``atomic``, ``bit``, ``cstring``, ``type_traits``
 
 - The headers ``<experimental/algorithm>`` and ``<experimental/functional>`` have been removed, since all the contents
   have been implemented in namespace ``std`` for at least two releases.
@@ -70,6 +73,15 @@ Deprecations and Removals
   from the Standard since it was never used, the proper specialization to use
   instead is ``template<size_t N> struct formatter<charT[N], charT>``.
 
+- Libc++ used to provide some C++11 tag type global variables in C++03 as an extension, which are removed in
+  this release. Those variables were ``std::allocator_arg``, ``std::defer_lock``, ``std::try_to_lock``,
+  ``std::adopt_lock``, and ``std::piecewise_construct``. Note that the types associated to those variables are
+  still provided in C++03 as an extension (e.g. ``std::piecewise_construct_t``). Providing those variables in
+  C++03 mode made it impossible to define them properly -- C++11 mandated that they be ``constexpr`` variables,
+  which is impossible in C++03 mode. Furthermore, C++17 mandated that they be ``inline constexpr`` variables,
+  which led to ODR violations when mixed with the C++03 definition. Cleaning this up is required for libc++ to
+  make progress on support for C++20 modules.
+
 Upcoming Deprecations and Removals
 ----------------------------------
 
@@ -91,3 +103,7 @@ ABI Affecting Changes
 
 Build System Changes
 --------------------
+
+- Building libc++ and libc++abi for Apple platforms now requires targeting macOS 10.13 and later.
+  This is relevant for vendors building the libc++ shared library and for folks statically linking
+  libc++ into an application that has back-deployment requirements on Apple platforms.
diff --git a/libcxx/docs/Status/Cxx20Issues.csv b/libcxx/docs/Status/Cxx20Issues.csv
index c2cee7e88b83a..c0143931be018 100644
--- a/libcxx/docs/Status/Cxx20Issues.csv
+++ b/libcxx/docs/Status/Cxx20Issues.csv
@@ -1,6 +1,6 @@
 "Issue #","Issue Name","Meeting","Status","First released version","Labels"
 "`2070 <https://wg21.link/LWG2070>`__","``allocate_shared``\  should use ``allocator_traits<A>::construct``\ ","Toronto","Resolved by `P0674R1 <https://wg21.link/P0674R1>`__",""
-"`2444 <https://wg21.link/LWG2444>`__","Inconsistent complexity for ``std::sort_heap``\ ","Toronto","",""
+"`2444 <https://wg21.link/LWG2444>`__","Inconsistent complexity for ``std::sort_heap``\ ","Toronto","|Nothing To Do|",""
 "`2593 <https://wg21.link/LWG2593>`__","Moved-from state of Allocators","Toronto","",""
 "`2597 <https://wg21.link/LWG2597>`__","``std::log``\  misspecified for complex numbers","Toronto","",""
 "`2783 <https://wg21.link/LWG2783>`__","``stack::emplace()``\  and ``queue::emplace()``\  should return ``decltype(auto)``\ ","Toronto","|Complete|",""
@@ -268,7 +268,7 @@
 "`3354 <https://wg21.link/LWG3354>`__","``has_strong_structural_equality``\  has a meaningless definition","Prague","|Nothing To Do|","","|spaceship|"
 "`3355 <https://wg21.link/LWG3355>`__","The memory algorithms should support move-only input iterators introduced by P1207","Prague","|Complete|","15.0","|ranges|"
 "`3356 <https://wg21.link/LWG3356>`__","``__cpp_lib_nothrow_convertible``\  should be ``__cpp_lib_is_nothrow_convertible``\ ","Prague","|Complete|","12.0"
-"`3358 <https://wg21.link/LWG3358>`__","|sect|\ [span.cons] is mistaken that ``to_address``\  can throw","Prague","",""
+"`3358 <https://wg21.link/LWG3358>`__","|sect|\ [span.cons] is mistaken that ``to_address``\  can throw","Prague","|Complete|","17.0"
 "`3359 <https://wg21.link/LWG3359>`__","``<chrono>``\  leap second support should allow for negative leap seconds","Prague","","","|chrono|"
 "`3360 <https://wg21.link/LWG3360>`__","``three_way_comparable_with``\  is inconsistent with similar concepts","Prague","|Nothing To Do|","","|spaceship|"
 "`3362 <https://wg21.link/LWG3362>`__","Strike ``stop_source``\ 's ``operator!=``\ ","Prague","",""
diff --git a/libcxx/docs/Status/Cxx2bIssues.csv b/libcxx/docs/Status/Cxx2bIssues.csv
index 324643b96a816..2a8a4dcce9dbd 100644
--- a/libcxx/docs/Status/Cxx2bIssues.csv
+++ b/libcxx/docs/Status/Cxx2bIssues.csv
@@ -256,7 +256,7 @@
 "`3807 <https://wg21.link/LWG3807>`__","The feature test macro for ``ranges::find_last`` should be renamed","February 2023","","","|ranges|"
 "`3811 <https://wg21.link/LWG3811>`__","``views::as_const`` on ``ref_view<T>`` should return ``ref_view<const T>``","February 2023","","","|ranges|"
 "`3820 <https://wg21.link/LWG3820>`__","``cartesian_product_view::iterator::prev`` is not quite right","February 2023","","","|ranges|"
-"`3825 <https://wg21.link/LWG3825>`__","Missing compile-time argument ``id`` check in ``basic_format_parse_context::next_arg_id``","February 2023","","","|format|"
+"`3825 <https://wg21.link/LWG3825>`__","Missing compile-time argument ``id`` check in ``basic_format_parse_context::next_arg_id``","February 2023","|Complete|","17.0","|format|"
 "`3204 <https://wg21.link/LWG3204>`__","``sub_match::swap`` only swaps the base class","February 2023","|Complete|","17.0",""
 "`3733 <https://wg21.link/LWG3733>`__","``ranges::to`` misuses ``cpp17-input-iterator``","February 2023","","","|ranges|"
 "`3742 <https://wg21.link/LWG3742>`__","``deque::prepend_range`` needs to permute","February 2023","","","|ranges|"
@@ -264,7 +264,7 @@
 "`3819 <https://wg21.link/LWG3819>`__","``reference_meows_from_temporary`` should not use ``is_meowible``","February 2023","","",""
 "`3821 <https://wg21.link/LWG3821>`__","``uses_allocator_construction_args`` should have overload for ``pair-like``","February 2023","","",""
 "`3834 <https://wg21.link/LWG3834>`__","Missing ``constexpr`` for ``std::intmax_t`` math functions in ``<cinttypes>``","February 2023","","",""
-"`3839 <https://wg21.link/LWG3839>`__","``range_formatter``'s ``set_separator``, ``set_brackets``, and ``underlying`` functions should be ``noexcept``","February 2023","","","|format|"
+"`3839 <https://wg21.link/LWG3839>`__","``range_formatter``'s ``set_separator``, ``set_brackets``, and ``underlying`` functions should be ``noexcept``","February 2023","|Complete|","17.0","|format|"
 "`3841 <https://wg21.link/LWG3841>`__","``<version>`` should not be ""all freestanding""","February 2023","","",""
 "`3842 <https://wg21.link/LWG3842>`__","Unclear wording for ``precision`` in ``chrono-format-spec``","February 2023","","","|format|"
 "`3848 <https://wg21.link/LWG3848>`__","``adjacent_view``, ``adjacent_transform_view`` and ``slide_view`` missing ``base`` accessor","February 2023","","","|ranges|"
@@ -294,7 +294,7 @@
 "`3843 <https://wg21.link/LWG3843>`__","``std::expected<T,E>::value() &`` assumes ``E`` is copy constructible","February 2023","","",""
 "`3847 <https://wg21.link/LWG3847>`__","``ranges::to`` can still return views","February 2023","","","|ranges|"
 "`3862 <https://wg21.link/LWG3862>`__","``basic_const_iterator``'s ``common_type`` specialization is underconstrained","February 2023","","",""
-"`3865 <https://wg21.link/LWG3865>`__","Sorting a range of ``pairs``","February 2023","","","|ranges|"
+"`3865 <https://wg21.link/LWG3865>`__","Sorting a range of ``pairs``","February 2023","|Complete|","17.0","|ranges|"
 "`3869 <https://wg21.link/LWG3869>`__","Deprecate ``std::errc`` constants related to UNIX STREAMS","February 2023","","",""
 "`3870 <https://wg21.link/LWG3870>`__","Remove ``voidify``","February 2023","","",""
 "`3871 <https://wg21.link/LWG3871>`__","Adjust note about ``terminate``","February 2023","","",""
@@ -305,6 +305,6 @@
 "`3878 <https://wg21.link/LWG3878>`__","import ``std;`` should guarantee initialization of standard iostreams objects","February 2023","","",""
 "`3879 <https://wg21.link/LWG3879>`__","``erase_if`` for ``flat_{,multi}set`` is incorrectly specified","February 2023","","",""
 "`3880 <https://wg21.link/LWG3880>`__","Clarify ``operator+=`` complexity for ``{chunk,stride}_view::iterator``","February 2023","","","|ranges|"
-"`3881 <https://wg21.link/LWG3881>`__","Incorrect formatting of container adapters backed by ``std::string``","February 2023","","","|format|"
+"`3881 <https://wg21.link/LWG3881>`__","Incorrect formatting of container adapters backed by ``std::string``","February 2023","|Complete|","17.0","|format|"
 "","","","","",""
 "`3343 <https://wg21.link/LWG3343>`__","Ordering of calls to ``unlock()`` and ``notify_all()`` in Effects element of ``notify_all_at_thread_exit()`` should be reversed","Not Yet Adopted","|Complete|","16.0",""
diff --git a/libcxx/docs/Status/Parallelism.rst b/libcxx/docs/Status/Parallelism.rst
new file mode 100644
index 0000000000000..4fce429bbe6ce
--- /dev/null
+++ b/libcxx/docs/Status/Parallelism.rst
@@ -0,0 +1,29 @@
+.. parallelism-status:
+
+====================================
+libc++ Parallelism TS Status (N4808)
+====================================
+
+.. include:: ../Helpers/Styles.rst
+
+.. contents::
+  :local:
+
+Overview
+========
+
+This document contains the status of the Parallelism TS implementation in libc++.
+It is used to track both the status of the sub-projects of the effort and who
+is assigned to these sub-projects. This avoids duplicating effort.
+
+If you are interested in contributing to this effort, please send a message
+to the #libcxx channel in the LLVM discord. Please *do not* start working
+on any items below that has already been assigned to someone else.
+
+Sub-projects in the Implementation Effort
+=========================================
+
+.. csv-table::
+  :file: ParallelismProjects.csv
+  :header-rows: 1
+  :widths: auto
diff --git a/libcxx/docs/Status/ParallelismProjects.csv b/libcxx/docs/Status/ParallelismProjects.csv
new file mode 100644
index 0000000000000..de59ad601b9a0
--- /dev/null
+++ b/libcxx/docs/Status/ParallelismProjects.csv
@@ -0,0 +1,12 @@
+Section,Description,Dependencies,Assignee,Complete
+| `[parallel.exceptions] <https://wg21.link/N4808>`_, "Parallel Exceptions", None, unassigned, |Not Started|
+| `[parallel.execpol] <https://wg21.link/N4808>`_, "Execution policies", None, unassigned, |Not Started|
+| `[parallel.alg] <https://wg21.link/N4808>`_, "Parallel algorithms", None, unassigned, |Not Started|
+| `[parallel.taskblock] <https://wg21.link/N4808>`_, "Task Block", None, unassigned, |Not Started|
+| `[parallel.simd.abi] <https://wg21.link/N4808>`_, "simd ABI tags", None, Yin Zhang, |In Progress|
+| `[parallel.simd.traits] <https://wg21.link/N4808>`_, "simd type traits", None, Yin Zhang, |In Progress|
+| `[parallel.simd.whereexpr] <https://wg21.link/N4808>`_, "Where expression class templates", None, Yin Zhang, |In Progress|
+| `[parallel.simd.class] <https://wg21.link/N4808>`_, "Class template simd", None, Yin Zhang, |In Progress|
+| `[parallel.simd.nonmembers] <https://wg21.link/N4808>`_, "simd non-member operations", None, Yin Zhang, |In Progress|
+| `[parallel.simd.mask.class] <https://wg21.link/N4808>`_, "Class template simd_mask", None, Yin Zhang, |In Progress|
+| `[parallel.simd.mask.nonmembers] <https://wg21.link/N4808>`_, "simd_mask non-member operations", None, Yin Zhang, |In Progress|
diff --git a/libcxx/docs/Status/SpaceshipProjects.csv b/libcxx/docs/Status/SpaceshipProjects.csv
index c043272071117..a3dc4ee81ec34 100644
--- a/libcxx/docs/Status/SpaceshipProjects.csv
+++ b/libcxx/docs/Status/SpaceshipProjects.csv
@@ -40,8 +40,8 @@ Section,Description,Dependencies,Assignee,Complete
 | `[forward.list.syn] <https://wg21.link/forward.list.syn>`_ (`general <https://wg21.link/container.opt.reqmts>`_),| `forward_list <https://reviews.llvm.org/D145172>`_,[expos.only.func],Hristo Hristov,|Complete|
 | `[list.syn] <https://wg21.link/list.syn>`_ (`general <https://wg21.link/container.opt.reqmts>`_),| `list <https://reviews.llvm.org/D132312>`_,[expos.only.func],Adrian Vogelsgesang,|Complete|
 | `[vector.syn] <https://wg21.link/vector.syn>`_ (`general <https://wg21.link/container.opt.reqmts>`_),| `vector <https://reviews.llvm.org/D132268>`_,[expos.only.func],Adrian Vogelsgesang,|In Progress|
-| `[associative.map.syn] <https://wg21.link/associative.map.syn>`_ (`general <https://wg21.link/container.opt.reqmts>`_),"| map
-| multimap",[expos.only.func],Unassigned,|Not Started|
+| `[associative.map.syn] <https://wg21.link/associative.map.syn>`_ (`general <https://wg21.link/container.opt.reqmts>`_),"| `map <https://reviews.llvm.org/D145976>`_
+| `multimap <https://reviews.llvm.org/D145976>`_",[expos.only.func],Hristo Hristov,|Complete|
 | `[associative.set.syn] <https://wg21.link/associative.set.syn>`_ (`general <https://wg21.link/container.opt.reqmts>`_),"| multiset
 | set",[expos.only.func],Unassigned,|Not Started|
 | `[queue.ops] <https://wg21.link/queue.ops>`_,| queue,None,Unassigned,|Not Started|
diff --git a/libcxx/docs/index.rst b/libcxx/docs/index.rst
index 41a87f4a90a67..abcdeabec4870 100644
--- a/libcxx/docs/index.rst
+++ b/libcxx/docs/index.rst
@@ -44,6 +44,7 @@ Getting Started with libc++
    Status/Cxx20
    Status/Cxx2b
    Status/Format
+   Status/Parallelism
    Status/Ranges
    Status/Spaceship
    Status/Zip
@@ -116,7 +117,7 @@ Libc++ also supports common platforms and architectures:
 =============== ========================= ============================
 Target platform Target architecture       Notes
 =============== ========================= ============================
-macOS 10.9+     i386, x86_64, arm64       Building the shared library itself requires targetting macOS 10.11+
+macOS 10.9+     i386, x86_64, arm64       Building the shared library itself requires targetting macOS 10.13+
 FreeBSD 12+     i386, x86_64, arm
 Linux           i386, x86_64, arm, arm64  Only glibc-2.24 and later and no other libc is officially supported
 Windows         i386, x86_64              Both MSVC and MinGW style environments
diff --git a/libcxx/include/CMakeLists.txt b/libcxx/include/CMakeLists.txt
index 34932dee692ed..f8c52328ccff0 100644
--- a/libcxx/include/CMakeLists.txt
+++ b/libcxx/include/CMakeLists.txt
@@ -225,10 +225,15 @@ set(files
   __bsd_locale_defaults.h
   __bsd_locale_fallbacks.h
   __charconv/chars_format.h
+  __charconv/from_chars_integral.h
   __charconv/from_chars_result.h
   __charconv/tables.h
+  __charconv/to_chars.h
   __charconv/to_chars_base_10.h
+  __charconv/to_chars_floating_point.h
+  __charconv/to_chars_integral.h
   __charconv/to_chars_result.h
+  __charconv/traits.h
   __chrono/calendar.h
   __chrono/concepts.h
   __chrono/convert_to_timespec.h
@@ -297,6 +302,11 @@ set(files
   __debug
   __debug_utils/randomize_range.h
   __errc
+  __exception/exception.h
+  __exception/exception_ptr.h
+  __exception/nested_exception.h
+  __exception/operations.h
+  __exception/terminate.h
   __expected/bad_expected_access.h
   __expected/expected.h
   __expected/unexpect.h
diff --git a/libcxx/include/__algorithm/make_projected.h b/libcxx/include/__algorithm/make_projected.h
index f4ca4c9d4a7a8..61055fcdf30d7 100644
--- a/libcxx/include/__algorithm/make_projected.h
+++ b/libcxx/include/__algorithm/make_projected.h
@@ -60,7 +60,7 @@ template <class _Pred,
           __enable_if_t<!(!is_member_pointer<typename decay<_Pred>::type>::value &&
                             __is_identity<typename decay<_Proj>::type>::value),
                         int> = 0>
-_LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR static _ProjectedPred<_Pred, _Proj>
+_LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR _ProjectedPred<_Pred, _Proj>
 __make_projected(_Pred& __pred, _Proj& __proj) {
   return _ProjectedPred<_Pred, _Proj>(__pred, __proj);
 }
@@ -73,7 +73,7 @@ template <class _Pred,
           __enable_if_t<!is_member_pointer<typename decay<_Pred>::type>::value &&
                           __is_identity<typename decay<_Proj>::type>::value,
                         int> = 0>
-_LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR static _Pred& __make_projected(_Pred& __pred, _Proj&) {
+_LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR _Pred& __make_projected(_Pred& __pred, _Proj&) {
   return __pred;
 }
 
@@ -86,7 +86,7 @@ _LIBCPP_BEGIN_NAMESPACE_STD
 namespace ranges {
 
 template <class _Comp, class _Proj1, class _Proj2>
-_LIBCPP_HIDE_FROM_ABI constexpr static
+_LIBCPP_HIDE_FROM_ABI constexpr
 decltype(auto) __make_projected_comp(_Comp& __comp, _Proj1& __proj1, _Proj2& __proj2) {
   if constexpr (__is_identity<decay_t<_Proj1>>::value && __is_identity<decay_t<_Proj2>>::value &&
                 !is_member_pointer_v<decay_t<_Comp>>) {
diff --git a/libcxx/include/__algorithm/ranges_binary_search.h b/libcxx/include/__algorithm/ranges_binary_search.h
index b43de468ddabd..d89597e1be31c 100644
--- a/libcxx/include/__algorithm/ranges_binary_search.h
+++ b/libcxx/include/__algorithm/ranges_binary_search.h
@@ -36,7 +36,7 @@ struct __fn {
   _LIBCPP_NODISCARD_EXT _LIBCPP_HIDE_FROM_ABI constexpr
   bool operator()(_Iter __first, _Sent __last, const _Type& __value, _Comp __comp = {}, _Proj __proj = {}) const {
     auto __ret = std::__lower_bound_impl<_RangeAlgPolicy>(__first, __last, __value, __comp, __proj);
-    return __ret != __last && !std::invoke(__comp, __value, std::invoke(__proj, *__first));
+    return __ret != __last && !std::invoke(__comp, __value, std::invoke(__proj, *__ret));
   }
 
   template <forward_range _Range, class _Type, class _Proj = identity,
@@ -46,7 +46,7 @@ struct __fn {
     auto __first = ranges::begin(__r);
     auto __last = ranges::end(__r);
     auto __ret = std::__lower_bound_impl<_RangeAlgPolicy>(__first, __last, __value, __comp, __proj);
-    return __ret != __last && !std::invoke(__comp, __value, std::invoke(__proj, *__first));
+    return __ret != __last && !std::invoke(__comp, __value, std::invoke(__proj, *__ret));
   }
 };
 } // namespace __binary_search
diff --git a/libcxx/include/__algorithm/ranges_find_if.h b/libcxx/include/__algorithm/ranges_find_if.h
index 0b27b8c5dee74..6e2bd4c3ddd4a 100644
--- a/libcxx/include/__algorithm/ranges_find_if.h
+++ b/libcxx/include/__algorithm/ranges_find_if.h
@@ -31,7 +31,7 @@ _LIBCPP_BEGIN_NAMESPACE_STD
 namespace ranges {
 
 template <class _Ip, class _Sp, class _Pred, class _Proj>
-_LIBCPP_HIDE_FROM_ABI static constexpr
+_LIBCPP_HIDE_FROM_ABI constexpr
 _Ip __find_if_impl(_Ip __first, _Sp __last, _Pred& __pred, _Proj& __proj) {
   for (; __first != __last; ++__first) {
     if (std::invoke(__pred, std::invoke(__proj, *__first)))
diff --git a/libcxx/include/__algorithm/ranges_max.h b/libcxx/include/__algorithm/ranges_max.h
index b3c7fb163de1e..2fd2970bbf35d 100644
--- a/libcxx/include/__algorithm/ranges_max.h
+++ b/libcxx/include/__algorithm/ranges_max.h
@@ -20,6 +20,7 @@
 #include <__iterator/projected.h>
 #include <__ranges/access.h>
 #include <__ranges/concepts.h>
+#include <__type_traits/is_trivially_copyable.h>
 #include <__utility/move.h>
 #include <initializer_list>
 
@@ -67,7 +68,7 @@ struct __fn {
 
     _LIBCPP_ASSERT(__first != __last, "range must contain at least one element");
 
-    if constexpr (forward_range<_Rp>) {
+    if constexpr (forward_range<_Rp> && !__is_cheap_to_copy<range_value_t<_Rp>>) {
       auto __comp_lhs_rhs_swapped = [&](auto&& __lhs, auto&& __rhs) { return std::invoke(__comp, __rhs, __lhs); };
       return *ranges::__min_element_impl(std::move(__first), std::move(__last), __comp_lhs_rhs_swapped, __proj);
     } else {
diff --git a/libcxx/include/__algorithm/ranges_min.h b/libcxx/include/__algorithm/ranges_min.h
index ba9dad5e9c200..5e941a1db06a8 100644
--- a/libcxx/include/__algorithm/ranges_min.h
+++ b/libcxx/include/__algorithm/ranges_min.h
@@ -20,6 +20,7 @@
 #include <__iterator/projected.h>
 #include <__ranges/access.h>
 #include <__ranges/concepts.h>
+#include <__type_traits/is_trivially_copyable.h>
 #include <initializer_list>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
@@ -61,10 +62,8 @@ struct __fn {
   range_value_t<_Rp> operator()(_Rp&& __r, _Comp __comp = {}, _Proj __proj = {}) const {
     auto __first = ranges::begin(__r);
     auto __last = ranges::end(__r);
-
     _LIBCPP_ASSERT(__first != __last, "range must contain at least one element");
-
-    if constexpr (forward_range<_Rp>) {
+    if constexpr (forward_range<_Rp> && !__is_cheap_to_copy<range_value_t<_Rp>>) {
       return *ranges::__min_element_impl(__first, __last, __comp, __proj);
     } else {
       range_value_t<_Rp> __result = *__first;
diff --git a/libcxx/include/__algorithm/ranges_min_element.h b/libcxx/include/__algorithm/ranges_min_element.h
index 6f0dec68f9b64..4c58adb3119f8 100644
--- a/libcxx/include/__algorithm/ranges_min_element.h
+++ b/libcxx/include/__algorithm/ranges_min_element.h
@@ -32,7 +32,7 @@ namespace ranges {
 
 // TODO(ranges): `ranges::min_element` can now simply delegate to `std::__min_element`.
 template <class _Ip, class _Sp, class _Proj, class _Comp>
-_LIBCPP_HIDE_FROM_ABI static constexpr
+_LIBCPP_HIDE_FROM_ABI constexpr
 _Ip __min_element_impl(_Ip __first, _Sp __last, _Comp& __comp, _Proj& __proj) {
   if (__first == __last)
     return __first;
diff --git a/libcxx/include/__algorithm/sort.h b/libcxx/include/__algorithm/sort.h
index c893270eeaa8a..ee627c4f6d172 100644
--- a/libcxx/include/__algorithm/sort.h
+++ b/libcxx/include/__algorithm/sort.h
@@ -276,7 +276,7 @@ void __insertion_sort(_BidirectionalIterator __first, _BidirectionalIterator __l
 
 // Sort the iterator range [__first, __last) using the comparator __comp using
 // the insertion sort algorithm.  Insertion sort has two loops, outer and inner.
-// The implementation below has not bounds check (unguarded) for the inner loop.
+// The implementation below has no bounds check (unguarded) for the inner loop.
 // Assumes that there is an element in the position (__first - 1) and that each
 // element in the input range is greater or equal to the element at __first - 1.
 template <class _AlgPolicy, class _Compare, class _RandomAccessIterator>
diff --git a/libcxx/include/__availability b/libcxx/include/__availability
index bd1c92ae73dbb..c03d373cafb51 100644
--- a/libcxx/include/__availability
+++ b/libcxx/include/__availability
@@ -239,7 +239,10 @@
 #   endif
 
 #   define _LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT                         \
-        __attribute__((unavailable))
+        __attribute__((availability(macos,strict,introduced=13.3)))             \
+        __attribute__((availability(ios,strict,introduced=16.3)))               \
+        __attribute__((availability(tvos,strict,introduced=16.3)))              \
+        __attribute__((availability(watchos,strict,introduced=9.3)))
 
 #   define _LIBCPP_AVAILABILITY_SYNC                                            \
         __attribute__((availability(macos,strict,introduced=11.0)))             \
diff --git a/libcxx/include/__bit_reference b/libcxx/include/__bit_reference
index c66dc4a8e5e97..95188ff68c823 100644
--- a/libcxx/include/__bit_reference
+++ b/libcxx/include/__bit_reference
@@ -19,7 +19,9 @@
 #include <__iterator/iterator_traits.h>
 #include <__memory/construct_at.h>
 #include <__memory/pointer_traits.h>
-#include <type_traits>
+#include <__type_traits/conditional.h>
+#include <__utility/swap.h>
+#include <cstring>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__charconv/chars_format.h b/libcxx/include/__charconv/chars_format.h
index 0d7538e4f6cce..0e781c047e39c 100644
--- a/libcxx/include/__charconv/chars_format.h
+++ b/libcxx/include/__charconv/chars_format.h
@@ -21,50 +21,37 @@ _LIBCPP_BEGIN_NAMESPACE_STD
 
 #if _LIBCPP_STD_VER >= 17
 
-enum class _LIBCPP_ENUM_VIS chars_format
-{
-    scientific = 0x1,
-    fixed = 0x2,
-    hex = 0x4,
-    general = fixed | scientific
-};
-
-inline _LIBCPP_INLINE_VISIBILITY constexpr chars_format
-operator~(chars_format __x) {
-  return chars_format(~_VSTD::__to_underlying(__x));
+enum class _LIBCPP_ENUM_VIS chars_format { scientific = 0x1, fixed = 0x2, hex = 0x4, general = fixed | scientific };
+
+inline _LIBCPP_HIDE_FROM_ABI constexpr chars_format operator~(chars_format __x) {
+  return chars_format(~std::__to_underlying(__x));
 }
 
-inline _LIBCPP_INLINE_VISIBILITY constexpr chars_format
-operator&(chars_format __x, chars_format __y) {
-  return chars_format(_VSTD::__to_underlying(__x) &
-                      _VSTD::__to_underlying(__y));
+inline _LIBCPP_HIDE_FROM_ABI constexpr chars_format operator&(chars_format __x, chars_format __y) {
+  return chars_format(std::__to_underlying(__x) & std::__to_underlying(__y));
 }
 
-inline _LIBCPP_INLINE_VISIBILITY constexpr chars_format
-operator|(chars_format __x, chars_format __y) {
-  return chars_format(_VSTD::__to_underlying(__x) |
-                      _VSTD::__to_underlying(__y));
+inline _LIBCPP_HIDE_FROM_ABI constexpr chars_format operator|(chars_format __x, chars_format __y) {
+  return chars_format(std::__to_underlying(__x) | std::__to_underlying(__y));
 }
 
-inline _LIBCPP_INLINE_VISIBILITY constexpr chars_format
-operator^(chars_format __x, chars_format __y) {
-  return chars_format(_VSTD::__to_underlying(__x) ^
-                      _VSTD::__to_underlying(__y));
+inline _LIBCPP_HIDE_FROM_ABI constexpr chars_format operator^(chars_format __x, chars_format __y) {
+  return chars_format(std::__to_underlying(__x) ^ std::__to_underlying(__y));
 }
 
-inline _LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR_SINCE_CXX14 chars_format&
+inline _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX14 chars_format&
 operator&=(chars_format& __x, chars_format __y) {
   __x = __x & __y;
   return __x;
 }
 
-inline _LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR_SINCE_CXX14 chars_format&
+inline _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX14 chars_format&
 operator|=(chars_format& __x, chars_format __y) {
   __x = __x | __y;
   return __x;
 }
 
-inline _LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR_SINCE_CXX14 chars_format&
+inline _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX14 chars_format&
 operator^=(chars_format& __x, chars_format __y) {
   __x = __x ^ __y;
   return __x;
diff --git a/libcxx/include/__charconv/from_chars_integral.h b/libcxx/include/__charconv/from_chars_integral.h
new file mode 100644
index 0000000000000..1b729850da3e4
--- /dev/null
+++ b/libcxx/include/__charconv/from_chars_integral.h
@@ -0,0 +1,239 @@
+// -*- C++ -*-
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _LIBCPP___CHARCONV_FROM_CHARS_INTEGRAL_H
+#define _LIBCPP___CHARCONV_FROM_CHARS_INTEGRAL_H
+
+#include <__algorithm/copy_n.h>
+#include <__charconv/from_chars_result.h>
+#include <__charconv/traits.h>
+#include <__config>
+#include <__errc>
+#include <__memory/addressof.h>
+#include <__type_traits/enable_if.h>
+#include <__type_traits/integral_constant.h>
+#include <__type_traits/is_integral.h>
+#include <__type_traits/is_unsigned.h>
+#include <__type_traits/make_unsigned.h>
+#include <limits>
+
+#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
+#  pragma GCC system_header
+#endif
+
+_LIBCPP_PUSH_MACROS
+#include <__undef_macros>
+
+_LIBCPP_BEGIN_NAMESPACE_STD
+
+#if _LIBCPP_STD_VER >= 17
+
+from_chars_result from_chars(const char*, const char*, bool, int = 10) = delete;
+
+template <typename _It, typename _Tp, typename _Fn, typename... _Ts>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI from_chars_result
+__sign_combinator(_It __first, _It __last, _Tp& __value, _Fn __f, _Ts... __args) {
+  using __tl = numeric_limits<_Tp>;
+  decltype(std::__to_unsigned_like(__value)) __x;
+
+  bool __neg = (__first != __last && *__first == '-');
+  auto __r   = __f(__neg ? __first + 1 : __first, __last, __x, __args...);
+  switch (__r.ec) {
+  case errc::invalid_argument:
+    return {__first, __r.ec};
+  case errc::result_out_of_range:
+    return __r;
+  default:
+    break;
+  }
+
+  if (__neg) {
+    if (__x <= std::__complement(std::__to_unsigned_like(__tl::min()))) {
+      __x = std::__complement(__x);
+      std::copy_n(std::addressof(__x), 1, std::addressof(__value));
+      return __r;
+    }
+  } else {
+    if (__x <= std::__to_unsigned_like(__tl::max())) {
+      __value = __x;
+      return __r;
+    }
+  }
+
+  return {__r.ptr, errc::result_out_of_range};
+}
+
+template <typename _Tp>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI bool __in_pattern(_Tp __c) {
+  return '0' <= __c && __c <= '9';
+}
+
+struct _LIBCPP_HIDDEN __in_pattern_result {
+  bool __ok;
+  int __val;
+
+  explicit _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI operator bool() const { return __ok; }
+};
+
+template <typename _Tp>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI __in_pattern_result __in_pattern(_Tp __c, int __base) {
+  if (__base <= 10)
+    return {'0' <= __c && __c < '0' + __base, __c - '0'};
+  else if (std::__in_pattern(__c))
+    return {true, __c - '0'};
+  else if ('a' <= __c && __c < 'a' + __base - 10)
+    return {true, __c - 'a' + 10};
+  else
+    return {'A' <= __c && __c < 'A' + __base - 10, __c - 'A' + 10};
+}
+
+template <typename _It, typename _Tp, typename _Fn, typename... _Ts>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI from_chars_result
+__subject_seq_combinator(_It __first, _It __last, _Tp& __value, _Fn __f, _Ts... __args) {
+  auto __find_non_zero = [](_It __firstit, _It __lastit) {
+    for (; __firstit != __lastit; ++__firstit)
+      if (*__firstit != '0')
+        break;
+    return __firstit;
+  };
+
+  auto __p = __find_non_zero(__first, __last);
+  if (__p == __last || !std::__in_pattern(*__p, __args...)) {
+    if (__p == __first)
+      return {__first, errc::invalid_argument};
+    else {
+      __value = 0;
+      return {__p, {}};
+    }
+  }
+
+  auto __r = __f(__p, __last, __value, __args...);
+  if (__r.ec == errc::result_out_of_range) {
+    for (; __r.ptr != __last; ++__r.ptr) {
+      if (!std::__in_pattern(*__r.ptr, __args...))
+        break;
+    }
+  }
+
+  return __r;
+}
+
+template <typename _Tp, typename enable_if<is_unsigned<_Tp>::value, int>::type = 0>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI from_chars_result
+__from_chars_atoi(const char* __first, const char* __last, _Tp& __value) {
+  using __tx          = __itoa::__traits<_Tp>;
+  using __output_type = typename __tx::type;
+
+  return std::__subject_seq_combinator(
+      __first, __last, __value, [](const char* __f, const char* __l, _Tp& __val) -> from_chars_result {
+        __output_type __a, __b;
+        auto __p = __tx::__read(__f, __l, __a, __b);
+        if (__p == __l || !std::__in_pattern(*__p)) {
+          __output_type __m = numeric_limits<_Tp>::max();
+          if (__m >= __a && __m - __a >= __b) {
+            __val = __a + __b;
+            return {__p, {}};
+          }
+        }
+        return {__p, errc::result_out_of_range};
+      });
+}
+
+template <typename _Tp, typename enable_if<is_signed<_Tp>::value, int>::type = 0>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI from_chars_result
+__from_chars_atoi(const char* __first, const char* __last, _Tp& __value) {
+  using __t = decltype(std::__to_unsigned_like(__value));
+  return std::__sign_combinator(__first, __last, __value, __from_chars_atoi<__t>);
+}
+
+/*
+// Code used to generate __from_chars_log2f_lut.
+#include <cmath>
+#include <format>
+#include <iostream>
+
+int main() {
+  for (int i = 2; i <= 36; ++i)
+    std::cout << std::format("{},\n", log2f(i));
+}
+*/
+/// log2f table for bases [2, 36].
+inline constexpr float __from_chars_log2f_lut[35] = {
+    1,         1.5849625, 2,         2.321928, 2.5849626, 2.807355, 3,        3.169925,  3.321928,
+    3.4594316, 3.5849626, 3.7004397, 3.807355, 3.9068906, 4,        4.087463, 4.169925,  4.2479277,
+    4.321928,  4.3923173, 4.4594316, 4.523562, 4.5849624, 4.643856, 4.70044,  4.7548876, 4.807355,
+    4.857981,  4.9068904, 4.9541965, 5,        5.044394,  5.087463, 5.129283, 5.169925};
+
+template <typename _Tp, typename enable_if<is_unsigned<_Tp>::value, int>::type = 0>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI from_chars_result
+__from_chars_integral(const char* __first, const char* __last, _Tp& __value, int __base) {
+  if (__base == 10)
+    return std::__from_chars_atoi(__first, __last, __value);
+
+  return std::__subject_seq_combinator(
+      __first,
+      __last,
+      __value,
+      [](const char* __p, const char* __lastp, _Tp& __val, int __b) -> from_chars_result {
+        using __tl = numeric_limits<_Tp>;
+        // __base is always between 2 and 36 inclusive.
+        auto __digits = __tl::digits / __from_chars_log2f_lut[__b - 2];
+        _Tp __x = __in_pattern(*__p++, __b).__val, __y = 0;
+
+        for (int __i = 1; __p != __lastp; ++__i, ++__p) {
+          if (auto __c = __in_pattern(*__p, __b)) {
+            if (__i < __digits - 1)
+              __x = __x * __b + __c.__val;
+            else {
+              if (!__itoa::__mul_overflowed(__x, __b, __x))
+                ++__p;
+              __y = __c.__val;
+              break;
+            }
+          } else
+            break;
+        }
+
+        if (__p == __lastp || !__in_pattern(*__p, __b)) {
+          if (__tl::max() - __x >= __y) {
+            __val = __x + __y;
+            return {__p, {}};
+          }
+        }
+        return {__p, errc::result_out_of_range};
+      },
+      __base);
+}
+
+template <typename _Tp, typename enable_if<is_signed<_Tp>::value, int>::type = 0>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI from_chars_result
+__from_chars_integral(const char* __first, const char* __last, _Tp& __value, int __base) {
+  using __t = decltype(std::__to_unsigned_like(__value));
+  return std::__sign_combinator(__first, __last, __value, __from_chars_integral<__t>, __base);
+}
+
+template <typename _Tp, typename enable_if<is_integral<_Tp>::value, int>::type = 0>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI from_chars_result
+from_chars(const char* __first, const char* __last, _Tp& __value) {
+  return std::__from_chars_atoi(__first, __last, __value);
+}
+
+template <typename _Tp, typename enable_if<is_integral<_Tp>::value, int>::type = 0>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI from_chars_result
+from_chars(const char* __first, const char* __last, _Tp& __value, int __base) {
+  _LIBCPP_ASSERT(2 <= __base && __base <= 36, "base not in [2, 36]");
+  return std::__from_chars_integral(__first, __last, __value, __base);
+}
+#endif // _LIBCPP_STD_VER >= 17
+
+_LIBCPP_END_NAMESPACE_STD
+
+_LIBCPP_POP_MACROS
+
+#endif // _LIBCPP___CHARCONV_FROM_CHARS_INTEGRAL_H
diff --git a/libcxx/include/__charconv/from_chars_result.h b/libcxx/include/__charconv/from_chars_result.h
index 5267463a7ad85..028f5142ceccd 100644
--- a/libcxx/include/__charconv/from_chars_result.h
+++ b/libcxx/include/__charconv/from_chars_result.h
@@ -21,12 +21,11 @@ _LIBCPP_BEGIN_NAMESPACE_STD
 
 #if _LIBCPP_STD_VER >= 17
 
-struct _LIBCPP_TYPE_VIS from_chars_result
-{
-    const char* ptr;
-    errc ec;
+struct _LIBCPP_TYPE_VIS from_chars_result {
+  const char* ptr;
+  errc ec;
 #  if _LIBCPP_STD_VER >= 20
-    _LIBCPP_HIDE_FROM_ABI friend bool operator==(const from_chars_result&, const from_chars_result&) = default;
+  _LIBCPP_HIDE_FROM_ABI friend bool operator==(const from_chars_result&, const from_chars_result&) = default;
 #  endif
 };
 
diff --git a/libcxx/include/__charconv/tables.h b/libcxx/include/__charconv/tables.h
index 81d9a8bada53a..6b93536b8c1ba 100644
--- a/libcxx/include/__charconv/tables.h
+++ b/libcxx/include/__charconv/tables.h
@@ -62,32 +62,41 @@ inline constexpr char __base_16_lut[512] = {
     'f', 'd', 'f', 'e', 'f', 'f'};
 
 inline constexpr uint32_t __pow10_32[10] = {
-    UINT32_C(0),      UINT32_C(10),      UINT32_C(100),      UINT32_C(1000),      UINT32_C(10000),
-    UINT32_C(100000), UINT32_C(1000000), UINT32_C(10000000), UINT32_C(100000000), UINT32_C(1000000000)};
-
-inline constexpr uint64_t __pow10_64[20] = {UINT64_C(0),
-                                               UINT64_C(10),
-                                               UINT64_C(100),
-                                               UINT64_C(1000),
-                                               UINT64_C(10000),
-                                               UINT64_C(100000),
-                                               UINT64_C(1000000),
-                                               UINT64_C(10000000),
-                                               UINT64_C(100000000),
-                                               UINT64_C(1000000000),
-                                               UINT64_C(10000000000),
-                                               UINT64_C(100000000000),
-                                               UINT64_C(1000000000000),
-                                               UINT64_C(10000000000000),
-                                               UINT64_C(100000000000000),
-                                               UINT64_C(1000000000000000),
-                                               UINT64_C(10000000000000000),
-                                               UINT64_C(100000000000000000),
-                                               UINT64_C(1000000000000000000),
-                                               UINT64_C(10000000000000000000)};
+    UINT32_C(0),
+    UINT32_C(10),
+    UINT32_C(100),
+    UINT32_C(1000),
+    UINT32_C(10000),
+    UINT32_C(100000),
+    UINT32_C(1000000),
+    UINT32_C(10000000),
+    UINT32_C(100000000),
+    UINT32_C(1000000000)};
+
+inline constexpr uint64_t __pow10_64[20] = {
+    UINT64_C(0),
+    UINT64_C(10),
+    UINT64_C(100),
+    UINT64_C(1000),
+    UINT64_C(10000),
+    UINT64_C(100000),
+    UINT64_C(1000000),
+    UINT64_C(10000000),
+    UINT64_C(100000000),
+    UINT64_C(1000000000),
+    UINT64_C(10000000000),
+    UINT64_C(100000000000),
+    UINT64_C(1000000000000),
+    UINT64_C(10000000000000),
+    UINT64_C(100000000000000),
+    UINT64_C(1000000000000000),
+    UINT64_C(10000000000000000),
+    UINT64_C(100000000000000000),
+    UINT64_C(1000000000000000000),
+    UINT64_C(10000000000000000000)};
 
 #  ifndef _LIBCPP_HAS_NO_INT128
-inline constexpr int __pow10_128_offset = 0;
+inline constexpr int __pow10_128_offset      = 0;
 inline constexpr __uint128_t __pow10_128[40] = {
     UINT64_C(0),
     UINT64_C(10),
diff --git a/libcxx/include/__charconv/to_chars.h b/libcxx/include/__charconv/to_chars.h
new file mode 100644
index 0000000000000..8ef09af737559
--- /dev/null
+++ b/libcxx/include/__charconv/to_chars.h
@@ -0,0 +1,25 @@
+// -*- C++ -*-
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _LIBCPP___CHARCONV_TO_CHARS
+#define _LIBCPP___CHARCONV_TO_CHARS
+
+#include <__charconv/to_chars_floating_point.h>
+#include <__charconv/to_chars_integral.h>
+#include <__config>
+
+#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
+#  pragma GCC system_header
+#endif
+
+_LIBCPP_BEGIN_NAMESPACE_STD
+
+_LIBCPP_END_NAMESPACE_STD
+
+#endif // _LIBCPP___CHARCONV_TO_CHARS
diff --git a/libcxx/include/__charconv/to_chars_base_10.h b/libcxx/include/__charconv/to_chars_base_10.h
index a11714da86cd5..028ff3352b65a 100644
--- a/libcxx/include/__charconv/to_chars_base_10.h
+++ b/libcxx/include/__charconv/to_chars_base_10.h
@@ -72,7 +72,8 @@ _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI char* __append10(char* __fir
                            static_cast<uint32_t>(__value % 100000000));
 }
 
-_LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI inline char* __base_10_u32(char* __first, uint32_t __value) noexcept {
+_LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI inline char*
+__base_10_u32(char* __first, uint32_t __value) noexcept {
   if (__value < 1000000) {
     if (__value < 10000) {
       if (__value < 100) {
@@ -107,7 +108,8 @@ _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI inline char* __base_10_u32(c
   return __itoa::__append10(__first, __value);
 }
 
-_LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI inline char* __base_10_u64(char* __buffer, uint64_t __value) noexcept {
+_LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI inline char*
+__base_10_u64(char* __buffer, uint64_t __value) noexcept {
   if (__value <= UINT32_MAX)
     return __itoa::__base_10_u32(__buffer, static_cast<uint32_t>(__value));
 
@@ -134,7 +136,8 @@ _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI inline __uint128_t __pow_10(
   return __pow10_128[__exp - __pow10_128_offset];
 }
 
-_LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI inline char* __base_10_u128(char* __buffer, __uint128_t __value) noexcept {
+_LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI inline char*
+__base_10_u128(char* __buffer, __uint128_t __value) noexcept {
   _LIBCPP_ASSERT(
       __value > numeric_limits<uint64_t>::max(), "The optimizations for this algorithm fail when this isn't true.");
 
@@ -159,8 +162,7 @@ _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI inline char* __base_10_u128(
     __value %= __itoa::__pow_10(29);
     __buffer = __itoa::__append10(__buffer, static_cast<uint64_t>(__value / __itoa::__pow_10(19)));
     __value %= __itoa::__pow_10(19);
-  }
-  else {
+  } else {
     // step 2
     // This version needs to determine the position of the leading non-zero digit.
     __buffer = __base_10_u64(__buffer, static_cast<uint64_t>(__value / __itoa::__pow_10(19)));
diff --git a/libcxx/include/__charconv/to_chars_floating_point.h b/libcxx/include/__charconv/to_chars_floating_point.h
new file mode 100644
index 0000000000000..6ede36e88dff0
--- /dev/null
+++ b/libcxx/include/__charconv/to_chars_floating_point.h
@@ -0,0 +1,56 @@
+// -*- C++ -*-
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _LIBCPP___CHARCONV_TO_CHARS_FLOATING_POINT_H
+#define _LIBCPP___CHARCONV_TO_CHARS_FLOATING_POINT_H
+
+#include <__availability>
+#include <__charconv/chars_format.h>
+#include <__charconv/to_chars_result.h>
+#include <__config>
+
+#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
+#  pragma GCC system_header
+#endif
+
+_LIBCPP_BEGIN_NAMESPACE_STD
+
+#if _LIBCPP_STD_VER >= 17
+
+_LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT _LIBCPP_FUNC_VIS to_chars_result
+to_chars(char* __first, char* __last, float __value);
+
+_LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT _LIBCPP_FUNC_VIS to_chars_result
+to_chars(char* __first, char* __last, double __value);
+
+_LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT _LIBCPP_FUNC_VIS to_chars_result
+to_chars(char* __first, char* __last, long double __value);
+
+_LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT _LIBCPP_FUNC_VIS to_chars_result
+to_chars(char* __first, char* __last, float __value, chars_format __fmt);
+
+_LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT _LIBCPP_FUNC_VIS to_chars_result
+to_chars(char* __first, char* __last, double __value, chars_format __fmt);
+
+_LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT _LIBCPP_FUNC_VIS to_chars_result
+to_chars(char* __first, char* __last, long double __value, chars_format __fmt);
+
+_LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT _LIBCPP_FUNC_VIS to_chars_result
+to_chars(char* __first, char* __last, float __value, chars_format __fmt, int __precision);
+
+_LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT _LIBCPP_FUNC_VIS to_chars_result
+to_chars(char* __first, char* __last, double __value, chars_format __fmt, int __precision);
+
+_LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT _LIBCPP_FUNC_VIS to_chars_result
+to_chars(char* __first, char* __last, long double __value, chars_format __fmt, int __precision);
+#endif // _LIBCPP_STD_VER >= 17
+
+_LIBCPP_END_NAMESPACE_STD
+
+#endif // _LIBCPP___CHARCONV_TO_CHARS_FLOATING_POINT_H
diff --git a/libcxx/include/__charconv/to_chars_integral.h b/libcxx/include/__charconv/to_chars_integral.h
new file mode 100644
index 0000000000000..32205e25c798e
--- /dev/null
+++ b/libcxx/include/__charconv/to_chars_integral.h
@@ -0,0 +1,326 @@
+// -*- C++ -*-
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _LIBCPP___CHARCONV_TO_CHARS_INTEGRAL_H
+#define _LIBCPP___CHARCONV_TO_CHARS_INTEGRAL_H
+
+#include <__algorithm/copy_n.h>
+#include <__bit/countl.h>
+#include <__charconv/tables.h>
+#include <__charconv/to_chars_base_10.h>
+#include <__charconv/to_chars_result.h>
+#include <__charconv/traits.h>
+#include <__config>
+#include <__errc>
+#include <__type_traits/enable_if.h>
+#include <__type_traits/integral_constant.h>
+#include <__type_traits/is_same.h>
+#include <__type_traits/make_32_64_or_128_bit.h>
+#include <__type_traits/make_unsigned.h>
+#include <__utility/unreachable.h>
+#include <cstddef>
+#include <cstdint>
+#include <limits>
+
+#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
+#  pragma GCC system_header
+#endif
+
+_LIBCPP_PUSH_MACROS
+#include <__undef_macros>
+
+_LIBCPP_BEGIN_NAMESPACE_STD
+
+#if _LIBCPP_STD_VER >= 17
+
+to_chars_result to_chars(char*, char*, bool, int = 10) = delete;
+
+template <typename _Tp>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
+__to_chars_itoa(char* __first, char* __last, _Tp __value, false_type);
+
+template <typename _Tp>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
+__to_chars_itoa(char* __first, char* __last, _Tp __value, true_type) {
+  auto __x = std::__to_unsigned_like(__value);
+  if (__value < 0 && __first != __last) {
+    *__first++ = '-';
+    __x        = std::__complement(__x);
+  }
+
+  return std::__to_chars_itoa(__first, __last, __x, false_type());
+}
+
+template <typename _Tp>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
+__to_chars_itoa(char* __first, char* __last, _Tp __value, false_type) {
+  using __tx  = __itoa::__traits<_Tp>;
+  auto __diff = __last - __first;
+
+  if (__tx::digits <= __diff || __tx::__width(__value) <= __diff)
+    return {__tx::__convert(__first, __value), errc(0)};
+  else
+    return {__last, errc::value_too_large};
+}
+
+#  ifndef _LIBCPP_HAS_NO_INT128
+template <>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
+__to_chars_itoa(char* __first, char* __last, __uint128_t __value, false_type) {
+  // When the value fits in 64-bits use the 64-bit code path. This reduces
+  // the number of expensive calculations on 128-bit values.
+  //
+  // NOTE the 128-bit code path requires this optimization.
+  if (__value <= numeric_limits<uint64_t>::max())
+    return __to_chars_itoa(__first, __last, static_cast<uint64_t>(__value), false_type());
+
+  using __tx  = __itoa::__traits<__uint128_t>;
+  auto __diff = __last - __first;
+
+  if (__tx::digits <= __diff || __tx::__width(__value) <= __diff)
+    return {__tx::__convert(__first, __value), errc(0)};
+  else
+    return {__last, errc::value_too_large};
+}
+#  endif
+
+template <class _Tp>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
+__to_chars_integral(char* __first, char* __last, _Tp __value, int __base, false_type);
+
+template <typename _Tp>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
+__to_chars_integral(char* __first, char* __last, _Tp __value, int __base, true_type) {
+  auto __x = std::__to_unsigned_like(__value);
+  if (__value < 0 && __first != __last) {
+    *__first++ = '-';
+    __x        = std::__complement(__x);
+  }
+
+  return std::__to_chars_integral(__first, __last, __x, __base, false_type());
+}
+
+namespace __itoa {
+
+template <unsigned _Base>
+struct _LIBCPP_HIDDEN __integral;
+
+template <>
+struct _LIBCPP_HIDDEN __integral<2> {
+  template <typename _Tp>
+  _LIBCPP_HIDE_FROM_ABI static constexpr int __width(_Tp __value) noexcept {
+    // If value == 0 still need one digit. If the value != this has no
+    // effect since the code scans for the most significant bit set. (Note
+    // that __libcpp_clz doesn't work for 0.)
+    return numeric_limits<_Tp>::digits - std::__libcpp_clz(__value | 1);
+  }
+
+  template <typename _Tp>
+  _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI static to_chars_result
+  __to_chars(char* __first, char* __last, _Tp __value) {
+    ptrdiff_t __cap = __last - __first;
+    int __n         = __width(__value);
+    if (__n > __cap)
+      return {__last, errc::value_too_large};
+
+    __last                   = __first + __n;
+    char* __p                = __last;
+    const unsigned __divisor = 16;
+    while (__value > __divisor) {
+      unsigned __c = __value % __divisor;
+      __value /= __divisor;
+      __p -= 4;
+      std::copy_n(&__base_2_lut[4 * __c], 4, __p);
+    }
+    do {
+      unsigned __c = __value % 2;
+      __value /= 2;
+      *--__p = "01"[__c];
+    } while (__value != 0);
+    return {__last, errc(0)};
+  }
+};
+
+template <>
+struct _LIBCPP_HIDDEN __integral<8> {
+  template <typename _Tp>
+  _LIBCPP_HIDE_FROM_ABI static constexpr int __width(_Tp __value) noexcept {
+    // If value == 0 still need one digit. If the value != this has no
+    // effect since the code scans for the most significat bit set. (Note
+    // that __libcpp_clz doesn't work for 0.)
+    return ((numeric_limits<_Tp>::digits - std::__libcpp_clz(__value | 1)) + 2) / 3;
+  }
+
+  template <typename _Tp>
+  _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI static to_chars_result
+  __to_chars(char* __first, char* __last, _Tp __value) {
+    ptrdiff_t __cap = __last - __first;
+    int __n         = __width(__value);
+    if (__n > __cap)
+      return {__last, errc::value_too_large};
+
+    __last             = __first + __n;
+    char* __p          = __last;
+    unsigned __divisor = 64;
+    while (__value > __divisor) {
+      unsigned __c = __value % __divisor;
+      __value /= __divisor;
+      __p -= 2;
+      std::copy_n(&__base_8_lut[2 * __c], 2, __p);
+    }
+    do {
+      unsigned __c = __value % 8;
+      __value /= 8;
+      *--__p = "01234567"[__c];
+    } while (__value != 0);
+    return {__last, errc(0)};
+  }
+};
+
+template <>
+struct _LIBCPP_HIDDEN __integral<16> {
+  template <typename _Tp>
+  _LIBCPP_HIDE_FROM_ABI static constexpr int __width(_Tp __value) noexcept {
+    // If value == 0 still need one digit. If the value != this has no
+    // effect since the code scans for the most significat bit set. (Note
+    // that __libcpp_clz doesn't work for 0.)
+    return (numeric_limits<_Tp>::digits - std::__libcpp_clz(__value | 1) + 3) / 4;
+  }
+
+  template <typename _Tp>
+  _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI static to_chars_result
+  __to_chars(char* __first, char* __last, _Tp __value) {
+    ptrdiff_t __cap = __last - __first;
+    int __n         = __width(__value);
+    if (__n > __cap)
+      return {__last, errc::value_too_large};
+
+    __last             = __first + __n;
+    char* __p          = __last;
+    unsigned __divisor = 256;
+    while (__value > __divisor) {
+      unsigned __c = __value % __divisor;
+      __value /= __divisor;
+      __p -= 2;
+      std::copy_n(&__base_16_lut[2 * __c], 2, __p);
+    }
+    if (__first != __last)
+      do {
+        unsigned __c = __value % 16;
+        __value /= 16;
+        *--__p = "0123456789abcdef"[__c];
+      } while (__value != 0);
+    return {__last, errc(0)};
+  }
+};
+
+} // namespace __itoa
+
+template <unsigned _Base, typename _Tp, typename enable_if<(sizeof(_Tp) >= sizeof(unsigned)), int>::type = 0>
+_LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI int __to_chars_integral_width(_Tp __value) {
+  return __itoa::__integral<_Base>::__width(__value);
+}
+
+template <unsigned _Base, typename _Tp, typename enable_if<(sizeof(_Tp) < sizeof(unsigned)), int>::type = 0>
+_LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI int __to_chars_integral_width(_Tp __value) {
+  return std::__to_chars_integral_width<_Base>(static_cast<unsigned>(__value));
+}
+
+template <unsigned _Base, typename _Tp, typename enable_if<(sizeof(_Tp) >= sizeof(unsigned)), int>::type = 0>
+_LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
+__to_chars_integral(char* __first, char* __last, _Tp __value) {
+  return __itoa::__integral<_Base>::__to_chars(__first, __last, __value);
+}
+
+template <unsigned _Base, typename _Tp, typename enable_if<(sizeof(_Tp) < sizeof(unsigned)), int>::type = 0>
+_LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
+__to_chars_integral(char* __first, char* __last, _Tp __value) {
+  return std::__to_chars_integral<_Base>(__first, __last, static_cast<unsigned>(__value));
+}
+
+template <typename _Tp>
+_LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI int __to_chars_integral_width(_Tp __value, unsigned __base) {
+  _LIBCPP_ASSERT(__value >= 0, "The function requires a non-negative value.");
+
+  unsigned __base_2 = __base * __base;
+  unsigned __base_3 = __base_2 * __base;
+  unsigned __base_4 = __base_2 * __base_2;
+
+  int __r = 0;
+  while (true) {
+    if (__value < __base)
+      return __r + 1;
+    if (__value < __base_2)
+      return __r + 2;
+    if (__value < __base_3)
+      return __r + 3;
+    if (__value < __base_4)
+      return __r + 4;
+
+    __value /= __base_4;
+    __r += 4;
+  }
+
+  __libcpp_unreachable();
+}
+
+template <typename _Tp>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
+__to_chars_integral(char* __first, char* __last, _Tp __value, int __base, false_type) {
+  if (__base == 10) [[likely]]
+    return std::__to_chars_itoa(__first, __last, __value, false_type());
+
+  switch (__base) {
+  case 2:
+    return std::__to_chars_integral<2>(__first, __last, __value);
+  case 8:
+    return std::__to_chars_integral<8>(__first, __last, __value);
+  case 16:
+    return std::__to_chars_integral<16>(__first, __last, __value);
+  }
+
+  ptrdiff_t __cap = __last - __first;
+  int __n         = std::__to_chars_integral_width(__value, __base);
+  if (__n > __cap)
+    return {__last, errc::value_too_large};
+
+  __last    = __first + __n;
+  char* __p = __last;
+  do {
+    unsigned __c = __value % __base;
+    __value /= __base;
+    *--__p = "0123456789abcdefghijklmnopqrstuvwxyz"[__c];
+  } while (__value != 0);
+  return {__last, errc(0)};
+}
+
+template <typename _Tp, typename enable_if<is_integral<_Tp>::value, int>::type = 0>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
+to_chars(char* __first, char* __last, _Tp __value) {
+  using _Type = __make_32_64_or_128_bit_t<_Tp>;
+  static_assert(!is_same<_Type, void>::value, "unsupported integral type used in to_chars");
+  return std::__to_chars_itoa(__first, __last, static_cast<_Type>(__value), is_signed<_Tp>());
+}
+
+template <typename _Tp, typename enable_if<is_integral<_Tp>::value, int>::type = 0>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
+to_chars(char* __first, char* __last, _Tp __value, int __base) {
+  _LIBCPP_ASSERT(2 <= __base && __base <= 36, "base not in [2, 36]");
+
+  using _Type = __make_32_64_or_128_bit_t<_Tp>;
+  return std::__to_chars_integral(__first, __last, static_cast<_Type>(__value), __base, is_signed<_Tp>());
+}
+
+#endif // _LIBCPP_STD_VER >= 17
+
+_LIBCPP_END_NAMESPACE_STD
+
+_LIBCPP_POP_MACROS
+
+#endif // _LIBCPP___CHARCONV_TO_CHARS_INTEGRAL_H
diff --git a/libcxx/include/__charconv/to_chars_result.h b/libcxx/include/__charconv/to_chars_result.h
index 4a39213b2372e..ef74f20b9d9ab 100644
--- a/libcxx/include/__charconv/to_chars_result.h
+++ b/libcxx/include/__charconv/to_chars_result.h
@@ -21,12 +21,11 @@ _LIBCPP_BEGIN_NAMESPACE_STD
 
 #if _LIBCPP_STD_VER >= 17
 
-struct _LIBCPP_TYPE_VIS to_chars_result
-{
-    char* ptr;
-    errc ec;
+struct _LIBCPP_TYPE_VIS to_chars_result {
+  char* ptr;
+  errc ec;
 #  if _LIBCPP_STD_VER >= 20
-    _LIBCPP_HIDE_FROM_ABI friend bool operator==(const to_chars_result&, const to_chars_result&) = default;
+  _LIBCPP_HIDE_FROM_ABI friend bool operator==(const to_chars_result&, const to_chars_result&) = default;
 #  endif
 };
 
diff --git a/libcxx/include/__charconv/traits.h b/libcxx/include/__charconv/traits.h
new file mode 100644
index 0000000000000..505a0b1ac1429
--- /dev/null
+++ b/libcxx/include/__charconv/traits.h
@@ -0,0 +1,199 @@
+// -*- C++ -*-
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _LIBCPP___CHARCONV_TRAITS
+#define _LIBCPP___CHARCONV_TRAITS
+
+#include <__bit/countl.h>
+#include <__charconv/tables.h>
+#include <__charconv/to_chars_base_10.h>
+#include <__config>
+#include <__type_traits/enable_if.h>
+#include <__type_traits/is_unsigned.h>
+#include <cstdint>
+#include <limits>
+
+#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
+#  pragma GCC system_header
+#endif
+
+_LIBCPP_PUSH_MACROS
+#include <__undef_macros>
+
+_LIBCPP_BEGIN_NAMESPACE_STD
+
+#if _LIBCPP_STD_VER >= 17
+
+namespace __itoa {
+
+template <typename _Tp, typename = void>
+struct _LIBCPP_HIDDEN __traits_base;
+
+template <typename _Tp>
+struct _LIBCPP_HIDDEN __traits_base<_Tp, __enable_if_t<sizeof(_Tp) <= sizeof(uint32_t)>> {
+  using type = uint32_t;
+
+  /// The width estimation using a log10 algorithm.
+  ///
+  /// The algorithm is based on
+  /// http://graphics.stanford.edu/~seander/bithacks.html#IntegerLog10
+  /// Instead of using IntegerLogBase2 it uses __libcpp_clz. Since that
+  /// function requires its input to have at least one bit set the value of
+  /// zero is set to one. This means the first element of the lookup table is
+  /// zero.
+  static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI int __width(_Tp __v) {
+    auto __t = (32 - std::__libcpp_clz(static_cast<type>(__v | 1))) * 1233 >> 12;
+    return __t - (__v < __itoa::__pow10_32[__t]) + 1;
+  }
+
+  static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI char* __convert(char* __p, _Tp __v) {
+    return __itoa::__base_10_u32(__p, __v);
+  }
+
+  static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI decltype(__pow10_32)& __pow() {
+    return __itoa::__pow10_32;
+  }
+};
+
+template <typename _Tp>
+struct _LIBCPP_HIDDEN __traits_base<_Tp, __enable_if_t<sizeof(_Tp) == sizeof(uint64_t)>> {
+  using type = uint64_t;
+
+  /// The width estimation using a log10 algorithm.
+  ///
+  /// The algorithm is based on
+  /// http://graphics.stanford.edu/~seander/bithacks.html#IntegerLog10
+  /// Instead of using IntegerLogBase2 it uses __libcpp_clz. Since that
+  /// function requires its input to have at least one bit set the value of
+  /// zero is set to one. This means the first element of the lookup table is
+  /// zero.
+  static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI int __width(_Tp __v) {
+    auto __t = (64 - std::__libcpp_clz(static_cast<type>(__v | 1))) * 1233 >> 12;
+    return __t - (__v < __itoa::__pow10_64[__t]) + 1;
+  }
+
+  static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI char* __convert(char* __p, _Tp __v) {
+    return __itoa::__base_10_u64(__p, __v);
+  }
+
+  static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI decltype(__pow10_64)& __pow() {
+    return __itoa::__pow10_64;
+  }
+};
+
+#  ifndef _LIBCPP_HAS_NO_INT128
+template <typename _Tp>
+struct _LIBCPP_HIDDEN __traits_base<_Tp, __enable_if_t<sizeof(_Tp) == sizeof(__uint128_t)> > {
+  using type = __uint128_t;
+
+  /// The width estimation using a log10 algorithm.
+  ///
+  /// The algorithm is based on
+  /// http://graphics.stanford.edu/~seander/bithacks.html#IntegerLog10
+  /// Instead of using IntegerLogBase2 it uses __libcpp_clz. Since that
+  /// function requires its input to have at least one bit set the value of
+  /// zero is set to one. This means the first element of the lookup table is
+  /// zero.
+  static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI int __width(_Tp __v) {
+    _LIBCPP_ASSERT(
+        __v > numeric_limits<uint64_t>::max(), "The optimizations for this algorithm fail when this isn't true.");
+    // There's always a bit set in the upper 64-bits.
+    auto __t = (128 - std::__libcpp_clz(static_cast<uint64_t>(__v >> 64))) * 1233 >> 12;
+    _LIBCPP_ASSERT(__t >= __itoa::__pow10_128_offset, "Index out of bounds");
+    // __t is adjusted since the lookup table misses the lower entries.
+    return __t - (__v < __itoa::__pow10_128[__t - __itoa::__pow10_128_offset]) + 1;
+  }
+
+  static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI char* __convert(char* __p, _Tp __v) {
+    return __itoa::__base_10_u128(__p, __v);
+  }
+
+  // TODO FMT This pow function should get an index.
+  // By moving this to its own header it can be reused by the pow function in to_chars_base_10.
+  static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI decltype(__pow10_128)& __pow() {
+    return __itoa::__pow10_128;
+  }
+};
+#  endif
+
+template <typename _Tp>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI bool
+__mul_overflowed(unsigned char __a, _Tp __b, unsigned char& __r) {
+  auto __c = __a * __b;
+  __r      = __c;
+  return __c > numeric_limits<unsigned char>::max();
+}
+
+template <typename _Tp>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI bool
+__mul_overflowed(unsigned short __a, _Tp __b, unsigned short& __r) {
+  auto __c = __a * __b;
+  __r      = __c;
+  return __c > numeric_limits<unsigned short>::max();
+}
+
+template <typename _Tp>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI bool __mul_overflowed(_Tp __a, _Tp __b, _Tp& __r) {
+  static_assert(is_unsigned<_Tp>::value, "");
+  return __builtin_mul_overflow(__a, __b, &__r);
+}
+
+template <typename _Tp, typename _Up>
+inline _LIBCPP_HIDE_FROM_ABI bool _LIBCPP_CONSTEXPR_SINCE_CXX23 __mul_overflowed(_Tp __a, _Up __b, _Tp& __r) {
+  return __itoa::__mul_overflowed(__a, static_cast<_Tp>(__b), __r);
+}
+
+template <typename _Tp>
+struct _LIBCPP_HIDDEN __traits : __traits_base<_Tp> {
+  static constexpr int digits = numeric_limits<_Tp>::digits10 + 1;
+  using __traits_base<_Tp>::__pow;
+  using typename __traits_base<_Tp>::type;
+
+  // precondition: at least one non-zero character available
+  static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI char const*
+  __read(char const* __p, char const* __ep, type& __a, type& __b) {
+    type __cprod[digits];
+    int __j = digits - 1;
+    int __i = digits;
+    do {
+      if (*__p < '0' || *__p > '9')
+        break;
+      __cprod[--__i] = *__p++ - '0';
+    } while (__p != __ep && __i != 0);
+
+    __a = __inner_product(__cprod + __i + 1, __cprod + __j, __pow() + 1, __cprod[__i]);
+    if (__itoa::__mul_overflowed(__cprod[__j], __pow()[__j - __i], __b))
+      --__p;
+    return __p;
+  }
+
+  template <typename _It1, typename _It2, class _Up>
+  static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI _Up
+  __inner_product(_It1 __first1, _It1 __last1, _It2 __first2, _Up __init) {
+    for (; __first1 < __last1; ++__first1, ++__first2)
+      __init = __init + *__first1 * *__first2;
+    return __init;
+  }
+};
+
+} // namespace __itoa
+
+template <typename _Tp>
+inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI _Tp __complement(_Tp __x) {
+  static_assert(is_unsigned<_Tp>::value, "cast to unsigned first");
+  return _Tp(~__x + 1);
+}
+
+#endif // _LIBCPP_STD_VER >= 17
+
+_LIBCPP_END_NAMESPACE_STD
+
+_LIBCPP_POP_MACROS
+
+#endif // _LIBCPP___CHARCONV_TRAITS
diff --git a/libcxx/include/__chrono/concepts.h b/libcxx/include/__chrono/concepts.h
index d7502ef9f84dc..38d6b0ccd538c 100644
--- a/libcxx/include/__chrono/concepts.h
+++ b/libcxx/include/__chrono/concepts.h
@@ -20,12 +20,12 @@
 
 _LIBCPP_BEGIN_NAMESPACE_STD
 
-#if _LIBCPP_STD_VER > 17
+#if _LIBCPP_STD_VER >= 20
 
 template <class _Tp>
 concept __is_hh_mm_ss = __is_specialization_v<_Tp, chrono::hh_mm_ss>;
 
-#endif // _LIBCPP_STD_VER > 17
+#endif // _LIBCPP_STD_VER >= 20
 
 _LIBCPP_END_NAMESPACE_STD
 
diff --git a/libcxx/include/__chrono/hh_mm_ss.h b/libcxx/include/__chrono/hh_mm_ss.h
index dd1c94ee66707..5bd452e57fa3c 100644
--- a/libcxx/include/__chrono/hh_mm_ss.h
+++ b/libcxx/include/__chrono/hh_mm_ss.h
@@ -13,8 +13,8 @@
 #include <__chrono/duration.h>
 #include <__chrono/time_point.h>
 #include <__config>
+#include <__type_traits/common_type.h>
 #include <ratio>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__compare/ordering.h b/libcxx/include/__compare/ordering.h
index c87871decc080..c348f0433a321 100644
--- a/libcxx/include/__compare/ordering.h
+++ b/libcxx/include/__compare/ordering.h
@@ -40,7 +40,7 @@ template<class _Tp, class... _Args>
 inline constexpr bool __one_of_v = (is_same_v<_Tp, _Args> || ...);
 
 struct _CmpUnspecifiedParam {
-  _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEVAL
+  _LIBCPP_HIDE_FROM_ABI constexpr
   _CmpUnspecifiedParam(int _CmpUnspecifiedParam::*) noexcept {}
 
   template<class _Tp, class = enable_if_t<!__one_of_v<_Tp, int, partial_ordering, weak_ordering, strong_ordering>>>
diff --git a/libcxx/include/__config b/libcxx/include/__config
index f531b1c057f84..3d3664eb83ae7 100644
--- a/libcxx/include/__config
+++ b/libcxx/include/__config
@@ -50,6 +50,7 @@
 #    define _LIBCPP_FREESTANDING
 #  endif
 
+// NOLINTBEGIN(libcpp-cpp-version-check)
 #  ifndef _LIBCPP_STD_VER
 #    if __cplusplus <= 201103L
 #      define _LIBCPP_STD_VER 11
@@ -64,6 +65,7 @@
 #      define _LIBCPP_STD_VER 23
 #    endif
 #  endif // _LIBCPP_STD_VER
+// NOLINTEND(libcpp-cpp-version-check)
 
 #  if defined(__ELF__)
 #    define _LIBCPP_OBJECT_FORMAT_ELF 1
@@ -164,6 +166,12 @@
 #    if defined(__FreeBSD__)
 #      define _LIBCPP_DEPRECATED_ABI_DISABLE_PAIR_TRIVIAL_COPY_CTOR
 #    endif
+// For XCOFF linkers, we have problems if we see a weak hidden version of a symbol
+// in user code (like you get with -fvisibility-inlines-hidden) and then a strong def
+// in the library, so we need to always rely on the library version.
+#    if defined(_AIX)
+#      define _LIBCPP_ABI_BAD_FUNCTION_CALL_KEY_FUNCTION
+#    endif
 #  endif
 
 #  if defined(_LIBCPP_BUILDING_LIBRARY) || _LIBCPP_ABI_VERSION >= 2
@@ -181,6 +189,7 @@
 #  define _LIBCPP_TOSTRING2(x) #x
 #  define _LIBCPP_TOSTRING(x) _LIBCPP_TOSTRING2(x)
 
+// NOLINTNEXTLINE(libcpp-cpp-version-check)
 #  if __cplusplus < 201103L
 #    define _LIBCPP_CXX03_LANG
 #  endif
@@ -686,12 +695,6 @@ _LIBCPP_BEGIN_NAMESPACE_STD _LIBCPP_END_NAMESPACE_STD
 #    define _LIBCPP_HAS_NO_INT128
 #  endif
 
-#  ifndef __cpp_consteval
-#    define _LIBCPP_CONSTEVAL _LIBCPP_CONSTEXPR
-#  else
-#    define _LIBCPP_CONSTEVAL consteval
-#  endif
-
 #  if __has_attribute(__malloc__)
 #    define _LIBCPP_NOALIAS __attribute__((__malloc__))
 #  else
@@ -1251,13 +1254,19 @@ _LIBCPP_BEGIN_NAMESPACE_STD _LIBCPP_END_NAMESPACE_STD
 // macro is used to mark them as such, which suppresses the
 // '-Wctad-maybe-unsupported' compiler warning when CTAD is used in user code
 // with these classes.
-#if _LIBCPP_STD_VER >= 17
-#    define _LIBCPP_CTAD_SUPPORTED_FOR_TYPE(_ClassName)                                                                \
-      template <class ..._Tag>                                                                                         \
-      _ClassName(typename _Tag::__allow_ctad...) -> _ClassName<_Tag...>
-#else
-#  define _LIBCPP_CTAD_SUPPORTED_FOR_TYPE(_ClassName) static_assert(true, "")
-#endif
+#  if _LIBCPP_STD_VER >= 17
+#    ifdef _LIBCPP_COMPILER_CLANG_BASED
+#      define _LIBCPP_CTAD_SUPPORTED_FOR_TYPE(_ClassName)                                                              \
+        template <class... _Tag>                                                                                       \
+        [[maybe_unused]] _ClassName(typename _Tag::__allow_ctad...)->_ClassName<_Tag...>
+#    else
+#      define _LIBCPP_CTAD_SUPPORTED_FOR_TYPE(ClassName)                                                               \
+        template <class... _Tag>                                                                                       \
+        ClassName(typename _Tag::__allow_ctad...)->ClassName<_Tag...>
+#    endif
+#  else
+#    define _LIBCPP_CTAD_SUPPORTED_FOR_TYPE(_ClassName) static_assert(true, "")
+#  endif
 
 // TODO(varconst): currently, there are bugs in Clang's intrinsics when handling Objective-C++ `id`, so don't use
 // compiler intrinsics in the Objective-C++ mode.
diff --git a/libcxx/include/__exception/exception.h b/libcxx/include/__exception/exception.h
new file mode 100644
index 0000000000000..49a58dd16999a
--- /dev/null
+++ b/libcxx/include/__exception/exception.h
@@ -0,0 +1,91 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _LIBCPP___EXCEPTION_EXCEPTION_H
+#define _LIBCPP___EXCEPTION_EXCEPTION_H
+
+#include <__config>
+
+// <vcruntime_exception.h> defines its own std::exception and std::bad_exception types,
+// which we use in order to be ABI-compatible with other STLs on Windows.
+#if defined(_LIBCPP_ABI_VCRUNTIME)
+#  include <vcruntime_exception.h>
+#endif
+
+#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
+#  pragma GCC system_header
+#endif
+
+namespace std { // purposefully not using versioning namespace
+
+#if defined(_LIBCPP_ABI_VCRUNTIME) && (!defined(_HAS_EXCEPTIONS) || _HAS_EXCEPTIONS != 0)
+// The std::exception class was already included above, but we're explicit about this condition here for clarity.
+
+#elif defined(_LIBCPP_ABI_VCRUNTIME) && _HAS_EXCEPTIONS == 0
+// However, <vcruntime_exception.h> does not define std::exception and std::bad_exception
+// when _HAS_EXCEPTIONS == 0.
+//
+// Since libc++ still wants to provide the std::exception hierarchy even when _HAS_EXCEPTIONS == 0
+// (after all those are simply types like any other), we define an ABI-compatible version
+// of the VCRuntime std::exception and std::bad_exception types in that mode.
+
+struct __std_exception_data {
+  char const* _What;
+  bool _DoFree;
+};
+
+class exception { // base of all library exceptions
+public:
+  exception() _NOEXCEPT : __data_() {}
+
+  explicit exception(char const* __message) _NOEXCEPT : __data_() {
+    __data_._What   = __message;
+    __data_._DoFree = true;
+  }
+
+  exception(exception const&) _NOEXCEPT {}
+
+  exception& operator=(exception const&) _NOEXCEPT { return *this; }
+
+  virtual ~exception() _NOEXCEPT {}
+
+  virtual char const* what() const _NOEXCEPT { return __data_._What ? __data_._What : "Unknown exception"; }
+
+private:
+  __std_exception_data __data_;
+};
+
+class bad_exception : public exception {
+public:
+  bad_exception() _NOEXCEPT : exception("bad exception") {}
+};
+
+#else  // !defined(_LIBCPP_ABI_VCRUNTIME)
+// On all other platforms, we define our own std::exception and std::bad_exception types
+// regardless of whether exceptions are turned on as a language feature.
+
+class _LIBCPP_EXCEPTION_ABI exception {
+public:
+  _LIBCPP_HIDE_FROM_ABI exception() _NOEXCEPT {}
+  _LIBCPP_HIDE_FROM_ABI exception(const exception&) _NOEXCEPT = default;
+
+  virtual ~exception() _NOEXCEPT;
+  virtual const char* what() const _NOEXCEPT;
+};
+
+class _LIBCPP_EXCEPTION_ABI bad_exception : public exception {
+public:
+  _LIBCPP_HIDE_FROM_ABI bad_exception() _NOEXCEPT {}
+  ~bad_exception() _NOEXCEPT override;
+  const char* what() const _NOEXCEPT override;
+};
+#endif // !_LIBCPP_ABI_VCRUNTIME
+
+} // namespace std
+
+#endif // _LIBCPP___EXCEPTION_EXCEPTION_H
diff --git a/libcxx/include/__exception/exception_ptr.h b/libcxx/include/__exception/exception_ptr.h
new file mode 100644
index 0000000000000..cac4a887d5a7c
--- /dev/null
+++ b/libcxx/include/__exception/exception_ptr.h
@@ -0,0 +1,109 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _LIBCPP___EXCEPTION_EXCEPTION_PTR_H
+#define _LIBCPP___EXCEPTION_EXCEPTION_PTR_H
+
+#include <__config>
+#include <__exception/operations.h>
+#include <__memory/addressof.h>
+#include <cstddef>
+#include <cstdlib>
+
+#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
+#  pragma GCC system_header
+#endif
+
+namespace std { // purposefully not using versioning namespace
+
+#ifndef _LIBCPP_ABI_MICROSOFT
+
+class _LIBCPP_TYPE_VIS exception_ptr {
+  void* __ptr_;
+
+public:
+  _LIBCPP_HIDE_FROM_ABI exception_ptr() _NOEXCEPT : __ptr_() {}
+  _LIBCPP_HIDE_FROM_ABI exception_ptr(nullptr_t) _NOEXCEPT : __ptr_() {}
+
+  exception_ptr(const exception_ptr&) _NOEXCEPT;
+  exception_ptr& operator=(const exception_ptr&) _NOEXCEPT;
+  ~exception_ptr() _NOEXCEPT;
+
+  _LIBCPP_HIDE_FROM_ABI explicit operator bool() const _NOEXCEPT { return __ptr_ != nullptr; }
+
+  friend _LIBCPP_HIDE_FROM_ABI bool operator==(const exception_ptr& __x, const exception_ptr& __y) _NOEXCEPT {
+    return __x.__ptr_ == __y.__ptr_;
+  }
+
+  friend _LIBCPP_HIDE_FROM_ABI bool operator!=(const exception_ptr& __x, const exception_ptr& __y) _NOEXCEPT {
+    return !(__x == __y);
+  }
+
+  friend _LIBCPP_FUNC_VIS exception_ptr current_exception() _NOEXCEPT;
+  friend _LIBCPP_FUNC_VIS void rethrow_exception(exception_ptr);
+};
+
+template <class _Ep>
+_LIBCPP_HIDE_FROM_ABI exception_ptr make_exception_ptr(_Ep __e) _NOEXCEPT {
+#  ifndef _LIBCPP_HAS_NO_EXCEPTIONS
+  try {
+    throw __e;
+  } catch (...) {
+    return current_exception();
+  }
+#  else
+  ((void)__e);
+  std::abort();
+#  endif
+}
+
+#else  // _LIBCPP_ABI_MICROSOFT
+
+class _LIBCPP_TYPE_VIS exception_ptr {
+  _LIBCPP_DIAGNOSTIC_PUSH
+  _LIBCPP_CLANG_DIAGNOSTIC_IGNORED("-Wunused-private-field")
+  void* __ptr1_;
+  void* __ptr2_;
+  _LIBCPP_DIAGNOSTIC_POP
+
+public:
+  exception_ptr() _NOEXCEPT;
+  exception_ptr(nullptr_t) _NOEXCEPT;
+  exception_ptr(const exception_ptr& __other) _NOEXCEPT;
+  exception_ptr& operator=(const exception_ptr& __other) _NOEXCEPT;
+  exception_ptr& operator=(nullptr_t) _NOEXCEPT;
+  ~exception_ptr() _NOEXCEPT;
+  explicit operator bool() const _NOEXCEPT;
+};
+
+_LIBCPP_FUNC_VIS bool operator==(const exception_ptr& __x, const exception_ptr& __y) _NOEXCEPT;
+
+inline _LIBCPP_HIDE_FROM_ABI bool operator!=(const exception_ptr& __x, const exception_ptr& __y) _NOEXCEPT {
+  return !(__x == __y);
+}
+
+_LIBCPP_FUNC_VIS void swap(exception_ptr&, exception_ptr&) _NOEXCEPT;
+
+_LIBCPP_FUNC_VIS exception_ptr __copy_exception_ptr(void* __except, const void* __ptr);
+_LIBCPP_FUNC_VIS exception_ptr current_exception() _NOEXCEPT;
+_LIBCPP_NORETURN _LIBCPP_FUNC_VIS void rethrow_exception(exception_ptr);
+
+// This is a built-in template function which automagically extracts the required
+// information.
+template <class _E>
+void* __GetExceptionInfo(_E);
+
+template <class _Ep>
+_LIBCPP_HIDE_FROM_ABI exception_ptr make_exception_ptr(_Ep __e) _NOEXCEPT {
+  return __copy_exception_ptr(std::addressof(__e), __GetExceptionInfo(__e));
+}
+
+#endif // _LIBCPP_ABI_MICROSOFT
+} // namespace std
+
+#endif // _LIBCPP___EXCEPTION_EXCEPTION_PTR_H
diff --git a/libcxx/include/__exception/nested_exception.h b/libcxx/include/__exception/nested_exception.h
new file mode 100644
index 0000000000000..b842dde03e16a
--- /dev/null
+++ b/libcxx/include/__exception/nested_exception.h
@@ -0,0 +1,101 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _LIBCPP___EXCEPTION_NESTED_EXCEPTION_H
+#define _LIBCPP___EXCEPTION_NESTED_EXCEPTION_H
+
+#include <__config>
+#include <__exception/exception_ptr.h>
+#include <__memory/addressof.h>
+#include <__type_traits/decay.h>
+#include <__type_traits/is_base_of.h>
+#include <__type_traits/is_class.h>
+#include <__type_traits/is_convertible.h>
+#include <__type_traits/is_copy_constructible.h>
+#include <__type_traits/is_final.h>
+#include <__type_traits/is_polymorphic.h>
+#include <__utility/forward.h>
+#include <cstddef>
+
+#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
+#  pragma GCC system_header
+#endif
+
+namespace std { // purposefully not using versioning namespace
+
+class _LIBCPP_EXCEPTION_ABI nested_exception {
+  exception_ptr __ptr_;
+
+public:
+  nested_exception() _NOEXCEPT;
+  //     nested_exception(const nested_exception&) noexcept = default;
+  //     nested_exception& operator=(const nested_exception&) noexcept = default;
+  virtual ~nested_exception() _NOEXCEPT;
+
+  // access functions
+  _LIBCPP_NORETURN void rethrow_nested() const;
+  _LIBCPP_HIDE_FROM_ABI exception_ptr nested_ptr() const _NOEXCEPT { return __ptr_; }
+};
+
+template <class _Tp>
+struct __nested : public _Tp, public nested_exception {
+  _LIBCPP_HIDE_FROM_ABI explicit __nested(const _Tp& __t) : _Tp(__t) {}
+};
+
+#ifndef _LIBCPP_HAS_NO_EXCEPTIONS
+template <class _Tp, class _Up, bool>
+struct __throw_with_nested;
+
+template <class _Tp, class _Up>
+struct __throw_with_nested<_Tp, _Up, true> {
+  _LIBCPP_NORETURN static inline _LIBCPP_INLINE_VISIBILITY void __do_throw(_Tp&& __t) {
+    throw __nested<_Up>(std::forward<_Tp>(__t));
+  }
+};
+
+template <class _Tp, class _Up>
+struct __throw_with_nested<_Tp, _Up, false> {
+  _LIBCPP_NORETURN static inline _LIBCPP_INLINE_VISIBILITY void __do_throw(_Tp&& __t) { throw std::forward<_Tp>(__t); }
+};
+#endif
+
+template <class _Tp>
+_LIBCPP_NORETURN _LIBCPP_HIDE_FROM_ABI void throw_with_nested(_Tp&& __t) {
+#ifndef _LIBCPP_HAS_NO_EXCEPTIONS
+  typedef typename decay<_Tp>::type _Up;
+  static_assert(is_copy_constructible<_Up>::value, "type thrown must be CopyConstructible");
+  __throw_with_nested<_Tp,
+                      _Up,
+                      is_class<_Up>::value && !is_base_of<nested_exception, _Up>::value &&
+                          !__libcpp_is_final<_Up>::value>::__do_throw(std::forward<_Tp>(__t));
+#else
+  ((void)__t);
+  // FIXME: Make this abort
+#endif
+}
+
+template <class _From, class _To>
+struct __can_dynamic_cast
+    : _BoolConstant< is_polymorphic<_From>::value &&
+                     (!is_base_of<_To, _From>::value || is_convertible<const _From*, const _To*>::value)> {};
+
+template <class _Ep>
+inline _LIBCPP_HIDE_FROM_ABI void
+rethrow_if_nested(const _Ep& __e, __enable_if_t< __can_dynamic_cast<_Ep, nested_exception>::value>* = 0) {
+  const nested_exception* __nep = dynamic_cast<const nested_exception*>(std::addressof(__e));
+  if (__nep)
+    __nep->rethrow_nested();
+}
+
+template <class _Ep>
+inline _LIBCPP_HIDE_FROM_ABI void
+rethrow_if_nested(const _Ep&, __enable_if_t<!__can_dynamic_cast<_Ep, nested_exception>::value>* = 0) {}
+
+} // namespace std
+
+#endif // _LIBCPP___EXCEPTION_NESTED_EXCEPTION_H
diff --git a/libcxx/include/__exception/operations.h b/libcxx/include/__exception/operations.h
new file mode 100644
index 0000000000000..e8c5ba61ee0e2
--- /dev/null
+++ b/libcxx/include/__exception/operations.h
@@ -0,0 +1,42 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _LIBCPP___EXCEPTION_OPERATIONS_H
+#define _LIBCPP___EXCEPTION_OPERATIONS_H
+
+#include <__availability>
+#include <__config>
+#include <cstddef>
+
+#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
+#  pragma GCC system_header
+#endif
+
+namespace std { // purposefully not using versioning namespace
+#if _LIBCPP_STD_VER <= 14 || defined(_LIBCPP_ENABLE_CXX17_REMOVED_UNEXPECTED_FUNCTIONS) ||                             \
+    defined(_LIBCPP_BUILDING_LIBRARY)
+using unexpected_handler = void (*)();
+_LIBCPP_FUNC_VIS unexpected_handler set_unexpected(unexpected_handler) _NOEXCEPT;
+_LIBCPP_FUNC_VIS unexpected_handler get_unexpected() _NOEXCEPT;
+_LIBCPP_NORETURN _LIBCPP_FUNC_VIS void unexpected();
+#endif
+
+using terminate_handler = void (*)();
+_LIBCPP_FUNC_VIS terminate_handler set_terminate(terminate_handler) _NOEXCEPT;
+_LIBCPP_FUNC_VIS terminate_handler get_terminate() _NOEXCEPT;
+
+_LIBCPP_FUNC_VIS bool uncaught_exception() _NOEXCEPT;
+_LIBCPP_FUNC_VIS _LIBCPP_AVAILABILITY_UNCAUGHT_EXCEPTIONS int uncaught_exceptions() _NOEXCEPT;
+
+class _LIBCPP_TYPE_VIS exception_ptr;
+
+_LIBCPP_FUNC_VIS exception_ptr current_exception() _NOEXCEPT;
+_LIBCPP_NORETURN _LIBCPP_FUNC_VIS void rethrow_exception(exception_ptr);
+} // namespace std
+
+#endif // _LIBCPP___EXCEPTION_OPERATIONS_H
diff --git a/libcxx/include/__exception/terminate.h b/libcxx/include/__exception/terminate.h
new file mode 100644
index 0000000000000..d8dd9642b88d1
--- /dev/null
+++ b/libcxx/include/__exception/terminate.h
@@ -0,0 +1,22 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _LIBCPP___EXCEPTION_TERMINATE_H
+#define _LIBCPP___EXCEPTION_TERMINATE_H
+
+#include <__config>
+
+#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
+#  pragma GCC system_header
+#endif
+
+namespace std { // purposefully not using versioning namespace
+_LIBCPP_NORETURN _LIBCPP_FUNC_VIS void terminate() _NOEXCEPT;
+} // namespace std
+
+#endif // _LIBCPP___EXCEPTION_TERMINATE_H
diff --git a/libcxx/include/__expected/bad_expected_access.h b/libcxx/include/__expected/bad_expected_access.h
index 361eab4b60187..e7a568c88f01c 100644
--- a/libcxx/include/__expected/bad_expected_access.h
+++ b/libcxx/include/__expected/bad_expected_access.h
@@ -10,10 +10,9 @@
 #define _LIBCPP___EXPECTED_BAD_EXPECTED_ACCESS_H
 
 #include <__config>
+#include <__exception/exception.h>
 #include <__utility/move.h>
 
-#include <exception>
-
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
 #endif
diff --git a/libcxx/include/__expected/expected.h b/libcxx/include/__expected/expected.h
index 8bb04071b1230..9a8d4047cdefa 100644
--- a/libcxx/include/__expected/expected.h
+++ b/libcxx/include/__expected/expected.h
@@ -49,6 +49,7 @@
 #include <__utility/in_place.h>
 #include <__utility/move.h>
 #include <__utility/swap.h>
+#include <__verbose_abort>
 #include <cstdlib> // for std::abort
 #include <initializer_list>
 
@@ -68,7 +69,7 @@ _LIBCPP_HIDE_FROM_ABI void __throw_bad_expected_access(_Arg&& __arg) {
   throw bad_expected_access<_Err>(std::forward<_Arg>(__arg));
 #  else
   (void)__arg;
-  std::abort();
+  _LIBCPP_VERBOSE_ABORT("bad_expected_access was thrown in -fno-exceptions mode");
 #  endif
 }
 
diff --git a/libcxx/include/__filesystem/filesystem_error.h b/libcxx/include/__filesystem/filesystem_error.h
index 70e39541e73ff..def7e3f6d17a0 100644
--- a/libcxx/include/__filesystem/filesystem_error.h
+++ b/libcxx/include/__filesystem/filesystem_error.h
@@ -15,10 +15,10 @@
 #include <__filesystem/path.h>
 #include <__memory/shared_ptr.h>
 #include <__utility/forward.h>
+#include <__verbose_abort>
 #include <iosfwd>
 #include <new>
 #include <system_error>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
@@ -92,7 +92,7 @@ void __throw_filesystem_error(_Args&&... __args) {
 }
 #else
 void __throw_filesystem_error(_Args&&...) {
-  _VSTD::abort();
+    _LIBCPP_VERBOSE_ABORT("filesystem_error was thrown in -fno-exceptions mode");
 }
 #endif
 _LIBCPP_AVAILABILITY_FILESYSTEM_POP
diff --git a/libcxx/include/__filesystem/path.h b/libcxx/include/__filesystem/path.h
index aad0beb0c6063..530821a9cb374 100644
--- a/libcxx/include/__filesystem/path.h
+++ b/libcxx/include/__filesystem/path.h
@@ -18,10 +18,13 @@
 #include <__fwd/hash.h>
 #include <__iterator/back_insert_iterator.h>
 #include <__iterator/iterator_traits.h>
+#include <__type_traits/decay.h>
+#include <__type_traits/is_pointer.h>
+#include <__type_traits/remove_const.h>
+#include <__type_traits/remove_pointer.h>
 #include <cstddef>
 #include <string>
 #include <string_view>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
 # include <iomanip> // for quoted
diff --git a/libcxx/include/__filesystem/u8path.h b/libcxx/include/__filesystem/u8path.h
index d35faa14bb8a0..ebc1159ee0b44 100644
--- a/libcxx/include/__filesystem/u8path.h
+++ b/libcxx/include/__filesystem/u8path.h
@@ -15,7 +15,6 @@
 #include <__config>
 #include <__filesystem/path.h>
 #include <string>
-#include <type_traits>
 
 // Only required on Windows for __widen_from_utf8, and included conservatively
 // because it requires support for localization.
diff --git a/libcxx/include/__format/buffer.h b/libcxx/include/__format/buffer.h
index c985dbc5ab01b..0f669ba9e2854 100644
--- a/libcxx/include/__format/buffer.h
+++ b/libcxx/include/__format/buffer.h
@@ -27,6 +27,8 @@
 #include <__iterator/incrementable_traits.h>
 #include <__iterator/iterator_traits.h>
 #include <__iterator/wrap_iter.h>
+#include <__memory/addressof.h>
+#include <__type_traits/add_pointer.h>
 #include <__utility/move.h>
 #include <cstddef>
 #include <string_view>
diff --git a/libcxx/include/__format/concepts.h b/libcxx/include/__format/concepts.h
index 0bc1b4740c57f..62552f8270583 100644
--- a/libcxx/include/__format/concepts.h
+++ b/libcxx/include/__format/concepts.h
@@ -65,9 +65,8 @@ concept formattable = __formattable<_Tp, _CharT>;
 // TODO FMT Add a test to validate we fail when using that concept after P2165
 // has been implemented.
 template <class _Tp>
-concept __fmt_pair_like = __is_specialization_v<_Tp, pair> ||
-                          // Use a requires since tuple_size_v may fail to instantiate,
-                          (__is_specialization_v<_Tp, tuple> && requires { tuple_size_v<_Tp> == 2; });
+concept __fmt_pair_like =
+    __is_specialization_v<_Tp, pair> || (__is_specialization_v<_Tp, tuple> && tuple_size_v<_Tp> == 2);
 
 #  endif //_LIBCPP_STD_VER >= 23
 #endif //_LIBCPP_STD_VER >= 20
diff --git a/libcxx/include/__format/container_adaptor.h b/libcxx/include/__format/container_adaptor.h
index baf062af1a809..9439b10c29914 100644
--- a/libcxx/include/__format/container_adaptor.h
+++ b/libcxx/include/__format/container_adaptor.h
@@ -19,6 +19,10 @@
 #include <__format/concepts.h>
 #include <__format/formatter.h>
 #include <__format/range_default_formatter.h>
+#include <__ranges/all.h>
+#include <__ranges/ref_view.h>
+#include <__type_traits/is_const.h>
+#include <__type_traits/maybe_const.h>
 #include <queue>
 #include <stack>
 
@@ -35,8 +39,9 @@ _LIBCPP_BEGIN_NAMESPACE_STD
 template <class _Adaptor, class _CharT>
 struct _LIBCPP_TEMPLATE_VIS _LIBCPP_AVAILABILITY_FORMAT __formatter_container_adaptor {
 private:
-  using __maybe_const_adaptor = __fmt_maybe_const<_Adaptor, _CharT>;
-  formatter<typename _Adaptor::container_type, _CharT> __underlying_;
+  using __maybe_const_container = __fmt_maybe_const<typename _Adaptor::container_type, _CharT>;
+  using __maybe_const_adaptor   = __maybe_const<is_const_v<__maybe_const_container>, _Adaptor>;
+  formatter<ranges::ref_view<__maybe_const_container>, _CharT> __underlying_;
 
 public:
   template <class _ParseContext>
diff --git a/libcxx/include/__format/format_arg.h b/libcxx/include/__format/format_arg.h
index 43fca44bee2fd..a25976a0795e7 100644
--- a/libcxx/include/__format/format_arg.h
+++ b/libcxx/include/__format/format_arg.h
@@ -18,6 +18,7 @@
 #include <__format/format_parse_context.h>
 #include <__functional/invoke.h>
 #include <__memory/addressof.h>
+#include <__utility/declval.h>
 #include <__utility/forward.h>
 #include <__utility/unreachable.h>
 #include <__variant/monostate.h>
diff --git a/libcxx/include/__format/format_arg_store.h b/libcxx/include/__format/format_arg_store.h
index 44e0e93850bf5..b9a36e58930ec 100644
--- a/libcxx/include/__format/format_arg_store.h
+++ b/libcxx/include/__format/format_arg_store.h
@@ -19,6 +19,9 @@
 #include <__config>
 #include <__format/concepts.h>
 #include <__format/format_arg.h>
+#include <__type_traits/conditional.h>
+#include <__type_traits/extent.h>
+#include <__type_traits/is_same.h>
 #include <__utility/forward.h>
 #include <string>
 #include <string_view>
@@ -140,21 +143,24 @@ consteval __arg_t __determine_arg_t() {
 //
 // Note this version can't be constrained avoiding ambiguous overloads.
 // That means it can be instantiated by disabled formatters. To solve this, a
-// constrained version for not formattable formatters is added. That overload
-// is marked as deleted to fail creating a storage type for disabled formatters.
+// constrained version for not formattable formatters is added.
 template <class _Context, class _Tp>
 consteval __arg_t __determine_arg_t() {
   return __arg_t::__handle;
 }
 
+// The overload for not formattable types allows triggering the static
+// assertion below.
 template <class _Context, class _Tp>
   requires(!__formattable<_Tp, typename _Context::char_type>)
-consteval __arg_t __determine_arg_t() = delete;
+consteval __arg_t __determine_arg_t() {
+  return __arg_t::__none;
+}
 
 template <class _Context, class _Tp>
 _LIBCPP_HIDE_FROM_ABI basic_format_arg<_Context> __create_format_arg(_Tp&& __value) noexcept {
   constexpr __arg_t __arg = __determine_arg_t<_Context, remove_cvref_t<_Tp>>();
-  static_assert(__arg != __arg_t::__none);
+  static_assert(__arg != __arg_t::__none, "the supplied type is not formattable");
 
   // Not all types can be used to directly initialize the
   // __basic_format_arg_value.  First handle all types needing adjustment, the
diff --git a/libcxx/include/__format/format_parse_context.h b/libcxx/include/__format/format_parse_context.h
index 2e2b47782a690..ac2f5a843405f 100644
--- a/libcxx/include/__format/format_parse_context.h
+++ b/libcxx/include/__format/format_parse_context.h
@@ -12,6 +12,7 @@
 
 #include <__config>
 #include <__format/format_error.h>
+#include <__type_traits/is_constant_evaluated.h>
 #include <string_view>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
@@ -58,6 +59,18 @@ class _LIBCPP_TEMPLATE_VIS _LIBCPP_AVAILABILITY_FORMAT basic_format_parse_contex
 
     if (__indexing_ == __unknown)
       __indexing_ = __automatic;
+
+    // Throws an exception to make the expression a non core constant
+    // expression as required by:
+    // [format.parse.ctx]/8
+    //   Remarks: Let cur-arg-id be the value of next_arg_id_ prior to this
+    //   call. Call expressions where cur-arg-id >= num_args_ is true are not
+    //   core constant expressions (7.7 [expr.const]).
+    // Note: the Throws clause [format.parse.ctx]/9 doesn't specify the
+    // behavior when id >= num_args_.
+    if (is_constant_evaluated() && __next_arg_id_ >= __num_args_)
+      std::__throw_format_error("Argument index outside the valid range");
+
     return __next_arg_id_++;
   }
   _LIBCPP_HIDE_FROM_ABI constexpr void check_arg_id(size_t __id) {
diff --git a/libcxx/include/__format/formatter_floating_point.h b/libcxx/include/__format/formatter_floating_point.h
index e95fba6d0a44e..31cd012e484ad 100644
--- a/libcxx/include/__format/formatter_floating_point.h
+++ b/libcxx/include/__format/formatter_floating_point.h
@@ -26,6 +26,7 @@
 #include <__format/formatter_output.h>
 #include <__format/parser_std_format_spec.h>
 #include <__memory/allocator.h>
+#include <__type_traits/conditional.h>
 #include <__utility/move.h>
 #include <__utility/unreachable.h>
 #include <charconv>
@@ -404,7 +405,6 @@ _LIBCPP_HIDE_FROM_ABI __float_result __format_buffer_general_lower_case(__float_
       // In fixed mode the algorithm truncates trailing spaces and possibly the
       // radix point. There's no good guess for the position of the radix point
       // therefore scan the output after the first digit.
-
       __result.__radix_point = _VSTD::find(__first, __result.__last, '.');
     }
   }
@@ -665,7 +665,7 @@ __format_floating_point(_Tp __value, auto& __ctx, __format_spec::__parsed_specif
       if (__result.__exponent == __result.__last)
         // if P > X >= -4, the conversion is with style f or F and precision P - 1 - X.
         // By including the radix point it calculates P - (1 + X)
-        __p -= __result.__radix_point - __buffer.begin();
+        __p -= __result.__radix_point - __result.__integral;
       else
         // otherwise, the conversion is with style e or E and precision P - 1.
         --__p;
diff --git a/libcxx/include/__format/formatter_integral.h b/libcxx/include/__format/formatter_integral.h
index 8a18049618846..cfb0644d0d2cf 100644
--- a/libcxx/include/__format/formatter_integral.h
+++ b/libcxx/include/__format/formatter_integral.h
@@ -17,11 +17,13 @@
 #include <__format/format_error.h>
 #include <__format/formatter_output.h>
 #include <__format/parser_std_format_spec.h>
+#include <__type_traits/make_unsigned.h>
 #include <__utility/unreachable.h>
 #include <array>
 #include <charconv>
 #include <limits>
 #include <string>
+#include <string_view>
 
 #ifndef _LIBCPP_HAS_NO_LOCALIZATION
 #  include <locale>
diff --git a/libcxx/include/__format/formatter_output.h b/libcxx/include/__format/formatter_output.h
index 6c8cadb76eec5..2b3c5ae97d71d 100644
--- a/libcxx/include/__format/formatter_output.h
+++ b/libcxx/include/__format/formatter_output.h
@@ -12,6 +12,7 @@
 
 #include <__algorithm/ranges_copy.h>
 #include <__algorithm/ranges_fill_n.h>
+#include <__algorithm/ranges_for_each.h>
 #include <__algorithm/ranges_transform.h>
 #include <__chrono/statically_widen.h>
 #include <__concepts/same_as.h>
@@ -503,36 +504,17 @@ __escape(basic_string<_CharT>& __str, basic_string_view<_CharT> __values, __esca
   __unicode::__code_point_view<_CharT> __view{__values.begin(), __values.end()};
 
   while (!__view.__at_end()) {
-    auto __first                                        = __view.__position();
-    typename __unicode::__consume_p2286_result __result = __view.__consume_p2286();
-    if (__result.__ill_formed_size == 0) {
-      if (!__formatter::__is_escaped_sequence_written(__str, __result.__value, __mark))
+    auto __first                                  = __view.__position();
+    typename __unicode::__consume_result __result = __view.__consume();
+    if (__result.__status == __unicode::__consume_result::__ok) {
+      if (!__formatter::__is_escaped_sequence_written(__str, __result.__code_point, __mark))
         // 2.2.1.3 - Add the character
         ranges::copy(__first, __view.__position(), std::back_insert_iterator(__str));
-
     } else {
       // 2.2.3 sequence of ill-formed code units
-      // The number of code-units in __result.__value depends on the character type being used.
-      if constexpr (sizeof(_CharT) == 1) {
-        _LIBCPP_ASSERT(__result.__ill_formed_size == 1 || __result.__ill_formed_size == 4,
-                       "illegal number of invalid code units.");
-        if (__result.__ill_formed_size == 1) // ill-formed, one code unit
-          __formatter::__write_escape_ill_formed_code_unit(__str, __result.__value & 0xff);
-        else { // out of valid range, four code units
-               // The code point was properly encoded, decode the value.
-          __formatter::__write_escape_ill_formed_code_unit(__str, __result.__value >> 18 | 0xf0);
-          __formatter::__write_escape_ill_formed_code_unit(__str, (__result.__value >> 12 & 0x3f) | 0x80);
-          __formatter::__write_escape_ill_formed_code_unit(__str, (__result.__value >> 6 & 0x3f) | 0x80);
-          __formatter::__write_escape_ill_formed_code_unit(__str, (__result.__value & 0x3f) | 0x80);
-        }
-      } else if constexpr (sizeof(_CharT) == 2) {
-        _LIBCPP_ASSERT(__result.__ill_formed_size == 1, "for UTF-16 at most one invalid code unit");
-        __formatter::__write_escape_ill_formed_code_unit(__str, __result.__value & 0xffff);
-      } else {
-        static_assert(sizeof(_CharT) == 4, "unsupported character width");
-        _LIBCPP_ASSERT(__result.__ill_formed_size == 1, "for UTF-32 one code unit is one code point");
-        __formatter::__write_escape_ill_formed_code_unit(__str, __result.__value);
-      }
+      ranges::for_each(__first, __view.__position(), [&](_CharT __value) {
+        __formatter::__write_escape_ill_formed_code_unit(__str, __formatter::__to_char32(__value));
+      });
     }
   }
 }
diff --git a/libcxx/include/__format/formatter_tuple.h b/libcxx/include/__format/formatter_tuple.h
index e1bfd7c5e11c3..d1874ddecb723 100644
--- a/libcxx/include/__format/formatter_tuple.h
+++ b/libcxx/include/__format/formatter_tuple.h
@@ -40,11 +40,11 @@ _LIBCPP_BEGIN_NAMESPACE_STD
 
 template <__fmt_char_type _CharT, class _Tuple, formattable<_CharT>... _Args>
 struct _LIBCPP_TEMPLATE_VIS _LIBCPP_AVAILABILITY_FORMAT __formatter_tuple {
-  _LIBCPP_HIDE_FROM_ABI constexpr void set_separator(basic_string_view<_CharT> __separator) {
+  _LIBCPP_HIDE_FROM_ABI constexpr void set_separator(basic_string_view<_CharT> __separator) noexcept {
     __separator_ = __separator;
   }
   _LIBCPP_HIDE_FROM_ABI constexpr void
-  set_brackets(basic_string_view<_CharT> __opening_bracket, basic_string_view<_CharT> __closing_bracket) {
+  set_brackets(basic_string_view<_CharT> __opening_bracket, basic_string_view<_CharT> __closing_bracket) noexcept {
     __opening_bracket_ = __opening_bracket;
     __closing_bracket_ = __closing_bracket;
   }
diff --git a/libcxx/include/__format/parser_std_format_spec.h b/libcxx/include/__format/parser_std_format_spec.h
index b88f39960184a..c500525932025 100644
--- a/libcxx/include/__format/parser_std_format_spec.h
+++ b/libcxx/include/__format/parser_std_format_spec.h
@@ -30,6 +30,8 @@
 #include <__format/unicode.h>
 #include <__iterator/concepts.h>
 #include <__iterator/readable_traits.h> // iter_value_t
+#include <__type_traits/common_type.h>
+#include <__type_traits/is_trivially_copyable.h>
 #include <__variant/monostate.h>
 #include <cstdint>
 #include <string_view>
diff --git a/libcxx/include/__format/range_default_formatter.h b/libcxx/include/__format/range_default_formatter.h
index 136ff063d0697..7fdb254de3b88 100644
--- a/libcxx/include/__format/range_default_formatter.h
+++ b/libcxx/include/__format/range_default_formatter.h
@@ -95,11 +95,11 @@ struct _LIBCPP_TEMPLATE_VIS _LIBCPP_AVAILABILITY_FORMAT __range_default_formatte
   range_formatter<remove_cvref_t<ranges::range_reference_t<__maybe_const_r>>, _CharT> __underlying_;
 
 public:
-  _LIBCPP_HIDE_FROM_ABI constexpr void set_separator(basic_string_view<_CharT> __separator) {
+  _LIBCPP_HIDE_FROM_ABI constexpr void set_separator(basic_string_view<_CharT> __separator) noexcept {
     __underlying_.set_separator(__separator);
   }
   _LIBCPP_HIDE_FROM_ABI constexpr void
-  set_brackets(basic_string_view<_CharT> __opening_bracket, basic_string_view<_CharT> __closing_bracket) {
+  set_brackets(basic_string_view<_CharT> __opening_bracket, basic_string_view<_CharT> __closing_bracket) noexcept {
     __underlying_.set_brackets(__opening_bracket, __closing_bracket);
   }
 
diff --git a/libcxx/include/__format/range_formatter.h b/libcxx/include/__format/range_formatter.h
index aac379536219c..f60151029c9c3 100644
--- a/libcxx/include/__format/range_formatter.h
+++ b/libcxx/include/__format/range_formatter.h
@@ -41,17 +41,17 @@ _LIBCPP_BEGIN_NAMESPACE_STD
 template <class _Tp, class _CharT = char>
   requires same_as<remove_cvref_t<_Tp>, _Tp> && formattable<_Tp, _CharT>
 struct _LIBCPP_TEMPLATE_VIS _LIBCPP_AVAILABILITY_FORMAT range_formatter {
-  _LIBCPP_HIDE_FROM_ABI constexpr void set_separator(basic_string_view<_CharT> __separator) {
+  _LIBCPP_HIDE_FROM_ABI constexpr void set_separator(basic_string_view<_CharT> __separator) noexcept {
     __separator_ = __separator;
   }
   _LIBCPP_HIDE_FROM_ABI constexpr void
-  set_brackets(basic_string_view<_CharT> __opening_bracket, basic_string_view<_CharT> __closing_bracket) {
+  set_brackets(basic_string_view<_CharT> __opening_bracket, basic_string_view<_CharT> __closing_bracket) noexcept {
     __opening_bracket_ = __opening_bracket;
     __closing_bracket_ = __closing_bracket;
   }
 
-  _LIBCPP_HIDE_FROM_ABI constexpr formatter<_Tp, _CharT>& underlying() { return __underlying_; }
-  _LIBCPP_HIDE_FROM_ABI constexpr const formatter<_Tp, _CharT>& underlying() const { return __underlying_; }
+  _LIBCPP_HIDE_FROM_ABI constexpr formatter<_Tp, _CharT>& underlying() noexcept { return __underlying_; }
+  _LIBCPP_HIDE_FROM_ABI constexpr const formatter<_Tp, _CharT>& underlying() const noexcept { return __underlying_; }
 
   template <class _ParseContext>
   _LIBCPP_HIDE_FROM_ABI constexpr typename _ParseContext::iterator parse(_ParseContext& __parse_ctx) {
diff --git a/libcxx/include/__format/unicode.h b/libcxx/include/__format/unicode.h
index 53b5320600dc1..12aed507990e8 100644
--- a/libcxx/include/__format/unicode.h
+++ b/libcxx/include/__format/unicode.h
@@ -31,23 +31,28 @@ _LIBCPP_BEGIN_NAMESPACE_STD
 
 namespace __unicode {
 
-#  if _LIBCPP_STD_VER >= 23
-
-/// The result of consuming a code point using P2286' semantics
-///
-/// TODO FMT Combine __consume and  __consume_p2286 in one function.
-struct __consume_p2286_result {
-  // A size of 0 means well formed. This to differenciate between
-  // a valid code point and a code unit that's invalid like 0b11111xxx.
-  int __ill_formed_size;
-
-  // If well formed the consumed code point.
-  // Otherwise the ill-formed code units as unsigned 8-bit values. They are
-  // stored in reverse order, to make it easier to extract the values.
-  char32_t __value;
+// Helper struct for the result of a consume operation.
+//
+// The status value for a correct code point is 0. This allows a valid value to
+// be used without masking.
+// When the decoding fails it know the number of code units affected. For the
+// current use-cases that value is not needed, therefore it is not stored.
+// The escape routine needs the number of code units for both a valid and
+// invalid character and keeps track of it itself. Doing it in this result
+// unconditionally would give some overhead when the value is unneeded.
+struct __consume_result {
+  // When __status == __ok it contains the decoded code point.
+  // Else it contains the replacement character U+FFFD
+  char32_t __code_point : 31;
+
+  enum : char32_t {
+    // Consumed a well-formed code point.
+    __ok = 0,
+    // Encountered invalid UTF-8
+    __error = 1
+  } __status : 1 {__ok};
 };
-
-#  endif // _LIBCPP_STD_VER >= 23
+static_assert(sizeof(__consume_result) == sizeof(char32_t));
 
 #  ifndef _LIBCPP_HAS_NO_UNICODE
 
@@ -66,6 +71,36 @@ struct __consume_p2286_result {
 
 inline constexpr char32_t __replacement_character = U'\ufffd';
 
+// The error of a consume operation.
+//
+// This sets the code point to the replacement character. This code point does
+// not participate in the grapheme clustering, so grapheme clustering code can
+// ignore the error status and always use the code point.
+inline constexpr __consume_result __consume_result_error{__replacement_character, __consume_result::__error};
+
+[[nodiscard]] _LIBCPP_HIDE_FROM_ABI constexpr bool __is_high_surrogate(char32_t __value) {
+  return __value >= 0xd800 && __value <= 0xdbff;
+}
+
+[[nodiscard]] _LIBCPP_HIDE_FROM_ABI constexpr bool __is_low_surrogate(char32_t __value) {
+  return __value >= 0xdc00 && __value <= 0xdfff;
+}
+
+// https://www.unicode.org/glossary/#surrogate_code_point
+[[nodiscard]] _LIBCPP_HIDE_FROM_ABI inline constexpr bool __is_surrogate(char32_t __value) {
+  return __value >= 0xd800 && __value <= 0xdfff;
+}
+
+// https://www.unicode.org/glossary/#code_point
+[[nodiscard]] _LIBCPP_HIDE_FROM_ABI inline constexpr bool __is_code_point(char32_t __value) {
+  return __value <= 0x10ffff;
+}
+
+// https://www.unicode.org/glossary/#unicode_scalar_value
+[[nodiscard]] _LIBCPP_HIDE_FROM_ABI inline constexpr bool __is_scalar_value(char32_t __value) {
+  return __unicode::__is_code_point(__value) && !__unicode::__is_surrogate(__value);
+}
+
 template <contiguous_iterator _Iterator>
   requires same_as<iter_value_t<_Iterator>, char>
 _LIBCPP_HIDE_FROM_ABI constexpr bool __is_continuation(_Iterator __char, int __count) {
@@ -97,122 +132,103 @@ class __code_point_view<char> {
   _LIBCPP_HIDE_FROM_ABI constexpr bool __at_end() const noexcept { return __first_ == __last_; }
   _LIBCPP_HIDE_FROM_ABI constexpr _Iterator __position() const noexcept { return __first_; }
 
-  _LIBCPP_HIDE_FROM_ABI constexpr char32_t __consume() noexcept {
+  // https://www.unicode.org/versions/latest/ch03.pdf#G7404
+  // Based on Table 3-7, Well-Formed UTF-8 Byte Sequences
+  //
+  // Code Points        First Byte Second Byte Third Byte Fourth Byte  Remarks
+  // U+0000..U+007F     00..7F                                         U+0000..U+007F 1 code unit range
+  //                    C0..C1     80..BF                              invalid overlong encoding
+  // U+0080..U+07FF     C2..DF     80..BF                              U+0080..U+07FF 2 code unit range
+  //                    E0         80..9F      80..BF                  invalid overlong encoding
+  // U+0800..U+0FFF     E0         A0..BF      80..BF                  U+0800..U+FFFF 3 code unit range
+  // U+1000..U+CFFF     E1..EC     80..BF      80..BF
+  // U+D000..U+D7FF     ED         80..9F      80..BF
+  // U+D800..U+DFFF     ED         A0..BF      80..BF                  invalid encoding of surrogate code point
+  // U+E000..U+FFFF     EE..EF     80..BF      80..BF
+  //                    F0         80..8F      80..BF     80..BF       invalid overlong encoding
+  // U+10000..U+3FFFF   F0         90..BF      80..BF     80..BF       U+10000..U+10FFFF 4 code unit range
+  // U+40000..U+FFFFF   F1..F3     80..BF      80..BF     80..BF
+  // U+100000..U+10FFFF F4         80..8F      80..BF     80..BF
+  //                    F4         90..BF      80..BF     80..BF       U+110000.. invalid code point range
+  //
+  // Unlike other parsers, these invalid entries are tested after decoding.
+  // - The parser always needs to consume these code units
+  // - The code is optimized for well-formed UTF-8
+  [[nodiscard]] _LIBCPP_HIDE_FROM_ABI constexpr __consume_result __consume() noexcept {
     _LIBCPP_ASSERT(__first_ != __last_, "can't move beyond the end of input");
 
     // Based on the number of leading 1 bits the number of code units in the
     // code point can be determined. See
     // https://en.wikipedia.org/wiki/UTF-8#Encoding
-    switch (_VSTD::countl_one(static_cast<unsigned char>(*__first_))) {
+    switch (std::countl_one(static_cast<unsigned char>(*__first_))) {
     case 0:
-      return *__first_++;
+      return {static_cast<unsigned char>(*__first_++)};
 
-    case 2:
+    case 2: {
       if (__last_ - __first_ < 2 || !__unicode::__is_continuation(__first_ + 1, 1)) [[unlikely]]
         break;
-      else {
-        char32_t __value = static_cast<unsigned char>(*__first_++) & 0x1f;
-        __value <<= 6;
-        __value |= static_cast<unsigned char>(*__first_++) & 0x3f;
-        return __value;
-      }
 
-    case 3:
-      if (__last_ - __first_ < 3 || !__unicode::__is_continuation(__first_ + 1, 2)) [[unlikely]]
-        break;
-      else {
-        char32_t __value = static_cast<unsigned char>(*__first_++) & 0x0f;
-        __value <<= 6;
-        __value |= static_cast<unsigned char>(*__first_++) & 0x3f;
-        __value <<= 6;
-        __value |= static_cast<unsigned char>(*__first_++) & 0x3f;
-        return __value;
-      }
+      char32_t __value = static_cast<unsigned char>(*__first_++) & 0x1f;
+      __value <<= 6;
+      __value |= static_cast<unsigned char>(*__first_++) & 0x3f;
 
-    case 4:
-      if (__last_ - __first_ < 4 || !__unicode::__is_continuation(__first_ + 1, 3)) [[unlikely]]
-        break;
-      else {
-        char32_t __value = static_cast<unsigned char>(*__first_++) & 0x07;
-        __value <<= 6;
-        __value |= static_cast<unsigned char>(*__first_++) & 0x3f;
-        __value <<= 6;
-        __value |= static_cast<unsigned char>(*__first_++) & 0x3f;
-        __value <<= 6;
-        __value |= static_cast<unsigned char>(*__first_++) & 0x3f;
-        return __value;
-      }
-    }
-    // An invalid number of leading ones can be garbage or a code unit in the
-    // middle of a code point. By consuming one code unit the parser may get
-    // "in sync" after a few code units.
-    ++__first_;
-    return __replacement_character;
-  }
-
-#    if _LIBCPP_STD_VER >= 23
-  _LIBCPP_HIDE_FROM_ABI constexpr __consume_p2286_result __consume_p2286() noexcept {
-    _LIBCPP_ASSERT(__first_ != __last_, "can't move beyond the end of input");
+      // These values should be encoded in 1 UTF-8 code unit.
+      if (__value < 0x0080) [[unlikely]]
+        return __consume_result_error;
 
-    // Based on the number of leading 1 bits the number of code units in the
-    // code point can be determined. See
-    // https://en.wikipedia.org/wiki/UTF-8#Encoding
-    switch (std::countl_one(static_cast<unsigned char>(*__first_))) {
-    case 0:
-      return {0, static_cast<unsigned char>(*__first_++)};
+      return {__value};
+    }
 
-    case 2:
-      if (__last_ - __first_ < 2) [[unlikely]]
+    case 3: {
+      if (__last_ - __first_ < 3 || !__unicode::__is_continuation(__first_ + 1, 2)) [[unlikely]]
         break;
 
-      if (__unicode::__is_continuation(__first_ + 1, 1)) {
-        char32_t __value = static_cast<unsigned char>(*__first_++) & 0x1f;
-        __value <<= 6;
-        __value |= static_cast<unsigned char>(*__first_++) & 0x3f;
-        return {0, __value};
-      }
-      break;
+      char32_t __value = static_cast<unsigned char>(*__first_++) & 0x0f;
+      __value <<= 6;
+      __value |= static_cast<unsigned char>(*__first_++) & 0x3f;
+      __value <<= 6;
+      __value |= static_cast<unsigned char>(*__first_++) & 0x3f;
 
-    case 3:
-      if (__last_ - __first_ < 3) [[unlikely]]
-        break;
+      // These values should be encoded in 1 or 2 UTF-8 code units.
+      if (__value < 0x0800) [[unlikely]]
+        return __consume_result_error;
 
-      if (__unicode::__is_continuation(__first_ + 1, 2)) {
-        char32_t __value = static_cast<unsigned char>(*__first_++) & 0x0f;
-        __value <<= 6;
-        __value |= static_cast<unsigned char>(*__first_++) & 0x3f;
-        __value <<= 6;
-        __value |= static_cast<unsigned char>(*__first_++) & 0x3f;
-        return {0, __value};
-      }
-      break;
+      // A surrogate value is always encoded in 3 UTF-8 code units.
+      if (__unicode::__is_surrogate(__value)) [[unlikely]]
+        return __consume_result_error;
+
+      return {__value};
+    }
 
-    case 4:
-      if (__last_ - __first_ < 4) [[unlikely]]
+    case 4: {
+      if (__last_ - __first_ < 4 || !__unicode::__is_continuation(__first_ + 1, 3)) [[unlikely]]
         break;
 
-      if (__unicode::__is_continuation(__first_ + 1, 3)) {
-        char32_t __value = static_cast<unsigned char>(*__first_++) & 0x07;
-        __value <<= 6;
-        __value |= static_cast<unsigned char>(*__first_++) & 0x3f;
-        __value <<= 6;
-        __value |= static_cast<unsigned char>(*__first_++) & 0x3f;
-        __value <<= 6;
-        __value |= static_cast<unsigned char>(*__first_++) & 0x3f;
+      char32_t __value = static_cast<unsigned char>(*__first_++) & 0x07;
+      __value <<= 6;
+      __value |= static_cast<unsigned char>(*__first_++) & 0x3f;
+      __value <<= 6;
+      __value |= static_cast<unsigned char>(*__first_++) & 0x3f;
+      __value <<= 6;
+      __value |= static_cast<unsigned char>(*__first_++) & 0x3f;
 
-        if (__value > 0x10FFFF) // Outside the valid Unicode range?
-          return {4, __value};
+      // These values should be encoded in 1, 2, or 3 UTF-8 code units.
+      if (__value < 0x10000) [[unlikely]]
+        return __consume_result_error;
 
-        return {0, __value};
-      }
-      break;
+      // A value too large is always encoded in 4 UTF-8 code units.
+      if (!__unicode::__is_code_point(__value)) [[unlikely]]
+        return __consume_result_error;
+
+      return {__value};
+    }
     }
     // An invalid number of leading ones can be garbage or a code unit in the
     // middle of a code point. By consuming one code unit the parser may get
     // "in sync" after a few code units.
-    return {1, static_cast<unsigned char>(*__first_++)};
+    ++__first_;
+    return __consume_result_error;
   }
-#    endif // _LIBCPP_STD_VER >= 23
 
 private:
   _Iterator __first_;
@@ -244,62 +260,33 @@ class __code_point_view<wchar_t> {
   _LIBCPP_HIDE_FROM_ABI constexpr _Iterator __position() const noexcept { return __first_; }
   _LIBCPP_HIDE_FROM_ABI constexpr bool __at_end() const noexcept { return __first_ == __last_; }
 
-  _LIBCPP_HIDE_FROM_ABI constexpr char32_t __consume() noexcept {
+  [[nodiscard]] _LIBCPP_HIDE_FROM_ABI constexpr __consume_result __consume() noexcept {
     _LIBCPP_ASSERT(__first_ != __last_, "can't move beyond the end of input");
 
+    char32_t __value = static_cast<char32_t>(*__first_++);
     if constexpr (sizeof(wchar_t) == 2) {
-      char32_t __result = *__first_++;
-      // Is the code unit part of a surrogate pair? See
-      // https://en.wikipedia.org/wiki/UTF-16#U+D800_to_U+DFFF
-      if (__result >= 0xd800 && __result <= 0xDfff) {
-        // Malformed Unicode.
-        if (__first_ == __last_) [[unlikely]]
-          return __replacement_character;
-
-        __result -= 0xd800;
-        __result <<= 10;
-        __result += *__first_++ - 0xdc00;
-        __result += 0x10000;
-      }
-      return __result;
+      if (__unicode::__is_low_surrogate(__value)) [[unlikely]]
+        return __consume_result_error;
 
-    } else if constexpr (sizeof(wchar_t) == 4) {
-      char32_t __result = *__first_++;
-      if (__result > 0x10FFFF) [[unlikely]]
-        return __replacement_character;
-      return __result;
-    } else {
-      __libcpp_unreachable();
-    }
-  }
+      if (__unicode::__is_high_surrogate(__value)) {
+        if (__first_ == __last_ || !__unicode::__is_low_surrogate(static_cast<char32_t>(*__first_))) [[unlikely]]
+          return __consume_result_error;
 
-#      if _LIBCPP_STD_VER >= 23
-  _LIBCPP_HIDE_FROM_ABI constexpr __consume_p2286_result __consume_p2286() noexcept {
-    _LIBCPP_ASSERT(__first_ != __last_, "can't move beyond the end of input");
+        __value -= 0xd800;
+        __value <<= 10;
+        __value += static_cast<char32_t>(*__first_++) - 0xdc00;
+        __value += 0x10000;
 
-    char32_t __result = *__first_++;
-    if constexpr (sizeof(wchar_t) == 2) {
-      // https://en.wikipedia.org/wiki/UTF-16#U+D800_to_U+DFFF
-      if (__is_surrogate_pair_high(__result)) {
-        // Malformed Unicode.
-        if (__first_ == __last_ || !__is_surrogate_pair_low(*(__first_ + 1))) [[unlikely]]
-          return {1, __result};
-
-        __result -= 0xd800;
-        __result <<= 10;
-        __result += *__first_++ - 0xdc00;
-        __result += 0x10000;
-      } else if (__is_surrogate_pair_low(__result))
-        // A code point shouldn't start with the low surrogate pair
-        return {1, __result};
+        if (!__unicode::__is_code_point(__value)) [[unlikely]]
+          return __consume_result_error;
+      }
     } else {
-      if (__result > 0x10FFFF) [[unlikely]]
-        return {1, __result};
+      if (!__unicode::__is_scalar_value(__value)) [[unlikely]]
+        return __consume_result_error;
     }
 
-    return {0, __result};
+    return {__value};
   }
-#      endif // _LIBCPP_STD_VER >= 23
 
 private:
   _Iterator __first_;
@@ -399,7 +386,7 @@ class __extended_grapheme_cluster_view {
 public:
   _LIBCPP_HIDE_FROM_ABI constexpr explicit __extended_grapheme_cluster_view(_Iterator __first, _Iterator __last)
       : __code_point_view_(__first, __last),
-        __next_code_point_(__code_point_view_.__consume()),
+        __next_code_point_(__code_point_view_.__consume().__code_point),
         __next_prop_(__extended_grapheme_custer_property_boundary::__get_property(__next_code_point_)) {}
 
   struct __cluster {
@@ -420,6 +407,7 @@ class __extended_grapheme_cluster_view {
     _LIBCPP_ASSERT(
         __next_prop_ != __extended_grapheme_custer_property_boundary::__property::__eot,
         "can't move beyond the end of input");
+
     char32_t __code_point = __next_code_point_;
     if (!__code_point_view_.__at_end())
       return {__code_point, __get_break()};
@@ -444,7 +432,7 @@ class __extended_grapheme_cluster_view {
         __next_prop_ = __extended_grapheme_custer_property_boundary::__property::__eot;
         return __result;
       }
-      __next_code_point_ = __code_point_view_.__consume();
+      __next_code_point_ = __code_point_view_.__consume().__code_point;
       __next_prop_       = __extended_grapheme_custer_property_boundary::__get_property(__next_code_point_);
 
       __has_extened_pictographic |=
@@ -474,18 +462,10 @@ class __code_point_view {
   _LIBCPP_HIDE_FROM_ABI constexpr bool __at_end() const noexcept { return __first_ == __last_; }
   _LIBCPP_HIDE_FROM_ABI constexpr _Iterator __position() const noexcept { return __first_; }
 
-  _LIBCPP_HIDE_FROM_ABI constexpr char32_t __consume() noexcept {
+  [[nodiscard]] _LIBCPP_HIDE_FROM_ABI constexpr __consume_result __consume() noexcept {
     _LIBCPP_ASSERT(__first_ != __last_, "can't move beyond the end of input");
-    return *__first_++;
-  }
-
-#    if _LIBCPP_STD_VER >= 23
-  _LIBCPP_HIDE_FROM_ABI constexpr __consume_p2286_result __consume_p2286() noexcept {
-    _LIBCPP_ASSERT(__first_ != __last_, "can't move beyond the end of input");
-
-    return {0, std::make_unsigned_t<_CharT>(*__first_++)};
+    return {static_cast<char32_t>(*__first_++)};
   }
-#    endif // _LIBCPP_STD_VER >= 23
 
 private:
   _Iterator __first_;
diff --git a/libcxx/include/__functional/bind.h b/libcxx/include/__functional/bind.h
index 30af9380eb6e1..0eb5ce7055895 100644
--- a/libcxx/include/__functional/bind.h
+++ b/libcxx/include/__functional/bind.h
@@ -15,7 +15,6 @@
 #include <__functional/weak_result_type.h>
 #include <cstddef>
 #include <tuple>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__functional/bind_back.h b/libcxx/include/__functional/bind_back.h
index 1476325055779..544315d6ea55f 100644
--- a/libcxx/include/__functional/bind_back.h
+++ b/libcxx/include/__functional/bind_back.h
@@ -16,7 +16,6 @@
 #include <__utility/forward.h>
 #include <__utility/integer_sequence.h>
 #include <tuple>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__functional/bind_front.h b/libcxx/include/__functional/bind_front.h
index 4e0e71d8df1f8..109a83c2104f6 100644
--- a/libcxx/include/__functional/bind_front.h
+++ b/libcxx/include/__functional/bind_front.h
@@ -14,7 +14,6 @@
 #include <__functional/invoke.h>
 #include <__functional/perfect_forward.h>
 #include <__utility/forward.h>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__functional/compose.h b/libcxx/include/__functional/compose.h
index 2a0d61d7eb0be..168131e1d9ea3 100644
--- a/libcxx/include/__functional/compose.h
+++ b/libcxx/include/__functional/compose.h
@@ -14,7 +14,6 @@
 #include <__functional/invoke.h>
 #include <__functional/perfect_forward.h>
 #include <__utility/forward.h>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__functional/function.h b/libcxx/include/__functional/function.h
index d7cc07e7355b6..3c6998cb8c8fa 100644
--- a/libcxx/include/__functional/function.h
+++ b/libcxx/include/__functional/function.h
@@ -12,6 +12,7 @@
 
 #include <__assert>
 #include <__config>
+#include <__exception/exception.h>
 #include <__functional/binary_function.h>
 #include <__functional/invoke.h>
 #include <__functional/unary_function.h>
@@ -23,15 +24,17 @@
 #include <__memory/builtin_new_allocator.h>
 #include <__memory/compressed_pair.h>
 #include <__memory/unique_ptr.h>
+#include <__type_traits/aligned_storage.h>
+#include <__type_traits/is_trivially_copy_constructible.h>
+#include <__type_traits/is_trivially_destructible.h>
 #include <__type_traits/strip_signature.h>
 #include <__utility/forward.h>
 #include <__utility/move.h>
 #include <__utility/piecewise_construct.h>
 #include <__utility/swap.h>
-#include <exception>
+#include <__verbose_abort>
 #include <new>
 #include <tuple>
-#include <type_traits>
 #include <typeinfo>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
@@ -71,7 +74,7 @@ void __throw_bad_function_call()
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
     throw bad_function_call();
 #else
-    _VSTD::abort();
+    _LIBCPP_VERBOSE_ABORT("bad_function_call was thrown in -fno-exceptions mode");
 #endif
 }
 
diff --git a/libcxx/include/__functional/is_transparent.h b/libcxx/include/__functional/is_transparent.h
index 4ff163f7b3071..c539a07d696dd 100644
--- a/libcxx/include/__functional/is_transparent.h
+++ b/libcxx/include/__functional/is_transparent.h
@@ -11,7 +11,8 @@
 #define _LIBCPP___FUNCTIONAL_IS_TRANSPARENT
 
 #include <__config>
-#include <type_traits>
+#include <__type_traits/integral_constant.h>
+#include <__type_traits/void_t.h>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__functional/mem_fn.h b/libcxx/include/__functional/mem_fn.h
index 8b51627079cf2..fe221dd128f42 100644
--- a/libcxx/include/__functional/mem_fn.h
+++ b/libcxx/include/__functional/mem_fn.h
@@ -15,7 +15,6 @@
 #include <__functional/invoke.h>
 #include <__functional/weak_result_type.h>
 #include <__utility/forward.h>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__functional/not_fn.h b/libcxx/include/__functional/not_fn.h
index 3c4f588508639..f148a408eeec9 100644
--- a/libcxx/include/__functional/not_fn.h
+++ b/libcxx/include/__functional/not_fn.h
@@ -14,7 +14,6 @@
 #include <__functional/invoke.h>
 #include <__functional/perfect_forward.h>
 #include <__utility/forward.h>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__functional/perfect_forward.h b/libcxx/include/__functional/perfect_forward.h
index 4497decab8b0f..af89797af2964 100644
--- a/libcxx/include/__functional/perfect_forward.h
+++ b/libcxx/include/__functional/perfect_forward.h
@@ -15,7 +15,6 @@
 #include <__utility/forward.h>
 #include <__utility/move.h>
 #include <tuple>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__hash_table b/libcxx/include/__hash_table
index 19a47e0d04ce0..e38a44221c89f 100644
--- a/libcxx/include/__hash_table
+++ b/libcxx/include/__hash_table
@@ -17,6 +17,7 @@
 #include <__config>
 #include <__debug>
 #include <__functional/hash.h>
+#include <__functional/invoke.h>
 #include <__iterator/iterator_traits.h>
 #include <__memory/addressof.h>
 #include <__memory/allocator_traits.h>
@@ -25,6 +26,16 @@
 #include <__memory/swap_allocator.h>
 #include <__memory/unique_ptr.h>
 #include <__type_traits/can_extract_key.h>
+#include <__type_traits/conditional.h>
+#include <__type_traits/is_const.h>
+#include <__type_traits/is_copy_constructible.h>
+#include <__type_traits/is_nothrow_constructible.h>
+#include <__type_traits/is_nothrow_default_constructible.h>
+#include <__type_traits/is_nothrow_move_constructible.h>
+#include <__type_traits/is_pointer.h>
+#include <__type_traits/is_reference.h>
+#include <__type_traits/remove_const.h>
+#include <__type_traits/remove_cvref.h>
 #include <__utility/forward.h>
 #include <__utility/move.h>
 #include <__utility/pair.h>
@@ -32,7 +43,6 @@
 #include <cmath>
 #include <cstring>
 #include <initializer_list>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__iterator/common_iterator.h b/libcxx/include/__iterator/common_iterator.h
index 3c8eb1c00e11c..aab04ce49717f 100644
--- a/libcxx/include/__iterator/common_iterator.h
+++ b/libcxx/include/__iterator/common_iterator.h
@@ -25,6 +25,7 @@
 #include <__iterator/iter_swap.h>
 #include <__iterator/iterator_traits.h>
 #include <__iterator/readable_traits.h>
+#include <__memory/addressof.h>
 #include <__type_traits/is_pointer.h>
 #include <__utility/declval.h>
 #include <variant>
diff --git a/libcxx/include/__memory/allocator.h b/libcxx/include/__memory/allocator.h
index de94cfb6a2353..244013cef7b64 100644
--- a/libcxx/include/__memory/allocator.h
+++ b/libcxx/include/__memory/allocator.h
@@ -11,6 +11,7 @@
 #define _LIBCPP___MEMORY_ALLOCATOR_H
 
 #include <__config>
+#include <__memory/addressof.h>
 #include <__memory/allocate_at_least.h>
 #include <__memory/allocator_traits.h>
 #include <__type_traits/is_constant_evaluated.h>
diff --git a/libcxx/include/__memory/allocator_arg_t.h b/libcxx/include/__memory/allocator_arg_t.h
index 15f8c98c4c90c..9e2d8c6311b64 100644
--- a/libcxx/include/__memory/allocator_arg_t.h
+++ b/libcxx/include/__memory/allocator_arg_t.h
@@ -25,9 +25,9 @@ _LIBCPP_BEGIN_NAMESPACE_STD
 
 struct _LIBCPP_TEMPLATE_VIS allocator_arg_t { explicit allocator_arg_t() = default; };
 
-#if defined(_LIBCPP_CXX03_LANG) || defined(_LIBCPP_BUILDING_LIBRARY)
+#if defined(_LIBCPP_BUILDING_LIBRARY)
 extern _LIBCPP_EXPORTED_FROM_ABI const allocator_arg_t allocator_arg;
-#else
+#elif !defined(_LIBCPP_CXX03_LANG)
 /* inline */ constexpr allocator_arg_t allocator_arg = allocator_arg_t();
 #endif
 
diff --git a/libcxx/include/__memory/shared_ptr.h b/libcxx/include/__memory/shared_ptr.h
index 3ef264dcfe48c..f22a918cd536e 100644
--- a/libcxx/include/__memory/shared_ptr.h
+++ b/libcxx/include/__memory/shared_ptr.h
@@ -29,9 +29,25 @@
 #include <__memory/pointer_traits.h>
 #include <__memory/uninitialized_algorithms.h>
 #include <__memory/unique_ptr.h>
+#include <__type_traits/add_lvalue_reference.h>
+#include <__type_traits/conditional.h>
+#include <__type_traits/conjunction.h>
+#include <__type_traits/disjunction.h>
+#include <__type_traits/is_array.h>
+#include <__type_traits/is_bounded_array.h>
+#include <__type_traits/is_convertible.h>
+#include <__type_traits/is_move_constructible.h>
+#include <__type_traits/is_reference.h>
+#include <__type_traits/is_unbounded_array.h>
+#include <__type_traits/nat.h>
+#include <__type_traits/negation.h>
+#include <__type_traits/remove_extent.h>
+#include <__type_traits/remove_reference.h>
+#include <__utility/declval.h>
 #include <__utility/forward.h>
 #include <__utility/move.h>
 #include <__utility/swap.h>
+#include <__verbose_abort>
 #include <cstddef>
 #include <cstdlib> // abort
 #include <iosfwd>
@@ -39,7 +55,7 @@
 #include <stdexcept>
 #include <typeinfo>
 #if !defined(_LIBCPP_HAS_NO_ATOMIC_HEADER)
-#  include <atomic>
+#  include <__atomic/memory_order.h>
 #endif
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
@@ -121,7 +137,7 @@ void __throw_bad_weak_ptr()
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
     throw bad_weak_ptr();
 #else
-    _VSTD::abort();
+    _LIBCPP_VERBOSE_ABORT("bad_weak_ptr was thrown in -fno-exceptions mode");
 #endif
 }
 
@@ -433,10 +449,10 @@ struct __is_array_deletable<_Ptr, decltype(delete[] std::declval<_Ptr>())> : tru
 
 template <class _Dp, class _Pt,
     class = decltype(std::declval<_Dp>()(std::declval<_Pt>()))>
-static true_type __well_formed_deleter_test(int);
+true_type __well_formed_deleter_test(int);
 
 template <class, class>
-static false_type __well_formed_deleter_test(...);
+false_type __well_formed_deleter_test(...);
 
 template <class _Dp, class _Pt>
 struct __well_formed_deleter : decltype(std::__well_formed_deleter_test<_Dp, _Pt>(0)) {};
diff --git a/libcxx/include/__mutex_base b/libcxx/include/__mutex_base
index 82d9fa6ecbacf..191955363a2d3 100644
--- a/libcxx/include/__mutex_base
+++ b/libcxx/include/__mutex_base
@@ -16,6 +16,8 @@
 #include <__chrono/time_point.h>
 #include <__config>
 #include <__threading_support>
+#include <__type_traits/is_floating_point.h>
+#include <__type_traits/is_nothrow_default_constructible.h>
 #include <ratio>
 #include <system_error>
 #include <time.h>
@@ -64,19 +66,15 @@ struct _LIBCPP_TYPE_VIS defer_lock_t { explicit defer_lock_t() = default; };
 struct _LIBCPP_TYPE_VIS try_to_lock_t { explicit try_to_lock_t() = default; };
 struct _LIBCPP_TYPE_VIS adopt_lock_t { explicit adopt_lock_t() = default; };
 
-#if defined(_LIBCPP_CXX03_LANG) || defined(_LIBCPP_BUILDING_LIBRARY)
-
-extern _LIBCPP_EXPORTED_FROM_ABI const defer_lock_t  defer_lock;
+#  if defined(_LIBCPP_BUILDING_LIBRARY)
+extern _LIBCPP_EXPORTED_FROM_ABI const defer_lock_t defer_lock;
 extern _LIBCPP_EXPORTED_FROM_ABI const try_to_lock_t try_to_lock;
-extern _LIBCPP_EXPORTED_FROM_ABI const adopt_lock_t  adopt_lock;
-
-#else
-
-/* inline */ constexpr defer_lock_t  defer_lock  = defer_lock_t();
+extern _LIBCPP_EXPORTED_FROM_ABI const adopt_lock_t adopt_lock;
+#  elif !defined(_LIBCPP_CXX03_LANG)
+/* inline */ constexpr defer_lock_t defer_lock = defer_lock_t();
 /* inline */ constexpr try_to_lock_t try_to_lock = try_to_lock_t();
-/* inline */ constexpr adopt_lock_t  adopt_lock  = adopt_lock_t();
-
-#endif
+/* inline */ constexpr adopt_lock_t adopt_lock = adopt_lock_t();
+#  endif
 
 template <class _Mutex>
 class _LIBCPP_TEMPLATE_VIS _LIBCPP_THREAD_SAFETY_ANNOTATION(scoped_lockable)
diff --git a/libcxx/include/__random/clamp_to_integral.h b/libcxx/include/__random/clamp_to_integral.h
index 85f8aa574ea12..c67d76d6e9bad 100644
--- a/libcxx/include/__random/clamp_to_integral.h
+++ b/libcxx/include/__random/clamp_to_integral.h
@@ -12,7 +12,6 @@
 #include <__config>
 #include <cmath>
 #include <limits>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__random/discard_block_engine.h b/libcxx/include/__random/discard_block_engine.h
index db33623d94851..077f1584d1e0e 100644
--- a/libcxx/include/__random/discard_block_engine.h
+++ b/libcxx/include/__random/discard_block_engine.h
@@ -11,10 +11,12 @@
 
 #include <__config>
 #include <__random/is_seed_sequence.h>
+#include <__type_traits/enable_if.h>
+#include <__type_traits/is_convertible.h>
 #include <__utility/move.h>
+#include <cstddef>
 #include <iosfwd>
 #include <limits>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__random/independent_bits_engine.h b/libcxx/include/__random/independent_bits_engine.h
index f4a23bbb5098b..2cd4096884222 100644
--- a/libcxx/include/__random/independent_bits_engine.h
+++ b/libcxx/include/__random/independent_bits_engine.h
@@ -12,10 +12,13 @@
 #include <__config>
 #include <__random/is_seed_sequence.h>
 #include <__random/log2.h>
+#include <__type_traits/conditional.h>
+#include <__type_traits/enable_if.h>
+#include <__type_traits/is_convertible.h>
 #include <__utility/move.h>
+#include <cstddef>
 #include <iosfwd>
 #include <limits>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__random/is_seed_sequence.h b/libcxx/include/__random/is_seed_sequence.h
index 3c82b2790b035..06621c3cee0d0 100644
--- a/libcxx/include/__random/is_seed_sequence.h
+++ b/libcxx/include/__random/is_seed_sequence.h
@@ -10,7 +10,9 @@
 #define _LIBCPP___RANDOM_IS_SEED_SEQUENCE_H
 
 #include <__config>
-#include <type_traits>
+#include <__type_traits/is_convertible.h>
+#include <__type_traits/is_same.h>
+#include <__type_traits/remove_cv.h>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__random/linear_congruential_engine.h b/libcxx/include/__random/linear_congruential_engine.h
index 0d91f14120b64..aa24a06c8c24b 100644
--- a/libcxx/include/__random/linear_congruential_engine.h
+++ b/libcxx/include/__random/linear_congruential_engine.h
@@ -11,9 +11,11 @@
 
 #include <__config>
 #include <__random/is_seed_sequence.h>
+#include <__type_traits/enable_if.h>
+#include <__type_traits/integral_constant.h>
+#include <__type_traits/is_unsigned.h>
 #include <cstdint>
 #include <iosfwd>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__random/mersenne_twister_engine.h b/libcxx/include/__random/mersenne_twister_engine.h
index eb5f56f42dba8..89df825619612 100644
--- a/libcxx/include/__random/mersenne_twister_engine.h
+++ b/libcxx/include/__random/mersenne_twister_engine.h
@@ -17,7 +17,6 @@
 #include <cstdint>
 #include <iosfwd>
 #include <limits>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__random/seed_seq.h b/libcxx/include/__random/seed_seq.h
index 330537fa00234..5c40d15899f7f 100644
--- a/libcxx/include/__random/seed_seq.h
+++ b/libcxx/include/__random/seed_seq.h
@@ -13,6 +13,7 @@
 #include <__algorithm/fill.h>
 #include <__algorithm/max.h>
 #include <__config>
+#include <cstdint>
 #include <initializer_list>
 #include <vector>
 
diff --git a/libcxx/include/__random/shuffle_order_engine.h b/libcxx/include/__random/shuffle_order_engine.h
index 86a60fc57b251..e07f230d21c77 100644
--- a/libcxx/include/__random/shuffle_order_engine.h
+++ b/libcxx/include/__random/shuffle_order_engine.h
@@ -12,10 +12,13 @@
 #include <__algorithm/equal.h>
 #include <__config>
 #include <__random/is_seed_sequence.h>
+#include <__type_traits/enable_if.h>
+#include <__type_traits/integral_constant.h>
+#include <__type_traits/is_convertible.h>
 #include <__utility/move.h>
+#include <cstddef>
 #include <cstdint>
 #include <iosfwd>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__random/subtract_with_carry_engine.h b/libcxx/include/__random/subtract_with_carry_engine.h
index b25ca84e3afd7..9294cc2f3811d 100644
--- a/libcxx/include/__random/subtract_with_carry_engine.h
+++ b/libcxx/include/__random/subtract_with_carry_engine.h
@@ -18,7 +18,6 @@
 #include <cstdint>
 #include <iosfwd>
 #include <limits>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__random/uniform_real_distribution.h b/libcxx/include/__random/uniform_real_distribution.h
index b52b68977f826..dd6d6b3335e29 100644
--- a/libcxx/include/__random/uniform_real_distribution.h
+++ b/libcxx/include/__random/uniform_real_distribution.h
@@ -14,7 +14,6 @@
 #include <__random/is_valid.h>
 #include <iosfwd>
 #include <limits>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/all.h b/libcxx/include/__ranges/all.h
index 383fd14b261b3..ecf908df1fe6a 100644
--- a/libcxx/include/__ranges/all.h
+++ b/libcxx/include/__ranges/all.h
@@ -18,10 +18,10 @@
 #include <__ranges/owning_view.h>
 #include <__ranges/range_adaptor.h>
 #include <__ranges/ref_view.h>
+#include <__type_traits/decay.h>
 #include <__utility/auto_cast.h>
 #include <__utility/declval.h>
 #include <__utility/forward.h>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/common_view.h b/libcxx/include/__ranges/common_view.h
index 2b1e72bcbbfbd..8fc68bada0c76 100644
--- a/libcxx/include/__ranges/common_view.h
+++ b/libcxx/include/__ranges/common_view.h
@@ -24,7 +24,6 @@
 #include <__ranges/view_interface.h>
 #include <__utility/forward.h>
 #include <__utility/move.h>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/copyable_box.h b/libcxx/include/__ranges/copyable_box.h
index c92e29390021a..1973bbffc3cf8 100644
--- a/libcxx/include/__ranges/copyable_box.h
+++ b/libcxx/include/__ranges/copyable_box.h
@@ -18,7 +18,6 @@
 #include <__memory/construct_at.h>
 #include <__utility/move.h>
 #include <optional>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/counted.h b/libcxx/include/__ranges/counted.h
index f50cd2b934dd5..882f90b1ed82e 100644
--- a/libcxx/include/__ranges/counted.h
+++ b/libcxx/include/__ranges/counted.h
@@ -19,10 +19,11 @@
 #include <__iterator/iterator_traits.h>
 #include <__memory/pointer_traits.h>
 #include <__ranges/subrange.h>
+#include <__type_traits/decay.h>
 #include <__utility/forward.h>
 #include <__utility/move.h>
+#include <cstddef>
 #include <span>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/drop_view.h b/libcxx/include/__ranges/drop_view.h
index 62de4458bb204..b97505b578552 100644
--- a/libcxx/include/__ranges/drop_view.h
+++ b/libcxx/include/__ranges/drop_view.h
@@ -33,10 +33,15 @@
 #include <__ranges/size.h>
 #include <__ranges/subrange.h>
 #include <__ranges/view_interface.h>
+#include <__type_traits/conditional.h>
+#include <__type_traits/decay.h>
+#include <__type_traits/is_nothrow_constructible.h>
+#include <__type_traits/make_unsigned.h>
+#include <__type_traits/remove_cvref.h>
 #include <__utility/auto_cast.h>
 #include <__utility/forward.h>
 #include <__utility/move.h>
-#include <type_traits>
+#include <cstddef>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/empty_view.h b/libcxx/include/__ranges/empty_view.h
index 1a48769d1b2fc..bb445b4e6f94f 100644
--- a/libcxx/include/__ranges/empty_view.h
+++ b/libcxx/include/__ranges/empty_view.h
@@ -13,7 +13,8 @@
 #include <__config>
 #include <__ranges/enable_borrowed_range.h>
 #include <__ranges/view_interface.h>
-#include <type_traits>
+#include <__type_traits/is_object.h>
+#include <cstddef>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/filter_view.h b/libcxx/include/__ranges/filter_view.h
index 895eb6e1fef27..bf1481b7f9156 100644
--- a/libcxx/include/__ranges/filter_view.h
+++ b/libcxx/include/__ranges/filter_view.h
@@ -35,7 +35,6 @@
 #include <__utility/forward.h>
 #include <__utility/in_place.h>
 #include <__utility/move.h>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/iota_view.h b/libcxx/include/__ranges/iota_view.h
index fae3b0b580c06..67cf0b73ecd65 100644
--- a/libcxx/include/__ranges/iota_view.h
+++ b/libcxx/include/__ranges/iota_view.h
@@ -30,9 +30,12 @@
 #include <__ranges/copyable_box.h>
 #include <__ranges/enable_borrowed_range.h>
 #include <__ranges/view_interface.h>
+#include <__type_traits/conditional.h>
+#include <__type_traits/is_nothrow_copy_constructible.h>
+#include <__type_traits/make_unsigned.h>
+#include <__type_traits/type_identity.h>
 #include <__utility/forward.h>
 #include <__utility/move.h>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/join_view.h b/libcxx/include/__ranges/join_view.h
index a9e5b998afa0b..83145ffb23a92 100644
--- a/libcxx/include/__ranges/join_view.h
+++ b/libcxx/include/__ranges/join_view.h
@@ -22,6 +22,7 @@
 #include <__iterator/iterator_traits.h>
 #include <__iterator/iterator_with_data.h>
 #include <__iterator/segmented_iterator.h>
+#include <__memory/addressof.h>
 #include <__ranges/access.h>
 #include <__ranges/all.h>
 #include <__ranges/concepts.h>
@@ -32,7 +33,6 @@
 #include <__type_traits/maybe_const.h>
 #include <__utility/forward.h>
 #include <optional>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/lazy_split_view.h b/libcxx/include/__ranges/lazy_split_view.h
index d84e4ce7990ca..b5b0e7ef02307 100644
--- a/libcxx/include/__ranges/lazy_split_view.h
+++ b/libcxx/include/__ranges/lazy_split_view.h
@@ -12,6 +12,7 @@
 
 #include <__algorithm/ranges_find.h>
 #include <__algorithm/ranges_mismatch.h>
+#include <__assert>
 #include <__concepts/constructible.h>
 #include <__concepts/convertible_to.h>
 #include <__concepts/derived_from.h>
@@ -34,10 +35,13 @@
 #include <__ranges/single_view.h>
 #include <__ranges/subrange.h>
 #include <__ranges/view_interface.h>
+#include <__type_traits/conditional.h>
+#include <__type_traits/decay.h>
+#include <__type_traits/is_nothrow_constructible.h>
 #include <__type_traits/maybe_const.h>
+#include <__type_traits/remove_reference.h>
 #include <__utility/forward.h>
 #include <__utility/move.h>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/non_propagating_cache.h b/libcxx/include/__ranges/non_propagating_cache.h
index 438d90d623dc7..2e6e72f94e177 100644
--- a/libcxx/include/__ranges/non_propagating_cache.h
+++ b/libcxx/include/__ranges/non_propagating_cache.h
@@ -16,7 +16,6 @@
 #include <__memory/addressof.h>
 #include <__utility/forward.h>
 #include <optional>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/owning_view.h b/libcxx/include/__ranges/owning_view.h
index ec46065d47b92..cb1d0d578279e 100644
--- a/libcxx/include/__ranges/owning_view.h
+++ b/libcxx/include/__ranges/owning_view.h
@@ -20,8 +20,8 @@
 #include <__ranges/enable_borrowed_range.h>
 #include <__ranges/size.h>
 #include <__ranges/view_interface.h>
+#include <__type_traits/remove_cvref.h>
 #include <__utility/move.h>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/range_adaptor.h b/libcxx/include/__ranges/range_adaptor.h
index fcb1f2f71214e..f8fd2f66baf3c 100644
--- a/libcxx/include/__ranges/range_adaptor.h
+++ b/libcxx/include/__ranges/range_adaptor.h
@@ -20,7 +20,6 @@
 #include <__ranges/concepts.h>
 #include <__utility/forward.h>
 #include <__utility/move.h>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/rbegin.h b/libcxx/include/__ranges/rbegin.h
index 720ae3d83355f..1ceb1116d695f 100644
--- a/libcxx/include/__ranges/rbegin.h
+++ b/libcxx/include/__ranges/rbegin.h
@@ -17,8 +17,11 @@
 #include <__iterator/readable_traits.h>
 #include <__iterator/reverse_iterator.h>
 #include <__ranges/access.h>
+#include <__type_traits/decay.h>
+#include <__type_traits/is_reference.h>
+#include <__type_traits/remove_cvref.h>
+#include <__type_traits/remove_reference.h>
 #include <__utility/auto_cast.h>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/ref_view.h b/libcxx/include/__ranges/ref_view.h
index a7fa4f27c3d26..be32b8124202e 100644
--- a/libcxx/include/__ranges/ref_view.h
+++ b/libcxx/include/__ranges/ref_view.h
@@ -24,8 +24,9 @@
 #include <__ranges/enable_borrowed_range.h>
 #include <__ranges/size.h>
 #include <__ranges/view_interface.h>
+#include <__type_traits/is_object.h>
+#include <__utility/declval.h>
 #include <__utility/forward.h>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/rend.h b/libcxx/include/__ranges/rend.h
index 15770d10c45c3..7ee574ccfa674 100644
--- a/libcxx/include/__ranges/rend.h
+++ b/libcxx/include/__ranges/rend.h
@@ -18,8 +18,11 @@
 #include <__iterator/reverse_iterator.h>
 #include <__ranges/access.h>
 #include <__ranges/rbegin.h>
+#include <__type_traits/decay.h>
+#include <__type_traits/is_reference.h>
+#include <__type_traits/remove_cvref.h>
+#include <__type_traits/remove_reference.h>
 #include <__utility/auto_cast.h>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/reverse_view.h b/libcxx/include/__ranges/reverse_view.h
index fc49eeadff236..01d1b97b3ce28 100644
--- a/libcxx/include/__ranges/reverse_view.h
+++ b/libcxx/include/__ranges/reverse_view.h
@@ -24,9 +24,10 @@
 #include <__ranges/size.h>
 #include <__ranges/subrange.h>
 #include <__ranges/view_interface.h>
+#include <__type_traits/conditional.h>
+#include <__type_traits/remove_cvref.h>
 #include <__utility/forward.h>
 #include <__utility/move.h>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/single_view.h b/libcxx/include/__ranges/single_view.h
index 7ea302a39b58f..ccab2c1664ac7 100644
--- a/libcxx/include/__ranges/single_view.h
+++ b/libcxx/include/__ranges/single_view.h
@@ -15,10 +15,12 @@
 #include <__ranges/copyable_box.h>
 #include <__ranges/range_adaptor.h>
 #include <__ranges/view_interface.h>
+#include <__type_traits/decay.h>
+#include <__type_traits/is_object.h>
 #include <__utility/forward.h>
 #include <__utility/in_place.h>
 #include <__utility/move.h>
-#include <type_traits>
+#include <cstddef>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/take_view.h b/libcxx/include/__ranges/take_view.h
index b8adedffa52ed..ec859e920ff17 100644
--- a/libcxx/include/__ranges/take_view.h
+++ b/libcxx/include/__ranges/take_view.h
@@ -34,11 +34,14 @@
 #include <__ranges/size.h>
 #include <__ranges/subrange.h>
 #include <__ranges/view_interface.h>
+#include <__type_traits/decay.h>
+#include <__type_traits/is_nothrow_constructible.h>
 #include <__type_traits/maybe_const.h>
+#include <__type_traits/remove_cvref.h>
 #include <__utility/auto_cast.h>
 #include <__utility/forward.h>
 #include <__utility/move.h>
-#include <type_traits>
+#include <cstddef>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/transform_view.h b/libcxx/include/__ranges/transform_view.h
index 9394fce6fc6c7..a71350f0c99dc 100644
--- a/libcxx/include/__ranges/transform_view.h
+++ b/libcxx/include/__ranges/transform_view.h
@@ -35,7 +35,6 @@
 #include <__utility/forward.h>
 #include <__utility/in_place.h>
 #include <__utility/move.h>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__ranges/zip_view.h b/libcxx/include/__ranges/zip_view.h
index b3cac719668c4..b2067f659e341 100644
--- a/libcxx/include/__ranges/zip_view.h
+++ b/libcxx/include/__ranges/zip_view.h
@@ -30,11 +30,11 @@
 #include <__ranges/enable_borrowed_range.h>
 #include <__ranges/size.h>
 #include <__ranges/view_interface.h>
+#include <__utility/declval.h>
 #include <__utility/forward.h>
 #include <__utility/integer_sequence.h>
 #include <__utility/move.h>
 #include <tuple>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__split_buffer b/libcxx/include/__split_buffer
index 86daddbb74bee..c4f1315ee5bf2 100644
--- a/libcxx/include/__split_buffer
+++ b/libcxx/include/__split_buffer
@@ -23,9 +23,18 @@
 #include <__memory/compressed_pair.h>
 #include <__memory/pointer_traits.h>
 #include <__memory/swap_allocator.h>
+#include <__type_traits/add_lvalue_reference.h>
+#include <__type_traits/enable_if.h>
+#include <__type_traits/integral_constant.h>
+#include <__type_traits/is_nothrow_default_constructible.h>
+#include <__type_traits/is_nothrow_move_assignable.h>
+#include <__type_traits/is_nothrow_move_constructible.h>
+#include <__type_traits/is_swappable.h>
+#include <__type_traits/is_trivially_destructible.h>
+#include <__type_traits/remove_reference.h>
 #include <__utility/forward.h>
 #include <__utility/move.h>
-#include <type_traits>
+#include <cstddef>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
diff --git a/libcxx/include/__tree b/libcxx/include/__tree
index 672b92c7796ea..8b2965ae9b51c 100644
--- a/libcxx/include/__tree
+++ b/libcxx/include/__tree
@@ -18,6 +18,7 @@
 #include <__iterator/distance.h>
 #include <__iterator/iterator_traits.h>
 #include <__iterator/next.h>
+#include <__memory/addressof.h>
 #include <__memory/allocator_traits.h>
 #include <__memory/compressed_pair.h>
 #include <__memory/pointer_traits.h>
diff --git a/libcxx/include/__type_traits/is_nothrow_convertible.h b/libcxx/include/__type_traits/is_nothrow_convertible.h
index 148c2d68ccabd..64f75e1fd68db 100644
--- a/libcxx/include/__type_traits/is_nothrow_convertible.h
+++ b/libcxx/include/__type_traits/is_nothrow_convertible.h
@@ -27,10 +27,10 @@ _LIBCPP_BEGIN_NAMESPACE_STD
 #if _LIBCPP_STD_VER >= 20
 
 template <typename _Tp>
-static void __test_noexcept(_Tp) noexcept;
+void __test_noexcept(_Tp) noexcept;
 
 template<typename _Fm, typename _To>
-static bool_constant<noexcept(_VSTD::__test_noexcept<_To>(std::declval<_Fm>()))>
+bool_constant<noexcept(_VSTD::__test_noexcept<_To>(std::declval<_Fm>()))>
 __is_nothrow_convertible_test();
 
 template <typename _Fm, typename _To>
diff --git a/libcxx/include/__type_traits/is_trivially_copyable.h b/libcxx/include/__type_traits/is_trivially_copyable.h
index 3e6d598bb2233..a725a0beb4ffa 100644
--- a/libcxx/include/__type_traits/is_trivially_copyable.h
+++ b/libcxx/include/__type_traits/is_trivially_copyable.h
@@ -11,6 +11,7 @@
 
 #include <__config>
 #include <__type_traits/integral_constant.h>
+#include <cstdint>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
@@ -27,6 +28,11 @@ template <class _Tp>
 inline constexpr bool is_trivially_copyable_v = __is_trivially_copyable(_Tp);
 #endif
 
+#if _LIBCPP_STD_VER >= 20
+template <class _Tp>
+inline constexpr bool __is_cheap_to_copy = is_trivially_copyable_v<_Tp> && sizeof(_Tp) <= sizeof(std::intmax_t);
+#endif
+
 _LIBCPP_END_NAMESPACE_STD
 
 #endif // _LIBCPP___TYPE_TRAITS_IS_TRIVIALLY_COPYABLE_H
diff --git a/libcxx/include/__utility/pair.h b/libcxx/include/__utility/pair.h
index 273c7e248aef8..30797005fc02f 100644
--- a/libcxx/include/__utility/pair.h
+++ b/libcxx/include/__utility/pair.h
@@ -424,22 +424,22 @@ pair(_T1, _T2) -> pair<_T1, _T2>;
 
 // [pairs.spec], specialized algorithms
 
-template <class _T1, class _T2>
+template <class _T1, class _T2, class _U1, class _U2>
 inline _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX14
 bool
-operator==(const pair<_T1,_T2>& __x, const pair<_T1,_T2>& __y)
+operator==(const pair<_T1,_T2>& __x, const pair<_U1,_U2>& __y)
 {
     return __x.first == __y.first && __x.second == __y.second;
 }
 
 #if _LIBCPP_STD_VER >= 20
 
-template <class _T1, class _T2>
+template <class _T1, class _T2, class _U1, class _U2>
 _LIBCPP_HIDE_FROM_ABI constexpr
 common_comparison_category_t<
-        __synth_three_way_result<_T1>,
-        __synth_three_way_result<_T2> >
-operator<=>(const pair<_T1,_T2>& __x, const pair<_T1,_T2>& __y)
+        __synth_three_way_result<_T1, _U1>,
+        __synth_three_way_result<_T2, _U2> >
+operator<=>(const pair<_T1,_T2>& __x, const pair<_U1,_U2>& __y)
 {
     if (auto __c = std::__synth_three_way(__x.first, __y.first); __c != 0) {
       return __c;
@@ -449,42 +449,42 @@ operator<=>(const pair<_T1,_T2>& __x, const pair<_T1,_T2>& __y)
 
 #else // _LIBCPP_STD_VER >= 20
 
-template <class _T1, class _T2>
+template <class _T1, class _T2, class _U1, class _U2>
 inline _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX14
 bool
-operator!=(const pair<_T1,_T2>& __x, const pair<_T1,_T2>& __y)
+operator!=(const pair<_T1,_T2>& __x, const pair<_U1,_U2>& __y)
 {
     return !(__x == __y);
 }
 
-template <class _T1, class _T2>
+template <class _T1, class _T2, class _U1, class _U2>
 inline _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX14
 bool
-operator< (const pair<_T1,_T2>& __x, const pair<_T1,_T2>& __y)
+operator< (const pair<_T1,_T2>& __x, const pair<_U1,_U2>& __y)
 {
     return __x.first < __y.first || (!(__y.first < __x.first) && __x.second < __y.second);
 }
 
-template <class _T1, class _T2>
+template <class _T1, class _T2, class _U1, class _U2>
 inline _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX14
 bool
-operator> (const pair<_T1,_T2>& __x, const pair<_T1,_T2>& __y)
+operator> (const pair<_T1,_T2>& __x, const pair<_U1,_U2>& __y)
 {
     return __y < __x;
 }
 
-template <class _T1, class _T2>
+template <class _T1, class _T2, class _U1, class _U2>
 inline _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX14
 bool
-operator>=(const pair<_T1,_T2>& __x, const pair<_T1,_T2>& __y)
+operator>=(const pair<_T1,_T2>& __x, const pair<_U1,_U2>& __y)
 {
     return !(__x < __y);
 }
 
-template <class _T1, class _T2>
+template <class _T1, class _T2, class _U1, class _U2>
 inline _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX14
 bool
-operator<=(const pair<_T1,_T2>& __x, const pair<_T1,_T2>& __y)
+operator<=(const pair<_T1,_T2>& __x, const pair<_U1,_U2>& __y)
 {
     return !(__y < __x);
 }
diff --git a/libcxx/include/__utility/piecewise_construct.h b/libcxx/include/__utility/piecewise_construct.h
index 3cc86fef2a0fb..071e207bd07ca 100644
--- a/libcxx/include/__utility/piecewise_construct.h
+++ b/libcxx/include/__utility/piecewise_construct.h
@@ -18,9 +18,10 @@
 _LIBCPP_BEGIN_NAMESPACE_STD
 
 struct _LIBCPP_TEMPLATE_VIS piecewise_construct_t { explicit piecewise_construct_t() = default; };
-#if defined(_LIBCPP_CXX03_LANG) || defined(_LIBCPP_BUILDING_LIBRARY)
-extern _LIBCPP_EXPORTED_FROM_ABI const piecewise_construct_t piecewise_construct;// = piecewise_construct_t();
-#else
+
+#if defined(_LIBCPP_BUILDING_LIBRARY)
+extern _LIBCPP_EXPORTED_FROM_ABI const piecewise_construct_t piecewise_construct;
+#elif !defined(_LIBCPP_CXX03_LANG)
 /* inline */ constexpr piecewise_construct_t piecewise_construct = piecewise_construct_t();
 #endif
 
diff --git a/libcxx/include/__verbose_abort b/libcxx/include/__verbose_abort
index a16d75d5ac4c3..40b0e07da5489 100644
--- a/libcxx/include/__verbose_abort
+++ b/libcxx/include/__verbose_abort
@@ -46,7 +46,8 @@ void __libcpp_verbose_abort(const char *__format, ...);
 #  endif
 
 #  if defined(_LIBCPP_HAS_NO_VERBOSE_ABORT_IN_LIBRARY)
-#    define _LIBCPP_VERBOSE_ABORT(...) __builtin_abort()
+// The decltype is there to suppress -Wunused warnings in this configuration.
+#    define _LIBCPP_VERBOSE_ABORT(...) (decltype(std::__libcpp_verbose_abort(__VA_ARGS__))(), __builtin_abort())
 #  else
 #    define _LIBCPP_VERBOSE_ABORT(...) ::std::__libcpp_verbose_abort(__VA_ARGS__)
 #  endif
diff --git a/libcxx/include/any b/libcxx/include/any
index 423628736b382..e139756f65f4c 100644
--- a/libcxx/include/any
+++ b/libcxx/include/any
@@ -87,13 +87,28 @@ namespace std {
 #include <__memory/allocator_destructor.h>
 #include <__memory/allocator_traits.h>
 #include <__memory/unique_ptr.h>
+#include <__type_traits/add_const.h>
+#include <__type_traits/add_pointer.h>
+#include <__type_traits/aligned_storage.h>
+#include <__type_traits/alignment_of.h>
+#include <__type_traits/conditional.h>
+#include <__type_traits/decay.h>
+#include <__type_traits/is_constructible.h>
+#include <__type_traits/is_copy_constructible.h>
+#include <__type_traits/is_function.h>
+#include <__type_traits/is_nothrow_move_constructible.h>
+#include <__type_traits/is_reference.h>
+#include <__type_traits/is_same.h>
+#include <__type_traits/remove_cv.h>
+#include <__type_traits/remove_cvref.h>
+#include <__type_traits/remove_reference.h>
 #include <__utility/forward.h>
 #include <__utility/in_place.h>
 #include <__utility/move.h>
 #include <__utility/unreachable.h>
+#include <__verbose_abort>
 #include <cstdlib>
 #include <initializer_list>
-#include <type_traits>
 #include <typeinfo>
 #include <version>
 
@@ -120,7 +135,7 @@ void __throw_bad_any_cast()
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
     throw bad_any_cast();
 #else
-    _VSTD::abort();
+    _LIBCPP_VERBOSE_ABORT("bad_any_cast was thrown in -fno-exceptions mode");
 #endif
 }
 
@@ -706,6 +721,7 @@ _LIBCPP_END_NAMESPACE_STD
 #  include <iosfwd>
 #  include <iterator>
 #  include <memory>
+#  include <type_traits>
 #  include <variant>
 #endif
 
diff --git a/libcxx/include/array b/libcxx/include/array
index 2a8d0dbc598f0..218ad3dd186f5 100644
--- a/libcxx/include/array
+++ b/libcxx/include/array
@@ -116,11 +116,20 @@ template <size_t I, class T, size_t N> const T&& get(const array<T, N>&&) noexce
 #include <__config>
 #include <__iterator/reverse_iterator.h>
 #include <__tuple_dir/sfinae_helpers.h>
+#include <__type_traits/conditional.h>
+#include <__type_traits/is_array.h>
+#include <__type_traits/is_const.h>
+#include <__type_traits/is_constructible.h>
+#include <__type_traits/is_move_constructible.h>
+#include <__type_traits/is_nothrow_constructible.h>
+#include <__type_traits/is_nothrow_move_constructible.h>
+#include <__type_traits/is_same.h>
+#include <__type_traits/is_swappable.h>
+#include <__type_traits/remove_cv.h>
 #include <__utility/integer_sequence.h>
 #include <__utility/move.h>
 #include <__utility/unreachable.h>
 #include <stdexcept>
-#include <type_traits>
 #include <version>
 
 // standard-mandated includes
@@ -536,6 +545,7 @@ _LIBCPP_END_NAMESPACE_STD
 #  include <algorithm>
 #  include <concepts>
 #  include <iterator>
+#  include <type_traits>
 #  include <utility>
 #endif
 
diff --git a/libcxx/include/atomic b/libcxx/include/atomic
index 3ebb2a93298b6..2f122a707bdc3 100644
--- a/libcxx/include/atomic
+++ b/libcxx/include/atomic
@@ -534,29 +534,24 @@ template <class T>
 #include <__atomic/kill_dependency.h>
 #include <__atomic/memory_order.h>
 #include <__config>
-#include <cstddef>
-#include <cstdint>
-#include <cstring>
 #include <version>
 
-#ifndef _LIBCPP_HAS_NO_THREADS
-# include <__threading_support>
-#endif
-
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
 #endif
 
 #ifdef _LIBCPP_HAS_NO_ATOMIC_HEADER
-# error <atomic> is not implemented
+#  error <atomic> is not implemented
 #endif
+
 #ifdef kill_dependency
-# error <atomic> is incompatible with <stdatomic.h> before C++23. Please compile with -std=c++23.
+#  error <atomic> is incompatible with <stdatomic.h> before C++23. Please compile with -std=c++23.
 #endif
 
 #if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
 #  include <cmath>
 #  include <compare>
+#  include <cstring>
 #  include <type_traits>
 #endif
 
diff --git a/libcxx/include/barrier b/libcxx/include/barrier
index 2e8906b71379c..41b3d8e0b7b40 100644
--- a/libcxx/include/barrier
+++ b/libcxx/include/barrier
@@ -46,13 +46,18 @@ namespace std
 */
 
 #include <__assert> // all public C++ headers provide the assertion handler
+#include <__atomic/atomic_base.h>
+#include <__atomic/memory_order.h>
 #include <__availability>
 #include <__config>
 #include <__memory/unique_ptr.h>
+#include <__thread/poll_with_backoff.h>
 #include <__thread/timed_backoff_policy.h>
 #include <__utility/move.h>
-#include <atomic>
+#include <cstddef>
+#include <cstdint>
 #include <limits>
+#include <version>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
@@ -329,6 +334,7 @@ _LIBCPP_END_NAMESPACE_STD
 _LIBCPP_POP_MACROS
 
 #if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <atomic>
 #  include <concepts>
 #  include <iterator>
 #  include <memory>
diff --git a/libcxx/include/bitset b/libcxx/include/bitset
index daf3d1ad2f733..ac866fa6e0c39 100644
--- a/libcxx/include/bitset
+++ b/libcxx/include/bitset
@@ -1154,6 +1154,7 @@ _LIBCPP_POP_MACROS
 
 #if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
 #  include <concepts>
+#  include <type_traits>
 #endif
 
 #endif // _LIBCPP_BITSET
diff --git a/libcxx/include/charconv b/libcxx/include/charconv
index df444d7e7a289..c30210fb52b0a 100644
--- a/libcxx/include/charconv
+++ b/libcxx/include/charconv
@@ -67,771 +67,38 @@ namespace std {
 
 */
 
-#include <__algorithm/copy_n.h>
 #include <__assert> // all public C++ headers provide the assertion handler
-#include <__availability>
-#include <__bit/countl.h>
 #include <__charconv/chars_format.h>
+#include <__charconv/from_chars_integral.h>
 #include <__charconv/from_chars_result.h>
 #include <__charconv/tables.h>
+#include <__charconv/to_chars.h>
 #include <__charconv/to_chars_base_10.h>
+#include <__charconv/to_chars_floating_point.h>
+#include <__charconv/to_chars_integral.h>
 #include <__charconv/to_chars_result.h>
+#include <__charconv/traits.h>
 #include <__config>
 #include <__debug>
 #include <__errc>
-#include <__memory/addressof.h>
-#include <__type_traits/make_32_64_or_128_bit.h>
-#include <__utility/unreachable.h>
 #include <cmath> // for log2f
 #include <cstdint>
 #include <cstdlib>
 #include <limits>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
 #endif
 
-_LIBCPP_PUSH_MACROS
-#include <__undef_macros>
-
 _LIBCPP_BEGIN_NAMESPACE_STD
 
-#if _LIBCPP_STD_VER >= 17
-
-to_chars_result to_chars(char*, char*, bool, int = 10) = delete;
-from_chars_result from_chars(const char*, const char*, bool, int = 10) = delete;
-
-namespace __itoa
-{
-
-template <typename _Tp, typename = void>
-struct _LIBCPP_HIDDEN __traits_base;
-
-template <typename _Tp>
-struct _LIBCPP_HIDDEN __traits_base<_Tp, __enable_if_t<sizeof(_Tp) <= sizeof(uint32_t)>>
-{
-    using type = uint32_t;
-
-    /// The width estimation using a log10 algorithm.
-    ///
-    /// The algorithm is based on
-    /// http://graphics.stanford.edu/~seander/bithacks.html#IntegerLog10
-    /// Instead of using IntegerLogBase2 it uses __libcpp_clz. Since that
-    /// function requires its input to have at least one bit set the value of
-    /// zero is set to one. This means the first element of the lookup table is
-    /// zero.
-    static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI int __width(_Tp __v)
-    {
-        auto __t = (32 - std::__libcpp_clz(static_cast<type>(__v | 1))) * 1233 >> 12;
-        return __t - (__v < __itoa::__pow10_32[__t]) + 1;
-    }
-
-    static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI char* __convert(char* __p, _Tp __v)
-    {
-        return __itoa::__base_10_u32(__p, __v);
-    }
-
-    static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI decltype(__pow10_32)& __pow() { return __itoa::__pow10_32; }
-};
-
-template <typename _Tp>
-struct _LIBCPP_HIDDEN
-    __traits_base<_Tp, __enable_if_t<sizeof(_Tp) == sizeof(uint64_t)>> {
-  using type = uint64_t;
-
-  /// The width estimation using a log10 algorithm.
-  ///
-  /// The algorithm is based on
-  /// http://graphics.stanford.edu/~seander/bithacks.html#IntegerLog10
-  /// Instead of using IntegerLogBase2 it uses __libcpp_clz. Since that
-  /// function requires its input to have at least one bit set the value of
-  /// zero is set to one. This means the first element of the lookup table is
-  /// zero.
-  static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI int __width(_Tp __v) {
-    auto __t = (64 - std::__libcpp_clz(static_cast<type>(__v | 1))) * 1233 >> 12;
-    return __t - (__v < __itoa::__pow10_64[__t]) + 1;
-  }
-
-  static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI char* __convert(char* __p, _Tp __v) { return __itoa::__base_10_u64(__p, __v); }
-
-  static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI decltype(__pow10_64)& __pow() { return __itoa::__pow10_64; }
-};
-
-
-#  ifndef _LIBCPP_HAS_NO_INT128
-template <typename _Tp>
-struct _LIBCPP_HIDDEN
-    __traits_base<_Tp, __enable_if_t<sizeof(_Tp) == sizeof(__uint128_t)> > {
-  using type = __uint128_t;
-
-  /// The width estimation using a log10 algorithm.
-  ///
-  /// The algorithm is based on
-  /// http://graphics.stanford.edu/~seander/bithacks.html#IntegerLog10
-  /// Instead of using IntegerLogBase2 it uses __libcpp_clz. Since that
-  /// function requires its input to have at least one bit set the value of
-  /// zero is set to one. This means the first element of the lookup table is
-  /// zero.
-  static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI int __width(_Tp __v) {
-    _LIBCPP_ASSERT(__v > numeric_limits<uint64_t>::max(), "The optimizations for this algorithm fail when this isn't true.");
-    // There's always a bit set in the upper 64-bits.
-    auto __t = (128 - std::__libcpp_clz(static_cast<uint64_t>(__v >> 64))) * 1233 >> 12;
-    _LIBCPP_ASSERT(__t >= __itoa::__pow10_128_offset, "Index out of bounds");
-    // __t is adjusted since the lookup table misses the lower entries.
-    return __t - (__v < __itoa::__pow10_128[__t - __itoa::__pow10_128_offset]) + 1;
-  }
-
-  static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI char* __convert(char* __p, _Tp __v) { return __itoa::__base_10_u128(__p, __v); }
-
-  // TODO FMT This pow function should get an index.
-  // By moving this to its own header it can be reused by the pow function in to_chars_base_10.
-  static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI decltype(__pow10_128)& __pow() { return __itoa::__pow10_128; }
-};
-#endif
-
-template <typename _Tp>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI bool
-__mul_overflowed(unsigned char __a, _Tp __b, unsigned char& __r)
-{
-    auto __c = __a * __b;
-    __r = __c;
-    return __c > numeric_limits<unsigned char>::max();
-}
-
-template <typename _Tp>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI bool
-__mul_overflowed(unsigned short __a, _Tp __b, unsigned short& __r)
-{
-    auto __c = __a * __b;
-    __r = __c;
-    return __c > numeric_limits<unsigned short>::max();
-}
-
-template <typename _Tp>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI bool
-__mul_overflowed(_Tp __a, _Tp __b, _Tp& __r)
-{
-    static_assert(is_unsigned<_Tp>::value, "");
-    return __builtin_mul_overflow(__a, __b, &__r);
-}
-
-template <typename _Tp, typename _Up>
-inline _LIBCPP_HIDE_FROM_ABI bool
-_LIBCPP_CONSTEXPR_SINCE_CXX23 __mul_overflowed(_Tp __a, _Up __b, _Tp& __r)
-{
-    return __itoa::__mul_overflowed(__a, static_cast<_Tp>(__b), __r);
-}
-
-template <typename _Tp>
-struct _LIBCPP_HIDDEN __traits : __traits_base<_Tp>
-{
-    static constexpr int digits = numeric_limits<_Tp>::digits10 + 1;
-    using __traits_base<_Tp>::__pow;
-    using typename __traits_base<_Tp>::type;
-
-    // precondition: at least one non-zero character available
-    static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI char const*
-    __read(char const* __p, char const* __ep, type& __a, type& __b)
-    {
-        type __cprod[digits];
-        int __j = digits - 1;
-        int __i = digits;
-        do
-        {
-            if (*__p < '0' || *__p > '9')
-                break;
-            __cprod[--__i] = *__p++ - '0';
-        } while (__p != __ep && __i != 0);
-
-        __a = __inner_product(__cprod + __i + 1, __cprod + __j, __pow() + 1,
-                              __cprod[__i]);
-        if (__itoa::__mul_overflowed(__cprod[__j], __pow()[__j - __i], __b))
-            --__p;
-        return __p;
-    }
-
-    template <typename _It1, typename _It2, class _Up>
-    static _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI _Up
-    __inner_product(_It1 __first1, _It1 __last1, _It2 __first2, _Up __init)
-    {
-        for (; __first1 < __last1; ++__first1, ++__first2)
-            __init = __init + *__first1 * *__first2;
-        return __init;
-    }
-};
-
-}  // namespace __itoa
-
-template <typename _Tp>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI _Tp
-__complement(_Tp __x)
-{
-    static_assert(is_unsigned<_Tp>::value, "cast to unsigned first");
-    return _Tp(~__x + 1);
-}
-
-template <typename _Tp>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
-__to_chars_itoa(char* __first, char* __last, _Tp __value, false_type);
-
-template <typename _Tp>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
-__to_chars_itoa(char* __first, char* __last, _Tp __value, true_type)
-{
-    auto __x = std::__to_unsigned_like(__value);
-    if (__value < 0 && __first != __last)
-    {
-        *__first++ = '-';
-        __x = std::__complement(__x);
-    }
-
-    return std::__to_chars_itoa(__first, __last, __x, false_type());
-}
-
-template <typename _Tp>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
-__to_chars_itoa(char* __first, char* __last, _Tp __value, false_type)
-{
-    using __tx = __itoa::__traits<_Tp>;
-    auto __diff = __last - __first;
-
-    if (__tx::digits <= __diff || __tx::__width(__value) <= __diff)
-        return {__tx::__convert(__first, __value), errc(0)};
-    else
-        return {__last, errc::value_too_large};
-}
-
-#  ifndef _LIBCPP_HAS_NO_INT128
-template <>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
-__to_chars_itoa(char* __first, char* __last, __uint128_t __value, false_type)
-{
-    // When the value fits in 64-bits use the 64-bit code path. This reduces
-    // the number of expensive calculations on 128-bit values.
-    //
-    // NOTE the 128-bit code path requires this optimization.
-    if(__value <= numeric_limits<uint64_t>::max())
-        return __to_chars_itoa(__first, __last, static_cast<uint64_t>(__value), false_type());
-
-    using __tx = __itoa::__traits<__uint128_t>;
-    auto __diff = __last - __first;
-
-    if (__tx::digits <= __diff || __tx::__width(__value) <= __diff)
-        return {__tx::__convert(__first, __value), errc(0)};
-    else
-        return {__last, errc::value_too_large};
-}
-#endif
-
-template <class _Tp>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
-__to_chars_integral(char* __first, char* __last, _Tp __value, int __base, false_type);
-
-template <typename _Tp>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
-__to_chars_integral(char* __first, char* __last, _Tp __value, int __base,
-                    true_type)
-{
-    auto __x = std::__to_unsigned_like(__value);
-    if (__value < 0 && __first != __last)
-    {
-        *__first++ = '-';
-        __x = std::__complement(__x);
-    }
-
-    return std::__to_chars_integral(__first, __last, __x, __base, false_type());
-}
-
-namespace __itoa {
-
-template <unsigned _Base>
-struct _LIBCPP_HIDDEN __integral;
-
-template <>
-struct _LIBCPP_HIDDEN __integral<2> {
-  template <typename _Tp>
-  _LIBCPP_HIDE_FROM_ABI static constexpr int __width(_Tp __value) noexcept {
-    // If value == 0 still need one digit. If the value != this has no
-    // effect since the code scans for the most significant bit set. (Note
-    // that __libcpp_clz doesn't work for 0.)
-    return numeric_limits<_Tp>::digits - std::__libcpp_clz(__value | 1);
-  }
-
-  template <typename _Tp>
-  _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI static to_chars_result __to_chars(char* __first, char* __last, _Tp __value) {
-    ptrdiff_t __cap = __last - __first;
-    int __n = __width(__value);
-    if (__n > __cap)
-      return {__last, errc::value_too_large};
-
-    __last = __first + __n;
-    char* __p = __last;
-    const unsigned __divisor = 16;
-    while (__value > __divisor) {
-      unsigned __c = __value % __divisor;
-      __value /= __divisor;
-      __p -= 4;
-      std::copy_n(&__base_2_lut[4 * __c], 4, __p);
-    }
-    do {
-      unsigned __c = __value % 2;
-      __value /= 2;
-      *--__p = "01"[__c];
-    } while (__value != 0);
-    return {__last, errc(0)};
-  }
-};
-
-template <>
-struct _LIBCPP_HIDDEN __integral<8> {
-  template <typename _Tp>
-  _LIBCPP_HIDE_FROM_ABI static constexpr int __width(_Tp __value) noexcept {
-    // If value == 0 still need one digit. If the value != this has no
-    // effect since the code scans for the most significat bit set. (Note
-    // that __libcpp_clz doesn't work for 0.)
-    return ((numeric_limits<_Tp>::digits - std::__libcpp_clz(__value | 1)) + 2) / 3;
-  }
-
-  template <typename _Tp>
-  _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI static to_chars_result __to_chars(char* __first, char* __last, _Tp __value) {
-    ptrdiff_t __cap = __last - __first;
-    int __n = __width(__value);
-    if (__n > __cap)
-      return {__last, errc::value_too_large};
-
-    __last = __first + __n;
-    char* __p = __last;
-    unsigned __divisor = 64;
-    while (__value > __divisor) {
-      unsigned __c = __value % __divisor;
-      __value /= __divisor;
-      __p -= 2;
-      std::copy_n(&__base_8_lut[2 * __c], 2, __p);
-    }
-    do {
-      unsigned __c = __value % 8;
-      __value /= 8;
-      *--__p = "01234567"[__c];
-    } while (__value != 0);
-    return {__last, errc(0)};
-  }
-
-};
-
-template <>
-struct _LIBCPP_HIDDEN __integral<16> {
-  template <typename _Tp>
-  _LIBCPP_HIDE_FROM_ABI static constexpr int __width(_Tp __value) noexcept {
-    // If value == 0 still need one digit. If the value != this has no
-    // effect since the code scans for the most significat bit set. (Note
-    // that __libcpp_clz doesn't work for 0.)
-    return (numeric_limits<_Tp>::digits - std::__libcpp_clz(__value | 1) + 3) / 4;
-  }
-
-  template <typename _Tp>
-  _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI static to_chars_result __to_chars(char* __first, char* __last, _Tp __value) {
-    ptrdiff_t __cap = __last - __first;
-    int __n = __width(__value);
-    if (__n > __cap)
-      return {__last, errc::value_too_large};
-
-    __last = __first + __n;
-    char* __p = __last;
-    unsigned __divisor = 256;
-    while (__value > __divisor) {
-      unsigned __c = __value % __divisor;
-      __value /= __divisor;
-      __p -= 2;
-      std::copy_n(&__base_16_lut[2 * __c], 2, __p);
-    }
-    if (__first != __last)
-      do {
-        unsigned __c = __value % 16;
-        __value /= 16;
-        *--__p = "0123456789abcdef"[__c];
-      } while (__value != 0);
-    return {__last, errc(0)};
-  }
-};
-
-} // namespace __itoa
-
-template <unsigned _Base, typename _Tp,
-          typename enable_if<(sizeof(_Tp) >= sizeof(unsigned)), int>::type = 0>
-_LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI int
-__to_chars_integral_width(_Tp __value) {
-  return __itoa::__integral<_Base>::__width(__value);
-}
-
-template <unsigned _Base, typename _Tp,
-          typename enable_if<(sizeof(_Tp) < sizeof(unsigned)), int>::type = 0>
-_LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI int
-__to_chars_integral_width(_Tp __value) {
-  return std::__to_chars_integral_width<_Base>(static_cast<unsigned>(__value));
-}
-
-template <unsigned _Base, typename _Tp,
-          typename enable_if<(sizeof(_Tp) >= sizeof(unsigned)), int>::type = 0>
-_LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
-__to_chars_integral(char* __first, char* __last, _Tp __value) {
-  return __itoa::__integral<_Base>::__to_chars(__first, __last, __value);
-}
-
-template <unsigned _Base, typename _Tp,
-          typename enable_if<(sizeof(_Tp) < sizeof(unsigned)), int>::type = 0>
-_LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
-__to_chars_integral(char* __first, char* __last, _Tp __value) {
-  return std::__to_chars_integral<_Base>(__first, __last, static_cast<unsigned>(__value));
-}
-
-template <typename _Tp>
-_LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI int
-__to_chars_integral_width(_Tp __value, unsigned __base) {
-  _LIBCPP_ASSERT(__value >= 0, "The function requires a non-negative value.");
-
-  unsigned __base_2 = __base * __base;
-  unsigned __base_3 = __base_2 * __base;
-  unsigned __base_4 = __base_2 * __base_2;
-
-  int __r = 0;
-  while (true) {
-    if (__value < __base)
-      return __r + 1;
-    if (__value < __base_2)
-      return __r + 2;
-    if (__value < __base_3)
-      return __r + 3;
-    if (__value < __base_4)
-      return __r + 4;
-
-    __value /= __base_4;
-    __r += 4;
-  }
-
-  __libcpp_unreachable();
-}
-
-template <typename _Tp>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
-__to_chars_integral(char* __first, char* __last, _Tp __value, int __base,
-                    false_type)
-{
-  if (__base == 10) [[likely]]
-    return std::__to_chars_itoa(__first, __last, __value, false_type());
-
-  switch (__base) {
-  case 2:
-    return std::__to_chars_integral<2>(__first, __last, __value);
-  case 8:
-    return std::__to_chars_integral<8>(__first, __last, __value);
-  case 16:
-    return std::__to_chars_integral<16>(__first, __last, __value);
-  }
-
-  ptrdiff_t __cap = __last - __first;
-  int __n = std::__to_chars_integral_width(__value, __base);
-  if (__n > __cap)
-    return {__last, errc::value_too_large};
-
-  __last = __first + __n;
-  char* __p = __last;
-  do {
-    unsigned __c = __value % __base;
-    __value /= __base;
-    *--__p = "0123456789abcdefghijklmnopqrstuvwxyz"[__c];
-  } while (__value != 0);
-  return {__last, errc(0)};
-}
-
-template <typename _Tp, typename enable_if<is_integral<_Tp>::value, int>::type = 0>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
-to_chars(char* __first, char* __last, _Tp __value)
-{
-  using _Type = __make_32_64_or_128_bit_t<_Tp>;
-  static_assert(!is_same<_Type, void>::value, "unsupported integral type used in to_chars");
-  return std::__to_chars_itoa(__first, __last, static_cast<_Type>(__value), is_signed<_Tp>());
-}
-
-template <typename _Tp, typename enable_if<is_integral<_Tp>::value, int>::type = 0>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI to_chars_result
-to_chars(char* __first, char* __last, _Tp __value, int __base)
-{
-  _LIBCPP_ASSERT(2 <= __base && __base <= 36, "base not in [2, 36]");
-
-  using _Type = __make_32_64_or_128_bit_t<_Tp>;
-  return std::__to_chars_integral(__first, __last, static_cast<_Type>(__value), __base, is_signed<_Tp>());
-}
-
-template <typename _It, typename _Tp, typename _Fn, typename... _Ts>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI from_chars_result
-__sign_combinator(_It __first, _It __last, _Tp& __value, _Fn __f, _Ts... __args)
-{
-    using __tl = numeric_limits<_Tp>;
-    decltype(std::__to_unsigned_like(__value)) __x;
-
-    bool __neg = (__first != __last && *__first == '-');
-    auto __r = __f(__neg ? __first + 1 : __first, __last, __x, __args...);
-    switch (__r.ec)
-    {
-    case errc::invalid_argument:
-        return {__first, __r.ec};
-    case errc::result_out_of_range:
-        return __r;
-    default:
-        break;
-    }
-
-    if (__neg)
-    {
-        if (__x <= std::__complement(std::__to_unsigned_like(__tl::min())))
-        {
-            __x = std::__complement(__x);
-            std::copy_n(std::addressof(__x), 1, std::addressof(__value));
-            return __r;
-        }
-    }
-    else
-    {
-        if (__x <= std::__to_unsigned_like(__tl::max()))
-        {
-            __value = __x;
-            return __r;
-        }
-    }
-
-    return {__r.ptr, errc::result_out_of_range};
-}
-
-template <typename _Tp>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI bool
-__in_pattern(_Tp __c)
-{
-    return '0' <= __c && __c <= '9';
-}
-
-struct _LIBCPP_HIDDEN __in_pattern_result
-{
-    bool __ok;
-    int __val;
-
-    explicit _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI operator bool() const { return __ok; }
-};
-
-template <typename _Tp>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI __in_pattern_result
-__in_pattern(_Tp __c, int __base)
-{
-    if (__base <= 10)
-        return {'0' <= __c && __c < '0' + __base, __c - '0'};
-    else if (std::__in_pattern(__c))
-        return {true, __c - '0'};
-    else if ('a' <= __c && __c < 'a' + __base - 10)
-        return {true, __c - 'a' + 10};
-    else
-        return {'A' <= __c && __c < 'A' + __base - 10, __c - 'A' + 10};
-}
-
-template <typename _It, typename _Tp, typename _Fn, typename... _Ts>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI from_chars_result
-__subject_seq_combinator(_It __first, _It __last, _Tp& __value, _Fn __f,
-                         _Ts... __args)
-{
-    auto __find_non_zero = [](_It __firstit, _It __lastit) {
-        for (; __firstit != __lastit; ++__firstit)
-            if (*__firstit != '0')
-                break;
-        return __firstit;
-    };
-
-    auto __p = __find_non_zero(__first, __last);
-    if (__p == __last || !std::__in_pattern(*__p, __args...))
-    {
-        if (__p == __first)
-            return {__first, errc::invalid_argument};
-        else
-        {
-            __value = 0;
-            return {__p, {}};
-        }
-    }
-
-    auto __r = __f(__p, __last, __value, __args...);
-    if (__r.ec == errc::result_out_of_range)
-    {
-        for (; __r.ptr != __last; ++__r.ptr)
-        {
-            if (!std::__in_pattern(*__r.ptr, __args...))
-                break;
-        }
-    }
-
-    return __r;
-}
-
-template <typename _Tp, typename enable_if<is_unsigned<_Tp>::value, int>::type = 0>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI from_chars_result
-__from_chars_atoi(const char* __first, const char* __last, _Tp& __value)
-{
-    using __tx = __itoa::__traits<_Tp>;
-    using __output_type = typename __tx::type;
-
-    return std::__subject_seq_combinator(
-        __first, __last, __value,
-        [](const char* __f, const char* __l,
-           _Tp& __val) -> from_chars_result {
-            __output_type __a, __b;
-            auto __p = __tx::__read(__f, __l, __a, __b);
-            if (__p == __l || !std::__in_pattern(*__p))
-            {
-                __output_type __m = numeric_limits<_Tp>::max();
-                if (__m >= __a && __m - __a >= __b)
-                {
-                    __val = __a + __b;
-                    return {__p, {}};
-                }
-            }
-            return {__p, errc::result_out_of_range};
-        });
-}
-
-template <typename _Tp, typename enable_if<is_signed<_Tp>::value, int>::type = 0>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI from_chars_result
-__from_chars_atoi(const char* __first, const char* __last, _Tp& __value)
-{
-    using __t = decltype(std::__to_unsigned_like(__value));
-    return std::__sign_combinator(__first, __last, __value, __from_chars_atoi<__t>);
-}
-
-
-/*
-// Code used to generate __from_chars_log2f_lut.
-#include <cmath>
-#include <iostream>
-#include <format>
-
-int main() {
-  for (int i = 2; i <= 36; ++i)
-    std::cout << std::format("{},\n", log2f(i));
-}
-*/
-/// log2f table for bases [2, 36].
-inline constexpr float __from_chars_log2f_lut[35] = {
-    1,         1.5849625, 2,         2.321928, 2.5849626, 2.807355, 3,        3.169925,  3.321928,
-    3.4594316, 3.5849626, 3.7004397, 3.807355, 3.9068906, 4,        4.087463, 4.169925,  4.2479277,
-    4.321928,  4.3923173, 4.4594316, 4.523562, 4.5849624, 4.643856, 4.70044,  4.7548876, 4.807355,
-    4.857981,  4.9068904, 4.9541965, 5,        5.044394,  5.087463, 5.129283, 5.169925};
-
-template <typename _Tp, typename enable_if<is_unsigned<_Tp>::value, int>::type = 0>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI from_chars_result
-__from_chars_integral(const char* __first, const char* __last, _Tp& __value,
-                      int __base)
-{
-    if (__base == 10)
-        return std::__from_chars_atoi(__first, __last, __value);
-
-    return std::__subject_seq_combinator(
-        __first, __last, __value,
-        [](const char* __p, const char* __lastp, _Tp& __val,
-           int __b) -> from_chars_result {
-            using __tl = numeric_limits<_Tp>;
-            // __base is always between 2 and 36 inclusive.
-            auto __digits = __tl::digits / __from_chars_log2f_lut[__b - 2];
-            _Tp __x = __in_pattern(*__p++, __b).__val, __y = 0;
-
-            for (int __i = 1; __p != __lastp; ++__i, ++__p)
-            {
-                if (auto __c = __in_pattern(*__p, __b))
-                {
-                    if (__i < __digits - 1)
-                        __x = __x * __b + __c.__val;
-                    else
-                    {
-                        if (!__itoa::__mul_overflowed(__x, __b, __x))
-                            ++__p;
-                        __y = __c.__val;
-                        break;
-                    }
-                }
-                else
-                    break;
-            }
-
-            if (__p == __lastp || !__in_pattern(*__p, __b))
-            {
-                if (__tl::max() - __x >= __y)
-                {
-                    __val = __x + __y;
-                    return {__p, {}};
-                }
-            }
-            return {__p, errc::result_out_of_range};
-        },
-        __base);
-}
-
-template <typename _Tp, typename enable_if<is_signed<_Tp>::value, int>::type = 0>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI from_chars_result
-__from_chars_integral(const char* __first, const char* __last, _Tp& __value,
-                      int __base)
-{
-    using __t = decltype(std::__to_unsigned_like(__value));
-    return std::__sign_combinator(__first, __last, __value,
-                                  __from_chars_integral<__t>, __base);
-}
-
-template <typename _Tp, typename enable_if<is_integral<_Tp>::value, int>::type = 0>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI from_chars_result
-from_chars(const char* __first, const char* __last, _Tp& __value)
-{
-    return std::__from_chars_atoi(__first, __last, __value);
-}
-
-template <typename _Tp, typename enable_if<is_integral<_Tp>::value, int>::type = 0>
-inline _LIBCPP_CONSTEXPR_SINCE_CXX23 _LIBCPP_HIDE_FROM_ABI from_chars_result
-from_chars(const char* __first, const char* __last, _Tp& __value, int __base)
-{
-    _LIBCPP_ASSERT(2 <= __base && __base <= 36, "base not in [2, 36]");
-    return std::__from_chars_integral(__first, __last, __value, __base);
-}
-
-_LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT _LIBCPP_FUNC_VIS
-to_chars_result to_chars(char* __first, char* __last, float __value);
-
-_LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT _LIBCPP_FUNC_VIS
-to_chars_result to_chars(char* __first, char* __last, double __value);
-
-_LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT _LIBCPP_FUNC_VIS
-to_chars_result to_chars(char* __first, char* __last, long double __value);
-
-_LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT _LIBCPP_FUNC_VIS
-to_chars_result to_chars(char* __first, char* __last, float __value, chars_format __fmt);
-
-_LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT _LIBCPP_FUNC_VIS
-to_chars_result to_chars(char* __first, char* __last, double __value, chars_format __fmt);
-
-_LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT _LIBCPP_FUNC_VIS
-to_chars_result to_chars(char* __first, char* __last, long double __value, chars_format __fmt);
-
-_LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT _LIBCPP_FUNC_VIS
-to_chars_result to_chars(char* __first, char* __last, float __value, chars_format __fmt, int __precision);
-
-_LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT _LIBCPP_FUNC_VIS
-to_chars_result to_chars(char* __first, char* __last, double __value, chars_format __fmt, int __precision);
-
-_LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT _LIBCPP_FUNC_VIS
-to_chars_result to_chars(char* __first, char* __last, long double __value, chars_format __fmt, int __precision);
-
-#endif // _LIBCPP_STD_VER >= 17
-
 _LIBCPP_END_NAMESPACE_STD
 
-_LIBCPP_POP_MACROS
-
 #if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
 #  include <concepts>
 #  include <cstring>
 #  include <iosfwd>
+#  include <type_traits>
 #endif
 
 #endif // _LIBCPP_CHARCONV
diff --git a/libcxx/include/complex b/libcxx/include/complex
index 5c07c1c7e91d7..26f189d8bc486 100644
--- a/libcxx/include/complex
+++ b/libcxx/include/complex
@@ -236,7 +236,6 @@ template<class T> complex<T> tanh (const complex<T>&);
 #include <cmath>
 #include <iosfwd>
 #include <stdexcept>
-#include <type_traits>
 #include <version>
 
 #if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
@@ -1565,4 +1564,8 @@ inline namespace literals
 
 _LIBCPP_END_NAMESPACE_STD
 
+#if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <type_traits>
+#endif
+
 #endif // _LIBCPP_COMPLEX
diff --git a/libcxx/include/condition_variable b/libcxx/include/condition_variable
index f13df1c012fa9..bb7b13c6fe3eb 100644
--- a/libcxx/include/condition_variable
+++ b/libcxx/include/condition_variable
@@ -202,7 +202,7 @@ condition_variable_any::wait(_Lock& __lock)
     unique_lock<mutex> __lk(*__mut);
     __lock.unlock();
     unique_ptr<_Lock, __lock_external> __lxx(&__lock);
-    lock_guard<unique_lock<mutex> > __lx(__lk, adopt_lock);
+    lock_guard<unique_lock<mutex> > __lx(__lk, adopt_lock_t());
     __cv_.wait(__lk);
 }  // __mut_.unlock(), __lock.lock()
 
@@ -224,7 +224,7 @@ condition_variable_any::wait_until(_Lock& __lock,
     unique_lock<mutex> __lk(*__mut);
     __lock.unlock();
     unique_ptr<_Lock, __lock_external> __lxx(&__lock);
-    lock_guard<unique_lock<mutex> > __lx(__lk, adopt_lock);
+    lock_guard<unique_lock<mutex> > __lx(__lk, adopt_lock_t());
     return __cv_.wait_until(__lk, __t);
 }  // __mut_.unlock(), __lock.lock()
 
@@ -269,6 +269,7 @@ _LIBCPP_END_NAMESPACE_STD
 #endif // !_LIBCPP_HAS_NO_THREADS
 
 #if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <atomic>
 #  include <concepts>
 #  include <type_traits>
 #endif
diff --git a/libcxx/include/deque b/libcxx/include/deque
index ed03f37cedabf..166a75c843fd1 100644
--- a/libcxx/include/deque
+++ b/libcxx/include/deque
@@ -181,6 +181,7 @@ template <class T, class Allocator, class Predicate>
 #include <__iterator/prev.h>
 #include <__iterator/reverse_iterator.h>
 #include <__iterator/segmented_iterator.h>
+#include <__memory/addressof.h>
 #include <__memory/allocator_destructor.h>
 #include <__memory/pointer_traits.h>
 #include <__memory/temp_value.h>
@@ -188,12 +189,14 @@ template <class T, class Allocator, class Predicate>
 #include <__memory_resource/polymorphic_allocator.h>
 #include <__split_buffer>
 #include <__type_traits/is_allocator.h>
+#include <__type_traits/is_convertible.h>
+#include <__type_traits/is_same.h>
+#include <__type_traits/type_identity.h>
 #include <__utility/forward.h>
 #include <__utility/move.h>
 #include <__utility/swap.h>
 #include <limits>
 #include <stdexcept>
-#include <type_traits>
 #include <version>
 
 // standard-mandated includes
@@ -2456,6 +2459,7 @@ _LIBCPP_POP_MACROS
 #  include <functional>
 #  include <iosfwd>
 #  include <iterator>
+#  include <type_traits>
 #  include <typeinfo>
 #endif
 
diff --git a/libcxx/include/exception b/libcxx/include/exception
index 2b08f11fbf332..97fee977690d0 100644
--- a/libcxx/include/exception
+++ b/libcxx/include/exception
@@ -77,307 +77,20 @@ template <class E> void rethrow_if_nested(const E& e);
 */
 
 #include <__assert> // all public C++ headers provide the assertion handler
-#include <__availability>
 #include <__config>
-#include <__memory/addressof.h>
-#include <__type_traits/decay.h>
-#include <__type_traits/is_base_of.h>
-#include <__type_traits/is_class.h>
-#include <__type_traits/is_convertible.h>
-#include <__type_traits/is_copy_constructible.h>
-#include <__type_traits/is_final.h>
-#include <__type_traits/is_polymorphic.h>
-#include <cstddef>
-#include <cstdlib>
+#include <__exception/exception.h>
+#include <__exception/exception_ptr.h>
+#include <__exception/nested_exception.h>
+#include <__exception/operations.h>
+#include <__exception/terminate.h>
 #include <version>
 
-// <vcruntime_exception.h> defines its own std::exception and std::bad_exception types,
-// which we use in order to be ABI-compatible with other STLs on Windows.
-#if defined(_LIBCPP_ABI_VCRUNTIME)
-#  include <vcruntime_exception.h>
-#endif
-
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
 #endif
 
-namespace std  // purposefully not using versioning namespace
-{
-
-#if defined(_LIBCPP_ABI_VCRUNTIME) && (!defined(_HAS_EXCEPTIONS) || _HAS_EXCEPTIONS != 0)
-// The std::exception class was already included above, but we're explicit about this condition here for clarity.
-
-#elif defined(_LIBCPP_ABI_VCRUNTIME) && _HAS_EXCEPTIONS == 0
-// However, <vcruntime_exception.h> does not define std::exception and std::bad_exception
-// when _HAS_EXCEPTIONS == 0.
-//
-// Since libc++ still wants to provide the std::exception hierarchy even when _HAS_EXCEPTIONS == 0
-// (after all those are simply types like any other), we define an ABI-compatible version
-// of the VCRuntime std::exception and std::bad_exception types in that mode.
-
-struct __std_exception_data {
-  char const* _What;
-  bool _DoFree;
-};
-
-class exception { // base of all library exceptions
-public:
-  exception() _NOEXCEPT : __data_() {}
-
-  explicit exception(char const* __message) _NOEXCEPT : __data_() {
-    __data_._What = __message;
-    __data_._DoFree = true;
-  }
-
-  exception(exception const&) _NOEXCEPT {}
-
-  exception& operator=(exception const&) _NOEXCEPT { return *this; }
-
-  virtual ~exception() _NOEXCEPT {}
-
-  virtual char const* what() const _NOEXCEPT { return __data_._What ? __data_._What : "Unknown exception"; }
-
-private:
-  __std_exception_data __data_;
-};
-
-class bad_exception : public exception {
-public:
-  bad_exception() _NOEXCEPT : exception("bad exception") {}
-};
-
-#else // !defined(_LIBCPP_ABI_VCRUNTIME)
-// On all other platforms, we define our own std::exception and std::bad_exception types
-// regardless of whether exceptions are turned on as a language feature.
-
-class _LIBCPP_EXCEPTION_ABI exception {
-public:
-  _LIBCPP_INLINE_VISIBILITY exception() _NOEXCEPT {}
-  _LIBCPP_INLINE_VISIBILITY exception(const exception&) _NOEXCEPT = default;
-
-  virtual ~exception() _NOEXCEPT;
-  virtual const char* what() const _NOEXCEPT;
-};
-
-class _LIBCPP_EXCEPTION_ABI bad_exception : public exception {
-public:
-  _LIBCPP_INLINE_VISIBILITY bad_exception() _NOEXCEPT {}
-  ~bad_exception() _NOEXCEPT override;
-  const char* what() const _NOEXCEPT override;
-};
-#endif // !_LIBCPP_ABI_VCRUNTIME
-
-#if _LIBCPP_STD_VER <= 14 \
-    || defined(_LIBCPP_ENABLE_CXX17_REMOVED_UNEXPECTED_FUNCTIONS) \
-    || defined(_LIBCPP_BUILDING_LIBRARY)
-typedef void (*unexpected_handler)();
-_LIBCPP_FUNC_VIS unexpected_handler set_unexpected(unexpected_handler) _NOEXCEPT;
-_LIBCPP_FUNC_VIS unexpected_handler get_unexpected() _NOEXCEPT;
-_LIBCPP_NORETURN _LIBCPP_FUNC_VIS void unexpected();
-#endif
-
-typedef void (*terminate_handler)();
-_LIBCPP_FUNC_VIS terminate_handler set_terminate(terminate_handler) _NOEXCEPT;
-_LIBCPP_FUNC_VIS terminate_handler get_terminate() _NOEXCEPT;
-_LIBCPP_NORETURN _LIBCPP_FUNC_VIS void terminate() _NOEXCEPT;
-
-_LIBCPP_FUNC_VIS bool uncaught_exception() _NOEXCEPT;
-_LIBCPP_FUNC_VIS _LIBCPP_AVAILABILITY_UNCAUGHT_EXCEPTIONS int uncaught_exceptions() _NOEXCEPT;
-
-class _LIBCPP_TYPE_VIS exception_ptr;
-
-_LIBCPP_FUNC_VIS exception_ptr current_exception() _NOEXCEPT;
-_LIBCPP_NORETURN _LIBCPP_FUNC_VIS void rethrow_exception(exception_ptr);
-
-#ifndef _LIBCPP_ABI_MICROSOFT
-
-class _LIBCPP_TYPE_VIS exception_ptr
-{
-    void* __ptr_;
-public:
-    _LIBCPP_INLINE_VISIBILITY exception_ptr() _NOEXCEPT : __ptr_() {}
-    _LIBCPP_INLINE_VISIBILITY exception_ptr(nullptr_t) _NOEXCEPT : __ptr_() {}
-
-    exception_ptr(const exception_ptr&) _NOEXCEPT;
-    exception_ptr& operator=(const exception_ptr&) _NOEXCEPT;
-    ~exception_ptr() _NOEXCEPT;
-
-    _LIBCPP_INLINE_VISIBILITY explicit operator bool() const _NOEXCEPT
-    {return __ptr_ != nullptr;}
-
-    friend _LIBCPP_INLINE_VISIBILITY
-    bool operator==(const exception_ptr& __x, const exception_ptr& __y) _NOEXCEPT
-        {return __x.__ptr_ == __y.__ptr_;}
-
-    friend _LIBCPP_INLINE_VISIBILITY
-    bool operator!=(const exception_ptr& __x, const exception_ptr& __y) _NOEXCEPT
-        {return !(__x == __y);}
-
-    friend _LIBCPP_FUNC_VIS exception_ptr current_exception() _NOEXCEPT;
-    friend _LIBCPP_FUNC_VIS void rethrow_exception(exception_ptr);
-};
-
-template<class _Ep>
-_LIBCPP_INLINE_VISIBILITY exception_ptr
-make_exception_ptr(_Ep __e) _NOEXCEPT
-{
-#ifndef _LIBCPP_HAS_NO_EXCEPTIONS
-    try
-    {
-        throw __e;
-    }
-    catch (...)
-    {
-        return current_exception();
-    }
-#else
-    ((void)__e);
-    _VSTD::abort();
-#endif
-}
-
-#else // _LIBCPP_ABI_MICROSOFT
-
-class _LIBCPP_TYPE_VIS exception_ptr
-{
-_LIBCPP_DIAGNOSTIC_PUSH
-_LIBCPP_CLANG_DIAGNOSTIC_IGNORED("-Wunused-private-field")
-    void* __ptr1_;
-    void* __ptr2_;
-_LIBCPP_DIAGNOSTIC_POP
-public:
-    exception_ptr() _NOEXCEPT;
-    exception_ptr(nullptr_t) _NOEXCEPT;
-    exception_ptr(const exception_ptr& __other) _NOEXCEPT;
-    exception_ptr& operator=(const exception_ptr& __other) _NOEXCEPT;
-    exception_ptr& operator=(nullptr_t) _NOEXCEPT;
-    ~exception_ptr() _NOEXCEPT;
-    explicit operator bool() const _NOEXCEPT;
-};
-
-_LIBCPP_FUNC_VIS
-bool operator==(const exception_ptr& __x, const exception_ptr& __y) _NOEXCEPT;
-
-inline _LIBCPP_INLINE_VISIBILITY
-bool operator!=(const exception_ptr& __x, const exception_ptr& __y) _NOEXCEPT
-    {return !(__x == __y);}
-
-_LIBCPP_FUNC_VIS void swap(exception_ptr&, exception_ptr&) _NOEXCEPT;
-
-_LIBCPP_FUNC_VIS exception_ptr __copy_exception_ptr(void *__except, const void* __ptr);
-_LIBCPP_FUNC_VIS exception_ptr current_exception() _NOEXCEPT;
-_LIBCPP_NORETURN _LIBCPP_FUNC_VIS void rethrow_exception(exception_ptr);
-
-// This is a built-in template function which automagically extracts the required
-// information.
-template <class _E> void *__GetExceptionInfo(_E);
-
-template<class _Ep>
-_LIBCPP_INLINE_VISIBILITY exception_ptr
-make_exception_ptr(_Ep __e) _NOEXCEPT
-{
-  return __copy_exception_ptr(_VSTD::addressof(__e), __GetExceptionInfo(__e));
-}
-
-#endif // _LIBCPP_ABI_MICROSOFT
-// nested_exception
-
-class _LIBCPP_EXCEPTION_ABI nested_exception
-{
-    exception_ptr __ptr_;
-public:
-    nested_exception() _NOEXCEPT;
-//     nested_exception(const nested_exception&) noexcept = default;
-//     nested_exception& operator=(const nested_exception&) noexcept = default;
-    virtual ~nested_exception() _NOEXCEPT;
-
-    // access functions
-    _LIBCPP_NORETURN void rethrow_nested() const;
-    _LIBCPP_INLINE_VISIBILITY exception_ptr nested_ptr() const _NOEXCEPT {return __ptr_;}
-};
-
-template <class _Tp>
-struct __nested
-    : public _Tp,
-      public nested_exception
-{
-    _LIBCPP_INLINE_VISIBILITY explicit __nested(const _Tp& __t) : _Tp(__t) {}
-};
-
-#ifndef _LIBCPP_HAS_NO_EXCEPTIONS
-template <class _Tp, class _Up, bool>
-struct __throw_with_nested;
-
-template <class _Tp, class _Up>
-struct __throw_with_nested<_Tp, _Up, true> {
-    _LIBCPP_NORETURN static inline _LIBCPP_INLINE_VISIBILITY void
-    __do_throw(_Tp&& __t)
-    {
-        throw __nested<_Up>(static_cast<_Tp&&>(__t));
-    }
-};
-
-template <class _Tp, class _Up>
-struct __throw_with_nested<_Tp, _Up, false> {
-    _LIBCPP_NORETURN static inline _LIBCPP_INLINE_VISIBILITY void
-#ifndef _LIBCPP_CXX03_LANG
-    __do_throw(_Tp&& __t)
-#else
-    __do_throw (_Tp& __t)
-#endif // _LIBCPP_CXX03_LANG
-    {
-        throw static_cast<_Tp&&>(__t);
-    }
-};
-#endif
-
-template <class _Tp>
-_LIBCPP_NORETURN _LIBCPP_HIDE_FROM_ABI
-void
-throw_with_nested(_Tp&& __t)
-{
-#ifndef _LIBCPP_HAS_NO_EXCEPTIONS
-    typedef typename decay<_Tp>::type _Up;
-    static_assert( is_copy_constructible<_Up>::value, "type thrown must be CopyConstructible");
-    __throw_with_nested<_Tp, _Up,
-        is_class<_Up>::value &&
-        !is_base_of<nested_exception, _Up>::value &&
-        !__libcpp_is_final<_Up>::value>::
-            __do_throw(static_cast<_Tp&&>(__t));
-#else
-    ((void)__t);
-    // FIXME: Make this abort
-#endif
-}
-
-template <class _From, class _To>
-struct __can_dynamic_cast : _BoolConstant<
-              is_polymorphic<_From>::value &&
-                 (!is_base_of<_To, _From>::value ||
-                   is_convertible<const _From*, const _To*>::value)> {};
-
-template <class _Ep>
-inline _LIBCPP_INLINE_VISIBILITY
-void
-rethrow_if_nested(const _Ep& __e,
-                  __enable_if_t< __can_dynamic_cast<_Ep, nested_exception>::value>* = 0)
-{
-    const nested_exception* __nep = dynamic_cast<const nested_exception*>(_VSTD::addressof(__e));
-    if (__nep)
-        __nep->rethrow_nested();
-}
-
-template <class _Ep>
-inline _LIBCPP_INLINE_VISIBILITY
-void
-rethrow_if_nested(const _Ep&,
-                  __enable_if_t<!__can_dynamic_cast<_Ep, nested_exception>::value>* = 0)
-{
-}
-
-} // namespace std
-
 #if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <cstdlib>
 #  include <type_traits>
 #endif
 
diff --git a/libcxx/include/experimental/__memory b/libcxx/include/experimental/__memory
index b36f31eebb7c6..b11369632ae15 100644
--- a/libcxx/include/experimental/__memory
+++ b/libcxx/include/experimental/__memory
@@ -12,9 +12,12 @@
 
 #include <__memory/allocator_arg_t.h>
 #include <__memory/uses_allocator.h>
+#include <__type_traits/conditional.h>
+#include <__type_traits/is_constructible.h>
+#include <__type_traits/is_convertible.h>
+#include <__type_traits/is_same.h>
 #include <experimental/__config>
 #include <experimental/utility> // for erased_type
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
@@ -90,7 +93,7 @@ template <class _Tp, class _Allocator, class... _Args>
 inline _LIBCPP_INLINE_VISIBILITY
 void __user_alloc_construct_impl (integral_constant<int, 1>, _Tp *__storage, const _Allocator &__a, _Args &&... __args )
 {
-    new (__storage) _Tp (allocator_arg, __a, _VSTD::forward<_Args>(__args)...);
+    new (__storage) _Tp (allocator_arg_t(), __a, _VSTD::forward<_Args>(__args)...);
 }
 
 // FIXME: This should have a version which takes a non-const alloc.
diff --git a/libcxx/include/experimental/memory_resource b/libcxx/include/experimental/memory_resource
index f107f109203f0..f9edd3077f558 100644
--- a/libcxx/include/experimental/memory_resource
+++ b/libcxx/include/experimental/memory_resource
@@ -66,6 +66,7 @@ namespace pmr {
 
 #include <__assert> // all public C++ headers provide the assertion handler
 #include <__memory/allocator_traits.h>
+#include <__type_traits/aligned_storage.h>
 #include <__utility/move.h>
 #include <cstddef>
 #include <cstdlib>
@@ -75,7 +76,6 @@ namespace pmr {
 #include <new>
 #include <stdexcept>
 #include <tuple>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
@@ -437,6 +437,7 @@ _LIBCPP_POP_MACROS
 #  include <iterator>
 #  include <memory>
 #  include <ratio>
+#  include <type_traits>
 #  include <variant>
 #endif
 
diff --git a/libcxx/include/experimental/propagate_const b/libcxx/include/experimental/propagate_const
index feb85d4a9b381..3efec5db49213 100644
--- a/libcxx/include/experimental/propagate_const
+++ b/libcxx/include/experimental/propagate_const
@@ -110,11 +110,26 @@
 #include <__assert> // all public C++ headers provide the assertion handler
 #include <__functional/operations.h>
 #include <__fwd/hash.h>
+#include <__type_traits/conditional.h>
+#include <__type_traits/decay.h>
+#include <__type_traits/enable_if.h>
+#include <__type_traits/is_array.h>
+#include <__type_traits/is_constructible.h>
+#include <__type_traits/is_convertible.h>
+#include <__type_traits/is_function.h>
+#include <__type_traits/is_pointer.h>
+#include <__type_traits/is_reference.h>
+#include <__type_traits/is_same.h>
+#include <__type_traits/is_swappable.h>
+#include <__type_traits/remove_cv.h>
+#include <__type_traits/remove_pointer.h>
+#include <__type_traits/remove_reference.h>
+#include <__utility/declval.h>
 #include <__utility/forward.h>
 #include <__utility/move.h>
 #include <__utility/swap.h>
+#include <cstddef>
 #include <experimental/__config>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
@@ -578,4 +593,9 @@ struct greater_equal<experimental::fundamentals_v2::propagate_const<_Tp>>
 _LIBCPP_END_NAMESPACE_STD
 
 #endif // _LIBCPP_STD_VER >= 14
+
+#if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <type_traits>
+#endif
+
 #endif // _LIBCPP_EXPERIMENTAL_PROPAGATE_CONST
diff --git a/libcxx/include/ext/hash_map b/libcxx/include/ext/hash_map
index 3ad4b166865af..f7f53bc3801ec 100644
--- a/libcxx/include/ext/hash_map
+++ b/libcxx/include/ext/hash_map
@@ -208,7 +208,6 @@ template <class Key, class T, class Hash, class Pred, class Alloc>
 #include <ext/__hash>
 #include <functional>
 #include <stdexcept>
-#include <type_traits>
 
 #if defined(__DEPRECATED) && __DEPRECATED
 #if defined(_LIBCPP_WARNING)
@@ -985,6 +984,7 @@ operator!=(const hash_multimap<_Key, _Tp, _Hash, _Pred, _Alloc>& __x,
 #if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
 #  include <concepts>
 #  include <iterator>
+#  include <type_traits>
 #endif
 
 #endif // _LIBCPP_HASH_MAP
diff --git a/libcxx/include/ext/hash_set b/libcxx/include/ext/hash_set
index dc8b786453842..9c678623ddd66 100644
--- a/libcxx/include/ext/hash_set
+++ b/libcxx/include/ext/hash_set
@@ -665,6 +665,7 @@ operator!=(const hash_multiset<_Value, _Hash, _Pred, _Alloc>& __x,
 #if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
 #  include <concepts>
 #  include <iterator>
+#  include <type_traits>
 #endif
 
 #endif // _LIBCPP_HASH_SET
diff --git a/libcxx/include/forward_list b/libcxx/include/forward_list
index f5a6120089fd3..e487d17919fad 100644
--- a/libcxx/include/forward_list
+++ b/libcxx/include/forward_list
@@ -202,11 +202,18 @@ template <class T, class Allocator, class Predicate>
 #include <__memory/swap_allocator.h>
 #include <__memory/unique_ptr.h>
 #include <__memory_resource/polymorphic_allocator.h>
+#include <__type_traits/conditional.h>
 #include <__type_traits/is_allocator.h>
+#include <__type_traits/is_const.h>
+#include <__type_traits/is_nothrow_default_constructible.h>
+#include <__type_traits/is_nothrow_move_assignable.h>
+#include <__type_traits/is_nothrow_move_constructible.h>
+#include <__type_traits/is_pointer.h>
+#include <__type_traits/is_same.h>
+#include <__type_traits/type_identity.h>
 #include <__utility/forward.h>
 #include <__utility/move.h>
 #include <limits>
-#include <type_traits>
 #include <version>
 
 // standard-mandated includes
@@ -1818,6 +1825,7 @@ _LIBCPP_POP_MACROS
 #  include <functional>
 #  include <iosfwd>
 #  include <iterator>
+#  include <type_traits>
 #  include <typeinfo>
 #endif
 
diff --git a/libcxx/include/functional b/libcxx/include/functional
index bb5db290f5a43..489b70d6462cf 100644
--- a/libcxx/include/functional
+++ b/libcxx/include/functional
@@ -543,9 +543,7 @@ POLICY:  For non-variadic implementations, the number of arguments is limited
 #include <__functional/unary_negate.h>
 #include <__functional/unwrap_ref.h>
 #include <__utility/forward.h>
-#include <exception>
 #include <memory> // TODO: find out why removing this breaks the modules build
-#include <type_traits>
 #include <typeinfo>
 #include <version>
 
@@ -554,8 +552,11 @@ POLICY:  For non-variadic implementations, the number of arguments is limited
 #endif
 
 #if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <atomic>
 #  include <concepts>
+#  include <exception>
 #  include <tuple>
+#  include <type_traits>
 #  include <utility>
 #endif
 
diff --git a/libcxx/include/future b/libcxx/include/future
index a82efeb84cef8..6f1eeff9cf1af 100644
--- a/libcxx/include/future
+++ b/libcxx/include/future
@@ -366,14 +366,18 @@ template <class R, class Alloc> struct uses_allocator<packaged_task<R>, Alloc>;
 #include <__chrono/duration.h>
 #include <__chrono/time_point.h>
 #include <__config>
+#include <__exception/exception_ptr.h>
 #include <__memory/allocator_arg_t.h>
 #include <__memory/allocator_destructor.h>
+#include <__memory/shared_ptr.h>
+#include <__memory/unique_ptr.h>
 #include <__memory/uses_allocator.h>
+#include <__type_traits/aligned_storage.h>
+#include <__type_traits/alignment_of.h>
 #include <__type_traits/strip_signature.h>
 #include <__utility/auto_cast.h>
 #include <__utility/forward.h>
 #include <__utility/move.h>
-#include <exception>
 #include <mutex>
 #include <new>
 #include <system_error>
@@ -528,8 +532,8 @@ void __throw_future_error(future_errc __ev)
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
     throw future_error(make_error_code(__ev));
 #else
-    ((void)__ev);
-    _VSTD::abort();
+    (void)__ev;
+    _LIBCPP_VERBOSE_ABORT("future_error was thrown in -fno-exceptions mode");
 #endif
 }
 
@@ -1902,8 +1906,8 @@ public:
               class = __enable_if_t<!is_same<__remove_cvref_t<_Fp>, packaged_task>::value> >
         _LIBCPP_INLINE_VISIBILITY
         packaged_task(allocator_arg_t, const _Allocator& __a, _Fp&& __f)
-             : __f_(allocator_arg, __a, _VSTD::forward<_Fp>(__f)),
-               __p_(allocator_arg, __a) {}
+             : __f_(allocator_arg_t(), __a, _VSTD::forward<_Fp>(__f)),
+               __p_(allocator_arg_t(), __a) {}
     // ~packaged_task() = default;
 
     // no copy
@@ -2017,8 +2021,8 @@ public:
               class = __enable_if_t<!is_same<__remove_cvref_t<_Fp>, packaged_task>::value> >
         _LIBCPP_INLINE_VISIBILITY
         packaged_task(allocator_arg_t, const _Allocator& __a, _Fp&& __f)
-             : __f_(allocator_arg, __a, _VSTD::forward<_Fp>(__f)),
-               __p_(allocator_arg, __a) {}
+             : __f_(allocator_arg_t(), __a, _VSTD::forward<_Fp>(__f)),
+               __p_(allocator_arg_t(), __a) {}
     // ~packaged_task() = default;
 
     // no copy
@@ -2454,4 +2458,9 @@ _LIBCPP_END_NAMESPACE_STD
 #  include <chrono>
 #endif
 
+#if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <atomic>
+#  include <exception>
+#endif
+
 #endif // _LIBCPP_FUTURE
diff --git a/libcxx/include/iomanip b/libcxx/include/iomanip
index bf8fbfe2a16f8..53445c72ba19a 100644
--- a/libcxx/include/iomanip
+++ b/libcxx/include/iomanip
@@ -513,8 +513,6 @@ put_time(const tm* __tm, const _CharT* __fmt)
     return __iom_t10<_CharT>(__tm, __fmt);
 }
 
-#if _LIBCPP_STD_VER >= 11
-
 template <class _CharT, class _Traits>
 _LIBCPP_HIDE_FROM_ABI basic_ostream<_CharT, _Traits>&
 __quoted_output(basic_ostream<_CharT, _Traits>& __os,
@@ -622,8 +620,6 @@ __quoted(basic_string<_CharT, _Traits, _Allocator>& __s, _CharT __delim = _CharT
     return __quoted_proxy<_CharT, _Traits, _Allocator>(__s, __delim, __escape);
 }
 
-#endif // _LIBCPP_STD_VER >= 11
-
 #if _LIBCPP_STD_VER >= 14
 
 template <class _CharT>
diff --git a/libcxx/include/ios b/libcxx/include/ios
index bc478a11bd22b..afe42a6866dda 100644
--- a/libcxx/include/ios
+++ b/libcxx/include/ios
@@ -220,6 +220,7 @@ storage-class-specifier const error_category& iostream_category() noexcept;
 #include <__ios/fpos.h>
 #include <__locale>
 #include <__utility/swap.h>
+#include <__verbose_abort>
 #include <system_error>
 #include <version>
 
@@ -229,7 +230,7 @@ storage-class-specifier const error_category& iostream_category() noexcept;
 #include <iosfwd>
 
 #if !defined(_LIBCPP_HAS_NO_ATOMIC_HEADER)
-#include <atomic>     // for __xindex_
+#  include <__atomic/atomic.h> // for __xindex_
 #endif
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
@@ -451,8 +452,7 @@ void __throw_failure(char const* __msg) {
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
     throw ios_base::failure(__msg);
 #else
-    ((void)__msg);
-    _VSTD::abort();
+    _LIBCPP_VERBOSE_ABORT("ios_base::failure was thrown in -fno-exceptions mode with message \"%s\"", __msg);
 #endif
 }
 
@@ -1039,6 +1039,7 @@ defaultfloat(ios_base& __str)
 _LIBCPP_END_NAMESPACE_STD
 
 #if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <atomic>
 #  include <concepts>
 #  include <cstddef>
 #  include <cstdlib>
diff --git a/libcxx/include/istream b/libcxx/include/istream
index 831d4ecaabe5f..7350e11fe129d 100644
--- a/libcxx/include/istream
+++ b/libcxx/include/istream
@@ -161,6 +161,10 @@ template <class Stream, class T>
 #include <__assert> // all public C++ headers provide the assertion handler
 #include <__config>
 #include <__iterator/istreambuf_iterator.h>
+#include <__type_traits/conjunction.h>
+#include <__type_traits/enable_if.h>
+#include <__type_traits/is_base_of.h>
+#include <__utility/declval.h>
 #include <__utility/forward.h>
 #include <ostream>
 #include <version>
diff --git a/libcxx/include/latch b/libcxx/include/latch
index ff17f8453ea50..d72cb4f7b0ebd 100644
--- a/libcxx/include/latch
+++ b/libcxx/include/latch
@@ -41,9 +41,12 @@ namespace std
 */
 
 #include <__assert> // all public C++ headers provide the assertion handler
+#include <__atomic/atomic_base.h>
+#include <__atomic/atomic_sync.h>
+#include <__atomic/memory_order.h>
 #include <__availability>
 #include <__config>
-#include <atomic>
+#include <cstddef>
 #include <limits>
 #include <version>
 
@@ -111,4 +114,8 @@ _LIBCPP_END_NAMESPACE_STD
 
 _LIBCPP_POP_MACROS
 
+#if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <atomic>
+#endif
+
 #endif //_LIBCPP_LATCH
diff --git a/libcxx/include/libcxx.imp b/libcxx/include/libcxx.imp
index 8e897c9bb66e1..d85ae36953a1e 100644
--- a/libcxx/include/libcxx.imp
+++ b/libcxx/include/libcxx.imp
@@ -24,6 +24,7 @@
   { include: [ "@<__concepts/.*>", "private", "<concepts>", "public" ] },
   { include: [ "@<__coroutine/.*>", "private", "<coroutine>", "public" ] },
   { include: [ "@<__debug_utils/.*>", "private", "<debug_utils>", "public" ] },
+  { include: [ "@<__exception/.*>", "private", "<exception>", "public" ] },
   { include: [ "@<__expected/.*>", "private", "<expected>", "public" ] },
   { include: [ "@<__filesystem/.*>", "private", "<filesystem>", "public" ] },
   { include: [ "@<__format/.*>", "private", "<format>", "public" ] },
diff --git a/libcxx/include/list b/libcxx/include/list
index 750ccf99d41fb..191f317341660 100644
--- a/libcxx/include/list
+++ b/libcxx/include/list
@@ -207,13 +207,19 @@ template <class T, class Allocator, class Predicate>
 #include <__memory/swap_allocator.h>
 #include <__memory/unique_ptr.h>
 #include <__memory_resource/polymorphic_allocator.h>
+#include <__type_traits/conditional.h>
 #include <__type_traits/is_allocator.h>
+#include <__type_traits/is_nothrow_default_constructible.h>
+#include <__type_traits/is_nothrow_move_assignable.h>
+#include <__type_traits/is_nothrow_move_constructible.h>
+#include <__type_traits/is_pointer.h>
+#include <__type_traits/is_same.h>
+#include <__type_traits/type_identity.h>
 #include <__utility/forward.h>
 #include <__utility/move.h>
 #include <__utility/swap.h>
 #include <cstring>
 #include <limits>
-#include <type_traits>
 #include <version>
 
 // standard-mandated includes
@@ -2399,6 +2405,7 @@ _LIBCPP_POP_MACROS
 #  include <functional>
 #  include <iosfwd>
 #  include <iterator>
+#  include <type_traits>
 #  include <typeinfo>
 #endif
 
diff --git a/libcxx/include/locale b/libcxx/include/locale
index 0b64c68ebfac3..0f862b12ed286 100644
--- a/libcxx/include/locale
+++ b/libcxx/include/locale
@@ -202,6 +202,7 @@ template <class charT> class messages_byname;
 #include <__iterator/ostreambuf_iterator.h>
 #include <__locale>
 #include <__memory/unique_ptr.h>
+#include <__type_traits/make_unsigned.h>
 #include <cstdio>
 #include <cstdlib>
 #include <ctime>
diff --git a/libcxx/include/map b/libcxx/include/map
index b1ba562cab2e9..98ee961559e27 100644
--- a/libcxx/include/map
+++ b/libcxx/include/map
@@ -250,27 +250,32 @@ operator==(const map<Key, T, Compare, Allocator>& x,
 template <class Key, class T, class Compare, class Allocator>
 bool
 operator< (const map<Key, T, Compare, Allocator>& x,
-           const map<Key, T, Compare, Allocator>& y);
+           const map<Key, T, Compare, Allocator>& y);      // removed in C++20
 
 template <class Key, class T, class Compare, class Allocator>
 bool
 operator!=(const map<Key, T, Compare, Allocator>& x,
-           const map<Key, T, Compare, Allocator>& y);
+           const map<Key, T, Compare, Allocator>& y);      // removed in C++20
 
 template <class Key, class T, class Compare, class Allocator>
 bool
 operator> (const map<Key, T, Compare, Allocator>& x,
-           const map<Key, T, Compare, Allocator>& y);
+           const map<Key, T, Compare, Allocator>& y);      // removed in C++20
 
 template <class Key, class T, class Compare, class Allocator>
 bool
 operator>=(const map<Key, T, Compare, Allocator>& x,
-           const map<Key, T, Compare, Allocator>& y);
+           const map<Key, T, Compare, Allocator>& y);      // removed in C++20
 
 template <class Key, class T, class Compare, class Allocator>
 bool
 operator<=(const map<Key, T, Compare, Allocator>& x,
-           const map<Key, T, Compare, Allocator>& y);
+           const map<Key, T, Compare, Allocator>& y);      // removed in C++20
+
+template<class Key, class T, class Compare, class Allocator>
+  synth-three-way-result<pair<const Key, T>>
+    operator<=>(const map<Key, T, Compare, Allocator>& x,
+                const map<Key, T, Compare, Allocator>& y); // since C++20
 
 // specialized algorithms:
 template <class Key, class T, class Compare, class Allocator>
@@ -491,27 +496,32 @@ operator==(const multimap<Key, T, Compare, Allocator>& x,
 template <class Key, class T, class Compare, class Allocator>
 bool
 operator< (const multimap<Key, T, Compare, Allocator>& x,
-           const multimap<Key, T, Compare, Allocator>& y);
+           const multimap<Key, T, Compare, Allocator>& y);      // removed in C++20
 
 template <class Key, class T, class Compare, class Allocator>
 bool
 operator!=(const multimap<Key, T, Compare, Allocator>& x,
-           const multimap<Key, T, Compare, Allocator>& y);
+           const multimap<Key, T, Compare, Allocator>& y);      // removed in C++20
 
 template <class Key, class T, class Compare, class Allocator>
 bool
 operator> (const multimap<Key, T, Compare, Allocator>& x,
-           const multimap<Key, T, Compare, Allocator>& y);
+           const multimap<Key, T, Compare, Allocator>& y);      // removed in C++20
 
 template <class Key, class T, class Compare, class Allocator>
 bool
 operator>=(const multimap<Key, T, Compare, Allocator>& x,
-           const multimap<Key, T, Compare, Allocator>& y);
+           const multimap<Key, T, Compare, Allocator>& y);      // removed in C++20
 
 template <class Key, class T, class Compare, class Allocator>
 bool
 operator<=(const multimap<Key, T, Compare, Allocator>& x,
-           const multimap<Key, T, Compare, Allocator>& y);
+           const multimap<Key, T, Compare, Allocator>& y);      // removed in C++20
+
+template<class Key, class T, class Compare, class Allocator>
+  synth-three-way-result<pair<const Key, T>>
+    operator<=>(const multimap<Key, T, Compare, Allocator>& x,
+                const multimap<Key, T, Compare, Allocator>& y); // since c++20
 
 // specialized algorithms:
 template <class Key, class T, class Compare, class Allocator>
@@ -530,6 +540,7 @@ erase_if(multimap<Key, T, Compare, Allocator>& c, Predicate pred);  // C++20
 
 #include <__algorithm/equal.h>
 #include <__algorithm/lexicographical_compare.h>
+#include <__algorithm/lexicographical_compare_three_way.h>
 #include <__assert> // all public C++ headers provide the assertion handler
 #include <__config>
 #include <__functional/binary_function.h>
@@ -538,6 +549,7 @@ erase_if(multimap<Key, T, Compare, Allocator>& c, Predicate pred);  // C++20
 #include <__iterator/erase_if_container.h>
 #include <__iterator/iterator_traits.h>
 #include <__iterator/reverse_iterator.h>
+#include <__memory/addressof.h>
 #include <__memory/allocator.h>
 #include <__memory_resource/polymorphic_allocator.h>
 #include <__node_handle>
@@ -547,7 +559,6 @@ erase_if(multimap<Key, T, Compare, Allocator>& c, Predicate pred);  // C++20
 #include <__utility/piecewise_construct.h>
 #include <__utility/swap.h>
 #include <tuple>
-#include <type_traits>
 #include <version>
 
 // standard-mandated includes
@@ -1677,6 +1688,8 @@ operator==(const map<_Key, _Tp, _Compare, _Allocator>& __x,
     return __x.size() == __y.size() && _VSTD::equal(__x.begin(), __x.end(), __y.begin());
 }
 
+#if _LIBCPP_STD_VER <= 17
+
 template <class _Key, class _Tp, class _Compare, class _Allocator>
 inline _LIBCPP_INLINE_VISIBILITY
 bool
@@ -1722,6 +1735,20 @@ operator<=(const map<_Key, _Tp, _Compare, _Allocator>& __x,
     return !(__y < __x);
 }
 
+#else // #if _LIBCPP_STD_VER <= 17
+
+template<class _Key, class _Tp, class _Compare, class _Allocator>
+inline _LIBCPP_HIDE_FROM_ABI
+__synth_three_way_result<pair<const _Key, _Tp>>
+operator<=>(const map<_Key, _Tp, _Compare, _Allocator>& __x,
+            const map<_Key, _Tp, _Compare, _Allocator>& __y)
+{
+    return std::lexicographical_compare_three_way(
+        __x.begin(), __x.end(), __y.begin(), __y.end(), __synth_three_way);
+}
+
+#endif // #if _LIBCPP_STD_VER <= 17
+
 template <class _Key, class _Tp, class _Compare, class _Allocator>
 inline _LIBCPP_INLINE_VISIBILITY
 void
@@ -2271,6 +2298,8 @@ operator==(const multimap<_Key, _Tp, _Compare, _Allocator>& __x,
     return __x.size() == __y.size() && _VSTD::equal(__x.begin(), __x.end(), __y.begin());
 }
 
+#if _LIBCPP_STD_VER <= 17
+
 template <class _Key, class _Tp, class _Compare, class _Allocator>
 inline _LIBCPP_INLINE_VISIBILITY
 bool
@@ -2316,6 +2345,20 @@ operator<=(const multimap<_Key, _Tp, _Compare, _Allocator>& __x,
     return !(__y < __x);
 }
 
+#else // #if _LIBCPP_STD_VER <= 17
+
+template<class _Key, class _Tp, class _Compare, class _Allocator>
+inline _LIBCPP_HIDE_FROM_ABI
+__synth_three_way_result<pair<const _Key, _Tp>>
+operator<=>(const multimap<_Key, _Tp, _Compare, _Allocator>& __x,
+            const multimap<_Key, _Tp, _Compare, _Allocator>& __y)
+{
+    return std::lexicographical_compare_three_way(
+        __x.begin(), __x.end(), __y.begin(), __y.end(), __synth_three_way);
+}
+
+#endif // #if _LIBCPP_STD_VER <= 17
+
 template <class _Key, class _Tp, class _Compare, class _Allocator>
 inline _LIBCPP_INLINE_VISIBILITY
 void
@@ -2355,6 +2398,7 @@ _LIBCPP_END_NAMESPACE_STD
 #  include <concepts>
 #  include <functional>
 #  include <iterator>
+#  include <type_traits>
 #  include <utility>
 #endif
 
diff --git a/libcxx/include/memory b/libcxx/include/memory
index 0a7787acf1a8e..f6063178edc08 100644
--- a/libcxx/include/memory
+++ b/libcxx/include/memory
@@ -917,6 +917,7 @@ template<size_t N, class T>
 #endif
 
 #if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <atomic>
 #  include <concepts>
 #  include <cstddef>
 #  include <cstdint>
diff --git a/libcxx/include/module.modulemap.in b/libcxx/include/module.modulemap.in
index da71f110d5b41..89a03cbb341d3 100644
--- a/libcxx/include/module.modulemap.in
+++ b/libcxx/include/module.modulemap.in
@@ -631,7 +631,10 @@ module std [system] {
     export *
 
     module __atomic {
-      module aliases             { private header "__atomic/aliases.h" }
+      module aliases             {
+        private header "__atomic/aliases.h"
+        export atomic
+      }
       module atomic              { private header "__atomic/atomic.h" }
       module atomic_base         { private header "__atomic/atomic_base.h" }
       module atomic_flag         { private header "__atomic/atomic_flag.h" }
@@ -684,11 +687,16 @@ module std [system] {
     export *
 
     module __charconv {
-      module chars_format      { private header "__charconv/chars_format.h" }
-      module from_chars_result { private header "__charconv/from_chars_result.h" }
-      module tables            { private header "__charconv/tables.h" }
-      module to_chars_base_10  { private header "__charconv/to_chars_base_10.h" }
-      module to_chars_result   { private header "__charconv/to_chars_result.h" }
+      module chars_format            { private header "__charconv/chars_format.h" }
+      module from_chars_integral     { private header "__charconv/from_chars_integral.h" }
+      module from_chars_result       { private header "__charconv/from_chars_result.h" }
+      module tables                  { private header "__charconv/tables.h" }
+      module to_chars                { private header "__charconv/to_chars.h" }
+      module to_chars_base_10        { private header "__charconv/to_chars_base_10.h" }
+      module to_chars_floating_point { private header "__charconv/to_chars_floating_point.h" }
+      module to_chars_integral       { private header "__charconv/to_chars_integral.h" }
+      module to_chars_result         { private header "__charconv/to_chars_result.h" }
+      module traits                  { private header "__charconv/traits.h" }
     }
 
   }
@@ -826,6 +834,14 @@ module std [system] {
   module exception {
     header "exception"
     export *
+
+    module __exception {
+      module exception        { private header "__exception/exception.h" }
+      module exception_ptr    { private header "__exception/exception_ptr.h" }
+      module nested_exception { private header "__exception/nested_exception.h" }
+      module operations       { private header "__exception/operations.h" }
+      module terminate        { private header "__exception/terminate.h" }
+    }
   }
   module execution {
     header "execution"
@@ -1620,7 +1636,10 @@ module std [system] {
 
     module __utility {
       module as_const            { private header "__utility/as_const.h" }
-      module auto_cast           { private header "__utility/auto_cast.h" }
+      module auto_cast           {
+        private header "__utility/auto_cast.h"
+        export type_traits.decay
+      }
       module cmp                 { private header "__utility/cmp.h" }
       module convert_to_integral { private header "__utility/convert_to_integral.h" }
       module declval             { private header "__utility/declval.h" }
diff --git a/libcxx/include/mutex b/libcxx/include/mutex
index 5f4f7d3791a88..9d24768d6c0e2 100644
--- a/libcxx/include/mutex
+++ b/libcxx/include/mutex
@@ -328,7 +328,7 @@ template <class _L0, class _L1>
 _LIBCPP_HIDE_FROM_ABI int
 try_lock(_L0& __l0, _L1& __l1)
 {
-    unique_lock<_L0> __u0(__l0, try_to_lock);
+    unique_lock<_L0> __u0(__l0, try_to_lock_t());
     if (__u0.owns_lock())
     {
         if (__l1.try_lock())
@@ -704,7 +704,9 @@ _LIBCPP_END_NAMESPACE_STD
 _LIBCPP_POP_MACROS
 
 #if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <atomic>
 #  include <concepts>
+#  include <ctime>
 #  include <functional>
 #  include <type_traits>
 #endif
diff --git a/libcxx/include/new b/libcxx/include/new
index be8033e33db91..af396cd4b8eb8 100644
--- a/libcxx/include/new
+++ b/libcxx/include/new
@@ -89,12 +89,14 @@ void  operator delete[](void* ptr, void*) noexcept;
 #include <__assert> // all public C++ headers provide the assertion handler
 #include <__availability>
 #include <__config>
+#include <__exception/exception.h>
+#include <__type_traits/alignment_of.h>
 #include <__type_traits/is_function.h>
 #include <__type_traits/is_same.h>
 #include <__type_traits/remove_cv.h>
+#include <__verbose_abort>
 #include <cstddef>
 #include <cstdlib>
-#include <exception>
 #include <version>
 
 #if defined(_LIBCPP_ABI_VCRUNTIME)
@@ -176,7 +178,7 @@ void __throw_bad_array_new_length()
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
     throw bad_array_new_length();
 #else
-    _VSTD::abort();
+    _LIBCPP_VERBOSE_ABORT("bad_array_new_length was thrown in -fno-exceptions mode");
 #endif
 }
 
@@ -364,6 +366,7 @@ inline constexpr size_t hardware_constructive_interference_size = __GCC_CONSTRUC
 _LIBCPP_END_NAMESPACE_STD
 
 #if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <exception>
 #  include <type_traits>
 #endif
 
diff --git a/libcxx/include/optional b/libcxx/include/optional
index bbc8229c03316..9f896bd16945a 100644
--- a/libcxx/include/optional
+++ b/libcxx/include/optional
@@ -165,16 +165,26 @@ template<class T>
 #include <__functional/hash.h>
 #include <__functional/invoke.h>
 #include <__functional/unary_function.h>
+#include <__memory/addressof.h>
 #include <__memory/construct_at.h>
 #include <__tuple_dir/sfinae_helpers.h>
+#include <__type_traits/conjunction.h>
+#include <__type_traits/disjunction.h>
+#include <__type_traits/is_trivially_copy_assignable.h>
+#include <__type_traits/is_trivially_copy_constructible.h>
+#include <__type_traits/is_trivially_destructible.h>
+#include <__type_traits/is_trivially_move_assignable.h>
+#include <__type_traits/is_trivially_move_constructible.h>
+#include <__type_traits/negation.h>
+#include <__utility/declval.h>
 #include <__utility/forward.h>
 #include <__utility/in_place.h>
 #include <__utility/move.h>
 #include <__utility/swap.h>
+#include <__verbose_abort>
 #include <initializer_list>
 #include <new>
 #include <stdexcept>
-#include <type_traits>
 #include <version>
 
 // standard-mandated includes
@@ -211,7 +221,7 @@ void __throw_bad_optional_access() {
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
         throw bad_optional_access();
 #else
-        _VSTD::abort();
+    _LIBCPP_VERBOSE_ABORT("bad_optional_access was thrown in -fno-exceptions mode");
 #endif
 }
 
@@ -1583,6 +1593,7 @@ _LIBCPP_END_NAMESPACE_STD
 #  include <memory>
 #  include <ratio>
 #  include <tuple>
+#  include <type_traits>
 #  include <typeinfo>
 #  include <utility>
 #  include <variant>
diff --git a/libcxx/include/ostream b/libcxx/include/ostream
index d2e422b328fd4..dbb54ac7e1a26 100644
--- a/libcxx/include/ostream
+++ b/libcxx/include/ostream
@@ -165,8 +165,14 @@ basic_ostream<wchar_t, traits>& operator<<(basic_ostream<wchar_t, traits>&, cons
 
 #include <__assert> // all public C++ headers provide the assertion handler
 #include <__config>
+#include <__exception/operations.h>
 #include <__memory/shared_ptr.h>
 #include <__memory/unique_ptr.h>
+#include <__type_traits/conjunction.h>
+#include <__type_traits/enable_if.h>
+#include <__type_traits/is_base_of.h>
+#include <__type_traits/void_t.h>
+#include <__utility/declval.h>
 #include <bitset>
 #include <ios>
 #include <locale>
@@ -1189,6 +1195,7 @@ extern template class _LIBCPP_EXTERN_TEMPLATE_TYPE_VIS basic_ostream<wchar_t>;
 _LIBCPP_END_NAMESPACE_STD
 
 #if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <atomic>
 #  include <concepts>
 #  include <iterator>
 #  include <type_traits>
diff --git a/libcxx/include/queue b/libcxx/include/queue
index f3ca8a696311d..35cceb5168959 100644
--- a/libcxx/include/queue
+++ b/libcxx/include/queue
@@ -227,7 +227,6 @@ template <class T, class Container, class Compare>
 #include <__memory/uses_allocator.h>
 #include <__utility/forward.h>
 #include <deque>
-#include <type_traits>
 #include <vector>
 #include <version>
 
@@ -965,6 +964,7 @@ _LIBCPP_END_NAMESPACE_STD
 #if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
 #  include <concepts>
 #  include <functional>
+#  include <type_traits>
 #endif
 
 #endif // _LIBCPP_QUEUE
diff --git a/libcxx/include/ranges b/libcxx/include/ranges
index c804709339e0b..f9a94a136cfbe 100644
--- a/libcxx/include/ranges
+++ b/libcxx/include/ranges
@@ -377,7 +377,6 @@ namespace std {
 #include <__ranges/view_interface.h>
 #include <__ranges/views.h>
 #include <__ranges/zip_view.h>
-#include <type_traits>
 #include <version>
 
 #if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
@@ -399,4 +398,8 @@ namespace std {
 #  pragma GCC system_header
 #endif
 
+#if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <type_traits>
+#endif
+
 #endif // _LIBCPP_RANGES
diff --git a/libcxx/include/regex b/libcxx/include/regex
index 1fe93a19a5157..9a367c3aabe79 100644
--- a/libcxx/include/regex
+++ b/libcxx/include/regex
@@ -776,6 +776,7 @@ typedef regex_token_iterator<wstring::const_iterator> wsregex_token_iterator;
 #include <__utility/move.h>
 #include <__utility/pair.h>
 #include <__utility/swap.h>
+#include <__verbose_abort>
 #include <deque>
 #include <stdexcept>
 #include <string>
@@ -1017,7 +1018,7 @@ void __throw_regex_error()
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
     throw regex_error(_Ev);
 #else
-    _VSTD::abort();
+    _LIBCPP_VERBOSE_ABORT("regex_error was thrown in -fno-exceptions mode");
 #endif
 }
 
diff --git a/libcxx/include/semaphore b/libcxx/include/semaphore
index 73cbc7f40a54a..cfbc2477a8baf 100644
--- a/libcxx/include/semaphore
+++ b/libcxx/include/semaphore
@@ -46,12 +46,15 @@ using binary_semaphore = counting_semaphore<1>;
 */
 
 #include <__assert> // all public C++ headers provide the assertion handler
+#include <__atomic/atomic_base.h>
+#include <__atomic/atomic_sync.h>
+#include <__atomic/memory_order.h>
 #include <__availability>
 #include <__chrono/time_point.h>
 #include <__config>
 #include <__thread/timed_backoff_policy.h>
 #include <__threading_support>
-#include <atomic>
+#include <cstddef>
 #include <limits>
 #include <version>
 
@@ -188,4 +191,8 @@ _LIBCPP_END_NAMESPACE_STD
 
 _LIBCPP_POP_MACROS
 
+#if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <atomic>
+#endif
+
 #endif //_LIBCPP_SEMAPHORE
diff --git a/libcxx/include/set b/libcxx/include/set
index 9bbb34cdcc912..289e35ae41bbc 100644
--- a/libcxx/include/set
+++ b/libcxx/include/set
@@ -1600,6 +1600,7 @@ _LIBCPP_END_NAMESPACE_STD
 #  include <concepts>
 #  include <functional>
 #  include <iterator>
+#  include <type_traits>
 #endif
 
 #endif // _LIBCPP_SET
diff --git a/libcxx/include/span b/libcxx/include/span
index 381404fe3a047..2ef1ea69d9587 100644
--- a/libcxx/include/span
+++ b/libcxx/include/span
@@ -141,11 +141,14 @@ template<class R>
 #include <__ranges/enable_borrowed_range.h>
 #include <__ranges/enable_view.h>
 #include <__ranges/size.h>
+#include <__type_traits/is_convertible.h>
+#include <__type_traits/remove_cvref.h>
+#include <__type_traits/remove_reference.h>
+#include <__type_traits/type_identity.h>
 #include <__utility/forward.h>
 #include <array>        // for array
 #include <cstddef>      // for byte
 #include <limits>
-#include <type_traits>  // for remove_cv, etc
 #include <version>
 
 // standard-mandated includes
@@ -238,10 +241,12 @@ public:
     template <__span_compatible_iterator<element_type> _It, __span_compatible_sentinel_for<_It> _End>
     _LIBCPP_INLINE_VISIBILITY
     constexpr explicit span(_It __first, _End __last) : __data_{_VSTD::to_address(__first)} {
-      (void)__last;
-      _LIBCPP_ASSERT((__last - __first >= 0), "invalid range in span's constructor (iterator, sentinel)");
-      _LIBCPP_ASSERT(__last - __first == _Extent,
-                     "invalid range in span's constructor (iterator, sentinel): last - first != extent");
+      // [span.cons]/10
+      // Throws: When and what last - first throws.
+      [[maybe_unused]] auto __dist = __last - __first;
+      _LIBCPP_ASSERT(__dist >= 0, "invalid range in span's constructor (iterator, sentinel)");
+      _LIBCPP_ASSERT(
+          __dist == _Extent, "invalid range in span's constructor (iterator, sentinel): last - first != extent");
     }
 
     _LIBCPP_INLINE_VISIBILITY constexpr span(type_identity_t<element_type> (&__arr)[_Extent]) noexcept : __data_{__arr} {}
@@ -593,6 +598,7 @@ _LIBCPP_POP_MACROS
 #  include <concepts>
 #  include <functional>
 #  include <iterator>
+#  include <type_traits>
 #endif
 
 #endif // _LIBCPP_SPAN
diff --git a/libcxx/include/stack b/libcxx/include/stack
index 3ee40742549e6..318c9c71ce24d 100644
--- a/libcxx/include/stack
+++ b/libcxx/include/stack
@@ -102,9 +102,9 @@ template <class T, class Container>
 #include <__config>
 #include <__iterator/iterator_traits.h>
 #include <__memory/uses_allocator.h>
+#include <__type_traits/is_same.h>
 #include <__utility/forward.h>
 #include <deque>
-#include <type_traits>
 #include <version>
 
 // standard-mandated includes
@@ -366,6 +366,7 @@ _LIBCPP_END_NAMESPACE_STD
 #if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
 #  include <concepts>
 #  include <functional>
+#  include <type_traits>
 #endif
 
 #endif // _LIBCPP_STACK
diff --git a/libcxx/include/stdexcept b/libcxx/include/stdexcept
index 6b7d10b5dc799..abdef87a5b0b3 100644
--- a/libcxx/include/stdexcept
+++ b/libcxx/include/stdexcept
@@ -43,8 +43,8 @@ public:
 
 #include <__assert> // all public C++ headers provide the assertion handler
 #include <__config>
+#include <__exception/exception.h>
 #include <cstdlib>
-#include <exception>
 #include <iosfwd>  // for string forward decl
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
@@ -223,8 +223,7 @@ void __throw_logic_error(const char*__msg)
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
     throw logic_error(__msg);
 #else
-    ((void)__msg);
-    _VSTD::abort();
+    _LIBCPP_VERBOSE_ABORT("logic_error was thrown in -fno-exceptions mode with message \"%s\"", __msg);
 #endif
 }
 
@@ -234,8 +233,7 @@ void __throw_domain_error(const char*__msg)
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
     throw domain_error(__msg);
 #else
-    ((void)__msg);
-    _VSTD::abort();
+    _LIBCPP_VERBOSE_ABORT("domain_error was thrown in -fno-exceptions mode with message \"%s\"", __msg);
 #endif
 }
 
@@ -245,8 +243,7 @@ void __throw_invalid_argument(const char*__msg)
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
     throw invalid_argument(__msg);
 #else
-    ((void)__msg);
-    _VSTD::abort();
+    _LIBCPP_VERBOSE_ABORT("invalid_argument was thrown in -fno-exceptions mode with message \"%s\"", __msg);
 #endif
 }
 
@@ -256,8 +253,7 @@ void __throw_length_error(const char*__msg)
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
     throw length_error(__msg);
 #else
-    ((void)__msg);
-    _VSTD::abort();
+    _LIBCPP_VERBOSE_ABORT("length_error was thrown in -fno-exceptions mode with message \"%s\"", __msg);
 #endif
 }
 
@@ -267,8 +263,7 @@ void __throw_out_of_range(const char*__msg)
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
     throw out_of_range(__msg);
 #else
-    ((void)__msg);
-    _VSTD::abort();
+    _LIBCPP_VERBOSE_ABORT("out_of_range was thrown in -fno-exceptions mode with message \"%s\"", __msg);
 #endif
 }
 
@@ -278,8 +273,7 @@ void __throw_range_error(const char*__msg)
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
     throw range_error(__msg);
 #else
-    ((void)__msg);
-    _VSTD::abort();
+    _LIBCPP_VERBOSE_ABORT("range_error was thrown in -fno-exceptions mode with message \"%s\"", __msg);
 #endif
 }
 
@@ -289,8 +283,7 @@ void __throw_overflow_error(const char*__msg)
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
     throw overflow_error(__msg);
 #else
-    ((void)__msg);
-    _VSTD::abort();
+    _LIBCPP_VERBOSE_ABORT("overflow_error was thrown in -fno-exceptions mode with message \"%s\"", __msg);
 #endif
 }
 
@@ -300,11 +293,14 @@ void __throw_underflow_error(const char*__msg)
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
     throw underflow_error(__msg);
 #else
-    ((void)__msg);
-    _VSTD::abort();
+    _LIBCPP_VERBOSE_ABORT("underflow_error was thrown in -fno-exceptions mode with message \"%s\"", __msg);
 #endif
 }
 
 _LIBCPP_END_NAMESPACE_STD
 
+#if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <exception>
+#endif
+
 #endif // _LIBCPP_STDEXCEPT
diff --git a/libcxx/include/string b/libcxx/include/string
index ede917f3431ed..8db92de1652f0 100644
--- a/libcxx/include/string
+++ b/libcxx/include/string
@@ -550,6 +550,7 @@ basic_string<char32_t> operator "" s( const char32_t *str, size_t len );
 #include <__iterator/iterator_traits.h>
 #include <__iterator/reverse_iterator.h>
 #include <__iterator/wrap_iter.h>
+#include <__memory/addressof.h>
 #include <__memory/allocate_at_least.h>
 #include <__memory/allocator.h>
 #include <__memory/allocator_traits.h>
@@ -561,7 +562,15 @@ basic_string<char32_t> operator "" s( const char32_t *str, size_t len );
 #include <__string/char_traits.h>
 #include <__string/extern_template_lists.h>
 #include <__type_traits/is_allocator.h>
+#include <__type_traits/is_array.h>
+#include <__type_traits/is_convertible.h>
+#include <__type_traits/is_nothrow_default_constructible.h>
+#include <__type_traits/is_nothrow_move_assignable.h>
+#include <__type_traits/is_same.h>
+#include <__type_traits/is_standard_layout.h>
+#include <__type_traits/is_trivial.h>
 #include <__type_traits/noexcept_move_assign_container.h>
+#include <__type_traits/remove_cvref.h>
 #include <__utility/auto_cast.h>
 #include <__utility/move.h>
 #include <__utility/swap.h>
@@ -574,7 +583,6 @@ basic_string<char32_t> operator "" s( const char32_t *str, size_t len );
 #include <limits>
 #include <stdexcept>
 #include <string_view>
-#include <type_traits>
 #include <version>
 
 #ifndef _LIBCPP_HAS_NO_WIDE_CHARACTERS
@@ -842,8 +850,23 @@ public:
     __default_init();
   }
 
-  _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string(const basic_string& __str);
-  _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string(const basic_string& __str, const allocator_type& __a);
+  _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string(const basic_string& __str)
+      : __r_(__default_init_tag(), __alloc_traits::select_on_container_copy_construction(__str.__alloc())) {
+    if (!__str.__is_long())
+      __r_.first() = __str.__r_.first();
+    else
+      __init_copy_ctor_external(std::__to_address(__str.__get_long_pointer()), __str.__get_long_size());
+    std::__debug_db_insert_c(this);
+  }
+
+  _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string(const basic_string& __str, const allocator_type& __a)
+      : __r_(__default_init_tag(), __a) {
+    if (!__str.__is_long())
+      __r_.first() = __str.__r_.first();
+    else
+      __init_copy_ctor_external(std::__to_address(__str.__get_long_pointer()), __str.__get_long_size());
+    std::__debug_db_insert_c(this);
+  }
 
 #ifndef _LIBCPP_CXX03_LANG
   _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string(basic_string&& __str)
@@ -875,7 +898,7 @@ public:
   }
 #endif // _LIBCPP_CXX03_LANG
 
-  template <class = __enable_if_t<__is_allocator<_Allocator>::value, nullptr_t> >
+  template <__enable_if_t<__is_allocator<_Allocator>::value, int> = 0>
   _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string(const _CharT* __s)
       : __r_(__default_init_tag(), __default_init_tag()) {
     _LIBCPP_ASSERT(__s != nullptr, "basic_string(const char*) detected nullptr");
@@ -883,8 +906,13 @@ public:
     std::__debug_db_insert_c(this);
   }
 
-  template <class = __enable_if_t<__is_allocator<_Allocator>::value, nullptr_t> >
-  _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string(const _CharT* __s, const _Allocator& __a);
+  template <__enable_if_t<__is_allocator<_Allocator>::value, int> = 0>
+  _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string(const _CharT* __s, const _Allocator& __a)
+      : __r_(__default_init_tag(), __a) {
+    _LIBCPP_ASSERT(__s != nullptr, "basic_string(const char*, allocator) detected nullptr");
+    __init(__s, traits_type::length(__s));
+    std::__debug_db_insert_c(this);
+  }
 
 #if _LIBCPP_STD_VER >= 23
   basic_string(nullptr_t) = delete;
@@ -941,11 +969,22 @@ public:
   }
 #endif
 
-  template <class = __enable_if_t<__is_allocator<_Allocator>::value, nullptr_t> >
-  _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string(size_type __n, _CharT __c, const _Allocator& __a);
+  template <__enable_if_t<__is_allocator<_Allocator>::value, int> = 0>
+  _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string(size_type __n, _CharT __c, const _Allocator& __a)
+      : __r_(__default_init_tag(), __a) {
+    __init(__n, __c);
+    std::__debug_db_insert_c(this);
+  }
 
   _LIBCPP_CONSTEXPR_SINCE_CXX20
-  basic_string(const basic_string& __str, size_type __pos, size_type __n, const _Allocator& __a = _Allocator());
+  basic_string(const basic_string& __str, size_type __pos, size_type __n, const _Allocator& __a = _Allocator())
+      : __r_(__default_init_tag(), __a) {
+    size_type __str_sz = __str.size();
+    if (__pos > __str_sz)
+      __throw_out_of_range();
+    __init(__str.data() + __pos, std::min(__n, __str_sz - __pos));
+    std::__debug_db_insert_c(this);
+  }
 
   _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
   basic_string(const basic_string& __str, size_type __pos, const _Allocator& __a = _Allocator())
@@ -958,31 +997,49 @@ public:
   }
 
   template <class _Tp,
-            class = __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
-                                  !__is_same_uncvref<_Tp, basic_string>::value> >
+            __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
+                              !__is_same_uncvref<_Tp, basic_string>::value,
+                          int> = 0>
   _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-  basic_string(const _Tp& __t, size_type __pos, size_type __n, const allocator_type& __a = allocator_type());
+  basic_string(const _Tp& __t, size_type __pos, size_type __n, const allocator_type& __a = allocator_type())
+      : __r_(__default_init_tag(), __a) {
+    __self_view __sv0 = __t;
+    __self_view __sv  = __sv0.substr(__pos, __n);
+    __init(__sv.data(), __sv.size());
+    std::__debug_db_insert_c(this);
+  }
 
   template <class _Tp,
-            class = __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
-                                  !__is_same_uncvref<_Tp, basic_string>::value> >
-  _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 explicit basic_string(
-      const _Tp& __t);
+            __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
+                              !__is_same_uncvref<_Tp, basic_string>::value,
+                          int> = 0>
+  _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 explicit basic_string(const _Tp& __t)
+      : __r_(__default_init_tag(), __default_init_tag()) {
+    __self_view __sv = __t;
+    __init(__sv.data(), __sv.size());
+    std::__debug_db_insert_c(this);
+  }
 
   template <class _Tp,
-            class = __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
-                                  !__is_same_uncvref<_Tp, basic_string>::value> >
+            __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
+                              !__is_same_uncvref<_Tp, basic_string>::value,
+                          int> = 0>
   _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 explicit basic_string(
-      const _Tp& __t, const allocator_type& __a);
+      const _Tp& __t, const allocator_type& __a)
+      : __r_(__default_init_tag(), __a) {
+    __self_view __sv = __t;
+    __init(__sv.data(), __sv.size());
+    std::__debug_db_insert_c(this);
+  }
 
-  template <class _InputIterator, class = __enable_if_t<__is_cpp17_input_iterator<_InputIterator>::value> >
+  template <class _InputIterator, __enable_if_t<__is_cpp17_input_iterator<_InputIterator>::value, int> = 0>
   _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string(_InputIterator __first, _InputIterator __last)
       : __r_(__default_init_tag(), __default_init_tag()) {
     __init(__first, __last);
     std::__debug_db_insert_c(this);
   }
 
-  template <class _InputIterator, class = __enable_if_t<__is_cpp17_input_iterator<_InputIterator>::value> >
+  template <class _InputIterator, __enable_if_t<__is_cpp17_input_iterator<_InputIterator>::value, int> = 0>
   _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
   basic_string(_InputIterator __first, _InputIterator __last, const allocator_type& __a)
       : __r_(__default_init_tag(), __a) {
@@ -1004,15 +1061,19 @@ public:
   }
 #endif // _LIBCPP_CXX03_LANG
 
-    inline _LIBCPP_CONSTEXPR_SINCE_CXX20 ~basic_string();
+  inline _LIBCPP_CONSTEXPR_SINCE_CXX20 ~basic_string() {
+    std::__debug_db_erase_c(this);
+    if (__is_long())
+      __alloc_traits::deallocate(__alloc(), __get_long_pointer(), __get_long_cap());
+  }
 
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
     operator __self_view() const _NOEXCEPT { return __self_view(data(), size()); }
 
     _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& operator=(const basic_string& __str);
 
-    template <class _Tp, class = __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
-                                               !__is_same_uncvref<_Tp, basic_string>::value> >
+    template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
+                                           !__is_same_uncvref<_Tp, basic_string>::value, int> = 0>
     _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& operator=(const _Tp& __t) {
       __self_view __sv = __t;
       return assign(__sv);
@@ -1132,14 +1193,11 @@ public:
         return append(__str);
     }
 
-    template <class _Tp>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value
-            && !__is_same_uncvref<_Tp, basic_string >::value,
-            basic_string&
-        >
+    template <class _Tp,
+              __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
+                                !__is_same_uncvref<_Tp, basic_string >::value,
+                            int> = 0>
+    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
     operator+=(const _Tp& __t) {
         __self_view __sv = __t; return append(__sv);
     }
@@ -1162,25 +1220,27 @@ public:
         return append(__str.data(), __str.size());
   }
 
-    template <class _Tp>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t<
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value
-            && !__is_same_uncvref<_Tp, basic_string>::value,
-            basic_string&
-        >
-                  append(const _Tp& __t) { __self_view __sv = __t; return append(__sv.data(), __sv.size()); }
+  template <class _Tp,
+            __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
+                              !__is_same_uncvref<_Tp, basic_string>::value,
+                          int> = 0>
+  _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
+  append(const _Tp& __t) {
+        __self_view __sv = __t;
+        return append(__sv.data(), __sv.size());
+  }
+
     _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& append(const basic_string& __str, size_type __pos, size_type __n=npos);
 
-    template <class _Tp>
+    template <class _Tp,
+              __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
+                                !__is_same_uncvref<_Tp, basic_string>::value,
+                            int> = 0>
     _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value
-            && !__is_same_uncvref<_Tp, basic_string>::value,
-            basic_string&
-        >
-                  append(const _Tp& __t, size_type __pos, size_type __n=npos);
+
+        basic_string&
+        append(const _Tp& __t, size_type __pos, size_type __n = npos);
+
     _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& append(const value_type* __s, size_type __n);
     _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& append(const value_type* __s);
     _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& append(size_type __n, value_type __c);
@@ -1188,27 +1248,16 @@ public:
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
     void __append_default_init(size_type __n);
 
-    template<class _InputIterator>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS
-    __enable_if_t
-        <
-            __is_exactly_cpp17_input_iterator<_InputIterator>::value,
-            basic_string&
-        >
-    _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
+    template <class _InputIterator, __enable_if_t<__is_exactly_cpp17_input_iterator<_InputIterator>::value, int> = 0>
+    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
     append(_InputIterator __first, _InputIterator __last) {
-      const basic_string __temp(__first, __last, __alloc());
-      append(__temp.data(), __temp.size());
-      return *this;
+        const basic_string __temp(__first, __last, __alloc());
+        append(__temp.data(), __temp.size());
+        return *this;
     }
-    template<class _ForwardIterator>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS
-    __enable_if_t
-        <
-            __is_cpp17_forward_iterator<_ForwardIterator>::value,
-            basic_string&
-        >
-    _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
+
+    template <class _ForwardIterator, __enable_if_t<__is_cpp17_forward_iterator<_ForwardIterator>::value, int> = 0>
+    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
     append(_ForwardIterator __first, _ForwardIterator __last);
 
 #ifndef _LIBCPP_CXX03_LANG
@@ -1239,14 +1288,13 @@ public:
     return *(data() + size() - 1);
   }
 
-    template <class _Tp>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-            basic_string&
-        >
-                 assign(const _Tp & __t) { __self_view __sv = __t; return assign(__sv.data(), __sv.size()); }
+  template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> = 0>
+  _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
+  assign(const _Tp& __t) {
+    __self_view __sv = __t;
+    return assign(__sv.data(), __sv.size());
+  }
+
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
     basic_string& assign(const basic_string& __str) { return *this = __str; }
 #ifndef _LIBCPP_CXX03_LANG
@@ -1256,34 +1304,25 @@ public:
         {*this = std::move(__str); return *this;}
 #endif
     _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& assign(const basic_string& __str, size_type __pos, size_type __n=npos);
-    template <class _Tp>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value
-            && !__is_same_uncvref<_Tp, basic_string>::value,
-            basic_string&
-        >
-                  assign(const _Tp & __t, size_type __pos, size_type __n=npos);
+
+    template <class _Tp,
+              __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
+                                !__is_same_uncvref<_Tp, basic_string>::value,
+                            int> = 0>
+    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
+    assign(const _Tp& __t, size_type __pos, size_type __n = npos);
+
     _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& assign(const value_type* __s, size_type __n);
     _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& assign(const value_type* __s);
     _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& assign(size_type __n, value_type __c);
-    template<class _InputIterator>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __is_exactly_cpp17_input_iterator<_InputIterator>::value,
-            basic_string&
-        >
-        assign(_InputIterator __first, _InputIterator __last);
-    template<class _ForwardIterator>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __is_cpp17_forward_iterator<_ForwardIterator>::value,
-            basic_string&
-        >
-        assign(_ForwardIterator __first, _ForwardIterator __last);
+    template <class _InputIterator, __enable_if_t<__is_exactly_cpp17_input_iterator<_InputIterator>::value, int> = 0>
+    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
+    assign(_InputIterator __first, _InputIterator __last);
+
+    template <class _ForwardIterator, __enable_if_t<__is_cpp17_forward_iterator<_ForwardIterator>::value, int> = 0>
+    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
+    assign(_ForwardIterator __first, _ForwardIterator __last);
+
 #ifndef _LIBCPP_CXX03_LANG
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
     basic_string& assign(initializer_list<value_type> __il) {return assign(__il.begin(), __il.size());}
@@ -1294,30 +1333,26 @@ public:
     return insert(__pos1, __str.data(), __str.size());
   }
 
-    template <class _Tp>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-            basic_string&
-        >
-                 insert(size_type __pos1, const _Tp& __t)
-    { __self_view __sv = __t; return insert(__pos1, __sv.data(), __sv.size()); }
-
-    template <class _Tp>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value && !__is_same_uncvref<_Tp, basic_string>::value,
-            basic_string&
-        >
-                  insert(size_type __pos1, const _Tp& __t, size_type __pos2, size_type __n=npos);
-    _LIBCPP_CONSTEXPR_SINCE_CXX20
-    basic_string& insert(size_type __pos1, const basic_string& __str, size_type __pos2, size_type __n=npos);
-    _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& insert(size_type __pos, const value_type* __s, size_type __n);
-    _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& insert(size_type __pos, const value_type* __s);
-    _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& insert(size_type __pos, size_type __n, value_type __c);
-    _LIBCPP_CONSTEXPR_SINCE_CXX20 iterator      insert(const_iterator __pos, value_type __c);
+  template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> = 0>
+  _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
+  insert(size_type __pos1, const _Tp& __t) {
+    __self_view __sv = __t;
+    return insert(__pos1, __sv.data(), __sv.size());
+  }
+
+  template <class _Tp,
+            __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
+                              !__is_same_uncvref<_Tp, basic_string>::value,
+                          int> = 0>
+  _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
+  insert(size_type __pos1, const _Tp& __t, size_type __pos2, size_type __n = npos);
+
+  _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
+  insert(size_type __pos1, const basic_string& __str, size_type __pos2, size_type __n = npos);
+  _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& insert(size_type __pos, const value_type* __s, size_type __n);
+  _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& insert(size_type __pos, const value_type* __s);
+  _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& insert(size_type __pos, size_type __n, value_type __c);
+  _LIBCPP_CONSTEXPR_SINCE_CXX20 iterator insert(const_iterator __pos, value_type __c);
 
   _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20 iterator
   insert(const_iterator __pos, size_type __n, value_type __c) {
@@ -1328,22 +1363,14 @@ public:
     return begin() + __p;
   }
 
-    template<class _InputIterator>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __is_exactly_cpp17_input_iterator<_InputIterator>::value,
-            iterator
-        >
-        insert(const_iterator __pos, _InputIterator __first, _InputIterator __last);
-    template<class _ForwardIterator>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __is_cpp17_forward_iterator<_ForwardIterator>::value,
-            iterator
-        >
-        insert(const_iterator __pos, _ForwardIterator __first, _ForwardIterator __last);
+  template <class _InputIterator, __enable_if_t<__is_exactly_cpp17_input_iterator<_InputIterator>::value, int> = 0>
+  _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 iterator
+  insert(const_iterator __pos, _InputIterator __first, _InputIterator __last);
+
+  template <class _ForwardIterator, __enable_if_t<__is_cpp17_forward_iterator<_ForwardIterator>::value, int> = 0>
+  _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 iterator
+  insert(const_iterator __pos, _ForwardIterator __first, _ForwardIterator __last);
+
 #ifndef _LIBCPP_CXX03_LANG
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
     iterator insert(const_iterator __pos, initializer_list<value_type> __il)
@@ -1361,28 +1388,27 @@ public:
     return replace(__pos1, __n1, __str.data(), __str.size());
   }
 
-    template <class _Tp>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-            basic_string&
-        >
-                  replace(size_type __pos1, size_type __n1, const _Tp& __t) { __self_view __sv = __t; return replace(__pos1, __n1, __sv.data(), __sv.size()); }
-    _LIBCPP_CONSTEXPR_SINCE_CXX20
-    basic_string& replace(size_type __pos1, size_type __n1, const basic_string& __str, size_type __pos2, size_type __n2=npos);
-    template <class _Tp>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value  && !__is_same_uncvref<_Tp, basic_string>::value,
-            basic_string&
-        >
-                  replace(size_type __pos1, size_type __n1, const _Tp& __t, size_type __pos2, size_type __n2=npos);
-    _LIBCPP_CONSTEXPR_SINCE_CXX20
-    basic_string& replace(size_type __pos, size_type __n1, const value_type* __s, size_type __n2);
-    _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& replace(size_type __pos, size_type __n1, const value_type* __s);
-    _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& replace(size_type __pos, size_type __n1, size_type __n2, value_type __c);
+  template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> = 0>
+  _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
+  replace(size_type __pos1, size_type __n1, const _Tp& __t) {
+    __self_view __sv = __t;
+    return replace(__pos1, __n1, __sv.data(), __sv.size());
+  }
+
+  _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
+  replace(size_type __pos1, size_type __n1, const basic_string& __str, size_type __pos2, size_type __n2 = npos);
+
+  template <class _Tp,
+            __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
+                              !__is_same_uncvref<_Tp, basic_string>::value,
+                          int> = 0>
+  _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
+  replace(size_type __pos1, size_type __n1, const _Tp& __t, size_type __pos2, size_type __n2 = npos);
+
+  _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
+  replace(size_type __pos, size_type __n1, const value_type* __s, size_type __n2);
+  _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& replace(size_type __pos, size_type __n1, const value_type* __s);
+  _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string& replace(size_type __pos, size_type __n1, size_type __n2, value_type __c);
 
   _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
   replace(const_iterator __i1, const_iterator __i2, const basic_string& __str) {
@@ -1390,14 +1416,12 @@ public:
         static_cast<size_type>(__i1 - begin()), static_cast<size_type>(__i2 - __i1), __str.data(), __str.size());
   }
 
-    template <class _Tp>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-            basic_string&
-        >
-                  replace(const_iterator __i1, const_iterator __i2, const _Tp& __t) { __self_view __sv = __t; return replace(__i1 - begin(), __i2 - __i1, __sv); }
+  template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> = 0>
+  _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
+  replace(const_iterator __i1, const_iterator __i2, const _Tp& __t) {
+    __self_view __sv = __t;
+    return replace(__i1 - begin(), __i2 - __i1, __sv);
+  }
 
   _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
   replace(const_iterator __i1, const_iterator __i2, const value_type* __s, size_type __n) {
@@ -1414,14 +1438,10 @@ public:
     return replace(static_cast<size_type>(__i1 - begin()), static_cast<size_type>(__i2 - __i1), __n, __c);
   }
 
-    template<class _InputIterator>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __is_cpp17_input_iterator<_InputIterator>::value,
-            basic_string&
-        >
-        replace(const_iterator __i1, const_iterator __i2, _InputIterator __j1, _InputIterator __j2);
+  template <class _InputIterator, __enable_if_t<__is_cpp17_input_iterator<_InputIterator>::value, int> = 0>
+  _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string&
+  replace(const_iterator __i1, const_iterator __i2, _InputIterator __j1, _InputIterator __j2);
+
 #ifndef _LIBCPP_CXX03_LANG
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
     basic_string& replace(const_iterator __i1, const_iterator __i2, initializer_list<value_type> __il)
@@ -1471,16 +1491,11 @@ public:
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
     size_type find(const basic_string& __str, size_type __pos = 0) const _NOEXCEPT;
 
-    template <class _Tp>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-            size_type
-        >
-              find(const _Tp& __t, size_type __pos = 0) const _NOEXCEPT;
-    _LIBCPP_CONSTEXPR_SINCE_CXX20
-    size_type find(const value_type* __s, size_type __pos, size_type __n) const _NOEXCEPT;
+    template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> = 0>
+    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 size_type
+    find(const _Tp& __t, size_type __pos = 0) const _NOEXCEPT;
+
+    _LIBCPP_CONSTEXPR_SINCE_CXX20 size_type find(const value_type* __s, size_type __pos, size_type __n) const _NOEXCEPT;
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
     size_type find(const value_type* __s, size_type __pos = 0) const _NOEXCEPT;
     _LIBCPP_CONSTEXPR_SINCE_CXX20 size_type find(value_type __c, size_type __pos = 0) const _NOEXCEPT;
@@ -1488,14 +1503,10 @@ public:
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
     size_type rfind(const basic_string& __str, size_type __pos = npos) const _NOEXCEPT;
 
-    template <class _Tp>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-            size_type
-        >
-              rfind(const _Tp& __t, size_type __pos = npos) const _NOEXCEPT;
+    template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> = 0>
+    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 size_type
+    rfind(const _Tp& __t, size_type __pos = npos) const _NOEXCEPT;
+
     _LIBCPP_CONSTEXPR_SINCE_CXX20
     size_type rfind(const value_type* __s, size_type __pos, size_type __n) const _NOEXCEPT;
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
@@ -1505,14 +1516,10 @@ public:
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
     size_type find_first_of(const basic_string& __str, size_type __pos = 0) const _NOEXCEPT;
 
-    template <class _Tp>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-            size_type
-        >
-              find_first_of(const _Tp& __t, size_type __pos = 0) const _NOEXCEPT;
+    template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> = 0>
+    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 size_type
+    find_first_of(const _Tp& __t, size_type __pos = 0) const _NOEXCEPT;
+
     _LIBCPP_CONSTEXPR_SINCE_CXX20
     size_type find_first_of(const value_type* __s, size_type __pos, size_type __n) const _NOEXCEPT;
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
@@ -1523,14 +1530,10 @@ public:
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
     size_type find_last_of(const basic_string& __str, size_type __pos = npos) const _NOEXCEPT;
 
-    template <class _Tp>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-            size_type
-        >
-              find_last_of(const _Tp& __t, size_type __pos = npos) const _NOEXCEPT;
+    template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> = 0>
+    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 size_type
+    find_last_of(const _Tp& __t, size_type __pos = npos) const _NOEXCEPT;
+
     _LIBCPP_CONSTEXPR_SINCE_CXX20
     size_type find_last_of(const value_type* __s, size_type __pos, size_type __n) const _NOEXCEPT;
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
@@ -1541,14 +1544,10 @@ public:
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
     size_type find_first_not_of(const basic_string& __str, size_type __pos = 0) const _NOEXCEPT;
 
-    template <class _Tp>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-            size_type
-        >
-              find_first_not_of(const _Tp &__t, size_type __pos = 0) const _NOEXCEPT;
+    template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> = 0>
+    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 size_type
+    find_first_not_of(const _Tp& __t, size_type __pos = 0) const _NOEXCEPT;
+
     _LIBCPP_CONSTEXPR_SINCE_CXX20
     size_type find_first_not_of(const value_type* __s, size_type __pos, size_type __n) const _NOEXCEPT;
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
@@ -1559,14 +1558,10 @@ public:
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
     size_type find_last_not_of(const basic_string& __str, size_type __pos = npos) const _NOEXCEPT;
 
-    template <class _Tp>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-            size_type
-        >
-              find_last_not_of(const _Tp& __t, size_type __pos = npos) const _NOEXCEPT;
+    template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> = 0>
+    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 size_type
+    find_last_not_of(const _Tp& __t, size_type __pos = npos) const _NOEXCEPT;
+
     _LIBCPP_CONSTEXPR_SINCE_CXX20
     size_type find_last_not_of(const value_type* __s, size_type __pos, size_type __n) const _NOEXCEPT;
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
@@ -1577,23 +1572,13 @@ public:
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
     int compare(const basic_string& __str) const _NOEXCEPT;
 
-    template <class _Tp>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-            int
-        >
-        compare(const _Tp &__t) const _NOEXCEPT;
-
-    template <class _Tp>
-    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-        <
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-            int
-        >
-         compare(size_type __pos1, size_type __n1, const _Tp& __t) const;
+    template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> = 0>
+    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 int
+    compare(const _Tp& __t) const _NOEXCEPT;
+
+    template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> = 0>
+    _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS _LIBCPP_CONSTEXPR_SINCE_CXX20 int
+    compare(size_type __pos1, size_type __n1, const _Tp& __t) const;
 
     _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
     int compare(size_type __pos1, size_type __n1, const basic_string& __str) const;
@@ -1601,14 +1586,13 @@ public:
     int compare(size_type __pos1, size_type __n1, const basic_string& __str, size_type __pos2,
                 size_type __n2 = npos) const;
 
-    template <class _Tp>
-    inline _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
-        __enable_if_t
-        <
-            __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value  && !__is_same_uncvref<_Tp, basic_string>::value,
-            int
-        >
-        compare(size_type __pos1, size_type __n1, const _Tp& __t, size_type __pos2, size_type __n2=npos) const;
+    template <class _Tp,
+              __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
+                                !__is_same_uncvref<_Tp, basic_string>::value,
+                            int> = 0>
+    inline _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20 int
+    compare(size_type __pos1, size_type __n1, const _Tp& __t, size_type __pos2, size_type __n2 = npos) const;
+
     _LIBCPP_CONSTEXPR_SINCE_CXX20 int compare(const value_type* __s) const _NOEXCEPT;
     _LIBCPP_CONSTEXPR_SINCE_CXX20 int compare(size_type __pos1, size_type __n1, const value_type* __s) const;
     _LIBCPP_CONSTEXPR_SINCE_CXX20
@@ -1839,21 +1823,11 @@ private:
     // to only inline the fast path code directly in the ctor.
     _LIBCPP_CONSTEXPR_SINCE_CXX20 void __init_copy_ctor_external(const value_type* __s, size_type __sz);
 
-    template <class _InputIterator>
-    inline _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-    <
-        __is_exactly_cpp17_input_iterator<_InputIterator>::value
-    >
-    __init(_InputIterator __first, _InputIterator __last);
+    template <class _InputIterator, __enable_if_t<__is_exactly_cpp17_input_iterator<_InputIterator>::value, int> = 0>
+    inline _LIBCPP_CONSTEXPR_SINCE_CXX20 void __init(_InputIterator __first, _InputIterator __last);
 
-    template <class _ForwardIterator>
-    inline _LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-    <
-        __is_cpp17_forward_iterator<_ForwardIterator>::value
-    >
-    __init(_ForwardIterator __first, _ForwardIterator __last);
+    template <class _ForwardIterator, __enable_if_t<__is_cpp17_forward_iterator<_ForwardIterator>::value, int> = 0>
+    inline _LIBCPP_CONSTEXPR_SINCE_CXX20 void __init(_ForwardIterator __first, _ForwardIterator __last);
 
     _LIBCPP_CONSTEXPR_SINCE_CXX20
     void __grow_by(size_type __old_cap, size_type __delta_cap, size_type __old_sz,
@@ -2128,44 +2102,6 @@ basic_string<_CharT, _Traits, _Allocator>::__init(const value_type* __s, size_ty
     traits_type::assign(__p[__sz], value_type());
 }
 
-template <class _CharT, class _Traits, class _Allocator>
-template <class>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-basic_string<_CharT, _Traits, _Allocator>::basic_string(const _CharT* __s, const _Allocator& __a)
-    : __r_(__default_init_tag(), __a)
-{
-    _LIBCPP_ASSERT(__s != nullptr, "basic_string(const char*, allocator) detected nullptr");
-    __init(__s, traits_type::length(__s));
-    std::__debug_db_insert_c(this);
-}
-
-template <class _CharT, class _Traits, class _Allocator>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-basic_string<_CharT, _Traits, _Allocator>::basic_string(const basic_string& __str)
-    : __r_(__default_init_tag(), __alloc_traits::select_on_container_copy_construction(__str.__alloc()))
-{
-    if (!__str.__is_long())
-        __r_.first() = __str.__r_.first();
-    else
-        __init_copy_ctor_external(std::__to_address(__str.__get_long_pointer()),
-                                  __str.__get_long_size());
-    std::__debug_db_insert_c(this);
-}
-
-template <class _CharT, class _Traits, class _Allocator>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-basic_string<_CharT, _Traits, _Allocator>::basic_string(
-    const basic_string& __str, const allocator_type& __a)
-    : __r_(__default_init_tag(), __a)
-{
-    if (!__str.__is_long())
-        __r_.first() = __str.__r_.first();
-    else
-        __init_copy_ctor_external(std::__to_address(__str.__get_long_pointer()),
-                                  __str.__get_long_size());
-    std::__debug_db_insert_c(this);
-}
-
 template <class _CharT, class _Traits, class _Allocator>
 _LIBCPP_CONSTEXPR_SINCE_CXX20
 void basic_string<_CharT, _Traits, _Allocator>::__init_copy_ctor_external(
@@ -2220,72 +2156,9 @@ basic_string<_CharT, _Traits, _Allocator>::__init(size_type __n, value_type __c)
 }
 
 template <class _CharT, class _Traits, class _Allocator>
-template <class>
+template <class _InputIterator, __enable_if_t<__is_exactly_cpp17_input_iterator<_InputIterator>::value, int> >
 _LIBCPP_CONSTEXPR_SINCE_CXX20
-basic_string<_CharT, _Traits, _Allocator>::basic_string(size_type __n, _CharT __c, const _Allocator& __a)
-    : __r_(__default_init_tag(), __a)
-{
-    __init(__n, __c);
-    std::__debug_db_insert_c(this);
-}
-
-template <class _CharT, class _Traits, class _Allocator>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-basic_string<_CharT, _Traits, _Allocator>::basic_string(const basic_string& __str,
-                                                        size_type __pos, size_type __n,
-                                                        const _Allocator& __a)
-    : __r_(__default_init_tag(), __a)
-{
-    size_type __str_sz = __str.size();
-    if (__pos > __str_sz)
-        __throw_out_of_range();
-    __init(__str.data() + __pos, std::min(__n, __str_sz - __pos));
-    std::__debug_db_insert_c(this);
-}
-
-template <class _CharT, class _Traits, class _Allocator>
-template <class _Tp, class>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-basic_string<_CharT, _Traits, _Allocator>::basic_string(
-             const _Tp& __t, size_type __pos, size_type __n, const allocator_type& __a)
-    : __r_(__default_init_tag(), __a)
-{
-    __self_view __sv0 = __t;
-    __self_view __sv = __sv0.substr(__pos, __n);
-    __init(__sv.data(), __sv.size());
-    std::__debug_db_insert_c(this);
-}
-
-template <class _CharT, class _Traits, class _Allocator>
-template <class _Tp, class>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-basic_string<_CharT, _Traits, _Allocator>::basic_string(const _Tp & __t)
-     : __r_(__default_init_tag(), __default_init_tag())
-{
-    __self_view __sv = __t;
-    __init(__sv.data(), __sv.size());
-    std::__debug_db_insert_c(this);
-}
-
-template <class _CharT, class _Traits, class _Allocator>
-template <class _Tp, class>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-basic_string<_CharT, _Traits, _Allocator>::basic_string(const _Tp & __t, const _Allocator& __a)
-    : __r_(__default_init_tag(), __a)
-{
-    __self_view __sv = __t;
-    __init(__sv.data(), __sv.size());
-    std::__debug_db_insert_c(this);
-}
-
-template <class _CharT, class _Traits, class _Allocator>
-template <class _InputIterator>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-    __is_exactly_cpp17_input_iterator<_InputIterator>::value
->
-basic_string<_CharT, _Traits, _Allocator>::__init(_InputIterator __first, _InputIterator __last)
+void basic_string<_CharT, _Traits, _Allocator>::__init(_InputIterator __first, _InputIterator __last)
 {
     __default_init();
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
@@ -2306,12 +2179,8 @@ basic_string<_CharT, _Traits, _Allocator>::__init(_InputIterator __first, _Input
 }
 
 template <class _CharT, class _Traits, class _Allocator>
-template <class _ForwardIterator>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-    __is_cpp17_forward_iterator<_ForwardIterator>::value
->
+template <class _ForwardIterator, __enable_if_t<__is_cpp17_forward_iterator<_ForwardIterator>::value, int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 void
 basic_string<_CharT, _Traits, _Allocator>::__init(_ForwardIterator __first, _ForwardIterator __last)
 {
     if (__libcpp_is_constant_evaluated())
@@ -2353,15 +2222,6 @@ basic_string<_CharT, _Traits, _Allocator>::__init(_ForwardIterator __first, _For
 #endif  // _LIBCPP_HAS_NO_EXCEPTIONS
 }
 
-template <class _CharT, class _Traits, class _Allocator>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-basic_string<_CharT, _Traits, _Allocator>::~basic_string()
-{
-    std::__debug_db_erase_c(this);
-    if (__is_long())
-        __alloc_traits::deallocate(__alloc(), __get_long_pointer(), __get_long_cap());
-}
-
 template <class _CharT, class _Traits, class _Allocator>
 _LIBCPP_CONSTEXPR_SINCE_CXX20
 void
@@ -2522,7 +2382,7 @@ _LIBCPP_CONSTEXPR_SINCE_CXX20
 basic_string<_CharT, _Traits, _Allocator>&
 basic_string<_CharT, _Traits, _Allocator>::operator=(const basic_string& __str)
 {
-  if (this != &__str) {
+  if (this != std::addressof(__str)) {
     __copy_assign_alloc(__str);
     if (!__is_long()) {
       if (!__str.__is_long()) {
@@ -2584,13 +2444,8 @@ basic_string<_CharT, _Traits, _Allocator>::__move_assign(basic_string& __str, tr
 #endif
 
 template <class _CharT, class _Traits, class _Allocator>
-template<class _InputIterator>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-     __is_exactly_cpp17_input_iterator<_InputIterator>::value,
-    basic_string<_CharT, _Traits, _Allocator>&
->
+template<class _InputIterator, __enable_if_t<__is_exactly_cpp17_input_iterator<_InputIterator>::value, int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string<_CharT, _Traits, _Allocator>&
 basic_string<_CharT, _Traits, _Allocator>::assign(_InputIterator __first, _InputIterator __last)
 {
     const basic_string __temp(__first, __last, __alloc());
@@ -2599,13 +2454,8 @@ basic_string<_CharT, _Traits, _Allocator>::assign(_InputIterator __first, _Input
 }
 
 template <class _CharT, class _Traits, class _Allocator>
-template<class _ForwardIterator>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-    __is_cpp17_forward_iterator<_ForwardIterator>::value,
-    basic_string<_CharT, _Traits, _Allocator>&
->
+template<class _ForwardIterator, __enable_if_t<__is_cpp17_forward_iterator<_ForwardIterator>::value, int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string<_CharT, _Traits, _Allocator>&
 basic_string<_CharT, _Traits, _Allocator>::assign(_ForwardIterator __first, _ForwardIterator __last)
 {
     size_type __cap = capacity();
@@ -2647,16 +2497,12 @@ basic_string<_CharT, _Traits, _Allocator>::assign(const basic_string& __str, siz
 }
 
 template <class _CharT, class _Traits, class _Allocator>
-template <class _Tp>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-    __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value
-    && !__is_same_uncvref<_Tp, basic_string<_CharT, _Traits, _Allocator> >::value,
-    basic_string<_CharT, _Traits, _Allocator>&
->
-basic_string<_CharT, _Traits, _Allocator>::assign(const _Tp & __t, size_type __pos, size_type __n)
-{
+template <class _Tp,
+          __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
+                            !__is_same_uncvref<_Tp, basic_string<_CharT, _Traits, _Allocator> >::value,
+                        int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string<_CharT, _Traits, _Allocator>&
+basic_string<_CharT, _Traits, _Allocator>::assign(const _Tp& __t, size_type __pos, size_type __n) {
     __self_view __sv = __t;
     size_type __sz = __sv.size();
     if (__pos > __sz)
@@ -2664,7 +2510,6 @@ basic_string<_CharT, _Traits, _Allocator>::assign(const _Tp & __t, size_type __p
     return assign(__sv.data() + __pos, std::min(__n, __sz - __pos));
 }
 
-
 template <class _CharT, class _Traits, class _Allocator>
 _LIBCPP_CONSTEXPR_SINCE_CXX20
 basic_string<_CharT, _Traits, _Allocator>&
@@ -2786,13 +2631,8 @@ basic_string<_CharT, _Traits, _Allocator>::push_back(value_type __c)
 }
 
 template <class _CharT, class _Traits, class _Allocator>
-template<class _ForwardIterator>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-    __is_cpp17_forward_iterator<_ForwardIterator>::value,
-    basic_string<_CharT, _Traits, _Allocator>&
->
+template<class _ForwardIterator, __enable_if_t<__is_cpp17_forward_iterator<_ForwardIterator>::value, int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string<_CharT, _Traits, _Allocator>&
 basic_string<_CharT, _Traits, _Allocator>::append(
     _ForwardIterator __first, _ForwardIterator __last)
 {
@@ -2833,15 +2673,12 @@ basic_string<_CharT, _Traits, _Allocator>::append(const basic_string& __str, siz
 }
 
 template <class _CharT, class _Traits, class _Allocator>
-template <class _Tp>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-    __enable_if_t
-    <
-        __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value  && !__is_same_uncvref<_Tp, basic_string<_CharT, _Traits, _Allocator> >::value,
-        basic_string<_CharT, _Traits, _Allocator>&
-    >
-basic_string<_CharT, _Traits, _Allocator>::append(const _Tp & __t, size_type __pos, size_type __n)
-{
+template <class _Tp,
+          __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
+                            !__is_same_uncvref<_Tp, basic_string<_CharT, _Traits, _Allocator> >::value,
+                        int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string<_CharT, _Traits, _Allocator>&
+basic_string<_CharT, _Traits, _Allocator>::append(const _Tp& __t, size_type __pos, size_type __n) {
     __self_view __sv = __t;
     size_type __sz = __sv.size();
     if (__pos > __sz)
@@ -2933,13 +2770,8 @@ basic_string<_CharT, _Traits, _Allocator>::insert(size_type __pos, size_type __n
 }
 
 template <class _CharT, class _Traits, class _Allocator>
-template<class _InputIterator>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-   __is_exactly_cpp17_input_iterator<_InputIterator>::value,
-   typename basic_string<_CharT, _Traits, _Allocator>::iterator
->
+template<class _InputIterator, __enable_if_t<__is_exactly_cpp17_input_iterator<_InputIterator>::value, int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 typename basic_string<_CharT, _Traits, _Allocator>::iterator
 basic_string<_CharT, _Traits, _Allocator>::insert(const_iterator __pos, _InputIterator __first, _InputIterator __last)
 {
   _LIBCPP_DEBUG_ASSERT(__get_const_db()->__find_c_from_i(&__pos) == this,
@@ -2950,13 +2782,8 @@ basic_string<_CharT, _Traits, _Allocator>::insert(const_iterator __pos, _InputIt
 }
 
 template <class _CharT, class _Traits, class _Allocator>
-template<class _ForwardIterator>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-    __is_cpp17_forward_iterator<_ForwardIterator>::value,
-    typename basic_string<_CharT, _Traits, _Allocator>::iterator
->
+template<class _ForwardIterator, __enable_if_t<__is_cpp17_forward_iterator<_ForwardIterator>::value, int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 typename basic_string<_CharT, _Traits, _Allocator>::iterator
 basic_string<_CharT, _Traits, _Allocator>::insert(const_iterator __pos, _ForwardIterator __first, _ForwardIterator __last)
 {
     _LIBCPP_DEBUG_ASSERT(__get_const_db()->__find_c_from_i(&__pos) == this,
@@ -2991,16 +2818,12 @@ basic_string<_CharT, _Traits, _Allocator>::insert(size_type __pos1, const basic_
 }
 
 template <class _CharT, class _Traits, class _Allocator>
-template <class _Tp>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-    __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value  && !__is_same_uncvref<_Tp, basic_string<_CharT, _Traits, _Allocator> >::value,
-    basic_string<_CharT, _Traits, _Allocator>&
->
-basic_string<_CharT, _Traits, _Allocator>::insert(size_type __pos1, const _Tp& __t,
-                                                  size_type __pos2, size_type __n)
-{
+template <class _Tp,
+          __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
+                            !__is_same_uncvref<_Tp, basic_string<_CharT, _Traits, _Allocator> >::value,
+                        int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string<_CharT, _Traits, _Allocator>&
+basic_string<_CharT, _Traits, _Allocator>::insert(size_type __pos1, const _Tp& __t, size_type __pos2, size_type __n) {
     __self_view __sv = __t;
     size_type __str_sz = __sv.size();
     if (__pos2 > __str_sz)
@@ -3135,13 +2958,8 @@ basic_string<_CharT, _Traits, _Allocator>::replace(size_type __pos, size_type __
 }
 
 template <class _CharT, class _Traits, class _Allocator>
-template<class _InputIterator>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-    __is_cpp17_input_iterator<_InputIterator>::value,
-    basic_string<_CharT, _Traits, _Allocator>&
->
+template<class _InputIterator, __enable_if_t<__is_cpp17_input_iterator<_InputIterator>::value, int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string<_CharT, _Traits, _Allocator>&
 basic_string<_CharT, _Traits, _Allocator>::replace(const_iterator __i1, const_iterator __i2,
                                                    _InputIterator __j1, _InputIterator __j2)
 {
@@ -3162,16 +2980,13 @@ basic_string<_CharT, _Traits, _Allocator>::replace(size_type __pos1, size_type _
 }
 
 template <class _CharT, class _Traits, class _Allocator>
-template <class _Tp>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-    __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value && !__is_same_uncvref<_Tp, basic_string<_CharT, _Traits, _Allocator> >::value,
-    basic_string<_CharT, _Traits, _Allocator>&
->
-basic_string<_CharT, _Traits, _Allocator>::replace(size_type __pos1, size_type __n1, const _Tp& __t,
-                                                   size_type __pos2, size_type __n2)
-{
+template <class _Tp,
+          __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
+                            !__is_same_uncvref<_Tp, basic_string<_CharT, _Traits, _Allocator> >::value,
+                        int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 basic_string<_CharT, _Traits, _Allocator>&
+basic_string<_CharT, _Traits, _Allocator>::replace(
+    size_type __pos1, size_type __n1, const _Tp& __t, size_type __pos2, size_type __n2) {
     __self_view __sv = __t;
     size_type __str_sz = __sv.size();
     if (__pos2 > __str_sz)
@@ -3495,13 +3310,8 @@ basic_string<_CharT, _Traits, _Allocator>::find(const basic_string& __str,
 }
 
 template<class _CharT, class _Traits, class _Allocator>
-template <class _Tp>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-    __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-    typename basic_string<_CharT, _Traits, _Allocator>::size_type
->
+template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 typename basic_string<_CharT, _Traits, _Allocator>::size_type
 basic_string<_CharT, _Traits, _Allocator>::find(const _Tp &__t,
                                                 size_type __pos) const _NOEXCEPT
 {
@@ -3556,13 +3366,8 @@ basic_string<_CharT, _Traits, _Allocator>::rfind(const basic_string& __str,
 }
 
 template<class _CharT, class _Traits, class _Allocator>
-template <class _Tp>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-    __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-    typename basic_string<_CharT, _Traits, _Allocator>::size_type
->
+template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 typename basic_string<_CharT, _Traits, _Allocator>::size_type
 basic_string<_CharT, _Traits, _Allocator>::rfind(const _Tp& __t,
                                                 size_type __pos) const _NOEXCEPT
 {
@@ -3617,13 +3422,8 @@ basic_string<_CharT, _Traits, _Allocator>::find_first_of(const basic_string& __s
 }
 
 template<class _CharT, class _Traits, class _Allocator>
-template <class _Tp>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-    __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-    typename basic_string<_CharT, _Traits, _Allocator>::size_type
->
+template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 typename basic_string<_CharT, _Traits, _Allocator>::size_type
 basic_string<_CharT, _Traits, _Allocator>::find_first_of(const _Tp& __t,
                                                 size_type __pos) const _NOEXCEPT
 {
@@ -3677,13 +3477,8 @@ basic_string<_CharT, _Traits, _Allocator>::find_last_of(const basic_string& __st
 }
 
 template<class _CharT, class _Traits, class _Allocator>
-template <class _Tp>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-    __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-    typename basic_string<_CharT, _Traits, _Allocator>::size_type
->
+template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 typename basic_string<_CharT, _Traits, _Allocator>::size_type
 basic_string<_CharT, _Traits, _Allocator>::find_last_of(const _Tp& __t,
                                                 size_type __pos) const _NOEXCEPT
 {
@@ -3737,13 +3532,8 @@ basic_string<_CharT, _Traits, _Allocator>::find_first_not_of(const basic_string&
 }
 
 template<class _CharT, class _Traits, class _Allocator>
-template <class _Tp>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-    __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-    typename basic_string<_CharT, _Traits, _Allocator>::size_type
->
+template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 typename basic_string<_CharT, _Traits, _Allocator>::size_type
 basic_string<_CharT, _Traits, _Allocator>::find_first_not_of(const _Tp& __t,
                                                 size_type __pos) const _NOEXCEPT
 {
@@ -3798,13 +3588,8 @@ basic_string<_CharT, _Traits, _Allocator>::find_last_not_of(const basic_string&
 }
 
 template<class _CharT, class _Traits, class _Allocator>
-template <class _Tp>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-    __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-    typename basic_string<_CharT, _Traits, _Allocator>::size_type
->
+template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 typename basic_string<_CharT, _Traits, _Allocator>::size_type
 basic_string<_CharT, _Traits, _Allocator>::find_last_not_of(const _Tp& __t,
                                                 size_type __pos) const _NOEXCEPT
 {
@@ -3837,13 +3622,8 @@ basic_string<_CharT, _Traits, _Allocator>::find_last_not_of(value_type __c,
 // compare
 
 template <class _CharT, class _Traits, class _Allocator>
-template <class _Tp>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-    __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-    int
->
+template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 int
 basic_string<_CharT, _Traits, _Allocator>::compare(const _Tp& __t) const _NOEXCEPT
 {
     __self_view __sv = __t;
@@ -3893,13 +3673,8 @@ basic_string<_CharT, _Traits, _Allocator>::compare(size_type __pos1,
 }
 
 template <class _CharT, class _Traits, class _Allocator>
-template <class _Tp>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-    __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value,
-    int
->
+template <class _Tp, __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value, int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 int
 basic_string<_CharT, _Traits, _Allocator>::compare(size_type __pos1,
                                                    size_type __n1,
                                                    const _Tp& __t) const
@@ -3919,20 +3694,12 @@ basic_string<_CharT, _Traits, _Allocator>::compare(size_type __pos1,
 }
 
 template <class _CharT, class _Traits, class _Allocator>
-template <class _Tp>
-_LIBCPP_CONSTEXPR_SINCE_CXX20
-__enable_if_t
-<
-    __can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value
-    && !__is_same_uncvref<_Tp, basic_string<_CharT, _Traits, _Allocator> >::value,
-    int
->
-basic_string<_CharT, _Traits, _Allocator>::compare(size_type __pos1,
-                                                   size_type __n1,
-                                                   const _Tp& __t,
-                                                   size_type __pos2,
-                                                   size_type __n2) const
-{
+template <class _Tp,
+          __enable_if_t<__can_be_converted_to_string_view<_CharT, _Traits, _Tp>::value &&
+                            !__is_same_uncvref<_Tp, basic_string<_CharT, _Traits, _Allocator> >::value,
+                        int> >
+_LIBCPP_CONSTEXPR_SINCE_CXX20 int basic_string<_CharT, _Traits, _Allocator>::compare(
+    size_type __pos1, size_type __n1, const _Tp& __t, size_type __pos2, size_type __n2) const {
     __self_view __sv = __t;
     return __self_view(*this).substr(__pos1, __n1).compare(__sv.substr(__pos2, __n2));
 }
@@ -3982,7 +3749,7 @@ basic_string<_CharT, _Traits, _Allocator>::__invariants() const
         return false;
     if (data() == nullptr)
         return false;
-    if (data()[size()] != value_type())
+    if (!_Traits::eq(data()[size()], value_type()))
         return false;
     return true;
 }
@@ -4625,6 +4392,7 @@ _LIBCPP_POP_MACROS
 #  include <functional>
 #  include <iterator>
 #  include <new>
+#  include <type_traits>
 #  include <typeinfo>
 #  include <utility>
 #  include <vector>
diff --git a/libcxx/include/string_view b/libcxx/include/string_view
index 052e0981618b8..3f5e1c4d1878e 100644
--- a/libcxx/include/string_view
+++ b/libcxx/include/string_view
@@ -219,10 +219,17 @@ namespace std {
 #include <__ranges/enable_view.h>
 #include <__ranges/size.h>
 #include <__string/char_traits.h>
+#include <__type_traits/is_array.h>
+#include <__type_traits/is_convertible.h>
+#include <__type_traits/is_same.h>
+#include <__type_traits/is_standard_layout.h>
+#include <__type_traits/is_trivial.h>
+#include <__type_traits/remove_cvref.h>
+#include <__type_traits/remove_reference.h>
+#include <__type_traits/type_identity.h>
 #include <iosfwd>
 #include <limits>
 #include <stdexcept>
-#include <type_traits>
 #include <version>
 
 // standard-mandated includes
@@ -299,6 +306,7 @@ public:
         : __data_(__s), __size_(__len)
     {
 #if _LIBCPP_STD_VER >= 14
+    _LIBCPP_ASSERT(__len <= static_cast<size_type>(numeric_limits<difference_type>::max()), "string_view::string_view(_CharT *, size_t): length does not fit in difference_type");
     _LIBCPP_ASSERT(__len == 0 || __s != nullptr, "string_view::string_view(_CharT *, size_t): received nullptr");
 #endif
     }
@@ -1040,6 +1048,7 @@ _LIBCPP_POP_MACROS
 #  include <concepts>
 #  include <functional>
 #  include <iterator>
+#  include <type_traits>
 #endif
 
 #endif // _LIBCPP_STRING_VIEW
diff --git a/libcxx/include/system_error b/libcxx/include/system_error
index 3d2b6815fca6b..8142aac05032e 100644
--- a/libcxx/include/system_error
+++ b/libcxx/include/system_error
@@ -152,7 +152,6 @@ template <> struct hash<std::error_condition>;
 #include <__memory/addressof.h>
 #include <stdexcept>
 #include <string>
-#include <type_traits>
 #include <version>
 
 // standard-mandated includes
@@ -545,4 +544,8 @@ void __throw_system_error(int __ev, const char* __what_arg);
 
 _LIBCPP_END_NAMESPACE_STD
 
+#if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <type_traits>
+#endif
+
 #endif // _LIBCPP_SYSTEM_ERROR
diff --git a/libcxx/include/thread b/libcxx/include/thread
index 2d4229db2290f..13e722ca9476a 100644
--- a/libcxx/include/thread
+++ b/libcxx/include/thread
@@ -85,7 +85,9 @@ void sleep_for(const chrono::duration<Rep, Period>& rel_time);
 
 #include <__assert> // all public C++ headers provide the assertion handler
 #include <__config>
+#include <__exception/terminate.h>
 #include <__functional/hash.h>
+#include <__memory/addressof.h>
 #include <__memory/unique_ptr.h>
 #include <__mutex_base>
 #include <__thread/poll_with_backoff.h>
@@ -96,7 +98,6 @@ void sleep_for(const chrono::duration<Rep, Period>& rel_time);
 #include <iosfwd>
 #include <system_error>
 #include <tuple>
-#include <type_traits>
 #include <version>
 
 // standard-mandated includes
diff --git a/libcxx/include/typeinfo b/libcxx/include/typeinfo
index b3ba8c2c5600a..1553be1c6e107 100644
--- a/libcxx/include/typeinfo
+++ b/libcxx/include/typeinfo
@@ -59,12 +59,12 @@ public:
 #include <__assert> // all public C++ headers provide the assertion handler
 #include <__availability>
 #include <__config>
+#include <__exception/exception.h>
 #include <__type_traits/is_constant_evaluated.h>
+#include <__verbose_abort>
 #include <cstddef>
 #include <cstdint>
 #include <cstdlib>
-#include <exception>
-#include <type_traits>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
@@ -409,9 +409,14 @@ void __throw_bad_cast()
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
     throw bad_cast();
 #else
-    _VSTD::abort();
+    _LIBCPP_VERBOSE_ABORT("bad_cast was thrown in -fno-exceptions mode");
 #endif
 }
 _LIBCPP_END_NAMESPACE_STD
 
+#if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <exception>
+#  include <type_traits>
+#endif
+
 #endif // __LIBCPP_TYPEINFO
diff --git a/libcxx/include/unordered_map b/libcxx/include/unordered_map
index 251a581277cb4..99d55d3bdd4da 100644
--- a/libcxx/include/unordered_map
+++ b/libcxx/include/unordered_map
@@ -2648,6 +2648,7 @@ _LIBCPP_END_NAMESPACE_STD
 #  include <bit>
 #  include <concepts>
 #  include <iterator>
+#  include <type_traits>
 #endif
 
 #endif // _LIBCPP_UNORDERED_MAP
diff --git a/libcxx/include/unordered_set b/libcxx/include/unordered_set
index 1fdaf96b64b34..3d4a1f7466613 100644
--- a/libcxx/include/unordered_set
+++ b/libcxx/include/unordered_set
@@ -1819,6 +1819,7 @@ _LIBCPP_END_NAMESPACE_STD
 #  include <concepts>
 #  include <functional>
 #  include <iterator>
+#  include <type_traits>
 #endif
 
 #endif // _LIBCPP_UNORDERED_SET
diff --git a/libcxx/include/utility b/libcxx/include/utility
index 845b542132d30..d2f4218549196 100644
--- a/libcxx/include/utility
+++ b/libcxx/include/utility
@@ -117,22 +117,22 @@ struct common_type<pair<T1, T2>, pair<U1, U2>>;
 
 template<class T1, class T2> pair(T1, T2) -> pair<T1, T2>;
 
-template <class T1, class T2>
-bool operator==(const pair<T1,T2>&, const pair<T1,T2>&);                         // constexpr in C++14
-template <class T1, class T2>
-bool operator!=(const pair<T1,T2>&, const pair<T1,T2>&);                         // constexpr in C++14, removed in C++20
-template <class T1, class T2>
-bool operator< (const pair<T1,T2>&, const pair<T1,T2>&);                         // constexpr in C++14, removed in C++20
-template <class T1, class T2>
-bool operator> (const pair<T1,T2>&, const pair<T1,T2>&);                         // constexpr in C++14, removed in C++20
-template <class T1, class T2>
-bool operator>=(const pair<T1,T2>&, const pair<T1,T2>&);                         // constexpr in C++14, removed in C++20
-template <class T1, class T2>
-bool operator<=(const pair<T1,T2>&, const pair<T1,T2>&);                         // constexpr in C++14, removed in C++20
-template <class T1, class T2>
-  constexpr common_comparison_type_t<synth-three-way-result<T1>,
-                                     synth-three-way-result<T2>>
-    operator<=>(const pair<T1,T2>&, const pair<T1,T2>&);                         // C++20
+template <class T1, class T2, class U1, class U2>
+bool operator==(const pair<T1,T2>&, const pair<U1,U2>&);                         // constexpr in C++14
+template <class T1, class T2, class U1, class U2>
+bool operator!=(const pair<T1,T2>&, const pair<U1,U2>&);                         // constexpr in C++14, removed in C++20
+template <class T1, class T2, class U1, class U2>
+bool operator< (const pair<T1,T2>&, const pair<U1,U2>&);                         // constexpr in C++14, removed in C++20
+template <class T1, class T2, class U1, class U2>
+bool operator> (const pair<T1,T2>&, const pair<U1,U2>&);                         // constexpr in C++14, removed in C++20
+template <class T1, class T2, class U1, class U2>
+bool operator>=(const pair<T1,T2>&, const pair<U1,U2>&);                         // constexpr in C++14, removed in C++20
+template <class T1, class T2, class U1, class U2>
+bool operator<=(const pair<T1,T2>&, const pair<U1,U2>&);                         // constexpr in C++14, removed in C++20
+template <class T1, class T2, class U1, class U2>
+  constexpr common_comparison_type_t<synth-three-way-result<T1,U1>,
+                                     synth-three-way-result<T2,U2>>
+    operator<=>(const pair<T1,T2>&, const pair<U1,U2>&);                         // C++20
 
 template <class T1, class T2> pair<V1, V2> make_pair(T1&&, T2&&);                // constexpr in C++14
 template <class T1, class T2>
diff --git a/libcxx/include/valarray b/libcxx/include/valarray
index a52fc1088e467..f305bac8bc764 100644
--- a/libcxx/include/valarray
+++ b/libcxx/include/valarray
@@ -351,8 +351,10 @@ template <class T> unspecified2 end(const valarray<T>& v);
 #include <__assert> // all public C++ headers provide the assertion handler
 #include <__config>
 #include <__functional/operations.h>
+#include <__memory/addressof.h>
 #include <__memory/allocator.h>
 #include <__memory/uninitialized_algorithms.h>
+#include <__type_traits/decay.h>
 #include <__type_traits/remove_reference.h>
 #include <__utility/move.h>
 #include <__utility/swap.h>
diff --git a/libcxx/include/variant b/libcxx/include/variant
index c5234692b4a38..425feadd120b0 100644
--- a/libcxx/include/variant
+++ b/libcxx/include/variant
@@ -210,10 +210,12 @@ namespace std {
 #include <__compare/compare_three_way_result.h>
 #include <__compare/three_way_comparable.h>
 #include <__config>
+#include <__exception/exception.h>
 #include <__functional/hash.h>
 #include <__functional/invoke.h>
 #include <__functional/operations.h>
 #include <__functional/unary_function.h>
+#include <__memory/addressof.h>
 #include <__type_traits/add_const.h>
 #include <__type_traits/add_cv.h>
 #include <__type_traits/add_pointer.h>
@@ -231,12 +233,13 @@ namespace std {
 #include <__type_traits/remove_const.h>
 #include <__type_traits/type_identity.h>
 #include <__type_traits/void_t.h>
+#include <__utility/declval.h>
 #include <__utility/forward.h>
 #include <__utility/in_place.h>
 #include <__utility/move.h>
 #include <__utility/swap.h>
 #include <__variant/monostate.h>
-#include <exception>
+#include <__verbose_abort>
 #include <initializer_list>
 #include <limits>
 #include <new>
@@ -288,7 +291,7 @@ void __throw_bad_variant_access() {
 #ifndef _LIBCPP_HAS_NO_EXCEPTIONS
         throw bad_variant_access();
 #else
-        _VSTD::abort();
+    _LIBCPP_VERBOSE_ABORT("bad_variant_access was thrown in -fno-exceptions mode");
 #endif
 }
 
@@ -1831,6 +1834,7 @@ _LIBCPP_END_NAMESPACE_STD
 _LIBCPP_POP_MACROS
 
 #if !defined(_LIBCPP_REMOVE_TRANSITIVE_INCLUDES) && _LIBCPP_STD_VER <= 20
+#  include <exception>
 #  include <type_traits>
 #  include <typeinfo>
 #  include <utility>
diff --git a/libcxx/include/vector b/libcxx/include/vector
index d1e4b136ae2d9..980af582558be 100644
--- a/libcxx/include/vector
+++ b/libcxx/include/vector
@@ -299,6 +299,7 @@ template<class T, class charT> requires is-vector-bool-reference<T> // Since C++
 #include <__iterator/iterator_traits.h>
 #include <__iterator/reverse_iterator.h>
 #include <__iterator/wrap_iter.h>
+#include <__memory/addressof.h>
 #include <__memory/allocate_at_least.h>
 #include <__memory/pointer_traits.h>
 #include <__memory/swap_allocator.h>
@@ -307,7 +308,10 @@ template<class T, class charT> requires is-vector-bool-reference<T> // Since C++
 #include <__memory_resource/polymorphic_allocator.h>
 #include <__split_buffer>
 #include <__type_traits/is_allocator.h>
+#include <__type_traits/is_constructible.h>
+#include <__type_traits/is_nothrow_move_assignable.h>
 #include <__type_traits/noexcept_move_assign_container.h>
+#include <__type_traits/type_identity.h>
 #include <__utility/exception_guard.h>
 #include <__utility/forward.h>
 #include <__utility/move.h>
@@ -318,7 +322,6 @@ template<class T, class charT> requires is-vector-bool-reference<T> // Since C++
 #include <iosfwd> // for forward declaration of vector
 #include <limits>
 #include <stdexcept>
-#include <type_traits>
 #include <version>
 
 // standard-mandated includes
@@ -439,11 +442,11 @@ private:
       _LIBCPP_CONSTEXPR __destroy_vector(vector& __vec) : __vec_(__vec) {}
 
       _LIBCPP_CONSTEXPR_SINCE_CXX20 _LIBCPP_HIDE_FROM_ABI void operator()() {
+          __vec_.__annotate_delete();
           std::__debug_db_erase_c(std::addressof(__vec_));
 
           if (__vec_.__begin_ != nullptr) {
             __vec_.__clear();
-            __vec_.__annotate_delete();
             __alloc_traits::deallocate(__vec_.__alloc(), __vec_.__begin_, __vec_.capacity());
           }
       }
@@ -743,25 +746,8 @@ private:
                                          const void *__old_mid,
                                          const void *__new_mid) const
     {
-#  if _LIBCPP_CLANG_VER >= 1600
-      if (!__libcpp_is_constant_evaluated() && __beg)
-      // Implementation of __sanitizer_annotate_contiguous_container function changed with commit
-      // rGdd1b7b797a116eed588fd752fbe61d34deeb24e4 and supports:
-      //  - unaligned beginnings of buffers,
-      //  - shared first/last granule (if memory granule is shared with a different object
-      //    just after the end of unaligned end/before the unaligned beginning, memory of that object won't be poisoned).
-      //
-      // Therefore, check for standard allocator is not necessary.
-#  else
-      // TODO LLVM18: Remove the special-casing
-      //
-      // Vectors annotations rely on __sanitizer_annotate_contiguous_container function,
-      // its implementation from LLVM15 (and earlier) requires that
-      // beginning of a containers data buffer is aligned to shadow granularity and
-      // memory just after is not shared with another object.
-      // Default allocator satisfies that.
+
       if (!__libcpp_is_constant_evaluated() && __beg && is_same<allocator_type, __default_allocator_type>::value)
-#  endif
         __sanitizer_annotate_contiguous_container(__beg, __end, __old_mid, __new_mid);
     }
 #else
@@ -883,7 +869,6 @@ private:
     if (__alloc() != __c.__alloc())
     {
       __clear();
-      __annotate_delete();
       __alloc_traits::deallocate(__alloc(), this->__begin_, capacity());
       this->__begin_ = this->__end_ = __end_cap() = nullptr;
     }
@@ -972,7 +957,6 @@ vector<_Tp, _Allocator>::__vdeallocate() _NOEXCEPT
     if (this->__begin_ != nullptr)
     {
         clear();
-        __annotate_delete();
         __alloc_traits::deallocate(this->__alloc(), this->__begin_, capacity());
         this->__begin_ = this->__end_ = this->__end_cap() = nullptr;
     }
@@ -3378,6 +3362,7 @@ _LIBCPP_POP_MACROS
 #  include <algorithm>
 #  include <atomic>
 #  include <concepts>
+#  include <type_traits>
 #  include <typeinfo>
 #  include <utility>
 #endif
diff --git a/libcxx/lib/abi/CHANGELOG.TXT b/libcxx/lib/abi/CHANGELOG.TXT
index f382c7e60e233..8661ab4598bb2 100644
--- a/libcxx/lib/abi/CHANGELOG.TXT
+++ b/libcxx/lib/abi/CHANGELOG.TXT
@@ -187,7 +187,7 @@ Version 14.0
   In fact, any program that would have attempted to refer to that symbol would
   have previously failed to link.
 
-  x86_64-apple-apple-darwin
+  <arch>-apple-darwin
   -------------------------
   Symbol added: _ZNSt3__15ctypeIcE10table_sizeE
 
@@ -366,7 +366,7 @@ Version 12.0
   Those were previously defined in both libc++ and libc++abi, which was an
   ODR violation.
 
-  <arch>-apple-apple-darwin
+  <arch>-apple-darwin
   -------------------------
   The following symbols are now re-exported from libc++abi instead of exported
   by libc++ directly (this should not be an ABI break):
@@ -424,8 +424,8 @@ Version 12.0
   We re-export some symbols that were exported from libc++abi but not from
   libc++. Exporting new symbols is not an ABI break.
 
-  x86_64-apple-apple-darwin
-  -------------------------
+  <arch>-apple-darwin
+  -------------------
   Symbol added: ___cxa_allocate_dependent_exception
   Symbol added: ___cxa_free_dependent_exception
 
@@ -439,12 +439,8 @@ Version 10.0
   adds a few symbols to the dylib. This is backwards-compatible since we're
   just adding new symbols.
 
-  x86_64-unknown-linux-gnu
-  ------------------------
-  TODO
-
-  <arch>-apple-apple-darwin
-  -------------------------
+  All platforms
+  -------------
   Symbol added: __ZNSt3__120__libcpp_atomic_waitEPVKNS_17__cxx_atomic_implIxNS_22__cxx_atomic_base_implIxEEEEx
   Symbol added: __ZNSt3__120__libcpp_atomic_waitEPVKvx
   Symbol added: __ZNSt3__123__cxx_atomic_notify_allEPVKNS_17__cxx_atomic_implIxNS_22__cxx_atomic_base_implIxEEEE
@@ -469,12 +465,8 @@ Version 9.0
   against an older version of libc++abi that doesn't provide those symbols,
   however doing so is not supported anyway.
 
-  x86_64-unknown-linux-gnu
-  ------------------------
-  TODO
-
-  x86_64-apple-apple-darwin
-  -------------------------
+  <arch>-apple-darwin
+  -------------------
   Symbol added: ___cxa_throw_bad_array_new_length
   Symbol added: ___cxa_uncaught_exceptions
 
@@ -594,11 +586,10 @@ Version 9.0
   Symbol removed: _ZTSNSt3__124__libcpp_debug_exceptionE
   Symbol removed: _ZTVNSt3__124__libcpp_debug_exceptionE
 
-
-    x86_64-apple-apple-darwin
-    -------------------------
-    Symbol added: __ZNSt3__111__libcpp_db10__insert_cEPvPFPNS_8__c_nodeES1_S1_S3_E
-    Symbol removed: __ZNSt3__111__libcpp_db10__insert_cEPv
+  <arch>-apple-darwin
+  -------------------
+  Symbol added: __ZNSt3__111__libcpp_db10__insert_cEPvPFPNS_8__c_nodeES1_S1_S3_E
+  Symbol removed: __ZNSt3__111__libcpp_db10__insert_cEPv
 
 * r355367 - Fix -fsanitize=vptr badness in <__debug>
 
@@ -626,8 +617,8 @@ Version 8.0
   users couldn't be relying on the functionality because it is marked as being
   unavailable using Clang's availability attribute.
 
-  x86_64-apple-darwin16.0
-  -----------------------
+  <arch>-apple-darwin
+  -------------------
   Symbol removed: __ZNKSt16bad_array_length4whatEv
   Symbol removed: __ZNKSt16bad_array_length4whatEv
   Symbol removed: __ZNSt16bad_array_lengthC1Ev
@@ -706,8 +697,8 @@ Version 8.0
   Symbol removed: _ZNSt3__121__thread_specific_ptrINS_15__thread_structEE16__at_thread_exitEPv
   Symbol removed: _ZNSt3__119__start_std_streamsE
 
-  x86_64-apple-darwin16.0
-  -----------------------
+  <arch>-apple-darwin
+  -------------------
   Symbol removed: __ZNSt3__221__thread_specific_ptrINS_15__thread_structEE16__at_thread_exitEPv
   Symbol removed: __ZNSt3__119__start_std_streamsE
 
@@ -724,8 +715,8 @@ Version 7.0
   Symbol added: _ZNSt3__16__itoa8__u64toaEmPc
   Symbol added: _ZNSt3__16__itoa8__u32toaEjPc
 
-  x86_64-apple-darwin16.0
-  -----------------------
+  <arch>-apple-darwin
+  -------------------
   Symbol added: __ZNSt3__16__itoa8__u64toaEyPc
   Symbol added: __ZNSt3__16__itoa8__u32toaEjPc
 
@@ -753,8 +744,8 @@ Version 5.0
   ----------------
   No changes
 
-  x86_64-apple-darwin16.0
-  -----------------------
+  <arch>-apple-darwin
+  -------------------
   Symbol added: ___cxa_deleted_virtual
 
 * r296729 - Remove std::num_get template methods which should be inline
@@ -785,8 +776,8 @@ Version 5.0
   Symbol removed: _ZNKSt3__17num_getIwNS_19istreambuf_iteratorIwNS_11char_traitsIwEEEEE23__do_get_floating_pointIeEES4_S4_S4_RNS_8ios_baseERjRT_
   Symbol removed: _ZNKSt3__17num_getIwNS_19istreambuf_iteratorIwNS_11char_traitsIwEEEEE23__do_get_floating_pointIfEES4_S4_S4_RNS_8ios_baseERjRT_
 
-  x86_64-apple-darwin16.0
-  -----------------------
+  <arch>-apple-darwin
+  -------------------
   No changes
 
 -----------
@@ -836,8 +827,8 @@ Version 4.0
   Symbol removed: _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6appendIPcEENS_9enable_ifIXaasr21__is_forward_iteratorIT_EE5valuesr38__libcpp_string_gets_noexcept_iteratorIS9_EE5valueERS5_E4typeES9_S9_
   Symbol removed: _ZNSt3__112basic_stringIwNS_11char_traitsIwEENS_9allocatorIwEEE6appendIPwEENS_9enable_ifIXaasr21__is_forward_iteratorIT_EE5valuesr38__libcpp_string_gets_noexcept_iteratorIS9_EE5valueERS5_E4typeES9_S9_
 
-  x86_64-apple-darwin16.0
-  -----------------------
+  <arch>-apple-darwin
+  -------------------
   No changes
 
 * r285101 - Add -fvisibility-inlines-hidden when building libc++.
@@ -869,8 +860,8 @@ Version 4.0
   Symbol removed: _ZNSt3__120__codecvt_utf8_utf16IDsED0Ev
   Symbol removed: _ZNSt3__120__codecvt_utf8_utf16IwED0Ev
 
-  x86_64-apple-darwin16.0
-  -----------------------
+  <arch>-apple-darwin
+  -------------------
   No Changes - inline symbols are already hidden
 
 * r284206 - Implement C++17 aligned allocation in <new>
@@ -909,8 +900,7 @@ Version 4.0
   Symbol added: _ZTSSt12bad_any_cast
   Symbol added: _ZTVSt12bad_any_cast
 
-* r295398 - Remove basic_string::insert and basic_string::replace template methods
-    which should be inline.
+* r295398 - Remove basic_string::insert and basic_string::replace template methods which should be inline.
 
   These functions should never have had visible definitions in the dylib but
   since they were previously not specified with 'inline' they accidentally
diff --git a/libcxx/src/chrono.cpp b/libcxx/src/chrono.cpp
index 0af89d6a526c4..8c85d96e5444b 100644
--- a/libcxx/src/chrono.cpp
+++ b/libcxx/src/chrono.cpp
@@ -167,59 +167,6 @@ system_clock::from_time_t(time_t t) noexcept
 
 #if defined(__APPLE__)
 
-// TODO(ldionne):
-// This old implementation of steady_clock is retained until Chrome drops supports
-// for macOS < 10.12. The issue is that they link libc++ statically into their
-// application, which means that libc++ must support being built for such deployment
-// targets. See https://llvm.org/D74489 for details.
-#if (defined(__ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__) && __ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__ < 101200) || \
-    (defined(__ENVIRONMENT_IPHONE_OS_VERSION_MIN_REQUIRED__) && __ENVIRONMENT_IPHONE_OS_VERSION_MIN_REQUIRED__ < 100000) || \
-    (defined(__ENVIRONMENT_TV_OS_VERSION_MIN_REQUIRED__) && __ENVIRONMENT_TV_OS_VERSION_MIN_REQUIRED__ < 100000) || \
-    (defined(__ENVIRONMENT_WATCH_OS_VERSION_MIN_REQUIRED__) && __ENVIRONMENT_WATCH_OS_VERSION_MIN_REQUIRED__ < 30000)
-# define _LIBCPP_USE_OLD_MACH_ABSOLUTE_TIME
-#endif
-
-#if defined(_LIBCPP_USE_OLD_MACH_ABSOLUTE_TIME)
-
-//   mach_absolute_time() * MachInfo.numer / MachInfo.denom is the number of
-//   nanoseconds since the computer booted up.  MachInfo.numer and MachInfo.denom
-//   are run time constants supplied by the OS.  This clock has no relationship
-//   to the Gregorian calendar.  It's main use is as a high resolution timer.
-
-// MachInfo.numer / MachInfo.denom is often 1 on the latest equipment.  Specialize
-//   for that case as an optimization.
-
-static steady_clock::rep steady_simplified() {
-    return static_cast<steady_clock::rep>(mach_absolute_time());
-}
-static double compute_steady_factor() {
-    mach_timebase_info_data_t MachInfo;
-    mach_timebase_info(&MachInfo);
-    return static_cast<double>(MachInfo.numer) / MachInfo.denom;
-}
-
-static steady_clock::rep steady_full() {
-    static const double factor = compute_steady_factor();
-    return static_cast<steady_clock::rep>(mach_absolute_time() * factor);
-}
-
-typedef steady_clock::rep (*FP)();
-
-static FP init_steady_clock() {
-    mach_timebase_info_data_t MachInfo;
-    mach_timebase_info(&MachInfo);
-    if (MachInfo.numer == MachInfo.denom)
-        return &steady_simplified;
-    return &steady_full;
-}
-
-static steady_clock::time_point __libcpp_steady_clock_now() {
-    static FP fp = init_steady_clock();
-    return steady_clock::time_point(steady_clock::duration(fp()));
-}
-
-#else // vvvvv default behavior for Apple platforms  vvvvv
-
 // On Apple platforms, only CLOCK_UPTIME_RAW, CLOCK_MONOTONIC_RAW or
 // mach_absolute_time are able to time functions in the nanosecond range.
 // Furthermore, only CLOCK_MONOTONIC_RAW is truly monotonic, because it
@@ -232,8 +179,6 @@ static steady_clock::time_point __libcpp_steady_clock_now() {
     return steady_clock::time_point(seconds(tp.tv_sec) + nanoseconds(tp.tv_nsec));
 }
 
-#endif
-
 #elif defined(_LIBCPP_WIN32API)
 
 // https://msdn.microsoft.com/en-us/library/windows/desktop/ms644905(v=vs.85).aspx says:
diff --git a/libcxx/src/filesystem/filesystem_common.h b/libcxx/src/filesystem/filesystem_common.h
index fb6f88783cd03..0d524262b1825 100644
--- a/libcxx/src/filesystem/filesystem_common.h
+++ b/libcxx/src/filesystem/filesystem_common.h
@@ -34,19 +34,16 @@
 # include <unistd.h>
 #endif // defined(_LIBCPP_WIN32API)
 
-#include "../include/apple_availability.h"
-
-#if !defined(__APPLE__)
-// We can use the presence of UTIME_OMIT to detect platforms that provide
-// utimensat.
+// We can use the presence of UTIME_OMIT to detect platforms that provide utimensat.
 #if defined(UTIME_OMIT)
-#define _LIBCPP_USE_UTIMENSAT
-#endif
+# define _LIBCPP_USE_UTIMENSAT
 #endif
 
+// TODO: Check whether these functions actually need internal linkage, or if they can be made normal header functions
 _LIBCPP_DIAGNOSTIC_PUSH
 _LIBCPP_GCC_DIAGNOSTIC_IGNORED("-Wunused-function")
 _LIBCPP_CLANG_DIAGNOSTIC_IGNORED("-Wunused-function")
+_LIBCPP_CLANG_DIAGNOSTIC_IGNORED("-Wunused-template")
 
 #if defined(_LIBCPP_WIN32API)
 #  define PATHSTR(x) (L##x)
diff --git a/libcxx/src/include/apple_availability.h b/libcxx/src/include/apple_availability.h
index 75bf79658d1e6..c8d158e990e27 100644
--- a/libcxx/src/include/apple_availability.h
+++ b/libcxx/src/include/apple_availability.h
@@ -11,24 +11,6 @@
 
 #if defined(__APPLE__)
 
-#if defined(__ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__)
-#if __ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__ >= 101300
-#define _LIBCPP_USE_UTIMENSAT
-#endif
-#elif defined(__ENVIRONMENT_IPHONE_OS_VERSION_MIN_REQUIRED__)
-#if __ENVIRONMENT_IPHONE_OS_VERSION_MIN_REQUIRED__ >= 110000
-#define _LIBCPP_USE_UTIMENSAT
-#endif
-#elif defined(__ENVIRONMENT_TV_OS_VERSION_MIN_REQUIRED__)
-#if __ENVIRONMENT_TV_OS_VERSION_MIN_REQUIRED__ >= 110000
-#define _LIBCPP_USE_UTIMENSAT
-#endif
-#elif defined(__ENVIRONMENT_WATCH_OS_VERSION_MIN_REQUIRED__)
-#if __ENVIRONMENT_WATCH_OS_VERSION_MIN_REQUIRED__ >= 40000
-#define _LIBCPP_USE_UTIMENSAT
-#endif
-#endif // __ENVIRONMENT_.*_VERSION_MIN_REQUIRED__
-
 #if defined(__ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__)
 #if __ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__ >= 101500
 #define _LIBCPP_USE_ULOCK
diff --git a/libcxx/test/configs/apple-libc++-backdeployment.cfg.in b/libcxx/test/configs/apple-libc++-backdeployment.cfg.in
index 512631981c685..b471c02709c06 100644
--- a/libcxx/test/configs/apple-libc++-backdeployment.cfg.in
+++ b/libcxx/test/configs/apple-libc++-backdeployment.cfg.in
@@ -54,6 +54,8 @@ config.substitutions.append(('%{exec}',
     '%{executor} --execdir %T --env DYLD_LIBRARY_PATH="%{cxx-runtime-root}:%{abi-runtime-root}:%{unwind-runtime-root}" -- '
 ))
 
+config.stdlib = 'apple-libc++'
+
 libcxx.test.config.configure(
     libcxx.test.params.DEFAULT_PARAMETERS + BACKDEPLOYMENT_PARAMETERS,
     libcxx.test.features.DEFAULT_FEATURES,
diff --git a/libcxx/test/configs/apple-libc++-shared.cfg.in b/libcxx/test/configs/apple-libc++-shared.cfg.in
index dba32a37085ee..af1926e3859b9 100644
--- a/libcxx/test/configs/apple-libc++-shared.cfg.in
+++ b/libcxx/test/configs/apple-libc++-shared.cfg.in
@@ -22,6 +22,8 @@ config.substitutions.append(('%{exec}',
     '%{executor} --execdir %T --env DYLD_LIBRARY_PATH=%{lib} -- '
 ))
 
+config.stdlib = 'apple-libc++'
+
 import os, site
 site.addsitedir(os.path.join('@LIBCXX_SOURCE_DIR@', 'utils'))
 import libcxx.test.params, libcxx.test.config
diff --git a/libcxx/test/libcxx/algorithms/alg.modifying.operations/alg.random.shuffle/random_shuffle.depr_in_cxx14.verify.cpp b/libcxx/test/libcxx/algorithms/alg.modifying.operations/alg.random.shuffle/random_shuffle.depr_in_cxx14.verify.cpp
index fc58955c1c2cb..057f126a93cfe 100644
--- a/libcxx/test/libcxx/algorithms/alg.modifying.operations/alg.random.shuffle/random_shuffle.depr_in_cxx14.verify.cpp
+++ b/libcxx/test/libcxx/algorithms/alg.modifying.operations/alg.random.shuffle/random_shuffle.depr_in_cxx14.verify.cpp
@@ -35,12 +35,9 @@ struct gen
 };
 
 
-int main(int, char**)
-{
+void f() {
     int v[1] = {1};
     std::random_shuffle(&v[0], &v[1]); // expected-warning {{'random_shuffle<int *>' is deprecated}}
     gen r;
     std::random_shuffle(&v[0], &v[1], r); // expected-warning {{'random_shuffle<int *, gen &>' is deprecated}}
-
-  return 0;
 }
diff --git a/libcxx/test/libcxx/atomics/atomics.align/align.pass.cpp b/libcxx/test/libcxx/atomics/atomics.align/align.pass.cpp
index e99af02e4f8e2..495d02fbe5c8d 100644
--- a/libcxx/test/libcxx/atomics/atomics.align/align.pass.cpp
+++ b/libcxx/test/libcxx/atomics/atomics.align/align.pass.cpp
@@ -37,6 +37,7 @@
 
 #include <atomic>
 #include <cassert>
+#include <cstddef>
 
 template <typename T>
 struct atomic_test : public std::__atomic_base<T> {
diff --git a/libcxx/test/libcxx/atomics/bit-int.verify.cpp b/libcxx/test/libcxx/atomics/bit-int.verify.cpp
index 612688d43a908..209ec71d9e9ce 100644
--- a/libcxx/test/libcxx/atomics/bit-int.verify.cpp
+++ b/libcxx/test/libcxx/atomics/bit-int.verify.cpp
@@ -18,10 +18,7 @@
 
 #include <atomic>
 
-int main(int, char**)
-{
+void f() {
   // expected-error@*:*1 {{_Atomic cannot be applied to integer type '_BitInt(32)'}}
   std::atomic<_BitInt(32)> x(42);
-
-  return 0;
 }
diff --git a/libcxx/test/libcxx/atomics/diagnose_invalid_memory_order.verify.cpp b/libcxx/test/libcxx/atomics/diagnose_invalid_memory_order.verify.cpp
index 3eb4cab840da0..a6c1160c7022f 100644
--- a/libcxx/test/libcxx/atomics/diagnose_invalid_memory_order.verify.cpp
+++ b/libcxx/test/libcxx/atomics/diagnose_invalid_memory_order.verify.cpp
@@ -18,7 +18,7 @@
 
 #include <atomic>
 
-int main(int, char**) {
+void f() {
     std::atomic<int> x(42);
     volatile std::atomic<int>& vx = x;
     int val1 = 1; ((void)val1);
@@ -123,6 +123,4 @@ int main(int, char**) {
         std::atomic_compare_exchange_strong_explicit(&x, &val1, val2, std::memory_order_seq_cst, std::memory_order_acquire);
         std::atomic_compare_exchange_strong_explicit(&x, &val1, val2, std::memory_order_seq_cst, std::memory_order_seq_cst);
     }
-
-  return 0;
 }
diff --git a/libcxx/test/libcxx/containers/associative/map/compare.three_way.pass.cpp b/libcxx/test/libcxx/containers/associative/map/compare.three_way.pass.cpp
new file mode 100644
index 0000000000000..0e1c57f123e58
--- /dev/null
+++ b/libcxx/test/libcxx/containers/associative/map/compare.three_way.pass.cpp
@@ -0,0 +1,28 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+// UNSUPPORTED: c++03, c++11, c++14, c++17
+
+// <map>
+
+// class map
+
+// template<class Key, class T, class Compare, class Allocator>
+//   synth-three-way-result<pair<const Key, T>>
+//     operator<=>(const map<Key, T, Compare, Allocator>& x,
+//                 const map<Key, T, Compare, Allocator>& y);
+
+#include <cassert>
+#include <map>
+
+#include "test_container_comparisons.h"
+
+int main(int, char**) {
+  assert(test_ordered_map_container_spaceship<std::map>());
+  // `std::map` is not constexpr, so no `static_assert` test here.
+  return 0;
+}
diff --git a/libcxx/test/libcxx/containers/associative/multimap/compare.three_way.pass.cpp b/libcxx/test/libcxx/containers/associative/multimap/compare.three_way.pass.cpp
new file mode 100644
index 0000000000000..c7febc362cd63
--- /dev/null
+++ b/libcxx/test/libcxx/containers/associative/multimap/compare.three_way.pass.cpp
@@ -0,0 +1,28 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+// UNSUPPORTED: c++03, c++11, c++14, c++17
+
+// <map>
+
+// class multimap
+
+// template<class Key, class T, class Compare, class Allocator>
+//   synth-three-way-result<pair<const Key, T>>
+//     operator<=>(const multimap<Key, T, Compare, Allocator>& x,
+//                 const multimap<Key, T, Compare, Allocator>& y);
+
+#include <cassert>
+#include <map>
+
+#include "test_container_comparisons.h"
+
+int main(int, char**) {
+  assert(test_ordered_map_container_spaceship<std::multimap>());
+  // `std::multimap` is not constexpr, so no `static_assert` test here.
+  return 0;
+}
diff --git a/libcxx/test/libcxx/containers/associative/non_const_comparator.incomplete.verify.cpp b/libcxx/test/libcxx/containers/associative/non_const_comparator.incomplete.verify.cpp
index eb4d4cfc20436..3f7086f7e37f3 100644
--- a/libcxx/test/libcxx/containers/associative/non_const_comparator.incomplete.verify.cpp
+++ b/libcxx/test/libcxx/containers/associative/non_const_comparator.incomplete.verify.cpp
@@ -17,9 +17,6 @@
 #include <map>
 #include <functional>
 
-#include "test_macros.h"
-
-
 template <template <typename ...> class Container>
 void test_set() {
   struct KeyBase { };
@@ -48,12 +45,10 @@ void test_map() {
   C c;
 }
 
-int main(int, char**) {
+void f() {
   // expected-no-diagnostics
   test_set<std::set>();
   test_set<std::multiset>();
   test_map<std::map>();
   test_map<std::multimap>();
-
-  return 0;
 }
diff --git a/libcxx/test/libcxx/containers/associative/non_const_comparator.verify.cpp b/libcxx/test/libcxx/containers/associative/non_const_comparator.verify.cpp
index 61cdc2f10bd99..3bb59f1920f63 100644
--- a/libcxx/test/libcxx/containers/associative/non_const_comparator.verify.cpp
+++ b/libcxx/test/libcxx/containers/associative/non_const_comparator.verify.cpp
@@ -23,7 +23,7 @@ struct BadCompare {
   }
 };
 
-int main(int, char**) {
+void f() {
   static_assert(!std::__invokable<BadCompare const&, int const&, int const&>::value, "");
   static_assert(std::__invokable<BadCompare&, int const&, int const&>::value, "");
 
@@ -45,6 +45,4 @@ int main(int, char**) {
     using C = std::multimap<long, int, BadCompare>;
     C s;
   }
-
-  return 0;
 }
diff --git a/libcxx/test/libcxx/containers/sequences/vector.bool/trivial_for_purposes_of_call.pass.cpp b/libcxx/test/libcxx/containers/sequences/vector.bool/trivial_for_purposes_of_call.pass.cpp
index d9d698fc7f708..d29b0f1a95abd 100644
--- a/libcxx/test/libcxx/containers/sequences/vector.bool/trivial_for_purposes_of_call.pass.cpp
+++ b/libcxx/test/libcxx/containers/sequences/vector.bool/trivial_for_purposes_of_call.pass.cpp
@@ -26,10 +26,9 @@
 
 // UNSUPPORTED: c++03
 
-#include <vector>
 #include <cassert>
-
-#include "test_macros.h"
+#include <type_traits>
+#include <vector>
 
 template <class T>
 using IsTrivialForCall = std::integral_constant<bool,
diff --git a/libcxx/test/libcxx/containers/sequences/vector/asan.pass.cpp b/libcxx/test/libcxx/containers/sequences/vector/asan.pass.cpp
index 4032e86b70d4b..40275f58cb32c 100644
--- a/libcxx/test/libcxx/containers/sequences/vector/asan.pass.cpp
+++ b/libcxx/test/libcxx/containers/sequences/vector/asan.pass.cpp
@@ -29,65 +29,40 @@ void do_exit() {
 
 int main(int, char**)
 {
-#if TEST_STD_VER >= 11 && TEST_CLANG_VER < 1600
-  // TODO LLVM18: Remove the special-test
-  {
-    typedef int T;
-    typedef std::vector<T, min_allocator<T>> C;
-    const T t[] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
-    C c(std::begin(t), std::end(t));
-    c.reserve(2 * c.size());
-    volatile T foo = c[c.size()]; // bad, but not caught by ASAN
-    ((void)foo);
-  }
+#if TEST_STD_VER >= 11
+    {
+        typedef int T;
+        typedef std::vector<T, min_allocator<T>> C;
+        const T t[] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
+        C c(std::begin(t), std::end(t));
+        c.reserve(2*c.size());
+        volatile T foo = c[c.size()];    // bad, but not caught by ASAN
+        ((void)foo);
+    }
 #endif
 
-#if TEST_STD_VER >= 11 && TEST_CLANG_VER >= 1600
-  // TODO LLVM18: Remove the special-casing
-  {
-    typedef int T;
-    typedef cpp17_input_iterator<T*> MyInputIter;
-    std::vector<T, min_allocator<T>> v;
-    v.reserve(1);
-    int i[] = {42};
-    v.insert(v.begin(), MyInputIter(i), MyInputIter(i + 1));
-    assert(v[0] == 42);
-    assert(is_contiguous_container_asan_correct(v));
-  }
-  {
-    typedef char T;
-    typedef cpp17_input_iterator<T*> MyInputIter;
-    std::vector<T, unaligned_allocator<T>> v;
-    v.reserve(1);
-    char i[] = {'a', 'b'};
-    v.insert(v.begin(), MyInputIter(i), MyInputIter(i + 2));
-    assert(v[0] == 'a');
-    assert(v[1] == 'b');
-    assert(is_contiguous_container_asan_correct(v));
-  }
-#endif
-  {
-    typedef cpp17_input_iterator<int*> MyInputIter;
-    // Sould not trigger ASan.
-    std::vector<int> v;
-    v.reserve(1);
-    int i[] = {42};
-    v.insert(v.begin(), MyInputIter(i), MyInputIter(i + 1));
-    assert(v[0] == 42);
-    assert(is_contiguous_container_asan_correct(v));
-  }
+    {
+        typedef cpp17_input_iterator<int*> MyInputIter;
+        // Sould not trigger ASan.
+        std::vector<int> v;
+        v.reserve(1);
+        int i[] = {42};
+        v.insert(v.begin(), MyInputIter(i), MyInputIter(i + 1));
+        assert(v[0] == 42);
+        assert(is_contiguous_container_asan_correct(v));
+    }
 
-  __sanitizer_set_death_callback(do_exit);
-  {
-    typedef int T;
-    typedef std::vector<T> C;
-    const T t[] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
-    C c(std::begin(t), std::end(t));
-    c.reserve(2 * c.size());
-    assert(is_contiguous_container_asan_correct(c));
-    assert(!__sanitizer_verify_contiguous_container(c.data(), c.data() + 1, c.data() + c.capacity()));
-    volatile T foo = c[c.size()]; // should trigger ASAN. Use volatile to prevent being optimized away.
-    assert(false);                // if we got here, ASAN didn't trigger
-    ((void)foo);
-  }
+    __sanitizer_set_death_callback(do_exit);
+    {
+        typedef int T;
+        typedef std::vector<T> C;
+        const T t[] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
+        C c(std::begin(t), std::end(t));
+        c.reserve(2*c.size());
+        assert(is_contiguous_container_asan_correct(c));
+        assert(!__sanitizer_verify_contiguous_container( c.data(), c.data() + 1, c.data() + c.capacity()));
+        volatile T foo = c[c.size()]; // should trigger ASAN. Use volatile to prevent being optimized away.
+        assert(false);          // if we got here, ASAN didn't trigger
+        ((void)foo);
+    }
 }
diff --git a/libcxx/test/libcxx/containers/unord/next_pow2.pass.cpp b/libcxx/test/libcxx/containers/unord/next_pow2.pass.cpp
index 5a29248741a2e..5146f41dbe49a 100644
--- a/libcxx/test/libcxx/containers/unord/next_pow2.pass.cpp
+++ b/libcxx/test/libcxx/containers/unord/next_pow2.pass.cpp
@@ -68,7 +68,7 @@ fuzz_unordered_map_reserve(unsigned num_inserts,
                            unsigned num_reserve1,
                            unsigned num_reserve2)
 {
-    std::unordered_map<uint64_t, unsigned long> m;
+    std::unordered_map<std::uint64_t, unsigned long> m;
     m.reserve(num_reserve1);
     for (unsigned I = 0; I < num_inserts; ++I) m[I] = 0;
     m.reserve(num_reserve2);
diff --git a/libcxx/test/libcxx/containers/unord/non_const_comparator.incomplete.verify.cpp b/libcxx/test/libcxx/containers/unord/non_const_comparator.incomplete.verify.cpp
index 603b4fcb50ffb..9fd4f698b5f96 100644
--- a/libcxx/test/libcxx/containers/unord/non_const_comparator.incomplete.verify.cpp
+++ b/libcxx/test/libcxx/containers/unord/non_const_comparator.incomplete.verify.cpp
@@ -48,12 +48,10 @@ void test_map() {
   C c;
 }
 
-int main(int, char**) {
+void f() {
   // expected-no-diagnostics
   test_set<std::unordered_set>();
   test_set<std::unordered_multiset>();
   test_map<std::unordered_map>();
   test_map<std::unordered_multimap>();
-
-  return 0;
 }
diff --git a/libcxx/test/libcxx/containers/unord/non_const_comparator.verify.cpp b/libcxx/test/libcxx/containers/unord/non_const_comparator.verify.cpp
index 2a389c0774c17..0452d080a7124 100644
--- a/libcxx/test/libcxx/containers/unord/non_const_comparator.verify.cpp
+++ b/libcxx/test/libcxx/containers/unord/non_const_comparator.verify.cpp
@@ -29,7 +29,7 @@ struct BadEqual {
   }
 };
 
-int main(int, char**) {
+void f() {
   static_assert(!std::__invokable<BadEqual const&, int const&, int const&>::value, "");
   static_assert(std::__invokable<BadEqual&, int const&, int const&>::value, "");
 
@@ -54,6 +54,4 @@ int main(int, char**) {
     using C = std::unordered_multimap<long, int, BadHash, BadEqual>;
     C s;
   }
-
-  return 0;
 }
diff --git a/libcxx/test/libcxx/depr/depr.auto.ptr/auto.ptr/auto_ptr.depr_in_cxx11.verify.cpp b/libcxx/test/libcxx/depr/depr.auto.ptr/auto.ptr/auto_ptr.depr_in_cxx11.verify.cpp
index 9fc83062c69a6..3fe7ced3f1d89 100644
--- a/libcxx/test/libcxx/depr/depr.auto.ptr/auto.ptr/auto_ptr.depr_in_cxx11.verify.cpp
+++ b/libcxx/test/libcxx/depr/depr.auto.ptr/auto.ptr/auto_ptr.depr_in_cxx11.verify.cpp
@@ -25,11 +25,6 @@
 #include <memory>
 #include "test_macros.h"
 
-int main(int, char**)
-{
-    typedef std::auto_ptr<int> AP; // expected-warning {{'auto_ptr<int>' is deprecated}}
-    typedef std::auto_ptr<void> APV; // expected-warning {{'auto_ptr<void>' is deprecated}}
-    typedef std::auto_ptr_ref<int> APR; // expected-warning {{'auto_ptr_ref<int>' is deprecated}}
-
-  return 0;
-}
+typedef std::auto_ptr<int> AP; // expected-warning {{'auto_ptr<int>' is deprecated}}
+typedef std::auto_ptr<void> APV; // expected-warning {{'auto_ptr<void>' is deprecated}}
+typedef std::auto_ptr_ref<int> APR; // expected-warning {{'auto_ptr_ref<int>' is deprecated}}
diff --git a/libcxx/test/libcxx/depr/depr.default.allocator/allocator.members/address.depr_in_cxx17.verify.cpp b/libcxx/test/libcxx/depr/depr.default.allocator/allocator.members/address.depr_in_cxx17.verify.cpp
index 4ccc3c099acd4..5b21cc607711e 100644
--- a/libcxx/test/libcxx/depr/depr.default.allocator/allocator.members/address.depr_in_cxx17.verify.cpp
+++ b/libcxx/test/libcxx/depr/depr.default.allocator/allocator.members/address.depr_in_cxx17.verify.cpp
@@ -19,15 +19,10 @@
 // ADDITIONAL_COMPILE_FLAGS: -D_LIBCPP_ENABLE_CXX20_REMOVED_ALLOCATOR_MEMBERS
 
 #include <memory>
-#include "test_macros.h"
 
-int main(int, char**)
-{
+void f() {
   int x = 0;
   std::allocator<int> a;
 
-  int* p = a.address(x); // expected-warning {{'address' is deprecated}}
-  (void)p;
-
-  return 0;
+  (void)a.address(x); // expected-warning {{'address' is deprecated}}
 }
diff --git a/libcxx/test/libcxx/depr/depr.default.allocator/allocator.members/allocate.cxx2a.verify.cpp b/libcxx/test/libcxx/depr/depr.default.allocator/allocator.members/allocate.cxx2a.verify.cpp
index c122be5996d0a..2289cd6cd4044 100644
--- a/libcxx/test/libcxx/depr/depr.default.allocator/allocator.members/allocate.cxx2a.verify.cpp
+++ b/libcxx/test/libcxx/depr/depr.default.allocator/allocator.members/allocate.cxx2a.verify.cpp
@@ -21,14 +21,8 @@
 // ADDITIONAL_COMPILE_FLAGS: -D_LIBCPP_DISABLE_DEPRECATION_WARNINGS
 
 #include <memory>
-#include <cassert>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::allocator<int> a;
     a.allocate(3, nullptr); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/libcxx/depr/depr.default.allocator/allocator.members/allocate.depr_in_cxx17.verify.cpp b/libcxx/test/libcxx/depr/depr.default.allocator/allocator.members/allocate.depr_in_cxx17.verify.cpp
index efb57b6e496ef..79fd61d24c4ce 100644
--- a/libcxx/test/libcxx/depr/depr.default.allocator/allocator.members/allocate.depr_in_cxx17.verify.cpp
+++ b/libcxx/test/libcxx/depr/depr.default.allocator/allocator.members/allocate.depr_in_cxx17.verify.cpp
@@ -18,12 +18,10 @@
 // ADDITIONAL_COMPILE_FLAGS: -D_LIBCPP_ENABLE_CXX20_REMOVED_ALLOCATOR_MEMBERS
 
 #include <memory>
+
 #include "test_macros.h"
 
-int main(int, char**)
-{
+void f() {
     std::allocator<int> a;
     TEST_IGNORE_NODISCARD a.allocate(3, nullptr); // expected-warning {{'allocate' is deprecated}}
-
-    return 0;
 }
diff --git a/libcxx/test/libcxx/depr/depr.function.objects/adaptors.depr_in_cxx11.verify.cpp b/libcxx/test/libcxx/depr/depr.function.objects/adaptors.depr_in_cxx11.verify.cpp
index 2b3a05eaad749..87d81d00ebaec 100644
--- a/libcxx/test/libcxx/depr/depr.function.objects/adaptors.depr_in_cxx11.verify.cpp
+++ b/libcxx/test/libcxx/depr/depr.function.objects/adaptors.depr_in_cxx11.verify.cpp
@@ -14,7 +14,6 @@
 
 #include <functional>
 #include <cassert>
-#include "test_macros.h"
 
 int identity(int v) { return v; }
 int sum(int a, int b) { return a + b; }
@@ -26,8 +25,7 @@ struct Foo {
     int identity(int v) { return v; }
 };
 
-int main(int, char**)
-{
+void f() {
     typedef std::pointer_to_unary_function<int, int> PUF; // expected-warning {{'pointer_to_unary_function<int, int>' is deprecated}}
     typedef std::pointer_to_binary_function<int, int, int> PBF; // expected-warning {{'pointer_to_binary_function<int, int, int>' is deprecated}}
     std::ptr_fun<int, int>(identity); // expected-warning {{'ptr_fun<int, int>' is deprecated}}
@@ -50,6 +48,4 @@ int main(int, char**)
     std::mem_fun_ref<int, Foo, int>(&Foo::identity); // expected-warning {{'mem_fun_ref<int, Foo, int>' is deprecated}}
     std::mem_fun_ref<int, Foo>(&Foo::const_zero); // expected-warning {{'mem_fun_ref<int, Foo>' is deprecated}}
     std::mem_fun_ref<int, Foo, int>(&Foo::const_identity); // expected-warning {{'mem_fun_ref<int, Foo, int>' is deprecated}}
-
-  return 0;
 }
diff --git a/libcxx/test/libcxx/diagnostics/nodiscard_aftercxx17.verify.cpp b/libcxx/test/libcxx/diagnostics/nodiscard_aftercxx17.verify.cpp
index 9e3796cb00787..242d935b7f00d 100644
--- a/libcxx/test/libcxx/diagnostics/nodiscard_aftercxx17.verify.cpp
+++ b/libcxx/test/libcxx/diagnostics/nodiscard_aftercxx17.verify.cpp
@@ -16,9 +16,6 @@
 
 _LIBCPP_NODISCARD_AFTER_CXX17 int foo() { return 6; }
 
-int main(int, char**)
-{
+void f() {
     foo(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/libcxx/diagnostics/nodiscard_extensions.compile.pass.cpp b/libcxx/test/libcxx/diagnostics/nodiscard_extensions.compile.pass.cpp
index e0d457bd844b8..0cc86e2a77cfd 100644
--- a/libcxx/test/libcxx/diagnostics/nodiscard_extensions.compile.pass.cpp
+++ b/libcxx/test/libcxx/diagnostics/nodiscard_extensions.compile.pass.cpp
@@ -183,12 +183,10 @@ void test_nontemplate_cast_wrappers()
 #endif
 }
 
-int main(int, char**) {
+void tests() {
   test_algorithms();
 
   int i = 42;
   test_template_cast_wrappers(i, std::move(i));
   test_nontemplate_cast_wrappers();
-
-  return 0;
 }
diff --git a/libcxx/test/libcxx/diagnostics/nodiscard_extensions.verify.cpp b/libcxx/test/libcxx/diagnostics/nodiscard_extensions.verify.cpp
index fe5fe24af5553..3d0dd2ab22ad6 100644
--- a/libcxx/test/libcxx/diagnostics/nodiscard_extensions.verify.cpp
+++ b/libcxx/test/libcxx/diagnostics/nodiscard_extensions.verify.cpp
@@ -349,12 +349,10 @@ void test_nontemplate_cast_wrappers()
 #endif
 }
 
-int main(int, char**) {
+void f() {
   test_algorithms();
 
   int i = 42;
   test_template_cast_wrappers(i, std::move(i));
   test_nontemplate_cast_wrappers();
-
-  return 0;
 }
diff --git a/libcxx/test/libcxx/fuzzing/random.pass.cpp b/libcxx/test/libcxx/fuzzing/random.pass.cpp
index e7545ffdf4da2..79ab7ac41151c 100644
--- a/libcxx/test/libcxx/fuzzing/random.pass.cpp
+++ b/libcxx/test/libcxx/fuzzing/random.pass.cpp
@@ -38,7 +38,7 @@ struct ParamTypeHelper {
   using ResultT = typename Dist::result_type;
   static_assert(std::is_same<ResultT, typename ParamT::distribution_type::result_type>::value, "");
 
-  static ParamT Create(const uint8_t* data, std::size_t size, bool &OK) {
+  static ParamT Create(const std::uint8_t* data, std::size_t size, bool &OK) {
     constexpr bool select_vector_result = std::is_constructible<ParamT, ResultT*, ResultT*, ResultT*>::value;
     constexpr bool select_vector_double = std::is_constructible<ParamT, double*, double*>::value;
     constexpr int selector = select_vector_result ? 0 : (select_vector_double ? 1 : 2);
diff --git a/libcxx/test/libcxx/input.output/filesystems/convert_file_time.pass.cpp b/libcxx/test/libcxx/input.output/filesystems/convert_file_time.pass.cpp
index 88054f3e1e7d4..b86282ccbedb5 100644
--- a/libcxx/test/libcxx/input.output/filesystems/convert_file_time.pass.cpp
+++ b/libcxx/test/libcxx/input.output/filesystems/convert_file_time.pass.cpp
@@ -280,12 +280,12 @@ using TestFileTimeT = time_point<TestClock<duration<IntType, Period> > >;
 int main(int, char**) {
   { assert((test_case<file_time_type, time_t, struct timespec>::test())); }
   {
-    assert((test_case<TestFileTimeT<int64_t>, int64_t,
-                      TestTimeSpec<int64_t, long> >::test()));
+    assert((test_case<TestFileTimeT<std::int64_t>, std::int64_t,
+                      TestTimeSpec<std::int64_t, long> >::test()));
   }
   {
-    assert((test_case<TestFileTimeT<long long>, int32_t,
-                      TestTimeSpec<int32_t, int32_t> >::test()));
+    assert((test_case<TestFileTimeT<long long>, std::int32_t,
+                      TestTimeSpec<std::int32_t, std::int32_t> >::test()));
   }
   {
     // Test that insane platforms like ppc64 linux, which use long double as time_t,
@@ -295,12 +295,12 @@ int main(int, char**) {
   }
 #ifndef TEST_HAS_NO_INT128_T
   {
-    assert((test_case<TestFileTimeT<__int128_t, std::nano>, int64_t,
-                      TestTimeSpec<int64_t, long> >::test()));
+    assert((test_case<TestFileTimeT<__int128_t, std::nano>, std::int64_t,
+                      TestTimeSpec<std::int64_t, long> >::test()));
   }
   {
-    assert((test_case<TestFileTimeT<__int128_t, std::nano>, int32_t,
-                      TestTimeSpec<int32_t, int32_t> >::test()));
+    assert((test_case<TestFileTimeT<__int128_t, std::nano>, std::int32_t,
+                      TestTimeSpec<std::int32_t, std::int32_t> >::test()));
   }
 #endif
 
diff --git a/libcxx/test/libcxx/iterators/predef.iterators/__unconstrained_reverse_iterator/reverse.iter.cons/ctor.iter.explicit.verify.cpp b/libcxx/test/libcxx/iterators/predef.iterators/__unconstrained_reverse_iterator/reverse.iter.cons/ctor.iter.explicit.verify.cpp
index 95626bef2e89b..6440e284f6a01 100644
--- a/libcxx/test/libcxx/iterators/predef.iterators/__unconstrained_reverse_iterator/reverse.iter.cons/ctor.iter.explicit.verify.cpp
+++ b/libcxx/test/libcxx/iterators/predef.iterators/__unconstrained_reverse_iterator/reverse.iter.cons/ctor.iter.explicit.verify.cpp
@@ -16,8 +16,7 @@
 
 #include <iterator>
 
-int main(int, char**) {
+void f() {
     char const* it = "";
     std::__unconstrained_reverse_iterator<char const*> r = it; // expected-error{{no viable conversion from 'const char *' to 'std::__unconstrained_reverse_iterator<const char *>'}}
-    return 0;
 }
diff --git a/libcxx/test/libcxx/iterators/predef.iterators/reverse.iterators/bad_template_argument.verify.cpp b/libcxx/test/libcxx/iterators/predef.iterators/reverse.iterators/bad_template_argument.verify.cpp
index 4ccf85ea528e2..1ad9ca9676ffb 100644
--- a/libcxx/test/libcxx/iterators/predef.iterators/reverse.iterators/bad_template_argument.verify.cpp
+++ b/libcxx/test/libcxx/iterators/predef.iterators/reverse.iterators/bad_template_argument.verify.cpp
@@ -16,9 +16,7 @@
 
 #include "test_iterators.h"
 
-int main(int, char**) {
+void f() {
   using BadIter = std::reverse_iterator<forward_iterator<int*>>;
   BadIter i; //expected-error-re@*:* {{{{(static_assert|static assertion)}} failed{{.*}}reverse_iterator<It> requires It to be a bidirectional iterator.}}
-
-  return 0;
 }
diff --git a/libcxx/test/libcxx/language.support/support.dynamic/aligned_alloc_availability.verify.cpp b/libcxx/test/libcxx/language.support/support.dynamic/aligned_alloc_availability.verify.cpp
index 6296bbddf547a..c22d04dbf37f9 100644
--- a/libcxx/test/libcxx/language.support/support.dynamic/aligned_alloc_availability.verify.cpp
+++ b/libcxx/test/libcxx/language.support/support.dynamic/aligned_alloc_availability.verify.cpp
@@ -23,8 +23,7 @@ constexpr auto OverAligned = __STDCPP_DEFAULT_NEW_ALIGNMENT__ * 2;
 
 struct alignas(OverAligned) A { };
 
-int main(int, char**)
-{
+void f() {
     // Normal versions
     {
         A *a1 = new A; // expected-error-re {{aligned allocation function of type {{.+}} is only available on}}
diff --git a/libcxx/test/libcxx/language.support/support.dynamic/new_faligned_allocation.pass.cpp b/libcxx/test/libcxx/language.support/support.dynamic/new_faligned_allocation.pass.cpp
index 8820e12b55e95..aecdcd6c787ae 100644
--- a/libcxx/test/libcxx/language.support/support.dynamic/new_faligned_allocation.pass.cpp
+++ b/libcxx/test/libcxx/language.support/support.dynamic/new_faligned_allocation.pass.cpp
@@ -21,10 +21,11 @@
 // REQUIRES: -faligned-allocation
 // ADDITIONAL_COMPILE_FLAGS: -faligned-allocation
 
+#include <cassert>
 #include <new>
-#include <typeinfo>
 #include <string>
-#include <cassert>
+#include <type_traits>
+#include <typeinfo>
 
 #include "test_macros.h"
 
diff --git a/libcxx/test/libcxx/modules_include.sh.cpp b/libcxx/test/libcxx/modules_include.sh.cpp
index c013e97353b3e..4a8f7293bb86f 100644
--- a/libcxx/test/libcxx/modules_include.sh.cpp
+++ b/libcxx/test/libcxx/modules_include.sh.cpp
@@ -13,8 +13,6 @@
 // The system-provided <uchar.h> seems to be broken on AIX
 // XFAIL: LIBCXX-AIX-FIXME
 
-// XFAIL: LIBCXX-FREEBSD-FIXME
-
 // GCC doesn't support -fcxx-modules
 // UNSUPPORTED: gcc
 
diff --git a/libcxx/test/libcxx/numerics/numeric.ops/midpoint.integer.pass.cpp b/libcxx/test/libcxx/numerics/numeric.ops/midpoint.integer.pass.cpp
index 054a682dc328f..a89a3d69476cd 100644
--- a/libcxx/test/libcxx/numerics/numeric.ops/midpoint.integer.pass.cpp
+++ b/libcxx/test/libcxx/numerics/numeric.ops/midpoint.integer.pass.cpp
@@ -40,10 +40,10 @@ int main(int, char**)
     test<long>();
     test<long long>();
 
-    test<int8_t>();
-    test<int16_t>();
-    test<int32_t>();
-    test<int64_t>();
+    test<std::int8_t>();
+    test<std::int16_t>();
+    test<std::int32_t>();
+    test<std::int64_t>();
 
     test<unsigned char>();
     test<unsigned short>();
@@ -51,10 +51,10 @@ int main(int, char**)
     test<unsigned long>();
     test<unsigned long long>();
 
-    test<uint8_t>();
-    test<uint16_t>();
-    test<uint32_t>();
-    test<uint64_t>();
+    test<std::uint8_t>();
+    test<std::uint16_t>();
+    test<std::uint32_t>();
+    test<std::uint64_t>();
 
 #ifndef TEST_HAS_NO_INT128
     test<__int128_t>();
diff --git a/libcxx/test/libcxx/numerics/rand/rand.device/has-no-random-device.verify.cpp b/libcxx/test/libcxx/numerics/rand/rand.device/has-no-random-device.verify.cpp
index 7559c88b59e9f..03d592774e572 100644
--- a/libcxx/test/libcxx/numerics/rand/rand.device/has-no-random-device.verify.cpp
+++ b/libcxx/test/libcxx/numerics/rand/rand.device/has-no-random-device.verify.cpp
@@ -13,7 +13,6 @@
 
 #include <random>
 
-int main(int, char**) {
+void f() {
   std::random_device d; // expected-error {{no type named 'random_device' in namespace 'std'}}
-  return 0;
 }
diff --git a/libcxx/test/libcxx/private_headers.verify.cpp b/libcxx/test/libcxx/private_headers.verify.cpp
index 0f611ce5eec75..dd2a655642e51 100644
--- a/libcxx/test/libcxx/private_headers.verify.cpp
+++ b/libcxx/test/libcxx/private_headers.verify.cpp
@@ -259,10 +259,15 @@ END-SCRIPT
 #include <__bit/rotate.h> // expected-error@*:* {{use of private header from outside its module: '__bit/rotate.h'}}
 #include <__bit_reference> // expected-error@*:* {{use of private header from outside its module: '__bit_reference'}}
 #include <__charconv/chars_format.h> // expected-error@*:* {{use of private header from outside its module: '__charconv/chars_format.h'}}
+#include <__charconv/from_chars_integral.h> // expected-error@*:* {{use of private header from outside its module: '__charconv/from_chars_integral.h'}}
 #include <__charconv/from_chars_result.h> // expected-error@*:* {{use of private header from outside its module: '__charconv/from_chars_result.h'}}
 #include <__charconv/tables.h> // expected-error@*:* {{use of private header from outside its module: '__charconv/tables.h'}}
+#include <__charconv/to_chars.h> // expected-error@*:* {{use of private header from outside its module: '__charconv/to_chars.h'}}
 #include <__charconv/to_chars_base_10.h> // expected-error@*:* {{use of private header from outside its module: '__charconv/to_chars_base_10.h'}}
+#include <__charconv/to_chars_floating_point.h> // expected-error@*:* {{use of private header from outside its module: '__charconv/to_chars_floating_point.h'}}
+#include <__charconv/to_chars_integral.h> // expected-error@*:* {{use of private header from outside its module: '__charconv/to_chars_integral.h'}}
 #include <__charconv/to_chars_result.h> // expected-error@*:* {{use of private header from outside its module: '__charconv/to_chars_result.h'}}
+#include <__charconv/traits.h> // expected-error@*:* {{use of private header from outside its module: '__charconv/traits.h'}}
 #include <__chrono/calendar.h> // expected-error@*:* {{use of private header from outside its module: '__chrono/calendar.h'}}
 #include <__chrono/concepts.h> // expected-error@*:* {{use of private header from outside its module: '__chrono/concepts.h'}}
 #include <__chrono/convert_to_timespec.h> // expected-error@*:* {{use of private header from outside its module: '__chrono/convert_to_timespec.h'}}
@@ -329,6 +334,11 @@ END-SCRIPT
 #include <__coroutine/trivial_awaitables.h> // expected-error@*:* {{use of private header from outside its module: '__coroutine/trivial_awaitables.h'}}
 #include <__debug_utils/randomize_range.h> // expected-error@*:* {{use of private header from outside its module: '__debug_utils/randomize_range.h'}}
 #include <__errc> // expected-error@*:* {{use of private header from outside its module: '__errc'}}
+#include <__exception/exception.h> // expected-error@*:* {{use of private header from outside its module: '__exception/exception.h'}}
+#include <__exception/exception_ptr.h> // expected-error@*:* {{use of private header from outside its module: '__exception/exception_ptr.h'}}
+#include <__exception/nested_exception.h> // expected-error@*:* {{use of private header from outside its module: '__exception/nested_exception.h'}}
+#include <__exception/operations.h> // expected-error@*:* {{use of private header from outside its module: '__exception/operations.h'}}
+#include <__exception/terminate.h> // expected-error@*:* {{use of private header from outside its module: '__exception/terminate.h'}}
 #include <__expected/bad_expected_access.h> // expected-error@*:* {{use of private header from outside its module: '__expected/bad_expected_access.h'}}
 #include <__expected/expected.h> // expected-error@*:* {{use of private header from outside its module: '__expected/expected.h'}}
 #include <__expected/unexpect.h> // expected-error@*:* {{use of private header from outside its module: '__expected/unexpect.h'}}
diff --git a/libcxx/test/libcxx/ranges/range.adaptors/range.join/segmented_iterator.compile.pass.cpp b/libcxx/test/libcxx/ranges/range.adaptors/range.join/segmented_iterator.compile.pass.cpp
index 509b0c89e7f10..82e8cab503a27 100644
--- a/libcxx/test/libcxx/ranges/range.adaptors/range.join/segmented_iterator.compile.pass.cpp
+++ b/libcxx/test/libcxx/ranges/range.adaptors/range.join/segmented_iterator.compile.pass.cpp
@@ -10,6 +10,7 @@
 // UNSUPPORTED: !c++experimental
 
 #include <ranges>
+#include <utility>
 #include <vector>
 
 using JoinView = decltype(std::views::join(std::declval<std::vector<std::vector<int>>&>()));
diff --git a/libcxx/test/libcxx/ranges/version.compile.pass.cpp b/libcxx/test/libcxx/ranges/version.compile.pass.cpp
index de955a91b8a4a..4b3aa3f164d43 100644
--- a/libcxx/test/libcxx/ranges/version.compile.pass.cpp
+++ b/libcxx/test/libcxx/ranges/version.compile.pass.cpp
@@ -16,6 +16,3 @@
 #ifndef _LIBCPP_VERSION
 #error _LIBCPP_VERSION not defined
 #endif
-
-// Required for MSVC internal test runner compatibility.
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/libcxx/selftest/compile.pass.mm/compile-error.compile.pass.mm b/libcxx/test/libcxx/selftest/compile.pass.mm/compile-error.compile.pass.mm
new file mode 100644
index 0000000000000..04bb4cc445266
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/compile.pass.mm/compile-error.compile.pass.mm
@@ -0,0 +1,18 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: objective-c++
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it fails at compile-time
+
+struct Foo { };
+typedef Foo::x x;
+
+int main(int, char**) { return 0; }
diff --git a/libcxx/test/libcxx/selftest/compile.pass.mm/compile-success.compile.pass.mm b/libcxx/test/libcxx/selftest/compile.pass.mm/compile-success.compile.pass.mm
new file mode 100644
index 0000000000000..9effc02650cb4
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/compile.pass.mm/compile-success.compile.pass.mm
@@ -0,0 +1,16 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: objective-c++
+
+// Make sure the test passes if it succeeds to compile
+
+@interface I;
+@end
+
+int main(int, char**) { return 0; }
diff --git a/libcxx/test/libcxx/selftest/compile.pass.mm/link-error.compile.pass.mm b/libcxx/test/libcxx/selftest/compile.pass.mm/link-error.compile.pass.mm
new file mode 100644
index 0000000000000..77f6a63c0c9a2
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/compile.pass.mm/link-error.compile.pass.mm
@@ -0,0 +1,21 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: objective-c++
+
+// Make sure the test passes even if there's a link error, i.e. it isn't linked.
+
+@interface I;
+@end
+
+extern void this_is_an_undefined_symbol();
+
+int main(int, char**) {
+    this_is_an_undefined_symbol();
+    return 0;
+}
diff --git a/libcxx/test/libcxx/selftest/compile.pass.mm/run-error.compile.pass.mm b/libcxx/test/libcxx/selftest/compile.pass.mm/run-error.compile.pass.mm
new file mode 100644
index 0000000000000..cdd3ef44d88b5
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/compile.pass.mm/run-error.compile.pass.mm
@@ -0,0 +1,18 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: objective-c++
+
+// Make sure the test passes even if there's a runtime error, i.e. it isn't run.
+
+@interface I;
+@end
+
+int main(int, char**) {
+    return 1;
+}
diff --git a/libcxx/test/libcxx/selftest/link.pass.mm/compile-error.link.pass.mm b/libcxx/test/libcxx/selftest/link.pass.mm/compile-error.link.pass.mm
new file mode 100644
index 0000000000000..6227641c54ed5
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/link.pass.mm/compile-error.link.pass.mm
@@ -0,0 +1,18 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: objective-c++
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it fails at compile-time.
+
+struct Foo { };
+typedef Foo::x x;
+
+int main(int, char**) { return 0; }
diff --git a/libcxx/test/libcxx/selftest/link.pass.mm/link-error.link.pass.mm b/libcxx/test/libcxx/selftest/link.pass.mm/link-error.link.pass.mm
new file mode 100644
index 0000000000000..e95594276e1ea
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/link.pass.mm/link-error.link.pass.mm
@@ -0,0 +1,20 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: objective-c++
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it fails to link.
+
+extern void this_is_an_undefined_symbol();
+
+int main(int, char**) {
+    this_is_an_undefined_symbol();
+    return 0;
+}
diff --git a/libcxx/test/libcxx/selftest/link.pass.mm/link-success.link.pass.mm b/libcxx/test/libcxx/selftest/link.pass.mm/link-success.link.pass.mm
new file mode 100644
index 0000000000000..a831d7f914f2e
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/link.pass.mm/link-success.link.pass.mm
@@ -0,0 +1,16 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: objective-c++
+
+// Make sure the test passes if it succeeds to link.
+
+@interface I;
+@end
+
+int main(int, char**) { return 0; }
diff --git a/libcxx/test/libcxx/selftest/link.pass.mm/run-error.link.pass.mm b/libcxx/test/libcxx/selftest/link.pass.mm/run-error.link.pass.mm
new file mode 100644
index 0000000000000..778fab3329ae6
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/link.pass.mm/run-error.link.pass.mm
@@ -0,0 +1,19 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: objective-c++
+
+// Make sure the test passes if it succeeds to link, even though it would have
+// failed at runtime.
+
+@interface I;
+@end
+
+int main(int, char**) {
+    return 1;
+}
diff --git a/libcxx/test/libcxx/selftest/verify.cpp/no-werror.verify.cpp b/libcxx/test/libcxx/selftest/verify.cpp/no-werror.verify.cpp
index b1234ae626744..e96d6be9087bc 100644
--- a/libcxx/test/libcxx/selftest/verify.cpp/no-werror.verify.cpp
+++ b/libcxx/test/libcxx/selftest/verify.cpp/no-werror.verify.cpp
@@ -10,7 +10,6 @@
 
 // ADDITIONAL_COMPILE_FLAGS: -Wunused-variable
 
-int main(int, char**) {
+void f() {
     int foo; // expected-warning {{unused variable}}
-    return 0;
 }
diff --git a/libcxx/test/libcxx/strings/basic.string/alignof.compile.pass.cpp b/libcxx/test/libcxx/strings/basic.string/alignof.compile.pass.cpp
index d679e7a1b7e31..1613fd7b5fdaa 100644
--- a/libcxx/test/libcxx/strings/basic.string/alignof.compile.pass.cpp
+++ b/libcxx/test/libcxx/strings/basic.string/alignof.compile.pass.cpp
@@ -18,7 +18,7 @@
 
 template <class T>
 class small_pointer {
-  uint16_t offset;
+  std::uint16_t offset;
 };
 
 template <class T>
@@ -26,8 +26,8 @@ class small_iter_allocator {
 public:
   using value_type = T;
   using pointer = small_pointer<T>;
-  using size_type = int16_t;
-  using difference_type = int16_t;
+  using size_type = std::int16_t;
+  using difference_type = std::int16_t;
 
   small_iter_allocator() TEST_NOEXCEPT {}
 
diff --git a/libcxx/test/libcxx/strings/basic.string/sizeof.compile.pass.cpp b/libcxx/test/libcxx/strings/basic.string/sizeof.compile.pass.cpp
index 9024d11c598e0..7a1835d0479f5 100644
--- a/libcxx/test/libcxx/strings/basic.string/sizeof.compile.pass.cpp
+++ b/libcxx/test/libcxx/strings/basic.string/sizeof.compile.pass.cpp
@@ -16,7 +16,7 @@
 
 template <class T>
 class small_pointer {
-  uint16_t offset;
+  std::uint16_t offset;
 };
 
 template <class T>
@@ -24,8 +24,8 @@ class small_iter_allocator {
 public:
   using value_type = T;
   using pointer = small_pointer<T>;
-  using size_type = int16_t;
-  using difference_type = int16_t;
+  using size_type = std::int16_t;
+  using difference_type = std::int16_t;
 
   small_iter_allocator() TEST_NOEXCEPT {}
 
diff --git a/libcxx/test/libcxx/strings/basic.string/string.iterators/debug.iterator.index.pass.cpp b/libcxx/test/libcxx/strings/basic.string/string.iterators/debug.iterator.index.pass.cpp
index 13a2301937309..2a120fb645770 100644
--- a/libcxx/test/libcxx/strings/basic.string/string.iterators/debug.iterator.index.pass.cpp
+++ b/libcxx/test/libcxx/strings/basic.string/string.iterators/debug.iterator.index.pass.cpp
@@ -20,7 +20,7 @@
 #include "min_allocator.h"
 
 int main(int, char**) {
-  using T = decltype(uint8_t() - uint8_t());
+  using T = decltype(std::uint8_t() - std::uint8_t());
   {
     typedef std::string C;
     C c(1, '\0');
diff --git a/libcxx/test/libcxx/strings/string.view/assert.ctor.length.pass.cpp b/libcxx/test/libcxx/strings/string.view/assert.ctor.length.pass.cpp
new file mode 100644
index 0000000000000..d6f0449d555ff
--- /dev/null
+++ b/libcxx/test/libcxx/strings/string.view/assert.ctor.length.pass.cpp
@@ -0,0 +1,27 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: has-unix-headers
+
+// UNSUPPORTED: c++03, c++11
+// XFAIL: use_system_cxx_lib && target={{.+}}-apple-macosx{{10.9|10.10|10.11|10.12|10.13|10.14|10.15|11.0|12.0}}
+// ADDITIONAL_COMPILE_FLAGS: -D_LIBCPP_ENABLE_ASSERTIONS=1
+
+// Construct a string_view from an invalid length
+// constexpr basic_string_view( const _CharT* s, size_type len )
+
+#include <string_view>
+
+#include "check_assertion.h"
+
+int main(int, char**) {
+  char c = 0;
+  TEST_LIBCPP_ASSERT_FAILURE(
+      std::string_view(&c, -1), "string_view::string_view(_CharT *, size_t): length does not fit in difference_type");
+  return 0;
+}
diff --git a/libcxx/test/libcxx/thread/atomic.availability.verify.cpp b/libcxx/test/libcxx/thread/atomic.availability.verify.cpp
index e96d461a45720..91e7257c60244 100644
--- a/libcxx/test/libcxx/thread/atomic.availability.verify.cpp
+++ b/libcxx/test/libcxx/thread/atomic.availability.verify.cpp
@@ -9,15 +9,12 @@
 // UNSUPPORTED: c++03, c++11
 // REQUIRES: use_system_cxx_lib && target={{.+}}-apple-macosx10.{{9|10|11|12|13|14|15}}
 
-
 // Test the availability markup on the C++20 Synchronization Library
 // additions to <atomic>.
 
 #include <atomic>
 
-
-int main(int, char**)
-{
+void f() {
     {
         std::atomic<int> i(3);
         std::memory_order m = std::memory_order_relaxed;
diff --git a/libcxx/test/libcxx/thread/barrier.availability.verify.cpp b/libcxx/test/libcxx/thread/barrier.availability.verify.cpp
index 8da39fe3c2dfd..a9043854e61f7 100644
--- a/libcxx/test/libcxx/thread/barrier.availability.verify.cpp
+++ b/libcxx/test/libcxx/thread/barrier.availability.verify.cpp
@@ -9,7 +9,6 @@
 // UNSUPPORTED: c++03, c++11
 // REQUIRES: use_system_cxx_lib && target={{.+}}-apple-macosx10.{{9|10|11|12|13|14|15}}
 
-
 // Test the availability markup on std::barrier.
 
 #include <barrier>
@@ -19,8 +18,7 @@ struct CompletionF {
     void operator()() { }
 };
 
-int main(int, char**)
-{
+void f() {
     // Availability markup on std::barrier<>
     {
         std::barrier<> b(10); // expected-error {{is unavailable}}
diff --git a/libcxx/test/libcxx/thread/latch.availability.verify.cpp b/libcxx/test/libcxx/thread/latch.availability.verify.cpp
index 8555e0e70267a..ceb2c19f79595 100644
--- a/libcxx/test/libcxx/thread/latch.availability.verify.cpp
+++ b/libcxx/test/libcxx/thread/latch.availability.verify.cpp
@@ -9,14 +9,11 @@
 // UNSUPPORTED: c++03, c++11
 // REQUIRES: use_system_cxx_lib && target={{.+}}-apple-macosx10.{{9|10|11|12|13|14|15}}
 
-
 // Test the availability markup on std::latch.
 
 #include <latch>
 
-
-int main(int, char**)
-{
+void f() {
     std::latch latch(10);
     latch.count_down(); // expected-error {{is unavailable}}
     latch.count_down(3); // expected-error {{is unavailable}}
diff --git a/libcxx/test/libcxx/thread/semaphore.availability.verify.cpp b/libcxx/test/libcxx/thread/semaphore.availability.verify.cpp
index a6d5c36678284..8434e0e76f60a 100644
--- a/libcxx/test/libcxx/thread/semaphore.availability.verify.cpp
+++ b/libcxx/test/libcxx/thread/semaphore.availability.verify.cpp
@@ -9,15 +9,12 @@
 // UNSUPPORTED: c++03, c++11
 // REQUIRES: use_system_cxx_lib && target={{.+}}-apple-macosx10.{{9|10|11|12|13|14|15}}
 
-
 // Test the availability markup on std::counting_semaphore and std::binary_semaphore.
 
 #include <chrono>
 #include <semaphore>
 
-
-int main(int, char**)
-{
+void f() {
     {
         // Tests for std::counting_semaphore with non-default template argument
         std::counting_semaphore<20> sem(10);
diff --git a/libcxx/test/libcxx/thread/thread.barrier/version.compile.pass.cpp b/libcxx/test/libcxx/thread/thread.barrier/version.compile.pass.cpp
index 7890787b6a2a1..fb80bb035bbcb 100644
--- a/libcxx/test/libcxx/thread/thread.barrier/version.compile.pass.cpp
+++ b/libcxx/test/libcxx/thread/thread.barrier/version.compile.pass.cpp
@@ -18,8 +18,3 @@
 #ifndef _LIBCPP_VERSION
 #error _LIBCPP_VERSION not defined
 #endif
-
-int main(int, char**)
-{
-  return 0;
-}
diff --git a/libcxx/test/libcxx/thread/thread.latch/version.compile.pass.cpp b/libcxx/test/libcxx/thread/thread.latch/version.compile.pass.cpp
index 365ee9276be17..559c827d81316 100644
--- a/libcxx/test/libcxx/thread/thread.latch/version.compile.pass.cpp
+++ b/libcxx/test/libcxx/thread/thread.latch/version.compile.pass.cpp
@@ -18,8 +18,3 @@
 #ifndef _LIBCPP_VERSION
 #error _LIBCPP_VERSION not defined
 #endif
-
-int main(int, char**)
-{
-  return 0;
-}
diff --git a/libcxx/test/libcxx/thread/thread.lock/thread.lock.guard/nodiscard.verify.cpp b/libcxx/test/libcxx/thread/thread.lock/thread.lock.guard/nodiscard.verify.cpp
index c0cad32da198b..5fe68c83b3d9f 100644
--- a/libcxx/test/libcxx/thread/thread.lock/thread.lock.guard/nodiscard.verify.cpp
+++ b/libcxx/test/libcxx/thread/thread.lock/thread.lock.guard/nodiscard.verify.cpp
@@ -23,9 +23,8 @@
 
 #include <mutex>
 
-int main(int, char**) {
+void f() {
     std::mutex m;
     std::lock_guard<std::mutex>{m}; // expected-warning {{ignoring temporary created by a constructor declared with 'nodiscard' attribute}}
     std::lock_guard<std::mutex>{m, std::adopt_lock}; // expected-warning {{ignoring temporary created by a constructor declared with 'nodiscard' attribute}}
-    return 0;
 }
diff --git a/libcxx/test/libcxx/thread/thread.mutex/thread_safety_annotations_not_enabled.pass.cpp b/libcxx/test/libcxx/thread/thread.mutex/thread_safety_annotations_not_enabled.pass.cpp
index 02072f5e6f8a7..5d40fa71a4152 100644
--- a/libcxx/test/libcxx/thread/thread.mutex/thread_safety_annotations_not_enabled.pass.cpp
+++ b/libcxx/test/libcxx/thread/thread.mutex/thread_safety_annotations_not_enabled.pass.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 // UNSUPPORTED: no-threads
+// UNSUPPORTED: c++03
 
 // <mutex>
 
diff --git a/libcxx/test/libcxx/thread/thread.semaphore/version.compile.pass.cpp b/libcxx/test/libcxx/thread/thread.semaphore/version.compile.pass.cpp
index 3f59448a8524a..9eaf1ad7e7bba 100644
--- a/libcxx/test/libcxx/thread/thread.semaphore/version.compile.pass.cpp
+++ b/libcxx/test/libcxx/thread/thread.semaphore/version.compile.pass.cpp
@@ -18,8 +18,3 @@
 #ifndef _LIBCPP_VERSION
 #error _LIBCPP_VERSION not defined
 #endif
-
-int main(int, char**)
-{
-  return 0;
-}
diff --git a/libcxx/test/libcxx/transitive_includes/cxx03.csv b/libcxx/test/libcxx/transitive_includes/cxx03.csv
index 846a465923605..92a84f59268bb 100644
--- a/libcxx/test/libcxx/transitive_includes/cxx03.csv
+++ b/libcxx/test/libcxx/transitive_includes/cxx03.csv
@@ -74,6 +74,7 @@ barrier ratio
 barrier stdexcept
 barrier type_traits
 barrier variant
+barrier version
 bit cstdint
 bit cstdlib
 bit iosfwd
@@ -162,6 +163,7 @@ condition_variable cstddef
 condition_variable cstdint
 condition_variable cstdlib
 condition_variable cstring
+condition_variable ctime
 condition_variable initializer_list
 condition_variable iosfwd
 condition_variable limits
@@ -211,7 +213,6 @@ exception version
 execution version
 expected cstddef
 expected cstdlib
-expected exception
 expected initializer_list
 expected new
 expected version
@@ -393,16 +394,24 @@ functional unordered_map
 functional utility
 functional vector
 functional version
+future atomic
 future chrono
 future cstddef
+future cstdint
+future cstdlib
+future cstring
 future exception
+future initializer_list
+future iosfwd
 future limits
 future mutex
 future new
 future ratio
+future stdexcept
 future system_error
 future thread
 future type_traits
+future typeinfo
 future version
 initializer_list cstddef
 iomanip istream
@@ -452,7 +461,14 @@ iterator utility
 iterator variant
 iterator version
 latch atomic
+latch cstddef
+latch cstdint
+latch cstring
+latch ctime
+latch iosfwd
 latch limits
+latch ratio
+latch type_traits
 latch version
 limits type_traits
 limits version
@@ -545,6 +561,7 @@ mutex cstddef
 mutex cstdint
 mutex cstdlib
 mutex cstring
+mutex ctime
 mutex functional
 mutex initializer_list
 mutex iosfwd
@@ -695,6 +712,8 @@ scoped_allocator variant
 scoped_allocator version
 semaphore atomic
 semaphore cstddef
+semaphore cstdint
+semaphore cstring
 semaphore ctime
 semaphore iosfwd
 semaphore limits
diff --git a/libcxx/test/libcxx/transitive_includes/cxx11.csv b/libcxx/test/libcxx/transitive_includes/cxx11.csv
index 76262c8add9a0..833e8ac19f8e9 100644
--- a/libcxx/test/libcxx/transitive_includes/cxx11.csv
+++ b/libcxx/test/libcxx/transitive_includes/cxx11.csv
@@ -74,6 +74,7 @@ barrier ratio
 barrier stdexcept
 barrier type_traits
 barrier variant
+barrier version
 bit cstdint
 bit cstdlib
 bit iosfwd
@@ -162,6 +163,7 @@ condition_variable cstddef
 condition_variable cstdint
 condition_variable cstdlib
 condition_variable cstring
+condition_variable ctime
 condition_variable initializer_list
 condition_variable iosfwd
 condition_variable limits
@@ -211,7 +213,6 @@ exception version
 execution version
 expected cstddef
 expected cstdlib
-expected exception
 expected initializer_list
 expected new
 expected version
@@ -393,16 +394,24 @@ functional unordered_map
 functional utility
 functional vector
 functional version
+future atomic
 future chrono
 future cstddef
+future cstdint
+future cstdlib
+future cstring
 future exception
+future initializer_list
+future iosfwd
 future limits
 future mutex
 future new
 future ratio
+future stdexcept
 future system_error
 future thread
 future type_traits
+future typeinfo
 future version
 initializer_list cstddef
 iomanip istream
@@ -452,7 +461,14 @@ iterator utility
 iterator variant
 iterator version
 latch atomic
+latch cstddef
+latch cstdint
+latch cstring
+latch ctime
+latch iosfwd
 latch limits
+latch ratio
+latch type_traits
 latch version
 limits type_traits
 limits version
@@ -545,6 +561,7 @@ mutex cstddef
 mutex cstdint
 mutex cstdlib
 mutex cstring
+mutex ctime
 mutex functional
 mutex initializer_list
 mutex iosfwd
@@ -696,6 +713,8 @@ scoped_allocator variant
 scoped_allocator version
 semaphore atomic
 semaphore cstddef
+semaphore cstdint
+semaphore cstring
 semaphore ctime
 semaphore iosfwd
 semaphore limits
diff --git a/libcxx/test/libcxx/transitive_includes/cxx14.csv b/libcxx/test/libcxx/transitive_includes/cxx14.csv
index 8537140af9f7b..c3c0ae6f71e8d 100644
--- a/libcxx/test/libcxx/transitive_includes/cxx14.csv
+++ b/libcxx/test/libcxx/transitive_includes/cxx14.csv
@@ -74,6 +74,7 @@ barrier ratio
 barrier stdexcept
 barrier type_traits
 barrier variant
+barrier version
 bit cstdint
 bit cstdlib
 bit iosfwd
@@ -162,6 +163,7 @@ condition_variable cstddef
 condition_variable cstdint
 condition_variable cstdlib
 condition_variable cstring
+condition_variable ctime
 condition_variable initializer_list
 condition_variable iosfwd
 condition_variable limits
@@ -211,7 +213,6 @@ exception version
 execution version
 expected cstddef
 expected cstdlib
-expected exception
 expected initializer_list
 expected new
 expected version
@@ -395,16 +396,24 @@ functional unordered_map
 functional utility
 functional vector
 functional version
+future atomic
 future chrono
 future cstddef
+future cstdint
+future cstdlib
+future cstring
 future exception
+future initializer_list
+future iosfwd
 future limits
 future mutex
 future new
 future ratio
+future stdexcept
 future system_error
 future thread
 future type_traits
+future typeinfo
 future version
 initializer_list cstddef
 iomanip istream
@@ -454,7 +463,14 @@ iterator utility
 iterator variant
 iterator version
 latch atomic
+latch cstddef
+latch cstdint
+latch cstring
+latch ctime
+latch iosfwd
 latch limits
+latch ratio
+latch type_traits
 latch version
 limits type_traits
 limits version
@@ -547,6 +563,7 @@ mutex cstddef
 mutex cstdint
 mutex cstdlib
 mutex cstring
+mutex ctime
 mutex functional
 mutex initializer_list
 mutex iosfwd
@@ -698,6 +715,8 @@ scoped_allocator variant
 scoped_allocator version
 semaphore atomic
 semaphore cstddef
+semaphore cstdint
+semaphore cstring
 semaphore ctime
 semaphore iosfwd
 semaphore limits
diff --git a/libcxx/test/libcxx/transitive_includes/cxx17.csv b/libcxx/test/libcxx/transitive_includes/cxx17.csv
index 8537140af9f7b..c3c0ae6f71e8d 100644
--- a/libcxx/test/libcxx/transitive_includes/cxx17.csv
+++ b/libcxx/test/libcxx/transitive_includes/cxx17.csv
@@ -74,6 +74,7 @@ barrier ratio
 barrier stdexcept
 barrier type_traits
 barrier variant
+barrier version
 bit cstdint
 bit cstdlib
 bit iosfwd
@@ -162,6 +163,7 @@ condition_variable cstddef
 condition_variable cstdint
 condition_variable cstdlib
 condition_variable cstring
+condition_variable ctime
 condition_variable initializer_list
 condition_variable iosfwd
 condition_variable limits
@@ -211,7 +213,6 @@ exception version
 execution version
 expected cstddef
 expected cstdlib
-expected exception
 expected initializer_list
 expected new
 expected version
@@ -395,16 +396,24 @@ functional unordered_map
 functional utility
 functional vector
 functional version
+future atomic
 future chrono
 future cstddef
+future cstdint
+future cstdlib
+future cstring
 future exception
+future initializer_list
+future iosfwd
 future limits
 future mutex
 future new
 future ratio
+future stdexcept
 future system_error
 future thread
 future type_traits
+future typeinfo
 future version
 initializer_list cstddef
 iomanip istream
@@ -454,7 +463,14 @@ iterator utility
 iterator variant
 iterator version
 latch atomic
+latch cstddef
+latch cstdint
+latch cstring
+latch ctime
+latch iosfwd
 latch limits
+latch ratio
+latch type_traits
 latch version
 limits type_traits
 limits version
@@ -547,6 +563,7 @@ mutex cstddef
 mutex cstdint
 mutex cstdlib
 mutex cstring
+mutex ctime
 mutex functional
 mutex initializer_list
 mutex iosfwd
@@ -698,6 +715,8 @@ scoped_allocator variant
 scoped_allocator version
 semaphore atomic
 semaphore cstddef
+semaphore cstdint
+semaphore cstring
 semaphore ctime
 semaphore iosfwd
 semaphore limits
diff --git a/libcxx/test/libcxx/transitive_includes/cxx20.csv b/libcxx/test/libcxx/transitive_includes/cxx20.csv
index 4cfdd9dc6c91c..a5be0f14d8589 100644
--- a/libcxx/test/libcxx/transitive_includes/cxx20.csv
+++ b/libcxx/test/libcxx/transitive_includes/cxx20.csv
@@ -72,6 +72,7 @@ barrier ratio
 barrier stdexcept
 barrier type_traits
 barrier variant
+barrier version
 bit cstdint
 bit cstdlib
 bit iosfwd
@@ -169,6 +170,7 @@ condition_variable cstddef
 condition_variable cstdint
 condition_variable cstdlib
 condition_variable cstring
+condition_variable ctime
 condition_variable initializer_list
 condition_variable iosfwd
 condition_variable limits
@@ -218,7 +220,6 @@ exception version
 execution version
 expected cstddef
 expected cstdlib
-expected exception
 expected initializer_list
 expected new
 expected version
@@ -402,15 +403,23 @@ functional unordered_map
 functional utility
 functional vector
 functional version
+future atomic
 future cstddef
+future cstdint
+future cstdlib
+future cstring
 future exception
+future initializer_list
+future iosfwd
 future limits
 future mutex
 future new
 future ratio
+future stdexcept
 future system_error
 future thread
 future type_traits
+future typeinfo
 future version
 initializer_list cstddef
 iomanip istream
@@ -460,7 +469,14 @@ iterator utility
 iterator variant
 iterator version
 latch atomic
+latch cstddef
+latch cstdint
+latch cstring
+latch ctime
+latch iosfwd
 latch limits
+latch ratio
+latch type_traits
 latch version
 limits type_traits
 limits version
@@ -553,6 +569,7 @@ mutex cstddef
 mutex cstdint
 mutex cstdlib
 mutex cstring
+mutex ctime
 mutex functional
 mutex initializer_list
 mutex iosfwd
@@ -704,6 +721,8 @@ scoped_allocator variant
 scoped_allocator version
 semaphore atomic
 semaphore cstddef
+semaphore cstdint
+semaphore cstring
 semaphore ctime
 semaphore iosfwd
 semaphore limits
diff --git a/libcxx/test/libcxx/transitive_includes/cxx2b.csv b/libcxx/test/libcxx/transitive_includes/cxx2b.csv
index 6ad66df42002e..8bed29fafc013 100644
--- a/libcxx/test/libcxx/transitive_includes/cxx2b.csv
+++ b/libcxx/test/libcxx/transitive_includes/cxx2b.csv
@@ -16,7 +16,6 @@ any initializer_list
 any limits
 any new
 any stdexcept
-any type_traits
 any typeinfo
 any version
 array compare
@@ -25,7 +24,6 @@ array cstdlib
 array initializer_list
 array limits
 array stdexcept
-array type_traits
 array version
 atomic cstddef
 atomic cstdint
@@ -36,7 +34,6 @@ atomic iosfwd
 atomic limits
 atomic ratio
 atomic version
-barrier atomic
 barrier cstddef
 barrier cstdint
 barrier cstring
@@ -45,6 +42,7 @@ barrier iosfwd
 barrier limits
 barrier new
 barrier ratio
+barrier version
 bit cstdint
 bit cstdlib
 bit limits
@@ -60,7 +58,6 @@ bitset limits
 bitset new
 bitset stdexcept
 bitset string
-bitset type_traits
 bitset version
 ccomplex complex
 charconv cerrno
@@ -70,7 +67,6 @@ charconv cstdint
 charconv cstdlib
 charconv initializer_list
 charconv limits
-charconv type_traits
 chrono array
 chrono charconv
 chrono cmath
@@ -90,7 +86,6 @@ chrono stdexcept
 chrono string
 chrono string_view
 chrono tuple
-chrono type_traits
 chrono vector
 chrono version
 cinttypes cstdint
@@ -111,15 +106,14 @@ complex cmath
 complex iosfwd
 complex sstream
 complex stdexcept
-complex type_traits
 complex version
 concepts cstddef
 concepts version
-condition_variable atomic
 condition_variable cstddef
 condition_variable cstdint
 condition_variable cstdlib
 condition_variable cstring
+condition_variable ctime
 condition_variable initializer_list
 condition_variable iosfwd
 condition_variable limits
@@ -150,7 +144,6 @@ deque limits
 deque new
 deque stdexcept
 deque tuple
-deque type_traits
 deque version
 exception cstddef
 exception cstdlib
@@ -158,7 +151,6 @@ exception version
 execution version
 expected cstddef
 expected cstdlib
-expected exception
 expected initializer_list
 expected new
 expected version
@@ -180,9 +172,7 @@ experimental/memory_resource limits
 experimental/memory_resource new
 experimental/memory_resource stdexcept
 experimental/memory_resource tuple
-experimental/memory_resource type_traits
 experimental/propagate_const cstddef
-experimental/propagate_const type_traits
 experimental/regex experimental/memory_resource
 experimental/regex experimental/string
 experimental/regex regex
@@ -213,7 +203,6 @@ ext/hash_map limits
 ext/hash_map new
 ext/hash_map stdexcept
 ext/hash_map string
-ext/hash_map type_traits
 ext/hash_set algorithm
 ext/hash_set cmath
 ext/hash_set cstddef
@@ -224,7 +213,6 @@ ext/hash_set initializer_list
 ext/hash_set limits
 ext/hash_set new
 ext/hash_set string
-ext/hash_set type_traits
 filesystem cerrno
 filesystem compare
 filesystem cstddef
@@ -240,7 +228,6 @@ filesystem ratio
 filesystem string
 filesystem string_view
 filesystem system_error
-filesystem type_traits
 filesystem version
 format array
 format charconv
@@ -269,7 +256,6 @@ forward_list limits
 forward_list new
 forward_list stdexcept
 forward_list tuple
-forward_list type_traits
 forward_list version
 fstream cctype
 fstream cstddef
@@ -286,12 +272,10 @@ fstream string
 fstream typeinfo
 fstream version
 functional array
-functional atomic
 functional cstddef
 functional cstdint
 functional cstdlib
 functional cstring
-functional exception
 functional initializer_list
 functional iosfwd
 functional limits
@@ -299,24 +283,28 @@ functional memory
 functional new
 functional stdexcept
 functional tuple
-functional type_traits
 functional typeinfo
 functional unordered_map
 functional vector
 functional version
 future cstddef
-future exception
+future cstdint
+future cstdlib
+future cstring
+future initializer_list
+future iosfwd
 future limits
 future mutex
 future new
 future ratio
+future stdexcept
 future system_error
 future thread
+future typeinfo
 future version
 initializer_list cstddef
 iomanip istream
 iomanip version
-ios atomic
 ios cctype
 ios cstddef
 ios cstdint
@@ -345,8 +333,13 @@ iterator iosfwd
 iterator limits
 iterator variant
 iterator version
-latch atomic
+latch cstddef
+latch cstdint
+latch cstring
+latch ctime
+latch iosfwd
 latch limits
+latch ratio
 latch version
 limits version
 list compare
@@ -359,7 +352,6 @@ list limits
 list new
 list stdexcept
 list tuple
-list type_traits
 list version
 locale cctype
 locale cstddef
@@ -386,9 +378,7 @@ map new
 map optional
 map stdexcept
 map tuple
-map type_traits
 map version
-memory atomic
 memory compare
 memory cstddef
 memory cstdint
@@ -410,11 +400,11 @@ memory_resource new
 memory_resource stdexcept
 memory_resource tuple
 memory_resource version
-mutex atomic
 mutex cstddef
 mutex cstdint
 mutex cstdlib
 mutex cstring
+mutex ctime
 mutex initializer_list
 mutex iosfwd
 mutex limits
@@ -427,7 +417,6 @@ mutex typeinfo
 mutex version
 new cstddef
 new cstdlib
-new exception
 new version
 numbers version
 numeric cmath
@@ -442,9 +431,7 @@ optional initializer_list
 optional limits
 optional new
 optional stdexcept
-optional type_traits
 optional version
-ostream atomic
 ostream bitset
 ostream cstddef
 ostream cstdint
@@ -466,7 +453,6 @@ queue cstdlib
 queue deque
 queue initializer_list
 queue limits
-queue type_traits
 queue vector
 queue version
 random cmath
@@ -478,7 +464,6 @@ random iosfwd
 random limits
 random numeric
 random string
-random type_traits
 random vector
 random version
 ranges compare
@@ -492,7 +477,6 @@ ranges new
 ranges optional
 ranges span
 ranges tuple
-ranges type_traits
 ranges variant
 ranges version
 ratio climits
@@ -517,8 +501,9 @@ scoped_allocator limits
 scoped_allocator new
 scoped_allocator tuple
 scoped_allocator version
-semaphore atomic
 semaphore cstddef
+semaphore cstdint
+semaphore cstring
 semaphore ctime
 semaphore iosfwd
 semaphore limits
@@ -533,7 +518,6 @@ set new
 set optional
 set stdexcept
 set tuple
-set type_traits
 set version
 shared_mutex cstddef
 shared_mutex ctime
@@ -548,7 +532,6 @@ span array
 span cstddef
 span initializer_list
 span limits
-span type_traits
 span version
 sstream cstddef
 sstream istream
@@ -559,10 +542,8 @@ stack compare
 stack cstddef
 stack deque
 stack initializer_list
-stack type_traits
 stack version
 stdexcept cstdlib
-stdexcept exception
 stdexcept iosfwd
 streambuf cstdint
 streambuf ios
@@ -583,7 +564,6 @@ string new
 string stdexcept
 string string_view
 string tuple
-string type_traits
 string version
 string_view compare
 string_view cstddef
@@ -596,7 +576,6 @@ string_view initializer_list
 string_view iosfwd
 string_view limits
 string_view stdexcept
-string_view type_traits
 string_view version
 strstream istream
 strstream ostream
@@ -609,7 +588,6 @@ system_error cstring
 system_error limits
 system_error stdexcept
 system_error string
-system_error type_traits
 system_error version
 thread compare
 thread cstddef
@@ -622,7 +600,6 @@ thread new
 thread ratio
 thread system_error
 thread tuple
-thread type_traits
 thread version
 tuple compare
 tuple cstddef
@@ -636,8 +613,6 @@ typeindex version
 typeinfo cstddef
 typeinfo cstdint
 typeinfo cstdlib
-typeinfo exception
-typeinfo type_traits
 unordered_map cmath
 unordered_map compare
 unordered_map cstddef
@@ -650,7 +625,6 @@ unordered_map new
 unordered_map optional
 unordered_map stdexcept
 unordered_map tuple
-unordered_map type_traits
 unordered_map version
 unordered_set cmath
 unordered_set compare
@@ -664,7 +638,6 @@ unordered_set new
 unordered_set optional
 unordered_set stdexcept
 unordered_set tuple
-unordered_set type_traits
 unordered_set version
 utility compare
 utility cstddef
@@ -673,6 +646,7 @@ utility limits
 utility version
 valarray cmath
 valarray cstddef
+valarray cstdint
 valarray cstdlib
 valarray initializer_list
 valarray limits
@@ -683,7 +657,6 @@ variant compare
 variant cstddef
 variant cstdint
 variant cstring
-variant exception
 variant initializer_list
 variant limits
 variant new
@@ -701,5 +674,4 @@ vector limits
 vector new
 vector stdexcept
 vector tuple
-vector type_traits
 vector version
diff --git a/libcxx/test/libcxx/type_traits/convert_to_integral.pass.cpp b/libcxx/test/libcxx/type_traits/convert_to_integral.pass.cpp
index 2972191221642..f1036b3929f06 100644
--- a/libcxx/test/libcxx/type_traits/convert_to_integral.pass.cpp
+++ b/libcxx/test/libcxx/type_traits/convert_to_integral.pass.cpp
@@ -94,12 +94,12 @@ int main(int, char**)
 #endif
   check_integral_types<char16_t, int>();
   // On some platforms, unsigned int and long are the same size.  These
-  // platforms have a choice of making uint32_t an int or a long.  However
+  // platforms have a choice of making std::uint32_t an int or a long.  However
   // char32_t must promote to an unsigned int on these platforms [conv.prom].
   // Use the following logic to make the test work on such platforms.
-  // (sizeof(uint32_t) == sizeof(unsigned int)) ? unsigned int : uint32_t;
-  typedef std::conditional<sizeof(uint32_t) == sizeof(unsigned int),
-                           unsigned int, uint32_t>::type char_integral;
+  // (sizeof(std::uint32_t) == sizeof(unsigned int)) ? unsigned int : std::uint32_t;
+  typedef std::conditional<sizeof(std::uint32_t) == sizeof(unsigned int),
+                           unsigned int, std::uint32_t>::type char_integral;
   check_integral_types<char32_t, char_integral>();
   check_integral_types<short, int>();
   check_integral_types<unsigned short, int>();
diff --git a/libcxx/test/libcxx/type_traits/is_always_bitcastable.compile.pass.cpp b/libcxx/test/libcxx/type_traits/is_always_bitcastable.compile.pass.cpp
index 3dc2da8f2d99a..9bbb85f2fe30c 100644
--- a/libcxx/test/libcxx/type_traits/is_always_bitcastable.compile.pass.cpp
+++ b/libcxx/test/libcxx/type_traits/is_always_bitcastable.compile.pass.cpp
@@ -71,7 +71,7 @@ constexpr void test() {
     // Bit-castable arithmetic types.
 
     // 8-bit types.
-    using integral_8 = types::type_list<char8_t, int8_t, uint8_t>;
+    using integral_8 = types::type_list<char8_t, std::int8_t, std::uint8_t>;
     using chars = types::type_list<char, unsigned char, signed char>;
 #if CHAR_BIT == 8
     check<true, types::concatenate_t<integral_8, chars>>();
@@ -81,7 +81,7 @@ constexpr void test() {
 #endif
 
     // 16-bit types.
-    using integral_16 = types::type_list<char16_t, int16_t, uint16_t>;
+    using integral_16 = types::type_list<char16_t, std::int16_t, std::uint16_t>;
 #if !defined(TEST_HAS_NO_WIDE_CHARACTERS) && __WCHAR_WIDTH__ == 16
     check<true, types::concatenate_t<integral_16, types::type_list<wchar_t>>>();
 #else
@@ -89,7 +89,7 @@ constexpr void test() {
 #endif
 
     // 32-bit types.
-    using integral_32 = types::type_list<char32_t, int32_t, uint32_t>;
+    using integral_32 = types::type_list<char32_t, std::int32_t, std::uint32_t>;
 #if !defined(TEST_HAS_NO_WIDE_CHARACTERS) && __WCHAR_WIDTH__ == 32
     check<true, types::concatenate_t<integral_32, types::type_list<wchar_t>>>();
 #else
@@ -97,7 +97,7 @@ constexpr void test() {
 #endif
 
     // 64-bit types.
-    using integral_64 = types::type_list<int64_t, uint64_t>;
+    using integral_64 = types::type_list<std::int64_t, std::uint64_t>;
     check<true, integral_64>();
 
     // 128-bit types.
diff --git a/libcxx/test/libcxx/utilities/format/format.arguments/format.arg/arg_t.compile.pass.cpp b/libcxx/test/libcxx/utilities/format/format.arguments/format.arg/arg_t.compile.pass.cpp
index 7f21f13f702b8..2775539ac8818 100644
--- a/libcxx/test/libcxx/utilities/format/format.arguments/format.arg/arg_t.compile.pass.cpp
+++ b/libcxx/test/libcxx/utilities/format/format.arguments/format.arg/arg_t.compile.pass.cpp
@@ -18,21 +18,21 @@
 
 #include "test_macros.h"
 
-static_assert(std::is_same_v<std::underlying_type_t<std::__format::__arg_t>, uint8_t>);
-
-static_assert(uint8_t(std::__format::__arg_t::__none) == 0);
-static_assert(uint8_t(std::__format::__arg_t::__boolean) == 1);
-static_assert(uint8_t(std::__format::__arg_t::__char_type) == 2);
-static_assert(uint8_t(std::__format::__arg_t::__int) == 3);
-static_assert(uint8_t(std::__format::__arg_t::__long_long) == 4);
-static_assert(uint8_t(std::__format::__arg_t::__i128) == 5);
-static_assert(uint8_t(std::__format::__arg_t::__unsigned) == 6);
-static_assert(uint8_t(std::__format::__arg_t::__unsigned_long_long) == 7);
-static_assert(uint8_t(std::__format::__arg_t::__u128) == 8);
-static_assert(uint8_t(std::__format::__arg_t::__float) == 9);
-static_assert(uint8_t(std::__format::__arg_t::__double) == 10);
-static_assert(uint8_t(std::__format::__arg_t::__long_double) == 11);
-static_assert(uint8_t(std::__format::__arg_t::__const_char_type_ptr) == 12);
-static_assert(uint8_t(std::__format::__arg_t::__string_view) == 13);
-static_assert(uint8_t(std::__format::__arg_t::__ptr) == 14);
-static_assert(uint8_t(std::__format::__arg_t::__handle) == 15);
+static_assert(std::is_same_v<std::underlying_type_t<std::__format::__arg_t>, std::uint8_t>);
+
+static_assert(std::uint8_t(std::__format::__arg_t::__none) == 0);
+static_assert(std::uint8_t(std::__format::__arg_t::__boolean) == 1);
+static_assert(std::uint8_t(std::__format::__arg_t::__char_type) == 2);
+static_assert(std::uint8_t(std::__format::__arg_t::__int) == 3);
+static_assert(std::uint8_t(std::__format::__arg_t::__long_long) == 4);
+static_assert(std::uint8_t(std::__format::__arg_t::__i128) == 5);
+static_assert(std::uint8_t(std::__format::__arg_t::__unsigned) == 6);
+static_assert(std::uint8_t(std::__format::__arg_t::__unsigned_long_long) == 7);
+static_assert(std::uint8_t(std::__format::__arg_t::__u128) == 8);
+static_assert(std::uint8_t(std::__format::__arg_t::__float) == 9);
+static_assert(std::uint8_t(std::__format::__arg_t::__double) == 10);
+static_assert(std::uint8_t(std::__format::__arg_t::__long_double) == 11);
+static_assert(std::uint8_t(std::__format::__arg_t::__const_char_type_ptr) == 12);
+static_assert(std::uint8_t(std::__format::__arg_t::__string_view) == 13);
+static_assert(std::uint8_t(std::__format::__arg_t::__ptr) == 14);
+static_assert(std::uint8_t(std::__format::__arg_t::__handle) == 15);
diff --git a/libcxx/test/libcxx/utilities/format/format.string/format.string.std/extended_grapheme_cluster.h b/libcxx/test/libcxx/utilities/format/format.string/format.string.std/extended_grapheme_cluster.h
index f28122255305a..cfe7f4eb394a1 100644
--- a/libcxx/test/libcxx/utilities/format/format.string/format.string.std/extended_grapheme_cluster.h
+++ b/libcxx/test/libcxx/utilities/format/format.string/format.string.std/extended_grapheme_cluster.h
@@ -8,7 +8,7 @@
 //===----------------------------------------------------------------------===//
 
 // WARNING, this entire header is generated by
-// utiles/generate_extended_grapheme_cluster_test.py
+// utils/generate_extended_grapheme_cluster_test.py
 // DO NOT MODIFY!
 
 // UNICODE, INC. LICENSE AGREEMENT - DATA FILES AND SOFTWARE
diff --git a/libcxx/test/libcxx/utilities/format/version.compile.pass.cpp b/libcxx/test/libcxx/utilities/format/version.compile.pass.cpp
index 4c1be774a9a6a..f4fa83951c146 100644
--- a/libcxx/test/libcxx/utilities/format/version.compile.pass.cpp
+++ b/libcxx/test/libcxx/utilities/format/version.compile.pass.cpp
@@ -17,6 +17,3 @@
 #ifndef _LIBCPP_VERSION
 #  error _LIBCPP_VERSION not defined
 #endif
-
-// Required for MSVC internal test runner compatibility.
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/libcxx/utilities/function.objects/unord.hash/murmur2_or_cityhash_ubsan_unsigned_overflow_ignored.pass.cpp b/libcxx/test/libcxx/utilities/function.objects/unord.hash/murmur2_or_cityhash_ubsan_unsigned_overflow_ignored.pass.cpp
index 1f295284b4a23..73ad7aa5d03cb 100644
--- a/libcxx/test/libcxx/utilities/function.objects/unord.hash/murmur2_or_cityhash_ubsan_unsigned_overflow_ignored.pass.cpp
+++ b/libcxx/test/libcxx/utilities/function.objects/unord.hash/murmur2_or_cityhash_ubsan_unsigned_overflow_ignored.pass.cpp
@@ -17,8 +17,8 @@
 
 #include "test_macros.h"
 
-typedef std::__murmur2_or_cityhash<uint32_t> Hash32;
-typedef std::__murmur2_or_cityhash<uint64_t> Hash64;
+typedef std::__murmur2_or_cityhash<std::uint32_t> Hash32;
+typedef std::__murmur2_or_cityhash<std::uint64_t> Hash64;
 
 void test(const void* key, int len) {
   for (int i=1; i <= len; ++i) {
diff --git a/libcxx/test/libcxx/utilities/utility/__murmur2_or_cityhash.abi-v1.pass.cpp b/libcxx/test/libcxx/utilities/utility/__murmur2_or_cityhash.abi-v1.pass.cpp
index 3f8557adb0364..cdaaacad282ef 100644
--- a/libcxx/test/libcxx/utilities/utility/__murmur2_or_cityhash.abi-v1.pass.cpp
+++ b/libcxx/test/libcxx/utilities/utility/__murmur2_or_cityhash.abi-v1.pass.cpp
@@ -33,14 +33,14 @@ std::string CityHash[] = {
 };
 
 int main(int, char**) {
-  const std::pair<std::string, uint64_t> TestCases[] = {
+  const std::pair<std::string, std::uint64_t> TestCases[] = {
       {CityHash[0], CHOOSE_BY_ENDIANESS(0x87c69099911bab7eULL, 0x297621d7fa436a3ULL)},
       {CityHash[1], CHOOSE_BY_ENDIANESS(0x87c69099911bab7eULL, 0xb17be531dde56e57ULL)},
       {CityHash[2], CHOOSE_BY_ENDIANESS(0x85322632e188694aULL, 0xe14f578b688e266dULL)},
       {CityHash[3], CHOOSE_BY_ENDIANESS(0x85322632e188694aULL, 0xca5a764a0450eac6ULL)},
   };
 
-  std::__murmur2_or_cityhash<uint64_t> h64;
+  std::__murmur2_or_cityhash<std::uint64_t> h64;
   for (const auto& test_case : TestCases) {
     assert(h64(test_case.first.data(), test_case.first.size()) == test_case.second);
   }
diff --git a/libcxx/test/libcxx/utilities/utility/__murmur2_or_cityhash.abi-v2.pass.cpp b/libcxx/test/libcxx/utilities/utility/__murmur2_or_cityhash.abi-v2.pass.cpp
index 01f8b88c81fe7..25c0b9f177dce 100644
--- a/libcxx/test/libcxx/utilities/utility/__murmur2_or_cityhash.abi-v2.pass.cpp
+++ b/libcxx/test/libcxx/utilities/utility/__murmur2_or_cityhash.abi-v2.pass.cpp
@@ -32,14 +32,14 @@ std::string CityHash[] = {
 };
 
 int main(int, char**) {
-  const std::pair<std::string, uint64_t> TestCases[] = {
+  const std::pair<std::string, std::uint64_t> TestCases[] = {
       {CityHash[0], CHOOSE_BY_ENDIANESS(0x4382a8d0fe8edb17ULL, 0xca84e809bef16fbcULL)},
       {CityHash[1], CHOOSE_BY_ENDIANESS(0xecefb080a6854061ULL, 0xd7feb824250272dcULL)},
       {CityHash[2], CHOOSE_BY_ENDIANESS(0x169ea3aebf908d6dULL, 0xea8cef3ca6f6e368ULL)},
       {CityHash[3], CHOOSE_BY_ENDIANESS(0xe18298a2760f09faULL, 0xf33a7700bb7a94a8ULL)},
   };
 
-  std::__murmur2_or_cityhash<uint64_t> h64;
+  std::__murmur2_or_cityhash<std::uint64_t> h64;
   for (const auto& test_case : TestCases) {
     assert(h64(test_case.first.data(), test_case.first.size()) == test_case.second);
   }
diff --git a/libcxx/test/libcxx/vendor/apple/system-install-properties.sh.cpp b/libcxx/test/libcxx/vendor/apple/system-install-properties.sh.cpp
index aaa9ad7a7d0e3..e155013ef3c02 100644
--- a/libcxx/test/libcxx/vendor/apple/system-install-properties.sh.cpp
+++ b/libcxx/test/libcxx/vendor/apple/system-install-properties.sh.cpp
@@ -15,9 +15,10 @@
 //
 // RUN: stat "%{include}/__config"
 
-// Make sure we install libc++.1.dylib in the right location.
+// Make sure we install libc++.1.dylib and libc++experimental.a in the right location.
 //
 // RUN: stat "%{lib}/libc++.1.dylib"
+// RUN: stat "%{lib}/libc++experimental.a"
 
 // Make sure we install a symlink from libc++.dylib to libc++.1.dylib.
 //
diff --git a/libcxx/test/libcxx/vendor/ibm/bad_function_call.pass.cpp b/libcxx/test/libcxx/vendor/ibm/bad_function_call.pass.cpp
new file mode 100644
index 0000000000000..2b684465650fa
--- /dev/null
+++ b/libcxx/test/libcxx/vendor/ibm/bad_function_call.pass.cpp
@@ -0,0 +1,45 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: target={{powerpc.*-ibm-aix.*}}
+// ADDITIONAL_COMPILE_FLAGS: -fvisibility-inlines-hidden
+
+// When there is a weak hidden symbol in user code and a strong definition
+// in the library, we test that the linker relies on the library version,
+// as the default weak resolution semantics don't favour weak local definitions
+// for XCOFF. This creates a conflict on std::bad_function_call, which is used
+// by the std::function template instantiated in main.
+#include <functional>
+#include "test_macros.h"
+#include "assert.h"
+
+void foo() {}
+
+void test_call() {
+  std::function<void()> r(foo);
+  r();
+}
+
+void test_throw() {
+#ifndef TEST_HAS_NO_EXCEPTIONS
+  std::function<int()> f;
+  try {
+    f();
+    assert(false);
+  } catch (const std::bad_function_call&) {
+    return;
+  }
+  assert(false);
+#endif // TEST_HAS_NO_EXCEPTIONS
+}
+
+int main(int, char**) {
+  test_call();
+  test_throw();
+  return 0;
+}
diff --git a/libcxx/test/std/algorithms/alg.sorting/alg.binary.search/binary.search/ranges.binary_search.pass.cpp b/libcxx/test/std/algorithms/alg.sorting/alg.binary.search/binary.search/ranges.binary_search.pass.cpp
index a1abdaff28b4c..cbeb7eb6fab53 100644
--- a/libcxx/test/std/algorithms/alg.sorting/alg.binary.search/binary.search/ranges.binary_search.pass.cpp
+++ b/libcxx/test/std/algorithms/alg.sorting/alg.binary.search/binary.search/ranges.binary_search.pass.cpp
@@ -131,6 +131,13 @@ constexpr void test_iterators() {
     auto range = std::ranges::subrange(It(a), Sent(It(a + 5)));
     assert(std::ranges::binary_search(range, 1));
   }
+
+  { // check that false is returned when the element doesn't exist, but an element with a greater value is in the range
+    int a[] = {1, 2, 4};
+    assert(!std::ranges::binary_search(It(a), Sent(It(a + 3)), 3));
+    auto range = std::ranges::subrange(It(a), Sent(It(a + 3)));
+    assert(!std::ranges::binary_search(range, 3));
+  }
 }
 
 constexpr bool test() {
diff --git a/libcxx/test/std/algorithms/alg.sorting/alg.binary.search/lower.bound/ranges.lower_bound.pass.cpp b/libcxx/test/std/algorithms/alg.sorting/alg.binary.search/lower.bound/ranges.lower_bound.pass.cpp
index e7af57bd0156f..cffce51334440 100644
--- a/libcxx/test/std/algorithms/alg.sorting/alg.binary.search/lower.bound/ranges.lower_bound.pass.cpp
+++ b/libcxx/test/std/algorithms/alg.sorting/alg.binary.search/lower.bound/ranges.lower_bound.pass.cpp
@@ -194,6 +194,22 @@ constexpr void test_iterators() {
       assert(base(ret) == a);
     }
   }
+
+  { // check that the middle of a range is returned when there are smaller and larger elements
+    {
+      int a[] = {1, 2, 3, 4, 6, 7, 8};
+      auto ret = std::ranges::lower_bound(It(a), It(a + 7), 5);
+      assert(base(ret) == a + 4);
+      assert(*ret == 6);
+    }
+    {
+      int a[] = {1, 2, 3, 4, 6, 7, 8};
+      auto range = std::ranges::subrange(It(a), It(a + 7));
+      auto ret = std::ranges::lower_bound(range, 5);
+      assert(base(ret) == a + 4);
+      assert(*ret == 6);
+    }
+  }
 }
 
 constexpr bool test() {
diff --git a/libcxx/test/std/algorithms/alg.sorting/alg.binary.search/upper.bound/ranges.upper_bound.pass.cpp b/libcxx/test/std/algorithms/alg.sorting/alg.binary.search/upper.bound/ranges.upper_bound.pass.cpp
index 762fd7808389c..cab7831c2b59b 100644
--- a/libcxx/test/std/algorithms/alg.sorting/alg.binary.search/upper.bound/ranges.upper_bound.pass.cpp
+++ b/libcxx/test/std/algorithms/alg.sorting/alg.binary.search/upper.bound/ranges.upper_bound.pass.cpp
@@ -193,6 +193,22 @@ constexpr void test_iterators() {
       assert(base(ret) == a + 1);
     }
   }
+
+  { // check that the middle of a range is returned when there are smaller and larger elements
+    {
+      int a[] = {1, 2, 3, 4, 6, 7, 8};
+      auto ret = std::ranges::upper_bound(It(a), It(a + 7), 5);
+      assert(base(ret) == a + 4);
+      assert(*ret == 6);
+    }
+    {
+      int a[] = {1, 2, 3, 4, 6, 7, 8};
+      auto range = std::ranges::subrange(It(a), It(a + 7));
+      auto ret = std::ranges::upper_bound(range, 5);
+      assert(base(ret) == a + 4);
+      assert(*ret == 6);
+    }
+  }
 }
 
 constexpr bool test() {
diff --git a/libcxx/test/libcxx/algorithms/alg.sorting/alg.heap.operations/sort.heap/complexity.pass.cpp b/libcxx/test/std/algorithms/alg.sorting/alg.heap.operations/sort.heap/complexity.pass.cpp
similarity index 63%
rename from libcxx/test/libcxx/algorithms/alg.sorting/alg.heap.operations/sort.heap/complexity.pass.cpp
rename to libcxx/test/std/algorithms/alg.sorting/alg.heap.operations/sort.heap/complexity.pass.cpp
index 0ed37a2b920e9..e3cb2339fddde 100644
--- a/libcxx/test/libcxx/algorithms/alg.sorting/alg.heap.operations/sort.heap/complexity.pass.cpp
+++ b/libcxx/test/std/algorithms/alg.sorting/alg.heap.operations/sort.heap/complexity.pass.cpp
@@ -21,8 +21,8 @@
 
 struct Stats {
   int compared = 0;
-  int copied = 0;
-  int moved = 0;
+  int copied   = 0;
+  int moved    = 0;
 } stats;
 
 struct MyInt {
@@ -46,30 +46,30 @@ struct MyInt {
   }
 };
 
-int main(int, char**)
-{
-  const int N = 100'000;
+int main(int, char**) {
+  constexpr int N = (1 << 20);
   std::vector<MyInt> v;
   v.reserve(N);
   std::mt19937 g;
-  for (int i = 0; i < N; ++i)
-    v.emplace_back(g());
-  std::make_heap(v.begin(), v.end());
-
-  // The exact stats of our current implementation are recorded here.
-  // If something changes to make them go a bit up or down, that's probably fine,
-  // and we can just update this test.
-  // But if they suddenly leap upward, that's a bad thing.
-
-  stats = {};
-  std::sort_heap(v.begin(), v.end());
-  assert(stats.copied == 0);
-  assert(stats.moved == 1'764'997);
+  for (int i = 0; i < N; ++i) {
+    v.emplace_back(i);
+  }
+  for (int logn = 10; logn <= 20; ++logn) {
+    const int n = (1 << logn);
+    auto first  = v.begin();
+    auto last   = v.begin() + n;
+    std::shuffle(first, last, g);
+    std::make_heap(first, last);
+    // The exact stats of our current implementation are recorded here.
+    stats = {};
+    std::sort_heap(first, last);
+    LIBCPP_ASSERT(stats.copied == 0);
+    LIBCPP_ASSERT(stats.moved <= 2 * n + n * logn);
 #ifndef _LIBCPP_ENABLE_DEBUG_MODE
-  assert(stats.compared == 1'534'701);
+    LIBCPP_ASSERT(stats.compared <= n * logn);
 #endif
-
-  assert(std::is_sorted(v.begin(), v.end()));
-
+    LIBCPP_ASSERT(std::is_sorted(first, last));
+    LIBCPP_ASSERT(stats.compared <= 2 * n * logn);
+  }
   return 0;
 }
diff --git a/libcxx/test/std/algorithms/alg.sorting/alg.heap.operations/sort.heap/ranges_sort_heap.pass.cpp b/libcxx/test/std/algorithms/alg.sorting/alg.heap.operations/sort.heap/ranges_sort_heap.pass.cpp
index faf1e2309eb67..abf317a753406 100644
--- a/libcxx/test/std/algorithms/alg.sorting/alg.heap.operations/sort.heap/ranges_sort_heap.pass.cpp
+++ b/libcxx/test/std/algorithms/alg.sorting/alg.heap.operations/sort.heap/ranges_sort_heap.pass.cpp
@@ -25,6 +25,7 @@
 #include <array>
 #include <concepts>
 #include <functional>
+#include <random>
 #include <ranges>
 
 #include "almost_satisfies_types.h"
@@ -212,9 +213,63 @@ constexpr bool test() {
   return true;
 }
 
+struct Stats {
+  int compared = 0;
+  int copied   = 0;
+  int moved    = 0;
+} stats;
+
+struct MyInt {
+  int value;
+  explicit MyInt(int xval) : value(xval) {}
+  MyInt(const MyInt& other) : value(other.value) { ++stats.copied; }
+  MyInt(MyInt&& other) : value(other.value) { ++stats.moved; }
+  MyInt& operator=(const MyInt& other) {
+    value = other.value;
+    ++stats.copied;
+    return *this;
+  }
+  MyInt& operator=(MyInt&& other) {
+    value = other.value;
+    ++stats.moved;
+    return *this;
+  }
+  static bool Comp(const MyInt& a, const MyInt& b) {
+    ++stats.compared;
+    return a.value < b.value;
+  }
+};
+
+void test_complexity() {
+  constexpr int N = (1 << 20);
+  std::vector<MyInt> v;
+  v.reserve(N);
+  std::mt19937 g;
+  for (int i = 0; i < N; ++i) {
+    v.emplace_back(i);
+  }
+  for (int logn = 10; logn <= 20; ++logn) {
+    const int n = (1 << logn);
+    auto first  = v.begin();
+    auto last   = v.begin() + n;
+    std::shuffle(first, last, g);
+    std::make_heap(first, last, &MyInt::Comp);
+    // The exact stats of our current implementation are recorded here.
+    stats = {};
+    std::ranges::sort_heap(first, last, &MyInt::Comp);
+    LIBCPP_ASSERT(stats.copied == 0);
+    LIBCPP_ASSERT(stats.moved <= 2 * n + n * logn);
+#ifndef _LIBCPP_ENABLE_DEBUG_MODE
+    LIBCPP_ASSERT(stats.compared <= n * logn);
+#endif
+    LIBCPP_ASSERT(std::is_sorted(first, last, &MyInt::Comp));
+    LIBCPP_ASSERT(stats.compared <= 2 * n * logn);
+  }
+}
+
 int main(int, char**) {
   test();
   static_assert(test());
-
+  test_complexity();
   return 0;
 }
diff --git a/libcxx/test/std/algorithms/alg.sorting/alg.min.max/ranges.max.pass.cpp b/libcxx/test/std/algorithms/alg.sorting/alg.min.max/ranges.max.pass.cpp
index 3e0d82daba7d8..e9e5240cf8cde 100644
--- a/libcxx/test/std/algorithms/alg.sorting/alg.min.max/ranges.max.pass.cpp
+++ b/libcxx/test/std/algorithms/alg.sorting/alg.min.max/ranges.max.pass.cpp
@@ -176,19 +176,38 @@ constexpr void test_initializer_list() {
 
 template <class It, class Sent = It>
 constexpr void test_range_types() {
-  int a[] = {7, 6, 9, 3, 5, 1, 2, 4};
+  std::iter_value_t<It> a[] = {7, 6, 9, 3, 5, 1, 2, 4};
   auto range = std::ranges::subrange(It(a), Sent(It(a + 8)));
-  int ret = std::ranges::max(range);
+  auto ret = std::ranges::max(range);
   assert(ret == 9);
 }
 
 constexpr void test_range() {
-  { // check that all range types work
-    test_range_types<cpp20_input_iterator<int*>, sentinel_wrapper<cpp20_input_iterator<int*>>>();
-    test_range_types<forward_iterator<int*>>();
-    test_range_types<bidirectional_iterator<int*>>();
-    test_range_types<random_access_iterator<int*>>();
-    test_range_types<contiguous_iterator<int*>>();
+  // check that all range types work
+  {
+    struct NonTrivialInt {
+      int val_;
+      constexpr NonTrivialInt(int val) : val_(val) {}
+      constexpr NonTrivialInt(const NonTrivialInt& other) : val_(other.val_) {}
+      constexpr NonTrivialInt& operator=(const NonTrivialInt& other) {
+        val_ = other.val_;
+        return *this;
+      }
+
+      constexpr ~NonTrivialInt() {}
+
+      auto operator<=>(const NonTrivialInt&) const = default;
+    };
+
+    auto call_with_sentinels = []<class Iter> {
+      if constexpr (std::forward_iterator<Iter>)
+        test_range_types<Iter, Iter>();
+      test_range_types<Iter, sentinel_wrapper<Iter>>();
+      test_range_types<Iter, sized_sentinel<Iter>>();
+    };
+
+    types::for_each(types::cpp20_input_iterator_list<int*>{}, call_with_sentinels);
+    types::for_each(types::cpp20_input_iterator_list<NonTrivialInt*>{}, call_with_sentinels);
   }
 
   int a[] = {7, 6, 9, 3, 5, 1, 2, 4};
diff --git a/libcxx/test/std/algorithms/alg.sorting/alg.min.max/ranges.min.pass.cpp b/libcxx/test/std/algorithms/alg.sorting/alg.min.max/ranges.min.pass.cpp
index a211fe6ed5269..3d92964c90daa 100644
--- a/libcxx/test/std/algorithms/alg.sorting/alg.min.max/ranges.min.pass.cpp
+++ b/libcxx/test/std/algorithms/alg.sorting/alg.min.max/ranges.min.pass.cpp
@@ -171,19 +171,38 @@ constexpr void test_initializer_list() {
 
 template <class It, class Sent = It>
 constexpr void test_range_types() {
-  int a[] = {7, 6, 9, 3, 5, 1, 2, 4};
+  std::iter_value_t<It> a[] = {7, 6, 9, 3, 5, 1, 2, 4};
   auto range = std::ranges::subrange(It(a), Sent(It(a + 8)));
-  int ret = std::ranges::min(range);
+  auto ret = std::ranges::min(range);
   assert(ret == 1);
 }
 
 constexpr void test_range() {
-  { // check that all range types work
-    test_range_types<cpp20_input_iterator<int*>, sentinel_wrapper<cpp20_input_iterator<int*>>>();
-    test_range_types<forward_iterator<int*>>();
-    test_range_types<bidirectional_iterator<int*>>();
-    test_range_types<random_access_iterator<int*>>();
-    test_range_types<contiguous_iterator<int*>>();
+  // check that all range types work
+  {
+    struct NonTrivialInt {
+      int val_;
+      constexpr NonTrivialInt(int val) : val_(val) {}
+      constexpr NonTrivialInt(const NonTrivialInt& other) : val_(other.val_) {}
+      constexpr NonTrivialInt& operator=(const NonTrivialInt& other) {
+        val_ = other.val_;
+        return *this;
+      }
+
+      constexpr ~NonTrivialInt() {}
+
+      auto operator<=>(const NonTrivialInt&) const = default;
+    };
+
+    auto call_with_sentinels = []<class Iter> {
+      if constexpr (std::forward_iterator<Iter>)
+        test_range_types<Iter, Iter>();
+      test_range_types<Iter, sentinel_wrapper<Iter>>();
+      test_range_types<Iter, sized_sentinel<Iter>>();
+    };
+
+    types::for_each(types::cpp20_input_iterator_list<int*>{}, call_with_sentinels);
+    types::for_each(types::cpp20_input_iterator_list<NonTrivialInt*>{}, call_with_sentinels);
   }
 
   int a[] = {7, 6, 9, 3, 5, 1, 2, 4};
diff --git a/libcxx/test/std/atomics/atomics.lockfree/isalwayslockfree.pass.cpp b/libcxx/test/std/atomics/atomics.lockfree/isalwayslockfree.pass.cpp
index de24bc2580ec6..b2d83f0a6fe88 100644
--- a/libcxx/test/std/atomics/atomics.lockfree/isalwayslockfree.pass.cpp
+++ b/libcxx/test/std/atomics/atomics.lockfree/isalwayslockfree.pass.cpp
@@ -14,6 +14,7 @@
 
 #include <atomic>
 #include <cassert>
+#include <cstddef>
 
 #include "test_macros.h"
 
diff --git a/libcxx/test/std/atomics/atomics.types.generic/integral.pass.cpp b/libcxx/test/std/atomics/atomics.types.generic/integral.pass.cpp
index 34559f0e1382c..058db2dc3ab04 100644
--- a/libcxx/test/std/atomics/atomics.types.generic/integral.pass.cpp
+++ b/libcxx/test/std/atomics/atomics.types.generic/integral.pass.cpp
@@ -84,8 +84,9 @@
 // };
 
 #include <atomic>
-#include <new>
 #include <cassert>
+#include <cstdint>
+#include <new>
 
 #include <cmpxchg_loop.h>
 
diff --git a/libcxx/test/std/atomics/atomics.types.generic/standard_layout.compile.pass.cpp b/libcxx/test/std/atomics/atomics.types.generic/standard_layout.compile.pass.cpp
index d7987ca2dc600..ca17657177f3c 100644
--- a/libcxx/test/std/atomics/atomics.types.generic/standard_layout.compile.pass.cpp
+++ b/libcxx/test/std/atomics/atomics.types.generic/standard_layout.compile.pass.cpp
@@ -25,10 +25,8 @@ struct CheckStandardLayout {
   }
 };
 
-int main(int, char**) {
+void tests() {
   TestEachIntegralType<CheckStandardLayout>()();
   TestEachFloatingPointType<CheckStandardLayout>()();
   TestEachPointerType<CheckStandardLayout>()();
-
-  return 0;
 }
diff --git a/libcxx/test/std/atomics/atomics.types.operations/atomics.types.operations.req/copy.assign.ptr.volatile.verify.cpp b/libcxx/test/std/atomics/atomics.types.operations/atomics.types.operations.req/copy.assign.ptr.volatile.verify.cpp
index 2d1acb623ed5a..d85f8a8443735 100644
--- a/libcxx/test/std/atomics/atomics.types.operations/atomics.types.operations.req/copy.assign.ptr.volatile.verify.cpp
+++ b/libcxx/test/std/atomics/atomics.types.operations/atomics.types.operations.req/copy.assign.ptr.volatile.verify.cpp
@@ -14,8 +14,7 @@
 
 #include <atomic>
 
-int main(int, char**)
-{
+void f() {
     volatile std::atomic<int*> obj1;
     std::atomic<int*> obj2;
     obj1 = obj2; // expected-error {{overload resolution selected deleted operator '='}}
diff --git a/libcxx/test/std/atomics/atomics.types.operations/atomics.types.operations.req/copy.assign.volatile.verify.cpp b/libcxx/test/std/atomics/atomics.types.operations/atomics.types.operations.req/copy.assign.volatile.verify.cpp
index 07e1181edf2bf..e72ccf1a26136 100644
--- a/libcxx/test/std/atomics/atomics.types.operations/atomics.types.operations.req/copy.assign.volatile.verify.cpp
+++ b/libcxx/test/std/atomics/atomics.types.operations/atomics.types.operations.req/copy.assign.volatile.verify.cpp
@@ -14,8 +14,7 @@
 
 #include <atomic>
 
-int main(int, char**)
-{
+void f() {
   volatile std::atomic<int> obj1;
   std::atomic<int> obj2;
   obj1 = obj2; // expected-error {{overload resolution selected deleted operator '='}}
diff --git a/libcxx/test/std/atomics/types.pass.cpp b/libcxx/test/std/atomics/types.pass.cpp
index f95f7a4c1d472..ee5cca2237745 100644
--- a/libcxx/test/std/atomics/types.pass.cpp
+++ b/libcxx/test/std/atomics/types.pass.cpp
@@ -119,32 +119,32 @@ int main(int, char**)
     test<wchar_t>            ();
 #endif
 
-    test<int_least8_t>   ();
-    test<uint_least8_t>  ();
-    test<int_least16_t>  ();
-    test<uint_least16_t> ();
-    test<int_least32_t>  ();
-    test<uint_least32_t> ();
-    test<int_least64_t>  ();
-    test<uint_least64_t> ();
-
-    test<int_fast8_t>   ();
-    test<uint_fast8_t>  ();
-    test<int_fast16_t>  ();
-    test<uint_fast16_t> ();
-    test<int_fast32_t>  ();
-    test<uint_fast32_t> ();
-    test<int_fast64_t>  ();
-    test<uint_fast64_t> ();
-
-    test< int8_t>  ();
-    test<uint8_t>  ();
-    test< int16_t> ();
-    test<uint16_t> ();
-    test< int32_t> ();
-    test<uint32_t> ();
-    test< int64_t> ();
-    test<uint64_t> ();
+    test<std::int_least8_t>   ();
+    test<std::uint_least8_t>  ();
+    test<std::int_least16_t>  ();
+    test<std::uint_least16_t> ();
+    test<std::int_least32_t>  ();
+    test<std::uint_least32_t> ();
+    test<std::int_least64_t>  ();
+    test<std::uint_least64_t> ();
+
+    test<std::int_fast8_t>   ();
+    test<std::uint_fast8_t>  ();
+    test<std::int_fast16_t>  ();
+    test<std::uint_fast16_t> ();
+    test<std::int_fast32_t>  ();
+    test<std::uint_fast32_t> ();
+    test<std::int_fast64_t>  ();
+    test<std::uint_fast64_t> ();
+
+    test< std::int8_t>  ();
+    test<std::uint8_t>  ();
+    test< std::int16_t> ();
+    test<std::uint16_t> ();
+    test< std::int32_t> ();
+    test<std::uint32_t> ();
+    test< std::int64_t> ();
+    test<std::uint64_t> ();
 
     test<intptr_t>  ();
     test<uintptr_t> ();
diff --git a/libcxx/test/std/concepts/concepts.compare/concept.equalitycomparable/equality_comparable.compile.pass.cpp b/libcxx/test/std/concepts/concepts.compare/concept.equalitycomparable/equality_comparable.compile.pass.cpp
index b58dae347db09..126606ef7ab3c 100644
--- a/libcxx/test/std/concepts/concepts.compare/concept.equalitycomparable/equality_comparable.compile.pass.cpp
+++ b/libcxx/test/std/concepts/concepts.compare/concept.equalitycomparable/equality_comparable.compile.pass.cpp
@@ -141,5 +141,3 @@ static_assert(std::equality_comparable<ge_returns_explicit_bool>);
 static_assert(std::equality_comparable<returns_true_type>);
 static_assert(std::equality_comparable<returns_int_ptr>);
 } // namespace types_fit_for_purpose
-
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/std/concepts/concepts.compare/concept.equalitycomparable/equality_comparable_with.compile.pass.cpp b/libcxx/test/std/concepts/concepts.compare/concept.equalitycomparable/equality_comparable_with.compile.pass.cpp
index ba8138f667f9d..e0edd1f332f81 100644
--- a/libcxx/test/std/concepts/concepts.compare/concept.equalitycomparable/equality_comparable_with.compile.pass.cpp
+++ b/libcxx/test/std/concepts/concepts.compare/concept.equalitycomparable/equality_comparable_with.compile.pass.cpp
@@ -1120,5 +1120,3 @@ static_assert(
 static_assert(
     !check_equality_comparable_with<one_way_ne, explicit_operators>());
 } // namespace types_fit_for_purpose
-
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/std/concepts/concepts.lang/concept.common/common_with.compile.pass.cpp b/libcxx/test/std/concepts/concepts.lang/concept.common/common_with.compile.pass.cpp
index c14634e79466d..5d80c1cba2655 100644
--- a/libcxx/test/std/concepts/concepts.lang/concept.common/common_with.compile.pass.cpp
+++ b/libcxx/test/std/concepts/concepts.lang/concept.common/common_with.compile.pass.cpp
@@ -990,5 +990,3 @@ struct common_type<const volatile int&, const volatile CommonWithIntButRefLong&>
     : common_type<CommonWithIntButRefLong&, int&> {};
 } // namespace std
 static_assert(CheckCommonWith<CommonWithIntButRefLong, int>());
-
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/std/concepts/concepts.lang/concept.commonref/common_reference.compile.pass.cpp b/libcxx/test/std/concepts/concepts.lang/concept.commonref/common_reference.compile.pass.cpp
index e7ab52025af12..7c37dafc2d691 100644
--- a/libcxx/test/std/concepts/concepts.lang/concept.commonref/common_reference.compile.pass.cpp
+++ b/libcxx/test/std/concepts/concepts.lang/concept.commonref/common_reference.compile.pass.cpp
@@ -343,5 +343,3 @@ struct basic_common_reference<int, HasCommonReference, X, Y> {
 } // namespace std
 static_assert(!std::common_reference_with<HasCommonReference, int>);
 static_assert(std::common_reference_with<HasCommonReference, int&>);
-
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/std/concepts/concepts.lang/concept.constructible/constructible_from.compile.pass.cpp b/libcxx/test/std/concepts/concepts.lang/concept.constructible/constructible_from.compile.pass.cpp
index 22a70bd7d6531..fe0ecece3382f 100644
--- a/libcxx/test/std/concepts/concepts.lang/concept.constructible/constructible_from.compile.pass.cpp
+++ b/libcxx/test/std/concepts/concepts.lang/concept.constructible/constructible_from.compile.pass.cpp
@@ -148,5 +148,3 @@ void test() {
   test<std::array<int, 1>, int>();
   test<std::array<int, 1>, int, int>();
 }
-
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/std/concepts/concepts.lang/concept.copyconstructible/copy_constructible.compile.pass.cpp b/libcxx/test/std/concepts/concepts.lang/concept.copyconstructible/copy_constructible.compile.pass.cpp
index cc349bef18483..4a14155eddff7 100644
--- a/libcxx/test/std/concepts/concepts.lang/concept.copyconstructible/copy_constructible.compile.pass.cpp
+++ b/libcxx/test/std/concepts/concepts.lang/concept.copyconstructible/copy_constructible.compile.pass.cpp
@@ -172,5 +172,3 @@ struct ExplicitlyCopyable {
 };
 static_assert(!std::copy_constructible<ExplicitlyCopyable>);
 } // namespace CopyConstructibleTests
-
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/std/concepts/concepts.lang/concept.default.init/default_initializable.compile.pass.cpp b/libcxx/test/std/concepts/concepts.lang/concept.default.init/default_initializable.compile.pass.cpp
index 340c29fe1b9c3..4921a48bcccc1 100644
--- a/libcxx/test/std/concepts/concepts.lang/concept.default.init/default_initializable.compile.pass.cpp
+++ b/libcxx/test/std/concepts/concepts.lang/concept.default.init/default_initializable.compile.pass.cpp
@@ -261,8 +261,3 @@ void test()
     test_true     <std::weak_ptr<int>>();
 
 }
-
-// Required for MSVC internal test runner compatibility.
-int main(int, char**) {
-    return 0;
-}
diff --git a/libcxx/test/std/concepts/concepts.lang/concept.default.init/default_initializable.verify.cpp b/libcxx/test/std/concepts/concepts.lang/concept.default.init/default_initializable.verify.cpp
index ec788eef1fe28..88a75d030c734 100644
--- a/libcxx/test/std/concepts/concepts.lang/concept.default.init/default_initializable.verify.cpp
+++ b/libcxx/test/std/concepts/concepts.lang/concept.default.init/default_initializable.verify.cpp
@@ -65,9 +65,3 @@ void test() {
     LIBCPP_STATIC_ASSERT(!std::__default_initializable<const int[]>);
     static_assert(!std::default_initializable<const int[]>);
 }
-
-int main(int, char**) {
-    test();
-
-    return 0;
-}
diff --git a/libcxx/test/std/concepts/concepts.lang/concept.destructible/destructible.compile.pass.cpp b/libcxx/test/std/concepts/concepts.lang/concept.destructible/destructible.compile.pass.cpp
index aa86f59efc78d..10c61336f67a7 100644
--- a/libcxx/test/std/concepts/concepts.lang/concept.destructible/destructible.compile.pass.cpp
+++ b/libcxx/test/std/concepts/concepts.lang/concept.destructible/destructible.compile.pass.cpp
@@ -73,6 +73,3 @@ void test() {
   test<int>();
   test<double>();
 }
-
-// Required for MSVC internal test runner compatibility.
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/std/concepts/concepts.lang/concept.moveconstructible/move_constructible.compile.pass.cpp b/libcxx/test/std/concepts/concepts.lang/concept.moveconstructible/move_constructible.compile.pass.cpp
index b4003d9cc69c6..d388248c05ac8 100644
--- a/libcxx/test/std/concepts/concepts.lang/concept.moveconstructible/move_constructible.compile.pass.cpp
+++ b/libcxx/test/std/concepts/concepts.lang/concept.moveconstructible/move_constructible.compile.pass.cpp
@@ -76,5 +76,3 @@ static_assert(std::move_constructible<const ImplicitlyDeletedMoveAssign&&>);
 static_assert(!std::move_constructible<NonMovable>);
 static_assert(!std::move_constructible<DerivedFromNonMovable>);
 static_assert(!std::move_constructible<HasANonMovable>);
-
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/std/concepts/concepts.object/copyable.compile.pass.cpp b/libcxx/test/std/concepts/concepts.object/copyable.compile.pass.cpp
index a92b6c4fe0072..accd6141dae84 100644
--- a/libcxx/test/std/concepts/concepts.object/copyable.compile.pass.cpp
+++ b/libcxx/test/std/concepts/concepts.object/copyable.compile.pass.cpp
@@ -113,5 +113,3 @@ static_assert(
     !std::assignable_from<deleted_assignment_from_const_rvalue&,
                           deleted_assignment_from_const_rvalue const>);
 static_assert(!std::copyable<deleted_assignment_from_const_rvalue>);
-
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/std/concepts/concepts.object/movable.compile.pass.cpp b/libcxx/test/std/concepts/concepts.object/movable.compile.pass.cpp
index c5879beec3df8..07a518d7fe1f3 100644
--- a/libcxx/test/std/concepts/concepts.object/movable.compile.pass.cpp
+++ b/libcxx/test/std/concepts/concepts.object/movable.compile.pass.cpp
@@ -133,5 +133,3 @@ static_assert(std::movable<deleted_assignment_from_const_rvalue>);
 
 // `move_constructible and assignable_from<T&, T>` implies `swappable<T>`,
 // so there's nothing to test for the case of non-swappable.
-
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/std/concepts/concepts.object/regular.compile.pass.cpp b/libcxx/test/std/concepts/concepts.object/regular.compile.pass.cpp
index b1a1709f51f54..ab5472f6c391d 100644
--- a/libcxx/test/std/concepts/concepts.object/regular.compile.pass.cpp
+++ b/libcxx/test/std/concepts/concepts.object/regular.compile.pass.cpp
@@ -163,5 +163,3 @@ struct is_equality_comparable {
   bool operator==(is_equality_comparable const&) const = default;
 };
 static_assert(std::regular<is_equality_comparable>);
-
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/std/concepts/concepts.object/semiregular.compile.pass.cpp b/libcxx/test/std/concepts/concepts.object/semiregular.compile.pass.cpp
index fed5f03f275bf..7151c1c8bb31d 100644
--- a/libcxx/test/std/concepts/concepts.object/semiregular.compile.pass.cpp
+++ b/libcxx/test/std/concepts/concepts.object/semiregular.compile.pass.cpp
@@ -126,5 +126,3 @@ static_assert(!std::semiregular<has_non_default_initializable>);
 static_assert(!std::semiregular<deleted_default_ctor>);
 static_assert(!std::semiregular<derived_from_deleted_default_ctor>);
 static_assert(!std::semiregular<has_deleted_default_ctor>);
-
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/std/containers/associative/map/map.access/empty.verify.cpp b/libcxx/test/std/containers/associative/map/map.access/empty.verify.cpp
index 1430591fdf6d1..27ca7c66ccebf 100644
--- a/libcxx/test/std/containers/associative/map/map.access/empty.verify.cpp
+++ b/libcxx/test/std/containers/associative/map/map.access/empty.verify.cpp
@@ -16,12 +16,7 @@
 
 #include <map>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::map<int, int> c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/containers/associative/multimap/empty.verify.cpp b/libcxx/test/std/containers/associative/multimap/empty.verify.cpp
index dcd3b58e0283a..8720497c35fdf 100644
--- a/libcxx/test/std/containers/associative/multimap/empty.verify.cpp
+++ b/libcxx/test/std/containers/associative/multimap/empty.verify.cpp
@@ -16,12 +16,7 @@
 
 #include <map>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::multimap<int, int> c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/containers/associative/multiset/empty.verify.cpp b/libcxx/test/std/containers/associative/multiset/empty.verify.cpp
index 87c02e4581d10..0de93052ad795 100644
--- a/libcxx/test/std/containers/associative/multiset/empty.verify.cpp
+++ b/libcxx/test/std/containers/associative/multiset/empty.verify.cpp
@@ -16,12 +16,7 @@
 
 #include <set>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::multiset<int> c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/containers/associative/set/empty.verify.cpp b/libcxx/test/std/containers/associative/set/empty.verify.cpp
index 0e7b28da152b9..e0405f70e67da 100644
--- a/libcxx/test/std/containers/associative/set/empty.verify.cpp
+++ b/libcxx/test/std/containers/associative/set/empty.verify.cpp
@@ -16,12 +16,7 @@
 
 #include <set>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::set<int> c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/containers/container.adaptors/container.adaptors.format/format.functions.tests.h b/libcxx/test/std/containers/container.adaptors/container.adaptors.format/format.functions.tests.h
index ecec920986e55..584ac2b2d8d10 100644
--- a/libcxx/test/std/containers/container.adaptors/container.adaptors.format/format.functions.tests.h
+++ b/libcxx/test/std/containers/container.adaptors/container.adaptors.format/format.functions.tests.h
@@ -250,6 +250,14 @@ void test_char(TestFunction check, ExceptionTest check_exception) {
   test_char_escaped_string<CharT>(
       check, check_exception, std::priority_queue{input.begin(), input.end(), std::greater{}});
   test_char_escaped_string<CharT>(check, check_exception, std::stack{input.begin(), input.end()});
+
+  // LWG3881 fixes formatting container adaptors backed by a std::string.
+  test_char_default<CharT>(check, check_exception, std::queue{std::basic_string<CharT>{input.begin(), input.end()}});
+  test_char_default<CharT>(
+      check,
+      check_exception,
+      std::priority_queue{std::greater{}, std::basic_string<CharT>{input.begin(), input.end()}});
+  test_char_default<CharT>(check, check_exception, std::stack{std::basic_string<CharT>{input.begin(), input.end()}});
 }
 
 //
diff --git a/libcxx/test/std/containers/container.adaptors/priority.queue/priqueue.members/empty.verify.cpp b/libcxx/test/std/containers/container.adaptors/priority.queue/priqueue.members/empty.verify.cpp
index 7976b96cd76af..793f47d12c20a 100644
--- a/libcxx/test/std/containers/container.adaptors/priority.queue/priqueue.members/empty.verify.cpp
+++ b/libcxx/test/std/containers/container.adaptors/priority.queue/priqueue.members/empty.verify.cpp
@@ -16,12 +16,7 @@
 
 #include <queue>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::priority_queue<int> c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/containers/container.adaptors/queue/queue.defn/empty.verify.cpp b/libcxx/test/std/containers/container.adaptors/queue/queue.defn/empty.verify.cpp
index 9a8761a51992a..5f24ef0e77fc9 100644
--- a/libcxx/test/std/containers/container.adaptors/queue/queue.defn/empty.verify.cpp
+++ b/libcxx/test/std/containers/container.adaptors/queue/queue.defn/empty.verify.cpp
@@ -16,12 +16,7 @@
 
 #include <queue>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::queue<int> c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/containers/container.adaptors/stack/stack.defn/empty.verify.cpp b/libcxx/test/std/containers/container.adaptors/stack/stack.defn/empty.verify.cpp
index 8264092b5fce4..e9af144c858b1 100644
--- a/libcxx/test/std/containers/container.adaptors/stack/stack.defn/empty.verify.cpp
+++ b/libcxx/test/std/containers/container.adaptors/stack/stack.defn/empty.verify.cpp
@@ -16,12 +16,7 @@
 
 #include <stack>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::stack<int> c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/containers/sequences/array/array.special/swap.pass.cpp b/libcxx/test/std/containers/sequences/array/array.special/swap.pass.cpp
index 617641df35ee7..94974295dfc10 100644
--- a/libcxx/test/std/containers/sequences/array/array.special/swap.pass.cpp
+++ b/libcxx/test/std/containers/sequences/array/array.special/swap.pass.cpp
@@ -12,6 +12,7 @@
 
 #include <array>
 #include <cassert>
+#include <utility>
 
 #include "test_macros.h"
 
diff --git a/libcxx/test/std/containers/sequences/array/contiguous.pass.cpp b/libcxx/test/std/containers/sequences/array/contiguous.pass.cpp
index 9589e63643941..fec48b325e90e 100644
--- a/libcxx/test/std/containers/sequences/array/contiguous.pass.cpp
+++ b/libcxx/test/std/containers/sequences/array/contiguous.pass.cpp
@@ -12,6 +12,7 @@
 
 #include <array>
 #include <cassert>
+#include <memory>
 
 #include "test_macros.h"
 
diff --git a/libcxx/test/std/containers/sequences/array/empty.verify.cpp b/libcxx/test/std/containers/sequences/array/empty.verify.cpp
index 6fa851f178c82..9cf8f1156990f 100644
--- a/libcxx/test/std/containers/sequences/array/empty.verify.cpp
+++ b/libcxx/test/std/containers/sequences/array/empty.verify.cpp
@@ -16,15 +16,9 @@
 
 #include <array>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
-
+void f() {
     std::array<int, 1> c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
     std::array<int, 0> c0;
     c0.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/containers/sequences/deque/abi.compile.pass.cpp b/libcxx/test/std/containers/sequences/deque/abi.compile.pass.cpp
index 74cbfe3ffaec2..36a76b2e11d61 100644
--- a/libcxx/test/std/containers/sequences/deque/abi.compile.pass.cpp
+++ b/libcxx/test/std/containers/sequences/deque/abi.compile.pass.cpp
@@ -14,7 +14,7 @@
 
 template <class T>
 class small_pointer {
-  uint16_t offset;
+  std::uint16_t offset;
 };
 
 template <class T>
@@ -22,8 +22,8 @@ class small_iter_allocator {
 public:
   using value_type      = T;
   using pointer         = small_pointer<T>;
-  using size_type       = int16_t;
-  using difference_type = int16_t;
+  using size_type       = std::int16_t;
+  using difference_type = std::int16_t;
 
   small_iter_allocator() TEST_NOEXCEPT {}
 
diff --git a/libcxx/test/std/containers/sequences/deque/deque.capacity/empty.verify.cpp b/libcxx/test/std/containers/sequences/deque/deque.capacity/empty.verify.cpp
index 01822e28f84d8..a2295088c0a9b 100644
--- a/libcxx/test/std/containers/sequences/deque/deque.capacity/empty.verify.cpp
+++ b/libcxx/test/std/containers/sequences/deque/deque.capacity/empty.verify.cpp
@@ -16,12 +16,7 @@
 
 #include <deque>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::deque<int> c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/containers/sequences/forwardlist/empty.verify.cpp b/libcxx/test/std/containers/sequences/forwardlist/empty.verify.cpp
index 4092f443f3874..366094b09e885 100644
--- a/libcxx/test/std/containers/sequences/forwardlist/empty.verify.cpp
+++ b/libcxx/test/std/containers/sequences/forwardlist/empty.verify.cpp
@@ -16,12 +16,7 @@
 
 #include <forward_list>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::forward_list<int> c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/containers/sequences/list/list.capacity/empty.verify.cpp b/libcxx/test/std/containers/sequences/list/list.capacity/empty.verify.cpp
index 4edf426c6eee3..77d4e5ab21387 100644
--- a/libcxx/test/std/containers/sequences/list/list.capacity/empty.verify.cpp
+++ b/libcxx/test/std/containers/sequences/list/list.capacity/empty.verify.cpp
@@ -16,12 +16,7 @@
 
 #include <list>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::list<int> c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/containers/sequences/vector.bool/empty.verify.cpp b/libcxx/test/std/containers/sequences/vector.bool/empty.verify.cpp
index 06c968e59c847..167e095b798f4 100644
--- a/libcxx/test/std/containers/sequences/vector.bool/empty.verify.cpp
+++ b/libcxx/test/std/containers/sequences/vector.bool/empty.verify.cpp
@@ -16,12 +16,7 @@
 
 #include <vector>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::vector<bool> c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/containers/sequences/vector/vector.capacity/empty.verify.cpp b/libcxx/test/std/containers/sequences/vector/vector.capacity/empty.verify.cpp
index df3b4960343e7..4cea844f1e357 100644
--- a/libcxx/test/std/containers/sequences/vector/vector.capacity/empty.verify.cpp
+++ b/libcxx/test/std/containers/sequences/vector/vector.capacity/empty.verify.cpp
@@ -16,12 +16,7 @@
 
 #include <vector>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::vector<int> c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/containers/sequences/vector/vector.cons/construct_iter_iter.pass.cpp b/libcxx/test/std/containers/sequences/vector/vector.cons/construct_iter_iter.pass.cpp
index 44364023c4e8e..5d71b8a20d1a1 100644
--- a/libcxx/test/std/containers/sequences/vector/vector.cons/construct_iter_iter.pass.cpp
+++ b/libcxx/test/std/containers/sequences/vector/vector.cons/construct_iter_iter.pass.cpp
@@ -190,8 +190,8 @@ TEST_CONSTEXPR_CXX20 void test_ctor_with_different_value_type() {
   }
   {
     // Though the types are different, initialization can be done with `memcpy`.
-    int32_t array[1] = { -1 };
-    std::vector<uint32_t> v(array, array + 1);
+    std::int32_t array[1] = { -1 };
+    std::vector<std::uint32_t> v(array, array + 1);
     assert(v[0] == 4294967295U);
   }
 }
diff --git a/libcxx/test/std/containers/sequences/vector/vector.cons/copy.move_only.verify.cpp b/libcxx/test/std/containers/sequences/vector/vector.cons/copy.move_only.verify.cpp
index 913010435d11e..f716eccf94e36 100644
--- a/libcxx/test/std/containers/sequences/vector/vector.cons/copy.move_only.verify.cpp
+++ b/libcxx/test/std/containers/sequences/vector/vector.cons/copy.move_only.verify.cpp
@@ -14,9 +14,7 @@
 
 #include "MoveOnly.h"
 
-int main(int, char**)
-{
+void f() {
     std::vector<MoveOnly> v;
     std::vector<MoveOnly> copy = v; // expected-error-re@* {{{{(no matching function for call to 'construct_at')|(call to implicitly-deleted copy constructor of 'MoveOnly')|(call to deleted constructor of 'MoveOnly')}}}}
-    return 0;
 }
diff --git a/libcxx/test/std/containers/unord/unord.map/empty.verify.cpp b/libcxx/test/std/containers/unord/unord.map/empty.verify.cpp
index 269250be9f650..ada96fdf89157 100644
--- a/libcxx/test/std/containers/unord/unord.map/empty.verify.cpp
+++ b/libcxx/test/std/containers/unord/unord.map/empty.verify.cpp
@@ -16,12 +16,7 @@
 
 #include <unordered_map>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::unordered_map<int, int> c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/containers/unord/unord.multimap/empty.verify.cpp b/libcxx/test/std/containers/unord/unord.multimap/empty.verify.cpp
index aead051866cb7..999d43fb63bc0 100644
--- a/libcxx/test/std/containers/unord/unord.multimap/empty.verify.cpp
+++ b/libcxx/test/std/containers/unord/unord.multimap/empty.verify.cpp
@@ -16,12 +16,7 @@
 
 #include <unordered_map>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::unordered_multimap<int, int> c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/containers/unord/unord.multiset/empty.verify.cpp b/libcxx/test/std/containers/unord/unord.multiset/empty.verify.cpp
index 547dd81b77481..8f5fa55c3aefb 100644
--- a/libcxx/test/std/containers/unord/unord.multiset/empty.verify.cpp
+++ b/libcxx/test/std/containers/unord/unord.multiset/empty.verify.cpp
@@ -16,12 +16,7 @@
 
 #include <unordered_set>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::unordered_multiset<int> c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/containers/unord/unord.set/empty.verify.cpp b/libcxx/test/std/containers/unord/unord.set/empty.verify.cpp
index 4a739a37a6493..e2164415aa182 100644
--- a/libcxx/test/std/containers/unord/unord.set/empty.verify.cpp
+++ b/libcxx/test/std/containers/unord/unord.set/empty.verify.cpp
@@ -16,12 +16,7 @@
 
 #include <unordered_set>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::unordered_set<int> c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/containers/views/views.span/span.cons/iterator_len.verify.cpp b/libcxx/test/std/containers/views/views.span/span.cons/iterator_len.verify.cpp
index 0372e00b789be..dc287cbe83114 100644
--- a/libcxx/test/std/containers/views/views.span/span.cons/iterator_len.verify.cpp
+++ b/libcxx/test/std/containers/views/views.span/span.cons/iterator_len.verify.cpp
@@ -22,7 +22,7 @@ std::span<T, extent> createImplicitSpan(T* ptr, size_t len) {
   return {ptr, len}; // expected-error {{chosen constructor is explicit in copy-initialization}}
 }
 
-int main(int, char**) {
+void f() {
   // explicit constructor necessary
   int arr[] = {1, 2, 3};
   createImplicitSpan<int, 1>(arr, 3);
@@ -31,6 +31,4 @@ int main(int, char**) {
   std::span<int, 2> sp2 = {0, 0}; // expected-error {{no matching constructor for initialization of 'std::span<int, 2>'}}
   std::span<const int> csp = {0, 0}; // expected-error {{no matching constructor for initialization of 'std::span<const int>'}}
   std::span<const int, 2> csp2 = {0, 0}; // expected-error {{no matching constructor for initialization of 'std::span<const int, 2>'}}
-
-  return 0;
 }
diff --git a/libcxx/test/std/containers/views/views.span/span.cons/iterator_sentinel.pass.cpp b/libcxx/test/std/containers/views/views.span/span.cons/iterator_sentinel.pass.cpp
index f744af789a8c7..335d2b850f6df 100644
--- a/libcxx/test/std/containers/views/views.span/span.cons/iterator_sentinel.pass.cpp
+++ b/libcxx/test/std/containers/views/views.span/span.cons/iterator_sentinel.pass.cpp
@@ -13,18 +13,22 @@
 // constexpr explicit(Extent != dynamic_extent) span(It first, End last);
 // Requires: [first, last) shall be a valid range.
 //   If Extent is not equal to dynamic_extent, then last - first shall be equal to Extent.
-//
+// Throws: When and what last - first throws.
 
+#include <array>
 #include <span>
 #include <cassert>
+#include <utility>
 
+#include "assert_macros.h"
 #include "test_iterators.h"
+#include "test_macros.h"
 
 template <class T, class Sentinel>
 constexpr bool test_ctor() {
   T val[2] = {};
-  auto s1 = std::span<T>(std::begin(val), Sentinel(std::end(val)));
-  auto s2 = std::span<T, 2>(std::begin(val), Sentinel(std::end(val)));
+  auto s1  = std::span<T>(std::begin(val), Sentinel(std::end(val)));
+  auto s2  = std::span<T, 2>(std::begin(val), Sentinel(std::end(val)));
   assert(s1.data() == std::data(val) && s1.size() == std::size(val));
   assert(s2.data() == std::data(val) && s2.size() == std::size(val));
   return true;
@@ -55,8 +59,85 @@ constexpr bool test() {
   return true;
 }
 
+#ifndef TEST_HAS_NO_EXCEPTIONS
+// A stripped down contiguous iterator that throws when using operator-.
+template <class It>
+class throw_operator_minus {
+  It it_;
+
+public:
+  typedef std::contiguous_iterator_tag iterator_category;
+  typedef typename std::iterator_traits<It>::value_type value_type;
+  typedef typename std::iterator_traits<It>::difference_type difference_type;
+  typedef It pointer;
+  typedef typename std::iterator_traits<It>::reference reference;
+  typedef typename std::remove_pointer<It>::type element_type;
+
+  throw_operator_minus() : it_() {}
+  explicit throw_operator_minus(It it) : it_(it) {}
+
+  reference operator*() const { return *it_; }
+  pointer operator->() const { return it_; }
+  reference operator[](difference_type n) const { return it_[n]; }
+
+  throw_operator_minus& operator++() {
+    ++it_;
+    return *this;
+  }
+  throw_operator_minus& operator--() {
+    --it_;
+    return *this;
+  }
+  throw_operator_minus operator++(int) { return throw_operator_minus(it_++); }
+  throw_operator_minus operator--(int) { return throw_operator_minus(it_--); }
+
+  throw_operator_minus& operator+=(difference_type n) {
+    it_ += n;
+    return *this;
+  }
+  throw_operator_minus& operator-=(difference_type n) {
+    it_ -= n;
+    return *this;
+  }
+  friend throw_operator_minus operator+(throw_operator_minus x, difference_type n) {
+    x += n;
+    return x;
+  }
+  friend throw_operator_minus operator+(difference_type n, throw_operator_minus x) {
+    x += n;
+    return x;
+  }
+  friend throw_operator_minus operator-(throw_operator_minus x, difference_type n) {
+    x -= n;
+    return x;
+  }
+  friend difference_type operator-(throw_operator_minus, throw_operator_minus) { throw 42; };
+
+  friend bool operator==(const throw_operator_minus& x, const throw_operator_minus& y) { return x.it_ == y.it_; }
+  friend bool operator<=>(const throw_operator_minus& x, const throw_operator_minus& y) { return x.it_ <=> y.it_; }
+};
+
+template <class It>
+throw_operator_minus(It) -> throw_operator_minus<It>;
+
+void test_exceptions() {
+  std::array a{42};
+  TEST_VALIDATE_EXCEPTION(
+      int,
+      [](int i) { assert(i == 42); },
+      (std::span<int>{throw_operator_minus{a.begin()}, throw_operator_minus{a.end()}}));
+  TEST_VALIDATE_EXCEPTION(
+      int,
+      [](int i) { assert(i == 42); },
+      (std::span<int, 1>{throw_operator_minus{a.begin()}, throw_operator_minus{a.end()}}));
+}
+#endif // TEST_HAS_NO_EXCEPTIONS
+
 int main(int, char**) {
   test();
+#ifndef TEST_HAS_NO_EXCEPTIONS
+  test_exceptions();
+#endif
   static_assert(test());
 
   return 0;
diff --git a/libcxx/test/std/containers/views/views.span/span.cons/iterator_sentinel.verify.cpp b/libcxx/test/std/containers/views/views.span/span.cons/iterator_sentinel.verify.cpp
index 9ac9a03da6ea6..d0d81df313d37 100644
--- a/libcxx/test/std/containers/views/views.span/span.cons/iterator_sentinel.verify.cpp
+++ b/libcxx/test/std/containers/views/views.span/span.cons/iterator_sentinel.verify.cpp
@@ -18,16 +18,13 @@
 #include <span>
 #include <iterator>
 
-
 template<class T, size_t Extent>
 std::span<T, Extent> createImplicitSpan(T* first, T* last) {
     return {first, last}; // expected-error {{chosen constructor is explicit in copy-initialization}}
 }
 
-int main(int, char**) {
+void f() {
   // explicit constructor necessary
   int arr[] = {1, 2, 3};
   createImplicitSpan<int, 1>(std::begin(arr), std::end(arr));
-
-  return 0;
 }
diff --git a/libcxx/test/std/depr/depr.lib.binders/depr.lib.bind.1st/bind1st.depr_in_cxx11.verify.cpp b/libcxx/test/std/depr/depr.lib.binders/depr.lib.bind.1st/bind1st.depr_in_cxx11.verify.cpp
index 93cfbb60a7d79..00ae30908da08 100644
--- a/libcxx/test/std/depr/depr.lib.binders/depr.lib.bind.1st/bind1st.depr_in_cxx11.verify.cpp
+++ b/libcxx/test/std/depr/depr.lib.binders/depr.lib.bind.1st/bind1st.depr_in_cxx11.verify.cpp
@@ -16,11 +16,7 @@
 #include <functional>
 
 #include "../test_func.h"
-#include "test_macros.h"
 
-int main(int, char**)
-{
+void f() {
     std::bind1st(test_func(1), 5); // expected-warning {{'bind1st<test_func, int>' is deprecated}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/depr/depr.lib.binders/depr.lib.bind.2nd/bind2nd.depr_in_cxx11.verify.cpp b/libcxx/test/std/depr/depr.lib.binders/depr.lib.bind.2nd/bind2nd.depr_in_cxx11.verify.cpp
index 3e55db60572dc..c1c78ecc62415 100644
--- a/libcxx/test/std/depr/depr.lib.binders/depr.lib.bind.2nd/bind2nd.depr_in_cxx11.verify.cpp
+++ b/libcxx/test/std/depr/depr.lib.binders/depr.lib.bind.2nd/bind2nd.depr_in_cxx11.verify.cpp
@@ -16,11 +16,7 @@
 #include <functional>
 
 #include "../test_func.h"
-#include "test_macros.h"
 
-int main(int, char**)
-{
+void f() {
     std::bind2nd(test_func(1), 5); // expected-warning {{'bind2nd<test_func, int>' is deprecated}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/depr/depr.lib.binders/depr.lib.binder.1st/binder1st.depr_in_cxx11.verify.cpp b/libcxx/test/std/depr/depr.lib.binders/depr.lib.binder.1st/binder1st.depr_in_cxx11.verify.cpp
index 5048d7b2ea1ef..c1b7c687a1962 100644
--- a/libcxx/test/std/depr/depr.lib.binders/depr.lib.binder.1st/binder1st.depr_in_cxx11.verify.cpp
+++ b/libcxx/test/std/depr/depr.lib.binders/depr.lib.binder.1st/binder1st.depr_in_cxx11.verify.cpp
@@ -16,11 +16,5 @@
 #include <functional>
 
 #include "../test_func.h"
-#include "test_macros.h"
 
-int main(int, char**)
-{
-    typedef std::binder1st<test_func> B1ST; // expected-warning {{'binder1st<test_func>' is deprecated}}
-
-    return 0;
-}
+typedef std::binder1st<test_func> B1ST; // expected-warning {{'binder1st<test_func>' is deprecated}}
diff --git a/libcxx/test/std/depr/depr.lib.binders/depr.lib.binder.2nd/binder2nd.depr_in_cxx11.verify.cpp b/libcxx/test/std/depr/depr.lib.binders/depr.lib.binder.2nd/binder2nd.depr_in_cxx11.verify.cpp
index 5a883ab040f47..fa25fa0e6826d 100644
--- a/libcxx/test/std/depr/depr.lib.binders/depr.lib.binder.2nd/binder2nd.depr_in_cxx11.verify.cpp
+++ b/libcxx/test/std/depr/depr.lib.binders/depr.lib.binder.2nd/binder2nd.depr_in_cxx11.verify.cpp
@@ -16,11 +16,5 @@
 #include <functional>
 
 #include "../test_func.h"
-#include "test_macros.h"
 
-int main(int, char**)
-{
-    typedef std::binder2nd<test_func> B2ND; // expected-warning {{'binder2nd<test_func>' is deprecated}}
-
-    return 0;
-}
+typedef std::binder2nd<test_func> B2ND; // expected-warning {{'binder2nd<test_func>' is deprecated}}
diff --git a/libcxx/test/std/experimental/simd/simd.abi/vector_extension.pass.cpp b/libcxx/test/std/experimental/simd/simd.abi/vector_extension.pass.cpp
index 35a45381a0156..e9182760ecec2 100644
--- a/libcxx/test/std/experimental/simd/simd.abi/vector_extension.pass.cpp
+++ b/libcxx/test/std/experimental/simd/simd.abi/vector_extension.pass.cpp
@@ -42,26 +42,26 @@ static_assert(
     sizeof(ex::simd<char, ex::__simd_abi<ex::_StorageKind::_VecExt, 12>>) == 16,
     "");
 static_assert(
-    sizeof(ex::simd<int32_t, ex::__simd_abi<ex::_StorageKind::_VecExt, 3>>) ==
+    sizeof(ex::simd<std::int32_t, ex::__simd_abi<ex::_StorageKind::_VecExt, 3>>) ==
         16,
     "");
 static_assert(
-    sizeof(ex::simd<int32_t, ex::__simd_abi<ex::_StorageKind::_VecExt, 5>>) ==
+    sizeof(ex::simd<std::int32_t, ex::__simd_abi<ex::_StorageKind::_VecExt, 5>>) ==
         32,
     "");
 static_assert(
-    std::is_same<ex::simd_abi::native<int8_t>,
+    std::is_same<ex::simd_abi::native<std::int8_t>,
                  ex::__simd_abi<ex::_StorageKind::_VecExt, reg_width()>>::value,
     "");
 #else
 static_assert(
-    std::is_same<ex::simd_abi::native<int8_t>,
+    std::is_same<ex::simd_abi::native<std::int8_t>,
                  ex::__simd_abi<ex::_StorageKind::_Array, reg_width()>>::value,
     "");
 
 #endif
 
-static_assert(std::is_same<ex::simd_abi::compatible<int8_t>,
+static_assert(std::is_same<ex::simd_abi::compatible<std::int8_t>,
                            ex::__simd_abi<ex::_StorageKind::_Array, 16>>::value,
               "");
 
diff --git a/libcxx/test/std/experimental/simd/simd.access/default.pass.cpp b/libcxx/test/std/experimental/simd/simd.access/default.pass.cpp
index 2ca5ec4482bf8..a8ec36c4f615b 100644
--- a/libcxx/test/std/experimental/simd/simd.access/default.pass.cpp
+++ b/libcxx/test/std/experimental/simd/simd.access/default.pass.cpp
@@ -26,7 +26,7 @@ template <typename SimdType>
 void test_access() {
   {
     SimdType a(42), b(4);
-    static_assert(std::is_convertible<decltype(a[0]), int8_t>::value, "");
+    static_assert(std::is_convertible<decltype(a[0]), std::int8_t>::value, "");
 
     assert(a[0] == 42);
     assert(!a[0] == !42);
diff --git a/libcxx/test/std/experimental/simd/simd.casts/simd_cast.pass.cpp b/libcxx/test/std/experimental/simd/simd.casts/simd_cast.pass.cpp
index 038af4be058ce..3a58f9f770b22 100644
--- a/libcxx/test/std/experimental/simd/simd.casts/simd_cast.pass.cpp
+++ b/libcxx/test/std/experimental/simd/simd.casts/simd_cast.pass.cpp
@@ -22,26 +22,26 @@
 namespace ex = std::experimental::parallelism_v2;
 
 static_assert(
-    std::is_same<decltype(ex::simd_cast<int32_t>(ex::native_simd<int32_t>())),
-                 ex::native_simd<int32_t>>::value,
+    std::is_same<decltype(ex::simd_cast<std::int32_t>(ex::native_simd<std::int32_t>())),
+                 ex::native_simd<std::int32_t>>::value,
     "");
 
-static_assert(std::is_same<decltype(ex::simd_cast<int64_t>(
-                               ex::fixed_size_simd<int32_t, 4>())),
-                           ex::fixed_size_simd<int64_t, 4>>::value,
+static_assert(std::is_same<decltype(ex::simd_cast<std::int64_t>(
+                               ex::fixed_size_simd<std::int32_t, 4>())),
+                           ex::fixed_size_simd<std::int64_t, 4>>::value,
               "");
 
 static_assert(
-    std::is_same<decltype(ex::simd_cast<ex::fixed_size_simd<int64_t, 1>>(
-                     ex::simd<int32_t, ex::simd_abi::scalar>())),
-                 ex::fixed_size_simd<int64_t, 1>>::value,
+    std::is_same<decltype(ex::simd_cast<ex::fixed_size_simd<std::int64_t, 1>>(
+                     ex::simd<std::int32_t, ex::simd_abi::scalar>())),
+                 ex::fixed_size_simd<std::int64_t, 1>>::value,
     "");
 
 static_assert(
     std::is_same<
-        decltype(ex::simd_cast<ex::simd<int64_t, ex::simd_abi::scalar>>(
-            ex::fixed_size_simd<int32_t, 1>())),
-        ex::simd<int64_t, ex::simd_abi::scalar>>::value,
+        decltype(ex::simd_cast<ex::simd<std::int64_t, ex::simd_abi::scalar>>(
+            ex::fixed_size_simd<std::int32_t, 1>())),
+        ex::simd<std::int64_t, ex::simd_abi::scalar>>::value,
     "");
 
 int main(int, char**) {
diff --git a/libcxx/test/std/experimental/simd/simd.cons/broadcast.pass.cpp b/libcxx/test/std/experimental/simd/simd.cons/broadcast.pass.cpp
index 94f9cb7d7b64e..02dd00379691d 100644
--- a/libcxx/test/std/experimental/simd/simd.cons/broadcast.pass.cpp
+++ b/libcxx/test/std/experimental/simd/simd.cons/broadcast.pass.cpp
@@ -37,37 +37,37 @@ template <class T>
 void supported_native_simd_ctor(...) = delete;
 
 void compile_narrowing_conversion() {
-  supported_native_simd_ctor<int8_t>(3);
-  supported_native_simd_ctor<int16_t>(3);
-  supported_native_simd_ctor<int32_t>(3);
-  supported_native_simd_ctor<int64_t>(3);
-  supported_native_simd_ctor<uint8_t>(3);
-  supported_native_simd_ctor<uint16_t>(3);
-  supported_native_simd_ctor<uint32_t>(3);
-  supported_native_simd_ctor<uint64_t>(3);
+  supported_native_simd_ctor<std::int8_t>(3);
+  supported_native_simd_ctor<std::int16_t>(3);
+  supported_native_simd_ctor<std::int32_t>(3);
+  supported_native_simd_ctor<std::int64_t>(3);
+  supported_native_simd_ctor<std::uint8_t>(3);
+  supported_native_simd_ctor<std::uint16_t>(3);
+  supported_native_simd_ctor<std::uint32_t>(3);
+  supported_native_simd_ctor<std::uint64_t>(3);
   supported_native_simd_ctor<float>(3.f);
   supported_native_simd_ctor<double>(3.);
   supported_native_simd_ctor<long double>(3.);
 
   not_supported_native_simd_ctor<float>(3.);
-  not_supported_native_simd_ctor<int8_t>(long(3));
+  not_supported_native_simd_ctor<std::int8_t>(long(3));
   not_supported_native_simd_ctor<float>(long(3));
   not_supported_native_simd_ctor<int>(3.);
 }
 
 void compile_convertible() {
   struct ConvertibleToInt {
-    operator int64_t() const;
+    operator std::int64_t() const;
   };
-  supported_native_simd_ctor<int64_t>(ConvertibleToInt());
+  supported_native_simd_ctor<std::int64_t>(ConvertibleToInt());
 
   struct NotConvertibleToInt {};
-  not_supported_native_simd_ctor<int64_t>(NotConvertibleToInt());
+  not_supported_native_simd_ctor<std::int64_t>(NotConvertibleToInt());
 }
 
 void compile_unsigned() {
   not_supported_native_simd_ctor<int>(3u);
-  supported_native_simd_ctor<uint16_t>(3u);
+  supported_native_simd_ctor<std::uint16_t>(3u);
 }
 
 template <typename SimdType>
diff --git a/libcxx/test/std/experimental/simd/simd.cons/default.pass.cpp b/libcxx/test/std/experimental/simd/simd.cons/default.pass.cpp
index feba23ee871b9..753917e1bcbe9 100644
--- a/libcxx/test/std/experimental/simd/simd.cons/default.pass.cpp
+++ b/libcxx/test/std/experimental/simd/simd.cons/default.pass.cpp
@@ -21,10 +21,10 @@
 namespace ex = std::experimental::parallelism_v2;
 
 int main(int, char**) {
-  static_assert(ex::native_simd<int32_t>().size() > 0, "");
-  static_assert(ex::fixed_size_simd<int32_t, 4>().size() == 4, "");
-  static_assert(ex::fixed_size_simd<int32_t, 5>().size() == 5, "");
-  static_assert(ex::fixed_size_simd<int32_t, 1>().size() == 1, "");
+  static_assert(ex::native_simd<std::int32_t>().size() > 0, "");
+  static_assert(ex::fixed_size_simd<std::int32_t, 4>().size() == 4, "");
+  static_assert(ex::fixed_size_simd<std::int32_t, 5>().size() == 5, "");
+  static_assert(ex::fixed_size_simd<std::int32_t, 1>().size() == 1, "");
   static_assert(ex::fixed_size_simd<char, 32>().size() == 32, "");
 
   return 0;
diff --git a/libcxx/test/std/experimental/simd/simd.cons/generator.pass.cpp b/libcxx/test/std/experimental/simd/simd.cons/generator.pass.cpp
index ace829fb7597a..5049d9b0de9a0 100644
--- a/libcxx/test/std/experimental/simd/simd.cons/generator.pass.cpp
+++ b/libcxx/test/std/experimental/simd/simd.cons/generator.pass.cpp
@@ -53,15 +53,15 @@ void compile_generator() {
 
 struct limited_identity {
   template <size_t value>
-  typename std::conditional<value <= 2, int32_t, int64_t>::type
+  typename std::conditional<value <= 2, std::int32_t, std::int64_t>::type
   operator()(std::integral_constant<size_t, value>) const {
     return value;
   }
 };
 
 void compile_limited_identity() {
-  supported_simd128_ctor<int64_t>(limited_identity());
-  not_supported_simd128_ctor<int32_t>(limited_identity());
+  supported_simd128_ctor<std::int64_t>(limited_identity());
+  not_supported_simd128_ctor<std::int32_t>(limited_identity());
 }
 
 template <typename SimdType>
@@ -84,9 +84,9 @@ void test_generator() {
 
 int main(int, char**) {
   // TODO: adjust the tests when this assertion fails.
-  assert(ex::native_simd<int32_t>::size() >= 4);
-  test_generator<ex::native_simd<int32_t>>();
-  test_generator<ex::fixed_size_simd<int32_t, 4>>();
+  assert(ex::native_simd<std::int32_t>::size() >= 4);
+  test_generator<ex::native_simd<std::int32_t>>();
+  test_generator<ex::fixed_size_simd<std::int32_t, 4>>();
 
   return 0;
 }
diff --git a/libcxx/test/std/experimental/simd/simd.cons/load.pass.cpp b/libcxx/test/std/experimental/simd/simd.cons/load.pass.cpp
index c4b352acf9808..ff29e7150e872 100644
--- a/libcxx/test/std/experimental/simd/simd.cons/load.pass.cpp
+++ b/libcxx/test/std/experimental/simd/simd.cons/load.pass.cpp
@@ -38,13 +38,13 @@ void supported_native_simd_ctor(...) = delete;
 
 void compile_load_ctor() {
   supported_native_simd_ctor<int>((int*)nullptr, ex::element_aligned_tag());
-  supported_native_simd_ctor<uint32_t>((int*)nullptr,
+  supported_native_simd_ctor<std::uint32_t>((int*)nullptr,
                                        ex::element_aligned_tag());
   supported_native_simd_ctor<double>((float*)nullptr,
                                      ex::element_aligned_tag());
-  supported_native_simd_ctor<uint16_t>((unsigned int*)nullptr,
+  supported_native_simd_ctor<std::uint16_t>((unsigned int*)nullptr,
                                        ex::element_aligned_tag());
-  supported_native_simd_ctor<uint32_t>((float*)nullptr,
+  supported_native_simd_ctor<std::uint32_t>((float*)nullptr,
                                        ex::element_aligned_tag());
 
   not_supported_native_simd_ctor<int>((int*)nullptr, int());
@@ -52,7 +52,7 @@ void compile_load_ctor() {
 
 template <typename SimdType>
 void test_load_ctor() {
-  alignas(32) int32_t buffer[] = {4, 3, 2, 1};
+  alignas(32) std::int32_t buffer[] = {4, 3, 2, 1};
   {
     SimdType a(buffer, ex::element_aligned_tag());
     assert(a[0] == 4);
@@ -110,11 +110,11 @@ void test_converting_load_ctor() {
 
 int main(int, char**) {
   // TODO: adjust the tests when this assertion fails.
-  assert(ex::native_simd<int32_t>::size() >= 4);
-  test_load_ctor<ex::native_simd<int32_t>>();
-  test_load_ctor<ex::fixed_size_simd<int32_t, 4>>();
-  test_converting_load_ctor<ex::native_simd<int32_t>>();
-  test_converting_load_ctor<ex::fixed_size_simd<int32_t, 4>>();
+  assert(ex::native_simd<std::int32_t>::size() >= 4);
+  test_load_ctor<ex::native_simd<std::int32_t>>();
+  test_load_ctor<ex::fixed_size_simd<std::int32_t, 4>>();
+  test_converting_load_ctor<ex::native_simd<std::int32_t>>();
+  test_converting_load_ctor<ex::fixed_size_simd<std::int32_t, 4>>();
 
   return 0;
 }
diff --git a/libcxx/test/std/experimental/simd/simd.mem/load.pass.cpp b/libcxx/test/std/experimental/simd/simd.mem/load.pass.cpp
index a8881176d5c36..c036d4f81db9c 100644
--- a/libcxx/test/std/experimental/simd/simd.mem/load.pass.cpp
+++ b/libcxx/test/std/experimental/simd/simd.mem/load.pass.cpp
@@ -39,17 +39,17 @@ void supported_load(...) = delete;
 
 void compile_load() {
   supported_load<int>((int*)nullptr, ex::element_aligned_tag());
-  supported_load<uint32_t>((int*)nullptr, ex::element_aligned_tag());
+  supported_load<std::uint32_t>((int*)nullptr, ex::element_aligned_tag());
   supported_load<double>((float*)nullptr, ex::element_aligned_tag());
-  supported_load<uint16_t>((unsigned int*)nullptr, ex::element_aligned_tag());
-  supported_load<uint32_t>((float*)nullptr, ex::element_aligned_tag());
+  supported_load<std::uint16_t>((unsigned int*)nullptr, ex::element_aligned_tag());
+  supported_load<std::uint32_t>((float*)nullptr, ex::element_aligned_tag());
 
   not_supported_load<int>((int*)nullptr, int());
 }
 
 template <typename SimdType>
 void test_load() {
-  alignas(32) int32_t buffer[] = {4, 3, 2, 1};
+  alignas(32) std::int32_t buffer[] = {4, 3, 2, 1};
   {
     SimdType a;
     a.copy_from(buffer, ex::element_aligned_tag());
@@ -114,11 +114,11 @@ void test_converting_load() {
 
 int main(int, char**) {
   // TODO: adjust the tests when this assertion fails.
-  assert(ex::native_simd<int32_t>::size() >= 4);
-  test_load<ex::native_simd<int32_t>>();
-  test_load<ex::fixed_size_simd<int32_t, 4>>();
-  test_converting_load<ex::native_simd<int32_t>>();
-  test_converting_load<ex::fixed_size_simd<int32_t, 4>>();
+  assert(ex::native_simd<std::int32_t>::size() >= 4);
+  test_load<ex::native_simd<std::int32_t>>();
+  test_load<ex::fixed_size_simd<std::int32_t, 4>>();
+  test_converting_load<ex::native_simd<std::int32_t>>();
+  test_converting_load<ex::fixed_size_simd<std::int32_t, 4>>();
 
   return 0;
 }
diff --git a/libcxx/test/std/experimental/simd/simd.mem/store.pass.cpp b/libcxx/test/std/experimental/simd/simd.mem/store.pass.cpp
index d3ead6094bb55..932ecfc5faeb2 100644
--- a/libcxx/test/std/experimental/simd/simd.mem/store.pass.cpp
+++ b/libcxx/test/std/experimental/simd/simd.mem/store.pass.cpp
@@ -25,7 +25,7 @@ template <typename SimdType>
 void test_store() {
   SimdType a([](int i) { return 4 - i; });
   {
-    alignas(32) int32_t buffer[4] = {0};
+    alignas(32) std::int32_t buffer[4] = {0};
     a.copy_to(buffer, ex::element_aligned_tag());
     assert(buffer[0] == 4);
     assert(buffer[1] == 3);
@@ -33,7 +33,7 @@ void test_store() {
     assert(buffer[3] == 1);
   }
   {
-    alignas(32) int32_t buffer[4] = {0};
+    alignas(32) std::int32_t buffer[4] = {0};
     a.copy_to(buffer, ex::vector_aligned_tag());
     assert(buffer[0] == 4);
     assert(buffer[1] == 3);
@@ -41,7 +41,7 @@ void test_store() {
     assert(buffer[3] == 1);
   }
   {
-    alignas(32) int32_t buffer[4] = {0};
+    alignas(32) std::int32_t buffer[4] = {0};
     a.copy_to(buffer, ex::overaligned_tag<32>());
     assert(buffer[0] == 4);
     assert(buffer[1] == 3);
@@ -50,7 +50,7 @@ void test_store() {
   }
 
   {
-    alignas(32) int32_t buffer[4] = {0};
+    alignas(32) std::int32_t buffer[4] = {0};
     a.copy_to(buffer, ex::element_aligned);
     assert(buffer[0] == 4);
     assert(buffer[1] == 3);
@@ -58,7 +58,7 @@ void test_store() {
     assert(buffer[3] == 1);
   }
   {
-    alignas(32) int32_t buffer[4] = {0};
+    alignas(32) std::int32_t buffer[4] = {0};
     a.copy_to(buffer, ex::vector_aligned);
     assert(buffer[0] == 4);
     assert(buffer[1] == 3);
@@ -66,7 +66,7 @@ void test_store() {
     assert(buffer[3] == 1);
   }
   {
-    alignas(32) int32_t buffer[4] = {0};
+    alignas(32) std::int32_t buffer[4] = {0};
     a.copy_to(buffer, ex::overaligned<32>);
     assert(buffer[0] == 4);
     assert(buffer[1] == 3);
@@ -88,10 +88,10 @@ void test_converting_store() {
 
 int main(int, char**) {
   // TODO: adjust the tests when this assertion fails.
-  test_store<ex::native_simd<int32_t>>();
-  test_store<ex::fixed_size_simd<int32_t, 4>>();
-  test_converting_store<ex::native_simd<int32_t>>();
-  test_converting_store<ex::fixed_size_simd<int32_t, 4>>();
+  test_store<ex::native_simd<std::int32_t>>();
+  test_store<ex::fixed_size_simd<std::int32_t, 4>>();
+  test_converting_store<ex::native_simd<std::int32_t>>();
+  test_converting_store<ex::fixed_size_simd<std::int32_t, 4>>();
 
   return 0;
 }
diff --git a/libcxx/test/std/experimental/simd/simd.traits/is_abi_tag.pass.cpp b/libcxx/test/std/experimental/simd/simd.traits/is_abi_tag.pass.cpp
index cfe816772604a..4d0883be45160 100644
--- a/libcxx/test/std/experimental/simd/simd.traits/is_abi_tag.pass.cpp
+++ b/libcxx/test/std/experimental/simd/simd.traits/is_abi_tag.pass.cpp
@@ -23,25 +23,25 @@ namespace ex = std::experimental::parallelism_v2;
 
 struct UserType {};
 
-static_assert(ex::is_abi_tag<ex::simd_abi::native<int8_t>>::value, "");
-static_assert(ex::is_abi_tag<ex::simd_abi::native<int16_t>>::value, "");
-static_assert(ex::is_abi_tag<ex::simd_abi::native<int32_t>>::value, "");
-static_assert(ex::is_abi_tag<ex::simd_abi::native<int64_t>>::value, "");
-static_assert(ex::is_abi_tag<ex::simd_abi::native<uint8_t>>::value, "");
-static_assert(ex::is_abi_tag<ex::simd_abi::native<uint16_t>>::value, "");
-static_assert(ex::is_abi_tag<ex::simd_abi::native<uint32_t>>::value, "");
-static_assert(ex::is_abi_tag<ex::simd_abi::native<uint64_t>>::value, "");
+static_assert(ex::is_abi_tag<ex::simd_abi::native<std::int8_t>>::value, "");
+static_assert(ex::is_abi_tag<ex::simd_abi::native<std::int16_t>>::value, "");
+static_assert(ex::is_abi_tag<ex::simd_abi::native<std::int32_t>>::value, "");
+static_assert(ex::is_abi_tag<ex::simd_abi::native<std::int64_t>>::value, "");
+static_assert(ex::is_abi_tag<ex::simd_abi::native<std::uint8_t>>::value, "");
+static_assert(ex::is_abi_tag<ex::simd_abi::native<std::uint16_t>>::value, "");
+static_assert(ex::is_abi_tag<ex::simd_abi::native<std::uint32_t>>::value, "");
+static_assert(ex::is_abi_tag<ex::simd_abi::native<std::uint64_t>>::value, "");
 static_assert(ex::is_abi_tag<ex::simd_abi::native<float>>::value, "");
 static_assert(ex::is_abi_tag<ex::simd_abi::native<double>>::value, "");
 
-static_assert(ex::is_abi_tag<ex::simd_abi::compatible<int8_t>>::value, "");
-static_assert(ex::is_abi_tag<ex::simd_abi::compatible<int16_t>>::value, "");
-static_assert(ex::is_abi_tag<ex::simd_abi::compatible<int32_t>>::value, "");
-static_assert(ex::is_abi_tag<ex::simd_abi::compatible<int64_t>>::value, "");
-static_assert(ex::is_abi_tag<ex::simd_abi::compatible<uint8_t>>::value, "");
-static_assert(ex::is_abi_tag<ex::simd_abi::compatible<uint16_t>>::value, "");
-static_assert(ex::is_abi_tag<ex::simd_abi::compatible<uint32_t>>::value, "");
-static_assert(ex::is_abi_tag<ex::simd_abi::compatible<uint64_t>>::value, "");
+static_assert(ex::is_abi_tag<ex::simd_abi::compatible<std::int8_t>>::value, "");
+static_assert(ex::is_abi_tag<ex::simd_abi::compatible<std::int16_t>>::value, "");
+static_assert(ex::is_abi_tag<ex::simd_abi::compatible<std::int32_t>>::value, "");
+static_assert(ex::is_abi_tag<ex::simd_abi::compatible<std::int64_t>>::value, "");
+static_assert(ex::is_abi_tag<ex::simd_abi::compatible<std::uint8_t>>::value, "");
+static_assert(ex::is_abi_tag<ex::simd_abi::compatible<std::uint16_t>>::value, "");
+static_assert(ex::is_abi_tag<ex::simd_abi::compatible<std::uint32_t>>::value, "");
+static_assert(ex::is_abi_tag<ex::simd_abi::compatible<std::uint64_t>>::value, "");
 static_assert(ex::is_abi_tag<ex::simd_abi::compatible<float>>::value, "");
 static_assert(ex::is_abi_tag<ex::simd_abi::compatible<double>>::value, "");
 
@@ -66,25 +66,25 @@ static_assert(!ex::is_abi_tag<ex::simd<float>>::value, "");
 static_assert(!ex::is_abi_tag<ex::simd_mask<int>>::value, "");
 static_assert(!ex::is_abi_tag<ex::simd_mask<float>>::value, "");
 
-static_assert(ex::is_abi_tag_v<ex::simd_abi::native<int8_t>>, "");
-static_assert(ex::is_abi_tag_v<ex::simd_abi::native<int16_t>>, "");
-static_assert(ex::is_abi_tag_v<ex::simd_abi::native<int32_t>>, "");
-static_assert(ex::is_abi_tag_v<ex::simd_abi::native<int64_t>>, "");
-static_assert(ex::is_abi_tag_v<ex::simd_abi::native<uint8_t>>, "");
-static_assert(ex::is_abi_tag_v<ex::simd_abi::native<uint16_t>>, "");
-static_assert(ex::is_abi_tag_v<ex::simd_abi::native<uint32_t>>, "");
-static_assert(ex::is_abi_tag_v<ex::simd_abi::native<uint64_t>>, "");
+static_assert(ex::is_abi_tag_v<ex::simd_abi::native<std::int8_t>>, "");
+static_assert(ex::is_abi_tag_v<ex::simd_abi::native<std::int16_t>>, "");
+static_assert(ex::is_abi_tag_v<ex::simd_abi::native<std::int32_t>>, "");
+static_assert(ex::is_abi_tag_v<ex::simd_abi::native<std::int64_t>>, "");
+static_assert(ex::is_abi_tag_v<ex::simd_abi::native<std::uint8_t>>, "");
+static_assert(ex::is_abi_tag_v<ex::simd_abi::native<std::uint16_t>>, "");
+static_assert(ex::is_abi_tag_v<ex::simd_abi::native<std::uint32_t>>, "");
+static_assert(ex::is_abi_tag_v<ex::simd_abi::native<std::uint64_t>>, "");
 static_assert(ex::is_abi_tag_v<ex::simd_abi::native<float>>, "");
 static_assert(ex::is_abi_tag_v<ex::simd_abi::native<double>>, "");
 
-static_assert(ex::is_abi_tag_v<ex::simd_abi::compatible<int8_t>>, "");
-static_assert(ex::is_abi_tag_v<ex::simd_abi::compatible<int16_t>>, "");
-static_assert(ex::is_abi_tag_v<ex::simd_abi::compatible<int32_t>>, "");
-static_assert(ex::is_abi_tag_v<ex::simd_abi::compatible<int64_t>>, "");
-static_assert(ex::is_abi_tag_v<ex::simd_abi::compatible<uint8_t>>, "");
-static_assert(ex::is_abi_tag_v<ex::simd_abi::compatible<uint16_t>>, "");
-static_assert(ex::is_abi_tag_v<ex::simd_abi::compatible<uint32_t>>, "");
-static_assert(ex::is_abi_tag_v<ex::simd_abi::compatible<uint64_t>>, "");
+static_assert(ex::is_abi_tag_v<ex::simd_abi::compatible<std::int8_t>>, "");
+static_assert(ex::is_abi_tag_v<ex::simd_abi::compatible<std::int16_t>>, "");
+static_assert(ex::is_abi_tag_v<ex::simd_abi::compatible<std::int32_t>>, "");
+static_assert(ex::is_abi_tag_v<ex::simd_abi::compatible<std::int64_t>>, "");
+static_assert(ex::is_abi_tag_v<ex::simd_abi::compatible<std::uint8_t>>, "");
+static_assert(ex::is_abi_tag_v<ex::simd_abi::compatible<std::uint16_t>>, "");
+static_assert(ex::is_abi_tag_v<ex::simd_abi::compatible<std::uint32_t>>, "");
+static_assert(ex::is_abi_tag_v<ex::simd_abi::compatible<std::uint64_t>>, "");
 static_assert(ex::is_abi_tag_v<ex::simd_abi::compatible<float>>, "");
 static_assert(ex::is_abi_tag_v<ex::simd_abi::compatible<double>>, "");
 
diff --git a/libcxx/test/std/experimental/simd/simd.traits/is_simd.pass.cpp b/libcxx/test/std/experimental/simd/simd.traits/is_simd.pass.cpp
index 6c7449eb6059d..dc20adf37b0cd 100644
--- a/libcxx/test/std/experimental/simd/simd.traits/is_simd.pass.cpp
+++ b/libcxx/test/std/experimental/simd/simd.traits/is_simd.pass.cpp
@@ -23,47 +23,47 @@ namespace ex = std::experimental::parallelism_v2;
 
 struct UserType {};
 
-static_assert(ex::is_simd<ex::native_simd<int8_t>>::value, "");
-static_assert(ex::is_simd<ex::native_simd<int16_t>>::value, "");
-static_assert(ex::is_simd<ex::native_simd<int32_t>>::value, "");
-static_assert(ex::is_simd<ex::native_simd<int64_t>>::value, "");
-static_assert(ex::is_simd<ex::native_simd<uint8_t>>::value, "");
-static_assert(ex::is_simd<ex::native_simd<uint16_t>>::value, "");
-static_assert(ex::is_simd<ex::native_simd<uint32_t>>::value, "");
-static_assert(ex::is_simd<ex::native_simd<uint64_t>>::value, "");
+static_assert(ex::is_simd<ex::native_simd<std::int8_t>>::value, "");
+static_assert(ex::is_simd<ex::native_simd<std::int16_t>>::value, "");
+static_assert(ex::is_simd<ex::native_simd<std::int32_t>>::value, "");
+static_assert(ex::is_simd<ex::native_simd<std::int64_t>>::value, "");
+static_assert(ex::is_simd<ex::native_simd<std::uint8_t>>::value, "");
+static_assert(ex::is_simd<ex::native_simd<std::uint16_t>>::value, "");
+static_assert(ex::is_simd<ex::native_simd<std::uint32_t>>::value, "");
+static_assert(ex::is_simd<ex::native_simd<std::uint64_t>>::value, "");
 static_assert(ex::is_simd<ex::native_simd<float>>::value, "");
 static_assert(ex::is_simd<ex::native_simd<double>>::value, "");
 
-static_assert(ex::is_simd<ex::fixed_size_simd<int8_t, 1>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<int16_t, 1>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<int32_t, 1>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<int64_t, 1>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<uint8_t, 1>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<uint16_t, 1>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<uint32_t, 1>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<uint64_t, 1>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::int8_t, 1>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::int16_t, 1>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::int32_t, 1>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::int64_t, 1>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::uint8_t, 1>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::uint16_t, 1>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::uint32_t, 1>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::uint64_t, 1>>::value, "");
 static_assert(ex::is_simd<ex::fixed_size_simd<float, 1>>::value, "");
 static_assert(ex::is_simd<ex::fixed_size_simd<double, 1>>::value, "");
 
-static_assert(ex::is_simd<ex::fixed_size_simd<int8_t, 3>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<int16_t, 3>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<int32_t, 3>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<int64_t, 3>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<uint8_t, 3>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<uint16_t, 3>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<uint32_t, 3>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<uint64_t, 3>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::int8_t, 3>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::int16_t, 3>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::int32_t, 3>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::int64_t, 3>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::uint8_t, 3>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::uint16_t, 3>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::uint32_t, 3>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::uint64_t, 3>>::value, "");
 static_assert(ex::is_simd<ex::fixed_size_simd<float, 3>>::value, "");
 static_assert(ex::is_simd<ex::fixed_size_simd<double, 3>>::value, "");
 
-static_assert(ex::is_simd<ex::fixed_size_simd<int8_t, 32>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<int16_t, 32>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<int32_t, 32>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<int64_t, 32>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<uint8_t, 32>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<uint16_t, 32>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<uint32_t, 32>>::value, "");
-static_assert(ex::is_simd<ex::fixed_size_simd<uint64_t, 32>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::int8_t, 32>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::int16_t, 32>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::int32_t, 32>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::int64_t, 32>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::uint8_t, 32>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::uint16_t, 32>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::uint32_t, 32>>::value, "");
+static_assert(ex::is_simd<ex::fixed_size_simd<std::uint64_t, 32>>::value, "");
 static_assert(ex::is_simd<ex::fixed_size_simd<float, 32>>::value, "");
 static_assert(ex::is_simd<ex::fixed_size_simd<double, 32>>::value, "");
 
@@ -74,47 +74,47 @@ static_assert(!ex::is_simd<ex::simd_mask<int>>::value, "");
 static_assert(!ex::is_simd<ex::simd_mask<float>>::value, "");
 static_assert(!ex::is_simd<UserType>::value, "");
 
-static_assert(ex::is_simd_v<ex::native_simd<int8_t>>, "");
-static_assert(ex::is_simd_v<ex::native_simd<int16_t>>, "");
-static_assert(ex::is_simd_v<ex::native_simd<int32_t>>, "");
-static_assert(ex::is_simd_v<ex::native_simd<int64_t>>, "");
-static_assert(ex::is_simd_v<ex::native_simd<uint8_t>>, "");
-static_assert(ex::is_simd_v<ex::native_simd<uint16_t>>, "");
-static_assert(ex::is_simd_v<ex::native_simd<uint32_t>>, "");
-static_assert(ex::is_simd_v<ex::native_simd<uint64_t>>, "");
+static_assert(ex::is_simd_v<ex::native_simd<std::int8_t>>, "");
+static_assert(ex::is_simd_v<ex::native_simd<std::int16_t>>, "");
+static_assert(ex::is_simd_v<ex::native_simd<std::int32_t>>, "");
+static_assert(ex::is_simd_v<ex::native_simd<std::int64_t>>, "");
+static_assert(ex::is_simd_v<ex::native_simd<std::uint8_t>>, "");
+static_assert(ex::is_simd_v<ex::native_simd<std::uint16_t>>, "");
+static_assert(ex::is_simd_v<ex::native_simd<std::uint32_t>>, "");
+static_assert(ex::is_simd_v<ex::native_simd<std::uint64_t>>, "");
 static_assert(ex::is_simd_v<ex::native_simd<float>>, "");
 static_assert(ex::is_simd_v<ex::native_simd<double>>, "");
 
-static_assert(ex::is_simd_v<ex::fixed_size_simd<int8_t, 1>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<int16_t, 1>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<int32_t, 1>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<int64_t, 1>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<uint8_t, 1>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<uint16_t, 1>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<uint32_t, 1>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<uint64_t, 1>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::int8_t, 1>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::int16_t, 1>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::int32_t, 1>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::int64_t, 1>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::uint8_t, 1>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::uint16_t, 1>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::uint32_t, 1>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::uint64_t, 1>>, "");
 static_assert(ex::is_simd_v<ex::fixed_size_simd<float, 1>>, "");
 static_assert(ex::is_simd_v<ex::fixed_size_simd<double, 1>>, "");
 
-static_assert(ex::is_simd_v<ex::fixed_size_simd<int8_t, 3>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<int16_t, 3>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<int32_t, 3>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<int64_t, 3>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<uint8_t, 3>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<uint16_t, 3>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<uint32_t, 3>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<uint64_t, 3>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::int8_t, 3>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::int16_t, 3>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::int32_t, 3>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::int64_t, 3>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::uint8_t, 3>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::uint16_t, 3>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::uint32_t, 3>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::uint64_t, 3>>, "");
 static_assert(ex::is_simd_v<ex::fixed_size_simd<float, 3>>, "");
 static_assert(ex::is_simd_v<ex::fixed_size_simd<double, 3>>, "");
 
-static_assert(ex::is_simd_v<ex::fixed_size_simd<int8_t, 32>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<int16_t, 32>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<int32_t, 32>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<int64_t, 32>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<uint8_t, 32>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<uint16_t, 32>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<uint32_t, 32>>, "");
-static_assert(ex::is_simd_v<ex::fixed_size_simd<uint64_t, 32>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::int8_t, 32>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::int16_t, 32>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::int32_t, 32>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::int64_t, 32>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::uint8_t, 32>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::uint16_t, 32>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::uint32_t, 32>>, "");
+static_assert(ex::is_simd_v<ex::fixed_size_simd<std::uint64_t, 32>>, "");
 static_assert(ex::is_simd_v<ex::fixed_size_simd<float, 32>>, "");
 static_assert(ex::is_simd_v<ex::fixed_size_simd<double, 32>>, "");
 
diff --git a/libcxx/test/std/experimental/simd/simd.traits/is_simd_flag_type.pass.cpp b/libcxx/test/std/experimental/simd/simd.traits/is_simd_flag_type.pass.cpp
index 1e7b0468867e0..e174a7c4d51d8 100644
--- a/libcxx/test/std/experimental/simd/simd.traits/is_simd_flag_type.pass.cpp
+++ b/libcxx/test/std/experimental/simd/simd.traits/is_simd_flag_type.pass.cpp
@@ -32,8 +32,8 @@ static_assert(!ex::is_simd_flag_type<void>::value, "");
 static_assert(!ex::is_simd_flag_type<int>::value, "");
 static_assert(!ex::is_simd_flag_type<float>::value, "");
 static_assert(!ex::is_simd_flag_type<UserType>::value, "");
-static_assert(!ex::is_simd_flag_type<ex::simd<int8_t>>::value, "");
-static_assert(!ex::is_simd_flag_type<ex::simd_mask<int8_t>>::value, "");
+static_assert(!ex::is_simd_flag_type<ex::simd<std::int8_t>>::value, "");
+static_assert(!ex::is_simd_flag_type<ex::simd_mask<std::int8_t>>::value, "");
 
 static_assert(ex::is_simd_flag_type_v<ex::element_aligned_tag>, "");
 static_assert(ex::is_simd_flag_type_v<ex::vector_aligned_tag>, "");
@@ -44,8 +44,8 @@ static_assert(!ex::is_simd_flag_type_v<void>, "");
 static_assert(!ex::is_simd_flag_type_v<int>, "");
 static_assert(!ex::is_simd_flag_type_v<float>, "");
 static_assert(!ex::is_simd_flag_type_v<UserType>, "");
-static_assert(!ex::is_simd_flag_type_v<ex::simd<int8_t>>, "");
-static_assert(!ex::is_simd_flag_type_v<ex::simd_mask<int8_t>>, "");
+static_assert(!ex::is_simd_flag_type_v<ex::simd<std::int8_t>>, "");
+static_assert(!ex::is_simd_flag_type_v<ex::simd_mask<std::int8_t>>, "");
 
 int main(int, char**) {
   return 0;
diff --git a/libcxx/test/std/experimental/simd/simd.traits/is_simd_mask.pass.cpp b/libcxx/test/std/experimental/simd/simd.traits/is_simd_mask.pass.cpp
index a495af43dcfea..b9d6cf84ea401 100644
--- a/libcxx/test/std/experimental/simd/simd.traits/is_simd_mask.pass.cpp
+++ b/libcxx/test/std/experimental/simd/simd.traits/is_simd_mask.pass.cpp
@@ -23,68 +23,68 @@ namespace ex = std::experimental::parallelism_v2;
 
 struct UserType {};
 
-static_assert(ex::is_simd_mask<ex::native_simd_mask<int8_t>>::value, "");
-static_assert(ex::is_simd_mask<ex::native_simd_mask<int16_t>>::value, "");
-static_assert(ex::is_simd_mask<ex::native_simd_mask<int32_t>>::value, "");
-static_assert(ex::is_simd_mask<ex::native_simd_mask<int64_t>>::value, "");
-static_assert(ex::is_simd_mask<ex::native_simd_mask<uint8_t>>::value, "");
-static_assert(ex::is_simd_mask<ex::native_simd_mask<uint16_t>>::value, "");
-static_assert(ex::is_simd_mask<ex::native_simd_mask<uint32_t>>::value, "");
-static_assert(ex::is_simd_mask<ex::native_simd_mask<uint64_t>>::value, "");
+static_assert(ex::is_simd_mask<ex::native_simd_mask<std::int8_t>>::value, "");
+static_assert(ex::is_simd_mask<ex::native_simd_mask<std::int16_t>>::value, "");
+static_assert(ex::is_simd_mask<ex::native_simd_mask<std::int32_t>>::value, "");
+static_assert(ex::is_simd_mask<ex::native_simd_mask<std::int64_t>>::value, "");
+static_assert(ex::is_simd_mask<ex::native_simd_mask<std::uint8_t>>::value, "");
+static_assert(ex::is_simd_mask<ex::native_simd_mask<std::uint16_t>>::value, "");
+static_assert(ex::is_simd_mask<ex::native_simd_mask<std::uint32_t>>::value, "");
+static_assert(ex::is_simd_mask<ex::native_simd_mask<std::uint64_t>>::value, "");
 static_assert(ex::is_simd_mask<ex::native_simd_mask<float>>::value, "");
 static_assert(ex::is_simd_mask<ex::native_simd_mask<double>>::value, "");
 
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<int8_t, 1>>::value, "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<int16_t, 1>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::int8_t, 1>>::value, "");
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::int16_t, 1>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<int32_t, 1>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::int32_t, 1>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<int64_t, 1>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::int64_t, 1>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<uint8_t, 1>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::uint8_t, 1>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<uint16_t, 1>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::uint16_t, 1>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<uint32_t, 1>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::uint32_t, 1>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<uint64_t, 1>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::uint64_t, 1>>::value,
               "");
 static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<float, 1>>::value, "");
 static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<double, 1>>::value, "");
 
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<int8_t, 3>>::value, "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<int16_t, 3>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::int8_t, 3>>::value, "");
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::int16_t, 3>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<int32_t, 3>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::int32_t, 3>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<int64_t, 3>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::int64_t, 3>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<uint8_t, 3>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::uint8_t, 3>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<uint16_t, 3>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::uint16_t, 3>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<uint32_t, 3>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::uint32_t, 3>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<uint64_t, 3>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::uint64_t, 3>>::value,
               "");
 static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<float, 3>>::value, "");
 static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<double, 3>>::value, "");
 
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<int8_t, 32>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::int8_t, 32>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<int16_t, 32>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::int16_t, 32>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<int32_t, 32>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::int32_t, 32>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<int64_t, 32>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::int64_t, 32>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<uint8_t, 32>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::uint8_t, 32>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<uint16_t, 32>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::uint16_t, 32>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<uint32_t, 32>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::uint32_t, 32>>::value,
               "");
-static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<uint64_t, 32>>::value,
+static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<std::uint64_t, 32>>::value,
               "");
 static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<float, 32>>::value, "");
 static_assert(ex::is_simd_mask<ex::fixed_size_simd_mask<double, 32>>::value,
@@ -97,47 +97,47 @@ static_assert(!ex::is_simd_mask<ex::simd<int>>::value, "");
 static_assert(!ex::is_simd_mask<ex::simd<float>>::value, "");
 static_assert(!ex::is_simd_mask<UserType>::value, "");
 
-static_assert(ex::is_simd_mask_v<ex::native_simd_mask<int8_t>>, "");
-static_assert(ex::is_simd_mask_v<ex::native_simd_mask<int16_t>>, "");
-static_assert(ex::is_simd_mask_v<ex::native_simd_mask<int32_t>>, "");
-static_assert(ex::is_simd_mask_v<ex::native_simd_mask<int64_t>>, "");
-static_assert(ex::is_simd_mask_v<ex::native_simd_mask<uint8_t>>, "");
-static_assert(ex::is_simd_mask_v<ex::native_simd_mask<uint16_t>>, "");
-static_assert(ex::is_simd_mask_v<ex::native_simd_mask<uint32_t>>, "");
-static_assert(ex::is_simd_mask_v<ex::native_simd_mask<uint64_t>>, "");
+static_assert(ex::is_simd_mask_v<ex::native_simd_mask<std::int8_t>>, "");
+static_assert(ex::is_simd_mask_v<ex::native_simd_mask<std::int16_t>>, "");
+static_assert(ex::is_simd_mask_v<ex::native_simd_mask<std::int32_t>>, "");
+static_assert(ex::is_simd_mask_v<ex::native_simd_mask<std::int64_t>>, "");
+static_assert(ex::is_simd_mask_v<ex::native_simd_mask<std::uint8_t>>, "");
+static_assert(ex::is_simd_mask_v<ex::native_simd_mask<std::uint16_t>>, "");
+static_assert(ex::is_simd_mask_v<ex::native_simd_mask<std::uint32_t>>, "");
+static_assert(ex::is_simd_mask_v<ex::native_simd_mask<std::uint64_t>>, "");
 static_assert(ex::is_simd_mask_v<ex::native_simd_mask<float>>, "");
 static_assert(ex::is_simd_mask_v<ex::native_simd_mask<double>>, "");
 
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<int8_t, 1>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<int16_t, 1>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<int32_t, 1>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<int64_t, 1>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<uint8_t, 1>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<uint16_t, 1>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<uint32_t, 1>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<uint64_t, 1>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::int8_t, 1>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::int16_t, 1>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::int32_t, 1>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::int64_t, 1>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::uint8_t, 1>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::uint16_t, 1>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::uint32_t, 1>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::uint64_t, 1>>, "");
 static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<float, 1>>, "");
 static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<double, 1>>, "");
 
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<int8_t, 3>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<int16_t, 3>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<int32_t, 3>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<int64_t, 3>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<uint8_t, 3>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<uint16_t, 3>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<uint32_t, 3>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<uint64_t, 3>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::int8_t, 3>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::int16_t, 3>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::int32_t, 3>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::int64_t, 3>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::uint8_t, 3>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::uint16_t, 3>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::uint32_t, 3>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::uint64_t, 3>>, "");
 static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<float, 3>>, "");
 static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<double, 3>>, "");
 
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<int8_t, 32>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<int16_t, 32>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<int32_t, 32>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<int64_t, 32>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<uint8_t, 32>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<uint16_t, 32>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<uint32_t, 32>>, "");
-static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<uint64_t, 32>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::int8_t, 32>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::int16_t, 32>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::int32_t, 32>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::int64_t, 32>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::uint8_t, 32>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::uint16_t, 32>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::uint32_t, 32>>, "");
+static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<std::uint64_t, 32>>, "");
 static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<float, 32>>, "");
 static_assert(ex::is_simd_mask_v<ex::fixed_size_simd_mask<double, 32>>, "");
 
diff --git a/libcxx/test/std/input.output/filesystems/class.path/path.member/path.decompose/empty.verify.cpp b/libcxx/test/std/input.output/filesystems/class.path/path.member/path.decompose/empty.verify.cpp
index 75e88e0b499d6..32bbc4c210731 100644
--- a/libcxx/test/std/input.output/filesystems/class.path/path.member/path.decompose/empty.verify.cpp
+++ b/libcxx/test/std/input.output/filesystems/class.path/path.member/path.decompose/empty.verify.cpp
@@ -16,12 +16,7 @@
 
 #include "filesystem_include.h"
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     fs::path c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.last_write_time/last_write_time.pass.cpp b/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.last_write_time/last_write_time.pass.cpp
index 51546f33a4f28..21d08f44bc47e 100644
--- a/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.last_write_time/last_write_time.pass.cpp
+++ b/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.last_write_time/last_write_time.pass.cpp
@@ -51,8 +51,8 @@ using std::chrono::duration_cast;
 
 #ifdef _WIN32
 struct TimeSpec {
-  int64_t tv_sec;
-  int64_t tv_nsec;
+  std::int64_t tv_sec;
+  std::int64_t tv_nsec;
 };
 struct StatT {
   TimeSpec st_atim;
@@ -60,7 +60,7 @@ struct StatT {
 };
 // There were 369 years and 89 leap days from the Windows epoch
 // (1601) to the Unix epoch (1970).
-#define FILE_TIME_OFFSET_SECS (uint64_t(369 * 365 + 89) * (24 * 60 * 60))
+#define FILE_TIME_OFFSET_SECS (std::uint64_t(369 * 365 + 89) * (24 * 60 * 60))
 static TimeSpec filetime_to_timespec(LARGE_INTEGER li) {
   TimeSpec ret;
   ret.tv_sec = li.QuadPart / 10000000 - FILE_TIME_OFFSET_SECS;
diff --git a/libcxx/test/std/input.output/iostream.format/output.streams/ostream.formatted/ostream.inserters.arithmetic/minmax_showbase.pass.cpp b/libcxx/test/std/input.output/iostream.format/output.streams/ostream.formatted/ostream.inserters.arithmetic/minmax_showbase.pass.cpp
index 126151221aa4b..3f522389ed065 100644
--- a/libcxx/test/std/input.output/iostream.format/output.streams/ostream.formatted/ostream.inserters.arithmetic/minmax_showbase.pass.cpp
+++ b/libcxx/test/std/input.output/iostream.format/output.streams/ostream.formatted/ostream.inserters.arithmetic/minmax_showbase.pass.cpp
@@ -67,9 +67,9 @@ int main(int, char**)
     test<unsigned int>(d, "4294967295");
     test<unsigned int>(x, "0xffffffff");
 
-    const bool long_is_32 = std::integral_constant<bool, sizeof(long) == sizeof(int32_t)>::value; // avoid compiler warnings
-    const bool long_is_64 = std::integral_constant<bool, sizeof(long) == sizeof(int64_t)>::value; // avoid compiler warnings
-    const bool long_long_is_64 = std::integral_constant<bool, sizeof(long long) == sizeof(int64_t)>::value; // avoid compiler warnings
+    const bool long_is_32 = std::integral_constant<bool, sizeof(long) == sizeof(std::int32_t)>::value; // avoid compiler warnings
+    const bool long_is_64 = std::integral_constant<bool, sizeof(long) == sizeof(std::int64_t)>::value; // avoid compiler warnings
+    const bool long_long_is_64 = std::integral_constant<bool, sizeof(long long) == sizeof(std::int64_t)>::value; // avoid compiler warnings
 
     if (long_is_32) {
         test<long>(o, "020000000000");
diff --git a/libcxx/test/std/input.output/iostream.format/output.streams/ostream.formatted/ostream.inserters.arithmetic/minus1.pass.cpp b/libcxx/test/std/input.output/iostream.format/output.streams/ostream.formatted/ostream.inserters.arithmetic/minus1.pass.cpp
index 8e3b77c420bea..1f3003a0945d7 100644
--- a/libcxx/test/std/input.output/iostream.format/output.streams/ostream.formatted/ostream.inserters.arithmetic/minus1.pass.cpp
+++ b/libcxx/test/std/input.output/iostream.format/output.streams/ostream.formatted/ostream.inserters.arithmetic/minus1.pass.cpp
@@ -63,16 +63,16 @@ void test_hex(const char *expected)
 int main(int, char**)
 {
 
-    test_octal<uint16_t>(                "177777");
-    test_octal< int16_t>(                "177777");
-    test_octal<uint32_t>(           "37777777777");
-    test_octal< int32_t>(           "37777777777");
-    test_octal<uint64_t>("1777777777777777777777");
-    test_octal< int64_t>("1777777777777777777777");
-    test_octal<uint64_t>("1777777777777777777777");
+    test_octal<std::uint16_t>(                "177777");
+    test_octal< std::int16_t>(                "177777");
+    test_octal<std::uint32_t>(           "37777777777");
+    test_octal< std::int32_t>(           "37777777777");
+    test_octal<std::uint64_t>("1777777777777777777777");
+    test_octal< std::int64_t>("1777777777777777777777");
+    test_octal<std::uint64_t>("1777777777777777777777");
 
-    const bool long_is_64 = std::integral_constant<bool, sizeof(long) == sizeof(int64_t)>::value; // avoid compiler warnings
-    const bool long_long_is_64 = std::integral_constant<bool, sizeof(long long) == sizeof(int64_t)>::value; // avoid compiler warnings
+    const bool long_is_64 = std::integral_constant<bool, sizeof(long) == sizeof(std::int64_t)>::value; // avoid compiler warnings
+    const bool long_long_is_64 = std::integral_constant<bool, sizeof(long long) == sizeof(std::int64_t)>::value; // avoid compiler warnings
 
     if (long_is_64) {
         test_octal< unsigned long>("1777777777777777777777");
@@ -83,12 +83,12 @@ int main(int, char**)
         test_octal<          long long>("1777777777777777777777");
     }
 
-    test_dec<uint16_t>(               "65535");
-    test_dec< int16_t>(                  "-1");
-    test_dec<uint32_t>(          "4294967295");
-    test_dec< int32_t>(                  "-1");
-    test_dec<uint64_t>("18446744073709551615");
-    test_dec< int64_t>(                  "-1");
+    test_dec<std::uint16_t>(               "65535");
+    test_dec< std::int16_t>(                  "-1");
+    test_dec<std::uint32_t>(          "4294967295");
+    test_dec< std::int32_t>(                  "-1");
+    test_dec<std::uint64_t>("18446744073709551615");
+    test_dec< std::int64_t>(                  "-1");
     if (long_is_64) {
         test_dec<unsigned long>("18446744073709551615");
         test_dec<         long>(                  "-1");
@@ -98,12 +98,12 @@ int main(int, char**)
         test_dec<         long long>(                  "-1");
     }
 
-    test_hex<uint16_t>(            "FFFF");
-    test_hex< int16_t>(            "FFFF");
-    test_hex<uint32_t>(        "FFFFFFFF");
-    test_hex< int32_t>(        "FFFFFFFF");
-    test_hex<uint64_t>("FFFFFFFFFFFFFFFF");
-    test_hex< int64_t>("FFFFFFFFFFFFFFFF");
+    test_hex<std::uint16_t>(            "FFFF");
+    test_hex< std::int16_t>(            "FFFF");
+    test_hex<std::uint32_t>(        "FFFFFFFF");
+    test_hex< std::int32_t>(        "FFFFFFFF");
+    test_hex<std::uint64_t>("FFFFFFFFFFFFFFFF");
+    test_hex< std::int64_t>("FFFFFFFFFFFFFFFF");
     if (long_is_64) {
         test_hex<unsigned long>("FFFFFFFFFFFFFFFF");
         test_hex<         long>("FFFFFFFFFFFFFFFF");
diff --git a/libcxx/test/std/input.output/iostream.format/quoted.manip/quoted_char.verify.cpp b/libcxx/test/std/input.output/iostream.format/quoted.manip/quoted_char.verify.cpp
index 4d9635183d663..22013cd6f4ed1 100644
--- a/libcxx/test/std/input.output/iostream.format/quoted.manip/quoted_char.verify.cpp
+++ b/libcxx/test/std/input.output/iostream.format/quoted.manip/quoted_char.verify.cpp
@@ -16,9 +16,6 @@
 #include <iomanip>
 #include <sstream>
 #include <string>
-#include <cassert>
-
-#include "test_macros.h"
 
 //  Test that mismatches between strings and wide streams are diagnosed
 
@@ -29,6 +26,6 @@ void round_trip ( const char *p ) {
     ss >> std::quoted(s); // expected-error {{invalid operands to binary expression}}
 }
 
-int main(int, char**) {
+void f() {
     round_trip("Hi Mom");
 }
diff --git a/libcxx/test/std/input.output/iostream.format/quoted.manip/quoted_traits.verify.cpp b/libcxx/test/std/input.output/iostream.format/quoted.manip/quoted_traits.verify.cpp
index 5a92485ced56f..e457adb5edda8 100644
--- a/libcxx/test/std/input.output/iostream.format/quoted.manip/quoted_traits.verify.cpp
+++ b/libcxx/test/std/input.output/iostream.format/quoted.manip/quoted_traits.verify.cpp
@@ -15,9 +15,6 @@
 #include <iomanip>
 #include <sstream>
 #include <string>
-#include <cassert>
-
-#include "test_macros.h"
 
 //  Test that mismatches in the traits between the quoted object and the dest string are diagnosed.
 
@@ -33,6 +30,6 @@ void round_trip ( const char *p ) {
     ss >> std::quoted(s); // expected-error {{invalid operands to binary expression}}
 }
 
-int main(int, char**) {
+void f() {
     round_trip("Hi Mom");
 }
diff --git a/libcxx/test/std/iterators/iterator.container/empty.array.verify.cpp b/libcxx/test/std/iterators/iterator.container/empty.array.verify.cpp
index e67d864670f0e..c60c752d413d0 100644
--- a/libcxx/test/std/iterators/iterator.container/empty.array.verify.cpp
+++ b/libcxx/test/std/iterators/iterator.container/empty.array.verify.cpp
@@ -13,15 +13,9 @@
 
 // UNSUPPORTED: c++03, c++11, c++14, c++17
 
-#include <vector>
 #include <iterator>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     int c[5];
     std::empty(c); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/iterators/iterator.container/empty.container.verify.cpp b/libcxx/test/std/iterators/iterator.container/empty.container.verify.cpp
index 44b102efa0d11..6fbaa052ac245 100644
--- a/libcxx/test/std/iterators/iterator.container/empty.container.verify.cpp
+++ b/libcxx/test/std/iterators/iterator.container/empty.container.verify.cpp
@@ -16,12 +16,7 @@
 #include <vector>
 #include <iterator>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::vector<int> c;
     std::empty(c); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/iterators/iterator.container/empty.initializer_list.verify.cpp b/libcxx/test/std/iterators/iterator.container/empty.initializer_list.verify.cpp
index f97534da42c0c..416931f1db688 100644
--- a/libcxx/test/std/iterators/iterator.container/empty.initializer_list.verify.cpp
+++ b/libcxx/test/std/iterators/iterator.container/empty.initializer_list.verify.cpp
@@ -16,12 +16,7 @@
 #include <initializer_list>
 #include <iterator>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::initializer_list<int> c = { 4 };
     std::empty(c); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/iterators/iterator.container/ssize.pass.cpp b/libcxx/test/std/iterators/iterator.container/ssize.pass.cpp
index 9f384c64ddf35..779e9ed8aa9dc 100644
--- a/libcxx/test/std/iterators/iterator.container/ssize.pass.cpp
+++ b/libcxx/test/std/iterators/iterator.container/ssize.pass.cpp
@@ -13,14 +13,16 @@
 //     -> common_type_t<ptrdiff_t, make_signed_t<decltype(c.size())>>;                    // C++20
 // template <class T, ptrdiff_t> constexpr ptrdiff_t ssize(const T (&array)[N]) noexcept; // C++20
 
-#include <iterator>
-#include <cassert>
-#include <vector>
 #include <array>
-#include <list>
+#include <cassert>
+#include <cstdint>
 #include <initializer_list>
-#include <string_view>
+#include <iterator>
 #include <limits>
+#include <list>
+#include <string_view>
+#include <type_traits>
+#include <vector>
 
 #include "test_macros.h"
 
@@ -28,7 +30,7 @@
 TEST_GCC_DIAGNOSTIC_IGNORED("-Wtype-limits")
 
 struct short_container {
-    uint16_t size() const { return 60000; } // not noexcept
+    std::uint16_t size() const { return 60000; } // not noexcept
 };
 
 template<typename C>
diff --git a/libcxx/test/std/iterators/iterator.requirements/iterator.cust/iterator.cust.move/iter_move.nodiscard.verify.cpp b/libcxx/test/std/iterators/iterator.requirements/iterator.cust/iterator.cust.move/iter_move.nodiscard.verify.cpp
index f531b3ab37fe3..ed4061f726b15 100644
--- a/libcxx/test/std/iterators/iterator.requirements/iterator.cust/iterator.cust.move/iter_move.nodiscard.verify.cpp
+++ b/libcxx/test/std/iterators/iterator.requirements/iterator.cust/iterator.cust.move/iter_move.nodiscard.verify.cpp
@@ -24,12 +24,10 @@ struct WithADL {
   friend constexpr auto iter_move(WithADL&) { return 0; }
 };
 
-int main(int, char**) {
+void f() {
   int* noADL = nullptr;
   std::ranges::iter_move(noADL); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
 
   WithADL adl;
   std::ranges::iter_move(adl); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-  return 0;
 }
diff --git a/libcxx/test/std/iterators/iterator.requirements/iterator.cust/iterator.cust.move/iter_move.pass.cpp b/libcxx/test/std/iterators/iterator.requirements/iterator.cust/iterator.cust.move/iter_move.pass.cpp
index 31ec0100e2eee..566638263e887 100644
--- a/libcxx/test/std/iterators/iterator.requirements/iterator.cust/iterator.cust.move/iter_move.pass.cpp
+++ b/libcxx/test/std/iterators/iterator.requirements/iterator.cust/iterator.cust.move/iter_move.pass.cpp
@@ -11,11 +11,11 @@
 // template<class I>
 // unspecified iter_move;
 
-#include <iterator>
-
-#include <array>
 #include <algorithm>
+#include <array>
 #include <cassert>
+#include <iterator>
+#include <type_traits>
 #include <utility>
 
 #include "../unqualified_lookup_wrapper.h"
diff --git a/libcxx/test/std/iterators/predef.iterators/move.iterators/move.iter.ops/move.iter.op.const/ctor.iter.explicit.verify.cpp b/libcxx/test/std/iterators/predef.iterators/move.iterators/move.iter.ops/move.iter.op.const/ctor.iter.explicit.verify.cpp
index b14111555966e..5b609f8cb6dcd 100644
--- a/libcxx/test/std/iterators/predef.iterators/move.iterators/move.iter.ops/move.iter.op.const/ctor.iter.explicit.verify.cpp
+++ b/libcxx/test/std/iterators/predef.iterators/move.iterators/move.iter.ops/move.iter.op.const/ctor.iter.explicit.verify.cpp
@@ -16,8 +16,7 @@
 
 #include <iterator>
 
-int main(int, char**) {
+void f() {
     char const* it = "";
     std::move_iterator<char const*> r = it; // expected-error{{no viable conversion from 'const char *' to 'std::move_iterator<const char *>'}}
-    return 0;
 }
diff --git a/libcxx/test/std/iterators/predef.iterators/move.iterators/move.iter.ops/move.iter.op.ref/deprecated.verify.cpp b/libcxx/test/std/iterators/predef.iterators/move.iterators/move.iter.ops/move.iter.op.ref/deprecated.verify.cpp
index 8c1684f0319dc..e4e22cb5ac7c1 100644
--- a/libcxx/test/std/iterators/predef.iterators/move.iterators/move.iter.ops/move.iter.op.ref/deprecated.verify.cpp
+++ b/libcxx/test/std/iterators/predef.iterators/move.iterators/move.iter.ops/move.iter.op.ref/deprecated.verify.cpp
@@ -12,9 +12,6 @@
 
 #include <iterator>
 
-int main(int, char**) {
-  (void)std::move_iterator<int*>().operator->();
-  // expected-warning@-1{{'operator->' is deprecated}}
-
-  return 0;
+void f() {
+  (void)std::move_iterator<int*>().operator->(); // expected-warning {{'operator->' is deprecated}}
 }
diff --git a/libcxx/test/std/iterators/predef.iterators/reverse.iterators/reverse.iter.cons/ctor.iter.explicit.verify.cpp b/libcxx/test/std/iterators/predef.iterators/reverse.iterators/reverse.iter.cons/ctor.iter.explicit.verify.cpp
index 940848b300eef..d69799de27dcc 100644
--- a/libcxx/test/std/iterators/predef.iterators/reverse.iterators/reverse.iter.cons/ctor.iter.explicit.verify.cpp
+++ b/libcxx/test/std/iterators/predef.iterators/reverse.iterators/reverse.iter.cons/ctor.iter.explicit.verify.cpp
@@ -16,8 +16,7 @@
 
 #include <iterator>
 
-int main(int, char**) {
+void f() {
     char const* it = "";
     std::reverse_iterator<char const*> r = it; // expected-error{{no viable conversion from 'const char *' to 'std::reverse_iterator<const char *>'}}
-    return 0;
 }
diff --git a/libcxx/test/std/iterators/stream.iterators/istream.iterator/istream.iterator.cons/default.pass.cpp b/libcxx/test/std/iterators/stream.iterators/istream.iterator/istream.iterator.cons/default.pass.cpp
index 7273223a7dbc1..d8377f0f4c5c8 100644
--- a/libcxx/test/std/iterators/stream.iterators/istream.iterator/istream.iterator.cons/default.pass.cpp
+++ b/libcxx/test/std/iterators/stream.iterators/istream.iterator/istream.iterator.cons/default.pass.cpp
@@ -14,9 +14,10 @@
 // C++17 says: If is_trivially_default_constructible_v<T> is true, then this
 //    constructor is a constexpr constructor.
 
-#include <iterator>
 #include <cassert>
+#include <iterator>
 #include <string>
+#include <type_traits>
 
 #include "test_macros.h"
 
diff --git a/libcxx/test/std/language.support/cmp/cmp.categories.pre/zero_type.verify.cpp b/libcxx/test/std/language.support/cmp/cmp.categories.pre/zero_type.verify.cpp
index 6d2687f599fad..8e3c793de4b92 100644
--- a/libcxx/test/std/language.support/cmp/cmp.categories.pre/zero_type.verify.cpp
+++ b/libcxx/test/std/language.support/cmp/cmp.categories.pre/zero_type.verify.cpp
@@ -50,9 +50,8 @@ void test_category(T v) {
   TEST_PASS(v, <=>);
 }
 
-int main(int, char**) {
+void f() {
   test_category(std::strong_ordering::equivalent);
   test_category(std::weak_ordering::equivalent);
   test_category(std::partial_ordering::equivalent);
-  return 0;
 }
diff --git a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.array/new_size.verify.cpp b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.array/new_size.verify.cpp
index efd84bb57967b..8e979165219dd 100644
--- a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.array/new_size.verify.cpp
+++ b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.array/new_size.verify.cpp
@@ -17,7 +17,6 @@
 
 #include <new>
 
-int main(int, char**) {
+void f() {
     ::operator new[](4);  // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-    return 0;
 }
diff --git a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.array/new_size_align.verify.cpp b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.array/new_size_align.verify.cpp
index 1551a057bac1e..530e152c5e41a 100644
--- a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.array/new_size_align.verify.cpp
+++ b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.array/new_size_align.verify.cpp
@@ -21,7 +21,6 @@
 
 #include <new>
 
-int main(int, char**) {
+void f() {
     ::operator new[](4, std::align_val_t{4});  // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-    return 0;
 }
diff --git a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.array/new_size_align_nothrow.verify.cpp b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.array/new_size_align_nothrow.verify.cpp
index ec819627aa5a7..33100736299c2 100644
--- a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.array/new_size_align_nothrow.verify.cpp
+++ b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.array/new_size_align_nothrow.verify.cpp
@@ -21,7 +21,6 @@
 
 #include <new>
 
-int main(int, char**) {
+void f() {
     ::operator new[](4, std::align_val_t{4}, std::nothrow);  // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-    return 0;
 }
diff --git a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.array/new_size_nothrow.verify.cpp b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.array/new_size_nothrow.verify.cpp
index 066ca1a23a106..fd954d4e70fca 100644
--- a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.array/new_size_nothrow.verify.cpp
+++ b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.array/new_size_nothrow.verify.cpp
@@ -17,7 +17,6 @@
 
 #include <new>
 
-int main(int, char**) {
+void f() {
     ::operator new[](4, std::nothrow);  // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-    return 0;
 }
diff --git a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.placement/new_array_ptr.verify.cpp b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.placement/new_array_ptr.verify.cpp
index b5ec8248b9fa6..b1b2f0f13882b 100644
--- a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.placement/new_array_ptr.verify.cpp
+++ b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.placement/new_array_ptr.verify.cpp
@@ -14,12 +14,7 @@
 
 #include <new>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     char buffer[100];
     ::operator new[](4, buffer); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.placement/new_ptr.verify.cpp b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.placement/new_ptr.verify.cpp
index 5029864349530..e98144ebd08dc 100644
--- a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.placement/new_ptr.verify.cpp
+++ b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.placement/new_ptr.verify.cpp
@@ -14,12 +14,7 @@
 
 #include <new>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     char buffer[100];
     ::operator new(4, buffer); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.single/new_size.verify.cpp b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.single/new_size.verify.cpp
index ed6607847f29f..d3ed46e7052b3 100644
--- a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.single/new_size.verify.cpp
+++ b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.single/new_size.verify.cpp
@@ -14,11 +14,6 @@
 
 #include <new>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     ::operator new(4); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.single/new_size_align.verify.cpp b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.single/new_size_align.verify.cpp
index 2a30e15640d96..42a88615734a2 100644
--- a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.single/new_size_align.verify.cpp
+++ b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.single/new_size_align.verify.cpp
@@ -21,7 +21,6 @@
 
 #include <new>
 
-int main(int, char**) {
+void f() {
     ::operator new(4, std::align_val_t{4});  // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-    return 0;
 }
diff --git a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.single/new_size_align_nothrow.verify.cpp b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.single/new_size_align_nothrow.verify.cpp
index 047e81d1c9757..8dcdbe3ca28ee 100644
--- a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.single/new_size_align_nothrow.verify.cpp
+++ b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.single/new_size_align_nothrow.verify.cpp
@@ -21,7 +21,6 @@
 
 #include <new>
 
-int main(int, char**) {
+void f() {
     ::operator new(4, std::align_val_t{4}, std::nothrow);  // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-    return 0;
 }
diff --git a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.single/new_size_nothrow.verify.cpp b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.single/new_size_nothrow.verify.cpp
index 4826cb41d5453..081769b7c4840 100644
--- a/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.single/new_size_nothrow.verify.cpp
+++ b/libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.single/new_size_nothrow.verify.cpp
@@ -14,11 +14,6 @@
 
 #include <new>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     ::operator new(4, std::nothrow); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/language.support/support.dynamic/ptr.launder/launder.nodiscard.verify.cpp b/libcxx/test/std/language.support/support.dynamic/ptr.launder/launder.nodiscard.verify.cpp
index 632ca0e1a23b9..f8e6cb2e55d24 100644
--- a/libcxx/test/std/language.support/support.dynamic/ptr.launder/launder.nodiscard.verify.cpp
+++ b/libcxx/test/std/language.support/support.dynamic/ptr.launder/launder.nodiscard.verify.cpp
@@ -11,17 +11,11 @@
 // template <class T> constexpr T* launder(T* p) noexcept;
 
 // UNSUPPORTED: c++03, c++11, c++14
-// UNSUPPORETD: c++17 && !stdlib=libc++
+// UNSUPPORTED: c++17 && !stdlib=libc++
 
 #include <new>
-#include <cassert>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     int *p = nullptr;
     std::launder(p); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/language.support/support.srcloc/general.pass.cpp b/libcxx/test/std/language.support/support.srcloc/general.pass.cpp
index 39ad936628ee3..70861ee351973 100644
--- a/libcxx/test/std/language.support/support.srcloc/general.pass.cpp
+++ b/libcxx/test/std/language.support/support.srcloc/general.pass.cpp
@@ -43,8 +43,8 @@ ASSERT_NOEXCEPT(empty.line());
 ASSERT_NOEXCEPT(empty.column());
 ASSERT_NOEXCEPT(empty.file_name());
 ASSERT_NOEXCEPT(empty.function_name());
-std::same_as<uint_least32_t> auto line   = empty.line();
-std::same_as<uint_least32_t> auto column = empty.column();
+std::same_as<std::uint_least32_t> auto line   = empty.line();
+std::same_as<std::uint_least32_t> auto column = empty.column();
 std::same_as<const char*> auto file      = empty.file_name();
 std::same_as<const char*> auto function  = empty.function_name();
 
diff --git a/libcxx/test/std/localization/locale.categories/category.ctype/locale.codecvt.byname/codecvt_byname_char16_t_char.depr_in_cxx20.verify.cpp b/libcxx/test/std/localization/locale.categories/category.ctype/locale.codecvt.byname/codecvt_byname_char16_t_char.depr_in_cxx20.verify.cpp
index c430321699a6d..5beeed95da488 100644
--- a/libcxx/test/std/localization/locale.categories/category.ctype/locale.codecvt.byname/codecvt_byname_char16_t_char.depr_in_cxx20.verify.cpp
+++ b/libcxx/test/std/localization/locale.categories/category.ctype/locale.codecvt.byname/codecvt_byname_char16_t_char.depr_in_cxx20.verify.cpp
@@ -16,14 +16,10 @@
 #include <locale>
 
 #include "../with_public_dtor.hpp"
-#include "test_macros.h"
 
-int main(int, char **)
-{
+void f() {
     // Don't test for the exact type since the underlying type of
     // std::mbstate_t depends on implementation details.
     with_public_dtor<std::codecvt_byname<char16_t, char, std::mbstate_t>> cvt("", 0); // expected-warning-re {{'codecvt_byname<char16_t, char, {{.*}}>' is deprecated}}
     (void)cvt;
-
-    return 0;
 }
diff --git a/libcxx/test/std/localization/locale.categories/category.ctype/locale.codecvt.byname/codecvt_byname_char32_t_char.depr_in_cxx20.verify.cpp b/libcxx/test/std/localization/locale.categories/category.ctype/locale.codecvt.byname/codecvt_byname_char32_t_char.depr_in_cxx20.verify.cpp
index ebace3a4f647f..1cf0ae3e244ea 100644
--- a/libcxx/test/std/localization/locale.categories/category.ctype/locale.codecvt.byname/codecvt_byname_char32_t_char.depr_in_cxx20.verify.cpp
+++ b/libcxx/test/std/localization/locale.categories/category.ctype/locale.codecvt.byname/codecvt_byname_char32_t_char.depr_in_cxx20.verify.cpp
@@ -16,14 +16,10 @@
 #include <locale>
 
 #include "../with_public_dtor.hpp"
-#include "test_macros.h"
 
-int main(int, char **)
-{
+void f() {
     // Don't test for the exact type since the underlying type of
     // std::mbstate_t depends on implementation details.
     with_public_dtor<std::codecvt_byname<char32_t, char, std::mbstate_t>> cvt("", 0); // expected-warning-re {{'codecvt_byname<char32_t, char, {{.*}}>' is deprecated}}
     (void)cvt;
-
-    return 0;
 }
diff --git a/libcxx/test/std/localization/locale.categories/category.ctype/locale.codecvt/codecvt_char16_t_char.depr_in_cxx20.verify.cpp b/libcxx/test/std/localization/locale.categories/category.ctype/locale.codecvt/codecvt_char16_t_char.depr_in_cxx20.verify.cpp
index 54f57c6b04985..cc9c355854dff 100644
--- a/libcxx/test/std/localization/locale.categories/category.ctype/locale.codecvt/codecvt_char16_t_char.depr_in_cxx20.verify.cpp
+++ b/libcxx/test/std/localization/locale.categories/category.ctype/locale.codecvt/codecvt_char16_t_char.depr_in_cxx20.verify.cpp
@@ -16,14 +16,10 @@
 #include <locale>
 
 #include "../with_public_dtor.hpp"
-#include "test_macros.h"
 
-int main(int, char **)
-{
+void f() {
     // Don't test for the exact type since the underlying type of
     // std::mbstate_t depends on implementation details.
     with_public_dtor<std::codecvt<char16_t, char, std::mbstate_t>> cvt("", 0); // expected-warning-re {{'codecvt<char16_t, char, {{.*}}>' is deprecated}}
     (void)cvt;
-
-    return 0;
 }
diff --git a/libcxx/test/std/localization/locale.categories/category.ctype/locale.codecvt/codecvt_char32_t_char.depr_in_cxx20.verify.cpp b/libcxx/test/std/localization/locale.categories/category.ctype/locale.codecvt/codecvt_char32_t_char.depr_in_cxx20.verify.cpp
index 9b7fdc5165b67..7cbfbd37591d6 100644
--- a/libcxx/test/std/localization/locale.categories/category.ctype/locale.codecvt/codecvt_char32_t_char.depr_in_cxx20.verify.cpp
+++ b/libcxx/test/std/localization/locale.categories/category.ctype/locale.codecvt/codecvt_char32_t_char.depr_in_cxx20.verify.cpp
@@ -16,14 +16,10 @@
 #include <locale>
 
 #include "../with_public_dtor.hpp"
-#include "test_macros.h"
 
-int main(int, char **)
-{
+void f() {
     // Don't test for the exact type since the underlying type of
     // std::mbstate_t depends on implementation details.
     with_public_dtor<std::codecvt<char32_t, char, std::mbstate_t>> cvt("", 0); // expected-warning-re {{'codecvt<char32_t, char, {{.*}}>' is deprecated}}
     (void)cvt;
-
-    return 0;
 }
diff --git a/libcxx/test/std/numerics/bit/bit.endian/endian.pass.cpp b/libcxx/test/std/numerics/bit/bit.endian/endian.pass.cpp
index d16c8c2bddd25..fa02afe9cf657 100644
--- a/libcxx/test/std/numerics/bit/bit.endian/endian.pass.cpp
+++ b/libcxx/test/std/numerics/bit/bit.endian/endian.pass.cpp
@@ -38,7 +38,7 @@ int main(int, char**) {
 
 //  Try to check at runtime
     {
-    uint32_t i = 0x01020304;
+    std::uint32_t i = 0x01020304;
     char c[4];
     static_assert(sizeof(i) == sizeof(c));
     std::memcpy(c, &i, sizeof(c));
diff --git a/libcxx/test/std/numerics/bit/bit.pow.two/bit_ceil.fail.cpp b/libcxx/test/std/numerics/bit/bit.pow.two/bit_ceil.fail.cpp
index fab7d78d1f607..c768d0fa9de26 100644
--- a/libcxx/test/std/numerics/bit/bit.pow.two/bit_ceil.fail.cpp
+++ b/libcxx/test/std/numerics/bit/bit.pow.two/bit_ceil.fail.cpp
@@ -39,10 +39,10 @@ int main(int, char**)
     static_assert(toobig<unsigned long>(),      ""); // expected-error-re {{{{(static_assert|static assertion)}} expression is not an integral constant expression}}
     static_assert(toobig<unsigned long long>(), ""); // expected-error-re {{{{(static_assert|static assertion)}} expression is not an integral constant expression}}
 
-    static_assert(toobig<uint8_t>(), "");   // expected-error-re {{{{(static_assert|static assertion)}} expression is not an integral constant expression}}
-    static_assert(toobig<uint16_t>(), "");  // expected-error-re {{{{(static_assert|static assertion)}} expression is not an integral constant expression}}
-    static_assert(toobig<uint32_t>(), "");  // expected-error-re {{{{(static_assert|static assertion)}} expression is not an integral constant expression}}
-    static_assert(toobig<uint64_t>(), "");  // expected-error-re {{{{(static_assert|static assertion)}} expression is not an integral constant expression}}
+    static_assert(toobig<std::uint8_t>(), "");   // expected-error-re {{{{(static_assert|static assertion)}} expression is not an integral constant expression}}
+    static_assert(toobig<std::uint16_t>(), "");  // expected-error-re {{{{(static_assert|static assertion)}} expression is not an integral constant expression}}
+    static_assert(toobig<std::uint32_t>(), "");  // expected-error-re {{{{(static_assert|static assertion)}} expression is not an integral constant expression}}
+    static_assert(toobig<std::uint64_t>(), "");  // expected-error-re {{{{(static_assert|static assertion)}} expression is not an integral constant expression}}
     static_assert(toobig<size_t>(), "");    // expected-error-re {{{{(static_assert|static assertion)}} expression is not an integral constant expression}}
     static_assert(toobig<uintmax_t>(), ""); // expected-error-re {{{{(static_assert|static assertion)}} expression is not an integral constant expression}}
     static_assert(toobig<uintptr_t>(), ""); // expected-error-re {{{{(static_assert|static assertion)}} expression is not an integral constant expression}}
diff --git a/libcxx/test/std/numerics/bit/bit.pow.two/bit_ceil.pass.cpp b/libcxx/test/std/numerics/bit/bit.pow.two/bit_ceil.pass.cpp
index f18d52a0ed574..7aa1265838dbb 100644
--- a/libcxx/test/std/numerics/bit/bit.pow.two/bit_ceil.pass.cpp
+++ b/libcxx/test/std/numerics/bit/bit.pow.two/bit_ceil.pass.cpp
@@ -84,10 +84,10 @@ int main(int, char**)
     static_assert(!std::is_invocable_v<L, __int128_t>);
 #endif
 
-    static_assert(!std::is_invocable_v<L, int8_t>);
-    static_assert(!std::is_invocable_v<L, int16_t>);
-    static_assert(!std::is_invocable_v<L, int32_t>);
-    static_assert(!std::is_invocable_v<L, int64_t>);
+    static_assert(!std::is_invocable_v<L, std::int8_t>);
+    static_assert(!std::is_invocable_v<L, std::int16_t>);
+    static_assert(!std::is_invocable_v<L, std::int32_t>);
+    static_assert(!std::is_invocable_v<L, std::int64_t>);
     static_assert(!std::is_invocable_v<L, intmax_t>);
     static_assert(!std::is_invocable_v<L, intptr_t>);
     static_assert(!std::is_invocable_v<L, ptrdiff_t>);
@@ -115,10 +115,10 @@ int main(int, char**)
 #ifndef TEST_HAS_NO_INT128
     static_assert(test<__uint128_t>());
 #endif
-    static_assert(test<uint8_t>());
-    static_assert(test<uint16_t>());
-    static_assert(test<uint32_t>());
-    static_assert(test<uint64_t>());
+    static_assert(test<std::uint8_t>());
+    static_assert(test<std::uint16_t>());
+    static_assert(test<std::uint32_t>());
+    static_assert(test<std::uint64_t>());
     static_assert(test<uintmax_t>());
     static_assert(test<uintptr_t>());
     static_assert(test<size_t>());
@@ -131,10 +131,10 @@ int main(int, char**)
 #ifndef TEST_HAS_NO_INT128
     test<__uint128_t>();
 #endif
-    test<uint8_t>();
-    test<uint16_t>();
-    test<uint32_t>();
-    test<uint64_t>();
+    test<std::uint8_t>();
+    test<std::uint16_t>();
+    test<std::uint32_t>();
+    test<std::uint64_t>();
     test<uintmax_t>();
     test<uintptr_t>();
     test<size_t>();
diff --git a/libcxx/test/std/numerics/bit/bit.pow.two/bit_floor.pass.cpp b/libcxx/test/std/numerics/bit/bit.pow.two/bit_floor.pass.cpp
index f843fff5c2810..7b8d67c6152f0 100644
--- a/libcxx/test/std/numerics/bit/bit.pow.two/bit_floor.pass.cpp
+++ b/libcxx/test/std/numerics/bit/bit.pow.two/bit_floor.pass.cpp
@@ -81,10 +81,10 @@ int main(int, char**)
     static_assert(!std::is_invocable_v<L, __int128_t>);
 #endif
 
-    static_assert(!std::is_invocable_v<L, int8_t>);
-    static_assert(!std::is_invocable_v<L, int16_t>);
-    static_assert(!std::is_invocable_v<L, int32_t>);
-    static_assert(!std::is_invocable_v<L, int64_t>);
+    static_assert(!std::is_invocable_v<L, std::int8_t>);
+    static_assert(!std::is_invocable_v<L, std::int16_t>);
+    static_assert(!std::is_invocable_v<L, std::int32_t>);
+    static_assert(!std::is_invocable_v<L, std::int64_t>);
     static_assert(!std::is_invocable_v<L, intmax_t>);
     static_assert(!std::is_invocable_v<L, intptr_t>);
     static_assert(!std::is_invocable_v<L, ptrdiff_t>);
@@ -113,10 +113,10 @@ int main(int, char**)
     static_assert(test<__uint128_t>());
 #endif
 
-    static_assert(test<uint8_t>());
-    static_assert(test<uint16_t>());
-    static_assert(test<uint32_t>());
-    static_assert(test<uint64_t>());
+    static_assert(test<std::uint8_t>());
+    static_assert(test<std::uint16_t>());
+    static_assert(test<std::uint32_t>());
+    static_assert(test<std::uint64_t>());
     static_assert(test<uintmax_t>());
     static_assert(test<uintptr_t>());
     static_assert(test<size_t>());
@@ -130,10 +130,10 @@ int main(int, char**)
     test<__uint128_t>();
 #endif
 
-    test<uint8_t>();
-    test<uint16_t>();
-    test<uint32_t>();
-    test<uint64_t>();
+    test<std::uint8_t>();
+    test<std::uint16_t>();
+    test<std::uint32_t>();
+    test<std::uint64_t>();
     test<uintmax_t>();
     test<uintptr_t>();
     test<size_t>();
diff --git a/libcxx/test/std/numerics/bit/bit.pow.two/bit_width.pass.cpp b/libcxx/test/std/numerics/bit/bit.pow.two/bit_width.pass.cpp
index 23aad275eead8..899d272f56d7c 100644
--- a/libcxx/test/std/numerics/bit/bit.pow.two/bit_width.pass.cpp
+++ b/libcxx/test/std/numerics/bit/bit.pow.two/bit_width.pass.cpp
@@ -86,10 +86,10 @@ int main(int, char**)
     static_assert(!std::is_invocable_v<L, __int128_t>);
 #endif
 
-    static_assert(!std::is_invocable_v<L, int8_t>);
-    static_assert(!std::is_invocable_v<L, int16_t>);
-    static_assert(!std::is_invocable_v<L, int32_t>);
-    static_assert(!std::is_invocable_v<L, int64_t>);
+    static_assert(!std::is_invocable_v<L, std::int8_t>);
+    static_assert(!std::is_invocable_v<L, std::int16_t>);
+    static_assert(!std::is_invocable_v<L, std::int32_t>);
+    static_assert(!std::is_invocable_v<L, std::int64_t>);
     static_assert(!std::is_invocable_v<L, intmax_t>);
     static_assert(!std::is_invocable_v<L, intptr_t>);
     static_assert(!std::is_invocable_v<L, ptrdiff_t>);
@@ -117,10 +117,10 @@ int main(int, char**)
 #ifndef TEST_HAS_NO_INT128
     static_assert(test<__uint128_t>());
 #endif
-    static_assert(test<uint8_t>());
-    static_assert(test<uint16_t>());
-    static_assert(test<uint32_t>());
-    static_assert(test<uint64_t>());
+    static_assert(test<std::uint8_t>());
+    static_assert(test<std::uint16_t>());
+    static_assert(test<std::uint32_t>());
+    static_assert(test<std::uint64_t>());
     static_assert(test<uintmax_t>());
     static_assert(test<uintptr_t>());
     static_assert(test<size_t>());
@@ -133,10 +133,10 @@ int main(int, char**)
 #ifndef TEST_HAS_NO_INT128
     test<__uint128_t>();
 #endif
-    test<uint8_t>();
-    test<uint16_t>();
-    test<uint32_t>();
-    test<uint64_t>();
+    test<std::uint8_t>();
+    test<std::uint16_t>();
+    test<std::uint32_t>();
+    test<std::uint64_t>();
     test<uintmax_t>();
     test<uintptr_t>();
     test<size_t>();
diff --git a/libcxx/test/std/numerics/bit/bit.pow.two/has_single_bit.pass.cpp b/libcxx/test/std/numerics/bit/bit.pow.two/has_single_bit.pass.cpp
index 1a20a4ca37194..2c33bda72982c 100644
--- a/libcxx/test/std/numerics/bit/bit.pow.two/has_single_bit.pass.cpp
+++ b/libcxx/test/std/numerics/bit/bit.pow.two/has_single_bit.pass.cpp
@@ -84,10 +84,10 @@ int main(int, char**)
     static_assert(!std::is_invocable_v<L, __int128_t>);
 #endif
 
-    static_assert(!std::is_invocable_v<L, int8_t>);
-    static_assert(!std::is_invocable_v<L, int16_t>);
-    static_assert(!std::is_invocable_v<L, int32_t>);
-    static_assert(!std::is_invocable_v<L, int64_t>);
+    static_assert(!std::is_invocable_v<L, std::int8_t>);
+    static_assert(!std::is_invocable_v<L, std::int16_t>);
+    static_assert(!std::is_invocable_v<L, std::int32_t>);
+    static_assert(!std::is_invocable_v<L, std::int64_t>);
     static_assert(!std::is_invocable_v<L, intmax_t>);
     static_assert(!std::is_invocable_v<L, intptr_t>);
     static_assert(!std::is_invocable_v<L, ptrdiff_t>);
@@ -115,10 +115,10 @@ int main(int, char**)
 #ifndef TEST_HAS_NO_INT128
     static_assert(test<__uint128_t>());
 #endif
-    static_assert(test<uint8_t>());
-    static_assert(test<uint16_t>());
-    static_assert(test<uint32_t>());
-    static_assert(test<uint64_t>());
+    static_assert(test<std::uint8_t>());
+    static_assert(test<std::uint16_t>());
+    static_assert(test<std::uint32_t>());
+    static_assert(test<std::uint64_t>());
     static_assert(test<uintmax_t>());
     static_assert(test<uintptr_t>());
     static_assert(test<size_t>());
@@ -131,10 +131,10 @@ int main(int, char**)
 #ifndef TEST_HAS_NO_INT128
     test<__uint128_t>();
 #endif
-    test<uint8_t>();
-    test<uint16_t>();
-    test<uint32_t>();
-    test<uint64_t>();
+    test<std::uint8_t>();
+    test<std::uint16_t>();
+    test<std::uint32_t>();
+    test<std::uint64_t>();
     test<uintmax_t>();
     test<uintptr_t>();
     test<size_t>();
diff --git a/libcxx/test/std/numerics/bit/bitops.count/countl_one.pass.cpp b/libcxx/test/std/numerics/bit/bitops.count/countl_one.pass.cpp
index ab82d03224f38..f9def0a4d3407 100644
--- a/libcxx/test/std/numerics/bit/bitops.count/countl_one.pass.cpp
+++ b/libcxx/test/std/numerics/bit/bitops.count/countl_one.pass.cpp
@@ -81,10 +81,10 @@ int main(int, char**)
     static_assert(!std::is_invocable_v<L, __int128_t>);
 #endif
 
-    static_assert(!std::is_invocable_v<L, int8_t>);
-    static_assert(!std::is_invocable_v<L, int16_t>);
-    static_assert(!std::is_invocable_v<L, int32_t>);
-    static_assert(!std::is_invocable_v<L, int64_t>);
+    static_assert(!std::is_invocable_v<L, std::int8_t>);
+    static_assert(!std::is_invocable_v<L, std::int16_t>);
+    static_assert(!std::is_invocable_v<L, std::int32_t>);
+    static_assert(!std::is_invocable_v<L, std::int64_t>);
     static_assert(!std::is_invocable_v<L, intmax_t>);
     static_assert(!std::is_invocable_v<L, intptr_t>);
     static_assert(!std::is_invocable_v<L, ptrdiff_t>);
@@ -112,10 +112,10 @@ int main(int, char**)
 #ifndef TEST_HAS_NO_INT128
     static_assert(test<__uint128_t>());
 #endif
-    static_assert(test<uint8_t>());
-    static_assert(test<uint16_t>());
-    static_assert(test<uint32_t>());
-    static_assert(test<uint64_t>());
+    static_assert(test<std::uint8_t>());
+    static_assert(test<std::uint16_t>());
+    static_assert(test<std::uint32_t>());
+    static_assert(test<std::uint64_t>());
     static_assert(test<uintmax_t>());
     static_assert(test<uintptr_t>());
     static_assert(test<size_t>());
@@ -128,10 +128,10 @@ int main(int, char**)
 #ifndef TEST_HAS_NO_INT128
     test<__uint128_t>();
 #endif
-    test<uint8_t>();
-    test<uint16_t>();
-    test<uint32_t>();
-    test<uint64_t>();
+    test<std::uint8_t>();
+    test<std::uint16_t>();
+    test<std::uint32_t>();
+    test<std::uint64_t>();
     test<uintmax_t>();
     test<uintptr_t>();
     test<size_t>();
diff --git a/libcxx/test/std/numerics/bit/bitops.count/countl_zero.pass.cpp b/libcxx/test/std/numerics/bit/bitops.count/countl_zero.pass.cpp
index 2a23e9bd39bb7..b76731eeb0469 100644
--- a/libcxx/test/std/numerics/bit/bitops.count/countl_zero.pass.cpp
+++ b/libcxx/test/std/numerics/bit/bitops.count/countl_zero.pass.cpp
@@ -80,10 +80,10 @@ int main(int, char**)
     static_assert(!std::is_invocable_v<L, __int128_t>);
 #endif
 
-    static_assert(!std::is_invocable_v<L, int8_t>);
-    static_assert(!std::is_invocable_v<L, int16_t>);
-    static_assert(!std::is_invocable_v<L, int32_t>);
-    static_assert(!std::is_invocable_v<L, int64_t>);
+    static_assert(!std::is_invocable_v<L, std::int8_t>);
+    static_assert(!std::is_invocable_v<L, std::int16_t>);
+    static_assert(!std::is_invocable_v<L, std::int32_t>);
+    static_assert(!std::is_invocable_v<L, std::int64_t>);
     static_assert(!std::is_invocable_v<L, intmax_t>);
     static_assert(!std::is_invocable_v<L, intptr_t>);
     static_assert(!std::is_invocable_v<L, ptrdiff_t>);
@@ -111,10 +111,10 @@ int main(int, char**)
 #ifndef TEST_HAS_NO_INT128
     static_assert(test<__uint128_t>());
 #endif
-    static_assert(test<uint8_t>());
-    static_assert(test<uint16_t>());
-    static_assert(test<uint32_t>());
-    static_assert(test<uint64_t>());
+    static_assert(test<std::uint8_t>());
+    static_assert(test<std::uint16_t>());
+    static_assert(test<std::uint32_t>());
+    static_assert(test<std::uint64_t>());
     static_assert(test<uintmax_t>());
     static_assert(test<uintptr_t>());
     static_assert(test<size_t>());
@@ -127,10 +127,10 @@ int main(int, char**)
 #ifndef TEST_HAS_NO_INT128
     test<__uint128_t>();
 #endif
-    test<uint8_t>();
-    test<uint16_t>();
-    test<uint32_t>();
-    test<uint64_t>();
+    test<std::uint8_t>();
+    test<std::uint16_t>();
+    test<std::uint32_t>();
+    test<std::uint64_t>();
     test<uintmax_t>();
     test<uintptr_t>();
     test<size_t>();
diff --git a/libcxx/test/std/numerics/bit/bitops.count/countr_one.pass.cpp b/libcxx/test/std/numerics/bit/bitops.count/countr_one.pass.cpp
index 467d0340ef33b..c1a9a4a0ebb5f 100644
--- a/libcxx/test/std/numerics/bit/bitops.count/countr_one.pass.cpp
+++ b/libcxx/test/std/numerics/bit/bitops.count/countr_one.pass.cpp
@@ -85,10 +85,10 @@ int main(int, char**)
     static_assert(!std::is_invocable_v<L, __int128_t>);
 #endif
 
-    static_assert(!std::is_invocable_v<L, int8_t>);
-    static_assert(!std::is_invocable_v<L, int16_t>);
-    static_assert(!std::is_invocable_v<L, int32_t>);
-    static_assert(!std::is_invocable_v<L, int64_t>);
+    static_assert(!std::is_invocable_v<L, std::int8_t>);
+    static_assert(!std::is_invocable_v<L, std::int16_t>);
+    static_assert(!std::is_invocable_v<L, std::int32_t>);
+    static_assert(!std::is_invocable_v<L, std::int64_t>);
     static_assert(!std::is_invocable_v<L, intmax_t>);
     static_assert(!std::is_invocable_v<L, intptr_t>);
     static_assert(!std::is_invocable_v<L, ptrdiff_t>);
@@ -116,10 +116,10 @@ int main(int, char**)
 #ifndef TEST_HAS_NO_INT128
     static_assert(test<__uint128_t>());
 #endif
-    static_assert(test<uint8_t>());
-    static_assert(test<uint16_t>());
-    static_assert(test<uint32_t>());
-    static_assert(test<uint64_t>());
+    static_assert(test<std::uint8_t>());
+    static_assert(test<std::uint16_t>());
+    static_assert(test<std::uint32_t>());
+    static_assert(test<std::uint64_t>());
     static_assert(test<uintmax_t>());
     static_assert(test<uintptr_t>());
     static_assert(test<size_t>());
@@ -132,10 +132,10 @@ int main(int, char**)
 #ifndef TEST_HAS_NO_INT128
     test<__uint128_t>();
 #endif
-    test<uint8_t>();
-    test<uint16_t>();
-    test<uint32_t>();
-    test<uint64_t>();
+    test<std::uint8_t>();
+    test<std::uint16_t>();
+    test<std::uint32_t>();
+    test<std::uint64_t>();
     test<uintmax_t>();
     test<uintptr_t>();
     test<size_t>();
diff --git a/libcxx/test/std/numerics/bit/bitops.count/countr_zero.pass.cpp b/libcxx/test/std/numerics/bit/bitops.count/countr_zero.pass.cpp
index 149ca27a74a30..639144309454c 100644
--- a/libcxx/test/std/numerics/bit/bitops.count/countr_zero.pass.cpp
+++ b/libcxx/test/std/numerics/bit/bitops.count/countr_zero.pass.cpp
@@ -82,10 +82,10 @@ int main(int, char**)
     static_assert(!std::is_invocable_v<L, __int128_t>);
 #endif
 
-    static_assert(!std::is_invocable_v<L, int8_t>);
-    static_assert(!std::is_invocable_v<L, int16_t>);
-    static_assert(!std::is_invocable_v<L, int32_t>);
-    static_assert(!std::is_invocable_v<L, int64_t>);
+    static_assert(!std::is_invocable_v<L, std::int8_t>);
+    static_assert(!std::is_invocable_v<L, std::int16_t>);
+    static_assert(!std::is_invocable_v<L, std::int32_t>);
+    static_assert(!std::is_invocable_v<L, std::int64_t>);
     static_assert(!std::is_invocable_v<L, intmax_t>);
     static_assert(!std::is_invocable_v<L, intptr_t>);
     static_assert(!std::is_invocable_v<L, ptrdiff_t>);
@@ -113,10 +113,10 @@ int main(int, char**)
 #ifndef TEST_HAS_NO_INT128
     static_assert(test<__uint128_t>());
 #endif
-    static_assert(test<uint8_t>());
-    static_assert(test<uint16_t>());
-    static_assert(test<uint32_t>());
-    static_assert(test<uint64_t>());
+    static_assert(test<std::uint8_t>());
+    static_assert(test<std::uint16_t>());
+    static_assert(test<std::uint32_t>());
+    static_assert(test<std::uint64_t>());
     static_assert(test<uintmax_t>());
     static_assert(test<uintptr_t>());
     static_assert(test<size_t>());
@@ -129,10 +129,10 @@ int main(int, char**)
 #ifndef TEST_HAS_NO_INT128
     test<__uint128_t>();
 #endif
-    test<uint8_t>();
-    test<uint16_t>();
-    test<uint32_t>();
-    test<uint64_t>();
+    test<std::uint8_t>();
+    test<std::uint16_t>();
+    test<std::uint32_t>();
+    test<std::uint64_t>();
     test<uintmax_t>();
     test<uintptr_t>();
     test<size_t>();
diff --git a/libcxx/test/std/numerics/bit/bitops.count/popcount.pass.cpp b/libcxx/test/std/numerics/bit/bitops.count/popcount.pass.cpp
index 96c5fd41c3e56..6b996ed5177a9 100644
--- a/libcxx/test/std/numerics/bit/bitops.count/popcount.pass.cpp
+++ b/libcxx/test/std/numerics/bit/bitops.count/popcount.pass.cpp
@@ -92,10 +92,10 @@ int main(int, char**)
     static_assert(!std::is_invocable_v<L, __int128_t>);
 #endif
 
-    static_assert(!std::is_invocable_v<L, int8_t>);
-    static_assert(!std::is_invocable_v<L, int16_t>);
-    static_assert(!std::is_invocable_v<L, int32_t>);
-    static_assert(!std::is_invocable_v<L, int64_t>);
+    static_assert(!std::is_invocable_v<L, std::int8_t>);
+    static_assert(!std::is_invocable_v<L, std::int16_t>);
+    static_assert(!std::is_invocable_v<L, std::int32_t>);
+    static_assert(!std::is_invocable_v<L, std::int64_t>);
     static_assert(!std::is_invocable_v<L, intmax_t>);
     static_assert(!std::is_invocable_v<L, intptr_t>);
     static_assert(!std::is_invocable_v<L, ptrdiff_t>);
@@ -123,10 +123,10 @@ int main(int, char**)
 #ifndef TEST_HAS_NO_INT128
     static_assert(test<__uint128_t>());
 #endif
-    static_assert(test<uint8_t>());
-    static_assert(test<uint16_t>());
-    static_assert(test<uint32_t>());
-    static_assert(test<uint64_t>());
+    static_assert(test<std::uint8_t>());
+    static_assert(test<std::uint16_t>());
+    static_assert(test<std::uint32_t>());
+    static_assert(test<std::uint64_t>());
     static_assert(test<uintmax_t>());
     static_assert(test<uintptr_t>());
     static_assert(test<size_t>());
@@ -139,10 +139,10 @@ int main(int, char**)
 #ifndef TEST_HAS_NO_INT128
     test<__uint128_t>();
 #endif
-    test<uint8_t>();
-    test<uint16_t>();
-    test<uint32_t>();
-    test<uint64_t>();
+    test<std::uint8_t>();
+    test<std::uint16_t>();
+    test<std::uint32_t>();
+    test<std::uint64_t>();
     test<uintmax_t>();
     test<uintptr_t>();
     test<size_t>();
diff --git a/libcxx/test/std/numerics/bit/bitops.rot/rotl.pass.cpp b/libcxx/test/std/numerics/bit/bitops.rot/rotl.pass.cpp
index e82388d018924..34859ae922197 100644
--- a/libcxx/test/std/numerics/bit/bitops.rot/rotl.pass.cpp
+++ b/libcxx/test/std/numerics/bit/bitops.rot/rotl.pass.cpp
@@ -80,10 +80,10 @@ int main(int, char**)
     static_assert(!std::is_invocable_v<L, __int128_t>);
 #endif
 
-    static_assert(!std::is_invocable_v<L, int8_t>);
-    static_assert(!std::is_invocable_v<L, int16_t>);
-    static_assert(!std::is_invocable_v<L, int32_t>);
-    static_assert(!std::is_invocable_v<L, int64_t>);
+    static_assert(!std::is_invocable_v<L, std::int8_t>);
+    static_assert(!std::is_invocable_v<L, std::int16_t>);
+    static_assert(!std::is_invocable_v<L, std::int32_t>);
+    static_assert(!std::is_invocable_v<L, std::int64_t>);
     static_assert(!std::is_invocable_v<L, intmax_t>);
     static_assert(!std::is_invocable_v<L, intptr_t>);
     static_assert(!std::is_invocable_v<L, ptrdiff_t>);
@@ -112,10 +112,10 @@ int main(int, char**)
     static_assert(test<__uint128_t>());
 #endif
 
-    static_assert(test<uint8_t>());
-    static_assert(test<uint16_t>());
-    static_assert(test<uint32_t>());
-    static_assert(test<uint64_t>());
+    static_assert(test<std::uint8_t>());
+    static_assert(test<std::uint16_t>());
+    static_assert(test<std::uint32_t>());
+    static_assert(test<std::uint64_t>());
     static_assert(test<uintmax_t>());
     static_assert(test<uintptr_t>());
     static_assert(test<size_t>());
@@ -129,10 +129,10 @@ int main(int, char**)
     test<__uint128_t>();
 #endif
 
-    test<uint8_t>();
-    test<uint16_t>();
-    test<uint32_t>();
-    test<uint64_t>();
+    test<std::uint8_t>();
+    test<std::uint16_t>();
+    test<std::uint32_t>();
+    test<std::uint64_t>();
     test<uintmax_t>();
     test<uintptr_t>();
     test<size_t>();
diff --git a/libcxx/test/std/numerics/bit/bitops.rot/rotr.pass.cpp b/libcxx/test/std/numerics/bit/bitops.rot/rotr.pass.cpp
index 054587487f136..a6851e72a5ad6 100644
--- a/libcxx/test/std/numerics/bit/bitops.rot/rotr.pass.cpp
+++ b/libcxx/test/std/numerics/bit/bitops.rot/rotr.pass.cpp
@@ -81,10 +81,10 @@ int main(int, char**)
     static_assert(!std::is_invocable_v<L, __int128_t>);
 #endif
 
-    static_assert(!std::is_invocable_v<L, int8_t>);
-    static_assert(!std::is_invocable_v<L, int16_t>);
-    static_assert(!std::is_invocable_v<L, int32_t>);
-    static_assert(!std::is_invocable_v<L, int64_t>);
+    static_assert(!std::is_invocable_v<L, std::int8_t>);
+    static_assert(!std::is_invocable_v<L, std::int16_t>);
+    static_assert(!std::is_invocable_v<L, std::int32_t>);
+    static_assert(!std::is_invocable_v<L, std::int64_t>);
     static_assert(!std::is_invocable_v<L, intmax_t>);
     static_assert(!std::is_invocable_v<L, intptr_t>);
     static_assert(!std::is_invocable_v<L, ptrdiff_t>);
@@ -113,10 +113,10 @@ int main(int, char**)
     static_assert(test<__uint128_t>());
 #endif
 
-    static_assert(test<uint8_t>());
-    static_assert(test<uint16_t>());
-    static_assert(test<uint32_t>());
-    static_assert(test<uint64_t>());
+    static_assert(test<std::uint8_t>());
+    static_assert(test<std::uint16_t>());
+    static_assert(test<std::uint32_t>());
+    static_assert(test<std::uint64_t>());
     static_assert(test<uintmax_t>());
     static_assert(test<uintptr_t>());
     static_assert(test<size_t>());
@@ -130,10 +130,10 @@ int main(int, char**)
     test<__uint128_t>();
 #endif
 
-    test<uint8_t>();
-    test<uint16_t>();
-    test<uint32_t>();
-    test<uint64_t>();
+    test<std::uint8_t>();
+    test<std::uint16_t>();
+    test<std::uint32_t>();
+    test<std::uint64_t>();
     test<uintmax_t>();
     test<uintptr_t>();
     test<size_t>();
diff --git a/libcxx/test/std/numerics/bit/byteswap.pass.cpp b/libcxx/test/std/numerics/bit/byteswap.pass.cpp
index bd73515a918a2..b87faf150177f 100644
--- a/libcxx/test/std/numerics/bit/byteswap.pass.cpp
+++ b/libcxx/test/std/numerics/bit/byteswap.pass.cpp
@@ -54,15 +54,15 @@ constexpr void test_implementation_defined_size() {
 }
 
 constexpr bool test() {
-  test_num<uint8_t>(0xAB, 0xAB);
-  test_num<uint16_t>(0xCDEF, 0xEFCD);
-  test_num<uint32_t>(0x01234567, 0x67452301);
-  test_num<uint64_t>(0x0123456789ABCDEF, 0xEFCDAB8967452301);
+  test_num<std::uint8_t>(0xAB, 0xAB);
+  test_num<std::uint16_t>(0xCDEF, 0xEFCD);
+  test_num<std::uint32_t>(0x01234567, 0x67452301);
+  test_num<std::uint64_t>(0x0123456789ABCDEF, 0xEFCDAB8967452301);
 
-  test_num<int8_t>(static_cast<int8_t>(0xAB), static_cast<int8_t>(0xAB));
-  test_num<int16_t>(static_cast<int16_t>(0xCDEF), static_cast<int16_t>(0xEFCD));
-  test_num<int32_t>(0x01234567, 0x67452301);
-  test_num<int64_t>(0x0123456789ABCDEF, 0xEFCDAB8967452301);
+  test_num<std::int8_t>(static_cast<std::int8_t>(0xAB), static_cast<std::int8_t>(0xAB));
+  test_num<std::int16_t>(static_cast<std::int16_t>(0xCDEF), static_cast<std::int16_t>(0xEFCD));
+  test_num<std::int32_t>(0x01234567, 0x67452301);
+  test_num<std::int64_t>(0x0123456789ABCDEF, 0xEFCDAB8967452301);
 
 #ifndef TEST_HAS_NO_INT128
   const auto in = static_cast<__uint128_t>(0x0123456789ABCDEF) << 64 | 0x13579BDF02468ACE;
diff --git a/libcxx/test/std/numerics/numbers/illformed.verify.cpp b/libcxx/test/std/numerics/numbers/illformed.verify.cpp
index 213f98cff952f..8f19718bdeed3 100644
--- a/libcxx/test/std/numerics/numbers/illformed.verify.cpp
+++ b/libcxx/test/std/numerics/numbers/illformed.verify.cpp
@@ -11,8 +11,7 @@
 #include <numbers>
 
 // Initializing the primary template is ill-formed.
-int log2e{std::numbers::log2e_v<
-    int>}; // expected-error-re@numbers:* {{{{(static_assert|static assertion)}} failed{{.*}}A program that instantiates a primary template of a mathematical constant variable template is ill-formed.}}
+int log2e{std::numbers::log2e_v<int>}; // expected-error-re@numbers:* {{{{(static_assert|static assertion)}} failed{{.*}}A program that instantiates a primary template of a mathematical constant variable template is ill-formed.}}
 int log10e{std::numbers::log10e_v<int>};
 int pi{std::numbers::pi_v<int>};
 int inv_pi{std::numbers::inv_pi_v<int>};
@@ -24,5 +23,3 @@ int sqrt3{std::numbers::sqrt3_v<int>};
 int inv_sqrt3{std::numbers::inv_sqrt3_v<int>};
 int egamma{std::numbers::egamma_v<int>};
 int phi{std::numbers::phi_v<int>};
-
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/std/numerics/numeric.ops/numeric.ops.midpoint/midpoint.integer.pass.cpp b/libcxx/test/std/numerics/numeric.ops/numeric.ops.midpoint/midpoint.integer.pass.cpp
index 599b32848531b..1862045a4ec90 100644
--- a/libcxx/test/std/numerics/numeric.ops/numeric.ops.midpoint/midpoint.integer.pass.cpp
+++ b/libcxx/test/std/numerics/numeric.ops/numeric.ops.midpoint/midpoint.integer.pass.cpp
@@ -18,6 +18,7 @@
 #include <numeric>
 #include <cassert>
 #include <cstddef>
+#include <cstdint>
 #include "test_macros.h"
 
 template <typename T>
@@ -114,10 +115,10 @@ int main(int, char**)
     signed_test<long>();
     signed_test<long long>();
 
-    signed_test<int8_t>();
-    signed_test<int16_t>();
-    signed_test<int32_t>();
-    signed_test<int64_t>();
+    signed_test<std::int8_t>();
+    signed_test<std::int16_t>();
+    signed_test<std::int32_t>();
+    signed_test<std::int64_t>();
 
     unsigned_test<unsigned char>();
     unsigned_test<unsigned short>();
@@ -125,10 +126,10 @@ int main(int, char**)
     unsigned_test<unsigned long>();
     unsigned_test<unsigned long long>();
 
-    unsigned_test<uint8_t>();
-    unsigned_test<uint16_t>();
-    unsigned_test<uint32_t>();
-    unsigned_test<uint64_t>();
+    unsigned_test<std::uint8_t>();
+    unsigned_test<std::uint16_t>();
+    unsigned_test<std::uint32_t>();
+    unsigned_test<std::uint64_t>();
 
 #ifndef TEST_HAS_NO_INT128
     unsigned_test<__uint128_t>();
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval.pass.cpp
index 2e8bfaebf3677..dbdd097241194 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval.pass.cpp
@@ -15,10 +15,12 @@
 
 // template<class _URNG> result_type operator()(_URNG& g);
 
-#include <random>
+#include <cassert>
+#include <cstdint>
 #include <numeric>
+#include <random>
+#include <type_traits>
 #include <vector>
-#include <cassert>
 
 #include "test_macros.h"
 
@@ -527,8 +529,8 @@ int main(int, char**) {
     tests<unsigned long long>();
 
 #if defined(_LIBCPP_VERSION) // extension
-    tests<int8_t>();
-    tests<uint8_t>();
+    tests<std::int8_t>();
+    tests<std::uint8_t>();
 #if !defined(TEST_HAS_NO_INT128)
     tests<__int128_t>();
     tests<__uint128_t>();
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.geo/eval.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.geo/eval.pass.cpp
index deccad1db6d5b..440334ed3488a 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.geo/eval.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.geo/eval.pass.cpp
@@ -15,10 +15,11 @@
 
 // template<class _URNG> result_type operator()(_URNG& g);
 
-#include <random>
+#include <cassert>
+#include <cstdint>
 #include <numeric>
+#include <random>
 #include <vector>
-#include <cassert>
 
 #include "test_macros.h"
 
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.pois/rand.dist.pois.poisson/eval.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.pois/rand.dist.pois.poisson/eval.pass.cpp
index 6aef9b46e48cf..5feb2e5802291 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.pois/rand.dist.pois.poisson/eval.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.pois/rand.dist.pois.poisson/eval.pass.cpp
@@ -15,13 +15,13 @@
 
 // template<class _URNG> result_type operator()(_URNG& g);
 
-#include <random>
-
 #include <cassert>
 #include <cmath>
+#include <cstdint>
 #include <limits>
-#include <vector>
 #include <numeric>
+#include <random>
+#include <vector>
 
 #include "test_macros.h"
 
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.samp/rand.dist.samp.discrete/eval.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.samp/rand.dist.samp.discrete/eval.pass.cpp
index d52e23e9dcd4a..578a66c7c1c0c 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.samp/rand.dist.samp.discrete/eval.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.samp/rand.dist.samp.discrete/eval.pass.cpp
@@ -15,9 +15,10 @@
 
 // template<class _URNG> result_type operator()(_URNG& g);
 
+#include <cassert>
+#include <cstdint>
 #include <random>
 #include <vector>
-#include <cassert>
 
 #include "test_macros.h"
 
@@ -305,8 +306,8 @@ int main(int, char**) {
     tests<unsigned long long>();
 
 #if defined(_LIBCPP_VERSION) // extension
-    tests<int8_t>();
-    tests<uint8_t>();
+    tests<std::int8_t>();
+    tests<std::uint8_t>();
 #if !defined(TEST_HAS_NO_INT128)
     tests<__int128_t>();
     tests<__uint128_t>();
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.uni/rand.dist.uni.int/eval.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.uni/rand.dist.uni.int/eval.pass.cpp
index d636c4c955c36..669f90d5292df 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.uni/rand.dist.uni.int/eval.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.uni/rand.dist.uni.int/eval.pass.cpp
@@ -15,12 +15,13 @@
 
 // template<class _URNG> result_type operator()(_URNG& g);
 
-#include <random>
 #include <cassert>
 #include <climits>
 #include <cstddef>
+#include <cstdint>
 #include <limits>
 #include <numeric>
+#include <random>
 #include <vector>
 
 #include "test_macros.h"
@@ -123,8 +124,8 @@ int main(int, char**)
     test_statistics<short, std::minstd_rand0>(SHRT_MIN, SHRT_MAX);
 
 #if defined(_LIBCPP_VERSION) // extension
-    test_statistics<int8_t, std::minstd_rand0>();
-    test_statistics<uint8_t, std::minstd_rand0>();
+    test_statistics<std::int8_t, std::minstd_rand0>();
+    test_statistics<std::uint8_t, std::minstd_rand0>();
 
 #if !defined(TEST_HAS_NO_INT128) && !defined(TEST_BUGGY_I128_FP)
     test_statistics<__int128_t, std::minstd_rand0>();
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.uni/rand.dist.uni.int/int128.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.uni/rand.dist.uni.int/int128.pass.cpp
index 9b5220522b055..9d499dbe22cb5 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.uni/rand.dist.uni.int/int128.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.uni/rand.dist.uni.int/int128.pass.cpp
@@ -15,10 +15,11 @@
 
 // template<class _URNG> result_type operator()(_URNG& g);
 
-#include <random>
 
 #include <cassert>
+#include <cstdint>
 #include <limits>
+#include <random>
 
 #include "test_macros.h"
 
diff --git a/libcxx/test/std/numerics/rand/rand.eng/rand.eng.mers/ctor_result_type.pass.cpp b/libcxx/test/std/numerics/rand/rand.eng/rand.eng.mers/ctor_result_type.pass.cpp
index a8140566f07af..02df0345525cb 100644
--- a/libcxx/test/std/numerics/rand/rand.eng/rand.eng.mers/ctor_result_type.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.eng/rand.eng.mers/ctor_result_type.pass.cpp
@@ -267,8 +267,8 @@ int main(int, char**)
     test1();
     test2();
 
-    test_implicit<std::mt19937, uint_fast32_t>();
-    test_implicit<std::mt19937_64, uint_fast64_t>();
+    test_implicit<std::mt19937, std::uint_fast32_t>();
+    test_implicit<std::mt19937_64, std::uint_fast64_t>();
 
     return 0;
 }
diff --git a/libcxx/test/std/numerics/rand/rand.eng/rand.eng.sub/ctor_result_type.pass.cpp b/libcxx/test/std/numerics/rand/rand.eng/rand.eng.sub/ctor_result_type.pass.cpp
index 291dbf70ff1e1..3c7ab48c8db0d 100644
--- a/libcxx/test/std/numerics/rand/rand.eng/rand.eng.sub/ctor_result_type.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.eng/rand.eng.sub/ctor_result_type.pass.cpp
@@ -76,8 +76,8 @@ int main(int, char**)
     static_assert(test_convertible<std::ranlux48_base>(), "");
     test_implicit_ctor<std::ranlux24_base>();
     test_implicit_ctor<std::ranlux48_base>();
-    static_assert(!test_convertible<std::ranlux24_base, uint_fast32_t>(), "");
-    static_assert(!test_convertible<std::ranlux48_base, uint_fast64_t>(), "");
+    static_assert(!test_convertible<std::ranlux24_base, std::uint_fast32_t>(), "");
+    static_assert(!test_convertible<std::ranlux48_base, std::uint_fast64_t>(), "");
 #endif
 
   return 0;
diff --git a/libcxx/test/std/numerics/rand/rand.req/rand.req.urng/uniform_random_bit_generator.compile.pass.cpp b/libcxx/test/std/numerics/rand/rand.req/rand.req.urng/uniform_random_bit_generator.compile.pass.cpp
index bfda9b79d56a9..9f2fe28d45415 100644
--- a/libcxx/test/std/numerics/rand/rand.req/rand.req.urng/uniform_random_bit_generator.compile.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.req/rand.req.urng/uniform_random_bit_generator.compile.pass.cpp
@@ -140,5 +140,3 @@ struct Works {
   static constexpr unsigned char max() { return 1; }
 };
 static_assert(std::uniform_random_bit_generator<Works>);
-
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/std/ranges/range.adaptors/range.as.rvalue/ctor.pass.cpp b/libcxx/test/std/ranges/range.adaptors/range.as.rvalue/ctor.pass.cpp
index 48321391dccb5..d9b14e4820abc 100644
--- a/libcxx/test/std/ranges/range.adaptors/range.as.rvalue/ctor.pass.cpp
+++ b/libcxx/test/std/ranges/range.adaptors/range.as.rvalue/ctor.pass.cpp
@@ -12,6 +12,7 @@
 
 #include <cassert>
 #include <ranges>
+#include <type_traits>
 #include <vector>
 
 struct DefaultConstructibleView : std::ranges::view_base {
diff --git a/libcxx/test/std/ranges/range.adaptors/range.take/ctor.default.pass.cpp b/libcxx/test/std/ranges/range.adaptors/range.take/ctor.default.pass.cpp
index 33071a94ad49b..98fd1e02ed7e1 100644
--- a/libcxx/test/std/ranges/range.adaptors/range.take/ctor.default.pass.cpp
+++ b/libcxx/test/std/ranges/range.adaptors/range.take/ctor.default.pass.cpp
@@ -10,8 +10,9 @@
 
 // take_view() requires default_initializable<V> = default;
 
-#include <ranges>
 #include <cassert>
+#include <ranges>
+#include <type_traits>
 
 int buff[8] = {1, 2, 3, 4, 5, 6, 7, 8};
 
diff --git a/libcxx/test/std/ranges/range.factories/range.iota.view/ctor.value.pass.cpp b/libcxx/test/std/ranges/range.factories/range.iota.view/ctor.value.pass.cpp
index ac5a25b9766df..cda372e1b7a35 100644
--- a/libcxx/test/std/ranges/range.factories/range.iota.view/ctor.value.pass.cpp
+++ b/libcxx/test/std/ranges/range.factories/range.iota.view/ctor.value.pass.cpp
@@ -10,8 +10,9 @@
 
 // constexpr explicit iota_view(W value);
 
-#include <ranges>
 #include <cassert>
+#include <ranges>
+#include <type_traits>
 
 #include "test_macros.h"
 #include "types.h"
diff --git a/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/ctor.value.pass.cpp b/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/ctor.value.pass.cpp
index 5a71837b0948d..a84cb91f7e9b3 100644
--- a/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/ctor.value.pass.cpp
+++ b/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/ctor.value.pass.cpp
@@ -10,8 +10,9 @@
 
 // constexpr explicit iterator(W value);
 
-#include <ranges>
 #include <cassert>
+#include <ranges>
+#include <type_traits>
 
 #include "test_macros.h"
 #include "../types.h"
diff --git a/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/decrement.pass.cpp b/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/decrement.pass.cpp
index 45b5e1761d2c2..3b1613e62193f 100644
--- a/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/decrement.pass.cpp
+++ b/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/decrement.pass.cpp
@@ -11,8 +11,9 @@
 // constexpr iterator& operator--() requires decrementable<W>;
 // constexpr iterator operator--(int) requires decrementable<W>;
 
-#include <ranges>
 #include <cassert>
+#include <ranges>
+#include <type_traits>
 
 #include "test_macros.h"
 #include "../types.h"
diff --git a/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/increment.pass.cpp b/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/increment.pass.cpp
index a979a2d42b301..d99e57f4b7aa1 100644
--- a/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/increment.pass.cpp
+++ b/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/increment.pass.cpp
@@ -12,8 +12,9 @@
 // constexpr void operator++(int);
 // constexpr iterator operator++(int) requires incrementable<W>;
 
-#include <ranges>
 #include <cassert>
+#include <ranges>
+#include <type_traits>
 
 #include "test_macros.h"
 #include "../types.h"
diff --git a/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/member_typedefs.compile.pass.cpp b/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/member_typedefs.compile.pass.cpp
index e009bfcfb70e3..c2f7fd14042a8 100644
--- a/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/member_typedefs.compile.pass.cpp
+++ b/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/member_typedefs.compile.pass.cpp
@@ -13,6 +13,7 @@
 #include <cassert>
 #include <cstdint>
 #include <ranges>
+#include <type_traits>
 
 #include "test_macros.h"
 #include "../types.h"
diff --git a/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/minus.pass.cpp b/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/minus.pass.cpp
index ff425baa3e359..705a744cb5978 100644
--- a/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/minus.pass.cpp
+++ b/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/minus.pass.cpp
@@ -16,6 +16,7 @@
 #include <cassert>
 #include <cstdint>
 #include <ranges>
+#include <type_traits>
 
 #include "test_macros.h"
 #include "../types.h"
diff --git a/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/minus_eq.pass.cpp b/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/minus_eq.pass.cpp
index 6b169826d63f6..b46a34dbc8907 100644
--- a/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/minus_eq.pass.cpp
+++ b/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/minus_eq.pass.cpp
@@ -11,8 +11,9 @@
 // constexpr iterator& operator-=(difference_type n)
 //   requires advanceable<W>;
 
-#include <ranges>
 #include <cassert>
+#include <ranges>
+#include <type_traits>
 
 #include "test_macros.h"
 #include "../types.h"
diff --git a/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/plus.pass.cpp b/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/plus.pass.cpp
index aaff730ee7d9e..b66dc78784217 100644
--- a/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/plus.pass.cpp
+++ b/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/plus.pass.cpp
@@ -13,8 +13,9 @@
 // friend constexpr iterator operator+(difference_type n, iterator i)
 //   requires advanceable<W>;
 
-#include <ranges>
 #include <cassert>
+#include <ranges>
+#include <type_traits>
 
 #include "test_macros.h"
 #include "../types.h"
diff --git a/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/plus_eq.pass.cpp b/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/plus_eq.pass.cpp
index ded3c975e4196..fa9135d179958 100644
--- a/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/plus_eq.pass.cpp
+++ b/libcxx/test/std/ranges/range.factories/range.iota.view/iterator/plus_eq.pass.cpp
@@ -11,8 +11,9 @@
 // constexpr iterator& operator+=(difference_type n)
 //   requires advanceable<W>;
 
-#include <ranges>
 #include <cassert>
+#include <ranges>
+#include <type_traits>
 
 #include "test_macros.h"
 #include "../types.h"
diff --git a/libcxx/test/std/ranges/range.factories/range.iota.view/sentinel/ctor.value.pass.cpp b/libcxx/test/std/ranges/range.factories/range.iota.view/sentinel/ctor.value.pass.cpp
index c465345d5d506..a56369b046dc1 100644
--- a/libcxx/test/std/ranges/range.factories/range.iota.view/sentinel/ctor.value.pass.cpp
+++ b/libcxx/test/std/ranges/range.factories/range.iota.view/sentinel/ctor.value.pass.cpp
@@ -10,8 +10,9 @@
 
 // constexpr explicit sentinel(Bound bound);
 
-#include <ranges>
 #include <cassert>
+#include <ranges>
+#include <type_traits>
 
 #include "test_macros.h"
 #include "../types.h"
diff --git a/libcxx/test/std/ranges/range.factories/range.iota.view/views_iota.pass.cpp b/libcxx/test/std/ranges/range.factories/range.iota.view/views_iota.pass.cpp
index d7310cc252e8a..a6b268fd6a8d8 100644
--- a/libcxx/test/std/ranges/range.factories/range.iota.view/views_iota.pass.cpp
+++ b/libcxx/test/std/ranges/range.factories/range.iota.view/views_iota.pass.cpp
@@ -10,9 +10,10 @@
 
 // views::iota
 
-#include <ranges>
 #include <cassert>
 #include <concepts>
+#include <ranges>
+#include <type_traits>
 
 #include "test_macros.h"
 #include "types.h"
diff --git a/libcxx/test/std/re/re.results/re.results.size/empty.verify.cpp b/libcxx/test/std/re/re.results/re.results.size/empty.verify.cpp
index 738ed88a57558..4087c24024ce3 100644
--- a/libcxx/test/std/re/re.results/re.results.size/empty.verify.cpp
+++ b/libcxx/test/std/re/re.results/re.results.size/empty.verify.cpp
@@ -15,12 +15,7 @@
 
 #include <regex>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::match_results<const char*> c;
     c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/strings/basic.string.literals/noexcept.compile.pass.cpp b/libcxx/test/std/strings/basic.string.literals/noexcept.compile.pass.cpp
index 7f040e58269a3..8e2cae74f3e02 100644
--- a/libcxx/test/std/strings/basic.string.literals/noexcept.compile.pass.cpp
+++ b/libcxx/test/std/strings/basic.string.literals/noexcept.compile.pass.cpp
@@ -9,6 +9,7 @@
 // UNSUPPORTED: c++03, c++11
 
 #include <string>
+#include <utility>
 
 #include "test_macros.h"
 
diff --git a/libcxx/test/std/strings/basic.string/string.capacity/empty.verify.cpp b/libcxx/test/std/strings/basic.string/string.capacity/empty.verify.cpp
index 26a4daf6047bd..8ba11ee406bcd 100644
--- a/libcxx/test/std/strings/basic.string/string.capacity/empty.verify.cpp
+++ b/libcxx/test/std/strings/basic.string/string.capacity/empty.verify.cpp
@@ -16,21 +16,7 @@
 
 #include <string>
 
-#include "test_macros.h"
-
-TEST_CONSTEXPR_CXX20 bool test() {
+void f() {
   std::string c;
   c.empty(); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-  return true;
-}
-
-int main(int, char**)
-{
-  test();
-#if TEST_STD_VER > 17
-  static_assert(test());
-#endif
-
-  return 0;
 }
diff --git a/libcxx/test/std/strings/basic.string/string.capacity/reserve.deprecated_in_cxx20.verify.cpp b/libcxx/test/std/strings/basic.string/string.capacity/reserve.deprecated_in_cxx20.verify.cpp
index 1b8a5da964587..632f7fd0523a1 100644
--- a/libcxx/test/std/strings/basic.string/string.capacity/reserve.deprecated_in_cxx20.verify.cpp
+++ b/libcxx/test/std/strings/basic.string/string.capacity/reserve.deprecated_in_cxx20.verify.cpp
@@ -14,9 +14,7 @@
 
 #include <string>
 
-int main(int, char**)
-{
+void f() {
     std::string s;
     s.reserve(); // expected-warning {{'reserve' is deprecated}}
-    return 0;
 }
diff --git a/libcxx/test/std/strings/basic.string/string.modifiers/string_append/initializer_list.pass.cpp b/libcxx/test/std/strings/basic.string/string.modifiers/string_append/initializer_list.pass.cpp
index ebe4e49e16cdc..6c5049f16bc61 100644
--- a/libcxx/test/std/strings/basic.string/string.modifiers/string_append/initializer_list.pass.cpp
+++ b/libcxx/test/std/strings/basic.string/string.modifiers/string_append/initializer_list.pass.cpp
@@ -17,25 +17,37 @@
 
 #include "test_macros.h"
 #include "min_allocator.h"
+#include "nasty_string.h"
+
+template <class S>
+TEST_CONSTEXPR_CXX20 void test() {
+  using CharT = typename S::value_type;
+
+  S s(CONVERT_TO_CSTRING(CharT, "123"));
+  s.append({CharT('a'), CharT('b'), CharT('c')});
+  assert(s == CONVERT_TO_CSTRING(CharT, "123abc"));
+}
 
 TEST_CONSTEXPR_CXX20 bool test() {
-  {
-    std::string s("123");
-    s.append({'a', 'b', 'c'});
-    assert(s == "123abc");
-  }
-  {
-    typedef std::basic_string<char, std::char_traits<char>, min_allocator<char>> S;
-    S s("123");
-    s.append({'a', 'b', 'c'});
-    assert(s == "123abc");
-  }
+  test<std::string>();
+#ifndef TEST_HAS_NO_WIDE_CHARACTERS
+  test<std::wstring>();
+#endif
+#if TEST_STD_VER >= 20
+  test<std::u8string>();
+#endif
+  test<std::u16string>();
+  test<std::u32string>();
+
+  test<std::basic_string<char, std::char_traits<char>, min_allocator<char>>>();
+#ifndef TEST_HAS_NO_NASTY_STRING
+  test<nasty_string>();
+#endif
 
   return true;
 }
 
-int main(int, char**)
-{
+int main(int, char**) {
   test();
 #if TEST_STD_VER > 17
   static_assert(test());
diff --git a/libcxx/test/std/strings/basic.string/string.modifiers/string_append/iterator.pass.cpp b/libcxx/test/std/strings/basic.string/string.modifiers/string_append/iterator.pass.cpp
index 83b8d8a1147a6..cf74053136474 100644
--- a/libcxx/test/std/strings/basic.string/string.modifiers/string_append/iterator.pass.cpp
+++ b/libcxx/test/std/strings/basic.string/string.modifiers/string_append/iterator.pass.cpp
@@ -154,7 +154,7 @@ TEST_CONSTEXPR_CXX20 bool test() {
 
   { // test appending a different type
     typedef std::string S;
-    const uint8_t p[] = "ABCD";
+    const std::uint8_t p[] = "ABCD";
 
     S s;
     s.append(p, p + 4);
diff --git a/libcxx/test/std/strings/basic.string/string.modifiers/string_assign/iterator.pass.cpp b/libcxx/test/std/strings/basic.string/string.modifiers/string_assign/iterator.pass.cpp
index 3b8d0dcc0a6b0..9af2f12bdb308 100644
--- a/libcxx/test/std/strings/basic.string/string.modifiers/string_assign/iterator.pass.cpp
+++ b/libcxx/test/std/strings/basic.string/string.modifiers/string_assign/iterator.pass.cpp
@@ -155,7 +155,7 @@ TEST_CONSTEXPR_CXX20 bool test() {
 
   { // test assigning a different type
     typedef std::string S;
-    const uint8_t p[] = "ABCD";
+    const std::uint8_t p[] = "ABCD";
 
     S s;
     s.assign(p, p + 4);
diff --git a/libcxx/test/std/strings/basic.string/string.modifiers/string_assign/string.pass.cpp b/libcxx/test/std/strings/basic.string/string.modifiers/string_assign/string.pass.cpp
index c0822d403bc6e..99503741743f7 100644
--- a/libcxx/test/std/strings/basic.string/string.modifiers/string_assign/string.pass.cpp
+++ b/libcxx/test/std/strings/basic.string/string.modifiers/string_assign/string.pass.cpp
@@ -15,91 +15,89 @@
 #include <cassert>
 
 #include "test_macros.h"
+#include "nasty_string.h"
 #include "min_allocator.h"
 #include "test_allocator.h"
 
 template <class S>
-TEST_CONSTEXPR_CXX20 void
-test(S s, S str, S expected)
-{
-    s.assign(str);
-    LIBCPP_ASSERT(s.__invariants());
-    assert(s == expected);
+TEST_CONSTEXPR_CXX20 void test(S dest, S src) {
+  dest.assign(src);
+  LIBCPP_ASSERT(dest.__invariants());
+  assert(dest == src);
 }
 
 template <class S>
 TEST_CONSTEXPR_CXX20 void
-testAlloc(S s, S str, const typename S::allocator_type& a)
+testAlloc(S dest, S src, const typename S::allocator_type& a)
 {
-    s.assign(str);
-    LIBCPP_ASSERT(s.__invariants());
-    assert(s == str);
-    assert(s.get_allocator() == a);
+    dest.assign(src);
+    LIBCPP_ASSERT(dest.__invariants());
+    assert(dest == src);
+    assert(dest.get_allocator() == a);
+}
+
+template <class S>
+TEST_CONSTEXPR_CXX20 void test_assign() {
+    using CharT = typename S::value_type;
+
+    test(S(), S());
+    test(S(), S(CONVERT_TO_CSTRING(CharT, "12345")));
+    test(S(), S(CONVERT_TO_CSTRING(CharT, "1234567890")));
+    test(S(), S(CONVERT_TO_CSTRING(CharT, "12345678901234567890")));
+
+    test(S(CONVERT_TO_CSTRING(CharT, "12345")), S());
+    test(S(CONVERT_TO_CSTRING(CharT, "12345")), S(CONVERT_TO_CSTRING(CharT, "12345")));
+    test(S(CONVERT_TO_CSTRING(CharT, "12345")), S(CONVERT_TO_CSTRING(CharT, "1234567890")));
+    test(S(CONVERT_TO_CSTRING(CharT, "12345")), S(CONVERT_TO_CSTRING(CharT, "12345678901234567890")));
+
+    test(S(CONVERT_TO_CSTRING(CharT, "1234567890")), S());
+    test(S(CONVERT_TO_CSTRING(CharT, "1234567890")), S(CONVERT_TO_CSTRING(CharT, "12345")));
+    test(S(CONVERT_TO_CSTRING(CharT, "1234567890")), S(CONVERT_TO_CSTRING(CharT, "1234567890")));
+    test(S(CONVERT_TO_CSTRING(CharT, "1234567890")), S(CONVERT_TO_CSTRING(CharT, "12345678901234567890")));
+
+    test(S(CONVERT_TO_CSTRING(CharT, "12345678901234567890")), S());
+    test(S(CONVERT_TO_CSTRING(CharT, "12345678901234567890")), S(CONVERT_TO_CSTRING(CharT, "12345")));
+    test(S(CONVERT_TO_CSTRING(CharT, "12345678901234567890")), S(CONVERT_TO_CSTRING(CharT, "1234567890")));
+    test(S(CONVERT_TO_CSTRING(CharT, "12345678901234567890")), S(CONVERT_TO_CSTRING(CharT, "12345678901234567890")));
 }
 
 TEST_CONSTEXPR_CXX20 bool test() {
-  {
-    typedef std::string S;
-    test(S(), S(), S());
-    test(S(), S("12345"), S("12345"));
-    test(S(), S("1234567890"), S("1234567890"));
-    test(S(), S("12345678901234567890"), S("12345678901234567890"));
-
-    test(S("12345"), S(), S());
-    test(S("12345"), S("12345"), S("12345"));
-    test(S("12345"), S("1234567890"), S("1234567890"));
-    test(S("12345"), S("12345678901234567890"), S("12345678901234567890"));
-
-    test(S("1234567890"), S(), S());
-    test(S("1234567890"), S("12345"), S("12345"));
-    test(S("1234567890"), S("1234567890"), S("1234567890"));
-    test(S("1234567890"), S("12345678901234567890"), S("12345678901234567890"));
-
-    test(S("12345678901234567890"), S(), S());
-    test(S("12345678901234567890"), S("12345"), S("12345"));
-    test(S("12345678901234567890"), S("1234567890"), S("1234567890"));
-    test(S("12345678901234567890"), S("12345678901234567890"),
-         S("12345678901234567890"));
-
-    testAlloc(S(), S(), std::allocator<char>());
-    testAlloc(S(), S("12345"), std::allocator<char>());
-    testAlloc(S(), S("1234567890"), std::allocator<char>());
-    testAlloc(S(), S("12345678901234567890"), std::allocator<char>());
-  }
+    test_assign<std::string>();
+#ifndef TEST_HAS_NO_WIDE_CHARACTERS
+    test_assign<std::wstring>();
+#endif
+#if TEST_STD_VER >= 20
+    test_assign<std::u8string>();
+#endif
+#if TEST_STD_VER >= 11
+    test_assign<std::u16string>();
+    test_assign<std::u32string>();
+#endif
+#ifndef TEST_HAS_NO_NASTY_STRING
+    test_assign<nasty_string>();
+#endif
 
-  { //  LWG#5579 make sure assign takes the allocators where appropriate
-    typedef other_allocator<char> A;  // has POCCA --> true
-    typedef std::basic_string<char, std::char_traits<char>, A> S;
-    testAlloc(S(A(5)), S(A(3)), A(3));
-    testAlloc(S(A(5)), S("1"), A());
-    testAlloc(S(A(5)), S("1", A(7)), A(7));
-    testAlloc(S(A(5)), S("1234567890123456789012345678901234567890123456789012345678901234567890", A(7)), A(7));
-  }
+    {
+      typedef std::string S;
+      testAlloc(S(), S(), std::allocator<char>());
+      testAlloc(S(), S("12345"), std::allocator<char>());
+      testAlloc(S(), S("1234567890"), std::allocator<char>());
+      testAlloc(S(), S("12345678901234567890"), std::allocator<char>());
+    }
+
+    {                                  //  LWG#5579 make sure assign takes the allocators where appropriate
+      typedef other_allocator<char> A; // has POCCA --> true
+      typedef std::basic_string<char, std::char_traits<char>, A> S;
+      testAlloc(S(A(5)), S(A(3)), A(3));
+      testAlloc(S(A(5)), S("1"), A());
+      testAlloc(S(A(5)), S("1", A(7)), A(7));
+      testAlloc(S(A(5)), S("1234567890123456789012345678901234567890123456789012345678901234567890", A(7)), A(7));
+    }
 
 #if TEST_STD_VER >= 11
   {
     typedef std::basic_string<char, std::char_traits<char>, min_allocator<char>> S;
-    test(S(), S(), S());
-    test(S(), S("12345"), S("12345"));
-    test(S(), S("1234567890"), S("1234567890"));
-    test(S(), S("12345678901234567890"), S("12345678901234567890"));
-
-    test(S("12345"), S(), S());
-    test(S("12345"), S("12345"), S("12345"));
-    test(S("12345"), S("1234567890"), S("1234567890"));
-    test(S("12345"), S("12345678901234567890"), S("12345678901234567890"));
-
-    test(S("1234567890"), S(), S());
-    test(S("1234567890"), S("12345"), S("12345"));
-    test(S("1234567890"), S("1234567890"), S("1234567890"));
-    test(S("1234567890"), S("12345678901234567890"), S("12345678901234567890"));
-
-    test(S("12345678901234567890"), S(), S());
-    test(S("12345678901234567890"), S("12345"), S("12345"));
-    test(S("12345678901234567890"), S("1234567890"), S("1234567890"));
-    test(S("12345678901234567890"), S("12345678901234567890"),
-         S("12345678901234567890"));
-
+    test_assign<S>();
     testAlloc(S(), S(), min_allocator<char>());
     testAlloc(S(), S("12345"), min_allocator<char>());
     testAlloc(S(), S("1234567890"), min_allocator<char>());
diff --git a/libcxx/test/std/strings/basic.string/string.modifiers/string_insert/iter_iter_iter.pass.cpp b/libcxx/test/std/strings/basic.string/string.modifiers/string_insert/iter_iter_iter.pass.cpp
index fb32892d10ada..d7f3ca3c4d4ca 100644
--- a/libcxx/test/std/strings/basic.string/string.modifiers/string_insert/iter_iter_iter.pass.cpp
+++ b/libcxx/test/std/strings/basic.string/string.modifiers/string_insert/iter_iter_iter.pass.cpp
@@ -145,7 +145,7 @@ TEST_CONSTEXPR_CXX20 bool test() {
 
   { // test assigning a different type
     typedef std::string S;
-    const uint8_t p[] = "ABCD";
+    const std::uint8_t p[] = "ABCD";
 
     S s;
     s.insert(s.begin(), p, p + 4);
diff --git a/libcxx/test/std/strings/basic.string/string.modifiers/string_replace/iter_iter_iter_iter.pass.cpp b/libcxx/test/std/strings/basic.string/string.modifiers/string_replace/iter_iter_iter_iter.pass.cpp
index 5a1282cb0c097..c70a43e2ca81e 100644
--- a/libcxx/test/std/strings/basic.string/string.modifiers/string_replace/iter_iter_iter_iter.pass.cpp
+++ b/libcxx/test/std/strings/basic.string/string.modifiers/string_replace/iter_iter_iter_iter.pass.cpp
@@ -1028,8 +1028,8 @@ TEST_CONSTEXPR_CXX20 bool test9() {
   }
 
   { // test assigning a different type
-    const uint8_t pc[] = "ABCD";
-    uint8_t        p[] = "EFGH";
+    const std::uint8_t pc[] = "ABCD";
+    std::uint8_t        p[] = "EFGH";
 
     S s;
     s.replace(s.begin(), s.end(), pc, pc + 4);
diff --git a/libcxx/test/std/strings/string.view/string.view.capacity/empty.verify.cpp b/libcxx/test/std/strings/string.view/string.view.capacity/empty.verify.cpp
index 9e6af24fdd5dd..73e0af7cb5e9c 100644
--- a/libcxx/test/std/strings/string.view/string.view.capacity/empty.verify.cpp
+++ b/libcxx/test/std/strings/string.view/string.view.capacity/empty.verify.cpp
@@ -16,12 +16,7 @@
 
 #include <string_view>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::string_view c;
     c.empty();  // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/strings/string.view/string.view.io/stream_insert_decl_present.compile.pass.cpp b/libcxx/test/std/strings/string.view/string.view.io/stream_insert_decl_present.compile.pass.cpp
index 42a83f8637cb0..9698ba514069b 100644
--- a/libcxx/test/std/strings/string.view/string.view.io/stream_insert_decl_present.compile.pass.cpp
+++ b/libcxx/test/std/strings/string.view/string.view.io/stream_insert_decl_present.compile.pass.cpp
@@ -15,8 +15,9 @@
 //   operator<<(basic_ostream<charT, traits>& os,
 //              const basic_string_view<charT,traits> str);
 
-#include <string_view>
 #include <iosfwd>
+#include <string_view>
+#include <utility>
 
 template <class SV, class = void>
 struct HasDecl : std::false_type {};
diff --git a/libcxx/test/std/thread/futures/futures.async/async.verify.cpp b/libcxx/test/std/thread/futures/futures.async/async.verify.cpp
index 997d27a8ceacc..e47ad82e79980 100644
--- a/libcxx/test/std/thread/futures/futures.async/async.verify.cpp
+++ b/libcxx/test/std/thread/futures/futures.async/async.verify.cpp
@@ -21,18 +21,10 @@
 
 
 #include <future>
-#include <atomic>
-#include <memory>
-#include <cassert>
-
-#include "test_macros.h"
 
 int foo (int x) { return x; }
 
-int main(int, char**)
-{
+void f() {
     std::async(                    foo, 3); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
     std::async(std::launch::async, foo, 3); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/thread/futures/futures.promise/copy_assign.verify.cpp b/libcxx/test/std/thread/futures/futures.promise/copy_assign.verify.cpp
index 96d7e572f2a06..84fa1c439cf69 100644
--- a/libcxx/test/std/thread/futures/futures.promise/copy_assign.verify.cpp
+++ b/libcxx/test/std/thread/futures/futures.promise/copy_assign.verify.cpp
@@ -16,10 +16,7 @@
 
 #include <future>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     {
         std::promise<int> p0, p;
         p = p0; // expected-error {{overload resolution selected deleted operator '='}}
@@ -32,6 +29,4 @@ int main(int, char**)
         std::promise<void> p0, p;
         p = p0; // expected-error {{overload resolution selected deleted operator '='}}
     }
-
-    return 0;
 }
diff --git a/libcxx/test/std/thread/futures/futures.promise/copy_ctor.verify.cpp b/libcxx/test/std/thread/futures/futures.promise/copy_ctor.verify.cpp
index b4f3d7f452f62..437642ba7c580 100644
--- a/libcxx/test/std/thread/futures/futures.promise/copy_ctor.verify.cpp
+++ b/libcxx/test/std/thread/futures/futures.promise/copy_ctor.verify.cpp
@@ -16,10 +16,7 @@
 
 #include <future>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     {
         std::promise<int> p0;
         std::promise<int> p(p0); // expected-error {{call to deleted constructor of 'std::promise<int>'}}
@@ -32,6 +29,4 @@ int main(int, char**)
         std::promise<void> p0;
         std::promise<void> p(p0); // expected-error {{call to deleted constructor of 'std::promise<void>'}}
     }
-
-    return 0;
 }
diff --git a/libcxx/test/std/thread/futures/futures.unique_future/copy_assign.verify.cpp b/libcxx/test/std/thread/futures/futures.unique_future/copy_assign.verify.cpp
index 284ccd86390a3..cb7e8552175b9 100644
--- a/libcxx/test/std/thread/futures/futures.unique_future/copy_assign.verify.cpp
+++ b/libcxx/test/std/thread/futures/futures.unique_future/copy_assign.verify.cpp
@@ -16,10 +16,7 @@
 
 #include <future>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     {
         std::future<int> f0, f;
         f = f0; // expected-error {{overload resolution selected deleted operator '='}}
@@ -32,6 +29,4 @@ int main(int, char**)
         std::future<void> f0, f;
         f = f0; // expected-error {{overload resolution selected deleted operator '='}}
     }
-
-    return 0;
 }
diff --git a/libcxx/test/std/thread/futures/futures.unique_future/copy_ctor.verify.cpp b/libcxx/test/std/thread/futures/futures.unique_future/copy_ctor.verify.cpp
index 442489025b61f..4ab51efbdfdc1 100644
--- a/libcxx/test/std/thread/futures/futures.unique_future/copy_ctor.verify.cpp
+++ b/libcxx/test/std/thread/futures/futures.unique_future/copy_ctor.verify.cpp
@@ -16,10 +16,7 @@
 
 #include <future>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     {
         std::future<int> f0;
         std::future<int> f = f0; // expected-error {{call to deleted constructor of 'std::future<int>'}}
@@ -32,6 +29,4 @@ int main(int, char**)
         std::future<void> f0;
         std::future<void> f = f0; // expected-error {{call to deleted constructor of 'std::future<void>'}}
     }
-
-    return 0;
 }
diff --git a/libcxx/test/std/thread/thread.condition/thread.condition.condvarany/wait_terminates.sh.cpp b/libcxx/test/std/thread/thread.condition/thread.condition.condvarany/wait_terminates.sh.cpp
index fd61634bd96e5..4e2086220e963 100644
--- a/libcxx/test/std/thread/thread.condition/thread.condition.condvarany/wait_terminates.sh.cpp
+++ b/libcxx/test/std/thread/thread.condition/thread.condition.condvarany/wait_terminates.sh.cpp
@@ -55,13 +55,14 @@
 //     that terminate has been called)
 
 
-#include <condition_variable>
 #include <atomic>
-#include <thread>
+#include <cassert>
 #include <chrono>
-#include <string>
+#include <condition_variable>
 #include <cstdlib>
-#include <cassert>
+#include <exception>
+#include <string>
+#include <thread>
 
 #include "make_test_thread.h"
 
diff --git a/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.guard/adopt_lock.pass.cpp b/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.guard/adopt_lock.pass.cpp
index cb105f20db597..4d11674f1e83c 100644
--- a/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.guard/adopt_lock.pass.cpp
+++ b/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.guard/adopt_lock.pass.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 //
 // UNSUPPORTED: no-threads
+// UNSUPPORTED: c++03
 
 // <mutex>
 
diff --git a/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.cons/mutex_adopt_lock.pass.cpp b/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.cons/mutex_adopt_lock.pass.cpp
index b2e80659abb35..4adbe26777d0b 100644
--- a/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.cons/mutex_adopt_lock.pass.cpp
+++ b/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.cons/mutex_adopt_lock.pass.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 //
 // UNSUPPORTED: no-threads
+// UNSUPPORTED: c++03
 
 // <mutex>
 
diff --git a/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.cons/mutex_defer_lock.pass.cpp b/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.cons/mutex_defer_lock.pass.cpp
index fa00da655b0df..06ef2043edcef 100644
--- a/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.cons/mutex_defer_lock.pass.cpp
+++ b/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.cons/mutex_defer_lock.pass.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 //
 // UNSUPPORTED: no-threads
+// UNSUPPORTED: c++03
 
 // <mutex>
 
diff --git a/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.cons/mutex_try_to_lock.pass.cpp b/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.cons/mutex_try_to_lock.pass.cpp
index 39d3bde9c12a8..f4f344ef9b2c3 100644
--- a/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.cons/mutex_try_to_lock.pass.cpp
+++ b/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.cons/mutex_try_to_lock.pass.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 // UNSUPPORTED: no-threads
+// UNSUPPORTED: c++03
 // ALLOW_RETRIES: 2
 
 // <mutex>
diff --git a/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.locking/lock.pass.cpp b/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.locking/lock.pass.cpp
index b1b5af08d4df9..fa43f5dd874a5 100644
--- a/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.locking/lock.pass.cpp
+++ b/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.locking/lock.pass.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 //
 // UNSUPPORTED: no-threads
+// UNSUPPORTED: c++03
 // ALLOW_RETRIES: 2
 
 // <mutex>
diff --git a/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.locking/try_lock.pass.cpp b/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.locking/try_lock.pass.cpp
index 83e1d7eaa62fd..9249959ad005a 100644
--- a/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.locking/try_lock.pass.cpp
+++ b/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.locking/try_lock.pass.cpp
@@ -7,7 +7,7 @@
 //===----------------------------------------------------------------------===//
 //
 // UNSUPPORTED: no-threads
-//
+// UNSUPPORTED: c++03
 // ALLOW_RETRIES: 2
 
 // <mutex>
diff --git a/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.locking/try_lock_for.pass.cpp b/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.locking/try_lock_for.pass.cpp
index 1a0fba6146f5c..fe29d1625069e 100644
--- a/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.locking/try_lock_for.pass.cpp
+++ b/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.locking/try_lock_for.pass.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 //
 // UNSUPPORTED: no-threads
+// UNSUPPORTED: c++03
 
 // <mutex>
 
diff --git a/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.locking/try_lock_until.pass.cpp b/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.locking/try_lock_until.pass.cpp
index 3f6f2137fe24a..a1e8553b965fe 100644
--- a/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.locking/try_lock_until.pass.cpp
+++ b/libcxx/test/std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.locking/try_lock_until.pass.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 //
 // UNSUPPORTED: no-threads
+// UNSUPPORTED: c++03
 
 // <mutex>
 
diff --git a/libcxx/test/std/thread/thread.mutex/thread.lock/types.pass.cpp b/libcxx/test/std/thread/thread.mutex/thread.lock/types.pass.cpp
index 2776e00aa31ad..bb4e9b4bf8af3 100644
--- a/libcxx/test/std/thread/thread.mutex/thread.lock/types.pass.cpp
+++ b/libcxx/test/std/thread/thread.mutex/thread.lock/types.pass.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 //
 // UNSUPPORTED: no-threads
+// UNSUPPORTED: c++03
 
 // <mutex>
 
diff --git a/libcxx/test/std/thread/thread.threads/thread.thread.class/thread.thread.destr/dtor.pass.cpp b/libcxx/test/std/thread/thread.threads/thread.thread.class/thread.thread.destr/dtor.pass.cpp
index 77665050d1848..1d959e96dbcad 100644
--- a/libcxx/test/std/thread/thread.threads/thread.thread.class/thread.thread.destr/dtor.pass.cpp
+++ b/libcxx/test/std/thread/thread.threads/thread.thread.class/thread.thread.destr/dtor.pass.cpp
@@ -15,10 +15,11 @@
 
 // ~thread();
 
-#include <thread>
-#include <new>
-#include <cstdlib>
 #include <cassert>
+#include <cstdlib>
+#include <exception>
+#include <new>
+#include <thread>
 
 #include "make_test_thread.h"
 #include "test_macros.h"
diff --git a/libcxx/test/std/time/time.hms/time.hms.members/seconds.pass.cpp b/libcxx/test/std/time/time.hms/time.hms.members/seconds.pass.cpp
index 7aa9f3c4a1035..f0723be01a74c 100644
--- a/libcxx/test/std/time/time.hms/time.hms.members/seconds.pass.cpp
+++ b/libcxx/test/std/time/time.hms/time.hms.members/seconds.pass.cpp
@@ -24,7 +24,7 @@
 #include "test_macros.h"
 
 template <typename Duration>
-constexpr int64_t check_seconds(Duration d)
+constexpr std::int64_t check_seconds(Duration d)
 {
     using HMS = std::chrono::hh_mm_ss<Duration>;
     ASSERT_SAME_TYPE(std::chrono::seconds, decltype(std::declval<HMS>().seconds()));
diff --git a/libcxx/test/std/time/time.hms/time.hms.members/subseconds.pass.cpp b/libcxx/test/std/time/time.hms/time.hms.members/subseconds.pass.cpp
index b3b46a974d4b3..6504d19c48358 100644
--- a/libcxx/test/std/time/time.hms/time.hms.members/subseconds.pass.cpp
+++ b/libcxx/test/std/time/time.hms/time.hms.members/subseconds.pass.cpp
@@ -24,7 +24,7 @@
 #include "test_macros.h"
 
 template <typename Duration>
-constexpr int64_t check_subseconds(Duration d)
+constexpr std::int64_t check_subseconds(Duration d)
 {
     using HMS = std::chrono::hh_mm_ss<Duration>;
     ASSERT_SAME_TYPE(typename HMS::precision, decltype(std::declval<HMS>().subseconds()));
diff --git a/libcxx/test/std/time/time.syn/formatter.hh_mm_ss.pass.cpp b/libcxx/test/std/time/time.syn/formatter.hh_mm_ss.pass.cpp
index 8814fa316337a..1bbe72a8e285e 100644
--- a/libcxx/test/std/time/time.syn/formatter.hh_mm_ss.pass.cpp
+++ b/libcxx/test/std/time/time.syn/formatter.hh_mm_ss.pass.cpp
@@ -149,7 +149,7 @@ static void test_valid_values() {
            "%EX='03:02:01'\t"
            "\n"),
         fmt,
-        std::chrono::hh_mm_ss(-(3h + 2min + 1s + std::chrono::duration<int64_t, std::pico>(123456789012))));
+        std::chrono::hh_mm_ss(-(3h + 2min + 1s + std::chrono::duration<std::int64_t, std::pico>(123456789012))));
 
   // The number of fractional seconds is 0 according to the Standard
   // TODO FMT Determine what to do.
@@ -260,7 +260,7 @@ static void test_valid_values() {
            "%EX='03:02:01'\t"
            "\n"),
         lfmt,
-        std::chrono::hh_mm_ss(-(3h + 2min + 1s + std::chrono::duration<int64_t, std::pico>(123456789012))));
+        std::chrono::hh_mm_ss(-(3h + 2min + 1s + std::chrono::duration<std::int64_t, std::pico>(123456789012))));
 
   check(SV("%H='01'\t"
            "%OH='01'\t"
@@ -379,7 +379,7 @@ static void test_valid_values() {
 #  endif
            "\n"),
         lfmt,
-        std::chrono::hh_mm_ss(-(3h + 2min + 1s + std::chrono::duration<int64_t, std::pico>(123456789012))));
+        std::chrono::hh_mm_ss(-(3h + 2min + 1s + std::chrono::duration<std::int64_t, std::pico>(123456789012))));
 
   check(loc,
         SV("%H='01'\t"
@@ -466,7 +466,7 @@ static void test_valid_values() {
            "%EX='03時02分01秒'\t"
            "\n"),
         lfmt,
-        std::chrono::hh_mm_ss(-(3h + 2min + 1s + std::chrono::duration<int64_t, std::pico>(123456789012))));
+        std::chrono::hh_mm_ss(-(3h + 2min + 1s + std::chrono::duration<std::int64_t, std::pico>(123456789012))));
 
   check(loc,
         SV("%H='01'\t"
diff --git a/libcxx/test/std/time/time.syn/formatter_tests.h b/libcxx/test/std/time/time.syn/formatter_tests.h
index 115d6cc7b93d8..42c176b3b47e5 100644
--- a/libcxx/test/std/time/time.syn/formatter_tests.h
+++ b/libcxx/test/std/time/time.syn/formatter_tests.h
@@ -15,6 +15,7 @@
 
 #include <algorithm>
 #include <cassert>
+#include <iostream>
 #include <set>
 #include <string>
 #include <string_view>
diff --git a/libcxx/test/std/utilities/allocator.adaptor/allocator.adaptor.members/allocate_size.verify.cpp b/libcxx/test/std/utilities/allocator.adaptor/allocator.adaptor.members/allocate_size.verify.cpp
index fd4d03e2c1c17..6bd5f8041f853 100644
--- a/libcxx/test/std/utilities/allocator.adaptor/allocator.adaptor.members/allocate_size.verify.cpp
+++ b/libcxx/test/std/utilities/allocator.adaptor/allocator.adaptor.members/allocate_size.verify.cpp
@@ -16,14 +16,10 @@
 // pointer allocate(size_type n);
 
 #include <scoped_allocator>
-#include <cassert>
 
 #include "allocators.h"
 
-int main(int, char**)
-{
+void f() {
     std::scoped_allocator_adaptor<A1<int>> a;
     a.allocate(10);  // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/utilities/allocator.adaptor/allocator.adaptor.members/allocate_size_hint.verify.cpp b/libcxx/test/std/utilities/allocator.adaptor/allocator.adaptor.members/allocate_size_hint.verify.cpp
index 6e60b6a3767a4..05588dffd4a8d 100644
--- a/libcxx/test/std/utilities/allocator.adaptor/allocator.adaptor.members/allocate_size_hint.verify.cpp
+++ b/libcxx/test/std/utilities/allocator.adaptor/allocator.adaptor.members/allocate_size_hint.verify.cpp
@@ -16,14 +16,10 @@
 // pointer allocate(size_type n, const_void_pointer hint);
 
 #include <scoped_allocator>
-#include <cassert>
 
 #include "allocators.h"
 
-int main(int, char**)
-{
+void f() {
     std::scoped_allocator_adaptor<A1<int>> a;
     a.allocate(10, (const void*)0); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/utilities/any/any.nonmembers/any.cast/const_correctness.verify.cpp b/libcxx/test/std/utilities/any/any.nonmembers/any.cast/const_correctness.verify.cpp
index a066c6af4998f..cc825762affb3 100644
--- a/libcxx/test/std/utilities/any/any.nonmembers/any.cast/const_correctness.verify.cpp
+++ b/libcxx/test/std/utilities/any/any.nonmembers/any.cast/const_correctness.verify.cpp
@@ -30,8 +30,7 @@
 struct TestType {};
 struct TestType2 {};
 
-int main(int, char**)
-{
+void f() {
     std::any a;
 
     // expected-error-re@any:* {{{{(static_assert|static assertion)}} failed{{.*}}ValueType is required to be a const lvalue reference or a CopyConstructible type}}
@@ -45,6 +44,4 @@ int main(int, char**)
 
     // expected-error-re@any:* {{{{(static_assert|static assertion)}} failed{{.*}}ValueType is required to be a const lvalue reference or a CopyConstructible type}}
     std::any_cast<TestType2 &&>(static_cast<std::any const&&>(a)); // expected-note {{requested here}}
-
-  return 0;
 }
diff --git a/libcxx/test/std/utilities/any/any.nonmembers/any.cast/not_copy_constructible.verify.cpp b/libcxx/test/std/utilities/any/any.nonmembers/any.cast/not_copy_constructible.verify.cpp
index a3b3b3b6734e4..394ebca1d769a 100644
--- a/libcxx/test/std/utilities/any/any.nonmembers/any.cast/not_copy_constructible.verify.cpp
+++ b/libcxx/test/std/utilities/any/any.nonmembers/any.cast/not_copy_constructible.verify.cpp
@@ -33,20 +33,19 @@
 
 #include <any>
 
-struct no_copy
-{
+struct no_copy {
     no_copy() {}
     no_copy(no_copy &&) {}
     no_copy(no_copy const &) = delete;
 };
 
 struct no_move {
-  no_move() {}
-  no_move(no_move&&) = delete;
-  no_move(no_move const&) {}
+    no_move() {}
+    no_move(no_move&&) = delete;
+    no_move(no_move const&) {}
 };
 
-int main(int, char**) {
+void f() {
     std::any a;
     // expected-error-re@any:* {{{{(static_assert|static assertion)}} failed{{.*}}ValueType is required to be an lvalue reference or a CopyConstructible type}}
     std::any_cast<no_copy>(static_cast<std::any&>(a)); // expected-note {{requested here}}
@@ -58,6 +57,4 @@ int main(int, char**) {
 
     // expected-error-re@any:* {{{{(static_assert|static assertion)}} failed{{.*}}ValueType is required to be an rvalue reference or a CopyConstructible type}}
     std::any_cast<no_move>(static_cast<std::any &&>(a));
-
-  return 0;
 }
diff --git a/libcxx/test/std/utilities/charconv/charconv.msvc/floating_point_test_cases.hpp b/libcxx/test/std/utilities/charconv/charconv.msvc/floating_point_test_cases.hpp
index 55c61db00d6ec..5ea71438c01cc 100644
--- a/libcxx/test/std/utilities/charconv/charconv.msvc/floating_point_test_cases.hpp
+++ b/libcxx/test/std/utilities/charconv/charconv.msvc/floating_point_test_cases.hpp
@@ -7,7 +7,7 @@
 #include <stdint.h>
 #include <utility>
 
-constexpr std::pair<const char*, uint64_t> floating_point_test_cases_double[] = {
+constexpr std::pair<const char*, std::uint64_t> floating_point_test_cases_double[] = {
     // Verify small exactly-representable integers:
     {"1", 0x3FF0000000000000ULL},
     {"2", 0x4000000000000000ULL},
@@ -202,7 +202,7 @@ constexpr std::pair<const char*, uint64_t> floating_point_test_cases_double[] =
     {"8.32116e+55", 0x4B8B2628393E02CDULL},
 };
 
-constexpr std::pair<const char*, uint32_t> floating_point_test_cases_float[] = {
+constexpr std::pair<const char*, std::uint32_t> floating_point_test_cases_float[] = {
     // Verify small exactly-representable integers:
     {"1", 0x3F800000U},
     {"2", 0x40000000U},
diff --git a/libcxx/test/std/utilities/charconv/charconv.msvc/test.cpp b/libcxx/test/std/utilities/charconv/charconv.msvc/test.cpp
index 69e11a6e60475..38ff3d8f29a62 100644
--- a/libcxx/test/std/utilities/charconv/charconv.msvc/test.cpp
+++ b/libcxx/test/std/utilities/charconv/charconv.msvc/test.cpp
@@ -54,7 +54,7 @@ void initialize_randomness(mt19937_64& mt64, const int argc, char** const /*argv
     static_assert(w % 32 == 0);
     constexpr size_t k = w / 32;
 
-    vector<uint32_t> vec(n * k);
+    vector<std::uint32_t> vec(n * k);
 
     puts("USAGE:");
     puts("test.exe              : generates seed data from random_device.");
@@ -187,12 +187,12 @@ constexpr const char* output_max_digit[] = {"skip0", "skip1", "11", "12", "13",
     "1t", "1u", "1v", "1w", "1x", "1y", "1z"};
 
 // https://www.wolframalpha.com : Table[BaseForm[k, n], {k, {MEOW, MEOW, MEOW}}, {n, 2, 36}]
-constexpr uint64_t stress_chunks_positive                          = 12000000345000678900ULL;
-constexpr pair<uint64_t, array<const char*, 37>> output_positive[] = {
+constexpr std::uint64_t stress_chunks_positive                          = 12000000345000678900ULL;
+constexpr pair<std::uint64_t, array<const char*, 37>> output_positive[] = {
     {123U, {{"skip0", "skip1", "1111011", "11120", "1323", "443", "323", "234", "173", "146", "123", "102", "a3", "96",
                "8b", "83", "7b", "74", "6f", "69", "63", "5i", "5d", "58", "53", "4n", "4j", "4f", "4b", "47", "43",
                "3u", "3r", "3o", "3l", "3i", "3f"}}},
-    {uint64_t{INT8_MAX}, {{"skip0", "skip1", "1111111", "11201", "1333", "1002", "331", "241", "177", "151", "127",
+    {std::uint64_t{INT8_MAX}, {{"skip0", "skip1", "1111111", "11201", "1333", "1002", "331", "241", "177", "151", "127",
                              "106", "a7", "9a", "91", "87", "7f", "78", "71", "6d", "67", "61", "5h", "5c", "57", "52",
                              "4n", "4j", "4f", "4b", "47", "43", "3v", "3s", "3p", "3m", "3j"}}},
     {161U, {{"skip0", "skip1", "10100001", "12222", "2201", "1121", "425", "320", "241", "188", "161", "137", "115",
@@ -204,7 +204,7 @@ constexpr pair<uint64_t, array<const char*, 37>> output_positive[] = {
     {1729U, {{"skip0", "skip1", "11011000001", "2101001", "123001", "23404", "12001", "5020", "3301", "2331", "1729",
                 "1332", "1001", "a30", "8b7", "7a4", "6c1", "5gc", "561", "4f0", "469", "3j7", "3cd", "364", "301",
                 "2j4", "2ed", "2a1", "25l", "21i", "1rj", "1oo", "1m1", "1jd", "1gt", "1ee", "1c1"}}},
-    {uint64_t{INT16_MAX}, {{"skip0", "skip1", "111111111111111", "1122221121", "13333333", "2022032", "411411",
+    {std::uint64_t{INT16_MAX}, {{"skip0", "skip1", "111111111111111", "1122221121", "13333333", "2022032", "411411",
                               "164350", "77777", "48847", "32767", "22689", "16b67", "11bb7", "bd27", "9a97", "7fff",
                               "6b68", "5b27", "4eeb", "41i7", "3b67", "31f9", "2flf", "28l7", "22ah", "1mc7", "1hpg",
                               "1dm7", "19rq", "16c7", "1330", "vvv", "u2v", "sbp", "qq7", "pa7"}}},
@@ -221,7 +221,7 @@ constexpr pair<uint64_t, array<const char*, 37>> output_positive[] = {
                     "1b9a06b2", "11973ba8", "9636514", "639e338", "43d49e6", "2g19gfb", "21b9d18", "19dec94", "124addi",
                     "h8f25b", "dhdfa6", "b13hg2", "8m91he", "7720j3", "5pgj58", "4pmelq", "43k17i", "3dg8ek", "2ro898",
                     "2f0et8", "23qif6", "1qw5lh", "1j7l7s", "1cdvli", "16cgrq"}}},
-    {uint64_t{INT32_MAX},
+    {std::uint64_t{INT32_MAX},
         {{"skip0", "skip1", "1111111111111111111111111111111", "12112122212110202101", "1333333333333333",
             "13344223434042", "553032005531", "104134211161", "17777777777", "5478773671", "2147483647", "a02220281",
             "4bb2308a7", "282ba4aaa", "1652ca931", "c87e66b7", "7fffffff", "53g7f548", "3928g3h1", "27c57h32",
@@ -247,7 +247,7 @@ constexpr pair<uint64_t, array<const char*, 37>> output_positive[] = {
             "4d0d5e232c53", "2d63h403i580", "1bf5h8185hdj", "kc3g550fkcg", "d41id5k9984", "8ef5n0him4g", "5i2dijfe1la",
             "3me22fm5fhi", "2hfmhgg73kd", "1ngpfabr53c", "18i7220bh11", "rm0lcjngpa", "kk1elesni1", "fgfge3c3fg",
             "bp4q5l6bjg", "8xna46jp0k", "6wejomvji5", "5di2s1qhv4"}}},
-    {uint64_t{INT64_MAX},
+    {std::uint64_t{INT64_MAX},
         {{"skip0", "skip1", "111111111111111111111111111111111111111111111111111111111111111",
             "2021110011022210012102010021220101220221", "13333333333333333333333333333333",
             "1104332401304422434310311212", "1540241003031030222122211", "22341010611245052052300",
@@ -290,8 +290,8 @@ constexpr pair<uint64_t, array<const char*, 37>> output_positive[] = {
 };
 
 // https://www.wolframalpha.com : Table[BaseForm[k, n], {k, {MEOW, MEOW, MEOW}}, {n, 2, 36}]
-constexpr int64_t stress_chunks_negative                          = -9000876000000054321LL;
-constexpr pair<int64_t, array<const char*, 37>> output_negative[] = {
+constexpr std::int64_t stress_chunks_negative                          = -9000876000000054321LL;
+constexpr pair<std::int64_t, array<const char*, 37>> output_negative[] = {
     {-85, {{"skip0", "skip1", "-1010101", "-10011", "-1111", "-320", "-221", "-151", "-125", "-104", "-85", "-78",
               "-71", "-67", "-61", "-5a", "-55", "-50", "-4d", "-49", "-45", "-41", "-3j", "-3g", "-3d", "-3a", "-37",
               "-34", "-31", "-2r", "-2p", "-2n", "-2l", "-2j", "-2h", "-2f", "-2d"}}},
@@ -360,7 +360,7 @@ void test_integer_to_chars() {
         test_integer_to_chars(static_cast<T>(base * 2 - 1), base, output_max_digit[base]);
 
         for (const auto& p : output_positive) {
-            if (p.first <= static_cast<uint64_t>(numeric_limits<T>::max())) {
+            if (p.first <= static_cast<std::uint64_t>(numeric_limits<T>::max())) {
                 test_integer_to_chars(static_cast<T>(p.first), base, p.second[static_cast<size_t>(base)]);
             }
         }
@@ -369,7 +369,7 @@ void test_integer_to_chars() {
             test_integer_to_chars(static_cast<T>(-1), base, "-1");
 
             for (const auto& p : output_negative) {
-                if (p.first >= static_cast<int64_t>(numeric_limits<T>::min())) {
+                if (p.first >= static_cast<std::int64_t>(numeric_limits<T>::min())) {
                     test_integer_to_chars(static_cast<T>(p.first), base, p.second[static_cast<size_t>(base)]);
                 }
             }
@@ -402,7 +402,7 @@ void test_from_chars(const string_view input, const BaseOrFmt base_or_fmt, const
     if (correct_ec == errc{} || (is_floating_point_v<T> && correct_ec == errc::result_out_of_range)) {
         if constexpr (is_floating_point_v<T>) {
             if (mode == TestFromCharsMode::Normal) {
-                using Uint = conditional_t<is_same_v<T, float>, uint32_t, uint64_t>;
+                using Uint = conditional_t<is_same_v<T, float>, std::uint32_t, std::uint64_t>;
                 assert(opt_correct.has_value());
                 assert(_Bit_cast<Uint>(dest) == _Bit_cast<Uint>(opt_correct.value()));
             } else {
@@ -543,7 +543,7 @@ void all_integer_tests() {
     test_from_chars<int>("-2147483650", 10, 11, out_ran); // beyond risky
 }
 
-void assert_message_bits(const bool b, const char* const msg, const uint32_t bits) {
+void assert_message_bits(const bool b, const char* const msg, const std::uint32_t bits) {
     if (!b) {
         fprintf(stderr, "%s failed for 0x%08zX\n", msg, static_cast<size_t>(bits));
         fprintf(stderr, "This is a randomized test.\n");
@@ -553,7 +553,7 @@ void assert_message_bits(const bool b, const char* const msg, const uint32_t bit
     }
 }
 
-void assert_message_bits(const bool b, const char* const msg, const uint64_t bits) {
+void assert_message_bits(const bool b, const char* const msg, const std::uint64_t bits) {
     if (!b) {
         // libc++ uses long for 64-bit values.
         fprintf(stderr, "%s failed for 0x%016llX\n", msg, static_cast<unsigned long long>(bits));
@@ -564,26 +564,26 @@ void assert_message_bits(const bool b, const char* const msg, const uint64_t bit
     }
 }
 
-constexpr uint32_t FractionBits = 10; // Tunable for test coverage vs. performance.
+constexpr std::uint32_t FractionBits = 10; // Tunable for test coverage vs. performance.
 static_assert(FractionBits >= 1, "Must test at least 1 fraction bit.");
 static_assert(FractionBits <= 23, "There are only 23 fraction bits in a float.");
 
-constexpr uint32_t Fractions = 1U << FractionBits;
-constexpr uint32_t Mask32    = ~((1U << FractionBits) - 1U);
-constexpr uint64_t Mask64    = ~((1ULL << FractionBits) - 1ULL);
+constexpr std::uint32_t Fractions = 1U << FractionBits;
+constexpr std::uint32_t Mask32    = ~((1U << FractionBits) - 1U);
+constexpr std::uint64_t Mask64    = ~((1ULL << FractionBits) - 1ULL);
 
-constexpr uint32_t PrefixesToTest = 100; // Tunable for test coverage vs. performance.
+constexpr std::uint32_t PrefixesToTest = 100; // Tunable for test coverage vs. performance.
 static_assert(PrefixesToTest >= 1, "Must test at least 1 prefix.");
 
-constexpr uint32_t PrefixLimit = 2 // sign bit
+constexpr std::uint32_t PrefixLimit = 2 // sign bit
                                * 255 // non-INF/NAN exponents for float
                                * (1U << (23 - FractionBits)); // fraction bits in prefix
 static_assert(PrefixesToTest <= PrefixLimit, "Too many prefixes.");
 
 template <bool IsDouble>
-void test_floating_prefix(const conditional_t<IsDouble, uint64_t, uint32_t> prefix) {
+void test_floating_prefix(const conditional_t<IsDouble, std::uint64_t, std::uint32_t> prefix) {
 
-    using UIntType     = conditional_t<IsDouble, uint64_t, uint32_t>;
+    using UIntType     = conditional_t<IsDouble, std::uint64_t, std::uint32_t>;
     using FloatingType = conditional_t<IsDouble, double, float>;
 
     // "-1.2345678901234567e-100" or "-1.23456789e-10"
@@ -607,7 +607,7 @@ void test_floating_prefix(const conditional_t<IsDouble, uint64_t, uint32_t> pref
     constexpr size_t stdio_buffer_size = 1 + (IsDouble ? 309 : 39) + 1;
     char stdio_buffer[stdio_buffer_size];
 
-    for (uint32_t frac = 0; frac < Fractions; ++frac) {
+    for (std::uint32_t frac = 0; frac < Fractions; ++frac) {
         const UIntType bits      = prefix + frac;
         const FloatingType input = _Bit_cast<FloatingType>(bits);
 
@@ -643,9 +643,9 @@ void test_floating_prefix(const conditional_t<IsDouble, uint64_t, uint32_t> pref
 }
 
 template <bool IsDouble>
-void test_floating_hex_prefix(const conditional_t<IsDouble, uint64_t, uint32_t> prefix) {
+void test_floating_hex_prefix(const conditional_t<IsDouble, std::uint64_t, std::uint32_t> prefix) {
 
-    using UIntType     = conditional_t<IsDouble, uint64_t, uint32_t>;
+    using UIntType     = conditional_t<IsDouble, std::uint64_t, std::uint32_t>;
     using FloatingType = conditional_t<IsDouble, double, float>;
 
     // The precision is the number of hexits after the decimal point.
@@ -661,7 +661,7 @@ void test_floating_hex_prefix(const conditional_t<IsDouble, uint64_t, uint32_t>
     FloatingType val;
 #endif
 
-    for (uint32_t frac = 0; frac < Fractions; ++frac) {
+    for (std::uint32_t frac = 0; frac < Fractions; ++frac) {
         const UIntType bits      = prefix + frac;
         const FloatingType input = _Bit_cast<FloatingType>(bits);
 
@@ -681,9 +681,9 @@ void test_floating_hex_prefix(const conditional_t<IsDouble, uint64_t, uint32_t>
 }
 
 template <bool IsDouble>
-void test_floating_precision_prefix(const conditional_t<IsDouble, uint64_t, uint32_t> prefix) {
+void test_floating_precision_prefix(const conditional_t<IsDouble, std::uint64_t, std::uint32_t> prefix) {
 
-    using UIntType     = conditional_t<IsDouble, uint64_t, uint32_t>;
+    using UIntType     = conditional_t<IsDouble, std::uint64_t, std::uint32_t>;
     using FloatingType = conditional_t<IsDouble, double, float>;
 
     // Precision for min subnormal in fixed notation. (More than enough for scientific notation.)
@@ -708,7 +708,7 @@ void test_floating_precision_prefix(const conditional_t<IsDouble, uint64_t, uint
     char general_buffer[general_buffer_size];
     char general_stdio_buffer[general_buffer_size + 1]; // + null terminator
 
-    for (uint32_t frac = 0; frac < Fractions; ++frac) {
+    for (std::uint32_t frac = 0; frac < Fractions; ++frac) {
         const UIntType bits      = prefix + frac;
         const FloatingType input = _Bit_cast<FloatingType>(bits);
 
@@ -748,10 +748,10 @@ void test_floating_precision_prefix(const conditional_t<IsDouble, uint64_t, uint
 
 void test_floating_prefixes(mt19937_64& mt64) {
     {
-        set<uint64_t> prefixes64;
+        set<std::uint64_t> prefixes64;
 
         while (prefixes64.size() < PrefixesToTest) {
-            const uint64_t val = mt64();
+            const std::uint64_t val = mt64();
 
             if ((val & 0x7FF0000000000000ULL) != 0x7FF0000000000000ULL) { // skip INF/NAN
                 prefixes64.insert(val & Mask64);
@@ -767,10 +767,10 @@ void test_floating_prefixes(mt19937_64& mt64) {
     }
 
     {
-        set<uint32_t> prefixes32;
+        set<std::uint32_t> prefixes32;
 
         while (prefixes32.size() < PrefixesToTest) {
-            const uint32_t val = static_cast<uint32_t>(mt64());
+            const std::uint32_t val = static_cast<std::uint32_t>(mt64());
 
             if ((val & 0x7F800000U) != 0x7F800000U) { // skip INF/NAN
                 prefixes32.insert(val & Mask32);
diff --git a/libcxx/test/std/utilities/charconv/charconv.syn/chars_format.pass.cpp b/libcxx/test/std/utilities/charconv/charconv.syn/chars_format.pass.cpp
index beee526422765..85b78c4db51b1 100644
--- a/libcxx/test/std/utilities/charconv/charconv.syn/chars_format.pass.cpp
+++ b/libcxx/test/std/utilities/charconv/charconv.syn/chars_format.pass.cpp
@@ -18,8 +18,9 @@
 //   general = fixed | scientific
 // };
 
-#include <charconv>
 #include <cassert>
+#include <charconv>
+#include <type_traits>
 
 #include "test_macros.h"
 
diff --git a/libcxx/test/std/utilities/charconv/charconv.to.chars/integral.pass.cpp b/libcxx/test/std/utilities/charconv/charconv.to.chars/integral.pass.cpp
index 4bdb8ca084716..3fc5670593058 100644
--- a/libcxx/test/std/utilities/charconv/charconv.to.chars/integral.pass.cpp
+++ b/libcxx/test/std/utilities/charconv/charconv.to.chars/integral.pass.cpp
@@ -23,26 +23,26 @@
 #include "charconv_test_helpers.h"
 
 #ifndef TEST_HAS_NO_INT128
-TEST_CONSTEXPR_CXX23 __uint128_t make_u128(__uint128_t a, uint64_t b) {
+TEST_CONSTEXPR_CXX23 __uint128_t make_u128(__uint128_t a, std::uint64_t b) {
   a *= 1000000000000000000UL;
   a *= 10;
   return a + b;
 }
 
-TEST_CONSTEXPR_CXX23 __uint128_t make_u128(__uint128_t a, uint64_t b, uint64_t c) {
+TEST_CONSTEXPR_CXX23 __uint128_t make_u128(__uint128_t a, std::uint64_t b, std::uint64_t c) {
   a *= 10000000000000ULL;
   a += b;
   a *= 10000000000000ULL;
   return a + c;
 }
 
-TEST_CONSTEXPR_CXX23 __int128_t make_i128(__int128_t a, int64_t b) {
+TEST_CONSTEXPR_CXX23 __int128_t make_i128(__int128_t a, std::int64_t b) {
   if (a < 0)
     return -make_u128(-a, b);
   return make_u128(a, b);
 }
 
-TEST_CONSTEXPR_CXX23 __int128_t make_i128(__int128_t a, __int128_t b, int64_t c) {
+TEST_CONSTEXPR_CXX23 __int128_t make_i128(__int128_t a, __int128_t b, std::int64_t c) {
   if (a < 0)
     return -make_u128(-a, b, c);
   return make_u128(a, b, c);
diff --git a/libcxx/test/std/utilities/format/format.arguments/format.args/types.compile.pass.cpp b/libcxx/test/std/utilities/format/format.arguments/format.args/types.compile.pass.cpp
index 9ca1d31e6abd1..4121d0b0e6b1f 100644
--- a/libcxx/test/std/utilities/format/format.arguments/format.args/types.compile.pass.cpp
+++ b/libcxx/test/std/utilities/format/format.arguments/format.args/types.compile.pass.cpp
@@ -26,6 +26,3 @@ static_assert(std::is_same_v<std::format_args,
 static_assert(std::is_same_v<std::wformat_args,
                              std::basic_format_args<std::wformat_context>>);
 #endif
-
-// Required for MSVC internal test runner compatibility.
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/std/utilities/format/format.formatter/format.formatter.spec/formatter.signed_integral.pass.cpp b/libcxx/test/std/utilities/format/format.formatter/format.formatter.spec/formatter.signed_integral.pass.cpp
index 9625c5bf55681..1bec16279948d 100644
--- a/libcxx/test/std/utilities/format/format.formatter/format.formatter.spec/formatter.signed_integral.pass.cpp
+++ b/libcxx/test/std/utilities/format/format.formatter/format.formatter.spec/formatter.signed_integral.pass.cpp
@@ -88,8 +88,8 @@ void test_signed_integral_type() {
     test_termination_condition(STR("2147483647"), STR("}"), A(2147483647));
   }
   if (sizeof(A) > 4) {
-    test_termination_condition(STR("-9223372036854775808"), STR("}"), A(std::numeric_limits<int64_t>::min()));
-    test_termination_condition(STR("9223372036854775807"), STR("}"), A(std::numeric_limits<int64_t>::max()));
+    test_termination_condition(STR("-9223372036854775808"), STR("}"), A(std::numeric_limits<std::int64_t>::min()));
+    test_termination_condition(STR("9223372036854775807"), STR("}"), A(std::numeric_limits<std::int64_t>::max()));
   }
 #ifndef TEST_HAS_NO_INT128
   if (sizeof(A) > 8) {
diff --git a/libcxx/test/std/utilities/format/format.formatter/format.parse.ctx/check_arg_id.verify.cpp b/libcxx/test/std/utilities/format/format.formatter/format.parse.ctx/check_arg_id.verify.cpp
index 74342c7d0fdb4..c3f68206c1b31 100644
--- a/libcxx/test/std/utilities/format/format.formatter/format.parse.ctx/check_arg_id.verify.cpp
+++ b/libcxx/test/std/utilities/format/format.formatter/format.parse.ctx/check_arg_id.verify.cpp
@@ -12,8 +12,6 @@
 
 #include <format>
 
-#include "test_macros.h"
-
 constexpr bool test() {
   // [format.parse.ctx]/11
   // Remarks: Call expressions where id >= num_args_ are not
@@ -24,9 +22,7 @@ constexpr bool test() {
   return true;
 }
 
-int main(int, char**) {
+void f() {
   // expected-error-re@+1 {{{{(static_assert|static assertion)}} expression is not an integral constant expression}}
   static_assert(test());
-
-  return 0;
 }
diff --git a/libcxx/test/std/utilities/format/format.formatter/format.parse.ctx/next_arg_id.pass.cpp b/libcxx/test/std/utilities/format/format.formatter/format.parse.ctx/next_arg_id.pass.cpp
index dbca19c1d5243..5ec3a943ba974 100644
--- a/libcxx/test/std/utilities/format/format.formatter/format.parse.ctx/next_arg_id.pass.cpp
+++ b/libcxx/test/std/utilities/format/format.formatter/format.parse.ctx/next_arg_id.pass.cpp
@@ -25,7 +25,7 @@
 #include "test_macros.h"
 
 constexpr bool test() {
-  std::format_parse_context context("");
+  std::format_parse_context context("", 10);
   for (size_t i = 0; i < 10; ++i)
     assert(i == context.next_arg_id());
 
diff --git a/libcxx/test/std/utilities/format/format.formatter/format.parse.ctx/next_arg_id.verify.cpp b/libcxx/test/std/utilities/format/format.formatter/format.parse.ctx/next_arg_id.verify.cpp
new file mode 100644
index 0000000000000..8448560838178
--- /dev/null
+++ b/libcxx/test/std/utilities/format/format.formatter/format.parse.ctx/next_arg_id.verify.cpp
@@ -0,0 +1,29 @@
+//===----------------------------------------------------------------------===//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// UNSUPPORTED: c++03, c++11, c++14, c++17
+// UNSUPPORTED: libcpp-has-no-incomplete-format
+
+// constexpr size_t next_arg_id()
+
+#include <format>
+
+constexpr bool test() {
+  // [format.parse.ctx]/8
+  // Let cur-arg-id be the value of next_arg_id_ prior to this call. Call
+  // expressions where cur-arg-id >= num_args_ is true are not core constant
+  // expressions (7.7 [expr.const]).
+  std::format_parse_context context("", 0);
+  context.next_arg_id();
+
+  return true;
+}
+
+void f() {
+  // expected-error-re@+1 {{{{(static_assert|static assertion)}} expression is not an integral constant expression}}
+  static_assert(test());
+}
diff --git a/libcxx/test/std/utilities/format/format.formatter/format.parse.ctx/types.compile.pass.cpp b/libcxx/test/std/utilities/format/format.formatter/format.parse.ctx/types.compile.pass.cpp
index eaa18a476aebf..f224daee70a5a 100644
--- a/libcxx/test/std/utilities/format/format.formatter/format.parse.ctx/types.compile.pass.cpp
+++ b/libcxx/test/std/utilities/format/format.formatter/format.parse.ctx/types.compile.pass.cpp
@@ -62,6 +62,3 @@ static_assert(std::is_same_v<std::format_parse_context,
 static_assert(std::is_same_v<std::wformat_parse_context,
                              std::basic_format_parse_context<wchar_t> >);
 #endif
-
-// Required for MSVC internal test runner compatibility.
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/std/utilities/format/format.functions/escaped_output.ascii.pass.cpp b/libcxx/test/std/utilities/format/format.functions/escaped_output.ascii.pass.cpp
index 5f87e3168da0a..d08133b98d250 100644
--- a/libcxx/test/std/utilities/format/format.functions/escaped_output.ascii.pass.cpp
+++ b/libcxx/test/std/utilities/format/format.functions/escaped_output.ascii.pass.cpp
@@ -224,7 +224,7 @@ void test_char() {
     // Unicode fitting in a 32-bit wchar_t
 
     constexpr wchar_t x  = 0x1ffff;
-    constexpr uint32_t y = 0x1ffff;
+    constexpr std::uint32_t y = 0x1ffff;
     static_assert(x == y);
 
     using V = std::basic_string_view<CharT>;
diff --git a/libcxx/test/std/utilities/format/format.functions/escaped_output.unicode.pass.cpp b/libcxx/test/std/utilities/format/format.functions/escaped_output.unicode.pass.cpp
index c94572ad6ff6b..9caa3a22d7503 100644
--- a/libcxx/test/std/utilities/format/format.functions/escaped_output.unicode.pass.cpp
+++ b/libcxx/test/std/utilities/format/format.functions/escaped_output.unicode.pass.cpp
@@ -202,14 +202,9 @@ void test_char() {
     test_format(V{L"'\\u{600}'"}, L"{:?}", L'\x600');   // ARABIC NUMBER SIGN
     test_format(V{L"'\\u{feff}'"}, L"{:?}", L'\xfeff'); // ZERO WIDTH NO-BREAK SPACE
 
-    if constexpr (sizeof(CharT) == 2) {
-      // Incomplete surrogate pair in UTF-16
-      test_format(V{L"'\\x{d800}'"}, L"{:?}", L'\xd800'); // <surrogate-D800>
-      test_format(V{L"'\\x{dfff}'"}, L"{:?}", L'\xdfff'); // <surrogate-DFFF>
-    } else {
-      test_format(V{L"'\\u{d800}'"}, L"{:?}", L'\xd800'); // <surrogate-D800>
-      test_format(V{L"'\\u{dfff}'"}, L"{:?}", L'\xdfff'); // <surrogate-DFFF>
-    }
+    // Incomplete surrogate pair in UTF-16
+    test_format(V{L"'\\x{d800}'"}, L"{:?}", L'\xd800'); // <surrogate-D800>
+    test_format(V{L"'\\x{dfff}'"}, L"{:?}", L'\xdfff'); // <surrogate-DFFF>
 
     // Private_Use
     test_format(V{L"'\\u{e000}'"}, L"{:?}", L'\xe000'); // <private-use-E000>
@@ -230,7 +225,7 @@ void test_char() {
     // Unicode fitting in a 32-bit wchar_t
 
     constexpr wchar_t x  = 0x1ffff;
-    constexpr uint32_t y = 0x1ffff;
+    constexpr std::uint32_t y = 0x1ffff;
     static_assert(x == y);
 
     using V = std::basic_string_view<CharT>;
@@ -277,6 +272,48 @@ void test_string() {
     // Ill-formend UTF-8
     test_format(SV(R"(["\x{c3}"])"), SV("[{:?}]"), "\xc3");
     test_format(SV(R"(["\x{c3}("])"), SV("[{:?}]"), "\xc3\x28");
+
+    /* U+0000..U+0007F 1 code unit range, encoded in 2 code units. */
+    test_format(SV(R"(["\x{c0}\x{80}"])"), SV("[{:?}]"), "\xc0\x80"); // U+0000
+    test_format(SV(R"(["\x{c1}\x{bf}"])"), SV("[{:?}]"), "\xc1\xbf"); // U+007F
+    test_format(SV(R"(["\u{80}"])"), SV("[{:?}]"), "\xc2\x80");       // U+0080 first valid (General_Category=Control)
+
+    /* U+0000..U+07FFF 1 and 2 code unit range, encoded in 3 code units. */
+    test_format(SV(R"(["\x{e0}\x{80}\x{80}"])"), SV("[{:?}]"), "\xe0\x80\x80"); // U+0000
+    test_format(SV(R"(["\x{e0}\x{81}\x{bf}"])"), SV("[{:?}]"), "\xe0\x81\xbf"); // U+007F
+    test_format(SV(R"(["\x{e0}\x{82}\x{80}"])"), SV("[{:?}]"), "\xe0\x82\x80"); // U+0080
+    test_format(SV(R"(["\x{e0}\x{9f}\x{bf}"])"), SV("[{:?}]"), "\xe0\x9f\xbf"); // U+07FF
+    test_format(SV("[\"\u0800\"]"), SV("[{:?}]"), "\xe0\xa0\x80");              // U+0800 first valid
+
+#if 0
+	// This code point is in the Hangul Jamo Extended-B block and at the time of writing
+	// it's unassigned. When it comes defined, this branch might become true.
+    test_format(SV("[\"\ud7ff\"]"), SV("[{:?}]"), "\xed\x9f\xbf");              // U+D7FF last valid
+#else
+    /* U+D800..D+DFFFF surrogate range */
+    test_format(SV(R"(["\u{d7ff}"])"), SV("[{:?}]"), "\xed\x9f\xbf");           // U+D7FF last valid
+#endif
+    test_format(SV(R"(["\x{ed}\x{a0}\x{80}"])"), SV("[{:?}]"), "\xed\xa0\x80"); // U+D800
+    test_format(SV(R"(["\x{ed}\x{af}\x{bf}"])"), SV("[{:?}]"), "\xed\xaf\xbf"); // U+DBFF
+    test_format(SV(R"(["\x{ed}\x{bf}\x{80}"])"), SV("[{:?}]"), "\xed\xbf\x80"); // U+DC00
+    test_format(SV(R"(["\x{ed}\x{bf}\x{bf}"])"), SV("[{:?}]"), "\xed\xbf\xbf"); // U+DFFF
+    test_format(SV(R"(["\u{e000}"])"), SV("[{:?}]"), "\xee\x80\x80");           // U+E000 first valid
+                                                                                // (in the Private Use Area block)
+
+    /* U+0000..U+FFFF 1, 2, and 3 code unit range */
+    test_format(SV(R"(["\x{f0}\x{80}\x{80}\x{80}"])"), SV("[{:?}]"), "\xf0\x80\x80\x80"); // U+0000
+    test_format(SV(R"(["\x{f0}\x{80}\x{81}\x{bf}"])"), SV("[{:?}]"), "\xf0\x80\x81\xbf"); // U+007F
+    test_format(SV(R"(["\x{f0}\x{80}\x{82}\x{80}"])"), SV("[{:?}]"), "\xf0\x80\x82\x80"); // U+0080
+    test_format(SV(R"(["\x{f0}\x{80}\x{9f}\x{bf}"])"), SV("[{:?}]"), "\xf0\x80\x9f\xbf"); // U+07FF
+    test_format(SV(R"(["\x{f0}\x{80}\x{a0}\x{80}"])"), SV("[{:?}]"), "\xf0\x80\xa0\x80"); // U+0800
+    test_format(SV(R"(["\x{f0}\x{8f}\x{bf}\x{bf}"])"), SV("[{:?}]"), "\xf0\x8f\xbf\xbf"); // U+FFFF
+    test_format(SV("[\"\U00010000\"]"), SV("[{:?}]"), "\xf0\x90\x80\x80");                // U+10000 first valid
+
+    /* U+10FFFF..U+1FFFFF invalid range */
+    test_format(SV(R"(["\u{10ffff}"])"), SV("[{:?}]"), "\xf4\x8f\xbf\xbf"); // U+10FFFF last valid
+                                                                            // (in Supplementary Private Use Area-B)
+    test_format(SV(R"(["\x{f4}\x{90}\x{80}\x{80}"])"), SV("[{:?}]"), "\xf4\x90\x80\x80"); // U+110000
+    test_format(SV(R"(["\x{f4}\x{bf}\x{bf}\x{bf}"])"), SV("[{:?}]"), "\xf4\xbf\xbf\xbf"); // U+11FFFF
   } else {
     // Valid UTF-16 and UTF-32
     test_format(SV("[\"\u00c3\"]"), SV("[{:?}]"), L"\xc3"); // LATIN CAPITAL LETTER A WITH TILDE
@@ -320,11 +357,8 @@ void test_string() {
     // Format
     test_format(V{LR"("\u{ad}\u{600}\u{feff}")"}, L"{:?}", L"\xad\x600\xfeff");
 
-    if constexpr (sizeof(CharT) == 2)
-      // Incomplete surrogate pair in UTF-16
-      test_format(V{LR"("\x{d800}")"}, L"{:?}", L"\xd800");
-    else
-      test_format(V{LR"("\u{d800}")"}, L"{:?}", L"\xd800");
+    // Incomplete surrogate pair in UTF-16
+    test_format(V{LR"("\x{d800}")"}, L"{:?}", L"\xd800");
 
     // Private_Use
     test_format(V{LR"("\u{e000}\u{f8ff}")"}, L"{:?}", L"\xe000\xf8ff");
@@ -341,7 +375,7 @@ void test_string() {
     // Unicode fitting in a 32-bit wchar_t
 
     constexpr wchar_t x  = 0x1ffff;
-    constexpr uint32_t y = 0x1ffff;
+    constexpr std::uint32_t y = 0x1ffff;
     static_assert(x == y);
 
     using V = std::basic_string_view<CharT>;
diff --git a/libcxx/test/std/utilities/format/format.functions/format_tests.h b/libcxx/test/std/utilities/format/format.functions/format_tests.h
index a62dd6a822950..0337d4c9767c1 100644
--- a/libcxx/test/std/utilities/format/format.functions/format_tests.h
+++ b/libcxx/test/std/utilities/format/format.functions/format_tests.h
@@ -663,27 +663,27 @@ void format_test_signed_integer(TestFunction check, ExceptionTest check_exceptio
   format_test_integer<__int128_t, CharT>(check, check_exception);
 #endif
   // *** check the minma and maxima ***
-  check(SV("-0b10000000"), SV("{:#b}"), std::numeric_limits<int8_t>::min());
-  check(SV("-0200"), SV("{:#o}"), std::numeric_limits<int8_t>::min());
-  check(SV("-128"), SV("{:#}"), std::numeric_limits<int8_t>::min());
-  check(SV("-0x80"), SV("{:#x}"), std::numeric_limits<int8_t>::min());
+  check(SV("-0b10000000"), SV("{:#b}"), std::numeric_limits<std::int8_t>::min());
+  check(SV("-0200"), SV("{:#o}"), std::numeric_limits<std::int8_t>::min());
+  check(SV("-128"), SV("{:#}"), std::numeric_limits<std::int8_t>::min());
+  check(SV("-0x80"), SV("{:#x}"), std::numeric_limits<std::int8_t>::min());
 
-  check(SV("-0b1000000000000000"), SV("{:#b}"), std::numeric_limits<int16_t>::min());
-  check(SV("-0100000"), SV("{:#o}"), std::numeric_limits<int16_t>::min());
-  check(SV("-32768"), SV("{:#}"), std::numeric_limits<int16_t>::min());
-  check(SV("-0x8000"), SV("{:#x}"), std::numeric_limits<int16_t>::min());
+  check(SV("-0b1000000000000000"), SV("{:#b}"), std::numeric_limits<std::int16_t>::min());
+  check(SV("-0100000"), SV("{:#o}"), std::numeric_limits<std::int16_t>::min());
+  check(SV("-32768"), SV("{:#}"), std::numeric_limits<std::int16_t>::min());
+  check(SV("-0x8000"), SV("{:#x}"), std::numeric_limits<std::int16_t>::min());
 
-  check(SV("-0b10000000000000000000000000000000"), SV("{:#b}"), std::numeric_limits<int32_t>::min());
-  check(SV("-020000000000"), SV("{:#o}"), std::numeric_limits<int32_t>::min());
-  check(SV("-2147483648"), SV("{:#}"), std::numeric_limits<int32_t>::min());
-  check(SV("-0x80000000"), SV("{:#x}"), std::numeric_limits<int32_t>::min());
+  check(SV("-0b10000000000000000000000000000000"), SV("{:#b}"), std::numeric_limits<std::int32_t>::min());
+  check(SV("-020000000000"), SV("{:#o}"), std::numeric_limits<std::int32_t>::min());
+  check(SV("-2147483648"), SV("{:#}"), std::numeric_limits<std::int32_t>::min());
+  check(SV("-0x80000000"), SV("{:#x}"), std::numeric_limits<std::int32_t>::min());
 
   check(SV("-0b1000000000000000000000000000000000000000000000000000000000000000"),
         SV("{:#b}"),
-        std::numeric_limits<int64_t>::min());
-  check(SV("-01000000000000000000000"), SV("{:#o}"), std::numeric_limits<int64_t>::min());
-  check(SV("-9223372036854775808"), SV("{:#}"), std::numeric_limits<int64_t>::min());
-  check(SV("-0x8000000000000000"), SV("{:#x}"), std::numeric_limits<int64_t>::min());
+        std::numeric_limits<std::int64_t>::min());
+  check(SV("-01000000000000000000000"), SV("{:#o}"), std::numeric_limits<std::int64_t>::min());
+  check(SV("-9223372036854775808"), SV("{:#}"), std::numeric_limits<std::int64_t>::min());
+  check(SV("-0x8000000000000000"), SV("{:#x}"), std::numeric_limits<std::int64_t>::min());
 
 #ifndef TEST_HAS_NO_INT128
   check(SV("-0b1000000000000000000000000000000000000000000000000000000000000000"
@@ -695,27 +695,27 @@ void format_test_signed_integer(TestFunction check, ExceptionTest check_exceptio
   check(SV("-0x80000000000000000000000000000000"), SV("{:#x}"), std::numeric_limits<__int128_t>::min());
 #endif
 
-  check(SV("0b1111111"), SV("{:#b}"), std::numeric_limits<int8_t>::max());
-  check(SV("0177"), SV("{:#o}"), std::numeric_limits<int8_t>::max());
-  check(SV("127"), SV("{:#}"), std::numeric_limits<int8_t>::max());
-  check(SV("0x7f"), SV("{:#x}"), std::numeric_limits<int8_t>::max());
+  check(SV("0b1111111"), SV("{:#b}"), std::numeric_limits<std::int8_t>::max());
+  check(SV("0177"), SV("{:#o}"), std::numeric_limits<std::int8_t>::max());
+  check(SV("127"), SV("{:#}"), std::numeric_limits<std::int8_t>::max());
+  check(SV("0x7f"), SV("{:#x}"), std::numeric_limits<std::int8_t>::max());
 
-  check(SV("0b111111111111111"), SV("{:#b}"), std::numeric_limits<int16_t>::max());
-  check(SV("077777"), SV("{:#o}"), std::numeric_limits<int16_t>::max());
-  check(SV("32767"), SV("{:#}"), std::numeric_limits<int16_t>::max());
-  check(SV("0x7fff"), SV("{:#x}"), std::numeric_limits<int16_t>::max());
+  check(SV("0b111111111111111"), SV("{:#b}"), std::numeric_limits<std::int16_t>::max());
+  check(SV("077777"), SV("{:#o}"), std::numeric_limits<std::int16_t>::max());
+  check(SV("32767"), SV("{:#}"), std::numeric_limits<std::int16_t>::max());
+  check(SV("0x7fff"), SV("{:#x}"), std::numeric_limits<std::int16_t>::max());
 
-  check(SV("0b1111111111111111111111111111111"), SV("{:#b}"), std::numeric_limits<int32_t>::max());
-  check(SV("017777777777"), SV("{:#o}"), std::numeric_limits<int32_t>::max());
-  check(SV("2147483647"), SV("{:#}"), std::numeric_limits<int32_t>::max());
-  check(SV("0x7fffffff"), SV("{:#x}"), std::numeric_limits<int32_t>::max());
+  check(SV("0b1111111111111111111111111111111"), SV("{:#b}"), std::numeric_limits<std::int32_t>::max());
+  check(SV("017777777777"), SV("{:#o}"), std::numeric_limits<std::int32_t>::max());
+  check(SV("2147483647"), SV("{:#}"), std::numeric_limits<std::int32_t>::max());
+  check(SV("0x7fffffff"), SV("{:#x}"), std::numeric_limits<std::int32_t>::max());
 
   check(SV("0b111111111111111111111111111111111111111111111111111111111111111"),
         SV("{:#b}"),
-        std::numeric_limits<int64_t>::max());
-  check(SV("0777777777777777777777"), SV("{:#o}"), std::numeric_limits<int64_t>::max());
-  check(SV("9223372036854775807"), SV("{:#}"), std::numeric_limits<int64_t>::max());
-  check(SV("0x7fffffffffffffff"), SV("{:#x}"), std::numeric_limits<int64_t>::max());
+        std::numeric_limits<std::int64_t>::max());
+  check(SV("0777777777777777777777"), SV("{:#o}"), std::numeric_limits<std::int64_t>::max());
+  check(SV("9223372036854775807"), SV("{:#}"), std::numeric_limits<std::int64_t>::max());
+  check(SV("0x7fffffffffffffff"), SV("{:#x}"), std::numeric_limits<std::int64_t>::max());
 
 #ifndef TEST_HAS_NO_INT128
   check(SV("0b111111111111111111111111111111111111111111111111111111111111111"
@@ -739,27 +739,27 @@ void format_test_unsigned_integer(TestFunction check, ExceptionTest check_except
   format_test_integer<__uint128_t, CharT>(check, check_exception);
 #endif
   // *** test the maxima ***
-  check(SV("0b11111111"), SV("{:#b}"), std::numeric_limits<uint8_t>::max());
-  check(SV("0377"), SV("{:#o}"), std::numeric_limits<uint8_t>::max());
-  check(SV("255"), SV("{:#}"), std::numeric_limits<uint8_t>::max());
-  check(SV("0xff"), SV("{:#x}"), std::numeric_limits<uint8_t>::max());
+  check(SV("0b11111111"), SV("{:#b}"), std::numeric_limits<std::uint8_t>::max());
+  check(SV("0377"), SV("{:#o}"), std::numeric_limits<std::uint8_t>::max());
+  check(SV("255"), SV("{:#}"), std::numeric_limits<std::uint8_t>::max());
+  check(SV("0xff"), SV("{:#x}"), std::numeric_limits<std::uint8_t>::max());
 
-  check(SV("0b1111111111111111"), SV("{:#b}"), std::numeric_limits<uint16_t>::max());
-  check(SV("0177777"), SV("{:#o}"), std::numeric_limits<uint16_t>::max());
-  check(SV("65535"), SV("{:#}"), std::numeric_limits<uint16_t>::max());
-  check(SV("0xffff"), SV("{:#x}"), std::numeric_limits<uint16_t>::max());
+  check(SV("0b1111111111111111"), SV("{:#b}"), std::numeric_limits<std::uint16_t>::max());
+  check(SV("0177777"), SV("{:#o}"), std::numeric_limits<std::uint16_t>::max());
+  check(SV("65535"), SV("{:#}"), std::numeric_limits<std::uint16_t>::max());
+  check(SV("0xffff"), SV("{:#x}"), std::numeric_limits<std::uint16_t>::max());
 
-  check(SV("0b11111111111111111111111111111111"), SV("{:#b}"), std::numeric_limits<uint32_t>::max());
-  check(SV("037777777777"), SV("{:#o}"), std::numeric_limits<uint32_t>::max());
-  check(SV("4294967295"), SV("{:#}"), std::numeric_limits<uint32_t>::max());
-  check(SV("0xffffffff"), SV("{:#x}"), std::numeric_limits<uint32_t>::max());
+  check(SV("0b11111111111111111111111111111111"), SV("{:#b}"), std::numeric_limits<std::uint32_t>::max());
+  check(SV("037777777777"), SV("{:#o}"), std::numeric_limits<std::uint32_t>::max());
+  check(SV("4294967295"), SV("{:#}"), std::numeric_limits<std::uint32_t>::max());
+  check(SV("0xffffffff"), SV("{:#x}"), std::numeric_limits<std::uint32_t>::max());
 
   check(SV("0b1111111111111111111111111111111111111111111111111111111111111111"),
         SV("{:#b}"),
-        std::numeric_limits<uint64_t>::max());
-  check(SV("01777777777777777777777"), SV("{:#o}"), std::numeric_limits<uint64_t>::max());
-  check(SV("18446744073709551615"), SV("{:#}"), std::numeric_limits<uint64_t>::max());
-  check(SV("0xffffffffffffffff"), SV("{:#x}"), std::numeric_limits<uint64_t>::max());
+        std::numeric_limits<std::uint64_t>::max());
+  check(SV("01777777777777777777777"), SV("{:#o}"), std::numeric_limits<std::uint64_t>::max());
+  check(SV("18446744073709551615"), SV("{:#}"), std::numeric_limits<std::uint64_t>::max());
+  check(SV("0xffffffffffffffff"), SV("{:#x}"), std::numeric_limits<std::uint64_t>::max());
 
 #ifndef TEST_HAS_NO_INT128
   check(SV("0b1111111111111111111111111111111111111111111111111111111111111111"
@@ -2367,6 +2367,7 @@ void format_test_floating_point_default_precision(TestFunction check) {
 
   // *** alternate form **
   // When precision is zero there's no decimal point except when the alternate form is specified.
+  // Note unlike the g and G option the trailing zeros are still removed.
   check(SV("answer is '0'"), SV("answer is '{:.0}'"), F(0));
   check(SV("answer is '0.'"), SV("answer is '{:#.0}'"), F(0));
 
@@ -2449,6 +2450,57 @@ void format_test_floating_point_default_precision(TestFunction check) {
   // See locale-specific_form.pass.cpp
 }
 
+template <class F, class CharT, class TestFunction>
+void format_test_floating_point_PR58714(TestFunction check) {
+  check(SV("+1234"), SV("{:+}"), F(1234.0));
+  check(SV("+1.348p+10"), SV("{:+a}"), F(1234.0));
+  check(SV("+1.234000e+03"), SV("{:+e}"), F(1234.0));
+  check(SV("+1234.000000"), SV("{:+f}"), F(1234.0));
+  check(SV("+1234"), SV("{:+g}"), F(1234.0));
+
+  check(SV("1234."), SV("{:#}"), F(1234.0));
+  check(SV("1.348p+10"), SV("{:#a}"), F(1234.0));
+  check(SV("1.234000e+03"), SV("{:#e}"), F(1234.0));
+  check(SV("1234.000000"), SV("{:#f}"), F(1234.0));
+  check(SV("1234.00"), SV("{:#g}"), F(1234.0));
+
+  check(SV("4.e+30"), SV("{:#}"), F(4.0e+30));
+  check(SV("1.p+102"), SV("{:#a}"), F(0x4.0p+100));
+  check(SV("4.000000e+30"), SV("{:#e}"), F(4.0e+30));
+  check(SV("5070602400912917605986812821504.000000"), SV("{:#f}"), F(0x4.0p+100));
+  check(SV("4.00000e+30"), SV("{:#g}"), F(4.0e+30));
+
+  check(SV("1234."), SV("{:#.6}"), F(1234.0)); // # does not restore zeros
+  check(SV("1.348000p+10"), SV("{:#.6a}"), F(1234.0));
+  check(SV("1.234000e+03"), SV("{:#.6e}"), F(1234.0));
+  check(SV("1234.000000"), SV("{:#.6f}"), F(1234.0));
+  check(SV("1234.00"), SV("{:#.6g}"), F(1234.0));
+
+  check(SV("-1234."), SV("{:#}"), F(-1234.0));
+  check(SV("-1.348p+10"), SV("{:#a}"), F(-1234.0));
+  check(SV("-1.234000e+03"), SV("{:#e}"), F(-1234.0));
+  check(SV("-1234.000000"), SV("{:#f}"), F(-1234.0));
+  check(SV("-1234.00"), SV("{:#g}"), F(-1234.0));
+
+  check(SV("-1234."), SV("{:#.6}"), F(-1234.0)); // # does not restore zeros
+  check(SV("-1.348000p+10"), SV("{:#.6a}"), F(-1234.0));
+  check(SV("-1.234000e+03"), SV("{:#.6e}"), F(-1234.0));
+  check(SV("-1234.000000"), SV("{:#.6f}"), F(-1234.0));
+  check(SV("-1234.00"), SV("{:#.6g}"), F(-1234.0));
+
+  check(SV("+1234."), SV("{:+#}"), F(1234.0));
+  check(SV("+1.348p+10"), SV("{:+#a}"), F(1234.0));
+  check(SV("+1.234000e+03"), SV("{:+#e}"), F(1234.0));
+  check(SV("+1234.000000"), SV("{:+#f}"), F(1234.0));
+  check(SV("+1234.00"), SV("{:+#g}"), F(1234.0));
+
+  check(SV("+1234."), SV("{:+#.6}"), F(1234.0)); // # does not restore zeros
+  check(SV("+1.348000p+10"), SV("{:+#.6a}"), F(1234.0));
+  check(SV("+1.234000e+03"), SV("{:+#.6e}"), F(1234.0));
+  check(SV("+1234.000000"), SV("{:+#.6f}"), F(1234.0));
+  check(SV("+1234.00"), SV("{:+#.6g}"), F(1234.0));
+}
+
 template <class F, class CharT, class TestFunction, class ExceptionTest>
 void format_test_floating_point(TestFunction check, ExceptionTest check_exception) {
   format_test_floating_point_hex_lower_case<F, CharT>(check);
@@ -2468,6 +2520,8 @@ void format_test_floating_point(TestFunction check, ExceptionTest check_exceptio
   format_test_floating_point_default<F, CharT>(check);
   format_test_floating_point_default_precision<F, CharT>(check);
 
+  format_test_floating_point_PR58714<F, CharT>(check);
+
   // *** type ***
   for (const auto& fmt : invalid_types<CharT>("aAeEfFgG"))
     check_exception("The format-spec type has a type not supported for a floating-point argument", fmt, F(1));
diff --git a/libcxx/test/std/utilities/format/format.range/format.range.fmtdef/format.pass.cpp b/libcxx/test/std/utilities/format/format.range/format.range.fmtdef/format.pass.cpp
new file mode 100644
index 0000000000000..31b7e5658b125
--- /dev/null
+++ b/libcxx/test/std/utilities/format/format.range/format.range.fmtdef/format.pass.cpp
@@ -0,0 +1,72 @@
+//===----------------------------------------------------------------------===//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// UNSUPPORTED: c++03, c++11, c++14, c++17, c++20
+// UNSUPPORTED: libcpp-has-no-incomplete-format
+
+// TODO FMT Fix this test using GCC, it currently times out.
+// UNSUPPORTED: gcc-12
+
+// This test requires the dylib support introduced in D92214.
+// XFAIL: use_system_cxx_lib && target={{.+}}-apple-macosx10.{{.+}}
+// XFAIL: use_system_cxx_lib && target={{.+}}-apple-macosx11.{{.+}}
+
+// <format>
+
+// template<ranges::input_range R, class charT>
+//   struct range-default-formatter<range_format::sequence, R, charT>
+
+// template<class FormatContext>
+//   typename FormatContext::iterator
+//     format(maybe-const-r& elems, FormatContext& ctx) const;
+
+#include <array>
+#include <cassert>
+#include <concepts>
+#include <format>
+#include <iterator>
+
+#include "test_format_context.h"
+#include "test_macros.h"
+#include "make_string.h"
+
+#define SV(S) MAKE_STRING_VIEW(CharT, S)
+
+template <class StringViewT>
+void test_format(StringViewT expected, std::array<int, 2> arg) {
+  using CharT      = typename StringViewT::value_type;
+  using String     = std::basic_string<CharT>;
+  using OutIt      = std::back_insert_iterator<String>;
+  using FormatCtxT = std::basic_format_context<OutIt, CharT>;
+
+  std::formatter<std::array<int, 2>, CharT> formatter;
+
+  String result;
+  OutIt out             = std::back_inserter(result);
+  FormatCtxT format_ctx = test_format_context_create<OutIt, CharT>(out, std::make_format_args<FormatCtxT>(arg));
+  formatter.format(arg, format_ctx);
+  assert(result == expected);
+}
+
+template <class CharT>
+void test_fmt() {
+  test_format(SV("[1, 42]"), std::array<int, 2>{{1, 42}});
+  test_format(SV("[0, 99]"), std::array<int, 2>{{0, 99}});
+}
+
+void test() {
+  test_fmt<char>();
+#ifndef TEST_HAS_NO_WIDE_CHARACTERS
+  test_fmt<wchar_t>();
+#endif
+}
+
+int main(int, char**) {
+  test();
+
+  return 0;
+}
diff --git a/libcxx/test/std/utilities/format/format.range/format.range.fmtdef/parse.pass.cpp b/libcxx/test/std/utilities/format/format.range/format.range.fmtdef/parse.pass.cpp
new file mode 100644
index 0000000000000..a7a160989b336
--- /dev/null
+++ b/libcxx/test/std/utilities/format/format.range/format.range.fmtdef/parse.pass.cpp
@@ -0,0 +1,72 @@
+//===----------------------------------------------------------------------===//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// UNSUPPORTED: c++03, c++11, c++14, c++17, c++20
+// UNSUPPORTED: libcpp-has-no-incomplete-format
+
+// TODO FMT Fix this test using GCC, it currently times out.
+// UNSUPPORTED: gcc-12
+
+// This test requires the dylib support introduced in D92214.
+// XFAIL: use_system_cxx_lib && target={{.+}}-apple-macosx10.{{.+}}
+// XFAIL: use_system_cxx_lib && target={{.+}}-apple-macosx11.{{.+}}
+
+// <format>
+
+// template<ranges::input_range R, class charT>
+//   struct range-default-formatter<range_format::sequence, R, charT>
+
+// template<class ParseContext>
+//   constexpr typename ParseContext::iterator
+//     parse(ParseContext& ctx);
+
+#include <array>
+#include <cassert>
+#include <concepts>
+#include <format>
+
+#include "test_format_context.h"
+#include "test_macros.h"
+#include "make_string.h"
+
+#define SV(S) MAKE_STRING_VIEW(CharT, S)
+
+template <class StringViewT>
+constexpr void test_parse(StringViewT fmt) {
+  using CharT    = typename StringViewT::value_type;
+  auto parse_ctx = std::basic_format_parse_context<CharT>(fmt);
+  std::formatter<std::array<int, 2>, CharT> formatter;
+  static_assert(std::semiregular<decltype(formatter)>);
+
+  std::same_as<typename StringViewT::iterator> auto it = formatter.parse(parse_ctx);
+  assert(it == fmt.end() - (!fmt.empty() && fmt.back() == '}'));
+}
+
+template <class CharT>
+constexpr void test_fmt() {
+  test_parse(SV(""));
+  test_parse(SV(":5"));
+
+  test_parse(SV("}"));
+  test_parse(SV(":5}"));
+}
+
+constexpr bool test() {
+  test_fmt<char>();
+#ifndef TEST_HAS_NO_WIDE_CHARACTERS
+  test_fmt<wchar_t>();
+#endif
+
+  return true;
+}
+
+int main(int, char**) {
+  test();
+  static_assert(test());
+
+  return 0;
+}
diff --git a/libcxx/test/std/utilities/format/format.range/format.range.fmtdef/set_brackets.pass.cpp b/libcxx/test/std/utilities/format/format.range/format.range.fmtdef/set_brackets.pass.cpp
new file mode 100644
index 0000000000000..6405b5ec22e51
--- /dev/null
+++ b/libcxx/test/std/utilities/format/format.range/format.range.fmtdef/set_brackets.pass.cpp
@@ -0,0 +1,72 @@
+//===----------------------------------------------------------------------===//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// UNSUPPORTED: c++03, c++11, c++14, c++17, c++20
+// UNSUPPORTED: libcpp-has-no-incomplete-format
+
+// TODO FMT Fix this test using GCC, it currently times out.
+// UNSUPPORTED: gcc-12
+
+// This test requires the dylib support introduced in D92214.
+// XFAIL: use_system_cxx_lib && target={{.+}}-apple-macosx10.{{.+}}
+// XFAIL: use_system_cxx_lib && target={{.+}}-apple-macosx11.{{.+}}
+
+// <format>
+
+// template<ranges::input_range R, class charT>
+//   struct range-default-formatter<range_format::sequence, R, charT>
+
+// constexpr void constexpr void set_brackets(basic_string_view<charT> opening,
+//                                            basic_string_view<charT> closing) noexcept;
+
+#include <format>
+#include <cassert>
+#include <iterator>
+#include <type_traits>
+#include <vector>
+
+#include "make_string.h"
+#include "test_format_context.h"
+
+#define SV(S) MAKE_STRING_VIEW(CharT, S)
+
+template <class CharT>
+constexpr void test_setter() {
+  std::formatter<std::vector<int>, CharT> formatter;
+  formatter.set_brackets(SV("open"), SV("close"));
+  // Note the SV macro may throw, so can't use it.
+  static_assert(noexcept(formatter.set_brackets(std::basic_string_view<CharT>{}, std::basic_string_view<CharT>{})));
+
+  // Note there is no direct way to validate this function modified the object.
+  if (!std::is_constant_evaluated()) {
+    using String     = std::basic_string<CharT>;
+    using OutIt      = std::back_insert_iterator<String>;
+    using FormatCtxT = std::basic_format_context<OutIt, CharT>;
+
+    String result;
+    OutIt out             = std::back_inserter(result);
+    FormatCtxT format_ctx = test_format_context_create<OutIt, CharT>(out, std::make_format_args<FormatCtxT>());
+    formatter.format(std::vector<int>{0, 42, 99}, format_ctx);
+    assert(result == SV("open0, 42, 99close"));
+  }
+}
+
+constexpr bool test() {
+  test_setter<char>();
+#ifndef TEST_HAS_NO_WIDE_CHARACTERS
+  test_setter<wchar_t>();
+#endif
+
+  return true;
+}
+
+int main(int, char**) {
+  test();
+  static_assert(test());
+
+  return 0;
+}
diff --git a/libcxx/test/std/utilities/format/format.range/format.range.fmtdef/set_separator.pass.cpp b/libcxx/test/std/utilities/format/format.range/format.range.fmtdef/set_separator.pass.cpp
new file mode 100644
index 0000000000000..21cee612bb2b0
--- /dev/null
+++ b/libcxx/test/std/utilities/format/format.range/format.range.fmtdef/set_separator.pass.cpp
@@ -0,0 +1,71 @@
+//===----------------------------------------------------------------------===//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// UNSUPPORTED: c++03, c++11, c++14, c++17, c++20
+// UNSUPPORTED: libcpp-has-no-incomplete-format
+
+// TODO FMT Fix this test using GCC, it currently times out.
+// UNSUPPORTED: gcc-12
+
+// This test requires the dylib support introduced in D92214.
+// XFAIL: use_system_cxx_lib && target={{.+}}-apple-macosx10.{{.+}}
+// XFAIL: use_system_cxx_lib && target={{.+}}-apple-macosx11.{{.+}}
+
+// <format>
+
+// template<ranges::input_range R, class charT>
+//   struct range-default-formatter<range_format::sequence, R, charT>
+
+// constexpr void set_separator(basic_string_view<charT> sep) noexcept;
+
+#include <format>
+#include <cassert>
+#include <iterator>
+#include <type_traits>
+#include <vector>
+
+#include "make_string.h"
+#include "test_format_context.h"
+
+#define SV(S) MAKE_STRING_VIEW(CharT, S)
+
+template <class CharT>
+constexpr void test_setter() {
+  std::formatter<std::vector<int>, CharT> formatter;
+  formatter.set_separator(SV("sep"));
+  // Note the SV macro may throw, so can't use it.
+  static_assert(noexcept(formatter.set_separator(std::basic_string_view<CharT>{})));
+
+  // Note there is no direct way to validate this function modified the object.
+  if (!std::is_constant_evaluated()) {
+    using String     = std::basic_string<CharT>;
+    using OutIt      = std::back_insert_iterator<String>;
+    using FormatCtxT = std::basic_format_context<OutIt, CharT>;
+
+    String result;
+    OutIt out             = std::back_inserter(result);
+    FormatCtxT format_ctx = test_format_context_create<OutIt, CharT>(out, std::make_format_args<FormatCtxT>());
+    formatter.format(std::vector<int>{0, 42, 99}, format_ctx);
+    assert(result == SV("[0sep42sep99]"));
+  }
+}
+
+constexpr bool test() {
+  test_setter<char>();
+#ifndef TEST_HAS_NO_WIDE_CHARACTERS
+  test_setter<wchar_t>();
+#endif
+
+  return true;
+}
+
+int main(int, char**) {
+  test();
+  static_assert(test());
+
+  return 0;
+}
diff --git a/libcxx/test/std/utilities/format/format.range/format.range.fmtset/format.functions.tests.h b/libcxx/test/std/utilities/format/format.range/format.range.fmtset/format.functions.tests.h
index a884ba7ec8585..cd0bc5ad1d81e 100644
--- a/libcxx/test/std/utilities/format/format.range/format.range.fmtset/format.functions.tests.h
+++ b/libcxx/test/std/utilities/format/format.range/format.range.fmtset/format.functions.tests.h
@@ -1260,7 +1260,7 @@ void test_tuple_int(TestFunction check, ExceptionTest check_exception) {
   check(SV("__(42), (99)___"), SV("{:_^15n}"), input);
 
   // *** type ***
-  check(SV("__{(42), (99)}___"), SV("{:_^17m}"), input);
+  check_exception("The range-format-spec type m requires two elements for a pair or tuple", SV("{:m}"), input);
   check_exception("The range-format-spec type s requires formatting a character type", SV("{:s}"), input);
   check_exception("The range-format-spec type ?s requires formatting a character type", SV("{:?s}"), input);
 
@@ -1366,7 +1366,7 @@ void test_tuple_int_int_int(TestFunction check, ExceptionTest check_exception) {
   check(SV("__(1, 10, 100), (42, 99, 0)___"), SV("{:_^30n}"), input);
 
   // *** type ***
-  check(SV("__{(1, 10, 100), (42, 99, 0)}___"), SV("{:_^32m}"), input);
+  check_exception("The range-format-spec type m requires two elements for a pair or tuple", SV("{:m}"), input);
   check_exception("The range-format-spec type s requires formatting a character type", SV("{:s}"), input);
   check_exception("The range-format-spec type ?s requires formatting a character type", SV("{:?s}"), input);
 
diff --git a/libcxx/test/std/utilities/format/format.range/format.range.formatter/format.functions.tests.h b/libcxx/test/std/utilities/format/format.range/format.range.formatter/format.functions.tests.h
index 927195133e0af..28ff1cb205099 100644
--- a/libcxx/test/std/utilities/format/format.range/format.range.formatter/format.functions.tests.h
+++ b/libcxx/test/std/utilities/format/format.range/format.range.formatter/format.functions.tests.h
@@ -1084,7 +1084,7 @@ void test_tuple_int(TestFunction check, ExceptionTest check_exception) {
   check(SV("__(42), (99)___"), SV("{:_^15n}"), input);
 
   // *** type ***
-  check(SV("__{(42), (99)}___"), SV("{:_^17m}"), input);
+  check_exception("The range-format-spec type m requires two elements for a pair or tuple", SV("{:m}"), input);
   check_exception("The range-format-spec type s requires formatting a character type", SV("{:s}"), input);
   check_exception("The range-format-spec type ?s requires formatting a character type", SV("{:?s}"), input);
   for (std::basic_string_view<CharT> fmt : fmt_invalid_types<CharT>("s"))
@@ -1185,7 +1185,7 @@ void test_tuple_int_int_int(TestFunction check, ExceptionTest check_exception) {
   check(SV("__(42, 99, 0), (1, 10, 100)___"), SV("{:_^30n}"), input);
 
   // *** type ***
-  check(SV("__{(42, 99, 0), (1, 10, 100)}___"), SV("{:_^32m}"), input);
+  check_exception("The range-format-spec type m requires two elements for a pair or tuple", SV("{:m}"), input);
   check_exception("The range-format-spec type s requires formatting a character type", SV("{:s}"), input);
   check_exception("The range-format-spec type ?s requires formatting a character type", SV("{:?s}"), input);
   for (std::basic_string_view<CharT> fmt : fmt_invalid_types<CharT>("s"))
diff --git a/libcxx/test/std/utilities/format/format.range/format.range.formatter/set_brackets.pass.cpp b/libcxx/test/std/utilities/format/format.range/format.range.formatter/set_brackets.pass.cpp
index 6a8989c583f5e..c399a81f2a041 100644
--- a/libcxx/test/std/utilities/format/format.range/format.range.formatter/set_brackets.pass.cpp
+++ b/libcxx/test/std/utilities/format/format.range/format.range.formatter/set_brackets.pass.cpp
@@ -22,7 +22,7 @@
 // class range_formatter
 
 // constexpr void constexpr void set_brackets(basic_string_view<charT> opening,
-//                                            basic_string_view<charT> closing);
+//                                            basic_string_view<charT> closing) noexcept;
 
 // Note this tests the basics of this function. It's tested in more detail in
 // the format functions test.
@@ -42,6 +42,8 @@ template <class CharT>
 constexpr void test_setter() {
   std::range_formatter<int, CharT> formatter;
   formatter.set_brackets(SV("open"), SV("close"));
+  // Note the SV macro may throw, so can't use it.
+  static_assert(noexcept(formatter.set_brackets(std::basic_string_view<CharT>{}, std::basic_string_view<CharT>{})));
 
   // Note there is no direct way to validate this function modified the object.
   if (!std::is_constant_evaluated()) {
diff --git a/libcxx/test/std/utilities/format/format.range/format.range.formatter/set_separator.pass.cpp b/libcxx/test/std/utilities/format/format.range/format.range.formatter/set_separator.pass.cpp
index e95f65edfbd4d..192ddcd16c44a 100644
--- a/libcxx/test/std/utilities/format/format.range/format.range.formatter/set_separator.pass.cpp
+++ b/libcxx/test/std/utilities/format/format.range/format.range.formatter/set_separator.pass.cpp
@@ -21,7 +21,7 @@
 //   requires same_as<remove_cvref_t<T>, T> && formattable<T, charT>
 // class range_formatter
 
-// constexpr void set_separator(basic_string_view<charT> sep);
+// constexpr void set_separator(basic_string_view<charT> sep) noexcept;
 
 // Note this tests the basics of this function. It's tested in more detail in
 // the format functions test.
@@ -41,6 +41,8 @@ template <class CharT>
 constexpr void test_setter() {
   std::range_formatter<int, CharT> formatter;
   formatter.set_separator(SV("sep"));
+  // Note the SV macro may throw, so can't use it.
+  static_assert(noexcept(formatter.set_separator(std::basic_string_view<CharT>{})));
 
   // Note there is no direct way to validate this function modified the object.
   if (!std::is_constant_evaluated()) {
diff --git a/libcxx/test/std/utilities/format/format.range/format.range.formatter/underlying.pass.cpp b/libcxx/test/std/utilities/format/format.range/format.range.formatter/underlying.pass.cpp
index 7d819d4ec6c37..10a330f0baf1e 100644
--- a/libcxx/test/std/utilities/format/format.range/format.range.formatter/underlying.pass.cpp
+++ b/libcxx/test/std/utilities/format/format.range/format.range.formatter/underlying.pass.cpp
@@ -21,8 +21,8 @@
 //   requires same_as<remove_cvref_t<T>, T> && formattable<T, charT>
 // class range_formatter
 
-// constexpr formatter<T, charT>& underlying();
-// constexpr const formatter<T, charT>& underlying() const;
+// constexpr formatter<T, charT>& underlying() noexcept;
+// constexpr const formatter<T, charT>& underlying() const noexcept;
 
 #include <concepts>
 #include <format>
@@ -34,10 +34,12 @@ constexpr void test_underlying() {
   {
     std::range_formatter<int, CharT> formatter;
     [[maybe_unused]] std::same_as<std::formatter<int, CharT>&> decltype(auto) underlying = formatter.underlying();
+    static_assert(noexcept(formatter.underlying()));
   }
   {
     const std::range_formatter<int, CharT> formatter;
     [[maybe_unused]] std::same_as<const std::formatter<int, CharT>&> decltype(auto) underlying = formatter.underlying();
+    static_assert(noexcept(formatter.underlying()));
   }
 }
 
diff --git a/libcxx/test/std/utilities/format/format.tuple/set_brackets.pass.cpp b/libcxx/test/std/utilities/format/format.tuple/set_brackets.pass.cpp
index 86c98455d0a8b..74af4f32fcf1d 100644
--- a/libcxx/test/std/utilities/format/format.tuple/set_brackets.pass.cpp
+++ b/libcxx/test/std/utilities/format/format.tuple/set_brackets.pass.cpp
@@ -21,7 +21,7 @@
 //   struct formatter<pair-or-tuple<Ts...>, charT>
 
 // constexpr void constexpr void set_brackets(basic_string_view<charT> opening,
-//                                            basic_string_view<charT> closing);
+//                                            basic_string_view<charT> closing) noexcept;
 
 // Note this tests the basics of this function. It's tested in more detail in
 // the format functions tests.
@@ -38,6 +38,8 @@ template <class CharT, class Arg>
 constexpr void test() {
   std::formatter<Arg, CharT> formatter;
   formatter.set_brackets(SV("open"), SV("close"));
+  // Note the SV macro may throw, so can't use it.
+  static_assert(noexcept(formatter.set_brackets(std::basic_string_view<CharT>{}, std::basic_string_view<CharT>{})));
 
   // Note there is no direct way to validate this function modified the object.
 }
diff --git a/libcxx/test/std/utilities/format/format.tuple/set_separator.pass.cpp b/libcxx/test/std/utilities/format/format.tuple/set_separator.pass.cpp
index fd2c827fe8f47..0258ae215ed22 100644
--- a/libcxx/test/std/utilities/format/format.tuple/set_separator.pass.cpp
+++ b/libcxx/test/std/utilities/format/format.tuple/set_separator.pass.cpp
@@ -21,7 +21,7 @@
 // template<class charT, formattable<charT>... Ts>
 //   struct formatter<pair-or-tuple<Ts...>, charT>
 
-// constexpr void set_separator(basic_string_view<charT> sep);
+// constexpr void set_separator(basic_string_view<charT> sep) noexcept;
 
 // Note this tests the basics of this function. It's tested in more detail in
 // the format functions tests.
@@ -38,6 +38,8 @@ template <class CharT, class Arg>
 constexpr void test() {
   std::formatter<Arg, CharT> formatter;
   formatter.set_separator(SV("sep"));
+  // Note the SV macro may throw, so can't use it.
+  static_assert(noexcept(formatter.set_separator(std::basic_string_view<CharT>{})));
 
   // Note there is no direct way to validate this function modified the object.
 }
diff --git a/libcxx/test/std/utilities/function.objects/func.bind_front/bind_front.verify.cpp b/libcxx/test/std/utilities/function.objects/func.bind_front/bind_front.verify.cpp
index f7d24998e09b5..ac9c97d039ba5 100644
--- a/libcxx/test/std/utilities/function.objects/func.bind_front/bind_front.verify.cpp
+++ b/libcxx/test/std/utilities/function.objects/func.bind_front/bind_front.verify.cpp
@@ -15,8 +15,6 @@
 
 #include <functional>
 
-#include "test_macros.h"
-
 constexpr int pass(const int n) { return n; }
 
 int simple(int n) { return n; }
@@ -34,8 +32,7 @@ struct NotMoveConst
 
 void testNotMoveConst(NotMoveConst) { }
 
-int main(int, char**)
-{
+void f() {
     int n = 1;
     const int c = 1;
 
@@ -45,6 +42,4 @@ int main(int, char**)
     auto d = std::bind_front(do_nothing, n); // expected-error {{no matching function for call to 'bind_front'}}
 
     auto t = std::bind_front(testNotMoveConst, NotMoveConst(0)); // expected-error {{no matching function for call to 'bind_front'}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc.verify.cpp b/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc.verify.cpp
index 49c8284b87c3c..b5d4a729cedac 100644
--- a/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc.verify.cpp
+++ b/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc.verify.cpp
@@ -17,12 +17,8 @@
 // This signature was removed in C++17
 
 #include <functional>
-#include <cassert>
+#include <memory>
 
-#include "min_allocator.h"
-
-int main(int, char**)
-{
+void f() {
     std::function<int(int)> f(std::allocator_arg, std::allocator<int>()); // expected-error {{no matching constructor for initialization of}}
-    return 0;
 }
diff --git a/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc_F.verify.cpp b/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc_F.verify.cpp
index 2353b51a2d9fc..88f81492c58f0 100644
--- a/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc_F.verify.cpp
+++ b/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc_F.verify.cpp
@@ -17,15 +17,10 @@
 // This signature was removed in C++17
 
 #include <functional>
-#include <cassert>
 #include <memory>
 
-#include "test_macros.h"
-
 void foo(int) {}
 
-int main(int, char**)
-{
+void f() {
     std::function<void(int)> f(std::allocator_arg, std::allocator<int>(), foo); // expected-error {{no matching constructor for initialization of}}
-    return 0;
 }
diff --git a/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc_function.verify.cpp b/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc_function.verify.cpp
index 4263eeef67de4..acb42fc169ab0 100644
--- a/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc_function.verify.cpp
+++ b/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc_function.verify.cpp
@@ -17,15 +17,10 @@
 // This signature was removed in C++17
 
 #include <functional>
-#include <cassert>
 #include <memory>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     typedef std::function<void(int)> F;
     F f1;
     F f2(std::allocator_arg, std::allocator<int>(), f1); // expected-error {{no matching constructor for initialization of}}
-    return 0;
 }
diff --git a/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc_nullptr.verify.cpp b/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc_nullptr.verify.cpp
index 6e9a6c44137ff..d457e21377a63 100644
--- a/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc_nullptr.verify.cpp
+++ b/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc_nullptr.verify.cpp
@@ -17,12 +17,8 @@
 // This signature was removed in C++17
 
 #include <functional>
-#include <cassert>
+#include <memory>
 
-#include "min_allocator.h"
-
-int main(int, char**)
-{
+void f() {
     std::function<int(int)> f(std::allocator_arg, std::allocator<int>(), nullptr); // expected-error {{no matching constructor for initialization of}}
-    return 0;
 }
diff --git a/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc_rfunction.verify.cpp b/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc_rfunction.verify.cpp
index b04b958f86a23..5fd6c96ce66e8 100644
--- a/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc_rfunction.verify.cpp
+++ b/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/alloc_rfunction.verify.cpp
@@ -16,13 +16,10 @@
 //
 // This signature was removed in C++17
 
-#include <cassert>
 #include <functional>
 #include <memory>
 #include <utility>
 
-#include "test_macros.h"
-
 class A
 {
     int data_[10];
@@ -52,9 +49,7 @@ int A::count = 0;
 
 int g(int) { return 0; }
 
-int main(int, char**)
-{
+void f() {
     std::function<int(int)> f = A();
     std::function<int(int)> f2(std::allocator_arg, std::allocator<A>(), std::move(f)); // expected-error {{no matching constructor for initialization of}}
-    return 0;
 }
diff --git a/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/deduct_F.verify.cpp b/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/deduct_F.verify.cpp
index cc20a01d2448f..520e5f055a71f 100644
--- a/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/deduct_F.verify.cpp
+++ b/libcxx/test/std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/deduct_F.verify.cpp
@@ -19,15 +19,13 @@
 // Make sure we stick to the specification.
 
 #include <functional>
-#include <type_traits>
 
 struct R { };
 struct f0 { R operator()() && { return {}; } };
 struct f1 { R operator()(int, ...) { return {}; } };
 
-int main(int, char**) {
+void f() {
     std::function f = f0{}; // expected-error{{no viable constructor or deduction guide for deduction of template arguments of 'function'}}
     std::function g = f1{}; // expected-error{{no viable constructor or deduction guide for deduction of template arguments of 'function'}}
     std::function h = nullptr; // expected-error{{no viable constructor or deduction guide for deduction of template arguments of 'function'}}
-    return 0;
 }
diff --git a/libcxx/test/std/utilities/function.objects/negators/binary_negate.depr_in_cxx17.verify.cpp b/libcxx/test/std/utilities/function.objects/negators/binary_negate.depr_in_cxx17.verify.cpp
index 1f6e153021e8a..5e7947f318f77 100644
--- a/libcxx/test/std/utilities/function.objects/negators/binary_negate.depr_in_cxx17.verify.cpp
+++ b/libcxx/test/std/utilities/function.objects/negators/binary_negate.depr_in_cxx17.verify.cpp
@@ -16,17 +16,13 @@
 
 #include <functional>
 
-#include "test_macros.h"
-
 struct Predicate {
     typedef int first_argument_type;
     typedef int second_argument_type;
     bool operator()(first_argument_type, second_argument_type) const { return true; }
 };
 
-int main(int, char**) {
+void test() {
     std::binary_negate<Predicate> f((Predicate())); // expected-warning {{'binary_negate<Predicate>' is deprecated}}
     (void)f;
-
-    return 0;
 }
diff --git a/libcxx/test/std/utilities/function.objects/negators/not1.depr_in_cxx17.verify.cpp b/libcxx/test/std/utilities/function.objects/negators/not1.depr_in_cxx17.verify.cpp
index e807a1ec6d4ce..4758a17f26328 100644
--- a/libcxx/test/std/utilities/function.objects/negators/not1.depr_in_cxx17.verify.cpp
+++ b/libcxx/test/std/utilities/function.objects/negators/not1.depr_in_cxx17.verify.cpp
@@ -16,15 +16,11 @@
 
 #include <functional>
 
-#include "test_macros.h"
-
 struct Predicate {
     typedef int argument_type;
     bool operator()(argument_type) const { return true; }
 };
 
-int main(int, char**) {
+void f() {
     std::not1(Predicate()); // expected-warning {{'not1<Predicate>' is deprecated}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/utilities/function.objects/negators/not2.depr_in_cxx17.verify.cpp b/libcxx/test/std/utilities/function.objects/negators/not2.depr_in_cxx17.verify.cpp
index 7e079f5d35f85..e6e1637f7e820 100644
--- a/libcxx/test/std/utilities/function.objects/negators/not2.depr_in_cxx17.verify.cpp
+++ b/libcxx/test/std/utilities/function.objects/negators/not2.depr_in_cxx17.verify.cpp
@@ -16,16 +16,12 @@
 
 #include <functional>
 
-#include "test_macros.h"
-
 struct Predicate {
     typedef int first_argument_type;
     typedef int second_argument_type;
     bool operator()(first_argument_type, second_argument_type) const { return true; }
 };
 
-int main(int, char**) {
+void f() {
     std::not2(Predicate()); // expected-warning {{'not2<Predicate>' is deprecated}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/utilities/function.objects/negators/unary_negate.depr_in_cxx17.verify.cpp b/libcxx/test/std/utilities/function.objects/negators/unary_negate.depr_in_cxx17.verify.cpp
index 1cf16297c2878..70c871fd47a90 100644
--- a/libcxx/test/std/utilities/function.objects/negators/unary_negate.depr_in_cxx17.verify.cpp
+++ b/libcxx/test/std/utilities/function.objects/negators/unary_negate.depr_in_cxx17.verify.cpp
@@ -16,16 +16,12 @@
 
 #include <functional>
 
-#include "test_macros.h"
-
 struct Predicate {
     typedef int argument_type;
     bool operator()(argument_type) const { return true; }
 };
 
-int main(int, char**) {
+void test() {
     std::unary_negate<Predicate> f((Predicate())); // expected-warning {{'unary_negate<Predicate>' is deprecated}}
     (void)f;
-
-  return 0;
 }
diff --git a/libcxx/test/std/utilities/function.objects/unord.hash/enum.pass.cpp b/libcxx/test/std/utilities/function.objects/unord.hash/enum.pass.cpp
index 6dd14dc77496a..54e9f25275cf2 100644
--- a/libcxx/test/std/utilities/function.objects/unord.hash/enum.pass.cpp
+++ b/libcxx/test/std/utilities/function.objects/unord.hash/enum.pass.cpp
@@ -23,7 +23,7 @@ enum class Colors { red, orange, yellow, green, blue, indigo, violet };
 enum class Cardinals { zero, one, two, three, five=5 };
 enum class LongColors : short { red, orange, yellow, green, blue, indigo, violet };
 enum class ShortColors : long { red, orange, yellow, green, blue, indigo, violet };
-enum class EightBitColors : uint8_t { red, orange, yellow, green, blue, indigo, violet };
+enum class EightBitColors : std::uint8_t { red, orange, yellow, green, blue, indigo, violet };
 
 enum Fruits { apple, pear, grape, mango, cantaloupe };
 
diff --git a/libcxx/test/std/utilities/function.objects/unord.hash/integral.pass.cpp b/libcxx/test/std/utilities/function.objects/unord.hash/integral.pass.cpp
index 2cfed3e2ba22d..622a99cf0b4e8 100644
--- a/libcxx/test/std/utilities/function.objects/unord.hash/integral.pass.cpp
+++ b/libcxx/test/std/utilities/function.objects/unord.hash/integral.pass.cpp
@@ -74,38 +74,38 @@ int main(int, char**)
     test<std::ptrdiff_t>();
     test<size_t>();
 
-    test<int8_t>();
-    test<int16_t>();
-    test<int32_t>();
-    test<int64_t>();
+    test<std::int8_t>();
+    test<std::int16_t>();
+    test<std::int32_t>();
+    test<std::int64_t>();
 
-    test<int_fast8_t>();
-    test<int_fast16_t>();
-    test<int_fast32_t>();
-    test<int_fast64_t>();
+    test<std::int_fast8_t>();
+    test<std::int_fast16_t>();
+    test<std::int_fast32_t>();
+    test<std::int_fast64_t>();
 
-    test<int_least8_t>();
-    test<int_least16_t>();
-    test<int_least32_t>();
-    test<int_least64_t>();
+    test<std::int_least8_t>();
+    test<std::int_least16_t>();
+    test<std::int_least32_t>();
+    test<std::int_least64_t>();
 
     test<intmax_t>();
     test<intptr_t>();
 
-    test<uint8_t>();
-    test<uint16_t>();
-    test<uint32_t>();
-    test<uint64_t>();
+    test<std::uint8_t>();
+    test<std::uint16_t>();
+    test<std::uint32_t>();
+    test<std::uint64_t>();
 
-    test<uint_fast8_t>();
-    test<uint_fast16_t>();
-    test<uint_fast32_t>();
-    test<uint_fast64_t>();
+    test<std::uint_fast8_t>();
+    test<std::uint_fast16_t>();
+    test<std::uint_fast32_t>();
+    test<std::uint_fast64_t>();
 
-    test<uint_least8_t>();
-    test<uint_least16_t>();
-    test<uint_least32_t>();
-    test<uint_least64_t>();
+    test<std::uint_least8_t>();
+    test<std::uint_least16_t>();
+    test<std::uint_least32_t>();
+    test<std::uint_least64_t>();
 
     test<uintmax_t>();
     test<uintptr_t>();
diff --git a/libcxx/test/std/utilities/memory/allocator.tag/allocator_arg.pass.cpp b/libcxx/test/std/utilities/memory/allocator.tag/allocator_arg.pass.cpp
index e22bec1e5144c..c1177445a3a6d 100644
--- a/libcxx/test/std/utilities/memory/allocator.tag/allocator_arg.pass.cpp
+++ b/libcxx/test/std/utilities/memory/allocator.tag/allocator_arg.pass.cpp
@@ -6,6 +6,8 @@
 //
 //===----------------------------------------------------------------------===//
 
+// UNSUPPORTED: c++03
+
 // <memory>
 
 // struct allocator_arg_t { explicit allocator_arg_t() = default; };
diff --git a/libcxx/test/std/utilities/memory/allocator.traits/allocator.traits.members/allocate.verify.cpp b/libcxx/test/std/utilities/memory/allocator.traits/allocator.traits.members/allocate.verify.cpp
index 7e3c41ced847e..adb4d697e1324 100644
--- a/libcxx/test/std/utilities/memory/allocator.traits/allocator.traits.members/allocate.verify.cpp
+++ b/libcxx/test/std/utilities/memory/allocator.traits/allocator.traits.members/allocate.verify.cpp
@@ -17,36 +17,18 @@
 
 // UNSUPPORTED: c++03, c++11, c++14, c++17
 
-#include <cassert>
 #include <cstddef>
-#include <cstdint>
 #include <memory>
 
-#include "test_macros.h"
-
 template <class T>
-struct A
-{
+struct A {
     typedef T value_type;
-
-    value_type* allocate(std::size_t n)
-    {
-        assert(n == 12);
-        return reinterpret_cast<value_type*>(static_cast<std::uintptr_t>(0xEEADBEEF));
-    }
-    value_type* allocate(std::size_t n, const void* p)
-    {
-        assert(n == 11);
-        assert(p == 0);
-        return reinterpret_cast<value_type*>(static_cast<std::uintptr_t>(0xFEADBEEF));
-    }
+    value_type* allocate(std::size_t n);
+    value_type* allocate(std::size_t n, const void* p);
 };
 
-int main(int, char**)
-{
+void f() {
     A<int> a;
     std::allocator_traits<A<int> >::allocate(a, 10);          // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
     std::allocator_traits<A<int> >::allocate(a, 10, nullptr); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/utilities/memory/allocator.uses/allocator.uses.trait/uses_allocator.pass.cpp b/libcxx/test/std/utilities/memory/allocator.uses/allocator.uses.trait/uses_allocator.pass.cpp
index dc52cb8635937..5d200a4bf808f 100644
--- a/libcxx/test/std/utilities/memory/allocator.uses/allocator.uses.trait/uses_allocator.pass.cpp
+++ b/libcxx/test/std/utilities/memory/allocator.uses/allocator.uses.trait/uses_allocator.pass.cpp
@@ -12,6 +12,7 @@
 
 #include <memory>
 #include <vector>
+#include <type_traits>
 
 #include "test_macros.h"
 
diff --git a/libcxx/test/std/utilities/memory/default.allocator/allocator.members/allocate.constexpr.size.verify.cpp b/libcxx/test/std/utilities/memory/default.allocator/allocator.members/allocate.constexpr.size.verify.cpp
index 717ba0d3b4336..71db96807f55f 100644
--- a/libcxx/test/std/utilities/memory/default.allocator/allocator.members/allocate.constexpr.size.verify.cpp
+++ b/libcxx/test/std/utilities/memory/default.allocator/allocator.members/allocate.constexpr.size.verify.cpp
@@ -13,7 +13,6 @@
 
 // UNSUPPORTED: c++03, c++11, c++14, c++17
 
-#include <cassert>
 #include <cstddef>
 #include <memory>
 
@@ -33,9 +32,7 @@ constexpr bool test()
     return true;
 }
 
-int main(int, char**)
-{
+void f() {
     static_assert(test<double>()); // expected-error-re {{{{(static_assert|static assertion)}} expression is not an integral constant expression}}
     LIBCPP_STATIC_ASSERT(test<const double>()); // expected-error-re {{{{(static_assert|static assertion)}} expression is not an integral constant expression}}
-    return 0;
 }
diff --git a/libcxx/test/std/utilities/memory/default.allocator/allocator.members/allocate.verify.cpp b/libcxx/test/std/utilities/memory/default.allocator/allocator.members/allocate.verify.cpp
index c19710a1596ea..b2b6f51e98245 100644
--- a/libcxx/test/std/utilities/memory/default.allocator/allocator.members/allocate.verify.cpp
+++ b/libcxx/test/std/utilities/memory/default.allocator/allocator.members/allocate.verify.cpp
@@ -14,14 +14,8 @@
 // T* allocate(size_t n);
 
 #include <memory>
-#include <cassert>
 
-#include "test_macros.h"
-
-int main(int, char**)
-{
+void f() {
     std::allocator<int> a;
     a.allocate(3); // expected-warning {{ignoring return value of function declared with 'nodiscard' attribute}}
-
-    return 0;
 }
diff --git a/libcxx/test/std/utilities/memory/default.allocator/allocator_types.deprecated_in_cxx17.verify.cpp b/libcxx/test/std/utilities/memory/default.allocator/allocator_types.deprecated_in_cxx17.verify.cpp
index 901f04730ee8f..917492929ecc2 100644
--- a/libcxx/test/std/utilities/memory/default.allocator/allocator_types.deprecated_in_cxx17.verify.cpp
+++ b/libcxx/test/std/utilities/memory/default.allocator/allocator_types.deprecated_in_cxx17.verify.cpp
@@ -27,7 +27,7 @@
 
 #include <memory>
 
-int main(int, char**) {
+void f() {
     {
         typedef std::allocator<char>::pointer Pointer;                  // expected-warning {{'pointer' is deprecated}}
         typedef std::allocator<char>::const_pointer ConstPointer;       // expected-warning {{'const_pointer' is deprecated}}
@@ -48,5 +48,4 @@ int main(int, char**) {
         // reference and const_reference are not provided by std::allocator<void>
         typedef std::allocator<void>::rebind<int>::other Rebind;        // expected-warning {{'rebind<int>' is deprecated}}
     }
-    return 0;
 }
diff --git a/libcxx/test/std/utilities/memory/default.allocator/allocator_types.removed_in_cxx20.verify.cpp b/libcxx/test/std/utilities/memory/default.allocator/allocator_types.removed_in_cxx20.verify.cpp
index 1d91a022867da..2fb7a60c9a88f 100644
--- a/libcxx/test/std/utilities/memory/default.allocator/allocator_types.removed_in_cxx20.verify.cpp
+++ b/libcxx/test/std/utilities/memory/default.allocator/allocator_types.removed_in_cxx20.verify.cpp
@@ -26,7 +26,6 @@
 // UNSUPPORTED: c++03, c++11, c++14, c++17
 
 #include <memory>
-#include "test_macros.h"
 
 template <typename T>
 void check()
@@ -38,10 +37,8 @@ void check()
     typedef typename std::allocator<T>::template rebind<int>::other ARO; // expected-error 3 {{no member named 'rebind'}}
 }
 
-int main(int, char**)
-{
+void f() {
     check<char>();
     check<char const>();
     check<void>();
-    return 0;
 }
diff --git a/libcxx/test/std/utilities/memory/util.smartptr/util.smartptr.shared/util.smartptr.shared.create/make_shared_for_overwrite.pass.cpp b/libcxx/test/std/utilities/memory/util.smartptr/util.smartptr.shared/util.smartptr.shared.create/make_shared_for_overwrite.pass.cpp
index 690a334557161..0027b0aa7c289 100644
--- a/libcxx/test/std/utilities/memory/util.smartptr/util.smartptr.shared/util.smartptr.shared.create/make_shared_for_overwrite.pass.cpp
+++ b/libcxx/test/std/utilities/memory/util.smartptr/util.smartptr.shared/util.smartptr.shared.create/make_shared_for_overwrite.pass.cpp
@@ -17,9 +17,9 @@
 
 #include <cassert>
 #include <concepts>
+#include <cstdlib>
 #include <cstring>
 #include <memory>
-#include <stdlib.h>
 #include <utility>
 
 #include "test_macros.h"
@@ -59,7 +59,7 @@ static_assert(!HasMakeSharedForOverwrite<Foo[], size_t, int>);
 constexpr char pattern = 0xDE;
 
 void* operator new(std::size_t count) {
-  void* ptr = malloc(count);
+  void* ptr = std::malloc(count);
   for (std::size_t i = 0; i < count; ++i) {
     *(reinterpret_cast<char*>(ptr) + i) = pattern;
   }
@@ -68,7 +68,7 @@ void* operator new(std::size_t count) {
 
 void* operator new[](std::size_t count) { return ::operator new(count); }
 
-void operator delete(void* ptr) noexcept { free(ptr); }
+void operator delete(void* ptr) noexcept { std::free(ptr); }
 
 void operator delete[](void* ptr) noexcept { ::operator delete(ptr); }
 
diff --git a/libcxx/test/std/utilities/meta/meta.trans/meta.trans.other/common_reference.compile.pass.cpp b/libcxx/test/std/utilities/meta/meta.trans/meta.trans.other/common_reference.compile.pass.cpp
index 643d5301baae5..9332865eaa245 100644
--- a/libcxx/test/std/utilities/meta/meta.trans/meta.trans.other/common_reference.compile.pass.cpp
+++ b/libcxx/test/std/utilities/meta/meta.trans/meta.trans.other/common_reference.compile.pass.cpp
@@ -217,5 +217,3 @@ static_assert(!has_type<std::common_reference<std::pair<int, const X2>, std::pai
 static_assert(!has_type<std::common_reference<std::pair<int, X2>, std::pair<float, Z2>>>);
 static_assert(!has_type<std::common_reference<std::pair<int, X2>, int, X2>>);
 #endif
-
-int main(int, char**) { return 0; }
diff --git a/libcxx/test/std/utilities/meta/meta.trans/objc_support.pass.mm b/libcxx/test/std/utilities/meta/meta.trans/objc_support.compile.pass.mm
similarity index 100%
rename from libcxx/test/std/utilities/meta/meta.trans/objc_support.pass.mm
rename to libcxx/test/std/utilities/meta/meta.trans/objc_support.compile.pass.mm
diff --git a/libcxx/test/std/utilities/smartptr/unique.ptr/unique.ptr.class/unique.ptr.observers/dereference.verify.cpp b/libcxx/test/std/utilities/smartptr/unique.ptr/unique.ptr.class/unique.ptr.observers/dereference.verify.cpp
index d125eb02f33f4..46506e5622622 100644
--- a/libcxx/test/std/utilities/smartptr/unique.ptr/unique.ptr.class/unique.ptr.observers/dereference.verify.cpp
+++ b/libcxx/test/std/utilities/smartptr/unique.ptr/unique.ptr.class/unique.ptr.observers/dereference.verify.cpp
@@ -13,13 +13,12 @@
 // test op*()
 
 #include <memory>
-#include <cassert>
 
-int main(int, char**) {
+#include "test_macros.h"
+
+void f() {
   std::unique_ptr<int[]> p(new int(3));
   const std::unique_ptr<int[]>& cp = p;
   TEST_IGNORE_NODISCARD(*p);  // expected-error-re {{indirection requires pointer operand ('std::unique_ptr<int{{[ ]*}}[]>' invalid)}}
   TEST_IGNORE_NODISCARD(*cp); // expected-error-re {{indirection requires pointer operand ('const std::unique_ptr<int{{[ ]*}}[]>' invalid)}}
-
-  return 0;
 }
diff --git a/libcxx/test/std/utilities/smartptr/unique.ptr/unique.ptr.class/unique.ptr.observers/op_arrow.verify.cpp b/libcxx/test/std/utilities/smartptr/unique.ptr/unique.ptr.class/unique.ptr.observers/op_arrow.verify.cpp
index 3d1a5e066a7a7..5361c1f055152 100644
--- a/libcxx/test/std/utilities/smartptr/unique.ptr/unique.ptr.class/unique.ptr.observers/op_arrow.verify.cpp
+++ b/libcxx/test/std/utilities/smartptr/unique.ptr/unique.ptr.class/unique.ptr.observers/op_arrow.verify.cpp
@@ -13,13 +13,12 @@
 // test op->()
 
 #include <memory>
-#include <cassert>
 
 struct V {
   int member;
 };
 
-int main(int, char**) {
+void f() {
   std::unique_ptr<V[]> p;
   std::unique_ptr<V[]> const& cp = p;
 
@@ -28,6 +27,4 @@ int main(int, char**) {
 
   cp->member; // expected-error-re {{member reference type 'const std::unique_ptr<V{{[ ]*}}[]>' is not a pointer}}
               // expected-error@-1 {{no member named 'member'}}
-
-  return 0;
 }
diff --git a/libcxx/test/std/utilities/smartptr/unique.ptr/unique.ptr.create/make_unique_for_overwrite.default_init.pass.cpp b/libcxx/test/std/utilities/smartptr/unique.ptr/unique.ptr.create/make_unique_for_overwrite.default_init.pass.cpp
index 5f4fbd4232fa8..8011a37be08ec 100644
--- a/libcxx/test/std/utilities/smartptr/unique.ptr/unique.ptr.create/make_unique_for_overwrite.default_init.pass.cpp
+++ b/libcxx/test/std/utilities/smartptr/unique.ptr/unique.ptr.create/make_unique_for_overwrite.default_init.pass.cpp
@@ -20,13 +20,13 @@
 #include <cassert>
 #include <concepts>
 #include <cstddef>
+#include <cstdlib>
 #include <memory>
-#include <stdlib.h>
 
 constexpr char pattern = 0xDE;
 
 void* operator new(std::size_t count) {
-  void* ptr = malloc(count);
+  void* ptr = std::malloc(count);
   for (std::size_t i = 0; i < count; ++i) {
     *(reinterpret_cast<char*>(ptr) + i) = pattern;
   }
@@ -35,7 +35,7 @@ void* operator new(std::size_t count) {
 
 void* operator new[](std::size_t count) { return ::operator new(count); }
 
-void operator delete(void* ptr) noexcept { free(ptr); }
+void operator delete(void* ptr) noexcept { std::free(ptr); }
 
 void operator delete[](void* ptr) noexcept { ::operator delete(ptr); }
 
diff --git a/libcxx/test/std/utilities/template.bitset/bitset.cons/string_ctor.pass.cpp b/libcxx/test/std/utilities/template.bitset/bitset.cons/string_ctor.pass.cpp
index 327765b71a06c..522b9678d52db 100644
--- a/libcxx/test/std/utilities/template.bitset/bitset.cons/string_ctor.pass.cpp
+++ b/libcxx/test/std/utilities/template.bitset/bitset.cons/string_ctor.pass.cpp
@@ -8,11 +8,12 @@
 
 // bitset(string, pos, n, zero, one); // constexpr since C++23
 
-#include <bitset>
 #include <algorithm> // for 'min' and 'max'
+#include <bitset>
 #include <cassert>
 #include <stdexcept> // for 'invalid_argument'
 #include <string>
+#include <type_traits>
 
 #include "test_macros.h"
 
diff --git a/libcxx/test/std/utilities/tuple/tuple.tuple/tuple.apply/make_from_tuple.pass.cpp b/libcxx/test/std/utilities/tuple/tuple.tuple/tuple.apply/make_from_tuple.pass.cpp
index 957bb67f4f003..5722044360140 100644
--- a/libcxx/test/std/utilities/tuple/tuple.tuple/tuple.apply/make_from_tuple.pass.cpp
+++ b/libcxx/test/std/utilities/tuple/tuple.tuple/tuple.apply/make_from_tuple.pass.cpp
@@ -61,12 +61,6 @@ constexpr bool do_constexpr_test(Tuple&& tup) {
     return std::make_from_tuple<Tp>(std::forward<Tuple>(tup)).args == tup;
 }
 
-// Needed by do_forwarding_test() since it compares pairs of different types.
-template <class T1, class T2, class U1, class U2>
-inline bool operator==(const std::pair<T1, T2>& lhs, const std::pair<U1, U2>& rhs) {
-    return lhs.first == rhs.first && lhs.second == rhs.second;
-}
-
 template <class ...ExpectTypes, class Tuple>
 bool do_forwarding_test(Tuple&& tup) {
     using RawTuple = std::decay_t<Tuple>;
diff --git a/libcxx/test/std/utilities/utility/pairs/pairs.spec/comparison.pass.cpp b/libcxx/test/std/utilities/utility/pairs/pairs.spec/comparison.pass.cpp
index 76c1509fe71ce..76f9771f2b99b 100644
--- a/libcxx/test/std/utilities/utility/pairs/pairs.spec/comparison.pass.cpp
+++ b/libcxx/test/std/utilities/utility/pairs/pairs.spec/comparison.pass.cpp
@@ -10,12 +10,12 @@
 
 // template <class T1, class T2> struct pair
 
-// template <class T1, class T2> bool operator==(const pair<T1,T2>&, const pair<T1,T2>&);
-// template <class T1, class T2> bool operator!=(const pair<T1,T2>&, const pair<T1,T2>&);
-// template <class T1, class T2> bool operator< (const pair<T1,T2>&, const pair<T1,T2>&);
-// template <class T1, class T2> bool operator> (const pair<T1,T2>&, const pair<T1,T2>&);
-// template <class T1, class T2> bool operator>=(const pair<T1,T2>&, const pair<T1,T2>&);
-// template <class T1, class T2> bool operator<=(const pair<T1,T2>&, const pair<T1,T2>&);
+// template <class T1, class T2, class U1, class U2> bool operator==(const pair<T1,T2>&, const pair<U1,U2>&);
+// template <class T1, class T2, class U1, class U2> bool operator!=(const pair<T1,T2>&, const pair<U1,U2>&);
+// template <class T1, class T2, class U1, class U2> bool operator< (const pair<T1,T2>&, const pair<U1,U2>&);
+// template <class T1, class T2, class U1, class U2> bool operator> (const pair<T1,T2>&, const pair<U1,U2>&);
+// template <class T1, class T2, class U1, class U2> bool operator>=(const pair<T1,T2>&, const pair<U1,U2>&);
+// template <class T1, class T2, class U1, class U2> bool operator<=(const pair<T1,T2>&, const pair<U1,U2>&);
 
 #include <utility>
 #include <cassert>
@@ -24,6 +24,18 @@
 
 int main(int, char**)
 {
+    {
+        typedef std::pair<int, short> P1;
+        typedef std::pair<long, long> P2;
+        P1 p1(3, static_cast<short>(4));
+        P2 p2(3, 4);
+        assert( (p1 == p2));
+        assert(!(p1 != p2));
+        assert(!(p1 <  p2));
+        assert( (p1 <= p2));
+        assert(!(p1 >  p2));
+        assert( (p1 >= p2));
+    }
     {
         typedef std::pair<int, short> P;
         P p1(3, static_cast<short>(4));
diff --git a/libcxx/test/std/utilities/utility/pairs/pairs.spec/three_way_comparison.pass.cpp b/libcxx/test/std/utilities/utility/pairs/pairs.spec/three_way_comparison.pass.cpp
index bd93d3f042784..5aa6efb72e9c4 100644
--- a/libcxx/test/std/utilities/utility/pairs/pairs.spec/three_way_comparison.pass.cpp
+++ b/libcxx/test/std/utilities/utility/pairs/pairs.spec/three_way_comparison.pass.cpp
@@ -10,7 +10,9 @@
 
 // template <class T1, class T2> struct pair
 
-// template <class T1, class T2> bool operator<=>(const pair<T1,T2>&, const pair<T1,T2>&);
+// template <class T1, class T2, class U1, class U2>
+//   constexpr common_comparison_category_t<synth-three-way-result<T1, U1>,synth-three-way-result<T2, U2>>
+//     operator<=>(const pair<T1,T2>&, const pair<U1,U2>&);
 
 // UNSUPPORTED: c++03, c++11, c++14, c++17
 
@@ -19,14 +21,37 @@
 #include <limits>
 #include <type_traits> // std::is_constant_evaluated
 #include <utility>
+#include <string>
 
 #include "test_macros.h"
 
-template<class T> concept HasEqual = requires(T t) { t == t; };
-template<class T> concept HasLess = requires(T t) { t < t; };
-template<class T> concept HasSpaceship = requires(T t) { t <=> t; };
+template <class T>
+concept HasEqual = requires(T t) { t == t; };
+template <class T>
+concept HasLess = requires(T t) { t < t; };
+template <class T, class U = T>
+concept HasSpaceship = requires(T t, U u) { t <=> u; };
 
 constexpr bool test() {
+  {
+    // Pairs of different types should compare with strong ordering.
+    using P1 = std::pair<int, int>;
+    using P2 = std::pair<long long, long long>;
+    ASSERT_SAME_TYPE(decltype(P1() <=> P2()), std::strong_ordering);
+    assert((P1(1, 1) <=> P2(1, 2)) == std::strong_ordering::less);
+    assert((P1(2, 1) <=> P2(1, 2)) == std::strong_ordering::greater);
+    assert((P1(0, 0) <=> P2(0, 0)) == std::strong_ordering::equal);
+  }
+  {
+    // Pairs of different types should compare with partial ordering.
+    using P1 = std::pair<int, int>;
+    using P2 = std::pair<double, double>;
+    ASSERT_SAME_TYPE(decltype(P1() <=> P2()), std::partial_ordering);
+    assert((P1(1, 1) <=> P2(1.0, 2.0)) == std::partial_ordering::less);
+    assert((P1(2, 1) <=> P2(1.0, 2.0)) == std::partial_ordering::greater);
+    assert((P1(0, 0) <=> P2(0.0, 0.0)) == std::partial_ordering::equivalent);
+  }
+  { static_assert(!HasSpaceship<std::pair<int, int>, std::pair<std::string, int>>); }
   {
     // Pairs of types that both have strong ordering should compare with strong ordering.
     using P = std::pair<int, int>;
@@ -50,7 +75,7 @@ constexpr bool test() {
   }
   {
     // Pairs of int (strongly ordered) and double (partially ordered) should compare with partial ordering.
-    using P = std::pair<int, double>;
+    using P              = std::pair<int, double>;
     constexpr double nan = std::numeric_limits<double>::quiet_NaN();
     ASSERT_SAME_TYPE(decltype(P() <=> P()), std::partial_ordering);
     assert((P(1, 1.0) <=> P(1, 2.0)) == std::partial_ordering::less);
@@ -62,7 +87,7 @@ constexpr bool test() {
     assert((P(1, nan) <=> P(1, nan)) == std::partial_ordering::unordered);
   }
   {
-    using P = std::pair<double, int>;
+    using P              = std::pair<double, int>;
     constexpr double nan = std::numeric_limits<double>::quiet_NaN();
     ASSERT_SAME_TYPE(decltype(P() <=> P()), std::partial_ordering);
     assert((P(2.0, 1) <=> P(1.0, 2)) == std::partial_ordering::greater);
@@ -93,12 +118,12 @@ constexpr bool test() {
 #endif
   {
     {
-      using P = std::pair<int, double>;
+      using P              = std::pair<int, double>;
       constexpr double nan = std::numeric_limits<double>::quiet_NaN();
       assert((P(1, 2.0) <=> P(1, nan)) == std::partial_ordering::unordered);
     }
     {
-      using P = std::pair<double, int>;
+      using P              = std::pair<double, int>;
       constexpr double nan = std::numeric_limits<double>::quiet_NaN();
       assert((P(1.0, 1) <=> P(nan, 2)) == std::partial_ordering::unordered);
     }
diff --git a/libcxx/test/std/utilities/utility/utility.intcmp/intcmp.in_range/in_range.pass.cpp b/libcxx/test/std/utilities/utility/utility.intcmp/intcmp.in_range/in_range.pass.cpp
index 1efa3d965a7f3..847cee2532581 100644
--- a/libcxx/test/std/utilities/utility/utility.intcmp/intcmp.in_range/in_range.pass.cpp
+++ b/libcxx/test/std/utilities/utility/utility.intcmp/intcmp.in_range/in_range.pass.cpp
@@ -47,14 +47,14 @@ constexpr void test_in_range1() {
 }
 
 constexpr void test_in_range() {
-  constexpr Tuple<uint8_t> utup8;
-  constexpr Tuple<int8_t> stup8;
-  assert(!std::in_range<int8_t>(utup8.max));
+  constexpr Tuple<std::uint8_t> utup8;
+  constexpr Tuple<std::int8_t> stup8;
+  assert(!std::in_range<std::int8_t>(utup8.max));
   assert(std::in_range<short>(utup8.max));
-  assert(!std::in_range<uint8_t>(stup8.min));
-  assert(std::in_range<int8_t>(utup8.mid));
-  assert(!std::in_range<uint8_t>(stup8.mid));
-  assert(!std::in_range<uint8_t>(-1));
+  assert(!std::in_range<std::uint8_t>(stup8.min));
+  assert(std::in_range<std::int8_t>(utup8.mid));
+  assert(!std::in_range<std::uint8_t>(stup8.mid));
+  assert(!std::in_range<std::uint8_t>(-1));
 }
 
 template <class... Ts>
diff --git a/libcxx/test/std/utilities/utility/utility.underlying/to_underlying.verify.cpp b/libcxx/test/std/utilities/utility/utility.underlying/to_underlying.verify.cpp
index b1b2881d888f0..9236d6e284b5d 100644
--- a/libcxx/test/std/utilities/utility/utility.underlying/to_underlying.verify.cpp
+++ b/libcxx/test/std/utilities/utility/utility.underlying/to_underlying.verify.cpp
@@ -16,9 +16,7 @@
 
 struct S {};
 
-int main(int, char**) {
+void f() {
   std::to_underlying(125); // expected-error {{no matching function for call}}
   std::to_underlying(S{}); // expected-error {{no matching function for call}}
-
-  return 0;
 }
diff --git a/libcxx/test/support/almost_satisfies_types.h b/libcxx/test/support/almost_satisfies_types.h
index 13f28c1bfd383..4c9583b0b582e 100644
--- a/libcxx/test/support/almost_satisfies_types.h
+++ b/libcxx/test/support/almost_satisfies_types.h
@@ -58,7 +58,7 @@ using InputRangeNotIndirectlyReadable = UncheckedRange<InputIteratorNotIndirectl
 static_assert(std::input_or_output_iterator<InputIteratorNotIndirectlyReadable>);
 static_assert(!std::indirectly_readable<InputIteratorNotIndirectlyReadable>);
 static_assert(!std::input_iterator<InputIteratorNotIndirectlyReadable>);
-static_assert(!std::ranges::input_range<InputIteratorNotIndirectlyReadable>);
+static_assert(!std::ranges::input_range<InputRangeNotIndirectlyReadable>);
 
 class InputIteratorNotInputOrOutputIterator {
 public:
@@ -287,7 +287,7 @@ using OutputRangeNotInputOrOutputIterator = UncheckedRange<InputIteratorNotInput
 static_assert(!std::input_or_output_iterator<OutputIteratorNotInputOrOutputIterator>);
 static_assert(std::indirectly_writable<OutputIteratorNotInputOrOutputIterator, int>);
 static_assert(!std::output_iterator<OutputIteratorNotInputOrOutputIterator, int>);
-static_assert(!std::ranges::input_range<OutputRangeNotInputOrOutputIterator>);
+static_assert(!std::ranges::output_range<OutputRangeNotInputOrOutputIterator, int>);
 
 class OutputIteratorNotIndirectlyWritable {
 public:
diff --git a/libcxx/test/support/assert_macros.h b/libcxx/test/support/assert_macros.h
index eb143896ba72a..bf382e19d5e04 100644
--- a/libcxx/test/support/assert_macros.h
+++ b/libcxx/test/support/assert_macros.h
@@ -29,14 +29,23 @@
 #include <cstdio>
 #include <cstdlib>
 
+// This function prints the given arguments to standard error.
+//
+// Keeping this as a separate function is important since it provides a single point for
+// downstreams to customize how errors are printed on exotic targets, if needed.
+template <class ...Args>
+void test_eprintf(char const* fmt, Args const& ...args) {
+  std::fprintf(stderr, fmt, args...);
+}
+
 void test_log(const char* condition, const char* file, int line, const char* message) {
   const char* msg = condition ? "Assertion failure: " : "Unconditional failure:";
-  std::fprintf(stderr, "%s%s %s %d\n%s", msg, condition, file, line, message);
+  test_eprintf("%s%s %s %d\n%s", msg, condition, file, line, message);
 }
 
 template <class F>
 void test_log(const char* condition, const char* file, int line, const F& functor) {
-  std::fprintf(stderr, "Assertion failure: %s %s %d\n", condition, file, line);
+  test_eprintf("Assertion failure: %s %s %d\n", condition, file, line);
   functor();
 }
 
diff --git a/libcxx/test/support/atomic_helpers.h b/libcxx/test/support/atomic_helpers.h
index dc38b7ba7768c..8add02d946b3f 100644
--- a/libcxx/test/support/atomic_helpers.h
+++ b/libcxx/test/support/atomic_helpers.h
@@ -10,6 +10,7 @@
 #define ATOMIC_HELPERS_H
 
 #include <cassert>
+#include <cstdint>
 
 #include "test_macros.h"
 
@@ -83,14 +84,14 @@ struct TestEachIntegralType {
 #endif
     TestFunctor<char16_t>()();
     TestFunctor<char32_t>()();
-    TestFunctor<int8_t>()();
-    TestFunctor<uint8_t>()();
-    TestFunctor<int16_t>()();
-    TestFunctor<uint16_t>()();
-    TestFunctor<int32_t>()();
-    TestFunctor<uint32_t>()();
-    TestFunctor<int64_t>()();
-    TestFunctor<uint64_t>()();
+    TestFunctor<std::int8_t>()();
+    TestFunctor<std::uint8_t>()();
+    TestFunctor<std::int16_t>()();
+    TestFunctor<std::uint16_t>()();
+    TestFunctor<std::int32_t>()();
+    TestFunctor<std::uint32_t>()();
+    TestFunctor<std::int64_t>()();
+    TestFunctor<std::uint64_t>()();
   }
 };
 
diff --git a/libcxx/test/support/charconv_test_helpers.h b/libcxx/test/support/charconv_test_helpers.h
index 1ba95e79a66c8..1e8010a3b97a7 100644
--- a/libcxx/test/support/charconv_test_helpers.h
+++ b/libcxx/test/support/charconv_test_helpers.h
@@ -16,6 +16,7 @@
 #include <numeric>
 #include <string.h>
 #include <stdlib.h>
+#include <type_traits>
 
 #include "test_macros.h"
 
diff --git a/libcxx/test/support/concat_macros.h b/libcxx/test/support/concat_macros.h
index 4dadf95a18eef..d6396a76dde73 100644
--- a/libcxx/test/support/concat_macros.h
+++ b/libcxx/test/support/concat_macros.h
@@ -12,6 +12,7 @@
 #include <cstdio>
 #include <string>
 
+#include "assert_macros.h"
 #include "test_macros.h"
 
 #ifndef TEST_HAS_NO_LOCALIZATION
@@ -46,7 +47,7 @@ std::string test_concat_message([[maybe_unused]] Args&&... args) {
 }
 
 // Writes its arguments to stderr, using the test_concat_message helper.
-#  define TEST_WRITE_CONCATENATED(...) [&] { ::std::fprintf(stderr, "%s", ::test_concat_message(__VA_ARGS__).c_str()); }
+#  define TEST_WRITE_CONCATENATED(...) [&] { ::test_eprintf("%s", ::test_concat_message(__VA_ARGS__).c_str()); }
 
 #endif // TEST_STD_VER > 17
 
diff --git a/libcxx/test/support/filesystem_test_helper.h b/libcxx/test/support/filesystem_test_helper.h
index 98429aaf003d8..766a973c23720 100644
--- a/libcxx/test/support/filesystem_test_helper.h
+++ b/libcxx/test/support/filesystem_test_helper.h
@@ -19,6 +19,7 @@
 #include <cstdio> // for printf
 #include <string>
 #include <system_error>
+#include <type_traits>
 #include <vector>
 
 #include "assert_macros.h"
@@ -101,7 +102,7 @@ namespace utils {
     // N.B. libc might define some of the foo[64] identifiers using macros from
     // foo64 -> foo or vice versa.
 #if defined(_WIN32)
-    using off64_t = int64_t;
+    using off64_t = std::int64_t;
 #elif defined(__MVS__) || defined(__LP64__)
     using off64_t = ::off_t;
 #else
diff --git a/libcxx/test/support/format.functions.common.h b/libcxx/test/support/format.functions.common.h
index 81ed04998a510..e70ae34200573 100644
--- a/libcxx/test/support/format.functions.common.h
+++ b/libcxx/test/support/format.functions.common.h
@@ -44,7 +44,7 @@ template <class T>
 using context_t = typename context<T>::type;
 
 // A user-defined type used to test the handle formatter.
-enum class status : uint16_t { foo = 0xAAAA, bar = 0x5555, foobar = 0xAA55 };
+enum class status : std::uint16_t { foo = 0xAAAA, bar = 0x5555, foobar = 0xAA55 };
 
 // The formatter for a user-defined type used to test the handle formatter.
 template <class CharT>
@@ -90,7 +90,7 @@ struct std::formatter<status, CharT> {
       begin = buffer;
       buffer[0] = '0';
       buffer[1] = 'x';
-      end = std::to_chars(&buffer[2], std::end(buffer), static_cast<uint16_t>(s), 16).ptr;
+      end = std::to_chars(&buffer[2], std::end(buffer), static_cast<std::uint16_t>(s), 16).ptr;
       buffer[6] = '\0';
       break;
 
@@ -98,7 +98,7 @@ struct std::formatter<status, CharT> {
       begin = buffer;
       buffer[0] = '0';
       buffer[1] = 'X';
-      end = std::to_chars(&buffer[2], std::end(buffer), static_cast<uint16_t>(s), 16).ptr;
+      end = std::to_chars(&buffer[2], std::end(buffer), static_cast<std::uint16_t>(s), 16).ptr;
       std::transform(static_cast<const char*>(&buffer[2]), end, &buffer[2], [](char c) {
         return static_cast<char>(std::toupper(c)); });
       buffer[6] = '\0';
diff --git a/libcxx/test/support/nasty_string.h b/libcxx/test/support/nasty_string.h
new file mode 100644
index 0000000000000..901700abef37c
--- /dev/null
+++ b/libcxx/test/support/nasty_string.h
@@ -0,0 +1,184 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef TEST_SUPPORT_NASTY_STRING_H
+#define TEST_SUPPORT_NASTY_STRING_H
+
+#include <algorithm>
+#include <string>
+#include <type_traits>
+
+#include "make_string.h"
+#include "test_macros.h"
+
+// This defines a nasty_string similar to nasty_containers. This string's
+// value_type does operator hijacking, which allows us to ensure that the
+// library uses the provided `CharTraits` instead of using operations on
+// the value_type directly.
+
+
+// When using the code during constant evaluation it relies on
+//   P2647R1 Permitting static constexpr variables in constexpr functions
+// This is a C++23 feature, which is not supported by all compilers yet.
+// * GCC >= 13
+// * Clang >= 16
+// * MSVC no support yet
+//
+// TODO After there is proper compiler support use TEST_STD_VER >= 23 instead
+// of this macro in the tests.
+#if TEST_STD_VER < 23 || __cpp_constexpr < 202211L
+#  define TEST_HAS_NO_NASTY_STRING
+#endif
+
+#ifndef TEST_HAS_NO_NASTY_STRING
+// Make sure the char-like operations in strings do not depend on the char-like type.
+struct nasty_char {
+  template <typename T>
+  friend auto operator<=>(T, T) = delete;
+
+  template <typename T>
+  friend void operator+(T&&) = delete;
+
+  template <typename T>
+  friend void operator-(T&&) = delete;
+
+  template <typename T>
+  friend void operator&(T&&) = delete;
+
+  char c;
+};
+
+static_assert(std::is_trivial<nasty_char>::value, "");
+static_assert(std::is_standard_layout<nasty_char>::value, "");
+
+// These traits are based on the constexpr_traits test class.
+struct nasty_char_traits {
+  typedef nasty_char char_type;
+  typedef int int_type;
+  typedef std::streamoff off_type;
+  typedef std::streampos pos_type;
+  typedef std::mbstate_t state_type;
+  // The comparison_category is omitted so the class will have weak_ordering
+  // in C++20. This is intentional.
+
+  static constexpr void assign(char_type& c1, const char_type& c2) noexcept { c1 = c2; }
+
+  static constexpr bool eq(char_type c1, char_type c2) noexcept { return c1.c == c2.c; }
+
+  static constexpr bool lt(char_type c1, char_type c2) noexcept { return c1.c < c2.c; }
+
+  static constexpr int compare(const char_type* s1, const char_type* s2, size_t n);
+  static constexpr size_t length(const char_type* s);
+  static constexpr const char_type* find(const char_type* s, size_t n, const char_type& a);
+  static constexpr char_type* move(char_type* s1, const char_type* s2, size_t n);
+  static constexpr char_type* copy(char_type* s1, const char_type* s2, size_t n);
+  static constexpr char_type* assign(char_type* s, size_t n, char_type a);
+
+  static constexpr int_type not_eof(int_type c) noexcept { return eq_int_type(c, eof()) ? ~eof() : c; }
+
+  static constexpr char_type to_char_type(int_type c) noexcept { return char_type(c); }
+
+  static constexpr int_type to_int_type(char_type c) noexcept { return int_type(c.c); }
+
+  static constexpr bool eq_int_type(int_type c1, int_type c2) noexcept { return c1 == c2; }
+
+  static constexpr int_type eof() noexcept { return int_type(EOF); }
+};
+
+constexpr int nasty_char_traits::compare(const nasty_char* s1, const nasty_char* s2, size_t n) {
+  for (; n; --n, ++s1, ++s2) {
+    if (lt(*s1, *s2))
+      return -1;
+    if (lt(*s2, *s1))
+      return 1;
+  }
+  return 0;
+}
+
+constexpr size_t nasty_char_traits::length(const nasty_char* s) {
+  size_t len = 0;
+  for (; !eq(*s, nasty_char(0)); ++s)
+    ++len;
+  return len;
+}
+
+constexpr const nasty_char* nasty_char_traits::find(const nasty_char* s, size_t n, const nasty_char& a) {
+  for (; n; --n) {
+    if (eq(*s, a))
+      return s;
+    ++s;
+  }
+  return 0;
+}
+
+constexpr nasty_char* nasty_char_traits::move(nasty_char* s1, const nasty_char* s2, size_t n) {
+  nasty_char* r = s1;
+  if (s1 < s2) {
+    for (; n; --n, ++s1, ++s2)
+      assign(*s1, *s2);
+  } else if (s2 < s1) {
+    s1 += n;
+    s2 += n;
+    for (; n; --n)
+      assign(*--s1, *--s2);
+  }
+  return r;
+}
+
+constexpr nasty_char* nasty_char_traits::copy(nasty_char* s1, const nasty_char* s2, size_t n) {
+  if (!std::is_constant_evaluated()) // fails in constexpr because we might be comparing unrelated pointers
+    assert(s2 < s1 || s2 >= s1 + n);
+  nasty_char* r = s1;
+  for (; n; --n, ++s1, ++s2)
+    assign(*s1, *s2);
+  return r;
+}
+
+constexpr nasty_char* nasty_char_traits::assign(nasty_char* s, size_t n, nasty_char a) {
+  nasty_char* r = s;
+  for (; n; --n, ++s)
+    assign(*s, a);
+  return r;
+}
+
+using nasty_string = std::basic_string<nasty_char, nasty_char_traits>;
+
+template <size_t N>
+struct ToNastyChar {
+  constexpr ToNastyChar(const char (&r)[N]) {
+    std::transform(r, r + N, std::addressof(text[0]), [](char c) { return nasty_char{c}; });
+  }
+  nasty_char text[N];
+};
+
+template <size_t N>
+ToNastyChar(const char (&)[N]) -> ToNastyChar<N>;
+
+template <ToNastyChar t>
+constexpr auto to_nasty_char() {
+  return t;
+}
+
+// A macro like MAKE_CSTRING
+//
+// The difference is this macro can convert the nasty_char too.
+//
+// The lambda is a template, so the 'if constexpr' false branch is not evaluated for the nasty_char.
+#  define CONVERT_TO_CSTRING(CHAR, STR)                                                                                \
+    []<class CharT> {                                                                                                  \
+      if constexpr (std::is_same_v<CharT, nasty_char>) {                                                               \
+        static constexpr auto result = to_nasty_char<STR>();                                                           \
+        return result.text;                                                                                            \
+      } else                                                                                                           \
+        return MAKE_CSTRING(CharT, STR);                                                                               \
+    }.template operator()<CHAR>() /* */
+#else                             // TEST_HAS_NO_NASTY_STRING
+#  define CONVERT_TO_CSTRING(CharT, STR) MAKE_CSTRING(CharT, STR)
+#endif                            // TEST_HAS_NO_NASTY_STRING
+
+#endif                            // TEST_SUPPORT_NASTY_STRING_H
diff --git a/libcxx/test/support/test.support/test_macros_header.no_exceptions.verify.cpp b/libcxx/test/support/test.support/test_macros_header.no_exceptions.verify.cpp
index cc156e2515110..efdf1c02dc67f 100644
--- a/libcxx/test/support/test.support/test_macros_header.no_exceptions.verify.cpp
+++ b/libcxx/test/support/test.support/test_macros_header.no_exceptions.verify.cpp
@@ -17,7 +17,6 @@
 #  error "TEST_HAS_NO_EXCEPTIONS should be defined"
 #endif
 
-int main(int, char**) {
+void f() {
     try { (void)0; } catch (...) { } // expected-error {{exceptions disabled}}
-    return 0;
 }
diff --git a/libcxx/test/support/test.support/test_macros_header.no_rtti.verify.cpp b/libcxx/test/support/test.support/test_macros_header.no_rtti.verify.cpp
index a7f35577c6beb..8790c73416e08 100644
--- a/libcxx/test/support/test.support/test_macros_header.no_rtti.verify.cpp
+++ b/libcxx/test/support/test.support/test_macros_header.no_rtti.verify.cpp
@@ -20,8 +20,7 @@
 struct A { virtual ~A() { } };
 struct B : A { };
 
-int main(int, char**) {
+void f() {
     A* ptr = new B;
     (void)dynamic_cast<B*>(ptr); // expected-error{{use of dynamic_cast requires -frtti}}
-    return 0;
 }
diff --git a/libcxx/test/support/test_container_comparisons.h b/libcxx/test/support/test_container_comparisons.h
index d4659bd1aa5d1..8687d9ae80d53 100644
--- a/libcxx/test/support/test_container_comparisons.h
+++ b/libcxx/test/support/test_container_comparisons.h
@@ -81,4 +81,128 @@ constexpr bool test_ordered_container_spaceship() {
   return true;
 }
 
+// Implementation detail of `test_ordered_map_container_spaceship`
+template <template <typename...> typename Container, typename Key, typename Val, typename Order>
+constexpr void test_ordered_map_container_spaceship_with_type() {
+  // Empty containers
+  {
+    Container<Key, Val> l1;
+    Container<Key, Val> l2;
+    assert(testOrder(l1, l2, Order::equivalent));
+  }
+  // Identical contents
+  {
+    Container<Key, Val> l1{{1, 1}, {2, 1}};
+    Container<Key, Val> l2{{1, 1}, {2, 1}};
+    assert(testOrder(l1, l2, Order::equivalent));
+  }
+  // Less, due to contained values
+  {
+    Container<Key, Val> l1{{1, 1}, {2, 1}};
+    Container<Key, Val> l2{{1, 1}, {2, 2}};
+    assert(testOrder(l1, l2, Order::less));
+  }
+  // Greater, due to contained values
+  {
+    Container<Key, Val> l1{{1, 1}, {2, 3}};
+    Container<Key, Val> l2{{1, 1}, {2, 2}};
+    assert(testOrder(l1, l2, Order::greater));
+  }
+  // Shorter list
+  {
+    Container<Key, Val> l1{{1, 1}};
+    Container<Key, Val> l2{{1, 1}, {2, 2}};
+    assert(testOrder(l1, l2, Order::less));
+  }
+  // Longer list
+  {
+    Container<Key, Val> l1{{1, 2}, {2, 2}};
+    Container<Key, Val> l2{{1, 1}};
+    assert(testOrder(l1, l2, Order::greater));
+  }
+  // Unordered
+  if constexpr (std::is_same_v<Val, PartialOrder>) {
+    Container<Key, Val> l1{{1, 1}, {2, std::numeric_limits<int>::min()}};
+    Container<Key, Val> l2{{1, 1}, {2, 2}};
+    assert(testOrder(l1, l2, Order::unordered));
+  }
+
+  // Identical contents
+  {
+    Container<Key, Val> l1{{1, 1}, {2, 1}, {2, 2}};
+    Container<Key, Val> l2{{1, 1}, {2, 1}, {2, 2}};
+    assert(testOrder(l1, l2, Order::equivalent));
+    Container<Key, Val> l3{{1, 1}, {2, 1}, {2, 2}};
+    Container<Key, Val> l4{{2, 1}, {2, 2}, {1, 1}};
+    assert(testOrder(l3, l4, Order::equivalent));
+  }
+  // Less, due to contained values
+  {
+    Container<Key, Val> l1{{1, 1}, {2, 1}, {2, 1}};
+    Container<Key, Val> l2{{1, 1}, {2, 2}, {2, 2}};
+    assert(testOrder(l1, l2, Order::less));
+    Container<Key, Val> l3{{1, 1}, {2, 1}, {2, 1}};
+    Container<Key, Val> l4{{2, 2}, {2, 2}, {1, 1}};
+    assert(testOrder(l3, l4, Order::less));
+  }
+  // Greater, due to contained values
+  {
+    Container<Key, Val> l1{{1, 1}, {2, 3}, {2, 3}};
+    Container<Key, Val> l2{{1, 1}, {2, 2}, {2, 2}};
+    assert(testOrder(l1, l2, Order::greater));
+    Container<Key, Val> l3{{1, 1}, {2, 3}, {2, 3}};
+    Container<Key, Val> l4{{2, 2}, {2, 2}, {1, 1}};
+    assert(testOrder(l3, l4, Order::greater));
+  }
+  // Shorter list
+  {
+    Container<Key, Val> l1{{1, 1}, {2, 2}};
+    Container<Key, Val> l2{{1, 1}, {2, 2}, {2, 2}, {3, 1}};
+    assert(testOrder(l1, l2, Order::less));
+    Container<Key, Val> l3{{1, 1}, {2, 2}};
+    Container<Key, Val> l4{{3, 1}, {2, 2}, {2, 2}, {1, 1}};
+    assert(testOrder(l3, l4, Order::less));
+  }
+  // Longer list
+  {
+    Container<Key, Val> l1{{1, 2}, {2, 2}, {2, 2}, {3, 1}};
+    Container<Key, Val> l2{{1, 1}, {2, 2}};
+    assert(testOrder(l1, l2, Order::greater));
+    Container<Key, Val> l3{{1, 2}, {2, 2}, {2, 2}, {3, 1}};
+    Container<Key, Val> l4{{2, 2}, {1, 1}};
+    assert(testOrder(l3, l4, Order::greater));
+  }
+  // Unordered
+  if constexpr (std::is_same_v<Val, PartialOrder>) {
+    Container<Key, Val> l1{{1, 1}, {2, std::numeric_limits<int>::min()}, {2, 3}};
+    Container<Key, Val> l2{{1, 1}, {2, 2}, {2, 3}};
+    assert(testOrder(l1, l2, Order::unordered));
+    Container<Key, Val> l3{{1, 1}, {2, std::numeric_limits<int>::min()}, {2, 3}};
+    Container<Key, Val> l4{{2, 3}, {2, 2}, {1, 1}};
+    assert(testOrder(l3, l4, Order::unordered));
+  }
+}
+
+// Tests the `operator<=>` on ordered containers
+template <template <typename...> typename Container>
+constexpr bool test_ordered_map_container_spaceship() {
+  // The container should fulfil `std::three_way_comparable`
+  static_assert(std::three_way_comparable<Container<int, int>>);
+
+  // Test different comparison categories
+  test_ordered_map_container_spaceship_with_type<Container, int, int, std::strong_ordering>();
+  test_ordered_map_container_spaceship_with_type<Container, int, StrongOrder, std::strong_ordering>();
+  test_ordered_map_container_spaceship_with_type<Container, int, WeakOrder, std::weak_ordering>();
+  test_ordered_map_container_spaceship_with_type<Container, int, PartialOrder, std::partial_ordering>();
+
+  // `LessAndEqComp` does not have `operator<=>`. ordering is sythesized based on `operator<`
+  test_ordered_map_container_spaceship_with_type<Container, int, LessAndEqComp, std::weak_ordering>();
+
+  // Thanks to SFINAE, the following is not a compiler error but returns `false`
+  struct NonComparable {};
+  static_assert(!std::three_way_comparable<Container<int, NonComparable>>);
+
+  return true;
+}
+
 #endif
diff --git a/libcxx/test/tools/clang_tidy_checks/CMakeLists.txt b/libcxx/test/tools/clang_tidy_checks/CMakeLists.txt
index 0e437d0a24308..bd1611fb0a9cc 100644
--- a/libcxx/test/tools/clang_tidy_checks/CMakeLists.txt
+++ b/libcxx/test/tools/clang_tidy_checks/CMakeLists.txt
@@ -17,6 +17,7 @@ endif()
 set(SOURCES
     abi_tag_on_virtual.cpp
     hide_from_abi.cpp
+    proper_version_checks.cpp
     qualify_declval.cpp
     robust_against_adl.cpp
     uglify_attributes.cpp
diff --git a/libcxx/test/tools/clang_tidy_checks/libcpp_module.cpp b/libcxx/test/tools/clang_tidy_checks/libcpp_module.cpp
index 685d34a86b54a..9ecb41e20a2da 100644
--- a/libcxx/test/tools/clang_tidy_checks/libcpp_module.cpp
+++ b/libcxx/test/tools/clang_tidy_checks/libcpp_module.cpp
@@ -11,8 +11,9 @@
 
 #include "abi_tag_on_virtual.hpp"
 #include "hide_from_abi.hpp"
-#include "robust_against_adl.hpp"
+#include "proper_version_checks.hpp"
 #include "qualify_declval.hpp"
+#include "robust_against_adl.hpp"
 #include "uglify_attributes.hpp"
 
 namespace {
@@ -21,6 +22,7 @@ class LibcxxTestModule : public clang::tidy::ClangTidyModule {
   void addCheckFactories(clang::tidy::ClangTidyCheckFactories& check_factories) override {
     check_factories.registerCheck<libcpp::abi_tag_on_virtual>("libcpp-avoid-abi-tag-on-virtual");
     check_factories.registerCheck<libcpp::hide_from_abi>("libcpp-hide-from-abi");
+    check_factories.registerCheck<libcpp::proper_version_checks>("libcpp-cpp-version-check");
     check_factories.registerCheck<libcpp::robust_against_adl_check>("libcpp-robust-against-adl");
     check_factories.registerCheck<libcpp::uglify_attributes>("libcpp-uglify-attributes");
     check_factories.registerCheck<libcpp::qualify_declval>("libcpp-qualify-declval");
diff --git a/libcxx/test/tools/clang_tidy_checks/proper_version_checks.cpp b/libcxx/test/tools/clang_tidy_checks/proper_version_checks.cpp
new file mode 100644
index 0000000000000..2abc5ceeecfb8
--- /dev/null
+++ b/libcxx/test/tools/clang_tidy_checks/proper_version_checks.cpp
@@ -0,0 +1,74 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "proper_version_checks.hpp"
+
+#include <clang/Lex/Lexer.h>
+#include <clang/Lex/PPCallbacks.h>
+#include <clang/Lex/Preprocessor.h>
+
+namespace libcpp {
+namespace {
+class proper_version_checks_callbacks : public clang::PPCallbacks {
+public:
+  proper_version_checks_callbacks(clang::Preprocessor& preprocessor, clang::tidy::ClangTidyCheck& check)
+      : preprocessor_(preprocessor), check_(check) {}
+
+  void If(clang::SourceLocation location, clang::SourceRange condition_range, ConditionValueKind) override {
+    check_condition(location, condition_range);
+  }
+
+  void Elif(clang::SourceLocation location,
+            clang::SourceRange condition_range,
+            ConditionValueKind,
+            clang::SourceLocation if_location) override {
+    check_condition(location, condition_range);
+  }
+
+private:
+  void check_condition(clang::SourceLocation location, clang::SourceRange condition_range) {
+    std::string_view condition = clang::Lexer::getSourceText(
+        clang::CharSourceRange::getTokenRange(condition_range),
+        preprocessor_.getSourceManager(),
+        preprocessor_.getLangOpts());
+    if (preprocessor_.getSourceManager().isInMainFile(location))
+      return;
+
+    if (condition.starts_with("_LIBCPP_STD_VER") && condition.find(">") != std::string_view::npos &&
+        condition.find(">=") == std::string_view::npos)
+      check_.diag(location, "_LIBCPP_STD_VER >= version should be used instead of _LIBCPP_STD_VER > prev_version");
+
+    else if (condition.starts_with("__cplusplus"))
+      check_.diag(location, "Use _LIBCPP_STD_VER instead of __cplusplus to constrain based on the C++ version");
+
+    else if (condition == "_LIBCPP_STD_VER >= 11")
+      check_.diag(location, "_LIBCPP_STD_VER >= 11 is always true. Did you mean '#ifndef _LIBCPP_CXX03_LANG'?");
+
+    else if (condition.starts_with("_LIBCPP_STD_VER >= ") &&
+             std::ranges::none_of(std::array{"14", "17", "20", "23"}, [&](auto val) {
+               return condition.find(val) != std::string_view::npos;
+             }))
+      check_.diag(location, "Not a valid value for _LIBCPP_STD_VER. Use 14, 17, 20 or 23");
+  }
+
+  clang::Preprocessor& preprocessor_;
+  clang::tidy::ClangTidyCheck& check_;
+};
+} // namespace
+
+proper_version_checks::proper_version_checks(llvm::StringRef name, clang::tidy::ClangTidyContext* context)
+    : clang::tidy::ClangTidyCheck(name, context) {}
+
+void proper_version_checks::registerPPCallbacks(
+    const clang::SourceManager& source_manager,
+    clang::Preprocessor* preprocessor,
+    clang::Preprocessor* module_expander) {
+  preprocessor->addPPCallbacks(std::make_unique<proper_version_checks_callbacks>(*preprocessor, *this));
+}
+
+} // namespace libcpp
diff --git a/libcxx/test/tools/clang_tidy_checks/proper_version_checks.hpp b/libcxx/test/tools/clang_tidy_checks/proper_version_checks.hpp
new file mode 100644
index 0000000000000..5c33b9551bf27
--- /dev/null
+++ b/libcxx/test/tools/clang_tidy_checks/proper_version_checks.hpp
@@ -0,0 +1,19 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang-tidy/ClangTidyCheck.h"
+
+namespace libcpp {
+class proper_version_checks : public clang::tidy::ClangTidyCheck {
+public:
+  proper_version_checks(llvm::StringRef, clang::tidy::ClangTidyContext*);
+  void registerPPCallbacks(const clang::SourceManager& source_manager,
+                           clang::Preprocessor* preprocessor,
+                           clang::Preprocessor* module_expander) override;
+};
+} // namespace libcpp
diff --git a/libcxx/utils/ci/Dockerfile b/libcxx/utils/ci/Dockerfile
index 083ab5828bcad..aa719d81a0ee5 100644
--- a/libcxx/utils/ci/Dockerfile
+++ b/libcxx/utils/ci/Dockerfile
@@ -76,14 +76,14 @@ RUN apt-get update && apt-get install -y clang-tidy-$(($LLVM_HEAD_VERSION - 2))
 RUN apt-get update && apt-get install -y llvm-$(($LLVM_HEAD_VERSION - 1))-dev     llvm-$(($LLVM_HEAD_VERSION - 2))-dev \
                                      libclang-$(($LLVM_HEAD_VERSION - 1))-dev libclang-$(($LLVM_HEAD_VERSION - 2))-dev
 
-# TODO LLVM16 Don't install llvm-16-dev explicitly
-RUN apt-get update && apt-get install -y llvm-16-dev libclang-16-dev
-
 # Install the most recent GCC, like clang install the previous version as a transition.
 ENV GCC_LATEST_VERSION=12
 RUN apt-get update && apt install -y gcc-$((GCC_LATEST_VERSION - 1)) g++-$((GCC_LATEST_VERSION - 1))
 RUN apt-get update && apt install -y gcc-$GCC_LATEST_VERSION g++-$GCC_LATEST_VERSION
 
+# Remove all, no longer needed, apt data
+RUN rm -rf /var/lib/apt/lists/*
+
 # Install a recent CMake
 RUN wget https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1-linux-x86_64.sh -O /tmp/install-cmake.sh
 RUN bash /tmp/install-cmake.sh --prefix=/usr --exclude-subdir --skip-license
diff --git a/libcxx/utils/ci/apple-install-libcxx.sh b/libcxx/utils/ci/apple-install-libcxx.sh
index dd349153ad6f7..c1d810256225a 100755
--- a/libcxx/utils/ci/apple-install-libcxx.sh
+++ b/libcxx/utils/ci/apple-install-libcxx.sh
@@ -119,7 +119,8 @@ for arch in ${architectures}; do
                 -DCMAKE_INSTALL_PREFIX="${build_dir}/${arch}-install" \
                 -DCMAKE_INSTALL_NAME_DIR="/usr/lib" \
                 -DCMAKE_OSX_ARCHITECTURES="${arch}" \
-                -DLIBCXXABI_LIBRARY_VERSION="${version}"
+                -DLIBCXXABI_LIBRARY_VERSION="${version}" \
+                -DLIBCXX_LIBRARY_VERSION="${version}"
 
     if [ "$headers_only" = true ]; then
         xcrun cmake --build "${build_dir}/${arch}" --target install-cxx-headers install-cxxabi-headers -- -v
@@ -150,6 +151,9 @@ if [ "$headers_only" != true ]; then
     universal_dylib libc++.1.dylib
     universal_dylib libc++abi.dylib
     (cd "${install_dir}/usr/lib" && ln -s "libc++.1.dylib" libc++.dylib)
+
+    experimental_libs=$(for arch in ${architectures}; do echo "${build_dir}/${arch}-install/lib/libc++experimental.a"; done)
+    xcrun lipo -create ${experimental_libs} -output "${install_dir}/usr/lib/libc++experimental.a"
 fi
 
 # Install the headers by copying the headers from one of the built architectures
diff --git a/libcxx/utils/ci/buildkite-pipeline.yml b/libcxx/utils/ci/buildkite-pipeline.yml
index 694d141ebebc2..474138d21f4f0 100644
--- a/libcxx/utils/ci/buildkite-pipeline.yml
+++ b/libcxx/utils/ci/buildkite-pipeline.yml
@@ -1025,3 +1025,22 @@ steps:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
+
+  - group: "FreeBSD"
+    steps:
+    - label: "FreeBSD 13 amd64"
+      command: "libcxx/utils/ci/run-buildbot generic-cxx2b"
+      artifact_paths:
+        - "**/test-results.xml"
+        - "**/*.abilist"
+      env:
+          CC: "clang15"
+          CXX: "clang++15"
+      agents:
+        queue: "libcxx-builders"
+        os: "freebsd"
+      retry:
+        automatic:
+          - exit_status: -1  # Agent was lost
+            limit: 2
+      timeout_in_minutes: 120
diff --git a/libcxx/utils/data/ignore_format.txt b/libcxx/utils/data/ignore_format.txt
index a4defede15520..a6838ccb2b022 100644
--- a/libcxx/utils/data/ignore_format.txt
+++ b/libcxx/utils/data/ignore_format.txt
@@ -212,7 +212,6 @@ libcxx/include/__algorithm/unwrap_range.h
 libcxx/include/__algorithm/upper_bound.h
 libcxx/include/any
 libcxx/include/array
-libcxx/include/atomic
 libcxx/include/__atomic/atomic_base.h
 libcxx/include/__atomic/atomic_flag.h
 libcxx/include/__atomic/atomic.h
@@ -240,12 +239,6 @@ libcxx/include/bitset
 libcxx/include/__bsd_locale_defaults.h
 libcxx/include/__bsd_locale_fallbacks.h
 libcxx/include/cctype
-libcxx/include/charconv
-libcxx/include/__charconv/chars_format.h
-libcxx/include/__charconv/from_chars_result.h
-libcxx/include/__charconv/tables.h
-libcxx/include/__charconv/to_chars_base_10.h
-libcxx/include/__charconv/to_chars_result.h
 libcxx/include/chrono
 libcxx/include/__chrono/calendar.h
 libcxx/include/__chrono/convert_to_timespec.h
@@ -318,7 +311,6 @@ libcxx/include/__debug_utils/randomize_range.h
 libcxx/include/deque
 libcxx/include/__errc
 libcxx/include/errno.h
-libcxx/include/exception
 libcxx/include/execution
 libcxx/include/expected
 libcxx/include/__expected/expected.h
diff --git a/libcxx/utils/generate_extended_grapheme_cluster_test.py b/libcxx/utils/generate_extended_grapheme_cluster_test.py
index 5061c80f8ed4c..94f669235a73b 100755
--- a/libcxx/utils/generate_extended_grapheme_cluster_test.py
+++ b/libcxx/utils/generate_extended_grapheme_cluster_test.py
@@ -99,7 +99,7 @@ def parseBreakTestLine(input: TextIO) -> Optional[BreakTestItem]:
 //===----------------------------------------------------------------------===//
 
 // WARNING, this entire header is generated by
-// utiles/generate_extended_grapheme_cluster_test.py
+// utils/generate_extended_grapheme_cluster_test.py
 // DO NOT MODIFY!
 
 // UNICODE, INC. LICENSE AGREEMENT - DATA FILES AND SOFTWARE
diff --git a/libcxx/utils/libcxx/test/format.py b/libcxx/utils/libcxx/test/format.py
index b491890dfec54..4d3466d6d3632 100644
--- a/libcxx/utils/libcxx/test/format.py
+++ b/libcxx/utils/libcxx/test/format.py
@@ -115,9 +115,11 @@ class CxxStandardLibraryTest(lit.formats.TestFormat):
     FOO.pass.mm             - Same as .pass.cpp, but for Objective-C++
 
     FOO.compile.pass.cpp    - Compiles successfully, link and run not attempted
+    FOO.compile.pass.mm     - Same as .compile.pass.cpp, but for Objective-C++
     FOO.compile.fail.cpp    - Does not compile successfully
 
     FOO.link.pass.cpp       - Compiles and links successfully, run not attempted
+    FOO.link.pass.mm        - Same as .link.pass.cpp, but for Objective-C++
     FOO.link.fail.cpp       - Compiles successfully, but fails to link
 
     FOO.sh.<anything>       - A builtin Lit Shell test
@@ -187,8 +189,10 @@ class CxxStandardLibraryTest(lit.formats.TestFormat):
     """
     def getTestsInDirectory(self, testSuite, pathInSuite, litConfig, localConfig):
         SUPPORTED_SUFFIXES = ['[.]pass[.]cpp$', '[.]pass[.]mm$',
-                              '[.]compile[.]pass[.]cpp$', '[.]compile[.]fail[.]cpp$',
-                              '[.]link[.]pass[.]cpp$', '[.]link[.]fail[.]cpp$',
+                              '[.]compile[.]pass[.]cpp$', '[.]compile[.]pass[.]mm$',
+                              '[.]compile[.]fail[.]cpp$',
+                              '[.]link[.]pass[.]cpp$', '[.]link[.]pass[.]mm$',
+                              '[.]link[.]fail[.]cpp$',
                               '[.]sh[.][^.]+$',
                               '[.]verify[.]cpp$',
                               '[.]fail[.]cpp$']
@@ -211,7 +215,7 @@ def execute(self, test, litConfig):
         if re.search('[.]sh[.][^.]+$', filename):
             steps = [ ] # The steps are already in the script
             return self._executeShTest(test, litConfig, steps)
-        elif filename.endswith('.compile.pass.cpp'):
+        elif filename.endswith('.compile.pass.cpp') or filename.endswith('.compile.pass.mm'):
             steps = [
                 "%dbg(COMPILED WITH) %{cxx} %s %{flags} %{compile_flags} -fsyntax-only"
             ]
@@ -221,7 +225,7 @@ def execute(self, test, litConfig):
                 "%dbg(COMPILED WITH) ! %{cxx} %s %{flags} %{compile_flags} -fsyntax-only"
             ]
             return self._executeShTest(test, litConfig, steps)
-        elif filename.endswith('.link.pass.cpp'):
+        elif filename.endswith('.link.pass.cpp') or filename.endswith('.link.pass.mm'):
             steps = [
                 "%dbg(COMPILED WITH) %{cxx} %s %{flags} %{compile_flags} %{link_flags} -o %t.exe"
             ]
diff --git a/libcxx/utils/libcxx/test/params.py b/libcxx/utils/libcxx/test/params.py
index 0574a00a737bd..fab60df0a3a72 100644
--- a/libcxx/utils/libcxx/test/params.py
+++ b/libcxx/utils/libcxx/test/params.py
@@ -17,6 +17,7 @@
   '-Wextra',
   '-Wshadow',
   '-Wundef',
+  '-Wunused-template',
   '-Wno-unused-command-line-argument',
   '-Wno-attributes',
   '-Wno-pessimizing-move',
@@ -124,8 +125,8 @@ def getStdFlag(cfg, std):
                  The Standard libraries currently supported are:
                  - llvm-libc++: The 'upstream' libc++ as shipped with LLVM.
                  - apple-libc++: libc++ as shipped by Apple. This is basically like the LLVM one, but
-                                 there are a few differences like installation paths and the use of
-                                 universal dylibs.
+                                 there are a few differences like installation paths, the use of
+                                 universal dylibs and the existence of availability markup.
                  - libstdc++: The GNU C++ library typically shipped with GCC.
                  - msvc: The Microsoft implementation of the C++ Standard Library.
                 """,
@@ -235,11 +236,9 @@ def getStdFlag(cfg, std):
     for a given deployment target. For example, this is the case when testing
     availability markup, where we want to make sure that using the annotated
     facility on a deployment target that doesn't support it will fail at compile
-    time, not at runtime. This can be achieved by creating a `.compile.pass.cpp`
-    and XFAILing it for the right deployment target. If the test doesn't fail at
-    compile-time like it's supposed to, the test will XPASS. Another option is to
-    create a `.verify.cpp` test that checks for the right errors, and mark that
-    test as requiring `use_system_cxx_lib && <target>`.
+    time, not at runtime. This can be achieved by creating a `.verify.cpp` test
+    that checks for the right errors, and mark that test as requiring
+    `use_system_cxx_lib && target=<target>`.
     """,
     actions=lambda useSystem: [
       AddFeature('use_system_cxx_lib')
diff --git a/libcxxabi/CMakeLists.txt b/libcxxabi/CMakeLists.txt
index 4dd9f092dbf00..9239274d7af1d 100644
--- a/libcxxabi/CMakeLists.txt
+++ b/libcxxabi/CMakeLists.txt
@@ -44,7 +44,7 @@ option(LIBCXXABI_ENABLE_EXCEPTIONS
   "Provide support for exceptions in the runtime.
   When disabled, libc++abi does not support stack unwinding and other exceptions-related features." ON)
 option(LIBCXXABI_ENABLE_ASSERTIONS "Enable assertions independent of build mode." ON)
-option(LIBCXXABI_ENABLE_PEDANTIC "Compile with pedantic enabled." ON)
+option(LIBCXXABI_ENABLE_PEDANTIC "Compile with pedantic enabled." OFF)
 option(LIBCXXABI_ENABLE_WERROR "Fail and stop if a warning is triggered." OFF)
 option(LIBCXXABI_USE_LLVM_UNWINDER "Build and use the LLVM unwinder." OFF)
 option(LIBCXXABI_ENABLE_STATIC_UNWINDER "Statically link the LLVM unwinder." OFF)
@@ -279,41 +279,6 @@ add_definitions(-D_LIBCXXABI_BUILDING_LIBRARY)
 # it is being built as part of libcxx.
 add_definitions(-D_LIBCPP_BUILDING_LIBRARY)
 
-add_compile_flags_if_supported(-Werror=return-type)
-
-# Get warning flags
-add_compile_flags_if_supported(-W)
-add_compile_flags_if_supported(-Wall)
-add_compile_flags_if_supported(-Wchar-subscripts)
-add_compile_flags_if_supported(-Wconversion)
-add_compile_flags_if_supported(-Wmismatched-tags)
-add_compile_flags_if_supported(-Wmissing-braces)
-add_compile_flags_if_supported(-Wnewline-eof)
-add_compile_flags_if_supported(-Wunused-function)
-add_compile_flags_if_supported(-Wshadow)
-add_compile_flags_if_supported(-Wshorten-64-to-32)
-add_compile_flags_if_supported(-Wsign-compare)
-add_compile_flags_if_supported(-Wsign-conversion)
-add_compile_flags_if_supported(-Wstrict-aliasing=2)
-add_compile_flags_if_supported(-Wstrict-overflow=4)
-add_compile_flags_if_supported(-Wunused-parameter)
-add_compile_flags_if_supported(-Wunused-variable)
-add_compile_flags_if_supported(-Wwrite-strings)
-add_compile_flags_if_supported(-Wundef)
-
-add_compile_flags_if_supported(-Wno-suggest-override)
-
-if (LIBCXXABI_ENABLE_WERROR)
-  add_compile_flags_if_supported(-Werror)
-  add_compile_flags_if_supported(-WX)
-else()
-  add_compile_flags_if_supported(-Wno-error)
-  add_compile_flags_if_supported(-WX-)
-endif()
-if (LIBCXXABI_ENABLE_PEDANTIC)
-  add_compile_flags_if_supported(-pedantic)
-endif()
-
 # Get feature flags.
 add_compile_flags_if_supported(-fstrict-aliasing)
 
diff --git a/libcxxabi/src/CMakeLists.txt b/libcxxabi/src/CMakeLists.txt
index 58df59a5725a3..4c27a25fc4755 100644
--- a/libcxxabi/src/CMakeLists.txt
+++ b/libcxxabi/src/CMakeLists.txt
@@ -158,8 +158,11 @@ if (NOT TARGET pstl::ParallelSTL)
   message(STATUS "Could not find ParallelSTL, libc++abi will not attempt to use it but the build may fail if the libc++ in use needs it to be available.")
 endif()
 
+include(WarningFlags)
+
 # Build the shared library.
 add_library(cxxabi_shared_objects OBJECT EXCLUDE_FROM_ALL ${LIBCXXABI_SOURCES} ${LIBCXXABI_HEADERS})
+cxx_add_warning_flags(cxxabi_shared_objects ${LIBCXXABI_ENABLE_WERROR} ${LIBCXXABI_ENABLE_PEDANTIC})
 if (LIBCXXABI_USE_LLVM_UNWINDER)
   if (LIBCXXABI_STATICALLY_LINK_UNWINDER_IN_SHARED_LIBRARY)
     target_link_libraries(cxxabi_shared_objects PUBLIC unwind_shared_objects) # propagate usage requirements
@@ -246,6 +249,7 @@ endif()
 
 # Build the static library.
 add_library(cxxabi_static_objects OBJECT EXCLUDE_FROM_ALL ${LIBCXXABI_SOURCES} ${LIBCXXABI_HEADERS})
+cxx_add_warning_flags(cxxabi_static_objects ${LIBCXXABI_ENABLE_WERROR} ${LIBCXXABI_ENABLE_PEDANTIC})
 if (LIBCXXABI_USE_LLVM_UNWINDER AND LIBCXXABI_STATICALLY_LINK_UNWINDER_IN_STATIC_LIBRARY)
   target_link_libraries(cxxabi_static_objects PUBLIC unwind_static_objects) # propagate usage requirements
   target_sources(cxxabi_static_objects PUBLIC $<TARGET_OBJECTS:unwind_static_objects>)
diff --git a/libcxxabi/src/demangle/ItaniumDemangle.h b/libcxxabi/src/demangle/ItaniumDemangle.h
index 66213c63f2f85..fd66e9570e7a9 100644
--- a/libcxxabi/src/demangle/ItaniumDemangle.h
+++ b/libcxxabi/src/demangle/ItaniumDemangle.h
@@ -19,6 +19,7 @@
 #include "DemangleConfig.h"
 #include "StringView.h"
 #include "Utility.h"
+#include <__cxxabi_config.h>
 #include <algorithm>
 #include <cassert>
 #include <cctype>
@@ -27,8 +28,14 @@
 #include <cstring>
 #include <limits>
 #include <new>
+#include <type_traits>
 #include <utility>
 
+#ifdef _LIBCXXABI_COMPILER_CLANG
+#pragma clang diagnostic push
+#pragma clang diagnostic ignored "-Wunused-template"
+#endif
+
 DEMANGLE_NAMESPACE_BEGIN
 
 template <class T, size_t N> class PODSmallVector {
@@ -537,6 +544,8 @@ struct AbiTagAttr : Node {
 
   template<typename Fn> void match(Fn F) const { F(Base, Tag); }
 
+  StringView getBaseName() const override { return Base->getBaseName(); }
+
   void printLeft(OutputBuffer &OB) const override {
     Base->printLeft(OB);
     OB += "[abi:";
@@ -5497,4 +5506,8 @@ struct ManglingParser : AbstractManglingParser<ManglingParser<Alloc>, Alloc> {
 
 DEMANGLE_NAMESPACE_END
 
+#ifdef _LIBCXXABI_COMPILER_CLANG
+#pragma clang diagnostic pop
+#endif
+
 #endif // DEMANGLE_ITANIUMDEMANGLE_H
diff --git a/libcxxabi/src/demangle/StringView.h b/libcxxabi/src/demangle/StringView.h
index 90890e3771102..07b7cf8575753 100644
--- a/libcxxabi/src/demangle/StringView.h
+++ b/libcxxabi/src/demangle/StringView.h
@@ -17,9 +17,16 @@
 #define DEMANGLE_STRINGVIEW_H
 
 #include "DemangleConfig.h"
+
+#include <__cxxabi_config.h>
 #include <cassert>
 #include <cstring>
 
+#ifdef _LIBCXXABI_COMPILER_CLANG
+#pragma clang diagnostic push
+#pragma clang diagnostic ignored "-Wunused-template"
+#endif
+
 DEMANGLE_NAMESPACE_BEGIN
 
 class StringView {
@@ -119,4 +126,8 @@ inline bool operator==(const StringView &LHS, const StringView &RHS) {
 
 DEMANGLE_NAMESPACE_END
 
+#ifdef _LIBCXXABI_COMPILER_CLANG
+#pragma clang diagnostic pop
+#endif
+
 #endif
diff --git a/libcxxabi/src/private_typeinfo.cpp b/libcxxabi/src/private_typeinfo.cpp
index e1086661c019d..83d1f9f130a39 100644
--- a/libcxxabi/src/private_typeinfo.cpp
+++ b/libcxxabi/src/private_typeinfo.cpp
@@ -41,6 +41,7 @@
 // Defining _LIBCXXABI_FORGIVING_DYNAMIC_CAST does not help since can_catch() calls
 // is_equal() with use_strcmp=false so the string names are not compared.
 
+#include <cstdint>
 #include <string.h>
 
 #ifdef _LIBCXXABI_FORGIVING_DYNAMIC_CAST
@@ -656,77 +657,138 @@ __dynamic_cast(const void *static_ptr, const __class_type_info *static_type,
     // Find out if we can use a giant short cut in the search
     if (is_equal(dynamic_type, dst_type, false))
     {
-        // Using giant short cut.  Add that information to info.
-        info.number_of_dst_type = 1;
-        // Do the  search
-        dynamic_type->search_above_dst(&info, dynamic_ptr, dynamic_ptr, public_path, false);
-#ifdef _LIBCXXABI_FORGIVING_DYNAMIC_CAST
-        // The following if should always be false because we should definitely
-        //   find (static_ptr, static_type), either on a public or private path
-        if (info.path_dst_ptr_to_static_ptr == unknown)
+        // We're downcasting from src_type to the complete object's dynamic
+        //   type. This is a really hot path that can be further optimized
+        //   with the `src2dst_offset` hint.
+        // In such a case, dynamic_ptr already gives the casting result if the
+        //   casting ever succeeds. All we have to do now is to check
+        //   static_ptr points to a public base sub-object of dynamic_ptr.
+
+        if (src2dst_offset >= 0)
         {
-            // We get here only if there is some kind of visibility problem
-            //   in client code.
-            static_assert(std::atomic<size_t>::is_always_lock_free, "");
-            static std::atomic<size_t> error_count(0);
-            size_t error_count_snapshot = error_count.fetch_add(1, std::memory_order_relaxed);
-            if ((error_count_snapshot & (error_count_snapshot-1)) == 0)
-                syslog(LOG_ERR, "dynamic_cast error 1: Both of the following type_info's "
-                        "should have public visibility. At least one of them is hidden. %s"
-                        ", %s.\n", static_type->name(), dynamic_type->name());
-            // Redo the search comparing type_info's using strcmp
-            info = {dst_type, static_ptr, static_type, src2dst_offset, 0};
-            info.number_of_dst_type = 1;
-            dynamic_type->search_above_dst(&info, dynamic_ptr, dynamic_ptr, public_path, true);
+            // The static type is a unique public non-virtual base type of
+            //   dst_type at offset `src2dst_offset` from the origin of dst.
+            // Note that there might be other non-public static_type bases. The
+            //   hint only guarantees that the public base is non-virtual and
+            //   unique. So we have to check whether static_ptr points to that
+            //   unique public base sub-object.
+            if (offset_to_derived == -src2dst_offset)
+                dst_ptr = dynamic_ptr;
+        }
+        else if (src2dst_offset == -2)
+        {
+            // static_type is not a public base of dst_type.
+            dst_ptr = nullptr;
         }
+        else
+        {
+            // If src2dst_offset == -3, then:
+            //   src_type is a multiple public base type but never a virtual
+            //   base type. We can't conclude that static_ptr points to those
+            //   public base sub-objects because there might be other non-
+            //   public static_type bases. The search is inevitable.
+
+            // Fallback to the slow path to check that static_type is a public
+            //   base type of dynamic_type.
+            // Using giant short cut.  Add that information to info.
+            info.number_of_dst_type = 1;
+            // Do the  search
+            dynamic_type->search_above_dst(&info, dynamic_ptr, dynamic_ptr, public_path, false);
+#ifdef _LIBCXXABI_FORGIVING_DYNAMIC_CAST
+            // The following if should always be false because we should
+            //   definitely find (static_ptr, static_type), either on a public
+            //   or private path
+            if (info.path_dst_ptr_to_static_ptr == unknown)
+            {
+                // We get here only if there is some kind of visibility problem
+                //   in client code.
+                static_assert(std::atomic<size_t>::is_always_lock_free, "");
+                static std::atomic<size_t> error_count(0);
+                size_t error_count_snapshot = error_count.fetch_add(1, std::memory_order_relaxed);
+                if ((error_count_snapshot & (error_count_snapshot-1)) == 0)
+                    syslog(LOG_ERR, "dynamic_cast error 1: Both of the following type_info's "
+                            "should have public visibility. At least one of them is hidden. %s"
+                            ", %s.\n", static_type->name(), dynamic_type->name());
+                // Redo the search comparing type_info's using strcmp
+                info = {dst_type, static_ptr, static_type, src2dst_offset, 0};
+                info.number_of_dst_type = 1;
+                dynamic_type->search_above_dst(&info, dynamic_ptr, dynamic_ptr, public_path, true);
+            }
 #endif // _LIBCXXABI_FORGIVING_DYNAMIC_CAST
-        // Query the search.
-        if (info.path_dst_ptr_to_static_ptr == public_path)
-            dst_ptr = dynamic_ptr;
+            // Query the search.
+            if (info.path_dst_ptr_to_static_ptr == public_path)
+                dst_ptr = dynamic_ptr;
+        }
     }
     else
     {
-        // Not using giant short cut.  Do the search
-        dynamic_type->search_below_dst(&info, dynamic_ptr, public_path, false);
- #ifdef _LIBCXXABI_FORGIVING_DYNAMIC_CAST
-        // The following if should always be false because we should definitely
-        //   find (static_ptr, static_type), either on a public or private path
-        if (info.path_dst_ptr_to_static_ptr == unknown &&
-            info.path_dynamic_ptr_to_static_ptr == unknown)
+        if (src2dst_offset >= 0)
         {
-            static_assert(std::atomic<size_t>::is_always_lock_free, "");
-            static std::atomic<size_t> error_count(0);
-            size_t error_count_snapshot = error_count.fetch_add(1, std::memory_order_relaxed);
-            if ((error_count_snapshot & (error_count_snapshot-1)) == 0)
-                syslog(LOG_ERR, "dynamic_cast error 2: One or more of the following type_info's "
-                                "has hidden visibility or is defined in more than one translation "
-                                "unit. They should all have public visibility. "
-                                "%s, %s, %s.\n", static_type->name(), dynamic_type->name(),
-                        dst_type->name());
-            // Redo the search comparing type_info's using strcmp
-            info = {dst_type, static_ptr, static_type, src2dst_offset, 0};
-            dynamic_type->search_below_dst(&info, dynamic_ptr, public_path, true);
+            // Optimize toward downcasting: dst_type has one unique public
+            //   static_type bases. Let's first try to do a downcast before
+            //   falling back to the slow path. The downcast succeeds if there
+            //   is at least one path regardless of visibility from
+            //   dynamic_type to dst_type.
+            const void* dst_ptr_to_static = reinterpret_cast<const char*>(static_ptr) - src2dst_offset;
+            if (reinterpret_cast<std::intptr_t>(dst_ptr_to_static) >= reinterpret_cast<std::intptr_t>(dynamic_ptr))
+            {
+                // Try to search a path from dynamic_type to dst_type.
+                __dynamic_cast_info dynamic_to_dst_info = {dynamic_type, dst_ptr_to_static, dst_type, src2dst_offset};
+                dynamic_to_dst_info.number_of_dst_type = 1;
+                dynamic_type->search_above_dst(&dynamic_to_dst_info, dynamic_ptr, dynamic_ptr, public_path, false);
+                if (dynamic_to_dst_info.path_dst_ptr_to_static_ptr != unknown) {
+                    // We have found at least one path from dynamic_ptr to
+                    //   dst_ptr. The downcast can succeed.
+                    dst_ptr = dst_ptr_to_static;
+                }
+            }
         }
-#endif // _LIBCXXABI_FORGIVING_DYNAMIC_CAST
-        // Query the search.
-        switch (info.number_to_static_ptr)
+
+        if (!dst_ptr)
         {
-        case 0:
-            if (info.number_to_dst_ptr == 1 &&
-                    info.path_dynamic_ptr_to_static_ptr == public_path &&
-                    info.path_dynamic_ptr_to_dst_ptr == public_path)
-                dst_ptr = info.dst_ptr_not_leading_to_static_ptr;
-            break;
-        case 1:
-            if (info.path_dst_ptr_to_static_ptr == public_path ||
-                   (
-                       info.number_to_dst_ptr == 0 &&
-                       info.path_dynamic_ptr_to_static_ptr == public_path &&
-                       info.path_dynamic_ptr_to_dst_ptr == public_path
-                   )
-               )
-                dst_ptr = info.dst_ptr_leading_to_static_ptr;
-            break;
+            // Not using giant short cut.  Do the search
+            dynamic_type->search_below_dst(&info, dynamic_ptr, public_path, false);
+#ifdef _LIBCXXABI_FORGIVING_DYNAMIC_CAST
+            // The following if should always be false because we should
+            //   definitely find (static_ptr, static_type), either on a public
+            //   or private path
+            if (info.path_dst_ptr_to_static_ptr == unknown &&
+                info.path_dynamic_ptr_to_static_ptr == unknown)
+            {
+                static_assert(std::atomic<size_t>::is_always_lock_free, "");
+                static std::atomic<size_t> error_count(0);
+                size_t error_count_snapshot = error_count.fetch_add(1, std::memory_order_relaxed);
+                if ((error_count_snapshot & (error_count_snapshot-1)) == 0)
+                    syslog(LOG_ERR, "dynamic_cast error 2: One or more of the following type_info's "
+                                    "has hidden visibility or is defined in more than one translation "
+                                    "unit. They should all have public visibility. "
+                                    "%s, %s, %s.\n", static_type->name(), dynamic_type->name(),
+                            dst_type->name());
+                // Redo the search comparing type_info's using strcmp
+                info = {dst_type, static_ptr, static_type, src2dst_offset, 0};
+                dynamic_type->search_below_dst(&info, dynamic_ptr, public_path, true);
+            }
+#endif // _LIBCXXABI_FORGIVING_DYNAMIC_CAST
+            // Query the search.
+            switch (info.number_to_static_ptr)
+            {
+            case 0:
+                if (info.number_to_dst_ptr == 1 &&
+                        info.path_dynamic_ptr_to_static_ptr == public_path &&
+                        info.path_dynamic_ptr_to_dst_ptr == public_path)
+                    dst_ptr = info.dst_ptr_not_leading_to_static_ptr;
+                break;
+            case 1:
+                if (info.path_dst_ptr_to_static_ptr == public_path ||
+                    (
+                        info.number_to_dst_ptr == 0 &&
+                        info.path_dynamic_ptr_to_static_ptr == public_path &&
+                        info.path_dynamic_ptr_to_dst_ptr == public_path
+                    )
+                )
+                    dst_ptr = info.dst_ptr_leading_to_static_ptr;
+                break;
+            }
         }
     }
     return const_cast<void*>(dst_ptr);
diff --git a/libcxxabi/test/configs/apple-libc++abi-backdeployment.cfg.in b/libcxxabi/test/configs/apple-libc++abi-backdeployment.cfg.in
index 4b30fe72b3250..2e9472163b9d5 100644
--- a/libcxxabi/test/configs/apple-libc++abi-backdeployment.cfg.in
+++ b/libcxxabi/test/configs/apple-libc++abi-backdeployment.cfg.in
@@ -55,6 +55,8 @@ config.substitutions.append(('%{exec}',
     '%{executor} --execdir %T --env DYLD_LIBRARY_PATH="%{cxx-runtime-root}:%{abi-runtime-root}:%{unwind-runtime-root}" -- '
 ))
 
+config.stdlib = 'apple-libc++'
+
 libcxx.test.config.configure(
     libcxx.test.params.DEFAULT_PARAMETERS + BACKDEPLOYMENT_PARAMETERS,
     libcxx.test.features.DEFAULT_FEATURES,
diff --git a/libcxxabi/test/configs/apple-libc++abi-shared.cfg.in b/libcxxabi/test/configs/apple-libc++abi-shared.cfg.in
index 8278ffc8242f9..ec0c93b0134a4 100644
--- a/libcxxabi/test/configs/apple-libc++abi-shared.cfg.in
+++ b/libcxxabi/test/configs/apple-libc++abi-shared.cfg.in
@@ -16,6 +16,8 @@ config.substitutions.append(('%{exec}',
     '%{executor} --execdir %T --env DYLD_LIBRARY_PATH=%{lib} -- '
 ))
 
+config.stdlib = 'apple-libc++'
+
 import os, site
 site.addsitedir(os.path.join('@LIBCXXABI_LIBCXX_PATH@', 'utils'))
 import libcxx.test.params, libcxx.test.config
diff --git a/libcxxabi/test/test_demangle.pass.cpp b/libcxxabi/test/test_demangle.pass.cpp
index 0607c4ef7efaa..00592eede9a12 100644
--- a/libcxxabi/test/test_demangle.pass.cpp
+++ b/libcxxabi/test/test_demangle.pass.cpp
@@ -30112,6 +30112,8 @@ const char* cases[][2] =
      "::basic_ostream()"},
     {"_ZNSsC1Ev", "std::basic_string<char, std::char_traits<char>,"
      " std::allocator<char>>::basic_string()"},
+    {"_ZN1SB8ctor_tagC2Ev", "S[abi:ctor_tag]::S()"},
+    {"_ZN1SB8ctor_tagD2Ev", "S[abi:ctor_tag]::~S()"},
 };
 
 const unsigned N = sizeof(cases) / sizeof(cases[0]);
diff --git a/libunwind/CMakeLists.txt b/libunwind/CMakeLists.txt
index f82fa0a70eff2..647b1c41955b5 100644
--- a/libunwind/CMakeLists.txt
+++ b/libunwind/CMakeLists.txt
@@ -169,28 +169,6 @@ if (LIBUNWIND_ENABLE_CET)
   endif()
 endif()
 
-# Get warning flags
-add_compile_flags_if_supported(-W)
-add_compile_flags_if_supported(-Wall)
-add_compile_flags_if_supported(-Wchar-subscripts)
-add_compile_flags_if_supported(-Wconversion)
-add_compile_flags_if_supported(-Wmismatched-tags)
-add_compile_flags_if_supported(-Wmissing-braces)
-add_compile_flags_if_supported(-Wnewline-eof)
-add_compile_flags_if_supported(-Wno-unused-function)
-add_compile_flags_if_supported(-Wshadow)
-add_compile_flags_if_supported(-Wshorten-64-to-32)
-add_compile_flags_if_supported(-Wsign-compare)
-add_compile_flags_if_supported(-Wsign-conversion)
-add_compile_flags_if_supported(-Wstrict-aliasing=2)
-add_compile_flags_if_supported(-Wstrict-overflow=4)
-add_compile_flags_if_supported(-Wunused-parameter)
-add_compile_flags_if_supported(-Wunused-variable)
-add_compile_flags_if_supported(-Wwrite-strings)
-add_compile_flags_if_supported(-Wundef)
-
-add_compile_flags_if_supported(-Wno-suggest-override)
-
 if (WIN32)
   # The headers lack matching dllexport attributes (_LIBUNWIND_EXPORT);
   # silence the warning instead of cluttering the headers (which aren't
@@ -199,18 +177,6 @@ if (WIN32)
   add_compile_flags_if_supported(-Wno-dll-attribute-on-redeclaration)
 endif()
 
-if (LIBUNWIND_ENABLE_WERROR)
-  add_compile_flags_if_supported(-Werror)
-  add_compile_flags_if_supported(-WX)
-else()
-  add_compile_flags_if_supported(-Wno-error)
-  add_compile_flags_if_supported(-WX-)
-endif()
-
-if (LIBUNWIND_ENABLE_PEDANTIC)
-  add_compile_flags_if_supported(-pedantic)
-endif()
-
 # Get feature flags.
 # Exceptions
 # Catches C++ exceptions only and tells the compiler to assume that extern C
diff --git a/libunwind/src/CMakeLists.txt b/libunwind/src/CMakeLists.txt
index df32e53d69e6e..cced6b4cf9e85 100644
--- a/libunwind/src/CMakeLists.txt
+++ b/libunwind/src/CMakeLists.txt
@@ -135,8 +135,11 @@ set_property(SOURCE ${LIBUNWIND_C_SOURCES}
 # ease, but does not rely on C++ at runtime.
 set(CMAKE_CXX_IMPLICIT_LINK_LIBRARIES "")
 
+include(WarningFlags)
+
 # Build the shared library.
 add_library(unwind_shared_objects OBJECT EXCLUDE_FROM_ALL ${LIBUNWIND_SOURCES} ${LIBUNWIND_HEADERS})
+cxx_add_warning_flags(unwind_shared_objects ${LIBUNWIND_ENABLE_WERROR} ${LIBUNWIND_ENABLE_PEDANTIC})
 if(CMAKE_C_COMPILER_ID STREQUAL MSVC)
   target_compile_options(unwind_shared_objects PRIVATE /GR-)
 else()
@@ -174,6 +177,7 @@ endif()
 
 # Build the static library.
 add_library(unwind_static_objects OBJECT EXCLUDE_FROM_ALL ${LIBUNWIND_SOURCES} ${LIBUNWIND_HEADERS})
+cxx_add_warning_flags(unwind_static_objects ${LIBUNWIND_ENABLE_WERROR} ${LIBUNWIND_ENABLE_PEDANTIC})
 if(CMAKE_C_COMPILER_ID STREQUAL MSVC)
   target_compile_options(unwind_static_objects PRIVATE /GR-)
 else()
diff --git a/libunwind/src/DwarfInstructions.hpp b/libunwind/src/DwarfInstructions.hpp
index 27432be56133b..9962c2ffa0ca3 100644
--- a/libunwind/src/DwarfInstructions.hpp
+++ b/libunwind/src/DwarfInstructions.hpp
@@ -224,7 +224,8 @@ int DwarfInstructions<A, R>::stepWithDwarf(A &addressSpace, pint_t pc,
         p &= ~0xfULL;
         // CFA is the bottom of the current stack frame.
         for (; p < cfa; p += 16) {
-          __asm__ __volatile__(".arch_extension memtag\n"
+          __asm__ __volatile__(".arch armv8.5-a\n"
+                               ".arch_extension memtag\n"
                                "stg %[Ptr], [%[Ptr]]\n"
                                :
                                : [Ptr] "r"(p)
diff --git a/libunwind/test/unwind_scalable_vectors.pass.cpp b/libunwind/test/unwind_scalable_vectors.pass.cpp
index 250e2c8fc7b1e..a5c5947c870fd 100644
--- a/libunwind/test/unwind_scalable_vectors.pass.cpp
+++ b/libunwind/test/unwind_scalable_vectors.pass.cpp
@@ -13,30 +13,8 @@
 #include <assert.h>
 #include <libunwind.h>
 
-// Check correct unwinding of frame with VLENB-sized objects (vector registers):
-// 1. Save return address (ra) in temporary register.
-// 2. Load VLENB (vector length in bytes) and substract it from current stack
-//    pointer (sp) - equivalent to one vector register on stack frame.
-// 3. Set DWARF cannonical frame address (CFA) to "sp + vlenb" expresssion so it
-//    can be correctly unwinded.
-// 4. Call stepper() function and check that 2 unwind steps are successful -
-//    from stepper() into foo() and from foo() into main().
-// 5. Restore stack pointer and return address.
-__attribute__((naked)) static void foo() {
-  __asm__(".cfi_startproc\n"
-          "mv s0, ra\n"
-          "csrr  s1, vlenb\n"
-          "sub sp, sp, s1\n"
-          "# .cfi_def_cfa_expression sp + vlenb\n"
-          ".cfi_escape 0x0f, 0x07, 0x72, 0x00, 0x92, 0xa2, 0x38, 0x00, 0x22\n"
-          "call stepper\n"
-          "add sp, sp, s1\n"
-          "mv ra, s0\n"
-          "ret\n"
-          ".cfi_endproc\n");
-}
-
-extern "C" void stepper() {
+#ifdef __riscv_vector
+__attribute__((noinline)) extern "C" void stepper() {
   unw_cursor_t cursor;
   unw_context_t uc;
   unw_getcontext(&uc);
@@ -47,4 +25,16 @@ extern "C" void stepper() {
   assert(unw_step(&cursor) > 0);
 }
 
+// Check correct unwinding of frame with VLENB-sized objects (vector registers).
+__attribute__((noinline)) static void foo() {
+  __rvv_int32m1_t v;
+  asm volatile("" : "=vr"(v)); // Dummy inline asm to def v.
+  stepper();                   // def-use of v has cross the function, so that
+                               // will triger spill/reload to/from the stack.
+  asm volatile("" ::"vr"(v));  // Dummy inline asm to use v.
+}
+
 int main() { foo(); }
+#else
+int main() { return 0; }
+#endif
diff --git a/lld/CMakeLists.txt b/lld/CMakeLists.txt
index 3d6225646fe61..1e9e47393196d 100644
--- a/lld/CMakeLists.txt
+++ b/lld/CMakeLists.txt
@@ -75,11 +75,9 @@ if(LLD_BUILT_STANDALONE)
         set(LLVM_UTILS_PROVIDED ON)
         set(LLD_TEST_DEPS FileCheck not)
       endif()
-      set(UNITTEST_DIR ${LLVM_THIRD_PARTY_DIR}/unittest)
-      if(EXISTS ${UNITTEST_DIR}/googletest/include/gtest/gtest.h
-          AND NOT EXISTS ${LLVM_LIBRARY_DIR}/${CMAKE_STATIC_LIBRARY_PREFIX}gtest${CMAKE_STATIC_LIBRARY_SUFFIX}
-          AND EXISTS ${UNITTEST_DIR}/CMakeLists.txt)
-        add_subdirectory(${UNITTEST_DIR} third-party/unittest)
+
+      if (NOT TARGET llvm_gtest)
+        message(FATAL_ERROR "llvm-gtest not found. Please install llvm-gtest or disable tests with -DLLVM_INCLUDE_TESTS=OFF")
       endif()
     else()
       # Seek installed Lit.
diff --git a/lld/COFF/Driver.cpp b/lld/COFF/Driver.cpp
index e0fc29879abee..29219662e5422 100644
--- a/lld/COFF/Driver.cpp
+++ b/lld/COFF/Driver.cpp
@@ -559,7 +559,7 @@ void LinkerDriver::detectWinSysRoot(const opt::InputArgList &Args) {
                                      WinSysRoot, vcToolChainPath, vsLayout) &&
       (Args.hasArg(OPT_lldignoreenv) ||
        !findVCToolChainViaEnvironment(*VFS, vcToolChainPath, vsLayout)) &&
-      !findVCToolChainViaSetupConfig(*VFS, vcToolChainPath, vsLayout) &&
+      !findVCToolChainViaSetupConfig(*VFS, {}, vcToolChainPath, vsLayout) &&
       !findVCToolChainViaRegistry(vcToolChainPath, vsLayout))
     return;
 
diff --git a/lld/COFF/MinGW.cpp b/lld/COFF/MinGW.cpp
index 1212569285678..2b403635b478e 100644
--- a/lld/COFF/MinGW.cpp
+++ b/lld/COFF/MinGW.cpp
@@ -49,6 +49,9 @@ AutoExporter::AutoExporter(
       "libclang_rt.profile-x86_64",
       "libc++",
       "libc++abi",
+      "libFortran_main",
+      "libFortranRuntime",
+      "libFortranDecimal",
       "libunwind",
       "libmsvcrt",
       "libucrtbase",
diff --git a/lld/ELF/Config.h b/lld/ELF/Config.h
index 61faea4e9bce4..249fe3c30eceb 100644
--- a/lld/ELF/Config.h
+++ b/lld/ELF/Config.h
@@ -73,7 +73,14 @@ enum class UnresolvedPolicy { ReportError, Warn, Ignore };
 enum class OrphanHandlingPolicy { Place, Warn, Error };
 
 // For --sort-section and linkerscript sorting rules.
-enum class SortSectionPolicy { Default, None, Alignment, Name, Priority };
+enum class SortSectionPolicy {
+  Default,
+  None,
+  Alignment,
+  Name,
+  Priority,
+  Reverse,
+};
 
 // For --target2
 enum class Target2Policy { Abs, Rel, GotRel };
diff --git a/lld/ELF/InputFiles.h b/lld/ELF/InputFiles.h
index 421ea6ed1a8ed..2ee15e89347ab 100644
--- a/lld/ELF/InputFiles.h
+++ b/lld/ELF/InputFiles.h
@@ -59,7 +59,6 @@ class InputFile {
   enum Kind : uint8_t {
     ObjKind,
     SharedKind,
-    ArchiveKind,
     BitcodeKind,
     BinaryKind,
   };
diff --git a/lld/ELF/LinkerScript.cpp b/lld/ELF/LinkerScript.cpp
index 668a79b5edd7f..9207751cb7d2f 100644
--- a/lld/ELF/LinkerScript.cpp
+++ b/lld/ELF/LinkerScript.cpp
@@ -457,6 +457,8 @@ static void sortSections(MutableArrayRef<InputSectionBase *> vec,
     return llvm::stable_sort(vec, nameComparator);
   case SortSectionPolicy::Priority:
     return llvm::stable_sort(vec, priorityComparator);
+  case SortSectionPolicy::Reverse:
+    return std::reverse(vec.begin(), vec.end());
   }
 }
 
diff --git a/lld/ELF/ScriptParser.cpp b/lld/ELF/ScriptParser.cpp
index e83779bcebff5..db7f0268cef61 100644
--- a/lld/ELF/ScriptParser.cpp
+++ b/lld/ELF/ScriptParser.cpp
@@ -660,6 +660,7 @@ StringMatcher ScriptParser::readFilePatterns() {
 
 SortSectionPolicy ScriptParser::peekSortKind() {
   return StringSwitch<SortSectionPolicy>(peek())
+      .Case("REVERSE", SortSectionPolicy::Reverse)
       .Cases("SORT", "SORT_BY_NAME", SortSectionPolicy::Name)
       .Case("SORT_BY_ALIGNMENT", SortSectionPolicy::Alignment)
       .Case("SORT_BY_INIT_PRIORITY", SortSectionPolicy::Priority)
diff --git a/lld/ELF/Writer.cpp b/lld/ELF/Writer.cpp
index 53a7c4b1e673b..534794bd835a3 100644
--- a/lld/ELF/Writer.cpp
+++ b/lld/ELF/Writer.cpp
@@ -846,7 +846,8 @@ enum RankFlags {
   RF_PPC_GOT = 1 << 3,
   RF_PPC_BRANCH_LT = 1 << 2,
   RF_MIPS_GPREL = 1 << 1,
-  RF_MIPS_NOT_GOT = 1 << 0
+  RF_MIPS_NOT_GOT = 1 << 0,
+  RF_RISCV_SDATA = 1 << 0,
 };
 
 static unsigned getSectionRank(const OutputSection &osec) {
@@ -972,6 +973,14 @@ static unsigned getSectionRank(const OutputSection &osec) {
       rank |= RF_MIPS_NOT_GOT;
   }
 
+  if (config->emachine == EM_RISCV) {
+    // .sdata and .sbss are placed closer to make GP relaxation more profitable
+    // and match GNU ld.
+    StringRef name = osec.name;
+    if (name == ".sdata" || (osec.type == SHT_NOBITS && name != ".sbss"))
+      rank |= RF_RISCV_SDATA;
+  }
+
   return rank;
 }
 
@@ -1087,8 +1096,12 @@ template <class ELFT> void Writer<ELFT>::setReservedSymbolSections() {
       ElfSym::end2->section = last->lastSec;
   }
 
-  if (ElfSym::bss)
-    ElfSym::bss->section = findSection(".bss");
+  if (ElfSym::bss) {
+    // On RISC-V, set __bss_start to the start of .sbss if present.
+    OutputSection *sbss =
+        config->emachine == EM_RISCV ? findSection(".sbss") : nullptr;
+    ElfSym::bss->section = sbss ? sbss : findSection(".bss");
+  }
 
   // Setup MIPS _gp_disp/__gnu_local_gp symbols which should
   // be equal to the _gp symbol's value.
@@ -1634,8 +1647,8 @@ template <class ELFT> void Writer<ELFT>::finalizeAddressDependentContent() {
     ++pass;
 
     // With Thunk Size much smaller than branch range we expect to
-    // converge quickly; if we get to 15 something has gone wrong.
-    if (changed && pass >= 15) {
+    // converge quickly; if we get to 30 something has gone wrong.
+    if (changed && pass >= 30) {
       error(target->needsThunks ? "thunk creation not converged"
                                 : "relaxation not converged");
       break;
diff --git a/lld/MachO/ConcatOutputSection.cpp b/lld/MachO/ConcatOutputSection.cpp
index d4f60b6dd7188..b68e55b78b07e 100644
--- a/lld/MachO/ConcatOutputSection.cpp
+++ b/lld/MachO/ConcatOutputSection.cpp
@@ -126,7 +126,7 @@ bool TextOutputSection::needsThunks() const {
     return false;
   uint64_t isecAddr = addr;
   for (ConcatInputSection *isec : inputs)
-    isecAddr = alignTo(isecAddr, isec->align) + isec->getSize();
+    isecAddr = alignToPowerOf2(isecAddr, isec->align) + isec->getSize();
   if (isecAddr - addr + in.stubs->getSize() <=
       std::min(target->backwardBranchRange, target->forwardBranchRange))
     return false;
@@ -172,7 +172,7 @@ uint64_t TextOutputSection::estimateStubsInRangeVA(size_t callIdx) const {
   uint64_t isecEnd = isecVA;
   for (size_t i = callIdx; i < inputs.size(); i++) {
     InputSection *isec = inputs[i];
-    isecEnd = alignTo(isecEnd, isec->align) + isec->getSize();
+    isecEnd = alignToPowerOf2(isecEnd, isec->align) + isec->getSize();
   }
   // Estimate the address after which call sites can safely call stubs
   // directly rather than through intermediary thunks.
@@ -194,8 +194,8 @@ uint64_t TextOutputSection::estimateStubsInRangeVA(size_t callIdx) const {
 }
 
 void ConcatOutputSection::finalizeOne(ConcatInputSection *isec) {
-  size = alignTo(size, isec->align);
-  fileSize = alignTo(fileSize, isec->align);
+  size = alignToPowerOf2(size, isec->align);
+  fileSize = alignToPowerOf2(fileSize, isec->align);
   isec->outSecOff = size;
   isec->isFinal = true;
   size += isec->getSize();
@@ -248,8 +248,9 @@ void TextOutputSection::finalize() {
     // grows. So leave room for a bunch of thunks.
     unsigned slop = 256 * thunkSize;
     while (finalIdx < endIdx) {
-      uint64_t expectedNewSize = alignTo(addr + size, inputs[finalIdx]->align) +
-                               inputs[finalIdx]->getSize();
+      uint64_t expectedNewSize =
+          alignToPowerOf2(addr + size, inputs[finalIdx]->align) +
+          inputs[finalIdx]->getSize();
       if (expectedNewSize >= isecVA + forwardBranchRange - slop)
         break;
       finalizeOne(inputs[finalIdx++]);
diff --git a/lld/MachO/InputFiles.cpp b/lld/MachO/InputFiles.cpp
index ed0d98a2ececd..db760d8f8382e 100644
--- a/lld/MachO/InputFiles.cpp
+++ b/lld/MachO/InputFiles.cpp
@@ -634,12 +634,6 @@ void ObjFile::parseRelocations(ArrayRef<SectionHeader> sectionHeaders,
   }
 }
 
-// Symbols with `l` or `L` as a prefix are linker-private and never appear in
-// the output.
-static bool isPrivateLabel(StringRef name) {
-  return name.startswith("l") || name.startswith("L");
-}
-
 template <class NList>
 static macho::Symbol *createDefined(const NList &sym, StringRef name,
                                     InputSection *isec, uint64_t value,
@@ -849,15 +843,12 @@ void ObjFile::parseSymbols(ArrayRef<typename LP::section> sectionHeaders,
     // We populate subsections by repeatedly splitting the last (highest
     // address) subsection.
     llvm::stable_sort(symbolIndices, [&](uint32_t lhs, uint32_t rhs) {
-      // Put private-label symbols that have no flags after other symbols at the
-      // same address.
-      StringRef lhsName = getSymName(nList[lhs]);
-      StringRef rhsName = getSymName(nList[rhs]);
-      if (nList[lhs].n_value == nList[rhs].n_value) {
-        if (isPrivateLabel(lhsName) && isPrivateLabel(rhsName))
-          return nList[lhs].n_desc > nList[rhs].n_desc;
-        return !isPrivateLabel(lhsName) && isPrivateLabel(rhsName);
-      }
+      // Put extern weak symbols after other symbols at the same address so
+      // that weak symbol coalescing works correctly. See
+      // SymbolTable::addDefined() for details.
+      if (nList[lhs].n_value == nList[rhs].n_value &&
+          nList[lhs].n_type & N_EXT && nList[rhs].n_type & N_EXT)
+        return !(nList[lhs].n_desc & N_WEAK_DEF) && (nList[rhs].n_desc & N_WEAK_DEF);
       return nList[lhs].n_value < nList[rhs].n_value;
     });
     for (size_t j = 0; j < symbolIndices.size(); ++j) {
@@ -883,17 +874,8 @@ void ObjFile::parseSymbols(ArrayRef<typename LP::section> sectionHeaders,
       if (!subsectionsViaSymbols || symbolOffset == 0 ||
           sym.n_desc & N_ALT_ENTRY || !isa<ConcatInputSection>(isec)) {
         isec->hasAltEntry = symbolOffset != 0;
-        // If we have an private-label symbol that's an alias, and that alias
-        // doesn't have any flags of its own, then we can just reuse the aliased
-        // symbol. Our sorting step above ensures that any such symbols will
-        // appear after the non-private-label ones. See weak-def-alias-ignored.s
-        // for the motivation behind this.
-        if (symbolOffset == 0 && isPrivateLabel(name) && j != 0 &&
-            sym.n_desc == 0)
-          symbols[symIndex] = symbols[symbolIndices[j - 1]];
-        else
-          symbols[symIndex] = createDefined(sym, name, isec, symbolOffset,
-                                            symbolSize, forceHidden);
+        symbols[symIndex] = createDefined(sym, name, isec, symbolOffset,
+                                          symbolSize, forceHidden);
         continue;
       }
       auto *concatIsec = cast<ConcatInputSection>(isec);
diff --git a/lld/MachO/MapFile.cpp b/lld/MachO/MapFile.cpp
index c16e046dcd1f7..f736360624ebd 100644
--- a/lld/MachO/MapFile.cpp
+++ b/lld/MachO/MapFile.cpp
@@ -122,8 +122,17 @@ static MapInfo gatherMapInfo() {
   return info;
 }
 
+// We use this instead of `toString(const InputFile *)` as we don't want to
+// include the dylib install name in our output.
+static void printFileName(raw_fd_ostream &os, const InputFile *f) {
+  if (f->archiveName.empty())
+    os << f->getName();
+  else
+    os << f->archiveName << "(" << path::filename(f->getName()) + ")";
+}
+
 // For printing the contents of the __stubs and __la_symbol_ptr sections.
-void printStubsEntries(
+static void printStubsEntries(
     raw_fd_ostream &os,
     const DenseMap<lld::macho::InputFile *, uint32_t> &readerToFileOrdinal,
     const OutputSection *osec, size_t entrySize) {
@@ -134,8 +143,8 @@ void printStubsEntries(
                  sym->getName().str().data());
 }
 
-void printNonLazyPointerSection(raw_fd_ostream &os,
-                                NonLazyPointerSectionBase *osec) {
+static void printNonLazyPointerSection(raw_fd_ostream &os,
+                                       NonLazyPointerSectionBase *osec) {
   // ld64 considers stubs to belong to particular files, but considers GOT
   // entries to be linker-synthesized. Not sure why they made that decision, but
   // I think we can follow suit unless there's demand for better symbol-to-file
@@ -171,7 +180,9 @@ void macho::writeMapFile() {
   uint32_t fileIndex = 1;
   DenseMap<lld::macho::InputFile *, uint32_t> readerToFileOrdinal;
   for (InputFile *file : info.files) {
-    os << format("[%3u] %s\n", fileIndex, file->getName().str().c_str());
+    os << format("[%3u] ", fileIndex);
+    printFileName(os, file);
+    os << "\n";
     readerToFileOrdinal[file] = fileIndex++;
   }
 
@@ -193,9 +204,10 @@ void macho::writeMapFile() {
       if (auto *concatOsec = dyn_cast<ConcatOutputSection>(osec)) {
         for (const InputSection *isec : concatOsec->inputs) {
           for (Defined *sym : isec->symbols)
-            os << format("0x%08llX\t0x%08llX\t[%3u] %s\n", sym->getVA(),
-                         sym->size, readerToFileOrdinal[sym->getFile()],
-                         sym->getName().str().data());
+            if (!(isPrivateLabel(sym->getName()) && sym->size == 0))
+              os << format("0x%08llX\t0x%08llX\t[%3u] %s\n", sym->getVA(),
+                           sym->size, readerToFileOrdinal[sym->getFile()],
+                           sym->getName().str().data());
         }
       } else if (osec == in.cStringSection || osec == in.objcMethnameSection) {
         const auto &liveCStrings = info.liveCStringsForSection.lookup(osec);
diff --git a/lld/MachO/SymbolTable.cpp b/lld/MachO/SymbolTable.cpp
index 03631239ccba1..09b3d8a376e60 100644
--- a/lld/MachO/SymbolTable.cpp
+++ b/lld/MachO/SymbolTable.cpp
@@ -61,6 +61,38 @@ struct DuplicateSymbolDiag {
 SmallVector<DuplicateSymbolDiag> dupSymDiags;
 } // namespace
 
+// Move symbols at \p fromOff in \p fromIsec into \p toIsec, unless that symbol
+// is \p skip.
+static void transplantSymbolsAtOffset(InputSection *fromIsec,
+                                      InputSection *toIsec, Defined *skip,
+                                      uint64_t fromOff, uint64_t toOff) {
+  // Ensure the symbols will still be in address order after our insertions.
+  auto insertIt = llvm::upper_bound(toIsec->symbols, toOff,
+                                    [](uint64_t off, const Symbol *s) {
+                                      return cast<Defined>(s)->value < off;
+                                    });
+  llvm::erase_if(fromIsec->symbols, [&](Symbol *s) {
+    auto *d = cast<Defined>(s);
+    if (d->value != fromOff)
+      return false;
+    if (d != skip) {
+      // This repeated insertion will be quadratic unless insertIt is the end
+      // iterator. However, that is typically the case for files that have
+      // .subsections_via_symbols set.
+      insertIt = toIsec->symbols.insert(insertIt, d);
+      d->isec = toIsec;
+      d->value = toOff;
+      // We don't want to have more than one unwindEntry at a given address, so
+      // drop the redundant ones. We We can safely drop the unwindEntries of
+      // the symbols in fromIsec since we will be adding another unwindEntry as
+      // we finish parsing toIsec's file. (We can assume that toIsec has its
+      // own unwindEntry because of the ODR.)
+      d->unwindEntry = nullptr;
+    }
+    return true;
+  });
+}
+
 Defined *SymbolTable::addDefined(StringRef name, InputFile *file,
                                  InputSection *isec, uint64_t value,
                                  uint64_t size, bool isWeakDef,
@@ -82,18 +114,27 @@ Defined *SymbolTable::addDefined(StringRef name, InputFile *file,
           defined->referencedDynamically |= isReferencedDynamically;
           defined->noDeadStrip |= noDeadStrip;
         }
-        // FIXME: Handle this for bitcode files.
-        if (auto concatIsec = dyn_cast_or_null<ConcatInputSection>(isec))
+        if (auto concatIsec = dyn_cast_or_null<ConcatInputSection>(isec)) {
           concatIsec->wasCoalesced = true;
+          // Any local symbols that alias the coalesced symbol should be moved
+          // into the prevailing section. Note that we have sorted the symbols
+          // in ObjFile::parseSymbols() such that extern weak symbols appear
+          // last, so we don't need to worry about subsequent symbols being
+          // added to an already-coalesced section.
+          if (defined->isec)
+            transplantSymbolsAtOffset(concatIsec, defined->isec,
+                                      /*skip=*/nullptr, value, defined->value);
+        }
         return defined;
       }
 
       if (defined->isWeakDef()) {
-        // FIXME: Handle this for bitcode files.
         if (auto concatIsec =
                 dyn_cast_or_null<ConcatInputSection>(defined->isec)) {
           concatIsec->wasCoalesced = true;
-          concatIsec->symbols.erase(llvm::find(concatIsec->symbols, defined));
+          if (isec)
+            transplantSymbolsAtOffset(concatIsec, isec, defined, defined->value,
+                                      value);
         }
       } else {
         std::string srcLoc1 = defined->getSourceLocation();
@@ -382,7 +423,7 @@ struct UndefinedDiag {
 };
 
 MapVector<const Undefined *, UndefinedDiag> undefs;
-}
+} // namespace
 
 void macho::reportPendingDuplicateSymbols() {
   for (const auto &duplicate : dupSymDiags) {
diff --git a/lld/MachO/Symbols.h b/lld/MachO/Symbols.h
index 6113f2d7b0ec6..d8e2f86e525b3 100644
--- a/lld/MachO/Symbols.h
+++ b/lld/MachO/Symbols.h
@@ -383,6 +383,12 @@ inline bool needsBinding(const Symbol *sym) {
     return defined->isExternalWeakDef() || defined->interposable;
   return false;
 }
+
+// Symbols with `l` or `L` as a prefix are linker-private and never appear in
+// the output.
+inline bool isPrivateLabel(StringRef name) {
+  return name.startswith("l") || name.startswith("L");
+}
 } // namespace macho
 
 std::string toString(const macho::Symbol &);
diff --git a/lld/MachO/SyntheticSections.cpp b/lld/MachO/SyntheticSections.cpp
index ca681eb9c90a4..95da58ca23c2d 100644
--- a/lld/MachO/SyntheticSections.cpp
+++ b/lld/MachO/SyntheticSections.cpp
@@ -95,7 +95,7 @@ uint64_t MachHeaderSection::getSize() const {
   // If we are emitting an encryptable binary, our load commands must have a
   // separate (non-encrypted) page to themselves.
   if (config->emitEncryptionInfo)
-    size = alignTo(size, target->getPageSize());
+    size = alignToPowerOf2(size, target->getPageSize());
   return size;
 }
 
@@ -1642,7 +1642,7 @@ void CStringSection::finalizeContents() {
       // handled.
       uint32_t pieceAlign = 1
                             << llvm::countr_zero(isec->align | piece.inSecOff);
-      offset = alignTo(offset, pieceAlign);
+      offset = alignToPowerOf2(offset, pieceAlign);
       piece.outSecOff = offset;
       isec->isFinal = true;
       StringRef string = isec->getStringRef(i);
@@ -1717,7 +1717,8 @@ void DeduplicatedCStringSection::finalizeContents() {
       assert(it != stringOffsetMap.end());
       StringOffset &offsetInfo = it->second;
       if (offsetInfo.outSecOff == UINT64_MAX) {
-        offsetInfo.outSecOff = alignTo(size, 1ULL << offsetInfo.trailingZeros);
+        offsetInfo.outSecOff =
+            alignToPowerOf2(size, 1ULL << offsetInfo.trailingZeros);
         size =
             offsetInfo.outSecOff + s.size() + 1; // account for null terminator
       }
diff --git a/lld/MachO/Writer.cpp b/lld/MachO/Writer.cpp
index 28c01d39764a9..b090ca2a72b76 100644
--- a/lld/MachO/Writer.cpp
+++ b/lld/MachO/Writer.cpp
@@ -1077,9 +1077,10 @@ void Writer::finalizeAddresses() {
     seg->addr = addr;
     assignAddresses(seg);
     // codesign / libstuff checks for segment ordering by verifying that
-    // `fileOff + fileSize == next segment fileOff`. So we call alignTo() before
-    // (instead of after) computing fileSize to ensure that the segments are
-    // contiguous. We handle addr / vmSize similarly for the same reason.
+    // `fileOff + fileSize == next segment fileOff`. So we call
+    // alignToPowerOf2() before (instead of after) computing fileSize to ensure
+    // that the segments are contiguous. We handle addr / vmSize similarly for
+    // the same reason.
     fileOff = alignToPowerOf2(fileOff, pageSize);
     addr = alignToPowerOf2(addr, pageSize);
     seg->vmSize = addr - seg->addr;
@@ -1122,8 +1123,8 @@ void Writer::assignAddresses(OutputSegment *seg) {
   for (OutputSection *osec : seg->getSections()) {
     if (!osec->isNeeded())
       continue;
-    addr = alignTo(addr, osec->align);
-    fileOff = alignTo(fileOff, osec->align);
+    addr = alignToPowerOf2(addr, osec->align);
+    fileOff = alignToPowerOf2(fileOff, osec->align);
     osec->addr = addr;
     osec->fileOff = isZeroFill(osec->flags) ? 0 : fileOff;
     osec->finalize();
diff --git a/lld/test/ELF/linkerscript/sort-init.s b/lld/test/ELF/linkerscript/sort-init.s
index 0a349fd9e57f3..e1f0c83e74e9f 100644
--- a/lld/test/ELF/linkerscript/sort-init.s
+++ b/lld/test/ELF/linkerscript/sort-init.s
@@ -9,6 +9,14 @@
 # CHECK:      Hex dump of section '.init_array':
 # CHECK-NEXT: 0x00000001 00010203 04050607
 
+## Test REVERSE can be used to reverse the order of .init_array and .ctors
+
+# RUN: ld.lld -T %t/reverse.lds %t.o -o %t2.out
+# RUN: llvm-readelf -x .init_array %t2.out | FileCheck %s --check-prefix=CHECK2
+
+# CHECK2:      Hex dump of section '.init_array':
+# CHECK2-NEXT: 0x00000001 04030201 00050706
+
 #--- asm
 .globl _start
 _start:
@@ -41,3 +49,11 @@ SECTIONS {
     *(.init_array .ctors)
   }
 }
+
+#--- reverse.lds
+SECTIONS {
+  .init_array : {
+    *(REVERSE(SORT_BY_INIT_PRIORITY(.init_array.* .ctors.*)) SORT_BY_INIT_PRIORITY(foo*))
+    *(REVERSE(.init_array .ctors))
+  }
+}
diff --git a/lld/test/ELF/riscv-gp-no-sdata.s b/lld/test/ELF/riscv-gp-no-sdata.s
deleted file mode 100644
index ee86438ec4f35..0000000000000
--- a/lld/test/ELF/riscv-gp-no-sdata.s
+++ /dev/null
@@ -1,15 +0,0 @@
-# REQUIRES: riscv
-# RUN: llvm-mc -filetype=obj -triple=riscv32 %s -o %t.32.o
-# RUN: ld.lld -pie %t.32.o -o %t.32
-# RUN: llvm-readelf -s %t.32 | FileCheck --check-prefix=SYM %s
-
-# RUN: llvm-mc -filetype=obj -triple=riscv64 %s -o %t.64.o
-# RUN: ld.lld -pie %t.64.o -o %t.64
-# RUN: llvm-readelf -s %t.64 | FileCheck --check-prefix=SYM %s
-
-## If there is an undefined reference to __global_pointer$ but .sdata doesn't
-## exist, define __global_pointer$ and set its st_shndx arbitrarily to 1.
-
-# SYM: {{0*}}00000800 0 NOTYPE GLOBAL DEFAULT 1 __global_pointer$
-
-lla gp, __global_pointer$
diff --git a/lld/test/ELF/riscv-section-layout.s b/lld/test/ELF/riscv-section-layout.s
new file mode 100644
index 0000000000000..bb9adf5eef054
--- /dev/null
+++ b/lld/test/ELF/riscv-section-layout.s
@@ -0,0 +1,58 @@
+# REQUIRES: riscv
+## Test RISC-V specific section layout. See also section-layout.s and riscv-gp.s.
+
+# RUN: llvm-mc -filetype=obj -triple=riscv32 %s -o %t.32.o
+# RUN: ld.lld -pie %t.32.o -o %t.32
+# RUN: llvm-readelf -S -s %t.32 | FileCheck %s --check-prefix=NOSDATA
+# RUN: llvm-mc -filetype=obj -triple=riscv32 --defsym=SDATA=1 %s -o %t.32s.o
+# RUN: ld.lld -pie %t.32s.o -o %t.32s
+# RUN: llvm-readelf -S -s %t.32s | FileCheck %s
+
+# RUN: llvm-mc -filetype=obj -triple=riscv64 %s -o %t.64.o
+# RUN: ld.lld -pie %t.64.o -o %t.64
+# RUN: llvm-readelf -S -s %t.64 | FileCheck %s --check-prefix=NOSDATA
+# RUN: llvm-mc -filetype=obj -triple=riscv64 --defsym=SDATA=1 %s -o %t.64s.o
+# RUN: ld.lld -pie %t.64s.o -o %t.64s
+# RUN: llvm-readelf -S -s %t.64s | FileCheck %s
+
+# NOSDATA:      .text
+# NOSDATA-NEXT: .tdata
+# NOSDATA-NEXT: .tbss
+# NOSDATA-NEXT: .dynamic
+# NOSDATA-NEXT: .got
+# NOSDATA-NEXT: .data    PROGBITS [[#%x,DATA:]]
+# NOSDATA-NEXT: .bss     NOBITS   [[#%x,BSS:]]
+
+## If there is an undefined reference to __global_pointer$ but .sdata doesn't
+## exist, define __global_pointer$ and set its st_shndx arbitrarily to 1.
+## The symbol value should not be used by the program.
+
+# NOSDATA-DAG:  [[#]]: {{0*}}[[#BSS]]         0 NOTYPE  GLOBAL DEFAULT [[#]] _edata
+# NOSDATA-DAG:  [[#]]: {{0*}}[[#BSS]]         0 NOTYPE  GLOBAL DEFAULT [[#]] __bss_start
+# NOSDATA-DAG:  [[#]]: {{0*}}800              0 NOTYPE  GLOBAL DEFAULT     1 __global_pointer$
+
+# CHECK:      .text
+# CHECK-NEXT: .tdata
+# CHECK-NEXT: .tbss
+# CHECK-NEXT: .dynamic
+# CHECK-NEXT: .got
+# CHECK-NEXT: .data
+# CHECK-NEXT: .sdata     PROGBITS [[#%x,SDATA:]]
+# CHECK-NEXT: .sbss      NOBITS   [[#%x,SBSS:]]
+# CHECK-NEXT: .bss
+
+# CHECK-DAG:  [[#]]: {{0*}}[[#SBSS]]        0 NOTYPE  GLOBAL DEFAULT [[#]] _edata
+# CHECK-DAG:  [[#]]: {{0*}}[[#SBSS]]        0 NOTYPE  GLOBAL DEFAULT [[#]] __bss_start
+# CHECK-DAG:  [[#]]: {{0*}}[[#SDATA+0x800]] 0 NOTYPE  GLOBAL DEFAULT [[#]] __global_pointer$
+
+.globl _edata, __bss_start
+  lla gp, __global_pointer$
+
+.section .data,"aw",@progbits; .long _GLOBAL_OFFSET_TABLE_ - .
+.section .bss,"aw",@nobits; .space 1
+.section .tdata,"awT",@progbits; .space 1
+.section .tbss,"awT",@nobits; .space 1
+.ifdef SDATA
+.section .sdata,"aw",@progbits; .space 1
+.section .sbss,"aw",@nobits; .space 1
+.endif
diff --git a/lld/test/MachO/local-alias-to-weak.s b/lld/test/MachO/local-alias-to-weak.s
new file mode 100644
index 0000000000000..feb4c0a2e4296
--- /dev/null
+++ b/lld/test/MachO/local-alias-to-weak.s
@@ -0,0 +1,149 @@
+# REQUIRES: x86
+## This test checks that when we coalesce weak definitions, their local symbol
+## aliases defs don't cause the coalesced data to be retained. This was
+## motivated by MC's aarch64 backend which automatically creates `ltmp<N>`
+## symbols at the start of each .text section. These symbols are frequently
+## aliases of other symbols created by clang or other inputs to MC. I've chosen
+## to explicitly create them here since we can then reference those symbols for
+## a more complete test.
+##
+## Not retaining the data matters for more than just size -- we have a use case
+## that depends on proper data coalescing to emit a valid file format. We also
+## need this behavior to properly deduplicate the __objc_protolist section;
+## failure to do this can result in dyld crashing on iOS 13.
+##
+## Finally, ld64 does all this regardless of whether .subsections_via_symbols is
+## specified. We don't. But again, given how rare the lack of that directive is
+## (I've only seen it from hand-written assembly inputs), I don't think we need
+## to worry about it.
+
+# RUN: rm -rf %t; split-file %s %t
+# RUN: llvm-mc -filetype=obj -triple=x86_64-apple-darwin %t/weak-then-local.s -o %t/weak-then-local.o
+# RUN: llvm-mc -filetype=obj -triple=x86_64-apple-darwin %t/local-then-weak.s -o %t/local-then-weak.o
+# RUN: llvm-mc -filetype=obj -triple=x86_64-apple-darwin %t/no-subsections.s -o %t/no-subsections.o
+# RUN: llvm-mc -filetype=obj -triple=x86_64-apple-darwin %t/no-dead-strip.s -o %t/no-dead-strip.o
+
+# RUN: %lld -lSystem -dylib %t/weak-then-local.o %t/local-then-weak.o -o %t/test1
+# RUN: llvm-objdump --macho --syms --section="__DATA,__data" --weak-bind %t/test1 | FileCheck %s
+# RUN: %lld -lSystem -dylib %t/local-then-weak.o %t/weak-then-local.o -o %t/test2
+# RUN: llvm-objdump --macho --syms --section="__DATA,__data" --weak-bind %t/test2 | FileCheck %s
+
+## Check that we only have one copy of 0x123 in the data, not two.
+# CHECK:       Contents of (__DATA,__data) section
+# CHECK-NEXT:  0000000000001000  23 01 00 00 00 00 00 00 00 10 00 00 00 00 00 00 {{$}}
+# CHECK-NEXT:  0000000000001010  00 10 00 00 00 00 00 00 {{$}}
+# CHECK-EMPTY:
+# CHECK-NEXT:  SYMBOL TABLE:
+# CHECK-NEXT:  0000000000001000 l     O __DATA,__data _alias
+# CHECK-NEXT:  0000000000001008 l     O __DATA,__data _ref
+# CHECK-NEXT:  0000000000001000 l     O __DATA,__data _alias
+# CHECK-NEXT:  0000000000001010 l     O __DATA,__data _ref
+# CHECK-NEXT:  0000000000001000  w    O __DATA,__data _weak
+# CHECK-NEXT:  0000000000000000         *UND* dyld_stub_binder
+# CHECK-EMPTY:
+## Even though the references were to the non-weak `_alias` symbols, ld64 still
+## emits weak binds as if they were the `_weak` symbol itself. We do not. I
+## don't know of any programs that rely on this behavior, so I'm just
+## documenting it here.
+# CHECK-NEXT:  Weak bind table:
+# CHECK-NEXT:  segment  section            address     type       addend   symbol
+# CHECK-EMPTY:
+
+# RUN: %lld -lSystem -dylib %t/local-then-weak.o %t/no-subsections.o -o %t/sub-nosub
+# RUN: llvm-objdump --macho --syms --section="__DATA,__data" %t/sub-nosub | FileCheck %s --check-prefix SUB-NOSUB
+
+## This test case demonstrates a shortcoming of LLD: If .subsections_via_symbols
+## isn't enabled, we don't elide the contents of coalesced weak symbols if they
+## are part of a section that has other non-coalesced symbols. In contrast, LD64
+## does elide the contents.
+# SUB-NOSUB:       Contents of (__DATA,__data) section
+# SUB-NOSUB-NEXT:  0000000000001000    23 01 00 00 00 00 00 00 00 10 00 00 00 00 00 00
+# SUB-NOSUB-NEXT:  0000000000001010    00 00 00 00 00 00 00 00 23 01 00 00 00 00 00 00
+# SUB-NOSUB-EMPTY:
+# SUB-NOSUB-NEXT:  SYMBOL TABLE:
+# SUB-NOSUB-NEXT:  0000000000001000 l     O __DATA,__data _alias
+# SUB-NOSUB-NEXT:  0000000000001008 l     O __DATA,__data _ref
+# SUB-NOSUB-NEXT:  0000000000001010 l     O __DATA,__data _zeros
+# SUB-NOSUB-NEXT:  0000000000001000 l     O __DATA,__data _alias
+# SUB-NOSUB-NEXT:  0000000000001000  w    O __DATA,__data _weak
+# SUB-NOSUB-NEXT:  0000000000000000         *UND* dyld_stub_binder
+
+# RUN: %lld -lSystem -dylib %t/no-subsections.o %t/local-then-weak.o -o %t/nosub-sub
+# RUN: llvm-objdump --macho --syms --section="__DATA,__data" %t/nosub-sub | FileCheck %s --check-prefix NOSUB-SUB
+
+# NOSUB-SUB:       Contents of (__DATA,__data) section
+# NOSUB-SUB-NEXT:  0000000000001000    00 00 00 00 00 00 00 00 23 01 00 00 00 00 00 00
+# NOSUB-SUB-NEXT:  0000000000001010    08 10 00 00 00 00 00 00 {{$}}
+# NOSUB-SUB-EMPTY:
+# NOSUB-SUB-NEXT:  SYMBOL TABLE:
+# NOSUB-SUB-NEXT:  0000000000001000 l     O __DATA,__data _zeros
+# NOSUB-SUB-NEXT:  0000000000001008 l     O __DATA,__data _alias
+# NOSUB-SUB-NEXT:  0000000000001008 l     O __DATA,__data _alias
+# NOSUB-SUB-NEXT:  0000000000001010 l     O __DATA,__data _ref
+# NOSUB-SUB-NEXT:  0000000000001008  w    O __DATA,__data _weak
+# NOSUB-SUB-NEXT:  0000000000000000         *UND* dyld_stub_binder
+
+## Verify that we don't drop any flags that the aliases have (such as
+## .no_dead_strip). This is a regression test. We previously had subsections
+## that were mistakenly stripped.
+
+# RUN: %lld -lSystem -dead_strip %t/no-dead-strip.o -o %t/no-dead-strip
+# RUN: llvm-objdump --macho --section-headers %t/no-dead-strip | FileCheck %s \
+# RUN:   --check-prefix=NO-DEAD-STRIP
+# NO-DEAD-STRIP: __data        00000010
+
+#--- weak-then-local.s
+.globl _weak
+.weak_definition _weak
+.data
+_weak:
+_alias:
+  .quad 0x123
+
+_ref:
+  .quad _alias
+
+.subsections_via_symbols
+
+#--- local-then-weak.s
+.globl _weak
+.weak_definition _weak
+.data
+_alias:
+_weak:
+  .quad 0x123
+
+_ref:
+  .quad _alias
+
+.subsections_via_symbols
+
+#--- no-subsections.s
+.globl _weak
+.weak_definition _weak
+.data
+_zeros:
+.space 8
+
+_weak:
+_alias:
+  .quad 0x123
+
+#--- no-dead-strip.s
+.globl _main
+
+_main:
+  ret
+
+.data
+.no_dead_strip l_foo, l_bar
+
+_foo:
+l_foo:
+  .quad 0x123
+
+l_bar:
+_bar:
+  .quad 0x123
+
+.subsections_via_symbols
diff --git a/lld/test/MachO/map-file.ll b/lld/test/MachO/map-file.ll
index 2f8332a2a3732..c8ab444370b3e 100644
--- a/lld/test/MachO/map-file.ll
+++ b/lld/test/MachO/map-file.ll
@@ -27,9 +27,9 @@
 ; FOOBAR-NEXT: # Arch: x86_64
 ; FOOBAR-NEXT: # Object files:
 ; FOOBAR-NEXT: [  0] linker synthesized
-; FOOBAR-NEXT: [  1] {{.*}}{{/|\\}}usr/lib{{/|\\}}libSystem.tbd
-; FOOBAR-NEXT: [  2] {{.*}}{{/|\\}}map-file.ll.tmp/foo.thinlto.o
-; FOOBAR-NEXT: [  3] {{.*}}{{/|\\}}map-file.ll.tmp/bar.thinlto.o
+; FOOBAR-NEXT: [  1] {{.*}}{{/|\\}}usr/lib{{/|\\}}libSystem.tbd{{$}}
+; FOOBAR-NEXT: [  2] {{.*}}{{/|\\}}map-file.ll.tmp/foo.thinlto.o{{$}}
+; FOOBAR-NEXT: [  3] {{.*}}{{/|\\}}map-file.ll.tmp/bar.thinlto.o{{$}}
 ; FOOBAR-NEXT: # Sections:
 ; FOOBAR:      # Symbols:
 ; FOOBAR-NEXT: # Address        Size             File  Name
diff --git a/lld/test/MachO/map-file.s b/lld/test/MachO/map-file.s
index 64c66830b21ef..aa9fff9938eb2 100644
--- a/lld/test/MachO/map-file.s
+++ b/lld/test/MachO/map-file.s
@@ -5,9 +5,11 @@
 # RUN: llvm-mc -filetype=obj -triple=x86_64-apple-darwin %t/c-string-literal.s -o %t/c-string-literal.o
 # RUN: llvm-mc -filetype=obj -triple=x86_64-apple-darwin %t/baz.s -o %t/baz.o
 
+# RUN: llvm-ar --format=darwin crs %t/libfoo.a %t/foo.o
 # RUN: %lld -dylib %t/baz.o -o %t/libbaz.dylib
-# RUN: %lld -demangle -map %t/map %t/test.o %t/foo.o %t/c-string-literal.o \
-# RUN:   %t/libbaz.dylib --time-trace -o %t/test --no-deduplicate-strings
+# RUN: %lld -demangle -map %t/map %t/test.o -force_load %t/libfoo.a \
+# RUN:   %t/c-string-literal.o %t/libbaz.dylib --time-trace -o %t/test \
+# RUN:   --no-deduplicate-strings
 # RUN: llvm-objdump --syms --section-headers %t/test > %t/objdump
 ## Check that symbols in cstring sections aren't emitted
 ## Also check that we don't have redundant EH_Frame symbols (regression test)
@@ -17,7 +19,7 @@
 
 # CHECK:       Sections:
 # CHECK-NEXT:  Idx  Name         Size     VMA               Type
-# CHECK-NEXT:  0 __text          0000001b [[#%x,TEXT:]]     TEXT
+# CHECK-NEXT:  0 __text          0000001c [[#%x,TEXT:]]     TEXT
 # CHECK-NEXT:  1 __stubs         0000000c [[#%x,STUBS:]]    TEXT
 # CHECK-NEXT:  2 __stub_helper   0000001a [[#%x,HELPER:]]   TEXT
 # CHECK-NEXT:  3 __cstring       0000002b [[#%x,CSTR:]]     DATA
@@ -42,11 +44,11 @@
 # CHECK-NEXT: # Arch: x86_64
 # CHECK-NEXT: # Object files:
 # CHECK-NEXT: [  0] linker synthesized
-# CHECK-NEXT: [  1] {{.*}}{{/|\\}}usr/lib{{/|\\}}libSystem.tbd
-# CHECK-NEXT: [  2] {{.*}}{{/|\\}}map-file.s.tmp/test.o
-# CHECK-NEXT: [  3] {{.*}}{{/|\\}}map-file.s.tmp/foo.o
-# CHECK-NEXT: [  4] {{.*}}{{/|\\}}map-file.s.tmp/c-string-literal.o
-# CHECK-NEXT: [  5] {{.*}}{{/|\\}}map-file.s.tmp/libbaz.dylib
+# CHECK-NEXT: [  1] {{.*}}{{/|\\}}usr/lib{{/|\\}}libSystem.tbd{{$}}
+# CHECK-NEXT: [  2] {{.*}}{{/|\\}}map-file.s.tmp/test.o{{$}}
+# CHECK-NEXT: [  3] {{.*}}{{/|\\}}map-file.s.tmp/libfoo.a(foo.o){{$}}
+# CHECK-NEXT: [  4] {{.*}}{{/|\\}}map-file.s.tmp/c-string-literal.o{{$}}
+# CHECK-NEXT: [  5] {{.*}}{{/|\\}}map-file.s.tmp/libbaz.dylib{{$}}
 
 # CHECK-NEXT: # Sections:
 # CHECK-NEXT: # Address           Size            Segment  Section
@@ -67,6 +69,7 @@
 # CHECK-NEXT: 0x[[#%X,MAIN]]           0x00000019  [  2] _main
 # CHECK-NEXT: 0x[[#%X,BAR]]            0x00000001  [  2] _bar
 # CHECK-NEXT: 0x[[#%X,FOO]]            0x00000001  [  3] __ZTIN3foo3bar4MethE
+# CHECK-NEXT: 0x[[#%X,FOO+1]]          0x00000001  [  3] ltmp1
 # CHECK-NEXT: 0x[[#%X,STUBS]]          0x00000006  [  5] _baz
 # CHECK-NEXT: 0x[[#%X,STUBS+6]]        0x00000006  [  2] _bar
 # CHECK-NEXT: 0x[[#%X,HELPER]]         0x0000001A  [  0] helper helper
@@ -84,11 +87,12 @@
 # CHECK-NEXT: 0x[[#%X,DYLD]]           0x00000000  [  0] __dyld_private
 # CHECK-NEXT: 0x[[#%X,TLVP]]           0x00000008  [  0] non-lazy-pointer-to-local: _baz_tlv
 # CHECK-NEXT: 0x[[#%X,BSS]]            0x00000001  [  2] _number
+# CHECK-EMPTY:
 
 # MAPFILE: "name":"Total Write map file"
 
-# RUN: %lld -demangle -dead_strip -map %t/stripped-map %t/test.o %t/foo.o \
-# RUN:   %t/c-string-literal.o %t/libbaz.dylib -o %t/stripped
+# RUN: %lld -demangle -dead_strip -map %t/stripped-map %t/test.o -force_load \
+# RUN:   %t/libfoo.a %t/c-string-literal.o %t/libbaz.dylib -o %t/stripped
 # RUN: FileCheck --check-prefix=STRIPPED %s < %t/stripped-map
 
 # STRIPPED-LABEL: Dead Stripped Symbols:
@@ -98,8 +102,8 @@
 # STRIPPED-DAG:   <<dead>>  0x0000000F  [  4] literal string: Hello, it's me
 # STRIPPED-DAG:   <<dead>>  0x0000000E  [  4] literal string: Hello world!\n
 
-# RUN: %lld --icf=all -map %t/icf-map %t/test.o %t/foo.o %t/c-string-literal.o \
-# RUN:   %t/libbaz.dylib -o /dev/null
+# RUN: %lld --icf=all -map %t/icf-map %t/test.o -force_load %t/libfoo.a \
+# RUN:   %t/c-string-literal.o %t/libbaz.dylib -o /dev/null
 # RUN: FileCheck --check-prefix=ICF %s < %t/icf-map
 
 ## Verify that folded symbols and cstrings have size zero. Note that ld64 prints
@@ -113,11 +117,19 @@
 
 #--- foo.s
 .globl __ZTIN3foo3bar4MethE
+## This should not appear in the map file since it is a zero-size private label
+## symbol.
+ltmp0:
 ## This C++ symbol makes it clear that we do not print the demangled name in
 ## the map file, even if `-demangle` is passed.
 __ZTIN3foo3bar4MethE:
   nop
 
+## This private label symbol will appear in the map file since it has nonzero
+## size.
+ltmp1:
+  nop
+
 .subsections_via_symbols
 
 #--- test.s
diff --git a/lld/test/MachO/private-label-alias.s b/lld/test/MachO/private-label-alias.s
deleted file mode 100644
index c5eb6277206c8..0000000000000
--- a/lld/test/MachO/private-label-alias.s
+++ /dev/null
@@ -1,105 +0,0 @@
-# REQUIRES: x86
-## This test checks that when we coalesce weak definitions, any flag-less
-## private-label aliases to those weak defs don't cause the coalesced data to be
-## retained. This test explicitly creates those private-label symbols, but it
-## was actually motivated by MC's aarch64 backend which automatically creates
-## them when emitting object files. I've chosen to explicitly create them here
-## since we can then reference those symbols for a more complete test.
-##
-## Not retaining the data matters for more than just size -- we have a use case
-## that depends on proper data coalescing to emit a valid file format.
-##
-## ld64 actually treats all local symbol aliases (not just the private ones) the
-## same way. But implementing this is harder -- we would have to create those
-## symbols first (so we can emit their names later), but we would have to
-## ensure the linker correctly shuffles them around when their aliasees get
-## coalesced. Emulating the behavior of weak binds for non-private symbols would
-## be even trickier. Let's just deal with private-label symbols for now until we
-## find a use case for more general local symbols.
-##
-## Finally, ld64 does all this regardless of whether .subsections_via_symbols is
-## specified. We don't. But again, given how rare the lack of that directive is
-## (I've only seen it from hand-written assembly inputs), I don't think we need
-## to worry about it.
-
-# RUN: rm -rf %t; split-file %s %t
-# RUN: llvm-mc -filetype=obj -triple=x86_64-apple-darwin %t/weak-then-private.s -o %t/weak-then-private.o
-# RUN: llvm-mc -filetype=obj -triple=x86_64-apple-darwin %t/private-then-weak.s -o %t/private-then-weak.o
-# RUN: llvm-mc -filetype=obj -triple=x86_64-apple-darwin %t/no-dead-strip.s -o %t/no-dead-strip.o
-# RUN: %lld -dylib %t/weak-then-private.o %t/private-then-weak.o -o %t/test1
-# RUN: %lld -dylib %t/private-then-weak.o %t/weak-then-private.o -o %t/test2
-# RUN: %lld -dead_strip %t/no-dead-strip.o -o %t/no-dead-strip
-
-# RUN: llvm-objdump --macho --syms --section="__DATA,__data" --weak-bind %t/test1 | FileCheck %s
-# RUN: llvm-objdump --macho --syms --section="__DATA,__data" --weak-bind %t/test2 | FileCheck %s
-
-## Check that we only have one copy of 0x123 in the data, not two.
-# CHECK:       Contents of (__DATA,__data) section
-# CHECK-NEXT:  0000000000001000  23 01 00 00 00 00 00 00 00 10 00 00 00 00 00 00 {{$}}
-# CHECK-NEXT:  0000000000001010  00 10 00 00 00 00 00 00 {{$}}
-# CHECK-EMPTY:
-# CHECK-NEXT:  SYMBOL TABLE:
-# CHECK-NEXT:  0000000000001008 l     O __DATA,__data _ref
-# CHECK-NEXT:  0000000000001010 l     O __DATA,__data _ref
-# CHECK-NEXT:  0000000000001000  w    O __DATA,__data _weak
-# CHECK-NEXT:  0000000000000000         *UND* dyld_stub_binder
-# CHECK-EMPTY:
-## Even though the references were to the non-weak `l_ignored` aliases, we
-## should still emit weak binds as if they were the `_weak` symbol itself.
-# CHECK-NEXT:  Weak bind table:
-# CHECK-NEXT:  segment  section            address     type       addend   symbol
-# CHECK-NEXT:  __DATA   __data             0x00001008 pointer         0   _weak
-# CHECK-NEXT:  __DATA   __data             0x00001010 pointer         0   _weak
-
-## Verify that we don't drop any flags that private-label aliases have (such as
-## .no_dead_strip). This is a regression test. We previously had subsections
-## that were mistakenly stripped.
-
-# RUN: llvm-objdump --macho --section-headers %t/no-dead-strip | FileCheck %s \
-# RUN:   --check-prefix=NO-DEAD-STRIP
-# NO-DEAD-STRIP: __data        00000010
-
-#--- weak-then-private.s
-.globl _weak
-.weak_definition _weak
-.data
-_weak:
-l_ignored:
-  .quad 0x123
-
-_ref:
-  .quad l_ignored
-
-.subsections_via_symbols
-
-#--- private-then-weak.s
-.globl _weak
-.weak_definition _weak
-.data
-l_ignored:
-_weak:
-  .quad 0x123
-
-_ref:
-  .quad l_ignored
-
-.subsections_via_symbols
-
-#--- no-dead-strip.s
-.globl _main
-
-_main:
-  ret
-
-.data
-.no_dead_strip l_foo, l_bar
-
-_foo:
-l_foo:
-  .quad 0x123
-
-l_bar:
-_bar:
-  .quad 0x123
-
-.subsections_via_symbols
diff --git a/lld/test/MachO/tools/generate-cfi-funcs.py b/lld/test/MachO/tools/generate-cfi-funcs.py
index 3f22008349f46..e2b75d0cd9a5a 100755
--- a/lld/test/MachO/tools/generate-cfi-funcs.py
+++ b/lld/test/MachO/tools/generate-cfi-funcs.py
@@ -23,7 +23,7 @@ def print_function(name):
   global lsda_odds
   have_lsda = (random.random() < lsda_odds)
   frame_size = random.randint(4, 64) * 16
-  frame_offset = -random.randint(0, (frame_size/16 - 4)) * 16
+  frame_offset = -random.randint(0, int(frame_size/16 - 4)) * 16
   global func_size_low, func_size_high
   func_size = random.randint(func_size_low, func_size_high) * 0x10
   func_size_high += 1
diff --git a/lld/test/wasm/lto/pic-empty.s b/lld/test/wasm/lto/pic-empty.s
index 51938e1ad429d..4fe4dffda1dcb 100644
--- a/lld/test/wasm/lto/pic-empty.s
+++ b/lld/test/wasm/lto/pic-empty.s
@@ -3,7 +3,7 @@
 ; We had a bug where the mutable-globals feature was not being added
 ; so the linker-generated import of `__stack_pointer` (which is currently
 ; mandatory for ; shared libraries) was generating a linker error.
-; See https://bugs.llvm.org/show_bug.cgi?id=52339
+; See https://github.com/llvm/llvm-project/issues/51681.
 
 ; RUN: llvm-as %s -o %t.o
 ; RUN: wasm-ld --lto-O2 --experimental-pic -shared --no-gc-sections --export=tls_int %t.o -o %t.so
diff --git a/lld/test/wasm/why-extract.s b/lld/test/wasm/why-extract.s
new file mode 100644
index 0000000000000..88be4b3fcae51
--- /dev/null
+++ b/lld/test/wasm/why-extract.s
@@ -0,0 +1,87 @@
+# RUN: rm -rf %t && split-file %s %t
+# RUN: llvm-mc -filetype=obj -triple=wasm32-unknown-unknown %t/main.s -o %t/main.o
+# RUN: llvm-mc -filetype=obj -triple=wasm32-unknown-unknown %t/a.s -o %t/a.o
+# RUN: llvm-mc -filetype=obj -triple=wasm32-unknown-unknown %t/a_b.s -o %t/a_b.o
+# RUN: llvm-mc -filetype=obj -triple=wasm32-unknown-unknown %t/b.s -o %t/b.o
+# RUN: llvm-ar rc %t/a.a %t/a.o
+# RUN: llvm-ar rc %t/a_b.a %t/a_b.o
+# RUN: llvm-ar rc %t/b.a %t/b.o
+# RUN: cd %t
+
+## Nothing is extracted from an archive. The file is created with just a header.
+# RUN: wasm-ld main.o a.o b.a -o /dev/null --why-extract=why1.txt
+# RUN: FileCheck %s --input-file=why1.txt --check-prefix=CHECK1 --match-full-lines --strict-whitespace
+
+#      CHECK1:reference	extracted	symbol
+#  CHECK1-NOT:{{.}}
+
+## Some archive members are extracted.
+# RUN: wasm-ld main.o a_b.a b.a -o /dev/null --why-extract=why2.txt
+# RUN: FileCheck %s --input-file=why2.txt --check-prefix=CHECK2 --match-full-lines --strict-whitespace
+
+#      CHECK2:reference	extracted	symbol
+# CHECK2-NEXT:main.o	a_b.a(a_b.o)	a
+# CHECK2-NEXT:a_b.a(a_b.o)	b.a(b.o)	b()
+
+## An undefined symbol error does not suppress the output.
+# RUN: not wasm-ld main.o a_b.a -o /dev/null --why-extract=why3.txt
+# RUN: FileCheck %s --input-file=why3.txt --check-prefix=CHECK3 --match-full-lines --strict-whitespace
+
+## Check that backward references are supported.
+## - means stdout.
+# RUN: wasm-ld b.a a_b.a main.o -o /dev/null --why-extract=- | FileCheck %s --check-prefix=CHECK4
+
+#      CHECK3:reference	extracted	symbol
+# CHECK3-NEXT:main.o	a_b.a(a_b.o)	a
+
+#      CHECK4:reference	extracted	symbol
+# CHECK4-NEXT:a_b.a(a_b.o)	b.a(b.o)	b()
+# CHECK4-NEXT:main.o	a_b.a(a_b.o)	a
+
+# RUN: wasm-ld main.o a_b.a b.a -o /dev/null --no-demangle --why-extract=- | FileCheck %s --check-prefix=MANGLED
+
+# MANGLED: a_b.a(a_b.o)	b.a(b.o)	_Z1bv
+
+# RUN: wasm-ld main.o a.a b.a -o /dev/null -u _Z1bv --why-extract=- | FileCheck %s --check-prefix=UNDEFINED
+
+## We insert -u symbol before processing other files, so its name is <internal>.
+## This is not ideal.
+# UNDEFINED: <internal>	b.a(b.o)	b()
+
+# RUN: wasm-ld main.o a.a b.a -o /dev/null -e _Z1bv --why-extract=- | FileCheck %s --check-prefix=ENTRY
+
+# ENTRY: --entry	b.a(b.o)	b()
+
+# SCRIPT: <internal>	b.a(b.o)	b()
+
+# RUN: not wasm-ld -shared main.o -o /dev/null --why-extract=/ 2>&1 | FileCheck %s --check-prefix=ERR
+
+# ERR: error: cannot open --why-extract= file /: {{.*}}
+
+#--- main.s
+.globl _start
+.functype a () -> ()
+_start:
+  .functype _start () -> ()
+  call a
+  end_function
+
+#--- a.s
+.globl a
+a:
+  .functype a () -> ()
+  end_function
+
+#--- a_b.s
+.functype _Z1bv () -> ()
+.globl a
+a:
+  .functype a () -> ()
+  call _Z1bv
+  end_function
+
+#--- b.s
+.globl _Z1bv
+_Z1bv:
+  .functype _Z1bv () -> ()
+  end_function
diff --git a/lld/wasm/Config.h b/lld/wasm/Config.h
index ea19a21172bb1..62cc14dc69865 100644
--- a/lld/wasm/Config.h
+++ b/lld/wasm/Config.h
@@ -20,8 +20,10 @@ namespace llvm::CodeGenOpt {
 enum Level : int;
 } // namespace llvm::CodeGenOpt
 
-namespace lld {
-namespace wasm {
+namespace lld::wasm {
+
+class InputFile;
+class Symbol;
 
 // For --unresolved-symbols.
 enum class UnresolvedPolicy { ReportError, Warn, Ignore, ImportDynamic };
@@ -81,6 +83,7 @@ struct Configuration {
   llvm::StringRef mapFile;
   llvm::StringRef outputFile;
   llvm::StringRef thinLTOCacheDir;
+  llvm::StringRef whyExtract;
 
   llvm::StringSet<> allowUndefinedSymbols;
   llvm::StringSet<> exportedSymbols;
@@ -92,7 +95,8 @@ struct Configuration {
   llvm::SmallVector<uint8_t, 0> buildIdVector;
 
   // The following config options do not directly correspond to any
-  // particular command line options.
+  // particular command line options, and should probably be moved to seperate
+  // Ctx struct as in ELF/Config.h
 
   // True if we are creating position-independent code.
   bool isPic;
@@ -110,12 +114,16 @@ struct Configuration {
   // Will be set to true if bss data segments should be emitted. In most cases
   // this is not necessary.
   bool emitBssSegments = false;
+
+  // A tuple of (reference, extractedFile, sym). Used by --why-extract=.
+  llvm::SmallVector<std::tuple<std::string, const InputFile *, const Symbol &>,
+                    0>
+      whyExtractRecords;
 };
 
 // The only instance of Configuration struct.
 extern Configuration *config;
 
-} // namespace wasm
-} // namespace lld
+} // namespace lld::wasm
 
 #endif
diff --git a/lld/wasm/Driver.cpp b/lld/wasm/Driver.cpp
index ea9299681d87f..e697f4b55ae6f 100644
--- a/lld/wasm/Driver.cpp
+++ b/lld/wasm/Driver.cpp
@@ -42,8 +42,7 @@ using namespace llvm::object;
 using namespace llvm::sys;
 using namespace llvm::wasm;
 
-namespace lld {
-namespace wasm {
+namespace lld::wasm {
 Configuration *config;
 
 namespace {
@@ -485,6 +484,7 @@ static void readConfigs(opt::InputArgList &args) {
       parseCachePruningPolicy(args.getLastArgValue(OPT_thinlto_cache_policy)),
       "--thinlto-cache-policy: invalid cache policy");
   config->unresolvedSymbols = getUnresolvedSymbolPolicy(args);
+  config->whyExtract = args.getLastArgValue(OPT_why_extract);
   errorHandler().verbose = args.hasArg(OPT_verbose);
   LLVM_DEBUG(errorHandler().verbose = true);
 
@@ -666,7 +666,7 @@ static const char *getReproduceOption(opt::InputArgList &args) {
 }
 
 // Force Sym to be entered in the output. Used for -u or equivalent.
-static Symbol *handleUndefined(StringRef name) {
+static Symbol *handleUndefined(StringRef name, const char *option) {
   Symbol *sym = symtab->find(name);
   if (!sym)
     return nullptr;
@@ -675,8 +675,11 @@ static Symbol *handleUndefined(StringRef name) {
   // eliminate it. Mark the symbol as "used" to prevent it.
   sym->isUsedInRegularObj = true;
 
-  if (auto *lazySym = dyn_cast<LazySymbol>(sym))
+  if (auto *lazySym = dyn_cast<LazySymbol>(sym)) {
     lazySym->fetch();
+    if (!config->whyExtract.empty())
+      config->whyExtractRecords.emplace_back(option, sym->getFile(), *sym);
+  }
 
   return sym;
 }
@@ -688,8 +691,31 @@ static void handleLibcall(StringRef name) {
 
   if (auto *lazySym = dyn_cast<LazySymbol>(sym)) {
     MemoryBufferRef mb = lazySym->getMemberBuffer();
-    if (isBitcode(mb))
+    if (isBitcode(mb)) {
+      if (!config->whyExtract.empty())
+        config->whyExtractRecords.emplace_back("<libcall>", sym->getFile(),
+                                               *sym);
       lazySym->fetch();
+    }
+  }
+}
+
+static void writeWhyExtract() {
+  if (config->whyExtract.empty())
+    return;
+
+  std::error_code ec;
+  raw_fd_ostream os(config->whyExtract, ec, sys::fs::OF_None);
+  if (ec) {
+    error("cannot open --why-extract= file " + config->whyExtract + ": " +
+          ec.message());
+    return;
+  }
+
+  os << "reference\textracted\tsymbol\n";
+  for (auto &entry : config->whyExtractRecords) {
+    os << std::get<0>(entry) << '\t' << toString(std::get<1>(entry)) << '\t'
+       << toString(std::get<2>(entry)) << '\n';
   }
 }
 
@@ -1070,16 +1096,16 @@ void LinkerDriver::linkerMain(ArrayRef<const char *> argsArr) {
 
   // Handle the `--undefined <sym>` options.
   for (auto *arg : args.filtered(OPT_undefined))
-    handleUndefined(arg->getValue());
+    handleUndefined(arg->getValue(), "<internal>");
 
   // Handle the `--export <sym>` options
   // This works like --undefined but also exports the symbol if its found
   for (auto &iter : config->exportedSymbols)
-    handleUndefined(iter.first());
+    handleUndefined(iter.first(), "--export");
 
   Symbol *entrySym = nullptr;
   if (!config->relocatable && !config->entry.empty()) {
-    entrySym = handleUndefined(config->entry);
+    entrySym = handleUndefined(config->entry, "--entry");
     if (entrySym && entrySym->isDefined())
       entrySym->forceExport = true;
     else
@@ -1096,7 +1122,8 @@ void LinkerDriver::linkerMain(ArrayRef<const char *> argsArr) {
       !WasmSym::callCtors->isUsedInRegularObj &&
       WasmSym::callCtors->getName() != config->entry &&
       !config->exportedSymbols.count(WasmSym::callCtors->getName())) {
-    if (Symbol *callDtors = handleUndefined("__wasm_call_dtors")) {
+    if (Symbol *callDtors =
+            handleUndefined("__wasm_call_dtors", "<internal>")) {
       if (auto *callDtorsFunc = dyn_cast<DefinedFunction>(callDtors)) {
         if (callDtorsFunc->signature &&
             (!callDtorsFunc->signature->Params.empty() ||
@@ -1131,6 +1158,8 @@ void LinkerDriver::linkerMain(ArrayRef<const char *> argsArr) {
   if (errorCount())
     return;
 
+  writeWhyExtract();
+
   // Do link-time optimization if given files are LLVM bitcode files.
   // This compiles bitcode files into real object files.
   symtab->compileBitcodeFiles();
@@ -1188,5 +1217,4 @@ void LinkerDriver::linkerMain(ArrayRef<const char *> argsArr) {
   writeResult();
 }
 
-} // namespace wasm
-} // namespace lld
+} // namespace wasm::lld
diff --git a/lld/wasm/InputFiles.cpp b/lld/wasm/InputFiles.cpp
index e8a3701232fb1..75760293bbaae 100644
--- a/lld/wasm/InputFiles.cpp
+++ b/lld/wasm/InputFiles.cpp
@@ -441,7 +441,7 @@ void ObjFile::parse(bool ignoreComdats) {
   // called directly (i.e. only address taken) don't have to match the defined
   // function's signature.  We cannot do this for directly called functions
   // because those signatures are checked at validation times.
-  // See https://bugs.llvm.org/show_bug.cgi?id=40412
+  // See https://github.com/llvm/llvm-project/issues/39758
   std::vector<bool> isCalledDirectly(wasmObj->getNumberOfSymbols(), false);
   for (const SectionRef &sec : wasmObj->sections()) {
     const WasmSection &section = wasmObj->getWasmSection(sec);
@@ -791,7 +791,8 @@ void BitcodeFile::parse() {
   }
   checkArch(t.getArch());
   std::vector<bool> keptComdats;
-  // TODO Support nodeduplicate https://bugs.llvm.org/show_bug.cgi?id=50531
+  // TODO Support nodeduplicate
+  // https://github.com/llvm/llvm-project/issues/49875
   for (std::pair<StringRef, Comdat::SelectionKind> s : obj->getComdatTable())
     keptComdats.push_back(symtab->addComdat(s.first));
 
diff --git a/lld/wasm/LTO.cpp b/lld/wasm/LTO.cpp
index d5ec3f70a1dfe..4b22e4ccc1dc3 100644
--- a/lld/wasm/LTO.cpp
+++ b/lld/wasm/LTO.cpp
@@ -37,8 +37,7 @@
 
 using namespace llvm;
 
-namespace lld {
-namespace wasm {
+namespace lld::wasm {
 static std::unique_ptr<lto::LTO> createLTO() {
   lto::Config c;
   c.Options = initTargetOptionsFromCodeGenFlags();
@@ -165,5 +164,4 @@ std::vector<StringRef> BitcodeCompiler::compile() {
   return ret;
 }
 
-} // namespace wasm
-} // namespace lld
+} // namespace lld::wasm
diff --git a/lld/wasm/LTO.h b/lld/wasm/LTO.h
index 9c1e1788ed385..bb57c6651394b 100644
--- a/lld/wasm/LTO.h
+++ b/lld/wasm/LTO.h
@@ -32,8 +32,7 @@ class LTO;
 }
 } // namespace llvm
 
-namespace lld {
-namespace wasm {
+namespace lld::wasm {
 
 class BitcodeFile;
 class InputFile;
@@ -51,7 +50,6 @@ class BitcodeCompiler {
   std::vector<SmallString<0>> buf;
   std::vector<std::unique_ptr<MemoryBuffer>> files;
 };
-} // namespace wasm
-} // namespace lld
+} // namespace lld::wasm
 
 #endif
diff --git a/lld/wasm/MapFile.h b/lld/wasm/MapFile.h
index ef2cc783a6c2c..c590d23238e43 100644
--- a/lld/wasm/MapFile.h
+++ b/lld/wasm/MapFile.h
@@ -11,11 +11,9 @@
 
 #include "llvm/ADT/ArrayRef.h"
 
-namespace lld {
-namespace wasm {
+namespace lld::wasm {
 class OutputSection;
 void writeMapFile(llvm::ArrayRef<OutputSection *> outputSections);
-} // namespace wasm
-} // namespace lld
+} // namespace lld::wasm
 
 #endif
diff --git a/lld/wasm/MarkLive.cpp b/lld/wasm/MarkLive.cpp
index 2b8c8e17787b8..a59a80ad2cc3a 100644
--- a/lld/wasm/MarkLive.cpp
+++ b/lld/wasm/MarkLive.cpp
@@ -30,8 +30,7 @@
 using namespace llvm;
 using namespace llvm::wasm;
 
-namespace lld {
-namespace wasm {
+namespace lld::wasm {
 
 namespace {
 
@@ -205,5 +204,4 @@ bool MarkLive::isCallCtorsLive() {
   return false;
 }
 
-} // namespace wasm
-} // namespace lld
+} // namespace lld::wasm
diff --git a/lld/wasm/MarkLive.h b/lld/wasm/MarkLive.h
index 7b88cb4859f8a..e788cc5fc132d 100644
--- a/lld/wasm/MarkLive.h
+++ b/lld/wasm/MarkLive.h
@@ -9,12 +9,10 @@
 #ifndef LLD_WASM_MARKLIVE_H
 #define LLD_WASM_MARKLIVE_H
 
-namespace lld {
-namespace wasm {
+namespace lld::wasm {
 
 void markLive();
 
-} // namespace wasm
-} // namespace lld
+} // namespace lld::wasm
 
 #endif // LLD_WASM_MARKLIVE_H
diff --git a/lld/wasm/Options.td b/lld/wasm/Options.td
index 5fe1e7e5c55e7..9670e5fb00847 100644
--- a/lld/wasm/Options.td
+++ b/lld/wasm/Options.td
@@ -226,6 +226,8 @@ defm whole_archive: B<"whole-archive",
     "Force load of all members in a static library",
     "Do not force load of all members in a static library (default)">;
 
+def why_extract: JJ<"why-extract=">, HelpText<"Print to a file about why archive members are extracted">;
+
 defm check_features: BB<"check-features",
     "Check feature compatibility of linked objects (default)",
     "Ignore feature compatibility of linked objects">;
diff --git a/lld/wasm/OutputSegment.cpp b/lld/wasm/OutputSegment.cpp
index 86b538403391b..ea052e89dbb55 100644
--- a/lld/wasm/OutputSegment.cpp
+++ b/lld/wasm/OutputSegment.cpp
@@ -15,9 +15,7 @@
 using namespace llvm;
 using namespace llvm::wasm;
 
-namespace lld {
-
-namespace wasm {
+namespace lld::wasm {
 
 void OutputSegment::addInputSegment(InputChunk *inSeg) {
   alignment = std::max(alignment, inSeg->alignment);
@@ -82,5 +80,4 @@ void OutputSegment::finalizeInputSegments() {
   }
 }
 
-} // namespace wasm
-} // namespace lld
+} // namespace lld::wasm
diff --git a/lld/wasm/OutputSegment.h b/lld/wasm/OutputSegment.h
index 3b7a0f50be94e..e1e43c1f9a36a 100644
--- a/lld/wasm/OutputSegment.h
+++ b/lld/wasm/OutputSegment.h
@@ -13,8 +13,7 @@
 #include "lld/Common/ErrorHandler.h"
 #include "llvm/Object/Wasm.h"
 
-namespace lld {
-namespace wasm {
+namespace lld::wasm {
 
 class InputSegment;
 
@@ -49,7 +48,6 @@ class OutputSegment {
   std::string header;
 };
 
-} // namespace wasm
-} // namespace lld
+} // namespace lld::wasm
 
 #endif // LLD_WASM_OUTPUT_SEGMENT_H
diff --git a/lld/wasm/Relocations.cpp b/lld/wasm/Relocations.cpp
index c7710a9baf320..2f6dd6af2d030 100644
--- a/lld/wasm/Relocations.cpp
+++ b/lld/wasm/Relocations.cpp
@@ -16,8 +16,7 @@
 using namespace llvm;
 using namespace llvm::wasm;
 
-namespace lld {
-namespace wasm {
+namespace lld::wasm {
 
 static bool requiresGOTAccess(const Symbol *sym) {
   if (!config->isPic &&
@@ -176,5 +175,4 @@ void scanRelocations(InputChunk *chunk) {
   }
 }
 
-} // namespace wasm
-} // namespace lld
+} // namespace lld::wasm
diff --git a/lld/wasm/Relocations.h b/lld/wasm/Relocations.h
index 02787f551d804..e493f363dab70 100644
--- a/lld/wasm/Relocations.h
+++ b/lld/wasm/Relocations.h
@@ -9,14 +9,12 @@
 #ifndef LLD_WASM_RELOCATIONS_H
 #define LLD_WASM_RELOCATIONS_H
 
-namespace lld {
-namespace wasm {
+namespace lld::wasm {
 
 class InputChunk;
 
 void scanRelocations(InputChunk *chunk);
 
-} // namespace wasm
-} // namespace lld
+} // namespace lld::wasm
 
 #endif
diff --git a/lld/wasm/SymbolTable.cpp b/lld/wasm/SymbolTable.cpp
index d14e9d5c010f1..881b1231ffdf9 100644
--- a/lld/wasm/SymbolTable.cpp
+++ b/lld/wasm/SymbolTable.cpp
@@ -20,8 +20,7 @@ using namespace llvm;
 using namespace llvm::wasm;
 using namespace llvm::object;
 
-namespace lld {
-namespace wasm {
+namespace lld::wasm {
 SymbolTable *symtab;
 
 void SymbolTable::addFile(InputFile *file) {
@@ -524,6 +523,9 @@ Symbol *SymbolTable::addUndefinedFunction(StringRef name,
       lazy->signature = sig;
     } else {
       lazy->fetch();
+      if (!config->whyExtract.empty())
+        config->whyExtractRecords.emplace_back(toString(file), s->getFile(),
+                                               *s);
     }
   } else {
     auto existingFunction = dyn_cast<FunctionSymbol>(s);
@@ -758,7 +760,10 @@ void SymbolTable::addLazy(ArchiveFile *file, const Archive::Symbol *sym) {
   }
 
   LLVM_DEBUG(dbgs() << "replacing existing undefined\n");
+  const InputFile *oldFile = s->getFile();
   file->addMember(sym);
+  if (!config->whyExtract.empty())
+    config->whyExtractRecords.emplace_back(toString(oldFile), s->getFile(), *s);
 }
 
 bool SymbolTable::addComdat(StringRef name) {
@@ -960,5 +965,4 @@ void SymbolTable::handleSymbolVariants() {
   }
 }
 
-} // namespace wasm
-} // namespace lld
+} // namespace wasm::lld
diff --git a/lld/wasm/SymbolTable.h b/lld/wasm/SymbolTable.h
index f624b8bdfd86a..5009e6039602b 100644
--- a/lld/wasm/SymbolTable.h
+++ b/lld/wasm/SymbolTable.h
@@ -18,8 +18,7 @@
 #include "llvm/BinaryFormat/WasmTraits.h"
 #include <optional>
 
-namespace lld {
-namespace wasm {
+namespace lld::wasm {
 
 class InputSegment;
 
@@ -144,7 +143,6 @@ class SymbolTable {
 
 extern SymbolTable *symtab;
 
-} // namespace wasm
-} // namespace lld
+} // namespace lld::wasm
 
 #endif
diff --git a/lld/wasm/SyntheticSections.cpp b/lld/wasm/SyntheticSections.cpp
index 85d1a99e88b5a..a1fd6bcf99017 100644
--- a/lld/wasm/SyntheticSections.cpp
+++ b/lld/wasm/SyntheticSections.cpp
@@ -22,8 +22,7 @@
 using namespace llvm;
 using namespace llvm::wasm;
 
-namespace lld {
-namespace wasm {
+namespace lld::wasm {
 
 OutStruct out;
 
@@ -900,6 +899,7 @@ static size_t getHashSize() {
   case BuildIdKind::None:
     return 0;
   }
+  llvm_unreachable("build id kind not implemented");
 }
 
 BuildIdSection::BuildIdSection()
@@ -922,5 +922,4 @@ void BuildIdSection::writeBuildId(llvm::ArrayRef<uint8_t> buf) {
   memcpy(hashPlaceholderPtr, buf.data(), hashSize);
 }
 
-} // namespace wasm
-} // namespace lld
+} // namespace wasm::lld
diff --git a/lld/wasm/SyntheticSections.h b/lld/wasm/SyntheticSections.h
index f4e990429bb9c..dd63931b974b0 100644
--- a/lld/wasm/SyntheticSections.h
+++ b/lld/wasm/SyntheticSections.h
@@ -24,8 +24,7 @@
 
 #define DEBUG_TYPE "lld"
 
-namespace lld {
-namespace wasm {
+namespace lld::wasm {
 
 // An init entry to be written to either the synthetic init func or the
 // linking metadata.
@@ -127,8 +126,7 @@ inline bool operator==(const ImportKey<T> &lhs, const ImportKey<T> &rhs) {
          lhs.importName == rhs.importName && lhs.type == rhs.type;
 }
 
-} // namespace wasm
-} // namespace lld
+} // namespace wasm::lld
 
 // `ImportKey<T>` can be used as a key in a `DenseMap` if `T` can be used as a
 // key in a `DenseMap`.
diff --git a/lld/wasm/Writer.cpp b/lld/wasm/Writer.cpp
index e82d70c9678d9..030ef7468791a 100644
--- a/lld/wasm/Writer.cpp
+++ b/lld/wasm/Writer.cpp
@@ -45,8 +45,7 @@
 using namespace llvm;
 using namespace llvm::wasm;
 
-namespace lld {
-namespace wasm {
+namespace lld::wasm {
 static constexpr int stackAlignment = 16;
 static constexpr int heapAlignment = 16;
 
@@ -568,7 +567,7 @@ void Writer::populateTargetFeatures() {
   if (config->isPic) {
     // This should not be necessary because all PIC objects should
     // contain the mutable-globals feature.
-    // TODO(https://bugs.llvm.org/show_bug.cgi?id=52339)
+    // TODO (https://github.com/llvm/llvm-project/issues/51681)
     allowed.insert("mutable-globals");
   }
 
@@ -1865,5 +1864,4 @@ void Writer::createHeader() {
 
 void writeResult() { Writer().run(); }
 
-} // namespace wasm
-} // namespace lld
+} // namespace wasm::lld
diff --git a/lld/wasm/Writer.h b/lld/wasm/Writer.h
index d83e4cbda628a..cd32ecffb37dc 100644
--- a/lld/wasm/Writer.h
+++ b/lld/wasm/Writer.h
@@ -9,12 +9,10 @@
 #ifndef LLD_WASM_WRITER_H
 #define LLD_WASM_WRITER_H
 
-namespace lld {
-namespace wasm {
+namespace lld::wasm {
 
 void writeResult();
 
-} // namespace wasm
-} // namespace lld
+} // namespace lld::wasm
 
 #endif
diff --git a/lldb/bindings/python/python-typemaps.swig b/lldb/bindings/python/python-typemaps.swig
index 1b1f2e1fe2664..b3e7dd4ee265a 100644
--- a/lldb/bindings/python/python-typemaps.swig
+++ b/lldb/bindings/python/python-typemaps.swig
@@ -384,6 +384,29 @@ template <> bool SetNumberFromPyObject<double>(double &number, PyObject *obj) {
   $1 = $1 || PyCallable_Check(reinterpret_cast<PyObject *>($input));
 }
 
+// For lldb::SBDebuggerDestroyCallback
+%typemap(in) (lldb::SBDebuggerDestroyCallback destroy_callback, void *baton) {
+  if (!($input == Py_None ||
+        PyCallable_Check(reinterpret_cast<PyObject *>($input)))) {
+    PyErr_SetString(PyExc_TypeError, "Need a callable object or None!");
+    SWIG_fail;
+  }
+
+  // FIXME (filcab): We can't currently check if our callback is already
+  // LLDBSwigPythonCallPythonSBDebuggerTerminateCallback (to DECREF the previous
+  // baton) nor can we just remove all traces of a callback, if we want to
+  // revert to a file logging mechanism.
+
+  // Don't lose the callback reference
+  Py_INCREF($input);
+  $1 = LLDBSwigPythonCallPythonSBDebuggerTerminateCallback;
+  $2 = $input;
+}
+
+%typemap(typecheck) (lldb::SBDebuggerDestroyCallback destroy_callback, void *baton) {
+  $1 = $input == Py_None;
+  $1 = $1 || PyCallable_Check(reinterpret_cast<PyObject *>($input));
+}
 
 %typemap(in) lldb::FileSP {
   PythonFile py_file(PyRefType::Borrowed, $input);
diff --git a/lldb/bindings/python/python-wrapper.swig b/lldb/bindings/python/python-wrapper.swig
index 26f79266538a5..d82c2dace18f3 100644
--- a/lldb/bindings/python/python-wrapper.swig
+++ b/lldb/bindings/python/python-wrapper.swig
@@ -1062,4 +1062,16 @@ static void LLDBSwigPythonCallPythonLogOutputCallback(const char *str,
     SWIG_PYTHON_THREAD_END_BLOCK;
   }
 }
+
+// For DebuggerTerminateCallback functions
+static void LLDBSwigPythonCallPythonSBDebuggerTerminateCallback(lldb::user_id_t debugger_id,
+                                                      void *baton) {
+  if (baton != Py_None) {
+    SWIG_PYTHON_THREAD_BEGIN_BLOCK;
+    PyObject *result = PyObject_CallFunction(
+        reinterpret_cast<PyObject *>(baton), const_cast<char *>("l"), debugger_id);
+    Py_XDECREF(result);
+    SWIG_PYTHON_THREAD_END_BLOCK;
+  }
+}
 %}
diff --git a/lldb/docs/index.rst b/lldb/docs/index.rst
index 02d446573900a..ad09129ee32b3 100644
--- a/lldb/docs/index.rst
+++ b/lldb/docs/index.rst
@@ -72,8 +72,7 @@ are welcome:
 * macOS debugging for i386, x86_64 and AArch64
 * iOS, tvOS, and watchOS simulator debugging on i386, x86_64 and AArch64
 * iOS, tvOS, and watchOS device debugging on ARM and AArch64
-* Linux user-space debugging for i386, x86_64, ARM, AArch64, MIPS64, PPC64le,
-  s390x
+* Linux user-space debugging for i386, x86_64, ARM, AArch64, PPC64le, s390x
 * FreeBSD user-space debugging for i386, x86_64, ARM, AArch64, MIPS64, PPC
 * NetBSD user-space debugging for i386 and x86_64
 * Windows user-space debugging for i386, x86_64, ARM and AArch64 (*)
diff --git a/lldb/docs/status/status.rst b/lldb/docs/status/status.rst
index 5ad19b898b97b..a046c0cf8cae6 100644
--- a/lldb/docs/status/status.rst
+++ b/lldb/docs/status/status.rst
@@ -11,8 +11,8 @@ Linux
 -----
 
 LLDB is improving on Linux. Linux is nearing feature completeness with Darwin
-to debug x86_64, i386, ARM, AArch64, IBM POWER (ppc64), IBM Z (s390x), and
-MIPS64 programs. For more details, see the Features by OS section below.
+to debug x86_64, i386, ARM, AArch64, IBM POWER (ppc64), and IBM Z (s390x)
+programs. For more details, see the Features by OS section below.
 
 macOS
 -----
diff --git a/lldb/include/lldb/API/SBCommandInterpreter.h b/lldb/include/lldb/API/SBCommandInterpreter.h
index 08ed989714d13..7c3f8275471e8 100644
--- a/lldb/include/lldb/API/SBCommandInterpreter.h
+++ b/lldb/include/lldb/API/SBCommandInterpreter.h
@@ -241,7 +241,20 @@ class SBCommandInterpreter {
                                        lldb::SBStringList &matches,
                                        lldb::SBStringList &descriptions);
 
+  /// Returns whether an interrupt flag was raised either by the SBDebugger - 
+  /// when the function is not running on the RunCommandInterpreter thread, or
+  /// by SBCommandInterpreter::InterruptCommand if it is.  If your code is doing
+  /// interruptible work, check this API periodically, and interrupt if it 
+  /// returns true.
   bool WasInterrupted() const;
+  
+  /// Interrupts the command currently executing in the RunCommandInterpreter
+  /// thread.
+  ///
+  /// \return
+  ///   \b true if there was a command in progress to recieve the interrupt.
+  ///   \b false if there's no command currently in flight.
+  bool InterruptCommand();
 
   // Catch commands before they execute by registering a callback that will get
   // called when the command gets executed. This allows GUI or command line
diff --git a/lldb/include/lldb/API/SBDebugger.h b/lldb/include/lldb/API/SBDebugger.h
index 80f76480363f3..bdcd5eeaaffdb 100644
--- a/lldb/include/lldb/API/SBDebugger.h
+++ b/lldb/include/lldb/API/SBDebugger.h
@@ -197,6 +197,10 @@ class LLDB_API SBDebugger {
   lldb::SBCommandInterpreter GetCommandInterpreter();
 
   void HandleCommand(const char *command);
+  
+  void RequestInterrupt();
+  void CancelInterruptRequest();
+  bool InterruptRequested();
 
   lldb::SBListener GetListener();
 
@@ -316,6 +320,9 @@ class LLDB_API SBDebugger {
 
   void SetLoggingCallback(lldb::LogOutputCallback log_callback, void *baton);
 
+  void SetDestroyCallback(lldb::SBDebuggerDestroyCallback destroy_callback,
+                          void *baton);
+
   // DEPRECATED
 #ifndef SWIG
   void DispatchInput(void *baton, const void *data, size_t data_len);
diff --git a/lldb/include/lldb/API/SBDefines.h b/lldb/include/lldb/API/SBDefines.h
index 848480d4f5e01..e7f8ca7b35f0d 100644
--- a/lldb/include/lldb/API/SBDefines.h
+++ b/lldb/include/lldb/API/SBDefines.h
@@ -110,6 +110,10 @@ class LLDB_API SBUnixSignals;
 typedef bool (*SBBreakpointHitCallback)(void *baton, SBProcess &process,
                                         SBThread &thread,
                                         lldb::SBBreakpointLocation &location);
+
+typedef void (*SBDebuggerDestroyCallback)(lldb::user_id_t debugger_id,
+                                          void *baton);
+
 typedef void *ScriptedObject;
 }
 
diff --git a/lldb/include/lldb/Core/Debugger.h b/lldb/include/lldb/Core/Debugger.h
index 9f83a37bc75cc..1d9d691bc2b06 100644
--- a/lldb/include/lldb/Core/Debugger.h
+++ b/lldb/include/lldb/Core/Debugger.h
@@ -28,6 +28,7 @@
 #include "lldb/Target/TargetList.h"
 #include "lldb/Utility/Broadcaster.h"
 #include "lldb/Utility/ConstString.h"
+#include "lldb/Utility/Diagnostics.h"
 #include "lldb/Utility/FileSpec.h"
 #include "lldb/Utility/Status.h"
 #include "lldb/Utility/UserID.h"
@@ -370,6 +371,48 @@ class Debugger : public std::enable_shared_from_this<Debugger>,
   bool IsHandlingEvents() const { return m_event_handler_thread.IsJoinable(); }
 
   Status RunREPL(lldb::LanguageType language, const char *repl_options);
+  
+  /// Interruption in LLDB:
+  /// 
+  /// This is a voluntary interruption mechanism, not preemptive.  Parts of lldb
+  /// that do work that can be safely interrupted call 
+  /// Debugger::InterruptRequested and if that returns true, they should return
+  /// at a safe point, shortcutting the rest of the work they were to do.
+  ///  
+  /// lldb clients can both offer a CommandInterpreter (through 
+  /// RunCommandInterpreter) and use the SB API's for their own purposes, so it
+  /// is convenient to separate "interrupting the CommandInterpreter execution"
+  /// and interrupting the work it is doing with the SB API's.  So there are two 
+  /// ways to cause an interrupt: 
+  ///   * CommandInterpreter::InterruptCommand: Interrupts the command currently
+  ///     running in the command interpreter IOHandler thread
+  ///   * Debugger::RequestInterrupt: Interrupts are active on anything but the
+  ///     CommandInterpreter thread till CancelInterruptRequest is called.
+  ///
+  /// Since the two checks are mutually exclusive, however, it's also convenient
+  /// to have just one function to check the interrupt state.
+
+
+  /// Bump the "interrupt requested" count on the debugger to support
+  /// cooperative interruption.  If this is non-zero, InterruptRequested will
+  /// return true.  Interruptible operations are expected to query the
+  /// InterruptRequested API periodically, and interrupt what they were doing
+  /// if it returns \b true.
+  ///
+  void RequestInterrupt();
+  
+  /// Decrement the "interrupt requested" counter.
+  void CancelInterruptRequest();
+  
+  /// This is the correct way to query the state of Interruption.
+  /// If you are on the RunCommandInterpreter thread, it will check the 
+  /// command interpreter state, and if it is on another thread it will 
+  /// check the debugger Interrupt Request state.
+  ///
+  /// \return
+  ///  A boolean value, if \b true an interruptible operation should interrupt
+  ///  itself.
+  bool InterruptRequested();
 
   // This is for use in the command interpreter, when you either want the
   // selected target, or if no target is present you want to prime the dummy
@@ -450,6 +493,10 @@ class Debugger : public std::enable_shared_from_this<Debugger>,
 
   static void ReportSymbolChange(const ModuleSpec &module_spec);
 
+  void
+  SetDestroyCallback(lldb_private::DebuggerDestroyCallback destroy_callback,
+                     void *baton);
+
 protected:
   friend class CommandInterpreter;
   friend class REPL;
@@ -494,6 +541,8 @@ class Debugger : public std::enable_shared_from_this<Debugger>,
                                    std::optional<lldb::user_id_t> debugger_id,
                                    std::once_flag *once);
 
+  void HandleDestroyCallback();
+
   void PrintProgress(const ProgressEventData &data);
 
   bool StartEventHandlerThread();
@@ -505,13 +554,19 @@ class Debugger : public std::enable_shared_from_this<Debugger>,
 
   bool PopIOHandler(const lldb::IOHandlerSP &reader_sp);
 
-  bool HasIOHandlerThread();
+  bool HasIOHandlerThread() const;
 
   bool StartIOHandlerThread();
 
   void StopIOHandlerThread();
+  
+  // Sets the IOHandler thread to the new_thread, and returns
+  // the previous IOHandler thread.
+  HostThread SetIOHandlerThread(HostThread &new_thread);
 
   void JoinIOHandlerThread();
+  
+  bool IsIOHandlerThreadCurrentThread() const;
 
   lldb::thread_result_t IOHandlerThread();
 
@@ -590,6 +645,13 @@ class Debugger : public std::enable_shared_from_this<Debugger>,
   lldb::ListenerSP m_forward_listener_sp;
   llvm::once_flag m_clear_once;
   lldb::TargetSP m_dummy_target_sp;
+  Diagnostics::CallbackID m_diagnostics_callback_id;
+
+  lldb_private::DebuggerDestroyCallback m_destroy_callback = nullptr;
+  void *m_destroy_callback_baton = nullptr;
+
+  uint32_t m_interrupt_requested = 0; ///< Tracks interrupt requests
+  std::mutex m_interrupt_mutex;
 
   // Events for m_sync_broadcaster
   enum {
diff --git a/lldb/include/lldb/Interpreter/CommandInterpreter.h b/lldb/include/lldb/Interpreter/CommandInterpreter.h
index 38ff061612b11..cf44412b13026 100644
--- a/lldb/include/lldb/Interpreter/CommandInterpreter.h
+++ b/lldb/include/lldb/Interpreter/CommandInterpreter.h
@@ -352,9 +352,10 @@ class CommandInterpreter : public Broadcaster,
                      CommandReturnObject &result);
 
   bool HandleCommand(const char *command_line, LazyBool add_to_history,
-                     CommandReturnObject &result);
+                     CommandReturnObject &result,
+                     bool force_repeat_command = false);
 
-  bool WasInterrupted() const;
+  bool InterruptCommand();
 
   /// Execute a list of commands in sequence.
   ///
@@ -639,6 +640,10 @@ class CommandInterpreter : public Broadcaster,
 protected:
   friend class Debugger;
 
+  // This checks just the RunCommandInterpreter interruption state.  It is only
+  // meant to be used in Debugger::InterruptRequested
+  bool WasInterrupted() const;
+
   // IOHandlerDelegate functions
   void IOHandlerInputComplete(IOHandler &io_handler,
                               std::string &line) override;
@@ -701,7 +706,6 @@ class CommandInterpreter : public Broadcaster,
 
   void StartHandlingCommand();
   void FinishHandlingCommand();
-  bool InterruptCommand();
 
   Debugger &m_debugger; // The debugger session that this interpreter is
                         // associated with
diff --git a/lldb/include/lldb/Interpreter/ScriptedInterface.h b/lldb/include/lldb/Interpreter/ScriptedInterface.h
index 8785e31f7b932..ab61425932643 100644
--- a/lldb/include/lldb/Interpreter/ScriptedInterface.h
+++ b/lldb/include/lldb/Interpreter/ScriptedInterface.h
@@ -40,12 +40,16 @@ class ScriptedInterface {
                               LLDBLog log_caterogy = LLDBLog::Process) {
     LLDB_LOGF(GetLog(log_caterogy), "%s ERROR = %s", caller_name.data(),
               error_msg.data());
-    llvm::Twine err = llvm::Twine(caller_name + llvm::Twine(" ERROR = ") +
-                                  llvm::Twine(error_msg));
+    std::string full_error_message =
+        llvm::Twine(caller_name + llvm::Twine(" ERROR = ") +
+                    llvm::Twine(error_msg))
+            .str();
     if (const char *detailed_error = error.AsCString())
-      err.concat(llvm::Twine(" (") + llvm::Twine(detailed_error) +
-                 llvm::Twine(")"));
-    error.SetErrorString(err.str());
+      full_error_message +=
+          llvm::Twine(llvm::Twine(" (") + llvm::Twine(detailed_error) +
+                      llvm::Twine(")"))
+              .str();
+    error.SetErrorString(full_error_message);
     return {};
   }
 
diff --git a/lldb/include/lldb/Symbol/CompilerDeclContext.h b/lldb/include/lldb/Symbol/CompilerDeclContext.h
index ca404a6641d5e..63e5f7b680e63 100644
--- a/lldb/include/lldb/Symbol/CompilerDeclContext.h
+++ b/lldb/include/lldb/Symbol/CompilerDeclContext.h
@@ -61,15 +61,21 @@ class CompilerDeclContext {
 
   /// Checks if this decl context represents a method of a class.
   ///
-  /// \param[out] language_object_name_ptr
-  ///     If non NULL and \b true is returned from this function,
-  ///     this will indicate if implicit object name for the language
-  ///     like "this" for C++, and "self" for Objective C.
-  ///
   /// \return
   ///     Returns true if this is a decl context that represents a method
   ///     in a struct, union or class.
-  bool IsClassMethod(ConstString *language_object_name_ptr = nullptr);
+  bool IsClassMethod();
+
+  /// Determines the original language of the decl context.
+  lldb::LanguageType GetLanguage();
+
+  /// Determines the name of the instance variable for the this decl context.
+  ///
+  /// For C++ the name is "this", for Objective-C the name is "self".
+  ///
+  /// \return
+  ///     Returns a string for the name of the instance variable.
+  ConstString GetInstanceVariableName(lldb::LanguageType language);
 
   /// Check if the given other decl context is contained in the lookup
   /// of this decl context (for example because the other context is a nested
diff --git a/lldb/include/lldb/Symbol/Symbol.h b/lldb/include/lldb/Symbol/Symbol.h
index 164fafd437dcb..44a2d560010fe 100644
--- a/lldb/include/lldb/Symbol/Symbol.h
+++ b/lldb/include/lldb/Symbol/Symbol.h
@@ -11,12 +11,23 @@
 
 #include "lldb/Core/AddressRange.h"
 #include "lldb/Core/Mangled.h"
+#include "lldb/Core/Section.h"
 #include "lldb/Symbol/SymbolContextScope.h"
 #include "lldb/Utility/UserID.h"
 #include "lldb/lldb-private.h"
+#include "llvm/Support/JSON.h"
 
 namespace lldb_private {
 
+struct JSONSymbol {
+  std::optional<uint64_t> address;
+  std::optional<uint64_t> value;
+  std::optional<uint64_t> size;
+  std::optional<uint64_t> id;
+  std::optional<lldb::SymbolType> type;
+  std::string name;
+};
+
 class Symbol : public SymbolContextScope {
 public:
   // ObjectFile readers can classify their symbol table entries and searches
@@ -39,6 +50,9 @@ class Symbol : public SymbolContextScope {
 
   const Symbol &operator=(const Symbol &rhs);
 
+  static llvm::Expected<Symbol> FromJSON(const JSONSymbol &symbol,
+                                         SectionList *section_list);
+
   void Clear();
 
   bool Compare(ConstString name, lldb::SymbolType type) const;
@@ -187,7 +201,7 @@ class Symbol : public SymbolContextScope {
 
   bool IsWeak() const { return m_is_weak; }
 
-  void SetIsWeak (bool b) { m_is_weak = b; }
+  void SetIsWeak(bool b) { m_is_weak = b; }
 
   bool GetByteSizeIsValid() const { return m_size_is_valid; }
 
@@ -332,4 +346,16 @@ class Symbol : public SymbolContextScope {
 
 } // namespace lldb_private
 
+namespace llvm {
+namespace json {
+
+bool fromJSON(const llvm::json::Value &value, lldb_private::JSONSymbol &symbol,
+              llvm::json::Path path);
+
+bool fromJSON(const llvm::json::Value &value, lldb::SymbolType &type,
+              llvm::json::Path path);
+
+} // namespace json
+} // namespace llvm
+
 #endif // LLDB_SYMBOL_SYMBOL_H
diff --git a/lldb/include/lldb/Symbol/SymbolContext.h b/lldb/include/lldb/Symbol/SymbolContext.h
index bb9e031daaaa4..73fa25514aff3 100644
--- a/lldb/include/lldb/Symbol/SymbolContext.h
+++ b/lldb/include/lldb/Symbol/SymbolContext.h
@@ -245,17 +245,13 @@ class SymbolContext {
   ///     represented by this symbol context object, nullptr otherwise.
   Block *GetFunctionBlock();
 
-  /// If this symbol context represents a function that is a method, return
-  /// true and provide information about the method.
+  /// Determines the name of the instance variable for the this decl context.
   ///
-  /// \param[out] language_object_name
-  ///     If \b true is returned, the name of the artificial variable
-  ///     for the language ("this" for C++, "self" for ObjC).
+  /// For C++ the name is "this", for Objective-C the name is "self".
   ///
   /// \return
-  ///     \b True if this symbol context represents a function that
-  ///     is a method of a class, \b false otherwise.
-  bool GetFunctionMethodInfo(ConstString &language_object_name);
+  ///     Returns a string for the name of the instance variable.
+  ConstString GetInstanceVariableName();
 
   /// Sorts the types in TypeMap according to SymbolContext to TypeList
   ///
diff --git a/lldb/include/lldb/Symbol/SymbolFile.h b/lldb/include/lldb/Symbol/SymbolFile.h
index 8548973bad3ff..d3e3166ab27ff 100644
--- a/lldb/include/lldb/Symbol/SymbolFile.h
+++ b/lldb/include/lldb/Symbol/SymbolFile.h
@@ -25,6 +25,7 @@
 #include "lldb/Utility/XcodeSDK.h"
 #include "lldb/lldb-private.h"
 #include "llvm/ADT/DenseSet.h"
+#include "llvm/ADT/SmallSet.h"
 #include "llvm/Support/Errc.h"
 
 #include <mutex>
@@ -146,6 +147,17 @@ class SymbolFile : public PluginInterface {
   virtual lldb::LanguageType ParseLanguage(CompileUnit &comp_unit) = 0;
   /// Return the Xcode SDK comp_unit was compiled against.
   virtual XcodeSDK ParseXcodeSDK(CompileUnit &comp_unit) { return {}; }
+
+  /// This function exists because SymbolFileDWARFDebugMap may extra compile
+  /// units which aren't exposed as "real" compile units. In every other
+  /// case this function should behave identically as ParseLanguage.
+  virtual llvm::SmallSet<lldb::LanguageType, 4>
+  ParseAllLanguages(CompileUnit &comp_unit) {
+    llvm::SmallSet<lldb::LanguageType, 4> langs;
+    langs.insert(ParseLanguage(comp_unit));
+    return langs;
+  }
+
   virtual size_t ParseFunctions(CompileUnit &comp_unit) = 0;
   virtual bool ParseLineTable(CompileUnit &comp_unit) = 0;
   virtual bool ParseDebugMacros(CompileUnit &comp_unit) = 0;
diff --git a/lldb/include/lldb/Symbol/TypeSystem.h b/lldb/include/lldb/Symbol/TypeSystem.h
index 7681a700766a2..0777d4d5ad6f3 100644
--- a/lldb/include/lldb/Symbol/TypeSystem.h
+++ b/lldb/include/lldb/Symbol/TypeSystem.h
@@ -127,13 +127,13 @@ class TypeSystem : public PluginInterface,
   virtual ConstString
   DeclContextGetScopeQualifiedName(void *opaque_decl_ctx) = 0;
 
-  virtual bool
-  DeclContextIsClassMethod(void *opaque_decl_ctx,
-                           ConstString *language_object_name_ptr) = 0;
+  virtual bool DeclContextIsClassMethod(void *opaque_decl_ctx) = 0;
 
   virtual bool DeclContextIsContainedInLookup(void *opaque_decl_ctx,
                                               void *other_opaque_decl_ctx) = 0;
 
+  virtual lldb::LanguageType DeclContextGetLanguage(void *opaque_decl_ctx) = 0;
+
   // Tests
 #ifndef NDEBUG
   /// Verify the integrity of the type to catch CompilerTypes that mix
@@ -202,6 +202,10 @@ class TypeSystem : public PluginInterface,
   // TypeSystems can support more than one language
   virtual bool SupportsLanguage(lldb::LanguageType language) = 0;
 
+  /// The name of the variable used for explicitly accessing data scoped to the
+  /// current instance (or type). C++ uses "this", ObjC uses "self".
+  virtual ConstString GetInstanceVariableName(lldb::LanguageType language) = 0;
+
   // Type Completion
 
   virtual bool GetCompleteType(lldb::opaque_compiler_type_t type) = 0;
diff --git a/lldb/include/lldb/Target/LanguageRuntime.h b/lldb/include/lldb/Target/LanguageRuntime.h
index 34f0fdb9cfbc9..9cc79dc976c4d 100644
--- a/lldb/include/lldb/Target/LanguageRuntime.h
+++ b/lldb/include/lldb/Target/LanguageRuntime.h
@@ -151,11 +151,6 @@ class LanguageRuntime : public Runtime, public PluginInterface {
   /// from the user interface.
   virtual bool IsAllowedRuntimeValue(ConstString name) { return false; }
 
-  /// Returns 'true' if we the variable with the specified 'name'
-  /// should be hidden from variable views (e.g., when listing variables in
-  /// 'frame variable' or 'target variable')
-  virtual bool ShouldHideVariable(llvm::StringRef name) const { return false; }
-
   virtual std::optional<CompilerType> GetRuntimeType(CompilerType base_type) {
     return std::nullopt;
   }
diff --git a/lldb/include/lldb/Target/Memory.h b/lldb/include/lldb/Target/Memory.h
index 8c564e8f5a8c5..2d1489a2c96ea 100644
--- a/lldb/include/lldb/Target/Memory.h
+++ b/lldb/include/lldb/Target/Memory.h
@@ -61,6 +61,8 @@ class MemoryCache {
 private:
   MemoryCache(const MemoryCache &) = delete;
   const MemoryCache &operator=(const MemoryCache &) = delete;
+
+  lldb::DataBufferSP GetL2CacheLine(lldb::addr_t addr, Status &error);
 };
 
     
diff --git a/lldb/include/lldb/Target/TraceDumper.h b/lldb/include/lldb/Target/TraceDumper.h
index ca3bf2c088088..ca08dc254182a 100644
--- a/lldb/include/lldb/Target/TraceDumper.h
+++ b/lldb/include/lldb/Target/TraceDumper.h
@@ -317,7 +317,7 @@ class TraceDumper {
     FunctionCall(const lldb::TraceCursorSP &cursor_sp,
                  const SymbolInfo &symbol_info);
 
-    /// Append a new traced segment to this funciton call.
+    /// Append a new traced segment to this function call.
     ///
     /// \param[in] cursor_sp
     ///   A cursor pointing to the first instruction of the new segment.
diff --git a/lldb/include/lldb/Interpreter/ScriptedMetadata.h b/lldb/include/lldb/Utility/ScriptedMetadata.h
similarity index 81%
rename from lldb/include/lldb/Interpreter/ScriptedMetadata.h
rename to lldb/include/lldb/Utility/ScriptedMetadata.h
index a4dfe54b762f4..69c83edce909a 100644
--- a/lldb/include/lldb/Interpreter/ScriptedMetadata.h
+++ b/lldb/include/lldb/Utility/ScriptedMetadata.h
@@ -9,9 +9,6 @@
 #ifndef LLDB_INTERPRETER_SCRIPTEDMETADATA_H
 #define LLDB_INTERPRETER_SCRIPTEDMETADATA_H
 
-#include "OptionGroupPythonClassWithDict.h"
-
-#include "lldb/Host/Host.h"
 #include "lldb/Utility/ProcessInfo.h"
 #include "lldb/Utility/StructuredData.h"
 
@@ -30,12 +27,6 @@ class ScriptedMetadata {
     }
   }
 
-  ScriptedMetadata(const OptionGroupPythonClassWithDict &option_group) {
-    auto opt_group = const_cast<OptionGroupPythonClassWithDict &>(option_group);
-    m_class_name = opt_group.GetName();
-    m_args_sp = opt_group.GetStructuredData();
-  }
-
   explicit operator bool() const { return !m_class_name.empty(); }
 
   llvm::StringRef GetClassName() const { return m_class_name; }
diff --git a/lldb/include/lldb/lldb-private-types.h b/lldb/include/lldb/lldb-private-types.h
index edc1c78985bdd..213e570667644 100644
--- a/lldb/include/lldb/lldb-private-types.h
+++ b/lldb/include/lldb/lldb-private-types.h
@@ -116,6 +116,8 @@ typedef struct type256 { uint64_t x[4]; } type256;
 using ValueObjectProviderTy =
     std::function<lldb::ValueObjectSP(ConstString, StackFrame *)>;
 
+typedef void (*DebuggerDestroyCallback)(lldb::user_id_t debugger_id,
+                                        void *baton);
 } // namespace lldb_private
 
 #endif // #if defined(__cplusplus)
diff --git a/lldb/source/API/SBAttachInfo.cpp b/lldb/source/API/SBAttachInfo.cpp
index ffc61b5860d05..0d2dacbf9c78c 100644
--- a/lldb/source/API/SBAttachInfo.cpp
+++ b/lldb/source/API/SBAttachInfo.cpp
@@ -11,9 +11,9 @@
 #include "lldb/API/SBFileSpec.h"
 #include "lldb/API/SBListener.h"
 #include "lldb/API/SBStructuredData.h"
-#include "lldb/Interpreter/ScriptedMetadata.h"
 #include "lldb/Target/Process.h"
 #include "lldb/Utility/Instrumentation.h"
+#include "lldb/Utility/ScriptedMetadata.h"
 
 using namespace lldb;
 using namespace lldb_private;
diff --git a/lldb/source/API/SBCommandInterpreter.cpp b/lldb/source/API/SBCommandInterpreter.cpp
index 35c55283855df..36b70bc1e03a7 100644
--- a/lldb/source/API/SBCommandInterpreter.cpp
+++ b/lldb/source/API/SBCommandInterpreter.cpp
@@ -141,7 +141,13 @@ bool SBCommandInterpreter::IsActive() {
 bool SBCommandInterpreter::WasInterrupted() const {
   LLDB_INSTRUMENT_VA(this);
 
-  return (IsValid() ? m_opaque_ptr->WasInterrupted() : false);
+  return (IsValid() ? m_opaque_ptr->GetDebugger().InterruptRequested() : false);
+}
+
+bool SBCommandInterpreter::InterruptCommand() {
+  LLDB_INSTRUMENT_VA(this);
+  
+  return (IsValid() ? m_opaque_ptr->InterruptCommand() : false);
 }
 
 const char *SBCommandInterpreter::GetIOHandlerControlSequence(char ch) {
diff --git a/lldb/source/API/SBDebugger.cpp b/lldb/source/API/SBDebugger.cpp
index 641c84f0bd159..2e7f06c7bb1b7 100644
--- a/lldb/source/API/SBDebugger.cpp
+++ b/lldb/source/API/SBDebugger.cpp
@@ -1684,9 +1684,39 @@ void SBDebugger::SetLoggingCallback(lldb::LogOutputCallback log_callback,
   }
 }
 
+void SBDebugger::SetDestroyCallback(
+    lldb::SBDebuggerDestroyCallback destroy_callback, void *baton) {
+  LLDB_INSTRUMENT_VA(this, destroy_callback, baton);
+  if (m_opaque_sp) {
+    return m_opaque_sp->SetDestroyCallback(
+        destroy_callback, baton);
+  }
+}
+
 SBTrace
 SBDebugger::LoadTraceFromFile(SBError &error,
                               const SBFileSpec &trace_description_file) {
   LLDB_INSTRUMENT_VA(this, error, trace_description_file);
   return SBTrace::LoadTraceFromFile(error, *this, trace_description_file);
 }
+
+void SBDebugger::RequestInterrupt() {
+  LLDB_INSTRUMENT_VA(this);
+  
+  if (m_opaque_sp)
+    m_opaque_sp->RequestInterrupt();  
+}
+void SBDebugger::CancelInterruptRequest()  {
+  LLDB_INSTRUMENT_VA(this);
+  
+  if (m_opaque_sp)
+    m_opaque_sp->CancelInterruptRequest();  
+}
+
+bool SBDebugger::InterruptRequested()   {
+  LLDB_INSTRUMENT_VA(this);
+  
+  if (m_opaque_sp)
+    return m_opaque_sp->InterruptRequested();
+  return false;
+}
diff --git a/lldb/source/API/SBFrame.cpp b/lldb/source/API/SBFrame.cpp
index bf833fbb95615..4cb086ed40ab4 100644
--- a/lldb/source/API/SBFrame.cpp
+++ b/lldb/source/API/SBFrame.cpp
@@ -16,6 +16,7 @@
 
 #include "Utils.h"
 #include "lldb/Core/Address.h"
+#include "lldb/Core/Debugger.h"
 #include "lldb/Core/StreamFile.h"
 #include "lldb/Core/ValueObjectRegister.h"
 #include "lldb/Core/ValueObjectVariable.h"
@@ -806,6 +807,7 @@ SBValueList SBFrame::GetVariables(const lldb::SBVariablesOptions &options) {
     if (stop_locker.TryLock(&process->GetRunLock())) {
       frame = exe_ctx.GetFramePtr();
       if (frame) {
+        Debugger &dbg = process->GetTarget().GetDebugger();
         VariableList *variable_list = nullptr;
         Status var_error;
         variable_list = frame->GetVariableList(true, &var_error);
@@ -815,6 +817,11 @@ SBValueList SBFrame::GetVariables(const lldb::SBVariablesOptions &options) {
           const size_t num_variables = variable_list->GetSize();
           if (num_variables) {
             for (const VariableSP &variable_sp : *variable_list) {
+              if (dbg.InterruptRequested()) {
+                Log *log = GetLog(LLDBLog::Host);
+                LLDB_LOG(log, "Interrupted SBFrame::GetVariables");
+                return {};
+              }
               if (variable_sp) {
                 bool add_variable = false;
                 switch (variable_sp->GetScope()) {
diff --git a/lldb/source/API/SBLaunchInfo.cpp b/lldb/source/API/SBLaunchInfo.cpp
index 5800ac63dddb6..21ae84b4e3cc4 100644
--- a/lldb/source/API/SBLaunchInfo.cpp
+++ b/lldb/source/API/SBLaunchInfo.cpp
@@ -17,7 +17,7 @@
 #include "lldb/API/SBStructuredData.h"
 #include "lldb/Core/StructuredDataImpl.h"
 #include "lldb/Host/ProcessLaunchInfo.h"
-#include "lldb/Interpreter/ScriptedMetadata.h"
+#include "lldb/Utility/ScriptedMetadata.h"
 
 using namespace lldb;
 using namespace lldb_private;
diff --git a/lldb/source/Commands/CommandObjectDWIMPrint.cpp b/lldb/source/Commands/CommandObjectDWIMPrint.cpp
index a348f416af22f..d8bc7a1e89696 100644
--- a/lldb/source/Commands/CommandObjectDWIMPrint.cpp
+++ b/lldb/source/Commands/CommandObjectDWIMPrint.cpp
@@ -32,6 +32,10 @@ CommandObjectDWIMPrint::CommandObjectDWIMPrint(CommandInterpreter &interpreter)
                        "Print a variable or expression.",
                        "dwim-print [<variable-name> | <expression>]",
                        eCommandProcessMustBePaused | eCommandTryTargetAPILock) {
+
+  CommandArgumentData var_name_arg(eArgTypeVarName, eArgRepeatPlain);
+  m_arguments.push_back({var_name_arg});
+
   m_option_group.Append(&m_format_options,
                         OptionGroupFormat::OPTION_GROUP_FORMAT |
                             OptionGroupFormat::OPTION_GROUP_GDB_FMT,
@@ -44,6 +48,13 @@ CommandObjectDWIMPrint::CommandObjectDWIMPrint(CommandInterpreter &interpreter)
 
 Options *CommandObjectDWIMPrint::GetOptions() { return &m_option_group; }
 
+void CommandObjectDWIMPrint::HandleArgumentCompletion(
+    CompletionRequest &request, OptionElementVector &opt_element_vector) {
+  CommandCompletions::InvokeCommonCompletionCallbacks(
+      GetCommandInterpreter(), CommandCompletions::eVariablePathCompletion,
+      request, nullptr);
+}
+
 bool CommandObjectDWIMPrint::DoExecute(StringRef command,
                                        CommandReturnObject &result) {
   m_option_group.NotifyOptionParsingStarting(&m_exe_ctx);
@@ -77,8 +88,10 @@ bool CommandObjectDWIMPrint::DoExecute(StringRef command,
   if (StackFrame *frame = m_exe_ctx.GetFramePtr()) {
     auto valobj_sp = frame->FindVariable(ConstString(expr));
     if (valobj_sp && valobj_sp->GetError().Success()) {
-      if (!eval_options.GetSuppressPersistentResult())
-        valobj_sp = valobj_sp->Persist();
+      if (!eval_options.GetSuppressPersistentResult()) {
+        if (auto persisted_valobj = valobj_sp->Persist())
+          valobj_sp = persisted_valobj;
+      }
 
       if (verbosity == eDWIMPrintVerbosityFull) {
         StringRef flags;
diff --git a/lldb/source/Commands/CommandObjectDWIMPrint.h b/lldb/source/Commands/CommandObjectDWIMPrint.h
index 8a14a6c7bb531..3fc6c01d47297 100644
--- a/lldb/source/Commands/CommandObjectDWIMPrint.h
+++ b/lldb/source/Commands/CommandObjectDWIMPrint.h
@@ -37,6 +37,12 @@ class CommandObjectDWIMPrint : public CommandObjectRaw {
 
   Options *GetOptions() override;
 
+  bool WantsCompletion() override { return true; }
+
+  void
+  HandleArgumentCompletion(CompletionRequest &request,
+                           OptionElementVector &opt_element_vector) override;
+
 private:
   bool DoExecute(llvm::StringRef command, CommandReturnObject &result) override;
 
diff --git a/lldb/source/Commands/CommandObjectPlatform.cpp b/lldb/source/Commands/CommandObjectPlatform.cpp
index 213631a250bd7..60d37121844b4 100644
--- a/lldb/source/Commands/CommandObjectPlatform.cpp
+++ b/lldb/source/Commands/CommandObjectPlatform.cpp
@@ -20,11 +20,11 @@
 #include "lldb/Interpreter/OptionGroupFile.h"
 #include "lldb/Interpreter/OptionGroupPlatform.h"
 #include "lldb/Interpreter/OptionGroupPythonClassWithDict.h"
-#include "lldb/Interpreter/ScriptedMetadata.h"
 #include "lldb/Target/ExecutionContext.h"
 #include "lldb/Target/Platform.h"
 #include "lldb/Target/Process.h"
 #include "lldb/Utility/Args.h"
+#include "lldb/Utility/ScriptedMetadata.h"
 
 #include "llvm/ADT/SmallString.h"
 
diff --git a/lldb/source/Commands/CommandObjectProcess.cpp b/lldb/source/Commands/CommandObjectProcess.cpp
index 4362b25f9663c..5b578454f1ad6 100644
--- a/lldb/source/Commands/CommandObjectProcess.cpp
+++ b/lldb/source/Commands/CommandObjectProcess.cpp
@@ -25,7 +25,6 @@
 #include "lldb/Interpreter/OptionArgParser.h"
 #include "lldb/Interpreter/OptionGroupPythonClassWithDict.h"
 #include "lldb/Interpreter/Options.h"
-#include "lldb/Interpreter/ScriptedMetadata.h"
 #include "lldb/Target/Platform.h"
 #include "lldb/Target/Process.h"
 #include "lldb/Target/StopInfo.h"
@@ -33,6 +32,7 @@
 #include "lldb/Target/Thread.h"
 #include "lldb/Target/UnixSignals.h"
 #include "lldb/Utility/Args.h"
+#include "lldb/Utility/ScriptedMetadata.h"
 #include "lldb/Utility/State.h"
 
 #include "llvm/ADT/ScopeExit.h"
diff --git a/lldb/source/Commands/CommandObjectRegexCommand.cpp b/lldb/source/Commands/CommandObjectRegexCommand.cpp
index 2d44ada0acc93..90b32b193e329 100644
--- a/lldb/source/Commands/CommandObjectRegexCommand.cpp
+++ b/lldb/source/Commands/CommandObjectRegexCommand.cpp
@@ -72,8 +72,9 @@ bool CommandObjectRegexCommand::DoExecute(llvm::StringRef command,
         result.GetOutputStream().Printf("%s\n", new_command->c_str());
       // We don't have to pass an override_context here, as the command that 
       // called us should have set up the context appropriately.
-      return m_interpreter.HandleCommand(new_command->c_str(),
-                                         eLazyBoolNo, result);
+      bool force_repeat_command = true;
+      return m_interpreter.HandleCommand(new_command->c_str(), eLazyBoolNo,
+                                         result, force_repeat_command);
     }
   }
   result.SetStatus(eReturnStatusFailed);
diff --git a/lldb/source/Commands/CommandObjectTarget.cpp b/lldb/source/Commands/CommandObjectTarget.cpp
index 490a7e061563a..5874453bca23f 100644
--- a/lldb/source/Commands/CommandObjectTarget.cpp
+++ b/lldb/source/Commands/CommandObjectTarget.cpp
@@ -2004,7 +2004,7 @@ class CommandObjectTargetModulesDumpSymtab
             result.GetOutputStream().EOL();
             result.GetOutputStream().EOL();
           }
-          if (m_interpreter.WasInterrupted())
+          if (GetDebugger().InterruptRequested())
             break;
           num_dumped++;
           DumpModuleSymtab(m_interpreter, result.GetOutputStream(),
@@ -2031,7 +2031,7 @@ class CommandObjectTargetModulesDumpSymtab
                 result.GetOutputStream().EOL();
                 result.GetOutputStream().EOL();
               }
-              if (m_interpreter.WasInterrupted())
+              if (GetDebugger().InterruptRequested())
                 break;
               num_dumped++;
               DumpModuleSymtab(m_interpreter, result.GetOutputStream(),
@@ -2092,7 +2092,7 @@ class CommandObjectTargetModulesDumpSections
       result.GetOutputStream().Format("Dumping sections for {0} modules.\n",
                                       num_modules);
       for (size_t image_idx = 0; image_idx < num_modules; ++image_idx) {
-        if (m_interpreter.WasInterrupted())
+        if (GetDebugger().InterruptRequested())
           break;
         num_dumped++;
         DumpModuleSections(
@@ -2110,7 +2110,7 @@ class CommandObjectTargetModulesDumpSections
             FindModulesByName(target, arg_cstr, module_list, true);
         if (num_matches > 0) {
           for (size_t i = 0; i < num_matches; ++i) {
-            if (m_interpreter.WasInterrupted())
+            if (GetDebugger().InterruptRequested())
               break;
             Module *module = module_list.GetModulePointerAtIndex(i);
             if (module) {
@@ -2224,7 +2224,7 @@ class CommandObjectTargetModulesDumpClangAST
       result.GetOutputStream().Format("Dumping clang ast for {0} modules.\n",
                                       num_modules);
       for (ModuleSP module_sp : module_list.ModulesNoLocking()) {
-        if (m_interpreter.WasInterrupted())
+        if (GetDebugger().InterruptRequested())
           break;
         if (SymbolFile *sf = module_sp->GetSymbolFile())
           sf->DumpClangAST(result.GetOutputStream());
@@ -2249,7 +2249,7 @@ class CommandObjectTargetModulesDumpClangAST
       }
 
       for (size_t i = 0; i < num_matches; ++i) {
-        if (m_interpreter.WasInterrupted())
+        if (GetDebugger().InterruptRequested())
           break;
         Module *m = module_list.GetModulePointerAtIndex(i);
         if (SymbolFile *sf = m->GetSymbolFile())
@@ -2298,7 +2298,7 @@ class CommandObjectTargetModulesDumpSymfile
       result.GetOutputStream().Format(
           "Dumping debug symbols for {0} modules.\n", num_modules);
       for (ModuleSP module_sp : target_modules.ModulesNoLocking()) {
-        if (m_interpreter.WasInterrupted())
+        if (GetDebugger().InterruptRequested())
           break;
         if (DumpModuleSymbolFile(result.GetOutputStream(), module_sp.get()))
           num_dumped++;
@@ -2314,7 +2314,7 @@ class CommandObjectTargetModulesDumpSymfile
             FindModulesByName(target, arg_cstr, module_list, true);
         if (num_matches > 0) {
           for (size_t i = 0; i < num_matches; ++i) {
-            if (m_interpreter.WasInterrupted())
+            if (GetDebugger().InterruptRequested())
               break;
             Module *module = module_list.GetModulePointerAtIndex(i);
             if (module) {
@@ -2381,7 +2381,7 @@ class CommandObjectTargetModulesDumpLineTable
         if (target_modules.GetSize() > 0) {
           uint32_t num_dumped = 0;
           for (ModuleSP module_sp : target_modules.ModulesNoLocking()) {
-            if (m_interpreter.WasInterrupted())
+            if (GetDebugger().InterruptRequested())
               break;
             if (DumpCompileUnitLineTable(
                     m_interpreter, result.GetOutputStream(), module_sp.get(),
diff --git a/lldb/source/Commands/CommandObjectThread.cpp b/lldb/source/Commands/CommandObjectThread.cpp
index 411f28d84a21f..f06d4bd7937f9 100644
--- a/lldb/source/Commands/CommandObjectThread.cpp
+++ b/lldb/source/Commands/CommandObjectThread.cpp
@@ -228,7 +228,7 @@ class CommandObjectThreadBacktrace : public CommandObjectIterateOverThreads {
           thread->GetIndexID());
       return false;
     }
-    if (m_options.m_extended_backtrace) {
+    if (m_options.m_extended_backtrace && !GetDebugger().InterruptRequested()) {
       DoExtendedBacktrace(thread, result);
     }
 
diff --git a/lldb/source/Commands/CommandObjectWatchpoint.cpp b/lldb/source/Commands/CommandObjectWatchpoint.cpp
index 7aa9614634181..8b8c196e4505b 100644
--- a/lldb/source/Commands/CommandObjectWatchpoint.cpp
+++ b/lldb/source/Commands/CommandObjectWatchpoint.cpp
@@ -29,12 +29,12 @@
 using namespace lldb;
 using namespace lldb_private;
 
-static void AddWatchpointDescription(Stream *s, Watchpoint *wp,
+static void AddWatchpointDescription(Stream &s, Watchpoint &wp,
                                      lldb::DescriptionLevel level) {
-  s->IndentMore();
-  wp->GetDescription(s, level);
-  s->IndentLess();
-  s->EOL();
+  s.IndentMore();
+  wp.GetDescription(&s, level);
+  s.IndentLess();
+  s.EOL();
 }
 
 static bool CheckTargetForWatchpointOperations(Target *target,
@@ -237,8 +237,8 @@ class CommandObjectWatchpointList : public CommandObjectParsed {
       // No watchpoint selected; show info about all currently set watchpoints.
       result.AppendMessage("Current watchpoints:");
       for (size_t i = 0; i < num_watchpoints; ++i) {
-        Watchpoint *wp = watchpoints.GetByIndex(i).get();
-        AddWatchpointDescription(&output_stream, wp, m_options.m_level);
+        WatchpointSP watch_sp = watchpoints.GetByIndex(i);
+        AddWatchpointDescription(output_stream, *watch_sp, m_options.m_level);
       }
       result.SetStatus(eReturnStatusSuccessFinishNoResult);
     } else {
@@ -252,9 +252,9 @@ class CommandObjectWatchpointList : public CommandObjectParsed {
 
       const size_t size = wp_ids.size();
       for (size_t i = 0; i < size; ++i) {
-        Watchpoint *wp = watchpoints.FindByID(wp_ids[i]).get();
-        if (wp)
-          AddWatchpointDescription(&output_stream, wp, m_options.m_level);
+        WatchpointSP watch_sp = watchpoints.FindByID(wp_ids[i]);
+        if (watch_sp)
+          AddWatchpointDescription(output_stream, *watch_sp, m_options.m_level);
         result.SetStatus(eReturnStatusSuccessFinishNoResult);
       }
     }
@@ -758,8 +758,8 @@ class CommandObjectWatchpointModify : public CommandObjectParsed {
     }
 
     if (command.GetArgumentCount() == 0) {
-      WatchpointSP wp_sp = target->GetLastCreatedWatchpoint();
-      wp_sp->SetCondition(m_options.m_condition.c_str());
+      WatchpointSP watch_sp = target->GetLastCreatedWatchpoint();
+      watch_sp->SetCondition(m_options.m_condition.c_str());
       result.SetStatus(eReturnStatusSuccessFinishNoResult);
     } else {
       // Particular watchpoints selected; set condition on them.
@@ -773,9 +773,9 @@ class CommandObjectWatchpointModify : public CommandObjectParsed {
       int count = 0;
       const size_t size = wp_ids.size();
       for (size_t i = 0; i < size; ++i) {
-        WatchpointSP wp_sp = watchpoints.FindByID(wp_ids[i]);
-        if (wp_sp) {
-          wp_sp->SetCondition(m_options.m_condition.c_str());
+        WatchpointSP watch_sp = watchpoints.FindByID(wp_ids[i]);
+        if (watch_sp) {
+          watch_sp->SetCondition(m_options.m_condition.c_str());
           ++count;
         }
       }
@@ -949,20 +949,19 @@ corresponding to the byte size of the data type.");
     uint32_t watch_type = m_option_watchpoint.watch_type;
 
     error.Clear();
-    Watchpoint *wp =
-        target->CreateWatchpoint(addr, size, &compiler_type, watch_type, error)
-            .get();
-    if (wp) {
-      wp->SetWatchSpec(command.GetArgumentAtIndex(0));
-      wp->SetWatchVariable(true);
+    WatchpointSP watch_sp =
+        target->CreateWatchpoint(addr, size, &compiler_type, watch_type, error);
+    if (watch_sp) {
+      watch_sp->SetWatchSpec(command.GetArgumentAtIndex(0));
+      watch_sp->SetWatchVariable(true);
       if (var_sp && var_sp->GetDeclaration().GetFile()) {
         StreamString ss;
         // True to show fullpath for declaration file.
         var_sp->GetDeclaration().DumpStopContext(&ss, true);
-        wp->SetDeclInfo(std::string(ss.GetString()));
+        watch_sp->SetDeclInfo(std::string(ss.GetString()));
       }
       output_stream.Printf("Watchpoint created: ");
-      wp->GetDescription(&output_stream, lldb::eDescriptionLevelFull);
+      watch_sp->GetDescription(&output_stream, lldb::eDescriptionLevelFull);
       output_stream.EOL();
       result.SetStatus(eReturnStatusSuccessFinishResult);
     } else {
@@ -1117,13 +1116,13 @@ class CommandObjectWatchpointSetExpression : public CommandObjectRaw {
     CompilerType compiler_type(valobj_sp->GetCompilerType());
 
     Status error;
-    Watchpoint *wp =
-        target->CreateWatchpoint(addr, size, &compiler_type, watch_type, error)
-            .get();
-    if (wp) {
+    WatchpointSP watch_sp =
+        target->CreateWatchpoint(addr, size, &compiler_type, watch_type, error);
+    if (watch_sp) {
+      watch_sp->SetWatchSpec(std::string(expr));
       Stream &output_stream = result.GetOutputStream();
       output_stream.Printf("Watchpoint created: ");
-      wp->GetDescription(&output_stream, lldb::eDescriptionLevelFull);
+      watch_sp->GetDescription(&output_stream, lldb::eDescriptionLevelFull);
       output_stream.EOL();
       result.SetStatus(eReturnStatusSuccessFinishResult);
     } else {
diff --git a/lldb/source/Core/CoreProperties.td b/lldb/source/Core/CoreProperties.td
index fc664be63f429..62729923f366b 100644
--- a/lldb/source/Core/CoreProperties.td
+++ b/lldb/source/Core/CoreProperties.td
@@ -193,7 +193,7 @@ let Definition = "debugger" in {
     Desc<"When displaying suggestion in a color-enabled terminal, use the ANSI terminal code specified in this format immediately after the suggestion.">;
   def DWIMPrintVerbosity: Property<"dwim-print-verbosity", "Enum">,
     Global,
-    DefaultEnumValue<"eDWIMPrintVerbosityExpression">,
+    DefaultEnumValue<"eDWIMPrintVerbosityNone">,
     EnumValues<"OptionEnumValues(g_dwim_print_verbosities)">,
     Desc<"The verbosity level used by dwim-print.">;
 }
diff --git a/lldb/source/Core/Debugger.cpp b/lldb/source/Core/Debugger.cpp
index 9c09da429d401..2926a29463f59 100644
--- a/lldb/source/Core/Debugger.cpp
+++ b/lldb/source/Core/Debugger.cpp
@@ -44,7 +44,6 @@
 #include "lldb/Target/Thread.h"
 #include "lldb/Target/ThreadList.h"
 #include "lldb/Utility/AnsiTerminal.h"
-#include "lldb/Utility/Diagnostics.h"
 #include "lldb/Utility/Event.h"
 #include "lldb/Utility/LLDBLog.h"
 #include "lldb/Utility/Listener.h"
@@ -560,6 +559,12 @@ void Debugger::Terminate() {
   assert(g_debugger_list_ptr &&
          "Debugger::Terminate called without a matching Debugger::Initialize!");
 
+  if (g_debugger_list_ptr && g_debugger_list_mutex_ptr) {
+    std::lock_guard<std::recursive_mutex> guard(*g_debugger_list_mutex_ptr);
+    for (const auto &debugger : *g_debugger_list_ptr)
+      debugger->HandleDestroyCallback();
+  }
+
   if (g_thread_pool) {
     // The destructor will wait for all the threads to complete.
     delete g_thread_pool;
@@ -679,10 +684,18 @@ DebuggerSP Debugger::CreateInstance(lldb::LogOutputCallback log_callback,
   return debugger_sp;
 }
 
+void Debugger::HandleDestroyCallback() {
+  if (m_destroy_callback) {
+    m_destroy_callback(GetID(), m_destroy_callback_baton);
+    m_destroy_callback = nullptr;
+  }
+}
+
 void Debugger::Destroy(DebuggerSP &debugger_sp) {
   if (!debugger_sp)
     return;
 
+  debugger_sp->HandleDestroyCallback();
   CommandInterpreter &cmd_interpreter = debugger_sp->GetCommandInterpreter();
 
   if (cmd_interpreter.GetSaveSessionOnQuit()) {
@@ -828,6 +841,22 @@ Debugger::Debugger(lldb::LogOutputCallback log_callback, void *baton)
   if (!GetOutputFile().GetIsTerminalWithColors())
     SetUseColor(false);
 
+  if (Diagnostics::Enabled()) {
+    m_diagnostics_callback_id = Diagnostics::Instance().AddCallback(
+        [this](const FileSpec &dir) -> llvm::Error {
+          for (auto &entry : m_stream_handlers) {
+            llvm::StringRef log_path = entry.first();
+            llvm::StringRef file_name = llvm::sys::path::filename(log_path);
+            FileSpec destination = dir.CopyByAppendingPathComponent(file_name);
+            std::error_code ec =
+                llvm::sys::fs::copy_file(log_path, destination.GetPath());
+            if (ec)
+              return llvm::errorCodeToError(ec);
+          }
+          return llvm::Error::success();
+        });
+  }
+
 #if defined(_WIN32) && defined(ENABLE_VIRTUAL_TERMINAL_PROCESSING)
   // Enabling use of ANSI color codes because LLDB is using them to highlight
   // text.
@@ -866,6 +895,9 @@ void Debugger::Clear() {
     GetInputFile().Close();
 
     m_command_interpreter_up->Clear();
+
+    if (Diagnostics::Enabled())
+      Diagnostics::Instance().RemoveCallback(m_diagnostics_callback_id);
   });
 }
 
@@ -1198,6 +1230,29 @@ StreamSP Debugger::GetAsyncErrorStream() {
   return std::make_shared<StreamAsynchronousIO>(*this, false, GetUseColor());
 }
 
+void Debugger::RequestInterrupt() {
+  std::lock_guard<std::mutex> guard(m_interrupt_mutex);
+  m_interrupt_requested++;
+}
+
+void Debugger::CancelInterruptRequest() {
+  std::lock_guard<std::mutex> guard(m_interrupt_mutex);
+  if (m_interrupt_requested > 0)
+    m_interrupt_requested--;
+}
+
+bool Debugger::InterruptRequested() {
+  // This is the one we should call internally.  This will return true either
+  // if there's a debugger interrupt and we aren't on the IOHandler thread, 
+  // or if we are on the IOHandler thread and there's a CommandInterpreter
+  // interrupt.
+  if (!IsIOHandlerThreadCurrentThread()) {
+    std::lock_guard<std::mutex> guard(m_interrupt_mutex);
+    return m_interrupt_requested != 0;
+  }
+  return GetCommandInterpreter().WasInterrupted();
+}
+
 size_t Debugger::GetNumDebuggers() {
   if (g_debugger_list_ptr && g_debugger_list_mutex_ptr) {
     std::lock_guard<std::recursive_mutex> guard(*g_debugger_list_mutex_ptr);
@@ -1285,6 +1340,12 @@ void Debugger::SetLoggingCallback(lldb::LogOutputCallback log_callback,
       std::make_shared<CallbackLogHandler>(log_callback, baton);
 }
 
+void Debugger::SetDestroyCallback(
+    lldb_private::DebuggerDestroyCallback destroy_callback, void *baton) {
+  m_destroy_callback = destroy_callback;
+  m_destroy_callback_baton = baton;
+}
+
 static void PrivateReportProgress(Debugger &debugger, uint64_t progress_id,
                                   std::string title, std::string details,
                                   uint64_t completed, uint64_t total,
@@ -1943,7 +2004,15 @@ void Debugger::HandleDiagnosticEvent(const lldb::EventSP &event_sp) {
   data->Dump(stream.get());
 }
 
-bool Debugger::HasIOHandlerThread() { return m_io_handler_thread.IsJoinable(); }
+bool Debugger::HasIOHandlerThread() const {
+  return m_io_handler_thread.IsJoinable(); 
+}
+
+HostThread Debugger::SetIOHandlerThread(HostThread &new_thread) {
+  HostThread old_host = m_io_handler_thread;
+  m_io_handler_thread = new_thread;
+  return old_host;
+}
 
 bool Debugger::StartIOHandlerThread() {
   if (!m_io_handler_thread.IsJoinable()) {
@@ -1975,6 +2044,12 @@ void Debugger::JoinIOHandlerThread() {
   }
 }
 
+bool Debugger::IsIOHandlerThreadCurrentThread() const {
+  if (!HasIOHandlerThread())
+    return false;
+  return m_io_handler_thread.EqualsThread(Host::GetCurrentThread());
+}
+
 Target &Debugger::GetSelectedOrDummyTarget(bool prefer_dummy) {
   if (!prefer_dummy) {
     if (TargetSP target = m_target_list.GetSelectedTarget())
diff --git a/lldb/source/Core/ValueObject.cpp b/lldb/source/Core/ValueObject.cpp
index f3ca16781e9ad..1be7e2c322f0d 100644
--- a/lldb/source/Core/ValueObject.cpp
+++ b/lldb/source/Core/ValueObject.cpp
@@ -1600,20 +1600,12 @@ bool ValueObject::IsRuntimeSupportValue() {
   if (!process)
     return false;
 
-  if (!GetVariable())
-    return false;
-
-  auto *runtime = process->GetLanguageRuntime(GetVariable()->GetLanguage());
-  if (runtime)
-    if (runtime->ShouldHideVariable(GetName().GetStringRef()))
-      return true;
-
   // We trust that the compiler did the right thing and marked runtime support
   // values as artificial.
-  if (!GetVariable()->IsArtificial())
+  if (!GetVariable() || !GetVariable()->IsArtificial())
     return false;
 
-  if (runtime)
+  if (auto *runtime = process->GetLanguageRuntime(GetVariable()->GetLanguage()))
     if (runtime->IsAllowedRuntimeValue(GetName()))
       return false;
 
@@ -1867,7 +1859,7 @@ ValueObjectSP ValueObject::GetDynamicValue(DynamicValueType use_dynamic) {
   if (!IsDynamic() && m_dynamic_value == nullptr) {
     CalculateDynamicValue(use_dynamic);
   }
-  if (m_dynamic_value)
+  if (m_dynamic_value && m_dynamic_value->GetError().Success())
     return m_dynamic_value->GetSP();
   else
     return ValueObjectSP();
diff --git a/lldb/source/Core/ValueObjectConstResult.cpp b/lldb/source/Core/ValueObjectConstResult.cpp
index 640abdddfcad5..17a725dcc7dd8 100644
--- a/lldb/source/Core/ValueObjectConstResult.cpp
+++ b/lldb/source/Core/ValueObjectConstResult.cpp
@@ -287,7 +287,7 @@ ValueObjectConstResult::GetDynamicValue(lldb::DynamicValueType use_dynamic) {
       if (process && process->IsPossibleDynamicValue(*this))
         m_dynamic_value = new ValueObjectDynamicValue(*this, use_dynamic);
     }
-    if (m_dynamic_value)
+    if (m_dynamic_value && m_dynamic_value->GetError().Success())
       return m_dynamic_value->GetSP();
   }
   return ValueObjectSP();
diff --git a/lldb/source/Core/ValueObjectDynamicValue.cpp b/lldb/source/Core/ValueObjectDynamicValue.cpp
index 0659c771918a5..9db50aeeec9ea 100644
--- a/lldb/source/Core/ValueObjectDynamicValue.cpp
+++ b/lldb/source/Core/ValueObjectDynamicValue.cpp
@@ -187,17 +187,19 @@ bool ValueObjectDynamicValue::UpdateValue() {
     m_type_impl.Clear();
   }
 
-  // If we don't have a dynamic type, then make ourselves just a echo of our
-  // parent. Or we could return false, and make ourselves an echo of our
-  // parent?
+  // If we don't have a dynamic type, set ourselves to be invalid and return
+  // false.  We used to try to produce a dynamic ValueObject that behaved "like"
+  // its parent, but that failed for ValueObjectConstResult, which is too 
+  // complex a beast to try to emulate.  If we return an invalid ValueObject,
+  // clients will end up getting the static value instead, which behaves
+  // correctly.
   if (!found_dynamic_type) {
     if (m_dynamic_type_info)
       SetValueDidChange(true);
     ClearDynamicTypeInformation();
     m_dynamic_type_info.Clear();
-    m_value = m_parent->GetValue();
-    m_error = m_value.GetValueAsData(&exe_ctx, m_data, GetModule().get());
-    return m_error.Success();
+    m_error.SetErrorString("no dynamic type found");
+    return false;
   }
 
   Value old_value(m_value);
diff --git a/lldb/source/Host/common/ProcessLaunchInfo.cpp b/lldb/source/Host/common/ProcessLaunchInfo.cpp
index af278cc1a03c3..8e09601b5715b 100644
--- a/lldb/source/Host/common/ProcessLaunchInfo.cpp
+++ b/lldb/source/Host/common/ProcessLaunchInfo.cpp
@@ -321,6 +321,8 @@ bool ProcessLaunchInfo::ConvertArgumentsForLaunchingInShell(
       } else {
         for (size_t i = 0; argv[i] != nullptr; ++i) {
           std::string safe_arg = Args::GetShellSafeArgument(m_shell, argv[i]);
+          if (safe_arg.empty())
+            safe_arg = "\"\"";
           // Add a space to separate this arg from the previous one.
           shell_command.PutCString(" ");
           shell_command.PutCString(safe_arg);
diff --git a/lldb/source/Interpreter/CommandInterpreter.cpp b/lldb/source/Interpreter/CommandInterpreter.cpp
index 1437ca464e760..cf3fff23005bf 100644
--- a/lldb/source/Interpreter/CommandInterpreter.cpp
+++ b/lldb/source/Interpreter/CommandInterpreter.cpp
@@ -413,17 +413,21 @@ void CommandInterpreter::Initialize() {
 
   alias_arguments_vector_sp = std::make_shared<OptionArgVector>();
 
-  cmd_obj_sp = GetCommandSPExact("expression");
+  cmd_obj_sp = GetCommandSPExact("dwim-print");
   if (cmd_obj_sp) {
     AddAlias("p", cmd_obj_sp, "--")->SetHelpLong("");
     AddAlias("print", cmd_obj_sp, "--")->SetHelpLong("");
-    AddAlias("call", cmd_obj_sp, "--")->SetHelpLong("");
     if (auto *po = AddAlias("po", cmd_obj_sp, "-O --")) {
       po->SetHelp("Evaluate an expression on the current thread.  Displays any "
                   "returned value with formatting "
                   "controlled by the type's author.");
       po->SetHelpLong("");
     }
+  }
+
+  cmd_obj_sp = GetCommandSPExact("expression");
+  if (cmd_obj_sp) {
+    AddAlias("call", cmd_obj_sp, "--")->SetHelpLong("");
     CommandAlias *parray_alias =
         AddAlias("parray", cmd_obj_sp, "--element-count %1 --");
     if (parray_alias) {
@@ -1870,8 +1874,8 @@ bool CommandInterpreter::HandleCommand(const char *command_line,
 
 bool CommandInterpreter::HandleCommand(const char *command_line,
                                        LazyBool lazy_add_to_history,
-                                       CommandReturnObject &result) {
-
+                                       CommandReturnObject &result,
+                                       bool force_repeat_command) {
   std::string command_string(command_line);
   std::string original_command_string(command_line);
 
@@ -1882,8 +1886,8 @@ bool CommandInterpreter::HandleCommand(const char *command_line,
   LLDB_LOGF(log, "Processing command: %s", command_line);
   LLDB_SCOPED_TIMERF("Processing command: %s.", command_line);
 
-  if (WasInterrupted()) {
-    result.AppendError("interrupted");
+  if (GetDebugger().InterruptRequested()) {
+    result.AppendError("... Interrupted");
     return false;
   }
 
@@ -2000,17 +2004,26 @@ bool CommandInterpreter::HandleCommand(const char *command_line,
   // arguments.
 
   if (cmd_obj != nullptr) {
+    bool generate_repeat_command = add_to_history;
     // If we got here when empty_command was true, then this command is a
     // stored "repeat command" which we should give a chance to produce it's
     // repeat command, even though we don't add repeat commands to the history.
-    if (add_to_history || empty_command) {
+    generate_repeat_command |= empty_command;
+    // For `command regex`, the regex command (ex `bt`) is added to history, but
+    // the resolved command (ex `thread backtrace`) is _not_ added to history.
+    // However, the resolved command must be given the opportunity to provide a
+    // repeat command. `force_repeat_command` supports this case.
+    generate_repeat_command |= force_repeat_command;
+    if (generate_repeat_command) {
       Args command_args(command_string);
       std::optional<std::string> repeat_command =
           cmd_obj->GetRepeatCommand(command_args, 0);
-      if (repeat_command)
+      if (repeat_command) {
+        LLDB_LOGF(log, "Repeat command: %s", repeat_command->data());
         m_repeat_command.assign(*repeat_command);
-      else
+      } else {
         m_repeat_command.assign(original_command_string);
+      }
     }
 
     if (add_to_history)
@@ -2542,7 +2555,7 @@ void CommandInterpreter::HandleCommands(const StringList &commands,
     m_debugger.SetAsyncExecution(false);
   }
 
-  for (size_t idx = 0; idx < num_lines && !WasInterrupted(); idx++) {
+  for (size_t idx = 0; idx < num_lines; idx++) {
     const char *cmd = commands.GetStringAtIndex(idx);
     if (cmd[0] == '\0')
       continue;
@@ -3022,6 +3035,9 @@ bool CommandInterpreter::InterruptCommand() {
 }
 
 bool CommandInterpreter::WasInterrupted() const {
+  if (!m_debugger.IsIOHandlerThreadCurrentThread())
+    return false;
+
   bool was_interrupted =
       (m_command_state == CommandHandlingState::eInterrupted);
   lldbassert(!was_interrupted || m_iohandler_nesting_level > 0);
@@ -3035,7 +3051,8 @@ void CommandInterpreter::PrintCommandOutput(IOHandler &io_handler,
   lldb::StreamFileSP stream = is_stdout ? io_handler.GetOutputStreamFileSP()
                                         : io_handler.GetErrorStreamFileSP();
   // Split the output into lines and poll for interrupt requests
-  while (!str.empty() && !WasInterrupted()) {
+  bool had_output = !str.empty();
+  while (!str.empty()) {
     llvm::StringRef line;
     std::tie(line, str) = str.split('\n');
     {
@@ -3046,7 +3063,7 @@ void CommandInterpreter::PrintCommandOutput(IOHandler &io_handler,
   }
 
   std::lock_guard<std::recursive_mutex> guard(io_handler.GetOutputMutex());
-  if (!str.empty())
+  if (had_output && GetDebugger().InterruptRequested())
     stream->Printf("\n... Interrupted.\n");
   stream->Flush();
 }
@@ -3359,7 +3376,13 @@ CommandInterpreterRunResult CommandInterpreter::RunCommandInterpreter(
   if (options.GetSpawnThread()) {
     m_debugger.StartIOHandlerThread();
   } else {
+    // If the current thread is not managed by a host thread, we won't detect
+    // that this IS the CommandInterpreter IOHandler thread, so make it so:
+    HostThread new_io_handler_thread(Host::GetCurrentThread());
+    HostThread old_io_handler_thread 
+        = m_debugger.SetIOHandlerThread(new_io_handler_thread);
     m_debugger.RunIOHandlers();
+    m_debugger.SetIOHandlerThread(old_io_handler_thread);
 
     if (options.GetAutoHandleEvents())
       m_debugger.StopEventHandlerThread();
diff --git a/lldb/source/Interpreter/OptionValueArgs.cpp b/lldb/source/Interpreter/OptionValueArgs.cpp
index bdb5f486835a6..0e20154907b02 100644
--- a/lldb/source/Interpreter/OptionValueArgs.cpp
+++ b/lldb/source/Interpreter/OptionValueArgs.cpp
@@ -17,8 +17,7 @@ size_t OptionValueArgs::GetArgs(Args &args) const {
   args.Clear();
   for (const auto &value : m_values) {
     llvm::StringRef string_value = value->GetStringValue();
-    if (!string_value.empty())
-      args.AppendArgument(string_value);
+    args.AppendArgument(string_value);
   }
 
   return args.GetArgumentCount();
diff --git a/lldb/source/Plugins/DynamicLoader/Darwin-Kernel/CMakeLists.txt b/lldb/source/Plugins/DynamicLoader/Darwin-Kernel/CMakeLists.txt
index fb838f42489db..a22a83363ffea 100644
--- a/lldb/source/Plugins/DynamicLoader/Darwin-Kernel/CMakeLists.txt
+++ b/lldb/source/Plugins/DynamicLoader/Darwin-Kernel/CMakeLists.txt
@@ -17,6 +17,7 @@ add_lldb_library(lldbPluginDynamicLoaderDarwinKernel PLUGIN
     lldbSymbol
     lldbTarget
     lldbUtility
+    lldbPluginObjectFileMachO
   )
 
 add_dependencies(lldbPluginDynamicLoaderDarwinKernel
diff --git a/lldb/source/Plugins/DynamicLoader/Darwin-Kernel/DynamicLoaderDarwinKernel.cpp b/lldb/source/Plugins/DynamicLoader/Darwin-Kernel/DynamicLoaderDarwinKernel.cpp
index fcbaf7e1ed7d2..ea2cb46af009f 100644
--- a/lldb/source/Plugins/DynamicLoader/Darwin-Kernel/DynamicLoaderDarwinKernel.cpp
+++ b/lldb/source/Plugins/DynamicLoader/Darwin-Kernel/DynamicLoaderDarwinKernel.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "Plugins/ObjectFile/Mach-O/ObjectFileMachO.h"
 #include "Plugins/Platform/MacOSX/PlatformDarwinKernel.h"
 #include "lldb/Breakpoint/StoppointCallbackContext.h"
 #include "lldb/Core/Debugger.h"
@@ -713,6 +714,7 @@ void DynamicLoaderDarwinKernel::KextImageInfo::SetIsKernel(bool is_kernel) {
 
 bool DynamicLoaderDarwinKernel::KextImageInfo::LoadImageUsingMemoryModule(
     Process *process) {
+  Log *log = GetLog(LLDBLog::DynamicLoader);
   if (IsLoaded())
     return true;
 
@@ -810,6 +812,27 @@ bool DynamicLoaderDarwinKernel::KextImageInfo::LoadImageUsingMemoryModule(
       }
     }
 
+    if (m_module_sp && m_uuid.IsValid() && m_module_sp->GetUUID() == m_uuid &&
+        m_module_sp->GetObjectFile()) {
+      if (ObjectFileMachO *ondisk_objfile_macho =
+              llvm::dyn_cast<ObjectFileMachO>(m_module_sp->GetObjectFile())) {
+        if (!IsKernel() && !ondisk_objfile_macho->IsKext()) {
+          // We have a non-kext, non-kernel binary.  If we already have this
+          // loaded in the Target with load addresses, don't re-load it again.
+          ModuleSP existing_module_sp = target.GetImages().FindModule(m_uuid);
+          if (existing_module_sp &&
+              existing_module_sp->IsLoadedInTarget(&target)) {
+            LLDB_LOGF(log,
+                      "'%s' with UUID %s is not a kext or kernel, and is "
+                      "already registered in target, not loading.",
+                      m_name.c_str(), m_uuid.GetAsString().c_str());
+            // It's already loaded, return true.
+            return true;
+          }
+        }
+      }
+    }
+
     // If we managed to find a module, append it to the target's list of
     // images. If we also have a memory module, require that they have matching
     // UUIDs
@@ -837,6 +860,34 @@ bool DynamicLoaderDarwinKernel::KextImageInfo::LoadImageUsingMemoryModule(
         // it.
         const bool ignore_linkedit = !IsKernel();
 
+        // Normally a kext will have its segment load commands
+        // (LC_SEGMENT vmaddrs) corrected in memory to have their
+        // actual segment addresses.
+        // Userland proceses have their libraries updated the same way
+        // by dyld.  The Mach-O load commands in memory are the canonical
+        // addresses.
+        //
+        // If the kernel gives us a binary where the in-memory segment
+        // vmaddr is incorrect, then this binary was put in memory without
+        // updating its Mach-O load commands.  We should assume a static
+        // slide value will be applied to every segment; we don't have the
+        // correct addresses for each individual segment.
+        addr_t fixed_slide = LLDB_INVALID_ADDRESS;
+        if (ObjectFileMachO *memory_objfile_macho =
+                llvm::dyn_cast<ObjectFileMachO>(memory_object_file)) {
+          if (Section *header_sect =
+                  memory_objfile_macho->GetMachHeaderSection()) {
+            if (header_sect->GetFileAddress() != m_load_address) {
+              fixed_slide = m_load_address - header_sect->GetFileAddress();
+              LLDB_LOGF(
+                  log,
+                  "kext %s in-memory LC_SEGMENT vmaddr is not correct, using a "
+                  "fixed slide of 0x%" PRIx64,
+                  m_name.c_str(), fixed_slide);
+            }
+          }
+        }
+
         SectionList *ondisk_section_list = ondisk_object_file->GetSectionList();
         SectionList *memory_section_list = memory_object_file->GetSectionList();
         if (memory_section_list && ondisk_section_list) {
@@ -865,14 +916,20 @@ bool DynamicLoaderDarwinKernel::KextImageInfo::LoadImageUsingMemoryModule(
                   ondisk_section_sp->GetName() == g_section_name_LINKEDIT)
                 continue;
 
-              const Section *memory_section =
-                  memory_section_list
-                      ->FindSectionByName(ondisk_section_sp->GetName())
-                      .get();
-              if (memory_section) {
-                target.SetSectionLoadAddress(ondisk_section_sp,
-                                             memory_section->GetFileAddress());
-                ++num_sections_loaded;
+              if (fixed_slide != LLDB_INVALID_ADDRESS) {
+                target.SetSectionLoadAddress(
+                    ondisk_section_sp,
+                    ondisk_section_sp->GetFileAddress() + fixed_slide);
+              } else {
+                const Section *memory_section =
+                    memory_section_list
+                        ->FindSectionByName(ondisk_section_sp->GetName())
+                        .get();
+                if (memory_section) {
+                  target.SetSectionLoadAddress(
+                      ondisk_section_sp, memory_section->GetFileAddress());
+                  ++num_sections_loaded;
+                }
               }
             }
           }
diff --git a/lldb/source/Plugins/ExpressionParser/Clang/ClangASTImporter.cpp b/lldb/source/Plugins/ExpressionParser/Clang/ClangASTImporter.cpp
index c084c3fbefc5d..2826b102625fe 100644
--- a/lldb/source/Plugins/ExpressionParser/Clang/ClangASTImporter.cpp
+++ b/lldb/source/Plugins/ExpressionParser/Clang/ClangASTImporter.cpp
@@ -113,7 +113,7 @@ class DeclContextOverride {
   llvm::DenseMap<clang::Decl *, Backup> m_backups;
 
   void OverrideOne(clang::Decl *decl) {
-    if (m_backups.find(decl) != m_backups.end()) {
+    if (m_backups.contains(decl)) {
       return;
     }
 
diff --git a/lldb/source/Plugins/ExpressionParser/Clang/ClangExpressionSourceCode.cpp b/lldb/source/Plugins/ExpressionParser/Clang/ClangExpressionSourceCode.cpp
index cc3abfdfde394..527055024a768 100644
--- a/lldb/source/Plugins/ExpressionParser/Clang/ClangExpressionSourceCode.cpp
+++ b/lldb/source/Plugins/ExpressionParser/Clang/ClangExpressionSourceCode.cpp
@@ -212,7 +212,7 @@ class TokenVerifier {
   /// Returns true iff the given expression body contained a token with the
   /// given content.
   bool hasToken(llvm::StringRef token) const {
-    return m_tokens.find(token) != m_tokens.end();
+    return m_tokens.contains(token);
   }
 };
 
diff --git a/lldb/source/Plugins/ExpressionParser/Clang/IRDynamicChecks.cpp b/lldb/source/Plugins/ExpressionParser/Clang/IRDynamicChecks.cpp
index 0a4af196857ca..0549868274685 100644
--- a/lldb/source/Plugins/ExpressionParser/Clang/IRDynamicChecks.cpp
+++ b/lldb/source/Plugins/ExpressionParser/Clang/IRDynamicChecks.cpp
@@ -334,20 +334,8 @@ class ValidPointerChecker : public Instrumenter {
     else
       return false;
 
-    // Insert an instruction to cast the loaded value to int8_t*
-
-    BitCastInst *bit_cast =
-        new BitCastInst(dereferenced_ptr, GetI8PtrTy(), "", inst);
-
     // Insert an instruction to call the helper with the result
-
-    llvm::Value *arg_array[1];
-
-    arg_array[0] = bit_cast;
-
-    llvm::ArrayRef<llvm::Value *> args(arg_array, 1);
-
-    CallInst::Create(m_valid_pointer_check_func, args, "", inst);
+    CallInst::Create(m_valid_pointer_check_func, dereferenced_ptr, "", inst);
 
     return true;
   }
@@ -425,16 +413,11 @@ class ObjcObjectChecker : public Instrumenter {
     assert(target_object);
     assert(selector);
 
-    // Insert an instruction to cast the receiver id to int8_t*
-
-    BitCastInst *bit_cast =
-        new BitCastInst(target_object, GetI8PtrTy(), "", inst);
-
     // Insert an instruction to call the helper with the result
 
     llvm::Value *arg_array[2];
 
-    arg_array[0] = bit_cast;
+    arg_array[0] = target_object;
     arg_array[1] = selector;
 
     ArrayRef<llvm::Value *> args(arg_array, 2);
diff --git a/lldb/source/Plugins/ExpressionParser/Clang/IRForTarget.cpp b/lldb/source/Plugins/ExpressionParser/Clang/IRForTarget.cpp
index ec0243d92099b..6fe46295770f8 100644
--- a/lldb/source/Plugins/ExpressionParser/Clang/IRForTarget.cpp
+++ b/lldb/source/Plugins/ExpressionParser/Clang/IRForTarget.cpp
@@ -476,8 +476,7 @@ bool IRForTarget::RewriteObjCConstString(llvm::GlobalVariable *ns_str,
     string_array = dyn_cast<ConstantDataSequential>(cstr->getInitializer());
 
   Constant *alloc_arg = Constant::getNullValue(i8_ptr_ty);
-  Constant *bytes_arg = cstr ? ConstantExpr::getBitCast(cstr, i8_ptr_ty)
-                             : Constant::getNullValue(i8_ptr_ty);
+  Constant *bytes_arg = cstr ? cstr : Constant::getNullValue(i8_ptr_ty);
   Constant *numBytes_arg = ConstantInt::get(
       m_intptr_ty, cstr ? (string_array->getNumElements() - 1) * string_array->getElementByteSize() : 0, false);
  int encoding_flags = 0;
@@ -821,17 +820,9 @@ bool IRForTarget::RewriteObjCSelector(Instruction *selector_load) {
                           ConstantExpr::getIntToPtr(srN_addr_int, srN_ptr_ty)};
   }
 
-  Value *argument_array[1];
-
-  Constant *omvn_pointer = ConstantExpr::getBitCast(
-      _objc_meth_var_name_, Type::getInt8PtrTy(m_module->getContext()));
-
-  argument_array[0] = omvn_pointer;
-
-  ArrayRef<Value *> srN_arguments(argument_array, 1);
-
-  CallInst *srN_call = CallInst::Create(m_sel_registerName, srN_arguments,
-                                        "sel_registerName", selector_load);
+  CallInst *srN_call =
+      CallInst::Create(m_sel_registerName, _objc_meth_var_name_,
+                       "sel_registerName", selector_load);
 
   // Replace the load with the call in all users
 
@@ -887,14 +878,12 @@ bool IRForTarget::RewriteObjCClassReference(Instruction *class_load) {
   // Unpack the class name from the reference.  In LLVM IR, a reference to an
   // Objective-C class gets represented as
   //
-  // %tmp     = load %struct._objc_class*,
-  //            %struct._objc_class** @OBJC_CLASS_REFERENCES_, align 4
+  // %tmp     = load ptr, ptr @OBJC_CLASS_REFERENCES_, align 4
   //
-  // @"OBJC_CLASS_REFERENCES_ is a bitcast of a character array called
+  // @OBJC_CLASS_REFERENCES_ is a reference to a character array called
   // @OBJC_CLASS_NAME_. @OBJC_CLASS_NAME contains the string.
 
-  // Find the pointer's initializer (a ConstantExpr with opcode BitCast) and
-  // get the string from its target
+  // Find the pointer's initializer and get the string from its target
 
   GlobalVariable *_objc_class_references_ =
       dyn_cast<GlobalVariable>(load->getPointerOperand());
@@ -903,23 +892,10 @@ bool IRForTarget::RewriteObjCClassReference(Instruction *class_load) {
       !_objc_class_references_->hasInitializer())
     return false;
 
-  Constant *ocr_initializer = _objc_class_references_->getInitializer();
-
-  ConstantExpr *ocr_initializer_expr = dyn_cast<ConstantExpr>(ocr_initializer);
-
-  if (!ocr_initializer_expr ||
-      ocr_initializer_expr->getOpcode() != Instruction::BitCast)
-    return false;
-
-  Value *ocr_initializer_base = ocr_initializer_expr->getOperand(0);
-
-  if (!ocr_initializer_base)
-    return false;
-
   // Find the string's initializer (a ConstantArray) and get the string from it
 
   GlobalVariable *_objc_class_name_ =
-      dyn_cast<GlobalVariable>(ocr_initializer_base);
+      dyn_cast<GlobalVariable>(_objc_class_references_->getInitializer());
 
   if (!_objc_class_name_ || !_objc_class_name_->hasInitializer())
     return false;
@@ -971,16 +947,7 @@ bool IRForTarget::RewriteObjCClassReference(Instruction *class_load) {
                        ConstantExpr::getIntToPtr(ogC_addr_int, ogC_ptr_ty)};
   }
 
-  Value *argument_array[1];
-
-  Constant *ocn_pointer = ConstantExpr::getBitCast(
-      _objc_class_name_, Type::getInt8PtrTy(m_module->getContext()));
-
-  argument_array[0] = ocn_pointer;
-
-  ArrayRef<Value *> ogC_arguments(argument_array, 1);
-
-  CallInst *ogC_call = CallInst::Create(m_objc_getClass, ogC_arguments,
+  CallInst *ogC_call = CallInst::Create(m_objc_getClass, _objc_class_name_,
                                         "objc_getClass", class_load);
 
   // Replace the load with the call in all users
@@ -1419,19 +1386,7 @@ bool IRForTarget::ResolveExternals(Function &llvm_function) {
 }
 
 static bool isGuardVariableRef(Value *V) {
-  Constant *Old = dyn_cast<Constant>(V);
-
-  if (!Old)
-    return false;
-
-  if (auto CE = dyn_cast<ConstantExpr>(V)) {
-    if (CE->getOpcode() != Instruction::BitCast)
-      return false;
-
-    Old = CE->getOperand(0);
-  }
-
-  GlobalVariable *GV = dyn_cast<GlobalVariable>(Old);
+  GlobalVariable *GV = dyn_cast<GlobalVariable>(V);
 
   if (!GV || !GV->hasName() || !isGuardVariableSymbol(GV->getName()))
     return false;
@@ -1732,19 +1687,12 @@ bool IRForTarget::ReplaceVariables(Function &llvm_function) {
                 int8Ty, argument, offset_int, "", entry_instruction);
 
             if (name == m_result_name && !m_result_is_pointer) {
-              BitCastInst *bit_cast = new BitCastInst(
-                  get_element_ptr, value->getType()->getPointerTo(), "",
-                  entry_instruction);
-
-              LoadInst *load = new LoadInst(value->getType(), bit_cast, "",
-                                            entry_instruction);
+              LoadInst *load = new LoadInst(value->getType(), get_element_ptr,
+                                            "", entry_instruction);
 
               return load;
             } else {
-              BitCastInst *bit_cast = new BitCastInst(
-                  get_element_ptr, value->getType(), "", entry_instruction);
-
-              return bit_cast;
+              return get_element_ptr;
             }
           });
 
diff --git a/lldb/source/Plugins/InstrumentationRuntime/TSan/InstrumentationRuntimeTSan.cpp b/lldb/source/Plugins/InstrumentationRuntime/TSan/InstrumentationRuntimeTSan.cpp
index 425b0baa453f8..873704723cced 100644
--- a/lldb/source/Plugins/InstrumentationRuntime/TSan/InstrumentationRuntimeTSan.cpp
+++ b/lldb/source/Plugins/InstrumentationRuntime/TSan/InstrumentationRuntimeTSan.cpp
@@ -87,6 +87,9 @@ extern "C"
     void *dlsym(void* handle, const char* symbol);
     int (*ptr__tsan_get_report_loc_object_type)(void *report, unsigned long idx, const char **object_type);
 }
+)";
+
+const char *thread_sanitizer_retrieve_report_data_command = R"(
 
 const int REPORT_TRACE_SIZE = 128;
 const int REPORT_ARRAY_SIZE = 4;
@@ -154,11 +157,7 @@ struct data {
         int idx;
         int tid;
     } unique_tids[REPORT_ARRAY_SIZE];
-};
-)";
-
-const char *thread_sanitizer_retrieve_report_data_command = R"(
-data t = {0};
+} t = {0};
 
 ptr__tsan_get_report_loc_object_type = (typeof(ptr__tsan_get_report_loc_object_type))(void *)dlsym((void*)-2 /*RTLD_DEFAULT*/, "__tsan_get_report_loc_object_type");
 
diff --git a/lldb/source/Plugins/InstrumentationRuntime/UBSan/InstrumentationRuntimeUBSan.cpp b/lldb/source/Plugins/InstrumentationRuntime/UBSan/InstrumentationRuntimeUBSan.cpp
index 7ea8b4697d204..0264a58e3a931 100644
--- a/lldb/source/Plugins/InstrumentationRuntime/UBSan/InstrumentationRuntimeUBSan.cpp
+++ b/lldb/source/Plugins/InstrumentationRuntime/UBSan/InstrumentationRuntimeUBSan.cpp
@@ -67,19 +67,18 @@ __ubsan_get_current_report_data(const char **OutIssueKind,
     const char **OutMessage, const char **OutFilename, unsigned *OutLine,
     unsigned *OutCol, char **OutMemoryAddr);
 }
+)";
 
-struct data {
+static const char *ub_sanitizer_retrieve_report_data_command = R"(
+struct {
   const char *issue_kind;
   const char *message;
   const char *filename;
   unsigned line;
   unsigned col;
   char *memory_addr;
-};
-)";
+} t;
 
-static const char *ub_sanitizer_retrieve_report_data_command = R"(
-data t;
 __ubsan_get_current_report_data(&t.issue_kind, &t.message, &t.filename, &t.line,
                                 &t.col, &t.memory_addr);
 t;
diff --git a/lldb/source/Plugins/Language/CPlusPlus/CPlusPlusNameParser.cpp b/lldb/source/Plugins/Language/CPlusPlus/CPlusPlusNameParser.cpp
index 617181a35d1d6..1f7d8a1b8598c 100644
--- a/lldb/source/Plugins/Language/CPlusPlus/CPlusPlusNameParser.cpp
+++ b/lldb/source/Plugins/Language/CPlusPlus/CPlusPlusNameParser.cpp
@@ -172,7 +172,7 @@ CPlusPlusNameParser::ParseFuncPtr(bool expect_return_type) {
   //
   // Consume inner function name. This will fail unless
   // we stripped all the pointers on the left hand side
-  // of the funciton name.
+  // of the function name.
   {
     Bookmark before_inner_function_pos = SetBookmark();
     auto maybe_inner_function_name = ParseFunctionImpl(false);
diff --git a/lldb/source/Plugins/Language/ClangCommon/ClangHighlighter.cpp b/lldb/source/Plugins/Language/ClangCommon/ClangHighlighter.cpp
index 6b78ce5de697a..6e56e29f8c31c 100644
--- a/lldb/source/Plugins/Language/ClangCommon/ClangHighlighter.cpp
+++ b/lldb/source/Plugins/Language/ClangCommon/ClangHighlighter.cpp
@@ -23,7 +23,7 @@
 using namespace lldb_private;
 
 bool ClangHighlighter::isKeyword(llvm::StringRef token) const {
-  return keywords.find(token) != keywords.end();
+  return keywords.contains(token);
 }
 
 ClangHighlighter::ClangHighlighter() {
diff --git a/lldb/source/Plugins/LanguageRuntime/CPlusPlus/CPPLanguageRuntime.cpp b/lldb/source/Plugins/LanguageRuntime/CPlusPlus/CPPLanguageRuntime.cpp
index 91c5283e5b459..0028a51412873 100644
--- a/lldb/source/Plugins/LanguageRuntime/CPlusPlus/CPPLanguageRuntime.cpp
+++ b/lldb/source/Plugins/LanguageRuntime/CPlusPlus/CPPLanguageRuntime.cpp
@@ -28,7 +28,6 @@
 #include "lldb/Target/StackFrame.h"
 #include "lldb/Target/ThreadPlanRunToAddress.h"
 #include "lldb/Target/ThreadPlanStepInRange.h"
-#include "lldb/Utility/RegularExpression.h"
 #include "lldb/Utility/Timer.h"
 
 using namespace lldb;
@@ -41,17 +40,6 @@ char CPPLanguageRuntime::ID = 0;
 CPPLanguageRuntime::CPPLanguageRuntime(Process *process)
     : LanguageRuntime(process) {}
 
-bool CPPLanguageRuntime::ShouldHideVariable(llvm::StringRef name) const {
-  // Matches the global function objects in std::ranges/std::ranges::views
-  // E.g.,
-  //   std::__1::ranges::views::__cpo::take
-  //   std::__1::ranges::__cpo::max_element
-  static RegularExpression ignore_global_ranges_pattern(
-      "std::__[[:alnum:]]+::ranges(::views)*::__cpo");
-
-  return ignore_global_ranges_pattern.Execute(name);
-}
-
 bool CPPLanguageRuntime::IsAllowedRuntimeValue(ConstString name) {
   return name == g_this;
 }
diff --git a/lldb/source/Plugins/LanguageRuntime/CPlusPlus/CPPLanguageRuntime.h b/lldb/source/Plugins/LanguageRuntime/CPlusPlus/CPPLanguageRuntime.h
index 097d1da6e909a..1be58b7bf9ea9 100644
--- a/lldb/source/Plugins/LanguageRuntime/CPlusPlus/CPPLanguageRuntime.h
+++ b/lldb/source/Plugins/LanguageRuntime/CPlusPlus/CPPLanguageRuntime.h
@@ -77,8 +77,6 @@ class CPPLanguageRuntime : public LanguageRuntime {
   lldb::ThreadPlanSP GetStepThroughTrampolinePlan(Thread &thread,
                                                   bool stop_others) override;
 
-  bool ShouldHideVariable(llvm::StringRef name) const override;
-
   bool IsAllowedRuntimeValue(ConstString name) override;
 protected:
   // Classes that inherit from CPPLanguageRuntime can see and modify these
diff --git a/lldb/source/Plugins/LanguageRuntime/CPlusPlus/ItaniumABI/ItaniumABILanguageRuntime.cpp b/lldb/source/Plugins/LanguageRuntime/CPlusPlus/ItaniumABI/ItaniumABILanguageRuntime.cpp
index f05997fa03af8..2879aa64fd7c3 100644
--- a/lldb/source/Plugins/LanguageRuntime/CPlusPlus/ItaniumABI/ItaniumABILanguageRuntime.cpp
+++ b/lldb/source/Plugins/LanguageRuntime/CPlusPlus/ItaniumABI/ItaniumABILanguageRuntime.cpp
@@ -583,7 +583,11 @@ ValueObjectSP ItaniumABILanguageRuntime::GetExceptionObjectForThread(
   ValueObjectSP exception = ValueObject::CreateValueObjectFromData(
       "exception", exception_isw.GetAsData(m_process->GetByteOrder()), exe_ctx,
       voidstar);
-  exception = exception->GetDynamicValue(eDynamicDontRunTarget);
+  ValueObjectSP dyn_exception 
+      = exception->GetDynamicValue(eDynamicDontRunTarget);
+  // If we succeed in making a dynamic value, return that:
+  if (dyn_exception) 
+     return dyn_exception;
 
   return exception;
 }
diff --git a/lldb/source/Plugins/MemoryHistory/asan/MemoryHistoryASan.cpp b/lldb/source/Plugins/MemoryHistory/asan/MemoryHistoryASan.cpp
index aaead88369b20..c1f5970badbde 100644
--- a/lldb/source/Plugins/MemoryHistory/asan/MemoryHistoryASan.cpp
+++ b/lldb/source/Plugins/MemoryHistory/asan/MemoryHistoryASan.cpp
@@ -67,8 +67,11 @@ const char *memory_history_asan_command_prefix = R"(
         size_t __asan_get_alloc_stack(void *addr, void **trace, size_t size, int *thread_id);
         size_t __asan_get_free_stack(void *addr, void **trace, size_t size, int *thread_id);
     }
+)";
 
-    struct data {
+const char *memory_history_asan_command_format =
+    R"(
+    struct {
         void *alloc_trace[256];
         size_t alloc_count;
         int alloc_tid;
@@ -76,12 +79,7 @@ const char *memory_history_asan_command_prefix = R"(
         void *free_trace[256];
         size_t free_count;
         int free_tid;
-    };
-)";
-
-const char *memory_history_asan_command_format =
-    R"(
-    data t;
+    } t;
 
     t.alloc_count = __asan_get_alloc_stack((void *)0x%)" PRIx64
     R"(, t.alloc_trace, 256, &t.alloc_tid);
diff --git a/lldb/source/Plugins/ObjectFile/CMakeLists.txt b/lldb/source/Plugins/ObjectFile/CMakeLists.txt
index 042ea826aa3b1..6b5ec86399117 100644
--- a/lldb/source/Plugins/ObjectFile/CMakeLists.txt
+++ b/lldb/source/Plugins/ObjectFile/CMakeLists.txt
@@ -1,8 +1,10 @@
 add_subdirectory(Breakpad)
 add_subdirectory(ELF)
 add_subdirectory(JIT)
+add_subdirectory(JSON)
 add_subdirectory(Mach-O)
 add_subdirectory(Minidump)
 add_subdirectory(PDB)
 add_subdirectory(PECOFF)
+add_subdirectory(Placeholder)
 add_subdirectory(wasm)
diff --git a/lldb/source/Plugins/ObjectFile/ELF/ObjectFileELF.cpp b/lldb/source/Plugins/ObjectFile/ELF/ObjectFileELF.cpp
index 9131367bf2230..5b75738e070c3 100644
--- a/lldb/source/Plugins/ObjectFile/ELF/ObjectFileELF.cpp
+++ b/lldb/source/Plugins/ObjectFile/ELF/ObjectFileELF.cpp
@@ -114,13 +114,13 @@ class ELFRelocation {
 
   static unsigned RelocSymbol64(const ELFRelocation &rel);
 
-  static unsigned RelocOffset32(const ELFRelocation &rel);
+  static elf_addr RelocOffset32(const ELFRelocation &rel);
 
-  static unsigned RelocOffset64(const ELFRelocation &rel);
+  static elf_addr RelocOffset64(const ELFRelocation &rel);
 
-  static unsigned RelocAddend32(const ELFRelocation &rel);
+  static elf_sxword RelocAddend32(const ELFRelocation &rel);
 
-  static unsigned RelocAddend64(const ELFRelocation &rel);
+  static elf_sxword RelocAddend64(const ELFRelocation &rel);
 
   bool IsRela() { return (reloc.is<ELFRela *>()); }
 
@@ -185,28 +185,28 @@ unsigned ELFRelocation::RelocSymbol64(const ELFRelocation &rel) {
     return ELFRela::RelocSymbol64(*rel.reloc.get<ELFRela *>());
 }
 
-unsigned ELFRelocation::RelocOffset32(const ELFRelocation &rel) {
+elf_addr ELFRelocation::RelocOffset32(const ELFRelocation &rel) {
   if (rel.reloc.is<ELFRel *>())
     return rel.reloc.get<ELFRel *>()->r_offset;
   else
     return rel.reloc.get<ELFRela *>()->r_offset;
 }
 
-unsigned ELFRelocation::RelocOffset64(const ELFRelocation &rel) {
+elf_addr ELFRelocation::RelocOffset64(const ELFRelocation &rel) {
   if (rel.reloc.is<ELFRel *>())
     return rel.reloc.get<ELFRel *>()->r_offset;
   else
     return rel.reloc.get<ELFRela *>()->r_offset;
 }
 
-unsigned ELFRelocation::RelocAddend32(const ELFRelocation &rel) {
+elf_sxword ELFRelocation::RelocAddend32(const ELFRelocation &rel) {
   if (rel.reloc.is<ELFRel *>())
     return 0;
   else
     return rel.reloc.get<ELFRela *>()->r_addend;
 }
 
-unsigned ELFRelocation::RelocAddend64(const ELFRelocation &rel) {
+elf_sxword  ELFRelocation::RelocAddend64(const ELFRelocation &rel) {
   if (rel.reloc.is<ELFRel *>())
     return 0;
   else
@@ -2593,6 +2593,50 @@ ObjectFileELF::ParseTrampolineSymbols(Symtab *symbol_table, user_id_t start_id,
                              rel_data, symtab_data, strtab_data);
 }
 
+static void ApplyELF64ABS64Relocation(Symtab *symtab, ELFRelocation &rel,
+                                      DataExtractor &debug_data,
+                                      Section *rel_section) {
+  Symbol *symbol = symtab->FindSymbolByID(ELFRelocation::RelocSymbol64(rel));
+  if (symbol) {
+    addr_t value = symbol->GetAddressRef().GetFileAddress();
+    DataBufferSP &data_buffer_sp = debug_data.GetSharedDataBuffer();
+    // ObjectFileELF creates a WritableDataBuffer in CreateInstance.
+    WritableDataBuffer *data_buffer =
+        llvm::cast<WritableDataBuffer>(data_buffer_sp.get());
+    uint64_t *dst = reinterpret_cast<uint64_t *>(
+        data_buffer->GetBytes() + rel_section->GetFileOffset() +
+        ELFRelocation::RelocOffset64(rel));
+    uint64_t val_offset = value + ELFRelocation::RelocAddend64(rel);
+    memcpy(dst, &val_offset, sizeof(uint64_t));
+  }
+}
+
+static void ApplyELF64ABS32Relocation(Symtab *symtab, ELFRelocation &rel,
+                                      DataExtractor &debug_data,
+                                      Section *rel_section, bool is_signed) {
+  Symbol *symbol = symtab->FindSymbolByID(ELFRelocation::RelocSymbol64(rel));
+  if (symbol) {
+    addr_t value = symbol->GetAddressRef().GetFileAddress();
+    value += ELFRelocation::RelocAddend32(rel);
+    if ((!is_signed && (value > UINT32_MAX)) ||
+        (is_signed &&
+         ((int64_t)value > INT32_MAX || (int64_t)value < INT32_MIN))) {
+      Log *log = GetLog(LLDBLog::Modules);
+      LLDB_LOGF(log, "Failed to apply debug info relocations");
+      return;
+    }
+    uint32_t truncated_addr = (value & 0xFFFFFFFF);
+    DataBufferSP &data_buffer_sp = debug_data.GetSharedDataBuffer();
+    // ObjectFileELF creates a WritableDataBuffer in CreateInstance.
+    WritableDataBuffer *data_buffer =
+        llvm::cast<WritableDataBuffer>(data_buffer_sp.get());
+    uint32_t *dst = reinterpret_cast<uint32_t *>(
+        data_buffer->GetBytes() + rel_section->GetFileOffset() +
+        ELFRelocation::RelocOffset32(rel));
+    memcpy(dst, &truncated_addr, sizeof(uint32_t));
+  }
+}
+
 unsigned ObjectFileELF::ApplyRelocations(
     Symtab *symtab, const ELFHeader *hdr, const ELFSectionHeader *rel_hdr,
     const ELFSectionHeader *symtab_hdr, const ELFSectionHeader *debug_hdr,
@@ -2656,55 +2700,50 @@ unsigned ObjectFileELF::ApplyRelocations(
                                  reloc_type(rel));
       }
     } else {
-      switch (reloc_type(rel)) {
-      case R_AARCH64_ABS64:
-      case R_X86_64_64: {
-        symbol = symtab->FindSymbolByID(reloc_symbol(rel));
-        if (symbol) {
-          addr_t value = symbol->GetAddressRef().GetFileAddress();
-          DataBufferSP &data_buffer_sp = debug_data.GetSharedDataBuffer();
-          // ObjectFileELF creates a WritableDataBuffer in CreateInstance.
-          WritableDataBuffer *data_buffer =
-              llvm::cast<WritableDataBuffer>(data_buffer_sp.get());
-          uint64_t *dst = reinterpret_cast<uint64_t *>(
-              data_buffer->GetBytes() + rel_section->GetFileOffset() +
-              ELFRelocation::RelocOffset64(rel));
-          uint64_t val_offset = value + ELFRelocation::RelocAddend64(rel);
-          memcpy(dst, &val_offset, sizeof(uint64_t));
+      switch (hdr->e_machine) {
+      case llvm::ELF::EM_AARCH64:
+        switch (reloc_type(rel)) {
+        case R_AARCH64_ABS64:
+          ApplyELF64ABS64Relocation(symtab, rel, debug_data, rel_section);
+          break;
+        case R_AARCH64_ABS32:
+          ApplyELF64ABS32Relocation(symtab, rel, debug_data, rel_section, true);
+          break;
+        default:
+          assert(false && "unexpected relocation type");
         }
         break;
-      }
-      case R_X86_64_32:
-      case R_X86_64_32S:
-      case R_AARCH64_ABS32: {
-        symbol = symtab->FindSymbolByID(reloc_symbol(rel));
-        if (symbol) {
-          addr_t value = symbol->GetAddressRef().GetFileAddress();
-          value += ELFRelocation::RelocAddend32(rel);
-          if ((reloc_type(rel) == R_X86_64_32 && (value > UINT32_MAX)) ||
-              (reloc_type(rel) == R_X86_64_32S &&
-               ((int64_t)value > INT32_MAX && (int64_t)value < INT32_MIN)) ||
-              (reloc_type(rel) == R_AARCH64_ABS32 &&
-               ((int64_t)value > INT32_MAX && (int64_t)value < INT32_MIN))) {
-            Log *log = GetLog(LLDBLog::Modules);
-            LLDB_LOGF(log, "Failed to apply debug info relocations");
-            break;
-          }
-          uint32_t truncated_addr = (value & 0xFFFFFFFF);
-          DataBufferSP &data_buffer_sp = debug_data.GetSharedDataBuffer();
-          // ObjectFileELF creates a WritableDataBuffer in CreateInstance.
-          WritableDataBuffer *data_buffer =
-              llvm::cast<WritableDataBuffer>(data_buffer_sp.get());
-          uint32_t *dst = reinterpret_cast<uint32_t *>(
-              data_buffer->GetBytes() + rel_section->GetFileOffset() +
-              ELFRelocation::RelocOffset32(rel));
-          memcpy(dst, &truncated_addr, sizeof(uint32_t));
+      case llvm::ELF::EM_LOONGARCH:
+        switch (reloc_type(rel)) {
+        case R_LARCH_64:
+          ApplyELF64ABS64Relocation(symtab, rel, debug_data, rel_section);
+          break;
+        case R_LARCH_32:
+          ApplyELF64ABS32Relocation(symtab, rel, debug_data, rel_section, true);
+          break;
+        default:
+          assert(false && "unexpected relocation type");
+        }
+        break;
+      case llvm::ELF::EM_X86_64:
+        switch (reloc_type(rel)) {
+        case R_X86_64_64:
+          ApplyELF64ABS64Relocation(symtab, rel, debug_data, rel_section);
+          break;
+        case R_X86_64_32:
+          ApplyELF64ABS32Relocation(symtab, rel, debug_data, rel_section,
+                                    false);
+          break;
+        case R_X86_64_32S:
+          ApplyELF64ABS32Relocation(symtab, rel, debug_data, rel_section, true);
+          break;
+        case R_X86_64_PC32:
+        default:
+          assert(false && "unexpected relocation type");
         }
         break;
-      }
-      case R_X86_64_PC32:
       default:
-        assert(false && "unexpected relocation type");
+        assert(false && "unsupported machine");
       }
     }
   }
diff --git a/lldb/source/Plugins/ObjectFile/JSON/CMakeLists.txt b/lldb/source/Plugins/ObjectFile/JSON/CMakeLists.txt
new file mode 100644
index 0000000000000..73a8697d13d15
--- /dev/null
+++ b/lldb/source/Plugins/ObjectFile/JSON/CMakeLists.txt
@@ -0,0 +1,12 @@
+add_lldb_library(lldbPluginObjectFileJSON PLUGIN
+  ObjectFileJSON.cpp
+
+  LINK_LIBS
+    lldbCore
+    lldbHost
+    lldbSymbol
+    lldbUtility
+  LINK_COMPONENTS
+    Support
+    TargetParser
+  )
diff --git a/lldb/source/Plugins/ObjectFile/JSON/ObjectFileJSON.cpp b/lldb/source/Plugins/ObjectFile/JSON/ObjectFileJSON.cpp
new file mode 100644
index 0000000000000..7cd836ab2460c
--- /dev/null
+++ b/lldb/source/Plugins/ObjectFile/JSON/ObjectFileJSON.cpp
@@ -0,0 +1,176 @@
+//===-- ObjectFileJSON.cpp ------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "Plugins/ObjectFile/JSON/ObjectFileJSON.h"
+#include "lldb/Core/Module.h"
+#include "lldb/Core/ModuleSpec.h"
+#include "lldb/Core/PluginManager.h"
+#include "lldb/Core/Section.h"
+#include "lldb/Symbol/Symbol.h"
+#include "lldb/Utility/LLDBLog.h"
+#include "lldb/Utility/Log.h"
+#include "llvm/ADT/DenseSet.h"
+#include <optional>
+
+using namespace llvm;
+using namespace lldb;
+using namespace lldb_private;
+
+LLDB_PLUGIN_DEFINE(ObjectFileJSON)
+
+char ObjectFileJSON::ID;
+
+void ObjectFileJSON::Initialize() {
+  PluginManager::RegisterPlugin(GetPluginNameStatic(),
+                                GetPluginDescriptionStatic(), CreateInstance,
+                                CreateMemoryInstance, GetModuleSpecifications);
+}
+
+void ObjectFileJSON::Terminate() {
+  PluginManager::UnregisterPlugin(CreateInstance);
+}
+
+ObjectFile *
+ObjectFileJSON::CreateInstance(const ModuleSP &module_sp, DataBufferSP data_sp,
+                               offset_t data_offset, const FileSpec *file,
+                               offset_t file_offset, offset_t length) {
+  if (!data_sp) {
+    data_sp = MapFileData(*file, length, file_offset);
+    if (!data_sp)
+      return nullptr;
+    data_offset = 0;
+  }
+
+  if (!MagicBytesMatch(data_sp, 0, data_sp->GetByteSize()))
+    return nullptr;
+
+  if (data_sp->GetByteSize() < length) {
+    data_sp = MapFileData(*file, length, file_offset);
+    if (!data_sp)
+      return nullptr;
+    data_offset = 0;
+  }
+
+  auto text =
+      llvm::StringRef(reinterpret_cast<const char *>(data_sp->GetBytes()));
+
+  Expected<json::Value> json = json::parse(text);
+  if (!json) {
+    llvm::consumeError(json.takeError());
+    return nullptr;
+  }
+
+  json::Path::Root root;
+  Header header;
+  if (!fromJSON(*json, header, root))
+    return nullptr;
+
+  ArchSpec arch(header.triple);
+  UUID uuid;
+  uuid.SetFromStringRef(header.uuid);
+
+  Body body;
+  fromJSON(*json, body, root);
+
+  return new ObjectFileJSON(module_sp, data_sp, data_offset, file, file_offset,
+                            length, std::move(arch), std::move(uuid),
+                            std::move(body.symbols));
+}
+
+ObjectFile *ObjectFileJSON::CreateMemoryInstance(const ModuleSP &module_sp,
+                                                 WritableDataBufferSP data_sp,
+                                                 const ProcessSP &process_sp,
+                                                 addr_t header_addr) {
+  return nullptr;
+}
+
+size_t ObjectFileJSON::GetModuleSpecifications(
+    const FileSpec &file, DataBufferSP &data_sp, offset_t data_offset,
+    offset_t file_offset, offset_t length, ModuleSpecList &specs) {
+
+  if (!MagicBytesMatch(data_sp, data_offset, data_sp->GetByteSize()))
+    return 0;
+
+  auto text =
+      llvm::StringRef(reinterpret_cast<const char *>(data_sp->GetBytes()));
+
+  Expected<json::Value> json = json::parse(text);
+  if (!json) {
+    llvm::consumeError(json.takeError());
+    return 0;
+  }
+
+  json::Path::Root root;
+  Header header;
+  if (!fromJSON(*json, header, root))
+    return 0;
+
+  ArchSpec arch(header.triple);
+  UUID uuid;
+  uuid.SetFromStringRef(header.uuid);
+
+  ModuleSpec spec(file, std::move(arch));
+  spec.GetUUID() = std::move(uuid);
+  specs.Append(spec);
+  return 1;
+}
+
+ObjectFileJSON::ObjectFileJSON(const ModuleSP &module_sp, DataBufferSP &data_sp,
+                               offset_t data_offset, const FileSpec *file,
+                               offset_t offset, offset_t length, ArchSpec arch,
+                               UUID uuid, std::vector<JSONSymbol> symbols)
+    : ObjectFile(module_sp, file, offset, length, data_sp, data_offset),
+      m_arch(std::move(arch)), m_uuid(std::move(uuid)),
+      m_symbols(std::move(symbols)) {}
+
+bool ObjectFileJSON::ParseHeader() {
+  // We already parsed the header during initialization.
+  return true;
+}
+
+void ObjectFileJSON::ParseSymtab(Symtab &symtab) {
+  Log *log = GetLog(LLDBLog::Symbols);
+  SectionList *section_list = GetModule()->GetSectionList();
+  for (JSONSymbol json_symbol : m_symbols) {
+    llvm::Expected<Symbol> symbol = Symbol::FromJSON(json_symbol, section_list);
+    if (!symbol) {
+      LLDB_LOG_ERROR(log, symbol.takeError(), "invalid symbol");
+      continue;
+    }
+    symtab.AddSymbol(*symbol);
+  }
+  symtab.Finalize();
+}
+
+void ObjectFileJSON::CreateSections(SectionList &unified_section_list) {}
+
+bool ObjectFileJSON::MagicBytesMatch(DataBufferSP data_sp,
+                                     lldb::addr_t data_offset,
+                                     lldb::addr_t data_length) {
+  DataExtractor data;
+  data.SetData(data_sp, data_offset, data_length);
+  lldb::offset_t offset = 0;
+  uint32_t magic = data.GetU8(&offset);
+  return magic == '{';
+}
+
+namespace lldb_private {
+
+bool fromJSON(const json::Value &value, ObjectFileJSON::Header &header,
+              json::Path path) {
+  json::ObjectMapper o(value, path);
+  return o && o.map("triple", header.triple) && o.map("uuid", header.uuid);
+}
+
+bool fromJSON(const json::Value &value, ObjectFileJSON::Body &body,
+              json::Path path) {
+  json::ObjectMapper o(value, path);
+  return o && o.map("symbols", body.symbols);
+}
+
+} // namespace lldb_private
diff --git a/lldb/source/Plugins/ObjectFile/JSON/ObjectFileJSON.h b/lldb/source/Plugins/ObjectFile/JSON/ObjectFileJSON.h
new file mode 100644
index 0000000000000..cb3a7d2261800
--- /dev/null
+++ b/lldb/source/Plugins/ObjectFile/JSON/ObjectFileJSON.h
@@ -0,0 +1,119 @@
+//===-- ObjectFileJSON.h -------------------------------------- -*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLDB_SOURCE_PLUGINS_OBJECTFILE_JSON_OBJECTFILEJSON_H
+#define LLDB_SOURCE_PLUGINS_OBJECTFILE_JSON_OBJECTFILEJSON_H
+
+#include "lldb/Symbol/ObjectFile.h"
+#include "lldb/Utility/ArchSpec.h"
+#include "llvm/Support/JSON.h"
+
+namespace lldb_private {
+
+class ObjectFileJSON : public ObjectFile {
+public:
+  static void Initialize();
+  static void Terminate();
+
+  static llvm::StringRef GetPluginNameStatic() { return "JSON"; }
+
+  static const char *GetPluginDescriptionStatic() {
+    return "JSON object file reader.";
+  }
+
+  static ObjectFile *
+  CreateInstance(const lldb::ModuleSP &module_sp, lldb::DataBufferSP data_sp,
+                 lldb::offset_t data_offset, const FileSpec *file,
+                 lldb::offset_t file_offset, lldb::offset_t length);
+
+  static ObjectFile *CreateMemoryInstance(const lldb::ModuleSP &module_sp,
+                                          lldb::WritableDataBufferSP data_sp,
+                                          const lldb::ProcessSP &process_sp,
+                                          lldb::addr_t header_addr);
+
+  static size_t GetModuleSpecifications(const FileSpec &file,
+                                        lldb::DataBufferSP &data_sp,
+                                        lldb::offset_t data_offset,
+                                        lldb::offset_t file_offset,
+                                        lldb::offset_t length,
+                                        ModuleSpecList &specs);
+
+  llvm::StringRef GetPluginName() override { return GetPluginNameStatic(); }
+
+  // LLVM RTTI support
+  static char ID;
+  bool isA(const void *ClassID) const override {
+    return ClassID == &ID || ObjectFile::isA(ClassID);
+  }
+  static bool classof(const ObjectFile *obj) { return obj->isA(&ID); }
+
+  bool ParseHeader() override;
+
+  lldb::ByteOrder GetByteOrder() const override {
+    return m_arch.GetByteOrder();
+  }
+
+  bool IsExecutable() const override { return false; }
+
+  uint32_t GetAddressByteSize() const override {
+    return m_arch.GetAddressByteSize();
+  }
+
+  AddressClass GetAddressClass(lldb::addr_t file_addr) override {
+    return AddressClass::eInvalid;
+  }
+
+  void ParseSymtab(lldb_private::Symtab &symtab) override;
+
+  bool IsStripped() override { return false; }
+
+  void CreateSections(SectionList &unified_section_list) override;
+
+  void Dump(Stream *s) override {}
+
+  ArchSpec GetArchitecture() override { return m_arch; }
+
+  UUID GetUUID() override { return m_uuid; }
+
+  uint32_t GetDependentModules(FileSpecList &files) override { return 0; }
+
+  Type CalculateType() override { return eTypeDebugInfo; }
+
+  Strata CalculateStrata() override { return eStrataUser; }
+
+  static bool MagicBytesMatch(lldb::DataBufferSP data_sp, lldb::addr_t offset,
+                              lldb::addr_t length);
+
+  struct Header {
+    std::string triple;
+    std::string uuid;
+  };
+
+  struct Body {
+    std::vector<JSONSymbol> symbols;
+  };
+
+private:
+  ArchSpec m_arch;
+  UUID m_uuid;
+  std::vector<JSONSymbol> m_symbols;
+
+  ObjectFileJSON(const lldb::ModuleSP &module_sp, lldb::DataBufferSP &data_sp,
+                 lldb::offset_t data_offset, const FileSpec *file,
+                 lldb::offset_t offset, lldb::offset_t length, ArchSpec arch,
+                 UUID uuid, std::vector<JSONSymbol> symbols);
+};
+
+bool fromJSON(const llvm::json::Value &value, ObjectFileJSON::Header &header,
+              llvm::json::Path path);
+
+bool fromJSON(const llvm::json::Value &value, ObjectFileJSON::Body &body,
+              llvm::json::Path path);
+
+} // namespace lldb_private
+#endif // LLDB_SOURCE_PLUGINS_OBJECTFILE_JSON_OBJECTFILEJSON_H
diff --git a/lldb/source/Plugins/ObjectFile/Mach-O/ObjectFileMachO.cpp b/lldb/source/Plugins/ObjectFile/Mach-O/ObjectFileMachO.cpp
index eaa47359ac882..1f89db0cb5ca0 100644
--- a/lldb/source/Plugins/ObjectFile/Mach-O/ObjectFileMachO.cpp
+++ b/lldb/source/Plugins/ObjectFile/Mach-O/ObjectFileMachO.cpp
@@ -1141,6 +1141,10 @@ bool ObjectFileMachO::IsSharedCacheBinary() const {
   return m_header.flags & MH_DYLIB_IN_CACHE;
 }
 
+bool ObjectFileMachO::IsKext() const {
+  return m_header.filetype == MH_KEXT_BUNDLE;
+}
+
 uint32_t ObjectFileMachO::GetAddressByteSize() const {
   return m_data.GetAddressByteSize();
 }
@@ -4552,7 +4556,7 @@ void ObjectFileMachO::ParseSymtab(Symtab &symtab) {
   // Count how many trie symbols we'll add to the symbol table
   int trie_symbol_table_augment_count = 0;
   for (auto &e : external_sym_trie_entries) {
-    if (symbols_added.find(e.entry.address) == symbols_added.end())
+    if (!symbols_added.contains(e.entry.address))
       trie_symbol_table_augment_count++;
   }
 
@@ -4599,8 +4603,7 @@ void ObjectFileMachO::ParseSymtab(Symtab &symtab) {
   if (function_starts_count > 0) {
     uint32_t num_synthetic_function_symbols = 0;
     for (i = 0; i < function_starts_count; ++i) {
-      if (symbols_added.find(function_starts.GetEntryRef(i).addr) ==
-          symbols_added.end())
+      if (!symbols_added.contains(function_starts.GetEntryRef(i).addr))
         ++num_synthetic_function_symbols;
     }
 
@@ -4612,7 +4615,7 @@ void ObjectFileMachO::ParseSymtab(Symtab &symtab) {
       for (i = 0; i < function_starts_count; ++i) {
         const FunctionStarts::Entry *func_start_entry =
             function_starts.GetEntryAtIndex(i);
-        if (symbols_added.find(func_start_entry->addr) == symbols_added.end()) {
+        if (!symbols_added.contains(func_start_entry->addr)) {
           addr_t symbol_file_addr = func_start_entry->addr;
           uint32_t symbol_flags = 0;
           if (func_start_entry->data)
@@ -7028,6 +7031,12 @@ bool ObjectFileMachO::LoadCoreFileImages(lldb_private::Process &process) {
           &process, image.filename, image.uuid, image.load_address,
           false /* value_is_offset */, image.currently_executing,
           false /* notify */);
+      if (module_sp) {
+        // We've already set the load address in the Target,
+        // don't do any more processing on this module.
+        added_modules.Append(module_sp, false /* notify */);
+        continue;
+      }
     }
 
     // If we have a slide, we need to find the original binary
@@ -7038,6 +7047,12 @@ bool ObjectFileMachO::LoadCoreFileImages(lldb_private::Process &process) {
           &process, image.filename, image.uuid, image.slide,
           true /* value_is_offset */, image.currently_executing,
           false /* notify */);
+      if (module_sp) {
+        // We've already set the load address in the Target,
+        // don't do any more processing on this module.
+        added_modules.Append(module_sp, false /* notify */);
+        continue;
+      }
     }
 
     // Try to find the binary by UUID or filename on the local
diff --git a/lldb/source/Plugins/ObjectFile/Mach-O/ObjectFileMachO.h b/lldb/source/Plugins/ObjectFile/Mach-O/ObjectFileMachO.h
index 3b458d8da8342..d4f42a8d61e3b 100644
--- a/lldb/source/Plugins/ObjectFile/Mach-O/ObjectFileMachO.h
+++ b/lldb/source/Plugins/ObjectFile/Mach-O/ObjectFileMachO.h
@@ -90,6 +90,8 @@ class ObjectFileMachO : public lldb_private::ObjectFile {
 
   bool IsSharedCacheBinary() const;
 
+  bool IsKext() const;
+
   uint32_t GetAddressByteSize() const override;
 
   lldb_private::AddressClass GetAddressClass(lldb::addr_t file_addr) override;
@@ -151,6 +153,8 @@ class ObjectFileMachO : public lldb_private::ObjectFile {
 
   bool AllowAssemblyEmulationUnwindPlans() override;
 
+  lldb_private::Section *GetMachHeaderSection();
+
   // PluginInterface protocol
   llvm::StringRef GetPluginName() override { return GetPluginNameStatic(); }
 
@@ -193,8 +197,6 @@ class ObjectFileMachO : public lldb_private::ObjectFile {
   /// should not be used.
   void GetLLDBSharedCacheUUID(lldb::addr_t &base_addir, lldb_private::UUID &uuid);
 
-  lldb_private::Section *GetMachHeaderSection();
-
   lldb::addr_t CalculateSectionLoadAddressForMemoryImage(
       lldb::addr_t mach_header_load_address,
       const lldb_private::Section *mach_header_section,
diff --git a/lldb/source/Plugins/ObjectFile/Placeholder/CMakeLists.txt b/lldb/source/Plugins/ObjectFile/Placeholder/CMakeLists.txt
new file mode 100644
index 0000000000000..d7f8762581016
--- /dev/null
+++ b/lldb/source/Plugins/ObjectFile/Placeholder/CMakeLists.txt
@@ -0,0 +1,12 @@
+add_lldb_library(lldbPluginObjectFilePlaceholder PLUGIN
+  ObjectFilePlaceholder.cpp
+
+  LINK_LIBS
+    lldbCore
+    lldbHost
+    lldbSymbol
+    lldbTarget
+  LINK_COMPONENTS
+    Object
+    Support
+  )
diff --git a/lldb/source/Plugins/ObjectFile/Placeholder/ObjectFilePlaceholder.cpp b/lldb/source/Plugins/ObjectFile/Placeholder/ObjectFilePlaceholder.cpp
new file mode 100644
index 0000000000000..ec1f3f61892d3
--- /dev/null
+++ b/lldb/source/Plugins/ObjectFile/Placeholder/ObjectFilePlaceholder.cpp
@@ -0,0 +1,70 @@
+//===-- ObjectFilePlaceholder.cpp----------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "ObjectFilePlaceholder.h"
+
+#include "lldb/Core/Module.h"
+#include "lldb/Core/ModuleSpec.h"
+#include "lldb/Core/PluginManager.h"
+#include "lldb/Core/Section.h"
+#include "lldb/Target/SectionLoadList.h"
+#include "lldb/Target/Target.h"
+
+#include <memory>
+
+using namespace lldb;
+using namespace lldb_private;
+
+LLDB_PLUGIN_DEFINE(ObjectFilePlaceholder)
+
+ObjectFilePlaceholder::ObjectFilePlaceholder(
+    const lldb::ModuleSP &module_sp,
+    const lldb_private::ModuleSpec &module_spec, lldb::addr_t base,
+    lldb::addr_t size)
+    : ObjectFile(module_sp, &module_spec.GetFileSpec(), /*file_offset*/ 0,
+                 /*length*/ 0, /*data_sp*/ nullptr, /*data_offset*/ 0),
+      m_arch(module_spec.GetArchitecture()), m_uuid(module_spec.GetUUID()),
+      m_base(base), m_size(size) {
+  m_symtab_up = std::make_unique<lldb_private::Symtab>(this);
+}
+
+void ObjectFilePlaceholder::CreateSections(
+    lldb_private::SectionList &unified_section_list) {
+  m_sections_up = std::make_unique<lldb_private::SectionList>();
+  auto section_sp = std::make_shared<lldb_private::Section>(
+      GetModule(), this, /*sect_id*/ 0,
+      lldb_private::ConstString(".module_image"), eSectionTypeOther, m_base,
+      m_size, /*file_offset*/ 0, /*file_size*/ 0,
+      /*log2align*/ 0, /*flags*/ 0);
+  section_sp->SetPermissions(ePermissionsReadable | ePermissionsExecutable);
+  m_sections_up->AddSection(section_sp);
+  unified_section_list.AddSection(std::move(section_sp));
+}
+
+lldb_private::Address ObjectFilePlaceholder::GetBaseAddress() {
+  return lldb_private::Address(m_sections_up->GetSectionAtIndex(0), 0);
+}
+
+bool ObjectFilePlaceholder::SetLoadAddress(Target &target, addr_t value,
+                                           bool value_is_offset) {
+  assert(!value_is_offset);
+  assert(value == m_base);
+
+  // Create sections if they haven't been created already.
+  GetModule()->GetSectionList();
+  assert(m_sections_up->GetNumSections(0) == 1);
+
+  target.GetSectionLoadList().SetSectionLoadAddress(
+      m_sections_up->GetSectionAtIndex(0), m_base);
+  return true;
+}
+
+void ObjectFilePlaceholder::Dump(lldb_private::Stream *s) {
+  s->Format("Placeholder object file for {0} loaded at [{1:x}-{2:x})\n",
+            GetFileSpec(), m_base, m_base + m_size);
+}
diff --git a/lldb/source/Plugins/ObjectFile/Placeholder/ObjectFilePlaceholder.h b/lldb/source/Plugins/ObjectFile/Placeholder/ObjectFilePlaceholder.h
new file mode 100644
index 0000000000000..8798bcc5259ee
--- /dev/null
+++ b/lldb/source/Plugins/ObjectFile/Placeholder/ObjectFilePlaceholder.h
@@ -0,0 +1,74 @@
+//===-- ObjectFilePlaceholder.h ---------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLDB_SOURCE_PLUGINS_OBJECTFILE_PLACEHOLDER_OBJECTFILEPLACEHOLDER_H
+#define LLDB_SOURCE_PLUGINS_OBJECTFILE_PLACEHOLDER_OBJECTFILEPLACEHOLDER_H
+
+#include "lldb/Symbol/ObjectFile.h"
+
+#include "lldb/Target/Target.h"
+#include "lldb/Utility/ArchSpec.h"
+#include "lldb/Utility/FileSpec.h"
+#include "lldb/Utility/UUID.h"
+#include "lldb/lldb-private.h"
+
+/// A minimal ObjectFile implementation providing a dummy object file for the
+/// cases when the real module binary is not available. This allows the module
+/// to show up in "image list" and symbols to be added to it.
+class ObjectFilePlaceholder : public lldb_private::ObjectFile {
+public:
+  // Static Functions
+  static void Initialize() {}
+
+  static void Terminate() {}
+
+  static llvm::StringRef GetPluginNameStatic() { return "placeholder"; }
+
+  ObjectFilePlaceholder(const lldb::ModuleSP &module_sp,
+                        const lldb_private::ModuleSpec &module_spec,
+                        lldb::addr_t base, lldb::addr_t size);
+
+  llvm::StringRef GetPluginName() override { return GetPluginNameStatic(); }
+  bool ParseHeader() override { return true; }
+  Type CalculateType() override { return eTypeUnknown; }
+  Strata CalculateStrata() override { return eStrataUnknown; }
+  uint32_t GetDependentModules(lldb_private::FileSpecList &file_list) override {
+    return 0;
+  }
+  bool IsExecutable() const override { return false; }
+  lldb_private::ArchSpec GetArchitecture() override { return m_arch; }
+  lldb_private::UUID GetUUID() override { return m_uuid; }
+  void ParseSymtab(lldb_private::Symtab &symtab) override {}
+  bool IsStripped() override { return true; }
+  lldb::ByteOrder GetByteOrder() const override {
+    return m_arch.GetByteOrder();
+  }
+
+  uint32_t GetAddressByteSize() const override {
+    return m_arch.GetAddressByteSize();
+  }
+
+  lldb_private::Address GetBaseAddress() override;
+
+  void CreateSections(lldb_private::SectionList &unified_section_list) override;
+
+  bool SetLoadAddress(lldb_private::Target &target, lldb::addr_t value,
+                      bool value_is_offset) override;
+
+  void Dump(lldb_private::Stream *s) override;
+
+  lldb::addr_t GetBaseImageAddress() const { return m_base; }
+
+private:
+  lldb_private::ArchSpec m_arch;
+  lldb_private::UUID m_uuid;
+  lldb::addr_t m_base;
+  lldb::addr_t m_size;
+};
+
+#endif // LLDB_SOURCE_PLUGINS_OBJECTFILE_PLACEHOLDER_OBJECTFILEPLACEHOLDER_H
diff --git a/lldb/source/Plugins/Process/FreeBSD/NativeThreadFreeBSD.cpp b/lldb/source/Plugins/Process/FreeBSD/NativeThreadFreeBSD.cpp
index 8e6399bcf9c79..449ec27e0da8f 100644
--- a/lldb/source/Plugins/Process/FreeBSD/NativeThreadFreeBSD.cpp
+++ b/lldb/source/Plugins/Process/FreeBSD/NativeThreadFreeBSD.cpp
@@ -90,8 +90,7 @@ void NativeThreadFreeBSD::SetStoppedBySignal(uint32_t signo,
     case SIGBUS:
     case SIGFPE:
     case SIGILL:
-      const auto reason = GetCrashReason(*info);
-      m_stop_description = GetCrashReasonString(reason, *info);
+      m_stop_description = GetCrashReasonString(*info);
       break;
     }
   }
diff --git a/lldb/source/Plugins/Process/Linux/NativeThreadLinux.cpp b/lldb/source/Plugins/Process/Linux/NativeThreadLinux.cpp
index 9572f617e60d3..b62e9f643fa79 100644
--- a/lldb/source/Plugins/Process/Linux/NativeThreadLinux.cpp
+++ b/lldb/source/Plugins/Process/Linux/NativeThreadLinux.cpp
@@ -294,24 +294,19 @@ void NativeThreadLinux::SetStoppedBySignal(uint32_t signo,
     case SIGBUS:
     case SIGFPE:
     case SIGILL:
-      // In case of MIPS64 target, SI_KERNEL is generated for invalid 64bit
-      // address.
-      const auto reason =
-          (info->si_signo == SIGBUS && info->si_code == SI_KERNEL)
-              ? CrashReason::eInvalidAddress
-              : GetCrashReason(*info);
-      m_stop_description = GetCrashReasonString(reason, *info);
-
-      if (reason == CrashReason::eSyncTagCheckFault) {
-        AnnotateSyncTagCheckFault(info);
-      }
-
+      m_stop_description = GetCrashReasonString(*info);
+#ifndef SEGV_MTESERR
+#define SEGV_MTESERR 9
+#endif
+      if (info->si_signo == SIGSEGV && info->si_code == SEGV_MTESERR)
+        AnnotateSyncTagCheckFault(
+            reinterpret_cast<lldb::addr_t>(info->si_addr));
       break;
     }
   }
 }
 
-void NativeThreadLinux::AnnotateSyncTagCheckFault(const siginfo_t *info) {
+void NativeThreadLinux::AnnotateSyncTagCheckFault(lldb::addr_t fault_addr) {
   int32_t allocation_tag_type = 0;
   switch (GetProcess().GetArchitecture().GetMachine()) {
   // aarch64_32 deliberately not here because there's no 32 bit MTE
@@ -336,7 +331,6 @@ void NativeThreadLinux::AnnotateSyncTagCheckFault(const siginfo_t *info) {
   m_stop_description.pop_back();
 
   std::stringstream ss;
-  lldb::addr_t fault_addr = reinterpret_cast<uintptr_t>(info->si_addr);
   std::unique_ptr<MemoryTagManager> manager(std::move(details->manager));
 
   ss << " logical tag: 0x" << std::hex << manager->GetLogicalTag(fault_addr);
diff --git a/lldb/source/Plugins/Process/Linux/NativeThreadLinux.h b/lldb/source/Plugins/Process/Linux/NativeThreadLinux.h
index 030a4012f46a7..1051c3ab68bf1 100644
--- a/lldb/source/Plugins/Process/Linux/NativeThreadLinux.h
+++ b/lldb/source/Plugins/Process/Linux/NativeThreadLinux.h
@@ -110,7 +110,7 @@ class NativeThreadLinux : public NativeThreadProtocol {
   /// Extend m_stop_description with logical and allocation tag values.
   /// If there is an error along the way just add the information we were able
   /// to get.
-  void AnnotateSyncTagCheckFault(const siginfo_t *info);
+  void AnnotateSyncTagCheckFault(lldb::addr_t fault_addr);
 
   // Member Variables
   lldb::StateType m_state;
diff --git a/lldb/source/Plugins/Process/NetBSD/NativeThreadNetBSD.cpp b/lldb/source/Plugins/Process/NetBSD/NativeThreadNetBSD.cpp
index 995fe3fa78e8b..f561c21b9d91c 100644
--- a/lldb/source/Plugins/Process/NetBSD/NativeThreadNetBSD.cpp
+++ b/lldb/source/Plugins/Process/NetBSD/NativeThreadNetBSD.cpp
@@ -90,8 +90,7 @@ void NativeThreadNetBSD::SetStoppedBySignal(uint32_t signo,
     case SIGBUS:
     case SIGFPE:
     case SIGILL:
-      const auto reason = GetCrashReason(*info);
-      m_stop_description = GetCrashReasonString(reason, *info);
+      m_stop_description = GetCrashReasonString(*info);
       break;
     }
   }
diff --git a/lldb/source/Plugins/Process/POSIX/CMakeLists.txt b/lldb/source/Plugins/Process/POSIX/CMakeLists.txt
index 2db4e795b96bb..81effaaf35585 100644
--- a/lldb/source/Plugins/Process/POSIX/CMakeLists.txt
+++ b/lldb/source/Plugins/Process/POSIX/CMakeLists.txt
@@ -1,7 +1,6 @@
 add_lldb_library(lldbPluginProcessPOSIX
   CrashReason.cpp
   NativeProcessELF.cpp
-  ProcessMessage.cpp
   ProcessPOSIXLog.cpp
 
   LINK_LIBS
diff --git a/lldb/source/Plugins/Process/POSIX/CrashReason.cpp b/lldb/source/Plugins/Process/POSIX/CrashReason.cpp
index c6ede61cfe1ba..2ae7f8390575d 100644
--- a/lldb/source/Plugins/Process/POSIX/CrashReason.cpp
+++ b/lldb/source/Plugins/Process/POSIX/CrashReason.cpp
@@ -12,17 +12,50 @@
 
 #include <sstream>
 
-namespace {
+enum class CrashReason {
+  eInvalidCrashReason,
 
-void AppendFaultAddr(std::string &str, lldb::addr_t addr) {
+  // SIGSEGV crash reasons.
+  eInvalidAddress,
+  ePrivilegedAddress,
+  eBoundViolation,
+  eAsyncTagCheckFault,
+  eSyncTagCheckFault,
+
+  // SIGILL crash reasons.
+  eIllegalOpcode,
+  eIllegalOperand,
+  eIllegalAddressingMode,
+  eIllegalTrap,
+  ePrivilegedOpcode,
+  ePrivilegedRegister,
+  eCoprocessorError,
+  eInternalStackError,
+
+  // SIGBUS crash reasons,
+  eIllegalAlignment,
+  eIllegalAddress,
+  eHardwareError,
+
+  // SIGFPE crash reasons,
+  eIntegerDivideByZero,
+  eIntegerOverflow,
+  eFloatDivideByZero,
+  eFloatOverflow,
+  eFloatUnderflow,
+  eFloatInexactResult,
+  eFloatInvalidOperation,
+  eFloatSubscriptRange
+};
+
+static void AppendFaultAddr(std::string &str, lldb::addr_t addr) {
   std::stringstream ss;
   ss << " (fault address: 0x" << std::hex << addr << ")";
   str += ss.str();
 }
 
-#if defined(si_lower) && defined(si_upper)
-void AppendBounds(std::string &str, lldb::addr_t lower_bound,
-                  lldb::addr_t upper_bound, lldb::addr_t addr) {
+static void AppendBounds(std::string &str, lldb::addr_t lower_bound,
+                         lldb::addr_t upper_bound, lldb::addr_t addr) {
   llvm::raw_string_ostream stream(str);
   if ((unsigned long)addr < lower_bound)
     stream << ": lower bound violation ";
@@ -37,12 +70,9 @@ void AppendBounds(std::string &str, lldb::addr_t lower_bound,
   stream << ")";
   stream.flush();
 }
-#endif
-
-CrashReason GetCrashReasonForSIGSEGV(const siginfo_t &info) {
-  assert(info.si_signo == SIGSEGV);
 
-  switch (info.si_code) {
+static CrashReason GetCrashReasonForSIGSEGV(int code) {
+  switch (code) {
 #ifdef SI_KERNEL
   case SI_KERNEL:
     // Some platforms will occasionally send nonstandard spurious SI_KERNEL
@@ -75,10 +105,8 @@ CrashReason GetCrashReasonForSIGSEGV(const siginfo_t &info) {
   return CrashReason::eInvalidCrashReason;
 }
 
-CrashReason GetCrashReasonForSIGILL(const siginfo_t &info) {
-  assert(info.si_signo == SIGILL);
-
-  switch (info.si_code) {
+static CrashReason GetCrashReasonForSIGILL(int code) {
+  switch (code) {
   case ILL_ILLOPC:
     return CrashReason::eIllegalOpcode;
   case ILL_ILLOPN:
@@ -100,10 +128,8 @@ CrashReason GetCrashReasonForSIGILL(const siginfo_t &info) {
   return CrashReason::eInvalidCrashReason;
 }
 
-CrashReason GetCrashReasonForSIGFPE(const siginfo_t &info) {
-  assert(info.si_signo == SIGFPE);
-
-  switch (info.si_code) {
+static CrashReason GetCrashReasonForSIGFPE(int code) {
+  switch (code) {
   case FPE_INTDIV:
     return CrashReason::eIntegerDivideByZero;
   case FPE_INTOVF:
@@ -125,10 +151,8 @@ CrashReason GetCrashReasonForSIGFPE(const siginfo_t &info) {
   return CrashReason::eInvalidCrashReason;
 }
 
-CrashReason GetCrashReasonForSIGBUS(const siginfo_t &info) {
-  assert(info.si_signo == SIGBUS);
-
-  switch (info.si_code) {
+static CrashReason GetCrashReasonForSIGBUS(int code) {
+  switch (code) {
   case BUS_ADRALN:
     return CrashReason::eIllegalAlignment;
   case BUS_ADRERR:
@@ -139,27 +163,9 @@ CrashReason GetCrashReasonForSIGBUS(const siginfo_t &info) {
 
   return CrashReason::eInvalidCrashReason;
 }
-}
-
-std::string GetCrashReasonString(CrashReason reason, const siginfo_t &info) {
-  std::string str;
-
-// make sure that siginfo_t has the bound fields available.
-#if defined(si_lower) && defined(si_upper)
-  if (reason == CrashReason::eBoundViolation) {
-    str = "signal SIGSEGV";
-    AppendBounds(str, reinterpret_cast<uintptr_t>(info.si_lower),
-                 reinterpret_cast<uintptr_t>(info.si_upper),
-                 reinterpret_cast<uintptr_t>(info.si_addr));
-    return str;
-  }
-#endif
-
-  return GetCrashReasonString(reason,
-                              reinterpret_cast<uintptr_t>(info.si_addr));
-}
 
-std::string GetCrashReasonString(CrashReason reason, lldb::addr_t fault_addr) {
+static std::string GetCrashReasonString(CrashReason reason,
+                                        lldb::addr_t fault_addr) {
   std::string str;
 
   switch (reason) {
@@ -247,109 +253,50 @@ std::string GetCrashReasonString(CrashReason reason, lldb::addr_t fault_addr) {
   return str;
 }
 
-const char *CrashReasonAsString(CrashReason reason) {
-  const char *str = nullptr;
-
-  switch (reason) {
-  case CrashReason::eInvalidCrashReason:
-    str = "eInvalidCrashReason";
-    break;
-
-  // SIGSEGV crash reasons.
-  case CrashReason::eInvalidAddress:
-    str = "eInvalidAddress";
-    break;
-  case CrashReason::ePrivilegedAddress:
-    str = "ePrivilegedAddress";
-    break;
-  case CrashReason::eBoundViolation:
-    str = "eBoundViolation";
-    break;
-  case CrashReason::eAsyncTagCheckFault:
-    str = "eAsyncTagCheckFault";
-    break;
-  case CrashReason::eSyncTagCheckFault:
-    str = "eSyncTagCheckFault";
-    break;
-
-  // SIGILL crash reasons.
-  case CrashReason::eIllegalOpcode:
-    str = "eIllegalOpcode";
-    break;
-  case CrashReason::eIllegalOperand:
-    str = "eIllegalOperand";
-    break;
-  case CrashReason::eIllegalAddressingMode:
-    str = "eIllegalAddressingMode";
-    break;
-  case CrashReason::eIllegalTrap:
-    str = "eIllegalTrap";
-    break;
-  case CrashReason::ePrivilegedOpcode:
-    str = "ePrivilegedOpcode";
-    break;
-  case CrashReason::ePrivilegedRegister:
-    str = "ePrivilegedRegister";
-    break;
-  case CrashReason::eCoprocessorError:
-    str = "eCoprocessorError";
-    break;
-  case CrashReason::eInternalStackError:
-    str = "eInternalStackError";
-    break;
-
-  // SIGBUS crash reasons:
-  case CrashReason::eIllegalAlignment:
-    str = "eIllegalAlignment";
-    break;
-  case CrashReason::eIllegalAddress:
-    str = "eIllegalAddress";
-    break;
-  case CrashReason::eHardwareError:
-    str = "eHardwareError";
-    break;
-
-  // SIGFPE crash reasons:
-  case CrashReason::eIntegerDivideByZero:
-    str = "eIntegerDivideByZero";
-    break;
-  case CrashReason::eIntegerOverflow:
-    str = "eIntegerOverflow";
-    break;
-  case CrashReason::eFloatDivideByZero:
-    str = "eFloatDivideByZero";
-    break;
-  case CrashReason::eFloatOverflow:
-    str = "eFloatOverflow";
-    break;
-  case CrashReason::eFloatUnderflow:
-    str = "eFloatUnderflow";
-    break;
-  case CrashReason::eFloatInexactResult:
-    str = "eFloatInexactResult";
-    break;
-  case CrashReason::eFloatInvalidOperation:
-    str = "eFloatInvalidOperation";
-    break;
-  case CrashReason::eFloatSubscriptRange:
-    str = "eFloatSubscriptRange";
-    break;
-  }
-  return str;
-}
-
-CrashReason GetCrashReason(const siginfo_t &info) {
-  switch (info.si_signo) {
+static CrashReason GetCrashReason(int signo, int code) {
+  switch (signo) {
   case SIGSEGV:
-    return GetCrashReasonForSIGSEGV(info);
+    return GetCrashReasonForSIGSEGV(code);
   case SIGBUS:
-    return GetCrashReasonForSIGBUS(info);
+    return GetCrashReasonForSIGBUS(code);
   case SIGFPE:
-    return GetCrashReasonForSIGFPE(info);
+    return GetCrashReasonForSIGFPE(code);
   case SIGILL:
-    return GetCrashReasonForSIGILL(info);
+    return GetCrashReasonForSIGILL(code);
   }
 
   assert(false && "unexpected signal");
   return CrashReason::eInvalidCrashReason;
 }
+
+static std::string GetCrashReasonString(int signo, int code, lldb::addr_t addr,
+                                        std::optional<lldb::addr_t> lower,
+                                        std::optional<lldb::addr_t> upper) {
+  CrashReason reason = GetCrashReason(signo, code);
+
+  if (lower && upper) {
+    std::string str;
+    if (reason == CrashReason::eBoundViolation) {
+      str = "signal SIGSEGV";
+      AppendBounds(str, *lower, *upper, addr);
+      return str;
+    }
+  }
+
+  return GetCrashReasonString(reason, addr);
+}
+
+std::string GetCrashReasonString(const siginfo_t &info) {
+#if defined(si_lower) && defined(si_upper)
+  std::optional<lldb::addr_t> lower =
+      reinterpret_cast<lldb::addr_t>(info.si_lower);
+  std::optional<lldb::addr_t> upper =
+      reinterpret_cast<lldb::addr_t>(info.si_upper);
+#else
+  std::optional<lldb::addr_t> lower;
+  std::optional<lldb::addr_t> upper;
+#endif
+  return GetCrashReasonString(info.si_signo, info.si_code,
+                              reinterpret_cast<uintptr_t>(info.si_addr), lower,
+                              upper);
+}
diff --git a/lldb/source/Plugins/Process/POSIX/CrashReason.h b/lldb/source/Plugins/Process/POSIX/CrashReason.h
index 24acdc08e9007..2177726b76f0e 100644
--- a/lldb/source/Plugins/Process/POSIX/CrashReason.h
+++ b/lldb/source/Plugins/Process/POSIX/CrashReason.h
@@ -15,47 +15,6 @@
 
 #include <string>
 
-enum class CrashReason {
-  eInvalidCrashReason,
-
-  // SIGSEGV crash reasons.
-  eInvalidAddress,
-  ePrivilegedAddress,
-  eBoundViolation,
-  eAsyncTagCheckFault,
-  eSyncTagCheckFault,
-
-  // SIGILL crash reasons.
-  eIllegalOpcode,
-  eIllegalOperand,
-  eIllegalAddressingMode,
-  eIllegalTrap,
-  ePrivilegedOpcode,
-  ePrivilegedRegister,
-  eCoprocessorError,
-  eInternalStackError,
-
-  // SIGBUS crash reasons,
-  eIllegalAlignment,
-  eIllegalAddress,
-  eHardwareError,
-
-  // SIGFPE crash reasons,
-  eIntegerDivideByZero,
-  eIntegerOverflow,
-  eFloatDivideByZero,
-  eFloatOverflow,
-  eFloatUnderflow,
-  eFloatInexactResult,
-  eFloatInvalidOperation,
-  eFloatSubscriptRange
-};
-
-std::string GetCrashReasonString(CrashReason reason, lldb::addr_t fault_addr);
-std::string GetCrashReasonString(CrashReason reason, const siginfo_t &info);
-
-const char *CrashReasonAsString(CrashReason reason);
-
-CrashReason GetCrashReason(const siginfo_t &info);
+std::string GetCrashReasonString(const siginfo_t &info);
 
 #endif // #ifndef liblldb_CrashReason_H_
diff --git a/lldb/source/Plugins/Process/POSIX/ProcessMessage.cpp b/lldb/source/Plugins/Process/POSIX/ProcessMessage.cpp
deleted file mode 100644
index 4e8c6f1ba2d2c..0000000000000
--- a/lldb/source/Plugins/Process/POSIX/ProcessMessage.cpp
+++ /dev/null
@@ -1,61 +0,0 @@
-//===-- ProcessMessage.cpp ------------------------------------------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#include "ProcessMessage.h"
-
-using namespace lldb_private;
-
-const char *ProcessMessage::PrintCrashReason() const {
-  return CrashReasonAsString(m_crash_reason);
-}
-
-const char *ProcessMessage::PrintKind(Kind kind) {
-  const char *str = nullptr;
-
-  switch (kind) {
-  case eInvalidMessage:
-    str = "eInvalidMessage";
-    break;
-  case eAttachMessage:
-    str = "eAttachMessage";
-    break;
-  case eExitMessage:
-    str = "eExitMessage";
-    break;
-  case eLimboMessage:
-    str = "eLimboMessage";
-    break;
-  case eSignalMessage:
-    str = "eSignalMessage";
-    break;
-  case eSignalDeliveredMessage:
-    str = "eSignalDeliveredMessage";
-    break;
-  case eTraceMessage:
-    str = "eTraceMessage";
-    break;
-  case eBreakpointMessage:
-    str = "eBreakpointMessage";
-    break;
-  case eWatchpointMessage:
-    str = "eWatchpointMessage";
-    break;
-  case eCrashMessage:
-    str = "eCrashMessage";
-    break;
-  case eNewThreadMessage:
-    str = "eNewThreadMessage";
-    break;
-  case eExecMessage:
-    str = "eExecMessage";
-    break;
-  }
-  return str;
-}
-
-const char *ProcessMessage::PrintKind() const { return PrintKind(m_kind); }
diff --git a/lldb/source/Plugins/Process/POSIX/ProcessMessage.h b/lldb/source/Plugins/Process/POSIX/ProcessMessage.h
deleted file mode 100644
index d9c10caaa95e9..0000000000000
--- a/lldb/source/Plugins/Process/POSIX/ProcessMessage.h
+++ /dev/null
@@ -1,168 +0,0 @@
-//===-- ProcessMessage.h ----------------------------------------*- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef liblldb_ProcessMessage_H_
-#define liblldb_ProcessMessage_H_
-
-#include "CrashReason.h"
-
-#include <cassert>
-#include <string>
-
-#include "lldb/lldb-defines.h"
-#include "lldb/lldb-types.h"
-
-class ProcessMessage {
-public:
-  /// The type of signal this message can correspond to.
-  enum Kind {
-    eInvalidMessage,
-    eAttachMessage,
-    eExitMessage,
-    eLimboMessage,
-    eSignalMessage,
-    eSignalDeliveredMessage,
-    eTraceMessage,
-    eBreakpointMessage,
-    eWatchpointMessage,
-    eCrashMessage,
-    eNewThreadMessage,
-    eExecMessage
-  };
-
-  ProcessMessage()
-      : m_tid(LLDB_INVALID_PROCESS_ID), m_kind(eInvalidMessage),
-        m_crash_reason(CrashReason::eInvalidCrashReason), m_status(0),
-        m_addr(0) {}
-
-  Kind GetKind() const { return m_kind; }
-
-  lldb::tid_t GetTID() const { return m_tid; }
-
-  /// Indicates that the process \p pid has successfully attached.
-  static ProcessMessage Attach(lldb::pid_t pid) {
-    return ProcessMessage(pid, eAttachMessage);
-  }
-
-  /// Indicates that the thread \p tid is about to exit with status \p status.
-  static ProcessMessage Limbo(lldb::tid_t tid, int status) {
-    return ProcessMessage(tid, eLimboMessage, status);
-  }
-
-  /// Indicates that the thread \p tid had the signal \p signum delivered.
-  static ProcessMessage Signal(lldb::tid_t tid, int signum) {
-    return ProcessMessage(tid, eSignalMessage, signum);
-  }
-
-  /// Indicates that a signal \p signum generated by the debugging process was
-  /// delivered to the thread \p tid.
-  static ProcessMessage SignalDelivered(lldb::tid_t tid, int signum) {
-    return ProcessMessage(tid, eSignalDeliveredMessage, signum);
-  }
-
-  /// Indicates that the thread \p tid encountered a trace point.
-  static ProcessMessage Trace(lldb::tid_t tid) {
-    return ProcessMessage(tid, eTraceMessage);
-  }
-
-  /// Indicates that the thread \p tid encountered a break point.
-  static ProcessMessage Break(lldb::tid_t tid) {
-    return ProcessMessage(tid, eBreakpointMessage);
-  }
-
-  static ProcessMessage Watch(lldb::tid_t tid, lldb::addr_t wp_addr) {
-    return ProcessMessage(tid, eWatchpointMessage, 0, wp_addr);
-  }
-
-  /// Indicates that the thread \p tid crashed.
-  static ProcessMessage Crash(lldb::pid_t pid, CrashReason reason, int signo,
-                              lldb::addr_t fault_addr) {
-    ProcessMessage message(pid, eCrashMessage, signo, fault_addr);
-    message.m_crash_reason = reason;
-    return message;
-  }
-
-  /// Indicates that the thread \p child_tid was spawned.
-  static ProcessMessage NewThread(lldb::tid_t parent_tid,
-                                  lldb::tid_t child_tid) {
-    return ProcessMessage(parent_tid, eNewThreadMessage, child_tid);
-  }
-
-  /// Indicates that the thread \p tid is about to exit with status \p status.
-  static ProcessMessage Exit(lldb::tid_t tid, int status) {
-    return ProcessMessage(tid, eExitMessage, status);
-  }
-
-  /// Indicates that the thread \p pid has exec'd.
-  static ProcessMessage Exec(lldb::tid_t tid) {
-    return ProcessMessage(tid, eExecMessage);
-  }
-
-  int GetExitStatus() const {
-    assert(GetKind() == eExitMessage || GetKind() == eLimboMessage);
-    return m_status;
-  }
-
-  int GetSignal() const {
-    assert(GetKind() == eSignalMessage || GetKind() == eCrashMessage ||
-           GetKind() == eSignalDeliveredMessage);
-    return m_status;
-  }
-
-  int GetStopStatus() const {
-    assert(GetKind() == eSignalMessage);
-    return m_status;
-  }
-
-  CrashReason GetCrashReason() const {
-    assert(GetKind() == eCrashMessage);
-    return m_crash_reason;
-  }
-
-  lldb::addr_t GetFaultAddress() const {
-    assert(GetKind() == eCrashMessage);
-    return m_addr;
-  }
-
-  lldb::addr_t GetHWAddress() const {
-    assert(GetKind() == eWatchpointMessage || GetKind() == eTraceMessage);
-    return m_addr;
-  }
-
-  lldb::tid_t GetChildTID() const {
-    assert(GetKind() == eNewThreadMessage);
-    return m_child_tid;
-  }
-
-  const char *PrintCrashReason() const;
-
-  const char *PrintKind() const;
-
-  static const char *PrintKind(Kind);
-
-private:
-  ProcessMessage(lldb::tid_t tid, Kind kind, int status = 0,
-                 lldb::addr_t addr = 0)
-      : m_tid(tid), m_kind(kind),
-        m_crash_reason(CrashReason::eInvalidCrashReason), m_status(status),
-        m_addr(addr), m_child_tid(0) {}
-
-  ProcessMessage(lldb::tid_t tid, Kind kind, lldb::tid_t child_tid)
-      : m_tid(tid), m_kind(kind),
-        m_crash_reason(CrashReason::eInvalidCrashReason), m_status(0),
-        m_addr(0), m_child_tid(child_tid) {}
-
-  lldb::tid_t m_tid;
-  Kind m_kind : 8;
-  CrashReason m_crash_reason;
-  int m_status;
-  lldb::addr_t m_addr;
-  lldb::tid_t m_child_tid;
-};
-
-#endif // #ifndef liblldb_ProcessMessage_H_
diff --git a/lldb/source/Plugins/Process/Utility/CMakeLists.txt b/lldb/source/Plugins/Process/Utility/CMakeLists.txt
index 912d763ab6632..edf4e851b653c 100644
--- a/lldb/source/Plugins/Process/Utility/CMakeLists.txt
+++ b/lldb/source/Plugins/Process/Utility/CMakeLists.txt
@@ -8,7 +8,6 @@ add_lldb_library(lldbPluginProcessUtility
   LinuxProcMaps.cpp
   LinuxSignals.cpp
   MemoryTagManagerAArch64MTE.cpp
-  MipsLinuxSignals.cpp
   NativeProcessSoftwareSingleStep.cpp
   NativeRegisterContextDBReg_arm64.cpp
   NativeRegisterContextDBReg_x86.cpp
diff --git a/lldb/source/Plugins/Process/Utility/MipsLinuxSignals.cpp b/lldb/source/Plugins/Process/Utility/MipsLinuxSignals.cpp
deleted file mode 100644
index fb51c56953f85..0000000000000
--- a/lldb/source/Plugins/Process/Utility/MipsLinuxSignals.cpp
+++ /dev/null
@@ -1,85 +0,0 @@
-//===-- MipsLinuxSignals.cpp ----------------------------------------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#include "MipsLinuxSignals.h"
-
-using namespace lldb_private;
-
-MipsLinuxSignals::MipsLinuxSignals() : UnixSignals() { Reset(); }
-
-void MipsLinuxSignals::Reset() {
-  m_signals.clear();
-  // clang-format off
-  //        SIGNO   NAME            SUPPRESS  STOP    NOTIFY  DESCRIPTION
-  //        ======  ==============  ========  ======  ======  ===================================================
-  AddSignal(1, "SIGHUP",            false,    true,   true,   "hangup");
-  AddSignal(2, "SIGINT",            true,     true,   true,   "interrupt");
-  AddSignal(3, "SIGQUIT",           false,    true,   true,   "quit");
-  AddSignal(4, "SIGILL",            false,    true,   true,   "illegal instruction");
-  AddSignal(5, "SIGTRAP",           true,     true,   true,   "trace trap (not reset when caught)");
-  AddSignal(6, "SIGABRT",           false,    true,   true,   "abort()/IOT trap", "SIGIOT");
-  AddSignal(7, "SIGEMT",            false,    true,   true,   "terminate process with core dump");
-  AddSignal(8, "SIGFPE",            false,    true,   true,   "floating point exception");
-  AddSignal(9, "SIGKILL",           false,    true,   true,   "kill");
-  AddSignal(10, "SIGBUS",           false,    true,   true,   "bus error");
-  AddSignal(11, "SIGSEGV",          false,    true,   true,   "segmentation violation");
-  AddSignal(12, "SIGSYS",           false,    true,   true,   "invalid system call");
-  AddSignal(13, "SIGPIPE",          false,    true,   true,   "write to pipe with reading end closed");
-  AddSignal(14, "SIGALRM",          false,    false,  false,  "alarm");
-  AddSignal(15, "SIGTERM",          false,    true,   true,   "termination requested");
-  AddSignal(16, "SIGUSR1",          false,    true,   true,   "user defined signal 1");
-  AddSignal(17, "SIGUSR2",          false,    true,   true,   "user defined signal 2");
-  AddSignal(18, "SIGCHLD",          false,    false,  true,   "child status has changed", "SIGCLD");
-  AddSignal(19, "SIGPWR",           false,    true,   true,   "power failure");
-  AddSignal(20, "SIGWINCH",         false,    true,   true,   "window size changes");
-  AddSignal(21, "SIGURG",           false,    true,   true,   "urgent data on socket");
-  AddSignal(22, "SIGIO",            false,    true,   true,   "input/output ready/Pollable event", "SIGPOLL");
-  AddSignal(23, "SIGSTOP",          true,     true,   true,   "process stop");
-  AddSignal(24, "SIGTSTP",          false,    true,   true,   "tty stop");
-  AddSignal(25, "SIGCONT",          false,    false,  true,   "process continue");
-  AddSignal(26, "SIGTTIN",          false,    true,   true,   "background tty read");
-  AddSignal(27, "SIGTTOU",          false,    true,   true,   "background tty write");
-  AddSignal(28, "SIGVTALRM",        false,    true,   true,   "virtual time alarm");
-  AddSignal(29, "SIGPROF",          false,    false,  false,  "profiling time alarm");
-  AddSignal(30, "SIGXCPU",          false,    true,   true,   "CPU resource exceeded");
-  AddSignal(31, "SIGXFSZ",          false,    true,   true,   "file size limit exceeded");
-  AddSignal(32, "SIG32",            false,    false,  false,  "threading library internal signal 1");
-  AddSignal(33, "SIG33",            false,    false,  false,  "threading library internal signal 2");
-  AddSignal(34, "SIGRTMIN",         false,    false,  false,  "real time signal 0");
-  AddSignal(35, "SIGRTMIN+1",       false,    false,  false,  "real time signal 1");
-  AddSignal(36, "SIGRTMIN+2",       false,    false,  false,  "real time signal 2");
-  AddSignal(37, "SIGRTMIN+3",       false,    false,  false,  "real time signal 3");
-  AddSignal(38, "SIGRTMIN+4",       false,    false,  false,  "real time signal 4");
-  AddSignal(39, "SIGRTMIN+5",       false,    false,  false,  "real time signal 5");
-  AddSignal(40, "SIGRTMIN+6",       false,    false,  false,  "real time signal 6");
-  AddSignal(41, "SIGRTMIN+7",       false,    false,  false,  "real time signal 7");
-  AddSignal(42, "SIGRTMIN+8",       false,    false,  false,  "real time signal 8");
-  AddSignal(43, "SIGRTMIN+9",       false,    false,  false,  "real time signal 9");
-  AddSignal(44, "SIGRTMIN+10",      false,    false,  false,  "real time signal 10");
-  AddSignal(45, "SIGRTMIN+11",      false,    false,  false,  "real time signal 11");
-  AddSignal(46, "SIGRTMIN+12",      false,    false,  false,  "real time signal 12");
-  AddSignal(47, "SIGRTMIN+13",      false,    false,  false,  "real time signal 13");
-  AddSignal(48, "SIGRTMIN+14",      false,    false,  false,  "real time signal 14");
-  AddSignal(49, "SIGRTMIN+15",      false,    false,  false,  "real time signal 15");
-  AddSignal(50, "SIGRTMAX-14",      false,    false,  false,  "real time signal 16"); // switching to SIGRTMAX-xxx to match "kill -l" output
-  AddSignal(51, "SIGRTMAX-13",      false,    false,  false,  "real time signal 17");
-  AddSignal(52, "SIGRTMAX-12",      false,    false,  false,  "real time signal 18");
-  AddSignal(53, "SIGRTMAX-11",      false,    false,  false,  "real time signal 19");
-  AddSignal(54, "SIGRTMAX-10",      false,    false,  false,  "real time signal 20");
-  AddSignal(55, "SIGRTMAX-9",       false,    false,  false,  "real time signal 21");
-  AddSignal(56, "SIGRTMAX-8",       false,    false,  false,  "real time signal 22");
-  AddSignal(57, "SIGRTMAX-7",       false,    false,  false,  "real time signal 23");
-  AddSignal(58, "SIGRTMAX-6",       false,    false,  false,  "real time signal 24");
-  AddSignal(59, "SIGRTMAX-5",       false,    false,  false,  "real time signal 25");
-  AddSignal(60, "SIGRTMAX-4",       false,    false,  false,  "real time signal 26");
-  AddSignal(61, "SIGRTMAX-3",       false,    false,  false,  "real time signal 27");
-  AddSignal(62, "SIGRTMAX-2",       false,    false,  false,  "real time signal 28");
-  AddSignal(63, "SIGRTMAX-1",       false,    false,  false,  "real time signal 29");
-  AddSignal(64, "SIGRTMAX",         false,    false,  false,  "real time signal 30");
-  // clang-format on
-}
diff --git a/lldb/source/Plugins/Process/Utility/MipsLinuxSignals.h b/lldb/source/Plugins/Process/Utility/MipsLinuxSignals.h
deleted file mode 100644
index 6b78fc72a91c6..0000000000000
--- a/lldb/source/Plugins/Process/Utility/MipsLinuxSignals.h
+++ /dev/null
@@ -1,28 +0,0 @@
-//===-- MipsLinuxSignals.h ------------------------------------------*- C++
-//-*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LLDB_SOURCE_PLUGINS_PROCESS_UTILITY_MIPSLINUXSIGNALS_H
-#define LLDB_SOURCE_PLUGINS_PROCESS_UTILITY_MIPSLINUXSIGNALS_H
-
-#include "lldb/Target/UnixSignals.h"
-
-namespace lldb_private {
-
-/// Linux specific set of Unix signals.
-class MipsLinuxSignals : public UnixSignals {
-public:
-  MipsLinuxSignals();
-
-private:
-  void Reset() override;
-};
-
-} // namespace lldb_private
-
-#endif // LLDB_SOURCE_PLUGINS_PROCESS_UTILITY_MIPSLINUXSIGNALS_H
diff --git a/lldb/source/Plugins/Process/minidump/CMakeLists.txt b/lldb/source/Plugins/Process/minidump/CMakeLists.txt
index afa3550dfe48d..9af816884c217 100644
--- a/lldb/source/Plugins/Process/minidump/CMakeLists.txt
+++ b/lldb/source/Plugins/Process/minidump/CMakeLists.txt
@@ -14,6 +14,7 @@ add_lldb_library(lldbPluginProcessMinidump PLUGIN
     lldbUtility
     lldbPluginProcessUtility
     lldbPluginProcessElfCore
+    lldbPluginObjectFilePlaceholder
   LINK_COMPONENTS
     BinaryFormat
     Object
diff --git a/lldb/source/Plugins/Process/minidump/ProcessMinidump.cpp b/lldb/source/Plugins/Process/minidump/ProcessMinidump.cpp
index 5911b137ef419..cf6166ecd77b9 100644
--- a/lldb/source/Plugins/Process/minidump/ProcessMinidump.cpp
+++ b/lldb/source/Plugins/Process/minidump/ProcessMinidump.cpp
@@ -34,6 +34,7 @@
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Threading.h"
 
+#include "Plugins/ObjectFile/Placeholder/ObjectFilePlaceholder.h"
 #include "Plugins/Process/Utility/StopInfoMachException.h"
 
 #include <memory>
@@ -47,84 +48,6 @@ LLDB_PLUGIN_DEFINE(ProcessMinidump)
 
 namespace {
 
-/// A minimal ObjectFile implementation providing a dummy object file for the
-/// cases when the real module binary is not available. This allows the module
-/// to show up in "image list" and symbols to be added to it.
-class PlaceholderObjectFile : public ObjectFile {
-public:
-  PlaceholderObjectFile(const lldb::ModuleSP &module_sp,
-                        const ModuleSpec &module_spec, lldb::addr_t base,
-                        lldb::addr_t size)
-      : ObjectFile(module_sp, &module_spec.GetFileSpec(), /*file_offset*/ 0,
-                   /*length*/ 0, /*data_sp*/ nullptr, /*data_offset*/ 0),
-        m_arch(module_spec.GetArchitecture()), m_uuid(module_spec.GetUUID()),
-        m_base(base), m_size(size) {
-    m_symtab_up = std::make_unique<Symtab>(this);
-  }
-
-  static ConstString GetStaticPluginName() {
-    return ConstString("placeholder");
-  }
-  llvm::StringRef GetPluginName() override {
-    return GetStaticPluginName().GetStringRef();
-  }
-  bool ParseHeader() override { return true; }
-  Type CalculateType() override { return eTypeUnknown; }
-  Strata CalculateStrata() override { return eStrataUnknown; }
-  uint32_t GetDependentModules(FileSpecList &file_list) override { return 0; }
-  bool IsExecutable() const override { return false; }
-  ArchSpec GetArchitecture() override { return m_arch; }
-  UUID GetUUID() override { return m_uuid; }
-  void ParseSymtab(lldb_private::Symtab &symtab) override {}
-  bool IsStripped() override { return true; }
-  ByteOrder GetByteOrder() const override { return m_arch.GetByteOrder(); }
-
-  uint32_t GetAddressByteSize() const override {
-    return m_arch.GetAddressByteSize();
-  }
-
-  Address GetBaseAddress() override {
-    return Address(m_sections_up->GetSectionAtIndex(0), 0);
-  }
-
-  void CreateSections(SectionList &unified_section_list) override {
-    m_sections_up = std::make_unique<SectionList>();
-    auto section_sp = std::make_shared<Section>(
-        GetModule(), this, /*sect_id*/ 0, ConstString(".module_image"),
-        eSectionTypeOther, m_base, m_size, /*file_offset*/ 0, /*file_size*/ 0,
-        /*log2align*/ 0, /*flags*/ 0);
-    section_sp->SetPermissions(ePermissionsReadable | ePermissionsExecutable);
-    m_sections_up->AddSection(section_sp);
-    unified_section_list.AddSection(std::move(section_sp));
-  }
-
-  bool SetLoadAddress(Target &target, addr_t value,
-                      bool value_is_offset) override {
-    assert(!value_is_offset);
-    assert(value == m_base);
-
-    // Create sections if they haven't been created already.
-    GetModule()->GetSectionList();
-    assert(m_sections_up->GetNumSections(0) == 1);
-
-    target.GetSectionLoadList().SetSectionLoadAddress(
-        m_sections_up->GetSectionAtIndex(0), m_base);
-    return true;
-  }
-
-  void Dump(Stream *s) override {
-    s->Format("Placeholder object file for {0} loaded at [{1:x}-{2:x})\n",
-              GetFileSpec(), m_base, m_base + m_size);
-  }
-
-  lldb::addr_t GetBaseImageAddress() const { return m_base; }
-private:
-  ArchSpec m_arch;
-  UUID m_uuid;
-  lldb::addr_t m_base;
-  lldb::addr_t m_size;
-};
-
 /// Duplicate the HashElfTextSection() from the breakpad sources.
 ///
 /// Breakpad, a Google crash log reporting tool suite, creates minidump files
@@ -578,12 +501,12 @@ void ProcessMinidump::ReadModuleList() {
       // Watch out for place holder modules that have different paths, but the
       // same UUID. If the base address is different, create a new module. If
       // we don't then we will end up setting the load address of a different
-      // PlaceholderObjectFile and an assertion will fire.
+      // ObjectFilePlaceholder and an assertion will fire.
       auto *objfile = module_sp->GetObjectFile();
       if (objfile &&
           objfile->GetPluginName() ==
-              PlaceholderObjectFile::GetStaticPluginName().GetStringRef()) {
-        if (((PlaceholderObjectFile *)objfile)->GetBaseImageAddress() !=
+              ObjectFilePlaceholder::GetPluginNameStatic()) {
+        if (((ObjectFilePlaceholder *)objfile)->GetBaseImageAddress() !=
             load_addr)
           module_sp.reset();
       }
@@ -601,7 +524,7 @@ void ProcessMinidump::ReadModuleList() {
                "placeholder module for: {0}",
                name);
 
-      module_sp = Module::CreateModuleFromObjectFile<PlaceholderObjectFile>(
+      module_sp = Module::CreateModuleFromObjectFile<ObjectFilePlaceholder>(
           module_spec, load_addr, load_size);
       GetTarget().GetImages().Append(module_sp, true /* notify */);
     }
diff --git a/lldb/source/Plugins/Process/scripted/ScriptedProcess.cpp b/lldb/source/Plugins/Process/scripted/ScriptedProcess.cpp
index 948ee691ecbe4..9a670276a8b82 100644
--- a/lldb/source/Plugins/Process/scripted/ScriptedProcess.cpp
+++ b/lldb/source/Plugins/Process/scripted/ScriptedProcess.cpp
@@ -18,11 +18,11 @@
 #include "lldb/Interpreter/OptionArgParser.h"
 #include "lldb/Interpreter/OptionGroupBoolean.h"
 #include "lldb/Interpreter/ScriptInterpreter.h"
-#include "lldb/Interpreter/ScriptedMetadata.h"
 #include "lldb/Target/MemoryRegionInfo.h"
 #include "lldb/Target/Queue.h"
 #include "lldb/Target/RegisterContext.h"
 #include "lldb/Utility/LLDBLog.h"
+#include "lldb/Utility/ScriptedMetadata.h"
 #include "lldb/Utility/State.h"
 
 #include <mutex>
diff --git a/lldb/source/Plugins/Process/scripted/ScriptedProcess.h b/lldb/source/Plugins/Process/scripted/ScriptedProcess.h
index 3601173e99cf3..368c54b1b4619 100644
--- a/lldb/source/Plugins/Process/scripted/ScriptedProcess.h
+++ b/lldb/source/Plugins/Process/scripted/ScriptedProcess.h
@@ -9,9 +9,9 @@
 #ifndef LLDB_SOURCE_PLUGINS_SCRIPTED_PROCESS_H
 #define LLDB_SOURCE_PLUGINS_SCRIPTED_PROCESS_H
 
-#include "lldb/Interpreter/ScriptedMetadata.h"
 #include "lldb/Target/Process.h"
 #include "lldb/Utility/ConstString.h"
+#include "lldb/Utility/ScriptedMetadata.h"
 #include "lldb/Utility/Status.h"
 
 #include "ScriptedThread.h"
diff --git a/lldb/source/Plugins/ScriptInterpreter/Python/ScriptedProcessPythonInterface.cpp b/lldb/source/Plugins/ScriptInterpreter/Python/ScriptedProcessPythonInterface.cpp
index cffa3bda3ab03..c985065835727 100644
--- a/lldb/source/Plugins/ScriptInterpreter/Python/ScriptedProcessPythonInterface.cpp
+++ b/lldb/source/Plugins/ScriptInterpreter/Python/ScriptedProcessPythonInterface.cpp
@@ -7,6 +7,10 @@
 //===----------------------------------------------------------------------===//
 
 #include "lldb/Host/Config.h"
+#if LLDB_ENABLE_PYTHON
+// LLDB Python header must be included first
+#include "lldb-python.h"
+#endif
 #include "lldb/Target/Process.h"
 #include "lldb/Utility/Log.h"
 #include "lldb/Utility/Status.h"
@@ -14,9 +18,6 @@
 
 #if LLDB_ENABLE_PYTHON
 
-// LLDB Python header must be included first
-#include "lldb-python.h"
-
 #include "SWIGPythonBridge.h"
 #include "ScriptInterpreterPythonImpl.h"
 #include "ScriptedProcessPythonInterface.h"
diff --git a/lldb/source/Plugins/SymbolFile/CMakeLists.txt b/lldb/source/Plugins/SymbolFile/CMakeLists.txt
index 5e18fb154823a..76b09172bd206 100644
--- a/lldb/source/Plugins/SymbolFile/CMakeLists.txt
+++ b/lldb/source/Plugins/SymbolFile/CMakeLists.txt
@@ -1,5 +1,6 @@
 add_subdirectory(Breakpad)
 add_subdirectory(DWARF)
+add_subdirectory(JSON)
 add_subdirectory(NativePDB)
 add_subdirectory(PDB)
 add_subdirectory(Symtab)
diff --git a/lldb/source/Plugins/SymbolFile/DWARF/DWARFASTParserClang.cpp b/lldb/source/Plugins/SymbolFile/DWARF/DWARFASTParserClang.cpp
index ca0982c1f9008..cf794854c8431 100644
--- a/lldb/source/Plugins/SymbolFile/DWARF/DWARFASTParserClang.cpp
+++ b/lldb/source/Plugins/SymbolFile/DWARF/DWARFASTParserClang.cpp
@@ -2394,12 +2394,12 @@ DWARFASTParserClang::ParseFunctionFromDWARF(CompileUnit &comp_unit,
   DWARFRangeList func_ranges;
   const char *name = nullptr;
   const char *mangled = nullptr;
-  int decl_file = 0;
-  int decl_line = 0;
-  int decl_column = 0;
-  int call_file = 0;
-  int call_line = 0;
-  int call_column = 0;
+  std::optional<int> decl_file;
+  std::optional<int> decl_line;
+  std::optional<int> decl_column;
+  std::optional<int> call_file;
+  std::optional<int> call_line;
+  std::optional<int> call_column;
   DWARFExpressionList frame_base;
 
   const dw_tag_t tag = die.Tag();
@@ -2429,9 +2429,10 @@ DWARFASTParserClang::ParseFunctionFromDWARF(CompileUnit &comp_unit,
 
     FunctionSP func_sp;
     std::unique_ptr<Declaration> decl_up;
-    if (decl_file != 0 || decl_line != 0 || decl_column != 0)
-      decl_up = std::make_unique<Declaration>(die.GetCU()->GetFile(decl_file),
-                                              decl_line, decl_column);
+    if (decl_file || decl_line || decl_column)
+      decl_up = std::make_unique<Declaration>(
+          die.GetCU()->GetFile(decl_file ? *decl_file : 0),
+          decl_line ? *decl_line : 0, decl_column ? *decl_column : 0);
 
     SymbolFileDWARF *dwarf = die.GetDWARF();
     // Supply the type _only_ if it has already been parsed
diff --git a/lldb/source/Plugins/SymbolFile/DWARF/DWARFDIE.cpp b/lldb/source/Plugins/SymbolFile/DWARF/DWARFDIE.cpp
index 7b9da6fa166f9..65fab503deb28 100644
--- a/lldb/source/Plugins/SymbolFile/DWARF/DWARFDIE.cpp
+++ b/lldb/source/Plugins/SymbolFile/DWARF/DWARFDIE.cpp
@@ -439,8 +439,9 @@ bool DWARFDIE::IsMethod() const {
 
 bool DWARFDIE::GetDIENamesAndRanges(
     const char *&name, const char *&mangled, DWARFRangeList &ranges,
-    int &decl_file, int &decl_line, int &decl_column, int &call_file,
-    int &call_line, int &call_column,
+    std::optional<int> &decl_file, std::optional<int> &decl_line,
+    std::optional<int> &decl_column, std::optional<int> &call_file,
+    std::optional<int> &call_line, std::optional<int> &call_column,
     lldb_private::DWARFExpressionList *frame_base) const {
   if (IsValid()) {
     return m_die->GetDIENamesAndRanges(
diff --git a/lldb/source/Plugins/SymbolFile/DWARF/DWARFDIE.h b/lldb/source/Plugins/SymbolFile/DWARF/DWARFDIE.h
index 3564f757dbd8c..031ea26ad4050 100644
--- a/lldb/source/Plugins/SymbolFile/DWARF/DWARFDIE.h
+++ b/lldb/source/Plugins/SymbolFile/DWARF/DWARFDIE.h
@@ -83,12 +83,12 @@ class DWARFDIE : public DWARFBaseDIE {
   DWARFDIE
   GetAttributeValueAsReferenceDIE(const dw_attr_t attr) const;
 
-  bool
-  GetDIENamesAndRanges(const char *&name, const char *&mangled,
-                       DWARFRangeList &ranges, int &decl_file, int &decl_line,
-                       int &decl_column, int &call_file, int &call_line,
-                       int &call_column,
-                       lldb_private::DWARFExpressionList *frame_base) const;
+  bool GetDIENamesAndRanges(
+      const char *&name, const char *&mangled, DWARFRangeList &ranges,
+      std::optional<int> &decl_file, std::optional<int> &decl_line,
+      std::optional<int> &decl_column, std::optional<int> &call_file,
+      std::optional<int> &call_line, std::optional<int> &call_column,
+      lldb_private::DWARFExpressionList *frame_base) const;
 
   /// The range of all the children of this DIE.
   llvm::iterator_range<child_iterator> children() const;
diff --git a/lldb/source/Plugins/SymbolFile/DWARF/DWARFDebugInfoEntry.cpp b/lldb/source/Plugins/SymbolFile/DWARF/DWARFDebugInfoEntry.cpp
index d1450d90dce17..bd2bb3d839c6a 100644
--- a/lldb/source/Plugins/SymbolFile/DWARF/DWARFDebugInfoEntry.cpp
+++ b/lldb/source/Plugins/SymbolFile/DWARF/DWARFDebugInfoEntry.cpp
@@ -234,9 +234,10 @@ static DWARFRangeList GetRangesOrReportError(DWARFUnit &unit,
 // DW_AT_low_pc/DW_AT_high_pc pair, DW_AT_entry_pc, or DW_AT_ranges attributes.
 bool DWARFDebugInfoEntry::GetDIENamesAndRanges(
     DWARFUnit *cu, const char *&name, const char *&mangled,
-    DWARFRangeList &ranges, int &decl_file, int &decl_line, int &decl_column,
-    int &call_file, int &call_line, int &call_column,
-    DWARFExpressionList *frame_base) const {
+    DWARFRangeList &ranges, std::optional<int> &decl_file,
+    std::optional<int> &decl_line, std::optional<int> &decl_column,
+    std::optional<int> &call_file, std::optional<int> &call_line,
+    std::optional<int> &call_column, DWARFExpressionList *frame_base) const {
   dw_addr_t lo_pc = LLDB_INVALID_ADDRESS;
   dw_addr_t hi_pc = LLDB_INVALID_ADDRESS;
   std::vector<DWARFDIE> dies;
@@ -315,32 +316,32 @@ bool DWARFDebugInfoEntry::GetDIENamesAndRanges(
           break;
 
         case DW_AT_decl_file:
-          if (decl_file == 0)
+          if (!decl_file)
             decl_file = form_value.Unsigned();
           break;
 
         case DW_AT_decl_line:
-          if (decl_line == 0)
+          if (!decl_line)
             decl_line = form_value.Unsigned();
           break;
 
         case DW_AT_decl_column:
-          if (decl_column == 0)
+          if (!decl_column)
             decl_column = form_value.Unsigned();
           break;
 
         case DW_AT_call_file:
-          if (call_file == 0)
+          if (!call_file)
             call_file = form_value.Unsigned();
           break;
 
         case DW_AT_call_line:
-          if (call_line == 0)
+          if (!call_line)
             call_line = form_value.Unsigned();
           break;
 
         case DW_AT_call_column:
-          if (call_column == 0)
+          if (!call_column)
             call_column = form_value.Unsigned();
           break;
 
diff --git a/lldb/source/Plugins/SymbolFile/DWARF/DWARFDebugInfoEntry.h b/lldb/source/Plugins/SymbolFile/DWARF/DWARFDebugInfoEntry.h
index 67bd0c2f3cca5..e1378952efbc3 100644
--- a/lldb/source/Plugins/SymbolFile/DWARF/DWARFDebugInfoEntry.h
+++ b/lldb/source/Plugins/SymbolFile/DWARF/DWARFDebugInfoEntry.h
@@ -103,8 +103,10 @@ class DWARFDebugInfoEntry {
 
   bool GetDIENamesAndRanges(
       DWARFUnit *cu, const char *&name, const char *&mangled,
-      DWARFRangeList &rangeList, int &decl_file, int &decl_line,
-      int &decl_column, int &call_file, int &call_line, int &call_column,
+      DWARFRangeList &rangeList, std::optional<int> &decl_file,
+      std::optional<int> &decl_line, std::optional<int> &decl_column,
+      std::optional<int> &call_file, std::optional<int> &call_line,
+      std::optional<int> &call_column,
       lldb_private::DWARFExpressionList *frame_base = nullptr) const;
 
   const DWARFAbbreviationDeclaration *
diff --git a/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp b/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp
index 689046443bdde..99a0152eaf6e6 100644
--- a/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp
+++ b/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp
@@ -1284,12 +1284,12 @@ size_t SymbolFileDWARF::ParseBlocksRecursive(
       const char *name = nullptr;
       const char *mangled_name = nullptr;
 
-      int decl_file = 0;
-      int decl_line = 0;
-      int decl_column = 0;
-      int call_file = 0;
-      int call_line = 0;
-      int call_column = 0;
+      std::optional<int> decl_file;
+      std::optional<int> decl_line;
+      std::optional<int> decl_column;
+      std::optional<int> call_file;
+      std::optional<int> call_line;
+      std::optional<int> call_column;
       if (die.GetDIENamesAndRanges(name, mangled_name, ranges, decl_file,
                                    decl_line, decl_column, call_file, call_line,
                                    call_column, nullptr)) {
@@ -1332,16 +1332,18 @@ size_t SymbolFileDWARF::ParseBlocksRecursive(
         if (tag != DW_TAG_subprogram &&
             (name != nullptr || mangled_name != nullptr)) {
           std::unique_ptr<Declaration> decl_up;
-          if (decl_file != 0 || decl_line != 0 || decl_column != 0)
+          if (decl_file || decl_line || decl_column)
             decl_up = std::make_unique<Declaration>(
-                comp_unit.GetSupportFiles().GetFileSpecAtIndex(decl_file),
-                decl_line, decl_column);
+                comp_unit.GetSupportFiles().GetFileSpecAtIndex(
+                    decl_file ? *decl_file : 0),
+                decl_line ? *decl_line : 0, decl_column ? *decl_column : 0);
 
           std::unique_ptr<Declaration> call_up;
-          if (call_file != 0 || call_line != 0 || call_column != 0)
+          if (call_file || call_line || call_column)
             call_up = std::make_unique<Declaration>(
-                comp_unit.GetSupportFiles().GetFileSpecAtIndex(call_file),
-                call_line, call_column);
+                comp_unit.GetSupportFiles().GetFileSpecAtIndex(
+                    call_file ? *call_file : 0),
+                call_line ? *call_line : 0, call_column ? *call_column : 0);
 
           block->SetInlinedFunctionInfo(name, mangled_name, decl_up.get(),
                                         call_up.get());
diff --git a/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDebugMap.cpp b/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDebugMap.cpp
index c8140f0f9148e..9b22b1b94aae4 100644
--- a/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDebugMap.cpp
+++ b/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDebugMap.cpp
@@ -683,6 +683,17 @@ XcodeSDK SymbolFileDWARFDebugMap::ParseXcodeSDK(CompileUnit &comp_unit) {
   return {};
 }
 
+llvm::SmallSet<lldb::LanguageType, 4>
+SymbolFileDWARFDebugMap::ParseAllLanguages(
+    lldb_private::CompileUnit &comp_unit) {
+  llvm::SmallSet<lldb::LanguageType, 4> langs;
+  auto *info = GetCompUnitInfo(comp_unit);
+  for (auto &comp_unit : info->compile_units_sps) {
+    langs.insert(comp_unit->GetLanguage());
+  }
+  return langs;
+}
+
 size_t SymbolFileDWARFDebugMap::ParseFunctions(CompileUnit &comp_unit) {
   std::lock_guard<std::recursive_mutex> guard(GetModuleMutex());
   SymbolFileDWARF *oso_dwarf = GetSymbolFile(comp_unit);
diff --git a/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDebugMap.h b/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDebugMap.h
index 485fb5b8f908d..84f78d87d64ca 100644
--- a/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDebugMap.h
+++ b/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDebugMap.h
@@ -62,6 +62,8 @@ class SymbolFileDWARFDebugMap : public lldb_private::SymbolFileCommon {
   ParseLanguage(lldb_private::CompileUnit &comp_unit) override;
   lldb_private::XcodeSDK
   ParseXcodeSDK(lldb_private::CompileUnit &comp_unit) override;
+  llvm::SmallSet<lldb::LanguageType, 4>
+  ParseAllLanguages(lldb_private::CompileUnit &comp_unit) override;
   size_t ParseFunctions(lldb_private::CompileUnit &comp_unit) override;
   bool ParseLineTable(lldb_private::CompileUnit &comp_unit) override;
   bool ParseDebugMacros(lldb_private::CompileUnit &comp_unit) override;
diff --git a/lldb/source/Plugins/SymbolFile/JSON/CMakeLists.txt b/lldb/source/Plugins/SymbolFile/JSON/CMakeLists.txt
new file mode 100644
index 0000000000000..7de8fef443976
--- /dev/null
+++ b/lldb/source/Plugins/SymbolFile/JSON/CMakeLists.txt
@@ -0,0 +1,7 @@
+add_lldb_library(lldbPluginSymbolFileJSON PLUGIN
+  SymbolFileJSON.cpp
+
+  LINK_LIBS
+    lldbCore
+    lldbSymbol
+  )
diff --git a/lldb/source/Plugins/SymbolFile/JSON/SymbolFileJSON.cpp b/lldb/source/Plugins/SymbolFile/JSON/SymbolFileJSON.cpp
new file mode 100644
index 0000000000000..f10ff59f3186d
--- /dev/null
+++ b/lldb/source/Plugins/SymbolFile/JSON/SymbolFileJSON.cpp
@@ -0,0 +1,105 @@
+//===-- SymbolFileJSON.cpp ----------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "SymbolFileJSON.h"
+
+#include "Plugins/ObjectFile/JSON/ObjectFileJSON.h"
+#include "lldb/Core/Module.h"
+#include "lldb/Core/PluginManager.h"
+#include "lldb/Symbol/CompileUnit.h"
+#include "lldb/Symbol/Function.h"
+#include "lldb/Symbol/ObjectFile.h"
+#include "lldb/Symbol/Symbol.h"
+#include "lldb/Symbol/SymbolContext.h"
+#include "lldb/Symbol/Symtab.h"
+#include "lldb/Symbol/TypeList.h"
+#include "lldb/Utility/LLDBLog.h"
+#include "lldb/Utility/Log.h"
+#include "lldb/Utility/RegularExpression.h"
+#include "lldb/Utility/Timer.h"
+#include "llvm/Support/MemoryBuffer.h"
+
+#include <memory>
+#include <optional>
+
+using namespace llvm;
+using namespace lldb;
+using namespace lldb_private;
+
+LLDB_PLUGIN_DEFINE(SymbolFileJSON)
+
+char SymbolFileJSON::ID;
+
+SymbolFileJSON::SymbolFileJSON(lldb::ObjectFileSP objfile_sp)
+    : SymbolFileCommon(std::move(objfile_sp)) {}
+
+void SymbolFileJSON::Initialize() {
+  PluginManager::RegisterPlugin(GetPluginNameStatic(),
+                                GetPluginDescriptionStatic(), CreateInstance);
+}
+
+void SymbolFileJSON::Terminate() {
+  PluginManager::UnregisterPlugin(CreateInstance);
+}
+
+llvm::StringRef SymbolFileJSON::GetPluginDescriptionStatic() {
+  return "Reads debug symbols from a textual symbol table.";
+}
+
+SymbolFile *SymbolFileJSON::CreateInstance(ObjectFileSP objfile_sp) {
+  return new SymbolFileJSON(std::move(objfile_sp));
+}
+
+uint32_t SymbolFileJSON::CalculateAbilities() {
+  if (!m_objfile_sp || !llvm::isa<ObjectFileJSON>(*m_objfile_sp))
+    return 0;
+
+  return GlobalVariables | Functions;
+}
+
+uint32_t SymbolFileJSON::ResolveSymbolContext(const Address &so_addr,
+                                              SymbolContextItem resolve_scope,
+                                              SymbolContext &sc) {
+  std::lock_guard<std::recursive_mutex> guard(GetModuleMutex());
+  if (m_objfile_sp->GetSymtab() == nullptr)
+    return 0;
+
+  uint32_t resolved_flags = 0;
+  if (resolve_scope & eSymbolContextSymbol) {
+    sc.symbol = m_objfile_sp->GetSymtab()->FindSymbolContainingFileAddress(
+        so_addr.GetFileAddress());
+    if (sc.symbol)
+      resolved_flags |= eSymbolContextSymbol;
+  }
+  return resolved_flags;
+}
+
+CompUnitSP SymbolFileJSON::ParseCompileUnitAtIndex(uint32_t idx) { return {}; }
+
+void SymbolFileJSON::GetTypes(SymbolContextScope *sc_scope, TypeClass type_mask,
+                              lldb_private::TypeList &type_list) {}
+
+void SymbolFileJSON::AddSymbols(Symtab &symtab) {
+  if (!m_objfile_sp)
+    return;
+
+  Symtab *json_symtab = m_objfile_sp->GetSymtab();
+  if (!json_symtab)
+    return;
+
+  if (&symtab == json_symtab)
+    return;
+
+  // Merge the two symbol tables.
+  const size_t num_new_symbols = json_symtab->GetNumSymbols();
+  for (size_t i = 0; i < num_new_symbols; ++i) {
+    Symbol *s = json_symtab->SymbolAtIndex(i);
+    symtab.AddSymbol(*s);
+  }
+  symtab.Finalize();
+}
diff --git a/lldb/source/Plugins/SymbolFile/JSON/SymbolFileJSON.h b/lldb/source/Plugins/SymbolFile/JSON/SymbolFileJSON.h
new file mode 100644
index 0000000000000..1fc41c739c027
--- /dev/null
+++ b/lldb/source/Plugins/SymbolFile/JSON/SymbolFileJSON.h
@@ -0,0 +1,110 @@
+//===-- SymbolFileJSON.h ----------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLDB_SOURCE_PLUGINS_SYMBOLFILE_JSON_SYMBOLFILETEXT_H
+#define LLDB_SOURCE_PLUGINS_SYMBOLFILE_JSON_SYMBOLFILETEXT_H
+
+#include <map>
+#include <optional>
+#include <vector>
+
+#include "lldb/Symbol/CompileUnit.h"
+#include "lldb/Symbol/SymbolFile.h"
+
+namespace lldb_private {
+
+class SymbolFileJSON : public lldb_private::SymbolFileCommon {
+  /// LLVM RTTI support.
+  static char ID;
+
+public:
+  /// LLVM RTTI support.
+  /// \{
+  bool isA(const void *ClassID) const override {
+    return ClassID == &ID || SymbolFileCommon::isA(ClassID);
+  }
+  static bool classof(const SymbolFile *obj) { return obj->isA(&ID); }
+  /// \}
+
+  SymbolFileJSON(lldb::ObjectFileSP objfile_sp);
+
+  static void Initialize();
+
+  static void Terminate();
+
+  static llvm::StringRef GetPluginNameStatic() { return "text"; }
+
+  static llvm::StringRef GetPluginDescriptionStatic();
+
+  static lldb_private::SymbolFile *
+  CreateInstance(lldb::ObjectFileSP objfile_sp);
+
+  llvm::StringRef GetPluginName() override { return GetPluginNameStatic(); }
+
+  uint32_t CalculateAbilities() override;
+
+  lldb::LanguageType ParseLanguage(CompileUnit &comp_unit) override {
+    return lldb::eLanguageTypeUnknown;
+  }
+
+  size_t ParseFunctions(CompileUnit &comp_unit) override { return 0; }
+
+  bool ParseLineTable(CompileUnit &comp_unit) override { return false; }
+
+  bool ParseDebugMacros(CompileUnit &comp_unit) override { return false; }
+
+  bool ParseSupportFiles(CompileUnit &comp_unit,
+                         FileSpecList &support_files) override {
+    return false;
+  }
+
+  size_t ParseTypes(CompileUnit &cu) override { return 0; }
+
+  bool ParseImportedModules(
+      const SymbolContext &sc,
+      std::vector<lldb_private::SourceModule> &imported_modules) override {
+    return false;
+  }
+
+  size_t ParseBlocksRecursive(Function &func) override { return 0; }
+
+  size_t ParseVariablesForContext(const SymbolContext &sc) override {
+    return 0;
+  }
+
+  uint32_t CalculateNumCompileUnits() override { return 0; }
+
+  lldb::CompUnitSP ParseCompileUnitAtIndex(uint32_t index) override;
+
+  Type *ResolveTypeUID(lldb::user_id_t type_uid) override { return nullptr; }
+  std::optional<ArrayInfo> GetDynamicArrayInfoForUID(
+      lldb::user_id_t type_uid,
+      const lldb_private::ExecutionContext *exe_ctx) override {
+    return std::nullopt;
+  }
+
+  bool CompleteType(CompilerType &compiler_type) override { return false; }
+
+  uint32_t ResolveSymbolContext(const lldb_private::Address &so_addr,
+                                lldb::SymbolContextItem resolve_scope,
+                                lldb_private::SymbolContext &sc) override;
+
+  void GetTypes(lldb_private::SymbolContextScope *sc_scope,
+                lldb::TypeClass type_mask,
+                lldb_private::TypeList &type_list) override;
+
+  void AddSymbols(Symtab &symtab) override;
+
+private:
+  lldb::addr_t GetBaseFileAddress();
+
+  std::vector<std::pair<uint64_t, std::string>> m_symbols;
+};
+} // namespace lldb_private
+
+#endif // LLDB_SOURCE_PLUGINS_SYMBOLFILE_JSON_SYMBOLFILETEXT_H
diff --git a/lldb/source/Plugins/SymbolFile/NativePDB/CompileUnitIndex.cpp b/lldb/source/Plugins/SymbolFile/NativePDB/CompileUnitIndex.cpp
index e2e06491e8c17..06cb720b1e9f7 100644
--- a/lldb/source/Plugins/SymbolFile/NativePDB/CompileUnitIndex.cpp
+++ b/lldb/source/Plugins/SymbolFile/NativePDB/CompileUnitIndex.cpp
@@ -162,9 +162,13 @@ CompilandIndexItem &CompileUnitIndex::GetOrCreateCompiland(uint16_t modi) {
   ParseExtendedInfo(m_index, *cci);
   ParseInlineeLineTableForCompileUnit(*cci);
 
-  cci->m_strings.initialize(cci->m_debug_stream.getSubsectionsArray());
-  PDBStringTable &strings = cantFail(m_index.pdb().getStringTable());
-  cci->m_strings.setStrings(strings.getStringTable());
+  auto strings = m_index.pdb().getStringTable();
+  if (strings) {
+    cci->m_strings.initialize(cci->m_debug_stream.getSubsectionsArray());
+    cci->m_strings.setStrings(strings->getStringTable());
+  } else {
+    consumeError(strings.takeError());
+  }
 
   // We want the main source file to always comes first.  Note that we can't
   // just push_back the main file onto the front because `GetMainSourceFile`
diff --git a/lldb/source/Plugins/SymbolFile/NativePDB/PdbUtil.cpp b/lldb/source/Plugins/SymbolFile/NativePDB/PdbUtil.cpp
index 164bfa8b3726a..888bd89a72625 100644
--- a/lldb/source/Plugins/SymbolFile/NativePDB/PdbUtil.cpp
+++ b/lldb/source/Plugins/SymbolFile/NativePDB/PdbUtil.cpp
@@ -703,8 +703,12 @@ static bool GetFrameDataProgram(PdbIndex &index,
   if (frame_data_it == new_fpo_data.end())
     return false;
 
-  PDBStringTable &strings = cantFail(index.pdb().getStringTable());
-  out_program = cantFail(strings.getStringForID(frame_data_it->FrameFunc));
+  auto strings = index.pdb().getStringTable();
+  if (!strings) {
+    consumeError(strings.takeError());
+    return false;
+  }
+  out_program = cantFail(strings->getStringForID(frame_data_it->FrameFunc));
   return true;
 }
 
diff --git a/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.cpp b/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.cpp
index 82d3707acfa4b..df558c842f9c1 100644
--- a/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.cpp
+++ b/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.cpp
@@ -76,6 +76,10 @@ static lldb::LanguageType TranslateLanguage(PDB_Lang lang) {
     return lldb::LanguageType::eLanguageTypeSwift;
   case PDB_Lang::Rust:
     return lldb::LanguageType::eLanguageTypeRust;
+  case PDB_Lang::ObjC:
+    return lldb::LanguageType::eLanguageTypeObjC;
+  case PDB_Lang::ObjCpp:
+    return lldb::LanguageType::eLanguageTypeObjC_plus_plus;
   default:
     return lldb::LanguageType::eLanguageTypeUnknown;
   }
@@ -1341,6 +1345,9 @@ bool SymbolFileNativePDB::ParseDebugMacros(CompileUnit &comp_unit) {
 llvm::Expected<uint32_t>
 SymbolFileNativePDB::GetFileIndex(const CompilandIndexItem &cii,
                                   uint32_t file_id) {
+  if (!cii.m_strings.hasChecksums() || !cii.m_strings.hasStrings())
+    return llvm::make_error<RawError>(raw_error_code::no_entry);
+
   const auto &checksums = cii.m_strings.checksums().getArray();
   const auto &strings = cii.m_strings.strings();
   // Indices in this structure are actually offsets of records in the
@@ -1389,7 +1396,7 @@ bool SymbolFileNativePDB::ParseImportedModules(
 void SymbolFileNativePDB::ParseInlineSite(PdbCompilandSymId id,
                                           Address func_addr) {
   lldb::user_id_t opaque_uid = toOpaqueUid(id);
-  if (m_inline_sites.find(opaque_uid) != m_inline_sites.end())
+  if (m_inline_sites.contains(opaque_uid))
     return;
 
   addr_t func_base = func_addr.GetFileAddress();
diff --git a/lldb/source/Plugins/SymbolFile/PDB/PDBASTParser.cpp b/lldb/source/Plugins/SymbolFile/PDB/PDBASTParser.cpp
index c9236da4842cf..da57338ffb58a 100644
--- a/lldb/source/Plugins/SymbolFile/PDB/PDBASTParser.cpp
+++ b/lldb/source/Plugins/SymbolFile/PDB/PDBASTParser.cpp
@@ -190,8 +190,7 @@ static ConstString GetPDBBuiltinTypeName(const PDBSymbolTypeBuiltin &pdb_type,
   return compiler_type.GetTypeName();
 }
 
-static bool GetDeclarationForSymbol(const PDBSymbol &symbol,
-                                    Declaration &decl) {
+static bool AddSourceInfoToDecl(const PDBSymbol &symbol, Declaration &decl) {
   auto &raw_sym = symbol.getRawSymbol();
   auto first_line_up = raw_sym.getSrcLineOnTypeDefn();
 
@@ -464,7 +463,7 @@ lldb::TypeSP PDBASTParser::CreateLLDBTypeFromPDBType(const PDBSymbol &type) {
     if (udt->isVolatileType())
       clang_type = clang_type.AddVolatileModifier();
 
-    GetDeclarationForSymbol(type, decl);
+    AddSourceInfoToDecl(type, decl);
     return m_ast.GetSymbolFile()->MakeType(
         type.getSymIndexId(), ConstString(name), udt->getLength(), nullptr,
         LLDB_INVALID_UID, lldb_private::Type::eEncodingIsUID, decl, clang_type,
@@ -533,7 +532,7 @@ lldb::TypeSP PDBASTParser::CreateLLDBTypeFromPDBType(const PDBSymbol &type) {
     if (enum_type->isVolatileType())
       ast_enum = ast_enum.AddVolatileModifier();
 
-    GetDeclarationForSymbol(type, decl);
+    AddSourceInfoToDecl(type, decl);
     return m_ast.GetSymbolFile()->MakeType(
         type.getSymIndexId(), ConstString(name), bytes, nullptr,
         LLDB_INVALID_UID, lldb_private::Type::eEncodingIsUID, decl, ast_enum,
@@ -579,7 +578,7 @@ lldb::TypeSP PDBASTParser::CreateLLDBTypeFromPDBType(const PDBSymbol &type) {
     if (type_def->isVolatileType())
       ast_typedef = ast_typedef.AddVolatileModifier();
 
-    GetDeclarationForSymbol(type, decl);
+    AddSourceInfoToDecl(type, decl);
     std::optional<uint64_t> size;
     if (type_def->getLength())
       size = type_def->getLength();
@@ -659,7 +658,7 @@ lldb::TypeSP PDBASTParser::CreateLLDBTypeFromPDBType(const PDBSymbol &type) {
         m_ast.CreateFunctionType(return_ast_type, arg_list.data(),
                                  arg_list.size(), is_variadic, type_quals, cc);
 
-    GetDeclarationForSymbol(type, decl);
+    AddSourceInfoToDecl(type, decl);
     return m_ast.GetSymbolFile()->MakeType(
         type.getSymIndexId(), ConstString(name), std::nullopt, nullptr,
         LLDB_INVALID_UID, lldb_private::Type::eEncodingIsUID, decl,
diff --git a/lldb/source/Plugins/SymbolFile/PDB/SymbolFilePDB.cpp b/lldb/source/Plugins/SymbolFile/PDB/SymbolFilePDB.cpp
index cb75dd59d3668..613e362071270 100644
--- a/lldb/source/Plugins/SymbolFile/PDB/SymbolFilePDB.cpp
+++ b/lldb/source/Plugins/SymbolFile/PDB/SymbolFilePDB.cpp
@@ -80,6 +80,10 @@ lldb::LanguageType TranslateLanguage(PDB_Lang lang) {
     return lldb::LanguageType::eLanguageTypeSwift;
   case PDB_Lang::Rust:
     return lldb::LanguageType::eLanguageTypeRust;
+  case PDB_Lang::ObjC:
+    return lldb::LanguageType::eLanguageTypeObjC;
+  case PDB_Lang::ObjCpp:
+    return lldb::LanguageType::eLanguageTypeObjC_plus_plus;
   default:
     return lldb::LanguageType::eLanguageTypeUnknown;
   }
diff --git a/lldb/source/Plugins/SymbolVendor/ELF/SymbolVendorELF.cpp b/lldb/source/Plugins/SymbolVendor/ELF/SymbolVendorELF.cpp
index 61e06cdfa02df..55a663bb1b963 100644
--- a/lldb/source/Plugins/SymbolVendor/ELF/SymbolVendorELF.cpp
+++ b/lldb/source/Plugins/SymbolVendor/ELF/SymbolVendorELF.cpp
@@ -111,6 +111,9 @@ SymbolVendorELF::CreateInstance(const lldb::ModuleSP &module_sp,
   SectionList *module_section_list = module_sp->GetSectionList();
   SectionList *objfile_section_list = dsym_objfile_sp->GetSectionList();
 
+  if (!module_section_list || !objfile_section_list)
+    return nullptr;
+
   static const SectionType g_sections[] = {
       eSectionTypeDWARFDebugAbbrev,     eSectionTypeDWARFDebugAddr,
       eSectionTypeDWARFDebugAranges,    eSectionTypeDWARFDebugCuIndex,
diff --git a/lldb/source/Plugins/SymbolVendor/wasm/SymbolVendorWasm.cpp b/lldb/source/Plugins/SymbolVendor/wasm/SymbolVendorWasm.cpp
index 6290843a89872..91b10ea64535d 100644
--- a/lldb/source/Plugins/SymbolVendor/wasm/SymbolVendorWasm.cpp
+++ b/lldb/source/Plugins/SymbolVendor/wasm/SymbolVendorWasm.cpp
@@ -109,6 +109,9 @@ SymbolVendorWasm::CreateInstance(const lldb::ModuleSP &module_sp,
   SectionList *module_section_list = module_sp->GetSectionList();
   SectionList *objfile_section_list = sym_objfile_sp->GetSectionList();
 
+  if (!module_section_list || !objfile_section_list)
+    return nullptr;
+
   static const SectionType g_sections[] = {
       eSectionTypeDWARFDebugAbbrev,   eSectionTypeDWARFDebugAddr,
       eSectionTypeDWARFDebugAranges,  eSectionTypeDWARFDebugCuIndex,
diff --git a/lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.cpp b/lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.cpp
index f26147e7d408e..a739494cffcc4 100644
--- a/lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.cpp
+++ b/lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.cpp
@@ -8,6 +8,8 @@
 
 #include "TypeSystemClang.h"
 
+#include "clang/AST/DeclBase.h"
+#include "llvm/Support/Casting.h"
 #include "llvm/Support/FormatAdapters.h"
 #include "llvm/Support/FormatVariadic.h"
 
@@ -3725,6 +3727,22 @@ bool TypeSystemClang::SupportsLanguage(lldb::LanguageType language) {
   return TypeSystemClangSupportsLanguage(language);
 }
 
+ConstString
+TypeSystemClang::GetInstanceVariableName(lldb::LanguageType language) {
+  switch (language) {
+  case LanguageType::eLanguageTypeC_plus_plus:
+  case LanguageType::eLanguageTypeC_plus_plus_03:
+  case LanguageType::eLanguageTypeC_plus_plus_11:
+  case LanguageType::eLanguageTypeC_plus_plus_14:
+    return ConstString("this");
+  case LanguageType::eLanguageTypeObjC:
+  case LanguageType::eLanguageTypeObjC_plus_plus:
+    return ConstString("self");
+  default:
+    return {};
+  }
+}
+
 std::optional<std::string>
 TypeSystemClang::GetCXXClassName(const CompilerType &type) {
   if (!type)
@@ -4972,10 +4990,10 @@ lldb::Encoding TypeSystemClang::GetEncoding(lldb::opaque_compiler_type_t type,
     case clang::BuiltinType::OCLIntelSubgroupAVCImeResult:
     case clang::BuiltinType::OCLIntelSubgroupAVCRefResult:
     case clang::BuiltinType::OCLIntelSubgroupAVCSicResult:
-    case clang::BuiltinType::OCLIntelSubgroupAVCImeResultSingleRefStreamout:
-    case clang::BuiltinType::OCLIntelSubgroupAVCImeResultDualRefStreamout:
-    case clang::BuiltinType::OCLIntelSubgroupAVCImeSingleRefStreamin:
-    case clang::BuiltinType::OCLIntelSubgroupAVCImeDualRefStreamin:
+    case clang::BuiltinType::OCLIntelSubgroupAVCImeResultSingleReferenceStreamout:
+    case clang::BuiltinType::OCLIntelSubgroupAVCImeResultDualReferenceStreamout:
+    case clang::BuiltinType::OCLIntelSubgroupAVCImeSingleReferenceStreamin:
+    case clang::BuiltinType::OCLIntelSubgroupAVCImeDualReferenceStreamin:
       break;
 
     // PowerPC -- Matrix Multiply Assist
@@ -4985,6 +5003,9 @@ lldb::Encoding TypeSystemClang::GetEncoding(lldb::opaque_compiler_type_t type,
 
     // ARM -- Scalable Vector Extension
     case clang::BuiltinType::SveBool:
+    case clang::BuiltinType::SveBoolx2:
+    case clang::BuiltinType::SveBoolx4:
+    case clang::BuiltinType::SveCount:
     case clang::BuiltinType::SveInt8:
     case clang::BuiltinType::SveInt8x2:
     case clang::BuiltinType::SveInt8x3:
@@ -9750,32 +9771,21 @@ TypeSystemClang::DeclContextGetScopeQualifiedName(void *opaque_decl_ctx) {
   return ConstString();
 }
 
-bool TypeSystemClang::DeclContextIsClassMethod(
-    void *opaque_decl_ctx, ConstString *language_object_name_ptr) {
-  if (opaque_decl_ctx) {
-    clang::DeclContext *decl_ctx = (clang::DeclContext *)opaque_decl_ctx;
-    if (ObjCMethodDecl *objc_method =
-            llvm::dyn_cast<clang::ObjCMethodDecl>(decl_ctx)) {
-      if (objc_method->isInstanceMethod())
-        if (language_object_name_ptr)
-          language_object_name_ptr->SetCString("self");
-      return true;
-    } else if (CXXMethodDecl *cxx_method =
-                   llvm::dyn_cast<clang::CXXMethodDecl>(decl_ctx)) {
-      if (cxx_method->isInstance())
-        if (language_object_name_ptr)
-          language_object_name_ptr->SetCString("this");
-      return true;
-    } else if (clang::FunctionDecl *function_decl =
-                   llvm::dyn_cast<clang::FunctionDecl>(decl_ctx)) {
-      ClangASTMetadata *metadata = GetMetadata(function_decl);
-      if (metadata && metadata->HasObjectPtr()) {
-        if (language_object_name_ptr)
-          language_object_name_ptr->SetCString(metadata->GetObjectPtrName());
-        return true;
-      }
-    }
+bool TypeSystemClang::DeclContextIsClassMethod(void *opaque_decl_ctx) {
+  if (!opaque_decl_ctx)
+    return false;
+
+  clang::DeclContext *decl_ctx = (clang::DeclContext *)opaque_decl_ctx;
+  if (llvm::isa<clang::ObjCMethodDecl>(decl_ctx)) {
+    return true;
+  } else if (llvm::isa<clang::CXXMethodDecl>(decl_ctx)) {
+    return true;
+  } else if (clang::FunctionDecl *fun_decl =
+                 llvm::dyn_cast<clang::FunctionDecl>(decl_ctx)) {
+    if (ClangASTMetadata *metadata = GetMetadata(fun_decl))
+      return metadata->HasObjectPtr();
   }
+
   return false;
 }
 
@@ -9796,6 +9806,24 @@ bool TypeSystemClang::DeclContextIsContainedInLookup(
   return false;
 }
 
+lldb::LanguageType
+TypeSystemClang::DeclContextGetLanguage(void *opaque_decl_ctx) {
+  if (!opaque_decl_ctx)
+    return eLanguageTypeUnknown;
+
+  auto *decl_ctx = (clang::DeclContext *)opaque_decl_ctx;
+  if (llvm::isa<clang::ObjCMethodDecl>(decl_ctx)) {
+    return eLanguageTypeObjC;
+  } else if (llvm::isa<clang::CXXMethodDecl>(decl_ctx)) {
+    return eLanguageTypeC_plus_plus;
+  } else if (auto *fun_decl = llvm::dyn_cast<clang::FunctionDecl>(decl_ctx)) {
+    if (ClangASTMetadata *metadata = GetMetadata(fun_decl))
+      return metadata->GetObjectPtrLanguage();
+  }
+
+  return eLanguageTypeUnknown;
+}
+
 static bool IsClangDeclContext(const CompilerDeclContext &dc) {
   return dc.IsValid() && isa<TypeSystemClang>(dc.GetTypeSystem());
 }
diff --git a/lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.h b/lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.h
index 99021f9b76bda..baddf6253beb4 100644
--- a/lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.h
+++ b/lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.h
@@ -579,12 +579,13 @@ class TypeSystemClang : public TypeSystem {
 
   ConstString DeclContextGetScopeQualifiedName(void *opaque_decl_ctx) override;
 
-  bool DeclContextIsClassMethod(void *opaque_decl_ctx,
-                                ConstString *language_object_name_ptr) override;
+  bool DeclContextIsClassMethod(void *opaque_decl_ctx) override;
 
   bool DeclContextIsContainedInLookup(void *opaque_decl_ctx,
                                       void *other_opaque_decl_ctx) override;
 
+  lldb::LanguageType DeclContextGetLanguage(void *opaque_decl_ctx) override;
+
   // Clang specific clang::DeclContext functions
 
   static clang::DeclContext *
@@ -710,6 +711,8 @@ class TypeSystemClang : public TypeSystem {
 
   bool SupportsLanguage(lldb::LanguageType language) override;
 
+  ConstString GetInstanceVariableName(lldb::LanguageType language) override;
+
   static std::optional<std::string> GetCXXClassName(const CompilerType &type);
 
   // Type Completion
diff --git a/lldb/source/Symbol/CompilerDeclContext.cpp b/lldb/source/Symbol/CompilerDeclContext.cpp
index 5b3049f74bbc8..36b9131055f83 100644
--- a/lldb/source/Symbol/CompilerDeclContext.cpp
+++ b/lldb/source/Symbol/CompilerDeclContext.cpp
@@ -34,13 +34,25 @@ ConstString CompilerDeclContext::GetScopeQualifiedName() const {
   return ConstString();
 }
 
-bool CompilerDeclContext::IsClassMethod(ConstString *language_object_name_ptr) {
+bool CompilerDeclContext::IsClassMethod() {
   if (IsValid())
-    return m_type_system->DeclContextIsClassMethod(m_opaque_decl_ctx,
-                                                   language_object_name_ptr);
+    return m_type_system->DeclContextIsClassMethod(m_opaque_decl_ctx);
   return false;
 }
 
+lldb::LanguageType CompilerDeclContext::GetLanguage() {
+  if (IsValid())
+    return m_type_system->DeclContextGetLanguage(m_opaque_decl_ctx);
+  return {};
+}
+
+ConstString
+CompilerDeclContext::GetInstanceVariableName(lldb::LanguageType language) {
+  if (IsValid())
+    return m_type_system->GetInstanceVariableName(language);
+  return {};
+}
+
 bool CompilerDeclContext::IsContainedInLookup(CompilerDeclContext other) const {
   if (!IsValid())
     return false;
diff --git a/lldb/source/Symbol/Symbol.cpp b/lldb/source/Symbol/Symbol.cpp
index 067b7b900d89d..ee42df9e4d469 100644
--- a/lldb/source/Symbol/Symbol.cpp
+++ b/lldb/source/Symbol/Symbol.cpp
@@ -19,6 +19,7 @@
 #include "lldb/Target/Target.h"
 #include "lldb/Utility/DataEncoder.h"
 #include "lldb/Utility/Stream.h"
+#include "llvm/ADT/StringSwitch.h"
 
 using namespace lldb;
 using namespace lldb_private;
@@ -95,6 +96,55 @@ const Symbol &Symbol::operator=(const Symbol &rhs) {
   return *this;
 }
 
+llvm::Expected<Symbol> Symbol::FromJSON(const JSONSymbol &symbol,
+                                        SectionList *section_list) {
+  if (!section_list)
+    return llvm::make_error<llvm::StringError>("no section list provided",
+                                               llvm::inconvertibleErrorCode());
+
+  if (!symbol.value && !symbol.address)
+    return llvm::make_error<llvm::StringError>(
+        "symbol must contain either a value or an address",
+        llvm::inconvertibleErrorCode());
+
+  if (symbol.value && symbol.address)
+    return llvm::make_error<llvm::StringError>(
+        "symbol cannot contain both a value and an address",
+        llvm::inconvertibleErrorCode());
+
+  const uint64_t size = symbol.size.value_or(0);
+  const bool is_artificial = false;
+  const bool is_trampoline = false;
+  const bool is_debug = false;
+  const bool external = false;
+  const bool size_is_valid = symbol.size.has_value();
+  const bool contains_linker_annotations = false;
+  const uint32_t flags = 0;
+
+  if (symbol.address) {
+    if (SectionSP section_sp =
+            section_list->FindSectionContainingFileAddress(*symbol.address)) {
+      const uint64_t offset = *symbol.address - section_sp->GetFileAddress();
+      return Symbol(symbol.id.value_or(0), Mangled(symbol.name),
+                    symbol.type.value_or(eSymbolTypeAny), external, is_debug,
+                    is_trampoline, is_artificial,
+                    AddressRange(section_sp, offset, size), size_is_valid,
+                    contains_linker_annotations, flags);
+    }
+    return llvm::make_error<llvm::StringError>(
+        llvm::formatv("no section found for address: {0:x}", *symbol.address),
+        llvm::inconvertibleErrorCode());
+  }
+
+  // Absolute symbols encode the integer value in the m_offset of the
+  // AddressRange object and the section is set to nothing.
+  return Symbol(symbol.id.value_or(0), Mangled(symbol.name),
+                symbol.type.value_or(eSymbolTypeAny), external, is_debug,
+                is_trampoline, is_artificial,
+                AddressRange(SectionSP(), *symbol.value, size), size_is_valid,
+                contains_linker_annotations, flags);
+}
+
 void Symbol::Clear() {
   m_uid = UINT32_MAX;
   m_mangled.Clear();
@@ -622,14 +672,14 @@ bool Symbol::Decode(const DataExtractor &data, lldb::offset_t *offset_ptr,
   const bool is_addr = data.GetU8(offset_ptr) != 0;
   const uint64_t value = data.GetU64(offset_ptr);
   if (is_addr) {
-    m_addr_range.GetBaseAddress().ResolveAddressUsingFileSections(
-        value, section_list);
+    m_addr_range.GetBaseAddress().ResolveAddressUsingFileSections(value,
+                                                                  section_list);
   } else {
     m_addr_range.GetBaseAddress().Clear();
     m_addr_range.GetBaseAddress().SetOffset(value);
   }
   m_addr_range.SetByteSize(data.GetU64(offset_ptr));
-  m_flags =  data.GetU32(offset_ptr);
+  m_flags = data.GetU32(offset_ptr);
   return true;
 }
 
@@ -723,3 +773,77 @@ bool Symbol::operator==(const Symbol &rhs) const {
     return false;
   return true;
 }
+
+namespace llvm {
+namespace json {
+
+bool fromJSON(const llvm::json::Value &value, lldb_private::JSONSymbol &symbol,
+              llvm::json::Path path) {
+  llvm::json::ObjectMapper o(value, path);
+  const bool mapped = o && o.map("value", symbol.value) &&
+                      o.map("address", symbol.address) &&
+                      o.map("size", symbol.size) && o.map("id", symbol.id) &&
+                      o.map("type", symbol.type) && o.map("name", symbol.name);
+
+  if (!mapped)
+    return false;
+
+  if (!symbol.value && !symbol.address) {
+    path.report("symbol must have either a value or an address");
+    return false;
+  }
+
+  if (symbol.value && symbol.address) {
+    path.report("symbol cannot have both a value and an address");
+    return false;
+  }
+
+  return true;
+}
+
+bool fromJSON(const llvm::json::Value &value, lldb::SymbolType &type,
+              llvm::json::Path path) {
+  if (auto str = value.getAsString()) {
+    type = llvm::StringSwitch<lldb::SymbolType>(*str)
+               .Case("absolute", eSymbolTypeAbsolute)
+               .Case("code", eSymbolTypeCode)
+               .Case("resolver", eSymbolTypeResolver)
+               .Case("data", eSymbolTypeData)
+               .Case("trampoline", eSymbolTypeTrampoline)
+               .Case("runtime", eSymbolTypeRuntime)
+               .Case("exception", eSymbolTypeException)
+               .Case("sourcefile", eSymbolTypeSourceFile)
+               .Case("headerfile", eSymbolTypeHeaderFile)
+               .Case("objectfile", eSymbolTypeObjectFile)
+               .Case("commonblock", eSymbolTypeCommonBlock)
+               .Case("block", eSymbolTypeBlock)
+               .Case("local", eSymbolTypeLocal)
+               .Case("param", eSymbolTypeParam)
+               .Case("variable", eSymbolTypeVariable)
+               .Case("variableType", eSymbolTypeVariableType)
+               .Case("lineentry", eSymbolTypeLineEntry)
+               .Case("lineheader", eSymbolTypeLineHeader)
+               .Case("scopebegin", eSymbolTypeScopeBegin)
+               .Case("scopeend", eSymbolTypeScopeEnd)
+               .Case("additional,", eSymbolTypeAdditional)
+               .Case("compiler", eSymbolTypeCompiler)
+               .Case("instrumentation", eSymbolTypeInstrumentation)
+               .Case("undefined", eSymbolTypeUndefined)
+               .Case("objcclass", eSymbolTypeObjCClass)
+               .Case("objcmetaClass", eSymbolTypeObjCMetaClass)
+               .Case("objcivar", eSymbolTypeObjCIVar)
+               .Case("reexporte", eSymbolTypeReExported)
+               .Default(eSymbolTypeInvalid);
+
+    if (type == eSymbolTypeInvalid) {
+      path.report("invalid symbol type");
+      return false;
+    }
+
+    return true;
+  }
+  path.report("expected string");
+  return false;
+}
+} // namespace json
+} // namespace llvm
diff --git a/lldb/source/Symbol/SymbolContext.cpp b/lldb/source/Symbol/SymbolContext.cpp
index 8453c1f116fef..5d4fb1cec6969 100644
--- a/lldb/source/Symbol/SymbolContext.cpp
+++ b/lldb/source/Symbol/SymbolContext.cpp
@@ -539,14 +539,16 @@ Block *SymbolContext::GetFunctionBlock() {
   return nullptr;
 }
 
-bool SymbolContext::GetFunctionMethodInfo(ConstString &language_object_name) {
-  Block *function_block = GetFunctionBlock();
-  if (function_block) {
-    CompilerDeclContext decl_ctx = function_block->GetDeclContext();
-    if (decl_ctx)
-      return decl_ctx.IsClassMethod(&language_object_name);
-  }
-  return false;
+ConstString SymbolContext::GetInstanceVariableName() {
+  if (Block *function_block = GetFunctionBlock())
+    if (CompilerDeclContext decl_ctx = function_block->GetDeclContext()) {
+      auto language = decl_ctx.GetLanguage();
+      if (language == eLanguageTypeUnknown)
+        language = GetLanguage();
+      return decl_ctx.GetInstanceVariableName(language);
+    }
+
+  return {};
 }
 
 void SymbolContext::SortTypeList(TypeMap &type_map, TypeList &type_list) const {
diff --git a/lldb/source/Target/Memory.cpp b/lldb/source/Target/Memory.cpp
index fdc01389721ed..45786415d23b1 100644
--- a/lldb/source/Target/Memory.cpp
+++ b/lldb/source/Target/Memory.cpp
@@ -123,18 +123,55 @@ bool MemoryCache::RemoveInvalidRange(lldb::addr_t base_addr,
   return false;
 }
 
+lldb::DataBufferSP MemoryCache::GetL2CacheLine(lldb::addr_t line_base_addr,
+                                               Status &error) {
+  // This function assumes that the address given is aligned correctly.
+  assert((line_base_addr % m_L2_cache_line_byte_size) == 0);
+
+  std::lock_guard<std::recursive_mutex> guard(m_mutex);
+  auto pos = m_L2_cache.find(line_base_addr);
+  if (pos != m_L2_cache.end())
+    return pos->second;
+
+  auto data_buffer_heap_sp =
+      std::make_shared<DataBufferHeap>(m_L2_cache_line_byte_size, 0);
+  size_t process_bytes_read = m_process.ReadMemoryFromInferior(
+      line_base_addr, data_buffer_heap_sp->GetBytes(),
+      data_buffer_heap_sp->GetByteSize(), error);
+
+  // If we failed a read, not much we can do.
+  if (process_bytes_read == 0)
+    return lldb::DataBufferSP();
+
+  // If we didn't get a complete read, we can still cache what we did get.
+  if (process_bytes_read < m_L2_cache_line_byte_size)
+    data_buffer_heap_sp->SetByteSize(process_bytes_read);
+
+  m_L2_cache[line_base_addr] = data_buffer_heap_sp;
+  return data_buffer_heap_sp;
+}
+
 size_t MemoryCache::Read(addr_t addr, void *dst, size_t dst_len,
                          Status &error) {
-  size_t bytes_left = dst_len;
-
-  // Check the L1 cache for a range that contain the entire memory read. If we
-  // find a range in the L1 cache that does, we use it. Else we fall back to
-  // reading memory in m_L2_cache_line_byte_size byte sized chunks. The L1
-  // cache contains chunks of memory that are not required to be
-  // m_L2_cache_line_byte_size bytes in size, so we don't try anything tricky
-  // when reading from them (no partial reads from the L1 cache).
+  if (!dst || dst_len == 0)
+    return 0;
 
   std::lock_guard<std::recursive_mutex> guard(m_mutex);
+  // FIXME: We should do a more thorough check to make sure that we're not
+  // overlapping with any invalid ranges (e.g. Read 0x100 - 0x200 but there's an
+  // invalid range 0x180 - 0x280). `FindEntryThatContains` has an implementation
+  // that takes a range, but it only checks to see if the argument is contained
+  // by an existing invalid range. It cannot check if the argument contains
+  // invalid ranges and cannot check for overlaps.
+  if (m_invalid_ranges.FindEntryThatContains(addr)) {
+    error.SetErrorStringWithFormat("memory read failed for 0x%" PRIx64, addr);
+    return 0;
+  }
+
+  // Check the L1 cache for a range that contains the entire memory read.
+  // L1 cache contains chunks of memory that are not required to be the size of
+  // an L2 cache line. We avoid trying to do partial reads from the L1 cache to
+  // simplify the implementation.
   if (!m_L1_cache.empty()) {
     AddrRange read_range(addr, dst_len);
     BlockMap::iterator pos = m_L1_cache.upper_bound(addr);
@@ -149,105 +186,82 @@ size_t MemoryCache::Read(addr_t addr, void *dst, size_t dst_len,
     }
   }
 
-  // If this memory read request is larger than the cache line size, then we
-  // (1) try to read as much of it at once as possible, and (2) don't add the
-  // data to the memory cache.  We don't want to split a big read up into more
-  // separate reads than necessary, and with a large memory read request, it is
-  // unlikely that the caller function will ask for the next
-  // 4 bytes after the large memory read - so there's little benefit to saving
-  // it in the cache.
-  if (dst && dst_len > m_L2_cache_line_byte_size) {
+  // If the size of the read is greater than the size of an L2 cache line, we'll
+  // just read from the inferior. If that read is successful, we'll cache what
+  // we read in the L1 cache for future use.
+  if (dst_len > m_L2_cache_line_byte_size) {
     size_t bytes_read =
         m_process.ReadMemoryFromInferior(addr, dst, dst_len, error);
-    // Add this non block sized range to the L1 cache if we actually read
-    // anything
     if (bytes_read > 0)
       AddL1CacheData(addr, dst, bytes_read);
     return bytes_read;
   }
 
-  if (dst && bytes_left > 0) {
-    const uint32_t cache_line_byte_size = m_L2_cache_line_byte_size;
-    uint8_t *dst_buf = (uint8_t *)dst;
-    addr_t curr_addr = addr - (addr % cache_line_byte_size);
-    addr_t cache_offset = addr - curr_addr;
-
-    while (bytes_left > 0) {
-      if (m_invalid_ranges.FindEntryThatContains(curr_addr)) {
-        error.SetErrorStringWithFormat("memory read failed for 0x%" PRIx64,
-                                       curr_addr);
-        return dst_len - bytes_left;
-      }
-
-      BlockMap::const_iterator pos = m_L2_cache.find(curr_addr);
-      BlockMap::const_iterator end = m_L2_cache.end();
-
-      if (pos != end) {
-        size_t curr_read_size = cache_line_byte_size - cache_offset;
-        if (curr_read_size > bytes_left)
-          curr_read_size = bytes_left;
-
-        memcpy(dst_buf + dst_len - bytes_left,
-               pos->second->GetBytes() + cache_offset, curr_read_size);
-
-        bytes_left -= curr_read_size;
-        curr_addr += curr_read_size + cache_offset;
-        cache_offset = 0;
-
-        if (bytes_left > 0) {
-          // Get sequential cache page hits
-          for (++pos; (pos != end) && (bytes_left > 0); ++pos) {
-            assert((curr_addr % cache_line_byte_size) == 0);
-
-            if (pos->first != curr_addr)
-              break;
-
-            curr_read_size = pos->second->GetByteSize();
-            if (curr_read_size > bytes_left)
-              curr_read_size = bytes_left;
+  // If the size of the read fits inside one L2 cache line, we'll try reading
+  // from the L2 cache. Note that if the range of memory we're reading sits
+  // between two contiguous cache lines, we'll touch two cache lines instead of
+  // just one.
+
+  // We're going to have all of our loads and reads be cache line aligned.
+  addr_t cache_line_offset = addr % m_L2_cache_line_byte_size;
+  addr_t cache_line_base_addr = addr - cache_line_offset;
+  DataBufferSP first_cache_line = GetL2CacheLine(cache_line_base_addr, error);
+  // If we get nothing, then the read to the inferior likely failed. Nothing to
+  // do here.
+  if (!first_cache_line)
+    return 0;
+
+  // If the cache line was not filled out completely and the offset is greater
+  // than what we have available, we can't do anything further here.
+  if (cache_line_offset >= first_cache_line->GetByteSize())
+    return 0;
+
+  uint8_t *dst_buf = (uint8_t *)dst;
+  size_t bytes_left = dst_len;
+  size_t read_size = first_cache_line->GetByteSize() - cache_line_offset;
+  if (read_size > bytes_left)
+    read_size = bytes_left;
+
+  memcpy(dst_buf + dst_len - bytes_left,
+         first_cache_line->GetBytes() + cache_line_offset, read_size);
+  bytes_left -= read_size;
+
+  // If the cache line was not filled out completely and we still have data to
+  // read, we can't do anything further.
+  if (first_cache_line->GetByteSize() < m_L2_cache_line_byte_size &&
+      bytes_left > 0)
+    return dst_len - bytes_left;
+
+  // We'll hit this scenario if our read straddles two cache lines.
+  if (bytes_left > 0) {
+    cache_line_base_addr += m_L2_cache_line_byte_size;
+
+    // FIXME: Until we are able to more thoroughly check for invalid ranges, we
+    // will have to check the second line to see if it is in an invalid range as
+    // well. See the check near the beginning of the function for more details.
+    if (m_invalid_ranges.FindEntryThatContains(cache_line_base_addr)) {
+      error.SetErrorStringWithFormat("memory read failed for 0x%" PRIx64,
+                                     cache_line_base_addr);
+      return dst_len - bytes_left;
+    }
 
-            memcpy(dst_buf + dst_len - bytes_left, pos->second->GetBytes(),
-                   curr_read_size);
+    DataBufferSP second_cache_line =
+        GetL2CacheLine(cache_line_base_addr, error);
+    if (!second_cache_line)
+      return dst_len - bytes_left;
 
-            bytes_left -= curr_read_size;
-            curr_addr += curr_read_size;
+    read_size = bytes_left;
+    if (read_size > second_cache_line->GetByteSize())
+      read_size = second_cache_line->GetByteSize();
 
-            // We have a cache page that succeeded to read some bytes but not
-            // an entire page. If this happens, we must cap off how much data
-            // we are able to read...
-            if (pos->second->GetByteSize() != cache_line_byte_size)
-              return dst_len - bytes_left;
-          }
-        }
-      }
+    memcpy(dst_buf + dst_len - bytes_left, second_cache_line->GetBytes(),
+           read_size);
+    bytes_left -= read_size;
 
-      // We need to read from the process
-
-      if (bytes_left > 0) {
-        assert((curr_addr % cache_line_byte_size) == 0);
-        std::unique_ptr<DataBufferHeap> data_buffer_heap_up(
-            new DataBufferHeap(cache_line_byte_size, 0));
-        size_t process_bytes_read = m_process.ReadMemoryFromInferior(
-            curr_addr, data_buffer_heap_up->GetBytes(),
-            data_buffer_heap_up->GetByteSize(), error);
-        if (process_bytes_read == 0)
-          return dst_len - bytes_left;
-
-        if (process_bytes_read != cache_line_byte_size) {
-          data_buffer_heap_up->SetByteSize(process_bytes_read);
-          if (process_bytes_read < data_buffer_heap_up->GetByteSize()) {
-            dst_len -= data_buffer_heap_up->GetByteSize() - process_bytes_read;
-            bytes_left = process_bytes_read;
-          }
-        }
-        m_L2_cache[curr_addr] = DataBufferSP(data_buffer_heap_up.release());
-        // We have read data and put it into the cache, continue through the
-        // loop again to get the data out of the cache...
-      }
-    }
+    return dst_len - bytes_left;
   }
 
-  return dst_len - bytes_left;
+  return dst_len;
 }
 
 AllocatedBlock::AllocatedBlock(lldb::addr_t addr, uint32_t byte_size,
diff --git a/lldb/source/Target/Process.cpp b/lldb/source/Target/Process.cpp
index 8bfebe3c8ff6a..77ee27687fefb 100644
--- a/lldb/source/Target/Process.cpp
+++ b/lldb/source/Target/Process.cpp
@@ -402,8 +402,7 @@ ConstString &Process::GetStaticBroadcasterClass() {
 }
 
 Process::Process(lldb::TargetSP target_sp, ListenerSP listener_sp)
-    : Process(target_sp, listener_sp,
-              UnixSignals::Create(HostInfo::GetArchitecture())) {
+    : Process(target_sp, listener_sp, UnixSignals::CreateForHost()) {
   // This constructor just delegates to the full Process constructor,
   // defaulting to using the Host's UnixSignals.
 }
diff --git a/lldb/source/Target/StackFrame.cpp b/lldb/source/Target/StackFrame.cpp
index fa74b5d0ac2c6..f828539057cbf 100644
--- a/lldb/source/Target/StackFrame.cpp
+++ b/lldb/source/Target/StackFrame.cpp
@@ -567,23 +567,20 @@ ValueObjectSP StackFrame::GetValueForVariableExpressionPath(
     // Check for direct ivars access which helps us with implicit access to
     // ivars using "this" or "self".
     GetSymbolContext(eSymbolContextFunction | eSymbolContextBlock);
-    ConstString method_object_name;
-    if (m_sc.GetFunctionMethodInfo(method_object_name)) {
-      if (method_object_name) {
-        var_sp = variable_list->FindVariable(method_object_name);
-        if (var_sp) {
-          separator_idx = 0;
-          if (Type *var_type = var_sp->GetType())
-            if (auto compiler_type = var_type->GetForwardCompilerType())
-              if (!compiler_type.IsPointerType())
-                var_expr_storage = ".";
-
-          if (var_expr_storage.empty())
-            var_expr_storage = "->";
-          var_expr_storage += var_expr;
-          var_expr = var_expr_storage;
-          synthetically_added_instance_object = true;
-        }
+    if (auto instance_var_name = m_sc.GetInstanceVariableName()) {
+      var_sp = variable_list->FindVariable(instance_var_name);
+      if (var_sp) {
+        separator_idx = 0;
+        if (Type *var_type = var_sp->GetType())
+          if (auto compiler_type = var_type->GetForwardCompilerType())
+            if (!compiler_type.IsPointerType())
+              var_expr_storage = ".";
+
+        if (var_expr_storage.empty())
+          var_expr_storage = "->";
+        var_expr_storage += var_expr;
+        var_expr = var_expr_storage;
+        synthetically_added_instance_object = true;
       }
     }
   }
diff --git a/lldb/source/Target/StackFrameList.cpp b/lldb/source/Target/StackFrameList.cpp
index c782b506a9cb8..2610b1cd6c296 100644
--- a/lldb/source/Target/StackFrameList.cpp
+++ b/lldb/source/Target/StackFrameList.cpp
@@ -9,6 +9,7 @@
 #include "lldb/Target/StackFrameList.h"
 #include "lldb/Breakpoint/Breakpoint.h"
 #include "lldb/Breakpoint/BreakpointLocation.h"
+#include "lldb/Core/Debugger.h"
 #include "lldb/Core/SourceManager.h"
 #include "lldb/Core/StreamFile.h"
 #include "lldb/Symbol/Block.h"
@@ -470,7 +471,15 @@ void StackFrameList::GetFramesUpTo(uint32_t end_idx) {
   }
 
   StackFrameSP unwind_frame_sp;
+  Debugger &dbg = m_thread.GetProcess()->GetTarget().GetDebugger();
   do {
+    // Check for interruption here when building the frames - this is the
+    // expensive part, Dump later on is cheap.
+    if (dbg.InterruptRequested()) {
+      Log *log = GetLog(LLDBLog::Host);
+      LLDB_LOG(log, "Interrupted %s", __FUNCTION__);
+      break;
+    }
     uint32_t idx = m_concrete_frames_fetched++;
     lldb::addr_t pc = LLDB_INVALID_ADDRESS;
     lldb::addr_t cfa = LLDB_INVALID_ADDRESS;
diff --git a/lldb/source/Target/UnixSignals.cpp b/lldb/source/Target/UnixSignals.cpp
index 858b1691f3188..02354be44cfe8 100644
--- a/lldb/source/Target/UnixSignals.cpp
+++ b/lldb/source/Target/UnixSignals.cpp
@@ -9,7 +9,6 @@
 #include "lldb/Target/UnixSignals.h"
 #include "Plugins/Process/Utility/FreeBSDSignals.h"
 #include "Plugins/Process/Utility/LinuxSignals.h"
-#include "Plugins/Process/Utility/MipsLinuxSignals.h"
 #include "Plugins/Process/Utility/NetBSDSignals.h"
 #include "lldb/Host/HostInfo.h"
 #include "lldb/Utility/ArchSpec.h"
@@ -33,17 +32,8 @@ UnixSignals::Signal::Signal(const char *name, bool default_suppress,
 lldb::UnixSignalsSP UnixSignals::Create(const ArchSpec &arch) {
   const auto &triple = arch.GetTriple();
   switch (triple.getOS()) {
-  case llvm::Triple::Linux: {
-    switch (triple.getArch()) {
-    case llvm::Triple::mips:
-    case llvm::Triple::mipsel:
-    case llvm::Triple::mips64:
-    case llvm::Triple::mips64el:
-      return std::make_shared<MipsLinuxSignals>();
-    default:
-      return std::make_shared<LinuxSignals>();
-    }
-  }
+  case llvm::Triple::Linux:
+    return std::make_shared<LinuxSignals>();
   case llvm::Triple::FreeBSD:
   case llvm::Triple::OpenBSD:
     return std::make_shared<FreeBSDSignals>();
diff --git a/lldb/source/Utility/ProcessInfo.cpp b/lldb/source/Utility/ProcessInfo.cpp
index 9ffc409a46959..595c0b090401a 100644
--- a/lldb/source/Utility/ProcessInfo.cpp
+++ b/lldb/source/Utility/ProcessInfo.cpp
@@ -8,8 +8,8 @@
 
 #include "lldb/Utility/ProcessInfo.h"
 
-#include "lldb/Interpreter/ScriptedMetadata.h"
 #include "lldb/Utility/ArchSpec.h"
+#include "lldb/Utility/ScriptedMetadata.h"
 #include "lldb/Utility/Stream.h"
 #include "lldb/Utility/StreamString.h"
 #include "lldb/Utility/UserIDResolver.h"
diff --git a/lldb/test/API/api/check_public_api_headers/TestPublicAPIHeaders.py b/lldb/test/API/api/check_public_api_headers/TestPublicAPIHeaders.py
index b74e395b3e671..d57817d228e6c 100644
--- a/lldb/test/API/api/check_public_api_headers/TestPublicAPIHeaders.py
+++ b/lldb/test/API/api/check_public_api_headers/TestPublicAPIHeaders.py
@@ -9,6 +9,8 @@
 
 
 @skipIfNoSBHeaders
+@skipIfRemote
+@skipUnlessDarwin
 class SBDirCheckerCase(TestBase):
     NO_DEBUG_INFO_TESTCASE = True
 
@@ -20,12 +22,8 @@ def setUp(self):
     def test_sb_api_directory(self):
         """Test the SB API directory and make sure there's no unwanted stuff."""
 
-        # Only proceed if this is an Apple OS, "x86_64", and local platform.
-        if not (self.platformIsDarwin() and self.getArchitecture() == "x86_64"):
+        if not self.isAArch64() and self.getArchitecture() != "x86_64":
             self.skipTest("This test is only for LLDB.framework built 64-bit")
-        if self.getArchitecture() == "i386":
-            self.skipTest(
-                "LLDB is 64-bit and cannot be linked to 32-bit test program.")
 
         exe_name = self.getBuildArtifact("a.out")
         self.buildDriver(self.source, exe_name)
@@ -33,7 +31,6 @@ def test_sb_api_directory(self):
 
     def sanity_check_executable(self, exe_name):
         """Sanity check executable compiled from the auto-generated program."""
-        exe_name = self.getBuildArtifact("a.out")
         exe = self.getBuildArtifact(exe_name)
         self.runCmd("file %s" % exe, CURRENT_EXECUTABLE_SET)
 
@@ -46,10 +43,6 @@ def sanity_check_executable(self, exe_name):
         if self.TraceOn():
             print("Set environment to: ", env_cmd)
         self.runCmd(env_cmd)
-        self.addTearDownHook(
-            lambda: self.dbg.HandleCommand(
-                "settings remove target.env-vars %s" %
-                self.dylibPath))
 
         lldbutil.run_break_set_by_file_and_line(
             self, self.source, self.line_to_break, num_expected_locations=-1)
diff --git a/lldb/test/API/commands/expression/namespace-alias/TestInlineNamespaceAlias.py b/lldb/test/API/commands/expression/namespace-alias/TestInlineNamespaceAlias.py
index 0c3f3d457f254..5220a6e0099cd 100644
--- a/lldb/test/API/commands/expression/namespace-alias/TestInlineNamespaceAlias.py
+++ b/lldb/test/API/commands/expression/namespace-alias/TestInlineNamespaceAlias.py
@@ -11,7 +11,8 @@
 from lldbsuite.test import lldbutil
 
 class TestInlineNamespace(TestBase):
-    def do_test(self, params):
+    @skipIf(compiler="clang", compiler_version=['<', '16.0'])
+    def test(self):
         self.build()
 
         lldbutil.run_to_source_breakpoint(self,
@@ -27,11 +28,3 @@ def do_test(self, params):
         self.expect_expr("E::D::a", result_type="int", result_value="-1")
         self.expect_expr("F::a", result_type="int", result_value="-1")
         self.expect_expr("G::a", result_type="int", result_value="-1")
-
-    @skipIf(debug_info=no_match(["dsym"]))
-    def test_dsym(self):
-        self.do_test({})
-
-    @skipIf(debug_info="dsym")
-    def test_dwarf(self):
-        self.do_test(dict(CFLAGS_EXTRAS="-gdwarf-5 -gpubnames"))
diff --git a/lldb/test/API/lang/cpp/hide_global_ranges_vars/Makefile b/lldb/test/API/commands/frame/var/direct-ivar/cpp/Makefile
similarity index 60%
rename from lldb/test/API/lang/cpp/hide_global_ranges_vars/Makefile
rename to lldb/test/API/commands/frame/var/direct-ivar/cpp/Makefile
index 4f79c0a900c3a..3d0b98f13f3d7 100644
--- a/lldb/test/API/lang/cpp/hide_global_ranges_vars/Makefile
+++ b/lldb/test/API/commands/frame/var/direct-ivar/cpp/Makefile
@@ -1,4 +1,2 @@
 CXX_SOURCES := main.cpp
-CXXFLAGS_EXTRAS := -std=c++20
-
 include Makefile.rules
diff --git a/lldb/test/API/commands/frame/var/direct-ivar/cpp/TestFrameVarDirectIvarCpp.py b/lldb/test/API/commands/frame/var/direct-ivar/cpp/TestFrameVarDirectIvarCpp.py
new file mode 100644
index 0000000000000..f483899a75509
--- /dev/null
+++ b/lldb/test/API/commands/frame/var/direct-ivar/cpp/TestFrameVarDirectIvarCpp.py
@@ -0,0 +1,11 @@
+import lldb
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test.decorators import *
+from lldbsuite.test import lldbutil
+
+
+class TestCase(TestBase):
+    def test_cpp_this(self):
+        self.build()
+        lldbutil.run_to_source_breakpoint(self, "// check this", lldb.SBFileSpec("main.cpp"))
+        self.expect("frame variable m_field", startstr="(int) m_field = 30")
diff --git a/lldb/test/API/commands/frame/var/direct-ivar/cpp/main.cpp b/lldb/test/API/commands/frame/var/direct-ivar/cpp/main.cpp
new file mode 100644
index 0000000000000..eadb02be38368
--- /dev/null
+++ b/lldb/test/API/commands/frame/var/direct-ivar/cpp/main.cpp
@@ -0,0 +1,12 @@
+struct Structure {
+  int m_field;
+  void fun() {
+    // check this
+  }
+};
+
+int main() {
+  Structure s;
+  s.m_field = 30;
+  s.fun();
+}
diff --git a/lldb/test/API/commands/frame/var/direct-ivar/objc/Makefile b/lldb/test/API/commands/frame/var/direct-ivar/objc/Makefile
new file mode 100644
index 0000000000000..c07887aa2f1df
--- /dev/null
+++ b/lldb/test/API/commands/frame/var/direct-ivar/objc/Makefile
@@ -0,0 +1,4 @@
+OBJC_SOURCES := main.m
+CFLAGS_EXTRAS := -fblocks -fobjc-arc
+LD_EXTRAS := -lobjc
+include Makefile.rules
diff --git a/lldb/test/API/commands/frame/var/direct-ivar/objc/TestFrameVarDirectIvarObjC.py b/lldb/test/API/commands/frame/var/direct-ivar/objc/TestFrameVarDirectIvarObjC.py
new file mode 100644
index 0000000000000..c93dc9ea1d9a0
--- /dev/null
+++ b/lldb/test/API/commands/frame/var/direct-ivar/objc/TestFrameVarDirectIvarObjC.py
@@ -0,0 +1,20 @@
+import lldb
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test.decorators import *
+from lldbsuite.test import lldbutil
+
+
+class TestCase(TestBase):
+    @skipUnlessDarwin
+    def test_objc_self(self):
+        self.build()
+        lldbutil.run_to_source_breakpoint(self, "// check self", lldb.SBFileSpec("main.m"))
+        self.expect("frame variable _ivar", startstr="(int) _ivar = 30")
+
+    @skipUnlessDarwin
+    def test_objc_self_capture_idiom(self):
+        self.build()
+        lldbutil.run_to_source_breakpoint(self, "// check idiomatic self", lldb.SBFileSpec("main.m"))
+        self.expect("frame variable weakSelf", startstr="(Classic *) weakSelf = 0x")
+        self.expect("frame variable self", startstr="(Classic *) self = 0x")
+        self.expect("frame variable _ivar", startstr="(int) _ivar = 30")
diff --git a/lldb/test/API/commands/frame/var/direct-ivar/objc/main.m b/lldb/test/API/commands/frame/var/direct-ivar/objc/main.m
new file mode 100644
index 0000000000000..156622ffb1e63
--- /dev/null
+++ b/lldb/test/API/commands/frame/var/direct-ivar/objc/main.m
@@ -0,0 +1,31 @@
+#include <objc/NSObject.h>
+
+@interface Classic : NSObject {
+@public
+  int _ivar;
+}
+@end
+
+@implementation Classic
+- (void)fun {
+  // check self
+}
+
+- (void)run {
+  __weak Classic *weakSelf = self;
+  ^{
+    Classic *self = weakSelf;
+    // check idiomatic self
+
+    // Use `self` to extend its lifetime (for lldb to inspect the variable).
+    [self copy];
+  }();
+}
+@end
+
+int main() {
+  Classic *c = [Classic new];
+  c->_ivar = 30;
+  [c fun];
+  [c run];
+}
diff --git a/lldb/test/API/commands/session/save/TestSessionSave.py b/lldb/test/API/commands/session/save/TestSessionSave.py
index 5474cb354dae0..e1b60af05dd26 100644
--- a/lldb/test/API/commands/session/save/TestSessionSave.py
+++ b/lldb/test/API/commands/session/save/TestSessionSave.py
@@ -30,7 +30,8 @@ def test_session_save(self):
         settings = [
           'settings set interpreter.echo-commands true',
           'settings set interpreter.echo-comment-commands true',
-          'settings set interpreter.stop-command-source-on-error false'
+          'settings set interpreter.stop-command-source-on-error false',
+          'settings set interpreter.open-transcript-in-editor false',
         ]
 
         for setting in settings:
@@ -102,6 +103,7 @@ def test_session_save_on_quit(self):
           'settings set interpreter.stop-command-source-on-error false',
           'settings set interpreter.save-session-on-quit true',
           'settings set interpreter.save-session-directory ' + td.name,
+          'settings set interpreter.open-transcript-in-editor false',
         ]
 
         for setting in settings:
diff --git a/lldb/test/API/commands/watchpoints/watchpoint_set_command/TestWatchLocationWithWatchSet.py b/lldb/test/API/commands/watchpoints/watchpoint_set_command/TestWatchLocationWithWatchSet.py
index aceb8f2bc4385..97e43c35127de 100644
--- a/lldb/test/API/commands/watchpoints/watchpoint_set_command/TestWatchLocationWithWatchSet.py
+++ b/lldb/test/API/commands/watchpoints/watchpoint_set_command/TestWatchLocationWithWatchSet.py
@@ -46,7 +46,7 @@ def test_watchlocation_using_watchpoint_set(self):
         self.setTearDownCleanup()
 
         exe = self.getBuildArtifact("a.out")
-        self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)
+        target = self.dbg.CreateTarget(exe)
 
         # Add a breakpoint to set a watchpoint when stopped on the breakpoint.
         lldbutil.run_break_set_by_file_and_line(
@@ -81,6 +81,12 @@ def test_watchlocation_using_watchpoint_set(self):
         self.expect("watchpoint list -v",
                     substrs=['hit_count = 0'])
 
+        # Check the underlying SBWatchpoint.
+        watchpoint = target.GetWatchpointAtIndex(0)
+        self.assertEqual(watchpoint.GetWatchSize(), 1)
+        self.assertEqual(watchpoint.GetHitCount(), 0)
+        self.assertEqual(watchpoint.GetWatchSpec(), "g_char_ptr + 7")
+
         self.runCmd("process continue")
 
         # We should be stopped again due to the watchpoint (write type), but
diff --git a/lldb/test/API/functionalities/alias/TestBtAliasRepeat.py b/lldb/test/API/functionalities/alias/TestBtAliasRepeat.py
new file mode 100644
index 0000000000000..42b5accddc0ba
--- /dev/null
+++ b/lldb/test/API/functionalities/alias/TestBtAliasRepeat.py
@@ -0,0 +1,19 @@
+import lldb
+from lldbsuite.test.lldbtest import TestBase
+from lldbsuite.test import lldbutil
+
+
+class TestCase(TestBase):
+    def test(self):
+        self.build()
+        lldbutil.run_to_source_breakpoint(self, "return", lldb.SBFileSpec("main.c"))
+
+        # Expect "frame #0" but not "frame #1".
+        self.expect("bt 1", inHistory=True, patterns=["frame #0", "^(?!.*frame #1)"])
+
+        # Run an empty command to run the repeat command for `bt`.
+        # The repeat command for `bt N` lists the subsequent N frames.
+        #
+        # In this case, after printing the frame 0 with `bt 1`, the repeat
+        # command will print "frame #1" (and won't print "frame #0").
+        self.expect("", patterns=["^(?!.*frame #0)", "frame #1"])
diff --git a/lldb/test/API/functionalities/completion/TestCompletion.py b/lldb/test/API/functionalities/completion/TestCompletion.py
index f2d3fb704900a..baaf9f57162ce 100644
--- a/lldb/test/API/functionalities/completion/TestCompletion.py
+++ b/lldb/test/API/functionalities/completion/TestCompletion.py
@@ -36,42 +36,55 @@ def test_de(self):
 
     def test_frame_variable(self):
         self.build()
-        self.main_source = "main.cpp"
-        self.main_source_spec = lldb.SBFileSpec(self.main_source)
 
-        (target, process, thread, bkpt) = lldbutil.run_to_source_breakpoint(self,
-                                          '// Break here', self.main_source_spec)
+        _, process, _, _ = lldbutil.run_to_source_breakpoint(
+            self, '// Break here', lldb.SBFileSpec("main.cpp"))
         self.assertState(process.GetState(), lldb.eStateStopped)
 
         # Since CommandInterpreter has been corrected to update the current execution
         # context at the beginning of HandleCompletion, we're here explicitly testing
         # the scenario where "frame var" is completed without any preceding commands.
+        self.do_test_variable_completion('frame variable')
 
-        self.complete_from_to('frame variable fo',
-                              'frame variable fooo')
-        self.complete_from_to('frame variable fooo.',
-                              'frame variable fooo.')
-        self.complete_from_to('frame variable fooo.dd',
-                              'frame variable fooo.dd')
-
-        self.complete_from_to('frame variable ptr_fooo->',
-                              'frame variable ptr_fooo->')
-        self.complete_from_to('frame variable ptr_fooo->dd',
-                              'frame variable ptr_fooo->dd')
-
-        self.complete_from_to('frame variable cont',
-                              'frame variable container')
-        self.complete_from_to('frame variable container.',
-                              'frame variable container.MemberVar')
-        self.complete_from_to('frame variable container.Mem',
-                              'frame variable container.MemberVar')
-
-        self.complete_from_to('frame variable ptr_cont',
-                              'frame variable ptr_container')
-        self.complete_from_to('frame variable ptr_container->',
-                              'frame variable ptr_container->MemberVar')
-        self.complete_from_to('frame variable ptr_container->Mem',
-                              'frame variable ptr_container->MemberVar')
+    def test_dwim_print(self):
+        self.build()
+
+        _, process, _, _ = lldbutil.run_to_source_breakpoint(
+            self, '// Break here', lldb.SBFileSpec("main.cpp"))
+        self.assertState(process.GetState(), lldb.eStateStopped)
+
+        # Since CommandInterpreter has been corrected to update the current execution
+        # context at the beginning of HandleCompletion, we're here explicitly testing
+        # the scenario where "frame var" is completed without any preceding commands.
+        self.do_test_variable_completion('dwim-print')
+
+
+    def do_test_variable_completion(self, command):
+        self.complete_from_to(f'{command} fo',
+                              f'{command} fooo')
+        self.complete_from_to(f'{command} fooo.',
+                              f'{command} fooo.')
+        self.complete_from_to(f'{command} fooo.dd',
+                              f'{command} fooo.dd')
+
+        self.complete_from_to(f'{command} ptr_fooo->',
+                              f'{command} ptr_fooo->')
+        self.complete_from_to(f'{command} ptr_fooo->dd',
+                              f'{command} ptr_fooo->dd')
+
+        self.complete_from_to(f'{command} cont',
+                              f'{command} container')
+        self.complete_from_to(f'{command} container.',
+                              f'{command} container.MemberVar')
+        self.complete_from_to(f'{command} container.Mem',
+                              f'{command} container.MemberVar')
+
+        self.complete_from_to(f'{command} ptr_cont',
+                              f'{command} ptr_container')
+        self.complete_from_to(f'{command} ptr_container->',
+                              f'{command} ptr_container->MemberVar')
+        self.complete_from_to(f'{command} ptr_container->Mem',
+                              f'{command} ptr_container->MemberVar')
 
     def test_process_attach_dash_dash_con(self):
         """Test that 'process attach --con' completes to 'process attach --continue '."""
diff --git a/lldb/test/API/functionalities/data-formatter/data-formatter-cpp/TestDataFormatterCpp.py b/lldb/test/API/functionalities/data-formatter/data-formatter-cpp/TestDataFormatterCpp.py
index 2085025629bec..e26ef1f61f9f3 100644
--- a/lldb/test/API/functionalities/data-formatter/data-formatter-cpp/TestDataFormatterCpp.py
+++ b/lldb/test/API/functionalities/data-formatter/data-formatter-cpp/TestDataFormatterCpp.py
@@ -11,7 +11,6 @@
 
 
 class CppDataFormatterTestCase(TestBase):
-
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
@@ -286,15 +285,23 @@ def cleanup():
             substrs=['(int) iAmInt = 0x00000001'])
         self.expect("frame variable iAmInt", substrs=['(int) iAmInt = 1'])
 
-        # Check that pointer to members are correctly formatted
+    @skipIfWindows
+    def test_mem_func_ptr_formats(self):
+        self.build()
+
+        lldbutil.run_to_source_breakpoint(self, "Break in has_local_mem_func_pointers", lldb.SBFileSpec("main.cpp"))
+
         self.expect(
             "frame variable member_ptr",
-            substrs=['member_ptr = 0x'])
+            patterns=['member_ptr = 0x[0-9a-z]+'])
         self.expect(
             "frame variable member_func_ptr",
-            substrs=['member_func_ptr = 0x',
-                     '(a.out`IUseCharStar::member_func(int) at main.cpp:61)'])
+            patterns=['member_func_ptr = 0x[0-9a-z]+'],
+            substrs=['(a.out`IUseCharStar::member_func(int) at main.cpp:61)'])
         self.expect(
             "frame variable ref_to_member_func_ptr",
-            substrs=['ref_to_member_func_ptr = 0x',
-                     '(a.out`IUseCharStar::member_func(int) at main.cpp:61)'])
+            patterns=['ref_to_member_func_ptr = 0x[0-9a-z]+'],
+            substrs=['(a.out`IUseCharStar::member_func(int) at main.cpp:61)'])
+        self.expect(
+            "frame variable virt_member_func_ptr",
+            patterns=['virt_member_func_ptr = 0x[0-9a-z]+$'])
diff --git a/lldb/test/API/functionalities/data-formatter/data-formatter-cpp/main.cpp b/lldb/test/API/functionalities/data-formatter/data-formatter-cpp/main.cpp
index f1cf507e47f87..1072fc262aa51 100644
--- a/lldb/test/API/functionalities/data-formatter/data-formatter-cpp/main.cpp
+++ b/lldb/test/API/functionalities/data-formatter/data-formatter-cpp/main.cpp
@@ -59,8 +59,21 @@ struct IUseCharStar
 	IUseCharStar() : pointer("Hello world") {}
 
         char const *member_func(int) { return ""; }
+        virtual void virt_member_func() {}
 };
 
+void has_local_mem_func_pointers() {
+  const char *IUseCharStar::*member_ptr = &IUseCharStar::pointer;
+  const char *(IUseCharStar::*member_func_ptr)(int) =
+      &IUseCharStar::member_func;
+  auto &ref_to_member_func_ptr = member_func_ptr;
+  
+  void (IUseCharStar::*virt_member_func_ptr)() =
+      &IUseCharStar::virt_member_func;
+
+  ::puts("Break in has_local_mem_func_pointers");
+}
+
 int main (int argc, const char * argv[])
 {
     
@@ -109,10 +122,7 @@ int main (int argc, const char * argv[])
     
     i_am_cooler the_coolest_guy(1,2,3.14,6.28,'E','G');
 
-    const char *IUseCharStar::*member_ptr = &IUseCharStar::pointer;
-    const char *(IUseCharStar::*member_func_ptr)(int) =
-        &IUseCharStar::member_func;
-    auto &ref_to_member_func_ptr = member_func_ptr;
+    has_local_mem_func_pointers();
 
     return 0; // Set break point at this line.
 }
diff --git a/lldb/test/API/lang/cpp/hide_global_ranges_vars/TestHideGlobalRangesVars.py b/lldb/test/API/lang/cpp/hide_global_ranges_vars/TestHideGlobalRangesVars.py
deleted file mode 100644
index f7b98a55341df..0000000000000
--- a/lldb/test/API/lang/cpp/hide_global_ranges_vars/TestHideGlobalRangesVars.py
+++ /dev/null
@@ -1,27 +0,0 @@
-"""Test that frame var and target var hide
-the global function objects in the libc++
-ranges implementation"""
-
-from lldbsuite.test.decorators import *
-from lldbsuite.test.lldbtest import *
-from lldbsuite.test import lldbutil
-
-class HideGlobalRangesVarsTestCase(TestBase):
-
-    @add_test_categories(["libc++"])
-    @skipIf(compiler=no_match("clang"))
-    @skipIf(compiler="clang", compiler_version=['<', '16.0'])
-    def test(self):
-        self.build()
-
-        lldbutil.run_to_source_breakpoint(self, "return", lldb.SBFileSpec('main.cpp', False))
-
-        self.expect("frame variable --show-globals",
-                    substrs=["::ranges::views::__cpo",
-                             "::ranges::__cpo"],
-                    matching=False)
-
-        self.expect("target variable",
-                    substrs=["::ranges::views::__cpo",
-                             "::ranges::__cpo"],
-                    matching=False)
diff --git a/lldb/test/API/lang/cpp/hide_global_ranges_vars/main.cpp b/lldb/test/API/lang/cpp/hide_global_ranges_vars/main.cpp
deleted file mode 100644
index 96a05f846396a..0000000000000
--- a/lldb/test/API/lang/cpp/hide_global_ranges_vars/main.cpp
+++ /dev/null
@@ -1,9 +0,0 @@
-#include <ranges>
-
-int main(int argc, char const *argv[]) {
-  int arr[3] = {1, 2, 3};
-  auto arr_view = std::ranges::views::all(arr);
-  auto arr_max = std::ranges::max_element(arr);
-
-  return 0;
-}
diff --git a/lldb/test/API/macosx/symbols/Makefile b/lldb/test/API/macosx/symbols/Makefile
new file mode 100644
index 0000000000000..aff841c364299
--- /dev/null
+++ b/lldb/test/API/macosx/symbols/Makefile
@@ -0,0 +1,8 @@
+C_SOURCES := main.c
+
+all: stripped.out
+
+stripped.out : a.out
+	strip a.out -o stripped.out
+
+include Makefile.rules
diff --git a/lldb/test/API/macosx/symbols/TestSymbolFileJSON.py b/lldb/test/API/macosx/symbols/TestSymbolFileJSON.py
new file mode 100644
index 0000000000000..3b25a3161149b
--- /dev/null
+++ b/lldb/test/API/macosx/symbols/TestSymbolFileJSON.py
@@ -0,0 +1,106 @@
+""" Testing symbol loading via JSON file. """
+import lldb
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+import json
+
+
+class TargetSymbolsFileJSON(TestBase):
+
+    def setUp(self):
+        TestBase.setUp(self)
+        self.source = 'main.c'
+
+    @no_debug_info_test
+    @skipIfWindows # No 'strip'
+    def test_symbol_file_json_address(self):
+        """Test that 'target symbols add' can load the symbols from a JSON file using file addresses."""
+
+        self.build()
+        stripped = self.getBuildArtifact("stripped.out")
+        unstripped = self.getBuildArtifact("a.out")
+
+        # Create a JSON symbol file from the unstripped target.
+        unstripped_target = self.dbg.CreateTarget(unstripped)
+        self.assertTrue(unstripped_target, VALID_TARGET)
+
+        unstripped_module = unstripped_target.GetModuleAtIndex(0)
+        main_symbol = unstripped_module.FindSymbol("main")
+        foo_symbol = unstripped_module.FindSymbol("foo")
+
+        data = {
+            "triple": unstripped_module.GetTriple(),
+            "uuid": unstripped_module.GetUUIDString(),
+            "symbols": list()
+        }
+        data['symbols'].append({
+            "name": "main",
+            "type": "code",
+            "size": main_symbol.GetSize(),
+            "address": main_symbol.addr.GetFileAddress(),
+        })
+        data['symbols'].append({
+            "name": "foo",
+            "type": "code",
+            "size": foo_symbol.GetSize(),
+            "address": foo_symbol.addr.GetFileAddress(),
+        })
+        data['symbols'].append({
+            "name": "bar",
+            "type": "code",
+            "size": 0,
+            "value": 0xFF,
+        })
+
+        json_object = json.dumps(data, indent=4)
+        json_symbol_file = self.getBuildArtifact("a.json")
+        with open(json_symbol_file, "w") as outfile:
+            outfile.write(json_object)
+
+        # Create a stripped target.
+        stripped_target = self.dbg.CreateTarget(stripped)
+        self.assertTrue(stripped_target, VALID_TARGET)
+
+        # Ensure there's no symbol for main and foo.
+        stripped_module = stripped_target.GetModuleAtIndex(0)
+        self.assertFalse(stripped_module.FindSymbol("main").IsValid())
+        self.assertFalse(stripped_module.FindSymbol("foo").IsValid())
+        self.assertFalse(stripped_module.FindSymbol("bar").IsValid())
+
+        main_bp = stripped_target.BreakpointCreateByName(
+            "main", "stripped.out")
+        self.assertTrue(main_bp, VALID_BREAKPOINT)
+        self.assertEqual(main_bp.num_locations, 0)
+
+        # Load the JSON symbol file.
+        self.runCmd("target symbols add -s %s %s" %
+                    (stripped, self.getBuildArtifact("a.json")))
+
+        stripped_main_symbol = stripped_module.FindSymbol("main")
+        stripped_foo_symbol = stripped_module.FindSymbol("foo")
+        stripped_bar_symbol = stripped_module.FindSymbol("bar")
+
+        # Ensure main and foo are available now.
+        self.assertTrue(stripped_main_symbol.IsValid())
+        self.assertTrue(stripped_foo_symbol.IsValid())
+        self.assertTrue(stripped_bar_symbol.IsValid())
+        self.assertEqual(main_bp.num_locations, 1)
+
+        # Ensure the file address matches between the stripped and unstripped target.
+        self.assertEqual(stripped_main_symbol.addr.GetFileAddress(),
+                         main_symbol.addr.GetFileAddress())
+        self.assertEqual(stripped_main_symbol.addr.GetFileAddress(),
+                         main_symbol.addr.GetFileAddress())
+
+        # Ensure the size matches.
+        self.assertEqual(stripped_main_symbol.GetSize(), main_symbol.GetSize())
+        self.assertEqual(stripped_main_symbol.GetSize(), main_symbol.GetSize())
+
+        # Ensure the type matches.
+        self.assertEqual(stripped_main_symbol.GetType(), main_symbol.GetType())
+        self.assertEqual(stripped_main_symbol.GetType(), main_symbol.GetType())
+
+        # Ensure the bar symbol has a fixed value of 10.
+        self.assertEqual(stripped_bar_symbol.GetValue(), 0xFF);
diff --git a/lldb/test/API/macosx/symbols/main.c b/lldb/test/API/macosx/symbols/main.c
new file mode 100644
index 0000000000000..88653f66d7153
--- /dev/null
+++ b/lldb/test/API/macosx/symbols/main.c
@@ -0,0 +1,2 @@
+int foo() { return 1; }
+int main() { return foo(); }
diff --git a/lldb/test/API/python_api/debugger/TestDebuggerAPI.py b/lldb/test/API/python_api/debugger/TestDebuggerAPI.py
index 6bc6b3bcbba81..dcc8e24774687 100644
--- a/lldb/test/API/python_api/debugger/TestDebuggerAPI.py
+++ b/lldb/test/API/python_api/debugger/TestDebuggerAPI.py
@@ -30,7 +30,7 @@ def test_debugger_api_boundary_condition(self):
         self.dbg.SetPrompt(None)
         self.dbg.SetCurrentPlatform(None)
         self.dbg.SetCurrentPlatformSDKRoot(None)
-        
+
         fresh_dbg = lldb.SBDebugger()
         self.assertEquals(len(fresh_dbg), 0)
 
@@ -146,3 +146,16 @@ def test_CreateTarget_arch(self):
         self.assertEqual(platform2.GetName(), expected_platform)
         self.assertTrue(platform2.GetWorkingDirectory().endswith("bar"),
                 platform2.GetWorkingDirectory())
+
+    def test_SetDestroyCallback(self):
+        destroy_dbg_id = None
+        def foo(dbg_id):
+            # Need nonlocal to modify closure variable.
+            nonlocal destroy_dbg_id
+            destroy_dbg_id = dbg_id
+
+        self.dbg.SetDestroyCallback(foo)
+
+        original_dbg_id = self.dbg.GetID()
+        self.dbg.Destroy(self.dbg)
+        self.assertEqual(destroy_dbg_id,  original_dbg_id)
diff --git a/lldb/test/API/python_api/value/addr_of_void_star/Makefile b/lldb/test/API/python_api/value/addr_of_void_star/Makefile
new file mode 100644
index 0000000000000..10495940055b6
--- /dev/null
+++ b/lldb/test/API/python_api/value/addr_of_void_star/Makefile
@@ -0,0 +1,3 @@
+C_SOURCES := main.c
+
+include Makefile.rules
diff --git a/lldb/test/API/python_api/value/addr_of_void_star/TestValueAPIAddressOfVoidStar.py b/lldb/test/API/python_api/value/addr_of_void_star/TestValueAPIAddressOfVoidStar.py
new file mode 100644
index 0000000000000..701b569e63576
--- /dev/null
+++ b/lldb/test/API/python_api/value/addr_of_void_star/TestValueAPIAddressOfVoidStar.py
@@ -0,0 +1,38 @@
+import lldb
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+class ValueAPIVoidStarTestCase(TestBase):
+
+    def test(self):
+        self.build()
+        
+        target, process, thread, _ = lldbutil.run_to_source_breakpoint(self,
+                                                                       "Break at this line",
+                                                                       lldb.SBFileSpec("main.c"))
+        frame = thread.GetFrameAtIndex(0)
+
+        # Verify that the expression result for a void * behaves the same way as the
+        # variable value.
+
+        var_val = frame.FindVariable("void_ptr")
+        self.assertSuccess(var_val.GetError(), "Var version made correctly")
+        
+        expr_val = frame.EvaluateExpression("void_ptr")
+        self.assertSuccess(expr_val.GetError(), "Expr version succeeds")
+
+        # The pointer values should be equal:
+        self.assertEqual(var_val.unsigned, expr_val.unsigned, "Values are equal")
+
+        # Both versions should have valid AddressOf, and they should be the same.
+
+        val_addr_of = var_val.AddressOf()
+        self.assertNotEqual(val_addr_of, lldb.LLDB_INVALID_ADDRESS, "Var addr of right")
+
+        expr_addr_of = expr_val.AddressOf()
+        self.assertNotEqual(expr_addr_of, lldb.LLDB_INVALID_ADDRESS, "Expr addr of right")
+
+        # The AddressOf values should also be equal.
+        self.assertEqual(expr_addr_of.unsigned, val_addr_of.unsigned, "Addr of equal")
+
diff --git a/lldb/test/API/python_api/value/addr_of_void_star/main.c b/lldb/test/API/python_api/value/addr_of_void_star/main.c
new file mode 100644
index 0000000000000..e60220a800f71
--- /dev/null
+++ b/lldb/test/API/python_api/value/addr_of_void_star/main.c
@@ -0,0 +1,6 @@
+int main (int argc, char const *argv[]) {
+  char *char_ptr = "Some pointer here";
+  void *void_ptr = &char_ptr;
+  
+  return 0; // Break at this line
+}
diff --git a/lldb/test/API/python_api/was_interrupted/Makefile b/lldb/test/API/python_api/was_interrupted/Makefile
new file mode 100644
index 0000000000000..695335e068c0c
--- /dev/null
+++ b/lldb/test/API/python_api/was_interrupted/Makefile
@@ -0,0 +1,4 @@
+C_SOURCES := main.c
+CFLAGS_EXTRAS := -std=c99
+
+include Makefile.rules
diff --git a/lldb/test/API/python_api/was_interrupted/TestDebuggerInterruption.py b/lldb/test/API/python_api/was_interrupted/TestDebuggerInterruption.py
new file mode 100644
index 0000000000000..3c519a65e613b
--- /dev/null
+++ b/lldb/test/API/python_api/was_interrupted/TestDebuggerInterruption.py
@@ -0,0 +1,330 @@
+"""
+Test SBDebugger.InterruptRequested and SBCommandInterpreter.WasInterrupted.
+"""
+
+import lldb
+import lldbsuite.test.lldbutil as lldbutil
+from lldbsuite.test.lldbtest import *
+import threading
+import os
+
+class TestDebuggerInterruption(TestBase):
+    """This test runs a command that starts up, rendevous with the test thread
+       using threading barriers, then checks whether it has been interrupted.
+
+    The command's first argument is either 'interp' or 'debugger', to test
+    InterruptRequested and WasInterrupted respectively.
+
+    The command has two modes, interrupt and check, the former is the one that
+    waits for an interrupt.  Then latter just returns whether an interrupt was
+    requested.  We use the latter to make sure we took down the flag correctly."""
+
+    NO_DEBUG_INFO_TESTCASE = True
+
+    class CommandRunner(threading.Thread):
+        """This class is for running a command, and for making a thread to run the command on.
+           It gets passed the test it is working on behalf of, and most of the important
+           objects come from the test. """
+        def __init__(self, test):
+            super().__init__()
+            self.test = test
+
+        def rendevous(self):
+            # We smuggle out barriers and event to the runner thread using thread local data:
+            import interruptible
+            interruptible.local_data = interruptible.BarrierContainer(self.test.before_interrupt_barrier,
+                                                                   self.test.after_interrupt_barrier,
+                                                                   self.test.event)
+            
+    class DirectCommandRunner(CommandRunner):
+        """"This version runs a single command using HandleCommand."""
+        def __init__(self, test, command):
+            super().__init__(test)
+            self.command = command
+        
+        def run(self):
+            self.rendevous()
+            result = self.test.dbg.GetCommandInterpreter().HandleCommand(self.command, self.test.result)
+            if self.test.result_barrier:
+                self.test.result_barrier.wait()
+
+    class CommandInterpreterRunner(CommandRunner):
+        """This version runs the CommandInterpreter and feeds the command to it."""
+        def __init__(self, test):
+            super().__init__(test)
+
+        def run(self):
+            self.rendevous()
+
+            test = self.test
+            
+            # We will use files for debugger input and output:
+            
+            # First write down the command:
+            with open(test.getBuildArtifact(test.in_filename), "w") as f:
+                f.write(f"{test.command}\n")
+
+            # Now set the debugger's stdout & stdin to our files, and run
+            # the CommandInterpreter:
+            with open(test.out_filename, "w") as outf, open(test.in_filename, "r") as inf:
+                outsbf = lldb.SBFile(outf.fileno(), "w", False)
+                orig_outf = test.dbg.GetOutputFile()
+                error = test.dbg.SetOutputFile(outsbf)
+                test.assertSuccess(error, "Could not set outfile")
+
+                insbf = lldb.SBFile(inf.fileno(), "r", False)
+                orig_inf = test.dbg.GetOutputFile()
+                error = test.dbg.SetInputFile(insbf)
+                test.assertSuccess(error, "Could not set infile")
+                      
+                options = lldb.SBCommandInterpreterRunOptions()
+                options.SetPrintResults(True)
+                options.SetEchoCommands(False)
+                
+                test.dbg.RunCommandInterpreter(True, False, options, 0, False, False)
+                test.dbg.GetOutputFile().Flush()
+
+                error = test.dbg.SetOutputFile(orig_outf)
+                test.assertSuccess(error, "Restored outfile")
+                test.dbg.SetInputFile(orig_inf)
+                test.assertSuccess(error, "Restored infile")
+            
+    def command_setup(self, args):
+        """Insert our command, if needed.  Then set up event and barriers if needed.
+           Then return the command to run."""
+
+        self.interp = self.dbg.GetCommandInterpreter()
+        self.command_name = "interruptible_command"
+        self.cmd_result = lldb.SBCommandReturnObject()
+
+        if not "check" in args:
+            self.event = threading.Event()
+            self.result_barrier = threading.Barrier(2, timeout=10)
+            self.before_interrupt_barrier = threading.Barrier(2, timeout=10)
+            self.after_interrupt_barrier = threading.Barrier(2, timeout=10)
+        else:
+            self.event = None
+            self.result_barrier = None
+            self.before_interrupt_barrier = None
+            self.after_interrupt_barrier = None
+
+        if not self.interp.UserCommandExists(self.command_name):
+            # Make the command we're going to use - it spins calling WasInterrupted:
+            cmd_filename = "interruptible"
+            cmd_filename = os.path.join(self.getSourceDir(), "interruptible.py")
+            self.runCmd(f"command script import {cmd_filename}")
+            cmd_string = f"command script add {self.command_name} --class interruptible.WelcomeCommand"
+            self.runCmd(cmd_string)
+
+        if len(args) == 0:
+            command = self.command_name
+        else:
+            command = self.command_name + " " + args
+        return command
+    
+    def run_single_command(self, command):
+        # Now start up a thread to run the command:
+        self.result.Clear()
+        self.runner = TestDebuggerInterruption.DirectCommandRunner(self, command)
+        self.runner.start()
+
+    def start_command_interp(self):
+        self.runner = TestDebuggerInterruption.CommandInterpreterRunner(self)
+        self.runner.start()
+        
+    def check_text(self, result_text, interrupted):
+        if interrupted: 
+            self.assertIn("Command was interrupted", result_text,
+                          "Got the interrupted message")
+        else:
+            self.assertIn("Command was not interrupted", result_text,
+                          "Got the not interrupted message")
+        
+    def gather_output(self):
+        # Now wait for the interrupt to interrupt the command:
+        self.runner.join(10.0)
+        finished = not self.runner.is_alive()
+        # Don't leave the runner thread stranded if the interrupt didn't work.
+        if not finished:
+            self.event.set()
+            self.runner.join(10.0)
+
+        self.assertTrue(finished, "We did finish the command")
+        
+    def check_result(self, interrupted = True):
+        self.gather_output()
+        self.check_text(self.result.GetOutput(), interrupted)
+
+    def check_result_output(self, interrupted = True):
+        self.gather_output()
+        buffer = ""
+        # Okay, now open the file for reading, and read.
+        with open(self.out_filename, "r") as f:
+            buffer = f.read()
+
+        self.assertNotEqual(len(buffer), 0, "No command data")
+        self.check_text(buffer, interrupted)
+        
+    def debugger_interrupt_test(self, use_interrupt_requested):
+        """Test that debugger interruption interrupts a command
+           running directly through HandleCommand.
+           If use_interrupt_requested is true, we'll check that API,
+           otherwise we'll check WasInterrupted.  They should both do
+           the same thing."""
+
+        if use_interrupt_requested:
+            command = self.command_setup("debugger")
+        else:
+            command = self.command_setup("interp")
+
+        self.result = lldb.SBCommandReturnObject()
+        self.run_single_command(command)
+
+        # Okay now wait till the command has gotten started to issue the interrupt:
+        self.before_interrupt_barrier.wait()
+        # I'm going to do it twice here to test that it works as a counter:
+        self.dbg.RequestInterrupt()
+        self.dbg.RequestInterrupt()
+        
+        def cleanup():
+            self.dbg.CancelInterruptRequest()
+        self.addTearDownHook(cleanup)
+        # Okay, now set both sides going:
+        self.after_interrupt_barrier.wait()
+        
+        # Check that the command was indeed interrupted.  First rendevous
+        # after the runner thread had a chance to execute the command:
+        self.result_barrier.wait()
+        self.assertTrue(self.result.Succeeded(), "Our command succeeded")
+        result_output = self.result.GetOutput()
+        self.check_result(True)
+
+        # Do it again to make sure that the counter is counting:
+        self.dbg.CancelInterruptRequest()
+        command = self.command_setup("debugger")
+        self.run_single_command(command)
+
+        # This time we won't even get to run the command, since HandleCommand
+        # checks for the interrupt state on entry, so we don't wait on the command
+        # barriers.
+        self.result_barrier.wait()
+        
+        # Again check that we were
+        self.assertFalse(self.result.Succeeded(), "Our command was not allowed to run")
+        error_output = self.result.GetError()
+        self.assertIn("... Interrupted", error_output, "Command was cut short by interrupt")
+        
+        # Now take down the flag, and make sure that we aren't interrupted:
+        self.dbg.CancelInterruptRequest()
+
+        # Now make sure that we really did take down the flag:
+        command = self.command_setup("debugger check")
+        self.run_single_command(command)
+        result_output = self.result.GetOutput()
+        self.check_result(False)
+
+    def test_debugger_interrupt_use_dbg(self):
+        self.debugger_interrupt_test(True)
+        
+    def test_debugger_interrupt_use_interp(self):
+        self.debugger_interrupt_test(False)
+        
+    def test_interp_doesnt_interrupt_debugger(self):
+        """Test that interpreter interruption does not interrupt a command
+           running directly through HandleCommand.
+           If use_interrupt_requested is true, we'll check that API,
+           otherwise we'll check WasInterrupted.  They should both do
+           the same thing."""
+
+        command = self.command_setup("debugger poll")
+
+        self.result = lldb.SBCommandReturnObject()
+        self.run_single_command(command)
+
+        # Now raise the debugger interrupt flag.  It will also interrupt the command:
+        self.before_interrupt_barrier.wait()
+        self.dbg.GetCommandInterpreter().InterruptCommand()
+        self.after_interrupt_barrier.wait()
+        
+        # Check that the command was indeed interrupted:
+        self.result_barrier.wait()
+        self.assertTrue(self.result.Succeeded(), "Our command succeeded")
+        result_output = self.result.GetOutput()
+        self.check_result(False)
+
+
+    def interruptible_command_test(self, use_interrupt_requested):
+        """Test that interpreter interruption interrupts a command
+           running in the RunCommandInterpreter loop.
+           If use_interrupt_requested is true, we'll check that API,
+           otherwise we'll check WasInterrupted.  They should both do
+           the same thing."""
+        
+        self.out_filename = self.getBuildArtifact("output")
+        self.in_filename = self.getBuildArtifact("input")
+        # We're going to overwrite the input file, but we
+        # don't want data accumulating in the output file.
+
+        if os.path.exists(self.out_filename):
+            os.unlink(self.out_filename)
+
+        # You should be able to use either check method interchangeably:
+        if use_interrupt_requested:
+            self.command = self.command_setup("debugger") + "\n"
+        else:
+            self.command = self.command_setup("interp") + "\n"
+
+        self.start_command_interp()
+
+        # Now give the interpreter a chance to run this command up
+        # to the first barrier
+        self.before_interrupt_barrier.wait()
+        # Then issue the interrupt:
+        sent_interrupt = self.dbg.GetCommandInterpreter().InterruptCommand()
+        self.assertTrue(sent_interrupt, "Did send command interrupt.")
+        # Now give the command a chance to finish:
+        self.after_interrupt_barrier.wait()
+        
+        self.check_result_output(True)
+
+        os.unlink(self.out_filename)
+
+        # Now send the check command, and make sure the flag is now down.
+        self.command = self.command_setup("interp check") + "\n"
+        self.start_command_interp()
+        
+        self.check_result_output(False)
+
+    def test_interruptible_command_check_dbg(self):
+        self.interruptible_command_test(True)
+        
+    def test_interruptible_command_check_interp(self):
+        self.interruptible_command_test(False)
+
+    def test_debugger_doesnt_interrupt_command(self):
+        """Test that debugger interruption doesn't interrupt a command
+           running in the RunCommandInterpreter loop."""
+        
+        self.out_filename = self.getBuildArtifact("output")
+        self.in_filename = self.getBuildArtifact("input")
+        # We're going to overwrite the input file, but we
+        # don't want data accumulating in the output file.
+
+        if os.path.exists(self.out_filename):
+            os.unlink(self.out_filename)
+
+        self.command = self.command_setup("interp poll") + "\n"
+
+        self.start_command_interp()
+
+        self.before_interrupt_barrier.wait()
+        self.dbg.RequestInterrupt()
+        def cleanup():
+            self.dbg.CancelInterruptRequest()
+        self.addTearDownHook(cleanup)
+        self.after_interrupt_barrier.wait()
+
+        self.check_result_output(False)
+
+        os.unlink(self.out_filename)
+
diff --git a/lldb/test/API/python_api/was_interrupted/interruptible.py b/lldb/test/API/python_api/was_interrupted/interruptible.py
new file mode 100644
index 0000000000000..8a6b4e96f3eee
--- /dev/null
+++ b/lldb/test/API/python_api/was_interrupted/interruptible.py
@@ -0,0 +1,90 @@
+import lldb
+import threading
+
+local_data = None
+
+class BarrierContainer(threading.local):
+    def __init__(self, before_interrupt_barrier, after_interrupt_barrier,  event):
+        self.event = event
+        self.before_interrupt_barrier = before_interrupt_barrier
+        self.after_interrupt_barrier = after_interrupt_barrier
+
+class WelcomeCommand(object):
+
+    def __init__(self, debugger, session_dict):
+        return
+
+    def get_short_help(self):
+        return "A command that waits for an interrupt before returning."
+
+    def check_was_interrupted(self, debugger, use_interpreter):
+        if use_interpreter:
+            self.was_interrupted = debugger.GetCommandInterpreter().WasInterrupted()
+        else:
+            self.was_interrupted = debugger.InterruptRequested()
+        if local_data.event:
+            self.was_canceled = local_data.event.is_set()
+
+    def __call__(self, debugger, args, exe_ctx, result):
+        """Command arguments:
+            {interp/debugger} - Whether to use SBCommandInterpreter::WasInterrupted
+                                of SBDebugger::InterruptRequested().
+            check - Don't do the rendevous, just check if an interrupt was requested.
+                    If check is not provided, we'll do the lock and then check.
+            poll  - Should we poll once after the rendevous or spin waiting for the
+                    interruption to happen.
+
+            For the interrupt cases, the command waits serially on the barriers
+            passed to it in local data, giving the test runner a chance to set the
+            interrupt.  Once the barriers are passed, it waits for the interrupt
+            or the event.
+            If it finds an interrupt, it returns "Command was interrupted". If it gets an
+            event before seeing the interrupt it returns "Command was not interrupted."
+            For the "poll" case, it waits on the rendevous, then checks once.
+            For the "check" case, it doesn't wait, but just returns whether there was
+            an interrupt in force or not."""
+            
+        if local_data == None:
+            result.SetError("local data was not set.")
+            result.SetStatus(lldb.eReturnStatusFailed)
+            return
+
+        use_interpreter = "interp" in args
+        if not use_interpreter:
+            if not "debugger" in args:
+                result.SetError("Must pass either 'interp' or 'debugger'")
+                result.SetStatus(lldb.eReturnStatusFailed)
+                return
+            
+        self.was_interrupted = False
+        self.was_canceled = False
+        
+        if "check" in args:
+            self.check_was_interrupted(debugger, use_interpreter)
+            if self.was_interrupted:
+                result.Print("Command was interrupted")
+            else:
+                result.Print("Command was not interrupted")
+        else:
+            # Wait here to rendevous in the test before it sets the interrupt.
+            local_data.before_interrupt_barrier.wait()
+            # Now the test will set the interrupt, and we can continue:
+            local_data.after_interrupt_barrier.wait()
+            
+            if "poll" in args:
+                self.check_was_interrupted(debugger, use_interpreter)
+            else:
+                while not self.was_interrupted and not self.was_canceled:
+                    self.check_was_interrupted(debugger, use_interpreter)
+
+            if self.was_interrupted:
+                result.Print("Command was interrupted")
+            else:
+                result.Print("Command was not interrupted")
+
+            if self.was_canceled:
+                result.Print("Command was canceled")
+        result.SetStatus(lldb.eReturnStatusSuccessFinishResult)
+        return True
+
+
diff --git a/lldb/test/API/python_api/was_interrupted/main.c b/lldb/test/API/python_api/was_interrupted/main.c
new file mode 100644
index 0000000000000..74b8a5995c461
--- /dev/null
+++ b/lldb/test/API/python_api/was_interrupted/main.c
@@ -0,0 +1,11 @@
+#include <stdio.h>
+
+int global_test_var = 10;
+
+int
+main()
+{
+  int test_var = 10;
+  printf ("Set a breakpoint here: %d.\n", test_var);
+  return global_test_var;
+}
diff --git a/lldb/test/Shell/Commands/command-source.test b/lldb/test/Shell/Commands/command-source.test
index fa389f2a12889..579e3989f6fd0 100644
--- a/lldb/test/Shell/Commands/command-source.test
+++ b/lldb/test/Shell/Commands/command-source.test
@@ -6,7 +6,7 @@
 # RUN: %lldb -x -b -o 'settings set interpreter.stop-command-source-on-error false' -o "command source %s" 2>&1 | FileCheck %s --check-prefix CONTINUE
 
 bogus
-p 10+1
+expression 10+1
 
 # CONTINUE: $0 = 11
 # STOP-NOT: $0 = 11
diff --git a/lldb/test/Shell/Commands/command-stop-hook-no-target.test b/lldb/test/Shell/Commands/command-stop-hook-no-target.test
index ee9ded164d745..cc5e71dff00cd 100644
--- a/lldb/test/Shell/Commands/command-stop-hook-no-target.test
+++ b/lldb/test/Shell/Commands/command-stop-hook-no-target.test
@@ -1,4 +1,4 @@
 # RUN: %clang_host -g %S/Inputs/main.c -o %t
-# RUN: %lldb -b -o 'target stop-hook add --name test --shlib test -o "p 95000 + 126"' -o 'file %t' -o 'b main' -o 'r' 2>&1 | FileCheck %s
+# RUN: %lldb -b -o 'target stop-hook add --name test --shlib test -o "expression 95000 + 126"' -o 'file %t' -o 'b main' -o 'r' 2>&1 | FileCheck %s
 # CHECK: Stop hook #1 added
 # CHECK-NOT: 95126
diff --git a/lldb/test/Shell/Diagnostics/Inputs/TestCopyLogs.in b/lldb/test/Shell/Diagnostics/Inputs/TestCopyLogs.in
new file mode 100644
index 0000000000000..ae9e9534adaff
--- /dev/null
+++ b/lldb/test/Shell/Diagnostics/Inputs/TestCopyLogs.in
@@ -0,0 +1 @@
+command alias logcommands log enable lldb commands
diff --git a/lldb/test/Shell/Diagnostics/TestCopyLogs.test b/lldb/test/Shell/Diagnostics/TestCopyLogs.test
new file mode 100644
index 0000000000000..4706a18bee0dd
--- /dev/null
+++ b/lldb/test/Shell/Diagnostics/TestCopyLogs.test
@@ -0,0 +1,7 @@
+# RUN: rm -rf %t
+# RUN: mkdir -p %t
+
+# RUN: %lldb -s %S/Inputs/TestCopyLogs.in -o 'logcommands -f %t/commands.log' -o 'diagnostics dump -d %t/diags'
+
+# RUN: cat %t/diags/commands.log | FileCheck %s
+# CHECK: Processing command: diagnostics dump
diff --git a/lldb/test/Shell/Driver/Inputs/dumpargs.c b/lldb/test/Shell/Driver/Inputs/dumpargs.c
new file mode 100644
index 0000000000000..5f0e6a4cb01d0
--- /dev/null
+++ b/lldb/test/Shell/Driver/Inputs/dumpargs.c
@@ -0,0 +1,8 @@
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  for (int i = 0; i < argc; i++) {
+    printf("argv[%d] = \"%s\"\n", i, argv[i]);
+  }
+  return 0;
+}
diff --git a/lldb/test/Shell/Driver/TestEmptyArgument.test b/lldb/test/Shell/Driver/TestEmptyArgument.test
new file mode 100644
index 0000000000000..ff327bdd38d01
--- /dev/null
+++ b/lldb/test/Shell/Driver/TestEmptyArgument.test
@@ -0,0 +1,7 @@
+# RUN: %clang_host %S/Inputs/dumpargs.c -o %t.out
+# RUN: %lldb -b -o "r" %t.out -- "one" "two" "" "three" | FileCheck %s
+
+# CHECK: argv[1] = "one"
+# CHECK: argv[2] = "two"
+# CHECK: argv[3] = ""
+# CHECK: argv[4] = "three"
diff --git a/lldb/test/Shell/Expr/nodefaultlib.cpp b/lldb/test/Shell/Expr/nodefaultlib.cpp
index fb53da7fcb08b..f97d8aad9a8db 100644
--- a/lldb/test/Shell/Expr/nodefaultlib.cpp
+++ b/lldb/test/Shell/Expr/nodefaultlib.cpp
@@ -7,10 +7,10 @@
 // XFAIL: system-netbsd || system-freebsd || system-darwin
 
 // RUN: %build %s --nodefaultlib -o %t
-// RUN: %lldb %t -o "b main" -o run -o "p call_me(5, 6)" -o exit \
+// RUN: %lldb %t -o "b main" -o run -o "expression call_me(5, 6)" -o exit \
 // RUN:   | FileCheck %s
 
-// CHECK: p call_me(5, 6)
+// CHECK: expression call_me(5, 6)
 // CHECK: (int) $0 = 30
 
 int call_me(int x, long y) { return x * y; }
diff --git a/lldb/test/Shell/ObjectFile/ELF/loongarch64-relocations.yaml b/lldb/test/Shell/ObjectFile/ELF/loongarch64-relocations.yaml
new file mode 100644
index 0000000000000..8fcc4d343de63
--- /dev/null
+++ b/lldb/test/Shell/ObjectFile/ELF/loongarch64-relocations.yaml
@@ -0,0 +1,49 @@
+# RUN: yaml2obj %s -o %t
+# RUN: lldb-test object-file -contents %t | FileCheck %s
+
+## Test that relocations are correctly applied to the .debug_info section on loongarch64.
+
+# CHECK:      Name: .debug_info
+# CHECK:      Data:  (
+## Before relocation:
+##                0000: 00000000 00000000 00000000 00000000 00000000
+## After relocation:
+# CHECK-NEXT:     0000: 34120000 88776655 44332211 8899AABB CCDDEEFF
+# CHECK-NEXT: )
+
+--- !ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_REL
+  Machine:         EM_LOONGARCH
+Sections:
+  - Name:            .debug_str
+    Type:            SHT_PROGBITS
+  - Name:            .debug_info
+    Type:            SHT_PROGBITS
+    Content:         0000000000000000000000000000000000000000
+  - Name:            .rela.debug_info
+    Type:            SHT_RELA
+    Info:            .debug_info
+    Relocations:
+      - Offset:          0x0000000000000000
+        Symbol:          .debug_str
+        Type:            R_LARCH_32
+        Addend:          0x1234
+      - Offset:          0x0000000000000004
+        Symbol:          .debug_str
+        Type:            R_LARCH_64
+        Addend:          0x1122334455667788
+      - Offset:          0x000000000000000C
+        Symbol:          .debug_str
+        Type:            R_LARCH_64
+        Addend:          0xFFEEDDCCBBAA9988
+Symbols:
+  - Name:            .debug_str
+    Type:            STT_SECTION
+    Section:         .debug_str
+  - Name:            .debug_info
+    Type:            STT_SECTION
+    Section:         .debug_info
+...
diff --git a/lldb/test/Shell/Register/x86-64-fp-read.test b/lldb/test/Shell/Register/x86-64-fp-read.test
index 39fdaf170f3f2..e23e333c204a5 100644
--- a/lldb/test/Shell/Register/x86-64-fp-read.test
+++ b/lldb/test/Shell/Register/x86-64-fp-read.test
@@ -9,9 +9,9 @@ process launch
 # CHECK: Process {{.*}} stopped
 
 # fdiv (%rbx) gets encoded into 2 bytes, int3 into 1 byte
-print (void*)($pc-3)
+expression (void*)($pc-3)
 # CHECK: (void *) $0 = [[FDIV:0x[0-9a-f]*]]
-print &zero
+expression &zero
 # CHECK: (uint32_t *) $1 = [[ZERO:0x[0-9a-f]*]]
 
 register read --all
@@ -32,9 +32,9 @@ register read --all
 # CHECK-DAG: st{{(mm)?}}7 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
 
 # legacy approach, superseded by fip/fdp registers
-print (void*)($fiseg*0x100000000 + $fioff)
+expression (void*)($fiseg*0x100000000 + $fioff)
 # CHECK: (void *) $2 = [[FDIV]]
-print (uint32_t*)($foseg * 0x100000000 + $fooff)
+expression (uint32_t*)($foseg * 0x100000000 + $fooff)
 # CHECK: (uint32_t *) $3 = [[ZERO]]
 
 process continue
diff --git a/lldb/test/Shell/Register/x86-db-read.test b/lldb/test/Shell/Register/x86-db-read.test
index f8380440c6270..85cdcd1cb8104 100644
--- a/lldb/test/Shell/Register/x86-db-read.test
+++ b/lldb/test/Shell/Register/x86-db-read.test
@@ -15,13 +15,13 @@ watchpoint set variable -w write g_32w
 watchpoint set variable -w read_write g_32rw
 # CHECK: Watchpoint created: Watchpoint 4: addr = 0x{{[0-9a-f]*}} size = 4 state = enabled type = rw
 
-print &g_8w
+expression &g_8w
 # CHECK: (uint8_t *) $0 = [[VAR8W:0x[0-9a-f]*]]
-print &g_16rw
+expression &g_16rw
 # CHECK: (uint16_t *) $1 = [[VAR16RW:0x[0-9a-f]*]]
-print &g_32w
+expression &g_32w
 # CHECK: (uint32_t *) $2 = [[VAR32W:0x[0-9a-f]*]]
-print &g_32rw
+expression &g_32rw
 # CHECK: (uint32_t *) $3 = [[VAR64RW:0x[0-9a-f]*]]
 
 register read --all
diff --git a/lldb/test/Shell/Register/x86-fp-read.test b/lldb/test/Shell/Register/x86-fp-read.test
index 566157b938367..f668b8cb78658 100644
--- a/lldb/test/Shell/Register/x86-fp-read.test
+++ b/lldb/test/Shell/Register/x86-fp-read.test
@@ -9,9 +9,9 @@ process launch
 # CHECK: Process {{.*}} stopped
 
 # fdiv (%rbx) gets encoded into 2 bytes, int3 into 1 byte
-print (void*)($pc-3)
+expression (void*)($pc-3)
 # CHECK: (void *) $0 = [[FDIV:0x[0-9a-f]*]]
-print &zero
+expression &zero
 # CHECK: (uint32_t *) $1 = [[ZERO:0x[0-9a-f]*]]
 
 register read --all
diff --git a/lldb/test/Shell/Settings/Inputs/DontStopCommandSource.in b/lldb/test/Shell/Settings/Inputs/DontStopCommandSource.in
index e33a1e329cad9..f3ef2a9e7016b 100644
--- a/lldb/test/Shell/Settings/Inputs/DontStopCommandSource.in
+++ b/lldb/test/Shell/Settings/Inputs/DontStopCommandSource.in
@@ -1,3 +1,3 @@
 settings set interpreter.stop-command-source-on-error false
 bogus
-print 123400000 + 56789
+expression 123400000 + 56789
diff --git a/lldb/test/Shell/Settings/Inputs/StopCommandSource.in b/lldb/test/Shell/Settings/Inputs/StopCommandSource.in
index 19974c0c444b1..1159554e3425f 100644
--- a/lldb/test/Shell/Settings/Inputs/StopCommandSource.in
+++ b/lldb/test/Shell/Settings/Inputs/StopCommandSource.in
@@ -1,3 +1,3 @@
 settings set interpreter.stop-command-source-on-error true
 bogus
-print 123400000 + 56789
+expression 123400000 + 56789
diff --git a/lldb/test/Shell/Settings/TestStopCommandSourceOnError.test b/lldb/test/Shell/Settings/TestStopCommandSourceOnError.test
index d734a0940a2d7..f99c9c204c19c 100644
--- a/lldb/test/Shell/Settings/TestStopCommandSourceOnError.test
+++ b/lldb/test/Shell/Settings/TestStopCommandSourceOnError.test
@@ -12,10 +12,10 @@
 # RUN: %lldb -b -o 'settings set interpreter.stop-command-source-on-error false' -s %S/Inputs/StopCommandSource.in | FileCheck %s --check-prefix CONTINUE
 
 # FIXME: Should continue
-# RUN: not %lldb -b -s %S/Inputs/DontStopCommandSource.in -o 'bogus' -o 'print 111100000 + 11111' | FileCheck %s --check-prefix STOP
+# RUN: not %lldb -b -s %S/Inputs/DontStopCommandSource.in -o 'bogus' -o 'expression 111100000 + 11111' | FileCheck %s --check-prefix STOP
 
 # FIXME: Should continue
-# RUN: not %lldb -b -o 'settings set interpreter.stop-command-source-on-error false' -o 'bogus' -o 'print 123400000 + 56789'  | FileCheck %s --check-prefix STOP
+# RUN: not %lldb -b -o 'settings set interpreter.stop-command-source-on-error false' -o 'bogus' -o 'expression 123400000 + 56789'  | FileCheck %s --check-prefix STOP
 
 # FIXME: Should continue
 # RUN: not %lldb -b -s %S/Inputs/DontStopCommandSource.in | FileCheck %s --check-prefix STOP
diff --git a/lldb/test/Shell/SymbolFile/DWARF/Inputs/inlined-file0-line0-col0.yaml b/lldb/test/Shell/SymbolFile/DWARF/Inputs/inlined-file0-line0-col0.yaml
new file mode 100644
index 0000000000000..0064b73c69698
--- /dev/null
+++ b/lldb/test/Shell/SymbolFile/DWARF/Inputs/inlined-file0-line0-col0.yaml
@@ -0,0 +1,699 @@
+--- !mach-o
+FileHeader:
+  magic:           0xFEEDFACF
+  cputype:         0x1000007
+  cpusubtype:      0x3
+  filetype:        0x1
+  ncmds:           3
+  sizeofcmds:      736
+  flags:           0x2000
+  reserved:        0x0
+LoadCommands:
+  - cmd:             LC_SEGMENT_64
+    cmdsize:         632
+    segname:         ''
+    vmaddr:          0
+    vmsize:          814
+    fileoff:         768
+    filesize:        814
+    maxprot:         7
+    initprot:        7
+    nsects:          7
+    flags:           0
+    Sections:
+      - sectname:        __text
+        segname:         __TEXT
+        addr:            0x0
+        size:            31
+        offset:          0x300
+        align:           4
+        reloff:          0x630
+        nreloc:          3
+        flags:           0x80000400
+        reserved1:       0x0
+        reserved2:       0x0
+        reserved3:       0x0
+        content:         554889E5833DFFFFFFFF067D0D833DFFFFFFFF067D0431C05DC3E800000000
+        relocations:
+          - address:         0x1B
+            symbolnum:       3
+            pcrel:           true
+            length:          2
+            extern:          true
+            type:            2
+            scattered:       false
+            value:           0
+          - address:         0xF
+            symbolnum:       1
+            pcrel:           true
+            length:          2
+            extern:          true
+            type:            6
+            scattered:       false
+            value:           0
+          - address:         0x6
+            symbolnum:       0
+            pcrel:           true
+            length:          2
+            extern:          true
+            type:            6
+            scattered:       false
+            value:           0
+      - sectname:        __data
+        segname:         __DATA
+        addr:            0x20
+        size:            8
+        offset:          0x320
+        align:           2
+        reloff:          0x0
+        nreloc:          0
+        flags:           0x0
+        reserved1:       0x0
+        reserved2:       0x0
+        reserved3:       0x0
+        content:         '0400000006000000'
+      - sectname:        __debug_abbrev
+        segname:         __DWARF
+        addr:            0x28
+        size:            182
+        offset:          0x328
+        align:           0
+        reloff:          0x0
+        nreloc:          0
+        flags:           0x2000000
+        reserved1:       0x0
+        reserved2:       0x0
+        reserved3:       0x0
+      - sectname:        __debug_info
+        segname:         __DWARF
+        addr:            0xDE
+        size:            300
+        offset:          0x3DE
+        align:           0
+        reloff:          0x648
+        nreloc:          11
+        flags:           0x2000000
+        reserved1:       0x0
+        reserved2:       0x0
+        reserved3:       0x0
+        relocations:
+          - address:         0x11B
+            symbolnum:       1
+            pcrel:           false
+            length:          3
+            extern:          false
+            type:            0
+            scattered:       false
+            value:           0
+          - address:         0x106
+            symbolnum:       1
+            pcrel:           false
+            length:          3
+            extern:          false
+            type:            0
+            scattered:       false
+            value:           0
+          - address:         0xF3
+            symbolnum:       1
+            pcrel:           false
+            length:          3
+            extern:          false
+            type:            0
+            scattered:       false
+            value:           0
+          - address:         0xDE
+            symbolnum:       1
+            pcrel:           false
+            length:          3
+            extern:          false
+            type:            0
+            scattered:       false
+            value:           0
+          - address:         0xCA
+            symbolnum:       1
+            pcrel:           false
+            length:          3
+            extern:          false
+            type:            0
+            scattered:       false
+            value:           0
+          - address:         0xB5
+            symbolnum:       1
+            pcrel:           false
+            length:          3
+            extern:          false
+            type:            0
+            scattered:       false
+            value:           0
+          - address:         0xA1
+            symbolnum:       1
+            pcrel:           false
+            length:          3
+            extern:          false
+            type:            0
+            scattered:       false
+            value:           0
+          - address:         0x84
+            symbolnum:       1
+            pcrel:           false
+            length:          3
+            extern:          false
+            type:            0
+            scattered:       false
+            value:           0
+          - address:         0x53
+            symbolnum:       2
+            pcrel:           false
+            length:          3
+            extern:          false
+            type:            0
+            scattered:       false
+            value:           0
+          - address:         0x37
+            symbolnum:       2
+            pcrel:           false
+            length:          3
+            extern:          false
+            type:            0
+            scattered:       false
+            value:           0
+          - address:         0x1E
+            symbolnum:       1
+            pcrel:           false
+            length:          3
+            extern:          false
+            type:            0
+            scattered:       false
+            value:           0
+      - sectname:        __debug_str
+        segname:         __DWARF
+        addr:            0x20A
+        size:            106
+        offset:          0x50A
+        align:           0
+        reloff:          0x0
+        nreloc:          0
+        flags:           0x2000000
+        reserved1:       0x0
+        reserved2:       0x0
+        reserved3:       0x0
+      - sectname:        __eh_frame
+        segname:         __TEXT
+        addr:            0x278
+        size:            72
+        offset:          0x578
+        align:           3
+        reloff:          0x0
+        nreloc:          0
+        flags:           0x6800000B
+        reserved1:       0x0
+        reserved2:       0x0
+        reserved3:       0x0
+        content:         1400000000000000017A520001781001100C0708900100002C0000001C00000068FDFFFFFFFFFFFF1F0000000000000000410E108602430D06550C0708410C061000000000000000
+      - sectname:        __debug_line
+        segname:         __DWARF
+        addr:            0x2C0
+        size:            110
+        offset:          0x5C0
+        align:           0
+        reloff:          0x6A0
+        nreloc:          1
+        flags:           0x2000000
+        reserved1:       0x0
+        reserved2:       0x0
+        reserved3:       0x0
+        relocations:
+          - address:         0x3F
+            symbolnum:       1
+            pcrel:           false
+            length:          3
+            extern:          false
+            type:            0
+            scattered:       false
+            value:           0
+  - cmd:             LC_SYMTAB
+    cmdsize:         24
+    symoff:          1704
+    nsyms:           4
+    stroff:          1768
+    strsize:         24
+  - cmd:             LC_DYSYMTAB
+    cmdsize:         80
+    ilocalsym:       0
+    nlocalsym:       0
+    iextdefsym:      0
+    nextdefsym:      3
+    iundefsym:       3
+    nundefsym:       1
+    tocoff:          0
+    ntoc:            0
+    modtaboff:       0
+    nmodtab:         0
+    extrefsymoff:    0
+    nextrefsyms:     0
+    indirectsymoff:  0
+    nindirectsyms:   0
+    extreloff:       0
+    nextrel:         0
+    locreloff:       0
+    nlocrel:         0
+LinkEditData:
+  NameList:
+    - n_strx:          18
+      n_type:          0xF
+      n_sect:          2
+      n_desc:          0
+      n_value:         32
+    - n_strx:          14
+      n_type:          0xF
+      n_sect:          2
+      n_desc:          0
+      n_value:         36
+    - n_strx:          8
+      n_type:          0xF
+      n_sect:          1
+      n_desc:          0
+      n_value:         0
+    - n_strx:          1
+      n_type:          0x1
+      n_sect:          0
+      n_desc:          0
+      n_value:         0
+  StringTable:
+    - ''
+    - _abort
+    - _main
+    - _g2
+    - _g1
+    - ''
+    - ''
+DWARF:
+  debug_str:
+    - clang version 16.0.0.prerel
+    - inlined-file0-line0-col0.c
+    - '/tmp/tmp.o5FdSk4Xkz'
+    - g1
+    - int
+    - g2
+    - bar
+    - q
+    - foo
+    - abort
+    - main
+  debug_abbrev:
+    - ID:              0
+      Table:
+        - Code:            0x1
+          Tag:             DW_TAG_compile_unit
+          Children:        DW_CHILDREN_yes
+          Attributes:
+            - Attribute:       DW_AT_producer
+              Form:            DW_FORM_strp
+            - Attribute:       DW_AT_language
+              Form:            DW_FORM_data2
+            - Attribute:       DW_AT_name
+              Form:            DW_FORM_strp
+            - Attribute:       DW_AT_stmt_list
+              Form:            DW_FORM_sec_offset
+            - Attribute:       DW_AT_comp_dir
+              Form:            DW_FORM_strp
+            - Attribute:       DW_AT_low_pc
+              Form:            DW_FORM_addr
+            - Attribute:       DW_AT_high_pc
+              Form:            DW_FORM_data4
+        - Code:            0x2
+          Tag:             DW_TAG_variable
+          Children:        DW_CHILDREN_no
+          Attributes:
+            - Attribute:       DW_AT_name
+              Form:            DW_FORM_strp
+            - Attribute:       DW_AT_type
+              Form:            DW_FORM_ref4
+            - Attribute:       DW_AT_external
+              Form:            DW_FORM_flag_present
+            - Attribute:       DW_AT_decl_file
+              Form:            DW_FORM_data1
+            - Attribute:       DW_AT_decl_line
+              Form:            DW_FORM_data1
+            - Attribute:       DW_AT_location
+              Form:            DW_FORM_exprloc
+        - Code:            0x3
+          Tag:             DW_TAG_base_type
+          Children:        DW_CHILDREN_no
+          Attributes:
+            - Attribute:       DW_AT_name
+              Form:            DW_FORM_strp
+            - Attribute:       DW_AT_encoding
+              Form:            DW_FORM_data1
+            - Attribute:       DW_AT_byte_size
+              Form:            DW_FORM_data1
+        - Code:            0x4
+          Tag:             DW_TAG_subprogram
+          Children:        DW_CHILDREN_yes
+          Attributes:
+            - Attribute:       DW_AT_name
+              Form:            DW_FORM_strp
+            - Attribute:       DW_AT_decl_file
+              Form:            DW_FORM_data1
+            - Attribute:       DW_AT_decl_line
+              Form:            DW_FORM_data1
+            - Attribute:       DW_AT_prototyped
+              Form:            DW_FORM_flag_present
+            - Attribute:       DW_AT_external
+              Form:            DW_FORM_flag_present
+            - Attribute:       DW_AT_inline
+              Form:            DW_FORM_data1
+        - Code:            0x5
+          Tag:             DW_TAG_formal_parameter
+          Children:        DW_CHILDREN_no
+          Attributes:
+            - Attribute:       DW_AT_name
+              Form:            DW_FORM_strp
+            - Attribute:       DW_AT_decl_file
+              Form:            DW_FORM_data1
+            - Attribute:       DW_AT_decl_line
+              Form:            DW_FORM_data1
+            - Attribute:       DW_AT_type
+              Form:            DW_FORM_ref4
+        - Code:            0x6
+          Tag:             DW_TAG_subprogram
+          Children:        DW_CHILDREN_yes
+          Attributes:
+            - Attribute:       DW_AT_low_pc
+              Form:            DW_FORM_addr
+            - Attribute:       DW_AT_high_pc
+              Form:            DW_FORM_data4
+            - Attribute:       DW_AT_frame_base
+              Form:            DW_FORM_exprloc
+            - Attribute:       DW_AT_GNU_all_call_sites
+              Form:            DW_FORM_flag_present
+            - Attribute:       DW_AT_name
+              Form:            DW_FORM_strp
+            - Attribute:       DW_AT_decl_file
+              Form:            DW_FORM_data1
+            - Attribute:       DW_AT_decl_line
+              Form:            DW_FORM_data1
+            - Attribute:       DW_AT_type
+              Form:            DW_FORM_ref4
+            - Attribute:       DW_AT_external
+              Form:            DW_FORM_flag_present
+        - Code:            0x7
+          Tag:             DW_TAG_inlined_subroutine
+          Children:        DW_CHILDREN_yes
+          Attributes:
+            - Attribute:       DW_AT_abstract_origin
+              Form:            DW_FORM_ref4
+            - Attribute:       DW_AT_low_pc
+              Form:            DW_FORM_addr
+            - Attribute:       DW_AT_high_pc
+              Form:            DW_FORM_data4
+            - Attribute:       DW_AT_call_file
+              Form:            DW_FORM_data1
+            - Attribute:       DW_AT_call_line
+              Form:            DW_FORM_data1
+            - Attribute:       DW_AT_call_column
+              Form:            DW_FORM_data1
+        - Code:            0x8
+          Tag:             DW_TAG_inlined_subroutine
+          Children:        DW_CHILDREN_no
+          Attributes:
+            - Attribute:       DW_AT_abstract_origin
+              Form:            DW_FORM_ref4
+            - Attribute:       DW_AT_low_pc
+              Form:            DW_FORM_addr
+            - Attribute:       DW_AT_high_pc
+              Form:            DW_FORM_data4
+            - Attribute:       DW_AT_call_file
+              Form:            DW_FORM_data1
+            - Attribute:       DW_AT_call_line
+              Form:            DW_FORM_data1
+            - Attribute:       DW_AT_call_column
+              Form:            DW_FORM_data1
+        - Code:            0x9
+          Tag:             DW_TAG_inlined_subroutine
+          Children:        DW_CHILDREN_yes
+          Attributes:
+            - Attribute:       DW_AT_abstract_origin
+              Form:            DW_FORM_ref4
+            - Attribute:       DW_AT_low_pc
+              Form:            DW_FORM_addr
+            - Attribute:       DW_AT_high_pc
+              Form:            DW_FORM_data4
+            - Attribute:       DW_AT_call_file
+              Form:            DW_FORM_data1
+            - Attribute:       DW_AT_call_line
+              Form:            DW_FORM_data1
+        - Code:            0xA
+          Tag:             DW_TAG_GNU_call_site
+          Children:        DW_CHILDREN_no
+          Attributes:
+            - Attribute:       DW_AT_abstract_origin
+              Form:            DW_FORM_ref4
+            - Attribute:       DW_AT_low_pc
+              Form:            DW_FORM_addr
+        - Code:            0xB
+          Tag:             DW_TAG_subprogram
+          Children:        DW_CHILDREN_no
+          Attributes:
+            - Attribute:       DW_AT_name
+              Form:            DW_FORM_strp
+            - Attribute:       DW_AT_decl_file
+              Form:            DW_FORM_data1
+            - Attribute:       DW_AT_decl_line
+              Form:            DW_FORM_data1
+            - Attribute:       DW_AT_prototyped
+              Form:            DW_FORM_flag_present
+            - Attribute:       DW_AT_declaration
+              Form:            DW_FORM_flag_present
+            - Attribute:       DW_AT_external
+              Form:            DW_FORM_flag_present
+            - Attribute:       DW_AT_noreturn
+              Form:            DW_FORM_flag_present
+  debug_info:
+    - Length:          0x128
+      Version:         4
+      AbbrevTableID:   0
+      AbbrOffset:      0x0
+      AddrSize:        8
+      Entries:
+        - AbbrCode:        0x1
+          Values:
+            - Value:           0x0
+            - Value:           0x1D
+            - Value:           0x1C
+            - Value:           0x0
+            - Value:           0x37
+            - Value:           0x0
+            - Value:           0x1F
+        - AbbrCode:        0x2
+          Values:
+            - Value:           0x4B
+            - Value:           0x3F
+            - Value:           0x1
+            - Value:           0x1
+            - Value:           0x2
+            - Value:           0x9
+              BlockData:       [ 0x3, 0x20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 
+                                 0x0 ]
+        - AbbrCode:        0x3
+          Values:
+            - Value:           0x4E
+            - Value:           0x5
+            - Value:           0x4
+        - AbbrCode:        0x2
+          Values:
+            - Value:           0x52
+            - Value:           0x3F
+            - Value:           0x1
+            - Value:           0x1
+            - Value:           0x2
+            - Value:           0x9
+              BlockData:       [ 0x3, 0x24, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 
+                                 0x0 ]
+        - AbbrCode:        0x4
+          Values:
+            - Value:           0x55
+            - Value:           0x1
+            - Value:           0x4
+            - Value:           0x1
+            - Value:           0x1
+            - Value:           0x1
+        - AbbrCode:        0x5
+          Values:
+            - Value:           0x59
+            - Value:           0x1
+            - Value:           0x4
+            - Value:           0x3F
+        - AbbrCode:        0x0
+        - AbbrCode:        0x4
+          Values:
+            - Value:           0x5B
+            - Value:           0x1
+            - Value:           0x9
+            - Value:           0x1
+            - Value:           0x1
+            - Value:           0x1
+        - AbbrCode:        0x5
+          Values:
+            - Value:           0x59
+            - Value:           0x1
+            - Value:           0x9
+            - Value:           0x3F
+        - AbbrCode:        0x0
+        - AbbrCode:        0x6
+          Values:
+            - Value:           0x0
+            - Value:           0x1F
+            - Value:           0x1
+              BlockData:       [ 0x56 ]
+            - Value:           0x1
+            - Value:           0x65
+            - Value:           0x1
+            - Value:           0xD
+            - Value:           0x3F
+            - Value:           0x1
+        - AbbrCode:        0x7
+          Values:
+            - Value:           0x6F
+            - Value:           0x4
+            - Value:           0x9
+            - Value:           0x1
+            - Value:           0xE
+            - Value:           0x3
+        - AbbrCode:        0x8
+          Values:
+            - Value:           0x5B
+            - Value:           0x4
+            - Value:           0x9
+            - Value:           0x1
+            - Value:           0xA
+            - Value:           0x3
+        - AbbrCode:        0x0
+        - AbbrCode:        0x7
+          Values:
+            - Value:           0x6F
+            - Value:           0xD
+            - Value:           0x9
+            - Value:           0x1
+            - Value:           0xF
+            - Value:           0x3
+        - AbbrCode:        0x8
+          Values:
+            - Value:           0x5B
+            - Value:           0xD
+            - Value:           0x9
+            - Value:           0x1
+            - Value:           0xA
+            - Value:           0x3
+        - AbbrCode:        0x0
+        - AbbrCode:        0x9
+          Values:
+            - Value:           0x6F
+            - Value:           0x1A
+            - Value:           0x5
+            - Value:           0x1
+            - Value:           0x0
+        - AbbrCode:        0x8
+          Values:
+            - Value:           0x5B
+            - Value:           0x1A
+            - Value:           0x5
+            - Value:           0x1
+            - Value:           0xA
+            - Value:           0x3
+        - AbbrCode:        0x0
+        - AbbrCode:        0xA
+          Values:
+            - Value:           0x124
+            - Value:           0x1F
+        - AbbrCode:        0x0
+        - AbbrCode:        0xB
+          Values:
+            - Value:           0x5F
+            - Value:           0x1
+            - Value:           0x1
+            - Value:           0x1
+            - Value:           0x1
+            - Value:           0x1
+            - Value:           0x1
+        - AbbrCode:        0x0
+  debug_line:
+    - Length:          106
+      Version:         4
+      PrologueLength:  50
+      MinInstLength:   1
+      MaxOpsPerInst:   1
+      DefaultIsStmt:   1
+      LineBase:        251
+      LineRange:       14
+      OpcodeBase:      13
+      StandardOpcodeLengths: [ 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1 ]
+      Files:
+        - Name:            inlined-file0-line0-col0.c
+          DirIdx:          0
+          ModTime:         0
+          Length:          0
+      Opcodes:
+        - Opcode:          DW_LNS_extended_op
+          ExtLen:          9
+          SubOpcode:       DW_LNE_set_address
+          Data:            0
+        - Opcode:          DW_LNS_advance_line
+          SData:           12
+          Data:            0
+        - Opcode:          DW_LNS_copy
+          Data:            0
+        - Opcode:          DW_LNS_set_column
+          Data:            9
+        - Opcode:          DW_LNS_set_prologue_end
+          Data:            0
+        - Opcode:          DW_LNS_advance_line
+          SData:           -8
+          Data:            0
+        - Opcode:          0x4A
+          Data:            0
+        - Opcode:          DW_LNS_set_column
+          Data:            7
+        - Opcode:          DW_LNS_negate_stmt
+          Data:            0
+        - Opcode:          0x74
+          Data:            0
+        - Opcode:          DW_LNS_set_column
+          Data:            9
+        - Opcode:          0x2E
+          Data:            0
+        - Opcode:          DW_LNS_set_column
+          Data:            7
+        - Opcode:          0x74
+          Data:            0
+        - Opcode:          DW_LNS_set_column
+          Data:            3
+        - Opcode:          DW_LNS_negate_stmt
+          Data:            0
+        - Opcode:          DW_LNS_advance_line
+          SData:           11
+          Data:            0
+        - Opcode:          0x2E
+          Data:            0
+        - Opcode:          DW_LNS_negate_stmt
+          Data:            0
+        - Opcode:          DW_LNS_set_epilogue_begin
+          Data:            0
+        - Opcode:          0x2E
+          Data:            0
+        - Opcode:          DW_LNS_set_column
+          Data:            5
+        - Opcode:          DW_LNS_negate_stmt
+          Data:            0
+        - Opcode:          DW_LNS_advance_line
+          SData:           -10
+          Data:            0
+        - Opcode:          0x2E
+          Data:            0
+        - Opcode:          DW_LNS_advance_pc
+          Data:            5
+        - Opcode:          DW_LNS_extended_op
+          ExtLen:          1
+          SubOpcode:       DW_LNE_end_sequence
+          Data:            0
+...
diff --git a/lldb/test/Shell/SymbolFile/DWARF/debug-types-expressions.test b/lldb/test/Shell/SymbolFile/DWARF/debug-types-expressions.test
index 0442cd8faa3b2..653af7b17c449 100644
--- a/lldb/test/Shell/SymbolFile/DWARF/debug-types-expressions.test
+++ b/lldb/test/Shell/SymbolFile/DWARF/debug-types-expressions.test
@@ -40,18 +40,18 @@ frame variable *a
 # CHECK-NEXT:   j = 42
 # CHECK-NEXT: }
 
-print a->f()
-# CHECK-LABEL: print a->f()
+expression a->f()
+# CHECK-LABEL: expression a->f()
 # CHECK: (int) $0 = 47
 
-print ns::A()
-# CHECK-LABEL: print ns::A()
+expression ns::A()
+# CHECK-LABEL: expression ns::A()
 # CHECK: (ns::A) $1 = (i = 147)
 
-print ns::A().i + a->i
-# CHECK-LABEL: print ns::A().i + a->i
+expression ns::A().i + a->i
+# CHECK-LABEL: expression ns::A().i + a->i
 # CHECK: (int) $2 = 194
 
-print ns::A().getA()
+expression ns::A().getA()
 # CHECK-LABEL: ns::A().getA()
 # CHECK: (A) $3 = (i = 146)
diff --git a/lldb/test/Shell/SymbolFile/DWARF/inlined-file0-line0-col0.test b/lldb/test/Shell/SymbolFile/DWARF/inlined-file0-line0-col0.test
new file mode 100644
index 0000000000000..cd56e6715f5a3
--- /dev/null
+++ b/lldb/test/Shell/SymbolFile/DWARF/inlined-file0-line0-col0.test
@@ -0,0 +1,34 @@
+# RUN: yaml2obj %S/Inputs/inlined-file0-line0-col0.yaml -o %t
+# RUN: %lldb %t -s %s -o exit | FileCheck %s
+
+#  1 void abort(void);
+#  2 int g1 = 4, g2 = 6;
+#  3
+#  4 inline __attribute__((always_inline)) void bar(int q) {
+#  5   if (q > 5)
+#  6     abort();
+#  7 }
+#  8
+#  9 inline __attribute__((always_inline)) void foo(int q) {
+# 10   bar(q);
+# 11 }
+# 12
+# 13 int main() {
+# 14   foo(g1);
+# 15   foo(g2);
+# 16   return 0;
+# 17 }
+
+# The input object file contains a single abort invocation for the two inlined
+# instances of foo() in main() at line 0. As the file, line and column numbers
+# are all 0, file and line number information would be missing for foo and main
+# in the lookup information.
+#
+# A line number 0 is not printed for main in this case, but the same holds
+# for a non-inlined location with line number 0.
+
+# CHECK: Summary: inlined-file0-line0-col0.test.tmp`main + 30 [inlined] bar + 4 at inlined-file0-line0-col0.c:6:5
+# CHECK-NEXT:     inlined-file0-line0-col0.test.tmp`main + 26 [inlined] foo at inlined-file0-line0-col0.c:10:3
+# CHECK-NEXT:     inlined-file0-line0-col0.test.tmp`main + 26 at inlined-file0-line0-col0.c
+
+image lookup -a 0x1e
diff --git a/lldb/test/Shell/SymbolFile/DWARF/split-dwarf-expression-eval-bug.cpp b/lldb/test/Shell/SymbolFile/DWARF/split-dwarf-expression-eval-bug.cpp
index ef1aa257665df..4a8004ddd287f 100644
--- a/lldb/test/Shell/SymbolFile/DWARF/split-dwarf-expression-eval-bug.cpp
+++ b/lldb/test/Shell/SymbolFile/DWARF/split-dwarf-expression-eval-bug.cpp
@@ -11,10 +11,10 @@
 // RUN: %clang_host -c -gsplit-dwarf -g %s -o %t2.o -DTWO
 // RUN: %clang_host -c -gsplit-dwarf -g %s -o %t3.o -DTHREE
 // RUN: %clang_host %t1.o %t2.o %t3.o -o %t
-// RUN: %lldb %t -o "br set -n foo" -o run -o "p bool_in_first_cu" -o exit \
+// RUN: %lldb %t -o "br set -n foo" -o run -o "expression bool_in_first_cu" -o exit \
 // RUN:   | FileCheck %s
 
-// CHECK: (lldb) p bool_in_first_cu
+// CHECK: (lldb) expression bool_in_first_cu
 // CHECK: (bool) $0 = true
 
 
diff --git a/lldb/test/Shell/SymbolFile/DWARF/x86/DW_TAG_GNU_call_site-DW_AT_low_pc.s b/lldb/test/Shell/SymbolFile/DWARF/x86/DW_TAG_GNU_call_site-DW_AT_low_pc.s
index 8cc1cc1ed2f95..c6e5ccda5f2ef 100644
--- a/lldb/test/Shell/SymbolFile/DWARF/x86/DW_TAG_GNU_call_site-DW_AT_low_pc.s
+++ b/lldb/test/Shell/SymbolFile/DWARF/x86/DW_TAG_GNU_call_site-DW_AT_low_pc.s
@@ -11,7 +11,7 @@
 # REQUIRES: target-x86_64, system-linux, lld
 
 # RUN: %clang_host -o %t %s
-# RUN: %lldb %t -o r -o 'p p' -o exit | FileCheck %s
+# RUN: %lldb %t -o r -o 'expression p' -o exit | FileCheck %s
 
 # CHECK: (int) $0 = 1
 
diff --git a/lldb/test/Shell/SymbolFile/DWARF/x86/DW_TAG_variable-DW_AT_const_value.s b/lldb/test/Shell/SymbolFile/DWARF/x86/DW_TAG_variable-DW_AT_const_value.s
index 7ab9a8c61bafd..1d9bec342de5b 100644
--- a/lldb/test/Shell/SymbolFile/DWARF/x86/DW_TAG_variable-DW_AT_const_value.s
+++ b/lldb/test/Shell/SymbolFile/DWARF/x86/DW_TAG_variable-DW_AT_const_value.s
@@ -1,7 +1,7 @@
 # This tests that lldb is able to print DW_TAG_variable using DW_AT_const_value.
 
 # RUN: llvm-mc -triple x86_64-unknown-linux-gnu %s -filetype=obj > %t.o
-# RUN: %lldb %t.o -o "p/x magic64" -o exit | FileCheck %s
+# RUN: %lldb %t.o -o "expression/x magic64" -o exit | FileCheck %s
 
 # CHECK: (const long) $0 = 0xed9a924c00011151
 
diff --git a/lldb/test/Shell/SymbolFile/DWARF/x86/debug-types-basic.test b/lldb/test/Shell/SymbolFile/DWARF/x86/debug-types-basic.test
index 947b43ba6b2ff..dc8005f73930e 100644
--- a/lldb/test/Shell/SymbolFile/DWARF/x86/debug-types-basic.test
+++ b/lldb/test/Shell/SymbolFile/DWARF/x86/debug-types-basic.test
@@ -51,12 +51,12 @@ type lookup EC
 # CHECK-NEXT:   e3
 # CHECK-NEXT: }
 
-print (E) 1
-# CHECK-LABEL: print (E) 1
+expression (E) 1
+# CHECK-LABEL: expression (E) 1
 # CHECK: (E) $0 = e2
 
-print (EC) 1
-# CHECK-LABEL: print (EC) 1
+expression (EC) 1
+# CHECK-LABEL: expression (EC) 1
 # CHECK: (EC) $1 = e2
 
 target variable a e ec
diff --git a/lldb/test/Shell/SymbolFile/DWARF/x86/debug-types-missing-signature.test b/lldb/test/Shell/SymbolFile/DWARF/x86/debug-types-missing-signature.test
index 8bdc2219cc1c9..e94b10a68d4e9 100644
--- a/lldb/test/Shell/SymbolFile/DWARF/x86/debug-types-missing-signature.test
+++ b/lldb/test/Shell/SymbolFile/DWARF/x86/debug-types-missing-signature.test
@@ -14,10 +14,10 @@ LOOKUPE: no type was found matching 'E'
 RUN: %lldb %t -b -o "type lookup EC" | FileCheck --check-prefix=LOOKUPEC %s
 LOOKUPEC: no type was found matching 'EC'
 
-RUN: not %lldb %t -b -o "print (E) 1" 2>&1 | FileCheck --check-prefix=PRINTE %s
+RUN: not %lldb %t -b -o "expression (E) 1" 2>&1 | FileCheck --check-prefix=PRINTE %s
 PRINTE: use of undeclared identifier 'E'
 
-RUN: not %lldb %t -b -o "print (EC) 1" 2>&1 | FileCheck --check-prefix=PRINTEC %s
+RUN: not %lldb %t -b -o "expression (EC) 1" 2>&1 | FileCheck --check-prefix=PRINTEC %s
 PRINTEC: use of undeclared identifier 'EC'
 
 RUN: %lldb %t -b -o "target variable a e ec" | FileCheck --check-prefix=VARS %s
diff --git a/lldb/test/Shell/SymbolFile/DWARF/x86/dwarf5-line-strp.s b/lldb/test/Shell/SymbolFile/DWARF/x86/dwarf5-line-strp.s
index 5c08f4a40528b..67f52454604f6 100755
--- a/lldb/test/Shell/SymbolFile/DWARF/x86/dwarf5-line-strp.s
+++ b/lldb/test/Shell/SymbolFile/DWARF/x86/dwarf5-line-strp.s
@@ -3,7 +3,7 @@
 # UNSUPPORTED: system-darwin, system-windows
 
 # RUN: llvm-mc -filetype=obj -o %t -triple x86_64-pc-linux %s
-# RUN: %lldb %t -o "p main" \
+# RUN: %lldb %t -o "expression main" \
 # RUN:   -o exit | FileCheck %s
 
 # CHECK: (void (*)()) $0 = 0x0000000000000000
diff --git a/lldb/test/Shell/SymbolFile/DWARF/x86/limit-debug-info.s b/lldb/test/Shell/SymbolFile/DWARF/x86/limit-debug-info.s
index 6a2140a147921..4bd0fc8a9b55e 100644
--- a/lldb/test/Shell/SymbolFile/DWARF/x86/limit-debug-info.s
+++ b/lldb/test/Shell/SymbolFile/DWARF/x86/limit-debug-info.s
@@ -3,10 +3,10 @@
 # RUN: llvm-mc --triple=x86_64-pc-windows --filetype=obj --defsym EXE=0 %s >%t.exe.o
 # RUN: lld-link /OUT:%t.dll %t.dll.o /SUBSYSTEM:console /dll /noentry /debug
 # RUN: lld-link /OUT:%t.exe %t.exe.o /SUBSYSTEM:console /debug /force
-# RUN: %lldb %t.exe -o "target modules add %t.dll" -o "p var" \
+# RUN: %lldb %t.exe -o "target modules add %t.dll" -o "expression var" \
 # RUN:   -o exit 2>&1 | FileCheck %s
 
-# CHECK: (lldb) p var
+# CHECK: (lldb) expression var
 # CHECK: (A) $0 = (member = 47)
 
         .section        .debug_abbrev,"dr"
diff --git a/lldb/test/Shell/SymbolFile/NativePDB/Inputs/inline_sites_live.lldbinit b/lldb/test/Shell/SymbolFile/NativePDB/Inputs/inline_sites_live.lldbinit
index 584d835977bf0..2291c7c452717 100644
--- a/lldb/test/Shell/SymbolFile/NativePDB/Inputs/inline_sites_live.lldbinit
+++ b/lldb/test/Shell/SymbolFile/NativePDB/Inputs/inline_sites_live.lldbinit
@@ -1,7 +1,7 @@
 br set -p BP_bar -f inline_sites_live.cpp
 br set -p BP_foo -f inline_sites_live.cpp
 run
-p param
+expression param
 continue
-p param
-p local
+expression param
+expression local
diff --git a/lldb/test/Shell/SymbolFile/NativePDB/Inputs/local-variables.lldbinit b/lldb/test/Shell/SymbolFile/NativePDB/Inputs/local-variables.lldbinit
index 1071edd25c92b..4c8073e7bd884 100644
--- a/lldb/test/Shell/SymbolFile/NativePDB/Inputs/local-variables.lldbinit
+++ b/lldb/test/Shell/SymbolFile/NativePDB/Inputs/local-variables.lldbinit
@@ -1,30 +1,30 @@
 break set -f local-variables.cpp -l 17
 run a b c d e f g
-p argc
+expression argc
 step
-p SomeLocal
+expression SomeLocal
 step
-p Param1
-p Param2
+expression Param1
+expression Param2
 step
-p Param1
-p Param2
-p Local1
+expression Param1
+expression Param2
+expression Local1
 step
-p Param1
-p Param2
-p Local1
-p Local2
+expression Param1
+expression Param2
+expression Local1
+expression Local2
 step
-p Param1
-p Param2
-p Local1
-p Local2
+expression Param1
+expression Param2
+expression Local1
+expression Local2
 step
-p Param1
-p Param2
-p Local1
-p Local2
+expression Param1
+expression Param2
+expression Local1
+expression Local2
 continue
 
 target modules dump ast
diff --git a/lldb/test/Shell/SymbolFile/NativePDB/incomplete-tag-type.cpp b/lldb/test/Shell/SymbolFile/NativePDB/incomplete-tag-type.cpp
index b6e7ab3a7b1a9..f27b186345224 100644
--- a/lldb/test/Shell/SymbolFile/NativePDB/incomplete-tag-type.cpp
+++ b/lldb/test/Shell/SymbolFile/NativePDB/incomplete-tag-type.cpp
@@ -6,13 +6,13 @@
 // RUN: lld-link /debug:full /nodefaultlib /entry:main %t1.obj %t2.obj /out:%t.exe /pdb:%t.pdb
 // RUN: env LLDB_USE_NATIVE_PDB_READER=1 %lldb -f %t.exe -o \
 // RUN:   "settings set interpreter.stop-command-source-on-error false" \
-// RUN:   -o "p b" -o "p d" -o "p static_e_ref" -o "exit" 2>&1 | FileCheck %s
+// RUN:   -o "expression b" -o "expression d" -o "expression static_e_ref" -o "exit" 2>&1 | FileCheck %s
 
-// CHECK: (lldb) p b
+// CHECK: (lldb) expression b
 // CHECK: (B) $0 = {}
-// CHECK: (lldb) p d
+// CHECK: (lldb) expression d
 // CHECK: (D) $1 = {}
-// CHECK: (lldb) p static_e_ref
+// CHECK: (lldb) expression static_e_ref
 // CHECK: error: expression failed to parse:
 // CHECK: error: {{.*}}: incomplete type 'E' where a complete type is required
 // CHECK: static_e_ref
diff --git a/lldb/test/Shell/SymbolFile/NativePDB/inline_sites_live.cpp b/lldb/test/Shell/SymbolFile/NativePDB/inline_sites_live.cpp
index bd3f80afd849c..767149ea18c46 100644
--- a/lldb/test/Shell/SymbolFile/NativePDB/inline_sites_live.cpp
+++ b/lldb/test/Shell/SymbolFile/NativePDB/inline_sites_live.cpp
@@ -24,11 +24,11 @@ int main(int argc, char** argv) {
 
 // CHECK:      * thread #1, stop reason = breakpoint 1
 // CHECK-NEXT:    frame #0: {{.*}}`main [inlined] bar(param=2)
-// CHECK:      (lldb) p param
+// CHECK:      (lldb) expression param
 // CHECK-NEXT: (int) $0 = 2
 // CHECK:      * thread #1, stop reason = breakpoint 2
 // CHECK-NEXT:    frame #0: {{.*}}`main [inlined] foo(param=1)
-// CHECK:      (lldb) p param
+// CHECK:      (lldb) expression param
 // CHECK-NEXT: (int) $1 = 1
-// CHECK-NEXT: (lldb) p local
+// CHECK-NEXT: (lldb) expression local
 // CHECK-NEXT: (int) $2 = 2
diff --git a/lldb/test/Shell/SymbolFile/NativePDB/local-variables.cpp b/lldb/test/Shell/SymbolFile/NativePDB/local-variables.cpp
index 4779a28c8b380..a1f147b4bda6d 100644
--- a/lldb/test/Shell/SymbolFile/NativePDB/local-variables.cpp
+++ b/lldb/test/Shell/SymbolFile/NativePDB/local-variables.cpp
@@ -37,7 +37,7 @@ int main(int argc, char **argv) {
 // CHECK-NEXT:    20
 
 // CHECK:      Process {{.*}} launched: '{{.*}}local-variables.cpp.tmp.exe'
-// CHECK-NEXT: (lldb) p argc
+// CHECK-NEXT: (lldb) expression argc
 // CHECK-NEXT: (int) $0 = 8
 // CHECK-NEXT: (lldb) step
 // CHECK-NEXT: Process {{.*}} stopped
@@ -50,7 +50,7 @@ int main(int argc, char **argv) {
 // CHECK-NEXT:    19 }
 // CHECK-NEXT:    20
 
-// CHECK:      (lldb) p SomeLocal
+// CHECK:      (lldb) expression SomeLocal
 // CHECK-NEXT: (int) $1 = 16
 // CHECK-NEXT: (lldb) step
 // CHECK-NEXT: Process {{.*}} stopped
@@ -64,9 +64,9 @@ int main(int argc, char **argv) {
 // CHECK-NEXT:    11     ++Local1;
 // CHECK-NEXT:    12     ++Local2;
 
-// CHECK:      (lldb) p Param1
+// CHECK:      (lldb) expression Param1
 // CHECK-NEXT: (int) $2 = 16
-// CHECK-NEXT: (lldb) p Param2
+// CHECK-NEXT: (lldb) expression Param2
 // CHECK-NEXT: (char) $3 = 'a'
 // CHECK-NEXT: (lldb) step
 // CHECK-NEXT: Process {{.*}} stopped
@@ -80,11 +80,11 @@ int main(int argc, char **argv) {
 // CHECK-NEXT:    12     ++Local2;
 // CHECK-NEXT:    13     return Local1;
 
-// CHECK:      (lldb) p Param1
+// CHECK:      (lldb) expression Param1
 // CHECK-NEXT: (int) $4 = 16
-// CHECK-NEXT: (lldb) p Param2
+// CHECK-NEXT: (lldb) expression Param2
 // CHECK-NEXT: (char) $5 = 'a'
-// CHECK-NEXT: (lldb) p Local1
+// CHECK-NEXT: (lldb) expression Local1
 // CHECK-NEXT: (unsigned int) $6 = 17
 // CHECK-NEXT: (lldb) step
 // CHECK-NEXT: Process {{.*}} stopped
@@ -98,13 +98,13 @@ int main(int argc, char **argv) {
 // CHECK-NEXT:    13     return Local1;
 // CHECK-NEXT:    14   }
 
-// CHECK:      (lldb) p Param1
+// CHECK:      (lldb) expression Param1
 // CHECK-NEXT: (int) $7 = 16
-// CHECK-NEXT: (lldb) p Param2
+// CHECK-NEXT: (lldb) expression Param2
 // CHECK-NEXT: (char) $8 = 'a'
-// CHECK-NEXT: (lldb) p Local1
+// CHECK-NEXT: (lldb) expression Local1
 // CHECK-NEXT: (unsigned int) $9 = 17
-// CHECK-NEXT: (lldb) p Local2
+// CHECK-NEXT: (lldb) expression Local2
 // CHECK-NEXT: (char) $10 = 'b'
 // CHECK-NEXT: (lldb) step
 // CHECK-NEXT: Process {{.*}} stopped
@@ -118,13 +118,13 @@ int main(int argc, char **argv) {
 // CHECK-NEXT:    14   }
 // CHECK-NEXT:    15
 
-// CHECK:      (lldb) p Param1
+// CHECK:      (lldb) expression Param1
 // CHECK-NEXT: (int) $11 = 16
-// CHECK-NEXT: (lldb) p Param2
+// CHECK-NEXT: (lldb) expression Param2
 // CHECK-NEXT: (char) $12 = 'a'
-// CHECK-NEXT: (lldb) p Local1
+// CHECK-NEXT: (lldb) expression Local1
 // CHECK-NEXT: (unsigned int) $13 = 18
-// CHECK-NEXT: (lldb) p Local2
+// CHECK-NEXT: (lldb) expression Local2
 // CHECK-NEXT: (char) $14 = 'b'
 // CHECK-NEXT: (lldb) step
 // CHECK-NEXT: Process {{.*}} stopped
@@ -138,13 +138,13 @@ int main(int argc, char **argv) {
 // CHECK-NEXT:    15
 // CHECK-NEXT:    16   int main(int argc, char **argv) {
 
-// CHECK:      (lldb) p Param1
+// CHECK:      (lldb) expression Param1
 // CHECK-NEXT: (int) $15 = 16
-// CHECK-NEXT: (lldb) p Param2
+// CHECK-NEXT: (lldb) expression Param2
 // CHECK-NEXT: (char) $16 = 'a'
-// CHECK-NEXT: (lldb) p Local1
+// CHECK-NEXT: (lldb) expression Local1
 // CHECK-NEXT: (unsigned int) $17 = 18
-// CHECK-NEXT: (lldb) p Local2
+// CHECK-NEXT: (lldb) expression Local2
 // CHECK-NEXT: (char) $18 = 'c'
 // CHECK-NEXT: (lldb) continue
 // CHECK-NEXT: Process {{.*}} resuming
diff --git a/lldb/test/Shell/SymbolFile/PDB/Inputs/ExpressionsTest0.script b/lldb/test/Shell/SymbolFile/PDB/Inputs/ExpressionsTest0.script
index d31a2abb68a2f..dcdaa9e1731d2 100644
--- a/lldb/test/Shell/SymbolFile/PDB/Inputs/ExpressionsTest0.script
+++ b/lldb/test/Shell/SymbolFile/PDB/Inputs/ExpressionsTest0.script
@@ -1,7 +1,7 @@
 breakpoint set --file ExpressionsTest.cpp --line 19
 run
-print result
-print N0::N1::sum(N0::N1::buf1, sizeof(N0::N1::buf1))
-print N1::sum(N1::buf1, sizeof(N1::buf1))
-print sum(buf1, sizeof(buf1))
-print sum(buf1, 1000000000)
+expression result
+expression N0::N1::sum(N0::N1::buf1, sizeof(N0::N1::buf1))
+expression N1::sum(N1::buf1, sizeof(N1::buf1))
+expression sum(buf1, sizeof(buf1))
+expression sum(buf1, 1000000000)
diff --git a/lldb/test/Shell/SymbolFile/PDB/Inputs/ExpressionsTest1.script b/lldb/test/Shell/SymbolFile/PDB/Inputs/ExpressionsTest1.script
index dac887faa5bba..d5d6292d3fdb7 100644
--- a/lldb/test/Shell/SymbolFile/PDB/Inputs/ExpressionsTest1.script
+++ b/lldb/test/Shell/SymbolFile/PDB/Inputs/ExpressionsTest1.script
@@ -1 +1 @@
-print sum(buf0, 1)
+expression sum(buf0, 1)
diff --git a/lldb/test/Shell/SymbolFile/PDB/Inputs/ExpressionsTest2.script b/lldb/test/Shell/SymbolFile/PDB/Inputs/ExpressionsTest2.script
index b19240baf99da..ce6af93463711 100644
--- a/lldb/test/Shell/SymbolFile/PDB/Inputs/ExpressionsTest2.script
+++ b/lldb/test/Shell/SymbolFile/PDB/Inputs/ExpressionsTest2.script
@@ -1,2 +1,2 @@
-print sum(buf0, result - 28)
-print sum(buf1 + 3, 3)
+expression sum(buf0, result - 28)
+expression sum(buf1 + 3, 3)
diff --git a/lldb/test/Shell/SymbolFile/PDB/Inputs/VBases.script b/lldb/test/Shell/SymbolFile/PDB/Inputs/VBases.script
index 8675890b76e1d..9206ccadafdb9 100644
--- a/lldb/test/Shell/SymbolFile/PDB/Inputs/VBases.script
+++ b/lldb/test/Shell/SymbolFile/PDB/Inputs/VBases.script
@@ -2,6 +2,6 @@ breakpoint set --file VBases.cpp --line 15
 
 run
 
-print c
+expression c
 
-frame variable c
\ No newline at end of file
+frame variable c
diff --git a/lldb/test/Shell/SymbolFile/PDB/expressions.test b/lldb/test/Shell/SymbolFile/PDB/expressions.test
index 819e2e180945c..89d7c94e7aa06 100644
--- a/lldb/test/Shell/SymbolFile/PDB/expressions.test
+++ b/lldb/test/Shell/SymbolFile/PDB/expressions.test
@@ -2,34 +2,34 @@ REQUIRES: system-windows, msvc
 RUN: %build --compiler=msvc --nodefaultlib --output=%t.exe %S/Inputs/ExpressionsTest.cpp
 RUN: not %lldb -b -s %S/Inputs/ExpressionsTest0.script -s %S/Inputs/ExpressionsTest1.script -s %S/Inputs/ExpressionsTest2.script -- %t.exe 2>&1 | FileCheck %s
 
-// Check the variable value through `print`
-CHECK: (lldb) print result
+// Check the variable value through `expression`
+CHECK: (lldb) expression result
 CHECK: (char) $0 = '\x1c'
 
 // Call the function just like in the code
-CHECK: (lldb) print N0::N1::sum(N0::N1::buf1, sizeof(N0::N1::buf1))
+CHECK: (lldb) expression N0::N1::sum(N0::N1::buf1, sizeof(N0::N1::buf1))
 CHECK: (char) $1 = '\x1c'
 
 // Try the relaxed namespaces search
-CHECK: (lldb) print N1::sum(N1::buf1, sizeof(N1::buf1))
+CHECK: (lldb) expression N1::sum(N1::buf1, sizeof(N1::buf1))
 CHECK: (char) $2 = '\x1c'
 
 // Try the relaxed variables and functions search
-CHECK: (lldb) print sum(buf1, sizeof(buf1))
+CHECK: (lldb) expression sum(buf1, sizeof(buf1))
 CHECK: (char) $3 = '\x1c'
 
 // Make a crash during expression calculation
-CHECK: (lldb) print sum(buf1, 1000000000)
+CHECK: (lldb) expression sum(buf1, 1000000000)
 CHECK: The process has been returned to the state before expression evaluation.
 
 // Make one more crash
-CHECK: (lldb) print sum(buf0, 1)
+CHECK: (lldb) expression sum(buf0, 1)
 CHECK: The process has been returned to the state before expression evaluation.
 
 // Check if the process state was restored succesfully
-CHECK: (lldb) print sum(buf0, result - 28)
+CHECK: (lldb) expression sum(buf0, result - 28)
 CHECK: (char) $4 = '\0'
 
 // Call the function with arbitrary parameters
-CHECK: (lldb) print sum(buf1 + 3, 3)
+CHECK: (lldb) expression sum(buf1 + 3, 3)
 CHECK: (char) $5 = '\f'
diff --git a/lldb/tools/lldb-vscode/lldb-vscode.cpp b/lldb/tools/lldb-vscode/lldb-vscode.cpp
index b70f5b0ec8bde..51a72f2b03c95 100644
--- a/lldb/tools/lldb-vscode/lldb-vscode.cpp
+++ b/lldb/tools/lldb-vscode/lldb-vscode.cpp
@@ -1065,8 +1065,9 @@ void request_completions(const llvm::json::Object &request) {
     text = text.substr(1);
     actual_column--;
   } else {
-    text = "p " + text;
-    actual_column += 2;
+    char command[] = "expression -- ";
+    text = command + text;
+    actual_column += strlen(command);
   }
   lldb::SBStringList matches;
   lldb::SBStringList descriptions;
diff --git a/lldb/unittests/Core/CommunicationTest.cpp b/lldb/unittests/Core/CommunicationTest.cpp
index 19ca4cb3d2493..df9ff089a0d73 100644
--- a/lldb/unittests/Core/CommunicationTest.cpp
+++ b/lldb/unittests/Core/CommunicationTest.cpp
@@ -86,8 +86,9 @@ static void CommunicationReadTest(bool use_read_thread) {
 
   // Test using Communication that is disconnected.
   ASSERT_EQ(comm.Disconnect(), lldb::eConnectionStatusSuccess);
-  if (use_read_thread)
+  if (use_read_thread) {
     ASSERT_TRUE(comm.StartReadThread());
+  }
   error.Clear();
   EXPECT_EQ(
       comm.Read(buf, sizeof(buf), std::chrono::seconds(5), status, &error), 0U);
@@ -97,8 +98,9 @@ static void CommunicationReadTest(bool use_read_thread) {
 
   // Test using Communication without a connection.
   comm.SetConnection(nullptr);
-  if (use_read_thread)
+  if (use_read_thread) {
     ASSERT_TRUE(comm.StartReadThread());
+  }
   error.Clear();
   EXPECT_EQ(
       comm.Read(buf, sizeof(buf), std::chrono::seconds(5), status, &error), 0U);
diff --git a/lldb/unittests/ScriptInterpreter/Python/PythonDataObjectsTests.cpp b/lldb/unittests/ScriptInterpreter/Python/PythonDataObjectsTests.cpp
index 2bf713e085760..733b919e28607 100644
--- a/lldb/unittests/ScriptInterpreter/Python/PythonDataObjectsTests.cpp
+++ b/lldb/unittests/ScriptInterpreter/Python/PythonDataObjectsTests.cpp
@@ -802,10 +802,13 @@ def main():
 
   PythonScript lol(script2);
 
-  EXPECT_THAT_EXPECTED(lol(),
-                       llvm::Failed<PythonException>(testing::Property(
-                           &PythonException::ReadBacktrace,
-                           testing::ContainsRegex("unprintable MyError"))));
+  EXPECT_THAT_EXPECTED(
+      lol(),
+      llvm::Failed<PythonException>(testing::Property(
+          &PythonException::ReadBacktrace,
+          testing::AnyOf(
+              testing::ContainsRegex("MyError: <exception str\\(\\) failed>"),
+              testing::ContainsRegex("unprintable MyError")))));
 
 #endif
 }
diff --git a/lldb/unittests/Symbol/CMakeLists.txt b/lldb/unittests/Symbol/CMakeLists.txt
index 60c0a9bdcf467..e1d24357e33db 100644
--- a/lldb/unittests/Symbol/CMakeLists.txt
+++ b/lldb/unittests/Symbol/CMakeLists.txt
@@ -1,4 +1,5 @@
 add_lldb_unittest(SymbolTests
+  JSONSymbolTest.cpp
   LocateSymbolFileTest.cpp
   MangledTest.cpp
   PostfixExpressionTest.cpp
diff --git a/lldb/unittests/Symbol/JSONSymbolTest.cpp b/lldb/unittests/Symbol/JSONSymbolTest.cpp
new file mode 100644
index 0000000000000..76c34b89f902f
--- /dev/null
+++ b/lldb/unittests/Symbol/JSONSymbolTest.cpp
@@ -0,0 +1,192 @@
+//===-- JSONSymbolTest.cpp ------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "lldb/Core/Section.h"
+#include "lldb/Symbol/Symbol.h"
+#include "llvm/Testing/Support/Error.h"
+
+#include "gtest/gtest.h"
+
+using namespace lldb;
+using namespace llvm;
+using namespace lldb_private;
+
+static std::string g_error_no_section_list = "no section list provided";
+static std::string g_error_both_value_and_address =
+    "symbol cannot contain both a value and an address";
+static std::string g_error_neither_value_or_address =
+    "symbol must contain either a value or an address";
+
+TEST(JSONSymbolTest, DeserializeCodeAddress) {
+  std::string text = R"(
+{
+  "name": "foo",
+  "type": "code",
+  "size": 32,
+  "address": 4096
+})";
+
+  Expected<json::Value> json = json::parse(text);
+  ASSERT_TRUE(static_cast<bool>(json));
+
+  json::Path::Root root;
+  JSONSymbol json_symbol;
+  ASSERT_TRUE(fromJSON(*json, json_symbol, root));
+
+  SectionSP sect_sp(new Section(
+      /*module_sp=*/ModuleSP(),
+      /*obj_file=*/nullptr,
+      /*sect_id=*/1,
+      /*name=*/ConstString(".text"),
+      /*sect_type=*/eSectionTypeCode,
+      /*file_vm_addr=*/0x1000,
+      /*vm_size=*/0x1000,
+      /*file_offset=*/0,
+      /*file_size=*/0,
+      /*log2align=*/5,
+      /*flags=*/0x10203040));
+  SectionList sect_list;
+  sect_list.AddSection(sect_sp);
+
+  Expected<Symbol> symbol = Symbol::FromJSON(json_symbol, &sect_list);
+  EXPECT_THAT_EXPECTED(symbol, llvm::Succeeded());
+  EXPECT_EQ(symbol->GetName(), ConstString("foo"));
+  EXPECT_EQ(symbol->GetFileAddress(), static_cast<lldb::addr_t>(0x1000));
+  EXPECT_EQ(symbol->GetType(), eSymbolTypeCode);
+}
+
+TEST(JSONSymbolTest, DeserializeCodeValue) {
+  std::string text = R"(
+{
+  "name": "foo",
+  "type": "code",
+  "size": 32,
+  "value": 4096
+})";
+
+  Expected<json::Value> json = json::parse(text);
+  EXPECT_THAT_EXPECTED(json, llvm::Succeeded());
+
+  json::Path::Root root;
+  JSONSymbol json_symbol;
+  ASSERT_TRUE(fromJSON(*json, json_symbol, root));
+
+  SectionList sect_list;
+
+  Expected<Symbol> symbol = Symbol::FromJSON(json_symbol, &sect_list);
+  EXPECT_THAT_EXPECTED(symbol, llvm::Succeeded());
+  EXPECT_EQ(symbol->GetName(), ConstString("foo"));
+  EXPECT_EQ(symbol->GetRawValue(), static_cast<lldb::addr_t>(0x1000));
+  EXPECT_EQ(symbol->GetType(), eSymbolTypeCode);
+}
+
+TEST(JSONSymbolTest, JSONInvalidValueAndAddress) {
+  std::string text = R"(
+{
+  "name": "foo",
+  "type": "code",
+  "size": 32,
+  "value": 4096,
+  "address": 4096
+})";
+
+  Expected<json::Value> json = json::parse(text);
+  EXPECT_THAT_EXPECTED(json, llvm::Succeeded());
+
+  json::Path::Root root;
+  JSONSymbol json_symbol;
+  ASSERT_FALSE(fromJSON(*json, json_symbol, root));
+}
+
+TEST(JSONSymbolTest, JSONInvalidNoValueOrAddress) {
+  std::string text = R"(
+{
+  "name": "foo",
+  "type": "code",
+  "size": 32
+})";
+
+  Expected<json::Value> json = json::parse(text);
+  EXPECT_THAT_EXPECTED(json, llvm::Succeeded());
+
+  json::Path::Root root;
+  JSONSymbol json_symbol;
+  ASSERT_FALSE(fromJSON(*json, json_symbol, root));
+}
+
+TEST(JSONSymbolTest, JSONInvalidType) {
+  std::string text = R"(
+{
+  "name": "foo",
+  "type": "bogus",
+  "value": 4096,
+  "size": 32
+})";
+
+  Expected<json::Value> json = json::parse(text);
+  EXPECT_THAT_EXPECTED(json, llvm::Succeeded());
+
+  json::Path::Root root;
+  JSONSymbol json_symbol;
+  ASSERT_FALSE(fromJSON(*json, json_symbol, root));
+}
+
+TEST(JSONSymbolTest, SymbolInvalidNoSectionList) {
+  JSONSymbol json_symbol;
+  json_symbol.value = 0x1;
+
+  Expected<Symbol> symbol = Symbol::FromJSON(json_symbol, nullptr);
+  EXPECT_THAT_EXPECTED(symbol,
+                       llvm::FailedWithMessage(g_error_no_section_list));
+}
+
+TEST(JSONSymbolTest, SymbolInvalidValueAndAddress) {
+  JSONSymbol json_symbol;
+  json_symbol.value = 0x1;
+  json_symbol.address = 0x2;
+
+  SectionList sect_list;
+
+  Expected<Symbol> symbol = Symbol::FromJSON(json_symbol, &sect_list);
+  EXPECT_THAT_EXPECTED(symbol,
+                       llvm::FailedWithMessage(g_error_both_value_and_address));
+}
+
+TEST(JSONSymbolTest, SymbolInvalidNoValueOrAddress) {
+  JSONSymbol json_symbol;
+
+  SectionList sect_list;
+
+  Expected<Symbol> symbol = Symbol::FromJSON(json_symbol, &sect_list);
+  EXPECT_THAT_EXPECTED(
+      symbol, llvm::FailedWithMessage(g_error_neither_value_or_address));
+}
+
+TEST(JSONSymbolTest, SymbolInvalidAddressNotInSection) {
+  JSONSymbol json_symbol;
+  json_symbol.address = 0x0fff;
+
+  SectionSP sect_sp(new Section(
+      /*module_sp=*/ModuleSP(),
+      /*obj_file=*/nullptr,
+      /*sect_id=*/1,
+      /*name=*/ConstString(".text"),
+      /*sect_type=*/eSectionTypeCode,
+      /*file_vm_addr=*/0x1000,
+      /*vm_size=*/0x1000,
+      /*file_offset=*/0,
+      /*file_size=*/0,
+      /*log2align=*/5,
+      /*flags=*/0x10203040));
+  SectionList sect_list;
+  sect_list.AddSection(sect_sp);
+
+  Expected<Symbol> symbol = Symbol::FromJSON(json_symbol, &sect_list);
+  EXPECT_THAT_EXPECTED(
+      symbol, llvm::FailedWithMessage("no section found for address: 0xfff"));
+}
diff --git a/lldb/unittests/Target/CMakeLists.txt b/lldb/unittests/Target/CMakeLists.txt
index c16f0fcfabe15..772d5847b11cc 100644
--- a/lldb/unittests/Target/CMakeLists.txt
+++ b/lldb/unittests/Target/CMakeLists.txt
@@ -3,6 +3,7 @@ add_lldb_unittest(TargetTests
   DynamicRegisterInfoTest.cpp
   ExecutionContextTest.cpp
   MemoryRegionInfoTest.cpp
+  MemoryTest.cpp
   MemoryTagMapTest.cpp
   ModuleCacheTest.cpp
   PathMappingListTest.cpp
@@ -15,6 +16,7 @@ add_lldb_unittest(TargetTests
       lldbHost
       lldbPluginObjectFileELF
       lldbPluginPlatformLinux
+      lldbPluginPlatformMacOSX
       lldbPluginSymbolFileSymtab
       lldbTarget
       lldbSymbol
diff --git a/lldb/unittests/Target/MemoryTest.cpp b/lldb/unittests/Target/MemoryTest.cpp
new file mode 100644
index 0000000000000..4d7c4d5b4c784
--- /dev/null
+++ b/lldb/unittests/Target/MemoryTest.cpp
@@ -0,0 +1,228 @@
+//===-- MemoryTest.cpp ----------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "lldb/Target/Memory.h"
+#include "Plugins/Platform/MacOSX/PlatformMacOSX.h"
+#include "Plugins/Platform/MacOSX/PlatformRemoteMacOSX.h"
+#include "lldb/Core/Debugger.h"
+#include "lldb/Host/FileSystem.h"
+#include "lldb/Host/HostInfo.h"
+#include "lldb/Target/Process.h"
+#include "lldb/Target/Target.h"
+#include "lldb/Utility/ArchSpec.h"
+#include "lldb/Utility/DataBufferHeap.h"
+#include "gtest/gtest.h"
+
+using namespace lldb_private;
+using namespace lldb_private::repro;
+using namespace lldb;
+
+namespace {
+class MemoryTest : public ::testing::Test {
+public:
+  void SetUp() override {
+    FileSystem::Initialize();
+    HostInfo::Initialize();
+    PlatformMacOSX::Initialize();
+  }
+  void TearDown() override {
+    PlatformMacOSX::Terminate();
+    HostInfo::Terminate();
+    FileSystem::Terminate();
+  }
+};
+
+class DummyProcess : public Process {
+public:
+  DummyProcess(lldb::TargetSP target_sp, lldb::ListenerSP listener_sp)
+      : Process(target_sp, listener_sp), m_bytes_left(0) {}
+
+  // Required overrides
+  bool CanDebug(lldb::TargetSP target, bool plugin_specified_by_name) override {
+    return true;
+  }
+  Status DoDestroy() override { return {}; }
+  void RefreshStateAfterStop() override {}
+  size_t DoReadMemory(lldb::addr_t vm_addr, void *buf, size_t size,
+                      Status &error) override {
+    if (m_bytes_left == 0)
+      return 0;
+
+    size_t num_bytes_to_write = size;
+    if (m_bytes_left < size) {
+      num_bytes_to_write = m_bytes_left;
+      m_bytes_left = 0;
+    } else {
+      m_bytes_left -= size;
+    }
+
+    memset(buf, 'B', num_bytes_to_write);
+    return num_bytes_to_write;
+  }
+  bool DoUpdateThreadList(ThreadList &old_thread_list,
+                          ThreadList &new_thread_list) override {
+    return false;
+  }
+  llvm::StringRef GetPluginName() override { return "Dummy"; }
+
+  // Test-specific additions
+  size_t m_bytes_left;
+  MemoryCache &GetMemoryCache() { return m_memory_cache; }
+  void SetMaxReadSize(size_t size) { m_bytes_left = size; }
+};
+} // namespace
+
+TargetSP CreateTarget(DebuggerSP &debugger_sp, ArchSpec &arch) {
+  PlatformSP platform_sp;
+  TargetSP target_sp;
+  debugger_sp->GetTargetList().CreateTarget(
+      *debugger_sp, "", arch, eLoadDependentsNo, platform_sp, target_sp);
+  return target_sp;
+}
+
+TEST_F(MemoryTest, TesetMemoryCacheRead) {
+  ArchSpec arch("x86_64-apple-macosx-");
+
+  Platform::SetHostPlatform(PlatformRemoteMacOSX::CreateInstance(true, &arch));
+
+  DebuggerSP debugger_sp = Debugger::CreateInstance();
+  ASSERT_TRUE(debugger_sp);
+
+  TargetSP target_sp = CreateTarget(debugger_sp, arch);
+  ASSERT_TRUE(target_sp);
+
+  ListenerSP listener_sp(Listener::MakeListener("dummy"));
+  ProcessSP process_sp = std::make_shared<DummyProcess>(target_sp, listener_sp);
+  ASSERT_TRUE(process_sp);
+
+  DummyProcess *process = static_cast<DummyProcess *>(process_sp.get());
+  MemoryCache &mem_cache = process->GetMemoryCache();
+  const uint64_t l2_cache_size = process->GetMemoryCacheLineSize();
+  Status error;
+  auto data_sp = std::make_shared<DataBufferHeap>(l2_cache_size * 2, '\0');
+  size_t bytes_read = 0;
+
+  // Cache empty, memory read fails, size > l2 cache size
+  process->SetMaxReadSize(0);
+  bytes_read = mem_cache.Read(0x1000, data_sp->GetBytes(),
+                              data_sp->GetByteSize(), error);
+  ASSERT_TRUE(bytes_read == 0);
+
+  // Cache empty, memory read fails, size <= l2 cache size
+  data_sp->SetByteSize(l2_cache_size);
+  bytes_read = mem_cache.Read(0x1000, data_sp->GetBytes(),
+                              data_sp->GetByteSize(), error);
+  ASSERT_TRUE(bytes_read == 0);
+
+  // Cache empty, memory read succeeds, size > l2 cache size
+  process->SetMaxReadSize(l2_cache_size * 4);
+  data_sp->SetByteSize(l2_cache_size * 2);
+  bytes_read = mem_cache.Read(0x1000, data_sp->GetBytes(),
+                              data_sp->GetByteSize(), error);
+  ASSERT_TRUE(bytes_read == data_sp->GetByteSize());
+  ASSERT_TRUE(process->m_bytes_left == l2_cache_size * 2);
+
+  // Reading data previously cached (not in L2 cache).
+  data_sp->SetByteSize(l2_cache_size + 1);
+  bytes_read = mem_cache.Read(0x1000, data_sp->GetBytes(),
+                              data_sp->GetByteSize(), error);
+  ASSERT_TRUE(bytes_read == data_sp->GetByteSize());
+  ASSERT_TRUE(process->m_bytes_left == l2_cache_size * 2); // Verify we didn't
+                                                           // read from the
+                                                           // inferior.
+
+  // Read from a different address, but make the size == l2 cache size.
+  // This should fill in a the L2 cache.
+  data_sp->SetByteSize(l2_cache_size);
+  bytes_read = mem_cache.Read(0x2000, data_sp->GetBytes(),
+                              data_sp->GetByteSize(), error);
+  ASSERT_TRUE(bytes_read == data_sp->GetByteSize());
+  ASSERT_TRUE(process->m_bytes_left == l2_cache_size);
+
+  // Read from that L2 cache entry but read less than size of the cache line.
+  // Additionally, read from an offset.
+  data_sp->SetByteSize(l2_cache_size - 5);
+  bytes_read = mem_cache.Read(0x2001, data_sp->GetBytes(),
+                              data_sp->GetByteSize(), error);
+  ASSERT_TRUE(bytes_read == data_sp->GetByteSize());
+  ASSERT_TRUE(process->m_bytes_left == l2_cache_size); // Verify we didn't read
+                                                       // from the inferior.
+
+  // What happens if we try to populate an L2 cache line but the read gives less
+  // than the size of a cache line?
+  process->SetMaxReadSize(l2_cache_size - 10);
+  data_sp->SetByteSize(l2_cache_size - 5);
+  bytes_read = mem_cache.Read(0x3000, data_sp->GetBytes(),
+                              data_sp->GetByteSize(), error);
+  ASSERT_TRUE(bytes_read == l2_cache_size - 10);
+  ASSERT_TRUE(process->m_bytes_left == 0);
+
+  // What happens if we have a partial L2 cache line filled in and we try to
+  // read the part that isn't filled in?
+  data_sp->SetByteSize(10);
+  bytes_read = mem_cache.Read(0x3000 + l2_cache_size - 10, data_sp->GetBytes(),
+                              data_sp->GetByteSize(), error);
+  ASSERT_TRUE(bytes_read == 0); // The last 10 bytes from this line are
+                                // missing and we should be reading nothing
+                                // here.
+
+  // What happens when we try to straddle 2 cache lines?
+  process->SetMaxReadSize(l2_cache_size * 2);
+  data_sp->SetByteSize(l2_cache_size);
+  bytes_read = mem_cache.Read(0x4001, data_sp->GetBytes(),
+                              data_sp->GetByteSize(), error);
+  ASSERT_TRUE(bytes_read == l2_cache_size);
+  ASSERT_TRUE(process->m_bytes_left == 0);
+
+  // What happens when we try to straddle 2 cache lines where the first one is
+  // only partially filled?
+  process->SetMaxReadSize(l2_cache_size - 1);
+  data_sp->SetByteSize(l2_cache_size);
+  bytes_read = mem_cache.Read(0x5005, data_sp->GetBytes(),
+                              data_sp->GetByteSize(), error);
+  ASSERT_TRUE(bytes_read == l2_cache_size - 6); // Ignoring the first 5 bytes,
+                                                // missing the last byte
+  ASSERT_TRUE(process->m_bytes_left == 0);
+
+  // What happens if we add an invalid range and try to do a read larger than
+  // a cache line?
+  mem_cache.AddInvalidRange(0x6000, l2_cache_size * 2);
+  process->SetMaxReadSize(l2_cache_size * 2);
+  data_sp->SetByteSize(l2_cache_size * 2);
+  bytes_read = mem_cache.Read(0x6000, data_sp->GetBytes(),
+                              data_sp->GetByteSize(), error);
+  ASSERT_TRUE(bytes_read == 0);
+  ASSERT_TRUE(process->m_bytes_left == l2_cache_size * 2);
+
+  // What happens if we add an invalid range and try to do a read lt/eq a
+  // cache line?
+  mem_cache.AddInvalidRange(0x7000, l2_cache_size);
+  process->SetMaxReadSize(l2_cache_size);
+  data_sp->SetByteSize(l2_cache_size);
+  bytes_read = mem_cache.Read(0x7000, data_sp->GetBytes(),
+                              data_sp->GetByteSize(), error);
+  ASSERT_TRUE(bytes_read == 0);
+  ASSERT_TRUE(process->m_bytes_left == l2_cache_size);
+
+  // What happens if we remove the invalid range and read again?
+  mem_cache.RemoveInvalidRange(0x7000, l2_cache_size);
+  bytes_read = mem_cache.Read(0x7000, data_sp->GetBytes(),
+                              data_sp->GetByteSize(), error);
+  ASSERT_TRUE(bytes_read == l2_cache_size);
+  ASSERT_TRUE(process->m_bytes_left == 0);
+
+  // What happens if we flush and read again?
+  process->SetMaxReadSize(l2_cache_size * 2);
+  mem_cache.Flush(0x7000, l2_cache_size);
+  bytes_read = mem_cache.Read(0x7000, data_sp->GetBytes(),
+                              data_sp->GetByteSize(), error);
+  ASSERT_TRUE(bytes_read == l2_cache_size);
+  ASSERT_TRUE(process->m_bytes_left == l2_cache_size); // Verify that we re-read
+                                                       // instead of using an
+                                                       // old cache
+}
diff --git a/lldb/unittests/TestingSupport/MockTildeExpressionResolver.cpp b/lldb/unittests/TestingSupport/MockTildeExpressionResolver.cpp
index b34e01bd01a05..0e64e1977ca44 100644
--- a/lldb/unittests/TestingSupport/MockTildeExpressionResolver.cpp
+++ b/lldb/unittests/TestingSupport/MockTildeExpressionResolver.cpp
@@ -21,7 +21,7 @@ MockTildeExpressionResolver::MockTildeExpressionResolver(StringRef CurrentUser,
 
 void MockTildeExpressionResolver::AddKnownUser(StringRef User,
                                                StringRef HomeDir) {
-  assert(UserDirectories.find(User) == UserDirectories.end());
+  assert(!UserDirectories.contains(User));
   UserDirectories.insert(std::make_pair(User, HomeDir));
 }
 
@@ -31,7 +31,7 @@ void MockTildeExpressionResolver::Clear() {
 }
 
 void MockTildeExpressionResolver::SetCurrentUser(StringRef User) {
-  assert(UserDirectories.find(User) != UserDirectories.end());
+  assert(UserDirectories.contains(User));
   CurrentUser = User;
 }
 
diff --git a/llvm/CMakeLists.txt b/llvm/CMakeLists.txt
index c993ff53d134e..bd23d6657829a 100644
--- a/llvm/CMakeLists.txt
+++ b/llvm/CMakeLists.txt
@@ -388,7 +388,7 @@ string(TOUPPER "${CMAKE_BUILD_TYPE}" uppercase_CMAKE_BUILD_TYPE)
 
 if (CMAKE_BUILD_TYPE AND
     NOT uppercase_CMAKE_BUILD_TYPE MATCHES "^(DEBUG|RELEASE|RELWITHDEBINFO|MINSIZEREL)$")
-  message(FATAL_ERROR "Invalid value for CMAKE_BUILD_TYPE: ${CMAKE_BUILD_TYPE}")
+  message(WARNING "Unknown value for CMAKE_BUILD_TYPE: ${CMAKE_BUILD_TYPE}")
 endif()
 
 # LLVM_INSTALL_PACKAGE_DIR needs to be declared prior to adding the tools
@@ -720,6 +720,10 @@ option(LLVM_BUILD_TESTS
   "Build LLVM unit tests. If OFF, just generate build targets." OFF)
 option(LLVM_INCLUDE_TESTS "Generate build targets for the LLVM unit tests." ON)
 
+option(LLVM_INSTALL_GTEST
+  "Install the llvm gtest library.  This should be on if you want to do
+   stand-alone builds of the other projects and run their unit tests." OFF)
+
 option(LLVM_BUILD_BENCHMARKS "Add LLVM benchmark targets to the list of default
 targets. If OFF, benchmarks still could be built using Benchmarks target." OFF)
 option(LLVM_INCLUDE_BENCHMARKS "Generate benchmark targets. If OFF, benchmarks can't be built." ON)
diff --git a/llvm/bindings/ocaml/llvm/llvm.ml b/llvm/bindings/ocaml/llvm/llvm.ml
index 6962dce737d24..77de9a6e46fae 100644
--- a/llvm/bindings/ocaml/llvm/llvm.ml
+++ b/llvm/bindings/ocaml/llvm/llvm.ml
@@ -685,8 +685,6 @@ external const_pointercast : llvalue -> lltype -> llvalue
 external const_intcast : llvalue -> lltype -> is_signed:bool -> llvalue
                        = "llvm_const_intcast"
 external const_fpcast : llvalue -> lltype -> llvalue = "llvm_const_fpcast"
-external const_select : llvalue -> llvalue -> llvalue -> llvalue
-                      = "llvm_const_select"
 external const_extractelement : llvalue -> llvalue -> llvalue
                               = "llvm_const_extractelement"
 external const_insertelement : llvalue -> llvalue -> llvalue -> llvalue
diff --git a/llvm/bindings/ocaml/llvm/llvm.mli b/llvm/bindings/ocaml/llvm/llvm.mli
index 59ac5b8551303..9c8b3b883e149 100644
--- a/llvm/bindings/ocaml/llvm/llvm.mli
+++ b/llvm/bindings/ocaml/llvm/llvm.mli
@@ -1268,11 +1268,6 @@ val const_intcast : llvalue -> lltype -> is_signed:bool -> llvalue
     See the method [llvm::ConstantExpr::getFPCast]. *)
 val const_fpcast : llvalue -> lltype -> llvalue
 
-(** [const_select cond t f] returns the constant conditional which returns value
-    [t] if the boolean constant [cond] is true and the value [f] otherwise.
-    See the method [llvm::ConstantExpr::getSelect]. *)
-val const_select : llvalue -> llvalue -> llvalue -> llvalue
-
 (** [const_extractelement vec i] returns the constant [i]th element of
     constant vector [vec]. [i] must be a constant [i32] value unsigned less than
     the size of the vector.
diff --git a/llvm/bindings/ocaml/llvm/llvm_ocaml.c b/llvm/bindings/ocaml/llvm/llvm_ocaml.c
index a9d25751fd2ca..0154b2f49c252 100644
--- a/llvm/bindings/ocaml/llvm/llvm_ocaml.c
+++ b/llvm/bindings/ocaml/llvm/llvm_ocaml.c
@@ -1385,13 +1385,6 @@ value llvm_const_fpcast(value CV, value T) {
   return to_val(Value);
 }
 
-/* llvalue -> llvalue -> llvalue -> llvalue */
-value llvm_const_select(value Cond, value IfTrue, value IfFalse) {
-  LLVMValueRef Value =
-      LLVMConstSelect(Value_val(Cond), Value_val(IfTrue), Value_val(IfFalse));
-  return to_val(Value);
-}
-
 /* llvalue -> llvalue -> llvalue */
 value llvm_const_extractelement(value V, value I) {
   LLVMValueRef Value = LLVMConstExtractElement(Value_val(V), Value_val(I));
diff --git a/llvm/cmake/modules/AddLLVM.cmake b/llvm/cmake/modules/AddLLVM.cmake
index 750838875cd0e..cb9254c023e6c 100644
--- a/llvm/cmake/modules/AddLLVM.cmake
+++ b/llvm/cmake/modules/AddLLVM.cmake
@@ -25,7 +25,9 @@ function(llvm_update_compile_flags name)
   else()
     if(LLVM_COMPILER_IS_GCC_COMPATIBLE)
       list(APPEND LLVM_COMPILE_FLAGS "-fno-exceptions")
-      if(NOT LLVM_ENABLE_UNWIND_TABLES)
+      if(LLVM_ENABLE_UNWIND_TABLES)
+        list(APPEND LLVM_COMPILE_FLAGS "-funwind-tables")
+      else()
         list(APPEND LLVM_COMPILE_FLAGS "-fno-unwind-tables")
         list(APPEND LLVM_COMPILE_FLAGS "-fno-asynchronous-unwind-tables")
       endif()
@@ -2047,13 +2049,19 @@ function(llvm_install_library_symlink name dest type)
   set(full_name ${CMAKE_${type}_LIBRARY_PREFIX}${name}${CMAKE_${type}_LIBRARY_SUFFIX})
   set(full_dest ${CMAKE_${type}_LIBRARY_PREFIX}${dest}${CMAKE_${type}_LIBRARY_SUFFIX})
 
+  if(LLVM_USE_SYMLINKS)
+    set(LLVM_LINK_OR_COPY create_symlink)
+  else()
+    set(LLVM_LINK_OR_COPY copy)
+  endif()
+
   set(output_dir lib${LLVM_LIBDIR_SUFFIX})
   if(WIN32 AND "${type}" STREQUAL "SHARED")
     set(output_dir "${CMAKE_INSTALL_BINDIR}")
   endif()
 
   install(SCRIPT ${INSTALL_SYMLINK}
-          CODE "install_symlink(\"${full_name}\" \"${full_dest}\" \"${output_dir}\")"
+          CODE "install_symlink(\"${full_name}\" \"${full_dest}\" \"${output_dir}\" \"${LLVM_LINK_OR_COPY}\")"
           COMPONENT ${component})
 
 endfunction()
@@ -2090,10 +2098,16 @@ function(llvm_install_symlink project name dest)
     set(full_dest llvm${CMAKE_EXECUTABLE_SUFFIX})
   endif()
 
+  if(LLVM_USE_SYMLINKS)
+    set(LLVM_LINK_OR_COPY create_symlink)
+  else()
+    set(LLVM_LINK_OR_COPY copy)
+  endif()
+
   set(output_dir "${${project}_TOOLS_INSTALL_DIR}")
 
   install(SCRIPT ${INSTALL_SYMLINK}
-          CODE "install_symlink(\"${full_name}\" \"${full_dest}\" \"${output_dir}\")"
+          CODE "install_symlink(\"${full_name}\" \"${full_dest}\" \"${output_dir}\" \"${LLVM_LINK_OR_COPY}\")"
           COMPONENT ${component})
 
   if (NOT LLVM_ENABLE_IDE AND NOT ARG_ALWAYS_GENERATE)
diff --git a/llvm/cmake/modules/HandleLLVMOptions.cmake b/llvm/cmake/modules/HandleLLVMOptions.cmake
index 4c4a1120651bf..578a3169178ee 100644
--- a/llvm/cmake/modules/HandleLLVMOptions.cmake
+++ b/llvm/cmake/modules/HandleLLVMOptions.cmake
@@ -1102,7 +1102,7 @@ if (CLANG_CL AND (LLVM_BUILD_INSTRUMENTED OR LLVM_USE_SANITIZER))
   endif()
   file(TO_CMAKE_PATH "${clang_compiler_rt_file}" clang_compiler_rt_file)
   get_filename_component(clang_runtime_dir "${clang_compiler_rt_file}" DIRECTORY)
-  append("/libpath:${clang_runtime_dir}"
+  append("/libpath:\"${clang_runtime_dir}\""
     CMAKE_EXE_LINKER_FLAGS
     CMAKE_MODULE_LINKER_FLAGS
     CMAKE_SHARED_LINKER_FLAGS)
diff --git a/llvm/cmake/modules/LLVMInstallSymlink.cmake b/llvm/cmake/modules/LLVMInstallSymlink.cmake
index 274cef64b3e79..0ef4b82474ce1 100644
--- a/llvm/cmake/modules/LLVMInstallSymlink.cmake
+++ b/llvm/cmake/modules/LLVMInstallSymlink.cmake
@@ -7,7 +7,10 @@
 set(CMAKE_INSTALL_LIBDIR "lib")
 include(GNUInstallDirs)
 
-function(install_symlink name target outdir)
+function(install_symlink name target outdir link_or_copy)
+  # link_or_copy is the "command" to pass to cmake -E.
+  # It should be either "create_symlink" or "copy".
+
   set(DESTDIR $ENV{DESTDIR})
   if(NOT IS_ABSOLUTE "${outdir}")
     set(outdir "${CMAKE_INSTALL_PREFIX}/${outdir}")
@@ -17,12 +20,7 @@ function(install_symlink name target outdir)
   message(STATUS "Creating ${name}")
 
   execute_process(
-    COMMAND "${CMAKE_COMMAND}" -E create_symlink "${target}" "${name}"
-    WORKING_DIRECTORY "${outdir}" ERROR_VARIABLE has_err)
-  if(CMAKE_HOST_WIN32 AND has_err)
-    execute_process(
-      COMMAND "${CMAKE_COMMAND}" -E copy "${target}" "${name}"
-      WORKING_DIRECTORY "${outdir}")
-  endif()
+    COMMAND "${CMAKE_COMMAND}" -E ${link_or_copy} "${target}" "${name}"
+    WORKING_DIRECTORY "${outdir}")
 
 endfunction()
diff --git a/llvm/docs/CMake.rst b/llvm/docs/CMake.rst
index 7926de258ec80..8b17d4c4406a1 100644
--- a/llvm/docs/CMake.rst
+++ b/llvm/docs/CMake.rst
@@ -191,9 +191,6 @@ used variables that control features of LLVM and enabled subprojects.
 
 **CMAKE_BUILD_TYPE**:STRING
   This configures the optimization level for ``make`` or ``ninja`` builds.
-  The default ``CMAKE_BUILD_TYPE`` is set to ``Debug`` but you should
-  carefully read the list below to figure out what configuration matches
-  your use case the best.
 
   Possible values:
 
diff --git a/llvm/docs/CommandGuide/index.rst b/llvm/docs/CommandGuide/index.rst
index bd3305c5d6720..52d2421678ecc 100644
--- a/llvm/docs/CommandGuide/index.rst
+++ b/llvm/docs/CommandGuide/index.rst
@@ -30,11 +30,13 @@ Basic Commands
    llvm-libtool-darwin
    llvm-link
    llvm-lipo
+   llvm-mc
    llvm-mca
    llvm-opt-report
    llvm-otool
    llvm-profdata
    llvm-readobj
+   llvm-reduce
    llvm-stress
    llvm-symbolizer
    opt
diff --git a/llvm/docs/CommandGuide/llvm-cov.rst b/llvm/docs/CommandGuide/llvm-cov.rst
index c4692b3c1e8f9..5841d4dbc4618 100644
--- a/llvm/docs/CommandGuide/llvm-cov.rst
+++ b/llvm/docs/CommandGuide/llvm-cov.rst
@@ -351,19 +351,20 @@ OPTIONS
 
 .. option:: -debuginfod
 
-Use debuginfod to look up coverage mapping for binary IDs present in the profile
-but not in any object given on the command line. Defaults to true if debuginfod
-is compiled in and configured via the DEBUGINFOD_URLS environment variable.
+ Use debuginfod to look up coverage mapping for binary IDs present in the
+ profile but not in any object given on the command line. Defaults to true if
+ debuginfod is compiled in and configured via the DEBUGINFOD_URLS environment
+ variable.
 
 .. option:: -debug-file-directory=<dir>
 
-Provides local directories to search for objects corresponding to binary IDs in
-the profile (as with debuginfod). Defaults to system build ID directories.
+ Provides local directories to search for objects corresponding to binary IDs in
+ the profile (as with debuginfod). Defaults to system build ID directories.
 
 .. option:: -check-binary-id
 
-Fail if an object file cannot be found for a binary ID present in the profile,
-neither on the command line nor via binary ID lookup.
+ Fail if an object file cannot be found for a binary ID present in the profile,
+ neither on the command line nor via binary ID lookup.
 
 .. program:: llvm-cov report
 
@@ -436,20 +437,20 @@ OPTIONS
 
 .. option:: -debuginfod
 
-Attempt to look up coverage mapping from objects using debuginfod. This is
-attempted by default for binary IDs present in the profile but not provided on
-the command line, so long as debuginfod is compiled in and configured via
-DEBUGINFOD_URLS.
+ Attempt to look up coverage mapping from objects using debuginfod. This is
+ attempted by default for binary IDs present in the profile but not provided on
+ the command line, so long as debuginfod is compiled in and configured via
+ DEBUGINFOD_URLS.
 
 .. option:: -debug-file-directory=<dir>
 
-Provides a directory to search for objects corresponding to binary IDs in the
-profile.
+ Provides a directory to search for objects corresponding to binary IDs in the
+ profile.
 
 .. option:: -check-binary-id
 
-Fail if an object file cannot be found for a binary ID present in the profile,
-neither on the command line nor via binary ID lookup.
+ Fail if an object file cannot be found for a binary ID present in the profile,
+ neither on the command line nor via binary ID lookup.
 
 .. program:: llvm-cov export
 
@@ -528,17 +529,17 @@ OPTIONS
 
 .. option:: -debuginfod
 
-Attempt to look up coverage mapping from objects using debuginfod. This is
-attempted by default for binary IDs present in the profile but not provided on
-the command line, so long as debuginfod is compiled in and configured via
-DEBUGINFOD_URLS.
+ Attempt to look up coverage mapping from objects using debuginfod. This is
+ attempted by default for binary IDs present in the profile but not provided on
+ the command line, so long as debuginfod is compiled in and configured via
+ DEBUGINFOD_URLS.
 
 .. option:: -debug-file-directory=<dir>
 
-Provides a directory to search for objects corresponding to binary IDs in the
-profile.
+ Provides a directory to search for objects corresponding to binary IDs in the
+ profile.
 
 .. option:: -check-binary-id
 
-Fail if an object file cannot be found for a binary ID present in the profile,
-neither on the command line nor via binary ID lookup.
+ Fail if an object file cannot be found for a binary ID present in the profile,
+ neither on the command line nor via binary ID lookup.
diff --git a/llvm/docs/CommandGuide/llvm-mc.rst b/llvm/docs/CommandGuide/llvm-mc.rst
new file mode 100644
index 0000000000000..b7a40a19c67fa
--- /dev/null
+++ b/llvm/docs/CommandGuide/llvm-mc.rst
@@ -0,0 +1,156 @@
+llvm-mc - LLVM Machine Code Playground
+======================================
+
+.. program:: llvm-mc
+
+SYNOPSIS
+--------
+
+:program:`llvm-mc` [*options*] [*filename*]
+
+DESCRIPTION
+-----------
+
+The :program:`llvm-mc` command take input as the assembly code for a 
+specified architecture and generate object file or executable as a output 
+for a specified architecture. 
+
+:program:`llvm-mc` provide powerful set of the tool for working with the machine code such 
+as encoding of their instruction and their internal representation, dissasemble 
+string to bytes etc. 
+
+The choice of architecture for the output assembly code is automatically
+determined from the input file, unless the :option:`--arch` option is used to
+override the default.
+
+OPTIONS
+-------
+
+If the :option:`-o` option is omitted, then :program:`llvm-mc` will send its output
+to standard output if the input is from standard input.  If the :option:`-o`
+option specifies "``-``", then the output will also be sent to standard output.
+
+If no :option:`-o` option is specified and an input file other than "``-``" is
+specified, then :program:`llvm-mc` creates the output filename by taking the input
+filename, removing any existing ``.s`` extension, and adding a ``.o`` suffix.
+
+Other :program:`llvm-mc` options are described below.
+
+End-user Options
+~~~~~~~~~~~~~~~~
+
+.. option:: --help
+
+ Display available options (--help-hidden for more).
+
+.. option:: -o <filename>
+
+ Use ``<filename>`` as the output filename. See the summary above for more
+ details.
+
+.. option:: --arch=<string>
+
+ Target arch to assemble for, see -version for available targets. 
+
+.. option:: --as-lex
+
+ Apply the assemblers "lexer" to break the input into tokens and print each of them out. 
+ This is intended to help develop and test an assembler implementation.
+
+.. option:: --assemble
+
+ Assemble assembly file (default), and print the result to assembly. 
+ This is useful to design and test instruction parsers, and can be a useful tool when combined with other llvm-mc flags. 
+ For example, this option may be useful to transcode assembly from different dialects, e.g. on Intel where you can use 
+ -output-asm-variant=1 to translate from AT&T to Intel assembly syntax. 
+ It can also be combined with --show-encoding to understand how instructions are encoded.
+
+.. option:: --disassemble
+
+ Parse a series of hex bytes, and print the result out as assembly syntax.
+
+
+
+.. option:: --mdis
+
+ Marked up disassembly of string of hex bytes.
+
+.. option:: -g
+
+ Generate DWARF debugging info for assembly source files.
+
+.. option:: --large-code-model
+
+ Create CFI directives that assume the code might be more than 2 GB.
+
+.. option:: --main-file-name=<string>  
+    
+ Specify the name we should consider the input file.
+
+ 
+.. option:: --masm-hexfloats
+    
+ Enable MASM-style hex float initializers (3F800000r).
+
+
+.. option:: -mattr=a1,+a2,-a3,...
+ Target specific attributes (-mattr=help for details).
+
+.. option:: --mcpu=<cpu-name> 
+
+ Target a specific cpu type (-mcpu=help for details).
+
+.. option::   --triple=<string>  
+
+ Target triple to assemble for, see -version for available targets.
+
+.. option::  --split-dwarf-file=<filename> 
+
+ DWO output filename.
+
+.. option:: --show-inst-operands 
+
+ Show instructions operands as parsed.
+
+.. option:: --show-inst
+
+ Show internal instruction representation.
+
+.. option::  --show-encoding   
+
+ Show instruction encodings.
+
+.. option:: --save-temp-labels   
+
+ Don't discard temporary labels.
+
+.. option::   --relax-relocations  
+
+ Emit R_X86_64_GOTPCRELX instead of R_X86_64_GOTPCREL.
+
+.. option:: --print-imm-hex      
+
+ Prefer hex format for immediate values.
+ For example, on x86 targets --output-asm-variant=0 prints in AT&T syntax, and --output-asm-variant=1 prints in 
+ Intel/MASM syntax.
+
+.. option::  --preserve-comments 
+
+ Preserve Comments in outputted assembly.
+
+.. option:: --output-asm-variant=<uint>
+
+ Syntax variant to use for output printing.
+
+
+.. option:: --compress-debug-sections=[none|zlib|zstd]
+
+ Choose DWARF debug sections compression.
+
+
+EXIT STATUS
+-----------
+
+If :program:`llvm-mc` succeeds, it will exit with 0.  Otherwise, if an error
+occurs, it will exit with a non-zero value.
+
diff --git a/llvm/docs/CommandGuide/llvm-reduce.rst b/llvm/docs/CommandGuide/llvm-reduce.rst
new file mode 100644
index 0000000000000..cd49ceba2d6b6
--- /dev/null
+++ b/llvm/docs/CommandGuide/llvm-reduce.rst
@@ -0,0 +1,108 @@
+llvm-reduce - LLVM automatic testcase reducer.
+==============================================
+
+.. program:: llvm-reduce
+
+SYNOPSIS
+--------
+
+:program:`llvm-reduce` [*options*] [*input...*]
+
+DESCRIPTION
+-----------
+
+The :program:`llvm-reduce` tool project that can be used for reducing the size of LLVM test cases.
+It works by removing redundant or unnecessary code from LLVM test cases while still preserving 
+their ability to detect bugs.
+
+If ``input`` is "``-``", :program:`llvm-reduce` reads from standard
+input. Otherwise, it will read from the specified ``filenames``.
+
+LLVM-Reduce is a useful tool for reducing the size and 
+complexity of LLVM test cases, making it easier to identify and debug issues in 
+the LLVM compiler infrastructure.
+
+GENERIC OPTIONS
+---------------
+
+
+.. option:: --help
+
+ Display available options (--help-hidden for more).
+
+.. option:: --abort-on-invalid-reduction
+
+ Abort if any reduction results in invalid IR
+
+.. option::--delta-passes=<string>  
+
+ Delta passes to run, separated by commas. By default, run all delta passes.
+
+
+.. option:: --in-place     
+
+ WARNING: This option will replace your input file with the reduced version!
+
+.. option:: --ir-passes=<string> 
+
+ A textual description of the pass pipeline, same as what's passed to `opt -passes`.
+
+.. option:: -j <uint>  
+
+ Maximum number of threads to use to process chunks. Set to 1 to disable parallelism.
+
+.. option::  --max-pass-iterations=<int>
+
+  Maximum number of times to run the full set of delta passes (default=5).
+
+.. option:: --mtriple=<string> 
+
+ Set the target triple.
+
+.. option:: --preserve-debug-environment
+
+ Don't disable features used for crash debugging (crash reports, llvm-symbolizer and core dumps)
+
+.. option:: --print-delta-passes  
+
+ Print list of delta passes, passable to --delta-passes as a comma separated liste.
+
+.. option:: --skip-delta-passes=<string>     
+
+ Delta passes to not run, separated by commas. By default, run all delta passes.
+
+.. option:: --starting-granularity-level=<uint>
+
+  Number of times to divide chunks prior to first test.
+
+  Note : Granularity refers to the level of detail at which the reduction process operates.
+  A lower granularity means that the reduction process operates at a more coarse-grained level,
+  while a higher granularity means that it operates at a more fine-grained level.
+
+.. option::  --test=<string> 
+
+ Name of the interesting-ness test to be run.
+
+.. option:: --test-arg=<string> 
+
+ Arguments passed onto the interesting-ness test.
+
+.. option:: --verbose    
+
+ Print extra debugging information.
+ 
+.. option::  --write-tmp-files-as-bitcode  
+
+ Always write temporary files as bitcode instead of textual IR.
+
+.. option:: -x={ir|mir}
+
+ Input language as ir or mir.
+
+EXIT STATUS
+------------
+
+:program:`llvm-reduce` returns 0 under normal operation. It returns a non-zero
+exit code if there were any errors.
+
+
diff --git a/llvm/docs/ConvergenceAndUniformity.rst b/llvm/docs/ConvergenceAndUniformity.rst
index c098ef8635257..ae78692b50103 100644
--- a/llvm/docs/ConvergenceAndUniformity.rst
+++ b/llvm/docs/ConvergenceAndUniformity.rst
@@ -393,15 +393,19 @@ instance is determined as follows:
 
 1. The semantics of the instruction may specify the output to be
    uniform.
-2. Otherwise, if it is a PHI node, its output is uniform if and only
-   if for every pair of converged dynamic instances produced by all
-   threads in ``S``:
+2. Otherwise, the output is divergent if the static instance is not
+   :ref:`m-converged <convergence-m-converged>`.
+3. Otherwise, if the static instance is m-converged:
 
-   a. Both instances choose the same output from converged
-      dynamic instances, and,
-   b. That output is uniform for all threads in ``S``.
-3. Otherwise, the output is uniform if and only if the input
-   operands are uniform for all threads in ``S``.
+   1. If it is a PHI node, its output is uniform if and only
+      if for every pair of converged dynamic instances produced by all
+      threads in ``S``:
+
+      a. Both instances choose the same output from converged
+         dynamic instances, and,
+      b. That output is uniform for all threads in ``S``.
+   2. Otherwise, the output is uniform if and only if the input
+      operands are uniform for all threads in ``S``.
 
 Divergent Cycle Exits
 ---------------------
@@ -433,6 +437,8 @@ nodes in irreducible cycles, and any uniformity analysis is limited to
 those static instances whose convergence is independent of the cycle
 hierarchy:
 
+.. _convergence-m-converged:
+
   **m-converged static instances:**
 
   A static instance ``X`` is *m-converged* for a given CFG if and only
@@ -474,9 +480,8 @@ only if:
    if the whole CFG is reducible, then all nodes in the CFG are
    m-converged.
 
-If a static instance is not m-converged, then every output is assumed
-to be divergent. Otherwise, for an m-converged static instance, the
-uniformity of each output is determined using the criteria
+The uniformity of each output of a static instance
+is determined using the criteria
 :ref:`described earlier <convergence-uniformity>`. The discovery of
 divergent outputs may cause their uses (including branches) to also
 become divergent. The analysis propagates this divergence until a
diff --git a/llvm/docs/GettingInvolved.rst b/llvm/docs/GettingInvolved.rst
index 486c6c5cf4246..9894ecc4af88f 100644
--- a/llvm/docs/GettingInvolved.rst
+++ b/llvm/docs/GettingInvolved.rst
@@ -318,11 +318,6 @@ don't find anyone present, chances are they happen to be off that day.
       `gcal <https://calendar.google.com/calendar/embed?src=48c4ad60290a4df218e51e1ceec1106fe317b0ebc76938d9273592053f38204e%40group.calendar.google.com>`__
     - `Google meet <https://meet.google.com/cbz-grrp-obs>`__
     - English, French
-  * - Paulo Matos (he/him)
-    - WebAssembly backend; LLVM IR;
-    - Monthly, 1st Monday of the month at 11:00am Europe/Berlin, for 30 minutes.
-    - `Igalia Jitsi <https://meetings.igalia.com/PauloMatosLLVMOfficeHour>`__
-    - English, Portuguese, German
   * - Phoebe Wang (she/her)
     - X86 backend, General questions to X86, women in compilers.
     - Monthly, 3rd Wednesday of the month at 8:30am Beijing time, for 30 minutes.
diff --git a/llvm/docs/GettingStarted.rst b/llvm/docs/GettingStarted.rst
index f2158a479d4f8..9247b3908aeb8 100644
--- a/llvm/docs/GettingStarted.rst
+++ b/llvm/docs/GettingStarted.rst
@@ -27,28 +27,23 @@ the `LLD linker <https://lld.llvm.org>`_, and more.
 Getting the Source Code and Building LLVM
 =========================================
 
-The LLVM Getting Started documentation may be out of date.  The `Clang
-Getting Started <https://clang.llvm.org/get_started.html>`_ page might have more
-accurate information.
-
-This is an example workflow and configuration to get and build the LLVM source:
-
-#. Checkout LLVM (including related subprojects like Clang):
+#. Check out LLVM (including subprojects like Clang):
 
    * ``git clone https://github.com/llvm/llvm-project.git``
-   * Or, on windows, ``git clone --config core.autocrlf=false
+   * Or, on windows:
+
+     ``git clone --config core.autocrlf=false
      https://github.com/llvm/llvm-project.git``
    * To save storage and speed-up the checkout time, you may want to do a
      `shallow clone <https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---depthltdepthgt>`_.
      For example, to get the latest revision of the LLVM project, use
+
      ``git clone --depth 1 https://github.com/llvm/llvm-project.git``
 
 #. Configure and build LLVM and Clang:
 
    * ``cd llvm-project``
-   * ``mkdir build``
-   * ``cd build``
-   * ``cmake -G <generator> -DCMAKE_BUILD_TYPE=<type> [options] ../llvm``
+   * ``cmake -S llvm -B build -G <generator> [options]``
 
      Some common build system generators are:
 
@@ -59,27 +54,40 @@ This is an example workflow and configuration to get and build the LLVM source:
        solutions.
      * ``Xcode`` --- for generating Xcode projects.
 
-     Some Common options:
+     * See the `CMake docs
+       <https://cmake.org/cmake/help/latest/manual/cmake-generators.7.html>`_
+       for a more comprehensive list.
+
+     Some common options:
 
      * ``-DLLVM_ENABLE_PROJECTS='...'`` --- semicolon-separated list of the LLVM
        subprojects you'd like to additionally build. Can include any of: clang,
-       clang-tools-extra, lldb, compiler-rt, lld, polly, or cross-project-tests.
+       clang-tools-extra, lldb, lld, polly, or cross-project-tests.
 
-       For example, to build LLVM, Clang, libcxx, and libcxxabi, use
-       ``-DLLVM_ENABLE_PROJECTS="clang" -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi"``.
+       For example, to build LLVM, Clang, and LLD, use
+       ``-DLLVM_ENABLE_PROJECTS="clang;lld"``.
 
      * ``-DCMAKE_INSTALL_PREFIX=directory`` --- Specify for *directory* the full
        pathname of where you want the LLVM tools and libraries to be installed
        (default ``/usr/local``).
 
-     * ``-DCMAKE_BUILD_TYPE=type`` --- Controls optimization level and debug information
-       of the build. The default value is ``Debug`` which fits people who want
-       to work on LLVM or its libraries. ``Release`` is a better fit for most
-       users of LLVM and Clang. For more detailed information see
-       :ref:`CMAKE_BUILD_TYPE <cmake_build_type>`.
+     * ``-DCMAKE_BUILD_TYPE=type`` --- Controls optimization level and debug
+       information of the build. Valid options for *type* are ``Debug``,
+       ``Release``, ``RelWithDebInfo``, and ``MinSizeRel``. For more detailed
+       information see :ref:`CMAKE_BUILD_TYPE <cmake_build_type>`.
+
+     * ``-DLLVM_ENABLE_ASSERTIONS=ON`` --- Compile with assertion checks enabled
+       (default is ON for Debug builds, OFF for all other build types).
 
-     * ``-DLLVM_ENABLE_ASSERTIONS=On`` --- Compile with assertion checks enabled
-       (default is Yes for Debug builds, No for all other build types).
+     * ``-DLLVM_USE_LINKER=lld`` --- Link with the `lld linker`_, assuming it
+       is installed on your system. This can dramatically speed up link times
+       if the default linker is slow.
+
+     * ``-DLLVM_PARALLEL_{COMPILE,LINK}_JOBS=N`` --- Limit the number of
+       compile/link jobs running in parallel at the same time. This is
+       especially important for linking since linking can use lots of memory. If
+       you run into memory issues building LLVM, try setting this to limit the
+       maximum number of compile/link jobs running at the same time.
 
    * ``cmake --build . [--target <target>]`` or the build system specified
      above directly.
@@ -98,10 +106,19 @@ This is an example workflow and configuration to get and build the LLVM source:
        option ``-j NN``, where ``NN`` is the number of parallel jobs, e.g. the
        number of available CPUs.
 
-   * For more information see `CMake <CMake.html>`__
+   * A basic CMake and build/test invocation which only builds LLVM and no other
+     subprojects:
+
+     ``cmake -S llvm -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug``
+
+     ``ninja -C build check-llvm``
+
+     This will setup an LLVM build with debugging info, then compile LLVM and
+     run LLVM tests.
+
+   * For more detailed information on CMake options, see `CMake <CMake.html>`__
 
-   * If you get an "internal compiler error (ICE)" or test failures, see
-     `below`_.
+   * If you get build or test failures, see `below`_.
 
 Consult the `Getting Started with LLVM`_ section for detailed information on
 configuring and compiling LLVM.  Go to `Directory Layout`_ to learn about the
@@ -601,8 +618,11 @@ used by people developing LLVM.
 |                         | out-of-tree targets. The default value includes:   |
 |                         | ``AArch64, AMDGPU, ARM, AVR, BPF, Hexagon, Lanai,  |
 |                         | Mips, MSP430, NVPTX, PowerPC, RISCV, Sparc,        |
-|                         | SystemZ, WebAssembly, X86, XCore``.                |
-|                         |                                                    |
+|                         | SystemZ, WebAssembly, X86, XCore``. Setting this   |
+|                         | to ``"host"`` will only compile the host           |
+|                         | architecture (e.g. equivalent to specifying ``X86``|
+|                         | on an x86 host machine) can                        |
+|                         | significantly speed up compile and test times.     |
 +-------------------------+----------------------------------------------------+
 | LLVM_ENABLE_DOXYGEN     | Build doxygen-based documentation from the source  |
 |                         | code This is disabled by default because it is     |
diff --git a/llvm/docs/HowToAddABuilder.rst b/llvm/docs/HowToAddABuilder.rst
index a8794707ccdb6..76763cd1f8c75 100644
--- a/llvm/docs/HowToAddABuilder.rst
+++ b/llvm/docs/HowToAddABuilder.rst
@@ -18,7 +18,8 @@ There are two buildmasters running.
   the build.
 * The staging buildmaster at `<https://lab.llvm.org/staging>`_. All builders
   attached to this machine will be completely silent by default when the build
-  is broken.
+  is broken. This buildmaster is reconfigured every two hours with any new
+  commits from the llvm-zorg repository.
 
 In order to remain connected to the main buildmaster (and thus notify
 developers of failures), a builbot must:
diff --git a/llvm/docs/JITLink.rst b/llvm/docs/JITLink.rst
index 55425eeb08ab4..8316892ba7945 100644
--- a/llvm/docs/JITLink.rst
+++ b/llvm/docs/JITLink.rst
@@ -419,7 +419,12 @@ Generic Link Algorithm
 ======================
 
 JITLink provides a generic link algorithm which can be extended / modified at
-certain points by the introduction of JITLink :ref:`passes`:
+certain points by the introduction of JITLink :ref:`passes`.
+
+At the end of each phase the linker packages its state into a *continuation*
+and calls the ``JITLinkContext`` object to perform a (potentialy high-latency)
+asynchronous operation: allocating memory, resolving external symbols, and
+finally transferring linked memory to the executing process.
 
 #. Phase 1
 
@@ -457,21 +462,18 @@ certain points by the introduction of JITLink :ref:`passes`:
       Notable use cases: Building Global Offset Table (GOT), Procedure Linkage
       Table (PLT), and Thread Local Variable (TLV) entries.
 
-   #. Sort blocks into segments.
-
-      Sorts all blocks by ordinal and then address. Collects sections with
-      matching permissions into segments and computes the size of these
-      segments for memory allocation.
-
-   #. Allocate segment memory, update node addresses.
+   #. Asynchronously allocate memory.
 
       Calls the ``JITLinkContext``'s ``JITLinkMemoryManager`` to allocate both
-      working and target memory for the graph, then updates all node addresses
-      to their assigned target address.
+      working and target memory for the graph. As part of this process the
+      ``JITLinkMemoryManager`` will update the the addresses of all nodes
+      defined in the graph to their assigned target address.
 
       Note: This step only updates the addresses of nodes defined in this graph.
       External symbols will still have null addresses.
 
+#. Phase 2
+
    #. Run post-allocation passes.
 
       These passes are run on the graph after working and target memory have
@@ -497,15 +499,9 @@ certain points by the introduction of JITLink :ref:`passes`:
    #. Identify external symbols and resolve their addresses asynchronously.
 
       Calls the ``JITLinkContext`` to resolve the target address of any external
-      symbols in the graph. This step is asynchronous -- JITLink will pack the
-      link state into a *continuation* to be run once the symbols are resolved.
+      symbols in the graph.
 
-      This is the final step of Phase 1.
-
-#. Phase 2
-
-   This phase is called by the continuation constructed at the end of the
-   external symbol resolution step above.
+#. Phase 3
 
    #. Apply external symbol resolution results.
 
@@ -541,21 +537,14 @@ certain points by the introduction of JITLink :ref:`passes`:
    #. Finalize memory asynchronously.
 
       Calls the ``JITLinkMemoryManager`` to copy working memory to the executor
-      process and apply the requested permissions. This step is asynchronous --
-      JITLink will pack the link state into a *continuation* to be run once
-      memory has been copied and protected.
-
-      This is the final step of Phase 2.
+      process and apply the requested permissions.
 
 #. Phase 3.
 
-   This phase is called by the continuation constructed at the end of the
-   memory finalization step above.
-
    #. Notify the context that the graph has been emitted.
 
       Calls ``JITLinkContext::notifyFinalized`` and hands off the
-      ``JITLinkMemoryManager::Allocation`` object for this graph's memory
+      ``JITLinkMemoryManager::FinalizedAlloc`` object for this graph's memory
       allocation. This allows the context to track/hold memory allocations and
       react to the newly emitted definitions. In ORC this is used to update the
       ``ExecutionSession`` instance's dependence graph, which may result in
@@ -664,58 +653,84 @@ the common case of running across processes on the same machine for security)
 via the host operating system's virtual memory management APIs.
 
 To satisfy these requirements ``JITLinkMemoryManager`` adopts the following
-design: The memory manager itself has just one virtual method that returns a
-``JITLinkMemoryManager::Allocation``:
+design: The memory manager itself has just two virtual methods for asynchronous
+operations (each with convenience overloads for calling synchronously):
 
 .. code-block:: c++
 
-  virtual Expected<std::unique_ptr<Allocation>>
-  allocate(const JITLinkDylib *JD, const SegmentsRequestMap &Request) = 0;
+  /// Called when allocation has been completed.
+  using OnAllocatedFunction =
+    unique_function<void(Expected<std::unique_ptr<InFlightAlloc>)>;
+
+  /// Called when deallocation has completed.
+  using OnDeallocatedFunction = unique_function<void(Error)>;
+
+  /// Call to allocate memory.
+  virtual void allocate(const JITLinkDylib *JD, LinkGraph &G,
+                        OnAllocatedFunction OnAllocated) = 0;
+
+  /// Call to deallocate memory.
+  virtual void deallocate(std::vector<FinalizedAlloc> Allocs,
+                          OnDeallocatedFunction OnDeallocated) = 0;
 
-This method takes a ``JITLinkDylib*`` representing the target simulated
-dylib, and the full set of sections that must be allocated for this object.
+The ``allocate`` method takes a ``JITLinkDylib*`` representing the target
+simulated dylib, a reference to the ``LinkGraph`` that must be allocated for,
+and a callback to run once an ``InFlightAlloc`` has been constructed.
 ``JITLinkMemoryManager`` implementations can (optionally) use the ``JD``
 argument to manage a per-simulated-dylib memory pool (since code model
 constraints are typically imposed on a per-dylib basis, and not across
-dylibs) [2]_. The ``Request`` argument, by describing all sections in the current
-object up-front, allows the implementer to allocate those sections as a
-single slab, either within a pre-allocated per-jitdylib pool or directly
-from system memory.
+dylibs) [2]_. The ``LinkGraph`` describes the object file that we need to
+allocate memory for. The allocator must allocate working memory for all of
+the Blocks defined in the graph, assign address space for each Block within the
+executing processes memory, and update the Blocks' addresses to reflect this
+assignment. Block content should be copied to working memory, but does not need
+to be transferred to executor memory yet (that will be done once the content is
+fixed up). ``JITLinkMemoryManager`` implementations can take full
+responsibility for these steps, or use the ``BasicLayout`` utility to reduce
+the task to allocating working and executor memory for *segments*: chunks of
+memory defined by permissions, alignments, content sizes, and zero-fill sizes.
+Once the allocation step is complete the memory manager should construct an
+``InFlightAlloc`` object to represent the allocation, and then pass this object
+to the ``OnAllocated`` callback.
+
+The ``InFlightAlloc`` object has two virtual methods:
 
-All subsequent operations are provided by the
-``JITLinkMemoryManager::Allocation`` interface:
-
-* ``virtual MutableArrayRef<char> getWorkingMemory(ProtectionFlags Seg)``
-
-  Should be overridden to return the address in working memory of the segment
-  with the given protection flags.
+.. code-block:: c++
 
-* ``virtual JITTargetAddress getTargetMemory(ProtectionFlags Seg)``
+    using OnFinalizedFunction = unique_function<void(Expected<FinalizedAlloc>)>;
+    using OnAbandonedFunction = unique_function<void(Error)>;
 
-  Should be overridden to return the address in the executor's address space of
-  the segment with the given protection flags.
+    /// Called prior to finalization if the allocation should be abandoned.
+    virtual void abandon(OnAbandonedFunction OnAbandoned) = 0;
 
-* ``virtual void finalizeAsync(FinalizeContinuation OnFinalize)``
+    /// Called to transfer working memory to the target and apply finalization.
+    virtual void finalize(OnFinalizedFunction OnFinalized) = 0;
 
-  Should be overridden to copy the contents of working memory to the target
-  address space and apply memory protections for all segments. Where working
-  memory and target memory are separate, this method should deallocate the
-  working memory.
+The linking process will call the ``finalize`` method on the ``InFlightAlloc``
+object if linking succeeds up to the finalization step, otherwise it will call
+``abandon`` to indicate that some error occurred during linking. A call to the
+``InFlightAlloc::finalize`` method should cause content for the allocation to be
+transferred from working to executor memory, and permissions to be run. A call
+to ``abandon`` should result in both kinds of memory being deallocated.
 
-* ``virtual Error deallocate()``
+On successful finalization, the ``InFlightAlloc::finalize`` method should
+construct a ``FinalizedAlloc`` object (an opaque uint64_t id that the
+``JITLinkMemoryManager`` can use to identify executor memory for deallocation)
+and pass it to the ``OnFinalized`` callback.
 
-  Should be overridden to deallocate memory in the target address space.
+Finalized allocations (represented by ``FinalizedAlloc`` objects) can be
+deallocated by calling the ``JITLinkMemoryManager::dealloc`` method. This method
+takes a vector of ``FinalizedAlloc`` objects, since it is common to deallocate
+multiple objects at the same time and this allows us to batch these requsets for
+transmission to the executing process.
 
 JITLink provides a simple in-process implementation of this interface:
 ``InProcessMemoryManager``. It allocates pages once and re-uses them as both
 working and target memory.
 
-ORC provides a cross-process ``JITLinkMemoryManager`` based on an ORC-RPC-based
-implementation of the ``orc::TargetProcessControl`` API:
-``OrcRPCTPCJITLinkMemoryManager``. This API uses TargetProcessControl API calls
-to allocate and manage memory in a remote process. The underlying communication
-channel is determined by the ORC-RPC channel type. Common options include unix
-sockets or TCP.
+ORC provides a cross-process-capable ``MapperJITLinkMemoryManager`` that can use
+shared memory or ORC-RPC-based communication to transfer content to the executing
+process.
 
 JITLinkMemoryManager and Security
 ---------------------------------
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 7f366d54a9fb4..3aa3e2e027fad 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -2474,8 +2474,8 @@ operand bundle to not miscompile programs containing it.
 - The bundle operands for an unknown operand bundle escape in unknown
   ways before control is transferred to the callee or invokee.
 - Calls and invokes with operand bundles have unknown read / write
-  effect on the heap on entry and exit (even if the call target is
-  ``readnone`` or ``readonly``), unless they're overridden with
+  effect on the heap on entry and exit (even if the call target specifies
+  a ``memory`` attribute), unless they're overridden with
   callsite specific attributes.
 - An operand bundle at a call site cannot change the implementation
   of the called function.  Inter-procedural optimizations work as
@@ -6472,14 +6472,14 @@ representing the maximum relative error, for example:
 ^^^^^^^^^^^^^^^^^^^^
 
 ``range`` metadata may be attached only to ``load``, ``call`` and ``invoke`` of
-integer types. It expresses the possible ranges the loaded value or the value
-returned by the called function at this call site is in. If the loaded or
-returned value is not in the specified range, a poison value is returned
-instead. The ranges are represented with a flattened list of integers. The
-loaded value or the value returned is known to be in the union of the ranges
+integer or vector of integer types. It expresses the possible ranges the loaded
+value or the value returned by the called function at this call site is in. If
+the loaded or returned value is not in the specified range, a poison value is
+returned instead. The ranges are represented with a flattened list of integers.
+The loaded value or the value returned is known to be in the union of the ranges
 defined by each consecutive pair. Each pair has the following properties:
 
--  The type must match the type loaded by the instruction.
+-  The type must match the scalar type of the instruction.
 -  The pair ``a,b`` represents the range ``[a,b)``.
 -  Both ``a`` and ``b`` are constants.
 -  The range is allowed to wrap.
@@ -6489,6 +6489,8 @@ defined by each consecutive pair. Each pair has the following properties:
 In addition, the pairs must be in signed order of the lower bound and
 they must be non-contiguous.
 
+For vector-typed instructions, the range is applied element-wise.
+
 Examples:
 
 .. code-block:: llvm
@@ -6498,6 +6500,7 @@ Examples:
       %c = call i8 @foo(),       !range !2 ; Can only be 0, 1, 3, 4 or 5
       %d = invoke i8 @bar() to label %cont
              unwind label %lpad, !range !3 ; Can only be -2, -1, 3, 4 or 5
+      %e = load <2 x i8>, ptr %x, !range 0 ; Can only be <0 or 1, 0 or 1>
     ...
     !0 = !{ i8 0, i8 2 }
     !1 = !{ i8 255, i8 2 }
@@ -7456,6 +7459,72 @@ Example:
 Clang emits ``kcfi_type`` metadata nodes for address-taken functions with
 ``-fsanitize=kcfi``.
 
+.. _md_memprof:
+
+'``memprof``' Metadata
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``memprof`` metadata is used to record memory profile data on heap
+allocation calls. Multiple context-sensitive profiles can be represented
+with a single ``memprof`` metadata attachment.
+
+Example:
+
+.. code-block:: text
+
+    %call = call ptr @_Znam(i64 10), !memprof !0, !callsite !5
+    !0 = !{!1, !3}
+    !1 = !{!2, !"cold"}
+    !2 = !{i64 4854880825882961848, i64 1905834578520680781}
+    !3 = !{!4, !"notcold"}
+    !4 = !{i64 4854880825882961848, i64 -6528110295079665978}
+    !5 = !{i64 4854880825882961848}
+
+Each operand in the ``memprof`` metadata attachment describes the profiled
+behavior of memory allocated by the associated allocation for a given context.
+In the above example, there were 2 profiled contexts, one allocating memory
+that was typically cold and one allocating memory that was typically not cold.
+
+The format of the metadata describing a context specific profile (e.g.
+``!1`` and ``!3`` above) requires a first operand that is a metadata node
+describing the context, followed by a list of string metadata tags describing
+the profile behavior (e.g. ``cold`` and ``notcold``) above. The metadata nodes
+describing the context (e.g. ``!2`` and ``!4`` above) are unique ids
+corresponding to callsites, which can be matched to associated IR calls via
+:ref:`callsite metadata<md_callsite>`. In practice these ids are formed via
+a hash of the callsite's debug info, and the associated call may be in a
+different module. The contexts are listed in order from leaf-most call (the
+allocation itself) to the outermost callsite context required for uniquely
+identifying the described profile behavior (note this may not be the top of
+the profiled call stack).
+
+.. _md_callsite:
+
+'``callsite``' Metadata
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``callsite`` metadata is used to identify callsites involved in memory
+profile contexts described in :ref:`memprof metadata<md_memprof>`.
+
+It is attached both to the profile allocation calls (see the example in
+:ref:`memprof metadata<md_memprof>`), as well as to other callsites
+in profiled contexts described in heap allocation ``memprof`` metadata.
+
+Example:
+
+.. code-block:: text
+
+    %call = call ptr @_Z1Bb(void), !callsite !0
+    !0 = !{i64 -6528110295079665978, i64 5462047985461644151}
+
+Each operand in the ``callsite`` metadata attachment is a unique id
+corresponding to a callsite (possibly inlined). In practice these ids are
+formed via a hash of the callsite's debug info. If the call was not inlined
+into any callers it will contain a single operand (id). If it was inlined
+it will contain a list of ids, including the ids of the callsites in the
+full inline sequence, in order from the leaf-most call's id to the outermost
+inlined call.
+
 Module Flags Metadata
 =====================
 
@@ -10758,14 +10827,14 @@ to LLVM:
       return &s[1].Z.B[5][13];
     }
 
-The LLVM code generated by Clang is:
+The LLVM code generated by Clang is approximately:
 
 .. code-block:: llvm
 
     %struct.RT = type { i8, [10 x [20 x i32]], i8 }
     %struct.ST = type { i32, double, %struct.RT }
 
-    define ptr @foo(ptr %s) nounwind uwtable readnone optsize ssp {
+    define ptr @foo(ptr %s) {
     entry:
       %arrayidx = getelementptr inbounds %struct.ST, ptr %s, i64 1, i32 2, i32 1, i64 5, i64 13
       ret ptr %arrayidx
@@ -12885,7 +12954,7 @@ Syntax:
       declare <pointer type>
         @llvm.experimental.gc.get.pointer.base(
           <pointer type> readnone nocapture %derived_ptr)
-          nounwind readnone willreturn
+          nounwind willreturn memory(none)
 
 Overview:
 """""""""
@@ -12923,7 +12992,7 @@ Syntax:
       declare i64
         @llvm.experimental.gc.get.pointer.offset(
           <pointer type> readnone nocapture %derived_ptr)
-          nounwind readnone willreturn
+          nounwind willreturn memory(none)
 
 Overview:
 """""""""
@@ -25592,7 +25661,7 @@ Syntax:
 
 ::
 
-      declare type @llvm.ssa.copy(type %operand) returned(1) readnone
+      declare type @llvm.ssa.copy(type returned %operand) memory(none)
 
 Arguments:
 """"""""""
@@ -25619,7 +25688,7 @@ Syntax:
 
 ::
 
-      declare i1 @llvm.type.test(ptr %ptr, metadata %type) nounwind readnone
+      declare i1 @llvm.type.test(ptr %ptr, metadata %type) nounwind memory(none)
 
 
 Arguments:
@@ -25644,7 +25713,7 @@ Syntax:
 
 ::
 
-      declare {ptr, i1} @llvm.type.checked.load(ptr %ptr, i32 %offset, metadata %type) argmemonly nounwind readonly
+      declare {ptr, i1} @llvm.type.checked.load(ptr %ptr, i32 %offset, metadata %type) nounwind memory(argmem: read)
 
 
 Arguments:
@@ -25734,7 +25803,7 @@ Syntax:
 
 ::
 
-      declare void @llvm.donothing() nounwind readnone
+      declare void @llvm.donothing() nounwind memory(none)
 
 Overview:
 """""""""
@@ -26030,7 +26099,7 @@ Syntax:
 
 ::
 
-      declare ptr @llvm.load.relative.iN(ptr %ptr, iN %offset) argmemonly nounwind readonly
+      declare ptr @llvm.load.relative.iN(ptr %ptr, iN %offset) nounwind memory(argmem: read)
 
 Overview:
 """""""""
@@ -26089,9 +26158,9 @@ This is an overloaded intrinsic. You can use llvm.is.constant with any argument
 
 ::
 
-      declare i1 @llvm.is.constant.i32(i32 %operand) nounwind readnone
-      declare i1 @llvm.is.constant.f32(float %operand) nounwind readnone
-      declare i1 @llvm.is.constant.TYPENAME(TYPE %operand) nounwind readnone
+      declare i1 @llvm.is.constant.i32(i32 %operand) nounwind memory(none)
+      declare i1 @llvm.is.constant.f32(float %operand) nounwind memory(none)
+      declare i1 @llvm.is.constant.TYPENAME(TYPE %operand) nounwind memory(none)
 
 Overview:
 """""""""
@@ -26134,7 +26203,7 @@ Syntax:
 
 ::
 
-      declare ptrty llvm.ptrmask(ptrty %ptr, intty %mask) readnone speculatable
+      declare ptrty llvm.ptrmask(ptrty %ptr, intty %mask) speculatable memory(none)
 
 Arguments:
 """"""""""
@@ -26170,7 +26239,7 @@ Syntax:
 
 ::
 
-      declare ptr @llvm.threadlocal.address(ptr) nounwind readnone willreturn
+      declare ptr @llvm.threadlocal.address(ptr) nounwind willreturn memory(none)
 
 Arguments:
 """"""""""
diff --git a/llvm/docs/Lexicon.rst b/llvm/docs/Lexicon.rst
index ea2f85967b3ea..80454191b6cc2 100644
--- a/llvm/docs/Lexicon.rst
+++ b/llvm/docs/Lexicon.rst
@@ -232,6 +232,9 @@ O
 P
 -
 
+**PGO**
+    Profile-Guided Optimization
+
 **PR**
     Problem report. A bug filed on `the LLVM Bug Tracking System
     <https://bugs.llvm.org/enter_bug.cgi>`_.
diff --git a/llvm/docs/LoopTerminology.rst b/llvm/docs/LoopTerminology.rst
index acc0475e5bb1b..3d3f2573e1403 100644
--- a/llvm/docs/LoopTerminology.rst
+++ b/llvm/docs/LoopTerminology.rst
@@ -43,7 +43,7 @@ The definition of a loop comes with some additional terminology:
 
 * An **entering block** (or **loop predecessor**) is a non-loop node
   that has an edge into the loop (necessarily the header). If there is
-  only one entering block entering block, and its only edge is to the
+  only one entering block, and its only edge is to the
   header, it is also called the loop's **preheader**. The preheader
   dominates the loop without itself being part of the loop.
 
diff --git a/llvm/docs/NewPassManager.rst b/llvm/docs/NewPassManager.rst
index a9fa72cf6eb83..7464110daa092 100644
--- a/llvm/docs/NewPassManager.rst
+++ b/llvm/docs/NewPassManager.rst
@@ -92,9 +92,9 @@ Loop, where going through a CGSCC is optional.
   MPM.addPass(createModuleToFunctionPassAdaptor(FunctionFooPass()));
 
   // loop -> function -> cgscc -> module
-  MPM.addPass(createModuleToCGSCCPassAdaptor(createCGSCCToFunctionPassAdaptor(createFunctionToLoopPassAdaptor(LoopFooPass()))));
+  MPM.addPass(createModuleToPostOrderCGSCCPassAdaptor(createCGSCCToFunctionPassAdaptor(createFunctionToLoopPassAdaptor(LoopFooPass()))));
   // function -> cgscc -> module
-  MPM.addPass(createModuleToCGSCCPassAdaptor(createCGSCCToFunctionPassAdaptor(FunctionFooPass())));
+  MPM.addPass(createModuleToPostOrderCGSCCPassAdaptor(createCGSCCToFunctionPassAdaptor(FunctionFooPass())));
 
 
 A pass manager of a specific IR unit is also a pass of that kind. For
@@ -386,7 +386,7 @@ checked if they are invalidated:
 .. code-block:: c++
 
   bool FooAnalysisResult::invalidate(Function &F, const PreservedAnalyses &PA,
-                                     FunctionAnalysisManager::Invalidator &) {
+                                     FunctionAnalysisManager::Invalidator &Inv) {
     auto PAC = PA.getChecker<FooAnalysis>();
     if (!PAC.preserved() && !PAC.preservedSet<AllAnalysesOn<Function>>())
       return true;
@@ -420,17 +420,6 @@ for more details.
 Invoking ``opt``
 ================
 
-To use the legacy pass manager:
-
-.. code-block:: shell
-
-  $ opt -enable-new-pm=0 -pass1 -pass2 /tmp/a.ll -S
-
-This will be removed once the legacy pass manager is deprecated and removed for
-the optimization pipeline.
-
-To use the new PM:
-
 .. code-block:: shell
 
   $ opt -passes='pass1,pass2' /tmp/a.ll -S
diff --git a/llvm/docs/ReleaseNotes.rst b/llvm/docs/ReleaseNotes.rst
index 10ffc60e8e8ea..9d7b1dbd79ef1 100644
--- a/llvm/docs/ReleaseNotes.rst
+++ b/llvm/docs/ReleaseNotes.rst
@@ -56,6 +56,20 @@ Changes to the LLVM IR
 * The ``nofpclass`` attribute was introduced. This allows more
   optimizations around special floating point value comparisons.
 
+* The constant expression variants of the following instructions have been
+  removed:
+
+  * ``select``
+
+Changes to LLVM infrastructure
+------------------------------
+
+* The legacy optimization pipeline has been removed.
+
+* Alloca merging in the inliner has been removed, since it only worked with the
+  legacy inliner pass. Backend stack coloring should handle cases alloca
+  merging initially set out to handle.
+
 Changes to building LLVM
 ------------------------
 
@@ -70,6 +84,12 @@ Changes to the AArch64 Backend
 
 Changes to the AMDGPU Backend
 -----------------------------
+* More fine-grained synchronization around barriers for newer architectures
+  (gfx90a+, gfx10+). The AMDGPU backend now omits previously automatically
+  generated waitcnt instructions before barriers, allowing for more precise
+  control. Users must now use memory fences to implement fine-grained
+  synchronization strategies around barriers. Refer to `AMDGPU memory model
+  <AMDGPUUsage.html#memory-model>`__.
 
 Changes to the ARM Backend
 --------------------------
@@ -147,6 +167,16 @@ Changes to the C API
   pointer, has been removed.
 * Functions for adding legacy passes like ``LLVMAddInstructionCombiningPass``
   have been removed.
+* Removed ``LLVMPassManagerBuilderRef`` and functions interacting with it.
+  These belonged to the no longer supported legacy pass manager.
+* As part of the opaque pointer transition, ``LLVMGetElementType`` no longer
+  gives the pointee type of a pointer type.
+* The following functions for creating constant expressions have been removed,
+  because the underlying constant expressions are no longer supported. Instead,
+  an instruction should be created using the ``LLVMBuildXYZ`` APIs, which will
+  constant fold the operands if possible and create an instruction otherwise:
+
+  * ``LLVMConstSelect``
 
 Changes to the FastISel infrastructure
 --------------------------------------
@@ -181,12 +211,15 @@ Changes to the LLVM tools
 ---------------------------------
 * llvm-lib now supports the /def option for generating a Windows import library from a definition file.
 
+* Made significant changes to JSON output format of `llvm-readobj`/`llvm-readelf`
+  to improve correctness and clarity.
+
 Changes to LLDB
 ---------------------------------
 
-* In the results of commands such as `expr` and `frame var`, type summaries will now
+* In the results of commands such as ``expr`` and ``frame var``, type summaries will now
   omit defaulted template parameters. The full template parameter list can still be
-  viewed with `expr --raw-output`/`frame var --raw-output`. (`D141828 <https://reviews.llvm.org/D141828>`_)
+  viewed with ``expr --raw-output``/``frame var --raw-output``. (`D141828 <https://reviews.llvm.org/D141828>`_)
 
 Changes to Sanitizers
 ---------------------
diff --git a/llvm/docs/SPIRVUsage.rst b/llvm/docs/SPIRVUsage.rst
index 0f75e001e661e..5ca035b68392f 100644
--- a/llvm/docs/SPIRVUsage.rst
+++ b/llvm/docs/SPIRVUsage.rst
@@ -75,3 +75,36 @@ to specify the target triple:
 Example:
 
 ``-target spirv64v1.0`` can be used to compile for SPIR-V version 1.0 with 64-bit pointer width.
+
+.. _spirv-types:
+
+Representing special types in SPIR-V
+====================================
+
+SPIR-V specifies several kinds of opaque types. These types are represented
+using target extension types. These types are represented as follows:
+
+  .. table:: SPIR-V Opaque Types
+
+     ================== ====================== =========================================================================================
+     SPIR-V Type        LLVM type name         LLVM type arguments
+     ================== ====================== =========================================================================================
+     OpTypeImage        ``spirv.Image``        sampled type, dimensionality, depth, arrayed, MS, sampled, image format, access qualifier
+     OpTypeSampler      ``spirv.Sampler``      (none)
+     OpTypeSampledImage ``spirv.SampledImage`` sampled type, dimensionality, depth, arrayed, MS, sampled, image format, access qualifier
+     OpTypeEvent        ``spirv.Event``        (none)
+     OpTypeDeviceEvent  ``spirv.DeviceEvent``  (none)
+     OpTypeReserveId    ``spirv.ReserveId``    (none)
+     OpTypeQueue        ``spirv.Queue``        (none)
+     OpTypePipe         ``spirv.Pipe``         access qualifier
+     OpTypePipeStorage  ``spirv.PipeStorage``  (none)
+     ================== ====================== =========================================================================================
+
+All integer arguments take the same value as they do in their `corresponding
+SPIR-V instruction <https://registry.khronos.org/SPIR-V/specs/unified1/SPIRV.html#_type_declaration_instructions>`_.
+For example, the OpenCL type ``image2d_depth_ro_t`` would be represented in
+SPIR-V IR as ``target("spirv.Image", void, 1, 1, 0, 0, 0, 0, 0)``, with its
+dimensionality parameter as ``1`` meaning 2D. Sampled image types include the
+parameters of its underlying image type, so that a sampled image for the
+previous type has the representation
+``target("spirv.SampledImage, void, 1, 1, 0, 0, 0, 0, 0)``.
diff --git a/llvm/docs/TableGen/ProgRef.rst b/llvm/docs/TableGen/ProgRef.rst
index 16d24a92ab1d8..8c74831505633 100644
--- a/llvm/docs/TableGen/ProgRef.rst
+++ b/llvm/docs/TableGen/ProgRef.rst
@@ -1630,6 +1630,12 @@ and non-0 as true.
     This operator produces 1 if the string, list, or DAG *a* is empty; 0 otherwise.
     A dag is empty if it has no arguments; the operator does not count.
 
+``!tolower(``\ *a*\ ``)``
+  This operator converts a string input *a* to lower case.
+
+``!toupper(``\ *a*\ ``)``
+  This operator converts a string input *a* to upper case.
+
 ``!eq(`` *a*\ `,` *b*\ ``)``
     This operator produces 1 if *a* is equal to *b*; 0 otherwise.
     The arguments must be ``bit``, ``bits``, ``int``, ``string``, or
diff --git a/llvm/docs/WritingAnLLVMPass.rst b/llvm/docs/WritingAnLLVMPass.rst
index 771185614a49f..3e6063e583343 100644
--- a/llvm/docs/WritingAnLLVMPass.rst
+++ b/llvm/docs/WritingAnLLVMPass.rst
@@ -36,11 +36,10 @@ advanced features are discussed.
 
 .. warning::
   This document deals with the legacy pass manager. LLVM uses the new pass
-  manager by default for the optimization pipeline (the codegen pipeline is
-  still using the legacy pass manager), which has its own way of defining
+  manager for the optimization pipeline (the codegen pipeline
+  still uses the legacy pass manager), which has its own way of defining
   passes. For more details, see :doc:`WritingAnLLVMNewPMPass` and
-  :doc:`NewPassManager`. To use the legacy pass manager with ``opt``, pass
-  the ``-enable-new-pm=0`` flag to all ``opt`` invocations.
+  :doc:`NewPassManager`.
 
 Quick Start --- Writing hello world
 ===================================
@@ -194,7 +193,6 @@ As a whole, the ``.cpp`` file looks like:
   #include "llvm/Support/raw_ostream.h"
 
   #include "llvm/IR/LegacyPassManager.h"
-  #include "llvm/Transforms/IPO/PassManagerBuilder.h"
 
   using namespace llvm;
 
diff --git a/llvm/examples/OrcV2Examples/LLJITWithOptimizingIRTransform/LLJITWithOptimizingIRTransform.cpp b/llvm/examples/OrcV2Examples/LLJITWithOptimizingIRTransform/LLJITWithOptimizingIRTransform.cpp
index a5fd384b65b54..069c8e8e80316 100644
--- a/llvm/examples/OrcV2Examples/LLJITWithOptimizingIRTransform/LLJITWithOptimizingIRTransform.cpp
+++ b/llvm/examples/OrcV2Examples/LLJITWithOptimizingIRTransform/LLJITWithOptimizingIRTransform.cpp
@@ -74,7 +74,6 @@ class MyOptimizationTransform {
 public:
   MyOptimizationTransform() : PM(std::make_unique<legacy::PassManager>()) {
     PM->add(createTailCallEliminationPass());
-    PM->add(createIndVarSimplifyPass());
     PM->add(createCFGSimplificationPass());
   }
 
diff --git a/llvm/include/llvm-c/Core.h b/llvm/include/llvm-c/Core.h
index df9db59503c3b..f2959ac8d22da 100644
--- a/llvm/include/llvm-c/Core.h
+++ b/llvm/include/llvm-c/Core.h
@@ -1411,8 +1411,6 @@ LLVMBool LLVMIsLiteralStruct(LLVMTypeRef StructTy);
 /**
  * Obtain the element type of an array or vector type.
  *
- * This currently also works for pointer types, but this usage is deprecated.
- *
  * @see llvm::SequentialType::getElementType()
  */
 LLVMTypeRef LLVMGetElementType(LLVMTypeRef Ty);
@@ -2259,9 +2257,6 @@ LLVMValueRef LLVMConstPointerCast(LLVMValueRef ConstantVal,
 LLVMValueRef LLVMConstIntCast(LLVMValueRef ConstantVal, LLVMTypeRef ToType,
                               LLVMBool isSigned);
 LLVMValueRef LLVMConstFPCast(LLVMValueRef ConstantVal, LLVMTypeRef ToType);
-LLVMValueRef LLVMConstSelect(LLVMValueRef ConstantCondition,
-                             LLVMValueRef ConstantIfTrue,
-                             LLVMValueRef ConstantIfFalse);
 LLVMValueRef LLVMConstExtractElement(LLVMValueRef VectorConstant,
                                      LLVMValueRef IndexConstant);
 LLVMValueRef LLVMConstInsertElement(LLVMValueRef VectorConstant,
diff --git a/llvm/include/llvm-c/Transforms/PassManagerBuilder.h b/llvm/include/llvm-c/Transforms/PassManagerBuilder.h
deleted file mode 100644
index 3ba75440129ae..0000000000000
--- a/llvm/include/llvm-c/Transforms/PassManagerBuilder.h
+++ /dev/null
@@ -1,81 +0,0 @@
-/*===-- llvm-c/Transform/PassManagerBuilder.h - PMB C Interface ---*- C -*-===*\
-|*                                                                            *|
-|* Part of the LLVM Project, under the Apache License v2.0 with LLVM          *|
-|* Exceptions.                                                                *|
-|* See https://llvm.org/LICENSE.txt for license information.                  *|
-|* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception                    *|
-|*                                                                            *|
-|*===----------------------------------------------------------------------===*|
-|*                                                                            *|
-|* This header declares the C interface to the PassManagerBuilder class.      *|
-|*                                                                            *|
-\*===----------------------------------------------------------------------===*/
-
-#ifndef LLVM_C_TRANSFORMS_PASSMANAGERBUILDER_H
-#define LLVM_C_TRANSFORMS_PASSMANAGERBUILDER_H
-
-#include "llvm-c/ExternC.h"
-#include "llvm-c/Types.h"
-
-typedef struct LLVMOpaquePassManagerBuilder *LLVMPassManagerBuilderRef;
-
-LLVM_C_EXTERN_C_BEGIN
-
-/**
- * @defgroup LLVMCTransformsPassManagerBuilder Pass manager builder
- * @ingroup LLVMCTransforms
- *
- * @{
- */
-
-/** See llvm::PassManagerBuilder. */
-LLVMPassManagerBuilderRef LLVMPassManagerBuilderCreate(void);
-void LLVMPassManagerBuilderDispose(LLVMPassManagerBuilderRef PMB);
-
-/** See llvm::PassManagerBuilder::OptLevel. */
-void
-LLVMPassManagerBuilderSetOptLevel(LLVMPassManagerBuilderRef PMB,
-                                  unsigned OptLevel);
-
-/** See llvm::PassManagerBuilder::SizeLevel. */
-void
-LLVMPassManagerBuilderSetSizeLevel(LLVMPassManagerBuilderRef PMB,
-                                   unsigned SizeLevel);
-
-/** See llvm::PassManagerBuilder::DisableUnitAtATime. */
-void
-LLVMPassManagerBuilderSetDisableUnitAtATime(LLVMPassManagerBuilderRef PMB,
-                                            LLVMBool Value);
-
-/** See llvm::PassManagerBuilder::DisableUnrollLoops. */
-void
-LLVMPassManagerBuilderSetDisableUnrollLoops(LLVMPassManagerBuilderRef PMB,
-                                            LLVMBool Value);
-
-/** See llvm::PassManagerBuilder::DisableSimplifyLibCalls */
-void
-LLVMPassManagerBuilderSetDisableSimplifyLibCalls(LLVMPassManagerBuilderRef PMB,
-                                                 LLVMBool Value);
-
-/** See llvm::PassManagerBuilder::Inliner. */
-void
-LLVMPassManagerBuilderUseInlinerWithThreshold(LLVMPassManagerBuilderRef PMB,
-                                              unsigned Threshold);
-
-/** See llvm::PassManagerBuilder::populateFunctionPassManager. */
-void
-LLVMPassManagerBuilderPopulateFunctionPassManager(LLVMPassManagerBuilderRef PMB,
-                                                  LLVMPassManagerRef PM);
-
-/** See llvm::PassManagerBuilder::populateModulePassManager. */
-void
-LLVMPassManagerBuilderPopulateModulePassManager(LLVMPassManagerBuilderRef PMB,
-                                                LLVMPassManagerRef PM);
-
-/**
- * @}
- */
-
-LLVM_C_EXTERN_C_END
-
-#endif
diff --git a/llvm/include/llvm/ADT/DenseMap.h b/llvm/include/llvm/ADT/DenseMap.h
index 12a6673cc62c1..8a680c031d302 100644
--- a/llvm/include/llvm/ADT/DenseMap.h
+++ b/llvm/include/llvm/ADT/DenseMap.h
@@ -141,10 +141,15 @@ class DenseMapBase : public DebugEpochBase {
     setNumTombstones(0);
   }
 
+  /// Return true if the specified key is in the map, false otherwise.
+  bool contains(const_arg_type_t<KeyT> Val) const {
+    const BucketT *TheBucket;
+    return LookupBucketFor(Val, TheBucket);
+  }
+
   /// Return 1 if the specified key is in the map, 0 otherwise.
   size_type count(const_arg_type_t<KeyT> Val) const {
-    const BucketT *TheBucket;
-    return LookupBucketFor(Val, TheBucket) ? 1 : 0;
+    return contains(Val) ? 1 : 0;
   }
 
   iterator find(const_arg_type_t<KeyT> Val) {
diff --git a/llvm/include/llvm/ADT/GenericSSAContext.h b/llvm/include/llvm/ADT/GenericSSAContext.h
index 7c02060636e96..40b2be7c34c31 100644
--- a/llvm/include/llvm/ADT/GenericSSAContext.h
+++ b/llvm/include/llvm/ADT/GenericSSAContext.h
@@ -53,6 +53,11 @@ template <typename _FunctionT> class GenericSSAContext {
   // indicated by the compiler.
   using FunctionT = typename _FunctionT::invalidTemplateInstanceError;
 
+  // A UseT represents a data-edge from the defining instruction to the using
+  // instruction.
+  //
+  // using UseT = ...
+
   // Initialize the SSA context with information about the FunctionT being
   // processed.
   //
diff --git a/llvm/include/llvm/ADT/GenericUniformityImpl.h b/llvm/include/llvm/ADT/GenericUniformityImpl.h
index 380da45437acf..7eff61b26ba56 100644
--- a/llvm/include/llvm/ADT/GenericUniformityImpl.h
+++ b/llvm/include/llvm/ADT/GenericUniformityImpl.h
@@ -330,6 +330,7 @@ template <typename ContextT> class GenericUniformityAnalysisImpl {
   using FunctionT = typename ContextT::FunctionT;
   using ValueRefT = typename ContextT::ValueRefT;
   using ConstValueRefT = typename ContextT::ConstValueRefT;
+  using UseT = typename ContextT::UseT;
   using InstructionT = typename ContextT::InstructionT;
   using DominatorTreeT = typename ContextT::DominatorTreeT;
 
@@ -384,6 +385,8 @@ template <typename ContextT> class GenericUniformityAnalysisImpl {
   /// \brief Whether \p Val is divergent at its definition.
   bool isDivergent(ConstValueRefT V) const { return DivergentValues.count(V); }
 
+  bool isDivergentUse(const UseT &U) const;
+
   bool hasDivergentTerminator(const BlockT &B) const {
     return DivergentTermBlocks.contains(&B);
   }
@@ -462,9 +465,9 @@ template <typename ContextT> class GenericUniformityAnalysisImpl {
 
   bool usesValueFromCycle(const InstructionT &I, const CycleT &DefCycle) const;
 
-  /// \brief Whether \p Val is divergent when read in \p ObservingBlock.
+  /// \brief Whether \p Def is divergent when read in \p ObservingBlock.
   bool isTemporalDivergent(const BlockT &ObservingBlock,
-                           ConstValueRefT Val) const;
+                           const InstructionT &Def) const;
 };
 
 template <typename ImplT>
@@ -784,6 +787,9 @@ bool GenericUniformityAnalysisImpl<ContextT>::markDivergent(
     return false;
   }
 
+  if (isAlwaysUniform(I))
+    return false;
+
   return markDefsDivergent(I);
 }
 
@@ -952,10 +958,6 @@ void GenericUniformityAnalysisImpl<ContextT>::taintAndPushAllDefs(
     if (I.isTerminator())
       break;
 
-    // Mark this as divergent. We don't check if the instruction is
-    // always uniform. In a cycle where the thread convergence is not
-    // statically known, the instruction is not statically converged,
-    // and its outputs cannot be statically uniform.
     if (markDivergent(I))
       Worklist.push_back(&I);
   }
@@ -1092,6 +1094,20 @@ getOutermostDivergentCycle(const CycleT *Cycle, const BlockT *DivTermBlock,
   return Ext;
 }
 
+template <typename ContextT>
+bool GenericUniformityAnalysisImpl<ContextT>::isTemporalDivergent(
+    const BlockT &ObservingBlock, const InstructionT &Def) const {
+  const BlockT *DefBlock = Def.getParent();
+  for (const CycleT *Cycle = CI.getCycle(DefBlock);
+       Cycle && !Cycle->contains(&ObservingBlock);
+       Cycle = Cycle->getParentCycle()) {
+    if (DivergentExitCycles.contains(Cycle)) {
+      return true;
+    }
+  }
+  return false;
+}
+
 template <typename ContextT>
 void GenericUniformityAnalysisImpl<ContextT>::analyzeControlDivergence(
     const InstructionT &Term) {
@@ -1274,6 +1290,11 @@ bool GenericUniformityInfo<ContextT>::isDivergent(const InstructionT *I) const {
   return DA->isDivergent(*I);
 }
 
+template <typename ContextT>
+bool GenericUniformityInfo<ContextT>::isDivergentUse(const UseT &U) const {
+  return DA->isDivergentUse(U);
+}
+
 template <typename ContextT>
 bool GenericUniformityInfo<ContextT>::hasDivergentTerminator(const BlockT &B) {
   return DA->hasDivergentTerminator(B);
diff --git a/llvm/include/llvm/ADT/GenericUniformityInfo.h b/llvm/include/llvm/ADT/GenericUniformityInfo.h
index 987239d2c2b08..add5f3c68225f 100644
--- a/llvm/include/llvm/ADT/GenericUniformityInfo.h
+++ b/llvm/include/llvm/ADT/GenericUniformityInfo.h
@@ -36,6 +36,7 @@ template <typename ContextT> class GenericUniformityInfo {
   using FunctionT = typename ContextT::FunctionT;
   using ValueRefT = typename ContextT::ValueRefT;
   using ConstValueRefT = typename ContextT::ConstValueRefT;
+  using UseT = typename ContextT::UseT;
   using InstructionT = typename ContextT::InstructionT;
   using DominatorTreeT = typename ContextT::DominatorTreeT;
   using ThisT = GenericUniformityInfo<ContextT>;
@@ -69,6 +70,10 @@ template <typename ContextT> class GenericUniformityInfo {
   bool isUniform(const InstructionT *I) const { return !isDivergent(I); };
   bool isDivergent(const InstructionT *I) const;
 
+  /// \brief Whether \p U is divergent. Uses of a uniform value can be
+  /// divergent.
+  bool isDivergentUse(const UseT &U) const;
+
   bool hasDivergentTerminator(const BlockT &B);
 
   void print(raw_ostream &Out) const;
diff --git a/llvm/include/llvm/ADT/MapVector.h b/llvm/include/llvm/ADT/MapVector.h
index 9d908f3af4ed0..a129b63f8be7a 100644
--- a/llvm/include/llvm/ADT/MapVector.h
+++ b/llvm/include/llvm/ADT/MapVector.h
@@ -140,10 +140,9 @@ class MapVector {
     return std::make_pair(begin() + I, false);
   }
 
-  size_type count(const KeyT &Key) const {
-    typename MapType::const_iterator Pos = Map.find(Key);
-    return Pos == Map.end()? 0 : 1;
-  }
+  bool contains(const KeyT &Key) const { return Map.find(Key) != Map.end(); }
+
+  size_type count(const KeyT &Key) const { return contains(Key) ? 1 : 0; }
 
   iterator find(const KeyT &Key) {
     typename MapType::const_iterator Pos = Map.find(Key);
diff --git a/llvm/include/llvm/ADT/STLExtras.h b/llvm/include/llvm/ADT/STLExtras.h
index 0ee260bc99f23..bf33d79801065 100644
--- a/llvm/include/llvm/ADT/STLExtras.h
+++ b/llvm/include/llvm/ADT/STLExtras.h
@@ -755,26 +755,26 @@ template<typename... Iters> struct ZipTupleType {
   using type = std::tuple<decltype(*declval<Iters>())...>;
 };
 
-template <typename ZipType, typename... Iters>
+template <typename ZipType, typename ReferenceTupleType, typename... Iters>
 using zip_traits = iterator_facade_base<
     ZipType,
     std::common_type_t<
         std::bidirectional_iterator_tag,
         typename std::iterator_traits<Iters>::iterator_category...>,
     // ^ TODO: Implement random access methods.
-    typename ZipTupleType<Iters...>::type,
+    ReferenceTupleType,
     typename std::iterator_traits<
         std::tuple_element_t<0, std::tuple<Iters...>>>::difference_type,
     // ^ FIXME: This follows boost::make_zip_iterator's assumption that all
     // inner iterators have the same difference_type. It would fail if, for
     // instance, the second field's difference_type were non-numeric while the
     // first is.
-    typename ZipTupleType<Iters...>::type *,
-    typename ZipTupleType<Iters...>::type>;
+    ReferenceTupleType *, ReferenceTupleType>;
 
-template <typename ZipType, typename... Iters>
-struct zip_common : public zip_traits<ZipType, Iters...> {
-  using Base = zip_traits<ZipType, Iters...>;
+template <typename ZipType, typename ReferenceTupleType, typename... Iters>
+struct zip_common : public zip_traits<ZipType, ReferenceTupleType, Iters...> {
+  using Base = zip_traits<ZipType, ReferenceTupleType, Iters...>;
+  using IndexSequence = std::index_sequence_for<Iters...>;
   using value_type = typename Base::value_type;
 
   std::tuple<Iters...> iterators;
@@ -784,19 +784,17 @@ struct zip_common : public zip_traits<ZipType, Iters...> {
     return value_type(*std::get<Ns>(iterators)...);
   }
 
-  template <size_t... Ns>
-  decltype(iterators) tup_inc(std::index_sequence<Ns...>) const {
-    return std::tuple<Iters...>(std::next(std::get<Ns>(iterators))...);
+  template <size_t... Ns> void tup_inc(std::index_sequence<Ns...>) {
+    (++std::get<Ns>(iterators), ...);
   }
 
-  template <size_t... Ns>
-  decltype(iterators) tup_dec(std::index_sequence<Ns...>) const {
-    return std::tuple<Iters...>(std::prev(std::get<Ns>(iterators))...);
+  template <size_t... Ns> void tup_dec(std::index_sequence<Ns...>) {
+    (--std::get<Ns>(iterators), ...);
   }
 
   template <size_t... Ns>
   bool test_all_equals(const zip_common &other,
-            std::index_sequence<Ns...>) const {
+                       std::index_sequence<Ns...>) const {
     return ((std::get<Ns>(this->iterators) == std::get<Ns>(other.iterators)) &&
             ...);
   }
@@ -804,55 +802,54 @@ struct zip_common : public zip_traits<ZipType, Iters...> {
 public:
   zip_common(Iters &&... ts) : iterators(std::forward<Iters>(ts)...) {}
 
-  value_type operator*() const {
-    return deref(std::index_sequence_for<Iters...>{});
-  }
+  value_type operator*() const { return deref(IndexSequence{}); }
 
   ZipType &operator++() {
-    iterators = tup_inc(std::index_sequence_for<Iters...>{});
-    return *reinterpret_cast<ZipType *>(this);
+    tup_inc(IndexSequence{});
+    return static_cast<ZipType &>(*this);
   }
 
   ZipType &operator--() {
     static_assert(Base::IsBidirectional,
                   "All inner iterators must be at least bidirectional.");
-    iterators = tup_dec(std::index_sequence_for<Iters...>{});
-    return *reinterpret_cast<ZipType *>(this);
+    tup_dec(IndexSequence{});
+    return static_cast<ZipType &>(*this);
   }
 
   /// Return true if all the iterator are matching `other`'s iterators.
   bool all_equals(zip_common &other) {
-    return test_all_equals(other, std::index_sequence_for<Iters...>{});
+    return test_all_equals(other, IndexSequence{});
   }
 };
 
 template <typename... Iters>
-struct zip_first : public zip_common<zip_first<Iters...>, Iters...> {
-  using Base = zip_common<zip_first<Iters...>, Iters...>;
+struct zip_first : zip_common<zip_first<Iters...>,
+                              typename ZipTupleType<Iters...>::type, Iters...> {
+  using zip_common<zip_first, typename ZipTupleType<Iters...>::type,
+                   Iters...>::zip_common;
 
-  bool operator==(const zip_first<Iters...> &other) const {
+  bool operator==(const zip_first &other) const {
     return std::get<0>(this->iterators) == std::get<0>(other.iterators);
   }
-
-  zip_first(Iters &&... ts) : Base(std::forward<Iters>(ts)...) {}
 };
 
 template <typename... Iters>
-class zip_shortest : public zip_common<zip_shortest<Iters...>, Iters...> {
-  template <size_t... Ns>
-  bool test(const zip_shortest<Iters...> &other,
-            std::index_sequence<Ns...>) const {
-    return ((std::get<Ns>(this->iterators) != std::get<Ns>(other.iterators)) &&
-            ...);
-  }
+struct zip_shortest
+    : zip_common<zip_shortest<Iters...>, typename ZipTupleType<Iters...>::type,
+                 Iters...> {
+  using zip_common<zip_shortest, typename ZipTupleType<Iters...>::type,
+                   Iters...>::zip_common;
 
-public:
-  using Base = zip_common<zip_shortest<Iters...>, Iters...>;
-
-  zip_shortest(Iters &&... ts) : Base(std::forward<Iters>(ts)...) {}
+  bool operator==(const zip_shortest &other) const {
+    return any_iterator_equals(other, std::index_sequence_for<Iters...>{});
+  }
 
-  bool operator==(const zip_shortest<Iters...> &other) const {
-    return !test(other, std::index_sequence_for<Iters...>{});
+private:
+  template <size_t... Ns>
+  bool any_iterator_equals(const zip_shortest &other,
+                           std::index_sequence<Ns...>) const {
+    return ((std::get<Ns>(this->iterators) == std::get<Ns>(other.iterators)) ||
+            ...);
   }
 };
 
@@ -1520,19 +1517,21 @@ template <typename ContainerTy> auto make_second_range(ContainerTy &&c) {
 //     Extra additions to <utility>
 //===----------------------------------------------------------------------===//
 
-/// Function object to check whether the first component of a std::pair
-/// compares less than the first component of another std::pair.
+/// Function object to check whether the first component of a container
+/// supported by std::get (like std::pair and std::tuple) compares less than the
+/// first component of another container.
 struct less_first {
   template <typename T> bool operator()(const T &lhs, const T &rhs) const {
-    return std::less<>()(lhs.first, rhs.first);
+    return std::less<>()(std::get<0>(lhs), std::get<0>(rhs));
   }
 };
 
-/// Function object to check whether the second component of a std::pair
-/// compares less than the second component of another std::pair.
+/// Function object to check whether the second component of a container
+/// supported by std::get (like std::pair and std::tuple) compares less than the
+/// second component of another container.
 struct less_second {
   template <typename T> bool operator()(const T &lhs, const T &rhs) const {
-    return std::less<>()(lhs.second, rhs.second);
+    return std::less<>()(std::get<1>(lhs), std::get<1>(rhs));
   }
 };
 
@@ -1911,11 +1910,40 @@ OutputIt move(R &&Range, OutputIt Out) {
   return std::move(adl_begin(Range), adl_end(Range), Out);
 }
 
-/// Wrapper function around std::find to detect if an element exists
-/// in a container.
+namespace detail {
+template <typename Range, typename Element>
+using check_has_member_contains_t =
+    decltype(std::declval<Range &>().contains(std::declval<const Element &>()));
+
+template <typename Range, typename Element>
+static constexpr bool HasMemberContains =
+    is_detected<check_has_member_contains_t, Range, Element>::value;
+
+template <typename Range, typename Element>
+using check_has_member_find_t =
+    decltype(std::declval<Range &>().find(std::declval<const Element &>()) !=
+             std::declval<Range &>().end());
+
+template <typename Range, typename Element>
+static constexpr bool HasMemberFind =
+    is_detected<check_has_member_find_t, Range, Element>::value;
+
+} // namespace detail
+
+/// Returns true if \p Element is found in \p Range. Delegates the check to
+/// either `.contains(Element)`, `.find(Element)`, or `std::find`, in this
+/// order of preference. This is intended as the canonical way to check if an
+/// element exists in a range in generic code or range type that does not
+/// expose a `.contains(Element)` member.
 template <typename R, typename E>
 bool is_contained(R &&Range, const E &Element) {
-  return std::find(adl_begin(Range), adl_end(Range), Element) != adl_end(Range);
+  if constexpr (detail::HasMemberContains<R, E>)
+    return Range.contains(Element);
+  else if constexpr (detail::HasMemberFind<R, E>)
+    return Range.find(Element) != Range.end();
+  else
+    return std::find(adl_begin(Range), adl_end(Range), Element) !=
+           adl_end(Range);
 }
 
 /// Returns true iff \p Element exists in \p Set. This overload takes \p Set as
@@ -2189,113 +2217,182 @@ template <typename T> struct deref {
 
 namespace detail {
 
-template <typename R> class enumerator_iter;
-
-template <typename R> struct result_pair {
-  using value_reference =
-      typename std::iterator_traits<IterOfRange<R>>::reference;
+/// Tuple-like type for `zip_enumerator` dereference.
+template <typename... Refs> struct enumerator_result;
 
-  friend class enumerator_iter<R>;
-
-  result_pair(std::size_t Index, IterOfRange<R> Iter)
-      : Index(Index), Iter(Iter) {}
-
-  std::size_t index() const { return Index; }
-  value_reference value() const { return *Iter; }
-
-private:
-  std::size_t Index = std::numeric_limits<std::size_t>::max();
-  IterOfRange<R> Iter;
-};
-
-template <std::size_t i, typename R>
-decltype(auto) get(const result_pair<R> &Pair) {
-  static_assert(i < 2);
-  if constexpr (i == 0) {
-    return Pair.index();
-  } else {
-    return Pair.value();
-  }
-}
-
-template <typename R>
-class enumerator_iter
-    : public iterator_facade_base<enumerator_iter<R>, std::forward_iterator_tag,
-                                  const result_pair<R>> {
-  using result_type = result_pair<R>;
-
-public:
-  explicit enumerator_iter(IterOfRange<R> EndIter)
-      : Result(std::numeric_limits<size_t>::max(), EndIter) {}
-
-  enumerator_iter(std::size_t Index, IterOfRange<R> Iter)
-      : Result(Index, Iter) {}
-
-  const result_type &operator*() const { return Result; }
+template <typename... Iters>
+using EnumeratorTupleType = enumerator_result<decltype(*declval<Iters>())...>;
+
+/// Zippy iterator that uses the second iterator for comparisons. For the
+/// increment to be safe, the second range has to be the shortest.
+/// Returns `enumerator_result` on dereference to provide `.index()` and
+/// `.value()` member functions.
+/// Note: Because the dereference operator returns `enumerator_result` as a
+/// value instead of a reference and does not strictly conform to the C++17's
+/// definition of forward iterator. However, it satisfies all the
+/// forward_iterator requirements that the `zip_common` and `zippy` depend on
+/// and fully conforms to the C++20 definition of forward iterator.
+/// This is similar to `std::vector<bool>::iterator` that returns bit reference
+/// wrappers on dereference.
+template <typename... Iters>
+struct zip_enumerator : zip_common<zip_enumerator<Iters...>,
+                                   EnumeratorTupleType<Iters...>, Iters...> {
+  static_assert(sizeof...(Iters) >= 2, "Expected at least two iteratees");
+  using zip_common<zip_enumerator<Iters...>, EnumeratorTupleType<Iters...>,
+                   Iters...>::zip_common;
 
-  enumerator_iter &operator++() {
-    assert(Result.Index != std::numeric_limits<size_t>::max());
-    ++Result.Iter;
-    ++Result.Index;
-    return *this;
+  bool operator==(const zip_enumerator &Other) const {
+    return std::get<1>(this->iterators) == std::get<1>(Other.iterators);
   }
+};
 
-  bool operator==(const enumerator_iter &RHS) const {
-    // Don't compare indices here, only iterators.  It's possible for an end
-    // iterator to have different indices depending on whether it was created
-    // by calling std::end() versus incrementing a valid iterator.
-    return Result.Iter == RHS.Result.Iter;
+template <typename... Refs> struct enumerator_result<std::size_t, Refs...> {
+  static constexpr std::size_t NumRefs = sizeof...(Refs);
+  static_assert(NumRefs != 0);
+  // `NumValues` includes the index.
+  static constexpr std::size_t NumValues = NumRefs + 1;
+
+  // Tuple type whose element types are references for each `Ref`.
+  using range_reference_tuple = std::tuple<Refs...>;
+  // Tuple type who elements are references to all values, including both
+  // the index and `Refs` reference types.
+  using value_reference_tuple = std::tuple<std::size_t, Refs...>;
+
+  enumerator_result(std::size_t Index, Refs &&...Rs)
+      : Idx(Index), Storage(std::forward<Refs>(Rs)...) {}
+
+  /// Returns the 0-based index of the current position within the original
+  /// input range(s).
+  std::size_t index() const { return Idx; }
+
+  /// Returns the value(s) for the current iterator. This does not include the
+  /// index.
+  decltype(auto) value() const {
+    if constexpr (NumRefs == 1)
+      return std::get<0>(Storage);
+    else
+      return Storage;
+  }
+
+  /// Returns the value at index `I`. This includes the index.
+  template <std::size_t I>
+  friend decltype(auto) get(const enumerator_result &Result) {
+    static_assert(I < NumValues, "Index out of bounds");
+    if constexpr (I == 0)
+      return Result.Idx;
+    else
+      return std::get<I - 1>(Result.Storage);
+  }
+
+  template <typename... Ts>
+  friend bool operator==(const enumerator_result &Result,
+                         const std::tuple<std::size_t, Ts...> &Other) {
+    static_assert(NumRefs == sizeof...(Ts), "Size mismatch");
+    if (Result.Idx != std::get<0>(Other))
+      return false;
+    return Result.is_value_equal(Other, std::make_index_sequence<NumRefs>{});
   }
 
 private:
-  result_type Result;
+  template <typename Tuple, std::size_t... Idx>
+  bool is_value_equal(const Tuple &Other, std::index_sequence<Idx...>) const {
+    return ((std::get<Idx>(Storage) == std::get<Idx + 1>(Other)) && ...);
+  }
+
+  std::size_t Idx;
+  // Make this tuple mutable to avoid casts that obfuscate const-correctness
+  // issues. Const-correctness of references is taken care of by `zippy` that
+  // defines const-non and const iterator types that will propagate down to
+  // `enumerator_result`'s `Refs`.
+  //  Note that unlike the results of `zip*` functions, `enumerate`'s result are
+  //  supposed to be modifiable even when defined as
+  // `const`.
+  mutable range_reference_tuple Storage;
 };
 
-template <typename R> class enumerator {
-public:
-  explicit enumerator(R &&Range) : TheRange(std::forward<R>(Range)) {}
+/// Infinite stream of increasing 0-based `size_t` indices.
+struct index_stream {
+  struct iterator : iterator_facade_base<iterator, std::forward_iterator_tag,
+                                         const iterator> {
+    iterator &operator++() {
+      assert(Index != std::numeric_limits<std::size_t>::max() &&
+             "Attempting to increment end iterator");
+      ++Index;
+      return *this;
+    }
 
-  enumerator_iter<R> begin() {
-    return enumerator_iter<R>(0, adl_begin(TheRange));
-  }
-  enumerator_iter<R> begin() const {
-    return enumerator_iter<R>(0, adl_begin(TheRange));
-  }
+    // Note: This dereference operator returns a value instead of a reference
+    // and does not strictly conform to the C++17's definition of forward
+    // iterator. However, it satisfies all the forward_iterator requirements
+    // that the `zip_common` depends on and fully conforms to the C++20
+    // definition of forward iterator.
+    std::size_t operator*() const { return Index; }
 
-  enumerator_iter<R> end() { return enumerator_iter<R>(adl_end(TheRange)); }
-  enumerator_iter<R> end() const {
-    return enumerator_iter<R>(adl_end(TheRange));
-  }
+    friend bool operator==(const iterator &Lhs, const iterator &Rhs) {
+      return Lhs.Index == Rhs.Index;
+    }
 
-private:
-  R TheRange;
+    std::size_t Index = 0;
+  };
+
+  iterator begin() const { return {}; }
+  iterator end() const {
+    // We approximate 'infinity' with the max size_t value, which should be good
+    // enough to index over any container.
+    iterator It;
+    It.Index = std::numeric_limits<std::size_t>::max();
+    return It;
+  }
 };
 
 } // end namespace detail
 
-/// Given an input range, returns a new range whose values are are pair (A,B)
-/// such that A is the 0-based index of the item in the sequence, and B is
-/// the value from the original sequence.  Example:
+/// Given two or more input ranges, returns a new range whose values are are
+/// tuples (A, B, C, ...), such that A is the 0-based index of the item in the
+/// sequence, and B, C, ..., are the values from the original input ranges. All
+/// input ranges are required to have equal lengths. Note that the returned
+/// iterator allows for the values (B, C, ...) to be modified.  Example:
+///
+/// ```c++
+/// std::vector<char> Letters = {'A', 'B', 'C', 'D'};
+/// std::vector<int> Vals = {10, 11, 12, 13};
 ///
-/// std::vector<char> Items = {'A', 'B', 'C', 'D'};
-/// for (auto X : enumerate(Items)) {
-///   printf("Item %zu - %c\n", X.index(), X.value());
+/// for (auto [Index, Letter, Value] : enumerate(Letters, Vals)) {
+///   printf("Item %zu - %c: %d\n", Index, Letter, Value);
+///   Value -= 10;
 /// }
+/// ```
 ///
-/// or using structured bindings:
+/// Output:
+///   Item 0 - A: 10
+///   Item 1 - B: 11
+///   Item 2 - C: 12
+///   Item 3 - D: 13
 ///
-/// for (auto [Index, Value] : enumerate(Items)) {
-///   printf("Item %zu - %c\n", Index, Value);
+/// or using an iterator:
+/// ```c++
+/// for (auto it : enumerate(Vals)) {
+///   it.value() += 10;
+///   printf("Item %zu: %d\n", it.index(), it.value());
 /// }
+/// ```
 ///
 /// Output:
-///   Item 0 - A
-///   Item 1 - B
-///   Item 2 - C
-///   Item 3 - D
+///   Item 0: 20
+///   Item 1: 21
+///   Item 2: 22
+///   Item 3: 23
 ///
-template <typename R> detail::enumerator<R> enumerate(R &&TheRange) {
-  return detail::enumerator<R>(std::forward<R>(TheRange));
+template <typename FirstRange, typename... RestRanges>
+auto enumerate(FirstRange &&First, RestRanges &&...Rest) {
+  assert((sizeof...(Rest) == 0 ||
+          all_equal({std::distance(adl_begin(First), adl_end(First)),
+                     std::distance(adl_begin(Rest), adl_end(Rest))...})) &&
+         "Ranges have different length");
+  using enumerator = detail::zippy<detail::zip_enumerator, detail::index_stream,
+                                   FirstRange, RestRanges...>;
+  return enumerator(detail::index_stream{}, std::forward<FirstRange>(First),
+                    std::forward<RestRanges>(Rest)...);
 }
 
 namespace detail {
@@ -2427,15 +2524,17 @@ template <class T> constexpr T *to_address(T *P) { return P; }
 } // end namespace llvm
 
 namespace std {
-template <typename R>
-struct tuple_size<llvm::detail::result_pair<R>>
-    : std::integral_constant<std::size_t, 2> {};
+template <typename... Refs>
+struct tuple_size<llvm::detail::enumerator_result<Refs...>>
+    : std::integral_constant<std::size_t, sizeof...(Refs)> {};
 
-template <std::size_t i, typename R>
-struct tuple_element<i, llvm::detail::result_pair<R>>
-    : std::conditional<i == 0, std::size_t,
-                       typename llvm::detail::result_pair<R>::value_reference> {
-};
+template <std::size_t I, typename... Refs>
+struct tuple_element<I, llvm::detail::enumerator_result<Refs...>>
+    : std::tuple_element<I, std::tuple<Refs...>> {};
+
+template <std::size_t I, typename... Refs>
+struct tuple_element<I, const llvm::detail::enumerator_result<Refs...>>
+    : std::tuple_element<I, std::tuple<Refs...>> {};
 
 } // namespace std
 
diff --git a/llvm/include/llvm/ADT/Sequence.h b/llvm/include/llvm/ADT/Sequence.h
index 1153352d8b24f..ddda9a95a7bc6 100644
--- a/llvm/include/llvm/ADT/Sequence.h
+++ b/llvm/include/llvm/ADT/Sequence.h
@@ -190,7 +190,7 @@ template <typename T, bool IsReverse> struct SafeIntIterator {
   using value_type = T;
   using difference_type = intmax_t;
   using pointer = T *;
-  using reference = T &;
+  using reference = value_type; // The iterator does not reference memory.
 
   // Construct from T.
   explicit SafeIntIterator(T Value) : SI(CheckedInt::from<T>(Value)) {}
@@ -198,9 +198,9 @@ template <typename T, bool IsReverse> struct SafeIntIterator {
   SafeIntIterator(const SafeIntIterator<T, !IsReverse> &O) : SI(O.SI) {}
 
   // Dereference
-  value_type operator*() const { return SI.to<T>(); }
+  reference operator*() const { return SI.to<T>(); }
   // Indexing
-  value_type operator[](intmax_t Offset) const { return *(*this + Offset); }
+  reference operator[](intmax_t Offset) const { return *(*this + Offset); }
 
   // Can be compared for equivalence using the equality/inequality operators.
   bool operator==(const SafeIntIterator &O) const { return SI == O.SI; }
diff --git a/llvm/include/llvm/ADT/StringMap.h b/llvm/include/llvm/ADT/StringMap.h
index 443714c02389f..17e81bcc2ab94 100644
--- a/llvm/include/llvm/ADT/StringMap.h
+++ b/llvm/include/llvm/ADT/StringMap.h
@@ -249,8 +249,11 @@ class StringMap : public StringMapImpl,
   /// if the key is not in the map.
   ValueTy &operator[](StringRef Key) { return try_emplace(Key).first->second; }
 
+  /// contains - Return true if the element is in the map, false otherwise.
+  bool contains(StringRef Key) const { return find(Key) != end(); }
+
   /// count - Return 1 if the element is in the map, 0 otherwise.
-  size_type count(StringRef Key) const { return find(Key) == end() ? 0 : 1; }
+  size_type count(StringRef Key) const { return contains(Key) ? 1 : 0; }
 
   template <typename InputTy>
   size_type count(const StringMapEntry<InputTy> &MapEntry) const {
diff --git a/llvm/include/llvm/ADT/StringRef.h b/llvm/include/llvm/ADT/StringRef.h
index 2d2f0bedfe1f8..868722e19001c 100644
--- a/llvm/include/llvm/ADT/StringRef.h
+++ b/llvm/include/llvm/ADT/StringRef.h
@@ -343,12 +343,11 @@ namespace llvm {
     /// \returns The index of the last occurrence of \p C, or npos if not
     /// found.
     [[nodiscard]] size_t rfind(char C, size_t From = npos) const {
-      From = std::min(From, Length);
-      size_t i = From;
-      while (i != 0) {
-        --i;
-        if (Data[i] == C)
-          return i;
+      size_t I = std::min(From, Length);
+      while (I) {
+        --I;
+        if (Data[I] == C)
+          return I;
       }
       return npos;
     }
@@ -449,8 +448,8 @@ namespace llvm {
     /// Return the number of occurrences of \p C in the string.
     [[nodiscard]] size_t count(char C) const {
       size_t Count = 0;
-      for (size_t i = 0, e = Length; i != e; ++i)
-        if (Data[i] == C)
+      for (size_t I = 0; I != Length; ++I)
+        if (Data[I] == C)
           ++Count;
       return Count;
     }
@@ -624,7 +623,7 @@ namespace llvm {
       if (!starts_with(Prefix))
         return false;
 
-      *this = drop_front(Prefix.size());
+      *this = substr(Prefix.size());
       return true;
     }
 
@@ -634,7 +633,7 @@ namespace llvm {
       if (!startswith_insensitive(Prefix))
         return false;
 
-      *this = drop_front(Prefix.size());
+      *this = substr(Prefix.size());
       return true;
     }
 
@@ -644,7 +643,7 @@ namespace llvm {
       if (!ends_with(Suffix))
         return false;
 
-      *this = drop_back(Suffix.size());
+      *this = substr(0, size() - Suffix.size());
       return true;
     }
 
@@ -654,7 +653,7 @@ namespace llvm {
       if (!endswith_insensitive(Suffix))
         return false;
 
-      *this = drop_back(Suffix.size());
+      *this = substr(0, size() - Suffix.size());
       return true;
     }
 
@@ -671,7 +670,7 @@ namespace llvm {
     /// be returned.
     [[nodiscard]] StringRef slice(size_t Start, size_t End) const {
       Start = std::min(Start, Length);
-      End = std::min(std::max(Start, End), Length);
+      End = std::clamp(End, Start, Length);
       return StringRef(Data + Start, End - Start);
     }
 
diff --git a/llvm/include/llvm/Analysis/CGSCCPassManager.h b/llvm/include/llvm/Analysis/CGSCCPassManager.h
index 5b914afcf366c..0264b36a1d12f 100644
--- a/llvm/include/llvm/Analysis/CGSCCPassManager.h
+++ b/llvm/include/llvm/Analysis/CGSCCPassManager.h
@@ -487,8 +487,16 @@ class CGSCCToFunctionPassAdaptor
   void printPipeline(raw_ostream &OS,
                      function_ref<StringRef(StringRef)> MapClassName2PassName) {
     OS << "function";
-    if (EagerlyInvalidate)
-      OS << "<eager-inv>";
+    if (EagerlyInvalidate || NoRerun) {
+      OS << "<";
+      if (EagerlyInvalidate)
+        OS << "eager-inv";
+      if (EagerlyInvalidate && NoRerun)
+        OS << ";";
+      if (NoRerun)
+        OS << "no-rerun";
+      OS << ">";
+    }
     OS << '(';
     Pass->printPipeline(OS, MapClassName2PassName);
     OS << ')';
diff --git a/llvm/include/llvm/Analysis/DependenceGraphBuilder.h b/llvm/include/llvm/Analysis/DependenceGraphBuilder.h
index e0dbdcdaa7496..f490f20e7c19a 100644
--- a/llvm/include/llvm/Analysis/DependenceGraphBuilder.h
+++ b/llvm/include/llvm/Analysis/DependenceGraphBuilder.h
@@ -156,7 +156,7 @@ template <class GraphType> class AbstractDependenceGraphBuilder {
 
   /// Given an instruction \p I return its associated ordinal number.
   size_t getOrdinal(Instruction &I) {
-    assert(InstOrdinalMap.find(&I) != InstOrdinalMap.end() &&
+    assert(InstOrdinalMap.contains(&I) &&
            "No ordinal computed for this instruction.");
     return InstOrdinalMap[&I];
   }
diff --git a/llvm/include/llvm/Analysis/IRSimilarityIdentifier.h b/llvm/include/llvm/Analysis/IRSimilarityIdentifier.h
index 91af95b3f002f..bbc2385fb4bbc 100644
--- a/llvm/include/llvm/Analysis/IRSimilarityIdentifier.h
+++ b/llvm/include/llvm/Analysis/IRSimilarityIdentifier.h
@@ -226,6 +226,11 @@ struct IRInstructionData
   void
   setPHIPredecessors(DenseMap<BasicBlock *, unsigned> &BasicBlockToInteger);
 
+  /// Get the BasicBlock based operands for PHINodes and BranchInsts.
+  ///
+  /// \returns A list of relevant BasicBlocks.
+  ArrayRef<Value *> getBlockOperVals();
+
   /// Hashes \p Value based on its opcode, types, and operand types.
   /// Two IRInstructionData instances produce the same hash when they perform
   /// the same operation.
diff --git a/llvm/include/llvm/Analysis/InlineOrder.h b/llvm/include/llvm/Analysis/InlineOrder.h
index 4c82d73df8a57..a1d25a25b8568 100644
--- a/llvm/include/llvm/Analysis/InlineOrder.h
+++ b/llvm/include/llvm/Analysis/InlineOrder.h
@@ -32,7 +32,52 @@ template <typename T> class InlineOrder {
 };
 
 std::unique_ptr<InlineOrder<std::pair<CallBase *, int>>>
-getInlineOrder(FunctionAnalysisManager &FAM, const InlineParams &Params);
+getDefaultInlineOrder(FunctionAnalysisManager &FAM, const InlineParams &Params,
+                      ModuleAnalysisManager &MAM, Module &M);
+
+std::unique_ptr<InlineOrder<std::pair<CallBase *, int>>>
+getInlineOrder(FunctionAnalysisManager &FAM, const InlineParams &Params,
+               ModuleAnalysisManager &MAM, Module &M);
+
+/// Used for dynamically loading instances of InlineOrder as plugins
+///
+/// Plugins must implement an InlineOrderFactory, for an example refer to:
+/// llvm/unittests/Analysis/InlineOrderPlugin/InlineOrderPlugin.cpp
+///
+/// If a PluginInlineOrderAnalysis has been registered with the
+/// current ModuleAnalysisManager, llvm::getInlineOrder returns an
+/// InlineOrder created by the PluginInlineOrderAnalysis' Factory.
+///
+class PluginInlineOrderAnalysis
+    : public AnalysisInfoMixin<PluginInlineOrderAnalysis> {
+public:
+  static AnalysisKey Key;
+
+  typedef std::unique_ptr<InlineOrder<std::pair<CallBase *, int>>> (
+      *InlineOrderFactory)(FunctionAnalysisManager &FAM,
+                           const InlineParams &Params,
+                           ModuleAnalysisManager &MAM, Module &M);
+
+  PluginInlineOrderAnalysis(InlineOrderFactory Factory) : Factory(Factory) {
+    HasBeenRegistered = true;
+    assert(Factory != nullptr &&
+           "The plugin inline order factory should not be a null pointer.");
+  }
+
+  struct Result {
+    InlineOrderFactory Factory;
+  };
+
+  Result run(Module &, ModuleAnalysisManager &) { return {Factory}; }
+  Result getResult() { return {Factory}; }
+
+  static bool isRegistered() { return HasBeenRegistered; }
+  static void unregister() { HasBeenRegistered = false; }
+
+private:
+  static bool HasBeenRegistered;
+  InlineOrderFactory Factory;
+};
 
 } // namespace llvm
 #endif // LLVM_ANALYSIS_INLINEORDER_H
diff --git a/llvm/include/llvm/Analysis/LazyCallGraph.h b/llvm/include/llvm/Analysis/LazyCallGraph.h
index d438cea9bf851..211a058aa0173 100644
--- a/llvm/include/llvm/Analysis/LazyCallGraph.h
+++ b/llvm/include/llvm/Analysis/LazyCallGraph.h
@@ -255,7 +255,7 @@ class LazyCallGraph {
     iterator end() { return iterator(Edges.end(), Edges.end()); }
 
     Edge &operator[](Node &N) {
-      assert(EdgeIndexMap.find(&N) != EdgeIndexMap.end() && "No such edge!");
+      assert(EdgeIndexMap.contains(&N) && "No such edge!");
       auto &E = Edges[EdgeIndexMap.find(&N)->second];
       assert(E && "Dead or null edge!");
       return E;
diff --git a/llvm/include/llvm/Analysis/LogicCombine.h b/llvm/include/llvm/Analysis/LogicCombine.h
deleted file mode 100644
index 53eda1f9fe62d..0000000000000
--- a/llvm/include/llvm/Analysis/LogicCombine.h
+++ /dev/null
@@ -1,73 +0,0 @@
-//===------------------ LogicCombine.h --------------------------*- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LLVM_ANALYSIS_LOGICCOMBINE_H
-#define LLVM_ANALYSIS_LOGICCOMBINE_H
-
-#include "LogicalExpr.h"
-#include "llvm/ADT/DenseMap.h"
-#include "llvm/ADT/SetVector.h"
-#include "llvm/ADT/Statistic.h"
-#include "llvm/IR/InstrTypes.h"
-#include "llvm/IR/Instruction.h"
-#include "llvm/Support/Allocator.h"
-
-namespace llvm {
-
-class LogicCombiner;
-
-class LogicalOpNode {
-private:
-  LogicCombiner *Helper;
-  Value *Val;
-  LogicalExpr Expr;
-  // TODO: Add weight to measure cost for more than one use value
-
-  void printAndChain(raw_ostream &OS, uint64_t LeafBits) const;
-
-public:
-  LogicalOpNode(LogicCombiner *OpsHelper, Value *SrcVal,
-                const LogicalExpr &SrcExpr)
-      : Helper(OpsHelper), Val(SrcVal), Expr(SrcExpr) {}
-  ~LogicalOpNode() {}
-
-  Value *getValue() const { return Val; }
-  const LogicalExpr &getExpr() const { return Expr; }
-  void print(raw_ostream &OS) const;
-};
-
-class LogicCombiner {
-public:
-  LogicCombiner() {}
-  ~LogicCombiner() { clear(); }
-
-  Value *simplify(Value *Root);
-
-private:
-  friend class LogicalOpNode;
-
-  SpecificBumpPtrAllocator<LogicalOpNode> Alloc;
-  SmallDenseMap<Value *, LogicalOpNode *, 16> LogicalOpNodes;
-  SmallSetVector<Value *, 8> LeafValues;
-
-  void clear();
-
-  LogicalOpNode *visitLeafNode(Value *Val, unsigned Depth);
-  LogicalOpNode *visitBinOp(BinaryOperator *BO, unsigned Depth);
-  LogicalOpNode *getLogicalOpNode(Value *Val, unsigned Depth = 0);
-  Value *logicalOpToValue(LogicalOpNode *Node);
-};
-
-inline raw_ostream &operator<<(raw_ostream &OS, const LogicalOpNode &I) {
-  I.print(OS);
-  return OS;
-}
-
-} // namespace llvm
-
-#endif // LLVM_ANALYSIS_LOGICCOMBINE_H
diff --git a/llvm/include/llvm/Analysis/LogicalExpr.h b/llvm/include/llvm/Analysis/LogicalExpr.h
deleted file mode 100644
index 4c99d282421ce..0000000000000
--- a/llvm/include/llvm/Analysis/LogicalExpr.h
+++ /dev/null
@@ -1,145 +0,0 @@
-//===------------------- LogicalExpr.h --------------------------*- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-/// \file
-/// This file defines LogicalExpr, a class that represent a logical value by
-/// a set of bitsets.
-///
-/// For a logical expression represented by bitset, the "and" logic
-/// operator represented by "&" is translated to "*" and is then evaluated as
-/// the "or" of the bitset. For example, pattern "a & b" is represented by the
-/// logical expression "01 * 10", and the expression is reduced to "11". So the
-/// operation "&" between two logical expressions (not "xor", only "and" chain)
-/// is actually bitwise "or" of the masks. There are two exceptions:
-///    If one of the operands is constant 0, the entire bitset represents 0.
-///    If one of the operands is constant -1, the result is the other one.
-///
-/// The evaluation of a pattern for bitwise "xor" is represented by a "+" math
-/// operator. But it also has one exception to normal math rules: if two masks
-/// are identical, we remove them. For example with "a ^ a", the logical
-/// expression is "1 + 1". We eliminate them from the logical expression.
-///
-/// We use commutative, associative, and distributive laws of arithmetic
-/// multiplication and addition to reduce the expression. An example for the
-/// LogicalExpr caculation:
-///     ((a & b) | (a ^ c)) ^ (!(b & c) & a)
-/// Mask for the leafs are: a --> 001, b --> 010, c -->100
-/// First step is expand the pattern to:
-///      (((a & b) & (a ^ c)) ^ (a & b) ^ (a ^ c)) ^ (((b & c) ^ -1) & a)
-/// Use logical expression to represent the pattern:
-///      001 * 010 * (001 + 100) + 001 * 010 + 001 + 100 + (010 * 100 + -1C) *
-///      001
-/// Expression after distributive laws:
-///      001 * 010 * 001 + 001 * 010 * 100 + 001 * 010 + 001 + 100 + 010 * 100 *
-///      001 + -1C * 001
-/// Calculate multiplication:
-///      011 + 111 + 011 + 001 + 100 + 111 + 001
-/// Calculate addition:
-///      100
-/// Restore to value
-///      c
-//===----------------------------------------------------------------------===//
-
-#ifndef LLVM_ANALYSIS_LOGICALEXPR_H
-#define LLVM_ANALYSIS_LOGICALEXPR_H
-
-#include "llvm/ADT/DenseSet.h"
-
-namespace llvm {
-// TODO: can we use APInt define the mask to enlarge the max leaf number?
-typedef SmallDenseSet<uint64_t, 8> ExprAddChain;
-
-class LogicalExpr {
-private:
-  ExprAddChain AddChain;
-
-public:
-  static const uint64_t ExprAllOne = 0x8000000000000000;
-
-  LogicalExpr() {}
-  LogicalExpr(uint64_t BitSet) {
-    if (BitSet != 0)
-      AddChain.insert(BitSet);
-  }
-  LogicalExpr(const ExprAddChain &SrcAddChain) : AddChain(SrcAddChain) {
-  }
-
-  unsigned size() const { return AddChain.size(); }
-  ExprAddChain::iterator begin() { return AddChain.begin(); }
-  ExprAddChain::iterator end() { return AddChain.end(); }
-  ExprAddChain::const_iterator begin() const { return AddChain.begin(); }
-  ExprAddChain::const_iterator end() const { return AddChain.end(); }
-
-  LogicalExpr &operator*=(const LogicalExpr &RHS) {
-    ExprAddChain NewChain;
-    for (auto LHS : AddChain) {
-      for (auto RHS : RHS.AddChain) {
-        uint64_t NewBitSet;
-        // Except the special case one value "*" -1 is just return itself, the
-        // other "*" operation is actually "|" LHS and RHS 's bitset. For
-        // example: ab * bd  = abd The expression ab * bd convert to bitset will
-        // be 0b0011 * 0b1010. The result abd convert to bitset will become
-        // 0b1011.
-        if (LHS == ExprAllOne)
-          NewBitSet = RHS;
-        else if (RHS == ExprAllOne)
-          NewBitSet = LHS;
-        else
-          NewBitSet = LHS | RHS;
-        assert(NewBitSet == ExprAllOne || (NewBitSet & ExprAllOne) == 0);
-        // a ^ a -> 0
-        auto InsertPair = NewChain.insert(NewBitSet);
-        if (!InsertPair.second)
-          NewChain.erase(InsertPair.first);
-      }
-    }
-
-    AddChain = NewChain;
-    return *this;
-  }
-
-  LogicalExpr &operator+=(const LogicalExpr &RHS) {
-    for (auto RHS : RHS.AddChain) {
-      // a ^ a -> 0
-      auto InsertPair = AddChain.insert(RHS);
-      if (!InsertPair.second)
-        AddChain.erase(InsertPair.first);
-    }
-    return *this;
-  }
-};
-
-inline LogicalExpr operator*(LogicalExpr a, const LogicalExpr &b) {
-  a *= b;
-  return a;
-}
-
-inline LogicalExpr operator+(LogicalExpr a, const LogicalExpr &b) {
-  a += b;
-  return a;
-}
-
-inline LogicalExpr operator&(const LogicalExpr &a, const LogicalExpr &b) {
-  return a * b;
-}
-
-inline LogicalExpr operator^(const LogicalExpr &a, const LogicalExpr &b) {
-  return a + b;
-}
-
-inline LogicalExpr operator|(const LogicalExpr &a, const LogicalExpr &b) {
-  return a * b + a + b;
-}
-
-inline LogicalExpr operator~(const LogicalExpr &a) {
-  LogicalExpr AllOneExpr(LogicalExpr::ExprAllOne);
-  return a + AllOneExpr;
-}
-
-} // namespace llvm
-
-#endif // LLVM_ANALYSIS_LOGICALEXPR_H
diff --git a/llvm/include/llvm/Analysis/LoopAccessAnalysis.h b/llvm/include/llvm/Analysis/LoopAccessAnalysis.h
index a49e24ada4406..759619396ccb8 100644
--- a/llvm/include/llvm/Analysis/LoopAccessAnalysis.h
+++ b/llvm/include/llvm/Analysis/LoopAccessAnalysis.h
@@ -785,6 +785,9 @@ class LoopAccessInfoManager {
   const LoopAccessInfo &getInfo(Loop &L);
 
   void clear() { LoopAccessInfoMap.clear(); }
+
+  bool invalidate(Function &F, const PreservedAnalyses &PA,
+                  FunctionAnalysisManager::Invalidator &Inv);
 };
 
 /// This analysis provides dependence information for the memory accesses
diff --git a/llvm/include/llvm/Analysis/LoopInfoImpl.h b/llvm/include/llvm/Analysis/LoopInfoImpl.h
index c509ee67cbacc..48f6281e48f8b 100644
--- a/llvm/include/llvm/Analysis/LoopInfoImpl.h
+++ b/llvm/include/llvm/Analysis/LoopInfoImpl.h
@@ -371,7 +371,7 @@ void LoopBase<BlockT, LoopT>::verifyLoop() const {
 
   // Check the parent loop pointer.
   if (ParentLoop) {
-    assert(is_contained(*ParentLoop, this) &&
+    assert(is_contained(ParentLoop->getSubLoops(), this) &&
            "Loop is not a subloop of its parent!");
   }
 #endif
diff --git a/llvm/include/llvm/Analysis/ScalarEvolution.h b/llvm/include/llvm/Analysis/ScalarEvolution.h
index ee1f2eabc77a3..8a25cf9b37c86 100644
--- a/llvm/include/llvm/Analysis/ScalarEvolution.h
+++ b/llvm/include/llvm/Analysis/ScalarEvolution.h
@@ -1133,7 +1133,7 @@ class ScalarEvolution {
   /// Compute the number of times the backedge of the specified loop will
   /// execute if its exit condition were a conditional branch of ExitCond.
   ///
-  /// \p ControlsExit is true if ExitCond directly controls the exit
+  /// \p ControlsOnlyExit is true if ExitCond directly controls the only exit
   /// branch. In this case, we can assume that the loop exits only if the
   /// condition is true and can infer that failing to meet the condition prior
   /// to integer wraparound results in undefined behavior.
@@ -1141,7 +1141,7 @@ class ScalarEvolution {
   /// If \p AllowPredicates is set, this call will try to use a minimal set of
   /// SCEV predicates in order to return an exact answer.
   ExitLimit computeExitLimitFromCond(const Loop *L, Value *ExitCond,
-                                     bool ExitIfTrue, bool ControlsExit,
+                                     bool ExitIfTrue, bool ControlsOnlyExit,
                                      bool AllowPredicates = false);
 
   /// A predicate is said to be monotonically increasing if may go from being
@@ -1200,11 +1200,9 @@ class ScalarEvolution {
   /// Simplify LHS and RHS in a comparison with predicate Pred. Return true
   /// iff any changes were made. If the operands are provably equal or
   /// unequal, LHS and RHS are set to the same value and Pred is set to either
-  /// ICMP_EQ or ICMP_NE. ControllingFiniteLoop is set if this comparison
-  /// controls the exit of a loop known to have a finite number of iterations.
+  /// ICMP_EQ or ICMP_NE.
   bool SimplifyICmpOperands(ICmpInst::Predicate &Pred, const SCEV *&LHS,
-                            const SCEV *&RHS, unsigned Depth = 0,
-                            bool ControllingFiniteLoop = false);
+                            const SCEV *&RHS, unsigned Depth = 0);
 
   /// Return the "disposition" of the given SCEV with respect to the given
   /// loop.
@@ -1771,11 +1769,11 @@ class ScalarEvolution {
   // complexity.
 
   class ExitLimitCache {
-    // It may look like we need key on the whole (L, ExitIfTrue, ControlsExit,
-    // AllowPredicates) tuple, but recursive calls to
+    // It may look like we need key on the whole (L, ExitIfTrue,
+    // ControlsOnlyExit, AllowPredicates) tuple, but recursive calls to
     // computeExitLimitFromCondCached from computeExitLimitFromCondImpl only
-    // vary the in \c ExitCond and \c ControlsExit parameters.  We remember the
-    // initial values of the other values to assert our assumption.
+    // vary the in \c ExitCond and \c ControlsOnlyExit parameters.  We remember
+    // the initial values of the other values to assert our assumption.
     SmallDenseMap<PointerIntPair<Value *, 1>, ExitLimit> TripCountMap;
 
     const Loop *L;
@@ -1787,11 +1785,12 @@ class ScalarEvolution {
         : L(L), ExitIfTrue(ExitIfTrue), AllowPredicates(AllowPredicates) {}
 
     std::optional<ExitLimit> find(const Loop *L, Value *ExitCond,
-                                  bool ExitIfTrue, bool ControlsExit,
+                                  bool ExitIfTrue, bool ControlsOnlyExit,
                                   bool AllowPredicates);
 
     void insert(const Loop *L, Value *ExitCond, bool ExitIfTrue,
-                bool ControlsExit, bool AllowPredicates, const ExitLimit &EL);
+                bool ControlsOnlyExit, bool AllowPredicates,
+                const ExitLimit &EL);
   };
 
   using ExitLimitCacheTy = ExitLimitCache;
@@ -1799,16 +1798,15 @@ class ScalarEvolution {
   ExitLimit computeExitLimitFromCondCached(ExitLimitCacheTy &Cache,
                                            const Loop *L, Value *ExitCond,
                                            bool ExitIfTrue,
-                                           bool ControlsExit,
+                                           bool ControlsOnlyExit,
                                            bool AllowPredicates);
   ExitLimit computeExitLimitFromCondImpl(ExitLimitCacheTy &Cache, const Loop *L,
                                          Value *ExitCond, bool ExitIfTrue,
-                                         bool ControlsExit,
+                                         bool ControlsOnlyExit,
                                          bool AllowPredicates);
-  std::optional<ScalarEvolution::ExitLimit>
-  computeExitLimitFromCondFromBinOp(ExitLimitCacheTy &Cache, const Loop *L,
-                                    Value *ExitCond, bool ExitIfTrue,
-                                    bool ControlsExit, bool AllowPredicates);
+  std::optional<ScalarEvolution::ExitLimit> computeExitLimitFromCondFromBinOp(
+      ExitLimitCacheTy &Cache, const Loop *L, Value *ExitCond, bool ExitIfTrue,
+      bool ControlsOnlyExit, bool AllowPredicates);
 
   /// Compute the number of times the backedge of the specified loop will
   /// execute if its exit condition were a conditional branch of the ICmpInst
@@ -1873,14 +1871,14 @@ class ScalarEvolution {
   ///
   /// \p isSigned specifies whether the less-than is signed.
   ///
-  /// \p ControlsExit is true when the LHS < RHS condition directly controls
+  /// \p ControlsOnlyExit is true when the LHS < RHS condition directly controls
   /// the branch (loops exits only if condition is true). In this case, we can
   /// use NoWrapFlags to skip overflow checks.
   ///
   /// If \p AllowPredicates is set, this call will try to use a minimal set of
   /// SCEV predicates in order to return an exact answer.
   ExitLimit howManyLessThans(const SCEV *LHS, const SCEV *RHS, const Loop *L,
-                             bool isSigned, bool ControlsExit,
+                             bool isSigned, bool ControlsOnlyExit,
                              bool AllowPredicates = false);
 
   ExitLimit howManyGreaterThans(const SCEV *LHS, const SCEV *RHS, const Loop *L,
diff --git a/llvm/include/llvm/Analysis/TargetFolder.h b/llvm/include/llvm/Analysis/TargetFolder.h
index 51f457ddb529a..3d9edf132dc10 100644
--- a/llvm/include/llvm/Analysis/TargetFolder.h
+++ b/llvm/include/llvm/Analysis/TargetFolder.h
@@ -116,6 +116,9 @@ class TargetFolder final : public IRBuilderFolder {
 
   Value *FoldGEP(Type *Ty, Value *Ptr, ArrayRef<Value *> IdxList,
                  bool IsInBounds = false) const override {
+    if (!ConstantExpr::isSupportedGetElementPtr(Ty))
+      return nullptr;
+
     if (auto *PC = dyn_cast<Constant>(Ptr)) {
       // Every index must be constant.
       if (any_of(IdxList, [](Value *V) { return !isa<Constant>(V); }))
diff --git a/llvm/include/llvm/Analysis/TargetLibraryInfo.h b/llvm/include/llvm/Analysis/TargetLibraryInfo.h
index 42eb11fc34e92..1626798ba1c21 100644
--- a/llvm/include/llvm/Analysis/TargetLibraryInfo.h
+++ b/llvm/include/llvm/Analysis/TargetLibraryInfo.h
@@ -138,7 +138,7 @@ class TargetLibraryInfoImpl {
     if (StandardNames[F] != Name) {
       setState(F, CustomName);
       CustomNames[F] = std::string(Name);
-      assert(CustomNames.find(F) != CustomNames.end());
+      assert(CustomNames.contains(F));
     } else {
       setState(F, StandardName);
     }
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfo.h b/llvm/include/llvm/Analysis/TargetTransformInfo.h
index 9742429203552..ba8977678d36f 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfo.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfo.h
@@ -279,6 +279,51 @@ class TargetTransformInfo {
              ArrayRef<const Value *> Operands,
              TargetCostKind CostKind = TCK_SizeAndLatency) const;
 
+  /// Describe known properties for a set of pointers.
+  struct PointersChainInfo {
+    /// All the GEPs in a set have same base address.
+    unsigned IsSameBaseAddress : 1;
+    /// These properties only valid if SameBaseAddress is set.
+    /// True if distance between any two neigbouring pointers is same value.
+    unsigned IsUniformStride : 1;
+    /// True if distance between any two neigbouring pointers is a known value.
+    unsigned IsKnownStride : 1;
+    unsigned Reserved : 29;
+
+    bool isSameBase() const { return IsSameBaseAddress; }
+    bool isUniformStride() const {
+      return IsSameBaseAddress && IsUniformStride;
+    }
+    bool isKnownStride() const { return IsSameBaseAddress && IsKnownStride; }
+
+    static PointersChainInfo getKnownUniformStrided() {
+      return {/*IsSameBaseAddress=*/1, /*IsUniformStride=*/1,
+              /*IsKnownStride=*/1, 0};
+    }
+    static PointersChainInfo getUniformStrided() {
+      return {/*IsSameBaseAddress=*/1, /*IsUniformStride=*/1,
+              /*IsKnownStride=*/0, 0};
+    }
+    static PointersChainInfo getKnownNonUniformStrided() {
+      return {/*IsSameBaseAddress=*/1, /*IsUniformStride=*/0,
+              /*IsKnownStride=*/1, 0};
+    }
+    static PointersChainInfo getNonUniformStrided() {
+      return {/*IsSameBaseAddress=*/1, /*IsUniformStride=*/0,
+              /*IsKnownStride=*/0, 0};
+    }
+  };
+  static_assert(sizeof(PointersChainInfo) == 4, "Was size increase justified?");
+
+  /// Estimate the cost of a chain of pointers (typically pointer operands of a
+  /// chain of loads or stores within same block) operations set when lowered.
+  InstructionCost
+  getPointersChainCost(ArrayRef<const Value *> Ptrs, const Value *Base,
+                       const PointersChainInfo &Info,
+                       TargetCostKind CostKind = TTI::TCK_RecipThroughput
+
+  ) const;
+
   /// \returns A value by which our inlining threshold should be multiplied.
   /// This is primarily used to bump up the inlining threshold wholesale on
   /// targets where calls are unusually expensive.
@@ -1612,6 +1657,10 @@ class TargetTransformInfo::Concept {
   virtual InstructionCost getGEPCost(Type *PointeeType, const Value *Ptr,
                                      ArrayRef<const Value *> Operands,
                                      TTI::TargetCostKind CostKind) = 0;
+  virtual InstructionCost
+  getPointersChainCost(ArrayRef<const Value *> Ptrs, const Value *Base,
+                       const TTI::PointersChainInfo &Info,
+                       TTI::TargetCostKind CostKind) = 0;
   virtual unsigned getInliningThresholdMultiplier() = 0;
   virtual unsigned adjustInliningThreshold(const CallBase *CB) = 0;
   virtual int getInlinerVectorBonusPercent() = 0;
@@ -1970,6 +2019,12 @@ class TargetTransformInfo::Model final : public TargetTransformInfo::Concept {
              TargetTransformInfo::TargetCostKind CostKind) override {
     return Impl.getGEPCost(PointeeType, Ptr, Operands, CostKind);
   }
+  InstructionCost getPointersChainCost(ArrayRef<const Value *> Ptrs,
+                                       const Value *Base,
+                                       const PointersChainInfo &Info,
+                                       TargetCostKind CostKind) override {
+    return Impl.getPointersChainCost(Ptrs, Base, Info, CostKind);
+  }
   unsigned getInliningThresholdMultiplier() override {
     return Impl.getInliningThresholdMultiplier();
   }
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
index 5a23ea2149ca1..18b31396f2744 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
@@ -499,6 +499,14 @@ class TargetTransformInfoImplBase {
       TTI::OperandValueInfo Opd1Info, TTI::OperandValueInfo Opd2Info,
       ArrayRef<const Value *> Args,
       const Instruction *CxtI = nullptr) const {
+    // Widenable conditions will eventually lower into constants, so some
+    // operations with them will be trivially optimized away.
+    auto IsWidenableCondition = [](const Value *V) {
+      if (auto *II = dyn_cast<IntrinsicInst>(V))
+        if (II->getIntrinsicID() == Intrinsic::experimental_widenable_condition)
+          return true;
+      return false;
+    };
     // FIXME: A number of transformation tests seem to require these values
     // which seems a little odd for how arbitary there are.
     switch (Opcode) {
@@ -512,6 +520,11 @@ class TargetTransformInfoImplBase {
     case Instruction::URem:
       // FIXME: Unlikely to be true for CodeSize.
       return TTI::TCC_Expensive;
+    case Instruction::And:
+    case Instruction::Or:
+      if (any_of(Args, IsWidenableCondition))
+        return TTI::TCC_Free;
+      break;
     }
 
     // Assume a 3cy latency for fp arithmetic ops.
@@ -654,6 +667,7 @@ class TargetTransformInfoImplBase {
     case Intrinsic::sideeffect:
     case Intrinsic::pseudoprobe:
     case Intrinsic::arithmetic_fence:
+    case Intrinsic::dbg_assign:
     case Intrinsic::dbg_declare:
     case Intrinsic::dbg_value:
     case Intrinsic::dbg_label:
@@ -680,6 +694,7 @@ class TargetTransformInfoImplBase {
     case Intrinsic::coro_suspend:
     case Intrinsic::coro_subfn_addr:
     case Intrinsic::threadlocal_address:
+    case Intrinsic::experimental_widenable_condition:
       // These intrinsics don't actually represent code after lowering.
       return 0;
     }
@@ -1023,6 +1038,42 @@ class TargetTransformInfoImplCRTPBase : public TargetTransformInfoImplBase {
     return TTI::TCC_Basic;
   }
 
+  InstructionCost getPointersChainCost(ArrayRef<const Value *> Ptrs,
+                                       const Value *Base,
+                                       const TTI::PointersChainInfo &Info,
+                                       TTI::TargetCostKind CostKind) {
+    InstructionCost Cost = TTI::TCC_Free;
+    // In the basic model we take into account GEP instructions only
+    // (although here can come alloca instruction, a value, constants and/or
+    // constant expressions, PHIs, bitcasts ... whatever allowed to be used as a
+    // pointer). Typically, if Base is a not a GEP-instruction and all the
+    // pointers are relative to the same base address, all the rest are
+    // either GEP instructions, PHIs, bitcasts or constants. When we have same
+    // base, we just calculate cost of each non-Base GEP as an ADD operation if
+    // any their index is a non-const.
+    // If no known dependecies between the pointers cost is calculated as a sum
+    // of costs of GEP instructions.
+    for (const Value *V : Ptrs) {
+      const auto *GEP = dyn_cast<GetElementPtrInst>(V);
+      if (!GEP)
+        continue;
+      if (Info.isSameBase() && V != Base) {
+        if (GEP->hasAllConstantIndices())
+          continue;
+        Cost += static_cast<T *>(this)->getArithmeticInstrCost(
+            Instruction::Add, GEP->getType(), CostKind,
+            {TTI::OK_AnyValue, TTI::OP_None}, {TTI::OK_AnyValue, TTI::OP_None},
+            std::nullopt);
+      } else {
+        SmallVector<const Value *> Indices(GEP->indices());
+        Cost += static_cast<T *>(this)->getGEPCost(GEP->getSourceElementType(),
+                                                   GEP->getPointerOperand(),
+                                                   Indices, CostKind);
+      }
+    }
+    return Cost;
+  }
+
   InstructionCost getInstructionCost(const User *U,
                                      ArrayRef<const Value *> Operands,
                                      TTI::TargetCostKind CostKind) {
diff --git a/llvm/include/llvm/Analysis/Utils/TrainingLogger.h b/llvm/include/llvm/Analysis/Utils/TrainingLogger.h
index 57a1637f59b5c..8f46779a732d1 100644
--- a/llvm/include/llvm/Analysis/Utils/TrainingLogger.h
+++ b/llvm/include/llvm/Analysis/Utils/TrainingLogger.h
@@ -128,7 +128,7 @@ class Logger final {
 
   /// Check if there is at least an observation for the context `Ctx`.
   bool hasAnyObservationForContext(StringRef Ctx) const {
-    return ObservationIDs.find(Ctx) != ObservationIDs.end();
+    return ObservationIDs.contains(Ctx);
   }
 
   template <typename T> void logReward(T Value) {
diff --git a/llvm/include/llvm/Analysis/ValueTracking.h b/llvm/include/llvm/Analysis/ValueTracking.h
index 041b97c4919d1..51ee3ad27fbf6 100644
--- a/llvm/include/llvm/Analysis/ValueTracking.h
+++ b/llvm/include/llvm/Analysis/ValueTracking.h
@@ -217,6 +217,106 @@ unsigned ComputeMaxSignificantBits(const Value *Op, const DataLayout &DL,
 Intrinsic::ID getIntrinsicForCallSite(const CallBase &CB,
                                       const TargetLibraryInfo *TLI);
 
+/// Returns a pair of values, which if passed to llvm.is.fpclass, returns the
+/// same result as an fcmp with the given operands.
+///
+/// If \p LookThroughSrc is true, consider the input value when computing the
+/// mask.
+///
+/// If \p LookThroughSrc is false, ignore the source value (i.e. the first pair
+/// element will always be LHS.
+std::pair<Value *, FPClassTest> fcmpToClassTest(CmpInst::Predicate Pred,
+                                                const Function &F, Value *LHS,
+                                                Value *RHS,
+                                                bool LookThroughSrc = true);
+
+struct KnownFPClass {
+  /// Floating-point classes the value could be one of.
+  FPClassTest KnownFPClasses = fcAllFlags;
+
+  /// std::nullopt if the sign bit is unknown, true if the sign bit is
+  /// definitely set or false if the sign bit is definitely unset.
+  std::optional<bool> SignBit;
+
+  KnownFPClass &operator|=(const KnownFPClass &RHS) {
+    KnownFPClasses = KnownFPClasses | RHS.KnownFPClasses;
+
+    if (SignBit != RHS.SignBit)
+      SignBit = std::nullopt;
+    return *this;
+  }
+
+  void knownNot(FPClassTest RuleOut) {
+    KnownFPClasses = KnownFPClasses & ~RuleOut;
+  }
+
+  void fneg() {
+    KnownFPClasses = llvm::fneg(KnownFPClasses);
+    if (SignBit)
+      SignBit = !*SignBit;
+  }
+
+  void fabs() {
+    KnownFPClasses = llvm::fabs(KnownFPClasses);
+    SignBit = false;
+  }
+
+  /// Assume the sign bit is zero.
+  void signBitIsZero() {
+    KnownFPClasses = (KnownFPClasses & fcPositive) |
+                     (KnownFPClasses & fcNan);
+    SignBit = false;
+  }
+
+  void copysign(const KnownFPClass &Sign) {
+    // Start assuming nothing about the sign.
+    SignBit = Sign.SignBit;
+    if (!SignBit)
+      return;
+
+    if (*SignBit)
+      KnownFPClasses = KnownFPClasses & fcNegative;
+    else
+      KnownFPClasses = KnownFPClasses & fcPositive;
+  }
+
+  void resetAll() { *this = KnownFPClass(); }
+};
+
+inline KnownFPClass operator|(KnownFPClass LHS, const KnownFPClass &RHS) {
+  LHS |= RHS;
+  return LHS;
+}
+
+inline KnownFPClass operator|(const KnownFPClass &LHS, KnownFPClass &&RHS) {
+  RHS |= LHS;
+  return std::move(RHS);
+}
+
+/// Determine which floating-point classes are valid for \p V, and return them
+/// in KnownFPClass bit sets.
+///
+/// This function is defined on values with floating-point type, values vectors
+/// of floating-point type, and arrays of floating-point type.
+
+/// \p InterestedClasses is a compile time optimization hint for which floating
+/// point classes should be queried. Queries not specified in \p
+/// InterestedClasses should be reliable if they are determined during the
+/// query.
+KnownFPClass computeKnownFPClass(
+    const Value *V, const APInt &DemandedElts, const DataLayout &DL,
+    FPClassTest InterestedClasses = fcAllFlags, unsigned Depth = 0,
+    const TargetLibraryInfo *TLI = nullptr, AssumptionCache *AC = nullptr,
+    const Instruction *CxtI = nullptr, const DominatorTree *DT = nullptr,
+    OptimizationRemarkEmitter *ORE = nullptr, bool UseInstrInfo = true);
+
+KnownFPClass computeKnownFPClass(
+    const Value *V, const DataLayout &DL,
+    FPClassTest InterestedClasses = fcAllFlags, unsigned Depth = 0,
+    const TargetLibraryInfo *TLI = nullptr, AssumptionCache *AC = nullptr,
+    const Instruction *CxtI = nullptr, const DominatorTree *DT = nullptr,
+    OptimizationRemarkEmitter *ORE = nullptr, bool UseInstrInfo = true);
+
 /// Return true if we can prove that the specified FP value is never equal to
 /// -0.0.
 bool CannotBeNegativeZero(const Value *V, const TargetLibraryInfo *TLI,
@@ -561,6 +661,10 @@ OverflowResult computeOverflowForSignedSub(const Value *LHS, const Value *RHS,
 bool isOverflowIntrinsicNoWrap(const WithOverflowInst *WO,
                                const DominatorTree &DT);
 
+/// Determine the possible constant range of vscale with the given bit width,
+/// based on the vscale_range function attribute.
+ConstantRange getVScaleRange(const Function *F, unsigned BitWidth);
+
 /// Determine the possible constant range of an integer or vector of integer
 /// value. This is intended as a cheap, non-recursive check.
 ConstantRange computeConstantRange(const Value *V, bool ForSigned,
diff --git a/llvm/include/llvm/Analysis/VecFuncs.def b/llvm/include/llvm/Analysis/VecFuncs.def
index 85d208b946252..34f5b3904cb07 100644
--- a/llvm/include/llvm/Analysis/VecFuncs.def
+++ b/llvm/include/llvm/Analysis/VecFuncs.def
@@ -525,7 +525,6 @@ TLI_DEFINE_VECFUNC( "sinh", "_ZGVnN2v_sinh", FIXED(2))
 TLI_DEFINE_VECFUNC( "llvm.sinh.f64", "_ZGVnN2v_sinh", FIXED(2))
 
 TLI_DEFINE_VECFUNC( "sqrt", "_ZGVnN2v_sqrt", FIXED(2))
-TLI_DEFINE_VECFUNC( "llvm.sqrt.f64", "_ZGVnN2v_sqrt", FIXED(2))
 
 TLI_DEFINE_VECFUNC( "tan", "_ZGVnN2v_tan", FIXED(2))
 TLI_DEFINE_VECFUNC( "llvm.tan.f64", "_ZGVnN2v_tan", FIXED(2))
@@ -595,7 +594,6 @@ TLI_DEFINE_VECFUNC( "sinhf", "_ZGVnN4v_sinhf", FIXED(4))
 TLI_DEFINE_VECFUNC( "llvm.sinh.f32", "_ZGVnN4v_sinhf", FIXED(4))
 
 TLI_DEFINE_VECFUNC( "sqrtf", "_ZGVnN4v_sqrtf", FIXED(4))
-TLI_DEFINE_VECFUNC( "llvm.sqrt.f32", "_ZGVnN4v_sqrtf", FIXED(4))
 
 TLI_DEFINE_VECFUNC( "tanf", "_ZGVnN4v_tanf", FIXED(4))
 TLI_DEFINE_VECFUNC( "llvm.tan.f32", "_ZGVnN4v_tanf", FIXED(4))
diff --git a/llvm/include/llvm/Analysis/VectorUtils.h b/llvm/include/llvm/Analysis/VectorUtils.h
index af2cd6f07aa9d..da223f30e81a7 100644
--- a/llvm/include/llvm/Analysis/VectorUtils.h
+++ b/llvm/include/llvm/Analysis/VectorUtils.h
@@ -815,7 +815,7 @@ class InterleavedAccessInfo {
 
   /// Check if \p Instr belongs to any interleave group.
   bool isInterleaved(Instruction *Instr) const {
-    return InterleaveGroupMap.find(Instr) != InterleaveGroupMap.end();
+    return InterleaveGroupMap.contains(Instr);
   }
 
   /// Get the interleave group that \p Instr belongs to.
@@ -979,8 +979,7 @@ class InterleavedAccessInfo {
 
     // If we know there is a dependence from source to sink, assume the
     // instructions can't be reordered. Otherwise, reordering is legal.
-    return Dependences.find(Src) == Dependences.end() ||
-           !Dependences.lookup(Src).count(Sink);
+    return !Dependences.contains(Src) || !Dependences.lookup(Src).count(Sink);
   }
 
   /// Collect the dependences from LoopAccessInfo.
diff --git a/llvm/include/llvm/CodeGen/AssignmentTrackingAnalysis.h b/llvm/include/llvm/CodeGen/AssignmentTrackingAnalysis.h
index 6e82b2b1c1582..3e00627e7ebda 100644
--- a/llvm/include/llvm/CodeGen/AssignmentTrackingAnalysis.h
+++ b/llvm/include/llvm/CodeGen/AssignmentTrackingAnalysis.h
@@ -3,6 +3,7 @@
 
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/DebugLoc.h"
+#include "llvm/IR/IntrinsicInst.h"
 #include "llvm/Pass.h"
 
 namespace llvm {
@@ -21,7 +22,7 @@ struct VarLocInfo {
   llvm::VariableID VariableID;
   DIExpression *Expr = nullptr;
   DebugLoc DL;
-  Value *V = nullptr; // TODO: Needs to be value_s_ for variadic expressions.
+  RawLocationWrapper Values = RawLocationWrapper();
 };
 
 /// Data structure describing the variable locations in a function. Used as the
diff --git a/llvm/include/llvm/CodeGen/BasicTTIImpl.h b/llvm/include/llvm/CodeGen/BasicTTIImpl.h
index 2ae01333b2c9a..81c007485a993 100644
--- a/llvm/include/llvm/CodeGen/BasicTTIImpl.h
+++ b/llvm/include/llvm/CodeGen/BasicTTIImpl.h
@@ -2334,6 +2334,7 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
   InstructionCost getArithmeticReductionCost(unsigned Opcode, VectorType *Ty,
                                              std::optional<FastMathFlags> FMF,
                                              TTI::TargetCostKind CostKind) {
+    assert(Ty && "Unknown reduction vector type");
     if (TTI::requiresOrderedReduction(FMF))
       return getOrderedReductionCost(Opcode, Ty, CostKind);
     return getTreeReductionCost(Opcode, Ty, CostKind);
diff --git a/llvm/include/llvm/CodeGen/CommandFlags.h b/llvm/include/llvm/CodeGen/CommandFlags.h
index 7b1ef60912f1a..475d87bdd5b13 100644
--- a/llvm/include/llvm/CodeGen/CommandFlags.h
+++ b/llvm/include/llvm/CodeGen/CommandFlags.h
@@ -94,8 +94,6 @@ std::string getTrapFuncName();
 
 bool getUseCtors();
 
-bool getLowerGlobalDtorsViaCxaAtExit();
-
 bool getRelaxELFRelocations();
 
 bool getDataSections();
diff --git a/llvm/include/llvm/CodeGen/ComplexDeinterleavingPass.h b/llvm/include/llvm/CodeGen/ComplexDeinterleavingPass.h
index 99df6e5ad1d71..3d11bf3651a36 100644
--- a/llvm/include/llvm/CodeGen/ComplexDeinterleavingPass.h
+++ b/llvm/include/llvm/CodeGen/ComplexDeinterleavingPass.h
@@ -38,7 +38,8 @@ enum class ComplexDeinterleavingOperation {
   CMulPartial,
   // The following 'operations' are used to represent internal states. Backends
   // are not expected to try and support these in any capacity.
-  Shuffle
+  Shuffle,
+  Symmetric
 };
 
 enum class ComplexDeinterleavingRotation {
diff --git a/llvm/include/llvm/CodeGen/DIE.h b/llvm/include/llvm/CodeGen/DIE.h
index 7f7372630dbec..a598e0ae06888 100644
--- a/llvm/include/llvm/CodeGen/DIE.h
+++ b/llvm/include/llvm/CodeGen/DIE.h
@@ -115,6 +115,11 @@ class DIEAbbrev : public FoldingSetNode {
     Data.push_back(DIEAbbrevData(Attribute, Value));
   }
 
+  /// Adds another set of attribute information to the abbreviation.
+  void AddAttribute(const DIEAbbrevData &AbbrevData) {
+    Data.push_back(AbbrevData);
+  }
+
   /// Used to gather unique data for the abbreviation folding set.
   void Profile(FoldingSetNodeID &ID) const;
 
diff --git a/llvm/include/llvm/CodeGen/FunctionLoweringInfo.h b/llvm/include/llvm/CodeGen/FunctionLoweringInfo.h
index f8156ce73196c..e880b6227ef0b 100644
--- a/llvm/include/llvm/CodeGen/FunctionLoweringInfo.h
+++ b/llvm/include/llvm/CodeGen/FunctionLoweringInfo.h
@@ -35,7 +35,6 @@ namespace llvm {
 class Argument;
 class BasicBlock;
 class BranchProbabilityInfo;
-class LegacyDivergenceAnalysis;
 class Function;
 class Instruction;
 class MachineFunction;
@@ -45,6 +44,11 @@ class MVT;
 class SelectionDAG;
 class TargetLowering;
 
+template <typename T> class GenericSSAContext;
+using SSAContext = GenericSSAContext<Function>;
+template <typename T> class GenericUniformityInfo;
+using UniformityInfo = GenericUniformityInfo<SSAContext>;
+
 //===--------------------------------------------------------------------===//
 /// FunctionLoweringInfo - This contains information that is global to a
 /// function that is used when lowering a region of the function.
@@ -56,7 +60,7 @@ class FunctionLoweringInfo {
   const TargetLowering *TLI;
   MachineRegisterInfo *RegInfo;
   BranchProbabilityInfo *BPI;
-  const LegacyDivergenceAnalysis *DA;
+  const UniformityInfo *UA;
   /// CanLowerReturn - true iff the function's return value can be lowered to
   /// registers.
   bool CanLowerReturn;
diff --git a/llvm/include/llvm/CodeGen/GlobalISel/IRTranslator.h b/llvm/include/llvm/CodeGen/GlobalISel/IRTranslator.h
index 67e884038b478..48c49c22f81d0 100644
--- a/llvm/include/llvm/CodeGen/GlobalISel/IRTranslator.h
+++ b/llvm/include/llvm/CodeGen/GlobalISel/IRTranslator.h
@@ -104,9 +104,7 @@ class IRTranslator : public MachineFunctionPass {
       return ValToVRegs.find(&V);
     }
 
-    bool contains(const Value &V) const {
-      return ValToVRegs.find(&V) != ValToVRegs.end();
-    }
+    bool contains(const Value &V) const { return ValToVRegs.contains(&V); }
 
     void reset() {
       ValToVRegs.clear();
@@ -117,7 +115,7 @@ class IRTranslator : public MachineFunctionPass {
 
   private:
     VRegListT *insertVRegs(const Value &V) {
-      assert(ValToVRegs.find(&V) == ValToVRegs.end() && "Value already exists");
+      assert(!ValToVRegs.contains(&V) && "Value already exists");
 
       // We placement new using our fast allocator since we never try to free
       // the vectors until translation is finished.
@@ -127,8 +125,7 @@ class IRTranslator : public MachineFunctionPass {
     }
 
     OffsetListT *insertOffsets(const Value &V) {
-      assert(TypeToOffsets.find(V.getType()) == TypeToOffsets.end() &&
-             "Type already exists");
+      assert(!TypeToOffsets.contains(V.getType()) && "Type already exists");
 
       auto *OffsetList = new (OffsetAlloc.Allocate()) OffsetListT();
       TypeToOffsets[V.getType()] = OffsetList;
diff --git a/llvm/include/llvm/CodeGen/LiveRangeEdit.h b/llvm/include/llvm/CodeGen/LiveRangeEdit.h
index 507258fe665a2..fe7af0a8fb45b 100644
--- a/llvm/include/llvm/CodeGen/LiveRangeEdit.h
+++ b/llvm/include/llvm/CodeGen/LiveRangeEdit.h
@@ -211,7 +211,7 @@ class LiveRangeEdit : private MachineRegisterInfo::Delegate {
   /// by new MI in the index map.
   /// Return the SlotIndex of the new instruction.
   SlotIndex rematerializeAt(MachineBasicBlock &MBB,
-                            MachineBasicBlock::iterator MI, unsigned DestReg,
+                            MachineBasicBlock::iterator MI, Register DestReg,
                             const Remat &RM, const TargetRegisterInfo &,
                             bool Late = false, unsigned SubIdx = 0,
                             MachineInstr *ReplaceIndexMI = nullptr);
diff --git a/llvm/include/llvm/CodeGen/MIRPrinter.h b/llvm/include/llvm/CodeGen/MIRPrinter.h
index 45e30686b6426..5e94418d5fe06 100644
--- a/llvm/include/llvm/CodeGen/MIRPrinter.h
+++ b/llvm/include/llvm/CodeGen/MIRPrinter.h
@@ -34,7 +34,7 @@ void printMIR(raw_ostream &OS, const MachineFunction &MF);
 /// you the correct list of successor blocks in most cases except for things
 /// like jump tables where the basic block references can't easily be found.
 /// The MIRPRinter will skip printing successors if they match the result of
-/// this funciton and the parser will use this function to construct a list if
+/// this function and the parser will use this function to construct a list if
 /// it is missing.
 void guessSuccessors(const MachineBasicBlock &MBB,
                      SmallVectorImpl<MachineBasicBlock*> &Result,
diff --git a/llvm/include/llvm/CodeGen/MachineRegisterInfo.h b/llvm/include/llvm/CodeGen/MachineRegisterInfo.h
index 7f0c24e4e1154..ce447be3af41f 100644
--- a/llvm/include/llvm/CodeGen/MachineRegisterInfo.h
+++ b/llvm/include/llvm/CodeGen/MachineRegisterInfo.h
@@ -452,7 +452,7 @@ class MachineRegisterInfo {
   }
 
   void insertVRegByName(StringRef Name, Register Reg) {
-    assert((Name.empty() || VRegNames.find(Name) == VRegNames.end()) &&
+    assert((Name.empty() || !VRegNames.contains(Name)) &&
            "Named VRegs Must be Unique.");
     if (!Name.empty()) {
       VRegNames.insert(Name);
diff --git a/llvm/include/llvm/CodeGen/MachineSSAContext.h b/llvm/include/llvm/CodeGen/MachineSSAContext.h
index 8bf963da9748f..a3788f1046ec9 100644
--- a/llvm/include/llvm/CodeGen/MachineSSAContext.h
+++ b/llvm/include/llvm/CodeGen/MachineSSAContext.h
@@ -45,6 +45,7 @@ template <> class GenericSSAContext<MachineFunction> {
   using ValueRefT = Register;
   using ConstValueRefT = Register;
   static const Register ValueRefNull;
+  using UseT = MachineOperand;
   using DominatorTreeT = DominatorTreeBase<BlockT, false>;
 
   void setFunction(MachineFunction &Fn);
diff --git a/llvm/include/llvm/CodeGen/RegAllocRegistry.h b/llvm/include/llvm/CodeGen/RegAllocRegistry.h
index e33d16c636277..cd81e084a859b 100644
--- a/llvm/include/llvm/CodeGen/RegAllocRegistry.h
+++ b/llvm/include/llvm/CodeGen/RegAllocRegistry.h
@@ -66,7 +66,7 @@ class RegisterRegAlloc : public RegisterRegAllocBase<RegisterRegAlloc> {
 
 /// RegisterRegAlloc's global Registry tracks allocator registration.
 template <class T>
-MachinePassRegistry<RegisterRegAlloc::FunctionPassCtor>
+MachinePassRegistry<typename RegisterRegAllocBase<T>::FunctionPassCtor>
 RegisterRegAllocBase<T>::Registry;
 
 } // end namespace llvm
diff --git a/llvm/include/llvm/CodeGen/SelectionDAG.h b/llvm/include/llvm/CodeGen/SelectionDAG.h
index 3f15eb6e05013..e032598a0a1f8 100644
--- a/llvm/include/llvm/CodeGen/SelectionDAG.h
+++ b/llvm/include/llvm/CodeGen/SelectionDAG.h
@@ -71,7 +71,6 @@ class FunctionLoweringInfo;
 class FunctionVarLocs;
 class GlobalValue;
 struct KnownBits;
-class LegacyDivergenceAnalysis;
 class LLVMContext;
 class MachineBasicBlock;
 class MachineConstantPoolValue;
@@ -89,6 +88,11 @@ class TargetMachine;
 class TargetSubtargetInfo;
 class Value;
 
+template <typename T> class GenericSSAContext;
+using SSAContext = GenericSSAContext<Function>;
+template <typename T> class GenericUniformityInfo;
+using UniformityInfo = GenericUniformityInfo<SSAContext>;
+
 class SDVTListNode : public FoldingSetNode {
   friend struct FoldingSetTrait<SDVTListNode>;
 
@@ -229,7 +233,7 @@ class SelectionDAG {
   LLVMContext *Context;
   CodeGenOpt::Level OptLevel;
 
-  LegacyDivergenceAnalysis * DA = nullptr;
+  UniformityInfo *UA = nullptr;
   FunctionLoweringInfo * FLI = nullptr;
 
   /// The function-level optimization remark emitter.  Used to emit remarks
@@ -451,7 +455,7 @@ class SelectionDAG {
   /// Prepare this SelectionDAG to process code in the given MachineFunction.
   void init(MachineFunction &NewMF, OptimizationRemarkEmitter &NewORE,
             Pass *PassPtr, const TargetLibraryInfo *LibraryInfo,
-            LegacyDivergenceAnalysis *Divergence, ProfileSummaryInfo *PSIin,
+            UniformityInfo *UA, ProfileSummaryInfo *PSIin,
             BlockFrequencyInfo *BFIin, FunctionVarLocs const *FnVarLocs);
 
   void setFunctionLoweringInfo(FunctionLoweringInfo * FuncInfo) {
@@ -474,7 +478,7 @@ class SelectionDAG {
   const TargetLowering &getTargetLoweringInfo() const { return *TLI; }
   const TargetLibraryInfo &getLibInfo() const { return *LibInfo; }
   const SelectionDAGTargetInfo &getSelectionDAGInfo() const { return *TSI; }
-  const LegacyDivergenceAnalysis *getDivergenceAnalysis() const { return DA; }
+  const UniformityInfo *getUniformityInfo() const { return UA; }
   /// Returns the result of the AssignmentTrackingAnalysis pass if it's
   /// available, otherwise return nullptr.
   const FunctionVarLocs *getFunctionVarLocs() const { return FnVarLocs; }
diff --git a/llvm/include/llvm/CodeGen/SelectionDAGNodes.h b/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
index 187d179e34038..32745ac660d69 100644
--- a/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
+++ b/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
@@ -1608,10 +1608,10 @@ class ConstantSDNode : public SDNode {
 
   bool isOne() const { return Value->isOne(); }
   bool isZero() const { return Value->isZero(); }
-  // NOTE: This is soft-deprecated.  Please use `isZero()` instead.
+  LLVM_DEPRECATED("use isZero instead", "isZero")
   bool isNullValue() const { return isZero(); }
   bool isAllOnes() const { return Value->isMinusOne(); }
-  // NOTE: This is soft-deprecated.  Please use `isAllOnes()` instead.
+  LLVM_DEPRECATED("use isAllOnes instead", "isAllOnes")
   bool isAllOnesValue() const { return isAllOnes(); }
   bool isMaxSignedValue() const { return Value->isMaxValue(true); }
   bool isMinSignedValue() const { return Value->isMinValue(true); }
@@ -1712,6 +1712,10 @@ SDValue peekThroughOneUseBitcasts(SDValue V);
 /// If \p V is not an extracted subvector, it is returned as-is.
 SDValue peekThroughExtractSubvectors(SDValue V);
 
+/// Return the non-truncated source operand of \p V if it exists.
+/// If \p V is not a truncation, it is returned as-is.
+SDValue peekThroughTruncates(SDValue V);
+
 /// Returns true if \p V is a bitwise not operation. Assumes that an all ones
 /// constant is canonicalized to be operand 1.
 bool isBitwiseNot(SDValue V, bool AllowUndefs = false);
diff --git a/llvm/include/llvm/CodeGen/SlotIndexes.h b/llvm/include/llvm/CodeGen/SlotIndexes.h
index 403a94c53dc43..7e013dbf2ab38 100644
--- a/llvm/include/llvm/CodeGen/SlotIndexes.h
+++ b/llvm/include/llvm/CodeGen/SlotIndexes.h
@@ -540,7 +540,7 @@ class raw_ostream;
     SlotIndex insertMachineInstrInMaps(MachineInstr &MI, bool Late = false) {
       assert(!MI.isInsideBundle() &&
              "Instructions inside bundles should use bundle start's slot.");
-      assert(mi2iMap.find(&MI) == mi2iMap.end() && "Instr already indexed.");
+      assert(!mi2iMap.contains(&MI) && "Instr already indexed.");
       // Numbering debug instructions could cause code generation to be
       // affected by debug information.
       assert(!MI.isDebugInstr() && "Cannot number debug instructions.");
diff --git a/llvm/include/llvm/CodeGen/StackProtector.h b/llvm/include/llvm/CodeGen/StackProtector.h
index a5a95c421b2e1..6bbdae69666fe 100644
--- a/llvm/include/llvm/CodeGen/StackProtector.h
+++ b/llvm/include/llvm/CodeGen/StackProtector.h
@@ -63,12 +63,6 @@ class StackProtector : public FunctionPass {
   /// protection when -fstack-protection is used.
   unsigned SSPBufferSize = DefaultSSPBufferSize;
 
-  /// VisitedPHIs - The set of PHI nodes visited when determining
-  /// if a variable's reference has been taken.  This set
-  /// is maintained to ensure we don't visit the same PHI node multiple
-  /// times.
-  SmallPtrSet<const PHINode *, 16> VisitedPHIs;
-
   // A prologue is generated.
   bool HasPrologue = false;
 
@@ -87,22 +81,6 @@ class StackProtector : public FunctionPass {
   /// check fails.
   BasicBlock *CreateFailBB();
 
-  /// ContainsProtectableArray - Check whether the type either is an array or
-  /// contains an array of sufficient size so that we need stack protectors
-  /// for it.
-  /// \param [out] IsLarge is set to true if a protectable array is found and
-  /// it is "large" ( >= ssp-buffer-size).  In the case of a structure with
-  /// multiple arrays, this gets set if any of them is large.
-  bool ContainsProtectableArray(Type *Ty, bool &IsLarge, bool Strong = false,
-                                bool InStruct = false) const;
-
-  /// Check whether a stack allocation has its address taken.
-  bool HasAddressTaken(const Instruction *AI, TypeSize AllocSize);
-
-  /// RequiresStackProtector - Check whether or not this function needs a
-  /// stack protector based upon the stack protector level.
-  bool RequiresStackProtector();
-
 public:
   static char ID; // Pass identification, replacement for typeid.
 
@@ -116,6 +94,11 @@ class StackProtector : public FunctionPass {
   bool runOnFunction(Function &Fn) override;
 
   void copyToMachineFrameInfo(MachineFrameInfo &MFI) const;
+
+  /// Check whether or not \p F needs a stack protector based upon the stack
+  /// protector level.
+  static bool requiresStackProtector(Function *F, SSPLayoutMap *Layout = nullptr);
+
 };
 
 } // end namespace llvm
diff --git a/llvm/include/llvm/CodeGen/TargetLowering.h b/llvm/include/llvm/CodeGen/TargetLowering.h
index 0624dd394275a..f94994a901b11 100644
--- a/llvm/include/llvm/CodeGen/TargetLowering.h
+++ b/llvm/include/llvm/CodeGen/TargetLowering.h
@@ -74,7 +74,6 @@ class GISelKnownBits;
 class IntrinsicInst;
 class IRBuilderBase;
 struct KnownBits;
-class LegacyDivergenceAnalysis;
 class LLVMContext;
 class MachineBasicBlock;
 class MachineFunction;
@@ -1527,15 +1526,16 @@ class TargetLoweringBase {
   EVT getMemValueType(const DataLayout &DL, Type *Ty,
                       bool AllowUnknown = false) const {
     // Lower scalar pointers to native pointer types.
-    if (PointerType *PTy = dyn_cast<PointerType>(Ty))
+    if (auto *PTy = dyn_cast<PointerType>(Ty))
       return getPointerMemTy(DL, PTy->getAddressSpace());
-    else if (VectorType *VTy = dyn_cast<VectorType>(Ty)) {
-      Type *Elm = VTy->getElementType();
-      if (PointerType *PT = dyn_cast<PointerType>(Elm)) {
-        EVT PointerTy(getPointerMemTy(DL, PT->getAddressSpace()));
-        Elm = PointerTy.getTypeForEVT(Ty->getContext());
+
+    if (auto *VTy = dyn_cast<VectorType>(Ty)) {
+      Type *EltTy = VTy->getElementType();
+      if (auto *PTy = dyn_cast<PointerType>(EltTy)) {
+        EVT PointerTy(getPointerMemTy(DL, PTy->getAddressSpace()));
+        EltTy = PointerTy.getTypeForEVT(Ty->getContext());
       }
-      return EVT::getVectorVT(Ty->getContext(), EVT::getEVT(Elm, false),
+      return EVT::getVectorVT(Ty->getContext(), EVT::getEVT(EltTy, false),
                               VTy->getElementCount());
     }
 
@@ -2847,6 +2847,13 @@ class TargetLoweringBase {
                       getApproximateEVTForLLT(ToTy, DL, Ctx));
   }
 
+  /// Return true if zero-extending the specific node Val to type VT2 is free
+  /// (either because it's implicitly zero-extended such as ARM ldrb / ldrh or
+  /// because it's folded such as X86 zero-extending loads).
+  virtual bool isZExtFree(SDValue Val, EVT VT2) const {
+    return isZExtFree(Val.getValueType(), VT2);
+  }
+
   /// Return true if sign-extension from FromTy to ToTy is cheaper than
   /// zero-extension.
   virtual bool isSExtCheaperThanZExt(EVT FromTy, EVT ToTy) const {
@@ -2932,13 +2939,6 @@ class TargetLoweringBase {
     return false;
   }
 
-  /// Return true if zero-extending the specific node Val to type VT2 is free
-  /// (either because it's implicitly zero-extended such as ARM ldrb / ldrh or
-  /// because it's folded such as X86 zero-extending loads).
-  virtual bool isZExtFree(SDValue Val, EVT VT2) const {
-    return isZExtFree(Val.getValueType(), VT2);
-  }
-
   /// Return true if an fpext operation is free (for instance, because
   /// single-precision floating-point numbers are implicitly extended to
   /// double-precision).
@@ -3055,10 +3055,10 @@ class TargetLoweringBase {
     return false;
   }
 
-  /// Return true if it's profitable to narrow operations of type VT1 to
-  /// VT2. e.g. on x86, it's profitable to narrow from i32 to i8 but not from
+  /// Return true if it's profitable to narrow operations of type SrcVT to
+  /// DestVT. e.g. on x86, it's profitable to narrow from i32 to i8 but not from
   /// i32 to i16.
-  virtual bool isNarrowingProfitable(EVT /*VT1*/, EVT /*VT2*/) const {
+  virtual bool isNarrowingProfitable(EVT SrcVT, EVT DestVT) const {
     return false;
   }
 
@@ -3537,7 +3537,7 @@ class TargetLowering : public TargetLoweringBase {
 
   virtual bool isSDNodeSourceOfDivergence(const SDNode *N,
                                           FunctionLoweringInfo *FLI,
-                                          LegacyDivergenceAnalysis *DA) const {
+                                          UniformityInfo *UA) const {
     return false;
   }
 
@@ -3718,7 +3718,8 @@ class TargetLowering : public TargetLoweringBase {
   /// Convert x+y to (VT)((SmallVT)x+(SmallVT)y) if the casts are free.  This
   /// uses isZExtFree and ZERO_EXTEND for the widening cast, but it could be
   /// generalized for targets with other types of implicit widening casts.
-  bool ShrinkDemandedOp(SDValue Op, unsigned BitWidth, const APInt &Demanded,
+  bool ShrinkDemandedOp(SDValue Op, unsigned BitWidth,
+                        const APInt &DemandedBits,
                         TargetLoweringOpt &TLO) const;
 
   /// Look at Op.  At this point, we know that only the DemandedBits bits of the
diff --git a/llvm/include/llvm/DWARFLinker/DWARFLinker.h b/llvm/include/llvm/DWARFLinker/DWARFLinker.h
index ebfac89d4e670..f58bb3f6ec32b 100644
--- a/llvm/include/llvm/DWARFLinker/DWARFLinker.h
+++ b/llvm/include/llvm/DWARFLinker/DWARFLinker.h
@@ -117,19 +117,32 @@ class DwarfEmitter {
   virtual void
   emitAppleTypes(AccelTable<AppleAccelTableStaticTypeData> &Table) = 0;
 
-  /// Emit debug ranges(.debug_ranges, .debug_rnglists) header.
+  /// Emit debug ranges (.debug_ranges, .debug_rnglists) header.
   virtual MCSymbol *emitDwarfDebugRangeListHeader(const CompileUnit &Unit) = 0;
 
-  /// Emit debug ranges(.debug_ranges, .debug_rnglists) fragment.
+  /// Emit debug ranges (.debug_ranges, .debug_rnglists) fragment.
   virtual void
   emitDwarfDebugRangeListFragment(const CompileUnit &Unit,
                                   const AddressRanges &LinkedRanges,
                                   PatchLocation Patch) = 0;
 
-  /// Emit debug ranges(.debug_ranges, .debug_rnglists) footer.
+  /// Emit debug ranges (.debug_ranges, .debug_rnglists) footer.
   virtual void emitDwarfDebugRangeListFooter(const CompileUnit &Unit,
                                              MCSymbol *EndLabel) = 0;
 
+  /// Emit debug locations (.debug_loc, .debug_loclists) header.
+  virtual MCSymbol *emitDwarfDebugLocListHeader(const CompileUnit &Unit) = 0;
+
+  /// Emit debug locations (.debug_loc, .debug_loclists) fragment.
+  virtual void emitDwarfDebugLocListFragment(
+      const CompileUnit &Unit,
+      const DWARFLocationExpressionsVector &LinkedLocationExpression,
+      PatchLocation Patch) = 0;
+
+  /// Emit debug locations (.debug_loc, .debug_loclists) footer.
+  virtual void emitDwarfDebugLocListFooter(const CompileUnit &Unit,
+                                           MCSymbol *EndLabel) = 0;
+
   /// Emit .debug_aranges entries for \p Unit
   virtual void
   emitDwarfDebugArangesTable(const CompileUnit &Unit,
@@ -159,14 +172,6 @@ class DwarfEmitter {
   virtual void emitFDE(uint32_t CIEOffset, uint32_t AddreSize, uint64_t Address,
                        StringRef Bytes) = 0;
 
-  /// Emit the .debug_loc contribution for \p Unit by copying the entries from
-  /// \p Dwarf and offsetting them. Update the location attributes to point to
-  /// the new entries.
-  virtual void emitLocationsForUnit(
-      const CompileUnit &Unit, DWARFContext &Dwarf,
-      std::function<void(StringRef, SmallVectorImpl<uint8_t> &)>
-          ProcessExpr) = 0;
-
   /// Emit the compilation unit header for \p Unit in the
   /// .debug_info section.
   ///
@@ -207,6 +212,9 @@ class DwarfEmitter {
 
   /// Returns size of generated .debug_macro section.
   virtual uint64_t getDebugMacroSectionSize() const = 0;
+
+  /// Returns size of generated .debug_loclists section.
+  virtual uint64_t getLocListsSectionSize() const = 0;
 };
 
 using UnitListTy = std::vector<std::unique_ptr<CompileUnit>>;
@@ -738,6 +746,14 @@ class DWARFLinker {
   /// .debug_rnglists) for \p Unit, patch the attributes referencing it.
   void generateUnitRanges(CompileUnit &Unit, const DWARFFile &File) const;
 
+  using ExpressionHandlerRef = function_ref<void(SmallVectorImpl<uint8_t> &,
+                                                 SmallVectorImpl<uint8_t> &)>;
+
+  /// Compute and emit debug locations (.debug_loc, .debug_loclists)
+  /// for \p Unit, patch the attributes referencing it.
+  void generateUnitLocations(CompileUnit &Unit, const DWARFFile &File,
+                             ExpressionHandlerRef ExprHandler) const;
+
   /// Extract the line tables from the original dwarf, extract the relevant
   /// parts according to the linked function ranges and emit the result in the
   /// .debug_line section.
diff --git a/llvm/include/llvm/DWARFLinker/DWARFLinkerCompileUnit.h b/llvm/include/llvm/DWARFLinker/DWARFLinkerCompileUnit.h
index 39305a3de8680..cfbd13414f703 100644
--- a/llvm/include/llvm/DWARFLinker/DWARFLinkerCompileUnit.h
+++ b/llvm/include/llvm/DWARFLinker/DWARFLinkerCompileUnit.h
@@ -44,6 +44,7 @@ struct PatchLocation {
 };
 
 using RngListAttributesTy = SmallVector<PatchLocation>;
+using LocListAttributesTy = SmallVector<std::pair<PatchLocation, int64_t>>;
 
 /// Stores all information relating to a compile unit, be it in its original
 /// instance in the object file to its brand new cloned and generated DIE tree.
@@ -153,8 +154,7 @@ class CompileUnit {
     return UnitRangeAttribute;
   }
 
-  const std::vector<std::pair<PatchLocation, int64_t>> &
-  getLocationAttributes() const {
+  const LocListAttributesTy &getLocationAttributes() const {
     return LocationAttributes;
   }
 
@@ -289,7 +289,7 @@ class CompileUnit {
   /// original debug_loc section to the liked one. They are stored
   /// along with the PC offset that is to be applied to their
   /// function's address.
-  std::vector<std::pair<PatchLocation, int64_t>> LocationAttributes;
+  LocListAttributesTy LocationAttributes;
 
   /// Accelerator entries for the unit, both for the pub*
   /// sections and the apple* ones.
diff --git a/llvm/include/llvm/DWARFLinker/DWARFStreamer.h b/llvm/include/llvm/DWARFLinker/DWARFStreamer.h
index 69d1f9348449a..c96dd4007c23f 100644
--- a/llvm/include/llvm/DWARFLinker/DWARFStreamer.h
+++ b/llvm/include/llvm/DWARFLinker/DWARFStreamer.h
@@ -105,6 +105,19 @@ class DwarfStreamer : public DwarfEmitter {
   void emitDwarfDebugRangeListFooter(const CompileUnit &Unit,
                                      MCSymbol *EndLabel) override;
 
+  /// Emit debug locations(.debug_loc, .debug_loclists) header.
+  MCSymbol *emitDwarfDebugLocListHeader(const CompileUnit &Unit) override;
+
+  /// Emit debug ranges(.debug_loc, .debug_loclists) fragment.
+  void emitDwarfDebugLocListFragment(
+      const CompileUnit &Unit,
+      const DWARFLocationExpressionsVector &LinkedLocationExpression,
+      PatchLocation Patch) override;
+
+  /// Emit debug ranges(.debug_loc, .debug_loclists) footer.
+  void emitDwarfDebugLocListFooter(const CompileUnit &Unit,
+                                   MCSymbol *EndLabel) override;
+
   /// Emit .debug_aranges entries for \p Unit
   void emitDwarfDebugArangesTable(const CompileUnit &Unit,
                                   const AddressRanges &LinkedRanges) override;
@@ -115,14 +128,6 @@ class DwarfStreamer : public DwarfEmitter {
     return RngListsSectionSize;
   }
 
-  /// Emit the debug_loc contribution for \p Unit by copying the entries from
-  /// \p Dwarf and offsetting them. Update the location attributes to point to
-  /// the new entries.
-  void emitLocationsForUnit(
-      const CompileUnit &Unit, DWARFContext &Dwarf,
-      std::function<void(StringRef, SmallVectorImpl<uint8_t> &)> ProcessExpr)
-      override;
-
   /// Emit the line table described in \p Rows into the debug_line section.
   void emitLineTableForUnit(MCDwarfLineTableParams Params,
                             StringRef PrologueBytes, unsigned MinInstLength,
@@ -181,6 +186,10 @@ class DwarfStreamer : public DwarfEmitter {
     return MacroSectionSize;
   }
 
+  uint64_t getLocListsSectionSize() const override {
+    return LocListsSectionSize;
+  }
+
   void emitMacroTables(DWARFContext *Context,
                        const Offset2UnitMap &UnitMacroMap,
                        OffsetsStringPool &StringPool) override;
@@ -210,6 +219,18 @@ class DwarfStreamer : public DwarfEmitter {
                                            const AddressRanges &LinkedRanges,
                                            PatchLocation Patch);
 
+  /// Emit piece of .debug_loc for \p LinkedRanges.
+  void emitDwarfDebugLocTableFragment(
+      const CompileUnit &Unit,
+      const DWARFLocationExpressionsVector &LinkedLocationExpression,
+      PatchLocation Patch);
+
+  /// Emit piece of .debug_loclists for \p LinkedRanges.
+  void emitDwarfDebugLocListsTableFragment(
+      const CompileUnit &Unit,
+      const DWARFLocationExpressionsVector &LinkedLocationExpression,
+      PatchLocation Patch);
+
   /// \defgroup MCObjects MC layer objects constructed by the streamer
   /// @{
   std::unique_ptr<MCRegisterInfo> MRI;
@@ -234,6 +255,7 @@ class DwarfStreamer : public DwarfEmitter {
   uint64_t RangesSectionSize = 0;
   uint64_t RngListsSectionSize = 0;
   uint64_t LocSectionSize = 0;
+  uint64_t LocListsSectionSize = 0;
   uint64_t LineSectionSize = 0;
   uint64_t FrameSectionSize = 0;
   uint64_t DebugInfoSectionSize = 0;
diff --git a/llvm/include/llvm/DWARFLinkerParallel/AddressesMap.h b/llvm/include/llvm/DWARFLinkerParallel/AddressesMap.h
new file mode 100644
index 0000000000000..eadd7120f70c7
--- /dev/null
+++ b/llvm/include/llvm/DWARFLinkerParallel/AddressesMap.h
@@ -0,0 +1,71 @@
+//===- AddressesMap.h -------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_DWARFLINKERPARALLEL_ADDRESSESMAP_H
+#define LLVM_DWARFLINKERPARALLEL_ADDRESSESMAP_H
+
+#include "llvm/ADT/AddressRanges.h"
+#include "llvm/DebugInfo/DWARF/DWARFDie.h"
+#include <cstdint>
+
+namespace llvm {
+namespace dwarflinker_parallel {
+
+/// Mapped value in the address map is the offset to apply to the
+/// linked address.
+using RangesTy = AddressRangesMap;
+
+/// AddressesMap represents information about valid addresses used
+/// by debug information. Valid addresses are those which points to
+/// live code sections. i.e. relocations for these addresses point
+/// into sections which would be/are placed into resulting binary.
+class AddressesMap {
+public:
+  virtual ~AddressesMap() = default;
+
+  /// Checks that there are valid relocations in the .debug_info
+  /// section.
+  virtual bool hasValidRelocs() = 0;
+
+  /// Checks that the specified variable \p DIE references the live code
+  /// section and returns the relocation adjustment value (to get the linked
+  /// address this value might be added to the source variable address).
+  /// Allowed kinds of input DIE: DW_TAG_variable, DW_TAG_constant.
+  /// \returns relocation adjustment value or std::nullopt if there is no
+  /// corresponding live address.
+  virtual std::optional<int64_t>
+  getVariableRelocAdjustment(const DWARFDie &DIE) = 0;
+
+  /// Checks that the specified subprogram \p DIE references the live code
+  /// section and returns the relocation adjustment value (to get the linked
+  /// address this value might be added to the source subprogram address).
+  /// Allowed kinds of input DIE: DW_TAG_subprogram, DW_TAG_label.
+  /// \returns relocation adjustment value or std::nullopt if there is no
+  /// corresponding live address.
+  virtual std::optional<int64_t>
+  getSubprogramRelocAdjustment(const DWARFDie &DIE) = 0;
+
+  /// Apply the valid relocations to the buffer \p Data, taking into
+  /// account that Data is at \p BaseOffset in the .debug_info section.
+  ///
+  /// \returns true whether any reloc has been applied.
+  virtual bool applyValidRelocs(MutableArrayRef<char> Data, uint64_t BaseOffset,
+                                bool IsLittleEndian) = 0;
+
+  /// Returns all valid functions address ranges (i.e., those ranges
+  /// which points to sections with code).
+  virtual RangesTy &getValidAddressRanges() = 0;
+
+  /// Erases all data.
+  virtual void clear() = 0;
+};
+
+} // end of namespace dwarflinker_parallel
+} // end namespace llvm
+
+#endif // LLVM_DWARFLINKERPARALLEL_ADDRESSESMAP_H
diff --git a/llvm/include/llvm/DWARFLinkerParallel/DWARFLinker.h b/llvm/include/llvm/DWARFLinkerParallel/DWARFLinker.h
index b1169fdd85412..b7f9b94dca0e6 100644
--- a/llvm/include/llvm/DWARFLinkerParallel/DWARFLinker.h
+++ b/llvm/include/llvm/DWARFLinkerParallel/DWARFLinker.h
@@ -9,6 +9,8 @@
 #ifndef LLVM_DWARFLINKERPARALLEL_DWARFLINKER_H
 #define LLVM_DWARFLINKERPARALLEL_DWARFLINKER_H
 
+#include "llvm/DWARFLinkerParallel/AddressesMap.h"
+
 namespace llvm {
 namespace dwarflinker_parallel {} // end namespace dwarflinker_parallel
 } // end namespace llvm
diff --git a/llvm/include/llvm/DebugInfo/CodeView/CodeView.h b/llvm/include/llvm/DebugInfo/CodeView/CodeView.h
index 010a82dd0e232..a9ad99a1d0a84 100644
--- a/llvm/include/llvm/DebugInfo/CodeView/CodeView.h
+++ b/llvm/include/llvm/DebugInfo/CodeView/CodeView.h
@@ -138,8 +138,8 @@ enum class CPUType : uint16_t {
   D3D11_Shader = 0x100,
 };
 
-/// These values correspond to the CV_CFL_LANG enumeration, and are documented
-/// here: https://msdn.microsoft.com/en-us/library/bw3aekw6.aspx
+/// These values correspond to the CV_CFL_LANG enumeration in the Microsoft
+/// Debug Interface Access SDK
 enum SourceLanguage : uint8_t {
   C = 0x00,
   Cpp = 0x01,
@@ -158,6 +158,8 @@ enum SourceLanguage : uint8_t {
   JScript = 0x0e,
   MSIL = 0x0f,
   HLSL = 0x10,
+  ObjC = 0x11,
+  ObjCpp = 0x12,
 
   Rust = 0x15,
 
diff --git a/llvm/include/llvm/DebugInfo/DWARF/DWARFAbbreviationDeclaration.h b/llvm/include/llvm/DebugInfo/DWARF/DWARFAbbreviationDeclaration.h
index cff13dcf2955f..44d77a3769ddc 100644
--- a/llvm/include/llvm/DebugInfo/DWARF/DWARFAbbreviationDeclaration.h
+++ b/llvm/include/llvm/DebugInfo/DWARF/DWARFAbbreviationDeclaration.h
@@ -12,6 +12,7 @@
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/iterator_range.h"
 #include "llvm/BinaryFormat/Dwarf.h"
+#include "llvm/DebugInfo/DWARF/DWARFFormValue.h"
 #include <cassert>
 #include <cstddef>
 #include <cstdint>
@@ -19,7 +20,6 @@
 namespace llvm {
 
 class DataExtractor;
-class DWARFFormValue;
 class DWARFUnit;
 class raw_ostream;
 
@@ -39,6 +39,13 @@ class DWARFAbbreviationDeclaration {
         this->ByteSize.ByteSize = *ByteSize;
     }
 
+    DWARFFormValue getFormValue() const {
+      if (Form == dwarf::DW_FORM_implicit_const)
+        return DWARFFormValue::createFromSValue(Form, getImplicitConstValue());
+
+      return DWARFFormValue(Form);
+    }
+
     dwarf::Attribute Attr;
     dwarf::Form Form;
 
diff --git a/llvm/include/llvm/DebugInfo/DWARF/DWARFContext.h b/llvm/include/llvm/DebugInfo/DWARF/DWARFContext.h
index df903b967ef68..4eba79a7215f7 100644
--- a/llvm/include/llvm/DebugInfo/DWARF/DWARFContext.h
+++ b/llvm/include/llvm/DebugInfo/DWARF/DWARFContext.h
@@ -445,7 +445,16 @@ class DWARFContext : public DIContext {
   /// address.
   /// TODO: change input parameter from "uint64_t Address"
   ///       into "SectionedAddress Address"
-  DWARFCompileUnit *getCompileUnitForAddress(uint64_t Address);
+  DWARFCompileUnit *getCompileUnitForCodeAddress(uint64_t Address);
+
+  /// Return the compile unit which contains data with the provided address.
+  /// Note: This is more expensive than `getCompileUnitForAddress`, as if
+  /// `Address` isn't found in the CU ranges (which is cheap), then it falls
+  /// back to an expensive O(n) walk of all CU's looking for data that spans the
+  /// address.
+  /// TODO: change input parameter from "uint64_t Address" into
+  ///       "SectionedAddress Address"
+  DWARFCompileUnit *getCompileUnitForDataAddress(uint64_t Address);
 
   /// Returns whether CU/TU should be populated manually. TU Index populated
   /// manually only for DWARF5.
diff --git a/llvm/include/llvm/DebugInfo/DWARF/DWARFFormValue.h b/llvm/include/llvm/DebugInfo/DWARF/DWARFFormValue.h
index 1951d085e5dc5..f22e4c37d1b12 100644
--- a/llvm/include/llvm/DebugInfo/DWARF/DWARFFormValue.h
+++ b/llvm/include/llvm/DebugInfo/DWARF/DWARFFormValue.h
@@ -349,6 +349,14 @@ toBlock(const std::optional<DWARFFormValue> &V) {
   return std::nullopt;
 }
 
+/// Check whether specified \p Form belongs to the \p FC class.
+/// \param Form an attribute form.
+/// \param FC an attribute form class to check.
+/// \param DwarfVersion the version of DWARF debug info keeping the attribute.
+/// \returns true if specified \p Form belongs to the \p FC class.
+bool doesFormBelongToClass(dwarf::Form Form, DWARFFormValue::FormClass FC,
+                           uint16_t DwarfVersion);
+
 } // end namespace dwarf
 
 } // end namespace llvm
diff --git a/llvm/include/llvm/DebugInfo/GSYM/FileWriter.h b/llvm/include/llvm/DebugInfo/GSYM/FileWriter.h
index 84b5687597227..74811240e0b5e 100644
--- a/llvm/include/llvm/DebugInfo/GSYM/FileWriter.h
+++ b/llvm/include/llvm/DebugInfo/GSYM/FileWriter.h
@@ -113,6 +113,8 @@ class FileWriter {
     return OS;
   }
 
+  llvm::support::endianness getByteOrder() const { return ByteOrder; }
+
 private:
   FileWriter(const FileWriter &rhs) = delete;
   void operator=(const FileWriter &rhs) = delete;
diff --git a/llvm/include/llvm/DebugInfo/GSYM/FunctionInfo.h b/llvm/include/llvm/DebugInfo/GSYM/FunctionInfo.h
index 713e3c239e95f..8198ffa8621b2 100644
--- a/llvm/include/llvm/DebugInfo/GSYM/FunctionInfo.h
+++ b/llvm/include/llvm/DebugInfo/GSYM/FunctionInfo.h
@@ -90,6 +90,10 @@ struct FunctionInfo {
   uint32_t Name; ///< String table offset in the string table.
   std::optional<LineTable> OptLineTable;
   std::optional<InlineInfo> Inline;
+  /// If we encode a FunctionInfo during segmenting so we know its size, we can
+  /// cache that encoding here so we don't need to re-encode it when saving the
+  /// GSYM file.
+  SmallString<32> EncodingCache;
 
   FunctionInfo(uint64_t Addr = 0, uint64_t Size = 0, uint32_t N = 0)
       : Range(Addr, Addr + Size), Name(N) {}
@@ -140,6 +144,17 @@ struct FunctionInfo {
   /// function info that was successfully written into the stream.
   llvm::Expected<uint64_t> encode(FileWriter &O) const;
 
+  /// Encode this function info into the internal byte cache and return the size
+  /// in bytes.
+  ///
+  /// When segmenting GSYM files we need to know how big each FunctionInfo will
+  /// encode into so we can generate segments of the right size. We don't want
+  /// to have to encode a FunctionInfo twice, so we can cache the encoded bytes
+  /// and re-use then when calling FunctionInfo::encode(...).
+  ///
+  /// \returns The size in bytes of the FunctionInfo if it were to be encoded
+  /// into a byte stream.
+  uint64_t cacheEncoding();
 
   /// Lookup an address within a FunctionInfo object's data stream.
   ///
diff --git a/llvm/include/llvm/DebugInfo/GSYM/GsymCreator.h b/llvm/include/llvm/DebugInfo/GSYM/GsymCreator.h
index 2eac8b43f006c..c7cee0593c035 100644
--- a/llvm/include/llvm/DebugInfo/GSYM/GsymCreator.h
+++ b/llvm/include/llvm/DebugInfo/GSYM/GsymCreator.h
@@ -137,6 +137,8 @@ class GsymCreator {
   StringTableBuilder StrTab;
   StringSet<> StringStorage;
   DenseMap<llvm::gsym::FileEntry, uint32_t> FileEntryToIndex;
+  // Needed for mapping string offsets back to the string stored in \a StrTab.
+  DenseMap<uint64_t, CachedHashStringRef> StringOffsetMap;
   std::vector<llvm::gsym::FileEntry> Files;
   std::vector<uint8_t> UUID;
   std::optional<AddressRanges> ValidTextRanges;
@@ -145,6 +147,149 @@ class GsymCreator {
   bool Finalized = false;
   bool Quiet;
 
+
+  /// Get the first function start address.
+  ///
+  /// \returns The start address of the first FunctionInfo or std::nullopt if
+  /// there are no function infos.
+  std::optional<uint64_t> getFirstFunctionAddress() const;
+
+  /// Get the last function address.
+  ///
+  /// \returns The start address of the last FunctionInfo or std::nullopt if
+  /// there are no function infos.
+  std::optional<uint64_t> getLastFunctionAddress() const;
+
+  /// Get the base address to use for this GSYM file.
+  ///
+  /// \returns The base address to put into the header and to use when creating
+  ///          the address offset table or std::nullpt if there are no valid
+  ///          function infos or if the base address wasn't specified.
+  std::optional<uint64_t> getBaseAddress() const;
+
+  /// Get the size of an address offset in the address offset table.
+  ///
+  /// GSYM files store offsets from the base address in the address offset table
+  /// and we store the size of the address offsets in the GSYM header. This
+  /// function will calculate the size in bytes of these address offsets based
+  /// on the current contents of the GSYM file.
+  ///
+  /// \returns The size in byets of the address offsets.
+  uint8_t getAddressOffsetSize() const;
+
+  /// Get the maximum address offset for the current address offset size.
+  ///
+  /// This is used when creating the address offset table to ensure we have
+  /// values that are in range so we don't end up truncating address offsets
+  /// when creating GSYM files as the code evolves.
+  ///
+  /// \returns The maximum address offset value that will be encoded into a GSYM
+  /// file.
+  uint64_t getMaxAddressOffset() const;
+
+  /// Calculate the byte size of the GSYM header and tables sizes.
+  ///
+  /// This function will calculate the exact size in bytes of the encocded GSYM
+  /// for the following items:
+  /// - The GSYM header
+  /// - The Address offset table
+  /// - The Address info offset table
+  /// - The file table
+  /// - The string table
+  ///
+  /// This is used to help split GSYM files into segments.
+  ///
+  /// \returns Size in bytes the GSYM header and tables.
+  uint64_t calculateHeaderAndTableSize() const;
+
+  /// Copy a FunctionInfo from the \a SrcGC GSYM creator into this creator.
+  ///
+  /// Copy the function info and only the needed files and strings and add a
+  /// converted FunctionInfo into this object. This is used to segment GSYM
+  /// files into separate files while only transferring the files and strings
+  /// that are needed from \a SrcGC.
+  ///
+  /// \param SrcGC The source gsym creator to copy from.
+  /// \param FuncInfoIdx The function info index within \a SrcGC to copy.
+  /// \returns The number of bytes it will take to encode the function info in
+  /// this GsymCreator. This helps calculate the size of the current GSYM
+  /// segment file.
+  uint64_t copyFunctionInfo(const GsymCreator &SrcGC, size_t FuncInfoIdx);
+
+  /// Copy a string from \a SrcGC into this object.
+  ///
+  /// Copy a string from \a SrcGC by string table offset into this GSYM creator.
+  /// If a string has already been copied, the uniqued string table offset will
+  /// be returned, otherwise the string will be copied and a unique offset will
+  /// be returned.
+  ///
+  /// \param SrcGC The source gsym creator to copy from.
+  /// \param StrOff The string table offset from \a SrcGC to copy.
+  /// \returns The new string table offset of the string within this object.
+  uint32_t copyString(const GsymCreator &SrcGC, uint32_t StrOff);
+
+  /// Copy a file from \a SrcGC into this object.
+  ///
+  /// Copy a file from \a SrcGC by file index into this GSYM creator. Files
+  /// consist of two string table entries, one for the directory and one for the
+  /// filename, this function will copy any needed strings ensure the file is
+  /// uniqued within this object. If a file already exists in this GSYM creator
+  /// the uniqued index will be returned, else the stirngs will be copied and
+  /// the new file index will be returned.
+  ///
+  /// \param SrcGC The source gsym creator to copy from.
+  /// \param FileIdx The 1 based file table index within \a SrcGC to copy. A
+  /// file index of zero will always return zero as the zero is a reserved file
+  /// index that means no file.
+  /// \returns The new file index of the file within this object.
+  uint32_t copyFile(const GsymCreator &SrcGC, uint32_t FileIdx);
+
+  /// Inserts a FileEntry into the file table.
+  ///
+  /// This is used to insert a file entry in a thread safe way into this object.
+  ///
+  /// \param FE A file entry object that contains valid string table offsets
+  /// from this object already.
+  uint32_t insertFileEntry(FileEntry FE);
+
+  /// Fixup any string and file references by updating any file indexes and
+  /// strings offsets in the InlineInfo parameter.
+  ///
+  /// When copying InlineInfo entries, we can simply make a copy of the object
+  /// and then fixup the files and strings for efficiency.
+  ///
+  /// \param SrcGC The source gsym creator to copy from.
+  /// \param II The inline info that contains file indexes and string offsets
+  /// that come from \a SrcGC. The entries will be updated by coping any files
+  /// and strings over into this object.
+  void fixupInlineInfo(const GsymCreator &SrcGC, InlineInfo &II);
+
+  /// Get the first function info address from this GSYM file.
+  ///
+  /// This is used to add a suffix to segmented GSYM files to indicate the first
+  /// address for the first function info within the file.
+  ///
+  /// \returns The first function info address.
+  uint64_t getFirstFunctionInfoAddress() const;
+
+  /// Save this GSYM file into segments that are roughly \a SegmentSize in size.
+  ///
+  /// When segemented GSYM files are saved to disk, they will use \a Path as a
+  /// prefix and then have the first function info address appended to the path
+  /// when each segment is saved. Each segmented GSYM file has a only the
+  /// strings and files that are needed to save the function infos that are in
+  /// each segment. These smaller files are easy to compress and download
+  /// separately and allow for efficient lookups with very large GSYM files and
+  /// segmenting them allows servers to download only the segments that are
+  /// needed.
+  ///
+  /// \param Path The path prefix to use when saving the GSYM files.
+  /// \param ByteOrder The endianness to use when saving the file.
+  /// \param SegmentSize The size in bytes to segment the GSYM file into.
+  llvm::Error saveSegments(StringRef Path,
+                           llvm::support::endianness ByteOrder,
+                           uint64_t SegmentSize) const;
+
 public:
   GsymCreator(bool Quiet = false);
 
@@ -152,8 +297,18 @@ class GsymCreator {
   ///
   /// \param Path The file path to save the GSYM file to.
   /// \param ByteOrder The endianness to use when saving the file.
+  /// \param SegmentSize The size in bytes to segment the GSYM file into. If
+  ///                    this option is set this function will create N segments
+  ///                    that are all around \a SegmentSize bytes in size. This
+  ///                    allows a very large GSYM file to be broken up into
+  ///                    shards. Each GSYM file will have its own file table,
+  ///                    and string table that only have the files and strings
+  ///                    needed for the shared. If this argument has no value,
+  ///                    a single GSYM file that contains all function
+  ///                    information will be created.
   /// \returns An error object that indicates success or failure of the save.
-  llvm::Error save(StringRef Path, llvm::support::endianness ByteOrder) const;
+  llvm::Error save(StringRef Path, llvm::support::endianness ByteOrder,
+                   std::optional<uint64_t> SegmentSize = std::nullopt) const;
 
   /// Encode a GSYM into the file writer stream at the current position.
   ///
@@ -291,6 +446,28 @@ class GsymCreator {
 
   /// Whether the transformation should be quiet, i.e. not output warnings.
   bool isQuiet() const { return Quiet; }
+
+
+  /// Create a segmented GSYM creator starting with function info index
+  /// \a FuncIdx.
+  ///
+  /// This function will create a GsymCreator object that will encode into
+  /// roughly \a SegmentSize bytes and return it. It is used by the private
+  /// saveSegments(...) function and also is used by the GSYM unit tests to test
+  /// segmenting of GSYM files. The returned GsymCreator can be finalized and
+  /// encoded.
+  ///
+  /// \param [in] SegmentSize The size in bytes to roughly segment the GSYM file
+  /// into.
+  /// \param [in,out] FuncIdx The index of the first function info to encode
+  /// into the returned GsymCreator. This index will be updated so it can be
+  /// used in subsequent calls to this function to allow more segments to be
+  /// created.
+  /// \returns An expected unique pointer to a GsymCreator or an error. The
+  /// returned unique pointer can be NULL if there are no more functions to
+  /// encode.
+  llvm::Expected<std::unique_ptr<GsymCreator>>
+  createSegment(uint64_t SegmentSize, size_t &FuncIdx) const;
 };
 
 } // namespace gsym
diff --git a/llvm/include/llvm/DebugInfo/GSYM/LookupResult.h b/llvm/include/llvm/DebugInfo/GSYM/LookupResult.h
index 44e58f5220028..9ccc96fbb4d5c 100644
--- a/llvm/include/llvm/DebugInfo/GSYM/LookupResult.h
+++ b/llvm/include/llvm/DebugInfo/GSYM/LookupResult.h
@@ -52,6 +52,16 @@ struct LookupResult {
   std::string getSourceFile(uint32_t Index) const;
 };
 
+inline bool operator==(const LookupResult &LHS, const LookupResult &RHS) {
+  if (LHS.LookupAddr != RHS.LookupAddr)
+    return false;
+  if (LHS.FuncRange != RHS.FuncRange)
+    return false;
+  if (LHS.FuncName != RHS.FuncName)
+    return false;
+  return LHS.Locations == RHS.Locations;
+}
+
 raw_ostream &operator<<(raw_ostream &OS, const LookupResult &R);
 
 } // namespace gsym
diff --git a/llvm/include/llvm/DebugInfo/MSF/MSFError.h b/llvm/include/llvm/DebugInfo/MSF/MSFError.h
index 88c823f603cf5..0d0a43102a9b0 100644
--- a/llvm/include/llvm/DebugInfo/MSF/MSFError.h
+++ b/llvm/include/llvm/DebugInfo/MSF/MSFError.h
@@ -64,6 +64,7 @@ class MSFError : public ErrorInfo<MSFError, StringError> {
     case msf_error_code::stream_directory_overflow:
       return true;
     }
+    llvm_unreachable("msf error code not implemented");
   }
 
   static char ID;
diff --git a/llvm/include/llvm/Demangle/ItaniumDemangle.h b/llvm/include/llvm/Demangle/ItaniumDemangle.h
index 374bbd196b113..acdf570e1a6ba 100644
--- a/llvm/include/llvm/Demangle/ItaniumDemangle.h
+++ b/llvm/include/llvm/Demangle/ItaniumDemangle.h
@@ -27,18 +27,18 @@
 #include <cstring>
 #include <limits>
 #include <new>
-#include <type_traits>
 #include <utility>
 
 DEMANGLE_NAMESPACE_BEGIN
 
 template <class T, size_t N> class PODSmallVector {
-  static_assert(std::is_trivial_v<T>, "T is required to be a trivial type");
+  static_assert(std::is_pod<T>::value,
+                "T is required to be a plain old data type");
 
   T *First = nullptr;
   T *Last = nullptr;
   T *Cap = nullptr;
-  T Inline[N] = {};
+  T Inline[N] = {0};
 
   bool isInline() const { return First == Inline; }
 
@@ -537,6 +537,8 @@ struct AbiTagAttr : Node {
 
   template<typename Fn> void match(Fn F) const { F(Base, Tag); }
 
+  StringView getBaseName() const override { return Base->getBaseName(); }
+
   void printLeft(OutputBuffer &OB) const override {
     Base->printLeft(OB);
     OB += "[abi:";
diff --git a/llvm/include/llvm/ExecutionEngine/JITLink/JITLink.h b/llvm/include/llvm/ExecutionEngine/JITLink/JITLink.h
index f9d6679ef2427..2b6696c7fdffd 100644
--- a/llvm/include/llvm/ExecutionEngine/JITLink/JITLink.h
+++ b/llvm/include/llvm/ExecutionEngine/JITLink/JITLink.h
@@ -704,11 +704,11 @@ class Section {
   /// Set the protection flags for this section.
   void setMemProt(orc::MemProt Prot) { this->Prot = Prot; }
 
-  /// Get the deallocation policy for this section.
-  orc::MemDeallocPolicy getMemDeallocPolicy() const { return MDP; }
+  /// Get the memory lifetime policy for this section.
+  orc::MemLifetimePolicy getMemLifetimePolicy() const { return MLP; }
 
-  /// Set the deallocation policy for this section.
-  void setMemDeallocPolicy(orc::MemDeallocPolicy MDP) { this->MDP = MDP; }
+  /// Set the memory lifetime policy for this section.
+  void setMemLifetimePolicy(orc::MemLifetimePolicy MLP) { this->MLP = MLP; }
 
   /// Returns the ordinal for this section.
   SectionOrdinal getOrdinal() const { return SecOrdinal; }
@@ -773,7 +773,7 @@ class Section {
 
   StringRef Name;
   orc::MemProt Prot;
-  orc::MemDeallocPolicy MDP = orc::MemDeallocPolicy::Standard;
+  orc::MemLifetimePolicy MLP = orc::MemLifetimePolicy::Standard;
   SectionOrdinal SecOrdinal = 0;
   BlockSet Blocks;
   SymbolSet Symbols;
@@ -826,7 +826,7 @@ class SectionRange {
 
 class LinkGraph {
 private:
-  using SectionList = std::vector<std::unique_ptr<Section>>;
+  using SectionMap = DenseMap<StringRef, std::unique_ptr<Section>>;
   using ExternalSymbolSet = DenseSet<Symbol *>;
   using BlockSet = DenseSet<Block *>;
 
@@ -865,7 +865,7 @@ class LinkGraph {
   }
 
   static iterator_range<Section::const_block_iterator>
-  getSectionConstBlocks(Section &S) {
+  getSectionConstBlocks(const Section &S) {
     return S.blocks();
   }
 
@@ -875,15 +875,27 @@ class LinkGraph {
   }
 
   static iterator_range<Section::const_symbol_iterator>
-  getSectionConstSymbols(Section &S) {
+  getSectionConstSymbols(const Section &S) {
     return S.symbols();
   }
 
+  struct GetSectionMapEntryValue {
+    Section &operator()(SectionMap::value_type &KV) const { return *KV.second; }
+  };
+
+  struct GetSectionMapEntryConstValue {
+    const Section &operator()(const SectionMap::value_type &KV) const {
+      return *KV.second;
+    }
+  };
+
 public:
   using external_symbol_iterator = ExternalSymbolSet::iterator;
 
-  using section_iterator = pointee_iterator<SectionList::iterator>;
-  using const_section_iterator = pointee_iterator<SectionList::const_iterator>;
+  using section_iterator =
+      mapped_iterator<SectionMap::iterator, GetSectionMapEntryValue>;
+  using const_section_iterator =
+      mapped_iterator<SectionMap::const_iterator, GetSectionMapEntryConstValue>;
 
   template <typename OuterItrT, typename InnerItrT, typename T,
             iterator_range<InnerItrT> getInnerRange(
@@ -933,18 +945,17 @@ class LinkGraph {
   };
 
   using defined_symbol_iterator =
-      nested_collection_iterator<const_section_iterator,
-                                 Section::symbol_iterator, Symbol *,
-                                 getSectionSymbols>;
+      nested_collection_iterator<section_iterator, Section::symbol_iterator,
+                                 Symbol *, getSectionSymbols>;
 
   using const_defined_symbol_iterator =
       nested_collection_iterator<const_section_iterator,
                                  Section::const_symbol_iterator, const Symbol *,
                                  getSectionConstSymbols>;
 
-  using block_iterator = nested_collection_iterator<const_section_iterator,
-                                                    Section::block_iterator,
-                                                    Block *, getSectionBlocks>;
+  using block_iterator =
+      nested_collection_iterator<section_iterator, Section::block_iterator,
+                                 Block *, getSectionBlocks>;
 
   using const_block_iterator =
       nested_collection_iterator<const_section_iterator,
@@ -1041,14 +1052,9 @@ class LinkGraph {
 
   /// Create a section with the given name, protection flags, and alignment.
   Section &createSection(StringRef Name, orc::MemProt Prot) {
-    assert(llvm::none_of(Sections,
-                         [&](std::unique_ptr<Section> &Sec) {
-                           return Sec->getName() == Name;
-                         }) &&
-           "Duplicate section name");
+    assert(!Sections.count(Name) && "Duplicate section name");
     std::unique_ptr<Section> Sec(new Section(Name, Prot, Sections.size()));
-    Sections.push_back(std::move(Sec));
-    return *Sections.back();
+    return *Sections.insert(std::make_pair(Name, std::move(Sec))).first->second;
   }
 
   /// Create a content block.
@@ -1207,29 +1213,39 @@ class LinkGraph {
   }
 
   iterator_range<section_iterator> sections() {
-    return make_range(section_iterator(Sections.begin()),
-                      section_iterator(Sections.end()));
+    return make_range(
+        section_iterator(Sections.begin(), GetSectionMapEntryValue()),
+        section_iterator(Sections.end(), GetSectionMapEntryValue()));
+  }
+
+  iterator_range<const_section_iterator> sections() const {
+    return make_range(
+        const_section_iterator(Sections.begin(),
+                               GetSectionMapEntryConstValue()),
+        const_section_iterator(Sections.end(), GetSectionMapEntryConstValue()));
   }
 
-  SectionList::size_type sections_size() const { return Sections.size(); }
+  size_t sections_size() const { return Sections.size(); }
 
   /// Returns the section with the given name if it exists, otherwise returns
   /// null.
   Section *findSectionByName(StringRef Name) {
-    for (auto &S : sections())
-      if (S.getName() == Name)
-        return &S;
-    return nullptr;
+    auto I = Sections.find(Name);
+    if (I == Sections.end())
+      return nullptr;
+    return I->second.get();
   }
 
   iterator_range<block_iterator> blocks() {
-    return make_range(block_iterator(Sections.begin(), Sections.end()),
-                      block_iterator(Sections.end(), Sections.end()));
+    auto Secs = sections();
+    return make_range(block_iterator(Secs.begin(), Secs.end()),
+                      block_iterator(Secs.end(), Secs.end()));
   }
 
   iterator_range<const_block_iterator> blocks() const {
-    return make_range(const_block_iterator(Sections.begin(), Sections.end()),
-                      const_block_iterator(Sections.end(), Sections.end()));
+    auto Secs = sections();
+    return make_range(const_block_iterator(Secs.begin(), Secs.end()),
+                      const_block_iterator(Secs.end(), Secs.end()));
   }
 
   iterator_range<external_symbol_iterator> external_symbols() {
@@ -1241,14 +1257,15 @@ class LinkGraph {
   }
 
   iterator_range<defined_symbol_iterator> defined_symbols() {
-    return make_range(defined_symbol_iterator(Sections.begin(), Sections.end()),
-                      defined_symbol_iterator(Sections.end(), Sections.end()));
+    auto Secs = sections();
+    return make_range(defined_symbol_iterator(Secs.begin(), Secs.end()),
+                      defined_symbol_iterator(Secs.end(), Secs.end()));
   }
 
   iterator_range<const_defined_symbol_iterator> defined_symbols() const {
-    return make_range(
-        const_defined_symbol_iterator(Sections.begin(), Sections.end()),
-        const_defined_symbol_iterator(Sections.end(), Sections.end()));
+    auto Secs = sections();
+    return make_range(const_defined_symbol_iterator(Secs.begin(), Secs.end()),
+                      const_defined_symbol_iterator(Secs.end(), Secs.end()));
   }
 
   /// Make the given symbol external (must not already be external).
@@ -1447,11 +1464,10 @@ class LinkGraph {
   /// Remove a section. The section reference is defunct after calling this
   /// function and should no longer be used.
   void removeSection(Section &Sec) {
-    auto I = llvm::find_if(Sections, [&Sec](const std::unique_ptr<Section> &S) {
-      return S.get() == &Sec;
-    });
-    assert(I != Sections.end() && "Section does not appear in this graph");
-    Sections.erase(I);
+    assert(Sections.count(Sec.getName()) && "Section not found");
+    assert(Sections.find(Sec.getName())->second.get() == &Sec &&
+           "Section map entry invalid");
+    Sections.erase(Sec.getName());
   }
 
   /// Accessor for the AllocActions object for this graph. This can be used to
@@ -1474,7 +1490,7 @@ class LinkGraph {
   unsigned PointerSize;
   support::endianness Endianness;
   GetEdgeKindNameFunction GetEdgeKindName = nullptr;
-  SectionList Sections;
+  DenseMap<StringRef, std::unique_ptr<Section>> Sections;
   ExternalSymbolSet ExternalSymbols;
   ExternalSymbolSet AbsoluteSymbols;
   orc::shared::AllocActions AAs;
@@ -1627,7 +1643,7 @@ class SymbolAddressMap {
 };
 
 /// A function for mutating LinkGraphs.
-using LinkGraphPassFunction = std::function<Error(LinkGraph &)>;
+using LinkGraphPassFunction = unique_function<Error(LinkGraph &)>;
 
 /// A list of LinkGraph passes.
 using LinkGraphPassList = std::vector<LinkGraphPassFunction>;
diff --git a/llvm/include/llvm/ExecutionEngine/JITLink/JITLinkMemoryManager.h b/llvm/include/llvm/ExecutionEngine/JITLink/JITLinkMemoryManager.h
index 6ef4a0bd0c982..09e0d71cf0bd2 100644
--- a/llvm/include/llvm/ExecutionEngine/JITLink/JITLinkMemoryManager.h
+++ b/llvm/include/llvm/ExecutionEngine/JITLink/JITLinkMemoryManager.h
@@ -291,6 +291,9 @@ class BasicLayout {
 /// Segment. Clients can obtain a pointer to the working memory and executor
 /// address of that block using the Segment's AllocGroup. Once memory has been
 /// populated, clients can call finalize to finalize the memory.
+///
+/// Note: Segments with MemLifetimePolicy::NoAlloc are not permitted, since
+/// they would not be useful, and their presence is likely to indicate a bug.
 class SimpleSegmentAlloc {
 public:
   /// Describes a segment to be allocated.
diff --git a/llvm/include/llvm/ExecutionEngine/Orc/COFFPlatform.h b/llvm/include/llvm/ExecutionEngine/Orc/COFFPlatform.h
index 767d336753581..969c6f0b15afc 100644
--- a/llvm/include/llvm/ExecutionEngine/Orc/COFFPlatform.h
+++ b/llvm/include/llvm/ExecutionEngine/Orc/COFFPlatform.h
@@ -138,7 +138,7 @@ class COFFPlatform : public Platform {
 
   COFFPlatform(ExecutionSession &ES, ObjectLinkingLayer &ObjLinkingLayer,
                JITDylib &PlatformJD, const char *OrcRuntimePath,
-               LoadDynamicLibrary LoadDynLibrary, bool StaticVCRuntime,
+               LoadDynamicLibrary LoadDynamicLibrary, bool StaticVCRuntime,
                const char *VCRuntimePath, Error &Err);
 
   // Associate COFFPlatform JIT-side runtime support functions with handlers.
diff --git a/llvm/include/llvm/ExecutionEngine/Orc/Core.h b/llvm/include/llvm/ExecutionEngine/Orc/Core.h
index adab75d3aaf15..33aa2cfe604f0 100644
--- a/llvm/include/llvm/ExecutionEngine/Orc/Core.h
+++ b/llvm/include/llvm/ExecutionEngine/Orc/Core.h
@@ -1211,14 +1211,14 @@ class JITDylib : public ThreadSafeRefCountedBase<JITDylib>,
         : Flags(Flags), State(static_cast<uint8_t>(SymbolState::NeverSearched)),
           MaterializerAttached(false), PendingRemoval(false) {}
 
-    JITTargetAddress getAddress() const { return Addr; }
+    ExecutorAddr getAddress() const { return Addr; }
     JITSymbolFlags getFlags() const { return Flags; }
     SymbolState getState() const { return static_cast<SymbolState>(State); }
 
     bool hasMaterializerAttached() const { return MaterializerAttached; }
     bool isPendingRemoval() const { return PendingRemoval; }
 
-    void setAddress(JITTargetAddress Addr) { this->Addr = Addr; }
+    void setAddress(ExecutorAddr Addr) { this->Addr = Addr; }
     void setFlags(JITSymbolFlags Flags) { this->Flags = Flags; }
     void setState(SymbolState State) {
       assert(static_cast<uint8_t>(State) < (1 << 6) &&
@@ -1235,11 +1235,11 @@ class JITDylib : public ThreadSafeRefCountedBase<JITDylib>,
     }
 
     JITEvaluatedSymbol getSymbol() const {
-      return JITEvaluatedSymbol(Addr, Flags);
+      return JITEvaluatedSymbol(Addr.getValue(), Flags);
     }
 
   private:
-    JITTargetAddress Addr = 0;
+    ExecutorAddr Addr;
     JITSymbolFlags Flags;
     uint8_t State : 6;
     uint8_t MaterializerAttached : 1;
diff --git a/llvm/include/llvm/ExecutionEngine/Orc/EPCDebugObjectRegistrar.h b/llvm/include/llvm/ExecutionEngine/Orc/EPCDebugObjectRegistrar.h
index 8bd762460dd27..7e143811326f4 100644
--- a/llvm/include/llvm/ExecutionEngine/Orc/EPCDebugObjectRegistrar.h
+++ b/llvm/include/llvm/ExecutionEngine/Orc/EPCDebugObjectRegistrar.h
@@ -51,7 +51,7 @@ class EPCDebugObjectRegistrar : public DebugObjectRegistrar {
 
 /// Create a ExecutorProcessControl-based DebugObjectRegistrar that emits debug
 /// objects to the GDB JIT interface. This will use the EPC's lookupSymbols
-/// method to find the registration/deregistration  funciton addresses by name.
+/// method to find the registration/deregistration  function addresses by name.
 ///
 /// If RegistrationFunctionsDylib is non-None then it will be searched to find
 /// the registration functions. If it is None then the process dylib will be
diff --git a/llvm/include/llvm/ExecutionEngine/Orc/EPCEHFrameRegistrar.h b/llvm/include/llvm/ExecutionEngine/Orc/EPCEHFrameRegistrar.h
index 0494705b462de..9772c84b682a5 100644
--- a/llvm/include/llvm/ExecutionEngine/Orc/EPCEHFrameRegistrar.h
+++ b/llvm/include/llvm/ExecutionEngine/Orc/EPCEHFrameRegistrar.h
@@ -27,7 +27,7 @@ class EPCEHFrameRegistrar : public jitlink::EHFrameRegistrar {
 public:
   /// Create from a ExecutorProcessControl instance alone. This will use
   /// the EPC's lookupSymbols method to find the registration/deregistration
-  /// funciton addresses by name.
+  /// function addresses by name.
   ///
   /// If RegistrationFunctionsDylib is non-None then it will be searched to
   /// find the registration functions. If it is None then the process dylib
diff --git a/llvm/include/llvm/ExecutionEngine/Orc/Shared/MemoryFlags.h b/llvm/include/llvm/ExecutionEngine/Orc/Shared/MemoryFlags.h
index 2642e6c241b6e..c20366cfbb388 100644
--- a/llvm/include/llvm/ExecutionEngine/Orc/Shared/MemoryFlags.h
+++ b/llvm/include/llvm/ExecutionEngine/Orc/Shared/MemoryFlags.h
@@ -65,26 +65,43 @@ inline MemProt fromSysMemoryProtectionFlags(sys::Memory::ProtectionFlags PF) {
   return MP;
 }
 
-/// Describes a memory deallocation policy for memory to be allocated by a
+/// Describes a memory lifetime policy for memory to be allocated by a
 /// JITLinkMemoryManager.
 ///
 /// All memory allocated by a call to JITLinkMemoryManager::allocate should be
 /// deallocated if a call is made to
 /// JITLinkMemoryManager::InFlightAllocation::abandon. The policies below apply
 /// to finalized allocations.
-enum class MemDeallocPolicy {
-  /// Standard memory should be deallocated when the deallocate method is called
-  /// for the finalized allocation.
+enum class MemLifetimePolicy {
+  /// Standard memory should be allocated by the allocator and then deallocated
+  /// when the deallocate method is called for the finalized allocation.
   Standard,
 
-  /// Finalize memory should be overwritten and then deallocated after all
-  /// finalization functions have been run.
-  Finalize
+  /// Finalize memory should be allocated by the allocator, and then be
+  /// overwritten and deallocated after all finalization functions have been
+  /// run.
+  Finalize,
+
+  /// NoAlloc memory should not be allocated by the JITLinkMemoryManager at
+  /// all. It is used for sections that don't need to be transferred to the
+  /// executor process, typically metadata sections.
+  NoAlloc
 };
 
 /// Print a MemDeallocPolicy.
-inline raw_ostream &operator<<(raw_ostream &OS, MemDeallocPolicy MDP) {
-  return OS << (MDP == MemDeallocPolicy::Standard ? "standard" : "finalize");
+inline raw_ostream &operator<<(raw_ostream &OS, MemLifetimePolicy MLP) {
+  switch (MLP) {
+  case MemLifetimePolicy::Standard:
+    OS << "standard";
+    break;
+  case MemLifetimePolicy::Finalize:
+    OS << "finalize";
+    break;
+  case MemLifetimePolicy::NoAlloc:
+    OS << "noalloc";
+    break;
+  }
+  return OS;
 }
 
 /// A pair of memory protections and allocation policies.
@@ -95,34 +112,34 @@ class AllocGroup {
 
   using underlying_type = uint8_t;
   static constexpr unsigned BitsForProt = 3;
-  static constexpr unsigned BitsForDeallocPolicy = 1;
+  static constexpr unsigned BitsForLifetimePolicy = 2;
   static constexpr unsigned MaxIdentifiers =
-      1U << (BitsForProt + BitsForDeallocPolicy);
+      1U << (BitsForProt + BitsForLifetimePolicy);
 
 public:
   static constexpr unsigned NumGroups = MaxIdentifiers;
 
   /// Create a default AllocGroup. No memory protections, standard
-  /// deallocation policy.
+  /// lifetime policy.
   AllocGroup() = default;
 
   /// Create an AllocGroup from a MemProt only -- uses
-  /// MemoryDeallocationPolicy::Standard.
+  /// MemLifetimePolicy::Standard.
   AllocGroup(MemProt MP) : Id(static_cast<underlying_type>(MP)) {}
 
-  /// Create an AllocGroup from a MemProt and a MemoryDeallocationPolicy.
-  AllocGroup(MemProt MP, MemDeallocPolicy MDP)
+  /// Create an AllocGroup from a MemProt and a MemLifetimePolicy.
+  AllocGroup(MemProt MP, MemLifetimePolicy MLP)
       : Id(static_cast<underlying_type>(MP) |
-           (static_cast<underlying_type>(MDP) << BitsForProt)) {}
+           (static_cast<underlying_type>(MLP) << BitsForProt)) {}
 
   /// Returns the MemProt for this group.
   MemProt getMemProt() const {
     return static_cast<MemProt>(Id & ((1U << BitsForProt) - 1));
   }
 
-  /// Returns the MemoryDeallocationPolicy for this group.
-  MemDeallocPolicy getMemDeallocPolicy() const {
-    return static_cast<MemDeallocPolicy>(Id >> BitsForProt);
+  /// Returns the MemLifetimePolicy for this group.
+  MemLifetimePolicy getMemLifetimePolicy() const {
+    return static_cast<MemLifetimePolicy>(Id >> BitsForProt);
   }
 
   friend bool operator==(const AllocGroup &LHS, const AllocGroup &RHS) {
@@ -186,7 +203,7 @@ template <typename T> class AllocGroupSmallMap {
 
 /// Print an AllocGroup.
 inline raw_ostream &operator<<(raw_ostream &OS, AllocGroup AG) {
-  return OS << '(' << AG.getMemProt() << ", " << AG.getMemDeallocPolicy()
+  return OS << '(' << AG.getMemProt() << ", " << AG.getMemLifetimePolicy()
             << ')';
 }
 
diff --git a/llvm/include/llvm/ExecutionEngine/Orc/Shared/TargetProcessControlTypes.h b/llvm/include/llvm/ExecutionEngine/Orc/Shared/TargetProcessControlTypes.h
index 565fb5477c4af..09c73db44a947 100644
--- a/llvm/include/llvm/ExecutionEngine/Orc/Shared/TargetProcessControlTypes.h
+++ b/llvm/include/llvm/ExecutionEngine/Orc/Shared/TargetProcessControlTypes.h
@@ -30,8 +30,24 @@ namespace llvm {
 namespace orc {
 namespace tpctypes {
 
+struct RemoteAllocGroup {
+  RemoteAllocGroup() = default;
+  RemoteAllocGroup(MemProt Prot) : Prot(Prot) {}
+  RemoteAllocGroup(MemProt Prot, bool FinalizeLifetime)
+      : Prot(Prot), FinalizeLifetime(FinalizeLifetime) {}
+  RemoteAllocGroup(const AllocGroup &AG) : Prot(AG.getMemProt()) {
+    assert(AG.getMemLifetimePolicy() != orc::MemLifetimePolicy::NoAlloc &&
+           "Cannot use no-alloc memory in a remote alloc request");
+    FinalizeLifetime =
+        AG.getMemLifetimePolicy() == orc::MemLifetimePolicy::Finalize;
+  }
+
+  MemProt Prot;
+  bool FinalizeLifetime = false;
+};
+
 struct SegFinalizeRequest {
-  AllocGroup AG;
+  RemoteAllocGroup RAG;
   ExecutorAddr Addr;
   uint64_t Size;
   ArrayRef<char> Content;
@@ -43,7 +59,7 @@ struct FinalizeRequest {
 };
 
 struct SharedMemorySegFinalizeRequest {
-  AllocGroup AG;
+  RemoteAllocGroup RAG;
   ExecutorAddr Addr;
   uint64_t Size;
 };
@@ -93,16 +109,16 @@ using LookupResult = std::vector<ExecutorAddr>;
 
 namespace shared {
 
-class SPSAllocGroup {};
+class SPSRemoteAllocGroup;
 
 using SPSSegFinalizeRequest =
-    SPSTuple<SPSAllocGroup, SPSExecutorAddr, uint64_t, SPSSequence<char>>;
+    SPSTuple<SPSRemoteAllocGroup, SPSExecutorAddr, uint64_t, SPSSequence<char>>;
 
 using SPSFinalizeRequest = SPSTuple<SPSSequence<SPSSegFinalizeRequest>,
                                     SPSSequence<SPSAllocActionCallPair>>;
 
 using SPSSharedMemorySegFinalizeRequest =
-    SPSTuple<SPSAllocGroup, SPSExecutorAddr, uint64_t>;
+    SPSTuple<SPSRemoteAllocGroup, SPSExecutorAddr, uint64_t>;
 
 using SPSSharedMemoryFinalizeRequest =
     SPSTuple<SPSSequence<SPSSharedMemorySegFinalizeRequest>,
@@ -118,7 +134,8 @@ using SPSMemoryAccessUInt64Write = SPSMemoryAccessUIntWrite<uint64_t>;
 
 using SPSMemoryAccessBufferWrite = SPSTuple<SPSExecutorAddr, SPSSequence<char>>;
 
-template <> class SPSSerializationTraits<SPSAllocGroup, AllocGroup> {
+template <>
+class SPSSerializationTraits<SPSRemoteAllocGroup, tpctypes::RemoteAllocGroup> {
   enum WireBits {
     ReadBit = 1 << 0,
     WriteBit = 1 << 1,
@@ -127,25 +144,26 @@ template <> class SPSSerializationTraits<SPSAllocGroup, AllocGroup> {
   };
 
 public:
-  static size_t size(const AllocGroup &AG) {
+  static size_t size(const tpctypes::RemoteAllocGroup &RAG) {
     // All AllocGroup values encode to the same size.
     return SPSArgList<uint8_t>::size(uint8_t(0));
   }
 
-  static bool serialize(SPSOutputBuffer &OB, const AllocGroup &AG) {
+  static bool serialize(SPSOutputBuffer &OB,
+                        const tpctypes::RemoteAllocGroup &RAG) {
     uint8_t WireValue = 0;
-    if ((AG.getMemProt() & MemProt::Read) != MemProt::None)
+    if ((RAG.Prot & MemProt::Read) != MemProt::None)
       WireValue |= ReadBit;
-    if ((AG.getMemProt() & MemProt::Write) != MemProt::None)
+    if ((RAG.Prot & MemProt::Write) != MemProt::None)
       WireValue |= WriteBit;
-    if ((AG.getMemProt() & MemProt::Exec) != MemProt::None)
+    if ((RAG.Prot & MemProt::Exec) != MemProt::None)
       WireValue |= ExecBit;
-    if (AG.getMemDeallocPolicy() == MemDeallocPolicy::Finalize)
+    if (RAG.FinalizeLifetime)
       WireValue |= FinalizeBit;
     return SPSArgList<uint8_t>::serialize(OB, WireValue);
   }
 
-  static bool deserialize(SPSInputBuffer &IB, AllocGroup &AG) {
+  static bool deserialize(SPSInputBuffer &IB, tpctypes::RemoteAllocGroup &RAG) {
     uint8_t Val;
     if (!SPSArgList<uint8_t>::deserialize(IB, Val))
       return false;
@@ -156,9 +174,8 @@ template <> class SPSSerializationTraits<SPSAllocGroup, AllocGroup> {
       MP |= MemProt::Write;
     if (Val & ExecBit)
       MP |= MemProt::Exec;
-    MemDeallocPolicy MDP = (Val & FinalizeBit) ? MemDeallocPolicy::Finalize
-                                               : MemDeallocPolicy::Standard;
-    AG = AllocGroup(MP, MDP);
+    bool FinalizeLifetime = (Val & FinalizeBit) ? true : false;
+    RAG = {MP, FinalizeLifetime};
     return true;
   }
 };
@@ -170,17 +187,17 @@ class SPSSerializationTraits<SPSSegFinalizeRequest,
 
 public:
   static size_t size(const tpctypes::SegFinalizeRequest &SFR) {
-    return SFRAL::size(SFR.AG, SFR.Addr, SFR.Size, SFR.Content);
+    return SFRAL::size(SFR.RAG, SFR.Addr, SFR.Size, SFR.Content);
   }
 
   static bool serialize(SPSOutputBuffer &OB,
                         const tpctypes::SegFinalizeRequest &SFR) {
-    return SFRAL::serialize(OB, SFR.AG, SFR.Addr, SFR.Size, SFR.Content);
+    return SFRAL::serialize(OB, SFR.RAG, SFR.Addr, SFR.Size, SFR.Content);
   }
 
   static bool deserialize(SPSInputBuffer &IB,
                           tpctypes::SegFinalizeRequest &SFR) {
-    return SFRAL::deserialize(IB, SFR.AG, SFR.Addr, SFR.Size, SFR.Content);
+    return SFRAL::deserialize(IB, SFR.RAG, SFR.Addr, SFR.Size, SFR.Content);
   }
 };
 
@@ -210,17 +227,17 @@ class SPSSerializationTraits<SPSSharedMemorySegFinalizeRequest,
 
 public:
   static size_t size(const tpctypes::SharedMemorySegFinalizeRequest &SFR) {
-    return SFRAL::size(SFR.AG, SFR.Addr, SFR.Size);
+    return SFRAL::size(SFR.RAG, SFR.Addr, SFR.Size);
   }
 
   static bool serialize(SPSOutputBuffer &OB,
                         const tpctypes::SharedMemorySegFinalizeRequest &SFR) {
-    return SFRAL::serialize(OB, SFR.AG, SFR.Addr, SFR.Size);
+    return SFRAL::serialize(OB, SFR.RAG, SFR.Addr, SFR.Size);
   }
 
   static bool deserialize(SPSInputBuffer &IB,
                           tpctypes::SharedMemorySegFinalizeRequest &SFR) {
-    return SFRAL::deserialize(IB, SFR.AG, SFR.Addr, SFR.Size);
+    return SFRAL::deserialize(IB, SFR.RAG, SFR.Addr, SFR.Size);
   }
 };
 
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
index 1dfa858421d93..b3c4f21274cb2 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
@@ -988,6 +988,7 @@ class OpenMPIRBuilder {
   /// Generate a target region entry call.
   ///
   /// \param Loc The location at which the request originated and is fulfilled.
+  /// \param AllocaIP The insertion point to be used for alloca instructions.
   /// \param Return Return value of the created function returned by reference.
   /// \param DeviceID Identifier for the device via the 'device' clause.
   /// \param NumTeams Numer of teams for the region via the 'num_teams' clause
@@ -995,7 +996,8 @@ class OpenMPIRBuilder {
   /// \param NumThreads Number of threads via the 'thread_limit' clause.
   /// \param HostPtr Pointer to the host-side pointer of the target kernel.
   /// \param KernelArgs Array of arguments to the kernel.
-  InsertPointTy emitTargetKernel(const LocationDescription &Loc, Value *&Return,
+  InsertPointTy emitTargetKernel(const LocationDescription &Loc,
+                                 InsertPointTy AllocaIP, Value *&Return,
                                  Value *Ident, Value *DeviceID, Value *NumTeams,
                                  Value *NumThreads, Value *HostPtr,
                                  ArrayRef<Value *> KernelArgs);
diff --git a/llvm/include/llvm/IR/Attributes.h b/llvm/include/llvm/IR/Attributes.h
index a77c4381b0fcf..be536da04c9d7 100644
--- a/llvm/include/llvm/IR/Attributes.h
+++ b/llvm/include/llvm/IR/Attributes.h
@@ -1265,6 +1265,10 @@ enum AttributeSafetyKind : uint8_t {
   ASK_ALL = ASK_SAFE_TO_DROP | ASK_UNSAFE_TO_DROP,
 };
 
+/// Returns true if this is a type legal for the 'nofpclass' attribute. This
+/// follows the same type rules as FPMathOperator.
+bool isNoFPClassCompatibleType(Type *Ty);
+
 /// Which attributes cannot be applied to a type. The argument \p ASK indicates,
 /// if only attributes that are known to be safely droppable are contained in
 /// the mask; only attributes that might be unsafe to drop (e.g., ABI-related
diff --git a/llvm/include/llvm/IR/ConstantFolder.h b/llvm/include/llvm/IR/ConstantFolder.h
index 4473187a1bf93..56da3d205fe4b 100644
--- a/llvm/include/llvm/IR/ConstantFolder.h
+++ b/llvm/include/llvm/IR/ConstantFolder.h
@@ -105,6 +105,9 @@ class ConstantFolder final : public IRBuilderFolder {
 
   Value *FoldGEP(Type *Ty, Value *Ptr, ArrayRef<Value *> IdxList,
                  bool IsInBounds = false) const override {
+    if (!ConstantExpr::isSupportedGetElementPtr(Ty))
+      return nullptr;
+
     if (auto *PC = dyn_cast<Constant>(Ptr)) {
       // Every index must be constant.
       if (any_of(IdxList, [](Value *V) { return !isa<Constant>(V); }))
diff --git a/llvm/include/llvm/IR/Constants.h b/llvm/include/llvm/IR/Constants.h
index 8aaedc3d082b0..9cc56ecf8e970 100644
--- a/llvm/include/llvm/IR/Constants.h
+++ b/llvm/include/llvm/IR/Constants.h
@@ -1027,16 +1027,6 @@ class ConstantExpr : public Constant {
   ///
   static Constant *getSizeOf(Type *Ty);
 
-  /// getOffsetOf constant expr - computes the offset of a struct field in a
-  /// target independent way (Note: the return type is an i64).
-  ///
-  static Constant *getOffsetOf(StructType *STy, unsigned FieldNo);
-
-  /// getOffsetOf constant expr - This is a generalized form of getOffsetOf,
-  /// which supports any aggregate type, and any Constant index.
-  ///
-  static Constant *getOffsetOf(Type *Ty, Constant *FieldNo);
-
   static Constant *getNeg(Constant *C, bool HasNUW = false,
                           bool HasNSW = false);
   static Constant *getNot(Constant *C);
@@ -1049,7 +1039,6 @@ class ConstantExpr : public Constant {
   static Constant *getAnd(Constant *C1, Constant *C2);
   static Constant *getOr(Constant *C1, Constant *C2);
   static Constant *getXor(Constant *C1, Constant *C2);
-  static Constant *getUMin(Constant *C1, Constant *C2);
   static Constant *getShl(Constant *C1, Constant *C2, bool HasNUW = false,
                           bool HasNSW = false);
   static Constant *getLShr(Constant *C1, Constant *C2, bool isExact = false);
@@ -1209,12 +1198,6 @@ class ConstantExpr : public Constant {
   /// Return true if this is a compare constant expression
   bool isCompare() const;
 
-  /// Select constant expr
-  ///
-  /// \param OnlyIfReducedTy see \a getWithOperands() docs.
-  static Constant *getSelect(Constant *C, Constant *V1, Constant *V2,
-                             Type *OnlyIfReducedTy = nullptr);
-
   /// get - Return a binary or shift operator constant expression,
   /// folding if possible.
   ///
@@ -1350,6 +1333,12 @@ class ConstantExpr : public Constant {
   /// supported.
   static bool isSupportedBinOp(unsigned Opcode);
 
+  /// Whether creating a constant expression for this getelementptr type is
+  /// supported.
+  static bool isSupportedGetElementPtr(const Type *SrcElemTy) {
+    return !SrcElemTy->isScalableTy();
+  }
+
   /// Methods for support type inquiry through isa, cast, and dyn_cast:
   static bool classof(const Value *V) {
     return V->getValueID() == ConstantExprVal;
diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index dd4de53bc0918..fb0a194e0c344 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -1599,254 +1599,6 @@ class DILocalScope : public DIScope {
   }
 };
 
-/// Debug location.
-///
-/// A debug location in source code, used for debug info and otherwise.
-class DILocation : public MDNode {
-  friend class LLVMContextImpl;
-  friend class MDNode;
-
-  DILocation(LLVMContext &C, StorageType Storage, unsigned Line,
-             unsigned Column, ArrayRef<Metadata *> MDs, bool ImplicitCode);
-  ~DILocation() { dropAllReferences(); }
-
-  static DILocation *getImpl(LLVMContext &Context, unsigned Line,
-                             unsigned Column, Metadata *Scope,
-                             Metadata *InlinedAt, bool ImplicitCode,
-                             StorageType Storage, bool ShouldCreate = true);
-  static DILocation *getImpl(LLVMContext &Context, unsigned Line,
-                             unsigned Column, DILocalScope *Scope,
-                             DILocation *InlinedAt, bool ImplicitCode,
-                             StorageType Storage, bool ShouldCreate = true) {
-    return getImpl(Context, Line, Column, static_cast<Metadata *>(Scope),
-                   static_cast<Metadata *>(InlinedAt), ImplicitCode, Storage,
-                   ShouldCreate);
-  }
-
-  TempDILocation cloneImpl() const {
-    // Get the raw scope/inlinedAt since it is possible to invoke this on
-    // a DILocation containing temporary metadata.
-    return getTemporary(getContext(), getLine(), getColumn(), getRawScope(),
-                        getRawInlinedAt(), isImplicitCode());
-  }
-
-public:
-  // Disallow replacing operands.
-  void replaceOperandWith(unsigned I, Metadata *New) = delete;
-
-  DEFINE_MDNODE_GET(DILocation,
-                    (unsigned Line, unsigned Column, Metadata *Scope,
-                     Metadata *InlinedAt = nullptr, bool ImplicitCode = false),
-                    (Line, Column, Scope, InlinedAt, ImplicitCode))
-  DEFINE_MDNODE_GET(DILocation,
-                    (unsigned Line, unsigned Column, DILocalScope *Scope,
-                     DILocation *InlinedAt = nullptr,
-                     bool ImplicitCode = false),
-                    (Line, Column, Scope, InlinedAt, ImplicitCode))
-
-  /// Return a (temporary) clone of this.
-  TempDILocation clone() const { return cloneImpl(); }
-
-  unsigned getLine() const { return SubclassData32; }
-  unsigned getColumn() const { return SubclassData16; }
-  DILocalScope *getScope() const { return cast<DILocalScope>(getRawScope()); }
-
-  DILocation *getInlinedAt() const {
-    return cast_or_null<DILocation>(getRawInlinedAt());
-  }
-
-  /// Check if the location corresponds to an implicit code.
-  /// When the ImplicitCode flag is true, it means that the Instruction
-  /// with this DILocation has been added by the front-end but it hasn't been
-  /// written explicitly by the user (e.g. cleanup stuff in C++ put on a closing
-  /// bracket). It's useful for code coverage to not show a counter on "empty"
-  /// lines.
-  bool isImplicitCode() const { return SubclassData1; }
-  void setImplicitCode(bool ImplicitCode) { SubclassData1 = ImplicitCode; }
-
-  DIFile *getFile() const { return getScope()->getFile(); }
-  StringRef getFilename() const { return getScope()->getFilename(); }
-  StringRef getDirectory() const { return getScope()->getDirectory(); }
-  std::optional<StringRef> getSource() const { return getScope()->getSource(); }
-
-  /// Get the scope where this is inlined.
-  ///
-  /// Walk through \a getInlinedAt() and return \a getScope() from the deepest
-  /// location.
-  DILocalScope *getInlinedAtScope() const {
-    if (auto *IA = getInlinedAt())
-      return IA->getInlinedAtScope();
-    return getScope();
-  }
-
-  /// Get the DWARF discriminator.
-  ///
-  /// DWARF discriminators distinguish identical file locations between
-  /// instructions that are on different basic blocks.
-  ///
-  /// There are 3 components stored in discriminator, from lower bits:
-  ///
-  /// Base discriminator: assigned by AddDiscriminators pass to identify IRs
-  ///                     that are defined by the same source line, but
-  ///                     different basic blocks.
-  /// Duplication factor: assigned by optimizations that will scale down
-  ///                     the execution frequency of the original IR.
-  /// Copy Identifier: assigned by optimizations that clones the IR.
-  ///                  Each copy of the IR will be assigned an identifier.
-  ///
-  /// Encoding:
-  ///
-  /// The above 3 components are encoded into a 32bit unsigned integer in
-  /// order. If the lowest bit is 1, the current component is empty, and the
-  /// next component will start in the next bit. Otherwise, the current
-  /// component is non-empty, and its content starts in the next bit. The
-  /// value of each components is either 5 bit or 12 bit: if the 7th bit
-  /// is 0, the bit 2~6 (5 bits) are used to represent the component; if the
-  /// 7th bit is 1, the bit 2~6 (5 bits) and 8~14 (7 bits) are combined to
-  /// represent the component. Thus, the number of bits used for a component
-  /// is either 0 (if it and all the next components are empty); 1 - if it is
-  /// empty; 7 - if its value is up to and including 0x1f (lsb and msb are both
-  /// 0); or 14, if its value is up to and including 0x1ff. Note that the last
-  /// component is also capped at 0x1ff, even in the case when both first
-  /// components are 0, and we'd technically have 29 bits available.
-  ///
-  /// For precise control over the data being encoded in the discriminator,
-  /// use encodeDiscriminator/decodeDiscriminator.
-
-  inline unsigned getDiscriminator() const;
-
-  // For the regular discriminator, it stands for all empty components if all
-  // the lowest 3 bits are non-zero and all higher 29 bits are unused(zero by
-  // default). Here we fully leverage the higher 29 bits for pseudo probe use.
-  // This is the format:
-  // [2:0] - 0x7
-  // [31:3] - pseudo probe fields guaranteed to be non-zero as a whole
-  // So if the lower 3 bits is non-zero and the others has at least one
-  // non-zero bit, it guarantees to be a pseudo probe discriminator
-  inline static bool isPseudoProbeDiscriminator(unsigned Discriminator) {
-    return ((Discriminator & 0x7) == 0x7) && (Discriminator & 0xFFFFFFF8);
-  }
-
-  /// Returns a new DILocation with updated \p Discriminator.
-  inline const DILocation *cloneWithDiscriminator(unsigned Discriminator) const;
-
-  /// Returns a new DILocation with updated base discriminator \p BD. Only the
-  /// base discriminator is set in the new DILocation, the other encoded values
-  /// are elided.
-  /// If the discriminator cannot be encoded, the function returns std::nullopt.
-  inline std::optional<const DILocation *>
-  cloneWithBaseDiscriminator(unsigned BD) const;
-
-  /// Returns the duplication factor stored in the discriminator, or 1 if no
-  /// duplication factor (or 0) is encoded.
-  inline unsigned getDuplicationFactor() const;
-
-  /// Returns the copy identifier stored in the discriminator.
-  inline unsigned getCopyIdentifier() const;
-
-  /// Returns the base discriminator stored in the discriminator.
-  inline unsigned getBaseDiscriminator() const;
-
-  /// Returns a new DILocation with duplication factor \p DF * current
-  /// duplication factor encoded in the discriminator. The current duplication
-  /// factor is as defined by getDuplicationFactor().
-  /// Returns std::nullopt if encoding failed.
-  inline std::optional<const DILocation *>
-  cloneByMultiplyingDuplicationFactor(unsigned DF) const;
-
-  /// When two instructions are combined into a single instruction we also
-  /// need to combine the original locations into a single location.
-  /// When the locations are the same we can use either location.
-  /// When they differ, we need a third location which is distinct from either.
-  /// If they share a common scope, use this scope and compare the line/column
-  /// pair of the locations with the common scope:
-  /// * if both match, keep the line and column;
-  /// * if only the line number matches, keep the line and set the column as 0;
-  /// * otherwise set line and column as 0.
-  /// If they do not share a common scope the location is ambiguous and can't be
-  /// represented in a line entry. In this case, set line and column as 0 and
-  /// use the scope of any location.
-  ///
-  /// \p LocA \p LocB: The locations to be merged.
-  static const DILocation *getMergedLocation(const DILocation *LocA,
-                                             const DILocation *LocB);
-
-  /// Try to combine the vector of locations passed as input in a single one.
-  /// This function applies getMergedLocation() repeatedly left-to-right.
-  ///
-  /// \p Locs: The locations to be merged.
-  static const DILocation *
-  getMergedLocations(ArrayRef<const DILocation *> Locs);
-
-  /// Return the masked discriminator value for an input discrimnator value D
-  /// (i.e. zero out the (B+1)-th and above bits for D (B is 0-base).
-  // Example: an input of (0x1FF, 7) returns 0xFF.
-  static unsigned getMaskedDiscriminator(unsigned D, unsigned B) {
-    return (D & getN1Bits(B));
-  }
-
-  /// Return the bits used for base discriminators.
-  static unsigned getBaseDiscriminatorBits() { return getBaseFSBitEnd(); }
-
-  /// Returns the base discriminator for a given encoded discriminator \p D.
-  static unsigned
-  getBaseDiscriminatorFromDiscriminator(unsigned D,
-                                        bool IsFSDiscriminator = false) {
-    if (IsFSDiscriminator)
-      return getMaskedDiscriminator(D, getBaseDiscriminatorBits());
-    return getUnsignedFromPrefixEncoding(D);
-  }
-
-  /// Raw encoding of the discriminator. APIs such as cloneWithDuplicationFactor
-  /// have certain special case behavior (e.g. treating empty duplication factor
-  /// as the value '1').
-  /// This API, in conjunction with cloneWithDiscriminator, may be used to
-  /// encode the raw values provided.
-  ///
-  /// \p BD: base discriminator
-  /// \p DF: duplication factor
-  /// \p CI: copy index
-  ///
-  /// The return is std::nullopt if the values cannot be encoded in 32 bits -
-  /// for example, values for BD or DF larger than 12 bits. Otherwise, the
-  /// return is the encoded value.
-  static std::optional<unsigned> encodeDiscriminator(unsigned BD, unsigned DF,
-                                                     unsigned CI);
-
-  /// Raw decoder for values in an encoded discriminator D.
-  static void decodeDiscriminator(unsigned D, unsigned &BD, unsigned &DF,
-                                  unsigned &CI);
-
-  /// Returns the duplication factor for a given encoded discriminator \p D, or
-  /// 1 if no value or 0 is encoded.
-  static unsigned getDuplicationFactorFromDiscriminator(unsigned D) {
-    if (EnableFSDiscriminator)
-      return 1;
-    D = getNextComponentInDiscriminator(D);
-    unsigned Ret = getUnsignedFromPrefixEncoding(D);
-    if (Ret == 0)
-      return 1;
-    return Ret;
-  }
-
-  /// Returns the copy identifier for a given encoded discriminator \p D.
-  static unsigned getCopyIdentifierFromDiscriminator(unsigned D) {
-    return getUnsignedFromPrefixEncoding(
-        getNextComponentInDiscriminator(getNextComponentInDiscriminator(D)));
-  }
-
-  Metadata *getRawScope() const { return getOperand(0); }
-  Metadata *getRawInlinedAt() const {
-    if (getNumOperands() == 2)
-      return getOperand(1);
-    return nullptr;
-  }
-
-  static bool classof(const Metadata *MD) {
-    return MD->getMetadataID() == DILocationKind;
-  }
-};
-
 /// Subprogram description.
 class DISubprogram : public DILocalScope {
   friend class LLVMContextImpl;
@@ -2115,6 +1867,266 @@ class DISubprogram : public DILocalScope {
   }
 };
 
+/// Debug location.
+///
+/// A debug location in source code, used for debug info and otherwise.
+class DILocation : public MDNode {
+  friend class LLVMContextImpl;
+  friend class MDNode;
+
+  DILocation(LLVMContext &C, StorageType Storage, unsigned Line,
+             unsigned Column, ArrayRef<Metadata *> MDs, bool ImplicitCode);
+  ~DILocation() { dropAllReferences(); }
+
+  static DILocation *getImpl(LLVMContext &Context, unsigned Line,
+                             unsigned Column, Metadata *Scope,
+                             Metadata *InlinedAt, bool ImplicitCode,
+                             StorageType Storage, bool ShouldCreate = true);
+  static DILocation *getImpl(LLVMContext &Context, unsigned Line,
+                             unsigned Column, DILocalScope *Scope,
+                             DILocation *InlinedAt, bool ImplicitCode,
+                             StorageType Storage, bool ShouldCreate = true) {
+    return getImpl(Context, Line, Column, static_cast<Metadata *>(Scope),
+                   static_cast<Metadata *>(InlinedAt), ImplicitCode, Storage,
+                   ShouldCreate);
+  }
+
+  TempDILocation cloneImpl() const {
+    // Get the raw scope/inlinedAt since it is possible to invoke this on
+    // a DILocation containing temporary metadata.
+    return getTemporary(getContext(), getLine(), getColumn(), getRawScope(),
+                        getRawInlinedAt(), isImplicitCode());
+  }
+
+public:
+  // Disallow replacing operands.
+  void replaceOperandWith(unsigned I, Metadata *New) = delete;
+
+  DEFINE_MDNODE_GET(DILocation,
+                    (unsigned Line, unsigned Column, Metadata *Scope,
+                     Metadata *InlinedAt = nullptr, bool ImplicitCode = false),
+                    (Line, Column, Scope, InlinedAt, ImplicitCode))
+  DEFINE_MDNODE_GET(DILocation,
+                    (unsigned Line, unsigned Column, DILocalScope *Scope,
+                     DILocation *InlinedAt = nullptr,
+                     bool ImplicitCode = false),
+                    (Line, Column, Scope, InlinedAt, ImplicitCode))
+
+  /// Return a (temporary) clone of this.
+  TempDILocation clone() const { return cloneImpl(); }
+
+  unsigned getLine() const { return SubclassData32; }
+  unsigned getColumn() const { return SubclassData16; }
+  DILocalScope *getScope() const { return cast<DILocalScope>(getRawScope()); }
+
+  /// Return the linkage name of Subprogram. If the linkage name is empty,
+  /// return scope name (the demangled name).
+  const StringRef getSubprogramLinkageName() const {
+    DISubprogram *SP = getScope()->getSubprogram();
+    if (!SP)
+      return "";
+    auto Name = SP->getLinkageName();
+    if (!Name.empty())
+      return Name;
+    return SP->getName();
+  }
+
+  DILocation *getInlinedAt() const {
+    return cast_or_null<DILocation>(getRawInlinedAt());
+  }
+
+  /// Check if the location corresponds to an implicit code.
+  /// When the ImplicitCode flag is true, it means that the Instruction
+  /// with this DILocation has been added by the front-end but it hasn't been
+  /// written explicitly by the user (e.g. cleanup stuff in C++ put on a closing
+  /// bracket). It's useful for code coverage to not show a counter on "empty"
+  /// lines.
+  bool isImplicitCode() const { return SubclassData1; }
+  void setImplicitCode(bool ImplicitCode) { SubclassData1 = ImplicitCode; }
+
+  DIFile *getFile() const { return getScope()->getFile(); }
+  StringRef getFilename() const { return getScope()->getFilename(); }
+  StringRef getDirectory() const { return getScope()->getDirectory(); }
+  std::optional<StringRef> getSource() const { return getScope()->getSource(); }
+
+  /// Get the scope where this is inlined.
+  ///
+  /// Walk through \a getInlinedAt() and return \a getScope() from the deepest
+  /// location.
+  DILocalScope *getInlinedAtScope() const {
+    if (auto *IA = getInlinedAt())
+      return IA->getInlinedAtScope();
+    return getScope();
+  }
+
+  /// Get the DWARF discriminator.
+  ///
+  /// DWARF discriminators distinguish identical file locations between
+  /// instructions that are on different basic blocks.
+  ///
+  /// There are 3 components stored in discriminator, from lower bits:
+  ///
+  /// Base discriminator: assigned by AddDiscriminators pass to identify IRs
+  ///                     that are defined by the same source line, but
+  ///                     different basic blocks.
+  /// Duplication factor: assigned by optimizations that will scale down
+  ///                     the execution frequency of the original IR.
+  /// Copy Identifier: assigned by optimizations that clones the IR.
+  ///                  Each copy of the IR will be assigned an identifier.
+  ///
+  /// Encoding:
+  ///
+  /// The above 3 components are encoded into a 32bit unsigned integer in
+  /// order. If the lowest bit is 1, the current component is empty, and the
+  /// next component will start in the next bit. Otherwise, the current
+  /// component is non-empty, and its content starts in the next bit. The
+  /// value of each components is either 5 bit or 12 bit: if the 7th bit
+  /// is 0, the bit 2~6 (5 bits) are used to represent the component; if the
+  /// 7th bit is 1, the bit 2~6 (5 bits) and 8~14 (7 bits) are combined to
+  /// represent the component. Thus, the number of bits used for a component
+  /// is either 0 (if it and all the next components are empty); 1 - if it is
+  /// empty; 7 - if its value is up to and including 0x1f (lsb and msb are both
+  /// 0); or 14, if its value is up to and including 0x1ff. Note that the last
+  /// component is also capped at 0x1ff, even in the case when both first
+  /// components are 0, and we'd technically have 29 bits available.
+  ///
+  /// For precise control over the data being encoded in the discriminator,
+  /// use encodeDiscriminator/decodeDiscriminator.
+
+  inline unsigned getDiscriminator() const;
+
+  // For the regular discriminator, it stands for all empty components if all
+  // the lowest 3 bits are non-zero and all higher 29 bits are unused(zero by
+  // default). Here we fully leverage the higher 29 bits for pseudo probe use.
+  // This is the format:
+  // [2:0] - 0x7
+  // [31:3] - pseudo probe fields guaranteed to be non-zero as a whole
+  // So if the lower 3 bits is non-zero and the others has at least one
+  // non-zero bit, it guarantees to be a pseudo probe discriminator
+  inline static bool isPseudoProbeDiscriminator(unsigned Discriminator) {
+    return ((Discriminator & 0x7) == 0x7) && (Discriminator & 0xFFFFFFF8);
+  }
+
+  /// Returns a new DILocation with updated \p Discriminator.
+  inline const DILocation *cloneWithDiscriminator(unsigned Discriminator) const;
+
+  /// Returns a new DILocation with updated base discriminator \p BD. Only the
+  /// base discriminator is set in the new DILocation, the other encoded values
+  /// are elided.
+  /// If the discriminator cannot be encoded, the function returns std::nullopt.
+  inline std::optional<const DILocation *>
+  cloneWithBaseDiscriminator(unsigned BD) const;
+
+  /// Returns the duplication factor stored in the discriminator, or 1 if no
+  /// duplication factor (or 0) is encoded.
+  inline unsigned getDuplicationFactor() const;
+
+  /// Returns the copy identifier stored in the discriminator.
+  inline unsigned getCopyIdentifier() const;
+
+  /// Returns the base discriminator stored in the discriminator.
+  inline unsigned getBaseDiscriminator() const;
+
+  /// Returns a new DILocation with duplication factor \p DF * current
+  /// duplication factor encoded in the discriminator. The current duplication
+  /// factor is as defined by getDuplicationFactor().
+  /// Returns std::nullopt if encoding failed.
+  inline std::optional<const DILocation *>
+  cloneByMultiplyingDuplicationFactor(unsigned DF) const;
+
+  /// When two instructions are combined into a single instruction we also
+  /// need to combine the original locations into a single location.
+  /// When the locations are the same we can use either location.
+  /// When they differ, we need a third location which is distinct from either.
+  /// If they share a common scope, use this scope and compare the line/column
+  /// pair of the locations with the common scope:
+  /// * if both match, keep the line and column;
+  /// * if only the line number matches, keep the line and set the column as 0;
+  /// * otherwise set line and column as 0.
+  /// If they do not share a common scope the location is ambiguous and can't be
+  /// represented in a line entry. In this case, set line and column as 0 and
+  /// use the scope of any location.
+  ///
+  /// \p LocA \p LocB: The locations to be merged.
+  static const DILocation *getMergedLocation(const DILocation *LocA,
+                                             const DILocation *LocB);
+
+  /// Try to combine the vector of locations passed as input in a single one.
+  /// This function applies getMergedLocation() repeatedly left-to-right.
+  ///
+  /// \p Locs: The locations to be merged.
+  static const DILocation *
+  getMergedLocations(ArrayRef<const DILocation *> Locs);
+
+  /// Return the masked discriminator value for an input discrimnator value D
+  /// (i.e. zero out the (B+1)-th and above bits for D (B is 0-base).
+  // Example: an input of (0x1FF, 7) returns 0xFF.
+  static unsigned getMaskedDiscriminator(unsigned D, unsigned B) {
+    return (D & getN1Bits(B));
+  }
+
+  /// Return the bits used for base discriminators.
+  static unsigned getBaseDiscriminatorBits() { return getBaseFSBitEnd(); }
+
+  /// Returns the base discriminator for a given encoded discriminator \p D.
+  static unsigned
+  getBaseDiscriminatorFromDiscriminator(unsigned D,
+                                        bool IsFSDiscriminator = false) {
+    if (IsFSDiscriminator)
+      return getMaskedDiscriminator(D, getBaseDiscriminatorBits());
+    return getUnsignedFromPrefixEncoding(D);
+  }
+
+  /// Raw encoding of the discriminator. APIs such as cloneWithDuplicationFactor
+  /// have certain special case behavior (e.g. treating empty duplication factor
+  /// as the value '1').
+  /// This API, in conjunction with cloneWithDiscriminator, may be used to
+  /// encode the raw values provided.
+  ///
+  /// \p BD: base discriminator
+  /// \p DF: duplication factor
+  /// \p CI: copy index
+  ///
+  /// The return is std::nullopt if the values cannot be encoded in 32 bits -
+  /// for example, values for BD or DF larger than 12 bits. Otherwise, the
+  /// return is the encoded value.
+  static std::optional<unsigned> encodeDiscriminator(unsigned BD, unsigned DF,
+                                                     unsigned CI);
+
+  /// Raw decoder for values in an encoded discriminator D.
+  static void decodeDiscriminator(unsigned D, unsigned &BD, unsigned &DF,
+                                  unsigned &CI);
+
+  /// Returns the duplication factor for a given encoded discriminator \p D, or
+  /// 1 if no value or 0 is encoded.
+  static unsigned getDuplicationFactorFromDiscriminator(unsigned D) {
+    if (EnableFSDiscriminator)
+      return 1;
+    D = getNextComponentInDiscriminator(D);
+    unsigned Ret = getUnsignedFromPrefixEncoding(D);
+    if (Ret == 0)
+      return 1;
+    return Ret;
+  }
+
+  /// Returns the copy identifier for a given encoded discriminator \p D.
+  static unsigned getCopyIdentifierFromDiscriminator(unsigned D) {
+    return getUnsignedFromPrefixEncoding(
+        getNextComponentInDiscriminator(getNextComponentInDiscriminator(D)));
+  }
+
+  Metadata *getRawScope() const { return getOperand(0); }
+  Metadata *getRawInlinedAt() const {
+    if (getNumOperands() == 2)
+      return getOperand(1);
+    return nullptr;
+  }
+
+  static bool classof(const Metadata *MD) {
+    return MD->getMetadataID() == DILocationKind;
+  }
+};
+
 class DILexicalBlockBase : public DILocalScope {
 protected:
   DILexicalBlockBase(LLVMContext &C, unsigned ID, StorageType Storage,
diff --git a/llvm/include/llvm/IR/IRBuilder.h b/llvm/include/llvm/IR/IRBuilder.h
index 3c78faa3f94f7..0546ddaffbfc6 100644
--- a/llvm/include/llvm/IR/IRBuilder.h
+++ b/llvm/include/llvm/IR/IRBuilder.h
@@ -893,6 +893,14 @@ class IRBuilderBase {
   /// will be the same type as that of \p Scaling.
   Value *CreateVScale(Constant *Scaling, const Twine &Name = "");
 
+  /// Create an expression which evaluates to the number of elements in \p EC
+  /// at runtime.
+  Value *CreateElementCount(Type *DstType, ElementCount EC);
+
+  /// Create an expression which evaluates to the number of units in \p Size
+  /// at runtime.  This works for both units of bits and bytes.
+  Value *CreateTypeSize(Type *DstType, TypeSize Size);
+
   /// Creates a vector of type \p DstType with the linear sequence <0, 1, ...>
   Value *CreateStepVector(Type *DstType, const Twine &Name = "");
 
diff --git a/llvm/include/llvm/IR/InstrTypes.h b/llvm/include/llvm/IR/InstrTypes.h
index 6c74842bf4886..f8c1df90d1139 100644
--- a/llvm/include/llvm/IR/InstrTypes.h
+++ b/llvm/include/llvm/IR/InstrTypes.h
@@ -1074,6 +1074,8 @@ struct OperandTraits<CmpInst> : public FixedNumOperandTraits<CmpInst, 2> {
 
 DEFINE_TRANSPARENT_OPERAND_ACCESSORS(CmpInst, Value)
 
+raw_ostream &operator<<(raw_ostream &OS, CmpInst::Predicate Pred);
+
 /// A lightweight accessor for an operand bundle meant to be passed
 /// around by value.
 struct OperandBundleUse {
diff --git a/llvm/include/llvm/IR/Instructions.h b/llvm/include/llvm/IR/Instructions.h
index fb6faec3ad848..46202e4570ff4 100644
--- a/llvm/include/llvm/IR/Instructions.h
+++ b/llvm/include/llvm/IR/Instructions.h
@@ -2430,6 +2430,37 @@ class ShuffleVectorInst : public Instruction {
     }
   }
 
+  /// Return if this shuffle interleaves its two input vectors together.
+  bool isInterleave(unsigned Factor);
+
+  /// Return true if the mask interleaves one or more input vectors together.
+  ///
+  /// I.e. <0, LaneLen, ... , LaneLen*(Factor - 1), 1, LaneLen + 1, ...>
+  /// E.g. For a Factor of 2 (LaneLen=4):
+  ///   <0, 4, 1, 5, 2, 6, 3, 7>
+  /// E.g. For a Factor of 3 (LaneLen=4):
+  ///   <4, 0, 9, 5, 1, 10, 6, 2, 11, 7, 3, 12>
+  /// E.g. For a Factor of 4 (LaneLen=2):
+  ///   <0, 2, 6, 4, 1, 3, 7, 5>
+  ///
+  /// NumInputElts is the total number of elements in the input vectors.
+  ///
+  /// StartIndexes are the first indexes of each vector being interleaved,
+  /// substituting any indexes that were undef
+  /// E.g. <4, -1, 2, 5, 1, 3> (Factor=3): StartIndexes=<4, 0, 2>
+  ///
+  /// Note that this does not check if the input vectors are consecutive:
+  /// It will return true for masks such as
+  /// <0, 4, 6, 1, 5, 7> (Factor=3, LaneLen=2)
+  static bool isInterleaveMask(ArrayRef<int> Mask, unsigned Factor,
+                               unsigned NumInputElts,
+                               SmallVectorImpl<unsigned> &StartIndexes);
+  static bool isInterleaveMask(ArrayRef<int> Mask, unsigned Factor,
+                               unsigned NumInputElts) {
+    SmallVector<unsigned, 8> StartIndexes;
+    return isInterleaveMask(Mask, Factor, NumInputElts, StartIndexes);
+  }
+
   // Methods for support type inquiry through isa, cast, and dyn_cast:
   static bool classof(const Instruction *I) {
     return I->getOpcode() == Instruction::ShuffleVector;
diff --git a/llvm/include/llvm/IR/IntrinsicInst.h b/llvm/include/llvm/IR/IntrinsicInst.h
index 2ce2d8c684f0a..1ddde547565a7 100644
--- a/llvm/include/llvm/IR/IntrinsicInst.h
+++ b/llvm/include/llvm/IR/IntrinsicInst.h
@@ -176,57 +176,114 @@ class DbgInfoIntrinsic : public IntrinsicInst {
   /// @}
 };
 
-/// This is the common base class for debug info intrinsics for variables.
-class DbgVariableIntrinsic : public DbgInfoIntrinsic {
+// Iterator for ValueAsMetadata that internally uses direct pointer iteration
+// over either a ValueAsMetadata* or a ValueAsMetadata**, dereferencing to the
+// ValueAsMetadata .
+class location_op_iterator
+    : public iterator_facade_base<location_op_iterator,
+                                  std::bidirectional_iterator_tag, Value *> {
+  PointerUnion<ValueAsMetadata *, ValueAsMetadata **> I;
+
 public:
-  // Iterator for ValueAsMetadata that internally uses direct pointer iteration
-  // over either a ValueAsMetadata* or a ValueAsMetadata**, dereferencing to the
-  // ValueAsMetadata .
-  class location_op_iterator
-      : public iterator_facade_base<location_op_iterator,
-                                    std::bidirectional_iterator_tag, Value *> {
-    PointerUnion<ValueAsMetadata *, ValueAsMetadata **> I;
-
-  public:
-    location_op_iterator(ValueAsMetadata *SingleIter) : I(SingleIter) {}
-    location_op_iterator(ValueAsMetadata **MultiIter) : I(MultiIter) {}
-
-    location_op_iterator(const location_op_iterator &R) : I(R.I) {}
-    location_op_iterator &operator=(const location_op_iterator &R) {
-      I = R.I;
-      return *this;
-    }
-    bool operator==(const location_op_iterator &RHS) const {
-      return I == RHS.I;
-    }
-    const Value *operator*() const {
-      ValueAsMetadata *VAM = I.is<ValueAsMetadata *>()
-                                 ? I.get<ValueAsMetadata *>()
-                                 : *I.get<ValueAsMetadata **>();
-      return VAM->getValue();
-    };
-    Value *operator*() {
-      ValueAsMetadata *VAM = I.is<ValueAsMetadata *>()
-                                 ? I.get<ValueAsMetadata *>()
-                                 : *I.get<ValueAsMetadata **>();
-      return VAM->getValue();
-    }
-    location_op_iterator &operator++() {
-      if (I.is<ValueAsMetadata *>())
-        I = I.get<ValueAsMetadata *>() + 1;
-      else
-        I = I.get<ValueAsMetadata **>() + 1;
-      return *this;
-    }
-    location_op_iterator &operator--() {
-      if (I.is<ValueAsMetadata *>())
-        I = I.get<ValueAsMetadata *>() - 1;
-      else
-        I = I.get<ValueAsMetadata **>() - 1;
-      return *this;
-    }
+  location_op_iterator(ValueAsMetadata *SingleIter) : I(SingleIter) {}
+  location_op_iterator(ValueAsMetadata **MultiIter) : I(MultiIter) {}
+
+  location_op_iterator(const location_op_iterator &R) : I(R.I) {}
+  location_op_iterator &operator=(const location_op_iterator &R) {
+    I = R.I;
+    return *this;
+  }
+  bool operator==(const location_op_iterator &RHS) const { return I == RHS.I; }
+  const Value *operator*() const {
+    ValueAsMetadata *VAM = I.is<ValueAsMetadata *>()
+                               ? I.get<ValueAsMetadata *>()
+                               : *I.get<ValueAsMetadata **>();
+    return VAM->getValue();
   };
+  Value *operator*() {
+    ValueAsMetadata *VAM = I.is<ValueAsMetadata *>()
+                               ? I.get<ValueAsMetadata *>()
+                               : *I.get<ValueAsMetadata **>();
+    return VAM->getValue();
+  }
+  location_op_iterator &operator++() {
+    if (I.is<ValueAsMetadata *>())
+      I = I.get<ValueAsMetadata *>() + 1;
+    else
+      I = I.get<ValueAsMetadata **>() + 1;
+    return *this;
+  }
+  location_op_iterator &operator--() {
+    if (I.is<ValueAsMetadata *>())
+      I = I.get<ValueAsMetadata *>() - 1;
+    else
+      I = I.get<ValueAsMetadata **>() - 1;
+    return *this;
+  }
+};
+
+/// Lightweight class that wraps the location operand metadata of a debug
+/// intrinsic. The raw location may be a ValueAsMetadata, an empty MDTuple,
+/// or a DIArgList.
+class RawLocationWrapper {
+  Metadata *RawLocation = nullptr;
+
+public:
+  RawLocationWrapper() = default;
+  explicit RawLocationWrapper(Metadata *RawLocation)
+      : RawLocation(RawLocation) {
+    // Allow ValueAsMetadata, empty MDTuple, DIArgList.
+    assert(RawLocation && "unexpected null RawLocation");
+    assert(isa<ValueAsMetadata>(RawLocation) || isa<DIArgList>(RawLocation) ||
+           (isa<MDNode>(RawLocation) &&
+            !cast<MDNode>(RawLocation)->getNumOperands()));
+  }
+  Metadata *getRawLocation() const { return RawLocation; }
+  /// Get the locations corresponding to the variable referenced by the debug
+  /// info intrinsic.  Depending on the intrinsic, this could be the
+  /// variable's value or its address.
+  iterator_range<location_op_iterator> location_ops() const;
+  Value *getVariableLocationOp(unsigned OpIdx) const;
+  unsigned getNumVariableLocationOps() const {
+    if (hasArgList())
+      return cast<DIArgList>(getRawLocation())->getArgs().size();
+    return 1;
+  }
+  bool hasArgList() const { return isa<DIArgList>(getRawLocation()); }
+  bool isKillLocation(const DIExpression *Expression) const {
+    return (getNumVariableLocationOps() == 0 && !Expression->isComplex()) ||
+           any_of(location_ops(), [](Value *V) { return isa<UndefValue>(V); });
+  }
+
+  friend bool operator==(const RawLocationWrapper &A,
+                         const RawLocationWrapper &B) {
+    return A.RawLocation == B.RawLocation;
+  }
+  friend bool operator!=(const RawLocationWrapper &A,
+                         const RawLocationWrapper &B) {
+    return !(A == B);
+  }
+  friend bool operator>(const RawLocationWrapper &A,
+                        const RawLocationWrapper &B) {
+    return A.RawLocation > B.RawLocation;
+  }
+  friend bool operator>=(const RawLocationWrapper &A,
+                         const RawLocationWrapper &B) {
+    return A.RawLocation >= B.RawLocation;
+  }
+  friend bool operator<(const RawLocationWrapper &A,
+                        const RawLocationWrapper &B) {
+    return A.RawLocation < B.RawLocation;
+  }
+  friend bool operator<=(const RawLocationWrapper &A,
+                         const RawLocationWrapper &B) {
+    return A.RawLocation <= B.RawLocation;
+  }
+};
 
+/// This is the common base class for debug info intrinsics for variables.
+class DbgVariableIntrinsic : public DbgInfoIntrinsic {
+public:
   /// Get the locations corresponding to the variable referenced by the debug
   /// info intrinsic.  Depending on the intrinsic, this could be the
   /// variable's value or its address.
@@ -251,12 +308,10 @@ class DbgVariableIntrinsic : public DbgInfoIntrinsic {
   }
 
   unsigned getNumVariableLocationOps() const {
-    if (hasArgList())
-      return cast<DIArgList>(getRawLocation())->getArgs().size();
-    return 1;
+    return getWrappedLocation().getNumVariableLocationOps();
   }
 
-  bool hasArgList() const { return isa<DIArgList>(getRawLocation()); }
+  bool hasArgList() const { return getWrappedLocation().hasArgList(); }
 
   /// Does this describe the address of a local variable. True for dbg.declare,
   /// but not dbg.value, which describes its value, or dbg.assign, which
@@ -278,9 +333,7 @@ class DbgVariableIntrinsic : public DbgInfoIntrinsic {
   }
 
   bool isKillLocation() const {
-    return (getNumVariableLocationOps() == 0 &&
-            !getExpression()->isComplex()) ||
-           any_of(location_ops(), [](Value *V) { return isa<UndefValue>(V); });
+    return getWrappedLocation().isKillLocation(getExpression());
   }
 
   DILocalVariable *getVariable() const {
@@ -295,6 +348,10 @@ class DbgVariableIntrinsic : public DbgInfoIntrinsic {
     return cast<MetadataAsValue>(getArgOperand(0))->getMetadata();
   }
 
+  RawLocationWrapper getWrappedLocation() const {
+    return RawLocationWrapper(getRawLocation());
+  }
+
   Metadata *getRawVariable() const {
     return cast<MetadataAsValue>(getArgOperand(1))->getMetadata();
   }
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 84a4ca228f185..0e32ca9c8c7e4 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -404,6 +404,7 @@ class MSBuiltin<string name> {
   string MSBuiltinName = name;
 }
 
+#ifndef TEST_INTRINSICS_SUPPRESS_DEFS
 
 //===--------------- Variable Argument Handling Intrinsics ----------------===//
 //
@@ -1966,7 +1967,7 @@ def int_memset_element_unordered_atomic
 
 //===------------------------ Reduction Intrinsics ------------------------===//
 //
-let IntrProperties = [IntrNoMem] in {
+let IntrProperties = [IntrNoMem, IntrSpeculatable] in {
 
   def int_vector_reduce_fadd : DefaultAttrsIntrinsic<[LLVMVectorElementType<0>],
                                          [LLVMVectorElementType<0>,
@@ -2203,3 +2204,5 @@ include "llvm/IR/IntrinsicsSPIRV.td"
 include "llvm/IR/IntrinsicsVE.td"
 include "llvm/IR/IntrinsicsDirectX.td"
 include "llvm/IR/IntrinsicsLoongArch.td"
+
+#endif // TEST_INTRINSICS_SUPPRESS_DEFS
diff --git a/llvm/include/llvm/IR/IntrinsicsRISCV.td b/llvm/include/llvm/IR/IntrinsicsRISCV.td
index b140cdbe64375..4791dc75b849a 100644
--- a/llvm/include/llvm/IR/IntrinsicsRISCV.td
+++ b/llvm/include/llvm/IR/IntrinsicsRISCV.td
@@ -1477,7 +1477,7 @@ let TargetPrefix = "riscv" in {
                                  LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>],
                                 [NoCapture<ArgIndex<1>>, IntrWriteMem]>;
 
-  // Segment loads for fixed vectors.
+  // Segment loads/stores for fixed vectors.
   foreach nf = [2, 3, 4, 5, 6, 7, 8] in {
     def int_riscv_seg # nf # _load
           : DefaultAttrsIntrinsic<!listconcat([llvm_anyvector_ty],
@@ -1485,6 +1485,13 @@ let TargetPrefix = "riscv" in {
                                               !add(nf, -1))),
                                   [llvm_anyptr_ty, llvm_anyint_ty],
                                   [NoCapture<ArgIndex<0>>, IntrReadMem]>;
+    def int_riscv_seg # nf # _store
+          : DefaultAttrsIntrinsic<[],
+                                  !listconcat([llvm_anyvector_ty],
+                                              !listsplat(LLVMMatchType<0>,
+                                                          !add(nf, -1)),
+                                              [llvm_anyptr_ty, llvm_anyint_ty]),
+                                  [NoCapture<ArgIndex<nf>>, IntrWriteMem]>;
   }
 
 } // TargetPrefix = "riscv"
diff --git a/llvm/include/llvm/IR/PassManager.h b/llvm/include/llvm/IR/PassManager.h
index a9d54bdd2aae2..5fe28f8556e5b 100644
--- a/llvm/include/llvm/IR/PassManager.h
+++ b/llvm/include/llvm/IR/PassManager.h
@@ -516,14 +516,14 @@ class PassManager : public PassInfoMixin<
 
       PreservedAnalyses PassPA = Pass->run(IR, AM, ExtraArgs...);
 
-      // Call onto PassInstrumentation's AfterPass callbacks immediately after
-      // running the pass.
-      PI.runAfterPass<IRUnitT>(*Pass, IR, PassPA);
-
       // Update the analysis manager as each pass runs and potentially
       // invalidates analyses.
       AM.invalidate(IR, PassPA);
 
+      // Call onto PassInstrumentation's AfterPass callbacks immediately after
+      // running the pass.
+      PI.runAfterPass<IRUnitT>(*Pass, IR, PassPA);
+
       // Finally, intersect the preserved analyses to compute the aggregate
       // preserved set for this pass manager.
       PA.intersect(std::move(PassPA));
diff --git a/llvm/include/llvm/IR/SSAContext.h b/llvm/include/llvm/IR/SSAContext.h
index 346c1fc67574c..5180c92d65565 100644
--- a/llvm/include/llvm/IR/SSAContext.h
+++ b/llvm/include/llvm/IR/SSAContext.h
@@ -44,6 +44,7 @@ template <> class GenericSSAContext<Function> {
   using ValueRefT = Value *;
   using ConstValueRefT = const Value *;
   static Value *ValueRefNull;
+  using UseT = Use;
   using DominatorTreeT = DominatorTreeBase<BlockT, false>;
 
   void setFunction(Function &Fn);
diff --git a/llvm/include/llvm/IR/StructuralHash.h b/llvm/include/llvm/IR/StructuralHash.h
index eb63a21403100..1bdeb85afa3c5 100644
--- a/llvm/include/llvm/IR/StructuralHash.h
+++ b/llvm/include/llvm/IR/StructuralHash.h
@@ -1,4 +1,4 @@
-//===- llvm/IR/StructuralHash.h - IR Hash for expensive checks --*- C++ -*-===//
+//===- llvm/IR/StructuralHash.h - IR Hashing --------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
@@ -14,11 +14,8 @@
 #ifndef LLVM_IR_STRUCTURALHASH_H
 #define LLVM_IR_STRUCTURALHASH_H
 
-#ifdef EXPENSIVE_CHECKS
-
 #include <cstdint>
 
-// This header is only meant to be used when -DEXPENSIVE_CHECKS is set
 namespace llvm {
 
 class Function;
@@ -30,5 +27,3 @@ uint64_t StructuralHash(const Module &M);
 } // end namespace llvm
 
 #endif
-
-#endif // LLVM_IR_STRUCTURALHASH_H
diff --git a/llvm/include/llvm/InitializePasses.h b/llvm/include/llvm/InitializePasses.h
index 818dd2e5c45ee..7bf013f43d1be 100644
--- a/llvm/include/llvm/InitializePasses.h
+++ b/llvm/include/llvm/InitializePasses.h
@@ -99,7 +99,6 @@ void initializeCycleInfoWrapperPassPass(PassRegistry &);
 void initializeDAEPass(PassRegistry&);
 void initializeDAHPass(PassRegistry&);
 void initializeDCELegacyPassPass(PassRegistry&);
-void initializeDFAJumpThreadingLegacyPassPass(PassRegistry &);
 void initializeDSELegacyPassPass(PassRegistry&);
 void initializeDeadMachineInstructionElimPass(PassRegistry&);
 void initializeDebugifyMachineModulePass(PassRegistry &);
@@ -137,14 +136,11 @@ void initializeFinalizeMachineBundlesPass(PassRegistry&);
 void initializeFixIrreduciblePass(PassRegistry &);
 void initializeFixupStatepointCallerSavedPass(PassRegistry&);
 void initializeFlattenCFGLegacyPassPass(PassRegistry &);
-void initializeFloat2IntLegacyPassPass(PassRegistry&);
 void initializeForceFunctionAttrsLegacyPassPass(PassRegistry&);
 void initializeFuncletLayoutPass(PassRegistry&);
 void initializeGCMachineCodeAnalysisPass(PassRegistry&);
 void initializeGCModuleInfoPass(PassRegistry&);
-void initializeGVNHoistLegacyPassPass(PassRegistry&);
 void initializeGVNLegacyPassPass(PassRegistry&);
-void initializeGVNSinkLegacyPassPass(PassRegistry&);
 void initializeGlobalDCELegacyPassPass(PassRegistry&);
 void initializeGlobalMergePass(PassRegistry&);
 void initializeGlobalOptLegacyPassPass(PassRegistry&);
@@ -189,7 +185,6 @@ void initializeLegacyLoopSinkPassPass(PassRegistry&);
 void initializeLegalizerPass(PassRegistry&);
 void initializeGISelCSEAnalysisWrapperPassPass(PassRegistry &);
 void initializeGISelKnownBitsAnalysisPass(PassRegistry &);
-void initializeLibCallsShrinkWrapLegacyPassPass(PassRegistry&);
 void initializeLintLegacyPassPass(PassRegistry &);
 void initializeLiveDebugValuesPass(PassRegistry&);
 void initializeLiveDebugVariablesPass(PassRegistry&);
@@ -215,7 +210,6 @@ void initializeLoopSimplifyCFGLegacyPassPass(PassRegistry&);
 void initializeLoopSimplifyPass(PassRegistry&);
 void initializeLoopStrengthReducePass(PassRegistry&);
 void initializeLoopUnrollPass(PassRegistry&);
-void initializeLoopVectorizePass(PassRegistry&);
 void initializeLowerAtomicLegacyPassPass(PassRegistry&);
 void initializeLowerConstantIntrinsicsPass(PassRegistry&);
 void initializeLowerEmuTLSPass(PassRegistry&);
@@ -263,7 +257,6 @@ void initializeMachineTraceMetricsPass(PassRegistry&);
 void initializeMachineUniformityInfoPrinterPassPass(PassRegistry &);
 void initializeMachineUniformityAnalysisPassPass(PassRegistry &);
 void initializeMachineVerifierPassPass(PassRegistry&);
-void initializeMemCpyOptLegacyPassPass(PassRegistry&);
 void initializeMemDepPrinterPass(PassRegistry&);
 void initializeMemDerefPrinterPass(PassRegistry&);
 void initializeMemoryDependenceWrapperPassPass(PassRegistry&);
@@ -278,7 +271,6 @@ void initializeModuloScheduleTestPass(PassRegistry&);
 void initializeMustExecutePrinterPass(PassRegistry&);
 void initializeMustBeExecutedContextPrinterPass(PassRegistry&);
 void initializeNaryReassociateLegacyPassPass(PassRegistry&);
-void initializeNewGVNLegacyPassPass(PassRegistry&);
 void initializeObjCARCContractLegacyPassPass(PassRegistry &);
 void initializeOptimizationRemarkEmitterWrapperPassPass(PassRegistry&);
 void initializeOptimizePHIsPass(PassRegistry&);
@@ -336,7 +328,6 @@ void initializeRewriteStatepointsForGCLegacyPassPass(PassRegistry &);
 void initializeRewriteSymbolsLegacyPassPass(PassRegistry&);
 void initializeSCCPLegacyPassPass(PassRegistry&);
 void initializeSCEVAAWrapperPassPass(PassRegistry&);
-void initializeSLPVectorizerPass(PassRegistry&);
 void initializeSROALegacyPassPass(PassRegistry&);
 void initializeSafeStackLegacyPassPass(PassRegistry&);
 void initializeSafepointIRVerifierPass(PassRegistry&);
diff --git a/llvm/include/llvm/LinkAllPasses.h b/llvm/include/llvm/LinkAllPasses.h
index 3f20916af2f30..55d2a35ca353d 100644
--- a/llvm/include/llvm/LinkAllPasses.h
+++ b/llvm/include/llvm/LinkAllPasses.h
@@ -69,7 +69,6 @@ namespace {
         return;
 
       (void) llvm::createAAEvalPass();
-      (void) llvm::createAggressiveDCEPass();
       (void)llvm::createBitTrackingDCEPass();
       (void) llvm::createAlignmentFromAssumptionsPass();
       (void) llvm::createBasicAAWrapperPass();
@@ -81,11 +80,9 @@ namespace {
       (void) llvm::createCallGraphViewerPass();
       (void) llvm::createCFGSimplificationPass();
       (void) llvm::createStructurizeCFGPass();
-      (void) llvm::createLibCallsShrinkWrapPass();
       (void) llvm::createCostModelAnalysisPass();
       (void) llvm::createDeadArgEliminationPass();
       (void) llvm::createDeadCodeEliminationPass();
-      (void) llvm::createDeadStoreEliminationPass();
       (void) llvm::createDependenceAnalysisWrapperPass();
       (void) llvm::createDomOnlyPrinterWrapperPassPass();
       (void) llvm::createDomPrinterWrapperPassPass();
@@ -96,7 +93,6 @@ namespace {
       (void) llvm::createGuardWideningPass();
       (void) llvm::createLoopGuardWideningPass();
       (void) llvm::createInductiveRangeCheckEliminationPass();
-      (void) llvm::createIndVarSimplifyPass();
       (void) llvm::createInstSimplifyLegacyPass();
       (void) llvm::createInstructionCombiningPass();
       (void) llvm::createJMCInstrumenterPass();
@@ -137,18 +133,13 @@ namespace {
       (void) llvm::createSingleLoopExtractorPass();
       (void) llvm::createTailCallEliminationPass();
       (void)llvm::createTLSVariableHoistPass();
-      (void) llvm::createJumpThreadingPass();
-      (void) llvm::createDFAJumpThreadingPass();
       (void) llvm::createUnifyFunctionExitNodesPass();
       (void) llvm::createInstCountPass();
       (void) llvm::createConstantHoistingPass();
       (void) llvm::createCodeGenPreparePass();
       (void) llvm::createEarlyCSEPass();
-      (void) llvm::createGVNHoistPass();
       (void) llvm::createMergedLoadStoreMotionPass();
       (void) llvm::createGVNPass();
-      (void) llvm::createNewGVNPass();
-      (void) llvm::createMemCpyOptPass();
       (void) llvm::createPostDomTree();
       (void) llvm::createInstructionNamerPass();
       (void) llvm::createMetaRenamerPass();
@@ -164,10 +155,7 @@ namespace {
       (void) llvm::createLintLegacyPassPass();
       (void) llvm::createSinkingPass();
       (void) llvm::createLowerAtomicPass();
-      (void) llvm::createCorrelatedValuePropagationPass();
       (void) llvm::createMemDepPrinter();
-      (void) llvm::createLoopVectorizePass();
-      (void) llvm::createSLPVectorizerPass();
       (void) llvm::createLoadStoreVectorizerPass();
       (void) llvm::createPartiallyInlineLibCallsPass();
       (void) llvm::createScalarizerPass();
@@ -179,7 +167,6 @@ namespace {
       (void) llvm::createMemDerefPrinter();
       (void) llvm::createMustExecutePrinter();
       (void) llvm::createMustBeExecutedContextPrinter();
-      (void) llvm::createFloat2IntPass();
       (void)llvm::createScalarizeMaskedMemIntrinLegacyPass();
       (void) llvm::createHardwareLoopsLegacyPass();
       (void) llvm::createInjectTLIMappingsLegacyPass();
diff --git a/llvm/include/llvm/MC/MCStreamer.h b/llvm/include/llvm/MC/MCStreamer.h
index 5331d9bd3cc45..aa39954d62868 100644
--- a/llvm/include/llvm/MC/MCStreamer.h
+++ b/llvm/include/llvm/MC/MCStreamer.h
@@ -157,7 +157,7 @@ class ARMTargetStreamer : public MCTargetStreamer {
   virtual void emitTextAttribute(unsigned Attribute, StringRef String);
   virtual void emitIntTextAttribute(unsigned Attribute, unsigned IntValue,
                                     StringRef StringValue = "");
-  virtual void emitFPU(unsigned FPU);
+  virtual void emitFPU(ARM::FPUKind FPU);
   virtual void emitArch(ARM::ArchKind Arch);
   virtual void emitArchExtension(uint64_t ArchExt);
   virtual void emitObjectArch(ARM::ArchKind Arch);
diff --git a/llvm/include/llvm/MCA/HardwareUnits/LSUnit.h b/llvm/include/llvm/MCA/HardwareUnits/LSUnit.h
index c05f770df8ebe..9882c4851d6e3 100644
--- a/llvm/include/llvm/MCA/HardwareUnits/LSUnit.h
+++ b/llvm/include/llvm/MCA/HardwareUnits/LSUnit.h
@@ -269,7 +269,7 @@ class LSUnitBase : public HardwareUnit {
   bool isLQFull() const { return LQSize && LQSize == UsedLQEntries; }
 
   bool isValidGroupID(unsigned Index) const {
-    return Index && (Groups.find(Index) != Groups.end());
+    return Index && Groups.contains(Index);
   }
 
   /// Check if a peviously dispatched instruction IR is now ready for execution.
diff --git a/llvm/include/llvm/Object/ELF.h b/llvm/include/llvm/Object/ELF.h
index 1664ff96542f8..0db0ebb5c7c32 100644
--- a/llvm/include/llvm/Object/ELF.h
+++ b/llvm/include/llvm/Object/ELF.h
@@ -14,6 +14,7 @@
 #define LLVM_OBJECT_ELF_H
 
 #include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/MapVector.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
@@ -391,7 +392,21 @@ class ELFFile {
   Expected<ArrayRef<T>> getSectionContentsAsArray(const Elf_Shdr &Sec) const;
   Expected<ArrayRef<uint8_t>> getSectionContents(const Elf_Shdr &Sec) const;
   Expected<ArrayRef<uint8_t>> getSegmentContents(const Elf_Phdr &Phdr) const;
-  Expected<std::vector<BBAddrMap>> decodeBBAddrMap(const Elf_Shdr &Sec) const;
+
+  /// Returns a vector of BBAddrMap structs corresponding to each function
+  /// within the text section that the SHT_LLVM_BB_ADDR_MAP section \p Sec
+  /// is associated with. If the current ELFFile is relocatable, a corresponding
+  /// \p RelaSec must be passed in as an argument.
+  Expected<std::vector<BBAddrMap>>
+  decodeBBAddrMap(const Elf_Shdr &Sec, const Elf_Shdr *RelaSec = nullptr) const;
+
+  /// Returns a map from every section matching \p IsMatch to its relocation
+  /// section, or \p nullptr if it has no relocation section. This function
+  /// returns an error if any of the \p IsMatch calls fail or if it fails to
+  /// retrieve the content section of any relocation section.
+  Expected<MapVector<const Elf_Shdr *, const Elf_Shdr *>>
+  getSectionAndRelocations(
+      std::function<Expected<bool>(const Elf_Shdr &)> IsMatch) const;
 
   void createFakeSections();
 };
diff --git a/llvm/include/llvm/Passes/PassBuilder.h b/llvm/include/llvm/Passes/PassBuilder.h
index 9d9f256f1ceda..40413aad1281c 100644
--- a/llvm/include/llvm/Passes/PassBuilder.h
+++ b/llvm/include/llvm/Passes/PassBuilder.h
@@ -259,11 +259,6 @@ class PassBuilder {
   /// optimization and code generation. It is particularly tuned to fit well
   /// when IR coming into the LTO phase was first run through \c
   /// addPreLinkLTODefaultPipeline, and the two coordinate closely.
-  ///
-  /// Note that \p Level cannot be `O0` here. The pipelines produced are
-  /// only intended for use when attempting to optimize code. If frontends
-  /// require some transformations for semantic reasons, they should explicitly
-  /// build them.
   ModulePassManager
   buildThinLTODefaultPipeline(OptimizationLevel Level,
                               const ModuleSummaryIndex *ImportSummary);
@@ -275,11 +270,6 @@ class PassBuilder {
   /// run. It works to minimize the IR which needs to be analyzed without
   /// making irreversible decisions which could be made better during the LTO
   /// run.
-  ///
-  /// Note that \p Level cannot be `O0` here. The pipelines produced are
-  /// only intended for use when attempting to optimize code. If frontends
-  /// require some transformations for semantic reasons, they should explicitly
-  /// build them.
   ModulePassManager buildLTOPreLinkDefaultPipeline(OptimizationLevel Level);
 
   /// Build an LTO default optimization pipeline to a pass manager.
@@ -288,11 +278,6 @@ class PassBuilder {
   /// optimization and code generation. It is particularly tuned to fit well
   /// when IR coming into the LTO phase was first run through \c
   /// addPreLinkLTODefaultPipeline, and the two coordinate closely.
-  ///
-  /// Note that \p Level cannot be `O0` here. The pipelines produced are
-  /// only intended for use when attempting to optimize code. If frontends
-  /// require some transformations for semantic reasons, they should explicitly
-  /// build them.
   ModulePassManager buildLTODefaultPipeline(OptimizationLevel Level,
                                             ModuleSummaryIndex *ExportSummary);
 
diff --git a/llvm/include/llvm/Passes/StandardInstrumentations.h b/llvm/include/llvm/Passes/StandardInstrumentations.h
index c8614ec496883..759b79dc2b0ba 100644
--- a/llvm/include/llvm/Passes/StandardInstrumentations.h
+++ b/llvm/include/llvm/Passes/StandardInstrumentations.h
@@ -152,9 +152,8 @@ class PreservedCFGCheckerInstrumentation {
   SmallVector<StringRef, 8> PassStack;
 #endif
 
-  static cl::opt<bool> VerifyPreservedCFG;
   void registerCallbacks(PassInstrumentationCallbacks &PIC,
-                         FunctionAnalysisManager &FAM);
+                         ModuleAnalysisManager &MAM);
 };
 
 // Base class for classes that report changes to the IR.
@@ -575,7 +574,7 @@ class StandardInstrumentations {
   // Register all the standard instrumentation callbacks. If \p FAM is nullptr
   // then PreservedCFGChecker is not enabled.
   void registerCallbacks(PassInstrumentationCallbacks &PIC,
-                         FunctionAnalysisManager *FAM = nullptr);
+                         ModuleAnalysisManager *MAM = nullptr);
 
   TimePassesHandler &getTimePasses() { return TimePasses; }
 };
diff --git a/llvm/include/llvm/ProfileData/MemProfData.inc b/llvm/include/llvm/ProfileData/MemProfData.inc
index c533073da751f..b82a4baf6dd74 100644
--- a/llvm/include/llvm/ProfileData/MemProfData.inc
+++ b/llvm/include/llvm/ProfileData/MemProfData.inc
@@ -19,6 +19,7 @@
  * synced up.
  *
 \*===----------------------------------------------------------------------===*/
+#include <string.h>
 
 #ifdef _MSC_VER
 #define PACKED(...) __pragma(pack(push,1)) __VA_ARGS__ __pragma(pack(pop))
@@ -32,7 +33,9 @@
    (uint64_t)'o' << 24 | (uint64_t)'f' << 16 | (uint64_t)'r' << 8 | (uint64_t)129)
 
 // The version number of the raw binary format.
-#define MEMPROF_RAW_VERSION 2ULL
+#define MEMPROF_RAW_VERSION 3ULL
+
+#define MEMPROF_BUILDID_MAX_SIZE 32ULL
 
 namespace llvm {
 namespace memprof {
@@ -46,37 +49,40 @@ PACKED(struct Header {
   uint64_t StackOffset;
 });
 
-
 // A struct describing the information necessary to describe a /proc/maps
 // segment entry for a particular binary/library identified by its build id.
 PACKED(struct SegmentEntry {
   uint64_t Start;
   uint64_t End;
   uint64_t Offset;
-  // This field is unused until sanitizer procmaps support for build ids for
-  // Linux-Elf is implemented.
-  uint8_t BuildId[32] = {0};
+  uint64_t BuildIdSize;
+  uint8_t BuildId[MEMPROF_BUILDID_MAX_SIZE] = {0};
 
-  SegmentEntry(uint64_t S, uint64_t E, uint64_t O) :
-    Start(S), End(E), Offset(O) {}
+  // This constructor is only used in tests so don't set the BuildId.
+  SegmentEntry(uint64_t S, uint64_t E, uint64_t O)
+      : Start(S), End(E), Offset(O), BuildIdSize(0) {}
 
   SegmentEntry(const SegmentEntry& S) {
     Start = S.Start;
     End = S.End;
     Offset = S.Offset;
+    BuildIdSize = S.BuildIdSize;
+    memcpy(BuildId, S.BuildId, S.BuildIdSize);
   }
 
   SegmentEntry& operator=(const SegmentEntry& S) {
     Start = S.Start;
     End = S.End;
     Offset = S.Offset;
+    BuildIdSize = S.BuildIdSize;
+    memcpy(BuildId, S.BuildId, S.BuildIdSize);
     return *this;
   }
 
   bool operator==(const SegmentEntry& S) const {
-    return Start == S.Start &&
-           End == S.End &&
-           Offset == S.Offset;
+    return Start == S.Start && End == S.End && Offset == S.Offset &&
+           BuildIdSize == S.BuildIdSize &&
+           memcmp(BuildId, S.BuildId, S.BuildIdSize) == 0;
   }
 });
 
diff --git a/llvm/include/llvm/Support/CommandLine.h b/llvm/include/llvm/Support/CommandLine.h
index 43c769c1fd0ad..408853a60876b 100644
--- a/llvm/include/llvm/Support/CommandLine.h
+++ b/llvm/include/llvm/Support/CommandLine.h
@@ -315,7 +315,7 @@ class Option {
   }
 
   bool isInAllSubCommands() const {
-    return llvm::is_contained(Subs, &SubCommand::getAll());
+    return Subs.contains(&SubCommand::getAll());
   }
 
   //-------------------------------------------------------------------------===
diff --git a/llvm/include/llvm/Support/RISCVISAInfo.h b/llvm/include/llvm/Support/RISCVISAInfo.h
index 2194359796985..331b6455f8ced 100644
--- a/llvm/include/llvm/Support/RISCVISAInfo.h
+++ b/llvm/include/llvm/Support/RISCVISAInfo.h
@@ -18,7 +18,6 @@
 
 namespace llvm {
 struct RISCVExtensionInfo {
-  std::string ExtName;
   unsigned MajorVersion;
   unsigned MinorVersion;
 };
diff --git a/llvm/include/llvm/Support/ScopedPrinter.h b/llvm/include/llvm/Support/ScopedPrinter.h
index b91acb576ba59..15965bf79ade5 100644
--- a/llvm/include/llvm/Support/ScopedPrinter.h
+++ b/llvm/include/llvm/Support/ScopedPrinter.h
@@ -682,7 +682,7 @@ class JSONScopedPrinter : public ScopedPrinter {
   void printFlagsImpl(StringRef Label, HexNumber Value,
                       ArrayRef<FlagEntry> Flags) override {
     JOS.attributeObject(Label, [&]() {
-      JOS.attribute("RawFlags", hexNumberToInt(Value));
+      JOS.attribute("Value", hexNumberToInt(Value));
       JOS.attributeArray("Flags", [&]() {
         for (const FlagEntry &Flag : Flags) {
           JOS.objectBegin();
@@ -697,7 +697,7 @@ class JSONScopedPrinter : public ScopedPrinter {
   void printFlagsImpl(StringRef Label, HexNumber Value,
                       ArrayRef<HexNumber> Flags) override {
     JOS.attributeObject(Label, [&]() {
-      JOS.attribute("RawFlags", hexNumberToInt(Value));
+      JOS.attribute("Value", hexNumberToInt(Value));
       JOS.attributeArray("Flags", [&]() {
         for (const HexNumber &Flag : Flags) {
           JOS.value(Flag.Value);
@@ -728,8 +728,8 @@ class JSONScopedPrinter : public ScopedPrinter {
 
   void printHexImpl(StringRef Label, StringRef Str, HexNumber Value) override {
     JOS.attributeObject(Label, [&]() {
-      JOS.attribute("Value", Str);
-      JOS.attribute("RawValue", hexNumberToInt(Value));
+      JOS.attribute("Name", Str);
+      JOS.attribute("Value", hexNumberToInt(Value));
     });
   }
 
@@ -744,8 +744,8 @@ class JSONScopedPrinter : public ScopedPrinter {
   void printNumberImpl(StringRef Label, StringRef Str,
                        StringRef Value) override {
     JOS.attributeObject(Label, [&]() {
-      JOS.attribute("Value", Str);
-      JOS.attributeBegin("RawValue");
+      JOS.attribute("Name", Str);
+      JOS.attributeBegin("Value");
       JOS.rawValueBegin() << Value;
       JOS.rawValueEnd();
       JOS.attributeEnd();
diff --git a/llvm/include/llvm/Support/X86FoldTablesUtils.h b/llvm/include/llvm/Support/X86FoldTablesUtils.h
new file mode 100644
index 0000000000000..4829700183139
--- /dev/null
+++ b/llvm/include/llvm/Support/X86FoldTablesUtils.h
@@ -0,0 +1,59 @@
+//===-- X86FoldTablesUtils.h ------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_X86FOLDTABLESUTILS_H
+#define LLVM_SUPPORT_X86FOLDTABLESUTILS_H
+
+namespace {
+enum {
+  // Select which memory operand is being unfolded.
+  // (stored in bits 0 - 2)
+  TB_INDEX_0    = 0,
+  TB_INDEX_1    = 1,
+  TB_INDEX_2    = 2,
+  TB_INDEX_3    = 3,
+  TB_INDEX_4    = 4,
+  TB_INDEX_MASK = 0x7,
+
+  // Do not insert the reverse map (MemOp -> RegOp) into the table.
+  // This may be needed because there is a many -> one mapping.
+  TB_NO_REVERSE   = 1 << 3,
+
+  // Do not insert the forward map (RegOp -> MemOp) into the table.
+  // This is needed for Native Client, which prohibits branch
+  // instructions from using a memory operand.
+  TB_NO_FORWARD   = 1 << 4,
+
+  TB_FOLDED_LOAD  = 1 << 5,
+  TB_FOLDED_STORE = 1 << 6,
+  TB_FOLDED_BCAST = 1 << 7,
+
+  // Minimum alignment required for load/store.
+  // Used for RegOp->MemOp conversion. Encoded as Log2(Align) + 1 to allow 0
+  // to mean align of 0.
+  // (stored in bits 8 - 11)
+  TB_ALIGN_SHIFT = 8,
+  TB_ALIGN_NONE  =   0 << TB_ALIGN_SHIFT,
+  TB_ALIGN_16    =   5 << TB_ALIGN_SHIFT,
+  TB_ALIGN_32    =   6 << TB_ALIGN_SHIFT,
+  TB_ALIGN_64    =   7 << TB_ALIGN_SHIFT,
+  TB_ALIGN_MASK  = 0xf << TB_ALIGN_SHIFT,
+
+  // Broadcast type.
+  // (stored in bits 12 - 13)
+  TB_BCAST_TYPE_SHIFT = 12,
+  TB_BCAST_D    =   0 << TB_BCAST_TYPE_SHIFT,
+  TB_BCAST_Q    =   1 << TB_BCAST_TYPE_SHIFT,
+  TB_BCAST_SS   =   2 << TB_BCAST_TYPE_SHIFT,
+  TB_BCAST_SD   =   3 << TB_BCAST_TYPE_SHIFT,
+  TB_BCAST_MASK = 0x3 << TB_BCAST_TYPE_SHIFT,
+
+  // Unused bits 14-15
+};
+}
+#endif // LLVM_SUPPORT_X86FOLDTABLESUTILS_H
\ No newline at end of file
diff --git a/llvm/include/llvm/TableGen/Record.h b/llvm/include/llvm/TableGen/Record.h
index ba89da3cbe925..6326d62d34865 100644
--- a/llvm/include/llvm/TableGen/Record.h
+++ b/llvm/include/llvm/TableGen/Record.h
@@ -785,7 +785,18 @@ class OpInit : public TypedInit {
 ///
 class UnOpInit : public OpInit, public FoldingSetNode {
 public:
-  enum UnaryOp : uint8_t { CAST, NOT, HEAD, TAIL, SIZE, EMPTY, GETDAGOP, LOG2 };
+  enum UnaryOp : uint8_t {
+    TOLOWER,
+    TOUPPER,
+    CAST,
+    NOT,
+    HEAD,
+    TAIL,
+    SIZE,
+    EMPTY,
+    GETDAGOP,
+    LOG2
+  };
 
 private:
   Init *LHS;
diff --git a/llvm/include/llvm/Target/Target.td b/llvm/include/llvm/Target/Target.td
index 181c8eb17511b..d71336466b5c6 100644
--- a/llvm/include/llvm/Target/Target.td
+++ b/llvm/include/llvm/Target/Target.td
@@ -14,11 +14,13 @@
 // Include all information about LLVM intrinsics.
 include "llvm/IR/Intrinsics.td"
 
+class Predicate; // Forward def
+
 //===----------------------------------------------------------------------===//
 // Register file description - These classes are used to fill in the target
 // description classes.
 
-class HwMode<string FS> {
+class HwMode<string FS, list<Predicate> Ps> {
   // A string representing subtarget features that turn on this HW mode.
   // For example, "+feat1,-feat2" will indicate that the mode is active
   // when "feat1" is enabled and "feat2" is disabled at the same time.
@@ -26,12 +28,15 @@ class HwMode<string FS> {
   // When multiple modes are used, they should be mutually exclusive,
   // otherwise the results are unpredictable.
   string Features = FS;
+
+  // A list of predicates that turn on this HW mode.
+  list<Predicate> Predicates = Ps;
 }
 
 // A special mode recognized by tablegen. This mode is considered active
 // when no other mode is active. For targets that do not use specific hw
 // modes, this is the only mode.
-def DefaultMode : HwMode<"">;
+def DefaultMode : HwMode<"", []>;
 
 // A class used to associate objects with HW modes. It is only intended to
 // be used as a base class, where the derived class should contain a member
@@ -446,8 +451,6 @@ include "llvm/Target/TargetInstrPredicate.td"
 //
 include "llvm/Target/TargetSchedule.td"
 
-class Predicate; // Forward def
-
 class InstructionEncoding {
   // Size of encoded instruction.
   int Size;
@@ -1058,45 +1061,6 @@ class InstrInfo {
   //
   // This option is a temporary migration help. It will go away.
   bit guessInstructionProperties = true;
-
-  // TableGen's instruction encoder generator has support for matching operands
-  // to bit-field variables both by name and by position. Support for matching
-  // by position is DEPRECATED, and WILL BE REMOVED. Positional matching is
-  // confusing to use, and makes it very easy to accidentally write buggy
-  // instruction definitions.
-  //
-  // In previous versions of LLVM, the ability to match operands by position was
-  // enabled unconditionally. It is now controllable by this option -- and
-  // disabled by default. The previous behavior can be restored by setting this
-  // option to true.
-  //
-  // This option is temporary, and will go away once all in-tree targets have
-  // migrated.
-  //
-  // TODO: clean up and remove these options.
-  bit useDeprecatedPositionallyEncodedOperands = false;
-
-  // If positional encoding rules are used for the encoder generator, they may
-  // also need to be used by the decoder generator -- if so, enable this
-  // variable.
-  //
-  // This option is a no-op unless useDeprecatedPositionallyEncodedOperands is
-  // true.
-  //
-  // This option is temporary, and will go away once all in-tree targets have
-  // migrated.
-  bit decodePositionallyEncodedOperands = false;
-
-  // When set, this indicates that there will be no overlap between those
-  // operands that are matched by ordering (positional operands) and those
-  // matched by name.
-  //
-  // This is a no-op unless useDeprecatedPositionallyEncodedOperands is true
-  // (though it does modify the "would've used positional operand XXX" error.)
-  //
-  // This option is temporary, and will go away once all in-tree targets have
-  // migrated.
-  bit noNamedPositionallyEncodedOperands = false;
 }
 
 // Standard Pseudo Instructions.
diff --git a/llvm/include/llvm/Target/TargetOptions.h b/llvm/include/llvm/Target/TargetOptions.h
index d8a4159189394..22e811653c6d4 100644
--- a/llvm/include/llvm/Target/TargetOptions.h
+++ b/llvm/include/llvm/Target/TargetOptions.h
@@ -130,7 +130,7 @@ namespace llvm {
           HonorSignDependentRoundingFPMathOption(false), NoZerosInBSS(false),
           GuaranteedTailCallOpt(false), StackSymbolOrdering(true),
           EnableFastISel(false), EnableGlobalISel(false), UseInitArray(false),
-          LowerGlobalDtorsViaCxaAtExit(false), DisableIntegratedAS(false),
+          DisableIntegratedAS(false),
           RelaxELFRelocations(true), FunctionSections(false),
           DataSections(false), IgnoreXCOFFVisibility(false),
           XCOFFTracebackTable(true), UniqueSectionNames(true),
@@ -247,10 +247,6 @@ namespace llvm {
     /// constructors.
     unsigned UseInitArray : 1;
 
-    /// Use __cxa_atexit to register global destructors; determines how
-    /// llvm.global_dtors is lowered.
-    unsigned LowerGlobalDtorsViaCxaAtExit : 1;
-
     /// Disable the integrated assembler.
     unsigned DisableIntegratedAS : 1;
 
diff --git a/llvm/include/llvm/TargetParser/AArch64TargetParser.h b/llvm/include/llvm/TargetParser/AArch64TargetParser.h
index 348606964ea4b..9cba6b4b19599 100644
--- a/llvm/include/llvm/TargetParser/AArch64TargetParser.h
+++ b/llvm/include/llvm/TargetParser/AArch64TargetParser.h
@@ -25,6 +25,9 @@ namespace llvm {
 class Triple;
 
 namespace AArch64 {
+// Function Multi Versioning CPU features. They must be kept in sync with
+// compiler-rt enum CPUFeatures in lib/builtins/cpu_model.c with FEAT_MAX as
+// sentinel.
 enum CPUFeatures {
   FEAT_RNG,
   FEAT_FLAGM,
@@ -87,6 +90,9 @@ enum CPUFeatures {
   FEAT_MAX
 };
 
+static_assert(FEAT_MAX <= 64,
+              "CPUFeatures enum must not have more than 64 entries");
+
 // Arch extension modifiers for CPUs. These are labelled with their Arm ARM
 // feature name (though the canonical reference for those is AArch64.td)
 // clang-format off
@@ -147,6 +153,7 @@ enum ArchExtKind : uint64_t {
   AEK_SPECRES2 =    1ULL << 53, // FEAT_SPECRES2
   AEK_RASv2 =       1ULL << 54, // FEAT_RASv2
   AEK_ITE =         1ULL << 55, // FEAT_ITE
+  AEK_GCS =         1ULL << 56, // FEAT_GCS
 };
 // clang-format on
 
@@ -155,17 +162,18 @@ enum ArchExtKind : uint64_t {
 // SubtargetFeature which may represent either an actual extension or some
 // internal LLVM property.
 struct ExtensionInfo {
-  StringRef Name;       // Human readable name, e.g. "profile".
-  ArchExtKind ID;       // Corresponding to the ArchExtKind, this extensions
-                        // representation in the bitfield.
-  StringRef Feature;    // -mattr enable string, e.g. "+spe"
-  StringRef NegFeature; // -mattr disable string, e.g. "-spe"
-
-  // FIXME These were added by D127812 FMV support and need documenting:
-  CPUFeatures CPUFeature; // Bitfield value set in __aarch64_cpu_features
-  StringRef DependentFeatures;
-  unsigned FmvPriority;
-  static constexpr unsigned MaxFMVPriority = 1000;
+  StringRef Name;              // Human readable name, e.g. "profile".
+  ArchExtKind ID;              // Corresponding to the ArchExtKind, this
+                               // extensions representation in the bitfield.
+  StringRef Feature;           // -mattr enable string, e.g. "+spe"
+  StringRef NegFeature;        // -mattr disable string, e.g. "-spe"
+  CPUFeatures CPUFeature;      // Function Multi Versioning (FMV) bitfield value
+                               // set in __aarch64_cpu_features
+  StringRef DependentFeatures; // FMV enabled features string,
+                               // e.g. "+dotprod,+fp-armv8,+neon"
+  unsigned FmvPriority;        // FMV feature priority
+  static constexpr unsigned MaxFMVPriority =
+      1000; // Maximum priority for FMV feature
 };
 
 // clang-format off
@@ -249,6 +257,7 @@ inline constexpr ExtensionInfo Extensions[] = {
     {"the", AArch64::AEK_THE, "+the", "-the", FEAT_MAX, "", 0},
     {"tme", AArch64::AEK_TME, "+tme", "-tme", FEAT_MAX, "", 0},
     {"wfxt", AArch64::AEK_NONE, {}, {}, FEAT_WFXT, "+wfxt", 550},
+    {"gcs", AArch64::AEK_GCS, "+gcs", "-gcs", FEAT_MAX, "", 0},
     // Special cases
     {"none", AArch64::AEK_NONE, {}, {}, FEAT_MAX, "", ExtensionInfo::MaxFMVPriority},
 };
@@ -288,6 +297,8 @@ struct ArchInfo {
       return this->Version > Other.Version;
     }
     if (this->Version.getMajor() == 9 && Other.Version.getMajor() == 8) {
+      assert(this->Version.getMinor() && Other.Version.getMinor() &&
+             "AArch64::ArchInfo should have a minor version.");
       return this->Version.getMinor().value_or(0) + 5 >=
              Other.Version.getMinor().value_or(0);
     }
@@ -557,6 +568,10 @@ std::optional<CpuInfo> parseCpu(StringRef Name);
 void fillValidCPUArchList(SmallVectorImpl<StringRef> &Values);
 
 bool isX18ReservedByDefault(const Triple &TT);
+
+// For given feature names, return a bitmask corresponding to the entries of
+// AArch64::CPUFeatures. The values in CPUFeatures are not bitmasks
+// themselves, they are sequential (0, 1, 2, 3, ...).
 uint64_t getCpuSupportsMask(ArrayRef<StringRef> FeatureStrs);
 
 } // namespace AArch64
diff --git a/llvm/include/llvm/TargetParser/ARMTargetParser.h b/llvm/include/llvm/TargetParser/ARMTargetParser.h
index 0723f4b2663c2..9a81415681fdb 100644
--- a/llvm/include/llvm/TargetParser/ARMTargetParser.h
+++ b/llvm/include/llvm/TargetParser/ARMTargetParser.h
@@ -181,7 +181,7 @@ struct ArchNames {
   StringRef Name;
   StringRef CPUAttr; // CPU class in build attributes.
   StringRef ArchFeature;
-  unsigned DefaultFPU;
+  FPUKind DefaultFPU;
   uint64_t ArchBaseExtensions;
   ArchKind ID;
   ARMBuildAttrs::CPUArch ArchAttr; // Arch ID in build attributes.
@@ -213,12 +213,12 @@ inline ArchKind &operator--(ArchKind &Kind) {
 }
 
 // Information by ID
-StringRef getFPUName(unsigned FPUKind);
-FPUVersion getFPUVersion(unsigned FPUKind);
-NeonSupportLevel getFPUNeonSupportLevel(unsigned FPUKind);
-FPURestriction getFPURestriction(unsigned FPUKind);
+StringRef getFPUName(FPUKind FPUKind);
+FPUVersion getFPUVersion(FPUKind FPUKind);
+NeonSupportLevel getFPUNeonSupportLevel(FPUKind FPUKind);
+FPURestriction getFPURestriction(FPUKind FPUKind);
 
-bool getFPUFeatures(unsigned FPUKind, std::vector<StringRef> &Features);
+bool getFPUFeatures(FPUKind FPUKind, std::vector<StringRef> &Features);
 bool getHWDivFeatures(uint64_t HWDivKind, std::vector<StringRef> &Features);
 bool getExtensionFeatures(uint64_t Extensions,
                           std::vector<StringRef> &Features);
@@ -231,11 +231,11 @@ StringRef getArchExtName(uint64_t ArchExtKind);
 StringRef getArchExtFeature(StringRef ArchExt);
 bool appendArchExtFeatures(StringRef CPU, ARM::ArchKind AK, StringRef ArchExt,
                            std::vector<StringRef> &Features,
-                           unsigned &ArgFPUKind);
+                           FPUKind &ArgFPUKind);
 ArchKind convertV9toV8(ArchKind AK);
 
 // Information by Name
-unsigned getDefaultFPU(StringRef CPU, ArchKind AK);
+FPUKind getDefaultFPU(StringRef CPU, ArchKind AK);
 uint64_t getDefaultExtensions(StringRef CPU, ArchKind AK);
 StringRef getDefaultCPU(StringRef Arch);
 StringRef getCanonicalArchName(StringRef Arch);
@@ -243,7 +243,7 @@ StringRef getFPUSynonym(StringRef FPU);
 
 // Parser
 uint64_t parseHWDiv(StringRef HWDiv);
-unsigned parseFPU(StringRef FPU);
+FPUKind parseFPU(StringRef FPU);
 ArchKind parseArch(StringRef Arch);
 uint64_t parseArchExt(StringRef ArchExt);
 ArchKind parseCPUArch(StringRef CPU);
diff --git a/llvm/include/llvm/Transforms/IPO/Attributor.h b/llvm/include/llvm/Transforms/IPO/Attributor.h
index 26859ff77ce4d..e8159d2c25fe2 100644
--- a/llvm/include/llvm/Transforms/IPO/Attributor.h
+++ b/llvm/include/llvm/Transforms/IPO/Attributor.h
@@ -2567,7 +2567,10 @@ template <typename base_ty = uint32_t, base_ty BestState = ~base_ty(0),
           base_ty WorstState = 0>
 struct BitIntegerState
     : public IntegerStateBase<base_ty, BestState, WorstState> {
+  using super = IntegerStateBase<base_ty, BestState, WorstState>;
   using base_t = base_ty;
+  BitIntegerState() = default;
+  BitIntegerState(base_t Assumed) : super(Assumed) {}
 
   /// Return true if the bits set in \p BitsEncoding are "known bits".
   bool isKnown(base_t BitsEncoding) const {
@@ -4840,6 +4843,39 @@ struct AANoUndef
   static const char ID;
 };
 
+struct AANoFPClass
+    : public IRAttribute<
+          Attribute::NoFPClass,
+          StateWrapper<BitIntegerState<uint32_t, fcAllFlags, fcNone>,
+                       AbstractAttribute>> {
+  using Base = StateWrapper<BitIntegerState<uint32_t, fcAllFlags, fcNone>,
+                            AbstractAttribute>;
+
+  AANoFPClass(const IRPosition &IRP, Attributor &A) : IRAttribute(IRP) {}
+
+  /// Return true if we assume that the underlying value is nofpclass.
+  FPClassTest getAssumedNoFPClass() const {
+    return static_cast<FPClassTest>(getAssumed());
+  }
+
+  /// Create an abstract attribute view for the position \p IRP.
+  static AANoFPClass &createForPosition(const IRPosition &IRP, Attributor &A);
+
+  /// See AbstractAttribute::getName()
+  const std::string getName() const override { return "AANoFPClass"; }
+
+  /// See AbstractAttribute::getIdAddr()
+  const char *getIdAddr() const override { return &ID; }
+
+  /// This function should return true if the type of the \p AA is AANoFPClass
+  static bool classof(const AbstractAttribute *AA) {
+    return (AA->getIdAddr() == &ID);
+  }
+
+  /// Unique ID (due to the unique address)
+  static const char ID;
+};
+
 struct AACallGraphNode;
 struct AACallEdges;
 
diff --git a/llvm/include/llvm/Transforms/IPO/DeadArgumentElimination.h b/llvm/include/llvm/Transforms/IPO/DeadArgumentElimination.h
index a71fa3bf404de..63e1ad043d49f 100644
--- a/llvm/include/llvm/Transforms/IPO/DeadArgumentElimination.h
+++ b/llvm/include/llvm/Transforms/IPO/DeadArgumentElimination.h
@@ -136,6 +136,7 @@ class DeadArgumentEliminationPass
   bool removeDeadStuffFromFunction(Function *F);
   bool deleteDeadVarargs(Function &F);
   bool removeDeadArgumentsFromCallers(Function &F);
+  void propagateVirtMustcallLiveness(const Module &M);
 };
 
 } // end namespace llvm
diff --git a/llvm/include/llvm/Transforms/IPO/FunctionAttrs.h b/llvm/include/llvm/Transforms/IPO/FunctionAttrs.h
index 1cab52b2cca82..0b93bbd6eaa2d 100644
--- a/llvm/include/llvm/Transforms/IPO/FunctionAttrs.h
+++ b/llvm/include/llvm/Transforms/IPO/FunctionAttrs.h
@@ -48,8 +48,16 @@ bool thinLTOPropagateFunctionAttrs(
 /// attribute. It also discovers function arguments that are not captured by
 /// the function and marks them with the nocapture attribute.
 struct PostOrderFunctionAttrsPass : PassInfoMixin<PostOrderFunctionAttrsPass> {
+  PostOrderFunctionAttrsPass(bool SkipNonRecursive = false)
+      : SkipNonRecursive(SkipNonRecursive) {}
   PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
                         LazyCallGraph &CG, CGSCCUpdateResult &UR);
+
+  void printPipeline(raw_ostream &OS,
+                     function_ref<StringRef(StringRef)> MapClassName2PassName);
+
+private:
+  bool SkipNonRecursive;
 };
 
 /// A pass to do RPO deduction and propagation of function attributes.
diff --git a/llvm/include/llvm/Transforms/IPO/FunctionSpecialization.h b/llvm/include/llvm/Transforms/IPO/FunctionSpecialization.h
index 8db246f739ab1..831717c765169 100644
--- a/llvm/include/llvm/Transforms/IPO/FunctionSpecialization.h
+++ b/llvm/include/llvm/Transforms/IPO/FunctionSpecialization.h
@@ -127,11 +127,7 @@ class FunctionSpecializer {
   std::function<TargetTransformInfo &(Function &)> GetTTI;
   std::function<AssumptionCache &(Function &)> GetAC;
 
-  // The number of functions specialised, used for collecting statistics and
-  // also in the cost model.
-  unsigned NbFunctionsSpecialized = 0;
-
-  SmallPtrSet<Function *, 32> SpecializedFuncs;
+  SmallPtrSet<Function *, 32> Specializations;
   SmallPtrSet<Function *, 32> FullySpecialized;
   DenseMap<Function *, CodeMetrics> FunctionMetrics;
 
@@ -144,13 +140,9 @@ class FunctionSpecializer {
       : Solver(Solver), M(M), FAM(FAM), GetTLI(GetTLI), GetTTI(GetTTI),
         GetAC(GetAC) {}
 
-  ~FunctionSpecializer() {
-    // Eliminate dead code.
-    removeDeadFunctions();
-    cleanUpSSA();
-  }
+  ~FunctionSpecializer();
 
-  bool isClonedFunction(Function *F) { return SpecializedFuncs.count(F); }
+  bool isClonedFunction(Function *F) { return Specializations.count(F); }
 
   bool run();
 
diff --git a/llvm/include/llvm/Transforms/IPO/PassManagerBuilder.h b/llvm/include/llvm/Transforms/IPO/PassManagerBuilder.h
deleted file mode 100644
index 3930cb8a0e05e..0000000000000
--- a/llvm/include/llvm/Transforms/IPO/PassManagerBuilder.h
+++ /dev/null
@@ -1,134 +0,0 @@
-// llvm/Transforms/IPO/PassManagerBuilder.h - Build Standard Pass -*- C++ -*-=//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// This file defines the PassManagerBuilder class, which is used to set up a
-// "standard" optimization sequence suitable for languages like C and C++.
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LLVM_TRANSFORMS_IPO_PASSMANAGERBUILDER_H
-#define LLVM_TRANSFORMS_IPO_PASSMANAGERBUILDER_H
-
-#include "llvm-c/Transforms/PassManagerBuilder.h"
-#include <functional>
-#include <string>
-#include <vector>
-
-namespace llvm {
-class ModuleSummaryIndex;
-class Pass;
-class TargetLibraryInfoImpl;
-
-// The old pass manager infrastructure is hidden in a legacy namespace now.
-namespace legacy {
-class FunctionPassManager;
-class PassManagerBase;
-}
-
-/// PassManagerBuilder - This class is used to set up a standard optimization
-/// sequence for languages like C and C++, allowing some APIs to customize the
-/// pass sequence in various ways. A simple example of using it would be:
-///
-///  PassManagerBuilder Builder;
-///  Builder.OptLevel = 2;
-///  Builder.populateFunctionPassManager(FPM);
-///  Builder.populateModulePassManager(MPM);
-///
-/// In addition to setting up the basic passes, PassManagerBuilder allows
-/// frontends to vend a plugin API, where plugins are allowed to add extensions
-/// to the default pass manager.  They do this by specifying where in the pass
-/// pipeline they want to be added, along with a callback function that adds
-/// the pass(es).  For example, a plugin that wanted to add a loop optimization
-/// could do something like this:
-///
-/// static void addMyLoopPass(const PMBuilder &Builder, PassManagerBase &PM) {
-///   if (Builder.getOptLevel() > 2 && Builder.getOptSizeLevel() == 0)
-///     PM.add(createMyAwesomePass());
-/// }
-///   ...
-///   Builder.addExtension(PassManagerBuilder::EP_LoopOptimizerEnd,
-///                        addMyLoopPass);
-///   ...
-class PassManagerBuilder {
-public:
-  /// Extensions are passed to the builder itself (so they can see how it is
-  /// configured) as well as the pass manager to add stuff to.
-  typedef std::function<void(const PassManagerBuilder &Builder,
-                             legacy::PassManagerBase &PM)>
-      ExtensionFn;
-  typedef int GlobalExtensionID;
-
-  /// The Optimization Level - Specify the basic optimization level.
-  ///    0 = -O0, 1 = -O1, 2 = -O2, 3 = -O3
-  unsigned OptLevel;
-
-  /// SizeLevel - How much we're optimizing for size.
-  ///    0 = none, 1 = -Os, 2 = -Oz
-  unsigned SizeLevel;
-
-  /// LibraryInfo - Specifies information about the runtime library for the
-  /// optimizer.  If this is non-null, it is added to both the function and
-  /// per-module pass pipeline.
-  TargetLibraryInfoImpl *LibraryInfo;
-
-  /// Inliner - Specifies the inliner to use.  If this is non-null, it is
-  /// added to the per-module passes.
-  Pass *Inliner;
-
-  /// The module summary index to use for exporting information from the
-  /// regular LTO phase, for example for the CFI and devirtualization type
-  /// tests.
-  ModuleSummaryIndex *ExportSummary = nullptr;
-
-  /// The module summary index to use for importing information to the
-  /// thin LTO backends, for example for the CFI and devirtualization type
-  /// tests.
-  const ModuleSummaryIndex *ImportSummary = nullptr;
-
-  bool DisableUnrollLoops;
-  bool CallGraphProfile;
-  bool SLPVectorize;
-  bool LoopVectorize;
-  bool LoopsInterleaved;
-  bool DisableGVNLoadPRE;
-  bool ForgetAllSCEVInLoopUnroll;
-  bool VerifyInput;
-  bool VerifyOutput;
-  bool DivergentTarget;
-  unsigned LicmMssaOptCap;
-  unsigned LicmMssaNoAccForPromotionCap;
-
-public:
-  PassManagerBuilder();
-  ~PassManagerBuilder();
-
-private:
-  void addInitialAliasAnalysisPasses(legacy::PassManagerBase &PM) const;
-  void addFunctionSimplificationPasses(legacy::PassManagerBase &MPM);
-  void addVectorPasses(legacy::PassManagerBase &PM, bool IsFullLTO);
-
-public:
-  /// populateFunctionPassManager - This fills in the function pass manager,
-  /// which is expected to be run on each function immediately as it is
-  /// generated.  The idea is to reduce the size of the IR in memory.
-  void populateFunctionPassManager(legacy::FunctionPassManager &FPM);
-
-  /// populateModulePassManager - This sets up the primary pass manager.
-  void populateModulePassManager(legacy::PassManagerBase &MPM);
-};
-
-inline PassManagerBuilder *unwrap(LLVMPassManagerBuilderRef P) {
-    return reinterpret_cast<PassManagerBuilder*>(P);
-}
-
-inline LLVMPassManagerBuilderRef wrap(PassManagerBuilder *P) {
-  return reinterpret_cast<LLVMPassManagerBuilderRef>(P);
-}
-
-} // end namespace llvm
-#endif
diff --git a/llvm/include/llvm/Transforms/Scalar.h b/llvm/include/llvm/Transforms/Scalar.h
index ae91ddca7b81e..c188df4b248fb 100644
--- a/llvm/include/llvm/Transforms/Scalar.h
+++ b/llvm/include/llvm/Transforms/Scalar.h
@@ -47,13 +47,6 @@ Pass *createRedundantDbgInstEliminationPass();
 //
 FunctionPass *createDeadCodeEliminationPass();
 
-//===----------------------------------------------------------------------===//
-//
-// DeadStoreElimination - This pass deletes stores that are post-dominated by
-// must-aliased stores and are not loaded used between the stores.
-//
-FunctionPass *createDeadStoreEliminationPass();
-
 
 //===----------------------------------------------------------------------===//
 //
@@ -61,14 +54,6 @@ FunctionPass *createDeadStoreEliminationPass();
 // values.
 FunctionPass *createCallSiteSplittingPass();
 
-//===----------------------------------------------------------------------===//
-//
-// AggressiveDCE - This pass uses the SSA based Aggressive DCE algorithm.  This
-// algorithm assumes instructions are dead until proven otherwise, which makes
-// it more successful are removing non-obviously dead instructions.
-//
-FunctionPass *createAggressiveDCEPass();
-
 //===----------------------------------------------------------------------===//
 //
 // GuardWidening - An optimization over the @llvm.experimental.guard intrinsic
@@ -108,13 +93,6 @@ FunctionPass *createSROAPass(bool PreserveCFG = true);
 //
 Pass *createInductiveRangeCheckEliminationPass();
 
-//===----------------------------------------------------------------------===//
-//
-// InductionVariableSimplify - Transform induction variables in a program to all
-// use a single canonical induction variable per loop.
-//
-Pass *createIndVarSimplifyPass();
-
 //===----------------------------------------------------------------------===//
 //
 // LICM - This pass is a loop invariant code motion and memory promotion pass.
@@ -188,22 +166,6 @@ extern char &DemoteRegisterToMemoryID;
 //
 FunctionPass *createReassociatePass();
 
-//===----------------------------------------------------------------------===//
-//
-// JumpThreading - Thread control through mult-pred/multi-succ blocks where some
-// preds always go to some succ. Thresholds other than minus one
-// override the internal BB duplication default threshold.
-//
-FunctionPass *createJumpThreadingPass(int Threshold = -1);
-
-//===----------------------------------------------------------------------===//
-//
-// DFAJumpThreading - When a switch statement inside a loop is used to
-// implement a deterministic finite automata we can jump thread the switch
-// statement reducing number of conditional jumps.
-//
-FunctionPass *createDFAJumpThreadingPass();
-
 //===----------------------------------------------------------------------===//
 //
 // CFGSimplification - Merge basic blocks, eliminate unreachable blocks,
@@ -243,20 +205,6 @@ FunctionPass *createTailCallEliminationPass();
 //
 FunctionPass *createEarlyCSEPass(bool UseMemorySSA = false);
 
-//===----------------------------------------------------------------------===//
-//
-// GVNHoist - This pass performs a simple and fast GVN pass over the dominator
-// tree to hoist common expressions from sibling branches.
-//
-FunctionPass *createGVNHoistPass();
-
-//===----------------------------------------------------------------------===//
-//
-// GVNSink - This pass uses an "inverted" value numbering to decide the
-// similarity of expressions and sinks similar expressions into successors.
-//
-FunctionPass *createGVNSinkPass();
-
 //===----------------------------------------------------------------------===//
 //
 // MergedLoadStoreMotion - This pass merges loads and stores in diamonds. Loads
@@ -264,20 +212,6 @@ FunctionPass *createGVNSinkPass();
 //
 FunctionPass *createMergedLoadStoreMotionPass(bool SplitFooterBB = false);
 
-//===----------------------------------------------------------------------===//
-//
-// GVN - This pass performs global value numbering and redundant load
-// elimination cotemporaneously.
-//
-FunctionPass *createNewGVNPass();
-
-//===----------------------------------------------------------------------===//
-//
-// MemCpyOpt - This pass performs optimizations related to eliminating memcpy
-// calls and/or combining multiple stores into memset's.
-//
-FunctionPass *createMemCpyOptPass();
-
 //===----------------------------------------------------------------------===//
 //
 // ConstantHoisting - This pass prepares a function for expensive constants.
@@ -327,12 +261,6 @@ Pass *createLowerWidenableConditionPass();
 //
 Pass *createMergeICmpsLegacyPass();
 
-//===----------------------------------------------------------------------===//
-//
-// ValuePropagation - Propagate CFG-derived value information
-//
-Pass *createCorrelatedValuePropagationPass();
-
 //===----------------------------------------------------------------------===//
 //
 // InferAddressSpaces - Modify users of addrspacecast instructions with values
@@ -410,12 +338,6 @@ FunctionPass *createPlaceSafepointsPass();
 //
 ModulePass *createRewriteStatepointsForGCLegacyPass();
 
-//===----------------------------------------------------------------------===//
-//
-// Float2Int - Demote floats to ints where possible.
-//
-FunctionPass *createFloat2IntPass();
-
 //===----------------------------------------------------------------------===//
 //
 // NaryReassociate - Simplify n-ary operations by reassociation.
@@ -428,13 +350,6 @@ FunctionPass *createNaryReassociatePass();
 //
 FunctionPass *createLoopDataPrefetchPass();
 
-//===----------------------------------------------------------------------===//
-//
-// LibCallsShrinkWrap - Shrink-wraps a call to function if the result is not
-// used.
-//
-FunctionPass *createLibCallsShrinkWrapPass();
-
 //===----------------------------------------------------------------------===//
 //
 // LoopSimplifyCFG - This pass performs basic CFG simplification on loops,
diff --git a/llvm/include/llvm/Transforms/Scalar/LoopPassManager.h b/llvm/include/llvm/Transforms/Scalar/LoopPassManager.h
index b27df810da533..c8c4d5e9a5392 100644
--- a/llvm/include/llvm/Transforms/Scalar/LoopPassManager.h
+++ b/llvm/include/llvm/Transforms/Scalar/LoopPassManager.h
@@ -426,7 +426,7 @@ std::optional<PreservedAnalyses> LoopPassManager::runSinglePass(
 /// The adaptor comes with two modes: the loop mode and the loop-nest mode, and
 /// the worklist updater lived inside will be in the same mode as the adaptor
 /// (refer to the documentation of \c LPMUpdater for more detailed explanation).
-/// Specifically, in loop mode, all loops in the funciton will be pushed into
+/// Specifically, in loop mode, all loops in the function will be pushed into
 /// the worklist and processed by \p Pass, while only top-level loops are
 /// processed in loop-nest mode. Please refer to the various specializations of
 /// \fn createLoopFunctionToLoopPassAdaptor to see when loop mode and loop-nest
diff --git a/llvm/include/llvm/Transforms/Utils/AssumeBundleBuilder.h b/llvm/include/llvm/Transforms/Utils/AssumeBundleBuilder.h
index 991ecb8efbd07..30752e9cd79fc 100644
--- a/llvm/include/llvm/Transforms/Utils/AssumeBundleBuilder.h
+++ b/llvm/include/llvm/Transforms/Utils/AssumeBundleBuilder.h
@@ -40,7 +40,8 @@ AssumeInst *buildAssumeFromInst(Instruction *I);
 /// become silently be invalid.
 /// The DominatorTree can optionally be provided to enable cross-block
 /// reasoning.
-void salvageKnowledge(Instruction *I, AssumptionCache *AC = nullptr,
+/// This returns if a change was made.
+bool salvageKnowledge(Instruction *I, AssumptionCache *AC = nullptr,
                       DominatorTree *DT = nullptr);
 
 /// Build and return a new assume created from the provided knowledge
diff --git a/llvm/include/llvm/Transforms/Utils/BasicBlockUtils.h b/llvm/include/llvm/Transforms/Utils/BasicBlockUtils.h
index c97baaf4afc2f..bc1bba1ff59ac 100644
--- a/llvm/include/llvm/Transforms/Utils/BasicBlockUtils.h
+++ b/llvm/include/llvm/Transforms/Utils/BasicBlockUtils.h
@@ -32,6 +32,7 @@ class BlockFrequencyInfo;
 class BranchProbabilityInfo;
 class DomTreeUpdater;
 class Function;
+class IRBuilderBase;
 class LoopInfo;
 class MDNode;
 class MemoryDependenceResults;
@@ -474,6 +475,25 @@ void SplitBlockAndInsertIfThenElse(Value *Cond, Instruction *SplitBefore,
                                    MDNode *BranchWeights = nullptr,
                                    DomTreeUpdater *DTU = nullptr);
 
+/// Insert a for (int i = 0; i < End; i++) loop structure (with the exception
+/// that \p End is assumed > 0, and thus not checked on entry) at \p
+/// SplitBefore.  Returns the first insert point in the loop body, and the
+/// PHINode for the induction variable (i.e. "i" above).
+std::pair<Instruction*, Value*>
+SplitBlockAndInsertSimpleForLoop(Value *End, Instruction *SplitBefore);
+
+/// Utility function for performing a given action on each lane of a vector
+/// with \p EC elements.  To simplify porting legacy code, this defaults to
+/// unrolling the implied loop for non-scalable element counts, but this is
+/// not considered to be part of the contract of this routine, and is
+/// expected to change in the future. The callback takes as arguments an
+/// IRBuilder whose insert point is correctly set for instantiating the
+/// given index, and a value which is (at runtime) the index to access.
+/// This index *may* be a constant.
+void SplitBlockAndInsertForEachLane(ElementCount EC, Type *IndexTy,
+    Instruction *InsertBefore,
+    std::function<void(IRBuilderBase&, Value*)> Func);
+
 /// Check whether BB is the merge point of a if-region.
 /// If so, return the branch instruction that determines which entry into
 /// BB will be taken.  Also, return by references the block that will be
diff --git a/llvm/include/llvm/Transforms/Utils/Debugify.h b/llvm/include/llvm/Transforms/Utils/Debugify.h
index 24b9eeab6ee45..d4440942a64ef 100644
--- a/llvm/include/llvm/Transforms/Utils/Debugify.h
+++ b/llvm/include/llvm/Transforms/Utils/Debugify.h
@@ -192,8 +192,8 @@ class DebugifyEachInstrumentation {
   DebugifyStatsMap *DIStatsMap = nullptr;
 
 public:
-
-  void registerCallbacks(PassInstrumentationCallbacks &PIC);
+  void registerCallbacks(PassInstrumentationCallbacks &PIC,
+                         ModuleAnalysisManager &MAM);
   // Used within DebugifyMode::SyntheticDebugInfo mode.
   void setDIStatsMap(DebugifyStatsMap &StatMap) { DIStatsMap = &StatMap; }
   const DebugifyStatsMap &getDebugifyStatsMap() const { return *DIStatsMap; }
diff --git a/llvm/include/llvm/Transforms/Utils/SimplifyLibCalls.h b/llvm/include/llvm/Transforms/Utils/SimplifyLibCalls.h
index 90a5ffc5b4fa0..11c971754a9e6 100644
--- a/llvm/include/llvm/Transforms/Utils/SimplifyLibCalls.h
+++ b/llvm/include/llvm/Transforms/Utils/SimplifyLibCalls.h
@@ -198,7 +198,7 @@ class LibCallSimplifier {
   Value *optimizeFMinFMax(CallInst *CI, IRBuilderBase &B);
   Value *optimizeLog(CallInst *CI, IRBuilderBase &B);
   Value *optimizeSqrt(CallInst *CI, IRBuilderBase &B);
-  Value *optimizeSinCosPi(CallInst *CI, IRBuilderBase &B);
+  Value *optimizeSinCosPi(CallInst *CI, bool IsSin, IRBuilderBase &B);
   Value *optimizeTan(CallInst *CI, IRBuilderBase &B);
   // Wrapper for all floating point library call optimizations
   Value *optimizeFloatingPointLibCall(CallInst *CI, LibFunc Func,
diff --git a/llvm/include/llvm/Transforms/Utils/ValueMapper.h b/llvm/include/llvm/Transforms/Utils/ValueMapper.h
index 95fd0b14dd511..5f15af7f99903 100644
--- a/llvm/include/llvm/Transforms/Utils/ValueMapper.h
+++ b/llvm/include/llvm/Transforms/Utils/ValueMapper.h
@@ -112,8 +112,9 @@ inline RemapFlags operator|(RemapFlags LHS, RemapFlags RHS) {
 /// There are a number of top-level entry points:
 /// - \a mapValue() (and \a mapConstant());
 /// - \a mapMetadata() (and \a mapMDNode());
-/// - \a remapInstruction(); and
-/// - \a remapFunction().
+/// - \a remapInstruction();
+/// - \a remapFunction(); and
+/// - \a remapGlobalObjectMetadata().
 ///
 /// The \a ValueMaterializer can be used as a callback, but cannot invoke any
 /// of these top-level functions recursively.  Instead, callbacks should use
@@ -175,6 +176,7 @@ class ValueMapper {
 
   void remapInstruction(Instruction &I);
   void remapFunction(Function &F);
+  void remapGlobalObjectMetadata(GlobalObject &GO);
 
   void scheduleMapGlobalInitializer(GlobalVariable &GV, Constant &Init,
                                     unsigned MappingContextID = 0);
diff --git a/llvm/include/llvm/Transforms/Vectorize.h b/llvm/include/llvm/Transforms/Vectorize.h
index bca78d0730033..b77ebbc55e7de 100644
--- a/llvm/include/llvm/Transforms/Vectorize.h
+++ b/llvm/include/llvm/Transforms/Vectorize.h
@@ -104,20 +104,6 @@ struct VectorizeConfig {
   VectorizeConfig();
 };
 
-//===----------------------------------------------------------------------===//
-//
-// LoopVectorize - Create a loop vectorization pass.
-//
-Pass *createLoopVectorizePass();
-Pass *createLoopVectorizePass(bool InterleaveOnlyWhenForced,
-                              bool VectorizeOnlyWhenForced);
-
-//===----------------------------------------------------------------------===//
-//
-// SLPVectorizer - Create a bottom-up SLP vectorizer pass.
-//
-Pass *createSLPVectorizerPass();
-
 //===----------------------------------------------------------------------===//
 /// Vectorize the BasicBlock.
 ///
diff --git a/llvm/include/llvm/Transforms/Vectorize/LoopVectorize.h b/llvm/include/llvm/Transforms/Vectorize/LoopVectorize.h
index 97011052ea20b..da9fec8d63e9a 100644
--- a/llvm/include/llvm/Transforms/Vectorize/LoopVectorize.h
+++ b/llvm/include/llvm/Transforms/Vectorize/LoopVectorize.h
@@ -15,7 +15,7 @@
 // 'wide' iteration. After this transformation the index is incremented
 // by the SIMD vector width, and not by one.
 //
-// This pass has three parts:
+// This pass has four parts:
 // 1. The main loop pass that drives the different parts.
 // 2. LoopVectorizationLegality - A unit that checks for the legality
 //    of the vectorization.
diff --git a/llvm/include/llvm/WindowsDriver/MSVCPaths.h b/llvm/include/llvm/WindowsDriver/MSVCPaths.h
index d52b33d651694..51ffd6b6bc2c2 100644
--- a/llvm/include/llvm/WindowsDriver/MSVCPaths.h
+++ b/llvm/include/llvm/WindowsDriver/MSVCPaths.h
@@ -90,11 +90,15 @@ bool findVCToolChainViaEnvironment(vfs::FileSystem &VFS, std::string &Path,
                                    ToolsetLayout &VSLayout);
 
 // Query the Setup Config server for installs, then pick the newest version
-// and find its default VC toolchain.
+// and find its default VC toolchain. If `VCToolsVersion` is specified, that
+// version is preferred over the latest version.
+//
 // This is the preferred way to discover new Visual Studios, as they're no
 // longer listed in the registry.
-bool findVCToolChainViaSetupConfig(vfs::FileSystem &VFS, std::string &Path,
-                                   ToolsetLayout &VSLayout);
+bool
+findVCToolChainViaSetupConfig(vfs::FileSystem &VFS,
+                              std::optional<llvm::StringRef> VCToolsVersion,
+                              std::string &Path, ToolsetLayout &VSLayout);
 
 // Look in the registry for Visual Studio installs, and use that to get
 // a toolchain path. VS2017 and newer don't get added to the registry.
diff --git a/llvm/lib/Analysis/AssumptionCache.cpp b/llvm/lib/Analysis/AssumptionCache.cpp
index 11796ef504446..b439dc1e6a76d 100644
--- a/llvm/lib/Analysis/AssumptionCache.cpp
+++ b/llvm/lib/Analysis/AssumptionCache.cpp
@@ -87,7 +87,7 @@ findAffectedValues(CallBase *CI, TargetTransformInfo *TTI,
   AddAffected(Cond);
 
   CmpInst::Predicate Pred;
-  if (match(Cond, m_ICmp(Pred, m_Value(A), m_Value(B)))) {
+  if (match(Cond, m_Cmp(Pred, m_Value(A), m_Value(B)))) {
     AddAffected(A);
     AddAffected(B);
 
@@ -128,7 +128,18 @@ findAffectedValues(CallBase *CI, TargetTransformInfo *TTI,
       if (match(A, m_Add(m_Value(X), m_ConstantInt())) &&
           match(B, m_ConstantInt()))
         AddAffected(X);
+    } else if (CmpInst::isFPPredicate(Pred)) {
+      // fcmp fneg(x), y
+      // fcmp fabs(x), y
+      // fcmp fneg(fabs(x)), y
+      if (match(A, m_FNeg(m_Value(A))))
+        AddAffected(A);
+      if (match(A, m_FAbs(m_Value(A))))
+        AddAffected(A);
     }
+  } else if (match(Cond, m_Intrinsic<Intrinsic::is_fpclass>(m_Value(A),
+                                                            m_Value(B)))) {
+    AddAffected(A);
   }
 
   if (TTI) {
diff --git a/llvm/lib/Analysis/BlockFrequencyInfo.cpp b/llvm/lib/Analysis/BlockFrequencyInfo.cpp
index dd84336da604b..b18d04cc73dbc 100644
--- a/llvm/lib/Analysis/BlockFrequencyInfo.cpp
+++ b/llvm/lib/Analysis/BlockFrequencyInfo.cpp
@@ -333,9 +333,10 @@ bool BlockFrequencyInfoWrapperPass::runOnFunction(Function &F) {
 AnalysisKey BlockFrequencyAnalysis::Key;
 BlockFrequencyInfo BlockFrequencyAnalysis::run(Function &F,
                                                FunctionAnalysisManager &AM) {
+  auto &BP = AM.getResult<BranchProbabilityAnalysis>(F);
+  auto &LI = AM.getResult<LoopAnalysis>(F);
   BlockFrequencyInfo BFI;
-  BFI.calculate(F, AM.getResult<BranchProbabilityAnalysis>(F),
-                AM.getResult<LoopAnalysis>(F));
+  BFI.calculate(F, BP, LI);
   return BFI;
 }
 
diff --git a/llvm/lib/Analysis/BranchProbabilityInfo.cpp b/llvm/lib/Analysis/BranchProbabilityInfo.cpp
index 7931001d0a2b2..d01c652333ff6 100644
--- a/llvm/lib/Analysis/BranchProbabilityInfo.cpp
+++ b/llvm/lib/Analysis/BranchProbabilityInfo.cpp
@@ -1163,7 +1163,7 @@ void BranchProbabilityInfo::copyEdgeProbabilities(BasicBlock *Src,
   assert(NumSuccessors == Dst->getTerminator()->getNumSuccessors());
   if (NumSuccessors == 0)
     return; // Nothing to set.
-  if (this->Probs.find(std::make_pair(Src, 0)) == this->Probs.end())
+  if (!this->Probs.contains(std::make_pair(Src, 0)))
     return; // No probability is set for edges from Src. Keep the same for Dst.
 
   Handles.insert(BasicBlockCallbackVH(Dst, this));
@@ -1303,11 +1303,12 @@ void BranchProbabilityInfoWrapperPass::print(raw_ostream &OS,
 AnalysisKey BranchProbabilityAnalysis::Key;
 BranchProbabilityInfo
 BranchProbabilityAnalysis::run(Function &F, FunctionAnalysisManager &AM) {
+  auto &LI = AM.getResult<LoopAnalysis>(F);
+  auto &TLI = AM.getResult<TargetLibraryAnalysis>(F);
+  auto &DT = AM.getResult<DominatorTreeAnalysis>(F);
+  auto &PDT = AM.getResult<PostDominatorTreeAnalysis>(F);
   BranchProbabilityInfo BPI;
-  BPI.calculate(F, AM.getResult<LoopAnalysis>(F),
-                &AM.getResult<TargetLibraryAnalysis>(F),
-                &AM.getResult<DominatorTreeAnalysis>(F),
-                &AM.getResult<PostDominatorTreeAnalysis>(F));
+  BPI.calculate(F, LI, &TLI, &DT, &PDT);
   return BPI;
 }
 
diff --git a/llvm/lib/Analysis/CFGPrinter.cpp b/llvm/lib/Analysis/CFGPrinter.cpp
index f8eba1a00f28d..f05dd6852d6dc 100644
--- a/llvm/lib/Analysis/CFGPrinter.cpp
+++ b/llvm/lib/Analysis/CFGPrinter.cpp
@@ -325,8 +325,7 @@ bool DOTGraphTraits<DOTFuncInfo *>::isNodeHidden(const BasicBlock *Node,
         return true;
     }
   if (HideUnreachablePaths || HideDeoptimizePaths) {
-    if (isOnDeoptOrUnreachablePath.find(Node) == 
-        isOnDeoptOrUnreachablePath.end())
+    if (!isOnDeoptOrUnreachablePath.contains(Node))
       computeDeoptOrUnreachablePaths(Node->getParent());
     return isOnDeoptOrUnreachablePath[Node];
   }
diff --git a/llvm/lib/Analysis/CGSCCPassManager.cpp b/llvm/lib/Analysis/CGSCCPassManager.cpp
index 2de19884014c3..facb9c897da3f 100644
--- a/llvm/lib/Analysis/CGSCCPassManager.cpp
+++ b/llvm/lib/Analysis/CGSCCPassManager.cpp
@@ -86,11 +86,6 @@ PassManager<LazyCallGraph::SCC, CGSCCAnalysisManager, LazyCallGraph &,
 
     PreservedAnalyses PassPA = Pass->run(*C, AM, G, UR);
 
-    if (UR.InvalidatedSCCs.count(C))
-      PI.runAfterPassInvalidated<LazyCallGraph::SCC>(*Pass, PassPA);
-    else
-      PI.runAfterPass<LazyCallGraph::SCC>(*Pass, *C, PassPA);
-
     // Update the SCC if necessary.
     C = UR.UpdatedC ? UR.UpdatedC : C;
     if (UR.UpdatedC) {
@@ -107,6 +102,7 @@ PassManager<LazyCallGraph::SCC, CGSCCAnalysisManager, LazyCallGraph &,
     // If the CGSCC pass wasn't able to provide a valid updated SCC, the
     // current SCC may simply need to be skipped if invalid.
     if (UR.InvalidatedSCCs.count(C)) {
+      PI.runAfterPassInvalidated<LazyCallGraph::SCC>(*Pass, PassPA);
       LLVM_DEBUG(dbgs() << "Skipping invalidated root or island SCC!\n");
       break;
     }
@@ -117,6 +113,8 @@ PassManager<LazyCallGraph::SCC, CGSCCAnalysisManager, LazyCallGraph &,
     // Update the analysis manager as each pass runs and potentially
     // invalidates analyses.
     AM.invalidate(*C, PassPA);
+
+    PI.runAfterPass<LazyCallGraph::SCC>(*Pass, *C, PassPA);
   }
 
   // Before we mark all of *this* SCC's analyses as preserved below, intersect
@@ -276,11 +274,6 @@ ModuleToPostOrderCGSCCPassAdaptor::run(Module &M, ModuleAnalysisManager &AM) {
 
           PreservedAnalyses PassPA = Pass->run(*C, CGAM, CG, UR);
 
-          if (UR.InvalidatedSCCs.count(C))
-            PI.runAfterPassInvalidated<LazyCallGraph::SCC>(*Pass, PassPA);
-          else
-            PI.runAfterPass<LazyCallGraph::SCC>(*Pass, *C, PassPA);
-
           // Update the SCC and RefSCC if necessary.
           C = UR.UpdatedC ? UR.UpdatedC : C;
 
@@ -301,6 +294,7 @@ ModuleToPostOrderCGSCCPassAdaptor::run(Module &M, ModuleAnalysisManager &AM) {
           // If the CGSCC pass wasn't able to provide a valid updated SCC,
           // the current SCC may simply need to be skipped if invalid.
           if (UR.InvalidatedSCCs.count(C)) {
+            PI.runAfterPassInvalidated<LazyCallGraph::SCC>(*Pass, PassPA);
             LLVM_DEBUG(dbgs() << "Skipping invalidated root or island SCC!\n");
             break;
           }
@@ -316,6 +310,8 @@ ModuleToPostOrderCGSCCPassAdaptor::run(Module &M, ModuleAnalysisManager &AM) {
           // processed.
           CGAM.invalidate(*C, PassPA);
 
+          PI.runAfterPass<LazyCallGraph::SCC>(*Pass, *C, PassPA);
+
           // The pass may have restructured the call graph and refined the
           // current SCC and/or RefSCC. We need to update our current SCC and
           // RefSCC pointers to follow these. Also, when the current SCC is
@@ -408,25 +404,27 @@ PreservedAnalyses DevirtSCCRepeatedPass::run(LazyCallGraph::SCC &InitialC,
 
     PreservedAnalyses PassPA = Pass->run(*C, AM, CG, UR);
 
-    if (UR.InvalidatedSCCs.count(C))
-      PI.runAfterPassInvalidated<LazyCallGraph::SCC>(*Pass, PassPA);
-    else
-      PI.runAfterPass<LazyCallGraph::SCC>(*Pass, *C, PassPA);
-
     PA.intersect(PassPA);
 
-    // If the SCC structure has changed, bail immediately and let the outer
-    // CGSCC layer handle any iteration to reflect the refined structure.
-    if (UR.UpdatedC && UR.UpdatedC != C)
-      break;
-
     // If the CGSCC pass wasn't able to provide a valid updated SCC, the
     // current SCC may simply need to be skipped if invalid.
     if (UR.InvalidatedSCCs.count(C)) {
+      PI.runAfterPassInvalidated<LazyCallGraph::SCC>(*Pass, PassPA);
       LLVM_DEBUG(dbgs() << "Skipping invalidated root or island SCC!\n");
       break;
     }
 
+    // Update the analysis manager with each run and intersect the total set
+    // of preserved analyses so we're ready to iterate.
+    AM.invalidate(*C, PassPA);
+
+    PI.runAfterPass<LazyCallGraph::SCC>(*Pass, *C, PassPA);
+
+    // If the SCC structure has changed, bail immediately and let the outer
+    // CGSCC layer handle any iteration to reflect the refined structure.
+    if (UR.UpdatedC && UR.UpdatedC != C)
+      break;
+
     assert(C->begin() != C->end() && "Cannot have an empty SCC!");
 
     // Check whether any of the handles were devirtualized.
@@ -490,10 +488,6 @@ PreservedAnalyses DevirtSCCRepeatedPass::run(LazyCallGraph::SCC &InitialC,
 
     // Move over the new call counts in preparation for iterating.
     CallCounts = std::move(NewCallCounts);
-
-    // Update the analysis manager with each run and intersect the total set
-    // of preserved analyses so we're ready to iterate.
-    AM.invalidate(*C, PassPA);
   }
 
   // Note that we don't add any preserved entries here unlike a more normal
@@ -539,14 +533,13 @@ PreservedAnalyses CGSCCToFunctionPassAdaptor::run(LazyCallGraph::SCC &C,
       continue;
 
     PreservedAnalyses PassPA = Pass->run(F, FAM);
-    PI.runAfterPass<Function>(*Pass, F, PassPA);
 
     // We know that the function pass couldn't have invalidated any other
     // function's analyses (that's the contract of a function pass), so
     // directly handle the function analysis manager's invalidation here.
     FAM.invalidate(F, EagerlyInvalidate ? PreservedAnalyses::none() : PassPA);
-    if (NoRerun)
-      (void)FAM.getResult<ShouldNotRunFunctionPassesAnalysis>(F);
+
+    PI.runAfterPass<Function>(*Pass, F, PassPA);
 
     // Then intersect the preserved set so that invalidation of module
     // analyses will eventually occur when the module pass completes.
diff --git a/llvm/lib/Analysis/CMakeLists.txt b/llvm/lib/Analysis/CMakeLists.txt
index 531787fcb13d7..d25eb5c702a7e 100644
--- a/llvm/lib/Analysis/CMakeLists.txt
+++ b/llvm/lib/Analysis/CMakeLists.txt
@@ -87,7 +87,6 @@ add_llvm_component_library(LLVMAnalysis
   Lint.cpp
   Loads.cpp
   Local.cpp
-  LogicCombine.cpp
   LoopAccessAnalysis.cpp
   LoopAnalysisManager.cpp
   LoopCacheAnalysis.cpp
diff --git a/llvm/lib/Analysis/CaptureTracking.cpp b/llvm/lib/Analysis/CaptureTracking.cpp
index 34a320435c0a7..e2931973112b4 100644
--- a/llvm/lib/Analysis/CaptureTracking.cpp
+++ b/llvm/lib/Analysis/CaptureTracking.cpp
@@ -80,7 +80,10 @@ namespace {
         const SmallPtrSetImpl<const Value *> &EphValues, bool ReturnCaptures)
         : EphValues(EphValues), ReturnCaptures(ReturnCaptures) {}
 
-    void tooManyUses() override { Captured = true; }
+    void tooManyUses() override {
+      LLVM_DEBUG(dbgs() << "Captured due to too many uses\n");
+      Captured = true;
+    }
 
     bool captured(const Use *U) override {
       if (isa<ReturnInst>(U->getUser()) && !ReturnCaptures)
@@ -89,6 +92,8 @@ namespace {
       if (EphValues.contains(U->getUser()))
         return false;
 
+      LLVM_DEBUG(dbgs() << "Captured by: " << *U->getUser() << "\n");
+
       Captured = true;
       return true;
     }
@@ -233,12 +238,16 @@ bool llvm::PointerMayBeCaptured(const Value *V, bool ReturnCaptures,
   // take advantage of this.
   (void)StoreCaptures;
 
+  LLVM_DEBUG(dbgs() << "Captured?: " << *V << " = ");
+
   SimpleCaptureTracker SCT(EphValues, ReturnCaptures);
   PointerMayBeCaptured(V, &SCT, MaxUsesToExplore);
   if (SCT.Captured)
     ++NumCaptured;
-  else
+  else {
     ++NumNotCaptured;
+    LLVM_DEBUG(dbgs() << "not captured\n");
+  }
   return SCT.Captured;
 }
 
diff --git a/llvm/lib/Analysis/ConstantFolding.cpp b/llvm/lib/Analysis/ConstantFolding.cpp
index d6ed643765653..65ea0dad90d5a 100644
--- a/llvm/lib/Analysis/ConstantFolding.cpp
+++ b/llvm/lib/Analysis/ConstantFolding.cpp
@@ -822,7 +822,8 @@ Constant *SymbolicallyEvaluateBinop(unsigned Opc, Constant *Op0, Constant *Op1,
 /// If array indices are not pointer-sized integers, explicitly cast them so
 /// that they aren't implicitly casted by the getelementptr.
 Constant *CastGEPIndices(Type *SrcElemTy, ArrayRef<Constant *> Ops,
-                         Type *ResultTy, std::optional<unsigned> InRangeIndex,
+                         Type *ResultTy, bool InBounds,
+                         std::optional<unsigned> InRangeIndex,
                          const DataLayout &DL, const TargetLibraryInfo *TLI) {
   Type *IntIdxTy = DL.getIndexType(ResultTy);
   Type *IntIdxScalarTy = IntIdxTy->getScalarType();
@@ -851,7 +852,7 @@ Constant *CastGEPIndices(Type *SrcElemTy, ArrayRef<Constant *> Ops,
     return nullptr;
 
   Constant *C = ConstantExpr::getGetElementPtr(
-      SrcElemTy, Ops[0], NewIdxs, /*InBounds=*/false, InRangeIndex);
+      SrcElemTy, Ops[0], NewIdxs, InBounds, InRangeIndex);
   return ConstantFoldConstant(C, DL, TLI);
 }
 
@@ -886,7 +887,8 @@ Constant *SymbolicallyEvaluateGEP(const GEPOperator *GEP,
     return nullptr;
 
   if (Constant *C = CastGEPIndices(SrcElemTy, Ops, ResTy,
-                                   GEP->getInRangeIndex(), DL, TLI))
+                                   GEP->isInBounds(), GEP->getInRangeIndex(),
+                                   DL, TLI))
     return C;
 
   Constant *Ptr = Ops[0];
@@ -1050,11 +1052,15 @@ Constant *ConstantFoldInstOperandsImpl(const Value *InstOrCE, unsigned Opcode,
     return ConstantFoldCastOperand(Opcode, Ops[0], DestTy, DL);
 
   if (auto *GEP = dyn_cast<GEPOperator>(InstOrCE)) {
+    Type *SrcElemTy = GEP->getSourceElementType();
+    if (!ConstantExpr::isSupportedGetElementPtr(SrcElemTy))
+      return nullptr;
+
     if (Constant *C = SymbolicallyEvaluateGEP(GEP, Ops, DL, TLI))
       return C;
 
-    return ConstantExpr::getGetElementPtr(GEP->getSourceElementType(), Ops[0],
-                                          Ops.slice(1), GEP->isInBounds(),
+    return ConstantExpr::getGetElementPtr(SrcElemTy, Ops[0], Ops.slice(1),
+                                          GEP->isInBounds(),
                                           GEP->getInRangeIndex());
   }
 
@@ -2630,18 +2636,18 @@ static Constant *ConstantFoldScalarCall2(StringRef Name,
     } else if (auto *Op2C = dyn_cast<ConstantInt>(Operands[1])) {
       switch (IntrinsicID) {
       case Intrinsic::is_fpclass: {
-        uint32_t Mask = Op2C->getZExtValue();
+        FPClassTest Mask = static_cast<FPClassTest>(Op2C->getZExtValue());
         bool Result =
           ((Mask & fcSNan) && Op1V.isNaN() && Op1V.isSignaling()) ||
           ((Mask & fcQNan) && Op1V.isNaN() && !Op1V.isSignaling()) ||
-          ((Mask & fcNegInf) && Op1V.isInfinity() && Op1V.isNegative()) ||
+          ((Mask & fcNegInf) && Op1V.isNegInfinity()) ||
           ((Mask & fcNegNormal) && Op1V.isNormal() && Op1V.isNegative()) ||
           ((Mask & fcNegSubnormal) && Op1V.isDenormal() && Op1V.isNegative()) ||
           ((Mask & fcNegZero) && Op1V.isZero() && Op1V.isNegative()) ||
           ((Mask & fcPosZero) && Op1V.isZero() && !Op1V.isNegative()) ||
           ((Mask & fcPosSubnormal) && Op1V.isDenormal() && !Op1V.isNegative()) ||
           ((Mask & fcPosNormal) && Op1V.isNormal() && !Op1V.isNegative()) ||
-          ((Mask & fcPosInf) && Op1V.isInfinity() && !Op1V.isNegative());
+          ((Mask & fcPosInf) && Op1V.isPosInfinity());
         return ConstantInt::get(Ty, Result);
       }
       default:
diff --git a/llvm/lib/Analysis/DDG.cpp b/llvm/lib/Analysis/DDG.cpp
index da64ef1539602..a0774096c5129 100644
--- a/llvm/lib/Analysis/DDG.cpp
+++ b/llvm/lib/Analysis/DDG.cpp
@@ -241,11 +241,10 @@ bool DataDependenceGraph::addNode(DDGNode &N) {
 }
 
 const PiBlockDDGNode *DataDependenceGraph::getPiBlock(const NodeType &N) const {
-  if (PiBlockMap.find(&N) == PiBlockMap.end())
+  if (!PiBlockMap.contains(&N))
     return nullptr;
   auto *Pi = PiBlockMap.find(&N)->second;
-  assert(PiBlockMap.find(Pi) == PiBlockMap.end() &&
-         "Nested pi-blocks detected.");
+  assert(!PiBlockMap.contains(Pi) && "Nested pi-blocks detected.");
   return Pi;
 }
 
diff --git a/llvm/lib/Analysis/DemandedBits.cpp b/llvm/lib/Analysis/DemandedBits.cpp
index e5d693a220d1e..35c5217fb8a5e 100644
--- a/llvm/lib/Analysis/DemandedBits.cpp
+++ b/llvm/lib/Analysis/DemandedBits.cpp
@@ -475,8 +475,7 @@ APInt DemandedBits::getDemandedBits(Use *U) {
 bool DemandedBits::isInstructionDead(Instruction *I) {
   performAnalysis();
 
-  return !Visited.count(I) && AliveBits.find(I) == AliveBits.end() &&
-    !isAlwaysLive(I);
+  return !Visited.count(I) && !AliveBits.contains(I) && !isAlwaysLive(I);
 }
 
 bool DemandedBits::isUseDead(Use *U) {
diff --git a/llvm/lib/Analysis/IRSimilarityIdentifier.cpp b/llvm/lib/Analysis/IRSimilarityIdentifier.cpp
index f471e32344cbc..930985a955456 100644
--- a/llvm/lib/Analysis/IRSimilarityIdentifier.cpp
+++ b/llvm/lib/Analysis/IRSimilarityIdentifier.cpp
@@ -14,6 +14,7 @@
 
 #include "llvm/Analysis/IRSimilarityIdentifier.h"
 #include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/SetOperations.h"
 #include "llvm/IR/Intrinsics.h"
 #include "llvm/IR/Operator.h"
 #include "llvm/IR/User.h"
@@ -97,7 +98,8 @@ void IRInstructionData::setBranchSuccessors(
 
   int CurrentBlockNumber = static_cast<int>(BBNumIt->second);
 
-  for (BasicBlock *Successor : BI->successors()) {
+  for (Value *V : getBlockOperVals()) {
+    BasicBlock *Successor = cast<BasicBlock>(V);
     BBNumIt = BasicBlockToInteger.find(Successor);
     assert(BBNumIt != BasicBlockToInteger.end() &&
            "Could not find number for BasicBlock!");
@@ -108,6 +110,25 @@ void IRInstructionData::setBranchSuccessors(
   }
 }
 
+ArrayRef<Value *> IRInstructionData::getBlockOperVals() {
+  assert((isa<BranchInst>(Inst) ||
+         isa<PHINode>(Inst)) && "Instruction must be branch or PHINode");
+  
+  if (BranchInst *BI = dyn_cast<BranchInst>(Inst))
+    return ArrayRef<Value *>(
+      std::next(OperVals.begin(), BI->isConditional() ? 1 : 0),
+      OperVals.end()
+    );
+
+  if (PHINode *PN = dyn_cast<PHINode>(Inst))
+    return ArrayRef<Value *>(
+      std::next(OperVals.begin(), PN->getNumIncomingValues()),
+      OperVals.end()
+    );
+
+  return ArrayRef<Value *>();
+}
+
 void IRInstructionData::setCalleeName(bool MatchByName) {
   CallInst *CI = dyn_cast<CallInst>(Inst);
   assert(CI && "Instruction must be call");
@@ -159,7 +180,6 @@ void IRInstructionData::setPHIPredecessors(
 
     int Relative = OtherBlockNumber - CurrentBlockNumber;
     RelativeBlockLocations.push_back(Relative);
-    RelativeBlockLocations.push_back(Relative);
   }
 }
 
@@ -439,7 +459,7 @@ IRSimilarityCandidate::IRSimilarityCandidate(unsigned StartIdx, unsigned Len,
     // Map the operand values to an unsigned integer if it does not already
     // have an unsigned integer assigned to it.
     for (Value *Arg : ID->OperVals)
-      if (ValueToNumber.find(Arg) == ValueToNumber.end()) {
+      if (!ValueToNumber.contains(Arg)) {
         ValueToNumber.try_emplace(Arg, LocalValNumber);
         NumberToValue.try_emplace(LocalValNumber, Arg);
         LocalValNumber++;
@@ -447,7 +467,7 @@ IRSimilarityCandidate::IRSimilarityCandidate(unsigned StartIdx, unsigned Len,
 
     // Mapping the instructions to an unsigned integer if it is not already
     // exist in the mapping.
-    if (ValueToNumber.find(ID->Inst) == ValueToNumber.end()) {
+    if (!ValueToNumber.contains(ID->Inst)) {
       ValueToNumber.try_emplace(ID->Inst, LocalValNumber);
       NumberToValue.try_emplace(LocalValNumber, ID->Inst);
       LocalValNumber++;
@@ -464,7 +484,7 @@ IRSimilarityCandidate::IRSimilarityCandidate(unsigned StartIdx, unsigned Len,
   DenseSet<BasicBlock *> BBSet;
   getBasicBlocks(BBSet);
   for (BasicBlock *BB : BBSet) {
-    if (ValueToNumber.find(BB) != ValueToNumber.end())
+    if (ValueToNumber.contains(BB))
       continue;
     
     ValueToNumber.try_emplace(BB, LocalValNumber);
@@ -701,8 +721,8 @@ bool IRSimilarityCandidate::compareCommutativeOperandMapping(
 bool IRSimilarityCandidate::checkRelativeLocations(RelativeLocMapping A,
                                                    RelativeLocMapping B) {
   // Get the basic blocks the label refers to.
-  BasicBlock *ABB = static_cast<BasicBlock *>(A.OperVal);
-  BasicBlock *BBB = static_cast<BasicBlock *>(B.OperVal);
+  BasicBlock *ABB = cast<BasicBlock>(A.OperVal);
+  BasicBlock *BBB = cast<BasicBlock>(B.OperVal);
 
   // Get the basic blocks contained in each region.
   DenseSet<BasicBlock *> BasicBlockA;
@@ -715,7 +735,7 @@ bool IRSimilarityCandidate::checkRelativeLocations(RelativeLocMapping A,
   bool BContained = BasicBlockB.contains(BBB);
 
   // Both blocks need to be contained in the region, or both need to be outside
-  // the reigon.
+  // the region.
   if (AContained != BContained)
     return false;
   
@@ -826,12 +846,22 @@ bool IRSimilarityCandidate::compareStructure(
 
     SmallVector<int, 4> &RelBlockLocsA = ItA->RelativeBlockLocations;
     SmallVector<int, 4> &RelBlockLocsB = ItB->RelativeBlockLocations;
+    ArrayRef<Value *> ABL = ItA->getBlockOperVals();
+    ArrayRef<Value *> BBL = ItB->getBlockOperVals();
+
+    // Check to make sure that the number of operands, and branching locations
+    // between BranchInsts is the same.
     if (RelBlockLocsA.size() != RelBlockLocsB.size() &&
-        OperValsA.size() != OperValsB.size())
+        ABL.size() != BBL.size())
       return false;
 
+    assert(RelBlockLocsA.size() == ABL.size() &&
+           "Block information vectors not the same size.");
+    assert(RelBlockLocsB.size() == BBL.size() &&
+           "Block information vectors not the same size.");
+
     ZippedRelativeLocationsT ZippedRelativeLocations =
-        zip(RelBlockLocsA, RelBlockLocsB, OperValsA, OperValsB);
+        zip(RelBlockLocsA, RelBlockLocsB, ABL, BBL);
     if (any_of(ZippedRelativeLocations,
                [&A, &B](std::tuple<int, int, Value *, Value *> R) {
                  return !checkRelativeLocations(
@@ -1026,7 +1056,7 @@ void IRSimilarityCandidate::createCanonicalRelationFrom(
 
     // We can skip the BasicBlock if the canonical numbering has already been
     // found in a separate instruction.
-    if (NumberToCanonNum.find(BBGVNForCurrCand) != NumberToCanonNum.end())
+    if (NumberToCanonNum.contains(BBGVNForCurrCand))
       continue;
 
     // If the basic block is the starting block, then the shared instruction may
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index 38b1a1d0b0bbd..156d141c461ed 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1107,7 +1107,7 @@ Value *RecurrenceDescriptor::getRecurrenceIdentity(RecurKind K, Type *Tp,
       return ConstantFP::get(Tp, 0.0L);
     return ConstantFP::get(Tp, -0.0L);
   case RecurKind::UMin:
-    return ConstantInt::get(Tp, -1);
+    return ConstantInt::get(Tp, -1, true);
   case RecurKind::UMax:
     return ConstantInt::get(Tp, 0);
   case RecurKind::SMin:
diff --git a/llvm/lib/Analysis/InlineCost.cpp b/llvm/lib/Analysis/InlineCost.cpp
index f303670c72868..6877e44813f56 100644
--- a/llvm/lib/Analysis/InlineCost.cpp
+++ b/llvm/lib/Analysis/InlineCost.cpp
@@ -142,11 +142,11 @@ static cl::opt<size_t>
                        cl::desc("Do not inline functions with a stack size "
                                 "that exceeds the specified limit"));
 
-static cl::opt<size_t>
-    RecurStackSizeThreshold("recursive-inline-max-stacksize", cl::Hidden,
-                       cl::init(InlineConstants::TotalAllocaSizeRecursiveCaller),
-                       cl::desc("Do not inline recursive functions with a stack "
-                                "size that exceeds the specified limit"));
+static cl::opt<size_t> RecurStackSizeThreshold(
+    "recursive-inline-max-stacksize", cl::Hidden,
+    cl::init(InlineConstants::TotalAllocaSizeRecursiveCaller),
+    cl::desc("Do not inline recursive functions with a stack "
+             "size that exceeds the specified limit"));
 
 static cl::opt<bool> OptComputeFullInlineCost(
     "inline-cost-full", cl::Hidden,
@@ -493,7 +493,7 @@ class CallAnalyzer : public InstVisitor<CallAnalyzer, bool> {
   InlineResult analyze();
 
   std::optional<Constant *> getSimplifiedValue(Instruction *I) {
-    if (SimplifiedValues.find(I) != SimplifiedValues.end())
+    if (SimplifiedValues.contains(I))
       return SimplifiedValues[I];
     return std::nullopt;
   }
@@ -1054,7 +1054,7 @@ class InlineCostCallAnalyzer final : public CallAnalyzer {
   void print(raw_ostream &OS);
 
   std::optional<InstructionCostDetail> getCostDetails(const Instruction *I) {
-    if (InstructionCostDetailMap.find(I) != InstructionCostDetailMap.end())
+    if (InstructionCostDetailMap.contains(I))
       return InstructionCostDetailMap[I];
     return std::nullopt;
   }
@@ -1607,7 +1607,7 @@ bool CallAnalyzer::simplifyIntrinsicCallIsConstant(CallBase &CB) {
 bool CallAnalyzer::simplifyIntrinsicCallObjectSize(CallBase &CB) {
   // As per the langref, "The fourth argument to llvm.objectsize determines if
   // the value should be evaluated at runtime."
-  if(cast<ConstantInt>(CB.getArgOperand(3))->isOne())
+  if (cast<ConstantInt>(CB.getArgOperand(3))->isOne())
     return false;
 
   Value *V = lowerObjectSizeCall(&cast<IntrinsicInst>(CB), DL, nullptr,
diff --git a/llvm/lib/Analysis/InlineOrder.cpp b/llvm/lib/Analysis/InlineOrder.cpp
index 8d0e49936901d..3b85820d7b8f1 100644
--- a/llvm/lib/Analysis/InlineOrder.cpp
+++ b/llvm/lib/Analysis/InlineOrder.cpp
@@ -33,8 +33,7 @@ static cl::opt<InlinePriorityMode> UseInlinePriority(
                           "Use inline cost priority."),
                clEnumValN(InlinePriorityMode::CostBenefit, "cost-benefit",
                           "Use cost-benefit ratio."),
-               clEnumValN(InlinePriorityMode::ML, "ml",
-                          "Use ML.")));
+               clEnumValN(InlinePriorityMode::ML, "ml", "Use ML.")));
 
 static cl::opt<int> ModuleInlinerTopPriorityThreshold(
     "moudle-inliner-top-priority-threshold", cl::Hidden, cl::init(0),
@@ -281,8 +280,13 @@ class PriorityInlineOrder : public InlineOrder<std::pair<CallBase *, int>> {
 
 } // namespace
 
+AnalysisKey llvm::PluginInlineOrderAnalysis::Key;
+bool llvm::PluginInlineOrderAnalysis::HasBeenRegistered;
+
 std::unique_ptr<InlineOrder<std::pair<CallBase *, int>>>
-llvm::getInlineOrder(FunctionAnalysisManager &FAM, const InlineParams &Params) {
+llvm::getDefaultInlineOrder(FunctionAnalysisManager &FAM,
+                            const InlineParams &Params,
+                            ModuleAnalysisManager &MAM, Module &M) {
   switch (UseInlinePriority) {
   case InlinePriorityMode::Size:
     LLVM_DEBUG(dbgs() << "    Current used priority: Size priority ---- \n");
@@ -295,11 +299,22 @@ llvm::getInlineOrder(FunctionAnalysisManager &FAM, const InlineParams &Params) {
   case InlinePriorityMode::CostBenefit:
     LLVM_DEBUG(
         dbgs() << "    Current used priority: cost-benefit priority ---- \n");
-    return std::make_unique<PriorityInlineOrder<CostBenefitPriority>>(FAM, Params);
+    return std::make_unique<PriorityInlineOrder<CostBenefitPriority>>(FAM,
+                                                                      Params);
   case InlinePriorityMode::ML:
-    LLVM_DEBUG(
-        dbgs() << "    Current used priority: ML priority ---- \n");
+    LLVM_DEBUG(dbgs() << "    Current used priority: ML priority ---- \n");
     return std::make_unique<PriorityInlineOrder<MLPriority>>(FAM, Params);
   }
   return nullptr;
 }
+
+std::unique_ptr<InlineOrder<std::pair<CallBase *, int>>>
+llvm::getInlineOrder(FunctionAnalysisManager &FAM, const InlineParams &Params,
+                     ModuleAnalysisManager &MAM, Module &M) {
+  if (llvm::PluginInlineOrderAnalysis::isRegistered()) {
+    LLVM_DEBUG(dbgs() << "    Current used priority: plugin ---- \n");
+    return MAM.getResult<PluginInlineOrderAnalysis>(M).Factory(FAM, Params, MAM,
+                                                               M);
+  }
+  return getDefaultInlineOrder(FAM, Params, MAM, M);
+}
\ No newline at end of file
diff --git a/llvm/lib/Analysis/InstructionPrecedenceTracking.cpp b/llvm/lib/Analysis/InstructionPrecedenceTracking.cpp
index 78e7f456ebc69..fba5859b74cef 100644
--- a/llvm/lib/Analysis/InstructionPrecedenceTracking.cpp
+++ b/llvm/lib/Analysis/InstructionPrecedenceTracking.cpp
@@ -47,9 +47,9 @@ const Instruction *InstructionPrecedenceTracking::getFirstSpecialInstruction(
     validate(BB);
 #endif
 
-  if (FirstSpecialInsts.find(BB) == FirstSpecialInsts.end()) {
+  if (!FirstSpecialInsts.contains(BB)) {
     fill(BB);
-    assert(FirstSpecialInsts.find(BB) != FirstSpecialInsts.end() && "Must be!");
+    assert(FirstSpecialInsts.contains(BB) && "Must be!");
   }
   return FirstSpecialInsts[BB];
 }
diff --git a/llvm/lib/Analysis/InstructionSimplify.cpp b/llvm/lib/Analysis/InstructionSimplify.cpp
index d6f3585fcc582..ebc5a77efd1f7 100644
--- a/llvm/lib/Analysis/InstructionSimplify.cpp
+++ b/llvm/lib/Analysis/InstructionSimplify.cpp
@@ -539,12 +539,16 @@ static Value *threadBinOpOverPHI(Instruction::BinaryOps Opcode, Value *LHS,
 
   // Evaluate the BinOp on the incoming phi values.
   Value *CommonValue = nullptr;
-  for (Value *Incoming : PI->incoming_values()) {
+  for (Use &Incoming : PI->incoming_values()) {
     // If the incoming value is the phi node itself, it can safely be skipped.
     if (Incoming == PI)
       continue;
-    Value *V = PI == LHS ? simplifyBinOp(Opcode, Incoming, RHS, Q, MaxRecurse)
-                         : simplifyBinOp(Opcode, LHS, Incoming, Q, MaxRecurse);
+    Instruction *InTI = PI->getIncomingBlock(Incoming)->getTerminator();
+    Value *V = PI == LHS
+                   ? simplifyBinOp(Opcode, Incoming, RHS,
+                                   Q.getWithInstruction(InTI), MaxRecurse)
+                   : simplifyBinOp(Opcode, LHS, Incoming,
+                                   Q.getWithInstruction(InTI), MaxRecurse);
     // If the operation failed to simplify, or simplified to a different value
     // to previously, then give up.
     if (!V || (CommonValue && V != CommonValue))
@@ -1445,10 +1449,11 @@ static Value *simplifyShlInst(Value *Op0, Value *Op1, bool IsNSW, bool IsNUW,
           simplifyShift(Instruction::Shl, Op0, Op1, IsNSW, Q, MaxRecurse))
     return V;
 
+  Type *Ty = Op0->getType();
   // undef << X -> 0
   // undef << X -> undef if (if it's NSW/NUW)
   if (Q.isUndefValue(Op0))
-    return IsNSW || IsNUW ? Op0 : Constant::getNullValue(Op0->getType());
+    return IsNSW || IsNUW ? Op0 : Constant::getNullValue(Ty);
 
   // (X >> A) << A -> X
   Value *X;
@@ -1462,6 +1467,13 @@ static Value *simplifyShlInst(Value *Op0, Value *Op1, bool IsNSW, bool IsNUW,
   // NOTE: could use computeKnownBits() / LazyValueInfo,
   // but the cost-benefit analysis suggests it isn't worth it.
 
+  // "nuw" guarantees that only zeros are shifted out, and "nsw" guarantees
+  // that the sign-bit does not change, so the only input that does not
+  // produce poison is 0, and "0 << (bitwidth-1) --> 0".
+  if (IsNSW && IsNUW &&
+      match(Op1, m_SpecificInt(Ty->getScalarSizeInBits() - 1)))
+    return Constant::getNullValue(Ty);
+
   return nullptr;
 }
 
@@ -4920,6 +4932,10 @@ static Value *simplifyGEPInst(Type *SrcTy, Value *Ptr,
       !all_of(Indices, [](Value *V) { return isa<Constant>(V); }))
     return nullptr;
 
+  if (!ConstantExpr::isSupportedGetElementPtr(SrcTy))
+    return ConstantFoldGetElementPtr(SrcTy, cast<Constant>(Ptr), InBounds,
+                                     std::nullopt, Indices);
+
   auto *CE = ConstantExpr::getGetElementPtr(SrcTy, cast<Constant>(Ptr), Indices,
                                             InBounds);
   return ConstantFoldConstant(CE, Q.DL);
diff --git a/llvm/lib/Analysis/LogicCombine.cpp b/llvm/lib/Analysis/LogicCombine.cpp
deleted file mode 100644
index ffe4c341cc2b4..0000000000000
--- a/llvm/lib/Analysis/LogicCombine.cpp
+++ /dev/null
@@ -1,207 +0,0 @@
-//===--------------------- LogicCombine.cpp -------------------------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-/// \file
-/// This file attempts to find the simplest expression for a bitwise logic
-/// operation chain. We canonicalize all other ops to "&"/"^".
-/// For example:
-///    a | b --> (a & b) ^ a ^ b
-///    c ? a : b --> (c & a) ^ ((c ^ true) & b)
-/// We use a set of bitset to represent the expression. Any value that is not a
-/// logic operation is a leaf node. Leaf node is 1 bit in the bitset. For
-/// example, we have source a, b, c. The bit for a is 1, b is 2, c is 4.
-///     a & b & c --> {0b111}
-///     a & b ^ c & a --> {0b011, 0b101}
-///     a & b ^ c & a ^ b --> {0b011, 0b101, 0b010}
-/// Every bitset is an "&" chain. The set of bitset is a "^" chain.
-/// Based on boolean ring, we can treat "&" as ring multiplication and "^" as
-/// ring addition. After that, any logic value can be represented as a chain of
-/// bitsets. For example:
-///     r1 = (a | b) & c -> r1 = (a * b * c) + (a * c) + (b * c) ->
-///     {0b111, 0b101, 0b110}
-/// Finally we need to rebuild the simplest pattern from the expression.
-///
-/// Reference: https://en.wikipedia.org/wiki/Boolean_ring
-///
-//===----------------------------------------------------------------------===//
-
-#include "llvm/Analysis/LogicCombine.h"
-#include "llvm/ADT/Statistic.h"
-#include "llvm/ADT/StringExtras.h"
-#include "llvm/IR/Constants.h"
-#include "llvm/Support/CommandLine.h"
-#include "llvm/Support/Debug.h"
-
-using namespace llvm;
-
-#define DEBUG_TYPE "logic-combine"
-
-STATISTIC(NumLogicalOpsSimplified, "Number of logical operations simplified");
-
-static cl::opt<unsigned> MaxLogicOpLeafsToScan(
-    "logic-combine-max-leafs", cl::init(8), cl::Hidden,
-    cl::desc("Max leafs of logic ops to scan for logical combine."));
-
-static cl::opt<unsigned> MaxDepthLogicOpsToScan(
-    "logic-combine-max-depth", cl::init(8), cl::Hidden,
-    cl::desc("Max depth of logic ops to scan for logical combine."));
-
-void LogicalOpNode::printAndChain(raw_ostream &OS, uint64_t LeafBits) const {
-  if (LeafBits == LogicalExpr::ExprAllOne) {
-    OS << "-1";
-    return;
-  }
-
-  if (LeafBits == 0)
-    return;
-
-  unsigned LeafCnt = popcount(LeafBits);
-  if (LeafCnt == 1) {
-    Helper->LeafValues[Log2_64(LeafBits)]->printAsOperand(OS, false);
-    return;
-  }
-
-  unsigned LeafIdx;
-  ListSeparator LS(" * ");
-  for (unsigned I = 0; I < LeafCnt; I++) {
-    LeafIdx = countr_zero(LeafBits);
-    OS << LS;
-    Helper->LeafValues[LeafIdx]->printAsOperand(OS, false);
-    LeafBits -= (1ULL << LeafIdx);
-  }
-}
-
-void LogicalOpNode::print(raw_ostream &OS) const {
-  Val->printAsOperand(OS, false);
-  OS << " --> ";
-  if (Expr.size() == 0) {
-    OS << "0\n";
-    return;
-  }
-
-  ListSeparator LS(" + ");
-  for (auto I = Expr.begin(); I != Expr.end(); I++) {
-    OS << LS;
-    printAndChain(OS, *I);
-  }
-
-  OS << "\n";
-}
-
-void LogicCombiner::clear() {
-  LogicalOpNodes.clear();
-  LeafValues.clear();
-}
-
-LogicalOpNode *LogicCombiner::visitLeafNode(Value *Val, unsigned Depth) {
-  // Depth is 0 means the root is not logical operation. We can't
-  // do anything for that.
-  if (Depth == 0 || LeafValues.size() >= MaxLogicOpLeafsToScan)
-    return nullptr;
-
-  uint64_t ExprVal = 1ULL << LeafValues.size();
-  // Constant Zero,AllOne are special leaf nodes. They involve
-  // LogicalExpr's calculation so we must detect them at first.
-  if (auto ConstVal = dyn_cast<ConstantInt>(Val)) {
-    if (ConstVal->isZero())
-      ExprVal = 0;
-    else if (ConstVal->isAllOnesValue())
-      ExprVal = LogicalExpr::ExprAllOne;
-  }
-  if (ExprVal != LogicalExpr::ExprAllOne && ExprVal != 0)
-    LeafValues.insert(Val);
-  LogicalOpNode *Node =
-      new (Alloc.Allocate()) LogicalOpNode(this, Val, LogicalExpr(ExprVal));
-  LogicalOpNodes[Val] = Node;
-  return Node;
-}
-
-LogicalOpNode *LogicCombiner::visitBinOp(BinaryOperator *BO, unsigned Depth) {
-  if (!BO->isBitwiseLogicOp())
-    return visitLeafNode(BO, Depth);
-
-  LogicalOpNode *LHS = getLogicalOpNode(BO->getOperand(0), Depth + 1);
-  if (LHS == nullptr)
-    return nullptr;
-
-  LogicalOpNode *RHS = getLogicalOpNode(BO->getOperand(1), Depth + 1);
-  if (RHS == nullptr)
-    return nullptr;
-
-  LogicalOpNode *Node;
-  if (BO->getOpcode() == Instruction::And)
-    Node = new (Alloc.Allocate())
-        LogicalOpNode(this, BO, LHS->getExpr() & RHS->getExpr());
-  else if (BO->getOpcode() == Instruction::Or)
-    Node = new (Alloc.Allocate())
-        LogicalOpNode(this, BO, LHS->getExpr() | RHS->getExpr());
-  else
-    Node = new (Alloc.Allocate())
-        LogicalOpNode(this, BO, LHS->getExpr() ^ RHS->getExpr());
-  LogicalOpNodes[BO] = Node;
-  return Node;
-}
-
-LogicalOpNode *LogicCombiner::getLogicalOpNode(Value *Val, unsigned Depth) {
-  if (Depth == MaxDepthLogicOpsToScan)
-    return nullptr;
-
-  if (LogicalOpNodes.find(Val) == LogicalOpNodes.end()) {
-    LogicalOpNode *Node;
-
-    // TODO: add select instruction support
-    if (auto *BO = dyn_cast<BinaryOperator>(Val))
-      Node = visitBinOp(BO, Depth);
-    else
-      Node = visitLeafNode(Val, Depth);
-
-    if (!Node)
-      return nullptr;
-    LLVM_DEBUG(dbgs() << *Node);
-  }
-  return LogicalOpNodes[Val];
-}
-
-Value *LogicCombiner::logicalOpToValue(LogicalOpNode *Node) {
-  const LogicalExpr &Expr = Node->getExpr();
-  // Empty when all leaf bits are erased from the set because a ^ a = 0.
-  if (Expr.size() == 0)
-    return Constant::getNullValue(Node->getValue()->getType());
-
-  if (Expr.size() == 1) {
-    uint64_t LeafBits = *Expr.begin();
-    if (LeafBits == 0)
-      return Constant::getNullValue(Node->getValue()->getType());
-    // ExprAllOne is not in the LeafValues
-    if (LeafBits == LogicalExpr::ExprAllOne)
-      return Constant::getAllOnesValue(Node->getValue()->getType());
-
-    if (popcount(LeafBits) == 1)
-      return LeafValues[Log2_64(LeafBits)];
-  }
-
-  // TODO: find the simplest form from logical expression when it is not
-  // only an "and" chain.
-
-  return nullptr;
-}
-
-Value *LogicCombiner::simplify(Value *Root) {
-  assert(MaxLogicOpLeafsToScan <= 63 &&
-         "Logical leaf node can't be larger than 63.");
-  LogicalOpNode *RootNode = getLogicalOpNode(Root);
-  if (RootNode == nullptr)
-    return nullptr;
-
-  Value *NewRoot = logicalOpToValue(RootNode);
-  if (NewRoot == nullptr || NewRoot == Root)
-    return nullptr;
-
-  LogicalOpNodes.erase(Root);
-  NumLogicalOpsSimplified++;
-  return NewRoot;
-}
diff --git a/llvm/lib/Analysis/LoopAccessAnalysis.cpp b/llvm/lib/Analysis/LoopAccessAnalysis.cpp
index 9e110567e98e4..d8d61d61b92f4 100644
--- a/llvm/lib/Analysis/LoopAccessAnalysis.cpp
+++ b/llvm/lib/Analysis/LoopAccessAnalysis.cpp
@@ -2704,12 +2704,32 @@ void LoopAccessLegacyAnalysis::getAnalysisUsage(AnalysisUsage &AU) const {
   AU.setPreservesAll();
 }
 
+bool LoopAccessInfoManager::invalidate(
+    Function &F, const PreservedAnalyses &PA,
+    FunctionAnalysisManager::Invalidator &Inv) {
+  // Check whether our analysis is preserved.
+  auto PAC = PA.getChecker<LoopAccessAnalysis>();
+  if (!PAC.preserved() && !PAC.preservedSet<AllAnalysesOn<Function>>())
+    // If not, give up now.
+    return true;
+
+  // Check whether the analyses we depend on became invalid for any reason.
+  // Skip checking TargetLibraryAnalysis as it is immutable and can't become
+  // invalid.
+  return Inv.invalidate<AAManager>(F, PA) ||
+         Inv.invalidate<ScalarEvolutionAnalysis>(F, PA) ||
+         Inv.invalidate<LoopAnalysis>(F, PA) ||
+         Inv.invalidate<DominatorTreeAnalysis>(F, PA);
+}
+
 LoopAccessInfoManager LoopAccessAnalysis::run(Function &F,
-                                              FunctionAnalysisManager &AM) {
-  return LoopAccessInfoManager(
-      AM.getResult<ScalarEvolutionAnalysis>(F), AM.getResult<AAManager>(F),
-      AM.getResult<DominatorTreeAnalysis>(F), AM.getResult<LoopAnalysis>(F),
-      &AM.getResult<TargetLibraryAnalysis>(F));
+                                              FunctionAnalysisManager &FAM) {
+  auto &SE = FAM.getResult<ScalarEvolutionAnalysis>(F);
+  auto &AA = FAM.getResult<AAManager>(F);
+  auto &DT = FAM.getResult<DominatorTreeAnalysis>(F);
+  auto &LI = FAM.getResult<LoopAnalysis>(F);
+  auto &TLI = FAM.getResult<TargetLibraryAnalysis>(F);
+  return LoopAccessInfoManager(SE, AA, DT, LI, &TLI);
 }
 
 char LoopAccessLegacyAnalysis::ID = 0;
diff --git a/llvm/lib/Analysis/ScalarEvolution.cpp b/llvm/lib/Analysis/ScalarEvolution.cpp
index 5da3aaac3c232..e3c4fc57c202c 100644
--- a/llvm/lib/Analysis/ScalarEvolution.cpp
+++ b/llvm/lib/Analysis/ScalarEvolution.cpp
@@ -1625,11 +1625,6 @@ const SCEV *ScalarEvolution::getZeroExtendExprImpl(const SCEV *Op, Type *Ty,
       unsigned BitWidth = getTypeSizeInBits(AR->getType());
       const Loop *L = AR->getLoop();
 
-      if (!AR->hasNoUnsignedWrap()) {
-        auto NewFlags = proveNoWrapViaConstantRanges(AR);
-        setNoWrapFlags(const_cast<SCEVAddRecExpr *>(AR), NewFlags);
-      }
-
       // If we have special knowledge that this addrec won't overflow,
       // we don't need to do any further analysis.
       if (AR->hasNoUnsignedWrap()) {
@@ -2009,11 +2004,6 @@ const SCEV *ScalarEvolution::getSignExtendExprImpl(const SCEV *Op, Type *Ty,
       unsigned BitWidth = getTypeSizeInBits(AR->getType());
       const Loop *L = AR->getLoop();
 
-      if (!AR->hasNoSignedWrap()) {
-        auto NewFlags = proveNoWrapViaConstantRanges(AR);
-        setNoWrapFlags(const_cast<SCEVAddRecExpr *>(AR), NewFlags);
-      }
-
       // If we have special knowledge that this addrec won't overflow,
       // we don't need to do any further analysis.
       if (AR->hasNoSignedWrap()) {
@@ -5679,6 +5669,12 @@ const SCEV *ScalarEvolution::createSimpleAffineAddRec(PHINode *PN,
   const SCEV *PHISCEV = getAddRecExpr(StartVal, Accum, L, Flags);
   insertValueToMap(PN, PHISCEV);
 
+  if (auto *AR = dyn_cast<SCEVAddRecExpr>(PHISCEV)) {
+    setNoWrapFlags(const_cast<SCEVAddRecExpr *>(AR),
+                   (SCEV::NoWrapFlags)(AR->getNoWrapFlags() |
+                                       proveNoWrapViaConstantRanges(AR)));
+  }
+
   // We can add Flags to the post-inc expression only if we
   // know that it is *undefined behavior* for BEValueV to
   // overflow.
@@ -5804,6 +5800,12 @@ const SCEV *ScalarEvolution::createAddRecFromPHI(PHINode *PN) {
         forgetMemoizedResults(SymbolicName);
         insertValueToMap(PN, PHISCEV);
 
+        if (auto *AR = dyn_cast<SCEVAddRecExpr>(PHISCEV)) {
+          setNoWrapFlags(const_cast<SCEVAddRecExpr *>(AR),
+                         (SCEV::NoWrapFlags)(AR->getNoWrapFlags() |
+                                             proveNoWrapViaConstantRanges(AR)));
+        }
+
         // We can add Flags to the post-inc expression only if we
         // know that it is *undefined behavior* for BEValueV to
         // overflow.
@@ -6485,7 +6487,7 @@ ScalarEvolution::getRangeRefIter(const SCEV *S,
   auto AddToWorklist = [&WorkList, &Seen, &Cache](const SCEV *Expr) {
     if (!Seen.insert(Expr).second)
       return;
-    if (Cache.find(Expr) != Cache.end())
+    if (Cache.contains(Expr))
       return;
     switch (Expr->getSCEVType()) {
     case scUnknown:
@@ -6598,7 +6600,7 @@ const ConstantRange &ScalarEvolution::getRangeRef(
   case scConstant:
     llvm_unreachable("Already handled above.");
   case scVScale:
-    return setRange(S, SignHint, std::move(ConservativeResult));
+    return setRange(S, SignHint, getVScaleRange(&F, BitWidth));
   case scTruncate: {
     const SCEVTruncateExpr *Trunc = cast<SCEVTruncateExpr>(S);
     ConstantRange X = getRangeRef(Trunc->getOperand(), SignHint, Depth + 1);
@@ -8005,6 +8007,8 @@ const SCEV *ScalarEvolution::createSCEV(Value *V) {
         // A start_loop_iterations or llvm.annotation or llvm.prt.annotation is
         // just eqivalent to the first operand for SCEV purposes.
         return getSCEV(II->getArgOperand(0));
+      case Intrinsic::vscale:
+        return getVScale(II->getType());
       default:
         break;
       }
@@ -8819,9 +8823,9 @@ ScalarEvolution::computeExitLimit(const Loop *L, BasicBlock *ExitingBlock,
     assert(ExitIfTrue == L->contains(BI->getSuccessor(1)) &&
            "It should have one successor in loop and one exit block!");
     // Proceed to the next level to examine the exit condition expression.
-    return computeExitLimitFromCond(
-        L, BI->getCondition(), ExitIfTrue,
-        /*ControlsExit=*/IsOnlyExit, AllowPredicates);
+    return computeExitLimitFromCond(L, BI->getCondition(), ExitIfTrue,
+                                    /*ControlsOnlyExit=*/IsOnlyExit,
+                                    AllowPredicates);
   }
 
   if (SwitchInst *SI = dyn_cast<SwitchInst>(Term)) {
@@ -8834,24 +8838,25 @@ ScalarEvolution::computeExitLimit(const Loop *L, BasicBlock *ExitingBlock,
         Exit = SBB;
       }
     assert(Exit && "Exiting block must have at least one exit");
-    return computeExitLimitFromSingleExitSwitch(L, SI, Exit,
-                                                /*ControlsExit=*/IsOnlyExit);
+    return computeExitLimitFromSingleExitSwitch(
+        L, SI, Exit,
+        /*ControlsOnlyExit=*/IsOnlyExit);
   }
 
   return getCouldNotCompute();
 }
 
 ScalarEvolution::ExitLimit ScalarEvolution::computeExitLimitFromCond(
-    const Loop *L, Value *ExitCond, bool ExitIfTrue,
-    bool ControlsExit, bool AllowPredicates) {
+    const Loop *L, Value *ExitCond, bool ExitIfTrue, bool ControlsOnlyExit,
+    bool AllowPredicates) {
   ScalarEvolution::ExitLimitCacheTy Cache(L, ExitIfTrue, AllowPredicates);
   return computeExitLimitFromCondCached(Cache, L, ExitCond, ExitIfTrue,
-                                        ControlsExit, AllowPredicates);
+                                        ControlsOnlyExit, AllowPredicates);
 }
 
 std::optional<ScalarEvolution::ExitLimit>
 ScalarEvolution::ExitLimitCache::find(const Loop *L, Value *ExitCond,
-                                      bool ExitIfTrue, bool ControlsExit,
+                                      bool ExitIfTrue, bool ControlsOnlyExit,
                                       bool AllowPredicates) {
   (void)this->L;
   (void)this->ExitIfTrue;
@@ -8860,7 +8865,7 @@ ScalarEvolution::ExitLimitCache::find(const Loop *L, Value *ExitCond,
   assert(this->L == L && this->ExitIfTrue == ExitIfTrue &&
          this->AllowPredicates == AllowPredicates &&
          "Variance in assumed invariant key components!");
-  auto Itr = TripCountMap.find({ExitCond, ControlsExit});
+  auto Itr = TripCountMap.find({ExitCond, ControlsOnlyExit});
   if (Itr == TripCountMap.end())
     return std::nullopt;
   return Itr->second;
@@ -8868,14 +8873,14 @@ ScalarEvolution::ExitLimitCache::find(const Loop *L, Value *ExitCond,
 
 void ScalarEvolution::ExitLimitCache::insert(const Loop *L, Value *ExitCond,
                                              bool ExitIfTrue,
-                                             bool ControlsExit,
+                                             bool ControlsOnlyExit,
                                              bool AllowPredicates,
                                              const ExitLimit &EL) {
   assert(this->L == L && this->ExitIfTrue == ExitIfTrue &&
          this->AllowPredicates == AllowPredicates &&
          "Variance in assumed invariant key components!");
 
-  auto InsertResult = TripCountMap.insert({{ExitCond, ControlsExit}, EL});
+  auto InsertResult = TripCountMap.insert({{ExitCond, ControlsOnlyExit}, EL});
   assert(InsertResult.second && "Expected successful insertion!");
   (void)InsertResult;
   (void)ExitIfTrue;
@@ -8883,36 +8888,37 @@ void ScalarEvolution::ExitLimitCache::insert(const Loop *L, Value *ExitCond,
 
 ScalarEvolution::ExitLimit ScalarEvolution::computeExitLimitFromCondCached(
     ExitLimitCacheTy &Cache, const Loop *L, Value *ExitCond, bool ExitIfTrue,
-    bool ControlsExit, bool AllowPredicates) {
+    bool ControlsOnlyExit, bool AllowPredicates) {
 
-  if (auto MaybeEL =
-          Cache.find(L, ExitCond, ExitIfTrue, ControlsExit, AllowPredicates))
+  if (auto MaybeEL = Cache.find(L, ExitCond, ExitIfTrue, ControlsOnlyExit,
+                                AllowPredicates))
     return *MaybeEL;
 
-  ExitLimit EL = computeExitLimitFromCondImpl(Cache, L, ExitCond, ExitIfTrue,
-                                              ControlsExit, AllowPredicates);
-  Cache.insert(L, ExitCond, ExitIfTrue, ControlsExit, AllowPredicates, EL);
+  ExitLimit EL = computeExitLimitFromCondImpl(
+      Cache, L, ExitCond, ExitIfTrue, ControlsOnlyExit, AllowPredicates);
+  Cache.insert(L, ExitCond, ExitIfTrue, ControlsOnlyExit, AllowPredicates, EL);
   return EL;
 }
 
 ScalarEvolution::ExitLimit ScalarEvolution::computeExitLimitFromCondImpl(
     ExitLimitCacheTy &Cache, const Loop *L, Value *ExitCond, bool ExitIfTrue,
-    bool ControlsExit, bool AllowPredicates) {
+    bool ControlsOnlyExit, bool AllowPredicates) {
   // Handle BinOp conditions (And, Or).
   if (auto LimitFromBinOp = computeExitLimitFromCondFromBinOp(
-          Cache, L, ExitCond, ExitIfTrue, ControlsExit, AllowPredicates))
+          Cache, L, ExitCond, ExitIfTrue, ControlsOnlyExit, AllowPredicates))
     return *LimitFromBinOp;
 
   // With an icmp, it may be feasible to compute an exact backedge-taken count.
   // Proceed to the next level to examine the icmp.
   if (ICmpInst *ExitCondICmp = dyn_cast<ICmpInst>(ExitCond)) {
     ExitLimit EL =
-        computeExitLimitFromICmp(L, ExitCondICmp, ExitIfTrue, ControlsExit);
+        computeExitLimitFromICmp(L, ExitCondICmp, ExitIfTrue, ControlsOnlyExit);
     if (EL.hasFullInfo() || !AllowPredicates)
       return EL;
 
     // Try again, but use SCEV predicates this time.
-    return computeExitLimitFromICmp(L, ExitCondICmp, ExitIfTrue, ControlsExit,
+    return computeExitLimitFromICmp(L, ExitCondICmp, ExitIfTrue,
+                                    ControlsOnlyExit,
                                     /*AllowPredicates=*/true);
   }
 
@@ -8948,8 +8954,9 @@ ScalarEvolution::ExitLimit ScalarEvolution::computeExitLimitFromCondImpl(
     if (Offset != 0)
       LHS = getAddExpr(LHS, getConstant(Offset));
     auto EL = computeExitLimitFromICmp(L, Pred, LHS, getConstant(NewRHSC),
-                                       ControlsExit, AllowPredicates);
-    if (EL.hasAnyInfo()) return EL;
+                                       ControlsOnlyExit, AllowPredicates);
+    if (EL.hasAnyInfo())
+      return EL;
   }
 
   // If it's not an integer or pointer comparison then compute it the hard way.
@@ -8959,7 +8966,7 @@ ScalarEvolution::ExitLimit ScalarEvolution::computeExitLimitFromCondImpl(
 std::optional<ScalarEvolution::ExitLimit>
 ScalarEvolution::computeExitLimitFromCondFromBinOp(
     ExitLimitCacheTy &Cache, const Loop *L, Value *ExitCond, bool ExitIfTrue,
-    bool ControlsExit, bool AllowPredicates) {
+    bool ControlsOnlyExit, bool AllowPredicates) {
   // Check if the controlling expression for this loop is an And or Or.
   Value *Op0, *Op1;
   bool IsAnd = false;
@@ -8974,12 +8981,12 @@ ScalarEvolution::computeExitLimitFromCondFromBinOp(
   //   br (and Op0 Op1), loop, exit
   //   br (or  Op0 Op1), exit, loop
   bool EitherMayExit = IsAnd ^ ExitIfTrue;
-  ExitLimit EL0 = computeExitLimitFromCondCached(Cache, L, Op0, ExitIfTrue,
-                                                 ControlsExit && !EitherMayExit,
-                                                 AllowPredicates);
-  ExitLimit EL1 = computeExitLimitFromCondCached(Cache, L, Op1, ExitIfTrue,
-                                                 ControlsExit && !EitherMayExit,
-                                                 AllowPredicates);
+  ExitLimit EL0 = computeExitLimitFromCondCached(
+      Cache, L, Op0, ExitIfTrue, ControlsOnlyExit && !EitherMayExit,
+      AllowPredicates);
+  ExitLimit EL1 = computeExitLimitFromCondCached(
+      Cache, L, Op1, ExitIfTrue, ControlsOnlyExit && !EitherMayExit,
+      AllowPredicates);
 
   // Be robust against unsimplified IR for the form "op i1 X, NeutralElement"
   const Constant *NeutralElement = ConstantInt::get(ExitCond->getType(), IsAnd);
@@ -9037,12 +9044,9 @@ ScalarEvolution::computeExitLimitFromCondFromBinOp(
                    { &EL0.Predicates, &EL1.Predicates });
 }
 
-ScalarEvolution::ExitLimit
-ScalarEvolution::computeExitLimitFromICmp(const Loop *L,
-                                          ICmpInst *ExitCond,
-                                          bool ExitIfTrue,
-                                          bool ControlsExit,
-                                          bool AllowPredicates) {
+ScalarEvolution::ExitLimit ScalarEvolution::computeExitLimitFromICmp(
+    const Loop *L, ICmpInst *ExitCond, bool ExitIfTrue, bool ControlsOnlyExit,
+    bool AllowPredicates) {
   // If the condition was exit on true, convert the condition to exit on false
   ICmpInst::Predicate Pred;
   if (!ExitIfTrue)
@@ -9054,9 +9058,10 @@ ScalarEvolution::computeExitLimitFromICmp(const Loop *L,
   const SCEV *LHS = getSCEV(ExitCond->getOperand(0));
   const SCEV *RHS = getSCEV(ExitCond->getOperand(1));
 
-  ExitLimit EL = computeExitLimitFromICmp(L, Pred, LHS, RHS, ControlsExit,
+  ExitLimit EL = computeExitLimitFromICmp(L, Pred, LHS, RHS, ControlsOnlyExit,
                                           AllowPredicates);
-  if (EL.hasAnyInfo()) return EL;
+  if (EL.hasAnyInfo())
+    return EL;
 
   auto *ExhaustiveCount =
       computeExitCountExhaustively(L, ExitCond, ExitIfTrue);
@@ -9067,12 +9072,9 @@ ScalarEvolution::computeExitLimitFromICmp(const Loop *L,
   return computeShiftCompareExitLimit(ExitCond->getOperand(0),
                                       ExitCond->getOperand(1), L, OriginalPred);
 }
-ScalarEvolution::ExitLimit
-ScalarEvolution::computeExitLimitFromICmp(const Loop *L,
-                                          ICmpInst::Predicate Pred,
-                                          const SCEV *LHS, const SCEV *RHS,
-                                          bool ControlsExit,
-                                          bool AllowPredicates) {
+ScalarEvolution::ExitLimit ScalarEvolution::computeExitLimitFromICmp(
+    const Loop *L, ICmpInst::Predicate Pred, const SCEV *LHS, const SCEV *RHS,
+    bool ControlsOnlyExit, bool AllowPredicates) {
 
   // Try to evaluate any dependencies out of the loop.
   LHS = getSCEVAtScope(LHS, L);
@@ -9086,12 +9088,10 @@ ScalarEvolution::computeExitLimitFromICmp(const Loop *L,
     Pred = ICmpInst::getSwappedPredicate(Pred);
   }
 
-  bool ControllingFiniteLoop =
-      ControlsExit && loopHasNoAbnormalExits(L) && loopIsFiniteByAssumption(L);
+  bool ControllingFiniteLoop = ControlsOnlyExit && loopHasNoAbnormalExits(L) &&
+                               loopIsFiniteByAssumption(L);
   // Simplify the operands before analyzing them.
-  (void)SimplifyICmpOperands(Pred, LHS, RHS, /*Depth=*/0,
-                             (EnableFiniteLoopControl ? ControllingFiniteLoop
-                                                     : false));
+  (void)SimplifyICmpOperands(Pred, LHS, RHS, /*Depth=*/0);
 
   // If we have a comparison of a chrec against a constant, try to use value
   // ranges to answer this query.
@@ -9143,9 +9143,10 @@ ScalarEvolution::computeExitLimitFromICmp(const Loop *L,
       if (isa<SCEVCouldNotCompute>(RHS))
         return RHS;
     }
-    ExitLimit EL = howFarToZero(getMinusSCEV(LHS, RHS), L, ControlsExit,
+    ExitLimit EL = howFarToZero(getMinusSCEV(LHS, RHS), L, ControlsOnlyExit,
                                 AllowPredicates);
-    if (EL.hasAnyInfo()) return EL;
+    if (EL.hasAnyInfo())
+      return EL;
     break;
   }
   case ICmpInst::ICMP_EQ: {                     // while (X == Y)
@@ -9164,21 +9165,40 @@ ScalarEvolution::computeExitLimitFromICmp(const Loop *L,
     if (EL.hasAnyInfo()) return EL;
     break;
   }
+  case ICmpInst::ICMP_SLE:
+  case ICmpInst::ICMP_ULE:
+    // Since the loop is finite, an invariant RHS cannot include the boundary
+    // value, otherwise it would loop forever.
+    if (!EnableFiniteLoopControl || !ControllingFiniteLoop ||
+        !isLoopInvariant(RHS, L))
+      break;
+    RHS = getAddExpr(getOne(RHS->getType()), RHS);
+    [[fallthrough]];
   case ICmpInst::ICMP_SLT:
-  case ICmpInst::ICMP_ULT: {                    // while (X < Y)
-    bool IsSigned = Pred == ICmpInst::ICMP_SLT;
-    ExitLimit EL = howManyLessThans(LHS, RHS, L, IsSigned, ControlsExit,
+  case ICmpInst::ICMP_ULT: { // while (X < Y)
+    bool IsSigned = ICmpInst::isSigned(Pred);
+    ExitLimit EL = howManyLessThans(LHS, RHS, L, IsSigned, ControlsOnlyExit,
                                     AllowPredicates);
-    if (EL.hasAnyInfo()) return EL;
+    if (EL.hasAnyInfo())
+      return EL;
     break;
   }
+  case ICmpInst::ICMP_SGE:
+  case ICmpInst::ICMP_UGE:
+    // Since the loop is finite, an invariant RHS cannot include the boundary
+    // value, otherwise it would loop forever.
+    if (!EnableFiniteLoopControl || !ControllingFiniteLoop ||
+        !isLoopInvariant(RHS, L))
+      break;
+    RHS = getAddExpr(getMinusOne(RHS->getType()), RHS);
+    [[fallthrough]];
   case ICmpInst::ICMP_SGT:
-  case ICmpInst::ICMP_UGT: {                    // while (X > Y)
-    bool IsSigned = Pred == ICmpInst::ICMP_SGT;
-    ExitLimit EL =
-        howManyGreaterThans(LHS, RHS, L, IsSigned, ControlsExit,
-                            AllowPredicates);
-    if (EL.hasAnyInfo()) return EL;
+  case ICmpInst::ICMP_UGT: { // while (X > Y)
+    bool IsSigned = ICmpInst::isSigned(Pred);
+    ExitLimit EL = howManyGreaterThans(LHS, RHS, L, IsSigned, ControlsOnlyExit,
+                                       AllowPredicates);
+    if (EL.hasAnyInfo())
+      return EL;
     break;
   }
   default:
@@ -9192,7 +9212,7 @@ ScalarEvolution::ExitLimit
 ScalarEvolution::computeExitLimitFromSingleExitSwitch(const Loop *L,
                                                       SwitchInst *Switch,
                                                       BasicBlock *ExitingBlock,
-                                                      bool ControlsExit) {
+                                                      bool ControlsOnlyExit) {
   assert(!L->contains(ExitingBlock) && "Not an exiting block!");
 
   // Give up if the exit is the default dest of a switch.
@@ -9205,7 +9225,7 @@ ScalarEvolution::computeExitLimitFromSingleExitSwitch(const Loop *L,
   const SCEV *RHS = getConstant(Switch->findCaseDest(ExitingBlock));
 
   // while (X != Y) --> while (X-Y != 0)
-  ExitLimit EL = howFarToZero(getMinusSCEV(LHS, RHS), L, ControlsExit);
+  ExitLimit EL = howFarToZero(getMinusSCEV(LHS, RHS), L, ControlsOnlyExit);
   if (EL.hasAnyInfo())
     return EL;
 
@@ -10364,9 +10384,10 @@ SolveQuadraticAddRecRange(const SCEVAddRecExpr *AddRec,
   return TruncIfPossible(MinOptional(SL.first, SU.first), BitWidth);
 }
 
-ScalarEvolution::ExitLimit
-ScalarEvolution::howFarToZero(const SCEV *V, const Loop *L, bool ControlsExit,
-                              bool AllowPredicates) {
+ScalarEvolution::ExitLimit ScalarEvolution::howFarToZero(const SCEV *V,
+                                                         const Loop *L,
+                                                         bool ControlsOnlyExit,
+                                                         bool AllowPredicates) {
 
   // This is only used for loops with a "x != y" exit test. The exit condition
   // is now expressed as a single expression, V = x-y. So the exit test is
@@ -10475,7 +10496,7 @@ ScalarEvolution::howFarToZero(const SCEV *V, const Loop *L, bool ControlsExit,
   // compute the backedge count.  In this case, the step may not divide the
   // distance, but we don't care because if the condition is "missed" the loop
   // will have undefined behavior due to wrapping.
-  if (ControlsExit && AddRec->hasNoSelfWrap() &&
+  if (ControlsOnlyExit && AddRec->hasNoSelfWrap() &&
       loopHasNoAbnormalExits(AddRec->getLoop())) {
     const SCEV *Exact =
         getUDivExpr(Distance, CountDown ? getNegativeSCEV(Step) : Step);
@@ -10570,8 +10591,7 @@ static bool HasSameValue(const SCEV *A, const SCEV *B) {
 
 bool ScalarEvolution::SimplifyICmpOperands(ICmpInst::Predicate &Pred,
                                            const SCEV *&LHS, const SCEV *&RHS,
-                                           unsigned Depth,
-                                           bool ControllingFiniteLoop) {
+                                           unsigned Depth) {
   bool Changed = false;
   // Simplifies ICMP to trivial true or false by turning it into '0 == 0' or
   // '0 != 0'.
@@ -10700,15 +10720,10 @@ bool ScalarEvolution::SimplifyICmpOperands(ICmpInst::Predicate &Pred,
   }
 
   // If possible, canonicalize GE/LE comparisons to GT/LT comparisons, by
-  // adding or subtracting 1 from one of the operands. This can be done for
-  // one of two reasons:
-  // 1) The range of the RHS does not include the (signed/unsigned) boundaries
-  // 2) The loop is finite, with this comparison controlling the exit. Since the
-  // loop is finite, the bound cannot include the corresponding boundary
-  // (otherwise it would loop forever).
+  // adding or subtracting 1 from one of the operands.
   switch (Pred) {
   case ICmpInst::ICMP_SLE:
-    if (ControllingFiniteLoop || !getSignedRangeMax(RHS).isMaxSignedValue()) {
+    if (!getSignedRangeMax(RHS).isMaxSignedValue()) {
       RHS = getAddExpr(getConstant(RHS->getType(), 1, true), RHS,
                        SCEV::FlagNSW);
       Pred = ICmpInst::ICMP_SLT;
@@ -10721,7 +10736,7 @@ bool ScalarEvolution::SimplifyICmpOperands(ICmpInst::Predicate &Pred,
     }
     break;
   case ICmpInst::ICMP_SGE:
-    if (ControllingFiniteLoop || !getSignedRangeMin(RHS).isMinSignedValue()) {
+    if (!getSignedRangeMin(RHS).isMinSignedValue()) {
       RHS = getAddExpr(getConstant(RHS->getType(), (uint64_t)-1, true), RHS,
                        SCEV::FlagNSW);
       Pred = ICmpInst::ICMP_SGT;
@@ -10734,7 +10749,7 @@ bool ScalarEvolution::SimplifyICmpOperands(ICmpInst::Predicate &Pred,
     }
     break;
   case ICmpInst::ICMP_ULE:
-    if (ControllingFiniteLoop || !getUnsignedRangeMax(RHS).isMaxValue()) {
+    if (!getUnsignedRangeMax(RHS).isMaxValue()) {
       RHS = getAddExpr(getConstant(RHS->getType(), 1, true), RHS,
                        SCEV::FlagNUW);
       Pred = ICmpInst::ICMP_ULT;
@@ -10746,7 +10761,7 @@ bool ScalarEvolution::SimplifyICmpOperands(ICmpInst::Predicate &Pred,
     }
     break;
   case ICmpInst::ICMP_UGE:
-    if (ControllingFiniteLoop || !getUnsignedRangeMin(RHS).isMinValue()) {
+    if (!getUnsignedRangeMin(RHS).isMinValue()) {
       RHS = getAddExpr(getConstant(RHS->getType(), (uint64_t)-1, true), RHS);
       Pred = ICmpInst::ICMP_UGT;
       Changed = true;
@@ -10766,8 +10781,7 @@ bool ScalarEvolution::SimplifyICmpOperands(ICmpInst::Predicate &Pred,
   // Recursively simplify until we either hit a recursion limit or nothing
   // changes.
   if (Changed)
-    return SimplifyICmpOperands(Pred, LHS, RHS, Depth + 1,
-                                ControllingFiniteLoop);
+    return SimplifyICmpOperands(Pred, LHS, RHS, Depth + 1);
 
   return Changed;
 }
@@ -12694,7 +12708,7 @@ const SCEV *ScalarEvolution::computeMaxBECountForLT(const SCEV *Start,
 ScalarEvolution::ExitLimit
 ScalarEvolution::howManyLessThans(const SCEV *LHS, const SCEV *RHS,
                                   const Loop *L, bool IsSigned,
-                                  bool ControlsExit, bool AllowPredicates) {
+                                  bool ControlsOnlyExit, bool AllowPredicates) {
   SmallPtrSet<const SCEVPredicate *, 4> Predicates;
 
   const SCEVAddRecExpr *IV = dyn_cast<SCEVAddRecExpr>(LHS);
@@ -12722,7 +12736,7 @@ ScalarEvolution::howManyLessThans(const SCEV *LHS, const SCEV *RHS,
     if (!StrideC || !StrideC->getAPInt().isPowerOf2())
       return false;
 
-    if (!ControlsExit || !loopHasNoAbnormalExits(L))
+    if (!ControlsOnlyExit || !loopHasNoAbnormalExits(L))
       return false;
 
     return loopIsFiniteByAssumption(L);
@@ -12797,7 +12811,7 @@ ScalarEvolution::howManyLessThans(const SCEV *LHS, const SCEV *RHS,
   // implicit/exceptional) which causes the loop to execute before the
   // exiting instruction we're analyzing would trigger UB.
   auto WrapType = IsSigned ? SCEV::FlagNSW : SCEV::FlagNUW;
-  bool NoWrap = ControlsExit && IV->getNoWrapFlags(WrapType);
+  bool NoWrap = ControlsOnlyExit && IV->getNoWrapFlags(WrapType);
   ICmpInst::Predicate Cond = IsSigned ? ICmpInst::ICMP_SLT : ICmpInst::ICMP_ULT;
 
   const SCEV *Stride = IV->getStepRecurrence(*this);
@@ -13117,10 +13131,9 @@ ScalarEvolution::howManyLessThans(const SCEV *LHS, const SCEV *RHS,
                    Predicates);
 }
 
-ScalarEvolution::ExitLimit
-ScalarEvolution::howManyGreaterThans(const SCEV *LHS, const SCEV *RHS,
-                                     const Loop *L, bool IsSigned,
-                                     bool ControlsExit, bool AllowPredicates) {
+ScalarEvolution::ExitLimit ScalarEvolution::howManyGreaterThans(
+    const SCEV *LHS, const SCEV *RHS, const Loop *L, bool IsSigned,
+    bool ControlsOnlyExit, bool AllowPredicates) {
   SmallPtrSet<const SCEVPredicate *, 4> Predicates;
   // We handle only IV > Invariant
   if (!isLoopInvariant(RHS, L))
@@ -13138,7 +13151,7 @@ ScalarEvolution::howManyGreaterThans(const SCEV *LHS, const SCEV *RHS,
     return getCouldNotCompute();
 
   auto WrapType = IsSigned ? SCEV::FlagNSW : SCEV::FlagNUW;
-  bool NoWrap = ControlsExit && IV->getNoWrapFlags(WrapType);
+  bool NoWrap = ControlsOnlyExit && IV->getNoWrapFlags(WrapType);
   ICmpInst::Predicate Cond = IsSigned ? ICmpInst::ICMP_SGT : ICmpInst::ICMP_UGT;
 
   const SCEV *Stride = getNegativeSCEV(IV->getStepRecurrence(*this));
@@ -14294,10 +14307,11 @@ AnalysisKey ScalarEvolutionAnalysis::Key;
 
 ScalarEvolution ScalarEvolutionAnalysis::run(Function &F,
                                              FunctionAnalysisManager &AM) {
-  return ScalarEvolution(F, AM.getResult<TargetLibraryAnalysis>(F),
-                         AM.getResult<AssumptionAnalysis>(F),
-                         AM.getResult<DominatorTreeAnalysis>(F),
-                         AM.getResult<LoopAnalysis>(F));
+  auto &TLI = AM.getResult<TargetLibraryAnalysis>(F);
+  auto &AC = AM.getResult<AssumptionAnalysis>(F);
+  auto &DT = AM.getResult<DominatorTreeAnalysis>(F);
+  auto &LI = AM.getResult<LoopAnalysis>(F);
+  return ScalarEvolution(F, TLI, AC, DT, LI);
 }
 
 PreservedAnalyses
@@ -14582,8 +14596,7 @@ void SCEVComparePredicate::print(raw_ostream &OS, unsigned Depth) const {
   if (Pred == ICmpInst::ICMP_EQ)
     OS.indent(Depth) << "Equal predicate: " << *LHS << " == " << *RHS << "\n";
   else
-    OS.indent(Depth) << "Compare predicate: " << *LHS
-                     << " " << CmpInst::getPredicateName(Pred) << ") "
+    OS.indent(Depth) << "Compare predicate: " << *LHS << " " << Pred << ") "
                      << *RHS << "\n";
 
 }
@@ -15058,57 +15071,93 @@ const SCEV *ScalarEvolution::applyLoopGuards(const SCEV *Expr, const Loop *L) {
       return I != RewriteMap.end() ? I->second : S;
     };
 
-    const SCEV *RewrittenLHS = GetMaybeRewritten(LHS);
-
-    const SCEV *RewrittenRHS = nullptr;
+    // Collect rewrites for LHS and its transitive operands based on the
+    // condition.
+    // For min/max expressions, also apply the guard to its operands:
+    //  'min(a, b) >= c'   ->   '(a >= c) and (b >= c)',
+    //  'min(a, b) >  c'   ->   '(a >  c) and (b >  c)',
+    //  'max(a, b) <= c'   ->   '(a <= c) and (b <= c)',
+    //  'max(a, b) <  c'   ->   '(a <  c) and (b <  c)'.
+
+    // We cannot express strict predicates in SCEV, so instead we replace them
+    // with non-strict ones against plus or minus one of RHS depending on the
+    // predicate.
+    const SCEV *One = getOne(RHS->getType());
     switch (Predicate) {
-    case CmpInst::ICMP_ULT: {
-      if (RHS->getType()->isPointerTy())
+      case CmpInst::ICMP_ULT:
+        if (RHS->getType()->isPointerTy())
+          return;
+        RHS = getUMaxExpr(RHS, One);
+        LLVM_FALLTHROUGH;
+      case CmpInst::ICMP_SLT: {
+        RHS = getMinusSCEV(RHS, One);
+        break;
+      }
+      case CmpInst::ICMP_UGT:
+      case CmpInst::ICMP_SGT:
+        RHS = getAddExpr(RHS, One);
+        break;
+      default:
         break;
-      const SCEV *One = getOne(RHS->getType());
-      RewrittenRHS =
-          getUMinExpr(RewrittenLHS, getMinusSCEV(getUMaxExpr(RHS, One), One));
-      break;
-    }
-    case CmpInst::ICMP_SLT:
-      RewrittenRHS =
-          getSMinExpr(RewrittenLHS, getMinusSCEV(RHS, getOne(RHS->getType())));
-      break;
-    case CmpInst::ICMP_ULE:
-      RewrittenRHS = getUMinExpr(RewrittenLHS, RHS);
-      break;
-    case CmpInst::ICMP_SLE:
-      RewrittenRHS = getSMinExpr(RewrittenLHS, RHS);
-      break;
-    case CmpInst::ICMP_UGT:
-      RewrittenRHS =
-          getUMaxExpr(RewrittenLHS, getAddExpr(RHS, getOne(RHS->getType())));
-      break;
-    case CmpInst::ICMP_SGT:
-      RewrittenRHS =
-          getSMaxExpr(RewrittenLHS, getAddExpr(RHS, getOne(RHS->getType())));
-      break;
-    case CmpInst::ICMP_UGE:
-      RewrittenRHS = getUMaxExpr(RewrittenLHS, RHS);
-      break;
-    case CmpInst::ICMP_SGE:
-      RewrittenRHS = getSMaxExpr(RewrittenLHS, RHS);
-      break;
-    case CmpInst::ICMP_EQ:
-      if (isa<SCEVConstant>(RHS))
-        RewrittenRHS = RHS;
-      break;
-    case CmpInst::ICMP_NE:
-      if (isa<SCEVConstant>(RHS) &&
-          cast<SCEVConstant>(RHS)->getValue()->isNullValue())
-        RewrittenRHS = getUMaxExpr(RewrittenLHS, getOne(RHS->getType()));
-      break;
-    default:
-      break;
     }
 
-    if (RewrittenRHS)
-      AddRewrite(LHS, RewrittenLHS, RewrittenRHS);
+    SmallVector<const SCEV *, 16> Worklist(1, LHS);
+    SmallPtrSet<const SCEV *, 16> Visited;
+
+    auto EnqueueOperands = [&Worklist](const SCEVNAryExpr *S) {
+      append_range(Worklist, S->operands());
+    };
+
+    while (!Worklist.empty()) {
+      const SCEV *From = Worklist.pop_back_val();
+      if (isa<SCEVConstant>(From))
+        continue;
+      if (!Visited.insert(From).second)
+        continue;
+      const SCEV *FromRewritten = GetMaybeRewritten(From);
+      const SCEV *To = nullptr;
+
+      switch (Predicate) {
+      case CmpInst::ICMP_ULT:
+      case CmpInst::ICMP_ULE:
+        To = getUMinExpr(FromRewritten, RHS);
+        if (auto *UMax = dyn_cast<SCEVUMaxExpr>(FromRewritten))
+          EnqueueOperands(UMax);
+        break;
+      case CmpInst::ICMP_SLT:
+      case CmpInst::ICMP_SLE:
+        To = getSMinExpr(FromRewritten, RHS);
+        if (auto *SMax = dyn_cast<SCEVSMaxExpr>(FromRewritten))
+          EnqueueOperands(SMax);
+        break;
+      case CmpInst::ICMP_UGT:
+      case CmpInst::ICMP_UGE:
+        To = getUMaxExpr(FromRewritten, RHS);
+        if (auto *UMin = dyn_cast<SCEVUMinExpr>(FromRewritten))
+          EnqueueOperands(UMin);
+        break;
+      case CmpInst::ICMP_SGT:
+      case CmpInst::ICMP_SGE:
+        To = getSMaxExpr(FromRewritten, RHS);
+        if (auto *SMin = dyn_cast<SCEVSMinExpr>(FromRewritten))
+          EnqueueOperands(SMin);
+        break;
+      case CmpInst::ICMP_EQ:
+        if (isa<SCEVConstant>(RHS))
+          To = RHS;
+        break;
+      case CmpInst::ICMP_NE:
+        if (isa<SCEVConstant>(RHS) &&
+            cast<SCEVConstant>(RHS)->getValue()->isNullValue())
+          To = getUMaxExpr(FromRewritten, One);
+        break;
+      default:
+        break;
+      }
+
+      if (To)
+        AddRewrite(From, FromRewritten, To);
+    }
   };
 
   BasicBlock *Header = L->getHeader();
diff --git a/llvm/lib/Analysis/StackLifetime.cpp b/llvm/lib/Analysis/StackLifetime.cpp
index 46e1f132c8d2d..3e1b5dea6f6c2 100644
--- a/llvm/lib/Analysis/StackLifetime.cpp
+++ b/llvm/lib/Analysis/StackLifetime.cpp
@@ -39,7 +39,7 @@ StackLifetime::getLiveRange(const AllocaInst *AI) const {
 }
 
 bool StackLifetime::isReachable(const Instruction *I) const {
-  return BlockInstRange.find(I->getParent()) != BlockInstRange.end();
+  return BlockInstRange.contains(I->getParent());
 }
 
 bool StackLifetime::isAliveAfter(const AllocaInst *AI,
diff --git a/llvm/lib/Analysis/TargetTransformInfo.cpp b/llvm/lib/Analysis/TargetTransformInfo.cpp
index 5ef329f600f3e..f444b39a4fe77 100644
--- a/llvm/lib/Analysis/TargetTransformInfo.cpp
+++ b/llvm/lib/Analysis/TargetTransformInfo.cpp
@@ -220,6 +220,14 @@ TargetTransformInfo::getGEPCost(Type *PointeeType, const Value *Ptr,
   return TTIImpl->getGEPCost(PointeeType, Ptr, Operands, CostKind);
 }
 
+InstructionCost TargetTransformInfo::getPointersChainCost(
+    ArrayRef<const Value *> Ptrs, const Value *Base,
+    const TTI::PointersChainInfo &Info, TTI::TargetCostKind CostKind) const {
+  assert((Base || !Info.isSameBase()) &&
+         "If pointers have same base address it has to be provided.");
+  return TTIImpl->getPointersChainCost(Ptrs, Base, Info, CostKind);
+}
+
 unsigned TargetTransformInfo::getEstimatedNumberOfCaseClusters(
     const SwitchInst &SI, unsigned &JTSize, ProfileSummaryInfo *PSI,
     BlockFrequencyInfo *BFI) const {
diff --git a/llvm/lib/Analysis/UniformityAnalysis.cpp b/llvm/lib/Analysis/UniformityAnalysis.cpp
index bf1d3a61f75d7..13a9c2b7e4438 100644
--- a/llvm/lib/Analysis/UniformityAnalysis.cpp
+++ b/llvm/lib/Analysis/UniformityAnalysis.cpp
@@ -32,12 +32,10 @@ bool llvm::GenericUniformityAnalysisImpl<SSAContext>::markDefsDivergent(
 
 template <> void llvm::GenericUniformityAnalysisImpl<SSAContext>::initialize() {
   for (auto &I : instructions(F)) {
-    if (TTI->isSourceOfDivergence(&I)) {
-      assert(!I.isTerminator());
+    if (TTI->isSourceOfDivergence(&I))
       markDivergent(I);
-    } else if (TTI->isAlwaysUniform(&I)) {
+    else if (TTI->isAlwaysUniform(&I))
       addUniformOverride(I);
-    }
   }
   for (auto &Arg : F.args()) {
     if (TTI->isSourceOfDivergence(&Arg)) {
@@ -50,13 +48,10 @@ template <>
 void llvm::GenericUniformityAnalysisImpl<SSAContext>::pushUsers(
     const Value *V) {
   for (const auto *User : V->users()) {
-    const auto *UserInstr = dyn_cast<const Instruction>(User);
-    if (!UserInstr)
-      continue;
-    if (isAlwaysUniform(*UserInstr))
-      continue;
-    if (markDivergent(*UserInstr)) {
-      Worklist.push_back(UserInstr);
+    if (const auto *UserInstr = dyn_cast<const Instruction>(User)) {
+      if (markDivergent(*UserInstr)) {
+        Worklist.push_back(UserInstr);
+      }
     }
   }
 }
@@ -83,6 +78,19 @@ bool llvm::GenericUniformityAnalysisImpl<SSAContext>::usesValueFromCycle(
   return false;
 }
 
+template <>
+bool llvm::GenericUniformityAnalysisImpl<SSAContext>::isDivergentUse(
+    const Use &U) const {
+  const auto *V = U.get();
+  if (isDivergent(V))
+    return true;
+  if (const auto *DefInstr = dyn_cast<Instruction>(V)) {
+    const auto *UseInstr = cast<Instruction>(U.getUser());
+    return isTemporalDivergent(*UseInstr->getParent(), *DefInstr);
+  }
+  return false;
+}
+
 // This ensures explicit instantiation of
 // GenericUniformityAnalysisImpl::ImplDeleter::operator()
 template class llvm::GenericUniformityInfo<SSAContext>;
@@ -127,6 +135,7 @@ UniformityInfoWrapperPass::UniformityInfoWrapperPass() : FunctionPass(ID) {
 INITIALIZE_PASS_BEGIN(UniformityInfoWrapperPass, "uniformity",
                       "Uniformity Analysis", true, true)
 INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
+INITIALIZE_PASS_DEPENDENCY(CycleInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
 INITIALIZE_PASS_END(UniformityInfoWrapperPass, "uniformity",
                     "Uniformity Analysis", true, true)
@@ -134,7 +143,7 @@ INITIALIZE_PASS_END(UniformityInfoWrapperPass, "uniformity",
 void UniformityInfoWrapperPass::getAnalysisUsage(AnalysisUsage &AU) const {
   AU.setPreservesAll();
   AU.addRequired<DominatorTreeWrapperPass>();
-  AU.addRequired<CycleInfoWrapperPass>();
+  AU.addRequiredTransitive<CycleInfoWrapperPass>();
   AU.addRequired<TargetTransformInfoWrapperPass>();
 }
 
diff --git a/llvm/lib/Analysis/ValueTracking.cpp b/llvm/lib/Analysis/ValueTracking.cpp
index 1ad875bec6e77..905b4ac6174a7 100644
--- a/llvm/lib/Analysis/ValueTracking.cpp
+++ b/llvm/lib/Analysis/ValueTracking.cpp
@@ -16,6 +16,7 @@
 #include "llvm/ADT/APInt.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/ScopeExit.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/SmallVector.h"
@@ -1151,6 +1152,25 @@ KnownBits llvm::analyzeKnownBitsFromAndXorOr(
       Query(DL, AC, safeCxtI(I, CxtI), DT, UseInstrInfo, ORE));
 }
 
+ConstantRange llvm::getVScaleRange(const Function *F, unsigned BitWidth) {
+  Attribute Attr = F->getFnAttribute(Attribute::VScaleRange);
+  // Without vscale_range, we only know that vscale is non-zero.
+  if (!Attr.isValid())
+    return ConstantRange(APInt(BitWidth, 1), APInt::getZero(BitWidth));
+
+  unsigned AttrMin = Attr.getVScaleRangeMin();
+  // Minimum is larger than vscale width, result is always poison.
+  if ((unsigned)llvm::bit_width(AttrMin) > BitWidth)
+    return ConstantRange::getEmpty(BitWidth);
+
+  APInt Min(BitWidth, AttrMin);
+  std::optional<unsigned> AttrMax = Attr.getVScaleRangeMax();
+  if (!AttrMax || (unsigned)llvm::bit_width(*AttrMax) > BitWidth)
+    return ConstantRange(Min, APInt::getZero(BitWidth));
+
+  return ConstantRange(Min, APInt(BitWidth, *AttrMax) + 1);
+}
+
 static void computeKnownBitsFromOperator(const Operator *I,
                                          const APInt &DemandedElts,
                                          KnownBits &Known, unsigned Depth,
@@ -1819,31 +1839,10 @@ static void computeKnownBitsFromOperator(const Operator *I,
           Known.Zero.setBitsFrom(17);
         break;
       case Intrinsic::vscale: {
-        if (!II->getParent() || !II->getFunction() ||
-            !II->getFunction()->hasFnAttribute(Attribute::VScaleRange))
-          break;
-
-        auto Attr = II->getFunction()->getFnAttribute(Attribute::VScaleRange);
-        std::optional<unsigned> VScaleMax = Attr.getVScaleRangeMax();
-
-        if (!VScaleMax)
-          break;
-
-        unsigned VScaleMin = Attr.getVScaleRangeMin();
-
-        // If vscale min = max then we know the exact value at compile time
-        // and hence we know the exact bits.
-        if (VScaleMin == VScaleMax) {
-          Known.One = VScaleMin;
-          Known.Zero = VScaleMin;
-          Known.Zero.flipAllBits();
+        if (!II->getParent() || !II->getFunction())
           break;
-        }
-
-        unsigned FirstZeroHighBit = llvm::bit_width(*VScaleMax);
-        if (FirstZeroHighBit < BitWidth)
-          Known.Zero.setBitsFrom(FirstZeroHighBit);
 
+        Known = getVScaleRange(II->getFunction(), BitWidth).toKnownBits();
         break;
       }
       }
@@ -4124,6 +4123,381 @@ bool llvm::isKnownNeverNaN(const Value *V, const TargetLibraryInfo *TLI,
   return false;
 }
 
+/// Return true if it's possible to assume IEEE treatment of input denormals in
+/// \p F for \p Val.
+static bool inputDenormalIsIEEE(const Function &F, const Value *Val) {
+  Type *Ty = Val->getType()->getScalarType();
+  return F.getDenormalMode(Ty->getFltSemantics()).Input == DenormalMode::IEEE;
+}
+
+/// Returns a pair of values, which if passed to llvm.is.fpclass, returns the
+/// same result as an fcmp with the given operands.
+std::pair<Value *, FPClassTest> llvm::fcmpToClassTest(FCmpInst::Predicate Pred,
+                                                      const Function &F,
+                                                      Value *LHS, Value *RHS,
+                                                      bool LookThroughSrc) {
+  const APFloat *ConstRHS;
+  if (!match(RHS, m_APFloat(ConstRHS)))
+    return {nullptr, fcNone};
+
+  if (ConstRHS->isZero()) {
+    // Compares with fcNone are only exactly equal to fcZero if input denormals are
+    // not flushed.
+    if (FCmpInst::isEquality(Pred) && !inputDenormalIsIEEE(F, LHS))
+      return {nullptr, fcNone};
+
+    switch (Pred) {
+    case FCmpInst::FCMP_OEQ: // Match x == 0.0
+      return {LHS, fcZero};
+    case FCmpInst::FCMP_UEQ: // Match isnan(x) || (x == 0.0)
+      return {LHS, fcZero | fcNan};
+    case FCmpInst::FCMP_UNE: // Match (x != 0.0)
+      return {LHS, ~fcZero};
+    case FCmpInst::FCMP_ONE: // Match !isnan(x) && x != 0.0
+      return {LHS, ~fcNan & ~fcZero};
+    case FCmpInst::FCMP_ORD:
+      // Canonical form of ord/uno is with a zero. We could also handle
+      // non-canonical other non-NaN constants or LHS == RHS.
+      return {LHS, ~fcNan};
+    case FCmpInst::FCMP_UNO:
+      return {LHS, fcNan};
+    default:
+      break;
+    }
+
+    return {nullptr, fcNone};
+  }
+
+  Value *Src = LHS;
+  const bool IsFabs = LookThroughSrc && match(LHS, m_FAbs(m_Value(Src)));
+
+  // Compute the test mask that would return true for the ordered comparisons.
+  FPClassTest Mask;
+
+  if (ConstRHS->isInfinity()) {
+    switch (Pred) {
+    case FCmpInst::FCMP_OEQ:
+    case FCmpInst::FCMP_UNE: {
+      // Match __builtin_isinf patterns
+      //
+      //   fcmp oeq x, +inf -> is_fpclass x, fcPosInf
+      //   fcmp oeq fabs(x), +inf -> is_fpclass x, fcInf
+      //   fcmp oeq x, -inf -> is_fpclass x, fcNegInf
+      //   fcmp oeq fabs(x), -inf -> is_fpclass x, 0 -> false
+      //
+      //   fcmp une x, +inf -> is_fpclass x, ~fcPosInf
+      //   fcmp une fabs(x), +inf -> is_fpclass x, ~fcInf
+      //   fcmp une x, -inf -> is_fpclass x, ~fcNegInf
+      //   fcmp une fabs(x), -inf -> is_fpclass x, fcAllFlags -> true
+
+      if (ConstRHS->isNegative()) {
+        Mask = fcNegInf;
+        if (IsFabs)
+          Mask = fcNone;
+      } else {
+        Mask = fcPosInf;
+        if (IsFabs)
+          Mask |= fcNegInf;
+      }
+
+      break;
+    }
+    case FCmpInst::FCMP_ONE:
+    case FCmpInst::FCMP_UEQ: {
+      // Match __builtin_isinf patterns
+      //   fcmp one x, -inf -> is_fpclass x, fcNegInf
+      //   fcmp one fabs(x), -inf -> is_fpclass x, ~fcNegInf & ~fcNan
+      //   fcmp one x, +inf -> is_fpclass x, ~fcNegInf & ~fcNan
+      //   fcmp one fabs(x), +inf -> is_fpclass x, ~fcInf & fcNan
+      //
+      //   fcmp ueq x, +inf -> is_fpclass x, fcPosInf|fcNan
+      //   fcmp ueq (fabs x), +inf -> is_fpclass x, fcInf|fcNan
+      //   fcmp ueq x, -inf -> is_fpclass x, fcNegInf|fcNan
+      //   fcmp ueq fabs(x), -inf -> is_fpclass x, fcNan
+      if (ConstRHS->isNegative()) {
+        Mask = ~fcNegInf & ~fcNan;
+        if (IsFabs)
+          Mask = ~fcNan;
+      } else {
+        Mask = ~fcPosInf & ~fcNan;
+        if (IsFabs)
+          Mask &= ~fcNegInf;
+      }
+
+      break;
+    }
+    case FCmpInst::FCMP_OLT:
+    case FCmpInst::FCMP_UGE: {
+      if (ConstRHS->isNegative()) // TODO
+        return {nullptr, fcNone};
+
+      // fcmp olt fabs(x), +inf -> fcFinite
+      // fcmp uge fabs(x), +inf -> ~fcFinite
+      // fcmp olt x, +inf -> fcFinite|fcNegInf
+      // fcmp uge x, +inf -> ~(fcFinite|fcNegInf)
+      Mask = fcFinite;
+      if (!IsFabs)
+        Mask |= fcNegInf;
+      break;
+    }
+    case FCmpInst::FCMP_OGE:
+    case FCmpInst::FCMP_ULT: {
+      if (ConstRHS->isNegative()) // TODO
+        return {nullptr, fcNone};
+
+      // fcmp oge fabs(x), +inf -> fcInf
+      // fcmp oge x, +inf -> fcPosInf
+      // fcmp ult fabs(x), +inf -> ~fcInf
+      // fcmp ult x, +inf -> ~fcPosInf
+      Mask = fcPosInf;
+      if (IsFabs)
+        Mask |= fcNegInf;
+      break;
+    }
+    default:
+      return {nullptr, fcNone};
+    }
+  } else if (ConstRHS->isSmallestNormalized() && !ConstRHS->isNegative()) {
+    // Match pattern that's used in __builtin_isnormal.
+    switch (Pred) {
+    case FCmpInst::FCMP_OLT:
+    case FCmpInst::FCMP_UGE: {
+      // fcmp olt x, smallest_normal -> fcNegInf|fcNegNormal|fcSubnormal|fcZero
+      // fcmp olt fabs(x), smallest_normal -> fcSubnormal|fcZero
+      // fcmp uge x, smallest_normal -> fcNan|fcPosNormal|fcPosInf
+      // fcmp uge fabs(x), smallest_normal -> ~(fcSubnormal|fcZero)
+      Mask = fcZero | fcSubnormal;
+      if (!IsFabs)
+        Mask |= fcNegNormal | fcNegInf;
+
+      break;
+    }
+    case FCmpInst::FCMP_OGE:
+    case FCmpInst::FCMP_ULT: {
+      // fcmp oge x, smallest_normal -> fcPosNormal | fcPosInf
+      // fcmp oge fabs(x), smallest_normal -> fcInf | fcNormal
+      // fcmp ult x, smallest_normal -> ~(fcPosNormal | fcPosInf)
+      // fcmp ult fabs(x), smallest_normal -> ~(fcInf | fcNormal)
+      Mask = fcPosInf | fcPosNormal;
+      if (IsFabs)
+        Mask |= fcNegInf | fcNegNormal;
+      break;
+    }
+    default:
+      return {nullptr, fcNone};
+    }
+  } else
+    return {nullptr, fcNone};
+
+  // Invert the comparison for the unordered cases.
+  if (FCmpInst::isUnordered(Pred))
+    Mask = ~Mask;
+
+  return {Src, Mask};
+}
+
+static FPClassTest computeKnownFPClassFromAssumes(const Value *V,
+                                                  const Query &Q) {
+  FPClassTest KnownFromAssume = fcNone;
+
+  // Try to restrict the floating-point classes based on information from
+  // assumptions.
+  for (auto &AssumeVH : Q.AC->assumptionsFor(V)) {
+    if (!AssumeVH)
+      continue;
+    CallInst *I = cast<CallInst>(AssumeVH);
+    const Function *F = I->getFunction();
+
+    assert(F == Q.CxtI->getParent()->getParent() &&
+           "Got assumption for the wrong function!");
+    assert(I->getCalledFunction()->getIntrinsicID() == Intrinsic::assume &&
+           "must be an assume intrinsic");
+
+    if (!isValidAssumeForContext(I, Q.CxtI, Q.DT))
+      continue;
+
+    CmpInst::Predicate Pred;
+    Value *LHS, *RHS;
+    uint64_t ClassVal = 0;
+    if (match(I->getArgOperand(0), m_FCmp(Pred, m_Value(LHS), m_Value(RHS)))) {
+      auto [TestedValue, TestedMask] =
+          fcmpToClassTest(Pred, *F, LHS, RHS, true);
+      // First see if we can fold in fabs/fneg into the test.
+      if (TestedValue == V)
+        KnownFromAssume |= TestedMask;
+      else {
+        // Try again without the lookthrough if we found a different source
+        // value.
+        auto [TestedValue, TestedMask] =
+            fcmpToClassTest(Pred, *F, LHS, RHS, false);
+        if (TestedValue == V)
+          KnownFromAssume |= TestedMask;
+      }
+    } else if (match(I->getArgOperand(0),
+                     m_Intrinsic<Intrinsic::is_fpclass>(
+                         m_Value(LHS), m_ConstantInt(ClassVal)))) {
+      KnownFromAssume |= static_cast<FPClassTest>(ClassVal);
+    }
+  }
+
+  return KnownFromAssume;
+}
+
+// TODO: Merge implementations of isKnownNeverNaN, isKnownNeverInfinity,
+// CannotBeNegativeZero, cannotBeOrderedLessThanZero into here.
+void computeKnownFPClass(const Value *V, const APInt &DemandedElts,
+                         FPClassTest InterestedClasses, KnownFPClass &Known,
+                         unsigned Depth, const Query &Q,
+                         const TargetLibraryInfo *TLI) {
+  if (!DemandedElts) {
+    // No demanded elts, better to assume we don't know anything.
+    Known.resetAll();
+    return;
+  }
+
+  assert(Depth <= MaxAnalysisRecursionDepth && "Limit Search Depth");
+
+  const APFloat *C;
+  if (match(V, m_APFloat(C))) {
+    // We know all of the classes for a scalar constant or a splat vector
+    // constant!
+    Known.KnownFPClasses = C->classify();
+    Known.SignBit = C->isNegative();
+    return;
+  }
+
+  FPClassTest KnownNotFromFlags = fcNone;
+  if (const auto *CB = dyn_cast<CallBase>(V))
+    KnownNotFromFlags |= CB->getRetNoFPClass();
+  else if (const auto *Arg = dyn_cast<Argument>(V))
+    KnownNotFromFlags |= Arg->getNoFPClass();
+
+  const Operator *Op = dyn_cast<Operator>(V);
+  if (const FPMathOperator *FPOp = dyn_cast_or_null<FPMathOperator>(Op)) {
+    if (FPOp->hasNoNaNs())
+      KnownNotFromFlags |= fcNan;
+    if (FPOp->hasNoInfs())
+      KnownNotFromFlags |= fcInf;
+  }
+
+  if (Q.AC)
+    KnownNotFromFlags |= computeKnownFPClassFromAssumes(V, Q);
+
+  // We no longer need to find out about these bits from inputs if we can
+  // assume this from flags/attributes.
+  InterestedClasses &= ~KnownNotFromFlags;
+
+  auto ClearClassesFromFlags = make_scope_exit([=, &Known] {
+    Known.knownNot(KnownNotFromFlags);
+  });
+
+  if (!Op)
+    return;
+
+  // All recursive calls that increase depth must come after this.
+  if (Depth == MaxAnalysisRecursionDepth)
+    return;
+
+  switch (Op->getOpcode()) {
+  case Instruction::FNeg: {
+    computeKnownFPClass(Op->getOperand(0), DemandedElts, InterestedClasses,
+                        Known, Depth + 1, Q, TLI);
+    Known.fneg();
+    break;
+  }
+  case Instruction::Select: {
+    KnownFPClass Known2;
+    computeKnownFPClass(Op->getOperand(1), DemandedElts, InterestedClasses,
+                        Known, Depth + 1, Q, TLI);
+    computeKnownFPClass(Op->getOperand(2), DemandedElts, InterestedClasses,
+                        Known2, Depth + 1, Q, TLI);
+    Known |= Known2;
+    break;
+  }
+  case Instruction::Call: {
+    if (const IntrinsicInst *II = dyn_cast<IntrinsicInst>(Op)) {
+      switch (II->getIntrinsicID()) {
+      case Intrinsic::fabs:
+        computeKnownFPClass(II->getArgOperand(0), DemandedElts,
+                            InterestedClasses, Known, Depth + 1, Q, TLI);
+        Known.fabs();
+        break;
+      case Intrinsic::copysign: {
+        KnownFPClass KnownSign;
+
+        computeKnownFPClass(II->getArgOperand(0), DemandedElts,
+                            InterestedClasses, Known, Depth + 1, Q, TLI);
+        computeKnownFPClass(II->getArgOperand(1), DemandedElts,
+                            InterestedClasses, KnownSign, Depth + 1, Q, TLI);
+        Known.copysign(KnownSign);
+        break;
+      }
+      default:
+        break;
+      }
+    }
+
+    break;
+  }
+  case Instruction::SIToFP:
+  case Instruction::UIToFP: {
+    // Cannot produce nan
+    Known.knownNot(fcNan);
+    if (Op->getOpcode() == Instruction::UIToFP)
+      Known.signBitIsZero();
+
+    if (InterestedClasses & fcInf) {
+      // Get width of largest magnitude integer (remove a bit if signed).
+      // This still works for a signed minimum value because the largest FP
+      // value is scaled by some fraction close to 2.0 (1.0 + 0.xxxx).
+      int IntSize = Op->getOperand(0)->getType()->getScalarSizeInBits();
+      if (Op->getOpcode() == Instruction::SIToFP)
+        --IntSize;
+
+      // If the exponent of the largest finite FP value can hold the largest
+      // integer, the result of the cast must be finite.
+      Type *FPTy = Op->getType()->getScalarType();
+      if (ilogb(APFloat::getLargest(FPTy->getFltSemantics())) >= IntSize)
+        Known.knownNot(fcInf);
+    }
+
+    break;
+  }
+  default:
+    break;
+  }
+
+  // TODO: Handle assumes
+}
+
+KnownFPClass llvm::computeKnownFPClass(
+    const Value *V, const APInt &DemandedElts, const DataLayout &DL,
+    FPClassTest InterestedClasses, unsigned Depth, const TargetLibraryInfo *TLI,
+    AssumptionCache *AC, const Instruction *CxtI, const DominatorTree *DT,
+    OptimizationRemarkEmitter *ORE, bool UseInstrInfo) {
+  KnownFPClass KnownClasses;
+  ::computeKnownFPClass(V, DemandedElts, InterestedClasses, KnownClasses, Depth,
+                        Query(DL, AC, safeCxtI(V, CxtI), DT, UseInstrInfo, ORE),
+                        TLI);
+  return KnownClasses;
+}
+
+KnownFPClass
+llvm::computeKnownFPClass(const Value *V, const DataLayout &DL,
+                          FPClassTest InterestedClasses, unsigned Depth,
+                          const TargetLibraryInfo *TLI, AssumptionCache *AC,
+                          const Instruction *CxtI, const DominatorTree *DT,
+                          OptimizationRemarkEmitter *ORE, bool UseInstrInfo) {
+  KnownFPClass Known;
+  auto *FVTy = dyn_cast<FixedVectorType>(V->getType());
+  APInt DemandedElts =
+      FVTy ? APInt::getAllOnes(FVTy->getNumElements()) : APInt(1, 1);
+  ::computeKnownFPClass(V, DemandedElts, InterestedClasses, Known, Depth,
+                        Query(DL, AC, safeCxtI(V, CxtI), DT, UseInstrInfo, ORE),
+                        TLI);
+  return Known;
+}
+
 Value *llvm::isBytewiseValue(Value *V, const DataLayout &DL) {
 
   // All byte-wide stores are splatable, even of arbitrary variables.
@@ -7354,67 +7728,66 @@ static void setLimitsForBinOp(const BinaryOperator &BO, APInt &Lower,
   }
 }
 
-static void setLimitsForIntrinsic(const IntrinsicInst &II, APInt &Lower,
-                                  APInt &Upper) {
-  unsigned Width = Lower.getBitWidth();
+static ConstantRange getRangeForIntrinsic(const IntrinsicInst &II) {
+  unsigned Width = II.getType()->getScalarSizeInBits();
   const APInt *C;
   switch (II.getIntrinsicID()) {
   case Intrinsic::ctpop:
   case Intrinsic::ctlz:
   case Intrinsic::cttz:
     // Maximum of set/clear bits is the bit width.
-    assert(Lower == 0 && "Expected lower bound to be zero");
-    Upper = Width + 1;
-    break;
+    return ConstantRange(APInt::getZero(Width), APInt(Width, Width + 1));
   case Intrinsic::uadd_sat:
     // uadd.sat(x, C) produces [C, UINT_MAX].
     if (match(II.getOperand(0), m_APInt(C)) ||
         match(II.getOperand(1), m_APInt(C)))
-      Lower = *C;
+      return ConstantRange::getNonEmpty(*C, APInt::getZero(Width));
     break;
   case Intrinsic::sadd_sat:
     if (match(II.getOperand(0), m_APInt(C)) ||
         match(II.getOperand(1), m_APInt(C))) {
-      if (C->isNegative()) {
+      if (C->isNegative())
         // sadd.sat(x, -C) produces [SINT_MIN, SINT_MAX + (-C)].
-        Lower = APInt::getSignedMinValue(Width);
-        Upper = APInt::getSignedMaxValue(Width) + *C + 1;
-      } else {
-        // sadd.sat(x, +C) produces [SINT_MIN + C, SINT_MAX].
-        Lower = APInt::getSignedMinValue(Width) + *C;
-        Upper = APInt::getSignedMaxValue(Width) + 1;
-      }
+        return ConstantRange::getNonEmpty(APInt::getSignedMinValue(Width),
+                                          APInt::getSignedMaxValue(Width) + *C +
+                                              1);
+
+      // sadd.sat(x, +C) produces [SINT_MIN + C, SINT_MAX].
+      return ConstantRange::getNonEmpty(APInt::getSignedMinValue(Width) + *C,
+                                        APInt::getSignedMaxValue(Width) + 1);
     }
     break;
   case Intrinsic::usub_sat:
     // usub.sat(C, x) produces [0, C].
     if (match(II.getOperand(0), m_APInt(C)))
-      Upper = *C + 1;
+      return ConstantRange::getNonEmpty(APInt::getZero(Width), *C + 1);
+
     // usub.sat(x, C) produces [0, UINT_MAX - C].
-    else if (match(II.getOperand(1), m_APInt(C)))
-      Upper = APInt::getMaxValue(Width) - *C + 1;
+    if (match(II.getOperand(1), m_APInt(C)))
+      return ConstantRange::getNonEmpty(APInt::getZero(Width),
+                                        APInt::getMaxValue(Width) - *C + 1);
     break;
   case Intrinsic::ssub_sat:
     if (match(II.getOperand(0), m_APInt(C))) {
-      if (C->isNegative()) {
+      if (C->isNegative())
         // ssub.sat(-C, x) produces [SINT_MIN, -SINT_MIN + (-C)].
-        Lower = APInt::getSignedMinValue(Width);
-        Upper = *C - APInt::getSignedMinValue(Width) + 1;
-      } else {
-        // ssub.sat(+C, x) produces [-SINT_MAX + C, SINT_MAX].
-        Lower = *C - APInt::getSignedMaxValue(Width);
-        Upper = APInt::getSignedMaxValue(Width) + 1;
-      }
+        return ConstantRange::getNonEmpty(APInt::getSignedMinValue(Width),
+                                          *C - APInt::getSignedMinValue(Width) +
+                                              1);
+
+      // ssub.sat(+C, x) produces [-SINT_MAX + C, SINT_MAX].
+      return ConstantRange::getNonEmpty(*C - APInt::getSignedMaxValue(Width),
+                                        APInt::getSignedMaxValue(Width) + 1);
     } else if (match(II.getOperand(1), m_APInt(C))) {
-      if (C->isNegative()) {
+      if (C->isNegative())
         // ssub.sat(x, -C) produces [SINT_MIN - (-C), SINT_MAX]:
-        Lower = APInt::getSignedMinValue(Width) - *C;
-        Upper = APInt::getSignedMaxValue(Width) + 1;
-      } else {
-        // ssub.sat(x, +C) produces [SINT_MIN, SINT_MAX - C].
-        Lower = APInt::getSignedMinValue(Width);
-        Upper = APInt::getSignedMaxValue(Width) - *C + 1;
-      }
+        return ConstantRange::getNonEmpty(APInt::getSignedMinValue(Width) - *C,
+                                          APInt::getSignedMaxValue(Width) + 1);
+
+      // ssub.sat(x, +C) produces [SINT_MIN, SINT_MAX - C].
+      return ConstantRange::getNonEmpty(APInt::getSignedMinValue(Width),
+                                        APInt::getSignedMaxValue(Width) - *C +
+                                            1);
     }
     break;
   case Intrinsic::umin:
@@ -7427,19 +7800,15 @@ static void setLimitsForIntrinsic(const IntrinsicInst &II, APInt &Lower,
 
     switch (II.getIntrinsicID()) {
     case Intrinsic::umin:
-      Upper = *C + 1;
-      break;
+      return ConstantRange::getNonEmpty(APInt::getZero(Width), *C + 1);
     case Intrinsic::umax:
-      Lower = *C;
-      break;
+      return ConstantRange::getNonEmpty(*C, APInt::getZero(Width));
     case Intrinsic::smin:
-      Lower = APInt::getSignedMinValue(Width);
-      Upper = *C + 1;
-      break;
+      return ConstantRange::getNonEmpty(APInt::getSignedMinValue(Width),
+                                        *C + 1);
     case Intrinsic::smax:
-      Lower = *C;
-      Upper = APInt::getSignedMaxValue(Width) + 1;
-      break;
+      return ConstantRange::getNonEmpty(*C,
+                                        APInt::getSignedMaxValue(Width) + 1);
     default:
       llvm_unreachable("Must be min/max intrinsic");
     }
@@ -7448,13 +7817,20 @@ static void setLimitsForIntrinsic(const IntrinsicInst &II, APInt &Lower,
     // If abs of SIGNED_MIN is poison, then the result is [0..SIGNED_MAX],
     // otherwise it is [0..SIGNED_MIN], as -SIGNED_MIN == SIGNED_MIN.
     if (match(II.getOperand(1), m_One()))
-      Upper = APInt::getSignedMaxValue(Width) + 1;
-    else
-      Upper = APInt::getSignedMinValue(Width) + 1;
-    break;
+      return ConstantRange(APInt::getZero(Width),
+                           APInt::getSignedMaxValue(Width) + 1);
+
+    return ConstantRange(APInt::getZero(Width),
+                         APInt::getSignedMinValue(Width) + 1);
+  case Intrinsic::vscale:
+    if (!II.getParent() || !II.getFunction())
+      break;
+    return getVScaleRange(II.getFunction(), Width);
   default:
     break;
   }
+
+  return ConstantRange::getFull(Width);
 }
 
 static void setLimitsForSelectPattern(const SelectInst &SI, APInt &Lower,
@@ -7543,18 +7919,28 @@ ConstantRange llvm::computeConstantRange(const Value *V, bool ForSigned,
 
   InstrInfoQuery IIQ(UseInstrInfo);
   unsigned BitWidth = V->getType()->getScalarSizeInBits();
-  APInt Lower = APInt(BitWidth, 0);
-  APInt Upper = APInt(BitWidth, 0);
-  if (auto *BO = dyn_cast<BinaryOperator>(V))
+  ConstantRange CR = ConstantRange::getFull(BitWidth);
+  if (auto *BO = dyn_cast<BinaryOperator>(V)) {
+    APInt Lower = APInt(BitWidth, 0);
+    APInt Upper = APInt(BitWidth, 0);
+    // TODO: Return ConstantRange.
     setLimitsForBinOp(*BO, Lower, Upper, IIQ, ForSigned);
-  else if (auto *II = dyn_cast<IntrinsicInst>(V))
-    setLimitsForIntrinsic(*II, Lower, Upper);
-  else if (auto *SI = dyn_cast<SelectInst>(V))
+    CR = ConstantRange::getNonEmpty(Lower, Upper);
+  } else if (auto *II = dyn_cast<IntrinsicInst>(V))
+    CR = getRangeForIntrinsic(*II);
+  else if (auto *SI = dyn_cast<SelectInst>(V)) {
+    APInt Lower = APInt(BitWidth, 0);
+    APInt Upper = APInt(BitWidth, 0);
+    // TODO: Return ConstantRange.
     setLimitsForSelectPattern(*SI, Lower, Upper, IIQ);
-  else if (isa<FPToUIInst>(V) || isa<FPToSIInst>(V))
+    CR = ConstantRange::getNonEmpty(Lower, Upper);
+  } else if (isa<FPToUIInst>(V) || isa<FPToSIInst>(V)) {
+    APInt Lower = APInt(BitWidth, 0);
+    APInt Upper = APInt(BitWidth, 0);
+    // TODO: Return ConstantRange.
     setLimitForFPToI(cast<Instruction>(V), Lower, Upper);
-
-  ConstantRange CR = ConstantRange::getNonEmpty(Lower, Upper);
+    CR = ConstantRange::getNonEmpty(Lower, Upper);
+  }
 
   if (auto *I = dyn_cast<Instruction>(V))
     if (auto *Range = IIQ.getMetadata(I, LLVMContext::MD_range))
diff --git a/llvm/lib/AsmParser/LLParser.cpp b/llvm/lib/AsmParser/LLParser.cpp
index 673a34ef7f181..8ee1b665c9b76 100644
--- a/llvm/lib/AsmParser/LLParser.cpp
+++ b/llvm/lib/AsmParser/LLParser.cpp
@@ -3330,6 +3330,12 @@ Value *LLParser::PerFunctionState::getVal(const std::string &Name, Type *Ty,
   } else {
     FwdVal = new Argument(Ty, Name);
   }
+  if (FwdVal->getName() != Name) {
+    P.error(Loc, "name is too long which can result in name collisions, "
+                 "consider making the name shorter or "
+                 "increasing -non-global-value-max-name-size");
+    return nullptr;
+  }
 
   ForwardRefVals[Name] = std::make_pair(FwdVal, Loc);
   return FwdVal;
@@ -3876,6 +3882,8 @@ bool LLParser::parseValID(ValID &ID, PerFunctionState *PFS, Type *ExpectedTy) {
     return error(ID.Loc, "frem constexprs are no longer supported");
   case lltok::kw_fneg:
     return error(ID.Loc, "fneg constexprs are no longer supported");
+  case lltok::kw_select:
+    return error(ID.Loc, "select constexprs are no longer supported");
   case lltok::kw_icmp:
   case lltok::kw_fcmp: {
     unsigned PredVal, Opc = Lex.getUIntVal();
@@ -4005,8 +4013,7 @@ bool LLParser::parseValID(ValID &ID, PerFunctionState *PFS, Type *ExpectedTy) {
   case lltok::kw_getelementptr:
   case lltok::kw_shufflevector:
   case lltok::kw_insertelement:
-  case lltok::kw_extractelement:
-  case lltok::kw_select: {
+  case lltok::kw_extractelement: {
     unsigned Opc = Lex.getUIntVal();
     SmallVector<Constant*, 16> Elts;
     bool InBounds = false;
@@ -4085,13 +4092,6 @@ bool LLParser::parseValID(ValID &ID, PerFunctionState *PFS, Type *ExpectedTy) {
 
       ID.ConstantVal = ConstantExpr::getGetElementPtr(Ty, Elts[0], Indices,
                                                       InBounds, InRangeOp);
-    } else if (Opc == Instruction::Select) {
-      if (Elts.size() != 3)
-        return error(ID.Loc, "expected three operands to select");
-      if (const char *Reason = SelectInst::areInvalidOperands(Elts[0], Elts[1],
-                                                              Elts[2]))
-        return error(ID.Loc, Reason);
-      ID.ConstantVal = ConstantExpr::getSelect(Elts[0], Elts[1], Elts[2]);
     } else if (Opc == Instruction::ShuffleVector) {
       if (Elts.size() != 3)
         return error(ID.Loc, "expected three operands to shufflevector");
diff --git a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
index 2c1f195119bcc..64616240d4a3c 100644
--- a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
+++ b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
@@ -1399,7 +1399,9 @@ unsigned BitcodeReader::getVirtualTypeID(Type *Ty,
   return TypeID;
 }
 
-static bool isConstExprSupported(uint8_t Opcode) {
+static bool isConstExprSupported(const BitcodeConstant *BC) {
+  uint8_t Opcode = BC->Opcode;
+
   // These are not real constant expressions, always consider them supported.
   if (Opcode >= BitcodeConstant::FirstSpecialOpcode)
     return true;
@@ -1412,7 +1414,16 @@ static bool isConstExprSupported(uint8_t Opcode) {
   if (Instruction::isBinaryOp(Opcode))
     return ConstantExpr::isSupportedBinOp(Opcode);
 
-  return Opcode != Instruction::FNeg;
+  if (Opcode == Instruction::GetElementPtr)
+    return ConstantExpr::isSupportedGetElementPtr(BC->SrcElemTy);
+
+  switch (Opcode) {
+  case Instruction::FNeg:
+  case Instruction::Select:
+    return false;
+  default:
+    return true;
+  }
 }
 
 Expected<Value *> BitcodeReader::materializeValue(unsigned StartValID,
@@ -1467,7 +1478,7 @@ Expected<Value *> BitcodeReader::materializeValue(unsigned StartValID,
         ConstOps.push_back(C);
 
     // Materialize as constant expression if possible.
-    if (isConstExprSupported(BC->Opcode) && ConstOps.size() == Ops.size()) {
+    if (isConstExprSupported(BC) && ConstOps.size() == Ops.size()) {
       Constant *C;
       if (Instruction::isCast(BC->Opcode)) {
         C = UpgradeBitCastExpr(BC->Opcode, ConstOps[0], BC->getType());
@@ -1544,9 +1555,6 @@ Expected<Value *> BitcodeReader::materializeValue(unsigned StartValID,
                                              ArrayRef(ConstOps).drop_front(),
                                              BC->Flags, BC->getInRangeIndex());
           break;
-        case Instruction::Select:
-          C = ConstantExpr::getSelect(ConstOps[0], ConstOps[1], ConstOps[2]);
-          break;
         case Instruction::ExtractElement:
           C = ConstantExpr::getExtractElement(ConstOps[0], ConstOps[1]);
           break;
diff --git a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
index 51756eb8ecb06..a368ac303bd91 100644
--- a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -2676,12 +2676,6 @@ void ModuleBitcodeWriter::writeConstants(unsigned FirstVal, unsigned LastVal,
         }
         break;
       }
-      case Instruction::Select:
-        Code = bitc::CST_CODE_CE_SELECT;
-        Record.push_back(VE.getValueID(C->getOperand(0)));
-        Record.push_back(VE.getValueID(C->getOperand(1)));
-        Record.push_back(VE.getValueID(C->getOperand(2)));
-        break;
       case Instruction::ExtractElement:
         Code = bitc::CST_CODE_CE_EXTRACTELT;
         Record.push_back(VE.getTypeID(C->getOperand(0)->getType()));
diff --git a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
index 4fbe1f4da9b10..c8a88a11a1ef3 100644
--- a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
@@ -1365,7 +1365,7 @@ void AsmPrinter::emitBBAddrMapSection(const MachineFunction &MF) {
   OutStreamer->AddComment("number of basic blocks");
   OutStreamer->emitULEB128IntValue(MF.size());
   const MCSymbol *PrevMBBEndSymbol = FunctionSymbol;
-  // Emit BB Information for each basic block in the funciton.
+  // Emit BB Information for each basic block in the function.
   for (const MachineBasicBlock &MBB : MF) {
     const MCSymbol *MBBSymbol =
         MBB.isEntryBlock() ? FunctionSymbol : MBB.getSymbol();
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
index 5542c560a490c..86cdd11ebd17e 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
@@ -569,7 +569,6 @@ static SourceLanguage MapDWLangToCVLang(unsigned DWLang) {
   case dwarf::DW_LANG_C89:
   case dwarf::DW_LANG_C99:
   case dwarf::DW_LANG_C11:
-  case dwarf::DW_LANG_ObjC:
     return SourceLanguage::C;
   case dwarf::DW_LANG_C_plus_plus:
   case dwarf::DW_LANG_C_plus_plus_03:
@@ -595,6 +594,10 @@ static SourceLanguage MapDWLangToCVLang(unsigned DWLang) {
     return SourceLanguage::Swift;
   case dwarf::DW_LANG_Rust:
     return SourceLanguage::Rust;
+  case dwarf::DW_LANG_ObjC:
+    return SourceLanguage::ObjC;
+  case dwarf::DW_LANG_ObjC_plus_plus:
+    return SourceLanguage::ObjCpp;
   default:
     // There's no CodeView representation for this language, and CV doesn't
     // have an "unknown" option for the language field, so we'll use MASM,
@@ -3405,7 +3408,7 @@ void CodeViewDebug::emitDebugInfoForGlobal(const CVGlobalVariable &CVGV) {
     OS.AddComment("DataOffset");
 
     uint64_t Offset = 0;
-    if (CVGlobalVariableOffsets.find(DIGV) != CVGlobalVariableOffsets.end())
+    if (CVGlobalVariableOffsets.contains(DIGV))
       // Use the offset seen while collecting info on globals.
       Offset = CVGlobalVariableOffsets[DIGV];
     OS.emitCOFFSecRel32(GVSym, Offset);
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfCompileUnit.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfCompileUnit.cpp
index 4c6a5bb5bf0dd..f336eae12518f 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfCompileUnit.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfCompileUnit.cpp
@@ -278,7 +278,16 @@ void DwarfCompileUnit::addLocationAttribute(
                    : FormAndOp{dwarf::DW_FORM_data8, dwarf::DW_OP_const8u};
       };
       if (Global->isThreadLocal()) {
-        if (Asm->TM.useEmulatedTLS()) {
+        if (Asm->TM.getTargetTriple().isWasm()) {
+          // FIXME This is not guaranteed, but in practice, in static linking,
+          // if present, __tls_base's index is 1. This doesn't hold for dynamic
+          // linking, so TLS variables used in dynamic linking won't have
+          // correct debug info for now. See
+          // https://github.com/llvm/llvm-project/blob/19afbfe33156d211fa959dadeea46cd17b9c723c/lld/wasm/Driver.cpp#L786-L823
+          addWasmRelocBaseGlobal(Loc, "__tls_base", 1);
+          addOpAddress(*Loc, Sym);
+          addUInt(*Loc, dwarf::DW_FORM_data1, dwarf::DW_OP_plus);
+        } else if (Asm->TM.useEmulatedTLS()) {
           // TODO: add debug info for emulated thread local mode.
         } else {
           // FIXME: Make this work with -gsplit-dwarf.
@@ -301,6 +310,14 @@ void DwarfCompileUnit::addLocationAttribute(
                   DD->useGNUTLSOpcode() ? dwarf::DW_OP_GNU_push_tls_address
                                         : dwarf::DW_OP_form_tls_address);
         }
+      } else if (Asm->TM.getTargetTriple().isWasm() &&
+                 Asm->TM.getRelocationModel() == Reloc::PIC_) {
+        // FIXME This is not guaranteed, but in practice, if present,
+        // __memory_base's index is 1. See
+        // https://github.com/llvm/llvm-project/blob/19afbfe33156d211fa959dadeea46cd17b9c723c/lld/wasm/Driver.cpp#L786-L823
+        addWasmRelocBaseGlobal(Loc, "__memory_base", 1);
+        addOpAddress(*Loc, Sym);
+        addUInt(*Loc, dwarf::DW_FORM_data1, dwarf::DW_OP_plus);
       } else if ((Asm->TM.getRelocationModel() == Reloc::RWPI ||
                   Asm->TM.getRelocationModel() == Reloc::ROPI_RWPI) &&
                  !Asm->getObjFileLowering()
@@ -449,6 +466,39 @@ DIE &DwarfCompileUnit::updateSubprogramScopeDIE(const DISubprogram *SP) {
   return ContextCU->updateSubprogramScopeDIEImpl(SP, SPDie);
 }
 
+// Add info for Wasm-global-based relocation.
+// 'GlobalIndex' is used for split dwarf, which currently relies on a few
+// assumptions that are not guaranteed in a formal way but work in practice.
+void DwarfCompileUnit::addWasmRelocBaseGlobal(DIELoc *Loc, StringRef GlobalName,
+                                              uint64_t GlobalIndex) {
+  // FIXME: duplicated from Target/WebAssembly/WebAssembly.h
+  // don't want to depend on target specific headers in this code?
+  const unsigned TI_GLOBAL_RELOC = 3;
+  unsigned PointerSize = Asm->getDataLayout().getPointerSize();
+  auto *Sym = cast<MCSymbolWasm>(Asm->GetExternalSymbolSymbol(GlobalName));
+  // FIXME: this repeats what WebAssemblyMCInstLower::
+  // GetExternalSymbolSymbol does, since if there's no code that
+  // refers to this symbol, we have to set it here.
+  Sym->setType(wasm::WASM_SYMBOL_TYPE_GLOBAL);
+  Sym->setGlobalType(wasm::WasmGlobalType{
+      static_cast<uint8_t>(PointerSize == 4 ? wasm::WASM_TYPE_I32
+                                            : wasm::WASM_TYPE_I64),
+      true});
+  addUInt(*Loc, dwarf::DW_FORM_data1, dwarf::DW_OP_WASM_location);
+  addSInt(*Loc, dwarf::DW_FORM_sdata, TI_GLOBAL_RELOC);
+  if (!isDwoUnit()) {
+    addLabel(*Loc, dwarf::DW_FORM_data4, Sym);
+  } else {
+    // FIXME: when writing dwo, we need to avoid relocations. Probably
+    // the "right" solution is to treat globals the way func and data
+    // symbols are (with entries in .debug_addr).
+    // For now we hardcode the indices in the callsites. Global indices are not
+    // fixed, but in practice a few are fixed; for example, __stack_pointer is
+    // always index 0.
+    addUInt(*Loc, dwarf::DW_FORM_data4, GlobalIndex);
+  }
+}
+
 DIE &DwarfCompileUnit::updateSubprogramScopeDIEImpl(const DISubprogram *SP,
                                                     DIE *SPDie) {
   SmallVector<RangeSpan, 2> BB_List;
@@ -485,35 +535,14 @@ DIE &DwarfCompileUnit::updateSubprogramScopeDIEImpl(const DISubprogram *SP,
     }
     case TargetFrameLowering::DwarfFrameBase::WasmFrameBase: {
       // FIXME: duplicated from Target/WebAssembly/WebAssembly.h
-      // don't want to depend on target specific headers in this code?
       const unsigned TI_GLOBAL_RELOC = 3;
       if (FrameBase.Location.WasmLoc.Kind == TI_GLOBAL_RELOC) {
         // These need to be relocatable.
-        assert(FrameBase.Location.WasmLoc.Index == 0);  // Only SP so far.
-        auto SPSym = cast<MCSymbolWasm>(
-          Asm->GetExternalSymbolSymbol("__stack_pointer"));
-        // FIXME: this repeats what WebAssemblyMCInstLower::
-        // GetExternalSymbolSymbol does, since if there's no code that
-        // refers to this symbol, we have to set it here.
-        SPSym->setType(wasm::WASM_SYMBOL_TYPE_GLOBAL);
-        SPSym->setGlobalType(wasm::WasmGlobalType{
-            uint8_t(Asm->getSubtargetInfo().getTargetTriple().getArch() ==
-                            Triple::wasm64
-                        ? wasm::WASM_TYPE_I64
-                        : wasm::WASM_TYPE_I32),
-            true});
         DIELoc *Loc = new (DIEValueAllocator) DIELoc;
-        addUInt(*Loc, dwarf::DW_FORM_data1, dwarf::DW_OP_WASM_location);
-        addSInt(*Loc, dwarf::DW_FORM_sdata, TI_GLOBAL_RELOC);
-        if (!isDwoUnit()) {
-          addLabel(*Loc, dwarf::DW_FORM_data4, SPSym);
-        } else {
-          // FIXME: when writing dwo, we need to avoid relocations. Probably
-          // the "right" solution is to treat globals the way func and data
-          // symbols are (with entries in .debug_addr).
-          // For now, since we only ever use index 0, this should work as-is.
-          addUInt(*Loc, dwarf::DW_FORM_data4, FrameBase.Location.WasmLoc.Index);
-        }
+        assert(FrameBase.Location.WasmLoc.Index == 0); // Only SP so far.
+        // For now, since we only ever use index 0, this should work as-is.
+        addWasmRelocBaseGlobal(Loc, "__stack_pointer",
+                               FrameBase.Location.WasmLoc.Index);
         addUInt(*Loc, dwarf::DW_FORM_data1, dwarf::DW_OP_stack_value);
         addBlock(*SPDie, dwarf::DW_AT_frame_base, Loc);
       } else {
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfCompileUnit.h b/llvm/lib/CodeGen/AsmPrinter/DwarfCompileUnit.h
index 7d87f35021bb4..5581b9144bab7 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfCompileUnit.h
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfCompileUnit.h
@@ -108,6 +108,10 @@ class DwarfCompileUnit final : public DwarfUnit {
 
   void finishNonUnitTypeDIE(DIE& D, const DICompositeType *CTy) override;
 
+  /// Add info for Wasm-global-based relocation.
+  void addWasmRelocBaseGlobal(DIELoc *Loc, StringRef GlobalName,
+                              uint64_t GlobalIndex);
+
 public:
   DwarfCompileUnit(unsigned UID, const DICompileUnit *Node, AsmPrinter *A,
                    DwarfDebug *DW, DwarfFile *DWU,
diff --git a/llvm/lib/CodeGen/AsmPrinter/EHStreamer.cpp b/llvm/lib/CodeGen/AsmPrinter/EHStreamer.cpp
index 67e2c0e070958..eef6b1d93f36e 100644
--- a/llvm/lib/CodeGen/AsmPrinter/EHStreamer.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/EHStreamer.cpp
@@ -410,7 +410,7 @@ MCSymbol *EHStreamer::emitExceptionTable() {
   computeActionsTable(LandingPads, Actions, FirstActions);
 
   // Compute the call-site table and call-site ranges. Normally, there is only
-  // one call-site-range which covers the whole funciton. With
+  // one call-site-range which covers the whole function. With
   // -basic-block-sections, there is one call-site-range per basic block
   // section.
   SmallVector<CallSiteEntry, 64> CallSites;
diff --git a/llvm/lib/CodeGen/AsmPrinter/PseudoProbePrinter.cpp b/llvm/lib/CodeGen/AsmPrinter/PseudoProbePrinter.cpp
index 3e75b4371033e..0540a4c2ae02a 100644
--- a/llvm/lib/CodeGen/AsmPrinter/PseudoProbePrinter.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/PseudoProbePrinter.cpp
@@ -32,11 +32,7 @@ void PseudoProbeHandler::emitPseudoProbe(uint64_t Guid, uint64_t Index,
   SmallVector<InlineSite, 8> ReversedInlineStack;
   auto *InlinedAt = DebugLoc ? DebugLoc->getInlinedAt() : nullptr;
   while (InlinedAt) {
-    const DISubprogram *SP = InlinedAt->getScope()->getSubprogram();
-    // Use linkage name for C++ if possible.
-    auto Name = SP->getLinkageName();
-    if (Name.empty())
-      Name = SP->getName();
+    auto Name = InlinedAt->getSubprogramLinkageName();
     // Use caching to avoid redundant md5 computation for build speed.
     uint64_t &CallerGuid = NameGuidMap[Name];
     if (!CallerGuid)
diff --git a/llvm/lib/CodeGen/AssignmentTrackingAnalysis.cpp b/llvm/lib/CodeGen/AssignmentTrackingAnalysis.cpp
index d9458c688ce5f..bda6ec9d3d042 100644
--- a/llvm/lib/CodeGen/AssignmentTrackingAnalysis.cpp
+++ b/llvm/lib/CodeGen/AssignmentTrackingAnalysis.cpp
@@ -105,23 +105,23 @@ class FunctionVarLocsBuilder {
 
   /// Add a def for a variable that is valid for its lifetime.
   void addSingleLocVar(DebugVariable Var, DIExpression *Expr, DebugLoc DL,
-                       Value *V) {
+                       RawLocationWrapper R) {
     VarLocInfo VarLoc;
     VarLoc.VariableID = insertVariable(Var);
     VarLoc.Expr = Expr;
     VarLoc.DL = DL;
-    VarLoc.V = V;
+    VarLoc.Values = R;
     SingleLocVars.emplace_back(VarLoc);
   }
 
   /// Add a def to the wedge of defs just before /p Before.
   void addVarLoc(Instruction *Before, DebugVariable Var, DIExpression *Expr,
-                 DebugLoc DL, Value *V) {
+                 DebugLoc DL, RawLocationWrapper R) {
     VarLocInfo VarLoc;
     VarLoc.VariableID = insertVariable(Var);
     VarLoc.Expr = Expr;
     VarLoc.DL = DL;
-    VarLoc.V = V;
+    VarLoc.Values = R;
     VarLocsBeforeInst[Before].emplace_back(VarLoc);
   }
 };
@@ -148,7 +148,11 @@ void FunctionVarLocs::print(raw_ostream &OS, const Function &Fn) const {
 
   auto PrintLoc = [&OS](const VarLocInfo &Loc) {
     OS << "DEF Var=[" << (unsigned)Loc.VariableID << "]"
-       << " Expr=" << *Loc.Expr << " V=" << *Loc.V << "\n";
+       << " Expr=" << *Loc.Expr << " Values=(";
+    for (auto *Op : Loc.Values.location_ops()) {
+      errs() << Op->getName() << " ";
+    }
+    errs() << ")\n";
   };
 
   // Print the single location variables.
@@ -317,7 +321,7 @@ class MemLocFragmentFill {
 
   /// IDs for memory location base addresses in maps. Use 0 to indicate that
   /// there's no memory location.
-  UniqueVector<Value *> Bases;
+  UniqueVector<RawLocationWrapper> Bases;
   UniqueVector<DebugAggregate> Aggregates;
   DenseMap<const BasicBlock *, VarFragMap> LiveIn;
   DenseMap<const BasicBlock *, VarFragMap> LiveOut;
@@ -370,7 +374,7 @@ class MemLocFragmentFill {
   /// Return a string for the value that \p BaseID represents.
   std::string toString(unsigned BaseID) {
     if (BaseID)
-      return Bases[BaseID]->getName().str();
+      return Bases[BaseID].getVariableLocationOp(0)->getName().str();
     else
       return "None";
   }
@@ -603,7 +607,7 @@ class MemLocFragmentFill {
     const auto DerefOffsetInBytes = getDerefOffsetInBytes(DIExpr);
     const unsigned Base =
         DerefOffsetInBytes && *DerefOffsetInBytes * 8 == StartBit
-            ? Bases.insert(VarLoc.V)
+            ? Bases.insert(VarLoc.Values)
             : 0;
     LLVM_DEBUG(dbgs() << "DEF " << DbgVar.getVariable()->getName() << " ["
                       << StartBit << ", " << EndBit << "): " << toString(Base)
@@ -1244,10 +1248,11 @@ void AssignmentTrackingLowering::emitDbgValue(
     const DbgVariableIntrinsic *Source, Instruction *After) {
 
   DILocation *DL = Source->getDebugLoc();
-  auto Emit = [this, Source, After, DL](Value *Val, DIExpression *Expr) {
+  auto Emit = [this, Source, After, DL](Metadata *Val, DIExpression *Expr) {
     assert(Expr);
     if (!Val)
-      Val = PoisonValue::get(Type::getInt1Ty(Source->getContext()));
+      Val = ValueAsMetadata::get(
+          PoisonValue::get(Type::getInt1Ty(Source->getContext())));
 
     // Find a suitable insert point.
     Instruction *InsertBefore = After->getNextNode();
@@ -1257,7 +1262,7 @@ void AssignmentTrackingLowering::emitDbgValue(
     VarLocInfo VarLoc;
     VarLoc.VariableID = static_cast<VariableID>(Var);
     VarLoc.Expr = Expr;
-    VarLoc.V = Val;
+    VarLoc.Values = RawLocationWrapper(Val);
     VarLoc.DL = DL;
     // Insert it into the map for later.
     InsertBeforeMap[InsertBefore].push_back(VarLoc);
@@ -1286,16 +1291,13 @@ void AssignmentTrackingLowering::emitDbgValue(
       // The address-expression has an implicit deref, add it now.
       std::tie(Val, Expr) =
           walkToAllocaAndPrependOffsetDeref(Layout, Val, Expr);
-      Emit(Val, Expr);
+      Emit(ValueAsMetadata::get(Val), Expr);
       return;
     }
   }
 
   if (Kind == LocKind::Val) {
-    /// Get the value component, converting to Undef if it is variadic.
-    Value *Val =
-        Source->hasArgList() ? nullptr : Source->getVariableLocationOp(0);
-    Emit(Val, Source->getExpression());
+    Emit(Source->getRawLocation(), Source->getExpression());
     return;
   }
 
@@ -1373,7 +1375,8 @@ void AssignmentTrackingLowering::processUntaggedInstruction(
     VarLocInfo VarLoc;
     VarLoc.VariableID = static_cast<VariableID>(Var);
     VarLoc.Expr = DIE;
-    VarLoc.V = const_cast<AllocaInst *>(Info.Base);
+    VarLoc.Values = RawLocationWrapper(
+        ValueAsMetadata::get(const_cast<AllocaInst *>(Info.Base)));
     VarLoc.DL = DILoc;
     // 3. Insert it into the map for later.
     InsertBeforeMap[InsertBefore].push_back(VarLoc);
@@ -1856,7 +1859,8 @@ static AssignmentTrackingLowering::OverlapMap buildOverlapMapAndRecordDeclares(
     for (auto &I : BB) {
       if (auto *DDI = dyn_cast<DbgDeclareInst>(&I)) {
         FnVarLocs->addSingleLocVar(DebugVariable(DDI), DDI->getExpression(),
-                                   DDI->getDebugLoc(), DDI->getAddress());
+                                   DDI->getDebugLoc(),
+                                   DDI->getWrappedLocation());
       } else if (auto *DII = dyn_cast<DbgVariableIntrinsic>(&I)) {
         DebugVariable DV = DebugVariable(DII);
         DebugAggregate DA = {DV.getVariable(), DV.getInlinedAt()};
@@ -2068,14 +2072,15 @@ bool AssignmentTrackingLowering::run(FunctionVarLocsBuilder *FnVarLocsBuilder) {
       //
       // Unless we've already done so, create the single location def now.
       if (AlwaysStackHomed.insert(Aggr).second) {
-        assert(isa<AllocaInst>(VarLoc.V));
+        assert(!VarLoc.Values.hasArgList() &&
+               isa<AllocaInst>(VarLoc.Values.getVariableLocationOp(0)));
         // TODO: When more complex cases are handled VarLoc.Expr should be
         // built appropriately rather than always using an empty DIExpression.
         // The assert below is a reminder.
         assert(Simple);
         VarLoc.Expr = DIExpression::get(Fn.getContext(), std::nullopt);
         DebugVariable Var = FnVarLocs->getVariable(VarLoc.VariableID);
-        FnVarLocs->addSingleLocVar(Var, VarLoc.Expr, VarLoc.DL, VarLoc.V);
+        FnVarLocs->addSingleLocVar(Var, VarLoc.Expr, VarLoc.DL, VarLoc.Values);
         InsertedAnyIntrinsics = true;
       }
     }
@@ -2118,20 +2123,11 @@ bool AssignmentTrackingLowering::emitPromotedVarLocs(
       // already.
       if (VarsWithStackSlot->contains(getAggregate(DVI)))
         continue;
-      // Wrapper to get a single value (or undef) from DVI.
-      auto GetValue = [DVI]() -> Value * {
-        // We can't handle variadic DIExpressions yet so treat those as
-        // kill locations.
-        if (DVI->isKillLocation() || DVI->getValue() == nullptr ||
-            DVI->hasArgList())
-          return PoisonValue::get(Type::getInt32Ty(DVI->getContext()));
-        return DVI->getValue();
-      };
       Instruction *InsertBefore = I.getNextNode();
       assert(InsertBefore && "Unexpected: debug intrinsics after a terminator");
       FnVarLocs->addVarLoc(InsertBefore, DebugVariable(DVI),
                            DVI->getExpression(), DVI->getDebugLoc(),
-                           GetValue());
+                           DVI->getWrappedLocation());
       InsertedAnyIntrinsics = true;
     }
   }
@@ -2213,7 +2209,8 @@ static bool
 removeRedundantDbgLocsUsingForwardScan(const BasicBlock *BB,
                                        FunctionVarLocsBuilder &FnVarLocs) {
   bool Changed = false;
-  DenseMap<DebugVariable, std::pair<Value *, DIExpression *>> VariableMap;
+  DenseMap<DebugVariable, std::pair<RawLocationWrapper, DIExpression *>>
+      VariableMap;
 
   // Scan over the entire block, not just over the instructions mapped by
   // FnVarLocs, because wedges in FnVarLocs may only be seperated by debug
@@ -2238,9 +2235,9 @@ removeRedundantDbgLocsUsingForwardScan(const BasicBlock *BB,
 
       // Update the map if we found a new value/expression describing the
       // variable, or if the variable wasn't mapped already.
-      if (VMI == VariableMap.end() || VMI->second.first != Loc.V ||
+      if (VMI == VariableMap.end() || VMI->second.first != Loc.Values ||
           VMI->second.second != Loc.Expr) {
-        VariableMap[Key] = {Loc.V, Loc.Expr};
+        VariableMap[Key] = {Loc.Values, Loc.Expr};
         NewDefs.push_back(Loc);
         continue;
       }
@@ -2320,7 +2317,7 @@ removeUndefDbgLocsFromEntryBlock(const BasicBlock *BB,
 
       // Remove undef entries that are encountered before any non-undef
       // intrinsics from the entry block.
-      if (isa<UndefValue>(Loc.V) && !HasDefinedBits(Aggr, Var)) {
+      if (Loc.Values.isKillLocation(Loc.Expr) && !HasDefinedBits(Aggr, Var)) {
         // Did not insert this Loc, which is the same as removing it.
         NumDefsRemoved++;
         ChangedThisWedge = true;
diff --git a/llvm/lib/CodeGen/CodeGenPrepare.cpp b/llvm/lib/CodeGen/CodeGenPrepare.cpp
index 54c71c1e39461..8d11f282516c8 100644
--- a/llvm/lib/CodeGen/CodeGenPrepare.cpp
+++ b/llvm/lib/CodeGen/CodeGenPrepare.cpp
@@ -2349,24 +2349,6 @@ bool CodeGenPrepare::optimizeCallInst(CallInst *CI, ModifyDT &ModifiedDT) {
     case Intrinsic::dbg_assign:
     case Intrinsic::dbg_value:
       return fixupDbgValue(II);
-    case Intrinsic::vscale: {
-      // If datalayout has no special restrictions on vector data layout,
-      // replace `llvm.vscale` by an equivalent constant expression
-      // to benefit from cheap constant propagation.
-      Type *ScalableVectorTy =
-          VectorType::get(Type::getInt8Ty(II->getContext()), 1, true);
-      if (DL->getTypeAllocSize(ScalableVectorTy).getKnownMinValue() == 8) {
-        auto *Null = Constant::getNullValue(ScalableVectorTy->getPointerTo());
-        auto *One = ConstantInt::getSigned(II->getType(), 1);
-        auto *CGep =
-            ConstantExpr::getGetElementPtr(ScalableVectorTy, Null, One);
-        replaceAllUsesWith(II, ConstantExpr::getPtrToInt(CGep, II->getType()),
-                           FreshBBs, IsHugeFunc);
-        II->eraseFromParent();
-        return true;
-      }
-      break;
-    }
     case Intrinsic::masked_gather:
       return optimizeGatherScatterInst(II, II->getArgOperand(0));
     case Intrinsic::masked_scatter:
@@ -2687,7 +2669,7 @@ void ExtAddrMode::print(raw_ostream &OS) const {
   if (InBounds)
     OS << "inbounds ";
   if (BaseGV) {
-    OS << (NeedPlus ? " + " : "") << "GV:";
+    OS << "GV:";
     BaseGV->printAsOperand(OS, /*PrintType=*/false);
     NeedPlus = true;
   }
@@ -3866,17 +3848,17 @@ class AddressingModeCombiner {
                         SimplificationTracker &ST) {
     while (!TraverseOrder.empty()) {
       Value *Current = TraverseOrder.pop_back_val();
-      assert(Map.find(Current) != Map.end() && "No node to fill!!!");
+      assert(Map.contains(Current) && "No node to fill!!!");
       Value *V = Map[Current];
 
       if (SelectInst *Select = dyn_cast<SelectInst>(V)) {
         // CurrentValue also must be Select.
         auto *CurrentSelect = cast<SelectInst>(Current);
         auto *TrueValue = CurrentSelect->getTrueValue();
-        assert(Map.find(TrueValue) != Map.end() && "No True Value!");
+        assert(Map.contains(TrueValue) && "No True Value!");
         Select->setTrueValue(ST.Get(Map[TrueValue]));
         auto *FalseValue = CurrentSelect->getFalseValue();
-        assert(Map.find(FalseValue) != Map.end() && "No False Value!");
+        assert(Map.contains(FalseValue) && "No False Value!");
         Select->setFalseValue(ST.Get(Map[FalseValue]));
       } else {
         // Must be a Phi node then.
@@ -3884,7 +3866,7 @@ class AddressingModeCombiner {
         // Fill the Phi node with values from predecessors.
         for (auto *B : predecessors(PHI->getParent())) {
           Value *PV = cast<PHINode>(Current)->getIncomingValueForBlock(B);
-          assert(Map.find(PV) != Map.end() && "No predecessor Value!");
+          assert(Map.contains(PV) && "No predecessor Value!");
           PHI->addIncoming(ST.Get(Map[PV]), B);
         }
       }
@@ -3908,7 +3890,7 @@ class AddressingModeCombiner {
     while (!Worklist.empty()) {
       Value *Current = Worklist.pop_back_val();
       // if it is already visited or it is an ending value then skip it.
-      if (Map.find(Current) != Map.end())
+      if (Map.contains(Current))
         continue;
       TraverseOrder.push_back(Current);
 
diff --git a/llvm/lib/CodeGen/CommandFlags.cpp b/llvm/lib/CodeGen/CommandFlags.cpp
index 73521ac46df45..51d259cea41b8 100644
--- a/llvm/lib/CodeGen/CommandFlags.cpp
+++ b/llvm/lib/CodeGen/CommandFlags.cpp
@@ -81,7 +81,6 @@ CGOPT(bool, StackSymbolOrdering)
 CGOPT(bool, StackRealign)
 CGOPT(std::string, TrapFuncName)
 CGOPT(bool, UseCtors)
-CGOPT(bool, LowerGlobalDtorsViaCxaAtExit)
 CGOPT(bool, RelaxELFRelocations)
 CGOPT_EXP(bool, DataSections)
 CGOPT_EXP(bool, FunctionSections)
@@ -349,12 +348,6 @@ codegen::RegisterCodeGenFlags::RegisterCodeGenFlags() {
                                 cl::init(false));
   CGBINDOPT(UseCtors);
 
-  static cl::opt<bool> LowerGlobalDtorsViaCxaAtExit(
-      "lower-global-dtors-via-cxa-atexit",
-      cl::desc("Lower llvm.global_dtors (global destructors) via __cxa_atexit"),
-      cl::init(true));
-  CGBINDOPT(LowerGlobalDtorsViaCxaAtExit);
-
   static cl::opt<bool> RelaxELFRelocations(
       "relax-elf-relocations",
       cl::desc(
@@ -538,7 +531,6 @@ codegen::InitTargetOptionsFromCodeGenFlags(const Triple &TheTriple) {
   Options.GuaranteedTailCallOpt = getEnableGuaranteedTailCallOpt();
   Options.StackSymbolOrdering = getStackSymbolOrdering();
   Options.UseInitArray = !getUseCtors();
-  Options.LowerGlobalDtorsViaCxaAtExit = getLowerGlobalDtorsViaCxaAtExit();
   Options.RelaxELFRelocations = getRelaxELFRelocations();
   Options.DataSections =
       getExplicitDataSections().value_or(TheTriple.hasDefaultDataSections());
diff --git a/llvm/lib/CodeGen/ComplexDeinterleavingPass.cpp b/llvm/lib/CodeGen/ComplexDeinterleavingPass.cpp
index 9b1f7117fa573..fcc25d900e6ed 100644
--- a/llvm/lib/CodeGen/ComplexDeinterleavingPass.cpp
+++ b/llvm/lib/CodeGen/ComplexDeinterleavingPass.cpp
@@ -254,6 +254,7 @@ class ComplexDeinterleavingGraph {
   /// 270: r: ar + bi
   ///      i: ai - br
   NodePtr identifyAdd(Instruction *Real, Instruction *Imag);
+  NodePtr identifySymmetricOperation(Instruction *Real, Instruction *Imag);
 
   NodePtr identifyNode(Instruction *I, Instruction *J);
 
@@ -271,8 +272,6 @@ class ComplexDeinterleavingGraph {
   bool identifyNodes(Instruction *RootI);
 
   /// Perform the actual replacement of the underlying instruction graph.
-  /// Returns false if the deinterleaving operation should be cancelled for the
-  /// current graph.
   void replaceNodes();
 };
 
@@ -598,8 +597,16 @@ ComplexDeinterleavingGraph::identifyPartialMul(Instruction *Real,
        Rotation == ComplexDeinterleavingRotation::Rotation_270)
           ? CommonOperand
           : nullptr);
-  NodePtr CNode = identifyNodeWithImplicitAdd(
-      cast<Instruction>(CR), cast<Instruction>(CI), PartialMatch);
+
+  auto *CRInst = dyn_cast<Instruction>(CR);
+  auto *CIInst = dyn_cast<Instruction>(CI);
+
+  if (!CRInst || !CIInst) {
+    LLVM_DEBUG(dbgs() << "  - Common operands are not instructions.\n");
+    return nullptr;
+  }
+
+  NodePtr CNode = identifyNodeWithImplicitAdd(CRInst, CIInst, PartialMatch);
   if (!CNode) {
     LLVM_DEBUG(dbgs() << "  - No cnode identified\n");
     return nullptr;
@@ -696,6 +703,59 @@ static bool isInstructionPairMul(Instruction *A, Instruction *B) {
   return match(A, Pattern) && match(B, Pattern);
 }
 
+static bool isInstructionPotentiallySymmetric(Instruction *I) {
+  switch (I->getOpcode()) {
+  case Instruction::FAdd:
+  case Instruction::FSub:
+  case Instruction::FMul:
+  case Instruction::FNeg:
+    return true;
+  default:
+    return false;
+  }
+}
+
+ComplexDeinterleavingGraph::NodePtr
+ComplexDeinterleavingGraph::identifySymmetricOperation(Instruction *Real,
+                                                       Instruction *Imag) {
+  if (Real->getOpcode() != Imag->getOpcode())
+    return nullptr;
+
+  if (!isInstructionPotentiallySymmetric(Real) ||
+      !isInstructionPotentiallySymmetric(Imag))
+    return nullptr;
+
+  auto *R0 = dyn_cast<Instruction>(Real->getOperand(0));
+  auto *I0 = dyn_cast<Instruction>(Imag->getOperand(0));
+
+  if (!R0 || !I0)
+    return nullptr;
+
+  NodePtr Op0 = identifyNode(R0, I0);
+  NodePtr Op1 = nullptr;
+  if (Op0 == nullptr)
+    return nullptr;
+
+  if (Real->isBinaryOp()) {
+    auto *R1 = dyn_cast<Instruction>(Real->getOperand(1));
+    auto *I1 = dyn_cast<Instruction>(Imag->getOperand(1));
+    if (!R1 || !I1)
+      return nullptr;
+
+    Op1 = identifyNode(R1, I1);
+    if (Op1 == nullptr)
+      return nullptr;
+  }
+
+  auto Node = prepareCompositeNode(ComplexDeinterleavingOperation::Symmetric,
+                                   Real, Imag);
+  Node->addOperand(Op0);
+  if (Real->isBinaryOp())
+    Node->addOperand(Op1);
+
+  return submitCompositeNode(Node);
+}
+
 ComplexDeinterleavingGraph::NodePtr
 ComplexDeinterleavingGraph::identifyNode(Instruction *Real, Instruction *Imag) {
   LLVM_DEBUG(dbgs() << "identifyNode on " << *Real << " / " << *Imag << "\n");
@@ -788,8 +848,10 @@ ComplexDeinterleavingGraph::identifyNode(Instruction *Real, Instruction *Imag) {
     PlaceholderNode->ReplacementNode = RealShuffle->getOperand(0);
     return submitCompositeNode(PlaceholderNode);
   }
-  if (RealShuffle || ImagShuffle)
+  if (RealShuffle || ImagShuffle) {
+    LLVM_DEBUG(dbgs() << " - There's a shuffle where there shouldn't be.\n");
     return nullptr;
+  }
 
   auto *VTy = cast<FixedVectorType>(Real->getType());
   auto *NewVTy =
@@ -807,7 +869,10 @@ ComplexDeinterleavingGraph::identifyNode(Instruction *Real, Instruction *Imag) {
     return identifyAdd(Real, Imag);
   }
 
-  return nullptr;
+  auto Symmetric = identifySymmetricOperation(Real, Imag);
+  LLVM_DEBUG(if (Symmetric == nullptr) dbgs()
+             << "  - Not recognised as a valid pattern.\n");
+  return Symmetric;
 }
 
 bool ComplexDeinterleavingGraph::identifyNodes(Instruction *RootI) {
@@ -839,21 +904,53 @@ bool ComplexDeinterleavingGraph::identifyNodes(Instruction *RootI) {
   return RootNode != nullptr;
 }
 
+static Value *replaceSymmetricNode(ComplexDeinterleavingGraph::RawNodePtr Node,
+                                   Value *InputA, Value *InputB) {
+  Instruction *I = Node->Real;
+  if (I->isUnaryOp())
+    assert(!InputB &&
+           "Unary symmetric operations need one input, but two were provided.");
+  else if (I->isBinaryOp())
+    assert(InputB && "Binary symmetric operations need two inputs, only one "
+                     "was provided.");
+
+  IRBuilder<> B(I);
+
+  switch (I->getOpcode()) {
+  case Instruction::FNeg:
+    return B.CreateFNeg(InputA);
+  case Instruction::FAdd:
+    return B.CreateFAdd(InputA, InputB);
+  case Instruction::FSub:
+    return B.CreateFSub(InputA, InputB);
+  case Instruction::FMul:
+    return B.CreateFMul(InputA, InputB);
+  }
+
+  return nullptr;
+}
+
 Value *ComplexDeinterleavingGraph::replaceNode(
     ComplexDeinterleavingGraph::RawNodePtr Node) {
   if (Node->ReplacementNode)
     return Node->ReplacementNode;
 
   Value *Input0 = replaceNode(Node->Operands[0]);
-  Value *Input1 = replaceNode(Node->Operands[1]);
+  Value *Input1 =
+      Node->Operands.size() > 1 ? replaceNode(Node->Operands[1]) : nullptr;
   Value *Accumulator =
       Node->Operands.size() > 2 ? replaceNode(Node->Operands[2]) : nullptr;
 
-  assert(Input0->getType() == Input1->getType() &&
-         "Node inputs need to be of the same type");
+  if (Input1)
+    assert(Input0->getType() == Input1->getType() &&
+           "Node inputs need to be of the same type");
 
-  Node->ReplacementNode = TL->createComplexDeinterleavingIR(
-      Node->Real, Node->Operation, Node->Rotation, Input0, Input1, Accumulator);
+  if (Node->Operation == ComplexDeinterleavingOperation::Symmetric)
+    Node->ReplacementNode = replaceSymmetricNode(Node, Input0, Input1);
+  else
+    Node->ReplacementNode = TL->createComplexDeinterleavingIR(
+        Node->Real, Node->Operation, Node->Rotation, Input0, Input1,
+        Accumulator);
 
   assert(Node->ReplacementNode && "Target failed to create Intrinsic call.");
   NumComplexTransformations += 1;
diff --git a/llvm/lib/CodeGen/DwarfEHPrepare.cpp b/llvm/lib/CodeGen/DwarfEHPrepare.cpp
index 32c94de7280cb..69768a17e0fa9 100644
--- a/llvm/lib/CodeGen/DwarfEHPrepare.cpp
+++ b/llvm/lib/CodeGen/DwarfEHPrepare.cpp
@@ -18,6 +18,7 @@
 #include "llvm/Analysis/DomTreeUpdater.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/CodeGen/RuntimeLibcalls.h"
+#include "llvm/CodeGen/StackProtector.h"
 #include "llvm/CodeGen/TargetLowering.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
@@ -28,6 +29,7 @@
 #include "llvm/IR/Dominators.h"
 #include "llvm/IR/EHPersonalities.h"
 #include "llvm/IR/Function.h"
+#include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/Module.h"
 #include "llvm/IR/Type.h"
@@ -37,6 +39,7 @@
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/TargetParser/Triple.h"
 #include "llvm/Transforms/Utils/Local.h"
+#include "llvm/Transforms/Utils/BasicBlockUtils.h"
 #include <cstddef>
 
 using namespace llvm;
@@ -166,7 +169,136 @@ size_t DwarfEHPrepare::pruneUnreachableResumes(
   return ResumesLeft;
 }
 
+/// If a landingpad block doesn't already have a cleanup case, add one
+/// that feeds directly into a resume instruction.
+static void addCleanupResumeToLandingPad(BasicBlock &BB, DomTreeUpdater *DTU) {
+  LandingPadInst *LP = BB.getLandingPadInst();
+  if (LP->isCleanup())
+    return;
+
+  // There will usually be code testing for the other kinds of exception
+  // immediately after the landingpad. Working out the far end of that chain is
+  // tricky, so put our test for the new cleanup case (i.e. selector == 0) at
+  // the beginning.
+  BasicBlock *ContBB = SplitBlock(&BB, LP->getNextNode(), DTU);
+  BB.getTerminator()->eraseFromParent();
+
+  LP->setCleanup(true);
+  IRBuilder<> B(&BB);
+  Value *Selector = B.CreateExtractValue(LP, 1);
+  Value *Cmp = B.CreateICmpEQ(Selector, ConstantInt::get(Selector->getType(), 0));
+
+  Function *F = BB.getParent();
+  LLVMContext &Ctx = F->getContext();
+  BasicBlock *ResumeBB = BasicBlock::Create(Ctx, "resume", F);
+  ResumeInst::Create(LP, ResumeBB);
+
+  B.CreateCondBr(Cmp, ResumeBB, ContBB);
+  if (DTU) {
+    SmallVector<DominatorTree::UpdateType> Updates;
+    Updates.push_back({DominatorTree::Insert, &BB, ResumeBB});
+    DTU->applyUpdates(Updates);
+  }
+}
+
+/// Create a basic block that has a `landingpad` instruction feeding
+/// directly into a `resume`. Will be set to the unwind destination of a new
+/// invoke.
+static BasicBlock *createCleanupResumeBB(Function &F,  Type *LandingPadTy) {
+  LLVMContext &Ctx = F.getContext();
+  BasicBlock *BB = BasicBlock::Create(Ctx, "cleanup_resume", &F);
+  IRBuilder<> B(BB);
+
+  // If this is going to be the only landingpad in the function, synthesize the
+  // standard type all ABIs use, which is essentially `{ ptr, i32 }`.
+  if (!LandingPadTy)
+    LandingPadTy =
+        StructType::get(Type::getInt8PtrTy(Ctx), IntegerType::get(Ctx, 32));
+
+  LandingPadInst *Except = B.CreateLandingPad(LandingPadTy, 0);
+  Except->setCleanup(true);
+  B.CreateResume(Except);
+  return BB;
+}
+
+/// Convert a call that might throw into an invoke that unwinds to the specified
+/// simple landingpad/resume block.
+static void changeCallToInvokeResume(CallInst &CI, BasicBlock *CleanupResumeBB,
+                                     DomTreeUpdater *DTU) {
+  BasicBlock *BB = CI.getParent();
+  BasicBlock *ContBB = SplitBlock(BB, &CI, DTU);
+  BB->getTerminator()->eraseFromParent();
+
+  IRBuilder<> B(BB);
+  SmallVector<Value *> Args(CI.args());
+  SmallVector<OperandBundleDef> Bundles;
+  CI.getOperandBundlesAsDefs(Bundles);
+  InvokeInst *NewCall =
+      B.CreateInvoke(CI.getFunctionType(), CI.getCalledOperand(), ContBB,
+                     CleanupResumeBB, Args, Bundles, CI.getName());
+  NewCall->setAttributes(CI.getAttributes());
+  NewCall->setCallingConv(CI.getCallingConv());
+  NewCall->copyMetadata(CI);
+
+  if (DTU) {
+    SmallVector<DominatorTree::UpdateType> Updates;
+    Updates.push_back({DominatorTree::Insert, BB, CleanupResumeBB});
+    DTU->applyUpdates(Updates);
+  }
+  CI.replaceAllUsesWith(NewCall);
+  CI.eraseFromParent();
+}
+
+/// Ensure that any call in this function that might throw has an associated
+/// cleanup/resume that the stack protector can instrument later. Existing
+/// invokes will get an added `cleanup` clause if needed, calls will be
+/// converted to an invoke with trivial unwind followup.
+static void addCleanupPathsForStackProtector(Function &F, DomTreeUpdater *DTU) {
+  // First add cleanup -> resume paths to all existing landingpads, noting what
+  // type landingpads in this function actually have along the way.
+  Type *LandingPadTy = nullptr;
+  for (Function::iterator FI = F.begin(); FI != F.end(); ++FI) {
+    BasicBlock &BB = *FI;
+    if (LandingPadInst *LP = BB.getLandingPadInst()) {
+      // We can assume the type is broadly compatible with { ptr, i32 } since
+      // other parts of this pass already try to extract values from it.
+      LandingPadTy = LP->getType();
+      addCleanupResumeToLandingPad(BB, DTU);
+    }
+  }
+
+  // Next convert any call that might throw into an invoke to a resume
+  // instruction for later instrumentation.
+  BasicBlock *CleanupResumeBB = nullptr;
+  for (Function::iterator FI = F.begin(); FI != F.end(); ++FI) {
+    BasicBlock &BB = *FI;
+    for (Instruction &I : BB) {
+      CallInst *CI = dyn_cast<CallInst>(&I);
+      if (!CI || CI->doesNotThrow())
+        continue;
+
+      // Tail calls cannot use our stack so no need to check whether it was
+      // corrupted.
+      if (CI->isTailCall())
+        continue;
+
+      if (!CleanupResumeBB)
+        CleanupResumeBB = createCleanupResumeBB(F, LandingPadTy);
+
+      changeCallToInvokeResume(*CI, CleanupResumeBB, DTU);
+
+      // This block has been split, start again on its continuation.
+      break;
+    }
+  }
+}
+
 bool DwarfEHPrepare::InsertUnwindResumeCalls() {
+  if (F.hasPersonalityFn() &&
+      !isScopedEHPersonality(classifyEHPersonality(F.getPersonalityFn())) &&
+      StackProtector::requiresStackProtector(&F, nullptr))
+    addCleanupPathsForStackProtector(F, DTU);
+
   SmallVector<ResumeInst *, 16> Resumes;
   SmallVector<LandingPadInst *, 16> CleanupLPads;
   if (F.doesNotThrow())
diff --git a/llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp b/llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp
index 55d939de426e3..0b32d69afeb28 100644
--- a/llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp
+++ b/llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp
@@ -388,7 +388,7 @@ class StatepointState {
       Register Reg = MO.getReg();
       assert(Reg.isPhysical() && "Only physical regs are expected");
 
-      if (isCalleeSaved(Reg) && (AllowGCPtrInCSR || !is_contained(GCRegs, Reg)))
+      if (isCalleeSaved(Reg) && (AllowGCPtrInCSR || !GCRegs.contains(Reg)))
         continue;
 
       LLVM_DEBUG(dbgs() << "Will spill " << printReg(Reg, &TRI) << " at index "
diff --git a/llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp b/llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp
index e68fa70862c19..f0a5b2ebcb3ae 100644
--- a/llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp
@@ -7359,14 +7359,14 @@ LegalizerHelper::lowerISFPCLASS(MachineInstr &MI) {
   Register SrcReg = MI.getOperand(1).getReg();
   LLT DstTy = MRI.getType(DstReg);
   LLT SrcTy = MRI.getType(SrcReg);
-  uint64_t Mask = MI.getOperand(2).getImm();
+  FPClassTest Mask = static_cast<FPClassTest>(MI.getOperand(2).getImm());
 
-  if (Mask == 0) {
+  if (Mask == fcNone) {
     MIRBuilder.buildConstant(DstReg, 0);
     MI.eraseFromParent();
     return Legalized;
   }
-  if ((Mask & fcAllFlags) == fcAllFlags) {
+  if (Mask == fcAllFlags) {
     MIRBuilder.buildConstant(DstReg, 1);
     MI.eraseFromParent();
     return Legalized;
@@ -7426,7 +7426,7 @@ LegalizerHelper::lowerISFPCLASS(MachineInstr &MI) {
   }
 
   // Check for individual classes.
-  if (unsigned PartialCheck = Mask & fcZero) {
+  if (FPClassTest PartialCheck = Mask & fcZero) {
     if (PartialCheck == fcPosZero)
       appendToRes(MIRBuilder.buildICmp(CmpInst::Predicate::ICMP_EQ, DstTy,
                                        AsInt, ZeroC));
@@ -7438,7 +7438,21 @@ LegalizerHelper::lowerISFPCLASS(MachineInstr &MI) {
                                        AsInt, SignBitC));
   }
 
-  if (unsigned PartialCheck = Mask & fcInf) {
+  if (FPClassTest PartialCheck = Mask & fcSubnormal) {
+    // issubnormal(V) ==> unsigned(abs(V) - 1) u< (all mantissa bits set)
+    // issubnormal(V) && V>0 ==> unsigned(V - 1) u< (all mantissa bits set)
+    auto V = (PartialCheck == fcPosSubnormal) ? AsInt : Abs;
+    auto OneC = MIRBuilder.buildConstant(IntTy, 1);
+    auto VMinusOne = MIRBuilder.buildSub(IntTy, V, OneC);
+    auto SubnormalRes =
+        MIRBuilder.buildICmp(CmpInst::Predicate::ICMP_ULT, DstTy, VMinusOne,
+                             MIRBuilder.buildConstant(IntTy, AllOneMantissa));
+    if (PartialCheck == fcNegSubnormal)
+      SubnormalRes = MIRBuilder.buildAnd(DstTy, SubnormalRes, Sign);
+    appendToRes(SubnormalRes);
+  }
+
+  if (FPClassTest PartialCheck = Mask & fcInf) {
     if (PartialCheck == fcPosInf)
       appendToRes(MIRBuilder.buildICmp(CmpInst::Predicate::ICMP_EQ, DstTy,
                                        AsInt, InfC));
@@ -7453,7 +7467,7 @@ LegalizerHelper::lowerISFPCLASS(MachineInstr &MI) {
     }
   }
 
-  if (unsigned PartialCheck = Mask & fcNan) {
+  if (FPClassTest PartialCheck = Mask & fcNan) {
     auto InfWithQnanBitC = MIRBuilder.buildConstant(IntTy, Inf | QNaNBitMask);
     if (PartialCheck == fcNan) {
       // isnan(V) ==> abs(V) u> int(inf)
@@ -7474,21 +7488,7 @@ LegalizerHelper::lowerISFPCLASS(MachineInstr &MI) {
     }
   }
 
-  if (unsigned PartialCheck = Mask & fcSubnormal) {
-    // issubnormal(V) ==> unsigned(abs(V) - 1) u< (all mantissa bits set)
-    // issubnormal(V) && V>0 ==> unsigned(V - 1) u< (all mantissa bits set)
-    auto V = (PartialCheck == fcPosSubnormal) ? AsInt : Abs;
-    auto OneC = MIRBuilder.buildConstant(IntTy, 1);
-    auto VMinusOne = MIRBuilder.buildSub(IntTy, V, OneC);
-    auto SubnormalRes =
-        MIRBuilder.buildICmp(CmpInst::Predicate::ICMP_ULT, DstTy, VMinusOne,
-                             MIRBuilder.buildConstant(IntTy, AllOneMantissa));
-    if (PartialCheck == fcNegSubnormal)
-      SubnormalRes = MIRBuilder.buildAnd(DstTy, SubnormalRes, Sign);
-    appendToRes(SubnormalRes);
-  }
-
-  if (unsigned PartialCheck = Mask & fcNormal) {
+  if (FPClassTest PartialCheck = Mask & fcNormal) {
     // isnormal(V) ==> (0 u< exp u< max_exp) ==> (unsigned(exp-1) u<
     // (max_exp-1))
     APInt ExpLSB = ExpMask & ~(ExpMask.shl(1));
diff --git a/llvm/lib/CodeGen/InlineSpiller.cpp b/llvm/lib/CodeGen/InlineSpiller.cpp
index cf4fff878ad1b..53117d63a39b3 100644
--- a/llvm/lib/CodeGen/InlineSpiller.cpp
+++ b/llvm/lib/CodeGen/InlineSpiller.cpp
@@ -1250,7 +1250,7 @@ void HoistSpillHelper::addToMergeableSpills(MachineInstr &Spill, int StackSlot,
   LiveInterval &OrigLI = LIS.getInterval(Original);
   // save a copy of LiveInterval in StackSlotToOrigLI because the original
   // LiveInterval may be cleared after all its references are spilled.
-  if (StackSlotToOrigLI.find(StackSlot) == StackSlotToOrigLI.end()) {
+  if (!StackSlotToOrigLI.contains(StackSlot)) {
     auto LI = std::make_unique<LiveInterval>(OrigLI.reg(), OrigLI.weight());
     LI->assign(OrigLI, Allocator);
     StackSlotToOrigLI[StackSlot] = std::move(LI);
@@ -1459,7 +1459,7 @@ void HoistSpillHelper::runHoistSpills(
     MachineBasicBlock *Block = (*RIt)->getBlock();
 
     // If Block contains an original spill, simply continue.
-    if (SpillsToKeep.find(*RIt) != SpillsToKeep.end() && !SpillsToKeep[*RIt]) {
+    if (SpillsToKeep.contains(*RIt) && !SpillsToKeep[*RIt]) {
       SpillsInSubTreeMap[*RIt].first.insert(*RIt);
       // SpillsInSubTreeMap[*RIt].second contains the cost of spill.
       SpillsInSubTreeMap[*RIt].second = MBFI.getBlockFreq(Block);
@@ -1469,7 +1469,7 @@ void HoistSpillHelper::runHoistSpills(
     // Collect spills in subtree of current node (*RIt) to
     // SpillsInSubTreeMap[*RIt].first.
     for (MachineDomTreeNode *Child : (*RIt)->children()) {
-      if (SpillsInSubTreeMap.find(Child) == SpillsInSubTreeMap.end())
+      if (!SpillsInSubTreeMap.contains(Child))
         continue;
       // The stmt "SpillsInSubTree = SpillsInSubTreeMap[*RIt].first" below
       // should be placed before getting the begin and end iterators of
@@ -1508,8 +1508,7 @@ void HoistSpillHelper::runHoistSpills(
       for (auto *const SpillBB : SpillsInSubTree) {
         // When SpillBB is a BB contains original spill, insert the spill
         // to SpillsToRm.
-        if (SpillsToKeep.find(SpillBB) != SpillsToKeep.end() &&
-            !SpillsToKeep[SpillBB]) {
+        if (SpillsToKeep.contains(SpillBB) && !SpillsToKeep[SpillBB]) {
           MachineInstr *SpillToRm = SpillBBToSpill[SpillBB];
           SpillsToRm.push_back(SpillToRm);
         }
diff --git a/llvm/lib/CodeGen/InterleavedAccessPass.cpp b/llvm/lib/CodeGen/InterleavedAccessPass.cpp
index 0582378be4cd8..95c4fafba8640 100644
--- a/llvm/lib/CodeGen/InterleavedAccessPass.cpp
+++ b/llvm/lib/CodeGen/InterleavedAccessPass.cpp
@@ -202,86 +202,15 @@ static bool isDeInterleaveMask(ArrayRef<int> Mask, unsigned &Factor,
 /// The particular case of an RE-interleave mask is:
 /// I.e. <0, LaneLen, ... , LaneLen*(Factor - 1), 1, LaneLen + 1, ...>
 /// E.g. For a Factor of 2 (LaneLen=4): <0, 4, 1, 5, 2, 6, 3, 7>
-static bool isReInterleaveMask(ArrayRef<int> Mask, unsigned &Factor,
-                               unsigned MaxFactor, unsigned OpNumElts) {
-  unsigned NumElts = Mask.size();
+static bool isReInterleaveMask(ShuffleVectorInst *SVI, unsigned &Factor,
+                               unsigned MaxFactor) {
+  unsigned NumElts = SVI->getShuffleMask().size();
   if (NumElts < 4)
     return false;
 
   // Check potential Factors.
   for (Factor = 2; Factor <= MaxFactor; Factor++) {
-    if (NumElts % Factor)
-      continue;
-
-    unsigned LaneLen = NumElts / Factor;
-    if (!isPowerOf2_32(LaneLen))
-      continue;
-
-    // Check whether each element matches the general interleaved rule.
-    // Ignore undef elements, as long as the defined elements match the rule.
-    // Outer loop processes all factors (x, y, z in the above example)
-    unsigned I = 0, J;
-    for (; I < Factor; I++) {
-      unsigned SavedLaneValue;
-      unsigned SavedNoUndefs = 0;
-
-      // Inner loop processes consecutive accesses (x, x+1... in the example)
-      for (J = 0; J < LaneLen - 1; J++) {
-        // Lane computes x's position in the Mask
-        unsigned Lane = J * Factor + I;
-        unsigned NextLane = Lane + Factor;
-        int LaneValue = Mask[Lane];
-        int NextLaneValue = Mask[NextLane];
-
-        // If both are defined, values must be sequential
-        if (LaneValue >= 0 && NextLaneValue >= 0 &&
-            LaneValue + 1 != NextLaneValue)
-          break;
-
-        // If the next value is undef, save the current one as reference
-        if (LaneValue >= 0 && NextLaneValue < 0) {
-          SavedLaneValue = LaneValue;
-          SavedNoUndefs = 1;
-        }
-
-        // Undefs are allowed, but defined elements must still be consecutive:
-        // i.e.: x,..., undef,..., x + 2,..., undef,..., undef,..., x + 5, ....
-        // Verify this by storing the last non-undef followed by an undef
-        // Check that following non-undef masks are incremented with the
-        // corresponding distance.
-        if (SavedNoUndefs > 0 && LaneValue < 0) {
-          SavedNoUndefs++;
-          if (NextLaneValue >= 0 &&
-              SavedLaneValue + SavedNoUndefs != (unsigned)NextLaneValue)
-            break;
-        }
-      }
-
-      if (J < LaneLen - 1)
-        break;
-
-      int StartMask = 0;
-      if (Mask[I] >= 0) {
-        // Check that the start of the I range (J=0) is greater than 0
-        StartMask = Mask[I];
-      } else if (Mask[(LaneLen - 1) * Factor + I] >= 0) {
-        // StartMask defined by the last value in lane
-        StartMask = Mask[(LaneLen - 1) * Factor + I] - J;
-      } else if (SavedNoUndefs > 0) {
-        // StartMask defined by some non-zero value in the j loop
-        StartMask = SavedLaneValue - (LaneLen - 1 - SavedNoUndefs);
-      }
-      // else StartMask remains set to 0, i.e. all elements are undefs
-
-      if (StartMask < 0)
-        break;
-      // We must stay within the vectors; This case can happen with undefs.
-      if (StartMask + LaneLen > OpNumElts*2)
-        break;
-    }
-
-    // Found an interleaved mask of current factor.
-    if (I == Factor)
+    if (SVI->isInterleave(Factor))
       return true;
   }
 
@@ -500,9 +429,7 @@ bool InterleavedAccess::lowerInterleavedStore(
 
   // Check if the shufflevector is RE-interleave shuffle.
   unsigned Factor;
-  unsigned OpNumElts =
-      cast<FixedVectorType>(SVI->getOperand(0)->getType())->getNumElements();
-  if (!isReInterleaveMask(SVI->getShuffleMask(), Factor, MaxFactor, OpNumElts))
+  if (!isReInterleaveMask(SVI, Factor, MaxFactor))
     return false;
 
   LLVM_DEBUG(dbgs() << "IA: Found an interleaved store: " << *SI << "\n");
diff --git a/llvm/lib/CodeGen/LiveDebugValues/InstrRefBasedImpl.cpp b/llvm/lib/CodeGen/LiveDebugValues/InstrRefBasedImpl.cpp
index 26aca0345dfd4..60a0db596139e 100644
--- a/llvm/lib/CodeGen/LiveDebugValues/InstrRefBasedImpl.cpp
+++ b/llvm/lib/CodeGen/LiveDebugValues/InstrRefBasedImpl.cpp
@@ -3179,7 +3179,7 @@ void InstrRefBasedLDV::buildVLocValueMap(
     SmallPtrSet<MachineBasicBlock *, 32> DefBlocks;
     for (const MachineBasicBlock *ExpMBB : BlocksToExplore) {
       auto &TransferFunc = AllTheVLocs[ExpMBB->getNumber()].Vars;
-      if (TransferFunc.find(Var) != TransferFunc.end())
+      if (TransferFunc.contains(Var))
         DefBlocks.insert(const_cast<MachineBasicBlock *>(ExpMBB));
     }
 
@@ -3295,7 +3295,7 @@ void InstrRefBasedLDV::buildVLocValueMap(
         // to be visited next time around.
         for (auto *s : MBB->successors()) {
           // Ignore out of scope / not-to-be-explored successors.
-          if (LiveInIdx.find(s) == LiveInIdx.end())
+          if (!LiveInIdx.contains(s))
             continue;
 
           if (BBToOrder[s] > BBToOrder[MBB]) {
@@ -3411,7 +3411,7 @@ void InstrRefBasedLDV::initialSetup(MachineFunction &MF) {
   for (MachineBasicBlock *MBB : RPOT)
     processMBB(MBB);
   for (MachineBasicBlock &MBB : MF)
-    if (BBToOrder.find(&MBB) == BBToOrder.end())
+    if (!BBToOrder.contains(&MBB))
       processMBB(&MBB);
 
   // Order value substitutions by their "source" operand pair, for quick lookup.
@@ -4195,7 +4195,7 @@ std::optional<ValueIDNum> InstrRefBasedLDV::resolveDbgPHIsImpl(
     // Are all these things actually defined?
     for (auto &PHIIt : PHI->IncomingValues) {
       // Any undef input means DBG_PHIs didn't dominate the use point.
-      if (Updater.UndefMap.find(&PHIIt.first->BB) != Updater.UndefMap.end())
+      if (Updater.UndefMap.contains(&PHIIt.first->BB))
         return std::nullopt;
 
       ValueIDNum ValueToCheck;
diff --git a/llvm/lib/CodeGen/LiveDebugValues/InstrRefBasedImpl.h b/llvm/lib/CodeGen/LiveDebugValues/InstrRefBasedImpl.h
index 2fdc37c6dda26..cb0008a5ce6ed 100644
--- a/llvm/lib/CodeGen/LiveDebugValues/InstrRefBasedImpl.h
+++ b/llvm/lib/CodeGen/LiveDebugValues/InstrRefBasedImpl.h
@@ -740,7 +740,7 @@ class MLocTracker {
   unsigned getLocID(SpillLocationNo Spill, StackSlotPos Idx) {
     unsigned SlotNo = Spill.id() - 1;
     SlotNo *= NumSlotIdxes;
-    assert(StackSlotIdxes.find(Idx) != StackSlotIdxes.end());
+    assert(StackSlotIdxes.contains(Idx));
     SlotNo += StackSlotIdxes[Idx];
     SlotNo += NumRegs;
     return SlotNo;
diff --git a/llvm/lib/CodeGen/LiveDebugValues/VarLocBasedImpl.cpp b/llvm/lib/CodeGen/LiveDebugValues/VarLocBasedImpl.cpp
index eec3de9ca33ae..959fea6ccb065 100644
--- a/llvm/lib/CodeGen/LiveDebugValues/VarLocBasedImpl.cpp
+++ b/llvm/lib/CodeGen/LiveDebugValues/VarLocBasedImpl.cpp
@@ -1347,7 +1347,7 @@ void VarLocBasedLDV::removeEntryValue(const MachineInstr &MI,
   // Try to get non-debug instruction responsible for the DBG_VALUE.
   const MachineInstr *TransferInst = nullptr;
   Register Reg = MI.getDebugOperand(0).getReg();
-  if (Reg.isValid() && RegSetInstrs.find(Reg) != RegSetInstrs.end())
+  if (Reg.isValid() && RegSetInstrs.contains(Reg))
     TransferInst = RegSetInstrs.find(Reg)->second;
 
   // Case of the parameter's DBG_VALUE at the start of entry MBB.
diff --git a/llvm/lib/CodeGen/LiveRangeEdit.cpp b/llvm/lib/CodeGen/LiveRangeEdit.cpp
index d8b024fbdfea4..67902686b90cd 100644
--- a/llvm/lib/CodeGen/LiveRangeEdit.cpp
+++ b/llvm/lib/CodeGen/LiveRangeEdit.cpp
@@ -82,7 +82,7 @@ void LiveRangeEdit::scanRemattable() {
   for (VNInfo *VNI : getParent().valnos) {
     if (VNI->isUnused())
       continue;
-    unsigned Original = VRM->getOriginal(getReg());
+    Register Original = VRM->getOriginal(getReg());
     LiveInterval &OrigLI = LIS.getInterval(Original);
     VNInfo *OrigVNI = OrigLI.getVNInfoAt(VNI->def);
     if (!OrigVNI)
@@ -181,11 +181,9 @@ bool LiveRangeEdit::canRematerializeAt(Remat &RM, VNInfo *OrigVNI,
 
 SlotIndex LiveRangeEdit::rematerializeAt(MachineBasicBlock &MBB,
                                          MachineBasicBlock::iterator MI,
-                                         unsigned DestReg,
-                                         const Remat &RM,
+                                         Register DestReg, const Remat &RM,
                                          const TargetRegisterInfo &tri,
-                                         bool Late,
-                                         unsigned SubIdx,
+                                         bool Late, unsigned SubIdx,
                                          MachineInstr *ReplaceIndexMI) {
   assert(RM.OrigMI && "Invalid remat");
   TII.reMaterialize(MBB, MI, DestReg, SubIdx, *RM.OrigMI, tri);
@@ -306,7 +304,7 @@ void LiveRangeEdit::eliminateDeadDef(MachineInstr *MI, ToShrinkSet &ToShrink) {
   LLVM_DEBUG(dbgs() << "Deleting dead def " << Idx << '\t' << *MI);
 
   // Collect virtual registers to be erased after MI is gone.
-  SmallVector<unsigned, 8> RegsToErase;
+  SmallVector<Register, 8> RegsToErase;
   bool ReadsPhysRegs = false;
   bool isOrigDef = false;
   Register Dest;
diff --git a/llvm/lib/CodeGen/MIRFSDiscriminator.cpp b/llvm/lib/CodeGen/MIRFSDiscriminator.cpp
index ad8a17f25ec55..e2166a4b484ac 100644
--- a/llvm/lib/CodeGen/MIRFSDiscriminator.cpp
+++ b/llvm/lib/CodeGen/MIRFSDiscriminator.cpp
@@ -22,6 +22,7 @@
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/raw_ostream.h"
+#include "llvm/Support/xxhash.h"
 #include "llvm/Transforms/Utils/SampleProfileLoaderBaseUtil.h"
 
 using namespace llvm;
@@ -30,6 +31,13 @@ using namespace sampleprofutil;
 
 #define DEBUG_TYPE "mirfs-discriminators"
 
+// TODO(xur): Remove this option and related code once we make true as the
+// default.
+cl::opt<bool> ImprovedFSDiscriminator(
+    "improved-fs-discriminator", cl::Hidden, cl::init(false),
+    cl::desc("New FS discriminators encoding (incompatible with the original "
+             "encoding)"));
+
 char MIRAddFSDiscriminators::ID = 0;
 
 INITIALIZE_PASS(MIRAddFSDiscriminators, DEBUG_TYPE,
@@ -42,11 +50,12 @@ FunctionPass *llvm::createMIRAddFSDiscriminatorsPass(FSDiscriminatorPass P) {
   return new MIRAddFSDiscriminators(P);
 }
 
+// TODO(xur): Remove this once we switch to ImprovedFSDiscriminator.
 // Compute a hash value using debug line number, and the line numbers from the
 // inline stack.
-static uint64_t getCallStackHash(const MachineBasicBlock &BB,
-                                 const MachineInstr &MI,
-                                 const DILocation *DIL) {
+static uint64_t getCallStackHashV0(const MachineBasicBlock &BB,
+                                   const MachineInstr &MI,
+                                   const DILocation *DIL) {
   auto updateHash = [](const StringRef &Str) -> uint64_t {
     if (Str.empty())
       return 0;
@@ -62,6 +71,19 @@ static uint64_t getCallStackHash(const MachineBasicBlock &BB,
   return Ret;
 }
 
+static uint64_t getCallStackHash(const DILocation *DIL) {
+  auto hashCombine = [](const uint64_t Seed, const uint64_t Val) {
+    std::hash<uint64_t> Hasher;
+    return Seed ^ (Hasher(Val) + 0x9e3779b9 + (Seed << 6) + (Seed >> 2));
+  };
+  uint64_t Ret = 0;
+  for (DIL = DIL->getInlinedAt(); DIL; DIL = DIL->getInlinedAt()) {
+    Ret = hashCombine(Ret, xxHash64(ArrayRef<uint8_t>(DIL->getLine())));
+    Ret = hashCombine(Ret, xxHash64(DIL->getSubprogramLinkageName()));
+  }
+  return Ret;
+}
+
 // Traverse the CFG and assign FD discriminators. If two instructions
 // have the same lineno and discriminator, but residing in different BBs,
 // the latter instruction will get a new discriminator value. The new
@@ -74,7 +96,8 @@ bool MIRAddFSDiscriminators::runOnMachineFunction(MachineFunction &MF) {
     return false;
 
   bool Changed = false;
-  using LocationDiscriminator = std::tuple<StringRef, unsigned, unsigned>;
+  using LocationDiscriminator =
+      std::tuple<StringRef, unsigned, unsigned, uint64_t>;
   using BBSet = DenseSet<const MachineBasicBlock *>;
   using LocationDiscriminatorBBMap = DenseMap<LocationDiscriminator, BBSet>;
   using LocationDiscriminatorCurrPassMap =
@@ -84,7 +107,12 @@ bool MIRAddFSDiscriminators::runOnMachineFunction(MachineFunction &MF) {
   LocationDiscriminatorCurrPassMap LDCM;
 
   // Mask of discriminators before this pass.
-  unsigned BitMaskBefore = getN1Bits(LowBit);
+  // TODO(xur): simplify this once we switch to ImprovedFSDiscriminator.
+  unsigned LowBitTemp = LowBit;
+  assert(LowBit > 0 && "LowBit in FSDiscriminator cannot be 0");
+  if (ImprovedFSDiscriminator)
+    LowBitTemp -= 1;
+  unsigned BitMaskBefore = getN1Bits(LowBitTemp);
   // Mask of discriminators including this pass.
   unsigned BitMaskNow = getN1Bits(HighBit);
   // Mask of discriminators for bits specific to this pass.
@@ -92,9 +120,14 @@ bool MIRAddFSDiscriminators::runOnMachineFunction(MachineFunction &MF) {
   unsigned NumNewD = 0;
 
   LLVM_DEBUG(dbgs() << "MIRAddFSDiscriminators working on Func: "
-                    << MF.getFunction().getName() << "\n");
+                    << MF.getFunction().getName() << " Highbit=" << HighBit
+                    << "\n");
+
   for (MachineBasicBlock &BB : MF) {
     for (MachineInstr &I : BB) {
+      if (ImprovedFSDiscriminator && I.isMetaInstruction()) {
+        continue;
+      }
       const DILocation *DIL = I.getDebugLoc().get();
       if (!DIL)
         continue;
@@ -102,7 +135,12 @@ bool MIRAddFSDiscriminators::runOnMachineFunction(MachineFunction &MF) {
       if (LineNo == 0)
         continue;
       unsigned Discriminator = DIL->getDiscriminator();
-      LocationDiscriminator LD{DIL->getFilename(), LineNo, Discriminator};
+      uint64_t CallStackHashVal = 0;
+      if (ImprovedFSDiscriminator)
+        CallStackHashVal = getCallStackHash(DIL);
+
+      LocationDiscriminator LD{DIL->getFilename(), LineNo, Discriminator,
+                               CallStackHashVal};
       auto &BBMap = LDBM[LD];
       auto R = BBMap.insert(&BB);
       if (BBMap.size() == 1)
@@ -111,7 +149,8 @@ bool MIRAddFSDiscriminators::runOnMachineFunction(MachineFunction &MF) {
       unsigned DiscriminatorCurrPass;
       DiscriminatorCurrPass = R.second ? ++LDCM[LD] : LDCM[LD];
       DiscriminatorCurrPass = DiscriminatorCurrPass << LowBit;
-      DiscriminatorCurrPass += getCallStackHash(BB, I, DIL);
+      if (!ImprovedFSDiscriminator)
+        DiscriminatorCurrPass += getCallStackHashV0(BB, I, DIL);
       DiscriminatorCurrPass &= BitMaskThisPass;
       unsigned NewD = Discriminator | DiscriminatorCurrPass;
       const auto *const NewDIL = DIL->cloneWithDiscriminator(NewD);
diff --git a/llvm/lib/CodeGen/MIRSampleProfile.cpp b/llvm/lib/CodeGen/MIRSampleProfile.cpp
index 81d86621b9df9..a821ad4ccea4c 100644
--- a/llvm/lib/CodeGen/MIRSampleProfile.cpp
+++ b/llvm/lib/CodeGen/MIRSampleProfile.cpp
@@ -58,6 +58,7 @@ static cl::opt<bool> ViewBFIAfter("fs-viewbfi-after", cl::Hidden,
                                   cl::init(false),
                                   cl::desc("View BFI after MIR loader"));
 
+extern cl::opt<bool> ImprovedFSDiscriminator;
 char MIRProfileLoaderPass::ID = 0;
 
 INITIALIZE_PASS_BEGIN(MIRProfileLoaderPass, DEBUG_TYPE,
@@ -165,6 +166,11 @@ class MIRProfileLoader final
   unsigned HighBit;
 
   bool ProfileIsValid = true;
+  ErrorOr<uint64_t> getInstWeight(const MachineInstr &MI) override {
+    if (ImprovedFSDiscriminator && MI.isMetaInstruction())
+      return std::error_code();
+    return getInstWeightImpl(MI);
+  }
 };
 
 template <>
diff --git a/llvm/lib/CodeGen/MIRVRegNamerUtils.cpp b/llvm/lib/CodeGen/MIRVRegNamerUtils.cpp
index e634a2b284c31..812d57984e6ca 100644
--- a/llvm/lib/CodeGen/MIRVRegNamerUtils.cpp
+++ b/llvm/lib/CodeGen/MIRVRegNamerUtils.cpp
@@ -39,7 +39,7 @@ VRegRenamer::getVRegRenameMap(const std::vector<NamedVReg> &VRegs) {
   StringMap<unsigned> VRegNameCollisionMap;
 
   auto GetUniqueVRegName = [&VRegNameCollisionMap](const NamedVReg &Reg) {
-    if (VRegNameCollisionMap.find(Reg.getName()) == VRegNameCollisionMap.end())
+    if (!VRegNameCollisionMap.contains(Reg.getName()))
       VRegNameCollisionMap[Reg.getName()] = 0;
     const unsigned Counter = ++VRegNameCollisionMap[Reg.getName()];
     return Reg.getName() + "__" + std::to_string(Counter);
diff --git a/llvm/lib/CodeGen/MachineBlockPlacement.cpp b/llvm/lib/CodeGen/MachineBlockPlacement.cpp
index ac4d59ead4861..4f9f5ded367c2 100644
--- a/llvm/lib/CodeGen/MachineBlockPlacement.cpp
+++ b/llvm/lib/CodeGen/MachineBlockPlacement.cpp
@@ -2017,7 +2017,7 @@ MachineBlockPlacement::FallThroughGains(
      for (MachineBasicBlock *Succ : BestPred->successors()) {
        if ((Succ == NewTop) || (Succ == BestPred) || !LoopBlockSet.count(Succ))
          continue;
-       if (ComputedEdges.find(Succ) != ComputedEdges.end())
+       if (ComputedEdges.contains(Succ))
          continue;
        BlockChain *SuccChain = BlockToChain[Succ];
        if ((SuccChain && (Succ != *SuccChain->begin())) ||
diff --git a/llvm/lib/CodeGen/MachineFunction.cpp b/llvm/lib/CodeGen/MachineFunction.cpp
index a0e20437fbf37..4a908493e58ae 100644
--- a/llvm/lib/CodeGen/MachineFunction.cpp
+++ b/llvm/lib/CodeGen/MachineFunction.cpp
@@ -427,8 +427,7 @@ void MachineFunction::deleteMachineInstr(MachineInstr *MI) {
   // be triggered during the implementation of support for the
   // call site info of a new architecture. If the assertion is triggered,
   // back trace will tell where to insert a call to updateCallSiteInfo().
-  assert((!MI->isCandidateForCallSiteEntry() ||
-          CallSitesInfo.find(MI) == CallSitesInfo.end()) &&
+  assert((!MI->isCandidateForCallSiteEntry() || !CallSitesInfo.contains(MI)) &&
          "Call site info was not updated!");
   // Strip it for parts. The operand array and the MI object itself are
   // independently recyclable.
diff --git a/llvm/lib/CodeGen/MachineLICM.cpp b/llvm/lib/CodeGen/MachineLICM.cpp
index 1c09c01df3aae..ed3b3aa32aa04 100644
--- a/llvm/lib/CodeGen/MachineLICM.cpp
+++ b/llvm/lib/CodeGen/MachineLICM.cpp
@@ -866,7 +866,7 @@ MachineLICMBase::calcRegisterCost(const MachineInstr *MI, bool ConsiderSeen,
       continue;
     const int *PS = TRI->getRegClassPressureSets(RC);
     for (; *PS != -1; ++PS) {
-      if (Cost.find(*PS) == Cost.end())
+      if (!Cost.contains(*PS))
         Cost[*PS] = RCCost;
       else
         Cost[*PS] += RCCost;
diff --git a/llvm/lib/CodeGen/MachineOperand.cpp b/llvm/lib/CodeGen/MachineOperand.cpp
index 1178a93961054..5a43f69fe7187 100644
--- a/llvm/lib/CodeGen/MachineOperand.cpp
+++ b/llvm/lib/CodeGen/MachineOperand.cpp
@@ -991,7 +991,7 @@ void MachineOperand::print(raw_ostream &OS, ModuleSlotTracker &MST,
   case MachineOperand::MO_Predicate: {
     auto Pred = static_cast<CmpInst::Predicate>(getPredicate());
     OS << (CmpInst::isIntPredicate(Pred) ? "int" : "float") << "pred("
-       << CmpInst::getPredicateName(Pred) << ')';
+       << Pred << ')';
     break;
   }
   case MachineOperand::MO_ShuffleMask:
diff --git a/llvm/lib/CodeGen/MachinePassManager.cpp b/llvm/lib/CodeGen/MachinePassManager.cpp
index 039634f3d0477..439ff8babcc67 100644
--- a/llvm/lib/CodeGen/MachinePassManager.cpp
+++ b/llvm/lib/CodeGen/MachinePassManager.cpp
@@ -91,8 +91,8 @@ Error MachineFunctionPassManager::run(Module &M,
 
         // TODO: EmitSizeRemarks
         PreservedAnalyses PassPA = P->run(MF, MFAM);
-        PI.runAfterPass(*P, MF, PassPA);
         MFAM.invalidate(MF, PassPA);
+        PI.runAfterPass(*P, MF, PassPA);
       }
     }
   } while (true);
diff --git a/llvm/lib/CodeGen/MachineUniformityAnalysis.cpp b/llvm/lib/CodeGen/MachineUniformityAnalysis.cpp
index 2aa09f73ed2c8..ef67bae1c1af0 100644
--- a/llvm/lib/CodeGen/MachineUniformityAnalysis.cpp
+++ b/llvm/lib/CodeGen/MachineUniformityAnalysis.cpp
@@ -75,8 +75,6 @@ void llvm::GenericUniformityAnalysisImpl<MachineSSAContext>::pushUsers(
     Register Reg) {
   const auto &RegInfo = F.getRegInfo();
   for (MachineInstr &UserInstr : RegInfo.use_instructions(Reg)) {
-    if (isAlwaysUniform(UserInstr))
-      continue;
     if (markDivergent(UserInstr))
       Worklist.push_back(&UserInstr);
   }
@@ -115,6 +113,26 @@ bool llvm::GenericUniformityAnalysisImpl<MachineSSAContext>::usesValueFromCycle(
   return false;
 }
 
+template <>
+bool llvm::GenericUniformityAnalysisImpl<MachineSSAContext>::isDivergentUse(
+    const MachineOperand &U) const {
+  if (!U.isReg())
+    return false;
+
+  auto Reg = U.getReg();
+  if (isDivergent(Reg))
+    return true;
+
+  const auto &RegInfo = F.getRegInfo();
+  auto *Def = RegInfo.getOneDef(Reg);
+  if (!Def)
+    return true;
+
+  auto *DefInstr = Def->getParent();
+  auto *UseInstr = U.getParent();
+  return isTemporalDivergent(*UseInstr->getParent(), *DefInstr);
+}
+
 // This ensures explicit instantiation of
 // GenericUniformityAnalysisImpl::ImplDeleter::operator()
 template class llvm::GenericUniformityInfo<MachineSSAContext>;
diff --git a/llvm/lib/CodeGen/PseudoProbeInserter.cpp b/llvm/lib/CodeGen/PseudoProbeInserter.cpp
index 86ea3ec671785..913e0035b046f 100644
--- a/llvm/lib/CodeGen/PseudoProbeInserter.cpp
+++ b/llvm/lib/CodeGen/PseudoProbeInserter.cpp
@@ -128,10 +128,7 @@ class PseudoProbeInserter : public MachineFunctionPass {
 
 private:
   uint64_t getFuncGUID(Module *M, DILocation *DL) {
-    auto *SP = DL->getScope()->getSubprogram();
-    auto Name = SP->getLinkageName();
-    if (Name.empty())
-      Name = SP->getName();
+    auto Name = DL->getSubprogramLinkageName();
     return Function::getGUID(Name);
   }
 
diff --git a/llvm/lib/CodeGen/RegAllocGreedy.cpp b/llvm/lib/CodeGen/RegAllocGreedy.cpp
index b43a4d2a4b855..a9c85e558bd3c 100644
--- a/llvm/lib/CodeGen/RegAllocGreedy.cpp
+++ b/llvm/lib/CodeGen/RegAllocGreedy.cpp
@@ -1960,7 +1960,7 @@ unsigned RAGreedy::tryLastChanceRecoloring(const LiveInterval &VirtReg,
     // don't add it to NewVRegs because its physical register will be restored
     // below. Other vregs in CurrentNewVRegs are created by calling
     // selectOrSplit and should be added into NewVRegs.
-    for (Register &R : CurrentNewVRegs) {
+    for (Register R : CurrentNewVRegs) {
       if (RecoloringCandidates.count(&LIS->getInterval(R)))
         continue;
       NewVRegs.push_back(R);
diff --git a/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp b/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp
index 54c088bae8f83..7667b38c744ca 100644
--- a/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp
@@ -30,6 +30,7 @@
 #include "llvm/Analysis/AliasAnalysis.h"
 #include "llvm/Analysis/MemoryLocation.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
+#include "llvm/Analysis/ValueTracking.h"
 #include "llvm/Analysis/VectorUtils.h"
 #include "llvm/CodeGen/DAGCombine.h"
 #include "llvm/CodeGen/ISDOpcodes.h"
@@ -362,6 +363,11 @@ namespace {
     SDValue SplitIndexingFromLoad(LoadSDNode *LD);
     bool SliceUpLoad(SDNode *N);
 
+    // Looks up the chain to find a unique (unaliased) store feeding the passed
+    // load. If no such store is found, returns a nullptr.
+    // Note: This will look past a CALLSEQ_START if the load is chained to it so
+    //       so that it can find stack stores for byval params.
+    StoreSDNode *getUniqueStoreFeeding(LoadSDNode *LD, int64_t &Offset);
     // Scalars have size 0 to distinguish from singleton vectors.
     SDValue ForwardStoreValueToDirectLoad(LoadSDNode *LD);
     bool getTruncatedStoreValue(StoreSDNode *ST, SDValue &Val);
@@ -722,6 +728,11 @@ namespace {
     SDValue getMergeStoreChains(SmallVectorImpl<MemOpLink> &StoreNodes,
                                 unsigned NumStores);
 
+    /// Helper function for mergeConsecutiveStores which checks if all the store
+    /// nodes have the same underlying object. We can still reuse the first
+    /// store's pointer info if all the stores are from the same object.
+    bool hasSameUnderlyingObj(ArrayRef<MemOpLink> StoreNodes);
+
     /// This is a helper function for mergeConsecutiveStores. When the source
     /// elements of the consecutive stores are all constants or all extracted
     /// vector elements, try to merge them into one larger store introducing
@@ -2582,6 +2593,12 @@ static bool isADDLike(SDValue V, const SelectionDAG &DAG) {
   return false;
 }
 
+static bool
+areBitwiseNotOfEachother(SDValue Op0, SDValue Op1) {
+  return (isBitwiseNot(Op0) && Op0.getOperand(0) == Op1) ||
+         (isBitwiseNot(Op1) && Op1.getOperand(0) == Op0);
+}
+
 /// Try to fold a node that behaves like an ADD (note that N isn't necessarily
 /// an ISD::ADD here, it could for example be an ISD::OR if we know that there
 /// are no common bits set in the operands).
@@ -2606,6 +2623,10 @@ SDValue DAGCombiner::visitADDLike(SDNode *N) {
       !DAG.isConstantIntBuildVectorOrConstantInt(N1))
     return DAG.getNode(ISD::ADD, DL, VT, N1, N0);
 
+  if (areBitwiseNotOfEachother(N0, N1))
+    return DAG.getConstant(APInt::getAllOnes(VT.getScalarSizeInBits()),
+                           SDLoc(N), VT);
+
   // fold vector ops
   if (VT.isVector()) {
     if (SDValue FoldedVOp = SimplifyVBinOp(N, DL))
@@ -2671,12 +2692,22 @@ SDValue DAGCombiner::visitADDLike(SDNode *N) {
     // equivalent to (add x, c).
     // Reassociate (add (xor x, c), y) -> (add add(x, y), c)) if (xor x, c) is
     // equivalent to (add x, c).
+    // Do this optimization only when adding c does not introduce instructions
+    // for adding carries.
     auto ReassociateAddOr = [&](SDValue N0, SDValue N1) {
       if (isADDLike(N0, DAG) && N0.hasOneUse() &&
           isConstantOrConstantVector(N0.getOperand(1), /* NoOpaque */ true)) {
-        return DAG.getNode(ISD::ADD, DL, VT,
-                           DAG.getNode(ISD::ADD, DL, VT, N1, N0.getOperand(0)),
-                           N0.getOperand(1));
+        // If N0's type does not split or is a sign mask, it does not introduce
+        // add carry.
+        auto TyActn = TLI.getTypeAction(*DAG.getContext(), N0.getValueType());
+        bool NoAddCarry = TyActn == TargetLoweringBase::TypeLegal ||
+                          TyActn == TargetLoweringBase::TypePromoteInteger ||
+                          isMinSignedConstant(N0.getOperand(1));
+        if (NoAddCarry)
+          return DAG.getNode(
+              ISD::ADD, DL, VT,
+              DAG.getNode(ISD::ADD, DL, VT, N1, N0.getOperand(0)),
+              N0.getOperand(1));
       }
       return SDValue();
     };
@@ -5560,8 +5591,7 @@ SDValue DAGCombiner::hoistLogicOpWithSameOpcodeHands(SDNode *N) {
   EVT VT = N0.getValueType();
   unsigned LogicOpcode = N->getOpcode();
   unsigned HandOpcode = N0.getOpcode();
-  assert((LogicOpcode == ISD::AND || LogicOpcode == ISD::OR ||
-          LogicOpcode == ISD::XOR) && "Expected logic opcode");
+  assert(ISD::isBitwiseLogicOp(LogicOpcode) && "Expected logic opcode");
   assert(HandOpcode == N1.getOpcode() && "Bad input!");
 
   // Bail early if none of these transforms apply.
@@ -6420,12 +6450,6 @@ SDValue DAGCombiner::unfoldExtremeBitClearingToShifts(SDNode *N) {
 static SDValue combineShiftAnd1ToBitTest(SDNode *And, SelectionDAG &DAG) {
   assert(And->getOpcode() == ISD::AND && "Expected an 'and' op");
 
-  // This is probably not worthwhile without a supported type.
-  EVT VT = And->getValueType(0);
-  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
-  if (!TLI.isTypeLegal(VT))
-    return SDValue();
-
   // Look through an optional extension.
   SDValue And0 = And->getOperand(0), And1 = And->getOperand(1);
   if (And0.getOpcode() == ISD::ANY_EXTEND && And0.hasOneUse())
@@ -6453,13 +6477,17 @@ static SDValue combineShiftAnd1ToBitTest(SDNode *And, SelectionDAG &DAG) {
   if (Src.getOpcode() != ISD::SRL || !Src.hasOneUse())
     return SDValue();
 
+  // This is probably not worthwhile without a supported type.
+  EVT SrcVT = Src.getValueType();
+  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+  if (!TLI.isTypeLegal(SrcVT))
+    return SDValue();
+
   // We might have looked through casts that make this transform invalid.
-  // TODO: If the source type is wider than the result type, do the mask and
-  //       compare in the source type.
-  unsigned VTBitWidth = VT.getScalarSizeInBits();
+  unsigned BitWidth = SrcVT.getScalarSizeInBits();
   SDValue ShiftAmt = Src.getOperand(1);
   auto *ShiftAmtC = dyn_cast<ConstantSDNode>(ShiftAmt);
-  if (!ShiftAmtC || !ShiftAmtC->getAPIntValue().ult(VTBitWidth))
+  if (!ShiftAmtC || !ShiftAmtC->getAPIntValue().ult(BitWidth))
     return SDValue();
 
   // Set source to shift source.
@@ -6480,14 +6508,15 @@ static SDValue combineShiftAnd1ToBitTest(SDNode *And, SelectionDAG &DAG) {
   // and (not (srl X, C)), 1 --> (and X, 1<<C) == 0
   // and (srl (not X), C)), 1 --> (and X, 1<<C) == 0
   SDLoc DL(And);
-  SDValue X = DAG.getZExtOrTrunc(Src, DL, VT);
-  EVT CCVT = TLI.getSetCCResultType(DAG.getDataLayout(), *DAG.getContext(), VT);
+  SDValue X = DAG.getZExtOrTrunc(Src, DL, SrcVT);
+  EVT CCVT =
+      TLI.getSetCCResultType(DAG.getDataLayout(), *DAG.getContext(), SrcVT);
   SDValue Mask = DAG.getConstant(
-      APInt::getOneBitSet(VTBitWidth, ShiftAmtC->getZExtValue()), DL, VT);
-  SDValue NewAnd = DAG.getNode(ISD::AND, DL, VT, X, Mask);
-  SDValue Zero = DAG.getConstant(0, DL, VT);
+      APInt::getOneBitSet(BitWidth, ShiftAmtC->getZExtValue()), DL, SrcVT);
+  SDValue NewAnd = DAG.getNode(ISD::AND, DL, SrcVT, X, Mask);
+  SDValue Zero = DAG.getConstant(0, DL, SrcVT);
   SDValue Setcc = DAG.getSetCC(DL, CCVT, NewAnd, Zero, ISD::SETEQ);
-  return DAG.getZExtOrTrunc(Setcc, DL, VT);
+  return DAG.getZExtOrTrunc(Setcc, DL, And->getValueType(0));
 }
 
 /// For targets that support usubsat, match a bit-hack form of that operation
@@ -6530,9 +6559,8 @@ static SDValue foldAndToUsubsat(SDNode *N, SelectionDAG &DAG) {
 static SDValue foldLogicOfShifts(SDNode *N, SDValue LogicOp, SDValue ShiftOp,
                                  SelectionDAG &DAG) {
   unsigned LogicOpcode = N->getOpcode();
-  assert((LogicOpcode == ISD::AND || LogicOpcode == ISD::OR ||
-          LogicOpcode == ISD::XOR)
-         && "Expected bitwise logic operation");
+  assert(ISD::isBitwiseLogicOp(LogicOpcode) &&
+         "Expected bitwise logic operation");
 
   if (!LogicOp.hasOneUse() || !ShiftOp.hasOneUse())
     return SDValue();
@@ -6579,8 +6607,8 @@ static SDValue foldLogicOfShifts(SDNode *N, SDValue LogicOp, SDValue ShiftOp,
 static SDValue foldLogicTreeOfShifts(SDNode *N, SDValue LeftHand,
                                      SDValue RightHand, SelectionDAG &DAG) {
   unsigned LogicOpcode = N->getOpcode();
-  assert((LogicOpcode == ISD::AND || LogicOpcode == ISD::OR ||
-          LogicOpcode == ISD::XOR));
+  assert(ISD::isBitwiseLogicOp(LogicOpcode) &&
+         "Expected bitwise logic operation");
   if (LeftHand.getOpcode() != LogicOpcode ||
       RightHand.getOpcode() != LogicOpcode)
     return SDValue();
@@ -6625,6 +6653,10 @@ SDValue DAGCombiner::visitAND(SDNode *N) {
       !DAG.isConstantIntBuildVectorOrConstantInt(N1))
     return DAG.getNode(ISD::AND, SDLoc(N), VT, N1, N0);
 
+  if (areBitwiseNotOfEachother(N0, N1))
+    return DAG.getConstant(APInt::getZero(VT.getScalarSizeInBits()), SDLoc(N),
+                           VT);
+
   // fold vector ops
   if (VT.isVector()) {
     if (SDValue FoldedVOp = SimplifyVBinOp(N, SDLoc(N)))
@@ -6702,13 +6734,27 @@ SDValue DAGCombiner::visitAND(SDNode *N) {
       ISD::matchBinaryPredicate(N0.getOperand(1), N1, MatchSubset))
     return N1;
 
-  // fold (and (any_ext V), c) -> (zero_ext V) if 'and' only clears top bits.
   if (N1C && N0.getOpcode() == ISD::ANY_EXTEND) {
     SDValue N0Op0 = N0.getOperand(0);
+    EVT SrcVT = N0Op0.getValueType();
+    unsigned SrcBitWidth = SrcVT.getScalarSizeInBits();
     APInt Mask = ~N1C->getAPIntValue();
-    Mask = Mask.trunc(N0Op0.getScalarValueSizeInBits());
+    Mask = Mask.trunc(SrcBitWidth);
+
+    // fold (and (any_ext V), c) -> (zero_ext V) if 'and' only clears top bits.
     if (DAG.MaskedValueIsZero(N0Op0, Mask))
-      return DAG.getNode(ISD::ZERO_EXTEND, SDLoc(N), N0.getValueType(), N0Op0);
+      return DAG.getNode(ISD::ZERO_EXTEND, SDLoc(N), VT, N0Op0);
+
+    // fold (and (any_ext V), c) -> (zero_ext (and (trunc V), c)) if profitable.
+    if (N1C->getAPIntValue().countLeadingZeros() >= (BitWidth - SrcBitWidth) &&
+        TLI.isTruncateFree(VT, SrcVT) && TLI.isZExtFree(SrcVT, VT) &&
+        TLI.isTypeDesirableForOp(ISD::AND, SrcVT) &&
+        TLI.isNarrowingProfitable(VT, SrcVT)) {
+      SDLoc DL(N);
+      return DAG.getNode(ISD::ZERO_EXTEND, DL, VT,
+                         DAG.getNode(ISD::AND, DL, SrcVT, N0Op0,
+                                     DAG.getZExtOrTrunc(N1, DL, SrcVT)));
+    }
   }
 
   // fold (and (ext (and V, c1)), c2) -> (and (ext V), (and c1, (ext c2)))
@@ -11794,6 +11840,38 @@ SDValue DAGCombiner::visitVSELECT(SDNode *N) {
       }
     }
 
+    // Match VSELECTs with absolute difference patterns.
+    // (vselect (setcc a, b, set?gt), (sub a, b), (sub b, a)) --> (abd? a, b)
+    // (vselect (setcc a, b, set?ge), (sub a, b), (sub b, a)) --> (abd? a, b)
+    // (vselect (setcc a, b, set?lt), (sub b, a), (sub a, b)) --> (abd? a, b)
+    // (vselect (setcc a, b, set?le), (sub b, a), (sub a, b)) --> (abd? a, b)
+    if (N1.getOpcode() == ISD::SUB && N2.getOpcode() == ISD::SUB &&
+        N1.getOperand(0) == N2.getOperand(1) &&
+        N1.getOperand(1) == N2.getOperand(0)) {
+      bool IsSigned = isSignedIntSetCC(CC);
+      unsigned ABDOpc = IsSigned ? ISD::ABDS : ISD::ABDU;
+      if (hasOperation(ABDOpc, VT)) {
+        switch (CC) {
+        case ISD::SETGT:
+        case ISD::SETGE:
+        case ISD::SETUGT:
+        case ISD::SETUGE:
+          if (LHS == N1.getOperand(0) && RHS == N1.getOperand(1))
+            return DAG.getNode(ABDOpc, DL, VT, LHS, RHS);
+          break;
+        case ISD::SETLT:
+        case ISD::SETLE:
+        case ISD::SETULT:
+        case ISD::SETULE:
+          if (RHS == N1.getOperand(0) && LHS == N1.getOperand(1) )
+            return DAG.getNode(ABDOpc, DL, VT, LHS, RHS);
+          break;
+        default:
+          break;
+        }
+      }
+    }
+
     // Match VSELECTs into add with unsigned saturation.
     if (hasOperation(ISD::UADDSAT, VT)) {
       // Check if one of the arms of the VSELECT is vector with all bits set.
@@ -12455,8 +12533,7 @@ SDValue DAGCombiner::CombineZExtLogicopShiftLoad(SDNode *N) {
 
   // and/or/xor
   SDValue N0 = N->getOperand(0);
-  if (!(N0.getOpcode() == ISD::AND || N0.getOpcode() == ISD::OR ||
-        N0.getOpcode() == ISD::XOR) ||
+  if (!ISD::isBitwiseLogicOp(N0.getOpcode()) ||
       N0.getOperand(1).getOpcode() != ISD::Constant ||
       (LegalOperations && !TLI.isOperationLegal(N0.getOpcode(), VT)))
     return SDValue();
@@ -12936,8 +13013,7 @@ SDValue DAGCombiner::visitSIGN_EXTEND(SDNode *N) {
 
   // fold (sext (and/or/xor (load x), cst)) ->
   //      (and/or/xor (sextload x), (sext cst))
-  if ((N0.getOpcode() == ISD::AND || N0.getOpcode() == ISD::OR ||
-       N0.getOpcode() == ISD::XOR) &&
+  if (ISD::isBitwiseLogicOp(N0.getOpcode()) &&
       isa<LoadSDNode>(N0.getOperand(0)) &&
       N0.getOperand(1).getOpcode() == ISD::Constant &&
       (!LegalOperations && TLI.isOperationLegal(N0.getOpcode(), VT))) {
@@ -13126,14 +13202,15 @@ static SDValue widenAbs(SDNode *Extend, SelectionDAG &DAG) {
 SDValue DAGCombiner::visitZERO_EXTEND(SDNode *N) {
   SDValue N0 = N->getOperand(0);
   EVT VT = N->getValueType(0);
+  SDLoc DL(N);
 
   if (VT.isVector())
-    if (SDValue FoldedVOp = SimplifyVCastOp(N, SDLoc(N)))
+    if (SDValue FoldedVOp = SimplifyVCastOp(N, DL))
       return FoldedVOp;
 
   // zext(undef) = 0
   if (N0.isUndef())
-    return DAG.getConstant(0, SDLoc(N), VT);
+    return DAG.getConstant(0, DL, VT);
 
   if (SDValue Res = tryToFoldExtendOfConstant(N, TLI, DAG, LegalTypes))
     return Res;
@@ -13141,8 +13218,7 @@ SDValue DAGCombiner::visitZERO_EXTEND(SDNode *N) {
   // fold (zext (zext x)) -> (zext x)
   // fold (zext (aext x)) -> (zext x)
   if (N0.getOpcode() == ISD::ZERO_EXTEND || N0.getOpcode() == ISD::ANY_EXTEND)
-    return DAG.getNode(ISD::ZERO_EXTEND, SDLoc(N), VT,
-                       N0.getOperand(0));
+    return DAG.getNode(ISD::ZERO_EXTEND, DL, VT, N0.getOperand(0));
 
   // fold (zext (truncate x)) -> (zext x) or
   //      (zext (truncate x)) -> (truncate x)
@@ -13158,7 +13234,7 @@ SDValue DAGCombiner::visitZERO_EXTEND(SDNode *N) {
                         std::min(Op.getScalarValueSizeInBits(),
                                  VT.getScalarSizeInBits()));
     if (TruncatedBits.isSubsetOf(Known.Zero))
-      return DAG.getZExtOrTrunc(Op, SDLoc(N), VT);
+      return DAG.getZExtOrTrunc(Op, DL, VT);
   }
 
   // fold (zext (truncate x)) -> (and x, mask)
@@ -13184,9 +13260,9 @@ SDValue DAGCombiner::visitZERO_EXTEND(SDNode *N) {
       if (!LegalOperations || (TLI.isOperationLegal(ISD::AND, SrcVT) &&
                                TLI.isOperationLegal(ISD::ZERO_EXTEND, VT))) {
         SDValue Op = N0.getOperand(0);
-        Op = DAG.getZeroExtendInReg(Op, SDLoc(N), MinVT);
+        Op = DAG.getZeroExtendInReg(Op, DL, MinVT);
         AddToWorklist(Op.getNode());
-        SDValue ZExtOrTrunc = DAG.getZExtOrTrunc(Op, SDLoc(N), VT);
+        SDValue ZExtOrTrunc = DAG.getZExtOrTrunc(Op, DL, VT);
         // Transfer the debug info; the new node is equivalent to N0.
         DAG.transferDbgValues(N0, ZExtOrTrunc);
         return ZExtOrTrunc;
@@ -13194,9 +13270,9 @@ SDValue DAGCombiner::visitZERO_EXTEND(SDNode *N) {
     }
 
     if (!LegalOperations || TLI.isOperationLegal(ISD::AND, VT)) {
-      SDValue Op = DAG.getAnyExtOrTrunc(N0.getOperand(0), SDLoc(N), VT);
+      SDValue Op = DAG.getAnyExtOrTrunc(N0.getOperand(0), DL, VT);
       AddToWorklist(Op.getNode());
-      SDValue And = DAG.getZeroExtendInReg(Op, SDLoc(N), MinVT);
+      SDValue And = DAG.getZeroExtendInReg(Op, DL, MinVT);
       // We may safely transfer the debug info describing the truncate node over
       // to the equivalent and operation.
       DAG.transferDbgValues(N0, And);
@@ -13215,7 +13291,6 @@ SDValue DAGCombiner::visitZERO_EXTEND(SDNode *N) {
     SDValue X = N0.getOperand(0).getOperand(0);
     X = DAG.getAnyExtOrTrunc(X, SDLoc(X), VT);
     APInt Mask = N0.getConstantOperandAPInt(1).zext(VT.getSizeInBits());
-    SDLoc DL(N);
     return DAG.getNode(ISD::AND, DL, VT,
                        X, DAG.getConstant(Mask, DL, VT));
   }
@@ -13240,8 +13315,7 @@ SDValue DAGCombiner::visitZERO_EXTEND(SDNode *N) {
   //      (and/or/xor (zextload x), (zext cst))
   // Unless (and (load x) cst) will match as a zextload already and has
   // additional users.
-  if ((N0.getOpcode() == ISD::AND || N0.getOpcode() == ISD::OR ||
-       N0.getOpcode() == ISD::XOR) &&
+  if (ISD::isBitwiseLogicOp(N0.getOpcode()) &&
       isa<LoadSDNode>(N0.getOperand(0)) &&
       N0.getOperand(1).getOpcode() == ISD::Constant &&
       (!LegalOperations && TLI.isOperationLegal(N0.getOpcode(), VT))) {
@@ -13269,7 +13343,6 @@ SDValue DAGCombiner::visitZERO_EXTEND(SDNode *N) {
                                          LN00->getMemoryVT(),
                                          LN00->getMemOperand());
         APInt Mask = N0.getConstantOperandAPInt(1).zext(VT.getSizeInBits());
-        SDLoc DL(N);
         SDValue And = DAG.getNode(N0.getOpcode(), DL, VT,
                                   ExtLoad, DAG.getConstant(Mask, DL, VT));
         ExtendSetCCUses(SetCCs, N0.getOperand(0), ExtLoad, ISD::ZERO_EXTEND);
@@ -13323,7 +13396,6 @@ SDValue DAGCombiner::visitZERO_EXTEND(SDNode *N) {
       // that matter). Check to see that they are the same size. If so, we know
       // that the element size of the sext'd result matches the element size of
       // the compare operands.
-      SDLoc DL(N);
       if (VT.getSizeInBits() == N00VT.getSizeInBits()) {
         // zext(setcc) -> zext_in_reg(vsetcc) for vectors.
         SDValue VSetCC = DAG.getNode(ISD::SETCC, DL, VT, N0.getOperand(0),
@@ -13343,7 +13415,6 @@ SDValue DAGCombiner::visitZERO_EXTEND(SDNode *N) {
     }
 
     // zext(setcc x,y,cc) -> zext(select x, y, true, false, cc)
-    SDLoc DL(N);
     EVT N0VT = N0.getValueType();
     EVT N00VT = N0.getOperand(0).getValueType();
     if (SDValue SCC = SimplifySelectCC(
@@ -13356,29 +13427,29 @@ SDValue DAGCombiner::visitZERO_EXTEND(SDNode *N) {
 
   // (zext (shl (zext x), cst)) -> (shl (zext x), cst)
   if ((N0.getOpcode() == ISD::SHL || N0.getOpcode() == ISD::SRL) &&
-      isa<ConstantSDNode>(N0.getOperand(1)) &&
-      N0.getOperand(0).getOpcode() == ISD::ZERO_EXTEND &&
-      N0.hasOneUse()) {
+      !TLI.isZExtFree(N0, VT)) {
+    SDValue ShVal = N0.getOperand(0);
     SDValue ShAmt = N0.getOperand(1);
-    if (N0.getOpcode() == ISD::SHL) {
-      SDValue InnerZExt = N0.getOperand(0);
-      // If the original shl may be shifting out bits, do not perform this
-      // transformation.
-      unsigned KnownZeroBits = InnerZExt.getValueSizeInBits() -
-        InnerZExt.getOperand(0).getValueSizeInBits();
-      if (cast<ConstantSDNode>(ShAmt)->getAPIntValue().ugt(KnownZeroBits))
-        return SDValue();
-    }
-
-    SDLoc DL(N);
+    if (auto *ShAmtC = dyn_cast<ConstantSDNode>(ShAmt)) {
+      if (ShVal.getOpcode() == ISD::ZERO_EXTEND && N0.hasOneUse()) {
+        if (N0.getOpcode() == ISD::SHL) {
+          // If the original shl may be shifting out bits, do not perform this
+          // transformation.
+          // TODO: Add MaskedValueIsZero check.
+          unsigned KnownZeroBits = ShVal.getValueSizeInBits() -
+                                   ShVal.getOperand(0).getValueSizeInBits();
+          if (ShAmtC->getAPIntValue().ugt(KnownZeroBits))
+            return SDValue();
+        }
 
-    // Ensure that the shift amount is wide enough for the shifted value.
-    if (Log2_32_Ceil(VT.getSizeInBits()) > ShAmt.getValueSizeInBits())
-      ShAmt = DAG.getNode(ISD::ZERO_EXTEND, DL, MVT::i32, ShAmt);
+        // Ensure that the shift amount is wide enough for the shifted value.
+        if (Log2_32_Ceil(VT.getSizeInBits()) > ShAmt.getValueSizeInBits())
+          ShAmt = DAG.getNode(ISD::ZERO_EXTEND, DL, MVT::i32, ShAmt);
 
-    return DAG.getNode(N0.getOpcode(), DL, VT,
-                       DAG.getNode(ISD::ZERO_EXTEND, DL, VT, N0.getOperand(0)),
-                       ShAmt);
+        return DAG.getNode(N0.getOpcode(), DL, VT,
+                           DAG.getNode(ISD::ZERO_EXTEND, DL, VT, ShVal), ShAmt);
+      }
+    }
   }
 
   if (SDValue NewVSel = matchVSelectOpSizesWithSetCC(N))
@@ -14630,6 +14701,22 @@ SDValue DAGCombiner::visitBITCAST(SDNode *N) {
   if (N0.getOpcode() == ISD::BITCAST)
     return DAG.getBitcast(VT, N0.getOperand(0));
 
+  // fold (conv (logicop (conv x), (c))) -> (logicop x, (conv c))
+  // iff the current bitwise logicop type isn't legal
+  if (ISD::isBitwiseLogicOp(N0.getOpcode()) && VT.isInteger() &&
+      !TLI.isTypeLegal(N0.getOperand(0).getValueType())) {
+    auto IsFreeBitcast = [VT](SDValue V) {
+      return (V.getOpcode() == ISD::BITCAST &&
+              V.getOperand(0).getValueType() == VT) ||
+             (ISD::isBuildVectorOfConstantSDNodes(V.getNode()) &&
+              V->hasOneUse());
+    };
+    if (IsFreeBitcast(N0.getOperand(0)) && IsFreeBitcast(N0.getOperand(1)))
+      return DAG.getNode(N0.getOpcode(), SDLoc(N), VT,
+                         DAG.getBitcast(VT, N0.getOperand(0)),
+                         DAG.getBitcast(VT, N0.getOperand(1)));
+  }
+
   // fold (conv (load x)) -> (load (conv*)x)
   // If the resultant load doesn't need a higher alignment than the original!
   if (ISD::isNormalLoad(N0.getNode()) && N0.hasOneUse() &&
@@ -17749,11 +17836,53 @@ bool DAGCombiner::extendLoadedValueToExtension(LoadSDNode *LD, SDValue &Val) {
   return false;
 }
 
+StoreSDNode *DAGCombiner::getUniqueStoreFeeding(LoadSDNode *LD,
+                                                int64_t &Offset) {
+  SDValue Chain = LD->getOperand(0);
+
+  // Look through CALLSEQ_START.
+  if (Chain.getOpcode() == ISD::CALLSEQ_START)
+    Chain = Chain->getOperand(0);
+
+  StoreSDNode *ST = nullptr;
+  SmallVector<SDValue, 8> Aliases;
+  if (Chain.getOpcode() == ISD::TokenFactor) {
+    // Look for unique store within the TokenFactor.
+    for (SDValue Op : Chain->ops()) {
+      StoreSDNode *Store = dyn_cast<StoreSDNode>(Op.getNode());
+      if (!Store)
+        continue;
+      BaseIndexOffset BasePtrLD = BaseIndexOffset::match(LD, DAG);
+      BaseIndexOffset BasePtrST = BaseIndexOffset::match(Store, DAG);
+      if (!BasePtrST.equalBaseIndex(BasePtrLD, DAG, Offset))
+        continue;
+      // Make sure the store is not aliased with any nodes in TokenFactor.
+      GatherAllAliases(Store, Chain, Aliases);
+      if (Aliases.empty() ||
+          (Aliases.size() == 1 && Aliases.front().getNode() == Store))
+        ST = Store;
+      break;
+    }
+  } else {
+    StoreSDNode *Store = dyn_cast<StoreSDNode>(Chain.getNode());
+    if (Store) {
+      BaseIndexOffset BasePtrLD = BaseIndexOffset::match(LD, DAG);
+      BaseIndexOffset BasePtrST = BaseIndexOffset::match(Store, DAG);
+      if (BasePtrST.equalBaseIndex(BasePtrLD, DAG, Offset))
+        ST = Store;
+    }
+  }
+
+  return ST;
+}
+
 SDValue DAGCombiner::ForwardStoreValueToDirectLoad(LoadSDNode *LD) {
   if (OptLevel == CodeGenOpt::None || !LD->isSimple())
     return SDValue();
   SDValue Chain = LD->getOperand(0);
-  StoreSDNode *ST = dyn_cast<StoreSDNode>(Chain.getNode());
+  int64_t Offset;
+
+  StoreSDNode *ST = getUniqueStoreFeeding(LD, Offset);
   // TODO: Relax this restriction for unordered atomics (see D66309)
   if (!ST || !ST->isSimple() || ST->getAddressSpace() != LD->getAddressSpace())
     return SDValue();
@@ -17781,12 +17910,6 @@ SDValue DAGCombiner::ForwardStoreValueToDirectLoad(LoadSDNode *LD) {
   if (LdStScalable && DAG.getDataLayout().isBigEndian())
     return SDValue();
 
-  BaseIndexOffset BasePtrLD = BaseIndexOffset::match(LD, DAG);
-  BaseIndexOffset BasePtrST = BaseIndexOffset::match(ST, DAG);
-  int64_t Offset;
-  if (!BasePtrST.equalBaseIndex(BasePtrLD, DAG, Offset))
-    return SDValue();
-
   // Normalize for Endianness. After this Offset=0 will denote that the least
   // significant bit in the loaded value maps to the least significant bit in
   // the stored value). With Offset=n (for n > 0) the loaded value starts at the
@@ -18988,6 +19111,30 @@ SDValue DAGCombiner::getMergeStoreChains(SmallVectorImpl<MemOpLink> &StoreNodes,
   return DAG.getTokenFactor(StoreDL, Chains);
 }
 
+bool DAGCombiner::hasSameUnderlyingObj(ArrayRef<MemOpLink> StoreNodes) {
+  const Value *UnderlyingObj = nullptr;
+  for (const auto &MemOp : StoreNodes) {
+    const MachineMemOperand *MMO = MemOp.MemNode->getMemOperand();
+    // Pseudo value like stack frame has its own frame index and size, should
+    // not use the first store's frame index for other frames.
+    if (MMO->getPseudoValue())
+      return false;
+
+    if (!MMO->getValue())
+      return false;
+
+    const Value *Obj = getUnderlyingObject(MMO->getValue());
+
+    if (UnderlyingObj && UnderlyingObj != Obj)
+      return false;
+
+    if (!UnderlyingObj)
+      UnderlyingObj = Obj;
+  }
+
+  return true;
+}
+
 bool DAGCombiner::mergeStoresOfConstantsOrVecElts(
     SmallVectorImpl<MemOpLink> &StoreNodes, EVT MemVT, unsigned NumStores,
     bool IsConstantSrc, bool UseVector, bool UseTrunc) {
@@ -19139,13 +19286,21 @@ bool DAGCombiner::mergeStoresOfConstantsOrVecElts(
 
   LSBaseSDNode *FirstInChain = StoreNodes[0].MemNode;
   SDValue NewChain = getMergeStoreChains(StoreNodes, NumStores);
+  bool CanReusePtrInfo = hasSameUnderlyingObj(StoreNodes);
 
   // make sure we use trunc store if it's necessary to be legal.
+  // When generate the new widen store, if the first store's pointer info can
+  // not be reused, discard the pointer info except the address space because
+  // now the widen store can not be represented by the original pointer info
+  // which is for the narrow memory object.
   SDValue NewStore;
   if (!UseTrunc) {
-    NewStore = DAG.getStore(NewChain, DL, StoredVal, FirstInChain->getBasePtr(),
-                            FirstInChain->getPointerInfo(),
-                            FirstInChain->getAlign(), *Flags, AAInfo);
+    NewStore = DAG.getStore(
+        NewChain, DL, StoredVal, FirstInChain->getBasePtr(),
+        CanReusePtrInfo
+            ? FirstInChain->getPointerInfo()
+            : MachinePointerInfo(FirstInChain->getPointerInfo().getAddrSpace()),
+        FirstInChain->getAlign(), *Flags, AAInfo);
   } else { // Must be realized as a trunc store
     EVT LegalizedStoredValTy =
         TLI.getTypeToTransformTo(*DAG.getContext(), StoredVal.getValueType());
@@ -19156,8 +19311,11 @@ bool DAGCombiner::mergeStoresOfConstantsOrVecElts(
                         LegalizedStoredValTy);
     NewStore = DAG.getTruncStore(
         NewChain, DL, ExtendedStoreVal, FirstInChain->getBasePtr(),
-        FirstInChain->getPointerInfo(), StoredVal.getValueType() /*TVT*/,
-        FirstInChain->getAlign(), *Flags, AAInfo);
+        CanReusePtrInfo
+            ? FirstInChain->getPointerInfo()
+            : MachinePointerInfo(FirstInChain->getPointerInfo().getAddrSpace()),
+        StoredVal.getValueType() /*TVT*/, FirstInChain->getAlign(), *Flags,
+        AAInfo);
   }
 
   // Replace all merged stores with the new store.
@@ -19850,6 +20008,7 @@ bool DAGCombiner::tryStoreMergeOfLoads(SmallVectorImpl<MemOpLink> &StoreNodes,
     // using the first's chain is acceptable.
 
     SDValue NewStoreChain = getMergeStoreChains(StoreNodes, NumElem);
+    bool CanReusePtrInfo = hasSameUnderlyingObj(StoreNodes);
     AddToWorklist(NewStoreChain.getNode());
 
     MachineMemOperand::Flags LdMMOFlags =
@@ -19879,7 +20038,9 @@ bool DAGCombiner::tryStoreMergeOfLoads(SmallVectorImpl<MemOpLink> &StoreNodes,
       }
       NewStore = DAG.getStore(
           NewStoreChain, StoreDL, StoreOp, FirstInChain->getBasePtr(),
-          FirstInChain->getPointerInfo(), FirstStoreAlign, StMMOFlags);
+          CanReusePtrInfo ? FirstInChain->getPointerInfo()
+                          : MachinePointerInfo(FirstStoreAS),
+          FirstStoreAlign, StMMOFlags);
     } else { // This must be the truncstore/extload case
       EVT ExtendedTy =
           TLI.getTypeToTransformTo(*DAG.getContext(), JointMemOpVT);
@@ -19889,8 +20050,10 @@ bool DAGCombiner::tryStoreMergeOfLoads(SmallVectorImpl<MemOpLink> &StoreNodes,
                                FirstLoadAlign, LdMMOFlags);
       NewStore = DAG.getTruncStore(
           NewStoreChain, StoreDL, NewLoad, FirstInChain->getBasePtr(),
-          FirstInChain->getPointerInfo(), JointMemOpVT,
-          FirstInChain->getAlign(), FirstInChain->getMemOperand()->getFlags());
+          CanReusePtrInfo ? FirstInChain->getPointerInfo()
+                          : MachinePointerInfo(FirstStoreAS),
+          JointMemOpVT, FirstInChain->getAlign(),
+          FirstInChain->getMemOperand()->getFlags());
     }
 
     // Transfer chain users from old loads to the new load.
@@ -20229,9 +20392,13 @@ SDValue DAGCombiner::visitSTORE(SDNode *N) {
   }
 
   // If this is a load followed by a store to the same location, then the store
-  // is dead/noop.
+  // is dead/noop. Peek through any truncates if canCombineTruncStore failed.
+  // TODO: Add big-endian truncate support with test coverage.
   // TODO: Can relax for unordered atomics (see D66309)
-  if (LoadSDNode *Ld = dyn_cast<LoadSDNode>(Value)) {
+  SDValue TruncVal = DAG.getDataLayout().isLittleEndian()
+                         ? peekThroughTruncates(Value)
+                         : Value;
+  if (auto *Ld = dyn_cast<LoadSDNode>(TruncVal)) {
     if (Ld->getBasePtr() == Ptr && ST->getMemoryVT() == Ld->getMemoryVT() &&
         ST->isUnindexed() && ST->isSimple() &&
         Ld->getAddressSpace() == ST->getAddressSpace() &&
@@ -20820,6 +20987,20 @@ SDValue DAGCombiner::visitINSERT_VECTOR_ELT(SDNode *N) {
             return NewShuffle;
       }
 
+      // If all insertions are zero value, try to convert to AND mask.
+      // TODO: Do this for -1 with OR mask?
+      if (!LegalOperations && llvm::isNullConstant(InVal) &&
+          all_of(Ops, [InVal](SDValue Op) { return !Op || Op == InVal; }) &&
+          count_if(Ops, [InVal](SDValue Op) { return Op == InVal; }) >= 2) {
+        SDValue Zero = DAG.getConstant(0, DL, MaxEltVT);
+        SDValue AllOnes = DAG.getAllOnesConstant(DL, MaxEltVT);
+        SmallVector<SDValue, 8> Mask(NumElts);
+        for (unsigned I = 0; I != NumElts; ++I)
+          Mask[I] = Ops[I] ? Zero : AllOnes;
+        return DAG.getNode(ISD::AND, DL, VT, CurVec,
+                           DAG.getBuildVector(VT, DL, Mask));
+      }
+
       // Failed to find a match in the chain - bail.
       break;
     }
@@ -23447,7 +23628,7 @@ SDValue DAGCombiner::visitEXTRACT_SUBVECTOR(SDNode *N) {
         if (NumElems == 1) {
           SDValue Src = V->getOperand(IdxVal);
           if (EltVT != Src.getValueType())
-            Src = DAG.getNode(ISD::TRUNCATE, SDLoc(N), InVT, Src);
+            Src = DAG.getNode(ISD::TRUNCATE, SDLoc(N), EltVT, Src);
           return DAG.getBitcast(NVT, Src);
         }
 
diff --git a/llvm/lib/CodeGen/SelectionDAG/FastISel.cpp b/llvm/lib/CodeGen/SelectionDAG/FastISel.cpp
index 2f2ae6e29855b..5f7e3fe8b1665 100644
--- a/llvm/lib/CodeGen/SelectionDAG/FastISel.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/FastISel.cpp
@@ -454,8 +454,7 @@ bool FastISel::selectBinaryOp(const User *I, unsigned ISDOpcode) {
   if (!TLI.isTypeLegal(VT)) {
     // MVT::i1 is special. Allow AND, OR, or XOR because they
     // don't require additional zeroing, which makes them easy.
-    if (VT == MVT::i1 && (ISDOpcode == ISD::AND || ISDOpcode == ISD::OR ||
-                          ISDOpcode == ISD::XOR))
+    if (VT == MVT::i1 && ISD::isBitwiseLogicOp(ISDOpcode))
       VT = TLI.getTypeToTransformTo(I->getContext(), VT);
     else
       return false;
diff --git a/llvm/lib/CodeGen/SelectionDAG/FunctionLoweringInfo.cpp b/llvm/lib/CodeGen/SelectionDAG/FunctionLoweringInfo.cpp
index 476648668e3ba..76ebdb471606c 100644
--- a/llvm/lib/CodeGen/SelectionDAG/FunctionLoweringInfo.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/FunctionLoweringInfo.cpp
@@ -13,7 +13,7 @@
 
 #include "llvm/CodeGen/FunctionLoweringInfo.h"
 #include "llvm/ADT/APInt.h"
-#include "llvm/Analysis/LegacyDivergenceAnalysis.h"
+#include "llvm/Analysis/UniformityAnalysis.h"
 #include "llvm/CodeGen/Analysis.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
@@ -83,7 +83,7 @@ void FunctionLoweringInfo::set(const Function &fn, MachineFunction &mf,
   TLI = MF->getSubtarget().getTargetLowering();
   RegInfo = &MF->getRegInfo();
   const TargetFrameLowering *TFI = MF->getSubtarget().getFrameLowering();
-  DA = DAG->getDivergenceAnalysis();
+  UA = DAG->getUniformityInfo();
 
   // Check whether the function can return without sret-demotion.
   SmallVector<ISD::OutputArg, 4> Outs;
@@ -381,8 +381,8 @@ Register FunctionLoweringInfo::CreateRegs(Type *Ty, bool isDivergent) {
 }
 
 Register FunctionLoweringInfo::CreateRegs(const Value *V) {
-  return CreateRegs(V->getType(), DA && DA->isDivergent(V) &&
-                    !TLI->requiresUniformRegister(*MF, V));
+  return CreateRegs(V->getType(), UA && UA->isDivergent(V) &&
+                                      !TLI->requiresUniformRegister(*MF, V));
 }
 
 /// GetLiveOutRegInfo - Gets LiveOutInfo for a register, returning NULL if the
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
index 29c7ddcd8b963..5cf9497069f78 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
@@ -1311,8 +1311,8 @@ SelectionDAG::SelectionDAG(const TargetMachine &tm, CodeGenOpt::Level OL)
 void SelectionDAG::init(MachineFunction &NewMF,
                         OptimizationRemarkEmitter &NewORE, Pass *PassPtr,
                         const TargetLibraryInfo *LibraryInfo,
-                        LegacyDivergenceAnalysis *Divergence,
-                        ProfileSummaryInfo *PSIin, BlockFrequencyInfo *BFIin,
+                        UniformityInfo *NewUA, ProfileSummaryInfo *PSIin,
+                        BlockFrequencyInfo *BFIin,
                         FunctionVarLocs const *VarLocs) {
   MF = &NewMF;
   SDAGISelPass = PassPtr;
@@ -1321,7 +1321,7 @@ void SelectionDAG::init(MachineFunction &NewMF,
   TSI = getSubtarget().getSelectionDAGInfo();
   LibInfo = LibraryInfo;
   Context = &MF->getFunction().getContext();
-  DA = Divergence;
+  UA = NewUA;
   PSI = PSIin;
   BFI = BFIin;
   FnVarLocs = VarLocs;
@@ -2175,7 +2175,7 @@ SDValue SelectionDAG::getRegister(unsigned RegNo, EVT VT) {
     return SDValue(E, 0);
 
   auto *N = newSDNode<RegisterSDNode>(RegNo, VT);
-  N->SDNodeBits.IsDivergent = TLI->isSDNodeSourceOfDivergence(N, FLI, DA);
+  N->SDNodeBits.IsDivergent = TLI->isSDNodeSourceOfDivergence(N, FLI, UA);
   CSEMap.InsertNode(N, IP);
   InsertNode(N);
   return SDValue(N, 0);
@@ -9422,8 +9422,8 @@ SDValue SelectionDAG::getNode(unsigned Opcode, const SDLoc &DL, SDVTList VTList,
            "STRICT_FP_EXTEND result type should be vector iff the operand "
            "type is vector!");
     assert((!VTList.VTs[0].isVector() ||
-            VTList.VTs[0].getVectorNumElements() ==
-            Ops[1].getValueType().getVectorNumElements()) &&
+            VTList.VTs[0].getVectorElementCount() ==
+                Ops[1].getValueType().getVectorElementCount()) &&
            "Vector element count mismatch!");
     assert(Ops[1].getValueType().bitsLT(VTList.VTs[0]) &&
            "Invalid fpext node, dst <= src!");
@@ -10659,11 +10659,11 @@ class RAUOVWUpdateListener : public SelectionDAG::DAGUpdateListener {
 
 bool SelectionDAG::calculateDivergence(SDNode *N) {
   if (TLI->isSDNodeAlwaysUniform(N)) {
-    assert(!TLI->isSDNodeSourceOfDivergence(N, FLI, DA) &&
+    assert(!TLI->isSDNodeSourceOfDivergence(N, FLI, UA) &&
            "Conflicting divergence information!");
     return false;
   }
-  if (TLI->isSDNodeSourceOfDivergence(N, FLI, DA))
+  if (TLI->isSDNodeSourceOfDivergence(N, FLI, UA))
     return true;
   for (const auto &Op : N->ops()) {
     if (Op.Val.getValueType() != MVT::Other && Op.getNode()->isDivergent())
@@ -11040,6 +11040,12 @@ SDValue llvm::peekThroughExtractSubvectors(SDValue V) {
   return V;
 }
 
+SDValue llvm::peekThroughTruncates(SDValue V) {
+  while (V.getOpcode() == ISD::TRUNCATE)
+    V = V.getOperand(0);
+  return V;
+}
+
 bool llvm::isBitwiseNot(SDValue V, bool AllowUndefs) {
   if (V.getOpcode() != ISD::XOR)
     return false;
@@ -12158,7 +12164,7 @@ void SelectionDAG::createOperands(SDNode *Node, ArrayRef<SDValue> Vals) {
   Node->NumOperands = Vals.size();
   Node->OperandList = Ops;
   if (!TLI->isSDNodeAlwaysUniform(Node)) {
-    IsDivergent |= TLI->isSDNodeSourceOfDivergence(Node, FLI, DA);
+    IsDivergent |= TLI->isSDNodeSourceOfDivergence(Node, FLI, UA);
     Node->SDNodeBits.IsDivergent = IsDivergent;
   }
   checkForCycles(Node);
@@ -12223,10 +12229,16 @@ void SelectionDAG::copyExtraInfo(SDNode *From, SDNode *To) {
 
   // Use of operator[] on the DenseMap may cause an insertion, which invalidates
   // the iterator, hence the need to make a copy to prevent a use-after-free.
-  NodeExtraInfo Copy = I->second;
-  if (LLVM_LIKELY(!Copy.PCSections)) {
+  NodeExtraInfo NEI = I->second;
+  if (LLVM_LIKELY(!NEI.PCSections)) {
     // No deep copy required for the types of extra info set.
-    SDEI[To] = std::move(Copy);
+    //
+    // FIXME: Investigate if other types of extra info also need deep copy. This
+    // depends on the types of nodes they can be attached to: if some extra info
+    // is only ever attached to nodes where a replacement To node is always the
+    // node where later use and propagation of the extra info has the intended
+    // semantics, no deep copy is required.
+    SDEI[To] = std::move(NEI);
     return;
   }
 
@@ -12239,41 +12251,68 @@ void SelectionDAG::copyExtraInfo(SDNode *From, SDNode *To) {
   // In the first step pre-populate the visited set with the nodes reachable
   // from the old From node. This avoids copying NodeExtraInfo to parts of the
   // DAG that is not new and should be left untouched.
-  DenseSet<const SDNode *> Visited;
-  auto VisitFrom = [&Visited](auto &&Self, SDNode *N, int Depth) {
-    constexpr int MaxDepth = 16;
-    if (Depth >= MaxDepth)
+  SmallVector<const SDNode *> Leafs{From}; // Leafs reachable with VisitFrom.
+  DenseSet<const SDNode *> FromReach; // The set of nodes reachable from From.
+  auto VisitFrom = [&](auto &&Self, const SDNode *N, int MaxDepth) {
+    if (MaxDepth == 0) {
+      // Remember this node in case we need to increase MaxDepth and continue
+      // populating FromReach from this node.
+      Leafs.emplace_back(N);
       return;
-    if (!Visited.insert(N).second)
+    }
+    if (!FromReach.insert(N).second)
       return;
     for (const SDValue &Op : N->op_values())
-      Self(Self, Op.getNode(), Depth + 1);
+      Self(Self, Op.getNode(), MaxDepth - 1);
   };
-  VisitFrom(VisitFrom, From, 0);
 
   // Copy extra info to To and all its transitive operands (that are new).
-  auto DeepCopyTo = [this, &Copy, &Visited](auto &&Self, SDNode *To) {
-    if (!Visited.insert(To).second)
+  SmallPtrSet<const SDNode *, 8> Visited;
+  auto DeepCopyTo = [&](auto &&Self, const SDNode *N) {
+    if (FromReach.contains(N))
+      return true;
+    if (!Visited.insert(N).second)
       return true;
-    if (getEntryNode().getNode() == To) {
-      // This should not happen - and if it did, that means From has a depth
-      // greater or equal to MaxDepth, and VisitFrom() could not visit all
-      // common operands. As a result, we're able to reach the entry node.
-      assert(false && "Too complex 'From' node - increase MaxDepth?");
+    if (getEntryNode().getNode() == N)
       return false;
-    }
-    for (const SDValue &Op : To->op_values()) {
+    for (const SDValue &Op : N->op_values()) {
       if (!Self(Self, Op.getNode()))
         return false;
     }
-    SDEI[To] = Copy;
+    // Copy only if entry node was not reached.
+    SDEI[N] = NEI;
     return true;
   };
 
-  if (LLVM_UNLIKELY(!DeepCopyTo(DeepCopyTo, To))) {
-    // Fallback - see assert above.
-    SDEI[To] = std::move(Copy);
-  }
+  // We first try with a lower MaxDepth, assuming that the path to common
+  // operands between From and To is relatively short. This significantly
+  // improves performance in the common case. The initial MaxDepth is big
+  // enough to avoid retry in the common case; the last MaxDepth is large
+  // enough to avoid having to use the fallback below (and protects from
+  // potential stack exhaustion from recursion).
+  for (int PrevDepth = 0, MaxDepth = 16; MaxDepth <= 1024;
+       PrevDepth = MaxDepth, MaxDepth *= 2, Visited.clear()) {
+    // StartFrom is the previous (or initial) set of leafs reachable at the
+    // previous maximum depth.
+    SmallVector<const SDNode *> StartFrom;
+    std::swap(StartFrom, Leafs);
+    for (const SDNode *N : StartFrom)
+      VisitFrom(VisitFrom, N, MaxDepth - PrevDepth);
+    if (LLVM_LIKELY(DeepCopyTo(DeepCopyTo, To)))
+      return;
+    // This should happen very rarely (reached the entry node).
+    LLVM_DEBUG(dbgs() << __func__ << ": MaxDepth=" << MaxDepth << " too low\n");
+    assert(!Leafs.empty());
+  }
+
+  // This should not happen - but if it did, that means the subgraph reachable
+  // from From has depth greater or equal to maximum MaxDepth, and VisitFrom()
+  // could not visit all reachable common operands. Consequently, we were able
+  // to reach the entry node.
+  errs() << "warning: incomplete propagation of SelectionDAG::NodeExtraInfo\n";
+  assert(false && "From subgraph too complex - increase max. MaxDepth?");
+  // Best-effort fallback if assertions disabled.
+  SDEI[To] = std::move(NEI);
 }
 
 #ifndef NDEBUG
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index aac315d081808..37fbe251cda40 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -1145,8 +1145,9 @@ void SelectionDAGBuilder::visit(const Instruction &I) {
          It != End; ++It) {
       auto *Var = FnVarLocs->getDILocalVariable(It->VariableID);
       dropDanglingDebugInfo(Var, It->Expr);
-      if (!handleDebugValue(It->V, Var, It->Expr, It->DL, SDNodeOrder,
-                            /*IsVariadic=*/false))
+      SmallVector<Value *> Values(It->Values.location_ops());
+      if (!handleDebugValue(Values, Var, It->Expr, It->DL, SDNodeOrder,
+                            It->Values.hasArgList()))
         addDanglingDebugInfo(It, SDNodeOrder);
     }
   }
@@ -1206,27 +1207,46 @@ void SelectionDAGBuilder::visit(unsigned Opcode, const User &I) {
   }
 }
 
+static bool handleDanglingVariadicDebugInfo(SelectionDAG &DAG,
+                                            DILocalVariable *Variable,
+                                            DebugLoc DL, unsigned Order,
+                                            RawLocationWrapper Values,
+                                            DIExpression *Expression) {
+  if (!Values.hasArgList())
+    return false;
+  // For variadic dbg_values we will now insert an undef.
+  // FIXME: We can potentially recover these!
+  SmallVector<SDDbgOperand, 2> Locs;
+  for (const Value *V : Values.location_ops()) {
+    auto *Undef = UndefValue::get(V->getType());
+    Locs.push_back(SDDbgOperand::fromConst(Undef));
+  }
+  SDDbgValue *SDV = DAG.getDbgValueList(Variable, Expression, Locs, {},
+                                        /*IsIndirect=*/false, DL, Order,
+                                        /*IsVariadic=*/true);
+  DAG.AddDbgValue(SDV, /*isParameter=*/false);
+  return true;
+}
+
 void SelectionDAGBuilder::addDanglingDebugInfo(const VarLocInfo *VarLoc,
                                                unsigned Order) {
-  DanglingDebugInfoMap[VarLoc->V].emplace_back(VarLoc, Order);
+  if (!handleDanglingVariadicDebugInfo(
+          DAG,
+          const_cast<DILocalVariable *>(DAG.getFunctionVarLocs()
+                                            ->getVariable(VarLoc->VariableID)
+                                            .getVariable()),
+          VarLoc->DL, Order, VarLoc->Values, VarLoc->Expr)) {
+    DanglingDebugInfoMap[VarLoc->Values.getVariableLocationOp(0)].emplace_back(
+        VarLoc, Order);
+  }
 }
 
 void SelectionDAGBuilder::addDanglingDebugInfo(const DbgValueInst *DI,
                                                unsigned Order) {
   // We treat variadic dbg_values differently at this stage.
-  if (DI->hasArgList()) {
-    // For variadic dbg_values we will now insert an undef.
-    // FIXME: We can potentially recover these!
-    SmallVector<SDDbgOperand, 2> Locs;
-    for (const Value *V : DI->getValues()) {
-      auto Undef = UndefValue::get(V->getType());
-      Locs.push_back(SDDbgOperand::fromConst(Undef));
-    }
-    SDDbgValue *SDV = DAG.getDbgValueList(
-        DI->getVariable(), DI->getExpression(), Locs, {},
-        /*IsIndirect=*/false, DI->getDebugLoc(), Order, /*IsVariadic=*/true);
-    DAG.AddDbgValue(SDV, /*isParameter=*/false);
-  } else {
+  if (!handleDanglingVariadicDebugInfo(
+          DAG, DI->getVariable(), DI->getDebugLoc(), Order,
+          DI->getWrappedLocation(), DI->getExpression())) {
     // TODO: Dangling debug info will eventually either be resolved or produce
     // an Undef DBG_VALUE. However in the resolution case, a gap may appear
     // between the original dbg.value location and its resolved DBG_VALUE,
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
index 8f6dae9cb4e86..3627d91005d91 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
@@ -131,7 +131,7 @@ class SelectionDAGBuilder {
     Value *getVariableLocationOp(unsigned Idx) const {
       assert(Idx == 0 && "Dangling variadic debug values not supported yet");
       if (Info.is<VarLocTy>())
-        return Info.get<VarLocTy>()->V;
+        return Info.get<VarLocTy>()->Values.getVariableLocationOp(Idx);
       return Info.get<DbgValTy>()->getVariableLocationOp(Idx);
     }
     DebugLoc getDebugLoc() const {
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp
index d4452fa8c9d0c..8d1cc04607bf4 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp
@@ -26,11 +26,11 @@
 #include "llvm/Analysis/BranchProbabilityInfo.h"
 #include "llvm/Analysis/CFG.h"
 #include "llvm/Analysis/LazyBlockFrequencyInfo.h"
-#include "llvm/Analysis/LegacyDivergenceAnalysis.h"
 #include "llvm/Analysis/OptimizationRemarkEmitter.h"
 #include "llvm/Analysis/ProfileSummaryInfo.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
+#include "llvm/Analysis/UniformityAnalysis.h"
 #include "llvm/CodeGen/AssignmentTrackingAnalysis.h"
 #include "llvm/CodeGen/CodeGenCommonISel.h"
 #include "llvm/CodeGen/FastISel.h"
@@ -425,9 +425,10 @@ bool SelectionDAGISel::runOnMachineFunction(MachineFunction &mf) {
 
   LLVM_DEBUG(dbgs() << "\n\n\n=== " << Fn.getName() << "\n");
 
-  CurDAG->init(*MF, *ORE, this, LibInfo,
-               getAnalysisIfAvailable<LegacyDivergenceAnalysis>(), PSI, BFI,
-               FnVarLocs);
+  UniformityInfo *UA = nullptr;
+  if (auto *UAPass = getAnalysisIfAvailable<UniformityInfoWrapperPass>())
+    UA = &UAPass->getUniformityInfo();
+  CurDAG->init(*MF, *ORE, this, LibInfo, UA, PSI, BFI, FnVarLocs);
   FuncInfo->set(Fn, *MF, CurDAG);
   SwiftError->setFunction(*MF);
 
@@ -1367,9 +1368,11 @@ static void processSingleLocVars(FunctionLoweringInfo &FuncInfo,
                                  FunctionVarLocs const *FnVarLocs) {
   for (auto It = FnVarLocs->single_locs_begin(),
             End = FnVarLocs->single_locs_end();
-       It != End; ++It)
-    processDbgDeclare(FuncInfo, It->V, It->Expr,
+       It != End; ++It) {
+    assert(!It->Values.hasArgList() && "Single loc variadic ops not supported");
+    processDbgDeclare(FuncInfo, It->Values.getVariableLocationOp(0), It->Expr,
                       FnVarLocs->getDILocalVariable(It->VariableID), It->DL);
+  }
 }
 
 void SelectionDAGISel::SelectAllBasicBlocks(const Function &Fn) {
diff --git a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
index b688463332107..6d0de75579a60 100644
--- a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
@@ -552,18 +552,19 @@ bool TargetLowering::ShrinkDemandedConstant(SDValue Op,
 /// This uses isZExtFree and ZERO_EXTEND for the widening cast, but it could be
 /// generalized for targets with other types of implicit widening casts.
 bool TargetLowering::ShrinkDemandedOp(SDValue Op, unsigned BitWidth,
-                                      const APInt &Demanded,
+                                      const APInt &DemandedBits,
                                       TargetLoweringOpt &TLO) const {
   assert(Op.getNumOperands() == 2 &&
          "ShrinkDemandedOp only supports binary operators!");
   assert(Op.getNode()->getNumValues() == 1 &&
          "ShrinkDemandedOp only supports nodes with one result!");
 
+  EVT VT = Op.getValueType();
   SelectionDAG &DAG = TLO.DAG;
   SDLoc dl(Op);
 
   // Early return, as this function cannot handle vector types.
-  if (Op.getValueType().isVector())
+  if (VT.isVector())
     return false;
 
   // Don't do this if the node has another user, which may require the
@@ -574,20 +575,18 @@ bool TargetLowering::ShrinkDemandedOp(SDValue Op, unsigned BitWidth,
   // Search for the smallest integer type with free casts to and from
   // Op's type. For expedience, just check power-of-2 integer types.
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
-  unsigned DemandedSize = Demanded.getActiveBits();
-  unsigned SmallVTBits = DemandedSize;
-  SmallVTBits = llvm::bit_ceil(SmallVTBits);
-  for (; SmallVTBits < BitWidth; SmallVTBits = NextPowerOf2(SmallVTBits)) {
+  unsigned DemandedSize = DemandedBits.getActiveBits();
+  for (unsigned SmallVTBits = llvm::bit_ceil(DemandedSize);
+       SmallVTBits < BitWidth; SmallVTBits = NextPowerOf2(SmallVTBits)) {
     EVT SmallVT = EVT::getIntegerVT(*DAG.getContext(), SmallVTBits);
-    if (TLI.isTruncateFree(Op.getValueType(), SmallVT) &&
-        TLI.isZExtFree(SmallVT, Op.getValueType())) {
+    if (TLI.isTruncateFree(VT, SmallVT) && TLI.isZExtFree(SmallVT, VT)) {
       // We found a type with free casts.
       SDValue X = DAG.getNode(
           Op.getOpcode(), dl, SmallVT,
           DAG.getNode(ISD::TRUNCATE, dl, SmallVT, Op.getOperand(0)),
           DAG.getNode(ISD::TRUNCATE, dl, SmallVT, Op.getOperand(1)));
       assert(DemandedSize <= SmallVTBits && "Narrowed below demanded bits?");
-      SDValue Z = DAG.getNode(ISD::ANY_EXTEND, dl, Op.getValueType(), X);
+      SDValue Z = DAG.getNode(ISD::ANY_EXTEND, dl, VT, X);
       return TLO.CombineTo(Op, Z);
     }
   }
@@ -7985,7 +7984,7 @@ SDValue TargetLowering::expandFMINNUM_FMAXNUM(SDNode *Node,
   }
 
   // If the target has FMINIMUM/FMAXIMUM but not FMINNUM/FMAXNUM use that
-  // instead if there are no NaNs and there can't be an incompatiable zero
+  // instead if there are no NaNs and there can't be an incompatible zero
   // compare: at least one operand isn't +/-0, or there are no signed-zeros.
   if (Node->getFlags().hasNoNaNs() &&
       (Node->getFlags().hasNoSignedZeros() ||
@@ -8150,6 +8149,19 @@ SDValue TargetLowering::expandIS_FPCLASS(EVT ResultVT, SDValue Op,
     appendResult(PartialRes);
   }
 
+  if (unsigned PartialCheck = Test & fcSubnormal) {
+    // issubnormal(V) ==> unsigned(abs(V) - 1) < (all mantissa bits set)
+    // issubnormal(V) && V>0 ==> unsigned(V - 1) < (all mantissa bits set)
+    SDValue V = (PartialCheck == fcPosSubnormal) ? OpAsInt : AbsV;
+    SDValue MantissaV = DAG.getConstant(AllOneMantissa, DL, IntVT);
+    SDValue VMinusOneV =
+        DAG.getNode(ISD::SUB, DL, IntVT, V, DAG.getConstant(1, DL, IntVT));
+    PartialRes = DAG.getSetCC(DL, ResultVT, VMinusOneV, MantissaV, ISD::SETULT);
+    if (PartialCheck == fcNegSubnormal)
+      PartialRes = DAG.getNode(ISD::AND, DL, ResultVT, PartialRes, SignV);
+    appendResult(PartialRes);
+  }
+
   if (unsigned PartialCheck = Test & fcInf) {
     if (PartialCheck == fcPosInf)
       PartialRes = DAG.getSetCC(DL, ResultVT, OpAsInt, InfV, ISD::SETEQ);
@@ -8194,19 +8206,6 @@ SDValue TargetLowering::expandIS_FPCLASS(EVT ResultVT, SDValue Op,
     appendResult(PartialRes);
   }
 
-  if (unsigned PartialCheck = Test & fcSubnormal) {
-    // issubnormal(V) ==> unsigned(abs(V) - 1) < (all mantissa bits set)
-    // issubnormal(V) && V>0 ==> unsigned(V - 1) < (all mantissa bits set)
-    SDValue V = (PartialCheck == fcPosSubnormal) ? OpAsInt : AbsV;
-    SDValue MantissaV = DAG.getConstant(AllOneMantissa, DL, IntVT);
-    SDValue VMinusOneV =
-        DAG.getNode(ISD::SUB, DL, IntVT, V, DAG.getConstant(1, DL, IntVT));
-    PartialRes = DAG.getSetCC(DL, ResultVT, VMinusOneV, MantissaV, ISD::SETULT);
-    if (PartialCheck == fcNegSubnormal)
-      PartialRes = DAG.getNode(ISD::AND, DL, ResultVT, PartialRes, SignV);
-    appendResult(PartialRes);
-  }
-
   if (unsigned PartialCheck = Test & fcNormal) {
     // isnormal(V) ==> (0 < exp < max_exp) ==> (unsigned(exp-1) < (max_exp-1))
     APInt ExpLSB = ExpMask & ~(ExpMask.shl(1));
diff --git a/llvm/lib/CodeGen/SlotIndexes.cpp b/llvm/lib/CodeGen/SlotIndexes.cpp
index ee3a0164564ed..47ee36971d0ea 100644
--- a/llvm/lib/CodeGen/SlotIndexes.cpp
+++ b/llvm/lib/CodeGen/SlotIndexes.cpp
@@ -215,7 +215,7 @@ void SlotIndexes::repairIndexesInRange(MachineBasicBlock *MBB,
         --MBBI;
       else
         pastStart = true;
-    } else if (MI && mi2iMap.find(MI) == mi2iMap.end()) {
+    } else if (MI && !mi2iMap.contains(MI)) {
       if (MBBI != Begin)
         --MBBI;
       else
@@ -232,7 +232,7 @@ void SlotIndexes::repairIndexesInRange(MachineBasicBlock *MBB,
   for (MachineBasicBlock::iterator I = End; I != Begin;) {
     --I;
     MachineInstr &MI = *I;
-    if (!MI.isDebugOrPseudoInstr() && mi2iMap.find(&MI) == mi2iMap.end())
+    if (!MI.isDebugOrPseudoInstr() && !mi2iMap.contains(&MI))
       insertMachineInstrInMaps(MI);
   }
 }
diff --git a/llvm/lib/CodeGen/StackProtector.cpp b/llvm/lib/CodeGen/StackProtector.cpp
index 52336cc1d609e..aa92dcb386560 100644
--- a/llvm/lib/CodeGen/StackProtector.cpp
+++ b/llvm/lib/CodeGen/StackProtector.cpp
@@ -96,7 +96,7 @@ bool StackProtector::runOnFunction(Function &Fn) {
 
   SSPBufferSize = Fn.getFnAttributeAsParsedInteger(
       "stack-protector-buffer-size", DefaultSSPBufferSize);
-  if (!RequiresStackProtector())
+  if (!requiresStackProtector(F, &Layout))
     return false;
 
   // TODO(etienneb): Functions with funclets are not correctly supported now.
@@ -121,9 +121,9 @@ bool StackProtector::runOnFunction(Function &Fn) {
 /// \param [out] IsLarge is set to true if a protectable array is found and
 /// it is "large" ( >= ssp-buffer-size).  In the case of a structure with
 /// multiple arrays, this gets set if any of them is large.
-bool StackProtector::ContainsProtectableArray(Type *Ty, bool &IsLarge,
-                                              bool Strong,
-                                              bool InStruct) const {
+static bool ContainsProtectableArray(Type *Ty, Module *M, unsigned SSPBufferSize,
+                                     bool &IsLarge, bool Strong,
+                                     bool InStruct) {
   if (!Ty)
     return false;
   if (ArrayType *AT = dyn_cast<ArrayType>(Ty)) {
@@ -132,7 +132,7 @@ bool StackProtector::ContainsProtectableArray(Type *Ty, bool &IsLarge,
       // add stack protectors unless the array is a character array.
       // However, in strong mode any array, regardless of type and size,
       // triggers a protector.
-      if (!Strong && (InStruct || !Trip.isOSDarwin()))
+      if (!Strong && (InStruct || !Triple(M->getTargetTriple()).isOSDarwin()))
         return false;
     }
 
@@ -154,7 +154,7 @@ bool StackProtector::ContainsProtectableArray(Type *Ty, bool &IsLarge,
 
   bool NeedsProtector = false;
   for (Type *ET : ST->elements())
-    if (ContainsProtectableArray(ET, IsLarge, Strong, true)) {
+    if (ContainsProtectableArray(ET, M, SSPBufferSize, IsLarge, Strong, true)) {
       // If the element is a protectable array and is large (>= SSPBufferSize)
       // then we are done.  If the protectable array is not large, then
       // keep looking in case a subsequent element is a large array.
@@ -166,8 +166,10 @@ bool StackProtector::ContainsProtectableArray(Type *Ty, bool &IsLarge,
   return NeedsProtector;
 }
 
-bool StackProtector::HasAddressTaken(const Instruction *AI,
-                                     TypeSize AllocSize) {
+/// Check whether a stack allocation has its address taken.
+static bool HasAddressTaken(const Instruction *AI, TypeSize AllocSize,
+                            Module *M,
+                            SmallPtrSet<const PHINode *, 16> &VisitedPHIs) {
   const DataLayout &DL = M->getDataLayout();
   for (const User *U : AI->users()) {
     const auto *I = cast<Instruction>(U);
@@ -221,14 +223,14 @@ bool StackProtector::HasAddressTaken(const Instruction *AI,
       // assume the scalable value is of minimum size.
       TypeSize NewAllocSize =
           TypeSize::Fixed(AllocSize.getKnownMinValue()) - OffsetSize;
-      if (HasAddressTaken(I, NewAllocSize))
+      if (HasAddressTaken(I, NewAllocSize, M, VisitedPHIs))
         return true;
       break;
     }
     case Instruction::BitCast:
     case Instruction::Select:
     case Instruction::AddrSpaceCast:
-      if (HasAddressTaken(I, AllocSize))
+      if (HasAddressTaken(I, AllocSize, M, VisitedPHIs))
         return true;
       break;
     case Instruction::PHI: {
@@ -236,7 +238,7 @@ bool StackProtector::HasAddressTaken(const Instruction *AI,
       // they are only visited once.
       const auto *PN = cast<PHINode>(I);
       if (VisitedPHIs.insert(PN).second)
-        if (HasAddressTaken(PN, AllocSize))
+        if (HasAddressTaken(PN, AllocSize, M, VisitedPHIs))
           return true;
       break;
     }
@@ -282,10 +284,19 @@ static const CallInst *findStackProtectorIntrinsic(Function &F) {
 /// functions with aggregates that contain any buffer regardless of type and
 /// size, and functions that contain stack-based variables that have had their
 /// address taken.
-bool StackProtector::RequiresStackProtector() {
+bool StackProtector::requiresStackProtector(Function *F, SSPLayoutMap *Layout) {
+  Module *M = F->getParent();
   bool Strong = false;
   bool NeedsProtector = false;
 
+  // The set of PHI nodes visited when determining if a variable's reference has
+  // been taken.  This set is maintained to ensure we don't visit the same PHI
+  // node multiple times.
+  SmallPtrSet<const PHINode *, 16> VisitedPHIs;
+
+  unsigned SSPBufferSize = F->getFnAttributeAsParsedInteger(
+      "stack-protector-buffer-size", DefaultSSPBufferSize);
+
   if (F->hasFnAttribute(Attribute::SafeStack))
     return false;
 
@@ -295,6 +306,8 @@ bool StackProtector::RequiresStackProtector() {
   OptimizationRemarkEmitter ORE(F);
 
   if (F->hasFnAttribute(Attribute::StackProtectReq)) {
+    if (!Layout)
+      return true;
     ORE.emit([&]() {
       return OptimizationRemark(DEBUG_TYPE, "StackProtectorRequested", F)
              << "Stack protection applied to function "
@@ -324,21 +337,27 @@ bool StackProtector::RequiresStackProtector() {
             if (CI->getLimitedValue(SSPBufferSize) >= SSPBufferSize) {
               // A call to alloca with size >= SSPBufferSize requires
               // stack protectors.
-              Layout.insert(std::make_pair(AI,
-                                           MachineFrameInfo::SSPLK_LargeArray));
+              if (!Layout)
+                return true;
+              Layout->insert(
+                  std::make_pair(AI, MachineFrameInfo::SSPLK_LargeArray));
               ORE.emit(RemarkBuilder);
               NeedsProtector = true;
             } else if (Strong) {
               // Require protectors for all alloca calls in strong mode.
-              Layout.insert(std::make_pair(AI,
-                                           MachineFrameInfo::SSPLK_SmallArray));
+              if (!Layout)
+                return true;
+              Layout->insert(
+                  std::make_pair(AI, MachineFrameInfo::SSPLK_SmallArray));
               ORE.emit(RemarkBuilder);
               NeedsProtector = true;
             }
           } else {
             // A call to alloca with a variable size requires protectors.
-            Layout.insert(std::make_pair(AI,
-                                         MachineFrameInfo::SSPLK_LargeArray));
+            if (!Layout)
+              return true;
+            Layout->insert(
+                std::make_pair(AI, MachineFrameInfo::SSPLK_LargeArray));
             ORE.emit(RemarkBuilder);
             NeedsProtector = true;
           }
@@ -346,10 +365,13 @@ bool StackProtector::RequiresStackProtector() {
         }
 
         bool IsLarge = false;
-        if (ContainsProtectableArray(AI->getAllocatedType(), IsLarge, Strong)) {
-          Layout.insert(std::make_pair(AI, IsLarge
-                                       ? MachineFrameInfo::SSPLK_LargeArray
-                                       : MachineFrameInfo::SSPLK_SmallArray));
+        if (ContainsProtectableArray(AI->getAllocatedType(), M, SSPBufferSize,
+                                     IsLarge, Strong, false)) {
+          if (!Layout)
+            return true;
+          Layout->insert(std::make_pair(
+              AI, IsLarge ? MachineFrameInfo::SSPLK_LargeArray
+                          : MachineFrameInfo::SSPLK_SmallArray));
           ORE.emit([&]() {
             return OptimizationRemark(DEBUG_TYPE, "StackProtectorBuffer", &I)
                    << "Stack protection applied to function "
@@ -361,10 +383,14 @@ bool StackProtector::RequiresStackProtector() {
           continue;
         }
 
-        if (Strong && HasAddressTaken(AI, M->getDataLayout().getTypeAllocSize(
-                                              AI->getAllocatedType()))) {
+        if (Strong &&
+            HasAddressTaken(
+                AI, M->getDataLayout().getTypeAllocSize(AI->getAllocatedType()),
+                M, VisitedPHIs)) {
           ++NumAddrTaken;
-          Layout.insert(std::make_pair(AI, MachineFrameInfo::SSPLK_AddrOf));
+          if (!Layout)
+            return true;
+          Layout->insert(std::make_pair(AI, MachineFrameInfo::SSPLK_AddrOf));
           ORE.emit([&]() {
             return OptimizationRemark(DEBUG_TYPE, "StackProtectorAddressTaken",
                                       &I)
diff --git a/llvm/lib/CodeGen/TargetInstrInfo.cpp b/llvm/lib/CodeGen/TargetInstrInfo.cpp
index e86724ad7c0f0..e50c9222b1ffe 100644
--- a/llvm/lib/CodeGen/TargetInstrInfo.cpp
+++ b/llvm/lib/CodeGen/TargetInstrInfo.cpp
@@ -1017,6 +1017,17 @@ void TargetInstrInfo::reassociateOps(
   InsInstrs.push_back(MIB2);
   DelInstrs.push_back(&Prev);
   DelInstrs.push_back(&Root);
+
+  // We transformed:
+  // B = A op X (Prev)
+  // C = B op Y (Root)
+  // Into:
+  // B = X op Y (MIB1)
+  // C = A op B (MIB2)
+  // C has the same value as before, B doesn't; as such, keep the debug number
+  // of C but not of B.
+  if (unsigned OldRootNum = Root.peekDebugInstrNum())
+    MIB2.getInstr()->setDebugInstrNum(OldRootNum);
 }
 
 void TargetInstrInfo::genAlternativeCodeSequence(
@@ -1038,15 +1049,13 @@ void TargetInstrInfo::genAlternativeCodeSequence(
     Prev = MRI.getUniqueVRegDef(Root.getOperand(2).getReg());
     break;
   default:
-    break;
+    llvm_unreachable("Unknown pattern for machine combiner");
   }
 
   // Don't reassociate if Prev and Root are in different blocks.
   if (Prev->getParent() != Root.getParent())
     return;
 
-  assert(Prev && "Unknown pattern for machine combiner");
-
   reassociateOps(Root, *Prev, Pattern, InsInstrs, DelInstrs, InstIdxForVirtReg);
 }
 
diff --git a/llvm/lib/CodeGen/TargetLoweringObjectFileImpl.cpp b/llvm/lib/CodeGen/TargetLoweringObjectFileImpl.cpp
index b78a4d2b2108b..e2fbe027f15b0 100644
--- a/llvm/lib/CodeGen/TargetLoweringObjectFileImpl.cpp
+++ b/llvm/lib/CodeGen/TargetLoweringObjectFileImpl.cpp
@@ -1202,11 +1202,7 @@ void TargetLoweringObjectFileMachO::Initialize(MCContext &Ctx,
 
 MCSection *TargetLoweringObjectFileMachO::getStaticDtorSection(
     unsigned Priority, const MCSymbol *KeySym) const {
-  // TODO(yln): Remove -lower-global-dtors-via-cxa-atexit fallback flag
-  // (LowerGlobalDtorsViaCxaAtExit) and always issue a fatal error here.
-  if (TM->Options.LowerGlobalDtorsViaCxaAtExit)
-    report_fatal_error("@llvm.global_dtors should have been lowered already");
-  return StaticDtorSection;
+  report_fatal_error("@llvm.global_dtors should have been lowered already");
 }
 
 void TargetLoweringObjectFileMachO::emitModuleMetadata(MCStreamer &Streamer,
@@ -1267,6 +1263,20 @@ MCSection *TargetLoweringObjectFileMachO::getExplicitSectionGlobal(
 
   StringRef SectionName = GO->getSection();
 
+  const GlobalVariable *GV = dyn_cast<GlobalVariable>(GO);
+  if (GV && GV->hasImplicitSection()) {
+    auto Attrs = GV->getAttributes();
+    if (Attrs.hasAttribute("bss-section") && Kind.isBSS()) {
+      SectionName = Attrs.getAttribute("bss-section").getValueAsString();
+    } else if (Attrs.hasAttribute("rodata-section") && Kind.isReadOnly()) {
+      SectionName = Attrs.getAttribute("rodata-section").getValueAsString();
+    } else if (Attrs.hasAttribute("relro-section") && Kind.isReadOnlyWithRel()) {
+      SectionName = Attrs.getAttribute("relro-section").getValueAsString();
+    } else if (Attrs.hasAttribute("data-section") && Kind.isData()) {
+      SectionName = Attrs.getAttribute("data-section").getValueAsString();
+    }
+  }
+
   const Function *F = dyn_cast<Function>(GO);
   if (F && F->hasFnAttribute("implicit-section-name")) {
     SectionName = F->getFnAttribute("implicit-section-name").getValueAsString();
diff --git a/llvm/lib/CodeGen/TargetPassConfig.cpp b/llvm/lib/CodeGen/TargetPassConfig.cpp
index c48b520d0cb25..32fa6cc17d53f 100644
--- a/llvm/lib/CodeGen/TargetPassConfig.cpp
+++ b/llvm/lib/CodeGen/TargetPassConfig.cpp
@@ -878,8 +878,7 @@ void TargetPassConfig::addIRPasses() {
 
   // For MachO, lower @llvm.global_dtors into @llvm.global_ctors with
   // __cxa_atexit() calls to avoid emitting the deprecated __mod_term_func.
-  if (TM->getTargetTriple().isOSBinFormatMachO() &&
-      TM->Options.LowerGlobalDtorsViaCxaAtExit)
+  if (TM->getTargetTriple().isOSBinFormatMachO())
     addPass(createLowerGlobalDtorsLegacyPass());
 
   // Make sure that no unreachable blocks are instruction selected.
diff --git a/llvm/lib/DWARFLinker/DWARFLinker.cpp b/llvm/lib/DWARFLinker/DWARFLinker.cpp
index 7640dfa40a682..789a7789c807a 100644
--- a/llvm/lib/DWARFLinker/DWARFLinker.cpp
+++ b/llvm/lib/DWARFLinker/DWARFLinker.cpp
@@ -928,7 +928,7 @@ void DWARFLinker::assignAbbrev(DIEAbbrev &Abbrev) {
     Abbreviations.push_back(
         std::make_unique<DIEAbbrev>(Abbrev.getTag(), Abbrev.hasChildren()));
     for (const auto &Attr : Abbrev.getData())
-      Abbreviations.back()->AddAttribute(Attr.getAttribute(), Attr.getForm());
+      Abbreviations.back()->AddAttribute(Attr);
     AbbreviationsSet.InsertNode(Abbreviations.back().get(), InsertToken);
     // Assign the unique abbreviation number.
     Abbrev.setNumber(Abbreviations.size());
@@ -1201,15 +1201,24 @@ unsigned DWARFLinker::DIECloner::cloneAddressAttribute(
     *Addr += Info.PCOffset;
   }
 
-  dwarf::Form Form = AttrSpec.Form;
-
-  // DWARFLinker does not use addrx forms since it generates relocated
-  // addresses. Replace DW_FORM_addrx with DW_FORM_addr here.
-  if (Form == dwarf::DW_FORM_addrx)
-    Form = dwarf::DW_FORM_addr;
+  switch (AttrSpec.Form) {
+  case dwarf::DW_FORM_addrx:
+  case dwarf::DW_FORM_addrx1:
+  case dwarf::DW_FORM_addrx2:
+  case dwarf::DW_FORM_addrx3:
+  case dwarf::DW_FORM_addrx4: {
+    // DWARFLinker does not use addrx forms since it generates relocated
+    // addresses. Replace DW_FORM_addrx* with DW_FORM_addr here.
+    AttrSpec.Form = dwarf::DW_FORM_addr;
+    break;
+  }
+  default:
+    // Nothing to do.
+    break;
+  }
 
   Die.addValue(DIEAlloc, static_cast<dwarf::Attribute>(AttrSpec.Attr),
-               static_cast<dwarf::Form>(Form), DIEInteger(*Addr));
+               AttrSpec.Form, DIEInteger(*Addr));
   return Unit.getOrigUnit().getAddressByteSize();
 }
 
@@ -1252,8 +1261,13 @@ unsigned DWARFLinker::DIECloner::cloneScalarAttribute(
     }
     if (AttrSpec.Attr == dwarf::DW_AT_declaration && Value)
       Info.IsDeclaration = true;
-    Die.addValue(DIEAlloc, dwarf::Attribute(AttrSpec.Attr),
-                 dwarf::Form(AttrSpec.Form), DIEInteger(Value));
+
+    if (AttrSpec.Form == dwarf::DW_FORM_loclistx)
+      Die.addValue(DIEAlloc, dwarf::Attribute(AttrSpec.Attr),
+                   dwarf::Form(AttrSpec.Form), DIELocList(Value));
+    else
+      Die.addValue(DIEAlloc, dwarf::Attribute(AttrSpec.Attr),
+                   dwarf::Form(AttrSpec.Form), DIEInteger(Value));
     return AttrSize;
   }
 
@@ -1276,6 +1290,27 @@ unsigned DWARFLinker::DIECloner::cloneScalarAttribute(
       return 0;
     }
 
+    Value = *Offset;
+    AttrSpec.Form = dwarf::DW_FORM_sec_offset;
+    AttrSize = Unit.getOrigUnit().getFormParams().getDwarfOffsetByteSize();
+  } else if (AttrSpec.Form == dwarf::DW_FORM_loclistx) {
+    // DWARFLinker does not generate .debug_addr table. Thus we need to change
+    // all "addrx" related forms to "addr" version. Change DW_FORM_loclistx
+    // to DW_FORM_sec_offset here.
+    std::optional<uint64_t> Index = Val.getAsSectionOffset();
+    if (!Index) {
+      Linker.reportWarning("Cannot read the attribute. Dropping.", File,
+                           &InputDIE);
+      return 0;
+    }
+    std::optional<uint64_t> Offset =
+        Unit.getOrigUnit().getLoclistOffset(*Index);
+    if (!Offset) {
+      Linker.reportWarning("Cannot read the attribute. Dropping.", File,
+                           &InputDIE);
+      return 0;
+    }
+
     Value = *Offset;
     AttrSpec.Form = dwarf::DW_FORM_sec_offset;
     AttrSize = Unit.getOrigUnit().getFormParams().getDwarfOffsetByteSize();
@@ -1298,6 +1333,7 @@ unsigned DWARFLinker::DIECloner::cloneScalarAttribute(
         &InputDIE);
     return 0;
   }
+
   PatchLocation Patch =
       Die.addValue(DIEAlloc, dwarf::Attribute(AttrSpec.Attr),
                    dwarf::Form(AttrSpec.Form), DIEInteger(Value));
@@ -1305,13 +1341,10 @@ unsigned DWARFLinker::DIECloner::cloneScalarAttribute(
       AttrSpec.Attr == dwarf::DW_AT_start_scope) {
     Unit.noteRangeAttribute(Die, Patch);
     Info.HasRanges = true;
-  }
-  // A more generic way to check for location attributes would be
-  // nice, but it's very unlikely that any other attribute needs a
-  // location list.
-  // FIXME: use DWARFAttribute::mayHaveLocationDescription().
-  else if (AttrSpec.Attr == dwarf::DW_AT_location ||
-           AttrSpec.Attr == dwarf::DW_AT_frame_base) {
+  } else if (DWARFAttribute::mayHaveLocationList(AttrSpec.Attr) &&
+             dwarf::doesFormBelongToClass(AttrSpec.Form,
+                                          DWARFFormValue::FC_SectionOffset,
+                                          Unit.getOrigUnit().getVersion())) {
     Unit.noteLocationAttribute(Patch, Info.PCOffset);
   } else if (AttrSpec.Attr == dwarf::DW_AT_declaration && Value)
     Info.IsDeclaration = true;
@@ -1358,6 +1391,10 @@ unsigned DWARFLinker::DIECloner::cloneAttribute(
                                IsLittleEndian);
   case dwarf::DW_FORM_addr:
   case dwarf::DW_FORM_addrx:
+  case dwarf::DW_FORM_addrx1:
+  case dwarf::DW_FORM_addrx2:
+  case dwarf::DW_FORM_addrx3:
+  case dwarf::DW_FORM_addrx4:
     return cloneAddressAttribute(Die, InputDIE, AttrSpec, AttrSize, Val, Unit,
                                  Info);
   case dwarf::DW_FORM_data1:
@@ -1370,6 +1407,8 @@ unsigned DWARFLinker::DIECloner::cloneAttribute(
   case dwarf::DW_FORM_flag:
   case dwarf::DW_FORM_flag_present:
   case dwarf::DW_FORM_rnglistx:
+  case dwarf::DW_FORM_loclistx:
+  case dwarf::DW_FORM_implicit_const:
     return cloneScalarAttribute(Die, InputDIE, File, Unit, AttrSpec, Val,
                                 AttrSize, Info);
   default:
@@ -1451,6 +1490,12 @@ static bool shouldSkipAttribute(
   case dwarf::DW_AT_str_offsets_base:
     // FIXME: Use the string offset table with Dwarf 5.
     return true;
+  case dwarf::DW_AT_loclists_base:
+    // In case !Update the .debug_addr table is not generated/preserved.
+    // Thus instead of DW_FORM_loclistx the DW_FORM_sec_offset is used.
+    // Since DW_AT_loclists_base is used for only DW_FORM_loclistx the
+    // DW_AT_loclists_base is removed.
+    return !Update;
   case dwarf::DW_AT_location:
   case dwarf::DW_AT_frame_base:
     // FIXME: for some reason dsymutil-classic keeps the location attributes
@@ -1552,7 +1597,7 @@ DIE *DWARFLinker::DIECloner::cloneDIE(const DWARFDie &InputDIE,
       continue;
     }
 
-    DWARFFormValue Val(AttrSpec.Form);
+    DWARFFormValue Val = AttrSpec.getFormValue();
     uint64_t AttrSize = Offset;
     Val.extractValue(Data, &Offset, U.getFormParams(), &U);
     AttrSize = Offset - AttrSize;
@@ -1719,6 +1764,61 @@ void DWARFLinker::generateUnitRanges(CompileUnit &Unit,
   }
 }
 
+void DWARFLinker::generateUnitLocations(
+    CompileUnit &Unit, const DWARFFile &File,
+    ExpressionHandlerRef ExprHandler) const {
+  if (LLVM_UNLIKELY(Options.Update))
+    return;
+
+  const LocListAttributesTy &AllLocListAttributes =
+      Unit.getLocationAttributes();
+
+  if (AllLocListAttributes.empty())
+    return;
+
+  // Emit locations list table header.
+  MCSymbol *EndLabel = TheDwarfEmitter->emitDwarfDebugLocListHeader(Unit);
+
+  for (auto &CurLocAttr : AllLocListAttributes) {
+    // Get location expressions vector corresponding to the current attribute
+    // from the source DWARF.
+    Expected<DWARFLocationExpressionsVector> OriginalLocations =
+        Unit.getOrigUnit().findLoclistFromOffset((CurLocAttr.first).get());
+
+    if (!OriginalLocations) {
+      llvm::consumeError(OriginalLocations.takeError());
+      reportWarning("Invalid location attribute ignored.", File);
+      continue;
+    }
+
+    DWARFLocationExpressionsVector LinkedLocationExpressions;
+    for (DWARFLocationExpression &CurExpression : *OriginalLocations) {
+
+      DWARFLocationExpression LinkedExpression;
+
+      if (CurExpression.Range) {
+        // Relocate address range.
+        LinkedExpression.Range = {
+            CurExpression.Range->LowPC + CurLocAttr.second,
+            CurExpression.Range->HighPC + CurLocAttr.second};
+      }
+
+      // Clone expression.
+      LinkedExpression.Expr.reserve(CurExpression.Expr.size());
+      ExprHandler(CurExpression.Expr, LinkedExpression.Expr);
+
+      LinkedLocationExpressions.push_back(LinkedExpression);
+    }
+
+    // Emit locations list table fragment corresponding to the CurLocAttr.
+    TheDwarfEmitter->emitDwarfDebugLocListFragment(
+        Unit, LinkedLocationExpressions, CurLocAttr.first);
+  }
+
+  // Emit locations list table footer.
+  TheDwarfEmitter->emitDwarfDebugLocListFooter(Unit, EndLabel);
+}
+
 /// Insert the new line info sequence \p Seq into the current
 /// set of already linked line info \p Rows.
 static void insertLineSequence(std::vector<DWARFDebugLine::Row> &Seq,
@@ -2301,17 +2401,17 @@ uint64_t DWARFLinker::DIECloner::cloneAllCompileUnits(
 
       Linker.generateUnitRanges(*CurrentUnit, File);
 
-      auto ProcessExpr = [&](StringRef Bytes,
-                             SmallVectorImpl<uint8_t> &Buffer) {
+      auto ProcessExpr = [&](SmallVectorImpl<uint8_t> &SrcBytes,
+                             SmallVectorImpl<uint8_t> &OutBytes) {
         DWARFUnit &OrigUnit = CurrentUnit->getOrigUnit();
-        DataExtractor Data(Bytes, IsLittleEndian,
+        DataExtractor Data(SrcBytes, IsLittleEndian,
                            OrigUnit.getAddressByteSize());
         cloneExpression(Data,
                         DWARFExpression(Data, OrigUnit.getAddressByteSize(),
                                         OrigUnit.getFormParams().Format),
-                        File, *CurrentUnit, Buffer);
+                        File, *CurrentUnit, OutBytes);
       };
-      Emitter->emitLocationsForUnit(*CurrentUnit, DwarfContext, ProcessExpr);
+      Linker.generateUnitLocations(*CurrentUnit, File, ProcessExpr);
     }
   }
 
@@ -2420,6 +2520,8 @@ void DWARFLinker::copyInvariantDebugSection(DWARFContext &Dwarf) {
       Dwarf.getDWARFObj().getAddrSection().Data, "debug_addr");
   TheDwarfEmitter->emitSectionContents(
       Dwarf.getDWARFObj().getRnglistsSection().Data, "debug_rnglists");
+  TheDwarfEmitter->emitSectionContents(
+      Dwarf.getDWARFObj().getLoclistsSection().Data, "debug_loclists");
 }
 
 void DWARFLinker::addObjectFile(DWARFFile &File, objFileLoader Loader,
@@ -2505,20 +2607,6 @@ Error DWARFLinker::link() {
       continue;
     }
 
-    // Check for unsupported sections. Following sections can be referenced
-    // from .debug_info section. Current DWARFLinker implementation does not
-    // support or update references to these tables. Thus we report warning
-    // and skip corresponding object file.
-    if (!OptContext.File.Dwarf->getDWARFObj()
-             .getLoclistsSection()
-             .Data.empty()) {
-      reportWarning("'.debug_loclists' is not currently supported: file "
-                    "will be skipped",
-                    OptContext.File);
-      OptContext.Skip = true;
-      continue;
-    }
-
     // In a first phase, just read in the debug info and load all clang modules.
     OptContext.CompileUnits.reserve(
         OptContext.File.Dwarf->getNumCompileUnits());
diff --git a/llvm/lib/DWARFLinker/DWARFStreamer.cpp b/llvm/lib/DWARFLinker/DWARFStreamer.cpp
index 3af7cde52509d..1593ea65065d5 100644
--- a/llvm/lib/DWARFLinker/DWARFStreamer.cpp
+++ b/llvm/lib/DWARFLinker/DWARFStreamer.cpp
@@ -110,6 +110,7 @@ bool DwarfStreamer::init(Triple TheTriple,
   RangesSectionSize = 0;
   RngListsSectionSize = 0;
   LocSectionSize = 0;
+  LocListsSectionSize = 0;
   LineSectionSize = 0;
   FrameSectionSize = 0;
   DebugInfoSectionSize = 0;
@@ -206,6 +207,8 @@ void DwarfStreamer::emitSectionContents(StringRef SecData, StringRef SecName) {
           .Case("debug_addr", MC->getObjectFileInfo()->getDwarfAddrSection())
           .Case("debug_rnglists",
                 MC->getObjectFileInfo()->getDwarfRnglistsSection())
+          .Case("debug_loclists",
+                MC->getObjectFileInfo()->getDwarfLoclistsSection())
           .Default(nullptr);
 
   if (Section) {
@@ -483,73 +486,149 @@ void DwarfStreamer::emitDwarfDebugRngListsTableFragment(
   RngListsSectionSize += 1;
 }
 
-/// Emit location lists for \p Unit and update attributes to point to the new
-/// entries.
-void DwarfStreamer::emitLocationsForUnit(
-    const CompileUnit &Unit, DWARFContext &Dwarf,
-    std::function<void(StringRef, SmallVectorImpl<uint8_t> &)> ProcessExpr) {
-  const auto &Attributes = Unit.getLocationAttributes();
+/// Emit debug locations(.debug_loc, .debug_loclists) header.
+MCSymbol *DwarfStreamer::emitDwarfDebugLocListHeader(const CompileUnit &Unit) {
+  if (Unit.getOrigUnit().getVersion() < 5)
+    return nullptr;
+
+  // Make .debug_loclists the current section.
+  MS->switchSection(MC->getObjectFileInfo()->getDwarfLoclistsSection());
+
+  MCSymbol *BeginLabel = Asm->createTempSymbol("Bloclists");
+  MCSymbol *EndLabel = Asm->createTempSymbol("Eloclists");
+  unsigned AddressSize = Unit.getOrigUnit().getAddressByteSize();
+
+  // Length
+  Asm->emitLabelDifference(EndLabel, BeginLabel, sizeof(uint32_t));
+  Asm->OutStreamer->emitLabel(BeginLabel);
+  LocListsSectionSize += sizeof(uint32_t);
+
+  // Version.
+  MS->emitInt16(5);
+  LocListsSectionSize += sizeof(uint16_t);
+
+  // Address size.
+  MS->emitInt8(AddressSize);
+  LocListsSectionSize++;
+
+  // Seg_size
+  MS->emitInt8(0);
+  LocListsSectionSize++;
 
-  if (Attributes.empty())
+  // Offset entry count
+  MS->emitInt32(0);
+  LocListsSectionSize += sizeof(uint32_t);
+
+  return EndLabel;
+}
+
+/// Emit debug locations(.debug_loc, .debug_loclists) fragment.
+void DwarfStreamer::emitDwarfDebugLocListFragment(
+    const CompileUnit &Unit,
+    const DWARFLocationExpressionsVector &LinkedLocationExpression,
+    PatchLocation Patch) {
+  if (Unit.getOrigUnit().getVersion() < 5) {
+    emitDwarfDebugLocTableFragment(Unit, LinkedLocationExpression, Patch);
     return;
+  }
 
-  MS->switchSection(MC->getObjectFileInfo()->getDwarfLocSection());
+  emitDwarfDebugLocListsTableFragment(Unit, LinkedLocationExpression, Patch);
+}
+
+/// Emit debug locations(.debug_loc, .debug_loclists) footer.
+void DwarfStreamer::emitDwarfDebugLocListFooter(const CompileUnit &Unit,
+                                                MCSymbol *EndLabel) {
+  if (Unit.getOrigUnit().getVersion() < 5)
+    return;
+
+  // Make .debug_loclists the current section.
+  MS->switchSection(MC->getObjectFileInfo()->getDwarfLoclistsSection());
+
+  if (EndLabel != nullptr)
+    Asm->OutStreamer->emitLabel(EndLabel);
+}
 
+/// Emit piece of .debug_loc for \p LinkedLocationExpression.
+void DwarfStreamer::emitDwarfDebugLocTableFragment(
+    const CompileUnit &Unit,
+    const DWARFLocationExpressionsVector &LinkedLocationExpression,
+    PatchLocation Patch) {
+  Patch.set(LocSectionSize);
+
+  // Make .debug_loc to be current section.
+  MS->switchSection(MC->getObjectFileInfo()->getDwarfLocSection());
   unsigned AddressSize = Unit.getOrigUnit().getAddressByteSize();
-  uint64_t BaseAddressMarker = (AddressSize == 8)
-                                   ? std::numeric_limits<uint64_t>::max()
-                                   : std::numeric_limits<uint32_t>::max();
-  const DWARFSection &InputSec = Dwarf.getDWARFObj().getLocSection();
-  DataExtractor Data(InputSec.Data, Dwarf.isLittleEndian(), AddressSize);
-  DWARFUnit &OrigUnit = Unit.getOrigUnit();
-  auto OrigUnitDie = OrigUnit.getUnitDIE(false);
-  int64_t UnitPcOffset = 0;
-  if (auto OrigLowPc =
-          dwarf::toAddress(OrigUnitDie.find(dwarf::DW_AT_low_pc))) {
-    assert(Unit.getLowPc());
-    UnitPcOffset = int64_t(*OrigLowPc) - *Unit.getLowPc();
+
+  // Emit ranges.
+  uint64_t BaseAddress = 0;
+  if (std::optional<uint64_t> LowPC = Unit.getLowPc())
+    BaseAddress = *LowPC;
+
+  for (const DWARFLocationExpression &LocExpression :
+       LinkedLocationExpression) {
+    if (LocExpression.Range) {
+      MS->emitIntValue(LocExpression.Range->LowPC - BaseAddress, AddressSize);
+      MS->emitIntValue(LocExpression.Range->HighPC - BaseAddress, AddressSize);
+
+      LocSectionSize += AddressSize;
+      LocSectionSize += AddressSize;
+    }
+
+    Asm->OutStreamer->emitIntValue(LocExpression.Expr.size(), 2);
+    Asm->OutStreamer->emitBytes(StringRef(
+        (const char *)LocExpression.Expr.data(), LocExpression.Expr.size()));
+    LocSectionSize += LocExpression.Expr.size() + 2;
   }
 
-  SmallVector<uint8_t, 32> Buffer;
-  for (const auto &Attr : Attributes) {
-    uint64_t Offset = Attr.first.get();
-    Attr.first.set(LocSectionSize);
-    // This is the quantity to add to the old location address to get
-    // the correct address for the new one.
-    int64_t LocPcOffset = Attr.second + UnitPcOffset;
-    while (Data.isValidOffset(Offset)) {
-      uint64_t Low = Data.getUnsigned(&Offset, AddressSize);
-      uint64_t High = Data.getUnsigned(&Offset, AddressSize);
-      LocSectionSize += 2 * AddressSize;
-      // End of list entry.
-      if (Low == 0 && High == 0) {
-        Asm->OutStreamer->emitIntValue(0, AddressSize);
-        Asm->OutStreamer->emitIntValue(0, AddressSize);
-        break;
-      }
-      // Base address selection entry.
-      if (Low == BaseAddressMarker) {
-        Asm->OutStreamer->emitIntValue(BaseAddressMarker, AddressSize);
-        Asm->OutStreamer->emitIntValue(High + Attr.second, AddressSize);
-        LocPcOffset = 0;
-        continue;
-      }
-      // Location list entry.
-      Asm->OutStreamer->emitIntValue(Low + LocPcOffset, AddressSize);
-      Asm->OutStreamer->emitIntValue(High + LocPcOffset, AddressSize);
-      uint64_t Length = Data.getU16(&Offset);
-      Asm->OutStreamer->emitIntValue(Length, 2);
-      // Copy the bytes into to the buffer, process them, emit them.
-      Buffer.reserve(Length);
-      Buffer.resize(0);
-      StringRef Input = InputSec.Data.substr(Offset, Length);
-      ProcessExpr(Input, Buffer);
-      Asm->OutStreamer->emitBytes(
-          StringRef((const char *)Buffer.data(), Length));
-      Offset += Length;
-      LocSectionSize += Length + 2;
+  // Add the terminator entry.
+  MS->emitIntValue(0, AddressSize);
+  MS->emitIntValue(0, AddressSize);
+
+  LocSectionSize += AddressSize;
+  LocSectionSize += AddressSize;
+}
+
+/// Emit piece of .debug_loclists for \p LinkedLocationExpression.
+void DwarfStreamer::emitDwarfDebugLocListsTableFragment(
+    const CompileUnit &Unit,
+    const DWARFLocationExpressionsVector &LinkedLocationExpression,
+    PatchLocation Patch) {
+  Patch.set(LocListsSectionSize);
+
+  // Make .debug_loclists the current section.
+  MS->switchSection(MC->getObjectFileInfo()->getDwarfLoclistsSection());
+
+  unsigned AddressSize = Unit.getOrigUnit().getAddressByteSize();
+
+  for (const DWARFLocationExpression &LocExpression :
+       LinkedLocationExpression) {
+    if (LocExpression.Range) {
+      // Emit type of entry.
+      MS->emitInt8(dwarf::DW_LLE_start_length);
+      LocListsSectionSize += 1;
+
+      // Emit start address.
+      MS->emitIntValue(LocExpression.Range->LowPC, AddressSize);
+      LocListsSectionSize += AddressSize;
+
+      // Emit length of the range.
+      LocListsSectionSize += MS->emitSLEB128IntValue(
+          LocExpression.Range->HighPC - LocExpression.Range->LowPC);
+    } else {
+      // Emit type of entry.
+      MS->emitInt8(dwarf::DW_LLE_default_location);
+      LocListsSectionSize += 1;
     }
+
+    LocListsSectionSize += MS->emitULEB128IntValue(LocExpression.Expr.size());
+    Asm->OutStreamer->emitBytes(StringRef(
+        (const char *)LocExpression.Expr.data(), LocExpression.Expr.size()));
+    LocListsSectionSize += LocExpression.Expr.size();
   }
+
+  // Emit the terminator entry.
+  MS->emitInt8(dwarf::DW_LLE_end_of_list);
+  LocListsSectionSize += 1;
 }
 
 void DwarfStreamer::emitLineTableForUnit(MCDwarfLineTableParams Params,
diff --git a/llvm/lib/DebugInfo/CodeView/EnumTables.cpp b/llvm/lib/DebugInfo/CodeView/EnumTables.cpp
index 78a258600696e..b2f0099bd01c5 100644
--- a/llvm/lib/DebugInfo/CodeView/EnumTables.cpp
+++ b/llvm/lib/DebugInfo/CodeView/EnumTables.cpp
@@ -105,6 +105,7 @@ static const EnumEntry<codeview::SourceLanguage> SourceLanguages[] = {
     CV_ENUM_ENT(SourceLanguage, JScript), CV_ENUM_ENT(SourceLanguage, MSIL),
     CV_ENUM_ENT(SourceLanguage, HLSL),    CV_ENUM_ENT(SourceLanguage, D),
     CV_ENUM_ENT(SourceLanguage, Swift),   CV_ENUM_ENT(SourceLanguage, Rust),
+    CV_ENUM_ENT(SourceLanguage, ObjC),    CV_ENUM_ENT(SourceLanguage, ObjCpp),
 };
 
 static const EnumEntry<uint32_t> CompileSym2FlagNames[] = {
diff --git a/llvm/lib/DebugInfo/DWARF/DWARFContext.cpp b/llvm/lib/DebugInfo/DWARF/DWARFContext.cpp
index dd86144d16e08..f648ef8ff7707 100644
--- a/llvm/lib/DebugInfo/DWARF/DWARFContext.cpp
+++ b/llvm/lib/DebugInfo/DWARF/DWARFContext.cpp
@@ -1118,14 +1118,17 @@ DWARFCompileUnit *DWARFContext::getCompileUnitForOffset(uint64_t Offset) {
       NormalUnits.getUnitForOffset(Offset));
 }
 
-DWARFCompileUnit *DWARFContext::getCompileUnitForAddress(uint64_t Address) {
-  // First, get the offset of the compile unit.
+DWARFCompileUnit *DWARFContext::getCompileUnitForCodeAddress(uint64_t Address) {
+  uint64_t CUOffset = getDebugAranges()->findAddress(Address);
+  return getCompileUnitForOffset(CUOffset);
+}
+
+DWARFCompileUnit *DWARFContext::getCompileUnitForDataAddress(uint64_t Address) {
   uint64_t CUOffset = getDebugAranges()->findAddress(Address);
-  // Retrieve the compile unit.
   if (DWARFCompileUnit *OffsetCU = getCompileUnitForOffset(CUOffset))
     return OffsetCU;
 
-  // Global variables are often not found by the above search, for one of two
+  // Global variables are often missed by the above search, for one of two
   // reasons:
   //   1. .debug_aranges may not include global variables. On clang, it seems we
   //      put the globals in the aranges, but this isn't true for gcc.
@@ -1146,7 +1149,7 @@ DWARFCompileUnit *DWARFContext::getCompileUnitForAddress(uint64_t Address) {
 DWARFContext::DIEsForAddress DWARFContext::getDIEsForAddress(uint64_t Address) {
   DIEsForAddress Result;
 
-  DWARFCompileUnit *CU = getCompileUnitForAddress(Address);
+  DWARFCompileUnit *CU = getCompileUnitForCodeAddress(Address);
   if (!CU)
     return Result;
 
@@ -1297,7 +1300,7 @@ void DWARFContext::addLocalsForDie(DWARFCompileUnit *CU, DWARFDie Subprogram,
 std::vector<DILocal>
 DWARFContext::getLocalsForAddress(object::SectionedAddress Address) {
   std::vector<DILocal> Result;
-  DWARFCompileUnit *CU = getCompileUnitForAddress(Address.Address);
+  DWARFCompileUnit *CU = getCompileUnitForCodeAddress(Address.Address);
   if (!CU)
     return Result;
 
@@ -1310,7 +1313,7 @@ DWARFContext::getLocalsForAddress(object::SectionedAddress Address) {
 DILineInfo DWARFContext::getLineInfoForAddress(object::SectionedAddress Address,
                                                DILineInfoSpecifier Spec) {
   DILineInfo Result;
-  DWARFCompileUnit *CU = getCompileUnitForAddress(Address.Address);
+  DWARFCompileUnit *CU = getCompileUnitForCodeAddress(Address.Address);
   if (!CU)
     return Result;
 
@@ -1331,7 +1334,7 @@ DILineInfo DWARFContext::getLineInfoForAddress(object::SectionedAddress Address,
 DILineInfo
 DWARFContext::getLineInfoForDataAddress(object::SectionedAddress Address) {
   DILineInfo Result;
-  DWARFCompileUnit *CU = getCompileUnitForAddress(Address.Address);
+  DWARFCompileUnit *CU = getCompileUnitForDataAddress(Address.Address);
   if (!CU)
     return Result;
 
@@ -1346,7 +1349,7 @@ DWARFContext::getLineInfoForDataAddress(object::SectionedAddress Address) {
 DILineInfoTable DWARFContext::getLineInfoForAddressRange(
     object::SectionedAddress Address, uint64_t Size, DILineInfoSpecifier Spec) {
   DILineInfoTable Lines;
-  DWARFCompileUnit *CU = getCompileUnitForAddress(Address.Address);
+  DWARFCompileUnit *CU = getCompileUnitForCodeAddress(Address.Address);
   if (!CU)
     return Lines;
 
@@ -1402,7 +1405,7 @@ DWARFContext::getInliningInfoForAddress(object::SectionedAddress Address,
                                         DILineInfoSpecifier Spec) {
   DIInliningInfo InliningInfo;
 
-  DWARFCompileUnit *CU = getCompileUnitForAddress(Address.Address);
+  DWARFCompileUnit *CU = getCompileUnitForCodeAddress(Address.Address);
   if (!CU)
     return InliningInfo;
 
diff --git a/llvm/lib/DebugInfo/DWARF/DWARFFormValue.cpp b/llvm/lib/DebugInfo/DWARF/DWARFFormValue.cpp
index 804b877291cea..0fc65a8809823 100644
--- a/llvm/lib/DebugInfo/DWARF/DWARFFormValue.cpp
+++ b/llvm/lib/DebugInfo/DWARF/DWARFFormValue.cpp
@@ -214,35 +214,7 @@ bool DWARFFormValue::skipValue(dwarf::Form Form, DataExtractor DebugInfoData,
 }
 
 bool DWARFFormValue::isFormClass(DWARFFormValue::FormClass FC) const {
-  // First, check DWARF5 form classes.
-  if (Form < ArrayRef(DWARF5FormClasses).size() &&
-      DWARF5FormClasses[Form] == FC)
-    return true;
-  // Check more forms from extensions and proposals.
-  switch (Form) {
-  case DW_FORM_GNU_ref_alt:
-    return (FC == FC_Reference);
-  case DW_FORM_GNU_addr_index:
-    return (FC == FC_Address);
-  case DW_FORM_GNU_str_index:
-  case DW_FORM_GNU_strp_alt:
-    return (FC == FC_String);
-  case DW_FORM_LLVM_addrx_offset:
-    return (FC == FC_Address);
-  default:
-    break;
-  }
-
-  if (FC == FC_SectionOffset) {
-    if (Form == DW_FORM_strp || Form == DW_FORM_line_strp)
-      return true;
-    // In DWARF3 DW_FORM_data4 and DW_FORM_data8 served also as a section
-    // offset. If we don't have a DWARFUnit, default to the old behavior.
-    if (Form == DW_FORM_data4 || Form == DW_FORM_data8)
-      return !U || U->getVersion() <= 3;
-  }
-
-  return false;
+  return doesFormBelongToClass(Form, FC, U ? U->getVersion() : 3);
 }
 
 bool DWARFFormValue::extractValue(const DWARFDataExtractor &Data,
@@ -778,3 +750,33 @@ DWARFFormValue::getAsFile(DILineInfoSpecifier::FileLineInfoKind Kind) const {
   }
   return std::nullopt;
 }
+
+bool llvm::dwarf::doesFormBelongToClass(dwarf::Form Form, DWARFFormValue::FormClass FC,
+                           uint16_t DwarfVersion) {
+  // First, check DWARF5 form classes.
+  if (Form < ArrayRef(DWARF5FormClasses).size() &&
+      DWARF5FormClasses[Form] == FC)
+    return true;
+  // Check more forms from extensions and proposals.
+  switch (Form) {
+  case DW_FORM_GNU_ref_alt:
+    return (FC == DWARFFormValue::FC_Reference);
+  case DW_FORM_GNU_addr_index:
+    return (FC == DWARFFormValue::FC_Address);
+  case DW_FORM_GNU_str_index:
+  case DW_FORM_GNU_strp_alt:
+    return (FC == DWARFFormValue::FC_String);
+  case DW_FORM_LLVM_addrx_offset:
+    return (FC == DWARFFormValue::FC_Address);
+  case DW_FORM_strp:
+  case DW_FORM_line_strp:
+    return (FC == DWARFFormValue::FC_SectionOffset);
+  case DW_FORM_data4:
+  case DW_FORM_data8:
+    // In DWARF3 DW_FORM_data4 and DW_FORM_data8 served also as a section
+    // offset.
+    return (FC == DWARFFormValue::FC_SectionOffset) && (DwarfVersion <= 3);
+  default:
+    return false;
+  }
+}
diff --git a/llvm/lib/DebugInfo/DWARF/DWARFUnit.cpp b/llvm/lib/DebugInfo/DWARF/DWARFUnit.cpp
index c199e0118a6f9..2fc26fd86ba3d 100644
--- a/llvm/lib/DebugInfo/DWARF/DWARFUnit.cpp
+++ b/llvm/lib/DebugInfo/DWARF/DWARFUnit.cpp
@@ -176,7 +176,7 @@ DWARFUnitVector::getUnitForIndexEntry(const DWARFUnitIndex::Entry &E) {
 
   auto U = Parser(Offset, DW_SECT_INFO, nullptr, &E);
   if (!U)
-    U = nullptr;
+    return nullptr;
 
   auto *NewCU = U.get();
   this->insert(CU, std::move(U));
diff --git a/llvm/lib/DebugInfo/GSYM/FunctionInfo.cpp b/llvm/lib/DebugInfo/GSYM/FunctionInfo.cpp
index 51058fc09cf19..145a43d3b381b 100644
--- a/llvm/lib/DebugInfo/GSYM/FunctionInfo.cpp
+++ b/llvm/lib/DebugInfo/GSYM/FunctionInfo.cpp
@@ -96,57 +96,83 @@ llvm::Expected<FunctionInfo> FunctionInfo::decode(DataExtractor &Data,
   return std::move(FI);
 }
 
-llvm::Expected<uint64_t> FunctionInfo::encode(FileWriter &O) const {
+uint64_t FunctionInfo::cacheEncoding() {
+  EncodingCache.clear();
+  if (!isValid())
+    return 0;
+  raw_svector_ostream OutStrm(EncodingCache);
+  FileWriter FW(OutStrm, support::endian::system_endianness());
+  llvm::Expected<uint64_t> Result = encode(FW);
+  if (!Result) {
+    EncodingCache.clear();
+    consumeError(Result.takeError());
+    return 0;
+  }
+  return EncodingCache.size();
+}
+
+llvm::Expected<uint64_t> FunctionInfo::encode(FileWriter &Out) const {
   if (!isValid())
     return createStringError(std::errc::invalid_argument,
         "attempted to encode invalid FunctionInfo object");
   // Align FunctionInfo data to a 4 byte alignment.
-  O.alignTo(4);
-  const uint64_t FuncInfoOffset = O.tell();
+  Out.alignTo(4);
+  const uint64_t FuncInfoOffset = Out.tell();
+  // Check if we have already encoded this function info into EncodingCache.
+  // This will be non empty when creating segmented GSYM files as we need to
+  // precompute exactly how big FunctionInfo objects encode into so we can
+  // accurately make segments of a specific size.
+  if (!EncodingCache.empty() &&
+      support::endian::system_endianness() == Out.getByteOrder()) {
+    // We already encoded this object, just write out the bytes.
+    Out.writeData(llvm::ArrayRef<uint8_t>((const uint8_t *)EncodingCache.data(),
+                                          EncodingCache.size()));
+    return FuncInfoOffset;
+  }
   // Write the size in bytes of this function as a uint32_t. This can be zero
   // if we just have a symbol from a symbol table and that symbol has no size.
-  O.writeU32(size());
+  Out.writeU32(size());
   // Write the name of this function as a uint32_t string table offset.
-  O.writeU32(Name);
+  Out.writeU32(Name);
 
   if (OptLineTable) {
-    O.writeU32(InfoType::LineTableInfo);
+    Out.writeU32(InfoType::LineTableInfo);
     // Write a uint32_t length as zero for now, we will fix this up after
     // writing the LineTable out with the number of bytes that were written.
-    O.writeU32(0);
-    const auto StartOffset = O.tell();
-    llvm::Error err = OptLineTable->encode(O, Range.start());
+    Out.writeU32(0);
+    const auto StartOffset = Out.tell();
+    llvm::Error err = OptLineTable->encode(Out, Range.start());
     if (err)
       return std::move(err);
-    const auto Length = O.tell() - StartOffset;
+    const auto Length = Out.tell() - StartOffset;
     if (Length > UINT32_MAX)
         return createStringError(std::errc::invalid_argument,
             "LineTable length is greater than UINT32_MAX");
     // Fixup the size of the LineTable data with the correct size.
-    O.fixup32(static_cast<uint32_t>(Length), StartOffset - 4);
+    Out.fixup32(static_cast<uint32_t>(Length), StartOffset - 4);
   }
 
   // Write out the inline function info if we have any and if it is valid.
   if (Inline) {
-    O.writeU32(InfoType::InlineInfo);
+    Out.writeU32(InfoType::InlineInfo);
     // Write a uint32_t length as zero for now, we will fix this up after
     // writing the LineTable out with the number of bytes that were written.
-    O.writeU32(0);
-    const auto StartOffset = O.tell();
-    llvm::Error err = Inline->encode(O, Range.start());
+    Out.writeU32(0);
+    const auto StartOffset = Out.tell();
+    llvm::Error err = Inline->encode(Out, Range.start());
     if (err)
       return std::move(err);
-    const auto Length = O.tell() - StartOffset;
+    const auto Length = Out.tell() - StartOffset;
     if (Length > UINT32_MAX)
         return createStringError(std::errc::invalid_argument,
             "InlineInfo length is greater than UINT32_MAX");
     // Fixup the size of the InlineInfo data with the correct size.
-    O.fixup32(static_cast<uint32_t>(Length), StartOffset - 4);
+    Out.fixup32(static_cast<uint32_t>(Length), StartOffset - 4);
   }
 
   // Terminate the data chunks with and end of list with zero size
-  O.writeU32(InfoType::EndOfList);
-  O.writeU32(0);
+  Out.writeU32(InfoType::EndOfList);
+  Out.writeU32(0);
   return FuncInfoOffset;
 }
 
diff --git a/llvm/lib/DebugInfo/GSYM/GsymCreator.cpp b/llvm/lib/DebugInfo/GSYM/GsymCreator.cpp
index 8281938770cf1..60b6dbc6a12d2 100644
--- a/llvm/lib/DebugInfo/GSYM/GsymCreator.cpp
+++ b/llvm/lib/DebugInfo/GSYM/GsymCreator.cpp
@@ -34,8 +34,10 @@ uint32_t GsymCreator::insertFile(StringRef Path, llvm::sys::path::Style Style) {
   // requirements.
   const uint32_t Dir = insertString(directory);
   const uint32_t Base = insertString(filename);
-  FileEntry FE(Dir, Base);
+  return insertFileEntry(FileEntry(Dir, Base));
+}
 
+uint32_t GsymCreator::insertFileEntry(FileEntry FE) {
   std::lock_guard<std::mutex> Guard(Mutex);
   const auto NextIndex = Files.size();
   // Find FE in hash map and insert if not present.
@@ -45,8 +47,26 @@ uint32_t GsymCreator::insertFile(StringRef Path, llvm::sys::path::Style Style) {
   return R.first->second;
 }
 
+uint32_t GsymCreator::copyFile(const GsymCreator &SrcGC, uint32_t FileIdx) {
+  // File index zero is reserved for a FileEntry with no directory and no
+  // filename. Any other file and we need to copy the strings for the directory
+  // and filename.
+  if (FileIdx == 0)
+    return 0;
+  const FileEntry SrcFE = SrcGC.Files[FileIdx];
+  // Copy the strings for the file and then add the newly converted file entry.
+  uint32_t Dir = StrTab.add(SrcGC.StringOffsetMap.find(SrcFE.Dir)->second);
+  uint32_t Base = StrTab.add(SrcGC.StringOffsetMap.find(SrcFE.Base)->second);
+  FileEntry DstFE(Dir, Base);
+  return insertFileEntry(DstFE);
+}
+
+
 llvm::Error GsymCreator::save(StringRef Path,
-                              llvm::support::endianness ByteOrder) const {
+                              llvm::support::endianness ByteOrder,
+                              std::optional<uint64_t> SegmentSize) const {
+  if (SegmentSize)
+    return saveSegments(Path, ByteOrder, *SegmentSize);
   std::error_code EC;
   raw_fd_ostream OutStrm(Path, EC);
   if (EC)
@@ -68,16 +88,17 @@ llvm::Error GsymCreator::encode(FileWriter &O) const {
     return createStringError(std::errc::invalid_argument,
                              "too many FunctionInfos");
 
-  const uint64_t MinAddr =
-      BaseAddress ? *BaseAddress : Funcs.front().startAddress();
-  const uint64_t MaxAddr = Funcs.back().startAddress();
-  const uint64_t AddrDelta = MaxAddr - MinAddr;
+  std::optional<uint64_t> BaseAddress = getBaseAddress();
+  // Base address should be valid if we have any functions.
+  if (!BaseAddress)
+    return createStringError(std::errc::invalid_argument,
+                             "invalid base address");
   Header Hdr;
   Hdr.Magic = GSYM_MAGIC;
   Hdr.Version = GSYM_VERSION;
-  Hdr.AddrOffSize = 0;
+  Hdr.AddrOffSize = getAddressOffsetSize();
   Hdr.UUIDSize = static_cast<uint8_t>(UUID.size());
-  Hdr.BaseAddress = MinAddr;
+  Hdr.BaseAddress = *BaseAddress;
   Hdr.NumAddresses = static_cast<uint32_t>(Funcs.size());
   Hdr.StrtabOffset = 0; // We will fix this up later.
   Hdr.StrtabSize = 0;   // We will fix this up later.
@@ -85,15 +106,6 @@ llvm::Error GsymCreator::encode(FileWriter &O) const {
   if (UUID.size() > sizeof(Hdr.UUID))
     return createStringError(std::errc::invalid_argument,
                              "invalid UUID size %u", (uint32_t)UUID.size());
-  // Set the address offset size correctly in the GSYM header.
-  if (AddrDelta <= UINT8_MAX)
-    Hdr.AddrOffSize = 1;
-  else if (AddrDelta <= UINT16_MAX)
-    Hdr.AddrOffSize = 2;
-  else if (AddrDelta <= UINT32_MAX)
-    Hdr.AddrOffSize = 4;
-  else
-    Hdr.AddrOffSize = 8;
   // Copy the UUID value if we have one.
   if (UUID.size() > 0)
     memcpy(Hdr.UUID, UUID.data(), UUID.size());
@@ -102,10 +114,17 @@ llvm::Error GsymCreator::encode(FileWriter &O) const {
   if (Err)
     return Err;
 
+  const uint64_t MaxAddressOffset = getMaxAddressOffset();
   // Write out the address offsets.
   O.alignTo(Hdr.AddrOffSize);
   for (const auto &FuncInfo : Funcs) {
     uint64_t AddrOffset = FuncInfo.startAddress() - Hdr.BaseAddress;
+    // Make sure we calculated the address offsets byte size correctly by
+    // verifying the current address offset is within ranges. We have seen bugs
+    // introduced when the code changes that can cause problems here so it is
+    // good to catch this during testing.
+    assert(AddrOffset <= MaxAddressOffset);
+    (void)MaxAddressOffset;
     switch (Hdr.AddrOffSize) {
     case 1:
       O.writeU8(static_cast<uint8_t>(AddrOffset));
@@ -142,7 +161,7 @@ llvm::Error GsymCreator::encode(FileWriter &O) const {
     O.writeU32(File.Base);
   }
 
-  // Write out the sting table.
+  // Write out the string table.
   const off_t StrtabOffset = O.tell();
   StrTab.write(O.get_stream());
   const off_t StrtabSize = O.tell() - StrtabOffset;
@@ -300,6 +319,13 @@ llvm::Error GsymCreator::finalize(llvm::raw_ostream &OS) {
   return Error::success();
 }
 
+uint32_t GsymCreator::copyString(const GsymCreator &SrcGC, uint32_t StrOff) {
+  // String offset at zero is always the empty string, no copying needed.
+  if (StrOff == 0)
+    return 0;
+  return StrTab.add(SrcGC.StringOffsetMap.find(StrOff)->second);
+}
+
 uint32_t GsymCreator::insertString(StringRef S, bool Copy) {
   if (S.empty())
     return 0;
@@ -318,7 +344,13 @@ uint32_t GsymCreator::insertString(StringRef S, bool Copy) {
       CHStr = CachedHashStringRef{StringStorage.insert(S).first->getKey(),
                                   CHStr.hash()};
   }
-  return StrTab.add(CHStr);
+  const uint32_t StrOff = StrTab.add(CHStr);
+  // Save a mapping of string offsets to the cached string reference in case
+  // we need to segment the GSYM file and copy string from one string table to
+  // another.
+  if (StringOffsetMap.count(StrOff) == 0)
+    StringOffsetMap.insert(std::make_pair(StrOff, CHStr));
+  return StrOff;
 }
 
 void GsymCreator::addFunctionInfo(FunctionInfo &&FI) {
@@ -360,3 +392,187 @@ bool GsymCreator::hasFunctionInfoForAddress(uint64_t Addr) const {
   std::lock_guard<std::mutex> Guard(Mutex);
   return Ranges.contains(Addr);
 }
+
+std::optional<uint64_t> GsymCreator::getFirstFunctionAddress() const {
+  if (Finalized && !Funcs.empty())
+    return std::optional<uint64_t>(Funcs.front().startAddress());
+  // This code gets used by the segmentation of GSYM files to help determine the
+  // size of the GSYM header while continually adding new FunctionInfo objects
+  // to this object, so we haven't finalized this object yet.
+  if (Ranges.empty())
+    return std::nullopt;
+  return std::optional<uint64_t>(Ranges.begin()->start());
+}
+
+std::optional<uint64_t> GsymCreator::getLastFunctionAddress() const {
+  if (Finalized && !Funcs.empty())
+    return std::optional<uint64_t>(Funcs.back().startAddress());
+  // This code gets used by the segmentation of GSYM files to help determine the
+  // size of the GSYM header while continually adding new FunctionInfo objects
+  // to this object, so we haven't finalized this object yet.
+  if (Ranges.empty())
+    return std::nullopt;
+  return std::optional<uint64_t>((Ranges.end() - 1)->end());
+}
+
+std::optional<uint64_t> GsymCreator::getBaseAddress() const {
+  if (BaseAddress)
+    return BaseAddress;
+  return getFirstFunctionAddress();
+}
+
+uint64_t GsymCreator::getMaxAddressOffset() const {
+  switch (getAddressOffsetSize()) {
+    case 1: return UINT8_MAX;
+    case 2: return UINT16_MAX;
+    case 4: return UINT32_MAX;
+    case 8: return UINT64_MAX;
+  }
+  llvm_unreachable("invalid address offset");
+}
+
+uint8_t GsymCreator::getAddressOffsetSize() const {
+  const std::optional<uint64_t> BaseAddress = getBaseAddress();
+  const std::optional<uint64_t> LastFuncAddr = getLastFunctionAddress();
+  if (BaseAddress && LastFuncAddr) {
+    const uint64_t AddrDelta = *LastFuncAddr - *BaseAddress;
+    if (AddrDelta <= UINT8_MAX)
+      return 1;
+    else if (AddrDelta <= UINT16_MAX)
+      return 2;
+    else if (AddrDelta <= UINT32_MAX)
+      return 4;
+    return 8;
+  }
+  return 1;
+}
+
+uint64_t GsymCreator::calculateHeaderAndTableSize() const {
+  uint64_t Size = sizeof(Header);
+  const size_t NumFuncs = Funcs.size();
+  // Add size of address offset table
+  Size += NumFuncs * getAddressOffsetSize();
+  // Add size of address info offsets which are 32 bit integers in version 1.
+  Size += NumFuncs * sizeof(uint32_t);
+  // Add file table size
+  Size += Files.size() * sizeof(FileEntry);
+  // Add string table size
+  Size += StrTab.getSize();
+
+  return Size;
+}
+
+// This function takes a InlineInfo class that was copy constructed from an
+// InlineInfo from the \a SrcGC and updates all members that point to strings
+// and files to point to strings and files from this GsymCreator.
+void GsymCreator::fixupInlineInfo(const GsymCreator &SrcGC, InlineInfo &II) {
+  II.Name = copyString(SrcGC, II.Name);
+  II.CallFile = copyFile(SrcGC, II.CallFile);
+  for (auto &ChildII: II.Children)
+    fixupInlineInfo(SrcGC, ChildII);
+}
+
+uint64_t GsymCreator::copyFunctionInfo(const GsymCreator &SrcGC, size_t FuncIdx) {
+  // To copy a function info we need to copy any files and strings over into
+  // this GsymCreator and then copy the function info and update the string
+  // table offsets to match the new offsets.
+  const FunctionInfo &SrcFI = SrcGC.Funcs[FuncIdx];
+  Ranges.insert(SrcFI.Range);
+
+  FunctionInfo DstFI;
+  DstFI.Range = SrcFI.Range;
+  DstFI.Name = copyString(SrcGC, SrcFI.Name);
+  // Copy the line table if there is one.
+  if (SrcFI.OptLineTable) {
+    // Copy the entire line table.
+    DstFI.OptLineTable = LineTable(SrcFI.OptLineTable.value());
+    // Fixup all LineEntry::File entries which are indexes in the the file table
+    // from SrcGC and must be converted to file indexes from this GsymCreator.
+    LineTable &DstLT = DstFI.OptLineTable.value();
+    const size_t NumLines = DstLT.size();
+    for (size_t I=0; I<NumLines; ++I) {
+      LineEntry &LE = DstLT.get(I);
+      LE.File = copyFile(SrcGC, LE.File);
+    }
+  }
+  // Copy the inline information if needed.
+  if (SrcFI.Inline) {
+    // Make a copy of the source inline information.
+    DstFI.Inline = SrcFI.Inline.value();
+    // Fixup all strings and files in the copied inline information.
+    fixupInlineInfo(SrcGC, *DstFI.Inline);
+  }
+  std::lock_guard<std::mutex> Guard(Mutex);
+  Funcs.push_back(DstFI);
+  return Funcs.back().cacheEncoding();
+}
+
+llvm::Error GsymCreator::saveSegments(StringRef Path,
+                                      llvm::support::endianness ByteOrder,
+                                      uint64_t SegmentSize) const {
+  if (SegmentSize == 0)
+    return createStringError(std::errc::invalid_argument,
+                             "invalid segment size zero");
+
+  size_t FuncIdx = 0;
+  const size_t NumFuncs = Funcs.size();
+  while (FuncIdx < NumFuncs) {
+    llvm::Expected<std::unique_ptr<GsymCreator>> ExpectedGC =
+        createSegment(SegmentSize, FuncIdx);
+    if (ExpectedGC) {
+      GsymCreator *GC = ExpectedGC->get();
+      if (GC == NULL)
+        break; // We had not more functions to encode.
+      raw_null_ostream ErrorStrm;
+      llvm::Error Err = GC->finalize(ErrorStrm);
+      if (Err)
+        return Err;
+      std::string SegmentedGsymPath;
+      raw_string_ostream SGP(SegmentedGsymPath);
+      std::optional<uint64_t> FirstFuncAddr = GC->getFirstFunctionAddress();
+      if (FirstFuncAddr) {
+        SGP << Path << "-" << llvm::format_hex(*FirstFuncAddr, 1);
+        SGP.flush();
+        Err = GC->save(SegmentedGsymPath, ByteOrder, std::nullopt);
+        if (Err)
+          return Err;
+      }
+    } else {
+      return ExpectedGC.takeError();
+    }
+  }
+  return Error::success();
+}
+
+llvm::Expected<std::unique_ptr<GsymCreator>>
+GsymCreator::createSegment(uint64_t SegmentSize, size_t &FuncIdx) const {
+  // No function entries, return empty unique pointer
+  if (FuncIdx >= Funcs.size())
+    return std::unique_ptr<GsymCreator>();
+
+  std::unique_ptr<GsymCreator> GC(new GsymCreator(/*Quiet=*/true));
+  // Set the base address if there is one.
+  if (BaseAddress)
+    GC->setBaseAddress(*BaseAddress);
+  // Copy the UUID value from this object into the new creator.
+  GC->setUUID(UUID);
+  const size_t NumFuncs = Funcs.size();
+  // Track how big the function infos are for the current segment so we can
+  // emit segments that are close to the requested size. It is quick math to
+  // determine the current header and tables sizes, so we can do that each loop.
+  uint64_t SegmentFuncInfosSize = 0;
+  for (; FuncIdx < NumFuncs; ++FuncIdx) {
+    const uint64_t HeaderAndTableSize = GC->calculateHeaderAndTableSize();
+    if (HeaderAndTableSize + SegmentFuncInfosSize >= SegmentSize) {
+      if (SegmentFuncInfosSize == 0)
+        return createStringError(std::errc::invalid_argument,
+                                 "a segment size of %" PRIu64 " is to small to "
+                                 "fit any function infos, specify a larger value",
+                                 SegmentSize);
+
+      break;
+    }
+    SegmentFuncInfosSize += alignTo(GC->copyFunctionInfo(*this, FuncIdx), 4);
+  }
+  return std::move(GC);
+}
diff --git a/llvm/lib/DebugInfo/PDB/PDBExtras.cpp b/llvm/lib/DebugInfo/PDB/PDBExtras.cpp
index 571510e6bad9c..2b318bf1c6488 100644
--- a/llvm/lib/DebugInfo/PDB/PDBExtras.cpp
+++ b/llvm/lib/DebugInfo/PDB/PDBExtras.cpp
@@ -232,6 +232,8 @@ raw_ostream &llvm::pdb::operator<<(raw_ostream &OS, const PDB_Lang &Lang) {
     CASE_OUTPUT_ENUM_CLASS_NAME(PDB_Lang, D, OS)
     CASE_OUTPUT_ENUM_CLASS_NAME(PDB_Lang, Swift, OS)
     CASE_OUTPUT_ENUM_CLASS_NAME(PDB_Lang, Rust, OS)
+    CASE_OUTPUT_ENUM_CLASS_NAME(PDB_Lang, ObjC, OS)
+    CASE_OUTPUT_ENUM_CLASS_STR(PDB_Lang, ObjCpp, "ObjC++", OS)
   }
   return OS;
 }
diff --git a/llvm/lib/DebugInfo/PDB/PDBSymbolCompiland.cpp b/llvm/lib/DebugInfo/PDB/PDBSymbolCompiland.cpp
index bd60489b6bed6..437b96677c0b5 100644
--- a/llvm/lib/DebugInfo/PDB/PDBSymbolCompiland.cpp
+++ b/llvm/lib/DebugInfo/PDB/PDBSymbolCompiland.cpp
@@ -102,6 +102,8 @@ std::string PDBSymbolCompiland::getSourceFileFullPath() const {
               .Case(".asm", Lang == PDB_Lang::Masm)
               .Case(".swift", Lang == PDB_Lang::Swift)
               .Case(".rs", Lang == PDB_Lang::Rust)
+              .Case(".m", Lang == PDB_Lang::ObjC)
+              .Case(".mm", Lang == PDB_Lang::ObjCpp)
               .Default(false))
         return File->getFileName();
     }
diff --git a/llvm/lib/Debuginfod/Debuginfod.cpp b/llvm/lib/Debuginfod/Debuginfod.cpp
index 2b0710b536bae..af84fe4176b75 100644
--- a/llvm/lib/Debuginfod/Debuginfod.cpp
+++ b/llvm/lib/Debuginfod/Debuginfod.cpp
@@ -251,17 +251,25 @@ Expected<std::string> getCachedOrDownloadArtifact(
 
     // Perform the HTTP request and if successful, write the response body to
     // the cache.
-    StreamedHTTPResponseHandler Handler(
-        [&]() { return CacheAddStream(Task, ""); }, Client);
-    HTTPRequest Request(ArtifactUrl);
-    Request.Headers = getHeaders();
-    Error Err = Client.perform(Request, Handler);
-    if (Err)
-      return std::move(Err);
+    {
+      StreamedHTTPResponseHandler Handler(
+          [&]() { return CacheAddStream(Task, ""); }, Client);
+      HTTPRequest Request(ArtifactUrl);
+      Request.Headers = getHeaders();
+      Error Err = Client.perform(Request, Handler);
+      if (Err)
+        return std::move(Err);
+
+      unsigned Code = Client.responseCode();
+      if (Code && Code != 200)
+        continue;
+    }
 
-    unsigned Code = Client.responseCode();
-    if (Code && Code != 200)
-      continue;
+    Expected<CachePruningPolicy> PruningPolicyOrErr =
+        parseCachePruningPolicy(std::getenv("DEBUGINFOD_CACHE_POLICY"));
+    if (!PruningPolicyOrErr)
+      return PruningPolicyOrErr.takeError();
+    pruneCache(CacheDirectoryPath, *PruningPolicyOrErr);
 
     // Return the path to the artifact on disk.
     return std::string(AbsCachedArtifactPath);
diff --git a/llvm/lib/ExecutionEngine/GDBRegistrationListener.cpp b/llvm/lib/ExecutionEngine/GDBRegistrationListener.cpp
index f1eeee3b3599d..b5b76130c55eb 100644
--- a/llvm/lib/ExecutionEngine/GDBRegistrationListener.cpp
+++ b/llvm/lib/ExecutionEngine/GDBRegistrationListener.cpp
@@ -176,7 +176,7 @@ void GDBJITRegistrationListener::notifyObjectLoaded(
   size_t      Size = DebugObj.getBinary()->getMemoryBufferRef().getBufferSize();
 
   std::lock_guard<llvm::sys::Mutex> locked(JITDebugLock);
-  assert(ObjectBufferMap.find(K) == ObjectBufferMap.end() &&
+  assert(!ObjectBufferMap.contains(K) &&
          "Second attempt to perform debug registration.");
   jit_code_entry* JITCodeEntry = new jit_code_entry();
 
diff --git a/llvm/lib/ExecutionEngine/Interpreter/ExternalFunctions.cpp b/llvm/lib/ExecutionEngine/Interpreter/ExternalFunctions.cpp
index 73b1023fffac1..4f8f883a75f32 100644
--- a/llvm/lib/ExecutionEngine/Interpreter/ExternalFunctions.cpp
+++ b/llvm/lib/ExecutionEngine/Interpreter/ExternalFunctions.cpp
@@ -346,8 +346,10 @@ static GenericValue lle_X_abort(FunctionType *FT, ArrayRef<GenericValue> Args) {
 
 // Silence warnings about sprintf. (See also
 // https://github.com/llvm/llvm-project/issues/58086)
+#if defined(__clang__)
 #pragma clang diagnostic push
 #pragma clang diagnostic ignored "-Wdeprecated-declarations"
+#endif
 // int sprintf(char *, const char *, ...) - a very rough implementation to make
 // output useful.
 static GenericValue lle_X_sprintf(FunctionType *FT,
@@ -429,7 +431,9 @@ static GenericValue lle_X_sprintf(FunctionType *FT,
   }
   return GV;
 }
+#if defined(__clang__)
 #pragma clang diagnostic pop
+#endif
 
 // int printf(const char *, ...) - a very rough implementation to make output
 // useful.
diff --git a/llvm/lib/ExecutionEngine/JITLink/COFFLinkGraphBuilder.cpp b/llvm/lib/ExecutionEngine/JITLink/COFFLinkGraphBuilder.cpp
index 481d6e72b245e..fe6252f9aa1f9 100644
--- a/llvm/lib/ExecutionEngine/JITLink/COFFLinkGraphBuilder.cpp
+++ b/llvm/lib/ExecutionEngine/JITLink/COFFLinkGraphBuilder.cpp
@@ -152,8 +152,11 @@ Error COFFLinkGraphBuilder::graphifySections() {
 
     // Look for existing sections first.
     auto *GraphSec = G->findSectionByName(SectionName);
-    if (!GraphSec)
+    if (!GraphSec) {
       GraphSec = &G->createSection(SectionName, Prot);
+      if ((*Sec)->Characteristics & COFF::IMAGE_SCN_LNK_REMOVE)
+        GraphSec->setMemLifetimePolicy(orc::MemLifetimePolicy::NoAlloc);
+    }
     if (GraphSec->getMemProt() != Prot)
       return make_error<JITLinkError>("MemProt should match");
 
diff --git a/llvm/lib/ExecutionEngine/JITLink/ELFLinkGraphBuilder.h b/llvm/lib/ExecutionEngine/JITLink/ELFLinkGraphBuilder.h
index 953a9f512784b..26feb8ea3277b 100644
--- a/llvm/lib/ExecutionEngine/JITLink/ELFLinkGraphBuilder.h
+++ b/llvm/lib/ExecutionEngine/JITLink/ELFLinkGraphBuilder.h
@@ -62,6 +62,14 @@ class ELFLinkGraphBuilder : public ELFLinkGraphBuilderBase {
                       StringRef FileName,
                       LinkGraph::GetEdgeKindNameFunction GetEdgeKindName);
 
+  /// Debug sections are included in the graph by default. Use
+  /// setProcessDebugSections(false) to ignore them if debug info is not
+  /// needed.
+  ELFLinkGraphBuilder &setProcessDebugSections(bool ProcessDebugSections) {
+    this->ProcessDebugSections = ProcessDebugSections;
+    return *this;
+  }
+
   /// Attempt to construct and return the LinkGraph.
   Expected<std::unique_ptr<LinkGraph>> buildGraph();
 
@@ -115,8 +123,7 @@ class ELFLinkGraphBuilder : public ELFLinkGraphBuilderBase {
   ///
   template <typename RelocHandlerMethod>
   Error forEachRelaRelocation(const typename ELFT::Shdr &RelSect,
-                              RelocHandlerMethod &&Func,
-                              bool ProcessDebugSections = false);
+                              RelocHandlerMethod &&Func);
 
   /// Traverse all matching ELFT::Rel relocation records in the given section.
   /// The handler function Func should be callable with this signature:
@@ -125,22 +132,19 @@ class ELFLinkGraphBuilder : public ELFLinkGraphBuilderBase {
   ///
   template <typename RelocHandlerMethod>
   Error forEachRelRelocation(const typename ELFT::Shdr &RelSect,
-                             RelocHandlerMethod &&Func,
-                             bool ProcessDebugSections = false);
+                             RelocHandlerMethod &&Func);
 
   /// Traverse all matching rela relocation records in the given section.
   /// Convenience wrapper to allow passing a member function for the handler.
   ///
   template <typename ClassT, typename RelocHandlerMethod>
   Error forEachRelaRelocation(const typename ELFT::Shdr &RelSect,
-                              ClassT *Instance, RelocHandlerMethod &&Method,
-                              bool ProcessDebugSections = false) {
+                              ClassT *Instance, RelocHandlerMethod &&Method) {
     return forEachRelaRelocation(
         RelSect,
         [Instance, Method](const auto &Rel, const auto &Target, auto &GS) {
           return (Instance->*Method)(Rel, Target, GS);
-        },
-        ProcessDebugSections);
+        });
   }
 
   /// Traverse all matching rel relocation records in the given section.
@@ -148,14 +152,12 @@ class ELFLinkGraphBuilder : public ELFLinkGraphBuilderBase {
   ///
   template <typename ClassT, typename RelocHandlerMethod>
   Error forEachRelRelocation(const typename ELFT::Shdr &RelSect,
-                             ClassT *Instance, RelocHandlerMethod &&Method,
-                             bool ProcessDebugSections = false) {
+                             ClassT *Instance, RelocHandlerMethod &&Method) {
     return forEachRelRelocation(
         RelSect,
         [Instance, Method](const auto &Rel, const auto &Target, auto &GS) {
           return (Instance->*Method)(Rel, Target, GS);
-        },
-        ProcessDebugSections);
+        });
   }
 
   const ELFFile &Obj;
@@ -163,6 +165,7 @@ class ELFLinkGraphBuilder : public ELFLinkGraphBuilderBase {
   typename ELFFile::Elf_Shdr_Range Sections;
   const typename ELFFile::Elf_Shdr *SymTabSec = nullptr;
   StringRef SectionStringTab;
+  bool ProcessDebugSections = true;
 
   // Maps ELF section indexes to LinkGraph Blocks.
   // Only SHF_ALLOC sections will have graph blocks.
@@ -308,22 +311,20 @@ template <typename ELFT> Error ELFLinkGraphBuilder<ELFT>::graphifySections() {
     if (!Name)
       return Name.takeError();
 
-    // If the name indicates that it's a debug section then skip it: We don't
-    // support those yet.
-    if (isDwarfSection(*Name)) {
+    // Skip null sections.
+    if (Sec.sh_type == ELF::SHT_NULL) {
       LLVM_DEBUG({
-        dbgs() << "    " << SecIndex << ": \"" << *Name
-               << "\" is a debug section: "
-                  "No graph section will be created.\n";
+        dbgs() << "    " << SecIndex << ": has type SHT_NULL. Skipping.\n";
       });
       continue;
     }
 
-    // Skip non-SHF_ALLOC sections
-    if (!(Sec.sh_flags & ELF::SHF_ALLOC)) {
+    // If the name indicates that it's a debug section then skip it: We don't
+    // support those yet.
+    if (!ProcessDebugSections && isDwarfSection(*Name)) {
       LLVM_DEBUG({
         dbgs() << "    " << SecIndex << ": \"" << *Name
-               << "\" is not an SHF_ALLOC section: "
+               << "\" is a debug section: "
                   "No graph section will be created.\n";
       });
       continue;
@@ -343,8 +344,18 @@ template <typename ELFT> Error ELFLinkGraphBuilder<ELFT>::graphifySections() {
 
     // Look for existing sections first.
     auto *GraphSec = G->findSectionByName(*Name);
-    if (!GraphSec)
+    if (!GraphSec) {
       GraphSec = &G->createSection(*Name, Prot);
+      // Non-SHF_ALLOC sections get NoAlloc memory lifetimes.
+      if (!(Sec.sh_flags & ELF::SHF_ALLOC)) {
+        GraphSec->setMemLifetimePolicy(orc::MemLifetimePolicy::NoAlloc);
+        LLVM_DEBUG({
+          dbgs() << "      " << SecIndex << ": \"" << *Name
+                 << "\" is not a SHF_ALLOC section. Using NoAlloc lifetime.\n";
+        });
+      }
+    }
+
     assert(GraphSec->getMemProt() == Prot && "MemProt should match");
 
     Block *B = nullptr;
@@ -514,8 +525,7 @@ template <typename ELFT> Error ELFLinkGraphBuilder<ELFT>::graphifySymbols() {
 template <typename ELFT>
 template <typename RelocHandlerFunction>
 Error ELFLinkGraphBuilder<ELFT>::forEachRelaRelocation(
-    const typename ELFT::Shdr &RelSect, RelocHandlerFunction &&Func,
-    bool ProcessDebugSections) {
+    const typename ELFT::Shdr &RelSect, RelocHandlerFunction &&Func) {
   // Only look into sections that store relocation entries.
   if (RelSect.sh_type != ELF::SHT_RELA)
     return Error::success();
@@ -561,8 +571,7 @@ Error ELFLinkGraphBuilder<ELFT>::forEachRelaRelocation(
 template <typename ELFT>
 template <typename RelocHandlerFunction>
 Error ELFLinkGraphBuilder<ELFT>::forEachRelRelocation(
-    const typename ELFT::Shdr &RelSect, RelocHandlerFunction &&Func,
-    bool ProcessDebugSections) {
+    const typename ELFT::Shdr &RelSect, RelocHandlerFunction &&Func) {
   // Only look into sections that store relocation entries.
   if (RelSect.sh_type != ELF::SHT_REL)
     return Error::success();
diff --git a/llvm/lib/ExecutionEngine/JITLink/ELF_aarch64.cpp b/llvm/lib/ExecutionEngine/JITLink/ELF_aarch64.cpp
index 5b9553ec8b116..3eb7e1bccde7e 100644
--- a/llvm/lib/ExecutionEngine/JITLink/ELF_aarch64.cpp
+++ b/llvm/lib/ExecutionEngine/JITLink/ELF_aarch64.cpp
@@ -552,6 +552,7 @@ void link_ELF_aarch64(std::unique_ptr<LinkGraph> G,
     Config.PrePrunePasses.push_back(EHFrameEdgeFixer(
         ".eh_frame", 8, aarch64::Pointer32, aarch64::Pointer64,
         aarch64::Delta32, aarch64::Delta64, aarch64::NegDelta32));
+    Config.PrePrunePasses.push_back(EHFrameNullTerminator(".eh_frame"));
 
     // Add a mark-live pass.
     if (auto MarkLive = Ctx->getMarkLivePass(TT))
diff --git a/llvm/lib/ExecutionEngine/JITLink/ELF_x86_64.cpp b/llvm/lib/ExecutionEngine/JITLink/ELF_x86_64.cpp
index c9359522c2480..ab2ca449ec8ee 100644
--- a/llvm/lib/ExecutionEngine/JITLink/ELF_x86_64.cpp
+++ b/llvm/lib/ExecutionEngine/JITLink/ELF_x86_64.cpp
@@ -108,6 +108,7 @@ class ELFLinkGraphBuilder_x86_64 : public ELFLinkGraphBuilder<object::ELF64LE> {
 
   enum ELFX86RelocationKind : Edge::Kind {
     Branch32 = Edge::FirstRelocation,
+    Pointer32,
     Pointer32Signed,
     Pointer64,
     PCRel32,
@@ -123,6 +124,8 @@ class ELFLinkGraphBuilder_x86_64 : public ELFLinkGraphBuilder<object::ELF64LE> {
 
   static Expected<ELFX86RelocationKind> getRelocationKind(const uint32_t Type) {
     switch (Type) {
+    case ELF::R_X86_64_32:
+      return ELFX86RelocationKind::Pointer32;
     case ELF::R_X86_64_32S:
       return ELFX86RelocationKind::Pointer32Signed;
     case ELF::R_X86_64_PC32:
@@ -207,6 +210,9 @@ class ELFLinkGraphBuilder_x86_64 : public ELFLinkGraphBuilder<object::ELF64LE> {
     case Delta64:
       Kind = x86_64::Delta64;
       break;
+    case Pointer32:
+      Kind = x86_64::Pointer32;
+      break;
     case Pointer32Signed:
       Kind = x86_64::Pointer32Signed;
       break;
diff --git a/llvm/lib/ExecutionEngine/JITLink/JITLinkGeneric.h b/llvm/lib/ExecutionEngine/JITLink/JITLinkGeneric.h
index 2c92445265364..e69eddd6e1194 100644
--- a/llvm/lib/ExecutionEngine/JITLink/JITLinkGeneric.h
+++ b/llvm/lib/ExecutionEngine/JITLink/JITLinkGeneric.h
@@ -123,26 +123,47 @@ template <typename LinkerImpl> class JITLinker : public JITLinkerBase {
   Error fixUpBlocks(LinkGraph &G) const override {
     LLVM_DEBUG(dbgs() << "Fixing up blocks:\n");
 
-    for (auto *B : G.blocks()) {
-      LLVM_DEBUG(dbgs() << "  " << *B << ":\n");
-
-      // Copy Block data and apply fixups.
-      LLVM_DEBUG(dbgs() << "    Applying fixups.\n");
-      assert((!B->isZeroFill() || all_of(B->edges(),
-                                         [](const Edge &E) {
-                                           return E.getKind() ==
-                                                  Edge::KeepAlive;
-                                         })) &&
-             "Non-KeepAlive edges in zero-fill block?");
-      for (auto &E : B->edges()) {
-
-        // Skip non-relocation edges.
-        if (!E.isRelocation())
-          continue;
-
-        // Dispatch to LinkerImpl for fixup.
-        if (auto Err = impl().applyFixup(G, *B, E))
-          return Err;
+    for (auto &Sec : G.sections()) {
+      bool NoAllocSection =
+          Sec.getMemLifetimePolicy() == orc::MemLifetimePolicy::NoAlloc;
+
+      for (auto *B : Sec.blocks()) {
+        LLVM_DEBUG(dbgs() << "  " << *B << ":\n");
+
+        // Copy Block data and apply fixups.
+        LLVM_DEBUG(dbgs() << "    Applying fixups.\n");
+        assert((!B->isZeroFill() || all_of(B->edges(),
+                                           [](const Edge &E) {
+                                             return E.getKind() ==
+                                                    Edge::KeepAlive;
+                                           })) &&
+               "Non-KeepAlive edges in zero-fill block?");
+
+        // If this is a no-alloc section then copy the block content into
+        // memory allocated on the Graph's allocator (if it hasn't been
+        // already).
+        if (NoAllocSection)
+          (void)B->getMutableContent(G);
+
+        for (auto &E : B->edges()) {
+
+          // Skip non-relocation edges.
+          if (!E.isRelocation())
+            continue;
+
+          // If B is a block in a Standard or Finalize section then make sure
+          // that no edges point to symbols in NoAlloc sections.
+          assert(
+              (NoAllocSection || !E.getTarget().isDefined() ||
+               E.getTarget().getBlock().getSection().getMemLifetimePolicy() !=
+                   orc::MemLifetimePolicy::NoAlloc) &&
+              "Block in allocated section has edge pointing to no-alloc "
+              "section");
+
+          // Dispatch to LinkerImpl for fixup.
+          if (auto Err = impl().applyFixup(G, *B, E))
+            return Err;
+        }
       }
     }
 
diff --git a/llvm/lib/ExecutionEngine/JITLink/JITLinkMemoryManager.cpp b/llvm/lib/ExecutionEngine/JITLink/JITLinkMemoryManager.cpp
index e74aa059f4054..f481504135a5f 100644
--- a/llvm/lib/ExecutionEngine/JITLink/JITLinkMemoryManager.cpp
+++ b/llvm/lib/ExecutionEngine/JITLink/JITLinkMemoryManager.cpp
@@ -24,11 +24,12 @@ JITLinkMemoryManager::InFlightAlloc::~InFlightAlloc() = default;
 BasicLayout::BasicLayout(LinkGraph &G) : G(G) {
 
   for (auto &Sec : G.sections()) {
-    // Skip empty sections.
-    if (Sec.blocks().empty())
+    // Skip empty sections, and sections with NoAlloc lifetime policies.
+    if (Sec.blocks().empty() ||
+        Sec.getMemLifetimePolicy() == orc::MemLifetimePolicy::NoAlloc)
       continue;
 
-    auto &Seg = Segments[{Sec.getMemProt(), Sec.getMemDeallocPolicy()}];
+    auto &Seg = Segments[{Sec.getMemProt(), Sec.getMemLifetimePolicy()}];
     for (auto *B : Sec.blocks())
       if (LLVM_LIKELY(!B->isZeroFill()))
         Seg.ContentBlocks.push_back(B);
@@ -89,7 +90,7 @@ BasicLayout::getContiguousPageBasedLayoutSizes(uint64_t PageSize) {
                                      inconvertibleErrorCode());
 
     uint64_t SegSize = alignTo(Seg.ContentSize + Seg.ZeroFillSize, PageSize);
-    if (AG.getMemDeallocPolicy() == orc::MemDeallocPolicy::Standard)
+    if (AG.getMemLifetimePolicy() == orc::MemLifetimePolicy::Standard)
       SegsSizes.StandardSegs += SegSize;
     else
       SegsSizes.FinalizeSegs += SegSize;
@@ -146,7 +147,7 @@ void SimpleSegmentAlloc::Create(JITLinkMemoryManager &MemMgr,
                                 const JITLinkDylib *JD, SegmentMap Segments,
                                 OnCreatedFunction OnCreated) {
 
-  static_assert(orc::AllocGroup::NumGroups == 16,
+  static_assert(orc::AllocGroup::NumGroups == 32,
                 "AllocGroup has changed. Section names below must be updated");
   StringRef AGSectionNames[] = {
       "__---.standard", "__R--.standard", "__-W-.standard", "__RW-.standard",
@@ -163,12 +164,15 @@ void SimpleSegmentAlloc::Create(JITLinkMemoryManager &MemMgr,
     auto &AG = KV.first;
     auto &Seg = KV.second;
 
+    assert(AG.getMemLifetimePolicy() != orc::MemLifetimePolicy::NoAlloc &&
+           "NoAlloc segments are not supported by SimpleSegmentAlloc");
+
     auto AGSectionName =
         AGSectionNames[static_cast<unsigned>(AG.getMemProt()) |
-                       static_cast<bool>(AG.getMemDeallocPolicy()) << 3];
+                       static_cast<bool>(AG.getMemLifetimePolicy()) << 3];
 
     auto &Sec = G->createSection(AGSectionName, AG.getMemProt());
-    Sec.setMemDeallocPolicy(AG.getMemDeallocPolicy());
+    Sec.setMemLifetimePolicy(AG.getMemLifetimePolicy());
 
     if (Seg.ContentSize != 0) {
       NextAddr =
@@ -416,7 +420,7 @@ void InProcessMemoryManager::allocate(const JITLinkDylib *JD, LinkGraph &G,
     auto &Seg = KV.second;
 
     auto &SegAddr =
-        (AG.getMemDeallocPolicy() == orc::MemDeallocPolicy::Standard)
+        (AG.getMemLifetimePolicy() == orc::MemLifetimePolicy::Standard)
             ? NextStandardSegAddr
             : NextFinalizeSegAddr;
 
diff --git a/llvm/lib/ExecutionEngine/JITLink/MachOLinkGraphBuilder.cpp b/llvm/lib/ExecutionEngine/JITLink/MachOLinkGraphBuilder.cpp
index bdf0eb6012892..385230c53f914 100644
--- a/llvm/lib/ExecutionEngine/JITLink/MachOLinkGraphBuilder.cpp
+++ b/llvm/lib/ExecutionEngine/JITLink/MachOLinkGraphBuilder.cpp
@@ -189,6 +189,10 @@ Error MachOLinkGraphBuilder::createNormalizedSections() {
     NSec.GraphSection = &G->createSection(
         StringRef(FullyQualifiedName.data(), FullyQualifiedName.size()), Prot);
 
+    // TODO: Are there any other criteria for NoAlloc lifetime?
+    if (NSec.Flags & MachO::S_ATTR_DEBUG)
+      NSec.GraphSection->setMemLifetimePolicy(orc::MemLifetimePolicy::NoAlloc);
+
     IndexToSection.insert(std::make_pair(SecIndex, std::move(NSec)));
   }
 
diff --git a/llvm/lib/ExecutionEngine/Orc/COFFPlatform.cpp b/llvm/lib/ExecutionEngine/Orc/COFFPlatform.cpp
index a3cef492ad018..dfafc429843fb 100644
--- a/llvm/lib/ExecutionEngine/Orc/COFFPlatform.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/COFFPlatform.cpp
@@ -352,11 +352,11 @@ bool COFFPlatform::supportedTarget(const Triple &TT) {
 COFFPlatform::COFFPlatform(ExecutionSession &ES,
                            ObjectLinkingLayer &ObjLinkingLayer,
                            JITDylib &PlatformJD, const char *OrcRuntimePath,
-                           LoadDynamicLibrary LoadDynLibrary,
+                           LoadDynamicLibrary LoadDynamicLibrary,
                            bool StaticVCRuntime, const char *VCRuntimePath,
                            Error &Err)
     : ES(ES), ObjLinkingLayer(ObjLinkingLayer),
-      LoadDynLibrary(std::move(LoadDynLibrary)),
+      LoadDynLibrary(std::move(LoadDynamicLibrary)),
       StaticVCRuntime(StaticVCRuntime),
       COFFHeaderStartSymbol(ES.intern("__ImageBase")) {
   ErrorAsOutParameter _(&Err);
diff --git a/llvm/lib/ExecutionEngine/Orc/COFFVCRuntimeSupport.cpp b/llvm/lib/ExecutionEngine/Orc/COFFVCRuntimeSupport.cpp
index d9316fab2de30..94f696fa20863 100644
--- a/llvm/lib/ExecutionEngine/Orc/COFFVCRuntimeSupport.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/COFFVCRuntimeSupport.cpp
@@ -160,7 +160,7 @@ COFFVCRuntimeBootstrapper::getMSVCToolchainPath() {
   if (!findVCToolChainViaCommandLine(*VFS, std::nullopt, std::nullopt,
                                      std::nullopt, VCToolChainPath, VSLayout) &&
       !findVCToolChainViaEnvironment(*VFS, VCToolChainPath, VSLayout) &&
-      !findVCToolChainViaSetupConfig(*VFS, VCToolChainPath, VSLayout) &&
+      !findVCToolChainViaSetupConfig(*VFS, {}, VCToolChainPath, VSLayout) &&
       !findVCToolChainViaRegistry(VCToolChainPath, VSLayout))
     return make_error<StringError>("Couldn't find msvc toolchain.",
                                    inconvertibleErrorCode());
diff --git a/llvm/lib/ExecutionEngine/Orc/Core.cpp b/llvm/lib/ExecutionEngine/Orc/Core.cpp
index f85719195532d..82fa4bed914e6 100644
--- a/llvm/lib/ExecutionEngine/Orc/Core.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/Core.cpp
@@ -965,7 +965,7 @@ Error JITDylib::resolve(MaterializationResponsibility &MR,
                  "Resolving symbol with materializer attached?");
           assert(SymI->second.getState() == SymbolState::Materializing &&
                  "Symbol should be materializing");
-          assert(SymI->second.getAddress() == 0 &&
+          assert(SymI->second.getAddress() == ExecutorAddr() &&
                  "Symbol has already been resolved");
 
           if (SymI->second.getFlags().hasError())
@@ -1000,7 +1000,7 @@ Error JITDylib::resolve(MaterializationResponsibility &MR,
 
           // Resolved symbols can not be weak: discard the weak flag.
           JITSymbolFlags ResolvedFlags = ResolvedSym.getFlags();
-          SymI->second.setAddress(ResolvedSym.getAddress());
+          SymI->second.setAddress(ExecutorAddr(ResolvedSym.getAddress()));
           SymI->second.setFlags(ResolvedFlags);
           SymI->second.setState(SymbolState::Resolved);
 
@@ -1441,7 +1441,7 @@ void JITDylib::dump(raw_ostream &OS) {
     for (auto &KV : Symbols) {
       OS << "    \"" << *KV.first << "\": ";
       if (auto Addr = KV.second.getAddress())
-        OS << format("0x%016" PRIx64, Addr);
+        OS << Addr;
       else
         OS << "<not resolved> ";
 
@@ -2667,7 +2667,7 @@ void ExecutionSession::OL_completeLookup(
             // whether it has a materializer attached, and if so prepare to run
             // it.
             if (SymI->second.hasMaterializerAttached()) {
-              assert(SymI->second.getAddress() == 0 &&
+              assert(SymI->second.getAddress() == ExecutorAddr() &&
                      "Symbol not resolved but already has address?");
               auto UMII = JD.UnmaterializedInfos.find(Name);
               assert(UMII != JD.UnmaterializedInfos.end() &&
diff --git a/llvm/lib/ExecutionEngine/Orc/EPCGenericRTDyldMemoryManager.cpp b/llvm/lib/ExecutionEngine/Orc/EPCGenericRTDyldMemoryManager.cpp
index ec82081937e22..fbe25d70c38a2 100644
--- a/llvm/lib/ExecutionEngine/Orc/EPCGenericRTDyldMemoryManager.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/EPCGenericRTDyldMemoryManager.cpp
@@ -235,7 +235,7 @@ bool EPCGenericRTDyldMemoryManager::finalizeMemory(std::string *ErrMsg) {
     for (unsigned I = 0; I != 3; ++I) {
       FR.Segments.push_back({});
       auto &Seg = FR.Segments.back();
-      Seg.AG = SegMemProts[I];
+      Seg.RAG = SegMemProts[I];
       Seg.Addr = RemoteAddrs[I]->Start;
       for (auto &SecAlloc : *SegSections[I]) {
         Seg.Size = alignTo(Seg.Size, SecAlloc.Align);
diff --git a/llvm/lib/ExecutionEngine/Orc/MemoryMapper.cpp b/llvm/lib/ExecutionEngine/Orc/MemoryMapper.cpp
index b457c7297bed6..ca4950077ffe9 100644
--- a/llvm/lib/ExecutionEngine/Orc/MemoryMapper.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/MemoryMapper.cpp
@@ -322,7 +322,8 @@ void SharedMemoryMapper::initialize(MemoryMapper::AllocInfo &AI,
     std::memset(Base + Segment.ContentSize, 0, Segment.ZeroFillSize);
 
     tpctypes::SharedMemorySegFinalizeRequest SegReq;
-    SegReq.AG = Segment.AG;
+    SegReq.RAG = {Segment.AG.getMemProt(), Segment.AG.getMemLifetimePolicy() ==
+                                               MemLifetimePolicy::Finalize};
     SegReq.Addr = AI.MappingBase + Segment.Offset;
     SegReq.Size = Segment.ContentSize + Segment.ZeroFillSize;
 
diff --git a/llvm/lib/ExecutionEngine/Orc/TargetProcess/ExecutorSharedMemoryMapperService.cpp b/llvm/lib/ExecutionEngine/Orc/TargetProcess/ExecutorSharedMemoryMapperService.cpp
index 147f915f61d6a..3f70dbf60437a 100644
--- a/llvm/lib/ExecutionEngine/Orc/TargetProcess/ExecutorSharedMemoryMapperService.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/TargetProcess/ExecutorSharedMemoryMapperService.cpp
@@ -132,11 +132,11 @@ Expected<ExecutorAddr> ExecutorSharedMemoryMapperService::initialize(
 #if defined(LLVM_ON_UNIX)
 
     int NativeProt = 0;
-    if ((Segment.AG.getMemProt() & MemProt::Read) == MemProt::Read)
+    if ((Segment.RAG.Prot & MemProt::Read) == MemProt::Read)
       NativeProt |= PROT_READ;
-    if ((Segment.AG.getMemProt() & MemProt::Write) == MemProt::Write)
+    if ((Segment.RAG.Prot & MemProt::Write) == MemProt::Write)
       NativeProt |= PROT_WRITE;
-    if ((Segment.AG.getMemProt() & MemProt::Exec) == MemProt::Exec)
+    if ((Segment.RAG.Prot & MemProt::Exec) == MemProt::Exec)
       NativeProt |= PROT_EXEC;
 
     if (mprotect(Segment.Addr.toPtr<void *>(), Segment.Size, NativeProt))
@@ -144,8 +144,7 @@ Expected<ExecutorAddr> ExecutorSharedMemoryMapperService::initialize(
 
 #elif defined(_WIN32)
 
-    DWORD NativeProt =
-        getWindowsProtectionFlags(Segment.AG.getMemProt());
+    DWORD NativeProt = getWindowsProtectionFlags(Segment.RAG.Prot);
 
     if (!VirtualProtect(Segment.Addr.toPtr<void *>(), Segment.Size, NativeProt,
                         &NativeProt))
@@ -153,7 +152,7 @@ Expected<ExecutorAddr> ExecutorSharedMemoryMapperService::initialize(
 
 #endif
 
-    if ((Segment.AG.getMemProt() & MemProt::Exec) == MemProt::Exec)
+    if ((Segment.RAG.Prot & MemProt::Exec) == MemProt::Exec)
       sys::Memory::InvalidateInstructionCache(Segment.Addr.toPtr<void *>(),
                                               Segment.Size);
   }
diff --git a/llvm/lib/ExecutionEngine/Orc/TargetProcess/SimpleExecutorMemoryManager.cpp b/llvm/lib/ExecutionEngine/Orc/TargetProcess/SimpleExecutorMemoryManager.cpp
index ce94bf1e039aa..4da031716e32a 100644
--- a/llvm/lib/ExecutionEngine/Orc/TargetProcess/SimpleExecutorMemoryManager.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/TargetProcess/SimpleExecutorMemoryManager.cpp
@@ -132,9 +132,9 @@ Error SimpleExecutorMemoryManager::finalize(tpctypes::FinalizeRequest &FR) {
     assert(Seg.Size <= std::numeric_limits<size_t>::max());
     if (auto EC = sys::Memory::protectMappedMemory(
             {Mem, static_cast<size_t>(Seg.Size)},
-            toSysMemoryProtectionFlags(Seg.AG.getMemProt())))
+            toSysMemoryProtectionFlags(Seg.RAG.Prot)))
       return BailOut(errorCodeToError(EC));
-    if ((Seg.AG.getMemProt() & MemProt::Exec) == MemProt::Exec)
+    if ((Seg.RAG.Prot & MemProt::Exec) == MemProt::Exec)
       sys::Memory::InvalidateInstructionCache(Mem, Seg.Size);
   }
 
diff --git a/llvm/lib/FileCheck/FileCheck.cpp b/llvm/lib/FileCheck/FileCheck.cpp
index ec963c2de45b2..ebebe0573b0b9 100644
--- a/llvm/lib/FileCheck/FileCheck.cpp
+++ b/llvm/lib/FileCheck/FileCheck.cpp
@@ -493,8 +493,7 @@ Expected<NumericVariable *> Pattern::parseNumericVariableDefinition(
 
   // Detect collisions between string and numeric variables when the latter
   // is created later than the former.
-  if (Context->DefinedVariableTable.find(Name) !=
-      Context->DefinedVariableTable.end())
+  if (Context->DefinedVariableTable.contains(Name))
     return ErrorDiagnostic::get(
         SM, Name, "string variable with name '" + Name + "' already exists");
 
@@ -1072,8 +1071,7 @@ bool Pattern::parsePattern(StringRef PatternStr, StringRef Prefix,
 
           // Detect collisions between string and numeric variables when the
           // former is created later than the latter.
-          if (Context->GlobalNumericVariableTable.find(Name) !=
-              Context->GlobalNumericVariableTable.end()) {
+          if (Context->GlobalNumericVariableTable.contains(Name)) {
             SM.PrintMessage(
                 SMLoc::getFromPointer(Name.data()), SourceMgr::DK_Error,
                 "numeric variable with name '" + Name + "' already exists");
@@ -2753,8 +2751,7 @@ Error FileCheckPatternContext::defineCmdlineVariables(
 
       // Detect collisions between string and numeric variables when the former
       // is created later than the latter.
-      if (GlobalNumericVariableTable.find(Name) !=
-          GlobalNumericVariableTable.end()) {
+      if (GlobalNumericVariableTable.contains(Name)) {
         Errs = joinErrors(std::move(Errs),
                           ErrorDiagnostic::get(SM, Name,
                                                "numeric variable with name '" +
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index 9381f42454f8b..fd4d2b5d51c86 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -813,14 +813,17 @@ void OpenMPIRBuilder::emitOffloadingEntry(Constant *Addr, StringRef Name,
 }
 
 OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::emitTargetKernel(
-    const LocationDescription &Loc, Value *&Return, Value *Ident,
-    Value *DeviceID, Value *NumTeams, Value *NumThreads, Value *HostPtr,
-    ArrayRef<Value *> KernelArgs) {
+    const LocationDescription &Loc, InsertPointTy AllocaIP, Value *&Return,
+    Value *Ident, Value *DeviceID, Value *NumTeams, Value *NumThreads,
+    Value *HostPtr, ArrayRef<Value *> KernelArgs) {
   if (!updateToLocation(Loc))
     return Loc.IP;
 
+  Builder.restoreIP(AllocaIP);
   auto *KernelArgsPtr =
       Builder.CreateAlloca(OpenMPIRBuilder::KernelArgs, nullptr, "kernel_args");
+  Builder.restoreIP(Loc.IP);
+
   for (unsigned I = 0, Size = KernelArgs.size(); I != Size; ++I) {
     llvm::Value *Arg =
         Builder.CreateStructGEP(OpenMPIRBuilder::KernelArgs, KernelArgsPtr, I);
diff --git a/llvm/lib/IR/AsmWriter.cpp b/llvm/lib/IR/AsmWriter.cpp
index d8e3d25ea15d1..d6e36927e73ef 100644
--- a/llvm/lib/IR/AsmWriter.cpp
+++ b/llvm/lib/IR/AsmWriter.cpp
@@ -1589,8 +1589,7 @@ static void WriteConstantInternal(raw_ostream &Out, const Constant *CV,
     Out << CE->getOpcodeName();
     WriteOptimizationInfo(Out, CE);
     if (CE->isCompare())
-      Out << ' ' << CmpInst::getPredicateName(
-                        static_cast<CmpInst::Predicate>(CE->getPredicate()));
+      Out << ' ' << static_cast<CmpInst::Predicate>(CE->getPredicate());
     Out << " (";
 
     std::optional<unsigned> InRangeOp;
@@ -4086,7 +4085,7 @@ void AssemblyWriter::printInstruction(const Instruction &I) {
 
   // Print out the compare instruction predicates
   if (const CmpInst *CI = dyn_cast<CmpInst>(&I))
-    Out << ' ' << CmpInst::getPredicateName(CI->getPredicate());
+    Out << ' ' << CI->getPredicate();
 
   // Print out the atomicrmw operation
   if (const AtomicRMWInst *RMWI = dyn_cast<AtomicRMWInst>(&I))
diff --git a/llvm/lib/IR/Attributes.cpp b/llvm/lib/IR/Attributes.cpp
index 3858f5f2484e1..57944c35836a2 100644
--- a/llvm/lib/IR/Attributes.cpp
+++ b/llvm/lib/IR/Attributes.cpp
@@ -266,16 +266,6 @@ bool Attribute::isExistingAttribute(StringRef Name) {
       .Default(false);
 }
 
-/// Returns true if this is a type legal for the 'nofpclass' attribute. This
-/// follows the same type rules as FPMathOperator.
-///
-/// TODO: Consider relaxing to any FP type struct fields.
-static bool isNoFPClassCompatibleType(Type *Ty) {
-  while (ArrayType *ArrTy = dyn_cast<ArrayType>(Ty))
-    Ty = ArrTy->getElementType();
-  return Ty->isFPOrFPVectorTy();
-}
-
 //===----------------------------------------------------------------------===//
 // Attribute Accessor Methods
 //===----------------------------------------------------------------------===//
@@ -1850,6 +1840,9 @@ AttrBuilder &AttrBuilder::addMemoryAttr(MemoryEffects ME) {
 }
 
 AttrBuilder &AttrBuilder::addNoFPClassAttr(FPClassTest Mask) {
+  if (Mask == fcNone)
+    return *this;
+
   return addRawIntAttr(Attribute::NoFPClass, Mask);
 }
 
@@ -1935,6 +1928,16 @@ bool AttrBuilder::operator==(const AttrBuilder &B) const {
 // AttributeFuncs Function Defintions
 //===----------------------------------------------------------------------===//
 
+/// Returns true if this is a type legal for the 'nofpclass' attribute. This
+/// follows the same type rules as FPMathOperator.
+///
+/// TODO: Consider relaxing to any FP type struct fields.
+bool AttributeFuncs::isNoFPClassCompatibleType(Type *Ty) {
+  while (ArrayType *ArrTy = dyn_cast<ArrayType>(Ty))
+    Ty = ArrTy->getElementType();
+  return Ty->isFPOrFPVectorTy();
+}
+
 /// Which attributes cannot be applied to a type.
 AttributeMask AttributeFuncs::typeIncompatible(Type *Ty,
                                                AttributeSafetyKind ASK) {
diff --git a/llvm/lib/IR/AutoUpgrade.cpp b/llvm/lib/IR/AutoUpgrade.cpp
index 9ca86c978ecc4..f68c492ee67b0 100644
--- a/llvm/lib/IR/AutoUpgrade.cpp
+++ b/llvm/lib/IR/AutoUpgrade.cpp
@@ -32,12 +32,18 @@
 #include "llvm/IR/Metadata.h"
 #include "llvm/IR/Module.h"
 #include "llvm/IR/Verifier.h"
+#include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/Regex.h"
 #include "llvm/TargetParser/Triple.h"
 #include <cstring>
+
 using namespace llvm;
 
+static cl::opt<bool>
+    DisableAutoUpgradeDebugInfo("disable-auto-upgrade-debug-info",
+                                cl::desc("Disable autoupgrade of debug info"));
+
 static void rename(GlobalValue *GV) { GV->setName(GV->getName() + ".old"); }
 
 // Upgrade the declarations of the SSE4.1 ptest intrinsics whose arguments have
@@ -4480,6 +4486,9 @@ Constant *llvm::UpgradeBitCastExpr(unsigned Opc, Constant *C, Type *DestTy) {
 /// Check the debug info version number, if it is out-dated, drop the debug
 /// info. Return true if module is modified.
 bool llvm::UpgradeDebugInfo(Module &M) {
+  if (DisableAutoUpgradeDebugInfo)
+    return false;
+
   unsigned Version = getDebugMetadataVersionFromModule(M);
   if (Version == DEBUG_METADATA_VERSION) {
     bool BrokenDebugInfo = false;
@@ -4988,7 +4997,6 @@ void llvm::UpgradeAttributes(AttrBuilder &B) {
 }
 
 void llvm::UpgradeOperandBundles(std::vector<OperandBundleDef> &Bundles) {
-
   // clang.arc.attachedcall bundles are now required to have an operand.
   // If they don't, it's okay to drop them entirely: when there is an operand,
   // the "attachedcall" is meaningful and required, but without an operand,
diff --git a/llvm/lib/IR/ConstantFold.cpp b/llvm/lib/IR/ConstantFold.cpp
index 553bc1e277e3b..59131a4264fc5 100644
--- a/llvm/lib/IR/ConstantFold.cpp
+++ b/llvm/lib/IR/ConstantFold.cpp
@@ -593,17 +593,6 @@ Constant *llvm::ConstantFoldSelectInstruction(Constant *Cond,
   if (isa<UndefValue>(V1) && NotPoison(V2)) return V2;
   if (isa<UndefValue>(V2) && NotPoison(V1)) return V1;
 
-  if (ConstantExpr *TrueVal = dyn_cast<ConstantExpr>(V1)) {
-    if (TrueVal->getOpcode() == Instruction::Select)
-      if (TrueVal->getOperand(0) == Cond)
-        return ConstantExpr::getSelect(Cond, TrueVal->getOperand(1), V2);
-  }
-  if (ConstantExpr *FalseVal = dyn_cast<ConstantExpr>(V2)) {
-    if (FalseVal->getOpcode() == Instruction::Select)
-      if (FalseVal->getOperand(0) == Cond)
-        return ConstantExpr::getSelect(Cond, V1, FalseVal->getOperand(2));
-  }
-
   return nullptr;
 }
 
diff --git a/llvm/lib/IR/Constants.cpp b/llvm/lib/IR/Constants.cpp
index 6e25e224b1dbc..ba68e6be05b52 100644
--- a/llvm/lib/IR/Constants.cpp
+++ b/llvm/lib/IR/Constants.cpp
@@ -547,8 +547,6 @@ void llvm::deleteConstant(Constant *C) {
       delete static_cast<CastConstantExpr *>(C);
     else if (isa<BinaryConstantExpr>(C))
       delete static_cast<BinaryConstantExpr *>(C);
-    else if (isa<SelectConstantExpr>(C))
-      delete static_cast<SelectConstantExpr *>(C);
     else if (isa<ExtractElementConstantExpr>(C))
       delete static_cast<ExtractElementConstantExpr *>(C);
     else if (isa<InsertElementConstantExpr>(C))
@@ -1488,8 +1486,6 @@ Constant *ConstantExpr::getWithOperands(ArrayRef<Constant *> Ops, Type *Ty,
   case Instruction::BitCast:
   case Instruction::AddrSpaceCast:
     return ConstantExpr::getCast(getOpcode(), Ops[0], Ty, OnlyIfReduced);
-  case Instruction::Select:
-    return ConstantExpr::getSelect(Ops[0], Ops[1], Ops[2], OnlyIfReducedTy);
   case Instruction::InsertElement:
     return ConstantExpr::getInsertElement(Ops[0], Ops[1], Ops[2],
                                           OnlyIfReducedTy);
@@ -2401,24 +2397,6 @@ Constant *ConstantExpr::getAlignOf(Type* Ty) {
                      Type::getInt64Ty(Ty->getContext()));
 }
 
-Constant *ConstantExpr::getOffsetOf(StructType* STy, unsigned FieldNo) {
-  return getOffsetOf(STy, ConstantInt::get(Type::getInt32Ty(STy->getContext()),
-                                           FieldNo));
-}
-
-Constant *ConstantExpr::getOffsetOf(Type* Ty, Constant *FieldNo) {
-  // offsetof is implemented as: (i64) gep (Ty*)null, 0, FieldNo
-  // Note that a non-inbounds gep is used, as null isn't within any object.
-  Constant *GEPIdx[] = {
-    ConstantInt::get(Type::getInt64Ty(Ty->getContext()), 0),
-    FieldNo
-  };
-  Constant *GEP = getGetElementPtr(
-      Ty, Constant::getNullValue(PointerType::getUnqual(Ty)), GEPIdx);
-  return getPtrToInt(GEP,
-                     Type::getInt64Ty(Ty->getContext()));
-}
-
 Constant *ConstantExpr::getCompare(unsigned short Predicate, Constant *C1,
                                    Constant *C2, bool OnlyIfReduced) {
   assert(C1->getType() == C2->getType() && "Op types should be identical!");
@@ -2441,29 +2419,13 @@ Constant *ConstantExpr::getCompare(unsigned short Predicate, Constant *C1,
   }
 }
 
-Constant *ConstantExpr::getSelect(Constant *C, Constant *V1, Constant *V2,
-                                  Type *OnlyIfReducedTy) {
-  assert(!SelectInst::areInvalidOperands(C, V1, V2)&&"Invalid select operands");
-
-  if (Constant *SC = ConstantFoldSelectInstruction(C, V1, V2))
-    return SC;        // Fold common cases
-
-  if (OnlyIfReducedTy == V1->getType())
-    return nullptr;
-
-  Constant *ArgVec[] = { C, V1, V2 };
-  ConstantExprKeyType Key(Instruction::Select, ArgVec);
-
-  LLVMContextImpl *pImpl = C->getContext().pImpl;
-  return pImpl->ExprConstants.getOrCreate(V1->getType(), Key);
-}
-
 Constant *ConstantExpr::getGetElementPtr(Type *Ty, Constant *C,
                                          ArrayRef<Value *> Idxs, bool InBounds,
                                          std::optional<unsigned> InRangeIndex,
                                          Type *OnlyIfReducedTy) {
   PointerType *OrigPtrTy = cast<PointerType>(C->getType()->getScalarType());
   assert(Ty && "Must specify element type");
+  assert(isSupportedGetElementPtr(Ty) && "Element type is unsupported!");
   assert(OrigPtrTy->isOpaqueOrPointeeTypeMatches(Ty));
 
   if (Constant *FC =
@@ -2690,11 +2652,6 @@ Constant *ConstantExpr::getXor(Constant *C1, Constant *C2) {
   return get(Instruction::Xor, C1, C2);
 }
 
-Constant *ConstantExpr::getUMin(Constant *C1, Constant *C2) {
-  Constant *Cmp = ConstantExpr::getICmp(CmpInst::ICMP_ULT, C1, C2);
-  return getSelect(Cmp, C1, C2);
-}
-
 Constant *ConstantExpr::getShl(Constant *C1, Constant *C2,
                                bool HasNUW, bool HasNSW) {
   unsigned Flags = (HasNUW ? OverflowingBinaryOperator::NoUnsignedWrap : 0) |
@@ -3443,8 +3400,6 @@ Instruction *ConstantExpr::getAsInstruction(Instruction *InsertBefore) const {
   case Instruction::AddrSpaceCast:
     return CastInst::Create((Instruction::CastOps)getOpcode(), Ops[0],
                             getType(), "", InsertBefore);
-  case Instruction::Select:
-    return SelectInst::Create(Ops[0], Ops[1], Ops[2], "", InsertBefore);
   case Instruction::InsertElement:
     return InsertElementInst::Create(Ops[0], Ops[1], Ops[2], "", InsertBefore);
   case Instruction::ExtractElement:
diff --git a/llvm/lib/IR/ConstantsContext.h b/llvm/lib/IR/ConstantsContext.h
index fbda443de7b2a..6023216a50708 100644
--- a/llvm/lib/IR/ConstantsContext.h
+++ b/llvm/lib/IR/ConstantsContext.h
@@ -90,32 +90,6 @@ class BinaryConstantExpr final : public ConstantExpr {
   }
 };
 
-/// SelectConstantExpr - This class is private to Constants.cpp, and is used
-/// behind the scenes to implement select constant exprs.
-class SelectConstantExpr final : public ConstantExpr {
-public:
-  SelectConstantExpr(Constant *C1, Constant *C2, Constant *C3)
-    : ConstantExpr(C2->getType(), Instruction::Select, &Op<0>(), 3) {
-    Op<0>() = C1;
-    Op<1>() = C2;
-    Op<2>() = C3;
-  }
-
-  // allocate space for exactly three operands
-  void *operator new(size_t S) { return User::operator new(S, 3); }
-  void operator delete(void *Ptr) { User::operator delete(Ptr); }
-
-  /// Transparently provide more efficient getOperand methods.
-  DECLARE_TRANSPARENT_OPERAND_ACCESSORS(Value);
-
-  static bool classof(const ConstantExpr *CE) {
-    return CE->getOpcode() == Instruction::Select;
-  }
-  static bool classof(const Value *V) {
-    return isa<ConstantExpr>(V) && classof(cast<ConstantExpr>(V));
-  }
-};
-
 /// ExtractElementConstantExpr - This class is private to
 /// Constants.cpp, and is used behind the scenes to implement
 /// extractelement constant exprs.
@@ -279,11 +253,6 @@ struct OperandTraits<BinaryConstantExpr>
     : public FixedNumOperandTraits<BinaryConstantExpr, 2> {};
 DEFINE_TRANSPARENT_OPERAND_ACCESSORS(BinaryConstantExpr, Value)
 
-template <>
-struct OperandTraits<SelectConstantExpr>
-    : public FixedNumOperandTraits<SelectConstantExpr, 3> {};
-DEFINE_TRANSPARENT_OPERAND_ACCESSORS(SelectConstantExpr, Value)
-
 template <>
 struct OperandTraits<ExtractElementConstantExpr>
     : public FixedNumOperandTraits<ExtractElementConstantExpr, 2> {};
@@ -523,8 +492,6 @@ struct ConstantExprKeyType {
         return new BinaryConstantExpr(Opcode, Ops[0], Ops[1],
                                       SubclassOptionalData);
       llvm_unreachable("Invalid ConstantExpr!");
-    case Instruction::Select:
-      return new SelectConstantExpr(Ops[0], Ops[1], Ops[2]);
     case Instruction::ExtractElement:
       return new ExtractElementConstantExpr(Ops[0], Ops[1]);
     case Instruction::InsertElement:
diff --git a/llvm/lib/IR/Core.cpp b/llvm/lib/IR/Core.cpp
index a0fade96daec4..283296b639ffc 100644
--- a/llvm/lib/IR/Core.cpp
+++ b/llvm/lib/IR/Core.cpp
@@ -811,8 +811,6 @@ LLVMTypeRef LLVMScalableVectorType(LLVMTypeRef ElementType,
 
 LLVMTypeRef LLVMGetElementType(LLVMTypeRef WrappedTy) {
   auto *Ty = unwrap(WrappedTy);
-  if (auto *PTy = dyn_cast<PointerType>(Ty))
-    return wrap(PTy->getNonOpaquePointerElementType());
   if (auto *ATy = dyn_cast<ArrayType>(Ty))
     return wrap(ATy->getElementType());
   return wrap(cast<VectorType>(Ty)->getElementType());
@@ -1801,14 +1799,6 @@ LLVMValueRef LLVMConstFPCast(LLVMValueRef ConstantVal, LLVMTypeRef ToType) {
                                       unwrap(ToType)));
 }
 
-LLVMValueRef LLVMConstSelect(LLVMValueRef ConstantCondition,
-                             LLVMValueRef ConstantIfTrue,
-                             LLVMValueRef ConstantIfFalse) {
-  return wrap(ConstantExpr::getSelect(unwrap<Constant>(ConstantCondition),
-                                      unwrap<Constant>(ConstantIfTrue),
-                                      unwrap<Constant>(ConstantIfFalse)));
-}
-
 LLVMValueRef LLVMConstExtractElement(LLVMValueRef VectorConstant,
                                      LLVMValueRef IndexConstant) {
   return wrap(ConstantExpr::getExtractElement(unwrap<Constant>(VectorConstant),
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index ad9540e51932a..479a51e1e0042 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -13,6 +13,7 @@
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "LLVMContextImpl.h"
 #include "MetadataImpl.h"
+#include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/BinaryFormat/Dwarf.h"
@@ -114,63 +115,122 @@ const DILocation *DILocation::getMergedLocation(const DILocation *LocA,
     return LocA;
 
   LLVMContext &C = LocA->getContext();
-  SmallDenseMap<std::pair<DILocalScope *, DILocation *>,
-                std::pair<unsigned, unsigned>, 4>
-      Locations;
-
-  DIScope *S = LocA->getScope();
-  DILocation *L = LocA->getInlinedAt();
-  unsigned Line = LocA->getLine();
-  unsigned Col = LocA->getColumn();
-
-  // Walk from the current source locaiton until the file scope;
-  // then, do the same for the inlined-at locations.
-  auto AdvanceToParentLoc = [&S, &L, &Line, &Col]() {
-    S = S->getScope();
-    if (!S && L) {
-      Line = L->getLine();
-      Col = L->getColumn();
-      S = L->getScope();
-      L = L->getInlinedAt();
-    }
-  };
 
-  while (S) {
-    if (auto *LS = dyn_cast<DILocalScope>(S))
-      Locations.try_emplace(std::make_pair(LS, L), std::make_pair(Line, Col));
-    AdvanceToParentLoc();
+  using LocVec = SmallVector<const DILocation *>;
+  LocVec ALocs;
+  LocVec BLocs;
+  SmallDenseMap<std::pair<const DISubprogram *, const DILocation *>, unsigned,
+                4>
+      ALookup;
+
+  // Walk through LocA and its inlined-at locations, populate them in ALocs and
+  // save the index for the subprogram and inlined-at pair, which we use to find
+  // a matching starting location in LocB's chain.
+  for (auto [L, I] = std::make_pair(LocA, 0U); L; L = L->getInlinedAt(), I++) {
+    ALocs.push_back(L);
+    auto Res = ALookup.try_emplace(
+        {L->getScope()->getSubprogram(), L->getInlinedAt()}, I);
+    assert(Res.second && "Multiple <SP, InlinedAt> pairs in a location chain?");
+    (void)Res;
   }
 
-  // Walk the source locations of LocB until a match with LocA is found.
-  S = LocB->getScope();
-  L = LocB->getInlinedAt();
-  Line = LocB->getLine();
-  Col = LocB->getColumn();
-  while (S) {
-    if (auto *LS = dyn_cast<DILocalScope>(S)) {
-      auto MatchLoc = Locations.find(std::make_pair(LS, L));
-      if (MatchLoc != Locations.end()) {
-        // If the lines match, keep the line, but set the column to '0'
-        // If the lines don't match, pick a "line 0" location but keep
-        // the current scope and inlined-at.
-        bool SameLine = Line == MatchLoc->second.first;
-        bool SameCol = Col == MatchLoc->second.second;
-        Line = SameLine ? Line : 0;
-        Col = SameLine && SameCol ? Col : 0;
-        break;
-      }
-    }
-    AdvanceToParentLoc();
+  LocVec::reverse_iterator ARIt = ALocs.rend();
+  LocVec::reverse_iterator BRIt = BLocs.rend();
+
+  // Populate BLocs and look for a matching starting location, the first
+  // location with the same subprogram and inlined-at location as in LocA's
+  // chain. Since the two locations have the same inlined-at location we do
+  // not need to look at those parts of the chains.
+  for (auto [L, I] = std::make_pair(LocB, 0U); L; L = L->getInlinedAt(), I++) {
+    BLocs.push_back(L);
+
+    if (ARIt != ALocs.rend())
+      // We have already found a matching starting location.
+      continue;
+
+    auto IT = ALookup.find({L->getScope()->getSubprogram(), L->getInlinedAt()});
+    if (IT == ALookup.end())
+      continue;
+
+    // The + 1 is to account for the &*rev_it = &(it - 1) relationship.
+    ARIt = LocVec::reverse_iterator(ALocs.begin() + IT->second + 1);
+    BRIt = LocVec::reverse_iterator(BLocs.begin() + I + 1);
+
+    // If we have found a matching starting location we do not need to add more
+    // locations to BLocs, since we will only look at location pairs preceding
+    // the matching starting location, and adding more elements to BLocs could
+    // invalidate the iterator that we initialized here.
+    break;
   }
 
-  if (!S) {
-    // If the two locations are irreconsilable, pick any scope,
-    // and return a "line 0" location.
-    Line = Col = 0;
-    S = LocA->getScope();
+  // Merge the two locations if possible, using the supplied
+  // inlined-at location for the created location.
+  auto MergeLocPair = [&C](const DILocation *L1, const DILocation *L2,
+                           DILocation *InlinedAt) -> DILocation * {
+    if (L1 == L2)
+      return DILocation::get(C, L1->getLine(), L1->getColumn(), L1->getScope(),
+                             InlinedAt);
+
+    // If the locations originate from different subprograms we can't produce
+    // a common location.
+    if (L1->getScope()->getSubprogram() != L2->getScope()->getSubprogram())
+      return nullptr;
+
+    // Return the nearest common scope inside a subprogram.
+    auto GetNearestCommonScope = [](DIScope *S1, DIScope *S2) -> DIScope * {
+      SmallPtrSet<DIScope *, 8> Scopes;
+      for (; S1; S1 = S1->getScope()) {
+        Scopes.insert(S1);
+        if (isa<DISubprogram>(S1))
+          break;
+      }
+
+      for (; S2; S2 = S2->getScope()) {
+        if (Scopes.count(S2))
+          return S2;
+        if (isa<DISubprogram>(S2))
+          break;
+      }
+
+      return nullptr;
+    };
+
+    auto Scope = GetNearestCommonScope(L1->getScope(), L2->getScope());
+    assert(Scope && "No common scope in the same subprogram?");
+
+    bool SameLine = L1->getLine() == L2->getLine();
+    bool SameCol = L1->getColumn() == L2->getColumn();
+    unsigned Line = SameLine ? L1->getLine() : 0;
+    unsigned Col = SameLine && SameCol ? L1->getColumn() : 0;
+
+    return DILocation::get(C, Line, Col, Scope, InlinedAt);
+  };
+
+  DILocation *Result = ARIt != ALocs.rend() ? (*ARIt)->getInlinedAt() : nullptr;
+
+  // If we have found a common starting location, walk up the inlined-at chains
+  // and try to produce common locations.
+  for (; ARIt != ALocs.rend() && BRIt != BLocs.rend(); ++ARIt, ++BRIt) {
+    DILocation *Tmp = MergeLocPair(*ARIt, *BRIt, Result);
+
+    if (!Tmp)
+      // We have walked up to a point in the chains where the two locations
+      // are irreconsilable. At this point Result contains the nearest common
+      // location in the inlined-at chains of LocA and LocB, so we break here.
+      break;
+
+    Result = Tmp;
   }
 
-  return DILocation::get(C, Line, Col, S, L);
+  if (Result)
+    return Result;
+
+  // We ended up with LocA and LocB as irreconsilable locations. Produce a
+  // location at 0:0 with one of the locations' scope. The function has
+  // historically picked A's scope, and a nullptr inlined-at location, so that
+  // behavior is mimicked here but I am not sure if this is always the correct
+  // way to handle this.
+  return DILocation::get(C, 0, 0, LocA->getScope(), nullptr);
 }
 
 std::optional<unsigned>
@@ -1607,7 +1667,7 @@ bool DIExpression::hasAllLocationOps(unsigned N) const {
     if (ExprOp.getOp() == dwarf::DW_OP_LLVM_arg)
       SeenOps.insert(ExprOp.getArg(0));
   for (uint64_t Idx = 0; Idx < N; ++Idx)
-    if (!is_contained(SeenOps, Idx))
+    if (!SeenOps.contains(Idx))
       return false;
   return true;
 }
diff --git a/llvm/lib/IR/IRBuilder.cpp b/llvm/lib/IR/IRBuilder.cpp
index f871205843a76..3ebd40a862191 100644
--- a/llvm/lib/IR/IRBuilder.cpp
+++ b/llvm/lib/IR/IRBuilder.cpp
@@ -107,6 +107,16 @@ Value *IRBuilderBase::CreateVScale(Constant *Scaling, const Twine &Name) {
              : CreateMul(CI, Scaling);
 }
 
+Value *IRBuilderBase::CreateElementCount(Type *DstType, ElementCount EC) {
+  Constant *MinEC = ConstantInt::get(DstType, EC.getKnownMinValue());
+  return EC.isScalable() ? CreateVScale(MinEC) : MinEC;
+}
+
+Value *IRBuilderBase::CreateTypeSize(Type *DstType, TypeSize Size) {
+  Constant *MinSize = ConstantInt::get(DstType, Size.getKnownMinValue());
+  return Size.isScalable() ? CreateVScale(MinSize) : MinSize;
+}
+
 Value *IRBuilderBase::CreateStepVector(Type *DstType, const Twine &Name) {
   Type *STy = DstType->getScalarType();
   if (isa<ScalableVectorType>(DstType)) {
diff --git a/llvm/lib/IR/Instructions.cpp b/llvm/lib/IR/Instructions.cpp
index da5b02b491a44..a1cc580998ee4 100644
--- a/llvm/lib/IR/Instructions.cpp
+++ b/llvm/lib/IR/Instructions.cpp
@@ -2728,6 +2728,98 @@ bool ShuffleVectorInst::isOneUseSingleSourceMask(int VF) const {
   return isOneUseSingleSourceMask(ShuffleMask, VF);
 }
 
+bool ShuffleVectorInst::isInterleave(unsigned Factor) {
+  FixedVectorType *OpTy = dyn_cast<FixedVectorType>(getOperand(0)->getType());
+  // shuffle_vector can only interleave fixed length vectors - for scalable
+  // vectors, see the @llvm.experimental.vector.interleave2 intrinsic
+  if (!OpTy)
+    return false;
+  unsigned OpNumElts = OpTy->getNumElements();
+
+  return isInterleaveMask(ShuffleMask, Factor, OpNumElts * 2);
+}
+
+bool ShuffleVectorInst::isInterleaveMask(
+    ArrayRef<int> Mask, unsigned Factor, unsigned NumInputElts,
+    SmallVectorImpl<unsigned> &StartIndexes) {
+  unsigned NumElts = Mask.size();
+  if (NumElts % Factor)
+    return false;
+
+  unsigned LaneLen = NumElts / Factor;
+  if (!isPowerOf2_32(LaneLen))
+    return false;
+
+  StartIndexes.resize(Factor);
+
+  // Check whether each element matches the general interleaved rule.
+  // Ignore undef elements, as long as the defined elements match the rule.
+  // Outer loop processes all factors (x, y, z in the above example)
+  unsigned I = 0, J;
+  for (; I < Factor; I++) {
+    unsigned SavedLaneValue;
+    unsigned SavedNoUndefs = 0;
+
+    // Inner loop processes consecutive accesses (x, x+1... in the example)
+    for (J = 0; J < LaneLen - 1; J++) {
+      // Lane computes x's position in the Mask
+      unsigned Lane = J * Factor + I;
+      unsigned NextLane = Lane + Factor;
+      int LaneValue = Mask[Lane];
+      int NextLaneValue = Mask[NextLane];
+
+      // If both are defined, values must be sequential
+      if (LaneValue >= 0 && NextLaneValue >= 0 &&
+          LaneValue + 1 != NextLaneValue)
+        break;
+
+      // If the next value is undef, save the current one as reference
+      if (LaneValue >= 0 && NextLaneValue < 0) {
+        SavedLaneValue = LaneValue;
+        SavedNoUndefs = 1;
+      }
+
+      // Undefs are allowed, but defined elements must still be consecutive:
+      // i.e.: x,..., undef,..., x + 2,..., undef,..., undef,..., x + 5, ....
+      // Verify this by storing the last non-undef followed by an undef
+      // Check that following non-undef masks are incremented with the
+      // corresponding distance.
+      if (SavedNoUndefs > 0 && LaneValue < 0) {
+        SavedNoUndefs++;
+        if (NextLaneValue >= 0 &&
+            SavedLaneValue + SavedNoUndefs != (unsigned)NextLaneValue)
+          break;
+      }
+    }
+
+    if (J < LaneLen - 1)
+      return false;
+
+    int StartMask = 0;
+    if (Mask[I] >= 0) {
+      // Check that the start of the I range (J=0) is greater than 0
+      StartMask = Mask[I];
+    } else if (Mask[(LaneLen - 1) * Factor + I] >= 0) {
+      // StartMask defined by the last value in lane
+      StartMask = Mask[(LaneLen - 1) * Factor + I] - J;
+    } else if (SavedNoUndefs > 0) {
+      // StartMask defined by some non-zero value in the j loop
+      StartMask = SavedLaneValue - (LaneLen - 1 - SavedNoUndefs);
+    }
+    // else StartMask remains set to 0, i.e. all elements are undefs
+
+    if (StartMask < 0)
+      return false;
+    // We must stay within the vectors; This case can happen with undefs.
+    if (StartMask + LaneLen > NumInputElts)
+      return false;
+
+    StartIndexes[I] = StartMask;
+  }
+
+  return true;
+}
+
 //===----------------------------------------------------------------------===//
 //                             InsertValueInst Class
 //===----------------------------------------------------------------------===//
@@ -4154,6 +4246,11 @@ StringRef CmpInst::getPredicateName(Predicate Pred) {
   }
 }
 
+raw_ostream &llvm::operator<<(raw_ostream &OS, CmpInst::Predicate Pred) {
+  OS << CmpInst::getPredicateName(Pred);
+  return OS;
+}
+
 ICmpInst::Predicate ICmpInst::getSignedPredicate(Predicate pred) {
   switch (pred) {
     default: llvm_unreachable("Unknown icmp predicate!");
diff --git a/llvm/lib/IR/IntrinsicInst.cpp b/llvm/lib/IR/IntrinsicInst.cpp
index 0bafa5f92b226..e120491fe3861 100644
--- a/llvm/lib/IR/IntrinsicInst.cpp
+++ b/llvm/lib/IR/IntrinsicInst.cpp
@@ -71,11 +71,9 @@ bool IntrinsicInst::mayLowerToFunctionCall(Intrinsic::ID IID) {
 /// intrinsics for variables.
 ///
 
-iterator_range<DbgVariableIntrinsic::location_op_iterator>
-DbgVariableIntrinsic::location_ops() const {
-  auto *MD = getRawLocation();
+iterator_range<location_op_iterator> RawLocationWrapper::location_ops() const {
+  Metadata *MD = getRawLocation();
   assert(MD && "First operand of DbgVariableIntrinsic should be non-null.");
-
   // If operand is ValueAsMetadata, return a range over just that operand.
   if (auto *VAM = dyn_cast<ValueAsMetadata>(MD)) {
     return {location_op_iterator(VAM), location_op_iterator(VAM + 1)};
@@ -89,8 +87,17 @@ DbgVariableIntrinsic::location_ops() const {
           location_op_iterator(static_cast<ValueAsMetadata *>(nullptr))};
 }
 
+iterator_range<location_op_iterator>
+DbgVariableIntrinsic::location_ops() const {
+  return getWrappedLocation().location_ops();
+}
+
 Value *DbgVariableIntrinsic::getVariableLocationOp(unsigned OpIdx) const {
-  auto *MD = getRawLocation();
+  return getWrappedLocation().getVariableLocationOp(OpIdx);
+}
+
+Value *RawLocationWrapper::getVariableLocationOp(unsigned OpIdx) const {
+  Metadata *MD = getRawLocation();
   assert(MD && "First operand of DbgVariableIntrinsic should be non-null.");
   if (auto *AL = dyn_cast<DIArgList>(MD))
     return AL->getArgs()[OpIdx]->getValue();
diff --git a/llvm/lib/IR/PassManager.cpp b/llvm/lib/IR/PassManager.cpp
index de46749a81e7a..92b729c44d215 100644
--- a/llvm/lib/IR/PassManager.cpp
+++ b/llvm/lib/IR/PassManager.cpp
@@ -122,13 +122,14 @@ PreservedAnalyses ModuleToFunctionPassAdaptor::run(Module &M,
       continue;
 
     PreservedAnalyses PassPA = Pass->run(F, FAM);
-    PI.runAfterPass(*Pass, F, PassPA);
 
     // We know that the function pass couldn't have invalidated any other
     // function's analyses (that's the contract of a function pass), so
     // directly handle the function analysis manager's invalidation here.
     FAM.invalidate(F, EagerlyInvalidate ? PreservedAnalyses::none() : PassPA);
 
+    PI.runAfterPass(*Pass, F, PassPA);
+
     // Then intersect the preserved set so that invalidation of module
     // analyses will eventually occur when the module pass completes.
     PA.intersect(std::move(PassPA));
diff --git a/llvm/lib/IR/SafepointIRVerifier.cpp b/llvm/lib/IR/SafepointIRVerifier.cpp
index 5d3fa28f7d0a2..ed99d05975c23 100644
--- a/llvm/lib/IR/SafepointIRVerifier.cpp
+++ b/llvm/lib/IR/SafepointIRVerifier.cpp
@@ -485,9 +485,7 @@ class GCPtrTracker {
                              InstructionVerifier &Verifier);
 
   /// Returns true for reachable and live blocks.
-  bool isMapped(const BasicBlock *BB) const {
-    return BlockMap.find(BB) != BlockMap.end();
-  }
+  bool isMapped(const BasicBlock *BB) const { return BlockMap.contains(BB); }
 
 private:
   /// Returns true if the instruction may be safely skipped during verification.
diff --git a/llvm/lib/IR/StructuralHash.cpp b/llvm/lib/IR/StructuralHash.cpp
index b6b9fe72cc355..077f8b6562f5e 100644
--- a/llvm/lib/IR/StructuralHash.cpp
+++ b/llvm/lib/IR/StructuralHash.cpp
@@ -1,13 +1,10 @@
-//===-- StructuralHash.cpp - IR Hash for expensive checks -------*- C++ -*-===//
+//===-- StructuralHash.cpp - IR Hashing -------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
-//
-
-#ifdef EXPENSIVE_CHECKS
 
 #include "llvm/IR/StructuralHash.h"
 #include "llvm/IR/Function.h"
@@ -15,19 +12,19 @@
 
 using namespace llvm;
 
-namespace details {
+namespace {
 
 // Basic hashing mechanism to detect structural change to the IR, used to verify
 // pass return status consistency with actual change. Loosely copied from
 // llvm/lib/Transforms/Utils/FunctionComparator.cpp
 
-class StructuralHash {
+class StructuralHashImpl {
   uint64_t Hash = 0x6acaa36bef8325c5ULL;
 
   void update(uint64_t V) { Hash = hashing::detail::hash_16_bytes(Hash, V); }
 
 public:
-  StructuralHash() = default;
+  StructuralHashImpl() = default;
 
   void update(const Function &F) {
     if (F.empty())
@@ -64,18 +61,16 @@ class StructuralHash {
   uint64_t getHash() const { return Hash; }
 };
 
-} // namespace details
+} // namespace
 
 uint64_t llvm::StructuralHash(const Function &F) {
-  ::details::StructuralHash H;
+  StructuralHashImpl H;
   H.update(F);
   return H.getHash();
 }
 
 uint64_t llvm::StructuralHash(const Module &M) {
-  ::details::StructuralHash H;
+  StructuralHashImpl H;
   H.update(M);
   return H.getHash();
 }
-
-#endif
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 83f8cfcc724fb..5cc90e0de938f 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3896,7 +3896,8 @@ void Verifier::visitRangeMetadata(Instruction &I, MDNode *Range, Type *Ty) {
     ConstantInt *High =
         mdconst::dyn_extract<ConstantInt>(Range->getOperand(2 * i + 1));
     Check(High, "The upper limit must be an integer!", High);
-    Check(High->getType() == Low->getType() && High->getType() == Ty,
+    Check(High->getType() == Low->getType() &&
+          High->getType() == Ty->getScalarType(),
           "Range types must match instruction type!", &I);
 
     APInt HighV = High->getValue();
@@ -6113,8 +6114,8 @@ void Verifier::visitConstrainedFPIntrinsic(ConstrainedFPIntrinsic &FPI) {
     Check(OperandTy->isVectorTy() == ResultTy->isVectorTy(),
           "Intrinsic first argument and result disagree on vector use", &FPI);
     if (OperandTy->isVectorTy()) {
-      Check(cast<FixedVectorType>(OperandTy)->getNumElements() ==
-                cast<FixedVectorType>(ResultTy)->getNumElements(),
+      Check(cast<VectorType>(OperandTy)->getElementCount() ==
+                cast<VectorType>(ResultTy)->getElementCount(),
             "Intrinsic first argument and result vector lengths must be equal",
             &FPI);
     }
diff --git a/llvm/lib/LTO/LTO.cpp b/llvm/lib/LTO/LTO.cpp
index 1b1cbe463ad7f..1f273a8e5025f 100644
--- a/llvm/lib/LTO/LTO.cpp
+++ b/llvm/lib/LTO/LTO.cpp
@@ -925,13 +925,16 @@ Error LTO::addThinLTO(BitcodeModule BM, ArrayRef<InputFile::Symbol> Syms,
     }
   }
 
+  uint64_t ModuleId = ThinLTO.ModuleMap.size();
   if (Error Err =
           BM.readSummary(ThinLTO.CombinedIndex, BM.getModuleIdentifier(),
-                         ThinLTO.ModuleMap.size(), [&](GlobalValue::GUID GUID) {
+                         ModuleId, [&](GlobalValue::GUID GUID) {
                            return ThinLTO.PrevailingModuleForGUID[GUID] ==
                                   BM.getModuleIdentifier();
                          }))
     return Err;
+  LLVM_DEBUG(dbgs() << "Module " << ModuleId << ": " << BM.getModuleIdentifier()
+                    << "\n");
 
   for (const InputFile::Symbol &Sym : Syms) {
     assert(ResI != ResE);
diff --git a/llvm/lib/LTO/LTOBackend.cpp b/llvm/lib/LTO/LTOBackend.cpp
index 4c41a382276a4..d574ae5737d55 100644
--- a/llvm/lib/LTO/LTOBackend.cpp
+++ b/llvm/lib/LTO/LTOBackend.cpp
@@ -260,7 +260,7 @@ static void runNewPMPasses(const Config &Conf, Module &Mod, TargetMachine *TM,
 
   PassInstrumentationCallbacks PIC;
   StandardInstrumentations SI(Mod.getContext(), Conf.DebugPassManager);
-  SI.registerCallbacks(PIC, &FAM);
+  SI.registerCallbacks(PIC, &MAM);
   PassBuilder PB(TM, Conf.PTO, PGOOpt, &PIC);
 
   RegisterPassPlugins(Conf.PassPlugins, PB);
diff --git a/llvm/lib/LTO/ThinLTOCodeGenerator.cpp b/llvm/lib/LTO/ThinLTOCodeGenerator.cpp
index 5b137a8f8cb34..144cc5a1e8971 100644
--- a/llvm/lib/LTO/ThinLTOCodeGenerator.cpp
+++ b/llvm/lib/LTO/ThinLTOCodeGenerator.cpp
@@ -150,7 +150,7 @@ static StringMap<lto::InputFile *>
 generateModuleMap(std::vector<std::unique_ptr<lto::InputFile>> &Modules) {
   StringMap<lto::InputFile *> ModuleMap;
   for (auto &M : Modules) {
-    assert(ModuleMap.find(M->getName()) == ModuleMap.end() &&
+    assert(!ModuleMap.contains(M->getName()) &&
            "Expect unique Buffer Identifier");
     ModuleMap[M->getName()] = M.get();
   }
@@ -245,7 +245,7 @@ static void optimizeModule(Module &TheModule, TargetMachine &TM,
 
   PassInstrumentationCallbacks PIC;
   StandardInstrumentations SI(TheModule.getContext(), DebugPassManager);
-  SI.registerCallbacks(PIC, &FAM);
+  SI.registerCallbacks(PIC, &MAM);
   PipelineTuningOptions PTO;
   PTO.LoopVectorization = true;
   PTO.SLPVectorization = true;
diff --git a/llvm/lib/Linker/IRMover.cpp b/llvm/lib/Linker/IRMover.cpp
index 721acb1ea02ca..97794ee7eb53e 100644
--- a/llvm/lib/Linker/IRMover.cpp
+++ b/llvm/lib/Linker/IRMover.cpp
@@ -409,6 +409,10 @@ class IRLinker {
   std::vector<GlobalValue *> Worklist;
   std::vector<std::pair<GlobalValue *, Value*>> RAUWWorklist;
 
+  /// Set of globals with eagerly copied metadata that may require remapping.
+  /// This remapping is performed after metadata linking.
+  DenseSet<GlobalObject *> UnmappedMetadata;
+
   void maybeAdd(GlobalValue *GV) {
     if (ValuesToLink.insert(GV).second)
       Worklist.push_back(GV);
@@ -750,8 +754,11 @@ GlobalValue *IRLinker::copyGlobalValueProto(const GlobalValue *SGV,
 
   if (auto *NewGO = dyn_cast<GlobalObject>(NewGV)) {
     // Metadata for global variables and function declarations is copied eagerly.
-    if (isa<GlobalVariable>(SGV) || SGV->isDeclaration())
+    if (isa<GlobalVariable>(SGV) || SGV->isDeclaration()) {
       NewGO->copyMetadata(cast<GlobalObject>(SGV), 0);
+      if (SGV->isDeclaration() && NewGO->hasMetadata())
+        UnmappedMetadata.insert(NewGO);
+    }
   }
 
   // Remove these copied constants in case this stays a declaration, since
@@ -1056,6 +1063,10 @@ Expected<Constant *> IRLinker::linkGlobalValueProto(GlobalValue *SGV,
   // as well.
   if (Function *F = dyn_cast<Function>(NewGV))
     if (auto Remangled = Intrinsic::remangleIntrinsicFunction(F)) {
+      // Note: remangleIntrinsicFunction does not copy metadata and as such
+      // F should not occur in the set of objects with unmapped metadata.
+      // If this assertion fails then remangleIntrinsicFunction needs updating.
+      assert(!UnmappedMetadata.count(F) && "intrinsic has unmapped metadata");
       NewGV->eraseFromParent();
       NewGV = *Remangled;
       NeedsRenaming = false;
@@ -1651,6 +1662,13 @@ Error IRLinker::run() {
   // are properly remapped.
   linkNamedMDNodes();
 
+  // Clean up any global objects with potentially unmapped metadata.
+  // Specifically declarations which did not become definitions.
+  for (GlobalObject *NGO : UnmappedMetadata) {
+    if (NGO->isDeclaration())
+      Mapper.remapGlobalObjectMetadata(*NGO);
+  }
+
   if (!IsPerformingImport && !SrcM->getModuleInlineAsm().empty()) {
     // Append the module inline asm string.
     DstM.appendModuleInlineAsm(adjustInlineAsm(SrcM->getModuleInlineAsm(),
diff --git a/llvm/lib/MC/MCParser/MasmParser.cpp b/llvm/lib/MC/MCParser/MasmParser.cpp
index 5d09630c3a32b..036d8218c8871 100644
--- a/llvm/lib/MC/MCParser/MasmParser.cpp
+++ b/llvm/lib/MC/MCParser/MasmParser.cpp
@@ -479,7 +479,7 @@ class MasmParser : public MCAsmParser {
   void addDirectiveHandler(StringRef Directive,
                            ExtensionDirectiveHandler Handler) override {
     ExtensionDirectiveMap[Directive] = Handler;
-    if (DirectiveKindMap.find(Directive) == DirectiveKindMap.end()) {
+    if (!DirectiveKindMap.contains(Directive)) {
       DirectiveKindMap[Directive] = DK_HANDLER_DIRECTIVE;
     }
   }
@@ -6260,9 +6260,9 @@ bool MasmParser::parseDirectiveIfdef(SMLoc DirectiveLoc, bool expect_defined) {
           parseEOL())
         return true;
 
-      if (BuiltinSymbolMap.find(Name.lower()) != BuiltinSymbolMap.end()) {
+      if (BuiltinSymbolMap.contains(Name.lower())) {
         is_defined = true;
-      } else if (Variables.find(Name.lower()) != Variables.end()) {
+      } else if (Variables.contains(Name.lower())) {
         is_defined = true;
       } else {
         MCSymbol *Sym = getContext().lookupSymbol(Name.lower());
@@ -6381,9 +6381,9 @@ bool MasmParser::parseDirectiveElseIfdef(SMLoc DirectiveLoc,
           parseEOL())
         return true;
 
-      if (BuiltinSymbolMap.find(Name.lower()) != BuiltinSymbolMap.end()) {
+      if (BuiltinSymbolMap.contains(Name.lower())) {
         is_defined = true;
-      } else if (Variables.find(Name.lower()) != Variables.end()) {
+      } else if (Variables.contains(Name.lower())) {
         is_defined = true;
       } else {
         MCSymbol *Sym = getContext().lookupSymbol(Name);
@@ -6551,9 +6551,9 @@ bool MasmParser::parseDirectiveErrorIfdef(SMLoc DirectiveLoc,
     if (check(parseIdentifier(Name), "expected identifier after '.errdef'"))
       return true;
 
-    if (BuiltinSymbolMap.find(Name.lower()) != BuiltinSymbolMap.end()) {
+    if (BuiltinSymbolMap.contains(Name.lower())) {
       IsDefined = true;
-    } else if (Variables.find(Name.lower()) != Variables.end()) {
+    } else if (Variables.contains(Name.lower())) {
       IsDefined = true;
     } else {
       MCSymbol *Sym = getContext().lookupSymbol(Name);
diff --git a/llvm/lib/MC/MCWin64EH.cpp b/llvm/lib/MC/MCWin64EH.cpp
index 1a55722133ccd..a2d61da722af8 100644
--- a/llvm/lib/MC/MCWin64EH.cpp
+++ b/llvm/lib/MC/MCWin64EH.cpp
@@ -1089,7 +1089,7 @@ static void ARM64ProcessEpilogs(WinEH::FrameInfo *info,
       FindMatchingEpilog(EpilogInstrs, AddedEpilogs, info);
     int PrologOffset;
     if (MatchingEpilog) {
-      assert(EpilogInfo.find(MatchingEpilog) != EpilogInfo.end() &&
+      assert(EpilogInfo.contains(MatchingEpilog) &&
              "Duplicate epilog not found");
       EpilogInfo[EpilogStart] = EpilogInfo.lookup(MatchingEpilog);
       // Clear the unwind codes in the EpilogMap, so that they don't get output
@@ -2369,7 +2369,7 @@ static void ARMEmitUnwindInfo(MCStreamer &streamer, WinEH::FrameInfo *info,
         FindMatchingEpilog(EpilogInstrs, AddedEpilogs, info);
     int PrologOffset;
     if (MatchingEpilog) {
-      assert(EpilogInfo.find(MatchingEpilog) != EpilogInfo.end() &&
+      assert(EpilogInfo.contains(MatchingEpilog) &&
              "Duplicate epilog not found");
       EpilogInfo[EpilogStart] = EpilogInfo.lookup(MatchingEpilog);
       // Clear the unwind codes in the EpilogMap, so that they don't get output
@@ -2449,7 +2449,7 @@ static void ARMEmitUnwindInfo(MCStreamer &streamer, WinEH::FrameInfo *info,
       else
         OffsetExpr = GetSubDivExpr(streamer, EpilogStart, info->Begin, 2);
 
-      assert(info->EpilogMap.find(EpilogStart) != info->EpilogMap.end());
+      assert(info->EpilogMap.contains(EpilogStart));
       unsigned Condition = info->EpilogMap[EpilogStart].Condition;
       assert(Condition <= 0xf);
 
diff --git a/llvm/lib/MC/WinCOFFObjectWriter.cpp b/llvm/lib/MC/WinCOFFObjectWriter.cpp
index c0b5e8bdc5039..4927ea6177fdb 100644
--- a/llvm/lib/MC/WinCOFFObjectWriter.cpp
+++ b/llvm/lib/MC/WinCOFFObjectWriter.cpp
@@ -716,7 +716,7 @@ void WinCOFFObjectWriter::recordRelocation(MCAssembler &Asm,
   MCSection *MCSec = Fragment->getParent();
 
   // Mark this symbol as requiring an entry in the symbol table.
-  assert(SectionMap.find(MCSec) != SectionMap.end() &&
+  assert(SectionMap.contains(MCSec) &&
          "Section must already have been defined in executePostLayoutBinding!");
 
   COFFSection *Sec = SectionMap[MCSec];
@@ -753,7 +753,7 @@ void WinCOFFObjectWriter::recordRelocation(MCAssembler &Asm,
   if (A.isTemporary()) {
     MCSection *TargetSection = &A.getSection();
     assert(
-        SectionMap.find(TargetSection) != SectionMap.end() &&
+        SectionMap.contains(TargetSection) &&
         "Section must already have been defined in executePostLayoutBinding!");
     COFFSection *Section = SectionMap[TargetSection];
     Reloc.Symb = Section->Symbol;
@@ -774,7 +774,7 @@ void WinCOFFObjectWriter::recordRelocation(MCAssembler &Asm,
     }
   } else {
     assert(
-        SymbolMap.find(&A) != SymbolMap.end() &&
+        SymbolMap.contains(&A) &&
         "Symbol must already have been defined in executePostLayoutBinding!");
     Reloc.Symb = SymbolMap[&A];
   }
@@ -1106,7 +1106,7 @@ uint64_t WinCOFFObjectWriter::writeObject(MCAssembler &Asm,
       }
 
       MCSection *TargetSection = &S->getSection();
-      assert(SectionMap.find(TargetSection) != SectionMap.end() &&
+      assert(SectionMap.contains(TargetSection) &&
              "Section must already have been defined in "
              "executePostLayoutBinding!");
       encodeULEB128(SectionMap[TargetSection]->Symbol->getIndex(), OS);
diff --git a/llvm/lib/MC/XCOFFObjectWriter.cpp b/llvm/lib/MC/XCOFFObjectWriter.cpp
index ab6acf085e7bc..c79bdeb2cac4c 100644
--- a/llvm/lib/MC/XCOFFObjectWriter.cpp
+++ b/llvm/lib/MC/XCOFFObjectWriter.cpp
@@ -488,8 +488,7 @@ void XCOFFObjectWriter::executePostLayoutBinding(MCAssembler &Asm,
                                                  const MCAsmLayout &Layout) {
   for (const auto &S : Asm) {
     const auto *MCSec = cast<const MCSectionXCOFF>(&S);
-    assert(SectionMap.find(MCSec) == SectionMap.end() &&
-           "Cannot add a section twice.");
+    assert(!SectionMap.contains(MCSec) && "Cannot add a section twice.");
 
     // If the name does not fit in the storage provided in the symbol table
     // entry, add it to the string table.
@@ -547,7 +546,7 @@ void XCOFFObjectWriter::executePostLayoutBinding(MCAssembler &Asm,
     if (!XSym->isExternal())
       continue;
 
-    assert(SectionMap.find(ContainingCsect) != SectionMap.end() &&
+    assert(SectionMap.contains(ContainingCsect) &&
            "Expected containing csect to exist in map");
     XCOFFSection *Csect = SectionMap[ContainingCsect];
     // Lookup the containing csect and add the symbol to it.
@@ -583,7 +582,7 @@ void XCOFFObjectWriter::recordRelocation(MCAssembler &Asm,
     // If we could not find the symbol directly in SymbolIndexMap, this symbol
     // could either be a temporary symbol or an undefined symbol. In this case,
     // we would need to have the relocation reference its csect instead.
-    return SymbolIndexMap.find(Sym) != SymbolIndexMap.end()
+    return SymbolIndexMap.contains(Sym)
                ? SymbolIndexMap[Sym]
                : SymbolIndexMap[ContainingCsect->getQualNameSymbol()];
   };
@@ -616,7 +615,7 @@ void XCOFFObjectWriter::recordRelocation(MCAssembler &Asm,
       TargetObjectWriter->getRelocTypeAndSignSize(Target, Fixup, IsPCRel);
 
   const MCSectionXCOFF *SymASec = getContainingCsect(cast<MCSymbolXCOFF>(SymA));
-  assert(SectionMap.find(SymASec) != SectionMap.end() &&
+  assert(SectionMap.contains(SymASec) &&
          "Expected containing csect to exist in map.");
 
   const uint32_t Index = getIndex(SymA, SymASec);
@@ -674,7 +673,7 @@ void XCOFFObjectWriter::recordRelocation(MCAssembler &Asm,
 
   XCOFFRelocation Reloc = {Index, FixupOffsetInCsect, SignAndSize, Type};
   MCSectionXCOFF *RelocationSec = cast<MCSectionXCOFF>(Fragment->getParent());
-  assert(SectionMap.find(RelocationSec) != SectionMap.end() &&
+  assert(SectionMap.contains(RelocationSec) &&
          "Expected containing csect to exist in map.");
   SectionMap[RelocationSec]->Relocations.push_back(Reloc);
 
@@ -686,7 +685,7 @@ void XCOFFObjectWriter::recordRelocation(MCAssembler &Asm,
     report_fatal_error("relocation for opposite term is not yet supported");
 
   const MCSectionXCOFF *SymBSec = getContainingCsect(cast<MCSymbolXCOFF>(SymB));
-  assert(SectionMap.find(SymBSec) != SectionMap.end() &&
+  assert(SectionMap.contains(SymBSec) &&
          "Expected containing csect to exist in map.");
   if (SymASec == SymBSec)
     report_fatal_error(
diff --git a/llvm/lib/MCA/InstrBuilder.cpp b/llvm/lib/MCA/InstrBuilder.cpp
index 7f3a4bb11b271..7ca55132117de 100644
--- a/llvm/lib/MCA/InstrBuilder.cpp
+++ b/llvm/lib/MCA/InstrBuilder.cpp
@@ -612,7 +612,7 @@ InstrBuilder::getOrCreateInstrDesc(const MCInst &MCI,
   unsigned CPUID = STI.getSchedModel().getProcessorID();
   SchedClassID = STI.resolveVariantSchedClass(SchedClassID, &MCI, &MCII, CPUID);
   auto VDKey = std::make_pair(&MCI, SchedClassID);
-  if (VariantDescriptors.find(VDKey) != VariantDescriptors.end())
+  if (VariantDescriptors.contains(VDKey))
     return *VariantDescriptors[VDKey];
 
   return createInstrDescImpl(MCI, IVec);
diff --git a/llvm/lib/Object/ELF.cpp b/llvm/lib/Object/ELF.cpp
index 81c9a097170d8..715cb5c7765e4 100644
--- a/llvm/lib/Object/ELF.cpp
+++ b/llvm/lib/Object/ELF.cpp
@@ -640,7 +640,26 @@ ELFFile<ELFT>::toMappedAddr(uint64_t VAddr, WarningHandler WarnHandler) const {
 
 template <class ELFT>
 Expected<std::vector<BBAddrMap>>
-ELFFile<ELFT>::decodeBBAddrMap(const Elf_Shdr &Sec) const {
+ELFFile<ELFT>::decodeBBAddrMap(const Elf_Shdr &Sec,
+                               const Elf_Shdr *RelaSec) const {
+  bool IsRelocatable = getHeader().e_type == ELF::ET_REL;
+
+  // This DenseMap maps the offset of each function (the location of the
+  // reference to the function in the SHT_LLVM_BB_ADDR_MAP section) to the
+  // addend (the location of the function in the text section).
+  llvm::DenseMap<uint64_t, uint64_t> FunctionOffsetTranslations;
+  if (IsRelocatable && RelaSec) {
+    assert(RelaSec &&
+           "Can't read a SHT_LLVM_BB_ADDR_MAP section in a relocatable "
+           "object file without providing a relocation section.");
+    Expected<Elf_Rela_Range> Relas = this->relas(*RelaSec);
+    if (!Relas)
+      return createError("unable to read relocations for section " +
+                         describe(*this, Sec) + ": " +
+                         toString(Relas.takeError()));
+    for (Elf_Rela Rela : *Relas)
+      FunctionOffsetTranslations[Rela.r_offset] = Rela.r_addend;
+  }
   Expected<ArrayRef<uint8_t>> ContentsOrErr = getSectionContents(Sec);
   if (!ContentsOrErr)
     return ContentsOrErr.takeError();
@@ -680,7 +699,20 @@ ELFFile<ELFT>::decodeBBAddrMap(const Elf_Shdr &Sec) const {
                            Twine(static_cast<int>(Version)));
       Data.getU8(Cur); // Feature byte
     }
+    uint64_t SectionOffset = Cur.tell();
     uintX_t Address = static_cast<uintX_t>(Data.getAddress(Cur));
+    if (!Cur)
+      return Cur.takeError();
+    if (IsRelocatable) {
+      assert(Address == 0);
+      auto FOTIterator = FunctionOffsetTranslations.find(SectionOffset);
+      if (FOTIterator == FunctionOffsetTranslations.end()) {
+        return createError("failed to get relocation data for offset: " +
+                           Twine::utohexstr(SectionOffset) + " in section " +
+                           describe(*this, Sec));
+      }
+      Address = FOTIterator->second;
+    }
     uint32_t NumBlocks = ReadULEB128AsUInt32();
     std::vector<BBAddrMap::BBEntry> BBEntries;
     uint32_t PrevBBEndOffset = 0;
@@ -706,6 +738,50 @@ ELFFile<ELFT>::decodeBBAddrMap(const Elf_Shdr &Sec) const {
   return FunctionEntries;
 }
 
+template <class ELFT>
+Expected<
+    MapVector<const typename ELFT::Shdr *, const typename ELFT::Shdr *>>
+ELFFile<ELFT>::getSectionAndRelocations(
+    std::function<Expected<bool>(const Elf_Shdr &)> IsMatch) const {
+  MapVector<const Elf_Shdr *, const Elf_Shdr *> SecToRelocMap;
+  Error Errors = Error::success();
+  for (const Elf_Shdr &Sec : cantFail(this->sections())) {
+    Expected<bool> DoesSectionMatch = IsMatch(Sec);
+    if (!DoesSectionMatch) {
+      Errors = joinErrors(std::move(Errors), DoesSectionMatch.takeError());
+      continue;
+    }
+    if (*DoesSectionMatch) {
+      if (SecToRelocMap.insert(std::make_pair(&Sec, (const Elf_Shdr *)nullptr))
+              .second)
+        continue;
+    }
+
+    if (Sec.sh_type != ELF::SHT_RELA && Sec.sh_type != ELF::SHT_REL)
+      continue;
+
+    Expected<const Elf_Shdr *> RelSecOrErr = this->getSection(Sec.sh_info);
+    if (!RelSecOrErr) {
+      Errors = joinErrors(std::move(Errors),
+                          createError(describe(*this, Sec) +
+                                      ": failed to get a relocated section: " +
+                                      toString(RelSecOrErr.takeError())));
+      continue;
+    }
+    const Elf_Shdr *ContentsSec = *RelSecOrErr;
+    Expected<bool> DoesRelTargetMatch = IsMatch(*ContentsSec);
+    if (!DoesRelTargetMatch) {
+      Errors = joinErrors(std::move(Errors), DoesRelTargetMatch.takeError());
+      continue;
+    }
+    if (*DoesRelTargetMatch)
+      SecToRelocMap[ContentsSec] = &Sec;
+  }
+  if(Errors)
+    return std::move(Errors);
+  return SecToRelocMap;
+}
+
 template class llvm::object::ELFFile<ELF32LE>;
 template class llvm::object::ELFFile<ELF32BE>;
 template class llvm::object::ELFFile<ELF64LE>;
diff --git a/llvm/lib/Object/ELFObjectFile.cpp b/llvm/lib/Object/ELFObjectFile.cpp
index 2cbfd8ac901b3..b9343d646db61 100644
--- a/llvm/lib/Object/ELFObjectFile.cpp
+++ b/llvm/lib/Object/ELFObjectFile.cpp
@@ -681,24 +681,39 @@ template <class ELFT>
 Expected<std::vector<BBAddrMap>> static readBBAddrMapImpl(
     const ELFFile<ELFT> &EF, std::optional<unsigned> TextSectionIndex) {
   using Elf_Shdr = typename ELFT::Shdr;
+  bool IsRelocatable = EF.getHeader().e_type == ELF::ET_REL;
   std::vector<BBAddrMap> BBAddrMaps;
+
   const auto &Sections = cantFail(EF.sections());
-  for (const Elf_Shdr &Sec : Sections) {
+  auto IsMatch = [&](const Elf_Shdr &Sec) -> Expected<bool> {
     if (Sec.sh_type != ELF::SHT_LLVM_BB_ADDR_MAP &&
         Sec.sh_type != ELF::SHT_LLVM_BB_ADDR_MAP_V0)
-      continue;
-    if (TextSectionIndex) {
-      Expected<const Elf_Shdr *> TextSecOrErr = EF.getSection(Sec.sh_link);
-      if (!TextSecOrErr)
-        return createError("unable to get the linked-to section for " +
-                           describe(EF, Sec) + ": " +
-                           toString(TextSecOrErr.takeError()));
-      if (*TextSectionIndex != std::distance(Sections.begin(), *TextSecOrErr))
-        continue;
-    }
-    Expected<std::vector<BBAddrMap>> BBAddrMapOrErr = EF.decodeBBAddrMap(Sec);
+      return false;
+    if (!TextSectionIndex)
+      return true;
+    Expected<const Elf_Shdr *> TextSecOrErr = EF.getSection(Sec.sh_link);
+    if (!TextSecOrErr)
+      return createError("unable to get the linked-to section for " +
+                         describe(EF, Sec) + ": " +
+                         toString(TextSecOrErr.takeError()));
+    if (*TextSectionIndex != std::distance(Sections.begin(), *TextSecOrErr))
+      return false;
+    return true;
+  };
+
+  Expected<MapVector<const Elf_Shdr *, const Elf_Shdr *>> SectionRelocMapOrErr =
+      EF.getSectionAndRelocations(IsMatch);
+  if (!SectionRelocMapOrErr)
+    return SectionRelocMapOrErr.takeError();
+
+  for (auto const &[Sec, RelocSec] : *SectionRelocMapOrErr) {
+    if (IsRelocatable && !RelocSec)
+      return createError("unable to get relocation section for " +
+                         describe(EF, *Sec));
+    Expected<std::vector<BBAddrMap>> BBAddrMapOrErr =
+        EF.decodeBBAddrMap(*Sec, RelocSec);
     if (!BBAddrMapOrErr)
-      return createError("unable to read " + describe(EF, Sec) + ": " +
+      return createError("unable to read " + describe(EF, *Sec) + ": " +
                          toString(BBAddrMapOrErr.takeError()));
     std::move(BBAddrMapOrErr->begin(), BBAddrMapOrErr->end(),
               std::back_inserter(BBAddrMaps));
diff --git a/llvm/lib/ObjectYAML/DWARFYAML.cpp b/llvm/lib/ObjectYAML/DWARFYAML.cpp
index 37116ada99015..2bddeed464135 100644
--- a/llvm/lib/ObjectYAML/DWARFYAML.cpp
+++ b/llvm/lib/ObjectYAML/DWARFYAML.cpp
@@ -59,22 +59,20 @@ Expected<DWARFYAML::Data::AbbrevTableInfo>
 DWARFYAML::Data::getAbbrevTableInfoByID(uint64_t ID) const {
   if (AbbrevTableInfoMap.empty()) {
     uint64_t AbbrevTableOffset = 0;
-    for (auto &AbbrevTable : enumerate(DebugAbbrev)) {
+    for (const auto &[Index, AbbrevTable] : enumerate(DebugAbbrev)) {
       // If the abbrev table's ID isn't specified, we use the index as its ID.
-      uint64_t AbbrevTableID =
-          AbbrevTable.value().ID.value_or(AbbrevTable.index());
+      uint64_t AbbrevTableID = AbbrevTable.ID.value_or(Index);
       auto It = AbbrevTableInfoMap.insert(
-          {AbbrevTableID, AbbrevTableInfo{/*Index=*/AbbrevTable.index(),
+          {AbbrevTableID, AbbrevTableInfo{/*Index=*/Index,
                                           /*Offset=*/AbbrevTableOffset}});
       if (!It.second)
         return createStringError(
             errc::invalid_argument,
             "the ID (%" PRIu64 ") of abbrev table with index %zu has been used "
             "by abbrev table with index %" PRIu64,
-            AbbrevTableID, AbbrevTable.index(), It.first->second.Index);
+            AbbrevTableID, Index, It.first->second.Index);
 
-      AbbrevTableOffset +=
-          getAbbrevTableContentByIndex(AbbrevTable.index()).size();
+      AbbrevTableOffset += getAbbrevTableContentByIndex(Index).size();
     }
   }
 
diff --git a/llvm/lib/ObjectYAML/MinidumpEmitter.cpp b/llvm/lib/ObjectYAML/MinidumpEmitter.cpp
index 1bda6f364b1bd..24b521a9925c7 100644
--- a/llvm/lib/ObjectYAML/MinidumpEmitter.cpp
+++ b/llvm/lib/ObjectYAML/MinidumpEmitter.cpp
@@ -236,8 +236,8 @@ bool yaml2minidump(MinidumpYAML::Object &Obj, raw_ostream &Out,
   Obj.Header.StreamDirectoryRVA = File.allocateArray(ArrayRef(StreamDirectory));
   Obj.Header.NumberOfStreams = StreamDirectory.size();
 
-  for (auto &Stream : enumerate(Obj.Streams))
-    StreamDirectory[Stream.index()] = layout(File, *Stream.value());
+  for (const auto &[Index, Stream] : enumerate(Obj.Streams))
+    StreamDirectory[Index] = layout(File, *Stream);
 
   File.writeTo(Out);
   return true;
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index e68d94ddbf8eb..89d2e6a4b2d1a 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -486,6 +486,28 @@ static std::optional<int> parseRepeatPassName(StringRef Name) {
   return Count;
 }
 
+static std::optional<std::pair<bool, bool>>
+parseFunctionPipelineName(StringRef Name) {
+  std::pair<bool, bool> Params;
+  if (!Name.consume_front("function"))
+    return std::nullopt;
+  if (Name.empty())
+    return Params;
+  if (!Name.consume_front("<") || !Name.consume_back(">"))
+    return std::nullopt;
+  while (!Name.empty()) {
+    auto [Front, Back] = Name.split(';');
+    Name = Back;
+    if (Front == "eager-inv")
+      Params.first = true;
+    else if (Front == "no-rerun")
+      Params.second = true;
+    else
+      return std::nullopt;
+  }
+  return Params;
+}
+
 static std::optional<int> parseDevirtPassName(StringRef Name) {
   if (!Name.consume_front("devirt<") || !Name.consume_back(">"))
     return std::nullopt;
@@ -657,6 +679,11 @@ Expected<bool> parseCoroSplitPassOptions(StringRef Params) {
   return parseSinglePassOption(Params, "reuse-storage", "CoroSplitPass");
 }
 
+Expected<bool> parsePostOrderFunctionAttrsPassOptions(StringRef Params) {
+  return parseSinglePassOption(Params, "skip-non-recursive",
+                               "PostOrderFunctionAttrs");
+}
+
 Expected<bool> parseEarlyCSEPassOptions(StringRef Params) {
   return parseSinglePassOption(Params, "memssa", "EarlyCSE");
 }
@@ -1004,12 +1031,14 @@ static bool isModulePassName(StringRef Name, CallbacksT &Callbacks) {
   if (startsWithDefaultPipelineAliasPrefix(Name))
     return DefaultAliasRegex.match(Name);
 
+  StringRef NameNoBracket = Name.take_until([](char C) { return C == '<'; });
+
   // Explicitly handle pass manager names.
   if (Name == "module")
     return true;
   if (Name == "cgscc")
     return true;
-  if (Name == "function" || Name == "function<eager-inv>")
+  if (NameNoBracket == "function")
     return true;
   if (Name == "coro-cond")
     return true;
@@ -1035,9 +1064,10 @@ static bool isModulePassName(StringRef Name, CallbacksT &Callbacks) {
 template <typename CallbacksT>
 static bool isCGSCCPassName(StringRef Name, CallbacksT &Callbacks) {
   // Explicitly handle pass manager names.
+  StringRef NameNoBracket = Name.take_until([](char C) { return C == '<'; });
   if (Name == "cgscc")
     return true;
-  if (Name == "function" || Name == "function<eager-inv>")
+  if (NameNoBracket == "function")
     return true;
 
   // Explicitly handle custom-parsed pass names.
@@ -1063,7 +1093,8 @@ static bool isCGSCCPassName(StringRef Name, CallbacksT &Callbacks) {
 template <typename CallbacksT>
 static bool isFunctionPassName(StringRef Name, CallbacksT &Callbacks) {
   // Explicitly handle pass manager names.
-  if (Name == "function" || Name == "function<eager-inv>")
+  StringRef NameNoBracket = Name.take_until([](char C) { return C == '<'; });
+  if (NameNoBracket == "function")
     return true;
   if (Name == "loop" || Name == "loop-mssa")
     return true;
@@ -1221,12 +1252,16 @@ Error PassBuilder::parseModulePass(ModulePassManager &MPM,
       MPM.addPass(createModuleToPostOrderCGSCCPassAdaptor(std::move(CGPM)));
       return Error::success();
     }
-    if (Name == "function" || Name == "function<eager-inv>") {
+    if (auto Params = parseFunctionPipelineName(Name)) {
+      if (Params->second)
+        return make_error<StringError>(
+            "cannot have a no-rerun module to function adaptor",
+            inconvertibleErrorCode());
       FunctionPassManager FPM;
       if (auto Err = parseFunctionPassPipeline(FPM, InnerPipeline))
         return Err;
-      MPM.addPass(createModuleToFunctionPassAdaptor(std::move(FPM),
-                                                    Name != "function"));
+      MPM.addPass(
+          createModuleToFunctionPassAdaptor(std::move(FPM), Params->first));
       return Error::success();
     }
     if (auto Count = parseRepeatPassName(Name)) {
@@ -1265,12 +1300,6 @@ Error PassBuilder::parseModulePass(ModulePassManager &MPM,
                               .Case("O3", OptimizationLevel::O3)
                               .Case("Os", OptimizationLevel::Os)
                               .Case("Oz", OptimizationLevel::Oz);
-    if (L == OptimizationLevel::O0 && Matches[1] != "thinlto" &&
-        Matches[1] != "lto") {
-      MPM.addPass(buildO0DefaultPipeline(L, Matches[1] == "thinlto-pre-link" ||
-                                                Matches[1] == "lto-pre-link"));
-      return Error::success();
-    }
 
     // This is consistent with old pass manager invoked via opt, but
     // inconsistent with clang. Clang doesn't enable loop vectorization
@@ -1395,13 +1424,13 @@ Error PassBuilder::parseCGSCCPass(CGSCCPassManager &CGPM,
       CGPM.addPass(std::move(NestedCGPM));
       return Error::success();
     }
-    if (Name == "function" || Name == "function<eager-inv>") {
+    if (auto Params = parseFunctionPipelineName(Name)) {
       FunctionPassManager FPM;
       if (auto Err = parseFunctionPassPipeline(FPM, InnerPipeline))
         return Err;
       // Add the nested pass manager with the appropriate adaptor.
-      CGPM.addPass(
-          createCGSCCToFunctionPassAdaptor(std::move(FPM), Name != "function"));
+      CGPM.addPass(createCGSCCToFunctionPassAdaptor(
+          std::move(FPM), Params->first, Params->second));
       return Error::success();
     }
     if (auto Count = parseRepeatPassName(Name)) {
diff --git a/llvm/lib/Passes/PassBuilderBindings.cpp b/llvm/lib/Passes/PassBuilderBindings.cpp
index a87c0e6dc0a37..2a49ae6e30e0a 100644
--- a/llvm/lib/Passes/PassBuilderBindings.cpp
+++ b/llvm/lib/Passes/PassBuilderBindings.cpp
@@ -66,7 +66,7 @@ LLVMErrorRef LLVMRunPasses(LLVMModuleRef M, const char *Passes,
   PB.crossRegisterProxies(LAM, FAM, CGAM, MAM);
 
   StandardInstrumentations SI(Mod->getContext(), Debug, VerifyEach);
-  SI.registerCallbacks(PIC, &FAM);
+  SI.registerCallbacks(PIC, &MAM);
   ModulePassManager MPM;
   if (VerifyEach) {
     MPM.addPass(VerifierPass());
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp b/llvm/lib/Passes/PassBuilderPipelines.cpp
index adb555ed21b9d..1d386139d9e6c 100644
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -831,8 +831,11 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
   if (AttributorRun & AttributorRunOption::CGSCC)
     MainCGPipeline.addPass(AttributorCGSCCPass());
 
-  // Now deduce any function attributes based in the current code.
-  MainCGPipeline.addPass(PostOrderFunctionAttrsPass());
+  // Deduce function attributes. We do another run of this after the function
+  // simplification pipeline, so this only needs to run when it could affect the
+  // function simplification pipeline, which is only the case with recursive
+  // functions.
+  MainCGPipeline.addPass(PostOrderFunctionAttrsPass(/*SkipNonRecursive*/ true));
 
   // When at O3 add argument promotion to the pass pipeline.
   // FIXME: It isn't at all clear why this should be limited to O3.
@@ -847,14 +850,25 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
   for (auto &C : CGSCCOptimizerLateEPCallbacks)
     C(MainCGPipeline, Level);
 
-  // Lastly, add the core function simplification pipeline nested inside the
+  // Add the core function simplification pipeline nested inside the
   // CGSCC walk.
   MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
       buildFunctionSimplificationPipeline(Level, Phase),
       PTO.EagerlyInvalidateAnalyses, /*NoRerun=*/true));
 
+  // Finally, deduce any function attributes based on the fully simplified
+  // function.
+  MainCGPipeline.addPass(PostOrderFunctionAttrsPass());
+
+  // Mark that the function is fully simplified and that it shouldn't be
+  // simplified again if we somehow revisit it due to CGSCC mutations unless
+  // it's been modified since.
+  MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
+      RequireAnalysisPass<ShouldNotRunFunctionPassesAnalysis, Function>()));
+
   MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0));
 
+  // Make sure we don't affect potential future NoRerun CGSCC adaptors.
   MIWP.addLateModulePass(createModuleToFunctionPassAdaptor(
       InvalidateAnalysisPass<ShouldNotRunFunctionPassesAnalysis>()));
 
@@ -954,14 +968,6 @@ PassBuilder::buildModuleSimplificationPipeline(OptimizationLevel Level,
   if (Level == OptimizationLevel::O3)
     EarlyFPM.addPass(CallSiteSplittingPass());
 
-  // In SamplePGO ThinLTO backend, we need instcombine before profile annotation
-  // to convert bitcast to direct calls so that they can be inlined during the
-  // profile annotation prepration step.
-  // More details about SamplePGO design can be found in:
-  // https://research.google.com/pubs/pub45290.html
-  // FIXME: revisit how SampleProfileLoad/Inliner/ICP is structured.
-  if (LoadSampleProfile)
-    EarlyFPM.addPass(InstCombinePass());
   MPM.addPass(createModuleToFunctionPassAdaptor(std::move(EarlyFPM),
                                                 PTO.EagerlyInvalidateAnalyses));
 
@@ -1006,9 +1012,12 @@ PassBuilder::buildModuleSimplificationPipeline(OptimizationLevel Level,
   // and prior to optimizing globals.
   // FIXME: This position in the pipeline hasn't been carefully considered in
   // years, it should be re-analyzed.
-  MPM.addPass(IPSCCPPass(IPSCCPOptions(/*AllowFuncSpec=*/
-                                       Level != OptimizationLevel::Os &&
-                                       Level != OptimizationLevel::Oz)));
+  MPM.addPass(IPSCCPPass(
+              IPSCCPOptions(/*AllowFuncSpec=*/
+                            Level != OptimizationLevel::Os &&
+                            Level != OptimizationLevel::Oz &&
+                            Phase != ThinOrFullLTOPhase::ThinLTOPreLink &&
+                            Phase != ThinOrFullLTOPhase::FullLTOPreLink)));
 
   // Attach metadata to indirect call sites indicating the set of functions
   // they may target at run-time. This should follow IPSCCP.
@@ -1058,6 +1067,14 @@ PassBuilder::buildModuleSimplificationPipeline(OptimizationLevel Level,
 
   MPM.addPass(CoroCleanupPass());
 
+  // Optimize globals now that functions are fully simplified.
+  MPM.addPass(GlobalOptPass());
+
+  // Remove dead code, except in the ThinLTO pre-link pipeline where we may want
+  // to keep available_externally functions.
+  if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+    MPM.addPass(GlobalDCEPass());
+
   if (EnableMemProfiler && Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
     MPM.addPass(createModuleToFunctionPassAdaptor(MemProfilerPass()));
     MPM.addPass(ModuleMemProfilerPass());
@@ -1213,10 +1230,6 @@ PassBuilder::buildModuleOptimizationPipeline(OptimizationLevel Level,
                            LTOPhase == ThinOrFullLTOPhase::FullLTOPreLink);
   ModulePassManager MPM;
 
-  // Optimize globals now that the module is fully simplified.
-  MPM.addPass(GlobalOptPass());
-  MPM.addPass(GlobalDCEPass());
-
   // Run partial inlining pass to partially inline functions that have
   // large bodies.
   if (RunPartialInlining)
@@ -1385,8 +1398,8 @@ PassBuilder::buildModuleOptimizationPipeline(OptimizationLevel Level,
 ModulePassManager
 PassBuilder::buildPerModuleDefaultPipeline(OptimizationLevel Level,
                                            bool LTOPreLink) {
-  assert(Level != OptimizationLevel::O0 &&
-         "Must request optimizations for the default pipeline!");
+  if (Level == OptimizationLevel::O0)
+    return buildO0DefaultPipeline(Level, LTOPreLink);
 
   ModulePassManager MPM;
 
@@ -1427,8 +1440,8 @@ PassBuilder::buildPerModuleDefaultPipeline(OptimizationLevel Level,
 
 ModulePassManager
 PassBuilder::buildThinLTOPreLinkDefaultPipeline(OptimizationLevel Level) {
-  assert(Level != OptimizationLevel::O0 &&
-         "Must request optimizations for the default pipeline!");
+  if (Level == OptimizationLevel::O0)
+    return buildO0DefaultPipeline(Level, /*LTOPreLink*/true);
 
   ModulePassManager MPM;
 
@@ -1461,9 +1474,6 @@ PassBuilder::buildThinLTOPreLinkDefaultPipeline(OptimizationLevel Level) {
   if (RunPartialInlining)
     MPM.addPass(PartialInlinerPass());
 
-  // Reduce the size of the IR as much as possible.
-  MPM.addPass(GlobalOptPass());
-
   if (PGOOpt && PGOOpt->PseudoProbeForProfiling &&
       PGOOpt->Action == PGOOptions::SampleUse)
     MPM.addPass(PseudoProbeUpdatePass());
@@ -1542,8 +1552,6 @@ ModulePassManager PassBuilder::buildThinLTODefaultPipeline(
 
 ModulePassManager
 PassBuilder::buildLTOPreLinkDefaultPipeline(OptimizationLevel Level) {
-  assert(Level != OptimizationLevel::O0 &&
-         "Must request optimizations for the default pipeline!");
   // FIXME: We should use a customized pre-link pipeline!
   return buildPerModuleDefaultPipeline(Level,
                                        /* LTOPreLink */ true);
@@ -1674,6 +1682,9 @@ PassBuilder::buildLTODefaultPipeline(OptimizationLevel Level,
   // keep one copy of each constant.
   MPM.addPass(ConstantMergePass());
 
+  // Remove unused arguments from functions.
+  MPM.addPass(DeadArgumentEliminationPass());
+
   // Reduce the code after globalopt and ipsccp.  Both can open up significant
   // simplification opportunities, and both can propagate functions through
   // function pointers.  When this happens, we often have to resolve varargs
@@ -1711,9 +1722,6 @@ PassBuilder::buildLTODefaultPipeline(OptimizationLevel Level,
   // transform it to pass arguments by value instead of by reference.
   MPM.addPass(createModuleToPostOrderCGSCCPassAdaptor(ArgumentPromotionPass()));
 
-  // Remove unused arguments from functions.
-  MPM.addPass(DeadArgumentEliminationPass());
-
   FunctionPassManager FPM;
   // The IPO Passes may leave cruft around. Clean up after them.
   FPM.addPass(InstCombinePass());
diff --git a/llvm/lib/Passes/PassRegistry.def b/llvm/lib/Passes/PassRegistry.def
index 36700574c1663..04d648580a8c5 100644
--- a/llvm/lib/Passes/PassRegistry.def
+++ b/llvm/lib/Passes/PassRegistry.def
@@ -184,7 +184,6 @@ CGSCC_ANALYSIS("pass-instrumentation", PassInstrumentationAnalysis(PIC))
 #endif
 CGSCC_PASS("argpromotion", ArgumentPromotionPass())
 CGSCC_PASS("invalidate<all>", InvalidateAllAnalysesPass())
-CGSCC_PASS("function-attrs", PostOrderFunctionAttrsPass())
 CGSCC_PASS("attributor-cgscc", AttributorCGSCCPass())
 CGSCC_PASS("openmp-opt-cgscc", OpenMPOptCGSCCPass())
 CGSCC_PASS("no-op-cgscc", NoOpCGSCCPass())
@@ -207,6 +206,13 @@ CGSCC_PASS_WITH_PARAMS("coro-split",
                        },
                        parseCoroSplitPassOptions,
                        "reuse-storage")
+CGSCC_PASS_WITH_PARAMS("function-attrs",
+                       "PostOrderFunctionAttrsPass",
+                       [](bool SkipNonRecursive) {
+                         return PostOrderFunctionAttrsPass(SkipNonRecursive);
+                       },
+                       parsePostOrderFunctionAttrsPassOptions,
+                       "skip-non-recursive")
 #undef CGSCC_PASS_WITH_PARAMS
 
 #ifndef FUNCTION_ANALYSIS
diff --git a/llvm/lib/Passes/StandardInstrumentations.cpp b/llvm/lib/Passes/StandardInstrumentations.cpp
index 065f85f4e11fb..f1ff58defd4ad 100644
--- a/llvm/lib/Passes/StandardInstrumentations.cpp
+++ b/llvm/lib/Passes/StandardInstrumentations.cpp
@@ -24,6 +24,7 @@
 #include "llvm/IR/PassInstrumentation.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/IR/PrintPasses.h"
+#include "llvm/IR/StructuralHash.h"
 #include "llvm/IR/Verifier.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/CrashRecoveryContext.h"
@@ -43,14 +44,14 @@
 
 using namespace llvm;
 
-cl::opt<bool> PreservedCFGCheckerInstrumentation::VerifyPreservedCFG(
-    "verify-cfg-preserved", cl::Hidden,
-#ifdef NDEBUG
-    cl::init(false)
+static cl::opt<bool> VerifyAnalysisInvalidation("verify-analysis-invalidation",
+                                                cl::Hidden,
+#ifdef EXPENSIVE_CHECKS
+                                                cl::init(true)
 #else
-    cl::init(true)
+                                                cl::init(false)
 #endif
-    );
+);
 
 // An option that supports the -print-changed option.  See
 // the description for -print-changed for an explanation of the use
@@ -1049,6 +1050,40 @@ struct PreservedCFGCheckerAnalysis
 
 AnalysisKey PreservedCFGCheckerAnalysis::Key;
 
+struct PreservedFunctionHashAnalysis
+    : public AnalysisInfoMixin<PreservedFunctionHashAnalysis> {
+  static AnalysisKey Key;
+
+  struct FunctionHash {
+    uint64_t Hash;
+  };
+
+  using Result = FunctionHash;
+
+  Result run(Function &F, FunctionAnalysisManager &FAM) {
+    return Result{StructuralHash(F)};
+  }
+};
+
+AnalysisKey PreservedFunctionHashAnalysis::Key;
+
+struct PreservedModuleHashAnalysis
+    : public AnalysisInfoMixin<PreservedModuleHashAnalysis> {
+  static AnalysisKey Key;
+
+  struct ModuleHash {
+    uint64_t Hash;
+  };
+
+  using Result = ModuleHash;
+
+  Result run(Module &F, ModuleAnalysisManager &FAM) {
+    return Result{StructuralHash(F)};
+  }
+};
+
+AnalysisKey PreservedModuleHashAnalysis::Key;
+
 bool PreservedCFGCheckerInstrumentation::CFG::invalidate(
     Function &F, const PreservedAnalyses &PA,
     FunctionAnalysisManager::Invalidator &) {
@@ -1057,39 +1092,52 @@ bool PreservedCFGCheckerInstrumentation::CFG::invalidate(
            PAC.preservedSet<CFGAnalyses>());
 }
 
-void PreservedCFGCheckerInstrumentation::registerCallbacks(
-    PassInstrumentationCallbacks &PIC, FunctionAnalysisManager &FAM) {
-  if (!VerifyPreservedCFG)
-    return;
-
-  FAM.registerPass([&] { return PreservedCFGCheckerAnalysis(); });
+static SmallVector<Function *, 1> GetFunctions(Any IR) {
+  SmallVector<Function *, 1> Functions;
 
-  auto checkCFG = [](StringRef Pass, StringRef FuncName, const CFG &GraphBefore,
-                     const CFG &GraphAfter) {
-    if (GraphAfter == GraphBefore)
-      return;
+  if (const auto **MaybeF = any_cast<const Function *>(&IR)) {
+    Functions.push_back(*const_cast<Function **>(MaybeF));
+  } else if (const auto **MaybeM = any_cast<const Module *>(&IR)) {
+    for (Function &F : **const_cast<Module **>(MaybeM))
+      Functions.push_back(&F);
+  }
+  return Functions;
+}
 
-    dbgs() << "Error: " << Pass
-           << " does not invalidate CFG analyses but CFG changes detected in "
-              "function @"
-           << FuncName << ":\n";
-    CFG::printDiff(dbgs(), GraphBefore, GraphAfter);
-    report_fatal_error(Twine("CFG unexpectedly changed by ", Pass));
-  };
+void PreservedCFGCheckerInstrumentation::registerCallbacks(
+    PassInstrumentationCallbacks &PIC, ModuleAnalysisManager &MAM) {
+  if (!VerifyAnalysisInvalidation)
+    return;
 
-  PIC.registerBeforeNonSkippedPassCallback(
-      [this, &FAM](StringRef P, Any IR) {
+  bool Registered = false;
+  PIC.registerBeforeNonSkippedPassCallback([this, &MAM, Registered](
+                                               StringRef P, Any IR) mutable {
 #ifdef LLVM_ENABLE_ABI_BREAKING_CHECKS
-        assert(&PassStack.emplace_back(P));
+    assert(&PassStack.emplace_back(P));
 #endif
-        (void)this;
-        const auto **F = any_cast<const Function *>(&IR);
-        if (!F)
-          return;
+    (void)this;
 
-        // Make sure a fresh CFG snapshot is available before the pass.
-        FAM.getResult<PreservedCFGCheckerAnalysis>(*const_cast<Function *>(*F));
-      });
+    auto &FAM = MAM.getResult<FunctionAnalysisManagerModuleProxy>(
+                       *const_cast<Module *>(unwrapModule(IR, /*Force=*/true)))
+                    .getManager();
+    if (!Registered) {
+      FAM.registerPass([&] { return PreservedCFGCheckerAnalysis(); });
+      FAM.registerPass([&] { return PreservedFunctionHashAnalysis(); });
+      MAM.registerPass([&] { return PreservedModuleHashAnalysis(); });
+      Registered = true;
+    }
+
+    for (Function *F : GetFunctions(IR)) {
+      // Make sure a fresh CFG snapshot is available before the pass.
+      FAM.getResult<PreservedCFGCheckerAnalysis>(*F);
+      FAM.getResult<PreservedFunctionHashAnalysis>(*F);
+    }
+
+    if (auto *MaybeM = any_cast<const Module *>(&IR)) {
+      Module &M = **const_cast<Module **>(MaybeM);
+      MAM.getResult<PreservedModuleHashAnalysis>(M);
+    }
+  });
 
   PIC.registerAfterPassInvalidatedCallback(
       [this](StringRef P, const PreservedAnalyses &PassPA) {
@@ -1100,27 +1148,60 @@ void PreservedCFGCheckerInstrumentation::registerCallbacks(
         (void)this;
       });
 
-  PIC.registerAfterPassCallback([this, &FAM,
-                                 checkCFG](StringRef P, Any IR,
-                                           const PreservedAnalyses &PassPA) {
+  PIC.registerAfterPassCallback([this, &MAM](StringRef P, Any IR,
+                                             const PreservedAnalyses &PassPA) {
 #ifdef LLVM_ENABLE_ABI_BREAKING_CHECKS
     assert(PassStack.pop_back_val() == P &&
            "Before and After callbacks must correspond");
 #endif
     (void)this;
 
-    const auto **F = any_cast<const Function *>(&IR);
-    if (!F)
-      return;
+    // We have to get the FAM via the MAM, rather than directly use a passed in
+    // FAM because if MAM has not cached the FAM, it won't invalidate function
+    // analyses in FAM.
+    auto &FAM = MAM.getResult<FunctionAnalysisManagerModuleProxy>(
+                       *const_cast<Module *>(unwrapModule(IR, /*Force=*/true)))
+                    .getManager();
+
+    for (Function *F : GetFunctions(IR)) {
+      if (auto *HashBefore =
+              FAM.getCachedResult<PreservedFunctionHashAnalysis>(*F)) {
+        if (HashBefore->Hash != StructuralHash(*F)) {
+          report_fatal_error(formatv(
+              "Function @{0} changed by {1} without invalidating analyses",
+              F->getName(), P));
+        }
+      }
 
-    if (!PassPA.allAnalysesInSetPreserved<CFGAnalyses>() &&
-        !PassPA.allAnalysesInSetPreserved<AllAnalysesOn<Function>>())
-      return;
+      auto CheckCFG = [](StringRef Pass, StringRef FuncName,
+                         const CFG &GraphBefore, const CFG &GraphAfter) {
+        if (GraphAfter == GraphBefore)
+          return;
 
-    if (auto *GraphBefore = FAM.getCachedResult<PreservedCFGCheckerAnalysis>(
-            *const_cast<Function *>(*F)))
-      checkCFG(P, (*F)->getName(), *GraphBefore,
-               CFG(*F, /* TrackBBLifetime */ false));
+        dbgs()
+            << "Error: " << Pass
+            << " does not invalidate CFG analyses but CFG changes detected in "
+               "function @"
+            << FuncName << ":\n";
+        CFG::printDiff(dbgs(), GraphBefore, GraphAfter);
+        report_fatal_error(Twine("CFG unexpectedly changed by ", Pass));
+      };
+
+      if (auto *GraphBefore =
+              FAM.getCachedResult<PreservedCFGCheckerAnalysis>(*F))
+        CheckCFG(P, F->getName(), *GraphBefore,
+                 CFG(F, /* TrackBBLifetime */ false));
+    }
+    if (auto *MaybeM = any_cast<const Module *>(&IR)) {
+      Module &M = **const_cast<Module **>(MaybeM);
+      if (auto *HashBefore =
+              MAM.getCachedResult<PreservedModuleHashAnalysis>(M)) {
+        if (HashBefore->Hash != StructuralHash(M)) {
+          report_fatal_error(formatv(
+              "Module changed by {0} without invalidating analyses", P));
+        }
+      }
+    }
   });
 }
 
@@ -2150,24 +2231,23 @@ void PrintCrashIRInstrumentation::registerCallbacks(
 }
 
 void StandardInstrumentations::registerCallbacks(
-    PassInstrumentationCallbacks &PIC, FunctionAnalysisManager *FAM) {
+    PassInstrumentationCallbacks &PIC, ModuleAnalysisManager *MAM) {
   PrintIR.registerCallbacks(PIC);
   PrintPass.registerCallbacks(PIC);
   TimePasses.registerCallbacks(PIC);
   OptNone.registerCallbacks(PIC);
   OptPassGate.registerCallbacks(PIC);
-  if (FAM)
-    PreservedCFGChecker.registerCallbacks(PIC, *FAM);
   PrintChangedIR.registerCallbacks(PIC);
   PseudoProbeVerification.registerCallbacks(PIC);
   if (VerifyEach)
     Verify.registerCallbacks(PIC);
   PrintChangedDiff.registerCallbacks(PIC);
   WebsiteChangeReporter.registerCallbacks(PIC);
-
   ChangeTester.registerCallbacks(PIC);
-
   PrintCrashIR.registerCallbacks(PIC);
+  if (MAM)
+    PreservedCFGChecker.registerCallbacks(PIC, *MAM);
+
   // TimeProfiling records the pass running time cost.
   // Its 'BeforePassCallback' can be appended at the tail of all the
   // BeforeCallbacks by calling `registerCallbacks` in the end.
diff --git a/llvm/lib/ProfileData/InstrProfWriter.cpp b/llvm/lib/ProfileData/InstrProfWriter.cpp
index af3c27ebac76d..8fd42b706e2d3 100644
--- a/llvm/lib/ProfileData/InstrProfWriter.cpp
+++ b/llvm/lib/ProfileData/InstrProfWriter.cpp
@@ -200,7 +200,7 @@ void InstrProfWriter::overlapRecord(NamedInstrProfRecord &&Other,
   auto Name = Other.Name;
   auto Hash = Other.Hash;
   Other.accumulateCounts(FuncLevelOverlap.Test);
-  if (FunctionData.find(Name) == FunctionData.end()) {
+  if (!FunctionData.contains(Name)) {
     Overlap.addOneUnique(FuncLevelOverlap.Test);
     return;
   }
diff --git a/llvm/lib/ProfileData/RawMemProfReader.cpp b/llvm/lib/ProfileData/RawMemProfReader.cpp
index 3081a04f26861..dd62a2f8a0f6c 100644
--- a/llvm/lib/ProfileData/RawMemProfReader.cpp
+++ b/llvm/lib/ProfileData/RawMemProfReader.cpp
@@ -149,20 +149,22 @@ Error report(Error E, const StringRef Context) {
 }
 
 bool isRuntimePath(const StringRef Path) {
-  return StringRef(llvm::sys::path::convert_to_slash(Path))
-      .contains("memprof/memprof_");
+  const StringRef Filename = llvm::sys::path::filename(Path);
+  // This list should be updated in case new files with additional interceptors
+  // are added to the memprof runtime.
+  return Filename.equals("memprof_malloc_linux.cpp") ||
+         Filename.equals("memprof_interceptors.cpp") ||
+         Filename.equals("memprof_new_delete.cpp");
 }
 
 std::string getBuildIdString(const SegmentEntry &Entry) {
-  constexpr size_t Size = sizeof(Entry.BuildId) / sizeof(uint8_t);
-  constexpr uint8_t Zeros[Size] = {0};
   // If the build id is unset print a helpful string instead of all zeros.
-  if (memcmp(Entry.BuildId, Zeros, Size) == 0)
+  if (Entry.BuildIdSize == 0)
     return "<None>";
 
   std::string Str;
   raw_string_ostream OS(Str);
-  for (size_t I = 0; I < Size; I++) {
+  for (size_t I = 0; I < Entry.BuildIdSize; I++) {
     OS << format_hex_no_prefix(Entry.BuildId[I], 2);
   }
   return OS.str();
diff --git a/llvm/lib/Support/CommandLine.cpp b/llvm/lib/Support/CommandLine.cpp
index 66632504d6fb6..d3efb8b67be5c 100644
--- a/llvm/lib/Support/CommandLine.cpp
+++ b/llvm/lib/Support/CommandLine.cpp
@@ -208,8 +208,7 @@ class CommandLineParser {
     bool HadErrors = false;
     if (O->hasArgStr()) {
       // If it's a DefaultOption, check to make sure it isn't already there.
-      if (O->isDefaultOption() &&
-          SC->OptionsMap.find(O->ArgStr) != SC->OptionsMap.end())
+      if (O->isDefaultOption() && SC->OptionsMap.contains(O->ArgStr))
         return;
 
       // Add argument to the argument map!
@@ -2758,7 +2757,7 @@ StringMap<Option *> &cl::getRegisteredOptions(SubCommand &Sub) {
   initCommonOptions();
   auto &Subs = GlobalParser->RegisteredSubCommands;
   (void)Subs;
-  assert(is_contained(Subs, &Sub));
+  assert(Subs.contains(&Sub));
   return Sub.OptionsMap;
 }
 
diff --git a/llvm/lib/Support/MemoryBuffer.cpp b/llvm/lib/Support/MemoryBuffer.cpp
index 0bb11725d2fcb..4cc4fe019b75b 100644
--- a/llvm/lib/Support/MemoryBuffer.cpp
+++ b/llvm/lib/Support/MemoryBuffer.cpp
@@ -11,6 +11,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "llvm/Support/MemoryBuffer.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/Config/config.h"
 #include "llvm/Support/Alignment.h"
@@ -22,6 +23,7 @@
 #include "llvm/Support/Process.h"
 #include "llvm/Support/Program.h"
 #include "llvm/Support/SmallVectorMemoryBuffer.h"
+#include <algorithm>
 #include <cassert>
 #include <cstring>
 #include <new>
@@ -132,10 +134,13 @@ MemoryBuffer::getMemBuffer(MemoryBufferRef Ref, bool RequiresNullTerminator) {
 
 static ErrorOr<std::unique_ptr<WritableMemoryBuffer>>
 getMemBufferCopyImpl(StringRef InputData, const Twine &BufferName) {
-  auto Buf = WritableMemoryBuffer::getNewUninitMemBuffer(InputData.size(), BufferName);
+  auto Buf =
+      WritableMemoryBuffer::getNewUninitMemBuffer(InputData.size(), BufferName);
   if (!Buf)
     return make_error_code(errc::not_enough_memory);
-  memcpy(Buf->getBufferStart(), InputData.data(), InputData.size());
+  // Calling memcpy with null src/dst is UB, and an empty StringRef is
+  // represented with {nullptr, 0}.
+  llvm::copy(InputData, Buf->getBufferStart());
   return std::move(Buf);
 }
 
diff --git a/llvm/lib/Support/RISCVISAInfo.cpp b/llvm/lib/Support/RISCVISAInfo.cpp
index d8702d7086189..2ef3ffa58079e 100644
--- a/llvm/lib/Support/RISCVISAInfo.cpp
+++ b/llvm/lib/Support/RISCVISAInfo.cpp
@@ -194,7 +194,6 @@ findDefaultVersion(StringRef ExtName) {
 void RISCVISAInfo::addExtension(StringRef ExtName, unsigned MajorVersion,
                                 unsigned MinorVersion) {
   RISCVExtensionInfo Ext;
-  Ext.ExtName = ExtName.str();
   Ext.MajorVersion = MajorVersion;
   Ext.MinorVersion = MinorVersion;
   Exts[ExtName.str()] = Ext;
@@ -622,10 +621,17 @@ RISCVISAInfo::parseArchString(StringRef Arch, bool EnableExperimentalExtension,
     break;
   case 'g':
     // g = imafd
+    if (Arch.size() > 5 && isdigit(Arch[5]))
+      return createStringError(errc::invalid_argument,
+                               "version not supported for 'g'");
     StdExts = StdExts.drop_front(4);
     break;
   }
 
+  if (Arch.back() == '_')
+    return createStringError(errc::invalid_argument,
+                             "extension name missing after separator '_'");
+
   // Skip rvxxx
   StringRef Exts = Arch.substr(5);
 
@@ -641,12 +647,10 @@ RISCVISAInfo::parseArchString(StringRef Arch, bool EnableExperimentalExtension,
   }
 
   unsigned Major, Minor, ConsumeLength;
-  if (auto E = getExtensionVersion(std::string(1, Baseline), Exts, Major, Minor,
-                                   ConsumeLength, EnableExperimentalExtension,
-                                   ExperimentalExtensionVersionCheck))
-    return std::move(E);
-
   if (Baseline == 'g') {
+    // Versions for g are disallowed, and this was checked for previously.
+    ConsumeLength = 0;
+
     // No matter which version is given to `g`, we always set imafd to default
     // version since the we don't have clear version scheme for that on
     // ISA spec.
@@ -655,9 +659,15 @@ RISCVISAInfo::parseArchString(StringRef Arch, bool EnableExperimentalExtension,
         ISAInfo->addExtension(Ext, Version->Major, Version->Minor);
       else
         llvm_unreachable("Default extension version not found?");
-  } else
+  } else {
     // Baseline is `i` or `e`
+    if (auto E = getExtensionVersion(std::string(1, Baseline), Exts, Major, Minor,
+                                     ConsumeLength, EnableExperimentalExtension,
+                                     ExperimentalExtensionVersionCheck))
+      return std::move(E);
+
     ISAInfo->addExtension(std::string(1, Baseline), Major, Minor);
+  }
 
   // Consume the base ISA version number and any '_' between rvxxx and the
   // first extension
@@ -804,6 +814,9 @@ RISCVISAInfo::parseArchString(StringRef Arch, bool EnableExperimentalExtension,
                                  Desc.str().c_str(), Name.str().c_str());
       }
 
+      if (IgnoreUnknown && !isSupportedExtension(Name))
+        continue;
+
       ISAInfo->addExtension(Name, Major, Minor);
       // Extension format is correct, keep parsing the extensions.
       // TODO: Save Type, Name, Major, Minor to avoid parsing them later.
@@ -839,35 +852,31 @@ Error RISCVISAInfo::checkDependency() {
         errc::invalid_argument,
         "standard user-level extension 'e' requires 'rv32'");
 
-  // It's illegal to specify the 'd' (double-precision floating point)
-  // extension without also specifying the 'f' (single precision
-  // floating-point) extension.
-  // TODO: This has been removed in later specs, which specify that D implies F
-  if (HasD && !HasF)
+  if (HasF && HasZfinx)
     return createStringError(errc::invalid_argument,
-                             "d requires f extension to also be specified");
+                             "'f' and 'zfinx' extensions are incompatible");
 
   if (HasZve32f && !HasF && !HasZfinx)
     return createStringError(
         errc::invalid_argument,
-        "zve32f requires f or zfinx extension to also be specified");
+        "'zve32f' requires 'f' or 'zfinx' extension to also be specified");
 
   if (HasZve64d && !HasD && !HasZdinx)
     return createStringError(
         errc::invalid_argument,
-        "zve64d requires d or zdinx extension to also be specified");
+        "'zve64d' requires 'd' or 'zdinx' extension to also be specified");
 
   if (Exts.count("zvfh") && !Exts.count("zfh") && !Exts.count("zfhmin") &&
       !Exts.count("zhinx") && !Exts.count("zhinxmin"))
     return createStringError(
         errc::invalid_argument,
-        "zvfh requires zfh, zfhmin, zhinx or zhinxmin extension to also be "
-        "specified");
+        "'zvfh' requires 'zfh', 'zfhmin', 'zhinx' or 'zhinxmin' extension to "
+        "also be specified");
 
   if (HasZvl && !HasVector)
     return createStringError(
         errc::invalid_argument,
-        "zvl*b requires v or zve* extension to also be specified");
+        "'zvl*b' requires 'v' or 'zve*' extension to also be specified");
 
   // Additional dependency checks.
   // TODO: The 'q' extension requires rv64.
@@ -876,6 +885,7 @@ Error RISCVISAInfo::checkDependency() {
   return Error::success();
 }
 
+static const char *ImpliedExtsD[] = {"f"};
 static const char *ImpliedExtsV[] = {"zvl128b", "zve64d", "f", "d"};
 static const char *ImpliedExtsZfhmin[] = {"f"};
 static const char *ImpliedExtsZfh[] = {"f"};
@@ -920,6 +930,7 @@ struct ImpliedExtsEntry {
 
 // Note: The table needs to be sorted by name.
 static constexpr ImpliedExtsEntry ImpliedExts[] = {
+    {{"d"}, {ImpliedExtsD}},
     {{"v"}, {ImpliedExtsV}},
     {{"xtheadvdot"}, {ImpliedExtsXTHeadVdot}},
     {{"zcb"}, {ImpliedExtsZcb}},
diff --git a/llvm/lib/Support/SpecialCaseList.cpp b/llvm/lib/Support/SpecialCaseList.cpp
index 0fb65accbf1db..46503477cdda4 100644
--- a/llvm/lib/Support/SpecialCaseList.cpp
+++ b/llvm/lib/Support/SpecialCaseList.cpp
@@ -175,7 +175,7 @@ bool SpecialCaseList::parse(const MemoryBuffer *MB,
     StringRef Category = SplitRegexp.second;
 
     // Create this section if it has not been seen before.
-    if (SectionsMap.find(Section) == SectionsMap.end()) {
+    if (!SectionsMap.contains(Section)) {
       std::unique_ptr<Matcher> M = std::make_unique<Matcher>();
       std::string REError;
       if (!M->insert(std::string(Section), LineNo, REError)) {
diff --git a/llvm/lib/TableGen/Record.cpp b/llvm/lib/TableGen/Record.cpp
index 654857fc19fe2..a3f09bd998fa3 100644
--- a/llvm/lib/TableGen/Record.cpp
+++ b/llvm/lib/TableGen/Record.cpp
@@ -778,6 +778,14 @@ void UnOpInit::Profile(FoldingSetNodeID &ID) const {
 Init *UnOpInit::Fold(Record *CurRec, bool IsFinal) const {
   RecordKeeper &RK = getRecordKeeper();
   switch (getOpcode()) {
+  case TOLOWER:
+    if (StringInit *LHSs = dyn_cast<StringInit>(LHS))
+      return StringInit::get(RK, LHSs->getValue().lower());
+    break;
+  case TOUPPER:
+    if (StringInit *LHSs = dyn_cast<StringInit>(LHS))
+      return StringInit::get(RK, LHSs->getValue().upper());
+    break;
   case CAST:
     if (isa<StringRecTy>(getType())) {
       if (StringInit *LHSs = dyn_cast<StringInit>(LHS))
@@ -792,37 +800,40 @@ Init *UnOpInit::Fold(Record *CurRec, bool IsFinal) const {
 
     } else if (isa<RecordRecTy>(getType())) {
       if (StringInit *Name = dyn_cast<StringInit>(LHS)) {
-        if (!CurRec && !IsFinal)
-          break;
-        assert(CurRec && "NULL pointer");
-        Record *D;
-
-        // Self-references are allowed, but their resolution is delayed until
-        // the final resolve to ensure that we get the correct type for them.
-        auto *Anonymous = dyn_cast<AnonymousNameInit>(CurRec->getNameInit());
-        if (Name == CurRec->getNameInit() ||
-            (Anonymous && Name == Anonymous->getNameInit())) {
-          if (!IsFinal)
-            break;
-          D = CurRec;
-        } else {
-          D = CurRec->getRecords().getDef(Name->getValue());
-          if (!D) {
-            if (IsFinal)
-              PrintFatalError(CurRec->getLoc(),
-                              Twine("Undefined reference to record: '") +
-                              Name->getValue() + "'\n");
-            break;
+        Record *D = RK.getDef(Name->getValue());
+        if (!D && CurRec) {
+          // Self-references are allowed, but their resolution is delayed until
+          // the final resolve to ensure that we get the correct type for them.
+          auto *Anonymous = dyn_cast<AnonymousNameInit>(CurRec->getNameInit());
+          if (Name == CurRec->getNameInit() ||
+              (Anonymous && Name == Anonymous->getNameInit())) {
+            if (!IsFinal)
+              break;
+            D = CurRec;
           }
         }
 
+        auto PrintFatalErrorHelper = [CurRec](const Twine &T) {
+          if (CurRec)
+            PrintFatalError(CurRec->getLoc(), T);
+          else
+            PrintFatalError(T);
+        };
+
+        if (!D) {
+          if (IsFinal) {
+            PrintFatalErrorHelper(Twine("Undefined reference to record: '") +
+                                  Name->getValue() + "'\n");
+          }
+          break;
+        }
+
         DefInit *DI = DefInit::get(D);
         if (!DI->getType()->typeIsA(getType())) {
-          PrintFatalError(CurRec->getLoc(),
-                          Twine("Expected type '") +
-                          getType()->getAsString() + "', got '" +
-                          DI->getType()->getAsString() + "' in: " +
-                          getAsString() + "\n");
+          PrintFatalErrorHelper(Twine("Expected type '") +
+                                getType()->getAsString() + "', got '" +
+                                DI->getType()->getAsString() + "' in: " +
+                                getAsString() + "\n");
         }
         return DI;
       }
@@ -927,6 +938,12 @@ std::string UnOpInit::getAsString() const {
   case EMPTY: Result = "!empty"; break;
   case GETDAGOP: Result = "!getdagop"; break;
   case LOG2 : Result = "!logtwo"; break;
+  case TOLOWER:
+    Result = "!tolower";
+    break;
+  case TOUPPER:
+    Result = "!toupper";
+    break;
   }
   return Result + "(" + LHS->getAsString() + ")";
 }
@@ -1776,7 +1793,7 @@ Init *ExistsOpInit::Fold(Record *CurRec, bool IsFinal) const {
     // Look up all defined records to see if we can find one.
     Record *D = CheckType->getRecordKeeper().getDef(Name->getValue());
     if (D) {
-      // Check if types are compatiable.
+      // Check if types are compatible.
       return IntInit::get(getRecordKeeper(),
                           DefInit::get(D)->getType()->typeIsA(CheckType));
     }
@@ -1791,7 +1808,7 @@ Init *ExistsOpInit::Fold(Record *CurRec, bool IsFinal) const {
           return const_cast<ExistsOpInit *>(this);
 
         // No doubt that there exists a record, so we should check if types are
-        // compatiable.
+        // compatible.
         return IntInit::get(getRecordKeeper(),
                             CurRec->getType()->typeIsA(CheckType));
       }
@@ -2259,7 +2276,7 @@ Init *CondOpInit::Fold(Record *CurRec) const {
   }
 
   PrintFatalError(CurRec->getLoc(),
-                  CurRec->getName() +
+                  CurRec->getNameInitAsString() +
                   " does not have any true condition in:" +
                   this->getAsString());
   return nullptr;
diff --git a/llvm/lib/TableGen/TGLexer.cpp b/llvm/lib/TableGen/TGLexer.cpp
index f2148b40a1b57..ba63d0b671ce6 100644
--- a/llvm/lib/TableGen/TGLexer.cpp
+++ b/llvm/lib/TableGen/TGLexer.cpp
@@ -549,50 +549,52 @@ tgtok::TokKind TGLexer::LexExclaim() {
 
   // Check to see which operator this is.
   tgtok::TokKind Kind =
-    StringSwitch<tgtok::TokKind>(StringRef(Start, CurPtr - Start))
-    .Case("eq", tgtok::XEq)
-    .Case("ne", tgtok::XNe)
-    .Case("le", tgtok::XLe)
-    .Case("lt", tgtok::XLt)
-    .Case("ge", tgtok::XGe)
-    .Case("gt", tgtok::XGt)
-    .Case("if", tgtok::XIf)
-    .Case("cond", tgtok::XCond)
-    .Case("isa", tgtok::XIsA)
-    .Case("head", tgtok::XHead)
-    .Case("tail", tgtok::XTail)
-    .Case("size", tgtok::XSize)
-    .Case("con", tgtok::XConcat)
-    .Case("dag", tgtok::XDag)
-    .Case("add", tgtok::XADD)
-    .Case("sub", tgtok::XSUB)
-    .Case("mul", tgtok::XMUL)
-    .Case("div", tgtok::XDIV)
-    .Case("not", tgtok::XNOT)
-    .Case("logtwo", tgtok::XLOG2)
-    .Case("and", tgtok::XAND)
-    .Case("or", tgtok::XOR)
-    .Case("xor", tgtok::XXOR)
-    .Case("shl", tgtok::XSHL)
-    .Case("sra", tgtok::XSRA)
-    .Case("srl", tgtok::XSRL)
-    .Case("cast", tgtok::XCast)
-    .Case("empty", tgtok::XEmpty)
-    .Case("subst", tgtok::XSubst)
-    .Case("foldl", tgtok::XFoldl)
-    .Case("foreach", tgtok::XForEach)
-    .Case("filter", tgtok::XFilter)
-    .Case("listconcat", tgtok::XListConcat)
-    .Case("listsplat", tgtok::XListSplat)
-    .Case("listremove", tgtok::XListRemove)
-    .Case("strconcat", tgtok::XStrConcat)
-    .Case("interleave", tgtok::XInterleave)
-    .Case("substr", tgtok::XSubstr)
-    .Case("find", tgtok::XFind)
-    .Cases("setdagop", "setop", tgtok::XSetDagOp) // !setop is deprecated.
-    .Cases("getdagop", "getop", tgtok::XGetDagOp) // !getop is deprecated.
-    .Case("exists", tgtok::XExists)
-    .Default(tgtok::Error);
+      StringSwitch<tgtok::TokKind>(StringRef(Start, CurPtr - Start))
+          .Case("eq", tgtok::XEq)
+          .Case("ne", tgtok::XNe)
+          .Case("le", tgtok::XLe)
+          .Case("lt", tgtok::XLt)
+          .Case("ge", tgtok::XGe)
+          .Case("gt", tgtok::XGt)
+          .Case("if", tgtok::XIf)
+          .Case("cond", tgtok::XCond)
+          .Case("isa", tgtok::XIsA)
+          .Case("head", tgtok::XHead)
+          .Case("tail", tgtok::XTail)
+          .Case("size", tgtok::XSize)
+          .Case("con", tgtok::XConcat)
+          .Case("dag", tgtok::XDag)
+          .Case("add", tgtok::XADD)
+          .Case("sub", tgtok::XSUB)
+          .Case("mul", tgtok::XMUL)
+          .Case("div", tgtok::XDIV)
+          .Case("not", tgtok::XNOT)
+          .Case("logtwo", tgtok::XLOG2)
+          .Case("and", tgtok::XAND)
+          .Case("or", tgtok::XOR)
+          .Case("xor", tgtok::XXOR)
+          .Case("shl", tgtok::XSHL)
+          .Case("sra", tgtok::XSRA)
+          .Case("srl", tgtok::XSRL)
+          .Case("cast", tgtok::XCast)
+          .Case("empty", tgtok::XEmpty)
+          .Case("subst", tgtok::XSubst)
+          .Case("foldl", tgtok::XFoldl)
+          .Case("foreach", tgtok::XForEach)
+          .Case("filter", tgtok::XFilter)
+          .Case("listconcat", tgtok::XListConcat)
+          .Case("listsplat", tgtok::XListSplat)
+          .Case("listremove", tgtok::XListRemove)
+          .Case("strconcat", tgtok::XStrConcat)
+          .Case("interleave", tgtok::XInterleave)
+          .Case("substr", tgtok::XSubstr)
+          .Case("find", tgtok::XFind)
+          .Cases("setdagop", "setop", tgtok::XSetDagOp) // !setop is deprecated.
+          .Cases("getdagop", "getop", tgtok::XGetDagOp) // !getop is deprecated.
+          .Case("exists", tgtok::XExists)
+          .Case("tolower", tgtok::XToLower)
+          .Case("toupper", tgtok::XToUpper)
+          .Default(tgtok::Error);
 
   return Kind != tgtok::Error ? Kind : ReturnError(Start-1, "Unknown operator");
 }
diff --git a/llvm/lib/TableGen/TGLexer.h b/llvm/lib/TableGen/TGLexer.h
index 284f1bade9de1..93e28a4417a47 100644
--- a/llvm/lib/TableGen/TGLexer.h
+++ b/llvm/lib/TableGen/TGLexer.h
@@ -29,11 +29,11 @@ class SourceMgr;
 class Twine;
 
 namespace tgtok {
-  enum TokKind {
-    // Markers
+enum TokKind {
+  // Markers
     Eof, Error,
 
-    // Tokens with no info.
+  // Tokens with no info.
     minus, plus,        // - +
     l_square, r_square, // [ ]
     l_brace, r_brace,   // { }
@@ -45,36 +45,36 @@ namespace tgtok {
     paste,              // #
     dotdotdot,          // ...
 
-    // Reserved keywords. ('ElseKW' is named to distinguish it from the
-    // existing 'Else' that means the preprocessor #else.)
+  // Reserved keywords. ('ElseKW' is named to distinguish it from the
+  // existing 'Else' that means the preprocessor #else.)
     Assert, Bit, Bits, Class, Code, Dag, Def, Defm, Defset, Defvar, ElseKW,
     FalseKW, Field, Foreach, If, In, Include, Int, Let, List, MultiClass,
     String, Then, TrueKW,
 
-    // Bang operators.
+  // Bang operators.
     XConcat, XADD, XSUB, XMUL, XDIV, XNOT, XLOG2, XAND, XOR, XXOR, XSRA, XSRL,
     XSHL, XListConcat, XListSplat, XStrConcat, XInterleave, XSubstr, XFind,
     XCast, XSubst, XForEach, XFilter, XFoldl, XHead, XTail, XSize, XEmpty, XIf,
     XCond, XEq, XIsA, XDag, XNe, XLe, XLt, XGe, XGt, XSetDagOp, XGetDagOp,
-    XExists, XListRemove,
+    XExists, XListRemove,  XToLower, XToUpper,
 
-    // Boolean literals.
+  // Boolean literals.
     TrueVal, FalseVal,
 
-    // Integer value.
-    IntVal,
+  // Integer value.
+  IntVal,
 
-    // Binary constant.  Note that these are sized according to the number of
-    // bits given.
-    BinaryIntVal,
+  // Binary constant.  Note that these are sized according to the number of
+  // bits given.
+  BinaryIntVal,
 
-    // String valued tokens.
+  // String valued tokens.
     Id, StrVal, VarName, CodeFragment,
 
-    // Preprocessing tokens for internal usage by the lexer.
-    // They are never returned as a result of Lex().
+  // Preprocessing tokens for internal usage by the lexer.
+  // They are never returned as a result of Lex().
     Ifdef, Ifndef, Else, Endif, Define
-  };
+};
 }
 
 /// TGLexer - TableGen Lexer class.
diff --git a/llvm/lib/TableGen/TGParser.cpp b/llvm/lib/TableGen/TGParser.cpp
index 735e565fc4174..6c52edc5dd413 100644
--- a/llvm/lib/TableGen/TGParser.cpp
+++ b/llvm/lib/TableGen/TGParser.cpp
@@ -969,6 +969,8 @@ Init *TGParser::ParseOperation(Record *CurRec, RecTy *ItemType) {
     TokError("unknown bang operator");
     return nullptr;
   case tgtok::XNOT:
+  case tgtok::XToLower:
+  case tgtok::XToUpper:
   case tgtok::XLOG2:
   case tgtok::XHead:
   case tgtok::XTail:
@@ -992,6 +994,16 @@ Init *TGParser::ParseOperation(Record *CurRec, RecTy *ItemType) {
         return nullptr;
       }
 
+      break;
+    case tgtok::XToLower:
+      Lex.Lex(); // eat the operation
+      Code = UnOpInit::TOLOWER;
+      Type = StringRecTy::get(Records);
+      break;
+    case tgtok::XToUpper:
+      Lex.Lex(); // eat the operation
+      Code = UnOpInit::TOUPPER;
+      Type = StringRecTy::get(Records);
       break;
     case tgtok::XNOT:
       Lex.Lex();  // eat the operation
@@ -2470,6 +2482,8 @@ Init *TGParser::ParseSimpleValue(Record *CurRec, RecTy *ItemType,
   case tgtok::XSize:
   case tgtok::XEmpty:
   case tgtok::XCast:
+  case tgtok::XToLower:
+  case tgtok::XToUpper:
   case tgtok::XGetDagOp: // Value ::= !unop '(' Value ')'
   case tgtok::XExists:
   case tgtok::XIsA:
diff --git a/llvm/lib/Target/AArch64/AArch64.td b/llvm/lib/Target/AArch64/AArch64.td
index cbe0b83f9e553..67b5cf10b14dd 100644
--- a/llvm/lib/Target/AArch64/AArch64.td
+++ b/llvm/lib/Target/AArch64/AArch64.td
@@ -522,6 +522,12 @@ def FeatureNoBTIAtReturnTwice : SubtargetFeature<"no-bti-at-return-twice",
                                                  "Don't place a BTI instruction "
                                                  "after a return-twice">;
 
+def FeatureCHK : SubtargetFeature<"chk", "HasCHK",
+    "true", "Enable Armv8.0-A Check Feature Status Extension (FEAT_CHK)">;
+
+def FeatureGCS : SubtargetFeature<"gcs", "HasGCS",
+    "true", "Enable Armv9.4-A Guarded Call Stack Extension", [FeatureCHK]>;
+
 def FeatureCLRBHB : SubtargetFeature<"clrbhb", "HasCLRBHB",
     "true", "Enable Clear BHB instruction (FEAT_CLRBHB)">;
 
@@ -603,7 +609,7 @@ def HasV8_8aOps : SubtargetFeature<
 def HasV8_9aOps : SubtargetFeature<
   "v8.9a", "HasV8_9aOps", "true", "Support ARM v8.9a instructions",
   [HasV8_8aOps, FeatureCLRBHB, FeaturePRFM_SLC, FeatureSPECRES2,
-   FeatureCSSC, FeatureRASv2]>;
+   FeatureCSSC, FeatureRASv2, FeatureCHK]>;
 
 def HasV9_0aOps : SubtargetFeature<
   "v9a", "HasV9_0aOps", "true", "Support ARM v9a instructions",
@@ -666,6 +672,7 @@ include "AArch64InstrInfo.td"
 include "AArch64SchedPredicates.td"
 include "AArch64SchedPredExynos.td"
 include "AArch64SchedPredAmpere.td"
+include "AArch64SchedPredNeoverse.td"
 include "AArch64Combine.td"
 
 def AArch64InstrInfo : InstrInfo;
diff --git a/llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp b/llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp
index 26d59acf60e77..a413c924c862a 100644
--- a/llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp
@@ -1177,6 +1177,8 @@ bool AArch64ExpandPseudo::expandMI(MachineBasicBlock &MBB,
         .add(MI.getOperand(2))
         .addImm(AArch64_AM::getShifterImm(AArch64_AM::LSL, 0));
     transferImpOps(MI, MIB1, MIB1);
+    if (auto DebugNumber = MI.peekDebugInstrNum())
+      NewMI->setDebugInstrNum(DebugNumber);
     MI.eraseFromParent();
     return true;
   }
diff --git a/llvm/lib/Target/AArch64/AArch64FrameLowering.cpp b/llvm/lib/Target/AArch64/AArch64FrameLowering.cpp
index 603dccdc82487..33b586e2c1b40 100644
--- a/llvm/lib/Target/AArch64/AArch64FrameLowering.cpp
+++ b/llvm/lib/Target/AArch64/AArch64FrameLowering.cpp
@@ -954,10 +954,10 @@ bool AArch64FrameLowering::shouldCombineCSRLocalStackBumpInEpilogue(
   switch (LastI->getOpcode()) {
   case AArch64::STGloop:
   case AArch64::STZGloop:
-  case AArch64::STGOffset:
-  case AArch64::STZGOffset:
-  case AArch64::ST2GOffset:
-  case AArch64::STZ2GOffset:
+  case AArch64::STGi:
+  case AArch64::STZGi:
+  case AArch64::ST2Gi:
+  case AArch64::STZ2Gi:
     return false;
   default:
     return true;
@@ -3455,8 +3455,8 @@ void TagStoreEdit::emitUnrolled(MachineBasicBlock::iterator InsertI) {
     int64_t InstrSize = (Size > 16) ? 32 : 16;
     unsigned Opcode =
         InstrSize == 16
-            ? (ZeroData ? AArch64::STZGOffset : AArch64::STGOffset)
-            : (ZeroData ? AArch64::STZ2GOffset : AArch64::ST2GOffset);
+            ? (ZeroData ? AArch64::STZGi : AArch64::STGi)
+            : (ZeroData ? AArch64::STZ2Gi : AArch64::ST2Gi);
     assert(BaseRegOffsetBytes % 16 == 0);
     MachineInstr *I = BuildMI(*MBB, InsertI, DL, TII->get(Opcode))
                           .addReg(AArch64::SP)
@@ -3638,8 +3638,8 @@ bool isMergeableStackTaggingInstruction(MachineInstr &MI, int64_t &Offset,
   const MachineFrameInfo &MFI = MF.getFrameInfo();
 
   unsigned Opcode = MI.getOpcode();
-  ZeroData = (Opcode == AArch64::STZGloop || Opcode == AArch64::STZGOffset ||
-              Opcode == AArch64::STZ2GOffset);
+  ZeroData = (Opcode == AArch64::STZGloop || Opcode == AArch64::STZGi ||
+              Opcode == AArch64::STZ2Gi);
 
   if (Opcode == AArch64::STGloop || Opcode == AArch64::STZGloop) {
     if (!MI.getOperand(0).isDead() || !MI.getOperand(1).isDead())
@@ -3651,9 +3651,9 @@ bool isMergeableStackTaggingInstruction(MachineInstr &MI, int64_t &Offset,
     return true;
   }
 
-  if (Opcode == AArch64::STGOffset || Opcode == AArch64::STZGOffset)
+  if (Opcode == AArch64::STGi || Opcode == AArch64::STZGi)
     Size = 16;
-  else if (Opcode == AArch64::ST2GOffset || Opcode == AArch64::STZ2GOffset)
+  else if (Opcode == AArch64::ST2Gi || Opcode == AArch64::STZ2Gi)
     Size = 32;
   else
     return false;
@@ -3903,10 +3903,10 @@ void AArch64FrameLowering::orderFrameObjects(
       case AArch64::STZGloop:
         OpIndex = 3;
         break;
-      case AArch64::STGOffset:
-      case AArch64::STZGOffset:
-      case AArch64::ST2GOffset:
-      case AArch64::STZ2GOffset:
+      case AArch64::STGi:
+      case AArch64::STZGi:
+      case AArch64::ST2Gi:
+      case AArch64::STZ2Gi:
         OpIndex = 1;
         break;
       default:
diff --git a/llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp b/llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp
index 401543f50416e..2e95966c5ae19 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp
@@ -6545,7 +6545,7 @@ bool AArch64DAGToDAGISel::SelectAddrModeIndexedSVE(SDNode *Root, SDValue N,
     int FI = cast<FrameIndexSDNode>(N)->getIndex();
     // We can only encode VL scaled offsets, so only fold in frame indexes
     // referencing SVE objects.
-    if (FI == 0 || MFI.getStackID(FI) == TargetStackID::ScalableVector) {
+    if (MFI.getStackID(FI) == TargetStackID::ScalableVector) {
       Base = CurDAG->getTargetFrameIndex(FI, TLI->getPointerTy(DL));
       OffImm = CurDAG->getTargetConstant(0, SDLoc(N), MVT::i64);
       return true;
@@ -6580,7 +6580,7 @@ bool AArch64DAGToDAGISel::SelectAddrModeIndexedSVE(SDNode *Root, SDValue N,
     int FI = cast<FrameIndexSDNode>(Base)->getIndex();
     // We can only encode VL scaled offsets, so only fold in frame indexes
     // referencing SVE objects.
-    if (FI == 0 || MFI.getStackID(FI) == TargetStackID::ScalableVector)
+    if (MFI.getStackID(FI) == TargetStackID::ScalableVector)
       Base = CurDAG->getTargetFrameIndex(FI, TLI->getPointerTy(DL));
   }
 
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
index aa1049cf09a08..0d3857743cb33 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
@@ -1302,12 +1302,10 @@ AArch64TargetLowering::AArch64TargetLowering(const TargetMachine &TM,
       setOperationAction(ISD::SDIVREM, VT, Expand);
       setOperationAction(ISD::UDIVREM, VT, Expand);
 
-      if (Subtarget->hasSVE2()) {
-        setOperationAction(ISD::AVGFLOORS, VT, Custom);
-        setOperationAction(ISD::AVGFLOORU, VT, Custom);
-        setOperationAction(ISD::AVGCEILS, VT, Custom);
-        setOperationAction(ISD::AVGCEILU, VT, Custom);
-      }
+      setOperationAction(ISD::AVGFLOORS, VT, Custom);
+      setOperationAction(ISD::AVGFLOORU, VT, Custom);
+      setOperationAction(ISD::AVGCEILS, VT, Custom);
+      setOperationAction(ISD::AVGCEILU, VT, Custom);
     }
 
     // Illegal unpacked integer vector types.
@@ -1495,11 +1493,11 @@ AArch64TargetLowering::AArch64TargetLowering(const TargetMachine &TM,
       setTruncStoreAction(MVT::v4f64, MVT::v4f32, Custom);
       for (MVT VT : {MVT::v8i8, MVT::v16i8, MVT::v4i16, MVT::v8i16, MVT::v2i32,
                      MVT::v4i32, MVT::v1i64, MVT::v2i64})
-        addTypeForStreamingSVE(VT);
+        addTypeForFixedLengthSVE(VT, /*StreamingSVE=*/ true);
 
       for (MVT VT :
            {MVT::v4f16, MVT::v8f16, MVT::v2f32, MVT::v4f32, MVT::v2f64})
-        addTypeForStreamingSVE(VT);
+        addTypeForFixedLengthSVE(VT, /*StreamingSVE=*/ true);
     }
 
     // NOTE: Currently this has to happen after computeRegisterProperties rather
@@ -1507,10 +1505,10 @@ AArch64TargetLowering::AArch64TargetLowering(const TargetMachine &TM,
     if (Subtarget->useSVEForFixedLengthVectors()) {
       for (MVT VT : MVT::integer_fixedlen_vector_valuetypes())
         if (useSVEForFixedLengthVectorVT(VT))
-          addTypeForFixedLengthSVE(VT);
+          addTypeForFixedLengthSVE(VT, /*StreamingSVE=*/ false);
       for (MVT VT : MVT::fp_fixedlen_vector_valuetypes())
         if (useSVEForFixedLengthVectorVT(VT))
-          addTypeForFixedLengthSVE(VT);
+          addTypeForFixedLengthSVE(VT, /*StreamingSVE=*/ false);
 
       // 64bit results can mean a bigger than NEON input.
       for (auto VT : {MVT::v8i8, MVT::v4i16})
@@ -1716,139 +1714,8 @@ bool AArch64TargetLowering::shouldExpandGetActiveLaneMask(EVT ResVT,
   return false;
 }
 
-void AArch64TargetLowering::addTypeForStreamingSVE(MVT VT) {
-  // By default set all operations to Expand,
-  // then change to Legal/Custom if needed.
-  for (unsigned Op = 0; Op < ISD::BUILTIN_OP_END; ++Op)
-    setOperationAction(Op, VT, Expand);
-
-  assert(VT.isFixedLengthVector() && "Expected fixed length vector type!");
-
-  if (VT.isFloatingPoint()) {
-    setCondCodeAction(ISD::SETO, VT, Expand);
-    setCondCodeAction(ISD::SETOLT, VT, Expand);
-    setCondCodeAction(ISD::SETOLE, VT, Expand);
-    setCondCodeAction(ISD::SETULT, VT, Expand);
-    setCondCodeAction(ISD::SETULE, VT, Expand);
-    setCondCodeAction(ISD::SETUGE, VT, Expand);
-    setCondCodeAction(ISD::SETUGT, VT, Expand);
-    setCondCodeAction(ISD::SETUEQ, VT, Expand);
-    setCondCodeAction(ISD::SETONE, VT, Expand);
-  }
-
-  // STORE, LOAD, SCALAR_TO_VECTOR and BITCAST are natively supported,
-  // so no need to Custom/Expand them.
-  setOperationAction(ISD::STORE, VT, Legal);
-  setOperationAction(ISD::LOAD, VT, Legal);
-  setOperationAction(ISD::SCALAR_TO_VECTOR, VT, Legal);
-  setOperationAction(ISD::BITCAST, VT, Legal);
-
-  // Mark integer truncating stores/extending loads as having custom lowering
-  if (VT.isInteger()) {
-    MVT InnerVT = VT.changeVectorElementType(MVT::i8);
-    while (InnerVT != VT) {
-      setTruncStoreAction(VT, InnerVT, Custom);
-      setLoadExtAction(ISD::ZEXTLOAD, VT, InnerVT, Custom);
-      setLoadExtAction(ISD::SEXTLOAD, VT, InnerVT, Custom);
-      InnerVT = InnerVT.changeVectorElementType(
-          MVT::getIntegerVT(2 * InnerVT.getScalarSizeInBits()));
-    }
-  }
-
-  // Mark floating-point truncating stores/extending loads as having custom
-  // lowering
-  if (VT.isFloatingPoint()) {
-    MVT InnerVT = VT.changeVectorElementType(MVT::f16);
-    while (InnerVT != VT) {
-      setTruncStoreAction(VT, InnerVT, Custom);
-      setLoadExtAction(ISD::EXTLOAD, VT, InnerVT, Custom);
-      InnerVT = InnerVT.changeVectorElementType(
-          MVT::getFloatingPointVT(2 * InnerVT.getScalarSizeInBits()));
-    }
-  }
-
-  setOperationAction(ISD::ABS, VT, Custom);
-  setOperationAction(ISD::ADD, VT, Custom);
-  setOperationAction(ISD::AND, VT, Custom);
-  setOperationAction(ISD::ANY_EXTEND, VT, Custom);
-  setOperationAction(ISD::BITREVERSE, VT, Custom);
-  setOperationAction(ISD::BSWAP, VT, Custom);
-  setOperationAction(ISD::BUILD_VECTOR, VT, Custom);
-  setOperationAction(ISD::CONCAT_VECTORS, VT, Custom);
-  setOperationAction(ISD::CTLZ, VT, Custom);
-  setOperationAction(ISD::CTPOP, VT, Custom);
-  setOperationAction(ISD::CTTZ, VT, Custom);
-  setOperationAction(ISD::EXTRACT_SUBVECTOR, VT, Custom);
-  setOperationAction(ISD::EXTRACT_VECTOR_ELT, VT, Custom);
-  setOperationAction(ISD::FABS, VT, Custom);
-  setOperationAction(ISD::FADD, VT, Custom);
-  setOperationAction(ISD::FCEIL, VT, Custom);
-  setOperationAction(ISD::FCOPYSIGN, VT, Custom);
-  setOperationAction(ISD::FDIV, VT, Custom);
-  setOperationAction(ISD::FFLOOR, VT, Custom);
-  setOperationAction(ISD::FMA, VT, Custom);
-  setOperationAction(ISD::FMAXIMUM, VT, Custom);
-  setOperationAction(ISD::FMAXNUM, VT, Custom);
-  setOperationAction(ISD::FMINIMUM, VT, Custom);
-  setOperationAction(ISD::FMINNUM, VT, Custom);
-  setOperationAction(ISD::FMUL, VT, Custom);
-  setOperationAction(ISD::FNEARBYINT, VT, Custom);
-  setOperationAction(ISD::FNEG, VT, Custom);
-  setOperationAction(ISD::FP_EXTEND, VT, Custom);
-  setOperationAction(ISD::FP_ROUND, VT, Custom);
-  setOperationAction(ISD::FP_TO_SINT, VT, Custom);
-  setOperationAction(ISD::FP_TO_UINT, VT, Custom);
-  setOperationAction(ISD::FRINT, VT, Custom);
-  setOperationAction(ISD::FROUND, VT, Custom);
-  setOperationAction(ISD::FROUNDEVEN, VT, Custom);
-  setOperationAction(ISD::FSQRT, VT, Custom);
-  setOperationAction(ISD::FSUB, VT, Custom);
-  setOperationAction(ISD::FTRUNC, VT, Custom);
-  setOperationAction(ISD::INSERT_VECTOR_ELT, VT, Custom);
-  setOperationAction(ISD::MLOAD, VT, Custom);
-  setOperationAction(ISD::MSTORE, VT, Custom);
-  setOperationAction(ISD::MUL, VT, Custom);
-  setOperationAction(ISD::MULHS, VT, Custom);
-  setOperationAction(ISD::MULHU, VT, Custom);
-  setOperationAction(ISD::OR, VT, Custom);
-  setOperationAction(ISD::SDIV, VT, Custom);
-  setOperationAction(ISD::SELECT, VT, Custom);
-  setOperationAction(ISD::SETCC, VT, Custom);
-  setOperationAction(ISD::SHL, VT, Custom);
-  setOperationAction(ISD::SIGN_EXTEND, VT, Custom);
-  setOperationAction(ISD::SIGN_EXTEND_INREG, VT, Custom);
-  setOperationAction(ISD::SINT_TO_FP, VT, Custom);
-  setOperationAction(ISD::SMAX, VT, Custom);
-  setOperationAction(ISD::SMIN, VT, Custom);
-  setOperationAction(ISD::SPLAT_VECTOR, VT, Custom);
-  setOperationAction(ISD::SRA, VT, Custom);
-  setOperationAction(ISD::SRL, VT, Custom);
-  setOperationAction(ISD::SUB, VT, Custom);
-  setOperationAction(ISD::TRUNCATE, VT, Custom);
-  setOperationAction(ISD::UDIV, VT, Custom);
-  setOperationAction(ISD::UINT_TO_FP, VT, Custom);
-  setOperationAction(ISD::UMAX, VT, Custom);
-  setOperationAction(ISD::UMIN, VT, Custom);
-  setOperationAction(ISD::VECREDUCE_ADD, VT, Custom);
-  setOperationAction(ISD::VECREDUCE_AND, VT, Custom);
-  setOperationAction(ISD::VECREDUCE_FADD, VT, Custom);
-  setOperationAction(ISD::VECREDUCE_FMAX, VT, Custom);
-  setOperationAction(ISD::VECREDUCE_FMIN, VT, Custom);
-  setOperationAction(ISD::VECREDUCE_OR, VT, Custom);
-  setOperationAction(ISD::VECREDUCE_SEQ_FADD, VT, Custom);
-  setOperationAction(ISD::VECREDUCE_SMAX, VT, Custom);
-  setOperationAction(ISD::VECREDUCE_SMIN, VT, Custom);
-  setOperationAction(ISD::VECREDUCE_UMAX, VT, Custom);
-  setOperationAction(ISD::VECREDUCE_UMIN, VT, Custom);
-  setOperationAction(ISD::VECREDUCE_XOR, VT, Custom);
-  setOperationAction(ISD::VECTOR_SHUFFLE, VT, Custom);
-  setOperationAction(ISD::VECTOR_SPLICE, VT, Custom);
-  setOperationAction(ISD::VSELECT, VT, Custom);
-  setOperationAction(ISD::XOR, VT, Custom);
-  setOperationAction(ISD::ZERO_EXTEND, VT, Custom);
-}
-
-void AArch64TargetLowering::addTypeForFixedLengthSVE(MVT VT) {
+void AArch64TargetLowering::addTypeForFixedLengthSVE(MVT VT,
+                                                     bool StreamingSVE) {
   assert(VT.isFixedLengthVector() && "Expected fixed length vector type!");
 
   // By default everything must be expanded.
@@ -1896,7 +1763,7 @@ void AArch64TargetLowering::addTypeForFixedLengthSVE(MVT VT) {
   setOperationAction(ISD::ADD, VT, Custom);
   setOperationAction(ISD::AND, VT, Custom);
   setOperationAction(ISD::ANY_EXTEND, VT, Custom);
-  setOperationAction(ISD::BITCAST, VT, Custom);
+  setOperationAction(ISD::BITCAST, VT, StreamingSVE ? Legal : Custom);
   setOperationAction(ISD::BITREVERSE, VT, Custom);
   setOperationAction(ISD::BSWAP, VT, Custom);
   setOperationAction(ISD::BUILD_VECTOR, VT, Custom);
@@ -1931,15 +1798,16 @@ void AArch64TargetLowering::addTypeForFixedLengthSVE(MVT VT) {
   setOperationAction(ISD::FSUB, VT, Custom);
   setOperationAction(ISD::FTRUNC, VT, Custom);
   setOperationAction(ISD::INSERT_VECTOR_ELT, VT, Custom);
-  setOperationAction(ISD::LOAD, VT, Custom);
-  setOperationAction(ISD::MGATHER, VT, Custom);
+  setOperationAction(ISD::LOAD, VT, StreamingSVE ? Legal : Custom);
+  setOperationAction(ISD::MGATHER, VT, StreamingSVE ? Expand : Custom);
   setOperationAction(ISD::MLOAD, VT, Custom);
-  setOperationAction(ISD::MSCATTER, VT, Custom);
+  setOperationAction(ISD::MSCATTER, VT, StreamingSVE ? Expand : Custom);
   setOperationAction(ISD::MSTORE, VT, Custom);
   setOperationAction(ISD::MUL, VT, Custom);
   setOperationAction(ISD::MULHS, VT, Custom);
   setOperationAction(ISD::MULHU, VT, Custom);
   setOperationAction(ISD::OR, VT, Custom);
+  setOperationAction(ISD::SCALAR_TO_VECTOR, VT, StreamingSVE ? Legal : Expand);
   setOperationAction(ISD::SDIV, VT, Custom);
   setOperationAction(ISD::SELECT, VT, Custom);
   setOperationAction(ISD::SETCC, VT, Custom);
@@ -1952,7 +1820,7 @@ void AArch64TargetLowering::addTypeForFixedLengthSVE(MVT VT) {
   setOperationAction(ISD::SPLAT_VECTOR, VT, Custom);
   setOperationAction(ISD::SRA, VT, Custom);
   setOperationAction(ISD::SRL, VT, Custom);
-  setOperationAction(ISD::STORE, VT, Custom);
+  setOperationAction(ISD::STORE, VT, StreamingSVE ? Legal : Custom);
   setOperationAction(ISD::SUB, VT, Custom);
   setOperationAction(ISD::TRUNCATE, VT, Custom);
   setOperationAction(ISD::UDIV, VT, Custom);
@@ -6107,13 +5975,13 @@ SDValue AArch64TargetLowering::LowerOperation(SDValue Op,
   case ISD::ABDU:
     return LowerToPredicatedOp(Op, DAG, AArch64ISD::ABDU_PRED);
   case ISD::AVGFLOORS:
-    return LowerToPredicatedOp(Op, DAG, AArch64ISD::HADDS_PRED);
+    return LowerAVG(Op, DAG, AArch64ISD::HADDS_PRED);
   case ISD::AVGFLOORU:
-    return LowerToPredicatedOp(Op, DAG, AArch64ISD::HADDU_PRED);
+    return LowerAVG(Op, DAG, AArch64ISD::HADDU_PRED);
   case ISD::AVGCEILS:
-    return LowerToPredicatedOp(Op, DAG, AArch64ISD::RHADDS_PRED);
+    return LowerAVG(Op, DAG, AArch64ISD::RHADDS_PRED);
   case ISD::AVGCEILU:
-    return LowerToPredicatedOp(Op, DAG, AArch64ISD::RHADDU_PRED);
+    return LowerAVG(Op, DAG, AArch64ISD::RHADDU_PRED);
   case ISD::BITREVERSE:
     return LowerBitreverse(Op, DAG);
   case ISD::BSWAP:
@@ -11184,23 +11052,17 @@ static SDValue GeneratePerfectShuffle(unsigned ID, SDValue V1,
                        DAG.getConstant(Imm, dl, MVT::i32));
   }
   case OP_VUZPL:
-    return DAG.getNode(AArch64ISD::UZP1, dl, DAG.getVTList(VT, VT), OpLHS,
-                       OpRHS);
+    return DAG.getNode(AArch64ISD::UZP1, dl, VT, OpLHS, OpRHS);
   case OP_VUZPR:
-    return DAG.getNode(AArch64ISD::UZP2, dl, DAG.getVTList(VT, VT), OpLHS,
-                       OpRHS);
+    return DAG.getNode(AArch64ISD::UZP2, dl, VT, OpLHS, OpRHS);
   case OP_VZIPL:
-    return DAG.getNode(AArch64ISD::ZIP1, dl, DAG.getVTList(VT, VT), OpLHS,
-                       OpRHS);
+    return DAG.getNode(AArch64ISD::ZIP1, dl, VT, OpLHS, OpRHS);
   case OP_VZIPR:
-    return DAG.getNode(AArch64ISD::ZIP2, dl, DAG.getVTList(VT, VT), OpLHS,
-                       OpRHS);
+    return DAG.getNode(AArch64ISD::ZIP2, dl, VT, OpLHS, OpRHS);
   case OP_VTRNL:
-    return DAG.getNode(AArch64ISD::TRN1, dl, DAG.getVTList(VT, VT), OpLHS,
-                       OpRHS);
+    return DAG.getNode(AArch64ISD::TRN1, dl, VT, OpLHS, OpRHS);
   case OP_VTRNR:
-    return DAG.getNode(AArch64ISD::TRN2, dl, DAG.getVTList(VT, VT), OpLHS,
-                       OpRHS);
+    return DAG.getNode(AArch64ISD::TRN2, dl, VT, OpLHS, OpRHS);
   }
 }
 
@@ -13374,6 +13236,57 @@ SDValue AArch64TargetLowering::LowerWindowsDYNAMIC_STACKALLOC(
   return Chain;
 }
 
+// When x and y are extended, lower:
+//   avgfloor(x, y) -> (x + y) >> 1
+//   avgceil(x, y)  -> (x + y + 1) >> 1
+
+// Otherwise, lower to:
+//   avgfloor(x, y) -> (x >> 1) + (y >> 1) + (x & y & 1)
+//   avgceil(x, y)  -> (x >> 1) + (y >> 1) + ((x || y) & 1)
+SDValue AArch64TargetLowering::LowerAVG(SDValue Op, SelectionDAG &DAG,
+                                        unsigned NewOp) const {
+  if (Subtarget->hasSVE2())
+    return LowerToPredicatedOp(Op, DAG, NewOp);
+
+  SDLoc dl(Op);
+  SDValue OpA = Op->getOperand(0);
+  SDValue OpB = Op->getOperand(1);
+  EVT VT = Op.getValueType();
+  bool IsCeil =
+      (Op->getOpcode() == ISD::AVGCEILS || Op->getOpcode() == ISD::AVGCEILU);
+  bool IsSigned =
+      (Op->getOpcode() == ISD::AVGFLOORS || Op->getOpcode() == ISD::AVGCEILS);
+  unsigned ShiftOpc = IsSigned ? ISD::SRA : ISD::SRL;
+
+  assert(VT.isScalableVector() && "Only expect to lower scalable vector op!");
+
+  auto IsZeroExtended = [&DAG](SDValue &Node) {
+    KnownBits Known = DAG.computeKnownBits(Node, 0);
+    return Known.Zero.isSignBitSet();
+  };
+
+  auto IsSignExtended = [&DAG](SDValue &Node) {
+    return (DAG.ComputeNumSignBits(Node, 0) > 1);
+  };
+
+  SDValue ConstantOne = DAG.getConstant(1, dl, VT);
+  if ((!IsSigned && IsZeroExtended(OpA) && IsZeroExtended(OpB)) ||
+      (IsSigned && IsSignExtended(OpA) && IsSignExtended(OpB))) {
+    SDValue Add = DAG.getNode(ISD::ADD, dl, VT, OpA, OpB);
+    if (IsCeil)
+      Add = DAG.getNode(ISD::ADD, dl, VT, Add, ConstantOne);
+    return DAG.getNode(ShiftOpc, dl, VT, Add, ConstantOne);
+  }
+
+  SDValue ShiftOpA = DAG.getNode(ShiftOpc, dl, VT, OpA, ConstantOne);
+  SDValue ShiftOpB = DAG.getNode(ShiftOpc, dl, VT, OpB, ConstantOne);
+
+  SDValue tmp = DAG.getNode(IsCeil ? ISD::OR : ISD::AND, dl, VT, OpA, OpB);
+  tmp = DAG.getNode(ISD::AND, dl, VT, tmp, ConstantOne);
+  SDValue Add = DAG.getNode(ISD::ADD, dl, VT, ShiftOpA, ShiftOpB);
+  return DAG.getNode(ISD::ADD, dl, VT, Add, tmp);
+}
+
 SDValue
 AArch64TargetLowering::LowerDYNAMIC_STACKALLOC(SDValue Op,
                                                SelectionDAG &DAG) const {
@@ -19348,6 +19261,15 @@ static SDValue performPostLD1Combine(SDNode *N,
       return SDValue();
   }
 
+  // If there is one use and it can splat the value, prefer that operation.
+  // TODO: This could be expanded to more operations if they reliably use the
+  // index variants.
+  if (N->hasOneUse()) {
+    unsigned UseOpc = N->use_begin()->getOpcode();
+    if (UseOpc == ISD::FMUL || UseOpc == ISD::FMA)
+      return SDValue();
+  }
+
   SDValue Addr = LD->getOperand(1);
   SDValue Vector = N->getOperand(0);
   // Search for a use of the address operand that is an increment.
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.h b/llvm/lib/Target/AArch64/AArch64ISelLowering.h
index a878b2b089473..c1b212776fbb7 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.h
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.h
@@ -932,8 +932,7 @@ class AArch64TargetLowering : public TargetLowering {
   bool isExtFreeImpl(const Instruction *Ext) const override;
 
   void addTypeForNEON(MVT VT);
-  void addTypeForStreamingSVE(MVT VT);
-  void addTypeForFixedLengthSVE(MVT VT);
+  void addTypeForFixedLengthSVE(MVT VT, bool StreamingSVE);
   void addDRTypeForNEON(MVT VT);
   void addQRTypeForNEON(MVT VT);
 
@@ -1091,6 +1090,7 @@ class AArch64TargetLowering : public TargetLowering {
   SDValue LowerWindowsDYNAMIC_STACKALLOC(SDValue Op, SDValue Chain,
                                          SDValue &Size,
                                          SelectionDAG &DAG) const;
+  SDValue LowerAVG(SDValue Op, SelectionDAG &DAG, unsigned NewOp) const;
 
   SDValue LowerFixedLengthVectorIntDivideToSVE(SDValue Op,
                                                SelectionDAG &DAG) const;
diff --git a/llvm/lib/Target/AArch64/AArch64InstrFormats.td b/llvm/lib/Target/AArch64/AArch64InstrFormats.td
index 0d22a2c623641..0cd03dbf47718 100644
--- a/llvm/lib/Target/AArch64/AArch64InstrFormats.td
+++ b/llvm/lib/Target/AArch64/AArch64InstrFormats.td
@@ -4673,7 +4673,7 @@ class BaseMemTagStore<bits<2> opc1, bits<2> opc2, string asm_insn,
 }
 
 multiclass MemTagStore<bits<2> opc1, string insn> {
-  def Offset :
+  def i :
     BaseMemTagStore<opc1, 0b10, insn, "\t$Rt, [$Rn, $offset]", "",
                     (outs), (ins GPR64sp:$Rt, GPR64sp:$Rn, simm9s16:$offset)>;
   def PreIndex :
@@ -4688,7 +4688,7 @@ multiclass MemTagStore<bits<2> opc1, string insn> {
                     (ins GPR64sp:$Rt, GPR64sp:$Rn, simm9s16:$offset)>;
 
   def : InstAlias<insn # "\t$Rt, [$Rn]",
-                  (!cast<Instruction>(NAME # "Offset") GPR64sp:$Rt, GPR64sp:$Rn, 0)>;
+                  (!cast<Instruction>(NAME # "i") GPR64sp:$Rt, GPR64sp:$Rn, 0)>;
 }
 
 //---
diff --git a/llvm/lib/Target/AArch64/AArch64InstrInfo.cpp b/llvm/lib/Target/AArch64/AArch64InstrInfo.cpp
index 6a1617910b5f9..54ad05bc4698b 100644
--- a/llvm/lib/Target/AArch64/AArch64InstrInfo.cpp
+++ b/llvm/lib/Target/AArch64/AArch64InstrInfo.cpp
@@ -2369,7 +2369,7 @@ unsigned AArch64InstrInfo::getLoadStoreImmIdx(unsigned Opc) {
   case AArch64::LDNF1D_IMM:
     return 3;
   case AArch64::ADDG:
-  case AArch64::STGOffset:
+  case AArch64::STGi:
   case AArch64::LDR_PXI:
   case AArch64::STR_PXI:
     return 2;
@@ -2874,8 +2874,8 @@ bool AArch64InstrInfo::getMemOpInfo(unsigned Opcode, TypeSize &Scale,
     MaxOffset = 63;
     break;
   case AArch64::LDG:
-  case AArch64::STGOffset:
-  case AArch64::STZGOffset:
+  case AArch64::STGi:
+  case AArch64::STZGi:
     Scale = TypeSize::Fixed(16);
     Width = 16;
     MinOffset = -256;
@@ -3033,8 +3033,8 @@ bool AArch64InstrInfo::getMemOpInfo(unsigned Opcode, TypeSize &Scale,
     MinOffset = -8;
     MaxOffset = 7;
     break;
-  case AArch64::ST2GOffset:
-  case AArch64::STZ2GOffset:
+  case AArch64::ST2Gi:
+  case AArch64::STZ2Gi:
     Scale = TypeSize::Fixed(16);
     Width = 32;
     MinOffset = -256;
@@ -3151,10 +3151,10 @@ int AArch64InstrInfo::getMemScale(unsigned Opc) {
   case AArch64::LDPQi:
   case AArch64::LDRQpre:
   case AArch64::STPQi:
-  case AArch64::STGOffset:
-  case AArch64::STZGOffset:
-  case AArch64::ST2GOffset:
-  case AArch64::STZ2GOffset:
+  case AArch64::STGi:
+  case AArch64::STZGi:
+  case AArch64::ST2Gi:
+  case AArch64::STZ2Gi:
   case AArch64::STGPi:
     return 16;
   }
diff --git a/llvm/lib/Target/AArch64/AArch64InstrInfo.td b/llvm/lib/Target/AArch64/AArch64InstrInfo.td
index 2ab27f5225ea2..c889bb72e6cb2 100644
--- a/llvm/lib/Target/AArch64/AArch64InstrInfo.td
+++ b/llvm/lib/Target/AArch64/AArch64InstrInfo.td
@@ -244,6 +244,10 @@ def HasLSE128        : Predicate<"Subtarget->hasLSE128()">,
                        AssemblerPredicateWithAll<(all_of FeatureLSE128), "lse128">;
 def HasD128          : Predicate<"Subtarget->hasD128()">,
                        AssemblerPredicateWithAll<(all_of FeatureD128), "d128">;
+def HasCHK           : Predicate<"Subtarget->hasCHK()">,
+                       AssemblerPredicateWithAll<(all_of FeatureCHK), "chk">;
+def HasGCS           : Predicate<"Subtarget->hasGCS()">,
+                       AssemblerPredicateWithAll<(all_of FeatureGCS), "gcs">;
 def IsLE             : Predicate<"Subtarget->isLittleEndian()">;
 def IsBE             : Predicate<"!Subtarget->isLittleEndian()">;
 def IsWindows        : Predicate<"Subtarget->isTargetWindows()">;
@@ -1050,6 +1054,71 @@ def BRB_INJ:  BRBEI<0b101, "\tinj">;
 def : TokenAlias<"INJ", "inj">;
 def : TokenAlias<"IALL", "iall">;
 
+
+// ARMv9.4-A Guarded Control Stack
+class GCSNoOp<bits<3> op2, string mnemonic>
+    : SimpleSystemI<0, (ins), mnemonic, "">, Sched<[]> {
+  let Inst{20-8} = 0b0100001110111;
+  let Inst{7-5} = op2;
+  let Predicates = [HasGCS];
+}
+def GCSPUSHX : GCSNoOp<0b100, "gcspushx">;
+def GCSPOPCX : GCSNoOp<0b101, "gcspopcx">;
+def GCSPOPX  : GCSNoOp<0b110, "gcspopx">;
+
+class GCSRtIn<bits<3> op1, bits<3> op2, string mnemonic,
+            list<dag> pattern = []>
+    : RtSystemI<0, (outs), (ins GPR64:$Rt), mnemonic, "\t$Rt", pattern> {
+  let Inst{20-19} = 0b01;
+  let Inst{18-16} = op1;
+  let Inst{15-8} = 0b01110111;
+  let Inst{7-5} = op2;
+  let Predicates = [HasGCS];
+}
+
+def GCSSS1   : GCSRtIn<0b011, 0b010, "gcsss1">;
+def GCSPUSHM : GCSRtIn<0b011, 0b000, "gcspushm">;
+
+class GCSRtOut<bits<3> op1, bits<3> op2, string mnemonic,
+            list<dag> pattern = []>
+    : RtSystemI<1, (outs GPR64:$Rt), (ins), mnemonic, "\t$Rt", pattern> {
+  let Inst{20-19} = 0b01;
+  let Inst{18-16} = op1;
+  let Inst{15-8} = 0b01110111;
+  let Inst{7-5} = op2;
+  let Predicates = [HasGCS];
+}
+
+def GCSSS2  : GCSRtOut<0b011, 0b011, "gcsss2">;
+def GCSPOPM : GCSRtOut<0b011, 0b001, "gcspopm">;
+def GCSPOPM_NoOp : InstAlias<"gcspopm", (GCSPOPM XZR)>, Requires<[HasGCS]>; // Rt defaults to XZR if absent
+
+def GCSB_DSYNC_disable : InstAlias<"gcsb\tdsync", (HINT 19), 0>;
+def GCSB_DSYNC         : InstAlias<"gcsb\tdsync", (HINT 19), 1>, Requires<[HasGCS]>;
+
+def : TokenAlias<"DSYNC", "dsync">;
+
+let Uses = [X16], Defs = [X16], CRm = 0b0101 in {
+  def CHKFEAT   : SystemNoOperands<0b000, "hint\t#40">;
+}
+def : InstAlias<"chkfeat\tx16", (CHKFEAT), 0>;
+def : InstAlias<"chkfeat\tx16", (CHKFEAT), 1>, Requires<[HasCHK]>;
+
+class GCSSt<string mnemonic, bits<3> op>
+    : I<(outs), (ins GPR64:$Rt, GPR64sp:$Rn), mnemonic, "\t$Rt, $Rn", "", []>, Sched<[]> {
+  bits<5> Rt;
+  bits<5> Rn;
+  let Inst{31-15} = 0b11011001000111110;
+  let Inst{14-12} = op;
+  let Inst{11-10} = 0b11;
+  let Inst{9-5} = Rn;
+  let Inst{4-0} = Rt;
+  let Predicates = [HasGCS];
+}
+def GCSSTR  : GCSSt<"gcsstr",  0b000>;
+def GCSSTTR : GCSSt<"gcssttr", 0b001>;
+
+
 // ARMv8.2-A Dot Product
 let Predicates = [HasDotProd] in {
 defm SDOT : SIMDThreeSameVectorDot<0, 0, "sdot", AArch64sdot>;
@@ -2128,20 +2197,20 @@ defm ST2G  : MemTagStore<0b10, "st2g">;
 defm STZ2G : MemTagStore<0b11, "stz2g">;
 
 def : Pat<(AArch64stg GPR64sp:$Rn, (am_indexeds9s128 GPR64sp:$Rm, simm9s16:$imm)),
-          (STGOffset $Rn, $Rm, $imm)>;
+          (STGi $Rn, $Rm, $imm)>;
 def : Pat<(AArch64stzg GPR64sp:$Rn, (am_indexeds9s128 GPR64sp:$Rm, simm9s16:$imm)),
-          (STZGOffset $Rn, $Rm, $imm)>;
+          (STZGi $Rn, $Rm, $imm)>;
 def : Pat<(AArch64st2g GPR64sp:$Rn, (am_indexeds9s128 GPR64sp:$Rm, simm9s16:$imm)),
-          (ST2GOffset $Rn, $Rm, $imm)>;
+          (ST2Gi $Rn, $Rm, $imm)>;
 def : Pat<(AArch64stz2g GPR64sp:$Rn, (am_indexeds9s128 GPR64sp:$Rm, simm9s16:$imm)),
-          (STZ2GOffset $Rn, $Rm, $imm)>;
+          (STZ2Gi $Rn, $Rm, $imm)>;
 
 defm STGP     : StorePairOffset <0b01, 0, GPR64z, simm7s16, "stgp">;
 def  STGPpre  : StorePairPreIdx <0b01, 0, GPR64z, simm7s16, "stgp">;
 def  STGPpost : StorePairPostIdx<0b01, 0, GPR64z, simm7s16, "stgp">;
 
 def : Pat<(int_aarch64_stg GPR64:$Rt, (am_indexeds9s128 GPR64sp:$Rn, simm9s16:$offset)),
-          (STGOffset GPR64:$Rt, GPR64sp:$Rn,  simm9s16:$offset)>;
+          (STGi GPR64:$Rt, GPR64sp:$Rn,  simm9s16:$offset)>;
 
 def : Pat<(int_aarch64_stgp (am_indexed7s128 GPR64sp:$Rn, simm7s16:$imm), GPR64:$Rt, GPR64:$Rt2),
           (STGPi $Rt, $Rt2, $Rn, $imm)>;
@@ -3326,8 +3395,8 @@ def : InstAlias<"ldrsw $Rt, [$Rn, $offset]",
 
 // A LDR will implicitly zero the rest of the vector, so vector_insert(zeros,
 // load, 0) can use a single load.
-multiclass LoadInsertZeroPatterns<SDPatternOperator LoadOp, ValueType VT, ValueType HVT, ValueType ScalarVT,
-                                  Instruction LoadInst, Instruction UnscaledLoadInst,
+multiclass LoadInsertZeroPatterns<SDPatternOperator LoadOp, ValueType VT, ValueType HVT, ValueType SVT,
+                                  ValueType ScalarVT, Instruction LoadInst, Instruction UnscaledLoadInst,
                                   ComplexPattern Addr, ComplexPattern UnscaledAddr, Operand AddrImm,
                                   SubRegIndex SubReg> {
   // Scaled
@@ -3347,23 +3416,32 @@ multiclass LoadInsertZeroPatterns<SDPatternOperator LoadOp, ValueType VT, ValueT
   def : Pat <(vector_insert (HVT immAllZerosV),
                  (ScalarVT (LoadOp (UnscaledAddr GPR64sp:$Rn, simm9:$offset))), (i64 0)),
              (SUBREG_TO_REG (i64 0), (UnscaledLoadInst GPR64sp:$Rn, simm9:$offset), SubReg)>;
+
+  // SVE patterns
+  def : Pat <(vector_insert (SVT immAllZerosV),
+                 (ScalarVT (LoadOp (Addr GPR64sp:$Rn, AddrImm:$offset))), (i64 0)),
+             (SUBREG_TO_REG (i64 0), (LoadInst GPR64sp:$Rn, AddrImm:$offset), SubReg)>;
+  // Unscaled
+  def : Pat <(vector_insert (SVT immAllZerosV),
+                 (ScalarVT (LoadOp (UnscaledAddr GPR64sp:$Rn, simm9:$offset))), (i64 0)),
+             (SUBREG_TO_REG (i64 0), (UnscaledLoadInst GPR64sp:$Rn, simm9:$offset), SubReg)>;
 }
 
-defm : LoadInsertZeroPatterns<extloadi8,  v16i8,  v8i8,   i32,  LDRBui, LDRBui,
+defm : LoadInsertZeroPatterns<extloadi8,  v16i8,  v8i8,   nxv16i8,  i32,  LDRBui, LDURBi,
                               am_indexed8,  am_unscaled8,  uimm12s1, bsub>;
-defm : LoadInsertZeroPatterns<extloadi16, v8i16,  v4i16,  i32,  LDRHui, LDURHi,
+defm : LoadInsertZeroPatterns<extloadi16, v8i16,  v4i16,  nxv8i16,  i32,  LDRHui, LDURHi,
                               am_indexed16, am_unscaled16, uimm12s2, hsub>;
-defm : LoadInsertZeroPatterns<load,       v4i32,  v2i32,  i32,  LDRSui, LDURSi,
+defm : LoadInsertZeroPatterns<load,       v4i32,  v2i32,  nxv4i32,  i32,  LDRSui, LDURSi,
                               am_indexed32, am_unscaled32, uimm12s4, ssub>;
-defm : LoadInsertZeroPatterns<load,       v2i64,  v1i64,  i64,  LDRDui, LDURDi,
+defm : LoadInsertZeroPatterns<load,       v2i64,  v1i64,  nxv2i64,  i64,  LDRDui, LDURDi,
                               am_indexed64, am_unscaled64, uimm12s8, dsub>;
-defm : LoadInsertZeroPatterns<load,       v8f16,  v4f16,  f16,  LDRHui, LDURHi,
+defm : LoadInsertZeroPatterns<load,       v8f16,  v4f16,  nxv8f16,  f16,  LDRHui, LDURHi,
                               am_indexed16, am_unscaled16, uimm12s2, hsub>;
-defm : LoadInsertZeroPatterns<load,       v8bf16, v4bf16, bf16, LDRHui, LDURHi,
+defm : LoadInsertZeroPatterns<load,       v8bf16, v4bf16, nxv8bf16, bf16, LDRHui, LDURHi,
                               am_indexed16, am_unscaled16, uimm12s2, hsub>;
-defm : LoadInsertZeroPatterns<load,       v4f32,  v2f32,  f32,  LDRSui, LDURSi,
+defm : LoadInsertZeroPatterns<load,       v4f32,  v2f32,  nxv4f32,  f32,  LDRSui, LDURSi,
                               am_indexed32, am_unscaled32, uimm12s4, ssub>;
-defm : LoadInsertZeroPatterns<load,       v2f64,  v1f64,  f64,  LDRDui, LDURDi,
+defm : LoadInsertZeroPatterns<load,       v2f64,  v1f64,  nxv2f64,  f64,  LDRDui, LDURDi,
                               am_indexed64, am_unscaled64, uimm12s8, dsub>;
 
 // Pre-fetch.
diff --git a/llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp b/llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp
index afea2b2f5f754..4c13c095ff69f 100644
--- a/llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp
+++ b/llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp
@@ -237,10 +237,10 @@ static bool isTagStore(const MachineInstr &MI) {
   switch (MI.getOpcode()) {
   default:
     return false;
-  case AArch64::STGOffset:
-  case AArch64::STZGOffset:
-  case AArch64::ST2GOffset:
-  case AArch64::STZ2GOffset:
+  case AArch64::STGi:
+  case AArch64::STZGi:
+  case AArch64::ST2Gi:
+  case AArch64::STZ2Gi:
     return true;
   }
 }
@@ -465,13 +465,13 @@ static unsigned getPreIndexedOpcode(unsigned Opc) {
     return AArch64::STPWpre;
   case AArch64::STPXi:
     return AArch64::STPXpre;
-  case AArch64::STGOffset:
+  case AArch64::STGi:
     return AArch64::STGPreIndex;
-  case AArch64::STZGOffset:
+  case AArch64::STZGi:
     return AArch64::STZGPreIndex;
-  case AArch64::ST2GOffset:
+  case AArch64::ST2Gi:
     return AArch64::ST2GPreIndex;
-  case AArch64::STZ2GOffset:
+  case AArch64::STZ2Gi:
     return AArch64::STZ2GPreIndex;
   case AArch64::STGPi:
     return AArch64::STGPpre;
@@ -544,13 +544,13 @@ static unsigned getPostIndexedOpcode(unsigned Opc) {
     return AArch64::STPWpost;
   case AArch64::STPXi:
     return AArch64::STPXpost;
-  case AArch64::STGOffset:
+  case AArch64::STGi:
     return AArch64::STGPostIndex;
-  case AArch64::STZGOffset:
+  case AArch64::STZGi:
     return AArch64::STZGPostIndex;
-  case AArch64::ST2GOffset:
+  case AArch64::ST2Gi:
     return AArch64::ST2GPostIndex;
-  case AArch64::STZ2GOffset:
+  case AArch64::STZ2Gi:
     return AArch64::STZ2GPostIndex;
   case AArch64::STGPi:
     return AArch64::STGPpost;
@@ -681,10 +681,10 @@ static bool isMergeableLdStUpdate(MachineInstr &MI) {
   case AArch64::LDRWui:
   case AArch64::LDRHHui:
   case AArch64::LDRBBui:
-  case AArch64::STGOffset:
-  case AArch64::STZGOffset:
-  case AArch64::ST2GOffset:
-  case AArch64::STZ2GOffset:
+  case AArch64::STGi:
+  case AArch64::STZGi:
+  case AArch64::ST2Gi:
+  case AArch64::STZ2Gi:
   case AArch64::STGPi:
   // Unscaled instructions.
   case AArch64::STURSi:
diff --git a/llvm/lib/Target/AArch64/AArch64PerfectShuffle.h b/llvm/lib/Target/AArch64/AArch64PerfectShuffle.h
index 4555f1a3ebb08..5846fd454b654 100644
--- a/llvm/lib/Target/AArch64/AArch64PerfectShuffle.h
+++ b/llvm/lib/Target/AArch64/AArch64PerfectShuffle.h
@@ -6590,11 +6590,11 @@ static unsigned getPerfectShuffleCost(llvm::ArrayRef<int> M) {
   assert(M.size() == 4 && "Expected a 4 entry perfect shuffle");
 
   // Special case zero-cost nop copies, from either LHS or RHS.
-  if (llvm::all_of(llvm::enumerate(M), [](auto &E) {
+  if (llvm::all_of(llvm::enumerate(M), [](const auto &E) {
         return E.value() < 0 || E.value() == (int)E.index();
       }))
     return 0;
-  if (llvm::all_of(llvm::enumerate(M), [](auto &E) {
+  if (llvm::all_of(llvm::enumerate(M), [](const auto &E) {
         return E.value() < 0 || E.value() == (int)E.index() + 4;
       }))
     return 0;
diff --git a/llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td b/llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td
index f87b9e38fbaae..278f74ef341f8 100644
--- a/llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td
+++ b/llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td
@@ -401,14 +401,17 @@ def AArch64sub_m1 : PatFrags<(ops node:$pred, node:$op1, node:$op2),
                               (sub node:$op1, (vselect node:$pred, node:$op2, (SVEDup0)))]>;
 def AArch64mla_m1 : PatFrags<(ops node:$pred, node:$op1, node:$op2, node:$op3),
                              [(int_aarch64_sve_mla node:$pred, node:$op1, node:$op2, node:$op3),
-                              (add node:$op1, (AArch64mul_p_oneuse node:$pred, node:$op2, node:$op3)),
                               // add(a, select(mask, mul(b, c), splat(0))) -> mla(a, mask, b, c)
                               (add node:$op1, (vselect node:$pred, (AArch64mul_p_oneuse (SVEAllActive), node:$op2, node:$op3), (SVEDup0)))]>;
+// pattern for generating pseudo for MLA_ZPmZZ/MAD_ZPmZZ
+def AArch64mla_p : PatFrags<(ops node:$pred, node:$op1, node:$op2, node:$op3),
+                            [(add node:$op1, (AArch64mul_p_oneuse node:$pred, node:$op2, node:$op3))]>;
 def AArch64mls_m1 : PatFrags<(ops node:$pred, node:$op1, node:$op2, node:$op3),
                              [(int_aarch64_sve_mls node:$pred, node:$op1, node:$op2, node:$op3),
-                              (sub node:$op1, (AArch64mul_p_oneuse node:$pred, node:$op2, node:$op3)),
                               // sub(a, select(mask, mul(b, c), splat(0))) -> mls(a, mask, b, c)
                               (sub node:$op1, (vselect node:$pred, (AArch64mul_p_oneuse (SVEAllActive), node:$op2, node:$op3), (SVEDup0)))]>;
+def AArch64mls_p : PatFrags<(ops node:$pred, node:$op1, node:$op2, node:$op3),
+                             [(sub node:$op1, (AArch64mul_p_oneuse node:$pred, node:$op2, node:$op3))]>;
 def AArch64eor3 : PatFrags<(ops node:$op1, node:$op2, node:$op3),
                            [(int_aarch64_sve_eor3 node:$op1, node:$op2, node:$op3),
                             (xor node:$op1, (xor node:$op2, node:$op3))]>;
@@ -483,10 +486,13 @@ let Predicates = [HasSVEorSME] in {
   defm SQSUB_ZI : sve_int_arith_imm0<0b110, "sqsub", ssubsat>;
   defm UQSUB_ZI : sve_int_arith_imm0<0b111, "uqsub", usubsat>;
 
-  defm MAD_ZPmZZ : sve_int_mladdsub_vvv_pred<0b0, "mad", int_aarch64_sve_mad>;
-  defm MSB_ZPmZZ : sve_int_mladdsub_vvv_pred<0b1, "msb", int_aarch64_sve_msb>;
-  defm MLA_ZPmZZ : sve_int_mlas_vvv_pred<0b0, "mla", AArch64mla_m1>;
-  defm MLS_ZPmZZ : sve_int_mlas_vvv_pred<0b1, "mls", AArch64mls_m1>;
+  defm MAD_ZPmZZ : sve_int_mladdsub_vvv_pred<0b0, "mad", int_aarch64_sve_mad, "MLA_ZPmZZ", /*isReverseInstr*/ 1>;
+  defm MSB_ZPmZZ : sve_int_mladdsub_vvv_pred<0b1, "msb", int_aarch64_sve_msb, "MLS_ZPmZZ", /*isReverseInstr*/ 1>;
+  defm MLA_ZPmZZ : sve_int_mlas_vvv_pred<0b0, "mla", AArch64mla_m1, "MLA_ZPZZZ", "MAD_ZPmZZ">;
+  defm MLS_ZPmZZ : sve_int_mlas_vvv_pred<0b1, "mls", AArch64mls_m1, "MLS_ZPZZZ", "MSB_ZPmZZ">;
+
+  defm MLA_ZPZZZ : sve_int_3op_p_mladdsub<AArch64mla_p>;
+  defm MLS_ZPZZZ : sve_int_3op_p_mladdsub<AArch64mls_p>;
 
   // SVE predicated integer reductions.
   defm SADDV_VPZ : sve_int_reduce_0_saddv<0b000, "saddv", AArch64saddv_p>;
diff --git a/llvm/lib/Target/AArch64/AArch64SchedNeoverseN2.td b/llvm/lib/Target/AArch64/AArch64SchedNeoverseN2.td
index d9d5c2be03092..ee5c9280095f6 100644
--- a/llvm/lib/Target/AArch64/AArch64SchedNeoverseN2.td
+++ b/llvm/lib/Target/AArch64/AArch64SchedNeoverseN2.td
@@ -609,6 +609,16 @@ def N2Write_11cyc_9L01_9S_9V : SchedWriteRes<[N2UnitL01, N2UnitL01, N2UnitL01,
   let NumMicroOps = 27;
 }
 
+//===----------------------------------------------------------------------===//
+// Define types for arithmetic and logical ops with short shifts
+def N2Write_Arith : SchedWriteVariant<[
+                      SchedVar<NeoverseCheapLSL, [N2Write_1cyc_1I]>,
+                      SchedVar<NoSchedPred,      [N2Write_2cyc_1M]>]>;
+
+def N2Write_Logical: SchedWriteVariant<[
+                       SchedVar<NeoverseNoLSL, [N2Write_1cyc_1I]>,
+                       SchedVar<NoSchedPred,   [N2Write_2cyc_1M]>]>;
+
 // Miscellaneous
 // -----------------------------------------------------------------------------
 
@@ -636,9 +646,20 @@ def : InstRW<[N2Write_1cyc_1B_1S], (instrs BL, BLR)>;
 def : SchedAlias<WriteI,     N2Write_1cyc_1I>;
 
 // ALU, extend and shift
-def : SchedAlias<WriteISReg, N2Write_2cyc_1M>;
 def : SchedAlias<WriteIEReg, N2Write_2cyc_1M>;
 
+// Arithmetic, LSL shift, shift <= 4
+// Arithmetic, flagset, LSL shift, shift <= 4
+// Arithmetic, LSR/ASR/ROR shift or LSL shift > 4
+def : SchedAlias<WriteISReg, N2Write_Arith>;
+
+// Logical, shift, no flagset
+def : InstRW<[N2Write_1cyc_1I],
+             (instregex "^(AND|BIC|EON|EOR|ORN|ORR)[WX]rs$")>;
+
+// Logical, shift, flagset
+def : InstRW<[N2Write_Logical], (instregex "^(AND|BIC)S[WX]rs$")>;
+
 // Arithmetic, immediate to logical address tag
 def : InstRW<[N2Write_2cyc_1M], (instrs ADDG, SUBG)>;
 
@@ -758,8 +779,8 @@ def : InstRW<[N2Write_1cyc_1L01_1D_1I], (instrs STGPreIndex, STGPostIndex,
 // Store allocation tag to two granules, zeroing, signed offset
 // Store allocation tag and reg pair to memory, signed offset
 // Store multiple allocation tags
-def : InstRW<[N2Write_1cyc_1L01_1D], (instrs STGOffset, ST2GOffset, STZGOffset,
-                                             STZ2GOffset, STGPi, STGM, STZGM)>;
+def : InstRW<[N2Write_1cyc_1L01_1D], (instrs STGi, ST2Gi, STZGi,
+                                             STZ2Gi, STGPi, STGM, STZGM)>;
 
 // FP data processing instructions
 // -----------------------------------------------------------------------------
diff --git a/llvm/lib/Target/AArch64/AArch64SchedPredNeoverse.td b/llvm/lib/Target/AArch64/AArch64SchedPredNeoverse.td
new file mode 100644
index 0000000000000..80a300236716f
--- /dev/null
+++ b/llvm/lib/Target/AArch64/AArch64SchedPredNeoverse.td
@@ -0,0 +1,30 @@
+//===- AArch64SchedPredNeoverse.td - AArch64 Sched Preds -----*- tablegen -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines scheduling predicate definitions that are used by the
+// AArch64 Neoverse processors.
+//
+//===----------------------------------------------------------------------===//
+
+// Auxiliary predicates.
+
+// Check for LSL shift <= 4
+def NeoverseCheapLSL : MCSchedPredicate<
+                         CheckAll<
+                           [CheckShiftLSL,
+                            CheckAny<
+                              [CheckShiftBy0,
+                               CheckShiftBy1,
+                               CheckShiftBy2,
+                               CheckShiftBy3,
+                               CheckShiftBy4]>]>>;
+
+// Check for LSL shift == 0
+def NeoverseNoLSL : MCSchedPredicate<
+                      CheckAll<[CheckShiftLSL,
+                                CheckShiftBy0]>>;
diff --git a/llvm/lib/Target/AArch64/AArch64StackTaggingPreRA.cpp b/llvm/lib/Target/AArch64/AArch64StackTaggingPreRA.cpp
index 2cbac9783bbd6..41cd405c891ee 100644
--- a/llvm/lib/Target/AArch64/AArch64StackTaggingPreRA.cpp
+++ b/llvm/lib/Target/AArch64/AArch64StackTaggingPreRA.cpp
@@ -276,8 +276,8 @@ std::optional<int> AArch64StackTaggingPreRA::findFirstSlotCandidate() {
       Register UseReg = WorkList.pop_back_val();
       for (auto &UseI : MRI->use_instructions(UseReg)) {
         unsigned Opcode = UseI.getOpcode();
-        if (Opcode == AArch64::STGOffset || Opcode == AArch64::ST2GOffset ||
-            Opcode == AArch64::STZGOffset || Opcode == AArch64::STZ2GOffset ||
+        if (Opcode == AArch64::STGi || Opcode == AArch64::ST2Gi ||
+            Opcode == AArch64::STZGi || Opcode == AArch64::STZ2Gi ||
             Opcode == AArch64::STGPi || Opcode == AArch64::STGloop ||
             Opcode == AArch64::STZGloop || Opcode == AArch64::STGloop_wback ||
             Opcode == AArch64::STZGloop_wback)
diff --git a/llvm/lib/Target/AArch64/AArch64SystemOperands.td b/llvm/lib/Target/AArch64/AArch64SystemOperands.td
index c7f404463c587..cbe02f4e15b9a 100644
--- a/llvm/lib/Target/AArch64/AArch64SystemOperands.td
+++ b/llvm/lib/Target/AArch64/AArch64SystemOperands.td
@@ -1761,6 +1761,19 @@ let Requires = [{ {AArch64::FeatureNMI} }] in {
   def : ROSysReg<"ICC_NMIAR1_EL1", 0b11, 0b000, 0b1100, 0b1001, 0b101>; // FEAT_GICv3_NMI
 }
 
+// v9.4a Guarded Control Stack Extension (GCS)
+//                            Op0   Op1    CRn     CRm     Op2
+def : RWSysReg<"GCSCR_EL1",   0b11, 0b000, 0b0010, 0b0101, 0b000>;
+def : RWSysReg<"GCSPR_EL1",   0b11, 0b000, 0b0010, 0b0101, 0b001>;
+def : RWSysReg<"GCSCRE0_EL1", 0b11, 0b000, 0b0010, 0b0101, 0b010>;
+def : RWSysReg<"GCSPR_EL0",   0b11, 0b011, 0b0010, 0b0101, 0b001>;
+def : RWSysReg<"GCSCR_EL2",   0b11, 0b100, 0b0010, 0b0101, 0b000>;
+def : RWSysReg<"GCSPR_EL2",   0b11, 0b100, 0b0010, 0b0101, 0b001>;
+def : RWSysReg<"GCSCR_EL12",  0b11, 0b101, 0b0010, 0b0101, 0b000>;
+def : RWSysReg<"GCSPR_EL12",  0b11, 0b101, 0b0010, 0b0101, 0b001>;
+def : RWSysReg<"GCSCR_EL3",   0b11, 0b110, 0b0010, 0b0101, 0b000>;
+def : RWSysReg<"GCSPR_EL3",   0b11, 0b110, 0b0010, 0b0101, 0b001>;
+
 // v8.9a/v9.4a Memory Attribute Index Enhancement (FEAT_AIE)
 //                            Op0   Op1    CRn     CRm     Op2
 def : RWSysReg<"AMAIR2_EL1",  0b11, 0b000, 0b1010, 0b0011, 0b001>;
diff --git a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp b/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
index 5cd00584db64a..3e0f419ca3f94 100644
--- a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
+++ b/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
@@ -1619,9 +1619,17 @@ AArch64TTIImpl::instCombineIntrinsic(InstCombiner &IC,
   case Intrinsic::aarch64_sve_fadd:
   case Intrinsic::aarch64_sve_add:
     return instCombineSVEVectorAdd(IC, II);
+  case Intrinsic::aarch64_sve_fadd_u:
+    return instCombineSVEVectorFuseMulAddSub<Intrinsic::aarch64_sve_fmul_u,
+                                             Intrinsic::aarch64_sve_fmla_u>(
+        IC, II, true);
   case Intrinsic::aarch64_sve_fsub:
   case Intrinsic::aarch64_sve_sub:
     return instCombineSVEVectorSub(IC, II);
+  case Intrinsic::aarch64_sve_fsub_u:
+    return instCombineSVEVectorFuseMulAddSub<Intrinsic::aarch64_sve_fmul_u,
+                                             Intrinsic::aarch64_sve_fmls_u>(
+        IC, II, true);
   case Intrinsic::aarch64_sve_tbl:
     return instCombineSVETBL(IC, II);
   case Intrinsic::aarch64_sve_uunpkhi:
@@ -3211,13 +3219,19 @@ InstructionCost AArch64TTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
 
   Kind = improveShuffleKindFromMask(Kind, Mask);
 
-  // Check for broadcast loads.
-  if (Kind == TTI::SK_Broadcast) {
+  // Check for broadcast loads, which are supported by the LD1R instruction.
+  // In terms of code-size, the shuffle vector is free when a load + dup get
+  // folded into a LD1R. That's what we check and return here. For performance
+  // and reciprocal throughput, a LD1R is not completely free. In this case, we
+  // return the cost for the broadcast below (i.e. 1 for most/all types), so
+  // that we model the load + dup sequence slightly higher because LD1R is a
+  // high latency instruction.
+  if (CostKind == TTI::TCK_CodeSize && Kind == TTI::SK_Broadcast) {
     bool IsLoad = !Args.empty() && isa<LoadInst>(Args[0]);
     if (IsLoad && LT.second.isVector() &&
         isLegalBroadcastLoad(Tp->getElementType(),
                              LT.second.getVectorElementCount()))
-      return 0; // broadcast is handled by ld1r
+      return 0;
   }
 
   // If we have 4 elements for the shuffle and a Mask, get the cost straight
@@ -3239,6 +3253,8 @@ InstructionCost AArch64TTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
         {TTI::SK_Broadcast, MVT::v2i32, 1},
         {TTI::SK_Broadcast, MVT::v4i32, 1},
         {TTI::SK_Broadcast, MVT::v2i64, 1},
+        {TTI::SK_Broadcast, MVT::v4f16, 1},
+        {TTI::SK_Broadcast, MVT::v8f16, 1},
         {TTI::SK_Broadcast, MVT::v2f32, 1},
         {TTI::SK_Broadcast, MVT::v4f32, 1},
         {TTI::SK_Broadcast, MVT::v2f64, 1},
@@ -3251,6 +3267,8 @@ InstructionCost AArch64TTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
         {TTI::SK_Transpose, MVT::v2i32, 1},
         {TTI::SK_Transpose, MVT::v4i32, 1},
         {TTI::SK_Transpose, MVT::v2i64, 1},
+        {TTI::SK_Transpose, MVT::v4f16, 1},
+        {TTI::SK_Transpose, MVT::v8f16, 1},
         {TTI::SK_Transpose, MVT::v2f32, 1},
         {TTI::SK_Transpose, MVT::v4f32, 1},
         {TTI::SK_Transpose, MVT::v2f64, 1},
diff --git a/llvm/lib/Target/AArch64/AsmParser/AArch64AsmParser.cpp b/llvm/lib/Target/AArch64/AsmParser/AArch64AsmParser.cpp
index 4f2833c1c4230..d5162d39ce43c 100644
--- a/llvm/lib/Target/AArch64/AsmParser/AArch64AsmParser.cpp
+++ b/llvm/lib/Target/AArch64/AsmParser/AArch64AsmParser.cpp
@@ -3677,6 +3677,7 @@ static const struct Extension {
     {"ite", {AArch64::FeatureITE}},
     {"cssc", {AArch64::FeatureCSSC}},
     {"rcpc3", {AArch64::FeatureRCPC3}},
+    {"gcs", {AArch64::FeatureGCS}},
     // FIXME: Unsupported extensions
     {"lor", {}},
     {"rdma", {}},
@@ -4841,11 +4842,6 @@ bool AArch64AsmParser::parseOperand(OperandVector &Operands, bool isCondCode,
     if (!parseOptionalMulOperand(Operands))
       return false;
 
-    // If this is an "smstart" or "smstop" instruction, parse its special
-    // keyword operand as an identifier.
-    if (Mnemonic == "smstart" || Mnemonic == "smstop")
-      return parseKeywordOperand(Operands);
-
     // This could be an optional "shift" or "extend" operand.
     OperandMatchResultTy GotShift = tryParseOptionalShiftExtend(Operands);
     // We can only continue if no tokens were eaten.
@@ -4854,7 +4850,8 @@ bool AArch64AsmParser::parseOperand(OperandVector &Operands, bool isCondCode,
 
     // If this is a two-word mnemonic, parse its special keyword
     // operand as an identifier.
-    if (Mnemonic == "brb")
+    if (Mnemonic == "brb" || Mnemonic == "smstart" || Mnemonic == "smstop" ||
+        Mnemonic == "gcsb")
       return parseKeywordOperand(Operands);
 
     // This was not a register so parse other operands that start with an
@@ -7621,61 +7618,21 @@ extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeAArch64AsmParser() {
 unsigned AArch64AsmParser::validateTargetOperandClass(MCParsedAsmOperand &AsmOp,
                                                       unsigned Kind) {
   AArch64Operand &Op = static_cast<AArch64Operand &>(AsmOp);
-  // If the kind is a token for a literal immediate, check if our asm
-  // operand matches. This is for InstAliases which have a fixed-value
-  // immediate in the syntax.
-  int64_t ExpectedVal;
+
+  auto MatchesOpImmediate = [&](int64_t ExpectedVal) -> MatchResultTy {
+    if (!Op.isImm())
+      return Match_InvalidOperand;
+    const MCConstantExpr *CE = dyn_cast<MCConstantExpr>(Op.getImm());
+    if (!CE)
+      return Match_InvalidOperand;
+    if (CE->getValue() == ExpectedVal)
+      return Match_Success;
+    return Match_InvalidOperand;
+  };
+
   switch (Kind) {
   default:
     return Match_InvalidOperand;
-  case MCK__HASH_0:
-    ExpectedVal = 0;
-    break;
-  case MCK__HASH_1:
-    ExpectedVal = 1;
-    break;
-  case MCK__HASH_12:
-    ExpectedVal = 12;
-    break;
-  case MCK__HASH_16:
-    ExpectedVal = 16;
-    break;
-  case MCK__HASH_2:
-    ExpectedVal = 2;
-    break;
-  case MCK__HASH_24:
-    ExpectedVal = 24;
-    break;
-  case MCK__HASH_3:
-    ExpectedVal = 3;
-    break;
-  case MCK__HASH_32:
-    ExpectedVal = 32;
-    break;
-  case MCK__HASH_4:
-    ExpectedVal = 4;
-    break;
-  case MCK__HASH_48:
-    ExpectedVal = 48;
-    break;
-  case MCK__HASH_6:
-    ExpectedVal = 6;
-    break;
-  case MCK__HASH_64:
-    ExpectedVal = 64;
-    break;
-  case MCK__HASH_8:
-    ExpectedVal = 8;
-    break;
-  case MCK__HASH__MINUS_4:
-    ExpectedVal = -4;
-    break;
-  case MCK__HASH__MINUS_8:
-    ExpectedVal = -8;
-    break;
-  case MCK__HASH__MINUS_16:
-    ExpectedVal = -16;
-    break;
   case MCK_MPR:
     // If the Kind is a token for the MPR register class which has the "za"
     // register (SME accumulator array), check if the asm is a literal "za"
@@ -7684,15 +7641,36 @@ unsigned AArch64AsmParser::validateTargetOperandClass(MCParsedAsmOperand &AsmOp,
     if (Op.isTokenEqual("za"))
       return Match_Success;
     return Match_InvalidOperand;
+
+    // If the kind is a token for a literal immediate, check if our asm
+    // operand matches. This is for InstAliases which have a fixed-value
+    // immediate in the syntax.
+#define MATCH_HASH(N)                                                          \
+  case MCK__HASH_##N:                                                          \
+    return MatchesOpImmediate(N);
+    MATCH_HASH(0)
+    MATCH_HASH(1)
+    MATCH_HASH(2)
+    MATCH_HASH(3)
+    MATCH_HASH(4)
+    MATCH_HASH(6)
+    MATCH_HASH(8)
+    MATCH_HASH(12)
+    MATCH_HASH(16)
+    MATCH_HASH(24)
+    MATCH_HASH(32)
+    MATCH_HASH(40)
+    MATCH_HASH(48)
+    MATCH_HASH(64)
+#undef MATCH_HASH
+#define MATCH_HASH_MINUS(N)                                                    \
+  case MCK__HASH__MINUS_##N:                                                   \
+    return MatchesOpImmediate(-N);
+    MATCH_HASH_MINUS(4)
+    MATCH_HASH_MINUS(8)
+    MATCH_HASH_MINUS(16)
+#undef MATCH_HASH_MINUS
   }
-  if (!Op.isImm())
-    return Match_InvalidOperand;
-  const MCConstantExpr *CE = dyn_cast<MCConstantExpr>(Op.getImm());
-  if (!CE)
-    return Match_InvalidOperand;
-  if (CE->getValue() == ExpectedVal)
-    return Match_Success;
-  return Match_InvalidOperand;
 }
 
 OperandMatchResultTy
diff --git a/llvm/lib/Target/AArch64/MCTargetDesc/AArch64InstPrinter.cpp b/llvm/lib/Target/AArch64/MCTargetDesc/AArch64InstPrinter.cpp
index 7c32c6f7361f8..2983e9a9be928 100644
--- a/llvm/lib/Target/AArch64/MCTargetDesc/AArch64InstPrinter.cpp
+++ b/llvm/lib/Target/AArch64/MCTargetDesc/AArch64InstPrinter.cpp
@@ -282,6 +282,23 @@ void AArch64InstPrinter::printInst(const MCInst *MI, uint64_t Address,
     return;
   }
 
+  auto PrintMovImm = [&](uint64_t Value, int RegWidth) {
+    int64_t SExtVal = SignExtend64(Value, RegWidth);
+    O << "\tmov\t";
+    printRegName(O, MI->getOperand(0).getReg());
+    O << ", " << markup("<imm:") << "#"
+      << formatImm(SExtVal) << markup(">");
+    if (CommentStream) {
+      // Do the opposite to that used for instruction operands.
+      if (getPrintImmHex())
+        *CommentStream << '=' << formatDec(SExtVal) << '\n';
+      else {
+        uint64_t Mask = maskTrailingOnes<uint64_t>(RegWidth);
+        *CommentStream << '=' << formatHex(SExtVal & Mask) << '\n';
+      }
+    }
+  };
+
   // MOVZ, MOVN and "ORR wzr, #imm" instructions are aliases for MOV, but their
   // domains overlap so they need to be prioritized. The chain is "MOVZ lsl #0 >
   // MOVZ lsl #N > MOVN lsl #0 > MOVN lsl #N > ORR". The highest instruction
@@ -295,10 +312,7 @@ void AArch64InstPrinter::printInst(const MCInst *MI, uint64_t Address,
 
     if (AArch64_AM::isMOVZMovAlias(Value, Shift,
                                    Opcode == AArch64::MOVZXi ? 64 : 32)) {
-      O << "\tmov\t";
-      printRegName(O, MI->getOperand(0).getReg());
-      O << ", " << markup("<imm:") << "#"
-        << formatImm(SignExtend64(Value, RegWidth)) << markup(">");
+      PrintMovImm(Value, RegWidth);
       return;
     }
   }
@@ -312,10 +326,7 @@ void AArch64InstPrinter::printInst(const MCInst *MI, uint64_t Address,
       Value = Value & 0xffffffff;
 
     if (AArch64_AM::isMOVNMovAlias(Value, Shift, RegWidth)) {
-      O << "\tmov\t";
-      printRegName(O, MI->getOperand(0).getReg());
-      O << ", " << markup("<imm:") << "#"
-        << formatImm(SignExtend64(Value, RegWidth)) << markup(">");
+      PrintMovImm(Value, RegWidth);
       return;
     }
   }
@@ -328,10 +339,7 @@ void AArch64InstPrinter::printInst(const MCInst *MI, uint64_t Address,
     uint64_t Value = AArch64_AM::decodeLogicalImmediate(
         MI->getOperand(2).getImm(), RegWidth);
     if (!AArch64_AM::isAnyMOVWMovAlias(Value, RegWidth)) {
-      O << "\tmov\t";
-      printRegName(O, MI->getOperand(0).getReg());
-      O << ", " << markup("<imm:") << "#"
-        << formatImm(SignExtend64(Value, RegWidth)) << markup(">");
+      PrintMovImm(Value, RegWidth);
       return;
     }
   }
diff --git a/llvm/lib/Target/AArch64/SVEInstrFormats.td b/llvm/lib/Target/AArch64/SVEInstrFormats.td
index 9c2913122f28b..4c97ae88f192a 100644
--- a/llvm/lib/Target/AArch64/SVEInstrFormats.td
+++ b/llvm/lib/Target/AArch64/SVEInstrFormats.td
@@ -47,6 +47,10 @@ def SDT_AArch64Setcc : SDTypeProfile<1, 4, [
 ]>;
 
 def AArch64setcc_z : SDNode<"AArch64ISD::SETCC_MERGE_ZERO", SDT_AArch64Setcc>;
+def AArch64setcc_z_oneuse : PatFrag<(ops node:$pg, node:$op1, node:$op2, node:$cc),
+                                    (AArch64setcc_z node:$pg, node:$op1, node:$op2, node:$cc), [{
+  return N->hasOneUse();
+}]>;
 
 def SVEPatternOperand : AsmOperandClass {
   let Name = "SVEPattern";
@@ -3123,11 +3127,16 @@ class sve_int_mladdsub_vvv_pred<bits<2> sz8_64, bits<1> opc, string asm,
   let hasSideEffects = 0;
 }
 
-multiclass sve_int_mladdsub_vvv_pred<bits<1> opc, string asm, SDPatternOperator op> {
-  def _B : sve_int_mladdsub_vvv_pred<0b00, opc, asm, ZPR8>;
-  def _H : sve_int_mladdsub_vvv_pred<0b01, opc, asm, ZPR16>;
-  def _S : sve_int_mladdsub_vvv_pred<0b10, opc, asm, ZPR32>;
-  def _D : sve_int_mladdsub_vvv_pred<0b11, opc, asm, ZPR64>;
+multiclass sve_int_mladdsub_vvv_pred<bits<1> opc, string asm, SDPatternOperator op,
+                                     string revname, bit isReverseInstr=0> {
+  def _B : sve_int_mladdsub_vvv_pred<0b00, opc, asm, ZPR8>,
+           SVEInstr2Rev<NAME # _B, revname # _B, isReverseInstr>;
+  def _H : sve_int_mladdsub_vvv_pred<0b01, opc, asm, ZPR16>,
+           SVEInstr2Rev<NAME # _H, revname # _H, isReverseInstr>;
+  def _S : sve_int_mladdsub_vvv_pred<0b10, opc, asm, ZPR32>,
+           SVEInstr2Rev<NAME # _S, revname # _S, isReverseInstr>;
+  def _D : sve_int_mladdsub_vvv_pred<0b11, opc, asm, ZPR64>,
+           SVEInstr2Rev<NAME # _D, revname # _D, isReverseInstr>;
 
   def : SVE_4_Op_Pat<nxv16i8, op, nxv16i1, nxv16i8, nxv16i8, nxv16i8, !cast<Instruction>(NAME # _B)>;
   def : SVE_4_Op_Pat<nxv8i16, op, nxv8i1, nxv8i16, nxv8i16, nxv8i16, !cast<Instruction>(NAME # _H)>;
@@ -3156,16 +3165,21 @@ class sve_int_mlas_vvv_pred<bits<2> sz8_64, bits<1> opc, string asm,
   let Inst{4-0}   = Zda;
 
   let Constraints = "$Zda = $_Zda";
-  let DestructiveInstType = DestructiveOther;
+  let DestructiveInstType = DestructiveTernaryCommWithRev;
   let ElementSize = zprty.ElementSize;
   let hasSideEffects = 0;
 }
 
-multiclass sve_int_mlas_vvv_pred<bits<1> opc, string asm, SDPatternOperator op> {
-  def _B : sve_int_mlas_vvv_pred<0b00, opc, asm, ZPR8>;
-  def _H : sve_int_mlas_vvv_pred<0b01, opc, asm, ZPR16>;
-  def _S : sve_int_mlas_vvv_pred<0b10, opc, asm, ZPR32>;
-  def _D : sve_int_mlas_vvv_pred<0b11, opc, asm, ZPR64>;
+multiclass sve_int_mlas_vvv_pred<bits<1> opc, string asm, SDPatternOperator op,
+                                 string Ps, string revname, bit isReverseInstr=0> {
+  def _B : sve_int_mlas_vvv_pred<0b00, opc, asm, ZPR8>,
+           SVEPseudo2Instr<Ps # _B, 1>, SVEInstr2Rev<NAME # _B, revname # _B, isReverseInstr>;
+  def _H : sve_int_mlas_vvv_pred<0b01, opc, asm, ZPR16>,
+           SVEPseudo2Instr<Ps # _H, 1>, SVEInstr2Rev<NAME # _H, revname # _H, isReverseInstr>;
+  def _S : sve_int_mlas_vvv_pred<0b10, opc, asm, ZPR32>,
+           SVEPseudo2Instr<Ps # _S, 1>, SVEInstr2Rev<NAME # _S, revname # _S, isReverseInstr>;
+  def _D : sve_int_mlas_vvv_pred<0b11, opc, asm, ZPR64>,
+           SVEPseudo2Instr<Ps # _D, 1>, SVEInstr2Rev<NAME # _D, revname # _D, isReverseInstr>;
 
   def : SVE_4_Op_Pat<nxv16i8, op, nxv16i1, nxv16i8, nxv16i8, nxv16i8, !cast<Instruction>(NAME # _B)>;
   def : SVE_4_Op_Pat<nxv8i16, op, nxv8i1, nxv8i16, nxv8i16, nxv8i16, !cast<Instruction>(NAME # _H)>;
@@ -3173,6 +3187,19 @@ multiclass sve_int_mlas_vvv_pred<bits<1> opc, string asm, SDPatternOperator op>
   def : SVE_4_Op_Pat<nxv2i64, op, nxv2i1, nxv2i64, nxv2i64, nxv2i64, !cast<Instruction>(NAME # _D)>;
 }
 
+//class for generating pseudo for SVE MLA/MAD/MLS/MSB
+multiclass sve_int_3op_p_mladdsub<SDPatternOperator op> {
+  def _UNDEF_B : PredThreeOpPseudo<NAME # _B, ZPR8,  FalseLanesUndef>;
+  def _UNDEF_H : PredThreeOpPseudo<NAME # _H, ZPR16, FalseLanesUndef>;
+  def _UNDEF_S : PredThreeOpPseudo<NAME # _S, ZPR32, FalseLanesUndef>;
+  def _UNDEF_D : PredThreeOpPseudo<NAME # _D, ZPR64, FalseLanesUndef>;
+
+  def : SVE_4_Op_Pat<nxv16i8, op, nxv16i1, nxv16i8, nxv16i8, nxv16i8, !cast<Instruction>(NAME # _UNDEF_B)>;
+  def : SVE_4_Op_Pat<nxv8i16, op, nxv8i1,  nxv8i16, nxv8i16, nxv8i16, !cast<Instruction>(NAME # _UNDEF_H)>;
+  def : SVE_4_Op_Pat<nxv4i32, op, nxv4i1,  nxv4i32, nxv4i32, nxv4i32, !cast<Instruction>(NAME # _UNDEF_S)>;
+  def : SVE_4_Op_Pat<nxv2i64, op, nxv2i1,  nxv2i64, nxv2i64, nxv2i64, !cast<Instruction>(NAME # _UNDEF_D)>;
+}
+
 //===----------------------------------------------------------------------===//
 // SVE2 Integer Multiply-Add - Unpredicated Group
 //===----------------------------------------------------------------------===//
@@ -5005,9 +5032,9 @@ multiclass SVE_SETCC_Pat<CondCode cc, CondCode invcc, ValueType predvt,
             (cmp $Op1, $Op2, $Op3)>;
   def : Pat<(predvt (AArch64setcc_z predvt:$Op1, intvt:$Op2, intvt:$Op3, invcc)),
             (cmp $Op1, $Op3, $Op2)>;
-  def : Pat<(predvt (and predvt:$Pg, (AArch64setcc_z (predvt (AArch64ptrue 31)), intvt:$Op2, intvt:$Op3, cc))),
+  def : Pat<(predvt (and predvt:$Pg, (AArch64setcc_z_oneuse (predvt (AArch64ptrue 31)), intvt:$Op2, intvt:$Op3, cc))),
             (cmp $Pg, $Op2, $Op3)>;
-  def : Pat<(predvt (and predvt:$Pg, (AArch64setcc_z (predvt (AArch64ptrue 31)), intvt:$Op2, intvt:$Op3, invcc))),
+  def : Pat<(predvt (and predvt:$Pg, (AArch64setcc_z_oneuse (predvt (AArch64ptrue 31)), intvt:$Op2, intvt:$Op3, invcc))),
             (cmp $Pg, $Op3, $Op2)>;
 }
 
@@ -5017,9 +5044,9 @@ multiclass SVE_SETCC_Pat_With_Zero<CondCode cc, CondCode invcc, ValueType predvt
             (cmp $Op1, $Op2)>;
   def : Pat<(predvt (AArch64setcc_z predvt:$Op1, (SVEDup0), intvt:$Op2, invcc)),
             (cmp $Op1, $Op2)>;
-  def : Pat<(predvt (and predvt:$Pg, (AArch64setcc_z (predvt (AArch64ptrue 31)), intvt:$Op1, (SVEDup0), cc))),
+  def : Pat<(predvt (and predvt:$Pg, (AArch64setcc_z_oneuse (predvt (AArch64ptrue 31)), intvt:$Op1, (SVEDup0), cc))),
             (cmp $Pg, $Op1)>;
-  def : Pat<(predvt (and predvt:$Pg, (AArch64setcc_z (predvt (AArch64ptrue 31)), (SVEDup0), intvt:$Op1, invcc))),
+  def : Pat<(predvt (and predvt:$Pg, (AArch64setcc_z_oneuse (predvt (AArch64ptrue 31)), (SVEDup0), intvt:$Op1, invcc))),
             (cmp $Pg, $Op1)>;
 }
 
@@ -5103,13 +5130,13 @@ multiclass SVE_SETCC_Imm_Pat<CondCode cc, CondCode commuted_cc,
                                     commuted_cc)),
             (cmp $Pg, $Zs1, immtype:$imm)>;
   def : Pat<(predvt (and predvt:$Pg,
-                         (AArch64setcc_z (predvt (AArch64ptrue 31)),
+                         (AArch64setcc_z_oneuse (predvt (AArch64ptrue 31)),
                                          (intvt ZPR:$Zs1),
                                          (intvt (splat_vector (immtype:$imm))),
                                          cc))),
             (cmp $Pg, $Zs1, immtype:$imm)>;
   def : Pat<(predvt (and predvt:$Pg,
-                         (AArch64setcc_z (predvt (AArch64ptrue 31)),
+                         (AArch64setcc_z_oneuse (predvt (AArch64ptrue 31)),
                                          (intvt (splat_vector (immtype:$imm))),
                                          (intvt ZPR:$Zs1),
                                          commuted_cc))),
diff --git a/llvm/lib/Target/AMDGPU/AMDGPU.td b/llvm/lib/Target/AMDGPU/AMDGPU.td
index 775dca90b8a04..35c83f4f89583 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPU.td
+++ b/llvm/lib/Target/AMDGPU/AMDGPU.td
@@ -18,10 +18,6 @@ def p4 : PtrValueType<i64, 4>;
 def p5 : PtrValueType<i32, 5>;
 def p6 : PtrValueType<i32, 6>;
 
-class BoolToList<bit Value> {
-  list<int> ret = !if(Value, [1]<int>, []<int>);
-}
-
 //===------------------------------------------------------------===//
 // Subtarget Features (device properties)
 //===------------------------------------------------------------===//
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp
index 0883e7a5ed3a3..82c57dfcef0d8 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp
@@ -251,9 +251,9 @@ void AMDGPUAsmPrinter::emitFunctionBodyEnd() {
       STM, KernelName, getAmdhsaKernelDescriptor(*MF, CurrentProgramInfo),
       CurrentProgramInfo.NumVGPRsForWavesPerEU,
       CurrentProgramInfo.NumSGPRsForWavesPerEU -
-          IsaInfo::getNumExtraSGPRs(&STM,
-                                    CurrentProgramInfo.VCCUsed,
-                                    CurrentProgramInfo.FlatUsed),
+          IsaInfo::getNumExtraSGPRs(
+              &STM, CurrentProgramInfo.VCCUsed, CurrentProgramInfo.FlatUsed,
+              getTargetStreamer()->getTargetID()->isXnackOnOrAny()),
       CurrentProgramInfo.VCCUsed, CurrentProgramInfo.FlatUsed,
       CodeObjectVersion);
 
@@ -721,7 +721,8 @@ void AMDGPUAsmPrinter::getSIProgramInfo(SIProgramInfo &ProgInfo,
   // duplicated in part in AMDGPUAsmParser::calculateGPRBlocks, and could be
   // unified.
   unsigned ExtraSGPRs = IsaInfo::getNumExtraSGPRs(
-      &STM, ProgInfo.VCCUsed, ProgInfo.FlatUsed);
+      &STM, ProgInfo.VCCUsed, ProgInfo.FlatUsed,
+      getTargetStreamer()->getTargetID()->isXnackOnOrAny());
 
   // Check the addressable register limit before we add ExtraSGPRs.
   if (STM.getGeneration() >= AMDGPUSubtarget::VOLCANIC_ISLANDS &&
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
index 28967bb8e5b1c..e4ecfc710d6d8 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
@@ -15,7 +15,7 @@
 
 #include "AMDGPU.h"
 #include "GCNSubtarget.h"
-#include "llvm/Analysis/LegacyDivergenceAnalysis.h"
+#include "llvm/Analysis/UniformityAnalysis.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/InstVisitor.h"
@@ -42,7 +42,7 @@ class AMDGPUAtomicOptimizer : public FunctionPass,
                               public InstVisitor<AMDGPUAtomicOptimizer> {
 private:
   SmallVector<ReplacementInfo, 8> ToReplace;
-  const LegacyDivergenceAnalysis *DA;
+  const UniformityInfo *UA;
   const DataLayout *DL;
   DominatorTree *DT;
   const GCNSubtarget *ST;
@@ -65,7 +65,7 @@ class AMDGPUAtomicOptimizer : public FunctionPass,
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.addPreserved<DominatorTreeWrapperPass>();
-    AU.addRequired<LegacyDivergenceAnalysis>();
+    AU.addRequired<UniformityInfoWrapperPass>();
     AU.addRequired<TargetPassConfig>();
   }
 
@@ -84,7 +84,7 @@ bool AMDGPUAtomicOptimizer::runOnFunction(Function &F) {
     return false;
   }
 
-  DA = &getAnalysis<LegacyDivergenceAnalysis>();
+  UA = &getAnalysis<UniformityInfoWrapperPass>().getUniformityInfo();
   DL = &F.getParent()->getDataLayout();
   DominatorTreeWrapperPass *const DTW =
       getAnalysisIfAvailable<DominatorTreeWrapperPass>();
@@ -139,11 +139,11 @@ void AMDGPUAtomicOptimizer::visitAtomicRMWInst(AtomicRMWInst &I) {
 
   // If the pointer operand is divergent, then each lane is doing an atomic
   // operation on a different address, and we cannot optimize that.
-  if (DA->isDivergentUse(&I.getOperandUse(PtrIdx))) {
+  if (UA->isDivergentUse(I.getOperandUse(PtrIdx))) {
     return;
   }
 
-  const bool ValDivergent = DA->isDivergentUse(&I.getOperandUse(ValIdx));
+  const bool ValDivergent = UA->isDivergentUse(I.getOperandUse(ValIdx));
 
   // If the value operand is divergent, each lane is contributing a different
   // value to the atomic calculation. We can only optimize divergent values if
@@ -217,7 +217,7 @@ void AMDGPUAtomicOptimizer::visitIntrinsicInst(IntrinsicInst &I) {
 
   const unsigned ValIdx = 0;
 
-  const bool ValDivergent = DA->isDivergentUse(&I.getOperandUse(ValIdx));
+  const bool ValDivergent = UA->isDivergentUse(I.getOperandUse(ValIdx));
 
   // If the value operand is divergent, each lane is contributing a different
   // value to the atomic calculation. We can only optimize divergent values if
@@ -231,7 +231,7 @@ void AMDGPUAtomicOptimizer::visitIntrinsicInst(IntrinsicInst &I) {
   // If any of the other arguments to the intrinsic are divergent, we can't
   // optimize the operation.
   for (unsigned Idx = 1; Idx < I.getNumOperands(); Idx++) {
-    if (DA->isDivergentUse(&I.getOperandUse(Idx))) {
+    if (UA->isDivergentUse(I.getOperandUse(Idx))) {
       return;
     }
   }
@@ -705,7 +705,7 @@ void AMDGPUAtomicOptimizer::optimizeAtomic(Instruction &I,
 
 INITIALIZE_PASS_BEGIN(AMDGPUAtomicOptimizer, DEBUG_TYPE,
                       "AMDGPU atomic optimizations", false, false)
-INITIALIZE_PASS_DEPENDENCY(LegacyDivergenceAnalysis)
+INITIALIZE_PASS_DEPENDENCY(UniformityInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(TargetPassConfig)
 INITIALIZE_PASS_END(AMDGPUAtomicOptimizer, DEBUG_TYPE,
                     "AMDGPU atomic optimizations", false, false)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp b/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp
index 346f201333c5d..b79ea7f2b5e22 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp
@@ -17,7 +17,7 @@
 #include "SIModeRegisterDefaults.h"
 #include "llvm/Analysis/AssumptionCache.h"
 #include "llvm/Analysis/ConstantFolding.h"
-#include "llvm/Analysis/LegacyDivergenceAnalysis.h"
+#include "llvm/Analysis/UniformityAnalysis.h"
 #include "llvm/Analysis/ValueTracking.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/IR/Dominators.h"
@@ -73,7 +73,7 @@ class AMDGPUCodeGenPrepare : public FunctionPass,
   const GCNSubtarget *ST = nullptr;
   AssumptionCache *AC = nullptr;
   DominatorTree *DT = nullptr;
-  LegacyDivergenceAnalysis *DA = nullptr;
+  UniformityInfo *UA = nullptr;
   Module *Mod = nullptr;
   const DataLayout *DL = nullptr;
   bool HasUnsafeFPMath = false;
@@ -224,7 +224,7 @@ class AMDGPUCodeGenPrepare : public FunctionPass,
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.addRequired<AssumptionCacheTracker>();
-    AU.addRequired<LegacyDivergenceAnalysis>();
+    AU.addRequired<UniformityInfoWrapperPass>();
 
     // FIXME: Division expansion needs to preserve the dominator tree.
     if (!ExpandDiv64InIR)
@@ -314,7 +314,7 @@ bool AMDGPUCodeGenPrepare::canWidenScalarExtLoad(LoadInst &I) const {
   int TySize = DL.getTypeSizeInBits(Ty);
   Align Alignment = DL.getValueOrABITypeAlignment(I.getAlign(), Ty);
 
-  return I.isSimple() && TySize < 32 && Alignment >= 4 && DA->isUniform(&I);
+  return I.isSimple() && TySize < 32 && Alignment >= 4 && UA->isUniform(&I);
 }
 
 bool AMDGPUCodeGenPrepare::promoteUniformOpToI32(BinaryOperator &I) const {
@@ -519,7 +519,7 @@ bool AMDGPUCodeGenPrepare::replaceMulWithMul24(BinaryOperator &I) const {
     return false;
 
   // Prefer scalar if this could be s_mul_i32
-  if (DA->isUniform(&I))
+  if (UA->isUniform(&I))
     return false;
 
   Value *LHS = I.getOperand(0);
@@ -1237,7 +1237,7 @@ bool AMDGPUCodeGenPrepare::visitBinaryOperator(BinaryOperator &I) {
     return true;
 
   if (ST->has16BitInsts() && needsPromotionToI32(I.getType()) &&
-      DA->isUniform(&I) && promoteUniformOpToI32(I))
+      UA->isUniform(&I) && promoteUniformOpToI32(I))
     return true;
 
   if (UseMul24Intrin && replaceMulWithMul24(I))
@@ -1367,7 +1367,7 @@ bool AMDGPUCodeGenPrepare::visitICmpInst(ICmpInst &I) {
   bool Changed = false;
 
   if (ST->has16BitInsts() && needsPromotionToI32(I.getOperand(0)->getType()) &&
-      DA->isUniform(&I))
+      UA->isUniform(&I))
     Changed |= promoteUniformOpToI32(I);
 
   return Changed;
@@ -1377,7 +1377,7 @@ bool AMDGPUCodeGenPrepare::visitSelectInst(SelectInst &I) {
   bool Changed = false;
 
   if (ST->has16BitInsts() && needsPromotionToI32(I.getType()) &&
-      DA->isUniform(&I))
+      UA->isUniform(&I))
     Changed |= promoteUniformOpToI32(I);
 
   return Changed;
@@ -1396,7 +1396,7 @@ bool AMDGPUCodeGenPrepare::visitBitreverseIntrinsicInst(IntrinsicInst &I) {
   bool Changed = false;
 
   if (ST->has16BitInsts() && needsPromotionToI32(I.getType()) &&
-      DA->isUniform(&I))
+      UA->isUniform(&I))
     Changed |= promoteUniformBitreverseToI32(I);
 
   return Changed;
@@ -1419,7 +1419,7 @@ bool AMDGPUCodeGenPrepare::runOnFunction(Function &F) {
   const AMDGPUTargetMachine &TM = TPC->getTM<AMDGPUTargetMachine>();
   ST = &TM.getSubtarget<GCNSubtarget>(F);
   AC = &getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
-  DA = &getAnalysis<LegacyDivergenceAnalysis>();
+  UA = &getAnalysis<UniformityInfoWrapperPass>().getUniformityInfo();
 
   auto *DTWP = getAnalysisIfAvailable<DominatorTreeWrapperPass>();
   DT = DTWP ? &DTWP->getDomTree() : nullptr;
@@ -1459,7 +1459,7 @@ bool AMDGPUCodeGenPrepare::runOnFunction(Function &F) {
 INITIALIZE_PASS_BEGIN(AMDGPUCodeGenPrepare, DEBUG_TYPE,
                       "AMDGPU IR optimizations", false, false)
 INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
-INITIALIZE_PASS_DEPENDENCY(LegacyDivergenceAnalysis)
+INITIALIZE_PASS_DEPENDENCY(UniformityInfoWrapperPass)
 INITIALIZE_PASS_END(AMDGPUCodeGenPrepare, DEBUG_TYPE, "AMDGPU IR optimizations",
                     false, false)
 
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUCtorDtorLowering.cpp b/llvm/lib/Target/AMDGPU/AMDGPUCtorDtorLowering.cpp
index ba5a8799792a7..39736aed9969f 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUCtorDtorLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUCtorDtorLowering.cpp
@@ -92,8 +92,8 @@ class AMDGPUCtorDtorLoweringLegacy final : public ModulePass {
 
 PreservedAnalyses AMDGPUCtorDtorLoweringPass::run(Module &M,
                                                   ModuleAnalysisManager &AM) {
-  lowerCtorsAndDtors(M);
-  return PreservedAnalyses::all();
+  return lowerCtorsAndDtors(M) ? PreservedAnalyses::none()
+                               : PreservedAnalyses::all();
 }
 
 char AMDGPUCtorDtorLoweringLegacy::ID = 0;
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp b/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp
index 28c26b2998f00..c55b2e4d8fa14 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp
@@ -20,7 +20,7 @@
 #include "MCTargetDesc/R600MCTargetDesc.h"
 #include "R600RegisterInfo.h"
 #include "SIMachineFunctionInfo.h"
-#include "llvm/Analysis/LegacyDivergenceAnalysis.h"
+#include "llvm/Analysis/UniformityAnalysis.h"
 #include "llvm/Analysis/ValueTracking.h"
 #include "llvm/CodeGen/FunctionLoweringInfo.h"
 #include "llvm/CodeGen/SelectionDAG.h"
@@ -101,7 +101,7 @@ INITIALIZE_PASS_BEGIN(AMDGPUDAGToDAGISel, "amdgpu-isel",
                       "AMDGPU DAG->DAG Pattern Instruction Selection", false, false)
 INITIALIZE_PASS_DEPENDENCY(AMDGPUArgumentUsageInfo)
 INITIALIZE_PASS_DEPENDENCY(AMDGPUPerfHintAnalysis)
-INITIALIZE_PASS_DEPENDENCY(LegacyDivergenceAnalysis)
+INITIALIZE_PASS_DEPENDENCY(UniformityInfoWrapperPass)
 #ifdef EXPENSIVE_CHECKS
 INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(LoopInfoWrapperPass)
@@ -199,7 +199,7 @@ bool AMDGPUDAGToDAGISel::fp16SrcZerosHighBits(unsigned Opc) const {
 
 void AMDGPUDAGToDAGISel::getAnalysisUsage(AnalysisUsage &AU) const {
   AU.addRequired<AMDGPUArgumentUsageInfo>();
-  AU.addRequired<LegacyDivergenceAnalysis>();
+  AU.addRequired<UniformityInfoWrapperPass>();
 #ifdef EXPENSIVE_CHECKS
   AU.addRequired<DominatorTreeWrapperPass>();
   AU.addRequired<LoopInfoWrapperPass>();
@@ -1135,6 +1135,15 @@ bool AMDGPUDAGToDAGISel::isDSOffset2Legal(SDValue Base, unsigned Offset0,
   return CurDAG->SignBitIsZero(Base);
 }
 
+bool AMDGPUDAGToDAGISel::isFlatScratchBaseLegal(SDValue Base,
+                                                uint64_t FlatVariant) const {
+  if (FlatVariant != SIInstrFlags::FlatScratch)
+    return true;
+  // When value in 32-bit Base can be negative calculate scratch offset using
+  // 32-bit add instruction, otherwise use Base(unsigned) + offset.
+  return CurDAG->SignBitIsZero(Base);
+}
+
 // TODO: If offset is too big, put low 16-bit into offset.
 bool AMDGPUDAGToDAGISel::SelectDS64Bit4ByteAligned(SDValue Addr, SDValue &Base,
                                                    SDValue &Offset0,
@@ -1528,7 +1537,8 @@ bool AMDGPUDAGToDAGISel::SelectFlatOffsetImpl(SDNode *N, SDValue Addr,
 
   if (Subtarget->hasFlatInstOffsets() && !CanHaveFlatSegmentOffsetBug) {
     SDValue N0, N1;
-    if (isBaseWithConstantOffset64(Addr, N0, N1)) {
+    if (isBaseWithConstantOffset64(Addr, N0, N1) &&
+        isFlatScratchBaseLegal(N0, FlatVariant)) {
       int64_t COffsetVal = cast<ConstantSDNode>(N1)->getSExtValue();
 
       const SIInstrInfo *TII = Subtarget->getInstrInfo();
@@ -1760,7 +1770,8 @@ bool AMDGPUDAGToDAGISel::SelectScratchSAddr(SDNode *Parent, SDValue Addr,
 
   int64_t COffsetVal = 0;
 
-  if (CurDAG->isBaseWithConstantOffset(Addr)) {
+  if (CurDAG->isBaseWithConstantOffset(Addr) &&
+      isFlatScratchBaseLegal(Addr.getOperand(0))) {
     COffsetVal = cast<ConstantSDNode>(Addr.getOperand(1))->getSExtValue();
     SAddr = Addr.getOperand(0);
   } else {
@@ -1838,6 +1849,8 @@ bool AMDGPUDAGToDAGISel::SelectScratchSVAddr(SDNode *N, SDValue Addr,
           CurDAG->getTargetConstant(RemainderOffset, SDLoc(), MVT::i32));
         VAddr = SDValue(VMov, 0);
         SAddr = LHS;
+        if (!isFlatScratchBaseLegal(SAddr) || !isFlatScratchBaseLegal(VAddr))
+          return false;
         if (checkFlatScratchSVSSwizzleBug(VAddr, SAddr, SplitImmOffset))
           return false;
         Offset = CurDAG->getTargetConstant(SplitImmOffset, SDLoc(), MVT::i16);
@@ -1862,6 +1875,9 @@ bool AMDGPUDAGToDAGISel::SelectScratchSVAddr(SDNode *N, SDValue Addr,
     return false;
   }
 
+  if (!isFlatScratchBaseLegal(SAddr) || !isFlatScratchBaseLegal(VAddr))
+    return false;
+
   if (checkFlatScratchSVSSwizzleBug(VAddr, SAddr, ImmOffset))
     return false;
   SAddr = SelectSAddrFI(CurDAG, SAddr);
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.h b/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.h
index 12912b77edaf0..ac5e9d0986011 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.h
+++ b/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.h
@@ -158,6 +158,9 @@ class AMDGPUDAGToDAGISel : public SelectionDAGISel {
   bool isDSOffsetLegal(SDValue Base, unsigned Offset) const;
   bool isDSOffset2Legal(SDValue Base, unsigned Offset0, unsigned Offset1,
                         unsigned Size) const;
+  bool isFlatScratchBaseLegal(
+      SDValue Base, uint64_t FlatVariant = SIInstrFlags::FlatScratch) const;
+
   bool SelectDS1Addr1Offset(SDValue Ptr, SDValue &Base, SDValue &Offset) const;
   bool SelectDS64Bit4ByteAligned(SDValue Ptr, SDValue &Base, SDValue &Offset0,
                                  SDValue &Offset1) const;
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp b/llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp
index f25dd8e003fd3..a20ad4056154d 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp
@@ -912,10 +912,6 @@ bool AMDGPUTargetLowering::isZExtFree(EVT Src, EVT Dest) const {
   return Src == MVT::i32 && Dest == MVT::i64;
 }
 
-bool AMDGPUTargetLowering::isZExtFree(SDValue Val, EVT VT2) const {
-  return isZExtFree(Val.getValueType(), VT2);
-}
-
 bool AMDGPUTargetLowering::isNarrowingProfitable(EVT SrcVT, EVT DestVT) const {
   // There aren't really 64-bit registers, but pairs of 32-bit ones and only a
   // limited number of native 64-bit operations. Shrinking an operation to fit
@@ -1329,15 +1325,10 @@ SDValue AMDGPUTargetLowering::LowerGlobalAddress(AMDGPUMachineFunction* MFI,
   GlobalAddressSDNode *G = cast<GlobalAddressSDNode>(Op);
   const GlobalValue *GV = G->getGlobal();
 
-  if (G->getAddressSpace() == AMDGPUAS::LOCAL_ADDRESS) {
-    if (!MFI->isModuleEntryFunction()) {
-      if (const GlobalVariable *GVar = dyn_cast<GlobalVariable>(GV)) {
-        if (AMDGPUMachineFunction::isKnownAddressLDSGlobal(*GVar)) {
-          unsigned Offset =
-              AMDGPUMachineFunction::calculateKnownAddressOfLDSGlobal(*GVar);
-          return DAG.getConstant(Offset, SDLoc(Op), Op.getValueType());
-        }
-      }
+  if (!MFI->isModuleEntryFunction()) {
+    if (std::optional<uint32_t> Address =
+            AMDGPUMachineFunction::getLDSAbsoluteAddress(*GV)) {
+      return DAG.getConstant(*Address, SDLoc(Op), Op.getValueType());
     }
   }
 
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUISelLowering.h b/llvm/lib/Target/AMDGPU/AMDGPUISelLowering.h
index d82f877098892..1e90255f0ec66 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUISelLowering.h
+++ b/llvm/lib/Target/AMDGPU/AMDGPUISelLowering.h
@@ -173,14 +173,13 @@ class AMDGPUTargetLowering : public TargetLowering {
 
   bool isZExtFree(Type *Src, Type *Dest) const override;
   bool isZExtFree(EVT Src, EVT Dest) const override;
-  bool isZExtFree(SDValue Val, EVT VT2) const override;
 
   SDValue getNegatedExpression(SDValue Op, SelectionDAG &DAG,
                                bool LegalOperations, bool ForCodeSize,
                                NegatibleCost &Cost,
                                unsigned Depth) const override;
 
-  bool isNarrowingProfitable(EVT VT1, EVT VT2) const override;
+  bool isNarrowingProfitable(EVT SrcVT, EVT DestVT) const override;
 
   bool isDesirableToCommuteWithShift(const SDNode *N,
                                      CombineLevel Level) const override;
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp
index 7d3536df7a0a0..cd1f81579c3c8 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp
@@ -3847,7 +3847,7 @@ AMDGPUInstructionSelector::selectFlatOffsetImpl(MachineOperand &Root,
   int64_t ConstOffset;
   std::tie(PtrBase, ConstOffset) =
       getPtrBaseWithConstantOffset(Root.getReg(), *MRI);
-  if (ConstOffset == 0)
+  if (ConstOffset == 0 || !isFlatScratchBaseLegal(PtrBase, FlatVariant))
     return Default;
 
   unsigned AddrSpace = (*MI->memoperands_begin())->getAddrSpace();
@@ -4010,7 +4010,7 @@ AMDGPUInstructionSelector::selectScratchSAddr(MachineOperand &Root) const {
   // possible.
   std::tie(PtrBase, ConstOffset) = getPtrBaseWithConstantOffset(Addr, *MRI);
 
-  if (ConstOffset != 0 &&
+  if (ConstOffset != 0 && isFlatScratchBaseLegal(PtrBase) &&
       TII.isLegalFLATOffset(ConstOffset, AMDGPUAS::PRIVATE_ADDRESS,
                             SIInstrFlags::FlatScratch)) {
     Addr = PtrBase;
@@ -4103,6 +4103,9 @@ AMDGPUInstructionSelector::selectScratchSVAddr(MachineOperand &Root) const {
   Register LHS = AddrDef->MI->getOperand(1).getReg();
   auto LHSDef = getDefSrcRegIgnoringCopies(LHS, *MRI);
 
+  if (!isFlatScratchBaseLegal(LHS) || !isFlatScratchBaseLegal(RHS))
+    return std::nullopt;
+
   if (checkFlatScratchSVSSwizzleBug(RHS, LHS, ImmOffset))
     return std::nullopt;
 
@@ -4234,6 +4237,16 @@ bool AMDGPUInstructionSelector::isDSOffset2Legal(Register Base, int64_t Offset0,
   return KnownBits->signBitIsZero(Base);
 }
 
+bool AMDGPUInstructionSelector::isFlatScratchBaseLegal(
+    Register Base, uint64_t FlatVariant) const {
+  if (FlatVariant != SIInstrFlags::FlatScratch)
+    return true;
+
+  // When value in 32-bit Base can be negative calculate scratch offset using
+  // 32-bit add instruction, otherwise use Base(unsigned) + offset.
+  return KnownBits->signBitIsZero(Base);
+}
+
 bool AMDGPUInstructionSelector::isUnneededShiftMask(const MachineInstr &MI,
                                                     unsigned ShAmtBits) const {
   assert(MI.getOpcode() == TargetOpcode::G_AND);
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h
index 0ccf02ba41cfd..9f1376e6d0288 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h
@@ -13,6 +13,7 @@
 #ifndef LLVM_LIB_TARGET_AMDGPU_AMDGPUINSTRUCTIONSELECTOR_H
 #define LLVM_LIB_TARGET_AMDGPU_AMDGPUINSTRUCTIONSELECTOR_H
 
+#include "SIDefines.h"
 #include "llvm/CodeGen/GlobalISel/InstructionSelector.h"
 #include "llvm/IR/InstrTypes.h"
 
@@ -236,6 +237,8 @@ class AMDGPUInstructionSelector final : public InstructionSelector {
   bool isDSOffsetLegal(Register Base, int64_t Offset) const;
   bool isDSOffset2Legal(Register Base, int64_t Offset0, int64_t Offset1,
                         unsigned Size) const;
+  bool isFlatScratchBaseLegal(
+      Register Base, uint64_t FlatVariant = SIInstrFlags::FlatScratch) const;
 
   std::pair<Register, unsigned>
   selectDS1Addr1OffsetImpl(MachineOperand &Root) const;
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInstructions.td b/llvm/lib/Target/AMDGPU/AMDGPUInstructions.td
index 8cbe3e4ca9846..e1a372f5f89fd 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInstructions.td
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInstructions.td
@@ -287,6 +287,10 @@ def srl_16 : PatFrag<
   (ops node:$src0), (srl_oneuse node:$src0, (i32 16))
 >;
 
+def clamp_s16_u8 : PatFrag<
+  (ops node:$src),
+  (i16 (AMDGPUsmed3 $src, (i16 0), (i16 255)))
+>;
 
 def hi_i16_elt : PatFrag<
   (ops node:$src0), (i16 (trunc (i32 (srl_16 node:$src0))))
diff --git a/llvm/lib/Target/AMDGPU/AMDGPULateCodeGenPrepare.cpp b/llvm/lib/Target/AMDGPU/AMDGPULateCodeGenPrepare.cpp
index 9e86bd0c2b970..2fc68a1423969 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPULateCodeGenPrepare.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPULateCodeGenPrepare.cpp
@@ -14,7 +14,7 @@
 
 #include "AMDGPU.h"
 #include "llvm/Analysis/AssumptionCache.h"
-#include "llvm/Analysis/LegacyDivergenceAnalysis.h"
+#include "llvm/Analysis/UniformityAnalysis.h"
 #include "llvm/Analysis/ValueTracking.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/InstVisitor.h"
@@ -46,7 +46,7 @@ class AMDGPULateCodeGenPrepare
   const DataLayout *DL = nullptr;
 
   AssumptionCache *AC = nullptr;
-  LegacyDivergenceAnalysis *DA = nullptr;
+  UniformityInfo *UA = nullptr;
 
 public:
   static char ID;
@@ -59,7 +59,7 @@ class AMDGPULateCodeGenPrepare
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.addRequired<AssumptionCacheTracker>();
-    AU.addRequired<LegacyDivergenceAnalysis>();
+    AU.addRequired<UniformityInfoWrapperPass>();
     AU.setPreservesAll();
   }
 
@@ -91,7 +91,7 @@ bool AMDGPULateCodeGenPrepare::runOnFunction(Function &F) {
     return false;
 
   AC = &getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
-  DA = &getAnalysis<LegacyDivergenceAnalysis>();
+  UA = &getAnalysis<UniformityInfoWrapperPass>().getUniformityInfo();
 
   bool Changed = false;
   for (auto &BB : F)
@@ -122,7 +122,7 @@ bool AMDGPULateCodeGenPrepare::canWidenScalarExtLoad(LoadInst &LI) const {
   if (LI.getAlign() < DL->getABITypeAlign(Ty))
     return false;
   // It should be uniform, i.e. a scalar load.
-  return DA->isUniform(&LI);
+  return UA->isUniform(&LI);
 }
 
 bool AMDGPULateCodeGenPrepare::visitLoadInst(LoadInst &LI) {
@@ -184,7 +184,7 @@ bool AMDGPULateCodeGenPrepare::visitLoadInst(LoadInst &LI) {
 INITIALIZE_PASS_BEGIN(AMDGPULateCodeGenPrepare, DEBUG_TYPE,
                       "AMDGPU IR late optimizations", false, false)
 INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
-INITIALIZE_PASS_DEPENDENCY(LegacyDivergenceAnalysis)
+INITIALIZE_PASS_DEPENDENCY(UniformityInfoWrapperPass)
 INITIALIZE_PASS_END(AMDGPULateCodeGenPrepare, DEBUG_TYPE,
                     "AMDGPU IR late optimizations", false, false)
 
diff --git a/llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp b/llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp
index 0804fda79d607..d44e280640ee0 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp
@@ -608,6 +608,19 @@ class AMDGPULowerModuleLDS : public ModulePass {
     return MostUsed.GV;
   }
 
+  static void recordLDSAbsoluteAddress(Module *M, GlobalVariable *GV,
+                                       uint32_t Address) {
+    // Write the specified address into metadata where it can be retrieved by
+    // the assembler. Format is a half open range, [Address Address+1)
+    LLVMContext &Ctx = M->getContext();
+    auto *IntTy =
+        M->getDataLayout().getIntPtrType(Ctx, AMDGPUAS::LOCAL_ADDRESS);
+    auto *MinC = ConstantAsMetadata::get(ConstantInt::get(IntTy, Address));
+    auto *MaxC = ConstantAsMetadata::get(ConstantInt::get(IntTy, Address + 1));
+    GV->setMetadata(LLVMContext::MD_absolute_symbol,
+                    MDNode::get(Ctx, {MinC, MaxC}));
+  }
+
   bool runOnModule(Module &M) override {
     LLVMContext &Ctx = M.getContext();
     CallGraph CG = CallGraph(M);
@@ -708,17 +721,21 @@ class AMDGPULowerModuleLDS : public ModulePass {
         kernelsThatIndirectlyAccessAnyOfPassedVariables(M, LDSUsesInfo,
                                                         TableLookupVariables);
 
+    GlobalVariable *MaybeModuleScopeStruct = nullptr;
     if (!ModuleScopeVariables.empty()) {
       LDSVariableReplacement ModuleScopeReplacement =
           createLDSVariableReplacement(M, "llvm.amdgcn.module.lds",
                                        ModuleScopeVariables);
-
+      MaybeModuleScopeStruct = ModuleScopeReplacement.SGV;
       appendToCompilerUsed(M,
                            {static_cast<GlobalValue *>(
                                ConstantExpr::getPointerBitCastOrAddrSpaceCast(
                                    cast<Constant>(ModuleScopeReplacement.SGV),
                                    Type::getInt8PtrTy(Ctx)))});
 
+      // module.lds will be allocated at zero in any kernel that allocates it
+      recordLDSAbsoluteAddress(&M, ModuleScopeReplacement.SGV, 0);
+
       // historic
       removeLocalVarsFromUsedLists(M, ModuleScopeVariables);
 
@@ -806,6 +823,33 @@ class AMDGPULowerModuleLDS : public ModulePass {
       auto Replacement =
           createLDSVariableReplacement(M, VarName, KernelUsedVariables);
 
+      // This struct is allocated at a predictable address that can be
+      // calculated now, recorded in metadata then used to lower references to
+      // it during codegen.
+      {
+        // frame layout, starting from 0
+        //{
+        //  module.lds
+        //  alignment padding
+        //  kernel instance
+        //}
+
+        if (!MaybeModuleScopeStruct ||
+            Func.hasFnAttribute("amdgpu-elide-module-lds")) {
+          // There's no module.lds for this kernel so this replacement struct
+          // goes first
+          recordLDSAbsoluteAddress(&M, Replacement.SGV, 0);
+        } else {
+          const DataLayout &DL = M.getDataLayout();
+          TypeSize ModuleSize =
+              DL.getTypeAllocSize(MaybeModuleScopeStruct->getValueType());
+          GlobalVariable *KernelStruct = Replacement.SGV;
+          Align KernelAlign = AMDGPU::getAlign(DL, KernelStruct);
+          recordLDSAbsoluteAddress(&M, Replacement.SGV,
+                                   alignTo(ModuleSize, KernelAlign));
+        }
+      }
+
       // remove preserves existing codegen
       removeLocalVarsFromUsedLists(M, KernelUsedVariables);
       KernelToReplacement[&Func] = Replacement;
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUMCInstLower.cpp b/llvm/lib/Target/AMDGPU/AMDGPUMCInstLower.cpp
index d88a2cd961b21..da75f5759a8d1 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUMCInstLower.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUMCInstLower.cpp
@@ -13,6 +13,7 @@
 //
 
 #include "AMDGPUMCInstLower.h"
+#include "AMDGPU.h"
 #include "AMDGPUAsmPrinter.h"
 #include "AMDGPUMachineFunction.h"
 #include "AMDGPUTargetMachine.h"
@@ -168,12 +169,11 @@ bool AMDGPUAsmPrinter::lowerOperand(const MachineOperand &MO,
 const MCExpr *AMDGPUAsmPrinter::lowerConstant(const Constant *CV) {
 
   // Intercept LDS variables with known addresses
-  if (const GlobalVariable *GV = dyn_cast<GlobalVariable>(CV)) {
-    if (AMDGPUMachineFunction::isKnownAddressLDSGlobal(*GV)) {
-      unsigned offset =
-          AMDGPUMachineFunction::calculateKnownAddressOfLDSGlobal(*GV);
-      Constant *C = ConstantInt::get(CV->getContext(), APInt(32, offset));
-      return AsmPrinter::lowerConstant(C);
+  if (const GlobalVariable *GV = dyn_cast<const GlobalVariable>(CV)) {
+    if (std::optional<uint32_t> Address =
+            AMDGPUMachineFunction::getLDSAbsoluteAddress(*GV)) {
+      auto *IntTy = Type::getInt32Ty(CV->getContext());
+      return AsmPrinter::lowerConstant(ConstantInt::get(IntTy, *Address));
     }
   }
 
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp b/llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp
index a6a32b98f44ca..e70afd72462ee 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp
@@ -11,7 +11,9 @@
 #include "AMDGPUPerfHintAnalysis.h"
 #include "AMDGPUSubtarget.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
+#include "llvm/IR/ConstantRange.h"
 #include "llvm/IR/Constants.h"
+#include "llvm/IR/Metadata.h"
 #include "llvm/Target/TargetMachine.h"
 
 using namespace llvm;
@@ -89,24 +91,7 @@ unsigned AMDGPUMachineFunction::allocateLDSGlobal(const DataLayout &DL,
 
 static constexpr StringLiteral ModuleLDSName = "llvm.amdgcn.module.lds";
 
-bool AMDGPUMachineFunction::isKnownAddressLDSGlobal(const GlobalVariable &GV) {
-  auto name = GV.getName();
-  return (name == ModuleLDSName) ||
-         (name.startswith("llvm.amdgcn.kernel.") && name.endswith(".lds"));
-}
-
-const Function *AMDGPUMachineFunction::getKernelLDSFunctionFromGlobal(
-    const GlobalVariable &GV) {
-  const Module &M = *GV.getParent();
-  StringRef N(GV.getName());
-  if (N.consume_front("llvm.amdgcn.kernel.") && N.consume_back(".lds")) {
-    return M.getFunction(N);
-  }
-  return nullptr;
-}
-
-const GlobalVariable *
-AMDGPUMachineFunction::getKernelLDSGlobalFromFunction(const Function &F) {
+static const GlobalVariable *getKernelLDSGlobalFromFunction(const Function &F) {
   const Module *M = F.getParent();
   std::string KernelLDSName = "llvm.amdgcn.kernel.";
   KernelLDSName += F.getName();
@@ -119,40 +104,8 @@ static bool canElideModuleLDS(const Function &F) {
   return F.hasFnAttribute("amdgpu-elide-module-lds");
 }
 
-unsigned AMDGPUMachineFunction::calculateKnownAddressOfLDSGlobal(
-    const GlobalVariable &GV) {
-  // module.lds, then alignment padding, then kernel.lds, then other variables
-  // if any
-
-  assert(isKnownAddressLDSGlobal(GV));
-  unsigned Offset = 0;
-
-  if (GV.getName() == ModuleLDSName) {
-    return 0;
-  }
-
-  const Module *M = GV.getParent();
-  const DataLayout &DL = M->getDataLayout();
-
-  const GlobalVariable *GVM = M->getNamedGlobal(ModuleLDSName);
-  const Function *f = getKernelLDSFunctionFromGlobal(GV);
-
-  // Account for module.lds if allocated for this function
-  if (GVM && f && !canElideModuleLDS(*f)) {
-    // allocator aligns this to var align, but it's zero to begin with
-    Offset += DL.getTypeAllocSize(GVM->getValueType());
-  }
-
-  // No dynamic LDS alignment done by allocateModuleLDSGlobal
-  Offset = alignTo(
-      Offset, DL.getValueOrABITypeAlignment(GV.getAlign(), GV.getValueType()));
-
-  return Offset;
-}
-
 void AMDGPUMachineFunction::allocateKnownAddressLDSGlobal(const Function &F) {
   const Module *M = F.getParent();
-
   // This function is called before allocating any other LDS so that it can
   // reliably put values at known addresses. Consequently, dynamic LDS, if
   // present, will not yet have been allocated
@@ -180,40 +133,60 @@ void AMDGPUMachineFunction::allocateKnownAddressLDSGlobal(const Function &F) {
     const GlobalVariable *KV = getKernelLDSGlobalFromFunction(F);
 
     if (GV && !canElideModuleLDS(F)) {
-      assert(isKnownAddressLDSGlobal(*GV));
       unsigned Offset = allocateLDSGlobal(M->getDataLayout(), *GV, Align());
-      (void)Offset;
-      assert(Offset == calculateKnownAddressOfLDSGlobal(*GV) &&
-             "Module LDS expected to be allocated before other LDS");
+      std::optional<uint32_t> Expect = getLDSAbsoluteAddress(*GV);
+      if (!Expect || (Offset != Expect)) {
+        report_fatal_error("Inconsistent metadata on module LDS variable");
+      }
     }
 
     if (KV) {
       // The per-kernel offset is deterministic because it is allocated
       // before any other non-module LDS variables.
-      assert(isKnownAddressLDSGlobal(*KV));
       unsigned Offset = allocateLDSGlobal(M->getDataLayout(), *KV, Align());
-      (void)Offset;
-      assert(Offset == calculateKnownAddressOfLDSGlobal(*KV) &&
-             "Kernel LDS expected to be immediately after module LDS");
+      std::optional<uint32_t> Expect = getLDSAbsoluteAddress(*KV);
+      if (!Expect || (Offset != Expect)) {
+        report_fatal_error("Inconsistent metadata on kernel LDS variable");
+      }
     }
   }
 }
 
 std::optional<uint32_t>
 AMDGPUMachineFunction::getLDSKernelIdMetadata(const Function &F) {
-  auto MD = F.getMetadata("llvm.amdgcn.lds.kernel.id");
+  // TODO: Would be more consistent with the abs symbols to use a range
+  MDNode *MD = F.getMetadata("llvm.amdgcn.lds.kernel.id");
   if (MD && MD->getNumOperands() == 1) {
-    ConstantInt *KnownSize = mdconst::extract<ConstantInt>(MD->getOperand(0));
-    if (KnownSize) {
-      uint64_t V = KnownSize->getZExtValue();
-      if (V <= UINT32_MAX) {
-        return V;
+    if (ConstantInt *KnownSize =
+            mdconst::extract<ConstantInt>(MD->getOperand(0))) {
+      uint64_t ZExt = KnownSize->getZExtValue();
+      if (ZExt <= UINT32_MAX) {
+        return ZExt;
       }
     }
   }
   return {};
 }
 
+std::optional<uint32_t>
+AMDGPUMachineFunction::getLDSAbsoluteAddress(const GlobalValue &GV) {
+  if (GV.getAddressSpace() != AMDGPUAS::LOCAL_ADDRESS)
+    return {};
+
+  std::optional<ConstantRange> AbsSymRange = GV.getAbsoluteSymbolRange();
+  if (!AbsSymRange)
+    return {};
+
+  if (const APInt *V = AbsSymRange->getSingleElement()) {
+    std::optional<uint64_t> ZExt = V->tryZExtValue();
+    if (ZExt && (*ZExt <= UINT32_MAX)) {
+      return *ZExt;
+    }
+  }
+
+  return {};
+}
+
 void AMDGPUMachineFunction::setDynLDSAlign(const DataLayout &DL,
                                            const GlobalVariable &GV) {
   assert(DL.getTypeAllocSize(GV.getValueType()).isZero());
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.h b/llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.h
index f27f8252a4d8f..ba4c55a58c866 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.h
+++ b/llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.h
@@ -106,20 +106,8 @@ class AMDGPUMachineFunction : public MachineFunctionInfo {
 
   void allocateKnownAddressLDSGlobal(const Function &F);
 
-  // A kernel function may have an associated LDS allocation, and a kernel-scope
-  // LDS allocation must have an associated kernel function
-
-  // LDS allocation should have an associated kernel function
-  static const Function *
-  getKernelLDSFunctionFromGlobal(const GlobalVariable &GV);
-  static const GlobalVariable *
-  getKernelLDSGlobalFromFunction(const Function &F);
-
-  // Module or kernel scope LDS variable
-  static bool isKnownAddressLDSGlobal(const GlobalVariable &GV);
-  static unsigned calculateKnownAddressOfLDSGlobal(const GlobalVariable &GV);
-
   static std::optional<uint32_t> getLDSKernelIdMetadata(const Function &F);
+  static std::optional<uint32_t> getLDSAbsoluteAddress(const GlobalValue &GV);
 
   Align getDynLDSAlign() const { return DynLDSAlign; }
 
diff --git a/llvm/lib/Target/AMDGPU/AMDGPURewriteUndefForPHI.cpp b/llvm/lib/Target/AMDGPU/AMDGPURewriteUndefForPHI.cpp
index ff34726fdf028..9c07851243c91 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPURewriteUndefForPHI.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPURewriteUndefForPHI.cpp
@@ -10,7 +10,7 @@
 // StructurizedCFG pass, and this pass has some additional limitation that make
 // it can only run after SIAnnotateControlFlow.
 //
-// To achieve optimal code generation for AMDGPU, we assume that divergence
+// To achieve optimal code generation for AMDGPU, we assume that uniformity
 // analysis reports the PHI in join block of divergent branch as uniform if
 // it has one unique uniform value plus additional undefined/poisoned incoming
 // value. That is to say the later compiler pipeline will ensure such PHI always
@@ -56,7 +56,7 @@
 // \---
 
 #include "AMDGPU.h"
-#include "llvm/Analysis/LegacyDivergenceAnalysis.h"
+#include "llvm/Analysis/UniformityAnalysis.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/Dominators.h"
@@ -81,11 +81,11 @@ class AMDGPURewriteUndefForPHI : public FunctionPass {
   }
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
-    AU.addRequired<LegacyDivergenceAnalysis>();
+    AU.addRequired<UniformityInfoWrapperPass>();
     AU.addRequired<DominatorTreeWrapperPass>();
 
     AU.addPreserved<DominatorTreeWrapperPass>();
-    AU.addPreserved<LegacyDivergenceAnalysis>();
+    AU.addPreserved<UniformityInfoWrapperPass>();
     AU.setPreservesCFG();
   }
 };
@@ -95,17 +95,17 @@ char AMDGPURewriteUndefForPHI::ID = 0;
 
 INITIALIZE_PASS_BEGIN(AMDGPURewriteUndefForPHI, DEBUG_TYPE,
                       "Rewrite undef for PHI", false, false)
-INITIALIZE_PASS_DEPENDENCY(LegacyDivergenceAnalysis)
+INITIALIZE_PASS_DEPENDENCY(UniformityInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
 INITIALIZE_PASS_END(AMDGPURewriteUndefForPHI, DEBUG_TYPE,
                     "Rewrite undef for PHI", false, false)
 
-bool rewritePHIs(Function &F, LegacyDivergenceAnalysis *DA, DominatorTree *DT) {
+bool rewritePHIs(Function &F, UniformityInfo &UA, DominatorTree *DT) {
   bool Changed = false;
   SmallVector<PHINode *> ToBeDeleted;
   for (auto &BB : F) {
     for (auto &PHI : BB.phis()) {
-      if (DA->isDivergent(&PHI))
+      if (UA.isDivergent(&PHI))
         continue;
 
       // The unique incoming value except undef/poison for the PHI node.
@@ -147,7 +147,7 @@ bool rewritePHIs(Function &F, LegacyDivergenceAnalysis *DA, DominatorTree *DT) {
       // TODO: We should still be able to replace undef value if the unique
       // value is a Constant.
       if (!UniqueDefinedIncoming || Undefs.empty() ||
-          !DA->isDivergent(DominateBB->getTerminator()))
+          !UA.isDivergent(DominateBB->getTerminator()))
         continue;
 
       // We only replace the undef when DominateBB truly dominates all the
@@ -171,9 +171,10 @@ bool rewritePHIs(Function &F, LegacyDivergenceAnalysis *DA, DominatorTree *DT) {
 }
 
 bool AMDGPURewriteUndefForPHI::runOnFunction(Function &F) {
-  LegacyDivergenceAnalysis *DA = &getAnalysis<LegacyDivergenceAnalysis>();
+  UniformityInfo &UA =
+      getAnalysis<UniformityInfoWrapperPass>().getUniformityInfo();
   DominatorTree *DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
-  return rewritePHIs(F, DA, DT);
+  return rewritePHIs(F, UA, DT);
 }
 
 FunctionPass *llvm::createAMDGPURewriteUndefForPHIPass() {
diff --git a/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp b/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp
index c9dc8aade7c7b..1cff11227cf94 100644
--- a/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp
+++ b/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp
@@ -5263,10 +5263,10 @@ bool AMDGPUAsmParser::ParseDirectiveAMDHSAKernel() {
 #undef PARSE_BITS_ENTRY
   }
 
-  if (Seen.find(".amdhsa_next_free_vgpr") == Seen.end())
+  if (!Seen.contains(".amdhsa_next_free_vgpr"))
     return TokError(".amdhsa_next_free_vgpr directive is required");
 
-  if (Seen.find(".amdhsa_next_free_sgpr") == Seen.end())
+  if (!Seen.contains(".amdhsa_next_free_sgpr"))
     return TokError(".amdhsa_next_free_sgpr directive is required");
 
   unsigned VGPRBlocks;
@@ -5304,7 +5304,7 @@ bool AMDGPUAsmParser::ParseDirectiveAMDHSAKernel() {
                   UserSGPRCount);
 
   if (isGFX90A()) {
-    if (Seen.find(".amdhsa_accum_offset") == Seen.end())
+    if (!Seen.contains(".amdhsa_accum_offset"))
       return TokError(".amdhsa_accum_offset directive is required");
     if (AccumOffset < 4 || AccumOffset > 256 || (AccumOffset & 3))
       return TokError("accum_offset should be in range [4..256] in "
@@ -5315,7 +5315,7 @@ bool AMDGPUAsmParser::ParseDirectiveAMDHSAKernel() {
                     (AccumOffset / 4 - 1));
   }
 
-  if (IVersion.Major == 10) {
+  if (IVersion.Major >= 10) {
     // SharedVGPRCount < 16 checked by PARSE_ENTRY_BITS
     if (SharedVGPRCount && EnableWavefrontSize32) {
       return TokError("shared_vgpr_count directive not valid on "
diff --git a/llvm/lib/Target/AMDGPU/BUFInstructions.td b/llvm/lib/Target/AMDGPU/BUFInstructions.td
index 37747c5f39261..aacdffad32392 100644
--- a/llvm/lib/Target/AMDGPU/BUFInstructions.td
+++ b/llvm/lib/Target/AMDGPU/BUFInstructions.td
@@ -2697,11 +2697,11 @@ multiclass MUBUF_Real_vi_gfx90a<bits<7> op, MUBUF_Pseudo ps, bit isTFE = 0> {
   def _vi :     MUBUF_Real_vi<op, ps>;
 
   if !not(isTFE) then {
-    foreach _ = BoolToList<!not(ps.FPAtomic)>.ret in
+    if !not(ps.FPAtomic) then
       def _gfx90a : MUBUF_Real_gfx90a<op, ps>;
   }
 
-  foreach _ = BoolToList<ps.FPAtomic>.ret in {
+  if ps.FPAtomic then {
     def _gfx90a : MUBUF_Real_gfx90a<op, ps, 0> {
       let SubtargetPredicate = isGFX90AOnly;
       let AssemblerPredicate = isGFX90AOnly;
diff --git a/llvm/lib/Target/AMDGPU/FLATInstructions.td b/llvm/lib/Target/AMDGPU/FLATInstructions.td
index 6972afebf8c47..d5123e0047304 100644
--- a/llvm/lib/Target/AMDGPU/FLATInstructions.td
+++ b/llvm/lib/Target/AMDGPU/FLATInstructions.td
@@ -1480,12 +1480,23 @@ defm : GlobalFLATAtomicPats <"GLOBAL_ATOMIC_XOR_X2", "atomic_load_xor_global", i
 let OtherPredicates = [isGFX10Plus] in {
 defm : GlobalFLATAtomicPats <"GLOBAL_ATOMIC_FMIN", "atomic_load_fmin_global", f32>;
 defm : GlobalFLATAtomicPats <"GLOBAL_ATOMIC_FMAX", "atomic_load_fmax_global", f32>;
-defm : GlobalFLATAtomicPats <"GLOBAL_ATOMIC_FMIN_X2", "atomic_load_fmin_global", f64>;
-defm : GlobalFLATAtomicPats <"GLOBAL_ATOMIC_FMAX_X2", "atomic_load_fmax_global", f64>;
 defm : GlobalFLATAtomicIntrPats <"GLOBAL_ATOMIC_FMIN", "int_amdgcn_global_atomic_fmin", f32>;
 defm : GlobalFLATAtomicIntrPats <"GLOBAL_ATOMIC_FMAX", "int_amdgcn_global_atomic_fmax", f32>;
+defm : FlatSignedAtomicPat <"FLAT_ATOMIC_FMIN", "atomic_load_fmin_flat", f32>;
+defm : FlatSignedAtomicPat <"FLAT_ATOMIC_FMAX", "atomic_load_fmax_flat", f32>;
+defm : FlatSignedAtomicIntrPat <"FLAT_ATOMIC_FMIN", "int_amdgcn_flat_atomic_fmin", f32>;
+defm : FlatSignedAtomicIntrPat <"FLAT_ATOMIC_FMAX", "int_amdgcn_flat_atomic_fmax", f32>;
+}
+
+let OtherPredicates = [isGFX10Only] in {
+defm : GlobalFLATAtomicPats <"GLOBAL_ATOMIC_FMIN_X2", "atomic_load_fmin_global", f64>;
+defm : GlobalFLATAtomicPats <"GLOBAL_ATOMIC_FMAX_X2", "atomic_load_fmax_global", f64>;
 defm : GlobalFLATAtomicIntrPats <"GLOBAL_ATOMIC_FMIN_X2", "int_amdgcn_global_atomic_fmin", f64>;
 defm : GlobalFLATAtomicIntrPats <"GLOBAL_ATOMIC_FMAX_X2", "int_amdgcn_global_atomic_fmax", f64>;
+defm : FlatSignedAtomicPat <"FLAT_ATOMIC_FMIN_X2", "atomic_load_fmin_flat", f64>;
+defm : FlatSignedAtomicPat <"FLAT_ATOMIC_FMAX_X2", "atomic_load_fmax_flat", f64>;
+defm : FlatSignedAtomicIntrPat <"FLAT_ATOMIC_FMIN_X2", "int_amdgcn_flat_atomic_fmin", f64>;
+defm : FlatSignedAtomicIntrPat <"FLAT_ATOMIC_FMAX_X2", "int_amdgcn_flat_atomic_fmax", f64>;
 }
 
 let OtherPredicates = [HasAtomicFaddNoRtnInsts] in {
diff --git a/llvm/lib/Target/AMDGPU/GCNRegPressure.cpp b/llvm/lib/Target/AMDGPU/GCNRegPressure.cpp
index f9bed9a76c6fb..4b055d883e389 100644
--- a/llvm/lib/Target/AMDGPU/GCNRegPressure.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNRegPressure.cpp
@@ -336,23 +336,38 @@ bool GCNDownwardRPTracker::advanceBeforeNext() {
   assert(SI.isValid());
 
   // Remove dead registers or mask bits.
-  for (auto &It : LiveRegs) {
-    const LiveInterval &LI = LIS.getInterval(It.first);
+  SmallSet<Register, 8> SeenRegs;
+  for (auto &MO : LastTrackedMI->operands()) {
+    if (!MO.isReg() || !MO.getReg().isVirtual())
+      continue;
+    if (MO.isUse() && !MO.readsReg())
+      continue;
+    if (!SeenRegs.insert(MO.getReg()).second)
+      continue;
+    const LiveInterval &LI = LIS.getInterval(MO.getReg());
     if (LI.hasSubRanges()) {
+      auto It = LiveRegs.end();
       for (const auto &S : LI.subranges()) {
         if (!S.liveAt(SI)) {
-          auto PrevMask = It.second;
-          It.second &= ~S.LaneMask;
-          CurPressure.inc(It.first, PrevMask, It.second, *MRI);
+          if (It == LiveRegs.end()) {
+            It = LiveRegs.find(MO.getReg());
+            if (It == LiveRegs.end())
+              llvm_unreachable("register isn't live");
+          }
+          auto PrevMask = It->second;
+          It->second &= ~S.LaneMask;
+          CurPressure.inc(MO.getReg(), PrevMask, It->second, *MRI);
         }
       }
+      if (It != LiveRegs.end() && It->second.none())
+        LiveRegs.erase(It);
     } else if (!LI.liveAt(SI)) {
-      auto PrevMask = It.second;
-      It.second = LaneBitmask::getNone();
-      CurPressure.inc(It.first, PrevMask, It.second, *MRI);
+      auto It = LiveRegs.find(MO.getReg());
+      if (It == LiveRegs.end())
+        llvm_unreachable("register isn't live");
+      CurPressure.inc(MO.getReg(), It->second, LaneBitmask::getNone(), *MRI);
+      LiveRegs.erase(It);
     }
-    if (It.second.none())
-      LiveRegs.erase(It.first);
   }
 
   MaxPressure = max(MaxPressure, CurPressure);
diff --git a/llvm/lib/Target/AMDGPU/GCNSchedStrategy.cpp b/llvm/lib/Target/AMDGPU/GCNSchedStrategy.cpp
index f9c58e8c33318..7f7d6e56874c2 100644
--- a/llvm/lib/Target/AMDGPU/GCNSchedStrategy.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNSchedStrategy.cpp
@@ -511,11 +511,19 @@ void GCNScheduleDAGMILive::computeBlockPressure(unsigned RegionIdx,
   // If the block has the only successor then live-ins of that successor are
   // live-outs of the current block. We can reuse calculated live set if the
   // successor will be sent to scheduling past current block.
+
+  // However, due to the bug in LiveInterval analysis it may happen that two
+  // predecessors of the same successor block have different lane bitmasks for
+  // a live-out register. Workaround that by sticking to one-to-one relationship
+  // i.e. one predecessor with one successor block.
   const MachineBasicBlock *OnlySucc = nullptr;
-  if (MBB->succ_size() == 1 && !(*MBB->succ_begin())->empty()) {
-    SlotIndexes *Ind = LIS->getSlotIndexes();
-    if (Ind->getMBBStartIdx(MBB) < Ind->getMBBStartIdx(*MBB->succ_begin()))
-      OnlySucc = *MBB->succ_begin();
+  if (MBB->succ_size() == 1) {
+    auto *Candidate = *MBB->succ_begin();
+    if (!Candidate->empty() && Candidate->pred_size() == 1) {
+      SlotIndexes *Ind = LIS->getSlotIndexes();
+      if (Ind->getMBBStartIdx(MBB) < Ind->getMBBStartIdx(Candidate))
+        OnlySucc = Candidate;
+    }
   }
 
   // Scheduler sends regions from the end of the block upwards.
diff --git a/llvm/lib/Target/AMDGPU/R600ISelLowering.cpp b/llvm/lib/Target/AMDGPU/R600ISelLowering.cpp
index 1a951c1ab02fb..d361df26841c3 100644
--- a/llvm/lib/Target/AMDGPU/R600ISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/R600ISelLowering.cpp
@@ -1656,7 +1656,7 @@ SDValue R600TargetLowering::OptimizeSwizzle(SDValue BuildVector, SDValue Swz[],
   BuildVector = CompactSwizzlableVector(DAG, BuildVector, SwizzleRemap);
   for (unsigned i = 0; i < 4; i++) {
     unsigned Idx = cast<ConstantSDNode>(Swz[i])->getZExtValue();
-    if (SwizzleRemap.find(Idx) != SwizzleRemap.end())
+    if (SwizzleRemap.contains(Idx))
       Swz[i] = DAG.getConstant(SwizzleRemap[Idx], DL, MVT::i32);
   }
 
@@ -1664,7 +1664,7 @@ SDValue R600TargetLowering::OptimizeSwizzle(SDValue BuildVector, SDValue Swz[],
   BuildVector = ReorganizeVector(DAG, BuildVector, SwizzleRemap);
   for (unsigned i = 0; i < 4; i++) {
     unsigned Idx = cast<ConstantSDNode>(Swz[i])->getZExtValue();
-    if (SwizzleRemap.find(Idx) != SwizzleRemap.end())
+    if (SwizzleRemap.contains(Idx))
       Swz[i] = DAG.getConstant(SwizzleRemap[Idx], DL, MVT::i32);
   }
 
diff --git a/llvm/lib/Target/AMDGPU/R600InstrInfo.cpp b/llvm/lib/Target/AMDGPU/R600InstrInfo.cpp
index 4056274cd4404..7f874b245b8f4 100644
--- a/llvm/lib/Target/AMDGPU/R600InstrInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/R600InstrInfo.cpp
@@ -328,7 +328,7 @@ R600InstrInfo::ExtractSrcs(MachineInstr &MI,
     if (Reg == R600::OQAP) {
       Result.push_back(std::pair(Index, 0U));
     }
-    if (PV.find(Reg) != PV.end()) {
+    if (PV.contains(Reg)) {
       // 255 is used to tells its a PS/PV reg
       Result.push_back(std::pair(255, 0U));
       continue;
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index d23ef959d93c9..d71ea740d3fb1 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -19,8 +19,8 @@
 #include "SIRegisterInfo.h"
 #include "llvm/ADT/FloatingPointMode.h"
 #include "llvm/ADT/Statistic.h"
-#include "llvm/Analysis/LegacyDivergenceAnalysis.h"
 #include "llvm/Analysis/OptimizationRemarkEmitter.h"
+#include "llvm/Analysis/UniformityAnalysis.h"
 #include "llvm/BinaryFormat/ELF.h"
 #include "llvm/CodeGen/Analysis.h"
 #include "llvm/CodeGen/FunctionLoweringInfo.h"
@@ -35,8 +35,8 @@
 #include "llvm/IR/IntrinsicsAMDGPU.h"
 #include "llvm/IR/IntrinsicsR600.h"
 #include "llvm/Support/CommandLine.h"
-#include "llvm/Support/ModRef.h"
 #include "llvm/Support/KnownBits.h"
+#include "llvm/Support/ModRef.h"
 
 using namespace llvm;
 
@@ -2423,11 +2423,14 @@ SDValue SITargetLowering::LowerFormalArguments(
 
   if (IsGraphics) {
     assert(!Info->hasDispatchPtr() && !Info->hasKernargSegmentPtr() &&
-           (!Info->hasFlatScratchInit() || Subtarget->enableFlatScratch()) &&
-           !Info->hasWorkGroupIDX() && !Info->hasWorkGroupIDY() &&
-           !Info->hasWorkGroupIDZ() && !Info->hasWorkGroupInfo() &&
-           !Info->hasLDSKernelId() && !Info->hasWorkItemIDX() &&
-           !Info->hasWorkItemIDY() && !Info->hasWorkItemIDZ());
+           !Info->hasWorkGroupInfo() && !Info->hasLDSKernelId() &&
+           !Info->hasWorkItemIDX() && !Info->hasWorkItemIDY() &&
+           !Info->hasWorkItemIDZ());
+    if (!Subtarget->enableFlatScratch())
+      assert(!Info->hasFlatScratchInit());
+    if (CallConv != CallingConv::AMDGPU_CS || !Subtarget->hasArchitectedSGPRs())
+      assert(!Info->hasWorkGroupIDX() && !Info->hasWorkGroupIDY() &&
+             !Info->hasWorkGroupIDZ());
   }
 
   if (CallConv == CallingConv::AMDGPU_PS) {
@@ -9468,13 +9471,34 @@ SDValue SITargetLowering::performUCharToFloatCombine(SDNode *N,
 
 SDValue SITargetLowering::performFCopySignCombine(SDNode *N,
                                                   DAGCombinerInfo &DCI) const {
+  SDValue MagnitudeOp = N->getOperand(0);
   SDValue SignOp = N->getOperand(1);
-  if (SignOp.getValueType() != MVT::f64)
-    return SDValue();
-
   SelectionDAG &DAG = DCI.DAG;
   SDLoc DL(N);
 
+  // f64 fcopysign is really an f32 copysign on the high bits, so replace the
+  // lower half with a copy.
+  // fcopysign f64:x, _:y -> x.lo32, (fcopysign (f32 x.hi32), _:y)
+  if (MagnitudeOp.getValueType() == MVT::f64) {
+    SDValue MagAsVector = DAG.getNode(ISD::BITCAST, DL, MVT::v2f32, MagnitudeOp);
+    SDValue MagLo =
+      DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, MVT::f32, MagAsVector,
+                  DAG.getConstant(0, DL, MVT::i32));
+    SDValue MagHi =
+      DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, MVT::f32, MagAsVector,
+                  DAG.getConstant(1, DL, MVT::i32));
+
+    SDValue HiOp =
+      DAG.getNode(ISD::FCOPYSIGN, DL, MVT::f32, MagHi, SignOp);
+
+    SDValue Vector = DAG.getNode(ISD::BUILD_VECTOR, DL, MVT::v2f32, MagLo, HiOp);
+
+    return DAG.getNode(ISD::BITCAST, DL, MVT::f64, Vector);
+  }
+
+  if (SignOp.getValueType() != MVT::f64)
+    return SDValue();
+
   // Reduce width of sign operand, we only need the highest bit.
   //
   // fcopysign f64:x, f64:y ->
@@ -10567,45 +10591,41 @@ static unsigned minMaxOpcToMin3Max3Opc(unsigned Opc) {
   }
 }
 
-SDValue SITargetLowering::performIntMed3ImmCombine(
-  SelectionDAG &DAG, const SDLoc &SL,
-  SDValue Op0, SDValue Op1, bool Signed) const {
-  ConstantSDNode *K1 = dyn_cast<ConstantSDNode>(Op1);
-  if (!K1)
-    return SDValue();
+SDValue SITargetLowering::performIntMed3ImmCombine(SelectionDAG &DAG,
+                                                   const SDLoc &SL, SDValue Src,
+                                                   SDValue MinVal,
+                                                   SDValue MaxVal,
+                                                   bool Signed) const {
 
-  ConstantSDNode *K0 = dyn_cast<ConstantSDNode>(Op0.getOperand(1));
-  if (!K0)
+  // med3 comes from
+  //    min(max(x, K0), K1), K0 < K1
+  //    max(min(x, K0), K1), K1 < K0
+  //
+  // "MinVal" and "MaxVal" respectively refer to the rhs of the
+  // min/max op.
+  ConstantSDNode *MinK = dyn_cast<ConstantSDNode>(MinVal);
+  ConstantSDNode *MaxK = dyn_cast<ConstantSDNode>(MaxVal);
+
+  if (!MinK || !MaxK)
     return SDValue();
 
   if (Signed) {
-    if (K0->getAPIntValue().sge(K1->getAPIntValue()))
+    if (MaxK->getAPIntValue().sge(MinK->getAPIntValue()))
       return SDValue();
   } else {
-    if (K0->getAPIntValue().uge(K1->getAPIntValue()))
+    if (MaxK->getAPIntValue().uge(MinK->getAPIntValue()))
       return SDValue();
   }
 
-  EVT VT = K0->getValueType(0);
+  EVT VT = MinK->getValueType(0);
   unsigned Med3Opc = Signed ? AMDGPUISD::SMED3 : AMDGPUISD::UMED3;
-  if (VT == MVT::i32 || (VT == MVT::i16 && Subtarget->hasMed3_16())) {
-    return DAG.getNode(Med3Opc, SL, VT,
-                       Op0.getOperand(0), SDValue(K0, 0), SDValue(K1, 0));
-  }
-
-  // If there isn't a 16-bit med3 operation, convert to 32-bit.
-  if (VT == MVT::i16) {
-    MVT NVT = MVT::i32;
-    unsigned ExtOp = Signed ? ISD::SIGN_EXTEND : ISD::ZERO_EXTEND;
-
-    SDValue Tmp1 = DAG.getNode(ExtOp, SL, NVT, Op0->getOperand(0));
-    SDValue Tmp2 = DAG.getNode(ExtOp, SL, NVT, Op0->getOperand(1));
-    SDValue Tmp3 = DAG.getNode(ExtOp, SL, NVT, Op1);
-
-    SDValue Med3 = DAG.getNode(Med3Opc, SL, NVT, Tmp1, Tmp2, Tmp3);
-    return DAG.getNode(ISD::TRUNCATE, SL, VT, Med3);
-  }
+  if (VT == MVT::i32 || (VT == MVT::i16 && Subtarget->hasMed3_16()))
+    return DAG.getNode(Med3Opc, SL, VT, Src, MaxVal, MinVal);
 
+  // Note: we could also extend to i32 and use i32 med3 if i16 med3 is
+  // not available, but this is unlikely to be profitable as constants
+  // will often need to be materialized & extended, especially on
+  // pre-GFX10 where VOP3 instructions couldn't take literal operands.
   return SDValue();
 }
 
@@ -10717,13 +10737,26 @@ SDValue SITargetLowering::performMinMaxCombine(SDNode *N,
   }
 
   // min(max(x, K0), K1), K0 < K1 -> med3(x, K0, K1)
+  // max(min(x, K0), K1), K1 < K0 -> med3(x, K1, K0)
   if (Opc == ISD::SMIN && Op0.getOpcode() == ISD::SMAX && Op0.hasOneUse()) {
-    if (SDValue Med3 = performIntMed3ImmCombine(DAG, SDLoc(N), Op0, Op1, true))
+    if (SDValue Med3 = performIntMed3ImmCombine(
+            DAG, SDLoc(N), Op0->getOperand(0), Op1, Op0->getOperand(1), true))
+      return Med3;
+  }
+  if (Opc == ISD::SMAX && Op0.getOpcode() == ISD::SMIN && Op0.hasOneUse()) {
+    if (SDValue Med3 = performIntMed3ImmCombine(
+            DAG, SDLoc(N), Op0->getOperand(0), Op0->getOperand(1), Op1, true))
       return Med3;
   }
 
   if (Opc == ISD::UMIN && Op0.getOpcode() == ISD::UMAX && Op0.hasOneUse()) {
-    if (SDValue Med3 = performIntMed3ImmCombine(DAG, SDLoc(N), Op0, Op1, false))
+    if (SDValue Med3 = performIntMed3ImmCombine(
+            DAG, SDLoc(N), Op0->getOperand(0), Op1, Op0->getOperand(1), false))
+      return Med3;
+  }
+  if (Opc == ISD::UMAX && Op0.getOpcode() == ISD::UMIN && Op0.hasOneUse()) {
+    if (SDValue Med3 = performIntMed3ImmCombine(
+            DAG, SDLoc(N), Op0->getOperand(0), Op0->getOperand(1), Op1, false))
       return Med3;
   }
 
@@ -12838,9 +12871,9 @@ static bool isCopyFromRegOfInlineAsm(const SDNode *N) {
   return false;
 }
 
-bool SITargetLowering::isSDNodeSourceOfDivergence(
-    const SDNode *N, FunctionLoweringInfo *FLI,
-    LegacyDivergenceAnalysis *KDA) const {
+bool SITargetLowering::isSDNodeSourceOfDivergence(const SDNode *N,
+                                                  FunctionLoweringInfo *FLI,
+                                                  UniformityInfo *UA) const {
   switch (N->getOpcode()) {
   case ISD::CopyFromReg: {
     const RegisterSDNode *R = cast<RegisterSDNode>(N->getOperand(1));
@@ -12853,7 +12886,7 @@ bool SITargetLowering::isSDNodeSourceOfDivergence(
       return !TRI->isSGPRReg(MRI, Reg);
 
     if (const Value *V = FLI->getValueFromVirtualReg(R->getReg()))
-      return KDA->isDivergent(V);
+      return UA->isDivergent(V);
 
     assert(Reg == FLI->DemoteRegister || isCopyFromRegOfInlineAsm(N));
     return !TRI->isSGPRReg(MRI, Reg);
@@ -13134,6 +13167,8 @@ SITargetLowering::getRegClassFor(MVT VT, bool isDivergent) const {
 // uniform values (as produced by the mask results of control flow intrinsics)
 // used outside of divergent blocks. The phi users need to also be treated as
 // always uniform.
+//
+// FIXME: DA is no longer in-use. Does this still apply to UniformityAnalysis?
 static bool hasCFUser(const Value *V, SmallPtrSet<const Value *, 16> &Visited,
                       unsigned WaveSize) {
   // FIXME: We assume we never cast the mask results of a control flow
@@ -13402,3 +13437,25 @@ void SITargetLowering::emitExpandAtomicRMW(AtomicRMWInst *AI) const {
   AI->replaceAllUsesWith(Loaded);
   AI->eraseFromParent();
 }
+
+LoadInst *
+SITargetLowering::lowerIdempotentRMWIntoFencedLoad(AtomicRMWInst *AI) const {
+  IRBuilder<> Builder(AI);
+  auto Order = AI->getOrdering();
+
+  // The optimization removes store aspect of the atomicrmw. Therefore, cache
+  // must be flushed if the atomic ordering had a release semantics. This is
+  // not necessary a fence, a release fence just coincides to do that flush.
+  // Avoid replacing of an atomicrmw with a release semantics.
+  if (isReleaseOrStronger(Order))
+    return nullptr;
+
+  LoadInst *LI = Builder.CreateAlignedLoad(
+      AI->getType(), AI->getPointerOperand(), AI->getAlign());
+  LI->setAtomic(Order, AI->getSyncScopeID());
+  LI->copyMetadata(*AI);
+  LI->takeName(AI);
+  AI->replaceAllUsesWith(LI);
+  AI->eraseFromParent();
+  return LI;
+}
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.h b/llvm/lib/Target/AMDGPU/SIISelLowering.h
index d9fde1c6adceb..5b5c4bfcdd5f0 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.h
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.h
@@ -193,7 +193,8 @@ class SITargetLowering final : public AMDGPUTargetLowering {
   SDValue performFPMed3ImmCombine(SelectionDAG &DAG, const SDLoc &SL,
                                   SDValue Op0, SDValue Op1) const;
   SDValue performIntMed3ImmCombine(SelectionDAG &DAG, const SDLoc &SL,
-                                   SDValue Op0, SDValue Op1, bool Signed) const;
+                                   SDValue Src, SDValue MinVal, SDValue MaxVal,
+                                   bool Signed) const;
   SDValue performMinMaxCombine(SDNode *N, DAGCombinerInfo &DCI) const;
   SDValue performFMed3Combine(SDNode *N, DAGCombinerInfo &DCI) const;
   SDValue performCvtPkRTZCombine(SDNode *N, DAGCombinerInfo &DCI) const;
@@ -465,8 +466,8 @@ class SITargetLowering final : public AMDGPUTargetLowering {
   Align computeKnownAlignForTargetInstr(GISelKnownBits &Analysis, Register R,
                                         const MachineRegisterInfo &MRI,
                                         unsigned Depth = 0) const override;
-  bool isSDNodeSourceOfDivergence(const SDNode *N,
-    FunctionLoweringInfo *FLI, LegacyDivergenceAnalysis *DA) const override;
+  bool isSDNodeSourceOfDivergence(const SDNode *N, FunctionLoweringInfo *FLI,
+                                  UniformityInfo *UA) const override;
 
   bool hasMemSDNodeUser(SDNode *N) const;
 
@@ -496,6 +497,9 @@ class SITargetLowering final : public AMDGPUTargetLowering {
   shouldExpandAtomicCmpXchgInIR(AtomicCmpXchgInst *AI) const override;
   void emitExpandAtomicRMW(AtomicRMWInst *AI) const override;
 
+  LoadInst *
+  lowerIdempotentRMWIntoFencedLoad(AtomicRMWInst *AI) const override;
+
   const TargetRegisterClass *getRegClassFor(MVT VT,
                                             bool isDivergent) const override;
   bool requiresUniformRegister(MachineFunction &MF,
diff --git a/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp b/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
index d75abd8fed750..a1d13cc491eb7 100644
--- a/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
@@ -4346,7 +4346,7 @@ bool SIInstrInfo::verifyInstruction(const MachineInstr &MI,
 
       // Adjust for packed 16 bit values
       if (D16 && D16->getImm() && !ST.hasUnpackedD16VMem())
-        RegCount >>= 1;
+        RegCount = divideCeil(RegCount, 2);
 
       // Adjust if using LWE or TFE
       if ((LWE && LWE->getImm()) || (TFE && TFE->getImm()))
@@ -4359,7 +4359,7 @@ bool SIInstrInfo::verifyInstruction(const MachineInstr &MI,
         const TargetRegisterClass *DstRC = getOpRegClass(MI, DstIdx);
         uint32_t DstSize = RI.getRegSizeInBits(*DstRC) / 32;
         if (RegCount > DstSize) {
-          ErrInfo = "MIMG instruction returns too many registers for dst "
+          ErrInfo = "Image instruction returns too many registers for dst "
                     "register class";
           return false;
         }
diff --git a/llvm/lib/Target/AMDGPU/SIInstructions.td b/llvm/lib/Target/AMDGPU/SIInstructions.td
index 0683deb71a8b7..c10bbe7367a16 100644
--- a/llvm/lib/Target/AMDGPU/SIInstructions.td
+++ b/llvm/lib/Target/AMDGPU/SIInstructions.td
@@ -10,9 +10,10 @@
 // that are not yet supported remain commented out.
 //===----------------------------------------------------------------------===//
 
-class GCNPat<dag pattern, dag result> : Pat<pattern, result>, GCNPredicateControl {
+class GCNPat<dag pattern, dag result> : Pat<pattern, result>, GCNPredicateControl, GISelFlags;
 
-}
+let GIIgnoreCopies = 1 in
+class GCNPatIgnoreCopies<dag pattern, dag result> : GCNPat<pattern, result>;
 
 class UniformSextInreg<ValueType VT> : PatFrag<
   (ops node:$src),
@@ -2925,6 +2926,31 @@ def : GCNPat <
   (v2i16 (V_LSHL_OR_B32_e64 $src1, (i32 16), (i32 (V_AND_B32_e64 (i32 (V_MOV_B32_e32 (i32 0xffff))), $src0))))
 >;
 
+multiclass V_SAT_PK_Pat<Instruction inst> {
+  def: GCNPat<
+    (v2i16 (DivergentBinFrag<build_vector> (clamp_s16_u8 i16:$lo), (clamp_s16_u8 i16:$hi))),
+    (inst
+      (V_LSHL_OR_B32_e64 VGPR_32:$hi, (S_MOV_B32 (i32 16)),
+        (V_AND_B32_e64 VGPR_32:$lo, (S_MOV_B32 (i32 0xFFFF)))))
+  >;
+
+  def: GCNPatIgnoreCopies<
+    (v2i16 (DivergentBinFrag<smin> (smax v2i16:$src, (build_vector (i16 0), (i16 0))), (build_vector (i16 255), (i16 255)))),
+    (inst VGPR_32:$src)
+  >;
+
+  def: GCNPatIgnoreCopies<
+    (v2i16 (DivergentBinFrag<smax> (smin v2i16:$src, (build_vector (i16 255), (i16 255))), (build_vector (i16 0), (i16 0)))),
+    (inst VGPR_32:$src)
+  >;
+}
+
+let OtherPredicates = [HasTrue16BitInsts] in
+defm : V_SAT_PK_Pat<V_SAT_PK_U8_I16_t16_e64>;
+
+let OtherPredicates = [NotHasTrue16BitInsts] in
+defm : V_SAT_PK_Pat<V_SAT_PK_U8_I16_e64>;
+
 // With multiple uses of the shift, this will duplicate the shift and
 // increase register pressure.
 def : GCNPat <
diff --git a/llvm/lib/Target/AMDGPU/SILoadStoreOptimizer.cpp b/llvm/lib/Target/AMDGPU/SILoadStoreOptimizer.cpp
index 962f3383146c0..536efbbfd9cb7 100644
--- a/llvm/lib/Target/AMDGPU/SILoadStoreOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/SILoadStoreOptimizer.cpp
@@ -2092,7 +2092,7 @@ bool SILoadStoreOptimizer::promoteConstantOffsetToImm(
   // Step1: Find the base-registers and a 64bit constant offset.
   MachineOperand &Base = *TII->getNamedOperand(MI, AMDGPU::OpName::vaddr);
   MemAddress MAddr;
-  if (Visited.find(&MI) == Visited.end()) {
+  if (!Visited.contains(&MI)) {
     processBaseWithConstOffset(Base, MAddr);
     Visited[&MI] = MAddr;
   } else
@@ -2155,7 +2155,7 @@ bool SILoadStoreOptimizer::promoteConstantOffsetToImm(
     const MachineOperand &BaseNext =
       *TII->getNamedOperand(MINext, AMDGPU::OpName::vaddr);
     MemAddress MAddrNext;
-    if (Visited.find(&MINext) == Visited.end()) {
+    if (!Visited.contains(&MINext)) {
       processBaseWithConstOffset(BaseNext, MAddrNext);
       Visited[&MINext] = MAddrNext;
     } else
diff --git a/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.cpp b/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.cpp
index b2a433dd3db9e..427a0d1fdf989 100644
--- a/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.cpp
@@ -119,7 +119,8 @@ SIMachineFunctionInfo::SIMachineFunctionInfo(const Function &F,
   else if (ST.isMesaGfxShader(F))
     ImplicitBufferPtr = true;
 
-  if (!AMDGPU::isGraphics(CC)) {
+  if (!AMDGPU::isGraphics(CC) ||
+      (CC == CallingConv::AMDGPU_CS && ST.hasArchitectedSGPRs())) {
     if (IsKernel || !F.hasFnAttribute("amdgpu-no-workgroup-id-x"))
       WorkGroupIDX = true;
 
@@ -128,7 +129,9 @@ SIMachineFunctionInfo::SIMachineFunctionInfo(const Function &F,
 
     if (!F.hasFnAttribute("amdgpu-no-workgroup-id-z"))
       WorkGroupIDZ = true;
+  }
 
+  if (!AMDGPU::isGraphics(CC)) {
     if (IsKernel || !F.hasFnAttribute("amdgpu-no-workitem-id-x"))
       WorkItemIDX = true;
 
diff --git a/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h b/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h
index 80d500d52c709..c7ac0de07a7b7 100644
--- a/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h
+++ b/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h
@@ -578,7 +578,7 @@ class SIMachineFunctionInfo final : public AMDGPUMachineFunction {
   // Check if an entry created for \p Reg in PrologEpilogSGPRSpills. Return true
   // on success and false otherwise.
   bool hasPrologEpilogSGPRSpillEntry(Register Reg) const {
-    return PrologEpilogSGPRSpills.find(Reg) != PrologEpilogSGPRSpills.end();
+    return PrologEpilogSGPRSpills.contains(Reg);
   }
 
   // Get the scratch SGPR if allocated to save/restore \p Reg.
diff --git a/llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp b/llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp
index 68e592b65845d..47b81d7f5bf8a 100644
--- a/llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp
+++ b/llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp
@@ -2209,8 +2209,13 @@ bool SIMemoryLegalizer::expandAtomicFence(const SIMemOpInfo &MOI,
   bool Changed = false;
 
   if (MOI.isAtomic()) {
-    if (MOI.getOrdering() == AtomicOrdering::Acquire ||
-        MOI.getOrdering() == AtomicOrdering::Release ||
+    if (MOI.getOrdering() == AtomicOrdering::Acquire)
+      Changed |= CC->insertWait(MI, MOI.getScope(), MOI.getOrderingAddrSpace(),
+                                SIMemOp::LOAD | SIMemOp::STORE,
+                                MOI.getIsCrossAddressSpaceOrdering(),
+                                Position::BEFORE);
+
+    if (MOI.getOrdering() == AtomicOrdering::Release ||
         MOI.getOrdering() == AtomicOrdering::AcquireRelease ||
         MOI.getOrdering() == AtomicOrdering::SequentiallyConsistent)
       /// TODO: This relies on a barrier always generating a waitcnt
diff --git a/llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp b/llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp
index f7ce581f97363..3bb93b72c3473 100644
--- a/llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp
@@ -654,8 +654,7 @@ BitVector SIRegisterInfo::getReservedRegs(const MachineFunction &MF) const {
 
   // Reserve all the AGPRs if there are no instructions to use it.
   if (!ST.hasMAIInsts()) {
-    for (unsigned i = 0; i < MaxNumAGPRs; ++i) {
-      unsigned Reg = AMDGPU::AGPR_32RegClass.getRegister(i);
+    for (MCRegister Reg : AMDGPU::AGPR_32RegClass) {
       reserveRegisterTuples(Reserved, Reg);
     }
   }
diff --git a/llvm/lib/Target/AMDGPU/SMInstructions.td b/llvm/lib/Target/AMDGPU/SMInstructions.td
index 715d4076c4e08..29f868ab18526 100644
--- a/llvm/lib/Target/AMDGPU/SMInstructions.td
+++ b/llvm/lib/Target/AMDGPU/SMInstructions.td
@@ -132,7 +132,6 @@ class SM_Store_Pseudo <string opName, RegisterClass baseClass,
                                    offsets.Ins, (ins CPol:$cpol)),
               " $sdata, $sbase, " # offsets.Asm # "$cpol"> {
   RegisterClass BaseClass = baseClass;
-  RegisterClass SrcClass = srcClass;
   let mayLoad = 0;
   let mayStore = 1;
   let has_glc = 1;
@@ -465,18 +464,14 @@ class SMRD_Real_si <bits<5> op, SM_Pseudo ps>
   let Inst{31-27} = 0x18; //encoding
 }
 
-multiclass SM_Real_Loads_si<bits<5> op, string ps,
-                            SM_Load_Pseudo immPs = !cast<SM_Load_Pseudo>(ps#_IMM),
-                            SM_Load_Pseudo sgprPs = !cast<SM_Load_Pseudo>(ps#_SGPR)> {
-
+multiclass SM_Real_Loads_si<bits<5> op, string ps> {
+  defvar immPs = !cast<SM_Load_Pseudo>(ps#_IMM);
   def _IMM_si : SMRD_Real_si <op, immPs> {
     let InOperandList = (ins immPs.BaseClass:$sbase, smrd_offset_8:$offset, CPol:$cpol);
   }
 
-  def _SGPR_si : SMRD_Real_si <op, sgprPs> {
-    let InOperandList = (ins sgprPs.BaseClass:$sbase, SReg_32:$soffset, CPol:$cpol);
-  }
-
+  defvar sgprPs = !cast<SM_Load_Pseudo>(ps#_SGPR);
+  def _SGPR_si : SMRD_Real_si <op, sgprPs>;
 }
 
 defm S_LOAD_DWORD           : SM_Real_Loads_si <0x00, "S_LOAD_DWORD">;
@@ -543,10 +538,7 @@ class SMEM_Real_vi <bits<8> op, SM_Pseudo ps>
 }
 
 class SMEM_Real_Load_vi<bits<8> op, string ps, OffsetMode offsets>
-    : SMEM_Real_vi<op, !cast<SM_Pseudo>(ps # offsets.Variant)> {
-  RegisterClass BaseClass = !cast<SM_Load_Pseudo>(ps # offsets.Variant).BaseClass;
-  let InOperandList = !con((ins BaseClass:$sbase), offsets.Ins, (ins CPol:$cpol));
-}
+    : SMEM_Real_vi<op, !cast<SM_Pseudo>(ps # offsets.Variant)>;
 
 // The alternative GFX9 SGPR encoding using soffset to encode the
 // offset register. Not available in assembler and goes to the GFX9
@@ -577,12 +569,7 @@ class SMEM_Real_Store_Base_vi <bits<8> op, SM_Pseudo ps> : SMEM_Real_vi <op, ps>
 }
 
 class SMEM_Real_Store_vi <bits<8> op, string ps, OffsetMode offsets>
-    : SMEM_Real_Store_Base_vi <op, !cast<SM_Pseudo>(ps # offsets.Variant)> {
-  RegisterClass SrcClass = !cast<SM_Store_Pseudo>(ps # offsets.Variant).SrcClass;
-  RegisterClass BaseClass = !cast<SM_Store_Pseudo>(ps # offsets.Variant).BaseClass;
-  let InOperandList = !con((ins SrcClass:$sdata, BaseClass:$sbase),
-                           offsets.Ins, (ins CPol:$cpol));
-}
+    : SMEM_Real_Store_Base_vi <op, !cast<SM_Pseudo>(ps # offsets.Variant)>;
 
 multiclass SM_Real_Stores_vi<bits<8> op, string ps> {
   def _IMM_vi : SMEM_Real_Store_vi <op, ps, IMM_Offset>;
@@ -982,10 +969,7 @@ class SMEM_Real_gfx10<bits<8> op, SM_Pseudo ps>
 }
 
 class SMEM_Real_Load_gfx10<bits<8> op, string ps, OffsetMode offsets>
-    : SMEM_Real_gfx10<op, !cast<SM_Pseudo>(ps # offsets.Variant)> {
-  RegisterClass BaseClass = !cast<SM_Load_Pseudo>(ps # offsets.Variant).BaseClass;
-  let InOperandList = !con((ins BaseClass:$sbase), offsets.Ins, (ins CPol:$cpol));
-}
+    : SMEM_Real_gfx10<op, !cast<SM_Pseudo>(ps # offsets.Variant)>;
 
 multiclass SM_Real_Loads_gfx10<bits<8> op, string ps> {
   def _IMM_gfx10 : SMEM_Real_Load_gfx10<op, ps, IMM_Offset>;
@@ -1000,21 +984,15 @@ class SMEM_Real_Store_gfx10<bits<8> op, SM_Pseudo ps> : SMEM_Real_gfx10<op, ps>
   let Inst{12-6} = !if(ps.has_sdst, sdata{6-0}, ?);
 }
 
-multiclass SM_Real_Stores_gfx10<bits<8> op, string ps,
-                                SM_Store_Pseudo immPs = !cast<SM_Store_Pseudo>(ps#_IMM),
-                                SM_Store_Pseudo sgprPs = !cast<SM_Store_Pseudo>(ps#_SGPR)> {
-  def _IMM_gfx10 : SMEM_Real_Store_gfx10 <op, immPs> {
-    let InOperandList = (ins immPs.SrcClass:$sdata, immPs.BaseClass:$sbase, smem_offset:$offset, CPol:$cpol);
-  }
+multiclass SM_Real_Stores_gfx10<bits<8> op, string ps> {
+  defvar immPs = !cast<SM_Store_Pseudo>(ps#_IMM);
+  def _IMM_gfx10 : SMEM_Real_Store_gfx10 <op, immPs>;
 
-  def _SGPR_gfx10 : SMEM_Real_Store_gfx10 <op, sgprPs> {
-    let InOperandList = (ins sgprPs.SrcClass:$sdata, sgprPs.BaseClass:$sbase, SReg_32:$soffset, CPol:$cpol);
-  }
+  defvar sgprPs = !cast<SM_Store_Pseudo>(ps#_SGPR);
+  def _SGPR_gfx10 : SMEM_Real_Store_gfx10 <op, sgprPs>;
 
-  def _SGPR_IMM_gfx10 : SMEM_Real_Store_gfx10 <op, !cast<SM_Store_Pseudo>(ps#_SGPR_IMM)> {
-    let InOperandList = (ins sgprPs.SrcClass:$sdata, sgprPs.BaseClass:$sbase,
-                             SReg_32:$soffset, smem_offset_mod:$offset, CPol:$cpol);
-  }
+  defvar sgprImmPs = !cast<SM_Store_Pseudo>(ps#_SGPR_IMM);
+  def _SGPR_IMM_gfx10 : SMEM_Real_Store_gfx10 <op, sgprImmPs>;
 }
 
 defm S_LOAD_DWORD            : SM_Real_Loads_gfx10<0x000, "S_LOAD_DWORD">;
@@ -1185,10 +1163,7 @@ class SMEM_Real_gfx11<bits<8> op, SM_Pseudo ps, string opName = ps.Mnemonic> :
 }
 
 class SMEM_Real_Load_gfx11<bits<8> op, string ps, string opName, OffsetMode offsets> :
-    SMEM_Real_gfx11<op, !cast<SM_Pseudo>(ps # offsets.Variant), opName> {
-  RegisterClass BaseClass = !cast<SM_Load_Pseudo>(ps # offsets.Variant).BaseClass;
-  let InOperandList = !con((ins BaseClass:$sbase), offsets.Ins, (ins CPol:$cpol));
-}
+    SMEM_Real_gfx11<op, !cast<SM_Pseudo>(ps # offsets.Variant), opName>;
 
 multiclass SM_Real_Loads_gfx11<bits<8> op, string ps, string opName> {
   def _IMM_gfx11 : SMEM_Real_Load_gfx11<op, ps, opName, IMM_Offset>;
diff --git a/llvm/lib/Target/AMDGPU/VOP1Instructions.td b/llvm/lib/Target/AMDGPU/VOP1Instructions.td
index e6f9034829951..72ed1cc95a112 100644
--- a/llvm/lib/Target/AMDGPU/VOP1Instructions.td
+++ b/llvm/lib/Target/AMDGPU/VOP1Instructions.td
@@ -124,24 +124,24 @@ multiclass VOP1Inst <string opName, VOPProfile P,
     def _e64 : VOP3InstBase <opName, P, node>;
   }
 
-  foreach _ = BoolToList<P.HasExtSDWA>.ret in
+  if P.HasExtSDWA then
     def _sdwa : VOP1_SDWA_Pseudo <opName, P>;
 
-  foreach _ = BoolToList<P.HasExtDPP>.ret in
+  if P.HasExtDPP then
     def _dpp : VOP1_DPP_Pseudo <opName, P>;
 
   let SubtargetPredicate = isGFX11Plus in {
-    foreach _ = BoolToList<P.HasExtVOP3DPP>.ret in
+    if P.HasExtVOP3DPP then
       def _e64_dpp  : VOP3_DPP_Pseudo <opName, P>;
   } // End SubtargetPredicate = isGFX11Plus
 
   def : MnemonicAlias<opName#"_e32", opName>, LetDummies;
   def : MnemonicAlias<opName#"_e64", opName>, LetDummies;
 
-  foreach _ = BoolToList<P.HasExtSDWA>.ret in
+  if P.HasExtSDWA then
     def : MnemonicAlias<opName#"_sdwa", opName>, LetDummies;
 
-  foreach _ = BoolToList<P.HasExtDPP>.ret in
+  if P.HasExtDPP then
     def : MnemonicAlias<opName#"_dpp", opName, AMDGPUAsmVariants.DPP>, LetDummies;
 }
 
@@ -844,7 +844,7 @@ let AssemblerPredicate = isGFX10Only, DecoderNamespace = "GFX10" in {
       VOP3e_gfx10<{0, 1, 1, op{6-0}}, !cast<VOP3_Pseudo>(NAME#"_e64").Pfl>;
   }
   multiclass VOP1_Real_sdwa_gfx10<bits<9> op> {
-    foreach _ = BoolToList<!cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA9>.ret in
+    if !cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA9 then
     def _sdwa_gfx10 :
       VOP_SDWA10_Real<!cast<VOP1_SDWA_Pseudo>(NAME#"_sdwa")>,
       VOP1_SDWA9Ae<op{7-0}, !cast<VOP1_SDWA_Pseudo>(NAME#"_sdwa").Pfl> {
@@ -852,13 +852,13 @@ let AssemblerPredicate = isGFX10Only, DecoderNamespace = "GFX10" in {
     }
   }
   multiclass VOP1_Real_dpp_gfx10<bits<9> op> {
-    foreach _ = BoolToList<!cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExt32BitDPP>.ret in
+    if !cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExt32BitDPP then
     def _dpp_gfx10 : VOP1_DPP16<op{7-0}, !cast<VOP1_DPP_Pseudo>(NAME#"_dpp"), SIEncodingFamily.GFX10> {
       let DecoderNamespace = "SDWA10";
     }
   }
   multiclass VOP1_Real_dpp8_gfx10<bits<9> op> {
-    foreach _ = BoolToList<!cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExt32BitDPP>.ret in
+    if !cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExt32BitDPP then
     def _dpp8_gfx10 : VOP1_DPP8<op{7-0}, !cast<VOP1_Pseudo>(NAME#"_e32")> {
       let DecoderNamespace = "DPP8";
     }
@@ -1064,17 +1064,17 @@ multiclass VOP1_Real_e32e64_vi <bits<10> op> {
 multiclass VOP1_Real_vi <bits<10> op> {
   defm NAME : VOP1_Real_e32e64_vi <op>;
 
-  foreach _ = BoolToList<!cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA>.ret in
+  if !cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA then
   def _sdwa_vi :
     VOP_SDWA_Real <!cast<VOP1_SDWA_Pseudo>(NAME#"_sdwa")>,
     VOP1_SDWAe <op{7-0}, !cast<VOP1_SDWA_Pseudo>(NAME#"_sdwa").Pfl>;
 
-  foreach _ = BoolToList<!cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA9>.ret in
+  if !cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA9 then
   def _sdwa_gfx9 :
     VOP_SDWA9_Real <!cast<VOP1_SDWA_Pseudo>(NAME#"_sdwa")>,
     VOP1_SDWA9Ae <op{7-0}, !cast<VOP1_SDWA_Pseudo>(NAME#"_sdwa").Pfl>;
 
-  foreach _ = BoolToList<!cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExtDPP>.ret in
+  if !cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExtDPP then
     def _dpp_vi :
       VOP_DPP_Real<!cast<VOP1_DPP_Pseudo>(NAME#"_dpp"), SIEncodingFamily.VI>,
       VOP1_DPPe<op{7-0}, !cast<VOP1_DPP_Pseudo>(NAME#"_dpp")>;
@@ -1238,12 +1238,12 @@ multiclass VOP1_Real_gfx9 <bits<10> op> {
     defm NAME : VOP1_Real_e32e64_vi <op>;
   }
 
-  foreach _ = BoolToList<!cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA9>.ret in
+  if !cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA9 then
   def _sdwa_gfx9 :
     VOP_SDWA9_Real <!cast<VOP1_SDWA_Pseudo>(NAME#"_sdwa")>,
     VOP1_SDWA9Ae <op{7-0}, !cast<VOP1_SDWA_Pseudo>(NAME#"_sdwa").Pfl>;
 
-  foreach _ = BoolToList<!cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExtDPP>.ret in
+  if !cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExtDPP then
     def _dpp_gfx9 :
       VOP_DPP_Real<!cast<VOP1_DPP_Pseudo>(NAME#"_dpp"), SIEncodingFamily.GFX9>,
       VOP1_DPPe<op{7-0}, !cast<VOP1_DPP_Pseudo>(NAME#"_dpp")>;
@@ -1255,14 +1255,14 @@ multiclass VOP1_Real_NoDstSel_SDWA_gfx9 <bits<10> op> {
     defm NAME : VOP1_Real_e32e64_vi <op>;
   }
 
-  foreach _ = BoolToList<!cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA9>.ret in
+  if !cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA9 then
   def _sdwa_gfx9 :
     VOP_SDWA9_Real <!cast<VOP1_SDWA_Pseudo>(NAME#"_sdwa")>,
     VOP1_SDWA9Ae <op{7-0}, !cast<VOP1_SDWA_Pseudo>(NAME#"_sdwa").Pfl> {
       let Inst{42-40} = 6;
     }
 
-  foreach _ = BoolToList<!cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExtDPP>.ret in
+  if !cast<VOP1_Pseudo>(NAME#"_e32").Pfl.HasExtDPP then
     def _dpp_gfx9 :
       VOP_DPP_Real<!cast<VOP1_DPP_Pseudo>(NAME#"_dpp"), SIEncodingFamily.GFX9>,
       VOP1_DPPe<op{7-0}, !cast<VOP1_DPP_Pseudo>(NAME#"_dpp")>;
diff --git a/llvm/lib/Target/AMDGPU/VOP2Instructions.td b/llvm/lib/Target/AMDGPU/VOP2Instructions.td
index d8d7682645032..446e7d01c1d81 100644
--- a/llvm/lib/Target/AMDGPU/VOP2Instructions.td
+++ b/llvm/lib/Target/AMDGPU/VOP2Instructions.td
@@ -157,7 +157,7 @@ multiclass VOP2Inst_e64<string opName,
                Commutable_REV<revOp#"_e64", !eq(revOp, opName)>;
 
     let SubtargetPredicate = isGFX11Plus in {
-      foreach _ = BoolToList<P.HasExtVOP3DPP>.ret in
+      if P.HasExtVOP3DPP then
         def _e64_dpp  : VOP3_DPP_Pseudo <opName, P>;
     } // End SubtargetPredicate = isGFX11Plus
   } // End renamedInGFX9 = GFX9Renamed
@@ -167,7 +167,7 @@ multiclass VOP2Inst_sdwa<string opName,
                          VOPProfile P,
                          bit GFX9Renamed = 0> {
   let renamedInGFX9 = GFX9Renamed in {
-    foreach _ = BoolToList<P.HasExtSDWA>.ret in
+    if P.HasExtSDWA then
       def _sdwa : VOP2_SDWA_Pseudo <opName, P>;
   } // End renamedInGFX9 = GFX9Renamed
 }
@@ -181,7 +181,7 @@ multiclass VOP2Inst<string opName,
     VOP2Inst_e64<opName, P, node, revOp, GFX9Renamed>,
     VOP2Inst_sdwa<opName, P, GFX9Renamed> {
   let renamedInGFX9 = GFX9Renamed in {
-    foreach _ = BoolToList<P.HasExtDPP>.ret in
+    if P.HasExtDPP then
       def _dpp  : VOP2_DPP_Pseudo <opName, P>;
   }
 }
@@ -227,7 +227,7 @@ multiclass VOP2Inst_VOPD<string opName,
     VOP2Inst_e64<opName, P, node, revOp, GFX9Renamed>,
     VOP2Inst_sdwa<opName, P, GFX9Renamed> {
   let renamedInGFX9 = GFX9Renamed in {
-    foreach _ = BoolToList<P.HasExtDPP>.ret in
+    if P.HasExtDPP then
       def _dpp  : VOP2_DPP_Pseudo <opName, P>;
   }
 }
@@ -246,11 +246,11 @@ multiclass VOP2bInst <string opName,
           let usesCustomInserter = true;
         }
 
-        foreach _ = BoolToList<P.HasExtSDWA>.ret in
+        if P.HasExtSDWA then
           def _sdwa  : VOP2_SDWA_Pseudo <opName, P> {
             let AsmMatchConverter = "cvtSdwaVOP2b";
           }
-        foreach _ = BoolToList<P.HasExtDPP>.ret in
+        if P.HasExtDPP then
           def _dpp  : VOP2_DPP_Pseudo <opName, P>;
       } // End Uses = !if(useSGPRInput, [VCC, EXEC], [EXEC]), Defs = [VCC]
 
@@ -258,7 +258,7 @@ multiclass VOP2bInst <string opName,
                  Commutable_REV<revOp#"_e64", !eq(revOp, opName)>;
 
       let SubtargetPredicate = isGFX11Plus in {
-        foreach _ = BoolToList<P.HasExtVOP3DPP>.ret in
+        if P.HasExtVOP3DPP then
           def _e64_dpp  : VOP3_DPP_Pseudo <opName, P>;
       } // End SubtargetPredicate = isGFX11Plus
     }
@@ -297,12 +297,12 @@ multiclass
                    Commutable_REV<revOp#"_e32", !eq(revOp, opName)>,
                    VOPD_Component<VOPDOp, VOPDName>;
 
-      foreach _ = BoolToList<P.HasExtSDWA>.ret in
+      if P.HasExtSDWA then
         def _sdwa : VOP2_SDWA_Pseudo <opName, P> {
           let AsmMatchConverter = "cvtSdwaVOP2e";
         }
 
-      foreach _ = BoolToList<P.HasExtDPP>.ret in
+      if P.HasExtDPP then
         def _dpp  : VOP2_DPP_Pseudo <opName, P>;
     }
 
@@ -312,7 +312,7 @@ multiclass
     }
 
     let SubtargetPredicate = isGFX11Plus in {
-      foreach _ = BoolToList<P.HasExtVOP3DPP>.ret in
+      if P.HasExtVOP3DPP then
         def _e64_dpp  : VOP3_DPP_Pseudo <opName, P>;
     } // End SubtargetPredicate = isGFX11Plus
   }
@@ -1265,13 +1265,13 @@ let AssemblerPredicate = isGFX11Only, DecoderNamespace = "GFX11" in {
       VOP3e_gfx11<{0, 1, 0, 0, op{5-0}}, !cast<VOP3_Pseudo>(NAME#"_e64").Pfl>;
   }
   multiclass VOP2_Real_dpp_gfx11<bits<6> op> {
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExtDPP>.ret in
+    if !cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExtDPP then
     def _dpp_gfx11 : VOP2_DPP16<op, !cast<VOP2_DPP_Pseudo>(NAME#"_dpp"), SIEncodingFamily.GFX11> {
       let DecoderNamespace = "DPPGFX11";
     }
   }
   multiclass VOP2_Real_dpp8_gfx11<bits<6> op> {
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExtDPP>.ret in
+    if !cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExtDPP then
     def _dpp8_gfx11 : VOP2_DPP8<op, !cast<VOP2_Pseudo>(NAME#"_e32")> {
       let DecoderNamespace = "DPP8GFX11";
     }
@@ -1301,7 +1301,7 @@ let AssemblerPredicate = isGFX11Only, DecoderNamespace = "GFX11" in {
   multiclass VOP2_Real_dpp_with_name_gfx11<bits<6> op, string opName,
                                            string asmName> {
     defvar ps = !cast<VOP2_Pseudo>(opName#"_e32");
-    foreach _ = BoolToList<ps.Pfl.HasExtDPP>.ret in
+    if ps.Pfl.HasExtDPP then
     def _dpp_gfx11 : VOP2_DPP16<op, !cast<VOP2_DPP_Pseudo>(opName#"_dpp"),
         SIEncodingFamily.GFX11> {
       let AsmString = asmName # ps.Pfl.AsmDPP16;
@@ -1311,7 +1311,7 @@ let AssemblerPredicate = isGFX11Only, DecoderNamespace = "GFX11" in {
   multiclass VOP2_Real_dpp8_with_name_gfx11<bits<6> op, string opName,
                                             string asmName> {
     defvar ps = !cast<VOP2_Pseudo>(opName#"_e32");
-    foreach _ = BoolToList<ps.Pfl.HasExtDPP>.ret in
+    if ps.Pfl.HasExtDPP then
     def _dpp8_gfx11 : VOP2_DPP8<op, ps> {
       let AsmString = asmName # ps.Pfl.AsmDPP8;
       let DecoderNamespace = "DPP8GFX11";
@@ -1328,14 +1328,14 @@ let AssemblerPredicate = isGFX11Only, DecoderNamespace = "GFX11" in {
       }
   }
   multiclass VOP2be_Real_dpp_gfx11<bits<6> op, string opName, string asmName> {
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtDPP>.ret in
+    if !cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtDPP then
     def _dpp_gfx11 :
       VOP2_DPP16<op, !cast<VOP2_DPP_Pseudo>(opName#"_dpp"), SIEncodingFamily.GFX11, asmName> {
         string AsmDPP = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP16;
         let AsmString = asmName # !subst(", vcc", "", AsmDPP);
         let DecoderNamespace = "DPPGFX11";
       }
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtDPP>.ret in
+    if !cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtDPP then
     def _dpp_w32_gfx11 :
       Base_VOP2_DPP16<op, !cast<VOP2_DPP_Pseudo>(opName#"_dpp"), asmName> {
         string AsmDPP = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP16;
@@ -1343,7 +1343,7 @@ let AssemblerPredicate = isGFX11Only, DecoderNamespace = "GFX11" in {
         let isAsmParserOnly = 1;
         let WaveSizePredicate = isWave32;
       }
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtDPP>.ret in
+    if !cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtDPP then
     def _dpp_w64_gfx11 :
       Base_VOP2_DPP16<op, !cast<VOP2_DPP_Pseudo>(opName#"_dpp"), asmName> {
         string AsmDPP = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP16;
@@ -1353,14 +1353,14 @@ let AssemblerPredicate = isGFX11Only, DecoderNamespace = "GFX11" in {
       }
   }
   multiclass VOP2be_Real_dpp8_gfx11<bits<6> op, string opName, string asmName> {
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtDPP>.ret in
+    if !cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtDPP then
     def _dpp8_gfx11 :
       VOP2_DPP8<op, !cast<VOP2_Pseudo>(opName#"_e32")> {
         string AsmDPP8 = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP8;
         let AsmString = asmName # !subst(", vcc", "", AsmDPP8);
         let DecoderNamespace = "DPP8GFX11";
       }
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtDPP>.ret in
+    if !cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtDPP then
     def _dpp8_w32_gfx11 :
       VOP2_DPP8<op, !cast<VOP2_Pseudo>(opName#"_e32")> {
         string AsmDPP8 = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP8;
@@ -1368,7 +1368,7 @@ let AssemblerPredicate = isGFX11Only, DecoderNamespace = "GFX11" in {
         let isAsmParserOnly = 1;
         let WaveSizePredicate = isWave32;
       }
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtDPP>.ret in
+    if !cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtDPP then
     def _dpp8_w64_gfx11 :
       VOP2_DPP8<op, !cast<VOP2_Pseudo>(opName#"_e32")> {
         string AsmDPP8 = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP8;
@@ -1532,7 +1532,7 @@ let AssemblerPredicate = isGFX10Only, DecoderNamespace = "GFX10" in {
       VOP3e_gfx10<{0, 1, 0, 0, op{5-0}}, !cast<VOP3_Pseudo>(NAME#"_e64").Pfl>;
   }
   multiclass VOP2_Real_sdwa_gfx10<bits<6> op> {
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA9>.ret in
+    if !cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA9 then
     def _sdwa_gfx10 :
       VOP_SDWA10_Real<!cast<VOP2_SDWA_Pseudo>(NAME#"_sdwa")>,
       VOP2_SDWA9Ae<op{5-0}, !cast<VOP2_SDWA_Pseudo>(NAME#"_sdwa").Pfl> {
@@ -1540,13 +1540,13 @@ let AssemblerPredicate = isGFX10Only, DecoderNamespace = "GFX10" in {
     }
   }
   multiclass VOP2_Real_dpp_gfx10<bits<6> op> {
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExt32BitDPP>.ret in
+    if !cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExt32BitDPP then
     def _dpp_gfx10 : VOP2_DPP16<op, !cast<VOP2_DPP_Pseudo>(NAME#"_dpp"), SIEncodingFamily.GFX10> {
       let DecoderNamespace = "SDWA10";
     }
   }
   multiclass VOP2_Real_dpp8_gfx10<bits<6> op> {
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExt32BitDPP>.ret in
+    if !cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExt32BitDPP then
     def _dpp8_gfx10 : VOP2_DPP8<op, !cast<VOP2_Pseudo>(NAME#"_e32")> {
       let DecoderNamespace = "DPP8";
     }
@@ -1575,7 +1575,7 @@ let AssemblerPredicate = isGFX10Only, DecoderNamespace = "GFX10" in {
   let DecoderNamespace = "SDWA10" in {
     multiclass VOP2_Real_sdwa_gfx10_with_name<bits<6> op, string opName,
                                               string asmName> {
-      foreach _ = BoolToList<!cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtSDWA9>.ret in
+      if !cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtSDWA9 then
       def _sdwa_gfx10 :
         VOP_SDWA10_Real<!cast<VOP2_SDWA_Pseudo>(opName#"_sdwa")>,
         VOP2_SDWA9Ae<op{5-0}, !cast<VOP2_SDWA_Pseudo>(opName#"_sdwa").Pfl> {
@@ -1585,7 +1585,7 @@ let AssemblerPredicate = isGFX10Only, DecoderNamespace = "GFX10" in {
     }
     multiclass VOP2_Real_dpp_gfx10_with_name<bits<6> op, string opName,
                                              string asmName> {
-      foreach _ = BoolToList<!cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExt32BitDPP>.ret in
+      if !cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExt32BitDPP then
       def _dpp_gfx10 : VOP2_DPP16<op, !cast<VOP2_DPP_Pseudo>(opName#"_dpp"), SIEncodingFamily.GFX10> {
         VOP2_Pseudo ps = !cast<VOP2_Pseudo>(opName#"_e32");
         let AsmString = asmName # ps.Pfl.AsmDPP16;
@@ -1593,7 +1593,7 @@ let AssemblerPredicate = isGFX10Only, DecoderNamespace = "GFX10" in {
     }
     multiclass VOP2_Real_dpp8_gfx10_with_name<bits<6> op, string opName,
                                               string asmName> {
-      foreach _ = BoolToList<!cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExt32BitDPP>.ret in
+      if !cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExt32BitDPP then
       def _dpp8_gfx10 : VOP2_DPP8<op, !cast<VOP2_Pseudo>(opName#"_e32")> {
         VOP2_Pseudo ps = !cast<VOP2_Pseudo>(opName#"_e32");
         let AsmString = asmName # ps.Pfl.AsmDPP8;
@@ -1621,7 +1621,7 @@ let AssemblerPredicate = isGFX10Only, DecoderNamespace = "GFX10" in {
       }
   }
   multiclass VOP2be_Real_sdwa_gfx10<bits<6> op, string opName, string asmName> {
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtSDWA9>.ret in
+    if !cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtSDWA9 then
     def _sdwa_gfx10 :
       VOP_SDWA10_Real<!cast<VOP2_SDWA_Pseudo>(opName#"_sdwa")>,
       VOP2_SDWA9Ae<op{5-0}, !cast<VOP2_SDWA_Pseudo>(opName#"_sdwa").Pfl> {
@@ -1629,7 +1629,7 @@ let AssemblerPredicate = isGFX10Only, DecoderNamespace = "GFX10" in {
         let AsmString = asmName # !subst(", vcc", "", Ps.AsmOperands);
         let DecoderNamespace = "SDWA10";
       }
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtSDWA9>.ret in
+    if !cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtSDWA9 then
     def _sdwa_w32_gfx10 :
       Base_VOP_SDWA10_Real<!cast<VOP2_SDWA_Pseudo>(opName#"_sdwa")>,
       VOP2_SDWA9Ae<op{5-0}, !cast<VOP2_SDWA_Pseudo>(opName#"_sdwa").Pfl> {
@@ -1639,7 +1639,7 @@ let AssemblerPredicate = isGFX10Only, DecoderNamespace = "GFX10" in {
         let DecoderNamespace = "SDWA10";
         let WaveSizePredicate = isWave32;
       }
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtSDWA9>.ret in
+    if !cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExtSDWA9 then
     def _sdwa_w64_gfx10 :
       Base_VOP_SDWA10_Real<!cast<VOP2_SDWA_Pseudo>(opName#"_sdwa")>,
       VOP2_SDWA9Ae<op{5-0}, !cast<VOP2_SDWA_Pseudo>(opName#"_sdwa").Pfl> {
@@ -1651,14 +1651,14 @@ let AssemblerPredicate = isGFX10Only, DecoderNamespace = "GFX10" in {
       }
   }
   multiclass VOP2be_Real_dpp_gfx10<bits<6> op, string opName, string asmName> {
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExt32BitDPP>.ret in
+    if !cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExt32BitDPP then
     def _dpp_gfx10 :
       VOP2_DPP16<op, !cast<VOP2_DPP_Pseudo>(opName#"_dpp"), SIEncodingFamily.GFX10, asmName> {
         string AsmDPP = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP16;
         let AsmString = asmName # !subst(", vcc", "", AsmDPP);
         let DecoderNamespace = "SDWA10";
       }
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExt32BitDPP>.ret in
+    if !cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExt32BitDPP then
     def _dpp_w32_gfx10 :
       Base_VOP2_DPP16<op, !cast<VOP2_DPP_Pseudo>(opName#"_dpp"), asmName> {
         string AsmDPP = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP16;
@@ -1666,7 +1666,7 @@ let AssemblerPredicate = isGFX10Only, DecoderNamespace = "GFX10" in {
         let isAsmParserOnly = 1;
         let WaveSizePredicate = isWave32;
       }
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExt32BitDPP>.ret in
+    if !cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExt32BitDPP then
     def _dpp_w64_gfx10 :
       Base_VOP2_DPP16<op, !cast<VOP2_DPP_Pseudo>(opName#"_dpp"), asmName> {
         string AsmDPP = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP16;
@@ -1676,14 +1676,14 @@ let AssemblerPredicate = isGFX10Only, DecoderNamespace = "GFX10" in {
       }
   }
   multiclass VOP2be_Real_dpp8_gfx10<bits<6> op, string opName, string asmName> {
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExt32BitDPP>.ret in
+    if !cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExt32BitDPP then
     def _dpp8_gfx10 :
       VOP2_DPP8<op, !cast<VOP2_Pseudo>(opName#"_e32")> {
         string AsmDPP8 = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP8;
         let AsmString = asmName # !subst(", vcc", "", AsmDPP8);
         let DecoderNamespace = "DPP8";
       }
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExt32BitDPP>.ret in
+    if !cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExt32BitDPP then
     def _dpp8_w32_gfx10 :
       VOP2_DPP8<op, !cast<VOP2_Pseudo>(opName#"_e32")> {
         string AsmDPP8 = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP8;
@@ -1691,7 +1691,7 @@ let AssemblerPredicate = isGFX10Only, DecoderNamespace = "GFX10" in {
         let isAsmParserOnly = 1;
         let WaveSizePredicate = isWave32;
       }
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExt32BitDPP>.ret in
+    if !cast<VOP2_Pseudo>(opName#"_e32").Pfl.HasExt32BitDPP then
     def _dpp8_w64_gfx10 :
       VOP2_DPP8<op, !cast<VOP2_Pseudo>(opName#"_e32")> {
         string AsmDPP8 = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP8;
@@ -2013,14 +2013,14 @@ multiclass Base_VOP2_Real_e32e64_vi <bits<6> op> :
 } // End AssemblerPredicate = isGFX8GFX9, DecoderNamespace = "GFX8"
 
 multiclass VOP2_SDWA_Real <bits<6> op> {
-  foreach _ = BoolToList<!cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA>.ret in
+  if !cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA then
   def _sdwa_vi :
     VOP_SDWA_Real <!cast<VOP2_SDWA_Pseudo>(NAME#"_sdwa")>,
     VOP2_SDWAe <op{5-0}, !cast<VOP2_SDWA_Pseudo>(NAME#"_sdwa").Pfl>;
 }
 
 multiclass VOP2_SDWA9_Real <bits<6> op> {
-  foreach _ = BoolToList<!cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA9>.ret in
+  if !cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA9 then
   def _sdwa_gfx9 :
     VOP_SDWA9_Real <!cast<VOP2_SDWA_Pseudo>(NAME#"_sdwa")>,
     VOP2_SDWA9Ae <op{5-0}, !cast<VOP2_SDWA_Pseudo>(NAME#"_sdwa").Pfl>;
@@ -2043,14 +2043,14 @@ multiclass VOP2be_Real_e32e64_vi_only <bits<6> op, string OpName, string AsmName
       let AsmString = AsmName # ps.AsmOperands;
       let DecoderNamespace = "GFX8";
     }
-  foreach _ = BoolToList<!cast<VOP2_Pseudo>(OpName#"_e32").Pfl.HasExtSDWA>.ret in
+  if !cast<VOP2_Pseudo>(OpName#"_e32").Pfl.HasExtSDWA then
     def _sdwa_vi :
       VOP_SDWA_Real <!cast<VOP2_SDWA_Pseudo>(OpName#"_sdwa")>,
       VOP2_SDWAe <op{5-0}, !cast<VOP2_SDWA_Pseudo>(OpName#"_sdwa").Pfl> {
         VOP2_SDWA_Pseudo ps = !cast<VOP2_SDWA_Pseudo>(OpName#"_sdwa");
         let AsmString = AsmName # ps.AsmOperands;
       }
-  foreach _ = BoolToList<!cast<VOP2_Pseudo>(OpName#"_e32").Pfl.HasExtDPP>.ret in
+  if !cast<VOP2_Pseudo>(OpName#"_e32").Pfl.HasExtDPP then
     def _dpp_vi :
       VOP_DPP_Real<!cast<VOP2_DPP_Pseudo>(OpName#"_dpp"), SIEncodingFamily.VI>,
       VOP2_DPPe<op, !cast<VOP2_DPP_Pseudo>(OpName#"_dpp")> {
@@ -2077,14 +2077,14 @@ multiclass VOP2be_Real_e32e64_gfx9 <bits<6> op, string OpName, string AsmName> {
       let AsmString = AsmName # ps.AsmOperands;
       let DecoderNamespace = "GFX9";
     }
-  foreach _ = BoolToList<!cast<VOP2_Pseudo>(OpName#"_e32").Pfl.HasExtSDWA9>.ret in
+  if !cast<VOP2_Pseudo>(OpName#"_e32").Pfl.HasExtSDWA9 then
     def _sdwa_gfx9 :
       VOP_SDWA9_Real <!cast<VOP2_SDWA_Pseudo>(OpName#"_sdwa")>,
       VOP2_SDWA9Ae <op{5-0}, !cast<VOP2_SDWA_Pseudo>(OpName#"_sdwa").Pfl> {
         VOP2_SDWA_Pseudo ps = !cast<VOP2_SDWA_Pseudo>(OpName#"_sdwa");
         let AsmString = AsmName # ps.AsmOperands;
       }
-  foreach _ = BoolToList<!cast<VOP2_Pseudo>(OpName#"_e32").Pfl.HasExtDPP>.ret in
+  if !cast<VOP2_Pseudo>(OpName#"_e32").Pfl.HasExtDPP then
     def _dpp_gfx9 :
       VOP_DPP_Real<!cast<VOP2_DPP_Pseudo>(OpName#"_dpp"), SIEncodingFamily.GFX9>,
       VOP2_DPPe<op, !cast<VOP2_DPP_Pseudo>(OpName#"_dpp")> {
@@ -2105,12 +2105,12 @@ multiclass VOP2_Real_e32e64_gfx9 <bits<6> op> {
     VOP3e_vi <{0, 1, 0, 0, op{5-0}}, !cast<VOP3_Pseudo>(NAME#"_e64").Pfl> {
       let DecoderNamespace = "GFX9";
     }
-  foreach _ = BoolToList<!cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA9>.ret in
+  if !cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA9 then
     def _sdwa_gfx9 :
       VOP_SDWA9_Real <!cast<VOP2_SDWA_Pseudo>(NAME#"_sdwa")>,
       VOP2_SDWA9Ae <op{5-0}, !cast<VOP2_SDWA_Pseudo>(NAME#"_sdwa").Pfl> {
       }
-  foreach _ = BoolToList<!cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExtDPP>.ret in
+  if !cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExtDPP then
     def _dpp_gfx9 :
       VOP_DPP_Real<!cast<VOP2_DPP_Pseudo>(NAME#"_dpp"), SIEncodingFamily.GFX9>,
       VOP2_DPPe<op, !cast<VOP2_DPP_Pseudo>(NAME#"_dpp")> {
@@ -2123,7 +2123,7 @@ multiclass VOP2_Real_e32e64_gfx9 <bits<6> op> {
 multiclass VOP2_Real_e32e64_vi <bits<6> op> :
   Base_VOP2_Real_e32e64_vi<op>, VOP2_SDWA_Real<op>, VOP2_SDWA9_Real<op> {
 
-  foreach _ = BoolToList<!cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExtDPP>.ret in
+  if !cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExtDPP then
     def _dpp_vi :
       VOP_DPP_Real<!cast<VOP2_DPP_Pseudo>(NAME#"_dpp"), SIEncodingFamily.VI>,
       VOP2_DPPe<op, !cast<VOP2_DPP_Pseudo>(NAME#"_dpp")>;
@@ -2270,7 +2270,7 @@ let AssemblerPredicate = isGFX90APlus, DecoderNamespace = "GFX90A" in {
   multiclass VOP2_Real_e32e64_gfx90a <bits<6> op> :
     Base_VOP2_Real_e32e64_gfx90a<op> {
 
-    foreach _ = BoolToList<!cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExtDPP>.ret in
+    if !cast<VOP2_Pseudo>(NAME#"_e32").Pfl.HasExtDPP then
       def _dpp_gfx90a :
         VOP_DPP_Real<!cast<VOP2_DPP_Pseudo>(NAME#"_dpp"), SIEncodingFamily.GFX90A>,
         VOP2_DPPe<op, !cast<VOP2_DPP_Pseudo>(NAME#"_dpp")> {
diff --git a/llvm/lib/Target/AMDGPU/VOP3Instructions.td b/llvm/lib/Target/AMDGPU/VOP3Instructions.td
index 70e50111790bb..65db790fedf1c 100644
--- a/llvm/lib/Target/AMDGPU/VOP3Instructions.td
+++ b/llvm/lib/Target/AMDGPU/VOP3Instructions.td
@@ -263,7 +263,7 @@ let SchedRW = [Write64Bit] in {
 
 def : GCNPat<
   (i32 (DivergentUnaryFrag<sext> i16:$src)),
-  (i32 (V_BFE_I32_e64 $src, (S_MOV_B32 (i32 0)), (S_MOV_B32 (i32 0x10))))
+  (i32 (V_BFE_I32_e64 i16:$src, (i32 0), (i32 0x10)))
 >;
 
 let isReMaterializable = 1 in {
diff --git a/llvm/lib/Target/AMDGPU/VOP3PInstructions.td b/llvm/lib/Target/AMDGPU/VOP3PInstructions.td
index 8f8c4489454db..422d02e237621 100644
--- a/llvm/lib/Target/AMDGPU/VOP3PInstructions.td
+++ b/llvm/lib/Target/AMDGPU/VOP3PInstructions.td
@@ -607,7 +607,7 @@ multiclass MAIInst<string OpName, string P, SDPatternOperator node,
                         MFMATable<0, NAME # "_vgprcd_e64">;
     }
 
-    foreach _ = BoolToList<NoDstOverlap>.ret in {
+    if NoDstOverlap then {
       let Constraints = !if(NoDstOverlap, "$vdst = $src2", ""),
           isConvertibleToThreeAddress = NoDstOverlap,
           Mnemonic = OpName in {
@@ -1015,7 +1015,7 @@ multiclass VOP3P_Real_MFMA_gfx940_aliases<string NameFrom, string NameTo, string
                                           VOPProfile Pfl_ACD = PS_ACD.Pfl,
                                           VOPProfile Pfl_VCD = PS_VCD.Pfl> {
   let Predicates = [isGFX940Plus] in {
-    foreach _ = BoolToList<!ne(NameFrom, NameTo)>.ret in {
+    if !ne(NameFrom, NameTo) then {
       def : InstAlias <NameTo # " " # PS_ACD.AsmOperands,
                        (!cast<VOP3P_Real>(Op # "_gfx940_acd") Pfl_ACD.DstRC:$vdst,
                            Pfl_ACD.Src0RC64:$src0, Pfl_ACD.Src1RC64:$src1, Pfl_ACD.Src2RC64:$src2,
@@ -1043,7 +1043,7 @@ multiclass VOP3P_Real_MFMA_gfx940<bits<7> op, string Name = !cast<VOP3_Pseudo>(N
 
   defm : VOP3P_Real_MFMA_gfx940_aliases<Name, PS_ACD.Mnemonic, NAME>;
 
-  foreach _ = BoolToList<!ne(!subst("_1k", "", PS_ACD.Mnemonic), PS_ACD.Mnemonic)>.ret in
+  if !ne(!subst("_1k", "", PS_ACD.Mnemonic), PS_ACD.Mnemonic) then
   defm : VOP3P_Real_MFMA_gfx940_aliases<Name, !subst("_1k", "", PS_ACD.Mnemonic), NAME>;
 }
 
diff --git a/llvm/lib/Target/AMDGPU/VOPCInstructions.td b/llvm/lib/Target/AMDGPU/VOPCInstructions.td
index 6f054f3deaea6..9e11775fa9f08 100644
--- a/llvm/lib/Target/AMDGPU/VOPCInstructions.td
+++ b/llvm/lib/Target/AMDGPU/VOPCInstructions.td
@@ -299,7 +299,7 @@ multiclass VOPC_Pseudos <string opName,
     let isCommutable = 1;
   }
 
-  foreach _ = BoolToList<P.HasExtSDWA>.ret in
+  if P.HasExtSDWA then
   def _sdwa : VOPC_SDWA_Pseudo <opName, P> {
     let Defs = !if(DefExec, [EXEC], []);
     let SchedRW = P.Schedule;
@@ -360,7 +360,7 @@ multiclass VOPCX_Pseudos <string opName,
     let IsVCMPX = 1;
   }
 
-  foreach _ = BoolToList<P_NoSDst.HasExtSDWA>.ret in
+  if P_NoSDst.HasExtSDWA then
   def _nosdst_sdwa : VOPC_SDWA_Pseudo <opName#"_nosdst", P_NoSDst> {
     let Defs = [EXEC];
     let SchedRW = P_NoSDst.Schedule;
@@ -854,7 +854,7 @@ multiclass VOPC_Class_Pseudos <string opName, VOPC_Profile p, bit DefExec,
     let SchedRW = p.Schedule;
   }
 
-  foreach _ = BoolToList<p.HasExtSDWA>.ret in
+  if p.HasExtSDWA then
   def _sdwa : VOPC_SDWA_Pseudo <opName, p> {
     let Defs = !if(DefExec, !if(DefVcc, [VCC, EXEC], [EXEC]),
                             !if(DefVcc, [VCC], []));
@@ -902,7 +902,7 @@ multiclass VOPCX_Class_Pseudos <string opName,
     let SubtargetPredicate = HasNoSdstCMPX;
   }
 
-  foreach _ = BoolToList<P_NoSDst.HasExtSDWA>.ret in
+  if P_NoSDst.HasExtSDWA then
   def _nosdst_sdwa : VOPC_SDWA_Pseudo <opName#"_nosdst", P_NoSDst> {
     let Defs = [EXEC];
     let SchedRW = P_NoSDst.Schedule;
@@ -1320,7 +1320,7 @@ let AssemblerPredicate = isGFX11Only in {
 
     defm : VOPCInstAliases<NAME, "gfx11">;
 
-    foreach _ = BoolToList<ps32.Pfl.HasExtDPP>.ret in {
+    if ps32.Pfl.HasExtDPP then {
       defvar psDPP = !cast<VOP_DPP_Pseudo>(NAME #"_e32" #"_dpp");
       defvar AsmDPP = ps32.Pfl.AsmDPP16;
       let DecoderNamespace = "DPPGFX11" in {
@@ -1352,7 +1352,7 @@ let AssemblerPredicate = isGFX11Only in {
         }
       }
     }
-    foreach _ = BoolToList<ps64.Pfl.HasExtVOP3DPP>.ret in {
+    if ps64.Pfl.HasExtVOP3DPP then {
       defvar psDPP = !cast<VOP_DPP_Pseudo>(NAME #"_e64" #"_dpp");
       defvar AsmDPP = ps64.Pfl.AsmVOP3DPP16;
       let DecoderNamespace = "DPPGFX11" in {
@@ -1419,7 +1419,7 @@ let AssemblerPredicate = isGFX11Only in {
 
     defm : VOPCInstAliases<OpName, "gfx11", NAME, asm_name>;
 
-    foreach _ = BoolToList<ps32.Pfl.HasExtDPP>.ret in {
+    if ps32.Pfl.HasExtDPP then {
       defvar psDPP = !cast<VOP_DPP_Pseudo>(OpName #"_e32" #"_dpp");
       defvar AsmDPP = ps32.Pfl.AsmDPP16;
       let DecoderNamespace = "DPPGFX11" in {
@@ -1456,7 +1456,7 @@ let AssemblerPredicate = isGFX11Only in {
       }
     }
 
-    foreach _ = BoolToList<ps64.Pfl.HasExtVOP3DPP>.ret in {
+    if ps64.Pfl.HasExtVOP3DPP then {
       defvar psDPP = !cast<VOP_DPP_Pseudo>(OpName #"_e64" #"_dpp");
       defvar AsmDPP = ps64.Pfl.AsmVOP3DPP16;
       let DecoderNamespace = "DPPGFX11" in {
@@ -1518,7 +1518,7 @@ let AssemblerPredicate = isGFX11Only in {
 
     defm : VOPCXInstAliases<NAME, "gfx11">;
 
-    foreach _ = BoolToList<ps32.Pfl.HasExtDPP>.ret in {
+    if ps32.Pfl.HasExtDPP then {
       defvar psDPP = !cast<VOP_DPP_Pseudo>(NAME #"_nosdst_e32" #"_dpp");
       defvar AsmDPP = ps32.Pfl.AsmDPP16;
       let DecoderNamespace = "DPPGFX11" in {
@@ -1535,7 +1535,7 @@ let AssemblerPredicate = isGFX11Only in {
       }
     }
 
-    foreach _ = BoolToList<ps64.Pfl.HasExtVOP3DPP>.ret in {
+    if ps64.Pfl.HasExtVOP3DPP then {
       defvar psDPP = !cast<VOP_DPP_Pseudo>(NAME #"_nosdst_e64" #"_dpp");
       defvar AsmDPP = ps64.Pfl.AsmVOP3DPP16;
       let DecoderNamespace = "DPPGFX11" in {
@@ -1584,7 +1584,7 @@ let AssemblerPredicate = isGFX11Only in {
 
     defm : VOPCXInstAliases<OpName, "gfx11", NAME, asm_name>;
 
-    foreach _ = BoolToList<ps32.Pfl.HasExtDPP>.ret in {
+    if ps32.Pfl.HasExtDPP then {
       defvar psDPP = !cast<VOP_DPP_Pseudo>(OpName#"_nosdst_e32"#"_dpp");
       let DecoderNamespace = "DPPGFX11" in {
         def _e32_dpp_gfx11 : VOPC_DPP16_SIMC<op{7-0}, psDPP,
@@ -1594,7 +1594,7 @@ let AssemblerPredicate = isGFX11Only in {
         def _e32_dpp8_gfx11 : VOPC_DPP8<op{7-0}, ps32, asm_name>;
       }
     }
-    foreach _ = BoolToList<ps64.Pfl.HasExtVOP3DPP>.ret in {
+    if ps64.Pfl.HasExtVOP3DPP then {
       defvar psDPP = !cast<VOP_DPP_Pseudo>(OpName#"_nosdst_e64"#"_dpp");
       defvar AsmDPP = ps64.Pfl.AsmVOP3DPP16;
       let DecoderNamespace = "DPPGFX11" in {
@@ -1821,7 +1821,7 @@ let AssemblerPredicate = isGFX10Only in {
       }
     } // End DecoderNamespace = "GFX10"
 
-    foreach _ = BoolToList<!cast<VOPC_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA9>.ret in
+    if !cast<VOPC_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA9 then
     def _sdwa_gfx10 :
       VOP_SDWA10_Real<!cast<VOPC_SDWA_Pseudo>(NAME#"_sdwa")>,
       VOPC_SDWA9e<op{7-0}, !cast<VOPC_SDWA_Pseudo>(NAME#"_sdwa").Pfl>;
@@ -1847,7 +1847,7 @@ let AssemblerPredicate = isGFX10Only in {
         }
     } // End DecoderNamespace = "GFX10"
 
-    foreach _ = BoolToList<!cast<VOPC_Pseudo>(NAME#"_nosdst_e32").Pfl.HasExtSDWA9>.ret in
+    if !cast<VOPC_Pseudo>(NAME#"_nosdst_e32").Pfl.HasExtSDWA9 then
     def _sdwa_gfx10 :
       VOP_SDWA10_Real<!cast<VOPC_SDWA_Pseudo>(NAME#"_nosdst_sdwa")>,
       VOPC_SDWA9e<op{7-0}, !cast<VOPC_SDWA_Pseudo>(NAME#"_nosdst_sdwa").Pfl> {
@@ -2174,12 +2174,12 @@ multiclass VOPC_Real_vi <bits<10> op> {
     }
   }
 
-  foreach _ = BoolToList<!cast<VOPC_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA>.ret in
+  if !cast<VOPC_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA then
   def _sdwa_vi :
     VOP_SDWA_Real <!cast<VOPC_SDWA_Pseudo>(NAME#"_sdwa")>,
     VOPC_SDWAe <op{7-0}, !cast<VOPC_SDWA_Pseudo>(NAME#"_sdwa").Pfl>;
 
-  foreach _ = BoolToList<!cast<VOPC_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA9>.ret in
+  if !cast<VOPC_Pseudo>(NAME#"_e32").Pfl.HasExtSDWA9 then
   def _sdwa_gfx9 :
     VOP_SDWA9_Real <!cast<VOPC_SDWA_Pseudo>(NAME#"_sdwa")>,
     VOPC_SDWA9e <op{7-0}, !cast<VOPC_SDWA_Pseudo>(NAME#"_sdwa").Pfl>;
diff --git a/llvm/lib/Target/AMDGPU/VOPInstructions.td b/llvm/lib/Target/AMDGPU/VOPInstructions.td
index d5c662ac05745..c5e1d4f62d076 100644
--- a/llvm/lib/Target/AMDGPU/VOPInstructions.td
+++ b/llvm/lib/Target/AMDGPU/VOPInstructions.td
@@ -1268,7 +1268,7 @@ class VOP3InstBase<string OpName, VOPProfile P, SDPatternOperator node = null_fr
 multiclass VOP3Inst<string OpName, VOPProfile P, SDPatternOperator node = null_frag> {
   def _e64 : VOP3InstBase<OpName, P, node>;
   let SubtargetPredicate = isGFX11Plus in {
-    foreach _ = BoolToList<P.HasExtVOP3DPP>.ret in
+    if P.HasExtVOP3DPP then
       def _e64_dpp : VOP3_DPP_Pseudo <OpName, P>;
   } // end SubtargetPredicate = isGFX11Plus
 }
@@ -1329,11 +1329,11 @@ let AssemblerPredicate = isGFX11Only,
                                   bit isSingle = 0> {
     defvar ps = !cast<VOP_Pseudo>(opName#"_e64");
     let IsSingle = !or(isSingle, ps.Pfl.IsSingle) in {
-    foreach _ = BoolToList<ps.Pfl.HasOpSel>.ret in
+    if ps.Pfl.HasOpSel then
       def _e64_gfx11 :
         VOP3_Real<ps, SIEncodingFamily.GFX11>,
         VOP3OpSel_gfx11<op, ps.Pfl>;
-    foreach _ = BoolToList<!not(ps.Pfl.HasOpSel)>.ret in
+    if !not(ps.Pfl.HasOpSel) then
       def _e64_gfx11 :
         VOP3_Real<ps, SIEncodingFamily.GFX11>,
         VOP3e_gfx11<op, ps.Pfl>;
@@ -1353,11 +1353,11 @@ let AssemblerPredicate = isGFX11Only,
     defvar ps = !cast<VOP_Pseudo>(opName#"_e64");
     let AsmString = asmName # ps.AsmOperands,
         IsSingle = !or(isSingle, ps.Pfl.IsSingle) in {
-    foreach _ = BoolToList<ps.Pfl.HasOpSel>.ret in
+    if ps.Pfl.HasOpSel then
       def _e64_gfx11 :
         VOP3_Real<ps, SIEncodingFamily.GFX11>,
         VOP3OpSel_gfx11<op, ps.Pfl>;
-    foreach _ = BoolToList<!not(ps.Pfl.HasOpSel)>.ret in
+    if !not(ps.Pfl.HasOpSel) then
       def _e64_gfx11 :
         VOP3_Real<ps, SIEncodingFamily.GFX11>,
         VOP3e_gfx11<op, ps.Pfl>;
diff --git a/llvm/lib/Target/ARM/ARMISelLowering.cpp b/llvm/lib/Target/ARM/ARMISelLowering.cpp
index 552a24e9b649b..126bbc61a7d30 100644
--- a/llvm/lib/Target/ARM/ARMISelLowering.cpp
+++ b/llvm/lib/Target/ARM/ARMISelLowering.cpp
@@ -4422,10 +4422,8 @@ void ARMTargetLowering::VarArgStyleRegisters(CCState &CCInfo, SelectionDAG &DAG,
 bool ARMTargetLowering::splitValueIntoRegisterParts(
     SelectionDAG &DAG, const SDLoc &DL, SDValue Val, SDValue *Parts,
     unsigned NumParts, MVT PartVT, std::optional<CallingConv::ID> CC) const {
-  bool IsABIRegCopy = CC.has_value();
   EVT ValueVT = Val.getValueType();
-  if (IsABIRegCopy && (ValueVT == MVT::f16 || ValueVT == MVT::bf16) &&
-      PartVT == MVT::f32) {
+  if ((ValueVT == MVT::f16 || ValueVT == MVT::bf16) && PartVT == MVT::f32) {
     unsigned ValueBits = ValueVT.getSizeInBits();
     unsigned PartBits = PartVT.getSizeInBits();
     Val = DAG.getNode(ISD::BITCAST, DL, MVT::getIntegerVT(ValueBits), Val);
@@ -4440,9 +4438,7 @@ bool ARMTargetLowering::splitValueIntoRegisterParts(
 SDValue ARMTargetLowering::joinRegisterPartsIntoValue(
     SelectionDAG &DAG, const SDLoc &DL, const SDValue *Parts, unsigned NumParts,
     MVT PartVT, EVT ValueVT, std::optional<CallingConv::ID> CC) const {
-  bool IsABIRegCopy = CC.has_value();
-  if (IsABIRegCopy && (ValueVT == MVT::f16 || ValueVT == MVT::bf16) &&
-      PartVT == MVT::f32) {
+  if ((ValueVT == MVT::f16 || ValueVT == MVT::bf16) && PartVT == MVT::f32) {
     unsigned ValueBits = ValueVT.getSizeInBits();
     unsigned PartBits = PartVT.getSizeInBits();
     SDValue Val = Parts[0];
@@ -15004,16 +15000,31 @@ static SDValue PerformVMOVhrCombine(SDNode *N,
   // FullFP16: half values are passed in S-registers, and we don't
   // need any of the bitcast and moves:
   //
-  //     t2: f32,ch = CopyFromReg t0, Register:f32 %0
+  //     t2: f32,ch1,gl1? = CopyFromReg ch, Register:f32 %0, gl?
   //   t5: i32 = bitcast t2
   // t18: f16 = ARMISD::VMOVhr t5
+  // =>
+  // tN: f16,ch2,gl2? = CopyFromReg ch, Register::f32 %0, gl?
   if (Op0->getOpcode() == ISD::BITCAST) {
     SDValue Copy = Op0->getOperand(0);
     if (Copy.getValueType() == MVT::f32 &&
         Copy->getOpcode() == ISD::CopyFromReg) {
-      SDValue Ops[] = {Copy->getOperand(0), Copy->getOperand(1)};
+      bool HasGlue = Copy->getNumOperands() == 3;
+      SDValue Ops[] = {Copy->getOperand(0), Copy->getOperand(1),
+                       HasGlue ? Copy->getOperand(2) : SDValue()};
+      EVT OutTys[] = {N->getValueType(0), MVT::Other, MVT::Glue};
       SDValue NewCopy =
-          DCI.DAG.getNode(ISD::CopyFromReg, SDLoc(N), N->getValueType(0), Ops);
+          DCI.DAG.getNode(ISD::CopyFromReg, SDLoc(N),
+                          DCI.DAG.getVTList(ArrayRef(OutTys, HasGlue ? 3 : 2)),
+                          ArrayRef(Ops, HasGlue ? 3 : 2));
+
+      // Update Users, Chains, and Potential Glue.
+      DCI.DAG.ReplaceAllUsesOfValueWith(SDValue(N, 0), NewCopy.getValue(0));
+      DCI.DAG.ReplaceAllUsesOfValueWith(Copy.getValue(1), NewCopy.getValue(1));
+      if (HasGlue)
+        DCI.DAG.ReplaceAllUsesOfValueWith(Copy.getValue(2),
+                                          NewCopy.getValue(2));
+
       return NewCopy;
     }
   }
@@ -20241,8 +20252,12 @@ RCPair ARMTargetLowering::getRegForInlineAsmConstraint(
     case 'w':
       if (VT == MVT::Other)
         break;
-      if (VT == MVT::f16 || VT == MVT::bf16)
-        return RCPair(0U, &ARM::HPRRegClass);
+      if (VT == MVT::f16)
+        return RCPair(0U, Subtarget->hasFullFP16() ? &ARM::HPRRegClass
+                                                   : &ARM::SPRRegClass);
+      if (VT == MVT::bf16)
+        return RCPair(0U, Subtarget->hasBF16() ? &ARM::HPRRegClass
+                                               : &ARM::SPRRegClass);
       if (VT == MVT::f32)
         return RCPair(0U, &ARM::SPRRegClass);
       if (VT.getSizeInBits() == 64)
@@ -20263,8 +20278,12 @@ RCPair ARMTargetLowering::getRegForInlineAsmConstraint(
     case 't':
       if (VT == MVT::Other)
         break;
-      if (VT == MVT::f16 || VT == MVT::bf16)
-        return RCPair(0U, &ARM::HPRRegClass);
+      if (VT == MVT::f16)
+        return RCPair(0U, Subtarget->hasFullFP16() ? &ARM::HPRRegClass
+                                                   : &ARM::SPRRegClass);
+      if (VT == MVT::bf16)
+        return RCPair(0U, Subtarget->hasBF16() ? &ARM::HPRRegClass
+                                               : &ARM::SPRRegClass);
       if (VT == MVT::f32 || VT == MVT::i32)
         return RCPair(0U, &ARM::SPRRegClass);
       if (VT.getSizeInBits() == 64)
diff --git a/llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp b/llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp
index c8b156a6f2c0f..1e0a7d11d8ac2 100644
--- a/llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp
+++ b/llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp
@@ -1249,7 +1249,7 @@ bool LowOverheadLoop::ValidateMVEInst(MachineInstr *MI) {
   const MCInstrDesc &MCID = MI->getDesc();
   bool IsUse = false;
   unsigned LastOpIdx = MI->getNumOperands() - 1;
-  for (auto &Op : enumerate(reverse(MCID.operands()))) {
+  for (const auto &Op : enumerate(reverse(MCID.operands()))) {
     const MachineOperand &MO = MI->getOperand(LastOpIdx - Op.index());
     if (!MO.isReg() || !MO.isUse() || MO.getReg() != ARM::VPR)
       continue;
diff --git a/llvm/lib/Target/ARM/AsmParser/ARMAsmParser.cpp b/llvm/lib/Target/ARM/AsmParser/ARMAsmParser.cpp
index 37852ea2a72e3..7ce9e04657c7d 100644
--- a/llvm/lib/Target/ARM/AsmParser/ARMAsmParser.cpp
+++ b/llvm/lib/Target/ARM/AsmParser/ARMAsmParser.cpp
@@ -11766,7 +11766,7 @@ bool ARMAsmParser::parseDirectiveFPU(SMLoc L) {
   SMLoc FPUNameLoc = getTok().getLoc();
   StringRef FPU = getParser().parseStringToEndOfStatement().trim();
 
-  unsigned ID = ARM::parseFPU(FPU);
+  ARM::FPUKind ID = ARM::parseFPU(FPU);
   std::vector<StringRef> Features;
   if (!ARM::getFPUFeatures(ID, Features))
     return Error(FPUNameLoc, "Unknown FPU name");
diff --git a/llvm/lib/Target/ARM/MCTargetDesc/ARMAsmBackend.cpp b/llvm/lib/Target/ARM/MCTargetDesc/ARMAsmBackend.cpp
index bfbbbb419511b..0bd715688b018 100644
--- a/llvm/lib/Target/ARM/MCTargetDesc/ARMAsmBackend.cpp
+++ b/llvm/lib/Target/ARM/MCTargetDesc/ARMAsmBackend.cpp
@@ -1157,7 +1157,7 @@ uint32_t ARMAsmBackendDarwin::generateCompactUnwindEncoding(
       // Directive not convertable to compact unwind, bail out.
       DEBUG_WITH_TYPE("compact-unwind",
                       llvm::dbgs()
-                          << "CFI directive not compatiable with comact "
+                          << "CFI directive not compatible with compact "
                              "unwind encoding, opcode=" << Inst.getOperation()
                           << "\n");
       return CU::UNWIND_ARM_MODE_DWARF;
diff --git a/llvm/lib/Target/ARM/MCTargetDesc/ARMELFStreamer.cpp b/llvm/lib/Target/ARM/MCTargetDesc/ARMELFStreamer.cpp
index 822bf7ebe4e15..25bbc4ee7eb59 100644
--- a/llvm/lib/Target/ARM/MCTargetDesc/ARMELFStreamer.cpp
+++ b/llvm/lib/Target/ARM/MCTargetDesc/ARMELFStreamer.cpp
@@ -93,7 +93,7 @@ class ARMTargetAsmStreamer : public ARMTargetStreamer {
   void emitArch(ARM::ArchKind Arch) override;
   void emitArchExtension(uint64_t ArchExt) override;
   void emitObjectArch(ARM::ArchKind Arch) override;
-  void emitFPU(unsigned FPU) override;
+  void emitFPU(ARM::FPUKind FPU) override;
   void emitInst(uint32_t Inst, char Suffix = '\0') override;
   void finishAttributeSection() override;
 
@@ -248,7 +248,7 @@ void ARMTargetAsmStreamer::emitObjectArch(ARM::ArchKind Arch) {
   OS << "\t.object_arch\t" << ARM::getArchName(Arch) << '\n';
 }
 
-void ARMTargetAsmStreamer::emitFPU(unsigned FPU) {
+void ARMTargetAsmStreamer::emitFPU(ARM::FPUKind FPU) {
   OS << "\t.fpu\t" << ARM::getFPUName(FPU) << "\n";
 }
 
@@ -382,7 +382,7 @@ void ARMTargetAsmStreamer::emitARMWinCFICustom(unsigned Opcode) {
 class ARMTargetELFStreamer : public ARMTargetStreamer {
 private:
   StringRef CurrentVendor;
-  unsigned FPU = ARM::FK_INVALID;
+  ARM::FPUKind FPU = ARM::FK_INVALID;
   ARM::ArchKind Arch = ARM::ArchKind::INVALID;
   ARM::ArchKind EmittedArch = ARM::ArchKind::INVALID;
 
@@ -414,7 +414,7 @@ class ARMTargetELFStreamer : public ARMTargetStreamer {
                             StringRef StringValue) override;
   void emitArch(ARM::ArchKind Arch) override;
   void emitObjectArch(ARM::ArchKind Arch) override;
-  void emitFPU(unsigned FPU) override;
+  void emitFPU(ARM::FPUKind FPU) override;
   void emitInst(uint32_t Inst, char Suffix = '\0') override;
   void finishAttributeSection() override;
   void emitLabel(MCSymbol *Symbol) override;
@@ -928,9 +928,7 @@ void ARMTargetELFStreamer::emitArchDefaultAttributes() {
   }
 }
 
-void ARMTargetELFStreamer::emitFPU(unsigned Value) {
-  FPU = Value;
-}
+void ARMTargetELFStreamer::emitFPU(ARM::FPUKind Value) { FPU = Value; }
 
 void ARMTargetELFStreamer::emitFPUDefaultAttributes() {
   ARMELFStreamer &S = getStreamer();
diff --git a/llvm/lib/Target/ARM/MCTargetDesc/ARMTargetStreamer.cpp b/llvm/lib/Target/ARM/MCTargetDesc/ARMTargetStreamer.cpp
index a0df44441fcbd..b65d1b24e63d3 100644
--- a/llvm/lib/Target/ARM/MCTargetDesc/ARMTargetStreamer.cpp
+++ b/llvm/lib/Target/ARM/MCTargetDesc/ARMTargetStreamer.cpp
@@ -111,7 +111,7 @@ void ARMTargetStreamer::emitIntTextAttribute(unsigned Attribute,
 void ARMTargetStreamer::emitArch(ARM::ArchKind Arch) {}
 void ARMTargetStreamer::emitArchExtension(uint64_t ArchExt) {}
 void ARMTargetStreamer::emitObjectArch(ARM::ArchKind Arch) {}
-void ARMTargetStreamer::emitFPU(unsigned FPU) {}
+void ARMTargetStreamer::emitFPU(ARM::FPUKind FPU) {}
 void ARMTargetStreamer::finishAttributeSection() {}
 void ARMTargetStreamer::annotateTLSDescriptorSequence(
     const MCSymbolRefExpr *SRE) {}
diff --git a/llvm/lib/Target/BPF/BTFDebug.cpp b/llvm/lib/Target/BPF/BTFDebug.cpp
index 9a873413db870..e1a3e08d95b63 100644
--- a/llvm/lib/Target/BPF/BTFDebug.cpp
+++ b/llvm/lib/Target/BPF/BTFDebug.cpp
@@ -782,6 +782,17 @@ void BTFDebug::visitCompositeType(const DICompositeType *CTy,
     visitEnumType(CTy, TypeId);
 }
 
+bool BTFDebug::IsForwardDeclCandidate(const DIType *Base) {
+  if (const auto *CTy = dyn_cast<DICompositeType>(Base)) {
+    auto CTag = CTy->getTag();
+    if ((CTag == dwarf::DW_TAG_structure_type ||
+         CTag == dwarf::DW_TAG_union_type) &&
+        !CTy->getName().empty() && !CTy->isForwardDecl())
+      return true;
+  }
+  return false;
+}
+
 /// Handle pointer, typedef, const, volatile, restrict and member types.
 void BTFDebug::visitDerivedType(const DIDerivedType *DTy, uint32_t &TypeId,
                                 bool CheckPointer, bool SeenPointer) {
@@ -796,20 +807,15 @@ void BTFDebug::visitDerivedType(const DIDerivedType *DTy, uint32_t &TypeId,
   if (CheckPointer && SeenPointer) {
     const DIType *Base = DTy->getBaseType();
     if (Base) {
-      if (const auto *CTy = dyn_cast<DICompositeType>(Base)) {
-        auto CTag = CTy->getTag();
-        if ((CTag == dwarf::DW_TAG_structure_type ||
-             CTag == dwarf::DW_TAG_union_type) &&
-            !CTy->getName().empty() && !CTy->isForwardDecl()) {
-          /// Find a candidate, generate a fixup. Later on the struct/union
-          /// pointee type will be replaced with either a real type or
-          /// a forward declaration.
-          auto TypeEntry = std::make_unique<BTFTypeDerived>(DTy, Tag, true);
-          auto &Fixup = FixupDerivedTypes[CTy];
-          Fixup.push_back(std::make_pair(DTy, TypeEntry.get()));
-          TypeId = addType(std::move(TypeEntry), DTy);
-          return;
-        }
+      if (IsForwardDeclCandidate(Base)) {
+        /// Find a candidate, generate a fixup. Later on the struct/union
+        /// pointee type will be replaced with either a real type or
+        /// a forward declaration.
+        auto TypeEntry = std::make_unique<BTFTypeDerived>(DTy, Tag, true);
+        auto &Fixup = FixupDerivedTypes[cast<DICompositeType>(Base)];
+        Fixup.push_back(std::make_pair(DTy, TypeEntry.get()));
+        TypeId = addType(std::move(TypeEntry), DTy);
+        return;
       }
     }
   }
@@ -844,6 +850,13 @@ void BTFDebug::visitDerivedType(const DIDerivedType *DTy, uint32_t &TypeId,
     visitTypeEntry(DTy->getBaseType(), TempTypeId, CheckPointer, SeenPointer);
 }
 
+/// Visit a type entry. CheckPointer is true if the type has
+/// one of its predecessors as one struct/union member. SeenPointer
+/// is true if CheckPointer is true and one of its predecessors
+/// is a pointer. The goal of CheckPointer and SeenPointer is to
+/// do pruning for struct/union types so some of these types
+/// will not be emitted in BTF and rather forward declarations
+/// will be generated.
 void BTFDebug::visitTypeEntry(const DIType *Ty, uint32_t &TypeId,
                               bool CheckPointer, bool SeenPointer) {
   if (!Ty || DIToIdMap.find(Ty) != DIToIdMap.end()) {
@@ -888,6 +901,11 @@ void BTFDebug::visitTypeEntry(const DIType *Ty, uint32_t &TypeId,
           if (DIToIdMap.find(BaseTy) != DIToIdMap.end()) {
             DTy = dyn_cast<DIDerivedType>(BaseTy);
           } else {
+            if (CheckPointer && DTy->getTag() == dwarf::DW_TAG_pointer_type) {
+              SeenPointer = true;
+              if (IsForwardDeclCandidate(BaseTy))
+                break;
+            }
             uint32_t TmpTypeId;
             visitTypeEntry(BaseTy, TmpTypeId, CheckPointer, SeenPointer);
             break;
@@ -964,7 +982,7 @@ std::string BTFDebug::populateFileContent(const DISubprogram *SP) {
     FileName = std::string(File->getFilename());
 
   // No need to populate the contends if it has been populated!
-  if (FileContent.find(FileName) != FileContent.end())
+  if (FileContent.contains(FileName))
     return FileName;
 
   std::vector<std::string> Content;
diff --git a/llvm/lib/Target/BPF/BTFDebug.h b/llvm/lib/Target/BPF/BTFDebug.h
index aa982babd4588..f0b42232f4d55 100644
--- a/llvm/lib/Target/BPF/BTFDebug.h
+++ b/llvm/lib/Target/BPF/BTFDebug.h
@@ -338,6 +338,9 @@ class BTFDebug : public DebugHandlerBase {
   void visitMapDefType(const DIType *Ty, uint32_t &TypeId);
   /// @}
 
+  /// Check whether the type is a forward declaration candidate or not.
+  bool IsForwardDeclCandidate(const DIType *Base);
+
   /// Get the file content for the subprogram. Certain lines of the file
   /// later may be put into string table and referenced by line info.
   std::string populateFileContent(const DISubprogram *SP);
diff --git a/llvm/lib/Target/Hexagon/Hexagon.td b/llvm/lib/Target/Hexagon/Hexagon.td
index 5fa9f1ad7211b..d31597158cc1c 100644
--- a/llvm/lib/Target/Hexagon/Hexagon.td
+++ b/llvm/lib/Target/Hexagon/Hexagon.td
@@ -160,8 +160,8 @@ def UseSmallData       : Predicate<"HST->useSmallData()">;
 def UseCabac           : Predicate<"HST->useCabac()">,
                          AssemblerPredicate<(any_of FeatureCabac)>;
 
-def Hvx64:  HwMode<"+hvx-length64b">;
-def Hvx128: HwMode<"+hvx-length128b">;
+def Hvx64:  HwMode<"+hvx-length64b", [UseHVX64B]>;
+def Hvx128: HwMode<"+hvx-length128b", [UseHVX128B]>;
 
 //===----------------------------------------------------------------------===//
 // Classes used for relation maps.
diff --git a/llvm/lib/Target/Hexagon/HexagonLoopIdiomRecognition.cpp b/llvm/lib/Target/Hexagon/HexagonLoopIdiomRecognition.cpp
index fe723a4ba3fbf..a7887712c29a4 100644
--- a/llvm/lib/Target/Hexagon/HexagonLoopIdiomRecognition.cpp
+++ b/llvm/lib/Target/Hexagon/HexagonLoopIdiomRecognition.cpp
@@ -336,7 +336,7 @@ void Simplifier::Context::initialize(Instruction *Exp) {
 
   while (!Q.empty()) {
     Value *V = Q.pop_front_val();
-    if (M.find(V) != M.end())
+    if (M.contains(V))
       continue;
     if (Instruction *U = dyn_cast<Instruction>(V)) {
       if (isa<PHINode>(U) || U->getParent() != Block)
diff --git a/llvm/lib/Target/Hexagon/HexagonVectorCombine.cpp b/llvm/lib/Target/Hexagon/HexagonVectorCombine.cpp
index 3ff6e9b469398..2629b54224f1d 100644
--- a/llvm/lib/Target/Hexagon/HexagonVectorCombine.cpp
+++ b/llvm/lib/Target/Hexagon/HexagonVectorCombine.cpp
@@ -304,6 +304,8 @@ raw_ostream &operator<<(raw_ostream &OS, const AlignVectors::AddrInfo &AI) {
 
 LLVM_ATTRIBUTE_UNUSED
 raw_ostream &operator<<(raw_ostream &OS, const AlignVectors::MoveGroup &MG) {
+  OS << "IsLoad:" << (MG.IsLoad ? "yes" : "no");
+  OS << ", IsHvx:" << (MG.IsHvx ? "yes" : "no") << '\n';
   OS << "Main\n";
   for (Instruction *I : MG.Main)
     OS << "  " << *I << '\n';
@@ -316,8 +318,14 @@ raw_ostream &operator<<(raw_ostream &OS, const AlignVectors::MoveGroup &MG) {
 LLVM_ATTRIBUTE_UNUSED
 raw_ostream &operator<<(raw_ostream &OS,
                         const AlignVectors::ByteSpan::Block &B) {
-  OS << "  @" << B.Pos << " [" << B.Seg.Start << ',' << B.Seg.Size << "] "
-     << *B.Seg.Val;
+  OS << "  @" << B.Pos << " [" << B.Seg.Start << ',' << B.Seg.Size << "] ";
+  if (B.Seg.Val == reinterpret_cast<const Value *>(&B)) {
+    OS << "(self:" << B.Seg.Val << ')';
+  } else if (B.Seg.Val != nullptr) {
+    OS << *B.Seg.Val;
+  } else {
+    OS << "(null)";
+  }
   return OS;
 }
 
@@ -573,26 +581,26 @@ auto AlignVectors::createAdjustedPointer(IRBuilderBase &Builder, Value *Ptr,
     Type *ElemTy = PtrTy->getNonOpaquePointerElementType();
     int ElemSize = HVC.getSizeOf(ElemTy, HVC.Alloc);
     if (Adjust % ElemSize == 0 && Adjust != 0) {
-      Value *Tmp0 =
-          Builder.CreateGEP(ElemTy, Ptr, HVC.getConstInt(Adjust / ElemSize));
-      return Builder.CreatePointerCast(Tmp0, ValTy->getPointerTo());
+      Value *Tmp0 = Builder.CreateGEP(
+          ElemTy, Ptr, HVC.getConstInt(Adjust / ElemSize), "gep");
+      return Builder.CreatePointerCast(Tmp0, ValTy->getPointerTo(), "cst");
     }
   }
 
   PointerType *CharPtrTy = Type::getInt8PtrTy(HVC.F.getContext());
-  Value *Tmp0 = Builder.CreatePointerCast(Ptr, CharPtrTy);
+  Value *Tmp0 = Builder.CreatePointerCast(Ptr, CharPtrTy, "cst");
   Value *Tmp1 = Builder.CreateGEP(Type::getInt8Ty(HVC.F.getContext()), Tmp0,
-                                  HVC.getConstInt(Adjust));
-  return Builder.CreatePointerCast(Tmp1, ValTy->getPointerTo());
+                                  HVC.getConstInt(Adjust), "gep");
+  return Builder.CreatePointerCast(Tmp1, ValTy->getPointerTo(), "cst");
 }
 
 auto AlignVectors::createAlignedPointer(IRBuilderBase &Builder, Value *Ptr,
                                         Type *ValTy, int Alignment) const
     -> Value * {
-  Value *AsInt = Builder.CreatePtrToInt(Ptr, HVC.getIntTy());
+  Value *AsInt = Builder.CreatePtrToInt(Ptr, HVC.getIntTy(), "pti");
   Value *Mask = HVC.getConstInt(-Alignment);
-  Value *And = Builder.CreateAnd(AsInt, Mask);
-  return Builder.CreateIntToPtr(And, ValTy->getPointerTo());
+  Value *And = Builder.CreateAnd(AsInt, Mask, "add");
+  return Builder.CreateIntToPtr(And, ValTy->getPointerTo(), "itp");
 }
 
 auto AlignVectors::createAlignedLoad(IRBuilderBase &Builder, Type *ValTy,
@@ -602,8 +610,9 @@ auto AlignVectors::createAlignedLoad(IRBuilderBase &Builder, Type *ValTy,
   if (HVC.isZero(Mask))
     return PassThru;
   if (Mask == ConstantInt::getTrue(Mask->getType()))
-    return Builder.CreateAlignedLoad(ValTy, Ptr, Align(Alignment));
-  return Builder.CreateMaskedLoad(ValTy, Ptr, Align(Alignment), Mask, PassThru);
+    return Builder.CreateAlignedLoad(ValTy, Ptr, Align(Alignment), "ald");
+  return Builder.CreateMaskedLoad(ValTy, Ptr, Align(Alignment), Mask, PassThru,
+                                  "mld");
 }
 
 auto AlignVectors::createAlignedStore(IRBuilderBase &Builder, Value *Val,
@@ -814,6 +823,8 @@ auto AlignVectors::realignLoadGroup(IRBuilderBase &Builder,
                                     const ByteSpan &VSpan, int ScLen,
                                     Value *AlignVal, Value *AlignAddr) const
     -> void {
+  LLVM_DEBUG(dbgs() << __func__ << "\n");
+
   Type *SecTy = HVC.getByteTy(ScLen);
   int NumSectors = (VSpan.extent() + ScLen - 1) / ScLen;
   bool DoAlign = !HVC.isZero(AlignVal);
@@ -869,7 +880,7 @@ auto AlignVectors::realignLoadGroup(IRBuilderBase &Builder,
       assert(I != nullptr && "Load used in a non-instruction?");
       // Make sure we only consider at users in this block, but we need
       // to remember if there were users outside the block too. This is
-      // because if there are no users, aligned loads will not be created.
+      // because if no users are found, aligned loads will not be created.
       if (I->getParent() == BaseBlock) {
         if (!isa<PHINode>(I))
           User = std::min(User, I, isEarlier);
@@ -888,6 +899,14 @@ auto AlignVectors::realignLoadGroup(IRBuilderBase &Builder,
     }
   }
 
+  LLVM_DEBUG({
+    dbgs() << "ASpan:\n" << ASpan << '\n';
+    dbgs() << "Earliest users of ASpan:\n";
+    for (auto &[Val, User] : EarliestUser) {
+      dbgs() << Val << "\n ->" << *User << '\n';
+    }
+  });
+
   auto createLoad = [&](IRBuilderBase &Builder, const ByteSpan &VSpan,
                         int Index) {
     Value *Ptr =
@@ -913,11 +932,14 @@ auto AlignVectors::realignLoadGroup(IRBuilderBase &Builder,
   };
 
   // Generate necessary loads at appropriate locations.
+  LLVM_DEBUG(dbgs() << "Creating loads for ASpan sectors\n");
   for (int Index = 0; Index != NumSectors + 1; ++Index) {
     // In ASpan, each block will be either a single aligned load, or a
     // valign of a pair of loads. In the latter case, an aligned load j
     // will belong to the current valign, and the one in the previous
     // block (for j > 0).
+    // Place the load at a location which will dominate the valign, assuming
+    // the valign will be placed right before the earliest user.
     Instruction *PrevAt =
         DoAlign && Index > 0 ? EarliestUser[&ASpan[Index - 1]] : nullptr;
     Instruction *ThisAt =
@@ -925,16 +947,20 @@ auto AlignVectors::realignLoadGroup(IRBuilderBase &Builder,
     if (auto *Where = std::min(PrevAt, ThisAt, isEarlier)) {
       Builder.SetInsertPoint(Where);
       Loads[Index] = createLoad(Builder, VSpan, Index);
-      // We know it's safe to put the load at BasePos, so if it's not safe
-      // to move it from this location to BasePos, then the current location
-      // is not valid.
+      // We know it's safe to put the load at BasePos, but we'd prefer to put
+      // it at "Where". To see if the load is safe to be placed at Where, put
+      // it there first and then check if it's safe to move it to BasePos.
+      // If not, then the load needs to be placed at BasePos.
       // We can't do this check proactively because we need the load to exist
       // in order to check legality.
       if (!HVC.isSafeToMoveBeforeInBB(*Loads[Index], BasePos))
         moveBefore(Loads[Index], &*BasePos);
+      LLVM_DEBUG(dbgs() << "Loads[" << Index << "]:" << *Loads[Index] << '\n');
     }
   }
+
   // Generate valigns if needed, and fill in proper values in ASpan
+  LLVM_DEBUG(dbgs() << "Creating values for ASpan sectors\n");
   for (int Index = 0; Index != NumSectors; ++Index) {
     ASpan[Index].Seg.Val = nullptr;
     if (auto *Where = EarliestUser[&ASpan[Index]]) {
@@ -947,6 +973,7 @@ auto AlignVectors::realignLoadGroup(IRBuilderBase &Builder,
         Val = HVC.vralignb(Builder, Val, NextLoad, AlignVal);
       }
       ASpan[Index].Seg.Val = Val;
+      LLVM_DEBUG(dbgs() << "ASpan[" << Index << "]:" << *Val << '\n');
     }
   }
 
@@ -955,15 +982,27 @@ auto AlignVectors::realignLoadGroup(IRBuilderBase &Builder,
     Value *Accum = UndefValue::get(HVC.getByteTy(B.Seg.Size));
     Builder.SetInsertPoint(cast<Instruction>(B.Seg.Val));
 
+    // We're generating a reduction, where each instruction depends on
+    // the previous one, so we need to order them according to the position
+    // of their inputs in the code.
+    std::vector<ByteSpan::Block *> ABlocks;
     for (ByteSpan::Block &S : ASection) {
-      if (S.Seg.Val == nullptr)
-        continue;
+      if (S.Seg.Val != nullptr)
+        ABlocks.push_back(&S);
+    }
+    llvm::sort(ABlocks,
+               [&](const ByteSpan::Block *A, const ByteSpan::Block *B) {
+                 return isEarlier(cast<Instruction>(A->Seg.Val),
+                                  cast<Instruction>(B->Seg.Val));
+               });
+    for (ByteSpan::Block *S : ABlocks) {
       // The processing of the data loaded by the aligned loads
       // needs to be inserted after the data is available.
-      Instruction *SegI = cast<Instruction>(S.Seg.Val);
+      Instruction *SegI = cast<Instruction>(S->Seg.Val);
       Builder.SetInsertPoint(&*std::next(SegI->getIterator()));
-      Value *Pay = HVC.vbytes(Builder, getPayload(S.Seg.Val));
-      Accum = HVC.insertb(Builder, Accum, Pay, S.Seg.Start, S.Seg.Size, S.Pos);
+      Value *Pay = HVC.vbytes(Builder, getPayload(S->Seg.Val));
+      Accum =
+          HVC.insertb(Builder, Accum, Pay, S->Seg.Start, S->Seg.Size, S->Pos);
     }
     // Instead of casting everything to bytes for the vselect, cast to the
     // original value type. This will avoid complications with casting masks.
@@ -972,9 +1011,9 @@ auto AlignVectors::realignLoadGroup(IRBuilderBase &Builder,
     // but if the mask is not exactly of HVX length, extra handling would be
     // needed to make it work.
     Type *ValTy = getPayload(B.Seg.Val)->getType();
-    Value *Cast = Builder.CreateBitCast(Accum, ValTy);
+    Value *Cast = Builder.CreateBitCast(Accum, ValTy, "cst");
     Value *Sel = Builder.CreateSelect(getMask(B.Seg.Val), Cast,
-                                      getPassThrough(B.Seg.Val));
+                                      getPassThrough(B.Seg.Val), "sel");
     B.Seg.Val->replaceAllUsesWith(Sel);
   }
 }
@@ -983,6 +1022,8 @@ auto AlignVectors::realignStoreGroup(IRBuilderBase &Builder,
                                      const ByteSpan &VSpan, int ScLen,
                                      Value *AlignVal, Value *AlignAddr) const
     -> void {
+  LLVM_DEBUG(dbgs() << __func__ << "\n");
+
   Type *SecTy = HVC.getByteTy(ScLen);
   int NumSectors = (VSpan.extent() + ScLen - 1) / ScLen;
   bool DoAlign = !HVC.isZero(AlignVal);
@@ -997,7 +1038,7 @@ auto AlignVectors::realignStoreGroup(IRBuilderBase &Builder,
     if (Ty->isVectorTy())
       return Val;
     auto *VecTy = VectorType::get(Ty, 1, /*Scalable=*/false);
-    return Builder.CreateBitCast(Val, VecTy);
+    return Builder.CreateBitCast(Val, VecTy, "cst");
   };
 
   // Create an extra "undef" sector at the beginning and at the end.
@@ -1050,6 +1091,8 @@ auto AlignVectors::realignStoreGroup(IRBuilderBase &Builder,
 }
 
 auto AlignVectors::realignGroup(const MoveGroup &Move) const -> bool {
+  LLVM_DEBUG(dbgs() << "Realigning group:\n" << Move << '\n');
+
   // TODO: Needs support for masked loads/stores of "scalar" vectors.
   if (!Move.IsHvx)
     return false;
@@ -1137,7 +1180,8 @@ auto AlignVectors::realignGroup(const MoveGroup &Move) const -> bool {
     // aligned instruction with an address that is not really aligned.
     AlignAddr = createAlignedPointer(Builder, WithMinOffset.Addr,
                                      WithMinOffset.ValTy, MinNeeded.value());
-    AlignVal = Builder.CreatePtrToInt(WithMinOffset.Addr, HVC.getIntTy());
+    AlignVal =
+        Builder.CreatePtrToInt(WithMinOffset.Addr, HVC.getIntTy(), "pti");
   }
 
   ByteSpan VSpan;
@@ -1154,6 +1198,13 @@ auto AlignVectors::realignGroup(const MoveGroup &Move) const -> bool {
   assert(!Move.IsHvx || ScLen == 64 || ScLen == 128);
   assert(Move.IsHvx || ScLen == 4 || ScLen == 8);
 
+  LLVM_DEBUG({
+    dbgs() << "ScLen:  " << ScLen << "\n";
+    dbgs() << "AlignVal:" << *AlignVal << "\n";
+    dbgs() << "AlignAddr:" << *AlignAddr << "\n";
+    dbgs() << "VSpan:\n" << VSpan << '\n';
+  });
+
   if (Move.IsLoad)
     realignLoadGroup(Builder, VSpan, ScLen, AlignVal, AlignAddr);
   else
@@ -1175,9 +1226,18 @@ auto AlignVectors::isSectorTy(Type *Ty) const -> bool {
 }
 
 auto AlignVectors::run() -> bool {
+  LLVM_DEBUG(dbgs() << "Running HVC::AlignVectors\n");
   if (!createAddressGroups())
     return false;
 
+  LLVM_DEBUG({
+    dbgs() << "Address groups(" << AddrGroups.size() << "):\n";
+    for (auto &[In, AL] : AddrGroups) {
+      for (const AddrInfo &AI : AL)
+        dbgs() << "---\n" << AI << '\n';
+    }
+  });
+
   bool Changed = false;
   MoveList LoadGroups, StoreGroups;
 
@@ -1186,11 +1246,22 @@ auto AlignVectors::run() -> bool {
     llvm::append_range(StoreGroups, createStoreGroups(G.second));
   }
 
+  LLVM_DEBUG({
+    dbgs() << "\nLoad groups(" << LoadGroups.size() << "):\n";
+    for (const MoveGroup &G : LoadGroups)
+      dbgs() << G << "\n";
+    dbgs() << "Store groups(" << StoreGroups.size() << "):\n";
+    for (const MoveGroup &G : StoreGroups)
+      dbgs() << G << "\n";
+  });
+
   for (auto &M : LoadGroups)
     Changed |= move(M);
   for (auto &M : StoreGroups)
     Changed |= move(M);
 
+  LLVM_DEBUG(dbgs() << "After move:\n" << HVC.F);
+
   for (auto &M : LoadGroups)
     Changed |= realignGroup(M);
   for (auto &M : StoreGroups)
@@ -1356,13 +1427,13 @@ auto HvxIdioms::processFxpMul(Instruction &In, const FxpOp &Op) const
 
   auto *ResizeTy = VectorType::get(HVC.getIntTy(Width), VecTy);
   if (Width < ElemWidth) {
-    X = Builder.CreateTrunc(X, ResizeTy);
-    Y = Builder.CreateTrunc(Y, ResizeTy);
+    X = Builder.CreateTrunc(X, ResizeTy, "trn");
+    Y = Builder.CreateTrunc(Y, ResizeTy, "trn");
   } else if (Width > ElemWidth) {
-    X = SignX == Signed ? Builder.CreateSExt(X, ResizeTy)
-                        : Builder.CreateZExt(X, ResizeTy);
-    Y = SignY == Signed ? Builder.CreateSExt(Y, ResizeTy)
-                        : Builder.CreateZExt(Y, ResizeTy);
+    X = SignX == Signed ? Builder.CreateSExt(X, ResizeTy, "sxt")
+                        : Builder.CreateZExt(X, ResizeTy, "zxt");
+    Y = SignY == Signed ? Builder.CreateSExt(Y, ResizeTy, "sxt")
+                        : Builder.CreateZExt(Y, ResizeTy, "zxt");
   };
 
   assert(X->getType() == Y->getType() && X->getType() == ResizeTy);
@@ -1387,8 +1458,8 @@ auto HvxIdioms::processFxpMul(Instruction &In, const FxpOp &Op) const
 
   Value *Cat = HVC.concat(Builder, Results);
   Value *Ext = SignX == Signed || SignY == Signed
-                   ? Builder.CreateSExt(Cat, VecTy)
-                   : Builder.CreateZExt(Cat, VecTy);
+                   ? Builder.CreateSExt(Cat, VecTy, "sxt")
+                   : Builder.CreateZExt(Cat, VecTy, "zxt");
   return Ext;
 }
 
@@ -1434,14 +1505,14 @@ auto HvxIdioms::processFxpMulChopped(IRBuilderBase &Builder, Instruction &In,
     Value *Prod32 = createMul16(Builder, Op.X, Op.Y);
     if (Rounding) {
       Value *RoundVal = HVC.getConstSplat(Prod32->getType(), 1 << *Op.RoundAt);
-      Prod32 = Builder.CreateAdd(Prod32, RoundVal);
+      Prod32 = Builder.CreateAdd(Prod32, RoundVal, "add");
     }
 
     Value *ShiftAmt = HVC.getConstSplat(Prod32->getType(), Op.Frac);
     Value *Shifted = Op.X.Sgn == Signed || Op.Y.Sgn == Signed
-               ? Builder.CreateAShr(Prod32, ShiftAmt)
-               : Builder.CreateLShr(Prod32, ShiftAmt);
-    return Builder.CreateTrunc(Shifted, InpTy);
+                         ? Builder.CreateAShr(Prod32, ShiftAmt, "asr")
+                         : Builder.CreateLShr(Prod32, ShiftAmt, "lsr");
+    return Builder.CreateTrunc(Shifted, InpTy, "trn");
   }
 
   // Width >= 32
@@ -1475,10 +1546,11 @@ auto HvxIdioms::processFxpMulChopped(IRBuilderBase &Builder, Instruction &In,
     if (Src + 1 < End) {
       Value *Hi = WordP[Src + 1];
       WordP[Dst] = Builder.CreateIntrinsic(HvxWordTy, Intrinsic::fshr,
-                                           {Hi, Lo, ShiftAmt});
+                                           {Hi, Lo, ShiftAmt},
+                                           /*FMFSource*/ nullptr, "int");
     } else {
       // The shift of the most significant word.
-      WordP[Dst] = Builder.CreateAShr(Lo, ShiftAmt);
+      WordP[Dst] = Builder.CreateAShr(Lo, ShiftAmt, "asr");
     }
   }
   if (SkipWords != 0)
@@ -1540,8 +1612,8 @@ auto HvxIdioms::createAddCarry(IRBuilderBase &Builder, Value *X, Value *Y,
     }
     Value *Ret = HVC.createHvxIntrinsic(Builder, AddCarry,
                                         /*RetTy=*/nullptr, Args);
-    Value *Result = Builder.CreateExtractValue(Ret, {0});
-    Value *CarryOut = Builder.CreateExtractValue(Ret, {1});
+    Value *Result = Builder.CreateExtractValue(Ret, {0}, "ext");
+    Value *CarryOut = Builder.CreateExtractValue(Ret, {1}, "ext");
     return {Result, CarryOut};
   }
 
@@ -1560,13 +1632,13 @@ auto HvxIdioms::createAddCarry(IRBuilderBase &Builder, Value *X, Value *Y,
     Value *ValueIn =
         HVC.createHvxIntrinsic(Builder, V6_vandqrt, /*RetTy=*/nullptr,
                                {CarryIn, HVC.getConstInt(Mask)});
-    Result1 = Builder.CreateAdd(X, ValueIn);
+    Result1 = Builder.CreateAdd(X, ValueIn, "add");
   }
 
-  Value *CarryOut1 = Builder.CreateCmp(CmpInst::ICMP_ULT, Result1, X);
-  Value *Result2 = Builder.CreateAdd(Result1, Y);
-  Value *CarryOut2 = Builder.CreateCmp(CmpInst::ICMP_ULT, Result2, Y);
-  return {Result2, Builder.CreateOr(CarryOut1, CarryOut2)};
+  Value *CarryOut1 = Builder.CreateCmp(CmpInst::ICMP_ULT, Result1, X, "cmp");
+  Value *Result2 = Builder.CreateAdd(Result1, Y, "add");
+  Value *CarryOut2 = Builder.CreateCmp(CmpInst::ICMP_ULT, Result2, Y, "cmp");
+  return {Result2, Builder.CreateOr(CarryOut1, CarryOut2, "orb")};
 }
 
 auto HvxIdioms::createMul16(IRBuilderBase &Builder, SValue X, SValue Y) const
@@ -1603,15 +1675,16 @@ auto HvxIdioms::createMulH16(IRBuilderBase &Builder, SValue X, SValue Y) const
   }
 
   Type *HvxP16Ty = HVC.getHvxTy(HVC.getIntTy(16), /*Pair=*/true);
-  Value *Pair16 = Builder.CreateBitCast(createMul16(Builder, X, Y), HvxP16Ty);
+  Value *Pair16 =
+      Builder.CreateBitCast(createMul16(Builder, X, Y), HvxP16Ty, "cst");
   unsigned Len = HVC.length(HvxP16Ty) / 2;
 
   SmallVector<int, 128> PickOdd(Len);
   for (int i = 0; i != static_cast<int>(Len); ++i)
     PickOdd[i] = 2 * i + 1;
 
-  return Builder.CreateShuffleVector(HVC.sublo(Builder, Pair16),
-                                     HVC.subhi(Builder, Pair16), PickOdd);
+  return Builder.CreateShuffleVector(
+      HVC.sublo(Builder, Pair16), HVC.subhi(Builder, Pair16), PickOdd, "shf");
 }
 
 auto HvxIdioms::createMul32(IRBuilderBase &Builder, SValue X, SValue Y) const
@@ -1632,8 +1705,8 @@ auto HvxIdioms::createMul32(IRBuilderBase &Builder, SValue X, SValue Y) const
 
   Value *Parts = HVC.createHvxIntrinsic(Builder, V6_vmpy_parts, nullptr,
                                         {X.Val, Y.Val}, {HvxI32Ty});
-  Value *Hi = Builder.CreateExtractValue(Parts, {0});
-  Value *Lo = Builder.CreateExtractValue(Parts, {1});
+  Value *Hi = Builder.CreateExtractValue(Parts, {0}, "ext");
+  Value *Lo = Builder.CreateExtractValue(Parts, {1}, "ext");
   return {Lo, Hi};
 }
 
@@ -1899,7 +1972,7 @@ auto HexagonVectorCombine::insertb(IRBuilderBase &Builder, Value *Dst,
         (Where <= i && i < Where + Length) ? P2Len + Start + (i - Where) : i;
   }
 
-  Value *P2Insert = Builder.CreateShuffleVector(P2Dst, P2Src, SMask);
+  Value *P2Insert = Builder.CreateShuffleVector(P2Dst, P2Src, SMask, "shf");
   return vresize(Builder, P2Insert, DstLen, Undef);
 }
 
@@ -1922,12 +1995,14 @@ auto HexagonVectorCombine::vlalignb(IRBuilderBase &Builder, Value *Lo,
 
   if (VecLen == 4) {
     Value *Pair = concat(Builder, {Lo, Hi});
-    Value *Shift = Builder.CreateLShr(Builder.CreateShl(Pair, Amt), 32);
-    Value *Trunc = Builder.CreateTrunc(Shift, Type::getInt32Ty(F.getContext()));
-    return Builder.CreateBitCast(Trunc, Hi->getType());
+    Value *Shift =
+        Builder.CreateLShr(Builder.CreateShl(Pair, Amt, "shl"), 32, "lsr");
+    Value *Trunc =
+        Builder.CreateTrunc(Shift, Type::getInt32Ty(F.getContext()), "trn");
+    return Builder.CreateBitCast(Trunc, Hi->getType(), "cst");
   }
   if (VecLen == 8) {
-    Value *Sub = Builder.CreateSub(getConstInt(VecLen), Amt);
+    Value *Sub = Builder.CreateSub(getConstInt(VecLen), Amt, "sub");
     return vralignb(Builder, Lo, Hi, Sub);
   }
   llvm_unreachable("Unexpected vector length");
@@ -1951,18 +2026,19 @@ auto HexagonVectorCombine::vralignb(IRBuilderBase &Builder, Value *Lo,
 
   if (VecLen == 4) {
     Value *Pair = concat(Builder, {Lo, Hi});
-    Value *Shift = Builder.CreateLShr(Pair, Amt);
-    Value *Trunc = Builder.CreateTrunc(Shift, Type::getInt32Ty(F.getContext()));
-    return Builder.CreateBitCast(Trunc, Lo->getType());
+    Value *Shift = Builder.CreateLShr(Pair, Amt, "lsr");
+    Value *Trunc =
+        Builder.CreateTrunc(Shift, Type::getInt32Ty(F.getContext()), "trn");
+    return Builder.CreateBitCast(Trunc, Lo->getType(), "cst");
   }
   if (VecLen == 8) {
     Type *Int64Ty = Type::getInt64Ty(F.getContext());
-    Value *Lo64 = Builder.CreateBitCast(Lo, Int64Ty);
-    Value *Hi64 = Builder.CreateBitCast(Hi, Int64Ty);
+    Value *Lo64 = Builder.CreateBitCast(Lo, Int64Ty, "cst");
+    Value *Hi64 = Builder.CreateBitCast(Hi, Int64Ty, "cst");
     Function *FI = Intrinsic::getDeclaration(F.getParent(),
                                              Intrinsic::hexagon_S2_valignrb);
-    Value *Call = Builder.CreateCall(FI, {Hi64, Lo64, Amt});
-    return Builder.CreateBitCast(Call, Lo->getType());
+    Value *Call = Builder.CreateCall(FI, {Hi64, Lo64, Amt}, "cup");
+    return Builder.CreateBitCast(Call, Lo->getType(), "cst");
   }
   llvm_unreachable("Unexpected vector length");
 }
@@ -1985,8 +2061,8 @@ auto HexagonVectorCombine::concat(IRBuilderBase &Builder,
     if (Work[ThisW].size() % 2 != 0)
       Work[ThisW].push_back(UndefValue::get(Ty));
     for (int i = 0, e = Work[ThisW].size(); i < e; i += 2) {
-      Value *Joined = Builder.CreateShuffleVector(Work[ThisW][i],
-                                                  Work[ThisW][i + 1], SMask);
+      Value *Joined = Builder.CreateShuffleVector(
+          Work[ThisW][i], Work[ThisW][i + 1], SMask, "shf");
       Work[OtherW].push_back(Joined);
     }
     std::swap(ThisW, OtherW);
@@ -1998,7 +2074,7 @@ auto HexagonVectorCombine::concat(IRBuilderBase &Builder,
   SMask.resize(Vecs.size() * length(Vecs.front()->getType()));
   std::iota(SMask.begin(), SMask.end(), 0);
   Value *Total = Work[ThisW].front();
-  return Builder.CreateShuffleVector(Total, SMask);
+  return Builder.CreateShuffleVector(Total, SMask, "shf");
 }
 
 auto HexagonVectorCombine::vresize(IRBuilderBase &Builder, Value *Val,
@@ -2017,8 +2093,8 @@ auto HexagonVectorCombine::vresize(IRBuilderBase &Builder, Value *Val,
   SmallVector<int, 128> SMask(NewSize);
   std::iota(SMask.begin(), SMask.begin() + CurSize, 0);
   std::fill(SMask.begin() + CurSize, SMask.end(), CurSize);
-  Value *PadVec = Builder.CreateVectorSplat(CurSize, Pad);
-  return Builder.CreateShuffleVector(Val, PadVec, SMask);
+  Value *PadVec = Builder.CreateVectorSplat(CurSize, Pad, "spt");
+  return Builder.CreateShuffleVector(Val, PadVec, SMask, "shf");
 }
 
 auto HexagonVectorCombine::rescale(IRBuilderBase &Builder, Value *Mask,
@@ -2048,11 +2124,11 @@ auto HexagonVectorCombine::rescale(IRBuilderBase &Builder, Value *Mask,
   // Mask <N x i1> -> sext to <N x FromTy> -> bitcast to <M x ToTy> ->
   // -> trunc to <M x i1>.
   Value *Ext = Builder.CreateSExt(
-      Mask, VectorType::get(FromITy, FromCount, /*Scalable=*/false));
+      Mask, VectorType::get(FromITy, FromCount, /*Scalable=*/false), "sxt");
   Value *Cast = Builder.CreateBitCast(
-      Ext, VectorType::get(ToITy, ToCount, /*Scalable=*/false));
+      Ext, VectorType::get(ToITy, ToCount, /*Scalable=*/false), "cst");
   return Builder.CreateTrunc(
-      Cast, VectorType::get(getBoolTy(), ToCount, /*Scalable=*/false));
+      Cast, VectorType::get(getBoolTy(), ToCount, /*Scalable=*/false), "trn");
 }
 
 // Bitcast to bytes, and return least significant bits.
@@ -2064,10 +2140,10 @@ auto HexagonVectorCombine::vlsb(IRBuilderBase &Builder, Value *Val) const
 
   Value *Bytes = vbytes(Builder, Val);
   if (auto *VecTy = dyn_cast<VectorType>(Bytes->getType()))
-    return Builder.CreateTrunc(Bytes, getBoolTy(getSizeOf(VecTy)));
+    return Builder.CreateTrunc(Bytes, getBoolTy(getSizeOf(VecTy)), "trn");
   // If Bytes is a scalar (i.e. Val was a scalar byte), return i1, not
   // <1 x i1>.
-  return Builder.CreateTrunc(Bytes, getBoolTy());
+  return Builder.CreateTrunc(Bytes, getBoolTy(), "trn");
 }
 
 // Bitcast to bytes for non-bool. For bool, convert i1 -> i8.
@@ -2078,11 +2154,11 @@ auto HexagonVectorCombine::vbytes(IRBuilderBase &Builder, Value *Val) const
     return Val;
 
   if (ScalarTy != getBoolTy())
-    return Builder.CreateBitCast(Val, getByteTy(getSizeOf(Val)));
+    return Builder.CreateBitCast(Val, getByteTy(getSizeOf(Val)), "cst");
   // For bool, return a sext from i1 to i8.
   if (auto *VecTy = dyn_cast<VectorType>(Val->getType()))
-    return Builder.CreateSExt(Val, VectorType::get(getByteTy(), VecTy));
-  return Builder.CreateSExt(Val, getByteTy());
+    return Builder.CreateSExt(Val, VectorType::get(getByteTy(), VecTy), "sxt");
+  return Builder.CreateSExt(Val, getByteTy(), "sxt");
 }
 
 auto HexagonVectorCombine::subvector(IRBuilderBase &Builder, Value *Val,
@@ -2116,7 +2192,7 @@ auto HexagonVectorCombine::vdeal(IRBuilderBase &Builder, Value *Val0,
     Mask[i] = 2 * i;           // Even
     Mask[i + Len] = 2 * i + 1; // Odd
   }
-  return Builder.CreateShuffleVector(Val0, Val1, Mask);
+  return Builder.CreateShuffleVector(Val0, Val1, Mask, "shf");
 }
 
 auto HexagonVectorCombine::vshuff(IRBuilderBase &Builder, Value *Val0,
@@ -2129,7 +2205,7 @@ auto HexagonVectorCombine::vshuff(IRBuilderBase &Builder, Value *Val0,
     Mask[2 * i + 0] = i;       // Val0
     Mask[2 * i + 1] = i + Len; // Val1
   }
-  return Builder.CreateShuffleVector(Val0, Val1, Mask);
+  return Builder.CreateShuffleVector(Val0, Val1, Mask, "shf");
 }
 
 auto HexagonVectorCombine::createHvxIntrinsic(IRBuilderBase &Builder,
@@ -2149,7 +2225,7 @@ auto HexagonVectorCombine::createHvxIntrinsic(IRBuilderBase &Builder,
 
     Type *BoolTy = Type::getInt1Ty(F.getContext());
     if (cast<VectorType>(SrcTy)->getElementType() != BoolTy)
-      return Builder.CreateBitCast(Val, DestTy);
+      return Builder.CreateBitCast(Val, DestTy, "cst");
 
     // Predicate HVX vector.
     unsigned HwLen = HST.getVectorLength();
@@ -2157,7 +2233,7 @@ auto HexagonVectorCombine::createHvxIntrinsic(IRBuilderBase &Builder,
                                    : Intrinsic::hexagon_V6_pred_typecast_128B;
     Function *FI =
         Intrinsic::getDeclaration(F.getParent(), TC, {DestTy, Val->getType()});
-    return Builder.CreateCall(FI, {Val});
+    return Builder.CreateCall(FI, {Val}, "cup");
   };
 
   Function *IntrFn = Intrinsic::getDeclaration(F.getParent(), IntID, ArgTys);
@@ -2173,7 +2249,7 @@ auto HexagonVectorCombine::createHvxIntrinsic(IRBuilderBase &Builder,
       IntrArgs.push_back(A);
     }
   }
-  Value *Call = Builder.CreateCall(IntrFn, IntrArgs);
+  Value *Call = Builder.CreateCall(IntrFn, IntrArgs, "cup");
 
   Type *CallTy = Call->getType();
   if (RetTy == nullptr || CallTy == RetTy)
@@ -2223,7 +2299,7 @@ auto HexagonVectorCombine::splitVectorElements(IRBuilderBase &Builder,
     unsigned Width = Val->getType()->getScalarSizeInBits();
 
     auto *VTy = VectorType::get(getIntTy(Width / 2), 2 * Length, false);
-    Value *VVal = Builder.CreateBitCast(Val, VTy);
+    Value *VVal = Builder.CreateBitCast(Val, VTy, "cst");
 
     Value *Res = vdeal(Builder, sublo(Builder, VVal), subhi(Builder, VVal));
 
@@ -2265,8 +2341,8 @@ auto HexagonVectorCombine::joinVectorElements(IRBuilderBase &Builder,
     // Having too many inputs is ok: drop the high bits (usual wrap-around).
     // If there are too few, fill them with the sign bit.
     Value *Last = Inputs.back();
-    Value *Sign =
-        Builder.CreateAShr(Last, getConstSplat(Last->getType(), Width - 1));
+    Value *Sign = Builder.CreateAShr(
+        Last, getConstSplat(Last->getType(), Width - 1), "asr");
     Inputs.resize(NeedInputs, Sign);
   }
 
@@ -2275,7 +2351,7 @@ auto HexagonVectorCombine::joinVectorElements(IRBuilderBase &Builder,
     auto *VTy = VectorType::get(getIntTy(Width), Length, false);
     for (int i = 0, e = Inputs.size(); i < e; i += 2) {
       Value *Res = vshuff(Builder, Inputs[i], Inputs[i + 1]);
-      Inputs[i / 2] = Builder.CreateBitCast(Res, VTy);
+      Inputs[i / 2] = Builder.CreateBitCast(Res, VTy, "cst");
     }
     Inputs.resize(Inputs.size() / 2);
   }
@@ -2468,7 +2544,7 @@ auto HexagonVectorCombine::getElementRange(IRBuilderBase &Builder, Value *Lo,
   assert(0 <= Start && size_t(Start + Length) < length(Lo) + length(Hi));
   SmallVector<int, 128> SMask(Length);
   std::iota(SMask.begin(), SMask.end(), Start);
-  return Builder.CreateShuffleVector(Lo, Hi, SMask);
+  return Builder.CreateShuffleVector(Lo, Hi, SMask, "shf");
 }
 
 // Pass management.
diff --git a/llvm/lib/Target/LoongArch/LoongArch.td b/llvm/lib/Target/LoongArch/LoongArch.td
index 7e5c3563f39a2..0c9f64d7c52ed 100644
--- a/llvm/lib/Target/LoongArch/LoongArch.td
+++ b/llvm/lib/Target/LoongArch/LoongArch.td
@@ -30,7 +30,7 @@ def IsLA32
                          "LA32 Basic Integer and Privilege Instruction Set">;
 
 defvar LA32 = DefaultMode;
-def LA64 : HwMode<"+64bit">;
+def LA64 : HwMode<"+64bit", [IsLA64]>;
 
 // Single Precision floating point
 def FeatureBasicF
diff --git a/llvm/lib/Target/LoongArch/LoongArchAsmPrinter.cpp b/llvm/lib/Target/LoongArch/LoongArchAsmPrinter.cpp
index 6d9cb5e174d9a..04fdd41d67730 100644
--- a/llvm/lib/Target/LoongArch/LoongArchAsmPrinter.cpp
+++ b/llvm/lib/Target/LoongArch/LoongArchAsmPrinter.cpp
@@ -35,6 +35,12 @@ void LoongArchAsmPrinter::emitInstruction(const MachineInstr *MI) {
   if (emitPseudoExpansionLowering(*OutStreamer, MI))
     return;
 
+  switch (MI->getOpcode()) {
+  case TargetOpcode::PATCHABLE_FUNCTION_ENTER:
+    LowerPATCHABLE_FUNCTION_ENTER(*MI);
+    return;
+  }
+
   MCInst TmpInst;
   if (!lowerLoongArchMachineInstrToMCInst(MI, TmpInst, *this))
     EmitToStreamer(*OutStreamer, TmpInst);
@@ -110,6 +116,22 @@ bool LoongArchAsmPrinter::PrintAsmMemoryOperand(const MachineInstr *MI,
   return false;
 }
 
+void LoongArchAsmPrinter::LowerPATCHABLE_FUNCTION_ENTER(
+    const MachineInstr &MI) {
+  const Function &F = MF->getFunction();
+  if (F.hasFnAttribute("patchable-function-entry")) {
+    unsigned Num;
+    if (F.getFnAttribute("patchable-function-entry")
+            .getValueAsString()
+            .getAsInteger(10, Num))
+      return;
+    emitNops(Num);
+    return;
+  }
+
+  // TODO: Emit sled here once we get support for XRay.
+}
+
 bool LoongArchAsmPrinter::runOnMachineFunction(MachineFunction &MF) {
   AsmPrinter::runOnMachineFunction(MF);
   return true;
diff --git a/llvm/lib/Target/LoongArch/LoongArchAsmPrinter.h b/llvm/lib/Target/LoongArch/LoongArchAsmPrinter.h
index 23e29354743e6..c8bf657f8de7c 100644
--- a/llvm/lib/Target/LoongArch/LoongArchAsmPrinter.h
+++ b/llvm/lib/Target/LoongArch/LoongArchAsmPrinter.h
@@ -41,6 +41,8 @@ class LLVM_LIBRARY_VISIBILITY LoongArchAsmPrinter : public AsmPrinter {
   bool PrintAsmMemoryOperand(const MachineInstr *MI, unsigned OpNo,
                              const char *ExtraCode, raw_ostream &OS) override;
 
+  void LowerPATCHABLE_FUNCTION_ENTER(const MachineInstr &MI);
+
   // tblgen'erated function.
   bool emitPseudoExpansionLowering(MCStreamer &OutStreamer,
                                    const MachineInstr *MI);
diff --git a/llvm/lib/Target/LoongArch/LoongArchInstrInfo.cpp b/llvm/lib/Target/LoongArch/LoongArchInstrInfo.cpp
index fbbb764b8be1b..3e19f3e2fe08e 100644
--- a/llvm/lib/Target/LoongArch/LoongArchInstrInfo.cpp
+++ b/llvm/lib/Target/LoongArch/LoongArchInstrInfo.cpp
@@ -17,6 +17,7 @@
 #include "MCTargetDesc/LoongArchMCTargetDesc.h"
 #include "MCTargetDesc/LoongArchMatInt.h"
 #include "llvm/CodeGen/RegisterScavenging.h"
+#include "llvm/MC/MCInstBuilder.h"
 
 using namespace llvm;
 
@@ -28,6 +29,13 @@ LoongArchInstrInfo::LoongArchInstrInfo(LoongArchSubtarget &STI)
                             LoongArch::ADJCALLSTACKUP),
       STI(STI) {}
 
+MCInst LoongArchInstrInfo::getNop() const {
+  return MCInstBuilder(LoongArch::ANDI)
+      .addReg(LoongArch::R0)
+      .addReg(LoongArch::R0)
+      .addImm(0);
+}
+
 void LoongArchInstrInfo::copyPhysReg(MachineBasicBlock &MBB,
                                      MachineBasicBlock::iterator MBBI,
                                      const DebugLoc &DL, MCRegister DstReg,
diff --git a/llvm/lib/Target/LoongArch/LoongArchInstrInfo.h b/llvm/lib/Target/LoongArch/LoongArchInstrInfo.h
index e2b80460fcaf5..cf83abf27a1e3 100644
--- a/llvm/lib/Target/LoongArch/LoongArchInstrInfo.h
+++ b/llvm/lib/Target/LoongArch/LoongArchInstrInfo.h
@@ -27,6 +27,8 @@ class LoongArchInstrInfo : public LoongArchGenInstrInfo {
 public:
   explicit LoongArchInstrInfo(LoongArchSubtarget &STI);
 
+  MCInst getNop() const override;
+
   void copyPhysReg(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
                    const DebugLoc &DL, MCRegister DstReg, MCRegister SrcReg,
                    bool KillSrc) const override;
diff --git a/llvm/lib/Target/LoongArch/LoongArchInstrInfo.td b/llvm/lib/Target/LoongArch/LoongArchInstrInfo.td
index 0606c65f9d3b1..9ba648dfa0481 100644
--- a/llvm/lib/Target/LoongArch/LoongArchInstrInfo.td
+++ b/llvm/lib/Target/LoongArch/LoongArchInstrInfo.td
@@ -847,6 +847,13 @@ def : PatGprGpr<mulhs, MULH_W>;
 def : PatGprGpr<mulhu, MULH_WU>;
 def : PatGprGpr<rotr, ROTR_W>;
 def : PatGprImm<rotr, ROTRI_W, uimm5>;
+
+foreach Idx = 1...3 in {
+  defvar ShamtA = !mul(8, Idx);
+  defvar ShamtB = !mul(8, !sub(4, Idx));
+  def : Pat<(or (shl GPR:$rk, (i32 ShamtA)), (srl GPR:$rj, (i32 ShamtB))),
+            (BYTEPICK_W GPR:$rj, GPR:$rk, Idx)>;
+}
 } // Predicates = [IsLA32]
 
 let Predicates = [IsLA64] in {
@@ -891,6 +898,24 @@ def : Pat<(add GPR:$rj, simm32_hi16_lo12:$imm),
 def : Pat<(sext_inreg (add GPR:$rj, simm32_hi16_lo12:$imm), i32),
           (ADDI_W (ADDU16I_D GPR:$rj, (HI16ForAddu16idAddiPair $imm)),
                   (LO12 $imm))>;
+
+foreach Idx = 1...7 in {
+  defvar ShamtA = !mul(8, Idx);
+  defvar ShamtB = !mul(8, !sub(8, Idx));
+  def : Pat<(or (shl GPR:$rk, (i64 ShamtA)), (srl GPR:$rj, (i64 ShamtB))),
+            (BYTEPICK_D GPR:$rj, GPR:$rk, Idx)>;
+}
+
+foreach Idx = 1...3 in {
+  defvar ShamtA = !mul(8, Idx);
+  defvar ShamtB = !mul(8, !sub(4, Idx));
+  // NOTE: the srl node would already be transformed into a loongarch_bstrpick
+  // by the time this pattern gets to execute, hence the weird construction.
+  def : Pat<(sext_inreg (or (shl GPR:$rk, (i64 ShamtA)),
+                            (loongarch_bstrpick GPR:$rj, (i64 31),
+                                                         (i64 ShamtB))), i32),
+            (BYTEPICK_W GPR:$rj, GPR:$rk, Idx)>;
+}
 } // Predicates = [IsLA64]
 
 def : PatGprGpr<and, AND>;
diff --git a/llvm/lib/Target/M68k/M68kAsmPrinter.cpp b/llvm/lib/Target/M68k/M68kAsmPrinter.cpp
index 4933d40f33885..f748450c170aa 100644
--- a/llvm/lib/Target/M68k/M68kAsmPrinter.cpp
+++ b/llvm/lib/Target/M68k/M68kAsmPrinter.cpp
@@ -76,6 +76,90 @@ bool M68kAsmPrinter::PrintAsmOperand(const MachineInstr *MI, unsigned OpNo,
   return AsmPrinter::PrintAsmOperand(MI, OpNo, ExtraCode, OS);
 }
 
+void M68kAsmPrinter::printDisp(const MachineInstr *MI, unsigned opNum,
+                               raw_ostream &O) {
+  // Print immediate displacement without the '#' predix
+  const MachineOperand &Op = MI->getOperand(opNum);
+  if (Op.isImm()) {
+    O << Op.getImm();
+    return;
+  }
+  // Displacement is relocatable, so we're pretty permissive about what
+  // can be put here.
+  printOperand(MI, opNum, O);
+}
+
+void M68kAsmPrinter::printAbsMem(const MachineInstr *MI, unsigned OpNum,
+                                 raw_ostream &O) {
+  const MachineOperand &MO = MI->getOperand(OpNum);
+  if (MO.isImm())
+    O << format("$%0" PRIx64, (uint64_t)MO.getImm());
+  else
+    PrintAsmMemoryOperand(MI, OpNum, nullptr, O);
+}
+
+bool M68kAsmPrinter::PrintAsmMemoryOperand(const MachineInstr *MI,
+                                           unsigned OpNo, const char *ExtraCode,
+                                           raw_ostream &OS) {
+  const MachineOperand &MO = MI->getOperand(OpNo);
+  switch (MO.getType()) {
+  case MachineOperand::MO_Immediate:
+    // Immediate value that goes here is the addressing mode kind we set
+    // in M68kDAGToDAGISel::SelectInlineAsmMemoryOperand.
+    using namespace M68k;
+    // Skip the addressing mode kind operand.
+    ++OpNo;
+    // Decode MemAddrModeKind.
+    switch (static_cast<MemAddrModeKind>(MO.getImm())) {
+    case MemAddrModeKind::j:
+      printARIMem(MI, OpNo, OS);
+      break;
+    case MemAddrModeKind::o:
+      printARIPIMem(MI, OpNo, OS);
+      break;
+    case MemAddrModeKind::e:
+      printARIPDMem(MI, OpNo, OS);
+      break;
+    case MemAddrModeKind::p:
+      printARIDMem(MI, OpNo, OS);
+      break;
+    case MemAddrModeKind::f:
+    case MemAddrModeKind::F:
+      printARIIMem(MI, OpNo, OS);
+      break;
+    case MemAddrModeKind::k:
+      printPCIMem(MI, 0, OpNo, OS);
+      break;
+    case MemAddrModeKind::q:
+      printPCDMem(MI, 0, OpNo, OS);
+      break;
+    case MemAddrModeKind::b:
+      printAbsMem(MI, OpNo, OS);
+      break;
+    default:
+      llvm_unreachable("Unrecognized memory addressing mode");
+    }
+    return false;
+  case MachineOperand::MO_GlobalAddress:
+    PrintSymbolOperand(MO, OS);
+    return false;
+  case MachineOperand::MO_BlockAddress:
+    GetBlockAddressSymbol(MO.getBlockAddress())->print(OS, MAI);
+    return false;
+  case MachineOperand::MO_Register:
+    // This is a special case where it is treated as a memory reference, with
+    // the register holding the address value. Thus, we print it as ARI here.
+    if (M68kII::isAddressRegister(MO.getReg())) {
+      printARIMem(MI, OpNo, OS);
+      return false;
+    }
+    break;
+  default:
+    break;
+  }
+  return AsmPrinter::PrintAsmMemoryOperand(MI, OpNo, ExtraCode, OS);
+}
+
 void M68kAsmPrinter::emitInstruction(const MachineInstr *MI) {
   M68k_MC::verifyInstructionPredicates(MI->getOpcode(),
                                        getSubtargetInfo().getFeatureBits());
diff --git a/llvm/lib/Target/M68k/M68kAsmPrinter.h b/llvm/lib/Target/M68k/M68kAsmPrinter.h
index 1a76e3bf4e276..7b4dbfef58c57 100644
--- a/llvm/lib/Target/M68k/M68kAsmPrinter.h
+++ b/llvm/lib/Target/M68k/M68kAsmPrinter.h
@@ -16,6 +16,7 @@
 
 #include "M68kMCInstLower.h"
 #include "M68kTargetMachine.h"
+#include "MCTargetDesc/M68kMemOperandPrinter.h"
 
 #include "llvm/CodeGen/AsmPrinter.h"
 #include "llvm/MC/MCStreamer.h"
@@ -34,12 +35,19 @@ class raw_ostream;
 class M68kSubtarget;
 class M68kMachineFunctionInfo;
 
-class LLVM_LIBRARY_VISIBILITY M68kAsmPrinter : public AsmPrinter {
+class LLVM_LIBRARY_VISIBILITY M68kAsmPrinter
+    : public AsmPrinter,
+      public M68kMemOperandPrinter<M68kAsmPrinter, MachineInstr> {
+
+  friend class M68kMemOperandPrinter;
 
   void EmitInstrWithMacroNoAT(const MachineInstr *MI);
 
   void printOperand(const MachineInstr *MI, int OpNum, raw_ostream &OS);
 
+  void printDisp(const MachineInstr *MI, unsigned OpNum, raw_ostream &OS);
+  void printAbsMem(const MachineInstr *MI, unsigned OpNum, raw_ostream &OS);
+
 public:
   const M68kSubtarget *Subtarget;
   const M68kMachineFunctionInfo *MMFI;
@@ -57,6 +65,8 @@ class LLVM_LIBRARY_VISIBILITY M68kAsmPrinter : public AsmPrinter {
 
   bool PrintAsmOperand(const MachineInstr *MI, unsigned OpNo,
                        const char *ExtraCode, raw_ostream &OS) override;
+  bool PrintAsmMemoryOperand(const MachineInstr *MI, unsigned OpNo,
+                             const char *ExtraCode, raw_ostream &OS) override;
 
   void emitInstruction(const MachineInstr *MI) override;
   void emitFunctionBodyStart() override;
diff --git a/llvm/lib/Target/M68k/M68kISelDAGToDAG.cpp b/llvm/lib/Target/M68k/M68kISelDAGToDAG.cpp
index f8335f3dcd770..346470ed60d80 100644
--- a/llvm/lib/Target/M68k/M68kISelDAGToDAG.cpp
+++ b/llvm/lib/Target/M68k/M68kISelDAGToDAG.cpp
@@ -227,6 +227,9 @@ class M68kDAGToDAGISel : public SelectionDAGISel {
   bool SelectPCD(SDNode *Parent, SDValue N, SDValue &Imm);
   bool SelectPCI(SDNode *Parent, SDValue N, SDValue &Imm, SDValue &Index);
 
+  bool SelectInlineAsmMemoryOperand(const SDValue &Op, unsigned ConstraintID,
+                                    std::vector<SDValue> &OutOps) override;
+
   // If Address Mode represents Frame Index store FI in Disp and
   // Displacement bit size in Base. These values are read symmetrically by
   // M68kRegisterInfo::eliminateFrameIndex method
@@ -931,3 +934,74 @@ bool M68kDAGToDAGISel::SelectARI(SDNode *Parent, SDValue N, SDValue &Base) {
 
   return false;
 }
+
+bool M68kDAGToDAGISel::SelectInlineAsmMemoryOperand(
+    const SDValue &Op, unsigned ConstraintID, std::vector<SDValue> &OutOps) {
+  // In order to tell AsmPrinter the exact addressing mode we select here, which
+  // might comprise of multiple SDValues (hence MachineOperands), a 32-bit
+  // immediate value is prepended to the list of selected SDValues to indicate
+  // the addressing mode kind.
+  using AMK = M68k::MemAddrModeKind;
+  auto addKind = [this](SDValue &Opnd, AMK Kind) -> bool {
+    Opnd = CurDAG->getTargetConstant(unsigned(Kind), SDLoc(), MVT::i32);
+    return true;
+  };
+
+  switch (ConstraintID) {
+  // Generic memory operand.
+  case InlineAsm::Constraint_m: {
+    // Try every supported (memory) addressing modes.
+    SDValue Operands[4];
+
+    // TODO: The ordering of the following SelectXXX is relatively...arbitrary,
+    // right now we simply sort them by descending complexity. Maybe we should
+    // adjust this by code model and/or relocation mode in the future.
+    if (SelectARII(nullptr, Op, Operands[1], Operands[2], Operands[3]) &&
+        addKind(Operands[0], AMK::f)) {
+      OutOps.insert(OutOps.end(), &Operands[0], Operands + 4);
+      return false;
+    }
+
+    if ((SelectPCI(nullptr, Op, Operands[1], Operands[2]) &&
+         addKind(Operands[0], AMK::k)) ||
+        (SelectARID(nullptr, Op, Operands[1], Operands[2]) &&
+         addKind(Operands[0], AMK::p))) {
+      OutOps.insert(OutOps.end(), &Operands[0], Operands + 3);
+      return false;
+    }
+
+    if ((SelectPCD(nullptr, Op, Operands[1]) && addKind(Operands[0], AMK::q)) ||
+        (SelectARI(nullptr, Op, Operands[1]) && addKind(Operands[0], AMK::j)) ||
+        (SelectAL(nullptr, Op, Operands[1]) && addKind(Operands[0], AMK::b))) {
+      OutOps.insert(OutOps.end(), {Operands[0], Operands[1]});
+      return false;
+    }
+
+    return true;
+  }
+  // 'Q': Address register indirect addressing.
+  case InlineAsm::Constraint_Q: {
+    SDValue AMKind, Base;
+    // 'j' addressing mode.
+    // TODO: Add support for 'o' and 'e' after their
+    // select functions are implemented.
+    if (SelectARI(nullptr, Op, Base) && addKind(AMKind, AMK::j)) {
+      OutOps.insert(OutOps.end(), {AMKind, Base});
+      return false;
+    }
+    return true;
+  }
+  // 'U': Address register indirect w/ constant offset addressing.
+  case InlineAsm::Constraint_Um: {
+    SDValue AMKind, Base, Offset;
+    // 'p' addressing mode.
+    if (SelectARID(nullptr, Op, Offset, Base) && addKind(AMKind, AMK::p)) {
+      OutOps.insert(OutOps.end(), {AMKind, Offset, Base});
+      return false;
+    }
+    return true;
+  }
+  default:
+    return true;
+  }
+}
diff --git a/llvm/lib/Target/M68k/M68kISelLowering.cpp b/llvm/lib/Target/M68k/M68kISelLowering.cpp
index c9bbb23f31df8..8fce8bcba548e 100644
--- a/llvm/lib/Target/M68k/M68kISelLowering.cpp
+++ b/llvm/lib/Target/M68k/M68kISelLowering.cpp
@@ -201,6 +201,14 @@ M68kTargetLowering::getExceptionSelectorRegister(const Constant *) const {
   return M68k::D1;
 }
 
+unsigned
+M68kTargetLowering::getInlineAsmMemConstraint(StringRef ConstraintCode) const {
+  return StringSwitch<unsigned>(ConstraintCode)
+      .Case("Q", InlineAsm::Constraint_Q)
+      .Case("U", InlineAsm::Constraint_Um) // We borrow Constraint_Um for 'U'.
+      .Default(TargetLowering::getInlineAsmMemConstraint(ConstraintCode));
+}
+
 EVT M68kTargetLowering::getSetCCResultType(const DataLayout &DL,
                                            LLVMContext &Context, EVT VT) const {
   // M68k SETcc producess either 0x00 or 0xFF
@@ -2764,6 +2772,9 @@ M68kTargetLowering::getConstraintType(StringRef Constraint) const {
           break;
         }
       break;
+    case 'Q':
+    case 'U':
+      return C_Memory;
     default:
       break;
     }
diff --git a/llvm/lib/Target/M68k/M68kISelLowering.h b/llvm/lib/Target/M68k/M68kISelLowering.h
index e6242f76c56fb..f998f60f4c9d9 100644
--- a/llvm/lib/Target/M68k/M68kISelLowering.h
+++ b/llvm/lib/Target/M68k/M68kISelLowering.h
@@ -187,6 +187,8 @@ class M68kTargetLowering : public TargetLowering {
   Register
   getExceptionSelectorRegister(const Constant *PersonalityFn) const override;
 
+  unsigned getInlineAsmMemConstraint(StringRef ConstraintCode) const override;
+
 private:
   unsigned GetAlignedArgumentStackSize(unsigned StackSize,
                                        SelectionDAG &DAG) const;
diff --git a/llvm/lib/Target/M68k/MCTargetDesc/M68kBaseInfo.h b/llvm/lib/Target/M68k/MCTargetDesc/M68kBaseInfo.h
index bd2964ab84b19..3703d86519de6 100644
--- a/llvm/lib/Target/M68k/MCTargetDesc/M68kBaseInfo.h
+++ b/llvm/lib/Target/M68k/MCTargetDesc/M68kBaseInfo.h
@@ -48,6 +48,32 @@ enum { MemDisp = 0, MemBase = 1, MemIndex = 2, MemOuter = 3 };
 /// ([bd,PC,Xn],od)
 enum { PCRelDisp = 0, PCRelIndex = 1, PCRelOuter = 2 };
 
+enum class MemAddrModeKind : unsigned {
+  j = 1, // (An)
+  o,     // (An)+
+  e,     // -(An)
+  p,     // (d,An)
+  f,     // (d,An,Xn.L)
+  F,     // (d,An,Xn.W)
+  g,     // (d,An,Xn.L,SCALE)
+  G,     // (d,An,Xn.W,SCALE)
+  u,     // ([bd,An],Xn.L,SCALE,od)
+  U,     // ([bd,An],Xn.W,SCALE,od)
+  v,     // ([bd,An,Xn.L,SCALE],od)
+  V,     // ([bd,An,Xn.W,SCALE],od)
+  b,     // abs.L
+  B,     // abs.W
+  q,     // (d,PC)
+  k,     // (d,PC,Xn.L)
+  K,     // (d,PC,Xn.W)
+  l,     // (d,PC,Xn.L,SCALE)
+  L,     // (d,PC,Xn.W,SCALE)
+  x,     // ([bd,PC],Xn.L,SCALE,od)
+  X,     // ([bd,PC],Xn.W,SCALE,od)
+  y,     // ([bd,PC,Xn.L,SCALE],od)
+  Y      // ([bd,PC,Xn.W,SCALE],od)
+};
+
 // On a LE host:
 // MSB                   LSB    MSB                   LSB
 // | 0x12 0x34 | 0xAB 0xCD | -> | 0xAB 0xCD | 0x12 0x34 |
diff --git a/llvm/lib/Target/M68k/MCTargetDesc/M68kInstPrinter.cpp b/llvm/lib/Target/M68k/MCTargetDesc/M68kInstPrinter.cpp
index 97a5af45de022..84800fc762cbb 100644
--- a/llvm/lib/Target/M68k/MCTargetDesc/M68kInstPrinter.cpp
+++ b/llvm/lib/Target/M68k/MCTargetDesc/M68kInstPrinter.cpp
@@ -147,47 +147,6 @@ void M68kInstPrinter::printDisp(const MCInst *MI, unsigned opNum,
   Op.getExpr()->print(O, &MAI);
 }
 
-void M68kInstPrinter::printARIMem(const MCInst *MI, unsigned opNum,
-                                  raw_ostream &O) {
-  O << '(';
-  printOperand(MI, opNum, O);
-  O << ')';
-}
-
-void M68kInstPrinter::printARIPIMem(const MCInst *MI, unsigned opNum,
-                                    raw_ostream &O) {
-  O << "(";
-  printOperand(MI, opNum, O);
-  O << ")+";
-}
-
-void M68kInstPrinter::printARIPDMem(const MCInst *MI, unsigned opNum,
-                                    raw_ostream &O) {
-  O << "-(";
-  printOperand(MI, opNum, O);
-  O << ")";
-}
-
-void M68kInstPrinter::printARIDMem(const MCInst *MI, unsigned opNum,
-                                   raw_ostream &O) {
-  O << '(';
-  printDisp(MI, opNum + M68k::MemDisp, O);
-  O << ',';
-  printOperand(MI, opNum + M68k::MemBase, O);
-  O << ')';
-}
-
-void M68kInstPrinter::printARIIMem(const MCInst *MI, unsigned opNum,
-                                   raw_ostream &O) {
-  O << '(';
-  printDisp(MI, opNum + M68k::MemDisp, O);
-  O << ',';
-  printOperand(MI, opNum + M68k::MemBase, O);
-  O << ',';
-  printOperand(MI, opNum + M68k::MemIndex, O);
-  O << ')';
-}
-
 // NOTE forcing (W,L) size available since M68020 only
 void M68kInstPrinter::printAbsMem(const MCInst *MI, unsigned opNum,
                                   raw_ostream &O) {
@@ -201,19 +160,3 @@ void M68kInstPrinter::printAbsMem(const MCInst *MI, unsigned opNum,
   assert(MO.isImm() && "absolute memory addressing needs an immediate");
   O << format("$%0" PRIx64, (uint64_t)MO.getImm());
 }
-
-void M68kInstPrinter::printPCDMem(const MCInst *MI, uint64_t Address,
-                                  unsigned opNum, raw_ostream &O) {
-  O << '(';
-  printDisp(MI, opNum + M68k::PCRelDisp, O);
-  O << ",%pc)";
-}
-
-void M68kInstPrinter::printPCIMem(const MCInst *MI, uint64_t Address,
-                                  unsigned opNum, raw_ostream &O) {
-  O << '(';
-  printDisp(MI, opNum + M68k::PCRelDisp, O);
-  O << ",%pc,";
-  printOperand(MI, opNum + M68k::PCRelIndex, O);
-  O << ')';
-}
diff --git a/llvm/lib/Target/M68k/MCTargetDesc/M68kInstPrinter.h b/llvm/lib/Target/M68k/MCTargetDesc/M68kInstPrinter.h
index 5e104856adb16..0963176304587 100644
--- a/llvm/lib/Target/M68k/MCTargetDesc/M68kInstPrinter.h
+++ b/llvm/lib/Target/M68k/MCTargetDesc/M68kInstPrinter.h
@@ -14,13 +14,17 @@
 #ifndef LLVM_LIB_TARGET_M68K_INSTPRINTER_M68KINSTPRINTER_H
 #define LLVM_LIB_TARGET_M68K_INSTPRINTER_M68KINSTPRINTER_H
 
+#include "M68kMemOperandPrinter.h"
 #include "llvm/MC/MCInstPrinter.h"
 
 namespace llvm {
 
 class TargetMachine;
 
-class M68kInstPrinter : public MCInstPrinter {
+class M68kInstPrinter : public MCInstPrinter,
+                        public M68kMemOperandPrinter<M68kInstPrinter, MCInst> {
+  friend class M68kMemOperandPrinter;
+
 public:
   M68kInstPrinter(const MCAsmInfo &MAI, const MCInstrInfo &MII,
                   const MCRegisterInfo &MRI)
@@ -48,16 +52,7 @@ class M68kInstPrinter : public MCInstPrinter {
   /// Print register mask for MOVEM instruction in order A7-A0,D7-D0
   void printMoveMaskR(const MCInst *MI, unsigned opNum, raw_ostream &O);
   void printDisp(const MCInst *MI, unsigned opNum, raw_ostream &O);
-  void printARIMem(const MCInst *MI, unsigned opNum, raw_ostream &O);
-  void printARIPIMem(const MCInst *MI, unsigned opNum, raw_ostream &O);
-  void printARIPDMem(const MCInst *MI, unsigned opNum, raw_ostream &O);
-  void printARIDMem(const MCInst *MI, unsigned opNum, raw_ostream &O);
-  void printARIIMem(const MCInst *MI, unsigned opNum, raw_ostream &O);
   void printAbsMem(const MCInst *MI, unsigned opNum, raw_ostream &O);
-  void printPCDMem(const MCInst *MI, uint64_t Address, unsigned opNum,
-                   raw_ostream &O);
-  void printPCIMem(const MCInst *MI, uint64_t Address, unsigned opNum,
-                   raw_ostream &O);
 
   //===----------------------------------------------------------------------===//
   // Specializations
diff --git a/llvm/lib/Target/M68k/MCTargetDesc/M68kMemOperandPrinter.h b/llvm/lib/Target/M68k/MCTargetDesc/M68kMemOperandPrinter.h
new file mode 100644
index 0000000000000..cc5cc7a37e857
--- /dev/null
+++ b/llvm/lib/Target/M68k/MCTargetDesc/M68kMemOperandPrinter.h
@@ -0,0 +1,80 @@
+//===-- M68kMemOperandPrinter.h - Memory operands printing ------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// This file contains memory operand printing logics shared between AsmPrinter
+//  and MCInstPrinter.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIB_TARGET_M68K_MEMOPERANDPRINTER_M68KINSTPRINTER_H
+#define LLVM_LIB_TARGET_M68K_MEMOPERANDPRINTER_M68KINSTPRINTER_H
+
+#include "M68kBaseInfo.h"
+
+#include "llvm/Support/raw_ostream.h"
+
+namespace llvm {
+template <class Derived, typename InstTy> class M68kMemOperandPrinter {
+  Derived &impl() { return *static_cast<Derived *>(this); }
+
+protected:
+  void printARIMem(const InstTy *MI, unsigned OpNum, raw_ostream &O) {
+    O << '(';
+    impl().printOperand(MI, OpNum, O);
+    O << ')';
+  }
+
+  void printARIPIMem(const InstTy *MI, unsigned OpNum, raw_ostream &O) {
+    O << "(";
+    impl().printOperand(MI, OpNum, O);
+    O << ")+";
+  }
+
+  void printARIPDMem(const InstTy *MI, unsigned OpNum, raw_ostream &O) {
+    O << "-(";
+    impl().printOperand(MI, OpNum, O);
+    O << ")";
+  }
+
+  void printARIDMem(const InstTy *MI, unsigned OpNum, raw_ostream &O) {
+    O << '(';
+    impl().printDisp(MI, OpNum + M68k::MemDisp, O);
+    O << ',';
+    impl().printOperand(MI, OpNum + M68k::MemBase, O);
+    O << ')';
+  }
+
+  void printARIIMem(const InstTy *MI, unsigned OpNum, raw_ostream &O) {
+    O << '(';
+    impl().printDisp(MI, OpNum + M68k::MemDisp, O);
+    O << ',';
+    impl().printOperand(MI, OpNum + M68k::MemBase, O);
+    O << ',';
+    impl().printOperand(MI, OpNum + M68k::MemIndex, O);
+    O << ')';
+  }
+
+  void printPCDMem(const InstTy *MI, uint64_t Address, unsigned OpNum,
+                   raw_ostream &O) {
+    O << '(';
+    impl().printDisp(MI, OpNum + M68k::PCRelDisp, O);
+    O << ",%pc)";
+  }
+
+  void printPCIMem(const InstTy *MI, uint64_t Address, unsigned OpNum,
+                   raw_ostream &O) {
+    O << '(';
+    impl().printDisp(MI, OpNum + M68k::PCRelDisp, O);
+    O << ",%pc,";
+    impl().printOperand(MI, OpNum + M68k::PCRelIndex, O);
+    O << ')';
+  }
+};
+} // end namespace llvm
+#endif
diff --git a/llvm/lib/Target/MSP430/MSP430ISelLowering.cpp b/llvm/lib/Target/MSP430/MSP430ISelLowering.cpp
index 27fafa16288ec..c0b4a492d3da2 100644
--- a/llvm/lib/Target/MSP430/MSP430ISelLowering.cpp
+++ b/llvm/lib/Target/MSP430/MSP430ISelLowering.cpp
@@ -1412,10 +1412,6 @@ bool MSP430TargetLowering::isZExtFree(EVT VT1, EVT VT2) const {
   return false && VT1 == MVT::i8 && VT2 == MVT::i16;
 }
 
-bool MSP430TargetLowering::isZExtFree(SDValue Val, EVT VT2) const {
-  return isZExtFree(Val.getValueType(), VT2);
-}
-
 //===----------------------------------------------------------------------===//
 //  Other Lowering Code
 //===----------------------------------------------------------------------===//
diff --git a/llvm/lib/Target/MSP430/MSP430ISelLowering.h b/llvm/lib/Target/MSP430/MSP430ISelLowering.h
index f23042a369fd6..17ee457b849ac 100644
--- a/llvm/lib/Target/MSP430/MSP430ISelLowering.h
+++ b/llvm/lib/Target/MSP430/MSP430ISelLowering.h
@@ -126,7 +126,6 @@ namespace llvm {
     /// out to 16 bits.
     bool isZExtFree(Type *Ty1, Type *Ty2) const override;
     bool isZExtFree(EVT VT1, EVT VT2) const override;
-    bool isZExtFree(SDValue Val, EVT VT2) const override;
 
     bool isLegalICmpImmediate(int64_t) const override;
     bool shouldAvoidTransformToShift(EVT VT, unsigned Amount) const override;
diff --git a/llvm/lib/Target/NVPTX/NVPTXAsmPrinter.cpp b/llvm/lib/Target/NVPTX/NVPTXAsmPrinter.cpp
index a4d77994dec5c..b2db77a531107 100644
--- a/llvm/lib/Target/NVPTX/NVPTXAsmPrinter.cpp
+++ b/llvm/lib/Target/NVPTX/NVPTXAsmPrinter.cpp
@@ -466,7 +466,7 @@ void NVPTXAsmPrinter::emitFunctionEntryLabel() {
 
   CurrentFnSym->print(O, MAI);
 
-  emitFunctionParamList(*MF, O);
+  emitFunctionParamList(F, O);
 
   if (isKernelFunction(*F))
     emitKernelFunctionDirectives(*F, O);
@@ -700,7 +700,7 @@ static bool useFuncSeen(const Constant *C,
       const Function *caller = bb->getParent();
       if (!caller)
         continue;
-      if (seenMap.find(caller) != seenMap.end())
+      if (seenMap.contains(caller))
         return true;
     }
   }
@@ -753,7 +753,7 @@ void NVPTXAsmPrinter::emitDeclarations(const Module &M, raw_ostream &O) {
       // If a caller has already been seen, then the caller is
       // appearing in the module before the callee. so print out
       // a declaration for the callee.
-      if (seenMap.find(caller) != seenMap.end()) {
+      if (seenMap.contains(caller)) {
         emitDeclaration(&F, O);
         break;
       }
@@ -1441,12 +1441,6 @@ void NVPTXAsmPrinter::emitPTXGlobalVariable(const GlobalVariable *GVar,
   }
 }
 
-void NVPTXAsmPrinter::printParamName(Function::const_arg_iterator I,
-                                     int paramIndex, raw_ostream &O) {
-  getSymbol(I->getParent())->print(O, MAI);
-  O << "_param_" << paramIndex;
-}
-
 void NVPTXAsmPrinter::emitFunctionParamList(const Function *F, raw_ostream &O) {
   const DataLayout &DL = getDataLayout();
   const AttributeList &PAL = F->getAttributes();
@@ -1485,24 +1479,21 @@ void NVPTXAsmPrinter::emitFunctionParamList(const Function *F, raw_ostream &O) {
               O << "\t.param .u64 .ptr .surfref ";
             else
               O << "\t.param .surfref ";
-            CurrentFnSym->print(O, MAI);
-            O << "_param_" << paramIndex;
+            O << TLI->getParamName(F, paramIndex);
           }
           else { // Default image is read_only
             if (hasImageHandles)
               O << "\t.param .u64 .ptr .texref ";
             else
               O << "\t.param .texref ";
-            CurrentFnSym->print(O, MAI);
-            O << "_param_" << paramIndex;
+            O << TLI->getParamName(F, paramIndex);
           }
         } else {
           if (hasImageHandles)
             O << "\t.param .u64 .ptr .samplerref ";
           else
             O << "\t.param .samplerref ";
-          CurrentFnSym->print(O, MAI);
-          O << "_param_" << paramIndex;
+          O << TLI->getParamName(F, paramIndex);
         }
         continue;
       }
@@ -1524,7 +1515,7 @@ void NVPTXAsmPrinter::emitFunctionParamList(const Function *F, raw_ostream &O) {
         Align OptimalAlign = getOptimalAlignForParam(Ty);
 
         O << "\t.param .align " << OptimalAlign.value() << " .b8 ";
-        printParamName(I, paramIndex, O);
+        O << TLI->getParamName(F, paramIndex);
         O << "[" << DL.getTypeAllocSize(Ty) << "]";
 
         continue;
@@ -1563,7 +1554,7 @@ void NVPTXAsmPrinter::emitFunctionParamList(const Function *F, raw_ostream &O) {
             Align ParamAlign = I->getParamAlign().valueOrOne();
             O << ".align " << ParamAlign.value() << " ";
           }
-          printParamName(I, paramIndex, O);
+          O << TLI->getParamName(F, paramIndex);
           continue;
         }
 
@@ -1575,7 +1566,7 @@ void NVPTXAsmPrinter::emitFunctionParamList(const Function *F, raw_ostream &O) {
         else
           O << getPTXFundamentalTypeStr(Ty);
         O << " ";
-        printParamName(I, paramIndex, O);
+        O << TLI->getParamName(F, paramIndex);
         continue;
       }
       // Non-kernel function, just print .param .b<size> for ABI
@@ -1598,7 +1589,7 @@ void NVPTXAsmPrinter::emitFunctionParamList(const Function *F, raw_ostream &O) {
         O << "\t.param .b" << sz << " ";
       else
         O << "\t.reg .b" << sz << " ";
-      printParamName(I, paramIndex, O);
+      O << TLI->getParamName(F, paramIndex);
       continue;
     }
 
@@ -1619,7 +1610,7 @@ void NVPTXAsmPrinter::emitFunctionParamList(const Function *F, raw_ostream &O) {
 
       unsigned sz = DL.getTypeAllocSize(ETy);
       O << "\t.param .align " << OptimalAlign.value() << " .b8 ";
-      printParamName(I, paramIndex, O);
+      O << TLI->getParamName(F, paramIndex);
       O << "[" << sz << "]";
       continue;
     } else {
@@ -1642,7 +1633,7 @@ void NVPTXAsmPrinter::emitFunctionParamList(const Function *F, raw_ostream &O) {
           if (elemtype.isInteger())
             sz = promoteScalarArgumentSize(sz);
           O << "\t.reg .b" << sz << " ";
-          printParamName(I, paramIndex, O);
+          O << TLI->getParamName(F, paramIndex);
           if (j < je - 1)
             O << ",\n";
           ++paramIndex;
@@ -1660,19 +1651,12 @@ void NVPTXAsmPrinter::emitFunctionParamList(const Function *F, raw_ostream &O) {
       O << ",\n";
     O << "\t.param .align " << STI.getMaxRequiredAlignment();
     O << " .b8 ";
-    getSymbol(F)->print(O, MAI);
-    O << "_vararg[]";
+    O << TLI->getParamName(F, /* vararg */ -1) << "[]";
   }
 
   O << "\n)\n";
 }
 
-void NVPTXAsmPrinter::emitFunctionParamList(const MachineFunction &MF,
-                                            raw_ostream &O) {
-  const Function &F = MF.getFunction();
-  emitFunctionParamList(&F, O);
-}
-
 void NVPTXAsmPrinter::setAndEmitFunctionVirtualRegisters(
     const MachineFunction &MF) {
   SmallString<128> Str;
diff --git a/llvm/lib/Target/NVPTX/NVPTXAsmPrinter.h b/llvm/lib/Target/NVPTX/NVPTXAsmPrinter.h
index 710c089e33251..673aad1408891 100644
--- a/llvm/lib/Target/NVPTX/NVPTXAsmPrinter.h
+++ b/llvm/lib/Target/NVPTX/NVPTXAsmPrinter.h
@@ -173,14 +173,11 @@ class LLVM_LIBRARY_VISIBILITY NVPTXAsmPrinter : public AsmPrinter {
                        const char *Modifier = nullptr);
   void printModuleLevelGV(const GlobalVariable *GVar, raw_ostream &O,
                           bool processDemoted, const NVPTXSubtarget &STI);
-  void printParamName(Function::const_arg_iterator I, int paramIndex,
-                      raw_ostream &O);
   void emitGlobals(const Module &M);
   void emitHeader(Module &M, raw_ostream &O, const NVPTXSubtarget &STI);
   void emitKernelFunctionDirectives(const Function &F, raw_ostream &O) const;
   void emitVirtualRegister(unsigned int vr, raw_ostream &);
   void emitFunctionParamList(const Function *, raw_ostream &O);
-  void emitFunctionParamList(const MachineFunction &MF, raw_ostream &O);
   void setAndEmitFunctionVirtualRegisters(const MachineFunction &MF);
   void printReturnValStr(const Function *, raw_ostream &O);
   void printReturnValStr(const MachineFunction &MF, raw_ostream &O);
diff --git a/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp b/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp
index 008a04aa2f638..93f2533bf1d8b 100644
--- a/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp
+++ b/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp
@@ -2614,18 +2614,8 @@ SDValue NVPTXTargetLowering::LowerSTOREi1(SDValue Op, SelectionDAG &DAG) const {
 // passing variable arguments.
 SDValue NVPTXTargetLowering::getParamSymbol(SelectionDAG &DAG, int idx,
                                             EVT v) const {
-  std::string ParamSym;
-  raw_string_ostream ParamStr(ParamSym);
-
-  ParamStr << DAG.getMachineFunction().getName();
-
-  if (idx < 0)
-    ParamStr << "_vararg";
-  else
-    ParamStr << "_param_" << idx;
-
-  StringRef SavedStr =
-    nvTM->getStrPool().save(ParamSym);
+  StringRef SavedStr = nvTM->getStrPool().save(
+      getParamName(&DAG.getMachineFunction().getFunction(), idx));
   return DAG.getTargetExternalSymbol(SavedStr.data(), v);
 }
 
@@ -2676,7 +2666,9 @@ SDValue NVPTXTargetLowering::LowerFormalArguments(
         SmallVector<EVT, 16> vtparts;
 
         ComputePTXValueVTs(*this, DAG.getDataLayout(), Ty, vtparts);
-        assert(vtparts.size() > 0 && "empty aggregate type not expected");
+        if (vtparts.empty())
+          report_fatal_error("Empty parameter types are not supported");
+
         for (unsigned parti = 0, parte = vtparts.size(); parti != parte;
              ++parti) {
           InVals.push_back(DAG.getNode(ISD::UNDEF, dl, Ins[InsIdx].VT));
@@ -2713,7 +2705,9 @@ SDValue NVPTXTargetLowering::LowerFormalArguments(
       SmallVector<EVT, 16> VTs;
       SmallVector<uint64_t, 16> Offsets;
       ComputePTXValueVTs(*this, DL, Ty, VTs, &Offsets, 0);
-      assert(VTs.size() > 0 && "Unexpected empty type.");
+      if (VTs.empty())
+        report_fatal_error("Empty parameter types are not supported");
+
       auto VectorInfo =
           VectorizePTXValueVTs(VTs, Offsets, DL.getABITypeAlign(Ty));
 
@@ -4522,6 +4516,23 @@ Align NVPTXTargetLowering::getFunctionByValParamAlign(
   return ArgAlign;
 }
 
+// Helper for getting a function parameter name. Name is composed from
+// its index and the function name. Negative index corresponds to special
+// parameter (unsized array) used for passing variable arguments.
+std::string NVPTXTargetLowering::getParamName(const Function *F,
+                                              int Idx) const {
+  std::string ParamName;
+  raw_string_ostream ParamStr(ParamName);
+
+  ParamStr << getTargetMachine().getSymbol(F)->getName();
+  if (Idx < 0)
+    ParamStr << "_vararg";
+  else
+    ParamStr << "_param_" << Idx;
+
+  return ParamName;
+}
+
 /// isLegalAddressingMode - Return true if the addressing mode represented
 /// by AM is legal for this target, for a load/store of the specified type.
 /// Used to guide target specific optimizations, like loop strength reduction
diff --git a/llvm/lib/Target/NVPTX/NVPTXISelLowering.h b/llvm/lib/Target/NVPTX/NVPTXISelLowering.h
index f48ec1740b0f8..27cbae6027e05 100644
--- a/llvm/lib/Target/NVPTX/NVPTXISelLowering.h
+++ b/llvm/lib/Target/NVPTX/NVPTXISelLowering.h
@@ -466,6 +466,11 @@ class NVPTXTargetLowering : public TargetLowering {
                                    Align InitialAlign,
                                    const DataLayout &DL) const;
 
+  // Helper for getting a function parameter name. Name is composed from
+  // its index and the function name. Negative index corresponds to special
+  // parameter (unsized array) used for passing variable arguments.
+  std::string getParamName(const Function *F, int Idx) const;
+
   /// isLegalAddressingMode - Return true if the addressing mode represented
   /// by AM is legal for this target, for a load/store of the specified type
   /// Used to guide target specific optimizations, like loop strength
diff --git a/llvm/lib/Target/PowerPC/MCTargetDesc/PPCELFStreamer.cpp b/llvm/lib/Target/PowerPC/MCTargetDesc/PPCELFStreamer.cpp
index dfc0409434de3..2f03aa37745f8 100644
--- a/llvm/lib/Target/PowerPC/MCTargetDesc/PPCELFStreamer.cpp
+++ b/llvm/lib/Target/PowerPC/MCTargetDesc/PPCELFStreamer.cpp
@@ -184,7 +184,7 @@ void PPCELFStreamer::emitGOTToPCRelLabel(const MCInst &Inst) {
   emitLabel(LabelSym, Inst.getLoc());
 }
 
-// This funciton checks if the parameter Inst is part of the setup for a link
+// This function checks if the parameter Inst is part of the setup for a link
 // time GOT PC Relative optimization. For example in this situation:
 // <MCInst PLDpc <MCOperand Reg:282> <MCOperand Expr:(glob_double@got@pcrel)>
 //   <MCOperand Imm:0> <MCOperand Expr:(.Lpcrel@<<invalid>>)>>
diff --git a/llvm/lib/Target/PowerPC/PPCAsmPrinter.cpp b/llvm/lib/Target/PowerPC/PPCAsmPrinter.cpp
index 9f86f1479d600..1ecaeabacf9f7 100644
--- a/llvm/lib/Target/PowerPC/PPCAsmPrinter.cpp
+++ b/llvm/lib/Target/PowerPC/PPCAsmPrinter.cpp
@@ -2096,8 +2096,9 @@ void PPCAIXAsmPrinter::emitTracebackTable() {
   // Set the 5th byte of mandatory field.
   uint32_t SecondHalfOfMandatoryField = 0;
 
-  // Always store back chain.
-  SecondHalfOfMandatoryField |= TracebackTable::IsBackChainStoredMask;
+  SecondHalfOfMandatoryField |= MF->getFrameInfo().getStackSize()
+                                    ? TracebackTable::IsBackChainStoredMask
+                                    : 0;
 
   uint32_t FPRSaved = 0;
   for (unsigned Reg = PPC::F14; Reg <= PPC::F31; ++Reg) {
diff --git a/llvm/lib/Target/PowerPC/PPCISelDAGToDAG.cpp b/llvm/lib/Target/PowerPC/PPCISelDAGToDAG.cpp
index 2c450a332ef4a..e5220aec4f388 100644
--- a/llvm/lib/Target/PowerPC/PPCISelDAGToDAG.cpp
+++ b/llvm/lib/Target/PowerPC/PPCISelDAGToDAG.cpp
@@ -2846,9 +2846,6 @@ class IntegerCompareEliminator {
   }
 };
 
-static bool isLogicOp(unsigned Opc) {
-  return Opc == ISD::AND || Opc == ISD::OR || Opc == ISD::XOR;
-}
 // The obvious case for wanting to keep the value in a GPR. Namely, the
 // result of the comparison is actually needed in a GPR.
 SDNode *IntegerCompareEliminator::tryEXTEND(SDNode *N) {
@@ -2858,7 +2855,7 @@ SDNode *IntegerCompareEliminator::tryEXTEND(SDNode *N) {
   SDValue WideRes;
   // If we are zero-extending the result of a logical operation on i1
   // values, we can keep the values in GPRs.
-  if (isLogicOp(N->getOperand(0).getOpcode()) &&
+  if (ISD::isBitwiseLogicOp(N->getOperand(0).getOpcode()) &&
       N->getOperand(0).getValueType() == MVT::i1 &&
       N->getOpcode() == ISD::ZERO_EXTEND)
     WideRes = computeLogicOpInGPR(N->getOperand(0));
@@ -2894,7 +2891,7 @@ SDNode *IntegerCompareEliminator::tryEXTEND(SDNode *N) {
 SDNode *IntegerCompareEliminator::tryLogicOpOfCompares(SDNode *N) {
   if (N->getValueType(0) != MVT::i1)
     return nullptr;
-  assert(isLogicOp(N->getOpcode()) &&
+  assert(ISD::isBitwiseLogicOp(N->getOpcode()) &&
          "Expected a logic operation on setcc results.");
   SDValue LoweredLogical = computeLogicOpInGPR(SDValue(N, 0));
   if (!LoweredLogical)
@@ -2974,7 +2971,7 @@ SDNode *IntegerCompareEliminator::tryLogicOpOfCompares(SDNode *N) {
 // There is also a special case that is handled (namely a complement operation
 // achieved with xor %a, -1).
 SDValue IntegerCompareEliminator::computeLogicOpInGPR(SDValue LogicOp) {
-  assert(isLogicOp(LogicOp.getOpcode()) &&
+  assert(ISD::isBitwiseLogicOp(LogicOp.getOpcode()) &&
         "Can only handle logic operations here.");
   assert(LogicOp.getValueType() == MVT::i1 &&
          "Can only handle logic operations on i1 values here.");
@@ -2999,7 +2996,7 @@ SDValue IntegerCompareEliminator::computeLogicOpInGPR(SDValue LogicOp) {
                                             PPC::RLDICL, dl, InVT, InputOp,
                                             S->getI64Imm(0, dl),
                                             S->getI64Imm(63, dl)), 0);
-    } else if (isLogicOp(OperandOpcode))
+    } else if (ISD::isBitwiseLogicOp(OperandOpcode))
       return computeLogicOpInGPR(Operand);
     return SDValue();
   };
@@ -3888,7 +3885,7 @@ static bool allUsesExtend(SDValue Compare, SelectionDAG *CurDAG) {
     if (CompareUse->getOpcode() != ISD::SIGN_EXTEND &&
         CompareUse->getOpcode() != ISD::ZERO_EXTEND &&
         CompareUse->getOpcode() != ISD::SELECT &&
-        !isLogicOp(CompareUse->getOpcode())) {
+        !ISD::isBitwiseLogicOp(CompareUse->getOpcode())) {
       OmittedForNonExtendUses++;
       return false;
     }
diff --git a/llvm/lib/Target/PowerPC/PPCISelLowering.cpp b/llvm/lib/Target/PowerPC/PPCISelLowering.cpp
index 03cac2fd6ba1f..03a387570e3c6 100644
--- a/llvm/lib/Target/PowerPC/PPCISelLowering.cpp
+++ b/llvm/lib/Target/PowerPC/PPCISelLowering.cpp
@@ -1390,10 +1390,6 @@ PPCTargetLowering::PPCTargetLowering(const PPCTargetMachine &TM,
     setTargetDAGCombine({ISD::TRUNCATE, ISD::SETCC, ISD::SELECT_CC});
   }
 
-  if (Subtarget.hasP9Altivec()) {
-    setTargetDAGCombine({ISD::VSELECT});
-  }
-
   setLibcallName(RTLIB::LOG_F128, "logf128");
   setLibcallName(RTLIB::LOG2_F128, "log2f128");
   setLibcallName(RTLIB::LOG10_F128, "log10f128");
@@ -9315,7 +9311,7 @@ SDValue PPCTargetLowering::LowerBUILD_VECTOR(SDValue Op,
       // Exclude somes case where LD_SPLAT is worse than scalar_to_vector:
       // Below cases should also happen for "lfiwzx/lfiwax + LE target + index
       // 1" and "lxvrhx + BE target + index 7" and "lxvrbx + BE target + index
-      // 15", but funciton IsValidSplatLoad() now will only return true when
+      // 15", but function IsValidSplatLoad() now will only return true when
       // the data at index 0 is not nullptr. So we will not get into trouble for
       // these cases.
       //
@@ -15787,16 +15783,37 @@ SDValue PPCTargetLowering::PerformDAGCombine(SDNode *N,
 
     break;
   case ISD::INTRINSIC_W_CHAIN:
-    // For little endian, VSX loads require generating lxvd2x/xxswapd.
-    // Not needed on ISA 3.0 based CPUs since we have a non-permuting load.
-    if (Subtarget.needsSwapsForVSXMemOps()) {
-      switch (cast<ConstantSDNode>(N->getOperand(1))->getZExtValue()) {
-      default:
-        break;
-      case Intrinsic::ppc_vsx_lxvw4x:
-      case Intrinsic::ppc_vsx_lxvd2x:
-        return expandVSXLoadForLE(N, DCI);
+    switch (cast<ConstantSDNode>(N->getOperand(1))->getZExtValue()) {
+    default:
+      break;
+    case Intrinsic::ppc_altivec_vsum4sbs:
+    case Intrinsic::ppc_altivec_vsum4shs:
+    case Intrinsic::ppc_altivec_vsum4ubs: {
+      // These sum-across intrinsics only have a chain due to the side effect
+      // that they may set the SAT bit. If we know the SAT bit will not be set
+      // for some inputs, we can replace any uses of their chain with the input
+      // chain.
+      if (BuildVectorSDNode *BVN =
+              dyn_cast<BuildVectorSDNode>(N->getOperand(3))) {
+        APInt APSplatBits, APSplatUndef;
+        unsigned SplatBitSize;
+        bool HasAnyUndefs;
+        bool BVNIsConstantSplat = BVN->isConstantSplat(
+            APSplatBits, APSplatUndef, SplatBitSize, HasAnyUndefs, 0,
+            !Subtarget.isLittleEndian());
+        // If the constant splat vector is 0, the SAT bit will not be set.
+        if (BVNIsConstantSplat && APSplatBits == 0)
+          DAG.ReplaceAllUsesOfValueWith(SDValue(N, 1), N->getOperand(0));
       }
+      return SDValue();
+    }
+    case Intrinsic::ppc_vsx_lxvw4x:
+    case Intrinsic::ppc_vsx_lxvd2x:
+      // For little endian, VSX loads require generating lxvd2x/xxswapd.
+      // Not needed on ISA 3.0 based CPUs since we have a non-permuting load.
+      if (Subtarget.needsSwapsForVSXMemOps())
+        return expandVSXLoadForLE(N, DCI);
+      break;
     }
     break;
   case ISD::INTRINSIC_VOID:
@@ -16057,8 +16074,6 @@ SDValue PPCTargetLowering::PerformDAGCombine(SDNode *N,
   }
   case ISD::BUILD_VECTOR:
     return DAGCombineBuildVector(N, DCI);
-  case ISD::VSELECT:
-    return combineVSelect(N, DCI);
   }
 
   return SDValue();
@@ -17684,69 +17699,6 @@ isMaskAndCmp0FoldingBeneficial(const Instruction &AndI) const {
   return true;
 }
 
-// For type v4i32/v8ii16/v16i8, transform
-// from (vselect (setcc a, b, setugt), (sub a, b), (sub b, a)) to (abdu a, b)
-// from (vselect (setcc a, b, setuge), (sub a, b), (sub b, a)) to (abdu a, b)
-// from (vselect (setcc a, b, setult), (sub b, a), (sub a, b)) to (abdu a, b)
-// from (vselect (setcc a, b, setule), (sub b, a), (sub a, b)) to (abdu a, b)
-// TODO: Move this to DAGCombiner?
-SDValue PPCTargetLowering::combineVSelect(SDNode *N,
-                                          DAGCombinerInfo &DCI) const {
-  assert((N->getOpcode() == ISD::VSELECT) && "Need VSELECT node here");
-  assert(Subtarget.hasP9Altivec() &&
-         "Only combine this when P9 altivec supported!");
-
-  SelectionDAG &DAG = DCI.DAG;
-  SDLoc dl(N);
-  SDValue Cond = N->getOperand(0);
-  SDValue TrueOpnd = N->getOperand(1);
-  SDValue FalseOpnd = N->getOperand(2);
-  EVT VT = N->getOperand(1).getValueType();
-
-  if (Cond.getOpcode() != ISD::SETCC || TrueOpnd.getOpcode() != ISD::SUB ||
-      FalseOpnd.getOpcode() != ISD::SUB)
-    return SDValue();
-
-  // ABSD only available for type v4i32/v8i16/v16i8
-  if (VT != MVT::v4i32 && VT != MVT::v8i16 && VT != MVT::v16i8)
-    return SDValue();
-
-  // At least to save one more dependent computation
-  if (!(Cond.hasOneUse() || TrueOpnd.hasOneUse() || FalseOpnd.hasOneUse()))
-    return SDValue();
-
-  ISD::CondCode CC = cast<CondCodeSDNode>(Cond.getOperand(2))->get();
-
-  // Can only handle unsigned comparison here
-  switch (CC) {
-  default:
-    return SDValue();
-  case ISD::SETUGT:
-  case ISD::SETUGE:
-    break;
-  case ISD::SETULT:
-  case ISD::SETULE:
-    std::swap(TrueOpnd, FalseOpnd);
-    break;
-  }
-
-  SDValue CmpOpnd1 = Cond.getOperand(0);
-  SDValue CmpOpnd2 = Cond.getOperand(1);
-
-  // SETCC CmpOpnd1 CmpOpnd2 cond
-  // TrueOpnd = CmpOpnd1 - CmpOpnd2
-  // FalseOpnd = CmpOpnd2 - CmpOpnd1
-  if (TrueOpnd.getOperand(0) == CmpOpnd1 &&
-      TrueOpnd.getOperand(1) == CmpOpnd2 &&
-      FalseOpnd.getOperand(0) == CmpOpnd2 &&
-      FalseOpnd.getOperand(1) == CmpOpnd1) {
-    return DAG.getNode(ISD::ABDU, dl, N->getOperand(1).getValueType(), CmpOpnd1,
-                       CmpOpnd2, DAG.getTargetConstant(0, dl, MVT::i32));
-  }
-
-  return SDValue();
-}
-
 /// getAddrModeForFlags - Based on the set of address flags, select the most
 /// optimal instruction format to match by.
 PPC::AddrMode PPCTargetLowering::getAddrModeForFlags(unsigned Flags) const {
diff --git a/llvm/lib/Target/PowerPC/PPCISelLowering.h b/llvm/lib/Target/PowerPC/PPCISelLowering.h
index 21fad5a12a8fe..9706fddb0fc0b 100644
--- a/llvm/lib/Target/PowerPC/PPCISelLowering.h
+++ b/llvm/lib/Target/PowerPC/PPCISelLowering.h
@@ -1416,7 +1416,6 @@ namespace llvm {
     SDValue combineFMALike(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineTRUNCATE(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineSetCC(SDNode *N, DAGCombinerInfo &DCI) const;
-    SDValue combineVSelect(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineVectorShuffle(ShuffleVectorSDNode *SVN,
                                  SelectionDAG &DAG) const;
     SDValue combineVReverseMemOP(ShuffleVectorSDNode *SVN, LSBaseSDNode *LSBase,
diff --git a/llvm/lib/Target/PowerPC/PPCLoopInstrFormPrep.cpp b/llvm/lib/Target/PowerPC/PPCLoopInstrFormPrep.cpp
index 999b0b06baa62..f29a7af1bdf1c 100644
--- a/llvm/lib/Target/PowerPC/PPCLoopInstrFormPrep.cpp
+++ b/llvm/lib/Target/PowerPC/PPCLoopInstrFormPrep.cpp
@@ -910,7 +910,7 @@ bool PPCLoopInstrFormPrep::prepareBaseForDispFormChain(Bucket &BucketChain,
       unsigned Remainder = cast<SCEVConstant>(BucketChain.Elements[j].Offset)
                                ->getAPInt()
                                .urem(Form);
-      if (RemainderOffsetInfo.find(Remainder) == RemainderOffsetInfo.end())
+      if (!RemainderOffsetInfo.contains(Remainder))
         RemainderOffsetInfo[Remainder] = std::make_pair(j, 1);
       else
         RemainderOffsetInfo[Remainder].second++;
@@ -933,7 +933,7 @@ bool PPCLoopInstrFormPrep::prepareBaseForDispFormChain(Bucket &BucketChain,
   // 1 X form.
   unsigned MaxCountRemainder = 0;
   for (unsigned j = 0; j < (unsigned)Form; j++)
-    if ((RemainderOffsetInfo.find(j) != RemainderOffsetInfo.end()) &&
+    if ((RemainderOffsetInfo.contains(j)) &&
         RemainderOffsetInfo[j].second >
             RemainderOffsetInfo[MaxCountRemainder].second)
       MaxCountRemainder = j;
@@ -1179,6 +1179,8 @@ Value *PPCLoopInstrFormPrep::getNodeForInc(Loop *L, Instruction *MemI,
 
     // Get the incoming value from the loop latch and check if the value has
     // the add form with the required increment.
+    if (CurrentPHINode->getBasicBlockIndex(LatchBB) < 0)
+      continue;
     if (Instruction *I = dyn_cast<Instruction>(
             CurrentPHINode->getIncomingValueForBlock(LatchBB))) {
       Value *StrippedBaseI = I;
diff --git a/llvm/lib/Target/PowerPC/PPCVSXSwapRemoval.cpp b/llvm/lib/Target/PowerPC/PPCVSXSwapRemoval.cpp
index 365ba524a757e..ea88021eec3df 100644
--- a/llvm/lib/Target/PowerPC/PPCVSXSwapRemoval.cpp
+++ b/llvm/lib/Target/PowerPC/PPCVSXSwapRemoval.cpp
@@ -618,7 +618,7 @@ void PPCVSXSwapRemoval::formWebs() {
         continue;
 
       MachineInstr* DefMI = MRI->getVRegDef(Reg);
-      assert(SwapMap.find(DefMI) != SwapMap.end() &&
+      assert(SwapMap.contains(DefMI) &&
              "Inconsistency: def of vector reg not found in swap map!");
       int DefIdx = SwapMap[DefMI];
       (void)EC->unionSets(SwapVector[DefIdx].VSEId,
diff --git a/llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp b/llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp
index 61a2749d6ec43..e8edc96951229 100644
--- a/llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp
+++ b/llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp
@@ -158,6 +158,7 @@ class RISCVAsmParser : public MCTargetAsmParser {
 #include "RISCVGenAsmMatcher.inc"
 
   OperandMatchResultTy parseCSRSystemRegister(OperandVector &Operands);
+  OperandMatchResultTy parseFPImm(OperandVector &Operands);
   OperandMatchResultTy parseImmediate(OperandVector &Operands);
   OperandMatchResultTy parseRegister(OperandVector &Operands,
                                      bool AllowParens = false);
@@ -274,6 +275,7 @@ struct RISCVOperand final : public MCParsedAsmOperand {
     Token,
     Register,
     Immediate,
+    FPImmediate,
     SystemRegister,
     VType,
     FRM,
@@ -290,6 +292,10 @@ struct RISCVOperand final : public MCParsedAsmOperand {
     bool IsRV64;
   };
 
+  struct FPImmOp {
+    uint64_t Val;
+  };
+
   struct SysRegOp {
     const char *Data;
     unsigned Length;
@@ -315,6 +321,7 @@ struct RISCVOperand final : public MCParsedAsmOperand {
     StringRef Tok;
     RegOp Reg;
     ImmOp Imm;
+    FPImmOp FPImm;
     struct SysRegOp SysReg;
     struct VTypeOp VType;
     struct FRMOp FRM;
@@ -335,6 +342,9 @@ struct RISCVOperand final : public MCParsedAsmOperand {
     case KindTy::Immediate:
       Imm = o.Imm;
       break;
+    case KindTy::FPImmediate:
+      FPImm = o.FPImm;
+      break;
     case KindTy::Token:
       Tok = o.Tok;
       break;
@@ -480,6 +490,19 @@ struct RISCVOperand final : public MCParsedAsmOperand {
   bool isFRMArg() const { return Kind == KindTy::FRM; }
   bool isRTZArg() const { return isFRMArg() && FRM.FRM == RISCVFPRndMode::RTZ; }
 
+  /// Return true if the operand is a valid fli.s floating-point immediate.
+  bool isLoadFPImm() const {
+    if (isImm())
+      return isUImm5();
+    if (Kind != KindTy::FPImmediate)
+      return false;
+    int Idx = RISCVLoadFPImm::getLoadFPImm(
+        APFloat(APFloat::IEEEdouble(), APInt(64, getFPConst())));
+    // Don't allow decimal version of the minimum value. It is a different value
+    // for each supported data type.
+    return Idx >= 0 && Idx != 1;
+  }
+
   bool isImmXLenLI() const {
     int64_t Imm;
     RISCVMCExpr::VariantKind VK = RISCVMCExpr::VK_RISCV_None;
@@ -538,10 +561,10 @@ struct RISCVOperand final : public MCParsedAsmOperand {
     return IsConstantImm && isUInt<N>(Imm) && VK == RISCVMCExpr::VK_RISCV_None;
   }
 
-  bool isUImm2() { return IsUImm<2>(); }
-  bool isUImm3() { return IsUImm<3>(); }
-  bool isUImm5() { return IsUImm<5>(); }
-  bool isUImm7() { return IsUImm<7>(); }
+  bool isUImm2() const { return IsUImm<2>(); }
+  bool isUImm3() const { return IsUImm<3>(); }
+  bool isUImm5() const { return IsUImm<5>(); }
+  bool isUImm7() const { return IsUImm<7>(); }
 
   bool isRnumArg() const {
     int64_t Imm;
@@ -793,6 +816,11 @@ struct RISCVOperand final : public MCParsedAsmOperand {
     return Imm.Val;
   }
 
+  uint64_t getFPConst() const {
+    assert(Kind == KindTy::FPImmediate && "Invalid type access!");
+    return FPImm.Val;
+  }
+
   StringRef getToken() const {
     assert(Kind == KindTy::Token && "Invalid type access!");
     return Tok;
@@ -825,6 +853,8 @@ struct RISCVOperand final : public MCParsedAsmOperand {
     case KindTy::Immediate:
       OS << *getImm();
       break;
+    case KindTy::FPImmediate:
+      break;
     case KindTy::Register:
       OS << "<register " << RegName(getReg()) << ">";
       break;
@@ -880,6 +910,14 @@ struct RISCVOperand final : public MCParsedAsmOperand {
     return Op;
   }
 
+  static std::unique_ptr<RISCVOperand> createFPImm(uint64_t Val, SMLoc S) {
+    auto Op = std::make_unique<RISCVOperand>(KindTy::FPImmediate);
+    Op->FPImm.Val = Val;
+    Op->StartLoc = S;
+    Op->EndLoc = S;
+    return Op;
+  }
+
   static std::unique_ptr<RISCVOperand> createSysReg(StringRef Str, SMLoc S,
                                                     unsigned Encoding) {
     auto Op = std::make_unique<RISCVOperand>(KindTy::SystemRegister);
@@ -940,6 +978,18 @@ struct RISCVOperand final : public MCParsedAsmOperand {
     addExpr(Inst, getImm(), isRV64Imm());
   }
 
+  void addFPImmOperands(MCInst &Inst, unsigned N) const {
+    assert(N == 1 && "Invalid number of operands!");
+    if (isImm()) {
+      addExpr(Inst, getImm(), isRV64Imm());
+      return;
+    }
+
+    int Imm = RISCVLoadFPImm::getLoadFPImm(
+        APFloat(APFloat::IEEEdouble(), APInt(64, getFPConst())));
+    Inst.addOperand(MCOperand::createImm(Imm));
+  }
+
   void addFenceArgOperands(MCInst &Inst, unsigned N) const {
     assert(N == 1 && "Invalid number of operands!");
     Inst.addOperand(MCOperand::createImm(Fence.Val));
@@ -1246,6 +1296,10 @@ bool RISCVAsmParser::MatchAndEmitInstruction(SMLoc IDLoc, unsigned &Opcode,
                                       "operand must be a valid system register "
                                       "name or an integer in the range");
   }
+  case Match_InvalidLoadFPImm: {
+    SMLoc ErrorLoc = ((RISCVOperand &)*Operands[ErrorInfo]).getStartLoc();
+    return Error(ErrorLoc, "operand must be a valid floating-point constant");
+  }
   case Match_InvalidBareSymbol: {
     SMLoc ErrorLoc = ((RISCVOperand &)*Operands[ErrorInfo]).getStartLoc();
     return Error(ErrorLoc, "operand must be a bare symbol name");
@@ -1519,6 +1573,63 @@ RISCVAsmParser::parseCSRSystemRegister(OperandVector &Operands) {
   return MatchOperand_NoMatch;
 }
 
+OperandMatchResultTy RISCVAsmParser::parseFPImm(OperandVector &Operands) {
+  SMLoc S = getLoc();
+
+  // Parse special floats (inf/nan/min) representation.
+  if (getTok().is(AsmToken::Identifier)) {
+    StringRef Identifier = getTok().getIdentifier();
+    if (Identifier.compare_insensitive("inf") == 0) {
+      Operands.push_back(
+          RISCVOperand::createImm(MCConstantExpr::create(30, getContext()), S,
+                                  getTok().getEndLoc(), isRV64()));
+    } else if (Identifier.compare_insensitive("nan") == 0) {
+      Operands.push_back(
+          RISCVOperand::createImm(MCConstantExpr::create(31, getContext()), S,
+                                  getTok().getEndLoc(), isRV64()));
+    } else if (Identifier.compare_insensitive("min") == 0) {
+      Operands.push_back(
+          RISCVOperand::createImm(MCConstantExpr::create(1, getContext()), S,
+                                  getTok().getEndLoc(), isRV64()));
+    } else {
+      TokError("invalid floating point literal");
+      return MatchOperand_ParseFail;
+    }
+
+    Lex(); // Eat the token.
+
+    return MatchOperand_Success;
+  }
+
+  // Handle negation, as that still comes through as a separate token.
+  bool IsNegative = parseOptionalToken(AsmToken::Minus);
+
+  const AsmToken &Tok = getTok();
+  if (!Tok.is(AsmToken::Real)) {
+    TokError("invalid floating point immediate");
+    return MatchOperand_ParseFail;
+  }
+
+  // Parse FP representation.
+  APFloat RealVal(APFloat::IEEEdouble());
+  auto StatusOrErr =
+      RealVal.convertFromString(Tok.getString(), APFloat::rmTowardZero);
+  if (errorToBool(StatusOrErr.takeError())) {
+    TokError("invalid floating point representation");
+    return MatchOperand_ParseFail;
+  }
+
+  if (IsNegative)
+    RealVal.changeSign();
+
+  Operands.push_back(RISCVOperand::createFPImm(
+      RealVal.bitcastToAPInt().getZExtValue(), S));
+
+  Lex(); // Eat the token.
+
+  return MatchOperand_Success;
+}
+
 OperandMatchResultTy RISCVAsmParser::parseImmediate(OperandVector &Operands) {
   SMLoc S = getLoc();
   SMLoc E;
diff --git a/llvm/lib/Target/RISCV/MCTargetDesc/RISCVBaseInfo.cpp b/llvm/lib/Target/RISCV/MCTargetDesc/RISCVBaseInfo.cpp
index 90734c921c22f..98c8e883e5960 100644
--- a/llvm/lib/Target/RISCV/MCTargetDesc/RISCVBaseInfo.cpp
+++ b/llvm/lib/Target/RISCV/MCTargetDesc/RISCVBaseInfo.cpp
@@ -214,4 +214,79 @@ bool RISCVRVC::uncompress(MCInst &OutInst, const MCInst &MI,
   return uncompressInst(OutInst, MI, STI);
 }
 
+// Lookup table for fli.s for entries 2-31.
+static constexpr std::pair<uint8_t, uint8_t> LoadFP32ImmArr[] = {
+    {0b01101111, 0b00}, {0b01110000, 0b00}, {0b01110111, 0b00},
+    {0b01111000, 0b00}, {0b01111011, 0b00}, {0b01111100, 0b00},
+    {0b01111101, 0b00}, {0b01111101, 0b01}, {0b01111101, 0b10},
+    {0b01111101, 0b11}, {0b01111110, 0b00}, {0b01111110, 0b01},
+    {0b01111110, 0b10}, {0b01111110, 0b11}, {0b01111111, 0b00},
+    {0b01111111, 0b01}, {0b01111111, 0b10}, {0b01111111, 0b11},
+    {0b10000000, 0b00}, {0b10000000, 0b01}, {0b10000000, 0b10},
+    {0b10000001, 0b00}, {0b10000010, 0b00}, {0b10000011, 0b00},
+    {0b10000110, 0b00}, {0b10000111, 0b00}, {0b10001110, 0b00},
+    {0b10001111, 0b00}, {0b11111111, 0b00}, {0b11111111, 0b10},
+};
+
+int RISCVLoadFPImm::getLoadFPImm(APFloat FPImm) {
+  assert((&FPImm.getSemantics() == &APFloat::IEEEsingle() ||
+          &FPImm.getSemantics() == &APFloat::IEEEdouble() ||
+          &FPImm.getSemantics() == &APFloat::IEEEhalf()) &&
+         "Unexpected semantics");
+
+  // Handle the minimum normalized value which is different for each type.
+  if (FPImm.isSmallestNormalized())
+    return 1;
+
+  // Convert to single precision to use its lookup table.
+  bool LosesInfo;
+  APFloat::opStatus Status = FPImm.convert(
+      APFloat::IEEEsingle(), APFloat::rmNearestTiesToEven, &LosesInfo);
+  if (Status != APFloat::opOK || LosesInfo)
+    return -1;
+
+  APInt Imm = FPImm.bitcastToAPInt();
+
+  if (Imm.extractBitsAsZExtValue(21, 0) != 0)
+    return -1;
+
+  bool Sign = Imm.extractBitsAsZExtValue(1, 31);
+  uint8_t Mantissa = Imm.extractBitsAsZExtValue(2, 21);
+  uint8_t Exp = Imm.extractBitsAsZExtValue(8, 23);
+
+  auto EMI = llvm::lower_bound(LoadFP32ImmArr, std::make_pair(Exp, Mantissa));
+  if (EMI == std::end(LoadFP32ImmArr) || EMI->first != Exp ||
+      EMI->second != Mantissa)
+    return -1;
+
+  // Table doesn't have entry 0 or 1.
+  int Entry = std::distance(std::begin(LoadFP32ImmArr), EMI) + 2;
+
+  // The only legal negative value is -1.0(entry 0). 1.0 is entry 16.
+  if (Sign) {
+    if (Entry == 16)
+      return 0;
+    return false;
+  }
+
+  return Entry;
+}
+
+float RISCVLoadFPImm::getFPImm(unsigned Imm) {
+  assert(Imm != 1 && Imm != 30 && Imm != 31 && "Unsupported immediate");
+
+  // Entry 0 is -1.0, the only negative value. Entry 16 is 1.0.
+  uint32_t Sign = 0;
+  if (Imm == 0) {
+    Sign = 0b1;
+    Imm = 16;
+  }
+
+  uint32_t Exp = LoadFP32ImmArr[Imm - 2].first;
+  uint32_t Mantissa = LoadFP32ImmArr[Imm - 2].second;
+
+  uint32_t I = Sign << 31 | Exp << 23 | Mantissa << 21;
+  return bit_cast<float>(I);
+}
+
 } // namespace llvm
diff --git a/llvm/lib/Target/RISCV/MCTargetDesc/RISCVBaseInfo.h b/llvm/lib/Target/RISCV/MCTargetDesc/RISCVBaseInfo.h
index c7a6fa6dd1010..70fdc0e4ff120 100644
--- a/llvm/lib/Target/RISCV/MCTargetDesc/RISCVBaseInfo.h
+++ b/llvm/lib/Target/RISCV/MCTargetDesc/RISCVBaseInfo.h
@@ -14,6 +14,8 @@
 #define LLVM_LIB_TARGET_RISCV_MCTARGETDESC_RISCVBASEINFO_H
 
 #include "MCTargetDesc/RISCVMCTargetDesc.h"
+#include "llvm/ADT/APFloat.h"
+#include "llvm/ADT/APInt.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/MC/MCInstrDesc.h"
@@ -340,6 +342,19 @@ inline static bool isValidRoundingMode(unsigned Mode) {
 }
 } // namespace RISCVFPRndMode
 
+//===----------------------------------------------------------------------===//
+// Floating-point Immediates
+//
+
+namespace RISCVLoadFPImm {
+float getFPImm(unsigned Imm);
+
+/// getLoadFPImm - Return a 5-bit binary encoding of the floating-point
+/// immediate value. If the value cannot be represented as a 5-bit binary
+/// encoding, then return -1.
+int getLoadFPImm(APFloat FPImm);
+} // namespace RISCVLoadFPImm
+
 namespace RISCVSysReg {
 struct SysReg {
   const char *Name;
diff --git a/llvm/lib/Target/RISCV/MCTargetDesc/RISCVELFObjectWriter.cpp b/llvm/lib/Target/RISCV/MCTargetDesc/RISCVELFObjectWriter.cpp
index ee88fe7fe858a..4cc60dfeb95c6 100644
--- a/llvm/lib/Target/RISCV/MCTargetDesc/RISCVELFObjectWriter.cpp
+++ b/llvm/lib/Target/RISCV/MCTargetDesc/RISCVELFObjectWriter.cpp
@@ -58,7 +58,7 @@ unsigned RISCVELFObjectWriter::getRelocType(MCContext &Ctx,
   if (IsPCRel) {
     switch (Kind) {
     default:
-      Ctx.reportError(Fixup.getLoc(), "Unsupported relocation type");
+      Ctx.reportError(Fixup.getLoc(), "unsupported relocation type");
       return ELF::R_RISCV_NONE;
     case FK_Data_4:
     case FK_PCRel_4:
@@ -110,7 +110,7 @@ unsigned RISCVELFObjectWriter::getRelocType(MCContext &Ctx,
 
   switch (Kind) {
   default:
-    Ctx.reportError(Fixup.getLoc(), "Unsupported relocation type");
+    Ctx.reportError(Fixup.getLoc(), "unsupported relocation type");
     return ELF::R_RISCV_NONE;
   case FK_Data_1:
     Ctx.reportError(Fixup.getLoc(), "1-byte data relocations not supported");
diff --git a/llvm/lib/Target/RISCV/MCTargetDesc/RISCVELFStreamer.cpp b/llvm/lib/Target/RISCV/MCTargetDesc/RISCVELFStreamer.cpp
index f69fc7cb24f72..a05254b0ae579 100644
--- a/llvm/lib/Target/RISCV/MCTargetDesc/RISCVELFStreamer.cpp
+++ b/llvm/lib/Target/RISCV/MCTargetDesc/RISCVELFStreamer.cpp
@@ -53,98 +53,28 @@ void RISCVTargetELFStreamer::emitDirectiveOptionRelax() {}
 void RISCVTargetELFStreamer::emitDirectiveOptionNoRelax() {}
 
 void RISCVTargetELFStreamer::emitAttribute(unsigned Attribute, unsigned Value) {
-  setAttributeItem(Attribute, Value, /*OverwriteExisting=*/true);
+  getStreamer().setAttributeItem(Attribute, Value, /*OverwriteExisting=*/true);
 }
 
 void RISCVTargetELFStreamer::emitTextAttribute(unsigned Attribute,
                                                StringRef String) {
-  setAttributeItem(Attribute, String, /*OverwriteExisting=*/true);
+  getStreamer().setAttributeItem(Attribute, String, /*OverwriteExisting=*/true);
 }
 
 void RISCVTargetELFStreamer::emitIntTextAttribute(unsigned Attribute,
                                                   unsigned IntValue,
                                                   StringRef StringValue) {
-  setAttributeItems(Attribute, IntValue, StringValue,
-                    /*OverwriteExisting=*/true);
+  getStreamer().setAttributeItems(Attribute, IntValue, StringValue,
+                                  /*OverwriteExisting=*/true);
 }
 
 void RISCVTargetELFStreamer::finishAttributeSection() {
-  if (Contents.empty())
+  RISCVELFStreamer &S = getStreamer();
+  if (S.Contents.empty())
     return;
 
-  if (AttributeSection) {
-    Streamer.switchSection(AttributeSection);
-  } else {
-    MCAssembler &MCA = getStreamer().getAssembler();
-    AttributeSection = MCA.getContext().getELFSection(
-        ".riscv.attributes", ELF::SHT_RISCV_ATTRIBUTES, 0);
-    Streamer.switchSection(AttributeSection);
-
-    Streamer.emitInt8(ELFAttrs::Format_Version);
-  }
-
-  // Vendor size + Vendor name + '\0'
-  const size_t VendorHeaderSize = 4 + CurrentVendor.size() + 1;
-
-  // Tag + Tag Size
-  const size_t TagHeaderSize = 1 + 4;
-
-  const size_t ContentsSize = calculateContentSize();
-
-  Streamer.emitInt32(VendorHeaderSize + TagHeaderSize + ContentsSize);
-  Streamer.emitBytes(CurrentVendor);
-  Streamer.emitInt8(0); // '\0'
-
-  Streamer.emitInt8(ELFAttrs::File);
-  Streamer.emitInt32(TagHeaderSize + ContentsSize);
-
-  // Size should have been accounted for already, now
-  // emit each field as its type (ULEB or String).
-  for (AttributeItem item : Contents) {
-    Streamer.emitULEB128IntValue(item.Tag);
-    switch (item.Type) {
-    default:
-      llvm_unreachable("Invalid attribute type");
-    case AttributeType::Numeric:
-      Streamer.emitULEB128IntValue(item.IntValue);
-      break;
-    case AttributeType::Text:
-      Streamer.emitBytes(item.StringValue);
-      Streamer.emitInt8(0); // '\0'
-      break;
-    case AttributeType::NumericAndText:
-      Streamer.emitULEB128IntValue(item.IntValue);
-      Streamer.emitBytes(item.StringValue);
-      Streamer.emitInt8(0); // '\0'
-      break;
-    }
-  }
-
-  Contents.clear();
-}
-
-size_t RISCVTargetELFStreamer::calculateContentSize() const {
-  size_t Result = 0;
-  for (AttributeItem item : Contents) {
-    switch (item.Type) {
-    case AttributeType::Hidden:
-      break;
-    case AttributeType::Numeric:
-      Result += getULEB128Size(item.Tag);
-      Result += getULEB128Size(item.IntValue);
-      break;
-    case AttributeType::Text:
-      Result += getULEB128Size(item.Tag);
-      Result += item.StringValue.size() + 1; // string + '\0'
-      break;
-    case AttributeType::NumericAndText:
-      Result += getULEB128Size(item.Tag);
-      Result += getULEB128Size(item.IntValue);
-      Result += item.StringValue.size() + 1; // string + '\0';
-      break;
-    }
-  }
-  return Result;
+  S.emitAttributesSection(CurrentVendor, ".riscv.attributes",
+                          ELF::SHT_RISCV_ATTRIBUTES, AttributeSection);
 }
 
 void RISCVTargetELFStreamer::finish() {
@@ -184,7 +114,6 @@ void RISCVTargetELFStreamer::finish() {
 
 void RISCVTargetELFStreamer::reset() {
   AttributeSection = nullptr;
-  Contents.clear();
 }
 
 void RISCVTargetELFStreamer::emitDirectiveVariantCC(MCSymbol &Symbol) {
@@ -211,6 +140,10 @@ bool RISCVELFStreamer::requiresFixups(MCContext &C, const MCExpr *Value,
                              MCConstantExpr::create(E.getConstant(), C), C);
   RHS = E.getSymB();
 
+  // Avoid ADD/SUB if Kind is not VK_None, e.g. A@plt - B + C.
+  if (E.getSymA()->getKind() != MCSymbolRefExpr::VK_None)
+    return false;
+
   // If either symbol is in a text section, we need to delay the relocation
   // evaluation as relaxation may alter the size of the symbol.
   //
@@ -223,6 +156,16 @@ bool RISCVELFStreamer::requiresFixups(MCContext &C, const MCExpr *Value,
   if (B.isInSection() && B.getSection().getKind().isText())
     return true;
 
+  // If A is undefined and B is defined, we should emit ADD/SUB for A-B.
+  // Unfortunately, A may be defined later, but this requiresFixups call has to
+  // eagerly make a decision. For now, emit ADD/SUB unless A is .L*. This
+  // heuristic handles many temporary label differences for .debug_* and
+  // .apple_types sections.
+  //
+  // TODO Implement delayed relocation decision.
+  if (!A.isInSection() && !A.isTemporary() && B.isInSection())
+    return true;
+
   // Support cross-section symbolic differences ...
   return A.isInSection() && B.isInSection() &&
          A.getSection().getName() != B.getSection().getName();
diff --git a/llvm/lib/Target/RISCV/MCTargetDesc/RISCVELFStreamer.h b/llvm/lib/Target/RISCV/MCTargetDesc/RISCVELFStreamer.h
index 80b8be2ddc2cf..cf758bc7dadf6 100644
--- a/llvm/lib/Target/RISCV/MCTargetDesc/RISCVELFStreamer.h
+++ b/llvm/lib/Target/RISCV/MCTargetDesc/RISCVELFStreamer.h
@@ -32,81 +32,16 @@ namespace llvm {
 
 class RISCVTargetELFStreamer : public RISCVTargetStreamer {
 private:
-  enum class AttributeType { Hidden, Numeric, Text, NumericAndText };
-
-  struct AttributeItem {
-    AttributeType Type;
-    unsigned Tag;
-    unsigned IntValue;
-    std::string StringValue;
-  };
-
   StringRef CurrentVendor;
-  SmallVector<AttributeItem, 64> Contents;
 
   MCSection *AttributeSection = nullptr;
   const MCSubtargetInfo &STI;
 
-  AttributeItem *getAttributeItem(unsigned Attribute) {
-    for (size_t i = 0; i < Contents.size(); ++i)
-      if (Contents[i].Tag == Attribute)
-        return &Contents[i];
-    return nullptr;
-  }
-
-  void setAttributeItem(unsigned Attribute, unsigned Value,
-                        bool OverwriteExisting) {
-    // Look for existing attribute item.
-    if (AttributeItem *Item = getAttributeItem(Attribute)) {
-      if (!OverwriteExisting)
-        return;
-      Item->Type = AttributeType::Numeric;
-      Item->IntValue = Value;
-      return;
-    }
-
-    // Create new attribute item.
-    Contents.push_back({AttributeType::Numeric, Attribute, Value, ""});
-  }
-
-  void setAttributeItem(unsigned Attribute, StringRef Value,
-                        bool OverwriteExisting) {
-    // Look for existing attribute item.
-    if (AttributeItem *Item = getAttributeItem(Attribute)) {
-      if (!OverwriteExisting)
-        return;
-      Item->Type = AttributeType::Text;
-      Item->StringValue = std::string(Value);
-      return;
-    }
-
-    // Create new attribute item.
-    Contents.push_back({AttributeType::Text, Attribute, 0, std::string(Value)});
-  }
-
-  void setAttributeItems(unsigned Attribute, unsigned IntValue,
-                         StringRef StringValue, bool OverwriteExisting) {
-    // Look for existing attribute item.
-    if (AttributeItem *Item = getAttributeItem(Attribute)) {
-      if (!OverwriteExisting)
-        return;
-      Item->Type = AttributeType::NumericAndText;
-      Item->IntValue = IntValue;
-      Item->StringValue = std::string(StringValue);
-      return;
-    }
-
-    // Create new attribute item.
-    Contents.push_back({AttributeType::NumericAndText, Attribute, IntValue,
-                        std::string(StringValue)});
-  }
-
   void emitAttribute(unsigned Attribute, unsigned Value) override;
   void emitTextAttribute(unsigned Attribute, StringRef String) override;
   void emitIntTextAttribute(unsigned Attribute, unsigned IntValue,
                             StringRef StringValue) override;
   void finishAttributeSection() override;
-  size_t calculateContentSize() const;
 
   void reset() override;
 
diff --git a/llvm/lib/Target/RISCV/MCTargetDesc/RISCVInstPrinter.cpp b/llvm/lib/Target/RISCV/MCTargetDesc/RISCVInstPrinter.cpp
index bae812eca595f..becd8aaf6bb91 100644
--- a/llvm/lib/Target/RISCV/MCTargetDesc/RISCVInstPrinter.cpp
+++ b/llvm/lib/Target/RISCV/MCTargetDesc/RISCVInstPrinter.cpp
@@ -154,6 +154,29 @@ void RISCVInstPrinter::printFRMArg(const MCInst *MI, unsigned OpNo,
   O << ", " << RISCVFPRndMode::roundingModeToString(FRMArg);
 }
 
+void RISCVInstPrinter::printFPImmOperand(const MCInst *MI, unsigned OpNo,
+                                         const MCSubtargetInfo &STI,
+                                         raw_ostream &O) {
+  unsigned Imm = MI->getOperand(OpNo).getImm();
+  if (Imm == 1) {
+    O << "min";
+  } else if (Imm == 30) {
+    O << "inf";
+  } else if (Imm == 31) {
+    O << "nan";
+  } else {
+    float FPVal = RISCVLoadFPImm::getFPImm(Imm);
+    // If the value is an integer, print a .0 fraction. Otherwise, use %g to
+    // which will not print trailing zeros and will use scientific notation
+    // if it is shorter than printing as a decimal. The smallest value requires
+    // 12 digits of precision including the decimal.
+    if (FPVal == (int)(FPVal))
+      O << format("%.1f", FPVal);
+    else
+      O << format("%.12g", FPVal);
+  }
+}
+
 void RISCVInstPrinter::printZeroOffsetMemOp(const MCInst *MI, unsigned OpNo,
                                             const MCSubtargetInfo &STI,
                                             raw_ostream &O) {
diff --git a/llvm/lib/Target/RISCV/MCTargetDesc/RISCVInstPrinter.h b/llvm/lib/Target/RISCV/MCTargetDesc/RISCVInstPrinter.h
index d7d93842e80c8..02a3b968b7663 100644
--- a/llvm/lib/Target/RISCV/MCTargetDesc/RISCVInstPrinter.h
+++ b/llvm/lib/Target/RISCV/MCTargetDesc/RISCVInstPrinter.h
@@ -40,6 +40,8 @@ class RISCVInstPrinter : public MCInstPrinter {
                      const MCSubtargetInfo &STI, raw_ostream &O);
   void printFRMArg(const MCInst *MI, unsigned OpNo, const MCSubtargetInfo &STI,
                    raw_ostream &O);
+  void printFPImmOperand(const MCInst *MI, unsigned OpNo,
+                         const MCSubtargetInfo &STI, raw_ostream &O);
   void printZeroOffsetMemOp(const MCInst *MI, unsigned OpNo,
                             const MCSubtargetInfo &STI, raw_ostream &O);
   void printVTypeI(const MCInst *MI, unsigned OpNo, const MCSubtargetInfo &STI,
diff --git a/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp b/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
index 30a26e404cbbe..6eefaa27539b3 100644
--- a/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
+++ b/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
@@ -178,18 +178,17 @@ bool RISCVAsmPrinter::PrintAsmMemoryOperand(const MachineInstr *MI,
                                             unsigned OpNo,
                                             const char *ExtraCode,
                                             raw_ostream &OS) {
-  if (!ExtraCode) {
-    const MachineOperand &MO = MI->getOperand(OpNo);
-    // For now, we only support register memory operands in registers and
-    // assume there is no addend
-    if (!MO.isReg())
-      return true;
-
-    OS << "0(" << RISCVInstPrinter::getRegisterName(MO.getReg()) << ")";
-    return false;
-  }
+  if (ExtraCode)
+    return AsmPrinter::PrintAsmMemoryOperand(MI, OpNo, ExtraCode, OS);
 
-  return AsmPrinter::PrintAsmMemoryOperand(MI, OpNo, ExtraCode, OS);
+  const MachineOperand &MO = MI->getOperand(OpNo);
+  // For now, we only support register memory operands in registers and
+  // assume there is no addend
+  if (!MO.isReg())
+    return true;
+
+  OS << "0(" << RISCVInstPrinter::getRegisterName(MO.getReg()) << ")";
+  return false;
 }
 
 bool RISCVAsmPrinter::runOnMachineFunction(MachineFunction &MF) {
diff --git a/llvm/lib/Target/RISCV/RISCVFeatures.td b/llvm/lib/Target/RISCV/RISCVFeatures.td
index b439563fc2b91..b0bb2992f6b42 100644
--- a/llvm/lib/Target/RISCV/RISCVFeatures.td
+++ b/llvm/lib/Target/RISCV/RISCVFeatures.td
@@ -424,6 +424,12 @@ def FeatureStdExtZvfh
                        "'Zvfh' (Vector Half-Precision Floating-Point)",
                        [FeatureStdExtZve32f]>;
 
+def HasStdExtZfhOrZvfh
+    : Predicate<"Subtarget->hasStdExtZfh() || Subtarget->hasStdExtZvfh()">,
+                AssemblerPredicate<(any_of FeatureStdExtZfh, FeatureStdExtZvfh),
+                                   "'Zfh' (Half-Precision Floating-Point) or "
+                                   "'Zvfh' (Vector Half-Precision Floating-Point)">;
+
 def FeatureStdExtZicbom
     : SubtargetFeature<"zicbom", "HasStdExtZicbom", "true",
                        "'Zicbom' (Cache-Block Management Instructions)">;
@@ -570,9 +576,9 @@ def HasVendorXTHeadVdot : Predicate<"Subtarget->hasVendorXTHeadVdot()">,
 // Feature32Bit exists to mark CPUs that support RV32 to distinquish them from
 // tuning CPU names.
 def Feature32Bit
-    : SubtargetFeature<"32bit", "HasRV32", "true", "Implements RV32">;
+    : SubtargetFeature<"32bit", "IsRV32", "true", "Implements RV32">;
 def Feature64Bit
-    : SubtargetFeature<"64bit", "HasRV64", "true", "Implements RV64">;
+    : SubtargetFeature<"64bit", "IsRV64", "true", "Implements RV64">;
 def IsRV64 : Predicate<"Subtarget->is64Bit()">,
                        AssemblerPredicate<(all_of Feature64Bit),
                                          "RV64I Base Instruction Set">;
@@ -581,7 +587,7 @@ def IsRV32 : Predicate<"!Subtarget->is64Bit()">,
                                           "RV32I Base Instruction Set">;
 
 defvar RV32 = DefaultMode;
-def RV64           : HwMode<"+64bit">;
+def RV64           : HwMode<"+64bit", [IsRV64]>;
 
 def FeatureRV32E
     : SubtargetFeature<"e", "IsRV32E", "true",
diff --git a/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp b/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp
index bb55c16bf135a..a68096fc20d10 100644
--- a/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp
+++ b/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp
@@ -36,7 +36,8 @@ static void emitSCSPrologue(MachineFunction &MF, MachineBasicBlock &MBB,
     return;
 
   const auto &STI = MF.getSubtarget<RISCVSubtarget>();
-  Register RAReg = STI.getRegisterInfo()->getRARegister();
+  const llvm::RISCVRegisterInfo *TRI = STI.getRegisterInfo();
+  Register RAReg = TRI->getRARegister();
 
   // Do not save RA to the SCS if it's not saved to the regular stack,
   // i.e. RA is not at risk of being overwritten.
@@ -78,6 +79,26 @@ static void emitSCSPrologue(MachineFunction &MF, MachineBasicBlock &MBB,
       .addReg(SCSPReg)
       .addImm(SlotSize)
       .setMIFlag(MachineInstr::FrameSetup);
+
+  // Emit a CFI instruction that causes SlotSize to be subtracted from the value
+  // of the shadow stack pointer when unwinding past this frame.
+  char DwarfSCSReg = TRI->getDwarfRegNum(SCSPReg, /*IsEH*/ true);
+  assert(DwarfSCSReg < 32 && "SCS Register should be < 32 (X18).");
+
+  char Offset = static_cast<char>(-SlotSize) & 0x7f;
+  const char CFIInst[] = {
+      dwarf::DW_CFA_val_expression,
+      DwarfSCSReg, // register
+      2,           // length
+      static_cast<char>(unsigned(dwarf::DW_OP_breg0 + DwarfSCSReg)),
+      Offset, // addend (sleb128)
+  };
+
+  unsigned CFIIndex = MF.addFrameInst(MCCFIInstruction::createEscape(
+      nullptr, StringRef(CFIInst, sizeof(CFIInst))));
+  BuildMI(MBB, MI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
+      .addCFIIndex(CFIIndex)
+      .setMIFlag(MachineInstr::FrameSetup);
 }
 
 static void emitSCSEpilogue(MachineFunction &MF, MachineBasicBlock &MBB,
@@ -128,6 +149,12 @@ static void emitSCSEpilogue(MachineFunction &MF, MachineBasicBlock &MBB,
       .addReg(SCSPReg)
       .addImm(-SlotSize)
       .setMIFlag(MachineInstr::FrameDestroy);
+  // Restore the SCS pointer
+  unsigned CFIIndex = MF.addFrameInst(MCCFIInstruction::createRestore(
+      nullptr, STI.getRegisterInfo()->getDwarfRegNum(SCSPReg, /*IsEH*/ true)));
+  BuildMI(MBB, MI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
+      .addCFIIndex(CFIIndex)
+      .setMIFlags(MachineInstr::FrameDestroy);
 }
 
 // Get the ID of the libcall used for spilling and restoring callee saved
diff --git a/llvm/lib/Target/RISCV/RISCVISelDAGToDAG.cpp b/llvm/lib/Target/RISCV/RISCVISelDAGToDAG.cpp
index ca63794db7b6e..f397ef12913db 100644
--- a/llvm/lib/Target/RISCV/RISCVISelDAGToDAG.cpp
+++ b/llvm/lib/Target/RISCV/RISCVISelDAGToDAG.cpp
@@ -842,6 +842,9 @@ void RISCVDAGToDAGISel::Select(SDNode *Node) {
   }
   case ISD::ConstantFP: {
     const APFloat &APF = cast<ConstantFPSDNode>(Node)->getValueAPF();
+    if (static_cast<const RISCVTargetLowering *>(TLI)->isLegalZfaFPImm(APF, VT))
+      break;
+
     bool NegZeroF64 = APF.isNegZero() && VT == MVT::f64;
     SDValue Imm;
     // For +0.0 or f64 -0.0 we need to start from X0. For all others, we will
@@ -2961,8 +2964,14 @@ bool RISCVDAGToDAGISel::selectFPImm(SDValue N, SDValue &Imm) {
   // td can handle +0.0 already.
   if (APF.isPosZero())
     return false;
+
+  MVT VT = CFP->getSimpleValueType(0);
+
+  if (static_cast<const RISCVTargetLowering *>(TLI)->isLegalZfaFPImm(APF, VT))
+    return false;
+
   MVT XLenVT = Subtarget->getXLenVT();
-  if (CFP->getValueType(0) == MVT::f64 && !Subtarget->is64Bit()) {
+  if (VT == MVT::f64 && !Subtarget->is64Bit()) {
     assert(APF.isNegZero() && "Unexpected constant.");
     return false;
   }
@@ -3158,10 +3167,15 @@ bool RISCVDAGToDAGISel::doPeepholeMaskedRVV(SDNode *N) {
   return true;
 }
 
-// Try to fold VMERGE_VVM with unmasked intrinsic to masked intrinsic. The
-// peephole only deals with VMERGE_VVM which is TU and has false operand same as
-// its true operand now. E.g. (VMERGE_VVM_M1_TU False, False, (VADD_M1 ...),
-// ...) -> (VADD_VV_M1_MASK)
+// Try to fold away VMERGE_VVM instructions. We handle these cases:
+// -Masked TU VMERGE_VVM combined with an unmasked TA instruction instruction
+//  folds to a masked TU instruction. VMERGE_VVM must have have merge operand
+//  same as false operand.
+// -Masked TA VMERGE_VVM combined with an unmasked TA instruction fold to a
+//  masked TA instruction.
+// -Unmasked TU VMERGE_VVM combined with a masked MU TA instruction folds to
+//  masked TU instruction. Both instructions must have the same merge operand.
+//  VMERGE_VVM must have have merge operand same as false operand.
 bool RISCVDAGToDAGISel::performCombineVMergeAndVOps(SDNode *N, bool IsTA) {
   unsigned Offset = IsTA ? 0 : 1;
   uint64_t Policy = IsTA ? RISCVII::TAIL_AGNOSTIC : /*TUMU*/ 0;
@@ -3184,25 +3198,44 @@ bool RISCVDAGToDAGISel::performCombineVMergeAndVOps(SDNode *N, bool IsTA) {
   unsigned TrueOpc = True.getMachineOpcode();
 
   // Skip if True has merge operand.
-  // TODO: Deal with True having same merge operand with N.
-  if (RISCVII::hasMergeOp(TII->get(TrueOpc).TSFlags))
-    return false;
+  uint64_t TrueTSFlags = TII->get(TrueOpc).TSFlags;
+  bool HasMergeOp = RISCVII::hasMergeOp(TrueTSFlags);
+
+  if (HasMergeOp) {
+    // The vmerge instruction must be TU.
+    if (IsTA)
+      return false;
+    SDValue MergeOpN = N->getOperand(0);
+    SDValue MergeOpTrue = True->getOperand(0);
+    // Both the vmerge instruction and the True instruction must have the same
+    // merge operand. The vmerge instruction must have an all 1s mask since
+    // we're going to keep the mask from the True instruction.
+    // FIXME: Support mask agnostic True instruction which would have an
+    // undef merge operand.
+    if (MergeOpN != MergeOpTrue || !usesAllOnesMask(N, /* MaskOpIdx */ 3))
+      return false;
+  }
 
   // Skip if True has side effect.
-  // TODO: Support velff and vlsegff.
+  // TODO: Support vleff and vlsegff.
   if (TII->get(TrueOpc).hasUnmodeledSideEffects())
     return false;
 
-  // Only deal with True when True is unmasked intrinsic now.
   const RISCV::RISCVMaskedPseudoInfo *Info =
-      RISCV::lookupMaskedIntrinsicByUnmaskedTA(TrueOpc);
+      HasMergeOp ? RISCV::getMaskedPseudoInfo(TrueOpc)
+                 : RISCV::lookupMaskedIntrinsicByUnmaskedTA(TrueOpc);
 
   if (!Info)
     return false;
 
-  // The last operand of unmasked intrinsic should be sew or chain.
+  // The last operand of a masked instruction may be glued.
+  bool HasGlueOp = True->getGluedNode() != nullptr;
+
+  // The chain operand may exist either before the glued operands or in the last
+  // position.
+  unsigned TrueChainOpIdx = True.getNumOperands() - HasGlueOp - 1;
   bool HasChainOp =
-      True.getOperand(True.getNumOperands() - 1).getValueType() == MVT::Other;
+      True.getOperand(TrueChainOpIdx).getValueType() == MVT::Other;
 
   if (HasChainOp) {
     // Avoid creating cycles in the DAG. We must ensure that none of the other
@@ -3218,8 +3251,10 @@ bool RISCVDAGToDAGISel::performCombineVMergeAndVOps(SDNode *N, bool IsTA) {
       return false;
   }
 
-  // Need True has same VL with N.
-  unsigned TrueVLIndex = True.getNumOperands() - HasChainOp - 2;
+  // The vector policy operand may be present for masked intrinsics
+  bool HasVecPolicyOp = RISCVII::hasVecPolicyOp(TrueTSFlags);
+  unsigned TrueVLIndex =
+      True.getNumOperands() - HasVecPolicyOp - HasChainOp - HasGlueOp - 2;
   SDValue TrueVL = True.getOperand(TrueVLIndex);
 
   auto IsNoFPExcept = [this](SDValue N) {
@@ -3241,18 +3276,26 @@ bool RISCVDAGToDAGISel::performCombineVMergeAndVOps(SDNode *N, bool IsTA) {
          "Expected instructions with mask have merge operand.");
 
   SmallVector<SDValue, 8> Ops;
-  Ops.push_back(False);
-  Ops.append(True->op_begin(), True->op_begin() + TrueVLIndex);
-  Ops.append({Mask, VL, /* SEW */ True.getOperand(TrueVLIndex + 1)});
-  Ops.push_back(CurDAG->getTargetConstant(Policy, DL, Subtarget->getXLenVT()));
-
-  // Result node should have chain operand of True.
-  if (HasChainOp)
-    Ops.push_back(True.getOperand(True.getNumOperands() - 1));
-
-  // Result node should take over glued node of N.
-  if (N->getGluedNode())
-    Ops.push_back(N->getOperand(N->getNumOperands() - 1));
+  if (HasMergeOp) {
+    Ops.append(True->op_begin(), True->op_begin() + TrueVLIndex);
+    Ops.append({VL, /* SEW */ True.getOperand(TrueVLIndex + 1)});
+    Ops.push_back(
+        CurDAG->getTargetConstant(Policy, DL, Subtarget->getXLenVT()));
+    Ops.append(True->op_begin() + TrueVLIndex + 3, True->op_end());
+  } else {
+    Ops.push_back(False);
+    Ops.append(True->op_begin(), True->op_begin() + TrueVLIndex);
+    Ops.append({Mask, VL, /* SEW */ True.getOperand(TrueVLIndex + 1)});
+    Ops.push_back(
+        CurDAG->getTargetConstant(Policy, DL, Subtarget->getXLenVT()));
+
+    // Result node should have chain operand of True.
+    if (HasChainOp)
+      Ops.push_back(True.getOperand(TrueChainOpIdx));
+
+    if (N->getGluedNode())
+      Ops.push_back(N->getOperand(N->getNumOperands() - 1));
+  }
 
   SDNode *Result =
       CurDAG->getMachineNode(MaskedOpc, DL, True->getVTList(), Ops);
diff --git a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
index 6b93d53d7dbf3..e56a2b3b08b54 100644
--- a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
+++ b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
@@ -31,6 +31,7 @@
 #include "llvm/IR/DiagnosticInfo.h"
 #include "llvm/IR/DiagnosticPrinter.h"
 #include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/Instructions.h"
 #include "llvm/IR/IntrinsicsRISCV.h"
 #include "llvm/IR/PatternMatch.h"
 #include "llvm/Support/CommandLine.h"
@@ -802,6 +803,11 @@ RISCVTargetLowering::RISCVTargetLowering(const TargetMachine &TM,
       setOperationAction({ISD::VECTOR_REVERSE, ISD::VECTOR_SPLICE}, VT, Custom);
 
       setOperationAction(FloatingPointVPOps, VT, Custom);
+
+      setOperationAction(ISD::STRICT_FP_EXTEND, VT, Custom);
+      setOperationAction({ISD::STRICT_FADD, ISD::STRICT_FSUB, ISD::STRICT_FMUL,
+                          ISD::STRICT_FDIV},
+                         VT, Legal);
     };
 
     // Sets common extload/truncstore actions on RVV floating-point vector
@@ -1014,6 +1020,11 @@ RISCVTargetLowering::RISCVTargetLowering(const TargetMachine &TM,
         setOperationAction(FloatingPointVecReduceOps, VT, Custom);
 
         setOperationAction(FloatingPointVPOps, VT, Custom);
+
+        setOperationAction(ISD::STRICT_FP_EXTEND, VT, Custom);
+        setOperationAction({ISD::STRICT_FADD, ISD::STRICT_FSUB,
+                            ISD::STRICT_FMUL, ISD::STRICT_FDIV},
+                           VT, Custom);
       }
 
       // Custom-legalize bitcasts from fixed-length vectors to scalar types.
@@ -1178,6 +1189,24 @@ bool RISCVTargetLowering::getTgtMemIntrinsic(IntrinsicInfo &Info,
     Info.size = MemoryLocation::UnknownSize;
     Info.flags |= MachineMemOperand::MOLoad;
     return true;
+  case Intrinsic::riscv_seg2_store:
+  case Intrinsic::riscv_seg3_store:
+  case Intrinsic::riscv_seg4_store:
+  case Intrinsic::riscv_seg5_store:
+  case Intrinsic::riscv_seg6_store:
+  case Intrinsic::riscv_seg7_store:
+  case Intrinsic::riscv_seg8_store:
+    Info.opc = ISD::INTRINSIC_VOID;
+    // Operands are (vec, ..., vec, ptr, vl, int_id)
+    Info.ptrVal = I.getArgOperand(I.getNumOperands() - 3);
+    Info.memVT =
+        getValueType(DL, I.getArgOperand(0)->getType()->getScalarType());
+    Info.align = Align(
+        DL.getTypeSizeInBits(I.getArgOperand(0)->getType()->getScalarType()) /
+        8);
+    Info.size = MemoryLocation::UnknownSize;
+    Info.flags |= MachineMemOperand::MOStore;
+    return true;
   }
 }
 
@@ -1516,14 +1545,39 @@ bool RISCVTargetLowering::isOffsetFoldingLegal(
   return false;
 }
 
+bool RISCVTargetLowering::isLegalZfaFPImm(const APFloat &Imm, EVT VT) const {
+  if (!Subtarget.hasStdExtZfa())
+    return false;
+
+  bool IsSupportedVT = false;
+  if (VT == MVT::f16) {
+    IsSupportedVT = Subtarget.hasStdExtZfh() || Subtarget.hasStdExtZvfh();
+  } else if (VT == MVT::f32) {
+    IsSupportedVT = true;
+  } else if (VT == MVT::f64) {
+    assert(Subtarget.hasStdExtD() && "Expect D extension");
+    IsSupportedVT = true;
+  }
+
+  return IsSupportedVT && RISCVLoadFPImm::getLoadFPImm(Imm) != -1;
+}
+
 bool RISCVTargetLowering::isFPImmLegal(const APFloat &Imm, EVT VT,
                                        bool ForCodeSize) const {
-  if (VT == MVT::f16 && !Subtarget.hasStdExtZfhOrZfhmin())
-    return false;
-  if (VT == MVT::f32 && !Subtarget.hasStdExtF())
-    return false;
-  if (VT == MVT::f64 && !Subtarget.hasStdExtD())
+  bool IsLegalVT = false;
+  if (VT == MVT::f16)
+    IsLegalVT = Subtarget.hasStdExtZfhOrZfhmin();
+  else if (VT == MVT::f32)
+    IsLegalVT = Subtarget.hasStdExtF();
+  else if (VT == MVT::f64)
+    IsLegalVT = Subtarget.hasStdExtD();
+
+  if (!IsLegalVT)
     return false;
+
+  if (isLegalZfaFPImm(Imm, VT))
+    return true;
+
   // Cannot create a 64 bit floating-point immediate value for rv32.
   if (Subtarget.getXLen() < VT.getScalarSizeInBits()) {
     // td can handle +0.0 or -0.0 already.
@@ -3014,46 +3068,19 @@ static bool isInterleaveShuffle(ArrayRef<int> Mask, MVT VT, int &EvenSrc,
     return false;
 
   int Size = Mask.size();
-  int HalfSize = Size / 2;
   assert(Size == (int)VT.getVectorNumElements() && "Unexpected mask size");
 
-  int Srcs[] = {-1, -1};
-  for (int i = 0; i != Size; ++i) {
-    // Ignore undef elements.
-    if (Mask[i] < 0)
-      continue;
-
-    // Is this an even or odd element.
-    int Pol = i % 2;
-
-    // Ensure we consistently use the same half source for this polarity.
-    int Src = alignDown(Mask[i], HalfSize);
-    if (Srcs[Pol] < 0)
-      Srcs[Pol] = Src;
-    if (Srcs[Pol] != Src)
-      return false;
+  SmallVector<unsigned, 2> StartIndexes;
+  if (!ShuffleVectorInst::isInterleaveMask(Mask, 2, Size * 2, StartIndexes))
+    return false;
 
-    // Make sure the element within the source is appropriate for this element
-    // in the destination.
-    int Elt = Mask[i] % HalfSize;
-    if (Elt != i / 2)
-      return false;
-  }
+  EvenSrc = StartIndexes[0] % 2 ? StartIndexes[1] : StartIndexes[0];
+  OddSrc = StartIndexes[0] % 2 ? StartIndexes[0] : StartIndexes[1];
 
   // One source should be low half of first vector.
-  if (Srcs[0] != 0 && Srcs[1] != 0)
-    return false;
-
-  // Other source should be the upper half of the first source or the lower
-  // half of the second source.
-  // FIXME: This is only a heuristic to avoid regressions.
-  if (Srcs[0] != HalfSize && Srcs[0] != Size && Srcs[1] != HalfSize &&
-      Srcs[1] != Size)
+  if (EvenSrc != 0 && OddSrc != 0)
     return false;
 
-  EvenSrc = Srcs[0];
-  OddSrc = Srcs[1];
-
   return true;
 }
 
@@ -4066,6 +4093,8 @@ SDValue RISCVTargetLowering::LowerOperation(SDValue Op,
     if (!Op.getValueType().isVector())
       return Op;
     return lowerVectorFPExtendOrRoundLike(Op, DAG);
+  case ISD::STRICT_FP_EXTEND:
+    return lowerStrictFPExtend(Op, DAG);
   case ISD::FP_TO_SINT:
   case ISD::FP_TO_UINT:
   case ISD::SINT_TO_FP:
@@ -4407,6 +4436,18 @@ SDValue RISCVTargetLowering::LowerOperation(SDValue Op,
     return lowerFixedLengthVectorSelectToRVV(Op, DAG);
   case ISD::FCOPYSIGN:
     return lowerFixedLengthVectorFCOPYSIGNToRVV(Op, DAG);
+  case ISD::STRICT_FADD:
+    return lowerToScalableOp(Op, DAG, RISCVISD::STRICT_FADD_VL,
+                             /*HasMergeOp*/ true);
+  case ISD::STRICT_FSUB:
+    return lowerToScalableOp(Op, DAG, RISCVISD::STRICT_FSUB_VL,
+                             /*HasMergeOp*/ true);
+  case ISD::STRICT_FMUL:
+    return lowerToScalableOp(Op, DAG, RISCVISD::STRICT_FMUL_VL,
+                             /*HasMergeOp*/ true);
+  case ISD::STRICT_FDIV:
+    return lowerToScalableOp(Op, DAG, RISCVISD::STRICT_FDIV_VL,
+                             /*HasMergeOp*/ true);
   case ISD::MGATHER:
   case ISD::VP_GATHER:
     return lowerMaskedGather(Op, DAG);
@@ -5283,6 +5324,46 @@ SDValue RISCVTargetLowering::lowerVectorTruncLike(SDValue Op,
   return Result;
 }
 
+SDValue RISCVTargetLowering::lowerStrictFPExtend(SDValue Op,
+                                                 SelectionDAG &DAG) const {
+  SDLoc DL(Op);
+  SDValue Chain = Op.getOperand(0);
+  SDValue Src = Op.getOperand(1);
+  MVT VT = Op.getSimpleValueType();
+  MVT SrcVT = Src.getSimpleValueType();
+
+  MVT ContainerVT = VT;
+  if (VT.isFixedLengthVector()) {
+    MVT SrcContainerVT = getContainerForFixedLengthVector(SrcVT);
+    ContainerVT =
+        SrcContainerVT.changeVectorElementType(VT.getVectorElementType());
+    Src = convertToScalableVector(SrcContainerVT, Src, DAG, Subtarget);
+  }
+
+  auto [Mask, VL] = getDefaultVLOps(SrcVT, ContainerVT, DL, DAG, Subtarget);
+
+  // RVV can only widen fp to types double the size as the source, so it needs
+  // two vfwcvt to achieve extending fp16 to fp64.
+  if (VT.getVectorElementType() == MVT::f64 &&
+      SrcVT.getVectorElementType() == MVT::f16) {
+    MVT InterVT = ContainerVT.changeVectorElementType(MVT::f32);
+    Src = DAG.getNode(RISCVISD::STRICT_FP_EXTEND_VL, DL,
+                      DAG.getVTList(InterVT, MVT::Other), Chain, Src, Mask, VL);
+    Chain = Src.getValue(1);
+  }
+
+  SDValue Res =
+      DAG.getNode(RISCVISD::STRICT_FP_EXTEND_VL, DL,
+                  DAG.getVTList(ContainerVT, MVT::Other), Chain, Src, Mask, VL);
+  if (VT.isFixedLengthVector()) {
+    // StrictFP operations have two result values. Their lowered result should
+    // have same result count.
+    SDValue SubVec = convertFromScalableVector(VT, Res, DAG, Subtarget);
+    Res = DAG.getMergeValues({SubVec, Res.getValue(1)}, DL);
+  }
+  return Res;
+}
+
 SDValue
 RISCVTargetLowering::lowerVectorFPExtendOrRoundLike(SDValue Op,
                                                     SelectionDAG &DAG) const {
@@ -6019,6 +6100,41 @@ SDValue RISCVTargetLowering::LowerINTRINSIC_VOID(SDValue Op,
                                    Ops, Store->getMemoryVT(),
                                    Store->getMemOperand());
   }
+  case Intrinsic::riscv_seg2_store:
+  case Intrinsic::riscv_seg3_store:
+  case Intrinsic::riscv_seg4_store:
+  case Intrinsic::riscv_seg5_store:
+  case Intrinsic::riscv_seg6_store:
+  case Intrinsic::riscv_seg7_store:
+  case Intrinsic::riscv_seg8_store: {
+    SDLoc DL(Op);
+    static const Intrinsic::ID VssegInts[] = {
+        Intrinsic::riscv_vsseg2, Intrinsic::riscv_vsseg3,
+        Intrinsic::riscv_vsseg4, Intrinsic::riscv_vsseg5,
+        Intrinsic::riscv_vsseg6, Intrinsic::riscv_vsseg7,
+        Intrinsic::riscv_vsseg8};
+    // Operands are (chain, int_id, vec*, ptr, vl)
+    unsigned NF = Op->getNumOperands() - 4;
+    assert(NF >= 2 && NF <= 8 && "Unexpected seg number");
+    MVT XLenVT = Subtarget.getXLenVT();
+    MVT VT = Op->getOperand(2).getSimpleValueType();
+    MVT ContainerVT = getContainerForFixedLengthVector(VT);
+
+    SDValue VL = getVLOp(VT.getVectorNumElements(), DL, DAG, Subtarget);
+    SDValue IntID = DAG.getTargetConstant(VssegInts[NF - 2], DL, XLenVT);
+    SDValue Ptr = Op->getOperand(NF + 2);
+
+    auto *FixedIntrinsic = cast<MemIntrinsicSDNode>(Op);
+    SmallVector<SDValue, 12> Ops = {FixedIntrinsic->getChain(), IntID};
+    for (unsigned i = 0; i < NF; i++)
+      Ops.push_back(convertToScalableVector(
+          ContainerVT, FixedIntrinsic->getOperand(2 + i), DAG, Subtarget));
+    Ops.append({Ptr, VL});
+
+    return DAG.getMemIntrinsicNode(
+        ISD::INTRINSIC_VOID, DL, DAG.getVTList(MVT::Other), Ops,
+        FixedIntrinsic->getMemoryVT(), FixedIntrinsic->getMemOperand());
+  }
   }
 
   return SDValue();
@@ -7247,6 +7363,16 @@ SDValue RISCVTargetLowering::lowerToScalableOp(SDValue Op, SelectionDAG &DAG,
     Ops.push_back(Mask);
   Ops.push_back(VL);
 
+  // StrictFP operations have two result values. Their lowered result should
+  // have same result count.
+  if (Op->isStrictFPOpcode()) {
+    SDValue ScalableRes =
+        DAG.getNode(NewOpc, DL, DAG.getVTList(ContainerVT, MVT::Other), Ops,
+                    Op->getFlags());
+    SDValue SubVec = convertFromScalableVector(VT, ScalableRes, DAG, Subtarget);
+    return DAG.getMergeValues({SubVec, ScalableRes.getValue(1)}, DL);
+  }
+
   SDValue ScalableRes =
       DAG.getNode(NewOpc, DL, ContainerVT, Ops, Op->getFlags());
   return convertFromScalableVector(VT, ScalableRes, DAG, Subtarget);
@@ -10785,6 +10911,18 @@ SDValue RISCVTargetLowering::PerformDAGCombine(SDNode *N,
         SDValue Neg = DAG.getNegative(C, DL, VT);
         return DAG.getNode(ISD::AND, DL, VT, Neg, TrueV);
       }
+      // (riscvisd::select_cc x, 0, ne, x, 1) -> (add x, (setcc x, 0, eq))
+      // (riscvisd::select_cc x, 0, eq, 1, x) -> (add x, (setcc x, 0, eq))
+      if (((isOneConstant(FalseV) && LHS == TrueV &&
+            CCVal == ISD::CondCode::SETNE) ||
+           (isOneConstant(TrueV) && LHS == FalseV &&
+            CCVal == ISD::CondCode::SETEQ)) &&
+          isNullConstant(RHS)) {
+        // freeze it to be safe.
+        LHS = DAG.getFreeze(LHS);
+        SDValue C = DAG.getSetCC(DL, VT, LHS, RHS, ISD::CondCode::SETEQ);
+        return DAG.getNode(ISD::ADD, DL, VT, LHS, C);
+      }
     }
 
     return SDValue();
@@ -12668,9 +12806,10 @@ static bool CC_RISCV(const DataLayout &DL, RISCVABI::ABI ABI, unsigned ValNo,
     return false;
   }
 
-  // When a floating-point value is passed on the stack, no bit-conversion is
-  // needed.
-  if (ValVT.isFloatingPoint()) {
+  // When a scalar floating-point value is passed on the stack, no
+  // bit-conversion is needed.
+  if (ValVT.isFloatingPoint() && LocInfo != CCValAssign::Indirect) {
+    assert(!ValVT.isVector());
     LocVT = ValVT;
     LocInfo = CCValAssign::Full;
   }
@@ -13729,11 +13868,11 @@ bool RISCVTargetLowering::isUsedByReturnOnly(SDNode *N, SDValue &Chain) const {
     return false;
 
   SDNode *Copy = *N->use_begin();
-  
+
   if (Copy->getOpcode() == ISD::BITCAST) {
     return isUsedByReturnOnly(Copy, Chain);
   }
-  
+
   // TODO: Handle additional opcodes in order to support tail-calling libcalls
   // with soft float ABIs.
   if (Copy->getOpcode() != ISD::CopyToReg) {
@@ -13898,6 +14037,11 @@ const char *RISCVTargetLowering::getTargetNodeName(unsigned Opcode) const {
   NODE_NAME_CASE(VFCVT_RM_F_X_VL)
   NODE_NAME_CASE(FP_EXTEND_VL)
   NODE_NAME_CASE(FP_ROUND_VL)
+  NODE_NAME_CASE(STRICT_FADD_VL)
+  NODE_NAME_CASE(STRICT_FSUB_VL)
+  NODE_NAME_CASE(STRICT_FMUL_VL)
+  NODE_NAME_CASE(STRICT_FDIV_VL)
+  NODE_NAME_CASE(STRICT_FP_EXTEND_VL)
   NODE_NAME_CASE(VWMUL_VL)
   NODE_NAME_CASE(VWMULU_VL)
   NODE_NAME_CASE(VWMULSU_VL)
diff --git a/llvm/lib/Target/RISCV/RISCVISelLowering.h b/llvm/lib/Target/RISCV/RISCVISelLowering.h
index 5514a085e0472..b3a202476751d 100644
--- a/llvm/lib/Target/RISCV/RISCVISelLowering.h
+++ b/llvm/lib/Target/RISCV/RISCVISelLowering.h
@@ -330,6 +330,11 @@ enum NodeType : unsigned {
   // result being sign extended to 64 bit. These saturate out of range inputs.
   STRICT_FCVT_W_RV64 = ISD::FIRST_TARGET_STRICTFP_OPCODE,
   STRICT_FCVT_WU_RV64,
+  STRICT_FADD_VL,
+  STRICT_FSUB_VL,
+  STRICT_FMUL_VL,
+  STRICT_FDIV_VL,
+  STRICT_FP_EXTEND_VL,
 
   // WARNING: Do not add anything in the end unless you want the node to
   // have memop! In fact, starting from FIRST_TARGET_MEMORY_OPCODE all
@@ -388,6 +393,7 @@ class RISCVTargetLowering : public TargetLowering {
                           SmallVectorImpl<Use *> &Ops) const override;
   bool shouldScalarizeBinop(SDValue VecOp) const override;
   bool isOffsetFoldingLegal(const GlobalAddressSDNode *GA) const override;
+  bool isLegalZfaFPImm(const APFloat &Imm, EVT VT) const;
   bool isFPImmLegal(const APFloat &Imm, EVT VT,
                     bool ForCodeSize) const override;
   bool isExtractSubvectorCheap(EVT ResVT, EVT SrcVT,
@@ -485,6 +491,14 @@ class RISCVTargetLowering : public TargetLowering {
   EVT getSetCCResultType(const DataLayout &DL, LLVMContext &Context,
                          EVT VT) const override;
 
+  bool shouldFormOverflowOp(unsigned Opcode, EVT VT,
+                            bool MathUsed) const override {
+    if (VT == MVT::i8 || VT == MVT::i16)
+      return false;
+
+    return TargetLowering::shouldFormOverflowOp(Opcode, VT, MathUsed);
+  }
+
   bool convertSetCCLogicToBitwiseLogic(EVT VT) const override {
     return VT.isScalarInteger();
   }
@@ -758,6 +772,8 @@ class RISCVTargetLowering : public TargetLowering {
   SDValue lowerEH_DWARF_CFA(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerCTLZ_CTTZ_ZERO_UNDEF(SDValue Op, SelectionDAG &DAG) const;
 
+  SDValue lowerStrictFPExtend(SDValue Op, SelectionDAG &DAG) const;
+
   SDValue expandUnalignedRVVLoad(SDValue Op, SelectionDAG &DAG) const;
   SDValue expandUnalignedRVVStore(SDValue Op, SelectionDAG &DAG) const;
 
diff --git a/llvm/lib/Target/RISCV/RISCVInstrInfo.cpp b/llvm/lib/Target/RISCV/RISCVInstrInfo.cpp
index 7d060ad77186f..d525365762293 100644
--- a/llvm/lib/Target/RISCV/RISCVInstrInfo.cpp
+++ b/llvm/lib/Target/RISCV/RISCVInstrInfo.cpp
@@ -79,19 +79,33 @@ MCInst RISCVInstrInfo::getNop() const {
 
 unsigned RISCVInstrInfo::isLoadFromStackSlot(const MachineInstr &MI,
                                              int &FrameIndex) const {
+  unsigned Dummy;
+  return isLoadFromStackSlot(MI, FrameIndex, Dummy);
+}
+
+unsigned RISCVInstrInfo::isLoadFromStackSlot(const MachineInstr &MI,
+                                             int &FrameIndex,
+                                             unsigned &MemBytes) const {
   switch (MI.getOpcode()) {
   default:
     return 0;
   case RISCV::LB:
   case RISCV::LBU:
+    MemBytes = 1;
+    break;
   case RISCV::LH:
   case RISCV::LHU:
   case RISCV::FLH:
+    MemBytes = 2;
+    break;
   case RISCV::LW:
   case RISCV::FLW:
   case RISCV::LWU:
+    MemBytes = 4;
+    break;
   case RISCV::LD:
   case RISCV::FLD:
+    MemBytes = 8;
     break;
   }
 
@@ -106,16 +120,30 @@ unsigned RISCVInstrInfo::isLoadFromStackSlot(const MachineInstr &MI,
 
 unsigned RISCVInstrInfo::isStoreToStackSlot(const MachineInstr &MI,
                                             int &FrameIndex) const {
+  unsigned Dummy;
+  return isStoreToStackSlot(MI, FrameIndex, Dummy);
+}
+
+unsigned RISCVInstrInfo::isStoreToStackSlot(const MachineInstr &MI,
+                                            int &FrameIndex,
+                                            unsigned &MemBytes) const {
   switch (MI.getOpcode()) {
   default:
     return 0;
   case RISCV::SB:
+    MemBytes = 1;
+    break;
   case RISCV::SH:
-  case RISCV::SW:
   case RISCV::FSH:
+    MemBytes = 2;
+    break;
+  case RISCV::SW:
   case RISCV::FSW:
+    MemBytes = 4;
+    break;
   case RISCV::SD:
   case RISCV::FSD:
+    MemBytes = 8;
     break;
   }
 
@@ -1623,11 +1651,11 @@ bool RISCVInstrInfo::verifyInstruction(const MachineInstr &MI,
                                        StringRef &ErrInfo) const {
   MCInstrDesc const &Desc = MI.getDesc();
 
-  for (auto &OI : enumerate(Desc.operands())) {
-    unsigned OpType = OI.value().OperandType;
+  for (const auto &[Index, Operand] : enumerate(Desc.operands())) {
+    unsigned OpType = Operand.OperandType;
     if (OpType >= RISCVOp::OPERAND_FIRST_RISCV_IMM &&
         OpType <= RISCVOp::OPERAND_LAST_RISCV_IMM) {
-      const MachineOperand &MO = MI.getOperand(OI.index());
+      const MachineOperand &MO = MI.getOperand(Index);
       if (MO.isImm()) {
         int64_t Imm = MO.getImm();
         bool Ok;
diff --git a/llvm/lib/Target/RISCV/RISCVInstrInfo.h b/llvm/lib/Target/RISCV/RISCVInstrInfo.h
index 515a0071b2b62..cc84e9c07d0d6 100644
--- a/llvm/lib/Target/RISCV/RISCVInstrInfo.h
+++ b/llvm/lib/Target/RISCV/RISCVInstrInfo.h
@@ -51,8 +51,12 @@ class RISCVInstrInfo : public RISCVGenInstrInfo {
 
   unsigned isLoadFromStackSlot(const MachineInstr &MI,
                                int &FrameIndex) const override;
+  unsigned isLoadFromStackSlot(const MachineInstr &MI, int &FrameIndex,
+                               unsigned &MemBytes) const override;
   unsigned isStoreToStackSlot(const MachineInstr &MI,
                               int &FrameIndex) const override;
+  unsigned isStoreToStackSlot(const MachineInstr &MI, int &FrameIndex,
+                              unsigned &MemBytes) const override;
 
   void copyPhysReg(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
                    const DebugLoc &DL, MCRegister DstReg, MCRegister SrcReg,
diff --git a/llvm/lib/Target/RISCV/RISCVInstrInfoVSDPatterns.td b/llvm/lib/Target/RISCV/RISCVInstrInfoVSDPatterns.td
index 4af99d4669469..23fb5364ab165 100644
--- a/llvm/lib/Target/RISCV/RISCVInstrInfoVSDPatterns.td
+++ b/llvm/lib/Target/RISCV/RISCVInstrInfoVSDPatterns.td
@@ -72,7 +72,7 @@ multiclass VPatUSLoadStoreMaskSDNode<MTypeInfo m>
             (store_instr VR:$rs2, GPR:$rs1, m.AVL, m.Log2SEW)>;
 }
 
-class VPatBinarySDNode_VV<SDNode vop,
+class VPatBinarySDNode_VV<SDPatternOperator vop,
                           string instruction_name,
                           ValueType result_type,
                           ValueType op_type,
@@ -88,7 +88,7 @@ class VPatBinarySDNode_VV<SDNode vop,
                      op_reg_class:$rs2,
                      avl, sew)>;
 
-class VPatBinarySDNode_XI<SDNode vop,
+class VPatBinarySDNode_XI<SDPatternOperator vop,
                           string instruction_name,
                           string suffix,
                           ValueType result_type,
@@ -107,7 +107,7 @@ class VPatBinarySDNode_XI<SDNode vop,
                      xop_kind:$rs2,
                      avl, sew)>;
 
-multiclass VPatBinarySDNode_VV_VX<SDNode vop, string instruction_name> {
+multiclass VPatBinarySDNode_VV_VX<SDPatternOperator vop, string instruction_name> {
   foreach vti = AllIntegerVectors in {
     def : VPatBinarySDNode_VV<vop, instruction_name,
                               vti.Vector, vti.Vector, vti.Log2SEW,
@@ -119,7 +119,7 @@ multiclass VPatBinarySDNode_VV_VX<SDNode vop, string instruction_name> {
   }
 }
 
-multiclass VPatBinarySDNode_VV_VX_VI<SDNode vop, string instruction_name,
+multiclass VPatBinarySDNode_VV_VX_VI<SDPatternOperator vop, string instruction_name,
                                      Operand ImmType = simm5>
     : VPatBinarySDNode_VV_VX<vop, instruction_name> {
   foreach vti = AllIntegerVectors in {
@@ -131,7 +131,7 @@ multiclass VPatBinarySDNode_VV_VX_VI<SDNode vop, string instruction_name,
   }
 }
 
-class VPatBinarySDNode_VF<SDNode vop,
+class VPatBinarySDNode_VF<SDPatternOperator vop,
                           string instruction_name,
                           ValueType result_type,
                           ValueType vop_type,
@@ -148,7 +148,7 @@ class VPatBinarySDNode_VF<SDNode vop,
                      (xop_type xop_kind:$rs2),
                      avl, sew)>;
 
-multiclass VPatBinaryFPSDNode_VV_VF<SDNode vop, string instruction_name> {
+multiclass VPatBinaryFPSDNode_VV_VF<SDPatternOperator vop, string instruction_name> {
   foreach vti = AllFloatVectors in {
     def : VPatBinarySDNode_VV<vop, instruction_name,
                               vti.Vector, vti.Vector, vti.Log2SEW,
@@ -160,7 +160,7 @@ multiclass VPatBinaryFPSDNode_VV_VF<SDNode vop, string instruction_name> {
   }
 }
 
-multiclass VPatBinaryFPSDNode_R_VF<SDNode vop, string instruction_name> {
+multiclass VPatBinaryFPSDNode_R_VF<SDPatternOperator vop, string instruction_name> {
   foreach fvti = AllFloatVectors in
     def : Pat<(fvti.Vector (vop (fvti.Vector (SplatFPOp fvti.Scalar:$rs2)),
                                 (fvti.Vector fvti.RegClass:$rs1))),
@@ -826,18 +826,18 @@ foreach mti = AllMasks in {
 let Predicates = [HasVInstructionsAnyF] in {
 
 // 13.2. Vector Single-Width Floating-Point Add/Subtract Instructions
-defm : VPatBinaryFPSDNode_VV_VF<fadd, "PseudoVFADD">;
-defm : VPatBinaryFPSDNode_VV_VF<fsub, "PseudoVFSUB">;
-defm : VPatBinaryFPSDNode_R_VF<fsub, "PseudoVFRSUB">;
+defm : VPatBinaryFPSDNode_VV_VF<any_fadd, "PseudoVFADD">;
+defm : VPatBinaryFPSDNode_VV_VF<any_fsub, "PseudoVFSUB">;
+defm : VPatBinaryFPSDNode_R_VF<any_fsub, "PseudoVFRSUB">;
 
 // 13.3. Vector Widening Floating-Point Add/Subtract Instructions
 defm : VPatWidenBinaryFPSDNode_VV_VF_WV_WF<fadd, "PseudoVFWADD">;
 defm : VPatWidenBinaryFPSDNode_VV_VF_WV_WF<fsub, "PseudoVFWSUB">;
 
 // 13.4. Vector Single-Width Floating-Point Multiply/Divide Instructions
-defm : VPatBinaryFPSDNode_VV_VF<fmul, "PseudoVFMUL">;
-defm : VPatBinaryFPSDNode_VV_VF<fdiv, "PseudoVFDIV">;
-defm : VPatBinaryFPSDNode_R_VF<fdiv, "PseudoVFRDIV">;
+defm : VPatBinaryFPSDNode_VV_VF<any_fmul, "PseudoVFMUL">;
+defm : VPatBinaryFPSDNode_VV_VF<any_fdiv, "PseudoVFDIV">;
+defm : VPatBinaryFPSDNode_R_VF<any_fdiv, "PseudoVFRDIV">;
 
 // 13.5. Vector Widening Floating-Point Multiply Instructions
 defm : VPatWidenBinaryFPSDNode_VV_VF<fmul, "PseudoVFWMUL">;
diff --git a/llvm/lib/Target/RISCV/RISCVInstrInfoVVLPatterns.td b/llvm/lib/Target/RISCV/RISCVInstrInfoVVLPatterns.td
index d68f070d4d092..5087628749cb3 100644
--- a/llvm/lib/Target/RISCV/RISCVInstrInfoVVLPatterns.td
+++ b/llvm/lib/Target/RISCV/RISCVInstrInfoVVLPatterns.td
@@ -107,6 +107,24 @@ def riscv_fcopysign_vl : SDNode<"RISCVISD::FCOPYSIGN_VL", SDT_RISCVCopySign_VL>;
 def riscv_fminnum_vl   : SDNode<"RISCVISD::FMINNUM_VL",  SDT_RISCVFPBinOp_VL, [SDNPCommutative]>;
 def riscv_fmaxnum_vl   : SDNode<"RISCVISD::FMAXNUM_VL",  SDT_RISCVFPBinOp_VL, [SDNPCommutative]>;
 
+def riscv_strict_fadd_vl  : SDNode<"RISCVISD::STRICT_FADD_VL",  SDT_RISCVFPBinOp_VL, [SDNPCommutative, SDNPHasChain]>;
+def riscv_strict_fsub_vl  : SDNode<"RISCVISD::STRICT_FSUB_VL",  SDT_RISCVFPBinOp_VL, [SDNPHasChain]>;
+def riscv_strict_fmul_vl  : SDNode<"RISCVISD::STRICT_FMUL_VL",  SDT_RISCVFPBinOp_VL, [SDNPCommutative, SDNPHasChain]>;
+def riscv_strict_fdiv_vl  : SDNode<"RISCVISD::STRICT_FDIV_VL",  SDT_RISCVFPBinOp_VL, [SDNPHasChain]>;
+
+def any_riscv_fadd_vl : PatFrags<(ops node:$lhs, node:$rhs, node:$merge, node:$mask, node:$vl),
+                        [(riscv_fadd_vl node:$lhs, node:$rhs, node:$merge, node:$mask, node:$vl),
+                         (riscv_strict_fadd_vl node:$lhs, node:$rhs, node:$merge, node:$mask, node:$vl)]>;
+def any_riscv_fsub_vl : PatFrags<(ops node:$lhs, node:$rhs, node:$merge, node:$mask, node:$vl),
+                        [(riscv_fsub_vl node:$lhs, node:$rhs, node:$merge, node:$mask, node:$vl),
+                         (riscv_strict_fsub_vl node:$lhs, node:$rhs, node:$merge, node:$mask, node:$vl)]>;
+def any_riscv_fmul_vl : PatFrags<(ops node:$lhs, node:$rhs, node:$merge, node:$mask, node:$vl),
+                        [(riscv_fmul_vl node:$lhs, node:$rhs, node:$merge, node:$mask, node:$vl),
+                         (riscv_strict_fmul_vl node:$lhs, node:$rhs, node:$merge, node:$mask, node:$vl)]>;
+def any_riscv_fdiv_vl : PatFrags<(ops node:$lhs, node:$rhs, node:$merge, node:$mask, node:$vl),
+                        [(riscv_fdiv_vl node:$lhs, node:$rhs, node:$merge, node:$mask, node:$vl),
+                         (riscv_strict_fdiv_vl node:$lhs, node:$rhs, node:$merge, node:$mask, node:$vl)]>;
+
 def SDT_RISCVVecFMA_VL : SDTypeProfile<1, 5, [SDTCisSameAs<0, 1>,
                                               SDTCisSameAs<0, 2>,
                                               SDTCisSameAs<0, 3>,
@@ -130,8 +148,13 @@ def SDT_RISCVFPExtendOp_VL  : SDTypeProfile<1, 3, [
 
 def riscv_fpround_vl : SDNode<"RISCVISD::FP_ROUND_VL", SDT_RISCVFPRoundOp_VL>;
 def riscv_fpextend_vl : SDNode<"RISCVISD::FP_EXTEND_VL", SDT_RISCVFPExtendOp_VL>;
+def riscv_strict_fpextend_vl : SDNode<"RISCVISD::STRICT_FP_EXTEND_VL", SDT_RISCVFPExtendOp_VL, [SDNPHasChain]>;
 def riscv_fncvt_rod_vl : SDNode<"RISCVISD::VFNCVT_ROD_VL", SDT_RISCVFPRoundOp_VL>;
 
+def any_riscv_fpextend_vl : PatFrags<(ops node:$src, node:$mask, node:$vl),
+                            [(riscv_fpextend_vl node:$src, node:$mask, node:$vl),
+                             (riscv_strict_fpextend_vl node:$src, node:$mask, node:$vl)]>;
+
 def SDT_RISCVFP2IOp_VL  : SDTypeProfile<1, 3, [
   SDTCisInt<0>, SDTCisFP<1>, SDTCisSameNumEltsAs<0, 1>,
   SDTCVecEltisVT<2, i1>, SDTCisSameNumEltsAs<1, 2>, SDTCisVT<3, XLenVT>
@@ -416,7 +439,7 @@ def sew16simm5 : ComplexPattern<XLenVT, 1, "selectRVVSimm5<16>", []>;
 def sew32simm5 : ComplexPattern<XLenVT, 1, "selectRVVSimm5<32>", []>;
 def sew64simm5 : ComplexPattern<XLenVT, 1, "selectRVVSimm5<64>", []>;
 
-multiclass VPatBinaryVL_V<SDNode vop,
+multiclass VPatBinaryVL_V<SDPatternOperator vop,
                           string instruction_name,
                           string suffix,
                           ValueType result_type,
@@ -475,7 +498,7 @@ multiclass VPatTiedBinaryNoMaskVL_V<SDNode vop,
                      GPR:$vl, sew, TAIL_UNDISTURBED_MASK_UNDISTURBED)>;
 }
 
-multiclass VPatBinaryVL_XI<SDNode vop,
+multiclass VPatBinaryVL_XI<SDPatternOperator vop,
                            string instruction_name,
                            string suffix,
                            ValueType result_type,
@@ -501,7 +524,7 @@ multiclass VPatBinaryVL_XI<SDNode vop,
                      (mask_type V0), GPR:$vl, sew, TAIL_AGNOSTIC)>;
 }
 
-multiclass VPatBinaryVL_VV_VX<SDNode vop, string instruction_name> {
+multiclass VPatBinaryVL_VV_VX<SDPatternOperator vop, string instruction_name> {
   foreach vti = AllIntegerVectors in {
     defm : VPatBinaryVL_V<vop, instruction_name, "VV",
                            vti.Vector, vti.Vector, vti.Vector, vti.Mask,
@@ -514,7 +537,7 @@ multiclass VPatBinaryVL_VV_VX<SDNode vop, string instruction_name> {
   }
 }
 
-multiclass VPatBinaryVL_VV_VX_VI<SDNode vop, string instruction_name,
+multiclass VPatBinaryVL_VV_VX_VI<SDPatternOperator vop, string instruction_name,
                                  Operand ImmType = simm5>
     : VPatBinaryVL_VV_VX<vop, instruction_name> {
   foreach vti = AllIntegerVectors in {
@@ -526,7 +549,7 @@ multiclass VPatBinaryVL_VV_VX_VI<SDNode vop, string instruction_name,
   }
 }
 
-multiclass VPatBinaryWVL_VV_VX<SDNode vop, string instruction_name> {
+multiclass VPatBinaryWVL_VV_VX<SDPatternOperator vop, string instruction_name> {
   foreach VtiToWti = AllWidenableIntVectors in {
     defvar vti = VtiToWti.Vti;
     defvar wti = VtiToWti.Wti;
@@ -540,7 +563,7 @@ multiclass VPatBinaryWVL_VV_VX<SDNode vop, string instruction_name> {
                            SplatPat, GPR>;
   }
 }
-multiclass VPatBinaryWVL_VV_VX_WV_WX<SDNode vop, SDNode vop_w,
+multiclass VPatBinaryWVL_VV_VX_WV_WX<SDPatternOperator vop, SDNode vop_w,
                                      string instruction_name>
     : VPatBinaryWVL_VV_VX<vop, instruction_name> {
   foreach VtiToWti = AllWidenableIntVectors in {
@@ -560,7 +583,7 @@ multiclass VPatBinaryWVL_VV_VX_WV_WX<SDNode vop, SDNode vop_w,
   }
 }
 
-multiclass VPatBinaryNVL_WV_WX_WI<SDNode vop, string instruction_name> {
+multiclass VPatBinaryNVL_WV_WX_WI<SDPatternOperator vop, string instruction_name> {
   foreach VtiToWti = AllWidenableIntVectors in {
     defvar vti = VtiToWti.Vti;
     defvar wti = VtiToWti.Wti;
@@ -580,7 +603,7 @@ multiclass VPatBinaryNVL_WV_WX_WI<SDNode vop, string instruction_name> {
   }
 }
 
-multiclass VPatBinaryVL_VF<SDNode vop,
+multiclass VPatBinaryVL_VF<SDPatternOperator vop,
                            string instruction_name,
                            ValueType result_type,
                            ValueType vop_type,
@@ -602,7 +625,7 @@ multiclass VPatBinaryVL_VF<SDNode vop,
                      (mask_type V0), GPR:$vl, sew, TAIL_AGNOSTIC)>;
 }
 
-multiclass VPatBinaryFPVL_VV_VF<SDNode vop, string instruction_name> {
+multiclass VPatBinaryFPVL_VV_VF<SDPatternOperator vop, string instruction_name> {
   foreach vti = AllFloatVectors in {
     defm : VPatBinaryVL_V<vop, instruction_name, "VV",
                           vti.Vector, vti.Vector, vti.Vector, vti.Mask,
@@ -615,7 +638,7 @@ multiclass VPatBinaryFPVL_VV_VF<SDNode vop, string instruction_name> {
   }
 }
 
-multiclass VPatBinaryFPVL_R_VF<SDNode vop, string instruction_name> {
+multiclass VPatBinaryFPVL_R_VF<SDPatternOperator vop, string instruction_name> {
   foreach fvti = AllFloatVectors in {
     def : Pat<(fvti.Vector (vop (SplatFPOp fvti.ScalarRegClass:$rs2),
                                 fvti.RegClass:$rs1,
@@ -1607,18 +1630,18 @@ defm : VPatBinaryVL_VV_VX<riscv_usubsat_vl, "PseudoVSSUBU">;
 let Predicates = [HasVInstructionsAnyF] in {
 
 // 13.2. Vector Single-Width Floating-Point Add/Subtract Instructions
-defm : VPatBinaryFPVL_VV_VF<riscv_fadd_vl, "PseudoVFADD">;
-defm : VPatBinaryFPVL_VV_VF<riscv_fsub_vl, "PseudoVFSUB">;
-defm : VPatBinaryFPVL_R_VF<riscv_fsub_vl, "PseudoVFRSUB">;
+defm : VPatBinaryFPVL_VV_VF<any_riscv_fadd_vl, "PseudoVFADD">;
+defm : VPatBinaryFPVL_VV_VF<any_riscv_fsub_vl, "PseudoVFSUB">;
+defm : VPatBinaryFPVL_R_VF<any_riscv_fsub_vl, "PseudoVFRSUB">;
 
 // 13.3. Vector Widening Floating-Point Add/Subtract Instructions
 defm : VPatWidenBinaryFPVL_VV_VF_WV_WF<riscv_fadd_vl, "PseudoVFWADD">;
 defm : VPatWidenBinaryFPVL_VV_VF_WV_WF<riscv_fsub_vl, "PseudoVFWSUB">;
 
 // 13.4. Vector Single-Width Floating-Point Multiply/Divide Instructions
-defm : VPatBinaryFPVL_VV_VF<riscv_fmul_vl, "PseudoVFMUL">;
-defm : VPatBinaryFPVL_VV_VF<riscv_fdiv_vl, "PseudoVFDIV">;
-defm : VPatBinaryFPVL_R_VF<riscv_fdiv_vl, "PseudoVFRDIV">;
+defm : VPatBinaryFPVL_VV_VF<any_riscv_fmul_vl, "PseudoVFMUL">;
+defm : VPatBinaryFPVL_VV_VF<any_riscv_fdiv_vl, "PseudoVFDIV">;
+defm : VPatBinaryFPVL_R_VF<any_riscv_fdiv_vl, "PseudoVFRDIV">;
 
 // 13.5. Vector Widening Floating-Point Multiply Instructions
 defm : VPatWidenBinaryFPVL_VV_VF<riscv_fmul_vl, riscv_fpextend_vl_oneuse, "PseudoVFWMUL">;
@@ -1844,9 +1867,10 @@ foreach fvti = AllFloatVectors in {
   foreach fvtiToFWti = AllWidenableFloatVectors in {
     defvar fvti = fvtiToFWti.Vti;
     defvar fwti = fvtiToFWti.Wti;
-    def : Pat<(fwti.Vector (riscv_fpextend_vl (fvti.Vector fvti.RegClass:$rs1),
-                                              (fvti.Mask V0),
-                                              VLOpFrag)),
+    def : Pat<(fwti.Vector (any_riscv_fpextend_vl
+                               (fvti.Vector fvti.RegClass:$rs1),
+                               (fvti.Mask V0),
+                               VLOpFrag)),
               (!cast<Instruction>("PseudoVFWCVT_F_F_V_"#fvti.LMul.MX#"_MASK")
                   (fwti.Vector (IMPLICIT_DEF)), fvti.RegClass:$rs1,
                   (fvti.Mask V0), GPR:$vl, fvti.Log2SEW, TA_MA)>;
diff --git a/llvm/lib/Target/RISCV/RISCVInstrInfoZfa.td b/llvm/lib/Target/RISCV/RISCVInstrInfoZfa.td
index 845718f8a1d4e..28348b14a5ef1 100644
--- a/llvm/lib/Target/RISCV/RISCVInstrInfoZfa.td
+++ b/llvm/lib/Target/RISCV/RISCVInstrInfoZfa.td
@@ -17,6 +17,29 @@
 // Operand and SDNode transformation definitions.
 //===----------------------------------------------------------------------===//
 
+// 5-bit floating-point immediate encodings.
+def LoadFPImmOperand : AsmOperandClass {
+  let Name = "LoadFPImm";
+  let ParserMethod = "parseFPImm";
+  let RenderMethod = "addFPImmOperands";
+  let DiagnosticType = "InvalidLoadFPImm";
+}
+
+def loadfp32imm : Operand<f32> {
+  let ParserMatchClass = LoadFPImmOperand;
+  let PrintMethod = "printFPImmOperand";
+}
+
+def loadfp64imm : Operand<f64> {
+  let ParserMatchClass = LoadFPImmOperand;
+  let PrintMethod = "printFPImmOperand";
+}
+
+def loadfp16imm : Operand<f16> {
+  let ParserMatchClass = LoadFPImmOperand;
+  let PrintMethod = "printFPImmOperand";
+}
+
 def RTZArg : AsmOperandClass {
   let Name = "RTZArg";
   let RenderMethod = "addFRMArgOperands";
@@ -35,11 +58,26 @@ def rtzarg : Operand<XLenVT> {
 //===----------------------------------------------------------------------===//
 
 let hasSideEffects = 0, mayLoad = 0, mayStore = 0, mayRaiseFPException = 1 in
-class FPBinaryOp_rr<bits<7> funct7, bits<3> funct3, DAGOperand rdty, 
+class FPBinaryOp_rr<bits<7> funct7, bits<3> funct3, DAGOperand rdty,
                     DAGOperand rsty, string opcodestr>
-    : RVInstR<funct7, funct3, OPC_OP_FP, (outs rdty:$rd), 
+    : RVInstR<funct7, funct3, OPC_OP_FP, (outs rdty:$rd),
               (ins rsty:$rs1, rsty:$rs2), opcodestr, "$rd, $rs1, $rs2">;
 
+let hasSideEffects = 0, mayLoad = 0, mayStore = 0, mayRaiseFPException = 1 in
+class FPUnaryOp_imm<bits<7> funct7, bits<5> rs2val, bits<3> funct3, RISCVOpcode opcode,
+                    dag outs, dag ins, string opcodestr, string argstr>
+    : RVInst<outs, ins, opcodestr, argstr, [], InstFormatI> {
+  bits<5> imm;
+  bits<5> rd;
+
+  let Inst{31-25} = funct7;
+  let Inst{24-20} = rs2val;
+  let Inst{19-15} = imm;
+  let Inst{14-12} = funct3;
+  let Inst{11-7} = rd;
+  let Opcode = opcode.Value;
+}
+
 let hasSideEffects = 0, mayLoad = 0, mayStore = 0, mayRaiseFPException = 1,
     UseNamedOperandTable = 1, hasPostISelHook = 1 in
 class FPUnaryOp_r_rtz<bits<7> funct7, bits<5> rs2val, DAGOperand rdty,
@@ -55,6 +93,10 @@ class FPUnaryOp_r_rtz<bits<7> funct7, bits<5> rs2val, DAGOperand rdty,
 //===----------------------------------------------------------------------===//
 
 let Predicates = [HasStdExtZfa] in {
+let isReMaterializable = 1, isAsCheapAsAMove = 1 in
+def FLI_S : FPUnaryOp_imm<0b1111000, 0b00001, 0b000, OPC_OP_FP, (outs FPR32:$rd),
+            (ins loadfp32imm:$imm), "fli.s", "$rd, $imm">;
+
 def FMINM_S: FPALU_rr<0b0010100, 0b010, "fminm.s", FPR32, /*Commutable*/ 1>;
 def FMAXM_S: FPALU_rr<0b0010100, 0b011, "fmaxm.s", FPR32, /*Commutable*/ 1>;
 
@@ -66,6 +108,10 @@ def FLEQ_S : FPCmp_rr<0b1010000, 0b100, "fleq.s", FPR32>;
 } // Predicates = [HasStdExtZfa]
 
 let Predicates = [HasStdExtZfa, HasStdExtD] in {
+let isReMaterializable = 1, isAsCheapAsAMove = 1 in
+def FLI_D : FPUnaryOp_imm<0b1111001, 0b00001, 0b000, OPC_OP_FP, (outs FPR64:$rd),
+            (ins loadfp64imm:$imm), "fli.d", "$rd, $imm">;
+
 def FMINM_D: FPALU_rr<0b0010101, 0b010, "fminm.d", FPR64, /*Commutable*/ 1>;
 def FMAXM_D: FPALU_rr<0b0010101, 0b011, "fmaxm.d", FPR64, /*Commutable*/ 1>;
 
@@ -91,6 +137,11 @@ def FMV_X_W_FPR64 : FPUnaryOp_r<0b1110000, 0b00000, 0b000, GPR, FPR64,
                     Sched<[WriteFMovF32ToI32, ReadFMovF32ToI32]>;
 } // Predicates = [HasStdExtZfa, HasStdExtD, IsRV32]
 
+let Predicates = [HasStdExtZfa, HasStdExtZfhOrZvfh] in
+let isReMaterializable = 1, isAsCheapAsAMove = 1 in
+def FLI_H : FPUnaryOp_imm<0b1111010, 0b00001, 0b000, OPC_OP_FP, (outs FPR16:$rd),
+            (ins loadfp16imm:$imm), "fli.h", "$rd, $imm">;
+
 let Predicates = [HasStdExtZfa, HasStdExtZfh] in {
 def FMINM_H: FPALU_rr<0b0010110, 0b010, "fminm.h", FPR16, /*Commutable*/ 1>;
 def FMAXM_H: FPALU_rr<0b0010110, 0b011, "fmaxm.h", FPR16, /*Commutable*/ 1>;
@@ -131,7 +182,14 @@ def : InstAlias<"fgeq.h $rd, $rs, $rt",
 // Codegen patterns
 //===----------------------------------------------------------------------===//
 
+def fpimm_to_loadfpimm : SDNodeXForm<fpimm, [{
+  return CurDAG->getTargetConstant(RISCVLoadFPImm::getLoadFPImm(N->getValueAPF()),
+                                   SDLoc(N), Subtarget->getXLenVT());}]>;
+
+
 let Predicates = [HasStdExtZfa] in {
+def : Pat<(f32 fpimm:$imm), (FLI_S (fpimm_to_loadfpimm fpimm:$imm))>;
+
 def: PatFprFpr<fminimum, FMINM_S, FPR32>;
 def: PatFprFpr<fmaximum, FMAXM_S, FPR32>;
 
@@ -154,6 +212,8 @@ def: PatSetCC<FPR32, strict_fsetcc, SETOLE, FLEQ_S>;
 } // Predicates = [HasStdExtZfa]
 
 let Predicates = [HasStdExtZfa, HasStdExtD] in {
+def : Pat<(f64 fpimm:$imm), (FLI_D (fpimm_to_loadfpimm fpimm:$imm))>;
+
 def: PatFprFpr<fminimum, FMINM_D, FPR64>;
 def: PatFprFpr<fmaximum, FMAXM_D, FPR64>;
 
@@ -182,6 +242,8 @@ def : Pat<(RISCVBuildPairF64 GPR:$rs1, GPR:$rs2),
 }
 
 let Predicates = [HasStdExtZfa, HasStdExtZfh] in {
+def : Pat<(f16 fpimm:$imm), (FLI_H (fpimm_to_loadfpimm fpimm:$imm))>;
+
 def: PatFprFpr<fminimum, FMINM_H, FPR16>;
 def: PatFprFpr<fmaximum, FMAXM_H, FPR16>;
 
diff --git a/llvm/lib/Target/RISCV/RISCVScheduleV.td b/llvm/lib/Target/RISCV/RISCVScheduleV.td
index ad3d4cdcbfdf6..faf690ae46fd5 100644
--- a/llvm/lib/Target/RISCV/RISCVScheduleV.td
+++ b/llvm/lib/Target/RISCV/RISCVScheduleV.td
@@ -45,12 +45,29 @@ multiclass LMULReadAdvanceImpl<string name, int val,
   }
 }
 
+// Define classes to define list containing all SchedWrites for each (name, LMUL)
+// pair for each LMUL in each of the SchedMxList variants above and name in
+// argument `names`. These classes can be used to construct a list of existing
+// definitions of writes corresponding to each (name, LMUL) pair, that are needed
+// by the ReadAdvance. For example:
+// ```
+//   defm "" : LMULReadAdvance<"ReadVIALUX", 1,
+//                             LMULSchedWriteList<["WriteVIMovVX"]>.value>;
+// ```
+class LMULSchedWriteListImpl<list<string> names, list<string> MxList> {
+  list<SchedWrite> value = !foldl([]<SchedWrite>,
+                                  !foreach(name, names,
+                                    !foreach(mx, MxList, !cast<SchedWrite>(name # "_" # mx))),
+                                  all, writes, !listconcat(all, writes));
+}
+
 multiclass LMULSchedWrites<string name> : LMULSchedWritesImpl<name, SchedMxList>;
 multiclass LMULSchedReads<string name> : LMULSchedReadsImpl<name, SchedMxList>;
 multiclass LMULWriteRes<string name, list<ProcResourceKind> resources>
   : LMULWriteResImpl<name, resources>;
 multiclass LMULReadAdvance<string name, int val, list<SchedWrite> writes = []>
   : LMULReadAdvanceImpl<name, val, writes>;
+class LMULSchedWriteList<list<string> names> : LMULSchedWriteListImpl<names, SchedMxList>;
 
 multiclass LMULSchedWritesW<string name> : LMULSchedWritesImpl<name, SchedMxListW>;
 multiclass LMULSchedReadsW<string name> : LMULSchedReadsImpl<name, SchedMxListW>;
@@ -58,6 +75,7 @@ multiclass LMULWriteResW<string name, list<ProcResourceKind> resources>
   : LMULWriteResImpl<name, resources>;
 multiclass LMULReadAdvanceW<string name, int val, list<SchedWrite> writes = []>
   : LMULReadAdvanceImpl<name, val, writes>;
+class LMULSchedWriteListW<list<string> names> : LMULSchedWriteListImpl<names, SchedMxListW>;
 
 multiclass LMULSchedWritesFW<string name> : LMULSchedWritesImpl<name, SchedMxListFW>;
 multiclass LMULSchedReadsFW<string name> : LMULSchedReadsImpl<name, SchedMxListFW>;
@@ -65,10 +83,12 @@ multiclass LMULWriteResFW<string name, list<ProcResourceKind> resources>
   : LMULWriteResImpl<name, resources>;
 multiclass LMULReadAdvanceFW<string name, int val, list<SchedWrite> writes = []>
   : LMULReadAdvanceImpl<name, val, writes>;
+class LMULSchedWriteListFW<list<string> names> : LMULSchedWriteListImpl<names, SchedMxListFW>;
 
 multiclass LMULSchedWritesFWRed<string name> : LMULSchedWritesImpl<name, SchedMxListFWRed>;
 multiclass LMULWriteResFWRed<string name, list<ProcResourceKind> resources>
   : LMULWriteResImpl<name, resources>;
+class LMULSchedWriteListFWRed<list<string> names> : LMULSchedWriteListImpl<names, SchedMxListFWRed>;
 
 
 // 3.6 Vector Byte Length vlenb
diff --git a/llvm/lib/Target/RISCV/RISCVStripWSuffix.cpp b/llvm/lib/Target/RISCV/RISCVStripWSuffix.cpp
index 14ab9c2dd6557..8324f16497413 100644
--- a/llvm/lib/Target/RISCV/RISCVStripWSuffix.cpp
+++ b/llvm/lib/Target/RISCV/RISCVStripWSuffix.cpp
@@ -69,16 +69,18 @@ bool RISCVStripWSuffix::runOnMachineFunction(MachineFunction &MF) {
     for (auto I = MBB.begin(), IE = MBB.end(); I != IE; ++I) {
       MachineInstr &MI = *I;
 
+      unsigned Opc;
       switch (MI.getOpcode()) {
-      case RISCV::ADDW:
-      case RISCV::SLLIW:
-        if (TII.hasAllWUsers(MI, MRI)) {
-          unsigned Opc =
-              MI.getOpcode() == RISCV::ADDW ? RISCV::ADD : RISCV::SLLI;
-          MI.setDesc(TII.get(Opc));
-          MadeChange = true;
-        }
-        break;
+      default:
+        continue;
+      case RISCV::ADDW:  Opc = RISCV::ADD;  break;
+      case RISCV::MULW:  Opc = RISCV::MUL;  break;
+      case RISCV::SLLIW: Opc = RISCV::SLLI; break;
+      }
+
+      if (TII.hasAllWUsers(MI, MRI)) {
+        MI.setDesc(TII.get(Opc));
+        MadeChange = true;
       }
     }
   }
diff --git a/llvm/lib/Target/RISCV/RISCVSubtarget.cpp b/llvm/lib/Target/RISCV/RISCVSubtarget.cpp
index a6700fa3f4ea7..47b47271cc9be 100644
--- a/llvm/lib/Target/RISCV/RISCVSubtarget.cpp
+++ b/llvm/lib/Target/RISCV/RISCVSubtarget.cpp
@@ -30,7 +30,7 @@ using namespace llvm;
 #include "RISCVGenSubtargetInfo.inc"
 
 static cl::opt<bool> EnableSubRegLiveness("riscv-enable-subreg-liveness",
-                                          cl::init(false), cl::Hidden);
+                                          cl::init(true), cl::Hidden);
 
 static cl::opt<unsigned> RVVVectorLMULMax(
     "riscv-v-fixed-length-vector-lmul-max",
diff --git a/llvm/lib/Target/RISCV/RISCVSubtarget.h b/llvm/lib/Target/RISCV/RISCVSubtarget.h
index 5e8acffb6fbcf..6a8e18cd1594c 100644
--- a/llvm/lib/Target/RISCV/RISCVSubtarget.h
+++ b/llvm/lib/Target/RISCV/RISCVSubtarget.h
@@ -114,7 +114,7 @@ class RISCVSubtarget : public RISCVGenSubtargetInfo {
   bool hasStdExtCOrZca() const { return HasStdExtC || HasStdExtZca; }
   bool hasStdExtZvl() const { return ZvlLen != 0; }
   bool hasStdExtZfhOrZfhmin() const { return HasStdExtZfh || HasStdExtZfhmin; }
-  bool is64Bit() const { return HasRV64; }
+  bool is64Bit() const { return IsRV64; }
   MVT getXLenVT() const { return XLenVT; }
   unsigned getXLen() const { return XLen; }
   unsigned getFLen() const {
diff --git a/llvm/lib/Target/RISCV/RISCVTargetMachine.cpp b/llvm/lib/Target/RISCV/RISCVTargetMachine.cpp
index 5fcb94c6c4fec..529256294ed3a 100644
--- a/llvm/lib/Target/RISCV/RISCVTargetMachine.cpp
+++ b/llvm/lib/Target/RISCV/RISCVTargetMachine.cpp
@@ -66,6 +66,11 @@ static cl::opt<int> RVVVectorBitsMinOpt(
              "autovectorization with fixed width vectors."),
     cl::init(-1), cl::Hidden);
 
+static cl::opt<bool> EnableRISCVCopyPropagation(
+    "riscv-enable-copy-propagation",
+    cl::desc("Enable the copy propagation with RISCV copy instr"),
+    cl::init(true), cl::Hidden);
+
 extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeRISCVTarget() {
   RegisterTargetMachine<RISCVTargetMachine> X(getTheRISCV32Target());
   RegisterTargetMachine<RISCVTargetMachine> Y(getTheRISCV64Target());
@@ -332,10 +337,19 @@ void RISCVPassConfig::addPreSched2() {}
 void RISCVPassConfig::addPreEmitPass() {
   addPass(&BranchRelaxationPassID);
   addPass(createRISCVMakeCompressibleOptPass());
+
+  // TODO: It would potentially be better to schedule copy propagation after
+  // expanding pseudos (in addPreEmitPass2). However, performing copy
+  // propagation after the machine outliner (which runs after addPreEmitPass)
+  // currently leads to incorrect code-gen, where copies to registers within
+  // outlined functions are removed erroneously.
+  if (TM->getOptLevel() >= CodeGenOpt::Default && EnableRISCVCopyPropagation)
+    addPass(createMachineCopyPropagationPass(true));
 }
 
 void RISCVPassConfig::addPreEmitPass2() {
   addPass(createRISCVExpandPseudoPass());
+
   // Schedule the expansion of AMOs at the last possible moment, avoiding the
   // possibility for other passes to break the requirements for forward
   // progress in the LR/SC block.
diff --git a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
index 6ae7b5ee1dbf4..a6a216392de74 100644
--- a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
+++ b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
@@ -8,10 +8,12 @@
 
 #include "RISCVTargetTransformInfo.h"
 #include "MCTargetDesc/RISCVMatInt.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/CodeGen/BasicTTIImpl.h"
 #include "llvm/CodeGen/CostTable.h"
 #include "llvm/CodeGen/TargetLowering.h"
+#include "llvm/IR/Instructions.h"
 #include <cmath>
 #include <optional>
 using namespace llvm;
@@ -241,52 +243,56 @@ RISCVTTIImpl::getRegisterBitWidth(TargetTransformInfo::RegisterKind K) const {
   llvm_unreachable("Unsupported register kind");
 }
 
-InstructionCost RISCVTTIImpl::getSpliceCost(VectorType *Tp, int Index) {
-  std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(Tp);
-
-  unsigned Cost = 2; // vslidedown+vslideup.
-  // TODO: Multiplying by LT.first implies this legalizes into multiple copies
-  // of similar code, but I think we expand through memory.
-  return Cost * LT.first * getLMULCost(LT.second);
-}
-
 InstructionCost RISCVTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
                                              VectorType *Tp, ArrayRef<int> Mask,
                                              TTI::TargetCostKind CostKind,
                                              int Index, VectorType *SubTp,
                                              ArrayRef<const Value *> Args) {
-  if (isa<ScalableVectorType>(Tp)) {
-    std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(Tp);
+  std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(Tp);
+
+  // First, handle cases where having a fixed length vector enables us to
+  // give a more accurate cost than falling back to generic scalable codegen.
+  // TODO: Each of these cases hints at a modeling gap around scalable vectors.
+  if (isa<FixedVectorType>(Tp)) {
     switch (Kind) {
     default:
-      // Fallthrough to generic handling.
-      // TODO: Most of these cases will return getInvalid in generic code, and
-      // must be implemented here.
       break;
-    case TTI::SK_Broadcast: {
-      return LT.first * 1;
+    case TTI::SK_PermuteSingleSrc: {
+      if (Mask.size() >= 2 && LT.second.isFixedLengthVector()) {
+        MVT EltTp = LT.second.getVectorElementType();
+        // If the size of the element is < ELEN then shuffles of interleaves and
+        // deinterleaves of 2 vectors can be lowered into the following
+        // sequences
+        if (EltTp.getScalarSizeInBits() < ST->getELEN()) {
+          // Example sequence:
+          //   vsetivli     zero, 4, e8, mf4, ta, ma (ignored)
+          //   vwaddu.vv    v10, v8, v9
+          //   li       a0, -1                   (ignored)
+          //   vwmaccu.vx   v10, a0, v9
+          if (ShuffleVectorInst::isInterleaveMask(Mask, 2, Mask.size()))
+            return 2 * LT.first * getLMULCost(LT.second);
+
+          if (Mask[0] == 0 || Mask[0] == 1) {
+            auto DeinterleaveMask = createStrideMask(Mask[0], 2, Mask.size());
+            // Example sequence:
+            //   vnsrl.wi   v10, v8, 0
+            if (equal(DeinterleaveMask, Mask))
+              return LT.first * getLMULCost(LT.second);
+          }
+        }
+      }
     }
-    case TTI::SK_Splice:
-      return getSpliceCost(Tp, Index);
-    case TTI::SK_Reverse:
-      // Most of the cost here is producing the vrgather index register
-      // Example sequence:
-      //   csrr a0, vlenb
-      //   srli a0, a0, 3
-      //   addi a0, a0, -1
-      //   vsetvli a1, zero, e8, mf8, ta, mu (ignored)
-      //   vid.v v9
-      //   vrsub.vx v10, v9, a0
-      //   vrgather.vv v9, v8, v10
-      if (Tp->getElementType()->isIntegerTy(1))
-        // Mask operation additionally required extend and truncate
-        return LT.first * 9;
-      return LT.first * 6;
     }
-  }
+  };
 
-  if (isa<FixedVectorType>(Tp) && Kind == TargetTransformInfo::SK_Broadcast) {
-    std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(Tp);
+  // Handle scalable vectors (and fixed vectors legalized to scalable vectors).
+  switch (Kind) {
+  default:
+    // Fallthrough to generic handling.
+    // TODO: Most of these cases will return getInvalid in generic code, and
+    // must be implemented here.
+    break;
+  case TTI::SK_Broadcast: {
     bool HasScalar = (Args.size() > 0) && (Operator::getOpcode(Args[0]) ==
                                            Instruction::InsertElement);
     if (LT.second.getScalarSizeInBits() == 1) {
@@ -322,7 +328,36 @@ InstructionCost RISCVTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
     // implementation-dependent.
     return LT.first * getLMULCost(LT.second);
   }
-
+  case TTI::SK_Splice:
+    // vslidedown+vslideup.
+    // TODO: Multiplying by LT.first implies this legalizes into multiple copies
+    // of similar code, but I think we expand through memory.
+    return 2 * LT.first * getLMULCost(LT.second);
+  case TTI::SK_Reverse: {
+    // TODO: Cases to improve here:
+    // * LMUL > 1
+    // * i64 on RV32
+    // * i1 vector
+
+    // Most of the cost here is producing the vrgather index register
+    // Example sequence:
+    //   csrr a0, vlenb
+    //   srli a0, a0, 3
+    //   addi a0, a0, -1
+    //   vsetvli a1, zero, e8, mf8, ta, mu (ignored)
+    //   vid.v v9
+    //   vrsub.vx v10, v9, a0
+    //   vrgather.vv v9, v8, v10
+    unsigned LenCost = 3;
+    if (LT.second.isFixedLengthVector())
+      // vrsub.vi has a 5 bit immediate field, otherwise an li suffices
+      LenCost = isInt<5>(LT.second.getVectorNumElements() - 1) ? 0 : 1;
+    if (Tp->getElementType()->isIntegerTy(1))
+      // Mask operation additionally required extend and truncate
+      return LT.first * (LenCost + 6);
+    return LT.first * (LenCost + 3);
+  }
+  }
   return BaseT::getShuffleCost(Kind, Tp, Mask, CostKind, Index, SubTp);
 }
 
@@ -338,6 +373,59 @@ RISCVTTIImpl::getMaskedMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
   return getMemoryOpCost(Opcode, Src, Alignment, AddressSpace, CostKind);
 }
 
+InstructionCost RISCVTTIImpl::getInterleavedMemoryOpCost(
+    unsigned Opcode, Type *VecTy, unsigned Factor, ArrayRef<unsigned> Indices,
+    Align Alignment, unsigned AddressSpace, TTI::TargetCostKind CostKind,
+    bool UseMaskForCond, bool UseMaskForGaps) {
+  auto *FVTy = cast<FixedVectorType>(VecTy);
+  InstructionCost MemCost =
+      getMemoryOpCost(Opcode, VecTy, Alignment, AddressSpace, CostKind);
+  unsigned VF = FVTy->getNumElements() / Factor;
+
+  // An interleaved load will look like this for Factor=3:
+  // %wide.vec = load <12 x i32>, ptr %3, align 4
+  // %strided.vec = shufflevector %wide.vec, poison, <4 x i32> <stride mask>
+  // %strided.vec1 = shufflevector %wide.vec, poison, <4 x i32> <stride mask>
+  // %strided.vec2 = shufflevector %wide.vec, poison, <4 x i32> <stride mask>
+  if (Opcode == Instruction::Load) {
+    InstructionCost Cost = MemCost;
+    for (unsigned Index : Indices) {
+      FixedVectorType *SubVecTy =
+          FixedVectorType::get(FVTy->getElementType(), VF);
+      auto Mask = createStrideMask(Index, Factor, VF);
+      InstructionCost ShuffleCost =
+          getShuffleCost(TTI::ShuffleKind::SK_PermuteSingleSrc, SubVecTy, Mask,
+                         CostKind, 0, nullptr, {});
+      Cost += ShuffleCost;
+    }
+    return Cost;
+  }
+
+  // TODO: Model for NF > 2
+  // We'll need to enhance getShuffleCost to model shuffles that are just
+  // inserts and extracts into subvectors, since they won't have the full cost
+  // of a vrgather.
+  // An interleaved store for 3 vectors of 4 lanes will look like
+  // %11 = shufflevector <4 x i32> %4, <4 x i32> %6, <8 x i32> <0...7>
+  // %12 = shufflevector <4 x i32> %9, <4 x i32> poison, <8 x i32> <0...3>
+  // %13 = shufflevector <8 x i32> %11, <8 x i32> %12, <12 x i32> <0...11>
+  // %interleaved.vec = shufflevector %13, poison, <12 x i32> <interleave mask>
+  // store <12 x i32> %interleaved.vec, ptr %10, align 4
+  if (Factor != 2)
+    return BaseT::getInterleavedMemoryOpCost(Opcode, VecTy, Factor, Indices,
+                                             Alignment, AddressSpace, CostKind,
+                                             UseMaskForCond, UseMaskForGaps);
+
+  assert(Opcode == Instruction::Store && "Opcode must be a store");
+  // For an interleaving store of 2 vectors, we perform one large interleaving
+  // shuffle that goes into the wide store
+  auto Mask = createInterleaveMask(VF, Factor);
+  InstructionCost ShuffleCost =
+      getShuffleCost(TTI::ShuffleKind::SK_PermuteSingleSrc, FVTy, Mask,
+                     CostKind, 0, nullptr, {});
+  return MemCost + ShuffleCost;
+}
+
 InstructionCost RISCVTTIImpl::getGatherScatterOpCost(
     unsigned Opcode, Type *DataTy, const Value *Ptr, bool VariableMask,
     Align Alignment, TTI::TargetCostKind CostKind, const Instruction *I) {
@@ -840,7 +928,9 @@ RISCVTTIImpl::getIntrinsicInstrCost(const IntrinsicCostAttributes &ICA,
   case Intrinsic::sadd_sat:
   case Intrinsic::ssub_sat:
   case Intrinsic::uadd_sat:
-  case Intrinsic::usub_sat: {
+  case Intrinsic::usub_sat:
+  case Intrinsic::fabs:
+  case Intrinsic::sqrt: {
     auto LT = getTypeLegalizationCost(RetTy);
     if (ST->hasVInstructions() && LT.second.isVector())
       return LT.first;
@@ -855,13 +945,6 @@ RISCVTTIImpl::getIntrinsicInstrCost(const IntrinsicCostAttributes &ICA,
     }
     break;
   }
-  case Intrinsic::fabs:
-  case Intrinsic::sqrt: {
-    auto LT = getTypeLegalizationCost(RetTy);
-    if (ST->hasVInstructions() && LT.second.isVector())
-      return LT.first;
-    break;
-  }
   // TODO: add more intrinsic
   case Intrinsic::experimental_stepvector: {
     unsigned Cost = 1; // vid
diff --git a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h
index 4785e8bf30f82..2067140f564c6 100644
--- a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h
+++ b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h
@@ -75,6 +75,7 @@ class RISCVTTIImpl : public BasicTTIImplBase<RISCVTTIImpl> {
 
   bool shouldExpandReduction(const IntrinsicInst *II) const;
   bool supportsScalableVectors() const { return ST->hasVInstructions(); }
+  bool enableOrderedReductions() const { return true; }
   bool enableScalableVectorization() const { return ST->hasVInstructions(); }
   TailFoldingStyle
   getPreferredTailFoldingStyle(bool IVUpdateMayOverflow) const {
@@ -112,7 +113,6 @@ class RISCVTTIImpl : public BasicTTIImplBase<RISCVTTIImpl> {
     return ST->useRVVForFixedLengthVectors() ? 16 : 0;
   }
 
-  InstructionCost getSpliceCost(VectorType *Tp, int Index);
   InstructionCost getShuffleCost(TTI::ShuffleKind Kind, VectorType *Tp,
                                  ArrayRef<int> Mask,
                                  TTI::TargetCostKind CostKind, int Index,
@@ -122,6 +122,11 @@ class RISCVTTIImpl : public BasicTTIImplBase<RISCVTTIImpl> {
   InstructionCost getIntrinsicInstrCost(const IntrinsicCostAttributes &ICA,
                                         TTI::TargetCostKind CostKind);
 
+  InstructionCost getInterleavedMemoryOpCost(
+      unsigned Opcode, Type *VecTy, unsigned Factor, ArrayRef<unsigned> Indices,
+      Align Alignment, unsigned AddressSpace, TTI::TargetCostKind CostKind,
+      bool UseMaskForCond = false, bool UseMaskForGaps = false);
+
   InstructionCost getGatherScatterOpCost(unsigned Opcode, Type *DataTy,
                                          const Value *Ptr, bool VariableMask,
                                          Align Alignment,
@@ -280,6 +285,8 @@ class RISCVTTIImpl : public BasicTTIImplBase<RISCVTTIImpl> {
     return VF.isScalar() ? 1 : ST->getMaxInterleaveFactor();
   }
 
+  bool enableInterleavedAccessVectorization() { return true; }
+
   enum RISCVRegisterClass { GPRRC, FPRRC, VRRC };
   unsigned getNumberOfRegisters(unsigned ClassID) const {
     switch (ClassID) {
diff --git a/llvm/lib/Target/SPIRV/SPIRVBuiltins.cpp b/llvm/lib/Target/SPIRV/SPIRVBuiltins.cpp
index a69e8c4612823..40b652057e87f 100644
--- a/llvm/lib/Target/SPIRV/SPIRVBuiltins.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVBuiltins.cpp
@@ -291,6 +291,7 @@ buildBoolRegister(MachineIRBuilder &MIRBuilder, const SPIRVType *ResultType,
 
   Register ResultRegister =
       MIRBuilder.getMRI()->createGenericVirtualRegister(Type);
+  MIRBuilder.getMRI()->setRegClass(ResultRegister, &SPIRV::IDRegClass);
   GR->assignSPIRVTypeToVReg(BoolType, ResultRegister, MIRBuilder.getMF());
   return std::make_tuple(ResultRegister, BoolType);
 }
@@ -305,7 +306,7 @@ static bool buildSelectInst(MachineIRBuilder &MIRBuilder,
 
   if (ReturnType->getOpcode() == SPIRV::OpTypeVector) {
     unsigned Bits = GR->getScalarOrVectorBitWidth(ReturnType);
-    uint64_t AllOnes = APInt::getAllOnesValue(Bits).getZExtValue();
+    uint64_t AllOnes = APInt::getAllOnes(Bits).getZExtValue();
     TrueConst = GR->getOrCreateConsIntVector(AllOnes, MIRBuilder, ReturnType);
     FalseConst = GR->getOrCreateConsIntVector(0, MIRBuilder, ReturnType);
   } else {
@@ -417,33 +418,41 @@ static Register buildConstantIntReg(uint64_t Val, MachineIRBuilder &MIRBuilder,
 }
 
 static Register buildScopeReg(Register CLScopeRegister,
+                              SPIRV::Scope::Scope Scope,
                               MachineIRBuilder &MIRBuilder,
                               SPIRVGlobalRegistry *GR,
-                              const MachineRegisterInfo *MRI) {
-  auto CLScope =
-      static_cast<SPIRV::CLMemoryScope>(getIConstVal(CLScopeRegister, MRI));
-  SPIRV::Scope::Scope Scope = getSPIRVScope(CLScope);
-
-  if (CLScope == static_cast<unsigned>(Scope))
-    return CLScopeRegister;
-
+                              MachineRegisterInfo *MRI) {
+  if (CLScopeRegister.isValid()) {
+    auto CLScope =
+        static_cast<SPIRV::CLMemoryScope>(getIConstVal(CLScopeRegister, MRI));
+    Scope = getSPIRVScope(CLScope);
+
+    if (CLScope == static_cast<unsigned>(Scope)) {
+      MRI->setRegClass(CLScopeRegister, &SPIRV::IDRegClass);
+      return CLScopeRegister;
+    }
+  }
   return buildConstantIntReg(Scope, MIRBuilder, GR);
 }
 
 static Register buildMemSemanticsReg(Register SemanticsRegister,
-                                     Register PtrRegister,
-                                     const MachineRegisterInfo *MRI,
+                                     Register PtrRegister, unsigned &Semantics,
+                                     MachineIRBuilder &MIRBuilder,
                                      SPIRVGlobalRegistry *GR) {
-  std::memory_order Order =
-      static_cast<std::memory_order>(getIConstVal(SemanticsRegister, MRI));
-  unsigned Semantics =
-      getSPIRVMemSemantics(Order) |
-      getMemSemanticsForStorageClass(GR->getPointerStorageClass(PtrRegister));
-
-  if (Order == Semantics)
-    return SemanticsRegister;
+  if (SemanticsRegister.isValid()) {
+    MachineRegisterInfo *MRI = MIRBuilder.getMRI();
+    std::memory_order Order =
+        static_cast<std::memory_order>(getIConstVal(SemanticsRegister, MRI));
+    Semantics =
+        getSPIRVMemSemantics(Order) |
+        getMemSemanticsForStorageClass(GR->getPointerStorageClass(PtrRegister));
 
-  return Register();
+    if (Order == Semantics) {
+      MRI->setRegClass(SemanticsRegister, &SPIRV::IDRegClass);
+      return SemanticsRegister;
+    }
+  }
+  return buildConstantIntReg(Semantics, MIRBuilder, GR);
 }
 
 /// Helper function for translating atomic init to OpStore.
@@ -451,7 +460,8 @@ static bool buildAtomicInitInst(const SPIRV::IncomingCall *Call,
                                 MachineIRBuilder &MIRBuilder) {
   assert(Call->Arguments.size() == 2 &&
          "Need 2 arguments for atomic init translation");
-
+  MIRBuilder.getMRI()->setRegClass(Call->Arguments[0], &SPIRV::IDRegClass);
+  MIRBuilder.getMRI()->setRegClass(Call->Arguments[1], &SPIRV::IDRegClass);
   MIRBuilder.buildInstr(SPIRV::OpStore)
       .addUse(Call->Arguments[0])
       .addUse(Call->Arguments[1]);
@@ -463,19 +473,22 @@ static bool buildAtomicLoadInst(const SPIRV::IncomingCall *Call,
                                 MachineIRBuilder &MIRBuilder,
                                 SPIRVGlobalRegistry *GR) {
   Register PtrRegister = Call->Arguments[0];
+  MIRBuilder.getMRI()->setRegClass(PtrRegister, &SPIRV::IDRegClass);
   // TODO: if true insert call to __translate_ocl_memory_sccope before
   // OpAtomicLoad and the function implementation. We can use Translator's
   // output for transcoding/atomic_explicit_arguments.cl as an example.
   Register ScopeRegister;
-  if (Call->Arguments.size() > 1)
+  if (Call->Arguments.size() > 1) {
     ScopeRegister = Call->Arguments[1];
-  else
+    MIRBuilder.getMRI()->setRegClass(ScopeRegister, &SPIRV::IDRegClass);
+  } else
     ScopeRegister = buildConstantIntReg(SPIRV::Scope::Device, MIRBuilder, GR);
 
   Register MemSemanticsReg;
   if (Call->Arguments.size() > 2) {
     // TODO: Insert call to __translate_ocl_memory_order before OpAtomicLoad.
     MemSemanticsReg = Call->Arguments[2];
+    MIRBuilder.getMRI()->setRegClass(MemSemanticsReg, &SPIRV::IDRegClass);
   } else {
     int Semantics =
         SPIRV::MemorySemantics::SequentiallyConsistent |
@@ -499,11 +512,12 @@ static bool buildAtomicStoreInst(const SPIRV::IncomingCall *Call,
   Register ScopeRegister =
       buildConstantIntReg(SPIRV::Scope::Device, MIRBuilder, GR);
   Register PtrRegister = Call->Arguments[0];
+  MIRBuilder.getMRI()->setRegClass(PtrRegister, &SPIRV::IDRegClass);
   int Semantics =
       SPIRV::MemorySemantics::SequentiallyConsistent |
       getMemSemanticsForStorageClass(GR->getPointerStorageClass(PtrRegister));
   Register MemSemanticsReg = buildConstantIntReg(Semantics, MIRBuilder, GR);
-
+  MIRBuilder.getMRI()->setRegClass(Call->Arguments[1], &SPIRV::IDRegClass);
   MIRBuilder.buildInstr(SPIRV::OpAtomicStore)
       .addUse(PtrRegister)
       .addUse(ScopeRegister)
@@ -525,6 +539,9 @@ static bool buildAtomicCompareExchangeInst(const SPIRV::IncomingCall *Call,
   Register ObjectPtr = Call->Arguments[0];   // Pointer (volatile A *object.)
   Register ExpectedArg = Call->Arguments[1]; // Comparator (C* expected).
   Register Desired = Call->Arguments[2];     // Value (C Desired).
+  MRI->setRegClass(ObjectPtr, &SPIRV::IDRegClass);
+  MRI->setRegClass(ExpectedArg, &SPIRV::IDRegClass);
+  MRI->setRegClass(Desired, &SPIRV::IDRegClass);
   SPIRVType *SpvDesiredTy = GR->getSPIRVTypeForVReg(Desired);
   LLT DesiredLLT = MRI->getType(Desired);
 
@@ -564,6 +581,8 @@ static bool buildAtomicCompareExchangeInst(const SPIRV::IncomingCall *Call,
       MemSemEqualReg = Call->Arguments[3];
     if (MemOrdNeq == MemSemEqual)
       MemSemUnequalReg = Call->Arguments[4];
+    MRI->setRegClass(Call->Arguments[3], &SPIRV::IDRegClass);
+    MRI->setRegClass(Call->Arguments[4], &SPIRV::IDRegClass);
   }
   if (!MemSemEqualReg.isValid())
     MemSemEqualReg = buildConstantIntReg(MemSemEqual, MIRBuilder, GR);
@@ -580,6 +599,7 @@ static bool buildAtomicCompareExchangeInst(const SPIRV::IncomingCall *Call,
     Scope = getSPIRVScope(ClScope);
     if (ClScope == static_cast<unsigned>(Scope))
       ScopeReg = Call->Arguments[5];
+    MRI->setRegClass(Call->Arguments[5], &SPIRV::IDRegClass);
   }
   if (!ScopeReg.isValid())
     ScopeReg = buildConstantIntReg(Scope, MIRBuilder, GR);
@@ -591,6 +611,8 @@ static bool buildAtomicCompareExchangeInst(const SPIRV::IncomingCall *Call,
   MRI->setType(Expected, DesiredLLT);
   Register Tmp = !IsCmpxchg ? MRI->createGenericVirtualRegister(DesiredLLT)
                             : Call->ReturnRegister;
+  if (!MRI->getRegClassOrNull(Tmp))
+    MRI->setRegClass(Tmp, &SPIRV::IDRegClass);
   GR->assignSPIRVTypeToVReg(SpvDesiredTy, Tmp, MIRBuilder.getMF());
 
   SPIRVType *IntTy = GR->getOrCreateSPIRVIntegerType(32, MIRBuilder);
@@ -614,30 +636,23 @@ static bool buildAtomicCompareExchangeInst(const SPIRV::IncomingCall *Call,
 static bool buildAtomicRMWInst(const SPIRV::IncomingCall *Call, unsigned Opcode,
                                MachineIRBuilder &MIRBuilder,
                                SPIRVGlobalRegistry *GR) {
-  const MachineRegisterInfo *MRI = MIRBuilder.getMRI();
-  SPIRV::Scope::Scope Scope = SPIRV::Scope::Workgroup;
-  Register ScopeRegister;
-
-  if (Call->Arguments.size() >= 4) {
-    assert(Call->Arguments.size() == 4 &&
-           "Too many args for explicit atomic RMW");
-    ScopeRegister = buildScopeReg(Call->Arguments[3], MIRBuilder, GR, MRI);
-  }
+  MachineRegisterInfo *MRI = MIRBuilder.getMRI();
+  Register ScopeRegister =
+      Call->Arguments.size() >= 4 ? Call->Arguments[3] : Register();
 
-  if (!ScopeRegister.isValid())
-    ScopeRegister = buildConstantIntReg(Scope, MIRBuilder, GR);
+  assert(Call->Arguments.size() <= 4 &&
+         "Too many args for explicit atomic RMW");
+  ScopeRegister = buildScopeReg(ScopeRegister, SPIRV::Scope::Workgroup,
+                                MIRBuilder, GR, MRI);
 
   Register PtrRegister = Call->Arguments[0];
   unsigned Semantics = SPIRV::MemorySemantics::None;
-  Register MemSemanticsReg;
-
-  if (Call->Arguments.size() >= 3)
-    MemSemanticsReg =
-        buildMemSemanticsReg(Call->Arguments[2], PtrRegister, MRI, GR);
-
-  if (!MemSemanticsReg.isValid())
-    MemSemanticsReg = buildConstantIntReg(Semantics, MIRBuilder, GR);
-
+  MRI->setRegClass(PtrRegister, &SPIRV::IDRegClass);
+  Register MemSemanticsReg =
+      Call->Arguments.size() >= 3 ? Call->Arguments[2] : Register();
+  MemSemanticsReg = buildMemSemanticsReg(MemSemanticsReg, PtrRegister,
+                                         Semantics, MIRBuilder, GR);
+  MRI->setRegClass(Call->Arguments[1], &SPIRV::IDRegClass);
   MIRBuilder.buildInstr(Opcode)
       .addDef(Call->ReturnRegister)
       .addUse(GR->getSPIRVTypeID(Call->ReturnType))
@@ -653,32 +668,23 @@ static bool buildAtomicRMWInst(const SPIRV::IncomingCall *Call, unsigned Opcode,
 static bool buildAtomicFlagInst(const SPIRV::IncomingCall *Call,
                                 unsigned Opcode, MachineIRBuilder &MIRBuilder,
                                 SPIRVGlobalRegistry *GR) {
-  const MachineRegisterInfo *MRI = MIRBuilder.getMRI();
-
+  MachineRegisterInfo *MRI = MIRBuilder.getMRI();
   Register PtrRegister = Call->Arguments[0];
   unsigned Semantics = SPIRV::MemorySemantics::SequentiallyConsistent;
-  Register MemSemanticsReg;
-
-  if (Call->Arguments.size() >= 2)
-    MemSemanticsReg =
-        buildMemSemanticsReg(Call->Arguments[1], PtrRegister, MRI, GR);
-
-  if (!MemSemanticsReg.isValid())
-    MemSemanticsReg = buildConstantIntReg(Semantics, MIRBuilder, GR);
+  Register MemSemanticsReg =
+      Call->Arguments.size() >= 2 ? Call->Arguments[1] : Register();
+  MemSemanticsReg = buildMemSemanticsReg(MemSemanticsReg, PtrRegister,
+                                         Semantics, MIRBuilder, GR);
 
   assert((Opcode != SPIRV::OpAtomicFlagClear ||
           (Semantics != SPIRV::MemorySemantics::Acquire &&
            Semantics != SPIRV::MemorySemantics::AcquireRelease)) &&
          "Invalid memory order argument!");
 
-  SPIRV::Scope::Scope Scope = SPIRV::Scope::Device;
-  Register ScopeRegister;
-
-  if (Call->Arguments.size() >= 3)
-    ScopeRegister = buildScopeReg(Call->Arguments[2], MIRBuilder, GR, MRI);
-
-  if (!ScopeRegister.isValid())
-    ScopeRegister = buildConstantIntReg(Scope, MIRBuilder, GR);
+  Register ScopeRegister =
+      Call->Arguments.size() >= 3 ? Call->Arguments[2] : Register();
+  ScopeRegister =
+      buildScopeReg(ScopeRegister, SPIRV::Scope::Device, MIRBuilder, GR, MRI);
 
   auto MIB = MIRBuilder.buildInstr(Opcode);
   if (Opcode == SPIRV::OpAtomicFlagTestAndSet)
@@ -694,7 +700,7 @@ static bool buildAtomicFlagInst(const SPIRV::IncomingCall *Call,
 static bool buildBarrierInst(const SPIRV::IncomingCall *Call, unsigned Opcode,
                              MachineIRBuilder &MIRBuilder,
                              SPIRVGlobalRegistry *GR) {
-  const MachineRegisterInfo *MRI = MIRBuilder.getMRI();
+  MachineRegisterInfo *MRI = MIRBuilder.getMRI();
   unsigned MemFlags = getIConstVal(Call->Arguments[0], MRI);
   unsigned MemSemantics = SPIRV::MemorySemantics::None;
 
@@ -716,9 +722,10 @@ static bool buildBarrierInst(const SPIRV::IncomingCall *Call, unsigned Opcode,
   }
 
   Register MemSemanticsReg;
-  if (MemFlags == MemSemantics)
+  if (MemFlags == MemSemantics) {
     MemSemanticsReg = Call->Arguments[0];
-  else
+    MRI->setRegClass(MemSemanticsReg, &SPIRV::IDRegClass);
+  } else
     MemSemanticsReg = buildConstantIntReg(MemSemantics, MIRBuilder, GR);
 
   Register ScopeReg;
@@ -738,8 +745,10 @@ static bool buildBarrierInst(const SPIRV::IncomingCall *Call, unsigned Opcode,
         (Opcode == SPIRV::OpMemoryBarrier))
       Scope = MemScope;
 
-    if (CLScope == static_cast<unsigned>(Scope))
+    if (CLScope == static_cast<unsigned>(Scope)) {
       ScopeReg = Call->Arguments[1];
+      MRI->setRegClass(ScopeReg, &SPIRV::IDRegClass);
+    }
   }
 
   if (!ScopeReg.isValid())
@@ -834,7 +843,7 @@ static bool generateGroupInst(const SPIRV::IncomingCall *Call,
   const SPIRV::DemangledBuiltin *Builtin = Call->Builtin;
   const SPIRV::GroupBuiltin *GroupBuiltin =
       SPIRV::lookupGroupBuiltin(Builtin->Name);
-  const MachineRegisterInfo *MRI = MIRBuilder.getMRI();
+  MachineRegisterInfo *MRI = MIRBuilder.getMRI();
   Register Arg0;
   if (GroupBuiltin->HasBoolArg) {
     Register ConstRegister = Call->Arguments[0];
@@ -876,8 +885,11 @@ static bool generateGroupInst(const SPIRV::IncomingCall *Call,
     MIB.addImm(GroupBuiltin->GroupOperation);
   if (Call->Arguments.size() > 0) {
     MIB.addUse(Arg0.isValid() ? Arg0 : Call->Arguments[0]);
-    for (unsigned i = 1; i < Call->Arguments.size(); i++)
+    MRI->setRegClass(Call->Arguments[0], &SPIRV::IDRegClass);
+    for (unsigned i = 1; i < Call->Arguments.size(); i++) {
       MIB.addUse(Call->Arguments[i]);
+      MRI->setRegClass(Call->Arguments[i], &SPIRV::IDRegClass);
+    }
   }
 
   // Build select instruction.
@@ -936,16 +948,17 @@ static bool genWorkgroupQuery(const SPIRV::IncomingCall *Call,
   // If it's out of range (max dimension is 3), we can just return the constant
   // default value (0 or 1 depending on which query function).
   if (IsConstantIndex && getIConstVal(IndexRegister, MRI) >= 3) {
-    Register defaultReg = Call->ReturnRegister;
+    Register DefaultReg = Call->ReturnRegister;
     if (PointerSize != ResultWidth) {
-      defaultReg = MRI->createGenericVirtualRegister(LLT::scalar(PointerSize));
-      GR->assignSPIRVTypeToVReg(PointerSizeType, defaultReg,
+      DefaultReg = MRI->createGenericVirtualRegister(LLT::scalar(PointerSize));
+      MRI->setRegClass(DefaultReg, &SPIRV::IDRegClass);
+      GR->assignSPIRVTypeToVReg(PointerSizeType, DefaultReg,
                                 MIRBuilder.getMF());
-      ToTruncate = defaultReg;
+      ToTruncate = DefaultReg;
     }
     auto NewRegister =
         GR->buildConstantInt(DefaultValue, MIRBuilder, PointerSizeType);
-    MIRBuilder.buildCopy(defaultReg, NewRegister);
+    MIRBuilder.buildCopy(DefaultReg, NewRegister);
   } else { // If it could be in range, we need to load from the given builtin.
     auto Vec3Ty =
         GR->getOrCreateSPIRVVectorType(PointerSizeType, 3, MIRBuilder);
@@ -956,6 +969,7 @@ static bool genWorkgroupQuery(const SPIRV::IncomingCall *Call,
     Register Extracted = Call->ReturnRegister;
     if (!IsConstantIndex || PointerSize != ResultWidth) {
       Extracted = MRI->createGenericVirtualRegister(LLT::scalar(PointerSize));
+      MRI->setRegClass(Extracted, &SPIRV::IDRegClass);
       GR->assignSPIRVTypeToVReg(PointerSizeType, Extracted, MIRBuilder.getMF());
     }
     // Use Intrinsic::spv_extractelt so dynamic vs static extraction is
@@ -974,6 +988,7 @@ static bool genWorkgroupQuery(const SPIRV::IncomingCall *Call,
 
       Register CompareRegister =
           MRI->createGenericVirtualRegister(LLT::scalar(1));
+      MRI->setRegClass(CompareRegister, &SPIRV::IDRegClass);
       GR->assignSPIRVTypeToVReg(BoolType, CompareRegister, MIRBuilder.getMF());
 
       // Use G_ICMP to check if idxVReg < 3.
@@ -990,6 +1005,7 @@ static bool genWorkgroupQuery(const SPIRV::IncomingCall *Call,
       if (PointerSize != ResultWidth) {
         SelectionResult =
             MRI->createGenericVirtualRegister(LLT::scalar(PointerSize));
+        MRI->setRegClass(SelectionResult, &SPIRV::IDRegClass);
         GR->assignSPIRVTypeToVReg(PointerSizeType, SelectionResult,
                                   MIRBuilder.getMF());
       }
@@ -1125,6 +1141,7 @@ static bool generateImageSizeQueryInst(const SPIRV::IncomingCall *Call,
   if (NumExpectedRetComponents != NumActualRetComponents) {
     QueryResult = MIRBuilder.getMRI()->createGenericVirtualRegister(
         LLT::fixed_vector(NumActualRetComponents, 32));
+    MIRBuilder.getMRI()->setRegClass(QueryResult, &SPIRV::IDRegClass);
     SPIRVType *IntTy = GR->getOrCreateSPIRVIntegerType(32, MIRBuilder);
     QueryResultType = GR->getOrCreateSPIRVVectorType(
         IntTy, NumActualRetComponents, MIRBuilder);
@@ -1133,6 +1150,7 @@ static bool generateImageSizeQueryInst(const SPIRV::IncomingCall *Call,
   bool IsDimBuf = ImgType->getOperand(2).getImm() == SPIRV::Dim::DIM_Buffer;
   unsigned Opcode =
       IsDimBuf ? SPIRV::OpImageQuerySize : SPIRV::OpImageQuerySizeLod;
+  MIRBuilder.getMRI()->setRegClass(Call->Arguments[0], &SPIRV::IDRegClass);
   auto MIB = MIRBuilder.buildInstr(Opcode)
                  .addDef(QueryResult)
                  .addUse(GR->getSPIRVTypeID(QueryResultType))
@@ -1177,6 +1195,7 @@ static bool generateImageMiscQueryInst(const SPIRV::IncomingCall *Call,
       SPIRV::lookupNativeBuiltin(Builtin->Name, Builtin->Set)->Opcode;
 
   Register Image = Call->Arguments[0];
+  MIRBuilder.getMRI()->setRegClass(Image, &SPIRV::IDRegClass);
   SPIRV::Dim::Dim ImageDimensionality = static_cast<SPIRV::Dim::Dim>(
       GR->getSPIRVTypeForVReg(Image)->getOperand(2).getImm());
 
@@ -1239,8 +1258,13 @@ static bool generateReadImageInst(const StringRef DemangledCall,
                                   SPIRVGlobalRegistry *GR) {
   Register Image = Call->Arguments[0];
   MachineRegisterInfo *MRI = MIRBuilder.getMRI();
-
-  if (DemangledCall.contains_insensitive("ocl_sampler")) {
+  MRI->setRegClass(Image, &SPIRV::IDRegClass);
+  MRI->setRegClass(Call->Arguments[1], &SPIRV::IDRegClass);
+  bool HasOclSampler = DemangledCall.contains_insensitive("ocl_sampler");
+  bool HasMsaa = DemangledCall.contains_insensitive("msaa");
+  if (HasOclSampler || HasMsaa)
+    MRI->setRegClass(Call->Arguments[2], &SPIRV::IDRegClass);
+  if (HasOclSampler) {
     Register Sampler = Call->Arguments[1];
 
     if (!GR->isScalarOfType(Sampler, SPIRV::OpTypeSampler) &&
@@ -1274,6 +1298,7 @@ static bool generateReadImageInst(const StringRef DemangledCall,
     }
     LLT LLType = LLT::scalar(GR->getScalarOrVectorBitWidth(TempType));
     Register TempRegister = MRI->createGenericVirtualRegister(LLType);
+    MRI->setRegClass(TempRegister, &SPIRV::IDRegClass);
     GR->assignSPIRVTypeToVReg(TempType, TempRegister, MIRBuilder.getMF());
 
     MIRBuilder.buildInstr(SPIRV::OpImageSampleExplicitLod)
@@ -1290,7 +1315,7 @@ static bool generateReadImageInst(const StringRef DemangledCall,
           .addUse(GR->getSPIRVTypeID(Call->ReturnType))
           .addUse(TempRegister)
           .addImm(0);
-  } else if (DemangledCall.contains_insensitive("msaa")) {
+  } else if (HasMsaa) {
     MIRBuilder.buildInstr(SPIRV::OpImageRead)
         .addDef(Call->ReturnRegister)
         .addUse(GR->getSPIRVTypeID(Call->ReturnType))
@@ -1311,6 +1336,9 @@ static bool generateReadImageInst(const StringRef DemangledCall,
 static bool generateWriteImageInst(const SPIRV::IncomingCall *Call,
                                    MachineIRBuilder &MIRBuilder,
                                    SPIRVGlobalRegistry *GR) {
+  MIRBuilder.getMRI()->setRegClass(Call->Arguments[0], &SPIRV::IDRegClass);
+  MIRBuilder.getMRI()->setRegClass(Call->Arguments[1], &SPIRV::IDRegClass);
+  MIRBuilder.getMRI()->setRegClass(Call->Arguments[2], &SPIRV::IDRegClass);
   MIRBuilder.buildInstr(SPIRV::OpImageWrite)
       .addUse(Call->Arguments[0])  // Image.
       .addUse(Call->Arguments[1])  // Coordinate.
@@ -1322,10 +1350,11 @@ static bool generateSampleImageInst(const StringRef DemangledCall,
                                     const SPIRV::IncomingCall *Call,
                                     MachineIRBuilder &MIRBuilder,
                                     SPIRVGlobalRegistry *GR) {
+  MachineRegisterInfo *MRI = MIRBuilder.getMRI();
   if (Call->Builtin->Name.contains_insensitive(
           "__translate_sampler_initializer")) {
     // Build sampler literal.
-    uint64_t Bitmask = getIConstVal(Call->Arguments[0], MIRBuilder.getMRI());
+    uint64_t Bitmask = getIConstVal(Call->Arguments[0], MRI);
     Register Sampler = GR->buildConstantSampler(
         Call->ReturnRegister, getSamplerAddressingModeFromBitmask(Bitmask),
         getSamplerParamFromBitmask(Bitmask),
@@ -1340,7 +1369,7 @@ static bool generateSampleImageInst(const StringRef DemangledCall,
     Register SampledImage =
         Call->ReturnRegister.isValid()
             ? Call->ReturnRegister
-            : MIRBuilder.getMRI()->createVirtualRegister(&SPIRV::IDRegClass);
+            : MRI->createVirtualRegister(&SPIRV::IDRegClass);
     MIRBuilder.buildInstr(SPIRV::OpSampledImage)
         .addDef(SampledImage)
         .addUse(GR->getSPIRVTypeID(SampledImageType))
@@ -1356,6 +1385,10 @@ static bool generateSampleImageInst(const StringRef DemangledCall,
       ReturnType = ReturnType.substr(0, ReturnType.find('('));
     }
     SPIRVType *Type = GR->getOrCreateSPIRVTypeByName(ReturnType, MIRBuilder);
+    MRI->setRegClass(Call->Arguments[0], &SPIRV::IDRegClass);
+    MRI->setRegClass(Call->Arguments[1], &SPIRV::IDRegClass);
+    MRI->setRegClass(Call->Arguments[3], &SPIRV::IDRegClass);
+
     MIRBuilder.buildInstr(SPIRV::OpImageSampleExplicitLod)
         .addDef(Call->ReturnRegister)
         .addUse(GR->getSPIRVTypeID(Type))
@@ -1431,6 +1464,75 @@ static bool generateSpecConstantInst(const SPIRV::IncomingCall *Call,
   }
 }
 
+static bool buildNDRange(const SPIRV::IncomingCall *Call,
+                         MachineIRBuilder &MIRBuilder,
+                         SPIRVGlobalRegistry *GR) {
+  MachineRegisterInfo *MRI = MIRBuilder.getMRI();
+  MRI->setRegClass(Call->Arguments[0], &SPIRV::IDRegClass);
+  SPIRVType *PtrType = GR->getSPIRVTypeForVReg(Call->Arguments[0]);
+  assert(PtrType->getOpcode() == SPIRV::OpTypePointer &&
+         PtrType->getOperand(2).isReg());
+  Register TypeReg = PtrType->getOperand(2).getReg();
+  SPIRVType *StructType = GR->getSPIRVTypeForVReg(TypeReg);
+  MachineFunction &MF = MIRBuilder.getMF();
+  Register TmpReg = MRI->createVirtualRegister(&SPIRV::IDRegClass);
+  GR->assignSPIRVTypeToVReg(StructType, TmpReg, MF);
+  // Skip the first arg, it's the destination pointer. OpBuildNDRange takes
+  // three other arguments, so pass zero constant on absence.
+  unsigned NumArgs = Call->Arguments.size();
+  assert(NumArgs >= 2);
+  Register GlobalWorkSize = Call->Arguments[NumArgs < 4 ? 1 : 2];
+  MRI->setRegClass(GlobalWorkSize, &SPIRV::IDRegClass);
+  Register LocalWorkSize =
+      NumArgs == 2 ? Register(0) : Call->Arguments[NumArgs < 4 ? 2 : 3];
+  if (LocalWorkSize.isValid())
+    MRI->setRegClass(LocalWorkSize, &SPIRV::IDRegClass);
+  Register GlobalWorkOffset = NumArgs <= 3 ? Register(0) : Call->Arguments[1];
+  if (GlobalWorkOffset.isValid())
+    MRI->setRegClass(GlobalWorkOffset, &SPIRV::IDRegClass);
+  if (NumArgs < 4) {
+    Register Const;
+    SPIRVType *SpvTy = GR->getSPIRVTypeForVReg(GlobalWorkSize);
+    if (SpvTy->getOpcode() == SPIRV::OpTypePointer) {
+      MachineInstr *DefInstr = MRI->getUniqueVRegDef(GlobalWorkSize);
+      assert(DefInstr && isSpvIntrinsic(*DefInstr, Intrinsic::spv_gep) &&
+             DefInstr->getOperand(3).isReg());
+      Register GWSPtr = DefInstr->getOperand(3).getReg();
+      if (!MRI->getRegClassOrNull(GWSPtr))
+        MRI->setRegClass(GWSPtr, &SPIRV::IDRegClass);
+      // TODO: Maybe simplify generation of the type of the fields.
+      unsigned Size = Call->Builtin->Name.equals("ndrange_3D") ? 3 : 2;
+      unsigned BitWidth = GR->getPointerSize() == 64 ? 64 : 32;
+      Type *BaseTy = IntegerType::get(MF.getFunction().getContext(), BitWidth);
+      Type *FieldTy = ArrayType::get(BaseTy, Size);
+      SPIRVType *SpvFieldTy = GR->getOrCreateSPIRVType(FieldTy, MIRBuilder);
+      GlobalWorkSize = MRI->createVirtualRegister(&SPIRV::IDRegClass);
+      GR->assignSPIRVTypeToVReg(SpvFieldTy, GlobalWorkSize, MF);
+      MIRBuilder.buildInstr(SPIRV::OpLoad)
+          .addDef(GlobalWorkSize)
+          .addUse(GR->getSPIRVTypeID(SpvFieldTy))
+          .addUse(GWSPtr);
+      Const = GR->getOrCreateConsIntArray(0, MIRBuilder, SpvFieldTy);
+    } else {
+      Const = GR->buildConstantInt(0, MIRBuilder, SpvTy);
+    }
+    if (!LocalWorkSize.isValid())
+      LocalWorkSize = Const;
+    if (!GlobalWorkOffset.isValid())
+      GlobalWorkOffset = Const;
+  }
+  assert(LocalWorkSize.isValid() && GlobalWorkOffset.isValid());
+  MIRBuilder.buildInstr(SPIRV::OpBuildNDRange)
+      .addDef(TmpReg)
+      .addUse(TypeReg)
+      .addUse(GlobalWorkSize)
+      .addUse(LocalWorkSize)
+      .addUse(GlobalWorkOffset);
+  return MIRBuilder.buildInstr(SPIRV::OpStore)
+      .addUse(Call->Arguments[0])
+      .addUse(TmpReg);
+}
+
 static MachineInstr *getBlockStructInstr(Register ParamReg,
                                          MachineRegisterInfo *MRI) {
   // We expect the following sequence of instructions:
@@ -1538,9 +1640,8 @@ static bool buildEnqueueKernel(const SPIRV::IncomingCall *Call,
     const SPIRVType *PointerSizeTy = GR->getOrCreateSPIRVPointerType(
         Int32Ty, MIRBuilder, SPIRV::StorageClass::Function);
     for (unsigned I = 0; I < LocalSizeNum; ++I) {
-      Register Reg =
-          MIRBuilder.getMRI()->createVirtualRegister(&SPIRV::IDRegClass);
-      MIRBuilder.getMRI()->setType(Reg, LLType);
+      Register Reg = MRI->createVirtualRegister(&SPIRV::IDRegClass);
+      MRI->setType(Reg, LLType);
       GR->assignSPIRVTypeToVReg(PointerSizeTy, Reg, MIRBuilder.getMF());
       auto GEPInst = MIRBuilder.buildIntrinsic(Intrinsic::spv_gep,
                                                ArrayRef<Register>{Reg}, true);
@@ -1605,6 +1706,7 @@ static bool generateEnqueueInst(const SPIRV::IncomingCall *Call,
   switch (Opcode) {
   case SPIRV::OpRetainEvent:
   case SPIRV::OpReleaseEvent:
+    MIRBuilder.getMRI()->setRegClass(Call->Arguments[0], &SPIRV::IDRegClass);
     return MIRBuilder.buildInstr(Opcode).addUse(Call->Arguments[0]);
   case SPIRV::OpCreateUserEvent:
   case SPIRV::OpGetDefaultQueue:
@@ -1612,77 +1714,27 @@ static bool generateEnqueueInst(const SPIRV::IncomingCall *Call,
         .addDef(Call->ReturnRegister)
         .addUse(GR->getSPIRVTypeID(Call->ReturnType));
   case SPIRV::OpIsValidEvent:
+    MIRBuilder.getMRI()->setRegClass(Call->Arguments[0], &SPIRV::IDRegClass);
     return MIRBuilder.buildInstr(Opcode)
         .addDef(Call->ReturnRegister)
         .addUse(GR->getSPIRVTypeID(Call->ReturnType))
         .addUse(Call->Arguments[0]);
   case SPIRV::OpSetUserEventStatus:
+    MIRBuilder.getMRI()->setRegClass(Call->Arguments[0], &SPIRV::IDRegClass);
+    MIRBuilder.getMRI()->setRegClass(Call->Arguments[1], &SPIRV::IDRegClass);
     return MIRBuilder.buildInstr(Opcode)
         .addUse(Call->Arguments[0])
         .addUse(Call->Arguments[1]);
   case SPIRV::OpCaptureEventProfilingInfo:
+    MIRBuilder.getMRI()->setRegClass(Call->Arguments[0], &SPIRV::IDRegClass);
+    MIRBuilder.getMRI()->setRegClass(Call->Arguments[1], &SPIRV::IDRegClass);
+    MIRBuilder.getMRI()->setRegClass(Call->Arguments[2], &SPIRV::IDRegClass);
     return MIRBuilder.buildInstr(Opcode)
         .addUse(Call->Arguments[0])
         .addUse(Call->Arguments[1])
         .addUse(Call->Arguments[2]);
-  case SPIRV::OpBuildNDRange: {
-    MachineRegisterInfo *MRI = MIRBuilder.getMRI();
-    SPIRVType *PtrType = GR->getSPIRVTypeForVReg(Call->Arguments[0]);
-    assert(PtrType->getOpcode() == SPIRV::OpTypePointer &&
-           PtrType->getOperand(2).isReg());
-    Register TypeReg = PtrType->getOperand(2).getReg();
-    SPIRVType *StructType = GR->getSPIRVTypeForVReg(TypeReg);
-    Register TmpReg = MRI->createVirtualRegister(&SPIRV::IDRegClass);
-    GR->assignSPIRVTypeToVReg(StructType, TmpReg, MIRBuilder.getMF());
-    // Skip the first arg, it's the destination pointer. OpBuildNDRange takes
-    // three other arguments, so pass zero constant on absence.
-    unsigned NumArgs = Call->Arguments.size();
-    assert(NumArgs >= 2);
-    Register GlobalWorkSize = Call->Arguments[NumArgs < 4 ? 1 : 2];
-    Register LocalWorkSize =
-        NumArgs == 2 ? Register(0) : Call->Arguments[NumArgs < 4 ? 2 : 3];
-    Register GlobalWorkOffset = NumArgs <= 3 ? Register(0) : Call->Arguments[1];
-    if (NumArgs < 4) {
-      Register Const;
-      SPIRVType *SpvTy = GR->getSPIRVTypeForVReg(GlobalWorkSize);
-      if (SpvTy->getOpcode() == SPIRV::OpTypePointer) {
-        MachineInstr *DefInstr = MRI->getUniqueVRegDef(GlobalWorkSize);
-        assert(DefInstr && isSpvIntrinsic(*DefInstr, Intrinsic::spv_gep) &&
-               DefInstr->getOperand(3).isReg());
-        Register GWSPtr = DefInstr->getOperand(3).getReg();
-        // TODO: Maybe simplify generation of the type of the fields.
-        unsigned Size = Call->Builtin->Name.equals("ndrange_3D") ? 3 : 2;
-        unsigned BitWidth = GR->getPointerSize() == 64 ? 64 : 32;
-        Type *BaseTy = IntegerType::get(
-            MIRBuilder.getMF().getFunction().getContext(), BitWidth);
-        Type *FieldTy = ArrayType::get(BaseTy, Size);
-        SPIRVType *SpvFieldTy = GR->getOrCreateSPIRVType(FieldTy, MIRBuilder);
-        GlobalWorkSize = MRI->createVirtualRegister(&SPIRV::IDRegClass);
-        GR->assignSPIRVTypeToVReg(SpvFieldTy, GlobalWorkSize,
-                                  MIRBuilder.getMF());
-        MIRBuilder.buildInstr(SPIRV::OpLoad)
-            .addDef(GlobalWorkSize)
-            .addUse(GR->getSPIRVTypeID(SpvFieldTy))
-            .addUse(GWSPtr);
-        Const = GR->getOrCreateConsIntArray(0, MIRBuilder, SpvFieldTy);
-      } else {
-        Const = GR->buildConstantInt(0, MIRBuilder, SpvTy);
-      }
-      if (!LocalWorkSize.isValid())
-        LocalWorkSize = Const;
-      if (!GlobalWorkOffset.isValid())
-        GlobalWorkOffset = Const;
-    }
-    MIRBuilder.buildInstr(Opcode)
-        .addDef(TmpReg)
-        .addUse(TypeReg)
-        .addUse(GlobalWorkSize)
-        .addUse(LocalWorkSize)
-        .addUse(GlobalWorkOffset);
-    return MIRBuilder.buildInstr(SPIRV::OpStore)
-        .addUse(Call->Arguments[0])
-        .addUse(TmpReg);
-  }
+  case SPIRV::OpBuildNDRange:
+    return buildNDRange(Call, MIRBuilder, GR);
   case SPIRV::OpEnqueueKernel:
     return buildEnqueueKernel(Call, MIRBuilder, GR);
   default:
@@ -1817,16 +1869,23 @@ static bool generateLoadStoreInst(const SPIRV::IncomingCall *Call,
   }
   // Add a pointer to the value to load/store.
   MIB.addUse(Call->Arguments[0]);
+  MachineRegisterInfo *MRI = MIRBuilder.getMRI();
+  MRI->setRegClass(Call->Arguments[0], &SPIRV::IDRegClass);
   // Add a value to store.
-  if (!IsLoad)
+  if (!IsLoad) {
     MIB.addUse(Call->Arguments[1]);
+    MRI->setRegClass(Call->Arguments[1], &SPIRV::IDRegClass);
+  }
   // Add optional memory attributes and an alignment.
-  MachineRegisterInfo *MRI = MIRBuilder.getMRI();
   unsigned NumArgs = Call->Arguments.size();
-  if ((IsLoad && NumArgs >= 2) || NumArgs >= 3)
+  if ((IsLoad && NumArgs >= 2) || NumArgs >= 3) {
     MIB.addImm(getConstFromIntrinsic(Call->Arguments[IsLoad ? 1 : 2], MRI));
-  if ((IsLoad && NumArgs >= 3) || NumArgs >= 4)
+    MRI->setRegClass(Call->Arguments[IsLoad ? 1 : 2], &SPIRV::IDRegClass);
+  }
+  if ((IsLoad && NumArgs >= 3) || NumArgs >= 4) {
     MIB.addImm(getConstFromIntrinsic(Call->Arguments[IsLoad ? 2 : 3], MRI));
+    MRI->setRegClass(Call->Arguments[IsLoad ? 2 : 3], &SPIRV::IDRegClass);
+  }
   return true;
 }
 
@@ -1846,6 +1905,8 @@ std::optional<bool> lowerBuiltin(const StringRef DemangledCall,
   SPIRVType *ReturnType = nullptr;
   if (OrigRetTy && !OrigRetTy->isVoidTy()) {
     ReturnType = GR->assignTypeToVReg(OrigRetTy, OrigRet, MIRBuilder);
+    if (!MIRBuilder.getMRI()->getRegClassOrNull(ReturnRegister))
+      MIRBuilder.getMRI()->setRegClass(ReturnRegister, &SPIRV::IDRegClass);
   } else if (OrigRetTy && OrigRetTy->isVoidTy()) {
     ReturnRegister = MIRBuilder.getMRI()->createVirtualRegister(&IDRegClass);
     MIRBuilder.getMRI()->setType(ReturnRegister, LLT::scalar(32));
diff --git a/llvm/lib/Target/SPIRV/SPIRVCallLowering.cpp b/llvm/lib/Target/SPIRV/SPIRVCallLowering.cpp
index 8b618686ee7da..47b25a1f83515 100644
--- a/llvm/lib/Target/SPIRV/SPIRVCallLowering.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVCallLowering.cpp
@@ -374,6 +374,7 @@ bool SPIRVCallLowering::lowerCall(MachineIRBuilder &MIRBuilder,
     FTy = getOriginalFunctionType(*CF);
   }
 
+  MachineRegisterInfo *MRI = MIRBuilder.getMRI();
   Register ResVReg =
       Info.OrigRet.Regs.empty() ? Register(0) : Info.OrigRet.Regs[0];
   std::string FuncName = Info.Callee.getGlobal()->getName().str();
@@ -410,8 +411,9 @@ bool SPIRVCallLowering::lowerCall(MachineIRBuilder &MIRBuilder,
     for (const Argument &Arg : CF->args()) {
       if (MIRBuilder.getDataLayout().getTypeStoreSize(Arg.getType()).isZero())
         continue; // Don't handle zero sized types.
-      ToInsert.push_back(
-          {MIRBuilder.getMRI()->createGenericVirtualRegister(LLT::scalar(32))});
+      Register Reg = MRI->createGenericVirtualRegister(LLT::scalar(32));
+      MRI->setRegClass(Reg, &SPIRV::IDRegClass);
+      ToInsert.push_back({Reg});
       VRegArgs.push_back(ToInsert.back());
     }
     // TODO: Reuse FunctionLoweringInfo
diff --git a/llvm/lib/Target/SPIRV/SPIRVGlobalRegistry.cpp b/llvm/lib/Target/SPIRV/SPIRVGlobalRegistry.cpp
index 7da239cd31ff7..c77a7f860eda2 100644
--- a/llvm/lib/Target/SPIRV/SPIRVGlobalRegistry.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVGlobalRegistry.cpp
@@ -80,6 +80,16 @@ SPIRVType *SPIRVGlobalRegistry::getOpTypeBool(MachineIRBuilder &MIRBuilder) {
 SPIRVType *SPIRVGlobalRegistry::getOpTypeInt(uint32_t Width,
                                              MachineIRBuilder &MIRBuilder,
                                              bool IsSigned) {
+  assert(Width <= 64 && "Unsupported integer width!");
+  if (Width <= 8)
+    Width = 8;
+  else if (Width <= 16)
+    Width = 16;
+  else if (Width <= 32)
+    Width = 32;
+  else if (Width <= 64)
+    Width = 64;
+
   auto MIB = MIRBuilder.buildInstr(SPIRV::OpTypeInt)
                  .addDef(createTypeVReg(MIRBuilder))
                  .addImm(Width)
@@ -133,6 +143,7 @@ SPIRVGlobalRegistry::getOrCreateConstIntReg(uint64_t Val, SPIRVType *SpvType,
     unsigned BitWidth = SpvType ? getScalarOrVectorBitWidth(SpvType) : 32;
     LLT LLTy = LLT::scalar(32);
     Res = CurMF->getRegInfo().createGenericVirtualRegister(LLTy);
+    CurMF->getRegInfo().setRegClass(Res, &SPIRV::IDRegClass);
     if (MIRBuilder)
       assignTypeToVReg(LLVMIntTy, Res, *MIRBuilder);
     else
@@ -192,6 +203,7 @@ Register SPIRVGlobalRegistry::buildConstantInt(uint64_t Val,
     unsigned BitWidth = SpvType ? getScalarOrVectorBitWidth(SpvType) : 32;
     LLT LLTy = LLT::scalar(EmitIR ? BitWidth : 32);
     Res = MF.getRegInfo().createGenericVirtualRegister(LLTy);
+    MF.getRegInfo().setRegClass(Res, &SPIRV::IDRegClass);
     assignTypeToVReg(LLVMIntTy, Res, MIRBuilder,
                      SPIRV::AccessQualifier::ReadWrite, EmitIR);
     DT.add(ConstInt, &MIRBuilder.getMF(), Res);
@@ -237,6 +249,7 @@ Register SPIRVGlobalRegistry::buildConstantFP(APFloat Val,
   if (!Res.isValid()) {
     unsigned BitWidth = SpvType ? getScalarOrVectorBitWidth(SpvType) : 32;
     Res = MF.getRegInfo().createGenericVirtualRegister(LLT::scalar(BitWidth));
+    MF.getRegInfo().setRegClass(Res, &SPIRV::IDRegClass);
     assignTypeToVReg(LLVMFPTy, Res, MIRBuilder);
     DT.add(ConstFP, &MF, Res);
     MIRBuilder.buildFConstant(Res, *ConstFP);
@@ -262,6 +275,7 @@ Register SPIRVGlobalRegistry::getOrCreateIntCompositeOrNull(
     LLT LLTy = LLT::scalar(32);
     Register SpvVecConst =
         CurMF->getRegInfo().createGenericVirtualRegister(LLTy);
+    CurMF->getRegInfo().setRegClass(SpvVecConst, &SPIRV::IDRegClass);
     assignSPIRVTypeToVReg(SpvType, SpvVecConst, *CurMF);
     DT.add(CA, CurMF, SpvVecConst);
     MachineInstrBuilder MIB;
@@ -333,6 +347,7 @@ Register SPIRVGlobalRegistry::getOrCreateIntCompositeOrNull(
     LLT LLTy = EmitIR ? LLT::fixed_vector(ElemCnt, BitWidth) : LLT::scalar(32);
     Register SpvVecConst =
         CurMF->getRegInfo().createGenericVirtualRegister(LLTy);
+    CurMF->getRegInfo().setRegClass(SpvVecConst, &SPIRV::IDRegClass);
     assignSPIRVTypeToVReg(SpvType, SpvVecConst, *CurMF);
     DT.add(CA, CurMF, SpvVecConst);
     if (EmitIR) {
@@ -401,6 +416,7 @@ SPIRVGlobalRegistry::getOrCreateConstNullPtr(MachineIRBuilder &MIRBuilder,
   if (!Res.isValid()) {
     LLT LLTy = LLT::pointer(LLVMPtrTy->getAddressSpace(), PointerSize);
     Res = CurMF->getRegInfo().createGenericVirtualRegister(LLTy);
+    CurMF->getRegInfo().setRegClass(Res, &SPIRV::IDRegClass);
     assignSPIRVTypeToVReg(SpvType, Res, *CurMF);
     MIRBuilder.buildInstr(SPIRV::OpConstantNull)
         .addDef(Res)
@@ -1080,6 +1096,7 @@ Register SPIRVGlobalRegistry::getOrCreateUndef(MachineInstr &I,
     return Res;
   LLT LLTy = LLT::scalar(32);
   Res = CurMF->getRegInfo().createGenericVirtualRegister(LLTy);
+  CurMF->getRegInfo().setRegClass(Res, &SPIRV::IDRegClass);
   assignSPIRVTypeToVReg(SpvType, Res, *CurMF);
   DT.add(UV, CurMF, Res);
 
diff --git a/llvm/lib/Target/SPIRV/SPIRVISelLowering.cpp b/llvm/lib/Target/SPIRV/SPIRVISelLowering.cpp
index c226b1ac7528f..c2582bc298b40 100644
--- a/llvm/lib/Target/SPIRV/SPIRVISelLowering.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVISelLowering.cpp
@@ -27,6 +27,8 @@ unsigned SPIRVTargetLowering::getNumRegistersForCallingConv(
       (VT.getVectorElementType() == MVT::i1 ||
        VT.getVectorElementType() == MVT::i8))
     return 1;
+  if (!VT.isVector() && VT.isInteger() && VT.getSizeInBits() <= 64)
+    return 1;
   return getNumRegisters(Context, VT);
 }
 
diff --git a/llvm/lib/Target/SPIRV/SPIRVInstrInfo.td b/llvm/lib/Target/SPIRV/SPIRVInstrInfo.td
index e1521d44e4e53..44b5536becf7f 100644
--- a/llvm/lib/Target/SPIRV/SPIRVInstrInfo.td
+++ b/llvm/lib/Target/SPIRV/SPIRVInstrInfo.td
@@ -213,7 +213,7 @@ def PseudoConstI: IntImmLeaf<i32, [{ return Imm.getBitWidth() <= 32; }], imm_to_
 def PseudoConstF: FPImmLeaf<f32, [{  return true; }], fimm_to_i32>;
 def ConstPseudoTrue: IntImmLeaf<i32, [{ return Imm.getBitWidth() == 1 && Imm.getZExtValue() == 1; }]>;
 def ConstPseudoFalse: IntImmLeaf<i32, [{ return Imm.getBitWidth() == 1 && Imm.getZExtValue() == 0; }]>;
-def ConstPseudoNull: IntImmLeaf<i64, [{ return Imm.isNullValue(); }]>;
+def ConstPseudoNull: IntImmLeaf<i64, [{ return Imm.isZero(); }]>;
 
 multiclass IntFPImm<bits<16> opCode, string name> {
   def I: Op<opCode, (outs ID:$dst), (ins TYPE:$type, ID:$src, variable_ops),
diff --git a/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp b/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
index f48b7032ba4f8..5507e9254faee 100644
--- a/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
@@ -1080,8 +1080,8 @@ Register SPIRVInstructionSelector::buildOnesVal(bool AllOnes,
                                                 const SPIRVType *ResType,
                                                 MachineInstr &I) const {
   unsigned BitWidth = GR.getScalarOrVectorBitWidth(ResType);
-  APInt One = AllOnes ? APInt::getAllOnesValue(BitWidth)
-                      : APInt::getOneBitSet(BitWidth, 0);
+  APInt One =
+      AllOnes ? APInt::getAllOnes(BitWidth) : APInt::getOneBitSet(BitWidth, 0);
   if (ResType->getOpcode() == SPIRV::OpTypeVector)
     return GR.getOrCreateConsIntVector(One.getZExtValue(), I, ResType, TII);
   return GR.getOrCreateConstInt(One.getZExtValue(), I, ResType, TII);
@@ -1180,15 +1180,16 @@ bool SPIRVInstructionSelector::selectConst(Register ResVReg,
                                            const APInt &Imm,
                                            MachineInstr &I) const {
   unsigned TyOpcode = ResType->getOpcode();
-  assert(TyOpcode != SPIRV::OpTypePointer || Imm.isNullValue());
+  assert(TyOpcode != SPIRV::OpTypePointer || Imm.isZero());
   MachineBasicBlock &BB = *I.getParent();
   if ((TyOpcode == SPIRV::OpTypePointer || TyOpcode == SPIRV::OpTypeEvent) &&
-      Imm.isNullValue())
+      Imm.isZero())
     return BuildMI(BB, I, I.getDebugLoc(), TII.get(SPIRV::OpConstantNull))
         .addDef(ResVReg)
         .addUse(GR.getSPIRVTypeID(ResType))
         .constrainAllUses(TII, TRI, RBI);
   if (TyOpcode == SPIRV::OpTypeInt) {
+    assert(Imm.getBitWidth() <= 64 && "Unsupported integer width!");
     Register Reg = GR.getOrCreateConstInt(Imm.getZExtValue(), I, ResType, TII);
     if (Reg == ResVReg)
       return true;
diff --git a/llvm/lib/Target/SPIRV/SPIRVPreLegalizer.cpp b/llvm/lib/Target/SPIRV/SPIRVPreLegalizer.cpp
index 27d0e8a976f0d..2818329ece3cb 100644
--- a/llvm/lib/Target/SPIRV/SPIRVPreLegalizer.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVPreLegalizer.cpp
@@ -85,6 +85,9 @@ static void addConstantsToTrack(MachineFunction &MF, SPIRVGlobalRegistry *GR) {
     Register Reg = MI->getOperand(2).getReg();
     if (RegsAlreadyAddedToDT.find(MI) != RegsAlreadyAddedToDT.end())
       Reg = RegsAlreadyAddedToDT[MI];
+    auto *RC = MRI.getRegClassOrNull(MI->getOperand(0).getReg());
+    if (!MRI.getRegClassOrNull(Reg) && RC)
+      MRI.setRegClass(Reg, RC);
     MRI.replaceRegWith(MI->getOperand(0).getReg(), Reg);
     MI->eraseFromParent();
   }
@@ -201,8 +204,12 @@ Register insertAssignInstr(Register Reg, Type *Ty, SPIRVType *SpirvTy,
                   (Def->getNextNode() ? Def->getNextNode()->getIterator()
                                       : Def->getParent()->end()));
   Register NewReg = MRI.createGenericVirtualRegister(MRI.getType(Reg));
-  if (auto *RC = MRI.getRegClassOrNull(Reg))
+  if (auto *RC = MRI.getRegClassOrNull(Reg)) {
     MRI.setRegClass(NewReg, RC);
+  } else {
+    MRI.setRegClass(NewReg, &SPIRV::IDRegClass);
+    MRI.setRegClass(Reg, &SPIRV::IDRegClass);
+  }
   SpirvTy = SpirvTy ? SpirvTy : GR->getOrCreateSPIRVType(Ty, MIB);
   GR->assignSPIRVTypeToVReg(SpirvTy, Reg, MIB.getMF());
   // This is to make it convenient for Legalizer to get the SPIRVType
@@ -217,7 +224,6 @@ Register insertAssignInstr(Register Reg, Type *Ty, SPIRVType *SpirvTy,
       .addUse(GR->getSPIRVTypeID(SpirvTy))
       .setMIFlags(Flags);
   Def->getOperand(0).setReg(NewReg);
-  MRI.setRegClass(Reg, &SPIRV::ANYIDRegClass);
   return NewReg;
 }
 } // namespace llvm
diff --git a/llvm/lib/Target/SPIRV/SPIRVPrepareFunctions.cpp b/llvm/lib/Target/SPIRV/SPIRVPrepareFunctions.cpp
index 20f32ffeba3bf..554e66988f090 100644
--- a/llvm/lib/Target/SPIRV/SPIRVPrepareFunctions.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVPrepareFunctions.cpp
@@ -7,8 +7,11 @@
 //===----------------------------------------------------------------------===//
 //
 // This pass modifies function signatures containing aggregate arguments
-// and/or return value. Also it substitutes some llvm intrinsic calls by
-// function calls, generating these functions as the translator does.
+// and/or return value before IRTranslator. Information about the original
+// signatures is stored in metadata. It is used during call lowering to
+// restore correct SPIR-V types of function arguments and return values.
+// This pass also substitutes some llvm intrinsic calls with calls to newly
+// generated functions (as the Khronos LLVM/SPIR-V Translator does).
 //
 // NOTE: this pass is a module-level one due to the necessity to modify
 // GVs/functions.
@@ -33,7 +36,8 @@ void initializeSPIRVPrepareFunctionsPass(PassRegistry &);
 namespace {
 
 class SPIRVPrepareFunctions : public ModulePass {
-  Function *processFunctionSignature(Function *F);
+  bool substituteIntrinsicCalls(Function *F);
+  Function *removeAggregateTypesFromSignature(Function *F);
 
 public:
   static char ID;
@@ -57,68 +61,6 @@ char SPIRVPrepareFunctions::ID = 0;
 INITIALIZE_PASS(SPIRVPrepareFunctions, "prepare-functions",
                 "SPIRV prepare functions", false, false)
 
-Function *SPIRVPrepareFunctions::processFunctionSignature(Function *F) {
-  IRBuilder<> B(F->getContext());
-
-  bool IsRetAggr = F->getReturnType()->isAggregateType();
-  bool HasAggrArg =
-      std::any_of(F->arg_begin(), F->arg_end(), [](Argument &Arg) {
-        return Arg.getType()->isAggregateType();
-      });
-  bool DoClone = IsRetAggr || HasAggrArg;
-  if (!DoClone)
-    return F;
-  SmallVector<std::pair<int, Type *>, 4> ChangedTypes;
-  Type *RetType = IsRetAggr ? B.getInt32Ty() : F->getReturnType();
-  if (IsRetAggr)
-    ChangedTypes.push_back(std::pair<int, Type *>(-1, F->getReturnType()));
-  SmallVector<Type *, 4> ArgTypes;
-  for (const auto &Arg : F->args()) {
-    if (Arg.getType()->isAggregateType()) {
-      ArgTypes.push_back(B.getInt32Ty());
-      ChangedTypes.push_back(
-          std::pair<int, Type *>(Arg.getArgNo(), Arg.getType()));
-    } else
-      ArgTypes.push_back(Arg.getType());
-  }
-  FunctionType *NewFTy =
-      FunctionType::get(RetType, ArgTypes, F->getFunctionType()->isVarArg());
-  Function *NewF =
-      Function::Create(NewFTy, F->getLinkage(), F->getName(), *F->getParent());
-
-  ValueToValueMapTy VMap;
-  auto NewFArgIt = NewF->arg_begin();
-  for (auto &Arg : F->args()) {
-    StringRef ArgName = Arg.getName();
-    NewFArgIt->setName(ArgName);
-    VMap[&Arg] = &(*NewFArgIt++);
-  }
-  SmallVector<ReturnInst *, 8> Returns;
-
-  CloneFunctionInto(NewF, F, VMap, CloneFunctionChangeType::LocalChangesOnly,
-                    Returns);
-  NewF->takeName(F);
-
-  NamedMDNode *FuncMD =
-      F->getParent()->getOrInsertNamedMetadata("spv.cloned_funcs");
-  SmallVector<Metadata *, 2> MDArgs;
-  MDArgs.push_back(MDString::get(B.getContext(), NewF->getName()));
-  for (auto &ChangedTyP : ChangedTypes)
-    MDArgs.push_back(MDNode::get(
-        B.getContext(),
-        {ConstantAsMetadata::get(B.getInt32(ChangedTyP.first)),
-         ValueAsMetadata::get(Constant::getNullValue(ChangedTyP.second))}));
-  MDNode *ThisFuncMD = MDNode::get(B.getContext(), MDArgs);
-  FuncMD->addOperand(ThisFuncMD);
-
-  for (auto *U : make_early_inc_range(F->users())) {
-    if (auto *CI = dyn_cast<CallInst>(U))
-      CI->mutateFunctionType(NewF->getFunctionType());
-    U->replaceUsesOfWith(F, NewF);
-  }
-  return NewF;
-}
-
 std::string lowerLLVMIntrinsicName(IntrinsicInst *II) {
   Function *IntrinsicFunc = II->getCalledFunction();
   assert(IntrinsicFunc && "Missing function");
@@ -142,15 +84,16 @@ static Function *getOrCreateFunction(Module *M, Type *RetTy,
   return NewF;
 }
 
-static void lowerIntrinsicToFunction(Module *M, IntrinsicInst *Intrinsic) {
+static bool lowerIntrinsicToFunction(IntrinsicInst *Intrinsic) {
   // For @llvm.memset.* intrinsic cases with constant value and length arguments
   // are emulated via "storing" a constant array to the destination. For other
   // cases we wrap the intrinsic in @spirv.llvm_memset_* function and expand the
   // intrinsic to a loop via expandMemSetAsLoop().
   if (auto *MSI = dyn_cast<MemSetInst>(Intrinsic))
     if (isa<Constant>(MSI->getValue()) && isa<ConstantInt>(MSI->getLength()))
-      return; // It is handled later using OpCopyMemorySized.
+      return false; // It is handled later using OpCopyMemorySized.
 
+  Module *M = Intrinsic->getModule();
   std::string FuncName = lowerLLVMIntrinsicName(Intrinsic);
   if (Intrinsic->isVolatile())
     FuncName += ".volatile";
@@ -158,7 +101,7 @@ static void lowerIntrinsicToFunction(Module *M, IntrinsicInst *Intrinsic) {
   Function *F = M->getFunction(FuncName);
   if (F) {
     Intrinsic->setCalledFunction(F);
-    return;
+    return true;
   }
   // TODO copy arguments attributes: nocapture writeonly.
   FunctionCallee FC =
@@ -202,14 +145,15 @@ static void lowerIntrinsicToFunction(Module *M, IntrinsicInst *Intrinsic) {
   default:
     break;
   }
-  return;
+  return true;
 }
 
-static void lowerFunnelShifts(Module *M, IntrinsicInst *FSHIntrinsic) {
+static void lowerFunnelShifts(IntrinsicInst *FSHIntrinsic) {
   // Get a separate function - otherwise, we'd have to rework the CFG of the
   // current one. Then simply replace the intrinsic uses with a call to the new
   // function.
   // Generate LLVM IR for  i* @spirv.llvm_fsh?_i* (i* %a, i* %b, i* %c)
+  Module *M = FSHIntrinsic->getModule();
   FunctionType *FSHFuncTy = FSHIntrinsic->getFunctionType();
   Type *FSHRetTy = FSHFuncTy->getReturnType();
   const std::string FuncName = lowerLLVMIntrinsicName(FSHIntrinsic);
@@ -265,12 +209,13 @@ static void lowerFunnelShifts(Module *M, IntrinsicInst *FSHIntrinsic) {
   FSHIntrinsic->setCalledFunction(FSHFunc);
 }
 
-static void buildUMulWithOverflowFunc(Module *M, Function *UMulFunc) {
+static void buildUMulWithOverflowFunc(Function *UMulFunc) {
   // The function body is already created.
   if (!UMulFunc->empty())
     return;
 
-  BasicBlock *EntryBB = BasicBlock::Create(M->getContext(), "entry", UMulFunc);
+  BasicBlock *EntryBB = BasicBlock::Create(UMulFunc->getParent()->getContext(),
+                                           "entry", UMulFunc);
   IRBuilder<> IRB(EntryBB);
   // Build the actual unsigned multiplication logic with the overflow
   // indication. Do unsigned multiplication Mul = A * B. Then check
@@ -288,65 +233,132 @@ static void buildUMulWithOverflowFunc(Module *M, Function *UMulFunc) {
   IRB.CreateRet(Res);
 }
 
-static void lowerUMulWithOverflow(Module *M, IntrinsicInst *UMulIntrinsic) {
+static void lowerUMulWithOverflow(IntrinsicInst *UMulIntrinsic) {
   // Get a separate function - otherwise, we'd have to rework the CFG of the
   // current one. Then simply replace the intrinsic uses with a call to the new
   // function.
+  Module *M = UMulIntrinsic->getModule();
   FunctionType *UMulFuncTy = UMulIntrinsic->getFunctionType();
   Type *FSHLRetTy = UMulFuncTy->getReturnType();
   const std::string FuncName = lowerLLVMIntrinsicName(UMulIntrinsic);
   Function *UMulFunc =
       getOrCreateFunction(M, FSHLRetTy, UMulFuncTy->params(), FuncName);
-  buildUMulWithOverflowFunc(M, UMulFunc);
+  buildUMulWithOverflowFunc(UMulFunc);
   UMulIntrinsic->setCalledFunction(UMulFunc);
 }
 
-static void substituteIntrinsicCalls(Module *M, Function *F) {
+// Substitutes calls to LLVM intrinsics with either calls to SPIR-V intrinsics
+// or calls to proper generated functions. Returns True if F was modified.
+bool SPIRVPrepareFunctions::substituteIntrinsicCalls(Function *F) {
+  bool Changed = false;
   for (BasicBlock &BB : *F) {
     for (Instruction &I : BB) {
       auto Call = dyn_cast<CallInst>(&I);
       if (!Call)
         continue;
-      Call->setTailCall(false);
       Function *CF = Call->getCalledFunction();
       if (!CF || !CF->isIntrinsic())
         continue;
       auto *II = cast<IntrinsicInst>(Call);
       if (II->getIntrinsicID() == Intrinsic::memset ||
           II->getIntrinsicID() == Intrinsic::bswap)
-        lowerIntrinsicToFunction(M, II);
+        Changed |= lowerIntrinsicToFunction(II);
       else if (II->getIntrinsicID() == Intrinsic::fshl ||
-               II->getIntrinsicID() == Intrinsic::fshr)
-        lowerFunnelShifts(M, II);
-      else if (II->getIntrinsicID() == Intrinsic::umul_with_overflow)
-        lowerUMulWithOverflow(M, II);
+               II->getIntrinsicID() == Intrinsic::fshr) {
+        lowerFunnelShifts(II);
+        Changed = true;
+      } else if (II->getIntrinsicID() == Intrinsic::umul_with_overflow) {
+        lowerUMulWithOverflow(II);
+        Changed = true;
+      }
     }
   }
+  return Changed;
+}
+
+// Returns F if aggregate argument/return types are not present or cloned F
+// function with the types replaced by i32 types. The change in types is
+// noted in 'spv.cloned_funcs' metadata for later restoration.
+Function *
+SPIRVPrepareFunctions::removeAggregateTypesFromSignature(Function *F) {
+  IRBuilder<> B(F->getContext());
+
+  bool IsRetAggr = F->getReturnType()->isAggregateType();
+  bool HasAggrArg =
+      std::any_of(F->arg_begin(), F->arg_end(), [](Argument &Arg) {
+        return Arg.getType()->isAggregateType();
+      });
+  bool DoClone = IsRetAggr || HasAggrArg;
+  if (!DoClone)
+    return F;
+  SmallVector<std::pair<int, Type *>, 4> ChangedTypes;
+  Type *RetType = IsRetAggr ? B.getInt32Ty() : F->getReturnType();
+  if (IsRetAggr)
+    ChangedTypes.push_back(std::pair<int, Type *>(-1, F->getReturnType()));
+  SmallVector<Type *, 4> ArgTypes;
+  for (const auto &Arg : F->args()) {
+    if (Arg.getType()->isAggregateType()) {
+      ArgTypes.push_back(B.getInt32Ty());
+      ChangedTypes.push_back(
+          std::pair<int, Type *>(Arg.getArgNo(), Arg.getType()));
+    } else
+      ArgTypes.push_back(Arg.getType());
+  }
+  FunctionType *NewFTy =
+      FunctionType::get(RetType, ArgTypes, F->getFunctionType()->isVarArg());
+  Function *NewF =
+      Function::Create(NewFTy, F->getLinkage(), F->getName(), *F->getParent());
+
+  ValueToValueMapTy VMap;
+  auto NewFArgIt = NewF->arg_begin();
+  for (auto &Arg : F->args()) {
+    StringRef ArgName = Arg.getName();
+    NewFArgIt->setName(ArgName);
+    VMap[&Arg] = &(*NewFArgIt++);
+  }
+  SmallVector<ReturnInst *, 8> Returns;
+
+  CloneFunctionInto(NewF, F, VMap, CloneFunctionChangeType::LocalChangesOnly,
+                    Returns);
+  NewF->takeName(F);
+
+  NamedMDNode *FuncMD =
+      F->getParent()->getOrInsertNamedMetadata("spv.cloned_funcs");
+  SmallVector<Metadata *, 2> MDArgs;
+  MDArgs.push_back(MDString::get(B.getContext(), NewF->getName()));
+  for (auto &ChangedTyP : ChangedTypes)
+    MDArgs.push_back(MDNode::get(
+        B.getContext(),
+        {ConstantAsMetadata::get(B.getInt32(ChangedTyP.first)),
+         ValueAsMetadata::get(Constant::getNullValue(ChangedTyP.second))}));
+  MDNode *ThisFuncMD = MDNode::get(B.getContext(), MDArgs);
+  FuncMD->addOperand(ThisFuncMD);
+
+  for (auto *U : make_early_inc_range(F->users())) {
+    if (auto *CI = dyn_cast<CallInst>(U))
+      CI->mutateFunctionType(NewF->getFunctionType());
+    U->replaceUsesOfWith(F, NewF);
+  }
+  return NewF;
 }
 
 bool SPIRVPrepareFunctions::runOnModule(Module &M) {
+  bool Changed = false;
   for (Function &F : M)
-    substituteIntrinsicCalls(&M, &F);
+    Changed |= substituteIntrinsicCalls(&F);
 
   std::vector<Function *> FuncsWorklist;
-  bool Changed = false;
   for (auto &F : M)
     FuncsWorklist.push_back(&F);
 
-  for (auto *Func : FuncsWorklist) {
-    Function *F = processFunctionSignature(Func);
-
-    bool CreatedNewF = F != Func;
+  for (auto *F : FuncsWorklist) {
+    Function *NewF = removeAggregateTypesFromSignature(F);
 
-    if (Func->isDeclaration()) {
-      Changed |= CreatedNewF;
-      continue;
+    if (NewF != F) {
+      F->eraseFromParent();
+      Changed = true;
     }
-
-    if (CreatedNewF)
-      Func->eraseFromParent();
   }
-
   return Changed;
 }
 
diff --git a/llvm/lib/Target/SPIRV/SPIRVUtils.cpp b/llvm/lib/Target/SPIRV/SPIRVUtils.cpp
index b6353ec2fb6b4..514dc6497c470 100644
--- a/llvm/lib/Target/SPIRV/SPIRVUtils.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVUtils.cpp
@@ -77,24 +77,19 @@ std::string getStringImm(const MachineInstr &MI, unsigned StartIndex) {
 
 void addNumImm(const APInt &Imm, MachineInstrBuilder &MIB) {
   const auto Bitwidth = Imm.getBitWidth();
-  switch (Bitwidth) {
-  case 1:
-    break; // Already handled.
-  case 8:
-  case 16:
-  case 32:
+  if (Bitwidth == 1)
+    return; // Already handled
+  else if (Bitwidth <= 32) {
     MIB.addImm(Imm.getZExtValue());
-    break;
-  case 64: {
+    return;
+  } else if (Bitwidth <= 64) {
     uint64_t FullImm = Imm.getZExtValue();
     uint32_t LowBits = FullImm & 0xffffffff;
     uint32_t HighBits = (FullImm >> 32) & 0xffffffff;
     MIB.addImm(LowBits).addImm(HighBits);
-    break;
-  }
-  default:
-    report_fatal_error("Unsupported constant bitwidth");
+    return;
   }
+  report_fatal_error("Unsupported constant bitwidth");
 }
 
 void buildOpName(Register Target, const StringRef &Name,
diff --git a/llvm/lib/Target/Sparc/MCTargetDesc/SparcInstPrinter.cpp b/llvm/lib/Target/Sparc/MCTargetDesc/SparcInstPrinter.cpp
index c78260d38c4ee..b2e784a0c2545 100644
--- a/llvm/lib/Target/Sparc/MCTargetDesc/SparcInstPrinter.cpp
+++ b/llvm/lib/Target/Sparc/MCTargetDesc/SparcInstPrinter.cpp
@@ -178,6 +178,8 @@ void SparcInstPrinter::printCCOperand(const MCInst *MI, int opNum,
   default: break;
   case SP::FBCOND:
   case SP::FBCONDA:
+  case SP::FBCOND_V9:
+  case SP::FBCONDA_V9:
   case SP::BPFCC:
   case SP::BPFCCA:
   case SP::BPFCCNT:
diff --git a/llvm/lib/Target/Sparc/SparcInstrInfo.cpp b/llvm/lib/Target/Sparc/SparcInstrInfo.cpp
index 63f662c41f93c..a3a09a36f1dd4 100644
--- a/llvm/lib/Target/Sparc/SparcInstrInfo.cpp
+++ b/llvm/lib/Target/Sparc/SparcInstrInfo.cpp
@@ -28,6 +28,10 @@ using namespace llvm;
 #define GET_INSTRINFO_CTOR_DTOR
 #include "SparcGenInstrInfo.inc"
 
+static cl::opt<unsigned> BPccDisplacementBits(
+    "sparc-bpcc-offset-bits", cl::Hidden, cl::init(19),
+    cl::desc("Restrict range of BPcc/FBPfcc instructions (DEBUG)"));
+
 // Pin the vtable to this file.
 void SparcInstrInfo::anchor() {}
 
@@ -73,11 +77,6 @@ unsigned SparcInstrInfo::isStoreToStackSlot(const MachineInstr &MI,
   return 0;
 }
 
-static bool IsIntegerCC(unsigned CC)
-{
-  return  (CC <= SPCC::ICC_VC);
-}
-
 static SPCC::CondCodes GetOppositeBranchCondition(SPCC::CondCodes CC)
 {
   switch(CC) {
@@ -155,9 +154,7 @@ static SPCC::CondCodes GetOppositeBranchCondition(SPCC::CondCodes CC)
   llvm_unreachable("Invalid cond code");
 }
 
-static bool isUncondBranchOpcode(int Opc) {
-  return Opc == SP::BA || Opc == SP::BPA;
-}
+static bool isUncondBranchOpcode(int Opc) { return Opc == SP::BA; }
 
 static bool isI32CondBranchOpcode(int Opc) {
   return Opc == SP::BCOND || Opc == SP::BPICC || Opc == SP::BPICCA ||
@@ -169,7 +166,10 @@ static bool isI64CondBranchOpcode(int Opc) {
          Opc == SP::BPXCCANT;
 }
 
-static bool isFCondBranchOpcode(int Opc) { return Opc == SP::FBCOND; }
+static bool isFCondBranchOpcode(int Opc) {
+  return Opc == SP::FBCOND || Opc == SP::FBCONDA || Opc == SP::FBCOND_V9 ||
+         Opc == SP::FBCONDA_V9;
+}
 
 static bool isCondBranchOpcode(int Opc) {
   return isI32CondBranchOpcode(Opc) || isI64CondBranchOpcode(Opc) ||
@@ -193,6 +193,34 @@ static void parseCondBranch(MachineInstr *LastInst, MachineBasicBlock *&Target,
   Target = LastInst->getOperand(0).getMBB();
 }
 
+MachineBasicBlock *
+SparcInstrInfo::getBranchDestBlock(const MachineInstr &MI) const {
+  switch (MI.getOpcode()) {
+  default:
+      llvm_unreachable("unexpected opcode!");
+  case SP::BA:
+  case SP::BCOND:
+  case SP::BCONDA:
+  case SP::FBCOND:
+  case SP::FBCONDA:
+  case SP::BPICC:
+  case SP::BPICCA:
+  case SP::BPICCNT:
+  case SP::BPICCANT:
+  case SP::BPXCC:
+  case SP::BPXCCA:
+  case SP::BPXCCNT:
+  case SP::BPXCCANT:
+  case SP::BPFCC:
+  case SP::BPFCCA:
+  case SP::BPFCCNT:
+  case SP::BPFCCANT:
+  case SP::FBCOND_V9:
+  case SP::FBCONDA_V9:
+      return MI.getOperand(0).getMBB();
+  }
+}
+
 bool SparcInstrInfo::analyzeBranch(MachineBasicBlock &MBB,
                                    MachineBasicBlock *&TBB,
                                    MachineBasicBlock *&FBB,
@@ -285,36 +313,37 @@ unsigned SparcInstrInfo::insertBranch(MachineBasicBlock &MBB,
   assert(TBB && "insertBranch must not be told to insert a fallthrough");
   assert((Cond.size() <= 2) &&
          "Sparc branch conditions should have at most two components!");
-  assert(!BytesAdded && "code size not handled");
 
   if (Cond.empty()) {
     assert(!FBB && "Unconditional branch with multiple successors!");
-    BuildMI(&MBB, DL, get(Subtarget.isV9() ? SP::BPA : SP::BA)).addMBB(TBB);
+    BuildMI(&MBB, DL, get(SP::BA)).addMBB(TBB);
+    if (BytesAdded)
+      *BytesAdded = 8;
     return 1;
   }
 
   // Conditional branch
   unsigned Opc = Cond[0].getImm();
   unsigned CC = Cond[1].getImm();
+  BuildMI(&MBB, DL, get(Opc)).addMBB(TBB).addImm(CC);
 
-  if (IsIntegerCC(CC)) {
-    BuildMI(&MBB, DL, get(Opc)).addMBB(TBB).addImm(CC);
-  } else {
-    BuildMI(&MBB, DL, get(SP::FBCOND)).addMBB(TBB).addImm(CC);
-  }
-  if (!FBB)
+  if (!FBB) {
+    if (BytesAdded)
+      *BytesAdded = 8;
     return 1;
+  }
 
-  BuildMI(&MBB, DL, get(Subtarget.isV9() ? SP::BPA : SP::BA)).addMBB(FBB);
+  BuildMI(&MBB, DL, get(SP::BA)).addMBB(FBB);
+  if (BytesAdded)
+    *BytesAdded = 16;
   return 2;
 }
 
 unsigned SparcInstrInfo::removeBranch(MachineBasicBlock &MBB,
                                       int *BytesRemoved) const {
-  assert(!BytesRemoved && "code size not handled");
-
   MachineBasicBlock::iterator I = MBB.end();
   unsigned Count = 0;
+  int Removed = 0;
   while (I != MBB.begin()) {
     --I;
 
@@ -325,10 +354,14 @@ unsigned SparcInstrInfo::removeBranch(MachineBasicBlock &MBB,
         !isUncondBranchOpcode(I->getOpcode()))
       break; // Not a branch
 
+    Removed += getInstSizeInBytes(*I);
     I->eraseFromParent();
     I = MBB.end();
     ++Count;
   }
+
+  if (BytesRemoved)
+    *BytesRemoved = Removed;
   return Count;
 }
 
@@ -340,6 +373,37 @@ bool SparcInstrInfo::reverseBranchCondition(
   return false;
 }
 
+bool SparcInstrInfo::isBranchOffsetInRange(unsigned BranchOpc,
+                                           int64_t Offset) const {
+  assert((Offset & 0b11) == 0 && "Malformed branch offset");
+  switch (BranchOpc) {
+  case SP::BA:
+  case SP::BCOND:
+  case SP::BCONDA:
+  case SP::FBCOND:
+  case SP::FBCONDA:
+    return isIntN(22, Offset >> 2);
+
+  case SP::BPICC:
+  case SP::BPICCA:
+  case SP::BPICCNT:
+  case SP::BPICCANT:
+  case SP::BPXCC:
+  case SP::BPXCCA:
+  case SP::BPXCCNT:
+  case SP::BPXCCANT:
+  case SP::BPFCC:
+  case SP::BPFCCA:
+  case SP::BPFCCNT:
+  case SP::BPFCCANT:
+  case SP::FBCOND_V9:
+  case SP::FBCONDA_V9:
+    return isIntN(BPccDisplacementBits, Offset >> 2);
+  }
+
+  llvm_unreachable("Unknown branch instruction!");
+}
+
 void SparcInstrInfo::copyPhysReg(MachineBasicBlock &MBB,
                                  MachineBasicBlock::iterator I,
                                  const DebugLoc &DL, MCRegister DestReg,
@@ -530,6 +594,23 @@ Register SparcInstrInfo::getGlobalBaseReg(MachineFunction *MF) const {
   return GlobalBaseReg;
 }
 
+unsigned SparcInstrInfo::getInstSizeInBytes(const MachineInstr &MI) const {
+  unsigned Opcode = MI.getOpcode();
+
+  if (MI.isInlineAsm()) {
+    const MachineFunction *MF = MI.getParent()->getParent();
+    const char *AsmStr = MI.getOperand(0).getSymbolName();
+    return getInlineAsmLength(AsmStr, *MF->getTarget().getMCAsmInfo());
+  }
+
+  // If the instruction has a delay slot, be conservative and also include
+  // it for sizing purposes. This is done so that the BranchRelaxation pass
+  // will not mistakenly mark out-of-range branches as in-range.
+  if (MI.hasDelaySlot())
+    return get(Opcode).getSize() * 2;
+  return get(Opcode).getSize();
+}
+
 bool SparcInstrInfo::expandPostRAPseudo(MachineInstr &MI) const {
   switch (MI.getOpcode()) {
   case TargetOpcode::LOAD_STACK_GUARD: {
diff --git a/llvm/lib/Target/Sparc/SparcInstrInfo.h b/llvm/lib/Target/Sparc/SparcInstrInfo.h
index 39cf791c21730..7056d6babe17b 100644
--- a/llvm/lib/Target/Sparc/SparcInstrInfo.h
+++ b/llvm/lib/Target/Sparc/SparcInstrInfo.h
@@ -64,6 +64,8 @@ class SparcInstrInfo : public SparcGenInstrInfo {
   unsigned isStoreToStackSlot(const MachineInstr &MI,
                               int &FrameIndex) const override;
 
+  MachineBasicBlock *getBranchDestBlock(const MachineInstr &MI) const override;
+
   bool analyzeBranch(MachineBasicBlock &MBB, MachineBasicBlock *&TBB,
                      MachineBasicBlock *&FBB,
                      SmallVectorImpl<MachineOperand> &Cond,
@@ -80,6 +82,9 @@ class SparcInstrInfo : public SparcGenInstrInfo {
   bool
   reverseBranchCondition(SmallVectorImpl<MachineOperand> &Cond) const override;
 
+  /// Determine if the branch target is in range.
+  bool isBranchOffsetInRange(unsigned BranchOpc, int64_t Offset) const override;
+
   void copyPhysReg(MachineBasicBlock &MBB, MachineBasicBlock::iterator I,
                    const DebugLoc &DL, MCRegister DestReg, MCRegister SrcReg,
                    bool KillSrc) const override;
@@ -99,6 +104,10 @@ class SparcInstrInfo : public SparcGenInstrInfo {
 
   Register getGlobalBaseReg(MachineFunction *MF) const;
 
+  /// GetInstSize - Return the number of bytes of code the specified
+  /// instruction may be.  This returns the maximum number of bytes.
+  unsigned getInstSizeInBytes(const MachineInstr &MI) const override;
+
   // Lower pseudo instructions after register allocation.
   bool expandPostRAPseudo(MachineInstr &MI) const override;
 };
diff --git a/llvm/lib/Target/Sparc/SparcInstrInfo.td b/llvm/lib/Target/Sparc/SparcInstrInfo.td
index 2c45a7218d045..2e95bc10337a2 100644
--- a/llvm/lib/Target/Sparc/SparcInstrInfo.td
+++ b/llvm/lib/Target/Sparc/SparcInstrInfo.td
@@ -850,15 +850,8 @@ class BranchPredictAlways<dag ins, string asmstr, list<dag> pattern>
   : F2_3<0b001, 0, 1, (outs), ins, asmstr, pattern>;
 }
 
-let cond = 8 in {
-  // If we're compiling for v9, prefer BPA rather than BA
-  // TODO: Disallow BA emission when FeatureV8Deprecated isn't enabled
-  let Predicates = [HasV9], cc = 0b00 in
-    def BPA : BranchPredictAlways<(ins bprtarget:$imm19),
-      "ba %icc, $imm19", [(br bb:$imm19)]>;
-
+let cond = 8 in
   def BA : BranchAlways<(ins brtarget:$imm22), "ba $imm22", [(br bb:$imm22)]>;
-}
 
 let isBranch = 1, isTerminator = 1, hasDelaySlot = 1 in {
 
diff --git a/llvm/lib/Target/Sparc/SparcTargetMachine.cpp b/llvm/lib/Target/Sparc/SparcTargetMachine.cpp
index 082bb44faaa32..577dc1351de9b 100644
--- a/llvm/lib/Target/Sparc/SparcTargetMachine.cpp
+++ b/llvm/lib/Target/Sparc/SparcTargetMachine.cpp
@@ -31,6 +31,10 @@ extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeSparcTarget() {
   initializeSparcDAGToDAGISelPass(PR);
 }
 
+static cl::opt<bool>
+    BranchRelaxation("sparc-enable-branch-relax", cl::Hidden, cl::init(true),
+                     cl::desc("Relax out of range conditional branches"));
+
 static std::string computeDataLayout(const Triple &T, bool is64Bit) {
   // Sparc is typically big endian, but some are little.
   std::string Ret = T.getArch() == Triple::sparcel ? "e" : "E";
@@ -181,6 +185,9 @@ bool SparcPassConfig::addInstSelector() {
 }
 
 void SparcPassConfig::addPreEmitPass(){
+  if (BranchRelaxation)
+    addPass(&BranchRelaxationPassID);
+
   addPass(createSparcDelaySlotFillerPass());
 
   if (this->getSparcTargetMachine().getSubtargetImpl()->insertNOPLoad())
diff --git a/llvm/lib/Target/SystemZ/SystemZISelLowering.cpp b/llvm/lib/Target/SystemZ/SystemZISelLowering.cpp
index 1fb05fd32bb29..dd5b885701fa9 100644
--- a/llvm/lib/Target/SystemZ/SystemZISelLowering.cpp
+++ b/llvm/lib/Target/SystemZ/SystemZISelLowering.cpp
@@ -1265,12 +1265,10 @@ SystemZTargetLowering::getRegForInlineAsmConstraint(
 Register
 SystemZTargetLowering::getRegisterByName(const char *RegName, LLT VT,
                                          const MachineFunction &MF) const {
-  const SystemZSubtarget *Subtarget = &MF.getSubtarget<SystemZSubtarget>();
-
   Register Reg =
       StringSwitch<Register>(RegName)
-          .Case("r4", Subtarget->isTargetXPLINK64() ? SystemZ::R4D : 0)
-          .Case("r15", Subtarget->isTargetELF() ? SystemZ::R15D : 0)
+          .Case("r4", Subtarget.isTargetXPLINK64() ? SystemZ::R4D : 0)
+          .Case("r15", Subtarget.isTargetELF() ? SystemZ::R15D : 0)
           .Default(0);
 
   if (Reg)
@@ -2425,6 +2423,12 @@ static void adjustForSubtraction(SelectionDAG &DAG, const SDLoc &DL,
       if (N->getOpcode() == ISD::SUB &&
           ((N->getOperand(0) == C.Op0 && N->getOperand(1) == C.Op1) ||
            (N->getOperand(0) == C.Op1 && N->getOperand(1) == C.Op0))) {
+        // Disable the nsw and nuw flags: the backend needs to handle
+        // overflow as well during comparison elimination.
+        SDNodeFlags Flags = N->getFlags();
+        Flags.setNoSignedWrap(false);
+        Flags.setNoUnsignedWrap(false);
+        N->setFlags(Flags);
         C.Op0 = SDValue(N, 0);
         C.Op1 = DAG.getConstant(0, DL, N->getValueType(0));
         return;
@@ -4375,8 +4379,7 @@ SystemZTargetLowering::getTargetMMOFlags(const Instruction &I) const {
 SDValue SystemZTargetLowering::lowerSTACKSAVE(SDValue Op,
                                               SelectionDAG &DAG) const {
   MachineFunction &MF = DAG.getMachineFunction();
-  const SystemZSubtarget *Subtarget = &MF.getSubtarget<SystemZSubtarget>();
-  auto *Regs = Subtarget->getSpecialRegisters();
+  auto *Regs = Subtarget.getSpecialRegisters();
   if (MF.getFunction().getCallingConv() == CallingConv::GHC)
     report_fatal_error("Variable-sized stack allocations are not supported "
                        "in GHC calling convention");
@@ -4387,8 +4390,7 @@ SDValue SystemZTargetLowering::lowerSTACKSAVE(SDValue Op,
 SDValue SystemZTargetLowering::lowerSTACKRESTORE(SDValue Op,
                                                  SelectionDAG &DAG) const {
   MachineFunction &MF = DAG.getMachineFunction();
-  const SystemZSubtarget *Subtarget = &MF.getSubtarget<SystemZSubtarget>();
-  auto *Regs = Subtarget->getSpecialRegisters();
+  auto *Regs = Subtarget.getSpecialRegisters();
   bool StoreBackchain = MF.getFunction().hasFnAttribute("backchain");
 
   if (MF.getFunction().getCallingConv() == CallingConv::GHC)
@@ -7578,10 +7580,10 @@ static void createPHIsForSelects(SmallVector<MachineInstr*, 8> &Selects,
     if (MI->getOperand(4).getImm() == (CCValid ^ CCMask))
       std::swap(TrueReg, FalseReg);
 
-    if (RegRewriteTable.find(TrueReg) != RegRewriteTable.end())
+    if (RegRewriteTable.contains(TrueReg))
       TrueReg = RegRewriteTable[TrueReg].first;
 
-    if (RegRewriteTable.find(FalseReg) != RegRewriteTable.end())
+    if (RegRewriteTable.contains(FalseReg))
       FalseReg = RegRewriteTable[FalseReg].second;
 
     DebugLoc DL = MI->getDebugLoc();
diff --git a/llvm/lib/Target/WebAssembly/WebAssemblyDebugFixup.cpp b/llvm/lib/Target/WebAssembly/WebAssemblyDebugFixup.cpp
index 9a6acd157a745..f3f54a5fb501a 100644
--- a/llvm/lib/Target/WebAssembly/WebAssemblyDebugFixup.cpp
+++ b/llvm/lib/Target/WebAssembly/WebAssemblyDebugFixup.cpp
@@ -65,14 +65,14 @@ FunctionPass *llvm::createWebAssemblyDebugFixup() {
 // Because Wasm cannot access values in LLVM virtual registers in the debugger,
 // these dangling DBG_VALUEs in effect kill the effect of any previous DBG_VALUE
 // associated with the variable, which will appear as "optimized out".
-static void nullifyDanglingDebugValues(MachineBasicBlock &MBB,
-                                       const TargetInstrInfo *TII) {
+static void setDanglingDebugValuesUndef(MachineBasicBlock &MBB,
+                                        const TargetInstrInfo *TII) {
   for (auto &MI : llvm::make_early_inc_range(MBB)) {
     if (MI.isDebugValue() && MI.getDebugOperand(0).isReg() &&
         !MI.isUndefDebugValue()) {
-      LLVM_DEBUG(dbgs() << "Warning: dangling DBG_VALUE nullified: " << MI
+      LLVM_DEBUG(dbgs() << "Warning: dangling DBG_VALUE set to undef: " << MI
                         << "\n");
-      MI.getDebugOperand(0).setReg(Register());
+      MI.setDebugValueUndef();
     }
   }
 }
@@ -154,7 +154,7 @@ bool WebAssemblyDebugFixup::runOnMachineFunction(MachineFunction &MF) {
     assert(Stack.empty() &&
            "WebAssemblyDebugFixup: Stack not empty at end of basic block!");
 
-    nullifyDanglingDebugValues(MBB, TII);
+    setDanglingDebugValuesUndef(MBB, TII);
   }
 
   return true;
diff --git a/llvm/lib/Target/WebAssembly/WebAssemblyExplicitLocals.cpp b/llvm/lib/Target/WebAssembly/WebAssemblyExplicitLocals.cpp
index 08483b7935b6e..84fd34d73b634 100644
--- a/llvm/lib/Target/WebAssembly/WebAssemblyExplicitLocals.cpp
+++ b/llvm/lib/Target/WebAssembly/WebAssemblyExplicitLocals.cpp
@@ -267,15 +267,42 @@ bool WebAssemblyExplicitLocals::runOnMachineFunction(MachineFunction &MF) {
       // Replace tee instructions with local.tee. The difference is that tee
       // instructions have two defs, while local.tee instructions have one def
       // and an index of a local to write to.
+      //
+      // - Before:
+      // TeeReg, Reg = TEE DefReg
+      // INST ..., TeeReg, ...
+      // INST ..., Reg, ...
+      // INST ..., Reg, ...
+      // * DefReg: may or may not be stackified
+      // * Reg: not stackified
+      // * TeeReg: stackified
+      //
+      // - After (when DefReg was already stackified):
+      // TeeReg = LOCAL_TEE LocalId1, DefReg
+      // INST ..., TeeReg, ...
+      // INST ..., Reg, ...
+      // INST ..., Reg, ...
+      // * Reg: mapped to LocalId1
+      // * TeeReg: stackified
+      //
+      // - After (when DefReg was not already stackified):
+      // NewReg = LOCAL_GET LocalId1
+      // TeeReg = LOCAL_TEE LocalId2, NewReg
+      // INST ..., TeeReg, ...
+      // INST ..., Reg, ...
+      // INST ..., Reg, ...
+      // * DefReg: mapped to LocalId1
+      // * Reg: mapped to LocalId2
+      // * TeeReg: stackified
       if (WebAssembly::isTee(MI.getOpcode())) {
         assert(MFI.isVRegStackified(MI.getOperand(0).getReg()));
         assert(!MFI.isVRegStackified(MI.getOperand(1).getReg()));
-        Register OldReg = MI.getOperand(2).getReg();
-        const TargetRegisterClass *RC = MRI.getRegClass(OldReg);
+        Register DefReg = MI.getOperand(2).getReg();
+        const TargetRegisterClass *RC = MRI.getRegClass(DefReg);
 
         // Stackify the input if it isn't stackified yet.
-        if (!MFI.isVRegStackified(OldReg)) {
-          unsigned LocalId = getLocalId(Reg2Local, MFI, CurLocal, OldReg);
+        if (!MFI.isVRegStackified(DefReg)) {
+          unsigned LocalId = getLocalId(Reg2Local, MFI, CurLocal, DefReg);
           Register NewReg = MRI.createVirtualRegister(RC);
           unsigned Opc = getLocalGetOpcode(RC);
           BuildMI(MBB, &MI, MI.getDebugLoc(), TII->get(Opc), NewReg)
diff --git a/llvm/lib/Target/WebAssembly/WebAssemblyLowerEmscriptenEHSjLj.cpp b/llvm/lib/Target/WebAssembly/WebAssemblyLowerEmscriptenEHSjLj.cpp
index 71b930022214e..0e03890088ef0 100644
--- a/llvm/lib/Target/WebAssembly/WebAssemblyLowerEmscriptenEHSjLj.cpp
+++ b/llvm/lib/Target/WebAssembly/WebAssemblyLowerEmscriptenEHSjLj.cpp
@@ -580,7 +580,7 @@ Function *WebAssemblyLowerEmscriptenEHSjLj::getInvokeWrapper(CallBase *CI) {
   FunctionType *CalleeFTy = CI->getFunctionType();
 
   std::string Sig = getSignature(CalleeFTy);
-  if (InvokeWrappers.find(Sig) != InvokeWrappers.end())
+  if (InvokeWrappers.contains(Sig))
     return InvokeWrappers[Sig];
 
   // Put the pointer to the callee as first argument
@@ -1217,7 +1217,7 @@ bool WebAssemblyLowerEmscriptenEHSjLj::runEHOnFunction(Function &F) {
     for (unsigned I = 0, E = LPI->getNumClauses(); I < E; ++I) {
       Constant *Clause = LPI->getClause(I);
       // TODO Handle filters (= exception specifications).
-      // https://bugs.llvm.org/show_bug.cgi?id=50396
+      // https://github.com/llvm/llvm-project/issues/49740
       if (LPI->isCatch(I))
         FMCArgs.push_back(Clause);
     }
diff --git a/llvm/lib/Target/WebAssembly/WebAssemblyNullifyDebugValueLists.cpp b/llvm/lib/Target/WebAssembly/WebAssemblyNullifyDebugValueLists.cpp
index 5d8c58dcc3347..b58f7a0152aeb 100644
--- a/llvm/lib/Target/WebAssembly/WebAssemblyNullifyDebugValueLists.cpp
+++ b/llvm/lib/Target/WebAssembly/WebAssemblyNullifyDebugValueLists.cpp
@@ -9,7 +9,7 @@
 /// \file
 /// Nullify DBG_VALUE_LISTs instructions as a temporary measure before we
 /// implement DBG_VALUE_LIST handling in WebAssemblyDebugValueManager.
-/// See https://bugs.llvm.org/show_bug.cgi?id=50361.
+/// See https://github.com/llvm/llvm-project/issues/49705.
 /// TODO Correctly handle DBG_VALUE_LISTs
 ///
 //===----------------------------------------------------------------------===//
@@ -48,22 +48,17 @@ bool WebAssemblyNullifyDebugValueLists::runOnMachineFunction(
   LLVM_DEBUG(dbgs() << "********** Nullify DBG_VALUE_LISTs **********\n"
                        "********** Function: "
                     << MF.getName() << '\n');
-  const auto &TII = *MF.getSubtarget<WebAssemblySubtarget>().getInstrInfo();
-  SmallVector<MachineInstr *, 2> DbgValueLists;
-  for (auto &MBB : MF)
-    for (auto &MI : MBB)
-      if (MI.getOpcode() == TargetOpcode::DBG_VALUE_LIST)
-        DbgValueLists.push_back(&MI);
-
+  bool Changed = false;
   // Our backend, including WebAssemblyDebugValueManager, currently cannot
-  // handle DBG_VALUE_LISTs correctly. So this converts DBG_VALUE_LISTs to
-  // "DBG_VALUE $noreg", which will appear as "optimized out".
-  for (auto *DVL : DbgValueLists) {
-    BuildMI(*DVL->getParent(), DVL, DVL->getDebugLoc(),
-            TII.get(TargetOpcode::DBG_VALUE), false, Register(),
-            DVL->getOperand(0).getMetadata(), DVL->getOperand(1).getMetadata());
-    DVL->eraseFromParent();
+  // handle DBG_VALUE_LISTs correctly. So this makes them undefined, which will
+  // appear as "optimized out".
+  for (auto &MBB : MF) {
+    for (auto &MI : MBB) {
+      if (MI.getOpcode() == TargetOpcode::DBG_VALUE_LIST) {
+        MI.setDebugValueUndef();
+        Changed = true;
+      }
+    }
   }
-
-  return !DbgValueLists.empty();
+  return Changed;
 }
diff --git a/llvm/lib/Target/WebAssembly/WebAssemblyRuntimeLibcallSignatures.cpp b/llvm/lib/Target/WebAssembly/WebAssemblyRuntimeLibcallSignatures.cpp
index 4fe339ce52931..72a53b6c388ea 100644
--- a/llvm/lib/Target/WebAssembly/WebAssemblyRuntimeLibcallSignatures.cpp
+++ b/llvm/lib/Target/WebAssembly/WebAssemblyRuntimeLibcallSignatures.cpp
@@ -501,7 +501,7 @@ struct StaticLibcallNameMap {
       if (NameLibcall.first != nullptr &&
           getRuntimeLibcallSignatures().Table[NameLibcall.second] !=
               unsupported) {
-        assert(Map.find(NameLibcall.first) == Map.end() &&
+        assert(!Map.contains(NameLibcall.first) &&
                "duplicate libcall names in name map");
         Map[NameLibcall.first] = NameLibcall.second;
       }
diff --git a/llvm/lib/Target/X86/AsmParser/X86Operand.h b/llvm/lib/Target/X86/AsmParser/X86Operand.h
index 075b800f9e20a..4661e73c3ef8e 100644
--- a/llvm/lib/Target/X86/AsmParser/X86Operand.h
+++ b/llvm/lib/Target/X86/AsmParser/X86Operand.h
@@ -380,6 +380,40 @@ struct X86Operand final : public MCParsedAsmOperand {
   bool isMem512_RC512() const {
     return isMem512() && isMemIndexReg(X86::ZMM0, X86::ZMM31);
   }
+  bool isMem512_GR16() const {
+    if (!isMem512())
+      return false;
+    if (getMemBaseReg() &&
+        !X86MCRegisterClasses[X86::GR16RegClassID].contains(getMemBaseReg()))
+      return false;
+    return true;
+  }
+  bool isMem512_GR32() const {
+    if (!isMem512())
+      return false;
+    if (getMemBaseReg() &&
+        !X86MCRegisterClasses[X86::GR32RegClassID].contains(getMemBaseReg()) &&
+        getMemBaseReg() != X86::EIP)
+      return false;
+    if (getMemIndexReg() &&
+        !X86MCRegisterClasses[X86::GR32RegClassID].contains(getMemIndexReg()) &&
+        getMemIndexReg() != X86::EIZ)
+      return false;
+    return true;
+  }
+  bool isMem512_GR64() const {
+    if (!isMem512())
+      return false;
+    if (getMemBaseReg() &&
+        !X86MCRegisterClasses[X86::GR64RegClassID].contains(getMemBaseReg()) &&
+        getMemBaseReg() != X86::RIP)
+      return false;
+    if (getMemIndexReg() &&
+        !X86MCRegisterClasses[X86::GR64RegClassID].contains(getMemIndexReg()) &&
+        getMemIndexReg() != X86::RIZ)
+      return false;
+    return true;
+  }
 
   bool isAbsMem() const {
     return Kind == Memory && !getMemSegReg() && !getMemBaseReg() &&
diff --git a/llvm/lib/Target/X86/X86.td b/llvm/lib/Target/X86/X86.td
index bff9ac91e0b83..0f0ee6c8c5b68 100644
--- a/llvm/lib/Target/X86/X86.td
+++ b/llvm/lib/Target/X86/X86.td
@@ -724,6 +724,7 @@ include "X86ScheduleSLM.td"
 include "X86ScheduleZnver1.td"
 include "X86ScheduleZnver2.td"
 include "X86ScheduleZnver3.td"
+include "X86ScheduleZnver4.td"
 include "X86ScheduleBdVer2.td"
 include "X86ScheduleBtVer2.td"
 include "X86SchedSkylakeClient.td"
@@ -1728,7 +1729,7 @@ def : ProcModel<"znver2", Znver2Model, ProcessorFeatures.ZN2Features,
                 ProcessorFeatures.ZN2Tuning>;
 def : ProcModel<"znver3", Znver3Model, ProcessorFeatures.ZN3Features,
                 ProcessorFeatures.ZN3Tuning>;
-def : Proc<"znver4",ProcessorFeatures.ZN4Features,
+def : ProcModel<"znver4", Znver4Model, ProcessorFeatures.ZN4Features,
            ProcessorFeatures.ZN4Tuning>;
 
 def : Proc<"geode",           [FeatureX87, FeatureCX8, Feature3DNowA],
diff --git a/llvm/lib/Target/X86/X86FixupInstTuning.cpp b/llvm/lib/Target/X86/X86FixupInstTuning.cpp
index 7d01037931429..abeadc5edcbdf 100644
--- a/llvm/lib/Target/X86/X86FixupInstTuning.cpp
+++ b/llvm/lib/Target/X86/X86FixupInstTuning.cpp
@@ -57,6 +57,7 @@ class X86FixupInstTuningPass : public MachineFunctionPass {
 private:
   const X86InstrInfo *TII = nullptr;
   const X86Subtarget *ST = nullptr;
+  const MCSchedModel *SM = nullptr;
 };
 } // end anonymous namespace
 
@@ -68,6 +69,14 @@ FunctionPass *llvm::createX86FixupInstTuning() {
   return new X86FixupInstTuningPass();
 }
 
+template <typename T>
+static std::optional<bool> CmpOptionals(T NewVal, T CurVal) {
+  if (NewVal.has_value() && CurVal.has_value() && *NewVal != *CurVal)
+    return *NewVal < *CurVal;
+
+  return std::nullopt;
+}
+
 bool X86FixupInstTuningPass::processInstruction(
     MachineFunction &MF, MachineBasicBlock &MBB,
     MachineBasicBlock::iterator &I) {
@@ -75,13 +84,58 @@ bool X86FixupInstTuningPass::processInstruction(
   unsigned Opc = MI.getOpcode();
   unsigned NumOperands = MI.getDesc().getNumOperands();
 
+  auto GetInstTput = [&](unsigned Opcode) -> std::optional<double> {
+    // We already checked that SchedModel exists in `NewOpcPreferable`.
+    return MCSchedModel::getReciprocalThroughput(
+        *ST, *(SM->getSchedClassDesc(TII->get(Opcode).getSchedClass())));
+  };
+
+  auto GetInstLat = [&](unsigned Opcode) -> std::optional<double> {
+    // We already checked that SchedModel exists in `NewOpcPreferable`.
+    return MCSchedModel::computeInstrLatency(
+        *ST, *(SM->getSchedClassDesc(TII->get(Opcode).getSchedClass())));
+  };
+
+  auto GetInstSize = [&](unsigned Opcode) -> std::optional<unsigned> {
+    if (unsigned Size = TII->get(Opcode).getSize())
+      return Size;
+    // Zero size means we where unable to compute it.
+    return std::nullopt;
+  };
+
+  auto NewOpcPreferable = [&](unsigned NewOpc,
+                              bool ReplaceInTie = true) -> bool {
+    std::optional<bool> Res;
+    if (SM->hasInstrSchedModel()) {
+      // Compare tput -> lat -> code size.
+      Res = CmpOptionals(GetInstTput(NewOpc), GetInstTput(Opc));
+      if (Res.has_value())
+        return *Res;
+
+      Res = CmpOptionals(GetInstLat(NewOpc), GetInstLat(Opc));
+      if (Res.has_value())
+        return *Res;
+    }
+
+    Res = CmpOptionals(GetInstSize(Opc), GetInstSize(NewOpc));
+    if (Res.has_value())
+      return *Res;
+
+    // We either have either were unable to get tput/lat/codesize or all values
+    // were equal. Return specified option for a tie.
+    return ReplaceInTie;
+  };
+
   // `vpermilps r, i` -> `vshufps r, r, i`
-  // `vshufps` is always as fast or faster than `vpermilps` and takes 1 less
-  // byte of code size.
+  // `vpermilps r, i, k` -> `vshufps r, r, i, k`
+  // `vshufps` is always as fast or faster than `vpermilps` and takes
+  // 1 less byte of code size for VEX and SSE encoding.
   auto ProcessVPERMILPSri = [&](unsigned NewOpc) -> bool {
+    if (!NewOpcPreferable(NewOpc))
+      return false;
     unsigned MaskImm = MI.getOperand(NumOperands - 1).getImm();
     MI.removeOperand(NumOperands - 1);
-    MI.addOperand(MI.getOperand(1));
+    MI.addOperand(MI.getOperand(NumOperands - 2));
     MI.setDesc(TII->get(NewOpc));
     MI.addOperand(MachineOperand::CreateImm(MaskImm));
     return true;
@@ -93,13 +147,36 @@ bool X86FixupInstTuningPass::processInstruction(
   auto ProcessVPERMILPSmi = [&](unsigned NewOpc) -> bool {
     // TODO: Might be work adding bypass delay if -Os/-Oz is enabled as
     // `vpshufd` saves a byte of code size.
-    if (!ST->hasNoDomainDelayShuffle())
+    if (!ST->hasNoDomainDelayShuffle() &&
+        !NewOpcPreferable(NewOpc, /*ReplaceInTie*/ false))
+      return false;
+    MI.setDesc(TII->get(NewOpc));
+    return true;
+  };
+
+  // `vunpcklpd/vmovlhps r, r` -> `vshufps r, r, 0x44`
+  // `vunpckhpd/vmovlhps r, r` -> `vshufps r, r, 0xee`
+  // `vunpcklpd r, r, k` -> `vshufps r, r, 0x44, k`
+  // `vunpckhpd r, r, k` -> `vshufps r, r, 0xee, k`
+  // iff `vshufps` is faster than `vunpck{l|h}pd`. Otherwise stick with
+  // `vunpck{l|h}pd` as it uses less code size.
+  // TODO: Look into using `{VP}UNPCK{L|H}QDQ{...}` instead of `{V}SHUF{...}PS`
+  // as the replacement. `{VP}UNPCK{L|H}QDQ{...}` has no codesize cost.
+  auto ProcessUNPCKPD = [&](unsigned NewOpc, unsigned MaskImm) -> bool {
+    if (!NewOpcPreferable(NewOpc, /*ReplaceInTie*/ false))
       return false;
+
     MI.setDesc(TII->get(NewOpc));
+    MI.addOperand(MachineOperand::CreateImm(MaskImm));
     return true;
   };
+  auto ProcessUNPCKLPDrr = [&](unsigned NewOpc) -> bool {
+    return ProcessUNPCKPD(NewOpc, 0x44);
+  };
+  auto ProcessUNPCKHPDrr = [&](unsigned NewOpc) -> bool {
+    return ProcessUNPCKPD(NewOpc, 0xee);
+  };
 
-  // TODO: Add masked predicate execution variants.
   switch (Opc) {
   case X86::VPERMILPSri:
     return ProcessVPERMILPSri(X86::VSHUFPSrri);
@@ -111,6 +188,18 @@ bool X86FixupInstTuningPass::processInstruction(
     return ProcessVPERMILPSri(X86::VSHUFPSZ256rri);
   case X86::VPERMILPSZri:
     return ProcessVPERMILPSri(X86::VSHUFPSZrri);
+  case X86::VPERMILPSZ128rikz:
+    return ProcessVPERMILPSri(X86::VSHUFPSZ128rrikz);
+  case X86::VPERMILPSZ256rikz:
+    return ProcessVPERMILPSri(X86::VSHUFPSZ256rrikz);
+  case X86::VPERMILPSZrikz:
+    return ProcessVPERMILPSri(X86::VSHUFPSZrrikz);
+  case X86::VPERMILPSZ128rik:
+    return ProcessVPERMILPSri(X86::VSHUFPSZ128rrik);
+  case X86::VPERMILPSZ256rik:
+    return ProcessVPERMILPSri(X86::VSHUFPSZ256rrik);
+  case X86::VPERMILPSZrik:
+    return ProcessVPERMILPSri(X86::VSHUFPSZrrik);
   case X86::VPERMILPSmi:
     return ProcessVPERMILPSmi(X86::VPSHUFDmi);
   case X86::VPERMILPSYmi:
@@ -123,6 +212,77 @@ bool X86FixupInstTuningPass::processInstruction(
     return ProcessVPERMILPSmi(X86::VPSHUFDZ256mi);
   case X86::VPERMILPSZmi:
     return ProcessVPERMILPSmi(X86::VPSHUFDZmi);
+  case X86::VPERMILPSZ128mikz:
+    return ProcessVPERMILPSmi(X86::VPSHUFDZ128mikz);
+  case X86::VPERMILPSZ256mikz:
+    return ProcessVPERMILPSmi(X86::VPSHUFDZ256mikz);
+  case X86::VPERMILPSZmikz:
+    return ProcessVPERMILPSmi(X86::VPSHUFDZmikz);
+  case X86::VPERMILPSZ128mik:
+    return ProcessVPERMILPSmi(X86::VPSHUFDZ128mik);
+  case X86::VPERMILPSZ256mik:
+    return ProcessVPERMILPSmi(X86::VPSHUFDZ256mik);
+  case X86::VPERMILPSZmik:
+    return ProcessVPERMILPSmi(X86::VPSHUFDZmik);
+
+    // TODO: {V}UNPCK{L|H}PD{...} is probably safe to transform to
+    // `{VP}UNPCK{L|H}QDQ{...}` which gets the same perf benefit as
+    // `{V}SHUF{...}PS` but 1) without increasing code size and 2) can also
+    // handle the `mr` case. ICL doesn't have a domain penalty for replacing
+    // float unpck -> int unpck, but at this time, I haven't verified the set of
+    // processors where its safe.
+  case X86::MOVLHPSrr:
+  case X86::UNPCKLPDrr:
+    return ProcessUNPCKLPDrr(X86::SHUFPSrri);
+  case X86::VMOVLHPSrr:
+  case X86::VUNPCKLPDrr:
+    return ProcessUNPCKLPDrr(X86::VSHUFPSrri);
+  case X86::VUNPCKLPDYrr:
+    return ProcessUNPCKLPDrr(X86::VSHUFPSYrri);
+    // VMOVLHPS is always 128 bits.
+  case X86::VMOVLHPSZrr:
+  case X86::VUNPCKLPDZ128rr:
+    return ProcessUNPCKLPDrr(X86::VSHUFPSZ128rri);
+  case X86::VUNPCKLPDZ256rr:
+    return ProcessUNPCKLPDrr(X86::VSHUFPSZ256rri);
+  case X86::VUNPCKLPDZrr:
+    return ProcessUNPCKLPDrr(X86::VSHUFPSZrri);
+  case X86::VUNPCKLPDZ128rrk:
+    return ProcessUNPCKLPDrr(X86::VSHUFPSZ128rrik);
+  case X86::VUNPCKLPDZ256rrk:
+    return ProcessUNPCKLPDrr(X86::VSHUFPSZ256rrik);
+  case X86::VUNPCKLPDZrrk:
+    return ProcessUNPCKLPDrr(X86::VSHUFPSZrrik);
+  case X86::VUNPCKLPDZ128rrkz:
+    return ProcessUNPCKLPDrr(X86::VSHUFPSZ128rrikz);
+  case X86::VUNPCKLPDZ256rrkz:
+    return ProcessUNPCKLPDrr(X86::VSHUFPSZ256rrikz);
+  case X86::VUNPCKLPDZrrkz:
+    return ProcessUNPCKLPDrr(X86::VSHUFPSZrrikz);
+  case X86::UNPCKHPDrr:
+    return ProcessUNPCKHPDrr(X86::SHUFPSrri);
+  case X86::VUNPCKHPDrr:
+    return ProcessUNPCKHPDrr(X86::VSHUFPSrri);
+  case X86::VUNPCKHPDYrr:
+    return ProcessUNPCKHPDrr(X86::VSHUFPSYrri);
+  case X86::VUNPCKHPDZ128rr:
+    return ProcessUNPCKHPDrr(X86::VSHUFPSZ128rri);
+  case X86::VUNPCKHPDZ256rr:
+    return ProcessUNPCKHPDrr(X86::VSHUFPSZ256rri);
+  case X86::VUNPCKHPDZrr:
+    return ProcessUNPCKHPDrr(X86::VSHUFPSZrri);
+  case X86::VUNPCKHPDZ128rrk:
+    return ProcessUNPCKHPDrr(X86::VSHUFPSZ128rrik);
+  case X86::VUNPCKHPDZ256rrk:
+    return ProcessUNPCKHPDrr(X86::VSHUFPSZ256rrik);
+  case X86::VUNPCKHPDZrrk:
+    return ProcessUNPCKHPDrr(X86::VSHUFPSZrrik);
+  case X86::VUNPCKHPDZ128rrkz:
+    return ProcessUNPCKHPDrr(X86::VSHUFPSZ128rrikz);
+  case X86::VUNPCKHPDZ256rrkz:
+    return ProcessUNPCKHPDrr(X86::VSHUFPSZ256rrikz);
+  case X86::VUNPCKHPDZrrkz:
+    return ProcessUNPCKHPDrr(X86::VSHUFPSZrrikz);
   default:
     return false;
   }
@@ -133,6 +293,8 @@ bool X86FixupInstTuningPass::runOnMachineFunction(MachineFunction &MF) {
   bool Changed = false;
   ST = &MF.getSubtarget<X86Subtarget>();
   TII = ST->getInstrInfo();
+  SM = &ST->getSchedModel();
+
   for (MachineBasicBlock &MBB : MF) {
     for (MachineBasicBlock::iterator I = MBB.begin(); I != MBB.end(); ++I) {
       if (processInstruction(MF, MBB, I)) {
diff --git a/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp b/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
index c13245ff7bea7..5e90a94819b6b 100644
--- a/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
+++ b/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
@@ -2489,34 +2489,60 @@ bool X86DAGToDAGISel::matchAddressRecursively(SDValue N, X86ISelAddressMode &AM,
     // match the shift as a scale factor.
     if (AM.IndexReg.getNode() != nullptr || AM.Scale != 1)
       break;
-    if (N.getOperand(0).getOpcode() != ISD::SHL || !N.getOperand(0).hasOneUse())
+
+    // Peek through mask: zext(and(shl(x,c1),c2))
+    SDValue Src = N.getOperand(0);
+    APInt Mask = APInt::getAllOnes(Src.getScalarValueSizeInBits());
+    if (Src.getOpcode() == ISD::AND && Src.hasOneUse())
+      if (auto *MaskC = dyn_cast<ConstantSDNode>(Src.getOperand(1))) {
+        Mask = MaskC->getAPIntValue();
+        Src = Src.getOperand(0);
+      }
+
+    if (Src.getOpcode() != ISD::SHL || !Src.hasOneUse())
       break;
 
     // Give up if the shift is not a valid scale factor [1,2,3].
-    SDValue Shl = N.getOperand(0);
-    auto *ShAmtC = dyn_cast<ConstantSDNode>(Shl.getOperand(1));
-    if (!ShAmtC || ShAmtC->getZExtValue() > 3)
+    SDValue ShlSrc = Src.getOperand(0);
+    SDValue ShlAmt = Src.getOperand(1);
+    auto *ShAmtC = dyn_cast<ConstantSDNode>(ShlAmt);
+    if (!ShAmtC)
+      break;
+    unsigned ShAmtV = ShAmtC->getZExtValue();
+    if (ShAmtV > 3)
       break;
 
     // The narrow shift must only shift out zero bits (it must be 'nuw').
     // That makes it safe to widen to the destination type.
-    APInt HighZeros = APInt::getHighBitsSet(Shl.getValueSizeInBits(),
-                                            ShAmtC->getZExtValue());
-    if (!CurDAG->MaskedValueIsZero(Shl.getOperand(0), HighZeros))
+    APInt HighZeros =
+        APInt::getHighBitsSet(ShlSrc.getValueSizeInBits(), ShAmtV);
+    if (!CurDAG->MaskedValueIsZero(ShlSrc, HighZeros & Mask))
       break;
 
-    // zext (shl nuw i8 %x, C) to i32 --> shl (zext i8 %x to i32), (zext C)
+    // zext (shl nuw i8 %x, C1) to i32
+    // --> shl (zext i8 %x to i32), (zext C1)
+    // zext (and (shl nuw i8 %x, C1), C2) to i32
+    // --> shl (zext i8 (and %x, C2 >> C1) to i32), (zext C1)
+    MVT SrcVT = ShlSrc.getSimpleValueType();
     MVT VT = N.getSimpleValueType();
     SDLoc DL(N);
-    SDValue Zext = CurDAG->getNode(ISD::ZERO_EXTEND, DL, VT, Shl.getOperand(0));
-    SDValue NewShl = CurDAG->getNode(ISD::SHL, DL, VT, Zext, Shl.getOperand(1));
+
+    SDValue Res = ShlSrc;
+    if (!Mask.isAllOnes()) {
+      Res = CurDAG->getConstant(Mask.lshr(ShAmtV), DL, SrcVT);
+      insertDAGNode(*CurDAG, N, Res);
+      Res = CurDAG->getNode(ISD::AND, DL, SrcVT, ShlSrc, Res);
+      insertDAGNode(*CurDAG, N, Res);
+    }
+    SDValue Zext = CurDAG->getNode(ISD::ZERO_EXTEND, DL, VT, Res);
+    insertDAGNode(*CurDAG, N, Zext);
+    SDValue NewShl = CurDAG->getNode(ISD::SHL, DL, VT, Zext, ShlAmt);
+    insertDAGNode(*CurDAG, N, NewShl);
 
     // Convert the shift to scale factor.
-    AM.Scale = 1 << ShAmtC->getZExtValue();
+    AM.Scale = 1 << ShAmtV;
     AM.IndexReg = Zext;
 
-    insertDAGNode(*CurDAG, N, Zext);
-    insertDAGNode(*CurDAG, N, NewShl);
     CurDAG->ReplaceAllUsesWith(N, NewShl);
     CurDAG->RemoveDeadNode(N.getNode());
     return false;
diff --git a/llvm/lib/Target/X86/X86ISelLowering.cpp b/llvm/lib/Target/X86/X86ISelLowering.cpp
index ac743753f0ba5..e9a069b7295e1 100644
--- a/llvm/lib/Target/X86/X86ISelLowering.cpp
+++ b/llvm/lib/Target/X86/X86ISelLowering.cpp
@@ -29839,6 +29839,17 @@ static SDValue LowerMINMAX(SDValue Op, const X86Subtarget &Subtarget,
   if (VT == MVT::v32i16 || VT == MVT::v64i8)
     return splitVectorIntBinary(Op, DAG);
 
+  // umax(x,1) --> sub(x,cmpeq(x,0))
+  // TODO: Move this to expandIntMINMAX?
+  if (VT.isVector() && Op.getOpcode() == ISD::UMAX &&
+      llvm::isOneOrOneSplat(Op.getOperand(1), true)) {
+    SDLoc DL(Op);
+    SDValue X = DAG.getFreeze(Op.getOperand(0));
+    SDValue Zero = getZeroVector(VT, Subtarget, DAG, DL);
+    return DAG.getNode(ISD::SUB, DL, VT, X,
+                       DAG.getSetCC(DL, VT, X, Zero, ISD::SETEQ));
+  }
+
   // Default to expand.
   return SDValue();
 }
@@ -35398,9 +35409,9 @@ bool X86TargetLowering::isFMAFasterThanFMulAndFAdd(const MachineFunction &MF,
   return false;
 }
 
-bool X86TargetLowering::isNarrowingProfitable(EVT VT1, EVT VT2) const {
+bool X86TargetLowering::isNarrowingProfitable(EVT SrcVT, EVT DestVT) const {
   // i16 instructions are longer (0x66 prefix) and potentially slower.
-  return !(VT1 == MVT::i32 && VT2 == MVT::i16);
+  return !(SrcVT == MVT::i32 && DestVT == MVT::i16);
 }
 
 bool X86TargetLowering::shouldFoldSelectWithIdentityConstant(unsigned Opcode,
@@ -35950,10 +35961,10 @@ static MachineInstrBuilder createPHIsForCMOVsInSinkBB(
     if (MIIt->getOperand(3).getImm() == OppCC)
       std::swap(Op1Reg, Op2Reg);
 
-    if (RegRewriteTable.find(Op1Reg) != RegRewriteTable.end())
+    if (RegRewriteTable.contains(Op1Reg))
       Op1Reg = RegRewriteTable[Op1Reg].first;
 
-    if (RegRewriteTable.find(Op2Reg) != RegRewriteTable.end())
+    if (RegRewriteTable.contains(Op2Reg))
       Op2Reg = RegRewriteTable[Op2Reg].second;
 
     MIB = BuildMI(*SinkMBB, SinkInsertionPoint, DL, TII->get(X86::PHI), DestReg)
@@ -43671,13 +43682,14 @@ static bool checkBitcastSrcVectorSize(SDValue Src, unsigned Size,
   case ISD::OR:
     return checkBitcastSrcVectorSize(Src.getOperand(0), Size, AllowTruncate) &&
            checkBitcastSrcVectorSize(Src.getOperand(1), Size, AllowTruncate);
+  case ISD::SELECT:
   case ISD::VSELECT:
     return Src.getOperand(0).getScalarValueSizeInBits() == 1 &&
            checkBitcastSrcVectorSize(Src.getOperand(1), Size, AllowTruncate) &&
            checkBitcastSrcVectorSize(Src.getOperand(2), Size, AllowTruncate);
   case ISD::BUILD_VECTOR:
-    return ISD::isBuildVectorAllZeros(Src.getNode());
-
+    return ISD::isBuildVectorAllZeros(Src.getNode()) ||
+           ISD::isBuildVectorAllOnes(Src.getNode());
   }
   return false;
 }
@@ -43742,6 +43754,7 @@ static SDValue signExtendBitcastSrcVector(SelectionDAG &DAG, EVT SExtVT,
         Src.getOpcode(), DL, SExtVT,
         signExtendBitcastSrcVector(DAG, SExtVT, Src.getOperand(0), DL),
         signExtendBitcastSrcVector(DAG, SExtVT, Src.getOperand(1), DL));
+  case ISD::SELECT:
   case ISD::VSELECT:
     return DAG.getSelect(
         DL, SExtVT, Src.getOperand(0),
@@ -47428,6 +47441,32 @@ static SDValue combineCMov(SDNode *N, SelectionDAG &DAG,
     }
   }
 
+  // Transform:
+  //
+  //   (cmov 1 T (uge T 2))
+  //
+  // to:
+  //
+  //   (adc T 0 (sub T 1))
+  if (CC == X86::COND_AE && isOneConstant(FalseOp) &&
+      Cond.getOpcode() == X86ISD::SUB && Cond->hasOneUse()) {
+    SDValue Cond0 = Cond.getOperand(0);
+    if (Cond0.getOpcode() == ISD::TRUNCATE)
+      Cond0 = Cond0.getOperand(0);
+    auto *Sub1C = dyn_cast<ConstantSDNode>(Cond.getOperand(1));
+    if (Cond0 == TrueOp && Sub1C && Sub1C->getZExtValue() == 2) {
+      EVT CondVT = Cond->getValueType(0);
+      EVT OuterVT = N->getValueType(0);
+      // Subtract 1 and generate a carry.
+      SDValue NewSub =
+          DAG.getNode(X86ISD::SUB, DL, Cond->getVTList(), Cond.getOperand(0),
+                      DAG.getConstant(1, DL, CondVT));
+      SDValue EFLAGS(NewSub.getNode(), 1);
+      return DAG.getNode(X86ISD::ADC, DL, DAG.getVTList(OuterVT, MVT::i32),
+                         TrueOp, DAG.getConstant(0, DL, OuterVT), EFLAGS);
+    }
+  }
+
   // Fold and/or of setcc's to double CMOV:
   //   (CMOV F, T, ((cc1 | cc2) != 0)) -> (CMOV (CMOV F, T, cc1), T, cc2)
   //   (CMOV F, T, ((cc1 & cc2) != 0)) -> (CMOV (CMOV T, F, !cc1), F, !cc2)
@@ -55854,7 +55893,7 @@ static SDValue combineConcatVectorOps(const SDLoc &DL, MVT VT,
   if (llvm::all_of(Ops, [Op0](SDValue Op) {
         return Op.getOpcode() == Op0.getOpcode();
       })) {
-    auto ConcatSubOperand = [&](MVT VT, ArrayRef<SDValue> SubOps, unsigned I) {
+    auto ConcatSubOperand = [&](EVT VT, ArrayRef<SDValue> SubOps, unsigned I) {
       SmallVector<SDValue> Subs;
       for (SDValue SubOp : SubOps)
         Subs.push_back(SubOp.getOperand(I));
@@ -56009,6 +56048,20 @@ static SDValue combineConcatVectorOps(const SDLoc &DL, MVT VT,
         }
       }
       break;
+    case ISD::TRUNCATE:
+      if (!IsSplat && NumOps == 2 && VT.is256BitVector()) {
+        EVT SrcVT = Ops[0].getOperand(0).getValueType();
+        if (SrcVT.is256BitVector() && SrcVT.isSimple() &&
+            SrcVT == Ops[1].getOperand(0).getValueType() &&
+            Subtarget.useAVX512Regs() &&
+            Subtarget.getPreferVectorWidth() >= 512 &&
+            (SrcVT.getScalarSizeInBits() > 16 || Subtarget.useBWIRegs())) {
+          EVT NewSrcVT = SrcVT.getDoubleNumVectorElementsVT(*DAG.getContext());
+          return DAG.getNode(ISD::TRUNCATE, DL, VT,
+                             ConcatSubOperand(NewSrcVT, Ops, 0));
+        }
+      }
+      break;
     case X86ISD::VSHLI:
     case X86ISD::VSRLI:
       // Special case: SHL/SRL AVX1 V4i64 by 32-bits can lower as a shuffle.
@@ -56080,6 +56133,20 @@ static SDValue combineConcatVectorOps(const SDLoc &DL, MVT VT,
                            ConcatSubOperand(VT, Ops, 1), Op0.getOperand(2));
       }
       break;
+    case ISD::ADD:
+    case ISD::SUB:
+    case ISD::MUL:
+      if (!IsSplat && ((VT.is256BitVector() && Subtarget.hasInt256()) ||
+                       (VT.is512BitVector() && Subtarget.useAVX512Regs() &&
+                        (EltSizeInBits >= 32 || Subtarget.useBWIRegs())))) {
+        MVT SrcVT = Op0.getOperand(0).getSimpleValueType();
+        SrcVT = MVT::getVectorVT(SrcVT.getScalarType(),
+                                 NumOps * SrcVT.getVectorNumElements());
+        return DAG.getNode(Op0.getOpcode(), DL, VT,
+                           ConcatSubOperand(SrcVT, Ops, 0),
+                           ConcatSubOperand(SrcVT, Ops, 1));
+      }
+      break;
     case ISD::FADD:
     case ISD::FSUB:
     case ISD::FMUL:
diff --git a/llvm/lib/Target/X86/X86ISelLowering.h b/llvm/lib/Target/X86/X86ISelLowering.h
index 2f143a553ecb4..ab0d6611edf3e 100644
--- a/llvm/lib/Target/X86/X86ISelLowering.h
+++ b/llvm/lib/Target/X86/X86ISelLowering.h
@@ -1352,10 +1352,10 @@ namespace llvm {
     bool isFMAFasterThanFMulAndFAdd(const MachineFunction &MF,
                                     EVT VT) const override;
 
-    /// Return true if it's profitable to narrow
-    /// operations of type VT1 to VT2. e.g. on x86, it's profitable to narrow
-    /// from i32 to i8 but not from i32 to i16.
-    bool isNarrowingProfitable(EVT VT1, EVT VT2) const override;
+    /// Return true if it's profitable to narrow operations of type SrcVT to
+    /// DestVT. e.g. on x86, it's profitable to narrow from i32 to i8 but not
+    /// from i32 to i16.
+    bool isNarrowingProfitable(EVT SrcVT, EVT DestVT) const override;
 
     bool shouldFoldSelectWithIdentityConstant(unsigned BinOpcode,
                                               EVT VT) const override;
diff --git a/llvm/lib/Target/X86/X86InstrCompiler.td b/llvm/lib/Target/X86/X86InstrCompiler.td
index 575cf4d3dced3..36324d1338864 100644
--- a/llvm/lib/Target/X86/X86InstrCompiler.td
+++ b/llvm/lib/Target/X86/X86InstrCompiler.td
@@ -896,15 +896,15 @@ multiclass ATOMIC_LOGIC_OP<Format Form, string s> {
 multiclass ATOMIC_LOGIC_OP_RM<bits<8> Opc8, string s> {
   let Defs = [EFLAGS], mayLoad = 1, mayStore = 1, isCodeGenOnly = 1,
       SchedRW = [WriteBitTestSetRegRMW]  in {
-    def 16rm : Ii8<Opc8, MRMDestMem, (outs), (ins i16mem:$src1, GR16:$src2),
+    def 16rm : I<Opc8, MRMDestMem, (outs), (ins i16mem:$src1, GR16:$src2),
                   !strconcat(s, "{w}\t{$src2, $src1|$src1, $src2}"),
                   [(set EFLAGS, (!cast<SDNode>("x86_rm_" # s) addr:$src1, GR16:$src2))]>,
                OpSize16, TB, LOCK;
-    def 32rm : Ii8<Opc8, MRMDestMem, (outs), (ins i32mem:$src1, GR32:$src2),
+    def 32rm : I<Opc8, MRMDestMem, (outs), (ins i32mem:$src1, GR32:$src2),
                   !strconcat(s, "{l}\t{$src2, $src1|$src1, $src2}"),
                   [(set EFLAGS, (!cast<SDNode>("x86_rm_" # s) addr:$src1, GR32:$src2))]>,
                OpSize32, TB, LOCK;
-    def 64rm : RIi8<Opc8, MRMDestMem, (outs), (ins i64mem:$src1, GR64:$src2),
+    def 64rm : RI<Opc8, MRMDestMem, (outs), (ins i64mem:$src1, GR64:$src2),
                    !strconcat(s, "{q}\t{$src2, $src1|$src1, $src2}"),
                    [(set EFLAGS, (!cast<SDNode>("x86_rm_" # s) addr:$src1, GR64:$src2))]>,
                TB, LOCK;
diff --git a/llvm/lib/Target/X86/X86InstrFoldTables.h b/llvm/lib/Target/X86/X86InstrFoldTables.h
index b7aca27ab2bb5..384369b9560ac 100644
--- a/llvm/lib/Target/X86/X86InstrFoldTables.h
+++ b/llvm/lib/Target/X86/X86InstrFoldTables.h
@@ -14,55 +14,10 @@
 #define LLVM_LIB_TARGET_X86_X86INSTRFOLDTABLES_H
 
 #include <cstdint>
+#include "llvm/Support/X86FoldTablesUtils.h"
 
 namespace llvm {
 
-enum {
-  // Select which memory operand is being unfolded.
-  // (stored in bits 0 - 2)
-  TB_INDEX_0    = 0,
-  TB_INDEX_1    = 1,
-  TB_INDEX_2    = 2,
-  TB_INDEX_3    = 3,
-  TB_INDEX_4    = 4,
-  TB_INDEX_MASK = 0x7,
-
-  // Do not insert the reverse map (MemOp -> RegOp) into the table.
-  // This may be needed because there is a many -> one mapping.
-  TB_NO_REVERSE   = 1 << 3,
-
-  // Do not insert the forward map (RegOp -> MemOp) into the table.
-  // This is needed for Native Client, which prohibits branch
-  // instructions from using a memory operand.
-  TB_NO_FORWARD   = 1 << 4,
-
-  TB_FOLDED_LOAD  = 1 << 5,
-  TB_FOLDED_STORE = 1 << 6,
-  TB_FOLDED_BCAST = 1 << 7,
-
-  // Minimum alignment required for load/store.
-  // Used for RegOp->MemOp conversion. Encoded as Log2(Align) + 1 to allow 0
-  // to mean align of 0.
-  // (stored in bits 8 - 11)
-  TB_ALIGN_SHIFT = 8,
-  TB_ALIGN_NONE  =   0 << TB_ALIGN_SHIFT,
-  TB_ALIGN_16    =   5 << TB_ALIGN_SHIFT,
-  TB_ALIGN_32    =   6 << TB_ALIGN_SHIFT,
-  TB_ALIGN_64    =   7 << TB_ALIGN_SHIFT,
-  TB_ALIGN_MASK  = 0xf << TB_ALIGN_SHIFT,
-
-  // Broadcast type.
-  // (stored in bits 12 - 13)
-  TB_BCAST_TYPE_SHIFT = 12,
-  TB_BCAST_D    =   0 << TB_BCAST_TYPE_SHIFT,
-  TB_BCAST_Q    =   1 << TB_BCAST_TYPE_SHIFT,
-  TB_BCAST_SS   =   2 << TB_BCAST_TYPE_SHIFT,
-  TB_BCAST_SD   =   3 << TB_BCAST_TYPE_SHIFT,
-  TB_BCAST_MASK = 0x3 << TB_BCAST_TYPE_SHIFT,
-
-  // Unused bits 14-15
-};
-
 // This struct is used for both the folding and unfold tables. They KeyOp
 // is used to determine the sorting order.
 struct X86MemoryFoldTableEntry {
diff --git a/llvm/lib/Target/X86/X86InstrInfo.td b/llvm/lib/Target/X86/X86InstrInfo.td
index b832b15c5b938..d295208c75f7b 100644
--- a/llvm/lib/Target/X86/X86InstrInfo.td
+++ b/llvm/lib/Target/X86/X86InstrInfo.td
@@ -380,6 +380,9 @@ let RenderMethod = "addMemOperands", SuperClasses = [X86MemAsmOperand] in {
   def X86Mem512_RC256XOperand : AsmOperandClass { let Name = "Mem512_RC256X"; }
   def X86Mem256_RC512Operand  : AsmOperandClass { let Name = "Mem256_RC512"; }
   def X86Mem512_RC512Operand  : AsmOperandClass { let Name = "Mem512_RC512"; }
+  def X86Mem512_GR16Operand : AsmOperandClass { let Name = "Mem512_GR16"; }
+  def X86Mem512_GR32Operand : AsmOperandClass { let Name = "Mem512_GR32"; }
+  def X86Mem512_GR64Operand : AsmOperandClass { let Name = "Mem512_GR64"; }
 
   def X86SibMemOperand : AsmOperandClass { let Name = "SibMem"; }
 }
@@ -432,6 +435,11 @@ def f128mem : X86MemOperand<"printxmmwordmem", X86Mem128AsmOperand, 128>;
 def f256mem : X86MemOperand<"printymmwordmem", X86Mem256AsmOperand, 256>;
 def f512mem : X86MemOperand<"printzmmwordmem", X86Mem512AsmOperand, 512>;
 
+// 32/64 mode specific mem operands
+def i512mem_GR16 : X86MemOperand<"printzmmwordmem", X86Mem512_GR16Operand, 512>;
+def i512mem_GR32 : X86MemOperand<"printzmmwordmem", X86Mem512_GR32Operand, 512>;
+def i512mem_GR64 : X86MemOperand<"printzmmwordmem", X86Mem512_GR64Operand, 512>;
+
 // Gather mem operands
 def vx64mem  : X86VMemOperand<VR128,  "printqwordmem",  X86Mem64_RC128Operand, 64>;
 def vx128mem : X86VMemOperand<VR128,  "printxmmwordmem", X86Mem128_RC128Operand, 128>;
diff --git a/llvm/lib/Target/X86/X86InstrMisc.td b/llvm/lib/Target/X86/X86InstrMisc.td
index 092a10812d60c..bf094f3ef2a3c 100644
--- a/llvm/lib/Target/X86/X86InstrMisc.td
+++ b/llvm/lib/Target/X86/X86InstrMisc.td
@@ -1521,14 +1521,14 @@ def MOVDIRI64 : RI<0xF9, MRMDestMem, (outs), (ins i64mem:$dst, GR64:$src),
 // MOVDIR64B - Move 64 bytes as direct store
 //
 let SchedRW = [WriteStore] in {
-def MOVDIR64B16 : I<0xF8, MRMSrcMem, (outs), (ins GR16:$dst, i512mem:$src),
+def MOVDIR64B16 : I<0xF8, MRMSrcMem, (outs), (ins GR16:$dst, i512mem_GR16:$src),
                     "movdir64b\t{$src, $dst|$dst, $src}", []>,
                    T8PD, AdSize16, Requires<[HasMOVDIR64B, Not64BitMode]>;
-def MOVDIR64B32 : I<0xF8, MRMSrcMem, (outs), (ins GR32:$dst, i512mem:$src),
+def MOVDIR64B32 : I<0xF8, MRMSrcMem, (outs), (ins GR32:$dst, i512mem_GR32:$src),
                     "movdir64b\t{$src, $dst|$dst, $src}",
                     [(int_x86_movdir64b GR32:$dst, addr:$src)]>,
                    T8PD, AdSize32, Requires<[HasMOVDIR64B]>;
-def MOVDIR64B64 : I<0xF8, MRMSrcMem, (outs), (ins GR64:$dst, i512mem:$src),
+def MOVDIR64B64 : I<0xF8, MRMSrcMem, (outs), (ins GR64:$dst, i512mem_GR64:$src),
                     "movdir64b\t{$src, $dst|$dst, $src}",
                     [(int_x86_movdir64b GR64:$dst, addr:$src)]>,
                    T8PD, AdSize64, Requires<[HasMOVDIR64B, In64BitMode]>;
diff --git a/llvm/lib/Target/X86/X86LoadValueInjectionLoadHardening.cpp b/llvm/lib/Target/X86/X86LoadValueInjectionLoadHardening.cpp
index 5d9a9231fea9b..079d260e1fb5e 100644
--- a/llvm/lib/Target/X86/X86LoadValueInjectionLoadHardening.cpp
+++ b/llvm/lib/Target/X86/X86LoadValueInjectionLoadHardening.cpp
@@ -362,7 +362,7 @@ X86LoadValueInjectionLoadHardeningPass::getGadgetGraph(
     SmallSet<NodeId, 8> UsesVisited, DefsVisited;
     std::function<void(NodeAddr<DefNode *>)> AnalyzeDefUseChain =
         [&](NodeAddr<DefNode *> Def) {
-          if (Transmitters.find(Def.Id) != Transmitters.end())
+          if (Transmitters.contains(Def.Id))
             return; // Already analyzed `Def`
 
           // Use RDF to find all the uses of `Def`
diff --git a/llvm/lib/Target/X86/X86MemFoldTables.inc b/llvm/lib/Target/X86/X86MemFoldTables.inc
index d5aa6ebfab961..8a1a14c2f0014 100644
--- a/llvm/lib/Target/X86/X86MemFoldTables.inc
+++ b/llvm/lib/Target/X86/X86MemFoldTables.inc
@@ -258,20 +258,21 @@ static const X86MemoryFoldTableEntry MemoryFoldTable0[] = {
   {X86::JMP32r_NT, X86::JMP32m_NT, TB_FOLDED_LOAD},
   {X86::JMP64r, X86::JMP64m, TB_FOLDED_LOAD},
   {X86::JMP64r_NT, X86::JMP64m_NT, TB_FOLDED_LOAD},
+  {X86::JMP64r_REX, X86::JMP64m_REX, TB_FOLDED_LOAD},
   {X86::MMX_MOVD64from64rr, X86::MMX_MOVQ64mr, TB_FOLDED_STORE},
   {X86::MMX_MOVD64grr, X86::MMX_MOVD64mr, TB_FOLDED_STORE},
   {X86::MOV16ri, X86::MOV16mi, TB_FOLDED_STORE},
-  {X86::MOV16rr, X86::MOV16mr, TB_FOLDED_STORE},
+  {X86::MOV16rr, X86::MOV16mr, TB_FOLDED_STORE|TB_NO_REVERSE},
   {X86::MOV32ri, X86::MOV32mi, TB_FOLDED_STORE},
-  {X86::MOV32rr, X86::MOV32mr, TB_FOLDED_STORE},
+  {X86::MOV32rr, X86::MOV32mr, TB_FOLDED_STORE|TB_NO_REVERSE},
   {X86::MOV64ri32, X86::MOV64mi32, TB_FOLDED_STORE},
-  {X86::MOV64rr, X86::MOV64mr, TB_FOLDED_STORE},
+  {X86::MOV64rr, X86::MOV64mr, TB_FOLDED_STORE|TB_NO_REVERSE},
   {X86::MOV64toSDrr, X86::MOV64mr, TB_FOLDED_STORE|TB_NO_REVERSE},
   {X86::MOV8ri, X86::MOV8mi, TB_FOLDED_STORE},
-  {X86::MOV8rr, X86::MOV8mr, TB_FOLDED_STORE},
-  {X86::MOV8rr_NOREX, X86::MOV8mr_NOREX, TB_FOLDED_STORE},
-  {X86::MOVAPDrr, X86::MOVAPDmr, TB_FOLDED_STORE|TB_ALIGN_16},
-  {X86::MOVAPSrr, X86::MOVAPSmr, TB_FOLDED_STORE|TB_ALIGN_16},
+  {X86::MOV8rr, X86::MOV8mr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::MOV8rr_NOREX, X86::MOV8mr_NOREX, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::MOVAPDrr, X86::MOVAPDmr, TB_FOLDED_STORE|TB_NO_REVERSE|TB_ALIGN_16},
+  {X86::MOVAPSrr, X86::MOVAPSmr, TB_FOLDED_STORE|TB_NO_REVERSE|TB_ALIGN_16},
   {X86::MOVDI2SSrr, X86::MOV32mr, TB_FOLDED_STORE|TB_NO_REVERSE},
   {X86::MOVDQArr, X86::MOVDQAmr, TB_FOLDED_STORE|TB_ALIGN_16},
   {X86::MOVDQUrr, X86::MOVDQUmr, TB_FOLDED_STORE},
@@ -279,8 +280,8 @@ static const X86MemoryFoldTableEntry MemoryFoldTable0[] = {
   {X86::MOVPQIto64rr, X86::MOVPQI2QImr, TB_FOLDED_STORE|TB_NO_REVERSE},
   {X86::MOVSDto64rr, X86::MOVSDmr, TB_FOLDED_STORE|TB_NO_REVERSE},
   {X86::MOVSS2DIrr, X86::MOVSSmr, TB_FOLDED_STORE},
-  {X86::MOVUPDrr, X86::MOVUPDmr, TB_FOLDED_STORE},
-  {X86::MOVUPSrr, X86::MOVUPSmr, TB_FOLDED_STORE},
+  {X86::MOVUPDrr, X86::MOVUPDmr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::MOVUPSrr, X86::MOVUPSmr, TB_FOLDED_STORE|TB_NO_REVERSE},
   {X86::MUL16r, X86::MUL16m, TB_FOLDED_LOAD},
   {X86::MUL32r, X86::MUL32m, TB_FOLDED_LOAD},
   {X86::MUL64r, X86::MUL64m, TB_FOLDED_LOAD},
@@ -327,38 +328,38 @@ static const X86MemoryFoldTableEntry MemoryFoldTable0[] = {
   {X86::VEXTRACTPSrr, X86::VEXTRACTPSmr, TB_FOLDED_STORE},
   {X86::VMOV64toSDZrr, X86::MOV64mr, TB_FOLDED_STORE|TB_NO_REVERSE},
   {X86::VMOV64toSDrr, X86::MOV64mr, TB_FOLDED_STORE|TB_NO_REVERSE},
-  {X86::VMOVAPDYrr, X86::VMOVAPDYmr, TB_FOLDED_STORE|TB_ALIGN_32},
-  {X86::VMOVAPDZ128rr, X86::VMOVAPDZ128mr, TB_FOLDED_STORE|TB_ALIGN_16},
-  {X86::VMOVAPDZ256rr, X86::VMOVAPDZ256mr, TB_FOLDED_STORE|TB_ALIGN_32},
-  {X86::VMOVAPDZrr, X86::VMOVAPDZmr, TB_FOLDED_STORE|TB_ALIGN_64},
-  {X86::VMOVAPDrr, X86::VMOVAPDmr, TB_FOLDED_STORE|TB_ALIGN_16},
-  {X86::VMOVAPSYrr, X86::VMOVAPSYmr, TB_FOLDED_STORE|TB_ALIGN_32},
-  {X86::VMOVAPSZ128rr, X86::VMOVAPSZ128mr, TB_FOLDED_STORE|TB_ALIGN_16},
-  {X86::VMOVAPSZ256rr, X86::VMOVAPSZ256mr, TB_FOLDED_STORE|TB_ALIGN_32},
-  {X86::VMOVAPSZrr, X86::VMOVAPSZmr, TB_FOLDED_STORE|TB_ALIGN_64},
-  {X86::VMOVAPSrr, X86::VMOVAPSmr, TB_FOLDED_STORE|TB_ALIGN_16},
+  {X86::VMOVAPDYrr, X86::VMOVAPDYmr, TB_FOLDED_STORE|TB_NO_REVERSE|TB_ALIGN_32},
+  {X86::VMOVAPDZ128rr, X86::VMOVAPDZ128mr, TB_FOLDED_STORE|TB_NO_REVERSE|TB_ALIGN_16},
+  {X86::VMOVAPDZ256rr, X86::VMOVAPDZ256mr, TB_FOLDED_STORE|TB_NO_REVERSE|TB_ALIGN_32},
+  {X86::VMOVAPDZrr, X86::VMOVAPDZmr, TB_FOLDED_STORE|TB_NO_REVERSE|TB_ALIGN_64},
+  {X86::VMOVAPDrr, X86::VMOVAPDmr, TB_FOLDED_STORE|TB_NO_REVERSE|TB_ALIGN_16},
+  {X86::VMOVAPSYrr, X86::VMOVAPSYmr, TB_FOLDED_STORE|TB_NO_REVERSE|TB_ALIGN_32},
+  {X86::VMOVAPSZ128rr, X86::VMOVAPSZ128mr, TB_FOLDED_STORE|TB_NO_REVERSE|TB_ALIGN_16},
+  {X86::VMOVAPSZ256rr, X86::VMOVAPSZ256mr, TB_FOLDED_STORE|TB_NO_REVERSE|TB_ALIGN_32},
+  {X86::VMOVAPSZrr, X86::VMOVAPSZmr, TB_FOLDED_STORE|TB_NO_REVERSE|TB_ALIGN_64},
+  {X86::VMOVAPSrr, X86::VMOVAPSmr, TB_FOLDED_STORE|TB_NO_REVERSE|TB_ALIGN_16},
   {X86::VMOVDI2SSZrr, X86::MOV32mr, TB_FOLDED_STORE|TB_NO_REVERSE},
   {X86::VMOVDI2SSrr, X86::MOV32mr, TB_FOLDED_STORE|TB_NO_REVERSE},
-  {X86::VMOVDQA32Z128rr, X86::VMOVDQA32Z128mr, TB_FOLDED_STORE|TB_ALIGN_16},
-  {X86::VMOVDQA32Z256rr, X86::VMOVDQA32Z256mr, TB_FOLDED_STORE|TB_ALIGN_32},
-  {X86::VMOVDQA32Zrr, X86::VMOVDQA32Zmr, TB_FOLDED_STORE|TB_ALIGN_64},
-  {X86::VMOVDQA64Z128rr, X86::VMOVDQA64Z128mr, TB_FOLDED_STORE|TB_ALIGN_16},
-  {X86::VMOVDQA64Z256rr, X86::VMOVDQA64Z256mr, TB_FOLDED_STORE|TB_ALIGN_32},
-  {X86::VMOVDQA64Zrr, X86::VMOVDQA64Zmr, TB_FOLDED_STORE|TB_ALIGN_64},
+  {X86::VMOVDQA32Z128rr, X86::VMOVDQA32Z128mr, TB_FOLDED_STORE|TB_NO_REVERSE|TB_ALIGN_16},
+  {X86::VMOVDQA32Z256rr, X86::VMOVDQA32Z256mr, TB_FOLDED_STORE|TB_NO_REVERSE|TB_ALIGN_32},
+  {X86::VMOVDQA32Zrr, X86::VMOVDQA32Zmr, TB_FOLDED_STORE|TB_NO_REVERSE|TB_ALIGN_64},
+  {X86::VMOVDQA64Z128rr, X86::VMOVDQA64Z128mr, TB_FOLDED_STORE|TB_NO_REVERSE|TB_ALIGN_16},
+  {X86::VMOVDQA64Z256rr, X86::VMOVDQA64Z256mr, TB_FOLDED_STORE|TB_NO_REVERSE|TB_ALIGN_32},
+  {X86::VMOVDQA64Zrr, X86::VMOVDQA64Zmr, TB_FOLDED_STORE|TB_NO_REVERSE|TB_ALIGN_64},
   {X86::VMOVDQAYrr, X86::VMOVDQAYmr, TB_FOLDED_STORE|TB_ALIGN_32},
   {X86::VMOVDQArr, X86::VMOVDQAmr, TB_FOLDED_STORE|TB_ALIGN_16},
-  {X86::VMOVDQU16Z128rr, X86::VMOVDQU16Z128mr, TB_FOLDED_STORE},
-  {X86::VMOVDQU16Z256rr, X86::VMOVDQU16Z256mr, TB_FOLDED_STORE},
-  {X86::VMOVDQU16Zrr, X86::VMOVDQU16Zmr, TB_FOLDED_STORE},
-  {X86::VMOVDQU32Z128rr, X86::VMOVDQU32Z128mr, TB_FOLDED_STORE},
-  {X86::VMOVDQU32Z256rr, X86::VMOVDQU32Z256mr, TB_FOLDED_STORE},
-  {X86::VMOVDQU32Zrr, X86::VMOVDQU32Zmr, TB_FOLDED_STORE},
-  {X86::VMOVDQU64Z128rr, X86::VMOVDQU64Z128mr, TB_FOLDED_STORE},
-  {X86::VMOVDQU64Z256rr, X86::VMOVDQU64Z256mr, TB_FOLDED_STORE},
-  {X86::VMOVDQU64Zrr, X86::VMOVDQU64Zmr, TB_FOLDED_STORE},
-  {X86::VMOVDQU8Z128rr, X86::VMOVDQU8Z128mr, TB_FOLDED_STORE},
-  {X86::VMOVDQU8Z256rr, X86::VMOVDQU8Z256mr, TB_FOLDED_STORE},
-  {X86::VMOVDQU8Zrr, X86::VMOVDQU8Zmr, TB_FOLDED_STORE},
+  {X86::VMOVDQU16Z128rr, X86::VMOVDQU16Z128mr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVDQU16Z256rr, X86::VMOVDQU16Z256mr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVDQU16Zrr, X86::VMOVDQU16Zmr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVDQU32Z128rr, X86::VMOVDQU32Z128mr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVDQU32Z256rr, X86::VMOVDQU32Z256mr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVDQU32Zrr, X86::VMOVDQU32Zmr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVDQU64Z128rr, X86::VMOVDQU64Z128mr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVDQU64Z256rr, X86::VMOVDQU64Z256mr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVDQU64Zrr, X86::VMOVDQU64Zmr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVDQU8Z128rr, X86::VMOVDQU8Z128mr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVDQU8Z256rr, X86::VMOVDQU8Z256mr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVDQU8Zrr, X86::VMOVDQU8Zmr, TB_FOLDED_STORE|TB_NO_REVERSE},
   {X86::VMOVDQUYrr, X86::VMOVDQUYmr, TB_FOLDED_STORE},
   {X86::VMOVDQUrr, X86::VMOVDQUmr, TB_FOLDED_STORE},
   {X86::VMOVPDI2DIZrr, X86::VMOVPDI2DIZmr, TB_FOLDED_STORE},
@@ -369,16 +370,16 @@ static const X86MemoryFoldTableEntry MemoryFoldTable0[] = {
   {X86::VMOVSDto64rr, X86::VMOVSDmr, TB_FOLDED_STORE|TB_NO_REVERSE},
   {X86::VMOVSS2DIZrr, X86::VMOVSSZmr, TB_FOLDED_STORE},
   {X86::VMOVSS2DIrr, X86::VMOVSSmr, TB_FOLDED_STORE},
-  {X86::VMOVUPDYrr, X86::VMOVUPDYmr, TB_FOLDED_STORE},
-  {X86::VMOVUPDZ128rr, X86::VMOVUPDZ128mr, TB_FOLDED_STORE},
-  {X86::VMOVUPDZ256rr, X86::VMOVUPDZ256mr, TB_FOLDED_STORE},
-  {X86::VMOVUPDZrr, X86::VMOVUPDZmr, TB_FOLDED_STORE},
-  {X86::VMOVUPDrr, X86::VMOVUPDmr, TB_FOLDED_STORE},
-  {X86::VMOVUPSYrr, X86::VMOVUPSYmr, TB_FOLDED_STORE},
-  {X86::VMOVUPSZ128rr, X86::VMOVUPSZ128mr, TB_FOLDED_STORE},
-  {X86::VMOVUPSZ256rr, X86::VMOVUPSZ256mr, TB_FOLDED_STORE},
-  {X86::VMOVUPSZrr, X86::VMOVUPSZmr, TB_FOLDED_STORE},
-  {X86::VMOVUPSrr, X86::VMOVUPSmr, TB_FOLDED_STORE},
+  {X86::VMOVUPDYrr, X86::VMOVUPDYmr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVUPDZ128rr, X86::VMOVUPDZ128mr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVUPDZ256rr, X86::VMOVUPDZ256mr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVUPDZrr, X86::VMOVUPDZmr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVUPDrr, X86::VMOVUPDmr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVUPSYrr, X86::VMOVUPSYmr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVUPSZ128rr, X86::VMOVUPSZ128mr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVUPSZ256rr, X86::VMOVUPSZ256mr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVUPSZrr, X86::VMOVUPSZmr, TB_FOLDED_STORE|TB_NO_REVERSE},
+  {X86::VMOVUPSrr, X86::VMOVUPSmr, TB_FOLDED_STORE|TB_NO_REVERSE},
   {X86::VPEXTRDZrr, X86::VPEXTRDZmr, TB_FOLDED_STORE},
   {X86::VPEXTRDrr, X86::VPEXTRDmr, TB_FOLDED_STORE},
   {X86::VPEXTRQZrr, X86::VPEXTRQZmr, TB_FOLDED_STORE},
@@ -488,6 +489,10 @@ static const X86MemoryFoldTableEntry MemoryFoldTable1[] = {
   {X86::IMUL32rri8, X86::IMUL32rmi8, 0},
   {X86::IMUL64rri32, X86::IMUL64rmi32, 0},
   {X86::IMUL64rri8, X86::IMUL64rmi8, 0},
+  {X86::KMOVBkk, X86::KMOVBkm, TB_NO_REVERSE},
+  {X86::KMOVDkk, X86::KMOVDkm, 0},
+  {X86::KMOVQkk, X86::KMOVQkm, 0},
+  {X86::KMOVWkk, X86::KMOVWkm, 0},
   {X86::LWPINS32rri, X86::LWPINS32rmi, 0},
   {X86::LWPINS64rri, X86::LWPINS64rmi, 0},
   {X86::LWPVAL32rri, X86::LWPVAL32rmi, 0},
@@ -500,6 +505,7 @@ static const X86MemoryFoldTableEntry MemoryFoldTable1[] = {
   {X86::MMX_CVTPS2PIrr, X86::MMX_CVTPS2PIrm, TB_NO_REVERSE},
   {X86::MMX_CVTTPD2PIrr, X86::MMX_CVTTPD2PIrm, TB_ALIGN_16},
   {X86::MMX_CVTTPS2PIrr, X86::MMX_CVTTPS2PIrm, TB_NO_REVERSE},
+  {X86::MMX_MOVD64rr, X86::MMX_MOVD64rm, 0},
   {X86::MMX_MOVD64to64rr, X86::MMX_MOVQ64rm, 0},
   {X86::MMX_PABSBrr, X86::MMX_PABSBrm, 0},
   {X86::MMX_PABSDrr, X86::MMX_PABSDrm, 0},
@@ -520,8 +526,10 @@ static const X86MemoryFoldTableEntry MemoryFoldTable1[] = {
   {X86::MOVDQUrr, X86::MOVDQUrm, 0},
   {X86::MOVSHDUPrr, X86::MOVSHDUPrm, TB_ALIGN_16},
   {X86::MOVSLDUPrr, X86::MOVSLDUPrm, TB_ALIGN_16},
+  {X86::MOVSX16rr32, X86::MOVSX16rm32, 0},
   {X86::MOVSX16rr8, X86::MOVSX16rm8, 0},
   {X86::MOVSX32rr16, X86::MOVSX32rm16, 0},
+  {X86::MOVSX32rr32, X86::MOVSX32rm32, 0},
   {X86::MOVSX32rr8, X86::MOVSX32rm8, 0},
   {X86::MOVSX32rr8_NOREX, X86::MOVSX32rm8_NOREX, 0},
   {X86::MOVSX64rr16, X86::MOVSX64rm16, 0},
@@ -620,6 +628,8 @@ static const X86MemoryFoldTableEntry MemoryFoldTable1[] = {
   {X86::VCOMISDZrr_Int, X86::VCOMISDZrm_Int, TB_NO_REVERSE},
   {X86::VCOMISDrr, X86::VCOMISDrm, 0},
   {X86::VCOMISDrr_Int, X86::VCOMISDrm_Int, TB_NO_REVERSE},
+  {X86::VCOMISHZrr, X86::VCOMISHZrm, 0},
+  {X86::VCOMISHZrr_Int, X86::VCOMISHZrm_Int, TB_NO_REVERSE},
   {X86::VCOMISSZrr, X86::VCOMISSZrm, 0},
   {X86::VCOMISSZrr_Int, X86::VCOMISSZrm_Int, TB_NO_REVERSE},
   {X86::VCOMISSrr, X86::VCOMISSrm, 0},
@@ -629,19 +639,27 @@ static const X86MemoryFoldTableEntry MemoryFoldTable1[] = {
   {X86::VCVTDQ2PDZ256rr, X86::VCVTDQ2PDZ256rm, 0},
   {X86::VCVTDQ2PDZrr, X86::VCVTDQ2PDZrm, 0},
   {X86::VCVTDQ2PDrr, X86::VCVTDQ2PDrm, TB_NO_REVERSE},
+  {X86::VCVTDQ2PHZ128rr, X86::VCVTDQ2PHZ128rm, 0},
+  {X86::VCVTDQ2PHZ256rr, X86::VCVTDQ2PHZ256rm, 0},
+  {X86::VCVTDQ2PHZrr, X86::VCVTDQ2PHZrm, 0},
   {X86::VCVTDQ2PSYrr, X86::VCVTDQ2PSYrm, 0},
   {X86::VCVTDQ2PSZ128rr, X86::VCVTDQ2PSZ128rm, 0},
   {X86::VCVTDQ2PSZ256rr, X86::VCVTDQ2PSZ256rm, 0},
   {X86::VCVTDQ2PSZrr, X86::VCVTDQ2PSZrm, 0},
   {X86::VCVTDQ2PSrr, X86::VCVTDQ2PSrm, 0},
+  {X86::VCVTNEPS2BF16Yrr, X86::VCVTNEPS2BF16Yrm, 0},
   {X86::VCVTNEPS2BF16Z128rr, X86::VCVTNEPS2BF16Z128rm, 0},
   {X86::VCVTNEPS2BF16Z256rr, X86::VCVTNEPS2BF16Z256rm, 0},
   {X86::VCVTNEPS2BF16Zrr, X86::VCVTNEPS2BF16Zrm, 0},
+  {X86::VCVTNEPS2BF16rr, X86::VCVTNEPS2BF16rm, 0},
   {X86::VCVTPD2DQYrr, X86::VCVTPD2DQYrm, 0},
   {X86::VCVTPD2DQZ128rr, X86::VCVTPD2DQZ128rm, 0},
   {X86::VCVTPD2DQZ256rr, X86::VCVTPD2DQZ256rm, 0},
   {X86::VCVTPD2DQZrr, X86::VCVTPD2DQZrm, 0},
   {X86::VCVTPD2DQrr, X86::VCVTPD2DQrm, 0},
+  {X86::VCVTPD2PHZ128rr, X86::VCVTPD2PHZ128rm, 0},
+  {X86::VCVTPD2PHZ256rr, X86::VCVTPD2PHZ256rm, 0},
+  {X86::VCVTPD2PHZrr, X86::VCVTPD2PHZrm, 0},
   {X86::VCVTPD2PSYrr, X86::VCVTPD2PSYrm, 0},
   {X86::VCVTPD2PSZ128rr, X86::VCVTPD2PSZ128rm, 0},
   {X86::VCVTPD2PSZ256rr, X86::VCVTPD2PSZ256rm, 0},
@@ -656,11 +674,35 @@ static const X86MemoryFoldTableEntry MemoryFoldTable1[] = {
   {X86::VCVTPD2UQQZ128rr, X86::VCVTPD2UQQZ128rm, 0},
   {X86::VCVTPD2UQQZ256rr, X86::VCVTPD2UQQZ256rm, 0},
   {X86::VCVTPD2UQQZrr, X86::VCVTPD2UQQZrm, 0},
+  {X86::VCVTPH2DQZ128rr, X86::VCVTPH2DQZ128rm, TB_NO_REVERSE},
+  {X86::VCVTPH2DQZ256rr, X86::VCVTPH2DQZ256rm, 0},
+  {X86::VCVTPH2DQZrr, X86::VCVTPH2DQZrm, 0},
+  {X86::VCVTPH2PDZ128rr, X86::VCVTPH2PDZ128rm, TB_NO_REVERSE},
+  {X86::VCVTPH2PDZ256rr, X86::VCVTPH2PDZ256rm, TB_NO_REVERSE},
+  {X86::VCVTPH2PDZrr, X86::VCVTPH2PDZrm, 0},
+  {X86::VCVTPH2PSXZ128rr, X86::VCVTPH2PSXZ128rm, TB_NO_REVERSE},
+  {X86::VCVTPH2PSXZ256rr, X86::VCVTPH2PSXZ256rm, 0},
+  {X86::VCVTPH2PSXZrr, X86::VCVTPH2PSXZrm, 0},
   {X86::VCVTPH2PSYrr, X86::VCVTPH2PSYrm, 0},
   {X86::VCVTPH2PSZ128rr, X86::VCVTPH2PSZ128rm, TB_NO_REVERSE},
   {X86::VCVTPH2PSZ256rr, X86::VCVTPH2PSZ256rm, 0},
   {X86::VCVTPH2PSZrr, X86::VCVTPH2PSZrm, 0},
   {X86::VCVTPH2PSrr, X86::VCVTPH2PSrm, TB_NO_REVERSE},
+  {X86::VCVTPH2QQZ128rr, X86::VCVTPH2QQZ128rm, TB_NO_REVERSE},
+  {X86::VCVTPH2QQZ256rr, X86::VCVTPH2QQZ256rm, TB_NO_REVERSE},
+  {X86::VCVTPH2QQZrr, X86::VCVTPH2QQZrm, 0},
+  {X86::VCVTPH2UDQZ128rr, X86::VCVTPH2UDQZ128rm, TB_NO_REVERSE},
+  {X86::VCVTPH2UDQZ256rr, X86::VCVTPH2UDQZ256rm, 0},
+  {X86::VCVTPH2UDQZrr, X86::VCVTPH2UDQZrm, 0},
+  {X86::VCVTPH2UQQZ128rr, X86::VCVTPH2UQQZ128rm, TB_NO_REVERSE},
+  {X86::VCVTPH2UQQZ256rr, X86::VCVTPH2UQQZ256rm, TB_NO_REVERSE},
+  {X86::VCVTPH2UQQZrr, X86::VCVTPH2UQQZrm, 0},
+  {X86::VCVTPH2UWZ128rr, X86::VCVTPH2UWZ128rm, 0},
+  {X86::VCVTPH2UWZ256rr, X86::VCVTPH2UWZ256rm, 0},
+  {X86::VCVTPH2UWZrr, X86::VCVTPH2UWZrm, 0},
+  {X86::VCVTPH2WZ128rr, X86::VCVTPH2WZ128rm, 0},
+  {X86::VCVTPH2WZ256rr, X86::VCVTPH2WZ256rm, 0},
+  {X86::VCVTPH2WZrr, X86::VCVTPH2WZrm, 0},
   {X86::VCVTPS2DQYrr, X86::VCVTPS2DQYrm, 0},
   {X86::VCVTPS2DQZ128rr, X86::VCVTPS2DQZ128rm, 0},
   {X86::VCVTPS2DQZ256rr, X86::VCVTPS2DQZ256rm, 0},
@@ -671,6 +713,9 @@ static const X86MemoryFoldTableEntry MemoryFoldTable1[] = {
   {X86::VCVTPS2PDZ256rr, X86::VCVTPS2PDZ256rm, 0},
   {X86::VCVTPS2PDZrr, X86::VCVTPS2PDZrm, 0},
   {X86::VCVTPS2PDrr, X86::VCVTPS2PDrm, TB_NO_REVERSE},
+  {X86::VCVTPS2PHXZ128rr, X86::VCVTPS2PHXZ128rm, 0},
+  {X86::VCVTPS2PHXZ256rr, X86::VCVTPS2PHXZ256rm, 0},
+  {X86::VCVTPS2PHXZrr, X86::VCVTPS2PHXZrm, 0},
   {X86::VCVTPS2QQZ128rr, X86::VCVTPS2QQZ128rm, TB_NO_REVERSE},
   {X86::VCVTPS2QQZ256rr, X86::VCVTPS2QQZ256rm, 0},
   {X86::VCVTPS2QQZrr, X86::VCVTPS2QQZrm, 0},
@@ -683,6 +728,9 @@ static const X86MemoryFoldTableEntry MemoryFoldTable1[] = {
   {X86::VCVTQQ2PDZ128rr, X86::VCVTQQ2PDZ128rm, 0},
   {X86::VCVTQQ2PDZ256rr, X86::VCVTQQ2PDZ256rm, 0},
   {X86::VCVTQQ2PDZrr, X86::VCVTQQ2PDZrm, 0},
+  {X86::VCVTQQ2PHZ128rr, X86::VCVTQQ2PHZ128rm, 0},
+  {X86::VCVTQQ2PHZ256rr, X86::VCVTQQ2PHZ256rm, 0},
+  {X86::VCVTQQ2PHZrr, X86::VCVTQQ2PHZrm, 0},
   {X86::VCVTQQ2PSZ128rr, X86::VCVTQQ2PSZ128rm, 0},
   {X86::VCVTQQ2PSZ256rr, X86::VCVTQQ2PSZ256rm, 0},
   {X86::VCVTQQ2PSZrr, X86::VCVTQQ2PSZrm, 0},
@@ -696,6 +744,10 @@ static const X86MemoryFoldTableEntry MemoryFoldTable1[] = {
   {X86::VCVTSD2SIrr_Int, X86::VCVTSD2SIrm_Int, TB_NO_REVERSE},
   {X86::VCVTSD2USI64Zrr_Int, X86::VCVTSD2USI64Zrm_Int, TB_NO_REVERSE},
   {X86::VCVTSD2USIZrr_Int, X86::VCVTSD2USIZrm_Int, TB_NO_REVERSE},
+  {X86::VCVTSH2SI64Zrr_Int, X86::VCVTSH2SI64Zrm_Int, TB_NO_REVERSE},
+  {X86::VCVTSH2SIZrr_Int, X86::VCVTSH2SIZrm_Int, TB_NO_REVERSE},
+  {X86::VCVTSH2USI64Zrr_Int, X86::VCVTSH2USI64Zrm_Int, TB_NO_REVERSE},
+  {X86::VCVTSH2USIZrr_Int, X86::VCVTSH2USIZrm_Int, TB_NO_REVERSE},
   {X86::VCVTSS2SI64Zrr, X86::VCVTSS2SI64Zrm, 0},
   {X86::VCVTSS2SI64Zrr_Int, X86::VCVTSS2SI64Zrm_Int, TB_NO_REVERSE},
   {X86::VCVTSS2SI64rr, X86::VCVTSS2SI64rm, 0},
@@ -720,6 +772,24 @@ static const X86MemoryFoldTableEntry MemoryFoldTable1[] = {
   {X86::VCVTTPD2UQQZ128rr, X86::VCVTTPD2UQQZ128rm, 0},
   {X86::VCVTTPD2UQQZ256rr, X86::VCVTTPD2UQQZ256rm, 0},
   {X86::VCVTTPD2UQQZrr, X86::VCVTTPD2UQQZrm, 0},
+  {X86::VCVTTPH2DQZ128rr, X86::VCVTTPH2DQZ128rm, TB_NO_REVERSE},
+  {X86::VCVTTPH2DQZ256rr, X86::VCVTTPH2DQZ256rm, 0},
+  {X86::VCVTTPH2DQZrr, X86::VCVTTPH2DQZrm, 0},
+  {X86::VCVTTPH2QQZ128rr, X86::VCVTTPH2QQZ128rm, TB_NO_REVERSE},
+  {X86::VCVTTPH2QQZ256rr, X86::VCVTTPH2QQZ256rm, TB_NO_REVERSE},
+  {X86::VCVTTPH2QQZrr, X86::VCVTTPH2QQZrm, 0},
+  {X86::VCVTTPH2UDQZ128rr, X86::VCVTTPH2UDQZ128rm, TB_NO_REVERSE},
+  {X86::VCVTTPH2UDQZ256rr, X86::VCVTTPH2UDQZ256rm, 0},
+  {X86::VCVTTPH2UDQZrr, X86::VCVTTPH2UDQZrm, 0},
+  {X86::VCVTTPH2UQQZ128rr, X86::VCVTTPH2UQQZ128rm, TB_NO_REVERSE},
+  {X86::VCVTTPH2UQQZ256rr, X86::VCVTTPH2UQQZ256rm, TB_NO_REVERSE},
+  {X86::VCVTTPH2UQQZrr, X86::VCVTTPH2UQQZrm, 0},
+  {X86::VCVTTPH2UWZ128rr, X86::VCVTTPH2UWZ128rm, 0},
+  {X86::VCVTTPH2UWZ256rr, X86::VCVTTPH2UWZ256rm, 0},
+  {X86::VCVTTPH2UWZrr, X86::VCVTTPH2UWZrm, 0},
+  {X86::VCVTTPH2WZ128rr, X86::VCVTTPH2WZ128rm, 0},
+  {X86::VCVTTPH2WZ256rr, X86::VCVTTPH2WZ256rm, 0},
+  {X86::VCVTTPH2WZrr, X86::VCVTTPH2WZrm, 0},
   {X86::VCVTTPS2DQYrr, X86::VCVTTPS2DQYrm, 0},
   {X86::VCVTTPS2DQZ128rr, X86::VCVTTPS2DQZ128rm, 0},
   {X86::VCVTTPS2DQZ256rr, X86::VCVTTPS2DQZ256rm, 0},
@@ -746,6 +816,14 @@ static const X86MemoryFoldTableEntry MemoryFoldTable1[] = {
   {X86::VCVTTSD2USI64Zrr_Int, X86::VCVTTSD2USI64Zrm_Int, TB_NO_REVERSE},
   {X86::VCVTTSD2USIZrr, X86::VCVTTSD2USIZrm, 0},
   {X86::VCVTTSD2USIZrr_Int, X86::VCVTTSD2USIZrm_Int, TB_NO_REVERSE},
+  {X86::VCVTTSH2SI64Zrr, X86::VCVTTSH2SI64Zrm, 0},
+  {X86::VCVTTSH2SI64Zrr_Int, X86::VCVTTSH2SI64Zrm_Int, TB_NO_REVERSE},
+  {X86::VCVTTSH2SIZrr, X86::VCVTTSH2SIZrm, 0},
+  {X86::VCVTTSH2SIZrr_Int, X86::VCVTTSH2SIZrm_Int, TB_NO_REVERSE},
+  {X86::VCVTTSH2USI64Zrr, X86::VCVTTSH2USI64Zrm, 0},
+  {X86::VCVTTSH2USI64Zrr_Int, X86::VCVTTSH2USI64Zrm_Int, TB_NO_REVERSE},
+  {X86::VCVTTSH2USIZrr, X86::VCVTTSH2USIZrm, 0},
+  {X86::VCVTTSH2USIZrr_Int, X86::VCVTTSH2USIZrm_Int, TB_NO_REVERSE},
   {X86::VCVTTSS2SI64Zrr, X86::VCVTTSS2SI64Zrm, 0},
   {X86::VCVTTSS2SI64Zrr_Int, X86::VCVTTSS2SI64Zrm_Int, TB_NO_REVERSE},
   {X86::VCVTTSS2SI64rr, X86::VCVTTSS2SI64rm, 0},
@@ -761,15 +839,27 @@ static const X86MemoryFoldTableEntry MemoryFoldTable1[] = {
   {X86::VCVTUDQ2PDZ128rr, X86::VCVTUDQ2PDZ128rm, TB_NO_REVERSE},
   {X86::VCVTUDQ2PDZ256rr, X86::VCVTUDQ2PDZ256rm, 0},
   {X86::VCVTUDQ2PDZrr, X86::VCVTUDQ2PDZrm, 0},
+  {X86::VCVTUDQ2PHZ128rr, X86::VCVTUDQ2PHZ128rm, 0},
+  {X86::VCVTUDQ2PHZ256rr, X86::VCVTUDQ2PHZ256rm, 0},
+  {X86::VCVTUDQ2PHZrr, X86::VCVTUDQ2PHZrm, 0},
   {X86::VCVTUDQ2PSZ128rr, X86::VCVTUDQ2PSZ128rm, 0},
   {X86::VCVTUDQ2PSZ256rr, X86::VCVTUDQ2PSZ256rm, 0},
   {X86::VCVTUDQ2PSZrr, X86::VCVTUDQ2PSZrm, 0},
   {X86::VCVTUQQ2PDZ128rr, X86::VCVTUQQ2PDZ128rm, 0},
   {X86::VCVTUQQ2PDZ256rr, X86::VCVTUQQ2PDZ256rm, 0},
   {X86::VCVTUQQ2PDZrr, X86::VCVTUQQ2PDZrm, 0},
+  {X86::VCVTUQQ2PHZ128rr, X86::VCVTUQQ2PHZ128rm, 0},
+  {X86::VCVTUQQ2PHZ256rr, X86::VCVTUQQ2PHZ256rm, 0},
+  {X86::VCVTUQQ2PHZrr, X86::VCVTUQQ2PHZrm, 0},
   {X86::VCVTUQQ2PSZ128rr, X86::VCVTUQQ2PSZ128rm, 0},
   {X86::VCVTUQQ2PSZ256rr, X86::VCVTUQQ2PSZ256rm, 0},
   {X86::VCVTUQQ2PSZrr, X86::VCVTUQQ2PSZrm, 0},
+  {X86::VCVTUW2PHZ128rr, X86::VCVTUW2PHZ128rm, 0},
+  {X86::VCVTUW2PHZ256rr, X86::VCVTUW2PHZ256rm, 0},
+  {X86::VCVTUW2PHZrr, X86::VCVTUW2PHZrm, 0},
+  {X86::VCVTW2PHZ128rr, X86::VCVTW2PHZ128rm, 0},
+  {X86::VCVTW2PHZ256rr, X86::VCVTW2PHZ256rm, 0},
+  {X86::VCVTW2PHZrr, X86::VCVTW2PHZrm, 0},
   {X86::VEXP2PDZr, X86::VEXP2PDZm, 0},
   {X86::VEXP2PSZr, X86::VEXP2PSZm, 0},
   {X86::VEXPANDPDZ128rr, X86::VEXPANDPDZ128rm, TB_NO_REVERSE},
@@ -879,6 +969,7 @@ static const X86MemoryFoldTableEntry MemoryFoldTable1[] = {
   {X86::VMOVUPSZ256rr, X86::VMOVUPSZ256rm, 0},
   {X86::VMOVUPSZrr, X86::VMOVUPSZrm, 0},
   {X86::VMOVUPSrr, X86::VMOVUPSrm, 0},
+  {X86::VMOVW2SHrr, X86::VMOVWrm, TB_NO_REVERSE},
   {X86::VMOVZPQILo2PQIZrr, X86::VMOVQI2PQIZrm, TB_NO_REVERSE},
   {X86::VMOVZPQILo2PQIrr, X86::VMOVQI2PQIrm, TB_NO_REVERSE},
   {X86::VPABSBYrr, X86::VPABSBYrm, 0},
@@ -1198,6 +1289,8 @@ static const X86MemoryFoldTableEntry MemoryFoldTable1[] = {
   {X86::VUCOMISDZrr_Int, X86::VUCOMISDZrm_Int, TB_NO_REVERSE},
   {X86::VUCOMISDrr, X86::VUCOMISDrm, 0},
   {X86::VUCOMISDrr_Int, X86::VUCOMISDrm_Int, TB_NO_REVERSE},
+  {X86::VUCOMISHZrr, X86::VUCOMISHZrm, 0},
+  {X86::VUCOMISHZrr_Int, X86::VUCOMISHZrm_Int, TB_NO_REVERSE},
   {X86::VUCOMISSZrr, X86::VUCOMISSZrm, 0},
   {X86::VUCOMISSZrr_Int, X86::VUCOMISSZrm_Int, TB_NO_REVERSE},
   {X86::VUCOMISSrr, X86::VUCOMISSrm, 0},
@@ -1659,6 +1752,9 @@ static const X86MemoryFoldTableEntry MemoryFoldTable2[] = {
   {X86::VCVTDQ2PDZ128rrkz, X86::VCVTDQ2PDZ128rmkz, TB_NO_REVERSE},
   {X86::VCVTDQ2PDZ256rrkz, X86::VCVTDQ2PDZ256rmkz, 0},
   {X86::VCVTDQ2PDZrrkz, X86::VCVTDQ2PDZrmkz, 0},
+  {X86::VCVTDQ2PHZ128rrkz, X86::VCVTDQ2PHZ128rmkz, 0},
+  {X86::VCVTDQ2PHZ256rrkz, X86::VCVTDQ2PHZ256rmkz, 0},
+  {X86::VCVTDQ2PHZrrkz, X86::VCVTDQ2PHZrmkz, 0},
   {X86::VCVTDQ2PSZ128rrkz, X86::VCVTDQ2PSZ128rmkz, 0},
   {X86::VCVTDQ2PSZ256rrkz, X86::VCVTDQ2PSZ256rmkz, 0},
   {X86::VCVTDQ2PSZrrkz, X86::VCVTDQ2PSZrmkz, 0},
@@ -1671,6 +1767,9 @@ static const X86MemoryFoldTableEntry MemoryFoldTable2[] = {
   {X86::VCVTPD2DQZ128rrkz, X86::VCVTPD2DQZ128rmkz, 0},
   {X86::VCVTPD2DQZ256rrkz, X86::VCVTPD2DQZ256rmkz, 0},
   {X86::VCVTPD2DQZrrkz, X86::VCVTPD2DQZrmkz, 0},
+  {X86::VCVTPD2PHZ128rrkz, X86::VCVTPD2PHZ128rmkz, 0},
+  {X86::VCVTPD2PHZ256rrkz, X86::VCVTPD2PHZ256rmkz, 0},
+  {X86::VCVTPD2PHZrrkz, X86::VCVTPD2PHZrmkz, 0},
   {X86::VCVTPD2PSZ128rrkz, X86::VCVTPD2PSZ128rmkz, 0},
   {X86::VCVTPD2PSZ256rrkz, X86::VCVTPD2PSZ256rmkz, 0},
   {X86::VCVTPD2PSZrrkz, X86::VCVTPD2PSZrmkz, 0},
@@ -1683,15 +1782,42 @@ static const X86MemoryFoldTableEntry MemoryFoldTable2[] = {
   {X86::VCVTPD2UQQZ128rrkz, X86::VCVTPD2UQQZ128rmkz, 0},
   {X86::VCVTPD2UQQZ256rrkz, X86::VCVTPD2UQQZ256rmkz, 0},
   {X86::VCVTPD2UQQZrrkz, X86::VCVTPD2UQQZrmkz, 0},
+  {X86::VCVTPH2DQZ128rrkz, X86::VCVTPH2DQZ128rmkz, TB_NO_REVERSE},
+  {X86::VCVTPH2DQZ256rrkz, X86::VCVTPH2DQZ256rmkz, 0},
+  {X86::VCVTPH2DQZrrkz, X86::VCVTPH2DQZrmkz, 0},
+  {X86::VCVTPH2PDZ128rrkz, X86::VCVTPH2PDZ128rmkz, TB_NO_REVERSE},
+  {X86::VCVTPH2PDZ256rrkz, X86::VCVTPH2PDZ256rmkz, TB_NO_REVERSE},
+  {X86::VCVTPH2PDZrrkz, X86::VCVTPH2PDZrmkz, 0},
+  {X86::VCVTPH2PSXZ128rrkz, X86::VCVTPH2PSXZ128rmkz, TB_NO_REVERSE},
+  {X86::VCVTPH2PSXZ256rrkz, X86::VCVTPH2PSXZ256rmkz, 0},
+  {X86::VCVTPH2PSXZrrkz, X86::VCVTPH2PSXZrmkz, 0},
   {X86::VCVTPH2PSZ128rrkz, X86::VCVTPH2PSZ128rmkz, TB_NO_REVERSE},
   {X86::VCVTPH2PSZ256rrkz, X86::VCVTPH2PSZ256rmkz, 0},
   {X86::VCVTPH2PSZrrkz, X86::VCVTPH2PSZrmkz, 0},
+  {X86::VCVTPH2QQZ128rrkz, X86::VCVTPH2QQZ128rmkz, TB_NO_REVERSE},
+  {X86::VCVTPH2QQZ256rrkz, X86::VCVTPH2QQZ256rmkz, TB_NO_REVERSE},
+  {X86::VCVTPH2QQZrrkz, X86::VCVTPH2QQZrmkz, 0},
+  {X86::VCVTPH2UDQZ128rrkz, X86::VCVTPH2UDQZ128rmkz, TB_NO_REVERSE},
+  {X86::VCVTPH2UDQZ256rrkz, X86::VCVTPH2UDQZ256rmkz, 0},
+  {X86::VCVTPH2UDQZrrkz, X86::VCVTPH2UDQZrmkz, 0},
+  {X86::VCVTPH2UQQZ128rrkz, X86::VCVTPH2UQQZ128rmkz, TB_NO_REVERSE},
+  {X86::VCVTPH2UQQZ256rrkz, X86::VCVTPH2UQQZ256rmkz, TB_NO_REVERSE},
+  {X86::VCVTPH2UQQZrrkz, X86::VCVTPH2UQQZrmkz, 0},
+  {X86::VCVTPH2UWZ128rrkz, X86::VCVTPH2UWZ128rmkz, 0},
+  {X86::VCVTPH2UWZ256rrkz, X86::VCVTPH2UWZ256rmkz, 0},
+  {X86::VCVTPH2UWZrrkz, X86::VCVTPH2UWZrmkz, 0},
+  {X86::VCVTPH2WZ128rrkz, X86::VCVTPH2WZ128rmkz, 0},
+  {X86::VCVTPH2WZ256rrkz, X86::VCVTPH2WZ256rmkz, 0},
+  {X86::VCVTPH2WZrrkz, X86::VCVTPH2WZrmkz, 0},
   {X86::VCVTPS2DQZ128rrkz, X86::VCVTPS2DQZ128rmkz, 0},
   {X86::VCVTPS2DQZ256rrkz, X86::VCVTPS2DQZ256rmkz, 0},
   {X86::VCVTPS2DQZrrkz, X86::VCVTPS2DQZrmkz, 0},
   {X86::VCVTPS2PDZ128rrkz, X86::VCVTPS2PDZ128rmkz, TB_NO_REVERSE},
   {X86::VCVTPS2PDZ256rrkz, X86::VCVTPS2PDZ256rmkz, 0},
   {X86::VCVTPS2PDZrrkz, X86::VCVTPS2PDZrmkz, 0},
+  {X86::VCVTPS2PHXZ128rrkz, X86::VCVTPS2PHXZ128rmkz, 0},
+  {X86::VCVTPS2PHXZ256rrkz, X86::VCVTPS2PHXZ256rmkz, 0},
+  {X86::VCVTPS2PHXZrrkz, X86::VCVTPS2PHXZrmkz, 0},
   {X86::VCVTPS2QQZ128rrkz, X86::VCVTPS2QQZ128rmkz, TB_NO_REVERSE},
   {X86::VCVTPS2QQZ256rrkz, X86::VCVTPS2QQZ256rmkz, 0},
   {X86::VCVTPS2QQZrrkz, X86::VCVTPS2QQZrmkz, 0},
@@ -1704,17 +1830,28 @@ static const X86MemoryFoldTableEntry MemoryFoldTable2[] = {
   {X86::VCVTQQ2PDZ128rrkz, X86::VCVTQQ2PDZ128rmkz, 0},
   {X86::VCVTQQ2PDZ256rrkz, X86::VCVTQQ2PDZ256rmkz, 0},
   {X86::VCVTQQ2PDZrrkz, X86::VCVTQQ2PDZrmkz, 0},
+  {X86::VCVTQQ2PHZ128rrkz, X86::VCVTQQ2PHZ128rmkz, 0},
+  {X86::VCVTQQ2PHZ256rrkz, X86::VCVTQQ2PHZ256rmkz, 0},
+  {X86::VCVTQQ2PHZrrkz, X86::VCVTQQ2PHZrmkz, 0},
   {X86::VCVTQQ2PSZ128rrkz, X86::VCVTQQ2PSZ128rmkz, 0},
   {X86::VCVTQQ2PSZ256rrkz, X86::VCVTQQ2PSZ256rmkz, 0},
   {X86::VCVTQQ2PSZrrkz, X86::VCVTQQ2PSZrmkz, 0},
+  {X86::VCVTSD2SHZrr, X86::VCVTSD2SHZrm, 0},
+  {X86::VCVTSD2SHZrr_Int, X86::VCVTSD2SHZrm_Int, TB_NO_REVERSE},
   {X86::VCVTSD2SSZrr, X86::VCVTSD2SSZrm, 0},
   {X86::VCVTSD2SSZrr_Int, X86::VCVTSD2SSZrm_Int, TB_NO_REVERSE},
   {X86::VCVTSD2SSrr, X86::VCVTSD2SSrm, 0},
   {X86::VCVTSD2SSrr_Int, X86::VCVTSD2SSrm_Int, TB_NO_REVERSE},
+  {X86::VCVTSH2SDZrr, X86::VCVTSH2SDZrm, 0},
+  {X86::VCVTSH2SDZrr_Int, X86::VCVTSH2SDZrm_Int, TB_NO_REVERSE},
+  {X86::VCVTSH2SSZrr, X86::VCVTSH2SSZrm, 0},
+  {X86::VCVTSH2SSZrr_Int, X86::VCVTSH2SSZrm_Int, TB_NO_REVERSE},
   {X86::VCVTSI2SDZrr, X86::VCVTSI2SDZrm, 0},
   {X86::VCVTSI2SDZrr_Int, X86::VCVTSI2SDZrm_Int, 0},
   {X86::VCVTSI2SDrr, X86::VCVTSI2SDrm, 0},
   {X86::VCVTSI2SDrr_Int, X86::VCVTSI2SDrm_Int, 0},
+  {X86::VCVTSI2SHZrr, X86::VCVTSI2SHZrm, 0},
+  {X86::VCVTSI2SHZrr_Int, X86::VCVTSI2SHZrm_Int, 0},
   {X86::VCVTSI2SSZrr, X86::VCVTSI2SSZrm, 0},
   {X86::VCVTSI2SSZrr_Int, X86::VCVTSI2SSZrm_Int, 0},
   {X86::VCVTSI2SSrr, X86::VCVTSI2SSrm, 0},
@@ -1723,6 +1860,8 @@ static const X86MemoryFoldTableEntry MemoryFoldTable2[] = {
   {X86::VCVTSI642SDZrr_Int, X86::VCVTSI642SDZrm_Int, 0},
   {X86::VCVTSI642SDrr, X86::VCVTSI642SDrm, 0},
   {X86::VCVTSI642SDrr_Int, X86::VCVTSI642SDrm_Int, 0},
+  {X86::VCVTSI642SHZrr, X86::VCVTSI642SHZrm, 0},
+  {X86::VCVTSI642SHZrr_Int, X86::VCVTSI642SHZrm_Int, 0},
   {X86::VCVTSI642SSZrr, X86::VCVTSI642SSZrm, 0},
   {X86::VCVTSI642SSZrr_Int, X86::VCVTSI642SSZrm_Int, 0},
   {X86::VCVTSI642SSrr, X86::VCVTSI642SSrm, 0},
@@ -1731,6 +1870,8 @@ static const X86MemoryFoldTableEntry MemoryFoldTable2[] = {
   {X86::VCVTSS2SDZrr_Int, X86::VCVTSS2SDZrm_Int, TB_NO_REVERSE},
   {X86::VCVTSS2SDrr, X86::VCVTSS2SDrm, 0},
   {X86::VCVTSS2SDrr_Int, X86::VCVTSS2SDrm_Int, TB_NO_REVERSE},
+  {X86::VCVTSS2SHZrr, X86::VCVTSS2SHZrm, 0},
+  {X86::VCVTSS2SHZrr_Int, X86::VCVTSS2SHZrm_Int, TB_NO_REVERSE},
   {X86::VCVTTPD2DQZ128rrkz, X86::VCVTTPD2DQZ128rmkz, 0},
   {X86::VCVTTPD2DQZ256rrkz, X86::VCVTTPD2DQZ256rmkz, 0},
   {X86::VCVTTPD2DQZrrkz, X86::VCVTTPD2DQZrmkz, 0},
@@ -1743,6 +1884,24 @@ static const X86MemoryFoldTableEntry MemoryFoldTable2[] = {
   {X86::VCVTTPD2UQQZ128rrkz, X86::VCVTTPD2UQQZ128rmkz, 0},
   {X86::VCVTTPD2UQQZ256rrkz, X86::VCVTTPD2UQQZ256rmkz, 0},
   {X86::VCVTTPD2UQQZrrkz, X86::VCVTTPD2UQQZrmkz, 0},
+  {X86::VCVTTPH2DQZ128rrkz, X86::VCVTTPH2DQZ128rmkz, TB_NO_REVERSE},
+  {X86::VCVTTPH2DQZ256rrkz, X86::VCVTTPH2DQZ256rmkz, 0},
+  {X86::VCVTTPH2DQZrrkz, X86::VCVTTPH2DQZrmkz, 0},
+  {X86::VCVTTPH2QQZ128rrkz, X86::VCVTTPH2QQZ128rmkz, TB_NO_REVERSE},
+  {X86::VCVTTPH2QQZ256rrkz, X86::VCVTTPH2QQZ256rmkz, TB_NO_REVERSE},
+  {X86::VCVTTPH2QQZrrkz, X86::VCVTTPH2QQZrmkz, 0},
+  {X86::VCVTTPH2UDQZ128rrkz, X86::VCVTTPH2UDQZ128rmkz, TB_NO_REVERSE},
+  {X86::VCVTTPH2UDQZ256rrkz, X86::VCVTTPH2UDQZ256rmkz, 0},
+  {X86::VCVTTPH2UDQZrrkz, X86::VCVTTPH2UDQZrmkz, 0},
+  {X86::VCVTTPH2UQQZ128rrkz, X86::VCVTTPH2UQQZ128rmkz, TB_NO_REVERSE},
+  {X86::VCVTTPH2UQQZ256rrkz, X86::VCVTTPH2UQQZ256rmkz, TB_NO_REVERSE},
+  {X86::VCVTTPH2UQQZrrkz, X86::VCVTTPH2UQQZrmkz, 0},
+  {X86::VCVTTPH2UWZ128rrkz, X86::VCVTTPH2UWZ128rmkz, 0},
+  {X86::VCVTTPH2UWZ256rrkz, X86::VCVTTPH2UWZ256rmkz, 0},
+  {X86::VCVTTPH2UWZrrkz, X86::VCVTTPH2UWZrmkz, 0},
+  {X86::VCVTTPH2WZ128rrkz, X86::VCVTTPH2WZ128rmkz, 0},
+  {X86::VCVTTPH2WZ256rrkz, X86::VCVTTPH2WZ256rmkz, 0},
+  {X86::VCVTTPH2WZrrkz, X86::VCVTTPH2WZrmkz, 0},
   {X86::VCVTTPS2DQZ128rrkz, X86::VCVTTPS2DQZ128rmkz, 0},
   {X86::VCVTTPS2DQZ256rrkz, X86::VCVTTPS2DQZ256rmkz, 0},
   {X86::VCVTTPS2DQZrrkz, X86::VCVTTPS2DQZrmkz, 0},
@@ -1758,23 +1917,39 @@ static const X86MemoryFoldTableEntry MemoryFoldTable2[] = {
   {X86::VCVTUDQ2PDZ128rrkz, X86::VCVTUDQ2PDZ128rmkz, TB_NO_REVERSE},
   {X86::VCVTUDQ2PDZ256rrkz, X86::VCVTUDQ2PDZ256rmkz, 0},
   {X86::VCVTUDQ2PDZrrkz, X86::VCVTUDQ2PDZrmkz, 0},
+  {X86::VCVTUDQ2PHZ128rrkz, X86::VCVTUDQ2PHZ128rmkz, 0},
+  {X86::VCVTUDQ2PHZ256rrkz, X86::VCVTUDQ2PHZ256rmkz, 0},
+  {X86::VCVTUDQ2PHZrrkz, X86::VCVTUDQ2PHZrmkz, 0},
   {X86::VCVTUDQ2PSZ128rrkz, X86::VCVTUDQ2PSZ128rmkz, 0},
   {X86::VCVTUDQ2PSZ256rrkz, X86::VCVTUDQ2PSZ256rmkz, 0},
   {X86::VCVTUDQ2PSZrrkz, X86::VCVTUDQ2PSZrmkz, 0},
   {X86::VCVTUQQ2PDZ128rrkz, X86::VCVTUQQ2PDZ128rmkz, 0},
   {X86::VCVTUQQ2PDZ256rrkz, X86::VCVTUQQ2PDZ256rmkz, 0},
   {X86::VCVTUQQ2PDZrrkz, X86::VCVTUQQ2PDZrmkz, 0},
+  {X86::VCVTUQQ2PHZ128rrkz, X86::VCVTUQQ2PHZ128rmkz, 0},
+  {X86::VCVTUQQ2PHZ256rrkz, X86::VCVTUQQ2PHZ256rmkz, 0},
+  {X86::VCVTUQQ2PHZrrkz, X86::VCVTUQQ2PHZrmkz, 0},
   {X86::VCVTUQQ2PSZ128rrkz, X86::VCVTUQQ2PSZ128rmkz, 0},
   {X86::VCVTUQQ2PSZ256rrkz, X86::VCVTUQQ2PSZ256rmkz, 0},
   {X86::VCVTUQQ2PSZrrkz, X86::VCVTUQQ2PSZrmkz, 0},
   {X86::VCVTUSI2SDZrr, X86::VCVTUSI2SDZrm, 0},
   {X86::VCVTUSI2SDZrr_Int, X86::VCVTUSI2SDZrm_Int, 0},
+  {X86::VCVTUSI2SHZrr, X86::VCVTUSI2SHZrm, 0},
+  {X86::VCVTUSI2SHZrr_Int, X86::VCVTUSI2SHZrm_Int, 0},
   {X86::VCVTUSI2SSZrr, X86::VCVTUSI2SSZrm, 0},
   {X86::VCVTUSI2SSZrr_Int, X86::VCVTUSI2SSZrm_Int, 0},
   {X86::VCVTUSI642SDZrr, X86::VCVTUSI642SDZrm, 0},
   {X86::VCVTUSI642SDZrr_Int, X86::VCVTUSI642SDZrm_Int, 0},
+  {X86::VCVTUSI642SHZrr, X86::VCVTUSI642SHZrm, 0},
+  {X86::VCVTUSI642SHZrr_Int, X86::VCVTUSI642SHZrm_Int, 0},
   {X86::VCVTUSI642SSZrr, X86::VCVTUSI642SSZrm, 0},
   {X86::VCVTUSI642SSZrr_Int, X86::VCVTUSI642SSZrm_Int, 0},
+  {X86::VCVTUW2PHZ128rrkz, X86::VCVTUW2PHZ128rmkz, 0},
+  {X86::VCVTUW2PHZ256rrkz, X86::VCVTUW2PHZ256rmkz, 0},
+  {X86::VCVTUW2PHZrrkz, X86::VCVTUW2PHZrmkz, 0},
+  {X86::VCVTW2PHZ128rrkz, X86::VCVTW2PHZ128rmkz, 0},
+  {X86::VCVTW2PHZ256rrkz, X86::VCVTW2PHZ256rmkz, 0},
+  {X86::VCVTW2PHZrrkz, X86::VCVTW2PHZrmkz, 0},
   {X86::VDBPSADBWZ128rri, X86::VDBPSADBWZ128rmi, 0},
   {X86::VDBPSADBWZ256rri, X86::VDBPSADBWZ256rmi, 0},
   {X86::VDBPSADBWZrri, X86::VDBPSADBWZrmi, 0},
@@ -1933,6 +2108,8 @@ static const X86MemoryFoldTableEntry MemoryFoldTable2[] = {
   {X86::VINSERTI64x2Z256rr, X86::VINSERTI64x2Z256rm, 0},
   {X86::VINSERTI64x2Zrr, X86::VINSERTI64x2Zrm, 0},
   {X86::VINSERTI64x4Zrr, X86::VINSERTI64x4Zrm, 0},
+  {X86::VINSERTPSZrr, X86::VINSERTPSZrm, TB_NO_REVERSE},
+  {X86::VINSERTPSrr, X86::VINSERTPSrm, TB_NO_REVERSE},
   {X86::VMAXCPDYrr, X86::VMAXCPDYrm, 0},
   {X86::VMAXCPDZ128rr, X86::VMAXCPDZ128rm, 0},
   {X86::VMAXCPDZ256rr, X86::VMAXCPDZ256rm, 0},
@@ -3137,6 +3314,9 @@ static const X86MemoryFoldTableEntry MemoryFoldTable3[] = {
   {X86::VCVTDQ2PDZ128rrk, X86::VCVTDQ2PDZ128rmk, TB_NO_REVERSE},
   {X86::VCVTDQ2PDZ256rrk, X86::VCVTDQ2PDZ256rmk, 0},
   {X86::VCVTDQ2PDZrrk, X86::VCVTDQ2PDZrmk, 0},
+  {X86::VCVTDQ2PHZ128rrk, X86::VCVTDQ2PHZ128rmk, 0},
+  {X86::VCVTDQ2PHZ256rrk, X86::VCVTDQ2PHZ256rmk, 0},
+  {X86::VCVTDQ2PHZrrk, X86::VCVTDQ2PHZrmk, 0},
   {X86::VCVTDQ2PSZ128rrk, X86::VCVTDQ2PSZ128rmk, 0},
   {X86::VCVTDQ2PSZ256rrk, X86::VCVTDQ2PSZ256rmk, 0},
   {X86::VCVTDQ2PSZrrk, X86::VCVTDQ2PSZrmk, 0},
@@ -3149,6 +3329,9 @@ static const X86MemoryFoldTableEntry MemoryFoldTable3[] = {
   {X86::VCVTPD2DQZ128rrk, X86::VCVTPD2DQZ128rmk, 0},
   {X86::VCVTPD2DQZ256rrk, X86::VCVTPD2DQZ256rmk, 0},
   {X86::VCVTPD2DQZrrk, X86::VCVTPD2DQZrmk, 0},
+  {X86::VCVTPD2PHZ128rrk, X86::VCVTPD2PHZ128rmk, 0},
+  {X86::VCVTPD2PHZ256rrk, X86::VCVTPD2PHZ256rmk, 0},
+  {X86::VCVTPD2PHZrrk, X86::VCVTPD2PHZrmk, 0},
   {X86::VCVTPD2PSZ128rrk, X86::VCVTPD2PSZ128rmk, 0},
   {X86::VCVTPD2PSZ256rrk, X86::VCVTPD2PSZ256rmk, 0},
   {X86::VCVTPD2PSZrrk, X86::VCVTPD2PSZrmk, 0},
@@ -3161,15 +3344,42 @@ static const X86MemoryFoldTableEntry MemoryFoldTable3[] = {
   {X86::VCVTPD2UQQZ128rrk, X86::VCVTPD2UQQZ128rmk, 0},
   {X86::VCVTPD2UQQZ256rrk, X86::VCVTPD2UQQZ256rmk, 0},
   {X86::VCVTPD2UQQZrrk, X86::VCVTPD2UQQZrmk, 0},
+  {X86::VCVTPH2DQZ128rrk, X86::VCVTPH2DQZ128rmk, TB_NO_REVERSE},
+  {X86::VCVTPH2DQZ256rrk, X86::VCVTPH2DQZ256rmk, 0},
+  {X86::VCVTPH2DQZrrk, X86::VCVTPH2DQZrmk, 0},
+  {X86::VCVTPH2PDZ128rrk, X86::VCVTPH2PDZ128rmk, TB_NO_REVERSE},
+  {X86::VCVTPH2PDZ256rrk, X86::VCVTPH2PDZ256rmk, TB_NO_REVERSE},
+  {X86::VCVTPH2PDZrrk, X86::VCVTPH2PDZrmk, 0},
+  {X86::VCVTPH2PSXZ128rrk, X86::VCVTPH2PSXZ128rmk, TB_NO_REVERSE},
+  {X86::VCVTPH2PSXZ256rrk, X86::VCVTPH2PSXZ256rmk, 0},
+  {X86::VCVTPH2PSXZrrk, X86::VCVTPH2PSXZrmk, 0},
   {X86::VCVTPH2PSZ128rrk, X86::VCVTPH2PSZ128rmk, TB_NO_REVERSE},
   {X86::VCVTPH2PSZ256rrk, X86::VCVTPH2PSZ256rmk, 0},
   {X86::VCVTPH2PSZrrk, X86::VCVTPH2PSZrmk, 0},
+  {X86::VCVTPH2QQZ128rrk, X86::VCVTPH2QQZ128rmk, TB_NO_REVERSE},
+  {X86::VCVTPH2QQZ256rrk, X86::VCVTPH2QQZ256rmk, TB_NO_REVERSE},
+  {X86::VCVTPH2QQZrrk, X86::VCVTPH2QQZrmk, 0},
+  {X86::VCVTPH2UDQZ128rrk, X86::VCVTPH2UDQZ128rmk, TB_NO_REVERSE},
+  {X86::VCVTPH2UDQZ256rrk, X86::VCVTPH2UDQZ256rmk, 0},
+  {X86::VCVTPH2UDQZrrk, X86::VCVTPH2UDQZrmk, 0},
+  {X86::VCVTPH2UQQZ128rrk, X86::VCVTPH2UQQZ128rmk, TB_NO_REVERSE},
+  {X86::VCVTPH2UQQZ256rrk, X86::VCVTPH2UQQZ256rmk, TB_NO_REVERSE},
+  {X86::VCVTPH2UQQZrrk, X86::VCVTPH2UQQZrmk, 0},
+  {X86::VCVTPH2UWZ128rrk, X86::VCVTPH2UWZ128rmk, 0},
+  {X86::VCVTPH2UWZ256rrk, X86::VCVTPH2UWZ256rmk, 0},
+  {X86::VCVTPH2UWZrrk, X86::VCVTPH2UWZrmk, 0},
+  {X86::VCVTPH2WZ128rrk, X86::VCVTPH2WZ128rmk, 0},
+  {X86::VCVTPH2WZ256rrk, X86::VCVTPH2WZ256rmk, 0},
+  {X86::VCVTPH2WZrrk, X86::VCVTPH2WZrmk, 0},
   {X86::VCVTPS2DQZ128rrk, X86::VCVTPS2DQZ128rmk, 0},
   {X86::VCVTPS2DQZ256rrk, X86::VCVTPS2DQZ256rmk, 0},
   {X86::VCVTPS2DQZrrk, X86::VCVTPS2DQZrmk, 0},
   {X86::VCVTPS2PDZ128rrk, X86::VCVTPS2PDZ128rmk, TB_NO_REVERSE},
   {X86::VCVTPS2PDZ256rrk, X86::VCVTPS2PDZ256rmk, 0},
   {X86::VCVTPS2PDZrrk, X86::VCVTPS2PDZrmk, 0},
+  {X86::VCVTPS2PHXZ128rrk, X86::VCVTPS2PHXZ128rmk, 0},
+  {X86::VCVTPS2PHXZ256rrk, X86::VCVTPS2PHXZ256rmk, 0},
+  {X86::VCVTPS2PHXZrrk, X86::VCVTPS2PHXZrmk, 0},
   {X86::VCVTPS2QQZ128rrk, X86::VCVTPS2QQZ128rmk, TB_NO_REVERSE},
   {X86::VCVTPS2QQZ256rrk, X86::VCVTPS2QQZ256rmk, 0},
   {X86::VCVTPS2QQZrrk, X86::VCVTPS2QQZrmk, 0},
@@ -3182,11 +3392,18 @@ static const X86MemoryFoldTableEntry MemoryFoldTable3[] = {
   {X86::VCVTQQ2PDZ128rrk, X86::VCVTQQ2PDZ128rmk, 0},
   {X86::VCVTQQ2PDZ256rrk, X86::VCVTQQ2PDZ256rmk, 0},
   {X86::VCVTQQ2PDZrrk, X86::VCVTQQ2PDZrmk, 0},
+  {X86::VCVTQQ2PHZ128rrk, X86::VCVTQQ2PHZ128rmk, 0},
+  {X86::VCVTQQ2PHZ256rrk, X86::VCVTQQ2PHZ256rmk, 0},
+  {X86::VCVTQQ2PHZrrk, X86::VCVTQQ2PHZrmk, 0},
   {X86::VCVTQQ2PSZ128rrk, X86::VCVTQQ2PSZ128rmk, 0},
   {X86::VCVTQQ2PSZ256rrk, X86::VCVTQQ2PSZ256rmk, 0},
   {X86::VCVTQQ2PSZrrk, X86::VCVTQQ2PSZrmk, 0},
+  {X86::VCVTSD2SHZrr_Intkz, X86::VCVTSD2SHZrm_Intkz, TB_NO_REVERSE},
   {X86::VCVTSD2SSZrr_Intkz, X86::VCVTSD2SSZrm_Intkz, TB_NO_REVERSE},
+  {X86::VCVTSH2SDZrr_Intkz, X86::VCVTSH2SDZrm_Intkz, TB_NO_REVERSE},
+  {X86::VCVTSH2SSZrr_Intkz, X86::VCVTSH2SSZrm_Intkz, TB_NO_REVERSE},
   {X86::VCVTSS2SDZrr_Intkz, X86::VCVTSS2SDZrm_Intkz, TB_NO_REVERSE},
+  {X86::VCVTSS2SHZrr_Intkz, X86::VCVTSS2SHZrm_Intkz, TB_NO_REVERSE},
   {X86::VCVTTPD2DQZ128rrk, X86::VCVTTPD2DQZ128rmk, 0},
   {X86::VCVTTPD2DQZ256rrk, X86::VCVTTPD2DQZ256rmk, 0},
   {X86::VCVTTPD2DQZrrk, X86::VCVTTPD2DQZrmk, 0},
@@ -3199,6 +3416,24 @@ static const X86MemoryFoldTableEntry MemoryFoldTable3[] = {
   {X86::VCVTTPD2UQQZ128rrk, X86::VCVTTPD2UQQZ128rmk, 0},
   {X86::VCVTTPD2UQQZ256rrk, X86::VCVTTPD2UQQZ256rmk, 0},
   {X86::VCVTTPD2UQQZrrk, X86::VCVTTPD2UQQZrmk, 0},
+  {X86::VCVTTPH2DQZ128rrk, X86::VCVTTPH2DQZ128rmk, TB_NO_REVERSE},
+  {X86::VCVTTPH2DQZ256rrk, X86::VCVTTPH2DQZ256rmk, 0},
+  {X86::VCVTTPH2DQZrrk, X86::VCVTTPH2DQZrmk, 0},
+  {X86::VCVTTPH2QQZ128rrk, X86::VCVTTPH2QQZ128rmk, TB_NO_REVERSE},
+  {X86::VCVTTPH2QQZ256rrk, X86::VCVTTPH2QQZ256rmk, TB_NO_REVERSE},
+  {X86::VCVTTPH2QQZrrk, X86::VCVTTPH2QQZrmk, 0},
+  {X86::VCVTTPH2UDQZ128rrk, X86::VCVTTPH2UDQZ128rmk, TB_NO_REVERSE},
+  {X86::VCVTTPH2UDQZ256rrk, X86::VCVTTPH2UDQZ256rmk, 0},
+  {X86::VCVTTPH2UDQZrrk, X86::VCVTTPH2UDQZrmk, 0},
+  {X86::VCVTTPH2UQQZ128rrk, X86::VCVTTPH2UQQZ128rmk, TB_NO_REVERSE},
+  {X86::VCVTTPH2UQQZ256rrk, X86::VCVTTPH2UQQZ256rmk, TB_NO_REVERSE},
+  {X86::VCVTTPH2UQQZrrk, X86::VCVTTPH2UQQZrmk, 0},
+  {X86::VCVTTPH2UWZ128rrk, X86::VCVTTPH2UWZ128rmk, 0},
+  {X86::VCVTTPH2UWZ256rrk, X86::VCVTTPH2UWZ256rmk, 0},
+  {X86::VCVTTPH2UWZrrk, X86::VCVTTPH2UWZrmk, 0},
+  {X86::VCVTTPH2WZ128rrk, X86::VCVTTPH2WZ128rmk, 0},
+  {X86::VCVTTPH2WZ256rrk, X86::VCVTTPH2WZ256rmk, 0},
+  {X86::VCVTTPH2WZrrk, X86::VCVTTPH2WZrmk, 0},
   {X86::VCVTTPS2DQZ128rrk, X86::VCVTTPS2DQZ128rmk, 0},
   {X86::VCVTTPS2DQZ256rrk, X86::VCVTTPS2DQZ256rmk, 0},
   {X86::VCVTTPS2DQZrrk, X86::VCVTTPS2DQZrmk, 0},
@@ -3214,15 +3449,27 @@ static const X86MemoryFoldTableEntry MemoryFoldTable3[] = {
   {X86::VCVTUDQ2PDZ128rrk, X86::VCVTUDQ2PDZ128rmk, TB_NO_REVERSE},
   {X86::VCVTUDQ2PDZ256rrk, X86::VCVTUDQ2PDZ256rmk, 0},
   {X86::VCVTUDQ2PDZrrk, X86::VCVTUDQ2PDZrmk, 0},
+  {X86::VCVTUDQ2PHZ128rrk, X86::VCVTUDQ2PHZ128rmk, 0},
+  {X86::VCVTUDQ2PHZ256rrk, X86::VCVTUDQ2PHZ256rmk, 0},
+  {X86::VCVTUDQ2PHZrrk, X86::VCVTUDQ2PHZrmk, 0},
   {X86::VCVTUDQ2PSZ128rrk, X86::VCVTUDQ2PSZ128rmk, 0},
   {X86::VCVTUDQ2PSZ256rrk, X86::VCVTUDQ2PSZ256rmk, 0},
   {X86::VCVTUDQ2PSZrrk, X86::VCVTUDQ2PSZrmk, 0},
   {X86::VCVTUQQ2PDZ128rrk, X86::VCVTUQQ2PDZ128rmk, 0},
   {X86::VCVTUQQ2PDZ256rrk, X86::VCVTUQQ2PDZ256rmk, 0},
   {X86::VCVTUQQ2PDZrrk, X86::VCVTUQQ2PDZrmk, 0},
+  {X86::VCVTUQQ2PHZ128rrk, X86::VCVTUQQ2PHZ128rmk, 0},
+  {X86::VCVTUQQ2PHZ256rrk, X86::VCVTUQQ2PHZ256rmk, 0},
+  {X86::VCVTUQQ2PHZrrk, X86::VCVTUQQ2PHZrmk, 0},
   {X86::VCVTUQQ2PSZ128rrk, X86::VCVTUQQ2PSZ128rmk, 0},
   {X86::VCVTUQQ2PSZ256rrk, X86::VCVTUQQ2PSZ256rmk, 0},
   {X86::VCVTUQQ2PSZrrk, X86::VCVTUQQ2PSZrmk, 0},
+  {X86::VCVTUW2PHZ128rrk, X86::VCVTUW2PHZ128rmk, 0},
+  {X86::VCVTUW2PHZ256rrk, X86::VCVTUW2PHZ256rmk, 0},
+  {X86::VCVTUW2PHZrrk, X86::VCVTUW2PHZrmk, 0},
+  {X86::VCVTW2PHZ128rrk, X86::VCVTW2PHZ128rmk, 0},
+  {X86::VCVTW2PHZ256rrk, X86::VCVTW2PHZ256rmk, 0},
+  {X86::VCVTW2PHZrrk, X86::VCVTW2PHZrmk, 0},
   {X86::VDBPSADBWZ128rrikz, X86::VDBPSADBWZ128rmikz, 0},
   {X86::VDBPSADBWZ256rrikz, X86::VDBPSADBWZ256rmikz, 0},
   {X86::VDBPSADBWZrrikz, X86::VDBPSADBWZrmikz, 0},
@@ -5729,3 +5976,4 @@ static const X86MemoryFoldTableEntry MemoryFoldTable4[] = {
   {X86::VXORPSZ256rrk, X86::VXORPSZ256rmk, 0},
   {X86::VXORPSZrrk, X86::VXORPSZrmk, 0},
 };
+
diff --git a/llvm/lib/Target/X86/X86OptimizeLEAs.cpp b/llvm/lib/Target/X86/X86OptimizeLEAs.cpp
index e0018a0ea58ba..3172896a8f609 100644
--- a/llvm/lib/Target/X86/X86OptimizeLEAs.cpp
+++ b/llvm/lib/Target/X86/X86OptimizeLEAs.cpp
@@ -321,8 +321,7 @@ int X86OptimizeLEAPass::calcInstrDist(const MachineInstr &First,
   // presented in InstrPos.
   assert(Last.getParent() == First.getParent() &&
          "Instructions are in different basic blocks");
-  assert(InstrPos.find(&First) != InstrPos.end() &&
-         InstrPos.find(&Last) != InstrPos.end() &&
+  assert(InstrPos.contains(&First) && InstrPos.contains(&Last) &&
          "Instructions' positions are undefined");
 
   return InstrPos[&Last] - InstrPos[&First];
diff --git a/llvm/lib/Target/X86/X86PfmCounters.td b/llvm/lib/Target/X86/X86PfmCounters.td
index d2460e12b0053..49ef6efc6aecf 100644
--- a/llvm/lib/Target/X86/X86PfmCounters.td
+++ b/llvm/lib/Target/X86/X86PfmCounters.td
@@ -290,4 +290,17 @@ def ZnVer3PfmCounters : ProcPfmCounters {
   ];
 }
 def : PfmCountersBinding<"znver3", ZnVer3PfmCounters>;
-def : PfmCountersBinding<"znver4", ZnVer3PfmCounters>;
+
+def ZnVer4PfmCounters : ProcPfmCounters {
+  let CycleCounter = PfmCounter<"cycles_not_in_halt">;
+  let UopsCounter = PfmCounter<"retired_ops">;
+  let IssueCounters = [
+    PfmIssueCounter<"Zn4Int", "ops_type_dispatched_from_decoder:int_disp_retire_mode">,
+    PfmIssueCounter<"Zn4FPU", "ops_type_dispatched_from_decoder:fp_disp_retire_mode">,
+    PfmIssueCounter<"Zn4Load", "ls_dispatch:ld_dispatch">,
+    PfmIssueCounter<"Zn4Store", "ls_dispatch:store_dispatch">,
+    PfmIssueCounter<"Zn4Divider", "div_op_count">,
+    PfmIssueCounter<"Zn4AGU", "ls_dispatch:ld_st_dispatch + ls_dispatch:ld_dispatch + ls_dispatch:store_dispatch">
+  ];
+}
+def : PfmCountersBinding<"znver4", ZnVer4PfmCounters>;
diff --git a/llvm/lib/Target/X86/X86SchedAlderlakeP.td b/llvm/lib/Target/X86/X86SchedAlderlakeP.td
index 3817eb0fc3261..2cf5c6fe9f46a 100644
--- a/llvm/lib/Target/X86/X86SchedAlderlakeP.td
+++ b/llvm/lib/Target/X86/X86SchedAlderlakeP.td
@@ -54,7 +54,6 @@ def ADLPPort00_01_05       : ProcResGroup<[ADLPPort00, ADLPPort01, ADLPPort05]>;
 def ADLPPort00_01_05_06    : ProcResGroup<[ADLPPort00, ADLPPort01, ADLPPort05, ADLPPort06]>;
 def ADLPPort00_05          : ProcResGroup<[ADLPPort00, ADLPPort05]>;
 def ADLPPort00_05_06       : ProcResGroup<[ADLPPort00, ADLPPort05, ADLPPort06]>;
-def ADLPPort00_05_06_10    : ProcResGroup<[ADLPPort00, ADLPPort05, ADLPPort06, ADLPPort10]>;
 def ADLPPort00_06          : ProcResGroup<[ADLPPort00, ADLPPort06]>;
 def ADLPPort01_05          : ProcResGroup<[ADLPPort01, ADLPPort05]>;
 def ADLPPort01_05_10       : ProcResGroup<[ADLPPort01, ADLPPort05, ADLPPort10]>;
@@ -124,66 +123,66 @@ multiclass ADLPWriteResPair<X86FoldableSchedWrite SchedRW,
 // Infered SchedWrite definition.
 def : WriteRes<WriteADC, [ADLPPort00_06]>;
 defm : X86WriteRes<WriteADCLd, [ADLPPort00_01_05_06_10, ADLPPort00_06], 11, [1, 1], 2>;
-defm : ADLPWriteResPair<WriteAESDecEnc, [ADLPPort00_01], 3, [1], 1, 8>;
-defm : ADLPWriteResPair<WriteAESIMC, [ADLPPort00_01], 6, [2], 2, 8>;
+defm : ADLPWriteResPair<WriteAESDecEnc, [ADLPPort00_01], 5, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteAESIMC, [ADLPPort00_01], 8, [2], 2, 7>;
 defm : X86WriteRes<WriteAESKeyGen, [ADLPPort00, ADLPPort00_01, ADLPPort00_01_05, ADLPPort00_06, ADLPPort01_05, ADLPPort05], 7, [4, 1, 1, 2, 3, 3], 14>;
 defm : X86WriteRes<WriteAESKeyGenLd, [ADLPPort00, ADLPPort00_01, ADLPPort00_06, ADLPPort01_05, ADLPPort02_03_11, ADLPPort05], 12, [4, 1, 2, 3, 1, 3], 14>;
 def : WriteRes<WriteALU, [ADLPPort00_01_05_06_10]>;
 def : WriteRes<WriteALULd, [ADLPPort00_01_05_06_10]> {
-  let Latency = 6;
+  let Latency = 11;
 }
-defm : ADLPWriteResPair<WriteBEXTR, [ADLPPort00_06, ADLPPort01], 2, [1, 1], 2>;
-defm : ADLPWriteResPair<WriteBLS, [ADLPPort01_05_10], 1, [1]>;
+defm : ADLPWriteResPair<WriteBEXTR, [ADLPPort00_06, ADLPPort01], 6, [1, 1], 2>;
+defm : ADLPWriteResPair<WriteBLS, [ADLPPort01_05_10], 2, [1]>;
 defm : ADLPWriteResPair<WriteBSF, [ADLPPort01], 3, [1]>;
 defm : ADLPWriteResPair<WriteBSR, [ADLPPort01], 3, [1]>;
-def : WriteRes<WriteBSWAP32, [ADLPPort00_01_05_06_10]>;
+def : WriteRes<WriteBSWAP32, [ADLPPort01]>;
 defm : X86WriteRes<WriteBSWAP64, [ADLPPort00_06, ADLPPort01], 2, [1, 1], 2>;
-defm : ADLPWriteResPair<WriteBZHI, [ADLPPort01], 1, [1]>;
+defm : ADLPWriteResPair<WriteBZHI, [ADLPPort01], 3, [1]>;
 def : WriteRes<WriteBitTest, [ADLPPort01]>;
 defm : X86WriteRes<WriteBitTestImmLd, [ADLPPort01, ADLPPort02_03_11], 6, [1, 1], 2>;
 defm : X86WriteRes<WriteBitTestRegLd, [ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort02_03_11], 11, [4, 2, 1, 2, 1], 10>;
 def : WriteRes<WriteBitTestSet, [ADLPPort01]>;
 def : WriteRes<WriteBitTestSetImmLd, [ADLPPort01]> {
-  let Latency = 6;
+  let Latency = 11;
 }
 defm : X86WriteRes<WriteBitTestSetRegLd, [ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10], 17, [3, 2, 1, 2], 8>;
-defm : ADLPWriteResPair<WriteBlend, [ADLPPort01_05], 1, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteBlendY, [ADLPPort00_01_05], 1, [1], 1, 7>;
-defm : ADLPWriteResPair<WriteCLMul, [ADLPPort05], 3, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteCMOV, [ADLPPort00_06], 2, [2], 1, 4>;
+defm : ADLPWriteResPair<WriteBlend, [ADLPPort01_05], 1, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteBlendY, [ADLPPort00_01_05], 1, [1], 1, 8>;
+defm : ADLPWriteResPair<WriteCLMul, [ADLPPort05], 3, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteCMOV, [ADLPPort00_06], 1, [1], 1, 6>;
 defm : X86WriteRes<WriteCMPXCHG, [ADLPPort00_01_05_06_10, ADLPPort00_06], 3, [3, 2], 5>;
 defm : X86WriteRes<WriteCMPXCHGRMW, [ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08], 12, [1, 2, 1, 1, 1], 6>;
 defm : ADLPWriteResPair<WriteCRC32, [ADLPPort01], 3, [1]>;
 defm : X86WriteRes<WriteCvtI2PD, [ADLPPort00_01, ADLPPort05], 5, [1, 1], 2>;
-defm : X86WriteRes<WriteCvtI2PDLd, [ADLPPort00_01, ADLPPort02_03_11], 10, [1, 1], 2>;
+defm : X86WriteRes<WriteCvtI2PDLd, [ADLPPort00_01, ADLPPort02_03_11], 11, [1, 1], 2>;
 defm : X86WriteRes<WriteCvtI2PDY, [ADLPPort00_01, ADLPPort05], 7, [1, 1], 2>;
 defm : X86WriteRes<WriteCvtI2PDYLd, [ADLPPort00_01, ADLPPort02_03_11], 12, [1, 1], 2>;
 defm : X86WriteResPairUnsupported<WriteCvtI2PDZ>;
-defm : ADLPWriteResPair<WriteCvtI2PS, [ADLPPort00_01], 4, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteCvtI2PSY, [ADLPPort00_01], 4, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteCvtI2PS, [ADLPPort00_01], 4, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteCvtI2PSY, [ADLPPort00_01], 4, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteCvtI2PSZ>;
 defm : X86WriteRes<WriteCvtI2SD, [ADLPPort00_01, ADLPPort05], 7, [1, 1], 2>;
 defm : X86WriteRes<WriteCvtI2SDLd, [ADLPPort00_01, ADLPPort02_03_11], 11, [1, 1], 2>;
 defm : X86WriteRes<WriteCvtI2SS, [ADLPPort00_01, ADLPPort05], 7, [1, 1], 2>;
 defm : X86WriteRes<WriteCvtI2SSLd, [ADLPPort00_01, ADLPPort02_03_11], 11, [1, 1], 2>;
-defm : ADLPWriteResPair<WriteCvtPD2I, [ADLPPort00_01, ADLPPort05], 5, [1, 1], 2, 6>;
-defm : ADLPWriteResPair<WriteCvtPD2IY, [ADLPPort00_01, ADLPPort05], 7, [1, 1], 2, 7>;
+defm : ADLPWriteResPair<WriteCvtPD2I, [ADLPPort00_01, ADLPPort05], 5, [1, 1], 2, 7>;
+defm : ADLPWriteResPair<WriteCvtPD2IY, [ADLPPort00_01, ADLPPort05], 7, [1, 1], 2, 8>;
 defm : X86WriteResPairUnsupported<WriteCvtPD2IZ>;
 defm : ADLPWriteResPair<WriteCvtPD2PS, [ADLPPort00_01, ADLPPort05], 5, [1, 1], 2, 7>;
-defm : ADLPWriteResPair<WriteCvtPD2PSY, [ADLPPort00_01, ADLPPort05], 7, [1, 1], 2, 7>;
+defm : ADLPWriteResPair<WriteCvtPD2PSY, [ADLPPort00_01, ADLPPort05], 7, [1, 1], 2, 8>;
 defm : X86WriteResPairUnsupported<WriteCvtPD2PSZ>;
 defm : X86WriteRes<WriteCvtPH2PS, [ADLPPort00_01, ADLPPort05], 6, [1, 1], 2>;
-defm : X86WriteRes<WriteCvtPH2PSLd, [ADLPPort00_01, ADLPPort02_03_11], 11, [1, 1], 2>;
+defm : X86WriteRes<WriteCvtPH2PSLd, [ADLPPort00_01, ADLPPort02_03_11], 12, [1, 1], 2>;
 defm : X86WriteRes<WriteCvtPH2PSY, [ADLPPort00_01, ADLPPort05], 8, [1, 1], 2>;
 defm : X86WriteRes<WriteCvtPH2PSYLd, [ADLPPort00_01, ADLPPort02_03_11], 12, [1, 1], 2>;
 defm : X86WriteResPairUnsupported<WriteCvtPH2PSZ>;
-defm : ADLPWriteResPair<WriteCvtPS2I, [ADLPPort00_01], 4, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteCvtPS2IY, [ADLPPort00_01], 4, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteCvtPS2I, [ADLPPort00_01], 4, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteCvtPS2IY, [ADLPPort00_01], 4, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteCvtPS2IZ>;
 defm : X86WriteRes<WriteCvtPS2PD, [ADLPPort00_01, ADLPPort05], 5, [1, 1], 2>;
-defm : X86WriteRes<WriteCvtPS2PDLd, [ADLPPort00_01, ADLPPort02_03_11], 10, [1, 1], 2>;
+defm : X86WriteRes<WriteCvtPS2PDLd, [ADLPPort00_01, ADLPPort02_03_11], 11, [1, 1], 2>;
 defm : X86WriteRes<WriteCvtPS2PDY, [ADLPPort00_01, ADLPPort05], 7, [1, 1], 2>;
-defm : X86WriteRes<WriteCvtPS2PDYLd, [ADLPPort00_01, ADLPPort02_03_11], 11, [1, 1], 2>;
+defm : X86WriteRes<WriteCvtPS2PDYLd, [ADLPPort00_01, ADLPPort02_03_11], 12, [1, 1], 2>;
 defm : X86WriteResPairUnsupported<WriteCvtPS2PDZ>;
 defm : X86WriteRes<WriteCvtPS2PH, [ADLPPort00_01, ADLPPort05], 6, [1, 1], 2>;
 defm : X86WriteRes<WriteCvtPS2PHSt, [ADLPPort00_01, ADLPPort04_09, ADLPPort07_08], 12, [1, 1, 1], 3>;
@@ -196,112 +195,105 @@ defm : ADLPWriteResPair<WriteCvtSD2SS, [ADLPPort00_01, ADLPPort05], 5, [1, 1], 2
 defm : ADLPWriteResPair<WriteCvtSS2I, [ADLPPort00, ADLPPort00_01], 7, [1, 1], 2>;
 defm : X86WriteRes<WriteCvtSS2SD, [ADLPPort00_01, ADLPPort05], 5, [1, 1], 2>;
 defm : X86WriteRes<WriteCvtSS2SDLd, [ADLPPort00_01, ADLPPort02_03_11], 11, [1, 1], 2>;
-defm : ADLPWriteResPair<WriteDPPD, [ADLPPort00_01, ADLPPort01_05], 9, [2, 1], 3, 6>;
+defm : ADLPWriteResPair<WriteDPPD, [ADLPPort00_01, ADLPPort01_05], 9, [2, 1], 3, 7>;
 defm : ADLPWriteResPair<WriteDPPS, [ADLPPort00_01, ADLPPort00_06, ADLPPort01_05, ADLPPort05], 14, [2, 1, 2, 1], 6, 7>;
 defm : ADLPWriteResPair<WriteDPPSY, [ADLPPort00_01, ADLPPort00_06, ADLPPort01_05, ADLPPort05], 14, [2, 1, 2, 1], 6, 8>;
 defm : ADLPWriteResPair<WriteDiv16, [ADLPPort00_01_05_06_10, ADLPPort01], 16, [1, 3], 4, 4>;
 defm : ADLPWriteResPair<WriteDiv32, [ADLPPort00_01_05_06_10, ADLPPort01], 15, [1, 3], 4, 4>;
-defm : X86WriteRes<WriteDiv64, [ADLPPort00_01_05_06_10, ADLPPort01], 18, [1, 3], 5>;
-defm : X86WriteRes<WriteDiv64Ld, [ADLPPort00_01_05_06_10, ADLPPort01, ADLPPort02_03_11], 23, [1, 3, 1], 5>;
-defm : ADLPWriteResPair<WriteDiv8, [ADLPPort00_01_05_06_10, ADLPPort01], 17, [1, 3], 4>;
+defm : ADLPWriteResPair<WriteDiv64, [ADLPPort01], 18, [3], 3>;
+defm : X86WriteRes<WriteDiv8, [ADLPPort01], 17, [3], 3>;
+defm : X86WriteRes<WriteDiv8Ld, [ADLPPort01], 22, [3], 3>;
 defm : X86WriteRes<WriteEMMS, [ADLPPort00, ADLPPort00_05, ADLPPort00_06], 10, [1, 8, 1], 10>;
 def : WriteRes<WriteFAdd, [ADLPPort05]> {
   let Latency = 3;
 }
-defm : X86WriteRes<WriteFAddLd, [ADLPPort02_03, ADLPPort05], 10, [1, 1], 2>;
+defm : X86WriteRes<WriteFAddLd, [ADLPPort01_05, ADLPPort02_03_11], 10, [1, 1], 2>;
 defm : ADLPWriteResPair<WriteFAdd64, [ADLPPort01_05], 3, [1], 1, 7>;
-defm : ADLPWriteResPair<WriteFAdd64X, [ADLPPort01_05], 2, [1], 1, 7>;
-defm : ADLPWriteResPair<WriteFAdd64Y, [ADLPPort01_05], 2, [1], 1, 8>;
+defm : ADLPWriteResPair<WriteFAdd64X, [ADLPPort01_05], 3, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFAdd64Y, [ADLPPort01_05], 3, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteFAdd64Z>;
-defm : ADLPWriteResPair<WriteFAddX, [ADLPPort01_05], 2, [1], 1, 7>;
-defm : ADLPWriteResPair<WriteFAddY, [ADLPPort01_05], 2, [1], 1, 8>;
+defm : ADLPWriteResPair<WriteFAddX, [ADLPPort01_05], 3, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFAddY, [ADLPPort01_05], 3, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteFAddZ>;
-defm : ADLPWriteResPair<WriteFBlend, [ADLPPort00_01_05], 1, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteFBlendY, [ADLPPort00_01_05], 1, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFBlend, [ADLPPort00_01_05], 1, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFBlendY, [ADLPPort00_01_05], 1, [1], 1, 8>;
 def : WriteRes<WriteFCMOV, [ADLPPort01]> {
   let Latency = 3;
 }
 defm : ADLPWriteResPair<WriteFCmp, [ADLPPort00_01], 4, [1], 1, 7>;
 defm : ADLPWriteResPair<WriteFCmp64, [ADLPPort00_01], 4, [1], 1, 7>;
-defm : ADLPWriteResPair<WriteFCmp64X, [ADLPPort00_01], 4, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteFCmp64Y, [ADLPPort00_01], 4, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFCmp64X, [ADLPPort00_01], 4, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFCmp64Y, [ADLPPort00_01], 4, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteFCmp64Z>;
-defm : ADLPWriteResPair<WriteFCmpX, [ADLPPort00_01], 4, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteFCmpY, [ADLPPort00_01], 4, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFCmpX, [ADLPPort00_01], 4, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFCmpY, [ADLPPort00_01], 4, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteFCmpZ>;
 def : WriteRes<WriteFCom, [ADLPPort05]>;
 defm : X86WriteRes<WriteFComLd, [ADLPPort02_03, ADLPPort05], 8, [1, 1], 2>;
-defm : ADLPWriteResPair<WriteFComX, [ADLPPort00], 3, [1], 1, 6>;
-def : WriteRes<WriteFDiv, [ADLPPort00]> {
-  let Latency = 11;
-}
-defm : X86WriteRes<WriteFDivLd, [ADLPPort00, ADLPPort02_03], 27, [1, 1], 2>;
+defm : ADLPWriteResPair<WriteFComX, [ADLPPort00], 3, [1]>;
+defm : ADLPWriteResPair<WriteFDiv, [ADLPPort00], 11, [1], 1, 7>;
 defm : ADLPWriteResPair<WriteFDiv64, [ADLPPort00], 14, [1]>;
 defm : ADLPWriteResPair<WriteFDiv64X, [ADLPPort00], 14, [1], 1, 6>;
 defm : ADLPWriteResPair<WriteFDiv64Y, [ADLPPort00], 14, [1], 1, 7>;
 defm : X86WriteResPairUnsupported<WriteFDiv64Z>;
-defm : ADLPWriteResPair<WriteFDivX, [ADLPPort00], 11, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteFDivY, [ADLPPort00], 11, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFDivX, [ADLPPort00], 11, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFDivY, [ADLPPort00], 11, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteFDivZ>;
-defm : ADLPWriteResPair<WriteFHAdd, [ADLPPort01_05, ADLPPort05], 5, [1, 2], 3, 6>;
-defm : ADLPWriteResPair<WriteFHAddY, [ADLPPort01_05, ADLPPort05], 5, [1, 2], 3, 7>;
+defm : ADLPWriteResPair<WriteFHAdd, [ADLPPort01_05, ADLPPort05], 6, [1, 2], 3, 6>;
+defm : ADLPWriteResPair<WriteFHAddY, [ADLPPort01_05, ADLPPort05], 5, [1, 2], 3, 8>;
 def : WriteRes<WriteFLD0, [ADLPPort00_05]>;
 defm : X86WriteRes<WriteFLD1, [ADLPPort00_05], 1, [2], 2>;
 defm : X86WriteRes<WriteFLDC, [ADLPPort00_05], 1, [2], 2>;
 def : WriteRes<WriteFLoad, [ADLPPort02_03_11]> {
-  let Latency = 6;
+  let Latency = 7;
 }
 def : WriteRes<WriteFLoadX, [ADLPPort02_03_11]> {
-  let Latency = 6;
+  let Latency = 7;
 }
 def : WriteRes<WriteFLoadY, [ADLPPort02_03_11]> {
-  let Latency = 7;
+  let Latency = 8;
 }
-defm : ADLPWriteResPair<WriteFLogic, [ADLPPort00_01_05], 1, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteFLogicY, [ADLPPort00_01_05], 1, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFLogic, [ADLPPort00_01_05], 1, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFLogicY, [ADLPPort00_01_05], 1, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteFLogicZ>;
 defm : ADLPWriteResPair<WriteFMA, [ADLPPort00_01], 4, [1], 1, 7>;
-defm : ADLPWriteResPair<WriteFMAX, [ADLPPort00_01], 4, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteFMAY, [ADLPPort00_01], 4, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFMAX, [ADLPPort00_01], 4, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFMAY, [ADLPPort00_01], 4, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteFMAZ>;
 def : WriteRes<WriteFMOVMSK, [ADLPPort00]> {
   let Latency = 3;
 }
-defm : X86WriteRes<WriteFMaskedLoad, [ADLPPort00_01_05, ADLPPort02_03_11], 7, [1, 1], 2>;
-defm : X86WriteRes<WriteFMaskedLoadY, [ADLPPort00_01_05, ADLPPort02_03_11], 8, [1, 1], 2>;
-defm : X86WriteRes<WriteFMaskedStore32, [ADLPPort00, ADLPPort04_09, ADLPPort07_08], 3, [1, 1, 1], 3>;
-defm : X86WriteRes<WriteFMaskedStore32Y, [ADLPPort00, ADLPPort04_09, ADLPPort07_08], 3, [1, 1, 1], 3>;
-defm : X86WriteRes<WriteFMaskedStore64, [ADLPPort00, ADLPPort04_09, ADLPPort07_08], 3, [1, 1, 1], 3>;
-defm : X86WriteRes<WriteFMaskedStore64Y, [ADLPPort00, ADLPPort04_09, ADLPPort07_08], 3, [1, 1, 1], 3>;
-def : WriteRes<WriteFMoveX, [ADLPPort00_01_05]> {
-  let Latency = 0;
-}
-def : WriteRes<WriteFMoveY, [ADLPPort00_01_05]> {
-  let Latency = 0;
-}
+defm : X86WriteRes<WriteFMaskedLoad, [ADLPPort00_01_05, ADLPPort02_03_11], 8, [1, 1], 2>;
+defm : X86WriteRes<WriteFMaskedLoadY, [ADLPPort00_01_05, ADLPPort02_03_11], 9, [1, 1], 2>;
+defm : X86WriteRes<WriteFMaskedStore32, [ADLPPort00, ADLPPort04_09, ADLPPort07_08], 14, [1, 1, 1], 3>;
+defm : X86WriteRes<WriteFMaskedStore32Y, [ADLPPort00, ADLPPort04_09, ADLPPort07_08], 14, [1, 1, 1], 3>;
+defm : X86WriteRes<WriteFMaskedStore64, [ADLPPort00, ADLPPort04_09, ADLPPort07_08], 14, [1, 1, 1], 3>;
+defm : X86WriteRes<WriteFMaskedStore64Y, [ADLPPort00, ADLPPort04_09, ADLPPort07_08], 14, [1, 1, 1], 3>;
+defm : X86WriteRes<WriteFMoveX, [], 1, [], 0>;
+defm : X86WriteRes<WriteFMoveY, [], 1, [], 0>;
 defm : X86WriteResUnsupported<WriteFMoveZ>;
 defm : ADLPWriteResPair<WriteFMul, [ADLPPort00_01], 4, [1], 1, 7>;
 defm : ADLPWriteResPair<WriteFMul64, [ADLPPort00_01], 4, [1], 1, 7>;
-defm : ADLPWriteResPair<WriteFMul64X, [ADLPPort00_01], 4, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteFMul64Y, [ADLPPort00_01], 4, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFMul64X, [ADLPPort00_01], 4, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFMul64Y, [ADLPPort00_01], 4, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteFMul64Z>;
-defm : ADLPWriteResPair<WriteFMulX, [ADLPPort00_01], 4, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteFMulY, [ADLPPort00_01], 4, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFMulX, [ADLPPort00_01], 4, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFMulY, [ADLPPort00_01], 4, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteFMulZ>;
 defm : ADLPWriteResPair<WriteFRcp, [ADLPPort00], 4, [1], 1, 7>;
-defm : ADLPWriteResPair<WriteFRcpX, [ADLPPort00], 4, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteFRcpY, [ADLPPort00], 4, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFRcpX, [ADLPPort00], 4, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFRcpY, [ADLPPort00], 4, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteFRcpZ>;
-defm : ADLPWriteResPair<WriteFRnd, [ADLPPort00_01], 8, [2], 2, 6>;
-defm : ADLPWriteResPair<WriteFRndY, [ADLPPort00_01], 8, [2], 2, 7>;
+defm : ADLPWriteResPair<WriteFRnd, [ADLPPort00_01], 8, [2], 2, 7>;
+defm : ADLPWriteResPair<WriteFRndY, [ADLPPort00_01], 8, [2], 2, 8>;
 defm : X86WriteResPairUnsupported<WriteFRndZ>;
 defm : ADLPWriteResPair<WriteFRsqrt, [ADLPPort00], 4, [1], 1, 7>;
-defm : ADLPWriteResPair<WriteFRsqrtX, [ADLPPort00], 4, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteFRsqrtY, [ADLPPort00], 4, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFRsqrtX, [ADLPPort00], 4, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFRsqrtY, [ADLPPort00], 4, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteFRsqrtZ>;
-defm : ADLPWriteResPair<WriteFShuffle, [ADLPPort05], 1, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteFShuffle256, [ADLPPort05], 3, [1], 1, 7>;
-defm : ADLPWriteResPair<WriteFShuffleY, [ADLPPort05], 1, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFShuffle, [ADLPPort05], 1, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFShuffle256, [ADLPPort05], 3, [1], 1, 8>;
+defm : ADLPWriteResPair<WriteFShuffleY, [ADLPPort05], 1, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteFShuffleZ>;
 def : WriteRes<WriteFSign, [ADLPPort00]>;
 defm : ADLPWriteResPair<WriteFSqrt, [ADLPPort00], 12, [1], 1, 7>;
@@ -313,32 +305,32 @@ def : WriteRes<WriteFSqrt80, [ADLPPortInvalid, ADLPPort00]> {
   let ResourceCycles = [7, 1];
   let Latency = 21;
 }
-defm : ADLPWriteResPair<WriteFSqrtX, [ADLPPort00], 12, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteFSqrtY, [ADLPPort00], 12, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFSqrtX, [ADLPPort00], 12, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFSqrtY, [ADLPPort00], 12, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteFSqrtZ>;
-defm : X86WriteRes<WriteFStore, [ADLPPort04_09, ADLPPort07_08], 1, [1, 1], 2>;
+defm : X86WriteRes<WriteFStore, [ADLPPort04_09, ADLPPort07_08], 12, [1, 1], 2>;
 defm : X86WriteResUnsupported<WriteFStoreNT>;
 defm : X86WriteRes<WriteFStoreNTX, [ADLPPort04_09, ADLPPort07_08], 518, [1, 1], 2>;
 defm : X86WriteRes<WriteFStoreNTY, [ADLPPort04_09, ADLPPort07_08], 542, [1, 1], 2>;
-defm : X86WriteRes<WriteFStoreX, [ADLPPort04_09, ADLPPort07_08], 1, [1, 1], 2>;
-defm : X86WriteRes<WriteFStoreY, [ADLPPort04_09, ADLPPort07_08], 1, [1, 1], 2>;
+defm : X86WriteRes<WriteFStoreX, [ADLPPort04_09, ADLPPort07_08], 12, [1, 1], 2>;
+defm : X86WriteRes<WriteFStoreY, [ADLPPort04_09, ADLPPort07_08], 12, [1, 1], 2>;
 defm : ADLPWriteResPair<WriteFTest, [ADLPPort00], 3, [1]>;
 defm : ADLPWriteResPair<WriteFTestY, [ADLPPort00], 5, [1], 1, 6>;
 defm : ADLPWriteResPair<WriteFVarBlend, [ADLPPort00_01_05], 1, [1], 1, 7>;
-defm : ADLPWriteResPair<WriteFVarBlendY, [ADLPPort00_01_05], 3, [3], 3, 6>;
+defm : ADLPWriteResPair<WriteFVarBlendY, [ADLPPort00_01_05], 3, [3], 3, 7>;
 defm : X86WriteResPairUnsupported<WriteFVarBlendZ>;
-defm : ADLPWriteResPair<WriteFVarShuffle, [ADLPPort05], 1, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteFVarShuffle256, [ADLPPort05], 3, [1], 1, 7>;
-defm : ADLPWriteResPair<WriteFVarShuffleY, [ADLPPort05], 1, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFVarShuffle, [ADLPPort05], 1, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteFVarShuffle256, [ADLPPort05], 3, [1], 1, 8>;
+defm : ADLPWriteResPair<WriteFVarShuffleY, [ADLPPort05], 1, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteFVarShuffleZ>;
 def : WriteRes<WriteFence, [ADLPPort00_06]> {
   let Latency = 2;
 }
 defm : ADLPWriteResPair<WriteIDiv16, [ADLPPort00_01_05_06_10, ADLPPort01], 16, [1, 3], 4, 4>;
 defm : ADLPWriteResPair<WriteIDiv32, [ADLPPort00_01_05_06_10, ADLPPort01], 15, [1, 3], 4, 4>;
-defm : X86WriteRes<WriteIDiv64, [ADLPPort00_01_05_06_10, ADLPPort01], 18, [1, 3], 5>;
-defm : X86WriteRes<WriteIDiv64Ld, [ADLPPort00_01_05_06_10, ADLPPort01, ADLPPort02_03_11], 23, [1, 3, 1], 5>;
-defm : ADLPWriteResPair<WriteIDiv8, [ADLPPort00_01_05_06_10, ADLPPort01], 17, [1, 3], 4>;
+defm : ADLPWriteResPair<WriteIDiv64, [ADLPPort01], 18, [3], 3>;
+defm : X86WriteRes<WriteIDiv8, [ADLPPort01], 17, [3], 3>;
+defm : X86WriteRes<WriteIDiv8Ld, [ADLPPort01], 22, [3], 3>;
 defm : ADLPWriteResPair<WriteIMul16, [ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01], 5, [2, 1, 1], 4>;
 defm : ADLPWriteResPair<WriteIMul16Imm, [ADLPPort00_01_05_06_10, ADLPPort01], 4, [1, 1], 2>;
 defm : ADLPWriteResPair<WriteIMul16Reg, [ADLPPort01], 3, [1]>;
@@ -360,8 +352,8 @@ defm : X86WriteRes<WriteJumpLd, [ADLPPort00_06, ADLPPort02_03], 6, [1, 1], 2>;
 def : WriteRes<WriteLAHFSAHF, [ADLPPort00_06]> {
   let Latency = 3;
 }
-defm : X86WriteRes<WriteLDMXCSR, [ADLPPort00, ADLPPort00_01_05, ADLPPort02_03_11, ADLPPort06], 7, [1, 1, 1, 1], 4>;
-def : WriteRes<WriteLEA, [ADLPPort00_01_05_06_10]>;
+defm : X86WriteRes<WriteLDMXCSR, [ADLPPort00, ADLPPort00_01_05, ADLPPort00_06, ADLPPort02_03_11], 7, [1, 1, 1, 1], 4>;
+def : WriteRes<WriteLEA, [ADLPPort01]>;
 defm : ADLPWriteResPair<WriteLZCNT, [ADLPPort01], 3, [1]>;
 def : WriteRes<WriteLoad, [ADLPPort02_03_11]> {
   let Latency = 5;
@@ -369,8 +361,8 @@ def : WriteRes<WriteLoad, [ADLPPort02_03_11]> {
 def : WriteRes<WriteMMXMOVMSK, [ADLPPort00]> {
   let Latency = 3;
 }
-defm : ADLPWriteResPair<WriteMPSAD, [ADLPPort01_05, ADLPPort05], 4, [1, 1], 2, 6>;
-defm : ADLPWriteResPair<WriteMPSADY, [ADLPPort01_05, ADLPPort05], 4, [1, 1], 2, 7>;
+defm : ADLPWriteResPair<WriteMPSAD, [ADLPPort01_05, ADLPPort05], 4, [1, 1], 2, 7>;
+defm : ADLPWriteResPair<WriteMPSADY, [ADLPPort01_05, ADLPPort05], 4, [1, 1], 2, 8>;
 defm : ADLPWriteResPair<WriteMULX32, [ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01], 4, [1, 1, 1], 2>;
 defm : ADLPWriteResPair<WriteMULX64, [ADLPPort01, ADLPPort05], 4, [1, 1]>;
 def : WriteRes<WriteMicrocoded, [ADLPPort00_01_05_06]> {
@@ -384,76 +376,75 @@ defm : X86WriteRes<WritePCmpEStrM, [ADLPPort00, ADLPPort00_01_05, ADLPPort00_06,
 defm : X86WriteRes<WritePCmpEStrMLd, [ADLPPort00, ADLPPort00_01_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort05], 17, [3, 2, 1, 1, 1, 1], 9>;
 defm : ADLPWriteResPair<WritePCmpIStrI, [ADLPPort00], 11, [3], 3, 20>;
 defm : ADLPWriteResPair<WritePCmpIStrM, [ADLPPort00], 11, [3], 3>;
-defm : ADLPWriteResPair<WritePHAdd, [ADLPPort00_05, ADLPPort05], 3, [1, 2], 3, 6>;
-defm : ADLPWriteResPair<WritePHAddX, [ADLPPort00_01_05, ADLPPort01_05], 2, [1, 2], 3, 6>;
-defm : ADLPWriteResPair<WritePHAddY, [ADLPPort00_01_05, ADLPPort01_05], 2, [1, 2], 3, 7>;
+defm : ADLPWriteResPair<WritePHAdd, [ADLPPort00_05, ADLPPort05], 3, [1, 2], 3, 8>;
+defm : ADLPWriteResPair<WritePHAddX, [ADLPPort00_01_05, ADLPPort01_05], 2, [1, 2], 3, 7>;
+defm : ADLPWriteResPair<WritePHAddY, [ADLPPort00_01_05, ADLPPort01_05], 2, [1, 2], 3, 8>;
 defm : ADLPWriteResPair<WritePHMINPOS, [ADLPPort00], 4, [1], 1, 7>;
-defm : ADLPWriteResPair<WritePMULLD, [ADLPPort00_01], 10, [2], 2, 6>;
-defm : ADLPWriteResPair<WritePMULLDY, [ADLPPort00_01], 10, [2], 2, 7>;
+defm : ADLPWriteResPair<WritePMULLD, [ADLPPort00_01], 10, [2], 2, 8>;
+defm : ADLPWriteResPair<WritePMULLDY, [ADLPPort00_01], 10, [2], 2, 8>;
 defm : X86WriteResPairUnsupported<WritePMULLDZ>;
 defm : ADLPWriteResPair<WritePOPCNT, [ADLPPort01], 3, [1]>;
-defm : ADLPWriteResPair<WritePSADBW, [ADLPPort05], 3, [1], 1, 6>;
-defm : ADLPWriteResPair<WritePSADBWX, [ADLPPort05], 3, [1], 1, 6>;
-defm : ADLPWriteResPair<WritePSADBWY, [ADLPPort05], 3, [1], 1, 7>;
+defm : ADLPWriteResPair<WritePSADBW, [ADLPPort05], 3, [1], 1, 8>;
+defm : ADLPWriteResPair<WritePSADBWX, [ADLPPort05], 3, [1], 1, 7>;
+defm : ADLPWriteResPair<WritePSADBWY, [ADLPPort05], 3, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WritePSADBWZ>;
 defm : X86WriteRes<WriteRMW, [ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08], 1, [1, 1, 1], 3>;
 defm : X86WriteRes<WriteRotate, [ADLPPort00_01_05_06_10, ADLPPort00_06], 2, [1, 2], 3>;
 defm : X86WriteRes<WriteRotateLd, [ADLPPort00_01_05_06_10, ADLPPort00_06], 12, [1, 2], 3>;
-defm : X86WriteRes<WriteRotateCL, [ADLPPort00_06], 1, [2], 2>;
-defm : X86WriteRes<WriteRotateCLLd, [ADLPPort00_06], 7, [2], 2>;
-def : WriteRes<WriteSETCC, [ADLPPort00_06]>;
-defm : X86WriteRes<WriteSETCCStore, [ADLPPort00_06, ADLPPort04_09, ADLPPort07_08], 2, [1, 1, 1], 3>;
+defm : X86WriteRes<WriteRotateCL, [ADLPPort00_06], 2, [2], 2>;
+defm : X86WriteRes<WriteRotateCLLd, [ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01], 19, [2, 3, 2], 7>;
+defm : X86WriteRes<WriteSETCC, [ADLPPort00_06], 2, [2], 2>;
+defm : X86WriteRes<WriteSETCCStore, [ADLPPort00_06, ADLPPort04_09, ADLPPort07_08], 13, [2, 1, 1], 4>;
 defm : X86WriteRes<WriteSHDmrcl, [ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08], 12, [1, 1, 1, 1, 1, 1], 6>;
 defm : X86WriteRes<WriteSHDmri, [ADLPPort00_01_05_06_10, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08], 12, [1, 1, 1, 1, 1], 5>;
-defm : X86WriteRes<WriteSHDrrcl, [ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01], 4, [1, 1, 1], 3>;
+defm : X86WriteRes<WriteSHDrrcl, [ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01], 5, [1, 1, 1], 3>;
 def : WriteRes<WriteSHDrri, [ADLPPort01]> {
   let Latency = 3;
 }
-defm : X86WriteRes<WriteSTMXCSR, [ADLPPort00, ADLPPort04_09, ADLPPort06, ADLPPort07_08], 12, [1, 1, 1, 1], 4>;
+defm : X86WriteRes<WriteSTMXCSR, [ADLPPort00, ADLPPort00_06, ADLPPort04_09, ADLPPort07_08], 12, [1, 1, 1, 1], 4>;
 def : WriteRes<WriteShift, [ADLPPort00_06]>;
 def : WriteRes<WriteShiftLd, [ADLPPort00_06]> {
-  let Latency = 7;
+  let Latency = 12;
 }
-defm : X86WriteRes<WriteShiftCL, [ADLPPort00_06], 1, [2], 2>;
-defm : X86WriteRes<WriteShiftCLLd, [ADLPPort00_06], 7, [2], 2>;
-defm : ADLPWriteResPair<WriteShuffle, [ADLPPort05], 1, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteShuffle256, [ADLPPort05], 3, [1], 1, 7>;
-defm : ADLPWriteResPair<WriteShuffleX, [ADLPPort01_05], 1, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteShuffleY, [ADLPPort01_05], 1, [1], 1, 7>;
+defm : X86WriteRes<WriteShiftCL, [ADLPPort00_06], 2, [2], 2>;
+defm : X86WriteRes<WriteShiftCLLd, [ADLPPort00_06], 12, [2], 2>;
+defm : ADLPWriteResPair<WriteShuffle, [ADLPPort05], 1, [1], 1, 8>;
+defm : ADLPWriteResPair<WriteShuffle256, [ADLPPort05], 3, [1], 1, 8>;
+defm : ADLPWriteResPair<WriteShuffleX, [ADLPPort01_05], 1, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteShuffleY, [ADLPPort01_05], 1, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteShuffleZ>;
-defm : X86WriteRes<WriteStore, [ADLPPort02_03_07, ADLPPort04, ADLPPort05], 4, [1, 1, 1], 3>;
+defm : X86WriteRes<WriteStore, [ADLPPort04_09, ADLPPort07_08], 12, [1, 1], 2>;
 defm : X86WriteRes<WriteStoreNT, [ADLPPort04_09, ADLPPort07_08], 512, [1, 1], 2>;
 def : WriteRes<WriteSystem, [ADLPPort00_01_05_06]> {
   let Latency = AlderlakePModel.MaxLatency;
 }
 defm : ADLPWriteResPair<WriteTZCNT, [ADLPPort01], 3, [1]>;
-defm : ADLPWriteResPair<WriteVPMOV256, [ADLPPort05], 3, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteVPMOV256, [ADLPPort05], 3, [1], 1, 8>;
 defm : ADLPWriteResPair<WriteVarBlend, [ADLPPort00_01_05], 1, [1], 1, 7>;
-defm : ADLPWriteResPair<WriteVarBlendY, [ADLPPort00_01_05], 3, [3], 3, 6>;
+defm : ADLPWriteResPair<WriteVarBlendY, [ADLPPort00_01_05], 3, [3], 3, 7>;
 defm : X86WriteResPairUnsupported<WriteVarBlendZ>;
-defm : X86WriteRes<WriteVarShuffle, [ADLPPort00, ADLPPort05], 3, [1, 1], 3>;
-defm : X86WriteRes<WriteVarShuffleLd, [ADLPPort00, ADLPPort02_03_11, ADLPPort05], 9, [1, 1, 1], 3>;
-defm : ADLPWriteResPair<WriteVarShuffle256, [ADLPPort05], 3, [1], 1, 7>;
-defm : ADLPWriteResPair<WriteVarShuffleX, [ADLPPort01_05], 1, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteVarShuffleY, [ADLPPort01_05], 1, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteVarShuffle, [ADLPPort00, ADLPPort05], 3, [1, 1], 2, 8>;
+defm : ADLPWriteResPair<WriteVarShuffle256, [ADLPPort05], 3, [1], 1, 8>;
+defm : ADLPWriteResPair<WriteVarShuffleX, [ADLPPort01_05], 1, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteVarShuffleY, [ADLPPort01_05], 1, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteVarShuffleZ>;
-defm : ADLPWriteResPair<WriteVarVecShift, [ADLPPort00_01], 1, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteVarVecShiftY, [ADLPPort00_01], 1, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteVarVecShift, [ADLPPort00_01], 1, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteVarVecShiftY, [ADLPPort00_01], 1, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteVarVecShiftZ>;
-defm : ADLPWriteResPair<WriteVecALU, [ADLPPort00], 1, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteVecALUX, [ADLPPort00_01], 1, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteVecALUY, [ADLPPort00_01], 1, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteVecALU, [ADLPPort00], 1, [1], 1, 8>;
+defm : ADLPWriteResPair<WriteVecALUX, [ADLPPort00_01], 1, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteVecALUY, [ADLPPort00_01], 1, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteVecALUZ>;
 defm : X86WriteRes<WriteVecExtract, [ADLPPort00, ADLPPort01_05], 4, [1, 1], 2>;
-defm : X86WriteRes<WriteVecExtractSt, [ADLPPort01_05, ADLPPort04_09, ADLPPort07_08], 2, [1, 1, 1], 3>;
+defm : X86WriteRes<WriteVecExtractSt, [ADLPPort01_05, ADLPPort04_09, ADLPPort07_08], 19, [1, 1, 1], 3>;
 defm : ADLPWriteResPair<WriteVecIMul, [ADLPPort00], 5, [1], 1, 8>;
-defm : ADLPWriteResPair<WriteVecIMulX, [ADLPPort00_01], 5, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteVecIMulY, [ADLPPort00_01], 5, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteVecIMulX, [ADLPPort00_01], 5, [1], 1, 8>;
+defm : ADLPWriteResPair<WriteVecIMulY, [ADLPPort00_01], 5, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteVecIMulZ>;
-defm : X86WriteRes<WriteVecInsert, [ADLPPort01_05, ADLPPort05], 2, [1, 1], 2>;
-defm : X86WriteRes<WriteVecInsertLd, [ADLPPort01_05, ADLPPort02_03_11], 7, [1, 1], 2>;
+defm : X86WriteRes<WriteVecInsert, [ADLPPort01_05, ADLPPort05], 4, [1, 1], 2>;
+defm : X86WriteRes<WriteVecInsertLd, [ADLPPort01_05, ADLPPort02_03_11], 8, [1, 1], 2>;
 def : WriteRes<WriteVecLoad, [ADLPPort02_03_11]> {
-  let Latency = 6;
+  let Latency = 7;
 }
 def : WriteRes<WriteVecLoadNT, [ADLPPort02_03_11]> {
   let Latency = 7;
@@ -462,14 +453,14 @@ def : WriteRes<WriteVecLoadNTY, [ADLPPort02_03_11]> {
   let Latency = 8;
 }
 def : WriteRes<WriteVecLoadX, [ADLPPort02_03_11]> {
-  let Latency = 6;
+  let Latency = 7;
 }
 def : WriteRes<WriteVecLoadY, [ADLPPort02_03_11]> {
-  let Latency = 7;
+  let Latency = 8;
 }
-defm : ADLPWriteResPair<WriteVecLogic, [ADLPPort00_05], 1, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteVecLogicX, [ADLPPort00_01_05], 1, [1], 1, 6>;
-defm : ADLPWriteResPair<WriteVecLogicY, [ADLPPort00_01_05], 1, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteVecLogic, [ADLPPort00_05], 1, [1], 1, 8>;
+defm : ADLPWriteResPair<WriteVecLogicX, [ADLPPort00_01_05], 1, [1], 1, 7>;
+defm : ADLPWriteResPair<WriteVecLogicY, [ADLPPort00_01_05], 1, [1], 1, 8>;
 defm : X86WriteResPairUnsupported<WriteVecLogicZ>;
 def : WriteRes<WriteVecMOVMSK, [ADLPPort00]> {
   let Latency = 3;
@@ -478,25 +469,23 @@ def : WriteRes<WriteVecMOVMSKY, [ADLPPort00]> {
   let Latency = 4;
 }
 defm : X86WriteRes<WriteVecMaskedGatherWriteback, [], 5, [], 0>;
-defm : X86WriteRes<WriteVecMaskedLoad, [ADLPPort00_01_05, ADLPPort02_03_11], 7, [1, 1], 2>;
-defm : X86WriteRes<WriteVecMaskedLoadY, [ADLPPort00_01_05, ADLPPort02_03_11], 8, [1, 1], 2>;
-defm : X86WriteRes<WriteVecMaskedStore32, [ADLPPort00, ADLPPort04_09, ADLPPort07_08], 3, [1, 1, 1], 3>;
-defm : X86WriteRes<WriteVecMaskedStore32Y, [ADLPPort00, ADLPPort04_09, ADLPPort07_08], 3, [1, 1, 1], 3>;
-defm : X86WriteRes<WriteVecMaskedStore64, [ADLPPort00, ADLPPort04_09, ADLPPort07_08], 3, [1, 1, 1], 3>;
-defm : X86WriteRes<WriteVecMaskedStore64Y, [ADLPPort00, ADLPPort04_09, ADLPPort07_08], 3, [1, 1, 1], 3>;
+defm : X86WriteRes<WriteVecMaskedLoad, [ADLPPort00_01_05, ADLPPort02_03_11], 8, [1, 1], 2>;
+defm : X86WriteRes<WriteVecMaskedLoadY, [ADLPPort00_01_05, ADLPPort02_03_11], 9, [1, 1], 2>;
+defm : X86WriteRes<WriteVecMaskedStore32, [ADLPPort00, ADLPPort04_09, ADLPPort07_08], 14, [1, 1, 1], 3>;
+defm : X86WriteRes<WriteVecMaskedStore32Y, [ADLPPort00, ADLPPort04_09, ADLPPort07_08], 14, [1, 1, 1], 3>;
+defm : X86WriteRes<WriteVecMaskedStore64, [ADLPPort00, ADLPPort04_09, ADLPPort07_08], 14, [1, 1, 1], 3>;
+defm : X86WriteRes<WriteVecMaskedStore64Y, [ADLPPort00, ADLPPort04_09, ADLPPort07_08], 14, [1, 1, 1], 3>;
 def : WriteRes<WriteVecMove, [ADLPPort00_05]>;
-def : WriteRes<WriteVecMoveFromGpr, [ADLPPort05]>;
-def : WriteRes<WriteVecMoveToGpr, [ADLPPort00]> {
+def : WriteRes<WriteVecMoveFromGpr, [ADLPPort05]> {
   let Latency = 3;
 }
-def : WriteRes<WriteVecMoveX, [ADLPPort00_01_05]> {
-  let Latency = 0;
-}
-def : WriteRes<WriteVecMoveY, [ADLPPort00_01_05]> {
-  let Latency = 0;
+def : WriteRes<WriteVecMoveToGpr, [ADLPPort00]> {
+  let Latency = 3;
 }
+defm : X86WriteRes<WriteVecMoveX, [], 1, [], 0>;
+defm : X86WriteRes<WriteVecMoveY, [], 1, [], 0>;
 defm : X86WriteResUnsupported<WriteVecMoveZ>;
-defm : ADLPWriteResPair<WriteVecShift, [ADLPPort00], 1, [1], 1, 6>;
+defm : ADLPWriteResPair<WriteVecShift, [ADLPPort00], 1, [1], 1, 8>;
 def : WriteRes<WriteVecShiftImm, [ADLPPort00]>;
 def : WriteRes<WriteVecShiftImmX, [ADLPPort00_01]>;
 defm : X86WriteResUnsupported<WriteVecShiftImmXLd>;
@@ -504,15 +493,15 @@ def : WriteRes<WriteVecShiftImmY, [ADLPPort00_01]>;
 defm : X86WriteResUnsupported<WriteVecShiftImmYLd>;
 defm : X86WriteResPairUnsupported<WriteVecShiftImmZ>;
 defm : X86WriteRes<WriteVecShiftX, [ADLPPort00_01, ADLPPort01_05], 2, [1, 1], 2>;
-defm : X86WriteRes<WriteVecShiftXLd, [ADLPPort00_01, ADLPPort02_03_11], 7, [1, 1], 2>;
+defm : X86WriteRes<WriteVecShiftXLd, [ADLPPort00_01, ADLPPort02_03_11], 8, [1, 1], 2>;
 defm : X86WriteRes<WriteVecShiftY, [ADLPPort00_01, ADLPPort05], 4, [1, 1], 2>;
-defm : X86WriteRes<WriteVecShiftYLd, [ADLPPort00_01, ADLPPort02_03_11], 8, [1, 1], 2>;
+defm : X86WriteRes<WriteVecShiftYLd, [ADLPPort00_01, ADLPPort02_03_11], 9, [1, 1], 2>;
 defm : X86WriteResPairUnsupported<WriteVecShiftZ>;
-defm : X86WriteRes<WriteVecStore, [ADLPPort04_09, ADLPPort07_08], 1, [1, 1], 2>;
+defm : X86WriteRes<WriteVecStore, [ADLPPort04_09, ADLPPort07_08], 12, [1, 1], 2>;
 defm : X86WriteRes<WriteVecStoreNT, [ADLPPort04_09, ADLPPort07_08], 511, [1, 1], 2>;
 defm : X86WriteRes<WriteVecStoreNTY, [ADLPPort04_09, ADLPPort07_08], 507, [1, 1], 2>;
-defm : X86WriteRes<WriteVecStoreX, [ADLPPort04_09, ADLPPort07_08], 1, [1, 1], 2>;
-defm : X86WriteRes<WriteVecStoreY, [ADLPPort04_09, ADLPPort07_08], 1, [1, 1], 2>;
+defm : X86WriteRes<WriteVecStoreX, [ADLPPort04_09, ADLPPort07_08], 12, [1, 1], 2>;
+defm : X86WriteRes<WriteVecStoreY, [ADLPPort04_09, ADLPPort07_08], 12, [1, 1], 2>;
 defm : ADLPWriteResPair<WriteVecTest, [ADLPPort00, ADLPPort05], 4, [1, 1], 2>;
 defm : ADLPWriteResPair<WriteVecTestY, [ADLPPort00, ADLPPort05], 6, [1, 1], 2, 6>;
 defm : X86WriteRes<WriteXCHG, [ADLPPort00_01_05_06_10], 2, [3], 3>;
@@ -520,231 +509,244 @@ def : WriteRes<WriteZero, []>;
 
 // Infered SchedWriteRes and InstRW definition.
 
-def ADLPWriteResGroup0 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
-  let ResourceCycles = [2, 1, 1, 1, 1];
+def ADLPWriteResGroup0 : SchedWriteRes<[ADLPPort00_01_05_06, ADLPPort02_03, ADLPPort02_03_07, ADLPPort04]> {
   let Latency = 7;
+  let NumMicroOps = 3;
+}
+def : InstRW<[ADLPWriteResGroup0], (instregex "^AA(D|N)D64mr$",
+                                              "^A(X?)OR64mr$")>;
+
+def ADLPWriteResGroup1 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
+  let ResourceCycles = [2, 1, 1, 1, 1];
+  let Latency = 12;
   let NumMicroOps = 6;
 }
-def : InstRW<[ADLPWriteResGroup0, ReadAfterLd, ReadAfterLd, ReadDefault, ReadDefault, ReadDefault, ReadDefault, ReadDefault], (instregex "^(ADC|SBB)(8|16|32|64)mr$")>;
+def : InstRW<[ADLPWriteResGroup1, ReadAfterLd, ReadAfterLd, ReadDefault, ReadDefault, ReadDefault, ReadDefault, ReadDefault], (instregex "^(ADC|SBB)(16|32|64)mr$")>;
 
-def ADLPWriteResGroup1 : SchedWriteRes<[ADLPPort00_06, ADLPPort02_03_11]> {
+def ADLPWriteResGroup2 : SchedWriteRes<[ADLPPort00_06, ADLPPort02_03_11]> {
   let Latency = 6;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup1], (instregex "^JMP(16|32)m$",
-                                              "^JMP(16|32|64)m_NT$",
+def : InstRW<[ADLPWriteResGroup2], (instregex "^JMP(16|32|64)m((_NT)?)$",
                                               "^RET(16|32)$",
                                               "^RORX(32|64)mi$")>;
-def : InstRW<[ADLPWriteResGroup1, ReadAfterLd, ReadDefault, ReadDefault, ReadDefault, ReadDefault, ReadDefault], (instregex "^S(A|H)RX(32|64)rm$",
-                                                                                                                            "^SHLX(32|64)rm$")>;
-def : InstRW<[ADLPWriteResGroup1, ReadAfterLd, ReadAfterLd, ReadDefault, ReadDefault, ReadDefault, ReadDefault, ReadDefault], (instregex "^(ADC|SBB)(8|16|32|64)rm$",
+def : InstRW<[ADLPWriteResGroup2, ReadAfterLd, ReadAfterLd, ReadDefault, ReadDefault, ReadDefault, ReadDefault, ReadDefault], (instregex "^(ADC|SBB)(8|16|32|64)rm$",
                                                                                                                                          "^AD(C|O)X(32|64)rm$")>;
 
-def ADLPWriteResGroup2 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup3 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 13;
   let NumMicroOps = 5;
 }
-def : InstRW<[ADLPWriteResGroup2], (instregex "^(ADC|SBB)8mi(8?)$")>;
+def : InstRW<[ADLPWriteResGroup3], (instregex "^(ADC|SBB)8mi(8?)$")>;
+
+def ADLPWriteResGroup4 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
+  let ResourceCycles = [2, 1, 1, 1, 1];
+  let Latency = 13;
+  let NumMicroOps = 6;
+}
+def : InstRW<[ADLPWriteResGroup4, ReadAfterLd, ReadAfterLd, ReadDefault, ReadDefault, ReadDefault, ReadDefault, ReadDefault], (instregex "^(ADC|SBB)8mr$")>;
 
-def ADLPWriteResGroup3 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11]> {
+def ADLPWriteResGroup5 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11]> {
   let Latency = 6;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup3], (instregex "^(CMP|TEST)(8|16|32)mi$",
+def : InstRW<[ADLPWriteResGroup5], (instregex "^CMP(8|16|32)mi$",
                                               "^CMP(8|16|32|64)mi8$",
-                                              "^(CMP|TEST)64mi32$",
-                                              "^MOV(8|16)(rm|ao64)$",
-                                              "^MOVSX16rm(16|32)$",
-                                              "^MOVZX16rm(8|16)$",
-                                              "^POP32r((mr)?)$")>;
-def : InstRW<[ADLPWriteResGroup3], (instrs MOV8rm_NOREX)>;
-def : InstRW<[ADLPWriteResGroup3, ReadAfterLd], (instregex "^A(D|N)D(8|16|32|64)rm$",
-                                                           "^(CMP|SUB)(8|16|32|64)rm$",
-                                                           "^(X?)OR(8|16|32|64)rm$")>;
-def : InstRW<[ADLPWriteResGroup3, ReadAfterLd, ReadDefault, ReadDefault, ReadDefault, ReadDefault, ReadDefault], (instregex "^(CMP|TEST)(8|16|32|64)mr$")>;
-
-def ADLPWriteResGroup4 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
-  let Latency = 13;
-  let NumMicroOps = 4;
+                                              "^MOV(8|16)rm$",
+                                              "^POP(16|32)r((mr)?)$")>;
+def : InstRW<[ADLPWriteResGroup5], (instrs CMP64mi32,
+                                           MOV8rm_NOREX,
+                                           MOVZX16rm8)>;
+def : InstRW<[ADLPWriteResGroup5, ReadAfterLd], (instregex "^(ADD|CMP|SUB)(8|16|32|64)rm$",
+                                                           "^AND(8|16|32)rm$",
+                                                           "^(X?)OR(8|16|32)rm$")>;
+def : InstRW<[ADLPWriteResGroup5, ReadAfterLd, ReadDefault, ReadDefault, ReadDefault, ReadDefault, ReadDefault], (instregex "^CMP(8|16|32|64)mr$")>;
+
+def ADLPWriteResGroup6 : SchedWriteRes<[]> {
+  let NumMicroOps = 0;
 }
-def : InstRW<[ADLPWriteResGroup4], (instregex "^A(D|N)D8mi8$",
-                                              "^(X?)OR8mi8$")>;
-def : InstRW<[ADLPWriteResGroup4], (instrs SUB8mi8)>;
+def : InstRW<[ADLPWriteResGroup6], (instregex "^(ADD|SUB)64ri8$",
+                                              "^(DE|IN)C64r$",
+                                              "^MOV64rr((_REV)?)$")>;
+def : InstRW<[ADLPWriteResGroup6], (instrs CLC,
+                                           JMP_2)>;
 
-def ADLPWriteResGroup5 : SchedWriteRes<[ADLPPort01_05, ADLPPort02_03_11]> {
-  let Latency = 10;
-  let NumMicroOps = 2;
+def ADLPWriteResGroup7 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
+  let Latency = 13;
+  let NumMicroOps = 4;
 }
-def : InstRW<[ADLPWriteResGroup5, ReadAfterVecXLd], (instregex "^(ADD|SUB)PSrm$")>;
-def : InstRW<[ADLPWriteResGroup5, ReadAfterVecLd], (instregex "^(V?)(ADD|SUB)SSrm$")>;
+def : InstRW<[ADLPWriteResGroup7], (instregex "^A(D|N)D8mi(8?)$",
+                                              "^(DE|IN)C8m$",
+                                              "^N(EG|OT)8m$",
+                                              "^(X?)OR8mi(8?)$",
+                                              "^SUB8mi(8?)$")>;
+def : InstRW<[ADLPWriteResGroup7, ReadAfterLd, ReadDefault, ReadDefault, ReadDefault, ReadDefault, ReadDefault], (instregex "^A(D|N)D8mr$",
+                                                                                                                            "^(X?)OR8mr$")>;
+def : InstRW<[ADLPWriteResGroup7, ReadAfterLd, ReadDefault, ReadDefault, ReadDefault, ReadDefault, ReadDefault], (instrs SUB8mr)>;
 
-def ADLPWriteResGroup6 : SchedWriteRes<[ADLPPort01_05]> {
+def ADLPWriteResGroup8 : SchedWriteRes<[ADLPPort01_05]> {
   let Latency = 3;
 }
-def : InstRW<[ADLPWriteResGroup6], (instregex "^(ADD|SUB)(P|S)Srr$",
-                                              "^V(ADD|SUB)SSrr$")>;
+def : InstRW<[ADLPWriteResGroup8], (instregex "^(V?)(ADD|SUB)SSrr((_Int)?)$")>;
 
-def ADLPWriteResGroup7 : SchedWriteRes<[ADLPPort01_05, ADLPPort02_03_11]> {
-  let Latency = 9;
+def ADLPWriteResGroup9 : SchedWriteRes<[ADLPPort02_03, ADLPPort05]> {
+  let Latency = 10;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup7, ReadAfterVecLd], (instregex "^(V?)(ADD|SUB)S(D|S)rm_Int$")>;
+def : InstRW<[ADLPWriteResGroup9], (instregex "^ADD_F(32|64)m$",
+                                              "^ILD_F(16|32|64)m$",
+                                              "^SUB(R?)_F(32|64)m$")>;
 
-def ADLPWriteResGroup8 : SchedWriteRes<[ADLPPort01_05]> {
-  let Latency = 2;
-}
-def : InstRW<[ADLPWriteResGroup8], (instregex "^(V?)(ADD|SUB)S(D|S)rr_Int$")>;
-
-def ADLPWriteResGroup9 : SchedWriteRes<[ADLPPort02_03, ADLPPort05]> {
+def ADLPWriteResGroup10 : SchedWriteRes<[ADLPPort02_03, ADLPPort05]> {
   let ResourceCycles = [1, 2];
   let Latency = 13;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup9], (instregex "^ADD_FI(16|32)m$",
-                                              "^SUB(R?)_FI(16|32)m$")>;
+def : InstRW<[ADLPWriteResGroup10], (instregex "^ADD_FI(16|32)m$",
+                                               "^SUB(R?)_FI(16|32)m$")>;
 
-def ADLPWriteResGroup10 : SchedWriteRes<[ADLPPort00_01_05_06_10]> {
+def ADLPWriteResGroup11 : SchedWriteRes<[ADLPPort00_01_05_06_10]> {
   let Latency = 2;
 }
-def : InstRW<[ADLPWriteResGroup10], (instregex "^(AND|TEST)(32|64)i32$",
+def : InstRW<[ADLPWriteResGroup11], (instregex "^AND(8|16|32|64)r(r|i8)$",
+                                               "^AND(8|16|32|64)rr_REV$",
+                                               "^(AND|TEST)(32|64)i32$",
+                                               "^(AND|TEST)(8|32)ri$",
+                                               "^(AND|TEST)64ri32$",
+                                               "^(AND|TEST)8i8$",
+                                               "^(X?)OR(8|16|32|64)r(r|i8)$",
+                                               "^(X?)OR(8|16|32|64)rr_REV$",
                                                "^(X?)OR(32|64)i32$",
-                                               "^(X?)OR8ri8$")>;
-def : InstRW<[ADLPWriteResGroup10], (instrs AND8ri8,
-                                            XOR8rr_NOREX)>;
+                                               "^(X?)OR(8|32)ri$",
+                                               "^(X?)OR64ri32$",
+                                               "^(X?)OR8i8$",
+                                               "^TEST(8|16|32|64)rr$")>;
+def : InstRW<[ADLPWriteResGroup11], (instrs XOR8rr_NOREX)>;
 
-def ADLPWriteResGroup11 : SchedWriteRes<[ADLPPort01_05_10, ADLPPort02_03_11]> {
-  let Latency = 6;
+def ADLPWriteResGroup12 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11]> {
+  let Latency = 7;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup11], (instrs MOVSX16rm8)>;
-def : InstRW<[ADLPWriteResGroup11, ReadAfterLd], (instregex "^ANDN(32|64)rm$")>;
+def : InstRW<[ADLPWriteResGroup12], (instregex "^TEST(8|16|32)mi$")>;
+def : InstRW<[ADLPWriteResGroup12], (instrs TEST64mi32)>;
+def : InstRW<[ADLPWriteResGroup12, ReadAfterLd], (instregex "^(X?)OR64rm$")>;
+def : InstRW<[ADLPWriteResGroup12, ReadAfterLd], (instrs AND64rm)>;
+def : InstRW<[ADLPWriteResGroup12, ReadAfterLd, ReadDefault, ReadDefault, ReadDefault, ReadDefault, ReadDefault], (instregex "^TEST(8|16|32|64)mr$")>;
 
-def ADLPWriteResGroup12 : SchedWriteRes<[ADLPPort01_05_10]>;
-def : InstRW<[ADLPWriteResGroup12], (instregex "^ANDN(32|64)rr$",
-                                               "^C(DQ|WD)E$",
-                                               "^MOVSX(16|32|64)rr(8|32)$",
-                                               "^MOVSX(32|64)rr16$")>;
-def : InstRW<[ADLPWriteResGroup12], (instrs CBW,
-                                            MOVSX32rr8_NOREX)>;
+def ADLPWriteResGroup13 : SchedWriteRes<[ADLPPort01_05_10, ADLPPort02_03_11]> {
+  let Latency = 7;
+  let NumMicroOps = 2;
+}
+def : InstRW<[ADLPWriteResGroup13, ReadAfterLd], (instregex "^ANDN(32|64)rm$")>;
 
-def ADLPWriteResGroup13 : SchedWriteRes<[ADLPPort01]>;
-def : InstRW<[ADLPWriteResGroup13], (instrs BSWAP32r,
-                                            LEA32r)>;
+def ADLPWriteResGroup14 : SchedWriteRes<[ADLPPort01_05_10]> {
+  let Latency = 2;
+}
+def : InstRW<[ADLPWriteResGroup14], (instregex "^ANDN(32|64)rr$")>;
 
-def ADLPWriteResGroup14 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11]> {
+def ADLPWriteResGroup15 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11]> {
   let ResourceCycles = [5, 2, 1, 1];
   let Latency = 10;
   let NumMicroOps = 9;
 }
-def : InstRW<[ADLPWriteResGroup14], (instrs BT64mr)>;
+def : InstRW<[ADLPWriteResGroup15], (instrs BT64mr)>;
+
+def ADLPWriteResGroup16 : SchedWriteRes<[ADLPPort01]> {
+  let Latency = 3;
+}
+def : InstRW<[ADLPWriteResGroup16], (instregex "^BT((C|R|S)?)64rr$",
+                                               "^P(DEP|EXT)(32|64)rr$")>;
 
-def ADLPWriteResGroup15 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup17 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
   let ResourceCycles = [4, 2, 1, 1, 1, 1];
   let Latency = 17;
   let NumMicroOps = 10;
 }
-def : InstRW<[ADLPWriteResGroup15], (instregex "^BT(C|R|S)64mr$")>;
+def : InstRW<[ADLPWriteResGroup17], (instregex "^BT(C|R|S)64mr$")>;
 
-def ADLPWriteResGroup16 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup18 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 7;
   let NumMicroOps = 5;
 }
-def : InstRW<[ADLPWriteResGroup16], (instregex "^CALL(16|32)m$",
-                                               "^CALL(16|32|64)m_NT$")>;
+def : InstRW<[ADLPWriteResGroup18], (instregex "^CALL(16|32|64)m((_NT)?)$")>;
 
-def ADLPWriteResGroup17 : SchedWriteRes<[ADLPPort00_06, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup19 : SchedWriteRes<[ADLPPort00_06, ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 3;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup17], (instregex "^CALL(16|32)r$",
-                                               "^CALL(16|32|64)r_NT$")>;
+def : InstRW<[ADLPWriteResGroup19], (instregex "^CALL(16|32|64)r((_NT)?)$")>;
 
-def ADLPWriteResGroup18 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11, ADLPPort04_09, ADLPPort06, ADLPPort07_08]> {
-  let Latency = 7;
-  let NumMicroOps = 5;
-}
-def : InstRW<[ADLPWriteResGroup18], (instrs CALL64m)>;
-
-def ADLPWriteResGroup19 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort04_09, ADLPPort06, ADLPPort07_08]> {
+def ADLPWriteResGroup20 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 3;
-  let NumMicroOps = 4;
+  let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup19], (instrs CALL64pcrel32,
-                                            CALL64r)>;
+def : InstRW<[ADLPWriteResGroup20], (instrs CALL64pcrel32,
+                                            MFENCE)>;
 
-def ADLPWriteResGroup20 : SchedWriteRes<[ADLPPort00_06]>;
-def : InstRW<[ADLPWriteResGroup20], (instregex "^C(DQ|QO)$",
-                                               "^(CL|ST)AC$",
-                                               "^ROL(8|16|32|64)ri$",
-                                               "^ROR(16|32|64)ri$")>;
+def ADLPWriteResGroup21 : SchedWriteRes<[ADLPPort01_05]>;
+def : InstRW<[ADLPWriteResGroup21], (instregex "^C(DQ|WD)E$",
+                                               "^(V?)MOVS(H|L)DUPrr$",
+                                               "^(V?)SHUFP(D|S)rri$",
+                                               "^VMOVS(H|L)DUPYrr$",
+                                               "^VSHUFP(D|S)Yrri$")>;
+def : InstRW<[ADLPWriteResGroup21], (instrs CBW,
+                                            VPBLENDWYrri)>;
 
-def ADLPWriteResGroup21 : SchedWriteRes<[]> {
-  let NumMicroOps = 0;
-}
-def : InstRW<[ADLPWriteResGroup21], (instrs CLC,
-                                            JMP_2)>;
+def ADLPWriteResGroup22 : SchedWriteRes<[ADLPPort00_06]>;
+def : InstRW<[ADLPWriteResGroup22], (instregex "^C(DQ|QO)$",
+                                               "^(CL|ST)AC$")>;
 
-def ADLPWriteResGroup22 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06]> {
+def ADLPWriteResGroup23 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06]> {
   let Latency = 3;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup22], (instrs CLD)>;
+def : InstRW<[ADLPWriteResGroup23], (instrs CLD)>;
 
-def ADLPWriteResGroup23 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup24 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 3;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup23], (instrs CLDEMOTE)>;
+def : InstRW<[ADLPWriteResGroup24], (instrs CLDEMOTE)>;
 
-def ADLPWriteResGroup24 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup25 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 2;
   let NumMicroOps = 4;
 }
-def : InstRW<[ADLPWriteResGroup24], (instrs CLFLUSH)>;
+def : InstRW<[ADLPWriteResGroup25], (instrs CLFLUSH)>;
 
-def ADLPWriteResGroup25 : SchedWriteRes<[ADLPPort00_05_06_10, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup26 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 2;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup25], (instrs CLFLUSHOPT)>;
+def : InstRW<[ADLPWriteResGroup26], (instrs CLFLUSHOPT)>;
 
-def ADLPWriteResGroup26 : SchedWriteRes<[ADLPPort01, ADLPPort06]> {
-  let ResourceCycles = [1, 2];
+def ADLPWriteResGroup27 : SchedWriteRes<[ADLPPort00_06, ADLPPort01]> {
+  let ResourceCycles = [2, 1];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup26], (instrs CLI)>;
+def : InstRW<[ADLPWriteResGroup27], (instrs CLI)>;
 
-def ADLPWriteResGroup27 : SchedWriteRes<[ADLPPort00_06, ADLPPort01, ADLPPort05]> {
+def ADLPWriteResGroup28 : SchedWriteRes<[ADLPPort00_06, ADLPPort01, ADLPPort05]> {
   let ResourceCycles = [6, 1, 3];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 10;
 }
-def : InstRW<[ADLPWriteResGroup27], (instrs CLTS)>;
+def : InstRW<[ADLPWriteResGroup28], (instrs CLTS)>;
 
-def ADLPWriteResGroup28 : SchedWriteRes<[ADLPPort00_05_06_10, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup29 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 5;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup28], (instrs CLWB)>;
+def : InstRW<[ADLPWriteResGroup29], (instregex "^MOV16o(16|32|64)a$")>;
+def : InstRW<[ADLPWriteResGroup29], (instrs CLWB)>;
 
-def ADLPWriteResGroup29 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11]> {
+def ADLPWriteResGroup30 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11]> {
   let ResourceCycles = [5, 2];
   let Latency = 6;
   let NumMicroOps = 7;
 }
-def : InstRW<[ADLPWriteResGroup29], (instregex "^CMPS(B|L|Q|W)$")>;
-
-def ADLPWriteResGroup30 : SchedWriteRes<[ADLPPort00_01, ADLPPort02_03_11]> {
-  let Latency = 10;
-  let NumMicroOps = 2;
-}
-def : InstRW<[ADLPWriteResGroup30, ReadAfterVecLd], (instregex "^(V?)CMPS(D|S)rm_Int$",
-                                                               "^(V?)CVTS(I|S)2SDrm_Int$",
-                                                               "^(V?)CVTSI2SSrm_Int$",
-                                                               "^(V?)CVTSI642SDrm_Int$",
-                                                               "^(V?)M(AX|IN|UL)S(D|S)rm_Int$")>;
-def : InstRW<[ADLPWriteResGroup30, ReadAfterVecLd, ReadAfterVecLd], (instregex "^VF(N?)M(ADD|SUB)(132|213|231)S(D|S)m_Int$")>;
+def : InstRW<[ADLPWriteResGroup30], (instregex "^CMPS(B|L|Q|W)$")>;
 
 def ADLPWriteResGroup31 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01_05, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [2, 7, 6, 2, 1, 1, 2, 1];
@@ -767,1870 +769,1708 @@ def ADLPWriteResGroup33 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06,
 }
 def : InstRW<[ADLPWriteResGroup33], (instrs CMPXCHG8rm)>;
 
-def ADLPWriteResGroup34 : SchedWriteRes<[ADLPPort00, ADLPPort02_03_11]> {
-  let Latency = 8;
-  let NumMicroOps = 2;
-}
-def : InstRW<[ADLPWriteResGroup34, ReadAfterVecLd], (instregex "^((U|V|VU)?)COMIS(D|S)rm_Int$")>;
-
-def ADLPWriteResGroup35 : SchedWriteRes<[ADLPPort00, ADLPPort00_01, ADLPPort00_06, ADLPPort01, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup34 : SchedWriteRes<[ADLPPort00, ADLPPort00_01, ADLPPort00_06, ADLPPort01, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [2, 1, 10, 6, 1, 5, 1];
   let Latency = 18;
   let NumMicroOps = 26;
 }
-def : InstRW<[ADLPWriteResGroup35], (instrs CPUID)>;
-
-def ADLPWriteResGroup36 : SchedWriteRes<[ADLPPort00_01, ADLPPort02_03_11]> {
-  let Latency = 11;
-  let NumMicroOps = 2;
-}
-def : InstRW<[ADLPWriteResGroup36], (instregex "^VPDP(BU|WS)SD(S?)rm$")>;
-def : InstRW<[ADLPWriteResGroup36], (instrs CVTDQ2PDrm)>;
-def : InstRW<[ADLPWriteResGroup36, ReadAfterVecXLd], (instregex "^(V?)M(AX|IN)CP(D|S)rm$")>;
-def : InstRW<[ADLPWriteResGroup36, ReadAfterVecXLd], (instrs VGF2P8MULBrm)>;
-def : InstRW<[ADLPWriteResGroup36, ReadAfterVecYLd], (instregex "^VGF2P8AFFINE((INV)?)QBYrmi$")>;
-def : InstRW<[ADLPWriteResGroup36, ReadAfterVecYLd], (instrs VGF2P8MULBYrm)>;
+def : InstRW<[ADLPWriteResGroup34], (instrs CPUID)>;
 
-def ADLPWriteResGroup37 : SchedWriteRes<[ADLPPort00_01, ADLPPort02_03_11, ADLPPort05]> {
-  let Latency = 12;
-  let NumMicroOps = 3;
-}
-def : InstRW<[ADLPWriteResGroup37], (instregex "^CVT(T?)PD2DQrm$")>;
-def : InstRW<[ADLPWriteResGroup37], (instrs CVTSI642SSrm)>;
-def : InstRW<[ADLPWriteResGroup37, ReadAfterVecLd], (instrs VCVTSI642SSrm)>;
-
-def ADLPWriteResGroup38 : SchedWriteRes<[ADLPPort00, ADLPPort00_01, ADLPPort02_03_11]> {
+def ADLPWriteResGroup35 : SchedWriteRes<[ADLPPort00, ADLPPort00_01, ADLPPort02_03_11]> {
   let Latency = 26;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup38], (instregex "^(V?)CVT(T?)SD2SIrm$")>;
+def : InstRW<[ADLPWriteResGroup35], (instregex "^(V?)CVT(T?)SD2SIrm((_Int)?)$")>;
 
-def ADLPWriteResGroup39 : SchedWriteRes<[ADLPPort00_01, ADLPPort02_03_11, ADLPPort05]> {
-  let Latency = 11;
+def ADLPWriteResGroup36 : SchedWriteRes<[ADLPPort00_01, ADLPPort02_03_11, ADLPPort05]> {
+  let Latency = 12;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup39], (instrs VCVTPD2PSrm)>;
-def : InstRW<[ADLPWriteResGroup39, ReadAfterVecLd], (instregex "^(V?)CVTS(D|I64)2SSrm_Int$")>;
+def : InstRW<[ADLPWriteResGroup36], (instrs CVTSI642SSrm)>;
+def : InstRW<[ADLPWriteResGroup36, ReadAfterVecLd], (instregex "^(V?)CVTSI642SSrm_Int$")>;
+def : InstRW<[ADLPWriteResGroup36, ReadAfterVecLd], (instrs VCVTSI642SSrm)>;
 
-def ADLPWriteResGroup40 : SchedWriteRes<[ADLPPort00_01, ADLPPort05]> {
-  let Latency = 5;
-  let NumMicroOps = 2;
-}
-def : InstRW<[ADLPWriteResGroup40, ReadDefault, ReadInt2Fpu], (instregex "^CVTSI((64)?)2SDrr_Int$")>;
-def : InstRW<[ADLPWriteResGroup40, ReadDefault, ReadInt2Fpu], (instrs CVTSI2SSrr_Int)>;
-
-def ADLPWriteResGroup41 : SchedWriteRes<[ADLPPort00_01, ADLPPort05]> {
+def ADLPWriteResGroup37 : SchedWriteRes<[ADLPPort00_01, ADLPPort05]> {
   let ResourceCycles = [1, 2];
   let Latency = 8;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup41, ReadInt2Fpu], (instrs CVTSI642SSrr)>;
-def : InstRW<[ADLPWriteResGroup41, ReadDefault, ReadInt2Fpu], (instregex "^VCVTSI642SSrr((_Int)?)$")>;
-
-def ADLPWriteResGroup42 : SchedWriteRes<[ADLPPort00_01, ADLPPort05]> {
-  let ResourceCycles = [1, 2];
-  let Latency = 6;
-  let NumMicroOps = 3;
-}
-def : InstRW<[ADLPWriteResGroup42, ReadDefault, ReadInt2Fpu], (instrs CVTSI642SSrr_Int)>;
+def : InstRW<[ADLPWriteResGroup37, ReadInt2Fpu], (instrs CVTSI642SSrr)>;
+def : InstRW<[ADLPWriteResGroup37, ReadDefault, ReadInt2Fpu], (instregex "^(V?)CVTSI642SSrr_Int$")>;
+def : InstRW<[ADLPWriteResGroup37, ReadDefault, ReadInt2Fpu], (instrs VCVTSI642SSrr)>;
 
-def ADLPWriteResGroup43 : SchedWriteRes<[ADLPPort00, ADLPPort00_01, ADLPPort05]> {
+def ADLPWriteResGroup38 : SchedWriteRes<[ADLPPort00, ADLPPort00_01, ADLPPort05]> {
   let Latency = 8;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup43], (instregex "^(V?)CVT(T?)SS2SI64rr_Int$")>;
-def : InstRW<[ADLPWriteResGroup43, ReadDefault], (instregex "^(V?)CVT(T?)SS2SI64rr$")>;
+def : InstRW<[ADLPWriteResGroup38], (instregex "^(V?)CVT(T?)SS2SI64rr_Int$")>;
+def : InstRW<[ADLPWriteResGroup38, ReadDefault], (instregex "^(V?)CVT(T?)SS2SI64rr$")>;
 
-def ADLPWriteResGroup44 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06]> {
+def ADLPWriteResGroup39 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06]> {
   let Latency = 2;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup44], (instregex "^J(E|R)CXZ$")>;
-def : InstRW<[ADLPWriteResGroup44], (instrs CWD)>;
+def : InstRW<[ADLPWriteResGroup39], (instregex "^J(E|R)CXZ$")>;
+def : InstRW<[ADLPWriteResGroup39], (instrs CWD)>;
 
-def ADLPWriteResGroup45 : SchedWriteRes<[ADLPPort00_01_05_06]>;
-def : InstRW<[ADLPWriteResGroup45], (instregex "^(LD|ST)_Frr$",
+def ADLPWriteResGroup40 : SchedWriteRes<[ADLPPort00_01_05_06]>;
+def : InstRW<[ADLPWriteResGroup40], (instregex "^(LD|ST)_Frr$",
                                                "^MOV16s(m|r)$",
                                                "^MOV(32|64)sr$")>;
-def : InstRW<[ADLPWriteResGroup45], (instrs DEC16r_alt,
+def : InstRW<[ADLPWriteResGroup40], (instrs DEC16r_alt,
                                             SALC,
                                             ST_FPrr,
                                             SYSCALL)>;
 
-def ADLPWriteResGroup46 : SchedWriteRes<[ADLPPort00_06, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup41 : SchedWriteRes<[ADLPPort00_06, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 7;
 }
-def : InstRW<[ADLPWriteResGroup46], (instrs DEC32r_alt)>;
+def : InstRW<[ADLPWriteResGroup41], (instrs DEC32r_alt)>;
 
-def ADLPWriteResGroup47 : SchedWriteRes<[ADLPPort00, ADLPPort02_03, ADLPPort05]> {
+def ADLPWriteResGroup42 : SchedWriteRes<[ADLPPort00, ADLPPort02_03]> {
+  let Latency = 27;
+  let NumMicroOps = 2;
+}
+def : InstRW<[ADLPWriteResGroup42], (instregex "^DIVR_F(32|64)m$")>;
+
+def ADLPWriteResGroup43 : SchedWriteRes<[ADLPPort00, ADLPPort02_03, ADLPPort05]> {
   let Latency = 30;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup47], (instregex "^DIVR_FI(16|32)m$")>;
+def : InstRW<[ADLPWriteResGroup43], (instregex "^DIVR_FI(16|32)m$")>;
 
-def ADLPWriteResGroup48 : SchedWriteRes<[ADLPPort00]> {
+def ADLPWriteResGroup44 : SchedWriteRes<[ADLPPort00]> {
   let Latency = 15;
 }
-def : InstRW<[ADLPWriteResGroup48], (instregex "^DIVR_F(P?)rST0$")>;
-def : InstRW<[ADLPWriteResGroup48], (instrs DIVR_FST0r)>;
+def : InstRW<[ADLPWriteResGroup44], (instregex "^DIVR_F(P?)rST0$")>;
+def : InstRW<[ADLPWriteResGroup44], (instrs DIVR_FST0r)>;
 
-def ADLPWriteResGroup49 : SchedWriteRes<[ADLPPort00, ADLPPort02_03_11]> {
+def ADLPWriteResGroup45 : SchedWriteRes<[ADLPPort00, ADLPPort02_03_11]> {
   let Latency = 20;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup49, ReadAfterVecLd], (instregex "^(V?)DIVSDrm_Int$")>;
+def : InstRW<[ADLPWriteResGroup45, ReadAfterVecLd], (instregex "^(V?)DIVSDrm_Int$")>;
 
-def ADLPWriteResGroup50 : SchedWriteRes<[ADLPPort00, ADLPPort02_03_11]> {
-  let Latency = 18;
-  let NumMicroOps = 2;
-}
-def : InstRW<[ADLPWriteResGroup50, ReadAfterVecLd], (instregex "^(V?)DIVSSrm$",
-                                                               "^(V?)SQRTSSm_Int$")>;
-
-def ADLPWriteResGroup51 : SchedWriteRes<[ADLPPort00, ADLPPort02_03_11]> {
-  let Latency = 17;
-  let NumMicroOps = 2;
-}
-def : InstRW<[ADLPWriteResGroup51, ReadAfterVecLd], (instregex "^(V?)DIVSSrm_Int$")>;
-
-def ADLPWriteResGroup52 : SchedWriteRes<[ADLPPort00, ADLPPort02_03]> {
+def ADLPWriteResGroup46 : SchedWriteRes<[ADLPPort00, ADLPPort02_03]> {
   let Latency = 22;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup52], (instregex "^DIV_F(32|64)m$")>;
+def : InstRW<[ADLPWriteResGroup46], (instregex "^DIV_F(32|64)m$")>;
 
-def ADLPWriteResGroup53 : SchedWriteRes<[ADLPPort00, ADLPPort02_03, ADLPPort05]> {
+def ADLPWriteResGroup47 : SchedWriteRes<[ADLPPort00, ADLPPort02_03, ADLPPort05]> {
   let Latency = 25;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup53], (instregex "^DIV_FI(16|32)m$")>;
+def : InstRW<[ADLPWriteResGroup47], (instregex "^DIV_FI(16|32)m$")>;
 
-def ADLPWriteResGroup54 : SchedWriteRes<[ADLPPort00]> {
+def ADLPWriteResGroup48 : SchedWriteRes<[ADLPPort00]> {
   let Latency = 20;
 }
-def : InstRW<[ADLPWriteResGroup54], (instregex "^DIV_F(P?)rST0$")>;
-def : InstRW<[ADLPWriteResGroup54], (instrs DIV_FST0r)>;
+def : InstRW<[ADLPWriteResGroup48], (instregex "^DIV_F(P?)rST0$")>;
+def : InstRW<[ADLPWriteResGroup48], (instrs DIV_FST0r)>;
 
-def ADLPWriteResGroup55 : SchedWriteRes<[ADLPPort00, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup49 : SchedWriteRes<[ADLPPort00, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [2, 21, 2, 14, 4, 9, 5];
   let Latency = 126;
   let NumMicroOps = 57;
 }
-def : InstRW<[ADLPWriteResGroup55], (instrs ENTER)>;
+def : InstRW<[ADLPWriteResGroup49], (instrs ENTER)>;
 
-def ADLPWriteResGroup56 : SchedWriteRes<[ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
-  let Latency = 2;
+def ADLPWriteResGroup50 : SchedWriteRes<[ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+  let Latency = 12;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup56], (instregex "^(V?)EXTRACTPSmr$")>;
+def : InstRW<[ADLPWriteResGroup50], (instregex "^(V?)EXTRACTPSmr$")>;
+def : InstRW<[ADLPWriteResGroup50], (instrs SMSW16m)>;
 
-def ADLPWriteResGroup57 : SchedWriteRes<[ADLPPort00, ADLPPort05]> {
+def ADLPWriteResGroup51 : SchedWriteRes<[ADLPPort00, ADLPPort05]> {
   let Latency = 4;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup57], (instregex "^(V?)EXTRACTPSrr$")>;
-def : InstRW<[ADLPWriteResGroup57], (instrs MMX_PEXTRWrr)>;
+def : InstRW<[ADLPWriteResGroup51], (instregex "^(V?)EXTRACTPSrr$")>;
+def : InstRW<[ADLPWriteResGroup51], (instrs MMX_PEXTRWrr)>;
 
-def ADLPWriteResGroup58 : SchedWriteRes<[ADLPPort00_01_05_06, ADLPPort02_03, ADLPPort02_03_07, ADLPPort04, ADLPPort06]> {
+def ADLPWriteResGroup52 : SchedWriteRes<[ADLPPort00_01_05_06, ADLPPort02_03, ADLPPort02_03_07, ADLPPort04, ADLPPort06]> {
   let Latency = 7;
   let NumMicroOps = 5;
 }
-def : InstRW<[ADLPWriteResGroup58], (instrs FARCALL64m)>;
+def : InstRW<[ADLPWriteResGroup52], (instrs FARCALL64m)>;
 
-def ADLPWriteResGroup59 : SchedWriteRes<[ADLPPort02_03, ADLPPort06]> {
+def ADLPWriteResGroup53 : SchedWriteRes<[ADLPPort02_03, ADLPPort06]> {
   let Latency = 6;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup59], (instrs FARJMP64m,
+def : InstRW<[ADLPWriteResGroup53], (instrs FARJMP64m,
                                             JMP64m_REX)>;
 
-def ADLPWriteResGroup60 : SchedWriteRes<[ADLPPort02_03_07, ADLPPort04]> {
+def ADLPWriteResGroup54 : SchedWriteRes<[ADLPPort02_03_07, ADLPPort04]> {
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup60], (instregex "^(V?)MASKMOVDQU((64)?)$",
+def : InstRW<[ADLPWriteResGroup54], (instregex "^(V?)MASKMOVDQU((64)?)$",
                                                "^ST_FP(32|64|80)m$")>;
-def : InstRW<[ADLPWriteResGroup60], (instrs FBSTPm,
+def : InstRW<[ADLPWriteResGroup54], (instrs FBSTPm,
                                             VMPTRSTm)>;
 
-def ADLPWriteResGroup61 : SchedWriteRes<[ADLPPort00_05]> {
+def ADLPWriteResGroup55 : SchedWriteRes<[ADLPPort00_05]> {
   let ResourceCycles = [2];
   let Latency = 2;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup61], (instrs FDECSTP)>;
+def : InstRW<[ADLPWriteResGroup55], (instrs FDECSTP)>;
 
-def ADLPWriteResGroup62 : SchedWriteRes<[ADLPPort02_03, ADLPPort05]> {
+def ADLPWriteResGroup56 : SchedWriteRes<[ADLPPort02_03, ADLPPort05]> {
   let ResourceCycles = [1, 2];
   let Latency = 11;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup62], (instregex "^FICOM(P?)(16|32)m$")>;
+def : InstRW<[ADLPWriteResGroup56], (instregex "^FICOM(P?)(16|32)m$")>;
 
-def ADLPWriteResGroup63 : SchedWriteRes<[ADLPPort00_05]>;
-def : InstRW<[ADLPWriteResGroup63], (instregex "^MMX_P(ADD|SUB)(B|D|Q|W)rr$")>;
-def : InstRW<[ADLPWriteResGroup63], (instrs FINCSTP,
+def ADLPWriteResGroup57 : SchedWriteRes<[ADLPPort00_05]>;
+def : InstRW<[ADLPWriteResGroup57], (instregex "^MMX_P(ADD|SUB)(B|D|Q|W)rr$")>;
+def : InstRW<[ADLPWriteResGroup57], (instrs FINCSTP,
                                             FNOP)>;
 
-def ADLPWriteResGroup64 : SchedWriteRes<[ADLPPort00, ADLPPort00_05, ADLPPort02_03]> {
+def ADLPWriteResGroup58 : SchedWriteRes<[ADLPPort00, ADLPPort00_05, ADLPPort02_03]> {
   let Latency = 7;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup64], (instrs FLDCW16m)>;
+def : InstRW<[ADLPWriteResGroup58], (instrs FLDCW16m)>;
 
-def ADLPWriteResGroup65 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06, ADLPPort00_05, ADLPPort00_06, ADLPPort02_03]> {
+def ADLPWriteResGroup59 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06, ADLPPort00_05, ADLPPort00_06, ADLPPort02_03]> {
   let ResourceCycles = [2, 39, 5, 10, 8];
   let Latency = 62;
   let NumMicroOps = 64;
 }
-def : InstRW<[ADLPWriteResGroup65], (instrs FLDENVm)>;
+def : InstRW<[ADLPWriteResGroup59], (instrs FLDENVm)>;
 
-def ADLPWriteResGroup66 : SchedWriteRes<[ADLPPort00_01_05_06]> {
+def ADLPWriteResGroup60 : SchedWriteRes<[ADLPPort00_01_05_06]> {
   let ResourceCycles = [4];
   let Latency = 4;
   let NumMicroOps = 4;
 }
-def : InstRW<[ADLPWriteResGroup66], (instrs FNCLEX)>;
+def : InstRW<[ADLPWriteResGroup60], (instrs FNCLEX)>;
 
-def ADLPWriteResGroup67 : SchedWriteRes<[ADLPPort00_01_05_06, ADLPPort00_05, ADLPPort05]> {
+def ADLPWriteResGroup61 : SchedWriteRes<[ADLPPort00_01_05_06, ADLPPort00_05, ADLPPort05]> {
   let ResourceCycles = [6, 3, 6];
   let Latency = 75;
   let NumMicroOps = 15;
 }
-def : InstRW<[ADLPWriteResGroup67], (instrs FNINIT)>;
+def : InstRW<[ADLPWriteResGroup61], (instrs FNINIT)>;
 
-def ADLPWriteResGroup68 : SchedWriteRes<[ADLPPort02_03_07, ADLPPort04, ADLPPort06]> {
+def ADLPWriteResGroup62 : SchedWriteRes<[ADLPPort02_03_07, ADLPPort04, ADLPPort06]> {
   let Latency = 2;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup68], (instrs FNSTCW16m)>;
+def : InstRW<[ADLPWriteResGroup62], (instrs FNSTCW16m)>;
 
-def ADLPWriteResGroup69 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06]> {
+def ADLPWriteResGroup63 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06]> {
   let Latency = 3;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup69], (instrs FNSTSW16r)>;
+def : InstRW<[ADLPWriteResGroup63], (instrs FNSTSW16r)>;
 
-def ADLPWriteResGroup70 : SchedWriteRes<[ADLPPort00, ADLPPort02_03_07, ADLPPort04]> {
+def ADLPWriteResGroup64 : SchedWriteRes<[ADLPPort00, ADLPPort02_03_07, ADLPPort04]> {
   let Latency = 3;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup70], (instrs FNSTSWm)>;
+def : InstRW<[ADLPWriteResGroup64], (instrs FNSTSWm)>;
 
-def ADLPWriteResGroup71 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06, ADLPPort00_06, ADLPPort01, ADLPPort02_03_07, ADLPPort04, ADLPPort05, ADLPPort06]> {
+def ADLPWriteResGroup65 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06, ADLPPort00_06, ADLPPort01, ADLPPort02_03_07, ADLPPort04, ADLPPort05, ADLPPort06]> {
   let ResourceCycles = [9, 30, 21, 1, 11, 11, 16, 1];
   let Latency = 106;
   let NumMicroOps = 100;
 }
-def : InstRW<[ADLPWriteResGroup71], (instrs FSTENVm)>;
+def : InstRW<[ADLPWriteResGroup65], (instrs FSTENVm)>;
 
-def ADLPWriteResGroup72 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06, ADLPPort00_05, ADLPPort00_06, ADLPPort01_05, ADLPPort02_03, ADLPPort06]> {
+def ADLPWriteResGroup66 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06, ADLPPort00_05, ADLPPort00_06, ADLPPort01_05, ADLPPort02_03, ADLPPort06]> {
   let ResourceCycles = [4, 47, 1, 2, 1, 33, 2];
   let Latency = 63;
   let NumMicroOps = 90;
 }
-def : InstRW<[ADLPWriteResGroup72], (instrs FXRSTOR)>;
+def : InstRW<[ADLPWriteResGroup66], (instrs FXRSTOR)>;
 
-def ADLPWriteResGroup73 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06, ADLPPort00_05, ADLPPort00_06, ADLPPort01_05, ADLPPort02_03, ADLPPort06]> {
+def ADLPWriteResGroup67 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06, ADLPPort00_05, ADLPPort00_06, ADLPPort01_05, ADLPPort02_03, ADLPPort06]> {
   let ResourceCycles = [4, 45, 1, 2, 1, 31, 4];
   let Latency = 63;
   let NumMicroOps = 88;
 }
-def : InstRW<[ADLPWriteResGroup73], (instrs FXRSTOR64)>;
+def : InstRW<[ADLPWriteResGroup67], (instrs FXRSTOR64)>;
 
-def ADLPWriteResGroup74 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup68 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [2, 5, 10, 10, 2, 38, 5, 38];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 110;
 }
-def : InstRW<[ADLPWriteResGroup74], (instregex "^FXSAVE((64)?)$")>;
+def : InstRW<[ADLPWriteResGroup68], (instregex "^FXSAVE((64)?)$")>;
 
-def ADLPWriteResGroup75 : SchedWriteRes<[ADLPPort00_01, ADLPPort02_03_11]> {
+def ADLPWriteResGroup69 : SchedWriteRes<[ADLPPort00_01, ADLPPort02_03_11]> {
   let Latency = 12;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup75], (instregex "^VPDP(BU|WS)SD(S?)Yrm$")>;
-def : InstRW<[ADLPWriteResGroup75, ReadAfterVecXLd], (instregex "^GF2P8AFFINE((INV)?)QBrmi$")>;
-def : InstRW<[ADLPWriteResGroup75, ReadAfterVecXLd], (instrs GF2P8MULBrm)>;
-def : InstRW<[ADLPWriteResGroup75, ReadAfterVecYLd], (instregex "^VM(AX|IN)CP(D|S)Yrm$")>;
+def : InstRW<[ADLPWriteResGroup69, ReadAfterVecXLd], (instregex "^(V?)GF2P8AFFINE((INV)?)QBrmi$",
+                                                                "^(V?)GF2P8MULBrm$")>;
+def : InstRW<[ADLPWriteResGroup69, ReadAfterVecYLd], (instregex "^VGF2P8AFFINE((INV)?)QBYrmi$")>;
+def : InstRW<[ADLPWriteResGroup69, ReadAfterVecYLd], (instrs VGF2P8MULBYrm)>;
 
-def ADLPWriteResGroup76 : SchedWriteRes<[ADLPPort00_01]> {
+def ADLPWriteResGroup70 : SchedWriteRes<[ADLPPort00_01]> {
   let Latency = 5;
 }
-def : InstRW<[ADLPWriteResGroup76], (instrs GF2P8MULBrr)>;
-
-def ADLPWriteResGroup77 : SchedWriteRes<[ADLPPort01_05, ADLPPort02_03_11, ADLPPort05]> {
-  let ResourceCycles = [1, 1, 2];
-  let Latency = 12;
-  let NumMicroOps = 4;
-}
-def : InstRW<[ADLPWriteResGroup77, ReadAfterVecXLd], (instregex "^(V?)H(ADD|SUB)PSrm$")>;
+def : InstRW<[ADLPWriteResGroup70], (instregex "^(V?)GF2P8MULBrr$")>;
+def : InstRW<[ADLPWriteResGroup70], (instrs VGF2P8MULBYrr)>;
 
-def ADLPWriteResGroup78 : SchedWriteRes<[ADLPPort01_05, ADLPPort05]> {
-  let ResourceCycles = [1, 2];
-  let Latency = 6;
-  let NumMicroOps = 3;
-}
-def : InstRW<[ADLPWriteResGroup78], (instregex "^H(ADD|SUB)PSrr$")>;
-
-def ADLPWriteResGroup79 : SchedWriteRes<[ADLPPort02_03, ADLPPort05]> {
-  let Latency = 10;
-  let NumMicroOps = 2;
-}
-def : InstRW<[ADLPWriteResGroup79], (instregex "^ILD_F(16|32|64)m$")>;
-
-def ADLPWriteResGroup80 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup71 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [7, 5, 26, 19, 2, 7, 21];
   let Latency = 35;
   let NumMicroOps = 87;
 }
-def : InstRW<[ADLPWriteResGroup80], (instrs IN16ri)>;
+def : InstRW<[ADLPWriteResGroup71], (instrs IN16ri)>;
 
-def ADLPWriteResGroup81 : SchedWriteRes<[ADLPPort00, ADLPPort00_01, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup72 : SchedWriteRes<[ADLPPort00, ADLPPort00_01, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [7, 1, 4, 26, 19, 3, 7, 20];
   let Latency = 35;
   let NumMicroOps = 87;
 }
-def : InstRW<[ADLPWriteResGroup81], (instrs IN16rr)>;
+def : InstRW<[ADLPWriteResGroup72], (instrs IN16rr)>;
 
-def ADLPWriteResGroup82 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup73 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [7, 6, 28, 21, 2, 10, 20];
   let Latency = 35;
   let NumMicroOps = 94;
 }
-def : InstRW<[ADLPWriteResGroup82], (instrs IN32ri)>;
+def : InstRW<[ADLPWriteResGroup73], (instrs IN32ri)>;
 
-def ADLPWriteResGroup83 : SchedWriteRes<[ADLPPort00_01_05_06_10]> {
-  let NumMicroOps = 2;
+def ADLPWriteResGroup74 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort02_03_11, ADLPPort05]> {
+  let ResourceCycles = [7, 9, 28, 21, 2, 11, 21];
+  let NumMicroOps = 99;
 }
-def : InstRW<[ADLPWriteResGroup83], (instrs IN32rr)>;
+def : InstRW<[ADLPWriteResGroup74], (instrs IN32rr)>;
 
-def ADLPWriteResGroup84 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup75 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [7, 6, 25, 19, 2, 8, 20];
   let Latency = 35;
   let NumMicroOps = 87;
 }
-def : InstRW<[ADLPWriteResGroup84], (instrs IN8ri)>;
+def : InstRW<[ADLPWriteResGroup75], (instrs IN8ri)>;
 
-def ADLPWriteResGroup85 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup76 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [7, 6, 25, 19, 2, 7, 20];
   let Latency = 35;
   let NumMicroOps = 86;
 }
-def : InstRW<[ADLPWriteResGroup85], (instrs IN8rr)>;
+def : InstRW<[ADLPWriteResGroup76], (instrs IN8rr)>;
 
-def ADLPWriteResGroup86 : SchedWriteRes<[ADLPPort00_06]> {
+def ADLPWriteResGroup77 : SchedWriteRes<[ADLPPort00_06]> {
   let NumMicroOps = 4;
 }
-def : InstRW<[ADLPWriteResGroup86], (instrs INC16r_alt)>;
+def : InstRW<[ADLPWriteResGroup77], (instrs INC16r_alt)>;
 
-def ADLPWriteResGroup87 : SchedWriteRes<[ADLPPort02_03_11]> {
+def ADLPWriteResGroup78 : SchedWriteRes<[ADLPPort02_03_11]> {
   let Latency = 7;
 }
-def : InstRW<[ADLPWriteResGroup87], (instregex "^(V?)MOV(D|SH|SL)DUPrm$",
-                                               "^(V?)MOVS(D|S)rm_alt$",
-                                               "^VBROADCAST(F|I)128$",
-                                               "^VBROADCASTS(D|S)Yrm$",
-                                               "^VMOV(D|SH|SL)DUPYrm$",
-                                               "^VPBROADCAST(D|Q)Yrm$")>;
-def : InstRW<[ADLPWriteResGroup87], (instrs INC32r_alt)>;
+def : InstRW<[ADLPWriteResGroup78], (instregex "^(V?)MOV(D|SH|SL)DUPrm$",
+                                               "^VPBROADCAST(D|Q)rm$")>;
+def : InstRW<[ADLPWriteResGroup78], (instrs INC32r_alt,
+                                            VBROADCASTSSrm)>;
 
-def ADLPWriteResGroup88 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup79 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [7, 6, 24, 17, 8, 1, 19, 1];
   let Latency = 20;
   let NumMicroOps = 83;
 }
-def : InstRW<[ADLPWriteResGroup88], (instrs INSB)>;
+def : InstRW<[ADLPWriteResGroup79], (instrs INSB)>;
 
-def ADLPWriteResGroup89 : SchedWriteRes<[ADLPPort02_03_11, ADLPPort05]> {
-  let Latency = 8;
-  let NumMicroOps = 2;
-}
-def : InstRW<[ADLPWriteResGroup89], (instregex "^VPBROADCAST(B|W)Yrm$")>;
-def : InstRW<[ADLPWriteResGroup89, ReadAfterVecXLd], (instrs INSERTPSrm)>;
-def : InstRW<[ADLPWriteResGroup89, ReadAfterVecYLd], (instrs VPALIGNRYrmi)>;
-
-def ADLPWriteResGroup90 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05, ADLPPort00_01_05_06_10, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup80 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05, ADLPPort00_01_05_06_10, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [7, 1, 5, 1, 27, 17, 11, 1, 21, 1];
   let Latency = 20;
   let NumMicroOps = 92;
 }
-def : InstRW<[ADLPWriteResGroup90], (instrs INSL)>;
+def : InstRW<[ADLPWriteResGroup80], (instrs INSL)>;
 
-def ADLPWriteResGroup91 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05, ADLPPort00_01_05_06_10, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup81 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05, ADLPPort00_01_05_06_10, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [7, 1, 4, 1, 25, 17, 1, 9, 1, 19, 1];
   let Latency = 20;
   let NumMicroOps = 86;
 }
-def : InstRW<[ADLPWriteResGroup91], (instrs INSW)>;
+def : InstRW<[ADLPWriteResGroup81], (instrs INSW)>;
 
-def ADLPWriteResGroup92 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup82 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [5, 4, 8, 6, 2, 5, 7, 5];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 42;
 }
-def : InstRW<[ADLPWriteResGroup92], (instrs INVLPG)>;
+def : InstRW<[ADLPWriteResGroup82], (instrs INVLPG)>;
 
-def ADLPWriteResGroup93 : SchedWriteRes<[ADLPPort00_01_05_06, ADLPPort00_06]> {
-  let Latency = 2;
-  let NumMicroOps = 2;
+def ADLPWriteResGroup83 : SchedWriteRes<[ADLPPort02_03_07, ADLPPort04, ADLPPort05]> {
+  let Latency = 4;
+  let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup93], (instrs JCXZ)>;
+def : InstRW<[ADLPWriteResGroup83], (instregex "^IST(T?)_FP(16|32|64)m$",
+                                               "^IST_F(16|32)m$")>;
 
-def ADLPWriteResGroup94 : SchedWriteRes<[ADLPPort02_03_11, ADLPPort06]> {
-  let Latency = 6;
+def ADLPWriteResGroup84 : SchedWriteRes<[ADLPPort00_01_05_06, ADLPPort00_06]> {
+  let Latency = 2;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup94], (instrs JMP64m)>;
+def : InstRW<[ADLPWriteResGroup84], (instrs JCXZ)>;
 
-def ADLPWriteResGroup95 : SchedWriteRes<[ADLPPort06]>;
-def : InstRW<[ADLPWriteResGroup95], (instregex "^JMP64r((_REX)?)$")>;
+def ADLPWriteResGroup85 : SchedWriteRes<[ADLPPort06]>;
+def : InstRW<[ADLPWriteResGroup85], (instrs JMP64r_REX)>;
 
-def ADLPWriteResGroup96 : SchedWriteRes<[ADLPPort06]> {
+def ADLPWriteResGroup86 : SchedWriteRes<[]> {
   let Latency = 0;
+  let NumMicroOps = 0;
 }
-def : InstRW<[ADLPWriteResGroup96], (instregex "^JMP_(1|4)$")>;
+def : InstRW<[ADLPWriteResGroup86], (instregex "^JMP_(1|4)$")>;
+def : InstRW<[ADLPWriteResGroup86], (instrs VZEROUPPER)>;
 
-def ADLPWriteResGroup97 : SchedWriteRes<[ADLPPort00, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup87 : SchedWriteRes<[ADLPPort00, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [8, 2, 14, 3, 1];
   let Latency = 198;
   let NumMicroOps = 81;
 }
-def : InstRW<[ADLPWriteResGroup97], (instrs LAR16rm)>;
+def : InstRW<[ADLPWriteResGroup87], (instrs LAR16rm)>;
 
-def ADLPWriteResGroup98 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup88 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [1, 3, 1, 8, 5, 1, 2, 1];
   let Latency = 66;
   let NumMicroOps = 22;
 }
-def : InstRW<[ADLPWriteResGroup98], (instrs LAR16rr)>;
+def : InstRW<[ADLPWriteResGroup88], (instrs LAR16rr)>;
 
-def ADLPWriteResGroup99 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup89 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [1, 2, 2, 9, 5, 3, 1];
   let Latency = 71;
   let NumMicroOps = 85;
 }
-def : InstRW<[ADLPWriteResGroup99], (instrs LAR32rm)>;
+def : InstRW<[ADLPWriteResGroup89], (instrs LAR32rm)>;
 
-def ADLPWriteResGroup100 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup90 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [1, 3, 1, 8, 5, 1, 2, 1];
   let Latency = 65;
   let NumMicroOps = 22;
 }
-def : InstRW<[ADLPWriteResGroup100], (instregex "^LAR(32|64)rr$")>;
+def : InstRW<[ADLPWriteResGroup90], (instregex "^LAR(32|64)rr$")>;
 
-def ADLPWriteResGroup101 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup91 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [1, 2, 2, 9, 5, 3, 1];
   let Latency = 71;
   let NumMicroOps = 87;
 }
-def : InstRW<[ADLPWriteResGroup101], (instrs LAR64rm)>;
+def : InstRW<[ADLPWriteResGroup91], (instrs LAR64rm)>;
 
-def ADLPWriteResGroup102 : SchedWriteRes<[ADLPPort02_03]> {
+def ADLPWriteResGroup92 : SchedWriteRes<[ADLPPort02_03]> {
   let Latency = 7;
 }
-def : InstRW<[ADLPWriteResGroup102], (instregex "^LD_F(32|64|80)m$")>;
+def : InstRW<[ADLPWriteResGroup92], (instregex "^LD_F(32|64|80)m$")>;
 
-def ADLPWriteResGroup103 : SchedWriteRes<[ADLPPort00_01_05_06_10]> {
-  let ResourceCycles = [2];
+def ADLPWriteResGroup93 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort01]> {
   let Latency = 2;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup103], (instrs LEA16r)>;
+def : InstRW<[ADLPWriteResGroup93], (instrs LEA16r)>;
 
-def ADLPWriteResGroup104 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11]> {
+def ADLPWriteResGroup94 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11]> {
   let ResourceCycles = [3, 1];
   let Latency = 6;
   let NumMicroOps = 4;
 }
-def : InstRW<[ADLPWriteResGroup104], (instregex "^LODS(B|W)$",
-                                                "^SCAS(B|L|Q|W)$")>;
-def : InstRW<[ADLPWriteResGroup104], (instrs LEAVE)>;
+def : InstRW<[ADLPWriteResGroup94], (instregex "^LODS(B|W)$",
+                                               "^SCAS(B|L|Q|W)$")>;
+def : InstRW<[ADLPWriteResGroup94], (instrs LEAVE)>;
 
-def ADLPWriteResGroup105 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11]> {
+def ADLPWriteResGroup95 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11]> {
   let ResourceCycles = [2, 1];
   let Latency = 6;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup105], (instregex "^POP16r((mr)?)$")>;
-def : InstRW<[ADLPWriteResGroup105], (instrs LEAVE64)>;
+def : InstRW<[ADLPWriteResGroup95], (instrs LEAVE64)>;
 
-def ADLPWriteResGroup106 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup96 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
   let ResourceCycles = [1, 2, 4, 3, 2, 1, 1];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 14;
 }
-def : InstRW<[ADLPWriteResGroup106], (instrs LGDT64m)>;
+def : InstRW<[ADLPWriteResGroup96], (instrs LGDT64m)>;
 
-def ADLPWriteResGroup107 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup97 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
   let ResourceCycles = [1, 1, 5, 3, 2, 1, 1];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 14;
 }
-def : InstRW<[ADLPWriteResGroup107], (instrs LIDT64m)>;
+def : InstRW<[ADLPWriteResGroup97], (instrs LIDT64m)>;
 
-def ADLPWriteResGroup108 : SchedWriteRes<[ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup98 : SchedWriteRes<[ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
   let ResourceCycles = [5, 3, 2, 1, 1];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 12;
 }
-def : InstRW<[ADLPWriteResGroup108], (instrs LLDT16m)>;
+def : InstRW<[ADLPWriteResGroup98], (instrs LLDT16m)>;
 
-def ADLPWriteResGroup109 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup99 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
   let ResourceCycles = [1, 4, 3, 1, 1, 1];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 11;
 }
-def : InstRW<[ADLPWriteResGroup109], (instrs LLDT16r)>;
+def : InstRW<[ADLPWriteResGroup99], (instrs LLDT16r)>;
 
-def ADLPWriteResGroup110 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup100 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [1, 1, 2, 8, 3, 1, 2, 7, 2];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 27;
 }
-def : InstRW<[ADLPWriteResGroup110], (instrs LMSW16m)>;
+def : InstRW<[ADLPWriteResGroup100], (instrs LMSW16m)>;
 
-def ADLPWriteResGroup111 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup101 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [5, 7, 1, 2, 5, 2];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 22;
 }
-def : InstRW<[ADLPWriteResGroup111], (instrs LMSW16r)>;
+def : InstRW<[ADLPWriteResGroup101], (instrs LMSW16r)>;
 
-def ADLPWriteResGroup112 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11]> {
+def ADLPWriteResGroup102 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11]> {
   let ResourceCycles = [2, 1];
   let Latency = 5;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup112], (instregex "^LODS(L|Q)$")>;
+def : InstRW<[ADLPWriteResGroup102], (instregex "^LODS(L|Q)$")>;
 
-def ADLPWriteResGroup113 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
+def ADLPWriteResGroup103 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
   let ResourceCycles = [2, 4, 1];
   let Latency = 3;
   let NumMicroOps = 7;
 }
-def : InstRW<[ADLPWriteResGroup113], (instrs LOOP)>;
+def : InstRW<[ADLPWriteResGroup103], (instrs LOOP)>;
 
-def ADLPWriteResGroup114 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
+def ADLPWriteResGroup104 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
   let ResourceCycles = [4, 6, 1];
   let Latency = 3;
   let NumMicroOps = 11;
 }
-def : InstRW<[ADLPWriteResGroup114], (instrs LOOPE)>;
+def : InstRW<[ADLPWriteResGroup104], (instrs LOOPE)>;
 
-def ADLPWriteResGroup115 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
+def ADLPWriteResGroup105 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
   let ResourceCycles = [4, 6, 1];
   let Latency = 2;
   let NumMicroOps = 11;
 }
-def : InstRW<[ADLPWriteResGroup115], (instrs LOOPNE)>;
+def : InstRW<[ADLPWriteResGroup105], (instrs LOOPNE)>;
 
-def ADLPWriteResGroup116 : SchedWriteRes<[ADLPPort00_01_05_06, ADLPPort02_03, ADLPPort06]> {
+def ADLPWriteResGroup106 : SchedWriteRes<[ADLPPort00_01_05_06, ADLPPort02_03, ADLPPort06]> {
   let Latency = 7;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup116], (instrs LRET64)>;
+def : InstRW<[ADLPWriteResGroup106], (instrs LRET64)>;
 
-def ADLPWriteResGroup117 : SchedWriteRes<[ADLPPort00, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup107 : SchedWriteRes<[ADLPPort00, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [1, 5, 3, 3, 1];
   let Latency = 70;
   let NumMicroOps = 13;
 }
-def : InstRW<[ADLPWriteResGroup117], (instregex "^LSL(16|32|64)rm$")>;
+def : InstRW<[ADLPWriteResGroup107], (instregex "^LSL(16|32|64)rm$")>;
 
-def ADLPWriteResGroup118 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup108 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [1, 4, 4, 3, 2, 1];
   let Latency = 63;
   let NumMicroOps = 15;
 }
-def : InstRW<[ADLPWriteResGroup118], (instregex "^LSL(16|32|64)rr$")>;
+def : InstRW<[ADLPWriteResGroup108], (instregex "^LSL(16|32|64)rr$")>;
 
-def ADLPWriteResGroup119 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
-  let Latency = 3;
-  let NumMicroOps = 2;
+def ADLPWriteResGroup109 : SchedWriteRes<[ADLPPort00_01, ADLPPort02_03_11, ADLPPort05]> {
+  let Latency = 24;
+  let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup119], (instrs MFENCE)>;
+def : InstRW<[ADLPWriteResGroup109], (instregex "^MMX_CVT(T?)PD2PIrm$")>;
 
-def ADLPWriteResGroup120 : SchedWriteRes<[ADLPPort00_01, ADLPPort05]> {
+def ADLPWriteResGroup110 : SchedWriteRes<[ADLPPort00_01, ADLPPort05]> {
   let Latency = 8;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup120], (instregex "^MMX_CVT(T?)PD2PIrr$")>;
+def : InstRW<[ADLPWriteResGroup110], (instregex "^MMX_CVT(T?)PD2PIrr$")>;
 
-def ADLPWriteResGroup121 : SchedWriteRes<[ADLPPort00_01, ADLPPort05]> {
+def ADLPWriteResGroup111 : SchedWriteRes<[ADLPPort00_01, ADLPPort05]> {
   let Latency = 6;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup121], (instrs MMX_CVTPI2PDrr)>;
+def : InstRW<[ADLPWriteResGroup111], (instrs MMX_CVTPI2PDrr)>;
 
-def ADLPWriteResGroup122 : SchedWriteRes<[ADLPPort00, ADLPPort00_01]> {
-  let Latency = 6;
+def ADLPWriteResGroup112 : SchedWriteRes<[ADLPPort00, ADLPPort00_01]> {
+  let Latency = 7;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup122], (instrs MMX_CVTPI2PSrr)>;
+def : InstRW<[ADLPWriteResGroup112], (instrs MMX_CVTPI2PSrr)>;
 
-def ADLPWriteResGroup123 : SchedWriteRes<[ADLPPort00, ADLPPort02_03_11]> {
-  let Latency = 11;
+def ADLPWriteResGroup113 : SchedWriteRes<[ADLPPort00, ADLPPort02_03_11]> {
+  let Latency = 13;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup123], (instregex "^MMX_CVT(T?)PS2PIrm$")>;
+def : InstRW<[ADLPWriteResGroup113], (instregex "^MMX_CVT(T?)PS2PIrm$")>;
 
-def ADLPWriteResGroup124 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05]> {
+def ADLPWriteResGroup114 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05]> {
   let Latency = 9;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup124], (instregex "^MMX_CVT(T?)PS2PIrr$")>;
+def : InstRW<[ADLPWriteResGroup114], (instregex "^MMX_CVT(T?)PS2PIrr$")>;
 
-def ADLPWriteResGroup125 : SchedWriteRes<[ADLPPort00, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup115 : SchedWriteRes<[ADLPPort00, ADLPPort04_09, ADLPPort07_08]> {
   let ResourceCycles = [2, 1, 1];
   let Latency = 12;
   let NumMicroOps = 4;
 }
-def : InstRW<[ADLPWriteResGroup125], (instregex "^MMX_MASKMOVQ((64)?)$")>;
+def : InstRW<[ADLPWriteResGroup115], (instregex "^MMX_MASKMOVQ((64)?)$")>;
 
-def ADLPWriteResGroup126 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort00_05]> {
+def ADLPWriteResGroup116 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
+  let Latency = 18;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup126], (instrs MMX_MOVDQ2Qrr)>;
+def : InstRW<[ADLPWriteResGroup116], (instrs MMX_MOVD64mr)>;
 
-def ADLPWriteResGroup127 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort00_05]> {
+def ADLPWriteResGroup117 : SchedWriteRes<[ADLPPort02_03_11]> {
+  let Latency = 8;
+}
+def : InstRW<[ADLPWriteResGroup117], (instregex "^MMX_MOV(D|Q)64rm$",
+                                                "^VBROADCAST(F|I)128$",
+                                                "^VBROADCASTS(D|S)Yrm$",
+                                                "^VMOV(D|SH|SL)DUPYrm$",
+                                                "^VPBROADCAST(D|Q)Yrm$")>;
+def : InstRW<[ADLPWriteResGroup117], (instrs MMX_MOVD64to64rm)>;
+
+def ADLPWriteResGroup118 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort00_05]> {
   let Latency = 3;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup127], (instrs MMX_MOVFR642Qrr)>;
+def : InstRW<[ADLPWriteResGroup118], (instregex "^MMX_MOV(DQ|FR64)2Qrr$")>;
 
-def ADLPWriteResGroup128 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05]> {
+def ADLPWriteResGroup119 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05]> {
   let Latency = 3;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup128], (instregex "^MMX_MOVQ2(DQ|FR64)rr$")>;
+def : InstRW<[ADLPWriteResGroup119], (instregex "^MMX_MOVQ2(DQ|FR64)rr$")>;
 
-def ADLPWriteResGroup129 : SchedWriteRes<[ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup120 : SchedWriteRes<[ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [1, 2];
-  let Latency = 10;
+  let Latency = 12;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup129, ReadAfterVecLd], (instregex "^MMX_PACKSS(DW|WB)rm$")>;
-def : InstRW<[ADLPWriteResGroup129, ReadAfterVecLd], (instrs MMX_PACKUSWBrm)>;
+def : InstRW<[ADLPWriteResGroup120, ReadAfterVecLd], (instregex "^MMX_PACKSS(DW|WB)rm$")>;
+def : InstRW<[ADLPWriteResGroup120, ReadAfterVecLd], (instrs MMX_PACKUSWBrm)>;
 
-def ADLPWriteResGroup130 : SchedWriteRes<[ADLPPort05]> {
+def ADLPWriteResGroup121 : SchedWriteRes<[ADLPPort05]> {
   let ResourceCycles = [2];
   let Latency = 4;
-  let NumMicroOps = 3;
+  let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup130], (instregex "^MMX_PACKSS(DW|WB)rr$")>;
-def : InstRW<[ADLPWriteResGroup130], (instrs MMX_PACKUSWBrr)>;
+def : InstRW<[ADLPWriteResGroup121], (instregex "^MMX_PACKSS(DW|WB)rr$")>;
+def : InstRW<[ADLPWriteResGroup121], (instrs MMX_PACKUSWBrr)>;
+def : InstRW<[ADLPWriteResGroup121, ReadDefault, ReadInt2Fpu], (instrs MMX_PINSRWrr)>;
 
-def ADLPWriteResGroup131 : SchedWriteRes<[ADLPPort00_05, ADLPPort02_03_11]> {
-  let Latency = 7;
+def ADLPWriteResGroup122 : SchedWriteRes<[ADLPPort00_05, ADLPPort02_03_11]> {
+  let Latency = 9;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup131, ReadAfterVecLd], (instregex "^MMX_P(ADD|SUB)(B|D|Q|W)rm$")>;
+def : InstRW<[ADLPWriteResGroup122, ReadAfterVecLd], (instregex "^MMX_P(ADD|SUB)(B|D|Q|W)rm$")>;
 
-def ADLPWriteResGroup132 : SchedWriteRes<[ADLPPort00, ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup123 : SchedWriteRes<[ADLPPort00, ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [1, 1, 2];
   let Latency = 11;
   let NumMicroOps = 4;
 }
-def : InstRW<[ADLPWriteResGroup132, ReadAfterVecLd], (instregex "^MMX_PH(ADD|SUB)SWrm$")>;
+def : InstRW<[ADLPWriteResGroup123, ReadAfterVecLd], (instregex "^MMX_PH(ADD|SUB)SWrm$")>;
 
-def ADLPWriteResGroup133 : SchedWriteRes<[ADLPPort00, ADLPPort05]> {
+def ADLPWriteResGroup124 : SchedWriteRes<[ADLPPort00, ADLPPort05]> {
   let ResourceCycles = [1, 2];
   let Latency = 3;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup133], (instregex "^MMX_PH(ADD|SUB)SWrr$")>;
-
-def ADLPWriteResGroup134 : SchedWriteRes<[ADLPPort02_03_11, ADLPPort05]> {
-  let Latency = 7;
-  let NumMicroOps = 2;
-}
-def : InstRW<[ADLPWriteResGroup134], (instregex "^VPBROADCAST(B|W)rm$")>;
-def : InstRW<[ADLPWriteResGroup134, ReadAfterLd], (instrs MMX_PINSRWrm)>;
-def : InstRW<[ADLPWriteResGroup134, ReadAfterVecXLd], (instregex "^(V?)PALIGNRrmi$")>;
+def : InstRW<[ADLPWriteResGroup124], (instregex "^MMX_PH(ADD|SUB)SWrr$")>;
 
-def ADLPWriteResGroup135 : SchedWriteRes<[ADLPPort05]> {
-  let ResourceCycles = [2];
-  let Latency = 2;
+def ADLPWriteResGroup125 : SchedWriteRes<[ADLPPort02_03_11, ADLPPort05]> {
+  let Latency = 9;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup135, ReadDefault, ReadInt2Fpu], (instrs MMX_PINSRWrr)>;
+def : InstRW<[ADLPWriteResGroup125], (instregex "^VPBROADCAST(B|W)Yrm$")>;
+def : InstRW<[ADLPWriteResGroup125, ReadAfterLd], (instrs MMX_PINSRWrm)>;
+def : InstRW<[ADLPWriteResGroup125, ReadAfterVecYLd], (instrs VPALIGNRYrmi)>;
 
-def ADLPWriteResGroup136 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11]> {
+def ADLPWriteResGroup126 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11]> {
   let Latency = 5;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup136], (instregex "^MOV16ao(16|32)$",
-                                                "^POP64r((mr)?)$")>;
+def : InstRW<[ADLPWriteResGroup126], (instregex "^MOV16ao(16|32|64)$")>;
 
-def ADLPWriteResGroup137 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
-  let NumMicroOps = 2;
-}
-def : InstRW<[ADLPWriteResGroup137], (instregex "^MOV(8|16|32)mi$",
-                                                "^MOV(8|16|32|64)mr$")>;
-def : InstRW<[ADLPWriteResGroup137], (instrs MOV64mi32)>;
-
-def ADLPWriteResGroup138 : SchedWriteRes<[ADLPPort01, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup127 : SchedWriteRes<[ADLPPort01, ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 12;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup138], (instregex "^PUSH(F|G)S(16|32)$")>;
-def : InstRW<[ADLPWriteResGroup138], (instrs MOV16ms,
+def : InstRW<[ADLPWriteResGroup127], (instregex "^PUSH(F|G)S(16|32)$")>;
+def : InstRW<[ADLPWriteResGroup127], (instrs MOV16ms,
                                              MOVBE32mr)>;
 
-def ADLPWriteResGroup139 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort04_09, ADLPPort07_08]> {
-  let Latency = 5;
-  let NumMicroOps = 3;
-}
-def : InstRW<[ADLPWriteResGroup139], (instregex "^MOV16o(16|32)a$")>;
-
-def ADLPWriteResGroup140 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort04_09, ADLPPort07_08]> {
-  let Latency = 2;
-  let NumMicroOps = 3;
-}
-def : InstRW<[ADLPWriteResGroup140], (instregex "^MOV(8|16|32|64)o64a$")>;
-
-def ADLPWriteResGroup141 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort01]> {
+def ADLPWriteResGroup128 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort01]> {
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup141], (instregex "^MOV(16|32|64)rs$",
+def : InstRW<[ADLPWriteResGroup128], (instregex "^MOV(16|32|64)rs$",
                                                 "^S(TR|LDT)16r$")>;
 
-def ADLPWriteResGroup142 : SchedWriteRes<[ADLPPort02_03_11]>;
-def : InstRW<[ADLPWriteResGroup142], (instregex "^MOV32ao(16|32)$")>;
+def ADLPWriteResGroup129 : SchedWriteRes<[ADLPPort02_03_11]>;
+def : InstRW<[ADLPWriteResGroup129], (instregex "^MOV32ao(16|32|64)$")>;
+def : InstRW<[ADLPWriteResGroup129], (instrs MOV64ao64)>;
 
-def ADLPWriteResGroup143 : SchedWriteRes<[ADLPPort02_03_11]> {
-  let Latency = 5;
-}
-def : InstRW<[ADLPWriteResGroup143], (instregex "^MOV(32|64)ao64$")>;
-def : InstRW<[ADLPWriteResGroup143], (instrs MOV64ao32)>;
-
-def ADLPWriteResGroup144 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup130 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort04_09, ADLPPort07_08]> {
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup144], (instregex "^MOV(8|32)o(16|32)a$",
-                                                "^PUSH64r((mr)?)$")>;
+def : InstRW<[ADLPWriteResGroup130], (instregex "^MOV(8|32)o(16|32)a$",
+                                                "^MOV(8|32|64)o64a$")>;
 
-def ADLPWriteResGroup145 : SchedWriteRes<[ADLPPort00_01_05_06_10]> {
+def ADLPWriteResGroup131 : SchedWriteRes<[ADLPPort00_01_05_06_10]> {
   let Latency = 0;
 }
-def : InstRW<[ADLPWriteResGroup145], (instregex "^MOV(32|64)rr((_REV)?)$",
+def : InstRW<[ADLPWriteResGroup131], (instregex "^MOV32rr((_REV)?)$",
                                                 "^MOVZX(32|64)rr8$")>;
-def : InstRW<[ADLPWriteResGroup145], (instrs MOVZX32rr8_NOREX)>;
+def : InstRW<[ADLPWriteResGroup131], (instrs MOVZX32rr8_NOREX)>;
+
+def ADLPWriteResGroup132 : SchedWriteRes<[ADLPPort02_03_11]> {
+  let Latency = 5;
+}
+def : InstRW<[ADLPWriteResGroup132], (instrs MOV64ao32)>;
 
-def ADLPWriteResGroup146 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_01_05, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup133 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_01_05, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [1, 2, 4, 16, 7, 2, 2, 12, 2];
   let Latency = 217;
   let NumMicroOps = 48;
 }
-def : InstRW<[ADLPWriteResGroup146], (instrs MOV64dr)>;
+def : InstRW<[ADLPWriteResGroup133], (instrs MOV64dr)>;
 
-def ADLPWriteResGroup147 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup134 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 12;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup147], (instregex "^PUSH(16|32)i8$",
-                                                "^PUSH32r((mr)?)$")>;
-def : InstRW<[ADLPWriteResGroup147], (instrs MOV64o32a,
-                                             MOVLPSmr,
-                                             PUSHi32)>;
+def : InstRW<[ADLPWriteResGroup134], (instrs MOV64o32a)>;
 
-def ADLPWriteResGroup148 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort05]> {
+def ADLPWriteResGroup135 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort05]> {
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup148], (instrs MOV64rc)>;
+def : InstRW<[ADLPWriteResGroup135], (instrs MOV64rc)>;
 
-def ADLPWriteResGroup149 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort05]> {
+def ADLPWriteResGroup136 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort05]> {
   let ResourceCycles = [3, 4, 8, 4, 2, 3];
   let Latency = 181;
   let NumMicroOps = 24;
 }
-def : InstRW<[ADLPWriteResGroup149], (instrs MOV64rd)>;
+def : InstRW<[ADLPWriteResGroup136], (instrs MOV64rd)>;
 
-def ADLPWriteResGroup150 : SchedWriteRes<[ADLPPort05]> {
-  let Latency = 3;
-}
-def : InstRW<[ADLPWriteResGroup150], (instregex "^(V?)MOV64toSDrr$",
-                                                "^(V?)MOVDI2SSrr$",
-                                                "^(V?)PACK(S|U)S(DW|WB)rr$",
-                                                "^(V?)PCMPGTQrr$",
-                                                "^VPACK(S|U)S(DW|WB)Yrr$")>;
-def : InstRW<[ADLPWriteResGroup150], (instrs VPCMPGTQYrr)>;
-
-def ADLPWriteResGroup151 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11]> {
+def ADLPWriteResGroup137 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11]> {
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup151], (instregex "^MOV8ao(16|32)$")>;
+def : InstRW<[ADLPWriteResGroup137], (instregex "^MOV8ao(16|32|64)$")>;
 
-def ADLPWriteResGroup152 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup138 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 13;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup152], (instrs MOV8mr_NOREX)>;
+def : InstRW<[ADLPWriteResGroup138], (instregex "^MOV8m(i|r)$")>;
+def : InstRW<[ADLPWriteResGroup138], (instrs MOV8mr_NOREX)>;
 
-def ADLPWriteResGroup153 : SchedWriteRes<[ADLPPort00_06, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup139 : SchedWriteRes<[ADLPPort00_06, ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 12;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup153], (instrs MOVBE16mr)>;
+def : InstRW<[ADLPWriteResGroup139], (instrs MOVBE16mr)>;
 
-def ADLPWriteResGroup154 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort02_03_11]> {
+def ADLPWriteResGroup140 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort02_03_11]> {
   let Latency = 7;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup154], (instrs MOVBE16rm)>;
+def : InstRW<[ADLPWriteResGroup140], (instrs MOVBE16rm)>;
 
-def ADLPWriteResGroup155 : SchedWriteRes<[ADLPPort01, ADLPPort02_03_11]> {
+def ADLPWriteResGroup141 : SchedWriteRes<[ADLPPort01, ADLPPort02_03_11]> {
   let Latency = 6;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup155], (instrs MOVBE32rm)>;
+def : InstRW<[ADLPWriteResGroup141], (instrs MOVBE32rm)>;
 
-def ADLPWriteResGroup156 : SchedWriteRes<[ADLPPort00_06, ADLPPort01, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup142 : SchedWriteRes<[ADLPPort00_06, ADLPPort01, ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 12;
   let NumMicroOps = 4;
 }
-def : InstRW<[ADLPWriteResGroup156], (instrs MOVBE64mr,
+def : InstRW<[ADLPWriteResGroup142], (instrs MOVBE64mr,
+                                             PUSHF16,
                                              SLDT16m,
                                              STRm)>;
 
-def ADLPWriteResGroup157 : SchedWriteRes<[ADLPPort00_06, ADLPPort01, ADLPPort02_03_11]> {
+def ADLPWriteResGroup143 : SchedWriteRes<[ADLPPort00_06, ADLPPort01, ADLPPort02_03_11]> {
   let Latency = 7;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup157], (instrs MOVBE64rm)>;
+def : InstRW<[ADLPWriteResGroup143], (instrs MOVBE64rm)>;
 
-def ADLPWriteResGroup158 : SchedWriteRes<[ADLPPort00_06, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup144 : SchedWriteRes<[ADLPPort00_06, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
   let NumMicroOps = 4;
 }
-def : InstRW<[ADLPWriteResGroup158], (instregex "^MOVDIR64B(16|32|64)$")>;
+def : InstRW<[ADLPWriteResGroup144], (instregex "^MOVDIR64B(16|32|64)$")>;
 
-def ADLPWriteResGroup159 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup145 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 511;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup159], (instrs MOVDIRI32)>;
+def : InstRW<[ADLPWriteResGroup145], (instrs MOVDIRI32)>;
 
-def ADLPWriteResGroup160 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup146 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 514;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup160], (instrs MOVDIRI64)>;
+def : InstRW<[ADLPWriteResGroup146], (instrs MOVDIRI64)>;
 
-def ADLPWriteResGroup161 : SchedWriteRes<[ADLPPort01_05, ADLPPort02_03_11]> {
-  let Latency = 7;
+def ADLPWriteResGroup147 : SchedWriteRes<[ADLPPort01_05, ADLPPort02_03_11]> {
+  let Latency = 8;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup161, ReadAfterVecXLd], (instregex "^(V?)MOVLP(D|S)rm$",
+def : InstRW<[ADLPWriteResGroup147, ReadAfterVecXLd], (instregex "^(V?)MOVLP(D|S)rm$",
                                                                  "^(V?)SHUFP(D|S)rmi$")>;
 
-def ADLPWriteResGroup162 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup148 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 512;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup162], (instrs MOVNTDQmr)>;
+def : InstRW<[ADLPWriteResGroup148], (instrs MOVNTDQmr)>;
 
-def ADLPWriteResGroup163 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup149 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 518;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup163], (instrs MOVNTImr)>;
+def : InstRW<[ADLPWriteResGroup149], (instrs MOVNTImr)>;
 
-def ADLPWriteResGroup164 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup150 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
   let ResourceCycles = [4, 1, 1, 1];
   let Latency = 8;
   let NumMicroOps = 7;
 }
-def : InstRW<[ADLPWriteResGroup164], (instrs MOVSB)>;
+def : InstRW<[ADLPWriteResGroup150], (instrs MOVSB)>;
 
-def ADLPWriteResGroup165 : SchedWriteRes<[ADLPPort00_01_05]>;
-def : InstRW<[ADLPWriteResGroup165], (instregex "^(V?)MOVS(D|S)rr((_REV)?)$",
+def ADLPWriteResGroup151 : SchedWriteRes<[ADLPPort00_01_05]>;
+def : InstRW<[ADLPWriteResGroup151], (instregex "^(V?)MOVS(D|S)rr((_REV)?)$",
                                                 "^(V?)P(ADD|SUB)(B|D|Q|W)rr$",
                                                 "^VP(ADD|SUB)(B|D|Q|W)Yrr$")>;
-def : InstRW<[ADLPWriteResGroup165], (instrs VPBLENDDrri)>;
+def : InstRW<[ADLPWriteResGroup151], (instrs VPBLENDDrri)>;
 
-def ADLPWriteResGroup166 : SchedWriteRes<[ADLPPort01_05]>;
-def : InstRW<[ADLPWriteResGroup166], (instregex "^(V?)MOVS(H|L)DUPrr$",
-                                                "^(V?)SHUFP(D|S)rri$",
-                                                "^VMOVS(H|L)DUPYrr$",
-                                                "^VSHUFP(D|S)Yrri$")>;
-def : InstRW<[ADLPWriteResGroup166], (instrs VPBLENDWYrri)>;
-
-def ADLPWriteResGroup167 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup152 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
   let ResourceCycles = [4, 1, 1, 1];
   let Latency = 7;
   let NumMicroOps = 7;
 }
-def : InstRW<[ADLPWriteResGroup167], (instregex "^MOVS(L|Q|W)$")>;
+def : InstRW<[ADLPWriteResGroup152], (instregex "^MOVS(L|Q|W)$")>;
 
-def ADLPWriteResGroup168 : SchedWriteRes<[ADLPPort02_03_11]> {
+def ADLPWriteResGroup153 : SchedWriteRes<[ADLPPort02_03_11]> {
   let Latency = 6;
 }
-def : InstRW<[ADLPWriteResGroup168], (instregex "^MOVSX(32|64)rm(8|16|32)$",
-                                                "^VPBROADCAST(D|Q)rm$")>;
-def : InstRW<[ADLPWriteResGroup168], (instrs MOVSX32rm8_NOREX,
-                                             VBROADCASTSSrm)>;
+def : InstRW<[ADLPWriteResGroup153], (instregex "^MOVSX(16|32|64)rm(16|32)$",
+                                                "^MOVSX(32|64)rm8$")>;
+def : InstRW<[ADLPWriteResGroup153], (instrs MOVSX32rm8_NOREX)>;
 
-def ADLPWriteResGroup169 : SchedWriteRes<[ADLPPort00, ADLPPort02_03]> {
+def ADLPWriteResGroup154 : SchedWriteRes<[ADLPPort01_05_10, ADLPPort02_03_11]> {
+  let Latency = 6;
+  let NumMicroOps = 2;
+}
+def : InstRW<[ADLPWriteResGroup154], (instrs MOVSX16rm8)>;
+
+def ADLPWriteResGroup155 : SchedWriteRes<[ADLPPort01_05_10]>;
+def : InstRW<[ADLPWriteResGroup155], (instregex "^MOVSX(16|32|64)rr(8|16|32)$")>;
+def : InstRW<[ADLPWriteResGroup155], (instrs MOVSX32rr8_NOREX)>;
+
+def ADLPWriteResGroup156 : SchedWriteRes<[ADLPPort00, ADLPPort02_03]> {
   let Latency = 11;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup169], (instregex "^MUL_F(32|64)m$")>;
+def : InstRW<[ADLPWriteResGroup156], (instregex "^MUL_F(32|64)m$")>;
 
-def ADLPWriteResGroup170 : SchedWriteRes<[ADLPPort00, ADLPPort02_03, ADLPPort05]> {
+def ADLPWriteResGroup157 : SchedWriteRes<[ADLPPort00, ADLPPort02_03, ADLPPort05]> {
   let Latency = 14;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup170], (instregex "^MUL_FI(16|32)m$")>;
+def : InstRW<[ADLPWriteResGroup157], (instregex "^MUL_FI(16|32)m$")>;
 
-def ADLPWriteResGroup171 : SchedWriteRes<[ADLPPort00]> {
+def ADLPWriteResGroup158 : SchedWriteRes<[ADLPPort00]> {
   let Latency = 4;
 }
-def : InstRW<[ADLPWriteResGroup171], (instregex "^MUL_F(P?)rST0$")>;
-def : InstRW<[ADLPWriteResGroup171], (instrs MUL_FST0r)>;
+def : InstRW<[ADLPWriteResGroup158], (instregex "^MUL_F(P?)rST0$")>;
+def : InstRW<[ADLPWriteResGroup158], (instrs MUL_FST0r)>;
 
-def ADLPWriteResGroup172 : SchedWriteRes<[ADLPPort00_01_05_06, ADLPPort05, ADLPPort06]> {
+def ADLPWriteResGroup159 : SchedWriteRes<[ADLPPort00_01_05_06, ADLPPort05, ADLPPort06]> {
   let ResourceCycles = [7, 1, 2];
   let Latency = 20;
   let NumMicroOps = 10;
 }
-def : InstRW<[ADLPWriteResGroup172], (instrs MWAITrr)>;
+def : InstRW<[ADLPWriteResGroup159], (instrs MWAITrr)>;
 
-def ADLPWriteResGroup173 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup160 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [6, 4, 1, 28, 15, 7, 1, 16, 1];
   let Latency = 35;
   let NumMicroOps = 79;
 }
-def : InstRW<[ADLPWriteResGroup173], (instrs OUT16ir)>;
+def : InstRW<[ADLPWriteResGroup160], (instrs OUT16ir)>;
 
-def ADLPWriteResGroup174 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup161 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [6, 6, 27, 15, 7, 1, 16, 1];
   let Latency = 35;
   let NumMicroOps = 79;
 }
-def : InstRW<[ADLPWriteResGroup174], (instrs OUT16rr)>;
+def : InstRW<[ADLPWriteResGroup161], (instrs OUT16rr)>;
 
-def ADLPWriteResGroup175 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup162 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [6, 4, 1, 30, 15, 9, 1, 18, 1];
   let Latency = 35;
   let NumMicroOps = 85;
 }
-def : InstRW<[ADLPWriteResGroup175], (instrs OUT32ir)>;
+def : InstRW<[ADLPWriteResGroup162], (instrs OUT32ir)>;
 
-def ADLPWriteResGroup176 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup163 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [6, 6, 29, 15, 9, 1, 18, 1];
   let Latency = 35;
   let NumMicroOps = 85;
 }
-def : InstRW<[ADLPWriteResGroup176], (instrs OUT32rr)>;
+def : InstRW<[ADLPWriteResGroup163], (instrs OUT32rr)>;
 
-def ADLPWriteResGroup177 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup164 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [5, 5, 1, 25, 15, 5, 1, 15, 1];
   let Latency = 35;
   let NumMicroOps = 73;
 }
-def : InstRW<[ADLPWriteResGroup177], (instrs OUT8ir)>;
+def : InstRW<[ADLPWriteResGroup164], (instrs OUT8ir)>;
 
-def ADLPWriteResGroup178 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup165 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [5, 5, 26, 15, 5, 1, 15, 1];
   let Latency = 35;
   let NumMicroOps = 73;
 }
-def : InstRW<[ADLPWriteResGroup178], (instrs OUT8rr)>;
+def : InstRW<[ADLPWriteResGroup165], (instrs OUT8rr)>;
 
-def ADLPWriteResGroup179 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup166 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [7, 6, 25, 16, 7, 1, 17, 1];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 80;
 }
-def : InstRW<[ADLPWriteResGroup179], (instrs OUTSB)>;
+def : InstRW<[ADLPWriteResGroup166], (instrs OUTSB)>;
 
-def ADLPWriteResGroup180 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup167 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [7, 6, 28, 16, 10, 1, 20, 1];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 89;
 }
-def : InstRW<[ADLPWriteResGroup180], (instrs OUTSL)>;
+def : InstRW<[ADLPWriteResGroup167], (instrs OUTSL)>;
 
-def ADLPWriteResGroup181 : SchedWriteRes<[ADLPPort00, ADLPPort00_01, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup168 : SchedWriteRes<[ADLPPort00, ADLPPort00_01, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [6, 1, 5, 27, 16, 8, 1, 18, 1];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 83;
 }
-def : InstRW<[ADLPWriteResGroup181], (instrs OUTSW)>;
+def : InstRW<[ADLPWriteResGroup168], (instrs OUTSW)>;
 
-def ADLPWriteResGroup182 : SchedWriteRes<[ADLPPort02_03_11, ADLPPort05]> {
-  let Latency = 9;
+def ADLPWriteResGroup169 : SchedWriteRes<[ADLPPort02_03_11, ADLPPort05]> {
+  let Latency = 10;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup182, ReadAfterVecXLd], (instregex "^(V?)PACK(S|U)S(DW|WB)rm$",
+def : InstRW<[ADLPWriteResGroup169, ReadAfterVecXLd], (instregex "^(V?)PACK(S|U)S(DW|WB)rm$",
                                                                  "^(V?)PCMPGTQrm$")>;
 
-def ADLPWriteResGroup183 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort02_03_11]> {
-  let Latency = 7;
+def ADLPWriteResGroup170 : SchedWriteRes<[ADLPPort05]> {
+  let Latency = 3;
+}
+def : InstRW<[ADLPWriteResGroup170], (instregex "^(V?)PACK(S|U)S(DW|WB)rr$",
+                                                "^(V?)PCMPGTQrr$",
+                                                "^VPACK(S|U)S(DW|WB)Yrr$")>;
+def : InstRW<[ADLPWriteResGroup170], (instrs VPCMPGTQYrr)>;
+
+def ADLPWriteResGroup171 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort02_03_11]> {
+  let Latency = 8;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup183, ReadAfterVecXLd], (instregex "^(V?)PADD(B|D|Q|W)rm$",
-                                                                 "^(V?)PSUB(B|D|W)rm$")>;
-def : InstRW<[ADLPWriteResGroup183, ReadAfterVecXLd], (instrs VPBLENDDrmi,
-                                                              VPSUBQrm)>;
+def : InstRW<[ADLPWriteResGroup171, ReadAfterVecXLd], (instregex "^(V?)P(ADD|SUB)(B|D|Q|W)rm$")>;
+def : InstRW<[ADLPWriteResGroup171, ReadAfterVecXLd], (instrs VPBLENDDrmi)>;
 
-def ADLPWriteResGroup184 : SchedWriteRes<[ADLPPort05]>;
-def : InstRW<[ADLPWriteResGroup184], (instregex "^(V?)PALIGNRrri$",
+def ADLPWriteResGroup172 : SchedWriteRes<[ADLPPort02_03_11, ADLPPort05]> {
+  let Latency = 8;
+  let NumMicroOps = 2;
+}
+def : InstRW<[ADLPWriteResGroup172], (instregex "^VPBROADCAST(B|W)rm$")>;
+def : InstRW<[ADLPWriteResGroup172, ReadAfterVecXLd], (instregex "^(V?)PALIGNRrmi$")>;
+
+def ADLPWriteResGroup173 : SchedWriteRes<[ADLPPort05]>;
+def : InstRW<[ADLPWriteResGroup173], (instregex "^(V?)PALIGNRrri$",
                                                 "^VPBROADCAST(B|D|Q|W)rr$")>;
-def : InstRW<[ADLPWriteResGroup184], (instrs VPALIGNRYrri)>;
+def : InstRW<[ADLPWriteResGroup173], (instrs VPALIGNRYrri)>;
 
-def ADLPWriteResGroup185 : SchedWriteRes<[ADLPPort00_06, ADLPPort05]> {
+def ADLPWriteResGroup174 : SchedWriteRes<[ADLPPort00_06, ADLPPort05]> {
   let Latency = 4;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup185], (instrs PAUSE)>;
+def : InstRW<[ADLPWriteResGroup174], (instrs PAUSE)>;
 
-def ADLPWriteResGroup186 : SchedWriteRes<[ADLPPort01, ADLPPort02_03_11]> {
+def ADLPWriteResGroup175 : SchedWriteRes<[ADLPPort01, ADLPPort02_03_11]> {
   let Latency = 8;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup186, ReadAfterLd], (instregex "^P(DEP|EXT)(32|64)rm$")>;
+def : InstRW<[ADLPWriteResGroup175, ReadAfterLd], (instregex "^P(DEP|EXT)(32|64)rm$")>;
 
-def ADLPWriteResGroup187 : SchedWriteRes<[ADLPPort01]> {
-  let Latency = 3;
-}
-def : InstRW<[ADLPWriteResGroup187], (instregex "^P(DEP|EXT)(32|64)rr$")>;
-
-def ADLPWriteResGroup188 : SchedWriteRes<[ADLPPort01_05, ADLPPort04_09, ADLPPort07_08]> {
-  let Latency = 19;
+def ADLPWriteResGroup176 : SchedWriteRes<[ADLPPort01_05, ADLPPort04_09, ADLPPort07_08]> {
+  let Latency = 12;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup188], (instrs PEXTRBmr)>;
+def : InstRW<[ADLPWriteResGroup176], (instregex "^(V?)PEXTR(D|Q)mr$")>;
 
-def ADLPWriteResGroup189 : SchedWriteRes<[ADLPPort00_01, ADLPPort01_05, ADLPPort02_03_11]> {
+def ADLPWriteResGroup177 : SchedWriteRes<[ADLPPort00_01, ADLPPort01_05, ADLPPort02_03_11]> {
   let ResourceCycles = [1, 2, 1];
-  let Latency = 8;
+  let Latency = 9;
   let NumMicroOps = 4;
 }
-def : InstRW<[ADLPWriteResGroup189, ReadAfterVecXLd], (instregex "^(V?)PH(ADD|SUB)SWrm$")>;
+def : InstRW<[ADLPWriteResGroup177, ReadAfterVecXLd], (instregex "^(V?)PH(ADD|SUB)SWrm$")>;
 
-def ADLPWriteResGroup190 : SchedWriteRes<[ADLPPort00_01, ADLPPort01_05]> {
+def ADLPWriteResGroup178 : SchedWriteRes<[ADLPPort00_01, ADLPPort01_05]> {
   let ResourceCycles = [1, 2];
   let Latency = 2;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup190], (instregex "^(V?)PH(ADD|SUB)SWrr$",
+def : InstRW<[ADLPWriteResGroup178], (instregex "^(V?)PH(ADD|SUB)SWrr$",
                                                 "^VPH(ADD|SUB)SWYrr$")>;
 
-def ADLPWriteResGroup191 : SchedWriteRes<[ADLPPort01_05, ADLPPort02_03_11]> {
-  let Latency = 8;
-  let NumMicroOps = 2;
-}
-def : InstRW<[ADLPWriteResGroup191, ReadAfterLd], (instrs PINSRWrm)>;
-def : InstRW<[ADLPWriteResGroup191, ReadAfterVecXLd], (instrs PUNPCKLQDQrm)>;
-def : InstRW<[ADLPWriteResGroup191, ReadAfterVecYLd], (instregex "^VSHUFP(D|S)Yrmi$")>;
-def : InstRW<[ADLPWriteResGroup191, ReadAfterVecYLd], (instrs VPBLENDWYrmi)>;
-
-def ADLPWriteResGroup192 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
-  let Latency = 12;
-  let NumMicroOps = 4;
-}
-def : InstRW<[ADLPWriteResGroup192], (instregex "^P(OP|USH)16rmm$")>;
-
-def ADLPWriteResGroup193 : SchedWriteRes<[ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup179 : SchedWriteRes<[ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 12;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup193], (instregex "^POP(32|64)rmm$")>;
-def : InstRW<[ADLPWriteResGroup193], (instrs PUSH32rmm)>;
+def : InstRW<[ADLPWriteResGroup179], (instregex "^POP(16|32|64)rmm$",
+                                                "^PUSH(16|32)rmm$")>;
 
-def ADLPWriteResGroup194 : SchedWriteRes<[ADLPPort02_03]> {
+def ADLPWriteResGroup180 : SchedWriteRes<[ADLPPort02_03]> {
   let Latency = 5;
 }
-def : InstRW<[ADLPWriteResGroup194], (instregex "^POPA(16|32)$")>;
-def : InstRW<[ADLPWriteResGroup194], (instrs POPF32)>;
+def : InstRW<[ADLPWriteResGroup180], (instregex "^POPA(16|32)$",
+                                                "^PREFETCHIT(0|1)$")>;
+def : InstRW<[ADLPWriteResGroup180], (instrs POPF32)>;
 
-def ADLPWriteResGroup195 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11]> {
+def ADLPWriteResGroup181 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11]> {
   let ResourceCycles = [6, 2, 1, 1];
   let Latency = 5;
   let NumMicroOps = 10;
 }
-def : InstRW<[ADLPWriteResGroup195], (instrs POPF16)>;
+def : InstRW<[ADLPWriteResGroup181], (instrs POPF16)>;
 
-def ADLPWriteResGroup196 : SchedWriteRes<[ADLPPort00_06, ADLPPort01, ADLPPort02_03_11]> {
+def ADLPWriteResGroup182 : SchedWriteRes<[ADLPPort00_06, ADLPPort01, ADLPPort02_03_11]> {
   let ResourceCycles = [2, 1, 1];
   let Latency = 5;
   let NumMicroOps = 7;
 }
-def : InstRW<[ADLPWriteResGroup196], (instrs POPF64)>;
+def : InstRW<[ADLPWriteResGroup182], (instrs POPF64)>;
 
-def ADLPWriteResGroup197 : SchedWriteRes<[ADLPPort02_03_11]> {
+def ADLPWriteResGroup183 : SchedWriteRes<[ADLPPort02_03_11]> {
   let Latency = 0;
 }
-def : InstRW<[ADLPWriteResGroup197], (instregex "^PREFETCHT(0|1|2)$")>;
-def : InstRW<[ADLPWriteResGroup197], (instrs PREFETCHNTA)>;
+def : InstRW<[ADLPWriteResGroup183], (instregex "^PREFETCHT(0|1|2)$")>;
+def : InstRW<[ADLPWriteResGroup183], (instrs PREFETCHNTA)>;
 
-def ADLPWriteResGroup198 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort02_03_11]> {
-  let Latency = 8;
-  let NumMicroOps = 2;
-}
-def : InstRW<[ADLPWriteResGroup198, ReadAfterVecXLd], (instrs PSUBQrm)>;
-def : InstRW<[ADLPWriteResGroup198, ReadAfterVecYLd], (instregex "^VP(ADD|SUB)(B|D|Q|W)Yrm$")>;
-
-def ADLPWriteResGroup199 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11, ADLPPort06]> {
+def ADLPWriteResGroup184 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11, ADLPPort06]> {
   let ResourceCycles = [1, 1, 2];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 4;
 }
-def : InstRW<[ADLPWriteResGroup199], (instregex "^PTWRITE((64)?)m$")>;
+def : InstRW<[ADLPWriteResGroup184], (instregex "^PTWRITE((64)?)m$")>;
 
-def ADLPWriteResGroup200 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort06]> {
+def ADLPWriteResGroup185 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort06]> {
   let ResourceCycles = [1, 2];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup200], (instrs PTWRITE64r)>;
+def : InstRW<[ADLPWriteResGroup185], (instrs PTWRITE64r)>;
 
-def ADLPWriteResGroup201 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort06]> {
+def ADLPWriteResGroup186 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort06]> {
   let ResourceCycles = [2, 2];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 4;
 }
-def : InstRW<[ADLPWriteResGroup201], (instrs PTWRITEr)>;
+def : InstRW<[ADLPWriteResGroup186], (instrs PTWRITEr)>;
 
-def ADLPWriteResGroup202 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort04_09, ADLPPort07_08]> {
-  let Latency = 12;
-  let NumMicroOps = 3;
+def ADLPWriteResGroup187 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
+  let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup202], (instregex "^PUSH16r((mr)?)$",
-                                                "^PUSH64i(8|32)$")>;
-def : InstRW<[ADLPWriteResGroup202], (instrs PUSHi16)>;
+def : InstRW<[ADLPWriteResGroup187], (instregex "^PUSH64r((mr)?)$")>;
 
-def ADLPWriteResGroup203 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
-  let NumMicroOps = 4;
+def ADLPWriteResGroup188 : SchedWriteRes<[ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
+  let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup203], (instrs PUSH64rmm)>;
+def : InstRW<[ADLPWriteResGroup188], (instrs PUSH64rmm)>;
 
-def ADLPWriteResGroup204 : SchedWriteRes<[ADLPPort02_03_07, ADLPPort04]>;
-def : InstRW<[ADLPWriteResGroup204], (instregex "^PUSHA(16|32)$",
+def ADLPWriteResGroup189 : SchedWriteRes<[ADLPPort02_03_07, ADLPPort04]>;
+def : InstRW<[ADLPWriteResGroup189], (instregex "^PUSHA(16|32)$",
                                                 "^ST_F(32|64)m$")>;
-def : InstRW<[ADLPWriteResGroup204], (instrs PUSHF32)>;
+def : InstRW<[ADLPWriteResGroup189], (instrs PUSHF32)>;
 
-def ADLPWriteResGroup205 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort04_09, ADLPPort07_08]> {
-  let Latency = 12;
-  let NumMicroOps = 5;
-}
-def : InstRW<[ADLPWriteResGroup205], (instrs PUSHF16)>;
-
-def ADLPWriteResGroup206 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup190 : SchedWriteRes<[ADLPPort00_06, ADLPPort01, ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 4;
-  let NumMicroOps = 5;
-}
-def : InstRW<[ADLPWriteResGroup206], (instrs PUSHF64)>;
-
-def ADLPWriteResGroup207 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort01, ADLPPort04_09, ADLPPort07_08]> {
   let NumMicroOps = 4;
 }
-def : InstRW<[ADLPWriteResGroup207], (instregex "^PUSH(F|G)S64$")>;
+def : InstRW<[ADLPWriteResGroup190], (instrs PUSHF64)>;
 
-def ADLPWriteResGroup208 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
-  let ResourceCycles = [2, 3, 2];
-  let Latency = 19;
-  let NumMicroOps = 7;
+def ADLPWriteResGroup191 : SchedWriteRes<[ADLPPort01, ADLPPort04_09, ADLPPort07_08]> {
+  let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup208, WriteRMW], (instregex "^RC(L|R)(16|32|64)mCL$")>;
+def : InstRW<[ADLPWriteResGroup191], (instregex "^PUSH(F|G)S64$")>;
 
-def ADLPWriteResGroup209 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
+def ADLPWriteResGroup192 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
   let ResourceCycles = [2, 3, 2];
   let Latency = 8;
   let NumMicroOps = 7;
 }
-def : InstRW<[ADLPWriteResGroup209], (instregex "^RC(L|R)(16|32|64)rCL$")>;
+def : InstRW<[ADLPWriteResGroup192], (instregex "^RC(L|R)(16|32|64)rCL$")>;
 
-def ADLPWriteResGroup210 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06]> {
+def ADLPWriteResGroup193 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06]> {
   let ResourceCycles = [1, 2];
   let Latency = 13;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup210, WriteRMW], (instregex "^RC(L|R)8m(1|i)$")>;
+def : InstRW<[ADLPWriteResGroup193, WriteRMW], (instregex "^RC(L|R)8m(1|i)$")>;
 
-def ADLPWriteResGroup211 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
+def ADLPWriteResGroup194 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
   let ResourceCycles = [1, 5, 2];
   let Latency = 20;
   let NumMicroOps = 8;
 }
-def : InstRW<[ADLPWriteResGroup211, WriteRMW], (instrs RCL8mCL)>;
+def : InstRW<[ADLPWriteResGroup194, WriteRMW], (instrs RCL8mCL)>;
 
-def ADLPWriteResGroup212 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
+def ADLPWriteResGroup195 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
   let ResourceCycles = [2, 5, 2];
   let Latency = 7;
   let NumMicroOps = 9;
 }
-def : InstRW<[ADLPWriteResGroup212], (instrs RCL8rCL)>;
+def : InstRW<[ADLPWriteResGroup195], (instrs RCL8rCL)>;
 
-def ADLPWriteResGroup213 : SchedWriteRes<[ADLPPort00, ADLPPort02_03_11]> {
-  let Latency = 10;
-  let NumMicroOps = 2;
-}
-def : InstRW<[ADLPWriteResGroup213, ReadAfterVecLd], (instregex "^(V?)R(CP|SQRT)SSm_Int$")>;
-
-def ADLPWriteResGroup214 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
+def ADLPWriteResGroup196 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
   let ResourceCycles = [2, 4, 3];
   let Latency = 20;
   let NumMicroOps = 9;
 }
-def : InstRW<[ADLPWriteResGroup214, WriteRMW], (instrs RCR8mCL)>;
+def : InstRW<[ADLPWriteResGroup196, WriteRMW], (instrs RCR8mCL)>;
 
-def ADLPWriteResGroup215 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
+def ADLPWriteResGroup197 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
   let ResourceCycles = [3, 4, 3];
   let Latency = 9;
   let NumMicroOps = 10;
 }
-def : InstRW<[ADLPWriteResGroup215], (instrs RCR8rCL)>;
+def : InstRW<[ADLPWriteResGroup197], (instrs RCR8rCL)>;
 
-def ADLPWriteResGroup216 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_01_05, ADLPPort00_05, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort01_05_10, ADLPPort05]> {
+def ADLPWriteResGroup198 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_01_05, ADLPPort00_05, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort01_05_10, ADLPPort05]> {
   let ResourceCycles = [1, 6, 1, 10, 20, 8, 5, 1, 2];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 54;
 }
-def : InstRW<[ADLPWriteResGroup216], (instrs RDMSR)>;
+def : InstRW<[ADLPWriteResGroup198], (instrs RDMSR)>;
 
-def ADLPWriteResGroup217 : SchedWriteRes<[ADLPPort01]> {
+def ADLPWriteResGroup199 : SchedWriteRes<[ADLPPort01]> {
   let Latency = AlderlakePModel.MaxLatency;
 }
-def : InstRW<[ADLPWriteResGroup217], (instrs RDPID64)>;
+def : InstRW<[ADLPWriteResGroup199], (instrs RDPID64)>;
 
-def ADLPWriteResGroup218 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
+def ADLPWriteResGroup200 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup218], (instrs RDPKRUr)>;
+def : InstRW<[ADLPWriteResGroup200], (instrs RDPKRUr)>;
 
-def ADLPWriteResGroup219 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort05]> {
+def ADLPWriteResGroup201 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort05]> {
   let ResourceCycles = [9, 6, 2, 1];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 18;
 }
-def : InstRW<[ADLPWriteResGroup219], (instrs RDPMC)>;
+def : InstRW<[ADLPWriteResGroup201], (instrs RDPMC)>;
 
-def ADLPWriteResGroup220 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup202 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [2, 3, 2, 5, 7, 3, 1, 2];
   let Latency = 1386;
   let NumMicroOps = 25;
 }
-def : InstRW<[ADLPWriteResGroup220], (instrs RDRAND16r)>;
+def : InstRW<[ADLPWriteResGroup202], (instrs RDRAND16r)>;
 
-def ADLPWriteResGroup221 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup203 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06_10, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [2, 3, 2, 5, 7, 3, 1, 2];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 25;
 }
-def : InstRW<[ADLPWriteResGroup221], (instregex "^RDRAND(32|64)r$")>;
+def : InstRW<[ADLPWriteResGroup203], (instregex "^RDRAND(32|64)r$")>;
 
-def ADLPWriteResGroup222 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup204 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [2, 3, 3, 5, 7, 1, 4];
   let Latency = 1381;
   let NumMicroOps = 25;
 }
-def : InstRW<[ADLPWriteResGroup222], (instrs RDSEED16r)>;
+def : InstRW<[ADLPWriteResGroup204], (instrs RDSEED16r)>;
 
-def ADLPWriteResGroup223 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup205 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [2, 3, 3, 5, 7, 1, 4];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 25;
 }
-def : InstRW<[ADLPWriteResGroup223], (instregex "^RDSEED(32|64)r$")>;
+def : InstRW<[ADLPWriteResGroup205], (instregex "^RDSEED(32|64)r$")>;
 
-def ADLPWriteResGroup224 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort05]> {
+def ADLPWriteResGroup206 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort05]> {
   let ResourceCycles = [5, 6, 3, 1];
   let Latency = 18;
   let NumMicroOps = 15;
 }
-def : InstRW<[ADLPWriteResGroup224], (instrs RDTSC)>;
+def : InstRW<[ADLPWriteResGroup206], (instrs RDTSC)>;
 
-def ADLPWriteResGroup225 : SchedWriteRes<[ADLPPort00, ADLPPort00_01, ADLPPort00_01_05, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort05]> {
+def ADLPWriteResGroup207 : SchedWriteRes<[ADLPPort00, ADLPPort00_01, ADLPPort00_01_05, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort05]> {
   let ResourceCycles = [2, 2, 1, 2, 7, 4, 3];
   let Latency = 42;
   let NumMicroOps = 21;
 }
-def : InstRW<[ADLPWriteResGroup225], (instrs RDTSCP)>;
+def : InstRW<[ADLPWriteResGroup207], (instrs RDTSCP)>;
 
-def ADLPWriteResGroup226 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11, ADLPPort06]> {
+def ADLPWriteResGroup208 : SchedWriteRes<[ADLPPort00_06, ADLPPort02_03_11]> {
   let Latency = 7;
-  let NumMicroOps = 3;
+  let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup226], (instrs RET64)>;
+def : InstRW<[ADLPWriteResGroup208], (instrs RET64)>;
 
-def ADLPWriteResGroup227 : SchedWriteRes<[ADLPPort00_06, ADLPPort02_03_11]> {
+def ADLPWriteResGroup209 : SchedWriteRes<[ADLPPort00_06, ADLPPort02_03_11]> {
   let ResourceCycles = [2, 1];
   let Latency = 6;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup227], (instregex "^RETI(16|32)$")>;
+def : InstRW<[ADLPWriteResGroup209], (instregex "^RETI(16|32|64)$")>;
 
-def ADLPWriteResGroup228 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11, ADLPPort06]> {
-  let ResourceCycles = [2, 1, 2];
-  let Latency = 6;
-  let NumMicroOps = 5;
-}
-def : InstRW<[ADLPWriteResGroup228], (instrs RETI64)>;
-
-def ADLPWriteResGroup229 : SchedWriteRes<[]>;
-def : InstRW<[ADLPWriteResGroup229], (instrs REX64_PREFIX)>;
+def ADLPWriteResGroup210 : SchedWriteRes<[]>;
+def : InstRW<[ADLPWriteResGroup210], (instrs REX64_PREFIX)>;
 
-def ADLPWriteResGroup230 : SchedWriteRes<[ADLPPort00_06]> {
+def ADLPWriteResGroup211 : SchedWriteRes<[ADLPPort00_06]> {
   let ResourceCycles = [2];
   let Latency = 12;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup230, WriteRMW], (instregex "^RO(L|R)(16|32|64)m1$")>;
+def : InstRW<[ADLPWriteResGroup211, WriteRMW], (instregex "^RO(L|R)(16|32|64)m(1|i|CL)$")>;
 
-def ADLPWriteResGroup231 : SchedWriteRes<[ADLPPort00_06]> {
+def ADLPWriteResGroup212 : SchedWriteRes<[ADLPPort00_06]> {
   let ResourceCycles = [2];
-  let Latency = 7;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup231, WriteRMW], (instregex "^RO(L|R)(8|16|32|64)mi$")>;
+def : InstRW<[ADLPWriteResGroup212], (instregex "^RO(L|R)(8|16|32|64)r(1|i)$")>;
 
-def ADLPWriteResGroup232 : SchedWriteRes<[ADLPPort00_06]> {
+def ADLPWriteResGroup213 : SchedWriteRes<[ADLPPort00_06]> {
   let ResourceCycles = [2];
+  let Latency = 13;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup232], (instregex "^RO(L|R)(8|16|32|64)r1$")>;
+def : InstRW<[ADLPWriteResGroup213, WriteRMW], (instregex "^RO(L|R)8m(1|i)$",
+                                                          "^(RO|SH)L8mCL$",
+                                                          "^(RO|SA|SH)R8mCL$")>;
 
-def ADLPWriteResGroup233 : SchedWriteRes<[ADLPPort00_06]> {
+def ADLPWriteResGroup214 : SchedWriteRes<[ADLPPort00_06]> {
   let ResourceCycles = [2];
-  let Latency = 13;
+  let Latency = 4;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup233, WriteRMW], (instregex "^RO(L|R)8m1$")>;
-
-def ADLPWriteResGroup234 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06]> {
-  let ResourceCycles = [2, 1];
-  let NumMicroOps = 3;
-}
-def : InstRW<[ADLPWriteResGroup234], (instrs ROR8ri)>;
+def : InstRW<[ADLPWriteResGroup214], (instrs SAHF)>;
 
-def ADLPWriteResGroup235 : SchedWriteRes<[ADLPPort00_01, ADLPPort02_03_11]> {
-  let ResourceCycles = [2, 1];
-  let Latency = 15;
-  let NumMicroOps = 3;
+def ADLPWriteResGroup215 : SchedWriteRes<[ADLPPort00_06]> {
+  let Latency = 13;
 }
-def : InstRW<[ADLPWriteResGroup235, ReadAfterVecXLd], (instregex "^(V?)ROUNDS(D|S)m$")>;
+def : InstRW<[ADLPWriteResGroup215, WriteRMW], (instregex "^S(A|H)R8m(1|i)$",
+                                                          "^SHL8m(1|i)$")>;
 
-def ADLPWriteResGroup236 : SchedWriteRes<[ADLPPort00_06]> {
-  let ResourceCycles = [2];
-  let Latency = 4;
+def ADLPWriteResGroup216 : SchedWriteRes<[ADLPPort00_06, ADLPPort02_03_11]> {
+  let Latency = 8;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup236], (instrs SAHF)>;
-
-def ADLPWriteResGroup237 : SchedWriteRes<[ADLPPort00_06]> {
-  let Latency = 12;
-}
-def : InstRW<[ADLPWriteResGroup237, WriteRMW], (instregex "^S(A|H)R(16|32|64)m1$",
-                                                          "^SHL(16|32|64)m1$")>;
+def : InstRW<[ADLPWriteResGroup216, ReadAfterLd, ReadDefault, ReadDefault, ReadDefault, ReadDefault, ReadDefault], (instregex "^S(A|H)RX(32|64)rm$",
+                                                                                                                              "^SHLX(32|64)rm$")>;
 
-def ADLPWriteResGroup238 : SchedWriteRes<[ADLPPort00_06]> {
-  let Latency = 13;
+def ADLPWriteResGroup217 : SchedWriteRes<[ADLPPort00_06]> {
+  let Latency = 3;
 }
-def : InstRW<[ADLPWriteResGroup238, WriteRMW], (instregex "^S(A|H)R8m1$")>;
-def : InstRW<[ADLPWriteResGroup238, WriteRMW], (instrs SHL8m1)>;
+def : InstRW<[ADLPWriteResGroup217], (instregex "^S(A|H)RX(32|64)rr$",
+                                                "^SHLX(32|64)rr$")>;
 
-def ADLPWriteResGroup239 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup218 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort04_09, ADLPPort07_08]> {
   let ResourceCycles = [2, 2, 1, 1, 1];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 7;
 }
-def : InstRW<[ADLPWriteResGroup239], (instrs SERIALIZE)>;
+def : InstRW<[ADLPWriteResGroup218], (instrs SERIALIZE)>;
 
-def ADLPWriteResGroup240 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup219 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 2;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup240], (instrs SFENCE)>;
+def : InstRW<[ADLPWriteResGroup219], (instrs SFENCE)>;
 
-def ADLPWriteResGroup241 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort01, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup220 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort01, ADLPPort04_09, ADLPPort07_08]> {
   let ResourceCycles = [1, 2, 2, 2];
   let Latency = 21;
   let NumMicroOps = 7;
 }
-def : InstRW<[ADLPWriteResGroup241], (instregex "^S(G|I)DT64m$")>;
+def : InstRW<[ADLPWriteResGroup220], (instregex "^S(G|I)DT64m$")>;
 
-def ADLPWriteResGroup242 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort02_03_11, ADLPPort05]> {
-  let Latency = 8;
+def ADLPWriteResGroup221 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort02_03_11, ADLPPort05]> {
+  let Latency = 9;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup242, ReadAfterVecXLd], (instrs SHA1MSG1rm)>;
+def : InstRW<[ADLPWriteResGroup221, ReadAfterVecXLd], (instrs SHA1MSG1rm)>;
 
-def ADLPWriteResGroup243 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort05]> {
+def ADLPWriteResGroup222 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort05]> {
   let Latency = 2;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup243], (instrs SHA1MSG1rr)>;
+def : InstRW<[ADLPWriteResGroup222], (instrs SHA1MSG1rr)>;
 
-def ADLPWriteResGroup244 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_01_05, ADLPPort00_06, ADLPPort01_05, ADLPPort02_03_11]> {
+def ADLPWriteResGroup223 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_01_05, ADLPPort00_06, ADLPPort01_05, ADLPPort02_03_11]> {
   let ResourceCycles = [2, 2, 1, 2, 1];
   let Latency = 13;
   let NumMicroOps = 8;
 }
-def : InstRW<[ADLPWriteResGroup244, ReadAfterVecXLd], (instrs SHA1MSG2rm)>;
+def : InstRW<[ADLPWriteResGroup223, ReadAfterVecXLd], (instrs SHA1MSG2rm)>;
 
-def ADLPWriteResGroup245 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_01_05, ADLPPort00_06, ADLPPort01_05]> {
+def ADLPWriteResGroup224 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_01_05, ADLPPort00_06, ADLPPort01_05]> {
   let ResourceCycles = [2, 2, 1, 2];
   let Latency = 6;
   let NumMicroOps = 7;
 }
-def : InstRW<[ADLPWriteResGroup245], (instrs SHA1MSG2rr)>;
+def : InstRW<[ADLPWriteResGroup224], (instrs SHA1MSG2rr)>;
 
-def ADLPWriteResGroup246 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_01_05, ADLPPort01_05, ADLPPort02_03_11]> {
+def ADLPWriteResGroup225 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_01_05, ADLPPort01_05, ADLPPort02_03_11]> {
   let Latency = 8;
   let NumMicroOps = 4;
 }
-def : InstRW<[ADLPWriteResGroup246, ReadAfterVecXLd], (instrs SHA1NEXTErm)>;
+def : InstRW<[ADLPWriteResGroup225, ReadAfterVecXLd], (instrs SHA1NEXTErm)>;
 
-def ADLPWriteResGroup247 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_01_05, ADLPPort01_05]> {
+def ADLPWriteResGroup226 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_01_05, ADLPPort01_05]> {
   let Latency = 3;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup247], (instrs SHA1NEXTErr)>;
+def : InstRW<[ADLPWriteResGroup226], (instrs SHA1NEXTErr)>;
 
-def ADLPWriteResGroup248 : SchedWriteRes<[ADLPPort02_03_11, ADLPPort05]> {
-  let Latency = 12;
+def ADLPWriteResGroup227 : SchedWriteRes<[ADLPPort02_03_11, ADLPPort05]> {
+  let Latency = 13;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup248, ReadAfterVecXLd], (instrs SHA1RNDS4rmi,
+def : InstRW<[ADLPWriteResGroup227, ReadAfterVecXLd], (instrs SHA1RNDS4rmi,
                                                               SHA256RNDS2rm)>;
 
-def ADLPWriteResGroup249 : SchedWriteRes<[ADLPPort05]> {
-  let Latency = 4;
+def ADLPWriteResGroup228 : SchedWriteRes<[ADLPPort05]> {
+  let Latency = 6;
 }
-def : InstRW<[ADLPWriteResGroup249], (instrs SHA1RNDS4rri,
+def : InstRW<[ADLPWriteResGroup228], (instrs SHA1RNDS4rri,
                                              SHA256RNDS2rr)>;
 
-def ADLPWriteResGroup250 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_01_05, ADLPPort00_06, ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup229 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_01_05, ADLPPort00_06, ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [3, 2, 1, 1, 1];
   let Latency = 12;
   let NumMicroOps = 8;
 }
-def : InstRW<[ADLPWriteResGroup250, ReadAfterVecXLd], (instrs SHA256MSG1rm)>;
+def : InstRW<[ADLPWriteResGroup229, ReadAfterVecXLd], (instrs SHA256MSG1rm)>;
 
-def ADLPWriteResGroup251 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_01_05, ADLPPort00_06, ADLPPort05]> {
+def ADLPWriteResGroup230 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_01_05, ADLPPort00_06, ADLPPort05]> {
   let ResourceCycles = [3, 2, 1, 1];
   let Latency = 5;
   let NumMicroOps = 7;
 }
-def : InstRW<[ADLPWriteResGroup251], (instrs SHA256MSG1rr)>;
+def : InstRW<[ADLPWriteResGroup230], (instrs SHA256MSG1rr)>;
 
-def ADLPWriteResGroup252 : SchedWriteRes<[ADLPPort02_03_11, ADLPPort05]> {
+def ADLPWriteResGroup231 : SchedWriteRes<[ADLPPort02_03_11, ADLPPort05]> {
   let ResourceCycles = [1, 2];
-  let Latency = 12;
+  let Latency = 13;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup252, ReadAfterVecXLd], (instrs SHA256MSG2rm)>;
+def : InstRW<[ADLPWriteResGroup231, ReadAfterVecXLd], (instrs SHA256MSG2rm)>;
 
-def ADLPWriteResGroup253 : SchedWriteRes<[ADLPPort05]> {
+def ADLPWriteResGroup232 : SchedWriteRes<[ADLPPort05]> {
   let ResourceCycles = [2];
   let Latency = 6;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup253], (instrs SHA256MSG2rr)>;
+def : InstRW<[ADLPWriteResGroup232], (instrs SHA256MSG2rr)>;
 
-def ADLPWriteResGroup254 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup233 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 13;
   let NumMicroOps = 5;
 }
-def : InstRW<[ADLPWriteResGroup254], (instrs SHRD16mri8)>;
-
-def ADLPWriteResGroup255 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
-  let Latency = 5;
-  let NumMicroOps = 3;
-}
-def : InstRW<[ADLPWriteResGroup255], (instrs SHRD16rrCL)>;
+def : InstRW<[ADLPWriteResGroup233], (instrs SHRD16mri8)>;
 
-def ADLPWriteResGroup256 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort01]> {
+def ADLPWriteResGroup234 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort01]> {
   let Latency = 6;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup256], (instregex "^SLDT(32|64)r$")>;
+def : InstRW<[ADLPWriteResGroup234], (instregex "^SLDT(32|64)r$")>;
 
-def ADLPWriteResGroup257 : SchedWriteRes<[ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
-  let Latency = 12;
-  let NumMicroOps = 3;
-}
-def : InstRW<[ADLPWriteResGroup257], (instrs SMSW16m)>;
-
-def ADLPWriteResGroup258 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort05]> {
+def ADLPWriteResGroup235 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort05]> {
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup258], (instrs SMSW16r)>;
+def : InstRW<[ADLPWriteResGroup235], (instrs SMSW16r)>;
 
-def ADLPWriteResGroup259 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort05]> {
+def ADLPWriteResGroup236 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort05]> {
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup259], (instregex "^SMSW(32|64)r$")>;
+def : InstRW<[ADLPWriteResGroup236], (instregex "^SMSW(32|64)r$")>;
 
-def ADLPWriteResGroup260 : SchedWriteRes<[ADLPPort00, ADLPPort02_03_11]> {
+def ADLPWriteResGroup237 : SchedWriteRes<[ADLPPort00, ADLPPort02_03_11]> {
   let Latency = 24;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup260, ReadAfterVecLd], (instregex "^(V?)SQRTSDm_Int$")>;
+def : InstRW<[ADLPWriteResGroup237, ReadAfterVecLd], (instregex "^(V?)SQRTSDm_Int$")>;
 
-def ADLPWriteResGroup261 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06]> {
+def ADLPWriteResGroup238 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06]> {
   let Latency = 6;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup261], (instrs STD)>;
+def : InstRW<[ADLPWriteResGroup238], (instrs STD)>;
 
-def ADLPWriteResGroup262 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
+def ADLPWriteResGroup239 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01]> {
   let ResourceCycles = [1, 4, 1];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 6;
 }
-def : InstRW<[ADLPWriteResGroup262], (instrs STI)>;
+def : InstRW<[ADLPWriteResGroup239], (instrs STI)>;
 
-def ADLPWriteResGroup263 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11]> {
-  let Latency = 6;
+def ADLPWriteResGroup240 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort04_09, ADLPPort07_08]> {
+  let ResourceCycles = [2, 1, 1];
+  let Latency = 8;
+  let NumMicroOps = 4;
 }
-def : InstRW<[ADLPWriteResGroup263], (instrs STOSB)>;
+def : InstRW<[ADLPWriteResGroup240], (instrs STOSB)>;
 
-def ADLPWriteResGroup264 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup241 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort04_09, ADLPPort07_08]> {
   let ResourceCycles = [2, 1, 1];
   let Latency = 7;
   let NumMicroOps = 4;
 }
-def : InstRW<[ADLPWriteResGroup264], (instregex "^STOS(L|Q|W)$")>;
+def : InstRW<[ADLPWriteResGroup241], (instregex "^STOS(L|Q|W)$")>;
 
-def ADLPWriteResGroup265 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort01]> {
+def ADLPWriteResGroup242 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort01]> {
   let Latency = 5;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup265], (instregex "^STR(32|64)r$")>;
+def : InstRW<[ADLPWriteResGroup242], (instregex "^STR(32|64)r$")>;
 
-def ADLPWriteResGroup266 : SchedWriteRes<[ADLPPort00]> {
+def ADLPWriteResGroup243 : SchedWriteRes<[ADLPPort00]> {
   let Latency = 2;
 }
-def : InstRW<[ADLPWriteResGroup266], (instregex "^(TST|XAM)_F$")>;
-def : InstRW<[ADLPWriteResGroup266], (instrs UCOM_FPPr)>;
+def : InstRW<[ADLPWriteResGroup243], (instregex "^(TST|XAM)_F$")>;
+def : InstRW<[ADLPWriteResGroup243], (instrs UCOM_FPPr)>;
 
-def ADLPWriteResGroup267 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort02_03_11]> {
+def ADLPWriteResGroup244 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort02_03_11]> {
   let ResourceCycles = [3, 1];
-  let Latency = 8;
+  let Latency = 9;
   let NumMicroOps = 4;
 }
-def : InstRW<[ADLPWriteResGroup267, ReadAfterVecXLd, ReadAfterVecXLd, ReadDefault, ReadDefault, ReadDefault, ReadDefault, ReadDefault], (instregex "^VBLENDVP(D|S)rm$")>;
-def : InstRW<[ADLPWriteResGroup267, ReadAfterVecXLd, ReadAfterVecXLd, ReadDefault, ReadDefault, ReadDefault, ReadDefault, ReadDefault], (instrs VPBLENDVBrm)>;
+def : InstRW<[ADLPWriteResGroup244, ReadAfterVecXLd, ReadAfterVecXLd, ReadDefault, ReadDefault, ReadDefault, ReadDefault, ReadDefault], (instregex "^VBLENDVP(D|S)rm$")>;
+def : InstRW<[ADLPWriteResGroup244, ReadAfterVecXLd, ReadAfterVecXLd, ReadDefault, ReadDefault, ReadDefault, ReadDefault, ReadDefault], (instrs VPBLENDVBrm)>;
 
-def ADLPWriteResGroup268 : SchedWriteRes<[ADLPPort00_01_05]> {
+def ADLPWriteResGroup245 : SchedWriteRes<[ADLPPort00_01_05]> {
   let ResourceCycles = [3];
   let Latency = 3;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup268], (instregex "^VBLENDVP(D|S)rr$")>;
-def : InstRW<[ADLPWriteResGroup268], (instrs VPBLENDVBrr)>;
+def : InstRW<[ADLPWriteResGroup245], (instregex "^VBLENDVP(D|S)rr$")>;
+def : InstRW<[ADLPWriteResGroup245], (instrs VPBLENDVBrr)>;
 
-def ADLPWriteResGroup269 : SchedWriteRes<[ADLPPort00, ADLPPort01, ADLPPort02_03_11]> {
+def ADLPWriteResGroup246 : SchedWriteRes<[ADLPPort00, ADLPPort01, ADLPPort02_03_11]> {
   let ResourceCycles = [6, 7, 18];
   let Latency = 81;
   let NumMicroOps = 31;
 }
-def : InstRW<[ADLPWriteResGroup269], (instrs VERRm)>;
+def : InstRW<[ADLPWriteResGroup246], (instrs VERRm)>;
 
-def ADLPWriteResGroup270 : SchedWriteRes<[ADLPPort00, ADLPPort01, ADLPPort02_03_11]> {
+def ADLPWriteResGroup247 : SchedWriteRes<[ADLPPort00, ADLPPort01, ADLPPort02_03_11]> {
   let ResourceCycles = [6, 7, 17];
   let Latency = 74;
   let NumMicroOps = 30;
 }
-def : InstRW<[ADLPWriteResGroup270], (instrs VERRr)>;
+def : InstRW<[ADLPWriteResGroup247], (instrs VERRr)>;
 
-def ADLPWriteResGroup271 : SchedWriteRes<[ADLPPort00, ADLPPort01, ADLPPort02_03_11]> {
+def ADLPWriteResGroup248 : SchedWriteRes<[ADLPPort00, ADLPPort01, ADLPPort02_03_11]> {
   let ResourceCycles = [5, 8, 21];
   let Latency = 81;
   let NumMicroOps = 34;
 }
-def : InstRW<[ADLPWriteResGroup271], (instrs VERWm)>;
+def : InstRW<[ADLPWriteResGroup248], (instrs VERWm)>;
 
-def ADLPWriteResGroup272 : SchedWriteRes<[ADLPPort00, ADLPPort01, ADLPPort02_03_11]> {
+def ADLPWriteResGroup249 : SchedWriteRes<[ADLPPort00, ADLPPort01, ADLPPort02_03_11]> {
   let ResourceCycles = [5, 8, 20];
   let Latency = 74;
   let NumMicroOps = 33;
 }
-def : InstRW<[ADLPWriteResGroup272], (instrs VERWr)>;
+def : InstRW<[ADLPWriteResGroup249], (instrs VERWr)>;
 
-def ADLPWriteResGroup273 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05, ADLPPort01_05, ADLPPort02_03_11]> {
+def ADLPWriteResGroup250 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05, ADLPPort01_05, ADLPPort02_03_11]> {
   let ResourceCycles = [1, 1, 2, 4];
   let Latency = 29;
   let NumMicroOps = 8;
 }
-def : InstRW<[ADLPWriteResGroup273, WriteVecMaskedGatherWriteback], (instregex "^VGATHER(D|Q)PDYrm$",
+def : InstRW<[ADLPWriteResGroup250, WriteVecMaskedGatherWriteback], (instregex "^VGATHER(D|Q)PDYrm$",
                                                                                "^VPGATHER(D|Q)QYrm$")>;
-def : InstRW<[ADLPWriteResGroup273, WriteVecMaskedGatherWriteback], (instrs VGATHERQPSYrm,
+def : InstRW<[ADLPWriteResGroup250, WriteVecMaskedGatherWriteback], (instrs VGATHERQPSYrm,
                                                                             VPGATHERQDYrm)>;
 
-def ADLPWriteResGroup274 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05, ADLPPort01_05, ADLPPort02_03_11]> {
+def ADLPWriteResGroup251 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05, ADLPPort01_05, ADLPPort02_03_11]> {
   let ResourceCycles = [1, 1, 1, 2];
   let Latency = 20;
   let NumMicroOps = 5;
 }
-def : InstRW<[ADLPWriteResGroup274, WriteVecMaskedGatherWriteback], (instregex "^VGATHER(D|Q)PDrm$",
+def : InstRW<[ADLPWriteResGroup251, WriteVecMaskedGatherWriteback], (instregex "^VGATHER(D|Q)PDrm$",
                                                                                "^VPGATHER(D|Q)Qrm$")>;
-def : InstRW<[ADLPWriteResGroup274, WriteVecMaskedGatherWriteback], (instrs VGATHERQPSrm,
+def : InstRW<[ADLPWriteResGroup251, WriteVecMaskedGatherWriteback], (instrs VGATHERQPSrm,
                                                                             VPGATHERQDrm)>;
 
-def ADLPWriteResGroup275 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05, ADLPPort01_05, ADLPPort02_03_11]> {
+def ADLPWriteResGroup252 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05, ADLPPort01_05, ADLPPort02_03_11]> {
   let ResourceCycles = [1, 1, 2, 8];
   let Latency = 30;
   let NumMicroOps = 12;
 }
-def : InstRW<[ADLPWriteResGroup275, WriteVecMaskedGatherWriteback], (instrs VGATHERDPSYrm,
+def : InstRW<[ADLPWriteResGroup252, WriteVecMaskedGatherWriteback], (instrs VGATHERDPSYrm,
                                                                             VPGATHERDDYrm)>;
 
-def ADLPWriteResGroup276 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05, ADLPPort01_05, ADLPPort02_03_11]> {
+def ADLPWriteResGroup253 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05, ADLPPort01_05, ADLPPort02_03_11]> {
   let ResourceCycles = [1, 1, 2, 4];
   let Latency = 28;
   let NumMicroOps = 8;
 }
-def : InstRW<[ADLPWriteResGroup276, WriteVecMaskedGatherWriteback], (instrs VGATHERDPSrm,
+def : InstRW<[ADLPWriteResGroup253, WriteVecMaskedGatherWriteback], (instrs VGATHERDPSrm,
                                                                             VPGATHERDDrm)>;
 
-def ADLPWriteResGroup277 : SchedWriteRes<[ADLPPort00_01]> {
-  let Latency = 3;
-}
-def : InstRW<[ADLPWriteResGroup277], (instregex "^VGF2P8AFFINE((INV)?)QB(Y?)rri$",
-                                                "^VGF2P8MULB(Y?)rr$")>;
-
-def ADLPWriteResGroup278 : SchedWriteRes<[ADLPPort01_05, ADLPPort02_03_11, ADLPPort05]> {
-  let ResourceCycles = [1, 1, 2];
-  let Latency = 13;
-  let NumMicroOps = 4;
+def ADLPWriteResGroup254 : SchedWriteRes<[ADLPPort01_05, ADLPPort05]> {
+  let ResourceCycles = [1, 2];
+  let Latency = 5;
+  let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup278, ReadAfterVecYLd], (instregex "^VH(ADD|SUB)PSYrm$")>;
+def : InstRW<[ADLPWriteResGroup254], (instregex "^VH(ADD|SUB)P(D|S)rr$")>;
 
-def ADLPWriteResGroup279 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort02_03_11]> {
+def ADLPWriteResGroup255 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort02_03_11]> {
   let Latency = 9;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup279, ReadAfterVecYLd], (instregex "^VINSERT(F|I)128rm$")>;
+def : InstRW<[ADLPWriteResGroup255, ReadAfterVecYLd], (instregex "^VINSERT(F|I)128rm$",
+                                                                 "^VP(ADD|SUB)(B|D|Q|W)Yrm$")>;
 
-def ADLPWriteResGroup280 : SchedWriteRes<[ADLPPort00, ADLPPort00_06, ADLPPort02_03_11]> {
+def ADLPWriteResGroup256 : SchedWriteRes<[ADLPPort00, ADLPPort00_06, ADLPPort02_03_11]> {
   let Latency = 7;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup280], (instrs VLDMXCSR)>;
+def : InstRW<[ADLPWriteResGroup256], (instrs VLDMXCSR)>;
 
-def ADLPWriteResGroup281 : SchedWriteRes<[ADLPPort00_01_05_06, ADLPPort01, ADLPPort01_05, ADLPPort02_03, ADLPPort02_03_07, ADLPPort04, ADLPPort05, ADLPPort06]> {
+def ADLPWriteResGroup257 : SchedWriteRes<[ADLPPort00_01_05_06, ADLPPort01, ADLPPort01_05, ADLPPort02_03, ADLPPort02_03_07, ADLPPort04, ADLPPort05, ADLPPort06]> {
   let ResourceCycles = [8, 1, 1, 1, 1, 1, 2, 3];
   let Latency = 40;
   let NumMicroOps = 18;
 }
-def : InstRW<[ADLPWriteResGroup281], (instrs VMCLEARm)>;
+def : InstRW<[ADLPWriteResGroup257], (instrs VMCLEARm)>;
 
-def ADLPWriteResGroup282 : SchedWriteRes<[ADLPPort00]> {
+def ADLPWriteResGroup258 : SchedWriteRes<[ADLPPort00]> {
   let Latency = 5;
 }
-def : InstRW<[ADLPWriteResGroup282], (instregex "^VMOVMSKP(D|S)Yrr$")>;
+def : InstRW<[ADLPWriteResGroup258], (instregex "^VMOVMSKP(D|S)Yrr$")>;
 
-def ADLPWriteResGroup283 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup259 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 521;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup283], (instrs VMOVNTDQmr)>;
+def : InstRW<[ADLPWriteResGroup259], (instrs VMOVNTDQmr)>;
 
-def ADLPWriteResGroup284 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup260 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 473;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup284], (instrs VMOVNTPDmr)>;
+def : InstRW<[ADLPWriteResGroup260], (instrs VMOVNTPDmr)>;
 
-def ADLPWriteResGroup285 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup261 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 494;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup285], (instrs VMOVNTPSYmr)>;
+def : InstRW<[ADLPWriteResGroup261], (instrs VMOVNTPSYmr)>;
 
-def ADLPWriteResGroup286 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
+def ADLPWriteResGroup262 : SchedWriteRes<[ADLPPort04_09, ADLPPort07_08]> {
   let Latency = 470;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup286], (instrs VMOVNTPSmr)>;
+def : InstRW<[ADLPWriteResGroup262], (instrs VMOVNTPSmr)>;
 
-def ADLPWriteResGroup287 : SchedWriteRes<[ADLPPort02_03_11, ADLPPort05]> {
-  let Latency = 10;
+def ADLPWriteResGroup263 : SchedWriteRes<[ADLPPort02_03_11, ADLPPort05]> {
+  let Latency = 11;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup287, ReadAfterVecYLd], (instregex "^VPACK(S|U)S(DW|WB)Yrm$")>;
-def : InstRW<[ADLPWriteResGroup287, ReadAfterVecYLd], (instrs VPCMPGTQYrm)>;
-def : InstRW<[ADLPWriteResGroup287, ReadAfterVecXLd], (instrs VPCLMULQDQYrm)>;
+def : InstRW<[ADLPWriteResGroup263, ReadAfterVecYLd], (instregex "^VPACK(S|U)S(DW|WB)Yrm$")>;
+def : InstRW<[ADLPWriteResGroup263, ReadAfterVecYLd], (instrs VPCMPGTQYrm)>;
+def : InstRW<[ADLPWriteResGroup263, ReadAfterVecXLd], (instrs VPCLMULQDQYrm)>;
 
-def ADLPWriteResGroup288 : SchedWriteRes<[ADLPPort00_01, ADLPPort01_05, ADLPPort02_03_11]> {
-  let ResourceCycles = [1, 2, 1];
+def ADLPWriteResGroup264 : SchedWriteRes<[ADLPPort01_05, ADLPPort02_03_11]> {
   let Latency = 9;
-  let NumMicroOps = 4;
+  let NumMicroOps = 2;
+}
+def : InstRW<[ADLPWriteResGroup264, ReadAfterVecYLd], (instregex "^VSHUFP(D|S)Yrmi$")>;
+def : InstRW<[ADLPWriteResGroup264, ReadAfterVecYLd], (instrs VPBLENDWYrmi)>;
+
+def ADLPWriteResGroup265 : SchedWriteRes<[ADLPPort00_01, ADLPPort02_03_11]> {
+  let Latency = 13;
+  let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup288, ReadAfterVecYLd], (instrs VPHADDSWYrm)>;
+def : InstRW<[ADLPWriteResGroup265], (instregex "^VPDP(BU|WS)SD((SY)?)rm$",
+                                                "^VPDP(BU|WS)SD(S|Y)rm$")>;
 
-def ADLPWriteResGroup289 : SchedWriteRes<[ADLPPort00_01, ADLPPort01_05, ADLPPort02_03_11]> {
+def ADLPWriteResGroup266 : SchedWriteRes<[ADLPPort00_01, ADLPPort01_05, ADLPPort02_03_11]> {
   let ResourceCycles = [1, 2, 1];
   let Latency = 10;
   let NumMicroOps = 4;
 }
-def : InstRW<[ADLPWriteResGroup289, ReadAfterVecYLd], (instrs VPHSUBSWYrm)>;
+def : InstRW<[ADLPWriteResGroup266, ReadAfterVecYLd], (instregex "^VPH(ADD|SUB)SWYrm$")>;
 
-def ADLPWriteResGroup290 : SchedWriteRes<[ADLPPort01_05, ADLPPort05]> {
-  let Latency = 4;
-  let NumMicroOps = 2;
-}
-def : InstRW<[ADLPWriteResGroup290, ReadDefault, ReadInt2Fpu], (instregex "^VPINSR(B|D|Q|W)rr$")>;
-
-def ADLPWriteResGroup291 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10]> {
+def ADLPWriteResGroup267 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10]> {
   let ResourceCycles = [1, 2, 3, 3, 1];
   let Latency = 16;
   let NumMicroOps = 10;
 }
-def : InstRW<[ADLPWriteResGroup291], (instrs VZEROALL)>;
-
-def ADLPWriteResGroup292 : SchedWriteRes<[]> {
-  let Latency = 0;
-  let NumMicroOps = 0;
-}
-def : InstRW<[ADLPWriteResGroup292], (instrs VZEROUPPER)>;
+def : InstRW<[ADLPWriteResGroup267], (instrs VZEROALL)>;
 
-def ADLPWriteResGroup293 : SchedWriteRes<[ADLPPort00_01_05_06]> {
+def ADLPWriteResGroup268 : SchedWriteRes<[ADLPPort00_01_05_06]> {
   let ResourceCycles = [2];
   let Latency = 2;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup293], (instrs WAIT)>;
+def : InstRW<[ADLPWriteResGroup268], (instrs WAIT)>;
 
-def ADLPWriteResGroup294 : SchedWriteRes<[ADLPPort00, ADLPPort00_01, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup269 : SchedWriteRes<[ADLPPort00, ADLPPort00_01, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [8, 6, 19, 63, 21, 15, 1, 10, 1];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 144;
 }
-def : InstRW<[ADLPWriteResGroup294], (instrs WRMSR)>;
+def : InstRW<[ADLPWriteResGroup269], (instrs WRMSR)>;
 
-def ADLPWriteResGroup295 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort05]> {
+def ADLPWriteResGroup270 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06, ADLPPort01, ADLPPort05]> {
   let ResourceCycles = [2, 1, 4, 1];
   let Latency = AlderlakePModel.MaxLatency;
   let NumMicroOps = 8;
 }
-def : InstRW<[ADLPWriteResGroup295], (instrs WRPKRUr)>;
+def : InstRW<[ADLPWriteResGroup270], (instrs WRPKRUr)>;
 
-def ADLPWriteResGroup296 : SchedWriteRes<[ADLPPort00_01_05_06_10]> {
+def ADLPWriteResGroup271 : SchedWriteRes<[ADLPPort00_01_05_06_10]> {
   let ResourceCycles = [2];
-  let Latency = 7;
+  let Latency = 12;
+  let NumMicroOps = 2;
+}
+def : InstRW<[ADLPWriteResGroup271, WriteRMW], (instregex "^XADD(16|32|64)rm$")>;
+
+def ADLPWriteResGroup272 : SchedWriteRes<[ADLPPort00_01_05_06_10]> {
+  let ResourceCycles = [2];
+  let Latency = 13;
   let NumMicroOps = 2;
 }
-def : InstRW<[ADLPWriteResGroup296, WriteRMW], (instregex "^XADD(8|16|32|64)rm$")>;
+def : InstRW<[ADLPWriteResGroup272, WriteRMW], (instrs XADD8rm)>;
 
-def ADLPWriteResGroup297 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06]> {
+def ADLPWriteResGroup273 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06]> {
   let ResourceCycles = [4, 1];
   let Latency = 39;
   let NumMicroOps = 5;
 }
-def : InstRW<[ADLPWriteResGroup297, WriteRMW], (instregex "^XCHG(16|32)rm$")>;
+def : InstRW<[ADLPWriteResGroup273, WriteRMW], (instregex "^XCHG(16|32)rm$")>;
 
-def ADLPWriteResGroup298 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06]> {
+def ADLPWriteResGroup274 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06]> {
   let ResourceCycles = [5, 1];
   let Latency = 39;
   let NumMicroOps = 6;
 }
-def : InstRW<[ADLPWriteResGroup298, WriteRMW], (instrs XCHG64rm)>;
+def : InstRW<[ADLPWriteResGroup274, WriteRMW], (instrs XCHG64rm)>;
 
-def ADLPWriteResGroup299 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06]> {
+def ADLPWriteResGroup275 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_06]> {
   let ResourceCycles = [4, 1];
   let Latency = 40;
   let NumMicroOps = 5;
 }
-def : InstRW<[ADLPWriteResGroup299, WriteRMW], (instrs XCHG8rm)>;
+def : InstRW<[ADLPWriteResGroup275, WriteRMW], (instrs XCHG8rm)>;
 
-def ADLPWriteResGroup300 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06, ADLPPort00_05, ADLPPort01, ADLPPort05, ADLPPort06]> {
+def ADLPWriteResGroup276 : SchedWriteRes<[ADLPPort00, ADLPPort00_01_05_06, ADLPPort00_05, ADLPPort01, ADLPPort05, ADLPPort06]> {
   let ResourceCycles = [2, 4, 2, 1, 2, 4];
   let Latency = 17;
   let NumMicroOps = 15;
 }
-def : InstRW<[ADLPWriteResGroup300], (instrs XCH_F)>;
+def : InstRW<[ADLPWriteResGroup276], (instrs XCH_F)>;
 
-def ADLPWriteResGroup301 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01]> {
+def ADLPWriteResGroup277 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01]> {
   let ResourceCycles = [7, 3, 8, 5];
   let Latency = 4;
   let NumMicroOps = 23;
 }
-def : InstRW<[ADLPWriteResGroup301], (instrs XGETBV)>;
+def : InstRW<[ADLPWriteResGroup277], (instrs XGETBV)>;
 
-def ADLPWriteResGroup302 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11]> {
+def ADLPWriteResGroup278 : SchedWriteRes<[ADLPPort00_01_05_06_10, ADLPPort02_03_11]> {
   let ResourceCycles = [2, 1];
   let Latency = 7;
   let NumMicroOps = 3;
 }
-def : InstRW<[ADLPWriteResGroup302], (instrs XLAT)>;
+def : InstRW<[ADLPWriteResGroup278], (instrs XLAT)>;
 
-def ADLPWriteResGroup303 : SchedWriteRes<[ADLPPort00_01_05_06, ADLPPort01, ADLPPort02_03, ADLPPort06]> {
+def ADLPWriteResGroup279 : SchedWriteRes<[ADLPPort00_01_05_06, ADLPPort01, ADLPPort02_03, ADLPPort06]> {
   let ResourceCycles = [21, 1, 1, 8];
   let Latency = 37;
   let NumMicroOps = 31;
 }
-def : InstRW<[ADLPWriteResGroup303], (instregex "^XRSTOR((S|64)?)$")>;
-def : InstRW<[ADLPWriteResGroup303], (instrs XRSTORS64)>;
+def : InstRW<[ADLPWriteResGroup279], (instregex "^XRSTOR((S|64)?)$")>;
+def : InstRW<[ADLPWriteResGroup279], (instrs XRSTORS64)>;
 
-def ADLPWriteResGroup304 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup280 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [14, 25, 44, 21, 21, 4, 1, 9, 1];
   let Latency = 42;
   let NumMicroOps = 140;
 }
-def : InstRW<[ADLPWriteResGroup304], (instrs XSAVE)>;
+def : InstRW<[ADLPWriteResGroup280], (instrs XSAVE)>;
 
-def ADLPWriteResGroup305 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup281 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [14, 25, 44, 21, 21, 4, 1, 9, 1];
   let Latency = 41;
   let NumMicroOps = 140;
 }
-def : InstRW<[ADLPWriteResGroup305], (instrs XSAVE64)>;
+def : InstRW<[ADLPWriteResGroup281], (instrs XSAVE64)>;
 
-def ADLPWriteResGroup306 : SchedWriteRes<[ADLPPort00, ADLPPort00_01, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup282 : SchedWriteRes<[ADLPPort00, ADLPPort00_01, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [1, 19, 36, 52, 23, 4, 2, 12, 2];
   let Latency = 42;
   let NumMicroOps = 151;
 }
-def : InstRW<[ADLPWriteResGroup306], (instrs XSAVEC)>;
+def : InstRW<[ADLPWriteResGroup282], (instrs XSAVEC)>;
 
-def ADLPWriteResGroup307 : SchedWriteRes<[ADLPPort00, ADLPPort00_01, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup283 : SchedWriteRes<[ADLPPort00, ADLPPort00_01, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [1, 19, 36, 53, 23, 4, 2, 12, 2];
   let Latency = 42;
   let NumMicroOps = 152;
 }
-def : InstRW<[ADLPWriteResGroup307], (instrs XSAVEC64)>;
+def : InstRW<[ADLPWriteResGroup283], (instrs XSAVEC64)>;
 
-def ADLPWriteResGroup308 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup284 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [25, 35, 52, 27, 4, 1, 10, 1];
   let Latency = 46;
   let NumMicroOps = 155;
 }
-def : InstRW<[ADLPWriteResGroup308], (instrs XSAVEOPT)>;
+def : InstRW<[ADLPWriteResGroup284], (instrs XSAVEOPT)>;
 
-def ADLPWriteResGroup309 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup285 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [25, 35, 53, 27, 4, 1, 10, 1];
   let Latency = 46;
   let NumMicroOps = 156;
 }
-def : InstRW<[ADLPWriteResGroup309], (instrs XSAVEOPT64)>;
+def : InstRW<[ADLPWriteResGroup285], (instrs XSAVEOPT64)>;
 
-def ADLPWriteResGroup310 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup286 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [23, 32, 53, 29, 30, 4, 2, 9, 2];
   let Latency = 42;
   let NumMicroOps = 184;
 }
-def : InstRW<[ADLPWriteResGroup310], (instrs XSAVES)>;
+def : InstRW<[ADLPWriteResGroup286], (instrs XSAVES)>;
 
-def ADLPWriteResGroup311 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
+def ADLPWriteResGroup287 : SchedWriteRes<[ADLPPort00_01, ADLPPort00_05, ADLPPort00_06, ADLPPort01, ADLPPort01_05, ADLPPort02_03_11, ADLPPort04_09, ADLPPort05, ADLPPort07_08]> {
   let ResourceCycles = [23, 33, 53, 29, 32, 4, 2, 8, 2];
   let Latency = 42;
   let NumMicroOps = 186;
 }
-def : InstRW<[ADLPWriteResGroup311], (instrs XSAVES64)>;
+def : InstRW<[ADLPWriteResGroup287], (instrs XSAVES64)>;
 
-def ADLPWriteResGroup312 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort00_01_05_06_10, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort05]> {
+def ADLPWriteResGroup288 : SchedWriteRes<[ADLPPort00_01_05, ADLPPort00_01_05_06_10, ADLPPort00_05_06, ADLPPort00_06, ADLPPort01, ADLPPort01_05_10, ADLPPort05]> {
   let ResourceCycles = [4, 23, 2, 14, 8, 1, 2];
   let Latency = 5;
   let NumMicroOps = 54;
 }
-def : InstRW<[ADLPWriteResGroup312], (instrs XSETBV)>;
+def : InstRW<[ADLPWriteResGroup288], (instrs XSETBV)>;
 
 }
diff --git a/llvm/lib/Target/X86/X86ScheduleZnver4.td b/llvm/lib/Target/X86/X86ScheduleZnver4.td
new file mode 100644
index 0000000000000..c3f08998419fc
--- /dev/null
+++ b/llvm/lib/Target/X86/X86ScheduleZnver4.td
@@ -0,0 +1,1957 @@
+//=- X86ScheduleZnver4.td - X86 Znver4 Scheduling ------------*- tablegen -*-=//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the machine model for Znver4 to support instruction
+// scheduling and other instruction cost heuristics.
+// Based on:
+//  * AMD Software Optimization Guide for AMD Family 19h Processors.
+//    https://www.amd.com/system/files/TechDocs/56665.zip
+//===----------------------------------------------------------------------===//
+
+def Znver4Model : SchedMachineModel {
+  // AMD SOG 19h, 2.9.6 Dispatch
+  // The processor may dispatch up to 6 macro ops per cycle
+  // into the execution engine.
+  let IssueWidth = 6;
+  // AMD SOG 19h, 2.10.3
+  // The retire control unit (RCU) tracks the completion status of all
+  // outstanding operations (integer, load/store, and floating-point) and is
+  // the final arbiter for exception processing and recovery.
+  // The unit can receive up to 6 macro ops dispatched per cycle and track up
+  // to 320 macro ops in-flight in non-SMT mode or 160 per thread in SMT mode.
+  let MicroOpBufferSize = 320;
+  // AMD SOG 19h, 2.9.1 Op Cache
+  // The op cache is organized as an associative cache with 64 sets and 8 ways.
+  // At each set-way intersection is an entry containing up to 8 macro ops.
+  // The maximum capacity of the op cache is 4K ops.
+  // Agner, 22.5 µop cache
+  // The size of the µop cache is big enough for holding most critical loops.
+  // FIXME: PR50584: MachineScheduler/PostRAScheduler have quadradic complexity,
+  //        with large values here the compilation of certain loops
+  //        ends up taking way too long.
+  // Ideally for znver4, we should have 6.75K. However we don't add that
+  // considerting the impact compile time and prefer using default values 
+  // instead.
+  // let LoopMicroOpBufferSize = 6750;
+  // AMD SOG 19h, 2.6.2 L1 Data Cache
+  // The L1 data cache has a 4- or 5- cycle integer load-to-use latency.
+  // AMD SOG 19h, 2.12 L1 Data Cache
+  // The AGU and LS pipelines are optimized for simple address generation modes.
+  // <...> and can achieve 4-cycle load-to-use integer load latency.
+  let LoadLatency = 4;
+  // AMD SOG 19h, 2.12 L1 Data Cache
+  // The AGU and LS pipelines are optimized for simple address generation modes.
+  // <...> and can achieve <...> 7-cycle load-to-use FP load latency.
+  int VecLoadLatency = 7;
+  // Latency of a simple store operation.
+  int StoreLatency = 1;
+  // FIXME:
+  let HighLatency = 25; // FIXME: any better choice?
+  // AMD SOG 19h, 2.8 Optimizing Branching
+  // The branch misprediction penalty is in the range from 11 to 18 cycles,
+  // <...>. The common case penalty is 13 cycles.
+  let MispredictPenalty = 13;
+
+  let PostRAScheduler = 1; // Enable Post RegAlloc Scheduler pass.
+
+  let CompleteModel = 1;
+}
+
+let SchedModel = Znver4Model in {
+
+
+//===----------------------------------------------------------------------===//
+// RCU
+//===----------------------------------------------------------------------===//
+
+// AMD SOG 19h, 2.10.3 Retire Control Unit
+// The unit can receive up to 6 macro ops dispatched per cycle and track up to
+// 320 macro ops in-flight in non-SMT mode or 128 per thread in SMT mode. <...>
+// The retire unit handles in-order commit of up to nine macro ops per cycle.
+def Zn4RCU : RetireControlUnit<Znver4Model.MicroOpBufferSize, 9>;
+
+//===----------------------------------------------------------------------===//
+// Integer Execution Unit
+//
+
+// AMD SOG 19h, 2.4 Superscalar Organization
+// The processor uses four decoupled independent integer scheduler queues,
+// each one servicing one ALU pipeline and one or two other pipelines
+
+//
+// Execution pipes
+//===----------------------------------------------------------------------===//
+
+// AMD SOG 19h, 2.10.2 Execution Units
+// The processor contains 4 general purpose integer execution pipes.
+// Each pipe has an ALU capable of general purpose integer operations.
+def Zn4ALU0 : ProcResource<1>;
+def Zn4ALU1 : ProcResource<1>;
+def Zn4ALU2 : ProcResource<1>;
+def Zn4ALU3 : ProcResource<1>;
+
+// AMD SOG 19h, 2.10.2 Execution Units
+// There is also a separate branch execution unit.
+def Zn4BRU1 : ProcResource<1>;
+
+// AMD SOG 19h, 2.10.2 Execution Units
+// There are three Address Generation Units (AGUs) for all load and store
+// address generation. There are also 3 store data movement units
+// associated with the same schedulers as the AGUs.
+def Zn4AGU0 : ProcResource<1>;
+def Zn4AGU1 : ProcResource<1>;
+def Zn4AGU2 : ProcResource<1>;
+
+//
+// Execution Units
+//===----------------------------------------------------------------------===//
+
+// AMD SOG 19h, 2.10.2 Execution Units
+// ALU0 additionally has divide <...> execution capability.
+defvar Zn4Divider = Zn4ALU0;
+
+// AMD SOG 19h, 2.10.2 Execution Units
+// ALU0 additionally has <...> branch execution capability.
+defvar Zn4BRU0 = Zn4ALU0;
+
+// Integer Multiplication issued on ALU1.
+defvar Zn4Multiplier = Zn4ALU1;
+
+// Execution pipeline grouping
+//===----------------------------------------------------------------------===//
+
+// General ALU operations
+def Zn4ALU0123 : ProcResGroup<[Zn4ALU0, Zn4ALU1, Zn4ALU2, Zn4ALU3]>;
+
+// General AGU operations
+def Zn4AGU012 : ProcResGroup<[Zn4AGU0, Zn4AGU1, Zn4AGU2]>;
+
+// Control flow: jumps, calls
+def Zn4BRU01 : ProcResGroup<[Zn4BRU0, Zn4BRU1]>;
+
+// Everything that isn't control flow, but still needs to access CC register,
+// namely: conditional moves, SETcc.
+def Zn4ALU03 : ProcResGroup<[Zn4ALU0, Zn4ALU3]>;
+
+// Zn4ALU1 handles complex bit twiddling: CRC/PDEP/PEXT
+
+// Simple bit twiddling: bit test, shift/rotate, bit extraction
+def Zn4ALU12 : ProcResGroup<[Zn4ALU1, Zn4ALU2]>;
+
+
+//
+// Scheduling
+//===----------------------------------------------------------------------===//
+
+// AMD SOG 19h, 2.10.3 Retire Control Unit
+// The integer physical register file (PRF) consists of 224 registers.
+def Zn4IntegerPRF : RegisterFile<224, [GR64, CCR], [1, 1], [1, 0],
+                              6,  // Max moves that can be eliminated per cycle.
+                              0>; // Restrict move elimination to zero regs.
+
+// anandtech, The integer scheduler has a 4*24 entry macro op capacity.
+// AMD SOG 19h, 2.10.1 Schedulers
+// The schedulers can receive up to six macro ops per cycle, with a limit of
+// two per scheduler. Each scheduler can issue one micro op per cycle into
+// each of its associated pipelines
+def Zn4Int : ProcResGroup<[Zn4ALU0, Zn4AGU0, Zn4BRU0, // scheduler 0
+                           Zn4ALU1, Zn4AGU1,          // scheduler 1
+                           Zn4ALU2, Zn4AGU2,          // scheduler 2
+                           Zn4ALU3,          Zn4BRU1  // scheduler 3
+                          ]> {
+  let BufferSize = !mul(4, 24);
+}
+
+
+//===----------------------------------------------------------------------===//
+// Floating-Point Unit
+//
+
+// AMD SOG 19h, 2.4 Superscalar Organization
+// The processor uses <...> two decoupled independent floating point schedulers
+// each servicing two FP pipelines and one store or FP-to-integer pipeline.
+
+//
+// Execution pipes
+//===----------------------------------------------------------------------===//
+
+// AMD SOG 19h, 2.10.1 Schedulers
+// <...>, and six FPU pipes.
+// Agner, 22.10 Floating point execution pipes
+// There are six floating point/vector execution pipes,
+def Zn4FP0  : ProcResource<1>;
+def Zn4FP1  : ProcResource<1>;
+def Zn4FP2  : ProcResource<1>;
+def Zn4FP3  : ProcResource<1>;
+def Zn4FP45 : ProcResource<2>;
+
+//
+// Execution Units
+//===----------------------------------------------------------------------===//
+// AMD SOG 19h, 2.11.1 Floating Point Execution Resources
+
+// (v)FMUL*, (v)FMA*, Floating Point Compares, Blendv(DQ)
+defvar Zn4FPFMul0 = Zn4FP0;
+defvar Zn4FPFMul1 = Zn4FP1;
+
+// (v)FADD*
+defvar Zn4FPFAdd0 = Zn4FP2;
+defvar Zn4FPFAdd1 = Zn4FP3;
+
+// All convert operations except pack/unpack
+defvar Zn4FPFCvt0 = Zn4FP2;
+defvar Zn4FPFCvt1 = Zn4FP3;
+
+// All Divide and Square Root except Reciprocal Approximation
+// AMD SOG 19h, 2.11.1 Floating Point Execution Resources
+// FDIV unit can support 2 simultaneous operations in flight
+// even though it occupies a single pipe.
+// FIXME: BufferSize=2 ?
+defvar Zn4FPFDiv = Zn4FP1;
+
+// Moves and Logical operations on Floating Point Data Types
+defvar Zn4FPFMisc0 = Zn4FP0;
+defvar Zn4FPFMisc1 = Zn4FP1;
+defvar Zn4FPFMisc2 = Zn4FP2;
+defvar Zn4FPFMisc3 = Zn4FP3;
+
+// Integer Adds, Subtracts, and Compares
+// Some complex VADD operations are not available in all pipes.
+defvar Zn4FPVAdd0 = Zn4FP0;
+defvar Zn4FPVAdd1 = Zn4FP1;
+defvar Zn4FPVAdd2 = Zn4FP2;
+defvar Zn4FPVAdd3 = Zn4FP3;
+
+// Integer Multiplies, SAD, Blendvb
+defvar Zn4FPVMul0 = Zn4FP0;
+defvar Zn4FPVMul1 = Zn4FP3;
+
+// Data Shuffles, Packs, Unpacks, Permute
+// Some complex shuffle operations are only available in pipe1.
+defvar Zn4FPVShuf = Zn4FP1;
+defvar Zn4FPVShufAux = Zn4FP2;
+
+// Bit Shift Left/Right operations
+defvar Zn4FPVShift0 = Zn4FP1;
+defvar Zn4FPVShift1 = Zn4FP2;
+
+// Moves and Logical operations on Packed Integer Data Types
+defvar Zn4FPVMisc0 = Zn4FP0;
+defvar Zn4FPVMisc1 = Zn4FP1;
+defvar Zn4FPVMisc2 = Zn4FP2;
+defvar Zn4FPVMisc3 = Zn4FP3;
+
+// *AES*
+defvar Zn4FPAES0 = Zn4FP0;
+defvar Zn4FPAES1 = Zn4FP1;
+
+// *CLM*
+defvar Zn4FPCLM0 = Zn4FP0;
+defvar Zn4FPCLM1 = Zn4FP1;
+
+// Execution pipeline grouping
+//===----------------------------------------------------------------------===//
+
+// AMD SOG 19h, 2.11 Floating-Point Unit
+// Stores and floating point to general purpose register transfer
+// have 2 dedicated pipelines (pipe 5 and 6).
+def Zn4FPU0123 : ProcResGroup<[Zn4FP0, Zn4FP1, Zn4FP2, Zn4FP3]>;
+
+// (v)FMUL*, (v)FMA*, Floating Point Compares, Blendv(DQ)
+def Zn4FPFMul01 : ProcResGroup<[Zn4FPFMul0, Zn4FPFMul1]>;
+
+// (v)FADD*
+// Some complex VADD operations are not available in all pipes.
+def Zn4FPFAdd01 : ProcResGroup<[Zn4FPFAdd0, Zn4FPFAdd1]>;
+
+// All convert operations except pack/unpack
+def Zn4FPFCvt01 : ProcResGroup<[Zn4FPFCvt0, Zn4FPFCvt1]>;
+
+// All Divide and Square Root except Reciprocal Approximation
+// def Zn4FPFDiv : ProcResGroup<[Zn4FPFDiv]>;
+
+// Moves and Logical operations on Floating Point Data Types
+def Zn4FPFMisc0123 : ProcResGroup<[Zn4FPFMisc0, Zn4FPFMisc1, Zn4FPFMisc2, Zn4FPFMisc3]>;
+
+// FIXUP and RANGE use FP01 pipelines
+def Zn4FPFMisc01 : ProcResGroup<[Zn4FPFMisc0, Zn4FPFMisc1]>;
+def Zn4FPFMisc12 : ProcResGroup<[Zn4FPFMisc1, Zn4FPFMisc2]>;
+// SCALE instructions use FP23 pipelines
+def Zn4FPFMisc23 : ProcResGroup<[Zn4FPFMisc2, Zn4FPFMisc3]>;
+def Zn4FPFMisc123 : ProcResGroup<[Zn4FPFMisc1,Zn4FPFMisc2, Zn4FPFMisc3]>;
+
+// Loads, Stores and Move to General Register (EX) Operations
+// AMD SOG 19h, 2.11 Floating-Point Unit
+// Stores and floating point to general purpose register transfer
+// have 2 dedicated pipelines (pipe 5 and 6).
+defvar Zn4FPLd01 = Zn4FP45;
+
+// AMD SOG 19h, 2.11 Floating-Point Unit
+// Note that FP stores are supported on two pipelines,
+// but throughput is limited to one per cycle.
+let Super = Zn4FP45 in
+def Zn4FPSt : ProcResource<1>;
+
+// Integer Adds, Subtracts, and Compares
+// Some complex VADD operations are not available in all pipes.
+def Zn4FPVAdd0123 : ProcResGroup<[Zn4FPVAdd0, Zn4FPVAdd1, Zn4FPVAdd2, Zn4FPVAdd3]>;
+
+def Zn4FPVAdd01: ProcResGroup<[Zn4FPVAdd0, Zn4FPVAdd1]>;
+def Zn4FPVAdd12: ProcResGroup<[Zn4FPVAdd1, Zn4FPVAdd2]>;
+
+// AVX512 Opmask pipelines
+def Zn4FPOpMask01: ProcResGroup<[Zn4FP2, Zn4FP3]>;
+def Zn4FPOpMask4: ProcResGroup<[Zn4FP45]>;
+
+// Integer Multiplies, SAD, Blendvb
+def Zn4FPVMul01 : ProcResGroup<[Zn4FPVMul0, Zn4FPVMul1]>;
+
+// Data Shuffles, Packs, Unpacks, Permute
+// Some complex shuffle operations are only available in pipe1.
+def Zn4FPVShuf01 : ProcResGroup<[Zn4FPVShuf, Zn4FPVShufAux]>;
+
+// Bit Shift Left/Right operations
+def Zn4FPVShift01 : ProcResGroup<[Zn4FPVShift0, Zn4FPVShift1]>;
+
+// Moves and Logical operations on Packed Integer Data Types
+def Zn4FPVMisc0123 : ProcResGroup<[Zn4FPVMisc0, Zn4FPVMisc1, Zn4FPVMisc2, Zn4FPVMisc3]>;
+
+// *AES*
+def Zn4FPAES01 : ProcResGroup<[Zn4FPAES0, Zn4FPAES1]>;
+
+// *CLM*
+def Zn4FPCLM01 : ProcResGroup<[Zn4FPCLM0, Zn4FPCLM1]>;
+
+
+//
+// Scheduling
+//===----------------------------------------------------------------------===//
+
+// Agner, 21.8 Register renaming and out-of-order schedulers
+// The floating point register file has 192 vector registers
+// of 512b each in zen4.
+def Zn4FpPRF : RegisterFile<192, [VR64, VR128, VR256, VR512], [1, 1, 1, 1], [0, 1, 1],
+                            6,  // Max moves that can be eliminated per cycle.
+                            0>; // Restrict move elimination to zero regs.
+
+// AMD SOG 19h, 2.11 Floating-Point Unit
+// The floating-point scheduler has a 2*32 entry macro op capacity.
+// AMD SOG 19h, 2.11 Floating-Point Unit
+// <...> the scheduler can issue 1 micro op per cycle for each pipe.
+// FIXME: those are two separate schedulers, not a single big one.
+def Zn4FP : ProcResGroup<[Zn4FP0, Zn4FP2,          /*Zn4FP4,*/ // scheduler 0
+                          Zn4FP1, Zn4FP3, Zn4FP45 /*Zn4FP5*/  // scheduler 1
+                         ]> {
+  let BufferSize = !mul(2, 32);
+}
+
+// AMD SOG 19h, 2.11 Floating-Point Unit
+// Macro ops can be dispatched to the 64 entry Non Scheduling Queue (NSQ)
+// even if floating-point scheduler is full.
+// FIXME: how to model this properly?
+
+
+//===----------------------------------------------------------------------===//
+// Load-Store Unit
+//
+
+// AMD SOG 19h, 2.12 Load-Store Unit
+// The LS unit contains three largely independent pipe-lines
+// enabling the execution of three 256-bit memory operations per cycle.
+def Zn4LSU : ProcResource<3>;
+
+// AMD SOG 19h, 2.12 Load-Store Unit
+// All three memory operations can be loads.
+let Super = Zn4LSU in
+def Zn4Load : ProcResource<3> {
+  // AMD SOG 19h, 2.12 Load-Store Unit
+  // The LS unit can process up to 72 out-of-order loads.
+  let BufferSize = 72;
+}
+
+def Zn4LoadQueue : LoadQueue<Zn4Load>;
+
+// AMD SOG 19h, 2.12 Load-Store Unit
+// A maximum of two of the memory operations can be stores.
+let Super = Zn4LSU in
+def Zn4Store : ProcResource<2> {
+  // AMD SOG 19h, 2.12 Load-Store Unit
+  // The LS unit utilizes a 64-entry store queue (STQ).
+  let BufferSize = 64;
+}
+
+def Zn4StoreQueue : StoreQueue<Zn4Store>;
+
+//===----------------------------------------------------------------------===//
+// Basic helper classes.
+//===----------------------------------------------------------------------===//
+
+// Many SchedWrites are defined in pairs with and without a folded load.
+// Instructions with folded loads are usually micro-fused, so they only appear
+// as two micro-ops when dispatched by the schedulers.
+// This multiclass defines the resource usage for variants with and without
+// folded loads.
+
+multiclass __Zn4WriteRes<SchedWrite SchedRW, list<ProcResourceKind> ExePorts,
+                         int Lat = 1, list<int> Res = [], int UOps = 1> {
+  def : WriteRes<SchedRW, ExePorts> {
+    let Latency = Lat;
+    let ResourceCycles = Res;
+    let NumMicroOps = UOps;
+  }
+}
+
+multiclass __Zn4WriteResPair<X86FoldableSchedWrite SchedRW,
+                             list<ProcResourceKind> ExePorts, int Lat,
+                             list<int> Res, int UOps, int LoadLat, int LoadUOps,
+                             ProcResourceKind AGU, int LoadRes> {
+  defm : __Zn4WriteRes<SchedRW, ExePorts, Lat, Res, UOps>;
+
+  defm : __Zn4WriteRes<SchedRW.Folded,
+                       !listconcat([AGU, Zn4Load], ExePorts),
+                       !add(Lat, LoadLat),
+                       !if(!and(!empty(Res), !eq(LoadRes, 1)),
+                         [],
+                         !listconcat([1, LoadRes],
+                           !if(!empty(Res),
+                             !listsplat(1, !size(ExePorts)),
+                             Res))),
+                       !add(UOps, LoadUOps)>;
+}
+
+// For classes without folded loads.
+multiclass Zn4WriteResInt<SchedWrite SchedRW,
+                          list<ProcResourceKind> ExePorts, int Lat = 1,
+                          list<int> Res = [], int UOps = 1> {
+  defm : __Zn4WriteRes<SchedRW, ExePorts, Lat, Res, UOps>;
+}
+
+multiclass Zn4WriteResXMM<SchedWrite SchedRW,
+                          list<ProcResourceKind> ExePorts, int Lat = 1,
+                          list<int> Res = [], int UOps = 1> {
+  defm : __Zn4WriteRes<SchedRW, ExePorts, Lat, Res, UOps>;
+}
+
+multiclass Zn4WriteResYMM<SchedWrite SchedRW,
+                          list<ProcResourceKind> ExePorts, int Lat = 1,
+                          list<int> Res = [], int UOps = 1> {
+  defm : __Zn4WriteRes<SchedRW, ExePorts, Lat, Res, UOps>;
+}
+
+multiclass Zn4WriteResZMM<SchedWrite SchedRW,
+                          list<ProcResourceKind> ExePorts, int Lat = 1,
+                          list<int> Res = [], int UOps = 1> {
+  defm : __Zn4WriteRes<SchedRW, ExePorts, Lat, Res, UOps>;
+}
+
+// For classes with folded loads.
+multiclass Zn4WriteResIntPair<X86FoldableSchedWrite SchedRW,
+                              list<ProcResourceKind> ExePorts, int Lat = 1,
+                              list<int> Res = [], int UOps = 1,
+                              int LoadUOps = 0, int LoadRes = 1> {
+  defm : __Zn4WriteResPair<SchedRW, ExePorts, Lat, Res, UOps,
+                           Znver4Model.LoadLatency,
+                           LoadUOps, Zn4AGU012, LoadRes>;
+}
+
+multiclass Zn4WriteResXMMPair<X86FoldableSchedWrite SchedRW,
+                              list<ProcResourceKind> ExePorts, int Lat = 1,
+                              list<int> Res = [], int UOps = 1,
+                              int LoadUOps = 0, int LoadRes = 1> {
+  defm : __Zn4WriteResPair<SchedRW, ExePorts, Lat, Res, UOps,
+                           Znver4Model.VecLoadLatency,
+                           LoadUOps, Zn4FPLd01, LoadRes>;
+}
+
+multiclass Zn4WriteResYMMPair<X86FoldableSchedWrite SchedRW,
+                              list<ProcResourceKind> ExePorts, int Lat = 1,
+                              list<int> Res = [], int UOps = 1,
+                              int LoadUOps = 0, int LoadRes = 1> {
+  defm : __Zn4WriteResPair<SchedRW, ExePorts, Lat, Res, UOps,
+                           Znver4Model.VecLoadLatency,
+                           LoadUOps, Zn4FPLd01, LoadRes>;
+}
+
+multiclass Zn4WriteResZMMPair<X86FoldableSchedWrite SchedRW,
+                              list<ProcResourceKind> ExePorts, int Lat = 1,
+                              list<int> Res = [], int UOps = 2,
+                              int LoadUOps = 0, int LoadRes = 1> {
+  defm : __Zn4WriteResPair<SchedRW, ExePorts, Lat, Res, UOps,
+                           Znver4Model.VecLoadLatency,
+                           LoadUOps, Zn4FPLd01, LoadRes>;
+}
+
+//===----------------------------------------------------------------------===//
+// Here be dragons.
+//===----------------------------------------------------------------------===//
+
+def : ReadAdvance<ReadAfterLd, Znver4Model.LoadLatency>;
+
+def : ReadAdvance<ReadAfterVecLd, Znver4Model.VecLoadLatency>;
+def : ReadAdvance<ReadAfterVecXLd, Znver4Model.VecLoadLatency>;
+def : ReadAdvance<ReadAfterVecYLd, Znver4Model.VecLoadLatency>;
+
+// AMD SOG 19h, 2.11 Floating-Point Unit
+// There is 1 cycle of added latency for a result to cross
+// from F to I or I to F domain.
+def : ReadAdvance<ReadInt2Fpu, -1>;
+
+// Instructions with both a load and a store folded are modeled as a folded
+// load + WriteRMW.
+defm : Zn4WriteResInt<WriteRMW, [Zn4AGU012, Zn4Store], Znver4Model.StoreLatency, [1, 1], 0>;
+
+// Loads, stores, and moves, not folded with other operations.
+defm : Zn4WriteResInt<WriteLoad, [Zn4AGU012, Zn4Load], !add(Znver4Model.LoadLatency, 1), [1, 1], 1>;
+
+// Model the effect of clobbering the read-write mask operand of the GATHER operation.
+// Does not cost anything by itself, only has latency, matching that of the WriteLoad,
+defm : Zn4WriteResInt<WriteVecMaskedGatherWriteback, [], !add(Znver4Model.LoadLatency, 1), [], 0>;
+
+def Zn4WriteMOVSlow : SchedWriteRes<[Zn4AGU012, Zn4Load]> {
+  let Latency = !add(Znver4Model.LoadLatency, 1);
+  let ResourceCycles = [3, 1];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteMOVSlow], (instrs MOV8rm, MOV8rm_NOREX, MOV16rm, MOVSX16rm16, MOVSX16rm32, MOVZX16rm16, MOVSX16rm8, MOVZX16rm8)>;
+
+defm : Zn4WriteResInt<WriteStore, [Zn4AGU012, Zn4Store], Znver4Model.StoreLatency, [1, 2], 1>;
+defm : Zn4WriteResInt<WriteStoreNT, [Zn4AGU012, Zn4Store], Znver4Model.StoreLatency, [1, 2], 1>;
+defm : Zn4WriteResInt<WriteMove, [Zn4ALU0123], 1, [4], 1>;
+
+// Treat misc copies as a move.
+def : InstRW<[WriteMove], (instrs COPY)>;
+
+def Zn4WriteMOVBE16rm : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4ALU0123]> {
+  let Latency = Znver4Model.LoadLatency;
+  let ResourceCycles = [1, 1, 4];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteMOVBE16rm], (instrs MOVBE16rm)>;
+
+def Zn4WriteMOVBEmr : SchedWriteRes<[Zn4ALU0123, Zn4AGU012, Zn4Store]> {
+  let Latency = Znver4Model.StoreLatency;
+  let ResourceCycles = [4, 1, 1];
+  let NumMicroOps = 2;
+}
+def : InstRW<[Zn4WriteMOVBEmr], (instrs MOVBE16mr, MOVBE32mr, MOVBE64mr)>;
+
+// Arithmetic.
+defm : Zn4WriteResIntPair<WriteALU, [Zn4ALU0123], 1, [1], 1>; // Simple integer ALU op.
+
+def Zn4WriteALUSlow : SchedWriteRes<[Zn4ALU0123]> {
+  let Latency = 1;
+  let ResourceCycles = [4];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteALUSlow], (instrs ADD8i8, ADD16i16, ADD32i32, ADD64i32,
+                                        AND8i8, AND16i16, AND32i32, AND64i32,
+                                         OR8i8,  OR16i16,  OR32i32,  OR64i32,
+                                        SUB8i8, SUB16i16, SUB32i32, SUB64i32,
+                                        XOR8i8, XOR16i16, XOR32i32, XOR64i32)>;
+
+def Zn4WriteMoveExtend : SchedWriteRes<[Zn4ALU0123]> {
+  let Latency = 1;
+  let ResourceCycles = [4];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteMoveExtend], (instrs MOVSX16rr16, MOVSX16rr32, MOVZX16rr16, MOVSX16rr8, MOVZX16rr8)>;
+
+def Zn4WriteMaterialize32bitImm: SchedWriteRes<[Zn4ALU0123]> {
+  let Latency = 1;
+  let ResourceCycles = [2];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteMaterialize32bitImm], (instrs MOV32ri, MOV32ri_alt, MOV64ri32)>;
+
+def Zn4WritePDEP_PEXT : SchedWriteRes<[Zn4ALU1]> {
+  let Latency = 3;
+  let ResourceCycles = [1];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WritePDEP_PEXT], (instrs PDEP32rr, PDEP64rr,
+                                          PEXT32rr, PEXT64rr)>;
+
+defm : Zn4WriteResIntPair<WriteADC, [Zn4ALU0123], 1, [4], 1>; // Integer ALU + flags op.
+
+def Zn4WriteADC8mr_SBB8mr : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4ALU0123, Zn4Store]> {
+  let Latency = 1;
+  let ResourceCycles = [1, 1, 7, 1];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteADC8mr_SBB8mr], (instrs ADC8mr, SBB8mr)>;
+
+// This is for simple LEAs with one or two input operands.
+defm : Zn4WriteResInt<WriteLEA, [Zn4AGU012], 1, [1], 1>;     // LEA instructions can't fold loads.
+
+// This write is used for slow LEA instructions.
+def Zn4Write3OpsLEA : SchedWriteRes<[Zn4ALU0123]> {
+  let Latency = 2;
+  let ResourceCycles = [1];
+  let NumMicroOps = 2;
+}
+
+// On Znver4, a slow LEA is either a 3Ops LEA (base, index, offset),
+// or an LEA with a `Scale` value different than 1.
+def Zn4SlowLEAPredicate : MCSchedPredicate<
+  CheckAny<[
+    // A 3-operand LEA (base, index, offset).
+    IsThreeOperandsLEAFn,
+    // An LEA with a "Scale" different than 1.
+    CheckAll<[
+      CheckIsImmOperand<2>,
+      CheckNot<CheckImmOperand<2, 1>>
+    ]>
+  ]>
+>;
+
+def Zn4WriteLEA : SchedWriteVariant<[
+    SchedVar<Zn4SlowLEAPredicate, [Zn4Write3OpsLEA]>,
+    SchedVar<NoSchedPred,         [WriteLEA]>
+]>;
+
+def : InstRW<[Zn4WriteLEA], (instrs LEA32r, LEA64r, LEA64_32r)>;
+
+def Zn4SlowLEA16r : SchedWriteRes<[Zn4ALU0123]> {
+  let Latency = 2; // FIXME: not from llvm-exegesis
+  let ResourceCycles = [4];
+  let NumMicroOps = 2;
+}
+
+def : InstRW<[Zn4SlowLEA16r], (instrs LEA16r)>;
+
+// Integer multiplication
+defm : Zn4WriteResIntPair<WriteIMul8, [Zn4Multiplier], 3, [3], 1>; // Integer 8-bit multiplication.
+defm : Zn4WriteResIntPair<WriteIMul16, [Zn4Multiplier], 3, [3], 3, /*LoadUOps=*/1>; // Integer 16-bit multiplication.
+defm : Zn4WriteResIntPair<WriteIMul16Imm, [Zn4Multiplier], 4, [4], 2>; // Integer 16-bit multiplication by immediate.
+defm : Zn4WriteResIntPair<WriteIMul16Reg, [Zn4Multiplier], 3, [1], 1>; // Integer 16-bit multiplication by register.
+defm : Zn4WriteResIntPair<WriteIMul32, [Zn4Multiplier], 3, [3], 2>;    // Integer 32-bit multiplication.
+defm : Zn4WriteResIntPair<WriteMULX32, [Zn4Multiplier], 3, [1], 2>;    // Integer 32-bit Unsigned Multiply Without Affecting Flags.
+defm : Zn4WriteResIntPair<WriteIMul32Imm, [Zn4Multiplier], 3, [1], 1>; // Integer 32-bit multiplication by immediate.
+defm : Zn4WriteResIntPair<WriteIMul32Reg, [Zn4Multiplier], 3, [1], 1>; // Integer 32-bit multiplication by register.
+defm : Zn4WriteResIntPair<WriteIMul64, [Zn4Multiplier], 3, [3], 2>;    // Integer 64-bit multiplication.
+defm : Zn4WriteResIntPair<WriteMULX64, [Zn4Multiplier], 3, [1], 2>;    // Integer 32-bit Unsigned Multiply Without Affecting Flags.
+defm : Zn4WriteResIntPair<WriteIMul64Imm, [Zn4Multiplier], 3, [1], 1>; // Integer 64-bit multiplication by immediate.
+defm : Zn4WriteResIntPair<WriteIMul64Reg, [Zn4Multiplier], 3, [1], 1>; // Integer 64-bit multiplication by register.
+defm : Zn4WriteResInt<WriteIMulHLd, [], !add(4, Znver4Model.LoadLatency), [], 0>;  // Integer multiplication, high part.
+defm : Zn4WriteResInt<WriteIMulH, [], 4, [], 0>;  // Integer multiplication, high part.
+
+defm : Zn4WriteResInt<WriteBSWAP32, [Zn4ALU0123], 1, [1], 1>; // Byte Order (Endianness) 32-bit Swap.
+defm : Zn4WriteResInt<WriteBSWAP64, [Zn4ALU0123], 1, [1], 1>; // Byte Order (Endianness) 64-bit Swap.
+
+defm : Zn4WriteResIntPair<WriteCMPXCHG, [Zn4ALU0123], 3, [12], 5>; // Compare and set, compare and swap.
+
+def Zn4WriteCMPXCHG8rr : SchedWriteRes<[Zn4ALU0123]> {
+  let Latency = 3;
+  let ResourceCycles = [12];
+  let NumMicroOps = 3;
+}
+def : InstRW<[Zn4WriteCMPXCHG8rr], (instrs CMPXCHG8rr)>;
+
+defm : Zn4WriteResInt<WriteCMPXCHGRMW, [Zn4ALU0123], 3, [12], 6>;     // Compare and set, compare and swap.
+
+def Zn4WriteCMPXCHG8rm_LCMPXCHG8 : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4ALU0123]> {
+  let Latency = !add(Znver4Model.LoadLatency, Zn4WriteCMPXCHG8rr.Latency);
+  let ResourceCycles = [1, 1, 12];
+  let NumMicroOps = !add(Zn4WriteCMPXCHG8rr.NumMicroOps, 2);
+}
+def : InstRW<[Zn4WriteCMPXCHG8rm_LCMPXCHG8], (instrs CMPXCHG8rm, LCMPXCHG8)>;
+
+def Zn4WriteCMPXCHG8B : SchedWriteRes<[Zn4ALU0123]> {
+  let Latency = 3; // FIXME: not from llvm-exegesis
+  let ResourceCycles = [24];
+  let NumMicroOps = 19;
+}
+def : InstRW<[Zn4WriteCMPXCHG8B], (instrs CMPXCHG8B)>;
+
+def Zn4WriteCMPXCHG16B_LCMPXCHG16B : SchedWriteRes<[Zn4ALU0123]> {
+  let Latency = 4; // FIXME: not from llvm-exegesis
+  let ResourceCycles = [59];
+  let NumMicroOps = 28;
+}
+def : InstRW<[Zn4WriteCMPXCHG16B_LCMPXCHG16B], (instrs CMPXCHG16B, LCMPXCHG16B)>;
+
+def Zn4WriteWriteXCHGUnrenameable : SchedWriteRes<[Zn4ALU0123]> {
+  let Latency = 1;
+  let ResourceCycles = [2];
+  let NumMicroOps = 2;
+}
+def : InstRW<[Zn4WriteWriteXCHGUnrenameable], (instrs XCHG8rr, XCHG16rr, XCHG16ar)>;
+
+def Zn4WriteXCHG8rm_XCHG16rm : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4ALU0123]> {
+  let Latency = !add(Znver4Model.LoadLatency, 3); // FIXME: not from llvm-exegesis
+  let ResourceCycles = [1, 1, 2];
+  let NumMicroOps = 5;
+}
+def : InstRW<[Zn4WriteXCHG8rm_XCHG16rm], (instrs XCHG8rm, XCHG16rm)>;
+
+def Zn4WriteXCHG32rm_XCHG64rm : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4ALU0123]> {
+  let Latency = !add(Znver4Model.LoadLatency, 2); // FIXME: not from llvm-exegesis
+  let ResourceCycles = [1, 1, 2];
+  let NumMicroOps = 2;
+}
+def : InstRW<[Zn4WriteXCHG32rm_XCHG64rm], (instrs XCHG32rm, XCHG64rm)>;
+
+// Integer division.
+// FIXME: uops for 8-bit division measures as 2. for others it's a guess.
+// FIXME: latency for 8-bit division measures as 10. for others it's a guess.
+defm : Zn4WriteResIntPair<WriteDiv8, [Zn4Divider], 10, [10], 2>;
+defm : Zn4WriteResIntPair<WriteDiv16, [Zn4Divider], 11, [11], 2>;
+defm : Zn4WriteResIntPair<WriteDiv32, [Zn4Divider], 13, [13], 2>;
+defm : Zn4WriteResIntPair<WriteDiv64, [Zn4Divider], 17, [17], 2>;
+defm : Zn4WriteResIntPair<WriteIDiv8, [Zn4Divider], 10, [10], 2>;
+defm : Zn4WriteResIntPair<WriteIDiv16, [Zn4Divider], 11, [11], 2>;
+defm : Zn4WriteResIntPair<WriteIDiv32, [Zn4Divider], 13, [13], 2>;
+defm : Zn4WriteResIntPair<WriteIDiv64, [Zn4Divider], 17, [17], 2>;
+
+defm : Zn4WriteResIntPair<WriteBSF, [Zn4ALU1], 1, [1], 6, /*LoadUOps=*/1>; // Bit scan forward.
+defm : Zn4WriteResIntPair<WriteBSR, [Zn4ALU1], 1, [1], 6, /*LoadUOps=*/1>; // Bit scan reverse.
+
+defm : Zn4WriteResIntPair<WritePOPCNT, [Zn4ALU0123], 1, [1], 1>; // Bit population count.
+
+def Zn4WritePOPCNT16rr : SchedWriteRes<[Zn4ALU0123]> {
+  let Latency = 1;
+  let ResourceCycles = [4];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WritePOPCNT16rr], (instrs POPCNT16rr)>;
+
+defm : Zn4WriteResIntPair<WriteLZCNT, [Zn4ALU0123], 1, [1], 1>; // Leading zero count.
+
+def Zn4WriteLZCNT16rr : SchedWriteRes<[Zn4ALU0123]> {
+  let Latency = 1;
+  let ResourceCycles = [4];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteLZCNT16rr], (instrs LZCNT16rr)>;
+
+defm : Zn4WriteResIntPair<WriteTZCNT, [Zn4ALU12], 2, [1], 2>; // Trailing zero count.
+
+def Zn4WriteTZCNT16rr : SchedWriteRes<[Zn4ALU0123]> {
+  let Latency = 2;
+  let ResourceCycles = [4];
+  let NumMicroOps = 2;
+}
+def : InstRW<[Zn4WriteTZCNT16rr], (instrs TZCNT16rr)>;
+
+defm : Zn4WriteResIntPair<WriteCMOV, [Zn4ALU03], 1, [1], 1>; // Conditional move.
+defm : Zn4WriteResInt<WriteFCMOV, [Zn4ALU0123], 7, [28], 7>; // FIXME: not from llvm-exegesis // X87 conditional move.
+defm : Zn4WriteResInt<WriteSETCC, [Zn4ALU03], 1, [2], 1>; // Set register based on condition code.
+defm : Zn4WriteResInt<WriteSETCCStore, [Zn4ALU03, Zn4AGU012, Zn4Store], 2, [2, 1, 1], 2>; // FIXME: latency not from llvm-exegesis
+defm : Zn4WriteResInt<WriteLAHFSAHF, [Zn4ALU3], 1, [1], 1>; // Load/Store flags in AH.
+
+defm : Zn4WriteResInt<WriteBitTest, [Zn4ALU12], 1, [1], 1>; // Bit Test
+defm : Zn4WriteResInt<WriteBitTestImmLd, [Zn4AGU012, Zn4Load, Zn4ALU12], !add(Znver4Model.LoadLatency, 1), [1, 1, 1], 2>;
+defm : Zn4WriteResInt<WriteBitTestRegLd, [Zn4AGU012, Zn4Load, Zn4ALU12], !add(Znver4Model.LoadLatency, 1), [1, 1, 1], 7>;
+
+defm : Zn4WriteResInt<WriteBitTestSet, [Zn4ALU12], 2, [2], 2>; // Bit Test + Set
+defm : Zn4WriteResInt<WriteBitTestSetImmLd, [Zn4AGU012, Zn4Load, Zn4ALU12], !add(Znver4Model.LoadLatency, 2), [1, 1, 1], 4>;
+defm : Zn4WriteResInt<WriteBitTestSetRegLd, [Zn4AGU012, Zn4Load, Zn4ALU12], !add(Znver4Model.LoadLatency, 2), [1, 1, 1], 9>;
+
+// Integer shifts and rotates.
+defm : Zn4WriteResIntPair<WriteShift, [Zn4ALU12], 1, [1], 1, /*LoadUOps=*/1>;
+defm : Zn4WriteResIntPair<WriteShiftCL, [Zn4ALU12], 1, [1], 1, /*LoadUOps=*/1>;
+defm : Zn4WriteResIntPair<WriteRotate, [Zn4ALU12], 1, [1], 1, /*LoadUOps=*/1>;
+
+def Zn4WriteRotateR1 : SchedWriteRes<[Zn4ALU12]> {
+  let Latency = 1;
+  let ResourceCycles = [2];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteRotateR1], (instrs RCL8r1, RCL16r1, RCL32r1, RCL64r1,
+                                         RCR8r1, RCR16r1, RCR32r1, RCR64r1)>;
+
+def Zn4WriteRotateM1 : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4ALU12]> {
+  let Latency = !add(Znver4Model.LoadLatency, Zn4WriteRotateR1.Latency);
+  let ResourceCycles = [1, 1, 2];
+  let NumMicroOps = !add(Zn4WriteRotateR1.NumMicroOps, 1);
+}
+def : InstRW<[Zn4WriteRotateM1], (instrs RCL8m1, RCL16m1, RCL32m1, RCL64m1,
+                                         RCR8m1, RCR16m1, RCR32m1, RCR64m1)>;
+
+def Zn4WriteRotateRightRI : SchedWriteRes<[Zn4ALU12]> {
+  let Latency = 3;
+  let ResourceCycles = [6];
+  let NumMicroOps = 7;
+}
+def : InstRW<[Zn4WriteRotateRightRI], (instrs RCR8ri, RCR16ri, RCR32ri, RCR64ri)>;
+
+def Zn4WriteRotateRightMI : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4ALU12]> {
+  let Latency = !add(Znver4Model.LoadLatency, Zn4WriteRotateRightRI.Latency);
+  let ResourceCycles = [1, 1, 8];
+  let NumMicroOps = !add(Zn4WriteRotateRightRI.NumMicroOps, 3);
+}
+def : InstRW<[Zn4WriteRotateRightMI], (instrs RCR8mi, RCR16mi, RCR32mi, RCR64mi)>;
+
+def Zn4WriteRotateLeftRI : SchedWriteRes<[Zn4ALU12]> {
+  let Latency = 4;
+  let ResourceCycles = [8];
+  let NumMicroOps = 9;
+}
+def : InstRW<[Zn4WriteRotateLeftRI], (instrs RCL8ri, RCL16ri, RCL32ri, RCL64ri)>;
+
+def Zn4WriteRotateLeftMI : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4ALU12]> {
+  let Latency = !add(Znver4Model.LoadLatency, Zn4WriteRotateLeftRI.Latency);
+  let ResourceCycles = [1, 1, 8];
+  let NumMicroOps = !add(Zn4WriteRotateLeftRI.NumMicroOps, 2);
+}
+def : InstRW<[Zn4WriteRotateLeftMI], (instrs RCL8mi, RCL16mi, RCL32mi, RCL64mi)>;
+
+defm : Zn4WriteResIntPair<WriteRotateCL, [Zn4ALU12], 1, [1], 1, /*LoadUOps=*/1>;
+
+def Zn4WriteRotateRightRCL : SchedWriteRes<[Zn4ALU12]> {
+  let Latency = 3;
+  let ResourceCycles = [6];
+  let NumMicroOps = 7;
+}
+def : InstRW<[Zn4WriteRotateRightRCL], (instrs RCR8rCL, RCR16rCL, RCR32rCL, RCR64rCL)>;
+
+def Zn4WriteRotateRightMCL : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4ALU12]> {
+  let Latency = !add(Znver4Model.LoadLatency, Zn4WriteRotateRightRCL.Latency);
+  let ResourceCycles = [1, 1, 8];
+  let NumMicroOps = !add(Zn4WriteRotateRightRCL.NumMicroOps, 2);
+}
+def : InstRW<[Zn4WriteRotateRightMCL], (instrs RCR8mCL, RCR16mCL, RCR32mCL, RCR64mCL)>;
+
+def Zn4WriteRotateLeftRCL : SchedWriteRes<[Zn4ALU12]> {
+  let Latency = 4;
+  let ResourceCycles = [8];
+  let NumMicroOps = 9;
+}
+def : InstRW<[Zn4WriteRotateLeftRCL], (instrs RCL8rCL, RCL16rCL, RCL32rCL, RCL64rCL)>;
+
+def Zn4WriteRotateLeftMCL : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4ALU12]> {
+  let Latency = !add(Znver4Model.LoadLatency, Zn4WriteRotateLeftRCL.Latency);
+  let ResourceCycles = [1, 1, 8];
+  let NumMicroOps = !add(Zn4WriteRotateLeftRCL.NumMicroOps, 2);
+}
+def : InstRW<[Zn4WriteRotateLeftMCL], (instrs RCL8mCL, RCL16mCL, RCL32mCL, RCL64mCL)>;
+
+// Double shift instructions.
+defm : Zn4WriteResInt<WriteSHDrri, [Zn4ALU12], 2, [3], 4>;
+defm : Zn4WriteResInt<WriteSHDrrcl, [Zn4ALU12], 2, [3], 5>;
+defm : Zn4WriteResInt<WriteSHDmri, [Zn4AGU012, Zn4Load, Zn4ALU12], !add(Znver4Model.LoadLatency, 2), [1, 1, 4], 6>;
+defm : Zn4WriteResInt<WriteSHDmrcl, [Zn4AGU012, Zn4Load, Zn4ALU12], !add(Znver4Model.LoadLatency, 2), [1, 1, 4], 6>;
+
+// BMI1 BEXTR/BLS, BMI2 BZHI
+defm : Zn4WriteResIntPair<WriteBEXTR, [Zn4ALU12], 1, [1], 1, /*LoadUOps=*/1>;
+defm : Zn4WriteResIntPair<WriteBLS, [Zn4ALU0123], 1, [1], 1, /*LoadUOps=*/1>;
+defm : Zn4WriteResIntPair<WriteBZHI, [Zn4ALU12], 1, [1], 1, /*LoadUOps=*/1>;
+
+// Idioms that clear a register, like xorps %xmm0, %xmm0.
+// These can often bypass execution ports completely.
+defm : Zn4WriteResInt<WriteZero, [Zn4ALU0123], 0, [0], 1>;
+
+// Branches don't produce values, so they have no latency, but they still
+// consume resources. Indirect branches can fold loads.
+defm : Zn4WriteResIntPair<WriteJump, [Zn4BRU01], 1, [1], 1>; // FIXME: not from llvm-exegesis
+
+// Floating point. This covers both scalar and vector operations.
+defm : Zn4WriteResInt<WriteFLD0, [Zn4FPLd01, Zn4Load, Zn4FP1], !add(Znver4Model.LoadLatency, 4), [1, 1, 1], 1>;
+defm : Zn4WriteResInt<WriteFLD1, [Zn4FPLd01, Zn4Load, Zn4FP1], !add(Znver4Model.LoadLatency, 7), [1, 1, 1], 1>;
+defm : Zn4WriteResInt<WriteFLDC, [Zn4FPLd01, Zn4Load, Zn4FP1], !add(Znver4Model.LoadLatency, 7), [1, 1, 1], 1>;
+defm : Zn4WriteResXMM<WriteFLoad, [Zn4FPLd01, Zn4Load], !add(Znver4Model.VecLoadLatency, 1), [1, 1], 1>;
+defm : Zn4WriteResXMM<WriteFLoadX, [Zn4FPLd01, Zn4Load], !add(Znver4Model.VecLoadLatency, 1), [1, 1], 1>;
+defm : Zn4WriteResYMM<WriteFLoadY, [Zn4FPLd01, Zn4Load], !add(Znver4Model.VecLoadLatency, 1), [1, 1], 1>;
+defm : Zn4WriteResXMM<WriteFMaskedLoad, [Zn4FPLd01, Zn4Load], !add(Znver4Model.VecLoadLatency, 1), [1, 1], 1>;
+defm : Zn4WriteResYMM<WriteFMaskedLoadY, [Zn4FPLd01, Zn4Load], !add(Znver4Model.VecLoadLatency, 1), [1, 1], 1>;
+defm : Zn4WriteResXMM<WriteFStore, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [1, 1], 1>;
+
+def Zn4WriteWriteFStoreMMX : SchedWriteRes<[Zn4FPSt, Zn4Store]> {
+  let Latency = 2; // FIXME: not from llvm-exegesis
+  let ResourceCycles = [1, 1];
+  let NumMicroOps = 2;
+}
+def : InstRW<[Zn4WriteWriteFStoreMMX], (instrs MOVHPDmr,  MOVHPSmr,
+                                               VMOVHPDmr, VMOVHPSmr)>;
+
+defm : Zn4WriteResXMM<WriteFStoreX, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [1, 1], 1>;
+defm : Zn4WriteResYMM<WriteFStoreY, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [1, 1], 1>;
+defm : Zn4WriteResXMM<WriteFStoreNT, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [1, 1], 1>;
+defm : Zn4WriteResXMM<WriteFStoreNTX, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [1, 1], 1>;
+defm : Zn4WriteResYMM<WriteFStoreNTY, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [1, 1], 1>;
+
+defm : Zn4WriteResXMM<WriteFMaskedStore32, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [6, 1], 18>;
+defm : Zn4WriteResXMM<WriteFMaskedStore64, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [4, 1], 10>;
+defm : Zn4WriteResYMM<WriteFMaskedStore32Y, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [12, 1], 42>;
+defm : Zn4WriteResYMM<WriteFMaskedStore64Y, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [6, 1], 18>;
+
+defm : Zn4WriteResXMMPair<WriteFAdd, [Zn4FPFAdd01], 3, [1], 1>;  // Floating point add/sub.
+
+def Zn4WriteX87Arith : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4FPU0123]> {
+  let Latency = !add(Znver4Model.LoadLatency, 1); // FIXME: not from llvm-exegesis
+  let ResourceCycles = [1, 1, 24];
+  let NumMicroOps = 2;
+}
+def : InstRW<[Zn4WriteX87Arith], (instrs ADD_FI16m, ADD_FI32m,
+                                         SUB_FI16m, SUB_FI32m,
+                                         SUBR_FI16m, SUBR_FI32m,
+                                         MUL_FI16m, MUL_FI32m)>;
+
+def Zn4WriteX87Div : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4FPU0123]> {
+  let Latency = !add(Znver4Model.LoadLatency, 1); // FIXME: not from llvm-exegesis
+  let ResourceCycles = [1, 1, 62];
+  let NumMicroOps = 2;
+}
+def : InstRW<[Zn4WriteX87Div], (instrs DIV_FI16m, DIV_FI32m,
+                                       DIVR_FI16m, DIVR_FI32m)>;
+
+defm : Zn4WriteResXMMPair<WriteFAddX, [Zn4FPFAdd01], 3, [1], 1>; // Floating point add/sub (XMM).
+defm : Zn4WriteResYMMPair<WriteFAddY, [Zn4FPFAdd01], 3, [1], 1>; // Floating point add/sub (YMM).
+defm : Zn4WriteResZMMPair<WriteFAddZ, [Zn4FPFAdd01], 3, [2], 1>; // Floating point add/sub (ZMM).
+defm : Zn4WriteResXMMPair<WriteFAdd64, [Zn4FPFAdd01], 3, [1], 1>;  // Floating point double add/sub.
+defm : Zn4WriteResXMMPair<WriteFAdd64X, [Zn4FPFAdd01], 3, [1], 1>; // Floating point double add/sub (XMM).
+defm : Zn4WriteResYMMPair<WriteFAdd64Y, [Zn4FPFAdd01], 3, [1], 1>; // Floating point double add/sub (YMM).
+defm : Zn4WriteResZMMPair<WriteFAdd64Z, [Zn4FPFAdd01], 3, [2], 1>; // Floating point double add/sub (ZMM).
+defm : Zn4WriteResXMMPair<WriteFCmp, [Zn4FPFMul01], 2, [2], 1>;  // Floating point compare.
+defm : Zn4WriteResXMMPair<WriteFCmpX, [Zn4FPFMul01], 1, [1], 1>; // Floating point compare (XMM).
+defm : Zn4WriteResYMMPair<WriteFCmpY, [Zn4FPFMul01], 2, [2], 1>; // Floating point compare (YMM).
+defm : Zn4WriteResZMMPair<WriteFCmpZ, [Zn4FPFMul01], 2, [4], 1>; // Floating point compare (ZMM).
+defm : Zn4WriteResXMMPair<WriteFCmp64, [Zn4FPFMul01], 1, [1], 1>;  // Floating point double compare.
+defm : Zn4WriteResXMMPair<WriteFCmp64X, [Zn4FPFMul01], 1, [1], 1>; // Floating point double compare (XMM).
+defm : Zn4WriteResYMMPair<WriteFCmp64Y, [Zn4FPFMul01], 2, [2], 1>; // Floating point double compare (YMM).
+defm : Zn4WriteResZMMPair<WriteFCmp64Z, [Zn4FPFMul01], 2, [4], 1>; // Floating point double compare (ZMM).
+defm : Zn4WriteResXMMPair<WriteFCom, [Zn4FPFMul01], 3, [2], 1>; // FIXME: latency not from llvm-exegesis  // Floating point compare to flags (X87).
+defm : Zn4WriteResXMMPair<WriteFComX, [Zn4FPFMul01], 4, [2], 2>;  // FIXME: latency not from llvm-exegesis // Floating point compare to flags (SSE).
+defm : Zn4WriteResXMMPair<WriteFMul, [Zn4FPFMul01], 3, [1], 1>;  // Floating point multiplication.
+defm : Zn4WriteResXMMPair<WriteFMulX, [Zn4FPFMul01], 3, [1], 1>; // Floating point multiplication (XMM).
+defm : Zn4WriteResYMMPair<WriteFMulY, [Zn4FPFMul01], 3, [1], 1>; // Floating point multiplication (YMM).
+defm : Zn4WriteResZMMPair<WriteFMulZ, [Zn4FPFMul01], 3, [2], 1>; // Floating point multiplication (ZMM).
+defm : Zn4WriteResXMMPair<WriteFMul64, [Zn4FPFMul01], 3, [1], 1>;  // Floating point double multiplication.
+defm : Zn4WriteResXMMPair<WriteFMul64X, [Zn4FPFMul01], 3, [1], 1>; // Floating point double multiplication (XMM).
+defm : Zn4WriteResYMMPair<WriteFMul64Y, [Zn4FPFMul01], 3, [1], 1>; // Floating point double multiplication (YMM).
+defm : Zn4WriteResZMMPair<WriteFMul64Z, [Zn4FPFMul01], 3, [2], 1>; // Floating point double multiplication (ZMM).
+defm : Zn4WriteResXMMPair<WriteFDiv, [Zn4FPFDiv], 11, [3], 1>;  // Floating point division.
+defm : Zn4WriteResXMMPair<WriteFDivX, [Zn4FPFDiv], 11, [3], 1>; // Floating point division (XMM).
+defm : Zn4WriteResYMMPair<WriteFDivY, [Zn4FPFDiv], 11, [3], 1>; // Floating point division (YMM).
+defm : Zn4WriteResZMMPair<WriteFDivZ, [Zn4FPFDiv], 11, [6], 1>; // Floating point division (ZMM).
+defm : Zn4WriteResXMMPair<WriteFDiv64, [Zn4FPFDiv], 13, [5], 1>;  // Floating point double division.
+defm : Zn4WriteResXMMPair<WriteFDiv64X, [Zn4FPFDiv], 13, [5], 1>; // Floating point double division (XMM).
+defm : Zn4WriteResYMMPair<WriteFDiv64Y, [Zn4FPFDiv], 13, [5], 1>; // Floating point double division (YMM).
+defm : Zn4WriteResZMMPair<WriteFDiv64Z, [Zn4FPFDiv], 13, [10], 1>; // Floating point double division (ZMM).
+defm : Zn4WriteResXMMPair<WriteFSqrt, [Zn4FPFDiv], 15, [5], 1>;   // Floating point square root.
+defm : Zn4WriteResXMMPair<WriteFSqrtX, [Zn4FPFDiv], 15, [5], 1>;  // Floating point square root (XMM).
+defm : Zn4WriteResYMMPair<WriteFSqrtY, [Zn4FPFDiv], 15, [5], 1>;  // Floating point square root (YMM).
+defm : Zn4WriteResZMMPair<WriteFSqrtZ, [Zn4FPFDiv], 15, [10], 1>;  // Floating point square root (ZMM).
+defm : Zn4WriteResXMMPair<WriteFSqrt64, [Zn4FPFDiv], 21, [9], 1>;  // Floating point double square root.
+defm : Zn4WriteResXMMPair<WriteFSqrt64X, [Zn4FPFDiv], 21, [9], 1>; // Floating point double square root (XMM).
+defm : Zn4WriteResYMMPair<WriteFSqrt64Y, [Zn4FPFDiv], 21, [9], 1>; // Floating point double square root (YMM).
+defm : Zn4WriteResZMMPair<WriteFSqrt64Z, [Zn4FPFDiv], 21, [18], 1>; // Floating point double square root (ZMM).
+defm : Zn4WriteResXMMPair<WriteFSqrt80, [Zn4FPFDiv], 22, [23], 1>; // FIXME: latency not from llvm-exegesis  // Floating point long double square root.
+defm : Zn4WriteResXMMPair<WriteFRcp, [Zn4FPFMul01], 4, [1], 1>;  // Floating point reciprocal estimate.
+defm : Zn4WriteResXMMPair<WriteFRcpX, [Zn4FPFMul01], 4, [1], 1>; // Floating point reciprocal estimate (XMM).
+defm : Zn4WriteResYMMPair<WriteFRcpY, [Zn4FPFMul01], 5, [1], 1>; // Floating point reciprocal estimate (YMM).
+defm : Zn4WriteResZMMPair<WriteFRcpZ, [Zn4FPFMul01], 5, [2], 1>; // Floating point reciprocal estimate (ZMM).
+defm : Zn4WriteResXMMPair<WriteFRsqrt, [Zn4FPFDiv], 4, [1], 1>;  // Floating point reciprocal square root estimate.
+defm : Zn4WriteResXMMPair<WriteFRsqrtX, [Zn4FPFDiv], 4, [1], 1>; // Floating point reciprocal square root estimate (XMM).
+defm : Zn4WriteResYMMPair<WriteFRsqrtY, [Zn4FPFDiv], 4, [1], 1>; // Floating point reciprocal square root estimate (YMM).
+defm : Zn4WriteResZMMPair<WriteFRsqrtZ, [Zn4FPFDiv], 5, [2], 1>; // Floating point reciprocal square root estimate (ZMM).
+defm : Zn4WriteResXMMPair<WriteFMA, [Zn4FPFMul01], 4, [2], 1>;  // Fused Multiply Add.
+defm : Zn4WriteResXMMPair<WriteFMAX, [Zn4FPFMul01], 4, [2], 1>; // Fused Multiply Add (XMM).
+defm : Zn4WriteResYMMPair<WriteFMAY, [Zn4FPFMul01], 4, [2], 1>; // Fused Multiply Add (YMM).
+defm : Zn4WriteResZMMPair<WriteFMAZ, [Zn4FPFMul01], 4, [4], 1>; // Fused Multiply Add (ZMM).
+defm : Zn4WriteResXMMPair<WriteDPPD, [Zn4FPFMul01], 7, [6], 3, /*LoadUOps=*/2>; // Floating point double dot product.
+defm : Zn4WriteResXMMPair<WriteDPPS, [Zn4FPFMul01], 11, [8], 8, /*LoadUOps=*/2>; // Floating point single dot product.
+defm : Zn4WriteResYMMPair<WriteDPPSY, [Zn4FPFMul01], 11, [8], 7, /*LoadUOps=*/1>; // Floating point single dot product (YMM).
+defm : Zn4WriteResXMMPair<WriteFSign, [Zn4FPFMul01], 1, [2], 1>; // FIXME: latency not from llvm-exegesis  // Floating point fabs/fchs.
+defm : Zn4WriteResXMMPair<WriteFRnd, [Zn4FPFCvt01], 3, [1], 1>; // Floating point rounding.
+defm : Zn4WriteResYMMPair<WriteFRndY, [Zn4FPFCvt01], 3, [1], 1>; // Floating point rounding (YMM).
+defm : Zn4WriteResZMMPair<WriteFRndZ, [Zn4FPFCvt01], 3, [2], 1>; // Floating point rounding (ZMM).
+
+defm : Zn4WriteResXMMPair<WriteFLogic, [Zn4FPVMisc0123], 1, [1], 1>; // Floating point and/or/xor logicals.
+defm : Zn4WriteResYMMPair<WriteFLogicY, [Zn4FPVMisc0123], 1, [1], 1>; // Floating point and/or/xor logicals (YMM).
+defm : Zn4WriteResZMMPair<WriteFLogicZ, [Zn4FPVMisc0123], 1, [2], 1>; // Floating point and/or/xor logicals (ZMM).
+defm : Zn4WriteResXMMPair<WriteFTest, [Zn4FPFMisc12], 1, [2], 2>; // FIXME: latency not from llvm-exegesis // Floating point TEST instructions.
+defm : Zn4WriteResYMMPair<WriteFTestY, [Zn4FPFMisc12], 1, [2], 2>; // FIXME: latency not from llvm-exegesis // Floating point TEST instructions (YMM).
+defm : Zn4WriteResZMMPair<WriteFTestZ, [Zn4FPFMisc12], 1, [4], 1>; // FIXME: latency not from llvm-exegesis // Floating point TEST instructions (ZMM).
+defm : Zn4WriteResXMMPair<WriteFShuffle, [Zn4FPVShuf01], 1, [1], 1>; // Floating point vector shuffles.
+defm : Zn4WriteResYMMPair<WriteFShuffleY, [Zn4FPVShuf01], 1, [1], 1>; // Floating point vector shuffles (YMM).
+defm : Zn4WriteResZMMPair<WriteFShuffleZ, [Zn4FPVShuf01], 1, [2], 1>; // Floating point vector shuffles (ZMM).
+defm : Zn4WriteResXMMPair<WriteFVarShuffle, [Zn4FPVShuf01], 3, [1], 1>; // Floating point vector variable shuffles.
+defm : Zn4WriteResYMMPair<WriteFVarShuffleY, [Zn4FPVShuf01], 3, [1], 1>; // Floating point vector variable shuffles (YMM).
+defm : Zn4WriteResZMMPair<WriteFVarShuffleZ, [Zn4FPVShuf01], 3, [2], 1>; // Floating point vector variable shuffles (ZMM).
+defm : Zn4WriteResXMMPair<WriteFBlend, [Zn4FPFMul01], 1, [1], 1>; // Floating point vector blends.
+defm : Zn4WriteResYMMPair<WriteFBlendY, [Zn4FPFMul01], 1, [1], 1>; // Floating point vector blends (YMM).
+defm : Zn4WriteResZMMPair<WriteFBlendZ, [Zn4FPFMul01], 1, [2], 1>; // Floating point vector blends (ZMM).
+defm : Zn4WriteResXMMPair<WriteFVarBlend, [Zn4FPFMul01], 1, [1], 1>; // Fp vector variable blends.
+defm : Zn4WriteResYMMPair<WriteFVarBlendY, [Zn4FPFMul01], 1, [1], 1>; // Fp vector variable blends (YMM).
+defm : Zn4WriteResZMMPair<WriteFVarBlendZ, [Zn4FPFMul01], 1, [2], 1>; // Fp vector variable blends (ZMM).
+
+// Horizontal Add/Sub (float and integer)
+defm : Zn4WriteResXMMPair<WriteFHAdd, [Zn4FPFAdd0], 4, [2], 3>;
+defm : Zn4WriteResYMMPair<WriteFHAddY, [Zn4FPFAdd0], 4, [2], 3, /*LoadUOps=*/1>;
+defm : Zn4WriteResZMMPair<WriteFHAddZ, [Zn4FPFAdd0], 6, [4], 3, /*LoadUOps=*/1>;
+defm : Zn4WriteResXMMPair<WritePHAdd, [Zn4FPVAdd0], 2, [2], 3, /*LoadUOps=*/1>;
+defm : Zn4WriteResXMMPair<WritePHAddX, [Zn4FPVAdd0], 2, [2], 3>;
+defm : Zn4WriteResYMMPair<WritePHAddY, [Zn4FPVAdd0], 3, [3], 3, /*LoadUOps=*/1>;
+defm : Zn4WriteResZMMPair<WritePHAddZ, [Zn4FPVAdd0], 2, [4], 3, /*LoadUOps=*/1>;
+
+// Vector integer operations.
+defm : Zn4WriteResXMM<WriteVecLoad, [Zn4FPLd01, Zn4Load], !add(Znver4Model.VecLoadLatency, 1), [1, 1], 1>;
+defm : Zn4WriteResXMM<WriteVecLoadX, [Zn4FPLd01, Zn4Load], !add(Znver4Model.VecLoadLatency, 1), [1, 1], 1>;
+defm : Zn4WriteResYMM<WriteVecLoadY, [Zn4FPLd01, Zn4Load], !add(Znver4Model.VecLoadLatency, 1), [1, 1], 1>;
+defm : Zn4WriteResXMM<WriteVecLoadNT, [Zn4FPLd01, Zn4Load], !add(Znver4Model.VecLoadLatency, 1), [1, 1], 1>;
+defm : Zn4WriteResYMM<WriteVecLoadNTY, [Zn4FPLd01, Zn4Load], !add(Znver4Model.VecLoadLatency, 1), [1, 1], 1>;
+defm : Zn4WriteResXMM<WriteVecMaskedLoad, [Zn4FPLd01, Zn4Load], !add(Znver4Model.VecLoadLatency, 1), [1, 1], 1>;
+defm : Zn4WriteResYMM<WriteVecMaskedLoadY, [Zn4FPLd01, Zn4Load], !add(Znver4Model.VecLoadLatency, 1), [1, 1], 1>;
+defm : Zn4WriteResXMM<WriteVecStore, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [1, 1], 1>;
+defm : Zn4WriteResXMM<WriteVecStoreX, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [1, 1], 1>;
+
+def Zn4WriteVEXTRACTF128rr_VEXTRACTI128rr : SchedWriteRes<[Zn4FPFMisc0]> {
+  let Latency = 4;
+  let ResourceCycles = [1];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteVEXTRACTF128rr_VEXTRACTI128rr], (instrs VEXTRACTF128rr, VEXTRACTI128rr)>;
+
+def Zn4WriteVEXTRACTI128mr : SchedWriteRes<[Zn4FPFMisc0, Zn4FPSt, Zn4Store]> {
+  let Latency = !add(Znver4Model.LoadLatency, Zn4WriteVEXTRACTF128rr_VEXTRACTI128rr.Latency);
+  let ResourceCycles = [1, 1, 1];
+  let NumMicroOps = !add(Zn4WriteVEXTRACTF128rr_VEXTRACTI128rr.NumMicroOps, 1);
+}
+def : InstRW<[Zn4WriteVEXTRACTI128mr], (instrs VEXTRACTI128mr, VEXTRACTF128mr)>;
+
+def Zn4WriteVINSERTF128rmr : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4FPFMisc0]> {
+  let Latency = !add(Znver4Model.LoadLatency, Zn4WriteVEXTRACTF128rr_VEXTRACTI128rr.Latency);
+  let ResourceCycles = [1, 1, 1];
+  let NumMicroOps = !add(Zn4WriteVEXTRACTF128rr_VEXTRACTI128rr.NumMicroOps, 0);
+}
+def : InstRW<[Zn4WriteVINSERTF128rmr], (instrs VINSERTF128rm)>;
+
+defm : Zn4WriteResYMM<WriteVecStoreY, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [1, 1], 1>;
+defm : Zn4WriteResXMM<WriteVecStoreNT, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [1, 1], 1>;
+defm : Zn4WriteResYMM<WriteVecStoreNTY, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [1, 1], 1>;
+defm : Zn4WriteResXMM<WriteVecMaskedStore32, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [6, 1], 18>;
+defm : Zn4WriteResXMM<WriteVecMaskedStore64, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [4, 1], 10>;
+defm : Zn4WriteResYMM<WriteVecMaskedStore32Y, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [12, 1], 42>;
+defm : Zn4WriteResYMM<WriteVecMaskedStore64Y, [Zn4FPSt, Zn4Store], Znver4Model.StoreLatency, [6, 1], 18>;
+
+defm : Zn4WriteResXMM<WriteVecMoveToGpr, [Zn4FPLd01], 1, [2], 1>;
+defm : Zn4WriteResXMM<WriteVecMoveFromGpr, [Zn4FPLd01], 1, [2], 1>;
+
+def Zn4WriteMOVMMX : SchedWriteRes<[Zn4FPLd01, Zn4FPFMisc0123]> {
+  let Latency = 1;
+  let ResourceCycles = [1, 2];
+  let NumMicroOps = 2;
+}
+def : InstRW<[Zn4WriteMOVMMX], (instrs MMX_MOVQ2FR64rr, MMX_MOVQ2DQrr)>;
+
+def Zn4WriteMOVMMXSlow : SchedWriteRes<[Zn4FPLd01, Zn4FPFMisc0123]> {
+  let Latency = 1;
+  let ResourceCycles = [1, 4];
+  let NumMicroOps = 2;
+}
+def : InstRW<[Zn4WriteMOVMMXSlow], (instrs MMX_MOVD64rr, MMX_MOVD64to64rr)>;
+
+defm : Zn4WriteResXMMPair<WriteVecALU, [Zn4FPVAdd0123], 1, [1], 1>;  // Vector integer ALU op, no logicals.
+
+def Zn4WriteEXTRQ_INSERTQ : SchedWriteRes<[Zn4FPVShuf01, Zn4FPLd01]> {
+  let Latency = 3;
+  let ResourceCycles = [1, 1];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteEXTRQ_INSERTQ], (instrs EXTRQ, INSERTQ)>;
+
+def Zn4WriteEXTRQI_INSERTQI : SchedWriteRes<[Zn4FPVShuf01, Zn4FPLd01]> {
+  let Latency = 3;
+  let ResourceCycles = [1, 1];
+  let NumMicroOps = 2;
+}
+def : InstRW<[Zn4WriteEXTRQI_INSERTQI], (instrs EXTRQI, INSERTQI)>;
+
+defm : Zn4WriteResXMMPair<WriteVecALUX, [Zn4FPVAdd0123], 1, [1], 1>; // Vector integer ALU op, no logicals (XMM).
+
+def Zn4WriteVecALUXSlow : SchedWriteRes<[Zn4FPVAdd01]> {
+  let Latency = 2;
+  let ResourceCycles = [2];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteVecALUXSlow], (instrs PABSBrr, PABSDrr, PABSWrr,
+                                            PADDSBrr, PADDSWrr, PADDUSBrr, PADDUSWrr,
+                                            PAVGBrr, PAVGWrr,
+                                            PSIGNBrr, PSIGNDrr, PSIGNWrr,
+                                            VPABSBrr, VPABSDrr, VPABSWrr,
+                                            VPADDSBrr, VPADDSWrr, VPADDUSBrr, VPADDUSWrr,
+                                            VPAVGBrr, VPAVGWrr,
+                                            VPCMPEQQrr,
+                                            VPSIGNBrr, VPSIGNDrr, VPSIGNWrr,
+                                            PSUBSBrr, PSUBSWrr, PSUBUSBrr, PSUBUSWrr, VPSUBSBrr, VPSUBSWrr, VPSUBUSBrr, VPSUBUSWrr)>;
+
+def Zn4WriteVecOpMask : SchedWriteRes<[Zn4FPOpMask01]> {
+  let Latency = 1;
+  let ResourceCycles = [1];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteVecOpMask], (instrs   KADDBrr, KADDDrr, KADDQrr, KADDWrr,
+                                            KANDBrr, KANDDrr, KANDQrr, KANDWrr,
+                                            KANDNBrr, KANDNDrr, KANDNQrr, KANDNWrr,
+                                            KMOVBkk, KMOVDkk, KMOVQkk, KMOVWkk,
+                                            KMOVBrk, KMOVDrk, KMOVQrk, KMOVWrk,
+                                            KNOTBrr, KNOTDrr, KNOTQrr, KNOTWrr,
+                                            KORBrr, KORDrr, KORQrr, KORWrr,
+                                            KORTESTBrr, KORTESTDrr, KORTESTQrr, KORTESTWrr,
+                                            KTESTBrr, KTESTDrr, KTESTQrr, KTESTWrr,
+                                            KUNPCKBWrr, KUNPCKDQrr, KUNPCKWDrr,
+                                            KXNORBrr, KXNORDrr, KXNORQrr, KXNORWrr,
+                                            KXORBrr, KXORDrr, KXORQrr, KXORWrr)>;
+
+def Zn4WriteVecOpMaskMemMov : SchedWriteRes<[Zn4FPOpMask4]> {
+  let Latency = 1;
+  let ResourceCycles = [1];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteVecOpMaskMemMov], (instrs KMOVBmk, KMOVDmk, KMOVQmk, KMOVWmk)>;
+
+def Zn4WriteVecOpMaskKRMov : SchedWriteRes<[Zn4FPOpMask4]> {
+  let Latency = 1;
+  let ResourceCycles = [1];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteVecOpMaskKRMov], (instrs KMOVBkr, KMOVDkr, KMOVQkr, KMOVWkr)>;
+
+def Zn4WriteVecALU2Slow : SchedWriteRes<[Zn4FPVAdd12]> {
+  // TODO: All align instructions are expected to be of 4 cycle latency
+  let Latency = 4;
+  let ResourceCycles = [1];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteVecALU2Slow], (instrs VALIGNDZrri, VALIGNDZ128rri, VALIGNDZ256rri,
+                                            VALIGNQZrri, VALIGNQZ128rri, VALIGNQZ256rri)
+                                            >;
+defm : Zn4WriteResYMMPair<WriteVecALUY, [Zn4FPVAdd0123], 1, [1], 1>; // Vector integer ALU op, no logicals (YMM).
+
+def Zn4WriteVecALUYSlow : SchedWriteRes<[Zn4FPVAdd01]> {
+  let Latency = 1;
+  let ResourceCycles = [1];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteVecALUYSlow], (instrs VPABSBYrr, VPABSDYrr, VPABSWYrr,
+                                            VPADDSBYrr, VPADDSWYrr, VPADDUSBYrr, VPADDUSWYrr,
+                                            VPSUBSBYrr, VPSUBSWYrr, VPSUBUSBYrr, VPSUBUSWYrr,
+                                            VPAVGBYrr, VPAVGWYrr,
+                                            VPCMPEQQYrr,
+                                            VPSIGNBYrr, VPSIGNDYrr, VPSIGNWYrr)>;
+
+defm : Zn4WriteResZMMPair<WriteVecALUZ, [Zn4FPVAdd0123], 1, [2], 1>; // Vector integer ALU op, no logicals (ZMM).
+
+defm : Zn4WriteResXMMPair<WriteVecLogic, [Zn4FPVMisc0123], 1, [1], 1>;  // Vector integer and/or/xor logicals.
+defm : Zn4WriteResXMMPair<WriteVecLogicX, [Zn4FPVMisc0123], 1, [1], 1>; // Vector integer and/or/xor logicals (XMM).
+defm : Zn4WriteResYMMPair<WriteVecLogicY, [Zn4FPVMisc0123], 1, [1], 1>; // Vector integer and/or/xor logicals (YMM).
+defm : Zn4WriteResZMMPair<WriteVecLogicZ, [Zn4FPVMisc0123], 1, [2], 1>; // Vector integer and/or/xor logicals (ZMM).
+defm : Zn4WriteResXMMPair<WriteVecTest, [Zn4FPVAdd12, Zn4FPSt], 1, [1, 1], 2>;  // FIXME: latency not from llvm-exegesis // Vector integer TEST instructions.
+defm : Zn4WriteResYMMPair<WriteVecTestY, [Zn4FPVAdd12, Zn4FPSt], 1, [1, 1], 2>; // FIXME: latency not from llvm-exegesis  // Vector integer TEST instructions (YMM).
+defm : Zn4WriteResZMMPair<WriteVecTestZ, [Zn4FPVAdd12, Zn4FPSt], 1, [2, 2], 2>; // FIXME: latency not from llvm-exegesis  // Vector integer TEST instructions (ZMM).
+defm : Zn4WriteResXMMPair<WriteVecShift, [Zn4FPVShift01], 1, [1], 1>;  // Vector integer shifts (default).
+defm : Zn4WriteResXMMPair<WriteVecShiftX, [Zn4FPVShift01], 2, [2], 1>; // Vector integer shifts (XMM).
+defm : Zn4WriteResYMMPair<WriteVecShiftY, [Zn4FPVShift01], 1, [1], 1>; // Vector integer shifts (YMM).
+defm : Zn4WriteResZMMPair<WriteVecShiftZ, [Zn4FPVShift01], 1, [2], 1>; // Vector integer shifts (ZMM).
+defm : Zn4WriteResXMMPair<WriteVecShiftImm, [Zn4FPVShift01], 1, [1], 1>;  // Vector integer immediate shifts (default).
+defm : Zn4WriteResXMMPair<WriteVecShiftImmX, [Zn4FPVShift01], 1, [1], 1>; // Vector integer immediate shifts (XMM).
+defm : Zn4WriteResYMMPair<WriteVecShiftImmY, [Zn4FPVShift01], 1, [1], 1>; // Vector integer immediate shifts (YMM).
+defm : Zn4WriteResZMMPair<WriteVecShiftImmZ, [Zn4FPVShift01], 1, [2], 1>; // Vector integer immediate shifts (ZMM).
+defm : Zn4WriteResXMMPair<WriteVecIMul, [Zn4FPVMul01], 3, [1], 1>;  // Vector integer multiply (default).
+defm : Zn4WriteResXMMPair<WriteVecIMulX, [Zn4FPVMul01], 3, [1], 1>; // Vector integer multiply (XMM).
+defm : Zn4WriteResYMMPair<WriteVecIMulY, [Zn4FPVMul01], 3, [1], 1>; // Vector integer multiply (YMM).
+defm : Zn4WriteResZMMPair<WriteVecIMulZ, [Zn4FPVMul01], 3, [2], 1>; // Vector integer multiply (ZMM).
+defm : Zn4WriteResXMMPair<WritePMULLD, [Zn4FPVMul01], 3, [1], 1>; // Vector PMULLD.
+defm : Zn4WriteResYMMPair<WritePMULLDY, [Zn4FPVMul01], 3, [1], 1>; // Vector PMULLD (YMM).
+defm : Zn4WriteResZMMPair<WritePMULLDZ, [Zn4FPVMul01], 3, [2], 1>; // Vector PMULLD (ZMM).
+defm : Zn4WriteResXMMPair<WriteShuffle, [Zn4FPVShuf01], 1, [1], 1>;  // Vector shuffles.
+defm : Zn4WriteResXMMPair<WriteShuffleX, [Zn4FPVShuf01], 1, [1], 1>; // Vector shuffles (XMM).
+defm : Zn4WriteResYMMPair<WriteShuffleY, [Zn4FPVShuf01], 1, [1], 1>; // Vector shuffles (YMM).
+defm : Zn4WriteResZMMPair<WriteShuffleZ, [Zn4FPVShuf01], 1, [2], 1>; // Vector shuffles (ZMM).
+defm : Zn4WriteResXMMPair<WriteVarShuffle, [Zn4FPVShuf01], 1, [1], 1>;  // Vector variable shuffles.
+defm : Zn4WriteResXMMPair<WriteVarShuffleX, [Zn4FPVShuf01], 1, [1], 1>; // Vector variable shuffles (XMM).
+defm : Zn4WriteResYMMPair<WriteVarShuffleY, [Zn4FPVShuf01], 1, [1], 1>; // Vector variable shuffles (YMM).
+defm : Zn4WriteResZMMPair<WriteVarShuffleZ, [Zn4FPVShuf01], 1, [2], 1>; // Vector variable shuffles (ZMM).
+defm : Zn4WriteResXMMPair<WriteBlend, [Zn4FPVMisc0123], 1, [1], 1>; // Vector blends.
+defm : Zn4WriteResYMMPair<WriteBlendY, [Zn4FPVMisc0123], 1, [1], 1>; // Vector blends (YMM).
+defm : Zn4WriteResZMMPair<WriteBlendZ, [Zn4FPVMisc0123], 1, [2], 1>; // Vector blends (ZMM).
+defm : Zn4WriteResXMMPair<WriteVarBlend, [Zn4FPVMul01], 1, [1], 1>; // Vector variable blends.
+defm : Zn4WriteResYMMPair<WriteVarBlendY, [Zn4FPVMul01], 1, [1], 1>; // Vector variable blends (YMM).
+defm : Zn4WriteResZMMPair<WriteVarBlendZ, [Zn4FPVMul01], 1, [2], 1>; // Vector variable blends (ZMM).
+defm : Zn4WriteResXMMPair<WritePSADBW, [Zn4FPVAdd0123], 3, [2], 1>;  // Vector PSADBW.
+defm : Zn4WriteResXMMPair<WritePSADBWX, [Zn4FPVAdd0123], 3, [2], 1>; // Vector PSADBW (XMM).
+defm : Zn4WriteResYMMPair<WritePSADBWY, [Zn4FPVAdd0123], 3, [2], 1>; // Vector PSADBW (YMM).
+defm : Zn4WriteResZMMPair<WritePSADBWZ, [Zn4FPVAdd0123], 4, [4], 1>; // Vector PSADBW (ZMM).
+defm : Zn4WriteResXMMPair<WriteMPSAD, [Zn4FPVAdd0123], 4, [8], 4, /*LoadUOps=*/2>; // Vector MPSAD.
+defm : Zn4WriteResYMMPair<WriteMPSADY, [Zn4FPVAdd0123], 4, [8], 3, /*LoadUOps=*/1>; // Vector MPSAD (YMM).
+defm : Zn4WriteResZMMPair<WriteMPSADZ, [Zn4FPVAdd0123], 4, [16], 3, /*LoadUOps=*/1>; // Vector MPSAD (ZMM).
+defm : Zn4WriteResXMMPair<WritePHMINPOS, [Zn4FPVAdd01], 3, [1], 1>;  // Vector PHMINPOS.
+
+// Vector insert/extract operations.
+defm : Zn4WriteResXMMPair<WriteVecInsert, [Zn4FPLd01], 1, [2], 2, /*LoadUOps=*/-1>; // Insert gpr to vector element.
+defm : Zn4WriteResXMM<WriteVecExtract, [Zn4FPLd01], 1, [2], 2>; // Extract vector element to gpr.
+defm : Zn4WriteResXMM<WriteVecExtractSt, [Zn4FPSt, Zn4Store], !add(1, Znver4Model.StoreLatency), [1, 1], 2>; // Extract vector element and store.
+
+// MOVMSK operations.
+defm : Zn4WriteResXMM<WriteFMOVMSK, [Zn4FPVMisc2], 1, [1], 1>;
+defm : Zn4WriteResXMM<WriteVecMOVMSK, [Zn4FPVMisc2], 1, [1], 1>;
+defm : Zn4WriteResYMM<WriteVecMOVMSKY, [Zn4FPVMisc2], 1, [1], 1>;
+defm : Zn4WriteResXMM<WriteMMXMOVMSK, [Zn4FPVMisc2], 1, [1], 1>;
+
+// Conversion between integer and float.
+defm : Zn4WriteResXMMPair<WriteCvtSD2I, [Zn4FPFCvt01], 1, [1], 1>;  // Double -> Integer.
+defm : Zn4WriteResXMMPair<WriteCvtPD2I, [Zn4FPFCvt01], 3, [2], 1>; // Double -> Integer (XMM).
+defm : Zn4WriteResYMMPair<WriteCvtPD2IY, [Zn4FPFCvt01], 3, [2], 2>; // Double -> Integer (YMM).
+defm : Zn4WriteResZMMPair<WriteCvtPD2IZ, [Zn4FPFCvt01], 3, [4], 2>; // Double -> Integer (ZMM).
+
+def Zn4WriteCvtPD2IMMX : SchedWriteRes<[Zn4FPFCvt01]> {
+  let Latency = 1;
+  let ResourceCycles = [2];
+  let NumMicroOps = 2;
+}
+defm : Zn4WriteResXMMPair<WriteCvtSS2I, [Zn4FPFCvt01], 5, [5], 2>;  // Float -> Integer.
+
+defm : Zn4WriteResXMMPair<WriteCvtPS2I, [Zn4FPFCvt01], 3, [1], 1>; // Float -> Integer (XMM).
+defm : Zn4WriteResYMMPair<WriteCvtPS2IY, [Zn4FPFCvt01], 4, [1], 1>; // Float -> Integer (YMM).
+defm : Zn4WriteResZMMPair<WriteCvtPS2IZ, [Zn4FPFCvt01], 4, [2], 2>; // Float -> Integer (ZMM).
+
+defm : Zn4WriteResXMMPair<WriteCvtI2SD, [Zn4FPFCvt01], 4, [2], 2, /*LoadUOps=*/-1>;  // Integer -> Double.
+defm : Zn4WriteResXMMPair<WriteCvtI2PD, [Zn4FPFCvt01], 3, [1], 1>; // Integer -> Double (XMM).
+defm : Zn4WriteResYMMPair<WriteCvtI2PDY, [Zn4FPFCvt01], 3, [2], 2, /*LoadUOps=*/-1>; // Integer -> Double (YMM).
+defm : Zn4WriteResZMMPair<WriteCvtI2PDZ, [Zn4FPFCvt01], 4, [4], 4, /*LoadUOps=*/-1>; // Integer -> Double (ZMM).
+
+def Zn4WriteCvtI2PDMMX : SchedWriteRes<[Zn4FPFCvt01]> {
+  let Latency = 2;
+  let ResourceCycles = [6];
+  let NumMicroOps = 2;
+}
+
+defm : Zn4WriteResXMMPair<WriteCvtI2SS, [Zn4FPFCvt01], 3, [2], 2, /*LoadUOps=*/-1>;  // Integer -> Float.
+defm : Zn4WriteResXMMPair<WriteCvtI2PS, [Zn4FPFCvt01], 3, [1], 1>; // Integer -> Float (XMM).
+defm : Zn4WriteResYMMPair<WriteCvtI2PSY, [Zn4FPFCvt01], 3, [1], 1>; // Integer -> Float (YMM).
+defm : Zn4WriteResZMMPair<WriteCvtI2PSZ, [Zn4FPFCvt01], 3, [2], 2>; // Integer -> Float (ZMM).
+
+def Zn4WriteCvtI2PSMMX : SchedWriteRes<[Zn4FPFCvt01]> {
+  let Latency = 3;
+  let ResourceCycles = [1];
+  let NumMicroOps = 2;
+}
+
+defm : Zn4WriteResXMMPair<WriteCvtSS2SD, [Zn4FPFCvt01], 3, [1], 1>;  // Float -> Double size conversion.
+defm : Zn4WriteResXMMPair<WriteCvtPS2PD, [Zn4FPFCvt01], 3, [1], 1>; // Float -> Double size conversion (XMM).
+defm : Zn4WriteResYMMPair<WriteCvtPS2PDY, [Zn4FPFCvt01], 4, [2], 2, /*LoadUOps=*/-1>; // Float -> Double size conversion (YMM).
+defm : Zn4WriteResZMMPair<WriteCvtPS2PDZ, [Zn4FPFCvt01], 6, [4], 4, /*LoadUOps=*/-1>; // Float -> Double size conversion (ZMM).
+
+defm : Zn4WriteResXMMPair<WriteCvtSD2SS, [Zn4FPFCvt01], 3, [1], 1>;  // Double -> Float size conversion.
+defm : Zn4WriteResXMMPair<WriteCvtPD2PS, [Zn4FPFCvt01], 3, [1], 1>; // Double -> Float size conversion (XMM).
+defm : Zn4WriteResYMMPair<WriteCvtPD2PSY, [Zn4FPFCvt01], 6, [2], 2>; // Double -> Float size conversion (YMM).
+defm : Zn4WriteResZMMPair<WriteCvtPD2PSZ, [Zn4FPFCvt01], 6, [4], 4>; // Double -> Float size conversion (ZMM).
+
+defm : Zn4WriteResXMMPair<WriteCvtPH2PS, [Zn4FPFCvt01], 3, [1], 1>; // Half -> Float size conversion.
+defm : Zn4WriteResYMMPair<WriteCvtPH2PSY, [Zn4FPFCvt01], 4, [2], 2, /*LoadUOps=*/-1>; // Half -> Float size conversion (YMM).
+defm : Zn4WriteResZMMPair<WriteCvtPH2PSZ, [Zn4FPFCvt01], 4, [4], 4, /*LoadUOps=*/-1>; // Half -> Float size conversion (ZMM).
+
+defm : Zn4WriteResXMM<WriteCvtPS2PH, [Zn4FPFCvt01], 3, [2], 1>; // Float -> Half size conversion.
+defm : Zn4WriteResYMM<WriteCvtPS2PHY, [Zn4FPFCvt01], 6, [2], 2>; // Float -> Half size conversion (YMM).
+defm : Zn4WriteResZMM<WriteCvtPS2PHZ, [Zn4FPFCvt01], 6, [2], 2>; // Float -> Half size conversion (ZMM).
+
+defm : Zn4WriteResXMM<WriteCvtPS2PHSt, [Zn4FPFCvt01, Zn4FPSt, Zn4Store], !add(3, Znver4Model.StoreLatency), [1, 1, 1], 2>; // Float -> Half + store size conversion.
+defm : Zn4WriteResYMM<WriteCvtPS2PHYSt, [Zn4FPFCvt01, Zn4FPSt, Zn4Store], !add(6, Znver4Model.StoreLatency), [2, 1, 1], 3>; // Float -> Half + store size conversion (YMM).
+defm : Zn4WriteResYMM<WriteCvtPS2PHZSt, [Zn4FPFCvt01, Zn4FPSt, Zn4Store], !add(6, Znver4Model.StoreLatency), [2, 1, 1], 3>; // Float -> Half + store size conversion (ZMM).
+
+// CRC32 instruction.
+defm : Zn4WriteResIntPair<WriteCRC32, [Zn4ALU1], 3, [1], 1>;
+
+def Zn4WriteSHA1MSG1rr : SchedWriteRes<[Zn4FPU0123]> {
+  let Latency = 2;
+  let ResourceCycles = [2];
+  let NumMicroOps = 2;
+}
+def : InstRW<[Zn4WriteSHA1MSG1rr], (instrs SHA1MSG1rr)>;
+
+def Zn4WriteSHA1MSG1rm : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4FPU0123]> {
+  let Latency = !add(Znver4Model.LoadLatency, Zn4WriteSHA1MSG1rr.Latency);
+  let ResourceCycles = [1, 1, 2];
+  let NumMicroOps = !add(Zn4WriteSHA1MSG1rr.NumMicroOps, 0);
+}
+def : InstRW<[Zn4WriteSHA1MSG1rm], (instrs SHA1MSG1rm)>;
+
+def Zn4WriteSHA1MSG2rr_SHA1NEXTErr : SchedWriteRes<[Zn4FPU0123]> {
+  let Latency = 1;
+  let ResourceCycles = [2];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteSHA1MSG2rr_SHA1NEXTErr], (instrs SHA1MSG2rr, SHA1NEXTErr)>;
+
+def Zn4Writerm_SHA1MSG2rm_SHA1NEXTErm : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4FPU0123]> {
+  let Latency = !add(Znver4Model.LoadLatency, Zn4WriteSHA1MSG2rr_SHA1NEXTErr.Latency);
+  let ResourceCycles = [1, 1, 2];
+  let NumMicroOps = !add(Zn4WriteSHA1MSG2rr_SHA1NEXTErr.NumMicroOps, 0);
+}
+def : InstRW<[Zn4Writerm_SHA1MSG2rm_SHA1NEXTErm], (instrs SHA1MSG2rm, SHA1NEXTErm)>;
+
+def Zn4WriteSHA256MSG1rr : SchedWriteRes<[Zn4FPU0123]> {
+  let Latency = 2;
+  let ResourceCycles = [3];
+  let NumMicroOps = 2;
+}
+def : InstRW<[Zn4WriteSHA256MSG1rr], (instrs SHA256MSG1rr)>;
+
+def Zn4Writerm_SHA256MSG1rm : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4FPU0123]> {
+  let Latency = !add(Znver4Model.LoadLatency, Zn4WriteSHA256MSG1rr.Latency);
+  let ResourceCycles = [1, 1, 3];
+  let NumMicroOps = !add(Zn4WriteSHA256MSG1rr.NumMicroOps, 0);
+}
+def : InstRW<[Zn4Writerm_SHA256MSG1rm], (instrs SHA256MSG1rm)>;
+
+def Zn4WriteSHA256MSG2rr : SchedWriteRes<[Zn4FPU0123]> {
+  let Latency = 3;
+  let ResourceCycles = [8];
+  let NumMicroOps = 4;
+}
+def : InstRW<[Zn4WriteSHA256MSG2rr], (instrs SHA256MSG2rr)>;
+
+def Zn4WriteSHA256MSG2rm : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4FPU0123]> {
+  let Latency = !add(Znver4Model.LoadLatency, Zn4WriteSHA256MSG2rr.Latency);
+  let ResourceCycles = [1, 1, 8];
+  let NumMicroOps = !add(Zn4WriteSHA256MSG2rr.NumMicroOps, 1);
+}
+def : InstRW<[Zn4WriteSHA256MSG2rm], (instrs SHA256MSG2rm)>;
+
+def Zn4WriteSHA1RNDS4rri : SchedWriteRes<[Zn4FPU0123]> {
+  let Latency = 6;
+  let ResourceCycles = [8];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteSHA1RNDS4rri], (instrs SHA1RNDS4rri)>;
+
+def Zn4WriteSHA256RNDS2rr : SchedWriteRes<[Zn4FPU0123]> {
+  let Latency = 4;
+  let ResourceCycles = [8];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteSHA256RNDS2rr], (instrs SHA256RNDS2rr)>;
+
+// Strings instructions.
+// Packed Compare Implicit Length Strings, Return Mask
+defm : Zn4WriteResXMMPair<WritePCmpIStrM, [Zn4FPVAdd0123], 6, [8], 3, /*LoadUOps=*/1>;
+// Packed Compare Explicit Length Strings, Return Mask
+defm : Zn4WriteResXMMPair<WritePCmpEStrM, [Zn4FPVAdd0123], 6, [12], 7, /*LoadUOps=*/5>;
+// Packed Compare Implicit Length Strings, Return Index
+defm : Zn4WriteResXMMPair<WritePCmpIStrI, [Zn4FPVAdd0123], 2, [8], 4>;
+// Packed Compare Explicit Length Strings, Return Index
+defm : Zn4WriteResXMMPair<WritePCmpEStrI, [Zn4FPVAdd0123], 6, [12], 8, /*LoadUOps=*/4>;
+
+// AES instructions.
+defm : Zn4WriteResXMMPair<WriteAESDecEnc, [Zn4FPAES01], 4, [1], 1>; // Decryption, encryption.
+defm : Zn4WriteResXMMPair<WriteAESIMC, [Zn4FPAES01], 4, [1], 1>; // InvMixColumn.
+defm : Zn4WriteResXMMPair<WriteAESKeyGen, [Zn4FPAES01], 4, [1], 1>; // Key Generation.
+
+// Carry-less multiplication instructions.
+defm : Zn4WriteResXMMPair<WriteCLMul, [Zn4FPCLM01], 4, [4], 4>;
+
+// EMMS/FEMMS
+defm : Zn4WriteResInt<WriteEMMS, [Zn4ALU0123], 2, [1], 1>; // FIXME: latency not from llvm-exegesis
+
+// Load/store MXCSR
+defm : Zn4WriteResInt<WriteLDMXCSR, [Zn4AGU012, Zn4Load, Zn4ALU0123], !add(Znver4Model.LoadLatency, 1), [1, 1, 6], 1>; // FIXME: latency not from llvm-exegesis
+defm : Zn4WriteResInt<WriteSTMXCSR, [Zn4ALU0123, Zn4AGU012, Zn4Store], !add(1, Znver4Model.StoreLatency), [60, 1, 1], 2>; // FIXME: latency not from llvm-exegesis
+
+// Catch-all for expensive system instructions.
+defm : Zn4WriteResInt<WriteSystem, [Zn4ALU0123], 100, [100], 100>;
+
+def Zn4WriteVZEROUPPER : SchedWriteRes<[Zn4FPU0123]> {
+  let Latency = 0; // FIXME: not from llvm-exegesis
+  let ResourceCycles = [1];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteVZEROUPPER], (instrs VZEROUPPER)>;
+
+def Zn4WriteVZEROALL : SchedWriteRes<[Zn4FPU0123]> {
+  let Latency = 10; // FIXME: not from llvm-exegesis
+  let ResourceCycles = [24];
+  let NumMicroOps = 18;
+}
+def : InstRW<[Zn4WriteVZEROALL], (instrs VZEROALL)>;
+
+// AVX2.
+defm : Zn4WriteResYMMPair<WriteFShuffle256, [Zn4FPVShuf], 2, [1], 1, /*LoadUOps=*/2>; // Fp 256-bit width vector shuffles.
+defm : Zn4WriteResYMMPair<WriteFVarShuffle256, [Zn4FPVShuf], 7, [1], 2, /*LoadUOps=*/1>; // Fp 256-bit width variable shuffles.
+defm : Zn4WriteResYMMPair<WriteShuffle256, [Zn4FPVShuf], 1, [1], 1>; // 256-bit width vector shuffles.
+
+def Zn4WriteVPERM2I128rr_VPERM2F128rr : SchedWriteRes<[Zn4FPVShuf]> {
+  let Latency = 3;
+  let ResourceCycles = [1];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteVPERM2I128rr_VPERM2F128rr], (instrs VPERM2I128rr, VPERM2F128rr)>;
+
+def Zn4WriteVPERM2F128rm : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4FPVShuf]> {
+  let Latency = !add(Znver4Model.LoadLatency, Zn4WriteVPERM2I128rr_VPERM2F128rr.Latency);
+  let ResourceCycles = [1, 1, 1];
+  let NumMicroOps = !add(Zn4WriteVPERM2I128rr_VPERM2F128rr.NumMicroOps, 0);
+}
+def : InstRW<[Zn4WriteVPERM2F128rm], (instrs VPERM2F128rm)>;
+
+def Zn4WriteVPERMPSYrr : SchedWriteRes<[Zn4FPVShuf]> {
+  let Latency = 7;
+  let ResourceCycles = [1];
+  let NumMicroOps = 2;
+}
+def : InstRW<[Zn4WriteVPERMPSYrr], (instrs VPERMPSYrr)>;
+
+def Zn4WriteVPERMPSYrm : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4FPVShuf]> {
+  let Latency = !add(Znver4Model.LoadLatency, Zn4WriteVPERMPSYrr.Latency);
+  let ResourceCycles = [1, 1, 2];
+  let NumMicroOps = !add(Zn4WriteVPERMPSYrr.NumMicroOps, 1);
+}
+def : InstRW<[Zn4WriteVPERMPSYrm], (instrs VPERMPSYrm)>;
+
+def Zn4WriteVPERMYri : SchedWriteRes<[Zn4FPVShuf]> {
+  let Latency = 6;
+  let ResourceCycles = [1];
+  let NumMicroOps = 2;
+}
+def : InstRW<[Zn4WriteVPERMYri], (instrs VPERMPDYri, VPERMQYri)>;
+
+def Zn4WriteVPERMPDYmi : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4FPVShuf]> {
+  let Latency = !add(Znver4Model.LoadLatency, Zn4WriteVPERMYri.Latency);
+  let ResourceCycles = [1, 1, 2];
+  let NumMicroOps = !add(Zn4WriteVPERMYri.NumMicroOps, 1);
+}
+def : InstRW<[Zn4WriteVPERMPDYmi], (instrs VPERMPDYmi)>;
+
+def Zn4WriteVPERMDYrr : SchedWriteRes<[Zn4FPVShuf]> {
+  let Latency = 5;
+  let ResourceCycles = [1];
+  let NumMicroOps = 2;
+}
+def : InstRW<[Zn4WriteVPERMDYrr], (instrs VPERMDYrr)>;
+
+def Zn4WriteVPERMYm : SchedWriteRes<[Zn4AGU012, Zn4Load, Zn4FPVShuf]> {
+  let Latency = !add(Znver4Model.LoadLatency, Zn4WriteVPERMDYrr.Latency);
+  let ResourceCycles = [1, 1, 2];
+  let NumMicroOps = !add(Zn4WriteVPERMDYrr.NumMicroOps, 0);
+}
+def : InstRW<[Zn4WriteVPERMYm], (instrs VPERMQYmi, VPERMDYrm)>;
+
+defm : Zn4WriteResYMMPair<WriteVPMOV256, [Zn4FPVShuf01], 4, [3], 2, /*LoadUOps=*/-1>; // 256-bit width packed vector width-changing move.
+defm : Zn4WriteResYMMPair<WriteVarShuffle256, [Zn4FPVShuf01], 1, [1], 2>; // 256-bit width vector variable shuffles.
+defm : Zn4WriteResXMMPair<WriteVarVecShift, [Zn4FPVShift01], 1, [1], 1>; // Variable vector shifts.
+defm : Zn4WriteResYMMPair<WriteVarVecShiftY, [Zn4FPVShift01], 1, [1], 1>; // Variable vector shifts (YMM).
+defm : Zn4WriteResZMMPair<WriteVarVecShiftZ, [Zn4FPVShift01], 1, [2], 2>; // Variable vector shifts (ZMM).
+
+// Old microcoded instructions that nobody use.
+defm : Zn4WriteResInt<WriteMicrocoded, [Zn4ALU0123], 100, [100], 100>;
+
+// Fence instructions.
+defm : Zn4WriteResInt<WriteFence, [Zn4ALU0123], 1, [100], 1>;
+
+def Zn4WriteLFENCE : SchedWriteRes<[Zn4LSU]> {
+  let Latency = 1;
+  let ResourceCycles = [30];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteLFENCE], (instrs LFENCE)>;
+
+def Zn4WriteSFENCE : SchedWriteRes<[Zn4LSU]> {
+  let Latency = 1;
+  let ResourceCycles = [1];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteSFENCE], (instrs SFENCE)>;
+
+// Nop, not very useful expect it provides a model for nops!
+defm : Zn4WriteResInt<WriteNop, [Zn4ALU0123], 0, [1], 1>; // FIXME: latency not from llvm-exegesis
+
+
+///////////////////////////////////////////////////////////////////////////////
+// Zero Cycle Move
+///////////////////////////////////////////////////////////////////////////////
+
+def Zn4WriteZeroLatency : SchedWriteRes<[]> {
+  let Latency = 0;
+  let ResourceCycles = [];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteZeroLatency], (instrs MOV32rr, MOV32rr_REV,
+                                               MOV64rr, MOV64rr_REV,
+                                               MOVSX32rr32)>;
+
+def Zn4WriteSwapRenameable : SchedWriteRes<[]> {
+  let Latency = 0;
+  let ResourceCycles = [];
+  let NumMicroOps = 2;
+}
+def : InstRW<[Zn4WriteSwapRenameable], (instrs XCHG32rr, XCHG32ar,
+                                               XCHG64rr, XCHG64ar)>;
+
+defm : Zn4WriteResInt<WriteXCHG, [Zn4ALU0123], 0, [8], 2>;        // Compare+Exchange - TODO RMW support.
+
+defm : Zn4WriteResXMM<WriteFMoveX, [], 0, [], 1>;
+defm : Zn4WriteResYMM<WriteFMoveY, [], 0, [], 1>;
+defm : Zn4WriteResYMM<WriteFMoveZ, [], 0, [], 1>;
+
+defm : Zn4WriteResXMM<WriteVecMove, [Zn4FPFMisc0123], 1, [1], 1>; // MMX
+defm : Zn4WriteResXMM<WriteVecMoveX, [], 0, [], 1>;
+defm : Zn4WriteResYMM<WriteVecMoveY, [], 0, [], 1>;
+defm : Zn4WriteResYMM<WriteVecMoveZ, [], 0, [], 1>;
+
+def : IsOptimizableRegisterMove<[
+  InstructionEquivalenceClass<[
+    // GPR variants.
+    MOV32rr, MOV32rr_REV,
+    MOV64rr, MOV64rr_REV,
+    MOVSX32rr32,
+    XCHG32rr, XCHG32ar,
+    XCHG64rr, XCHG64ar,
+
+    // MMX variants.
+    // MMX moves are *NOT* eliminated.
+
+    // SSE variants.
+    MOVAPSrr, MOVAPSrr_REV,
+    MOVUPSrr, MOVUPSrr_REV,
+    MOVAPDrr, MOVAPDrr_REV,
+    MOVUPDrr, MOVUPDrr_REV,
+    MOVDQArr, MOVDQArr_REV,
+    MOVDQUrr, MOVDQUrr_REV,
+
+    // AVX variants.
+    VMOVAPSrr, VMOVAPSrr_REV,
+    VMOVUPSrr, VMOVUPSrr_REV,
+    VMOVAPDrr, VMOVAPDrr_REV,
+    VMOVUPDrr, VMOVUPDrr_REV,
+    VMOVDQArr, VMOVDQArr_REV,
+    VMOVDQUrr, VMOVDQUrr_REV,
+
+    // AVX YMM variants.
+    VMOVAPSYrr, VMOVAPSYrr_REV,
+    VMOVUPSYrr, VMOVUPSYrr_REV,
+    VMOVAPDYrr, VMOVAPDYrr_REV,
+    VMOVUPDYrr, VMOVUPDYrr_REV,
+    VMOVDQAYrr, VMOVDQAYrr_REV,
+    VMOVDQUYrr, VMOVDQUYrr_REV,
+  ], TruePred >
+]>;
+
+// FIXUP and RANGE Instructions
+def Zn4WriteVFIXUPIMMPDZrr_VRANGESDrr : SchedWriteRes<[Zn4FPFMisc01]> {
+  let Latency = 2;
+  let ResourceCycles = [2];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteVFIXUPIMMPDZrr_VRANGESDrr], (instregex
+	"VFIXUPIMM(S|P)(S|D)(Z|Z128|Z256?)rrik", "VFIXUPIMM(S|P)(S|D)(Z?|Z128?|Z256?)rrikz", 
+        "VFIXUPIMM(S|P)(S|D)(Z128|Z256?)rri",  "VRANGE(S|P)(S|D)(Z?|Z128?|Z256?)rri(b?)",
+	"VRANGE(S|P)(S|D)(Z|Z128|Z256?)rri(b?)k","VRANGE(S|P)(S|D)(Z?|Z128?|Z256?)rri(b?)kz"
+	)>;
+
+// SCALE & REDUCE instructions
+def Zn4WriteSCALErr: SchedWriteRes<[Zn4FPFMisc23]> {
+  let Latency = 6;
+  let ResourceCycles = [6];
+  let NumMicroOps = 2;
+}
+def : InstRW<[Zn4WriteSCALErr], (instregex
+        "V(SCALEF|REDUCE)(S|P)(S|D)(Z?|Z128?|Z256?)(rr|rrb|rrkz|rrik|rrikz|rri)(_Int?|_Intkz?)",
+        "(V?)REDUCE(PD|PS|SD|SS)(Z?|Z128?)(rri|rrikz|rrib)"
+	)>;
+
+//BF16PS Instructions
+def Zn4WriteBF16: SchedWriteRes<[Zn4FPFMisc23]> {
+  let Latency = 6;
+  let ResourceCycles = [6];
+  let NumMicroOps = 2;
+}
+def : InstRW<[Zn4WriteBF16], (instregex
+        "(V?)DPBF16PS(Z?|Z128?|Z256?)(r|rk|rkz)"
+	)>;
+
+// BUSD and VPMADD Instructions
+def Zn4WriteBUSDr_VPMADDr: SchedWriteRes<[Zn4FPFMisc01]> {
+  let Latency = 4;
+  let ResourceCycles = [4];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteBUSDr_VPMADDr], (instregex
+	"VPDP(BU|WS)(S|P)(S|D|DS)(Z|Z128|Z256)(r|rk|rkz)",
+        "VPMADD52(H|L)UQ(Z|Z128|Z256)(r|rk|rkz)"
+	)>;
+
+// SHIFT instructions
+def Zn4WriteSHIFTrr: SchedWriteRes<[Zn4FPFMisc01]> {
+  let Latency = 2;
+  let ResourceCycles = [2];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteSHIFTrr], (instregex
+        "VP(LZCNT|SHLD|SHRD?)(D|Q|W|VD|VQ|VW?)(Z?|Z128?|Z256?)(rr|rk|rrk|rrkz|rri|rrik|rrikz)",
+        "(V?)P(SLL|SRL|SRA)(D|Q|W|DQ)(Y?|Z?|Z128?|Z256?)(rr|rrk|rrkz)",
+        "(V?)P(SLL|SRL|SRA)DQYri",
+        "(V?)P(SLL|SRL)DQ(Z?|Z256?)ri",
+        "(V?)P(SHUFB)(Y|Z|Z128|Z256?)(rr|rrk|rrkz)",
+        "(V?)P(ROL|ROR)(D|Q|VD|VQ)(Z?|Z128?|Z256?)(rr|rrk|rrkz)",
+        "(V?)P(ROL|ROR)(D|Q|VD|VQ)(Z256?)(ri|rik|rikz)",
+        "(V?)P(ROL|ROR)(D|Q)(Z?|Z128?)(ri|rik|rikz)",
+	"VPSHUFBITQMBZ128rr", "VFMSUB231SSZr_Intkz"
+	)>;
+
+def Zn4WriteSHIFTri: SchedWriteRes<[Zn4FPFMisc01]> {
+  let Latency = 1;
+  let ResourceCycles = [1];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteSHIFTri], (instregex
+        "VP(SLL|SRL|SRA)(D|Q|W)(Z|Z128|Z256?)(ri|rik|rikz)"
+	)>;
+
+// ALIGN Instructions
+def Zn4WriteALIGN: SchedWriteRes<[Zn4FPFMisc12]> {
+  let Latency = 2;
+  let ResourceCycles = [2];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteALIGN], (instregex
+        "(V?)PALIGNR(Z?|Z128?|Z256?)(rri|rrik|rrikz)"
+	)>;
+
+//PACK Instructions
+def Zn4WritePACK: SchedWriteRes<[Zn4FPFMisc12]> {
+  let Latency = 2;
+  let ResourceCycles = [2];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WritePACK], (instregex
+        "(V?)PACK(SS|US)(DW|WB)(Z?|Z128?|Z256?)(rr|rrk|rrkz)"
+	)>;
+
+// MAX and MIN Instructions
+def Zn4WriteFCmp64: SchedWriteRes<[Zn4FPFMisc01]> {
+  let Latency = 2;
+  let ResourceCycles = [2];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4WriteFCmp64], (instregex
+        "(V?)CMP(S|P)(S|D)(rr|rri|rr_Int)",
+        "(V?|VP?)(MAX|MIN|MINC|MAXC)(S|P|U)(S|D|Q)(Z?|Z128?|Z256?)(rr|rri|rrk|rrkz)(_Int?)",
+        "VP(MAX|MIN)(SQ|UQ)(Z|Z128|Z256)(rr|rrk|rrkz)",
+        "(V?)(MAX|MAXC|MIN|MINC)PD(Z|Z128|Z256?)(rr|rrk|rrkz)"
+	)>;
+
+// MOV Instructions
+def Zn4MOVS: SchedWriteRes<[Zn4FPFMisc12]> {
+  let Latency = 2;
+  let ResourceCycles = [2];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4MOVS], (instregex
+        "(V?)PMOV(SX|ZX)(BD|BQ|BW|WD|WQ|DQ)(Z128?|Z256?)(rr|rrk|rrkz)",
+        "(V?)PMOV(SX|QD|UZ|ZX)(BD|BQ|BW?)(Y|Z128?)(rr|rrk|rrkz)",
+        "(V?)PMOV(SX|US|ZX)(DQ|WD|QW|WQ?)(Y|Z128?)(rr|rrk|rrkz)",
+        "(V?)VMOVDDUP(Z|Z128|Z256)(rr|rrk|rrkz)",
+        "VPMOV(DB|DW|QB|QD|QW|SDB|SDW|SQB|SQD|SQW|SWB|USDB|USDW|USQB|USQD|USWB|WB)(Z128?)(rr|rrk|rrkz)"
+	)>;
+
+def Zn4MOVSZ: SchedWriteRes<[Zn4FPFMisc12]> {
+  let Latency = 4;
+  let ResourceCycles = [4];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4MOVSZ], (instregex
+        "(V?)PMOV(SX|ZX)(BD|BQ|BW|WD|WQ|DQ)(Z?)(rr|rrk|rrkz)"
+	)>;
+
+def Zn4MOVSrr: SchedWriteRes<[Zn4FPFMisc12]> {
+  let Latency = 5;
+  let ResourceCycles = [5];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4MOVSrr], (instregex
+        "(V?)PMOV(DB|QB|QW|SDB|SQB|SQW|USDB|USQB|USQW)(Z?)(rr|rrk|rrkz)"
+	)>;
+
+
+//VPTEST Instructions
+def Zn4VPTESTZ128: SchedWriteRes<[Zn4FPFMisc01]> {
+  let Latency = 3;
+  let ResourceCycles = [3];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4VPTESTZ128], (instregex
+        "(V?)PTEST(N?)(MB|MD|MQ|MW)(Z128?)(rrk)"
+	)>;
+
+def Zn4VPTESTZ256: SchedWriteRes<[Zn4FPFMisc01]> {
+  let Latency = 4;
+  let ResourceCycles = [4];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4VPTESTZ256], (instregex
+        "(V?)PTEST(N?)(MB|MD|MQ|MW)(Z256?)(rr|rrk)"
+	)>;
+
+def Zn4VPTESTZ: SchedWriteRes<[Zn4FPFMisc01]> {
+  let Latency = 5;
+  let ResourceCycles = [5];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4VPTESTZ], (instregex
+        "(V?)PTEST(N?)(MB|MD|MQ|MW)(Z?)(rrk)"
+	)>;
+
+// CONFLICT Instructions
+def Zn4CONFLICTZ128: SchedWriteRes<[Zn4FPFMisc01]> {
+  let Latency = 2;
+  let ResourceCycles = [2];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4CONFLICTZ128], (instregex
+        "VPCONFLICT(D|Q)(Z128)(rr|rrk|rrkz)"
+	)>;
+
+def Zn4CONFLICTrr: SchedWriteRes<[Zn4FPFMisc01,Zn4FPFMisc12,Zn4FPFMisc23]> {
+  let Latency = 6;
+  let ResourceCycles = [2,2,2];
+  let NumMicroOps = 4;
+}
+def : InstRW<[Zn4CONFLICTrr], (instregex
+        "VPCONFLICT(D|Q)(Z|Z256)(rr|rrkz)"
+	)>;
+
+// RSQRT Instructions
+def Zn4VRSQRT14PDZ256: SchedWriteRes<[Zn4FPFMisc01]> {
+  let Latency = 5;
+  let ResourceCycles = [2];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4VRSQRT14PDZ256], (instregex
+        "VRSQRT14(PD|PS)(Z?|Z128?|Z256?)(r|rr|rk|rrk|rkz|rrkz)"
+	)>;
+
+
+// PERM Instructions
+def Zn4PERMILP: SchedWriteRes<[Zn4FPFMisc123]> {
+  let Latency = 2;
+  let ResourceCycles = [2];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4PERMILP], (instregex
+        "VPERMILP(S|D)(Y|Z|Z128|Z256)(rr|rrk|rrkz)"
+	)>;
+
+def Zn4PERMIT2_128: SchedWriteRes<[Zn4FPFMisc12]> {
+  let Latency = 3;
+  let ResourceCycles = [2];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4PERMIT2_128], (instregex
+        "VPERM(I2|T2)(PS|PD|W)128(rr|rrk|rrkz)",
+        "VPERM(I2|T2)(B|D|Q)128(rr|rrk|rrkz)"
+	)>;
+
+def Zn4PERMIT2_128rr:SchedWriteRes<[Zn4FPFMisc12]> {
+  let Latency = 2;
+  let ResourceCycles = [2];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4PERMIT2_128rr], (instregex
+	"V(P?)COMPRESS(B|W|D|Q|PD|PS|SD|SQ)Z128(rr|rrk|rrkz)",
+	"VPERM(B|D|Q|W)(Z128?)(rr|rrk|rrkz)"
+	)>;
+
+def Zn4PERMIT2_256: SchedWriteRes<[Zn4FPFMisc12]> {
+  let Latency = 4;
+  let ResourceCycles = [2];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4PERMIT2_256], (instregex
+        "VPERM(I2|T2)(PS|PD|W)256(rr|rrk|rrkz)",
+	"VPERMP(S|D)Z256(rr|rrk|rrkz)",
+	"V(P?)COMPRESS(B|W|D|Q|PD|PS|SD|SQ)Z256(rr|rrk|rrkz)",
+	"VPERM(B|D|Q|W)Z256(rr|rrk|rrkz)",
+	"VPERM(I2|Q|T2)(B|D|Q)(Z?)256(rr|rrk|rrkz)",
+	"VPEXPAND(B|W)Z256(rr|rrk|rrkz)"
+	)>;
+
+def Zn4PERMIT2Z: SchedWriteRes<[Zn4FPFMisc12]> {
+  let Latency = 5;
+  let ResourceCycles = [2];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4PERMIT2Z], (instregex
+        "VPERM(I2|T2)(PS|PD|W)(rr|rrk|rrkz)",
+	"VPERM(B|D|W)Z(rr|rrk|rrkz)",
+	"VPERM(I2|Q|T2)(B|D|Q)(Z?)(rr|rrk|rrkz)",
+	"V(P?)COMPRESS(B|W|D|Q|PD|PS|SD|SQ)Z(rr|rrk|rrkz)",
+	"VPEXPAND(B|W)Z(rr|rrk|rrkz)",
+        "VPERMP(S|D)Z(rr|rrk|rrkz)"
+	)>;
+
+// ALU SLOW Misc Instructions
+def Zn4VecALUZSlow: SchedWriteRes<[Zn4FPFMisc01]> {
+  let Latency = 2;
+  let ResourceCycles = [2];
+  let NumMicroOps = 1;
+}
+def : InstRW<[Zn4VecALUZSlow], (instrs 
+	VPABSBZ128rr,      VPABSBZ128rrk,  VPABSBZ128rrkz,   VPABSDZ128rr, 
+	VPABSDZ128rrk,     VPABSDZ128rrkz, VPABSQZ128rr,     VPABSQZ128rrk, 
+	VPABSQZ128rrkz,    VPABSWZ128rr,   VPABSWZ128rrk,    VPABSWZ128rrkz, 
+	VPADDSBZ128rr,     VPADDSBZ128rrk, VPADDSBZ128rrkz,  VPADDSWZ128rr, 
+	VPADDSWZ128rrk,    VPADDSWZ128rrkz,VPADDUSBZ128rr,   VPADDUSBZ128rrk, 
+	VPADDUSBZ128rrkz,  VPADDUSWZ128rr, VPADDUSWZ128rrk,  VPADDUSWZ128rrkz, 
+	VPAVGBZ128rr,      VPAVGBZ128rrk,  VPAVGBZ128rrkz,   VPAVGWZ128rr, 
+	VPAVGWZ128rrk,     VPAVGWZ128rrkz, VPOPCNTBZ128rr,   VPOPCNTBZ128rrk, 
+	VPOPCNTBZ128rrkz,  VPOPCNTDZ128rr, VPOPCNTDZ128rrk,  VPOPCNTDZ128rrkz, 
+	VPOPCNTQZ128rr,    VPOPCNTQZ128rrk,VPOPCNTQZ128rrkz, VPOPCNTWZ128rr, 
+	VPOPCNTWZ128rrk,   VPOPCNTWZ128rrkz,VPSUBSBZ128rr,   VPSUBSBZ128rrk, 
+	VPSUBSBZ128rrkz,   VPSUBSWZ128rr,   VPSUBSWZ128rrk,  VPSUBSWZ128rrkz, 
+	VPSUBUSBZ128rr,    VPSUBUSBZ128rrk, VPSUBUSBZ128rrkz,VPSUBUSWZ128rr, 
+	VPSUBUSWZ128rrk,   VPSUBUSWZ128rrkz
+	)>;
+
+
+///////////////////////////////////////////////////////////////////////////////
+// Dependency breaking instructions.
+///////////////////////////////////////////////////////////////////////////////
+
+def Zn4WriteZeroIdiom : SchedWriteVariant<[
+    SchedVar<MCSchedPredicate<ZeroIdiomPredicate>, [Zn4WriteZeroLatency]>,
+    SchedVar<NoSchedPred,                          [WriteALU]>
+]>;
+def : InstRW<[Zn4WriteZeroIdiom], (instrs XOR32rr, XOR32rr_REV,
+                                          XOR64rr, XOR64rr_REV,
+                                          SUB32rr, SUB32rr_REV,
+                                          SUB64rr, SUB64rr_REV)>;
+
+def Zn4WriteZeroIdiomEFLAGS : SchedWriteVariant<[
+    SchedVar<MCSchedPredicate<CheckSameRegOperand<0, 1>>, [Zn4WriteZeroLatency]>,
+    SchedVar<NoSchedPred,                                 [WriteALU]>
+]>;
+def : InstRW<[Zn4WriteZeroIdiomEFLAGS], (instrs CMP8rr,  CMP8rr_REV,
+                                                CMP16rr, CMP16rr_REV,
+                                                CMP32rr, CMP32rr_REV,
+                                                CMP64rr, CMP64rr_REV)>;
+
+def Zn4WriteFZeroIdiom : SchedWriteVariant<[
+    SchedVar<MCSchedPredicate<ZeroIdiomPredicate>, [Zn4WriteZeroLatency]>,
+    SchedVar<NoSchedPred,                          [WriteFLogic]>
+]>;
+// NOTE: XORPSrr, XORPDrr are not zero-cycle!
+def : InstRW<[Zn4WriteFZeroIdiom], (instrs VXORPSrr, VXORPDrr,
+                                           VANDNPSrr, VANDNPDrr)>;
+
+def Zn4WriteFZeroIdiomY : SchedWriteVariant<[
+    SchedVar<MCSchedPredicate<ZeroIdiomPredicate>, [Zn4WriteZeroLatency]>,
+    SchedVar<NoSchedPred,                          [WriteFLogicY]>
+]>;
+def : InstRW<[Zn4WriteFZeroIdiomY], (instrs VXORPSYrr, VXORPDYrr,
+                                            VANDNPSYrr, VANDNPDYrr)>;
+
+def Zn4WriteVZeroIdiomLogicX : SchedWriteVariant<[
+    SchedVar<MCSchedPredicate<ZeroIdiomPredicate>, [Zn4WriteZeroLatency]>,
+    SchedVar<NoSchedPred,                          [WriteVecLogicX]>
+]>;
+// NOTE: PXORrr,PANDNrr are not zero-cycle!
+def : InstRW<[Zn4WriteVZeroIdiomLogicX], (instrs VPXORrr, VPANDNrr)>;
+
+def Zn4WriteVZeroIdiomLogicY : SchedWriteVariant<[
+    SchedVar<MCSchedPredicate<ZeroIdiomPredicate>, [Zn4WriteZeroLatency]>,
+    SchedVar<NoSchedPred,                          [WriteVecLogicY]>
+]>;
+def : InstRW<[Zn4WriteVZeroIdiomLogicY], (instrs VPXORYrr, VPANDNYrr)>;
+
+def Zn4WriteVZeroIdiomALUX : SchedWriteVariant<[
+    SchedVar<MCSchedPredicate<ZeroIdiomPredicate>, [Zn4WriteZeroLatency]>,
+    SchedVar<NoSchedPred,                          [WriteVecALUX]>
+]>;
+// NOTE: PSUBBrr, PSUBWrr, PSUBDrr, PSUBQrr,
+//       PCMPGTBrr, PCMPGTWrr, PCMPGTDrr, PCMPGTQrr are not zero-cycle!
+def : InstRW<[Zn4WriteVZeroIdiomALUX],
+             (instrs VPSUBBrr, VPSUBWrr, VPSUBDrr, VPSUBQrr,
+                     VPCMPGTBrr, VPCMPGTWrr, VPCMPGTDrr, VPCMPGTQrr)>;
+
+def Zn4WriteVZeroIdiomALUY : SchedWriteVariant<[
+    SchedVar<MCSchedPredicate<ZeroIdiomPredicate>, [Zn4WriteZeroLatency]>,
+    SchedVar<NoSchedPred,                          [WriteVecALUY]>
+]>;
+def : InstRW<[Zn4WriteVZeroIdiomALUY],
+             (instrs VPSUBBYrr, VPSUBWYrr, VPSUBDYrr, VPSUBQYrr,
+                     VPCMPGTBYrr, VPCMPGTWYrr, VPCMPGTDYrr, VPCMPGTQYrr)>;
+
+def : IsZeroIdiomFunction<[
+  // GPR Zero-idioms.
+  DepBreakingClass<[ XOR32rr, XOR32rr_REV,
+                     XOR64rr, XOR64rr_REV,
+                     SUB32rr, SUB32rr_REV,
+                     SUB64rr, SUB64rr_REV ], ZeroIdiomPredicate>,
+
+  // SSE XMM Zero-idioms.
+  DepBreakingClass<[
+    // fp variants.
+    XORPSrr, XORPDrr,
+    ANDNPSrr, ANDNPDrr,
+
+    // int variants.
+    PXORrr,
+    PANDNrr,
+    PSUBBrr, PSUBWrr, PSUBDrr, PSUBQrr,
+    PSUBSBrr, PSUBSWrr,
+    PSUBUSBrr, PSUBUSWrr,
+    PCMPGTBrr, PCMPGTWrr, PCMPGTDrr, PCMPGTQrr
+  ], ZeroIdiomPredicate>,
+
+  // AVX XMM Zero-idioms.
+  DepBreakingClass<[
+    // fp variants.
+    VXORPSrr, VXORPDrr,
+    VANDNPSrr, VANDNPDrr,
+
+    // int variants.
+    VPXORrr,
+    VPANDNrr,
+    VPSUBBrr, VPSUBWrr, VPSUBDrr, VPSUBQrr,
+    VPSUBSBrr, VPSUBSWrr,
+    VPSUBUSBrr, VPSUBUSWrr,
+    VPCMPGTBrr, VPCMPGTWrr, VPCMPGTDrr, VPCMPGTQrr,
+  ], ZeroIdiomPredicate>,
+
+  // AVX YMM Zero-idioms.
+  DepBreakingClass<[
+    // fp variants.
+    VXORPSYrr, VXORPDYrr,
+    VANDNPSYrr, VANDNPDYrr,
+
+    // int variants.
+    VPXORYrr,
+    VPANDNYrr,
+    VPSUBBYrr, VPSUBWYrr, VPSUBDYrr, VPSUBQYrr,
+    VPSUBSBYrr, VPSUBSWYrr,
+    VPSUBUSBYrr, VPSUBUSWYrr,
+    VPCMPGTBYrr, VPCMPGTWYrr, VPCMPGTDYrr, VPCMPGTQYrr
+  ], ZeroIdiomPredicate>,
+]>;
+
+def : IsDepBreakingFunction<[
+  // GPR
+  DepBreakingClass<[ SBB32rr, SBB32rr_REV,
+                     SBB64rr, SBB64rr_REV ], ZeroIdiomPredicate>,
+  DepBreakingClass<[ CMP8rr,  CMP8rr_REV,
+                     CMP16rr, CMP16rr_REV,
+                     CMP32rr, CMP32rr_REV,
+                     CMP64rr, CMP64rr_REV ], CheckSameRegOperand<0, 1> >,
+  // SSE
+  DepBreakingClass<[
+    PCMPEQBrr, PCMPEQWrr, PCMPEQDrr, PCMPEQQrr
+  ], ZeroIdiomPredicate>,
+
+  // AVX XMM
+  DepBreakingClass<[
+    VPCMPEQBrr, VPCMPEQWrr, VPCMPEQDrr, VPCMPEQQrr
+  ], ZeroIdiomPredicate>,
+
+  // AVX YMM
+  DepBreakingClass<[
+    VPCMPEQBYrr, VPCMPEQWYrr, VPCMPEQDYrr, VPCMPEQQYrr
+  ], ZeroIdiomPredicate>,
+]>;
+
+} // SchedModel
+
diff --git a/llvm/lib/Target/X86/X86SelectionDAGInfo.cpp b/llvm/lib/Target/X86/X86SelectionDAGInfo.cpp
index f47d8a6e83487..43e9f3d1dfbc1 100644
--- a/llvm/lib/Target/X86/X86SelectionDAGInfo.cpp
+++ b/llvm/lib/Target/X86/X86SelectionDAGInfo.cpp
@@ -182,7 +182,8 @@ static SDValue emitRepmovsB(const X86Subtarget &Subtarget, SelectionDAG &DAG,
 
 /// Returns the best type to use with repmovs depending on alignment.
 static MVT getOptimalRepmovsType(const X86Subtarget &Subtarget,
-                                 uint64_t Align) {
+                                 Align Alignment) {
+  uint64_t Align = Alignment.value();
   assert((Align != 0) && "Align is normalized");
   assert(isPowerOf2_64(Align) && "Align is a power of 2");
   switch (Align) {
@@ -204,7 +205,7 @@ static MVT getOptimalRepmovsType(const X86Subtarget &Subtarget,
 static SDValue emitConstantSizeRepmov(
     SelectionDAG &DAG, const X86Subtarget &Subtarget, const SDLoc &dl,
     SDValue Chain, SDValue Dst, SDValue Src, uint64_t Size, EVT SizeVT,
-    unsigned Align, bool isVolatile, bool AlwaysInline,
+    Align Alignment, bool isVolatile, bool AlwaysInline,
     MachinePointerInfo DstPtrInfo, MachinePointerInfo SrcPtrInfo) {
 
   /// TODO: Revisit next line: big copy with ERMSB on march >= haswell are very
@@ -219,10 +220,10 @@ static SDValue emitConstantSizeRepmov(
   assert(!Subtarget.hasERMSB() && "No efficient RepMovs");
   /// We assume runtime memcpy will do a better job for unaligned copies when
   /// ERMS is not present.
-  if (!AlwaysInline && (Align & 3) != 0)
+  if (!AlwaysInline && (Alignment.value() & 3) != 0)
     return SDValue();
 
-  const MVT BlockType = getOptimalRepmovsType(Subtarget, Align);
+  const MVT BlockType = getOptimalRepmovsType(Subtarget, Alignment);
   const uint64_t BlockBytes = BlockType.getSizeInBits() / 8;
   const uint64_t BlockCount = Size / BlockBytes;
   const uint64_t BytesLeft = Size % BlockBytes;
@@ -251,7 +252,7 @@ static SDValue emitConstantSizeRepmov(
       Chain, dl,
       DAG.getNode(ISD::ADD, dl, DstVT, Dst, DAG.getConstant(Offset, dl, DstVT)),
       DAG.getNode(ISD::ADD, dl, SrcVT, Src, DAG.getConstant(Offset, dl, SrcVT)),
-      DAG.getConstant(BytesLeft, dl, SizeVT), llvm::Align(Align), isVolatile,
+      DAG.getConstant(BytesLeft, dl, SizeVT), Alignment, isVolatile,
       /*AlwaysInline*/ true, /*isTailCall*/ false,
       DstPtrInfo.getWithOffset(Offset), SrcPtrInfo.getWithOffset(Offset)));
   return DAG.getNode(ISD::TokenFactor, dl, MVT::Other, Results);
@@ -281,10 +282,10 @@ SDValue X86SelectionDAGInfo::EmitTargetCodeForMemcpy(
 
   /// Handle constant sizes,
   if (ConstantSDNode *ConstantSize = dyn_cast<ConstantSDNode>(Size))
-    return emitConstantSizeRepmov(
-        DAG, Subtarget, dl, Chain, Dst, Src, ConstantSize->getZExtValue(),
-        Size.getValueType(), Alignment.value(), isVolatile, AlwaysInline,
-        DstPtrInfo, SrcPtrInfo);
+    return emitConstantSizeRepmov(DAG, Subtarget, dl, Chain, Dst, Src,
+                                  ConstantSize->getZExtValue(),
+                                  Size.getValueType(), Alignment, isVolatile,
+                                  AlwaysInline, DstPtrInfo, SrcPtrInfo);
 
   return SDValue();
 }
diff --git a/llvm/lib/Target/X86/X86TargetTransformInfo.cpp b/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
index 4e6acf4a1020a..e2678f337384d 100644
--- a/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
+++ b/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
@@ -4396,11 +4396,6 @@ InstructionCost X86TTIImpl::getVectorInstrCost(unsigned Opcode, Type *Val,
     return ShuffleCost + IntOrFpCost + RegisterFileMoveCost;
   }
 
-  // Add to the base cost if we know that the extracted element of a vector is
-  // destined to be moved to and used in the integer register file.
-  if (Opcode == Instruction::ExtractElement && ScalarType->isPointerTy())
-    RegisterFileMoveCost += 1;
-
   return BaseT::getVectorInstrCost(Opcode, Val, CostKind, Index, Op0, Op1) +
          RegisterFileMoveCost;
 }
@@ -4899,6 +4894,23 @@ X86TTIImpl::getMaskedMemoryOpCost(unsigned Opcode, Type *SrcTy, Align Alignment,
   return Cost + LT.first;
 }
 
+InstructionCost X86TTIImpl::getPointersChainCost(
+    ArrayRef<const Value *> Ptrs, const Value *Base,
+    const TTI::PointersChainInfo &Info, TTI::TargetCostKind CostKind) {
+  if (Info.isSameBase() && Info.isKnownStride()) {
+    // If all the pointers have known stride all the differences are translated
+    // into constants. X86 memory addressing allows encoding it into
+    // displacement. So we just need to take the base GEP cost.
+    if (const auto *BaseGEP = dyn_cast<GetElementPtrInst>(Base)) {
+      SmallVector<const Value *> Indices(BaseGEP->indices());
+      return getGEPCost(BaseGEP->getSourceElementType(),
+                        BaseGEP->getPointerOperand(), Indices, CostKind);
+    }
+    return TTI::TCC_Free;
+  }
+  return BaseT::getPointersChainCost(Ptrs, Base, Info, CostKind);
+}
+
 InstructionCost X86TTIImpl::getAddressComputationCost(Type *Ty,
                                                       ScalarEvolution *SE,
                                                       const SCEV *Ptr) {
diff --git a/llvm/lib/Target/X86/X86TargetTransformInfo.h b/llvm/lib/Target/X86/X86TargetTransformInfo.h
index 41f7095c8787d..6893b86056d23 100644
--- a/llvm/lib/Target/X86/X86TargetTransformInfo.h
+++ b/llvm/lib/Target/X86/X86TargetTransformInfo.h
@@ -177,6 +177,10 @@ class X86TTIImpl : public BasicTTIImplBase<X86TTIImpl> {
                                          Align Alignment,
                                          TTI::TargetCostKind CostKind,
                                          const Instruction *I);
+  InstructionCost getPointersChainCost(ArrayRef<const Value *> Ptrs,
+                                       const Value *Base,
+                                       const TTI::PointersChainInfo &Info,
+                                       TTI::TargetCostKind CostKind);
   InstructionCost getAddressComputationCost(Type *PtrTy, ScalarEvolution *SE,
                                             const SCEV *Ptr);
 
diff --git a/llvm/lib/TargetParser/ARMTargetParser.cpp b/llvm/lib/TargetParser/ARMTargetParser.cpp
index 740219d852684..785e9a4fe3fb9 100644
--- a/llvm/lib/TargetParser/ARMTargetParser.cpp
+++ b/llvm/lib/TargetParser/ARMTargetParser.cpp
@@ -147,7 +147,8 @@ ARM::ProfileKind ARM::parseArchProfile(StringRef Arch) {
   return getProfileKind(parseArch(Arch));
 }
 
-bool ARM::getFPUFeatures(unsigned FPUKind, std::vector<StringRef> &Features) {
+bool ARM::getFPUFeatures(ARM::FPUKind FPUKind,
+                         std::vector<StringRef> &Features) {
 
   if (FPUKind >= FK_LAST || FPUKind == FK_INVALID)
     return false;
@@ -211,7 +212,7 @@ bool ARM::getFPUFeatures(unsigned FPUKind, std::vector<StringRef> &Features) {
   return true;
 }
 
-unsigned ARM::parseFPU(StringRef FPU) {
+ARM::FPUKind ARM::parseFPU(StringRef FPU) {
   StringRef Syn = getFPUSynonym(FPU);
   for (const auto &F : FPUNames) {
     if (Syn == F.Name)
@@ -220,7 +221,7 @@ unsigned ARM::parseFPU(StringRef FPU) {
   return FK_INVALID;
 }
 
-ARM::NeonSupportLevel ARM::getFPUNeonSupportLevel(unsigned FPUKind) {
+ARM::NeonSupportLevel ARM::getFPUNeonSupportLevel(ARM::FPUKind FPUKind) {
   if (FPUKind >= FK_LAST)
     return NeonSupportLevel::None;
   return FPUNames[FPUKind].NeonSupport;
@@ -243,33 +244,33 @@ StringRef ARM::getFPUSynonym(StringRef FPU) {
       .Default(FPU);
 }
 
-StringRef ARM::getFPUName(unsigned FPUKind) {
+StringRef ARM::getFPUName(ARM::FPUKind FPUKind) {
   if (FPUKind >= FK_LAST)
     return StringRef();
   return FPUNames[FPUKind].Name;
 }
 
-ARM::FPUVersion ARM::getFPUVersion(unsigned FPUKind) {
+ARM::FPUVersion ARM::getFPUVersion(ARM::FPUKind FPUKind) {
   if (FPUKind >= FK_LAST)
     return FPUVersion::NONE;
   return FPUNames[FPUKind].FPUVer;
 }
 
-ARM::FPURestriction ARM::getFPURestriction(unsigned FPUKind) {
+ARM::FPURestriction ARM::getFPURestriction(ARM::FPUKind FPUKind) {
   if (FPUKind >= FK_LAST)
     return FPURestriction::None;
   return FPUNames[FPUKind].Restriction;
 }
 
-unsigned ARM::getDefaultFPU(StringRef CPU, ARM::ArchKind AK) {
+ARM::FPUKind ARM::getDefaultFPU(StringRef CPU, ARM::ArchKind AK) {
   if (CPU == "generic")
     return ARM::ARMArchNames[static_cast<unsigned>(AK)].DefaultFPU;
 
-  return StringSwitch<unsigned>(CPU)
+  return StringSwitch<ARM::FPUKind>(CPU)
 #define ARM_CPU_NAME(NAME, ID, DEFAULT_FPU, IS_DEFAULT, DEFAULT_EXT)           \
   .Case(NAME, DEFAULT_FPU)
 #include "llvm/TargetParser/ARMTargetParser.def"
-   .Default(ARM::FK_INVALID);
+      .Default(ARM::FK_INVALID);
 }
 
 uint64_t ARM::getDefaultExtensions(StringRef CPU, ARM::ArchKind AK) {
@@ -362,7 +363,7 @@ StringRef ARM::getArchExtFeature(StringRef ArchExt) {
   return StringRef();
 }
 
-static unsigned findDoublePrecisionFPU(unsigned InputFPUKind) {
+static ARM::FPUKind findDoublePrecisionFPU(ARM::FPUKind InputFPUKind) {
   const ARM::FPUName &InputFPU = ARM::FPUNames[InputFPUKind];
 
   // If the input FPU already supports double-precision, then there
@@ -394,7 +395,7 @@ static unsigned findDoublePrecisionFPU(unsigned InputFPUKind) {
 bool ARM::appendArchExtFeatures(StringRef CPU, ARM::ArchKind AK,
                                 StringRef ArchExt,
                                 std::vector<StringRef> &Features,
-                                unsigned &ArgFPUID) {
+                                ARM::FPUKind &ArgFPUKind) {
 
   size_t StartingNumFeatures = Features.size();
   const bool Negated = stripNegationPrefix(ArchExt);
@@ -417,7 +418,7 @@ bool ARM::appendArchExtFeatures(StringRef CPU, ARM::ArchKind AK,
     CPU = "generic";
 
   if (ArchExt == "fp" || ArchExt == "fp.dp") {
-    unsigned FPUKind;
+    ARM::FPUKind FPUKind;
     if (ArchExt == "fp.dp") {
       if (Negated) {
         Features.push_back("-fp64");
@@ -429,7 +430,7 @@ bool ARM::appendArchExtFeatures(StringRef CPU, ARM::ArchKind AK,
     } else {
       FPUKind = getDefaultFPU(CPU, AK);
     }
-    ArgFPUID = FPUKind;
+    ArgFPUKind = FPUKind;
     return ARM::getFPUFeatures(FPUKind, Features);
   }
   return StartingNumFeatures != Features.size();
@@ -523,7 +524,7 @@ StringRef ARM::computeDefaultTargetABI(const Triple &TT, StringRef CPU) {
   default:
     if (TT.isOSNetBSD())
       return "apcs-gnu";
-    if (TT.isOSOpenBSD() || TT.isOHOSFamily())
+    if (TT.isOSFreeBSD() || TT.isOSOpenBSD() || TT.isOHOSFamily())
       return "aapcs-linux";
     return "aapcs";
   }
diff --git a/llvm/lib/TargetParser/CMakeLists.txt b/llvm/lib/TargetParser/CMakeLists.txt
index 392c67478007f..0bfea7b14e690 100644
--- a/llvm/lib/TargetParser/CMakeLists.txt
+++ b/llvm/lib/TargetParser/CMakeLists.txt
@@ -1,3 +1,13 @@
+# Ensure that libLLVMTargetParser does not carry any static global initializer.
+# ManagedStatic can be used to enable lazy-initialization of globals.
+#
+# HAS_WERROR_GLOBAL_CTORS and LLVM_HAS_NOGLOBAL_CTOR_MUTEX should have been set
+# by llvm/lib/Support/CMakeLists.txt (which provides the required Support
+# dependency).
+if (HAS_WERROR_GLOBAL_CTORS AND NOT LLVM_HAS_NOGLOBAL_CTOR_MUTEX)
+  SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Werror=global-constructors")
+endif()
+
 add_llvm_component_library(LLVMTargetParser
   AArch64TargetParser.cpp
   ARMTargetParserCommon.cpp
diff --git a/llvm/lib/TargetParser/RISCVTargetParser.cpp b/llvm/lib/TargetParser/RISCVTargetParser.cpp
index 933a82b7c6cb2..3445e339ac606 100644
--- a/llvm/lib/TargetParser/RISCVTargetParser.cpp
+++ b/llvm/lib/TargetParser/RISCVTargetParser.cpp
@@ -103,7 +103,7 @@ bool getCPUFeaturesExceptStdExt(CPUKind Kind,
 
 bool isX18ReservedByDefault(const Triple &TT) {
   // X18 is reserved for the ShadowCallStack ABI (even when not enabled).
-  return TT.isOSFuchsia();
+  return TT.isOSFuchsia() || TT.isAndroid();
 }
 
 } // namespace RISCV
diff --git a/llvm/lib/Testing/Annotations/CMakeLists.txt b/llvm/lib/Testing/Annotations/CMakeLists.txt
index 75e006207edbd..1d8dba8b6a414 100644
--- a/llvm/lib/Testing/Annotations/CMakeLists.txt
+++ b/llvm/lib/Testing/Annotations/CMakeLists.txt
@@ -1,11 +1,14 @@
 # Do not build unittest libraries automatically, they will be pulled in
 # by unittests if these are built.
-set(EXCLUDE_FROM_ALL ON)
+if (NOT ${LLVM_INSTALL_GTEST})
+  set (BUILDTREE_ONLY BUILDTREE_ONLY)
+  set(EXCLUDE_FROM_ALL ON)
+endif()
 
 add_llvm_library(LLVMTestingAnnotations
   Annotations.cpp
 
-  BUILDTREE_ONLY
+  ${BUILDTREE_ONLY}
 
   ADDITIONAL_HEADER_DIRS
   ${LLVM_MAIN_INCLUDE_DIR}/llvm/Testing/Support
diff --git a/llvm/lib/Testing/Support/CMakeLists.txt b/llvm/lib/Testing/Support/CMakeLists.txt
index 766c64b6fdd6e..6955271239ca6 100644
--- a/llvm/lib/Testing/Support/CMakeLists.txt
+++ b/llvm/lib/Testing/Support/CMakeLists.txt
@@ -1,12 +1,15 @@
 # Do not build unittest libraries automatically, they will be pulled in
 # by unittests if these are built.
-set(EXCLUDE_FROM_ALL ON)
+if (NOT ${LLVM_INSTALL_GTEST})
+  set (BUILDTREE_ONLY BUILDTREE_ONLY)
+  set(EXCLUDE_FROM_ALL ON)
+endif()
 
 add_llvm_library(LLVMTestingSupport
   Error.cpp
   SupportHelpers.cpp
 
-  BUILDTREE_ONLY
+  ${BUILDTREE_ONLY}
 
   ADDITIONAL_HEADER_DIRS
   ${LLVM_MAIN_INCLUDE_DIR}/llvm/Testing/Support
diff --git a/llvm/lib/Transforms/AggressiveInstCombine/AggressiveInstCombine.cpp b/llvm/lib/Transforms/AggressiveInstCombine/AggressiveInstCombine.cpp
index 615d201afd2e2..473b41241b8a6 100644
--- a/llvm/lib/Transforms/AggressiveInstCombine/AggressiveInstCombine.cpp
+++ b/llvm/lib/Transforms/AggressiveInstCombine/AggressiveInstCombine.cpp
@@ -19,7 +19,6 @@
 #include "llvm/Analysis/AssumptionCache.h"
 #include "llvm/Analysis/BasicAliasAnalysis.h"
 #include "llvm/Analysis/GlobalsModRef.h"
-#include "llvm/Analysis/LogicCombine.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/Analysis/ValueTracking.h"
@@ -825,40 +824,13 @@ static bool foldConsecutiveLoads(Instruction &I, const DataLayout &DL,
   return true;
 }
 
-/// Reduce bitwise logic sequences.
-static bool foldBitwiseLogic(Function &F, DominatorTree &DT) {
-  bool MadeChange = false;
-  for (BasicBlock &BB : F) {
-    // Ignore unreachable basic blocks.
-    if (!DT.isReachableFromEntry(&BB))
-      continue;
-
-    // TODO: Combining at the function-level would allow more caching of nodes
-    // which saves on compile-time, but it may hit the max value limits before
-    // finding a solution. We could split the combiner based on types to make
-    // the code more efficient, adjust the value of max depth/values, or use
-    // APInt to support tracking more than 63 leaf values.
-    LogicCombiner LC;
-    for (Instruction &I : BB) {
-      if (I.isBitwiseLogicOp()) {
-        Value *NewV = LC.simplify(&I);
-        if (NewV) {
-          MadeChange = true;
-          I.replaceAllUsesWith(NewV);
-        }
-      }
-    }
-  }
-  return MadeChange;
-}
-
 /// This is the entry point for folds that could be implemented in regular
 /// InstCombine, but they are separated because they are not expected to
 /// occur frequently and/or have more than a constant-length pattern match.
 static bool foldUnusualPatterns(Function &F, DominatorTree &DT,
                                 TargetTransformInfo &TTI,
                                 TargetLibraryInfo &TLI, AliasAnalysis &AA) {
-  bool MadeChange = foldBitwiseLogic(F, DT);
+  bool MadeChange = false;
   for (BasicBlock &BB : F) {
     // Ignore unreachable basic blocks.
     if (!DT.isReachableFromEntry(&BB))
diff --git a/llvm/lib/Transforms/Coroutines/CoroCleanup.cpp b/llvm/lib/Transforms/Coroutines/CoroCleanup.cpp
index 81b43a2ab2c22..29978bef661c6 100644
--- a/llvm/lib/Transforms/Coroutines/CoroCleanup.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroCleanup.cpp
@@ -127,10 +127,16 @@ PreservedAnalyses CoroCleanupPass::run(Module &M,
   FunctionPassManager FPM;
   FPM.addPass(SimplifyCFGPass());
 
+  PreservedAnalyses FuncPA;
+  FuncPA.preserveSet<CFGAnalyses>();
+
   Lowerer L(M);
-  for (auto &F : M)
-    if (L.lower(F))
+  for (auto &F : M) {
+    if (L.lower(F)) {
+      FAM.invalidate(F, FuncPA);
       FPM.run(F, FAM);
+    }
+  }
 
   return PreservedAnalyses::none();
 }
diff --git a/llvm/lib/Transforms/Coroutines/CoroElide.cpp b/llvm/lib/Transforms/Coroutines/CoroElide.cpp
index f032c568449b3..d78ab1c1ea284 100644
--- a/llvm/lib/Transforms/Coroutines/CoroElide.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroElide.cpp
@@ -12,6 +12,7 @@
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Analysis/AliasAnalysis.h"
 #include "llvm/Analysis/InstructionSimplify.h"
+#include "llvm/Analysis/OptimizationRemarkEmitter.h"
 #include "llvm/IR/Dominators.h"
 #include "llvm/IR/InstIterator.h"
 #include "llvm/Support/ErrorHandling.h"
@@ -46,7 +47,8 @@ struct Lowerer : coro::LowererBase {
                             AAResults &AA);
   bool shouldElide(Function *F, DominatorTree &DT) const;
   void collectPostSplitCoroIds(Function *F);
-  bool processCoroId(CoroIdInst *, AAResults &AA, DominatorTree &DT);
+  bool processCoroId(CoroIdInst *, AAResults &AA, DominatorTree &DT,
+                     OptimizationRemarkEmitter &ORE);
   bool hasEscapePath(const CoroBeginInst *,
                      const SmallPtrSetImpl<BasicBlock *> &) const;
 };
@@ -299,7 +301,7 @@ void Lowerer::collectPostSplitCoroIds(Function *F) {
 }
 
 bool Lowerer::processCoroId(CoroIdInst *CoroId, AAResults &AA,
-                            DominatorTree &DT) {
+                            DominatorTree &DT, OptimizationRemarkEmitter &ORE) {
   CoroBegins.clear();
   CoroAllocs.clear();
   ResumeAddr.clear();
@@ -343,6 +345,24 @@ bool Lowerer::processCoroId(CoroIdInst *CoroId, AAResults &AA,
   replaceWithConstant(ResumeAddrConstant, ResumeAddr);
 
   bool ShouldElide = shouldElide(CoroId->getFunction(), DT);
+  if (!ShouldElide)
+    ORE.emit([&]() {
+      if (auto FrameSizeAndAlign =
+              getFrameLayout(cast<Function>(ResumeAddrConstant)))
+        return OptimizationRemarkMissed(DEBUG_TYPE, "CoroElide", CoroId)
+               << "'" << ore::NV("callee", CoroId->getCoroutine()->getName())
+               << "' not elided in '"
+               << ore::NV("caller", CoroId->getFunction()->getName())
+               << "' (frame_size="
+               << ore::NV("frame_size", FrameSizeAndAlign->first) << ", align="
+               << ore::NV("align", FrameSizeAndAlign->second.value()) << ")";
+      else
+        return OptimizationRemarkMissed(DEBUG_TYPE, "CoroElide", CoroId)
+               << "'" << ore::NV("callee", CoroId->getCoroutine()->getName())
+               << "' not elided in '"
+               << ore::NV("caller", CoroId->getFunction()->getName())
+               << "' (frame_size=unknown, align=unknown)";
+    });
 
   auto *DestroyAddrConstant = Resumers->getAggregateElement(
       ShouldElide ? CoroSubFnInst::CleanupIndex : CoroSubFnInst::DestroyIndex);
@@ -363,6 +383,23 @@ bool Lowerer::processCoroId(CoroIdInst *CoroId, AAResults &AA,
             << "Elide " << CoroId->getCoroutine()->getName() << " in "
             << CoroId->getFunction()->getName() << "\n";
 #endif
+      ORE.emit([&]() {
+        return OptimizationRemark(DEBUG_TYPE, "CoroElide", CoroId)
+               << "'" << ore::NV("callee", CoroId->getCoroutine()->getName())
+               << "' elided in '"
+               << ore::NV("caller", CoroId->getFunction()->getName())
+               << "' (frame_size="
+               << ore::NV("frame_size", FrameSizeAndAlign->first) << ", align="
+               << ore::NV("align", FrameSizeAndAlign->second.value()) << ")";
+      });
+    } else {
+      ORE.emit([&]() {
+        return OptimizationRemarkMissed(DEBUG_TYPE, "CoroElide", CoroId)
+               << "'" << ore::NV("callee", CoroId->getCoroutine()->getName())
+               << "' not elided in '"
+               << ore::NV("caller", CoroId->getFunction()->getName())
+               << "' (frame_size=unknown, align=unknown)";
+      });
     }
   }
 
@@ -387,10 +424,11 @@ PreservedAnalyses CoroElidePass::run(Function &F, FunctionAnalysisManager &AM) {
 
   AAResults &AA = AM.getResult<AAManager>(F);
   DominatorTree &DT = AM.getResult<DominatorTreeAnalysis>(F);
+  auto &ORE = AM.getResult<OptimizationRemarkEmitterAnalysis>(F);
 
   bool Changed = false;
   for (auto *CII : L.CoroIds)
-    Changed |= L.processCoroId(CII, AA, DT);
+    Changed |= L.processCoroId(CII, AA, DT, ORE);
 
   return Changed ? PreservedAnalyses::none() : PreservedAnalyses::all();
 }
diff --git a/llvm/lib/Transforms/Coroutines/CoroFrame.cpp b/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
index d15f73326631c..542a5a3bd6c95 100644
--- a/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
@@ -943,7 +943,7 @@ void FrameTypeBuilder::finish(StructType *Ty) {
 static void cacheDIVar(FrameDataInfo &FrameData,
                        DenseMap<Value *, DILocalVariable *> &DIVarCache) {
   for (auto *V : FrameData.getAllDefs()) {
-    if (DIVarCache.find(V) != DIVarCache.end())
+    if (DIVarCache.contains(V))
       continue;
 
     auto DDIs = FindDbgDeclareUses(V);
@@ -1166,7 +1166,7 @@ static void buildFrameDebugInfo(Function &F, coro::Shape &Shape,
                                   dwarf::DW_ATE_unsigned_char)});
 
   for (auto *V : FrameData.getAllDefs()) {
-    if (DIVarCache.find(V) == DIVarCache.end())
+    if (!DIVarCache.contains(V))
       continue;
 
     auto Index = FrameData.getFieldIndex(V);
@@ -1198,7 +1198,7 @@ static void buildFrameDebugInfo(Function &F, coro::Shape &Shape,
   // fields confilicts with each other.
   unsigned UnknownTypeNum = 0;
   for (unsigned Index = 0; Index < FrameTy->getNumElements(); Index++) {
-    if (OffsetCache.find(Index) == OffsetCache.end())
+    if (!OffsetCache.contains(Index))
       continue;
 
     std::string Name;
@@ -1213,7 +1213,7 @@ static void buildFrameDebugInfo(Function &F, coro::Shape &Shape,
     AlignInBits = OffsetCache[Index].first * 8;
     OffsetInBits = OffsetCache[Index].second * 8;
 
-    if (NameCache.find(Index) != NameCache.end()) {
+    if (NameCache.contains(Index)) {
       Name = NameCache[Index].str();
       DITy = TyCache[Index];
     } else {
diff --git a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
index 3a8139d256a82..589cf193f35d3 100644
--- a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
@@ -31,6 +31,7 @@
 #include "llvm/Analysis/CallGraph.h"
 #include "llvm/Analysis/ConstantFolding.h"
 #include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/Analysis/OptimizationRemarkEmitter.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/BinaryFormat/Dwarf.h"
 #include "llvm/IR/Argument.h"
@@ -2142,11 +2143,19 @@ PreservedAnalyses CoroSplitPass::run(LazyCallGraph::SCC &C,
     F.setSplittedCoroutine();
 
     SmallVector<Function *, 4> Clones;
+    auto &ORE = FAM.getResult<OptimizationRemarkEmitterAnalysis>(F);
     const coro::Shape Shape =
         splitCoroutine(F, Clones, FAM.getResult<TargetIRAnalysis>(F),
                        OptimizeFrame, MaterializableCallback);
     updateCallGraphAfterCoroutineSplit(*N, Shape, Clones, C, CG, AM, UR, FAM);
 
+    ORE.emit([&]() {
+      return OptimizationRemark(DEBUG_TYPE, "CoroSplit", &F)
+             << "Split '" << ore::NV("function", F.getName())
+             << "' (frame_size=" << ore::NV("frame_size", Shape.FrameSize)
+             << ", align=" << ore::NV("align", Shape.FrameAlign.value()) << ")";
+    });
+
     if (!Shape.CoroSuspends.empty()) {
       // Run the CGSCC pipeline on the original and newly split functions.
       UR.CWorklist.insert(&C);
diff --git a/llvm/lib/Transforms/IPO/Annotation2Metadata.cpp b/llvm/lib/Transforms/IPO/Annotation2Metadata.cpp
index 7156e65f45a5e..a2649be8e4909 100644
--- a/llvm/lib/Transforms/IPO/Annotation2Metadata.cpp
+++ b/llvm/lib/Transforms/IPO/Annotation2Metadata.cpp
@@ -66,6 +66,6 @@ static bool convertAnnotation2Metadata(Module &M) {
 
 PreservedAnalyses Annotation2MetadataPass::run(Module &M,
                                                ModuleAnalysisManager &AM) {
-  convertAnnotation2Metadata(M);
-  return PreservedAnalyses::all();
+  return convertAnnotation2Metadata(M) ? PreservedAnalyses::none()
+                                       : PreservedAnalyses::all();
 }
diff --git a/llvm/lib/Transforms/IPO/Attributor.cpp b/llvm/lib/Transforms/IPO/Attributor.cpp
index 6cb74cd4aacd6..46b29009437c5 100644
--- a/llvm/lib/Transforms/IPO/Attributor.cpp
+++ b/llvm/lib/Transforms/IPO/Attributor.cpp
@@ -3240,6 +3240,8 @@ void Attributor::identifyDefaultAbstractAttributes(Function &F) {
       // Every function with pointer return type might be marked
       // dereferenceable.
       getOrCreateAAFor<AADereferenceable>(RetPos);
+    } else if (AttributeFuncs::isNoFPClassCompatibleType(ReturnType)) {
+      getOrCreateAAFor<AANoFPClass>(RetPos);
     }
   }
 
@@ -3284,6 +3286,8 @@ void Attributor::identifyDefaultAbstractAttributes(Function &F) {
 
       // Every argument with pointer type might be privatizable (or promotable)
       getOrCreateAAFor<AAPrivatizablePtr>(ArgPos);
+    } else if (AttributeFuncs::isNoFPClassCompatibleType(Arg.getType())) {
+      getOrCreateAAFor<AANoFPClass>(ArgPos);
     }
   }
 
@@ -3312,11 +3316,13 @@ void Attributor::identifyDefaultAbstractAttributes(Function &F) {
       return true;
 
     if (!Callee->getReturnType()->isVoidTy() && !CB.use_empty()) {
-
       IRPosition CBRetPos = IRPosition::callsite_returned(CB);
       bool UsedAssumedInformation = false;
       getAssumedSimplified(CBRetPos, nullptr, UsedAssumedInformation,
                            AA::Intraprocedural);
+
+      if (AttributeFuncs::isNoFPClassCompatibleType(Callee->getReturnType()))
+        getOrCreateAAFor<AANoFPClass>(CBInstPos);
     }
 
     for (int I = 0, E = CB.arg_size(); I < E; ++I) {
@@ -3336,8 +3342,14 @@ void Attributor::identifyDefaultAbstractAttributes(Function &F) {
       // Every call site argument might be marked "noundef".
       getOrCreateAAFor<AANoUndef>(CBArgPos);
 
-      if (!CB.getArgOperand(I)->getType()->isPointerTy())
+      Type *ArgTy = CB.getArgOperand(I)->getType();
+
+      if (!ArgTy->isPointerTy()) {
+        if (AttributeFuncs::isNoFPClassCompatibleType(ArgTy))
+          getOrCreateAAFor<AANoFPClass>(CBArgPos);
+
         continue;
+      }
 
       // Call site argument attribute "non-null".
       getOrCreateAAFor<AANonNull>(CBArgPos);
diff --git a/llvm/lib/Transforms/IPO/AttributorAttributes.cpp b/llvm/lib/Transforms/IPO/AttributorAttributes.cpp
index 100f43fbb389d..c4f10f30bd163 100644
--- a/llvm/lib/Transforms/IPO/AttributorAttributes.cpp
+++ b/llvm/lib/Transforms/IPO/AttributorAttributes.cpp
@@ -177,6 +177,7 @@ PIPE_OPERATOR(AAUndefinedBehavior)
 PIPE_OPERATOR(AAPotentialConstantValues)
 PIPE_OPERATOR(AAPotentialValues)
 PIPE_OPERATOR(AANoUndef)
+PIPE_OPERATOR(AANoFPClass)
 PIPE_OPERATOR(AACallEdges)
 PIPE_OPERATOR(AAInterFnReachability)
 PIPE_OPERATOR(AAPointerInfo)
@@ -10205,6 +10206,163 @@ struct AANoUndefCallSiteReturned final
   void trackStatistics() const override { STATS_DECLTRACK_CSRET_ATTR(noundef) }
 };
 
+/// ------------------------ NoFPClass Attribute -------------------------------
+
+struct AANoFPClassImpl : AANoFPClass {
+  AANoFPClassImpl(const IRPosition &IRP, Attributor &A) : AANoFPClass(IRP, A) {}
+
+  void initialize(Attributor &A) override {
+    const IRPosition &IRP = getIRPosition();
+
+    Value &V = IRP.getAssociatedValue();
+    if (isa<UndefValue>(V)) {
+      indicateOptimisticFixpoint();
+      return;
+    }
+
+    SmallVector<Attribute> Attrs;
+    IRP.getAttrs({Attribute::NoFPClass}, Attrs, false, &A);
+    if (!Attrs.empty()) {
+      addKnownBits(Attrs[0].getNoFPClass());
+      return;
+    }
+
+    const DataLayout &DL = A.getDataLayout();
+    if (getPositionKind() != IRPosition::IRP_RETURNED) {
+      KnownFPClass KnownFPClass = computeKnownFPClass(&V, DL);
+      addKnownBits(~KnownFPClass.KnownFPClasses);
+    }
+
+    if (Instruction *CtxI = getCtxI())
+      followUsesInMBEC(*this, A, getState(), *CtxI);
+  }
+
+  /// See followUsesInMBEC
+  bool followUseInMBEC(Attributor &A, const Use *U, const Instruction *I,
+                       AANoFPClass::StateType &State) {
+    const Value *UseV = U->get();
+    const DominatorTree *DT = nullptr;
+    AssumptionCache *AC = nullptr;
+    const TargetLibraryInfo *TLI = nullptr;
+    InformationCache &InfoCache = A.getInfoCache();
+
+    if (Function *F = getAnchorScope()) {
+      DT = InfoCache.getAnalysisResultForFunction<DominatorTreeAnalysis>(*F);
+      AC = InfoCache.getAnalysisResultForFunction<AssumptionAnalysis>(*F);
+      TLI = InfoCache.getTargetLibraryInfoForFunction(*F);
+    }
+
+    const DataLayout &DL = A.getDataLayout();
+
+    KnownFPClass KnownFPClass =
+        computeKnownFPClass(UseV, DL,
+                            /*InterestedClasses=*/fcAllFlags,
+                            /*Depth=*/0, TLI, AC, I, DT);
+    State.addKnownBits(~KnownFPClass.KnownFPClasses);
+
+    bool TrackUse = false;
+    return TrackUse;
+  }
+
+  const std::string getAsStr() const override {
+    std::string Result = "nofpclass";
+    raw_string_ostream OS(Result);
+    OS << getAssumedNoFPClass();
+    return Result;
+  }
+
+  void getDeducedAttributes(LLVMContext &Ctx,
+                            SmallVectorImpl<Attribute> &Attrs) const override {
+    Attrs.emplace_back(Attribute::getWithNoFPClass(Ctx, getAssumedNoFPClass()));
+  }
+};
+
+struct AANoFPClassFloating : public AANoFPClassImpl {
+  AANoFPClassFloating(const IRPosition &IRP, Attributor &A)
+      : AANoFPClassImpl(IRP, A) {}
+
+  /// See AbstractAttribute::initialize(...).
+  void initialize(Attributor &A) override {
+    AANoFPClassImpl::initialize(A);
+  }
+
+  /// See AbstractAttribute::updateImpl(...).
+  ChangeStatus updateImpl(Attributor &A) override {
+    SmallVector<AA::ValueAndContext> Values;
+    bool UsedAssumedInformation = false;
+    if (!A.getAssumedSimplifiedValues(getIRPosition(), *this, Values,
+                                      AA::AnyScope, UsedAssumedInformation)) {
+      Values.push_back({getAssociatedValue(), getCtxI()});
+    }
+
+    StateType T;
+    auto VisitValueCB = [&](Value &V, const Instruction *CtxI) -> bool {
+      const auto &AA = A.getAAFor<AANoFPClass>(*this, IRPosition::value(V),
+                                               DepClassTy::REQUIRED);
+      if (this == &AA) {
+        T.indicatePessimisticFixpoint();
+      } else {
+        const AANoFPClass::StateType &S =
+            static_cast<const AANoFPClass::StateType &>(AA.getState());
+        T ^= S;
+      }
+      return T.isValidState();
+    };
+
+    for (const auto &VAC : Values)
+      if (!VisitValueCB(*VAC.getValue(), VAC.getCtxI()))
+        return indicatePessimisticFixpoint();
+
+    return clampStateAndIndicateChange(getState(), T);
+  }
+
+  /// See AbstractAttribute::trackStatistics()
+  void trackStatistics() const override {
+    STATS_DECLTRACK_FNRET_ATTR(nofpclass)
+  }
+};
+
+struct AANoFPClassReturned final
+    : AAReturnedFromReturnedValues<AANoFPClass, AANoFPClassImpl> {
+  AANoFPClassReturned(const IRPosition &IRP, Attributor &A)
+      : AAReturnedFromReturnedValues<AANoFPClass, AANoFPClassImpl>(IRP, A) {}
+
+  /// See AbstractAttribute::trackStatistics()
+  void trackStatistics() const override {
+    STATS_DECLTRACK_FNRET_ATTR(nofpclass)
+  }
+};
+
+struct AANoFPClassArgument final
+    : AAArgumentFromCallSiteArguments<AANoFPClass, AANoFPClassImpl> {
+  AANoFPClassArgument(const IRPosition &IRP, Attributor &A)
+      : AAArgumentFromCallSiteArguments<AANoFPClass, AANoFPClassImpl>(IRP, A) {}
+
+  /// See AbstractAttribute::trackStatistics()
+  void trackStatistics() const override { STATS_DECLTRACK_ARG_ATTR(nofpclass) }
+};
+
+struct AANoFPClassCallSiteArgument final : AANoFPClassFloating {
+  AANoFPClassCallSiteArgument(const IRPosition &IRP, Attributor &A)
+      : AANoFPClassFloating(IRP, A) {}
+
+  /// See AbstractAttribute::trackStatistics()
+  void trackStatistics() const override {
+    STATS_DECLTRACK_CSARG_ATTR(nofpclass)
+  }
+};
+
+struct AANoFPClassCallSiteReturned final
+    : AACallSiteReturnedFromReturned<AANoFPClass, AANoFPClassImpl> {
+  AANoFPClassCallSiteReturned(const IRPosition &IRP, Attributor &A)
+      : AACallSiteReturnedFromReturned<AANoFPClass, AANoFPClassImpl>(IRP, A) {}
+
+  /// See AbstractAttribute::trackStatistics()
+  void trackStatistics() const override {
+    STATS_DECLTRACK_CSRET_ATTR(nofpclass)
+  }
+};
+
 struct AACallEdgesImpl : public AACallEdges {
   AACallEdgesImpl(const IRPosition &IRP, Attributor &A) : AACallEdges(IRP, A) {}
 
@@ -11629,6 +11787,7 @@ const char AAValueConstantRange::ID = 0;
 const char AAPotentialConstantValues::ID = 0;
 const char AAPotentialValues::ID = 0;
 const char AANoUndef::ID = 0;
+const char AANoFPClass::ID = 0;
 const char AACallEdges::ID = 0;
 const char AAInterFnReachability::ID = 0;
 const char AAPointerInfo::ID = 0;
@@ -11749,6 +11908,7 @@ CREATE_VALUE_ABSTRACT_ATTRIBUTE_FOR_POSITION(AAValueConstantRange)
 CREATE_VALUE_ABSTRACT_ATTRIBUTE_FOR_POSITION(AAPotentialConstantValues)
 CREATE_VALUE_ABSTRACT_ATTRIBUTE_FOR_POSITION(AAPotentialValues)
 CREATE_VALUE_ABSTRACT_ATTRIBUTE_FOR_POSITION(AANoUndef)
+CREATE_VALUE_ABSTRACT_ATTRIBUTE_FOR_POSITION(AANoFPClass)
 CREATE_VALUE_ABSTRACT_ATTRIBUTE_FOR_POSITION(AAPointerInfo)
 
 CREATE_ALL_ABSTRACT_ATTRIBUTE_FOR_POSITION(AAValueSimplify)
diff --git a/llvm/lib/Transforms/IPO/CMakeLists.txt b/llvm/lib/Transforms/IPO/CMakeLists.txt
index 34a1e4c3ebc0b..063a9a60d0cb5 100644
--- a/llvm/lib/Transforms/IPO/CMakeLists.txt
+++ b/llvm/lib/Transforms/IPO/CMakeLists.txt
@@ -31,7 +31,6 @@ add_llvm_component_library(LLVMipo
   ModuleInliner.cpp
   OpenMPOpt.cpp
   PartialInlining.cpp
-  PassManagerBuilder.cpp
   SampleContextTracker.cpp
   SampleProfile.cpp
   SampleProfileProbe.cpp
diff --git a/llvm/lib/Transforms/IPO/DeadArgumentElimination.cpp b/llvm/lib/Transforms/IPO/DeadArgumentElimination.cpp
index bf2c65a2402cb..d6dc0f94bfcf0 100644
--- a/llvm/lib/Transforms/IPO/DeadArgumentElimination.cpp
+++ b/llvm/lib/Transforms/IPO/DeadArgumentElimination.cpp
@@ -85,6 +85,11 @@ class DAE : public ModulePass {
   virtual bool shouldHackArguments() const { return false; }
 };
 
+bool isMustTailCalleeAnalyzable(const CallBase &CB) {
+  assert(CB.isMustTailCall());
+  return CB.getCalledFunction() && !CB.getCalledFunction()->isDeclaration();
+}
+
 } // end anonymous namespace
 
 char DAE::ID = 0;
@@ -520,8 +525,16 @@ void DeadArgumentEliminationPass::surveyFunction(const Function &F) {
   for (const BasicBlock &BB : F) {
     // If we have any returns of `musttail` results - the signature can't
     // change
-    if (BB.getTerminatingMustTailCall() != nullptr)
+    if (const auto *TC = BB.getTerminatingMustTailCall()) {
       HasMustTailCalls = true;
+      // In addition, if the called function is not locally defined (or unknown,
+      // if this is an indirect call), we can't change the callsite and thus
+      // can't change this function's signature either.
+      if (!isMustTailCalleeAnalyzable(*TC)) {
+        markLive(F);
+        return;
+      }
+    }
   }
 
   if (HasMustTailCalls) {
@@ -1081,6 +1094,26 @@ bool DeadArgumentEliminationPass::removeDeadStuffFromFunction(Function *F) {
   return true;
 }
 
+void DeadArgumentEliminationPass::propagateVirtMustcallLiveness(
+    const Module &M) {
+  // If a function was marked "live", and it has musttail callers, they in turn
+  // can't change either.
+  LiveFuncSet NewLiveFuncs(LiveFunctions);
+  while (!NewLiveFuncs.empty()) {
+    LiveFuncSet Temp;
+    for (const auto *F : NewLiveFuncs)
+      for (const auto *U : F->users())
+        if (const auto *CB = dyn_cast<CallBase>(U))
+          if (CB->isMustTailCall())
+            if (!LiveFunctions.count(CB->getParent()->getParent()))
+              Temp.insert(CB->getParent()->getParent());
+    NewLiveFuncs.clear();
+    NewLiveFuncs.insert(Temp.begin(), Temp.end());
+    for (const auto *F : Temp)
+      markLive(*F);
+  }
+}
+
 PreservedAnalyses DeadArgumentEliminationPass::run(Module &M,
                                                    ModuleAnalysisManager &) {
   bool Changed = false;
@@ -1101,6 +1134,8 @@ PreservedAnalyses DeadArgumentEliminationPass::run(Module &M,
   for (auto &F : M)
     surveyFunction(F);
 
+  propagateVirtMustcallLiveness(M);
+
   // Now, remove all dead arguments and return values from each function in
   // turn.  We use make_early_inc_range here because functions will probably get
   // removed (i.e. replaced by new ones).
diff --git a/llvm/lib/Transforms/IPO/FunctionAttrs.cpp b/llvm/lib/Transforms/IPO/FunctionAttrs.cpp
index 68a4860dc1902..9d87ce5714e52 100644
--- a/llvm/lib/Transforms/IPO/FunctionAttrs.cpp
+++ b/llvm/lib/Transforms/IPO/FunctionAttrs.cpp
@@ -1763,6 +1763,13 @@ PreservedAnalyses PostOrderFunctionAttrsPass::run(LazyCallGraph::SCC &C,
                                                   CGSCCAnalysisManager &AM,
                                                   LazyCallGraph &CG,
                                                   CGSCCUpdateResult &) {
+  // Skip non-recursive functions if requested.
+  if (C.size() == 1 && SkipNonRecursive) {
+    LazyCallGraph::Node &N = *C.begin();
+    if (!N->lookup(N))
+      return PreservedAnalyses::all();
+  }
+
   FunctionAnalysisManager &FAM =
       AM.getResult<FunctionAnalysisManagerCGSCCProxy>(C, CG).getManager();
 
@@ -1808,6 +1815,14 @@ PreservedAnalyses PostOrderFunctionAttrsPass::run(LazyCallGraph::SCC &C,
   return PA;
 }
 
+void PostOrderFunctionAttrsPass::printPipeline(
+    raw_ostream &OS, function_ref<StringRef(StringRef)> MapClassName2PassName) {
+  static_cast<PassInfoMixin<PostOrderFunctionAttrsPass> *>(this)->printPipeline(
+      OS, MapClassName2PassName);
+  if (SkipNonRecursive)
+    OS << "<skip-non-recursive>";
+}
+
 template <typename AARGetterT>
 static bool runImpl(CallGraphSCC &SCC, AARGetterT AARGetter) {
   SmallVector<Function *, 8> Functions;
diff --git a/llvm/lib/Transforms/IPO/FunctionImport.cpp b/llvm/lib/Transforms/IPO/FunctionImport.cpp
index 222b719c1b89d..4875c18e1facf 100644
--- a/llvm/lib/Transforms/IPO/FunctionImport.cpp
+++ b/llvm/lib/Transforms/IPO/FunctionImport.cpp
@@ -1435,7 +1435,7 @@ static bool doImportingForModule(Module &M) {
   if (renameModuleForThinLTO(M, *Index, /*ClearDSOLocalOnDeclarations=*/false,
                              /*GlobalsToImport=*/nullptr)) {
     errs() << "Error renaming module\n";
-    return false;
+    return true;
   }
 
   // Perform the import now.
@@ -1450,10 +1450,10 @@ static bool doImportingForModule(Module &M) {
   if (!Result) {
     logAllUnhandledErrors(Result.takeError(), errs(),
                           "Error importing module: ");
-    return false;
+    return true;
   }
 
-  return *Result;
+  return true;
 }
 
 PreservedAnalyses FunctionImportPass::run(Module &M,
diff --git a/llvm/lib/Transforms/IPO/FunctionSpecialization.cpp b/llvm/lib/Transforms/IPO/FunctionSpecialization.cpp
index 51bc1786fdc79..8f5878feab60f 100644
--- a/llvm/lib/Transforms/IPO/FunctionSpecialization.cpp
+++ b/llvm/lib/Transforms/IPO/FunctionSpecialization.cpp
@@ -64,42 +64,39 @@ using namespace llvm;
 
 #define DEBUG_TYPE "function-specialization"
 
-STATISTIC(NumFuncSpecialized, "Number of functions specialized");
-
-static cl::opt<bool> ForceFunctionSpecialization(
-    "force-function-specialization", cl::init(false), cl::Hidden,
-    cl::desc("Force function specialization for every call site with a "
-             "constant argument"));
-
-static cl::opt<unsigned> MaxClonesThreshold(
-    "func-specialization-max-clones", cl::Hidden,
-    cl::desc("The maximum number of clones allowed for a single function "
-             "specialization"),
-    cl::init(3));
-
-static cl::opt<unsigned> SmallFunctionThreshold(
-    "func-specialization-size-threshold", cl::Hidden,
-    cl::desc("Don't specialize functions that have less than this theshold "
-             "number of instructions"),
-    cl::init(100));
-
-static cl::opt<unsigned>
-    AvgLoopIterationCount("func-specialization-avg-iters-cost", cl::Hidden,
-                          cl::desc("Average loop iteration count cost"),
-                          cl::init(10));
-
-static cl::opt<bool> SpecializeOnAddresses(
-    "func-specialization-on-address", cl::init(false), cl::Hidden,
-    cl::desc("Enable function specialization on the address of global values"));
+STATISTIC(NumSpecsCreated, "Number of specializations created");
+
+static cl::opt<bool> ForceSpecialization(
+    "force-specialization", cl::init(false), cl::Hidden, cl::desc(
+    "Force function specialization for every call site with a constant "
+    "argument"));
+
+static cl::opt<unsigned> MaxClones(
+    "funcspec-max-clones", cl::init(3), cl::Hidden, cl::desc(
+    "The maximum number of clones allowed for a single function "
+    "specialization"));
+
+static cl::opt<unsigned> MinFunctionSize(
+    "funcspec-min-function-size", cl::init(100), cl::Hidden, cl::desc(
+    "Don't specialize functions that have less than this number of "
+    "instructions"));
+
+static cl::opt<unsigned> AvgLoopIters(
+    "funcspec-avg-loop-iters", cl::init(10), cl::Hidden, cl::desc(
+    "Average loop iteration count"));
+
+static cl::opt<bool> SpecializeOnAddress(
+    "funcspec-on-address", cl::init(false), cl::Hidden, cl::desc(
+    "Enable function specialization on the address of global values"));
 
 // Disabled by default as it can significantly increase compilation times.
 //
 // https://llvm-compile-time-tracker.com
 // https://github.com/nikic/llvm-compile-time-tracker
-static cl::opt<bool> EnableSpecializationForLiteralConstant(
-    "function-specialization-for-literal-constant", cl::init(false), cl::Hidden,
-    cl::desc("Enable specialization of functions that take a literal constant "
-             "as an argument."));
+static cl::opt<bool> SpecializeLiteralConstant(
+    "funcspec-for-literal-constant", cl::init(false), cl::Hidden, cl::desc(
+    "Enable specialization of functions that take a literal constant as an "
+    "argument"));
 
 Constant *FunctionSpecializer::getPromotableAlloca(AllocaInst *Alloca,
                                                    CallInst *Call) {
@@ -234,7 +231,7 @@ static void removeSSACopy(Function &F) {
 
 /// Remove any ssa_copy intrinsics that may have been introduced.
 void FunctionSpecializer::cleanUpSSA() {
-  for (Function *F : SpecializedFuncs)
+  for (Function *F : Specializations)
     removeSSACopy(*F);
 }
 
@@ -253,6 +250,16 @@ template <> struct llvm::DenseMapInfo<SpecSig> {
   }
 };
 
+FunctionSpecializer::~FunctionSpecializer() {
+  LLVM_DEBUG(
+    if (NumSpecsCreated > 0)
+      dbgs() << "FnSpecialization: Created " << NumSpecsCreated
+             << " specializations in module " << M.getName() << "\n");
+  // Eliminate dead code.
+  removeDeadFunctions();
+  cleanUpSSA();
+}
+
 /// Attempt to specialize functions in the module to enable constant
 /// propagation across function boundaries.
 ///
@@ -300,7 +307,7 @@ bool FunctionSpecializer::run() {
     return AllSpecs[I].Gain > AllSpecs[J].Gain;
   };
   const unsigned NSpecs =
-      std::min(NumCandidates * MaxClonesThreshold, unsigned(AllSpecs.size()));
+      std::min(NumCandidates * MaxClones, unsigned(AllSpecs.size()));
   SmallVector<unsigned> BestSpecs(NSpecs + 1);
   std::iota(BestSpecs.begin(), BestSpecs.begin() + NSpecs, 0);
   if (AllSpecs.size() > NSpecs) {
@@ -358,11 +365,6 @@ bool FunctionSpecializer::run() {
   }
 
   promoteConstantStackValues();
-  LLVM_DEBUG(if (NbFunctionsSpecialized) dbgs()
-             << "FnSpecialization: Specialized " << NbFunctionsSpecialized
-             << " functions in module " << M.getName() << "\n");
-
-  NumFuncSpecialized += NbFunctionsSpecialized;
   return true;
 }
 
@@ -477,7 +479,7 @@ bool FunctionSpecializer::findSpecializations(Function *F, InstructionCost Cost,
             getSpecializationBonus(A.Formal, A.Actual, Solver.getLoopInfo(*F));
 
       // Discard unprofitable specialisations.
-      if (!ForceFunctionSpecialization && Gain <= 0)
+      if (!ForceSpecialization && Gain <= 0)
         continue;
 
       // Create a new specialisation entry.
@@ -506,7 +508,7 @@ bool FunctionSpecializer::isCandidateFunction(Function *F) {
     return false;
 
   // Do not specialize the cloned function again.
-  if (SpecializedFuncs.contains(F))
+  if (Specializations.contains(F))
     return false;
 
   // If we're optimizing the function for size, we shouldn't specialize it.
@@ -540,8 +542,8 @@ Function *FunctionSpecializer::createSpecialization(Function *F, const SpecSig &
   Solver.markBlockExecutable(&Clone->front());
 
   // Mark all the specialized functions
-  SpecializedFuncs.insert(Clone);
-  NbFunctionsSpecialized++;
+  Specializations.insert(Clone);
+  ++NumSpecsCreated;
 
   return Clone;
 }
@@ -554,9 +556,8 @@ InstructionCost FunctionSpecializer::getSpecializationCost(Function *F) {
   // Or if the lines of codes implies that this function is easy to get
   // inlined so that we shouldn't specialize it.
   if (Metrics.notDuplicatable || !Metrics.NumInsts.isValid() ||
-      (!ForceFunctionSpecialization &&
-       !F->hasFnAttribute(Attribute::NoInline) &&
-       Metrics.NumInsts < SmallFunctionThreshold))
+      (!ForceSpecialization && !F->hasFnAttribute(Attribute::NoInline) &&
+       Metrics.NumInsts < MinFunctionSize))
     return InstructionCost::getInvalid();
 
   // Otherwise, set the specialization cost to be the cost of all the
@@ -578,7 +579,7 @@ static InstructionCost getUserBonus(User *U, llvm::TargetTransformInfo &TTI,
 
   // Increase the cost if it is inside the loop.
   unsigned LoopDepth = LI.getLoopDepth(I->getParent());
-  Cost *= std::pow((double)AvgLoopIterationCount, LoopDepth);
+  Cost *= std::pow((double)AvgLoopIters, LoopDepth);
 
   // Traverse recursively if there are more uses.
   // TODO: Any other instructions to be added here?
@@ -673,7 +674,7 @@ bool FunctionSpecializer::isArgumentInteresting(Argument *A) {
 
   // Specialization of integer and floating point types needs to be explicitly
   // enabled.
-  if (!EnableSpecializationForLiteralConstant &&
+  if (!SpecializeLiteralConstant &&
       (ArgTy->isIntegerTy() || ArgTy->isFloatingPointTy()))
     return false;
 
@@ -709,7 +710,7 @@ Constant *FunctionSpecializer::getCandidateConstant(Value *V) {
   if (auto *GV = dyn_cast<GlobalVariable>(V)) {
     // Check if we want to specialize on the address of non-constant
     // global values.
-    if (!GV->isConstant() && !SpecializeOnAddresses)
+    if (!GV->isConstant() && !SpecializeOnAddress)
       return nullptr;
 
     if (!GV->getValueType()->isSingleValueType())
diff --git a/llvm/lib/Transforms/IPO/GlobalOpt.cpp b/llvm/lib/Transforms/IPO/GlobalOpt.cpp
index 48166bb90a85f..1d5661119df84 100644
--- a/llvm/lib/Transforms/IPO/GlobalOpt.cpp
+++ b/llvm/lib/Transforms/IPO/GlobalOpt.cpp
@@ -342,6 +342,7 @@ static bool CleanupConstantGlobalUsers(GlobalVariable *GV,
 /// loads and stores with the given type.
 struct GlobalPart {
   Type *Ty;
+  Constant *Initializer = nullptr;
   bool IsLoaded = false;
   bool IsStored = false;
 };
@@ -384,16 +385,41 @@ static bool collectSRATypes(DenseMap<uint64_t, GlobalPart> &Parts,
       // TODO: We currently require that all accesses at a given offset must
       // use the same type. This could be relaxed.
       Type *Ty = getLoadStoreType(V);
-      auto It = Parts.try_emplace(Offset.getZExtValue(), GlobalPart{Ty}).first;
+      const auto &[It, Inserted] =
+          Parts.try_emplace(Offset.getZExtValue(), GlobalPart{Ty});
       if (Ty != It->second.Ty)
         return false;
 
+      if (Inserted) {
+        It->second.Initializer =
+            ConstantFoldLoadFromConst(GV->getInitializer(), Ty, Offset, DL);
+        if (!It->second.Initializer) {
+          LLVM_DEBUG(dbgs() << "Global SRA: Failed to evaluate initializer of "
+                            << *GV << " with type " << *Ty << " at offset "
+                            << Offset.getZExtValue());
+          return false;
+        }
+      }
+
       // Scalable types not currently supported.
       if (isa<ScalableVectorType>(Ty))
         return false;
 
+      auto IsStored = [](Value *V, Constant *Initializer) {
+        auto *SI = dyn_cast<StoreInst>(V);
+        if (!SI)
+          return false;
+
+        Constant *StoredConst = dyn_cast<Constant>(SI->getOperand(0));
+        if (!StoredConst)
+          return true;
+
+        // Don't consider stores that only write the initializer value.
+        return Initializer != StoredConst;
+      };
+
       It->second.IsLoaded |= isa<LoadInst>(V);
-      It->second.IsStored |= isa<StoreInst>(V);
+      It->second.IsStored |= IsStored(V, It->second.Initializer);
       continue;
     }
 
@@ -423,6 +449,7 @@ static void transferSRADebugInfo(GlobalVariable *GV, GlobalVariable *NGV,
     DIExpression *Expr = GVE->getExpression();
     int64_t CurVarOffsetInBytes = 0;
     uint64_t CurVarOffsetInBits = 0;
+    uint64_t FragmentEndInBits = FragmentOffsetInBits + FragmentSizeInBits;
 
     // Calculate the offset (Bytes), Continue if unknown.
     if (!Expr->extractIfOffset(CurVarOffsetInBytes))
@@ -436,27 +463,50 @@ static void transferSRADebugInfo(GlobalVariable *GV, GlobalVariable *NGV,
     CurVarOffsetInBits = CHAR_BIT * (uint64_t)CurVarOffsetInBytes;
 
     // Current var starts after the fragment, ignore.
-    if (CurVarOffsetInBits >= (FragmentOffsetInBits + FragmentSizeInBits))
+    if (CurVarOffsetInBits >= FragmentEndInBits)
       continue;
 
     uint64_t CurVarSize = Var->getType()->getSizeInBits();
+    uint64_t CurVarEndInBits = CurVarOffsetInBits + CurVarSize;
     // Current variable ends before start of fragment, ignore.
-    if (CurVarSize != 0 &&
-        (CurVarOffsetInBits + CurVarSize) <= FragmentOffsetInBits)
+    if (CurVarSize != 0 && /* CurVarSize is known */
+        CurVarEndInBits <= FragmentOffsetInBits)
       continue;
 
-    // Current variable fits in the fragment.
-    if (CurVarOffsetInBits == FragmentOffsetInBits &&
-        CurVarSize == FragmentSizeInBits)
-      Expr = DIExpression::get(Expr->getContext(), {});
-    // If the FragmentSize is smaller than the variable,
+    // Current variable fits in (not greater than) the fragment,
+    // does not need fragment expression.
+    if (CurVarSize != 0 && /* CurVarSize is known */
+        CurVarOffsetInBits >= FragmentOffsetInBits &&
+        CurVarEndInBits <= FragmentEndInBits) {
+      uint64_t CurVarOffsetInFragment =
+          (CurVarOffsetInBits - FragmentOffsetInBits) / 8;
+      if (CurVarOffsetInFragment != 0)
+        Expr = DIExpression::get(Expr->getContext(), {dwarf::DW_OP_plus_uconst,
+                                                      CurVarOffsetInFragment});
+      else
+        Expr = DIExpression::get(Expr->getContext(), {});
+      auto *NGVE =
+          DIGlobalVariableExpression::get(GVE->getContext(), Var, Expr);
+      NGV->addDebugInfo(NGVE);
+      continue;
+    }
+    // Current variable does not fit in single fragment,
     // emit a fragment expression.
-    else if (FragmentSizeInBits < VarSize) {
+    if (FragmentSizeInBits < VarSize) {
+      if (CurVarOffsetInBits > FragmentOffsetInBits)
+        continue;
+      uint64_t CurVarFragmentOffsetInBits =
+          FragmentOffsetInBits - CurVarOffsetInBits;
+      uint64_t CurVarFragmentSizeInBits = FragmentSizeInBits;
+      if (CurVarSize != 0 && CurVarEndInBits < FragmentEndInBits)
+        CurVarFragmentSizeInBits -= (FragmentEndInBits - CurVarEndInBits);
+      if (CurVarOffsetInBits)
+        Expr = DIExpression::get(Expr->getContext(), {});
       if (auto E = DIExpression::createFragmentExpression(
-              Expr, FragmentOffsetInBits, FragmentSizeInBits))
+              Expr, CurVarFragmentOffsetInBits, CurVarFragmentSizeInBits))
         Expr = *E;
       else
-        return;
+        continue;
     }
     auto *NGVE = DIGlobalVariableExpression::get(GVE->getContext(), Var, Expr);
     NGV->addDebugInfo(NGVE);
@@ -490,40 +540,27 @@ static GlobalVariable *SRAGlobal(GlobalVariable *GV, const DataLayout &DL) {
     return nullptr;
 
   // Sort by offset.
-  SmallVector<std::pair<uint64_t, Type *>, 16> TypesVector;
-  for (const auto &Pair : Parts)
-    TypesVector.push_back({Pair.first, Pair.second.Ty});
+  SmallVector<std::tuple<uint64_t, Type *, Constant *>, 16> TypesVector;
+  for (const auto &Pair : Parts) {
+    TypesVector.push_back(
+        {Pair.first, Pair.second.Ty, Pair.second.Initializer});
+  }
   sort(TypesVector, llvm::less_first());
 
   // Check that the types are non-overlapping.
   uint64_t Offset = 0;
-  for (const auto &Pair : TypesVector) {
+  for (const auto &[OffsetForTy, Ty, _] : TypesVector) {
     // Overlaps with previous type.
-    if (Pair.first < Offset)
+    if (OffsetForTy < Offset)
       return nullptr;
 
-    Offset = Pair.first + DL.getTypeAllocSize(Pair.second);
+    Offset = OffsetForTy + DL.getTypeAllocSize(Ty);
   }
 
   // Some accesses go beyond the end of the global, don't bother.
   if (Offset > DL.getTypeAllocSize(GV->getValueType()))
     return nullptr;
 
-  // Collect initializers for new globals.
-  Constant *OrigInit = GV->getInitializer();
-  DenseMap<uint64_t, Constant *> Initializers;
-  for (const auto &Pair : TypesVector) {
-    Constant *NewInit = ConstantFoldLoadFromConst(OrigInit, Pair.second,
-                                                  APInt(64, Pair.first), DL);
-    if (!NewInit) {
-      LLVM_DEBUG(dbgs() << "Global SRA: Failed to evaluate initializer of "
-                        << *GV << " with type " << *Pair.second << " at offset "
-                        << Pair.first << "\n");
-      return nullptr;
-    }
-    Initializers.insert({Pair.first, NewInit});
-  }
-
   LLVM_DEBUG(dbgs() << "PERFORMING GLOBAL SRA ON: " << *GV << "\n");
 
   // Get the alignment of the global, either explicit or target-specific.
@@ -534,26 +571,24 @@ static GlobalVariable *SRAGlobal(GlobalVariable *GV, const DataLayout &DL) {
   // Create replacement globals.
   DenseMap<uint64_t, GlobalVariable *> NewGlobals;
   unsigned NameSuffix = 0;
-  for (auto &Pair : TypesVector) {
-    uint64_t Offset = Pair.first;
-    Type *Ty = Pair.second;
+  for (auto &[OffsetForTy, Ty, Initializer] : TypesVector) {
     GlobalVariable *NGV = new GlobalVariable(
         *GV->getParent(), Ty, false, GlobalVariable::InternalLinkage,
-        Initializers[Offset], GV->getName() + "." + Twine(NameSuffix++), GV,
+        Initializer, GV->getName() + "." + Twine(NameSuffix++), GV,
         GV->getThreadLocalMode(), GV->getAddressSpace());
     NGV->copyAttributesFrom(GV);
-    NewGlobals.insert({Offset, NGV});
+    NewGlobals.insert({OffsetForTy, NGV});
 
     // Calculate the known alignment of the field.  If the original aggregate
     // had 256 byte alignment for example, something might depend on that:
     // propagate info to each field.
-    Align NewAlign = commonAlignment(StartAlignment, Offset);
+    Align NewAlign = commonAlignment(StartAlignment, OffsetForTy);
     if (NewAlign > DL.getABITypeAlign(Ty))
       NGV->setAlignment(NewAlign);
 
     // Copy over the debug info for the variable.
-    transferSRADebugInfo(GV, NGV, Offset * 8, DL.getTypeAllocSizeInBits(Ty),
-                         VarSize);
+    transferSRADebugInfo(GV, NGV, OffsetForTy * 8,
+                         DL.getTypeAllocSizeInBits(Ty), VarSize);
   }
 
   // Replace uses of the original global with uses of the new global.
diff --git a/llvm/lib/Transforms/IPO/IROutliner.cpp b/llvm/lib/Transforms/IPO/IROutliner.cpp
index 82f890a50f059..788dc9b2ba9f6 100644
--- a/llvm/lib/Transforms/IPO/IROutliner.cpp
+++ b/llvm/lib/Transforms/IPO/IROutliner.cpp
@@ -588,7 +588,7 @@ collectRegionsConstants(OutlinableRegion &Region,
       // While this value is a register, it might not have been previously,
       // make sure we don't already have a constant mapped to this global value
       // number.
-      if (GVNToConstant.find(GVN) != GVNToConstant.end())
+      if (GVNToConstant.contains(GVN))
         ConstantsTheSame = false;
 
       NotSame.insert(GVN);
@@ -816,7 +816,7 @@ static void mapInputsToGVNs(IRSimilarityCandidate &C,
   // replacement.
   for (Value *Input : CurrentInputs) {
     assert(Input && "Have a nullptr as an input");
-    if (OutputMappings.find(Input) != OutputMappings.end())
+    if (OutputMappings.contains(Input))
       Input = OutputMappings.find(Input)->second;
     assert(C.getGVN(Input) && "Could not find a numbering for the given input");
     EndInputNumbers.push_back(*C.getGVN(Input));
@@ -838,7 +838,7 @@ remapExtractedInputs(const ArrayRef<Value *> ArgInputs,
   // Get the global value number for each input that will be extracted as an
   // argument by the code extractor, remapping if needed for reloaded values.
   for (Value *Input : ArgInputs) {
-    if (OutputMappings.find(Input) != OutputMappings.end())
+    if (OutputMappings.contains(Input))
       Input = OutputMappings.find(Input)->second;
     RemappedArgInputs.insert(Input);
   }
@@ -1481,8 +1481,7 @@ CallInst *replaceCalledFunction(Module &M, OutlinableRegion &Region) {
     }
 
     // If it is a constant, we simply add it to the argument list as a value.
-    if (Region.AggArgToConstant.find(AggArgIdx) !=
-        Region.AggArgToConstant.end()) {
+    if (Region.AggArgToConstant.contains(AggArgIdx)) {
       Constant *CST = Region.AggArgToConstant.find(AggArgIdx)->second;
       LLVM_DEBUG(dbgs() << "Setting argument " << AggArgIdx << " to value "
                         << *CST << "\n");
@@ -1816,8 +1815,7 @@ replaceArgumentUses(OutlinableRegion &Region,
 
   for (unsigned ArgIdx = 0; ArgIdx < Region.ExtractedFunction->arg_size();
        ArgIdx++) {
-    assert(Region.ExtractedArgToAgg.find(ArgIdx) !=
-               Region.ExtractedArgToAgg.end() &&
+    assert(Region.ExtractedArgToAgg.contains(ArgIdx) &&
            "No mapping from extracted to outlined?");
     unsigned AggArgIdx = Region.ExtractedArgToAgg.find(ArgIdx)->second;
     Argument *AggArg = Group.OutlinedFunction->getArg(AggArgIdx);
@@ -2698,7 +2696,7 @@ void IROutliner::updateOutputMapping(OutlinableRegion &Region,
   if (!OutputIdx)
     return;
 
-  if (OutputMappings.find(Outputs[*OutputIdx]) == OutputMappings.end()) {
+  if (!OutputMappings.contains(Outputs[*OutputIdx])) {
     LLVM_DEBUG(dbgs() << "Mapping extracted output " << *LI << " to "
                       << *Outputs[*OutputIdx] << "\n");
     OutputMappings.insert(std::make_pair(LI, Outputs[*OutputIdx]));
diff --git a/llvm/lib/Transforms/IPO/Inliner.cpp b/llvm/lib/Transforms/IPO/Inliner.cpp
index 748a27e3fa5a5..7340edcbd6bec 100644
--- a/llvm/lib/Transforms/IPO/Inliner.cpp
+++ b/llvm/lib/Transforms/IPO/Inliner.cpp
@@ -73,18 +73,6 @@ using namespace llvm;
 STATISTIC(NumInlined, "Number of functions inlined");
 STATISTIC(NumCallsDeleted, "Number of call sites deleted, not inlined");
 STATISTIC(NumDeleted, "Number of functions deleted because all callers found");
-STATISTIC(NumMergedAllocas, "Number of allocas merged together");
-
-/// Flag to disable manual alloca merging.
-///
-/// Merging of allocas was originally done as a stack-size saving technique
-/// prior to LLVM's code generator having support for stack coloring based on
-/// lifetime markers. It is now in the process of being removed. To experiment
-/// with disabling it and relying fully on lifetime marker based stack
-/// coloring, you can pass this flag to LLVM.
-static cl::opt<bool>
-    DisableInlinedAllocaMerging("disable-inlined-alloca-merging",
-                                cl::init(false), cl::Hidden);
 
 static cl::opt<int> IntraSCCCostMultiplier(
     "intra-scc-cost-multiplier", cl::init(2), cl::Hidden,
@@ -181,121 +169,6 @@ void LegacyInlinerBase::getAnalysisUsage(AnalysisUsage &AU) const {
 
 using InlinedArrayAllocasTy = DenseMap<ArrayType *, std::vector<AllocaInst *>>;
 
-/// Look at all of the allocas that we inlined through this call site.  If we
-/// have already inlined other allocas through other calls into this function,
-/// then we know that they have disjoint lifetimes and that we can merge them.
-///
-/// There are many heuristics possible for merging these allocas, and the
-/// different options have different tradeoffs.  One thing that we *really*
-/// don't want to hurt is SRoA: once inlining happens, often allocas are no
-/// longer address taken and so they can be promoted.
-///
-/// Our "solution" for that is to only merge allocas whose outermost type is an
-/// array type.  These are usually not promoted because someone is using a
-/// variable index into them.  These are also often the most important ones to
-/// merge.
-///
-/// A better solution would be to have real memory lifetime markers in the IR
-/// and not have the inliner do any merging of allocas at all.  This would
-/// allow the backend to do proper stack slot coloring of all allocas that
-/// *actually make it to the backend*, which is really what we want.
-///
-/// Because we don't have this information, we do this simple and useful hack.
-static void mergeInlinedArrayAllocas(Function *Caller, InlineFunctionInfo &IFI,
-                                     InlinedArrayAllocasTy &InlinedArrayAllocas,
-                                     int InlineHistory) {
-  SmallPtrSet<AllocaInst *, 16> UsedAllocas;
-
-  // When processing our SCC, check to see if the call site was inlined from
-  // some other call site.  For example, if we're processing "A" in this code:
-  //   A() { B() }
-  //   B() { x = alloca ... C() }
-  //   C() { y = alloca ... }
-  // Assume that C was not inlined into B initially, and so we're processing A
-  // and decide to inline B into A.  Doing this makes an alloca available for
-  // reuse and makes a callsite (C) available for inlining.  When we process
-  // the C call site we don't want to do any alloca merging between X and Y
-  // because their scopes are not disjoint.  We could make this smarter by
-  // keeping track of the inline history for each alloca in the
-  // InlinedArrayAllocas but this isn't likely to be a significant win.
-  if (InlineHistory != -1) // Only do merging for top-level call sites in SCC.
-    return;
-
-  // Loop over all the allocas we have so far and see if they can be merged with
-  // a previously inlined alloca.  If not, remember that we had it.
-  for (unsigned AllocaNo = 0, E = IFI.StaticAllocas.size(); AllocaNo != E;
-       ++AllocaNo) {
-    AllocaInst *AI = IFI.StaticAllocas[AllocaNo];
-
-    // Don't bother trying to merge array allocations (they will usually be
-    // canonicalized to be an allocation *of* an array), or allocations whose
-    // type is not itself an array (because we're afraid of pessimizing SRoA).
-    ArrayType *ATy = dyn_cast<ArrayType>(AI->getAllocatedType());
-    if (!ATy || AI->isArrayAllocation())
-      continue;
-
-    // Get the list of all available allocas for this array type.
-    std::vector<AllocaInst *> &AllocasForType = InlinedArrayAllocas[ATy];
-
-    // Loop over the allocas in AllocasForType to see if we can reuse one.  Note
-    // that we have to be careful not to reuse the same "available" alloca for
-    // multiple different allocas that we just inlined, we use the 'UsedAllocas'
-    // set to keep track of which "available" allocas are being used by this
-    // function.  Also, AllocasForType can be empty of course!
-    bool MergedAwayAlloca = false;
-    for (AllocaInst *AvailableAlloca : AllocasForType) {
-      Align Align1 = AI->getAlign();
-      Align Align2 = AvailableAlloca->getAlign();
-
-      // The available alloca has to be in the right function, not in some other
-      // function in this SCC.
-      if (AvailableAlloca->getParent() != AI->getParent())
-        continue;
-
-      // If the inlined function already uses this alloca then we can't reuse
-      // it.
-      if (!UsedAllocas.insert(AvailableAlloca).second)
-        continue;
-
-      // Otherwise, we *can* reuse it, RAUW AI into AvailableAlloca and declare
-      // success!
-      LLVM_DEBUG(dbgs() << "    ***MERGED ALLOCA: " << *AI
-                        << "\n\t\tINTO: " << *AvailableAlloca << '\n');
-
-      // Move affected dbg.declare calls immediately after the new alloca to
-      // avoid the situation when a dbg.declare precedes its alloca.
-      if (auto *L = LocalAsMetadata::getIfExists(AI))
-        if (auto *MDV = MetadataAsValue::getIfExists(AI->getContext(), L))
-          for (User *U : MDV->users())
-            if (DbgDeclareInst *DDI = dyn_cast<DbgDeclareInst>(U))
-              DDI->moveBefore(AvailableAlloca->getNextNode());
-
-      AI->replaceAllUsesWith(AvailableAlloca);
-
-      if (Align1 > Align2)
-        AvailableAlloca->setAlignment(AI->getAlign());
-
-      AI->eraseFromParent();
-      MergedAwayAlloca = true;
-      ++NumMergedAllocas;
-      IFI.StaticAllocas[AllocaNo] = nullptr;
-      break;
-    }
-
-    // If we already nuked the alloca, we're done with it.
-    if (MergedAwayAlloca)
-      continue;
-
-    // If we were unable to merge away the alloca either because there are no
-    // allocas of the right type available or because we reused them all
-    // already, remember that this alloca came from an inlined function and mark
-    // it used so we don't reuse it for other allocas from this inline
-    // operation.
-    AllocasForType.push_back(AI);
-    UsedAllocas.insert(AI);
-  }
-}
-
 /// If it is possible to inline the specified call site,
 /// do so and update the CallGraph for this operation.
 ///
@@ -325,9 +198,6 @@ static InlineResult inlineCallIfPossible(
   if (InlinerFunctionImportStats != InlinerFunctionImportStatsOpts::No)
     ImportedFunctionsStats.recordInline(*Caller, *Callee);
 
-  if (!DisableInlinedAllocaMerging)
-    mergeInlinedArrayAllocas(Caller, IFI, InlinedArrayAllocas, InlineHistory);
-
   return IR; // success
 }
 
@@ -555,7 +425,8 @@ inlineCallsImpl(CallGraphSCC &SCC, CallGraph &CG,
 
       // If we inlined or deleted the last possible call site to the function,
       // delete the function body now.
-      if (Callee && Callee->use_empty() && Callee->hasLocalLinkage() &&
+      assert(Callee && "Expected to be non-null due to check at start of loop");
+      if (Callee->use_empty() && Callee->hasLocalLinkage() &&
           // TODO: Can remove if in SCC now.
           !SCCFunctions.count(Callee) &&
           // The function may be apparently dead, but if there are indirect
@@ -729,8 +600,7 @@ InlinerPass::getAdvisor(const ModuleAnalysisManagerCGSCCProxy::Result &MAM,
                                 CGSCCInlineReplayFallback,
                                 {CGSCCInlineReplayFormat}},
           /*EmitRemarks=*/true,
-          InlineContext{LTOPhase,
-                              InlinePass::ReplayCGSCCInliner});
+          InlineContext{LTOPhase, InlinePass::ReplayCGSCCInliner});
 
     return *OwnedAdvisor;
   }
@@ -871,8 +741,8 @@ PreservedAnalyses InlinerPass::run(LazyCallGraph::SCC &InitialC,
 
       if (InlineHistoryID != -1 &&
           inlineHistoryIncludes(&Callee, InlineHistoryID, InlineHistory)) {
-        LLVM_DEBUG(dbgs() << "Skipping inlining due to history: "
-                          << F.getName() << " -> " << Callee.getName() << "\n");
+        LLVM_DEBUG(dbgs() << "Skipping inlining due to history: " << F.getName()
+                          << " -> " << Callee.getName() << "\n");
         setInlineRemark(*CB, "recursive");
         continue;
       }
diff --git a/llvm/lib/Transforms/IPO/LowerTypeTests.cpp b/llvm/lib/Transforms/IPO/LowerTypeTests.cpp
index ccb824595cfcf..8380afaaaf982 100644
--- a/llvm/lib/Transforms/IPO/LowerTypeTests.cpp
+++ b/llvm/lib/Transforms/IPO/LowerTypeTests.cpp
@@ -1370,15 +1370,25 @@ void LowerTypeTestsModule::replaceWeakDeclarationWithJumpTablePtr(
   replaceCfiUses(F, PlaceholderFn, IsJumpTableCanonical);
 
   convertUsersOfConstantsToInstructions(PlaceholderFn);
-  for (Use &U : make_early_inc_range(PlaceholderFn->uses())) {
-    auto *I = dyn_cast<Instruction>(U.getUser());
-    assert(I && "Non-instruction users should have been eliminated");
-    IRBuilder Builder(I);
+  // Don't use range based loop, because use list will be modified.
+  while (!PlaceholderFn->use_empty()) {
+    Use &U = *PlaceholderFn->use_begin();
+    auto *InsertPt = dyn_cast<Instruction>(U.getUser());
+    assert(InsertPt && "Non-instruction users should have been eliminated");
+    auto *PN = dyn_cast<PHINode>(InsertPt);
+    if (PN)
+      InsertPt = PN->getIncomingBlock(U)->getTerminator();
+    IRBuilder Builder(InsertPt);
     Value *ICmp = Builder.CreateICmp(CmpInst::ICMP_NE, F,
                                      Constant::getNullValue(F->getType()));
     Value *Select = Builder.CreateSelect(ICmp, JT,
                                          Constant::getNullValue(F->getType()));
-    U.set(Select);
+    // For phi nodes, we need to update the incoming value for all operands
+    // with the same predecessor.
+    if (PN)
+      PN->setIncomingValueForBlock(InsertPt->getParent(), Select);
+    else
+      U.set(Select);
   }
   PlaceholderFn->eraseFromParent();
 }
diff --git a/llvm/lib/Transforms/IPO/ModuleInliner.cpp b/llvm/lib/Transforms/IPO/ModuleInliner.cpp
index ee382657f5e69..956a8597c8941 100644
--- a/llvm/lib/Transforms/IPO/ModuleInliner.cpp
+++ b/llvm/lib/Transforms/IPO/ModuleInliner.cpp
@@ -138,7 +138,7 @@ PreservedAnalyses ModuleInlinerPass::run(Module &M,
   //
   // TODO: Here is a huge amount duplicate code between the module inliner and
   // the SCC inliner, which need some refactoring.
-  auto Calls = getInlineOrder(FAM, Params);
+  auto Calls = getInlineOrder(FAM, Params, MAM, M);
   assert(Calls != nullptr && "Expected an initialized InlineOrder");
 
   // Populate the initial list of calls in this module.
diff --git a/llvm/lib/Transforms/IPO/PassManagerBuilder.cpp b/llvm/lib/Transforms/IPO/PassManagerBuilder.cpp
deleted file mode 100644
index 5fec9dcbadb35..0000000000000
--- a/llvm/lib/Transforms/IPO/PassManagerBuilder.cpp
+++ /dev/null
@@ -1,428 +0,0 @@
-//===- PassManagerBuilder.cpp - Build Standard Pass -----------------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// This file defines the PassManagerBuilder class, which is used to set up a
-// "standard" optimization sequence suitable for languages like C and C++.
-//
-//===----------------------------------------------------------------------===//
-
-#include "llvm/Transforms/IPO/PassManagerBuilder.h"
-#include "llvm-c/Transforms/PassManagerBuilder.h"
-#include "llvm/ADT/STLExtras.h"
-#include "llvm/ADT/SmallVector.h"
-#include "llvm/Analysis/GlobalsModRef.h"
-#include "llvm/Analysis/ScopedNoAliasAA.h"
-#include "llvm/Analysis/TargetLibraryInfo.h"
-#include "llvm/Analysis/TypeBasedAliasAnalysis.h"
-#include "llvm/IR/LegacyPassManager.h"
-#include "llvm/Support/CommandLine.h"
-#include "llvm/Support/ManagedStatic.h"
-#include "llvm/Target/CGPassBuilderOption.h"
-#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
-#include "llvm/Transforms/IPO.h"
-#include "llvm/Transforms/IPO/Attributor.h"
-#include "llvm/Transforms/IPO/ForceFunctionAttrs.h"
-#include "llvm/Transforms/IPO/FunctionAttrs.h"
-#include "llvm/Transforms/IPO/InferFunctionAttrs.h"
-#include "llvm/Transforms/InstCombine/InstCombine.h"
-#include "llvm/Transforms/Instrumentation.h"
-#include "llvm/Transforms/Scalar.h"
-#include "llvm/Transforms/Scalar/GVN.h"
-#include "llvm/Transforms/Scalar/LICM.h"
-#include "llvm/Transforms/Scalar/LoopUnrollPass.h"
-#include "llvm/Transforms/Scalar/SimpleLoopUnswitch.h"
-#include "llvm/Transforms/Utils.h"
-#include "llvm/Transforms/Vectorize.h"
-
-using namespace llvm;
-
-PassManagerBuilder::PassManagerBuilder() {
-    OptLevel = 2;
-    SizeLevel = 0;
-    LibraryInfo = nullptr;
-    Inliner = nullptr;
-    DisableUnrollLoops = false;
-    SLPVectorize = false;
-    LoopVectorize = true;
-    LoopsInterleaved = true;
-    LicmMssaOptCap = SetLicmMssaOptCap;
-    LicmMssaNoAccForPromotionCap = SetLicmMssaNoAccForPromotionCap;
-    DisableGVNLoadPRE = false;
-    ForgetAllSCEVInLoopUnroll = ForgetSCEVInLoopUnroll;
-    VerifyInput = false;
-    VerifyOutput = false;
-    DivergentTarget = false;
-    CallGraphProfile = true;
-}
-
-PassManagerBuilder::~PassManagerBuilder() {
-  delete LibraryInfo;
-  delete Inliner;
-}
-
-void PassManagerBuilder::addInitialAliasAnalysisPasses(
-    legacy::PassManagerBase &PM) const {
-  // Add TypeBasedAliasAnalysis before BasicAliasAnalysis so that
-  // BasicAliasAnalysis wins if they disagree. This is intended to help
-  // support "obvious" type-punning idioms.
-  PM.add(createTypeBasedAAWrapperPass());
-  PM.add(createScopedNoAliasAAWrapperPass());
-}
-
-void PassManagerBuilder::populateFunctionPassManager(
-    legacy::FunctionPassManager &FPM) {
-  // Add LibraryInfo if we have some.
-  if (LibraryInfo)
-    FPM.add(new TargetLibraryInfoWrapperPass(*LibraryInfo));
-
-  if (OptLevel == 0) return;
-
-  addInitialAliasAnalysisPasses(FPM);
-
-  // Lower llvm.expect to metadata before attempting transforms.
-  // Compare/branch metadata may alter the behavior of passes like SimplifyCFG.
-  FPM.add(createLowerExpectIntrinsicPass());
-  FPM.add(createCFGSimplificationPass());
-  FPM.add(createSROAPass());
-  FPM.add(createEarlyCSEPass());
-}
-
-void PassManagerBuilder::addFunctionSimplificationPasses(
-    legacy::PassManagerBase &MPM) {
-  // Start of function pass.
-  // Break up aggregate allocas, using SSAUpdater.
-  assert(OptLevel >= 1 && "Calling function optimizer with no optimization level!");
-  MPM.add(createSROAPass());
-  MPM.add(createEarlyCSEPass(true /* Enable mem-ssa. */)); // Catch trivial redundancies
-
-  if (OptLevel > 1) {
-    // Speculative execution if the target has divergent branches; otherwise nop.
-    MPM.add(createSpeculativeExecutionIfHasBranchDivergencePass());
-
-    MPM.add(createJumpThreadingPass());         // Thread jumps.
-    MPM.add(createCorrelatedValuePropagationPass()); // Propagate conditionals
-  }
-  MPM.add(
-      createCFGSimplificationPass(SimplifyCFGOptions().convertSwitchRangeToICmp(
-          true))); // Merge & remove BBs
-  // Combine silly seq's
-  MPM.add(createInstructionCombiningPass());
-  if (SizeLevel == 0)
-    MPM.add(createLibCallsShrinkWrapPass());
-
-  // TODO: Investigate the cost/benefit of tail call elimination on debugging.
-  if (OptLevel > 1)
-    MPM.add(createTailCallEliminationPass()); // Eliminate tail calls
-  MPM.add(
-      createCFGSimplificationPass(SimplifyCFGOptions().convertSwitchRangeToICmp(
-          true)));                            // Merge & remove BBs
-  MPM.add(createReassociatePass());           // Reassociate expressions
-
-  // Begin the loop pass pipeline.
-
-  // The simple loop unswitch pass relies on separate cleanup passes. Schedule
-  // them first so when we re-process a loop they run before other loop
-  // passes.
-  MPM.add(createLoopInstSimplifyPass());
-  MPM.add(createLoopSimplifyCFGPass());
-
-  // Try to remove as much code from the loop header as possible,
-  // to reduce amount of IR that will have to be duplicated. However,
-  // do not perform speculative hoisting the first time as LICM
-  // will destroy metadata that may not need to be destroyed if run
-  // after loop rotation.
-  // TODO: Investigate promotion cap for O1.
-  MPM.add(createLICMPass(LicmMssaOptCap, LicmMssaNoAccForPromotionCap,
-                         /*AllowSpeculation=*/false));
-  // Rotate Loop - disable header duplication at -Oz
-  MPM.add(createLoopRotatePass(SizeLevel == 2 ? 0 : -1, false));
-  // TODO: Investigate promotion cap for O1.
-  MPM.add(createLICMPass(LicmMssaOptCap, LicmMssaNoAccForPromotionCap,
-                         /*AllowSpeculation=*/true));
-  MPM.add(createSimpleLoopUnswitchLegacyPass(OptLevel == 3));
-  // FIXME: We break the loop pass pipeline here in order to do full
-  // simplifycfg. Eventually loop-simplifycfg should be enhanced to replace the
-  // need for this.
-  MPM.add(createCFGSimplificationPass(
-      SimplifyCFGOptions().convertSwitchRangeToICmp(true)));
-  MPM.add(createInstructionCombiningPass());
-  // We resume loop passes creating a second loop pipeline here.
-  MPM.add(createIndVarSimplifyPass());        // Canonicalize indvars
-
-  // Unroll small loops and perform peeling.
-  MPM.add(createSimpleLoopUnrollPass(OptLevel, DisableUnrollLoops,
-                                     ForgetAllSCEVInLoopUnroll));
-  // This ends the loop pass pipelines.
-
-  // Break up allocas that may now be splittable after loop unrolling.
-  MPM.add(createSROAPass());
-
-  if (OptLevel > 1) {
-    MPM.add(createMergedLoadStoreMotionPass()); // Merge ld/st in diamonds
-    MPM.add(createGVNPass(DisableGVNLoadPRE));  // Remove redundancies
-  }
-
-  // Delete dead bit computations (instcombine runs after to fold away the dead
-  // computations, and then ADCE will run later to exploit any new DCE
-  // opportunities that creates).
-  MPM.add(createBitTrackingDCEPass());        // Delete dead bit computations
-
-  // Run instcombine after redundancy elimination to exploit opportunities
-  // opened up by them.
-  MPM.add(createInstructionCombiningPass());
-  if (OptLevel > 1) {
-    MPM.add(createJumpThreadingPass());         // Thread jumps
-    MPM.add(createCorrelatedValuePropagationPass());
-  }
-  MPM.add(createAggressiveDCEPass()); // Delete dead instructions
-
-  MPM.add(createMemCpyOptPass());               // Remove memcpy / form memset
-  // TODO: Investigate if this is too expensive at O1.
-  if (OptLevel > 1) {
-    MPM.add(createDeadStoreEliminationPass());  // Delete dead stores
-    MPM.add(createLICMPass(LicmMssaOptCap, LicmMssaNoAccForPromotionCap,
-                           /*AllowSpeculation=*/true));
-  }
-
-  // Merge & remove BBs and sink & hoist common instructions.
-  MPM.add(createCFGSimplificationPass(
-      SimplifyCFGOptions().hoistCommonInsts(true).sinkCommonInsts(true)));
-  // Clean up after everything.
-  MPM.add(createInstructionCombiningPass());
-}
-
-/// FIXME: Should LTO cause any differences to this set of passes?
-void PassManagerBuilder::addVectorPasses(legacy::PassManagerBase &PM,
-                                         bool IsFullLTO) {
-  PM.add(createLoopVectorizePass(!LoopsInterleaved, !LoopVectorize));
-
-  if (IsFullLTO) {
-    // The vectorizer may have significantly shortened a loop body; unroll
-    // again. Unroll small loops to hide loop backedge latency and saturate any
-    // parallel execution resources of an out-of-order processor. We also then
-    // need to clean up redundancies and loop invariant code.
-    // FIXME: It would be really good to use a loop-integrated instruction
-    // combiner for cleanup here so that the unrolling and LICM can be pipelined
-    // across the loop nests.
-    PM.add(createLoopUnrollPass(OptLevel, DisableUnrollLoops,
-                                ForgetAllSCEVInLoopUnroll));
-  }
-
-  // Cleanup after the loop optimization passes.
-  PM.add(createInstructionCombiningPass());
-
-  // Now that we've formed fast to execute loop structures, we do further
-  // optimizations. These are run afterward as they might block doing complex
-  // analyses and transforms such as what are needed for loop vectorization.
-
-  // Cleanup after loop vectorization, etc. Simplification passes like CVP and
-  // GVN, loop transforms, and others have already run, so it's now better to
-  // convert to more optimized IR using more aggressive simplify CFG options.
-  // The extra sinking transform can create larger basic blocks, so do this
-  // before SLP vectorization.
-  PM.add(createCFGSimplificationPass(SimplifyCFGOptions()
-                                         .forwardSwitchCondToPhi(true)
-                                         .convertSwitchRangeToICmp(true)
-                                         .convertSwitchToLookupTable(true)
-                                         .needCanonicalLoops(false)
-                                         .hoistCommonInsts(true)
-                                         .sinkCommonInsts(true)));
-
-  if (IsFullLTO) {
-    PM.add(createInstructionCombiningPass()); // Clean up again
-    PM.add(createBitTrackingDCEPass());
-  }
-
-  // Optimize parallel scalar instruction chains into SIMD instructions.
-  if (SLPVectorize) {
-    PM.add(createSLPVectorizerPass());
-  }
-
-  if (!IsFullLTO) {
-    PM.add(createInstructionCombiningPass());
-
-    // Unroll small loops
-    PM.add(createLoopUnrollPass(OptLevel, DisableUnrollLoops,
-                                ForgetAllSCEVInLoopUnroll));
-
-    if (!DisableUnrollLoops) {
-      // LoopUnroll may generate some redundency to cleanup.
-      PM.add(createInstructionCombiningPass());
-
-      // Runtime unrolling will introduce runtime check in loop prologue. If the
-      // unrolled loop is a inner loop, then the prologue will be inside the
-      // outer loop. LICM pass can help to promote the runtime check out if the
-      // checked value is loop invariant.
-      PM.add(createLICMPass(LicmMssaOptCap, LicmMssaNoAccForPromotionCap,
-                            /*AllowSpeculation=*/true));
-    }
-  }
-
-  // After vectorization and unrolling, assume intrinsics may tell us more
-  // about pointer alignments.
-  PM.add(createAlignmentFromAssumptionsPass());
-
-  if (IsFullLTO)
-    PM.add(createInstructionCombiningPass());
-}
-
-void PassManagerBuilder::populateModulePassManager(
-    legacy::PassManagerBase &MPM) {
-  // If all optimizations are disabled, just run the always-inline pass and,
-  // if enabled, the function merging pass.
-  if (OptLevel == 0) {
-    if (Inliner) {
-      MPM.add(Inliner);
-      Inliner = nullptr;
-    }
-
-    return;
-  }
-
-  // Add LibraryInfo if we have some.
-  if (LibraryInfo)
-    MPM.add(new TargetLibraryInfoWrapperPass(*LibraryInfo));
-
-  addInitialAliasAnalysisPasses(MPM);
-
-  if (OptLevel > 2)
-    MPM.add(createCallSiteSplittingPass());
-
-  // Promote any localized global vars.
-  MPM.add(createPromoteMemoryToRegisterPass());
-
-  MPM.add(createDeadArgEliminationPass()); // Dead argument elimination
-
-  MPM.add(createInstructionCombiningPass()); // Clean up after IPCP & DAE
-  MPM.add(
-      createCFGSimplificationPass(SimplifyCFGOptions().convertSwitchRangeToICmp(
-          true))); // Clean up after IPCP & DAE
-
-  // We add a module alias analysis pass here. In part due to bugs in the
-  // analysis infrastructure this "works" in that the analysis stays alive
-  // for the entire SCC pass run below.
-  MPM.add(createGlobalsAAWrapperPass());
-
-  // Start of CallGraph SCC passes.
-  if (Inliner) {
-    MPM.add(Inliner);
-    Inliner = nullptr;
-  }
-
-  addFunctionSimplificationPasses(MPM);
-
-  // FIXME: This is a HACK! The inliner pass above implicitly creates a CGSCC
-  // pass manager that we are specifically trying to avoid. To prevent this
-  // we must insert a no-op module pass to reset the pass manager.
-  MPM.add(createBarrierNoopPass());
-
-  // We add a fresh GlobalsModRef run at this point. This is particularly
-  // useful as the above will have inlined, DCE'ed, and function-attr
-  // propagated everything. We should at this point have a reasonably minimal
-  // and richly annotated call graph. By computing aliasing and mod/ref
-  // information for all local globals here, the late loop passes and notably
-  // the vectorizer will be able to use them to help recognize vectorizable
-  // memory operations.
-  //
-  // Note that this relies on a bug in the pass manager which preserves
-  // a module analysis into a function pass pipeline (and throughout it) so
-  // long as the first function pass doesn't invalidate the module analysis.
-  // Thus both Float2Int and LoopRotate have to preserve AliasAnalysis for
-  // this to work. Fortunately, it is trivial to preserve AliasAnalysis
-  // (doing nothing preserves it as it is required to be conservatively
-  // correct in the face of IR changes).
-  MPM.add(createGlobalsAAWrapperPass());
-
-  MPM.add(createFloat2IntPass());
-  MPM.add(createLowerConstantIntrinsicsPass());
-
-  // Re-rotate loops in all our loop nests. These may have fallout out of
-  // rotated form due to GVN or other transformations, and the vectorizer relies
-  // on the rotated form. Disable header duplication at -Oz.
-  MPM.add(createLoopRotatePass(SizeLevel == 2 ? 0 : -1, false));
-
-  addVectorPasses(MPM, /* IsFullLTO */ false);
-
-  // LoopSink pass sinks instructions hoisted by LICM, which serves as a
-  // canonicalization pass that enables other optimizations. As a result,
-  // LoopSink pass needs to be a very late IR pass to avoid undoing LICM
-  // result too early.
-  MPM.add(createLoopSinkPass());
-  // Get rid of LCSSA nodes.
-  MPM.add(createInstSimplifyLegacyPass());
-
-  // LoopSink (and other loop passes since the last simplifyCFG) might have
-  // resulted in single-entry-single-exit or empty blocks. Clean up the CFG.
-  MPM.add(createCFGSimplificationPass(
-      SimplifyCFGOptions().convertSwitchRangeToICmp(true)));
-}
-
-LLVMPassManagerBuilderRef LLVMPassManagerBuilderCreate() {
-  PassManagerBuilder *PMB = new PassManagerBuilder();
-  return wrap(PMB);
-}
-
-void LLVMPassManagerBuilderDispose(LLVMPassManagerBuilderRef PMB) {
-  PassManagerBuilder *Builder = unwrap(PMB);
-  delete Builder;
-}
-
-void
-LLVMPassManagerBuilderSetOptLevel(LLVMPassManagerBuilderRef PMB,
-                                  unsigned OptLevel) {
-  PassManagerBuilder *Builder = unwrap(PMB);
-  Builder->OptLevel = OptLevel;
-}
-
-void
-LLVMPassManagerBuilderSetSizeLevel(LLVMPassManagerBuilderRef PMB,
-                                   unsigned SizeLevel) {
-  PassManagerBuilder *Builder = unwrap(PMB);
-  Builder->SizeLevel = SizeLevel;
-}
-
-void
-LLVMPassManagerBuilderSetDisableUnitAtATime(LLVMPassManagerBuilderRef PMB,
-                                            LLVMBool Value) {
-  // NOTE: The DisableUnitAtATime switch has been removed.
-}
-
-void
-LLVMPassManagerBuilderSetDisableUnrollLoops(LLVMPassManagerBuilderRef PMB,
-                                            LLVMBool Value) {
-  PassManagerBuilder *Builder = unwrap(PMB);
-  Builder->DisableUnrollLoops = Value;
-}
-
-void
-LLVMPassManagerBuilderSetDisableSimplifyLibCalls(LLVMPassManagerBuilderRef PMB,
-                                                 LLVMBool Value) {
-  // NOTE: The simplify-libcalls pass has been removed.
-}
-
-void
-LLVMPassManagerBuilderUseInlinerWithThreshold(LLVMPassManagerBuilderRef PMB,
-                                              unsigned Threshold) {
-  // TODO: remove this
-}
-
-void
-LLVMPassManagerBuilderPopulateFunctionPassManager(LLVMPassManagerBuilderRef PMB,
-                                                  LLVMPassManagerRef PM) {
-  PassManagerBuilder *Builder = unwrap(PMB);
-  legacy::FunctionPassManager *FPM = unwrap<legacy::FunctionPassManager>(PM);
-  Builder->populateFunctionPassManager(*FPM);
-}
-
-void
-LLVMPassManagerBuilderPopulateModulePassManager(LLVMPassManagerBuilderRef PMB,
-                                                LLVMPassManagerRef PM) {
-  PassManagerBuilder *Builder = unwrap(PMB);
-  legacy::PassManagerBase *MPM = unwrap(PM);
-  Builder->populateModulePassManager(*MPM);
-}
diff --git a/llvm/lib/Transforms/IPO/SCCP.cpp b/llvm/lib/Transforms/IPO/SCCP.cpp
index c736c298c7dae..51114d184881b 100644
--- a/llvm/lib/Transforms/IPO/SCCP.cpp
+++ b/llvm/lib/Transforms/IPO/SCCP.cpp
@@ -42,8 +42,8 @@ STATISTIC(NumDeadBlocks , "Number of basic blocks unreachable");
 STATISTIC(NumInstReplaced,
           "Number of instructions replaced with (simpler) instruction");
 
-static cl::opt<unsigned> FuncSpecializationMaxIters(
-    "func-specialization-max-iters", cl::init(1), cl::Hidden, cl::desc(
+static cl::opt<unsigned> FuncSpecMaxIters(
+    "funcspec-max-iters", cl::init(1), cl::Hidden, cl::desc(
     "The maximum number of iterations function specialization is run"));
 
 static void findReturnsToZap(Function &F,
@@ -158,7 +158,7 @@ static bool runIPSCCP(
 
   if (IsFuncSpecEnabled) {
     unsigned Iters = 0;
-    while (Iters++ < FuncSpecializationMaxIters && Specializer.run());
+    while (Iters++ < FuncSpecMaxIters && Specializer.run());
   }
 
   // Iterate over all of the instructions in the module, replacing them with
@@ -292,13 +292,6 @@ static bool runIPSCCP(
         if (!CB || CB->getCalledFunction() != F)
           continue;
 
-        // Limit to cases where the return value is guaranteed to be neither
-        // poison nor undef. Poison will be outside any range and currently
-        // values outside of the specified range cause immediate undefined
-        // behavior.
-        if (!isGuaranteedNotToBeUndefOrPoison(CB, nullptr, CB))
-          continue;
-
         // Do not touch existing metadata for now.
         // TODO: We should be able to take the intersection of the existing
         // metadata and the inferred range.
diff --git a/llvm/lib/Transforms/IPO/SampleProfileProbe.cpp b/llvm/lib/Transforms/IPO/SampleProfileProbe.cpp
index a819bd9fb6e03..7a40ddee81798 100644
--- a/llvm/lib/Transforms/IPO/SampleProfileProbe.cpp
+++ b/llvm/lib/Transforms/IPO/SampleProfileProbe.cpp
@@ -56,11 +56,7 @@ static uint64_t getCallStackHash(const DILocation *DIL) {
   while (InlinedAt) {
     Hash ^= MD5Hash(std::to_string(InlinedAt->getLine()));
     Hash ^= MD5Hash(std::to_string(InlinedAt->getColumn()));
-    const DISubprogram *SP = InlinedAt->getScope()->getSubprogram();
-    // Use linkage name for C++ if possible.
-    auto Name = SP->getLinkageName();
-    if (Name.empty())
-      Name = SP->getName();
+    auto Name = InlinedAt->getSubprogramLinkageName();
     Hash ^= MD5Hash(Name);
     InlinedAt = InlinedAt->getInlinedAt();
   }
diff --git a/llvm/lib/Transforms/IPO/StripSymbols.cpp b/llvm/lib/Transforms/IPO/StripSymbols.cpp
index 03bc4f3139600..5ad71187b391f 100644
--- a/llvm/lib/Transforms/IPO/StripSymbols.cpp
+++ b/llvm/lib/Transforms/IPO/StripSymbols.cpp
@@ -265,23 +265,31 @@ static bool stripDeadDebugInfoImpl(Module &M) {
 PreservedAnalyses StripSymbolsPass::run(Module &M, ModuleAnalysisManager &AM) {
   StripDebugInfo(M);
   StripSymbolNames(M, false);
-  return PreservedAnalyses::all();
+  PreservedAnalyses PA;
+  PA.preserveSet<CFGAnalyses>();
+  return PA;
 }
 
 PreservedAnalyses StripNonDebugSymbolsPass::run(Module &M,
                                                 ModuleAnalysisManager &AM) {
   StripSymbolNames(M, true);
-  return PreservedAnalyses::all();
+  PreservedAnalyses PA;
+  PA.preserveSet<CFGAnalyses>();
+  return PA;
 }
 
 PreservedAnalyses StripDebugDeclarePass::run(Module &M,
                                              ModuleAnalysisManager &AM) {
   stripDebugDeclareImpl(M);
-  return PreservedAnalyses::all();
+  PreservedAnalyses PA;
+  PA.preserveSet<CFGAnalyses>();
+  return PA;
 }
 
 PreservedAnalyses StripDeadDebugInfoPass::run(Module &M,
                                               ModuleAnalysisManager &AM) {
   stripDeadDebugInfoImpl(M);
-  return PreservedAnalyses::all();
+  PreservedAnalyses PA;
+  PA.preserveSet<CFGAnalyses>();
+  return PA;
 }
diff --git a/llvm/lib/Transforms/IPO/WholeProgramDevirt.cpp b/llvm/lib/Transforms/IPO/WholeProgramDevirt.cpp
index d0ef84b724aed..e380b47c735fe 100644
--- a/llvm/lib/Transforms/IPO/WholeProgramDevirt.cpp
+++ b/llvm/lib/Transforms/IPO/WholeProgramDevirt.cpp
@@ -764,7 +764,7 @@ PreservedAnalyses WholeProgramDevirtPass::run(Module &M,
     return FAM.getResult<DominatorTreeAnalysis>(F);
   };
   if (UseCommandLine) {
-    if (DevirtModule::runForTesting(M, AARGetter, OREGetter, LookupDomTree))
+    if (!DevirtModule::runForTesting(M, AARGetter, OREGetter, LookupDomTree))
       return PreservedAnalyses::all();
     return PreservedAnalyses::none();
   }
@@ -895,8 +895,7 @@ static Error checkCombinedSummaryForTesting(ModuleSummaryIndex *Summary) {
   // DevirtIndex::run, not to DevirtModule::run used by opt/runForTesting.
   const auto &ModPaths = Summary->modulePaths();
   if (ClSummaryAction != PassSummaryAction::Import &&
-      ModPaths.find(ModuleSummaryIndex::getRegularLTOModuleName()) ==
-          ModPaths.end())
+      !ModPaths.contains(ModuleSummaryIndex::getRegularLTOModuleName()))
     return createStringError(
         errc::invalid_argument,
         "combined summary should contain Regular LTO module");
diff --git a/llvm/lib/Transforms/InstCombine/InstCombineAddSub.cpp b/llvm/lib/Transforms/InstCombine/InstCombineAddSub.cpp
index 0d8fa5c5a2c5d..2c2b767b2ced1 100644
--- a/llvm/lib/Transforms/InstCombine/InstCombineAddSub.cpp
+++ b/llvm/lib/Transforms/InstCombine/InstCombineAddSub.cpp
@@ -2406,6 +2406,18 @@ Instruction *InstCombinerImpl::visitSub(BinaryOperator &I) {
     return replaceInstUsesWith(I, Mul);
   }
 
+  // max(X,Y) nsw/nuw - min(X,Y) --> abs(X nsw - Y)
+  if (match(Op0, m_OneUse(m_c_SMax(m_Value(X), m_Value(Y)))) &&
+      match(Op1, m_OneUse(m_c_SMin(m_Specific(X), m_Specific(Y))))) {
+    if (I.hasNoUnsignedWrap() || I.hasNoSignedWrap()) {
+      Value *Sub =
+          Builder.CreateSub(X, Y, "sub", /*HasNUW=*/false, /*HasNSW=*/true);
+      Value *Call =
+          Builder.CreateBinaryIntrinsic(Intrinsic::abs, Sub, Builder.getTrue());
+      return replaceInstUsesWith(I, Call);
+    }
+  }
+
   return TryToNarrowDeduceFlags();
 }
 
diff --git a/llvm/lib/Transforms/InstCombine/InstCombineAndOrXor.cpp b/llvm/lib/Transforms/InstCombine/InstCombineAndOrXor.cpp
index 0a9da9220731f..b9cc104f13fda 100644
--- a/llvm/lib/Transforms/InstCombine/InstCombineAndOrXor.cpp
+++ b/llvm/lib/Transforms/InstCombine/InstCombineAndOrXor.cpp
@@ -1273,161 +1273,6 @@ static Value *matchIsFiniteTest(InstCombiner::BuilderTy &Builder, FCmpInst *LHS,
   return Builder.CreateFCmp(FCmpInst::getOrderedPredicate(PredR), RHS0, RHS1);
 }
 
-/// Returns a pair of values, which if passed to llvm.is.fpclass, returns the
-/// same result as an fcmp with the given operands.
-static std::pair<Value *, unsigned> fcmpToClassTest(FCmpInst::Predicate Pred,
-                                                    Value *LHS, Value *RHS) {
-  const APFloat *ConstRHS;
-  if (!match(RHS, m_APFloat(ConstRHS)))
-    return {nullptr, 0};
-
-  if (ConstRHS->isZero()) {
-    switch (Pred) {
-    // TODO: Compares eq/ne with 0 depends on the denormal handling mode.
-    case FCmpInst::FCMP_ORD:
-      // Canonical form of ord/uno is with a zero. We could also handle
-      // non-canonical other non-NaN constants or LHS == RHS.
-      return {LHS, ~fcNan & fcAllFlags};
-    case FCmpInst::FCMP_UNO:
-      return {LHS, fcNan};
-    default:
-      break;
-    }
-
-    return {nullptr, 0};
-  }
-
-  Value *Src = LHS;
-  const bool IsFabs = match(LHS, m_FAbs(m_Value(Src)));
-
-  // Compute the test mask that would return true for the ordered comparisons.
-  unsigned Mask;
-
-  if (ConstRHS->isInfinity()) {
-    switch (Pred) {
-    case FCmpInst::FCMP_OEQ:
-    case FCmpInst::FCMP_UNE: {
-      // Match __builtin_isinf patterns
-      //
-      //   fcmp oeq x, +inf -> is_fpclass x, fcPosInf
-      //   fcmp oeq fabs(x), +inf -> is_fpclass x, fcInf
-      //   fcmp oeq x, -inf -> is_fpclass x, fcNegInf
-      //   fcmp oeq fabs(x), -inf -> is_fpclass x, 0 -> false
-      //
-      //   fcmp une x, +inf -> is_fpclass x, ~fcPosInf
-      //   fcmp une fabs(x), +inf -> is_fpclass x, ~fcInf
-      //   fcmp une x, -inf -> is_fpclass x, ~fcNegInf
-      //   fcmp une fabs(x), -inf -> is_fpclass x, fcAllFlags -> true
-
-      if (ConstRHS->isNegative()) {
-        Mask = fcNegInf;
-        if (IsFabs)
-          Mask = 0;
-      } else {
-        Mask = fcPosInf;
-        if (IsFabs)
-          Mask |= fcNegInf;
-      }
-
-      break;
-    }
-    case FCmpInst::FCMP_ONE:
-    case FCmpInst::FCMP_UEQ: {
-      // Match __builtin_isinf patterns
-      //   fcmp one x, -inf -> is_fpclass x, fcNegInf
-      //   fcmp one fabs(x), -inf -> is_fpclass x, ~fcNegInf & ~fcNan
-      //   fcmp one x, +inf -> is_fpclass x, ~fcNegInf & ~fcNan
-      //   fcmp one fabs(x), +inf -> is_fpclass x, ~fcInf & fcNan
-      //
-      //   fcmp ueq x, +inf -> is_fpclass x, fcPosInf|fcNan
-      //   fcmp ueq (fabs x), +inf -> is_fpclass x, fcInf|fcNan
-      //   fcmp ueq x, -inf -> is_fpclass x, fcNegInf|fcNan
-      //   fcmp ueq fabs(x), -inf -> is_fpclass x, fcNan
-      if (ConstRHS->isNegative()) {
-        Mask = ~fcNegInf & ~fcNan;
-        if (IsFabs)
-          Mask = ~fcNan;
-      } else {
-        Mask = ~fcPosInf & ~fcNan;
-        if (IsFabs)
-          Mask &= ~fcNegInf;
-      }
-
-      Mask &= fcAllFlags;
-      break;
-    }
-    case FCmpInst::FCMP_OLT:
-    case FCmpInst::FCMP_UGE: {
-      if (ConstRHS->isNegative()) // TODO
-        return {nullptr, 0};
-
-      // fcmp olt fabs(x), +inf -> fcFinite
-      // fcmp uge fabs(x), +inf -> ~fcFinite
-      // fcmp olt x, +inf -> fcFinite|fcNegInf
-      // fcmp uge x, +inf -> ~(fcFinite|fcNegInf)
-      Mask = fcFinite;
-      if (!IsFabs)
-        Mask |= fcNegInf;
-      break;
-    }
-    case FCmpInst::FCMP_OGE:
-    case FCmpInst::FCMP_ULT: {
-      if (ConstRHS->isNegative()) // TODO
-        return {nullptr, 0};
-
-      // fcmp oge fabs(x), +inf -> fcInf
-      // fcmp oge x, +inf -> fcPosInf
-      // fcmp ult fabs(x), +inf -> ~fcInf
-      // fcmp ult x, +inf -> ~fcPosInf
-      Mask = fcPosInf;
-      if (IsFabs)
-        Mask |= fcNegInf;
-      break;
-    }
-    default:
-      return {nullptr, 0};
-    }
-  } else if (ConstRHS->isSmallestNormalized() && !ConstRHS->isNegative()) {
-    // Match pattern that's used in __builtin_isnormal.
-    switch (Pred) {
-    case FCmpInst::FCMP_OLT:
-    case FCmpInst::FCMP_UGE: {
-      // fcmp olt x, smallest_normal -> fcNegInf|fcNegNormal|fcSubnormal|fcZero
-      // fcmp olt fabs(x), smallest_normal -> fcSubnormal|fcZero
-      // fcmp uge x, smallest_normal -> fcNan|fcPosNormal|fcPosInf
-      // fcmp uge fabs(x), smallest_normal -> ~(fcSubnormal|fcZero)
-      Mask = fcZero | fcSubnormal;
-      if (!IsFabs)
-        Mask |= fcNegNormal | fcNegInf;
-
-      break;
-    }
-    case FCmpInst::FCMP_OGE:
-    case FCmpInst::FCMP_ULT: {
-      // fcmp oge x, smallest_normal -> fcPosNormal | fcPosInf
-      // fcmp oge fabs(x), smallest_normal -> fcInf | fcNormal
-      // fcmp ult x, smallest_normal -> ~(fcPosNormal | fcPosInf)
-      // fcmp ult fabs(x), smallest_normal -> ~(fcInf | fcNormal)
-      Mask = fcPosInf | fcPosNormal;
-      if (IsFabs)
-        Mask |= fcNegInf | fcNegNormal;
-      break;
-    }
-    default:
-      return {nullptr, 0};
-    }
-  } else
-    return {nullptr, 0};
-
-  // Invert the comparison for the unordered cases.
-  if (FCmpInst::isUnordered(Pred))
-    Mask = ~Mask & fcAllFlags;
-
-  assert((Mask & ~fcAllFlags) == 0);
-
-  return {Src, Mask};
-}
-
 Value *InstCombinerImpl::foldLogicOfFCmps(FCmpInst *LHS, FCmpInst *RHS,
                                           bool IsAnd, bool IsLogicalSelect) {
   Value *LHS0 = LHS->getOperand(0), *LHS1 = LHS->getOperand(1);
@@ -1501,9 +1346,11 @@ Value *InstCombinerImpl::foldLogicOfFCmps(FCmpInst *LHS, FCmpInst *RHS,
   // potentially eliminate 4-6 instructions. If we can represent a test with a
   // single fcmp with fneg and fabs, that's likely a better canonical form.
   if (LHS->hasOneUse() && RHS->hasOneUse()) {
-    auto [ClassValRHS, ClassMaskRHS] = fcmpToClassTest(PredR, RHS0, RHS1);
+    auto [ClassValRHS, ClassMaskRHS] =
+        fcmpToClassTest(PredR, *RHS->getFunction(), RHS0, RHS1);
     if (ClassValRHS) {
-      auto [ClassValLHS, ClassMaskLHS] = fcmpToClassTest(PredL, LHS0, LHS1);
+      auto [ClassValLHS, ClassMaskLHS] =
+          fcmpToClassTest(PredL, *LHS->getFunction(), LHS0, LHS1);
       if (ClassValLHS == ClassValRHS) {
         unsigned CombinedMask = IsAnd ? (ClassMaskLHS & ClassMaskRHS)
                                       : (ClassMaskLHS | ClassMaskRHS);
@@ -1525,8 +1372,9 @@ static bool matchIsFPClassLikeFCmp(Value *Op, Value *&ClassVal,
   if (!FCmp || !FCmp->hasOneUse())
     return false;
 
-  std::tie(ClassVal, ClassMask) = fcmpToClassTest(
-      FCmp->getPredicate(), FCmp->getOperand(0), FCmp->getOperand(1));
+  std::tie(ClassVal, ClassMask) =
+      fcmpToClassTest(FCmp->getPredicate(), *FCmp->getParent()->getParent(),
+                      FCmp->getOperand(0), FCmp->getOperand(1));
   return ClassVal != nullptr;
 }
 
diff --git a/llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp b/llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp
index 916476a2d912e..2b61b58dbc36a 100644
--- a/llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp
+++ b/llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp
@@ -439,9 +439,7 @@ Instruction *InstCombinerImpl::simplifyMaskedScatter(IntrinsicInst &II) {
       Align Alignment = cast<ConstantInt>(II.getArgOperand(2))->getAlignValue();
       VectorType *WideLoadTy = cast<VectorType>(II.getArgOperand(1)->getType());
       ElementCount VF = WideLoadTy->getElementCount();
-      Constant *EC =
-          ConstantInt::get(Builder.getInt32Ty(), VF.getKnownMinValue());
-      Value *RunTimeVF = VF.isScalable() ? Builder.CreateVScale(EC) : EC;
+      Value *RunTimeVF = Builder.CreateElementCount(Builder.getInt32Ty(), VF);
       Value *LastLane = Builder.CreateSub(RunTimeVF, Builder.getInt32(1));
       Value *Extract =
           Builder.CreateExtractElement(II.getArgOperand(0), LastLane);
@@ -599,8 +597,7 @@ static Instruction *foldCttzCtlz(IntrinsicInst &II, InstCombinerImpl &IC) {
   }
 
   // Add range metadata since known bits can't completely reflect what we know.
-  // TODO: Handle splat vectors.
-  auto *IT = dyn_cast<IntegerType>(Op0->getType());
+  auto *IT = cast<IntegerType>(Op0->getType()->getScalarType());
   if (IT && IT->getBitWidth() != 1 && !II.getMetadata(LLVMContext::MD_range)) {
     Metadata *LowAndHigh[] = {
         ConstantAsMetadata::get(ConstantInt::get(IT, DefiniteZeros)),
@@ -683,12 +680,8 @@ static Instruction *foldCtpop(IntrinsicInst &II, InstCombinerImpl &IC) {
                                                   Constant::getNullValue(Ty)),
                             Ty);
 
-  // FIXME: Try to simplify vectors of integers.
-  auto *IT = dyn_cast<IntegerType>(Ty);
-  if (!IT)
-    return nullptr;
-
   // Add range metadata since known bits can't completely reflect what we know.
+  auto *IT = cast<IntegerType>(Ty->getScalarType());
   unsigned MinCount = Known.countMinPopulation();
   unsigned MaxCount = Known.countMaxPopulation();
   if (IT->getBitWidth() != 1 && !II.getMetadata(LLVMContext::MD_range)) {
@@ -830,11 +823,33 @@ InstCombinerImpl::foldIntrinsicWithOverflowCommon(IntrinsicInst *II) {
   return nullptr;
 }
 
+/// \returns true if the test performed by llvm.is.fpclass(x, \p Mask) is
+/// equivalent to fcmp oeq x, 0.0 with the floating-point environment assumed
+/// for \p F for type \p Ty
+static bool fpclassTestIsFCmp0(FPClassTest Mask, const Function &F, Type *Ty) {
+  if (Mask == fcZero)
+    return F.getDenormalMode(Ty->getScalarType()->getFltSemantics()).Input ==
+           DenormalMode::IEEE;
+
+  if (Mask == (fcZero | fcSubnormal)) {
+    DenormalMode::DenormalModeKind InputMode =
+        F.getDenormalMode(Ty->getScalarType()->getFltSemantics()).Input;
+    return InputMode == DenormalMode::PreserveSign ||
+           InputMode == DenormalMode::PositiveZero;
+  }
+
+  return false;
+}
+
 Instruction *InstCombinerImpl::foldIntrinsicIsFPClass(IntrinsicInst &II) {
   Value *Src0 = II.getArgOperand(0);
   Value *Src1 = II.getArgOperand(1);
   const ConstantInt *CMask = cast<ConstantInt>(Src1);
   FPClassTest Mask = static_cast<FPClassTest>(CMask->getZExtValue());
+  const bool IsUnordered = (Mask & fcNan) == fcNan;
+  const bool IsOrdered = (Mask & fcNan) == fcNone;
+  const FPClassTest OrderedMask = Mask & ~fcNan;
+  const FPClassTest OrderedInvertedMask = ~OrderedMask & ~fcNan;
 
   const bool IsStrict = II.isStrictFP();
 
@@ -852,6 +867,37 @@ Instruction *InstCombinerImpl::foldIntrinsicIsFPClass(IntrinsicInst &II) {
     return replaceOperand(II, 0, FAbsSrc);
   }
 
+  // TODO: is.fpclass(x, fcInf) -> fabs(x) == inf
+
+  if ((OrderedMask == fcPosInf || OrderedMask == fcNegInf) &&
+      (IsOrdered || IsUnordered) && !IsStrict) {
+    // is.fpclass(x, fcPosInf) -> fcmp oeq x, +inf
+    // is.fpclass(x, fcNegInf) -> fcmp oeq x, -inf
+    // is.fpclass(x, fcPosInf|fcNan) -> fcmp ueq x, +inf
+    // is.fpclass(x, fcNegInf|fcNan) -> fcmp ueq x, -inf
+    Constant *Inf =
+        ConstantFP::getInfinity(Src0->getType(), OrderedMask == fcNegInf);
+    Value *EqInf = IsUnordered ? Builder.CreateFCmpUEQ(Src0, Inf)
+                               : Builder.CreateFCmpOEQ(Src0, Inf);
+
+    EqInf->takeName(&II);
+    return replaceInstUsesWith(II, EqInf);
+  }
+
+  if ((OrderedInvertedMask == fcPosInf || OrderedInvertedMask == fcNegInf) &&
+      (IsOrdered || IsUnordered) && !IsStrict) {
+    // is.fpclass(x, ~fcPosInf) -> fcmp one x, +inf
+    // is.fpclass(x, ~fcNegInf) -> fcmp one x, -inf
+    // is.fpclass(x, ~fcPosInf|fcNan) -> fcmp une x, +inf
+    // is.fpclass(x, ~fcNegInf|fcNan) -> fcmp une x, -inf
+    Constant *Inf = ConstantFP::getInfinity(Src0->getType(),
+                                            OrderedInvertedMask == fcNegInf);
+    Value *NeInf = IsUnordered ? Builder.CreateFCmpUNE(Src0, Inf)
+                               : Builder.CreateFCmpONE(Src0, Inf);
+    NeInf->takeName(&II);
+    return replaceInstUsesWith(II, NeInf);
+  }
+
   if (Mask == fcNan && !IsStrict) {
     // Equivalent of isnan. Replace with standard fcmp if we don't care about FP
     // exceptions.
@@ -869,6 +915,29 @@ Instruction *InstCombinerImpl::foldIntrinsicIsFPClass(IntrinsicInst &II) {
     return replaceInstUsesWith(II, FCmp);
   }
 
+  if (!IsStrict && (IsOrdered || IsUnordered) &&
+      fpclassTestIsFCmp0(OrderedMask, *II.getFunction(), Src0->getType())) {
+    Constant *Zero = ConstantFP::getZero(Src0->getType());
+    // Equivalent of == 0.
+    Value *FCmp = IsUnordered ? Builder.CreateFCmpUEQ(Src0, Zero)
+                              : Builder.CreateFCmpOEQ(Src0, Zero);
+    FCmp->takeName(&II);
+    return replaceInstUsesWith(II, FCmp);
+  }
+
+  if (!IsStrict && (IsOrdered || IsUnordered) &&
+      fpclassTestIsFCmp0(OrderedInvertedMask, *II.getFunction(),
+                         Src0->getType())) {
+    Constant *Zero = ConstantFP::getZero(Src0->getType());
+
+    // Equivalent of !(x == 0).
+    Value *FCmp = IsUnordered ? Builder.CreateFCmpUNE(Src0, Zero)
+                              : Builder.CreateFCmpONE(Src0, Zero);
+
+    FCmp->takeName(&II);
+    return replaceInstUsesWith(II, FCmp);
+  }
+
   // fp_class (nnan x), qnan|snan|other -> fp_class (nnan x), other
   if ((Mask & fcNan) && isKnownNeverNaN(Src0, &getTargetLibraryInfo())) {
     II.setArgOperand(1, ConstantInt::get(Src1->getType(), Mask & ~fcNan));
diff --git a/llvm/lib/Transforms/InstCombine/InstCombineCasts.cpp b/llvm/lib/Transforms/InstCombine/InstCombineCasts.cpp
index f399810ead2f6..2130177e8cf87 100644
--- a/llvm/lib/Transforms/InstCombine/InstCombineCasts.cpp
+++ b/llvm/lib/Transforms/InstCombine/InstCombineCasts.cpp
@@ -918,11 +918,18 @@ Instruction *InstCombinerImpl::visitTrunc(TruncInst &Trunc) {
     // removed by the trunc.
     if (match(C, m_SpecificInt_ICMP(ICmpInst::ICMP_ULE,
                                     APInt(SrcWidth, MaxShiftAmt)))) {
+      auto GetNewShAmt = [&](unsigned Width) {
+        Constant *MaxAmt = ConstantInt::get(SrcTy, Width - 1, false);
+        Constant *Cmp =
+            ConstantFoldCompareInstOperands(ICmpInst::ICMP_ULT, C, MaxAmt, DL);
+        Constant *ShAmt = ConstantFoldSelectInstruction(Cmp, C, MaxAmt);
+        return ConstantFoldCastOperand(Instruction::Trunc, ShAmt, A->getType(),
+                                       DL);
+      };
+
       // trunc (lshr (sext A), C) --> ashr A, C
       if (A->getType() == DestTy) {
-        Constant *MaxAmt = ConstantInt::get(SrcTy, DestWidth - 1, false);
-        Constant *ShAmt = ConstantExpr::getUMin(C, MaxAmt);
-        ShAmt = ConstantExpr::getTrunc(ShAmt, A->getType());
+        Constant *ShAmt = GetNewShAmt(DestWidth);
         ShAmt = Constant::mergeUndefsWith(ShAmt, C);
         return IsExact ? BinaryOperator::CreateExactAShr(A, ShAmt)
                        : BinaryOperator::CreateAShr(A, ShAmt);
@@ -930,9 +937,7 @@ Instruction *InstCombinerImpl::visitTrunc(TruncInst &Trunc) {
       // The types are mismatched, so create a cast after shifting:
       // trunc (lshr (sext A), C) --> sext/trunc (ashr A, C)
       if (Src->hasOneUse()) {
-        Constant *MaxAmt = ConstantInt::get(SrcTy, AWidth - 1, false);
-        Constant *ShAmt = ConstantExpr::getUMin(C, MaxAmt);
-        ShAmt = ConstantExpr::getTrunc(ShAmt, A->getType());
+        Constant *ShAmt = GetNewShAmt(AWidth);
         Value *Shift = Builder.CreateAShr(A, ShAmt, "", IsExact);
         return CastInst::CreateIntegerCast(Shift, DestTy, true);
       }
diff --git a/llvm/lib/Transforms/InstCombine/InstCombineCompares.cpp b/llvm/lib/Transforms/InstCombine/InstCombineCompares.cpp
index 1616c0e90ecc0..b1d59d355eb73 100644
--- a/llvm/lib/Transforms/InstCombine/InstCombineCompares.cpp
+++ b/llvm/lib/Transforms/InstCombine/InstCombineCompares.cpp
@@ -563,7 +563,7 @@ static Value *rewriteGEPAsOffset(Type *ElemTy, Value *Start, Value *Base,
   // Create all the other instructions.
   for (Value *Val : Explored) {
 
-    if (NewInsts.find(Val) != NewInsts.end())
+    if (NewInsts.contains(Val))
       continue;
 
     if (auto *CI = dyn_cast<CastInst>(Val)) {
@@ -611,7 +611,7 @@ static Value *rewriteGEPAsOffset(Type *ElemTy, Value *Start, Value *Base,
       for (unsigned I = 0, E = PHI->getNumIncomingValues(); I < E; ++I) {
         Value *NewIncoming = PHI->getIncomingValue(I);
 
-        if (NewInsts.find(NewIncoming) != NewInsts.end())
+        if (NewInsts.contains(NewIncoming))
           NewIncoming = NewInsts[NewIncoming];
 
         NewPhi->addIncoming(NewIncoming, PHI->getIncomingBlock(I));
@@ -3239,16 +3239,6 @@ Instruction *InstCombinerImpl::foldICmpBinOpEqualityWithConstant(
     }
     break;
   }
-  case Instruction::And: {
-    const APInt *BOC;
-    if (match(BOp1, m_APInt(BOC))) {
-      // If we have ((X & C) == C), turn it into ((X & C) != 0).
-      if (C == *BOC && C.isPowerOf2())
-        return new ICmpInst(isICMP_NE ? ICmpInst::ICMP_EQ : ICmpInst::ICMP_NE,
-                            BO, Constant::getNullValue(RHS->getType()));
-    }
-    break;
-  }
   case Instruction::UDiv:
     if (C.isZero()) {
       // (icmp eq/ne (udiv A, B), 0) -> (icmp ugt/ule i32 B, A)
@@ -3282,6 +3272,11 @@ Instruction *InstCombinerImpl::foldICmpEqIntrinsicWithConstant(
     return new ICmpInst(Pred, II->getArgOperand(0),
                         ConstantInt::get(Ty, C.byteSwap()));
 
+  case Intrinsic::bitreverse:
+    // bitreverse(A) == C  ->  A == bitreverse(C)
+    return new ICmpInst(Pred, II->getArgOperand(0),
+                        ConstantInt::get(Ty, C.reverseBits()));
+
   case Intrinsic::ctlz:
   case Intrinsic::cttz: {
     // ctz(A) == bitwidth(A)  ->  A == 0 and likewise for !=
@@ -5766,6 +5761,12 @@ Instruction *InstCombinerImpl::foldICmpUsingKnownBits(ICmpInst &I) {
         }
       }
     }
+
+    // Op0 eq C_Pow2 -> Op0 ne 0 if Op0 is known to be C_Pow2 or zero.
+    if (Op1Known.isConstant() && Op1Known.getConstant().isPowerOf2() &&
+        (Op0Known & Op1Known) == Op0Known)
+      return new ICmpInst(CmpInst::getInversePredicate(Pred), Op0,
+                          ConstantInt::getNullValue(Op1->getType()));
     break;
   }
   case ICmpInst::ICMP_ULT: {
@@ -5863,8 +5864,7 @@ Instruction *InstCombinerImpl::foldICmpUsingBoolRange(ICmpInst &I) {
     return BinaryOperator::CreateOr(Builder.CreateIsNull(X), Y);
 
   const APInt *C;
-  if (match(I.getOperand(0),
-            m_OneUse(m_c_Add(m_ZExt(m_Value(X)), m_SExt(m_Value(Y))))) &&
+  if (match(I.getOperand(0), m_c_Add(m_ZExt(m_Value(X)), m_SExt(m_Value(Y)))) &&
       match(I.getOperand(1), m_APInt(C)) &&
       X->getType()->isIntOrIntVectorTy(1) &&
       Y->getType()->isIntOrIntVectorTy(1)) {
@@ -5880,38 +5880,40 @@ Instruction *InstCombinerImpl::foldICmpUsingBoolRange(ICmpInst &I) {
         (C->slt(MinusOne) && Pred == ICmpInst::ICMP_SGT))
       return replaceInstUsesWith(I, ConstantInt::getTrue(I.getType()));
 
-    APInt NewC = *C;
-    // canonicalize predicate to eq/ne
-    if ((*C == Zero && Pred == ICmpInst::ICMP_SLT) ||
-        (*C != Zero && *C != MinusOne && Pred == ICmpInst::ICMP_UGT)) {
-      // x s< 0 in [-1, 1] --> x == -1
-      // x u> 1(or any const !=0 !=-1) in [-1, 1] --> x == -1
-      NewC = MinusOne;
-      Pred = ICmpInst::ICMP_EQ;
-    } else if ((*C == MinusOne && Pred == ICmpInst::ICMP_SGT) ||
-               (*C != Zero && *C != One && Pred == ICmpInst::ICMP_ULT)) {
-      // x s> -1 in [-1, 1] --> x != -1
-      // x u< -1 in [-1, 1] --> x != -1
-      Pred = ICmpInst::ICMP_NE;
-    } else if (*C == Zero && Pred == ICmpInst::ICMP_SGT) {
-      // x s> 0 in [-1, 1] --> x == 1
-      NewC = One;
-      Pred = ICmpInst::ICMP_EQ;
-    } else if (*C == One && Pred == ICmpInst::ICMP_SLT) {
-      // x s< 1 in [-1, 1] --> x != 1
-      Pred = ICmpInst::ICMP_NE;
-    }
-
-    if (NewC == MinusOne) {
-      if (Pred == ICmpInst::ICMP_EQ)
-        return BinaryOperator::CreateAnd(Builder.CreateNot(X), Y);
-      if (Pred == ICmpInst::ICMP_NE)
-        return BinaryOperator::CreateOr(X, Builder.CreateNot(Y));
-    } else if (NewC == One) {
-      if (Pred == ICmpInst::ICMP_EQ)
-        return BinaryOperator::CreateAnd(X, Builder.CreateNot(Y));
-      if (Pred == ICmpInst::ICMP_NE)
-        return BinaryOperator::CreateOr(Builder.CreateNot(X), Y);
+    if (I.getOperand(0)->hasOneUse()) {
+      APInt NewC = *C;
+      // canonicalize predicate to eq/ne
+      if ((*C == Zero && Pred == ICmpInst::ICMP_SLT) ||
+          (*C != Zero && *C != MinusOne && Pred == ICmpInst::ICMP_UGT)) {
+        // x s< 0 in [-1, 1] --> x == -1
+        // x u> 1(or any const !=0 !=-1) in [-1, 1] --> x == -1
+        NewC = MinusOne;
+        Pred = ICmpInst::ICMP_EQ;
+      } else if ((*C == MinusOne && Pred == ICmpInst::ICMP_SGT) ||
+                 (*C != Zero && *C != One && Pred == ICmpInst::ICMP_ULT)) {
+        // x s> -1 in [-1, 1] --> x != -1
+        // x u< -1 in [-1, 1] --> x != -1
+        Pred = ICmpInst::ICMP_NE;
+      } else if (*C == Zero && Pred == ICmpInst::ICMP_SGT) {
+        // x s> 0 in [-1, 1] --> x == 1
+        NewC = One;
+        Pred = ICmpInst::ICMP_EQ;
+      } else if (*C == One && Pred == ICmpInst::ICMP_SLT) {
+        // x s< 1 in [-1, 1] --> x != 1
+        Pred = ICmpInst::ICMP_NE;
+      }
+
+      if (NewC == MinusOne) {
+        if (Pred == ICmpInst::ICMP_EQ)
+          return BinaryOperator::CreateAnd(Builder.CreateNot(X), Y);
+        if (Pred == ICmpInst::ICMP_NE)
+          return BinaryOperator::CreateOr(X, Builder.CreateNot(Y));
+      } else if (NewC == One) {
+        if (Pred == ICmpInst::ICMP_EQ)
+          return BinaryOperator::CreateAnd(X, Builder.CreateNot(Y));
+        if (Pred == ICmpInst::ICMP_NE)
+          return BinaryOperator::CreateOr(Builder.CreateNot(X), Y);
+      }
     }
   }
 
diff --git a/llvm/lib/Transforms/InstCombine/InstCombineMulDivRem.cpp b/llvm/lib/Transforms/InstCombine/InstCombineMulDivRem.cpp
index 57e90076e1874..5768f71265ccd 100644
--- a/llvm/lib/Transforms/InstCombine/InstCombineMulDivRem.cpp
+++ b/llvm/lib/Transforms/InstCombine/InstCombineMulDivRem.cpp
@@ -1698,6 +1698,63 @@ Instruction *InstCombinerImpl::visitFDiv(BinaryOperator &I) {
   return nullptr;
 }
 
+// Variety of transform for (urem/srem (mul/shl X, Y), (mul/shl X, Z))
+static Instruction *simplifyIRemMulShl(BinaryOperator &I,
+                                       InstCombinerImpl &IC) {
+  Value *Op0 = I.getOperand(0), *Op1 = I.getOperand(1), *X;
+  const APInt *Y, *Z;
+  if (!(match(Op0, m_Mul(m_Value(X), m_APInt(Y))) &&
+        match(Op1, m_c_Mul(m_Specific(X), m_APInt(Z)))) &&
+      !(match(Op0, m_Mul(m_APInt(Y), m_Value(X))) &&
+        match(Op1, m_c_Mul(m_Specific(X), m_APInt(Z)))))
+    return nullptr;
+
+  bool IsSRem = I.getOpcode() == Instruction::SRem;
+
+  OverflowingBinaryOperator *BO0 = cast<OverflowingBinaryOperator>(Op0);
+  // TODO: We may be able to deduce more about nsw/nuw of BO0/BO1 based on Y >=
+  // Z or Z >= Y.
+  bool BO0HasNSW = BO0->hasNoSignedWrap();
+  bool BO0HasNUW = BO0->hasNoUnsignedWrap();
+  bool BO0NoWrap = IsSRem ? BO0HasNSW : BO0HasNUW;
+
+  APInt RemYZ = IsSRem ? Y->srem(*Z) : Y->urem(*Z);
+  // (rem (mul nuw/nsw X, Y), (mul X, Z))
+  //      if (rem Y, Z) == 0
+  //          -> 0
+  if (RemYZ.isZero() && BO0NoWrap)
+    return IC.replaceInstUsesWith(I, ConstantInt::getNullValue(I.getType()));
+
+  OverflowingBinaryOperator *BO1 = cast<OverflowingBinaryOperator>(Op1);
+  bool BO1HasNSW = BO1->hasNoSignedWrap();
+  bool BO1HasNUW = BO1->hasNoUnsignedWrap();
+  bool BO1NoWrap = IsSRem ? BO1HasNSW : BO1HasNUW;
+  // (rem (mul X, Y), (mul nuw/nsw X, Z))
+  //      if (rem Y, Z) == Y
+  //          -> (mul nuw/nsw X, Y)
+  if (RemYZ == *Y && BO1NoWrap) {
+    BinaryOperator *BO =
+        BinaryOperator::CreateMul(X, ConstantInt::get(I.getType(), *Y));
+    // Copy any overflow flags from Op0.
+    BO->setHasNoSignedWrap(IsSRem || BO0HasNSW);
+    BO->setHasNoUnsignedWrap(!IsSRem || BO0HasNUW);
+    return BO;
+  }
+
+  // (rem (mul nuw/nsw X, Y), (mul {nsw} X, Z))
+  //      if Y >= Z
+  //          -> (mul {nuw} nsw X, (rem Y, Z))
+  if (Y->uge(*Z) && (IsSRem ? (BO0HasNSW && BO1HasNSW) : BO0HasNUW)) {
+    BinaryOperator *BO =
+        BinaryOperator::CreateMul(X, ConstantInt::get(I.getType(), RemYZ));
+    BO->setHasNoSignedWrap();
+    BO->setHasNoUnsignedWrap(BO0HasNUW);
+    return BO;
+  }
+
+  return nullptr;
+}
+
 /// This function implements the transforms common to both integer remainder
 /// instructions (urem and srem). It is called by the visitors to those integer
 /// remainder instructions.
@@ -1750,6 +1807,9 @@ Instruction *InstCombinerImpl::commonIRemTransforms(BinaryOperator &I) {
     }
   }
 
+  if (Instruction *R = simplifyIRemMulShl(I, *this))
+    return R;
+
   return nullptr;
 }
 
diff --git a/llvm/lib/Transforms/InstCombine/InstCombinePHI.cpp b/llvm/lib/Transforms/InstCombine/InstCombinePHI.cpp
index d5584b77aa826..2565abcffd71e 100644
--- a/llvm/lib/Transforms/InstCombine/InstCombinePHI.cpp
+++ b/llvm/lib/Transforms/InstCombine/InstCombinePHI.cpp
@@ -1389,11 +1389,10 @@ Instruction *InstCombinerImpl::visitPHINode(PHINode &PN) {
 
   // If all PHI operands are the same operation, pull them through the PHI,
   // reducing code size.
-  if (isa<Instruction>(PN.getIncomingValue(0)) &&
-      isa<Instruction>(PN.getIncomingValue(1)) &&
-      cast<Instruction>(PN.getIncomingValue(0))->getOpcode() ==
-          cast<Instruction>(PN.getIncomingValue(1))->getOpcode() &&
-      PN.getIncomingValue(0)->hasOneUser())
+  auto *Inst0 = dyn_cast<Instruction>(PN.getIncomingValue(0));
+  auto *Inst1 = dyn_cast<Instruction>(PN.getIncomingValue(1));
+  if (Inst0 && Inst1 && Inst0->getOpcode() == Inst1->getOpcode() &&
+      Inst0->hasOneUser())
     if (Instruction *Result = foldPHIArgOpIntoPHI(PN))
       return Result;
 
diff --git a/llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp b/llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp
index e5916981fc3e5..1f2441bc9fcf9 100644
--- a/llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp
+++ b/llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp
@@ -950,6 +950,47 @@ static Value *canonicalizeSaturatedAdd(ICmpInst *Cmp, Value *TVal, Value *FVal,
   return nullptr;
 }
 
+/// Try to match patterns with select and subtract as absolute difference.
+static Value *foldAbsDiff(ICmpInst *Cmp, Value *TVal, Value *FVal,
+                          InstCombiner::BuilderTy &Builder) {
+  auto *TI = dyn_cast<Instruction>(TVal);
+  auto *FI = dyn_cast<Instruction>(FVal);
+  if (!TI || !FI)
+    return nullptr;
+
+  // Normalize predicate to gt/lt rather than ge/le.
+  ICmpInst::Predicate Pred = Cmp->getStrictPredicate();
+  Value *A = Cmp->getOperand(0);
+  Value *B = Cmp->getOperand(1);
+
+  // Normalize "A - B" as the true value of the select.
+  if (match(FI, m_Sub(m_Specific(A), m_Specific(B)))) {
+    std::swap(FI, TI);
+    Pred = ICmpInst::getSwappedPredicate(Pred);
+  }
+
+  // With any pair of no-wrap subtracts:
+  // (A > B) ? (A - B) : (B - A) --> abs(A - B)
+  if (Pred == CmpInst::ICMP_SGT &&
+      match(TI, m_Sub(m_Specific(A), m_Specific(B))) &&
+      match(FI, m_Sub(m_Specific(B), m_Specific(A))) &&
+      (TI->hasNoSignedWrap() || TI->hasNoUnsignedWrap()) &&
+      (FI->hasNoSignedWrap() || FI->hasNoUnsignedWrap())) {
+    // The remaining subtract is not "nuw" any more.
+    // If there's one use of the subtract (no other use than the use we are
+    // about to replace), then we know that the sub is "nsw" in this context
+    // even if it was only "nuw" before. If there's another use, then we can't
+    // add "nsw" to the existing instruction because it may not be safe in the
+    // other user's context.
+    TI->setHasNoUnsignedWrap(false);
+    if (!TI->hasNoSignedWrap())
+      TI->setHasNoSignedWrap(TI->hasOneUse());
+    return Builder.CreateBinaryIntrinsic(Intrinsic::abs, TI, Builder.getTrue());
+  }
+
+  return nullptr;
+}
+
 /// Fold the following code sequence:
 /// \code
 ///   int a = ctlz(x & -x);
@@ -1687,7 +1728,7 @@ Instruction *InstCombinerImpl::foldSelectInstWithICmp(SelectInst &SI,
   ICmpInst::Predicate Pred = ICI->getPredicate();
   Value *CmpLHS = ICI->getOperand(0);
   Value *CmpRHS = ICI->getOperand(1);
-  if (CmpRHS != CmpLHS && isa<Constant>(CmpRHS)) {
+  if (CmpRHS != CmpLHS && isa<Constant>(CmpRHS) && !isa<Constant>(CmpLHS)) {
     if (CmpLHS == TrueVal && Pred == ICmpInst::ICMP_EQ) {
       // Transform (X == C) ? X : Y -> (X == C) ? C : Y
       SI.setOperand(1, CmpRHS);
@@ -1790,6 +1831,9 @@ Instruction *InstCombinerImpl::foldSelectInstWithICmp(SelectInst &SI,
   if (Value *V = canonicalizeSaturatedAdd(ICI, TrueVal, FalseVal, Builder))
     return replaceInstUsesWith(SI, V);
 
+  if (Value *V = foldAbsDiff(ICI, TrueVal, FalseVal, Builder))
+    return replaceInstUsesWith(SI, V);
+
   return Changed ? &SI : nullptr;
 }
 
@@ -2999,6 +3043,14 @@ Instruction *InstCombinerImpl::foldSelectOfBools(SelectInst &SI) {
   if (match(CondVal, m_Select(m_Value(A), m_Value(B), m_Zero())) &&
       match(TrueVal, m_Specific(B)) && match(FalseVal, m_Zero()))
     return replaceOperand(SI, 0, A);
+  // select a, (select ~a, true, b), false -> select a, b, false
+  if (match(TrueVal, m_c_LogicalOr(m_Not(m_Specific(CondVal)), m_Value(B))) &&
+      match(FalseVal, m_Zero()))
+    return replaceOperand(SI, 1, B);
+  // select a, true, (select ~a, b, false) -> select a, true, b
+  if (match(FalseVal, m_c_LogicalAnd(m_Not(m_Specific(CondVal)), m_Value(B))) &&
+      match(TrueVal, m_One()))
+    return replaceOperand(SI, 2, B);
 
   // ~(A & B) & (A | B) --> A ^ B
   if (match(&SI, m_c_LogicalAnd(m_Not(m_LogicalAnd(m_Value(A), m_Value(B))),
diff --git a/llvm/lib/Transforms/InstCombine/InstCombineShifts.cpp b/llvm/lib/Transforms/InstCombine/InstCombineShifts.cpp
index 0f8a327177ec6..6f69e123014c2 100644
--- a/llvm/lib/Transforms/InstCombine/InstCombineShifts.cpp
+++ b/llvm/lib/Transforms/InstCombine/InstCombineShifts.cpp
@@ -483,6 +483,9 @@ Instruction *InstCombinerImpl::commonShiftTransforms(BinaryOperator &I) {
   if (Instruction *Logic = foldShiftOfShiftedBinOp(I, Builder))
     return Logic;
 
+  if (match(Op1, m_Or(m_Value(), m_SpecificInt(BitWidth - 1))))
+    return replaceOperand(I, 1, ConstantInt::get(Ty, BitWidth - 1));
+
   return nullptr;
 }
 
diff --git a/llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp b/llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp
index ea9aba449c749..5851e76454334 100644
--- a/llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp
+++ b/llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp
@@ -2936,7 +2936,7 @@ Instruction *InstCombinerImpl::visitShuffleVectorInst(ShuffleVectorInst &SVI) {
       unsigned SrcElemsPerTgtElem = TgtElemBitWidth / SrcElemBitWidth;
       assert(SrcElemsPerTgtElem);
       BegIdx /= SrcElemsPerTgtElem;
-      bool BCAlreadyExists = NewBCs.find(CastSrcTy) != NewBCs.end();
+      bool BCAlreadyExists = NewBCs.contains(CastSrcTy);
       auto *NewBC =
           BCAlreadyExists
               ? NewBCs[CastSrcTy]
diff --git a/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp b/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp
index a539c476a8c04..b3d4c40655562 100644
--- a/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp
+++ b/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp
@@ -667,17 +667,8 @@ struct AddressSanitizer {
     assert(this->UseAfterReturn != AsanDetectStackUseAfterReturnMode::Invalid);
   }
 
-  uint64_t getAllocaSizeInBytes(const AllocaInst &AI) const {
-    uint64_t ArraySize = 1;
-    if (AI.isArrayAllocation()) {
-      const ConstantInt *CI = dyn_cast<ConstantInt>(AI.getArraySize());
-      assert(CI && "non-constant array size");
-      ArraySize = CI->getZExtValue();
-    }
-    Type *Ty = AI.getAllocatedType();
-    uint64_t SizeInBytes =
-        AI.getModule()->getDataLayout().getTypeAllocSize(Ty);
-    return SizeInBytes * ArraySize;
+  TypeSize getAllocaSizeInBytes(const AllocaInst &AI) const {
+    return *AI.getAllocationSize(AI.getModule()->getDataLayout());
   }
 
   /// Check if we want (and can) handle this alloca.
@@ -1040,7 +1031,9 @@ struct FunctionStackPoisoner : public InstVisitor<FunctionStackPoisoner> {
 
   /// Collect Alloca instructions we want (and can) handle.
   void visitAllocaInst(AllocaInst &AI) {
-    if (!ASan.isInterestingAlloca(AI)) {
+    // FIXME: Handle scalable vectors instead of ignoring them.
+    if (!ASan.isInterestingAlloca(AI) ||
+        isa<ScalableVectorType>(AI.getAllocatedType())) {
       if (AI.isStaticAlloca()) {
         // Skip over allocas that are present *before* the first instrumented
         // alloca, we don't want to move those around.
@@ -1254,7 +1247,7 @@ bool AddressSanitizer::isInterestingAlloca(const AllocaInst &AI) {
   bool IsInteresting =
       (AI.getAllocatedType()->isSized() &&
        // alloca() may be called with 0 size, ignore it.
-       ((!AI.isStaticAlloca()) || getAllocaSizeInBytes(AI) > 0) &&
+       ((!AI.isStaticAlloca()) || !getAllocaSizeInBytes(AI).isZero()) &&
        // We are only interested in allocas not promotable to registers.
        // Promotable allocas are common under -O0.
        (!ClSkipPromotableAllocas || !isAllocaPromotable(&AI)) &&
@@ -1421,11 +1414,20 @@ static void doInstrumentAddress(AddressSanitizer *Pass, Instruction *I,
                                 uint32_t Exp) {
   // Instrument a 1-, 2-, 4-, 8-, or 16- byte access with one check
   // if the data is properly aligned.
-  if ((TypeStoreSize == 8 || TypeStoreSize == 16 || TypeStoreSize == 32 || TypeStoreSize == 64 ||
-       TypeStoreSize == 128) &&
-      (!Alignment || *Alignment >= Granularity || *Alignment >= TypeStoreSize / 8))
-    return Pass->instrumentAddress(I, InsertBefore, Addr, TypeStoreSize, IsWrite,
-                                   nullptr, UseCalls, Exp);
+  if (!TypeStoreSize.isScalable()) {
+    const auto FixedSize = TypeStoreSize.getFixedValue();
+    switch (FixedSize) {
+    case 8:
+    case 16:
+    case 32:
+    case 64:
+    case 128:
+      if (!Alignment || *Alignment >= Granularity ||
+          *Alignment >= FixedSize / 8)
+        return Pass->instrumentAddress(I, InsertBefore, Addr, FixedSize,
+                                       IsWrite, nullptr, UseCalls, Exp);
+    }
+  }
   Pass->instrumentUnusualSizeOrAlignment(I, InsertBefore, Addr, TypeStoreSize,
                                          IsWrite, nullptr, UseCalls, Exp);
 }
@@ -1437,36 +1439,30 @@ static void instrumentMaskedLoadOrStore(AddressSanitizer *Pass,
                                         unsigned Granularity, Type *OpType,
                                         bool IsWrite, Value *SizeArgument,
                                         bool UseCalls, uint32_t Exp) {
-  auto *VTy = cast<FixedVectorType>(OpType);
-  uint64_t ElemTypeSize = DL.getTypeStoreSizeInBits(VTy->getScalarType());
-  unsigned Num = VTy->getNumElements();
+  auto *VTy = cast<VectorType>(OpType);
+
+  TypeSize ElemTypeSize = DL.getTypeStoreSizeInBits(VTy->getScalarType());
   auto Zero = ConstantInt::get(IntptrTy, 0);
-  for (unsigned Idx = 0; Idx < Num; ++Idx) {
-    Value *InstrumentedAddress = nullptr;
-    Instruction *InsertBefore = I;
-    if (auto *Vector = dyn_cast<ConstantVector>(Mask)) {
-      // dyn_cast as we might get UndefValue
-      if (auto *Masked = dyn_cast<ConstantInt>(Vector->getOperand(Idx))) {
-        if (Masked->isZero())
-          // Mask is constant false, so no instrumentation needed.
-          continue;
-        // If we have a true or undef value, fall through to doInstrumentAddress
-        // with InsertBefore == I
-      }
+
+  SplitBlockAndInsertForEachLane(VTy->getElementCount(), IntptrTy, I,
+                                 [&](IRBuilderBase &IRB, Value *Index) {
+    Value *MaskElem = IRB.CreateExtractElement(Mask, Index);
+    if (auto *MaskElemC = dyn_cast<ConstantInt>(MaskElem)) {
+      if (MaskElemC->isZero())
+        // No check
+        return;
+      // Unconditional check
     } else {
-      IRBuilder<> IRB(I);
-      Value *MaskElem = IRB.CreateExtractElement(Mask, Idx);
-      Instruction *ThenTerm = SplitBlockAndInsertIfThen(MaskElem, I, false);
-      InsertBefore = ThenTerm;
+      // Conditional check
+      Instruction *ThenTerm = SplitBlockAndInsertIfThen(MaskElem, &*IRB.GetInsertPoint(), false);
+      IRB.SetInsertPoint(ThenTerm);
     }
 
-    IRBuilder<> IRB(InsertBefore);
-    InstrumentedAddress =
-        IRB.CreateGEP(VTy, Addr, {Zero, ConstantInt::get(IntptrTy, Idx)});
-    doInstrumentAddress(Pass, I, InsertBefore, InstrumentedAddress, Alignment,
-                        Granularity, TypeSize::Fixed(ElemTypeSize), IsWrite,
-                        SizeArgument, UseCalls, Exp);
-  }
+    Value *InstrumentedAddress = IRB.CreateGEP(VTy, Addr, {Zero, Index});
+    doInstrumentAddress(Pass, I, &*IRB.GetInsertPoint(), InstrumentedAddress, Alignment,
+                        Granularity, ElemTypeSize, IsWrite, SizeArgument,
+                        UseCalls, Exp);
+  });
 }
 
 void AddressSanitizer::instrumentMop(ObjectSizeOffsetVisitor &ObjSizeVis,
@@ -1676,7 +1672,9 @@ void AddressSanitizer::instrumentUnusualSizeOrAlignment(
     Instruction *I, Instruction *InsertBefore, Value *Addr, TypeSize TypeStoreSize,
     bool IsWrite, Value *SizeArgument, bool UseCalls, uint32_t Exp) {
   IRBuilder<> IRB(InsertBefore);
-  Value *Size = ConstantInt::get(IntptrTy, TypeStoreSize / 8);
+  Value *NumBits = IRB.CreateTypeSize(IntptrTy, TypeStoreSize);
+  Value *Size = IRB.CreateLShr(NumBits, ConstantInt::get(IntptrTy, 3));
+
   Value *AddrLong = IRB.CreatePointerCast(Addr, IntptrTy);
   if (UseCalls) {
     if (Exp == 0)
@@ -1686,8 +1684,9 @@ void AddressSanitizer::instrumentUnusualSizeOrAlignment(
       IRB.CreateCall(AsanMemoryAccessCallbackSized[IsWrite][1],
                      {AddrLong, Size, ConstantInt::get(IRB.getInt32Ty(), Exp)});
   } else {
+    Value *SizeMinusOne = IRB.CreateSub(Size, ConstantInt::get(IntptrTy, 1));
     Value *LastByte = IRB.CreateIntToPtr(
-        IRB.CreateAdd(AddrLong, ConstantInt::get(IntptrTy, TypeStoreSize / 8 - 1)),
+        IRB.CreateAdd(AddrLong, SizeMinusOne),
         Addr->getType());
     instrumentAddress(I, InsertBefore, Addr, 8, IsWrite, Size, false, Exp);
     instrumentAddress(I, InsertBefore, LastByte, 8, IsWrite, Size, false, Exp);
@@ -3486,6 +3485,10 @@ void FunctionStackPoisoner::handleDynamicAllocaCall(AllocaInst *AI) {
 // constant inbounds index.
 bool AddressSanitizer::isSafeAccess(ObjectSizeOffsetVisitor &ObjSizeVis,
                                     Value *Addr, TypeSize TypeStoreSize) const {
+  if (TypeStoreSize.isScalable())
+    // TODO: We can use vscale_range to convert a scalable value to an
+    // upper bound on the access size.
+    return false;
   SizeOffsetType SizeOffset = ObjSizeVis.compute(Addr);
   if (!ObjSizeVis.bothKnown(SizeOffset)) return false;
   uint64_t Size = SizeOffset.first.getZExtValue();
diff --git a/llvm/lib/Transforms/Instrumentation/ControlHeightReduction.cpp b/llvm/lib/Transforms/Instrumentation/ControlHeightReduction.cpp
index 83e662906bceb..c5f0d2d207684 100644
--- a/llvm/lib/Transforms/Instrumentation/ControlHeightReduction.cpp
+++ b/llvm/lib/Transforms/Instrumentation/ControlHeightReduction.cpp
@@ -1888,8 +1888,7 @@ void CHR::fixupBranch(Region *R, CHRScope *Scope,
   assert((IsTrueBiased || Scope->FalseBiasedRegions.count(R)) &&
          "Must be truthy or falsy");
   auto *BI = cast<BranchInst>(R->getEntry()->getTerminator());
-  assert(BranchBiasMap.find(R) != BranchBiasMap.end() &&
-         "Must be in the bias map");
+  assert(BranchBiasMap.contains(R) && "Must be in the bias map");
   BranchProbability Bias = BranchBiasMap[R];
   assert(Bias >= getCHRBiasThreshold() && "Must be highly biased");
   // Take the min.
@@ -1931,8 +1930,7 @@ void CHR::fixupSelect(SelectInst *SI, CHRScope *Scope,
   bool IsTrueBiased = Scope->TrueBiasedSelects.count(SI);
   assert((IsTrueBiased ||
           Scope->FalseBiasedSelects.count(SI)) && "Must be biased");
-  assert(SelectBiasMap.find(SI) != SelectBiasMap.end() &&
-         "Must be in the bias map");
+  assert(SelectBiasMap.contains(SI) && "Must be in the bias map");
   BranchProbability Bias = SelectBiasMap[SI];
   assert(Bias >= getCHRBiasThreshold() && "Must be highly biased");
   // Take the min.
@@ -1962,11 +1960,8 @@ void CHR::addToMergedCondition(bool IsTrueBiased, Value *Cond,
       Cond = IRB.CreateXor(ConstantInt::getTrue(F.getContext()), Cond);
   }
 
-  // Select conditions can be poison, while branching on poison is immediate
-  // undefined behavior. As such, we need to freeze potentially poisonous
-  // conditions derived from selects.
-  if (isa<SelectInst>(BranchOrSelect) &&
-      !isGuaranteedNotToBeUndefOrPoison(Cond))
+  // Freeze potentially poisonous conditions.
+  if (!isGuaranteedNotToBeUndefOrPoison(Cond))
     Cond = IRB.CreateFreeze(Cond);
 
   // Use logical and to avoid propagating poison from later conditions.
diff --git a/llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp b/llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp
index 2690c30212606..ca498d08422f7 100644
--- a/llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp
+++ b/llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp
@@ -282,7 +282,7 @@ class HWAddressSanitizer {
 
   void setSSI(const StackSafetyGlobalInfo *S) { SSI = S; }
 
-  bool sanitizeFunction(Function &F, FunctionAnalysisManager &FAM);
+  void sanitizeFunction(Function &F, FunctionAnalysisManager &FAM);
   void initializeModule();
   void createHwasanCtorComdat();
 
@@ -420,12 +420,9 @@ PreservedAnalyses HWAddressSanitizerPass::run(Module &M,
     SSI = &MAM.getResult<StackSafetyGlobalAnalysis>(M);
 
   HWAddressSanitizer HWASan(M, Options.CompileKernel, Options.Recover, SSI);
-  bool Modified = false;
   auto &FAM = MAM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
   for (Function &F : M)
-    Modified |= HWASan.sanitizeFunction(F, FAM);
-  if (!Modified)
-    return PreservedAnalyses::all();
+    HWASan.sanitizeFunction(F, FAM);
 
   PreservedAnalyses PA = PreservedAnalyses::none();
   // GlobalsAA is considered stateless and does not get invalidated unless
@@ -1402,13 +1399,13 @@ bool HWAddressSanitizer::instrumentStack(memtag::StackInfo &SInfo,
   return true;
 }
 
-bool HWAddressSanitizer::sanitizeFunction(Function &F,
+void HWAddressSanitizer::sanitizeFunction(Function &F,
                                           FunctionAnalysisManager &FAM) {
   if (&F == HwasanCtorFunction)
-    return false;
+    return;
 
   if (!F.hasFnAttribute(Attribute::SanitizeHWAddress))
-    return false;
+    return;
 
   LLVM_DEBUG(dbgs() << "Function: " << F.getName() << "\n");
 
@@ -1436,22 +1433,19 @@ bool HWAddressSanitizer::sanitizeFunction(Function &F,
 
   initializeCallbacks(*F.getParent());
 
-  bool Changed = false;
-
   if (!LandingPadVec.empty())
-    Changed |= instrumentLandingPads(LandingPadVec);
+    instrumentLandingPads(LandingPadVec);
 
   if (SInfo.AllocasToInstrument.empty() && F.hasPersonalityFn() &&
       F.getPersonalityFn()->getName() == kHwasanPersonalityThunkName) {
     // __hwasan_personality_thunk is a no-op for functions without an
     // instrumented stack, so we can drop it.
     F.setPersonalityFn(nullptr);
-    Changed = true;
   }
 
   if (SInfo.AllocasToInstrument.empty() && OperandsToInstrument.empty() &&
       IntrinToInstrument.empty())
-    return Changed;
+    return;
 
   assert(!ShadowBase);
 
@@ -1495,8 +1489,6 @@ bool HWAddressSanitizer::sanitizeFunction(Function &F,
   ShadowBase = nullptr;
   StackBaseTag = nullptr;
   CachedSP = nullptr;
-
-  return true;
 }
 
 void HWAddressSanitizer::instrumentGlobal(GlobalVariable *GV, uint8_t Tag) {
diff --git a/llvm/lib/Transforms/Instrumentation/MemorySanitizer.cpp b/llvm/lib/Transforms/Instrumentation/MemorySanitizer.cpp
index 23c85ca87de3d..75cb9e0c0e63d 100644
--- a/llvm/lib/Transforms/Instrumentation/MemorySanitizer.cpp
+++ b/llvm/lib/Transforms/Instrumentation/MemorySanitizer.cpp
@@ -1056,10 +1056,14 @@ struct MemorySanitizerVisitor;
 static VarArgHelper *CreateVarArgHelper(Function &Func, MemorySanitizer &Msan,
                                         MemorySanitizerVisitor &Visitor);
 
-static unsigned TypeSizeToSizeIndex(unsigned TypeSize) {
-  if (TypeSize <= 8)
+static unsigned TypeSizeToSizeIndex(TypeSize TS) {
+  if (TS.isScalable())
+    // Scalable types unconditionally take slowpaths.
+    return kNumberOfAccessSizes;
+  unsigned TypeSizeFixed = TS.getFixedValue();
+  if (TypeSizeFixed <= 8)
     return 0;
-  return Log2_32_Ceil((TypeSize + 7) / 8);
+  return Log2_32_Ceil((TypeSizeFixed + 7) / 8);
 }
 
 namespace {
@@ -1178,7 +1182,8 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
 
   /// Fill memory range with the given origin value.
   void paintOrigin(IRBuilder<> &IRB, Value *Origin, Value *OriginPtr,
-                   unsigned Size, Align Alignment) {
+                   TypeSize TS, Align Alignment) {
+    unsigned Size = TS.getFixedValue();
     const DataLayout &DL = F.getParent()->getDataLayout();
     const Align IntptrAlignment = DL.getABITypeAlign(MS.IntptrTy);
     unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);
@@ -1212,7 +1217,7 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
                    Value *OriginPtr, Align Alignment) {
     const DataLayout &DL = F.getParent()->getDataLayout();
     const Align OriginAlignment = std::max(kMinOriginAlignment, Alignment);
-    unsigned StoreSize = DL.getTypeStoreSize(Shadow->getType());
+    TypeSize StoreSize = DL.getTypeStoreSize(Shadow->getType());
     Value *ConvertedShadow = convertShadowToScalar(Shadow, IRB);
     if (auto *ConstantShadow = dyn_cast<Constant>(ConvertedShadow)) {
       if (!ClCheckConstantShadow || ConstantShadow->isZeroValue()) {
@@ -1229,7 +1234,7 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
       // Fallback to runtime check, which still can be optimized out later.
     }
 
-    unsigned TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType());
+    TypeSize TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType());
     unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits);
     if (instrumentWithCalls(ConvertedShadow) &&
         SizeIndex < kNumberOfAccessSizes && !MS.CompileKernel) {
@@ -1325,7 +1330,7 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
   void materializeOneCheck(IRBuilder<> &IRB, Value *ConvertedShadow,
                            Value *Origin) {
     const DataLayout &DL = F.getParent()->getDataLayout();
-    unsigned TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType());
+    TypeSize TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType());
     unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits);
     if (instrumentWithCalls(ConvertedShadow) &&
         SizeIndex < kNumberOfAccessSizes && !MS.CompileKernel) {
@@ -1505,8 +1510,8 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
     const DataLayout &DL = F.getParent()->getDataLayout();
     if (VectorType *VT = dyn_cast<VectorType>(OrigTy)) {
       uint32_t EltSize = DL.getTypeSizeInBits(VT->getElementType());
-      return FixedVectorType::get(IntegerType::get(*MS.C, EltSize),
-                                  cast<FixedVectorType>(VT)->getNumElements());
+      return VectorType::get(IntegerType::get(*MS.C, EltSize),
+                             VT->getElementCount());
     }
     if (ArrayType *AT = dyn_cast<ArrayType>(OrigTy)) {
       return ArrayType::get(getShadowTy(AT->getElementType()),
@@ -1524,14 +1529,6 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
     return IntegerType::get(*MS.C, TypeSize);
   }
 
-  /// Flatten a vector type.
-  Type *getShadowTyNoVec(Type *ty) {
-    if (VectorType *vt = dyn_cast<VectorType>(ty))
-      return IntegerType::get(*MS.C,
-                              vt->getPrimitiveSizeInBits().getFixedValue());
-    return ty;
-  }
-
   /// Extract combined shadow of struct elements as a bool
   Value *collapseStructShadow(StructType *Struct, Value *Shadow,
                               IRBuilder<> &IRB) {
@@ -1541,8 +1538,7 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
     for (unsigned Idx = 0; Idx < Struct->getNumElements(); Idx++) {
       // Combine by ORing together each element's bool shadow
       Value *ShadowItem = IRB.CreateExtractValue(Shadow, Idx);
-      Value *ShadowInner = convertShadowToScalar(ShadowItem, IRB);
-      Value *ShadowBool = convertToBool(ShadowInner, IRB);
+      Value *ShadowBool = convertToBool(ShadowItem, IRB);
 
       if (Aggregator != FalseVal)
         Aggregator = IRB.CreateOr(Aggregator, ShadowBool);
@@ -1578,11 +1574,12 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
       return collapseStructShadow(Struct, V, IRB);
     if (ArrayType *Array = dyn_cast<ArrayType>(V->getType()))
       return collapseArrayShadow(Array, V, IRB);
-    Type *Ty = V->getType();
-    Type *NoVecTy = getShadowTyNoVec(Ty);
-    if (Ty == NoVecTy)
-      return V;
-    return IRB.CreateBitCast(V, NoVecTy);
+    if (isa<VectorType>(V->getType())) {
+      unsigned BitWidth =
+        V->getType()->getPrimitiveSizeInBits().getFixedValue();
+      return IRB.CreateBitCast(V, IntegerType::get(*MS.C, BitWidth));
+    }
+    return V;
   }
 
   // Convert a scalar value to an i1 by comparing with 0
@@ -1597,28 +1594,28 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
   }
 
   Type *ptrToIntPtrType(Type *PtrTy) const {
-    if (FixedVectorType *VectTy = dyn_cast<FixedVectorType>(PtrTy)) {
-      return FixedVectorType::get(ptrToIntPtrType(VectTy->getElementType()),
-                                  VectTy->getNumElements());
+    if (VectorType *VectTy = dyn_cast<VectorType>(PtrTy)) {
+      return VectorType::get(ptrToIntPtrType(VectTy->getElementType()),
+                             VectTy->getElementCount());
     }
     assert(PtrTy->isIntOrPtrTy());
     return MS.IntptrTy;
   }
 
   Type *getPtrToShadowPtrType(Type *IntPtrTy, Type *ShadowTy) const {
-    if (FixedVectorType *VectTy = dyn_cast<FixedVectorType>(IntPtrTy)) {
-      return FixedVectorType::get(
+    if (VectorType *VectTy = dyn_cast<VectorType>(IntPtrTy)) {
+      return VectorType::get(
           getPtrToShadowPtrType(VectTy->getElementType(), ShadowTy),
-          VectTy->getNumElements());
+          VectTy->getElementCount());
     }
     assert(IntPtrTy == MS.IntptrTy);
     return ShadowTy->getPointerTo();
   }
 
   Constant *constToIntPtr(Type *IntPtrTy, uint64_t C) const {
-    if (FixedVectorType *VectTy = dyn_cast<FixedVectorType>(IntPtrTy)) {
-      return ConstantDataVector::getSplat(
-          VectTy->getNumElements(), constToIntPtr(VectTy->getElementType(), C));
+    if (VectorType *VectTy = dyn_cast<VectorType>(IntPtrTy)) {
+      return ConstantVector::getSplat(
+          VectTy->getElementCount(), constToIntPtr(VectTy->getElementType(), C));
     }
     assert(IntPtrTy == MS.IntptrTy);
     return ConstantInt::get(MS.IntptrTy, C);
@@ -1687,7 +1684,7 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
                                                             bool isStore) {
     Value *ShadowOriginPtrs;
     const DataLayout &DL = F.getParent()->getDataLayout();
-    int Size = DL.getTypeStoreSize(ShadowTy);
+    TypeSize Size = DL.getTypeStoreSize(ShadowTy);
 
     FunctionCallee Getter = MS.getKmsanShadowOriginAccessFn(isStore, Size);
     Value *AddrCast =
@@ -1714,14 +1711,14 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
                                                        IRBuilder<> &IRB,
                                                        Type *ShadowTy,
                                                        bool isStore) {
-    FixedVectorType *VectTy = dyn_cast<FixedVectorType>(Addr->getType());
+    VectorType *VectTy = dyn_cast<VectorType>(Addr->getType());
     if (!VectTy) {
       assert(Addr->getType()->isPointerTy());
       return getShadowOriginPtrKernelNoVec(Addr, IRB, ShadowTy, isStore);
     }
 
     // TODO: Support callbacs with vectors of addresses.
-    unsigned NumElements = VectTy->getNumElements();
+    unsigned NumElements = cast<FixedVectorType>(VectTy)->getNumElements();
     Value *ShadowPtrs = ConstantInt::getNullValue(
         FixedVectorType::get(ShadowTy->getPointerTo(), NumElements));
     Value *OriginPtrs = nullptr;
@@ -2367,9 +2364,7 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
           Constant *ConstOrigin = dyn_cast<Constant>(OpOrigin);
           // No point in adding something that might result in 0 origin value.
           if (!ConstOrigin || !ConstOrigin->isNullValue()) {
-            Value *FlatShadow = MSV->convertShadowToScalar(OpShadow, IRB);
-            Value *Cond =
-                IRB.CreateICmpNE(FlatShadow, MSV->getCleanShadow(FlatShadow));
+            Value *Cond = MSV->convertToBool(OpShadow, IRB);
             Origin = IRB.CreateSelect(Cond, OpOrigin, Origin);
           }
         }
@@ -2434,8 +2429,8 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
     if (dstTy->isIntegerTy() && srcTy->isIntegerTy())
       return IRB.CreateIntCast(V, dstTy, Signed);
     if (dstTy->isVectorTy() && srcTy->isVectorTy() &&
-        cast<FixedVectorType>(dstTy)->getNumElements() ==
-            cast<FixedVectorType>(srcTy)->getNumElements())
+        cast<VectorType>(dstTy)->getElementCount() ==
+            cast<VectorType>(srcTy)->getElementCount())
       return IRB.CreateIntCast(V, dstTy, Signed);
     Value *V1 = IRB.CreateBitCast(V, Type::getIntNTy(*MS.C, srcSizeInBits));
     Value *V2 =
@@ -3356,7 +3351,7 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
     }
 
     Type *ShadowTy = getShadowTy(&I);
-    Type *ElementShadowTy = cast<FixedVectorType>(ShadowTy)->getElementType();
+    Type *ElementShadowTy = cast<VectorType>(ShadowTy)->getElementType();
     auto [ShadowPtr, OriginPtr] =
         getShadowOriginPtr(Ptr, IRB, ElementShadowTy, {}, /*isStore*/ false);
 
@@ -3382,7 +3377,7 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
 
     Value *Shadow = getShadow(Values);
     Type *ElementShadowTy =
-        getShadowTy(cast<FixedVectorType>(Values->getType())->getElementType());
+        getShadowTy(cast<VectorType>(Values->getType())->getElementType());
     auto [ShadowPtr, OriginPtrs] =
         getShadowOriginPtr(Ptr, IRB, ElementShadowTy, {}, /*isStore*/ true);
 
@@ -3415,7 +3410,7 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
     }
 
     Type *ShadowTy = getShadowTy(&I);
-    Type *ElementShadowTy = cast<FixedVectorType>(ShadowTy)->getElementType();
+    Type *ElementShadowTy = cast<VectorType>(ShadowTy)->getElementType();
     auto [ShadowPtrs, OriginPtrs] = getShadowOriginPtr(
         Ptrs, IRB, ElementShadowTy, Alignment, /*isStore*/ false);
 
@@ -3448,7 +3443,7 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
 
     Value *Shadow = getShadow(Values);
     Type *ElementShadowTy =
-        getShadowTy(cast<FixedVectorType>(Values->getType())->getElementType());
+        getShadowTy(cast<VectorType>(Values->getType())->getElementType());
     auto [ShadowPtrs, OriginPtrs] = getShadowOriginPtr(
         Ptrs, IRB, ElementShadowTy, Alignment, /*isStore*/ true);
 
@@ -3520,8 +3515,7 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
     Value *MaskedPassThruShadow = IRB.CreateAnd(
         getShadow(PassThru), IRB.CreateSExt(IRB.CreateNeg(Mask), ShadowTy));
 
-    Value *ConvertedShadow = convertShadowToScalar(MaskedPassThruShadow, IRB);
-    Value *NotNull = convertToBool(ConvertedShadow, IRB, "_mscmp");
+    Value *NotNull = convertToBool(MaskedPassThruShadow, IRB, "_mscmp");
 
     Value *PtrOrigin = IRB.CreateLoad(MS.OriginTy, OriginPtr);
     Value *Origin = IRB.CreateSelect(NotNull, getOrigin(PassThru), PtrOrigin);
@@ -4391,11 +4385,8 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
       // Origins are always i32, so any vector conditions must be flattened.
       // FIXME: consider tracking vector origins for app vectors?
       if (B->getType()->isVectorTy()) {
-        Type *FlatTy = getShadowTyNoVec(B->getType());
-        B = IRB.CreateICmpNE(IRB.CreateBitCast(B, FlatTy),
-                             ConstantInt::getNullValue(FlatTy));
-        Sb = IRB.CreateICmpNE(IRB.CreateBitCast(Sb, FlatTy),
-                              ConstantInt::getNullValue(FlatTy));
+        B = convertToBool(B, IRB);
+        Sb = convertToBool(Sb, IRB);
       }
       // a = select b, c, d
       // Oa = Sb ? Ob : (b ? Oc : Od)
@@ -4490,9 +4481,9 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
     }
     if (!ElemTy->isSized())
       return;
-    int Size = DL.getTypeStoreSize(ElemTy);
     Value *Ptr = IRB.CreatePointerCast(Operand, IRB.getInt8PtrTy());
-    Value *SizeVal = ConstantInt::get(MS.IntptrTy, Size);
+    Value *SizeVal =
+      IRB.CreateTypeSize(MS.IntptrTy, DL.getTypeStoreSize(ElemTy));
     IRB.CreateCall(MS.MsanInstrumentAsmStoreFn, {Ptr, SizeVal});
   }
 
@@ -4721,7 +4712,7 @@ struct VarArgAMD64Helper : public VarArgHelper {
         IRB.CreateAlignedStore(Shadow, ShadowBase, kShadowTLSAlignment);
         if (MS.TrackOrigins) {
           Value *Origin = MSV.getOrigin(A);
-          unsigned StoreSize = DL.getTypeStoreSize(Shadow->getType());
+          TypeSize StoreSize = DL.getTypeStoreSize(Shadow->getType());
           MSV.paintOrigin(IRB, Origin, OriginBase, StoreSize,
                           std::max(kShadowTLSAlignment, kMinOriginAlignment));
         }
@@ -5586,7 +5577,7 @@ struct VarArgSystemZHelper : public VarArgHelper {
       IRB.CreateStore(Shadow, ShadowBase);
       if (MS.TrackOrigins) {
         Value *Origin = MSV.getOrigin(A);
-        unsigned StoreSize = DL.getTypeStoreSize(Shadow->getType());
+        TypeSize StoreSize = DL.getTypeStoreSize(Shadow->getType());
         MSV.paintOrigin(IRB, Origin, OriginBase, StoreSize,
                         kMinOriginAlignment);
       }
diff --git a/llvm/lib/Transforms/Instrumentation/PGOInstrumentation.cpp b/llvm/lib/Transforms/Instrumentation/PGOInstrumentation.cpp
index d49f893b1045a..b7ee55dfefb15 100644
--- a/llvm/lib/Transforms/Instrumentation/PGOInstrumentation.cpp
+++ b/llvm/lib/Transforms/Instrumentation/PGOInstrumentation.cpp
@@ -345,7 +345,7 @@ static std::string getBranchCondString(Instruction *TI) {
 
   std::string result;
   raw_string_ostream OS(result);
-  OS << CmpInst::getPredicateName(CI->getPredicate()) << "_";
+  OS << CI->getPredicate() << "_";
   CI->getOperand(0)->getType()->print(OS, true);
 
   Value *RHS = CI->getOperand(1);
diff --git a/llvm/lib/Transforms/Instrumentation/SanitizerBinaryMetadata.cpp b/llvm/lib/Transforms/Instrumentation/SanitizerBinaryMetadata.cpp
index 563d3e4039a9b..a1309a4ec0f45 100644
--- a/llvm/lib/Transforms/Instrumentation/SanitizerBinaryMetadata.cpp
+++ b/llvm/lib/Transforms/Instrumentation/SanitizerBinaryMetadata.cpp
@@ -131,6 +131,8 @@ class SanitizerBinaryMetadata {
         IRB(M.getContext()) {
     // FIXME: Make it work with other formats.
     assert(TargetTriple.isOSBinFormatELF() && "ELF only");
+    assert(!(TargetTriple.isNVPTX() || TargetTriple.isAMDGPU()) &&
+           "Device targets are not supported");
   }
 
   bool run();
diff --git a/llvm/lib/Transforms/Scalar/ADCE.cpp b/llvm/lib/Transforms/Scalar/ADCE.cpp
index d03a5033f403c..b96feb9056596 100644
--- a/llvm/lib/Transforms/Scalar/ADCE.cpp
+++ b/llvm/lib/Transforms/Scalar/ADCE.cpp
@@ -26,6 +26,7 @@
 #include "llvm/Analysis/DomTreeUpdater.h"
 #include "llvm/Analysis/GlobalsModRef.h"
 #include "llvm/Analysis/IteratedDominanceFrontier.h"
+#include "llvm/Analysis/MemorySSA.h"
 #include "llvm/Analysis/PostDominators.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/CFG.h"
@@ -115,6 +116,7 @@ struct BlockInfoType {
 
 struct ADCEChanged {
   bool ChangedAnything = false;
+  bool ChangedNonDebugInstr = false;
   bool ChangedControlFlow = false;
 };
 
@@ -560,6 +562,8 @@ ADCEChanged AggressiveDeadCodeElimination::removeDeadInstructions() {
         continue;
 
       // Fallthrough and drop the intrinsic.
+    } else {
+      Changed.ChangedNonDebugInstr = true;
     }
 
     // Prepare to delete.
@@ -713,57 +717,19 @@ PreservedAnalyses ADCEPass::run(Function &F, FunctionAnalysisManager &FAM) {
     return PreservedAnalyses::all();
 
   PreservedAnalyses PA;
-  if (!Changed.ChangedControlFlow)
+  if (!Changed.ChangedControlFlow) {
     PA.preserveSet<CFGAnalyses>();
+    if (!Changed.ChangedNonDebugInstr) {
+      // Only removing debug instructions does not affect MemorySSA.
+      //
+      // Therefore we preserve MemorySSA when only removing debug instructions
+      // since otherwise later passes may behave differently which then makes
+      // the presence of debug info affect code generation.
+      PA.preserve<MemorySSAAnalysis>();
+    }
+  }
   PA.preserve<DominatorTreeAnalysis>();
   PA.preserve<PostDominatorTreeAnalysis>();
 
   return PA;
 }
-
-namespace {
-
-struct ADCELegacyPass : public FunctionPass {
-  static char ID; // Pass identification, replacement for typeid
-
-  ADCELegacyPass() : FunctionPass(ID) {
-    initializeADCELegacyPassPass(*PassRegistry::getPassRegistry());
-  }
-
-  bool runOnFunction(Function &F) override {
-    if (skipFunction(F))
-      return false;
-
-    // ADCE does not need DominatorTree, but require DominatorTree here
-    // to update analysis if it is already available.
-    auto *DTWP = getAnalysisIfAvailable<DominatorTreeWrapperPass>();
-    auto *DT = DTWP ? &DTWP->getDomTree() : nullptr;
-    auto &PDT = getAnalysis<PostDominatorTreeWrapperPass>().getPostDomTree();
-    ADCEChanged Changed =
-        AggressiveDeadCodeElimination(F, DT, PDT).performDeadCodeElimination();
-    return Changed.ChangedAnything;
-  }
-
-  void getAnalysisUsage(AnalysisUsage &AU) const override {
-    AU.addRequired<PostDominatorTreeWrapperPass>();
-    if (!RemoveControlFlowFlag)
-      AU.setPreservesCFG();
-    else {
-      AU.addPreserved<DominatorTreeWrapperPass>();
-      AU.addPreserved<PostDominatorTreeWrapperPass>();
-    }
-    AU.addPreserved<GlobalsAAWrapperPass>();
-  }
-};
-
-} // end anonymous namespace
-
-char ADCELegacyPass::ID = 0;
-
-INITIALIZE_PASS_BEGIN(ADCELegacyPass, "adce",
-                      "Aggressive Dead Code Elimination", false, false)
-INITIALIZE_PASS_DEPENDENCY(PostDominatorTreeWrapperPass)
-INITIALIZE_PASS_END(ADCELegacyPass, "adce", "Aggressive Dead Code Elimination",
-                    false, false)
-
-FunctionPass *llvm::createAggressiveDCEPass() { return new ADCELegacyPass(); }
diff --git a/llvm/lib/Transforms/Scalar/ConstraintElimination.cpp b/llvm/lib/Transforms/Scalar/ConstraintElimination.cpp
index e5ec35545ca33..3d4724e5094dd 100644
--- a/llvm/lib/Transforms/Scalar/ConstraintElimination.cpp
+++ b/llvm/lib/Transforms/Scalar/ConstraintElimination.cpp
@@ -32,6 +32,7 @@
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/DebugCounter.h"
+#include "llvm/Support/KnownBits.h"
 #include "llvm/Support/MathExtras.h"
 #include "llvm/Transforms/Utils/Cloning.h"
 #include "llvm/Transforms/Utils/ValueMapper.h"
@@ -369,10 +370,17 @@ static Decomposition decompose(Value *V,
     return MergeResults(Op0, CI, true);
   }
 
+  // Decompose or as an add if there are no common bits between the operands.
+  if (match(V, m_Or(m_Value(Op0), m_ConstantInt(CI))) &&
+      haveNoCommonBitsSet(Op0, CI, DL)) {
+    return MergeResults(Op0, CI, IsSigned);
+  }
+
   if (match(V, m_NUWShl(m_Value(Op1), m_ConstantInt(CI))) && canUseSExt(CI)) {
-    int64_t Mult = int64_t(std::pow(int64_t(2), CI->getSExtValue()));
+    if (CI->getSExtValue() < 0 || CI->getSExtValue() >= 64)
+      return {V, IsKnownNonNegative};
     auto Result = decompose(Op1, Preconditions, IsSigned, DL);
-    Result.mul(Mult);
+    Result.mul(int64_t{1} << CI->getSExtValue());
     return Result;
   }
 
@@ -507,8 +515,8 @@ ConstraintInfo::getConstraint(CmpInst::Predicate Pred, Value *Op0, Value *Op1,
 
   // Add extra constraints for variables that are known positive.
   for (auto &KV : KnownNonNegativeVariables) {
-    if (!KV.second || (Value2Index.find(KV.first) == Value2Index.end() &&
-                       NewIndexMap.find(KV.first) == NewIndexMap.end()))
+    if (!KV.second ||
+        (!Value2Index.contains(KV.first) && !NewIndexMap.contains(KV.first)))
       continue;
     SmallVector<int64_t, 8> C(Value2Index.size() + NewVariables.size() + 1, 0);
     C[GetOrAddIndex(KV.first)] = -1;
@@ -799,7 +807,7 @@ static void generateReproducer(CmpInst *Cond, Module *M,
         continue;
 
       auto *I = dyn_cast<Instruction>(V);
-      if (Value2Index.find(V) != Value2Index.end() || !I ||
+      if (Value2Index.contains(V) || !I ||
           !isa<CmpInst, BinaryOperator, GetElementPtrInst, CastInst>(V)) {
         Old2New[V] = V;
         Args.push_back(V);
@@ -849,7 +857,7 @@ static void generateReproducer(CmpInst *Cond, Module *M,
         continue;
 
       auto *I = dyn_cast<Instruction>(V);
-      if (Value2Index.find(V) == Value2Index.end() && I) {
+      if (!Value2Index.contains(V) && I) {
         Old2New[V] = nullptr;
         ToClone.push_back(I);
         append_range(WorkList, I->operands());
@@ -975,7 +983,7 @@ void ConstraintInfo::addFact(CmpInst::Predicate Pred, Value *A, Value *B,
   if (!R.isValid(*this))
     return;
 
-  LLVM_DEBUG(dbgs() << "Adding '" << CmpInst::getPredicateName(Pred) << " ";
+  LLVM_DEBUG(dbgs() << "Adding '" << Pred << " ";
              A->printAsOperand(dbgs(), false); dbgs() << ", ";
              B->printAsOperand(dbgs(), false); dbgs() << "'\n");
   bool Added = false;
diff --git a/llvm/lib/Transforms/Scalar/CorrelatedValuePropagation.cpp b/llvm/lib/Transforms/Scalar/CorrelatedValuePropagation.cpp
index a6293ee671237..1323d724a97c2 100644
--- a/llvm/lib/Transforms/Scalar/CorrelatedValuePropagation.cpp
+++ b/llvm/lib/Transforms/Scalar/CorrelatedValuePropagation.cpp
@@ -97,43 +97,6 @@ STATISTIC(NumMinMax, "Number of llvm.[us]{min,max} intrinsics removed");
 STATISTIC(NumUDivURemsNarrowedExpanded,
           "Number of bound udiv's/urem's expanded");
 
-namespace {
-
-  class CorrelatedValuePropagation : public FunctionPass {
-  public:
-    static char ID;
-
-    CorrelatedValuePropagation(): FunctionPass(ID) {
-     initializeCorrelatedValuePropagationPass(*PassRegistry::getPassRegistry());
-    }
-
-    bool runOnFunction(Function &F) override;
-
-    void getAnalysisUsage(AnalysisUsage &AU) const override {
-      AU.addRequired<DominatorTreeWrapperPass>();
-      AU.addRequired<LazyValueInfoWrapperPass>();
-      AU.addPreserved<GlobalsAAWrapperPass>();
-      AU.addPreserved<DominatorTreeWrapperPass>();
-      AU.addPreserved<LazyValueInfoWrapperPass>();
-    }
-  };
-
-} // end anonymous namespace
-
-char CorrelatedValuePropagation::ID = 0;
-
-INITIALIZE_PASS_BEGIN(CorrelatedValuePropagation, "correlated-propagation",
-                "Value Propagation", false, false)
-INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(LazyValueInfoWrapperPass)
-INITIALIZE_PASS_END(CorrelatedValuePropagation, "correlated-propagation",
-                "Value Propagation", false, false)
-
-// Public interface to the Value Propagation pass
-Pass *llvm::createCorrelatedValuePropagationPass() {
-  return new CorrelatedValuePropagation();
-}
-
 static bool processSelect(SelectInst *S, LazyValueInfo *LVI) {
   if (S->getType()->isVectorTy()) return false;
   if (isa<Constant>(S->getCondition())) return false;
@@ -1221,16 +1184,6 @@ static bool runImpl(Function &F, LazyValueInfo *LVI, DominatorTree *DT,
   return FnChanged;
 }
 
-bool CorrelatedValuePropagation::runOnFunction(Function &F) {
-  if (skipFunction(F))
-    return false;
-
-  LazyValueInfo *LVI = &getAnalysis<LazyValueInfoWrapperPass>().getLVI();
-  DominatorTree *DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
-
-  return runImpl(F, LVI, DT, getBestSimplifyQuery(*this, F));
-}
-
 PreservedAnalyses
 CorrelatedValuePropagationPass::run(Function &F, FunctionAnalysisManager &AM) {
   LazyValueInfo *LVI = &AM.getResult<LazyValueAnalysis>(F);
diff --git a/llvm/lib/Transforms/Scalar/DFAJumpThreading.cpp b/llvm/lib/Transforms/Scalar/DFAJumpThreading.cpp
index 658d0fcb53fa5..ede98fc69e649 100644
--- a/llvm/lib/Transforms/Scalar/DFAJumpThreading.cpp
+++ b/llvm/lib/Transforms/Scalar/DFAJumpThreading.cpp
@@ -168,51 +168,8 @@ class DFAJumpThreading {
   OptimizationRemarkEmitter *ORE;
 };
 
-class DFAJumpThreadingLegacyPass : public FunctionPass {
-public:
-  static char ID; // Pass identification
-  DFAJumpThreadingLegacyPass() : FunctionPass(ID) {}
-
-  void getAnalysisUsage(AnalysisUsage &AU) const override {
-    AU.addRequired<AssumptionCacheTracker>();
-    AU.addRequired<DominatorTreeWrapperPass>();
-    AU.addPreserved<DominatorTreeWrapperPass>();
-    AU.addRequired<TargetTransformInfoWrapperPass>();
-    AU.addRequired<OptimizationRemarkEmitterWrapperPass>();
-  }
-
-  bool runOnFunction(Function &F) override {
-    if (skipFunction(F))
-      return false;
-
-    AssumptionCache *AC =
-        &getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
-    DominatorTree *DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
-    TargetTransformInfo *TTI =
-        &getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
-    OptimizationRemarkEmitter *ORE =
-        &getAnalysis<OptimizationRemarkEmitterWrapperPass>().getORE();
-
-    return DFAJumpThreading(AC, DT, TTI, ORE).run(F);
-  }
-};
 } // end anonymous namespace
 
-char DFAJumpThreadingLegacyPass::ID = 0;
-INITIALIZE_PASS_BEGIN(DFAJumpThreadingLegacyPass, "dfa-jump-threading",
-                      "DFA Jump Threading", false, false)
-INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
-INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(OptimizationRemarkEmitterWrapperPass)
-INITIALIZE_PASS_END(DFAJumpThreadingLegacyPass, "dfa-jump-threading",
-                    "DFA Jump Threading", false, false)
-
-// Public interface to the DFA Jump Threading pass
-FunctionPass *llvm::createDFAJumpThreadingPass() {
-  return new DFAJumpThreadingLegacyPass();
-}
-
 namespace {
 
 /// Create a new basic block and sink \p SIToSink into it.
diff --git a/llvm/lib/Transforms/Scalar/DeadStoreElimination.cpp b/llvm/lib/Transforms/Scalar/DeadStoreElimination.cpp
index d3757341ad20e..9a7ad8d49222c 100644
--- a/llvm/lib/Transforms/Scalar/DeadStoreElimination.cpp
+++ b/llvm/lib/Transforms/Scalar/DeadStoreElimination.cpp
@@ -2240,79 +2240,3 @@ PreservedAnalyses DSEPass::run(Function &F, FunctionAnalysisManager &AM) {
   PA.preserve<LoopAnalysis>();
   return PA;
 }
-
-namespace {
-
-/// A legacy pass for the legacy pass manager that wraps \c DSEPass.
-class DSELegacyPass : public FunctionPass {
-public:
-  static char ID; // Pass identification, replacement for typeid
-
-  DSELegacyPass() : FunctionPass(ID) {
-    initializeDSELegacyPassPass(*PassRegistry::getPassRegistry());
-  }
-
-  bool runOnFunction(Function &F) override {
-    if (skipFunction(F))
-      return false;
-
-    AliasAnalysis &AA = getAnalysis<AAResultsWrapperPass>().getAAResults();
-    DominatorTree &DT = getAnalysis<DominatorTreeWrapperPass>().getDomTree();
-    const TargetLibraryInfo &TLI =
-        getAnalysis<TargetLibraryInfoWrapperPass>().getTLI(F);
-    MemorySSA &MSSA = getAnalysis<MemorySSAWrapperPass>().getMSSA();
-    PostDominatorTree &PDT =
-        getAnalysis<PostDominatorTreeWrapperPass>().getPostDomTree();
-    AssumptionCache &AC =
-        getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
-    LoopInfo &LI = getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
-
-    bool Changed = eliminateDeadStores(F, AA, MSSA, DT, PDT, AC, TLI, LI);
-
-#ifdef LLVM_ENABLE_STATS
-    if (AreStatisticsEnabled())
-      for (auto &I : instructions(F))
-        NumRemainingStores += isa<StoreInst>(&I);
-#endif
-
-    return Changed;
-  }
-
-  void getAnalysisUsage(AnalysisUsage &AU) const override {
-    AU.setPreservesCFG();
-    AU.addRequired<AAResultsWrapperPass>();
-    AU.addRequired<TargetLibraryInfoWrapperPass>();
-    AU.addPreserved<GlobalsAAWrapperPass>();
-    AU.addRequired<DominatorTreeWrapperPass>();
-    AU.addPreserved<DominatorTreeWrapperPass>();
-    AU.addRequired<PostDominatorTreeWrapperPass>();
-    AU.addRequired<MemorySSAWrapperPass>();
-    AU.addPreserved<PostDominatorTreeWrapperPass>();
-    AU.addPreserved<MemorySSAWrapperPass>();
-    AU.addRequired<LoopInfoWrapperPass>();
-    AU.addPreserved<LoopInfoWrapperPass>();
-    AU.addRequired<AssumptionCacheTracker>();
-  }
-};
-
-} // end anonymous namespace
-
-char DSELegacyPass::ID = 0;
-
-INITIALIZE_PASS_BEGIN(DSELegacyPass, "dse", "Dead Store Elimination", false,
-                      false)
-INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(PostDominatorTreeWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(AAResultsWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(GlobalsAAWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(MemorySSAWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(MemoryDependenceWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(LoopInfoWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
-INITIALIZE_PASS_END(DSELegacyPass, "dse", "Dead Store Elimination", false,
-                    false)
-
-FunctionPass *llvm::createDeadStoreEliminationPass() {
-  return new DSELegacyPass();
-}
diff --git a/llvm/lib/Transforms/Scalar/Float2Int.cpp b/llvm/lib/Transforms/Scalar/Float2Int.cpp
index 7d27b783f1372..35123606a15dd 100644
--- a/llvm/lib/Transforms/Scalar/Float2Int.cpp
+++ b/llvm/lib/Transforms/Scalar/Float2Int.cpp
@@ -49,35 +49,6 @@ MaxIntegerBW("float2int-max-integer-bw", cl::init(64), cl::Hidden,
              cl::desc("Max integer bitwidth to consider in float2int"
                       "(default=64)"));
 
-namespace {
-  struct Float2IntLegacyPass : public FunctionPass {
-    static char ID; // Pass identification, replacement for typeid
-    Float2IntLegacyPass() : FunctionPass(ID) {
-      initializeFloat2IntLegacyPassPass(*PassRegistry::getPassRegistry());
-    }
-
-    bool runOnFunction(Function &F) override {
-      if (skipFunction(F))
-        return false;
-
-      const DominatorTree &DT = getAnalysis<DominatorTreeWrapperPass>().getDomTree();
-      return Impl.runImpl(F, DT);
-    }
-
-    void getAnalysisUsage(AnalysisUsage &AU) const override {
-      AU.setPreservesCFG();
-      AU.addRequired<DominatorTreeWrapperPass>();
-      AU.addPreserved<GlobalsAAWrapperPass>();
-    }
-
-  private:
-    Float2IntPass Impl;
-  };
-}
-
-char Float2IntLegacyPass::ID = 0;
-INITIALIZE_PASS(Float2IntLegacyPass, "float2int", "Float to int", false, false)
-
 // Given a FCmp predicate, return a matching ICmp predicate if one
 // exists, otherwise return BAD_ICMP_PREDICATE.
 static CmpInst::Predicate mapFCmpPred(CmpInst::Predicate P) {
@@ -187,7 +158,7 @@ void Float2IntPass::walkBackwards() {
     Instruction *I = Worklist.back();
     Worklist.pop_back();
 
-    if (SeenInsts.find(I) != SeenInsts.end())
+    if (SeenInsts.contains(I))
       // Seen already.
       continue;
 
@@ -371,7 +342,7 @@ bool Float2IntPass::validateAndTransform() {
           ConvertedToTy = I->getType();
         for (User *U : I->users()) {
           Instruction *UI = dyn_cast<Instruction>(U);
-          if (!UI || SeenInsts.find(UI) == SeenInsts.end()) {
+          if (!UI || !SeenInsts.contains(UI)) {
             LLVM_DEBUG(dbgs() << "F2I: Failing because of " << *U << "\n");
             Fail = true;
             break;
@@ -428,7 +399,7 @@ bool Float2IntPass::validateAndTransform() {
 }
 
 Value *Float2IntPass::convert(Instruction *I, Type *ToTy) {
-  if (ConvertedInsts.find(I) != ConvertedInsts.end())
+  if (ConvertedInsts.contains(I))
     // Already converted this instruction.
     return ConvertedInsts[I];
 
@@ -529,9 +500,6 @@ bool Float2IntPass::runImpl(Function &F, const DominatorTree &DT) {
   return Modified;
 }
 
-namespace llvm {
-FunctionPass *createFloat2IntPass() { return new Float2IntLegacyPass(); }
-
 PreservedAnalyses Float2IntPass::run(Function &F, FunctionAnalysisManager &AM) {
   const DominatorTree &DT = AM.getResult<DominatorTreeAnalysis>(F);
   if (!runImpl(F, DT))
@@ -541,4 +509,3 @@ PreservedAnalyses Float2IntPass::run(Function &F, FunctionAnalysisManager &AM) {
   PA.preserveSet<CFGAnalyses>();
   return PA;
 }
-} // End namespace llvm
diff --git a/llvm/lib/Transforms/Scalar/GVNHoist.cpp b/llvm/lib/Transforms/Scalar/GVNHoist.cpp
index bbff497b7d92d..4e290015bc5e7 100644
--- a/llvm/lib/Transforms/Scalar/GVNHoist.cpp
+++ b/llvm/lib/Transforms/Scalar/GVNHoist.cpp
@@ -519,39 +519,6 @@ class GVNHoist {
   std::pair<unsigned, unsigned> hoistExpressions(Function &F);
 };
 
-class GVNHoistLegacyPass : public FunctionPass {
-public:
-  static char ID;
-
-  GVNHoistLegacyPass() : FunctionPass(ID) {
-    initializeGVNHoistLegacyPassPass(*PassRegistry::getPassRegistry());
-  }
-
-  bool runOnFunction(Function &F) override {
-    if (skipFunction(F))
-      return false;
-    auto &DT = getAnalysis<DominatorTreeWrapperPass>().getDomTree();
-    auto &PDT = getAnalysis<PostDominatorTreeWrapperPass>().getPostDomTree();
-    auto &AA = getAnalysis<AAResultsWrapperPass>().getAAResults();
-    auto &MD = getAnalysis<MemoryDependenceWrapperPass>().getMemDep();
-    auto &MSSA = getAnalysis<MemorySSAWrapperPass>().getMSSA();
-
-    GVNHoist G(&DT, &PDT, &AA, &MD, &MSSA);
-    return G.run(F);
-  }
-
-  void getAnalysisUsage(AnalysisUsage &AU) const override {
-    AU.addRequired<DominatorTreeWrapperPass>();
-    AU.addRequired<PostDominatorTreeWrapperPass>();
-    AU.addRequired<AAResultsWrapperPass>();
-    AU.addRequired<MemoryDependenceWrapperPass>();
-    AU.addRequired<MemorySSAWrapperPass>();
-    AU.addPreserved<DominatorTreeWrapperPass>();
-    AU.addPreserved<MemorySSAWrapperPass>();
-    AU.addPreserved<GlobalsAAWrapperPass>();
-  }
-};
-
 bool GVNHoist::run(Function &F) {
   NumFuncArgs = F.arg_size();
   VN.setDomTree(DT);
@@ -808,15 +775,20 @@ bool GVNHoist::valueAnticipable(CHIArgs C, Instruction *TI) const {
 void GVNHoist::checkSafety(CHIArgs C, BasicBlock *BB, GVNHoist::InsKind K,
                            SmallVectorImpl<CHIArg> &Safe) {
   int NumBBsOnAllPaths = MaxNumberOfBBSInPath;
+  const Instruction *T = BB->getTerminator();
   for (auto CHI : C) {
     Instruction *Insn = CHI.I;
     if (!Insn) // No instruction was inserted in this CHI.
       continue;
+    // If the Terminator is some kind of "exotic terminator" that produces a
+    // value (such as InvokeInst, CallBrInst, or CatchSwitchInst) which the CHI
+    // uses, it is not safe to hoist the use above the def.
+    if (!T->use_empty() && is_contained(Insn->operands(), cast<const Value>(T)))
+      continue;
     if (K == InsKind::Scalar) {
       if (safeToHoistScalar(BB, Insn->getParent(), NumBBsOnAllPaths))
         Safe.push_back(CHI);
     } else {
-      auto *T = BB->getTerminator();
       if (MemoryUseOrDef *UD = MSSA->getMemoryAccess(Insn))
         if (safeToHoistLdSt(T, Insn, UD, K, NumBBsOnAllPaths))
           Safe.push_back(CHI);
@@ -1251,17 +1223,3 @@ PreservedAnalyses GVNHoistPass::run(Function &F, FunctionAnalysisManager &AM) {
   PA.preserve<MemorySSAAnalysis>();
   return PA;
 }
-
-char GVNHoistLegacyPass::ID = 0;
-
-INITIALIZE_PASS_BEGIN(GVNHoistLegacyPass, "gvn-hoist",
-                      "Early GVN Hoisting of Expressions", false, false)
-INITIALIZE_PASS_DEPENDENCY(MemoryDependenceWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(MemorySSAWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(PostDominatorTreeWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(AAResultsWrapperPass)
-INITIALIZE_PASS_END(GVNHoistLegacyPass, "gvn-hoist",
-                    "Early GVN Hoisting of Expressions", false, false)
-
-FunctionPass *llvm::createGVNHoistPass() { return new GVNHoistLegacyPass(); }
diff --git a/llvm/lib/Transforms/Scalar/GVNSink.cpp b/llvm/lib/Transforms/Scalar/GVNSink.cpp
index 5fb8a77051fb9..d0933ec418acf 100644
--- a/llvm/lib/Transforms/Scalar/GVNSink.cpp
+++ b/llvm/lib/Transforms/Scalar/GVNSink.cpp
@@ -154,7 +154,7 @@ class LockstepReverseIterator {
 
   void restrictToBlocks(SmallSetVector<BasicBlock *, 4> &Blocks) {
     for (auto II = Insts.begin(); II != Insts.end();) {
-      if (!llvm::is_contained(Blocks, (*II)->getParent())) {
+      if (!Blocks.contains((*II)->getParent())) {
         ActiveBlocks.remove((*II)->getParent());
         II = Insts.erase(II);
       } else {
@@ -272,7 +272,7 @@ class ModelledPHI {
     auto VI = Values.begin();
     while (BI != Blocks.end()) {
       assert(VI != Values.end());
-      if (!llvm::is_contained(NewBlocks, *BI)) {
+      if (!NewBlocks.contains(*BI)) {
         BI = Blocks.erase(BI);
         VI = Values.erase(VI);
       } else {
@@ -886,29 +886,6 @@ void GVNSink::sinkLastInstruction(ArrayRef<BasicBlock *> Blocks,
   NumRemoved += Insts.size() - 1;
 }
 
-////////////////////////////////////////////////////////////////////////////////
-// Pass machinery / boilerplate
-
-class GVNSinkLegacyPass : public FunctionPass {
-public:
-  static char ID;
-
-  GVNSinkLegacyPass() : FunctionPass(ID) {
-    initializeGVNSinkLegacyPassPass(*PassRegistry::getPassRegistry());
-  }
-
-  bool runOnFunction(Function &F) override {
-    if (skipFunction(F))
-      return false;
-    GVNSink G;
-    return G.run(F);
-  }
-
-  void getAnalysisUsage(AnalysisUsage &AU) const override {
-    AU.addPreserved<GlobalsAAWrapperPass>();
-  }
-};
-
 } // end anonymous namespace
 
 PreservedAnalyses GVNSinkPass::run(Function &F, FunctionAnalysisManager &AM) {
@@ -917,14 +894,3 @@ PreservedAnalyses GVNSinkPass::run(Function &F, FunctionAnalysisManager &AM) {
     return PreservedAnalyses::all();
   return PreservedAnalyses::none();
 }
-
-char GVNSinkLegacyPass::ID = 0;
-
-INITIALIZE_PASS_BEGIN(GVNSinkLegacyPass, "gvn-sink",
-                      "Early GVN sinking of Expressions", false, false)
-INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(PostDominatorTreeWrapperPass)
-INITIALIZE_PASS_END(GVNSinkLegacyPass, "gvn-sink",
-                    "Early GVN sinking of Expressions", false, false)
-
-FunctionPass *llvm::createGVNSinkPass() { return new GVNSinkLegacyPass(); }
diff --git a/llvm/lib/Transforms/Scalar/GuardWidening.cpp b/llvm/lib/Transforms/Scalar/GuardWidening.cpp
index 064c7b1d0ad12..e354b70357cd9 100644
--- a/llvm/lib/Transforms/Scalar/GuardWidening.cpp
+++ b/llvm/lib/Transforms/Scalar/GuardWidening.cpp
@@ -460,27 +460,55 @@ GuardWideningImpl::computeWideningScore(Instruction *DominatedInstr,
   if (HoistingOutOfLoop)
     return WS_Positive;
 
-  // Returns true if we might be hoisting above explicit control flow.  Note
-  // that this completely ignores implicit control flow (guards, calls which
-  // throw, etc...).  That choice appears arbitrary.
-  auto MaybeHoistingOutOfIf = [&]() {
-    auto *DominatingBlock = DominatingGuard->getParent();
-    auto *DominatedBlock = DominatedInstr->getParent();
-    if (isGuardAsWidenableBranch(DominatingGuard))
-      DominatingBlock = cast<BranchInst>(DominatingGuard)->getSuccessor(0);
+  // For a given basic block \p BB, return its successor which is guaranteed or
+  // highly likely will be taken as its successor.
+  auto GetLikelySuccessor = [](const BasicBlock * BB)->const BasicBlock * {
+    if (auto *UniqueSucc = BB->getUniqueSuccessor())
+      return UniqueSucc;
+    auto *Term = BB->getTerminator();
+    Value *Cond = nullptr;
+    const BasicBlock *IfTrue = nullptr, *IfFalse = nullptr;
+    using namespace PatternMatch;
+    if (!match(Term, m_Br(m_Value(Cond), m_BasicBlock(IfTrue),
+                          m_BasicBlock(IfFalse))))
+      return nullptr;
+    // For constant conditions, only one dynamical successor is possible
+    if (auto *ConstCond = dyn_cast<ConstantInt>(Cond))
+      return ConstCond->isAllOnesValue() ? IfTrue : IfFalse;
+    // If one of successors ends with deopt, another one is likely.
+    if (IfFalse->getPostdominatingDeoptimizeCall())
+      return IfTrue;
+    if (IfTrue->getPostdominatingDeoptimizeCall())
+      return IfFalse;
+    // TODO: Use branch frequency metatada to allow hoisting through non-deopt
+    // branches?
+    return nullptr;
+  };
+
+  // Returns true if we might be hoisting above explicit control flow into a
+  // considerably hotter block.  Note that this completely ignores implicit
+  // control flow (guards, calls which throw, etc...).  That choice appears
+  // arbitrary (we assume that implicit control flow exits are all rare).
+  auto MaybeHoistingToHotterBlock = [&]() {
+    const auto *DominatingBlock = DominatingGuard->getParent();
+    const auto *DominatedBlock = DominatedInstr->getParent();
+
+    // Descent as low as we can, always taking the likely successor.
+    while (DominatedBlock != DominatingBlock)
+      if (auto *LikelySucc = GetLikelySuccessor(DominatingBlock))
+        DominatingBlock = LikelySucc;
+      else
+        break;
 
     // Same Block?
     if (DominatedBlock == DominatingBlock)
       return false;
-    // Obvious successor (common loop header/preheader case)
-    if (DominatedBlock == DominatingBlock->getUniqueSuccessor())
-      return false;
     // TODO: diamond, triangle cases
     if (!PDT) return true;
     return !PDT->dominates(DominatedBlock, DominatingBlock);
   };
 
-  return MaybeHoistingOutOfIf() ? WS_IllegalOrNegative : WS_Neutral;
+  return MaybeHoistingToHotterBlock() ? WS_IllegalOrNegative : WS_Neutral;
 }
 
 bool GuardWideningImpl::canBeHoistedTo(
diff --git a/llvm/lib/Transforms/Scalar/IndVarSimplify.cpp b/llvm/lib/Transforms/Scalar/IndVarSimplify.cpp
index 0c21ae26297d4..1f79db79b0347 100644
--- a/llvm/lib/Transforms/Scalar/IndVarSimplify.cpp
+++ b/llvm/lib/Transforms/Scalar/IndVarSimplify.cpp
@@ -93,15 +93,6 @@ STATISTIC(NumLFTR        , "Number of loop exit tests replaced");
 STATISTIC(NumElimExt     , "Number of IV sign/zero extends eliminated");
 STATISTIC(NumElimIV      , "Number of congruent IVs eliminated");
 
-// Trip count verification can be enabled by default under NDEBUG if we
-// implement a strong expression equivalence checker in SCEV. Until then, we
-// use the verify-indvars flag, which may assert in some cases.
-static cl::opt<bool> VerifyIndvars(
-    "verify-indvars", cl::Hidden,
-    cl::desc("Verify the ScalarEvolution result after running indvars. Has no "
-             "effect in release builds. (Note: this adds additional SCEV "
-             "queries potentially changing the analysis result)"));
-
 static cl::opt<ReplaceExitVal> ReplaceExitValue(
     "replexitval", cl::Hidden, cl::init(OnlyCheapRepl),
     cl::desc("Choose the strategy to replace exit value in IndVarSimplify"),
@@ -2023,16 +2014,6 @@ bool IndVarSimplify::run(Loop *L) {
   if (!L->isLoopSimplifyForm())
     return false;
 
-#ifndef NDEBUG
-  // Used below for a consistency check only
-  // Note: Since the result returned by ScalarEvolution may depend on the order
-  // in which previous results are added to its cache, the call to
-  // getBackedgeTakenCount() may change following SCEV queries.
-  const SCEV *BackedgeTakenCount;
-  if (VerifyIndvars)
-    BackedgeTakenCount = SE->getBackedgeTakenCount(L);
-#endif
-
   bool Changed = false;
   // If there are any floating-point recurrences, attempt to
   // transform them to use integer recurrences.
@@ -2181,27 +2162,8 @@ bool IndVarSimplify::run(Loop *L) {
   // Check a post-condition.
   assert(L->isRecursivelyLCSSAForm(*DT, *LI) &&
          "Indvars did not preserve LCSSA!");
-
-  // Verify that LFTR, and any other change have not interfered with SCEV's
-  // ability to compute trip count.  We may have *changed* the exit count, but
-  // only by reducing it.
-#ifndef NDEBUG
-  if (VerifyIndvars && !isa<SCEVCouldNotCompute>(BackedgeTakenCount)) {
-    SE->forgetLoop(L);
-    const SCEV *NewBECount = SE->getBackedgeTakenCount(L);
-    if (SE->getTypeSizeInBits(BackedgeTakenCount->getType()) <
-        SE->getTypeSizeInBits(NewBECount->getType()))
-      NewBECount = SE->getTruncateOrNoop(NewBECount,
-                                         BackedgeTakenCount->getType());
-    else
-      BackedgeTakenCount = SE->getTruncateOrNoop(BackedgeTakenCount,
-                                                 NewBECount->getType());
-    assert(!SE->isKnownPredicate(ICmpInst::ICMP_ULT, BackedgeTakenCount,
-                                 NewBECount) && "indvars must preserve SCEV");
-  }
   if (VerifyMemorySSA && MSSAU)
     MSSAU->getMemorySSA()->verifyMemorySSA();
-#endif
 
   return Changed;
 }
@@ -2223,54 +2185,3 @@ PreservedAnalyses IndVarSimplifyPass::run(Loop &L, LoopAnalysisManager &AM,
     PA.preserve<MemorySSAAnalysis>();
   return PA;
 }
-
-namespace {
-
-struct IndVarSimplifyLegacyPass : public LoopPass {
-  static char ID; // Pass identification, replacement for typeid
-
-  IndVarSimplifyLegacyPass() : LoopPass(ID) {
-    initializeIndVarSimplifyLegacyPassPass(*PassRegistry::getPassRegistry());
-  }
-
-  bool runOnLoop(Loop *L, LPPassManager &LPM) override {
-    if (skipLoop(L))
-      return false;
-
-    auto *LI = &getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
-    auto *SE = &getAnalysis<ScalarEvolutionWrapperPass>().getSE();
-    auto *DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
-    auto *TLIP = getAnalysisIfAvailable<TargetLibraryInfoWrapperPass>();
-    auto *TLI = TLIP ? &TLIP->getTLI(*L->getHeader()->getParent()) : nullptr;
-    auto *TTIP = getAnalysisIfAvailable<TargetTransformInfoWrapperPass>();
-    auto *TTI = TTIP ? &TTIP->getTTI(*L->getHeader()->getParent()) : nullptr;
-    const DataLayout &DL = L->getHeader()->getModule()->getDataLayout();
-    auto *MSSAAnalysis = getAnalysisIfAvailable<MemorySSAWrapperPass>();
-    MemorySSA *MSSA = nullptr;
-    if (MSSAAnalysis)
-      MSSA = &MSSAAnalysis->getMSSA();
-
-    IndVarSimplify IVS(LI, SE, DT, DL, TLI, TTI, MSSA, AllowIVWidening);
-    return IVS.run(L);
-  }
-
-  void getAnalysisUsage(AnalysisUsage &AU) const override {
-    AU.setPreservesCFG();
-    AU.addPreserved<MemorySSAWrapperPass>();
-    getLoopAnalysisUsage(AU);
-  }
-};
-
-} // end anonymous namespace
-
-char IndVarSimplifyLegacyPass::ID = 0;
-
-INITIALIZE_PASS_BEGIN(IndVarSimplifyLegacyPass, "indvars",
-                      "Induction Variable Simplification", false, false)
-INITIALIZE_PASS_DEPENDENCY(LoopPass)
-INITIALIZE_PASS_END(IndVarSimplifyLegacyPass, "indvars",
-                    "Induction Variable Simplification", false, false)
-
-Pass *llvm::createIndVarSimplifyPass() {
-  return new IndVarSimplifyLegacyPass();
-}
diff --git a/llvm/lib/Transforms/Scalar/InductiveRangeCheckElimination.cpp b/llvm/lib/Transforms/Scalar/InductiveRangeCheckElimination.cpp
index 52a4bc8a9f24a..463ea1d4e3197 100644
--- a/llvm/lib/Transforms/Scalar/InductiveRangeCheckElimination.cpp
+++ b/llvm/lib/Transforms/Scalar/InductiveRangeCheckElimination.cpp
@@ -671,8 +671,7 @@ static bool isSafeDecreasingBound(const SCEV *Start,
   LLVM_DEBUG(dbgs() << "irce: Start: " << *Start << "\n");
   LLVM_DEBUG(dbgs() << "irce: Step: " << *Step << "\n");
   LLVM_DEBUG(dbgs() << "irce: BoundSCEV: " << *BoundSCEV << "\n");
-  LLVM_DEBUG(dbgs() << "irce: Pred: " << ICmpInst::getPredicateName(Pred)
-                    << "\n");
+  LLVM_DEBUG(dbgs() << "irce: Pred: " << Pred << "\n");
   LLVM_DEBUG(dbgs() << "irce: LatchExitBrIdx: " << LatchBrExitIdx << "\n");
 
   bool IsSigned = ICmpInst::isSigned(Pred);
@@ -719,8 +718,7 @@ static bool isSafeIncreasingBound(const SCEV *Start,
   LLVM_DEBUG(dbgs() << "irce: Start: " << *Start << "\n");
   LLVM_DEBUG(dbgs() << "irce: Step: " << *Step << "\n");
   LLVM_DEBUG(dbgs() << "irce: BoundSCEV: " << *BoundSCEV << "\n");
-  LLVM_DEBUG(dbgs() << "irce: Pred: " << ICmpInst::getPredicateName(Pred)
-                    << "\n");
+  LLVM_DEBUG(dbgs() << "irce: Pred: " << Pred << "\n");
   LLVM_DEBUG(dbgs() << "irce: LatchExitBrIdx: " << LatchBrExitIdx << "\n");
 
   bool IsSigned = ICmpInst::isSigned(Pred);
diff --git a/llvm/lib/Transforms/Scalar/InferAddressSpaces.cpp b/llvm/lib/Transforms/Scalar/InferAddressSpaces.cpp
index 114738a35fd1e..b6713730bfa9d 100644
--- a/llvm/lib/Transforms/Scalar/InferAddressSpaces.cpp
+++ b/llvm/lib/Transforms/Scalar/InferAddressSpaces.cpp
@@ -694,18 +694,6 @@ static Value *cloneConstantExprWithNewAddressSpace(
     return ConstantExpr::getAddrSpaceCast(CE, TargetType);
   }
 
-  if (CE->getOpcode() == Instruction::Select) {
-    Constant *Src0 = CE->getOperand(1);
-    Constant *Src1 = CE->getOperand(2);
-    if (Src0->getType()->getPointerAddressSpace() ==
-        Src1->getType()->getPointerAddressSpace()) {
-
-      return ConstantExpr::getSelect(
-          CE->getOperand(0), ConstantExpr::getAddrSpaceCast(Src0, TargetType),
-          ConstantExpr::getAddrSpaceCast(Src1, TargetType));
-    }
-  }
-
   if (CE->getOpcode() == Instruction::IntToPtr) {
     assert(isNoopPtrIntCastPair(cast<Operator>(CE), *DL, TTI));
     Constant *Src = cast<ConstantExpr>(CE->getOperand(0))->getOperand(0);
diff --git a/llvm/lib/Transforms/Scalar/JumpThreading.cpp b/llvm/lib/Transforms/Scalar/JumpThreading.cpp
index eac41f05ca14f..35daa8f3874d1 100644
--- a/llvm/lib/Transforms/Scalar/JumpThreading.cpp
+++ b/llvm/lib/Transforms/Scalar/JumpThreading.cpp
@@ -115,64 +115,6 @@ static cl::opt<bool> ThreadAcrossLoopHeaders(
     cl::desc("Allow JumpThreading to thread across loop headers, for testing"),
     cl::init(false), cl::Hidden);
 
-
-namespace {
-
-  /// This pass performs 'jump threading', which looks at blocks that have
-  /// multiple predecessors and multiple successors.  If one or more of the
-  /// predecessors of the block can be proven to always jump to one of the
-  /// successors, we forward the edge from the predecessor to the successor by
-  /// duplicating the contents of this block.
-  ///
-  /// An example of when this can occur is code like this:
-  ///
-  ///   if () { ...
-  ///     X = 4;
-  ///   }
-  ///   if (X < 3) {
-  ///
-  /// In this case, the unconditional branch at the end of the first if can be
-  /// revectored to the false side of the second if.
-  class JumpThreading : public FunctionPass {
-  public:
-    static char ID; // Pass identification
-
-    JumpThreading(int T = -1) : FunctionPass(ID) {
-      initializeJumpThreadingPass(*PassRegistry::getPassRegistry());
-    }
-
-    bool runOnFunction(Function &F) override;
-
-    void getAnalysisUsage(AnalysisUsage &AU) const override {
-      AU.addRequired<DominatorTreeWrapperPass>();
-      AU.addPreserved<DominatorTreeWrapperPass>();
-      AU.addRequired<AAResultsWrapperPass>();
-      AU.addRequired<LazyValueInfoWrapperPass>();
-      AU.addPreserved<LazyValueInfoWrapperPass>();
-      AU.addPreserved<GlobalsAAWrapperPass>();
-      AU.addRequired<TargetLibraryInfoWrapperPass>();
-      AU.addRequired<TargetTransformInfoWrapperPass>();
-    }
-  };
-
-} // end anonymous namespace
-
-char JumpThreading::ID = 0;
-
-INITIALIZE_PASS_BEGIN(JumpThreading, "jump-threading",
-                "Jump Threading", false, false)
-INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(LazyValueInfoWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(AAResultsWrapperPass)
-INITIALIZE_PASS_END(JumpThreading, "jump-threading",
-                "Jump Threading", false, false)
-
-// Public interface to the Jump Threading pass
-FunctionPass *llvm::createJumpThreadingPass(int Threshold) {
-  return new JumpThreading(Threshold);
-}
-
 JumpThreadingPass::JumpThreadingPass(int T) {
   DefaultBBDupThreshold = (T == -1) ? BBDuplicateThreshold : unsigned(T);
 }
@@ -303,35 +245,6 @@ static void updatePredecessorProfileMetadata(PHINode *PN, BasicBlock *BB) {
   }
 }
 
-/// runOnFunction - Toplevel algorithm.
-bool JumpThreading::runOnFunction(Function &F) {
-  if (skipFunction(F))
-    return false;
-  auto TTI = &getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
-  // Jump Threading has no sense for the targets with divergent CF
-  if (TTI->hasBranchDivergence())
-    return false;
-  auto TLI = &getAnalysis<TargetLibraryInfoWrapperPass>().getTLI(F);
-  auto DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
-  auto LVI = &getAnalysis<LazyValueInfoWrapperPass>().getLVI();
-  auto AA = &getAnalysis<AAResultsWrapperPass>().getAAResults();
-
-  LoopInfo LI{*DT};
-  auto BPI = std::make_unique<BranchProbabilityInfo>(F, LI, TLI);
-  auto BFI = std::make_unique<BlockFrequencyInfo>(F, *BPI, LI);
-
-  JumpThreadingPass Impl;
-  bool Changed = Impl.runImpl(F, nullptr, TLI, TTI, LVI, AA,
-                              std::make_unique<DomTreeUpdater>(
-                                  DT, DomTreeUpdater::UpdateStrategy::Lazy),
-                              BFI.get(), BPI.get());
-  if (PrintLVIAfterJumpThreading) {
-    dbgs() << "LVI for function '" << F.getName() << "':\n";
-    LVI->printLVI(F, Impl.getDomTreeUpdater()->getDomTree(), dbgs());
-  }
-  return Changed;
-}
-
 PreservedAnalyses JumpThreadingPass::run(Function &F,
                                          FunctionAnalysisManager &AM) {
   auto &TTI = AM.getResult<TargetIRAnalysis>(F);
diff --git a/llvm/lib/Transforms/Scalar/LICM.cpp b/llvm/lib/Transforms/Scalar/LICM.cpp
index 376f20a5a6131..d74e8fb4fa24a 100644
--- a/llvm/lib/Transforms/Scalar/LICM.cpp
+++ b/llvm/lib/Transforms/Scalar/LICM.cpp
@@ -68,6 +68,7 @@
 #include "llvm/IR/Dominators.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/IR/Metadata.h"
 #include "llvm/IR/PatternMatch.h"
@@ -102,6 +103,8 @@ STATISTIC(NumMovedCalls, "Number of call insts hoisted or sunk");
 STATISTIC(NumPromotionCandidates, "Number of promotion candidates");
 STATISTIC(NumLoadPromoted, "Number of load-only promotions");
 STATISTIC(NumLoadStorePromoted, "Number of load and store promotions");
+STATISTIC(NumMinMaxHoisted,
+          "Number min/max expressions hoisted out of the loop");
 
 /// Memory promotion is enabled by default.
 static cl::opt<bool>
@@ -167,6 +170,11 @@ static bool pointerInvalidatedByLoop(MemorySSA *MSSA, MemoryUse *MU,
                                      bool InvariantGroup);
 static bool pointerInvalidatedByBlock(BasicBlock &BB, MemorySSA &MSSA,
                                       MemoryUse &MU);
+/// Try to simplify things like (A < INV_1 AND icmp A < INV_2) into (A <
+/// min(INV_1, INV_2)), if INV_1 and INV_2 are both loop invariants and their
+/// minimun can be computed outside of loop, and X is not a loop-invariant.
+static bool hoistMinMax(Instruction &I, Loop &L, ICFLoopSafetyInfo &SafetyInfo,
+                        MemorySSAUpdater &MSSAU);
 static Instruction *cloneInstructionInExitBlock(
     Instruction &I, BasicBlock &ExitBlock, PHINode &PN, const LoopInfo *LI,
     const LoopSafetyInfo *SafetyInfo, MemorySSAUpdater &MSSAU);
@@ -981,6 +989,15 @@ bool llvm::hoistRegion(DomTreeNode *N, AAResults *AA, LoopInfo *LI,
         }
       }
 
+      // Optimize complex patterns, such as (x < INV1 && x < INV2), turning them
+      // into (x < min(INV1, INV2)), and hoisting the invariant part of this
+      // expression out of the loop.
+      if (hoistMinMax(I, *CurLoop, *SafetyInfo, MSSAU)) {
+        ++NumMinMaxHoisted;
+        Changed = true;
+        continue;
+      }
+
       // Remember possibly hoistable branches so we can actually hoist them
       // later if needed.
       if (BranchInst *BI = dyn_cast<BranchInst>(&I))
@@ -1212,10 +1229,6 @@ bool llvm::canSinkOrHoistInst(Instruction &I, AAResults *AA, DominatorTree *DT,
       // Assumes don't actually alias anything or throw
       return true;
 
-    if (match(CI, m_Intrinsic<Intrinsic::experimental_widenable_condition>()))
-      // Widenable conditions don't actually alias anything or throw
-      return true;
-
     // Handle simple cases by querying alias analysis.
     MemoryEffects Behavior = AA->getMemoryEffects(CI);
     if (Behavior.doesNotAccessMemory())
@@ -2396,6 +2409,84 @@ bool pointerInvalidatedByBlock(BasicBlock &BB, MemorySSA &MSSA, MemoryUse &MU) {
   return false;
 }
 
+static bool hoistMinMax(Instruction &I, Loop &L, ICFLoopSafetyInfo &SafetyInfo,
+                        MemorySSAUpdater &MSSAU) {
+  bool Inverse = false;
+  bool IsLogical = false;
+  using namespace PatternMatch;
+  Value *Cond1, *Cond2;
+  if (match(&I, m_Or(m_Value(Cond1), m_Value(Cond2))))
+    Inverse = true;
+  else if (match(&I, m_LogicalOr(m_Value(Cond1), m_Value(Cond2)))) {
+    Inverse = true;
+    IsLogical = true;
+  } else if (match(&I, m_And(m_Value(Cond1), m_Value(Cond2)))) {
+    // Do nothing
+  } else if (match(&I, m_LogicalAnd(m_Value(Cond1), m_Value(Cond2))))
+    IsLogical = true;
+  else
+    return false;
+
+  auto MatchICmpAgainstInvariant = [&](Value *C, ICmpInst::Predicate &P,
+                                       Value *&LHS, Value *&RHS) {
+    if (!match(C, m_OneUse(m_ICmp(P, m_Value(LHS), m_Value(RHS)))))
+      return false;
+    if (!LHS->getType()->isIntegerTy())
+      return false;
+    if (!ICmpInst::isRelational(P))
+      return false;
+    if (L.isLoopInvariant(LHS)) {
+      std::swap(LHS, RHS);
+      P = ICmpInst::getSwappedPredicate(P);
+    }
+    if (L.isLoopInvariant(LHS) || !L.isLoopInvariant(RHS))
+      return false;
+    if (Inverse)
+      P = ICmpInst::getInversePredicate(P);
+    return true;
+  };
+  ICmpInst::Predicate P1, P2;
+  Value *LHS1, *LHS2, *RHS1, *RHS2;
+  if (!MatchICmpAgainstInvariant(Cond1, P1, LHS1, RHS1) ||
+      !MatchICmpAgainstInvariant(Cond2, P2, LHS2, RHS2))
+    return false;
+  if (P1 != P2 || LHS1 != LHS2)
+    return false;
+
+  // Everything is fine, we can do the transform.
+  bool UseMin = ICmpInst::isLT(P1) || ICmpInst::isLE(P1);
+  assert(
+      (UseMin || ICmpInst::isGT(P1) || ICmpInst::isGE(P1)) &&
+      "Relational predicate is either less (or equal) or greater (or equal)!");
+  Intrinsic::ID id = ICmpInst::isSigned(P1)
+                         ? (UseMin ? Intrinsic::smin : Intrinsic::smax)
+                         : (UseMin ? Intrinsic::umin : Intrinsic::umax);
+  auto *Preheader = L.getLoopPreheader();
+  assert(Preheader && "Loop is not in simplify form?");
+  IRBuilder<> Builder(Preheader->getTerminator());
+  // We are about to create a new guaranteed use for RHS2 which might not exist
+  // before (if it was a non-taken input of logical and/or instruction). If it
+  // was poison, we need to freeze it. Note that no new use for LHS and RHS1 are
+  // introduced, so they don't need this.
+  if (IsLogical)
+    RHS2 = Builder.CreateFreeze(RHS2, RHS2->getName() + ".fr");
+  Value *NewRHS = Builder.CreateBinaryIntrinsic(
+      id, RHS1, RHS2, nullptr, StringRef("invariant.") +
+                                   (ICmpInst::isSigned(P1) ? "s" : "u") +
+                                   (UseMin ? "min" : "max"));
+  Builder.SetInsertPoint(&I);
+  ICmpInst::Predicate P = P1;
+  if (Inverse)
+    P = ICmpInst::getInversePredicate(P);
+  Value *NewCond = Builder.CreateICmp(P, LHS1, NewRHS);
+  NewCond->takeName(&I);
+  I.replaceAllUsesWith(NewCond);
+  eraseInstruction(I, SafetyInfo, MSSAU);
+  eraseInstruction(*cast<Instruction>(Cond1), SafetyInfo, MSSAU);
+  eraseInstruction(*cast<Instruction>(Cond2), SafetyInfo, MSSAU);
+  return true;
+}
+
 /// Little predicate that returns true if the specified basic block is in
 /// a subloop of the current one, not the current one itself.
 ///
diff --git a/llvm/lib/Transforms/Scalar/LoopInterchange.cpp b/llvm/lib/Transforms/Scalar/LoopInterchange.cpp
index 66c498aae808a..c256099557a58 100644
--- a/llvm/lib/Transforms/Scalar/LoopInterchange.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopInterchange.cpp
@@ -1104,8 +1104,7 @@ LoopInterchangeProfitability::isProfitablePerLoopCacheAnalysis(
   // This is the new cost model returned from loop cache analysis.
   // A smaller index means the loop should be placed an outer loop, and vice
   // versa.
-  if (CostMap.find(InnerLoop) != CostMap.end() &&
-      CostMap.find(OuterLoop) != CostMap.end()) {
+  if (CostMap.contains(InnerLoop) && CostMap.contains(OuterLoop)) {
     unsigned InnerIndex = 0, OuterIndex = 0;
     InnerIndex = CostMap.find(InnerLoop)->second;
     OuterIndex = CostMap.find(OuterLoop)->second;
diff --git a/llvm/lib/Transforms/Scalar/LoopStrengthReduce.cpp b/llvm/lib/Transforms/Scalar/LoopStrengthReduce.cpp
index e5da0652a4ab7..805e4c0ec4b57 100644
--- a/llvm/lib/Transforms/Scalar/LoopStrengthReduce.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopStrengthReduce.cpp
@@ -6706,8 +6706,8 @@ canFoldTermCondOfLoop(Loop *L, ScalarEvolution &SE, DominatorTree &DT,
   BranchInst *BI = cast<BranchInst>(LoopLatch->getTerminator());
   if (BI->isUnconditional())
     return std::nullopt;
-  Value *TermCond = BI->getCondition();
-  if (!isa<ICmpInst>(TermCond) || !cast<ICmpInst>(TermCond)->isEquality()) {
+  auto *TermCond = dyn_cast<ICmpInst>(BI->getCondition());
+  if (!TermCond || !TermCond->isEquality()) {
     LLVM_DEBUG(dbgs() << "Cannot fold on branching condition that is not an "
                          "ICmpInst::eq / ICmpInst::ne\n");
     return std::nullopt;
@@ -6800,8 +6800,7 @@ canFoldTermCondOfLoop(Loop *L, ScalarEvolution &SE, DominatorTree &DT,
                            "terminating condition folding.\n");
       continue;
     }
-    const SCEV *S = SE.getSCEV(&PN);
-    const SCEVAddRecExpr *AddRec = dyn_cast<SCEVAddRecExpr>(S);
+    const SCEVAddRecExpr *AddRec = dyn_cast<SCEVAddRecExpr>(SE.getSCEV(&PN));
     // Only speculate on affine AddRec
     if (!AddRec || !AddRec->isAffine()) {
       LLVM_DEBUG(dbgs() << "SCEV of phi '" << PN
diff --git a/llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp b/llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp
index 64846484f9368..1a83c713092da 100644
--- a/llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp
+++ b/llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp
@@ -255,54 +255,6 @@ void MemsetRanges::addRange(int64_t Start, int64_t Size, Value *Ptr,
 //                         MemCpyOptLegacyPass Pass
 //===----------------------------------------------------------------------===//
 
-namespace {
-
-class MemCpyOptLegacyPass : public FunctionPass {
-  MemCpyOptPass Impl;
-
-public:
-  static char ID; // Pass identification, replacement for typeid
-
-  MemCpyOptLegacyPass() : FunctionPass(ID) {
-    initializeMemCpyOptLegacyPassPass(*PassRegistry::getPassRegistry());
-  }
-
-  bool runOnFunction(Function &F) override;
-
-private:
-  // This transformation requires dominator postdominator info
-  void getAnalysisUsage(AnalysisUsage &AU) const override {
-    AU.setPreservesCFG();
-    AU.addRequired<AssumptionCacheTracker>();
-    AU.addRequired<DominatorTreeWrapperPass>();
-    AU.addPreserved<DominatorTreeWrapperPass>();
-    AU.addPreserved<GlobalsAAWrapperPass>();
-    AU.addRequired<TargetLibraryInfoWrapperPass>();
-    AU.addRequired<AAResultsWrapperPass>();
-    AU.addPreserved<AAResultsWrapperPass>();
-    AU.addRequired<MemorySSAWrapperPass>();
-    AU.addPreserved<MemorySSAWrapperPass>();
-  }
-};
-
-} // end anonymous namespace
-
-char MemCpyOptLegacyPass::ID = 0;
-
-/// The public interface to this file...
-FunctionPass *llvm::createMemCpyOptPass() { return new MemCpyOptLegacyPass(); }
-
-INITIALIZE_PASS_BEGIN(MemCpyOptLegacyPass, "memcpyopt", "MemCpy Optimization",
-                      false, false)
-INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
-INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(AAResultsWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(GlobalsAAWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(MemorySSAWrapperPass)
-INITIALIZE_PASS_END(MemCpyOptLegacyPass, "memcpyopt", "MemCpy Optimization",
-                    false, false)
-
 // Check that V is either not accessible by the caller, or unwinding cannot
 // occur between Start and End.
 static bool mayBeVisibleThroughUnwinding(Value *V, Instruction *Start,
@@ -1730,17 +1682,3 @@ bool MemCpyOptPass::runImpl(Function &F, TargetLibraryInfo *TLI_,
 
   return MadeChange;
 }
-
-/// This is the main transformation entry point for a function.
-bool MemCpyOptLegacyPass::runOnFunction(Function &F) {
-  if (skipFunction(F))
-    return false;
-
-  auto *TLI = &getAnalysis<TargetLibraryInfoWrapperPass>().getTLI(F);
-  auto *AA = &getAnalysis<AAResultsWrapperPass>().getAAResults();
-  auto *AC = &getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
-  auto *DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
-  auto *MSSA = &getAnalysis<MemorySSAWrapperPass>().getMSSA();
-
-  return Impl.runImpl(F, TLI, AA, AC, DT, MSSA);
-}
diff --git a/llvm/lib/Transforms/Scalar/NewGVN.cpp b/llvm/lib/Transforms/Scalar/NewGVN.cpp
index d3dba0c5f1d5f..92e9becf7afbd 100644
--- a/llvm/lib/Transforms/Scalar/NewGVN.cpp
+++ b/llvm/lib/Transforms/Scalar/NewGVN.cpp
@@ -4208,61 +4208,6 @@ bool NewGVN::shouldSwapOperandsForIntrinsic(const Value *A, const Value *B,
   return false;
 }
 
-namespace {
-
-class NewGVNLegacyPass : public FunctionPass {
-public:
-  // Pass identification, replacement for typeid.
-  static char ID;
-
-  NewGVNLegacyPass() : FunctionPass(ID) {
-    initializeNewGVNLegacyPassPass(*PassRegistry::getPassRegistry());
-  }
-
-  bool runOnFunction(Function &F) override;
-
-private:
-  void getAnalysisUsage(AnalysisUsage &AU) const override {
-    AU.addRequired<AssumptionCacheTracker>();
-    AU.addRequired<DominatorTreeWrapperPass>();
-    AU.addRequired<TargetLibraryInfoWrapperPass>();
-    AU.addRequired<MemorySSAWrapperPass>();
-    AU.addRequired<AAResultsWrapperPass>();
-    AU.addPreserved<DominatorTreeWrapperPass>();
-    AU.addPreserved<GlobalsAAWrapperPass>();
-  }
-};
-
-} // end anonymous namespace
-
-bool NewGVNLegacyPass::runOnFunction(Function &F) {
-  if (skipFunction(F))
-    return false;
-  return NewGVN(F, &getAnalysis<DominatorTreeWrapperPass>().getDomTree(),
-                &getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F),
-                &getAnalysis<TargetLibraryInfoWrapperPass>().getTLI(F),
-                &getAnalysis<AAResultsWrapperPass>().getAAResults(),
-                &getAnalysis<MemorySSAWrapperPass>().getMSSA(),
-                F.getParent()->getDataLayout())
-      .runGVN();
-}
-
-char NewGVNLegacyPass::ID = 0;
-
-INITIALIZE_PASS_BEGIN(NewGVNLegacyPass, "newgvn", "Global Value Numbering",
-                      false, false)
-INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
-INITIALIZE_PASS_DEPENDENCY(MemorySSAWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(AAResultsWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(GlobalsAAWrapperPass)
-INITIALIZE_PASS_END(NewGVNLegacyPass, "newgvn", "Global Value Numbering", false,
-                    false)
-
-// createGVNPass - The public interface to this file.
-FunctionPass *llvm::createNewGVNPass() { return new NewGVNLegacyPass(); }
-
 PreservedAnalyses NewGVNPass::run(Function &F, AnalysisManager<Function> &AM) {
   // Apparently the order in which we get these results matter for
   // the old GVN (see Chandler's comment in GVN.cpp). I'll keep
diff --git a/llvm/lib/Transforms/Scalar/Reassociate.cpp b/llvm/lib/Transforms/Scalar/Reassociate.cpp
index 21628b61edd62..f12b635a4da84 100644
--- a/llvm/lib/Transforms/Scalar/Reassociate.cpp
+++ b/llvm/lib/Transforms/Scalar/Reassociate.cpp
@@ -2048,7 +2048,7 @@ void ReassociatePass::EraseInst(Instruction *I) {
       // blocks because it's a waste of time and also because it can
       // lead to infinite loop due to LLVM's non-standard definition
       // of dominance.
-      if (ValueRankMap.find(Op) != ValueRankMap.end())
+      if (ValueRankMap.contains(Op))
         RedoInsts.insert(Op);
     }
 
diff --git a/llvm/lib/Transforms/Scalar/RewriteStatepointsForGC.cpp b/llvm/lib/Transforms/Scalar/RewriteStatepointsForGC.cpp
index 77afefcf37a40..cdb5639ef04e7 100644
--- a/llvm/lib/Transforms/Scalar/RewriteStatepointsForGC.cpp
+++ b/llvm/lib/Transforms/Scalar/RewriteStatepointsForGC.cpp
@@ -695,7 +695,7 @@ static Value *findBaseDefiningValue(Value *I, DefiningValueMapTy &Cache,
 /// Returns the base defining value for this value.
 static Value *findBaseDefiningValueCached(Value *I, DefiningValueMapTy &Cache,
                                           IsKnownBaseMapTy &KnownBases) {
-  if (Cache.find(I) == Cache.end()) {
+  if (!Cache.contains(I)) {
     auto *BDV = findBaseDefiningValue(I, Cache, KnownBases);
     Cache[I] = BDV;
     LLVM_DEBUG(dbgs() << "fBDV-cached: " << I->getName() << " -> "
@@ -703,7 +703,7 @@ static Value *findBaseDefiningValueCached(Value *I, DefiningValueMapTy &Cache,
                       << KnownBases[I] << "\n");
   }
   assert(Cache[I] != nullptr);
-  assert(KnownBases.find(Cache[I]) != KnownBases.end() &&
+  assert(KnownBases.contains(Cache[I]) &&
          "Cached value must be present in known bases map");
   return Cache[I];
 }
diff --git a/llvm/lib/Transforms/Scalar/SROA.cpp b/llvm/lib/Transforms/Scalar/SROA.cpp
index e2e0d036eee5d..f221ecf548ffe 100644
--- a/llvm/lib/Transforms/Scalar/SROA.cpp
+++ b/llvm/lib/Transforms/Scalar/SROA.cpp
@@ -1976,6 +1976,7 @@ static VectorType *isVectorPromotionViable(Partition &P, const DataLayout &DL) {
   // Collect the candidate types for vector-based promotion. Also track whether
   // we have different element types.
   SmallVector<VectorType *, 4> CandidateTys;
+  SetVector<Type *> LoadStoreTys;
   Type *CommonEltTy = nullptr;
   VectorType *CommonVecPtrTy = nullptr;
   bool HaveVecPtrTy = false;
@@ -2009,15 +2010,40 @@ static VectorType *isVectorPromotionViable(Partition &P, const DataLayout &DL) {
       }
     }
   };
-  // Consider any loads or stores that are the exact size of the slice.
-  for (const Slice &S : P)
-    if (S.beginOffset() == P.beginOffset() &&
-        S.endOffset() == P.endOffset()) {
-      if (auto *LI = dyn_cast<LoadInst>(S.getUse()->getUser()))
-        CheckCandidateType(LI->getType());
-      else if (auto *SI = dyn_cast<StoreInst>(S.getUse()->getUser()))
-        CheckCandidateType(SI->getValueOperand()->getType());
+  // Put load and store types into a set for de-duplication.
+  for (const Slice &S : P) {
+    Type *Ty;
+    if (auto *LI = dyn_cast<LoadInst>(S.getUse()->getUser()))
+      Ty = LI->getType();
+    else if (auto *SI = dyn_cast<StoreInst>(S.getUse()->getUser()))
+      Ty = SI->getValueOperand()->getType();
+    else
+      continue;
+    LoadStoreTys.insert(Ty);
+    // Consider any loads or stores that are the exact size of the slice.
+    if (S.beginOffset() == P.beginOffset() && S.endOffset() == P.endOffset())
+      CheckCandidateType(Ty);
+  }
+  // Consider additional vector types where the element type size is a
+  // multiple of load/store element size.
+  for (Type *Ty : LoadStoreTys) {
+    if (!VectorType::isValidElementType(Ty))
+      continue;
+    unsigned TypeSize = DL.getTypeSizeInBits(Ty).getFixedValue();
+    // Make a copy of CandidateTys and iterate through it, because we might
+    // append to CandidateTys in the loop.
+    SmallVector<VectorType *, 4> CandidateTysCopy = CandidateTys;
+    for (VectorType *&VTy : CandidateTysCopy) {
+      unsigned VectorSize = DL.getTypeSizeInBits(VTy).getFixedValue();
+      unsigned ElementSize =
+          DL.getTypeSizeInBits(VTy->getElementType()).getFixedValue();
+      if (TypeSize != VectorSize && TypeSize != ElementSize &&
+          VectorSize % TypeSize == 0) {
+        VectorType *NewVTy = VectorType::get(Ty, VectorSize / TypeSize, false);
+        CheckCandidateType(NewVTy);
+      }
     }
+  }
 
   // If we didn't find a vector type, nothing to do here.
   if (CandidateTys.empty())
@@ -2045,7 +2071,7 @@ static VectorType *isVectorPromotionViable(Partition &P, const DataLayout &DL) {
 
     // Rank the remaining candidate vector types. This is easy because we know
     // they're all integer vectors. We sort by ascending number of elements.
-    auto RankVectorTypes = [&DL](VectorType *RHSTy, VectorType *LHSTy) {
+    auto RankVectorTypesComp = [&DL](VectorType *RHSTy, VectorType *LHSTy) {
       (void)DL;
       assert(DL.getTypeSizeInBits(RHSTy).getFixedValue() ==
                  DL.getTypeSizeInBits(LHSTy).getFixedValue() &&
@@ -2057,10 +2083,22 @@ static VectorType *isVectorPromotionViable(Partition &P, const DataLayout &DL) {
       return cast<FixedVectorType>(RHSTy)->getNumElements() <
              cast<FixedVectorType>(LHSTy)->getNumElements();
     };
-    llvm::sort(CandidateTys, RankVectorTypes);
-    CandidateTys.erase(
-        std::unique(CandidateTys.begin(), CandidateTys.end(), RankVectorTypes),
-        CandidateTys.end());
+    auto RankVectorTypesEq = [&DL](VectorType *RHSTy, VectorType *LHSTy) {
+      (void)DL;
+      assert(DL.getTypeSizeInBits(RHSTy).getFixedValue() ==
+                 DL.getTypeSizeInBits(LHSTy).getFixedValue() &&
+             "Cannot have vector types of different sizes!");
+      assert(RHSTy->getElementType()->isIntegerTy() &&
+             "All non-integer types eliminated!");
+      assert(LHSTy->getElementType()->isIntegerTy() &&
+             "All non-integer types eliminated!");
+      return cast<FixedVectorType>(RHSTy)->getNumElements() ==
+             cast<FixedVectorType>(LHSTy)->getNumElements();
+    };
+    llvm::sort(CandidateTys, RankVectorTypesComp);
+    CandidateTys.erase(std::unique(CandidateTys.begin(), CandidateTys.end(),
+                                   RankVectorTypesEq),
+                       CandidateTys.end());
   } else {
 // The only way to have the same element type in every vector type is to
 // have the same vector type. Check that and remove all but one.
@@ -5069,8 +5107,9 @@ PreservedAnalyses SROAPass::runImpl(Function &F, DominatorTree &RunDT,
 }
 
 PreservedAnalyses SROAPass::run(Function &F, FunctionAnalysisManager &AM) {
-  return runImpl(F, AM.getResult<DominatorTreeAnalysis>(F),
-                 AM.getResult<AssumptionAnalysis>(F));
+  DominatorTree &DT = AM.getResult<DominatorTreeAnalysis>(F);
+  AssumptionCache &AC = AM.getResult<AssumptionAnalysis>(F);
+  return runImpl(F, DT, AC);
 }
 
 void SROAPass::printPipeline(
diff --git a/llvm/lib/Transforms/Scalar/Scalar.cpp b/llvm/lib/Transforms/Scalar/Scalar.cpp
index 9a3d243859be9..5c8018956b43a 100644
--- a/llvm/lib/Transforms/Scalar/Scalar.cpp
+++ b/llvm/lib/Transforms/Scalar/Scalar.cpp
@@ -29,31 +29,22 @@ using namespace llvm;
 /// initializeScalarOptsPasses - Initialize all passes linked into the
 /// ScalarOpts library.
 void llvm::initializeScalarOpts(PassRegistry &Registry) {
-  initializeADCELegacyPassPass(Registry);
   initializeBDCELegacyPassPass(Registry);
   initializeAlignmentFromAssumptionsPass(Registry);
   initializeCallSiteSplittingLegacyPassPass(Registry);
   initializeConstantHoistingLegacyPassPass(Registry);
-  initializeCorrelatedValuePropagationPass(Registry);
   initializeDCELegacyPassPass(Registry);
   initializeScalarizerLegacyPassPass(Registry);
-  initializeDSELegacyPassPass(Registry);
   initializeGuardWideningLegacyPassPass(Registry);
   initializeLoopGuardWideningLegacyPassPass(Registry);
   initializeGVNLegacyPassPass(Registry);
-  initializeNewGVNLegacyPassPass(Registry);
   initializeEarlyCSELegacyPassPass(Registry);
   initializeEarlyCSEMemSSALegacyPassPass(Registry);
   initializeMakeGuardsExplicitLegacyPassPass(Registry);
-  initializeGVNHoistLegacyPassPass(Registry);
-  initializeGVNSinkLegacyPassPass(Registry);
   initializeFlattenCFGLegacyPassPass(Registry);
   initializeIRCELegacyPassPass(Registry);
-  initializeIndVarSimplifyLegacyPassPass(Registry);
   initializeInferAddressSpacesPass(Registry);
   initializeInstSimplifyLegacyPassPass(Registry);
-  initializeJumpThreadingPass(Registry);
-  initializeDFAJumpThreadingLegacyPassPass(Registry);
   initializeLegacyLICMPassPass(Registry);
   initializeLegacyLoopSinkPassPass(Registry);
   initializeLoopDataPrefetchLegacyPassPass(Registry);
@@ -70,7 +61,6 @@ void llvm::initializeScalarOpts(PassRegistry &Registry) {
   initializeLowerMatrixIntrinsicsLegacyPassPass(Registry);
   initializeLowerMatrixIntrinsicsMinimalLegacyPassPass(Registry);
   initializeLowerWidenableConditionLegacyPassPass(Registry);
-  initializeMemCpyOptLegacyPassPass(Registry);
   initializeMergeICmpsLegacyPassPass(Registry);
   initializeMergedLoadStoreMotionLegacyPassPass(Registry);
   initializeNaryReassociateLegacyPassPass(Registry);
@@ -92,7 +82,6 @@ void llvm::initializeScalarOpts(PassRegistry &Registry) {
   initializeStraightLineStrengthReduceLegacyPassPass(Registry);
   initializePlaceBackedgeSafepointsLegacyPassPass(Registry);
   initializePlaceSafepointsLegacyPassPass(Registry);
-  initializeFloat2IntLegacyPassPass(Registry);
   initializeLoopSimplifyCFGLegacyPassPass(Registry);
 }
 
diff --git a/llvm/lib/Transforms/Scalar/StructurizeCFG.cpp b/llvm/lib/Transforms/Scalar/StructurizeCFG.cpp
index 81d151c2904e8..a7609f0b60069 100644
--- a/llvm/lib/Transforms/Scalar/StructurizeCFG.cpp
+++ b/llvm/lib/Transforms/Scalar/StructurizeCFG.cpp
@@ -15,10 +15,10 @@
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/Analysis/InstructionSimplify.h"
-#include "llvm/Analysis/LegacyDivergenceAnalysis.h"
 #include "llvm/Analysis/RegionInfo.h"
 #include "llvm/Analysis/RegionIterator.h"
 #include "llvm/Analysis/RegionPass.h"
+#include "llvm/Analysis/UniformityAnalysis.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/CFG.h"
 #include "llvm/IR/Constants.h"
@@ -244,7 +244,7 @@ class StructurizeCFG {
   Function *Func;
   Region *ParentRegion;
 
-  LegacyDivergenceAnalysis *DA = nullptr;
+  UniformityInfo *UA = nullptr;
   DominatorTree *DT;
 
   SmallVector<RegionNode *, 8> Order;
@@ -319,7 +319,7 @@ class StructurizeCFG {
 public:
   void init(Region *R);
   bool run(Region *R, DominatorTree *DT);
-  bool makeUniformRegion(Region *R, LegacyDivergenceAnalysis *DA);
+  bool makeUniformRegion(Region *R, UniformityInfo &UA);
 };
 
 class StructurizeCFGLegacyPass : public RegionPass {
@@ -339,8 +339,9 @@ class StructurizeCFGLegacyPass : public RegionPass {
     StructurizeCFG SCFG;
     SCFG.init(R);
     if (SkipUniformRegions) {
-      LegacyDivergenceAnalysis *DA = &getAnalysis<LegacyDivergenceAnalysis>();
-      if (SCFG.makeUniformRegion(R, DA))
+      UniformityInfo &UA =
+          getAnalysis<UniformityInfoWrapperPass>().getUniformityInfo();
+      if (SCFG.makeUniformRegion(R, UA))
         return false;
     }
     DominatorTree *DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
@@ -351,7 +352,7 @@ class StructurizeCFGLegacyPass : public RegionPass {
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     if (SkipUniformRegions)
-      AU.addRequired<LegacyDivergenceAnalysis>();
+      AU.addRequired<UniformityInfoWrapperPass>();
     AU.addRequiredID(LowerSwitchID);
     AU.addRequired<DominatorTreeWrapperPass>();
 
@@ -366,7 +367,7 @@ char StructurizeCFGLegacyPass::ID = 0;
 
 INITIALIZE_PASS_BEGIN(StructurizeCFGLegacyPass, "structurizecfg",
                       "Structurize the CFG", false, false)
-INITIALIZE_PASS_DEPENDENCY(LegacyDivergenceAnalysis)
+INITIALIZE_PASS_DEPENDENCY(UniformityInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(LowerSwitchLegacyPass)
 INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(RegionInfoPass)
@@ -798,8 +799,6 @@ void StructurizeCFG::killTerminator(BasicBlock *BB) {
   for (BasicBlock *Succ : successors(BB))
     delPhiValues(BB, Succ);
 
-  if (DA)
-    DA->removeValue(Term);
   Term->eraseFromParent();
 }
 
@@ -1064,7 +1063,7 @@ void StructurizeCFG::rebuildSSA() {
 }
 
 static bool hasOnlyUniformBranches(Region *R, unsigned UniformMDKindID,
-                                   const LegacyDivergenceAnalysis &DA) {
+                                   const UniformityInfo &UA) {
   // Bool for if all sub-regions are uniform.
   bool SubRegionsAreUniform = true;
   // Count of how many direct children are conditional.
@@ -1076,7 +1075,7 @@ static bool hasOnlyUniformBranches(Region *R, unsigned UniformMDKindID,
       if (!Br || !Br->isConditional())
         continue;
 
-      if (!DA.isUniform(Br))
+      if (!UA.isUniform(Br))
         return false;
 
       // One of our direct children is conditional.
@@ -1086,7 +1085,7 @@ static bool hasOnlyUniformBranches(Region *R, unsigned UniformMDKindID,
                         << " has uniform terminator\n");
     } else {
       // Explicitly refuse to treat regions as uniform if they have non-uniform
-      // subregions. We cannot rely on DivergenceAnalysis for branches in
+      // subregions. We cannot rely on UniformityAnalysis for branches in
       // subregions because those branches may have been removed and re-created,
       // so we look for our metadata instead.
       //
@@ -1128,15 +1127,15 @@ void StructurizeCFG::init(Region *R) {
   BoolFalse = ConstantInt::getFalse(Context);
   BoolUndef = UndefValue::get(Boolean);
 
-  this->DA = nullptr;
+  this->UA = nullptr;
 }
 
-bool StructurizeCFG::makeUniformRegion(Region *R,
-                                       LegacyDivergenceAnalysis *DA) {
+bool StructurizeCFG::makeUniformRegion(Region *R, UniformityInfo &UA) {
   if (R->isTopLevelRegion())
     return false;
 
-  this->DA = DA;
+  this->UA = &UA;
+
   // TODO: We could probably be smarter here with how we handle sub-regions.
   // We currently rely on the fact that metadata is set by earlier invocations
   // of the pass on sub-regions, and that this metadata doesn't get lost --
@@ -1144,7 +1143,7 @@ bool StructurizeCFG::makeUniformRegion(Region *R,
   unsigned UniformMDKindID =
       R->getEntry()->getContext().getMDKindID("structurizecfg.uniform");
 
-  if (hasOnlyUniformBranches(R, UniformMDKindID, *DA)) {
+  if (hasOnlyUniformBranches(R, UniformMDKindID, UA)) {
     LLVM_DEBUG(dbgs() << "Skipping region with uniform control flow: " << *R
                       << '\n');
 
diff --git a/llvm/lib/Transforms/Utils/AssumeBundleBuilder.cpp b/llvm/lib/Transforms/Utils/AssumeBundleBuilder.cpp
index d17c399ba7989..7eb615e90b177 100644
--- a/llvm/lib/Transforms/Utils/AssumeBundleBuilder.cpp
+++ b/llvm/lib/Transforms/Utils/AssumeBundleBuilder.cpp
@@ -290,17 +290,20 @@ AssumeInst *llvm::buildAssumeFromInst(Instruction *I) {
   return Builder.build();
 }
 
-void llvm::salvageKnowledge(Instruction *I, AssumptionCache *AC,
+bool llvm::salvageKnowledge(Instruction *I, AssumptionCache *AC,
                             DominatorTree *DT) {
   if (!EnableKnowledgeRetention || I->isTerminator())
-    return;
+    return false;
+  bool Changed = false;
   AssumeBuilderState Builder(I->getModule(), I, AC, DT);
   Builder.addInstruction(I);
   if (auto *Intr = Builder.build()) {
     Intr->insertBefore(I);
+    Changed = true;
     if (AC)
       AC->registerAssumption(Intr);
   }
+  return Changed;
 }
 
 AssumeInst *
@@ -563,9 +566,12 @@ PreservedAnalyses AssumeSimplifyPass::run(Function &F,
                                           FunctionAnalysisManager &AM) {
   if (!EnableKnowledgeRetention)
     return PreservedAnalyses::all();
-  simplifyAssumes(F, &AM.getResult<AssumptionAnalysis>(F),
-                  AM.getCachedResult<DominatorTreeAnalysis>(F));
-  return PreservedAnalyses::all();
+  if (!simplifyAssumes(F, &AM.getResult<AssumptionAnalysis>(F),
+                       AM.getCachedResult<DominatorTreeAnalysis>(F)))
+    return PreservedAnalyses::all();
+  PreservedAnalyses PA;
+  PA.preserveSet<CFGAnalyses>();
+  return PA;
 }
 
 namespace {
@@ -611,9 +617,14 @@ PreservedAnalyses AssumeBuilderPass::run(Function &F,
                                          FunctionAnalysisManager &AM) {
   AssumptionCache *AC = &AM.getResult<AssumptionAnalysis>(F);
   DominatorTree* DT = AM.getCachedResult<DominatorTreeAnalysis>(F);
+  bool Changed = false;
   for (Instruction &I : instructions(F))
-    salvageKnowledge(&I, AC, DT);
-  return PreservedAnalyses::all();
+    Changed |= salvageKnowledge(&I, AC, DT);
+  if (!Changed)
+    PreservedAnalyses::all();
+  PreservedAnalyses PA;
+  PA.preserveSet<CFGAnalyses>();
+  return PA;
 }
 
 namespace {
diff --git a/llvm/lib/Transforms/Utils/BasicBlockUtils.cpp b/llvm/lib/Transforms/Utils/BasicBlockUtils.cpp
index 0dd385766dea8..f0e24650cd5e2 100644
--- a/llvm/lib/Transforms/Utils/BasicBlockUtils.cpp
+++ b/llvm/lib/Transforms/Utils/BasicBlockUtils.cpp
@@ -32,6 +32,7 @@
 #include "llvm/IR/Instruction.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/IR/Type.h"
 #include "llvm/IR/User.h"
@@ -1599,6 +1600,57 @@ void llvm::SplitBlockAndInsertIfThenElse(Value *Cond, Instruction *SplitBefore,
   }
 }
 
+std::pair<Instruction*, Value*>
+llvm::SplitBlockAndInsertSimpleForLoop(Value *End, Instruction *SplitBefore) {
+  BasicBlock *LoopPred = SplitBefore->getParent();
+  BasicBlock *LoopBody = SplitBlock(SplitBefore->getParent(), SplitBefore);
+  BasicBlock *LoopExit = SplitBlock(SplitBefore->getParent(), SplitBefore);
+
+  auto *Ty = End->getType();
+  auto &DL = SplitBefore->getModule()->getDataLayout();
+  const unsigned Bitwidth = DL.getTypeSizeInBits(Ty);
+
+  IRBuilder<> Builder(LoopBody->getTerminator());
+  auto *IV = Builder.CreatePHI(Ty, 2, "iv");
+  auto *IVNext =
+    Builder.CreateAdd(IV, ConstantInt::get(Ty, 1), IV->getName() + ".next",
+                      /*HasNUW=*/true, /*HasNSW=*/Bitwidth != 2);
+  auto *IVCheck = Builder.CreateICmpEQ(IVNext, End,
+                                       IV->getName() + ".check");
+  Builder.CreateCondBr(IVCheck, LoopExit, LoopBody);
+  LoopBody->getTerminator()->eraseFromParent();
+
+  // Populate the IV PHI.
+  IV->addIncoming(ConstantInt::get(Ty, 0), LoopPred);
+  IV->addIncoming(IVNext, LoopBody);
+
+  return std::make_pair(LoopBody->getFirstNonPHI(), IV);
+}
+
+void llvm::SplitBlockAndInsertForEachLane(ElementCount EC,
+     Type *IndexTy, Instruction *InsertBefore,
+     std::function<void(IRBuilderBase&, Value*)> Func) {
+
+  IRBuilder<> IRB(InsertBefore);
+
+  if (EC.isScalable()) {
+    Value *NumElements = IRB.CreateElementCount(IndexTy, EC);
+
+    auto [BodyIP, Index] =
+      SplitBlockAndInsertSimpleForLoop(NumElements, InsertBefore);
+
+    IRB.SetInsertPoint(BodyIP);
+    Func(IRB, Index);
+    return;
+  }
+
+  unsigned Num = EC.getFixedValue();
+  for (unsigned Idx = 0; Idx < Num; ++Idx) {
+    IRB.SetInsertPoint(InsertBefore);
+    Func(IRB, ConstantInt::get(IndexTy, Idx));
+  }
+}
+
 BranchInst *llvm::GetIfCondition(BasicBlock *BB, BasicBlock *&IfTrue,
                                  BasicBlock *&IfFalse) {
   PHINode *SomePHI = dyn_cast<PHINode>(BB->begin());
diff --git a/llvm/lib/Transforms/Utils/Debugify.cpp b/llvm/lib/Transforms/Utils/Debugify.cpp
index 989473693a0bc..93cad0888a56d 100644
--- a/llvm/lib/Transforms/Utils/Debugify.cpp
+++ b/llvm/lib/Transforms/Utils/Debugify.cpp
@@ -979,7 +979,9 @@ PreservedAnalyses NewPMDebugifyPass::run(Module &M, ModuleAnalysisManager &) {
     collectDebugInfoMetadata(M, M.functions(), *DebugInfoBeforePass,
                              "ModuleDebugify (original debuginfo)",
                               NameOfWrappedPass);
-  return PreservedAnalyses::all();
+  PreservedAnalyses PA;
+  PA.preserveSet<CFGAnalyses>();
+  return PA;
 }
 
 ModulePass *createCheckDebugifyModulePass(
@@ -1027,45 +1029,58 @@ static bool isIgnoredPass(StringRef PassID) {
 }
 
 void DebugifyEachInstrumentation::registerCallbacks(
-    PassInstrumentationCallbacks &PIC) {
-  PIC.registerBeforeNonSkippedPassCallback([this](StringRef P, Any IR) {
-    if (isIgnoredPass(P))
-      return;
-    if (const auto **F = any_cast<const Function *>(&IR))
-      applyDebugify(*const_cast<Function *>(*F),
-                    Mode, DebugInfoBeforePass, P);
-    else if (const auto **M = any_cast<const Module *>(&IR))
-      applyDebugify(*const_cast<Module *>(*M),
-                    Mode, DebugInfoBeforePass, P);
-  });
-  PIC.registerAfterPassCallback([this](StringRef P, Any IR,
-                                       const PreservedAnalyses &PassPA) {
+    PassInstrumentationCallbacks &PIC, ModuleAnalysisManager &MAM) {
+  PIC.registerBeforeNonSkippedPassCallback([this, &MAM](StringRef P, Any IR) {
     if (isIgnoredPass(P))
       return;
+    PreservedAnalyses PA;
+    PA.preserveSet<CFGAnalyses>();
     if (const auto **CF = any_cast<const Function *>(&IR)) {
-      auto &F = *const_cast<Function *>(*CF);
-      Module &M = *F.getParent();
-      auto It = F.getIterator();
-      if (Mode == DebugifyMode::SyntheticDebugInfo)
-        checkDebugifyMetadata(M, make_range(It, std::next(It)), P,
-                              "CheckFunctionDebugify", /*Strip=*/true, DIStatsMap);
-      else
-        checkDebugInfoMetadata(
-          M, make_range(It, std::next(It)), *DebugInfoBeforePass,
-          "CheckModuleDebugify (original debuginfo)",
-          P, OrigDIVerifyBugsReportFilePath);
+      Function &F = *const_cast<Function *>(*CF);
+      applyDebugify(F, Mode, DebugInfoBeforePass, P);
+      MAM.getResult<FunctionAnalysisManagerModuleProxy>(*F.getParent())
+          .getManager()
+          .invalidate(F, PA);
     } else if (const auto **CM = any_cast<const Module *>(&IR)) {
-      auto &M = *const_cast<Module *>(*CM);
-      if (Mode == DebugifyMode::SyntheticDebugInfo)
-       checkDebugifyMetadata(M, M.functions(), P, "CheckModuleDebugify",
-                            /*Strip=*/true, DIStatsMap);
-      else
-        checkDebugInfoMetadata(
-          M, M.functions(), *DebugInfoBeforePass,
-          "CheckModuleDebugify (original debuginfo)",
-          P, OrigDIVerifyBugsReportFilePath);
+      Module &M = *const_cast<Module *>(*CM);
+      applyDebugify(M, Mode, DebugInfoBeforePass, P);
+      MAM.invalidate(M, PA);
     }
   });
+  PIC.registerAfterPassCallback(
+      [this, &MAM](StringRef P, Any IR, const PreservedAnalyses &PassPA) {
+        if (isIgnoredPass(P))
+          return;
+        PreservedAnalyses PA;
+        PA.preserveSet<CFGAnalyses>();
+        if (const auto **CF = any_cast<const Function *>(&IR)) {
+          auto &F = *const_cast<Function *>(*CF);
+          Module &M = *F.getParent();
+          auto It = F.getIterator();
+          if (Mode == DebugifyMode::SyntheticDebugInfo)
+            checkDebugifyMetadata(M, make_range(It, std::next(It)), P,
+                                  "CheckFunctionDebugify", /*Strip=*/true,
+                                  DIStatsMap);
+          else
+            checkDebugInfoMetadata(M, make_range(It, std::next(It)),
+                                   *DebugInfoBeforePass,
+                                   "CheckModuleDebugify (original debuginfo)",
+                                   P, OrigDIVerifyBugsReportFilePath);
+          MAM.getResult<FunctionAnalysisManagerModuleProxy>(*F.getParent())
+              .getManager()
+              .invalidate(F, PA);
+        } else if (const auto **CM = any_cast<const Module *>(&IR)) {
+          Module &M = *const_cast<Module *>(*CM);
+          if (Mode == DebugifyMode::SyntheticDebugInfo)
+            checkDebugifyMetadata(M, M.functions(), P, "CheckModuleDebugify",
+                                  /*Strip=*/true, DIStatsMap);
+          else
+            checkDebugInfoMetadata(M, M.functions(), *DebugInfoBeforePass,
+                                   "CheckModuleDebugify (original debuginfo)",
+                                   P, OrigDIVerifyBugsReportFilePath);
+          MAM.invalidate(M, PA);
+        }
+      });
 }
 
 char DebugifyModulePass::ID = 0;
diff --git a/llvm/lib/Transforms/Utils/InlineFunction.cpp b/llvm/lib/Transforms/Utils/InlineFunction.cpp
index 20617383cd452..474691643eb25 100644
--- a/llvm/lib/Transforms/Utils/InlineFunction.cpp
+++ b/llvm/lib/Transforms/Utils/InlineFunction.cpp
@@ -1537,7 +1537,8 @@ static void UpdateCallGraphAfterInlining(CallBase &CB,
 
 static void HandleByValArgumentInit(Type *ByValType, Value *Dst, Value *Src,
                                     Module *M, BasicBlock *InsertBlock,
-                                    InlineFunctionInfo &IFI) {
+                                    InlineFunctionInfo &IFI,
+                                    Function *CalledFunc) {
   IRBuilder<> Builder(InsertBlock, InsertBlock->begin());
 
   Value *Size =
@@ -1546,8 +1547,15 @@ static void HandleByValArgumentInit(Type *ByValType, Value *Dst, Value *Src,
   // Always generate a memcpy of alignment 1 here because we don't know
   // the alignment of the src pointer.  Other optimizations can infer
   // better alignment.
-  Builder.CreateMemCpy(Dst, /*DstAlign*/ Align(1), Src,
-                       /*SrcAlign*/ Align(1), Size);
+  CallInst *CI = Builder.CreateMemCpy(Dst, /*DstAlign*/ Align(1), Src,
+                                      /*SrcAlign*/ Align(1), Size);
+
+  // The verifier requires that all calls of debug-info-bearing functions
+  // from debug-info-bearing functions have a debug location (for inlining
+  // purposes). Assign a dummy location to satisfy the constraint.
+  if (!CI->getDebugLoc() && InsertBlock->getParent()->getSubprogram())
+    if (DISubprogram *SP = CalledFunc->getSubprogram())
+      CI->setDebugLoc(DILocation::get(SP->getContext(), 0, 0, SP));
 }
 
 /// When inlining a call site that has a byval argument,
@@ -2242,7 +2250,7 @@ llvm::InlineResult llvm::InlineFunction(CallBase &CB, InlineFunctionInfo &IFI,
     // Inject byval arguments initialization.
     for (ByValInit &Init : ByValInits)
       HandleByValArgumentInit(Init.Ty, Init.Dst, Init.Src, Caller->getParent(),
-                              &*FirstNewBlock, IFI);
+                              &*FirstNewBlock, IFI, CalledFunc);
 
     std::optional<OperandBundleUse> ParentDeopt =
         CB.getOperandBundle(LLVMContext::OB_deopt);
diff --git a/llvm/lib/Transforms/Utils/LibCallsShrinkWrap.cpp b/llvm/lib/Transforms/Utils/LibCallsShrinkWrap.cpp
index 5dd469c7af4b8..ff2d5a0175904 100644
--- a/llvm/lib/Transforms/Utils/LibCallsShrinkWrap.cpp
+++ b/llvm/lib/Transforms/Utils/LibCallsShrinkWrap.cpp
@@ -50,27 +50,6 @@ using namespace llvm;
 STATISTIC(NumWrappedOneCond, "Number of One-Condition Wrappers Inserted");
 STATISTIC(NumWrappedTwoCond, "Number of Two-Condition Wrappers Inserted");
 
-namespace {
-class LibCallsShrinkWrapLegacyPass : public FunctionPass {
-public:
-  static char ID; // Pass identification, replacement for typeid
-  explicit LibCallsShrinkWrapLegacyPass() : FunctionPass(ID) {
-    initializeLibCallsShrinkWrapLegacyPassPass(
-        *PassRegistry::getPassRegistry());
-  }
-  void getAnalysisUsage(AnalysisUsage &AU) const override;
-  bool runOnFunction(Function &F) override;
-};
-}
-
-char LibCallsShrinkWrapLegacyPass::ID = 0;
-INITIALIZE_PASS_BEGIN(LibCallsShrinkWrapLegacyPass, "libcalls-shrinkwrap",
-                      "Conditionally eliminate dead library calls", false,
-                      false)
-INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
-INITIALIZE_PASS_END(LibCallsShrinkWrapLegacyPass, "libcalls-shrinkwrap",
-                    "Conditionally eliminate dead library calls", false, false)
-
 namespace {
 class LibCallsShrinkWrap : public InstVisitor<LibCallsShrinkWrap> {
 public:
@@ -515,12 +494,6 @@ bool LibCallsShrinkWrap::perform(CallInst *CI) {
   return performCallErrors(CI, Func);
 }
 
-void LibCallsShrinkWrapLegacyPass::getAnalysisUsage(AnalysisUsage &AU) const {
-  AU.addPreserved<DominatorTreeWrapperPass>();
-  AU.addPreserved<GlobalsAAWrapperPass>();
-  AU.addRequired<TargetLibraryInfoWrapperPass>();
-}
-
 static bool runImpl(Function &F, const TargetLibraryInfo &TLI,
                     DominatorTree *DT) {
   if (F.hasFnAttribute(Attribute::OptimizeForSize))
@@ -534,21 +507,6 @@ static bool runImpl(Function &F, const TargetLibraryInfo &TLI,
   return Changed;
 }
 
-bool LibCallsShrinkWrapLegacyPass::runOnFunction(Function &F) {
-  auto &TLI = getAnalysis<TargetLibraryInfoWrapperPass>().getTLI(F);
-  auto *DTWP = getAnalysisIfAvailable<DominatorTreeWrapperPass>();
-  auto *DT = DTWP ? &DTWP->getDomTree() : nullptr;
-  return runImpl(F, TLI, DT);
-}
-
-namespace llvm {
-char &LibCallsShrinkWrapPassID = LibCallsShrinkWrapLegacyPass::ID;
-
-// Public interface to LibCallsShrinkWrap pass.
-FunctionPass *createLibCallsShrinkWrapPass() {
-  return new LibCallsShrinkWrapLegacyPass();
-}
-
 PreservedAnalyses LibCallsShrinkWrapPass::run(Function &F,
                                               FunctionAnalysisManager &FAM) {
   auto &TLI = FAM.getResult<TargetLibraryAnalysis>(F);
@@ -559,4 +517,3 @@ PreservedAnalyses LibCallsShrinkWrapPass::run(Function &F,
   PA.preserve<DominatorTreeAnalysis>();
   return PA;
 }
-}
diff --git a/llvm/lib/Transforms/Utils/Local.cpp b/llvm/lib/Transforms/Utils/Local.cpp
index 9c4f59897e811..3f486147564ee 100644
--- a/llvm/lib/Transforms/Utils/Local.cpp
+++ b/llvm/lib/Transforms/Utils/Local.cpp
@@ -1493,11 +1493,9 @@ static bool PhiHasDebugValue(DILocalVariable *DIVar,
 static bool valueCoversEntireFragment(Type *ValTy, DbgVariableIntrinsic *DII) {
   const DataLayout &DL = DII->getModule()->getDataLayout();
   TypeSize ValueSize = DL.getTypeAllocSizeInBits(ValTy);
-  if (std::optional<uint64_t> FragmentSize = DII->getFragmentSizeInBits()) {
-    assert(!ValueSize.isScalable() &&
-           "Fragments don't work on scalable types.");
-    return ValueSize.getFixedValue() >= *FragmentSize;
-  }
+  if (std::optional<uint64_t> FragmentSize = DII->getFragmentSizeInBits())
+    return TypeSize::isKnownGE(ValueSize, TypeSize::getFixed(*FragmentSize));
+
   // We can't always calculate the size of the DI variable (e.g. if it is a
   // VLA). Try to use the size of the alloca that the dbg intrinsic describes
   // intead.
@@ -1915,17 +1913,14 @@ void llvm::salvageDebugInfoForDbgValues(
     bool IsValidSalvageExpr = SalvagedExpr->getNumElements() <= MaxExpressionSize;
     if (AdditionalValues.empty() && IsValidSalvageExpr) {
       DII->setExpression(SalvagedExpr);
-    } else if (isa<DbgValueInst>(DII) && !isa<DbgAssignIntrinsic>(DII) &&
-               IsValidSalvageExpr &&
+    } else if (isa<DbgValueInst>(DII) && IsValidSalvageExpr &&
                DII->getNumVariableLocationOps() + AdditionalValues.size() <=
                    MaxDebugArgs) {
       DII->addVariableLocationOps(AdditionalValues, SalvagedExpr);
     } else {
       // Do not salvage using DIArgList for dbg.declare, as it is not currently
-      // supported in those instructions. Do not salvage using DIArgList for
-      // dbg.assign yet. FIXME: support this.
-      // Also do not salvage if the resulting DIArgList would contain an
-      // unreasonably large number of values.
+      // supported in those instructions. Also do not salvage if the resulting
+      // DIArgList would contain an unreasonably large number of values.
       DII->setKillLocation();
     }
     LLVM_DEBUG(dbgs() << "SALVAGE: " << *DII << '\n');
diff --git a/llvm/lib/Transforms/Utils/LoopUnrollAndJam.cpp b/llvm/lib/Transforms/Utils/LoopUnrollAndJam.cpp
index b125e952ec941..ed8e34f84e5aa 100644
--- a/llvm/lib/Transforms/Utils/LoopUnrollAndJam.cpp
+++ b/llvm/lib/Transforms/Utils/LoopUnrollAndJam.cpp
@@ -757,11 +757,11 @@ checkDependencies(Loop &Root, const BasicBlockSet &SubLoopBlocks,
                   DependenceInfo &DI, LoopInfo &LI) {
   SmallVector<BasicBlockSet, 8> AllBlocks;
   for (Loop *L : Root.getLoopsInPreorder())
-    if (ForeBlocksMap.find(L) != ForeBlocksMap.end())
+    if (ForeBlocksMap.contains(L))
       AllBlocks.push_back(ForeBlocksMap.lookup(L));
   AllBlocks.push_back(SubLoopBlocks);
   for (Loop *L : Root.getLoopsInPreorder())
-    if (AftBlocksMap.find(L) != AftBlocksMap.end())
+    if (AftBlocksMap.contains(L))
       AllBlocks.push_back(AftBlocksMap.lookup(L));
 
   unsigned LoopDepth = Root.getLoopDepth();
diff --git a/llvm/lib/Transforms/Utils/MetaRenamer.cpp b/llvm/lib/Transforms/Utils/MetaRenamer.cpp
index 0987fdc33fdd5..0ab750ad43cd1 100644
--- a/llvm/lib/Transforms/Utils/MetaRenamer.cpp
+++ b/llvm/lib/Transforms/Utils/MetaRenamer.cpp
@@ -115,7 +115,7 @@ void MetaRename(Function &F) {
 
     for (auto &I : BB)
       if (!I.getType()->isVoidTy())
-        I.setName("inst");
+        I.setName(I.getOpcodeName());
   }
 }
 
diff --git a/llvm/lib/Transforms/Utils/ScalarEvolutionExpander.cpp b/llvm/lib/Transforms/Utils/ScalarEvolutionExpander.cpp
index 902eee26a4567..b83700cc258d0 100644
--- a/llvm/lib/Transforms/Utils/ScalarEvolutionExpander.cpp
+++ b/llvm/lib/Transforms/Utils/ScalarEvolutionExpander.cpp
@@ -174,7 +174,7 @@ Value *SCEVExpander::InsertNoopCastOfTo(Value *V, Type *Ty) {
       assert(DL.getTypeAllocSize(Builder.getInt8Ty()) == 1 &&
              "alloc size of i8 must by 1 byte for the GEP to be correct");
       auto *GEP = Builder.CreateGEP(
-          Builder.getInt8Ty(), Constant::getNullValue(Int8PtrTy), V, "uglygep");
+          Builder.getInt8Ty(), Constant::getNullValue(Int8PtrTy), V, "scevgep");
       return Builder.CreateBitCast(GEP, Ty);
     }
   }
@@ -613,7 +613,7 @@ Value *SCEVExpander::expandAddToGEP(const SCEV *const *op_begin,
     }
 
     // Emit a GEP.
-    return Builder.CreateGEP(Builder.getInt8Ty(), V, Idx, "uglygep");
+    return Builder.CreateGEP(Builder.getInt8Ty(), V, Idx, "scevgep");
   }
 
   {
diff --git a/llvm/lib/Transforms/Utils/SimplifyCFG.cpp b/llvm/lib/Transforms/Utils/SimplifyCFG.cpp
index 28a09e820d6ca..187f11a0a0897 100644
--- a/llvm/lib/Transforms/Utils/SimplifyCFG.cpp
+++ b/llvm/lib/Transforms/Utils/SimplifyCFG.cpp
@@ -7110,34 +7110,11 @@ static bool passingValueIsAlwaysUndefined(Value *V, Instruction *I, bool PtrValu
     // Look through GEPs. A load from a GEP derived from NULL is still undefined
     if (GetElementPtrInst *GEP = dyn_cast<GetElementPtrInst>(Use))
       if (GEP->getPointerOperand() == I) {
-        // The current base address is null, there are four cases to consider:
-        // getelementptr (TY, null, 0)                 -> null
-        // getelementptr (TY, null, not zero)          -> may be modified
-        // getelementptr inbounds (TY, null, 0)        -> null
-        // getelementptr inbounds (TY, null, not zero) -> poison iff null is
-        // undefined?
-        if (!GEP->hasAllZeroIndices() &&
-            (!GEP->isInBounds() ||
-             NullPointerIsDefined(GEP->getFunction(),
-                                  GEP->getPointerAddressSpace())))
+        if (!GEP->isInBounds() || !GEP->hasAllZeroIndices())
           PtrValueMayBeModified = true;
         return passingValueIsAlwaysUndefined(V, GEP, PtrValueMayBeModified);
       }
 
-    // Look through return.
-    if (ReturnInst *Ret = dyn_cast<ReturnInst>(Use)) {
-      bool HasNoUndefAttr =
-          Ret->getFunction()->hasRetAttribute(Attribute::NoUndef);
-      // Return undefined to a noundef return value is undefined.
-      if (isa<UndefValue>(C) && HasNoUndefAttr)
-        return true;
-      // Return null to a nonnull+noundef return value is undefined.
-      if (C->isNullValue() && HasNoUndefAttr &&
-          Ret->getFunction()->hasRetAttribute(Attribute::NonNull)) {
-        return !PtrValueMayBeModified;
-      }
-    }
-
     // Look through bitcasts.
     if (BitCastInst *BC = dyn_cast<BitCastInst>(Use))
       return passingValueIsAlwaysUndefined(V, BC, PtrValueMayBeModified);
diff --git a/llvm/lib/Transforms/Utils/SimplifyLibCalls.cpp b/llvm/lib/Transforms/Utils/SimplifyLibCalls.cpp
index 746f61f4796a1..b6e5f84a30d8e 100644
--- a/llvm/lib/Transforms/Utils/SimplifyLibCalls.cpp
+++ b/llvm/lib/Transforms/Utils/SimplifyLibCalls.cpp
@@ -2588,7 +2588,7 @@ static bool insertSinCosCall(IRBuilderBase &B, Function *OrigCallee, Value *Arg,
   return true;
 }
 
-Value *LibCallSimplifier::optimizeSinCosPi(CallInst *CI, IRBuilderBase &B) {
+Value *LibCallSimplifier::optimizeSinCosPi(CallInst *CI, bool IsSin, IRBuilderBase &B) {
   // Make sure the prototype is as expected, otherwise the rest of the
   // function is probably invalid and likely to abort.
   if (!isTrigLibCall(CI))
@@ -2627,7 +2627,7 @@ Value *LibCallSimplifier::optimizeSinCosPi(CallInst *CI, IRBuilderBase &B) {
   replaceTrigInsts(CosCalls, Cos);
   replaceTrigInsts(SinCosCalls, SinCos);
 
-  return nullptr;
+  return IsSin ? Sin : Cos;
 }
 
 void LibCallSimplifier::classifyArgUse(
@@ -3470,9 +3470,10 @@ Value *LibCallSimplifier::optimizeFloatingPointLibCall(CallInst *CI,
   switch (Func) {
   case LibFunc_sinpif:
   case LibFunc_sinpi:
+    return optimizeSinCosPi(CI, /*IsSin*/true, Builder);
   case LibFunc_cospif:
   case LibFunc_cospi:
-    return optimizeSinCosPi(CI, Builder);
+    return optimizeSinCosPi(CI, /*IsSin*/false, Builder);
   case LibFunc_powf:
   case LibFunc_pow:
   case LibFunc_powl:
diff --git a/llvm/lib/Transforms/Utils/StripNonLineTableDebugInfo.cpp b/llvm/lib/Transforms/Utils/StripNonLineTableDebugInfo.cpp
index 01aa6d15948f9..65b5402008890 100644
--- a/llvm/lib/Transforms/Utils/StripNonLineTableDebugInfo.cpp
+++ b/llvm/lib/Transforms/Utils/StripNonLineTableDebugInfo.cpp
@@ -17,5 +17,7 @@ using namespace llvm;
 PreservedAnalyses
 StripNonLineTableDebugInfoPass::run(Module &M, ModuleAnalysisManager &AM) {
   llvm::stripNonLineTableDebugInfo(M);
-  return PreservedAnalyses::all();
+  PreservedAnalyses PA;
+  PA.preserveSet<CFGAnalyses>();
+  return PA;
 }
diff --git a/llvm/lib/Transforms/Utils/Utils.cpp b/llvm/lib/Transforms/Utils/Utils.cpp
index d86f20c7bb6fe..59d1b8e42dc19 100644
--- a/llvm/lib/Transforms/Utils/Utils.cpp
+++ b/llvm/lib/Transforms/Utils/Utils.cpp
@@ -30,7 +30,6 @@ void llvm::initializeTransformUtils(PassRegistry &Registry) {
   initializeCanonicalizeFreezeInLoopsPass(Registry);
   initializeInstNamerPass(Registry);
   initializeLCSSAWrapperPassPass(Registry);
-  initializeLibCallsShrinkWrapLegacyPassPass(Registry);
   initializeLoopSimplifyPass(Registry);
   initializeLowerGlobalDtorsLegacyPassPass(Registry);
   initializeLowerInvokeLegacyPassPass(Registry);
diff --git a/llvm/lib/Transforms/Utils/ValueMapper.cpp b/llvm/lib/Transforms/Utils/ValueMapper.cpp
index 2a7aba0bcee79..6fd608737ec40 100644
--- a/llvm/lib/Transforms/Utils/ValueMapper.cpp
+++ b/llvm/lib/Transforms/Utils/ValueMapper.cpp
@@ -1181,6 +1181,10 @@ void ValueMapper::remapFunction(Function &F) {
   FlushingMapper(pImpl)->remapFunction(F);
 }
 
+void ValueMapper::remapGlobalObjectMetadata(GlobalObject &GO) {
+  FlushingMapper(pImpl)->remapGlobalObjectMetadata(GO);
+}
+
 void ValueMapper::scheduleMapGlobalInitializer(GlobalVariable &GV,
                                                Constant &Init,
                                                unsigned MCID) {
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
index a7cd0c9821869..91539dd6f0786 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
@@ -902,7 +902,7 @@ bool LoopVectorizationLegality::canVectorizeInstrs() {
   if (any_of(FixedOrderRecurrences, [LoopLatch, this](const PHINode *Phi) {
         Instruction *V =
             cast<Instruction>(Phi->getIncomingValueForBlock(LoopLatch));
-        return SinkAfter.find(V) != SinkAfter.end();
+        return SinkAfter.contains(V);
       }))
     return false;
 
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 7fadb5d9a34b7..1054f6aab0748 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -520,7 +520,7 @@ class InnerLoopVectorizer {
   /// Instr's operands.
   void scalarizeInstruction(const Instruction *Instr,
                             VPReplicateRecipe *RepRecipe,
-                            const VPIteration &Instance, bool IfPredicateInstr,
+                            const VPIteration &Instance,
                             VPTransformState &State);
 
   /// Construct the vector value of a scalarized value \p V one lane at a time.
@@ -975,14 +975,12 @@ namespace llvm {
 Value *createStepForVF(IRBuilderBase &B, Type *Ty, ElementCount VF,
                        int64_t Step) {
   assert(Ty->isIntegerTy() && "Expected an integer step");
-  Constant *StepVal = ConstantInt::get(Ty, Step * VF.getKnownMinValue());
-  return VF.isScalable() ? B.CreateVScale(StepVal) : StepVal;
+  return B.CreateElementCount(Ty, VF.multiplyCoefficientBy(Step));
 }
 
 /// Return the runtime value for VF.
 Value *getRuntimeVF(IRBuilderBase &B, Type *Ty, ElementCount VF) {
-  Constant *EC = ConstantInt::get(Ty, VF.getKnownMinValue());
-  return VF.isScalable() ? B.CreateVScale(EC) : EC;
+  return B.CreateElementCount(Ty, VF);
 }
 
 const SCEV *createTripCountSCEV(Type *IdxTy, PredicatedScalarEvolution &PSE) {
@@ -1302,7 +1300,7 @@ class LoopVectorizationCostModel {
     auto Scalars = InstsToScalarize.find(VF);
     assert(Scalars != InstsToScalarize.end() &&
            "VF not yet analyzed for scalarization profitability");
-    return Scalars->second.find(I) != Scalars->second.end();
+    return Scalars->second.contains(I);
   }
 
   /// Returns true if \p I is known to be uniform after vectorization.
@@ -1346,7 +1344,7 @@ class LoopVectorizationCostModel {
   /// \returns True if instruction \p I can be truncated to a smaller bitwidth
   /// for vectorization factor \p VF.
   bool canTruncateToMinimalBitwidth(Instruction *I, ElementCount VF) const {
-    return VF.isVector() && MinBWs.find(I) != MinBWs.end() &&
+    return VF.isVector() && MinBWs.contains(I) &&
            !isProfitableToScalarize(I, VF) &&
            !isScalarAfterVectorization(I, VF);
   }
@@ -1409,7 +1407,7 @@ class LoopVectorizationCostModel {
   InstructionCost getWideningCost(Instruction *I, ElementCount VF) {
     assert(VF.isVector() && "Expected VF >=2");
     std::pair<Instruction *, ElementCount> InstOnVF = std::make_pair(I, VF);
-    assert(WideningDecisions.find(InstOnVF) != WideningDecisions.end() &&
+    assert(WideningDecisions.contains(InstOnVF) &&
            "The cost is not calculated");
     return WideningDecisions[InstOnVF].second;
   }
@@ -1449,7 +1447,7 @@ class LoopVectorizationCostModel {
   /// that may be vectorized as interleave, gather-scatter or scalarized.
   void collectUniformsAndScalars(ElementCount VF) {
     // Do the analysis once.
-    if (VF.isScalar() || Uniforms.find(VF) != Uniforms.end())
+    if (VF.isScalar() || Uniforms.contains(VF))
       return;
     setCostBasedWideningDecision(VF);
     collectLoopUniforms(VF);
@@ -1833,8 +1831,7 @@ class LoopVectorizationCostModel {
     // the scalars are collected. That should be a safe assumption in most
     // cases, because we check if the operands have vectorizable types
     // beforehand in LoopVectorizationLegality.
-    return Scalars.find(VF) == Scalars.end() ||
-           !isScalarAfterVectorization(I, VF);
+    return !Scalars.contains(VF) || !isScalarAfterVectorization(I, VF);
   };
 
   /// Returns a range containing only operands needing to be extracted.
@@ -2250,73 +2247,6 @@ static void collectSupportedLoops(Loop &L, LoopInfo *LI,
     collectSupportedLoops(*InnerL, LI, ORE, V);
 }
 
-namespace {
-
-/// The LoopVectorize Pass.
-struct LoopVectorize : public FunctionPass {
-  /// Pass identification, replacement for typeid
-  static char ID;
-
-  LoopVectorizePass Impl;
-
-  explicit LoopVectorize(bool InterleaveOnlyWhenForced = false,
-                         bool VectorizeOnlyWhenForced = false)
-      : FunctionPass(ID),
-        Impl({InterleaveOnlyWhenForced, VectorizeOnlyWhenForced}) {
-    initializeLoopVectorizePass(*PassRegistry::getPassRegistry());
-  }
-
-  bool runOnFunction(Function &F) override {
-    if (skipFunction(F))
-      return false;
-
-    auto *SE = &getAnalysis<ScalarEvolutionWrapperPass>().getSE();
-    auto *LI = &getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
-    auto *TTI = &getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
-    auto *DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
-    auto *BFI = &getAnalysis<BlockFrequencyInfoWrapperPass>().getBFI();
-    auto *TLIP = getAnalysisIfAvailable<TargetLibraryInfoWrapperPass>();
-    auto *TLI = TLIP ? &TLIP->getTLI(F) : nullptr;
-    auto *AC = &getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
-    auto &LAIs = getAnalysis<LoopAccessLegacyAnalysis>().getLAIs();
-    auto *DB = &getAnalysis<DemandedBitsWrapperPass>().getDemandedBits();
-    auto *ORE = &getAnalysis<OptimizationRemarkEmitterWrapperPass>().getORE();
-    auto *PSI = &getAnalysis<ProfileSummaryInfoWrapperPass>().getPSI();
-
-    return Impl
-        .runImpl(F, *SE, *LI, *TTI, *DT, BFI, TLI, *DB, *AC, LAIs, *ORE, PSI)
-        .MadeAnyChange;
-  }
-
-  void getAnalysisUsage(AnalysisUsage &AU) const override {
-    AU.addRequired<AssumptionCacheTracker>();
-    AU.addRequired<BlockFrequencyInfoWrapperPass>();
-    AU.addRequired<DominatorTreeWrapperPass>();
-    AU.addRequired<LoopInfoWrapperPass>();
-    AU.addRequired<ScalarEvolutionWrapperPass>();
-    AU.addRequired<TargetTransformInfoWrapperPass>();
-    AU.addRequired<LoopAccessLegacyAnalysis>();
-    AU.addRequired<DemandedBitsWrapperPass>();
-    AU.addRequired<OptimizationRemarkEmitterWrapperPass>();
-    AU.addRequired<InjectTLIMappingsLegacy>();
-
-    // We currently do not preserve loopinfo/dominator analyses with outer loop
-    // vectorization. Until this is addressed, mark these analyses as preserved
-    // only for non-VPlan-native path.
-    // TODO: Preserve Loop and Dominator analyses for VPlan-native path.
-    if (!EnableVPlanNativePath) {
-      AU.addPreserved<LoopInfoWrapperPass>();
-      AU.addPreserved<DominatorTreeWrapperPass>();
-    }
-
-    AU.addPreserved<BasicAAWrapperPass>();
-    AU.addPreserved<GlobalsAAWrapperPass>();
-    AU.addRequired<ProfileSummaryInfoWrapperPass>();
-  }
-};
-
-} // end anonymous namespace
-
 //===----------------------------------------------------------------------===//
 // Implementation of LoopVectorizationLegality, InnerLoopVectorizer and
 // LoopVectorizationCostModel and LoopVectorizationPlanner.
@@ -2885,7 +2815,6 @@ void InnerLoopVectorizer::vectorizeInterleaveGroup(
 void InnerLoopVectorizer::scalarizeInstruction(const Instruction *Instr,
                                                VPReplicateRecipe *RepRecipe,
                                                const VPIteration &Instance,
-                                               bool IfPredicateInstr,
                                                VPTransformState &State) {
   assert(!Instr->getType()->isAggregateType() && "Can't handle vectors");
 
@@ -2935,6 +2864,7 @@ void InnerLoopVectorizer::scalarizeInstruction(const Instruction *Instr,
     AC->registerAssumption(II);
 
   // End if-block.
+  bool IfPredicateInstr = RepRecipe->getParent()->getParent()->isReplicator();
   if (IfPredicateInstr)
     PredicatedInstructions.push_back(Cloned);
 }
@@ -3022,10 +2952,10 @@ InnerLoopVectorizer::getOrCreateVectorTripCount(BasicBlock *InsertBlock) {
 Value *InnerLoopVectorizer::createBitOrPointerCast(Value *V, VectorType *DstVTy,
                                                    const DataLayout &DL) {
   // Verify that V is a vector type with same number of elements as DstVTy.
-  auto *DstFVTy = cast<FixedVectorType>(DstVTy);
-  unsigned VF = DstFVTy->getNumElements();
-  auto *SrcVecTy = cast<FixedVectorType>(V->getType());
-  assert((VF == SrcVecTy->getNumElements()) && "Vector dimensions do not match");
+  auto *DstFVTy = cast<VectorType>(DstVTy);
+  auto VF = DstFVTy->getElementCount();
+  auto *SrcVecTy = cast<VectorType>(V->getType());
+  assert(VF == SrcVecTy->getElementCount() && "Vector dimensions do not match");
   Type *SrcElemTy = SrcVecTy->getElementType();
   Type *DstElemTy = DstFVTy->getElementType();
   assert((DL.getTypeSizeInBits(SrcElemTy) == DL.getTypeSizeInBits(DstElemTy)) &&
@@ -3045,7 +2975,7 @@ Value *InnerLoopVectorizer::createBitOrPointerCast(Value *V, VectorType *DstVTy,
          "Only one type should be a floating point type");
   Type *IntTy =
       IntegerType::getIntNTy(V->getContext(), DL.getTypeSizeInBits(SrcElemTy));
-  auto *VecIntTy = FixedVectorType::get(IntTy, VF);
+  auto *VecIntTy = VectorType::get(IntTy, VF);
   Value *CastVal = Builder.CreateBitOrPointerCast(V, VecIntTy);
   return Builder.CreateBitOrPointerCast(CastVal, DstFVTy);
 }
@@ -4302,7 +4232,7 @@ void LoopVectorizationCostModel::collectLoopScalars(ElementCount VF) {
   // We should not collect Scalars more than once per VF. Right now, this
   // function is called from collectUniformsAndScalars(), which already does
   // this check. Collecting Scalars for VF=1 does not make any sense.
-  assert(VF.isVector() && Scalars.find(VF) == Scalars.end() &&
+  assert(VF.isVector() && !Scalars.contains(VF) &&
          "This function should not be visited twice for the same VF");
 
   // This avoids any chances of creating a REPLICATE recipe during planning
@@ -4729,7 +4659,7 @@ void LoopVectorizationCostModel::collectLoopUniforms(ElementCount VF) {
   // already does this check. Collecting Uniforms for VF=1 does not make any
   // sense.
 
-  assert(VF.isVector() && Uniforms.find(VF) == Uniforms.end() &&
+  assert(VF.isVector() && !Uniforms.contains(VF) &&
          "This function should not be visited twice for the same VF");
 
   // Visit the list of Uniforms. If we'll not find any uniform value, we'll
@@ -4804,11 +4734,17 @@ void LoopVectorizationCostModel::collectLoopUniforms(ElementCount VF) {
             WideningDecision == CM_Interleave);
   };
 
-
   // Returns true if Ptr is the pointer operand of a memory access instruction
-  // I, and I is known to not require scalarization.
+  // I, I is known to not require scalarization, and the pointer is not also
+  // stored.
   auto isVectorizedMemAccessUse = [&](Instruction *I, Value *Ptr) -> bool {
-    return getLoadStorePointerOperand(I) == Ptr && isUniformDecision(I, VF);
+    auto GetStoredValue = [I]() -> Value * {
+      if (!isa<StoreInst>(I))
+        return nullptr;
+      return I->getOperand(0);
+    };
+    return getLoadStorePointerOperand(I) == Ptr && isUniformDecision(I, VF) &&
+           GetStoredValue() != Ptr;
   };
 
   // Holds a list of values which are known to have at least one uniform use.
@@ -4854,8 +4790,8 @@ void LoopVectorizationCostModel::collectLoopUniforms(ElementCount VF) {
       if (isUniformMemOpUse(&I))
         addToWorklistIfAllowed(&I);
 
-      if (isUniformDecision(&I, VF)) {
-        assert(isVectorizedMemAccessUse(&I, Ptr) && "consistency check");
+      if (isVectorizedMemAccessUse(&I, Ptr)) {
+        assert(isUniformDecision(&I, VF) && "consistency check");
         HasUniformUse.insert(Ptr);
       }
     }
@@ -6072,7 +6008,7 @@ LoopVectorizationCostModel::calculateRegisterUsage(ArrayRef<ElementCount> VFs) {
   // Saves the list of values that are used in the loop but are defined outside
   // the loop (not including non-instruction values such as arguments and
   // constants).
-  SmallPtrSet<Instruction *, 8> LoopInvariants;
+  SmallSetVector<Instruction *, 8> LoopInvariants;
 
   for (BasicBlock *BB : make_range(DFS.beginRPO(), DFS.endRPO())) {
     for (Instruction &I : BB->instructionsWithoutDebug()) {
@@ -6258,8 +6194,7 @@ void LoopVectorizationCostModel::collectInstsToScalarize(ElementCount VF) {
   // instructions to scalarize, there's nothing to do. Collection may already
   // have occurred if we have a user-selected VF and are now computing the
   // expected cost for interleaving.
-  if (VF.isScalar() || VF.isZero() ||
-      InstsToScalarize.find(VF) != InstsToScalarize.end())
+  if (VF.isScalar() || VF.isZero() || InstsToScalarize.contains(VF))
     return;
 
   // Initialize a mapping for VF in InstsToScalalarize. If we find that it's
@@ -6348,7 +6283,7 @@ InstructionCost LoopVectorizationCostModel::computePredInstDiscount(
     Instruction *I = Worklist.pop_back_val();
 
     // If we've already analyzed the instruction, there's nothing to do.
-    if (ScalarCosts.find(I) != ScalarCosts.end())
+    if (ScalarCosts.contains(I))
       continue;
 
     // Compute the cost of the vector instruction. Note that this cost already
@@ -7463,37 +7398,6 @@ LoopVectorizationCostModel::getInstructionCost(Instruction *I, ElementCount VF,
   } // end of switch.
 }
 
-char LoopVectorize::ID = 0;
-
-static const char lv_name[] = "Loop Vectorization";
-
-INITIALIZE_PASS_BEGIN(LoopVectorize, LV_NAME, lv_name, false, false)
-INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(BasicAAWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(GlobalsAAWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
-INITIALIZE_PASS_DEPENDENCY(BlockFrequencyInfoWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(ScalarEvolutionWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(LoopInfoWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(LoopAccessLegacyAnalysis)
-INITIALIZE_PASS_DEPENDENCY(DemandedBitsWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(OptimizationRemarkEmitterWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(ProfileSummaryInfoWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(InjectTLIMappingsLegacy)
-INITIALIZE_PASS_END(LoopVectorize, LV_NAME, lv_name, false, false)
-
-namespace llvm {
-
-Pass *createLoopVectorizePass() { return new LoopVectorize(); }
-
-Pass *createLoopVectorizePass(bool InterleaveOnlyWhenForced,
-                              bool VectorizeOnlyWhenForced) {
-  return new LoopVectorize(InterleaveOnlyWhenForced, VectorizeOnlyWhenForced);
-}
-
-} // end namespace llvm
-
 void LoopVectorizationCostModel::collectValuesToIgnore() {
   // Ignore ephemeral values.
   CodeMetrics::collectEphemeralValues(TheLoop, AC, ValuesToIgnore);
@@ -8622,9 +8526,9 @@ void VPRecipeBuilder::fixHeaderPhis() {
   }
 }
 
-VPBasicBlock *VPRecipeBuilder::handleReplication(Instruction *I, VFRange &Range,
-                                                 VPBasicBlock *VPBB,
-                                                 VPlan &Plan) {
+VPRecipeOrVPValueTy VPRecipeBuilder::handleReplication(Instruction *I,
+                                                       VFRange &Range,
+                                                       VPlan &Plan) {
   bool IsUniform = LoopVectorizationPlanner::getDecisionAndClampRange(
       [&](ElementCount VF) { return CM.isUniformAfterVectorization(I, VF); },
       Range);
@@ -8661,68 +8565,22 @@ VPBasicBlock *VPRecipeBuilder::handleReplication(Instruction *I, VFRange &Range,
       break;
     }
   }
-
-  auto *Recipe = new VPReplicateRecipe(I, Plan.mapToVPValues(I->operands()),
-                                       IsUniform, IsPredicated);
-
-  // Finalize the recipe for Instr, first if it is not predicated.
+  VPValue *BlockInMask = nullptr;
   if (!IsPredicated) {
+    // Finalize the recipe for Instr, first if it is not predicated.
     LLVM_DEBUG(dbgs() << "LV: Scalarizing:" << *I << "\n");
-    setRecipe(I, Recipe);
-    Plan.addVPValue(I, Recipe);
-    VPBB->appendRecipe(Recipe);
-    return VPBB;
-  }
-  LLVM_DEBUG(dbgs() << "LV: Scalarizing and predicating:" << *I << "\n");
-
-  VPBlockBase *SingleSucc = VPBB->getSingleSuccessor();
-  assert(SingleSucc && "VPBB must have a single successor when handling "
-                       "predicated replication.");
-  VPBlockUtils::disconnectBlocks(VPBB, SingleSucc);
-  // Record predicated instructions for above packing optimizations.
-  VPBlockBase *Region = createReplicateRegion(Recipe, Plan);
-  VPBlockUtils::insertBlockAfter(Region, VPBB);
-  auto *RegSucc = new VPBasicBlock();
-  VPBlockUtils::insertBlockAfter(RegSucc, Region);
-  VPBlockUtils::connectBlocks(RegSucc, SingleSucc);
-  return RegSucc;
-}
-
-VPRegionBlock *
-VPRecipeBuilder::createReplicateRegion(VPReplicateRecipe *PredRecipe,
-                                       VPlan &Plan) {
-  Instruction *Instr = PredRecipe->getUnderlyingInstr();
-  // Instructions marked for predication are replicated and placed under an
-  // if-then construct to prevent side-effects.
-  // Generate recipes to compute the block mask for this region.
-  VPValue *BlockInMask = createBlockInMask(Instr->getParent(), Plan);
-
-  // Build the triangular if-then region.
-  std::string RegionName = (Twine("pred.") + Instr->getOpcodeName()).str();
-  assert(Instr->getParent() && "Predicated instruction not in any basic block");
-  auto *BOMRecipe = new VPBranchOnMaskRecipe(BlockInMask);
-  auto *Entry = new VPBasicBlock(Twine(RegionName) + ".entry", BOMRecipe);
-  auto *PHIRecipe = Instr->getType()->isVoidTy()
-                        ? nullptr
-                        : new VPPredInstPHIRecipe(PredRecipe);
-  if (PHIRecipe) {
-    setRecipe(Instr, PHIRecipe);
-    Plan.addVPValue(Instr, PHIRecipe);
   } else {
-    setRecipe(Instr, PredRecipe);
-    Plan.addVPValue(Instr, PredRecipe);
+    LLVM_DEBUG(dbgs() << "LV: Scalarizing and predicating:" << *I << "\n");
+    // Instructions marked for predication are replicated and a mask operand is
+    // added initially. Masked replicate recipes will later be placed under an
+    // if-then construct to prevent side-effects. Generate recipes to compute
+    // the block mask for this region.
+    BlockInMask = createBlockInMask(I->getParent(), Plan);
   }
 
-  auto *Exiting = new VPBasicBlock(Twine(RegionName) + ".continue", PHIRecipe);
-  auto *Pred = new VPBasicBlock(Twine(RegionName) + ".if", PredRecipe);
-  VPRegionBlock *Region = new VPRegionBlock(Entry, Exiting, RegionName, true);
-
-  // Note: first set Entry as region entry and then connect successors starting
-  // from it in order, to propagate the "parent" of each VPBasicBlock.
-  VPBlockUtils::insertTwoBlocksAfter(Pred, Exiting, Entry);
-  VPBlockUtils::connectBlocks(Pred, Exiting);
-
-  return Region;
+  auto *Recipe = new VPReplicateRecipe(I, Plan.mapToVPValues(I->operands()),
+                                       IsUniform, BlockInMask);
+  return toVPRecipeResult(Recipe);
 }
 
 VPRecipeOrVPValueTy
@@ -8798,13 +8656,11 @@ VPRecipeBuilder::tryToCreateWidenRecipe(Instruction *Instr,
 
   if (auto GEP = dyn_cast<GetElementPtrInst>(Instr))
     return toVPRecipeResult(new VPWidenGEPRecipe(
-        GEP, make_range(Operands.begin(), Operands.end()), OrigLoop));
+        GEP, make_range(Operands.begin(), Operands.end())));
 
   if (auto *SI = dyn_cast<SelectInst>(Instr)) {
-    bool InvariantCond =
-        PSE.getSE()->isLoopInvariant(PSE.getSCEV(SI->getOperand(0)), OrigLoop);
     return toVPRecipeResult(new VPWidenSelectRecipe(
-        *SI, make_range(Operands.begin(), Operands.end()), InvariantCond));
+        *SI, make_range(Operands.begin(), Operands.end())));
   }
 
   return toVPRecipeResult(tryToWiden(Instr, Operands, VPBB, Plan));
@@ -9046,7 +8902,6 @@ VPlanPtr LoopVectorizationPlanner::buildVPlanWithVPRecipes(
   for (BasicBlock *BB : make_range(DFS.beginRPO(), DFS.endRPO())) {
     // Relevant instructions from basic block BB will be grouped into VPRecipe
     // ingredients and fill a new VPBasicBlock.
-    unsigned VPBBsForBB = 0;
     if (VPBB != HeaderVPBB)
       VPBB->setName(BB->getName());
     Builder.setInsertPoint(VPBB);
@@ -9078,46 +8933,36 @@ VPlanPtr LoopVectorizationPlanner::buildVPlanWithVPRecipes(
           Legal->isInvariantAddressOfReduction(SI->getPointerOperand()))
         continue;
 
-      if (auto RecipeOrValue = RecipeBuilder.tryToCreateWidenRecipe(
-              Instr, Operands, Range, VPBB, Plan)) {
-        // If Instr can be simplified to an existing VPValue, use it.
-        if (RecipeOrValue.is<VPValue *>()) {
-          auto *VPV = RecipeOrValue.get<VPValue *>();
-          Plan->addVPValue(Instr, VPV);
-          // If the re-used value is a recipe, register the recipe for the
-          // instruction, in case the recipe for Instr needs to be recorded.
-          if (VPRecipeBase *R = VPV->getDefiningRecipe())
-            RecipeBuilder.setRecipe(Instr, R);
-          continue;
-        }
-        // Otherwise, add the new recipe.
-        VPRecipeBase *Recipe = RecipeOrValue.get<VPRecipeBase *>();
-        for (auto *Def : Recipe->definedValues()) {
-          auto *UV = Def->getUnderlyingValue();
-          Plan->addVPValue(UV, Def);
-        }
-
-        RecipeBuilder.setRecipe(Instr, Recipe);
-        if (isa<VPWidenIntOrFpInductionRecipe>(Recipe) &&
-            HeaderVPBB->getFirstNonPhi() != VPBB->end()) {
-          // Move VPWidenIntOrFpInductionRecipes for optimized truncates to the
-          // phi section of HeaderVPBB.
-          assert(isa<TruncInst>(Instr));
-          Recipe->insertBefore(*HeaderVPBB, HeaderVPBB->getFirstNonPhi());
-        } else
-          VPBB->appendRecipe(Recipe);
+      auto RecipeOrValue = RecipeBuilder.tryToCreateWidenRecipe(
+          Instr, Operands, Range, VPBB, Plan);
+      if (!RecipeOrValue)
+        RecipeOrValue = RecipeBuilder.handleReplication(Instr, Range, *Plan);
+      // If Instr can be simplified to an existing VPValue, use it.
+      if (RecipeOrValue.is<VPValue *>()) {
+        auto *VPV = RecipeOrValue.get<VPValue *>();
+        Plan->addVPValue(Instr, VPV);
+        // If the re-used value is a recipe, register the recipe for the
+        // instruction, in case the recipe for Instr needs to be recorded.
+        if (VPRecipeBase *R = VPV->getDefiningRecipe())
+          RecipeBuilder.setRecipe(Instr, R);
         continue;
       }
-
-      // Otherwise, if all widening options failed, Instruction is to be
-      // replicated. This may create a successor for VPBB.
-      VPBasicBlock *NextVPBB =
-          RecipeBuilder.handleReplication(Instr, Range, VPBB, *Plan);
-      if (NextVPBB != VPBB) {
-        VPBB = NextVPBB;
-        VPBB->setName(BB->hasName() ? BB->getName() + "." + Twine(VPBBsForBB++)
-                                    : "");
+      // Otherwise, add the new recipe.
+      VPRecipeBase *Recipe = RecipeOrValue.get<VPRecipeBase *>();
+      for (auto *Def : Recipe->definedValues()) {
+        auto *UV = Def->getUnderlyingValue();
+        Plan->addVPValue(UV, Def);
       }
+
+      RecipeBuilder.setRecipe(Instr, Recipe);
+      if (isa<VPWidenIntOrFpInductionRecipe>(Recipe) &&
+          HeaderVPBB->getFirstNonPhi() != VPBB->end()) {
+        // Move VPWidenIntOrFpInductionRecipes for optimized truncates to the
+        // phi section of HeaderVPBB.
+        assert(isa<TruncInst>(Instr));
+        Recipe->insertBefore(*HeaderVPBB, HeaderVPBB->getFirstNonPhi());
+      } else
+        VPBB->appendRecipe(Recipe);
     }
 
     VPBlockUtils::insertBlockAfter(new VPBasicBlock(), VPBB);
@@ -9194,13 +9039,7 @@ VPlanPtr LoopVectorizationPlanner::buildVPlanWithVPRecipes(
   VPlanTransforms::optimizeInductions(*Plan, *PSE.getSE());
   VPlanTransforms::removeDeadRecipes(*Plan);
 
-  bool ShouldSimplify = true;
-  while (ShouldSimplify) {
-    ShouldSimplify = VPlanTransforms::sinkScalarOperands(*Plan);
-    ShouldSimplify |=
-        VPlanTransforms::mergeReplicateRegionsIntoSuccessors(*Plan);
-    ShouldSimplify |= VPlanTransforms::mergeBlocksIntoPredecessors(*Plan);
-  }
+  VPlanTransforms::createAndOptimizeReplicateRegions(*Plan);
 
   VPlanTransforms::removeRedundantExpandSCEVRecipes(*Plan);
   VPlanTransforms::mergeBlocksIntoPredecessors(*Plan);
@@ -9230,7 +9069,7 @@ VPlanPtr LoopVectorizationPlanner::buildVPlan(VFRange &Range) {
 
   SmallPtrSet<Instruction *, 1> DeadInstructions;
   VPlanTransforms::VPInstructionsToVPRecipes(
-      OrigLoop, Plan,
+      Plan,
       [this](PHINode *P) { return Legal->getIntOrFpInductionDescriptor(P); },
       DeadInstructions, *PSE.getSE(), *TLI);
 
@@ -9683,8 +9522,7 @@ void VPReplicateRecipe::execute(VPTransformState &State) {
   Instruction *UI = getUnderlyingInstr();
   if (State.Instance) { // Generate a single instance.
     assert(!State.VF.isScalable() && "Can't scalarize a scalable vector");
-    State.ILV->scalarizeInstruction(UI, this, *State.Instance,
-                                    IsPredicated, State);
+    State.ILV->scalarizeInstruction(UI, this, *State.Instance, State);
     // Insert scalar instance packing it into a vector.
     if (State.VF.isVector() && shouldPack()) {
       // If we're constructing lane 0, initialize to start from poison.
@@ -9706,8 +9544,7 @@ void VPReplicateRecipe::execute(VPTransformState &State) {
         all_of(operands(), [](VPValue *Op) {
           return Op->isDefinedOutsideVectorRegions();
         })) {
-      State.ILV->scalarizeInstruction(UI, this, VPIteration(0, 0), IsPredicated,
-                                      State);
+      State.ILV->scalarizeInstruction(UI, this, VPIteration(0, 0), State);
       if (user_begin() != user_end()) {
         for (unsigned Part = 1; Part < State.UF; ++Part)
           State.set(this, State.get(this, VPIteration(0, 0)),
@@ -9719,8 +9556,7 @@ void VPReplicateRecipe::execute(VPTransformState &State) {
     // Uniform within VL means we need to generate lane 0 only for each
     // unrolled copy.
     for (unsigned Part = 0; Part < State.UF; ++Part)
-      State.ILV->scalarizeInstruction(UI, this, VPIteration(Part, 0),
-                                      IsPredicated, State);
+      State.ILV->scalarizeInstruction(UI, this, VPIteration(Part, 0), State);
     return;
   }
 
@@ -9728,7 +9564,7 @@ void VPReplicateRecipe::execute(VPTransformState &State) {
   // needs only the last copy of the store.
   if (isa<StoreInst>(UI) && !getOperand(1)->hasDefiningRecipe()) {
     auto Lane = VPLane::getLastLaneForVF(State.VF);
-    State.ILV->scalarizeInstruction(UI, this, VPIteration(State.UF - 1, Lane), IsPredicated,
+    State.ILV->scalarizeInstruction(UI, this, VPIteration(State.UF - 1, Lane),
                                     State);
     return;
   }
@@ -9738,8 +9574,7 @@ void VPReplicateRecipe::execute(VPTransformState &State) {
   const unsigned EndLane = State.VF.getKnownMinValue();
   for (unsigned Part = 0; Part < State.UF; ++Part)
     for (unsigned Lane = 0; Lane < EndLane; ++Lane)
-      State.ILV->scalarizeInstruction(UI, this, VPIteration(Part, Lane),
-                                      IsPredicated, State);
+      State.ILV->scalarizeInstruction(UI, this, VPIteration(Part, Lane), State);
 }
 
 void VPWidenMemoryInstructionRecipe::execute(VPTransformState &State) {
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 3eaa627e0244f..9da2fd4bbabe5 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -349,16 +349,16 @@ enum class UseMask {
 static SmallBitVector buildUseMask(int VF, ArrayRef<int> Mask,
                                    UseMask MaskArg) {
   SmallBitVector UseMask(VF, true);
-  for (auto P : enumerate(Mask)) {
-    if (P.value() == UndefMaskElem) {
+  for (auto [Idx, Value] : enumerate(Mask)) {
+    if (Value == UndefMaskElem) {
       if (MaskArg == UseMask::UndefsAsMask)
-        UseMask.reset(P.index());
+        UseMask.reset(Idx);
       continue;
     }
-    if (MaskArg == UseMask::FirstArg && P.value() < VF)
-      UseMask.reset(P.value());
-    else if (MaskArg == UseMask::SecondArg && P.value() >= VF)
-      UseMask.reset(P.value() - VF);
+    if (MaskArg == UseMask::FirstArg && Value < VF)
+      UseMask.reset(Value);
+    else if (MaskArg == UseMask::SecondArg && Value >= VF)
+      UseMask.reset(Value - VF);
   }
   return UseMask;
 }
@@ -1182,6 +1182,7 @@ class BoUpSLP {
     MinBWs.clear();
     InstrElementSize.clear();
     UserIgnoreList = nullptr;
+    PostponedGathers.clear();
   }
 
   unsigned getTreeSize() const { return VectorizableTree.size(); }
@@ -2488,8 +2489,9 @@ class BoUpSLP {
   /// the bundle
   void setInsertPointAfterBundle(const TreeEntry *E);
 
-  /// \returns a vector from a collection of scalars in \p VL.
-  Value *gather(ArrayRef<Value *> VL);
+  /// \returns a vector from a collection of scalars in \p VL. if \p Root is not
+  /// specified, the starting vector value is poison.
+  Value *gather(ArrayRef<Value *> VL, Value *Root = nullptr);
 
   /// \returns whether the VectorizableTree is fully vectorizable and will
   /// be beneficial even the tree height is tiny.
@@ -2938,6 +2940,11 @@ class BoUpSLP {
   /// pre-gather them before.
   DenseMap<const TreeEntry *, Instruction *> EntryToLastInstruction;
 
+  /// List of gather nodes, depending on other gather/vector nodes, which should
+  /// be emitted after the vector instruction emission process to correctly
+  /// handle order of the vector instructions and shuffles.
+  SetVector<const TreeEntry *> PostponedGathers;
+
   /// This POD struct describes one external user in the vectorized tree.
   struct ExternalUser {
     ExternalUser(Value *S, llvm::User *U, int L)
@@ -3953,7 +3960,7 @@ bool clusterSortPtrAccesses(ArrayRef<Value *> VL, Type *ElemTy,
         return std::get<1>(X) < std::get<1>(Y);
       });
       int InitialOffset = std::get<1>(Vec[0]);
-      AnyConsecutive |= all_of(enumerate(Vec), [InitialOffset](auto &P) {
+      AnyConsecutive |= all_of(enumerate(Vec), [InitialOffset](const auto &P) {
         return std::get<1>(P.value()) == int(P.index()) + InitialOffset;
       });
     }
@@ -6590,9 +6597,10 @@ class BaseShuffleAnalysis {
         LocalVF = SVOpTy->getNumElements();
       SmallVector<int> ExtMask(Mask.size(), UndefMaskElem);
       for (auto [Idx, I] : enumerate(Mask)) {
-         if (I == UndefMaskElem)
-           continue;
-         ExtMask[Idx] = SV->getMaskValue(I);
+        if (I == UndefMaskElem ||
+            static_cast<unsigned>(I) >= SV->getShuffleMask().size())
+          continue;
+        ExtMask[Idx] = SV->getMaskValue(I);
       }
       bool IsOp1Undef =
           isUndefVector(SV->getOperand(0),
@@ -6623,7 +6631,8 @@ class BaseShuffleAnalysis {
         Op = SV->getOperand(1);
     }
     if (auto *OpTy = dyn_cast<FixedVectorType>(Op->getType());
-        !OpTy || !isIdentityMask(Mask, OpTy, SinglePermute)) {
+        !OpTy || !isIdentityMask(Mask, OpTy, SinglePermute) ||
+        ShuffleVectorInst::isZeroEltSplatMask(Mask)) {
       if (IdentityOp) {
         V = IdentityOp;
         assert(Mask.size() == IdentityMask.size() &&
@@ -6655,6 +6664,8 @@ class BaseShuffleAnalysis {
   static Value *createShuffle(Value *V1, Value *V2, ArrayRef<int> Mask,
                               ShuffleBuilderTy &Builder) {
     assert(V1 && "Expected at least one vector value.");
+    if (V2)
+      Builder.resizeToMatch(V1, V2);
     int VF = Mask.size();
     if (auto *FTy = dyn_cast<FixedVectorType>(V1->getType()))
       VF = FTy->getNumElements();
@@ -6742,6 +6753,15 @@ class BaseShuffleAnalysis {
           CombinedMask1[I] = CombinedMask2[I] + (Op1 == Op2 ? 0 : VF);
         }
       }
+      const int Limit = CombinedMask1.size() * 2;
+      if (Op1 == Op2 && Limit == 2 * VF &&
+          all_of(CombinedMask1, [=](int Idx) { return Idx < Limit; }) &&
+          (ShuffleVectorInst::isIdentityMask(CombinedMask1) ||
+           (ShuffleVectorInst::isZeroEltSplatMask(CombinedMask1) &&
+            isa<ShuffleVectorInst>(Op1) &&
+            cast<ShuffleVectorInst>(Op1)->getShuffleMask() ==
+                ArrayRef(CombinedMask1))))
+        return Op1;
       return Builder.createShuffleVector(
           Op1, Op1 == Op2 ? PoisonValue::get(Op1->getType()) : Op2,
           CombinedMask1);
@@ -6989,7 +7009,8 @@ InstructionCost BoUpSLP::getEntryCost(const TreeEntry *E,
         return TTI::TCC_Free;
       // Add broadcast for non-identity shuffle only.
       bool NeedShuffle =
-          VL.front() != *It || !all_of(VL.drop_front(), UndefValue::classof);
+          count(VL, *It) > 1 &&
+          (VL.front() != *It || !all_of(VL.drop_front(), UndefValue::classof));
       InstructionCost InsertCost = TTI->getVectorInstrCost(
           Instruction::InsertElement, VecTy, CostKind,
           NeedShuffle ? 0 : std::distance(VL.begin(), It),
@@ -6999,7 +7020,7 @@ InstructionCost BoUpSLP::getEntryCost(const TreeEntry *E,
                                      TargetTransformInfo::SK_Broadcast, VecTy,
                                      /*Mask=*/std::nullopt, CostKind,
                                      /*Index=*/0,
-                                     /*SubTp=*/nullptr, /*Args=*/VL[0])
+                                     /*SubTp=*/nullptr, /*Args=*/*It)
                                : TTI::TCC_Free);
     }
     InstructionCost ReuseShuffleCost = 0;
@@ -7151,37 +7172,85 @@ InstructionCost BoUpSLP::getEntryCost(const TreeEntry *E,
   // Calculate cost difference from vectorizing set of GEPs.
   // Negative value means vectorizing is profitable.
   auto GetGEPCostDiff = [=](ArrayRef<Value *> Ptrs, Value *BasePtr) {
-    InstructionCost CostSavings = 0;
-    for (Value *V : Ptrs) {
-      if (V == BasePtr)
-        continue;
-      auto *Ptr = dyn_cast<GetElementPtrInst>(V);
-      // GEPs may contain just addresses without instructions, considered free.
-      // GEPs with all constant indices also considered to have zero cost.
-      if (!Ptr || Ptr->hasAllConstantIndices())
-        continue;
-
-      // Here we differentiate two cases: when GEPs represent a regular
-      // vectorization tree node (and hence vectorized) and when the set is
-      // arguments of a set of loads or stores being vectorized. In the former
-      // case all the scalar GEPs will be removed as a result of vectorization.
+    InstructionCost ScalarCost = 0;
+    InstructionCost VecCost = 0;
+    // Here we differentiate two cases: (1) when Ptrs represent a regular
+    // vectorization tree node (as they are pointer arguments of scattered
+    // loads) or (2) when Ptrs are the arguments of loads or stores being
+    // vectorized as plane wide unit-stride load/store since all the
+    // loads/stores are known to be from/to adjacent locations.
+    assert(E->State == TreeEntry::Vectorize &&
+           "Entry state expected to be Vectorize here.");
+    if (isa<LoadInst, StoreInst>(VL0)) {
+      // Case 2: estimate costs for pointer related costs when vectorizing to
+      // a wide load/store.
+      // Scalar cost is estimated as a set of pointers with known relationship
+      // between them.
+      // For vector code we will use BasePtr as argument for the wide load/store
+      // but we also need to account all the instructions which are going to
+      // stay in vectorized code due to uses outside of these scalar
+      // loads/stores.
+      ScalarCost = TTI->getPointersChainCost(
+          Ptrs, BasePtr, TTI::PointersChainInfo::getKnownUniformStrided(),
+          CostKind);
+
+      SmallVector<const Value *> PtrsRetainedInVecCode;
+      for (Value *V : Ptrs) {
+        if (V == BasePtr) {
+          PtrsRetainedInVecCode.push_back(V);
+          continue;
+        }
+        auto *Ptr = dyn_cast<GetElementPtrInst>(V);
+        // For simplicity assume Ptr to stay in vectorized code if it's not a
+        // GEP instruction. We don't care since it's cost considered free.
+        // TODO: We should check for any uses outside of vectorizable tree
+        // rather than just single use.
+        if (!Ptr || !Ptr->hasOneUse())
+          PtrsRetainedInVecCode.push_back(V);
+      }
+
+      if (PtrsRetainedInVecCode.size() == Ptrs.size()) {
+        // If all pointers stay in vectorized code then we don't have
+        // any savings on that.
+        LLVM_DEBUG(dumpTreeCosts(E, 0, ScalarCost, ScalarCost,
+                                 "Calculated GEPs cost for Tree"));
+        return InstructionCost{TTI::TCC_Free};
+      }
+      VecCost = TTI->getPointersChainCost(
+          PtrsRetainedInVecCode, BasePtr,
+          TTI::PointersChainInfo::getKnownNonUniformStrided(), CostKind);
+    } else {
+      // Case 1: Ptrs are the arguments of loads that we are going to transform
+      // into masked gather load intrinsic.
+      // All the scalar GEPs will be removed as a result of vectorization.
       // For any external uses of some lanes extract element instructions will
-      // be generated (which cost is estimated separately). For the latter case
-      // since the set of GEPs itself is not vectorized those used more than
-      // once will remain staying in vectorized code as well. So we should not
-      // count them as savings.
-      if (!Ptr->hasOneUse() && isa<LoadInst, StoreInst>(VL0))
-        continue;
+      // be generated (which cost is estimated separately).
+      TTI::PointersChainInfo PtrsInfo =
+          all_of(Ptrs,
+                 [](const Value *V) {
+                   auto *Ptr = dyn_cast<GetElementPtrInst>(V);
+                   return Ptr && !Ptr->hasAllConstantIndices();
+                 })
+              ? TTI::PointersChainInfo::getNonUniformStrided()
+              : TTI::PointersChainInfo::getKnownNonUniformStrided();
+
+      ScalarCost = TTI->getPointersChainCost(Ptrs, BasePtr, PtrsInfo, CostKind);
 
-      // TODO: it is target dependent, so need to implement and then use a TTI
-      // interface.
-      CostSavings += TTI->getArithmeticInstrCost(Instruction::Add,
-                                                 Ptr->getType(), CostKind);
+      // Remark: it not quite correct to use scalar GEP cost for a vector GEP,
+      // but it's not clear how to do that without having vector GEP arguments
+      // ready.
+      // Perhaps using just TTI::TCC_Free/TTI::TCC_Basic would be better option.
+      if (const auto *Base = dyn_cast<GetElementPtrInst>(BasePtr)) {
+        SmallVector<const Value *> Indices(Base->indices());
+        VecCost = TTI->getGEPCost(Base->getSourceElementType(),
+                                  Base->getPointerOperand(), Indices, CostKind);
+      }
     }
-    LLVM_DEBUG(dbgs() << "SLP: Calculated GEPs cost savings or Tree:\n";
-               E->dump());
-    LLVM_DEBUG(dbgs() << "SLP:     GEP cost saving = " << CostSavings << "\n");
-    return InstructionCost() - CostSavings;
+
+    LLVM_DEBUG(dumpTreeCosts(E, 0, VecCost, ScalarCost,
+                             "Calculated GEPs cost for Tree"));
+
+    return VecCost - ScalarCost;
   };
 
   switch (ShuffleOrOp) {
@@ -8363,7 +8432,9 @@ BoUpSLP::isGatherShuffledEntry(const TreeEntry *TE, ArrayRef<Value *> VL,
            "Expected only single user of the gather node.");
     Instruction &EntryUserInst =
         getLastInstructionInBundle(EntryPtr->UserTreeIndices.front().UserTE);
-    if (&UserInst == &EntryUserInst) {
+    PHINode *EntryPHI = dyn_cast<PHINode>(
+        EntryPtr->UserTreeIndices.front().UserTE->getMainOp());
+    if (&UserInst == &EntryUserInst && !EntryPHI) {
       // If 2 gathers are operands of the same entry, compare operands indices,
       // use the earlier one as the base.
       if (TE->UserTreeIndices.front().UserTE ==
@@ -8374,7 +8445,6 @@ BoUpSLP::isGatherShuffledEntry(const TreeEntry *TE, ArrayRef<Value *> VL,
     }
     // Check if the user node of the TE comes after user node of EntryPtr,
     // otherwise EntryPtr depends on TE.
-    auto *EntryPHI = dyn_cast<PHINode>(&EntryUserInst);
     auto *EntryI =
         EntryPHI
             ? EntryPHI
@@ -8396,7 +8466,7 @@ BoUpSLP::isGatherShuffledEntry(const TreeEntry *TE, ArrayRef<Value *> VL,
   // have a permutation of 2 input vectors.
   SmallVector<SmallPtrSet<const TreeEntry *, 4>> UsedTEs;
   DenseMap<Value *, int> UsedValuesEntry;
-  for (Value *V : TE->Scalars) {
+  for (Value *V : VL) {
     if (isConstant(V))
       continue;
     // Build a list of tree entries where V is used.
@@ -8404,8 +8474,12 @@ BoUpSLP::isGatherShuffledEntry(const TreeEntry *TE, ArrayRef<Value *> VL,
     auto It = ValueToTEs.find(V);
     if (It != ValueToTEs.end())
       VToTEs = It->second;
-    if (const TreeEntry *VTE = getTreeEntry(V))
+    if (const TreeEntry *VTE = getTreeEntry(V)) {
+      Instruction &EntryUserInst = getLastInstructionInBundle(VTE);
+      if (&EntryUserInst == &UserInst || !CheckOrdering(&EntryUserInst))
+        continue;
       VToTEs.insert(VTE);
+    }
     if (VToTEs.empty())
       continue;
     if (UsedTEs.empty()) {
@@ -8461,7 +8535,7 @@ BoUpSLP::isGatherShuffledEntry(const TreeEntry *TE, ArrayRef<Value *> VL,
     auto *It = find_if(FirstEntries, [=](const TreeEntry *EntryPtr) {
       return EntryPtr->isSame(VL) || EntryPtr->isSame(TE->Scalars);
     });
-    if (It != FirstEntries.end()) {
+    if (It != FirstEntries.end() && (*It)->getVectorFactor() == VL.size()) {
       Entries.push_back(*It);
       std::iota(Mask.begin(), Mask.end(), 0);
       // Clear undef scalars.
@@ -8503,10 +8577,18 @@ BoUpSLP::isGatherShuffledEntry(const TreeEntry *TE, ArrayRef<Value *> VL,
         break;
       }
     }
-    // No 2 source vectors with the same vector factor - give up and do regular
-    // gather.
-    if (Entries.empty())
-      return std::nullopt;
+    // No 2 source vectors with the same vector factor - just choose 2 with max
+    // index.
+    if (Entries.empty()) {
+      Entries.push_back(
+          *std::max_element(UsedTEs.front().begin(), UsedTEs.front().end(),
+                            [](const TreeEntry *TE1, const TreeEntry *TE2) {
+                              return TE1->Idx < TE2->Idx;
+                            }));
+      Entries.push_back(SecondEntries.front());
+      VF = std::max(Entries.front()->getVectorFactor(),
+                    Entries.back()->getVectorFactor());
+    }
   }
 
   bool IsSplatOrUndefs = isSplat(VL) || all_of(VL, UndefValue::classof);
@@ -8842,7 +8924,7 @@ void BoUpSLP::setInsertPointAfterBundle(const TreeEntry *E) {
   Builder.SetCurrentDebugLocation(Front->getDebugLoc());
 }
 
-Value *BoUpSLP::gather(ArrayRef<Value *> VL) {
+Value *BoUpSLP::gather(ArrayRef<Value *> VL, Value *Root) {
   // List of instructions/lanes from current block and/or the blocks which are
   // part of the current loop. These instructions will be inserted at the end to
   // make it possible to optimize loops and hoist invariant instructions out of
@@ -8859,7 +8941,8 @@ Value *BoUpSLP::gather(ArrayRef<Value *> VL) {
   for (int I = 0, E = VL.size(); I < E; ++I) {
     if (auto *Inst = dyn_cast<Instruction>(VL[I]))
       if ((CheckPredecessor(Inst->getParent(), Builder.GetInsertBlock()) ||
-           getTreeEntry(Inst) || (L && (L->contains(Inst)))) &&
+           getTreeEntry(Inst) ||
+           (L && (!Root || L->isLoopInvariant(Root)) && L->contains(Inst))) &&
           PostponedIndices.insert(I).second)
         PostponedInsts.emplace_back(Inst, I);
   }
@@ -8882,7 +8965,7 @@ Value *BoUpSLP::gather(ArrayRef<Value *> VL) {
   Value *Val0 =
       isa<StoreInst>(VL[0]) ? cast<StoreInst>(VL[0])->getValueOperand() : VL[0];
   FixedVectorType *VecTy = FixedVectorType::get(Val0->getType(), VL.size());
-  Value *Vec = PoisonValue::get(VecTy);
+  Value *Vec = Root ? Root : PoisonValue::get(VecTy);
   SmallVector<int> NonConsts;
   // Insert constant values at first.
   for (int I = 0, E = VL.size(); I < E; ++I) {
@@ -8892,6 +8975,18 @@ Value *BoUpSLP::gather(ArrayRef<Value *> VL) {
       NonConsts.push_back(I);
       continue;
     }
+    if (Root) {
+      if (!isa<UndefValue>(VL[I])) {
+        NonConsts.push_back(I);
+        continue;
+      }
+      if (isa<PoisonValue>(VL[I]))
+        continue;
+      if (auto *SV = dyn_cast<ShuffleVectorInst>(Root)) {
+        if (SV->getMaskValue(I) == UndefMaskElem)
+          continue;
+      }
+    }
     Vec = CreateInsertElement(Vec, VL[I], I);
   }
   // Insert non-constant values.
@@ -9130,9 +9225,34 @@ class BoUpSLP::ShuffleInstructionBuilder final : public BaseShuffleAnalysis {
     add(V1, NewMask);
   }
   /// Finalize emission of the shuffles.
+  /// \param Action the action (if any) to be performed before final applying of
+  /// the \p ExtMask mask.
   Value *
-  finalize(ArrayRef<int> ExtMask = std::nullopt) {
+  finalize(ArrayRef<int> ExtMask, unsigned VF = 0,
+           function_ref<void(Value *&, SmallVectorImpl<int> &)> Action = {}) {
     IsFinalized = true;
+    if (Action) {
+      Value *Vec = InVectors.front();
+      if (InVectors.size() == 2) {
+        Vec = createShuffle(Vec, InVectors.back(), CommonMask);
+        InVectors.pop_back();
+      } else {
+        Vec = createShuffle(Vec, nullptr, CommonMask);
+      }
+      for (unsigned Idx = 0, Sz = CommonMask.size(); Idx < Sz; ++Idx)
+        if (CommonMask[Idx] != UndefMaskElem)
+          CommonMask[Idx] = Idx;
+      assert(VF > 0 &&
+             "Expected vector length for the final value before action.");
+      unsigned VecVF = cast<FixedVectorType>(Vec->getType())->getNumElements();
+      if (VecVF < VF) {
+        SmallVector<int> ResizeMask(VF, UndefMaskElem);
+        std::iota(ResizeMask.begin(), std::next(ResizeMask.begin(), VecVF), 0);
+        Vec = createShuffle(Vec, nullptr, ResizeMask);
+      }
+      Action(Vec, CommonMask);
+      InVectors.front() = Vec;
+    }
     if (!ExtMask.empty()) {
       if (CommonMask.empty()) {
         CommonMask.assign(ExtMask.begin(), ExtMask.end());
@@ -9261,94 +9381,345 @@ Value *BoUpSLP::createBuildVector(const TreeEntry *E) {
   assert(E->State == TreeEntry::NeedToGather && "Expected gather node.");
   unsigned VF = E->getVectorFactor();
 
-  ShuffleInstructionBuilder ShuffleBuilder(Builder, *this);
-  SmallVector<Value *> Gathered(
-      VF, PoisonValue::get(E->Scalars.front()->getType()));
+  auto AdjustExtracts = [&](const TreeEntry *E, ArrayRef<int> Mask) {
+    Value *VecBase = nullptr;
+    for (int I = 0, Sz = Mask.size(); I < Sz; ++I) {
+      int Idx = Mask[I];
+      if (Idx == UndefMaskElem)
+        continue;
+      auto *EI = cast<ExtractElementInst>(E->Scalars[I]);
+      VecBase = EI->getVectorOperand();
+      // TODO: EI can be erased, if all its users are vectorized. But need to
+      // emit shuffles for such extractelement instructions.
+    }
+    return VecBase;
+  };
+  auto NeedToDelay = [=](const TreeEntry *E,
+                         ArrayRef<const TreeEntry *> Deps) -> Value * {
+    // No need to delay emission if all deps are ready.
+    if (all_of(Deps, [](const TreeEntry *TE) { return TE->VectorizedValue; }))
+      return nullptr;
+    // Postpone gather emission, will be emitted after the end of the
+    // process to keep correct order.
+    auto *VecTy = FixedVectorType::get(E->Scalars.front()->getType(),
+                                       E->getVectorFactor());
+    Value *Vec = Builder.CreateAlignedLoad(
+        VecTy, PoisonValue::get(VecTy->getPointerTo()), MaybeAlign());
+    return Vec;
+  };
+
   bool NeedFreeze = false;
-  SmallVector<Value *> VL(E->Scalars.begin(), E->Scalars.end());
-  // Build a mask out of the redorder indices and reorder scalars per this mask.
+  SmallVector<int> ReuseShuffleIndicies(E->ReuseShuffleIndices.begin(),
+                                        E->ReuseShuffleIndices.end());
+  SmallVector<Value *> GatheredScalars(E->Scalars.begin(), E->Scalars.end());
+  // Build a mask out of the reorder indices and reorder scalars per this
+  // mask.
   SmallVector<int> ReorderMask;
   inversePermutation(E->ReorderIndices, ReorderMask);
   if (!ReorderMask.empty())
-    reorderScalars(VL, ReorderMask);
-  SmallVector<int> ReuseMask(VF, UndefMaskElem);
-  if (!allConstant(VL)) {
+    reorderScalars(GatheredScalars, ReorderMask);
+  auto FindReusedSplat = [&](SmallVectorImpl<int> &Mask) {
+    if (!isSplat(E->Scalars) || none_of(E->Scalars, [](Value *V) {
+          return isa<UndefValue>(V) && !isa<PoisonValue>(V);
+        }))
+      return false;
+    TreeEntry *UserTE = E->UserTreeIndices.back().UserTE;
+    unsigned EdgeIdx = E->UserTreeIndices.back().EdgeIdx;
+    if (UserTE->getNumOperands() != 2)
+      return false;
+    auto *It =
+        find_if(VectorizableTree, [=](const std::unique_ptr<TreeEntry> &TE) {
+          return find_if(TE->UserTreeIndices, [=](const EdgeInfo &EI) {
+                   return EI.UserTE == UserTE && EI.EdgeIdx != EdgeIdx;
+                 }) != TE->UserTreeIndices.end();
+        });
+    if (It == VectorizableTree.end())
+      return false;
+    unsigned I =
+        *find_if_not(Mask, [](int Idx) { return Idx == UndefMaskElem; });
+    int Sz = Mask.size();
+    if (all_of(Mask, [Sz](int Idx) { return Idx < 2 * Sz; }) &&
+        ShuffleVectorInst::isIdentityMask(Mask))
+      std::iota(Mask.begin(), Mask.end(), 0);
+    else
+      std::fill(Mask.begin(), Mask.end(), I);
+    return true;
+  };
+  ShuffleInstructionBuilder ShuffleBuilder(Builder, *this);
+  Value *Vec = nullptr;
+  SmallVector<int> Mask;
+  SmallVector<int> ExtractMask;
+  SmallVector<int> ReuseMask;
+  std::optional<TargetTransformInfo::ShuffleKind> ExtractShuffle;
+  std::optional<TargetTransformInfo::ShuffleKind> GatherShuffle;
+  SmallVector<const TreeEntry *> Entries;
+  Type *ScalarTy = GatheredScalars.front()->getType();
+  if (!all_of(GatheredScalars, UndefValue::classof)) {
+    // Check for gathered extracts.
+    ExtractShuffle = tryToGatherExtractElements(GatheredScalars, ExtractMask);
+    SmallVector<Value *> IgnoredVals;
+    if (UserIgnoreList)
+      IgnoredVals.assign(UserIgnoreList->begin(), UserIgnoreList->end());
+    bool Resized = false;
+    if (Value *VecBase = AdjustExtracts(E, ExtractMask))
+      if (auto *VecBaseTy = dyn_cast<FixedVectorType>(VecBase->getType()))
+        if (VF == VecBaseTy->getNumElements() && GatheredScalars.size() != VF) {
+          Resized = true;
+          GatheredScalars.append(VF - GatheredScalars.size(),
+                                 PoisonValue::get(ScalarTy));
+        }
+    // Gather extracts after we check for full matched gathers only.
+    if (ExtractShuffle || E->getOpcode() != Instruction::Load ||
+        E->isAltShuffle() ||
+        all_of(E->Scalars, [this](Value *V) { return getTreeEntry(V); }) ||
+        isSplat(E->Scalars) ||
+        (E->Scalars != GatheredScalars && GatheredScalars.size() <= 2)) {
+      GatherShuffle = isGatherShuffledEntry(E, GatheredScalars, Mask, Entries);
+    }
+    if (GatherShuffle) {
+      if (Value *Delayed = NeedToDelay(E, Entries)) {
+        // Delay emission of gathers which are not ready yet.
+        PostponedGathers.insert(E);
+        // Postpone gather emission, will be emitted after the end of the
+        // process to keep correct order.
+        return Delayed;
+      }
+      assert((Entries.size() == 1 || Entries.size() == 2) &&
+             "Expected shuffle of 1 or 2 entries.");
+      if (!Resized) {
+        unsigned VF1 = Entries.front()->getVectorFactor();
+        unsigned VF2 = Entries.back()->getVectorFactor();
+        if ((VF == VF1 || VF == VF2) && GatheredScalars.size() != VF)
+          GatheredScalars.append(VF - GatheredScalars.size(),
+                                 PoisonValue::get(ScalarTy));
+      }
+      // Remove shuffled elements from list of gathers.
+      for (int I = 0, Sz = Mask.size(); I < Sz; ++I) {
+        if (Mask[I] != UndefMaskElem)
+          GatheredScalars[I] = PoisonValue::get(ScalarTy);
+      }
+    }
+  }
+  if (ExtractShuffle || GatherShuffle) {
+    bool IsNonPoisoned = true;
+    bool IsUsedInExpr = false;
+    Value *Vec1 = nullptr;
+    if (ExtractShuffle) {
+      // Gather of extractelements can be represented as just a shuffle of
+      // a single/two vectors the scalars are extracted from.
+      // Find input vectors.
+      Value *Vec2 = nullptr;
+      for (unsigned I = 0, Sz = ExtractMask.size(); I < Sz; ++I) {
+        if (ExtractMask[I] == UndefMaskElem ||
+            (!Mask.empty() && Mask[I] != UndefMaskElem)) {
+          ExtractMask[I] = UndefMaskElem;
+          continue;
+        }
+        if (isa<UndefValue>(E->Scalars[I]))
+          continue;
+        auto *EI = cast<ExtractElementInst>(E->Scalars[I]);
+        if (!Vec1) {
+          Vec1 = EI->getVectorOperand();
+        } else if (Vec1 != EI->getVectorOperand()) {
+          assert((!Vec2 || Vec2 == EI->getVectorOperand()) &&
+                 "Expected only 1 or 2 vectors shuffle.");
+          Vec2 = EI->getVectorOperand();
+        }
+      }
+      if (Vec2) {
+        ShuffleBuilder.add(Vec1, Vec2, ExtractMask);
+      } else if (Vec1) {
+        ShuffleBuilder.add(Vec1, ExtractMask);
+        IsNonPoisoned &= isGuaranteedNotToBePoison(Vec1);
+        IsUsedInExpr = FindReusedSplat(ExtractMask);
+      } else {
+        ShuffleBuilder.add(PoisonValue::get(FixedVectorType::get(
+                               ScalarTy, GatheredScalars.size())),
+                           ExtractMask);
+      }
+    }
+    if (GatherShuffle) {
+      if (Entries.size() == 1) {
+        ShuffleBuilder.add(Entries.front()->VectorizedValue, Mask);
+        IsNonPoisoned &=
+            isGuaranteedNotToBePoison(Entries.front()->VectorizedValue);
+        IsUsedInExpr = FindReusedSplat(Mask);
+      } else {
+        ShuffleBuilder.add(Entries.front()->VectorizedValue,
+                           Entries.back()->VectorizedValue, Mask);
+      }
+    }
+    // Try to figure out best way to combine values: build a shuffle and insert
+    // elements or just build several shuffles.
+    // Insert non-constant scalars.
+    SmallVector<Value *> NonConstants(GatheredScalars);
+    int EMSz = ExtractMask.size();
+    int MSz = Mask.size();
+    // Try to build constant vector and shuffle with it only if currently we
+    // have a single permutation and more than 1 scalar constants.
+    bool IsSingleShuffle = !ExtractShuffle || !GatherShuffle;
+    bool IsIdentityShuffle =
+        (ExtractShuffle.value_or(TTI::SK_PermuteTwoSrc) ==
+             TTI::SK_PermuteSingleSrc &&
+         none_of(ExtractMask, [&](int I) { return I >= EMSz; }) &&
+         ShuffleVectorInst::isIdentityMask(ExtractMask)) ||
+        (GatherShuffle.value_or(TTI::SK_PermuteTwoSrc) ==
+             TTI::SK_PermuteSingleSrc &&
+         none_of(Mask, [&](int I) { return I >= MSz; }) &&
+         ShuffleVectorInst::isIdentityMask(Mask));
+    bool EnoughConstsForShuffle =
+        IsSingleShuffle &&
+        (none_of(GatheredScalars,
+                 [](Value *V) {
+                   return isa<UndefValue>(V) && !isa<PoisonValue>(V);
+                 }) ||
+         any_of(GatheredScalars,
+                [](Value *V) {
+                  return isa<Constant>(V) && !isa<UndefValue>(V);
+                })) &&
+        (!IsIdentityShuffle ||
+         (GatheredScalars.size() == 2 &&
+          any_of(GatheredScalars,
+                 [](Value *V) { return !isa<UndefValue>(V); })) ||
+         count_if(GatheredScalars, [](Value *V) {
+           return isa<Constant>(V) && !isa<PoisonValue>(V);
+         }) > 1);
+    // NonConstants array contains just non-constant values, GatheredScalars
+    // contains only constant to build final vector and then shuffle.
+    for (int I = 0, Sz = GatheredScalars.size(); I < Sz; ++I) {
+      if (EnoughConstsForShuffle && isa<Constant>(GatheredScalars[I]))
+        NonConstants[I] = PoisonValue::get(ScalarTy);
+      else
+        GatheredScalars[I] = PoisonValue::get(ScalarTy);
+    }
+    // Generate constants for final shuffle and build a mask for them.
+    if (!all_of(GatheredScalars, PoisonValue::classof)) {
+      SmallVector<int> BVMask(GatheredScalars.size(), UndefMaskElem);
+      Value *BV = gather(GatheredScalars);
+      for (int I = 0, Sz = GatheredScalars.size(); I < Sz; ++I) {
+        if (!isa<PoisonValue>(GatheredScalars[I]))
+          BVMask[I] = I;
+      }
+      ShuffleBuilder.add(BV, BVMask);
+    }
+    if (all_of(NonConstants, [=](Value *V) {
+          return isa<PoisonValue>(V) ||
+                 (IsSingleShuffle && ((IsIdentityShuffle &&
+                  IsNonPoisoned) || IsUsedInExpr) && isa<UndefValue>(V));
+        }))
+      Vec = ShuffleBuilder.finalize(E->ReuseShuffleIndices);
+    else
+      Vec = ShuffleBuilder.finalize(
+          E->ReuseShuffleIndices, E->Scalars.size(),
+          [&](Value *&Vec, SmallVectorImpl<int> &Mask) {
+            Vec = gather(NonConstants, Vec);
+            for (unsigned I = 0, Sz = Mask.size(); I < Sz; ++I)
+              if (!isa<PoisonValue>(NonConstants[I]))
+                Mask[I] = I;
+          });
+  } else if (!allConstant(E->Scalars)) {
+    // TODO: remove this code once able to combine shuffled vectors and build
+    // vector elements.
+    copy(E->Scalars, GatheredScalars.begin());
     // For splats with can emit broadcasts instead of gathers, so try to find
     // such sequences.
-    bool IsSplat = isSplat(VL) && (VL.size() > 2 || VL.front() == VL.back());
+    bool IsSplat = isSplat(GatheredScalars) &&
+                   (GatheredScalars.size() > 2 ||
+                    GatheredScalars.front() == GatheredScalars.back());
+    GatheredScalars.append(VF - GatheredScalars.size(),
+                           PoisonValue::get(ScalarTy));
+    ReuseMask.assign(VF, UndefMaskElem);
     SmallVector<int> UndefPos;
     DenseMap<Value *, unsigned> UniquePositions;
     // Gather unique non-const values and all constant values.
     // For repeated values, just shuffle them.
-    for (auto [I, V] : enumerate(VL)) {
+    int NumNonConsts = 0;
+    int SinglePos = 0;
+    for (auto [I, V] : enumerate(GatheredScalars)) {
       if (isa<UndefValue>(V)) {
         if (!isa<PoisonValue>(V)) {
-          Gathered[I] = V;
           ReuseMask[I] = I;
           UndefPos.push_back(I);
         }
         continue;
       }
       if (isConstant(V)) {
-        Gathered[I] = V;
         ReuseMask[I] = I;
         continue;
       }
+      ++NumNonConsts;
+      SinglePos = I;
+      Value *OrigV = V;
+      GatheredScalars[I] = PoisonValue::get(ScalarTy);
       if (IsSplat) {
-        Gathered.front() = V;
+        GatheredScalars.front() = OrigV;
         ReuseMask[I] = 0;
       } else {
-        const auto Res = UniquePositions.try_emplace(V, I);
-        Gathered[Res.first->second] = V;
+        const auto Res = UniquePositions.try_emplace(OrigV, I);
+        GatheredScalars[Res.first->second] = OrigV;
         ReuseMask[I] = Res.first->second;
       }
     }
-    if (!UndefPos.empty() && IsSplat) {
+    if (NumNonConsts == 1) {
+      // Restore single insert element.
+      if (IsSplat) {
+        ReuseMask.assign(VF, UndefMaskElem);
+        std::swap(GatheredScalars.front(), GatheredScalars[SinglePos]);
+        if (!UndefPos.empty() && UndefPos.front() == 0)
+          GatheredScalars.front() = UndefValue::get(ScalarTy);
+      }
+      ReuseMask[SinglePos] = SinglePos;
+    } else if (!UndefPos.empty() && IsSplat) {
       // For undef values, try to replace them with the simple broadcast.
       // We can do it if the broadcasted value is guaranteed to be
       // non-poisonous, or by freezing the incoming scalar value first.
-      auto *It = find_if(Gathered, [this, E](Value *V) {
+      auto *It = find_if(GatheredScalars, [this, E](Value *V) {
         return !isa<UndefValue>(V) &&
                (getTreeEntry(V) || isGuaranteedNotToBePoison(V) ||
-                any_of(V->uses(), [E](const Use &U) {
-                  // Check if the value already used in the same operation in
-                  // one of the nodes already.
-                  return E->UserTreeIndices.size() == 1 &&
-                         is_contained(
-                             E->UserTreeIndices.front().UserTE->Scalars,
-                             U.getUser()) &&
-                         E->UserTreeIndices.front().EdgeIdx != U.getOperandNo();
-                }));
+                (E->UserTreeIndices.size() == 1 &&
+                 any_of(V->uses(), [E](const Use &U) {
+                   // Check if the value already used in the same operation in
+                   // one of the nodes already.
+                   return E->UserTreeIndices.front().EdgeIdx !=
+                              U.getOperandNo() &&
+                          is_contained(
+                              E->UserTreeIndices.front().UserTE->Scalars,
+                              U.getUser());
+                 })));
       });
-      if (It != Gathered.end()) {
+      if (It != GatheredScalars.end()) {
         // Replace undefs by the non-poisoned scalars and emit broadcast.
-        int Pos = std::distance(Gathered.begin(), It);
+        int Pos = std::distance(GatheredScalars.begin(), It);
         for_each(UndefPos, [&](int I) {
           // Set the undef position to the non-poisoned scalar.
           ReuseMask[I] = Pos;
-          // Replace the undef by the poison, in the mask it is replaced by non-poisoned scalar already.
+          // Replace the undef by the poison, in the mask it is replaced by
+          // non-poisoned scalar already.
           if (I != Pos)
-            Gathered[I] = PoisonValue::get(Gathered[I]->getType());
+            GatheredScalars[I] = PoisonValue::get(ScalarTy);
         });
       } else {
         // Replace undefs by the poisons, emit broadcast and then emit
         // freeze.
         for_each(UndefPos, [&](int I) {
           ReuseMask[I] = UndefMaskElem;
-          if (isa<UndefValue>(Gathered[I]))
-            Gathered[I] = PoisonValue::get(Gathered[I]->getType());
+          if (isa<UndefValue>(GatheredScalars[I]))
+            GatheredScalars[I] = PoisonValue::get(ScalarTy);
         });
         NeedFreeze = true;
       }
     }
+    // Gather unique scalars and all constants.
+    Vec = gather(GatheredScalars);
+    ShuffleBuilder.add(Vec, ReuseMask);
+    Vec = ShuffleBuilder.finalize(E->ReuseShuffleIndices);
   } else {
-    ReuseMask.clear();
-    copy(VL, Gathered.begin());
+    // Gather all constants.
+    Vec = gather(E->Scalars);
+    ShuffleBuilder.add(Vec, ReuseMask);
+    Vec = ShuffleBuilder.finalize(E->ReuseShuffleIndices);
   }
-  // Gather unique scalars and all constants.
-  Value *Vec = gather(Gathered);
-  ShuffleBuilder.add(Vec, ReuseMask);
-  Vec = ShuffleBuilder.finalize(E->ReuseShuffleIndices);
+
   if (NeedFreeze)
     Vec = Builder.CreateFreeze(Vec);
   return Vec;
@@ -9362,10 +9733,17 @@ Value *BoUpSLP::vectorizeTree(TreeEntry *E) {
     return E->VectorizedValue;
   }
 
+  if (E->State == TreeEntry::NeedToGather) {
+    if (E->getMainOp() && E->Idx == 0)
+      setInsertPointAfterBundle(E);
+    Value *Vec = createBuildVector(E);
+    E->VectorizedValue = Vec;
+    return Vec;
+  }
+
   auto FinalShuffle = [&](Value *V, const TreeEntry *E) {
     ShuffleInstructionBuilder ShuffleBuilder(Builder, *this);
-    if (E->State != TreeEntry::NeedToGather &&
-        E->getOpcode() == Instruction::Store) {
+    if (E->getOpcode() == Instruction::Store) {
       ArrayRef<int> Mask =
           ArrayRef(reinterpret_cast<const int *>(E->ReorderIndices.begin()),
                    E->ReorderIndices.size());
@@ -9376,170 +9754,6 @@ Value *BoUpSLP::vectorizeTree(TreeEntry *E) {
     return ShuffleBuilder.finalize(E->ReuseShuffleIndices);
   };
 
-  if (E->State == TreeEntry::NeedToGather) {
-    if (E->Idx > 0) {
-      // We are in the middle of a vectorizable chain. We need to gather the
-      // scalars from the users.
-      Value *Vec = createBuildVector(E);
-      E->VectorizedValue = Vec;
-      return Vec;
-    }
-    if (E->getMainOp())
-      setInsertPointAfterBundle(E);
-    unsigned VF = E->getVectorFactor();
-    auto AdjustExtracts = [&](const TreeEntry *E, ArrayRef<int> Mask) {
-      Value *VecBase = nullptr;
-      for (int I = 0, Sz = Mask.size(); I < Sz; ++I) {
-        int Idx = Mask[I];
-        if (Idx == UndefMaskElem)
-          continue;
-        auto *EI = cast<ExtractElementInst>(E->Scalars[I]);
-        VecBase = EI->getVectorOperand();
-        // If all users are vectorized - can delete the extractelement
-        // itself.
-        if (any_of(EI->users(),
-                   [&](User *U) { return !ScalarToTreeEntry.count(U); }))
-          continue;
-        eraseInstruction(EI);
-      }
-      return VecBase;
-    };
-    auto CreateShuffle = [&](Value *V1, Value *V2, ArrayRef<int> Mask) {
-      if (V1->getType() != V2->getType()) {
-        unsigned VF1 = cast<FixedVectorType>(V1->getType())->getNumElements();
-        unsigned VF2 = cast<FixedVectorType>(V2->getType())->getNumElements();
-        unsigned VF = std::max(VF1, VF2);
-        SmallVector<int> ExtMask(VF, UndefMaskElem);
-        std::iota(ExtMask.begin(),
-                  std::next(ExtMask.begin(), std::min(VF1, VF2)), 0);
-        if (VF1 < VF2) {
-          V1 = Builder.CreateShuffleVector(V1, ExtMask);
-          if (auto *I = dyn_cast<Instruction>(V1)) {
-            GatherShuffleExtractSeq.insert(I);
-            CSEBlocks.insert(I->getParent());
-          }
-        } else {
-          V2 = Builder.CreateShuffleVector(V2, ExtMask);
-          if (auto *I = dyn_cast<Instruction>(V2)) {
-            GatherShuffleExtractSeq.insert(I);
-            CSEBlocks.insert(I->getParent());
-          }
-        }
-      }
-      Value *Vec = Builder.CreateShuffleVector(V1, V2, Mask);
-      if (auto *I = dyn_cast<Instruction>(Vec)) {
-        GatherShuffleExtractSeq.insert(I);
-        CSEBlocks.insert(I->getParent());
-      }
-      return Vec;
-    };
-
-    SmallVector<int> ReuseShuffleIndicies(E->ReuseShuffleIndices.begin(),
-                                          E->ReuseShuffleIndices.end());
-    SmallVector<Value *> GatheredScalars(E->Scalars.begin(), E->Scalars.end());
-    // Build a mask out of the reorder indices and reorder scalars per this
-    // mask.
-    SmallVector<int> ReorderMask;
-    inversePermutation(E->ReorderIndices, ReorderMask);
-    if (!ReorderMask.empty())
-      reorderScalars(GatheredScalars, ReorderMask);
-    Value *Vec = nullptr;
-    SmallVector<int> Mask;
-    SmallVector<int> ExtractMask;
-    std::optional<TargetTransformInfo::ShuffleKind> ExtractShuffle;
-    std::optional<TargetTransformInfo::ShuffleKind> GatherShuffle;
-    SmallVector<const TreeEntry *> Entries;
-    Type *ScalarTy = GatheredScalars.front()->getType();
-    // Check for gathered extracts.
-    ExtractShuffle = tryToGatherExtractElements(GatheredScalars, ExtractMask);
-    SmallVector<Value *> IgnoredVals;
-    if (UserIgnoreList)
-      IgnoredVals.assign(UserIgnoreList->begin(), UserIgnoreList->end());
-    bool Resized = false;
-    if (Value *VecBase = AdjustExtracts(E, ExtractMask))
-      if (auto *VecBaseTy = dyn_cast<FixedVectorType>(VecBase->getType()))
-        if (VF == VecBaseTy->getNumElements() && GatheredScalars.size() != VF) {
-          Resized = true;
-          GatheredScalars.append(VF - GatheredScalars.size(),
-                                 PoisonValue::get(ScalarTy));
-        }
-    // Gather extracts after we check for full matched gathers only.
-    if (ExtractShuffle || E->getOpcode() != Instruction::Load ||
-        E->isAltShuffle() ||
-        all_of(E->Scalars, [this](Value *V) { return getTreeEntry(V); }) ||
-        isSplat(E->Scalars) ||
-        (E->Scalars != GatheredScalars && GatheredScalars.size() <= 2))
-      GatherShuffle = isGatherShuffledEntry(E, GatheredScalars, Mask, Entries);
-    if (GatherShuffle) {
-      assert((Entries.size() == 1 || Entries.size() == 2) &&
-             "Expected shuffle of 1 or 2 entries.");
-      if (!Resized) {
-        unsigned VF1 = Entries.front()->getVectorFactor();
-        unsigned VF2 = Entries.back()->getVectorFactor();
-        if ((VF == VF1 || VF == VF2) && GatheredScalars.size() != VF)
-          GatheredScalars.append(VF - GatheredScalars.size(),
-                                 PoisonValue::get(ScalarTy));
-      }
-      // Remove shuffled elements from list of gathers.
-      for (int I = 0, Sz = Mask.size(); I < Sz; ++I) {
-        if (Mask[I] != UndefMaskElem)
-          GatheredScalars[I] = PoisonValue::get(ScalarTy);
-      }
-    }
-    if ((ExtractShuffle || GatherShuffle) &&
-        all_of(GatheredScalars, PoisonValue::classof)) {
-      Value *Vec1 = nullptr;
-      if (ExtractShuffle) {
-        // Gather of extractelements can be represented as just a shuffle of
-        // a single/two vectors the scalars are extracted from.
-        // Find input vectors.
-        Value *Vec2 = nullptr;
-        for (unsigned I = 0, Sz = ExtractMask.size(); I < Sz; ++I) {
-          if (ExtractMask[I] == UndefMaskElem ||
-              (!Mask.empty() && Mask[I] != UndefMaskElem)) {
-            ExtractMask[I] = UndefMaskElem;
-            continue;
-          }
-          if (isa<UndefValue>(E->Scalars[I]))
-            continue;
-          auto *EI = cast<ExtractElementInst>(E->Scalars[I]);
-          if (!Vec1) {
-            Vec1 = EI->getVectorOperand();
-          } else if (Vec1 != EI->getVectorOperand()) {
-            assert((!Vec2 || Vec2 == EI->getVectorOperand()) &&
-                   "Expected only 1 or 2 vectors shuffle.");
-            Vec2 = EI->getVectorOperand();
-          }
-        }
-        if (Vec2)
-          Vec1 = CreateShuffle(Vec1, Vec2, ExtractMask);
-        else if (Vec1)
-          Vec1 = CreateShuffle(Vec1, Vec1, ExtractMask);
-      }
-      if (GatherShuffle) {
-        Vec = CreateShuffle(Entries.front()->VectorizedValue,
-                            Entries.back()->VectorizedValue, Mask);
-        if (Vec1) {
-          // Build final mask.
-          for (auto [I, Idx] : enumerate(Mask)) {
-            if (ExtractMask[I] != UndefMaskElem)
-              Idx = I;
-            else if (Idx != UndefMaskElem)
-              Idx = I + VF;
-          }
-          Vec = CreateShuffle(Vec1, Vec, Mask);
-        }
-      } else {
-        Vec = Vec1;
-      }
-    } else {
-      Vec = gather(E->Scalars);
-    }
-    Vec = FinalShuffle(Vec, E);
-    E->VectorizedValue = Vec;
-    return Vec;
-  }
-
   assert((E->State == TreeEntry::Vectorize ||
           E->State == TreeEntry::ScatterVectorize) &&
          "Unhandled state");
@@ -10177,6 +10391,36 @@ Value *BoUpSLP::vectorizeTree(ExtraValueToDebugLocsMap &ExternallyUsedValues,
   Builder.SetInsertPoint(ReductionRoot ? ReductionRoot
                                        : &F->getEntryBlock().front());
   auto *VectorRoot = vectorizeTree(VectorizableTree[0].get());
+  // Run through the list of postponed gathers and emit them, replacing the temp
+  // emitted allocas with actual vector instructions.
+  ArrayRef<const TreeEntry *> PostponedNodes = PostponedGathers.getArrayRef();
+  DenseMap<Value *, SmallVector<TreeEntry *>> PostponedValues;
+  for (const TreeEntry *E : PostponedNodes) {
+    auto *TE = const_cast<TreeEntry *>(E);
+    if (auto *VecTE = getTreeEntry(TE->Scalars.front()))
+      if (VecTE->isSame(TE->UserTreeIndices.front().UserTE->getOperand(
+              TE->UserTreeIndices.front().EdgeIdx)))
+        // Found gather node which is absolutely the same as one of the
+        // vectorized nodes. It may happen after reordering.
+        continue;
+    auto *PrevVec = cast<Instruction>(TE->VectorizedValue);
+    TE->VectorizedValue = nullptr;
+    auto *UserI =
+        cast<Instruction>(TE->UserTreeIndices.front().UserTE->VectorizedValue);
+    Builder.SetInsertPoint(PrevVec);
+    Builder.SetCurrentDebugLocation(UserI->getDebugLoc());
+    Value *Vec = vectorizeTree(TE);
+    PrevVec->replaceAllUsesWith(Vec);
+    PostponedValues.try_emplace(Vec).first->second.push_back(TE);
+    // Replace the stub vector node, if it was used before for one of the
+    // buildvector nodes already.
+    auto It = PostponedValues.find(PrevVec);
+    if (It != PostponedValues.end()) {
+      for (TreeEntry *VTE : It->getSecond())
+        VTE->VectorizedValue = Vec;
+    }
+    eraseInstruction(PrevVec);
+  }
 
   // If the vectorized tree can be rewritten in a smaller type, we truncate the
   // vectorized root. InstCombine will then rewrite the entire expression. We
@@ -11567,60 +11811,6 @@ void BoUpSLP::computeMinimumValueSizes() {
     MinBWs[Scalar] = std::make_pair(MaxBitWidth, !IsKnownPositive);
 }
 
-namespace {
-
-/// The SLPVectorizer Pass.
-struct SLPVectorizer : public FunctionPass {
-  SLPVectorizerPass Impl;
-
-  /// Pass identification, replacement for typeid
-  static char ID;
-
-  explicit SLPVectorizer() : FunctionPass(ID) {
-    initializeSLPVectorizerPass(*PassRegistry::getPassRegistry());
-  }
-
-  bool doInitialization(Module &M) override { return false; }
-
-  bool runOnFunction(Function &F) override {
-    if (skipFunction(F))
-      return false;
-
-    auto *SE = &getAnalysis<ScalarEvolutionWrapperPass>().getSE();
-    auto *TTI = &getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
-    auto *TLIP = getAnalysisIfAvailable<TargetLibraryInfoWrapperPass>();
-    auto *TLI = TLIP ? &TLIP->getTLI(F) : nullptr;
-    auto *AA = &getAnalysis<AAResultsWrapperPass>().getAAResults();
-    auto *LI = &getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
-    auto *DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
-    auto *AC = &getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
-    auto *DB = &getAnalysis<DemandedBitsWrapperPass>().getDemandedBits();
-    auto *ORE = &getAnalysis<OptimizationRemarkEmitterWrapperPass>().getORE();
-
-    return Impl.runImpl(F, SE, TTI, TLI, AA, LI, DT, AC, DB, ORE);
-  }
-
-  void getAnalysisUsage(AnalysisUsage &AU) const override {
-    FunctionPass::getAnalysisUsage(AU);
-    AU.addRequired<AssumptionCacheTracker>();
-    AU.addRequired<ScalarEvolutionWrapperPass>();
-    AU.addRequired<AAResultsWrapperPass>();
-    AU.addRequired<TargetTransformInfoWrapperPass>();
-    AU.addRequired<LoopInfoWrapperPass>();
-    AU.addRequired<DominatorTreeWrapperPass>();
-    AU.addRequired<DemandedBitsWrapperPass>();
-    AU.addRequired<OptimizationRemarkEmitterWrapperPass>();
-    AU.addRequired<InjectTLIMappingsLegacy>();
-    AU.addPreserved<LoopInfoWrapperPass>();
-    AU.addPreserved<DominatorTreeWrapperPass>();
-    AU.addPreserved<AAResultsWrapperPass>();
-    AU.addPreserved<GlobalsAAWrapperPass>();
-    AU.setPreservesCFG();
-  }
-};
-
-} // end anonymous namespace
-
 PreservedAnalyses SLPVectorizerPass::run(Function &F, FunctionAnalysisManager &AM) {
   auto *SE = &AM.getResult<ScalarEvolutionAnalysis>(F);
   auto *TTI = &AM.getResult<TargetIRAnalysis>(F);
@@ -12235,6 +12425,7 @@ class HorizontalReduction {
   static Value *createOp(IRBuilder<> &Builder, RecurKind Kind, Value *LHS,
                          Value *RHS, const Twine &Name, bool UseSelect) {
     unsigned RdxOpcode = RecurrenceDescriptor::getOpcode(Kind);
+    bool IsConstant = isConstant(LHS) && isConstant(RHS);
     switch (Kind) {
     case RecurKind::Or:
       if (UseSelect &&
@@ -12256,29 +12447,37 @@ class HorizontalReduction {
       return Builder.CreateBinOp((Instruction::BinaryOps)RdxOpcode, LHS, RHS,
                                  Name);
     case RecurKind::FMax:
+      if (IsConstant)
+        return ConstantFP::get(LHS->getType(),
+                               maxnum(cast<ConstantFP>(LHS)->getValueAPF(),
+                                      cast<ConstantFP>(RHS)->getValueAPF()));
       return Builder.CreateBinaryIntrinsic(Intrinsic::maxnum, LHS, RHS);
     case RecurKind::FMin:
+      if (IsConstant)
+        return ConstantFP::get(LHS->getType(),
+                               minnum(cast<ConstantFP>(LHS)->getValueAPF(),
+                                      cast<ConstantFP>(RHS)->getValueAPF()));
       return Builder.CreateBinaryIntrinsic(Intrinsic::minnum, LHS, RHS);
     case RecurKind::SMax:
-      if (UseSelect) {
+      if (IsConstant || UseSelect) {
         Value *Cmp = Builder.CreateICmpSGT(LHS, RHS, Name);
         return Builder.CreateSelect(Cmp, LHS, RHS, Name);
       }
       return Builder.CreateBinaryIntrinsic(Intrinsic::smax, LHS, RHS);
     case RecurKind::SMin:
-      if (UseSelect) {
+      if (IsConstant || UseSelect) {
         Value *Cmp = Builder.CreateICmpSLT(LHS, RHS, Name);
         return Builder.CreateSelect(Cmp, LHS, RHS, Name);
       }
       return Builder.CreateBinaryIntrinsic(Intrinsic::smin, LHS, RHS);
     case RecurKind::UMax:
-      if (UseSelect) {
+      if (IsConstant || UseSelect) {
         Value *Cmp = Builder.CreateICmpUGT(LHS, RHS, Name);
         return Builder.CreateSelect(Cmp, LHS, RHS, Name);
       }
       return Builder.CreateBinaryIntrinsic(Intrinsic::umax, LHS, RHS);
     case RecurKind::UMin:
-      if (UseSelect) {
+      if (IsConstant || UseSelect) {
         Value *Cmp = Builder.CreateICmpULT(LHS, RHS, Name);
         return Builder.CreateSelect(Cmp, LHS, RHS, Name);
       }
@@ -14513,20 +14712,3 @@ bool SLPVectorizerPass::vectorizeStoreChains(BoUpSLP &R) {
   }
   return Changed;
 }
-
-char SLPVectorizer::ID = 0;
-
-static const char lv_name[] = "SLP Vectorizer";
-
-INITIALIZE_PASS_BEGIN(SLPVectorizer, SV_NAME, lv_name, false, false)
-INITIALIZE_PASS_DEPENDENCY(AAResultsWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
-INITIALIZE_PASS_DEPENDENCY(ScalarEvolutionWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(LoopSimplify)
-INITIALIZE_PASS_DEPENDENCY(DemandedBitsWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(OptimizationRemarkEmitterWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(InjectTLIMappingsLegacy)
-INITIALIZE_PASS_END(SLPVectorizer, SV_NAME, lv_name, false, false)
-
-Pass *llvm::createSLPVectorizerPass() { return new SLPVectorizer(); }
diff --git a/llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h b/llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h
index 1d2a96f1ad375..8313b2ca4fdbb 100644
--- a/llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h
+++ b/llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h
@@ -159,17 +159,11 @@ class VPRecipeBuilder {
     return Ingredient2Recipe[I];
   }
 
-  /// Create a replicating region for \p PredRecipe.
-  VPRegionBlock *createReplicateRegion(VPReplicateRecipe *PredRecipe,
-                                       VPlan &Plan);
-
-  /// Build a VPReplicationRecipe for \p I and enclose it within a Region if it
-  /// is predicated. \return \p VPBB augmented with this new recipe if \p I is
-  /// not predicated, otherwise \return a new VPBasicBlock that succeeds the new
-  /// Region. Range.End may be decreased to ensure same recipe behavior from \p
-  /// Range.Start to \p Range.End.
-  VPBasicBlock *handleReplication(Instruction *I, VFRange &Range,
-                                  VPBasicBlock *VPBB, VPlan &Plan);
+  /// Build a VPReplicationRecipe for \p I. If it is predicated, add the mask as
+  /// last operand. Range.End may be decreased to ensure same recipe behavior
+  /// from \p Range.Start to \p Range.End.
+  VPRecipeOrVPValueTy handleReplication(Instruction *I, VFRange &Range,
+                                        VPlan &Plan);
 
   /// Add the incoming values from the backedge to reduction & first-order
   /// recurrence cross-iteration phis.
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.cpp b/llvm/lib/Transforms/Vectorize/VPlan.cpp
index 04f08811f5f5d..20484c31b7357 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlan.cpp
@@ -748,18 +748,31 @@ void VPlan::print(raw_ostream &O) const {
 
   O << "VPlan '" << getName() << "' {";
 
+  bool AnyLiveIn = false;
   if (VectorTripCount.getNumUsers() > 0) {
     O << "\nLive-in ";
     VectorTripCount.printAsOperand(O, SlotTracker);
-    O << " = vector-trip-count\n";
+    O << " = vector-trip-count";
+    AnyLiveIn = true;
+  }
+
+  if (TripCount && TripCount->getNumUsers() > 0) {
+    O << "\nLive-in ";
+    TripCount->printAsOperand(O, SlotTracker);
+    O << " = original trip-count";
+    AnyLiveIn = true;
   }
 
   if (BackedgeTakenCount && BackedgeTakenCount->getNumUsers()) {
     O << "\nLive-in ";
     BackedgeTakenCount->printAsOperand(O, SlotTracker);
-    O << " = backedge-taken count\n";
+    O << " = backedge-taken count";
+    AnyLiveIn = true;
   }
 
+  if (AnyLiveIn)
+    O << "\n";
+
   for (const VPBlockBase *Block : vp_depth_first_shallow(getEntry())) {
     O << '\n';
     Block->print(O, "", SlotTracker);
@@ -1089,7 +1102,7 @@ VPInterleavedAccessInfo::VPInterleavedAccessInfo(VPlan &Plan,
 }
 
 void VPSlotTracker::assignSlot(const VPValue *V) {
-  assert(Slots.find(V) == Slots.end() && "VPValue already has a slot!");
+  assert(!Slots.contains(V) && "VPValue already has a slot!");
   Slots[V] = NextSlot++;
 }
 
@@ -1101,6 +1114,8 @@ void VPSlotTracker::assignSlots(const VPlan &Plan) {
   assignSlot(&Plan.VectorTripCount);
   if (Plan.BackedgeTakenCount)
     assignSlot(Plan.BackedgeTakenCount);
+  if (Plan.TripCount)
+    assignSlot(Plan.TripCount);
 
   ReversePostOrderTraversal<VPBlockDeepTraversalWrapper<const VPBlockBase *>>
       RPOT(VPBlockDeepTraversalWrapper<const VPBlockBase *>(Plan.getEntry()));
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.h b/llvm/lib/Transforms/Vectorize/VPlan.h
index efcfc135894f6..77a32d854be14 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.h
+++ b/llvm/lib/Transforms/Vectorize/VPlan.h
@@ -248,7 +248,7 @@ struct VPTransformState {
   }
 
   bool hasAnyVectorValue(VPValue *Def) const {
-    return Data.PerPartOutput.find(Def) != Data.PerPartOutput.end();
+    return Data.PerPartOutput.contains(Def);
   }
 
   bool hasScalarValue(VPValue *Def, VPIteration Instance) {
@@ -962,17 +962,10 @@ class VPWidenCallRecipe : public VPRecipeBase, public VPValue {
 };
 
 /// A recipe for widening select instructions.
-class VPWidenSelectRecipe : public VPRecipeBase, public VPValue {
-
-  /// Is the condition of the select loop invariant?
-  bool InvariantCond;
-
-public:
+struct VPWidenSelectRecipe : public VPRecipeBase, public VPValue {
   template <typename IterT>
-  VPWidenSelectRecipe(SelectInst &I, iterator_range<IterT> Operands,
-                      bool InvariantCond)
-      : VPRecipeBase(VPDef::VPWidenSelectSC, Operands), VPValue(this, &I),
-        InvariantCond(InvariantCond) {}
+  VPWidenSelectRecipe(SelectInst &I, iterator_range<IterT> Operands)
+      : VPRecipeBase(VPDef::VPWidenSelectSC, Operands), VPValue(this, &I) {}
 
   ~VPWidenSelectRecipe() override = default;
 
@@ -986,29 +979,37 @@ class VPWidenSelectRecipe : public VPRecipeBase, public VPValue {
   void print(raw_ostream &O, const Twine &Indent,
              VPSlotTracker &SlotTracker) const override;
 #endif
+
+  VPValue *getCond() const {
+    return getOperand(0);
+  }
+
+  bool isInvariantCond() const {
+    return getCond()->isDefinedOutsideVectorRegions();
+  }
 };
 
 /// A recipe for handling GEP instructions.
 class VPWidenGEPRecipe : public VPRecipeBase, public VPValue {
-  bool IsPtrLoopInvariant;
-  SmallBitVector IsIndexLoopInvariant;
+  bool isPointerLoopInvariant() const {
+    return getOperand(0)->isDefinedOutsideVectorRegions();
+  }
+
+  bool isIndexLoopInvariant(unsigned I) const {
+    return getOperand(I + 1)->isDefinedOutsideVectorRegions();
+  }
+
+  bool areAllOperandsInvariant() const {
+    return all_of(operands(), [](VPValue *Op) {
+      return Op->isDefinedOutsideVectorRegions();
+    });
+  }
 
 public:
   template <typename IterT>
   VPWidenGEPRecipe(GetElementPtrInst *GEP, iterator_range<IterT> Operands)
-      : VPRecipeBase(VPDef::VPWidenGEPSC, Operands), VPValue(this, GEP),
-        IsIndexLoopInvariant(GEP->getNumIndices(), false) {}
+      : VPRecipeBase(VPDef::VPWidenGEPSC, Operands), VPValue(this, GEP) {}
 
-  template <typename IterT>
-  VPWidenGEPRecipe(GetElementPtrInst *GEP, iterator_range<IterT> Operands,
-                   Loop *OrigLoop)
-      : VPRecipeBase(VPDef::VPWidenGEPSC, Operands), VPValue(this, GEP),
-        IsIndexLoopInvariant(GEP->getNumIndices(), false) {
-    IsPtrLoopInvariant = OrigLoop->isLoopInvariant(GEP->getPointerOperand());
-    for (auto Index : enumerate(GEP->indices()))
-      IsIndexLoopInvariant[Index.index()] =
-          OrigLoop->isLoopInvariant(Index.value().get());
-  }
   ~VPWidenGEPRecipe() override = default;
 
   VP_CLASSOF_IMPL(VPDef::VPWidenGEPSC)
@@ -1023,78 +1024,6 @@ class VPWidenGEPRecipe : public VPRecipeBase, public VPValue {
 #endif
 };
 
-/// A recipe for handling phi nodes of integer and floating-point inductions,
-/// producing their vector values.
-class VPWidenIntOrFpInductionRecipe : public VPRecipeBase, public VPValue {
-  PHINode *IV;
-  const InductionDescriptor &IndDesc;
-  bool NeedsVectorIV;
-
-public:
-  VPWidenIntOrFpInductionRecipe(PHINode *IV, VPValue *Start, VPValue *Step,
-                                const InductionDescriptor &IndDesc,
-                                bool NeedsVectorIV)
-      : VPRecipeBase(VPDef::VPWidenIntOrFpInductionSC, {Start, Step}),
-        VPValue(this, IV), IV(IV), IndDesc(IndDesc),
-        NeedsVectorIV(NeedsVectorIV) {}
-
-  VPWidenIntOrFpInductionRecipe(PHINode *IV, VPValue *Start, VPValue *Step,
-                                const InductionDescriptor &IndDesc,
-                                TruncInst *Trunc, bool NeedsVectorIV)
-      : VPRecipeBase(VPDef::VPWidenIntOrFpInductionSC, {Start, Step}),
-        VPValue(this, Trunc), IV(IV), IndDesc(IndDesc),
-        NeedsVectorIV(NeedsVectorIV) {}
-
-  ~VPWidenIntOrFpInductionRecipe() override = default;
-
-  VP_CLASSOF_IMPL(VPDef::VPWidenIntOrFpInductionSC)
-
-  /// Generate the vectorized and scalarized versions of the phi node as
-  /// needed by their users.
-  void execute(VPTransformState &State) override;
-
-#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
-  /// Print the recipe.
-  void print(raw_ostream &O, const Twine &Indent,
-             VPSlotTracker &SlotTracker) const override;
-#endif
-
-  /// Returns the start value of the induction.
-  VPValue *getStartValue() { return getOperand(0); }
-  const VPValue *getStartValue() const { return getOperand(0); }
-
-  /// Returns the step value of the induction.
-  VPValue *getStepValue() { return getOperand(1); }
-  const VPValue *getStepValue() const { return getOperand(1); }
-
-  /// Returns the first defined value as TruncInst, if it is one or nullptr
-  /// otherwise.
-  TruncInst *getTruncInst() {
-    return dyn_cast_or_null<TruncInst>(getVPValue(0)->getUnderlyingValue());
-  }
-  const TruncInst *getTruncInst() const {
-    return dyn_cast_or_null<TruncInst>(getVPValue(0)->getUnderlyingValue());
-  }
-
-  PHINode *getPHINode() { return IV; }
-
-  /// Returns the induction descriptor for the recipe.
-  const InductionDescriptor &getInductionDescriptor() const { return IndDesc; }
-
-  /// Returns true if the induction is canonical, i.e. starting at 0 and
-  /// incremented by UF * VF (= the original IV is incremented by 1).
-  bool isCanonical() const;
-
-  /// Returns the scalar type of the induction.
-  const Type *getScalarType() const {
-    const TruncInst *TruncI = getTruncInst();
-    return TruncI ? TruncI->getType() : IV->getType();
-  }
-
-  /// Returns true if a vector phi needs to be created for the induction.
-  bool needsVectorIV() const { return NeedsVectorIV; }
-};
-
 /// A pure virtual base class for all recipes modeling header phis, including
 /// phis for first order recurrences, pointer inductions and reductions. The
 /// start value is the first operand of the recipe and the incoming value from
@@ -1120,9 +1049,9 @@ class VPWidenIntOrFpInductionRecipe : public VPRecipeBase, public VPValue {
 ///    per-lane based on the canonical induction.
 class VPHeaderPHIRecipe : public VPRecipeBase, public VPValue {
 protected:
-  VPHeaderPHIRecipe(unsigned char VPDefID, PHINode *Phi,
+  VPHeaderPHIRecipe(unsigned char VPDefID, Instruction *UnderlyingInstr,
                     VPValue *Start = nullptr)
-      : VPRecipeBase(VPDefID, {}), VPValue(this, Phi) {
+      : VPRecipeBase(VPDefID, {}), VPValue(this, UnderlyingInstr) {
     if (Start)
       addOperand(Start);
   }
@@ -1133,12 +1062,12 @@ class VPHeaderPHIRecipe : public VPRecipeBase, public VPValue {
   /// Method to support type inquiry through isa, cast, and dyn_cast.
   static inline bool classof(const VPRecipeBase *B) {
     return B->getVPDefID() >= VPDef::VPFirstHeaderPHISC &&
-           B->getVPDefID() <= VPDef::VPLastPHISC;
+           B->getVPDefID() <= VPDef::VPLastHeaderPHISC;
   }
   static inline bool classof(const VPValue *V) {
     auto *B = V->getDefiningRecipe();
     return B && B->getVPDefID() >= VPRecipeBase::VPFirstHeaderPHISC &&
-           B->getVPDefID() <= VPRecipeBase::VPLastPHISC;
+           B->getVPDefID() <= VPRecipeBase::VPLastHeaderPHISC;
   }
 
   /// Generate the phi nodes.
@@ -1162,17 +1091,97 @@ class VPHeaderPHIRecipe : public VPRecipeBase, public VPValue {
   void setStartValue(VPValue *V) { setOperand(0, V); }
 
   /// Returns the incoming value from the loop backedge.
-  VPValue *getBackedgeValue() {
+  virtual VPValue *getBackedgeValue() {
     return getOperand(1);
   }
 
   /// Returns the backedge value as a recipe. The backedge value is guaranteed
   /// to be a recipe.
-  VPRecipeBase &getBackedgeRecipe() {
+  virtual VPRecipeBase &getBackedgeRecipe() {
     return *getBackedgeValue()->getDefiningRecipe();
   }
 };
 
+/// A recipe for handling phi nodes of integer and floating-point inductions,
+/// producing their vector values.
+class VPWidenIntOrFpInductionRecipe : public VPHeaderPHIRecipe {
+  PHINode *IV;
+  TruncInst *Trunc;
+  const InductionDescriptor &IndDesc;
+  bool NeedsVectorIV;
+
+public:
+  VPWidenIntOrFpInductionRecipe(PHINode *IV, VPValue *Start, VPValue *Step,
+                                const InductionDescriptor &IndDesc,
+                                bool NeedsVectorIV)
+      : VPHeaderPHIRecipe(VPDef::VPWidenIntOrFpInductionSC, IV, Start), IV(IV),
+        Trunc(nullptr), IndDesc(IndDesc), NeedsVectorIV(NeedsVectorIV) {
+    addOperand(Step);
+  }
+
+  VPWidenIntOrFpInductionRecipe(PHINode *IV, VPValue *Start, VPValue *Step,
+                                const InductionDescriptor &IndDesc,
+                                TruncInst *Trunc, bool NeedsVectorIV)
+      : VPHeaderPHIRecipe(VPDef::VPWidenIntOrFpInductionSC, Trunc, Start),
+        IV(IV), Trunc(Trunc), IndDesc(IndDesc), NeedsVectorIV(NeedsVectorIV) {
+    addOperand(Step);
+  }
+
+  ~VPWidenIntOrFpInductionRecipe() override = default;
+
+  VP_CLASSOF_IMPL(VPDef::VPWidenIntOrFpInductionSC)
+
+  /// Generate the vectorized and scalarized versions of the phi node as
+  /// needed by their users.
+  void execute(VPTransformState &State) override;
+
+#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
+  /// Print the recipe.
+  void print(raw_ostream &O, const Twine &Indent,
+             VPSlotTracker &SlotTracker) const override;
+#endif
+
+  VPValue *getBackedgeValue() override {
+    // TODO: All operands of base recipe must exist and be at same index in
+    // derived recipe.
+    llvm_unreachable(
+        "VPWidenIntOrFpInductionRecipe generates its own backedge value");
+  }
+
+  VPRecipeBase &getBackedgeRecipe() override {
+    // TODO: All operands of base recipe must exist and be at same index in
+    // derived recipe.
+    llvm_unreachable(
+        "VPWidenIntOrFpInductionRecipe generates its own backedge value");
+  }
+
+  /// Returns the step value of the induction.
+  VPValue *getStepValue() { return getOperand(1); }
+  const VPValue *getStepValue() const { return getOperand(1); }
+
+  /// Returns the first defined value as TruncInst, if it is one or nullptr
+  /// otherwise.
+  TruncInst *getTruncInst() { return Trunc; }
+  const TruncInst *getTruncInst() const { return Trunc; }
+
+  PHINode *getPHINode() { return IV; }
+
+  /// Returns the induction descriptor for the recipe.
+  const InductionDescriptor &getInductionDescriptor() const { return IndDesc; }
+
+  /// Returns true if the induction is canonical, i.e. starting at 0 and
+  /// incremented by UF * VF (= the original IV is incremented by 1).
+  bool isCanonical() const;
+
+  /// Returns the scalar type of the induction.
+  const Type *getScalarType() const {
+    return Trunc ? Trunc->getType() : IV->getType();
+  }
+
+  /// Returns true if a vector phi needs to be created for the induction.
+  bool needsVectorIV() const { return NeedsVectorIV; }
+};
+
 class VPWidenPointerInductionRecipe : public VPHeaderPHIRecipe {
   const InductionDescriptor &IndDesc;
 
@@ -1508,9 +1517,12 @@ class VPReplicateRecipe : public VPRecipeBase, public VPValue {
 public:
   template <typename IterT>
   VPReplicateRecipe(Instruction *I, iterator_range<IterT> Operands,
-                    bool IsUniform, bool IsPredicated = false)
+                    bool IsUniform, VPValue *Mask = nullptr)
       : VPRecipeBase(VPDef::VPReplicateSC, Operands), VPValue(this, I),
-        IsUniform(IsUniform), IsPredicated(IsPredicated) {}
+        IsUniform(IsUniform), IsPredicated(Mask) {
+    if (Mask)
+      addOperand(Mask);
+  }
 
   ~VPReplicateRecipe() override = default;
 
@@ -1549,6 +1561,12 @@ class VPReplicateRecipe : public VPRecipeBase, public VPValue {
   /// VPPredInstPHIRecipe. In this case, the scalar values should also be packed
   /// in a vector.
   bool shouldPack() const;
+
+  /// Return the mask of a predicated VPReplicateRecipe.
+  VPValue *getMask() {
+    assert(isPredicated() && "Trying to get the mask of a unpredicated recipe");
+    return getOperand(getNumOperands() - 1);
+  }
 };
 
 /// A recipe for generating conditional branches on the bits of a mask.
diff --git a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
index 935a1565822ab..edc1e362d93d2 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
@@ -553,7 +553,7 @@ void VPWidenSelectRecipe::print(raw_ostream &O, const Twine &Indent,
   getOperand(1)->printAsOperand(O, SlotTracker);
   O << ", ";
   getOperand(2)->printAsOperand(O, SlotTracker);
-  O << (InvariantCond ? " (condition is loop invariant)" : "");
+  O << (isInvariantCond() ? " (condition is loop invariant)" : "");
 }
 #endif
 
@@ -566,10 +566,10 @@ void VPWidenSelectRecipe::execute(VPTransformState &State) {
   // We have to take the 'vectorized' value and pick the first lane.
   // Instcombine will make this a no-op.
   auto *InvarCond =
-      InvariantCond ? State.get(getOperand(0), VPIteration(0, 0)) : nullptr;
+      isInvariantCond() ? State.get(getCond(), VPIteration(0, 0)) : nullptr;
 
   for (unsigned Part = 0; Part < State.UF; ++Part) {
-    Value *Cond = InvarCond ? InvarCond : State.get(getOperand(0), Part);
+    Value *Cond = InvarCond ? InvarCond : State.get(getCond(), Part);
     Value *Op0 = State.get(getOperand(1), Part);
     Value *Op1 = State.get(getOperand(2), Part);
     Value *Sel = State.Builder.CreateSelect(Cond, Op0, Op1);
@@ -714,7 +714,7 @@ void VPWidenRecipe::print(raw_ostream &O, const Twine &Indent,
   const Instruction *UI = getUnderlyingInstr();
   O << " = " << UI->getOpcodeName() << " ";
   if (auto *Cmp = dyn_cast<CmpInst>(UI))
-    O << CmpInst::getPredicateName(Cmp->getPredicate()) << " ";
+    O << Cmp->getPredicate() << " ";
   printOperands(O, SlotTracker);
 }
 
@@ -775,7 +775,7 @@ void VPWidenGEPRecipe::execute(VPTransformState &State) {
   // is vector-typed. Thus, to keep the representation compact, we only use
   // vector-typed operands for loop-varying values.
 
-  if (State.VF.isVector() && IsPtrLoopInvariant && IsIndexLoopInvariant.all()) {
+  if (State.VF.isVector() && areAllOperandsInvariant()) {
     // If we are vectorizing, but the GEP has only loop-invariant operands,
     // the GEP we build (by only using vector-typed operands for
     // loop-varying values) would be a scalar pointer. Thus, to ensure we
@@ -805,7 +805,7 @@ void VPWidenGEPRecipe::execute(VPTransformState &State) {
     for (unsigned Part = 0; Part < State.UF; ++Part) {
       // The pointer operand of the new GEP. If it's loop-invariant, we
       // won't broadcast it.
-      auto *Ptr = IsPtrLoopInvariant
+      auto *Ptr = isPointerLoopInvariant()
                       ? State.get(getOperand(0), VPIteration(0, 0))
                       : State.get(getOperand(0), Part);
 
@@ -814,7 +814,7 @@ void VPWidenGEPRecipe::execute(VPTransformState &State) {
       SmallVector<Value *, 4> Indices;
       for (unsigned I = 1, E = getNumOperands(); I < E; I++) {
         VPValue *Operand = getOperand(I);
-        if (IsIndexLoopInvariant[I - 1])
+        if (isIndexLoopInvariant(I - 1))
           Indices.push_back(State.get(Operand, VPIteration(0, 0)));
         else
           Indices.push_back(State.get(Operand, Part));
@@ -844,10 +844,9 @@ void VPWidenGEPRecipe::execute(VPTransformState &State) {
 void VPWidenGEPRecipe::print(raw_ostream &O, const Twine &Indent,
                              VPSlotTracker &SlotTracker) const {
   O << Indent << "WIDEN-GEP ";
-  O << (IsPtrLoopInvariant ? "Inv" : "Var");
-  size_t IndicesNumber = IsIndexLoopInvariant.size();
-  for (size_t I = 0; I < IndicesNumber; ++I)
-    O << "[" << (IsIndexLoopInvariant[I] ? "Inv" : "Var") << "]";
+  O << (isPointerLoopInvariant() ? "Inv" : "Var");
+  for (size_t I = 0; I < getNumOperands() - 1; ++I)
+    O << "[" << (isIndexLoopInvariant(I) ? "Inv" : "Var") << "]";
 
   O << " ";
   printAsOperand(O, SlotTracker);
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
index d3f84dade4715..99f600002336d 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
@@ -24,7 +24,7 @@
 using namespace llvm;
 
 void VPlanTransforms::VPInstructionsToVPRecipes(
-    Loop *OrigLoop, VPlanPtr &Plan,
+    VPlanPtr &Plan,
     function_ref<const InductionDescriptor *(PHINode *)>
         GetIntOrFpInductionDescriptor,
     SmallPtrSetImpl<Instruction *> &DeadInstructions, ScalarEvolution &SE,
@@ -76,17 +76,15 @@ void VPlanTransforms::VPInstructionsToVPRecipes(
               Plan->getOrAddVPValue(Store->getValueOperand()), nullptr /*Mask*/,
               false /*Consecutive*/, false /*Reverse*/);
         } else if (GetElementPtrInst *GEP = dyn_cast<GetElementPtrInst>(Inst)) {
-          NewRecipe = new VPWidenGEPRecipe(
-              GEP, Plan->mapToVPValues(GEP->operands()), OrigLoop);
+          NewRecipe =
+              new VPWidenGEPRecipe(GEP, Plan->mapToVPValues(GEP->operands()));
         } else if (CallInst *CI = dyn_cast<CallInst>(Inst)) {
           NewRecipe =
               new VPWidenCallRecipe(*CI, Plan->mapToVPValues(CI->args()),
                                     getVectorIntrinsicIDForCall(CI, &TLI));
         } else if (SelectInst *SI = dyn_cast<SelectInst>(Inst)) {
-          bool InvariantCond =
-              SE.isLoopInvariant(SE.getSCEV(SI->getOperand(0)), OrigLoop);
-          NewRecipe = new VPWidenSelectRecipe(
-              *SI, Plan->mapToVPValues(SI->operands()), InvariantCond);
+          NewRecipe =
+              new VPWidenSelectRecipe(*SI, Plan->mapToVPValues(SI->operands()));
         } else {
           NewRecipe =
               new VPWidenRecipe(*Inst, Plan->mapToVPValues(Inst->operands()));
@@ -108,7 +106,7 @@ void VPlanTransforms::VPInstructionsToVPRecipes(
   }
 }
 
-bool VPlanTransforms::sinkScalarOperands(VPlan &Plan) {
+static bool sinkScalarOperands(VPlan &Plan) {
   auto Iter = vp_depth_first_deep(Plan.getEntry());
   bool Changed = false;
   // First, collect the operands of all recipes in replicate blocks as seeds for
@@ -169,8 +167,7 @@ bool VPlanTransforms::sinkScalarOperands(VPlan &Plan) {
         continue;
       Instruction *I = cast<Instruction>(
           cast<VPReplicateRecipe>(SinkCandidate)->getUnderlyingValue());
-      auto *Clone =
-          new VPReplicateRecipe(I, SinkCandidate->operands(), true, false);
+      auto *Clone = new VPReplicateRecipe(I, SinkCandidate->operands(), true);
       // TODO: add ".cloned" suffix to name of Clone's VPValue.
 
       Clone->insertBefore(SinkCandidate);
@@ -226,7 +223,10 @@ static VPBasicBlock *getPredicatedThenBlock(VPRegionBlock *R) {
   return nullptr;
 }
 
-bool VPlanTransforms::mergeReplicateRegionsIntoSuccessors(VPlan &Plan) {
+// Merge replicate regions in their successor region, if a replicate region
+// is connected to a successor replicate region with the same predicate by a
+// single, empty VPBasicBlock.
+static bool mergeReplicateRegionsIntoSuccessors(VPlan &Plan) {
   SetVector<VPRegionBlock *> DeletedRegions;
 
   // Collect replicate regions followed by an empty block, followed by another
@@ -314,6 +314,81 @@ bool VPlanTransforms::mergeReplicateRegionsIntoSuccessors(VPlan &Plan) {
   return !DeletedRegions.empty();
 }
 
+static VPRegionBlock *createReplicateRegion(VPReplicateRecipe *PredRecipe,
+                                            VPlan &Plan) {
+  Instruction *Instr = PredRecipe->getUnderlyingInstr();
+  // Build the triangular if-then region.
+  std::string RegionName = (Twine("pred.") + Instr->getOpcodeName()).str();
+  assert(Instr->getParent() && "Predicated instruction not in any basic block");
+  auto *BlockInMask = PredRecipe->getMask();
+  auto *BOMRecipe = new VPBranchOnMaskRecipe(BlockInMask);
+  auto *Entry = new VPBasicBlock(Twine(RegionName) + ".entry", BOMRecipe);
+
+  // Replace predicated replicate recipe with a replicate recipe without a
+  // mask but in the replicate region.
+  auto *RecipeWithoutMask = new VPReplicateRecipe(
+      PredRecipe->getUnderlyingInstr(),
+      make_range(PredRecipe->op_begin(), std::prev(PredRecipe->op_end())),
+      PredRecipe->isUniform());
+  auto *Pred = new VPBasicBlock(Twine(RegionName) + ".if", RecipeWithoutMask);
+
+  VPPredInstPHIRecipe *PHIRecipe = nullptr;
+  if (PredRecipe->getNumUsers() != 0) {
+    PHIRecipe = new VPPredInstPHIRecipe(RecipeWithoutMask);
+    PredRecipe->replaceAllUsesWith(PHIRecipe);
+    PHIRecipe->setOperand(0, RecipeWithoutMask);
+  }
+  PredRecipe->eraseFromParent();
+  auto *Exiting = new VPBasicBlock(Twine(RegionName) + ".continue", PHIRecipe);
+  VPRegionBlock *Region = new VPRegionBlock(Entry, Exiting, RegionName, true);
+
+  // Note: first set Entry as region entry and then connect successors starting
+  // from it in order, to propagate the "parent" of each VPBasicBlock.
+  VPBlockUtils::insertTwoBlocksAfter(Pred, Exiting, Entry);
+  VPBlockUtils::connectBlocks(Pred, Exiting);
+
+  return Region;
+}
+
+static void addReplicateRegions(VPlan &Plan) {
+  SmallVector<VPReplicateRecipe *> WorkList;
+  for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(
+           vp_depth_first_deep(Plan.getEntry()))) {
+    for (VPRecipeBase &R : *VPBB)
+      if (auto *RepR = dyn_cast<VPReplicateRecipe>(&R)) {
+        if (RepR->isPredicated())
+          WorkList.push_back(RepR);
+      }
+  }
+
+  unsigned BBNum = 0;
+  for (VPReplicateRecipe *RepR : WorkList) {
+    VPBasicBlock *CurrentBlock = RepR->getParent();
+    VPBasicBlock *SplitBlock = CurrentBlock->splitAt(RepR->getIterator());
+
+    BasicBlock *OrigBB = RepR->getUnderlyingInstr()->getParent();
+    SplitBlock->setName(
+        OrigBB->hasName() ? OrigBB->getName() + "." + Twine(BBNum++) : "");
+    // Record predicated instructions for above packing optimizations.
+    VPBlockBase *Region = createReplicateRegion(RepR, Plan);
+    Region->setParent(CurrentBlock->getParent());
+    VPBlockUtils::disconnectBlocks(CurrentBlock, SplitBlock);
+    VPBlockUtils::connectBlocks(CurrentBlock, Region);
+    VPBlockUtils::connectBlocks(Region, SplitBlock);
+  }
+}
+
+void VPlanTransforms::createAndOptimizeReplicateRegions(VPlan &Plan) {
+  // Convert masked VPReplicateRecipes to if-then region blocks.
+  addReplicateRegions(Plan);
+
+  bool ShouldSimplify = true;
+  while (ShouldSimplify) {
+    ShouldSimplify = sinkScalarOperands(Plan);
+    ShouldSimplify |= mergeReplicateRegionsIntoSuccessors(Plan);
+    ShouldSimplify |= VPlanTransforms::mergeBlocksIntoPredecessors(Plan);
+  }
+}
 bool VPlanTransforms::mergeBlocksIntoPredecessors(VPlan &Plan) {
   SmallVector<VPBasicBlock *> WorkList;
   for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(
@@ -620,58 +695,11 @@ sinkRecurrenceUsersAfterPrevious(VPFirstOrderRecurrencePHIRecipe *FOR,
   });
 
   for (VPRecipeBase *SinkCandidate : WorkList) {
-    // VPPredInstPHIRecipes don't need sinking, because they will be sunk when
-    // sinking the containing replicate region.
-    if (isa<VPPredInstPHIRecipe>(SinkCandidate) || SinkCandidate == FOR)
+    if (SinkCandidate == FOR)
       continue;
 
-    VPRecipeBase *Target = Previous;
+    SinkCandidate->moveAfter(Previous);
     Previous = SinkCandidate;
-    auto *TargetRegion = GetReplicateRegion(Target);
-    auto *SinkRegion = GetReplicateRegion(SinkCandidate);
-    if (!SinkRegion) {
-      // If the sink source is not a replicate region, sink the recipe
-      // directly.
-      if (TargetRegion) {
-        // The target is in a replication region, make sure to move Sink to
-        // the block after it, not into the replication region itself.
-        VPBasicBlock *NextBlock =
-            cast<VPBasicBlock>(TargetRegion->getSuccessors().front());
-        SinkCandidate->moveBefore(*NextBlock, NextBlock->getFirstNonPhi());
-      } else
-        SinkCandidate->moveAfter(Target);
-      continue;
-    }
-    // The sink source is in a replicate region. Unhook the region from the
-    // CFG.
-    auto *SinkPred = SinkRegion->getSinglePredecessor();
-    auto *SinkSucc = SinkRegion->getSingleSuccessor();
-    VPBlockUtils::disconnectBlocks(SinkPred, SinkRegion);
-    VPBlockUtils::disconnectBlocks(SinkRegion, SinkSucc);
-    VPBlockUtils::connectBlocks(SinkPred, SinkSucc);
-
-    if (TargetRegion) {
-      // The target recipe is also in a replicate region, move the sink
-      // region after the target region.
-      auto *TargetSucc = TargetRegion->getSingleSuccessor();
-      VPBlockUtils::disconnectBlocks(TargetRegion, TargetSucc);
-      VPBlockUtils::connectBlocks(TargetRegion, SinkRegion);
-      VPBlockUtils::connectBlocks(SinkRegion, TargetSucc);
-    } else {
-      // The sink source is in a replicate region, we need to move the whole
-      // replicate region, which should only contain a single recipe in the
-      // main block.
-      auto *SplitBlock =
-          Target->getParent()->splitAt(std::next(Target->getIterator()));
-
-      auto *SplitPred = SplitBlock->getSinglePredecessor();
-
-      VPBlockUtils::disconnectBlocks(SplitPred, SplitBlock);
-      VPBlockUtils::connectBlocks(SplitPred, SinkRegion);
-      VPBlockUtils::connectBlocks(SinkRegion, SplitBlock);
-    }
-    // We modified the CFG, update dominator tree.
-    VPDT.recalculate(*SinkRegion->getPlan());
   }
 }
 
@@ -703,14 +731,7 @@ void VPlanTransforms::adjustFixedOrderRecurrences(VPlan &Plan,
     // Introduce a recipe to combine the incoming and previous values of a
     // fixed-order recurrence.
     VPBasicBlock *InsertBlock = Previous->getParent();
-    auto *Region = GetReplicateRegion(Previous);
-    if (Region)
-      InsertBlock = dyn_cast<VPBasicBlock>(Region->getSingleSuccessor());
-    if (!InsertBlock) {
-      InsertBlock = new VPBasicBlock(Region->getName() + ".succ");
-      VPBlockUtils::insertBlockAfter(InsertBlock, Region);
-    }
-    if (Region || isa<VPHeaderPHIRecipe>(Previous))
+    if (isa<VPHeaderPHIRecipe>(Previous))
       Builder.setInsertPoint(InsertBlock, InsertBlock->getFirstNonPhi());
     else
       Builder.setInsertPoint(InsertBlock, std::next(Previous->getIterator()));
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.h b/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
index bc1094d9c5128..e5bd1a42f8778 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
@@ -26,23 +26,23 @@ class Loop;
 class PredicatedScalarEvolution;
 class TargetLibraryInfo;
 class VPBuilder;
+class VPRecipeBuilder;
 
 struct VPlanTransforms {
   /// Replaces the VPInstructions in \p Plan with corresponding
   /// widen recipes.
   static void
-  VPInstructionsToVPRecipes(Loop *OrigLoop, VPlanPtr &Plan,
+  VPInstructionsToVPRecipes(VPlanPtr &Plan,
                             function_ref<const InductionDescriptor *(PHINode *)>
                                 GetIntOrFpInductionDescriptor,
                             SmallPtrSetImpl<Instruction *> &DeadInstructions,
                             ScalarEvolution &SE, const TargetLibraryInfo &TLI);
 
-  static bool sinkScalarOperands(VPlan &Plan);
-
-  /// Merge replicate regions in their successor region, if a replicate region
-  /// is connected to a successor replicate region with the same predicate by a
-  /// single, empty VPBasicBlock.
-  static bool mergeReplicateRegionsIntoSuccessors(VPlan &Plan);
+  /// Wrap predicated VPReplicateRecipes with a mask operand in an if-then
+  /// region block and remove the mask operand. Optimize the created regions by
+  /// iteratively sinking scalar operands into the region, followed by merging
+  /// regions until no improvements are remaining.
+  static void createAndOptimizeReplicateRegions(VPlan &Plan);
 
   /// Remove redundant VPBasicBlocks by merging them into their predecessor if
   /// the predecessor has a single successor.
diff --git a/llvm/lib/Transforms/Vectorize/VPlanValue.h b/llvm/lib/Transforms/Vectorize/VPlanValue.h
index 62ec65cbfe5dd..fee004cc92d5e 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanValue.h
+++ b/llvm/lib/Transforms/Vectorize/VPlanValue.h
@@ -346,11 +346,11 @@ class VPDef {
     VPWidenMemoryInstructionSC,
     VPWidenSC,
     VPWidenSelectSC,
-
-    // Phi-like recipes. Need to be kept together.
+    // START: Phi-like recipes. Need to be kept together.
     VPBlendSC,
     VPPredInstPHISC,
-    // Header-phi recipes. Need to be kept together.
+    // START: SubclassID for recipes that inherit VPHeaderPHIRecipe.
+    // VPHeaderPHIRecipe need to be kept together.
     VPCanonicalIVPHISC,
     VPActiveLaneMaskPHISC,
     VPFirstOrderRecurrencePHISC,
@@ -358,8 +358,11 @@ class VPDef {
     VPWidenIntOrFpInductionSC,
     VPWidenPointerInductionSC,
     VPReductionPHISC,
+    // END: SubclassID for recipes that inherit VPHeaderPHIRecipe
+    // END: Phi-like recipes
     VPFirstPHISC = VPBlendSC,
     VPFirstHeaderPHISC = VPCanonicalIVPHISC,
+    VPLastHeaderPHISC = VPReductionPHISC,
     VPLastPHISC = VPReductionPHISC,
   };
 
diff --git a/llvm/lib/Transforms/Vectorize/Vectorize.cpp b/llvm/lib/Transforms/Vectorize/Vectorize.cpp
index 6a335e9120bbe..cea47334d5077 100644
--- a/llvm/lib/Transforms/Vectorize/Vectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/Vectorize.cpp
@@ -22,8 +22,6 @@ using namespace llvm;
 
 /// Initialize all passes linked into the Vectorization library.
 void llvm::initializeVectorization(PassRegistry &Registry) {
-  initializeLoopVectorizePass(Registry);
-  initializeSLPVectorizerPass(Registry);
   initializeLoadStoreVectorizerLegacyPassPass(Registry);
 }
 
diff --git a/llvm/lib/WindowsDriver/MSVCPaths.cpp b/llvm/lib/WindowsDriver/MSVCPaths.cpp
index 0aae1bb3a7746..6e8496da17c89 100644
--- a/llvm/lib/WindowsDriver/MSVCPaths.cpp
+++ b/llvm/lib/WindowsDriver/MSVCPaths.cpp
@@ -609,8 +609,9 @@ bool findVCToolChainViaEnvironment(vfs::FileSystem &VFS, std::string &Path,
   return false;
 }
 
-bool findVCToolChainViaSetupConfig(vfs::FileSystem &VFS, std::string &Path,
-                                   ToolsetLayout &VSLayout) {
+bool findVCToolChainViaSetupConfig(vfs::FileSystem &VFS,
+                                   std::optional<StringRef> VCToolsVersion,
+                                   std::string &Path, ToolsetLayout &VSLayout) {
 #if !defined(USE_MSVC_SETUP_API)
   return false;
 #else
@@ -677,17 +678,24 @@ bool findVCToolChainViaSetupConfig(vfs::FileSystem &VFS, std::string &Path,
   std::string VCRootPath;
   convertWideToUTF8(std::wstring(VCPathWide), VCRootPath);
 
-  SmallString<256> ToolsVersionFilePath(VCRootPath);
-  sys::path::append(ToolsVersionFilePath, "Auxiliary", "Build",
-                    "Microsoft.VCToolsVersion.default.txt");
+  std::string ToolsVersion;
+  if (VCToolsVersion.has_value()) {
+    ToolsVersion = *VCToolsVersion;
+  } else {
+    SmallString<256> ToolsVersionFilePath(VCRootPath);
+    sys::path::append(ToolsVersionFilePath, "Auxiliary", "Build",
+                      "Microsoft.VCToolsVersion.default.txt");
+
+    auto ToolsVersionFile = MemoryBuffer::getFile(ToolsVersionFilePath);
+    if (!ToolsVersionFile)
+      return false;
+
+    ToolsVersion = ToolsVersionFile->get()->getBuffer().rtrim();
+  }
 
-  auto ToolsVersionFile = MemoryBuffer::getFile(ToolsVersionFilePath);
-  if (!ToolsVersionFile)
-    return false;
 
   SmallString<256> ToolchainPath(VCRootPath);
-  sys::path::append(ToolchainPath, "Tools", "MSVC",
-                    ToolsVersionFile->get()->getBuffer().rtrim());
+  sys::path::append(ToolchainPath, "Tools", "MSVC", ToolsVersion);
   auto Status = VFS.status(ToolchainPath);
   if (!Status || !Status->isDirectory())
     return false;
diff --git a/llvm/test/Analysis/CostModel/AArch64/shuffle-load.ll b/llvm/test/Analysis/CostModel/AArch64/shuffle-load.ll
index adf908de47f0c..b048f0e86be19 100644
--- a/llvm/test/Analysis/CostModel/AArch64/shuffle-load.ll
+++ b/llvm/test/Analysis/CostModel/AArch64/shuffle-load.ll
@@ -1,5 +1,6 @@
 ; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py
 ; RUN: opt < %s -mtriple=aarch64--linux-gnu -passes="print<cost-model>" 2>&1 -disable-output | FileCheck %s
+; RUN: opt < %s -mtriple=aarch64--linux-gnu -passes="print<cost-model>" -cost-kind=code-size 2>&1 -disable-output | FileCheck %s --check-prefix=CODESIZE
 
 ; These tests check the costs of ld1r instructions, through the
 ; isLegalBroadcastLoad method.
@@ -15,46 +16,93 @@ define void @shuffle() {
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %lv4i8 = load <4 x i8>, ptr undef, align 4
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %sv4i8 = shufflevector <4 x i8> %lv4i8, <4 x i8> undef, <4 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv8i8 = load <8 x i8>, ptr undef, align 8
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv8i8 = shufflevector <8 x i8> %lv8i8, <8 x i8> undef, <8 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %sv8i8 = shufflevector <8 x i8> %lv8i8, <8 x i8> undef, <8 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv16i8 = load <16 x i8>, ptr undef, align 16
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv16i8 = shufflevector <16 x i8> %lv16i8, <16 x i8> undef, <16 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %sv16i8 = shufflevector <16 x i8> %lv16i8, <16 x i8> undef, <16 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %lv2i16 = load <2 x i16>, ptr undef, align 4
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %sv2i16 = shufflevector <2 x i16> %lv2i16, <2 x i16> undef, <2 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv4i16 = load <4 x i16>, ptr undef, align 8
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv4i16 = shufflevector <4 x i16> %lv4i16, <4 x i16> undef, <4 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %sv4i16 = shufflevector <4 x i16> %lv4i16, <4 x i16> undef, <4 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv8i16 = load <8 x i16>, ptr undef, align 16
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv8i16 = shufflevector <8 x i16> %lv8i16, <8 x i16> undef, <8 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %sv8i16 = shufflevector <8 x i16> %lv8i16, <8 x i16> undef, <8 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %lv16i16 = load <16 x i16>, ptr undef, align 32
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv16i16 = shufflevector <16 x i16> %lv16i16, <16 x i16> undef, <16 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %sv16i16 = shufflevector <16 x i16> %lv16i16, <16 x i16> undef, <16 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv2i32 = load <2 x i32>, ptr undef, align 8
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv2i32 = shufflevector <2 x i32> %lv2i32, <2 x i32> undef, <2 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %sv2i32 = shufflevector <2 x i32> %lv2i32, <2 x i32> undef, <2 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv4i32 = load <4 x i32>, ptr undef, align 16
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv4i32 = shufflevector <4 x i32> %lv4i32, <4 x i32> undef, <4 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %sv4i32 = shufflevector <4 x i32> %lv4i32, <4 x i32> undef, <4 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %lv8i32 = load <8 x i32>, ptr undef, align 32
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv8i32 = shufflevector <8 x i32> %lv8i32, <8 x i32> undef, <8 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %sv8i32 = shufflevector <8 x i32> %lv8i32, <8 x i32> undef, <8 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv2i64 = load <2 x i64>, ptr undef, align 16
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv2i64 = shufflevector <2 x i64> %lv2i64, <2 x i64> undef, <2 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %sv2i64 = shufflevector <2 x i64> %lv2i64, <2 x i64> undef, <2 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %lv4i64 = load <4 x i64>, ptr undef, align 32
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv4i64 = shufflevector <4 x i64> %lv4i64, <4 x i64> undef, <4 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %sv4i64 = shufflevector <4 x i64> %lv4i64, <4 x i64> undef, <4 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv2f16 = load <2 x half>, ptr undef, align 4
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv2f16 = shufflevector <2 x half> %lv2f16, <2 x half> undef, <2 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %sv2f16 = shufflevector <2 x half> %lv2f16, <2 x half> undef, <2 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv4f16 = load <4 x half>, ptr undef, align 8
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv4f16 = shufflevector <4 x half> %lv4f16, <4 x half> undef, <4 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %sv4f16 = shufflevector <4 x half> %lv4f16, <4 x half> undef, <4 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv8f16 = load <8 x half>, ptr undef, align 16
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv8f16 = shufflevector <8 x half> %lv8f16, <8 x half> undef, <8 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %sv8f16 = shufflevector <8 x half> %lv8f16, <8 x half> undef, <8 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %lv16f16 = load <16 x half>, ptr undef, align 32
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv16f16 = shufflevector <16 x half> %lv16f16, <16 x half> undef, <16 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %sv16f16 = shufflevector <16 x half> %lv16f16, <16 x half> undef, <16 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv2f32 = load <2 x float>, ptr undef, align 8
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv2f32 = shufflevector <2 x float> %lv2f32, <2 x float> undef, <2 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %sv2f32 = shufflevector <2 x float> %lv2f32, <2 x float> undef, <2 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv4f32 = load <4 x float>, ptr undef, align 16
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv4f32 = shufflevector <4 x float> %lv4f32, <4 x float> undef, <4 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %sv4f32 = shufflevector <4 x float> %lv4f32, <4 x float> undef, <4 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %lv8f32 = load <8 x float>, ptr undef, align 32
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv8f32 = shufflevector <8 x float> %lv8f32, <8 x float> undef, <8 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %sv8f32 = shufflevector <8 x float> %lv8f32, <8 x float> undef, <8 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv2f64 = load <2 x double>, ptr undef, align 16
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv2f64 = shufflevector <2 x double> %lv2f64, <2 x double> undef, <2 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %sv2f64 = shufflevector <2 x double> %lv2f64, <2 x double> undef, <2 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %lv4f64 = load <4 x double>, ptr undef, align 32
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv4f64 = shufflevector <4 x double> %lv4f64, <4 x double> undef, <4 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %sv4f64 = shufflevector <4 x double> %lv4f64, <4 x double> undef, <4 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; CODESIZE-LABEL: 'shuffle'
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv2i8 = load <2 x i8>, ptr undef, align 2
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %sv2i8 = shufflevector <2 x i8> %lv2i8, <2 x i8> undef, <2 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv4i8 = load <4 x i8>, ptr undef, align 4
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %sv4i8 = shufflevector <4 x i8> %lv4i8, <4 x i8> undef, <4 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv8i8 = load <8 x i8>, ptr undef, align 8
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv8i8 = shufflevector <8 x i8> %lv8i8, <8 x i8> undef, <8 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv16i8 = load <16 x i8>, ptr undef, align 16
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv16i8 = shufflevector <16 x i8> %lv16i8, <16 x i8> undef, <16 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv2i16 = load <2 x i16>, ptr undef, align 4
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %sv2i16 = shufflevector <2 x i16> %lv2i16, <2 x i16> undef, <2 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv4i16 = load <4 x i16>, ptr undef, align 8
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv4i16 = shufflevector <4 x i16> %lv4i16, <4 x i16> undef, <4 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv8i16 = load <8 x i16>, ptr undef, align 16
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv8i16 = shufflevector <8 x i16> %lv8i16, <8 x i16> undef, <8 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %lv16i16 = load <16 x i16>, ptr undef, align 32
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv16i16 = shufflevector <16 x i16> %lv16i16, <16 x i16> undef, <16 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv2i32 = load <2 x i32>, ptr undef, align 8
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv2i32 = shufflevector <2 x i32> %lv2i32, <2 x i32> undef, <2 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv4i32 = load <4 x i32>, ptr undef, align 16
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv4i32 = shufflevector <4 x i32> %lv4i32, <4 x i32> undef, <4 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %lv8i32 = load <8 x i32>, ptr undef, align 32
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv8i32 = shufflevector <8 x i32> %lv8i32, <8 x i32> undef, <8 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv2i64 = load <2 x i64>, ptr undef, align 16
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv2i64 = shufflevector <2 x i64> %lv2i64, <2 x i64> undef, <2 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %lv4i64 = load <4 x i64>, ptr undef, align 32
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv4i64 = shufflevector <4 x i64> %lv4i64, <4 x i64> undef, <4 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv2f16 = load <2 x half>, ptr undef, align 4
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv2f16 = shufflevector <2 x half> %lv2f16, <2 x half> undef, <2 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv4f16 = load <4 x half>, ptr undef, align 8
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv4f16 = shufflevector <4 x half> %lv4f16, <4 x half> undef, <4 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv8f16 = load <8 x half>, ptr undef, align 16
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv8f16 = shufflevector <8 x half> %lv8f16, <8 x half> undef, <8 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %lv16f16 = load <16 x half>, ptr undef, align 32
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv16f16 = shufflevector <16 x half> %lv16f16, <16 x half> undef, <16 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv2f32 = load <2 x float>, ptr undef, align 8
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv2f32 = shufflevector <2 x float> %lv2f32, <2 x float> undef, <2 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv4f32 = load <4 x float>, ptr undef, align 16
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv4f32 = shufflevector <4 x float> %lv4f32, <4 x float> undef, <4 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %lv8f32 = load <8 x float>, ptr undef, align 32
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv8f32 = shufflevector <8 x float> %lv8f32, <8 x float> undef, <8 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lv2f64 = load <2 x double>, ptr undef, align 16
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv2f64 = shufflevector <2 x double> %lv2f64, <2 x double> undef, <2 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %lv4f64 = load <4 x double>, ptr undef, align 32
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %sv4f64 = shufflevector <4 x double> %lv4f64, <4 x double> undef, <4 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret void
 ;
   %lv2i8 = load <2 x i8>, ptr undef
   %sv2i8 = shufflevector <2 x i8> %lv2i8, <2 x i8> undef, <2 x i32> zeroinitializer
@@ -119,6 +167,12 @@ define <4 x half> @ld1r_4h_float_shuff(ptr nocapture %x) {
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <4 x half> %tmp1, <4 x half> undef, <4 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret <4 x half> %lane
 ;
+; CODESIZE-LABEL: 'ld1r_4h_float_shuff'
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %tmp = load half, ptr %x, align 2
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = insertelement <4 x half> undef, half %tmp, i32 0
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <4 x half> %tmp1, <4 x half> undef, <4 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret <4 x half> %lane
+;
 entry:
   %tmp = load half, ptr %x, align 2
   %tmp1 = insertelement <4 x half> undef, half %tmp, i32 0
@@ -130,9 +184,15 @@ define <8 x half> @ld1r_8h_float_shuff(ptr nocapture %x) {
 ; CHECK-LABEL: 'ld1r_8h_float_shuff'
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %tmp = load half, ptr %x, align 2
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = insertelement <8 x half> undef, half %tmp, i32 0
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 21 for instruction: %lane = shufflevector <8 x half> %tmp1, <8 x half> undef, <8 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <8 x half> %tmp1, <8 x half> undef, <8 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret <8 x half> %lane
 ;
+; CODESIZE-LABEL: 'ld1r_8h_float_shuff'
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %tmp = load half, ptr %x, align 2
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = insertelement <8 x half> undef, half %tmp, i32 0
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <8 x half> %tmp1, <8 x half> undef, <8 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret <8 x half> %lane
+;
 entry:
   %tmp = load half, ptr %x, align 2
   %tmp1 = insertelement <8 x half> undef, half %tmp, i32 0
@@ -147,6 +207,12 @@ define <2 x float> @ld1r_2s_float_shuff(ptr nocapture %x) {
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <2 x float> %tmp1, <2 x float> undef, <2 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret <2 x float> %lane
 ;
+; CODESIZE-LABEL: 'ld1r_2s_float_shuff'
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %tmp = load float, ptr %x, align 4
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = insertelement <2 x float> undef, float %tmp, i32 0
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <2 x float> %tmp1, <2 x float> undef, <2 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret <2 x float> %lane
+;
 entry:
   %tmp = load float, ptr %x, align 4
   %tmp1 = insertelement <2 x float> undef, float %tmp, i32 0
@@ -161,6 +227,12 @@ define <4 x float> @ld1r_4s_float_shuff(ptr nocapture %x) {
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <4 x float> %tmp1, <4 x float> undef, <4 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret <4 x float> %lane
 ;
+; CODESIZE-LABEL: 'ld1r_4s_float_shuff'
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %tmp = load float, ptr %x, align 4
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = insertelement <4 x float> undef, float %tmp, i32 0
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <4 x float> %tmp1, <4 x float> undef, <4 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret <4 x float> %lane
+;
 entry:
   %tmp = load float, ptr %x, align 4
   %tmp1 = insertelement <4 x float> undef, float %tmp, i32 0
@@ -175,6 +247,12 @@ define <2 x double> @ld1r_2d_double_shuff(ptr nocapture %x) {
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <2 x double> %tmp1, <2 x double> undef, <2 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret <2 x double> %lane
 ;
+; CODESIZE-LABEL: 'ld1r_2d_double_shuff'
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %tmp = load double, ptr %x, align 4
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = insertelement <2 x double> undef, double %tmp, i32 0
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <2 x double> %tmp1, <2 x double> undef, <2 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret <2 x double> %lane
+;
 entry:
   %tmp = load double, ptr %x, align 4
   %tmp1 = insertelement <2 x double> undef, double %tmp, i32 0
@@ -191,6 +269,12 @@ define <8 x i8> @ld1r_8b_int_shuff(ptr nocapture %x) {
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <8 x i8> %tmp1, <8 x i8> undef, <8 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i8> %lane
 ;
+; CODESIZE-LABEL: 'ld1r_8b_int_shuff'
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %tmp = load i8, ptr %x, align 2
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %tmp1 = insertelement <8 x i8> undef, i8 %tmp, i8 0
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <8 x i8> %tmp1, <8 x i8> undef, <8 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret <8 x i8> %lane
+;
 entry:
   %tmp = load i8, ptr %x, align 2
   %tmp1 = insertelement <8 x i8> undef, i8 %tmp, i8 0
@@ -205,6 +289,12 @@ define <16 x i8> @ld1r_16b_int_shuff(ptr nocapture %x) {
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <16 x i8> %tmp1, <16 x i8> undef, <16 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i8> %lane
 ;
+; CODESIZE-LABEL: 'ld1r_16b_int_shuff'
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %tmp = load i8, ptr %x, align 2
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %tmp1 = insertelement <16 x i8> undef, i8 %tmp, i8 0
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <16 x i8> %tmp1, <16 x i8> undef, <16 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret <16 x i8> %lane
+;
 entry:
   %tmp = load i8, ptr %x, align 2
   %tmp1 = insertelement <16 x i8> undef, i8 %tmp, i8 0
@@ -219,6 +309,12 @@ define <4 x i16> @ld1r_4h_int_shuff(ptr nocapture %x) {
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <4 x i16> %tmp1, <4 x i16> undef, <4 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i16> %lane
 ;
+; CODESIZE-LABEL: 'ld1r_4h_int_shuff'
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %tmp = load i16, ptr %x, align 2
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %tmp1 = insertelement <4 x i16> undef, i16 %tmp, i16 0
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <4 x i16> %tmp1, <4 x i16> undef, <4 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret <4 x i16> %lane
+;
 entry:
   %tmp = load i16, ptr %x, align 2
   %tmp1 = insertelement <4 x i16> undef, i16 %tmp, i16 0
@@ -233,6 +329,12 @@ define <8 x i16> @ld1r_8h_int_shuff(ptr nocapture %x) {
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <8 x i16> %tmp1, <8 x i16> undef, <8 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i16> %lane
 ;
+; CODESIZE-LABEL: 'ld1r_8h_int_shuff'
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %tmp = load i16, ptr %x, align 2
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %tmp1 = insertelement <8 x i16> undef, i16 %tmp, i16 0
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <8 x i16> %tmp1, <8 x i16> undef, <8 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret <8 x i16> %lane
+;
 entry:
   %tmp = load i16, ptr %x, align 2
   %tmp1 = insertelement <8 x i16> undef, i16 %tmp, i16 0
@@ -247,6 +349,12 @@ define <2 x i32> @ld1r_2s_int_shuff(ptr nocapture %x) {
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <2 x i32> %tmp1, <2 x i32> undef, <2 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i32> %lane
 ;
+; CODESIZE-LABEL: 'ld1r_2s_int_shuff'
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %tmp = load i32, ptr %x, align 4
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %tmp1 = insertelement <2 x i32> undef, i32 %tmp, i32 0
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <2 x i32> %tmp1, <2 x i32> undef, <2 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret <2 x i32> %lane
+;
 entry:
   %tmp = load i32, ptr %x, align 4
   %tmp1 = insertelement <2 x i32> undef, i32 %tmp, i32 0
@@ -261,6 +369,12 @@ define <4 x i32> @ld1r_4s_int_shuff(ptr nocapture %x) {
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <4 x i32> %tmp1, <4 x i32> undef, <4 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i32> %lane
 ;
+; CODESIZE-LABEL: 'ld1r_4s_int_shuff'
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %tmp = load i32, ptr %x, align 4
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %tmp1 = insertelement <4 x i32> undef, i32 %tmp, i32 0
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <4 x i32> %tmp1, <4 x i32> undef, <4 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret <4 x i32> %lane
+;
 entry:
   %tmp = load i32, ptr %x, align 4
   %tmp1 = insertelement <4 x i32> undef, i32 %tmp, i32 0
@@ -275,6 +389,12 @@ define <2 x i64> @ld1r_2d_int_shuff(ptr nocapture %x) {
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <2 x i64> %tmp1, <2 x i64> undef, <2 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i64> %lane
 ;
+; CODESIZE-LABEL: 'ld1r_2d_int_shuff'
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %tmp = load i64, ptr %x, align 8
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %tmp1 = insertelement <2 x i64> undef, i64 %tmp, i32 0
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %lane = shufflevector <2 x i64> %tmp1, <2 x i64> undef, <2 x i32> zeroinitializer
+; CODESIZE-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret <2 x i64> %lane
+;
 entry:
   %tmp = load i64, ptr %x, align 8
   %tmp1 = insertelement <2 x i64> undef, i64 %tmp, i32 0
diff --git a/llvm/test/Analysis/CostModel/AArch64/shuffle-transpose.ll b/llvm/test/Analysis/CostModel/AArch64/shuffle-transpose.ll
index 8ed09368b3fa4..6680f70aafe4d 100644
--- a/llvm/test/Analysis/CostModel/AArch64/shuffle-transpose.ll
+++ b/llvm/test/Analysis/CostModel/AArch64/shuffle-transpose.ll
@@ -182,3 +182,39 @@ define <2 x double> @trn2.v2f64(<2 x double> %v0, <2 x double> %v1) {
   %tmp0 = shufflevector <2 x double> %v0, <2 x double> %v1, <2 x i32> <i32 1, i32 3>
   ret <2 x double> %tmp0
 }
+
+; COST-LABEL: trn1.v4f16
+; COST:       Found an estimated cost of 1 for instruction: %tmp0 = shufflevector <4 x half> %v0, <4 x half> %v1, <4 x i32> <i32 0, i32 4, i32 2, i32 6>
+; CODE-LABEL: trn1.v4f16
+; CODE:       trn1 v0.4h, v0.4h, v1.4h
+define <4 x half> @trn1.v4f16(<4 x half> %v0, <4 x half> %v1) {
+  %tmp0 = shufflevector <4 x half> %v0, <4 x half> %v1, <4 x i32> <i32 0, i32 4, i32 2, i32 6>
+  ret <4 x half> %tmp0
+}
+
+; COST-LABEL: trn2.v4f16
+; COST:       Found an estimated cost of 1 for instruction: %tmp0 = shufflevector <4 x half> %v0, <4 x half> %v1, <4 x i32> <i32 1, i32 5, i32 3, i32 7>
+; CODE-LABEL: trn2.v4f16
+; CODE:       trn2 v0.4h, v0.4h, v1.4h
+define <4 x half> @trn2.v4f16(<4 x half> %v0, <4 x half> %v1) {
+  %tmp0 = shufflevector <4 x half> %v0, <4 x half> %v1, <4 x i32> <i32 1, i32 5, i32 3, i32 7>
+  ret <4 x half> %tmp0
+}
+
+; COST-LABEL: trn1.v8f16
+; COST:       Found an estimated cost of 1 for instruction: %tmp0 = shufflevector <8 x half> %v0, <8 x half> %v1, <8 x i32> <i32 0, i32 8, i32 2, i32 10, i32 4, i32 12, i32 6, i32 14>
+; CODE-LABEL: trn1.v8f16
+; CODE:       trn1 v0.8h, v0.8h, v1.8h
+define <8 x half> @trn1.v8f16(<8 x half> %v0, <8 x half> %v1) {
+  %tmp0 = shufflevector <8 x half> %v0, <8 x half> %v1, <8 x i32> <i32 0, i32 8, i32 2, i32 10, i32 4, i32 12, i32 6, i32 14>
+  ret <8 x half> %tmp0
+}
+
+; COST-LABEL: trn2.v8f16
+; COST:       Found an estimated cost of 1 for instruction: %tmp0 = shufflevector <8 x half> %v0, <8 x half> %v1, <8 x i32> <i32 1, i32 9, i32 3, i32 11, i32 5, i32 13, i32 7, i32 15>
+; CODE-LABEL: trn2.v8f16
+; CODE:       trn2 v0.8h, v0.8h, v1.8h
+define <8 x half> @trn2.v8f16(<8 x half> %v0, <8 x half> %v1) {
+  %tmp0 = shufflevector <8 x half> %v0, <8 x half> %v1, <8 x i32> <i32 1, i32 9, i32 3, i32 11, i32 5, i32 13, i32 7, i32 15>
+  ret <8 x half> %tmp0
+}
diff --git a/llvm/test/Analysis/CostModel/RISCV/rvv-shuffle.ll b/llvm/test/Analysis/CostModel/RISCV/rvv-shuffle.ll
index 29655e70de45a..f8b8f65135e93 100644
--- a/llvm/test/Analysis/CostModel/RISCV/rvv-shuffle.ll
+++ b/llvm/test/Analysis/CostModel/RISCV/rvv-shuffle.ll
@@ -6,17 +6,17 @@
 define void  @vector_broadcast() {
 ; CHECK-LABEL: 'vector_broadcast'
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %zero = shufflevector <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %1 = shufflevector <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %1 = shufflevector <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %2 = shufflevector <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %3 = shufflevector <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %3 = shufflevector <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %4 = shufflevector <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %5 = shufflevector <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %5 = shufflevector <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %6 = shufflevector <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %7 = shufflevector <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %8 = shufflevector <vscale x 16 x i1> undef, <vscale x 16 x i1> undef, <vscale x 16 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %9 = shufflevector <vscale x 8 x i1> undef, <vscale x 8 x i1> undef, <vscale x 8 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %10 = shufflevector <vscale x 4 x i1> undef, <vscale x 4 x i1> undef, <vscale x 4 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %11 = shufflevector <vscale x 2 x i1> undef, <vscale x 2 x i1> undef, <vscale x 2 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %7 = shufflevector <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 12 for instruction: %8 = shufflevector <vscale x 16 x i1> undef, <vscale x 16 x i1> undef, <vscale x 16 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 6 for instruction: %9 = shufflevector <vscale x 8 x i1> undef, <vscale x 8 x i1> undef, <vscale x 8 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 6 for instruction: %10 = shufflevector <vscale x 4 x i1> undef, <vscale x 4 x i1> undef, <vscale x 4 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 6 for instruction: %11 = shufflevector <vscale x 2 x i1> undef, <vscale x 2 x i1> undef, <vscale x 2 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret void
 ;
   %zero = shufflevector <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i32> zeroinitializer
diff --git a/llvm/test/Analysis/CostModel/RISCV/shuffle-broadcast.ll b/llvm/test/Analysis/CostModel/RISCV/shuffle-broadcast.ll
index e7b0d2abe53cc..8435bd7840bc6 100644
--- a/llvm/test/Analysis/CostModel/RISCV/shuffle-broadcast.ll
+++ b/llvm/test/Analysis/CostModel/RISCV/shuffle-broadcast.ll
@@ -6,47 +6,47 @@ define void  @broadcast() #0{
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %zero = shufflevector <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %1 = shufflevector <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %2 = shufflevector <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %3 = shufflevector <vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %4 = shufflevector <vscale x 16 x half> undef, <vscale x 16 x half> undef, <vscale x 16 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %5 = shufflevector <vscale x 32 x half> undef, <vscale x 32 x half> undef, <vscale x 32 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %3 = shufflevector <vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %4 = shufflevector <vscale x 16 x half> undef, <vscale x 16 x half> undef, <vscale x 16 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 8 for instruction: %5 = shufflevector <vscale x 32 x half> undef, <vscale x 32 x half> undef, <vscale x 32 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %6 = shufflevector <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %7 = shufflevector <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %8 = shufflevector <vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %9 = shufflevector <vscale x 8 x float> undef, <vscale x 8 x float> undef, <vscale x 8 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %10 = shufflevector <vscale x 16 x float> undef, <vscale x 16 x float> undef, <vscale x 16 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %8 = shufflevector <vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %9 = shufflevector <vscale x 8 x float> undef, <vscale x 8 x float> undef, <vscale x 8 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 8 for instruction: %10 = shufflevector <vscale x 16 x float> undef, <vscale x 16 x float> undef, <vscale x 16 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %11 = shufflevector <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %12 = shufflevector <vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %13 = shufflevector <vscale x 4 x double> undef, <vscale x 4 x double> undef, <vscale x 4 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %14 = shufflevector <vscale x 8 x double> undef, <vscale x 8 x double> undef, <vscale x 8 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %12 = shufflevector <vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %13 = shufflevector <vscale x 4 x double> undef, <vscale x 4 x double> undef, <vscale x 4 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 8 for instruction: %14 = shufflevector <vscale x 8 x double> undef, <vscale x 8 x double> undef, <vscale x 8 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %15 = shufflevector <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %16 = shufflevector <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %17 = shufflevector <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %18 = shufflevector <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %19 = shufflevector <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %20 = shufflevector <vscale x 32 x i8> undef, <vscale x 32 x i8> undef, <vscale x 32 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %21 = shufflevector <vscale x 64 x i8> undef, <vscale x 64 x i8> undef, <vscale x 64 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %19 = shufflevector <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %20 = shufflevector <vscale x 32 x i8> undef, <vscale x 32 x i8> undef, <vscale x 32 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 8 for instruction: %21 = shufflevector <vscale x 64 x i8> undef, <vscale x 64 x i8> undef, <vscale x 64 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %22 = shufflevector <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %23 = shufflevector <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %24 = shufflevector <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %25 = shufflevector <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %26 = shufflevector <vscale x 16 x i16> undef, <vscale x 16 x i16> undef, <vscale x 16 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %27 = shufflevector <vscale x 32 x i16> undef, <vscale x 32 x i16> undef, <vscale x 32 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %25 = shufflevector <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %26 = shufflevector <vscale x 16 x i16> undef, <vscale x 16 x i16> undef, <vscale x 16 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 8 for instruction: %27 = shufflevector <vscale x 32 x i16> undef, <vscale x 32 x i16> undef, <vscale x 32 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %28 = shufflevector <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %29 = shufflevector <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %30 = shufflevector <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %31 = shufflevector <vscale x 8 x i32> undef, <vscale x 8 x i32> undef, <vscale x 8 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %32 = shufflevector <vscale x 16 x i32> undef, <vscale x 16 x i32> undef, <vscale x 16 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %30 = shufflevector <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %31 = shufflevector <vscale x 8 x i32> undef, <vscale x 8 x i32> undef, <vscale x 8 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 8 for instruction: %32 = shufflevector <vscale x 16 x i32> undef, <vscale x 16 x i32> undef, <vscale x 16 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %33 = shufflevector <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %34 = shufflevector <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %35 = shufflevector <vscale x 4 x i64> undef, <vscale x 4 x i64> undef, <vscale x 4 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %36 = shufflevector <vscale x 8 x i64> undef, <vscale x 8 x i64> undef, <vscale x 8 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %37 = shufflevector <vscale x 1 x i1> undef, <vscale x 1 x i1> undef, <vscale x 1 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %38 = shufflevector <vscale x 2 x i1> undef, <vscale x 2 x i1> undef, <vscale x 2 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %39 = shufflevector <vscale x 4 x i1> undef, <vscale x 4 x i1> undef, <vscale x 4 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %40 = shufflevector <vscale x 8 x i1> undef, <vscale x 8 x i1> undef, <vscale x 8 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %41 = shufflevector <vscale x 16 x i1> undef, <vscale x 16 x i1> undef, <vscale x 16 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %42 = shufflevector <vscale x 32 x i1> undef, <vscale x 32 x i1> undef, <vscale x 32 x i32> zeroinitializer
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %43 = shufflevector <vscale x 64 x i1> undef, <vscale x 64 x i1> undef, <vscale x 64 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %34 = shufflevector <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %35 = shufflevector <vscale x 4 x i64> undef, <vscale x 4 x i64> undef, <vscale x 4 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 8 for instruction: %36 = shufflevector <vscale x 8 x i64> undef, <vscale x 8 x i64> undef, <vscale x 8 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 6 for instruction: %37 = shufflevector <vscale x 1 x i1> undef, <vscale x 1 x i1> undef, <vscale x 1 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 6 for instruction: %38 = shufflevector <vscale x 2 x i1> undef, <vscale x 2 x i1> undef, <vscale x 2 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 6 for instruction: %39 = shufflevector <vscale x 4 x i1> undef, <vscale x 4 x i1> undef, <vscale x 4 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 6 for instruction: %40 = shufflevector <vscale x 8 x i1> undef, <vscale x 8 x i1> undef, <vscale x 8 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 12 for instruction: %41 = shufflevector <vscale x 16 x i1> undef, <vscale x 16 x i1> undef, <vscale x 16 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 24 for instruction: %42 = shufflevector <vscale x 32 x i1> undef, <vscale x 32 x i1> undef, <vscale x 32 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 48 for instruction: %43 = shufflevector <vscale x 64 x i1> undef, <vscale x 64 x i1> undef, <vscale x 64 x i32> zeroinitializer
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret void
 ;
   %zero = shufflevector <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x i32> zeroinitializer
diff --git a/llvm/test/Analysis/CostModel/RISCV/shuffle-interleave.ll b/llvm/test/Analysis/CostModel/RISCV/shuffle-interleave.ll
new file mode 100644
index 0000000000000..97480d8e813c6
--- /dev/null
+++ b/llvm/test/Analysis/CostModel/RISCV/shuffle-interleave.ll
@@ -0,0 +1,70 @@
+; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py UTC_ARGS: --version 2
+; RUN: opt < %s -passes="print<cost-model>" 2>&1 -disable-output -mtriple=riscv32 -mattr=+v | FileCheck %s -check-prefixes=CHECK,RV32
+; RUN: opt < %s -passes="print<cost-model>" 2>&1 -disable-output -mtriple=riscv64 -mattr=+v | FileCheck %s -check-prefixes=CHECK,RV64
+
+; The mask here interleaves (%v1, %v0), not (%v0, %v1): it should still be cheap.
+define <4 x i8> @interleave2_v2i8(<2 x i8> %v0, <2 x i8> %v1) {
+; CHECK-LABEL: 'interleave2_v2i8'
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 7 for instruction: %concat = shufflevector <2 x i8> %v0, <2 x i8> %v1, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %res = shufflevector <4 x i8> %concat, <4 x i8> poison, <4 x i32> <i32 2, i32 0, i32 3, i32 1>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret <4 x i8> %res
+;
+  %concat = shufflevector <2 x i8> %v0, <2 x i8> %v1, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+  %res = shufflevector <4 x i8> %concat, <4 x i8> poison, <4 x i32> <i32 2, i32 0, i32 3, i32 1>
+  ret <4 x i8> %res
+}
+
+define <8 x i8> @interleave2_v8i8(<4 x i8> %v0, <4 x i8> %v1) {
+; CHECK-LABEL: 'interleave2_v8i8'
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 15 for instruction: %concat = shufflevector <4 x i8> %v0, <4 x i8> %v1, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %res = shufflevector <8 x i8> %concat, <8 x i8> poison, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret <8 x i8> %res
+;
+  %concat = shufflevector <4 x i8> %v0, <4 x i8> %v1, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+  %res = shufflevector <8 x i8> %concat, <8 x i8> poison, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
+  ret <8 x i8> %res
+}
+
+define <8 x i32> @interleave2_v8i32(<4 x i32> %v0, <4 x i32> %v1) {
+; CHECK-LABEL: 'interleave2_v8i32'
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 15 for instruction: %concat = shufflevector <4 x i32> %v0, <4 x i32> %v1, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %res = shufflevector <8 x i32> %concat, <8 x i32> poison, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret <8 x i32> %res
+;
+  %concat = shufflevector <4 x i32> %v0, <4 x i32> %v1, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+  %res = shufflevector <8 x i32> %concat, <8 x i32> poison, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
+  ret <8 x i32> %res
+}
+
+; Should be expensive on RV32 because it can't widen
+define <8 x i64> @interleave2_v8i64(<4 x i64> %v0, <4 x i64> %v1) {
+; RV32-LABEL: 'interleave2_v8i64'
+; RV32-NEXT:  Cost Model: Found an estimated cost of 35 for instruction: %concat = shufflevector <4 x i64> %v0, <4 x i64> %v1, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+; RV32-NEXT:  Cost Model: Found an estimated cost of 70 for instruction: %res = shufflevector <8 x i64> %concat, <8 x i64> poison, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
+; RV32-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret <8 x i64> %res
+;
+; RV64-LABEL: 'interleave2_v8i64'
+; RV64-NEXT:  Cost Model: Found an estimated cost of 15 for instruction: %concat = shufflevector <4 x i64> %v0, <4 x i64> %v1, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+; RV64-NEXT:  Cost Model: Found an estimated cost of 30 for instruction: %res = shufflevector <8 x i64> %concat, <8 x i64> poison, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
+; RV64-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret <8 x i64> %res
+;
+  %concat = shufflevector <4 x i64> %v0, <4 x i64> %v1, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+  %res = shufflevector <8 x i64> %concat, <8 x i64> poison, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
+  ret <8 x i64> %res
+}
+
+; TODO: getInstructionCost doesn't call getShuffleCost here because the shuffle changes length
+define {<4 x i8>, <4 x i8>} @deinterleave_2(<8 x i8> %v) {
+; CHECK-LABEL: 'deinterleave_2'
+; CHECK-NEXT:  Cost Model: Found an estimated cost of -1 for instruction: %v0 = shufflevector <8 x i8> %v, <8 x i8> poison, <4 x i32> <i32 0, i32 2, i32 4, i32 6>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of -1 for instruction: %v1 = shufflevector <8 x i8> %v, <8 x i8> poison, <4 x i32> <i32 1, i32 3, i32 5, i32 7>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of -1 for instruction: %res0 = insertvalue { <4 x i8>, <4 x i8> } poison, <4 x i8> %v0, 0
+; CHECK-NEXT:  Cost Model: Found an estimated cost of -1 for instruction: %res1 = insertvalue { <4 x i8>, <4 x i8> } %res0, <4 x i8> %v1, 1
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret { <4 x i8>, <4 x i8> } %res1
+;
+  %v0 = shufflevector <8 x i8> %v, <8 x i8> poison, <4 x i32> <i32 0, i32 2, i32 4, i32 6>
+  %v1 = shufflevector <8 x i8> %v, <8 x i8> poison, <4 x i32> <i32 1, i32 3, i32 5, i32 7>
+  %res0 = insertvalue {<4 x i8>, <4 x i8>} poison, <4 x i8> %v0, 0
+  %res1 = insertvalue {<4 x i8>, <4 x i8>} %res0, <4 x i8> %v1, 1
+  ret {<4 x i8>, <4 x i8>} %res1
+}
diff --git a/llvm/test/Analysis/CostModel/RISCV/shuffle-reverse.ll b/llvm/test/Analysis/CostModel/RISCV/shuffle-reverse.ll
index 7de34781a02b9..ae9d34dc84358 100644
--- a/llvm/test/Analysis/CostModel/RISCV/shuffle-reverse.ll
+++ b/llvm/test/Analysis/CostModel/RISCV/shuffle-reverse.ll
@@ -8,28 +8,28 @@
 define void @reverse() {
 ;
 ; CHECK-LABEL: 'reverse'
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 6 for instruction: %v2i8 = shufflevector <2 x i8> undef, <2 x i8> undef, <2 x i32> <i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 14 for instruction: %v4i8 = shufflevector <4 x i8> undef, <4 x i8> undef, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 30 for instruction: %v8i8 = shufflevector <8 x i8> undef, <8 x i8> undef, <8 x i32> <i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 62 for instruction: %v16i8 = shufflevector <16 x i8> undef, <16 x i8> undef, <16 x i32> <i32 15, i32 14, i32 13, i32 12, i32 11, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 6 for instruction: %v2i16 = shufflevector <2 x i16> undef, <2 x i16> undef, <2 x i32> <i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 14 for instruction: %v4i16 = shufflevector <4 x i16> undef, <4 x i16> undef, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 30 for instruction: %v8i16 = shufflevector <8 x i16> undef, <8 x i16> undef, <8 x i32> <i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 62 for instruction: %v16i16 = shufflevector <16 x i16> undef, <16 x i16> undef, <16 x i32> <i32 15, i32 14, i32 13, i32 12, i32 11, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 6 for instruction: %v2i32 = shufflevector <2 x i32> undef, <2 x i32> undef, <2 x i32> <i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 14 for instruction: %v4i32 = shufflevector <4 x i32> undef, <4 x i32> undef, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 30 for instruction: %v8i32 = shufflevector <8 x i32> undef, <8 x i32> undef, <8 x i32> <i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 16 for instruction: %v2i64 = shufflevector <2 x i64> undef, <2 x i64> undef, <2 x i32> <i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 34 for instruction: %v4i64 = shufflevector <4 x i64> undef, <4 x i64> undef, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 6 for instruction: %v2f16 = shufflevector <2 x half> undef, <2 x half> undef, <2 x i32> <i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 14 for instruction: %v4f16 = shufflevector <4 x half> undef, <4 x half> undef, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 30 for instruction: %v8f16 = shufflevector <8 x half> undef, <8 x half> undef, <8 x i32> <i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 62 for instruction: %v16f16 = shufflevector <16 x half> undef, <16 x half> undef, <16 x i32> <i32 15, i32 14, i32 13, i32 12, i32 11, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 6 for instruction: %v2f32 = shufflevector <2 x float> undef, <2 x float> undef, <2 x i32> <i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 14 for instruction: %v4f32 = shufflevector <4 x float> undef, <4 x float> undef, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 30 for instruction: %v8f32 = shufflevector <8 x float> undef, <8 x float> undef, <8 x i32> <i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 6 for instruction: %v2f64 = shufflevector <2 x double> undef, <2 x double> undef, <2 x i32> <i32 1, i32 0>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 14 for instruction: %v4f64 = shufflevector <4 x double> undef, <4 x double> undef, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v2i8 = shufflevector <2 x i8> undef, <2 x i8> undef, <2 x i32> <i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v4i8 = shufflevector <4 x i8> undef, <4 x i8> undef, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v8i8 = shufflevector <8 x i8> undef, <8 x i8> undef, <8 x i32> <i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v16i8 = shufflevector <16 x i8> undef, <16 x i8> undef, <16 x i32> <i32 15, i32 14, i32 13, i32 12, i32 11, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v2i16 = shufflevector <2 x i16> undef, <2 x i16> undef, <2 x i32> <i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v4i16 = shufflevector <4 x i16> undef, <4 x i16> undef, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v8i16 = shufflevector <8 x i16> undef, <8 x i16> undef, <8 x i32> <i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v16i16 = shufflevector <16 x i16> undef, <16 x i16> undef, <16 x i32> <i32 15, i32 14, i32 13, i32 12, i32 11, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v2i32 = shufflevector <2 x i32> undef, <2 x i32> undef, <2 x i32> <i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v4i32 = shufflevector <4 x i32> undef, <4 x i32> undef, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v8i32 = shufflevector <8 x i32> undef, <8 x i32> undef, <8 x i32> <i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v2i64 = shufflevector <2 x i64> undef, <2 x i64> undef, <2 x i32> <i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v4i64 = shufflevector <4 x i64> undef, <4 x i64> undef, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v2f16 = shufflevector <2 x half> undef, <2 x half> undef, <2 x i32> <i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v4f16 = shufflevector <4 x half> undef, <4 x half> undef, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v8f16 = shufflevector <8 x half> undef, <8 x half> undef, <8 x i32> <i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v16f16 = shufflevector <16 x half> undef, <16 x half> undef, <16 x i32> <i32 15, i32 14, i32 13, i32 12, i32 11, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v2f32 = shufflevector <2 x float> undef, <2 x float> undef, <2 x i32> <i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v4f32 = shufflevector <4 x float> undef, <4 x float> undef, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v8f32 = shufflevector <8 x float> undef, <8 x float> undef, <8 x i32> <i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v2f64 = shufflevector <2 x double> undef, <2 x double> undef, <2 x i32> <i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %v4f64 = shufflevector <4 x double> undef, <4 x double> undef, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret void
 ;
   %v2i8 = shufflevector <2 x i8> undef, <2 x i8> undef, <2 x i32> <i32 1, i32 0>
diff --git a/llvm/test/Analysis/CostModel/X86/free-intrinsics.ll b/llvm/test/Analysis/CostModel/X86/free-intrinsics.ll
index 4d31a7ccd4279..3c39e456eed5f 100644
--- a/llvm/test/Analysis/CostModel/X86/free-intrinsics.ll
+++ b/llvm/test/Analysis/CostModel/X86/free-intrinsics.ll
@@ -6,8 +6,11 @@ define i32 @trivially_free() {
 ; CHECK-SIZE-LABEL: 'trivially_free'
 ; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %a0 = call i32 @llvm.annotation.i32.p0(i32 undef, ptr undef, ptr undef, i32 undef)
 ; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.assume(i1 undef)
-; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.experimental.noalias.scope.decl(metadata !0)
+; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.experimental.noalias.scope.decl(metadata !3)
 ; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.sideeffect()
+; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.dbg.assign(metadata ptr undef, metadata !6, metadata !DIExpression(), metadata !8, metadata ptr undef, metadata !DIExpression()), !dbg !9
+; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.dbg.declare(metadata ptr undef, metadata !6, metadata !DIExpression()), !dbg !9
+; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.dbg.label(metadata !10), !dbg !9
 ; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %a1 = call ptr @llvm.invariant.start.p0(i64 1, ptr undef)
 ; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.invariant.end.p0(ptr undef, i64 1, ptr undef)
 ; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %a2 = call ptr @llvm.launder.invariant.group.p0(ptr undef)
@@ -23,8 +26,11 @@ define i32 @trivially_free() {
 ; CHECK-THROUGHPUT-LABEL: 'trivially_free'
 ; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %a0 = call i32 @llvm.annotation.i32.p0(i32 undef, ptr undef, ptr undef, i32 undef)
 ; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.assume(i1 undef)
-; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.experimental.noalias.scope.decl(metadata !0)
+; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.experimental.noalias.scope.decl(metadata !3)
 ; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.sideeffect()
+; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.dbg.assign(metadata ptr undef, metadata !6, metadata !DIExpression(), metadata !8, metadata ptr undef, metadata !DIExpression()), !dbg !9
+; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.dbg.declare(metadata ptr undef, metadata !6, metadata !DIExpression()), !dbg !9
+; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.dbg.label(metadata !10), !dbg !9
 ; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %a1 = call ptr @llvm.invariant.start.p0(i64 1, ptr undef)
 ; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.invariant.end.p0(ptr undef, i64 1, ptr undef)
 ; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %a2 = call ptr @llvm.launder.invariant.group.p0(ptr undef)
@@ -41,9 +47,10 @@ define i32 @trivially_free() {
   call void @llvm.assume(i1 undef)
   call void @llvm.experimental.noalias.scope.decl(metadata !4)
   call void @llvm.sideeffect()
-  call void @llvm.dbg.declare(metadata ptr undef, metadata !0, metadata !DIExpression())
-  call void @llvm.dbg.value(metadata i64 undef, i64 undef, metadata !DIExpression(), metadata !DIExpression())
-  call void @llvm.dbg.label(metadata !2)
+  call void @llvm.dbg.assign(metadata ptr undef, metadata !0, metadata !DIExpression(), metadata !10, metadata ptr undef, metadata !DIExpression()), !dbg !8
+  call void @llvm.dbg.declare(metadata ptr undef, metadata !0, metadata !DIExpression()), !dbg !8
+  call void @llvm.dbg.value(metadata i64 undef, i64 undef, metadata !DIExpression(), metadata !DIExpression()), !dbg !8
+  call void @llvm.dbg.label(metadata !2), !dbg !8
   %a1 = call ptr @llvm.invariant.start.p0(i64 1, ptr undef)
   call void @llvm.invariant.end.p0(ptr undef, i64 1, ptr undef)
   %a2 = call ptr @llvm.launder.invariant.group.p0(ptr undef)
@@ -61,6 +68,7 @@ declare i32 @llvm.annotation.i32(i32, ptr, ptr, i32)
 declare void @llvm.assume(i1)
 declare void @llvm.experimental.noalias.scope.decl(metadata)
 declare void @llvm.sideeffect()
+declare void @llvm.dbg.assign(metadata, metadata, metadata, metadata, metadata, metadata)
 declare void @llvm.dbg.declare(metadata, metadata, metadata)
 declare void @llvm.dbg.value(metadata, i64, metadata, metadata)
 declare void @llvm.dbg.label(metadata)
@@ -75,11 +83,17 @@ declare i64 @llvm.objectsize.i64.p0(ptr, i1, i1, i1)
 declare ptr @llvm.ptr.annotation.p0(ptr, ptr, ptr, i32, ptr)
 declare void @llvm.var.annotation(ptr, ptr, ptr, i32, ptr)
 
+!llvm.dbg.cu = !{!9}
+!llvm.module.flags = !{!7}
 
 !0 = !DILocalVariable(scope: !1)
-!1 = distinct !DISubprogram(name: "dummy", line: 79, isLocal: true, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: true)
+!1 = distinct !DISubprogram(name: "dummy", line: 79, isLocal: true, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: true, file: !3, unit: !9)
 !2 = !DILabel(scope: !1, name: "label", file: !3, line: 7)
 !3 = !DIFile(filename: "debug-label.c", directory: "./")
 !4 = !{ !5 }
 !5 = distinct !{ !5, !6, !"foo: var" }
 !6 = distinct !{ !6, !"foo" }
+!7 = !{i32 2, !"Debug Info Version", i32 3}
+!8 = !DILocation(line: 0, scope: !1)
+!9 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, producer: "clang")
+!10 = distinct !DIAssignID()
diff --git a/llvm/test/Analysis/CostModel/free-intrinsics-datalayout.ll b/llvm/test/Analysis/CostModel/free-intrinsics-datalayout.ll
index 249b610791990..b5f204c2d9b26 100644
--- a/llvm/test/Analysis/CostModel/free-intrinsics-datalayout.ll
+++ b/llvm/test/Analysis/CostModel/free-intrinsics-datalayout.ll
@@ -8,8 +8,11 @@ define i32 @trivially_free() {
 ; CHECK-SIZE-LABEL: 'trivially_free'
 ; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %a0 = call i32 @llvm.annotation.i32.p0(i32 undef, ptr undef, ptr undef, i32 undef)
 ; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.assume(i1 undef)
-; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.experimental.noalias.scope.decl(metadata !0)
+; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.experimental.noalias.scope.decl(metadata !3)
 ; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.sideeffect()
+; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.dbg.assign(metadata ptr undef, metadata !6, metadata !DIExpression(), metadata !8, metadata ptr undef, metadata !DIExpression()), !dbg !9
+; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.dbg.declare(metadata ptr undef, metadata !6, metadata !DIExpression()), !dbg !9
+; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.dbg.label(metadata !10), !dbg !9
 ; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %a1 = call ptr @llvm.invariant.start.p0(i64 1, ptr undef)
 ; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.invariant.end.p0(ptr undef, i64 1, ptr undef)
 ; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %a2 = call ptr @llvm.launder.invariant.group.p0(ptr undef)
@@ -25,8 +28,11 @@ define i32 @trivially_free() {
 ; CHECK-THROUGHPUT-LABEL: 'trivially_free'
 ; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %a0 = call i32 @llvm.annotation.i32.p0(i32 undef, ptr undef, ptr undef, i32 undef)
 ; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.assume(i1 undef)
-; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.experimental.noalias.scope.decl(metadata !0)
+; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.experimental.noalias.scope.decl(metadata !3)
 ; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.sideeffect()
+; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.dbg.assign(metadata ptr undef, metadata !6, metadata !DIExpression(), metadata !8, metadata ptr undef, metadata !DIExpression()), !dbg !9
+; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.dbg.declare(metadata ptr undef, metadata !6, metadata !DIExpression()), !dbg !9
+; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.dbg.label(metadata !10), !dbg !9
 ; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %a1 = call ptr @llvm.invariant.start.p0(i64 1, ptr undef)
 ; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.invariant.end.p0(ptr undef, i64 1, ptr undef)
 ; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %a2 = call ptr @llvm.launder.invariant.group.p0(ptr undef)
@@ -43,9 +49,10 @@ define i32 @trivially_free() {
   call void @llvm.assume(i1 undef)
   call void @llvm.experimental.noalias.scope.decl(metadata !4)
   call void @llvm.sideeffect()
-  call void @llvm.dbg.declare(metadata ptr undef, metadata !0, metadata !DIExpression())
-  call void @llvm.dbg.value(metadata i64 undef, i64 undef, metadata !DIExpression(), metadata !DIExpression())
-  call void @llvm.dbg.label(metadata !2)
+  call void @llvm.dbg.assign(metadata ptr undef, metadata !0, metadata !DIExpression(), metadata !10, metadata ptr undef, metadata !DIExpression()), !dbg !8
+  call void @llvm.dbg.declare(metadata ptr undef, metadata !0, metadata !DIExpression()), !dbg !8
+  call void @llvm.dbg.value(metadata i64 undef, i64 undef, metadata !DIExpression(), metadata !DIExpression()), !dbg !8
+  call void @llvm.dbg.label(metadata !2), !dbg !8
   %a1 = call ptr @llvm.invariant.start.p0(i64 1, ptr undef)
   call void @llvm.invariant.end.p0(ptr undef, i64 1, ptr undef)
   %a2 = call ptr @llvm.launder.invariant.group.p0(ptr undef)
@@ -63,6 +70,7 @@ declare i32 @llvm.annotation.i32(i32, ptr, ptr, i32)
 declare void @llvm.assume(i1)
 declare void @llvm.experimental.noalias.scope.decl(metadata)
 declare void @llvm.sideeffect()
+declare void @llvm.dbg.assign(metadata, metadata, metadata, metadata, metadata, metadata)
 declare void @llvm.dbg.declare(metadata, metadata, metadata)
 declare void @llvm.dbg.value(metadata, i64, metadata, metadata)
 declare void @llvm.dbg.label(metadata)
@@ -77,11 +85,17 @@ declare i64 @llvm.objectsize.i64.p0(ptr, i1, i1, i1)
 declare ptr @llvm.ptr.annotation.p0(ptr, ptr, ptr, i32, ptr)
 declare void @llvm.var.annotation(ptr, ptr, ptr, i32, ptr)
 
+!llvm.dbg.cu = !{!9}
+!llvm.module.flags = !{!7}
 
 !0 = !DILocalVariable(scope: !1)
-!1 = distinct !DISubprogram(name: "dummy", line: 79, isLocal: true, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: true)
+!1 = distinct !DISubprogram(name: "dummy", line: 79, isLocal: true, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: true, file: !3, unit: !9)
 !2 = !DILabel(scope: !1, name: "label", file: !3, line: 7)
 !3 = !DIFile(filename: "debug-label.c", directory: "./")
 !4 = !{ !5 }
 !5 = distinct !{ !5, !6, !"foo: var" }
 !6 = distinct !{ !6, !"foo" }
+!7 = !{i32 2, !"Debug Info Version", i32 3}
+!8 = !DILocation(line: 0, scope: !1)
+!9 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, producer: "clang")
+!10 = distinct !DIAssignID()
diff --git a/llvm/test/Analysis/CostModel/free-intrinsics-no_info.ll b/llvm/test/Analysis/CostModel/free-intrinsics-no_info.ll
index 715b124ae1fd0..5d4dcaa7cbe2f 100644
--- a/llvm/test/Analysis/CostModel/free-intrinsics-no_info.ll
+++ b/llvm/test/Analysis/CostModel/free-intrinsics-no_info.ll
@@ -6,8 +6,11 @@ define i32 @trivially_free() {
 ; CHECK-SIZE-LABEL: 'trivially_free'
 ; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %a0 = call i32 @llvm.annotation.i32.p0(i32 undef, ptr undef, ptr undef, i32 undef)
 ; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.assume(i1 undef)
-; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.experimental.noalias.scope.decl(metadata !0)
+; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.experimental.noalias.scope.decl(metadata !3)
 ; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.sideeffect()
+; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.dbg.assign(metadata ptr undef, metadata !6, metadata !DIExpression(), metadata !8, metadata ptr undef, metadata !DIExpression()), !dbg !9
+; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.dbg.declare(metadata ptr undef, metadata !6, metadata !DIExpression()), !dbg !9
+; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.dbg.label(metadata !10), !dbg !9
 ; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %a1 = call ptr @llvm.invariant.start.p0(i64 1, ptr undef)
 ; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.invariant.end.p0(ptr undef, i64 1, ptr undef)
 ; CHECK-SIZE-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %a2 = call ptr @llvm.launder.invariant.group.p0(ptr undef)
@@ -23,8 +26,11 @@ define i32 @trivially_free() {
 ; CHECK-THROUGHPUT-LABEL: 'trivially_free'
 ; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %a0 = call i32 @llvm.annotation.i32.p0(i32 undef, ptr undef, ptr undef, i32 undef)
 ; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.assume(i1 undef)
-; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.experimental.noalias.scope.decl(metadata !0)
+; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.experimental.noalias.scope.decl(metadata !3)
 ; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.sideeffect()
+; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.dbg.assign(metadata ptr undef, metadata !6, metadata !DIExpression(), metadata !8, metadata ptr undef, metadata !DIExpression()), !dbg !9
+; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.dbg.declare(metadata ptr undef, metadata !6, metadata !DIExpression()), !dbg !9
+; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.dbg.label(metadata !10), !dbg !9
 ; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %a1 = call ptr @llvm.invariant.start.p0(i64 1, ptr undef)
 ; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: call void @llvm.invariant.end.p0(ptr undef, i64 1, ptr undef)
 ; CHECK-THROUGHPUT-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %a2 = call ptr @llvm.launder.invariant.group.p0(ptr undef)
@@ -41,9 +47,10 @@ define i32 @trivially_free() {
   call void @llvm.assume(i1 undef)
   call void @llvm.experimental.noalias.scope.decl(metadata !4)
   call void @llvm.sideeffect()
-  call void @llvm.dbg.declare(metadata ptr undef, metadata !0, metadata !DIExpression())
-  call void @llvm.dbg.value(metadata i64 undef, i64 undef, metadata !DIExpression(), metadata !DIExpression())
-  call void @llvm.dbg.label(metadata !2)
+  call void @llvm.dbg.assign(metadata ptr undef, metadata !0, metadata !DIExpression(), metadata !10, metadata ptr undef, metadata !DIExpression()), !dbg !8
+  call void @llvm.dbg.declare(metadata ptr undef, metadata !0, metadata !DIExpression()), !dbg !8
+  call void @llvm.dbg.value(metadata i64 undef, i64 undef, metadata !DIExpression(), metadata !DIExpression()), !dbg !8
+  call void @llvm.dbg.label(metadata !2), !dbg !8
   %a1 = call ptr @llvm.invariant.start.p0(i64 1, ptr undef)
   call void @llvm.invariant.end.p0(ptr undef, i64 1, ptr undef)
   %a2 = call ptr @llvm.launder.invariant.group.p0(ptr undef)
@@ -61,6 +68,7 @@ declare i32 @llvm.annotation.i32(i32, ptr, ptr, i32)
 declare void @llvm.assume(i1)
 declare void @llvm.experimental.noalias.scope.decl(metadata)
 declare void @llvm.sideeffect()
+declare void @llvm.dbg.assign(metadata, metadata, metadata, metadata, metadata, metadata)
 declare void @llvm.dbg.declare(metadata, metadata, metadata)
 declare void @llvm.dbg.value(metadata, i64, metadata, metadata)
 declare void @llvm.dbg.label(metadata)
@@ -75,11 +83,17 @@ declare i64 @llvm.objectsize.i64.p0(ptr, i1, i1, i1)
 declare ptr @llvm.ptr.annotation.p0(ptr, ptr, ptr, i32, ptr)
 declare void @llvm.var.annotation(ptr, ptr, ptr, i32, ptr)
 
+!llvm.dbg.cu = !{!9}
+!llvm.module.flags = !{!7}
 
 !0 = !DILocalVariable(scope: !1)
-!1 = distinct !DISubprogram(name: "dummy", line: 79, isLocal: true, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: true)
+!1 = distinct !DISubprogram(name: "dummy", line: 79, isLocal: true, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: true, file: !3, unit: !9)
 !2 = !DILabel(scope: !1, name: "label", file: !3, line: 7)
 !3 = !DIFile(filename: "debug-label.c", directory: "./")
 !4 = !{ !5 }
 !5 = distinct !{ !5, !6, !"foo: var" }
 !6 = distinct !{ !6, !"foo" }
+!7 = !{i32 2, !"Debug Info Version", i32 3}
+!8 = !DILocation(line: 0, scope: !1)
+!9 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, producer: "clang")
+!10 = distinct !DIAssignID()
diff --git a/llvm/test/Analysis/Delinearization/multidim_only_ivs_3d_cast.ll b/llvm/test/Analysis/Delinearization/multidim_only_ivs_3d_cast.ll
index 4820811f68f24..e8ee7283e4040 100644
--- a/llvm/test/Analysis/Delinearization/multidim_only_ivs_3d_cast.ll
+++ b/llvm/test/Analysis/Delinearization/multidim_only_ivs_3d_cast.ll
@@ -10,7 +10,7 @@
 ; AddRec: {{{%A,+,(8 * (zext i32 %m to i64) * (zext i32 %o to i64))}<%for.i>,+,(8 * (zext i32 %o to i64))}<%for.j>,+,8}<%for.k>
 ; CHECK: Base offset: %A
 ; CHECK: ArrayDecl[UnknownSize][(zext i32 %m to i64)][(zext i32 %o to i64)] with elements of 8 bytes.
-; CHECK: ArrayRef[{0,+,1}<%for.i>][{0,+,1}<%for.j>][{0,+,1}<%for.k>]
+; CHECK: ArrayRef[{0,+,1}<nuw><nsw><%for.i>][{0,+,1}<nuw><nsw><%for.j>][{0,+,1}<nuw><nsw><%for.k>]
 
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
diff --git a/llvm/test/Analysis/DivergenceAnalysis/AMDGPU/irreducible/irreducible-2.ll b/llvm/test/Analysis/DivergenceAnalysis/AMDGPU/irreducible/irreducible-2.ll
index e1e87dec7e0db..6b8e7a1a0bb54 100644
--- a/llvm/test/Analysis/DivergenceAnalysis/AMDGPU/irreducible/irreducible-2.ll
+++ b/llvm/test/Analysis/DivergenceAnalysis/AMDGPU/irreducible/irreducible-2.ll
@@ -569,6 +569,37 @@ X:
   ret void
 }
 
+define amdgpu_kernel void @always_uniform() {
+; CHECK-LABEL: UniformityInfo for function 'always_uniform':
+; CHECK: CYCLES ASSSUMED DIVERGENT:
+; CHECK:   depth=1: entries(bb2 bb3)
+
+bb:
+  %inst = tail call i32 @llvm.amdgcn.mbcnt.hi(i32 0, i32 0)
+  %inst1 = icmp ugt i32 %inst, 0
+  br i1 %inst1, label %bb3, label %bb2
+; CHECK:   DIVERGENT:   %inst = tail call i32 @llvm.amdgcn.mbcnt.hi(i32 0, i32 0)
+; CHECK:   DIVERGENT:   %inst1 = icmp ugt i32 %inst, 0
+; CHECK:   DIVERGENT:   br i1 %inst1, label %bb3, label %bb2
+
+bb2:                                              ; preds = %bb3, %bb
+  br label %bb3
+
+bb3:                                              ; preds = %bb2, %bb
+  %inst4 = tail call i64 @llvm.amdgcn.icmp.i64.i16(i16 0, i16 0, i32 0)
+  %inst5 = trunc i64 %inst4 to i32
+  %inst6 = and i32 0, %inst5
+  br label %bb2
+; CHECK-LABEL: BLOCK bb3
+; CHECK-NOT: DIVERGENT: {{.*}} call i64 @llvm.amdgcn.icmp.i64.i16
+; CHECK:   DIVERGENT:   %inst5 = trunc i64 %inst4 to i32
+; CHECK:   DIVERGENT:   %inst6 = and i32 0, %inst5
+}
+
+declare i32 @llvm.amdgcn.mbcnt.hi(i32, i32)
+
+declare i64 @llvm.amdgcn.icmp.i64.i16(i16, i16, i32 immarg)
+
 declare i32 @llvm.amdgcn.workitem.id.x() #0
 
 attributes #0 = { nounwind readnone }
diff --git a/llvm/test/Analysis/LoopAccessAnalysis/invalidation.ll b/llvm/test/Analysis/LoopAccessAnalysis/invalidation.ll
new file mode 100644
index 0000000000000..fb3af609dd2c6
--- /dev/null
+++ b/llvm/test/Analysis/LoopAccessAnalysis/invalidation.ll
@@ -0,0 +1,64 @@
+; Test that access-info gets invalidated when the analyses it depends on are
+; invalidated.
+
+; This is a reproducer for https://github.com/llvm/llvm-project/issues/61324.
+; We want to see that LoopAccessAnalysis+AAManger is being updated at the end,
+; instead of crashing when using a stale AAManager.
+;
+; RUN: opt < %s -disable-output -debug-pass-manager -passes='function(require<access-info>,invalidate<aa>),print<access-info>' 2>&1 | FileCheck %s --check-prefix=CHECK-AA
+;
+; CHECK-AA: Running pass: RequireAnalysisPass
+; CHECK-AA-NEXT: Running analysis: LoopAccessAnalysis on foo
+; CHECK-AA: Running pass: InvalidateAnalysisPass
+; CHECK-AA-NEXT: Invalidating analysis: AAManager on foo
+; CHECK-AA-NEXT: Invalidating analysis: LoopAccessAnalysis on foo
+; CHECK-AA-NEXT: Running pass: LoopAccessInfoPrinterPass on foo
+; CHECK-AA-NEXT: Running analysis: LoopAccessAnalysis on foo
+; CHECK-AA-NEXT: Running analysis: AAManager on foo
+
+
+; Verify that an explicit invalidate request for access-info result in an
+; invalidation.
+;
+; RUN: opt < %s -disable-output -debug-pass-manager -passes='function(require<access-info>,invalidate<access-info>)' 2>&1 | FileCheck %s --check-prefix=CHECK-INV-AA
+;
+; CHECK-INV-AA: Running pass: RequireAnalysisPass
+; CHECK-INV-AA-NEXT: Running analysis: LoopAccessAnalysis on foo
+; CHECK-INV-AA: Running pass: InvalidateAnalysisPass
+; CHECK-INV-AA-NEXT: Invalidating analysis: LoopAccessAnalysis on foo
+
+
+; Invalidation of scalar-evolution should transitively invalidate access-info.
+;
+; RUN: opt < %s -disable-output -debug-pass-manager -passes='function(require<access-info>,invalidate<scalar-evolution>)' 2>&1 | FileCheck %s --check-prefix=CHECK-SCEV
+;
+; CHECK-SCEV: Running pass: RequireAnalysisPass
+; CHECK-SCEV-NEXT: Running analysis: LoopAccessAnalysis on foo
+; CHECK-SCEV: Running pass: InvalidateAnalysisPass
+; CHECK-SCEV-NEXT: Invalidating analysis: ScalarEvolutionAnalysis on foo
+; CHECK-SCEV-NEXT: Invalidating analysis: LoopAccessAnalysis on foo
+
+
+; Invalidation of domtree should transitively invalidate access-info.
+;
+; RUN: opt < %s -disable-output -debug-pass-manager -passes='function(require<access-info>,invalidate<domtree>)' 2>&1 | FileCheck %s --check-prefix=CHECK-DT
+;
+; CHECK-DT: Running pass: RequireAnalysisPass
+; CHECK-DT-NEXT: Running analysis: LoopAccessAnalysis on foo
+; CHECK-DT: Running pass: InvalidateAnalysisPass
+; CHECK-DT-NEXT: Invalidating analysis: DominatorTreeAnalysis on foo
+; CHECK-DT: Invalidating analysis: LoopAccessAnalysis on foo
+
+
+define void @foo(ptr nocapture writeonly %a, ptr nocapture writeonly %b) memory(argmem: write) {
+entry:
+  br label %for.cond1
+
+for.cond1:
+  store i16 0, ptr %b, align 1
+  store i16 0, ptr %a, align 1
+  br i1 true, label %for.end6, label %for.cond1
+
+for.end6:
+  ret void
+}
diff --git a/llvm/test/Analysis/MemoryDependenceAnalysis/invalidation.ll b/llvm/test/Analysis/MemoryDependenceAnalysis/invalidation.ll
index 894e60d319f71..cf933cc109f55 100644
--- a/llvm/test/Analysis/MemoryDependenceAnalysis/invalidation.ll
+++ b/llvm/test/Analysis/MemoryDependenceAnalysis/invalidation.ll
@@ -1,16 +1,19 @@
 ; Test that memdep gets invalidated when the analyses it depends on are
 ; invalidated.
 ;
-; Check AA. AA is stateless, there's nothing to invalidate.
+; Check AA. AA is stateless, but given an explicit invalidate (abandon) the
+; AAManager is invalidated and we must invalidate memdep as well given
+; the transitive dependency.
 ; RUN: opt -disable-output -debug-pass-manager -aa-pipeline='basic-aa' %s 2>&1 \
 ; RUN:     -passes='require<memdep>,invalidate<aa>,gvn' \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-AA-INVALIDATE
 ; CHECK-AA-INVALIDATE: Running pass: RequireAnalysisPass
 ; CHECK-AA-INVALIDATE: Running analysis: MemoryDependenceAnalysis
 ; CHECK-AA-INVALIDATE: Running pass: InvalidateAnalysisPass
-; CHECK-NOT-AA-INVALIDATE: Invalidating analysis: MemoryDependenceAnalysis
-; CHECK-AA-INVALIDATE: Running pass: GVN
-; CHECK-NOT-AA-INVALIDATE: Running analysis: MemoryDependenceAnalysis
+; CHECK-AA-INVALIDATE: Invalidating analysis: AAManager
+; CHECK-AA-INVALIDATE: Invalidating analysis: MemoryDependenceAnalysis
+; CHECK-AA-INVALIDATE: Running pass: GVNPass
+; CHECK-AA-INVALIDATE: Running analysis: MemoryDependenceAnalysis
 ;
 ; Check domtree specifically.
 ; RUN: opt -disable-output -debug-pass-manager %s 2>&1 \
@@ -21,7 +24,7 @@
 ; CHECK-DT-INVALIDATE: Running pass: InvalidateAnalysisPass
 ; CHECK-DT-INVALIDATE: Invalidating analysis: DominatorTreeAnalysis
 ; CHECK-DT-INVALIDATE: Invalidating analysis: MemoryDependenceAnalysis
-; CHECK-DT-INVALIDATE: Running pass: GVN
+; CHECK-DT-INVALIDATE: Running pass: GVNPass
 ; CHECK-DT-INVALIDATE: Running analysis: MemoryDependenceAnalysis
 ;
 
diff --git a/llvm/test/Analysis/ScalarEvolution/addrec-computed-during-addrec-calculation.ll b/llvm/test/Analysis/ScalarEvolution/addrec-computed-during-addrec-calculation.ll
index b03133ac6bea7..56be4e0726f8e 100644
--- a/llvm/test/Analysis/ScalarEvolution/addrec-computed-during-addrec-calculation.ll
+++ b/llvm/test/Analysis/ScalarEvolution/addrec-computed-during-addrec-calculation.ll
@@ -12,19 +12,19 @@ define void @test(ptr %p) {
 ; CHECK-NEXT:    %iv = phi i32 [ 0, %entry ], [ %iv.next, %loop.latch ]
 ; CHECK-NEXT:    --> %iv U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop.header: Variant, %loop2: Invariant, %loop3: Invariant }
 ; CHECK-NEXT:    %iv2 = phi i32 [ %iv, %loop.header ], [ %iv2.next, %loop2 ]
-; CHECK-NEXT:    --> {%iv,+,1}<nsw><%loop2> U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop2: Computable, %loop.header: Variant }
+; CHECK-NEXT:    --> {%iv,+,1}<%loop2> U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop2: Computable, %loop.header: Variant }
 ; CHECK-NEXT:    %iv2.next = add i32 %iv2, 1
-; CHECK-NEXT:    --> {(1 + %iv),+,1}<nw><%loop2> U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop2: Computable, %loop.header: Variant }
+; CHECK-NEXT:    --> {(1 + %iv),+,1}<%loop2> U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop2: Computable, %loop.header: Variant }
 ; CHECK-NEXT:    %v = load i32, ptr %p, align 4
 ; CHECK-NEXT:    --> %v U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop2: Variant, %loop.header: Variant }
 ; CHECK-NEXT:    %iv2.ext = sext i32 %iv2 to i64
-; CHECK-NEXT:    --> (sext i32 {%iv,+,1}<nsw><%loop2> to i64) U: [-2147483648,2147483648) S: [-2147483648,2147483648) Exits: <<Unknown>> LoopDispositions: { %loop.header: Variant, %loop2: Computable, %loop3: Invariant }
+; CHECK-NEXT:    --> (sext i32 {%iv,+,1}<%loop2> to i64) U: [-2147483648,2147483648) S: [-2147483648,2147483648) Exits: <<Unknown>> LoopDispositions: { %loop.header: Variant, %loop2: Computable, %loop3: Invariant }
 ; CHECK-NEXT:    %iv3 = phi i64 [ %iv2.ext, %loop2.end ], [ %iv3.next, %loop3 ]
-; CHECK-NEXT:    --> {(sext i32 {%iv,+,1}<nsw><%loop2> to i64),+,1}<nsw><%loop3> U: [-2147483648,2147483648) S: [-2147483648,2147483648) Exits: (sext i32 {%iv,+,1}<nsw><%loop2> to i64) LoopDispositions: { %loop3: Computable, %loop.header: Variant }
+; CHECK-NEXT:    --> {(sext i32 {%iv,+,1}<%loop2> to i64),+,1}<nsw><%loop3> U: [-2147483648,2147483648) S: [-2147483648,2147483648) Exits: (sext i32 {%iv,+,1}<%loop2> to i64) LoopDispositions: { %loop3: Computable, %loop.header: Variant }
 ; CHECK-NEXT:    %iv3.next = add nsw i64 %iv3, 1
-; CHECK-NEXT:    --> {(1 + (sext i32 {%iv,+,1}<nsw><%loop2> to i64))<nsw>,+,1}<nsw><%loop3> U: [-2147483647,2147483649) S: [-2147483647,2147483649) Exits: (1 + (sext i32 {%iv,+,1}<nsw><%loop2> to i64))<nsw> LoopDispositions: { %loop3: Computable, %loop.header: Variant }
+; CHECK-NEXT:    --> {(1 + (sext i32 {%iv,+,1}<%loop2> to i64))<nsw>,+,1}<nsw><%loop3> U: [-2147483647,2147483649) S: [-2147483647,2147483649) Exits: (1 + (sext i32 {%iv,+,1}<%loop2> to i64))<nsw> LoopDispositions: { %loop3: Computable, %loop.header: Variant }
 ; CHECK-NEXT:    %iv.next = trunc i64 %iv3 to i32
-; CHECK-NEXT:    --> {{\{}}{%iv,+,1}<nsw><%loop2>,+,1}<%loop3> U: full-set S: full-set --> {%iv,+,1}<nsw><%loop2> U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop.header: Variant, %loop2: Variant, %loop3: Computable }
+; CHECK-NEXT:    --> {{{.*}}{%iv,+,1}<%loop2>,+,1}<%loop3> U: full-set S: full-set --> {%iv,+,1}<%loop2> U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop.header: Variant, %loop2: Variant, %loop3: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test
 ; CHECK-NEXT:  Loop %loop2: Unpredictable backedge-taken count.
 ; CHECK-NEXT:  Loop %loop2: constant max backedge-taken count is -1
diff --git a/llvm/test/Analysis/ScalarEvolution/becount-invalidation.ll b/llvm/test/Analysis/ScalarEvolution/becount-invalidation.ll
index 31a21099d440e..418f85613ef58 100644
--- a/llvm/test/Analysis/ScalarEvolution/becount-invalidation.ll
+++ b/llvm/test/Analysis/ScalarEvolution/becount-invalidation.ll
@@ -15,7 +15,7 @@ define void @test(ptr %arg) {
 ; CHECK-NEXT:    %ptr1.next = phi ptr [ %ptr2, %loop.header ], [ %ptr1.next.next, %loop2.latch ]
 ; CHECK-NEXT:    --> {%ptr2,+,8}<nuw><%loop2.header> U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop2.header: Computable, %loop.header: Variant }
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %loop.header ], [ %iv.next, %loop2.latch ]
-; CHECK-NEXT:    --> {0,+,1}<%loop2.header> U: [0,1) S: [0,1) Exits: <<Unknown>> LoopDispositions: { %loop2.header: Computable, %loop.header: Variant }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop2.header> U: [0,1) S: [0,1) Exits: <<Unknown>> LoopDispositions: { %loop2.header: Computable, %loop.header: Variant }
 ; CHECK-NEXT:    %ptr1.dummy = getelementptr inbounds i64, ptr %ptr1.next, i64 0
 ; CHECK-NEXT:    --> {%ptr2,+,8}<nuw><%loop2.header> U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop2.header: Computable, %loop.header: Variant }
 ; CHECK-NEXT:    %val = load i64, ptr %ptr1.dummy, align 8
@@ -23,7 +23,7 @@ define void @test(ptr %arg) {
 ; CHECK-NEXT:    %ptr1.next.next = getelementptr inbounds i64, ptr %ptr1.next, i64 1
 ; CHECK-NEXT:    --> {(8 + %ptr2),+,8}<nw><%loop2.header> U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop2.header: Computable, %loop.header: Variant }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop2.header> U: [1,2) S: [1,2) Exits: <<Unknown>> LoopDispositions: { %loop2.header: Computable, %loop.header: Variant }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop2.header> U: [1,2) S: [1,2) Exits: <<Unknown>> LoopDispositions: { %loop2.header: Computable, %loop.header: Variant }
 ; CHECK-NEXT:    %ptr2.next = phi ptr [ %ptr1, %if ], [ %arg, %else ]
 ; CHECK-NEXT:    --> %ptr2.next U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop.header: Variant, %loop2.header: Invariant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test
diff --git a/llvm/test/Analysis/ScalarEvolution/different-loops-recs.ll b/llvm/test/Analysis/ScalarEvolution/different-loops-recs.ll
index 5e996c8f756c1..44081f32d0af6 100644
--- a/llvm/test/Analysis/ScalarEvolution/different-loops-recs.ll
+++ b/llvm/test/Analysis/ScalarEvolution/different-loops-recs.ll
@@ -18,17 +18,17 @@ define void @test_00() {
 ; CHECK:       %sum4 = add i32 %sum3, %phi6
 ; CHECK-NEXT:  -->  {159,+,6}<%loop2>
 ; CHECK:       %s1 = add i32 %phi1, %phi4
-; CHECK-NEXT:  -->  {{{{}}73,+,1}<%loop1>,+,1}<%loop2>
+; CHECK-NEXT:  -->  {{{{}}73,+,1}<nuw><nsw><%loop1>,+,1}<nw><%loop2>
 ; CHECK:       %s2 = add i32 %phi5, %phi2
-; CHECK-NEXT:  -->  {{{{}}57,+,2}<%loop1>,+,2}<%loop2>
+; CHECK-NEXT:  -->  {{{{}}57,+,2}<nuw><nsw><%loop1>,+,2}<nw><%loop2>
 ; CHECK:       %s3 = add i32 %sum1, %sum3
 ; CHECK-NEXT:  -->  {{{{}}130,+,3}<%loop1>,+,3}<%loop2>
 ; CHECK:       %s4 = add i32 %sum4, %sum2
 ; CHECK-NEXT:  -->  {{{{}}179,+,6}<%loop1>,+,6}<%loop2>
 ; CHECK:       %s5 = add i32 %phi3, %sum3
-; CHECK-NEXT:  -->  {{{{}}122,+,3}<%loop1>,+,3}<%loop2>
+; CHECK-NEXT:  -->  {{{{}}122,+,3}<nuw><nsw><%loop1>,+,3}<%loop2>
 ; CHECK:       %s6 = add i32 %sum2, %phi6
-; CHECK-NEXT:  -->  {{{{}}63,+,6}<%loop1>,+,3}<%loop2>
+; CHECK-NEXT:  -->  {{{{}}63,+,6}<%loop1>,+,3}<nw><%loop2>
 
 entry:
   br label %loop1
@@ -92,7 +92,7 @@ define void @test_01(i32 %a, i32 %b) {
 ; CHECK:       %s2 = add i32 %is2, %phi4
 ; CHECK-NEXT:  -->  {(222 + %b),+,7}<%loop2>
 ; CHECK:       %s3 = add i32 %is1, %phi5
-; CHECK-NEXT:  -->  {{{{}}(59 + (2 * %a) + %b),+,6}<%loop1>,+,2}<%loop2>
+; CHECK-NEXT:  -->  {{{{}}(59 + (2 * %a) + %b),+,6}<%loop1>,+,2}<nw><%loop2>
 ; CHECK:       %s4 = add i32 %phi2, %is2
 ; CHECK-NEXT:  -->  {{{{}}(159 + (2 * %b)),+,2}<%loop1>,+,6}<%loop2>
 ; CHECK:       %s5 = add i32 %is1, %is2
@@ -357,17 +357,17 @@ define void @test_06() {
 
 ; CHECK-LABEL: Classifying expressions for: @test_06
 ; CHECK:       %s1 = add i32 %phi1, %phi2
-; CHECK-NEXT:  -->  {{{{}}30,+,1}<%loop1>,+,2}<%loop2>
+; CHECK-NEXT:  -->  {{{{}}30,+,1}<nuw><nsw><%loop1>,+,2}<nw><%loop2>
 ; CHECK:       %s2 = add i32 %phi2, %phi1
-; CHECK-NEXT:  -->  {{{{}}30,+,1}<%loop1>,+,2}<%loop2>
+; CHECK-NEXT:  -->  {{{{}}30,+,1}<nuw><nsw><%loop1>,+,2}<nw><%loop2>
 ; CHECK:       %s3 = add i32 %phi1, %phi3
-; CHECK-NEXT:  -->  {{{{}}40,+,1}<%loop1>,+,3}<%loop3>
+; CHECK-NEXT:  -->  {{{{}}40,+,1}<nuw><nsw><%loop1>,+,3}<nw><%loop3>
 ; CHECK:       %s4 = add i32 %phi3, %phi1
-; CHECK-NEXT:  -->  {{{{}}40,+,1}<%loop1>,+,3}<%loop3>
+; CHECK-NEXT:  -->  {{{{}}40,+,1}<nuw><nsw><%loop1>,+,3}<nw><%loop3>
 ; CHECK:       %s5 = add i32 %phi2, %phi3
-; CHECK-NEXT:  -->  {{{{}}50,+,2}<%loop2>,+,3}<%loop3>
+; CHECK-NEXT:  -->  {{{{}}50,+,2}<nuw><nsw><%loop2>,+,3}<nw><%loop3>
 ; CHECK:       %s6 = add i32 %phi3, %phi2
-; CHECK-NEXT:  -->  {{{{}}50,+,2}<%loop2>,+,3}<%loop3>
+; CHECK-NEXT:  -->  {{{{}}50,+,2}<nuw><nsw><%loop2>,+,3}<nw><%loop3>
 
 entry:
   br label %loop1
@@ -409,17 +409,17 @@ define void @test_07() {
 
 ; CHECK-LABEL: Classifying expressions for: @test_07
 ; CHECK:       %s1 = add i32 %phi1, %phi2
-; CHECK-NEXT:  -->  {{{{}}30,+,1}<%loop1>,+,2}<%loop2>
+; CHECK-NEXT:  -->  {{{{}}30,+,1}<nuw><nsw><%loop1>,+,2}<nw><%loop2>
 ; CHECK:       %s2 = add i32 %phi2, %phi1
-; CHECK-NEXT:  -->  {{{{}}30,+,1}<%loop1>,+,2}<%loop2>
+; CHECK-NEXT:  -->  {{{{}}30,+,1}<nuw><nsw><%loop1>,+,2}<nw><%loop2>
 ; CHECK:       %s3 = add i32 %phi1, %phi3
-; CHECK-NEXT:  -->  {{{{}}40,+,3}<%loop3>,+,1}<%loop1>
+; CHECK-NEXT:  -->  {{{{}}40,+,3}<nuw><nsw><%loop3>,+,1}<nw><%loop1>
 ; CHECK:       %s4 = add i32 %phi3, %phi1
-; CHECK-NEXT:  -->  {{{{}}40,+,3}<%loop3>,+,1}<%loop1>
+; CHECK-NEXT:  -->  {{{{}}40,+,3}<nuw><nsw><%loop3>,+,1}<nw><%loop1>
 ; CHECK:       %s5 = add i32 %phi2, %phi3
-; CHECK-NEXT:  -->  {{{{}}50,+,3}<%loop3>,+,2}<%loop2>
+; CHECK-NEXT:  -->  {{{{}}50,+,3}<nuw><nsw><%loop3>,+,2}<nw><%loop2>
 ; CHECK:       %s6 = add i32 %phi3, %phi2
-; CHECK-NEXT:  -->  {{{{}}50,+,3}<%loop3>,+,2}<%loop2>
+; CHECK-NEXT:  -->  {{{{}}50,+,3}<nuw><nsw><%loop3>,+,2}<nw><%loop2>
 
 entry:
   br label %loop3
@@ -461,7 +461,7 @@ define void @test_08() {
 
 ; CHECK-LABEL: Classifying expressions for: @test_08
 ; CHECK:       %tmp11 = add i64 %iv.2.2, %iv.2.1
-; CHECK-NEXT:  -->  ({0,+,-1}<nsw><%loop_2> + %iv.2.1)
+; CHECK-NEXT:  -->  ({0,+,-1}<nuw><nsw><%loop_2> + %iv.2.1)
 ; CHECK:       %tmp12 = trunc i64 %tmp11 to i32
 ; CHECK-NEXT:  -->  ((trunc i64 %iv.2.1 to i32) + {0,+,-1}<%loop_2>)
 ; CHECK:       %tmp14 = mul i32 %tmp12, %tmp7
@@ -569,7 +569,7 @@ define i64 @test_10(i32 %param) {
 
 ; CHECK-LABEL: Classifying expressions for: @test_10
 ; CHECK:       %uncle = phi i64 [ %uncle.outer.next, %uncle.loop.backedge ], [ 0, %outer.loop ]
-; CHECK-NEXT:  -->  {0,+,1}<%uncle.loop>
+; CHECK-NEXT:  -->  {0,+,1}<nuw><nsw><%uncle.loop>
 ; CHECK:       %iv1 = phi i64 [ %iv1.next, %guarded ], [ 0, %uncle.loop ]
 ; CHECK-NEXT:  -->  {0,+,1}<nuw><nsw><%loop1>
 ; CHECK:       %iv1.trunc = trunc i64 %iv1 to i32
@@ -577,7 +577,7 @@ define i64 @test_10(i32 %param) {
 ; CHECK:       %iv1.next = add nuw nsw i64 %iv1, 1
 ; CHECK-NEXT:  -->  {1,+,1}<nuw><nsw><%loop1>
 ; CHECK:       %uncle.outer.next = add i64 %uncle, 1
-; CHECK-NEXT:  -->  {1,+,1}<%uncle.loop>
+; CHECK-NEXT:  -->  {1,+,1}<nuw><nsw><%uncle.loop>
 ; CHECK:       %iv2 = phi i32 [ %iv2.next, %loop2 ], [ %param, %loop2.preheader ]
 ; CHECK-NEXT:  -->  {%param,+,1}<%loop2>
 ; CHECK:       %iv2.next = add i32 %iv2, 1
diff --git a/llvm/test/Analysis/ScalarEvolution/exit-count-select-safe.ll b/llvm/test/Analysis/ScalarEvolution/exit-count-select-safe.ll
index ad89bb5e3a877..cf52c1b6cf8a2 100644
--- a/llvm/test/Analysis/ScalarEvolution/exit-count-select-safe.ll
+++ b/llvm/test/Analysis/ScalarEvolution/exit-count-select-safe.ll
@@ -426,9 +426,9 @@ define i32 @logical_and_2ops_and_constant(i32 %n, i32 %m, i32 %k) {
 ; CHECK-LABEL: 'logical_and_2ops_and_constant'
 ; CHECK-NEXT:  Classifying expressions for: @logical_and_2ops_and_constant
 ; CHECK-NEXT:    %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,43) S: [0,43) Exits: (42 umin %n) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,43) S: [0,43) Exits: (42 umin %n) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %i.next = add i32 %i, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,44) S: [1,44) Exits: (1 + (42 umin %n))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,44) S: [1,44) Exits: (1 + (42 umin %n))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %umin = call i32 @llvm.umin.i32(i32 %n, i32 42)
 ; CHECK-NEXT:    --> (42 umin %n) U: [0,43) S: [0,43) Exits: (42 umin %n) LoopDispositions: { %loop: Invariant }
 ; CHECK-NEXT:    %cond = select i1 %cond_p1, i1 true, i1 %cond_p0
@@ -469,9 +469,9 @@ define i32 @computeSCEVAtScope(i32 %d.0) {
 ; CHECK-NEXT:    %inc3 = add nsw i32 %e.1, 1
 ; CHECK-NEXT:    --> {(1 + %d.0),+,1}<nw><%for.cond> U: full-set S: full-set Exits: 1 LoopDispositions: { %for.cond: Computable, %while.cond: Variant }
 ; CHECK-NEXT:    %f.1 = phi i32 [ %inc8, %for.body5 ], [ 0, %for.cond4.preheader ]
-; CHECK-NEXT:    --> {0,+,1}<%for.cond4> U: [0,1) S: [0,1) Exits: 0 LoopDispositions: { %for.cond4: Computable, %while.cond: Variant }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%for.cond4> U: [0,1) S: [0,1) Exits: 0 LoopDispositions: { %for.cond4: Computable, %while.cond: Variant }
 ; CHECK-NEXT:    %inc8 = add i32 %f.1, 1
-; CHECK-NEXT:    --> {1,+,1}<%for.cond4> U: [1,2) S: [1,2) Exits: 1 LoopDispositions: { %for.cond4: Computable, %while.cond: Variant }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%for.cond4> U: [1,2) S: [1,2) Exits: 1 LoopDispositions: { %for.cond4: Computable, %while.cond: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @computeSCEVAtScope
 ; CHECK-NEXT:  Loop %for.cond: backedge-taken count is (-1 * %d.0)
 ; CHECK-NEXT:  Loop %for.cond: constant max backedge-taken count is -1
@@ -983,9 +983,9 @@ define i32 @logical_and_not_zero(i16 %n, i32 %m) {
 ; CHECK-NEXT:    %n1 = add i32 %n.ext, 1
 ; CHECK-NEXT:    --> (1 + (zext i16 %n to i32))<nuw><nsw> U: [1,65537) S: [1,65537)
 ; CHECK-NEXT:    %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,65537) S: [0,65537) Exits: ((1 + (zext i16 %n to i32))<nuw><nsw> umin %m) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,65537) S: [0,65537) Exits: ((1 + (zext i16 %n to i32))<nuw><nsw> umin %m) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %i.next = add i32 %i, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,65538) S: [1,65538) Exits: (1 + ((1 + (zext i16 %n to i32))<nuw><nsw> umin %m))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,65538) S: [1,65538) Exits: (1 + ((1 + (zext i16 %n to i32))<nuw><nsw> umin %m))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %cond = select i1 %cond_p0, i1 %cond_p1, i1 false
 ; CHECK-NEXT:    --> (%cond_p0 umin_seq %cond_p1) U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @logical_and_not_zero
@@ -1019,9 +1019,9 @@ define i32 @logical_and_not_zero_wrong_order(i16 %n, i32 %m) {
 ; CHECK-NEXT:    %n1 = add i32 %n.ext, 1
 ; CHECK-NEXT:    --> (1 + (zext i16 %n to i32))<nuw><nsw> U: [1,65537) S: [1,65537)
 ; CHECK-NEXT:    %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,65537) S: [0,65537) Exits: (%m umin_seq (1 + (zext i16 %n to i32))<nuw><nsw>) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,65537) S: [0,65537) Exits: (%m umin_seq (1 + (zext i16 %n to i32))<nuw><nsw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %i.next = add i32 %i, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,65538) S: [1,65538) Exits: (1 + (%m umin_seq (1 + (zext i16 %n to i32))<nuw><nsw>))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,65538) S: [1,65538) Exits: (1 + (%m umin_seq (1 + (zext i16 %n to i32))<nuw><nsw>))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %cond = select i1 %cond_p0, i1 %cond_p1, i1 false
 ; CHECK-NEXT:    --> (%cond_p0 umin_seq %cond_p1) U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @logical_and_not_zero_wrong_order
@@ -1090,9 +1090,9 @@ define i32 @logical_and_known_smaller(i16 %n, i16 %m) {
 ; CHECK-NEXT:    %m.add = add i32 %m.ext, 65536
 ; CHECK-NEXT:    --> (65536 + (zext i16 %m to i32))<nuw><nsw> U: [65536,131072) S: [65536,131072)
 ; CHECK-NEXT:    %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,65536) S: [0,65536) Exits: (zext i16 %n to i32) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,65536) S: [0,65536) Exits: (zext i16 %n to i32) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %i.next = add i32 %i, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,65537) S: [1,65537) Exits: (1 + (zext i16 %n to i32))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,65537) S: [1,65537) Exits: (1 + (zext i16 %n to i32))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %cond = select i1 %cond_p0, i1 %cond_p1, i1 false
 ; CHECK-NEXT:    --> (%cond_p0 umin_seq %cond_p1) U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @logical_and_known_smaller
@@ -1129,9 +1129,9 @@ define i32 @logical_and_known_smaller_equal(i16 %n, i16 %m) {
 ; CHECK-NEXT:    %m.add = add i32 %m.ext, 65535
 ; CHECK-NEXT:    --> (65535 + (zext i16 %m to i32))<nuw><nsw> U: [65535,131071) S: [65535,131071)
 ; CHECK-NEXT:    %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,65536) S: [0,65536) Exits: (zext i16 %n to i32) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,65536) S: [0,65536) Exits: (zext i16 %n to i32) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %i.next = add i32 %i, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,65537) S: [1,65537) Exits: (1 + (zext i16 %n to i32))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,65537) S: [1,65537) Exits: (1 + (zext i16 %n to i32))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %cond = select i1 %cond_p0, i1 %cond_p1, i1 false
 ; CHECK-NEXT:    --> (%cond_p0 umin_seq %cond_p1) U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @logical_and_known_smaller_equal
@@ -1168,9 +1168,9 @@ define i32 @logical_and_not_known_smaller_equal(i16 %n, i16 %m) {
 ; CHECK-NEXT:    %m.add = add i32 %m.ext, 65534
 ; CHECK-NEXT:    --> (65534 + (zext i16 %m to i32))<nuw><nsw> U: [65534,131070) S: [65534,131070)
 ; CHECK-NEXT:    %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,65536) S: [0,65536) Exits: ((zext i16 %n to i32) umin_seq (65534 + (zext i16 %m to i32))<nuw><nsw>) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,65536) S: [0,65536) Exits: ((zext i16 %n to i32) umin_seq (65534 + (zext i16 %m to i32))<nuw><nsw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %i.next = add i32 %i, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,65537) S: [1,65537) Exits: (1 + ((zext i16 %n to i32) umin_seq (65534 + (zext i16 %m to i32))<nuw><nsw>))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,65537) S: [1,65537) Exits: (1 + ((zext i16 %n to i32) umin_seq (65534 + (zext i16 %m to i32))<nuw><nsw>))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %cond = select i1 %cond_p0, i1 %cond_p1, i1 false
 ; CHECK-NEXT:    --> (%cond_p0 umin_seq %cond_p1) U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @logical_and_not_known_smaller_equal
@@ -1207,9 +1207,9 @@ define i32 @logical_and_known_greater(i16 %n, i16 %m) {
 ; CHECK-NEXT:    %m.add = add i32 %m.ext, 65536
 ; CHECK-NEXT:    --> (65536 + (zext i16 %m to i32))<nuw><nsw> U: [65536,131072) S: [65536,131072)
 ; CHECK-NEXT:    %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,65536) S: [0,65536) Exits: (zext i16 %n to i32) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,65536) S: [0,65536) Exits: (zext i16 %n to i32) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %i.next = add i32 %i, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,65537) S: [1,65537) Exits: (1 + (zext i16 %n to i32))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,65537) S: [1,65537) Exits: (1 + (zext i16 %n to i32))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %cond = select i1 %cond_p0, i1 %cond_p1, i1 false
 ; CHECK-NEXT:    --> (%cond_p0 umin_seq %cond_p1) U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @logical_and_known_greater
@@ -1246,9 +1246,9 @@ define i32 @logical_and_known_greater_equal(i16 %n, i16 %m) {
 ; CHECK-NEXT:    %m.add = add i32 %m.ext, 65535
 ; CHECK-NEXT:    --> (65535 + (zext i16 %m to i32))<nuw><nsw> U: [65535,131071) S: [65535,131071)
 ; CHECK-NEXT:    %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,65536) S: [0,65536) Exits: (zext i16 %n to i32) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,65536) S: [0,65536) Exits: (zext i16 %n to i32) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %i.next = add i32 %i, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,65537) S: [1,65537) Exits: (1 + (zext i16 %n to i32))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,65537) S: [1,65537) Exits: (1 + (zext i16 %n to i32))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %cond = select i1 %cond_p0, i1 %cond_p1, i1 false
 ; CHECK-NEXT:    --> (%cond_p0 umin_seq %cond_p1) U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @logical_and_known_greater_equal
@@ -1285,9 +1285,9 @@ define i32 @logical_and_not_known_greater_equal(i16 %n, i16 %m) {
 ; CHECK-NEXT:    %m.add = add i32 %m.ext, 65534
 ; CHECK-NEXT:    --> (65534 + (zext i16 %m to i32))<nuw><nsw> U: [65534,131070) S: [65534,131070)
 ; CHECK-NEXT:    %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,65536) S: [0,65536) Exits: ((zext i16 %n to i32) umin (65534 + (zext i16 %m to i32))<nuw><nsw>) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,65536) S: [0,65536) Exits: ((zext i16 %n to i32) umin (65534 + (zext i16 %m to i32))<nuw><nsw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %i.next = add i32 %i, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,65537) S: [1,65537) Exits: (1 + ((zext i16 %n to i32) umin (65534 + (zext i16 %m to i32))<nuw><nsw>))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,65537) S: [1,65537) Exits: (1 + ((zext i16 %n to i32) umin (65534 + (zext i16 %m to i32))<nuw><nsw>))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %cond = select i1 %cond_p0, i1 %cond_p1, i1 false
 ; CHECK-NEXT:    --> (%cond_p0 umin_seq %cond_p1) U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @logical_and_not_known_greater_equal
@@ -1318,9 +1318,9 @@ define i32 @logical_and_zero_arg1(i32 %n) {
 ; CHECK-LABEL: 'logical_and_zero_arg1'
 ; CHECK-NEXT:  Classifying expressions for: @logical_and_zero_arg1
 ; CHECK-NEXT:    %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,1) S: [0,1) Exits: 0 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,1) S: [0,1) Exits: 0 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %i.next = add i32 %i, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,2) S: [1,2) Exits: 1 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,2) S: [1,2) Exits: 1 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %cond = select i1 %cond_p0, i1 %cond_p1, i1 false
 ; CHECK-NEXT:    --> (%cond_p0 umin_seq %cond_p1) U: full-set S: full-set Exits: false LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @logical_and_zero_arg1
@@ -1348,9 +1348,9 @@ define i32 @logical_and_zero_arg2(i32 %n) {
 ; CHECK-LABEL: 'logical_and_zero_arg2'
 ; CHECK-NEXT:  Classifying expressions for: @logical_and_zero_arg2
 ; CHECK-NEXT:    %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,1) S: [0,1) Exits: 0 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,1) S: [0,1) Exits: 0 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %i.next = add i32 %i, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,2) S: [1,2) Exits: 1 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,2) S: [1,2) Exits: 1 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %cond = select i1 %cond_p0, i1 %cond_p1, i1 false
 ; CHECK-NEXT:    --> (%cond_p1 umin %cond_p0) U: full-set S: full-set Exits: false LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @logical_and_zero_arg2
diff --git a/llvm/test/Analysis/ScalarEvolution/exit-count-select.ll b/llvm/test/Analysis/ScalarEvolution/exit-count-select.ll
index 6fbd7f2dfde0a..f8a90ddaf08ee 100644
--- a/llvm/test/Analysis/ScalarEvolution/exit-count-select.ll
+++ b/llvm/test/Analysis/ScalarEvolution/exit-count-select.ll
@@ -7,9 +7,9 @@ define void @logical_and_m_const(i32 %n) {
 ; CHECK-LABEL: 'logical_and_m_const'
 ; CHECK-NEXT:  Classifying expressions for: @logical_and_m_const
 ; CHECK-NEXT:    %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,3) S: [0,3) Exits: (2 umin %n) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,3) S: [0,3) Exits: (2 umin %n) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %i.next = add i32 %i, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,4) S: [1,4) Exits: (1 + (2 umin %n))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,4) S: [1,4) Exits: (1 + (2 umin %n))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %cond = select i1 %cond_i, i1 %cond_i2, i1 false
 ; CHECK-NEXT:    --> (%cond_i2 umin %cond_i) U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @logical_and_m_const
@@ -39,9 +39,9 @@ define void @logical_and_nonzero(i32 %m) {
 ; CHECK-LABEL: 'logical_and_nonzero'
 ; CHECK-NEXT:  Classifying expressions for: @logical_and_nonzero
 ; CHECK-NEXT:    %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,3) S: [0,3) Exits: (2 umin %m) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,3) S: [0,3) Exits: (2 umin %m) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %i.next = add i32 %i, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,4) S: [1,4) Exits: (1 + (2 umin %m))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,4) S: [1,4) Exits: (1 + (2 umin %m))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %cond = select i1 %cond_i, i1 %cond_i2, i1 false
 ; CHECK-NEXT:    --> (%cond_i umin_seq %cond_i2) U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @logical_and_nonzero
@@ -72,9 +72,9 @@ define void @logical_and_zero(i32 %m) {
 ; CHECK-LABEL: 'logical_and_zero'
 ; CHECK-NEXT:  Classifying expressions for: @logical_and_zero
 ; CHECK-NEXT:    %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,1) S: [0,1) Exits: 0 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,1) S: [0,1) Exits: 0 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %i.next = add i32 %i, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,2) S: [1,2) Exits: 1 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,2) S: [1,2) Exits: 1 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %cond = select i1 %cond_i, i1 %cond_i2, i1 false
 ; CHECK-NEXT:    --> (%cond_i umin_seq %cond_i2) U: full-set S: full-set Exits: false LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @logical_and_zero
@@ -136,9 +136,9 @@ define void @logical_or_m_const(i32 %n) {
 ; CHECK-LABEL: 'logical_or_m_const'
 ; CHECK-NEXT:  Classifying expressions for: @logical_or_m_const
 ; CHECK-NEXT:    %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,3) S: [0,3) Exits: (2 umin %n) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,3) S: [0,3) Exits: (2 umin %n) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %i.next = add i32 %i, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,4) S: [1,4) Exits: (1 + (2 umin %n))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,4) S: [1,4) Exits: (1 + (2 umin %n))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %cond = select i1 %cond_i, i1 true, i1 %cond_i2
 ; CHECK-NEXT:    --> (true + ((true + %cond_i) umin (true + %cond_i2))) U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @logical_or_m_const
@@ -168,9 +168,9 @@ define void @logical_or_nonzero(i32 %m) {
 ; CHECK-LABEL: 'logical_or_nonzero'
 ; CHECK-NEXT:  Classifying expressions for: @logical_or_nonzero
 ; CHECK-NEXT:    %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,3) S: [0,3) Exits: (2 umin %m) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,3) S: [0,3) Exits: (2 umin %m) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %i.next = add i32 %i, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,4) S: [1,4) Exits: (1 + (2 umin %m))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,4) S: [1,4) Exits: (1 + (2 umin %m))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %cond = select i1 %cond_i, i1 true, i1 %cond_i2
 ; CHECK-NEXT:    --> (true + ((true + %cond_i) umin_seq (true + %cond_i2))) U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @logical_or_nonzero
@@ -201,9 +201,9 @@ define void @logical_or_zero(i32 %m) {
 ; CHECK-LABEL: 'logical_or_zero'
 ; CHECK-NEXT:  Classifying expressions for: @logical_or_zero
 ; CHECK-NEXT:    %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,1) S: [0,1) Exits: 0 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,1) S: [0,1) Exits: 0 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %i.next = add i32 %i, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,2) S: [1,2) Exits: 1 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,2) S: [1,2) Exits: 1 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %cond = select i1 %cond_i, i1 true, i1 %cond_i2
 ; CHECK-NEXT:    --> (true + ((true + %cond_i) umin_seq (true + %cond_i2))) U: full-set S: full-set Exits: true LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @logical_or_zero
diff --git a/llvm/test/Analysis/ScalarEvolution/finite-trip-count.ll b/llvm/test/Analysis/ScalarEvolution/finite-trip-count.ll
index 9eefbd7e2dc45..502201bd657f8 100644
--- a/llvm/test/Analysis/ScalarEvolution/finite-trip-count.ll
+++ b/llvm/test/Analysis/ScalarEvolution/finite-trip-count.ll
@@ -9,10 +9,10 @@ declare void @non_exit_use(i32 %i) #0
 define void @SLE(i32 %len) willreturn {
 ; CHECK-LABEL: 'SLE'
 ; CHECK-NEXT:  Determining loop execution counts for: @SLE
-; CHECK-NEXT:  Loop %for.body: backedge-taken count is (0 smax (1 + %len)<nsw>)
+; CHECK-NEXT:  Loop %for.body: backedge-taken count is (0 smax (1 + %len))
 ; CHECK-NEXT:  Loop %for.body: constant max backedge-taken count is 2147483647
-; CHECK-NEXT:  Loop %for.body: symbolic max backedge-taken count is (0 smax (1 + %len)<nsw>)
-; CHECK-NEXT:  Loop %for.body: Predicated backedge-taken count is (0 smax (1 + %len)<nsw>)
+; CHECK-NEXT:  Loop %for.body: symbolic max backedge-taken count is (0 smax (1 + %len))
+; CHECK-NEXT:  Loop %for.body: Predicated backedge-taken count is (0 smax (1 + %len))
 ; CHECK-NEXT:   Predicates:
 ; CHECK:       Loop %for.body: Trip multiple is 1
 ;
@@ -55,10 +55,10 @@ for.end:
 define void @ULE(i32 %len) willreturn {
 ; CHECK-LABEL: 'ULE'
 ; CHECK-NEXT:  Determining loop execution counts for: @ULE
-; CHECK-NEXT:  Loop %for.body: backedge-taken count is (1 + %len)<nuw>
+; CHECK-NEXT:  Loop %for.body: backedge-taken count is (1 + %len)
 ; CHECK-NEXT:  Loop %for.body: constant max backedge-taken count is -1
-; CHECK-NEXT:  Loop %for.body: symbolic max backedge-taken count is (1 + %len)<nuw>
-; CHECK-NEXT:  Loop %for.body: Predicated backedge-taken count is (1 + %len)<nuw>
+; CHECK-NEXT:  Loop %for.body: symbolic max backedge-taken count is (1 + %len)
+; CHECK-NEXT:  Loop %for.body: Predicated backedge-taken count is (1 + %len)
 ; CHECK-NEXT:   Predicates:
 ; CHECK:       Loop %for.body: Trip multiple is 1
 ;
@@ -101,10 +101,10 @@ for.end:
 define void @SGE(i32 %end) willreturn {
 ; CHECK-LABEL: 'SGE'
 ; CHECK-NEXT:  Determining loop execution counts for: @SGE
-; CHECK-NEXT:  Loop %for.body: backedge-taken count is (100 + (-1 * (100 smin (-1 + %end)<nsw>)))
+; CHECK-NEXT:  Loop %for.body: backedge-taken count is (100 + (-1 * (100 smin (-1 + %end))))
 ; CHECK-NEXT:  Loop %for.body: constant max backedge-taken count is -2147483548
-; CHECK-NEXT:  Loop %for.body: symbolic max backedge-taken count is (100 + (-1 * (100 smin (-1 + %end)<nsw>)))
-; CHECK-NEXT:  Loop %for.body: Predicated backedge-taken count is (100 + (-1 * (100 smin (-1 + %end)<nsw>)))
+; CHECK-NEXT:  Loop %for.body: symbolic max backedge-taken count is (100 + (-1 * (100 smin (-1 + %end))))
+; CHECK-NEXT:  Loop %for.body: Predicated backedge-taken count is (100 + (-1 * (100 smin (-1 + %end))))
 ; CHECK-NEXT:   Predicates:
 ; CHECK:       Loop %for.body: Trip multiple is 1
 ;
@@ -189,3 +189,27 @@ for.body:
 for.end:
   ret void
 }
+
+define void @pr54191(i64 %n) mustprogress {
+; CHECK-LABEL: 'pr54191'
+; CHECK-NEXT:  Determining loop execution counts for: @pr54191
+; CHECK-NEXT:  Loop %loop: backedge-taken count is ((-2 + %n) /u 2)
+; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 4611686018427387902
+; CHECK-NEXT:  Loop %loop: symbolic max backedge-taken count is ((-2 + %n) /u 2)
+; CHECK-NEXT:  Loop %loop: Predicated backedge-taken count is ((-2 + %n) /u 2)
+; CHECK-NEXT:   Predicates:
+; CHECK:       Loop %loop: Trip multiple is 1
+;
+entry:
+  %guard = icmp sgt i64 %n, 1
+  br i1 %guard, label %loop, label %exit
+
+loop:
+  %iv = phi i64 [ 2, %entry ], [ %iv.next, %loop ]
+  %iv.next = add nuw nsw i64 %iv, 2
+  %cmp = icmp sle i64 %iv.next, %n
+  br i1 %cmp, label %loop, label %exit
+
+exit:
+  ret void
+}
diff --git a/llvm/test/Analysis/ScalarEvolution/flags-from-poison.ll b/llvm/test/Analysis/ScalarEvolution/flags-from-poison.ll
index c07f4c08d004b..c52eb940f1490 100644
--- a/llvm/test/Analysis/ScalarEvolution/flags-from-poison.ll
+++ b/llvm/test/Analysis/ScalarEvolution/flags-from-poison.ll
@@ -1130,7 +1130,7 @@ define void @test-shl-nsw(ptr %input, i32 %start, i32 %numIterations) {
 ; CHECK-NEXT:    %ptr = getelementptr inbounds float, ptr %input, i64 %index64
 ; CHECK-NEXT:    --> {((4 * (sext i32 (256 * %start) to i64))<nsw> + %input),+,1024}<nw><%loop> U: full-set S: full-set Exits: ((4 * (sext i32 (256 * %start) to i64))<nsw> + (1024 * (zext i32 (-1 + (-1 * %start) + %numIterations) to i64))<nuw><nsw> + %input) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %nexti = add nsw i32 %i, 1
-; CHECK-NEXT:    --> {(1 + %start)<nsw>,+,1}<nsw><%loop> U: full-set S: full-set Exits: %numIterations LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(1 + %start)<nsw>,+,1}<nsw><%loop> U: [-2147483647,-2147483648) S: [-2147483647,-2147483648) Exits: %numIterations LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test-shl-nsw
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (-1 + (-1 * %start) + %numIterations)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is -1
@@ -1170,7 +1170,7 @@ define void @test-shl-nuw-edgecase(ptr %input, i32 %start, i32 %numIterations) {
 ; CHECK-NEXT:    %ptr = getelementptr inbounds float, ptr %input, i64 %index64
 ; CHECK-NEXT:    --> ((4 * (sext i32 {(-2147483648 * %start),+,-2147483648}<%loop> to i64))<nsw> + %input) U: full-set S: full-set Exits: ((4 * (sext i32 (-2147483648 + (-2147483648 * %numIterations)) to i64))<nsw> + %input) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %nexti = add nsw i32 %i, 1
-; CHECK-NEXT:    --> {(1 + %start)<nsw>,+,1}<nsw><%loop> U: full-set S: full-set Exits: %numIterations LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(1 + %start)<nsw>,+,1}<nsw><%loop> U: [-2147483647,-2147483648) S: [-2147483647,-2147483648) Exits: %numIterations LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test-shl-nuw-edgecase
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (-1 + (-1 * %start) + %numIterations)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is -1
@@ -1210,7 +1210,7 @@ define void @test-shl-nuw-nsw(ptr %input, i32 %start, i32 %numIterations) {
 ; CHECK-NEXT:    %ptr = getelementptr inbounds float, ptr %input, i64 %index64
 ; CHECK-NEXT:    --> {((4 * (sext i32 (-2147483648 * %start) to i64))<nsw> + %input),+,-8589934592}<nw><%loop> U: full-set S: full-set Exits: ((4 * (sext i32 (-2147483648 * %start) to i64))<nsw> + (-8589934592 * (zext i32 (-1 + (-1 * %start) + %numIterations) to i64)) + %input) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %nexti = add nsw i32 %i, 1
-; CHECK-NEXT:    --> {(1 + %start)<nsw>,+,1}<nsw><%loop> U: full-set S: full-set Exits: %numIterations LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(1 + %start)<nsw>,+,1}<nsw><%loop> U: [-2147483647,-2147483648) S: [-2147483647,-2147483648) Exits: %numIterations LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test-shl-nuw-nsw
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (-1 + (-1 * %start) + %numIterations)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is -1
@@ -1250,7 +1250,7 @@ define void @test-shl-no-nsw(ptr %input, i32 %start, i32 %numIterations) {
 ; CHECK-NEXT:    %ptr = getelementptr inbounds float, ptr %input, i64 %index64
 ; CHECK-NEXT:    --> ((4 * (sext i32 {(-2147483648 * %start),+,-2147483648}<%loop> to i64))<nsw> + %input) U: full-set S: full-set Exits: ((4 * (sext i32 (-2147483648 + (-2147483648 * %numIterations)) to i64))<nsw> + %input) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %nexti = add nsw i32 %i, 1
-; CHECK-NEXT:    --> {(1 + %start)<nsw>,+,1}<nsw><%loop> U: full-set S: full-set Exits: %numIterations LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(1 + %start)<nsw>,+,1}<nsw><%loop> U: [-2147483647,-2147483648) S: [-2147483647,-2147483648) Exits: %numIterations LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test-shl-no-nsw
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (-1 + (-1 * %start) + %numIterations)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is -1
@@ -1290,7 +1290,7 @@ define void @test-shl-nsw-edgecase(ptr %input, i32 %start, i32 %numIterations) {
 ; CHECK-NEXT:    %ptr = getelementptr inbounds float, ptr %input, i64 %index64
 ; CHECK-NEXT:    --> {((4 * (sext i32 (1073741824 * %start) to i64))<nsw> + %input),+,4294967296}<nw><%loop> U: full-set S: full-set Exits: ((4 * (sext i32 (1073741824 * %start) to i64))<nsw> + (4294967296 * (zext i32 (-1 + (-1 * %start) + %numIterations) to i64))<nuw> + %input) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %nexti = add nsw i32 %i, 1
-; CHECK-NEXT:    --> {(1 + %start)<nsw>,+,1}<nsw><%loop> U: full-set S: full-set Exits: %numIterations LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(1 + %start)<nsw>,+,1}<nsw><%loop> U: [-2147483647,-2147483648) S: [-2147483647,-2147483648) Exits: %numIterations LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test-shl-nsw-edgecase
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (-1 + (-1 * %start) + %numIterations)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is -1
@@ -1369,7 +1369,7 @@ define void @test-sub-no-nsw(ptr %input, i32 %start, i32 %sub, i32 %numIteration
 ; CHECK-NEXT:    %ptr = getelementptr inbounds float, ptr %input, i64 %index64
 ; CHECK-NEXT:    --> {((4 * (sext i32 %start to i64))<nsw> + (-4 * (sext i32 %sub to i64))<nsw> + %input),+,4}<nw><%loop> U: full-set S: full-set Exits: ((4 * (zext i32 (-1 + (-1 * %start) + %numIterations) to i64))<nuw><nsw> + (4 * (sext i32 %start to i64))<nsw> + (-4 * (sext i32 %sub to i64))<nsw> + %input) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %nexti = add nsw i32 %i, 1
-; CHECK-NEXT:    --> {(1 + %start)<nsw>,+,1}<nsw><%loop> U: full-set S: full-set Exits: %numIterations LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(1 + %start)<nsw>,+,1}<nsw><%loop> U: [-2147483647,-2147483648) S: [-2147483647,-2147483648) Exits: %numIterations LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test-sub-no-nsw
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (-1 + (-1 * %start) + %numIterations)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is -1
@@ -1411,7 +1411,7 @@ define void @test-sub-nsw(ptr %input, i32 %start, i32 %sub, i32 %numIterations)
 ; CHECK-NEXT:    %ptr = getelementptr inbounds float, ptr %input, i64 %index64
 ; CHECK-NEXT:    --> {((4 * (sext i32 %start to i64))<nsw> + (-4 * (sext i32 %halfsub to i64))<nsw> + %input),+,4}<nw><%loop> U: full-set S: full-set Exits: ((4 * (zext i32 (-1 + (-1 * %start) + %numIterations) to i64))<nuw><nsw> + (4 * (sext i32 %start to i64))<nsw> + (-4 * (sext i32 %halfsub to i64))<nsw> + %input) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %nexti = add nsw i32 %i, 1
-; CHECK-NEXT:    --> {(1 + %start)<nsw>,+,1}<nsw><%loop> U: full-set S: full-set Exits: %numIterations LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(1 + %start)<nsw>,+,1}<nsw><%loop> U: [-2147483647,-2147483648) S: [-2147483647,-2147483648) Exits: %numIterations LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test-sub-nsw
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (-1 + (-1 * %start) + %numIterations)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is -1
diff --git a/llvm/test/Analysis/ScalarEvolution/fold.ll b/llvm/test/Analysis/ScalarEvolution/fold.ll
index 6a57e3d6ed807..2096d309b2435 100644
--- a/llvm/test/Analysis/ScalarEvolution/fold.ll
+++ b/llvm/test/Analysis/ScalarEvolution/fold.ll
@@ -55,17 +55,17 @@ define void @test4(i32 %x, i32 %y) {
 ; CHECK-NEXT:    %Y = and i32 %y, 3
 ; CHECK-NEXT:    --> (zext i2 (trunc i32 %y to i2) to i32) U: [0,4) S: [0,4)
 ; CHECK-NEXT:    %A = phi i32 [ 0, %entry ], [ %I, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,21) S: [0,21) Exits: 20 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,21) S: [0,21) Exits: 20 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %Z1 = select i1 %rand1, i32 %A, i32 %Y
-; CHECK-NEXT:    --> ((zext i2 (trunc i32 %y to i2) to i32) smax {0,+,1}<%loop>) U: [0,21) S: [0,21) Exits: 20 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> ((zext i2 (trunc i32 %y to i2) to i32) smax {0,+,1}<nuw><nsw><%loop>) U: [0,21) S: [0,21) Exits: 20 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %Z2 = select i1 %rand2, i32 %A, i32 %Z1
-; CHECK-NEXT:    --> ({0,+,1}<%loop> umax ((zext i2 (trunc i32 %y to i2) to i32) smax {0,+,1}<%loop>)) U: [0,21) S: [0,21) Exits: 20 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> ({0,+,1}<nuw><nsw><%loop> umax ((zext i2 (trunc i32 %y to i2) to i32) smax {0,+,1}<nuw><nsw><%loop>)) U: [0,21) S: [0,21) Exits: 20 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %B = trunc i32 %Z2 to i16
-; CHECK-NEXT:    --> (trunc i32 ({0,+,1}<%loop> umax ((zext i2 (trunc i32 %y to i2) to i32) smax {0,+,1}<%loop>)) to i16) U: [0,21) S: [0,21) Exits: 20 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> (trunc i32 ({0,+,1}<nuw><nsw><%loop> umax ((zext i2 (trunc i32 %y to i2) to i32) smax {0,+,1}<nuw><nsw><%loop>)) to i16) U: [0,21) S: [0,21) Exits: 20 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %C = sext i16 %B to i30
-; CHECK-NEXT:    --> (trunc i32 ({0,+,1}<%loop> umax ((zext i2 (trunc i32 %y to i2) to i32) smax {0,+,1}<%loop>)) to i30) U: [0,21) S: [0,21) Exits: 20 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> (trunc i32 ({0,+,1}<nuw><nsw><%loop> umax ((zext i2 (trunc i32 %y to i2) to i32) smax {0,+,1}<nuw><nsw><%loop>)) to i30) U: [0,21) S: [0,21) Exits: 20 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %D = sext i16 %B to i32
-; CHECK-NEXT:    --> ({0,+,1}<%loop> umax ((zext i2 (trunc i32 %y to i2) to i32) smax {0,+,1}<%loop>)) U: [0,21) S: [0,21) Exits: 20 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> ({0,+,1}<nuw><nsw><%loop> umax ((zext i2 (trunc i32 %y to i2) to i32) smax {0,+,1}<nuw><nsw><%loop>)) U: [0,21) S: [0,21) Exits: 20 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %E = sext i16 %B to i34
 ; CHECK-NEXT:    --> ((zext i32 ((zext i2 (trunc i32 %y to i2) to i32) smax {0,+,1}<nuw><nsw><%loop>) to i34) umax {0,+,1}<nuw><nsw><%loop>) U: [0,21) S: [0,21) Exits: 20 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %F = zext i16 %B to i30
diff --git a/llvm/test/Analysis/ScalarEvolution/incorrect-exit-count.ll b/llvm/test/Analysis/ScalarEvolution/incorrect-exit-count.ll
index 765bfb2b5e546..98b832dedacea 100644
--- a/llvm/test/Analysis/ScalarEvolution/incorrect-exit-count.ll
+++ b/llvm/test/Analysis/ScalarEvolution/incorrect-exit-count.ll
@@ -17,11 +17,11 @@ define dso_local i32 @f() {
 ; CHECK-NEXT:    %storemerge23 = phi i32 [ 3, %entry ], [ %dec16, %for.inc13.3 ]
 ; CHECK-NEXT:    --> {3,+,-1}<nsw><%outer.loop> U: [1,4) S: [1,4) Exits: <<Unknown>> LoopDispositions: { %outer.loop: Computable, %for.cond6: Invariant, %inner.loop: Invariant }
 ; CHECK-NEXT:    %storemerge1921 = phi i32 [ 3, %outer.loop ], [ %dec, %for.end ]
-; CHECK-NEXT:    --> {3,+,-1}<nuw><nsw><%for.cond6> U: [3,4) S: [3,4) Exits: <<Unknown>> LoopDispositions: { %for.cond6: Computable, %outer.loop: Variant }
+; CHECK-NEXT:    --> {3,+,-1}<nsw><%for.cond6> U: [3,4) S: [3,4) Exits: <<Unknown>> LoopDispositions: { %for.cond6: Computable, %outer.loop: Variant }
 ; CHECK-NEXT:    %idxprom20 = zext i32 %storemerge1921 to i64
-; CHECK-NEXT:    --> {3,+,4294967295}<nuw><nsw><%for.cond6> U: [3,4) S: [3,4) Exits: <<Unknown>> LoopDispositions: { %for.cond6: Computable, %outer.loop: Variant }
+; CHECK-NEXT:    --> (zext i32 {3,+,-1}<nsw><%for.cond6> to i64) U: [3,4) S: [3,4) Exits: <<Unknown>> LoopDispositions: { %for.cond6: Computable, %outer.loop: Variant }
 ; CHECK-NEXT:    %arrayidx7 = getelementptr inbounds [1 x [4 x i16]], ptr @__const.f.g, i64 0, i64 0, i64 %idxprom20
-; CHECK-NEXT:    --> {(6 + @__const.f.g),+,8589934590}<nuw><%for.cond6> U: [0,-1) S: [-9223372036854775808,9223372036854775807) Exits: <<Unknown>> LoopDispositions: { %for.cond6: Computable, %outer.loop: Variant }
+; CHECK-NEXT:    --> ((2 * (zext i32 {3,+,-1}<nsw><%for.cond6> to i64))<nuw><nsw> + @__const.f.g)<nuw> U: [6,-1) S: [-9223372036854775808,9223372036854775807) Exits: <<Unknown>> LoopDispositions: { %for.cond6: Computable, %outer.loop: Variant }
 ; CHECK-NEXT:    %i = load i16, ptr %arrayidx7, align 2
 ; CHECK-NEXT:    --> %i U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %for.cond6: Variant, %outer.loop: Variant }
 ; CHECK-NEXT:    %storemerge1822.lcssa.ph = phi i32 [ 0, %for.cond6 ]
@@ -41,11 +41,11 @@ define dso_local i32 @f() {
 ; CHECK-NEXT:    %retval.0 = phi i32 [ %i1, %if.end ], [ 0, %cleanup.loopexit ]
 ; CHECK-NEXT:    --> %retval.0 U: full-set S: full-set
 ; CHECK-NEXT:    %storemerge1921.3 = phi i32 [ 3, %for.end ], [ %dec.3, %for.end.3 ]
-; CHECK-NEXT:    --> {3,+,-1}<nuw><nsw><%inner.loop> U: [3,4) S: [3,4) Exits: <<Unknown>> LoopDispositions: { %inner.loop: Computable, %outer.loop: Variant }
+; CHECK-NEXT:    --> {3,+,-1}<nsw><%inner.loop> U: [3,4) S: [3,4) Exits: <<Unknown>> LoopDispositions: { %inner.loop: Computable, %outer.loop: Variant }
 ; CHECK-NEXT:    %idxprom20.3 = zext i32 %storemerge1921.3 to i64
-; CHECK-NEXT:    --> {3,+,4294967295}<nuw><nsw><%inner.loop> U: [3,4) S: [3,4) Exits: <<Unknown>> LoopDispositions: { %inner.loop: Computable, %outer.loop: Variant }
+; CHECK-NEXT:    --> (zext i32 {3,+,-1}<nsw><%inner.loop> to i64) U: [3,4) S: [3,4) Exits: <<Unknown>> LoopDispositions: { %inner.loop: Computable, %outer.loop: Variant }
 ; CHECK-NEXT:    %arrayidx7.3 = getelementptr inbounds [1 x [4 x i16]], ptr @__const.f.g, i64 0, i64 0, i64 %idxprom20.3
-; CHECK-NEXT:    --> {(6 + @__const.f.g),+,8589934590}<nuw><%inner.loop> U: [0,-1) S: [-9223372036854775808,9223372036854775807) Exits: <<Unknown>> LoopDispositions: { %inner.loop: Computable, %outer.loop: Variant }
+; CHECK-NEXT:    --> ((2 * (zext i32 {3,+,-1}<nsw><%inner.loop> to i64))<nuw><nsw> + @__const.f.g)<nuw> U: [6,-1) S: [-9223372036854775808,9223372036854775807) Exits: <<Unknown>> LoopDispositions: { %inner.loop: Computable, %outer.loop: Variant }
 ; CHECK-NEXT:    %i7 = load i16, ptr %arrayidx7.3, align 2
 ; CHECK-NEXT:    --> %i7 U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %inner.loop: Variant, %outer.loop: Variant }
 ; CHECK-NEXT:    %i8 = load volatile i32, ptr @b, align 4
@@ -53,7 +53,7 @@ define dso_local i32 @f() {
 ; CHECK-NEXT:    %dec.3 = add nsw i32 %storemerge1921.3, -1
 ; CHECK-NEXT:    --> {2,+,-1}<nsw><%inner.loop> U: [2,3) S: [2,3) Exits: <<Unknown>> LoopDispositions: { %inner.loop: Computable, %outer.loop: Variant }
 ; CHECK-NEXT:    %storemerge1921.lcssa25.3 = phi i32 [ %storemerge1921.3, %for.end.3 ]
-; CHECK-NEXT:    --> {3,+,-1}<nuw><nsw><%inner.loop> U: [3,4) S: [3,4) Exits: <<Unknown>> LoopDispositions: { %outer.loop: Variant, %for.cond6: Variant, %inner.loop: Computable }
+; CHECK-NEXT:    --> {3,+,-1}<nsw><%inner.loop> U: [3,4) S: [3,4) Exits: <<Unknown>> LoopDispositions: { %outer.loop: Variant, %for.cond6: Variant, %inner.loop: Computable }
 ; CHECK-NEXT:    %dec16 = add nsw i32 %storemerge23, -1
 ; CHECK-NEXT:    --> {2,+,-1}<nsw><%outer.loop> U: [0,3) S: [0,3) Exits: <<Unknown>> LoopDispositions: { %outer.loop: Computable, %for.cond6: Invariant, %inner.loop: Invariant }
 ; CHECK-NEXT:  Determining loop execution counts for: @f
diff --git a/llvm/test/Analysis/ScalarEvolution/increasing-or-decreasing-iv.ll b/llvm/test/Analysis/ScalarEvolution/increasing-or-decreasing-iv.ll
index f2d0ccac329b4..fe448b39e1265 100644
--- a/llvm/test/Analysis/ScalarEvolution/increasing-or-decreasing-iv.ll
+++ b/llvm/test/Analysis/ScalarEvolution/increasing-or-decreasing-iv.ll
@@ -9,13 +9,13 @@ define void @f0(i1 %c) {
 ; CHECK-NEXT:    %step = select i1 %c, i32 -1, i32 1
 ; CHECK-NEXT:    --> %step U: [1,0) S: [-2,2)
 ; CHECK-NEXT:    %loop.iv = phi i32 [ 0, %entry ], [ %loop.iv.inc, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,128) S: [0,128) Exits: 127 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,128) S: [0,128) Exits: 127 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv = phi i32 [ %start, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {%start,+,%step}<%loop> U: [0,128) S: [0,128) Exits: ((127 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {%start,+,%step}<nsw><%loop> U: [0,128) S: [0,128) Exits: ((127 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i32 %iv, %step
-; CHECK-NEXT:    --> {(%step + %start),+,%step}<%loop> U: [-256,256) S: [-256,256) Exits: ((128 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(%step + %start),+,%step}<nw><%loop> U: [-256,256) S: [-256,256) Exits: ((128 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %loop.iv.inc = add i32 %loop.iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,129) S: [1,129) Exits: 128 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,129) S: [1,129) Exits: 128 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @f0
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is 127
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 127
@@ -49,41 +49,41 @@ define void @f1(i1 %c) {
 ; CHECK-NEXT:    %step = select i1 %c, i32 -8, i32 8
 ; CHECK-NEXT:    --> %step U: [8,-7) S: [-16,16)
 ; CHECK-NEXT:    %loop.iv = phi i32 [ 0, %entry ], [ %loop.iv.inc, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,16) S: [0,16) Exits: 15 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,16) S: [0,16) Exits: 15 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv = phi i32 [ %start, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {%start,+,%step}<%loop> U: [0,121) S: [0,121) Exits: ((15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {%start,+,%step}<nsw><%loop> U: [0,121) S: [0,121) Exits: ((15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.1 = add i32 %iv, 1
-; CHECK-NEXT:    --> {(1 + %start)<nuw><nsw>,+,%step}<%loop> U: [1,122) S: [1,122) Exits: (1 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(1 + %start)<nuw><nsw>,+,%step}<nsw><%loop> U: [1,122) S: [1,122) Exits: (1 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.2 = add i32 %iv, 2
-; CHECK-NEXT:    --> {(2 + %start)<nuw><nsw>,+,%step}<%loop> U: [2,123) S: [2,123) Exits: (2 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(2 + %start)<nuw><nsw>,+,%step}<nsw><%loop> U: [2,123) S: [2,123) Exits: (2 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.3 = add i32 %iv, 3
-; CHECK-NEXT:    --> {(3 + %start)<nuw><nsw>,+,%step}<%loop> U: [3,124) S: [3,124) Exits: (3 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(3 + %start)<nuw><nsw>,+,%step}<nsw><%loop> U: [3,124) S: [3,124) Exits: (3 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.4 = add i32 %iv, 4
-; CHECK-NEXT:    --> {(4 + %start)<nuw><nsw>,+,%step}<%loop> U: [4,125) S: [4,125) Exits: (4 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(4 + %start)<nuw><nsw>,+,%step}<nsw><%loop> U: [4,125) S: [4,125) Exits: (4 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.5 = add i32 %iv, 5
-; CHECK-NEXT:    --> {(5 + %start)<nuw><nsw>,+,%step}<%loop> U: [5,126) S: [5,126) Exits: (5 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(5 + %start)<nuw><nsw>,+,%step}<nsw><%loop> U: [5,126) S: [5,126) Exits: (5 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.6 = add i32 %iv, 6
-; CHECK-NEXT:    --> {(6 + %start)<nuw><nsw>,+,%step}<%loop> U: [6,127) S: [6,127) Exits: (6 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(6 + %start)<nuw><nsw>,+,%step}<nsw><%loop> U: [6,127) S: [6,127) Exits: (6 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.7 = add i32 %iv, 7
-; CHECK-NEXT:    --> {(7 + %start)<nuw><nsw>,+,%step}<%loop> U: [7,128) S: [7,128) Exits: (7 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(7 + %start)<nuw><nsw>,+,%step}<nsw><%loop> U: [7,128) S: [7,128) Exits: (7 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.m1 = sub i32 %iv, 1
-; CHECK-NEXT:    --> {(-1 + %start)<nsw>,+,%step}<%loop> U: [-1,120) S: [-1,120) Exits: (-1 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(-1 + %start)<nsw>,+,%step}<nsw><%loop> U: [-1,120) S: [-1,120) Exits: (-1 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.m2 = sub i32 %iv, 2
-; CHECK-NEXT:    --> {(-2 + %start)<nsw>,+,%step}<%loop> U: [0,-1) S: [-2,119) Exits: (-2 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(-2 + %start)<nsw>,+,%step}<nsw><%loop> U: [0,-1) S: [-2,119) Exits: (-2 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.m3 = sub i32 %iv, 3
-; CHECK-NEXT:    --> {(-3 + %start)<nsw>,+,%step}<%loop> U: [-3,118) S: [-3,118) Exits: (-3 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(-3 + %start)<nsw>,+,%step}<nsw><%loop> U: [-3,118) S: [-3,118) Exits: (-3 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.m4 = sub i32 %iv, 4
-; CHECK-NEXT:    --> {(-4 + %start)<nsw>,+,%step}<%loop> U: [0,-3) S: [-4,117) Exits: (-4 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(-4 + %start)<nsw>,+,%step}<nsw><%loop> U: [0,-3) S: [-4,117) Exits: (-4 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.m5 = sub i32 %iv, 5
-; CHECK-NEXT:    --> {(-5 + %start)<nsw>,+,%step}<%loop> U: [-5,116) S: [-5,116) Exits: (-5 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(-5 + %start)<nsw>,+,%step}<nsw><%loop> U: [-5,116) S: [-5,116) Exits: (-5 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.m6 = sub i32 %iv, 6
-; CHECK-NEXT:    --> {(-6 + %start)<nsw>,+,%step}<%loop> U: [0,-1) S: [-6,115) Exits: (-6 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(-6 + %start)<nsw>,+,%step}<nsw><%loop> U: [0,-1) S: [-6,115) Exits: (-6 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.m7 = sub i32 %iv, 7
-; CHECK-NEXT:    --> {(-7 + %start)<nsw>,+,%step}<%loop> U: [-7,114) S: [-7,114) Exits: (-7 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(-7 + %start)<nsw>,+,%step}<nsw><%loop> U: [-7,114) S: [-7,114) Exits: (-7 + (15 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i32 %iv, %step
-; CHECK-NEXT:    --> {(%step + %start),+,%step}<%loop> U: [0,-7) S: [-256,361) Exits: ((16 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(%step + %start),+,%step}<nw><%loop> U: [0,-7) S: [-256,361) Exits: ((16 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %loop.iv.inc = add i32 %loop.iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,17) S: [1,17) Exits: 16 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,17) S: [1,17) Exits: 16 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @f1
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is 15
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 15
@@ -136,15 +136,15 @@ define void @f2(i1 %c) {
 ; CHECK-NEXT:    %step = select i1 %c, i32 -1, i32 1
 ; CHECK-NEXT:    --> %step U: [1,0) S: [-2,2)
 ; CHECK-NEXT:    %loop.iv = phi i32 [ 0, %entry ], [ %loop.iv.inc, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,128) S: [0,128) Exits: 127 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,128) S: [0,128) Exits: 127 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv = phi i32 [ %start, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {%start,+,%step}<%loop> U: [0,128) S: [0,128) Exits: ((127 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {%start,+,%step}<nsw><%loop> U: [0,128) S: [0,128) Exits: ((127 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.sext = sext i32 %iv to i64
 ; CHECK-NEXT:    --> {(zext i32 %start to i64),+,(sext i32 %step to i64)}<nsw><%loop> U: [0,128) S: [0,128) Exits: ((zext i32 %start to i64) + (127 * (sext i32 %step to i64))<nsw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i32 %iv, %step
 ; CHECK-NEXT:    --> {(%step + %start),+,%step}<nw><%loop> U: [-256,256) S: [-256,256) Exits: ((128 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %loop.iv.inc = add i32 %loop.iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,129) S: [1,129) Exits: 128 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,129) S: [1,129) Exits: 128 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @f2
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is 127
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 127
@@ -179,15 +179,15 @@ define void @f3(i1 %c) {
 ; CHECK-NEXT:    %step = select i1 %c, i16 1, i16 509
 ; CHECK-NEXT:    --> %step U: [1,510) S: [1,510)
 ; CHECK-NEXT:    %loop.iv = phi i16 [ 0, %entry ], [ %loop.iv.inc, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,128) S: [0,128) Exits: 127 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,128) S: [0,128) Exits: 127 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv = phi i16 [ %start, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {%start,+,%step}<%loop> U: [0,-892) S: [0,-892) Exits: ((127 * %step)<nuw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {%start,+,%step}<nuw><%loop> U: [0,-892) S: [0,-892) Exits: ((127 * %step)<nuw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.zext = zext i16 %iv to i64
 ; CHECK-NEXT:    --> {(zext i16 %start to i64),+,(zext i16 %step to i64)}<nuw><%loop> U: [0,64644) S: [0,64644) Exits: ((zext i16 %start to i64) + (127 * (zext i16 %step to i64))<nuw><nsw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i16 %iv, %step
 ; CHECK-NEXT:    --> {(%step + %start),+,%step}<nw><%loop> U: full-set S: full-set Exits: ((128 * %step)<nuw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %loop.iv.inc = add i16 %loop.iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,129) S: [1,129) Exits: 128 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,129) S: [1,129) Exits: 128 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @f3
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is 127
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 127
@@ -229,15 +229,15 @@ define void @f4(i1 %c) {
 ; CHECK-NEXT:    %step = select i1 %c, i32 -1, i32 1
 ; CHECK-NEXT:    --> %step U: [1,0) S: [-2,2)
 ; CHECK-NEXT:    %loop.iv = phi i32 [ 0, %entry ], [ %loop.iv.inc, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,128) S: [0,128) Exits: 127 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,128) S: [0,128) Exits: 127 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv = phi i32 [ %start, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {%start,+,%step}<%loop> U: [0,128) S: [0,128) Exits: ((127 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {%start,+,%step}<nsw><%loop> U: [0,128) S: [0,128) Exits: ((127 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.trunc = trunc i32 %iv to i16
 ; CHECK-NEXT:    --> {(trunc i32 %start to i16),+,(trunc i32 %step to i16)}<%loop> U: full-set S: full-set Exits: ((trunc i32 %start to i16) + (127 * (trunc i32 %step to i16))<nsw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i32 %iv, %step
-; CHECK-NEXT:    --> {(%step + %start),+,%step}<%loop> U: [-256,256) S: [-256,256) Exits: ((128 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(%step + %start),+,%step}<nw><%loop> U: [-256,256) S: [-256,256) Exits: ((128 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %loop.iv.inc = add i32 %loop.iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,129) S: [1,129) Exits: 128 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,129) S: [1,129) Exits: 128 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @f4
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is 127
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 127
@@ -278,15 +278,15 @@ define void @f5(i1 %c) {
 ; CHECK-NEXT:    %step = select i1 %c, i32 -1, i32 1
 ; CHECK-NEXT:    --> %step U: [1,0) S: [-2,2)
 ; CHECK-NEXT:    %loop.iv = phi i16 [ 0, %entry ], [ %loop.iv.inc, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,128) S: [0,128) Exits: 127 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,128) S: [0,128) Exits: 127 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv = phi i32 [ %start, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {%start,+,%step}<%loop> U: [0,128) S: [0,128) Exits: ((127 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {%start,+,%step}<nsw><%loop> U: [0,128) S: [0,128) Exits: ((127 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.trunc = trunc i32 %iv to i16
 ; CHECK-NEXT:    --> {(trunc i32 %start to i16),+,(trunc i32 %step to i16)}<%loop> U: [0,128) S: [0,128) Exits: ((trunc i32 %start to i16) + (127 * (trunc i32 %step to i16))<nsw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i32 %iv, %step
-; CHECK-NEXT:    --> {(%step + %start),+,%step}<%loop> U: [-256,256) S: [-256,256) Exits: ((128 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(%step + %start),+,%step}<nw><%loop> U: [-256,256) S: [-256,256) Exits: ((128 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %loop.iv.inc = add i16 %loop.iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,129) S: [1,129) Exits: 128 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,129) S: [1,129) Exits: 128 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @f5
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is 127
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 127
@@ -322,17 +322,17 @@ define void @f6(i1 %c) {
 ; CHECK-NEXT:    %step = select i1 %c, i32 -2, i32 0
 ; CHECK-NEXT:    --> %step U: [0,-1) S: [-2,2)
 ; CHECK-NEXT:    %loop.iv = phi i16 [ 0, %entry ], [ %loop.iv.inc, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,128) S: [0,128) Exits: 127 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,128) S: [0,128) Exits: 127 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv = phi i32 [ %start, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {%start,+,(1 + %step)<nuw><nsw>}<%loop> U: [0,128) S: [0,128) Exits: (127 + (127 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {%start,+,(1 + %step)<nuw><nsw>}<nsw><%loop> U: [0,128) S: [0,128) Exits: (127 + (127 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %step.plus.one = add i32 %step, 1
 ; CHECK-NEXT:    --> (1 + %step)<nuw><nsw> U: [1,0) S: [-1,3) Exits: (1 + %step)<nuw><nsw> LoopDispositions: { %loop: Invariant }
 ; CHECK-NEXT:    %iv.next = add i32 %iv, %step.plus.one
-; CHECK-NEXT:    --> {(1 + %step + %start),+,(1 + %step)<nuw><nsw>}<%loop> U: [-128,384) S: [-128,384) Exits: (128 + (128 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(1 + %step + %start),+,(1 + %step)<nuw><nsw>}<nw><%loop> U: [-128,384) S: [-128,384) Exits: (128 + (128 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.sext = sext i32 %iv to i64
 ; CHECK-NEXT:    --> {(zext i32 %start to i64),+,(1 + (sext i32 %step to i64))<nuw><nsw>}<nsw><%loop> U: [0,128) S: [0,128) Exits: (127 + (zext i32 %start to i64) + (127 * (sext i32 %step to i64))<nsw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %loop.iv.inc = add i16 %loop.iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,129) S: [1,129) Exits: 128 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,129) S: [1,129) Exits: 128 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @f6
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is 127
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 127
@@ -369,19 +369,19 @@ define void @f7(i1 %c) {
 ; CHECK-NEXT:    %step = select i1 %c, i32 -1, i32 1
 ; CHECK-NEXT:    --> %step U: [1,0) S: [-2,2)
 ; CHECK-NEXT:    %loop.iv = phi i16 [ 0, %entry ], [ %loop.iv.inc, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,128) S: [0,128) Exits: 127 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,128) S: [0,128) Exits: 127 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv = phi i32 [ %start, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {%start,+,%step}<%loop> U: [0,128) S: [0,128) Exits: ((127 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {%start,+,%step}<nsw><%loop> U: [0,128) S: [0,128) Exits: ((127 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.trunc = trunc i32 %iv to i16
 ; CHECK-NEXT:    --> {(trunc i32 %start to i16),+,(trunc i32 %step to i16)}<%loop> U: [0,128) S: [0,128) Exits: ((trunc i32 %start to i16) + (127 * (trunc i32 %step to i16))<nsw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i32 %iv, %step
-; CHECK-NEXT:    --> {(%step + %start),+,%step}<%loop> U: [-256,256) S: [-256,256) Exits: ((128 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(%step + %start),+,%step}<nw><%loop> U: [-256,256) S: [-256,256) Exits: ((128 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.trunc.plus.one = add i16 %iv.trunc, 1
 ; CHECK-NEXT:    --> {(1 + (trunc i32 %start to i16))<nuw><nsw>,+,(trunc i32 %step to i16)}<%loop> U: [1,129) S: [1,129) Exits: (1 + (trunc i32 %start to i16) + (127 * (trunc i32 %step to i16))<nsw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.trunc.plus.two = add i16 %iv.trunc, 2
 ; CHECK-NEXT:    --> {(2 + (trunc i32 %start to i16))<nuw><nsw>,+,(trunc i32 %step to i16)}<%loop> U: [2,130) S: [2,130) Exits: (2 + (trunc i32 %start to i16) + (127 * (trunc i32 %step to i16))<nsw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %loop.iv.inc = add i16 %loop.iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,129) S: [1,129) Exits: 128 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,129) S: [1,129) Exits: 128 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @f7
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is 127
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 127
diff --git a/llvm/test/Analysis/ScalarEvolution/limit-depth.ll b/llvm/test/Analysis/ScalarEvolution/limit-depth.ll
index bf7d2ae1c53eb..b053eb3cccc63 100644
--- a/llvm/test/Analysis/ScalarEvolution/limit-depth.ll
+++ b/llvm/test/Analysis/ScalarEvolution/limit-depth.ll
@@ -87,7 +87,7 @@ exit:
 define void @test_zext(i32 %a, i32 %b, i32 %c, i32 %d, i32 %e, i32 %f) {
 ; CHECK-LABEL: @test_zext
 ; CHECK:          %ze2 = zext i64 %iv2.inc to i128
-; CHECK-NEXT:     -->  {(1 + (zext i64 {7,+,1}<nuw><nsw><%loop> to i128))<nuw><nsw>,+,1}<nuw><%loop2>
+; CHECK-NEXT:     -->  {(1 + (zext i64 {7,+,1}<nuw><nsw><%loop> to i128))<nuw><nsw>,+,1}<nuw><nsw><%loop2>
 entry:
   br label %loop
 
diff --git a/llvm/test/Analysis/ScalarEvolution/logical-operations.ll b/llvm/test/Analysis/ScalarEvolution/logical-operations.ll
index 1919ee67bfc66..d557f50326d53 100644
--- a/llvm/test/Analysis/ScalarEvolution/logical-operations.ll
+++ b/llvm/test/Analysis/ScalarEvolution/logical-operations.ll
@@ -410,11 +410,14 @@ define ptr @select_between_constantptrs(i1 %c, ptr %x) {
 define ptr @tautological_select() {
 ; CHECK-LABEL: 'tautological_select'
 ; CHECK-NEXT:  Classifying expressions for: @tautological_select
-; CHECK-NEXT:    %r = getelementptr i8, ptr @constant, i32 0
+; CHECK-NEXT:    %s = select i1 true, ptr @constant, ptr @another_constant
+; CHECK-NEXT:    --> @constant U: [0,-3) S: [-9223372036854775808,9223372036854775805)
+; CHECK-NEXT:    %r = getelementptr i8, ptr %s
 ; CHECK-NEXT:    --> @constant U: [0,-3) S: [-9223372036854775808,9223372036854775805)
 ; CHECK-NEXT:  Determining loop execution counts for: @tautological_select
 ;
-  %r = getelementptr i8, ptr select (i1 true, ptr @constant, ptr @another_constant), i32 0
+  %s = select i1 true, ptr @constant, ptr @another_constant
+  %r = getelementptr i8, ptr %s
   ret ptr %r
 }
 
@@ -422,11 +425,11 @@ define ptr @tautological_select_like_phi(i32 %tc) {
 ; CHECK-LABEL: 'tautological_select_like_phi'
 ; CHECK-NEXT:  Classifying expressions for: @tautological_select_like_phi
 ; CHECK-NEXT:    %iv = phi i32 [ 0, %entry ], [ %iv.next, %latch ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,101) S: [0,101) Exits: 100 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,101) S: [0,101) Exits: 100 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %r = phi ptr [ @constant, %truebb ], [ @another_constant, %falsebb ]
 ; CHECK-NEXT:    --> %r U: [0,-3) S: [-9223372036854775808,9223372036854775805) Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i32 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,102) S: [1,102) Exits: 101 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,102) S: [1,102) Exits: 101 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @tautological_select_like_phi
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is 100
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 100
diff --git a/llvm/test/Analysis/ScalarEvolution/max-backedge-taken-count-guard-info-rewrite-expressions.ll b/llvm/test/Analysis/ScalarEvolution/max-backedge-taken-count-guard-info-rewrite-expressions.ll
index a686ec231cc9e..49613180bad1a 100644
--- a/llvm/test/Analysis/ScalarEvolution/max-backedge-taken-count-guard-info-rewrite-expressions.ll
+++ b/llvm/test/Analysis/ScalarEvolution/max-backedge-taken-count-guard-info-rewrite-expressions.ll
@@ -54,11 +54,11 @@ define i32 @rewrite_zext_min_max(i32 %N, ptr %arr) {
 ; CHECK-NEXT:    %n.vec = and i64 %ext, 28
 ; CHECK-NEXT:    --> (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw> U: [0,17) S: [0,17)
 ; CHECK-NEXT:    %index = phi i64 [ 0, %loop.ph ], [ %index.next, %loop ]
-; CHECK-NEXT:    --> {0,+,4}<nuw><%loop> U: [0,13) S: [0,13) Exits: (4 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,4}<nuw><nsw><%loop> U: [0,13) S: [0,13) Exits: (4 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %gep = getelementptr inbounds i32, ptr %arr, i64 %index
 ; CHECK-NEXT:    --> {%arr,+,16}<nuw><%loop> U: full-set S: full-set Exits: ((16 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4)) + %arr) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %index.next = add nuw i64 %index, 4
-; CHECK-NEXT:    --> {4,+,4}<nuw><%loop> U: [4,17) S: [4,17) Exits: (4 + (4 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {4,+,4}<nuw><nsw><%loop> U: [4,17) S: [4,17) Exits: (4 + (4 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @rewrite_zext_min_max
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 3
@@ -102,11 +102,11 @@ define i32 @rewrite_min_max_zext(i32 %N, ptr %arr) {
 ; CHECK-NEXT:    %n.vec = and i64 %umin, 28
 ; CHECK-NEXT:    --> (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw> U: [0,17) S: [0,17)
 ; CHECK-NEXT:    %index = phi i64 [ 0, %loop.ph ], [ %index.next, %loop ]
-; CHECK-NEXT:    --> {0,+,4}<nuw><%loop> U: [0,13) S: [0,13) Exits: (4 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,4}<nuw><nsw><%loop> U: [0,13) S: [0,13) Exits: (4 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %gep = getelementptr inbounds i32, ptr %arr, i64 %index
 ; CHECK-NEXT:    --> {%arr,+,16}<nuw><%loop> U: full-set S: full-set Exits: ((16 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4)) + %arr) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %index.next = add nuw i64 %index, 4
-; CHECK-NEXT:    --> {4,+,4}<nuw><%loop> U: [4,17) S: [4,17) Exits: (4 + (4 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {4,+,4}<nuw><nsw><%loop> U: [4,17) S: [4,17) Exits: (4 + (4 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @rewrite_min_max_zext
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 3
@@ -250,9 +250,9 @@ define i32 @rewrite_zext_with_info_from_icmp_ne(i32 %N) {
 ; CHECK-NEXT:    %n.vec = and i64 %n.rnd.up, 8589934588
 ; CHECK-NEXT:    --> (4 * ((4 + (zext i32 (-1 + (zext i2 (trunc i32 %N to i2) to i32))<nsw> to i64))<nuw><nsw> /u 4))<nuw><nsw> U: [4,4294967297) S: [4,4294967297)
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %loop.ph ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,4}<%loop> U: [0,1) S: [0,1) Exits: 0 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,4}<nuw><nsw><%loop> U: [0,1) S: [0,1) Exits: 0 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 4
-; CHECK-NEXT:    --> {4,+,4}<%loop> U: [4,5) S: [4,5) Exits: 4 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {4,+,4}<nuw><nsw><%loop> U: [4,5) S: [4,5) Exits: 4 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @rewrite_zext_with_info_from_icmp_ne
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is 0
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 0
@@ -300,9 +300,9 @@ define i32 @rewrite_zext_no_icmp_ne(i32 %N) {
 ; CHECK-NEXT:    %n.vec = and i64 %n.rnd.up, 8589934588
 ; CHECK-NEXT:    --> (4 * ((4 + (zext i32 (-1 + (zext i2 (trunc i32 %N to i2) to i32))<nsw> to i64))<nuw><nsw> /u 4))<nuw><nsw> U: [4,4294967297) S: [4,4294967297)
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %loop.ph ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,4}<%loop> U: [0,4294967293) S: [0,4294967293) Exits: (4 * ((-4 + (4 * ((4 + (zext i32 (-1 + (zext i2 (trunc i32 %N to i2) to i32))<nsw> to i64))<nuw><nsw> /u 4))<nuw><nsw>)<nsw> /u 4))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,4}<nuw><nsw><%loop> U: [0,4294967293) S: [0,4294967293) Exits: (4 * ((-4 + (4 * ((4 + (zext i32 (-1 + (zext i2 (trunc i32 %N to i2) to i32))<nsw> to i64))<nuw><nsw> /u 4))<nuw><nsw>)<nsw> /u 4))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 4
-; CHECK-NEXT:    --> {4,+,4}<%loop> U: [4,4294967297) S: [4,4294967297) Exits: (4 + (4 * ((-4 + (4 * ((4 + (zext i32 (-1 + (zext i2 (trunc i32 %N to i2) to i32))<nsw> to i64))<nuw><nsw> /u 4))<nuw><nsw>)<nsw> /u 4))<nuw><nsw>)<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {4,+,4}<nuw><nsw><%loop> U: [4,4294967297) S: [4,4294967297) Exits: (4 + (4 * ((-4 + (4 * ((4 + (zext i32 (-1 + (zext i2 (trunc i32 %N to i2) to i32))<nsw> to i64))<nuw><nsw> /u 4))<nuw><nsw>)<nsw> /u 4))<nuw><nsw>)<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @rewrite_zext_no_icmp_ne
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is ((-4 + (4 * ((4 + (zext i32 (-1 + (zext i2 (trunc i32 %N to i2) to i32))<nsw> to i64))<nuw><nsw> /u 4))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 1073741823
@@ -430,9 +430,9 @@ define void @guard_pessimizes_analysis_step2(i1 %c, i32 %N) {
 ; CHECK-NEXT:    %init = phi i64 [ 2, %entry ], [ 4, %bb1 ]
 ; CHECK-NEXT:    --> %init U: [2,5) S: [2,5)
 ; CHECK-NEXT:    %iv = phi i64 [ %iv.next, %loop ], [ %init, %loop.ph ]
-; CHECK-NEXT:    --> {%init,+,2}<%loop> U: [2,17) S: [2,17) Exits: ((2 * ((14 + (-1 * %init)<nsw>)<nsw> /u 2))<nuw><nsw> + %init) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {%init,+,2}<nuw><nsw><%loop> U: [2,17) S: [2,17) Exits: ((2 * ((14 + (-1 * %init)<nsw>)<nsw> /u 2))<nuw><nsw> + %init) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 2
-; CHECK-NEXT:    --> {(2 + %init)<nuw><nsw>,+,2}<%loop> U: [4,19) S: [4,19) Exits: (2 + (2 * ((14 + (-1 * %init)<nsw>)<nsw> /u 2))<nuw><nsw> + %init) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(2 + %init)<nuw><nsw>,+,2}<nuw><nsw><%loop> U: [4,19) S: [4,19) Exits: (2 + (2 * ((14 + (-1 * %init)<nsw>)<nsw> /u 2))<nuw><nsw> + %init) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @guard_pessimizes_analysis_step2
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is ((14 + (-1 * %init)<nsw>)<nsw> /u 2)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 6
@@ -476,14 +476,14 @@ define i32 @rewrite_sext_slt_narrow_check(i32 %N, ptr %arr) {
 ; CHECK-NEXT:    %n.vec = and i64 %ext, 28
 ; CHECK-NEXT:    --> (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw> U: [0,29) S: [0,29)
 ; CHECK-NEXT:    %index = phi i64 [ 0, %loop.ph ], [ %index.next, %loop ]
-; CHECK-NEXT:    --> {0,+,4}<nuw><%loop> U: [0,-3) S: [-9223372036854775808,9223372036854775805) Exits: (4 * ((-4 + (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,4}<nuw><nsw><%loop> U: [0,13) S: [0,13) Exits: (4 * ((-4 + (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %gep = getelementptr inbounds i32, ptr %arr, i64 %index
-; CHECK-NEXT:    --> {%arr,+,16}<%loop> U: full-set S: full-set Exits: ((16 * ((-4 + (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)) + %arr) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {%arr,+,16}<nuw><%loop> U: full-set S: full-set Exits: ((16 * ((-4 + (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)) + %arr) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %index.next = add nuw i64 %index, 4
-; CHECK-NEXT:    --> {4,+,4}<nuw><%loop> U: [4,-3) S: [-9223372036854775808,9223372036854775805) Exits: (4 + (4 * ((-4 + (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {4,+,4}<nuw><nsw><%loop> U: [4,17) S: [4,17) Exits: (4 + (4 * ((-4 + (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @rewrite_sext_slt_narrow_check
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is ((-4 + (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)
-; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 4611686018427387903
+; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 3
 ; CHECK-NEXT:  Loop %loop: symbolic max backedge-taken count is ((-4 + (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:  Loop %loop: Predicated backedge-taken count is ((-4 + (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:   Predicates:
@@ -522,14 +522,14 @@ define i32 @rewrite_zext_ult_narrow_check(i32 %N, ptr %arr) {
 ; CHECK-NEXT:    %n.vec = and i64 %ext, 28
 ; CHECK-NEXT:    --> (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw> U: [0,29) S: [0,29)
 ; CHECK-NEXT:    %index = phi i64 [ 0, %loop.ph ], [ %index.next, %loop ]
-; CHECK-NEXT:    --> {0,+,4}<nuw><%loop> U: [0,-3) S: [-9223372036854775808,9223372036854775805) Exits: (4 * ((-4 + (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,4}<nuw><nsw><%loop> U: [0,13) S: [0,13) Exits: (4 * ((-4 + (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %gep = getelementptr inbounds i32, ptr %arr, i64 %index
-; CHECK-NEXT:    --> {%arr,+,16}<%loop> U: full-set S: full-set Exits: ((16 * ((-4 + (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)) + %arr) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {%arr,+,16}<nuw><%loop> U: full-set S: full-set Exits: ((16 * ((-4 + (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)) + %arr) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %index.next = add nuw i64 %index, 4
-; CHECK-NEXT:    --> {4,+,4}<nuw><%loop> U: [4,-3) S: [-9223372036854775808,9223372036854775805) Exits: (4 + (4 * ((-4 + (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {4,+,4}<nuw><nsw><%loop> U: [4,17) S: [4,17) Exits: (4 + (4 * ((-4 + (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @rewrite_zext_ult_narrow_check
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is ((-4 + (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)
-; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 4611686018427387903
+; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 3
 ; CHECK-NEXT:  Loop %loop: symbolic max backedge-taken count is ((-4 + (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:  Loop %loop: Predicated backedge-taken count is ((-4 + (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:   Predicates:
@@ -568,14 +568,14 @@ define i32 @rewrite_zext_ule_narrow_check(i32 %N, ptr %arr) {
 ; CHECK-NEXT:    %n.vec = and i64 %ext, 28
 ; CHECK-NEXT:    --> (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw> U: [0,29) S: [0,29)
 ; CHECK-NEXT:    %index = phi i64 [ 0, %loop.ph ], [ %index.next, %loop ]
-; CHECK-NEXT:    --> {0,+,4}<nuw><%loop> U: [0,-3) S: [-9223372036854775808,9223372036854775805) Exits: (4 * ((-4 + (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,4}<nuw><nsw><%loop> U: [0,13) S: [0,13) Exits: (4 * ((-4 + (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %gep = getelementptr inbounds i32, ptr %arr, i64 %index
-; CHECK-NEXT:    --> {%arr,+,16}<%loop> U: full-set S: full-set Exits: ((16 * ((-4 + (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)) + %arr) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {%arr,+,16}<nuw><%loop> U: full-set S: full-set Exits: ((16 * ((-4 + (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)) + %arr) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %index.next = add nuw i64 %index, 4
-; CHECK-NEXT:    --> {4,+,4}<nuw><%loop> U: [4,-3) S: [-9223372036854775808,9223372036854775805) Exits: (4 + (4 * ((-4 + (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {4,+,4}<nuw><nsw><%loop> U: [4,17) S: [4,17) Exits: (4 + (4 * ((-4 + (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @rewrite_zext_ule_narrow_check
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is ((-4 + (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)
-; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 4611686018427387903
+; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 3
 ; CHECK-NEXT:  Loop %loop: symbolic max backedge-taken count is ((-4 + (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:  Loop %loop: Predicated backedge-taken count is ((-4 + (4 * (zext i3 (trunc i64 ((4 umax (zext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:   Predicates:
@@ -614,14 +614,14 @@ define i32 @rewrite_zext_sle_narrow_check(i32 %N, ptr %arr) {
 ; CHECK-NEXT:    %n.vec = and i64 %ext, 28
 ; CHECK-NEXT:    --> (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw> U: [0,29) S: [0,29)
 ; CHECK-NEXT:    %index = phi i64 [ 0, %loop.ph ], [ %index.next, %loop ]
-; CHECK-NEXT:    --> {0,+,4}<nuw><%loop> U: [0,-3) S: [-9223372036854775808,9223372036854775805) Exits: (4 * ((-4 + (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,4}<nuw><nsw><%loop> U: [0,13) S: [0,13) Exits: (4 * ((-4 + (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %gep = getelementptr inbounds i32, ptr %arr, i64 %index
-; CHECK-NEXT:    --> {%arr,+,16}<%loop> U: full-set S: full-set Exits: ((16 * ((-4 + (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)) + %arr) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {%arr,+,16}<nuw><%loop> U: full-set S: full-set Exits: ((16 * ((-4 + (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)) + %arr) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %index.next = add nuw i64 %index, 4
-; CHECK-NEXT:    --> {4,+,4}<nuw><%loop> U: [4,-3) S: [-9223372036854775808,9223372036854775805) Exits: (4 + (4 * ((-4 + (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {4,+,4}<nuw><nsw><%loop> U: [4,17) S: [4,17) Exits: (4 + (4 * ((-4 + (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @rewrite_zext_sle_narrow_check
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is ((-4 + (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)
-; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 4611686018427387903
+; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 3
 ; CHECK-NEXT:  Loop %loop: symbolic max backedge-taken count is ((-4 + (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:  Loop %loop: Predicated backedge-taken count is ((-4 + (4 * (zext i3 (trunc i64 ((zext i32 (4 smax %N) to i64) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:   Predicates:
@@ -660,14 +660,14 @@ define i32 @rewrite_zext_uge_narrow_check(i32 %N, ptr %arr) {
 ; CHECK-NEXT:    %n.vec = and i64 %ext, 28
 ; CHECK-NEXT:    --> (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw> U: [0,17) S: [0,17)
 ; CHECK-NEXT:    %index = phi i64 [ 0, %loop.ph ], [ %index.next, %loop ]
-; CHECK-NEXT:    --> {0,+,4}<nuw><%loop> U: [0,-3) S: [-9223372036854775808,9223372036854775805) Exits: (4 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,4}<nuw><nsw><%loop> U: [0,13) S: [0,13) Exits: (4 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %gep = getelementptr inbounds i32, ptr %arr, i64 %index
-; CHECK-NEXT:    --> {%arr,+,16}<%loop> U: full-set S: full-set Exits: ((16 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4)) + %arr) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {%arr,+,16}<nuw><%loop> U: full-set S: full-set Exits: ((16 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4)) + %arr) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %index.next = add nuw i64 %index, 4
-; CHECK-NEXT:    --> {4,+,4}<nuw><%loop> U: [4,-3) S: [-9223372036854775808,9223372036854775805) Exits: (4 + (4 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {4,+,4}<nuw><nsw><%loop> U: [4,17) S: [4,17) Exits: (4 + (4 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @rewrite_zext_uge_narrow_check
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4)
-; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 4611686018427387903
+; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 3
 ; CHECK-NEXT:  Loop %loop: symbolic max backedge-taken count is ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:  Loop %loop: Predicated backedge-taken count is ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:   Predicates:
@@ -706,14 +706,14 @@ define i32 @rewrite_sext_sge_narrow_check(i32 %N, ptr %arr) {
 ; CHECK-NEXT:    %n.vec = and i64 %ext, 28
 ; CHECK-NEXT:    --> (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw> U: [0,29) S: [0,29)
 ; CHECK-NEXT:    %index = phi i64 [ 0, %loop.ph ], [ %index.next, %loop ]
-; CHECK-NEXT:    --> {0,+,4}<nuw><%loop> U: [0,-3) S: [-9223372036854775808,9223372036854775805) Exits: (4 * ((-4 + (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,4}<nuw><nsw><%loop> U: [0,13) S: [0,13) Exits: (4 * ((-4 + (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %gep = getelementptr inbounds i32, ptr %arr, i64 %index
-; CHECK-NEXT:    --> {%arr,+,16}<%loop> U: full-set S: full-set Exits: ((16 * ((-4 + (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)) + %arr) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {%arr,+,16}<nuw><%loop> U: full-set S: full-set Exits: ((16 * ((-4 + (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)) + %arr) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %index.next = add nuw i64 %index, 4
-; CHECK-NEXT:    --> {4,+,4}<nuw><%loop> U: [4,-3) S: [-9223372036854775808,9223372036854775805) Exits: (4 + (4 * ((-4 + (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {4,+,4}<nuw><nsw><%loop> U: [4,17) S: [4,17) Exits: (4 + (4 * ((-4 + (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @rewrite_sext_sge_narrow_check
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is ((-4 + (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)
-; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 4611686018427387903
+; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 3
 ; CHECK-NEXT:  Loop %loop: symbolic max backedge-taken count is ((-4 + (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:  Loop %loop: Predicated backedge-taken count is ((-4 + (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:   Predicates:
@@ -752,14 +752,14 @@ define i32 @rewrite_zext_ugt_narrow_check(i32 %N, ptr %arr) {
 ; CHECK-NEXT:    %n.vec = and i64 %ext, 28
 ; CHECK-NEXT:    --> (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw> U: [0,17) S: [0,17)
 ; CHECK-NEXT:    %index = phi i64 [ 0, %loop.ph ], [ %index.next, %loop ]
-; CHECK-NEXT:    --> {0,+,4}<nuw><%loop> U: [0,-3) S: [-9223372036854775808,9223372036854775805) Exits: (4 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,4}<nuw><nsw><%loop> U: [0,13) S: [0,13) Exits: (4 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %gep = getelementptr inbounds i32, ptr %arr, i64 %index
-; CHECK-NEXT:    --> {%arr,+,16}<%loop> U: full-set S: full-set Exits: ((16 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4)) + %arr) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {%arr,+,16}<nuw><%loop> U: full-set S: full-set Exits: ((16 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4)) + %arr) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %index.next = add nuw i64 %index, 4
-; CHECK-NEXT:    --> {4,+,4}<nuw><%loop> U: [4,-3) S: [-9223372036854775808,9223372036854775805) Exits: (4 + (4 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {4,+,4}<nuw><nsw><%loop> U: [4,17) S: [4,17) Exits: (4 + (4 * ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @rewrite_zext_ugt_narrow_check
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4)
-; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 4611686018427387903
+; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 3
 ; CHECK-NEXT:  Loop %loop: symbolic max backedge-taken count is ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:  Loop %loop: Predicated backedge-taken count is ((-4 + (4 * ((16 umin (zext i32 %N to i64)) /u 4))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:   Predicates:
@@ -798,14 +798,14 @@ define i32 @rewrite_sext_sgt_narrow_check(i32 %N, ptr %arr) {
 ; CHECK-NEXT:    %n.vec = and i64 %ext, 28
 ; CHECK-NEXT:    --> (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw> U: [0,29) S: [0,29)
 ; CHECK-NEXT:    %index = phi i64 [ 0, %loop.ph ], [ %index.next, %loop ]
-; CHECK-NEXT:    --> {0,+,4}<nuw><%loop> U: [0,-3) S: [-9223372036854775808,9223372036854775805) Exits: (4 * ((-4 + (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,4}<nuw><nsw><%loop> U: [0,13) S: [0,13) Exits: (4 * ((-4 + (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %gep = getelementptr inbounds i32, ptr %arr, i64 %index
-; CHECK-NEXT:    --> {%arr,+,16}<%loop> U: full-set S: full-set Exits: ((16 * ((-4 + (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)) + %arr) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {%arr,+,16}<nuw><%loop> U: full-set S: full-set Exits: ((16 * ((-4 + (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)) + %arr) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %index.next = add nuw i64 %index, 4
-; CHECK-NEXT:    --> {4,+,4}<nuw><%loop> U: [4,-3) S: [-9223372036854775808,9223372036854775805) Exits: (4 + (4 * ((-4 + (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {4,+,4}<nuw><nsw><%loop> U: [4,17) S: [4,17) Exits: (4 + (4 * ((-4 + (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4))<nuw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @rewrite_sext_sgt_narrow_check
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is ((-4 + (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)
-; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 4611686018427387903
+; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 3
 ; CHECK-NEXT:  Loop %loop: symbolic max backedge-taken count is ((-4 + (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:  Loop %loop: Predicated backedge-taken count is ((-4 + (4 * (zext i3 (trunc i64 ((16 smin (sext i32 %N to i64)) /u 4) to i3) to i64))<nuw><nsw>)<nsw> /u 4)
 ; CHECK-NEXT:   Predicates:
diff --git a/llvm/test/Analysis/ScalarEvolution/max-backedge-taken-count-guard-info.ll b/llvm/test/Analysis/ScalarEvolution/max-backedge-taken-count-guard-info.ll
index 161ac6d3fc0e3..3f900eeecd34a 100644
--- a/llvm/test/Analysis/ScalarEvolution/max-backedge-taken-count-guard-info.ll
+++ b/llvm/test/Analysis/ScalarEvolution/max-backedge-taken-count-guard-info.ll
@@ -424,11 +424,11 @@ define void @test_guard_ult_ne(ptr nocapture readonly %data, i64 %count) {
 ; CHECK-LABEL: 'test_guard_ult_ne'
 ; CHECK-NEXT:  Classifying expressions for: @test_guard_ult_ne
 ; CHECK-NEXT:    %iv = phi i64 [ %iv.next, %loop ], [ 0, %guardbb ]
-; CHECK-NEXT:    --> {0,+,1}<nuw><%loop> U: [0,4) S: [0,4) Exits: (-1 + %count) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,4) S: [0,4) Exits: (-1 + %count) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %idx = getelementptr inbounds i32, ptr %data, i64 %iv
 ; CHECK-NEXT:    --> {%data,+,4}<nuw><%loop> U: full-set S: full-set Exits: (-4 + (4 * %count) + %data) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add nuw i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<nuw><%loop> U: [1,5) S: [1,5) Exits: %count LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,5) S: [1,5) Exits: %count LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_guard_ult_ne
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (-1 + %count)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 3
@@ -461,11 +461,11 @@ define void @test_guard_ne_ult(ptr nocapture readonly %data, i64 %count) {
 ; CHECK-LABEL: 'test_guard_ne_ult'
 ; CHECK-NEXT:  Classifying expressions for: @test_guard_ne_ult
 ; CHECK-NEXT:    %iv = phi i64 [ %iv.next, %loop ], [ 0, %guardbb ]
-; CHECK-NEXT:    --> {0,+,1}<nuw><%loop> U: [0,4) S: [0,4) Exits: (-1 + %count) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,4) S: [0,4) Exits: (-1 + %count) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %idx = getelementptr inbounds i32, ptr %data, i64 %iv
 ; CHECK-NEXT:    --> {%data,+,4}<nuw><%loop> U: full-set S: full-set Exits: (-4 + (4 * %count) + %data) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add nuw i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<nuw><%loop> U: [1,5) S: [1,5) Exits: %count LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,5) S: [1,5) Exits: %count LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_guard_ne_ult
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (-1 + %count)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 3
@@ -500,11 +500,11 @@ define void @test_guard_if_and_enter(ptr nocapture readonly %data, i64 %count) {
 ; CHECK-NEXT:    %cmp.and = and i1 %cmp.ult, %cmp.ne
 ; CHECK-NEXT:    --> (%cmp.ult umin %cmp.ne) U: full-set S: full-set
 ; CHECK-NEXT:    %iv = phi i64 [ %iv.next, %loop ], [ 0, %entry ]
-; CHECK-NEXT:    --> {0,+,1}<nuw><%loop> U: [0,4) S: [0,4) Exits: (-1 + %count) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,4) S: [0,4) Exits: (-1 + %count) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %idx = getelementptr inbounds i32, ptr %data, i64 %iv
 ; CHECK-NEXT:    --> {%data,+,4}<nuw><%loop> U: full-set S: full-set Exits: (-4 + (4 * %count) + %data) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add nuw i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<nuw><%loop> U: [1,5) S: [1,5) Exits: %count LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,5) S: [1,5) Exits: %count LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_guard_if_and_enter
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (-1 + %count)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 3
@@ -576,11 +576,11 @@ define void @test_guard_if_and_and(ptr nocapture readonly %data, i64 %count, i1
 ; CHECK-NEXT:    %cmp.and = and i1 %cmp.ult, %cmp.and1
 ; CHECK-NEXT:    --> (%c umin %cmp.ult umin %cmp.ne) U: full-set S: full-set
 ; CHECK-NEXT:    %iv = phi i64 [ %iv.next, %loop ], [ 0, %entry ]
-; CHECK-NEXT:    --> {0,+,1}<nuw><%loop> U: [0,4) S: [0,4) Exits: (-1 + %count) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,4) S: [0,4) Exits: (-1 + %count) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %idx = getelementptr inbounds i32, ptr %data, i64 %iv
 ; CHECK-NEXT:    --> {%data,+,4}<nuw><%loop> U: full-set S: full-set Exits: (-4 + (4 * %count) + %data) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add nuw i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<nuw><%loop> U: [1,5) S: [1,5) Exits: %count LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,5) S: [1,5) Exits: %count LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_guard_if_and_and
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (-1 + %count)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 3
@@ -654,11 +654,11 @@ define void @test_guard_if_or_skip(ptr nocapture readonly %data, i64 %count) {
 ; CHECK-NEXT:    %cmp.or = or i1 %cmp.uge, %cmp.eq
 ; CHECK-NEXT:    --> (%cmp.uge umax %cmp.eq) U: full-set S: full-set
 ; CHECK-NEXT:    %iv = phi i64 [ %iv.next, %loop ], [ 0, %entry ]
-; CHECK-NEXT:    --> {0,+,1}<nuw><%loop> U: [0,4) S: [0,4) Exits: (-1 + %count) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,4) S: [0,4) Exits: (-1 + %count) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %idx = getelementptr inbounds i32, ptr %data, i64 %iv
 ; CHECK-NEXT:    --> {%data,+,4}<nuw><%loop> U: full-set S: full-set Exits: (-4 + (4 * %count) + %data) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add nuw i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<nuw><%loop> U: [1,5) S: [1,5) Exits: %count LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,5) S: [1,5) Exits: %count LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_guard_if_or_skip
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (-1 + %count)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 3
@@ -730,11 +730,11 @@ define void @test_guard_if_or_or(ptr nocapture readonly %data, i64 %count, i1 %c
 ; CHECK-NEXT:    %cmp.or = or i1 %cmp.uge, %cmp.or1
 ; CHECK-NEXT:    --> (%c umax %cmp.uge umax %cmp.eq) U: full-set S: full-set
 ; CHECK-NEXT:    %iv = phi i64 [ %iv.next, %loop ], [ 0, %entry ]
-; CHECK-NEXT:    --> {0,+,1}<nuw><%loop> U: [0,4) S: [0,4) Exits: (-1 + %count) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,4) S: [0,4) Exits: (-1 + %count) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %idx = getelementptr inbounds i32, ptr %data, i64 %iv
 ; CHECK-NEXT:    --> {%data,+,4}<nuw><%loop> U: full-set S: full-set Exits: (-4 + (4 * %count) + %data) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add nuw i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<nuw><%loop> U: [1,5) S: [1,5) Exits: %count LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,5) S: [1,5) Exits: %count LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_guard_if_or_or
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (-1 + %count)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 3
@@ -809,11 +809,11 @@ define void @test_guard_and_assume(ptr nocapture readonly %data, i64 %count) {
 ; CHECK-LABEL: 'test_guard_and_assume'
 ; CHECK-NEXT:  Classifying expressions for: @test_guard_and_assume
 ; CHECK-NEXT:    %iv = phi i64 [ %iv.next, %loop ], [ 0, %entry ]
-; CHECK-NEXT:    --> {0,+,1}<nuw><%loop> U: [0,4) S: [0,4) Exits: (-1 + %count) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,4) S: [0,4) Exits: (-1 + %count) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %idx = getelementptr inbounds i32, ptr %data, i64 %iv
 ; CHECK-NEXT:    --> {%data,+,4}<nuw><%loop> U: full-set S: full-set Exits: (-4 + (4 * %count) + %data) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add nuw i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<nuw><%loop> U: [1,5) S: [1,5) Exits: %count LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,5) S: [1,5) Exits: %count LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_guard_and_assume
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (-1 + %count)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 3
@@ -846,11 +846,11 @@ define void @test_guard_assume_and(ptr nocapture readonly %data, i64 %count) {
 ; CHECK-NEXT:    %cmp.and = and i1 %cmp.ult, %cmp.ne
 ; CHECK-NEXT:    --> (%cmp.ult umin %cmp.ne) U: full-set S: full-set
 ; CHECK-NEXT:    %iv = phi i64 [ %iv.next, %loop ], [ 0, %entry ]
-; CHECK-NEXT:    --> {0,+,1}<nuw><%loop> U: [0,4) S: [0,4) Exits: (-1 + %count) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,4) S: [0,4) Exits: (-1 + %count) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %idx = getelementptr inbounds i32, ptr %data, i64 %iv
 ; CHECK-NEXT:    --> {%data,+,4}<nuw><%loop> U: full-set S: full-set Exits: (-4 + (4 * %count) + %data) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add nuw i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<nuw><%loop> U: [1,5) S: [1,5) Exits: %count LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,5) S: [1,5) Exits: %count LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_guard_assume_and
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (-1 + %count)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 3
@@ -887,9 +887,9 @@ define void @guard_pessimizes_analysis_step1(i1 %c, i32 %N) {
 ; CHECK-NEXT:    %init = phi i32 [ 2, %entry ], [ 3, %bb1 ]
 ; CHECK-NEXT:    --> %init U: [2,4) S: [2,4)
 ; CHECK-NEXT:    %iv = phi i32 [ %iv.next, %loop ], [ %init, %loop.ph ]
-; CHECK-NEXT:    --> {%init,+,1}<%loop> U: [2,11) S: [2,11) Exits: 9 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {%init,+,1}<nuw><nsw><%loop> U: [2,11) S: [2,11) Exits: 9 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i32 %iv, 1
-; CHECK-NEXT:    --> {(1 + %init)<nuw><nsw>,+,1}<%loop> U: [3,12) S: [3,12) Exits: 10 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {(1 + %init)<nuw><nsw>,+,1}<nuw><nsw><%loop> U: [3,12) S: [3,12) Exits: 10 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @guard_pessimizes_analysis_step1
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (9 + (-1 * %init)<nsw>)<nsw>
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 7
@@ -1024,9 +1024,9 @@ define void @test_guard_uge(i32 %blockSize) {
 ; CHECK-NEXT:    %shr = lshr i32 %blockSize, 2
 ; CHECK-NEXT:    --> (%blockSize /u 4) U: [0,1073741824) S: [0,1073741824)
 ; CHECK-NEXT:    %iv = phi i32 [ %dec, %while.body ], [ %shr, %while.body.preheader ]
-; CHECK-NEXT:    --> {(%blockSize /u 4),+,-1}<%while.body> U: [-1073741822,1073741824) S: [-1073741822,1073741824) Exits: 1 LoopDispositions: { %while.body: Computable }
+; CHECK-NEXT:    --> {(%blockSize /u 4),+,-1}<nsw><%while.body> U: [-1073741822,1073741824) S: [-1073741822,1073741824) Exits: 1 LoopDispositions: { %while.body: Computable }
 ; CHECK-NEXT:    %dec = add i32 %iv, -1
-; CHECK-NEXT:    --> {(-1 + (%blockSize /u 4))<nsw>,+,-1}<%while.body> U: [-1073741823,1073741823) S: [-1073741823,1073741823) Exits: 0 LoopDispositions: { %while.body: Computable }
+; CHECK-NEXT:    --> {(-1 + (%blockSize /u 4))<nsw>,+,-1}<nsw><%while.body> U: [-1073741823,1073741823) S: [-1073741823,1073741823) Exits: 0 LoopDispositions: { %while.body: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_guard_uge
 ; CHECK-NEXT:  Loop %while.body: backedge-taken count is (-1 + (%blockSize /u 4))<nsw>
 ; CHECK-NEXT:  Loop %while.body: constant max backedge-taken count is 1073741822
@@ -1061,9 +1061,9 @@ define void @test_guard_ugt(i32 %blockSize) {
 ; CHECK-NEXT:    %shr = lshr i32 %blockSize, 2
 ; CHECK-NEXT:    --> (%blockSize /u 4) U: [0,1073741824) S: [0,1073741824)
 ; CHECK-NEXT:    %iv = phi i32 [ %dec, %while.body ], [ %shr, %while.body.preheader ]
-; CHECK-NEXT:    --> {(%blockSize /u 4),+,-1}<%while.body> U: [-1073741822,1073741824) S: [-1073741822,1073741824) Exits: 1 LoopDispositions: { %while.body: Computable }
+; CHECK-NEXT:    --> {(%blockSize /u 4),+,-1}<nsw><%while.body> U: [-1073741822,1073741824) S: [-1073741822,1073741824) Exits: 1 LoopDispositions: { %while.body: Computable }
 ; CHECK-NEXT:    %dec = add i32 %iv, -1
-; CHECK-NEXT:    --> {(-1 + (%blockSize /u 4))<nsw>,+,-1}<%while.body> U: [-1073741823,1073741823) S: [-1073741823,1073741823) Exits: 0 LoopDispositions: { %while.body: Computable }
+; CHECK-NEXT:    --> {(-1 + (%blockSize /u 4))<nsw>,+,-1}<nsw><%while.body> U: [-1073741823,1073741823) S: [-1073741823,1073741823) Exits: 0 LoopDispositions: { %while.body: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_guard_ugt
 ; CHECK-NEXT:  Loop %while.body: backedge-taken count is (-1 + (%blockSize /u 4))<nsw>
 ; CHECK-NEXT:  Loop %while.body: constant max backedge-taken count is 1073741822
@@ -1098,9 +1098,9 @@ define void @test_guard_uge_and_ule(i32 %blockSize) {
 ; CHECK-NEXT:    %shr = lshr i32 %blockSize, 2
 ; CHECK-NEXT:    --> (%blockSize /u 4) U: [0,1073741824) S: [0,1073741824)
 ; CHECK-NEXT:    %iv = phi i32 [ %dec, %while.body ], [ %shr, %while.body.preheader ]
-; CHECK-NEXT:    --> {(%blockSize /u 4),+,-1}<%while.body> U: [-255,1073741824) S: [-255,1073741824) Exits: 1 LoopDispositions: { %while.body: Computable }
+; CHECK-NEXT:    --> {(%blockSize /u 4),+,-1}<nsw><%while.body> U: [-255,1073741824) S: [-255,1073741824) Exits: 1 LoopDispositions: { %while.body: Computable }
 ; CHECK-NEXT:    %dec = add i32 %iv, -1
-; CHECK-NEXT:    --> {(-1 + (%blockSize /u 4))<nsw>,+,-1}<%while.body> U: [-256,1073741823) S: [-256,1073741823) Exits: 0 LoopDispositions: { %while.body: Computable }
+; CHECK-NEXT:    --> {(-1 + (%blockSize /u 4))<nsw>,+,-1}<nsw><%while.body> U: [-256,1073741823) S: [-256,1073741823) Exits: 0 LoopDispositions: { %while.body: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_guard_uge_and_ule
 ; CHECK-NEXT:  Loop %while.body: backedge-taken count is (-1 + (%blockSize /u 4))<nsw>
 ; CHECK-NEXT:  Loop %while.body: constant max backedge-taken count is 255
@@ -1139,9 +1139,9 @@ define void @test_guard_ugt_and_ult(i32 %blockSize) {
 ; CHECK-NEXT:    %shr = lshr i32 %blockSize, 2
 ; CHECK-NEXT:    --> (%blockSize /u 4) U: [0,1073741824) S: [0,1073741824)
 ; CHECK-NEXT:    %iv = phi i32 [ %dec, %while.body ], [ %shr, %while.body.preheader ]
-; CHECK-NEXT:    --> {(%blockSize /u 4),+,-1}<%while.body> U: [-255,1073741824) S: [-255,1073741824) Exits: 1 LoopDispositions: { %while.body: Computable }
+; CHECK-NEXT:    --> {(%blockSize /u 4),+,-1}<nsw><%while.body> U: [-255,1073741824) S: [-255,1073741824) Exits: 1 LoopDispositions: { %while.body: Computable }
 ; CHECK-NEXT:    %dec = add i32 %iv, -1
-; CHECK-NEXT:    --> {(-1 + (%blockSize /u 4))<nsw>,+,-1}<%while.body> U: [-256,1073741823) S: [-256,1073741823) Exits: 0 LoopDispositions: { %while.body: Computable }
+; CHECK-NEXT:    --> {(-1 + (%blockSize /u 4))<nsw>,+,-1}<nsw><%while.body> U: [-256,1073741823) S: [-256,1073741823) Exits: 0 LoopDispositions: { %while.body: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_guard_ugt_and_ult
 ; CHECK-NEXT:  Loop %while.body: backedge-taken count is (-1 + (%blockSize /u 4))<nsw>
 ; CHECK-NEXT:  Loop %while.body: constant max backedge-taken count is 255
@@ -1558,9 +1558,9 @@ define i32 @sle_sgt_ult_umax_to_smax(i32 %num) {
 ; CHECK-LABEL: 'sle_sgt_ult_umax_to_smax'
 ; CHECK-NEXT:  Classifying expressions for: @sle_sgt_ult_umax_to_smax
 ; CHECK-NEXT:    %iv = phi i32 [ 0, %guard.3 ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,4}<nuw><%loop> U: [0,25) S: [0,25) Exits: (4 * ((-4 + %num) /u 4))<nuw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,4}<nuw><nsw><%loop> U: [0,25) S: [0,25) Exits: (4 * ((-4 + %num) /u 4))<nuw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add nuw i32 %iv, 4
-; CHECK-NEXT:    --> {4,+,4}<nuw><%loop> U: [4,29) S: [4,29) Exits: (4 + (4 * ((-4 + %num) /u 4))<nuw>) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {4,+,4}<nuw><nsw><%loop> U: [4,29) S: [4,29) Exits: (4 + (4 * ((-4 + %num) /u 4))<nuw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @sle_sgt_ult_umax_to_smax
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is ((-4 + %num) /u 4)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 6
diff --git a/llvm/test/Analysis/ScalarEvolution/max-trip-count.ll b/llvm/test/Analysis/ScalarEvolution/max-trip-count.ll
index eab605e08d2b2..1f8746d0a4a27 100644
--- a/llvm/test/Analysis/ScalarEvolution/max-trip-count.ll
+++ b/llvm/test/Analysis/ScalarEvolution/max-trip-count.ll
@@ -342,7 +342,7 @@ loop.exit:
 define void @changing_end_bound3(i32 %start, ptr %n_addr, ptr %addr) {
 ; CHECK-LABEL: Determining loop execution counts for: @changing_end_bound3
 ; CHECK: Loop %loop: Unpredictable backedge-taken count.
-; CHECK: Loop %loop: constant max backedge-taken count is 1073741823
+; CHECK: Loop %loop: constant max backedge-taken count is 1073741822
 entry:
   br label %loop
 
diff --git a/llvm/test/Analysis/ScalarEvolution/nsw.ll b/llvm/test/Analysis/ScalarEvolution/nsw.ll
index 876b7cec4d9b6..5b2378502c521 100644
--- a/llvm/test/Analysis/ScalarEvolution/nsw.ll
+++ b/llvm/test/Analysis/ScalarEvolution/nsw.ll
@@ -282,16 +282,16 @@ define void @test4(i32 %arg) {
 ; CHECK-NEXT:    %sub = add nsw i32 %index, -2
 ; CHECK-NEXT:    --> {(-2 + %arg)<nsw>,+,1}<nsw><%for.body> U: full-set S: full-set Exits: (-3 + (10 smax (1 + %arg)<nsw>))<nsw> LoopDispositions: { %for.body: Computable }
 ; CHECK-NEXT:    %idxprom = sext i32 %sub to i64
-; CHECK-NEXT:    --> {(-2 + (sext i32 %arg to i64))<nsw>,+,1}<nsw><%for.body> U: [-2147483650,4294967304) S: [-2147483650,4294967304) Exits: (-2 + (zext i32 (-1 + (-1 * %arg) + (10 smax (1 + %arg)<nsw>)) to i64) + (sext i32 %arg to i64)) LoopDispositions: { %for.body: Computable }
+; CHECK-NEXT:    --> {(-2 + (sext i32 %arg to i64))<nsw>,+,1}<nsw><%for.body> U: [-2147483650,4294967303) S: [-2147483650,4294967303) Exits: (-2 + (zext i32 (-1 + (-1 * %arg) + (10 smax (1 + %arg)<nsw>)) to i64) + (sext i32 %arg to i64)) LoopDispositions: { %for.body: Computable }
 ; CHECK-NEXT:    %arrayidx = getelementptr inbounds [10 x i32], ptr %array, i64 0, i64 %idxprom
 ; CHECK-NEXT:    --> {(-8 + (4 * (sext i32 %arg to i64))<nsw> + %array),+,4}<nw><%for.body> U: [0,-3) S: [-9223372036854775808,9223372036854775805) Exits: (-8 + (4 * (zext i32 (-1 + (-1 * %arg) + (10 smax (1 + %arg)<nsw>)) to i64))<nuw><nsw> + (4 * (sext i32 %arg to i64))<nsw> + %array) LoopDispositions: { %for.body: Computable }
 ; CHECK-NEXT:    %data = load i32, ptr %arrayidx, align 4
 ; CHECK-NEXT:    --> %data U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %for.body: Variant }
 ; CHECK-NEXT:    %inc5 = add nsw i32 %index, 1
-; CHECK-NEXT:    --> {(1 + %arg)<nsw>,+,1}<nsw><%for.body> U: full-set S: full-set Exits: (10 smax (1 + %arg)<nsw>) LoopDispositions: { %for.body: Computable }
+; CHECK-NEXT:    --> {(1 + %arg)<nsw>,+,1}<nsw><%for.body> U: [-2147483647,-2147483648) S: [-2147483647,-2147483648) Exits: (10 smax (1 + %arg)<nsw>) LoopDispositions: { %for.body: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @test4
 ; CHECK-NEXT:  Loop %for.body: backedge-taken count is (-1 + (-1 * %arg) + (10 smax (1 + %arg)<nsw>))
-; CHECK-NEXT:  Loop %for.body: constant max backedge-taken count is -2147483638
+; CHECK-NEXT:  Loop %for.body: constant max backedge-taken count is -2147483639
 ; CHECK-NEXT:  Loop %for.body: symbolic max backedge-taken count is (-1 + (-1 * %arg) + (10 smax (1 + %arg)<nsw>))
 ; CHECK-NEXT:  Loop %for.body: Predicated backedge-taken count is (-1 + (-1 * %arg) + (10 smax (1 + %arg)<nsw>))
 ; CHECK-NEXT:   Predicates:
diff --git a/llvm/test/Analysis/ScalarEvolution/overflow-intrinsics.ll b/llvm/test/Analysis/ScalarEvolution/overflow-intrinsics.ll
index cc24410a111b8..cd740531f8335 100644
--- a/llvm/test/Analysis/ScalarEvolution/overflow-intrinsics.ll
+++ b/llvm/test/Analysis/ScalarEvolution/overflow-intrinsics.ll
@@ -44,7 +44,7 @@ for.cond.cleanup:                                 ; preds = %cont
 
 for.body:                                         ; preds = %entry, %cont
 ; CHECK:  %i.04 = phi i32 [ 0, %entry ], [ %tmp2, %cont ]
-; CHECK-NEXT:  -->  {0,+,1}<%for.body> U: [0,16) S: [0,16)
+; CHECK-NEXT:  -->  {0,+,1}<nuw><nsw><%for.body> U: [0,16) S: [0,16)
 
 ; SCEV can prove <nsw> for the above induction variable; but it does
 ; not bother so before it sees the sext below since it is not a 100%
@@ -198,7 +198,7 @@ for.cond.cleanup:                                 ; preds = %cont
 
 for.body:                                         ; preds = %entry, %cont
 ; CHECK:  %i.04 = phi i32 [ 0, %entry ], [ %tmp2, %cont ]
-; CHECK-NEXT:  -->  {0,+,1}<nuw><%for.body> U: [0,16) S: [0,16)
+; CHECK-NEXT:  -->  {0,+,1}<nuw><nsw><%for.body> U: [0,16) S: [0,16)
 
   %i.04 = phi i32 [ 0, %entry ], [ %tmp2, %cont ]
   %idxprom = sext i32 %i.04 to i64
@@ -229,7 +229,7 @@ for.cond.cleanup:                                 ; preds = %cont
 
 for.body:                                         ; preds = %entry, %cont
 ; CHECK:  %i.04 = phi i32 [ 15, %entry ], [ %tmp2, %cont ]
-; CHECK-NEXT:  -->  {15,+,-1}<%for.body> U: [0,16) S: [0,16)
+; CHECK-NEXT:  -->  {15,+,-1}<nsw><%for.body> U: [0,16) S: [0,16)
 
   %i.04 = phi i32 [ 15, %entry ], [ %tmp2, %cont ]
   %idxprom = sext i32 %i.04 to i64
@@ -260,7 +260,7 @@ for.cond.cleanup:                                 ; preds = %cont
 
 for.body:                                         ; preds = %entry, %cont
 ; CHECK:  %i.04 = phi i32 [ 15, %entry ], [ %tmp2, %cont ]
-; CHECK-NEXT:  -->  {15,+,-1}<%for.body> U: [0,16) S: [0,16)
+; CHECK-NEXT:  -->  {15,+,-1}<nsw><%for.body> U: [0,16) S: [0,16)
 
   %i.04 = phi i32 [ 15, %entry ], [ %tmp2, %cont ]
   %idxprom = sext i32 %i.04 to i64
diff --git a/llvm/test/Analysis/ScalarEvolution/range-signedness.ll b/llvm/test/Analysis/ScalarEvolution/range-signedness.ll
index 43c62338f175a..7022892dea715 100644
--- a/llvm/test/Analysis/ScalarEvolution/range-signedness.ll
+++ b/llvm/test/Analysis/ScalarEvolution/range-signedness.ll
@@ -35,9 +35,9 @@ define void @y(ptr %addr) {
 ; CHECK-LABEL: 'y'
 ; CHECK-NEXT:  Classifying expressions for: @y
 ; CHECK-NEXT:    %idx = phi i8 [ -5, %entry ], [ %idx.inc, %loop ]
-; CHECK-NEXT:    --> {-5,+,1}<%loop> U: [-5,6) S: [-5,6) Exits: 5 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {-5,+,1}<nsw><%loop> U: [-5,6) S: [-5,6) Exits: 5 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %idx.inc = add i8 %idx, 1
-; CHECK-NEXT:    --> {-4,+,1}<%loop> U: [-4,7) S: [-4,7) Exits: 6 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {-4,+,1}<nsw><%loop> U: [-4,7) S: [-4,7) Exits: 6 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @y
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is 10
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 10
diff --git a/llvm/test/Analysis/ScalarEvolution/scalable-vector.ll b/llvm/test/Analysis/ScalarEvolution/scalable-vector.ll
index b5bd724cb269c..13648ebdaf868 100644
--- a/llvm/test/Analysis/ScalarEvolution/scalable-vector.ll
+++ b/llvm/test/Analysis/ScalarEvolution/scalable-vector.ll
@@ -1,16 +1,76 @@
 ; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py
 ; RUN: opt "-passes=print<scalar-evolution>" -disable-output < %s 2>&1 | FileCheck %s
 
-define void @a(ptr %p) {
-; CHECK-LABEL: 'a'
-; CHECK-NEXT:  Classifying expressions for: @a
+define void @vscale_gep(ptr %p) {
+; CHECK-LABEL: 'vscale_gep'
+; CHECK-NEXT:  Classifying expressions for: @vscale_gep
 ; CHECK-NEXT:    %1 = getelementptr <vscale x 4 x i32>, ptr null, i32 3
 ; CHECK-NEXT:    --> ((48 * vscale) + null) U: [0,-15) S: [-9223372036854775808,9223372036854775793)
 ; CHECK-NEXT:    %2 = getelementptr <vscale x 1 x i64>, ptr %p, i32 1
 ; CHECK-NEXT:    --> ((8 * vscale) + %p) U: full-set S: full-set
-; CHECK-NEXT:  Determining loop execution counts for: @a
+; CHECK-NEXT:  Determining loop execution counts for: @vscale_gep
 ;
   getelementptr <vscale x 4 x i32>, ptr null, i32 3
   getelementptr <vscale x 1 x i64>, ptr %p, i32 1
   ret void
 }
+
+define void @vscale_gep_range(ptr %p) vscale_range(2, 16) {
+; CHECK-LABEL: 'vscale_gep_range'
+; CHECK-NEXT:  Classifying expressions for: @vscale_gep_range
+; CHECK-NEXT:    %1 = getelementptr <vscale x 4 x i32>, ptr null, i32 3
+; CHECK-NEXT:    --> ((48 * vscale)<nuw><nsw> + null) U: [96,769) S: [96,769)
+; CHECK-NEXT:    %2 = getelementptr <vscale x 1 x i64>, ptr %p, i32 1
+; CHECK-NEXT:    --> ((8 * vscale)<nuw><nsw> + %p) U: full-set S: full-set
+; CHECK-NEXT:  Determining loop execution counts for: @vscale_gep_range
+;
+  getelementptr <vscale x 4 x i32>, ptr null, i32 3
+  getelementptr <vscale x 1 x i64>, ptr %p, i32 1
+  ret void
+}
+
+define i64 @vscale_no_range() {
+; CHECK-LABEL: 'vscale_no_range'
+; CHECK-NEXT:  Classifying expressions for: @vscale_no_range
+; CHECK-NEXT:    %vscale = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    --> vscale U: [1,0) S: [1,0)
+; CHECK-NEXT:  Determining loop execution counts for: @vscale_no_range
+;
+  %vscale = call i64 @llvm.vscale.i64()
+  ret i64 %vscale
+}
+
+define i64 @vscale_min_max_range() vscale_range(2, 16) {
+; CHECK-LABEL: 'vscale_min_max_range'
+; CHECK-NEXT:  Classifying expressions for: @vscale_min_max_range
+; CHECK-NEXT:    %vscale = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    --> vscale U: [2,17) S: [2,17)
+; CHECK-NEXT:  Determining loop execution counts for: @vscale_min_max_range
+;
+  %vscale = call i64 @llvm.vscale.i64()
+  ret i64 %vscale
+}
+
+define i64 @vscale_min_range() vscale_range(2, 0) {
+; CHECK-LABEL: 'vscale_min_range'
+; CHECK-NEXT:  Classifying expressions for: @vscale_min_range
+; CHECK-NEXT:    %vscale = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    --> vscale U: [2,0) S: [2,0)
+; CHECK-NEXT:  Determining loop execution counts for: @vscale_min_range
+;
+  %vscale = call i64 @llvm.vscale.i64()
+  ret i64 %vscale
+}
+
+define i64 @vscale_exact_range() vscale_range(2) {
+; CHECK-LABEL: 'vscale_exact_range'
+; CHECK-NEXT:  Classifying expressions for: @vscale_exact_range
+; CHECK-NEXT:    %vscale = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    --> vscale U: [2,3) S: [2,3)
+; CHECK-NEXT:  Determining loop execution counts for: @vscale_exact_range
+;
+  %vscale = call i64 @llvm.vscale.i64()
+  ret i64 %vscale
+}
+
+declare i64 @llvm.vscale.i64()
diff --git a/llvm/test/Analysis/ScalarEvolution/scev-expander-reuse-gep.ll b/llvm/test/Analysis/ScalarEvolution/scev-expander-reuse-gep.ll
index bea8559dd4b75..32d6a958198e6 100644
--- a/llvm/test/Analysis/ScalarEvolution/scev-expander-reuse-gep.ll
+++ b/llvm/test/Analysis/ScalarEvolution/scev-expander-reuse-gep.ll
@@ -3,7 +3,7 @@
 ; RUN: opt -mtriple=i386-apple-macosx10.12.0 < %s -loop-reduce -S | FileCheck %s
 
 ; CHECK: %ptr4.ptr1 = select i1 %cmp.i, ptr %ptr4, ptr %ptr1
-; CHECK-NEXT: %uglygep = getelementptr i8, ptr %ptr4.ptr1, i32 1
+; CHECK-NEXT: %scevgep = getelementptr i8, ptr %ptr4.ptr1, i32 1
 ; CHECK-NEXT: br label %while.cond.i
 
 target datalayout = "e-m:o-p:32:32-f64:32:64-f80:128-n8:16:32-S128"
diff --git a/llvm/test/Analysis/ScalarEvolution/sext-iv-0.ll b/llvm/test/Analysis/ScalarEvolution/sext-iv-0.ll
index 268e9b5cfb6d9..e8284f8e6a747 100644
--- a/llvm/test/Analysis/ScalarEvolution/sext-iv-0.ll
+++ b/llvm/test/Analysis/ScalarEvolution/sext-iv-0.ll
@@ -13,7 +13,7 @@ bb1.thread:
 bb1:		; preds = %bb1, %bb1.thread
 	%i.0.reg2mem.0 = phi i64 [ -128, %bb1.thread ], [ %8, %bb1 ]		; <i64> [#uses=3]
 ; CHECK: %i.0.reg2mem.0
-; CHECK-NEXT: -->  {-128,+,1}<%bb1>{{ U: [^ ]+ S: [^ ]+}}{{ *}}Exits: 127
+; CHECK-NEXT: -->  {-128,+,1}<nsw><%bb1>{{ U: [^ ]+ S: [^ ]+}}{{ *}}Exits: 127
 	%0 = trunc i64 %i.0.reg2mem.0 to i8		; <i8> [#uses=1]
 ; CHECK: %0
 ; CHECK-NEXT: -->  {-128,+,1}<%bb1>{{ U: [^ ]+ S: [^ ]+}}{{ *}}Exits: 127
diff --git a/llvm/test/Analysis/ScalarEvolution/sext-iv-2.ll b/llvm/test/Analysis/ScalarEvolution/sext-iv-2.ll
index 7337f7f787a64..2e87063fa11ac 100644
--- a/llvm/test/Analysis/ScalarEvolution/sext-iv-2.ll
+++ b/llvm/test/Analysis/ScalarEvolution/sext-iv-2.ll
@@ -3,7 +3,7 @@
 ; CHECK: %tmp3 = sext i8 %tmp2 to i32
 ; CHECK: -->  (sext i8 {0,+,1}<%bb1> to i32){{ U: [^ ]+ S: [^ ]+}}{{ *}}Exits: -1
 ; CHECK: %tmp4 = mul i32 %tmp3, %i.02
-; CHECK: -->  ((sext i8 {0,+,1}<%bb1> to i32) * {0,+,1}<%bb>){{ U: [^ ]+ S: [^ ]+}}{{ *}}Exits: {0,+,-1}<%bb>
+; CHECK: -->  ((sext i8 {0,+,1}<%bb1> to i32) * {0,+,1}<nuw><nsw><%bb>){{ U: [^ ]+ S: [^ ]+}}{{ *}}Exits: {0,+,-1}<nw><%bb>
 
 ; These sexts are not foldable.
 
diff --git a/llvm/test/Analysis/ScalarEvolution/sext-to-zext.ll b/llvm/test/Analysis/ScalarEvolution/sext-to-zext.ll
index 645bc23f2e397..2eca58b32483b 100644
--- a/llvm/test/Analysis/ScalarEvolution/sext-to-zext.ll
+++ b/llvm/test/Analysis/ScalarEvolution/sext-to-zext.ll
@@ -11,9 +11,9 @@ define void @f(i1 %c) {
 ; CHECK-NEXT:    %iv = phi i32 [ %start, %entry ], [ %iv.dec, %loop ]
 ; CHECK-NEXT:    --> {%start,+,%step}<nsw><%loop> U: [0,101) S: [0,101) Exits: ((99 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.tc = phi i32 [ 0, %entry ], [ %iv.tc.inc, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,100) S: [0,100) Exits: 99 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,100) S: [0,100) Exits: 99 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.tc.inc = add i32 %iv.tc, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,101) S: [1,101) Exits: 100 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,101) S: [1,101) Exits: 100 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.dec = add nsw i32 %iv, %step
 ; CHECK-NEXT:    --> {(%step + %start),+,%step}<nw><%loop> U: [-200,201) S: [-200,201) Exits: ((100 * %step)<nsw> + %start) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.sext = sext i32 %iv to i64
diff --git a/llvm/test/Analysis/ScalarEvolution/shift-recurrences.ll b/llvm/test/Analysis/ScalarEvolution/shift-recurrences.ll
index b1cdd3b062fad..a7af3e88ca100 100644
--- a/llvm/test/Analysis/ScalarEvolution/shift-recurrences.ll
+++ b/llvm/test/Analysis/ScalarEvolution/shift-recurrences.ll
@@ -198,11 +198,11 @@ define void @test_shl2() {
 ; CHECK-LABEL: 'test_shl2'
 ; CHECK-NEXT:  Classifying expressions for: @test_shl2
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.shl = phi i64 [ 4, %entry ], [ %iv.shl.next, %loop ]
 ; CHECK-NEXT:    --> %iv.shl U: [4,65) S: [4,65) Exits: 64 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.shl.next = shl i64 %iv.shl, 1
 ; CHECK-NEXT:    --> (2 * %iv.shl)<nuw><nsw> U: [8,129) S: [8,129) Exits: 128 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_shl2
@@ -233,11 +233,11 @@ define void @test_shl3(i1 %c) {
 ; CHECK-NEXT:    %shiftamt = select i1 %c, i64 1, i64 0
 ; CHECK-NEXT:    --> %shiftamt U: [0,2) S: [0,2)
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.shl = phi i64 [ 4, %entry ], [ %iv.shl.next, %loop ]
 ; CHECK-NEXT:    --> %iv.shl U: [4,65) S: [4,65) Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.shl.next = shl i64 %iv.shl, %shiftamt
 ; CHECK-NEXT:    --> %iv.shl.next U: [0,-3) S: [-9223372036854775808,9223372036854775805) Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_shl3
@@ -267,11 +267,11 @@ define void @test_shl4() {
 ; CHECK-LABEL: 'test_shl4'
 ; CHECK-NEXT:  Classifying expressions for: @test_shl4
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,61) S: [0,61) Exits: 60 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,61) S: [0,61) Exits: 60 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.shl = phi i64 [ 4, %entry ], [ %iv.shl.next, %loop ]
 ; CHECK-NEXT:    --> %iv.shl U: [4,4611686018427387905) S: [4,4611686018427387905) Exits: 4611686018427387904 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,62) S: [1,62) Exits: 61 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,62) S: [1,62) Exits: 61 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.shl.next = shl i64 %iv.shl, 1
 ; CHECK-NEXT:    --> (2 * %iv.shl)<nuw> U: [8,-9223372036854775807) S: [-9223372036854775808,9223372036854775801) Exits: -9223372036854775808 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_shl4
@@ -300,11 +300,11 @@ define void @test_shl5() {
 ; CHECK-LABEL: 'test_shl5'
 ; CHECK-NEXT:  Classifying expressions for: @test_shl5
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,62) S: [0,62) Exits: 61 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,62) S: [0,62) Exits: 61 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.shl = phi i64 [ 4, %entry ], [ %iv.shl.next, %loop ]
 ; CHECK-NEXT:    --> %iv.shl U: [0,-3) S: [-9223372036854775808,9223372036854775801) Exits: -9223372036854775808 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,63) S: [1,63) Exits: 62 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,63) S: [1,63) Exits: 62 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.shl.next = shl i64 %iv.shl, 1
 ; CHECK-NEXT:    --> (2 * %iv.shl) U: [0,-7) S: [-9223372036854775808,9223372036854775801) Exits: 0 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_shl5
@@ -333,11 +333,11 @@ define void @test_shl6(i1 %c) {
 ; CHECK-LABEL: 'test_shl6'
 ; CHECK-NEXT:  Classifying expressions for: @test_shl6
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.shl = phi i64 [ 4, %entry ], [ %iv.shl.next, %loop ]
 ; CHECK-NEXT:    --> %iv.shl U: [4,65) S: [4,65) Exits: 16 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %shiftamt = and i64 %iv, 1
 ; CHECK-NEXT:    --> (zext i1 {false,+,true}<%loop> to i64) U: [0,2) S: [0,2) Exits: 0 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.shl.next = shl i64 %iv.shl, %shiftamt
@@ -369,11 +369,11 @@ define void @test_shl7(i1 %c, i64 %shiftamt) {
 ; CHECK-LABEL: 'test_shl7'
 ; CHECK-NEXT:  Classifying expressions for: @test_shl7
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.shl = phi i64 [ 4, %entry ], [ %iv.shl.next, %loop ]
 ; CHECK-NEXT:    --> %iv.shl U: [0,-3) S: [-9223372036854775808,9223372036854775805) Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.shl.next = shl i64 %iv.shl, %shiftamt
 ; CHECK-NEXT:    --> %iv.shl.next U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_shl7
@@ -516,11 +516,11 @@ define void @test_ashr_tc_positive() {
 ; CHECK-LABEL: 'test_ashr_tc_positive'
 ; CHECK-NEXT:  Classifying expressions for: @test_ashr_tc_positive
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.ashr = phi i64 [ 1023, %entry ], [ %iv.ashr.next, %loop ]
 ; CHECK-NEXT:    --> %iv.ashr U: [63,1024) S: [63,1024) Exits: 63 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.ashr.next = ashr i64 %iv.ashr, 1
 ; CHECK-NEXT:    --> %iv.ashr.next U: [0,512) S: [0,512) Exits: 31 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_ashr_tc_positive
@@ -548,11 +548,11 @@ define void @test_ashr_tc_negative() {
 ; CHECK-LABEL: 'test_ashr_tc_negative'
 ; CHECK-NEXT:  Classifying expressions for: @test_ashr_tc_negative
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.ashr = phi i8 [ -128, %entry ], [ %iv.ashr.next, %loop ]
 ; CHECK-NEXT:    --> %iv.ashr U: [-128,-7) S: [-128,-7) Exits: -8 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.ashr.next = ashr i8 %iv.ashr, 1
 ; CHECK-NEXT:    --> %iv.ashr.next U: [-64,0) S: [-64,0) Exits: -4 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_ashr_tc_negative
@@ -582,11 +582,11 @@ define void @test_ashr_tc_either(i1 %a) {
 ; CHECK-NEXT:    %start = sext i1 %a to i8
 ; CHECK-NEXT:    --> (sext i1 %a to i8) U: [-1,1) S: [-1,1)
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,61) S: [0,61) Exits: 60 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,61) S: [0,61) Exits: 60 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.ashr = phi i8 [ %start, %entry ], [ %iv.ashr.next, %loop ]
 ; CHECK-NEXT:    --> %iv.ashr U: [-16,16) S: [-16,16) Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,62) S: [1,62) Exits: 61 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,62) S: [1,62) Exits: 61 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.ashr.next = ashr i8 %iv.ashr, 1
 ; CHECK-NEXT:    --> %iv.ashr.next U: [-16,16) S: [-16,16) Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_ashr_tc_either
@@ -615,11 +615,11 @@ define void @test_ashr_zero_shift() {
 ; CHECK-LABEL: 'test_ashr_zero_shift'
 ; CHECK-NEXT:  Classifying expressions for: @test_ashr_zero_shift
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.ashr = phi i64 [ 1023, %entry ], [ %iv.ashr.next, %loop ]
 ; CHECK-NEXT:    --> %iv.ashr U: [1023,1024) S: [1023,1024) Exits: 1023 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.ashr.next = ashr i64 %iv.ashr, 0
 ; CHECK-NEXT:    --> %iv.ashr U: [1023,1024) S: [1023,1024) Exits: 1023 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_ashr_zero_shift
@@ -647,11 +647,11 @@ define void @test_lshr_tc_positive() {
 ; CHECK-LABEL: 'test_lshr_tc_positive'
 ; CHECK-NEXT:  Classifying expressions for: @test_lshr_tc_positive
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.lshr = phi i64 [ 1023, %entry ], [ %iv.lshr.next, %loop ]
 ; CHECK-NEXT:    --> %iv.lshr U: [63,1024) S: [63,1024) Exits: 63 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.lshr.next = lshr i64 %iv.lshr, 1
 ; CHECK-NEXT:    --> (%iv.lshr /u 2) U: [31,512) S: [31,512) Exits: 31 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_lshr_tc_positive
@@ -679,11 +679,11 @@ define void @test_lshr_tc_negative() {
 ; CHECK-LABEL: 'test_lshr_tc_negative'
 ; CHECK-NEXT:  Classifying expressions for: @test_lshr_tc_negative
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.lshr = phi i8 [ -1, %entry ], [ %iv.lshr.next, %loop ]
 ; CHECK-NEXT:    --> %iv.lshr U: [15,0) S: [-1,-128) Exits: 15 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.lshr.next = lshr i8 %iv.lshr, 1
 ; CHECK-NEXT:    --> (%iv.lshr /u 2) U: [7,-128) S: [7,-128) Exits: 7 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_lshr_tc_negative
@@ -713,11 +713,11 @@ define void @test_lshr_tc_either(i1 %a) {
 ; CHECK-NEXT:    %start = sext i1 %a to i8
 ; CHECK-NEXT:    --> (sext i1 %a to i8) U: [-1,1) S: [-1,1)
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.lshr = phi i8 [ %start, %entry ], [ %iv.lshr.next, %loop ]
 ; CHECK-NEXT:    --> %iv.lshr U: [-1,-128) S: [-1,-128) Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.lshr.next = lshr i8 %iv.lshr, 1
 ; CHECK-NEXT:    --> (%iv.lshr /u 2) U: [0,-128) S: [0,-128) Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_lshr_tc_either
@@ -746,11 +746,11 @@ define void @test_lshr_zero_shift() {
 ; CHECK-LABEL: 'test_lshr_zero_shift'
 ; CHECK-NEXT:  Classifying expressions for: @test_lshr_zero_shift
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.lshr = phi i64 [ 1023, %entry ], [ %iv.lshr.next, %loop ]
 ; CHECK-NEXT:    --> %iv.lshr U: [1023,1024) S: [1023,1024) Exits: 1023 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.lshr.next = lshr i64 %iv.lshr, 0
 ; CHECK-NEXT:    --> %iv.lshr U: [1023,1024) S: [1023,1024) Exits: 1023 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_lshr_zero_shift
@@ -779,11 +779,11 @@ define void @test_lshr_power_of_2_start() {
 ; CHECK-LABEL: 'test_lshr_power_of_2_start'
 ; CHECK-NEXT:  Classifying expressions for: @test_lshr_power_of_2_start
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.lshr = phi i64 [ 1024, %entry ], [ %iv.lshr.next, %loop ]
 ; CHECK-NEXT:    --> %iv.lshr U: [4,1025) S: [4,1025) Exits: 4 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.lshr.next = lshr i64 %iv.lshr, 2
 ; CHECK-NEXT:    --> (%iv.lshr /u 4) U: [1,257) S: [1,257) Exits: 1 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_lshr_power_of_2_start
@@ -812,11 +812,11 @@ define void @test_lshr_arbitrary_start() {
 ; CHECK-LABEL: 'test_lshr_arbitrary_start'
 ; CHECK-NEXT:  Classifying expressions for: @test_lshr_arbitrary_start
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.lshr = phi i64 [ 957, %entry ], [ %iv.lshr.next, %loop ]
 ; CHECK-NEXT:    --> %iv.lshr U: [3,958) S: [3,958) Exits: 3 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.lshr.next = lshr i64 %iv.lshr, 2
 ; CHECK-NEXT:    --> (%iv.lshr /u 4) U: [0,240) S: [0,240) Exits: 0 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_lshr_arbitrary_start
@@ -844,11 +844,11 @@ define void @test_lshr_start_power_of_2_plus_one() {
 ; CHECK-LABEL: 'test_lshr_start_power_of_2_plus_one'
 ; CHECK-NEXT:  Classifying expressions for: @test_lshr_start_power_of_2_plus_one
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,5) S: [0,5) Exits: 4 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.lshr = phi i64 [ 1025, %entry ], [ %iv.lshr.next, %loop ]
 ; CHECK-NEXT:    --> %iv.lshr U: [4,1026) S: [4,1026) Exits: 4 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,6) S: [1,6) Exits: 5 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.lshr.next = lshr i64 %iv.lshr, 2
 ; CHECK-NEXT:    --> (%iv.lshr /u 4) U: [1,257) S: [1,257) Exits: 1 LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_lshr_start_power_of_2_plus_one
diff --git a/llvm/test/Analysis/ScalarEvolution/solve-quadratic-overflow.ll b/llvm/test/Analysis/ScalarEvolution/solve-quadratic-overflow.ll
index 1537c49040d41..2f831827b549f 100644
--- a/llvm/test/Analysis/ScalarEvolution/solve-quadratic-overflow.ll
+++ b/llvm/test/Analysis/ScalarEvolution/solve-quadratic-overflow.ll
@@ -11,13 +11,13 @@ define signext i32 @f0() {
 ; CHECK-LABEL: 'f0'
 ; CHECK-NEXT:  Classifying expressions for: @f0
 ; CHECK-NEXT:    %v1 = phi i16 [ 0, %b0 ], [ %v2, %b1 ]
-; CHECK-NEXT:    --> {0,+,-1}<%b1> U: [-255,1) S: [-255,1) Exits: -255 LoopDispositions: { %b1: Computable }
+; CHECK-NEXT:    --> {0,+,-1}<nsw><%b1> U: [-255,1) S: [-255,1) Exits: -255 LoopDispositions: { %b1: Computable }
 ; CHECK-NEXT:    %v2 = add i16 %v1, -1
-; CHECK-NEXT:    --> {-1,+,-1}<%b1> U: [-256,0) S: [-256,0) Exits: -256 LoopDispositions: { %b1: Computable }
+; CHECK-NEXT:    --> {-1,+,-1}<nsw><%b1> U: [-256,0) S: [-256,0) Exits: -256 LoopDispositions: { %b1: Computable }
 ; CHECK-NEXT:    %v3 = mul i16 %v2, %v2
 ; CHECK-NEXT:    --> {1,+,3,+,2}<%b1> U: full-set S: full-set Exits: 0 LoopDispositions: { %b1: Computable }
 ; CHECK-NEXT:    %v5 = phi i16 [ %v2, %b1 ]
-; CHECK-NEXT:    --> {-1,+,-1}<%b1> U: [-256,0) S: [-256,0) --> -256 U: [-256,-255) S: [-256,-255)
+; CHECK-NEXT:    --> {-1,+,-1}<nsw><%b1> U: [-256,0) S: [-256,0) --> -256 U: [-256,-255) S: [-256,-255)
 ; CHECK-NEXT:    %v6 = phi i16 [ %v3, %b1 ]
 ; CHECK-NEXT:    --> {1,+,3,+,2}<%b1> U: full-set S: full-set --> 0 U: [0,1) S: [0,1)
 ; CHECK-NEXT:    %v7 = sext i16 %v5 to i32
diff --git a/llvm/test/Analysis/ScalarEvolution/symbolic_max_exit_count.ll b/llvm/test/Analysis/ScalarEvolution/symbolic_max_exit_count.ll
index d45cf2dd65fd3..9cbb6263e3935 100644
--- a/llvm/test/Analysis/ScalarEvolution/symbolic_max_exit_count.ll
+++ b/llvm/test/Analysis/ScalarEvolution/symbolic_max_exit_count.ll
@@ -230,9 +230,9 @@ define i32 @test_mixup_constant_symbolic(i32 %end, i32 %len) {
 ; CHECK-LABEL: 'test_mixup_constant_symbolic'
 ; CHECK-NEXT:  Classifying expressions for: @test_mixup_constant_symbolic
 ; CHECK-NEXT:    %iv = phi i32 [ 0, %entry ], [ %iv.next, %backedge ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,1001) S: [0,1001) Exits: <<Unknown>> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,1001) S: [0,1001) Exits: <<Unknown>> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i32 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,1002) S: [1,1002) Exits: <<Unknown>> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,1002) S: [1,1002) Exits: <<Unknown>> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %loop_cond = call i1 @cond()
 ; CHECK-NEXT:    --> %loop_cond U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_mixup_constant_symbolic
@@ -278,11 +278,11 @@ define i32 @test_mixup_constant_symbolic_merged(i32 %end, i32 %len) {
 ; CHECK-LABEL: 'test_mixup_constant_symbolic_merged'
 ; CHECK-NEXT:  Classifying expressions for: @test_mixup_constant_symbolic_merged
 ; CHECK-NEXT:    %iv = phi i32 [ 0, %entry ], [ %iv.next, %backedge ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,1001) S: [0,1001) Exits: <<Unknown>> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,1001) S: [0,1001) Exits: <<Unknown>> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %and = and i1 %zero_check, %range_check
 ; CHECK-NEXT:    --> (%range_check umin %zero_check) U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv.next = add i32 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,1002) S: [1,1002) Exits: <<Unknown>> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,1002) S: [1,1002) Exits: <<Unknown>> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %loop_cond = call i1 @cond()
 ; CHECK-NEXT:    --> %loop_cond U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @test_mixup_constant_symbolic_merged
diff --git a/llvm/test/Analysis/ScalarEvolution/trip-count15.ll b/llvm/test/Analysis/ScalarEvolution/trip-count15.ll
index 028b1b44b0934..e043357aa09c3 100644
--- a/llvm/test/Analysis/ScalarEvolution/trip-count15.ll
+++ b/llvm/test/Analysis/ScalarEvolution/trip-count15.ll
@@ -7,9 +7,9 @@ define void @umin_unsigned_check(i64 %n) {
 ; CHECK-NEXT:    %min.n = select i1 %min.cmp, i64 4096, i64 %n
 ; CHECK-NEXT:    --> (4096 umin %n) U: [0,4097) S: [0,4097)
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,4098) S: [0,4098) Exits: (1 + (4096 umin %n))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,4098) S: [0,4098) Exits: (1 + (4096 umin %n))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,4099) S: [1,4099) Exits: (2 + (4096 umin %n))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,4099) S: [1,4099) Exits: (2 + (4096 umin %n))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @umin_unsigned_check
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (1 + (4096 umin %n))<nuw><nsw>
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 4097
@@ -38,9 +38,9 @@ define void @umin_signed_check(i64 %n) {
 ; CHECK-NEXT:    %min.n = select i1 %min.cmp, i64 4096, i64 %n
 ; CHECK-NEXT:    --> (4096 umin %n) U: [0,4097) S: [0,4097)
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,4098) S: [0,4098) Exits: (1 + (4096 umin %n))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,4098) S: [0,4098) Exits: (1 + (4096 umin %n))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,4099) S: [1,4099) Exits: (2 + (4096 umin %n))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,4099) S: [1,4099) Exits: (2 + (4096 umin %n))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @umin_signed_check
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (1 + (4096 umin %n))<nuw><nsw>
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 4097
@@ -69,9 +69,9 @@ define void @smin_signed_check(i64 %n) {
 ; CHECK-NEXT:    %min.n = select i1 %min.cmp, i64 4096, i64 %n
 ; CHECK-NEXT:    --> (4096 smin %n) U: [-9223372036854775808,4097) S: [-9223372036854775808,4097)
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,4098) S: [0,4098) Exits: (0 smax (1 + (4096 smin %n))<nsw>) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,4098) S: [0,4098) Exits: (0 smax (1 + (4096 smin %n))<nsw>) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,4099) S: [1,4099) Exits: (1 + (0 smax (1 + (4096 smin %n))<nsw>))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,4099) S: [1,4099) Exits: (1 + (0 smax (1 + (4096 smin %n))<nsw>))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @smin_signed_check
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is (0 smax (1 + (4096 smin %n))<nsw>)
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 4097
diff --git a/llvm/test/Analysis/ScalarEvolution/trivial-phis.ll b/llvm/test/Analysis/ScalarEvolution/trivial-phis.ll
index a2d979dd45271..ecd5d46e84f01 100644
--- a/llvm/test/Analysis/ScalarEvolution/trivial-phis.ll
+++ b/llvm/test/Analysis/ScalarEvolution/trivial-phis.ll
@@ -117,21 +117,21 @@ bb48:                                             ; preds = %bb47, %bb
 ; CHECK-LABEL: @test3
 
 ; CHECK:      %tmp14 = phi i64 [ %tmp40, %bb39 ], [ 1, %bb8 ]
-; CHECK-NEXT: -->  {1,+,1}<%bb13> U: [1,9223372036854775807) S: [1,9223372036854775807)
+; CHECK-NEXT: --> {1,+,1}<nuw><nsw><%bb13> U: [1,9223372036854775807) S: [1,9223372036854775807)
 ; CHECK-SAME:      Exits: (-2 + %arg)       LoopDispositions: { %bb13: Computable, %bb8: Variant, %bb17_a: Invariant, %bb27: Invariant }
 ; CHECK:      %tmp18 = phi i64 [ %tmp20, %bb17 ], [ 0, %bb13 ]
 ; CHECK-NEXT: -->  {0,+,1}<nuw><nsw><%bb17_a> U: [0,9223372036854775807) S: [0,9223372036854775807)
 ; CHECK-SAME:      Exits: (-1 + %arg)       LoopDispositions: { %bb17_a: Computable, %bb13: Variant, %bb8: Variant }
 
 ; CHECK:      %tmp24 = phi i64 [ %tmp14, %bb13 ], [ %tmp14, %bb17 ]
-; CHECK-NEXT: -->  {1,+,1}<%bb13> U: [1,9223372036854775807) S: [1,9223372036854775807)
+; CHECK-NEXT: -->  {1,+,1}<nuw><nsw><%bb13> U: [1,9223372036854775807) S: [1,9223372036854775807)
 ; CHECK-SAME:      Exits: (-2 + %arg)       LoopDispositions: { %bb13: Computable, %bb8: Variant, %bb17_a: Invariant, %bb27: Invariant }
 ; CHECK:       %tmp28 = phi i64 [ %tmp34, %bb27 ], [ 0, %bb23 ]
 ; CHECK-NEXT:  -->  {0,+,1}<nuw><nsw><%bb27> U: [0,9223372036854775807) S: [0,9223372036854775807)
 ; CHECK-SAME:       Exits: (-1 + %arg)      LoopDispositions: { %bb27: Computable, %bb13: Variant, %bb8: Variant }
 
 ; CHECK:      %tmp38 = phi i64 [ %tmp24, %bb23 ], [ %tmp24, %bb27 ]
-; CHECK-NEXT: -->  {1,+,1}<%bb13> U: [1,9223372036854775807) S: [1,9223372036854775807)
+; CHECK-NEXT: --> {1,+,1}<nuw><nsw><%bb13> U: [1,9223372036854775807) S: [1,9223372036854775807)
 ; CHECK-SAME:      Exits: (-2 + %arg)       LoopDispositions: { %bb13: Computable, %bb8: Variant, %bb17_a: Invariant, %bb27: Invariant }
 
 define void @test3(i64 %arg, ptr %arg1) {
diff --git a/llvm/test/Analysis/ScalarEvolution/umin-umax-folds.ll b/llvm/test/Analysis/ScalarEvolution/umin-umax-folds.ll
index 8e72000ee4343..74c07c2933e3f 100644
--- a/llvm/test/Analysis/ScalarEvolution/umin-umax-folds.ll
+++ b/llvm/test/Analysis/ScalarEvolution/umin-umax-folds.ll
@@ -9,9 +9,9 @@ define void @umin_sext_x_zext_x(i32 %len) {
 ; CHECK-NEXT:    %len.sext = sext i32 %len to i64
 ; CHECK-NEXT:    --> (sext i32 %len to i64) U: [-2147483648,2147483648) S: [-2147483648,2147483648)
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,4294967296) S: [0,4294967296) Exits: (zext i32 %len to i64) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,4294967296) S: [0,4294967296) Exits: (zext i32 %len to i64) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,4294967297) S: [1,4294967297) Exits: (1 + (zext i32 %len to i64))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,4294967297) S: [1,4294967297) Exits: (1 + (zext i32 %len to i64))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %and = and i1 %cmp1, %cmp2
 ; CHECK-NEXT:    --> (%cmp1 umin %cmp2) U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:  Determining loop execution counts for: @umin_sext_x_zext_x
@@ -45,9 +45,9 @@ define void @ule_sext_x_zext_x(i32 %len) {
 ; CHECK-NEXT:    %len.sext = sext i32 %len to i64
 ; CHECK-NEXT:    --> (sext i32 %len to i64) U: [-2147483648,2147483648) S: [-2147483648,2147483648)
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,4294967296) S: [0,4294967296) Exits: (zext i32 %len to i64) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,4294967296) S: [0,4294967296) Exits: (zext i32 %len to i64) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,4294967297) S: [1,4294967297) Exits: (1 + (zext i32 %len to i64))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,4294967297) S: [1,4294967297) Exits: (1 + (zext i32 %len to i64))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %sel = select i1 %cmp1, i64 %len.zext, i64 %len.sext
 ; CHECK-NEXT:    --> (zext i32 %len to i64) U: [0,4294967296) S: [0,4294967296) Exits: (zext i32 %len to i64) LoopDispositions: { %loop: Invariant }
 ; CHECK-NEXT:  Determining loop execution counts for: @ule_sext_x_zext_x
@@ -117,9 +117,9 @@ define void @ult_sext_x_zext_x(i32 %len) {
 ; CHECK-NEXT:    %len.sext = sext i32 %len to i64
 ; CHECK-NEXT:    --> (sext i32 %len to i64) U: [-2147483648,2147483648) S: [-2147483648,2147483648)
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,4294967296) S: [0,4294967296) Exits: (zext i32 %len to i64) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,4294967296) S: [0,4294967296) Exits: (zext i32 %len to i64) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,4294967297) S: [1,4294967297) Exits: (1 + (zext i32 %len to i64))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,4294967297) S: [1,4294967297) Exits: (1 + (zext i32 %len to i64))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %umin = select i1 %cmp1, i64 %len.zext, i64 %len.sext
 ; CHECK-NEXT:    --> (zext i32 %len to i64) U: [0,4294967296) S: [0,4294967296) Exits: (zext i32 %len to i64) LoopDispositions: { %loop: Invariant }
 ; CHECK-NEXT:  Determining loop execution counts for: @ult_sext_x_zext_x
@@ -189,9 +189,9 @@ define void @sle_sext_x_zext_x(i32 %len) {
 ; CHECK-NEXT:    %len.sext = sext i32 %len to i64
 ; CHECK-NEXT:    --> (sext i32 %len to i64) U: [-2147483648,2147483648) S: [-2147483648,2147483648)
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,4294967296) S: [0,4294967296) Exits: (zext i32 %len to i64) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,4294967296) S: [0,4294967296) Exits: (zext i32 %len to i64) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,4294967297) S: [1,4294967297) Exits: (1 + (zext i32 %len to i64))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,4294967297) S: [1,4294967297) Exits: (1 + (zext i32 %len to i64))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %sel = select i1 %cmp1, i64 %len.zext, i64 %len.sext
 ; CHECK-NEXT:    --> (zext i32 %len to i64) U: [0,4294967296) S: [0,4294967296) Exits: (zext i32 %len to i64) LoopDispositions: { %loop: Invariant }
 ; CHECK-NEXT:  Determining loop execution counts for: @sle_sext_x_zext_x
@@ -225,9 +225,9 @@ define void @sge_sext_x_zext_x(i32 %len) {
 ; CHECK-NEXT:    %len.sext = sext i32 %len to i64
 ; CHECK-NEXT:    --> (sext i32 %len to i64) U: [-2147483648,2147483648) S: [-2147483648,2147483648)
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,4294967296) S: [0,4294967296) Exits: (zext i32 %len to i64) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,4294967296) S: [0,4294967296) Exits: (zext i32 %len to i64) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,4294967297) S: [1,4294967297) Exits: (1 + (zext i32 %len to i64))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,4294967297) S: [1,4294967297) Exits: (1 + (zext i32 %len to i64))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %sel = select i1 %cmp1, i64 %len.zext, i64 %len.sext
 ; CHECK-NEXT:    --> (zext i32 %len to i64) U: [0,4294967296) S: [0,4294967296) Exits: (zext i32 %len to i64) LoopDispositions: { %loop: Invariant }
 ; CHECK-NEXT:  Determining loop execution counts for: @sge_sext_x_zext_x
@@ -297,9 +297,9 @@ define void @sgt_sext_x_zext_x(i32 %len) {
 ; CHECK-NEXT:    %len.sext = sext i32 %len to i64
 ; CHECK-NEXT:    --> (sext i32 %len to i64) U: [-2147483648,2147483648) S: [-2147483648,2147483648)
 ; CHECK-NEXT:    %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,4294967296) S: [0,4294967296) Exits: (zext i32 %len to i64) LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,4294967296) S: [0,4294967296) Exits: (zext i32 %len to i64) LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i64 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,4294967297) S: [1,4294967297) Exits: (1 + (zext i32 %len to i64))<nuw><nsw> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,4294967297) S: [1,4294967297) Exits: (1 + (zext i32 %len to i64))<nuw><nsw> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %umax = select i1 %cmp1, i64 %len.zext, i64 %len.sext
 ; CHECK-NEXT:    --> (zext i32 %len to i64) U: [0,4294967296) S: [0,4294967296) Exits: (zext i32 %len to i64) LoopDispositions: { %loop: Invariant }
 ; CHECK-NEXT:  Determining loop execution counts for: @sgt_sext_x_zext_x
diff --git a/llvm/test/Analysis/ScalarEvolution/unknown_phis.ll b/llvm/test/Analysis/ScalarEvolution/unknown_phis.ll
index 2ab7788e62ccc..d0fab9b9447cb 100644
--- a/llvm/test/Analysis/ScalarEvolution/unknown_phis.ll
+++ b/llvm/test/Analysis/ScalarEvolution/unknown_phis.ll
@@ -43,9 +43,9 @@ define void @merge_values_with_ranges_looped(ptr %a_len_ptr, ptr %b_len_ptr) {
 ; CHECK-NEXT:    %p2 = phi i32 [ %len_b, %entry ], [ %p1, %loop ]
 ; CHECK-NEXT:    --> %p2 U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %iv = phi i32 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,100) S: [0,100) Exits: 99 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,100) S: [0,100) Exits: 99 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i32 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,101) S: [1,101) Exits: 100 LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,101) S: [1,101) Exits: 100 LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:  Determining loop execution counts for: @merge_values_with_ranges_looped
 ; CHECK-NEXT:  Loop %loop: backedge-taken count is 99
 ; CHECK-NEXT:  Loop %loop: constant max backedge-taken count is 99
diff --git a/llvm/test/Analysis/ScalarEvolution/widenable-condition.ll b/llvm/test/Analysis/ScalarEvolution/widenable-condition.ll
index b7e58bf5067eb..7d8b01b9effd4 100644
--- a/llvm/test/Analysis/ScalarEvolution/widenable-condition.ll
+++ b/llvm/test/Analysis/ScalarEvolution/widenable-condition.ll
@@ -14,9 +14,9 @@ define i32 @wc_max() {
 ; CHECK-LABEL: 'wc_max'
 ; CHECK-NEXT:  Classifying expressions for: @wc_max
 ; CHECK-NEXT:    %iv = phi i32 [ 0, %entry ], [ %iv.next, %loop ]
-; CHECK-NEXT:    --> {0,+,1}<%loop> U: [0,2000) S: [0,2000) Exits: <<Unknown>> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {0,+,1}<nuw><nsw><%loop> U: [0,2000) S: [0,2000) Exits: <<Unknown>> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %iv.next = add i32 %iv, 1
-; CHECK-NEXT:    --> {1,+,1}<%loop> U: [1,2001) S: [1,2001) Exits: <<Unknown>> LoopDispositions: { %loop: Computable }
+; CHECK-NEXT:    --> {1,+,1}<nuw><nsw><%loop> U: [1,2001) S: [1,2001) Exits: <<Unknown>> LoopDispositions: { %loop: Computable }
 ; CHECK-NEXT:    %widenable_cond3 = call i1 @llvm.experimental.widenable.condition()
 ; CHECK-NEXT:    --> %widenable_cond3 U: full-set S: full-set Exits: <<Unknown>> LoopDispositions: { %loop: Variant }
 ; CHECK-NEXT:    %exiplicit_guard_cond4 = and i1 %cond_1, %widenable_cond3
diff --git a/llvm/test/Assembler/ConstantExprFoldSelect.ll b/llvm/test/Assembler/ConstantExprFoldSelect.ll
index 5d218a9570b9b..1ece1f87c3403 100644
--- a/llvm/test/Assembler/ConstantExprFoldSelect.ll
+++ b/llvm/test/Assembler/ConstantExprFoldSelect.ll
@@ -1,9 +1,9 @@
-; RUN: llvm-as < %s | llvm-dis | FileCheck %s
+; RUN: opt -S -passes=instsimplify < %s | FileCheck %s
 ; RUN: verify-uselistorder %s
 ; PR18319
 
-define void @function() {
-  %c = trunc <4 x i16> select (<4 x i1> <i1 undef, i1 undef, i1 false, i1 true>, <4 x i16> <i16 undef, i16 2, i16 3, i16 4>, <4 x i16> <i16 -1, i16 -2, i16 -3, i16 -4>) to <4 x i8>
+define <4 x i16> @function() {
+  %s = select <4 x i1> <i1 undef, i1 undef, i1 false, i1 true>, <4 x i16> <i16 undef, i16 2, i16 3, i16 4>, <4 x i16> <i16 -1, i16 -2, i16 -3, i16 -4>
 ; CHECK: <i16 undef, i16 -2, i16 -3, i16 4>
-  ret void
+  ret <4 x i16> %s
 }
diff --git a/llvm/test/Assembler/non-global-value-max-name-size.ll b/llvm/test/Assembler/non-global-value-max-name-size.ll
index 4536ce2056862..dd56286baa973 100644
--- a/llvm/test/Assembler/non-global-value-max-name-size.ll
+++ b/llvm/test/Assembler/non-global-value-max-name-size.ll
@@ -1,10 +1,15 @@
-; RUN: opt < %s -S -non-global-value-max-name-size=4
-; Test that local value name lookup works if the name is capped
+; RUN: opt < %s -S -non-global-value-max-name-size=5
+; RUN: not opt < %s -S -non-global-value-max-name-size=4 2>&1 | FileCheck %s
+
+; CHECK: name is too long
 
 define void @f() {
 bb0:
   br label %testz
 
 testz:
+  br label %testa
+
+testa:
   br label %testz
 }
diff --git a/llvm/test/Bindings/OCaml/core.ml b/llvm/test/Bindings/OCaml/core.ml
index 33841deb42cfa..d2dd52c076084 100644
--- a/llvm/test/Bindings/OCaml/core.ml
+++ b/llvm/test/Bindings/OCaml/core.ml
@@ -333,7 +333,6 @@ let test_constants () =
   group "misc constants";
   (* CHECK: const_size_of{{.*}}getelementptr{{.*}}null
    * CHECK: const_gep{{.*}}getelementptr
-   * CHECK: const_select{{.*}}select
    * CHECK: const_extractelement{{.*}}extractelement
    * CHECK: const_insertelement{{.*}}insertelement
    * CHECK: const_shufflevector = global <4 x i32> <i32 0, i32 1, i32 1, i32 0>
@@ -341,10 +340,6 @@ let test_constants () =
   ignore (define_global "const_size_of" (size_of (pointer_type context)) m);
   ignore (define_global "const_gep" (const_gep i8_type foldbomb_gv [| five |])
           m);
-  ignore (define_global "const_select" (const_select
-    (const_icmp Icmp.Sle foldbomb five)
-    (const_int i8_type (-1))
-    (const_int i8_type 0)) m);
   let zero = const_int i32_type 0 in
   let one  = const_int i32_type 1 in
   ignore (define_global "const_extractelement" (const_extractelement
diff --git a/llvm/test/Bitcode/constexpr-scalable-vector-gep.ll b/llvm/test/Bitcode/constexpr-scalable-vector-gep.ll
new file mode 100644
index 0000000000000..df660b5812163
--- /dev/null
+++ b/llvm/test/Bitcode/constexpr-scalable-vector-gep.ll
@@ -0,0 +1,10 @@
+; RUN: llvm-dis < %s.bc | FileCheck %s
+
+@g = extern_weak global [32 x i8]
+
+define ptr @test_scalable_vector_gep() {
+; CHECK-LABEL: define ptr @test_scalable_vector_gep() {
+; CHECK-NEXT: %constexpr = getelementptr <vscale x 1 x i8>, ptr @g, i64 1
+; CHECK-NEXT: ret ptr %constexpr
+  ret ptr getelementptr (<vscale x 1 x i8>, ptr @g, i64 1)
+}
diff --git a/llvm/test/Bitcode/constexpr-scalable-vector-gep.ll.bc b/llvm/test/Bitcode/constexpr-scalable-vector-gep.ll.bc
new file mode 100644
index 0000000000000..de80a0e554d7c
Binary files /dev/null and b/llvm/test/Bitcode/constexpr-scalable-vector-gep.ll.bc differ
diff --git a/llvm/test/Bitcode/invalid.test b/llvm/test/Bitcode/invalid.test
index cb0677cf64303..464cf787feddf 100644
--- a/llvm/test/Bitcode/invalid.test
+++ b/llvm/test/Bitcode/invalid.test
@@ -1,3 +1,4 @@
+RUN: export LSAN_OPTIONS=detect_leaks=0
 RUN: not llvm-dis -disable-output %p/Inputs/invalid-empty.bc 2>&1 | \
 RUN:   FileCheck --check-prefix=INVALID-EMPTY %s
 RUN: not llvm-dis -disable-output %p/Inputs/invalid-pr20485.bc 2>&1 | \
diff --git a/llvm/test/Bitcode/select.ll b/llvm/test/Bitcode/select.ll
deleted file mode 100644
index 7e62361822a87..0000000000000
--- a/llvm/test/Bitcode/select.ll
+++ /dev/null
@@ -1,18 +0,0 @@
-; RUN: llvm-as < %s | llvm-dis | FileCheck %s
-; RUN: verify-uselistorder < %s
-
-define <2 x i32> @main() {
-  ret <2 x i32> select (<2 x i1> <i1 false, i1 undef>, <2 x i32> zeroinitializer, <2 x i32> <i32 0, i32 undef>)
-}
-
-; CHECK: define <2 x i32> @main() {
-; CHECK:   ret <2 x i32> <i32 0, i32 undef>
-; CHECK: }
-
-define <2 x float> @f() {
-  ret <2 x float> select (i1 ptrtoint (<2 x float> ()* @f to i1), <2 x float> <float 1.000000e+00, float 0.000000e+00>, <2 x float> zeroinitializer)
-}
-
-; CHECK: define <2 x float> @f() {
-; CHECK:   ret <2 x float> select (i1 ptrtoint (ptr @f to i1), <2 x float> <float 1.000000e+00, float 0.000000e+00>, <2 x float> zeroinitializer)
-; CHECK: }
diff --git a/llvm/test/Bitcode/thinlto-function-summary-callgraph-cast.ll b/llvm/test/Bitcode/thinlto-function-summary-callgraph-cast.ll
index 1926603b8b1f2..4d52edd6a085b 100644
--- a/llvm/test/Bitcode/thinlto-function-summary-callgraph-cast.ll
+++ b/llvm/test/Bitcode/thinlto-function-summary-callgraph-cast.ll
@@ -9,7 +9,7 @@
 ; "op7" is a call to "callee" function.
 ; CHECK-NEXT:    <PERMODULE {{.*}} op7=3 op8=[[ALIASID:[0-9]+]]/>
 ; "another_caller" has only references but no calls.
-; CHECK-NEXT:    <PERMODULE {{.*}} op4=3 {{.*}} op9={{[0-9]+}}/>
+; CHECK-NEXT:    <PERMODULE {{.*}}/>
 ; CHECK-NEXT:    <PERMODULE {{.*}} op0=[[ALIASEEID:[0-9]+]]
 ; CHECK-NEXT:    <ALIAS {{.*}} op0=[[ALIASID]] {{.*}} op2=[[ALIASEEID]]/>
 ; CHECK-NEXT:    <BLOCK_COUNT op0=3/>
@@ -27,7 +27,7 @@ define void @caller() {
 
 define void @another_caller() {
     ; Test calls that aren't handled either as direct or indirect.
-    call void select (i1 icmp eq (ptr @global, ptr null), ptr @f, ptr @g)()
+    call void getelementptr (i8, ptr @f, i64 ptrtoint (ptr @g to i64))()
     ret void
 }
 
diff --git a/llvm/test/Bitcode/vscale-round-trip.ll b/llvm/test/Bitcode/vscale-round-trip.ll
index 62a3479a37242..1962e90fa4645 100644
--- a/llvm/test/Bitcode/vscale-round-trip.ll
+++ b/llvm/test/Bitcode/vscale-round-trip.ll
@@ -36,14 +36,14 @@ define <vscale x 4 x i32> @non_const_shufflevector(<vscale x 4 x i32> %lhs,
 }
 
 ; CHECK-LABEL: define <vscale x 4 x i32> @const_select()
-; CHECK: <vscale x 4 x i32> select (<vscale x 4 x i1>
+; CHECK: select <vscale x 4 x i1>
 
 define <vscale x 4 x i32> @const_select() {
-  ret <vscale x 4 x i32> select
-    (<vscale x 4 x i1> insertelement
+  %s = select <vscale x 4 x i1> insertelement
       (<vscale x 4 x i1> undef,
        i1 icmp ne (i32* @important_val, i32* null),
        i32 0),
      <vscale x 4 x i32> zeroinitializer,
-     <vscale x 4 x i32> insertelement (<vscale x 4 x i32> undef, i32 1, i32 0))
+     <vscale x 4 x i32> insertelement (<vscale x 4 x i32> undef, i32 1, i32 0)
+  ret <vscale x 4 x i32> %s
 }
diff --git a/llvm/test/BugPoint/unsymbolized.ll b/llvm/test/BugPoint/unsymbolized.ll
index d0658b1b6b3a5..b181cffd43703 100644
--- a/llvm/test/BugPoint/unsymbolized.ll
+++ b/llvm/test/BugPoint/unsymbolized.ll
@@ -3,7 +3,7 @@
 ; RUN: echo "print('args = ' + str(sys.argv))" >> %t.py
 ; RUN: echo "exit(1)" >> %t.py
 ; RUN: not bugpoint -load %llvmshlibdir/BugpointPasses%pluginext %s -output-prefix %t -bugpoint-crashcalls -opt-command=%python -opt-args %t.py | FileCheck %s
-; RUN: not --crash opt -enable-new-pm=0 -load %llvmshlibdir/BugpointPasses%pluginext %s -bugpoint-crashcalls -disable-symbolication 2>&1 | FileCheck --check-prefix=CRASH %s
+; RUN: not --crash opt -bugpoint-enable-legacy-pm -load %llvmshlibdir/BugpointPasses%pluginext %s -bugpoint-crashcalls -disable-symbolication 2>&1 | FileCheck --check-prefix=CRASH %s
 ; RUN: not bugpoint -load %llvmshlibdir/BugpointPasses%pluginext %s -output-prefix %t -bugpoint-crashcalls -opt-command=%t.non.existent.opt.binary -opt-args %t.py 2>&1 | FileCheck %s --check-prefix=BAD-OPT
 
 ; Test that bugpoint disables symbolication on the opt tool to reduce runtime overhead when opt crashes
diff --git a/llvm/test/CodeGen/AArch64/abd-combine.ll b/llvm/test/CodeGen/AArch64/abd-combine.ll
index dad3fd08a7b9a..a7e0c26fd7a15 100644
--- a/llvm/test/CodeGen/AArch64/abd-combine.ll
+++ b/llvm/test/CodeGen/AArch64/abd-combine.ll
@@ -107,10 +107,7 @@ define <8 x i16> @abdu_undef(<8 x i16> %src1) {
 define <8 x i16> @abdu_ugt(<8 x i16>, <8 x i16>) {
 ; CHECK-LABEL: abdu_ugt:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmhi v2.8h, v0.8h, v1.8h
-; CHECK-NEXT:    sub v3.8h, v0.8h, v1.8h
-; CHECK-NEXT:    sub v0.8h, v1.8h, v0.8h
-; CHECK-NEXT:    bit v0.16b, v3.16b, v2.16b
+; CHECK-NEXT:    uabd v0.8h, v0.8h, v1.8h
 ; CHECK-NEXT:    ret
   %3 = icmp ugt <8 x i16> %0, %1
   %4 = sub <8 x i16> %0, %1
@@ -122,10 +119,7 @@ define <8 x i16> @abdu_ugt(<8 x i16>, <8 x i16>) {
 define <8 x i16> @abdu_uge(<8 x i16>, <8 x i16>) {
 ; CHECK-LABEL: abdu_uge:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmhs v2.8h, v0.8h, v1.8h
-; CHECK-NEXT:    sub v3.8h, v0.8h, v1.8h
-; CHECK-NEXT:    sub v0.8h, v1.8h, v0.8h
-; CHECK-NEXT:    bit v0.16b, v3.16b, v2.16b
+; CHECK-NEXT:    uabd v0.8h, v0.8h, v1.8h
 ; CHECK-NEXT:    ret
   %3 = icmp uge <8 x i16> %0, %1
   %4 = sub <8 x i16> %0, %1
@@ -137,10 +131,7 @@ define <8 x i16> @abdu_uge(<8 x i16>, <8 x i16>) {
 define <8 x i16> @abdu_ult(<8 x i16>, <8 x i16>) {
 ; CHECK-LABEL: abdu_ult:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmhi v2.8h, v1.8h, v0.8h
-; CHECK-NEXT:    sub v3.8h, v0.8h, v1.8h
-; CHECK-NEXT:    sub v0.8h, v1.8h, v0.8h
-; CHECK-NEXT:    bif v0.16b, v3.16b, v2.16b
+; CHECK-NEXT:    uabd v0.8h, v0.8h, v1.8h
 ; CHECK-NEXT:    ret
   %3 = icmp ult <8 x i16> %0, %1
   %4 = sub <8 x i16> %0, %1
@@ -152,10 +143,7 @@ define <8 x i16> @abdu_ult(<8 x i16>, <8 x i16>) {
 define <8 x i16> @abdu_ule(<8 x i16>, <8 x i16>) {
 ; CHECK-LABEL: abdu_ule:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmhs v2.8h, v1.8h, v0.8h
-; CHECK-NEXT:    sub v3.8h, v0.8h, v1.8h
-; CHECK-NEXT:    sub v0.8h, v1.8h, v0.8h
-; CHECK-NEXT:    bif v0.16b, v3.16b, v2.16b
+; CHECK-NEXT:    uabd v0.8h, v0.8h, v1.8h
 ; CHECK-NEXT:    ret
   %3 = icmp ule <8 x i16> %0, %1
   %4 = sub <8 x i16> %0, %1
@@ -167,10 +155,7 @@ define <8 x i16> @abdu_ule(<8 x i16>, <8 x i16>) {
 define <8 x i16> @abds_sgt(<8 x i16>, <8 x i16>) {
 ; CHECK-LABEL: abds_sgt:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmgt v2.8h, v0.8h, v1.8h
-; CHECK-NEXT:    sub v3.8h, v0.8h, v1.8h
-; CHECK-NEXT:    sub v0.8h, v1.8h, v0.8h
-; CHECK-NEXT:    bit v0.16b, v3.16b, v2.16b
+; CHECK-NEXT:    sabd v0.8h, v0.8h, v1.8h
 ; CHECK-NEXT:    ret
   %3 = icmp sgt <8 x i16> %0, %1
   %4 = sub <8 x i16> %0, %1
@@ -182,10 +167,7 @@ define <8 x i16> @abds_sgt(<8 x i16>, <8 x i16>) {
 define <8 x i16> @abds_sge(<8 x i16>, <8 x i16>) {
 ; CHECK-LABEL: abds_sge:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmge v2.8h, v0.8h, v1.8h
-; CHECK-NEXT:    sub v3.8h, v0.8h, v1.8h
-; CHECK-NEXT:    sub v0.8h, v1.8h, v0.8h
-; CHECK-NEXT:    bit v0.16b, v3.16b, v2.16b
+; CHECK-NEXT:    sabd v0.8h, v0.8h, v1.8h
 ; CHECK-NEXT:    ret
   %3 = icmp sge <8 x i16> %0, %1
   %4 = sub <8 x i16> %0, %1
@@ -197,10 +179,7 @@ define <8 x i16> @abds_sge(<8 x i16>, <8 x i16>) {
 define <8 x i16> @abds_slt(<8 x i16>, <8 x i16>) {
 ; CHECK-LABEL: abds_slt:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmgt v2.8h, v1.8h, v0.8h
-; CHECK-NEXT:    sub v3.8h, v0.8h, v1.8h
-; CHECK-NEXT:    sub v0.8h, v1.8h, v0.8h
-; CHECK-NEXT:    bif v0.16b, v3.16b, v2.16b
+; CHECK-NEXT:    sabd v0.8h, v0.8h, v1.8h
 ; CHECK-NEXT:    ret
   %3 = icmp slt <8 x i16> %0, %1
   %4 = sub <8 x i16> %0, %1
@@ -212,10 +191,7 @@ define <8 x i16> @abds_slt(<8 x i16>, <8 x i16>) {
 define <8 x i16> @abds_sle(<8 x i16>, <8 x i16>) {
 ; CHECK-LABEL: abds_sle:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmge v2.8h, v1.8h, v0.8h
-; CHECK-NEXT:    sub v3.8h, v0.8h, v1.8h
-; CHECK-NEXT:    sub v0.8h, v1.8h, v0.8h
-; CHECK-NEXT:    bif v0.16b, v3.16b, v2.16b
+; CHECK-NEXT:    sabd v0.8h, v0.8h, v1.8h
 ; CHECK-NEXT:    ret
   %3 = icmp sle <8 x i16> %0, %1
   %4 = sub <8 x i16> %0, %1
diff --git a/llvm/test/CodeGen/AArch64/add-i256.ll b/llvm/test/CodeGen/AArch64/add-i256.ll
new file mode 100644
index 0000000000000..e299d1672fe7a
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/add-i256.ll
@@ -0,0 +1,65 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mcpu=neoverse-n1 < %s | FileCheck %s
+target triple = "aarch64-linux-unknown"
+
+define void @add_i256(i64 %x0, i64 %x1, i64 %x2, i64 %x3, i64 %y1, i64 %y2, i64 %y3, i8* %store_addr_ptr) {
+; CHECK-LABEL: add_i256:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    adds x8, x0, #1
+; CHECK-NEXT:    adcs x9, x1, x4
+; CHECK-NEXT:    stp x8, x9, [x7]
+; CHECK-NEXT:    adcs x8, x2, x5
+; CHECK-NEXT:    adc x9, x3, x6
+; CHECK-NEXT:    stp x8, x9, [x7, #16]
+; CHECK-NEXT:    ret
+entry:
+  ; Build x_256 = x0 | x1 << 64 | x2 << 128 | x3 << 192
+  %temp = zext i64 %x0 to i256
+  %temp57 = zext i64 %x1 to i256
+  %temp58 = zext i64 %x2 to i256
+  %temp59 = zext i64 %x3 to i256
+  %temp_shifted = shl i256 %temp, 0
+  %temp_shifted60 = shl i256 %temp57, 64
+  %temp_shifted61 = shl i256 %temp58, 128
+  %temp_shifted62 = shl i256 %temp59, 192
+  %x = or i256 %temp_shifted, %temp_shifted60
+  %x63 = or i256 %x, %temp_shifted61
+  %x_big = or i256 %x63, %temp_shifted62
+
+  ; Build y_256 = 1 | y1 << 64 | y2 << 128 | y3 << 192
+  %temp65 = zext i64 %y1 to i256
+  %temp66 = zext i64 %y2 to i256
+  %temp67 = zext i64 %y3 to i256
+  %temp_shifted68 = shl i256 %temp65, 64
+  %temp_shifted69 = shl i256 %temp66, 128
+  %temp_shifted70 = shl i256 %temp67, 192
+  %y = or i256 1, %temp_shifted68
+  %y71 = or i256 %y, %temp_shifted69
+  %y_big = or i256 %y71, %temp_shifted70
+
+  ; z_256 = x_256 + y_256
+  %z_256 = add i256 %x_big, %y_big
+
+  ; split z_256 into 4 64-bit registers
+  %split_64bits = lshr i256 %z_256, 0
+  %z0 = trunc i256 %split_64bits to i64
+  %split_64bits74 = lshr i256 %z_256, 64
+  %z1 = trunc i256 %split_64bits74 to i64
+  %split_64bits76 = lshr i256 %z_256, 128
+  %z2 = trunc i256 %split_64bits76 to i64
+  %split_64bits78 = lshr i256 %z_256, 192
+  %z3 = trunc i256 %split_64bits78 to i64
+
+  %outptr0 = bitcast i8* %store_addr_ptr to i64*
+  store i64 %z0, i64* %outptr0, align 4
+  %gep = getelementptr i8, i8* %store_addr_ptr, i64 8
+  %outptr1 = bitcast i8* %gep to i64*
+  store i64 %z1, i64* %outptr1, align 4
+  %store_addr_ofs = getelementptr i8, i8* %store_addr_ptr, i64 16
+  %outptr081 = bitcast i8* %store_addr_ofs to i64*
+  store i64 %z2, i64* %outptr081, align 4
+  %gep82 = getelementptr i8, i8* %store_addr_ofs, i64 8
+  %outptr183 = bitcast i8* %gep82 to i64*
+  store i64 %z3, i64* %outptr183, align 4
+  ret void
+}
diff --git a/llvm/test/CodeGen/AArch64/argument-blocks-array-of-struct.ll b/llvm/test/CodeGen/AArch64/argument-blocks-array-of-struct.ll
index e90f89f359ac7..407a8e2120ccc 100644
--- a/llvm/test/CodeGen/AArch64/argument-blocks-array-of-struct.ll
+++ b/llvm/test/CodeGen/AArch64/argument-blocks-array-of-struct.ll
@@ -503,8 +503,8 @@ define void @argument_in_memory() {
 ; CHECK-NEXT:    add x8, x8, :lo12:in_memory_store
 ; CHECK-NEXT:    ldp q0, q1, [x8]
 ; CHECK-NEXT:    ldp q2, q3, [x8, #32]
-; CHECK-NEXT:    stp q0, q1, [sp]
 ; CHECK-NEXT:    ldr d4, [x8, #64]
+; CHECK-NEXT:    stp q0, q1, [sp]
 ; CHECK-NEXT:    stp q2, q3, [sp, #32]
 ; CHECK-NEXT:    str d4, [sp, #64]
 ; CHECK-NEXT:    bl callee_in_memory
diff --git a/llvm/test/CodeGen/AArch64/arm64-zip.ll b/llvm/test/CodeGen/AArch64/arm64-zip.ll
index 02508c2dfdc69..05847394e0f96 100644
--- a/llvm/test/CodeGen/AArch64/arm64-zip.ll
+++ b/llvm/test/CodeGen/AArch64/arm64-zip.ll
@@ -274,3 +274,67 @@ define <16 x i8> @combine_v8i16_8firstundef(<8 x i8> %0, <8 x i8> %1) {
   %3 = shufflevector <8 x i8> %1, <8 x i8> %0, <16 x i32> <i32 8, i32 0, i32 9, i32 1, i32 10, i32 2, i32 11, i32 3, i32 12, i32 4, i32 13, i32 5, i32 14, i32 6, i32 15, i32 undef>
   ret <16 x i8> %3
 }
+
+define <4 x float> @shuffle_zip1(<4 x float> %arg) {
+; CHECK-LABEL: shuffle_zip1:
+; CHECK:       // %bb.0: // %bb
+; CHECK-NEXT:    movi.2d v1, #0000000000000000
+; CHECK-NEXT:    fcmgt.4s v0, v0, v1
+; CHECK-NEXT:    uzp1.8h v1, v0, v0
+; CHECK-NEXT:    xtn.4h v0, v0
+; CHECK-NEXT:    xtn.4h v1, v1
+; CHECK-NEXT:    zip2.4h v0, v0, v1
+; CHECK-NEXT:    fmov.4s v1, #1.00000000
+; CHECK-NEXT:    zip1.4h v0, v0, v0
+; CHECK-NEXT:    sshll.4s v0, v0, #0
+; CHECK-NEXT:    and.16b v0, v1, v0
+; CHECK-NEXT:    ret
+bb:
+  %inst = fcmp olt <4 x float> zeroinitializer, %arg
+  %inst1 = shufflevector <4 x i1> %inst, <4 x i1> zeroinitializer, <2 x i32> <i32 2, i32 0>
+  %inst2 = shufflevector <2 x i1> %inst1, <2 x i1> zeroinitializer, <4 x i32> <i32 0, i32 0, i32 1, i32 1>
+  %inst3 = select <4 x i1> %inst2, <4 x float> <float 1.000000e+00, float 1.000000e+00, float 1.000000e+00, float 1.000000e+00>, <4 x float> zeroinitializer
+  ret <4 x float> %inst3
+}
+
+define <4 x i32> @shuffle_zip2(<4 x i32> %arg) {
+; CHECK-LABEL: shuffle_zip2:
+; CHECK:       // %bb.0: // %bb
+; CHECK-NEXT:    cmtst.4s v0, v0, v0
+; CHECK-NEXT:    uzp1.8h v1, v0, v0
+; CHECK-NEXT:    xtn.4h v0, v0
+; CHECK-NEXT:    xtn.4h v1, v1
+; CHECK-NEXT:    zip2.4h v0, v0, v1
+; CHECK-NEXT:    movi.4s v1, #1
+; CHECK-NEXT:    zip1.4h v0, v0, v0
+; CHECK-NEXT:    ushll.4s v0, v0, #0
+; CHECK-NEXT:    and.16b v0, v0, v1
+; CHECK-NEXT:    ret
+bb:
+  %inst = icmp ult <4 x i32> zeroinitializer, %arg
+  %inst1 = shufflevector <4 x i1> %inst, <4 x i1> zeroinitializer, <2 x i32> <i32 2, i32 0>
+  %inst2 = shufflevector <2 x i1> %inst1, <2 x i1> zeroinitializer, <4 x i32> <i32 0, i32 0, i32 1, i32 1>
+  %inst3 = select <4 x i1> %inst2, <4 x i32> <i32 1, i32 1, i32 1, i32 1>, <4 x i32> zeroinitializer
+  ret <4 x i32> %inst3
+}
+
+define <4 x i32> @shuffle_zip3(<4 x i32> %arg) {
+; CHECK-LABEL: shuffle_zip3:
+; CHECK:       // %bb.0: // %bb
+; CHECK-NEXT:    cmgt.4s v0, v0, #0
+; CHECK-NEXT:    uzp1.8h v1, v0, v0
+; CHECK-NEXT:    xtn.4h v0, v0
+; CHECK-NEXT:    xtn.4h v1, v1
+; CHECK-NEXT:    zip2.4h v0, v0, v1
+; CHECK-NEXT:    movi.4s v1, #1
+; CHECK-NEXT:    zip1.4h v0, v0, v0
+; CHECK-NEXT:    ushll.4s v0, v0, #0
+; CHECK-NEXT:    and.16b v0, v0, v1
+; CHECK-NEXT:    ret
+bb:
+  %inst = icmp slt <4 x i32> zeroinitializer, %arg
+  %inst1 = shufflevector <4 x i1> %inst, <4 x i1> zeroinitializer, <2 x i32> <i32 2, i32 0>
+  %inst2 = shufflevector <2 x i1> %inst1, <2 x i1> zeroinitializer, <4 x i32> <i32 0, i32 0, i32 1, i32 1>
+  %inst3 = select <4 x i1> %inst2, <4 x i32> <i32 1, i32 1, i32 1, i32 1>, <4 x i32> zeroinitializer
+  ret <4 x i32> %inst3
+}
diff --git a/llvm/test/CodeGen/AArch64/complex-deinterleaving-mixed-cases.ll b/llvm/test/CodeGen/AArch64/complex-deinterleaving-mixed-cases.ll
index d8b30fd9776e8..fd989180ae152 100644
--- a/llvm/test/CodeGen/AArch64/complex-deinterleaving-mixed-cases.ll
+++ b/llvm/test/CodeGen/AArch64/complex-deinterleaving-mixed-cases.ll
@@ -353,3 +353,155 @@ entry:
   %interleaved.vec = shufflevector <2 x float> %11, <2 x float> %8, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
   ret <4 x float> %interleaved.vec
 }
+
+; Expected to transform
+define <4 x float> @mul_addequal(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
+; CHECK-LABEL: mul_addequal:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    movi v3.2d, #0000000000000000
+; CHECK-NEXT:    fcmla v3.4s, v0.4s, v1.4s, #0
+; CHECK-NEXT:    fcmla v3.4s, v0.4s, v1.4s, #90
+; CHECK-NEXT:    fadd v0.4s, v3.4s, v2.4s
+; CHECK-NEXT:    ret
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %c.real = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %c.imag = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %6 = fadd fast <2 x float> %5, %c.real
+  %7 = fadd fast <2 x float> %2, %c.imag
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}
+
+; Expected to transform
+define <4 x float> @mul_subequal(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
+; CHECK-LABEL: mul_subequal:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    movi v3.2d, #0000000000000000
+; CHECK-NEXT:    fcmla v3.4s, v0.4s, v1.4s, #0
+; CHECK-NEXT:    fcmla v3.4s, v0.4s, v1.4s, #90
+; CHECK-NEXT:    fsub v0.4s, v3.4s, v2.4s
+; CHECK-NEXT:    ret
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %c.real = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %c.imag = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %6 = fsub fast <2 x float> %5, %c.real
+  %7 = fsub fast <2 x float> %2, %c.imag
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}
+
+
+; Expected to transform
+define <4 x float> @mul_mulequal(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
+; CHECK-LABEL: mul_mulequal:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    movi v3.2d, #0000000000000000
+; CHECK-NEXT:    fcmla v3.4s, v0.4s, v1.4s, #0
+; CHECK-NEXT:    fcmla v3.4s, v0.4s, v1.4s, #90
+; CHECK-NEXT:    fmul v0.4s, v3.4s, v2.4s
+; CHECK-NEXT:    ret
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %c.real = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %c.imag = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %6 = fmul fast <2 x float> %5, %c.real
+  %7 = fmul fast <2 x float> %2, %c.imag
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}
+
+; Expected to not transform
+define <4 x float> @mul_divequal(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
+; CHECK-LABEL: mul_divequal:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ext v3.16b, v1.16b, v1.16b, #8
+; CHECK-NEXT:    ext v4.16b, v0.16b, v0.16b, #8
+; CHECK-NEXT:    ext v16.16b, v2.16b, v2.16b, #8
+; CHECK-NEXT:    zip2 v5.2s, v1.2s, v3.2s
+; CHECK-NEXT:    zip1 v1.2s, v1.2s, v3.2s
+; CHECK-NEXT:    zip2 v6.2s, v0.2s, v4.2s
+; CHECK-NEXT:    zip1 v0.2s, v0.2s, v4.2s
+; CHECK-NEXT:    zip1 v4.2s, v2.2s, v16.2s
+; CHECK-NEXT:    zip2 v2.2s, v2.2s, v16.2s
+; CHECK-NEXT:    fmul v7.2s, v6.2s, v5.2s
+; CHECK-NEXT:    fneg v3.2s, v7.2s
+; CHECK-NEXT:    fmla v3.2s, v0.2s, v1.2s
+; CHECK-NEXT:    fmul v0.2s, v5.2s, v0.2s
+; CHECK-NEXT:    fmla v0.2s, v6.2s, v1.2s
+; CHECK-NEXT:    fdiv v3.2s, v3.2s, v4.2s
+; CHECK-NEXT:    fdiv v0.2s, v0.2s, v2.2s
+; CHECK-NEXT:    zip1 v0.4s, v3.4s, v0.4s
+; CHECK-NEXT:    ret
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %c.real = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %c.imag = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %6 = fdiv fast <2 x float> %5, %c.real
+  %7 = fdiv fast <2 x float> %2, %c.imag
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}
+
+; Expected to transform
+define <4 x float> @mul_negequal(<4 x float> %a, <4 x float> %b) {
+; CHECK-LABEL: mul_negequal:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    movi v2.2d, #0000000000000000
+; CHECK-NEXT:    fcmla v2.4s, v0.4s, v1.4s, #0
+; CHECK-NEXT:    fcmla v2.4s, v0.4s, v1.4s, #90
+; CHECK-NEXT:    fneg v0.4s, v2.4s
+; CHECK-NEXT:    ret
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %6 = fneg fast <2 x float> %5
+  %7 = fneg fast <2 x float> %2
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}
diff --git a/llvm/test/CodeGen/AArch64/custom-sections.ll b/llvm/test/CodeGen/AArch64/custom-sections.ll
new file mode 100644
index 0000000000000..d8d9ed48853b9
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/custom-sections.ll
@@ -0,0 +1,28 @@
+; RUN: llc -mtriple=arm64-apple-macosx %s -o - | FileCheck %s
+
+; CHECK-LABEL: .section __MY_DATA,__my_data
+; CHECK:       .globl _data
+; CHECK: _data:
+; CHECK:       .long 42
+@data = global i32 42 #0
+
+; CHECK-LABEL: .section __MY_BSS,__my_bss
+; CHECK:       .globl _bss
+; CHECK: _bss:
+; CHECK:       .long 0
+@bss = global i32 0 #0
+
+; CHECK-LABEL: .section __MY_RODATA,__my_rodata
+; CHECK:       .globl _const
+; CHECK: _const:
+; CHECK:       .long 42
+@const = constant i32 42 #0
+
+; CHECK-LABEL: .section __MY_RELRO,__my_relro
+; CHECK:       .globl _vars_relro
+; CHECK: _vars_relro:
+; CHECK:       .quad _data
+; CHECK:       .quad _bss
+@vars_relro = hidden constant [2 x ptr] [ptr @data, ptr @bss], align 16 #0
+
+attributes #0 = { "data-section"="__MY_DATA,__my_data" "bss-section"="__MY_BSS,__my_bss" "rodata-section"="__MY_RODATA,__my_rodata" "relro-section"="__MY_RELRO,__my_relro" }
diff --git a/llvm/test/CodeGen/AArch64/expand-subs-pseudo.mir b/llvm/test/CodeGen/AArch64/expand-subs-pseudo.mir
index b09e3b2665280..c4a97649801db 100644
--- a/llvm/test/CodeGen/AArch64/expand-subs-pseudo.mir
+++ b/llvm/test/CodeGen/AArch64/expand-subs-pseudo.mir
@@ -3,8 +3,8 @@
 ---
 # CHECK-LABEL:  name: test
 # CHECK-LABEL:  bb.0:
-# CHECK:          $w5 = SUBSWrs renamable $w3, renamable $w2, 0, implicit-def dead $nzcv
-# CHECK-NEXT:     $w6 = SUBSWrs renamable $w5, renamable $w3, 0, implicit-def $nzcv
+# CHECK:          $w5 = SUBSWrs renamable $w3, renamable $w2, 0, implicit-def dead $nzcv, debug-instr-number 1
+# CHECK-NEXT:     $w6 = SUBSWrs renamable $w5, renamable $w3, 0, implicit-def $nzcv, debug-instr-number 2
 # CHECK-NEXT:     RET undef $lr
 #
 name:            test
@@ -14,8 +14,8 @@ body:             |
   bb.0:
     liveins: $w5, $w6, $x2, $x3
 
-    renamable $w5 = nsw SUBSWrr renamable $w3, renamable $w2, implicit-def dead $nzcv
-    renamable $w6 = nsw SUBSWrr renamable $w5, renamable $w3, implicit-def $nzcv
+    renamable $w5 = nsw SUBSWrr renamable $w3, renamable $w2, implicit-def dead $nzcv, debug-instr-number 1
+    renamable $w6 = nsw SUBSWrr renamable $w5, renamable $w3, implicit-def $nzcv, debug-instr-number 2
     RET_ReallyLR
 
 ...
diff --git a/llvm/test/CodeGen/AArch64/ld1postmul.ll b/llvm/test/CodeGen/AArch64/ld1postmul.ll
new file mode 100644
index 0000000000000..44f6ee093c5ce
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/ld1postmul.ll
@@ -0,0 +1,174 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-none-eabi -o - %s | FileCheck %s --check-prefixes=CHECK,CHECK-NOFP16
+; RUN: llc -mtriple=aarch64-none-eabi -mattr=+fullfp16 -o - %s | FileCheck %s --check-prefixes=CHECK,CHECK-FP16
+
+define ptr @mul_v16i8(ptr %p, ptr %ps, <16 x i8> %t) {
+; CHECK-LABEL: mul_v16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1r { v1.16b }, [x0], #1
+; CHECK-NEXT:    mul v0.16b, v1.16b, v0.16b
+; CHECK-NEXT:    str q0, [x1]
+; CHECK-NEXT:    ret
+  %l = load i8, ptr %p
+  %i = insertelement <16 x i8> undef, i8 %l, i32 0
+  %s = shufflevector <16 x i8> %i, <16 x i8> undef, <16 x i32> zeroinitializer
+  %m = mul <16 x i8> %s, %t
+  store <16 x i8> %m, ptr %ps
+  %g = getelementptr i8, ptr %p, i64 1
+  ret ptr %g
+}
+
+define ptr @mul_v8i16(ptr %p, ptr %ps, <8 x i16> %t) {
+; CHECK-LABEL: mul_v8i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1r { v1.8h }, [x0], #2
+; CHECK-NEXT:    mul v0.8h, v1.8h, v0.8h
+; CHECK-NEXT:    str q0, [x1]
+; CHECK-NEXT:    ret
+  %l = load i16, ptr %p
+  %i = insertelement <8 x i16> undef, i16 %l, i32 0
+  %s = shufflevector <8 x i16> %i, <8 x i16> undef, <8 x i32> zeroinitializer
+  %m = mul <8 x i16> %s, %t
+  store <8 x i16> %m, ptr %ps
+  %g = getelementptr i16, ptr %p, i64 1
+  ret ptr %g
+}
+
+define ptr @mul_v4i32(ptr %p, ptr %ps, <4 x i32> %t) {
+; CHECK-LABEL: mul_v4i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1r { v1.4s }, [x0], #4
+; CHECK-NEXT:    mul v0.4s, v1.4s, v0.4s
+; CHECK-NEXT:    str q0, [x1]
+; CHECK-NEXT:    ret
+  %l = load i32, ptr %p
+  %i = insertelement <4 x i32> undef, i32 %l, i32 0
+  %s = shufflevector <4 x i32> %i, <4 x i32> undef, <4 x i32> zeroinitializer
+  %m = mul <4 x i32> %s, %t
+  store <4 x i32> %m, ptr %ps
+  %g = getelementptr i32, ptr %p, i64 1
+  ret ptr %g
+}
+
+define ptr @fmul_v4f16(ptr %p, ptr %ps, <4 x half> %t) {
+; CHECK-NOFP16-LABEL: fmul_v4f16:
+; CHECK-NOFP16:       // %bb.0:
+; CHECK-NOFP16-NEXT:    ld1r { v1.4h }, [x0], #2
+; CHECK-NOFP16-NEXT:    fcvtl v0.4s, v0.4h
+; CHECK-NOFP16-NEXT:    fcvtl v1.4s, v1.4h
+; CHECK-NOFP16-NEXT:    fmul v0.4s, v1.4s, v0.4s
+; CHECK-NOFP16-NEXT:    fcvtn v0.4h, v0.4s
+; CHECK-NOFP16-NEXT:    str d0, [x1]
+; CHECK-NOFP16-NEXT:    ret
+;
+; CHECK-FP16-LABEL: fmul_v4f16:
+; CHECK-FP16:       // %bb.0:
+; CHECK-FP16-NEXT:    ldr h1, [x0], #2
+; CHECK-FP16-NEXT:    fmul v0.4h, v0.4h, v1.h[0]
+; CHECK-FP16-NEXT:    str d0, [x1]
+; CHECK-FP16-NEXT:    ret
+  %l = load half, ptr %p
+  %i = insertelement <4 x half> undef, half %l, i32 0
+  %s = shufflevector <4 x half> %i, <4 x half> undef, <4 x i32> zeroinitializer
+  %m = fmul <4 x half> %s, %t
+  store <4 x half> %m, ptr %ps
+  %g = getelementptr half, ptr %p, i64 1
+  ret ptr %g
+}
+
+define ptr @fmla_v4f16(ptr %p, ptr %ps, <4 x half> %t, <4 x half> %u) {
+; CHECK-NOFP16-LABEL: fmla_v4f16:
+; CHECK-NOFP16:       // %bb.0:
+; CHECK-NOFP16-NEXT:    ld1r { v2.4h }, [x0], #2
+; CHECK-NOFP16-NEXT:    fcvtl v0.4s, v0.4h
+; CHECK-NOFP16-NEXT:    fcvtl v2.4s, v2.4h
+; CHECK-NOFP16-NEXT:    fcvtl v1.4s, v1.4h
+; CHECK-NOFP16-NEXT:    fmul v0.4s, v2.4s, v0.4s
+; CHECK-NOFP16-NEXT:    fcvtn v0.4h, v0.4s
+; CHECK-NOFP16-NEXT:    fcvtl v0.4s, v0.4h
+; CHECK-NOFP16-NEXT:    fadd v0.4s, v0.4s, v1.4s
+; CHECK-NOFP16-NEXT:    fcvtn v0.4h, v0.4s
+; CHECK-NOFP16-NEXT:    str d0, [x1]
+; CHECK-NOFP16-NEXT:    ret
+;
+; CHECK-FP16-LABEL: fmla_v4f16:
+; CHECK-FP16:       // %bb.0:
+; CHECK-FP16-NEXT:    ldr h2, [x0], #2
+; CHECK-FP16-NEXT:    fmla v1.4h, v0.4h, v2.h[0]
+; CHECK-FP16-NEXT:    str d1, [x1]
+; CHECK-FP16-NEXT:    ret
+  %l = load half, ptr %p
+  %i = insertelement <4 x half> undef, half %l, i32 0
+  %s = shufflevector <4 x half> %i, <4 x half> undef, <4 x i32> zeroinitializer
+  %m = fmul fast <4 x half> %s, %t
+  %a = fadd fast <4 x half> %m, %u
+  store <4 x half> %a, ptr %ps
+  %g = getelementptr half, ptr %p, i64 1
+  ret ptr %g
+}
+
+define ptr @fmul_v4f32(ptr %p, ptr %ps, <4 x float> %t) {
+; CHECK-LABEL: fmul_v4f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr s1, [x0], #4
+; CHECK-NEXT:    fmul v0.4s, v0.4s, v1.s[0]
+; CHECK-NEXT:    str q0, [x1]
+; CHECK-NEXT:    ret
+  %l = load float, ptr %p
+  %i = insertelement <4 x float> undef, float %l, i32 0
+  %s = shufflevector <4 x float> %i, <4 x float> undef, <4 x i32> zeroinitializer
+  %m = fmul <4 x float> %s, %t
+  store <4 x float> %m, ptr %ps
+  %g = getelementptr float, ptr %p, i64 1
+  ret ptr %g
+}
+
+define ptr @fmla_v4f32(ptr %p, ptr %ps, <4 x float> %t, <4 x float> %u) {
+; CHECK-LABEL: fmla_v4f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr s2, [x0], #4
+; CHECK-NEXT:    fmla v1.4s, v0.4s, v2.s[0]
+; CHECK-NEXT:    str q1, [x1]
+; CHECK-NEXT:    ret
+  %l = load float, ptr %p
+  %i = insertelement <4 x float> undef, float %l, i32 0
+  %s = shufflevector <4 x float> %i, <4 x float> undef, <4 x i32> zeroinitializer
+  %m = fmul fast <4 x float> %s, %t
+  %a = fadd fast <4 x float> %m, %u
+  store <4 x float> %a, ptr %ps
+  %g = getelementptr float, ptr %p, i64 1
+  ret ptr %g
+}
+
+define ptr @fmul_v2f64(ptr %p, ptr %ps, <2 x double> %t) {
+; CHECK-LABEL: fmul_v2f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr d1, [x0], #8
+; CHECK-NEXT:    fmul v0.2d, v0.2d, v1.d[0]
+; CHECK-NEXT:    str q0, [x1]
+; CHECK-NEXT:    ret
+  %l = load double, ptr %p
+  %i = insertelement <2 x double> undef, double %l, i32 0
+  %s = shufflevector <2 x double> %i, <2 x double> undef, <2 x i32> zeroinitializer
+  %m = fmul <2 x double> %s, %t
+  store <2 x double> %m, ptr %ps
+  %g = getelementptr double, ptr %p, i64 1
+  ret ptr %g
+}
+
+define ptr @fmla_v2f64(ptr %p, ptr %ps, <2 x double> %t, <2 x double> %u) {
+; CHECK-LABEL: fmla_v2f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr d2, [x0], #8
+; CHECK-NEXT:    fmla v1.2d, v0.2d, v2.d[0]
+; CHECK-NEXT:    str q1, [x1]
+; CHECK-NEXT:    ret
+  %l = load double, ptr %p
+  %i = insertelement <2 x double> undef, double %l, i32 0
+  %s = shufflevector <2 x double> %i, <2 x double> undef, <2 x i32> zeroinitializer
+  %m = fmul fast <2 x double> %s, %t
+  %a = fadd fast <2 x double> %m, %u
+  store <2 x double> %a, ptr %ps
+  %g = getelementptr double, ptr %p, i64 1
+  ret ptr %g
+}
diff --git a/llvm/test/CodeGen/AArch64/ldst-opt-mte-with-dbg.mir b/llvm/test/CodeGen/AArch64/ldst-opt-mte-with-dbg.mir
index 66a8a8e53928d..66b342432c2cd 100644
--- a/llvm/test/CodeGen/AArch64/ldst-opt-mte-with-dbg.mir
+++ b/llvm/test/CodeGen/AArch64/ldst-opt-mte-with-dbg.mir
@@ -14,7 +14,7 @@ body: |
   bb.0.entry:
     liveins: $x0
 
-    STGOffset $x0, $x0, 0
+    STGi $x0, $x0, 0
     DBG_VALUE $x0, 0
     DBG_VALUE $x0, 0
     $x0 = ADDXri $x0, 112, 0
@@ -30,7 +30,7 @@ body: |
   bb.0.entry:
     liveins: $x0, $x1
 
-    STGOffset $x1, $x0, 0
+    STGi $x1, $x0, 0
     DBG_VALUE $x0, 0
     DBG_VALUE $x0, 0
     $x0 = ADDXri $x0, 112, 0
@@ -40,14 +40,14 @@ body: |
 ...
 
 # CHECK-LABEL: name: test_STG_post_unaligned
-# CHECK:      STGOffset $x0, $x0, 0
+# CHECK:      STGi $x0, $x0, 0
 # CHECK-NEXT: ADDXri $x0, 8, 0
 name: test_STG_post_unaligned
 body: |
   bb.0.entry:
     liveins: $x0
 
-    STGOffset $x0, $x0, 0
+    STGi $x0, $x0, 0
     DBG_VALUE $x0, 0
     DBG_VALUE $x0, 0
     $x0 = ADDXri $x0, 8, 0
@@ -63,7 +63,7 @@ body: |
   bb.0.entry:
     liveins: $x0
 
-    STGOffset $x0, $x0, 0
+    STGi $x0, $x0, 0
     DBG_VALUE $x0, 0
     DBG_VALUE $x0, 0
     $x0 = SUBXri $x0, 4096, 0
@@ -73,14 +73,14 @@ body: |
 ...
 
 # CHECK-LABEL: name: test_STG_post3
-# CHECK:      STGOffset $x0, $x0, 0
+# CHECK:      STGi $x0, $x0, 0
 # CHECK-NEXT: SUBXri $x0, 4112, 0
 name: test_STG_post3
 body: |
   bb.0.entry:
     liveins: $x0
 
-    STGOffset $x0, $x0, 0
+    STGi $x0, $x0, 0
     DBG_VALUE $x0, 0
     DBG_VALUE $x0, 0
     $x0 = SUBXri $x0, 4112, 0
@@ -96,7 +96,7 @@ body: |
   bb.0.entry:
     liveins: $x0
 
-    STGOffset $x0, $x0, 0
+    STGi $x0, $x0, 0
     DBG_VALUE $x0, 0
     DBG_VALUE $x0, 0
     $x0 = ADDXri $x0, 4080, 0
@@ -106,14 +106,14 @@ body: |
 ...
 
 # CHECK-LABEL: name: test_STG_post5
-# CHECK:      STGOffset $x0, $x0, 0
+# CHECK:      STGi $x0, $x0, 0
 # CHECK-NEXT: ADDXri $x0, 4096, 0
 name: test_STG_post5
 body: |
   bb.0.entry:
     liveins: $x0
 
-    STGOffset $x0, $x0, 0
+    STGi $x0, $x0, 0
     DBG_VALUE $x0, 0
     DBG_VALUE $x0, 0
     $x0 = ADDXri $x0, 4096, 0
@@ -131,7 +131,7 @@ body: |
   bb.0.entry:
     liveins: $x0
 
-    STZGOffset $x0, $x0, 0
+    STZGi $x0, $x0, 0
     DBG_VALUE $x0, 0
     DBG_VALUE $x0, 0
     $x0 = ADDXri $x0, 112, 0
@@ -147,7 +147,7 @@ body: |
   bb.0.entry:
     liveins: $x0
 
-    ST2GOffset $x0, $x0, 0
+    ST2Gi $x0, $x0, 0
     DBG_VALUE $x0, 0
     DBG_VALUE $x0, 0
     $x0 = ADDXri $x0, 112, 0
@@ -163,7 +163,7 @@ body: |
   bb.0.entry:
     liveins: $x0
 
-    STZ2GOffset $x0, $x0, 0
+    STZ2Gi $x0, $x0, 0
     DBG_VALUE $x0, 0
     DBG_VALUE $x0, 0
     $x0 = ADDXri $x0, 112, 0
@@ -265,7 +265,7 @@ body: |
   bb.0.entry:
     liveins: $x0
 
-    STGOffset $x0, $x0, 10
+    STGi $x0, $x0, 10
     DBG_VALUE $x0, 0
     DBG_VALUE $x0, 0
     $x0 = ADDXri $x0, 160, 0
@@ -302,7 +302,7 @@ body: |
     $x0 = ADDXri $x0, 32, 0
     DBG_VALUE $x0, 0
     DBG_VALUE $x0, 0
-    STGOffset $x0, $x0, 0
+    STGi $x0, $x0, 0
     DBG_VALUE $x0, 0
     DBG_VALUE $x0, 0
     RET_ReallyLR implicit $x0
diff --git a/llvm/test/CodeGen/AArch64/ldst-opt-mte.mir b/llvm/test/CodeGen/AArch64/ldst-opt-mte.mir
index bc24224d897c1..382f79c048a50 100644
--- a/llvm/test/CodeGen/AArch64/ldst-opt-mte.mir
+++ b/llvm/test/CodeGen/AArch64/ldst-opt-mte.mir
@@ -11,7 +11,7 @@ body: |
   bb.0.entry:
     liveins: $x0
 
-    STGOffset $x0, $x0, 0
+    STGi $x0, $x0, 0
     $x0 = ADDXri $x0, 112, 0
     RET_ReallyLR implicit $x0
 ...
@@ -23,20 +23,20 @@ body: |
   bb.0.entry:
     liveins: $x0, $x1
 
-    STGOffset $x1, $x0, 0
+    STGi $x1, $x0, 0
     $x0 = ADDXri $x0, 112, 0
     RET_ReallyLR implicit $x0
 ...
 
 # CHECK-LABEL: name: test_STG_post_unaligned
-# CHECK:      STGOffset $x0, $x0, 0
+# CHECK:      STGi $x0, $x0, 0
 # CHECK-NEXT: ADDXri $x0, 8, 0
 name: test_STG_post_unaligned
 body: |
   bb.0.entry:
     liveins: $x0
 
-    STGOffset $x0, $x0, 0
+    STGi $x0, $x0, 0
     $x0 = ADDXri $x0, 8, 0
     RET_ReallyLR implicit $x0
 ...
@@ -48,20 +48,20 @@ body: |
   bb.0.entry:
     liveins: $x0
 
-    STGOffset $x0, $x0, 0
+    STGi $x0, $x0, 0
     $x0 = SUBXri $x0, 4096, 0
     RET_ReallyLR implicit $x0
 ...
 
 # CHECK-LABEL: name: test_STG_post3
-# CHECK:      STGOffset $x0, $x0, 0
+# CHECK:      STGi $x0, $x0, 0
 # CHECK-NEXT: SUBXri $x0, 4112, 0
 name: test_STG_post3
 body: |
   bb.0.entry:
     liveins: $x0
 
-    STGOffset $x0, $x0, 0
+    STGi $x0, $x0, 0
     $x0 = SUBXri $x0, 4112, 0
     RET_ReallyLR implicit $x0
 ...
@@ -73,20 +73,20 @@ body: |
   bb.0.entry:
     liveins: $x0
 
-    STGOffset $x0, $x0, 0
+    STGi $x0, $x0, 0
     $x0 = ADDXri $x0, 4080, 0
     RET_ReallyLR implicit $x0
 ...
 
 # CHECK-LABEL: name: test_STG_post5
-# CHECK:      STGOffset $x0, $x0, 0
+# CHECK:      STGi $x0, $x0, 0
 # CHECK-NEXT: ADDXri $x0, 4096, 0
 name: test_STG_post5
 body: |
   bb.0.entry:
     liveins: $x0
 
-    STGOffset $x0, $x0, 0
+    STGi $x0, $x0, 0
     $x0 = ADDXri $x0, 4096, 0
     RET_ReallyLR implicit $x0
 ...
@@ -100,7 +100,7 @@ body: |
   bb.0.entry:
     liveins: $x0
 
-    STZGOffset $x0, $x0, 0
+    STZGi $x0, $x0, 0
     $x0 = ADDXri $x0, 112, 0
     RET_ReallyLR implicit $x0
 ...
@@ -112,7 +112,7 @@ body: |
   bb.0.entry:
     liveins: $x0
 
-    ST2GOffset $x0, $x0, 0
+    ST2Gi $x0, $x0, 0
     $x0 = ADDXri $x0, 112, 0
     RET_ReallyLR implicit $x0
 ...
@@ -124,7 +124,7 @@ body: |
   bb.0.entry:
     liveins: $x0
 
-    STZ2GOffset $x0, $x0, 0
+    STZ2Gi $x0, $x0, 0
     $x0 = ADDXri $x0, 112, 0
     RET_ReallyLR implicit $x0
 ...
@@ -202,7 +202,7 @@ body: |
   bb.0.entry:
     liveins: $x0
 
-    STGOffset $x0, $x0, 10
+    STGi $x0, $x0, 10
     $x0 = ADDXri $x0, 160, 0
     RET_ReallyLR implicit $x0
 ...
@@ -229,7 +229,7 @@ body: |
     liveins: $x0
 
     $x0 = ADDXri $x0, 32, 0
-    STGOffset $x0, $x0, 0
+    STGi $x0, $x0, 0
     RET_ReallyLR implicit $x0
 ...
 
diff --git a/llvm/test/CodeGen/AArch64/load-insert-zero.ll b/llvm/test/CodeGen/AArch64/load-insert-zero.ll
index 79eaf2c5b07f9..bc21d8b5201a2 100644
--- a/llvm/test/CodeGen/AArch64/load-insert-zero.ll
+++ b/llvm/test/CodeGen/AArch64/load-insert-zero.ll
@@ -1,5 +1,5 @@
 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc < %s -mtriple=aarch64-none-eabi -mattr=+fullfp16,+bf16 | FileCheck %s
+; RUN: llc < %s -mtriple=aarch64-none-eabi -mattr=+fullfp16,+bf16,+sve | FileCheck %s
 
 define <8 x i8> @loadv8i8(ptr %p) {
 ; CHECK-LABEL: loadv8i8:
@@ -301,6 +301,162 @@ define <2 x double> @loadv2f64_offset(ptr %p) {
 }
 
 
+define <8 x i8> @loadv8i8_noffset(ptr %p) {
+; CHECK-LABEL: loadv8i8_noffset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur b0, [x0, #-1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 -1
+  %l = load i8, ptr %g
+  %v = insertelement <8 x i8> zeroinitializer, i8 %l, i32 0
+  ret <8 x i8> %v
+}
+
+define <16 x i8> @loadv16i8_noffset(ptr %p) {
+; CHECK-LABEL: loadv16i8_noffset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur b0, [x0, #-1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 -1
+  %l = load i8, ptr %g
+  %v = insertelement <16 x i8> zeroinitializer, i8 %l, i32 0
+  ret <16 x i8> %v
+}
+
+define <4 x i16> @loadv4i16_noffset(ptr %p) {
+; CHECK-LABEL: loadv4i16_noffset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur h0, [x0, #-1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 -1
+  %l = load i16, ptr %g
+  %v = insertelement <4 x i16> zeroinitializer, i16 %l, i32 0
+  ret <4 x i16> %v
+}
+
+define <8 x i16> @loadv8i16_noffset(ptr %p) {
+; CHECK-LABEL: loadv8i16_noffset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur h0, [x0, #-1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 -1
+  %l = load i16, ptr %g
+  %v = insertelement <8 x i16> zeroinitializer, i16 %l, i32 0
+  ret <8 x i16> %v
+}
+
+define <2 x i32> @loadv2i32_noffset(ptr %p) {
+; CHECK-LABEL: loadv2i32_noffset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur s0, [x0, #-1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 -1
+  %l = load i32, ptr %g
+  %v = insertelement <2 x i32> zeroinitializer, i32 %l, i32 0
+  ret <2 x i32> %v
+}
+
+define <4 x i32> @loadv4i32_noffset(ptr %p) {
+; CHECK-LABEL: loadv4i32_noffset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur s0, [x0, #-1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 -1
+  %l = load i32, ptr %g
+  %v = insertelement <4 x i32> zeroinitializer, i32 %l, i32 0
+  ret <4 x i32> %v
+}
+
+define <2 x i64> @loadv2i64_noffset(ptr %p) {
+; CHECK-LABEL: loadv2i64_noffset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur d0, [x0, #-1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 -1
+  %l = load i64, ptr %g
+  %v = insertelement <2 x i64> zeroinitializer, i64 %l, i32 0
+  ret <2 x i64> %v
+}
+
+
+define <4 x half> @loadv4f16_noffset(ptr %p) {
+; CHECK-LABEL: loadv4f16_noffset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur h0, [x0, #-1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 -1
+  %l = load half, ptr %g
+  %v = insertelement <4 x half> zeroinitializer, half %l, i32 0
+  ret <4 x half> %v
+}
+
+define <8 x half> @loadv8f16_noffset(ptr %p) {
+; CHECK-LABEL: loadv8f16_noffset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur h0, [x0, #-1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 -1
+  %l = load half, ptr %g
+  %v = insertelement <8 x half> zeroinitializer, half %l, i32 0
+  ret <8 x half> %v
+}
+
+define <4 x bfloat> @loadv4bf16_noffset(ptr %p) {
+; CHECK-LABEL: loadv4bf16_noffset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur h0, [x0, #-1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 -1
+  %l = load bfloat, ptr %g
+  %v = insertelement <4 x bfloat> zeroinitializer, bfloat %l, i32 0
+  ret <4 x bfloat> %v
+}
+
+define <8 x bfloat> @loadv8bf16_noffset(ptr %p) {
+; CHECK-LABEL: loadv8bf16_noffset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur h0, [x0, #-1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 -1
+  %l = load bfloat, ptr %g
+  %v = insertelement <8 x bfloat> zeroinitializer, bfloat %l, i32 0
+  ret <8 x bfloat> %v
+}
+
+define <2 x float> @loadv2f32_noffset(ptr %p) {
+; CHECK-LABEL: loadv2f32_noffset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur s0, [x0, #-1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 -1
+  %l = load float, ptr %g
+  %v = insertelement <2 x float> zeroinitializer, float %l, i32 0
+  ret <2 x float> %v
+}
+
+define <4 x float> @loadv4f32_noffset(ptr %p) {
+; CHECK-LABEL: loadv4f32_noffset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur s0, [x0, #-1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 -1
+  %l = load float, ptr %g
+  %v = insertelement <4 x float> zeroinitializer, float %l, i32 0
+  ret <4 x float> %v
+}
+
+define <2 x double> @loadv2f64_noffset(ptr %p) {
+; CHECK-LABEL: loadv2f64_noffset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur d0, [x0, #-1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 -1
+  %l = load double, ptr %g
+  %v = insertelement <2 x double> zeroinitializer, double %l, i32 0
+  ret <2 x double> %v
+}
+
+
 define void @predictor_4x4_neon(ptr nocapture noundef writeonly %0, i64 noundef %1, ptr nocapture noundef readonly %2, ptr nocapture noundef readnone %3) {
 ; CHECK-LABEL: predictor_4x4_neon:
 ; CHECK:       // %bb.0:
@@ -448,5 +604,366 @@ define void @predictor_4x4_neon_new(ptr nocapture noundef writeonly %0, i64 noun
   ret void
 }
 
+
+define <vscale x 8 x i8> @loadnxv8i8(ptr %p) {
+; CHECK-LABEL: loadnxv8i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldrb w8, [x0]
+; CHECK-NEXT:    ptrue p0.h, vl1
+; CHECK-NEXT:    mov z0.h, #0 // =0x0
+; CHECK-NEXT:    mov z0.h, p0/m, w8
+; CHECK-NEXT:    ret
+  %l = load i8, ptr %p
+  %v = insertelement <vscale x 8 x i8> zeroinitializer, i8 %l, i32 0
+  ret <vscale x 8 x i8> %v
+}
+
+define <vscale x 16 x i8> @loadnxv16i8(ptr %p) {
+; CHECK-LABEL: loadnxv16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr b0, [x0]
+; CHECK-NEXT:    ret
+  %l = load i8, ptr %p
+  %v = insertelement <vscale x 16 x i8> zeroinitializer, i8 %l, i32 0
+  ret <vscale x 16 x i8> %v
+}
+
+define <vscale x 4 x i16> @loadnxv4i16(ptr %p) {
+; CHECK-LABEL: loadnxv4i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldrh w8, [x0]
+; CHECK-NEXT:    ptrue p0.s, vl1
+; CHECK-NEXT:    mov z0.s, #0 // =0x0
+; CHECK-NEXT:    mov z0.s, p0/m, w8
+; CHECK-NEXT:    ret
+  %l = load i16, ptr %p
+  %v = insertelement <vscale x 4 x i16> zeroinitializer, i16 %l, i32 0
+  ret <vscale x 4 x i16> %v
+}
+
+define <vscale x 8 x i16> @loadnxv8i16(ptr %p) {
+; CHECK-LABEL: loadnxv8i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr h0, [x0]
+; CHECK-NEXT:    ret
+  %l = load i16, ptr %p
+  %v = insertelement <vscale x 8 x i16> zeroinitializer, i16 %l, i32 0
+  ret <vscale x 8 x i16> %v
+}
+
+define <vscale x 2 x i32> @loadnxv2i32(ptr %p) {
+; CHECK-LABEL: loadnxv2i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr w8, [x0]
+; CHECK-NEXT:    ptrue p0.d, vl1
+; CHECK-NEXT:    mov z0.d, #0 // =0x0
+; CHECK-NEXT:    mov z0.d, p0/m, x8
+; CHECK-NEXT:    ret
+  %l = load i32, ptr %p
+  %v = insertelement <vscale x 2 x i32> zeroinitializer, i32 %l, i32 0
+  ret <vscale x 2 x i32> %v
+}
+
+define <vscale x 4 x i32> @loadnxv4i32(ptr %p) {
+; CHECK-LABEL: loadnxv4i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr s0, [x0]
+; CHECK-NEXT:    ret
+  %l = load i32, ptr %p
+  %v = insertelement <vscale x 4 x i32> zeroinitializer, i32 %l, i32 0
+  ret <vscale x 4 x i32> %v
+}
+
+define <vscale x 2 x i64> @loadnxv2i64(ptr %p) {
+; CHECK-LABEL: loadnxv2i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr d0, [x0]
+; CHECK-NEXT:    ret
+  %l = load i64, ptr %p
+  %v = insertelement <vscale x 2 x i64> zeroinitializer, i64 %l, i32 0
+  ret <vscale x 2 x i64> %v
+}
+
+
+define <vscale x 4 x half> @loadnxv4f16(ptr %p) {
+; CHECK-LABEL: loadnxv4f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov w8, wzr
+; CHECK-NEXT:    ldr h1, [x0]
+; CHECK-NEXT:    index z0.s, #0, #1
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    mov z2.s, w8
+; CHECK-NEXT:    cmpeq p0.s, p0/z, z0.s, z2.s
+; CHECK-NEXT:    mov z0.h, #0 // =0x0
+; CHECK-NEXT:    mov z0.h, p0/m, h1
+; CHECK-NEXT:    ret
+  %l = load half, ptr %p
+  %v = insertelement <vscale x 4 x half> zeroinitializer, half %l, i32 0
+  ret <vscale x 4 x half> %v
+}
+
+define <vscale x 8 x half> @loadnxv8f16(ptr %p) {
+; CHECK-LABEL: loadnxv8f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr h0, [x0]
+; CHECK-NEXT:    ret
+  %l = load half, ptr %p
+  %v = insertelement <vscale x 8 x half> zeroinitializer, half %l, i32 0
+  ret <vscale x 8 x half> %v
+}
+
+define <vscale x 4 x bfloat> @loadnxv4bf16(ptr %p) {
+; CHECK-LABEL: loadnxv4bf16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov w8, wzr
+; CHECK-NEXT:    ldr h1, [x0]
+; CHECK-NEXT:    index z0.s, #0, #1
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    mov z2.s, w8
+; CHECK-NEXT:    cmpeq p0.s, p0/z, z0.s, z2.s
+; CHECK-NEXT:    mov z0.h, #0 // =0x0
+; CHECK-NEXT:    mov z0.h, p0/m, h1
+; CHECK-NEXT:    ret
+  %l = load bfloat, ptr %p
+  %v = insertelement <vscale x 4 x bfloat> zeroinitializer, bfloat %l, i32 0
+  ret <vscale x 4 x bfloat> %v
+}
+
+define <vscale x 8 x bfloat> @loadnxv8bf16(ptr %p) {
+; CHECK-LABEL: loadnxv8bf16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr h0, [x0]
+; CHECK-NEXT:    ret
+  %l = load bfloat, ptr %p
+  %v = insertelement <vscale x 8 x bfloat> zeroinitializer, bfloat %l, i32 0
+  ret <vscale x 8 x bfloat> %v
+}
+
+define <vscale x 2 x float> @loadnxv2f32(ptr %p) {
+; CHECK-LABEL: loadnxv2f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov x8, xzr
+; CHECK-NEXT:    ldr s1, [x0]
+; CHECK-NEXT:    index z0.d, #0, #1
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    mov z2.d, x8
+; CHECK-NEXT:    cmpeq p0.d, p0/z, z0.d, z2.d
+; CHECK-NEXT:    mov z0.s, #0 // =0x0
+; CHECK-NEXT:    mov z0.s, p0/m, s1
+; CHECK-NEXT:    ret
+  %l = load float, ptr %p
+  %v = insertelement <vscale x 2 x float> zeroinitializer, float %l, i32 0
+  ret <vscale x 2 x float> %v
+}
+
+define <vscale x 4 x float> @loadnxv4f32(ptr %p) {
+; CHECK-LABEL: loadnxv4f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr s0, [x0]
+; CHECK-NEXT:    ret
+  %l = load float, ptr %p
+  %v = insertelement <vscale x 4 x float> zeroinitializer, float %l, i32 0
+  ret <vscale x 4 x float> %v
+}
+
+define <vscale x 2 x double> @loadnxv2f64(ptr %p) {
+; CHECK-LABEL: loadnxv2f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr d0, [x0]
+; CHECK-NEXT:    ret
+  %l = load double, ptr %p
+  %v = insertelement <vscale x 2 x double> zeroinitializer, double %l, i32 0
+  ret <vscale x 2 x double> %v
+}
+
+
+; Unscaled
+
+define <vscale x 8 x i8> @loadnxv8i8_offset(ptr %p) {
+; CHECK-LABEL: loadnxv8i8_offset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldrb w8, [x0, #1]
+; CHECK-NEXT:    ptrue p0.h, vl1
+; CHECK-NEXT:    mov z0.h, #0 // =0x0
+; CHECK-NEXT:    mov z0.h, p0/m, w8
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 1
+  %l = load i8, ptr %g
+  %v = insertelement <vscale x 8 x i8> zeroinitializer, i8 %l, i32 0
+  ret <vscale x 8 x i8> %v
+}
+
+define <vscale x 16 x i8> @loadnxv16i8_offset(ptr %p) {
+; CHECK-LABEL: loadnxv16i8_offset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr b0, [x0, #1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 1
+  %l = load i8, ptr %g
+  %v = insertelement <vscale x 16 x i8> zeroinitializer, i8 %l, i32 0
+  ret <vscale x 16 x i8> %v
+}
+
+define <vscale x 4 x i16> @loadnxv4i16_offset(ptr %p) {
+; CHECK-LABEL: loadnxv4i16_offset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldurh w8, [x0, #1]
+; CHECK-NEXT:    ptrue p0.s, vl1
+; CHECK-NEXT:    mov z0.s, #0 // =0x0
+; CHECK-NEXT:    mov z0.s, p0/m, w8
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 1
+  %l = load i16, ptr %g
+  %v = insertelement <vscale x 4 x i16> zeroinitializer, i16 %l, i32 0
+  ret <vscale x 4 x i16> %v
+}
+
+define <vscale x 8 x i16> @loadnxv8i16_offset(ptr %p) {
+; CHECK-LABEL: loadnxv8i16_offset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur h0, [x0, #1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 1
+  %l = load i16, ptr %g
+  %v = insertelement <vscale x 8 x i16> zeroinitializer, i16 %l, i32 0
+  ret <vscale x 8 x i16> %v
+}
+
+define <vscale x 2 x i32> @loadnxv2i32_offset(ptr %p) {
+; CHECK-LABEL: loadnxv2i32_offset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur w8, [x0, #1]
+; CHECK-NEXT:    ptrue p0.d, vl1
+; CHECK-NEXT:    mov z0.d, #0 // =0x0
+; CHECK-NEXT:    mov z0.d, p0/m, x8
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 1
+  %l = load i32, ptr %g
+  %v = insertelement <vscale x 2 x i32> zeroinitializer, i32 %l, i32 0
+  ret <vscale x 2 x i32> %v
+}
+
+define <vscale x 4 x i32> @loadnxv4i32_offset(ptr %p) {
+; CHECK-LABEL: loadnxv4i32_offset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur s0, [x0, #1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 1
+  %l = load i32, ptr %g
+  %v = insertelement <vscale x 4 x i32> zeroinitializer, i32 %l, i32 0
+  ret <vscale x 4 x i32> %v
+}
+
+define <vscale x 2 x i64> @loadnxv2i64_offset(ptr %p) {
+; CHECK-LABEL: loadnxv2i64_offset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur d0, [x0, #1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 1
+  %l = load i64, ptr %g
+  %v = insertelement <vscale x 2 x i64> zeroinitializer, i64 %l, i32 0
+  ret <vscale x 2 x i64> %v
+}
+
+
+define <vscale x 4 x half> @loadnxv4f16_offset(ptr %p) {
+; CHECK-LABEL: loadnxv4f16_offset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov w8, wzr
+; CHECK-NEXT:    ldur h1, [x0, #1]
+; CHECK-NEXT:    index z0.s, #0, #1
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    mov z2.s, w8
+; CHECK-NEXT:    cmpeq p0.s, p0/z, z0.s, z2.s
+; CHECK-NEXT:    mov z0.h, #0 // =0x0
+; CHECK-NEXT:    mov z0.h, p0/m, h1
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 1
+  %l = load half, ptr %g
+  %v = insertelement <vscale x 4 x half> zeroinitializer, half %l, i32 0
+  ret <vscale x 4 x half> %v
+}
+
+define <vscale x 8 x half> @loadnxv8f16_offset(ptr %p) {
+; CHECK-LABEL: loadnxv8f16_offset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur h0, [x0, #1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 1
+  %l = load half, ptr %g
+  %v = insertelement <vscale x 8 x half> zeroinitializer, half %l, i32 0
+  ret <vscale x 8 x half> %v
+}
+
+define <vscale x 4 x bfloat> @loadnxv4bf16_offset(ptr %p) {
+; CHECK-LABEL: loadnxv4bf16_offset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov w8, wzr
+; CHECK-NEXT:    ldur h1, [x0, #1]
+; CHECK-NEXT:    index z0.s, #0, #1
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    mov z2.s, w8
+; CHECK-NEXT:    cmpeq p0.s, p0/z, z0.s, z2.s
+; CHECK-NEXT:    mov z0.h, #0 // =0x0
+; CHECK-NEXT:    mov z0.h, p0/m, h1
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 1
+  %l = load bfloat, ptr %g
+  %v = insertelement <vscale x 4 x bfloat> zeroinitializer, bfloat %l, i32 0
+  ret <vscale x 4 x bfloat> %v
+}
+
+define <vscale x 8 x bfloat> @loadnxv8bf16_offset(ptr %p) {
+; CHECK-LABEL: loadnxv8bf16_offset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur h0, [x0, #1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 1
+  %l = load bfloat, ptr %g
+  %v = insertelement <vscale x 8 x bfloat> zeroinitializer, bfloat %l, i32 0
+  ret <vscale x 8 x bfloat> %v
+}
+
+define <vscale x 2 x float> @loadnxv2f32_offset(ptr %p) {
+; CHECK-LABEL: loadnxv2f32_offset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov x8, xzr
+; CHECK-NEXT:    ldur s1, [x0, #1]
+; CHECK-NEXT:    index z0.d, #0, #1
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    mov z2.d, x8
+; CHECK-NEXT:    cmpeq p0.d, p0/z, z0.d, z2.d
+; CHECK-NEXT:    mov z0.s, #0 // =0x0
+; CHECK-NEXT:    mov z0.s, p0/m, s1
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 1
+  %l = load float, ptr %g
+  %v = insertelement <vscale x 2 x float> zeroinitializer, float %l, i32 0
+  ret <vscale x 2 x float> %v
+}
+
+define <vscale x 4 x float> @loadnxv4f32_offset(ptr %p) {
+; CHECK-LABEL: loadnxv4f32_offset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur s0, [x0, #1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 1
+  %l = load float, ptr %g
+  %v = insertelement <vscale x 4 x float> zeroinitializer, float %l, i32 0
+  ret <vscale x 4 x float> %v
+}
+
+define <vscale x 2 x double> @loadnxv2f64_offset(ptr %p) {
+; CHECK-LABEL: loadnxv2f64_offset:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur d0, [x0, #1]
+; CHECK-NEXT:    ret
+  %g = getelementptr inbounds i8, ptr %p, i64 1
+  %l = load double, ptr %g
+  %v = insertelement <vscale x 2 x double> zeroinitializer, double %l, i32 0
+  ret <vscale x 2 x double> %v
+}
+
+
 declare <8 x i8> @llvm.aarch64.neon.rshrn.v8i8(<8 x i16>, i32) #1
 declare <8 x i8> @llvm.aarch64.neon.urhadd.v8i8(<8 x i8>, <8 x i8>) #1
diff --git a/llvm/test/CodeGen/AArch64/movw-consts.ll b/llvm/test/CodeGen/AArch64/movw-consts.ll
index d585f74199cc2..a351f0dfcd69a 100644
--- a/llvm/test/CodeGen/AArch64/movw-consts.ll
+++ b/llvm/test/CodeGen/AArch64/movw-consts.ll
@@ -13,7 +13,7 @@ define i64 @test0() {
 define i64 @test1() {
 ; CHECK-LABEL: test1:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    mov w0, #1
+; CHECK-NEXT:    mov w0, #1 ; =0x1
 ; CHECK-NEXT:    ret
   ret i64 1
 }
@@ -21,7 +21,7 @@ define i64 @test1() {
 define i64 @test2() {
 ; CHECK-LABEL: test2:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    mov w0, #65535
+; CHECK-NEXT:    mov w0, #65535 ; =0xffff
 ; CHECK-NEXT:    ret
   ret i64 65535
 }
@@ -29,7 +29,7 @@ define i64 @test2() {
 define i64 @test3() {
 ; CHECK-LABEL: test3:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    mov w0, #65536
+; CHECK-NEXT:    mov w0, #65536 ; =0x10000
 ; CHECK-NEXT:    ret
   ret i64 65536
 }
@@ -37,7 +37,7 @@ define i64 @test3() {
 define i64 @test4() {
 ; CHECK-LABEL: test4:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    mov w0, #-65536
+; CHECK-NEXT:    mov w0, #-65536 ; =0xffff0000
 ; CHECK-NEXT:    ret
   ret i64 4294901760
 }
@@ -45,7 +45,7 @@ define i64 @test4() {
 define i64 @test5() {
 ; CHECK-LABEL: test5:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    mov x0, #4294967296
+; CHECK-NEXT:    mov x0, #4294967296 ; =0x100000000
 ; CHECK-NEXT:    ret
   ret i64 4294967296
 }
@@ -53,7 +53,7 @@ define i64 @test5() {
 define i64 @test6() {
 ; CHECK-LABEL: test6:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    mov x0, #281470681743360
+; CHECK-NEXT:    mov x0, #281470681743360 ; =0xffff00000000
 ; CHECK-NEXT:    ret
   ret i64 281470681743360
 }
@@ -61,7 +61,7 @@ define i64 @test6() {
 define i64 @test7() {
 ; CHECK-LABEL: test7:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    mov x0, #281474976710656
+; CHECK-NEXT:    mov x0, #281474976710656 ; =0x1000000000000
 ; CHECK-NEXT:    ret
   ret i64 281474976710656
 }
@@ -71,7 +71,7 @@ define i64 @test7() {
 define i64 @test8() {
 ; CHECK-LABEL: test8:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    mov w0, #-60876
+; CHECK-NEXT:    mov w0, #-60876 ; =0xffff1234
 ; CHECK-NEXT:    ret
   ret i64 4294906420
 }
@@ -79,7 +79,7 @@ define i64 @test8() {
 define i64 @test9() {
 ; CHECK-LABEL: test9:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    mov x0, #-1
+; CHECK-NEXT:    mov x0, #-1 ; =0xffffffffffffffff
 ; CHECK-NEXT:    ret
   ret i64 -1
 }
@@ -87,7 +87,7 @@ define i64 @test9() {
 define i64 @test10() {
 ; CHECK-LABEL: test10:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    mov x0, #-3989504001
+; CHECK-NEXT:    mov x0, #-3989504001 ; =0xffffffff1234ffff
 ; CHECK-NEXT:    ret
   ret i64 18446744069720047615
 }
@@ -110,7 +110,7 @@ define void @test12() {
 ; CHECK-LABEL: test12:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    adrp x8, _var32@PAGE
-; CHECK-NEXT:    mov w9, #1
+; CHECK-NEXT:    mov w9, #1 ; =0x1
 ; CHECK-NEXT:    str w9, [x8, _var32@PAGEOFF]
 ; CHECK-NEXT:    ret
   store i32 1, ptr @var32
@@ -121,7 +121,7 @@ define void @test13() {
 ; CHECK-LABEL: test13:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    adrp x8, _var32@PAGE
-; CHECK-NEXT:    mov w9, #65535
+; CHECK-NEXT:    mov w9, #65535 ; =0xffff
 ; CHECK-NEXT:    str w9, [x8, _var32@PAGEOFF]
 ; CHECK-NEXT:    ret
   store i32 65535, ptr @var32
@@ -132,7 +132,7 @@ define void @test14() {
 ; CHECK-LABEL: test14:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    adrp x8, _var32@PAGE
-; CHECK-NEXT:    mov w9, #65536
+; CHECK-NEXT:    mov w9, #65536 ; =0x10000
 ; CHECK-NEXT:    str w9, [x8, _var32@PAGEOFF]
 ; CHECK-NEXT:    ret
   store i32 65536, ptr @var32
@@ -143,7 +143,7 @@ define void @test15() {
 ; CHECK-LABEL: test15:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    adrp x8, _var32@PAGE
-; CHECK-NEXT:    mov w9, #-65536
+; CHECK-NEXT:    mov w9, #-65536 ; =0xffff0000
 ; CHECK-NEXT:    str w9, [x8, _var32@PAGEOFF]
 ; CHECK-NEXT:    ret
   store i32 4294901760, ptr @var32
@@ -154,7 +154,7 @@ define void @test16() {
 ; CHECK-LABEL: test16:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    adrp x8, _var32@PAGE
-; CHECK-NEXT:    mov w9, #-1
+; CHECK-NEXT:    mov w9, #-1 ; =0xffffffff
 ; CHECK-NEXT:    str w9, [x8, _var32@PAGEOFF]
 ; CHECK-NEXT:    ret
   store i32 -1, ptr @var32
@@ -164,7 +164,7 @@ define void @test16() {
 define i64 @test17() {
 ; CHECK-LABEL: test17:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    mov x0, #-3
+; CHECK-NEXT:    mov x0, #-3 ; =0xfffffffffffffffd
 ; CHECK-NEXT:    ret
 
   ; Mustn't MOVN w0 here.
diff --git a/llvm/test/CodeGen/AArch64/neon-dotreduce.ll b/llvm/test/CodeGen/AArch64/neon-dotreduce.ll
index 78d169da82114..18537798940fc 100644
--- a/llvm/test/CodeGen/AArch64/neon-dotreduce.ll
+++ b/llvm/test/CodeGen/AArch64/neon-dotreduce.ll
@@ -130,16 +130,15 @@ entry:
 define i32 @test_udot_v5i8(ptr nocapture readonly %a, ptr nocapture readonly %b, i32 %sum) {
 ; CHECK-LABEL: test_udot_v5i8:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ldr d0, [x1]
-; CHECK-NEXT:    ldr d1, [x0]
-; CHECK-NEXT:    ushll v0.8h, v0.8b, #0
+; CHECK-NEXT:    ldr d1, [x1]
+; CHECK-NEXT:    ldr d2, [x0]
+; CHECK-NEXT:    movi v0.2d, #0000000000000000
 ; CHECK-NEXT:    ushll v1.8h, v1.8b, #0
-; CHECK-NEXT:    umull2 v2.4s, v0.8h, v1.8h
-; CHECK-NEXT:    mov v2.s[1], wzr
-; CHECK-NEXT:    mov v2.s[2], wzr
-; CHECK-NEXT:    mov v2.s[3], wzr
-; CHECK-NEXT:    umlal v2.4s, v0.4h, v1.4h
-; CHECK-NEXT:    addv s0, v2.4s
+; CHECK-NEXT:    ushll v2.8h, v2.8b, #0
+; CHECK-NEXT:    umull2 v3.4s, v1.8h, v2.8h
+; CHECK-NEXT:    mov v0.s[0], v3.s[0]
+; CHECK-NEXT:    umlal v0.4s, v1.4h, v2.4h
+; CHECK-NEXT:    addv s0, v0.4s
 ; CHECK-NEXT:    fmov w8, s0
 ; CHECK-NEXT:    add w0, w8, w2
 ; CHECK-NEXT:    ret
@@ -157,13 +156,12 @@ entry:
 define i32 @test_udot_v5i8_nomla(ptr nocapture readonly %a1) {
 ; CHECK-LABEL: test_udot_v5i8_nomla:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ldr d0, [x0]
-; CHECK-NEXT:    ushll v0.8h, v0.8b, #0
-; CHECK-NEXT:    ushll2 v1.4s, v0.8h, #0
-; CHECK-NEXT:    mov v1.s[1], wzr
-; CHECK-NEXT:    mov v1.s[2], wzr
-; CHECK-NEXT:    mov v1.s[3], wzr
-; CHECK-NEXT:    uaddw v0.4s, v1.4s, v0.4h
+; CHECK-NEXT:    ldr d1, [x0]
+; CHECK-NEXT:    movi v0.2d, #0000000000000000
+; CHECK-NEXT:    ushll v1.8h, v1.8b, #0
+; CHECK-NEXT:    ushll2 v2.4s, v1.8h, #0
+; CHECK-NEXT:    mov v0.s[0], v2.s[0]
+; CHECK-NEXT:    uaddw v0.4s, v0.4s, v1.4h
 ; CHECK-NEXT:    addv s0, v0.4s
 ; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
@@ -176,16 +174,15 @@ entry:
 define i32 @test_sdot_v5i8(ptr nocapture readonly %a, ptr nocapture readonly %b, i32 %sum) {
 ; CHECK-LABEL: test_sdot_v5i8:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ldr d0, [x1]
-; CHECK-NEXT:    ldr d1, [x0]
-; CHECK-NEXT:    sshll v0.8h, v0.8b, #0
+; CHECK-NEXT:    ldr d1, [x1]
+; CHECK-NEXT:    ldr d2, [x0]
+; CHECK-NEXT:    movi v0.2d, #0000000000000000
 ; CHECK-NEXT:    sshll v1.8h, v1.8b, #0
-; CHECK-NEXT:    smull2 v2.4s, v0.8h, v1.8h
-; CHECK-NEXT:    mov v2.s[1], wzr
-; CHECK-NEXT:    mov v2.s[2], wzr
-; CHECK-NEXT:    mov v2.s[3], wzr
-; CHECK-NEXT:    smlal v2.4s, v0.4h, v1.4h
-; CHECK-NEXT:    addv s0, v2.4s
+; CHECK-NEXT:    sshll v2.8h, v2.8b, #0
+; CHECK-NEXT:    smull2 v3.4s, v1.8h, v2.8h
+; CHECK-NEXT:    mov v0.s[0], v3.s[0]
+; CHECK-NEXT:    smlal v0.4s, v1.4h, v2.4h
+; CHECK-NEXT:    addv s0, v0.4s
 ; CHECK-NEXT:    fmov w8, s0
 ; CHECK-NEXT:    add w0, w8, w2
 ; CHECK-NEXT:    ret
@@ -207,17 +204,15 @@ define i32 @test_sdot_v5i8_double(<5 x i8> %a, <5 x i8> %b, <5 x i8> %c, <5 x i8
 ; CHECK-NEXT:    sshll v0.8h, v0.8b, #0
 ; CHECK-NEXT:    sshll v1.8h, v1.8b, #0
 ; CHECK-NEXT:    sshll v3.8h, v3.8b, #0
-; CHECK-NEXT:    smull2 v4.4s, v0.8h, v1.8h
-; CHECK-NEXT:    smull2 v5.4s, v2.8h, v3.8h
-; CHECK-NEXT:    mov v4.s[1], wzr
-; CHECK-NEXT:    mov v5.s[1], wzr
-; CHECK-NEXT:    mov v4.s[2], wzr
-; CHECK-NEXT:    mov v5.s[2], wzr
-; CHECK-NEXT:    mov v4.s[3], wzr
-; CHECK-NEXT:    mov v5.s[3], wzr
-; CHECK-NEXT:    smlal v4.4s, v0.4h, v1.4h
-; CHECK-NEXT:    smlal v5.4s, v2.4h, v3.4h
-; CHECK-NEXT:    add v0.4s, v4.4s, v5.4s
+; CHECK-NEXT:    movi v4.2d, #0000000000000000
+; CHECK-NEXT:    smull2 v5.4s, v0.8h, v1.8h
+; CHECK-NEXT:    movi v6.2d, #0000000000000000
+; CHECK-NEXT:    smull2 v7.4s, v2.8h, v3.8h
+; CHECK-NEXT:    mov v6.s[0], v5.s[0]
+; CHECK-NEXT:    mov v4.s[0], v7.s[0]
+; CHECK-NEXT:    smlal v6.4s, v0.4h, v1.4h
+; CHECK-NEXT:    smlal v4.4s, v2.4h, v3.4h
+; CHECK-NEXT:    add v0.4s, v6.4s, v4.4s
 ; CHECK-NEXT:    addv s0, v0.4s
 ; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
@@ -237,18 +232,16 @@ entry:
 define i32 @test_sdot_v5i8_double_nomla(<5 x i8> %a, <5 x i8> %b, <5 x i8> %c, <5 x i8> %d) {
 ; CHECK-LABEL: test_sdot_v5i8_double_nomla:
 ; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    movi v1.2d, #0000000000000000
+; CHECK-NEXT:    movi v3.2d, #0000000000000000
 ; CHECK-NEXT:    sshll v0.8h, v0.8b, #0
-; CHECK-NEXT:    sshll v1.8h, v2.8b, #0
-; CHECK-NEXT:    sshll2 v2.4s, v0.8h, #0
-; CHECK-NEXT:    sshll2 v3.4s, v1.8h, #0
-; CHECK-NEXT:    mov v2.s[1], wzr
-; CHECK-NEXT:    mov v3.s[1], wzr
-; CHECK-NEXT:    mov v2.s[2], wzr
-; CHECK-NEXT:    mov v3.s[2], wzr
-; CHECK-NEXT:    mov v2.s[3], wzr
-; CHECK-NEXT:    mov v3.s[3], wzr
-; CHECK-NEXT:    saddw v0.4s, v2.4s, v0.4h
-; CHECK-NEXT:    saddw v1.4s, v3.4s, v1.4h
+; CHECK-NEXT:    sshll v2.8h, v2.8b, #0
+; CHECK-NEXT:    sshll2 v4.4s, v0.8h, #0
+; CHECK-NEXT:    sshll2 v5.4s, v2.8h, #0
+; CHECK-NEXT:    mov v3.s[0], v4.s[0]
+; CHECK-NEXT:    mov v1.s[0], v5.s[0]
+; CHECK-NEXT:    saddw v0.4s, v3.4s, v0.4h
+; CHECK-NEXT:    saddw v1.4s, v1.4s, v2.4h
 ; CHECK-NEXT:    add v0.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    addv s0, v0.4s
 ; CHECK-NEXT:    fmov w0, s0
@@ -1005,28 +998,27 @@ entry:
 define i32 @test_udot_v25i8(ptr nocapture readonly %a, ptr nocapture readonly %b, i32 %sum) {
 ; CHECK-LABEL: test_udot_v25i8:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ldp q3, q0, [x1]
-; CHECK-NEXT:    ushll v6.8h, v3.8b, #0
-; CHECK-NEXT:    ushll2 v3.8h, v3.16b, #0
-; CHECK-NEXT:    ldp q2, q1, [x0]
-; CHECK-NEXT:    ushll2 v5.8h, v0.16b, #0
-; CHECK-NEXT:    ushll v0.8h, v0.8b, #0
-; CHECK-NEXT:    ushll2 v4.8h, v1.16b, #0
-; CHECK-NEXT:    ushll v1.8h, v1.8b, #0
-; CHECK-NEXT:    umull v4.4s, v5.4h, v4.4h
-; CHECK-NEXT:    ushll2 v5.8h, v2.16b, #0
-; CHECK-NEXT:    ushll v2.8h, v2.8b, #0
-; CHECK-NEXT:    mov v4.s[1], wzr
-; CHECK-NEXT:    umull2 v7.4s, v6.8h, v2.8h
-; CHECK-NEXT:    umull v2.4s, v6.4h, v2.4h
-; CHECK-NEXT:    mov v4.s[2], wzr
-; CHECK-NEXT:    umlal2 v7.4s, v0.8h, v1.8h
-; CHECK-NEXT:    umlal v2.4s, v0.4h, v1.4h
-; CHECK-NEXT:    mov v4.s[3], wzr
-; CHECK-NEXT:    umlal2 v7.4s, v3.8h, v5.8h
-; CHECK-NEXT:    umlal v4.4s, v3.4h, v5.4h
-; CHECK-NEXT:    add v0.4s, v2.4s, v7.4s
-; CHECK-NEXT:    add v0.4s, v0.4s, v4.4s
+; CHECK-NEXT:    ldp q1, q4, [x1]
+; CHECK-NEXT:    movi v0.2d, #0000000000000000
+; CHECK-NEXT:    ldp q3, q2, [x0]
+; CHECK-NEXT:    ushll2 v7.8h, v4.16b, #0
+; CHECK-NEXT:    ushll v4.8h, v4.8b, #0
+; CHECK-NEXT:    ushll2 v6.8h, v3.16b, #0
+; CHECK-NEXT:    ushll v3.8h, v3.8b, #0
+; CHECK-NEXT:    ushll v5.8h, v2.8b, #0
+; CHECK-NEXT:    ushll2 v2.8h, v2.16b, #0
+; CHECK-NEXT:    umull v2.4s, v7.4h, v2.4h
+; CHECK-NEXT:    ushll v7.8h, v1.8b, #0
+; CHECK-NEXT:    ushll2 v1.8h, v1.16b, #0
+; CHECK-NEXT:    umull2 v16.4s, v7.8h, v3.8h
+; CHECK-NEXT:    mov v0.s[0], v2.s[0]
+; CHECK-NEXT:    umull v2.4s, v7.4h, v3.4h
+; CHECK-NEXT:    umlal2 v16.4s, v4.8h, v5.8h
+; CHECK-NEXT:    umlal v0.4s, v1.4h, v6.4h
+; CHECK-NEXT:    umlal v2.4s, v4.4h, v5.4h
+; CHECK-NEXT:    umlal2 v16.4s, v1.8h, v6.8h
+; CHECK-NEXT:    add v0.4s, v2.4s, v0.4s
+; CHECK-NEXT:    add v0.4s, v0.4s, v16.4s
 ; CHECK-NEXT:    addv s0, v0.4s
 ; CHECK-NEXT:    fmov w8, s0
 ; CHECK-NEXT:    add w0, w8, w2
@@ -1045,21 +1037,20 @@ entry:
 define i32 @test_udot_v25i8_nomla(ptr nocapture readonly %a1) {
 ; CHECK-LABEL: test_udot_v25i8_nomla:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ldp q1, q0, [x0]
-; CHECK-NEXT:    ushll v3.8h, v1.8b, #0
-; CHECK-NEXT:    ushll2 v1.8h, v1.16b, #0
-; CHECK-NEXT:    ushll2 v2.8h, v0.16b, #0
-; CHECK-NEXT:    ushll v0.8h, v0.8b, #0
-; CHECK-NEXT:    ushll v2.4s, v2.4h, #0
-; CHECK-NEXT:    uaddl2 v4.4s, v3.8h, v0.8h
-; CHECK-NEXT:    mov v2.s[1], wzr
-; CHECK-NEXT:    uaddl v0.4s, v3.4h, v0.4h
-; CHECK-NEXT:    uaddw2 v3.4s, v4.4s, v1.8h
-; CHECK-NEXT:    mov v2.s[2], wzr
-; CHECK-NEXT:    add v0.4s, v0.4s, v3.4s
-; CHECK-NEXT:    mov v2.s[3], wzr
-; CHECK-NEXT:    uaddw v1.4s, v2.4s, v1.4h
-; CHECK-NEXT:    add v0.4s, v0.4s, v1.4s
+; CHECK-NEXT:    ldp q1, q2, [x0]
+; CHECK-NEXT:    movi v0.2d, #0000000000000000
+; CHECK-NEXT:    ushll2 v3.8h, v1.16b, #0
+; CHECK-NEXT:    ushll v1.8h, v1.8b, #0
+; CHECK-NEXT:    ushll2 v4.8h, v2.16b, #0
+; CHECK-NEXT:    ushll v2.8h, v2.8b, #0
+; CHECK-NEXT:    ushll v4.4s, v4.4h, #0
+; CHECK-NEXT:    uaddl2 v5.4s, v1.8h, v2.8h
+; CHECK-NEXT:    mov v0.s[0], v4.s[0]
+; CHECK-NEXT:    uaddl v1.4s, v1.4h, v2.4h
+; CHECK-NEXT:    uaddw2 v2.4s, v5.4s, v3.8h
+; CHECK-NEXT:    uaddw v0.4s, v0.4s, v3.4h
+; CHECK-NEXT:    add v1.4s, v1.4s, v2.4s
+; CHECK-NEXT:    add v0.4s, v1.4s, v0.4s
 ; CHECK-NEXT:    addv s0, v0.4s
 ; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
@@ -1072,28 +1063,27 @@ entry:
 define i32 @test_sdot_v25i8(ptr nocapture readonly %a, ptr nocapture readonly %b, i32 %sum) {
 ; CHECK-LABEL: test_sdot_v25i8:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ldp q3, q0, [x1]
-; CHECK-NEXT:    sshll v6.8h, v3.8b, #0
-; CHECK-NEXT:    sshll2 v3.8h, v3.16b, #0
-; CHECK-NEXT:    ldp q2, q1, [x0]
-; CHECK-NEXT:    sshll2 v5.8h, v0.16b, #0
-; CHECK-NEXT:    sshll v0.8h, v0.8b, #0
-; CHECK-NEXT:    sshll2 v4.8h, v1.16b, #0
-; CHECK-NEXT:    sshll v1.8h, v1.8b, #0
-; CHECK-NEXT:    smull v4.4s, v5.4h, v4.4h
-; CHECK-NEXT:    sshll2 v5.8h, v2.16b, #0
-; CHECK-NEXT:    sshll v2.8h, v2.8b, #0
-; CHECK-NEXT:    mov v4.s[1], wzr
-; CHECK-NEXT:    smull2 v7.4s, v6.8h, v2.8h
-; CHECK-NEXT:    smull v2.4s, v6.4h, v2.4h
-; CHECK-NEXT:    mov v4.s[2], wzr
-; CHECK-NEXT:    smlal2 v7.4s, v0.8h, v1.8h
-; CHECK-NEXT:    smlal v2.4s, v0.4h, v1.4h
-; CHECK-NEXT:    mov v4.s[3], wzr
-; CHECK-NEXT:    smlal2 v7.4s, v3.8h, v5.8h
-; CHECK-NEXT:    smlal v4.4s, v3.4h, v5.4h
-; CHECK-NEXT:    add v0.4s, v2.4s, v7.4s
-; CHECK-NEXT:    add v0.4s, v0.4s, v4.4s
+; CHECK-NEXT:    ldp q1, q4, [x1]
+; CHECK-NEXT:    movi v0.2d, #0000000000000000
+; CHECK-NEXT:    ldp q3, q2, [x0]
+; CHECK-NEXT:    sshll2 v7.8h, v4.16b, #0
+; CHECK-NEXT:    sshll v4.8h, v4.8b, #0
+; CHECK-NEXT:    sshll2 v6.8h, v3.16b, #0
+; CHECK-NEXT:    sshll v3.8h, v3.8b, #0
+; CHECK-NEXT:    sshll v5.8h, v2.8b, #0
+; CHECK-NEXT:    sshll2 v2.8h, v2.16b, #0
+; CHECK-NEXT:    smull v2.4s, v7.4h, v2.4h
+; CHECK-NEXT:    sshll v7.8h, v1.8b, #0
+; CHECK-NEXT:    sshll2 v1.8h, v1.16b, #0
+; CHECK-NEXT:    smull2 v16.4s, v7.8h, v3.8h
+; CHECK-NEXT:    mov v0.s[0], v2.s[0]
+; CHECK-NEXT:    smull v2.4s, v7.4h, v3.4h
+; CHECK-NEXT:    smlal2 v16.4s, v4.8h, v5.8h
+; CHECK-NEXT:    smlal v0.4s, v1.4h, v6.4h
+; CHECK-NEXT:    smlal v2.4s, v4.4h, v5.4h
+; CHECK-NEXT:    smlal2 v16.4s, v1.8h, v6.8h
+; CHECK-NEXT:    add v0.4s, v2.4s, v0.4s
+; CHECK-NEXT:    add v0.4s, v0.4s, v16.4s
 ; CHECK-NEXT:    addv s0, v0.4s
 ; CHECK-NEXT:    fmov w8, s0
 ; CHECK-NEXT:    add w0, w8, w2
@@ -1118,220 +1108,218 @@ define i32 @test_sdot_v25i8_double(<25 x i8> %a, <25 x i8> %b, <25 x i8> %c, <25
 ; CHECK-NEXT:    ldr b2, [sp, #80]
 ; CHECK-NEXT:    add x8, sp, #88
 ; CHECK-NEXT:    ldr b0, [sp, #16]
-; CHECK-NEXT:    add x9, sp, #96
-; CHECK-NEXT:    ldr b4, [sp, #280]
-; CHECK-NEXT:    add x10, sp, #288
+; CHECK-NEXT:    add x9, sp, #24
+; CHECK-NEXT:    add x10, sp, #40
+; CHECK-NEXT:    add x11, sp, #128
 ; CHECK-NEXT:    ld1 { v2.b }[1], [x8]
-; CHECK-NEXT:    add x8, sp, #24
-; CHECK-NEXT:    add x11, sp, #304
-; CHECK-NEXT:    add x12, sp, #56
-; CHECK-NEXT:    ld1 { v4.b }[1], [x10]
-; CHECK-NEXT:    add x10, sp, #296
-; CHECK-NEXT:    ld1 { v0.b }[1], [x8]
-; CHECK-NEXT:    add x8, sp, #104
-; CHECK-NEXT:    ld1 { v2.b }[2], [x9]
+; CHECK-NEXT:    add x8, sp, #96
+; CHECK-NEXT:    ld1 { v0.b }[1], [x9]
 ; CHECK-NEXT:    add x9, sp, #32
-; CHECK-NEXT:    ldr b1, [sp, #216]
-; CHECK-NEXT:    fmov s6, w0
-; CHECK-NEXT:    ld1 { v4.b }[2], [x10]
-; CHECK-NEXT:    add x10, sp, #128
+; CHECK-NEXT:    ldr b17, [sp, #152]
+; CHECK-NEXT:    fmov s4, w0
+; CHECK-NEXT:    ldr b6, [sp, #280]
+; CHECK-NEXT:    add x12, sp, #224
+; CHECK-NEXT:    ld1 { v2.b }[2], [x8]
+; CHECK-NEXT:    add x8, sp, #104
 ; CHECK-NEXT:    ld1 { v0.b }[2], [x9]
 ; CHECK-NEXT:    add x9, sp, #112
+; CHECK-NEXT:    ldr b1, [sp, #216]
+; CHECK-NEXT:    mov v4.b[1], w1
+; CHECK-NEXT:    ldr b3, [sp, #480]
 ; CHECK-NEXT:    ld1 { v2.b }[3], [x8]
-; CHECK-NEXT:    add x8, sp, #40
-; CHECK-NEXT:    ldr b16, [sp, #152]
-; CHECK-NEXT:    ld1 { v4.b }[3], [x11]
-; CHECK-NEXT:    add x11, sp, #224
-; CHECK-NEXT:    ld1 { v0.b }[3], [x8]
 ; CHECK-NEXT:    add x8, sp, #120
+; CHECK-NEXT:    ld1 { v0.b }[3], [x10]
+; CHECK-NEXT:    add x10, sp, #48
+; CHECK-NEXT:    ld1 { v1.b }[1], [x12]
+; CHECK-NEXT:    mov v4.b[2], w2
+; CHECK-NEXT:    ldr b18, [sp, #352]
 ; CHECK-NEXT:    ld1 { v2.b }[4], [x9]
-; CHECK-NEXT:    add x9, sp, #48
-; CHECK-NEXT:    ld1 { v1.b }[1], [x11]
-; CHECK-NEXT:    add x11, sp, #328
-; CHECK-NEXT:    ldr b3, [sp, #480]
-; CHECK-NEXT:    ld1 { v0.b }[4], [x9]
-; CHECK-NEXT:    add x9, sp, #64
+; CHECK-NEXT:    add x9, sp, #56
+; CHECK-NEXT:    ld1 { v0.b }[4], [x10]
+; CHECK-NEXT:    add x10, sp, #288
+; CHECK-NEXT:    ldr b20, [sp, #680]
+; CHECK-NEXT:    mov v4.b[3], w3
+; CHECK-NEXT:    ldr b5, [sp, #144]
 ; CHECK-NEXT:    ld1 { v2.b }[5], [x8]
+; CHECK-NEXT:    add x8, sp, #160
+; CHECK-NEXT:    ld1 { v0.b }[5], [x9]
+; CHECK-NEXT:    add x9, sp, #64
+; CHECK-NEXT:    ld1 { v6.b }[1], [x10]
+; CHECK-NEXT:    add x10, sp, #296
+; CHECK-NEXT:    ld1 { v17.b }[1], [x8]
 ; CHECK-NEXT:    add x8, sp, #136
-; CHECK-NEXT:    mov v6.b[1], w1
-; CHECK-NEXT:    ldr b7, [sp, #352]
-; CHECK-NEXT:    ldr b19, [sp, #552]
-; CHECK-NEXT:    ld1 { v0.b }[5], [x12]
-; CHECK-NEXT:    add x12, sp, #72
-; CHECK-NEXT:    ld1 { v2.b }[6], [x10]
-; CHECK-NEXT:    add x10, sp, #160
-; CHECK-NEXT:    mov v6.b[2], w2
-; CHECK-NEXT:    ldr b5, [sp, #144]
-; CHECK-NEXT:    ldr b17, [sp, #344]
+; CHECK-NEXT:    ld1 { v2.b }[6], [x11]
+; CHECK-NEXT:    add x11, sp, #320
 ; CHECK-NEXT:    ld1 { v0.b }[6], [x9]
-; CHECK-NEXT:    add x9, sp, #232
-; CHECK-NEXT:    ld1 { v16.b }[1], [x10]
-; CHECK-NEXT:    add x10, sp, #312
+; CHECK-NEXT:    add x9, sp, #72
+; CHECK-NEXT:    ld1 { v6.b }[2], [x10]
+; CHECK-NEXT:    add x10, sp, #304
+; CHECK-NEXT:    mov v4.b[4], w4
 ; CHECK-NEXT:    ld1 { v2.b }[7], [x8]
 ; CHECK-NEXT:    add x8, sp, #168
-; CHECK-NEXT:    ld1 { v1.b }[2], [x9]
-; CHECK-NEXT:    add x9, sp, #240
-; CHECK-NEXT:    ld1 { v4.b }[4], [x10]
-; CHECK-NEXT:    add x10, sp, #320
-; CHECK-NEXT:    ld1 { v16.b }[2], [x8]
+; CHECK-NEXT:    ld1 { v0.b }[7], [x9]
+; CHECK-NEXT:    add x9, sp, #232
+; CHECK-NEXT:    ld1 { v6.b }[3], [x10]
+; CHECK-NEXT:    add x10, sp, #312
+; CHECK-NEXT:    ld1 { v17.b }[2], [x8]
 ; CHECK-NEXT:    add x8, sp, #176
-; CHECK-NEXT:    mov v6.b[3], w3
-; CHECK-NEXT:    ldr b20, [sp, #544]
-; CHECK-NEXT:    ld1 { v1.b }[3], [x9]
-; CHECK-NEXT:    add x9, sp, #248
-; CHECK-NEXT:    ld1 { v4.b }[5], [x10]
-; CHECK-NEXT:    add x10, sp, #488
-; CHECK-NEXT:    ld1 { v16.b }[3], [x8]
+; CHECK-NEXT:    ld1 { v1.b }[2], [x9]
+; CHECK-NEXT:    add x9, sp, #488
+; CHECK-NEXT:    mov v4.b[5], w5
+; CHECK-NEXT:    ld1 { v6.b }[4], [x10]
+; CHECK-NEXT:    add x10, sp, #240
+; CHECK-NEXT:    ld1 { v3.b }[1], [x9]
+; CHECK-NEXT:    add x9, sp, #496
+; CHECK-NEXT:    ld1 { v17.b }[3], [x8]
 ; CHECK-NEXT:    add x8, sp, #184
-; CHECK-NEXT:    mov v6.b[4], w4
-; CHECK-NEXT:    ld1 { v0.b }[7], [x12]
-; CHECK-NEXT:    ld1 { v1.b }[4], [x9]
-; CHECK-NEXT:    add x9, sp, #256
-; CHECK-NEXT:    ld1 { v3.b }[1], [x10]
-; CHECK-NEXT:    add x10, sp, #496
-; CHECK-NEXT:    ld1 { v16.b }[4], [x8]
+; CHECK-NEXT:    ld1 { v1.b }[3], [x10]
+; CHECK-NEXT:    add x10, sp, #248
+; CHECK-NEXT:    mov v4.b[6], w6
+; CHECK-NEXT:    ld1 { v6.b }[5], [x11]
+; CHECK-NEXT:    ld1 { v3.b }[2], [x9]
+; CHECK-NEXT:    add x9, sp, #504
+; CHECK-NEXT:    ld1 { v17.b }[4], [x8]
 ; CHECK-NEXT:    add x8, sp, #192
-; CHECK-NEXT:    mov v6.b[5], w5
-; CHECK-NEXT:    ld1 { v4.b }[6], [x11]
-; CHECK-NEXT:    ld1 { v1.b }[5], [x9]
-; CHECK-NEXT:    add x9, sp, #264
-; CHECK-NEXT:    ld1 { v3.b }[2], [x10]
-; CHECK-NEXT:    add x10, sp, #504
-; CHECK-NEXT:    ld1 { v16.b }[5], [x8]
+; CHECK-NEXT:    ld1 { v1.b }[4], [x10]
+; CHECK-NEXT:    add x10, sp, #256
+; CHECK-NEXT:    add x11, sp, #328
+; CHECK-NEXT:    ld1 { v3.b }[3], [x9]
+; CHECK-NEXT:    add x9, sp, #512
+; CHECK-NEXT:    ld1 { v17.b }[5], [x8]
 ; CHECK-NEXT:    add x8, sp, #200
-; CHECK-NEXT:    mov v6.b[6], w6
+; CHECK-NEXT:    ld1 { v1.b }[5], [x10]
+; CHECK-NEXT:    add x10, sp, #264
+; CHECK-NEXT:    mov v4.b[7], w7
+; CHECK-NEXT:    ld1 { v6.b }[6], [x11]
+; CHECK-NEXT:    ld1 { v3.b }[4], [x9]
+; CHECK-NEXT:    add x9, sp, #520
+; CHECK-NEXT:    ld1 { v17.b }[6], [x8]
+; CHECK-NEXT:    add x8, sp, #208
+; CHECK-NEXT:    ld1 { v1.b }[6], [x10]
 ; CHECK-NEXT:    add x11, sp, #336
-; CHECK-NEXT:    ld1 { v1.b }[6], [x9]
-; CHECK-NEXT:    add x9, sp, #272
-; CHECK-NEXT:    ld1 { v3.b }[3], [x10]
-; CHECK-NEXT:    add x10, sp, #528
-; CHECK-NEXT:    ld1 { v16.b }[6], [x8]
-; CHECK-NEXT:    add x8, sp, #512
-; CHECK-NEXT:    mov v6.b[7], w7
-; CHECK-NEXT:    ld1 { v4.b }[7], [x11]
-; CHECK-NEXT:    ld1 { v1.b }[7], [x9]
-; CHECK-NEXT:    add x9, sp, #208
-; CHECK-NEXT:    ld1 { v3.b }[4], [x8]
-; CHECK-NEXT:    add x8, sp, #520
-; CHECK-NEXT:    sshll v5.8h, v5.8b, #0
+; CHECK-NEXT:    add x10, sp, #272
+; CHECK-NEXT:    ld1 { v3.b }[5], [x9]
+; CHECK-NEXT:    add x9, sp, #536
+; CHECK-NEXT:    ld1 { v17.b }[7], [x8]
+; CHECK-NEXT:    add x8, sp, #528
+; CHECK-NEXT:    sshll v19.8h, v4.8b, #0
+; CHECK-NEXT:    ldr b4, [sp, #416]
+; CHECK-NEXT:    ld1 { v6.b }[7], [x11]
 ; CHECK-NEXT:    add x11, sp, #688
-; CHECK-NEXT:    ld1 { v16.b }[7], [x9]
-; CHECK-NEXT:    add x9, sp, #360
-; CHECK-NEXT:    sshll v6.8h, v6.8b, #0
-; CHECK-NEXT:    ld1 { v3.b }[5], [x8]
-; CHECK-NEXT:    add x8, sp, #368
-; CHECK-NEXT:    ld1 { v7.b }[1], [x9]
-; CHECK-NEXT:    add x9, sp, #560
-; CHECK-NEXT:    sshll v16.8h, v16.8b, #0
+; CHECK-NEXT:    ld1 { v3.b }[6], [x8]
+; CHECK-NEXT:    add x8, sp, #424
+; CHECK-NEXT:    ld1 { v1.b }[7], [x10]
+; CHECK-NEXT:    add x10, sp, #360
+; CHECK-NEXT:    sshll v7.8h, v2.8b, #0
+; CHECK-NEXT:    ldr b2, [sp, #344]
+; CHECK-NEXT:    ld1 { v4.b }[1], [x8]
+; CHECK-NEXT:    add x8, sp, #432
 ; CHECK-NEXT:    sshll v17.8h, v17.8b, #0
+; CHECK-NEXT:    ld1 { v18.b }[1], [x10]
+; CHECK-NEXT:    sshll v16.8h, v6.8b, #0
+; CHECK-NEXT:    ld1 { v3.b }[7], [x9]
+; CHECK-NEXT:    sshll v6.8h, v2.8b, #0
+; CHECK-NEXT:    add x9, sp, #560
+; CHECK-NEXT:    smull v2.4s, v19.4h, v17.4h
+; CHECK-NEXT:    ld1 { v4.b }[2], [x8]
+; CHECK-NEXT:    smull2 v17.4s, v19.8h, v17.8h
+; CHECK-NEXT:    ldr b19, [sp, #552]
+; CHECK-NEXT:    add x8, sp, #368
+; CHECK-NEXT:    add x10, sp, #440
+; CHECK-NEXT:    ld1 { v20.b }[1], [x11]
+; CHECK-NEXT:    add x11, sp, #696
 ; CHECK-NEXT:    ld1 { v19.b }[1], [x9]
 ; CHECK-NEXT:    add x9, sp, #376
-; CHECK-NEXT:    ld1 { v7.b }[2], [x8]
+; CHECK-NEXT:    ld1 { v18.b }[2], [x8]
 ; CHECK-NEXT:    add x8, sp, #568
-; CHECK-NEXT:    smull2 v18.4s, v6.8h, v16.8h
-; CHECK-NEXT:    ld1 { v3.b }[6], [x10]
-; CHECK-NEXT:    smull v6.4s, v6.4h, v16.4h
-; CHECK-NEXT:    ldr b16, [sp, #416]
+; CHECK-NEXT:    ld1 { v4.b }[3], [x10]
+; CHECK-NEXT:    add x10, sp, #448
+; CHECK-NEXT:    ld1 { v20.b }[2], [x11]
+; CHECK-NEXT:    add x11, sp, #704
 ; CHECK-NEXT:    ld1 { v19.b }[2], [x8]
 ; CHECK-NEXT:    add x8, sp, #576
-; CHECK-NEXT:    ld1 { v7.b }[3], [x9]
+; CHECK-NEXT:    ld1 { v18.b }[3], [x9]
 ; CHECK-NEXT:    add x9, sp, #384
-; CHECK-NEXT:    add x10, sp, #424
-; CHECK-NEXT:    sshll v2.8h, v2.8b, #0
+; CHECK-NEXT:    smlal v2.4s, v7.4h, v16.4h
+; CHECK-NEXT:    ld1 { v4.b }[4], [x10]
+; CHECK-NEXT:    smlal2 v17.4s, v7.8h, v16.8h
+; CHECK-NEXT:    ldr b7, [sp, #616]
 ; CHECK-NEXT:    ld1 { v19.b }[3], [x8]
 ; CHECK-NEXT:    add x8, sp, #584
-; CHECK-NEXT:    ld1 { v7.b }[4], [x9]
+; CHECK-NEXT:    ld1 { v18.b }[4], [x9]
 ; CHECK-NEXT:    add x9, sp, #392
-; CHECK-NEXT:    sshll v4.8h, v4.8b, #0
-; CHECK-NEXT:    ld1 { v16.b }[1], [x10]
-; CHECK-NEXT:    smull v5.4s, v5.4h, v17.4h
-; CHECK-NEXT:    ldr b17, [sp, #680]
+; CHECK-NEXT:    add x10, sp, #456
+; CHECK-NEXT:    ld1 { v20.b }[3], [x11]
+; CHECK-NEXT:    add x11, sp, #400
 ; CHECK-NEXT:    ld1 { v19.b }[4], [x8]
 ; CHECK-NEXT:    add x8, sp, #592
-; CHECK-NEXT:    ld1 { v7.b }[5], [x9]
-; CHECK-NEXT:    add x9, sp, #400
-; CHECK-NEXT:    add x10, sp, #432
-; CHECK-NEXT:    ld1 { v17.b }[1], [x11]
-; CHECK-NEXT:    smlal v6.4s, v2.4h, v4.4h
-; CHECK-NEXT:    add x11, sp, #696
+; CHECK-NEXT:    ld1 { v18.b }[5], [x9]
+; CHECK-NEXT:    add x9, sp, #624
+; CHECK-NEXT:    ld1 { v4.b }[5], [x10]
+; CHECK-NEXT:    add x10, sp, #712
+; CHECK-NEXT:    sshll v5.8h, v5.8b, #0
+; CHECK-NEXT:    ld1 { v7.b }[1], [x9]
+; CHECK-NEXT:    add x9, sp, #632
 ; CHECK-NEXT:    ld1 { v19.b }[5], [x8]
 ; CHECK-NEXT:    add x8, sp, #600
-; CHECK-NEXT:    ld1 { v7.b }[6], [x9]
-; CHECK-NEXT:    add x9, sp, #408
-; CHECK-NEXT:    smlal2 v18.4s, v2.8h, v4.8h
-; CHECK-NEXT:    ldr b2, [sp, #616]
-; CHECK-NEXT:    ld1 { v16.b }[2], [x10]
-; CHECK-NEXT:    add x10, sp, #440
+; CHECK-NEXT:    ld1 { v20.b }[4], [x10]
+; CHECK-NEXT:    add x10, sp, #720
+; CHECK-NEXT:    ld1 { v18.b }[6], [x11]
+; CHECK-NEXT:    add x11, sp, #408
+; CHECK-NEXT:    ld1 { v7.b }[2], [x9]
+; CHECK-NEXT:    add x9, sp, #640
 ; CHECK-NEXT:    ld1 { v19.b }[6], [x8]
 ; CHECK-NEXT:    add x8, sp, #608
-; CHECK-NEXT:    ld1 { v7.b }[7], [x9]
-; CHECK-NEXT:    add x9, sp, #624
-; CHECK-NEXT:    ld1 { v17.b }[2], [x11]
-; CHECK-NEXT:    add x11, sp, #536
-; CHECK-NEXT:    ld1 { v16.b }[3], [x10]
-; CHECK-NEXT:    add x10, sp, #704
-; CHECK-NEXT:    ld1 { v2.b }[1], [x9]
-; CHECK-NEXT:    add x9, sp, #632
-; CHECK-NEXT:    ld1 { v19.b }[7], [x8]
-; CHECK-NEXT:    add x8, sp, #448
-; CHECK-NEXT:    ld1 { v17.b }[3], [x10]
-; CHECK-NEXT:    add x10, sp, #712
-; CHECK-NEXT:    sshll v4.8h, v7.8b, #0
-; CHECK-NEXT:    ld1 { v3.b }[7], [x11]
-; CHECK-NEXT:    ld1 { v2.b }[2], [x9]
-; CHECK-NEXT:    add x9, sp, #640
-; CHECK-NEXT:    sshll v7.8h, v19.8b, #0
-; CHECK-NEXT:    ld1 { v16.b }[4], [x8]
-; CHECK-NEXT:    ld1 { v17.b }[4], [x10]
-; CHECK-NEXT:    add x10, sp, #720
-; CHECK-NEXT:    smull2 v19.4s, v4.8h, v7.8h
-; CHECK-NEXT:    add x8, sp, #456
-; CHECK-NEXT:    ld1 { v2.b }[3], [x9]
-; CHECK-NEXT:    add x9, sp, #648
-; CHECK-NEXT:    smull v4.4s, v4.4h, v7.4h
-; CHECK-NEXT:    ldr b7, [sp, #744]
-; CHECK-NEXT:    sshll v20.8h, v20.8b, #0
-; CHECK-NEXT:    ld1 { v17.b }[5], [x10]
+; CHECK-NEXT:    ld1 { v20.b }[5], [x10]
 ; CHECK-NEXT:    add x10, sp, #728
-; CHECK-NEXT:    ld1 { v16.b }[5], [x8]
-; CHECK-NEXT:    sshll v7.8h, v7.8b, #0
-; CHECK-NEXT:    ld1 { v2.b }[4], [x9]
-; CHECK-NEXT:    add x9, sp, #656
-; CHECK-NEXT:    add x8, sp, #464
-; CHECK-NEXT:    smull v7.4s, v20.4h, v7.4h
-; CHECK-NEXT:    ld1 { v17.b }[6], [x10]
-; CHECK-NEXT:    mov v5.s[1], wzr
-; CHECK-NEXT:    add x10, sp, #736
-; CHECK-NEXT:    ld1 { v2.b }[5], [x9]
+; CHECK-NEXT:    ld1 { v18.b }[7], [x11]
+; CHECK-NEXT:    add x11, sp, #464
+; CHECK-NEXT:    ld1 { v7.b }[3], [x9]
 ; CHECK-NEXT:    add x9, sp, #664
-; CHECK-NEXT:    ld1 { v16.b }[6], [x8]
+; CHECK-NEXT:    ld1 { v19.b }[7], [x8]
+; CHECK-NEXT:    add x8, sp, #648
+; CHECK-NEXT:    ld1 { v20.b }[6], [x10]
+; CHECK-NEXT:    add x10, sp, #736
+; CHECK-NEXT:    sshll v16.8h, v18.8b, #0
+; CHECK-NEXT:    ld1 { v4.b }[6], [x11]
+; CHECK-NEXT:    ld1 { v7.b }[4], [x8]
+; CHECK-NEXT:    add x8, sp, #656
+; CHECK-NEXT:    sshll v18.8h, v19.8b, #0
+; CHECK-NEXT:    ld1 { v20.b }[7], [x10]
+; CHECK-NEXT:    smull v19.4s, v16.4h, v18.4h
+; CHECK-NEXT:    ld1 { v7.b }[5], [x8]
 ; CHECK-NEXT:    add x8, sp, #472
-; CHECK-NEXT:    mov v7.s[1], wzr
-; CHECK-NEXT:    ld1 { v17.b }[7], [x10]
-; CHECK-NEXT:    mov v5.s[2], wzr
-; CHECK-NEXT:    ld1 { v2.b }[6], [x9]
-; CHECK-NEXT:    add x9, sp, #672
-; CHECK-NEXT:    ld1 { v16.b }[7], [x8]
-; CHECK-NEXT:    mov v7.s[2], wzr
+; CHECK-NEXT:    smull2 v16.4s, v16.8h, v18.8h
+; CHECK-NEXT:    ldr b18, [sp, #544]
+; CHECK-NEXT:    smull v5.4s, v5.4h, v6.4h
+; CHECK-NEXT:    ldr b6, [sp, #744]
 ; CHECK-NEXT:    sshll v3.8h, v3.8b, #0
-; CHECK-NEXT:    ld1 { v2.b }[7], [x9]
-; CHECK-NEXT:    sshll v17.8h, v17.8b, #0
-; CHECK-NEXT:    mov v5.s[3], wzr
-; CHECK-NEXT:    mov v7.s[3], wzr
-; CHECK-NEXT:    smlal v4.4s, v3.4h, v17.4h
-; CHECK-NEXT:    smlal2 v19.4s, v3.8h, v17.8h
+; CHECK-NEXT:    ld1 { v4.b }[7], [x8]
+; CHECK-NEXT:    sshll v20.8h, v20.8b, #0
+; CHECK-NEXT:    ld1 { v7.b }[6], [x9]
+; CHECK-NEXT:    sshll v18.8h, v18.8b, #0
+; CHECK-NEXT:    add x9, sp, #672
+; CHECK-NEXT:    sshll v6.8h, v6.8b, #0
+; CHECK-NEXT:    smlal v19.4s, v3.4h, v20.4h
+; CHECK-NEXT:    smlal2 v16.4s, v3.8h, v20.8h
+; CHECK-NEXT:    ld1 { v7.b }[7], [x9]
+; CHECK-NEXT:    movi v3.2d, #0000000000000000
+; CHECK-NEXT:    smull v6.4s, v18.4h, v6.4h
+; CHECK-NEXT:    movi v18.2d, #0000000000000000
 ; CHECK-NEXT:    sshll v0.8h, v0.8b, #0
+; CHECK-NEXT:    mov v3.s[0], v5.s[0]
 ; CHECK-NEXT:    sshll v1.8h, v1.8b, #0
-; CHECK-NEXT:    sshll v3.8h, v16.8b, #0
-; CHECK-NEXT:    sshll v2.8h, v2.8b, #0
-; CHECK-NEXT:    smlal v5.4s, v0.4h, v1.4h
-; CHECK-NEXT:    smlal v7.4s, v3.4h, v2.4h
-; CHECK-NEXT:    smlal2 v18.4s, v0.8h, v1.8h
-; CHECK-NEXT:    smlal2 v19.4s, v3.8h, v2.8h
-; CHECK-NEXT:    add v0.4s, v6.4s, v5.4s
-; CHECK-NEXT:    add v1.4s, v4.4s, v7.4s
-; CHECK-NEXT:    add v0.4s, v0.4s, v18.4s
-; CHECK-NEXT:    add v1.4s, v1.4s, v19.4s
+; CHECK-NEXT:    mov v18.s[0], v6.s[0]
+; CHECK-NEXT:    sshll v4.8h, v4.8b, #0
+; CHECK-NEXT:    sshll v5.8h, v7.8b, #0
+; CHECK-NEXT:    smlal v3.4s, v0.4h, v1.4h
+; CHECK-NEXT:    smlal v18.4s, v4.4h, v5.4h
+; CHECK-NEXT:    smlal2 v17.4s, v0.8h, v1.8h
+; CHECK-NEXT:    smlal2 v16.4s, v4.8h, v5.8h
+; CHECK-NEXT:    add v0.4s, v2.4s, v3.4s
+; CHECK-NEXT:    add v1.4s, v19.4s, v18.4s
+; CHECK-NEXT:    add v0.4s, v0.4s, v17.4s
+; CHECK-NEXT:    add v1.4s, v1.4s, v16.4s
 ; CHECK-NEXT:    add v0.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    addv s0, v0.4s
 ; CHECK-NEXT:    fmov w0, s0
@@ -1360,115 +1348,113 @@ define i32 @test_sdot_v25i8_double_nomla(<25 x i8> %a, <25 x i8> %b, <25 x i8> %
 ; CHECK-NEXT:    add x8, sp, #88
 ; CHECK-NEXT:    ldr b2, [sp, #16]
 ; CHECK-NEXT:    add x9, sp, #24
+; CHECK-NEXT:    fmov s1, w0
 ; CHECK-NEXT:    ldr b3, [sp, #480]
-; CHECK-NEXT:    add x10, sp, #488
 ; CHECK-NEXT:    ld1 { v0.b }[1], [x8]
 ; CHECK-NEXT:    add x8, sp, #96
 ; CHECK-NEXT:    ld1 { v2.b }[1], [x9]
 ; CHECK-NEXT:    add x9, sp, #32
-; CHECK-NEXT:    add x11, sp, #120
-; CHECK-NEXT:    ld1 { v3.b }[1], [x10]
-; CHECK-NEXT:    add x10, sp, #496
-; CHECK-NEXT:    add x12, sp, #48
+; CHECK-NEXT:    mov v1.b[1], w1
+; CHECK-NEXT:    add x10, sp, #488
+; CHECK-NEXT:    add x11, sp, #496
+; CHECK-NEXT:    ldr b4, [sp, #352]
 ; CHECK-NEXT:    ld1 { v0.b }[2], [x8]
 ; CHECK-NEXT:    add x8, sp, #104
 ; CHECK-NEXT:    ld1 { v2.b }[2], [x9]
 ; CHECK-NEXT:    add x9, sp, #40
-; CHECK-NEXT:    ldr b4, [sp, #144]
-; CHECK-NEXT:    fmov s1, w0
-; CHECK-NEXT:    ld1 { v3.b }[2], [x10]
-; CHECK-NEXT:    add x10, sp, #64
+; CHECK-NEXT:    ld1 { v3.b }[1], [x10]
+; CHECK-NEXT:    add x10, sp, #48
+; CHECK-NEXT:    mov v1.b[2], w2
+; CHECK-NEXT:    ldr b6, [sp, #416]
 ; CHECK-NEXT:    ld1 { v0.b }[3], [x8]
 ; CHECK-NEXT:    add x8, sp, #112
 ; CHECK-NEXT:    ld1 { v2.b }[3], [x9]
-; CHECK-NEXT:    add x9, sp, #136
-; CHECK-NEXT:    sshll v4.8h, v4.8b, #0
-; CHECK-NEXT:    ldr b6, [sp, #416]
-; CHECK-NEXT:    mov v1.b[1], w1
-; CHECK-NEXT:    ldr b16, [sp, #544]
+; CHECK-NEXT:    add x9, sp, #128
+; CHECK-NEXT:    ld1 { v3.b }[2], [x11]
+; CHECK-NEXT:    add x11, sp, #56
+; CHECK-NEXT:    mov v1.b[3], w3
+; CHECK-NEXT:    add x12, sp, #504
 ; CHECK-NEXT:    ld1 { v0.b }[4], [x8]
-; CHECK-NEXT:    add x8, sp, #128
-; CHECK-NEXT:    ld1 { v2.b }[4], [x12]
-; CHECK-NEXT:    add x12, sp, #56
-; CHECK-NEXT:    sshll v5.4s, v4.4h, #0
-; CHECK-NEXT:    ldr b4, [sp, #352]
-; CHECK-NEXT:    mov v1.b[2], w2
-; CHECK-NEXT:    ld1 { v0.b }[5], [x11]
-; CHECK-NEXT:    add x11, sp, #504
-; CHECK-NEXT:    ld1 { v2.b }[5], [x12]
-; CHECK-NEXT:    sshll v16.8h, v16.8b, #0
-; CHECK-NEXT:    ld1 { v3.b }[3], [x11]
+; CHECK-NEXT:    add x8, sp, #120
+; CHECK-NEXT:    ld1 { v2.b }[4], [x10]
+; CHECK-NEXT:    add x10, sp, #64
+; CHECK-NEXT:    ldr b5, [sp, #144]
+; CHECK-NEXT:    mov v1.b[4], w4
+; CHECK-NEXT:    ld1 { v3.b }[3], [x12]
+; CHECK-NEXT:    ld1 { v0.b }[5], [x8]
+; CHECK-NEXT:    add x8, sp, #136
+; CHECK-NEXT:    ld1 { v2.b }[5], [x11]
 ; CHECK-NEXT:    add x11, sp, #360
-; CHECK-NEXT:    ld1 { v0.b }[6], [x8]
-; CHECK-NEXT:    add x8, sp, #512
+; CHECK-NEXT:    add x12, sp, #72
+; CHECK-NEXT:    mov v1.b[5], w5
+; CHECK-NEXT:    ld1 { v0.b }[6], [x9]
+; CHECK-NEXT:    add x9, sp, #424
+; CHECK-NEXT:    ld1 { v4.b }[1], [x11]
+; CHECK-NEXT:    add x11, sp, #512
 ; CHECK-NEXT:    ld1 { v2.b }[6], [x10]
 ; CHECK-NEXT:    add x10, sp, #368
-; CHECK-NEXT:    ld1 { v4.b }[1], [x11]
-; CHECK-NEXT:    add x11, sp, #432
-; CHECK-NEXT:    ld1 { v3.b }[4], [x8]
-; CHECK-NEXT:    add x8, sp, #520
-; CHECK-NEXT:    ld1 { v0.b }[7], [x9]
-; CHECK-NEXT:    add x9, sp, #424
-; CHECK-NEXT:    mov v1.b[3], w3
-; CHECK-NEXT:    ld1 { v4.b }[2], [x10]
-; CHECK-NEXT:    add x10, sp, #376
 ; CHECK-NEXT:    ld1 { v6.b }[1], [x9]
-; CHECK-NEXT:    add x9, sp, #72
-; CHECK-NEXT:    ld1 { v3.b }[5], [x8]
-; CHECK-NEXT:    add x8, sp, #528
-; CHECK-NEXT:    mov v1.b[4], w4
-; CHECK-NEXT:    ld1 { v2.b }[7], [x9]
-; CHECK-NEXT:    add x9, sp, #440
-; CHECK-NEXT:    ld1 { v6.b }[2], [x11]
-; CHECK-NEXT:    ld1 { v4.b }[3], [x10]
-; CHECK-NEXT:    add x10, sp, #384
-; CHECK-NEXT:    ld1 { v3.b }[6], [x8]
-; CHECK-NEXT:    add x8, sp, #392
-; CHECK-NEXT:    mov v1.b[5], w5
-; CHECK-NEXT:    ld1 { v6.b }[3], [x9]
-; CHECK-NEXT:    add x9, sp, #448
-; CHECK-NEXT:    ld1 { v4.b }[4], [x10]
-; CHECK-NEXT:    add x10, sp, #456
-; CHECK-NEXT:    sshll v16.4s, v16.4h, #0
+; CHECK-NEXT:    add x9, sp, #376
+; CHECK-NEXT:    ld1 { v0.b }[7], [x8]
+; CHECK-NEXT:    add x8, sp, #432
+; CHECK-NEXT:    ld1 { v4.b }[2], [x10]
+; CHECK-NEXT:    add x10, sp, #520
 ; CHECK-NEXT:    mov v1.b[6], w6
-; CHECK-NEXT:    ld1 { v6.b }[4], [x9]
-; CHECK-NEXT:    add x9, sp, #536
-; CHECK-NEXT:    ld1 { v4.b }[5], [x8]
-; CHECK-NEXT:    add x8, sp, #400
-; CHECK-NEXT:    mov v5.s[1], wzr
-; CHECK-NEXT:    mov v16.s[1], wzr
-; CHECK-NEXT:    ld1 { v3.b }[7], [x9]
-; CHECK-NEXT:    ld1 { v6.b }[5], [x10]
-; CHECK-NEXT:    add x10, sp, #464
-; CHECK-NEXT:    ld1 { v4.b }[6], [x8]
-; CHECK-NEXT:    add x8, sp, #408
+; CHECK-NEXT:    ld1 { v2.b }[7], [x12]
+; CHECK-NEXT:    ld1 { v6.b }[2], [x8]
+; CHECK-NEXT:    add x8, sp, #440
+; CHECK-NEXT:    sshll v5.8h, v5.8b, #0
+; CHECK-NEXT:    ld1 { v3.b }[4], [x11]
+; CHECK-NEXT:    ld1 { v4.b }[3], [x9]
+; CHECK-NEXT:    add x9, sp, #384
+; CHECK-NEXT:    movi v7.2d, #0000000000000000
+; CHECK-NEXT:    ld1 { v6.b }[3], [x8]
+; CHECK-NEXT:    add x8, sp, #448
 ; CHECK-NEXT:    mov v1.b[7], w7
-; CHECK-NEXT:    add x9, sp, #472
-; CHECK-NEXT:    mov v5.s[2], wzr
-; CHECK-NEXT:    ld1 { v6.b }[6], [x10]
-; CHECK-NEXT:    mov v16.s[2], wzr
-; CHECK-NEXT:    ld1 { v4.b }[7], [x8]
+; CHECK-NEXT:    ld1 { v3.b }[5], [x10]
+; CHECK-NEXT:    sshll v5.4s, v5.4h, #0
+; CHECK-NEXT:    ld1 { v4.b }[4], [x9]
+; CHECK-NEXT:    add x9, sp, #392
+; CHECK-NEXT:    add x10, sp, #528
+; CHECK-NEXT:    ld1 { v6.b }[4], [x8]
+; CHECK-NEXT:    add x8, sp, #456
+; CHECK-NEXT:    mov v7.s[0], v5.s[0]
 ; CHECK-NEXT:    sshll v0.8h, v0.8b, #0
+; CHECK-NEXT:    ld1 { v4.b }[5], [x9]
 ; CHECK-NEXT:    sshll v1.8h, v1.8b, #0
-; CHECK-NEXT:    ld1 { v6.b }[7], [x9]
-; CHECK-NEXT:    mov v5.s[3], wzr
-; CHECK-NEXT:    mov v16.s[3], wzr
+; CHECK-NEXT:    add x9, sp, #400
+; CHECK-NEXT:    sshll v2.8h, v2.8b, #0
+; CHECK-NEXT:    ld1 { v6.b }[5], [x8]
+; CHECK-NEXT:    add x8, sp, #464
+; CHECK-NEXT:    ld1 { v3.b }[6], [x10]
+; CHECK-NEXT:    saddw v5.4s, v7.4s, v2.4h
+; CHECK-NEXT:    ld1 { v4.b }[6], [x9]
 ; CHECK-NEXT:    saddl v7.4s, v1.4h, v0.4h
+; CHECK-NEXT:    add x10, sp, #536
+; CHECK-NEXT:    add x9, sp, #408
+; CHECK-NEXT:    ld1 { v6.b }[6], [x8]
+; CHECK-NEXT:    add x8, sp, #472
+; CHECK-NEXT:    add v5.4s, v7.4s, v5.4s
+; CHECK-NEXT:    ldr b7, [sp, #544]
 ; CHECK-NEXT:    saddl2 v0.4s, v1.8h, v0.8h
-; CHECK-NEXT:    sshll v1.8h, v3.8b, #0
-; CHECK-NEXT:    sshll v3.8h, v4.8b, #0
-; CHECK-NEXT:    sshll v2.8h, v2.8b, #0
-; CHECK-NEXT:    sshll v4.8h, v6.8b, #0
-; CHECK-NEXT:    saddl v6.4s, v3.4h, v1.4h
-; CHECK-NEXT:    saddl2 v1.4s, v3.8h, v1.8h
-; CHECK-NEXT:    saddw v5.4s, v5.4s, v2.4h
-; CHECK-NEXT:    saddw v3.4s, v16.4s, v4.4h
+; CHECK-NEXT:    ld1 { v3.b }[7], [x10]
+; CHECK-NEXT:    movi v1.2d, #0000000000000000
+; CHECK-NEXT:    ld1 { v4.b }[7], [x9]
+; CHECK-NEXT:    sshll v7.8h, v7.8b, #0
+; CHECK-NEXT:    ld1 { v6.b }[7], [x8]
 ; CHECK-NEXT:    saddw2 v0.4s, v0.4s, v2.8h
-; CHECK-NEXT:    saddw2 v1.4s, v1.4s, v4.8h
-; CHECK-NEXT:    add v5.4s, v7.4s, v5.4s
-; CHECK-NEXT:    add v2.4s, v6.4s, v3.4s
+; CHECK-NEXT:    sshll v7.4s, v7.4h, #0
+; CHECK-NEXT:    sshll v3.8h, v3.8b, #0
+; CHECK-NEXT:    mov v1.s[0], v7.s[0]
+; CHECK-NEXT:    sshll v4.8h, v4.8b, #0
+; CHECK-NEXT:    sshll v6.8h, v6.8b, #0
+; CHECK-NEXT:    saddl v7.4s, v4.4h, v3.4h
+; CHECK-NEXT:    saddl2 v3.4s, v4.8h, v3.8h
+; CHECK-NEXT:    saddw v1.4s, v1.4s, v6.4h
 ; CHECK-NEXT:    add v0.4s, v5.4s, v0.4s
-; CHECK-NEXT:    add v1.4s, v2.4s, v1.4s
+; CHECK-NEXT:    saddw2 v2.4s, v3.4s, v6.8h
+; CHECK-NEXT:    add v1.4s, v7.4s, v1.4s
+; CHECK-NEXT:    add v1.4s, v1.4s, v2.4s
 ; CHECK-NEXT:    add v0.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    addv s0, v0.4s
 ; CHECK-NEXT:    fmov w0, s0
@@ -1601,34 +1587,33 @@ entry:
 define i32 @test_udot_v33i8(ptr nocapture readonly %a, ptr nocapture readonly %b, i32 %sum) {
 ; CHECK-LABEL: test_udot_v33i8:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ldr b0, [x1, #32]
-; CHECK-NEXT:    ldr b1, [x0, #32]
-; CHECK-NEXT:    ldp q2, q3, [x0]
-; CHECK-NEXT:    ushll v0.8h, v0.8b, #0
+; CHECK-NEXT:    ldr b1, [x1, #32]
+; CHECK-NEXT:    ldr b2, [x0, #32]
+; CHECK-NEXT:    ldp q3, q4, [x0]
 ; CHECK-NEXT:    ushll v1.8h, v1.8b, #0
-; CHECK-NEXT:    umull v0.4s, v0.4h, v1.4h
-; CHECK-NEXT:    ushll v5.8h, v2.8b, #0
-; CHECK-NEXT:    ushll2 v4.8h, v3.16b, #0
-; CHECK-NEXT:    ushll v1.8h, v3.8b, #0
-; CHECK-NEXT:    mov v0.s[1], wzr
-; CHECK-NEXT:    ldp q3, q6, [x1]
-; CHECK-NEXT:    ushll2 v2.8h, v2.16b, #0
-; CHECK-NEXT:    mov v0.s[2], wzr
-; CHECK-NEXT:    ushll2 v16.8h, v3.16b, #0
-; CHECK-NEXT:    ushll v3.8h, v3.8b, #0
-; CHECK-NEXT:    umull2 v17.4s, v16.8h, v2.8h
-; CHECK-NEXT:    umull2 v18.4s, v3.8h, v5.8h
-; CHECK-NEXT:    mov v0.s[3], wzr
-; CHECK-NEXT:    ushll2 v7.8h, v6.16b, #0
+; CHECK-NEXT:    ushll v2.8h, v2.8b, #0
+; CHECK-NEXT:    movi v0.2d, #0000000000000000
+; CHECK-NEXT:    umull v1.4s, v1.4h, v2.4h
+; CHECK-NEXT:    ushll v16.8h, v3.8b, #0
+; CHECK-NEXT:    ldp q5, q6, [x1]
+; CHECK-NEXT:    ushll2 v3.8h, v3.16b, #0
+; CHECK-NEXT:    mov v0.s[0], v1.s[0]
+; CHECK-NEXT:    ushll2 v7.8h, v4.16b, #0
+; CHECK-NEXT:    ushll2 v2.8h, v5.16b, #0
+; CHECK-NEXT:    ushll v5.8h, v5.8b, #0
+; CHECK-NEXT:    umull2 v18.4s, v2.8h, v3.8h
+; CHECK-NEXT:    umull2 v1.4s, v5.8h, v16.8h
+; CHECK-NEXT:    ushll v4.8h, v4.8b, #0
+; CHECK-NEXT:    ushll2 v17.8h, v6.16b, #0
 ; CHECK-NEXT:    ushll v6.8h, v6.8b, #0
-; CHECK-NEXT:    umull v2.4s, v16.4h, v2.4h
-; CHECK-NEXT:    umlal2 v17.4s, v7.8h, v4.8h
-; CHECK-NEXT:    umlal2 v18.4s, v6.8h, v1.8h
-; CHECK-NEXT:    umlal v0.4s, v3.4h, v5.4h
-; CHECK-NEXT:    umlal v2.4s, v7.4h, v4.4h
-; CHECK-NEXT:    add v3.4s, v18.4s, v17.4s
-; CHECK-NEXT:    umlal v0.4s, v6.4h, v1.4h
-; CHECK-NEXT:    add v1.4s, v2.4s, v3.4s
+; CHECK-NEXT:    umull v2.4s, v2.4h, v3.4h
+; CHECK-NEXT:    umlal2 v18.4s, v17.8h, v7.8h
+; CHECK-NEXT:    umlal2 v1.4s, v6.8h, v4.8h
+; CHECK-NEXT:    umlal v0.4s, v5.4h, v16.4h
+; CHECK-NEXT:    umlal v2.4s, v17.4h, v7.4h
+; CHECK-NEXT:    add v1.4s, v1.4s, v18.4s
+; CHECK-NEXT:    umlal v0.4s, v6.4h, v4.4h
+; CHECK-NEXT:    add v1.4s, v2.4s, v1.4s
 ; CHECK-NEXT:    add v0.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    addv s0, v0.4s
 ; CHECK-NEXT:    fmov w8, s0
@@ -1648,23 +1633,22 @@ entry:
 define i32 @test_udot_v33i8_nomla(ptr nocapture readonly %a1) {
 ; CHECK-LABEL: test_udot_v33i8_nomla:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ldr b0, [x0, #32]
-; CHECK-NEXT:    ldp q2, q1, [x0]
-; CHECK-NEXT:    ushll v0.8h, v0.8b, #0
-; CHECK-NEXT:    ushll v0.4s, v0.4h, #0
-; CHECK-NEXT:    ushll2 v4.8h, v2.16b, #0
-; CHECK-NEXT:    mov v0.s[1], wzr
-; CHECK-NEXT:    ushll v3.8h, v1.8b, #0
-; CHECK-NEXT:    ushll2 v1.8h, v1.16b, #0
-; CHECK-NEXT:    ushll v2.8h, v2.8b, #0
-; CHECK-NEXT:    mov v0.s[2], wzr
-; CHECK-NEXT:    uaddl2 v5.4s, v4.8h, v1.8h
-; CHECK-NEXT:    uaddl2 v6.4s, v2.8h, v3.8h
-; CHECK-NEXT:    uaddl v1.4s, v4.4h, v1.4h
-; CHECK-NEXT:    mov v0.s[3], wzr
-; CHECK-NEXT:    uaddw v0.4s, v0.4s, v2.4h
-; CHECK-NEXT:    add v2.4s, v6.4s, v5.4s
-; CHECK-NEXT:    uaddw v0.4s, v0.4s, v3.4h
+; CHECK-NEXT:    ldr b1, [x0, #32]
+; CHECK-NEXT:    ldp q3, q2, [x0]
+; CHECK-NEXT:    movi v0.2d, #0000000000000000
+; CHECK-NEXT:    ushll v1.8h, v1.8b, #0
+; CHECK-NEXT:    ushll v1.4s, v1.4h, #0
+; CHECK-NEXT:    ushll2 v5.8h, v3.16b, #0
+; CHECK-NEXT:    mov v0.s[0], v1.s[0]
+; CHECK-NEXT:    ushll v4.8h, v2.8b, #0
+; CHECK-NEXT:    ushll2 v2.8h, v2.16b, #0
+; CHECK-NEXT:    ushll v1.8h, v3.8b, #0
+; CHECK-NEXT:    uaddl2 v3.4s, v5.8h, v2.8h
+; CHECK-NEXT:    uaddl2 v6.4s, v1.8h, v4.8h
+; CHECK-NEXT:    uaddw v0.4s, v0.4s, v1.4h
+; CHECK-NEXT:    uaddl v1.4s, v5.4h, v2.4h
+; CHECK-NEXT:    add v2.4s, v6.4s, v3.4s
+; CHECK-NEXT:    uaddw v0.4s, v0.4s, v4.4h
 ; CHECK-NEXT:    add v1.4s, v1.4s, v2.4s
 ; CHECK-NEXT:    add v0.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    addv s0, v0.4s
@@ -1679,34 +1663,33 @@ entry:
 define i32 @test_sdot_v33i8(ptr nocapture readonly %a, ptr nocapture readonly %b, i32 %sum) {
 ; CHECK-LABEL: test_sdot_v33i8:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ldr b0, [x1, #32]
-; CHECK-NEXT:    ldr b1, [x0, #32]
-; CHECK-NEXT:    ldp q2, q3, [x0]
-; CHECK-NEXT:    sshll v0.8h, v0.8b, #0
+; CHECK-NEXT:    ldr b1, [x1, #32]
+; CHECK-NEXT:    ldr b2, [x0, #32]
+; CHECK-NEXT:    ldp q3, q4, [x0]
 ; CHECK-NEXT:    sshll v1.8h, v1.8b, #0
-; CHECK-NEXT:    smull v0.4s, v0.4h, v1.4h
-; CHECK-NEXT:    sshll v5.8h, v2.8b, #0
-; CHECK-NEXT:    sshll2 v4.8h, v3.16b, #0
-; CHECK-NEXT:    sshll v1.8h, v3.8b, #0
-; CHECK-NEXT:    mov v0.s[1], wzr
-; CHECK-NEXT:    ldp q3, q6, [x1]
-; CHECK-NEXT:    sshll2 v2.8h, v2.16b, #0
-; CHECK-NEXT:    mov v0.s[2], wzr
-; CHECK-NEXT:    sshll2 v16.8h, v3.16b, #0
-; CHECK-NEXT:    sshll v3.8h, v3.8b, #0
-; CHECK-NEXT:    smull2 v17.4s, v16.8h, v2.8h
-; CHECK-NEXT:    smull2 v18.4s, v3.8h, v5.8h
-; CHECK-NEXT:    mov v0.s[3], wzr
-; CHECK-NEXT:    sshll2 v7.8h, v6.16b, #0
+; CHECK-NEXT:    sshll v2.8h, v2.8b, #0
+; CHECK-NEXT:    movi v0.2d, #0000000000000000
+; CHECK-NEXT:    smull v1.4s, v1.4h, v2.4h
+; CHECK-NEXT:    sshll v16.8h, v3.8b, #0
+; CHECK-NEXT:    ldp q5, q6, [x1]
+; CHECK-NEXT:    sshll2 v3.8h, v3.16b, #0
+; CHECK-NEXT:    mov v0.s[0], v1.s[0]
+; CHECK-NEXT:    sshll2 v7.8h, v4.16b, #0
+; CHECK-NEXT:    sshll2 v2.8h, v5.16b, #0
+; CHECK-NEXT:    sshll v5.8h, v5.8b, #0
+; CHECK-NEXT:    smull2 v18.4s, v2.8h, v3.8h
+; CHECK-NEXT:    smull2 v1.4s, v5.8h, v16.8h
+; CHECK-NEXT:    sshll v4.8h, v4.8b, #0
+; CHECK-NEXT:    sshll2 v17.8h, v6.16b, #0
 ; CHECK-NEXT:    sshll v6.8h, v6.8b, #0
-; CHECK-NEXT:    smull v2.4s, v16.4h, v2.4h
-; CHECK-NEXT:    smlal2 v17.4s, v7.8h, v4.8h
-; CHECK-NEXT:    smlal2 v18.4s, v6.8h, v1.8h
-; CHECK-NEXT:    smlal v0.4s, v3.4h, v5.4h
-; CHECK-NEXT:    smlal v2.4s, v7.4h, v4.4h
-; CHECK-NEXT:    add v3.4s, v18.4s, v17.4s
-; CHECK-NEXT:    smlal v0.4s, v6.4h, v1.4h
-; CHECK-NEXT:    add v1.4s, v2.4s, v3.4s
+; CHECK-NEXT:    smull v2.4s, v2.4h, v3.4h
+; CHECK-NEXT:    smlal2 v18.4s, v17.8h, v7.8h
+; CHECK-NEXT:    smlal2 v1.4s, v6.8h, v4.8h
+; CHECK-NEXT:    smlal v0.4s, v5.4h, v16.4h
+; CHECK-NEXT:    smlal v2.4s, v17.4h, v7.4h
+; CHECK-NEXT:    add v1.4s, v1.4s, v18.4s
+; CHECK-NEXT:    smlal v0.4s, v6.4h, v4.4h
+; CHECK-NEXT:    add v1.4s, v2.4s, v1.4s
 ; CHECK-NEXT:    add v0.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    addv s0, v0.4s
 ; CHECK-NEXT:    fmov w8, s0
@@ -1731,291 +1714,289 @@ define i32 @test_sdot_v33i8_double(<33 x i8> %a, <33 x i8> %b, <33 x i8> %c, <33
 ; CHECK-NEXT:    .cfi_offset w29, -16
 ; CHECK-NEXT:    ldr b0, [sp, #80]
 ; CHECK-NEXT:    add x8, sp, #88
+; CHECK-NEXT:    ldr b1, [sp, #144]
+; CHECK-NEXT:    add x9, sp, #96
 ; CHECK-NEXT:    ldr b3, [sp, #16]
-; CHECK-NEXT:    add x9, sp, #152
-; CHECK-NEXT:    ldr b2, [sp, #144]
 ; CHECK-NEXT:    add x10, sp, #104
 ; CHECK-NEXT:    ld1 { v0.b }[1], [x8]
-; CHECK-NEXT:    add x8, sp, #96
+; CHECK-NEXT:    add x8, sp, #152
 ; CHECK-NEXT:    ldr b4, [sp, #344]
-; CHECK-NEXT:    add x11, sp, #368
-; CHECK-NEXT:    ld1 { v2.b }[1], [x9]
-; CHECK-NEXT:    add x9, sp, #160
-; CHECK-NEXT:    ldr b7, [sp, #216]
-; CHECK-NEXT:    fmov s1, w0
-; CHECK-NEXT:    ld1 { v0.b }[2], [x8]
-; CHECK-NEXT:    add x8, sp, #24
+; CHECK-NEXT:    fmov s2, w0
+; CHECK-NEXT:    ldr b6, [sp, #216]
+; CHECK-NEXT:    add x11, sp, #136
+; CHECK-NEXT:    ld1 { v1.b }[1], [x8]
+; CHECK-NEXT:    add x8, sp, #160
+; CHECK-NEXT:    ld1 { v0.b }[2], [x9]
+; CHECK-NEXT:    add x9, sp, #24
+; CHECK-NEXT:    mov v2.b[1], w1
 ; CHECK-NEXT:    ldr b17, [sp, #280]
-; CHECK-NEXT:    ld1 { v2.b }[2], [x9]
-; CHECK-NEXT:    add x9, sp, #168
-; CHECK-NEXT:    ld1 { v3.b }[1], [x8]
-; CHECK-NEXT:    add x8, sp, #32
+; CHECK-NEXT:    ldr b7, [sp, #408]
+; CHECK-NEXT:    ld1 { v1.b }[2], [x8]
+; CHECK-NEXT:    add x8, sp, #168
+; CHECK-NEXT:    ld1 { v3.b }[1], [x9]
+; CHECK-NEXT:    add x9, sp, #32
 ; CHECK-NEXT:    ld1 { v0.b }[3], [x10]
 ; CHECK-NEXT:    add x10, sp, #112
-; CHECK-NEXT:    ldr b16, [sp, #408]
-; CHECK-NEXT:    ld1 { v2.b }[3], [x9]
-; CHECK-NEXT:    add x9, sp, #176
-; CHECK-NEXT:    ld1 { v3.b }[2], [x8]
-; CHECK-NEXT:    add x8, sp, #40
+; CHECK-NEXT:    mov v2.b[2], w2
+; CHECK-NEXT:    ldr b5, [sp, #208]
+; CHECK-NEXT:    ld1 { v1.b }[3], [x8]
+; CHECK-NEXT:    add x8, sp, #176
+; CHECK-NEXT:    ld1 { v3.b }[2], [x9]
+; CHECK-NEXT:    add x9, sp, #40
 ; CHECK-NEXT:    ld1 { v0.b }[4], [x10]
 ; CHECK-NEXT:    add x10, sp, #120
-; CHECK-NEXT:    ldr b5, [sp, #208]
-; CHECK-NEXT:    ld1 { v2.b }[4], [x9]
-; CHECK-NEXT:    add x9, sp, #184
-; CHECK-NEXT:    ld1 { v3.b }[3], [x8]
-; CHECK-NEXT:    add x8, sp, #48
+; CHECK-NEXT:    mov v2.b[3], w3
+; CHECK-NEXT:    ld1 { v1.b }[4], [x8]
+; CHECK-NEXT:    add x8, sp, #184
+; CHECK-NEXT:    ld1 { v3.b }[3], [x9]
+; CHECK-NEXT:    add x9, sp, #128
 ; CHECK-NEXT:    ld1 { v0.b }[5], [x10]
-; CHECK-NEXT:    add x10, sp, #128
-; CHECK-NEXT:    sshll v5.8h, v5.8b, #0
-; CHECK-NEXT:    ldr b20, [sp, #872]
-; CHECK-NEXT:    ld1 { v2.b }[5], [x9]
-; CHECK-NEXT:    add x9, sp, #192
-; CHECK-NEXT:    ld1 { v3.b }[4], [x8]
-; CHECK-NEXT:    add x8, sp, #56
-; CHECK-NEXT:    ld1 { v0.b }[6], [x10]
-; CHECK-NEXT:    add x10, sp, #64
-; CHECK-NEXT:    mov v1.b[1], w1
-; CHECK-NEXT:    ldr b21, [sp, #744]
-; CHECK-NEXT:    ld1 { v2.b }[6], [x9]
-; CHECK-NEXT:    add x9, sp, #200
-; CHECK-NEXT:    ld1 { v3.b }[5], [x8]
-; CHECK-NEXT:    add x8, sp, #136
-; CHECK-NEXT:    mov v1.b[2], w2
-; CHECK-NEXT:    ld1 { v0.b }[7], [x8]
-; CHECK-NEXT:    add x8, sp, #72
-; CHECK-NEXT:    ld1 { v3.b }[6], [x10]
-; CHECK-NEXT:    add x10, sp, #288
-; CHECK-NEXT:    ld1 { v2.b }[7], [x9]
-; CHECK-NEXT:    add x9, sp, #224
-; CHECK-NEXT:    mov v1.b[3], w3
-; CHECK-NEXT:    ld1 { v17.b }[1], [x10]
-; CHECK-NEXT:    add x10, sp, #240
-; CHECK-NEXT:    ld1 { v3.b }[7], [x8]
+; CHECK-NEXT:    add x10, sp, #48
+; CHECK-NEXT:    mov v2.b[4], w4
+; CHECK-NEXT:    ld1 { v1.b }[5], [x8]
+; CHECK-NEXT:    add x8, sp, #192
+; CHECK-NEXT:    ld1 { v3.b }[4], [x10]
+; CHECK-NEXT:    add x10, sp, #360
+; CHECK-NEXT:    ld1 { v0.b }[6], [x9]
+; CHECK-NEXT:    add x9, sp, #56
+; CHECK-NEXT:    mov v2.b[5], w5
+; CHECK-NEXT:    ld1 { v1.b }[6], [x8]
+; CHECK-NEXT:    add x8, sp, #200
+; CHECK-NEXT:    ld1 { v3.b }[5], [x9]
+; CHECK-NEXT:    add x9, sp, #64
+; CHECK-NEXT:    ld1 { v0.b }[7], [x11]
+; CHECK-NEXT:    add x11, sp, #232
+; CHECK-NEXT:    mov v2.b[6], w6
+; CHECK-NEXT:    ld1 { v1.b }[7], [x8]
 ; CHECK-NEXT:    add x8, sp, #352
-; CHECK-NEXT:    ld1 { v7.b }[1], [x9]
-; CHECK-NEXT:    add x9, sp, #416
-; CHECK-NEXT:    sshll v19.8h, v2.8b, #0
-; CHECK-NEXT:    ldr b2, [sp, #472]
+; CHECK-NEXT:    ld1 { v3.b }[6], [x9]
+; CHECK-NEXT:    add x9, sp, #72
+; CHECK-NEXT:    sshll v5.8h, v5.8b, #0
 ; CHECK-NEXT:    ld1 { v4.b }[1], [x8]
-; CHECK-NEXT:    add x8, sp, #360
-; CHECK-NEXT:    ld1 { v16.b }[1], [x9]
+; CHECK-NEXT:    add x8, sp, #224
+; CHECK-NEXT:    mov v2.b[7], w7
+; CHECK-NEXT:    ld1 { v3.b }[7], [x9]
+; CHECK-NEXT:    add x9, sp, #416
+; CHECK-NEXT:    ld1 { v6.b }[1], [x8]
+; CHECK-NEXT:    add x8, sp, #288
+; CHECK-NEXT:    ld1 { v4.b }[2], [x10]
+; CHECK-NEXT:    add x10, sp, #368
+; CHECK-NEXT:    ld1 { v7.b }[1], [x9]
 ; CHECK-NEXT:    add x9, sp, #424
-; CHECK-NEXT:    sshll v3.8h, v3.8b, #0
-; CHECK-NEXT:    mov v1.b[4], w4
-; CHECK-NEXT:    ld1 { v4.b }[2], [x8]
-; CHECK-NEXT:    add x8, sp, #232
-; CHECK-NEXT:    ld1 { v16.b }[2], [x9]
-; CHECK-NEXT:    add x9, sp, #432
-; CHECK-NEXT:    sshll v0.8h, v0.8b, #0
-; CHECK-NEXT:    ld1 { v7.b }[2], [x8]
+; CHECK-NEXT:    ld1 { v17.b }[1], [x8]
 ; CHECK-NEXT:    add x8, sp, #296
-; CHECK-NEXT:    ld1 { v4.b }[3], [x11]
-; CHECK-NEXT:    add x11, sp, #376
-; CHECK-NEXT:    ld1 { v16.b }[3], [x9]
-; CHECK-NEXT:    add x9, sp, #440
+; CHECK-NEXT:    ld1 { v6.b }[2], [x11]
+; CHECK-NEXT:    add x11, sp, #240
+; CHECK-NEXT:    ld1 { v4.b }[3], [x10]
+; CHECK-NEXT:    add x10, sp, #376
+; CHECK-NEXT:    ld1 { v7.b }[2], [x9]
+; CHECK-NEXT:    add x9, sp, #432
 ; CHECK-NEXT:    ld1 { v17.b }[2], [x8]
 ; CHECK-NEXT:    add x8, sp, #304
-; CHECK-NEXT:    ld1 { v7.b }[3], [x10]
-; CHECK-NEXT:    add x10, sp, #248
-; CHECK-NEXT:    ld1 { v4.b }[4], [x11]
-; CHECK-NEXT:    add x11, sp, #384
-; CHECK-NEXT:    ld1 { v16.b }[4], [x9]
-; CHECK-NEXT:    add x9, sp, #448
+; CHECK-NEXT:    ld1 { v6.b }[3], [x11]
+; CHECK-NEXT:    add x11, sp, #248
+; CHECK-NEXT:    ld1 { v4.b }[4], [x10]
+; CHECK-NEXT:    add x10, sp, #384
+; CHECK-NEXT:    ld1 { v7.b }[3], [x9]
+; CHECK-NEXT:    add x9, sp, #440
 ; CHECK-NEXT:    ld1 { v17.b }[3], [x8]
 ; CHECK-NEXT:    add x8, sp, #312
-; CHECK-NEXT:    ld1 { v7.b }[4], [x10]
-; CHECK-NEXT:    add x10, sp, #256
-; CHECK-NEXT:    ld1 { v4.b }[5], [x11]
-; CHECK-NEXT:    add x11, sp, #392
-; CHECK-NEXT:    ld1 { v16.b }[5], [x9]
-; CHECK-NEXT:    add x9, sp, #456
+; CHECK-NEXT:    ld1 { v6.b }[4], [x11]
+; CHECK-NEXT:    add x11, sp, #256
+; CHECK-NEXT:    ld1 { v4.b }[5], [x10]
+; CHECK-NEXT:    add x10, sp, #392
+; CHECK-NEXT:    ld1 { v7.b }[4], [x9]
+; CHECK-NEXT:    add x9, sp, #448
 ; CHECK-NEXT:    ld1 { v17.b }[4], [x8]
 ; CHECK-NEXT:    add x8, sp, #320
-; CHECK-NEXT:    ld1 { v7.b }[5], [x10]
-; CHECK-NEXT:    add x10, sp, #264
-; CHECK-NEXT:    ld1 { v4.b }[6], [x11]
-; CHECK-NEXT:    add x11, sp, #400
-; CHECK-NEXT:    ld1 { v16.b }[6], [x9]
-; CHECK-NEXT:    add x9, sp, #464
+; CHECK-NEXT:    ld1 { v6.b }[5], [x11]
+; CHECK-NEXT:    add x11, sp, #264
+; CHECK-NEXT:    sshll v19.8h, v2.8b, #0
+; CHECK-NEXT:    ld1 { v4.b }[6], [x10]
+; CHECK-NEXT:    ld1 { v7.b }[5], [x9]
+; CHECK-NEXT:    add x9, sp, #456
 ; CHECK-NEXT:    ld1 { v17.b }[5], [x8]
 ; CHECK-NEXT:    add x8, sp, #328
-; CHECK-NEXT:    ld1 { v7.b }[6], [x10]
-; CHECK-NEXT:    add x10, sp, #272
-; CHECK-NEXT:    ld1 { v4.b }[7], [x11]
-; CHECK-NEXT:    add x11, sp, #496
-; CHECK-NEXT:    ld1 { v16.b }[7], [x9]
-; CHECK-NEXT:    add x9, sp, #488
+; CHECK-NEXT:    ld1 { v6.b }[6], [x11]
+; CHECK-NEXT:    add x11, sp, #272
+; CHECK-NEXT:    sshll v2.8h, v1.8b, #0
+; CHECK-NEXT:    ldr b1, [sp, #608]
+; CHECK-NEXT:    ld1 { v7.b }[6], [x9]
+; CHECK-NEXT:    add x9, sp, #464
 ; CHECK-NEXT:    ld1 { v17.b }[6], [x8]
 ; CHECK-NEXT:    add x8, sp, #336
-; CHECK-NEXT:    ld1 { v7.b }[7], [x10]
-; CHECK-NEXT:    add x10, sp, #640
-; CHECK-NEXT:    sshll v6.8h, v4.8b, #0
-; CHECK-NEXT:    sshll v16.8h, v16.8b, #0
+; CHECK-NEXT:    ld1 { v6.b }[7], [x11]
+; CHECK-NEXT:    add x10, sp, #400
+; CHECK-NEXT:    sshll v16.8h, v3.8b, #0
+; CHECK-NEXT:    add x11, sp, #648
+; CHECK-NEXT:    ld1 { v7.b }[7], [x9]
+; CHECK-NEXT:    add x9, sp, #624
 ; CHECK-NEXT:    ld1 { v17.b }[7], [x8]
 ; CHECK-NEXT:    add x8, sp, #616
-; CHECK-NEXT:    sshll v18.8h, v7.8b, #0
-; CHECK-NEXT:    ldr b7, [sp, #608]
-; CHECK-NEXT:    mov v1.b[5], w5
-; CHECK-NEXT:    ld1 { v7.b }[1], [x8]
-; CHECK-NEXT:    add x8, sp, #624
-; CHECK-NEXT:    sshll v4.8h, v17.8b, #0
-; CHECK-NEXT:    sshll v17.8h, v2.8b, #0
-; CHECK-NEXT:    smull2 v2.4s, v3.8h, v4.8h
-; CHECK-NEXT:    smull v3.4s, v3.4h, v4.4h
-; CHECK-NEXT:    ld1 { v7.b }[2], [x8]
-; CHECK-NEXT:    add x8, sp, #632
-; CHECK-NEXT:    smull v4.4s, v5.4h, v17.4h
-; CHECK-NEXT:    ldr b17, [sp, #544]
-; CHECK-NEXT:    smlal2 v2.4s, v19.8h, v16.8h
-; CHECK-NEXT:    smlal v3.4s, v19.4h, v16.4h
-; CHECK-NEXT:    ldr b16, [sp, #480]
-; CHECK-NEXT:    ld1 { v7.b }[3], [x8]
-; CHECK-NEXT:    add x8, sp, #552
-; CHECK-NEXT:    ldr b19, [sp, #672]
-; CHECK-NEXT:    ld1 { v16.b }[1], [x9]
-; CHECK-NEXT:    add x9, sp, #680
+; CHECK-NEXT:    sshll v21.8h, v6.8b, #0
+; CHECK-NEXT:    ldr b6, [sp, #472]
+; CHECK-NEXT:    ld1 { v4.b }[7], [x10]
+; CHECK-NEXT:    add x10, sp, #552
+; CHECK-NEXT:    ld1 { v1.b }[1], [x8]
+; CHECK-NEXT:    add x8, sp, #488
+; CHECK-NEXT:    sshll v18.8h, v17.8b, #0
+; CHECK-NEXT:    ldr b17, [sp, #480]
+; CHECK-NEXT:    sshll v6.8h, v6.8b, #0
+; CHECK-NEXT:    sshll v3.8h, v4.8b, #0
 ; CHECK-NEXT:    ld1 { v17.b }[1], [x8]
-; CHECK-NEXT:    add x8, sp, #560
-; CHECK-NEXT:    ld1 { v7.b }[4], [x10]
-; CHECK-NEXT:    add x10, sp, #648
-; CHECK-NEXT:    ld1 { v19.b }[1], [x9]
-; CHECK-NEXT:    add x9, sp, #688
-; CHECK-NEXT:    ld1 { v16.b }[2], [x11]
-; CHECK-NEXT:    add x11, sp, #504
+; CHECK-NEXT:    add x8, sp, #496
+; CHECK-NEXT:    ld1 { v1.b }[2], [x9]
+; CHECK-NEXT:    add x9, sp, #632
+; CHECK-NEXT:    sshll v4.8h, v7.8b, #0
+; CHECK-NEXT:    smull v20.4s, v5.4h, v6.4h
+; CHECK-NEXT:    movi v7.2d, #0000000000000000
 ; CHECK-NEXT:    ld1 { v17.b }[2], [x8]
-; CHECK-NEXT:    add x8, sp, #568
-; CHECK-NEXT:    ld1 { v7.b }[5], [x10]
-; CHECK-NEXT:    add x10, sp, #656
-; CHECK-NEXT:    ld1 { v19.b }[2], [x9]
-; CHECK-NEXT:    add x9, sp, #696
-; CHECK-NEXT:    ld1 { v16.b }[3], [x11]
-; CHECK-NEXT:    add x11, sp, #512
+; CHECK-NEXT:    smull v5.4s, v16.4h, v18.4h
+; CHECK-NEXT:    ld1 { v1.b }[3], [x9]
+; CHECK-NEXT:    smull2 v16.4s, v16.8h, v18.8h
+; CHECK-NEXT:    ldr b18, [sp, #544]
+; CHECK-NEXT:    add x8, sp, #504
+; CHECK-NEXT:    add x9, sp, #640
+; CHECK-NEXT:    mov v7.s[0], v20.s[0]
+; CHECK-NEXT:    ldr b20, [sp, #672]
+; CHECK-NEXT:    ld1 { v18.b }[1], [x10]
+; CHECK-NEXT:    add x10, sp, #680
 ; CHECK-NEXT:    ld1 { v17.b }[3], [x8]
-; CHECK-NEXT:    add x8, sp, #576
-; CHECK-NEXT:    mov v4.s[1], wzr
-; CHECK-NEXT:    ld1 { v7.b }[6], [x10]
-; CHECK-NEXT:    ld1 { v19.b }[3], [x9]
-; CHECK-NEXT:    add x9, sp, #704
-; CHECK-NEXT:    ld1 { v16.b }[4], [x11]
-; CHECK-NEXT:    add x11, sp, #520
-; CHECK-NEXT:    ld1 { v17.b }[4], [x8]
-; CHECK-NEXT:    add x8, sp, #584
-; CHECK-NEXT:    mov v1.b[6], w6
-; CHECK-NEXT:    add x10, sp, #664
-; CHECK-NEXT:    ld1 { v19.b }[4], [x9]
-; CHECK-NEXT:    add x9, sp, #712
-; CHECK-NEXT:    mov v4.s[2], wzr
-; CHECK-NEXT:    ld1 { v16.b }[5], [x11]
-; CHECK-NEXT:    ld1 { v17.b }[5], [x8]
-; CHECK-NEXT:    add x8, sp, #880
-; CHECK-NEXT:    mov v1.b[7], w7
-; CHECK-NEXT:    ld1 { v7.b }[7], [x10]
-; CHECK-NEXT:    ld1 { v19.b }[5], [x9]
-; CHECK-NEXT:    add x9, sp, #720
-; CHECK-NEXT:    ld1 { v20.b }[1], [x8]
+; CHECK-NEXT:    add x8, sp, #560
+; CHECK-NEXT:    ld1 { v1.b }[4], [x9]
+; CHECK-NEXT:    add x9, sp, #512
+; CHECK-NEXT:    ld1 { v20.b }[1], [x10]
+; CHECK-NEXT:    add x10, sp, #520
+; CHECK-NEXT:    ld1 { v18.b }[2], [x8]
+; CHECK-NEXT:    add x8, sp, #688
+; CHECK-NEXT:    ld1 { v17.b }[4], [x9]
+; CHECK-NEXT:    add x9, sp, #568
+; CHECK-NEXT:    smull2 v6.4s, v19.8h, v21.8h
+; CHECK-NEXT:    ld1 { v1.b }[5], [x11]
+; CHECK-NEXT:    ld1 { v20.b }[2], [x8]
+; CHECK-NEXT:    add x8, sp, #696
+; CHECK-NEXT:    ld1 { v18.b }[3], [x9]
+; CHECK-NEXT:    add x9, sp, #576
+; CHECK-NEXT:    ld1 { v17.b }[5], [x10]
 ; CHECK-NEXT:    add x10, sp, #528
-; CHECK-NEXT:    add x8, sp, #888
-; CHECK-NEXT:    add x11, sp, #592
-; CHECK-NEXT:    mov v4.s[3], wzr
-; CHECK-NEXT:    ld1 { v19.b }[6], [x9]
-; CHECK-NEXT:    add x9, sp, #728
-; CHECK-NEXT:    ld1 { v16.b }[6], [x10]
+; CHECK-NEXT:    smlal v7.4s, v19.4h, v21.4h
+; CHECK-NEXT:    ldr b19, [sp, #872]
+; CHECK-NEXT:    ld1 { v20.b }[3], [x8]
+; CHECK-NEXT:    add x8, sp, #704
+; CHECK-NEXT:    ld1 { v18.b }[4], [x9]
+; CHECK-NEXT:    add x9, sp, #584
+; CHECK-NEXT:    ld1 { v17.b }[6], [x10]
 ; CHECK-NEXT:    add x10, sp, #536
-; CHECK-NEXT:    ld1 { v20.b }[2], [x8]
-; CHECK-NEXT:    add x8, sp, #752
-; CHECK-NEXT:    sshll v1.8h, v1.8b, #0
-; CHECK-NEXT:    ld1 { v17.b }[6], [x11]
-; CHECK-NEXT:    ld1 { v19.b }[7], [x9]
-; CHECK-NEXT:    add x9, sp, #944
-; CHECK-NEXT:    smull2 v5.4s, v1.8h, v18.8h
-; CHECK-NEXT:    ld1 { v21.b }[1], [x8]
-; CHECK-NEXT:    smlal v4.4s, v1.4h, v18.4h
-; CHECK-NEXT:    ldr b1, [sp, #936]
-; CHECK-NEXT:    add x11, sp, #600
-; CHECK-NEXT:    ld1 { v16.b }[7], [x10]
-; CHECK-NEXT:    add x10, sp, #896
-; CHECK-NEXT:    add x8, sp, #760
-; CHECK-NEXT:    ld1 { v1.b }[1], [x9]
+; CHECK-NEXT:    ldr b21, [sp, #936]
+; CHECK-NEXT:    add x11, sp, #656
+; CHECK-NEXT:    ld1 { v20.b }[4], [x8]
+; CHECK-NEXT:    add x8, sp, #712
+; CHECK-NEXT:    ld1 { v18.b }[5], [x9]
+; CHECK-NEXT:    add x9, sp, #592
+; CHECK-NEXT:    ld1 { v17.b }[7], [x10]
+; CHECK-NEXT:    add x10, sp, #880
+; CHECK-NEXT:    sshll v0.8h, v0.8b, #0
+; CHECK-NEXT:    ld1 { v1.b }[6], [x11]
+; CHECK-NEXT:    ld1 { v20.b }[5], [x8]
+; CHECK-NEXT:    add x8, sp, #600
+; CHECK-NEXT:    ld1 { v18.b }[6], [x9]
+; CHECK-NEXT:    add x9, sp, #720
+; CHECK-NEXT:    ld1 { v19.b }[1], [x10]
+; CHECK-NEXT:    add x10, sp, #944
+; CHECK-NEXT:    smlal2 v6.4s, v0.8h, v3.8h
+; CHECK-NEXT:    add x11, sp, #664
+; CHECK-NEXT:    ld1 { v20.b }[6], [x9]
+; CHECK-NEXT:    add x9, sp, #888
+; CHECK-NEXT:    ld1 { v18.b }[7], [x8]
+; CHECK-NEXT:    add x8, sp, #728
+; CHECK-NEXT:    ld1 { v21.b }[1], [x10]
+; CHECK-NEXT:    add x10, sp, #752
+; CHECK-NEXT:    ld1 { v19.b }[2], [x9]
 ; CHECK-NEXT:    add x9, sp, #952
-; CHECK-NEXT:    ld1 { v17.b }[7], [x11]
+; CHECK-NEXT:    ld1 { v20.b }[7], [x8]
+; CHECK-NEXT:    add x8, sp, #896
+; CHECK-NEXT:    smlal v7.4s, v0.4h, v3.4h
+; CHECK-NEXT:    ldr b0, [sp, #744]
+; CHECK-NEXT:    ld1 { v21.b }[2], [x9]
+; CHECK-NEXT:    add x9, sp, #904
+; CHECK-NEXT:    ld1 { v19.b }[3], [x8]
+; CHECK-NEXT:    add x8, sp, #960
+; CHECK-NEXT:    ld1 { v0.b }[1], [x10]
+; CHECK-NEXT:    add x10, sp, #760
+; CHECK-NEXT:    ld1 { v1.b }[7], [x11]
 ; CHECK-NEXT:    add x11, sp, #816
-; CHECK-NEXT:    smlal2 v5.4s, v0.8h, v6.8h
-; CHECK-NEXT:    ld1 { v20.b }[3], [x10]
-; CHECK-NEXT:    smlal v4.4s, v0.4h, v6.4h
-; CHECK-NEXT:    ld1 { v21.b }[2], [x8]
-; CHECK-NEXT:    ldr b0, [sp, #808]
-; CHECK-NEXT:    add x10, sp, #904
-; CHECK-NEXT:    add x8, sp, #768
-; CHECK-NEXT:    ld1 { v1.b }[2], [x9]
-; CHECK-NEXT:    add x9, sp, #960
-; CHECK-NEXT:    ldr b18, [sp, #736]
-; CHECK-NEXT:    ld1 { v0.b }[1], [x11]
-; CHECK-NEXT:    add x11, sp, #824
-; CHECK-NEXT:    ld1 { v20.b }[4], [x10]
-; CHECK-NEXT:    add x10, sp, #912
 ; CHECK-NEXT:    ld1 { v21.b }[3], [x8]
-; CHECK-NEXT:    add x8, sp, #776
-; CHECK-NEXT:    ld1 { v1.b }[3], [x9]
-; CHECK-NEXT:    add x9, sp, #968
-; CHECK-NEXT:    ld1 { v0.b }[2], [x11]
-; CHECK-NEXT:    add x11, sp, #832
-; CHECK-NEXT:    ld1 { v20.b }[5], [x10]
-; CHECK-NEXT:    add x10, sp, #920
+; CHECK-NEXT:    add x8, sp, #968
+; CHECK-NEXT:    ldr b3, [sp, #808]
+; CHECK-NEXT:    ld1 { v19.b }[4], [x9]
+; CHECK-NEXT:    add x9, sp, #912
+; CHECK-NEXT:    ld1 { v0.b }[2], [x10]
+; CHECK-NEXT:    add x10, sp, #768
+; CHECK-NEXT:    ld1 { v3.b }[1], [x11]
+; CHECK-NEXT:    add x11, sp, #824
 ; CHECK-NEXT:    ld1 { v21.b }[4], [x8]
-; CHECK-NEXT:    add x8, sp, #784
-; CHECK-NEXT:    sshll v6.8h, v7.8b, #0
-; CHECK-NEXT:    ld1 { v1.b }[4], [x9]
-; CHECK-NEXT:    sshll v7.8h, v16.8b, #0
-; CHECK-NEXT:    ld1 { v0.b }[3], [x11]
-; CHECK-NEXT:    sshll v16.8h, v18.8b, #0
-; CHECK-NEXT:    ldr b18, [sp, #1000]
-; CHECK-NEXT:    ld1 { v20.b }[6], [x10]
-; CHECK-NEXT:    add x9, sp, #976
+; CHECK-NEXT:    add x8, sp, #976
+; CHECK-NEXT:    ld1 { v19.b }[5], [x9]
+; CHECK-NEXT:    add x9, sp, #920
+; CHECK-NEXT:    ld1 { v0.b }[3], [x10]
+; CHECK-NEXT:    add x10, sp, #776
+; CHECK-NEXT:    ld1 { v3.b }[2], [x11]
+; CHECK-NEXT:    add x11, sp, #832
 ; CHECK-NEXT:    ld1 { v21.b }[5], [x8]
-; CHECK-NEXT:    add x8, sp, #840
-; CHECK-NEXT:    sshll v18.8h, v18.8b, #0
-; CHECK-NEXT:    add x10, sp, #928
-; CHECK-NEXT:    ld1 { v1.b }[5], [x9]
-; CHECK-NEXT:    add x9, sp, #792
-; CHECK-NEXT:    smull v16.4s, v16.4h, v18.4h
-; CHECK-NEXT:    ld1 { v0.b }[4], [x8]
-; CHECK-NEXT:    ld1 { v20.b }[7], [x10]
-; CHECK-NEXT:    add x10, sp, #984
-; CHECK-NEXT:    add x8, sp, #848
-; CHECK-NEXT:    ld1 { v21.b }[6], [x9]
-; CHECK-NEXT:    add x9, sp, #800
-; CHECK-NEXT:    mov v16.s[1], wzr
-; CHECK-NEXT:    ld1 { v1.b }[6], [x10]
-; CHECK-NEXT:    ld1 { v0.b }[5], [x8]
-; CHECK-NEXT:    add x10, sp, #992
-; CHECK-NEXT:    add x8, sp, #856
-; CHECK-NEXT:    ld1 { v21.b }[7], [x9]
-; CHECK-NEXT:    sshll v18.8h, v19.8b, #0
-; CHECK-NEXT:    mov v16.s[2], wzr
-; CHECK-NEXT:    ld1 { v1.b }[7], [x10]
+; CHECK-NEXT:    add x8, sp, #984
+; CHECK-NEXT:    ld1 { v19.b }[6], [x9]
+; CHECK-NEXT:    add x9, sp, #928
+; CHECK-NEXT:    ld1 { v0.b }[4], [x10]
+; CHECK-NEXT:    add x10, sp, #848
+; CHECK-NEXT:    ld1 { v3.b }[3], [x11]
+; CHECK-NEXT:    add x11, sp, #840
+; CHECK-NEXT:    ld1 { v21.b }[6], [x8]
+; CHECK-NEXT:    add x8, sp, #992
+; CHECK-NEXT:    ld1 { v19.b }[7], [x9]
+; CHECK-NEXT:    add x9, sp, #784
+; CHECK-NEXT:    smlal2 v16.4s, v2.8h, v4.8h
+; CHECK-NEXT:    ld1 { v3.b }[4], [x11]
+; CHECK-NEXT:    ld1 { v21.b }[7], [x8]
+; CHECK-NEXT:    add x8, sp, #792
+; CHECK-NEXT:    ld1 { v0.b }[5], [x9]
+; CHECK-NEXT:    add x9, sp, #856
+; CHECK-NEXT:    smlal v5.4s, v2.4h, v4.4h
+; CHECK-NEXT:    ldr b2, [sp, #736]
+; CHECK-NEXT:    sshll v4.8h, v20.8b, #0
+; CHECK-NEXT:    ldr b20, [sp, #1000]
+; CHECK-NEXT:    ld1 { v3.b }[5], [x10]
+; CHECK-NEXT:    sshll v2.8h, v2.8b, #0
 ; CHECK-NEXT:    ld1 { v0.b }[6], [x8]
-; CHECK-NEXT:    add x8, sp, #864
-; CHECK-NEXT:    sshll v19.8h, v20.8b, #0
+; CHECK-NEXT:    sshll v20.8h, v20.8b, #0
+; CHECK-NEXT:    add x8, sp, #800
 ; CHECK-NEXT:    sshll v21.8h, v21.8b, #0
-; CHECK-NEXT:    mov v16.s[3], wzr
-; CHECK-NEXT:    sshll v1.8h, v1.8b, #0
+; CHECK-NEXT:    smull v2.4s, v2.4h, v20.4h
+; CHECK-NEXT:    ld1 { v3.b }[6], [x9]
+; CHECK-NEXT:    smull v20.4s, v4.4h, v21.4h
 ; CHECK-NEXT:    ld1 { v0.b }[7], [x8]
+; CHECK-NEXT:    smull2 v4.4s, v4.8h, v21.8h
+; CHECK-NEXT:    add x9, sp, #864
+; CHECK-NEXT:    movi v21.2d, #0000000000000000
+; CHECK-NEXT:    sshll v1.8h, v1.8b, #0
 ; CHECK-NEXT:    sshll v17.8h, v17.8b, #0
-; CHECK-NEXT:    smull v20.4s, v18.4h, v1.4h
-; CHECK-NEXT:    smull2 v1.4s, v18.8h, v1.8h
-; CHECK-NEXT:    smull2 v18.4s, v6.8h, v19.8h
-; CHECK-NEXT:    smlal v16.4s, v7.4h, v21.4h
+; CHECK-NEXT:    ld1 { v3.b }[7], [x9]
+; CHECK-NEXT:    sshll v19.8h, v19.8b, #0
+; CHECK-NEXT:    mov v21.s[0], v2.s[0]
 ; CHECK-NEXT:    sshll v0.8h, v0.8b, #0
-; CHECK-NEXT:    smlal2 v18.4s, v7.8h, v21.8h
-; CHECK-NEXT:    smlal2 v1.4s, v17.8h, v0.8h
-; CHECK-NEXT:    smlal v20.4s, v17.4h, v0.4h
-; CHECK-NEXT:    smlal v16.4s, v6.4h, v19.4h
-; CHECK-NEXT:    add v0.4s, v5.4s, v2.4s
-; CHECK-NEXT:    add v2.4s, v4.4s, v3.4s
-; CHECK-NEXT:    add v1.4s, v18.4s, v1.4s
-; CHECK-NEXT:    add v3.4s, v16.4s, v20.4s
-; CHECK-NEXT:    add v0.4s, v2.4s, v0.4s
-; CHECK-NEXT:    add v1.4s, v3.4s, v1.4s
+; CHECK-NEXT:    smull2 v2.4s, v1.8h, v19.8h
+; CHECK-NEXT:    sshll v18.8h, v18.8b, #0
+; CHECK-NEXT:    smlal v21.4s, v17.4h, v0.4h
+; CHECK-NEXT:    sshll v3.8h, v3.8b, #0
+; CHECK-NEXT:    smlal2 v2.4s, v17.8h, v0.8h
+; CHECK-NEXT:    smlal2 v4.4s, v18.8h, v3.8h
+; CHECK-NEXT:    smlal v20.4s, v18.4h, v3.4h
+; CHECK-NEXT:    smlal v21.4s, v1.4h, v19.4h
+; CHECK-NEXT:    add v0.4s, v6.4s, v16.4s
+; CHECK-NEXT:    add v1.4s, v7.4s, v5.4s
+; CHECK-NEXT:    add v2.4s, v2.4s, v4.4s
+; CHECK-NEXT:    add v3.4s, v21.4s, v20.4s
+; CHECK-NEXT:    add v0.4s, v1.4s, v0.4s
+; CHECK-NEXT:    add v1.4s, v3.4s, v2.4s
 ; CHECK-NEXT:    add v0.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    addv s0, v0.4s
 ; CHECK-NEXT:    fmov w0, s0
@@ -2042,154 +2023,152 @@ define i32 @test_sdot_v33i8_double_nomla(<33 x i8> %a, <33 x i8> %b, <33 x i8> %
 ; CHECK-NEXT:    .cfi_offset w29, -16
 ; CHECK-NEXT:    ldr b0, [sp, #80]
 ; CHECK-NEXT:    add x8, sp, #88
-; CHECK-NEXT:    ldr b3, [sp, #144]
-; CHECK-NEXT:    add x10, sp, #96
-; CHECK-NEXT:    ldr b5, [sp, #16]
-; CHECK-NEXT:    add x9, sp, #24
+; CHECK-NEXT:    ldr b2, [sp, #144]
+; CHECK-NEXT:    add x9, sp, #152
+; CHECK-NEXT:    fmov s3, w0
+; CHECK-NEXT:    ldr b4, [sp, #16]
 ; CHECK-NEXT:    ld1 { v0.b }[1], [x8]
-; CHECK-NEXT:    add x8, sp, #152
-; CHECK-NEXT:    fmov s2, w0
-; CHECK-NEXT:    ldr b1, [sp, #208]
-; CHECK-NEXT:    ld1 { v5.b }[1], [x9]
-; CHECK-NEXT:    add x9, sp, #32
-; CHECK-NEXT:    ld1 { v3.b }[1], [x8]
-; CHECK-NEXT:    add x8, sp, #160
-; CHECK-NEXT:    ld1 { v0.b }[2], [x10]
+; CHECK-NEXT:    add x8, sp, #96
 ; CHECK-NEXT:    add x10, sp, #104
-; CHECK-NEXT:    mov v2.b[1], w1
-; CHECK-NEXT:    ldr b17, [sp, #672]
-; CHECK-NEXT:    ld1 { v5.b }[2], [x9]
-; CHECK-NEXT:    add x9, sp, #40
-; CHECK-NEXT:    ld1 { v3.b }[2], [x8]
-; CHECK-NEXT:    add x8, sp, #168
+; CHECK-NEXT:    ld1 { v2.b }[1], [x9]
+; CHECK-NEXT:    mov v3.b[1], w1
+; CHECK-NEXT:    add x9, sp, #160
+; CHECK-NEXT:    add x11, sp, #128
+; CHECK-NEXT:    ldr b1, [sp, #208]
+; CHECK-NEXT:    ld1 { v0.b }[2], [x8]
+; CHECK-NEXT:    add x8, sp, #24
+; CHECK-NEXT:    ld1 { v2.b }[2], [x9]
+; CHECK-NEXT:    add x9, sp, #168
+; CHECK-NEXT:    mov v3.b[2], w2
+; CHECK-NEXT:    ld1 { v4.b }[1], [x8]
+; CHECK-NEXT:    add x8, sp, #112
 ; CHECK-NEXT:    ld1 { v0.b }[3], [x10]
-; CHECK-NEXT:    add x10, sp, #112
-; CHECK-NEXT:    mov v2.b[2], w2
-; CHECK-NEXT:    add x11, sp, #648
-; CHECK-NEXT:    ld1 { v5.b }[3], [x9]
-; CHECK-NEXT:    add x9, sp, #48
-; CHECK-NEXT:    ld1 { v3.b }[3], [x8]
-; CHECK-NEXT:    add x8, sp, #176
-; CHECK-NEXT:    ld1 { v0.b }[4], [x10]
+; CHECK-NEXT:    add x10, sp, #32
+; CHECK-NEXT:    ld1 { v2.b }[3], [x9]
+; CHECK-NEXT:    add x9, sp, #176
+; CHECK-NEXT:    mov v3.b[3], w3
+; CHECK-NEXT:    ld1 { v4.b }[2], [x10]
 ; CHECK-NEXT:    add x10, sp, #120
-; CHECK-NEXT:    mov v2.b[3], w3
-; CHECK-NEXT:    ld1 { v5.b }[4], [x9]
-; CHECK-NEXT:    add x9, sp, #56
-; CHECK-NEXT:    ld1 { v3.b }[4], [x8]
-; CHECK-NEXT:    add x8, sp, #184
+; CHECK-NEXT:    ld1 { v0.b }[4], [x8]
+; CHECK-NEXT:    add x8, sp, #40
+; CHECK-NEXT:    ld1 { v2.b }[4], [x9]
+; CHECK-NEXT:    add x9, sp, #184
+; CHECK-NEXT:    mov v3.b[4], w4
+; CHECK-NEXT:    ld1 { v4.b }[3], [x8]
+; CHECK-NEXT:    add x8, sp, #48
 ; CHECK-NEXT:    ld1 { v0.b }[5], [x10]
-; CHECK-NEXT:    add x10, sp, #128
-; CHECK-NEXT:    mov v2.b[4], w4
-; CHECK-NEXT:    ld1 { v5.b }[5], [x9]
-; CHECK-NEXT:    add x9, sp, #64
-; CHECK-NEXT:    ld1 { v3.b }[5], [x8]
-; CHECK-NEXT:    add x8, sp, #192
-; CHECK-NEXT:    ld1 { v0.b }[6], [x10]
 ; CHECK-NEXT:    add x10, sp, #136
-; CHECK-NEXT:    sshll v4.8h, v1.8b, #0
-; CHECK-NEXT:    mov v2.b[5], w5
-; CHECK-NEXT:    ld1 { v5.b }[6], [x9]
-; CHECK-NEXT:    ld1 { v3.b }[6], [x8]
-; CHECK-NEXT:    add x8, sp, #200
+; CHECK-NEXT:    ld1 { v2.b }[5], [x9]
+; CHECK-NEXT:    add x9, sp, #192
+; CHECK-NEXT:    mov v3.b[5], w5
+; CHECK-NEXT:    ld1 { v4.b }[4], [x8]
+; CHECK-NEXT:    add x8, sp, #56
+; CHECK-NEXT:    ld1 { v0.b }[6], [x11]
+; CHECK-NEXT:    add x11, sp, #632
+; CHECK-NEXT:    ld1 { v2.b }[6], [x9]
+; CHECK-NEXT:    add x9, sp, #200
+; CHECK-NEXT:    mov v3.b[6], w6
+; CHECK-NEXT:    ld1 { v4.b }[5], [x8]
+; CHECK-NEXT:    add x8, sp, #64
+; CHECK-NEXT:    sshll v1.8h, v1.8b, #0
 ; CHECK-NEXT:    ld1 { v0.b }[7], [x10]
-; CHECK-NEXT:    add x9, sp, #72
+; CHECK-NEXT:    ld1 { v2.b }[7], [x9]
+; CHECK-NEXT:    add x9, sp, #552
+; CHECK-NEXT:    mov v3.b[7], w7
 ; CHECK-NEXT:    add x10, sp, #680
-; CHECK-NEXT:    mov v2.b[6], w6
-; CHECK-NEXT:    ld1 { v3.b }[7], [x8]
+; CHECK-NEXT:    ld1 { v4.b }[6], [x8]
+; CHECK-NEXT:    add x8, sp, #72
+; CHECK-NEXT:    movi v6.2d, #0000000000000000
+; CHECK-NEXT:    sshll v5.4s, v1.4h, #0
+; CHECK-NEXT:    ldr b1, [sp, #608]
+; CHECK-NEXT:    sshll v0.8h, v0.8b, #0
+; CHECK-NEXT:    ld1 { v4.b }[7], [x8]
 ; CHECK-NEXT:    add x8, sp, #616
-; CHECK-NEXT:    sshll v1.8h, v0.8b, #0
-; CHECK-NEXT:    ldr b0, [sp, #608]
-; CHECK-NEXT:    ld1 { v5.b }[7], [x9]
-; CHECK-NEXT:    add x9, sp, #488
-; CHECK-NEXT:    mov v2.b[7], w7
-; CHECK-NEXT:    ld1 { v17.b }[1], [x10]
-; CHECK-NEXT:    ld1 { v0.b }[1], [x8]
-; CHECK-NEXT:    add x8, sp, #624
+; CHECK-NEXT:    sshll v2.8h, v2.8b, #0
 ; CHECK-NEXT:    sshll v7.8h, v3.8b, #0
-; CHECK-NEXT:    add x10, sp, #504
-; CHECK-NEXT:    sshll v16.8h, v5.8b, #0
-; CHECK-NEXT:    ldr b5, [sp, #480]
-; CHECK-NEXT:    sshll v6.8h, v2.8b, #0
-; CHECK-NEXT:    ld1 { v0.b }[2], [x8]
-; CHECK-NEXT:    add x8, sp, #632
-; CHECK-NEXT:    saddl2 v2.4s, v16.8h, v7.8h
-; CHECK-NEXT:    ld1 { v5.b }[1], [x9]
-; CHECK-NEXT:    saddl v7.4s, v16.4h, v7.4h
-; CHECK-NEXT:    ldr b16, [sp, #544]
-; CHECK-NEXT:    add x9, sp, #552
-; CHECK-NEXT:    ld1 { v0.b }[3], [x8]
-; CHECK-NEXT:    add x8, sp, #640
-; CHECK-NEXT:    sshll v4.4s, v4.4h, #0
-; CHECK-NEXT:    ld1 { v16.b }[1], [x9]
-; CHECK-NEXT:    add x9, sp, #560
-; CHECK-NEXT:    mov v4.s[1], wzr
-; CHECK-NEXT:    ld1 { v0.b }[4], [x8]
+; CHECK-NEXT:    ld1 { v1.b }[1], [x8]
+; CHECK-NEXT:    add x8, sp, #624
+; CHECK-NEXT:    sshll v3.8h, v4.8b, #0
+; CHECK-NEXT:    mov v6.s[0], v5.s[0]
+; CHECK-NEXT:    saddl2 v5.4s, v3.8h, v2.8h
+; CHECK-NEXT:    saddl2 v16.4s, v7.8h, v0.8h
+; CHECK-NEXT:    ld1 { v1.b }[2], [x8]
+; CHECK-NEXT:    add x8, sp, #488
+; CHECK-NEXT:    saddw v4.4s, v6.4s, v7.4h
+; CHECK-NEXT:    ldr b6, [sp, #480]
+; CHECK-NEXT:    add v5.4s, v16.4s, v5.4s
+; CHECK-NEXT:    ldr b7, [sp, #544]
+; CHECK-NEXT:    ldr b16, [sp, #672]
+; CHECK-NEXT:    ld1 { v6.b }[1], [x8]
 ; CHECK-NEXT:    add x8, sp, #496
-; CHECK-NEXT:    saddl2 v3.4s, v6.8h, v1.8h
-; CHECK-NEXT:    ld1 { v16.b }[2], [x9]
+; CHECK-NEXT:    ld1 { v7.b }[1], [x9]
+; CHECK-NEXT:    add x9, sp, #560
+; CHECK-NEXT:    ld1 { v16.b }[1], [x10]
+; CHECK-NEXT:    add x10, sp, #688
+; CHECK-NEXT:    ld1 { v1.b }[3], [x11]
+; CHECK-NEXT:    add x11, sp, #640
+; CHECK-NEXT:    ld1 { v6.b }[2], [x8]
+; CHECK-NEXT:    add x8, sp, #504
+; CHECK-NEXT:    ld1 { v7.b }[2], [x9]
 ; CHECK-NEXT:    add x9, sp, #568
-; CHECK-NEXT:    ld1 { v5.b }[2], [x8]
-; CHECK-NEXT:    add x8, sp, #688
-; CHECK-NEXT:    mov v4.s[2], wzr
-; CHECK-NEXT:    ld1 { v0.b }[5], [x11]
-; CHECK-NEXT:    add x11, sp, #656
-; CHECK-NEXT:    ld1 { v17.b }[2], [x8]
-; CHECK-NEXT:    add x8, sp, #696
-; CHECK-NEXT:    ld1 { v5.b }[3], [x10]
-; CHECK-NEXT:    add x10, sp, #512
-; CHECK-NEXT:    mov v4.s[3], wzr
-; CHECK-NEXT:    ld1 { v16.b }[3], [x9]
+; CHECK-NEXT:    ld1 { v16.b }[2], [x10]
+; CHECK-NEXT:    add x10, sp, #696
+; CHECK-NEXT:    ld1 { v1.b }[4], [x11]
+; CHECK-NEXT:    add x11, sp, #648
+; CHECK-NEXT:    ld1 { v6.b }[3], [x8]
+; CHECK-NEXT:    add x8, sp, #512
+; CHECK-NEXT:    ld1 { v7.b }[3], [x9]
 ; CHECK-NEXT:    add x9, sp, #576
-; CHECK-NEXT:    ld1 { v0.b }[6], [x11]
-; CHECK-NEXT:    ld1 { v17.b }[3], [x8]
-; CHECK-NEXT:    add x8, sp, #704
-; CHECK-NEXT:    ld1 { v5.b }[4], [x10]
-; CHECK-NEXT:    add x11, sp, #520
-; CHECK-NEXT:    saddw v4.4s, v4.4s, v6.4h
-; CHECK-NEXT:    ldr b6, [sp, #736]
-; CHECK-NEXT:    ld1 { v16.b }[4], [x9]
+; CHECK-NEXT:    ld1 { v16.b }[3], [x10]
+; CHECK-NEXT:    add x10, sp, #704
+; CHECK-NEXT:    ld1 { v1.b }[5], [x11]
+; CHECK-NEXT:    add x11, sp, #656
+; CHECK-NEXT:    ld1 { v6.b }[4], [x8]
+; CHECK-NEXT:    add x8, sp, #520
+; CHECK-NEXT:    ld1 { v7.b }[4], [x9]
 ; CHECK-NEXT:    add x9, sp, #584
-; CHECK-NEXT:    ld1 { v17.b }[4], [x8]
-; CHECK-NEXT:    add x8, sp, #712
-; CHECK-NEXT:    add x10, sp, #664
-; CHECK-NEXT:    ld1 { v5.b }[5], [x11]
-; CHECK-NEXT:    saddw v1.4s, v4.4s, v1.4h
-; CHECK-NEXT:    sshll v4.8h, v6.8b, #0
-; CHECK-NEXT:    ld1 { v16.b }[5], [x9]
-; CHECK-NEXT:    ld1 { v17.b }[5], [x8]
-; CHECK-NEXT:    add x9, sp, #592
-; CHECK-NEXT:    ld1 { v0.b }[7], [x10]
+; CHECK-NEXT:    ld1 { v16.b }[4], [x10]
+; CHECK-NEXT:    add x10, sp, #712
+; CHECK-NEXT:    ld1 { v1.b }[6], [x11]
+; CHECK-NEXT:    add x11, sp, #664
+; CHECK-NEXT:    ld1 { v6.b }[5], [x8]
 ; CHECK-NEXT:    add x8, sp, #528
+; CHECK-NEXT:    ld1 { v7.b }[5], [x9]
+; CHECK-NEXT:    add x9, sp, #592
+; CHECK-NEXT:    ld1 { v16.b }[5], [x10]
 ; CHECK-NEXT:    add x10, sp, #720
-; CHECK-NEXT:    sshll v4.4s, v4.4h, #0
-; CHECK-NEXT:    ld1 { v16.b }[6], [x9]
-; CHECK-NEXT:    ld1 { v5.b }[6], [x8]
-; CHECK-NEXT:    add x9, sp, #600
-; CHECK-NEXT:    mov v4.s[1], wzr
-; CHECK-NEXT:    ld1 { v17.b }[6], [x10]
-; CHECK-NEXT:    add x10, sp, #728
-; CHECK-NEXT:    add x8, sp, #536
-; CHECK-NEXT:    ld1 { v16.b }[7], [x9]
-; CHECK-NEXT:    add v2.4s, v3.4s, v2.4s
-; CHECK-NEXT:    mov v4.s[2], wzr
-; CHECK-NEXT:    ld1 { v17.b }[7], [x10]
-; CHECK-NEXT:    ld1 { v5.b }[7], [x8]
-; CHECK-NEXT:    sshll v0.8h, v0.8b, #0
-; CHECK-NEXT:    sshll v3.8h, v16.8b, #0
-; CHECK-NEXT:    mov v4.s[3], wzr
-; CHECK-NEXT:    sshll v6.8h, v17.8b, #0
-; CHECK-NEXT:    sshll v5.8h, v5.8b, #0
-; CHECK-NEXT:    add v1.4s, v1.4s, v7.4s
-; CHECK-NEXT:    saddl2 v7.4s, v6.8h, v3.8h
-; CHECK-NEXT:    saddl2 v16.4s, v0.8h, v5.8h
-; CHECK-NEXT:    saddw v4.4s, v4.4s, v5.4h
-; CHECK-NEXT:    saddl v3.4s, v6.4h, v3.4h
-; CHECK-NEXT:    add v5.4s, v16.4s, v7.4s
+; CHECK-NEXT:    saddl v2.4s, v3.4h, v2.4h
+; CHECK-NEXT:    ldr b3, [sp, #736]
+; CHECK-NEXT:    ld1 { v6.b }[6], [x8]
+; CHECK-NEXT:    add x8, sp, #600
 ; CHECK-NEXT:    saddw v0.4s, v4.4s, v0.4h
-; CHECK-NEXT:    add v3.4s, v3.4s, v5.4s
+; CHECK-NEXT:    ld1 { v7.b }[6], [x9]
+; CHECK-NEXT:    ld1 { v16.b }[6], [x10]
+; CHECK-NEXT:    add x9, sp, #728
+; CHECK-NEXT:    add x10, sp, #536
+; CHECK-NEXT:    ld1 { v1.b }[7], [x11]
+; CHECK-NEXT:    movi v4.2d, #0000000000000000
+; CHECK-NEXT:    add v0.4s, v0.4s, v2.4s
+; CHECK-NEXT:    ld1 { v7.b }[7], [x8]
+; CHECK-NEXT:    sshll v2.8h, v3.8b, #0
+; CHECK-NEXT:    ld1 { v16.b }[7], [x9]
+; CHECK-NEXT:    ld1 { v6.b }[7], [x10]
+; CHECK-NEXT:    sshll v2.4s, v2.4h, #0
+; CHECK-NEXT:    sshll v1.8h, v1.8b, #0
+; CHECK-NEXT:    mov v4.s[0], v2.s[0]
+; CHECK-NEXT:    sshll v3.8h, v7.8b, #0
+; CHECK-NEXT:    sshll v7.8h, v16.8b, #0
+; CHECK-NEXT:    sshll v2.8h, v6.8b, #0
+; CHECK-NEXT:    saddl2 v6.4s, v7.8h, v3.8h
+; CHECK-NEXT:    saddl2 v16.4s, v1.8h, v2.8h
+; CHECK-NEXT:    saddw v2.4s, v4.4s, v2.4h
+; CHECK-NEXT:    saddl v3.4s, v7.4h, v3.4h
+; CHECK-NEXT:    add v4.4s, v16.4s, v6.4s
+; CHECK-NEXT:    saddw v1.4s, v2.4s, v1.4h
+; CHECK-NEXT:    add v2.4s, v3.4s, v4.4s
+; CHECK-NEXT:    add v0.4s, v0.4s, v5.4s
 ; CHECK-NEXT:    add v1.4s, v1.4s, v2.4s
-; CHECK-NEXT:    add v0.4s, v0.4s, v3.4s
-; CHECK-NEXT:    add v0.4s, v1.4s, v0.4s
+; CHECK-NEXT:    add v0.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    addv s0, v0.4s
 ; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ldr x29, [sp], #16 // 8-byte Folded Reload
diff --git a/llvm/test/CodeGen/AArch64/replace-intrinsics-with-veclib-sqrt.ll b/llvm/test/CodeGen/AArch64/replace-intrinsics-with-veclib-sqrt.ll
new file mode 100644
index 0000000000000..03fca2ffe3e32
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/replace-intrinsics-with-veclib-sqrt.ll
@@ -0,0 +1,50 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -vector-library=sleefgnuabi -replace-with-veclib -S < %s | FileCheck %s
+
+target triple = "aarch64-unknown-linux-gnu"
+
+define <2 x double> @llvm_sqrt_f64(<2 x double> %in) {
+; CHECK-LABEL: define {{[^@]+}}@llvm_sqrt_f64
+; CHECK-SAME: (<2 x double> [[IN:%.*]]) {
+; CHECK-NEXT:    [[TMP1:%.*]] = call fast <2 x double> @llvm.sqrt.v2f64(<2 x double> [[IN]])
+; CHECK-NEXT:    ret <2 x double> [[TMP1]]
+;
+  %1 = call fast <2 x double> @llvm.sqrt.v2f64(<2 x double> %in)
+  ret <2 x double> %1
+}
+
+define <4 x float> @llvm_sqrt_f32(<4 x float> %in) {
+; CHECK-LABEL: define {{[^@]+}}@llvm_sqrt_f32
+; CHECK-SAME: (<4 x float> [[IN:%.*]]) {
+; CHECK-NEXT:    [[TMP1:%.*]] = call fast <4 x float> @llvm.sqrt.v4f32(<4 x float> [[IN]])
+; CHECK-NEXT:    ret <4 x float> [[TMP1]]
+;
+  %1 = call fast <4 x float> @llvm.sqrt.v4f32(<4 x float> %in)
+  ret <4 x float> %1
+}
+
+define <vscale x 2 x double> @llvm_sqrt_vscale_f64(<vscale x 2 x double> %in) {
+; CHECK-LABEL: define {{[^@]+}}@llvm_sqrt_vscale_f64
+; CHECK-SAME: (<vscale x 2 x double> [[IN:%.*]]) {
+; CHECK-NEXT:    [[TMP1:%.*]] = call fast <vscale x 2 x double> @llvm.sqrt.nxv2f64(<vscale x 2 x double> [[IN]])
+; CHECK-NEXT:    ret <vscale x 2 x double> [[TMP1]]
+;
+  %1 = call fast <vscale x 2 x double> @llvm.sqrt.nxv2f64(<vscale x 2 x double> %in)
+  ret <vscale x 2 x double> %1
+}
+
+define <vscale x 4 x float> @llvm_sqrt_vscale_f32(<vscale x 4 x float> %in) {
+; CHECK-LABEL: define {{[^@]+}}@llvm_sqrt_vscale_f32
+; CHECK-SAME: (<vscale x 4 x float> [[IN:%.*]]) {
+; CHECK-NEXT:    [[TMP1:%.*]] = call fast <vscale x 4 x float> @llvm.sqrt.nxv4f32(<vscale x 4 x float> [[IN]])
+; CHECK-NEXT:    ret <vscale x 4 x float> [[TMP1]]
+;
+  %1 = call fast <vscale x 4 x float> @llvm.sqrt.nxv4f32(<vscale x 4 x float> %in)
+  ret <vscale x 4 x float> %1
+}
+
+
+declare <2 x double> @llvm.sqrt.v2f64(<2 x double>)
+declare <4 x float> @llvm.sqrt.v4f32(<4 x float>)
+declare <vscale x 2 x double> @llvm.sqrt.nxv2f64(<vscale x 2 x double>)
+declare <vscale x 4 x float> @llvm.sqrt.nxv4f32(<vscale x 4 x float>)
diff --git a/llvm/test/CodeGen/AArch64/safestack-unwind.ll b/llvm/test/CodeGen/AArch64/safestack-unwind.ll
new file mode 100644
index 0000000000000..66ff7409a68c7
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/safestack-unwind.ll
@@ -0,0 +1,140 @@
+; RUN: opt --dwarfehprepare %s -S -o - -mtriple=aarch64-linux-gnu | FileCheck %s
+
+define void @call() sspreq personality ptr @__gxx_personality_v0 {
+; CHECK-LABEL: define void @call()
+; CHECK:     invoke void @bar()
+; CHECK:             to label %[[CONT:.*]] unwind label %[[CLEANUP:.*]]
+; CHECK: [[CONT]]:
+; CHECK:     ret void
+; CHECK: [[CLEANUP]]:
+; CHECK:     [[LP:%.*]] = landingpad { ptr, i32 }
+; CHECK:             cleanup
+; CHECK:     [[EXN:%.*]] = extractvalue { ptr, i32 } [[LP]], 0
+; CHECK:     call void @_Unwind_Resume(ptr [[EXN]])
+; CHECK:     unreachable
+  call void @bar()
+  ret void
+}
+
+define void @invoke_no_cleanup() sspreq personality ptr @__gxx_personality_v0 {
+; CHECK-LABEL: define void @invoke_no_cleanup
+; CHECK:   invoke void @bar()
+; CHECK:           to label %done unwind label %catch
+
+; CHECK: catch:
+; CHECK:   [[LP:%.*]] = landingpad { ptr, i32 }
+; CHECK:           cleanup
+; CHECK:           catch ptr null
+; CHECK:   [[SEL:%.*]] = extractvalue { ptr, i32 } [[LP]], 1
+; CHECK:   [[CMP:%.*]] = icmp eq i32 [[SEL]], 0
+; CHECK:   br i1 [[CMP]], label %[[RESUME:.*]], label %[[SPLIT:.*]]
+
+; CHECK: [[SPLIT]]:
+; CHECK:   br label %done
+
+; CHECK: done:
+; CHECK:   ret void
+
+; CHECK: [[RESUME]]:
+; CHECK:   [[EXN:%.*]] = extractvalue { ptr, i32 } [[LP]], 0
+; CHECK:   call void @_Unwind_Resume(ptr [[EXN]])
+; CHECK:   unreachable
+  invoke void @bar() to label %done unwind label %catch
+
+catch:
+  %lp = landingpad { ptr, i32 }
+          catch ptr null
+  br label %done
+
+done:
+  ret void
+}
+
+define void @invoke_no_cleanup_catches() sspreq personality ptr @__gxx_personality_v0 {
+; CHECK-LABEL: define void @invoke_no_cleanup_catches
+; CHECK:   invoke void @bar()
+; CHECK:           to label %done unwind label %catch
+
+; CHECK: catch:
+; CHECK:   [[LP:%.*]] = landingpad { ptr, i32 }
+; CHECK:           cleanup
+; CHECK:           catch ptr null
+; CHECK:   [[SEL:%.*]] = extractvalue { ptr, i32 } [[LP]], 1
+; CEHCK:   [[CMP:%.*]] = icmp eq i32 [[SEL]], 0
+; CEHCK:   br i1 [[CMP]], label %[[RESUME:.*]], label %[[SPLIT:.*]]
+
+; CHECK: [[SPLIT]]:
+; CHECK:   %exn = extractvalue { ptr, i32 } %lp, 0
+; CHECK:   invoke ptr @__cxa_begin_catch(ptr %exn)
+; CHECK:            to label %[[SPLIT2:.*]] unwind label %[[CLEANUP_RESUME:.*]]
+
+; CHECK: [[SPLIT2]]:
+; CHECK:   invoke void @__cxa_end_catch()
+; CHECK:            to label  %[[SPLIT3:.*]] unwind label %[[CLEANUP_RESUME:.*]]
+
+; CHECK: [[SPLIT3]]:
+; CHECK:   br label %done
+
+; CHECK: done:
+; CHECK:   ret void
+
+; CHECK: [[RESUME]]:
+; CHECK:   [[EXN1:%.*]] = extractvalue { ptr, i32 } [[LP]], 0
+; CHECK:   br label %[[RESUME_MERGE:.*]]
+
+; CHECK: [[CLEANUP_RESUME]]:
+; CHECK:   [[LP:%.*]] = landingpad { ptr, i32 }
+; CHECK:           cleanup
+; CHECK:   [[EXN2:%.*]] = extractvalue { ptr, i32 } [[LP]], 0
+; CHECK:   br label %[[RESUME_MERGE]]
+
+; CHECK: [[RESUME_MERGE]]:
+; CHECK:   [[EXN_PHI:%.*]] = phi ptr [ [[EXN1]], %[[RESUME]] ], [ [[EXN2]], %[[CLEANUP_RESUME]] ]
+; CHECK:   call void @_Unwind_Resume(ptr [[EXN_PHI]])
+; CHECK:   unreachable
+  invoke void @bar() to label %done unwind label %catch
+
+catch:
+  %lp = landingpad { ptr, i32 }
+          catch ptr null
+  %exn = extractvalue { ptr, i32 } %lp, 0
+  call ptr @__cxa_begin_catch(ptr %exn)
+  call void @__cxa_end_catch()
+  br label %done
+
+done:
+  ret void
+}
+
+; Don't try to invoke any intrinsics.
+define ptr @call_intrinsic() sspreq personality ptr @__gxx_personality_v0 {
+; CHECK-LABEL: define ptr @call_intrinsic
+; CHECK: call ptr @llvm.frameaddress.p0(i32 0)
+  %res = call ptr @llvm.frameaddress.p0(i32 0)
+  ret ptr %res
+}
+
+; Check we go along with the existing landingpad type, even if it's a bit
+; outside the normal.
+define void @weird_landingpad() sspreq personality ptr @__gxx_personality_v0 {
+; CHECK-LABEL: define void @weird_landingpad
+; CHECK: landingpad { ptr, i64 }
+; CHECK: landingpad { ptr, i64 }
+  invoke void @bar() to label %done unwind label %catch
+
+catch:
+  %lp = landingpad { ptr, i64 }
+           catch ptr null
+  resume { ptr, i64 } %lp
+;  br label %done
+
+done:
+  call void @bar()
+  ret void
+}
+
+declare void @bar()
+declare i32 @__gxx_personality_v0(...)
+declare ptr @__cxa_begin_catch(ptr)
+declare void @__cxa_end_catch()
+declare ptr @llvm.frameaddress.p0(i32)
diff --git a/llvm/test/CodeGen/AArch64/settag-merge.mir b/llvm/test/CodeGen/AArch64/settag-merge.mir
index 991e0a23e319d..03524e780df02 100644
--- a/llvm/test/CodeGen/AArch64/settag-merge.mir
+++ b/llvm/test/CodeGen/AArch64/settag-merge.mir
@@ -30,8 +30,8 @@
 # A sequence of STG with a register copy in the middle.
 # Can be merged into ST2G + ST2G.
 # CHECK-LABEL: name:{{.*}}stg16_16_16_16_ret
-# CHECK-DAG: ST2GOffset $sp, $sp, 2
-# CHECK-DAG: ST2GOffset $sp, $sp, 0
+# CHECK-DAG: ST2Gi $sp, $sp, 2
+# CHECK-DAG: ST2Gi $sp, $sp, 0
 # CHECK-DAG: $w0 = COPY $wzr
 # CHECK-DAG: RET_ReallyLR implicit killed $w0
 
@@ -44,11 +44,11 @@ stack:
   - { id: 3, name: d, size: 16, alignment: 16 }
 body:             |
   bb.0.entry:
-    STGOffset $sp, %stack.0.a, 0 :: (store (s128) into %ir.a)
-    STGOffset $sp, %stack.1.b, 0 :: (store (s128) into %ir.b)
-    STGOffset $sp, %stack.2.c, 0 :: (store (s128) into %ir.c)
+    STGi $sp, %stack.0.a, 0 :: (store (s128) into %ir.a)
+    STGi $sp, %stack.1.b, 0 :: (store (s128) into %ir.b)
+    STGi $sp, %stack.2.c, 0 :: (store (s128) into %ir.c)
     $w0 = COPY $wzr
-    STGOffset $sp, %stack.3.d, 0 :: (store (s128) into %ir.d)
+    STGi $sp, %stack.3.d, 0 :: (store (s128) into %ir.d)
     RET_ReallyLR implicit killed $w0
 
 ...
@@ -56,12 +56,12 @@ body:             |
 ---
 # A store in the middle prevents merging.
 # CHECK-LABEL: name:{{.*}}stg16_store_128
-# CHECK: ST2GOffset $sp, $sp, 2
-# CHECK: ST2GOffset $sp, $sp, 4
-# CHECK: ST2GOffset $sp, $sp, 6
-# CHECK: STGOffset  $sp, $sp, 8
+# CHECK: ST2Gi $sp, $sp, 2
+# CHECK: ST2Gi $sp, $sp, 4
+# CHECK: ST2Gi $sp, $sp, 6
+# CHECK: STGi  $sp, $sp, 8
 # CHECK: STRBBui
-# CHECK: ST2GOffset $sp, $sp, 0
+# CHECK: ST2Gi $sp, $sp, 0
 # CHECK: RET_ReallyLR
 
 name:            stg16_store_128
@@ -71,13 +71,13 @@ stack:
   - { id: 1, name: b, size: 128, alignment: 16 }
 body:             |
   bb.0.entry:
-    STGOffset $sp, %stack.0.a, 0 :: (store (s128) into %ir.a)
+    STGi $sp, %stack.0.a, 0 :: (store (s128) into %ir.a)
     renamable $w8 = MOVi32imm 42
-    ST2GOffset $sp, %stack.1.b, 6 :: (store (s256) into %ir.b + 96, align 16)
-    ST2GOffset $sp, %stack.1.b, 4 :: (store (s256) into %ir.b + 64, align 16)
-    ST2GOffset $sp, %stack.1.b, 2 :: (store (s256) into %ir.b + 32, align 16)
+    ST2Gi $sp, %stack.1.b, 6 :: (store (s256) into %ir.b + 96, align 16)
+    ST2Gi $sp, %stack.1.b, 4 :: (store (s256) into %ir.b + 64, align 16)
+    ST2Gi $sp, %stack.1.b, 2 :: (store (s256) into %ir.b + 32, align 16)
     STRBBui killed renamable $w8, %stack.0.a, 0 :: (store (s8) into %ir.a, align 16)
-    ST2GOffset $sp, %stack.1.b, 0 :: (store (s256) into %ir.b, align 16)
+    ST2Gi $sp, %stack.1.b, 0 :: (store (s256) into %ir.b, align 16)
     RET_ReallyLR
 
 ...
diff --git a/llvm/test/CodeGen/AArch64/sve-fcmp.ll b/llvm/test/CodeGen/AArch64/sve-fcmp.ll
index f4b38cf771f01..35cbe65c6a8b8 100644
--- a/llvm/test/CodeGen/AArch64/sve-fcmp.ll
+++ b/llvm/test/CodeGen/AArch64/sve-fcmp.ll
@@ -486,3 +486,61 @@ define <vscale x 2 x i1> @une_zero_pred(<vscale x 2 x i1> %pg, <vscale x 2 x dou
   %z = and <vscale x 2 x i1> %pg, %y
   ret <vscale x 2 x i1> %z
 }
+
+%svboolx2 = type { <vscale x 4 x i1>, <vscale x 4 x i1> }
+
+define %svboolx2 @and_of_multiuse_fcmp_ogt(<vscale x 4 x i1> %pg, <vscale x 4 x float> %x, <vscale x 4 x float> %y) {
+; CHECK-LABEL: and_of_multiuse_fcmp_ogt:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p1.s
+; CHECK-NEXT:    fcmgt p1.s, p1/z, z0.s, z1.s
+; CHECK-NEXT:    and p0.b, p0/z, p0.b, p1.b
+; CHECK-NEXT:    ret
+  %cmp = fcmp ogt <vscale x 4 x float> %x, %y
+  %and = and <vscale x 4 x i1> %pg, %cmp
+  %ins.1 = insertvalue %svboolx2 poison, <vscale x 4 x i1> %and, 0
+  %ins.2 = insertvalue %svboolx2 %ins.1, <vscale x 4 x i1> %cmp, 1
+  ret %svboolx2 %ins.2
+}
+
+define %svboolx2 @and_of_multiuse_fcmp_ogt_zero(<vscale x 4 x i1> %pg, <vscale x 4 x float> %x) {
+; CHECK-LABEL: and_of_multiuse_fcmp_ogt_zero:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p1.s
+; CHECK-NEXT:    fcmgt p1.s, p1/z, z0.s, #0.0
+; CHECK-NEXT:    and p0.b, p0/z, p0.b, p1.b
+; CHECK-NEXT:    ret
+  %cmp = fcmp ogt <vscale x 4 x float> %x, zeroinitializer
+  %and = and <vscale x 4 x i1> %pg, %cmp
+  %ins.1 = insertvalue %svboolx2 poison, <vscale x 4 x i1> %and, 0
+  %ins.2 = insertvalue %svboolx2 %ins.1, <vscale x 4 x i1> %cmp, 1
+  ret %svboolx2 %ins.2
+}
+
+define %svboolx2 @and_of_multiuse_fcmp_olt(<vscale x 4 x i1> %pg, <vscale x 4 x float> %x, <vscale x 4 x float> %y) {
+; CHECK-LABEL: and_of_multiuse_fcmp_olt:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p1.s
+; CHECK-NEXT:    fcmgt p1.s, p1/z, z1.s, z0.s
+; CHECK-NEXT:    and p0.b, p0/z, p0.b, p1.b
+; CHECK-NEXT:    ret
+  %cmp = fcmp olt <vscale x 4 x float> %x, %y
+  %and = and <vscale x 4 x i1> %pg, %cmp
+  %ins.1 = insertvalue %svboolx2 poison, <vscale x 4 x i1> %and, 0
+  %ins.2 = insertvalue %svboolx2 %ins.1, <vscale x 4 x i1> %cmp, 1
+  ret %svboolx2 %ins.2
+}
+
+define %svboolx2 @and_of_multiuse_fcmp_olt_zero(<vscale x 4 x i1> %pg, <vscale x 4 x float> %x) {
+; CHECK-LABEL: and_of_multiuse_fcmp_olt_zero:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p1.s
+; CHECK-NEXT:    fcmlt p1.s, p1/z, z0.s, #0.0
+; CHECK-NEXT:    and p0.b, p0/z, p0.b, p1.b
+; CHECK-NEXT:    ret
+  %cmp = fcmp olt <vscale x 4 x float> %x, zeroinitializer
+  %and = and <vscale x 4 x i1> %pg, %cmp
+  %ins.1 = insertvalue %svboolx2 poison, <vscale x 4 x i1> %and, 0
+  %ins.2 = insertvalue %svboolx2 %ins.1, <vscale x 4 x i1> %cmp, 1
+  ret %svboolx2 %ins.2
+}
diff --git a/llvm/test/CodeGen/AArch64/sve-fixed-ld2-alloca.ll b/llvm/test/CodeGen/AArch64/sve-fixed-ld2-alloca.ll
index 0fc1f2ddc33b1..eaa99239b09e8 100644
--- a/llvm/test/CodeGen/AArch64/sve-fixed-ld2-alloca.ll
+++ b/llvm/test/CodeGen/AArch64/sve-fixed-ld2-alloca.ll
@@ -8,16 +8,19 @@ declare void @def(ptr)
 define void @st1d_fixed(ptr %ptr) #0 {
 ; CHECK-LABEL: st1d_fixed:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    sub sp, sp, #144
-; CHECK-NEXT:    stp x30, x19, [sp, #128] // 16-byte Folded Spill
+; CHECK-NEXT:    sub sp, sp, #160
+; CHECK-NEXT:    stp x20, x19, [sp, #144] // 16-byte Folded Spill
 ; CHECK-NEXT:    mov x19, x0
 ; CHECK-NEXT:    mov x0, sp
+; CHECK-NEXT:    str x30, [sp, #128] // 8-byte Folded Spill
+; CHECK-NEXT:    mov x20, sp
 ; CHECK-NEXT:    bl def
 ; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    ld2d { z0.d, z1.d }, p0/z, [sp]
+; CHECK-NEXT:    ld2d { z0.d, z1.d }, p0/z, [x20]
+; CHECK-NEXT:    ldr x30, [sp, #128] // 8-byte Folded Reload
 ; CHECK-NEXT:    st1d { z0.d }, p0, [x19]
-; CHECK-NEXT:    ldp x30, x19, [sp, #128] // 16-byte Folded Reload
-; CHECK-NEXT:    add sp, sp, #144
+; CHECK-NEXT:    ldp x20, x19, [sp, #144] // 16-byte Folded Reload
+; CHECK-NEXT:    add sp, sp, #160
 ; CHECK-NEXT:    ret
   %alloc = alloca [16 x double]
   call void @def(ptr %alloc)
diff --git a/llvm/test/CodeGen/AArch64/sve-fixed-length-frame-offests.ll b/llvm/test/CodeGen/AArch64/sve-fixed-length-frame-offests.ll
index 320d171eb6682..281ffff8e5eeb 100644
--- a/llvm/test/CodeGen/AArch64/sve-fixed-length-frame-offests.ll
+++ b/llvm/test/CodeGen/AArch64/sve-fixed-length-frame-offests.ll
@@ -9,13 +9,14 @@ target triple = "aarch64-unknown-linux-gnu"
 ; accessing fixed width objects.
 define void @foo(ptr %a) #0 {
 ; CHECK-LABEL: foo:
-; CHECK:       SelectionDAG has 14 nodes:
+; CHECK:       SelectionDAG has 15 nodes:
 ; CHECK-NEXT:    t0: ch,glue = EntryToken
 ; CHECK-NEXT:    t12: nxv2i1 = PTRUE_D TargetConstant:i32<31>
 ; CHECK-NEXT:    t2: i64,ch = CopyFromReg t0, Register:i64 %0
 ; CHECK-NEXT:    t18: nxv2i64,ch = LD1D_IMM<Mem:(volatile load (s512) from %ir.a)> t12, t2, TargetConstant:i64<0>, t0
 ; CHECK-NEXT:    t8: i64 = ADDXri TargetFrameIndex:i64<1>, TargetConstant:i32<0>, TargetConstant:i32<0>
-; CHECK-NEXT:    t17: ch = ST1D_IMM<Mem:(volatile store (s512) into %ir.r0)> t18, t12, TargetFrameIndex:i64<0>, TargetConstant:i64<0>, t18:1
+; CHECK-NEXT:    t6: i64 = ADDXri TargetFrameIndex:i64<0>, TargetConstant:i32<0>, TargetConstant:i32<0>
+; CHECK-NEXT:    t17: ch = ST1D_IMM<Mem:(volatile store (s512) into %ir.r0)> t18, t12, t6, TargetConstant:i64<0>, t18:1
 ; CHECK-NEXT:    t16: ch = ST1D_IMM<Mem:(volatile store (s512) into %ir.r1)> t18, t12, t8, TargetConstant:i64<0>, t17
 ; CHECK-NEXT:    t10: ch = RET_ReallyLR t16
 ; CHECK-EMPTY:
diff --git a/llvm/test/CodeGen/AArch64/sve-fixed-length-permute-rev.ll b/llvm/test/CodeGen/AArch64/sve-fixed-length-permute-rev.ll
index a64e6bfec573d..abeb88a0a317b 100644
--- a/llvm/test/CodeGen/AArch64/sve-fixed-length-permute-rev.ll
+++ b/llvm/test/CodeGen/AArch64/sve-fixed-length-permute-rev.ll
@@ -205,14 +205,15 @@ define void @test_rev_elts_fail(ptr %a) #1 {
 ; CHECK-NEXT:    ptrue p0.d
 ; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; CHECK-NEXT:    mov z1.d, z0.d[2]
-; CHECK-NEXT:    mov z2.d, z0.d[3]
-; CHECK-NEXT:    mov x10, v0.d[1]
-; CHECK-NEXT:    fmov x8, d1
-; CHECK-NEXT:    fmov x9, d2
 ; CHECK-NEXT:    fmov x11, d0
+; CHECK-NEXT:    fmov x8, d1
+; CHECK-NEXT:    mov z1.d, z0.d[3]
+; CHECK-NEXT:    fmov x9, d1
+; CHECK-NEXT:    mov x10, v0.d[1]
 ; CHECK-NEXT:    stp x9, x8, [sp, #16]
+; CHECK-NEXT:    mov x8, sp
 ; CHECK-NEXT:    stp x10, x11, [sp]
-; CHECK-NEXT:    ld1d { z0.d }, p0/z, [sp]
+; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x8]
 ; CHECK-NEXT:    st1d { z0.d }, p0, [x0]
 ; CHECK-NEXT:    mov sp, x29
 ; CHECK-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
@@ -273,9 +274,9 @@ define void @test_revv8i32(ptr %a) #0 {
 ; CHECK-NEXT:    ptrue p0.s, vl8
 ; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; CHECK-NEXT:    mov w8, v0.s[1]
+; CHECK-NEXT:    fmov w10, s0
 ; CHECK-NEXT:    mov w9, v0.s[2]
 ; CHECK-NEXT:    mov w11, v0.s[3]
-; CHECK-NEXT:    fmov w10, s0
 ; CHECK-NEXT:    mov z1.s, z0.s[4]
 ; CHECK-NEXT:    mov z2.s, z0.s[5]
 ; CHECK-NEXT:    mov z3.s, z0.s[6]
@@ -287,8 +288,9 @@ define void @test_revv8i32(ptr %a) #0 {
 ; CHECK-NEXT:    fmov w9, s3
 ; CHECK-NEXT:    fmov w11, s0
 ; CHECK-NEXT:    stp w8, w10, [sp, #8]
+; CHECK-NEXT:    mov x8, sp
 ; CHECK-NEXT:    stp w11, w9, [sp]
-; CHECK-NEXT:    ld1w { z0.s }, p0/z, [sp]
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x8]
 ; CHECK-NEXT:    st1w { z0.s }, p0, [x0]
 ; CHECK-NEXT:    mov sp, x29
 ; CHECK-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
@@ -392,19 +394,21 @@ define void @test_rev_fail(ptr %a) #1 {
 ; CHECK-NEXT:    mov z1.h, z0.h[8]
 ; CHECK-NEXT:    fmov w8, s0
 ; CHECK-NEXT:    fmov w9, s1
+; CHECK-NEXT:    mov z4.h, z0.h[11]
 ; CHECK-NEXT:    mov z5.h, z0.h[12]
 ; CHECK-NEXT:    mov z2.h, z0.h[9]
+; CHECK-NEXT:    strh w8, [sp, #14]
+; CHECK-NEXT:    fmov w8, s4
 ; CHECK-NEXT:    mov z3.h, z0.h[10]
-; CHECK-NEXT:    mov z4.h, z0.h[11]
-; CHECK-NEXT:    fmov w11, s2
 ; CHECK-NEXT:    strh w9, [sp, #30]
 ; CHECK-NEXT:    fmov w9, s5
+; CHECK-NEXT:    mov z16.h, z0.h[15]
+; CHECK-NEXT:    fmov w11, s2
 ; CHECK-NEXT:    fmov w12, s3
-; CHECK-NEXT:    strh w8, [sp, #14]
-; CHECK-NEXT:    fmov w8, s4
+; CHECK-NEXT:    strh w8, [sp, #24]
+; CHECK-NEXT:    fmov w8, s16
 ; CHECK-NEXT:    mov z6.h, z0.h[13]
 ; CHECK-NEXT:    mov z7.h, z0.h[14]
-; CHECK-NEXT:    mov z16.h, z0.h[15]
 ; CHECK-NEXT:    umov w10, v0.h[1]
 ; CHECK-NEXT:    strh w9, [sp, #22]
 ; CHECK-NEXT:    umov w9, v0.h[2]
@@ -412,24 +416,23 @@ define void @test_rev_fail(ptr %a) #1 {
 ; CHECK-NEXT:    fmov w11, s6
 ; CHECK-NEXT:    strh w12, [sp, #26]
 ; CHECK-NEXT:    fmov w12, s7
-; CHECK-NEXT:    strh w8, [sp, #24]
-; CHECK-NEXT:    fmov w8, s16
+; CHECK-NEXT:    strh w8, [sp, #16]
+; CHECK-NEXT:    umov w8, v0.h[5]
 ; CHECK-NEXT:    strh w10, [sp, #12]
 ; CHECK-NEXT:    strh w11, [sp, #20]
 ; CHECK-NEXT:    umov w11, v0.h[3]
 ; CHECK-NEXT:    strh w12, [sp, #18]
 ; CHECK-NEXT:    umov w12, v0.h[4]
-; CHECK-NEXT:    strh w8, [sp, #16]
-; CHECK-NEXT:    umov w8, v0.h[5]
 ; CHECK-NEXT:    umov w10, v0.h[6]
 ; CHECK-NEXT:    strh w9, [sp, #10]
 ; CHECK-NEXT:    umov w9, v0.h[7]
+; CHECK-NEXT:    strh w8, [sp, #4]
+; CHECK-NEXT:    mov x8, sp
 ; CHECK-NEXT:    strh w11, [sp, #8]
 ; CHECK-NEXT:    strh w12, [sp, #6]
-; CHECK-NEXT:    strh w8, [sp, #4]
 ; CHECK-NEXT:    strh w10, [sp, #2]
 ; CHECK-NEXT:    strh w9, [sp]
-; CHECK-NEXT:    ld1h { z0.h }, p0/z, [sp]
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x8]
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x0]
 ; CHECK-NEXT:    mov sp, x29
 ; CHECK-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
@@ -457,39 +460,39 @@ define void @test_revv8i16v8i16(ptr %a, ptr %b, ptr %c) #1 {
 ; CHECK-NEXT:    orr x9, x8, #0x1e
 ; CHECK-NEXT:    orr x10, x8, #0x1c
 ; CHECK-NEXT:    ldr q1, [x0]
-; CHECK-NEXT:    orr x12, x8, #0x10
 ; CHECK-NEXT:    orr x11, x8, #0x18
+; CHECK-NEXT:    orr x12, x8, #0x10
 ; CHECK-NEXT:    str h0, [sp, #22]
 ; CHECK-NEXT:    st1 { v0.h }[4], [x9]
 ; CHECK-NEXT:    orr x9, x8, #0xe
 ; CHECK-NEXT:    st1 { v0.h }[5], [x10]
 ; CHECK-NEXT:    orr x10, x8, #0xc
-; CHECK-NEXT:    st1 { v0.h }[3], [x12]
-; CHECK-NEXT:    mov w12, #26
-; CHECK-NEXT:    st1 { v1.h }[4], [x9]
-; CHECK-NEXT:    orr x9, x8, #0x8
 ; CHECK-NEXT:    st1 { v0.h }[7], [x11]
-; CHECK-NEXT:    orr x11, x8, #0x2
+; CHECK-NEXT:    orr x11, x8, #0x8
+; CHECK-NEXT:    st1 { v1.h }[4], [x9]
+; CHECK-NEXT:    orr x9, x8, #0x4
 ; CHECK-NEXT:    st1 { v1.h }[5], [x10]
-; CHECK-NEXT:    orr x10, x8, #0x4
-; CHECK-NEXT:    st1 { v1.h }[7], [x9]
+; CHECK-NEXT:    mov w10, #26
+; CHECK-NEXT:    orr x10, x8, x10
+; CHECK-NEXT:    st1 { v0.h }[3], [x12]
+; CHECK-NEXT:    st1 { v1.h }[1], [x9]
+; CHECK-NEXT:    orr x9, x8, #0x2
+; CHECK-NEXT:    st1 { v1.h }[7], [x11]
+; CHECK-NEXT:    mov w11, #20
+; CHECK-NEXT:    mov w12, #18
+; CHECK-NEXT:    st1 { v0.h }[6], [x10]
+; CHECK-NEXT:    mov w10, #10
+; CHECK-NEXT:    orr x11, x8, x11
+; CHECK-NEXT:    st1 { v1.h }[2], [x9]
 ; CHECK-NEXT:    orr x9, x8, x12
-; CHECK-NEXT:    st1 { v1.h }[2], [x11]
-; CHECK-NEXT:    mov w11, #10
-; CHECK-NEXT:    st1 { v1.h }[1], [x10]
-; CHECK-NEXT:    mov w10, #18
-; CHECK-NEXT:    st1 { v0.h }[6], [x9]
-; CHECK-NEXT:    mov w9, #20
-; CHECK-NEXT:    orr x9, x8, x9
 ; CHECK-NEXT:    orr x10, x8, x10
 ; CHECK-NEXT:    st1 { v1.h }[3], [x8]
-; CHECK-NEXT:    orr x8, x8, x11
-; CHECK-NEXT:    str h1, [sp, #6]
+; CHECK-NEXT:    st1 { v0.h }[1], [x11]
 ; CHECK-NEXT:    ptrue p0.h
-; CHECK-NEXT:    st1 { v0.h }[1], [x9]
-; CHECK-NEXT:    st1 { v0.h }[2], [x10]
-; CHECK-NEXT:    st1 { v1.h }[6], [x8]
-; CHECK-NEXT:    ld1h { z0.h }, p0/z, [sp]
+; CHECK-NEXT:    st1 { v0.h }[2], [x9]
+; CHECK-NEXT:    st1 { v1.h }[6], [x10]
+; CHECK-NEXT:    str h1, [sp, #6]
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x8]
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    mov sp, x29
 ; CHECK-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
diff --git a/llvm/test/CodeGen/AArch64/sve-fixed-length-permute-zip-uzp-trn.ll b/llvm/test/CodeGen/AArch64/sve-fixed-length-permute-zip-uzp-trn.ll
index 0946b776d003a..8387f6a47aea1 100644
--- a/llvm/test/CodeGen/AArch64/sve-fixed-length-permute-zip-uzp-trn.ll
+++ b/llvm/test/CodeGen/AArch64/sve-fixed-length-permute-zip-uzp-trn.ll
@@ -140,6 +140,7 @@ define void @zip_v4f64(ptr %a, ptr %b) #0 {
 ; VBITS_EQ_512-NEXT:    sub x9, sp, #48
 ; VBITS_EQ_512-NEXT:    and sp, x9, #0xffffffffffffffe0
 ; VBITS_EQ_512-NEXT:    ptrue p0.d, vl4
+; VBITS_EQ_512-NEXT:    mov x8, sp
 ; VBITS_EQ_512-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_EQ_512-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_EQ_512-NEXT:    mov z2.d, z1.d[3]
@@ -149,7 +150,7 @@ define void @zip_v4f64(ptr %a, ptr %b) #0 {
 ; VBITS_EQ_512-NEXT:    mov z3.d, z0.d[2]
 ; VBITS_EQ_512-NEXT:    zip1 z0.d, z0.d, z1.d
 ; VBITS_EQ_512-NEXT:    stp d3, d2, [sp]
-; VBITS_EQ_512-NEXT:    ld1d { z2.d }, p0/z, [sp]
+; VBITS_EQ_512-NEXT:    ld1d { z2.d }, p0/z, [x8]
 ; VBITS_EQ_512-NEXT:    fadd z0.d, p0/m, z0.d, z2.d
 ; VBITS_EQ_512-NEXT:    st1d { z0.d }, p0, [x0]
 ; VBITS_EQ_512-NEXT:    mov sp, x29
@@ -657,6 +658,7 @@ define void @zip_vscale2_4(ptr %a, ptr %b) #2 {
 ; CHECK-NEXT:    sub x9, sp, #48
 ; CHECK-NEXT:    and sp, x9, #0xffffffffffffffe0
 ; CHECK-NEXT:    ptrue p0.d, vl4
+; CHECK-NEXT:    mov x8, sp
 ; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; CHECK-NEXT:    mov z2.d, z1.d[3]
@@ -666,7 +668,7 @@ define void @zip_vscale2_4(ptr %a, ptr %b) #2 {
 ; CHECK-NEXT:    mov z3.d, z0.d[2]
 ; CHECK-NEXT:    zip1 z0.d, z0.d, z1.d
 ; CHECK-NEXT:    stp d3, d2, [sp]
-; CHECK-NEXT:    ld1d { z2.d }, p0/z, [sp]
+; CHECK-NEXT:    ld1d { z2.d }, p0/z, [x8]
 ; CHECK-NEXT:    fadd z0.d, p0/m, z0.d, z2.d
 ; CHECK-NEXT:    st1d { z0.d }, p0, [x0]
 ; CHECK-NEXT:    mov sp, x29
diff --git a/llvm/test/CodeGen/AArch64/sve-fixed-length-vector-shuffle.ll b/llvm/test/CodeGen/AArch64/sve-fixed-length-vector-shuffle.ll
index 3df909fb23b2b..b3660cadba4d0 100644
--- a/llvm/test/CodeGen/AArch64/sve-fixed-length-vector-shuffle.ll
+++ b/llvm/test/CodeGen/AArch64/sve-fixed-length-vector-shuffle.ll
@@ -918,6 +918,7 @@ define void @shuffle_ext_invalid(ptr %a, ptr %b) vscale_range(2,0) #0 {
 ; CHECK-NEXT:    sub x9, sp, #48
 ; CHECK-NEXT:    and sp, x9, #0xffffffffffffffe0
 ; CHECK-NEXT:    ptrue p0.d, vl4
+; CHECK-NEXT:    mov x8, sp
 ; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; CHECK-NEXT:    mov z2.d, z1.d[1]
@@ -925,7 +926,7 @@ define void @shuffle_ext_invalid(ptr %a, ptr %b) vscale_range(2,0) #0 {
 ; CHECK-NEXT:    mov z1.d, z0.d[3]
 ; CHECK-NEXT:    mov z0.d, z0.d[2]
 ; CHECK-NEXT:    stp d0, d1, [sp]
-; CHECK-NEXT:    ld1d { z0.d }, p0/z, [sp]
+; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x8]
 ; CHECK-NEXT:    st1d { z0.d }, p0, [x0]
 ; CHECK-NEXT:    mov sp, x29
 ; CHECK-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
diff --git a/llvm/test/CodeGen/AArch64/sve-fp-int-min-max.ll b/llvm/test/CodeGen/AArch64/sve-fp-int-min-max.ll
index 2e6f704975bd3..2d4d0b74a08e1 100644
--- a/llvm/test/CodeGen/AArch64/sve-fp-int-min-max.ll
+++ b/llvm/test/CodeGen/AArch64/sve-fp-int-min-max.ll
@@ -23,12 +23,11 @@ define i64 @scalable_int_min_max(ptr %arg, ptr %arg1, <vscale x 2 x ptr> %i37, <
 ; CHECK-NEXT:    sxtw z5.d, p0/m, z6.d
 ; CHECK-NEXT:    smin z4.d, p0/m, z4.d, z5.d
 ; CHECK-NEXT:    cmpne p1.d, p0/z, z4.d, #0
-; CHECK-NEXT:    ld1w { z5.d }, p1/z, [x1]
+; CHECK-NEXT:    ld1w { z4.d }, p1/z, [x1]
 ; CHECK-NEXT:    ld1w { z0.d }, p1/z, [z0.d]
-; CHECK-NEXT:    fadd z0.s, p0/m, z0.s, z5.s
+; CHECK-NEXT:    fadd z0.s, p0/m, z0.s, z4.s
 ; CHECK-NEXT:    fcmge p2.s, p0/z, z0.s, z3.s
-; CHECK-NEXT:    not p2.b, p0/z, p2.b
-; CHECK-NEXT:    cmpne p2.d, p2/z, z4.d, #0
+; CHECK-NEXT:    bic p2.b, p1/z, p1.b, p2.b
 ; CHECK-NEXT:    mov z1.d, p2/m, #0 // =0x0
 ; CHECK-NEXT:    add z2.d, p1/m, z2.d, z1.d
 ; CHECK-NEXT:    uaddv d0, p0, z2.d
diff --git a/llvm/test/CodeGen/AArch64/sve-gather-scatter-addr-opts.ll b/llvm/test/CodeGen/AArch64/sve-gather-scatter-addr-opts.ll
index b32dad71bbcb5..f96e411885df0 100644
--- a/llvm/test/CodeGen/AArch64/sve-gather-scatter-addr-opts.ll
+++ b/llvm/test/CodeGen/AArch64/sve-gather-scatter-addr-opts.ll
@@ -72,19 +72,18 @@ define void @scatter_f16_index_offset_var(ptr %base, i64 %offset, i64 %scale, <v
 ; CHECK-LABEL: scatter_f16_index_offset_var:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    index z1.d, #0, #1
-; CHECK-NEXT:    mov z3.d, x1
-; CHECK-NEXT:    mov z2.d, z1.d
-; CHECK-NEXT:    mov z4.d, z3.d
 ; CHECK-NEXT:    ptrue p1.d
+; CHECK-NEXT:    mov z2.d, z1.d
+; CHECK-NEXT:    mov z3.d, x1
 ; CHECK-NEXT:    incd z2.d
-; CHECK-NEXT:    mla z3.d, p1/m, z1.d, z3.d
-; CHECK-NEXT:    mla z4.d, p1/m, z2.d, z4.d
+; CHECK-NEXT:    mad z1.d, p1/m, z3.d, z3.d
+; CHECK-NEXT:    mad z2.d, p1/m, z3.d, z3.d
 ; CHECK-NEXT:    punpklo p1.h, p0.b
-; CHECK-NEXT:    uunpklo z1.d, z0.s
+; CHECK-NEXT:    uunpklo z3.d, z0.s
 ; CHECK-NEXT:    punpkhi p0.h, p0.b
 ; CHECK-NEXT:    uunpkhi z0.d, z0.s
-; CHECK-NEXT:    st1h { z1.d }, p1, [x0, z3.d, lsl #1]
-; CHECK-NEXT:    st1h { z0.d }, p0, [x0, z4.d, lsl #1]
+; CHECK-NEXT:    st1h { z3.d }, p1, [x0, z1.d, lsl #1]
+; CHECK-NEXT:    st1h { z0.d }, p0, [x0, z2.d, lsl #1]
 ; CHECK-NEXT:    ret
   %t0 = insertelement <vscale x 4 x i64> undef, i64 %offset, i32 0
   %t1 = shufflevector <vscale x 4 x i64> %t0, <vscale x 4 x i64> undef, <vscale x 4 x i32> zeroinitializer
diff --git a/llvm/test/CodeGen/AArch64/sve-hadd.ll b/llvm/test/CodeGen/AArch64/sve-hadd.ll
new file mode 100644
index 0000000000000..82f9e51d9ca2e
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/sve-hadd.ll
@@ -0,0 +1,1295 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=aarch64-none-linux-gnu  -mattr=+sve | FileCheck %s -check-prefixes=CHECK,SVE
+; RUN: llc < %s -mtriple=aarch64-none-linux-gnu  -mattr=+sve2 | FileCheck %s -check-prefixes=CHECK,SVE2
+
+define <vscale x 2 x i64> @hadds_v2i64(<vscale x 2 x i64> %s0, <vscale x 2 x i64> %s1) {
+; SVE-LABEL: hadds_v2i64:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    asr z2.d, z1.d, #1
+; SVE-NEXT:    asr z3.d, z0.d, #1
+; SVE-NEXT:    and z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.d, z3.d, z2.d
+; SVE-NEXT:    and z0.d, z0.d, #0x1
+; SVE-NEXT:    add z0.d, z1.d, z0.d
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: hadds_v2i64:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.d
+; SVE2-NEXT:    shadd z0.d, p0/m, z0.d, z1.d
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 2 x i64> %s0 to <vscale x 2 x i128>
+  %s1s = sext <vscale x 2 x i64> %s1 to <vscale x 2 x i128>
+  %m = add nsw <vscale x 2 x i128> %s0s, %s1s
+  %s = ashr <vscale x 2 x i128> %m, shufflevector (<vscale x 2 x i128> insertelement (<vscale x 2 x i128> poison, i128 1, i32 0), <vscale x 2 x i128> poison, <vscale x 2 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 2 x i128> %s to <vscale x 2 x i64>
+  ret <vscale x 2 x i64> %s2
+}
+
+define <vscale x 2 x i64> @hadds_v2i64_lsh(<vscale x 2 x i64> %s0, <vscale x 2 x i64> %s1) {
+; SVE-LABEL: hadds_v2i64_lsh:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    asr z2.d, z1.d, #1
+; SVE-NEXT:    asr z3.d, z0.d, #1
+; SVE-NEXT:    and z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.d, z3.d, z2.d
+; SVE-NEXT:    and z0.d, z0.d, #0x1
+; SVE-NEXT:    add z0.d, z1.d, z0.d
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: hadds_v2i64_lsh:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.d
+; SVE2-NEXT:    shadd z0.d, p0/m, z0.d, z1.d
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 2 x i64> %s0 to <vscale x 2 x i128>
+  %s1s = sext <vscale x 2 x i64> %s1 to <vscale x 2 x i128>
+  %m = add nsw <vscale x 2 x i128> %s0s, %s1s
+  %s = lshr <vscale x 2 x i128> %m, shufflevector (<vscale x 2 x i128> insertelement (<vscale x 2 x i128> poison, i128 1, i32 0), <vscale x 2 x i128> poison, <vscale x 2 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 2 x i128> %s to <vscale x 2 x i64>
+  ret <vscale x 2 x i64> %s2
+}
+
+define <vscale x 2 x i64> @haddu_v2i64(<vscale x 2 x i64> %s0, <vscale x 2 x i64> %s1) {
+; SVE-LABEL: haddu_v2i64:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    lsr z2.d, z1.d, #1
+; SVE-NEXT:    lsr z3.d, z0.d, #1
+; SVE-NEXT:    and z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.d, z3.d, z2.d
+; SVE-NEXT:    and z0.d, z0.d, #0x1
+; SVE-NEXT:    add z0.d, z1.d, z0.d
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: haddu_v2i64:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.d
+; SVE2-NEXT:    uhadd z0.d, p0/m, z0.d, z1.d
+; SVE2-NEXT:    ret
+entry:
+  %s0s = zext <vscale x 2 x i64> %s0 to <vscale x 2 x i128>
+  %s1s = zext <vscale x 2 x i64> %s1 to <vscale x 2 x i128>
+  %m = add nuw nsw <vscale x 2 x i128> %s0s, %s1s
+  %s = lshr <vscale x 2 x i128> %m, shufflevector (<vscale x 2 x i128> insertelement (<vscale x 2 x i128> poison, i128 1, i32 0), <vscale x 2 x i128> poison, <vscale x 2 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 2 x i128> %s to <vscale x 2 x i64>
+  ret <vscale x 2 x i64> %s2
+}
+
+define <vscale x 2 x i32> @hadds_v2i32(<vscale x 2 x i32> %s0, <vscale x 2 x i32> %s1) {
+; SVE-LABEL: hadds_v2i32:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    ptrue p0.d
+; SVE-NEXT:    sxtw z0.d, p0/m, z0.d
+; SVE-NEXT:    adr z0.d, [z0.d, z1.d, sxtw]
+; SVE-NEXT:    asr z0.d, z0.d, #1
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: hadds_v2i32:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.d
+; SVE2-NEXT:    sxtw z0.d, p0/m, z0.d
+; SVE2-NEXT:    sxtw z1.d, p0/m, z1.d
+; SVE2-NEXT:    shadd z0.d, p0/m, z0.d, z1.d
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 2 x i32> %s0 to <vscale x 2 x i64>
+  %s1s = sext <vscale x 2 x i32> %s1 to <vscale x 2 x i64>
+  %m = add nsw <vscale x 2 x i64> %s0s, %s1s
+  %s = ashr <vscale x 2 x i64> %m, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 1, i32 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 2 x i64> %s to <vscale x 2 x i32>
+  ret <vscale x 2 x i32> %s2
+}
+
+define <vscale x 2 x i32> @hadds_v2i32_lsh(<vscale x 2 x i32> %s0, <vscale x 2 x i32> %s1) {
+; CHECK-LABEL: hadds_v2i32_lsh:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    sxtw z0.d, p0/m, z0.d
+; CHECK-NEXT:    adr z0.d, [z0.d, z1.d, sxtw]
+; CHECK-NEXT:    lsr z0.d, z0.d, #1
+; CHECK-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 2 x i32> %s0 to <vscale x 2 x i64>
+  %s1s = sext <vscale x 2 x i32> %s1 to <vscale x 2 x i64>
+  %m = add nsw <vscale x 2 x i64> %s0s, %s1s
+  %s = lshr <vscale x 2 x i64> %m, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 1, i32 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 2 x i64> %s to <vscale x 2 x i32>
+  ret <vscale x 2 x i32> %s2
+}
+
+define <vscale x 2 x i32> @haddu_v2i32(<vscale x 2 x i32> %s0, <vscale x 2 x i32> %s1) {
+; SVE-LABEL: haddu_v2i32:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    and z0.d, z0.d, #0xffffffff
+; SVE-NEXT:    adr z0.d, [z0.d, z1.d, uxtw]
+; SVE-NEXT:    lsr z0.d, z0.d, #1
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: haddu_v2i32:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.d
+; SVE2-NEXT:    and z0.d, z0.d, #0xffffffff
+; SVE2-NEXT:    and z1.d, z1.d, #0xffffffff
+; SVE2-NEXT:    uhadd z0.d, p0/m, z0.d, z1.d
+; SVE2-NEXT:    ret
+entry:
+  %s0s = zext <vscale x 2 x i32> %s0 to <vscale x 2 x i64>
+  %s1s = zext <vscale x 2 x i32> %s1 to <vscale x 2 x i64>
+  %m = add nuw nsw <vscale x 2 x i64> %s0s, %s1s
+  %s = lshr <vscale x 2 x i64> %m, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 1, i32 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 2 x i64> %s to <vscale x 2 x i32>
+  ret <vscale x 2 x i32> %s2
+}
+
+define <vscale x 4 x i32> @hadds_v4i32(<vscale x 4 x i32> %s0, <vscale x 4 x i32> %s1) {
+; SVE-LABEL: hadds_v4i32:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    asr z2.s, z1.s, #1
+; SVE-NEXT:    asr z3.s, z0.s, #1
+; SVE-NEXT:    and z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.s, z3.s, z2.s
+; SVE-NEXT:    and z0.s, z0.s, #0x1
+; SVE-NEXT:    add z0.s, z1.s, z0.s
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: hadds_v4i32:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.s
+; SVE2-NEXT:    shadd z0.s, p0/m, z0.s, z1.s
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 4 x i32> %s0 to <vscale x 4 x i64>
+  %s1s = sext <vscale x 4 x i32> %s1 to <vscale x 4 x i64>
+  %m = add nsw <vscale x 4 x i64> %s0s, %s1s
+  %s = ashr <vscale x 4 x i64> %m, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 1, i32 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 4 x i64> %s to <vscale x 4 x i32>
+  ret <vscale x 4 x i32> %s2
+}
+
+define <vscale x 4 x i32> @hadds_v4i32_lsh(<vscale x 4 x i32> %s0, <vscale x 4 x i32> %s1) {
+; SVE-LABEL: hadds_v4i32_lsh:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    asr z2.s, z1.s, #1
+; SVE-NEXT:    asr z3.s, z0.s, #1
+; SVE-NEXT:    and z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.s, z3.s, z2.s
+; SVE-NEXT:    and z0.s, z0.s, #0x1
+; SVE-NEXT:    add z0.s, z1.s, z0.s
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: hadds_v4i32_lsh:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.s
+; SVE2-NEXT:    shadd z0.s, p0/m, z0.s, z1.s
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 4 x i32> %s0 to <vscale x 4 x i64>
+  %s1s = sext <vscale x 4 x i32> %s1 to <vscale x 4 x i64>
+  %m = add nsw <vscale x 4 x i64> %s0s, %s1s
+  %s = lshr <vscale x 4 x i64> %m, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 1, i32 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 4 x i64> %s to <vscale x 4 x i32>
+  ret <vscale x 4 x i32> %s2
+}
+
+define <vscale x 4 x i32> @haddu_v4i32(<vscale x 4 x i32> %s0, <vscale x 4 x i32> %s1) {
+; SVE-LABEL: haddu_v4i32:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    lsr z2.s, z1.s, #1
+; SVE-NEXT:    lsr z3.s, z0.s, #1
+; SVE-NEXT:    and z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.s, z3.s, z2.s
+; SVE-NEXT:    and z0.s, z0.s, #0x1
+; SVE-NEXT:    add z0.s, z1.s, z0.s
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: haddu_v4i32:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.s
+; SVE2-NEXT:    uhadd z0.s, p0/m, z0.s, z1.s
+; SVE2-NEXT:    ret
+entry:
+  %s0s = zext <vscale x 4 x i32> %s0 to <vscale x 4 x i64>
+  %s1s = zext <vscale x 4 x i32> %s1 to <vscale x 4 x i64>
+  %m = add nuw nsw <vscale x 4 x i64> %s0s, %s1s
+  %s = lshr <vscale x 4 x i64> %m, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 1, i32 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 4 x i64> %s to <vscale x 4 x i32>
+  ret <vscale x 4 x i32> %s2
+}
+
+define <vscale x 2 x i16> @hadds_v2i16(<vscale x 2 x i16> %s0, <vscale x 2 x i16> %s1) {
+; SVE-LABEL: hadds_v2i16:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    ptrue p0.d
+; SVE-NEXT:    sxth z0.d, p0/m, z0.d
+; SVE-NEXT:    sxth z1.d, p0/m, z1.d
+; SVE-NEXT:    add z0.d, z0.d, z1.d
+; SVE-NEXT:    asr z0.d, z0.d, #1
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: hadds_v2i16:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.d
+; SVE2-NEXT:    sxth z0.d, p0/m, z0.d
+; SVE2-NEXT:    sxth z1.d, p0/m, z1.d
+; SVE2-NEXT:    shadd z0.d, p0/m, z0.d, z1.d
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 2 x i16> %s0 to <vscale x 2 x i32>
+  %s1s = sext <vscale x 2 x i16> %s1 to <vscale x 2 x i32>
+  %m = add nsw <vscale x 2 x i32> %s0s, %s1s
+  %s = ashr <vscale x 2 x i32> %m, shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> poison, i32 1, i32 0), <vscale x 2 x i32> poison, <vscale x 2 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 2 x i32> %s to <vscale x 2 x i16>
+  ret <vscale x 2 x i16> %s2
+}
+
+define <vscale x 2 x i16> @hadds_v2i16_lsh(<vscale x 2 x i16> %s0, <vscale x 2 x i16> %s1) {
+; CHECK-LABEL: hadds_v2i16_lsh:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    sxth z0.d, p0/m, z0.d
+; CHECK-NEXT:    sxth z1.d, p0/m, z1.d
+; CHECK-NEXT:    add z0.d, z0.d, z1.d
+; CHECK-NEXT:    and z0.d, z0.d, #0xffffffff
+; CHECK-NEXT:    lsr z0.d, z0.d, #1
+; CHECK-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 2 x i16> %s0 to <vscale x 2 x i32>
+  %s1s = sext <vscale x 2 x i16> %s1 to <vscale x 2 x i32>
+  %m = add nsw <vscale x 2 x i32> %s0s, %s1s
+  %s = lshr <vscale x 2 x i32> %m, shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> poison, i32 1, i32 0), <vscale x 2 x i32> poison, <vscale x 2 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 2 x i32> %s to <vscale x 2 x i16>
+  ret <vscale x 2 x i16> %s2
+}
+
+define <vscale x 2 x i16> @haddu_v2i16(<vscale x 2 x i16> %s0, <vscale x 2 x i16> %s1) {
+; SVE-LABEL: haddu_v2i16:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    and z0.d, z0.d, #0xffff
+; SVE-NEXT:    and z1.d, z1.d, #0xffff
+; SVE-NEXT:    add z0.d, z0.d, z1.d
+; SVE-NEXT:    lsr z0.d, z0.d, #1
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: haddu_v2i16:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.d
+; SVE2-NEXT:    and z0.d, z0.d, #0xffff
+; SVE2-NEXT:    and z1.d, z1.d, #0xffff
+; SVE2-NEXT:    uhadd z0.d, p0/m, z0.d, z1.d
+; SVE2-NEXT:    ret
+entry:
+  %s0s = zext <vscale x 2 x i16> %s0 to <vscale x 2 x i32>
+  %s1s = zext <vscale x 2 x i16> %s1 to <vscale x 2 x i32>
+  %m = add nuw nsw <vscale x 2 x i32> %s0s, %s1s
+  %s = lshr <vscale x 2 x i32> %m, shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> poison, i32 1, i32 0), <vscale x 2 x i32> poison, <vscale x 2 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 2 x i32> %s to <vscale x 2 x i16>
+  ret <vscale x 2 x i16> %s2
+}
+
+define <vscale x 4 x i16> @hadds_v4i16(<vscale x 4 x i16> %s0, <vscale x 4 x i16> %s1) {
+; SVE-LABEL: hadds_v4i16:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    ptrue p0.s
+; SVE-NEXT:    sxth z0.s, p0/m, z0.s
+; SVE-NEXT:    sxth z1.s, p0/m, z1.s
+; SVE-NEXT:    add z0.s, z0.s, z1.s
+; SVE-NEXT:    asr z0.s, z0.s, #1
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: hadds_v4i16:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.s
+; SVE2-NEXT:    sxth z0.s, p0/m, z0.s
+; SVE2-NEXT:    sxth z1.s, p0/m, z1.s
+; SVE2-NEXT:    shadd z0.s, p0/m, z0.s, z1.s
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 4 x i16> %s0 to <vscale x 4 x i32>
+  %s1s = sext <vscale x 4 x i16> %s1 to <vscale x 4 x i32>
+  %m = add nsw <vscale x 4 x i32> %s0s, %s1s
+  %s = ashr <vscale x 4 x i32> %m, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 1, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 4 x i32> %s to <vscale x 4 x i16>
+  ret <vscale x 4 x i16> %s2
+}
+
+define <vscale x 4 x i16> @hadds_v4i16_lsh(<vscale x 4 x i16> %s0, <vscale x 4 x i16> %s1) {
+; CHECK-LABEL: hadds_v4i16_lsh:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    sxth z0.s, p0/m, z0.s
+; CHECK-NEXT:    sxth z1.s, p0/m, z1.s
+; CHECK-NEXT:    add z0.s, z0.s, z1.s
+; CHECK-NEXT:    lsr z0.s, z0.s, #1
+; CHECK-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 4 x i16> %s0 to <vscale x 4 x i32>
+  %s1s = sext <vscale x 4 x i16> %s1 to <vscale x 4 x i32>
+  %m = add nsw <vscale x 4 x i32> %s0s, %s1s
+  %s = lshr <vscale x 4 x i32> %m, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 1, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 4 x i32> %s to <vscale x 4 x i16>
+  ret <vscale x 4 x i16> %s2
+}
+
+define <vscale x 4 x i16> @haddu_v4i16(<vscale x 4 x i16> %s0, <vscale x 4 x i16> %s1) {
+; SVE-LABEL: haddu_v4i16:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    and z0.s, z0.s, #0xffff
+; SVE-NEXT:    and z1.s, z1.s, #0xffff
+; SVE-NEXT:    add z0.s, z0.s, z1.s
+; SVE-NEXT:    lsr z0.s, z0.s, #1
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: haddu_v4i16:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.s
+; SVE2-NEXT:    and z0.s, z0.s, #0xffff
+; SVE2-NEXT:    and z1.s, z1.s, #0xffff
+; SVE2-NEXT:    uhadd z0.s, p0/m, z0.s, z1.s
+; SVE2-NEXT:    ret
+entry:
+  %s0s = zext <vscale x 4 x i16> %s0 to <vscale x 4 x i32>
+  %s1s = zext <vscale x 4 x i16> %s1 to <vscale x 4 x i32>
+  %m = add nuw nsw <vscale x 4 x i32> %s0s, %s1s
+  %s = lshr <vscale x 4 x i32> %m, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 1, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 4 x i32> %s to <vscale x 4 x i16>
+  ret <vscale x 4 x i16> %s2
+}
+
+define <vscale x 8 x i16> @hadds_v8i16(<vscale x 8 x i16> %s0, <vscale x 8 x i16> %s1) {
+; SVE-LABEL: hadds_v8i16:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    asr z2.h, z1.h, #1
+; SVE-NEXT:    asr z3.h, z0.h, #1
+; SVE-NEXT:    and z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.h, z3.h, z2.h
+; SVE-NEXT:    and z0.h, z0.h, #0x1
+; SVE-NEXT:    add z0.h, z1.h, z0.h
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: hadds_v8i16:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.h
+; SVE2-NEXT:    shadd z0.h, p0/m, z0.h, z1.h
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 8 x i16> %s0 to <vscale x 8 x i32>
+  %s1s = sext <vscale x 8 x i16> %s1 to <vscale x 8 x i32>
+  %m = add nsw <vscale x 8 x i32> %s0s, %s1s
+  %s = ashr <vscale x 8 x i32> %m, shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 1, i32 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 8 x i32> %s to <vscale x 8 x i16>
+  ret <vscale x 8 x i16> %s2
+}
+
+define <vscale x 8 x i16> @hadds_v8i16_lsh(<vscale x 8 x i16> %s0, <vscale x 8 x i16> %s1) {
+; SVE-LABEL: hadds_v8i16_lsh:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    asr z2.h, z1.h, #1
+; SVE-NEXT:    asr z3.h, z0.h, #1
+; SVE-NEXT:    and z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.h, z3.h, z2.h
+; SVE-NEXT:    and z0.h, z0.h, #0x1
+; SVE-NEXT:    add z0.h, z1.h, z0.h
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: hadds_v8i16_lsh:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.h
+; SVE2-NEXT:    shadd z0.h, p0/m, z0.h, z1.h
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 8 x i16> %s0 to <vscale x 8 x i32>
+  %s1s = sext <vscale x 8 x i16> %s1 to <vscale x 8 x i32>
+  %m = add nsw <vscale x 8 x i32> %s0s, %s1s
+  %s = lshr <vscale x 8 x i32> %m, shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 1, i32 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 8 x i32> %s to <vscale x 8 x i16>
+  ret <vscale x 8 x i16> %s2
+}
+
+define <vscale x 8 x i16> @haddu_v8i16(<vscale x 8 x i16> %s0, <vscale x 8 x i16> %s1) {
+; SVE-LABEL: haddu_v8i16:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    lsr z2.h, z1.h, #1
+; SVE-NEXT:    lsr z3.h, z0.h, #1
+; SVE-NEXT:    and z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.h, z3.h, z2.h
+; SVE-NEXT:    and z0.h, z0.h, #0x1
+; SVE-NEXT:    add z0.h, z1.h, z0.h
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: haddu_v8i16:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.h
+; SVE2-NEXT:    uhadd z0.h, p0/m, z0.h, z1.h
+; SVE2-NEXT:    ret
+entry:
+  %s0s = zext <vscale x 8 x i16> %s0 to <vscale x 8 x i32>
+  %s1s = zext <vscale x 8 x i16> %s1 to <vscale x 8 x i32>
+  %m = add nuw nsw <vscale x 8 x i32> %s0s, %s1s
+  %s = lshr <vscale x 8 x i32> %m, shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 1, i32 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 8 x i32> %s to <vscale x 8 x i16>
+  ret <vscale x 8 x i16> %s2
+}
+
+define <vscale x 4 x i8> @hadds_v4i8(<vscale x 4 x i8> %s0, <vscale x 4 x i8> %s1) {
+; SVE-LABEL: hadds_v4i8:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    ptrue p0.s
+; SVE-NEXT:    sxtb z0.s, p0/m, z0.s
+; SVE-NEXT:    sxtb z1.s, p0/m, z1.s
+; SVE-NEXT:    add z0.s, z0.s, z1.s
+; SVE-NEXT:    asr z0.s, z0.s, #1
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: hadds_v4i8:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.s
+; SVE2-NEXT:    sxtb z0.s, p0/m, z0.s
+; SVE2-NEXT:    sxtb z1.s, p0/m, z1.s
+; SVE2-NEXT:    shadd z0.s, p0/m, z0.s, z1.s
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 4 x i8> %s0 to <vscale x 4 x i16>
+  %s1s = sext <vscale x 4 x i8> %s1 to <vscale x 4 x i16>
+  %m = add nsw <vscale x 4 x i16> %s0s, %s1s
+  %s = ashr <vscale x 4 x i16> %m, shufflevector (<vscale x 4 x i16> insertelement (<vscale x 4 x i16> poison, i16 1, i32 0), <vscale x 4 x i16> poison, <vscale x 4 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 4 x i16> %s to <vscale x 4 x i8>
+  ret <vscale x 4 x i8> %s2
+}
+
+define <vscale x 4 x i8> @hadds_v4i8_lsh(<vscale x 4 x i8> %s0, <vscale x 4 x i8> %s1) {
+; CHECK-LABEL: hadds_v4i8_lsh:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    sxtb z0.s, p0/m, z0.s
+; CHECK-NEXT:    sxtb z1.s, p0/m, z1.s
+; CHECK-NEXT:    add z0.s, z0.s, z1.s
+; CHECK-NEXT:    and z0.s, z0.s, #0xffff
+; CHECK-NEXT:    lsr z0.s, z0.s, #1
+; CHECK-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 4 x i8> %s0 to <vscale x 4 x i16>
+  %s1s = sext <vscale x 4 x i8> %s1 to <vscale x 4 x i16>
+  %m = add nsw <vscale x 4 x i16> %s0s, %s1s
+  %s = lshr <vscale x 4 x i16> %m, shufflevector (<vscale x 4 x i16> insertelement (<vscale x 4 x i16> poison, i16 1, i32 0), <vscale x 4 x i16> poison, <vscale x 4 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 4 x i16> %s to <vscale x 4 x i8>
+  ret <vscale x 4 x i8> %s2
+}
+
+define <vscale x 4 x i8> @haddu_v4i8(<vscale x 4 x i8> %s0, <vscale x 4 x i8> %s1) {
+; SVE-LABEL: haddu_v4i8:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    and z0.s, z0.s, #0xff
+; SVE-NEXT:    and z1.s, z1.s, #0xff
+; SVE-NEXT:    add z0.s, z0.s, z1.s
+; SVE-NEXT:    lsr z0.s, z0.s, #1
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: haddu_v4i8:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.s
+; SVE2-NEXT:    and z0.s, z0.s, #0xff
+; SVE2-NEXT:    and z1.s, z1.s, #0xff
+; SVE2-NEXT:    uhadd z0.s, p0/m, z0.s, z1.s
+; SVE2-NEXT:    ret
+entry:
+  %s0s = zext <vscale x 4 x i8> %s0 to <vscale x 4 x i16>
+  %s1s = zext <vscale x 4 x i8> %s1 to <vscale x 4 x i16>
+  %m = add nuw nsw <vscale x 4 x i16> %s0s, %s1s
+  %s = lshr <vscale x 4 x i16> %m, shufflevector (<vscale x 4 x i16> insertelement (<vscale x 4 x i16> poison, i16 1, i32 0), <vscale x 4 x i16> poison, <vscale x 4 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 4 x i16> %s to <vscale x 4 x i8>
+  ret <vscale x 4 x i8> %s2
+}
+
+define <vscale x 8 x i8> @hadds_v8i8(<vscale x 8 x i8> %s0, <vscale x 8 x i8> %s1) {
+; SVE-LABEL: hadds_v8i8:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    ptrue p0.h
+; SVE-NEXT:    sxtb z0.h, p0/m, z0.h
+; SVE-NEXT:    sxtb z1.h, p0/m, z1.h
+; SVE-NEXT:    add z0.h, z0.h, z1.h
+; SVE-NEXT:    asr z0.h, z0.h, #1
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: hadds_v8i8:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.h
+; SVE2-NEXT:    sxtb z0.h, p0/m, z0.h
+; SVE2-NEXT:    sxtb z1.h, p0/m, z1.h
+; SVE2-NEXT:    shadd z0.h, p0/m, z0.h, z1.h
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 8 x i8> %s0 to <vscale x 8 x i16>
+  %s1s = sext <vscale x 8 x i8> %s1 to <vscale x 8 x i16>
+  %m = add nsw <vscale x 8 x i16> %s0s, %s1s
+  %s = ashr <vscale x 8 x i16> %m, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 1, i32 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 8 x i16> %s to <vscale x 8 x i8>
+  ret <vscale x 8 x i8> %s2
+}
+
+define <vscale x 8 x i8> @hadds_v8i8_lsh(<vscale x 8 x i8> %s0, <vscale x 8 x i8> %s1) {
+; CHECK-LABEL: hadds_v8i8_lsh:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    sxtb z0.h, p0/m, z0.h
+; CHECK-NEXT:    sxtb z1.h, p0/m, z1.h
+; CHECK-NEXT:    add z0.h, z0.h, z1.h
+; CHECK-NEXT:    lsr z0.h, z0.h, #1
+; CHECK-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 8 x i8> %s0 to <vscale x 8 x i16>
+  %s1s = sext <vscale x 8 x i8> %s1 to <vscale x 8 x i16>
+  %m = add nsw <vscale x 8 x i16> %s0s, %s1s
+  %s = lshr <vscale x 8 x i16> %m, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 1, i32 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 8 x i16> %s to <vscale x 8 x i8>
+  ret <vscale x 8 x i8> %s2
+}
+
+define <vscale x 8 x i8> @haddu_v8i8(<vscale x 8 x i8> %s0, <vscale x 8 x i8> %s1) {
+; SVE-LABEL: haddu_v8i8:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    and z0.h, z0.h, #0xff
+; SVE-NEXT:    and z1.h, z1.h, #0xff
+; SVE-NEXT:    add z0.h, z0.h, z1.h
+; SVE-NEXT:    lsr z0.h, z0.h, #1
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: haddu_v8i8:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.h
+; SVE2-NEXT:    and z0.h, z0.h, #0xff
+; SVE2-NEXT:    and z1.h, z1.h, #0xff
+; SVE2-NEXT:    uhadd z0.h, p0/m, z0.h, z1.h
+; SVE2-NEXT:    ret
+entry:
+  %s0s = zext <vscale x 8 x i8> %s0 to <vscale x 8 x i16>
+  %s1s = zext <vscale x 8 x i8> %s1 to <vscale x 8 x i16>
+  %m = add nuw nsw <vscale x 8 x i16> %s0s, %s1s
+  %s = lshr <vscale x 8 x i16> %m, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 1, i32 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 8 x i16> %s to <vscale x 8 x i8>
+  ret <vscale x 8 x i8> %s2
+}
+
+define <vscale x 16 x i8> @hadds_v16i8(<vscale x 16 x i8> %s0, <vscale x 16 x i8> %s1) {
+; SVE-LABEL: hadds_v16i8:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    asr z2.b, z1.b, #1
+; SVE-NEXT:    asr z3.b, z0.b, #1
+; SVE-NEXT:    and z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.b, z3.b, z2.b
+; SVE-NEXT:    and z0.b, z0.b, #0x1
+; SVE-NEXT:    add z0.b, z1.b, z0.b
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: hadds_v16i8:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.b
+; SVE2-NEXT:    shadd z0.b, p0/m, z0.b, z1.b
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 16 x i8> %s0 to <vscale x 16 x i16>
+  %s1s = sext <vscale x 16 x i8> %s1 to <vscale x 16 x i16>
+  %m = add nsw <vscale x 16 x i16> %s0s, %s1s
+  %s = ashr <vscale x 16 x i16> %m, shufflevector (<vscale x 16 x i16> insertelement (<vscale x 16 x i16> poison, i16 1, i32 0), <vscale x 16 x i16> poison, <vscale x 16 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 16 x i16> %s to <vscale x 16 x i8>
+  ret <vscale x 16 x i8> %s2
+}
+
+define <vscale x 16 x i8> @hadds_v16i8_lsh(<vscale x 16 x i8> %s0, <vscale x 16 x i8> %s1) {
+; SVE-LABEL: hadds_v16i8_lsh:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    asr z2.b, z1.b, #1
+; SVE-NEXT:    asr z3.b, z0.b, #1
+; SVE-NEXT:    and z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.b, z3.b, z2.b
+; SVE-NEXT:    and z0.b, z0.b, #0x1
+; SVE-NEXT:    add z0.b, z1.b, z0.b
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: hadds_v16i8_lsh:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.b
+; SVE2-NEXT:    shadd z0.b, p0/m, z0.b, z1.b
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 16 x i8> %s0 to <vscale x 16 x i16>
+  %s1s = sext <vscale x 16 x i8> %s1 to <vscale x 16 x i16>
+  %m = add nsw <vscale x 16 x i16> %s0s, %s1s
+  %s = lshr <vscale x 16 x i16> %m, shufflevector (<vscale x 16 x i16> insertelement (<vscale x 16 x i16> poison, i16 1, i32 0), <vscale x 16 x i16> poison, <vscale x 16 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 16 x i16> %s to <vscale x 16 x i8>
+  ret <vscale x 16 x i8> %s2
+}
+
+define <vscale x 16 x i8> @haddu_v16i8(<vscale x 16 x i8> %s0, <vscale x 16 x i8> %s1) {
+; SVE-LABEL: haddu_v16i8:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    lsr z2.b, z1.b, #1
+; SVE-NEXT:    lsr z3.b, z0.b, #1
+; SVE-NEXT:    and z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.b, z3.b, z2.b
+; SVE-NEXT:    and z0.b, z0.b, #0x1
+; SVE-NEXT:    add z0.b, z1.b, z0.b
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: haddu_v16i8:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.b
+; SVE2-NEXT:    uhadd z0.b, p0/m, z0.b, z1.b
+; SVE2-NEXT:    ret
+entry:
+  %s0s = zext <vscale x 16 x i8> %s0 to <vscale x 16 x i16>
+  %s1s = zext <vscale x 16 x i8> %s1 to <vscale x 16 x i16>
+  %m = add nuw nsw <vscale x 16 x i16> %s0s, %s1s
+  %s = lshr <vscale x 16 x i16> %m, shufflevector (<vscale x 16 x i16> insertelement (<vscale x 16 x i16> poison, i16 1, i32 0), <vscale x 16 x i16> poison, <vscale x 16 x i32> zeroinitializer)
+  %s2 = trunc <vscale x 16 x i16> %s to <vscale x 16 x i8>
+  ret <vscale x 16 x i8> %s2
+}
+
+define <vscale x 2 x i64> @rhadds_v2i64(<vscale x 2 x i64> %s0, <vscale x 2 x i64> %s1) {
+; SVE-LABEL: rhadds_v2i64:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    asr z2.d, z1.d, #1
+; SVE-NEXT:    asr z3.d, z0.d, #1
+; SVE-NEXT:    orr z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.d, z3.d, z2.d
+; SVE-NEXT:    and z0.d, z0.d, #0x1
+; SVE-NEXT:    add z0.d, z1.d, z0.d
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: rhadds_v2i64:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.d
+; SVE2-NEXT:    srhadd z0.d, p0/m, z0.d, z1.d
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 2 x i64> %s0 to <vscale x 2 x i128>
+  %s1s = sext <vscale x 2 x i64> %s1 to <vscale x 2 x i128>
+  %add = add <vscale x 2 x i128> %s0s, shufflevector (<vscale x 2 x i128> insertelement (<vscale x 2 x i128> poison, i128 1, i32 0), <vscale x 2 x i128> poison, <vscale x 2 x i32> zeroinitializer)
+  %add2 = add <vscale x 2 x i128> %add, %s1s
+  %s = ashr <vscale x 2 x i128> %add2, shufflevector (<vscale x 2 x i128> insertelement (<vscale x 2 x i128> poison, i128 1, i32 0), <vscale x 2 x i128> poison, <vscale x 2 x i32> zeroinitializer)
+  %result = trunc <vscale x 2 x i128> %s to <vscale x 2 x i64>
+  ret <vscale x 2 x i64> %result
+}
+
+define <vscale x 2 x i64> @rhadds_v2i64_lsh(<vscale x 2 x i64> %s0, <vscale x 2 x i64> %s1) {
+; SVE-LABEL: rhadds_v2i64_lsh:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    asr z2.d, z1.d, #1
+; SVE-NEXT:    asr z3.d, z0.d, #1
+; SVE-NEXT:    orr z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.d, z3.d, z2.d
+; SVE-NEXT:    and z0.d, z0.d, #0x1
+; SVE-NEXT:    add z0.d, z1.d, z0.d
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: rhadds_v2i64_lsh:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.d
+; SVE2-NEXT:    srhadd z0.d, p0/m, z0.d, z1.d
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 2 x i64> %s0 to <vscale x 2 x i128>
+  %s1s = sext <vscale x 2 x i64> %s1 to <vscale x 2 x i128>
+  %add = add <vscale x 2 x i128> %s0s, shufflevector (<vscale x 2 x i128> insertelement (<vscale x 2 x i128> poison, i128 1, i32 0), <vscale x 2 x i128> poison, <vscale x 2 x i32> zeroinitializer)
+  %add2 = add <vscale x 2 x i128> %add, %s1s
+  %s = lshr <vscale x 2 x i128> %add2, shufflevector (<vscale x 2 x i128> insertelement (<vscale x 2 x i128> poison, i128 1, i32 0), <vscale x 2 x i128> poison, <vscale x 2 x i32> zeroinitializer)
+  %result = trunc <vscale x 2 x i128> %s to <vscale x 2 x i64>
+  ret <vscale x 2 x i64> %result
+}
+
+define <vscale x 2 x i64> @rhaddu_v2i64(<vscale x 2 x i64> %s0, <vscale x 2 x i64> %s1) {
+; SVE-LABEL: rhaddu_v2i64:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    lsr z2.d, z1.d, #1
+; SVE-NEXT:    lsr z3.d, z0.d, #1
+; SVE-NEXT:    orr z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.d, z3.d, z2.d
+; SVE-NEXT:    and z0.d, z0.d, #0x1
+; SVE-NEXT:    add z0.d, z1.d, z0.d
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: rhaddu_v2i64:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.d
+; SVE2-NEXT:    urhadd z0.d, p0/m, z0.d, z1.d
+; SVE2-NEXT:    ret
+entry:
+  %s0s = zext <vscale x 2 x i64> %s0 to <vscale x 2 x i128>
+  %s1s = zext <vscale x 2 x i64> %s1 to <vscale x 2 x i128>
+  %add = add nuw nsw <vscale x 2 x i128> %s0s, shufflevector (<vscale x 2 x i128> insertelement (<vscale x 2 x i128> poison, i128 1, i32 0), <vscale x 2 x i128> poison, <vscale x 2 x i32> zeroinitializer)
+  %add2 = add nuw nsw <vscale x 2 x i128> %add, %s1s
+  %s = lshr <vscale x 2 x i128> %add2, shufflevector (<vscale x 2 x i128> insertelement (<vscale x 2 x i128> poison, i128 1, i32 0), <vscale x 2 x i128> poison, <vscale x 2 x i32> zeroinitializer)
+  %result = trunc <vscale x 2 x i128> %s to <vscale x 2 x i64>
+  ret <vscale x 2 x i64> %result
+}
+
+define <vscale x 2 x i32> @rhadds_v2i32(<vscale x 2 x i32> %s0, <vscale x 2 x i32> %s1) {
+; CHECK-LABEL: rhadds_v2i32:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    mov z2.d, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    sxtw z0.d, p0/m, z0.d
+; CHECK-NEXT:    sxtw z1.d, p0/m, z1.d
+; CHECK-NEXT:    eor z0.d, z0.d, z2.d
+; CHECK-NEXT:    sub z0.d, z1.d, z0.d
+; CHECK-NEXT:    asr z0.d, z0.d, #1
+; CHECK-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 2 x i32> %s0 to <vscale x 2 x i64>
+  %s1s = sext <vscale x 2 x i32> %s1 to <vscale x 2 x i64>
+  %add = add <vscale x 2 x i64> %s0s, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 1, i32 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+  %add2 = add <vscale x 2 x i64> %add, %s1s
+  %s = ashr <vscale x 2 x i64> %add2, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 1, i32 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+  %result = trunc <vscale x 2 x i64> %s to <vscale x 2 x i32>
+  ret <vscale x 2 x i32> %result
+}
+
+define <vscale x 2 x i32> @rhadds_v2i32_lsh(<vscale x 2 x i32> %s0, <vscale x 2 x i32> %s1) {
+; CHECK-LABEL: rhadds_v2i32_lsh:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    mov z2.d, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    sxtw z0.d, p0/m, z0.d
+; CHECK-NEXT:    sxtw z1.d, p0/m, z1.d
+; CHECK-NEXT:    eor z0.d, z0.d, z2.d
+; CHECK-NEXT:    sub z0.d, z1.d, z0.d
+; CHECK-NEXT:    lsr z0.d, z0.d, #1
+; CHECK-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 2 x i32> %s0 to <vscale x 2 x i64>
+  %s1s = sext <vscale x 2 x i32> %s1 to <vscale x 2 x i64>
+  %add = add <vscale x 2 x i64> %s0s, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 1, i32 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+  %add2 = add <vscale x 2 x i64> %add, %s1s
+  %s = lshr <vscale x 2 x i64> %add2, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 1, i32 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+  %result = trunc <vscale x 2 x i64> %s to <vscale x 2 x i32>
+  ret <vscale x 2 x i32> %result
+}
+
+define <vscale x 2 x i32> @rhaddu_v2i32(<vscale x 2 x i32> %s0, <vscale x 2 x i32> %s1) {
+; SVE-LABEL: rhaddu_v2i32:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    mov z2.d, #-1 // =0xffffffffffffffff
+; SVE-NEXT:    and z0.d, z0.d, #0xffffffff
+; SVE-NEXT:    and z1.d, z1.d, #0xffffffff
+; SVE-NEXT:    eor z0.d, z0.d, z2.d
+; SVE-NEXT:    sub z0.d, z1.d, z0.d
+; SVE-NEXT:    lsr z0.d, z0.d, #1
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: rhaddu_v2i32:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.d
+; SVE2-NEXT:    and z0.d, z0.d, #0xffffffff
+; SVE2-NEXT:    and z1.d, z1.d, #0xffffffff
+; SVE2-NEXT:    urhadd z0.d, p0/m, z0.d, z1.d
+; SVE2-NEXT:    ret
+entry:
+  %s0s = zext <vscale x 2 x i32> %s0 to <vscale x 2 x i64>
+  %s1s = zext <vscale x 2 x i32> %s1 to <vscale x 2 x i64>
+  %add = add nuw nsw <vscale x 2 x i64> %s0s, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 1, i32 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+  %add2 = add nuw nsw <vscale x 2 x i64> %add, %s1s
+  %s = lshr <vscale x 2 x i64> %add2, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 1, i32 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+  %result = trunc <vscale x 2 x i64> %s to <vscale x 2 x i32>
+  ret <vscale x 2 x i32> %result
+}
+
+define <vscale x 4 x i32> @rhadds_v4i32(<vscale x 4 x i32> %s0, <vscale x 4 x i32> %s1) {
+; SVE-LABEL: rhadds_v4i32:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    asr z2.s, z1.s, #1
+; SVE-NEXT:    asr z3.s, z0.s, #1
+; SVE-NEXT:    orr z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.s, z3.s, z2.s
+; SVE-NEXT:    and z0.s, z0.s, #0x1
+; SVE-NEXT:    add z0.s, z1.s, z0.s
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: rhadds_v4i32:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.s
+; SVE2-NEXT:    srhadd z0.s, p0/m, z0.s, z1.s
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 4 x i32> %s0 to <vscale x 4 x i64>
+  %s1s = sext <vscale x 4 x i32> %s1 to <vscale x 4 x i64>
+  %add = add <vscale x 4 x i64> %s0s, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 1, i32 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer)
+  %add2 = add <vscale x 4 x i64> %add, %s1s
+  %s = ashr <vscale x 4 x i64> %add2, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 1, i32 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer)
+  %result = trunc <vscale x 4 x i64> %s to <vscale x 4 x i32>
+  ret <vscale x 4 x i32> %result
+}
+
+define <vscale x 4 x i32> @rhadds_v4i32_lsh(<vscale x 4 x i32> %s0, <vscale x 4 x i32> %s1) {
+; SVE-LABEL: rhadds_v4i32_lsh:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    asr z2.s, z1.s, #1
+; SVE-NEXT:    asr z3.s, z0.s, #1
+; SVE-NEXT:    orr z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.s, z3.s, z2.s
+; SVE-NEXT:    and z0.s, z0.s, #0x1
+; SVE-NEXT:    add z0.s, z1.s, z0.s
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: rhadds_v4i32_lsh:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.s
+; SVE2-NEXT:    srhadd z0.s, p0/m, z0.s, z1.s
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 4 x i32> %s0 to <vscale x 4 x i64>
+  %s1s = sext <vscale x 4 x i32> %s1 to <vscale x 4 x i64>
+  %add = add <vscale x 4 x i64> %s0s, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 1, i32 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer)
+  %add2 = add <vscale x 4 x i64> %add, %s1s
+  %s = lshr <vscale x 4 x i64> %add2, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 1, i32 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer)
+  %result = trunc <vscale x 4 x i64> %s to <vscale x 4 x i32>
+  ret <vscale x 4 x i32> %result
+}
+
+define <vscale x 4 x i32> @rhaddu_v4i32(<vscale x 4 x i32> %s0, <vscale x 4 x i32> %s1) {
+; SVE-LABEL: rhaddu_v4i32:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    lsr z2.s, z1.s, #1
+; SVE-NEXT:    lsr z3.s, z0.s, #1
+; SVE-NEXT:    orr z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.s, z3.s, z2.s
+; SVE-NEXT:    and z0.s, z0.s, #0x1
+; SVE-NEXT:    add z0.s, z1.s, z0.s
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: rhaddu_v4i32:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.s
+; SVE2-NEXT:    urhadd z0.s, p0/m, z0.s, z1.s
+; SVE2-NEXT:    ret
+entry:
+  %s0s = zext <vscale x 4 x i32> %s0 to <vscale x 4 x i64>
+  %s1s = zext <vscale x 4 x i32> %s1 to <vscale x 4 x i64>
+  %add = add nuw nsw <vscale x 4 x i64> %s0s, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 1, i32 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer)
+  %add2 = add nuw nsw <vscale x 4 x i64> %add, %s1s
+  %s = lshr <vscale x 4 x i64> %add2, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 1, i32 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer)
+  %result = trunc <vscale x 4 x i64> %s to <vscale x 4 x i32>
+  ret <vscale x 4 x i32> %result
+}
+
+define <vscale x 2 x i16> @rhadds_v2i16(<vscale x 2 x i16> %s0, <vscale x 2 x i16> %s1) {
+; CHECK-LABEL: rhadds_v2i16:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    mov z2.d, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    sxth z0.d, p0/m, z0.d
+; CHECK-NEXT:    sxth z1.d, p0/m, z1.d
+; CHECK-NEXT:    eor z0.d, z0.d, z2.d
+; CHECK-NEXT:    sub z0.d, z1.d, z0.d
+; CHECK-NEXT:    asr z0.d, z0.d, #1
+; CHECK-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 2 x i16> %s0 to <vscale x 2 x i32>
+  %s1s = sext <vscale x 2 x i16> %s1 to <vscale x 2 x i32>
+  %add = add <vscale x 2 x i32> %s0s, shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> poison, i32 1, i32 0), <vscale x 2 x i32> poison, <vscale x 2 x i32> zeroinitializer)
+  %add2 = add <vscale x 2 x i32> %add, %s1s
+  %s = ashr <vscale x 2 x i32> %add2, shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> poison, i32 1, i32 0), <vscale x 2 x i32> poison, <vscale x 2 x i32> zeroinitializer)
+  %result = trunc <vscale x 2 x i32> %s to <vscale x 2 x i16>
+  ret <vscale x 2 x i16> %result
+}
+
+define <vscale x 2 x i16> @rhadds_v2i16_lsh(<vscale x 2 x i16> %s0, <vscale x 2 x i16> %s1) {
+; CHECK-LABEL: rhadds_v2i16_lsh:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    mov z2.d, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    sxth z0.d, p0/m, z0.d
+; CHECK-NEXT:    sxth z1.d, p0/m, z1.d
+; CHECK-NEXT:    eor z0.d, z0.d, z2.d
+; CHECK-NEXT:    sub z0.d, z1.d, z0.d
+; CHECK-NEXT:    and z0.d, z0.d, #0xffffffff
+; CHECK-NEXT:    lsr z0.d, z0.d, #1
+; CHECK-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 2 x i16> %s0 to <vscale x 2 x i32>
+  %s1s = sext <vscale x 2 x i16> %s1 to <vscale x 2 x i32>
+  %add = add <vscale x 2 x i32> %s0s, shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> poison, i32 1, i32 0), <vscale x 2 x i32> poison, <vscale x 2 x i32> zeroinitializer)
+  %add2 = add <vscale x 2 x i32> %add, %s1s
+  %s = lshr <vscale x 2 x i32> %add2, shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> poison, i32 1, i32 0), <vscale x 2 x i32> poison, <vscale x 2 x i32> zeroinitializer)
+  %result = trunc <vscale x 2 x i32> %s to <vscale x 2 x i16>
+  ret <vscale x 2 x i16> %result
+}
+
+define <vscale x 2 x i16> @rhaddu_v2i16(<vscale x 2 x i16> %s0, <vscale x 2 x i16> %s1) {
+; CHECK-LABEL: rhaddu_v2i16:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    mov z2.d, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    and z0.d, z0.d, #0xffff
+; CHECK-NEXT:    and z1.d, z1.d, #0xffff
+; CHECK-NEXT:    eor z0.d, z0.d, z2.d
+; CHECK-NEXT:    sub z0.d, z1.d, z0.d
+; CHECK-NEXT:    lsr z0.d, z0.d, #1
+; CHECK-NEXT:    ret
+entry:
+  %s0s = zext <vscale x 2 x i16> %s0 to <vscale x 2 x i32>
+  %s1s = zext <vscale x 2 x i16> %s1 to <vscale x 2 x i32>
+  %add = add nuw nsw <vscale x 2 x i32> %s0s, shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> poison, i32 1, i32 0), <vscale x 2 x i32> poison, <vscale x 2 x i32> zeroinitializer)
+  %add2 = add nuw nsw <vscale x 2 x i32> %add, %s1s
+  %s = lshr <vscale x 2 x i32> %add2, shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> poison, i32 1, i32 0), <vscale x 2 x i32> poison, <vscale x 2 x i32> zeroinitializer)
+  %result = trunc <vscale x 2 x i32> %s to <vscale x 2 x i16>
+  ret <vscale x 2 x i16> %result
+}
+
+define <vscale x 4 x i16> @rhadds_v4i16(<vscale x 4 x i16> %s0, <vscale x 4 x i16> %s1) {
+; CHECK-LABEL: rhadds_v4i16:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    mov z2.s, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    sxth z0.s, p0/m, z0.s
+; CHECK-NEXT:    sxth z1.s, p0/m, z1.s
+; CHECK-NEXT:    eor z0.d, z0.d, z2.d
+; CHECK-NEXT:    sub z0.s, z1.s, z0.s
+; CHECK-NEXT:    asr z0.s, z0.s, #1
+; CHECK-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 4 x i16> %s0 to <vscale x 4 x i32>
+  %s1s = sext <vscale x 4 x i16> %s1 to <vscale x 4 x i32>
+  %add = add <vscale x 4 x i32> %s0s, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 1, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
+  %add2 = add <vscale x 4 x i32> %add, %s1s
+  %s = ashr <vscale x 4 x i32> %add2, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 1, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
+  %result = trunc <vscale x 4 x i32> %s to <vscale x 4 x i16>
+  ret <vscale x 4 x i16> %result
+}
+
+define <vscale x 4 x i16> @rhadds_v4i16_lsh(<vscale x 4 x i16> %s0, <vscale x 4 x i16> %s1) {
+; CHECK-LABEL: rhadds_v4i16_lsh:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    mov z2.s, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    sxth z0.s, p0/m, z0.s
+; CHECK-NEXT:    sxth z1.s, p0/m, z1.s
+; CHECK-NEXT:    eor z0.d, z0.d, z2.d
+; CHECK-NEXT:    sub z0.s, z1.s, z0.s
+; CHECK-NEXT:    lsr z0.s, z0.s, #1
+; CHECK-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 4 x i16> %s0 to <vscale x 4 x i32>
+  %s1s = sext <vscale x 4 x i16> %s1 to <vscale x 4 x i32>
+  %add = add <vscale x 4 x i32> %s0s, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 1, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
+  %add2 = add <vscale x 4 x i32> %add, %s1s
+  %s = lshr <vscale x 4 x i32> %add2, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 1, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
+  %result = trunc <vscale x 4 x i32> %s to <vscale x 4 x i16>
+  ret <vscale x 4 x i16> %result
+}
+
+define <vscale x 4 x i16> @rhaddu_v4i16(<vscale x 4 x i16> %s0, <vscale x 4 x i16> %s1) {
+; SVE-LABEL: rhaddu_v4i16:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    mov z2.s, #-1 // =0xffffffffffffffff
+; SVE-NEXT:    and z0.s, z0.s, #0xffff
+; SVE-NEXT:    and z1.s, z1.s, #0xffff
+; SVE-NEXT:    eor z0.d, z0.d, z2.d
+; SVE-NEXT:    sub z0.s, z1.s, z0.s
+; SVE-NEXT:    lsr z0.s, z0.s, #1
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: rhaddu_v4i16:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.s
+; SVE2-NEXT:    and z0.s, z0.s, #0xffff
+; SVE2-NEXT:    and z1.s, z1.s, #0xffff
+; SVE2-NEXT:    urhadd z0.s, p0/m, z0.s, z1.s
+; SVE2-NEXT:    ret
+entry:
+  %s0s = zext <vscale x 4 x i16> %s0 to <vscale x 4 x i32>
+  %s1s = zext <vscale x 4 x i16> %s1 to <vscale x 4 x i32>
+  %add = add nuw nsw <vscale x 4 x i32> %s0s, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 1, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
+  %add2 = add nuw nsw <vscale x 4 x i32> %add, %s1s
+  %s = lshr <vscale x 4 x i32> %add2, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 1, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
+  %result = trunc <vscale x 4 x i32> %s to <vscale x 4 x i16>
+  ret <vscale x 4 x i16> %result
+}
+
+define <vscale x 8 x i16> @rhadds_v8i16(<vscale x 8 x i16> %s0, <vscale x 8 x i16> %s1) {
+; SVE-LABEL: rhadds_v8i16:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    asr z2.h, z1.h, #1
+; SVE-NEXT:    asr z3.h, z0.h, #1
+; SVE-NEXT:    orr z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.h, z3.h, z2.h
+; SVE-NEXT:    and z0.h, z0.h, #0x1
+; SVE-NEXT:    add z0.h, z1.h, z0.h
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: rhadds_v8i16:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.h
+; SVE2-NEXT:    srhadd z0.h, p0/m, z0.h, z1.h
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 8 x i16> %s0 to <vscale x 8 x i32>
+  %s1s = sext <vscale x 8 x i16> %s1 to <vscale x 8 x i32>
+  %add = add <vscale x 8 x i32> %s0s, shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 1, i32 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer)
+  %add2 = add <vscale x 8 x i32> %add, %s1s
+  %s = ashr <vscale x 8 x i32> %add2, shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 1, i32 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer)
+  %result = trunc <vscale x 8 x i32> %s to <vscale x 8 x i16>
+  ret <vscale x 8 x i16> %result
+}
+
+define <vscale x 8 x i16> @rhadds_v8i16_lsh(<vscale x 8 x i16> %s0, <vscale x 8 x i16> %s1) {
+; SVE-LABEL: rhadds_v8i16_lsh:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    asr z2.h, z1.h, #1
+; SVE-NEXT:    asr z3.h, z0.h, #1
+; SVE-NEXT:    orr z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.h, z3.h, z2.h
+; SVE-NEXT:    and z0.h, z0.h, #0x1
+; SVE-NEXT:    add z0.h, z1.h, z0.h
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: rhadds_v8i16_lsh:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.h
+; SVE2-NEXT:    srhadd z0.h, p0/m, z0.h, z1.h
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 8 x i16> %s0 to <vscale x 8 x i32>
+  %s1s = sext <vscale x 8 x i16> %s1 to <vscale x 8 x i32>
+  %add = add <vscale x 8 x i32> %s0s, shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 1, i32 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer)
+  %add2 = add <vscale x 8 x i32> %add, %s1s
+  %s = lshr <vscale x 8 x i32> %add2, shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 1, i32 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer)
+  %result = trunc <vscale x 8 x i32> %s to <vscale x 8 x i16>
+  ret <vscale x 8 x i16> %result
+}
+
+define <vscale x 8 x i16> @rhaddu_v8i16(<vscale x 8 x i16> %s0, <vscale x 8 x i16> %s1) {
+; SVE-LABEL: rhaddu_v8i16:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    lsr z2.h, z1.h, #1
+; SVE-NEXT:    lsr z3.h, z0.h, #1
+; SVE-NEXT:    orr z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.h, z3.h, z2.h
+; SVE-NEXT:    and z0.h, z0.h, #0x1
+; SVE-NEXT:    add z0.h, z1.h, z0.h
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: rhaddu_v8i16:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.h
+; SVE2-NEXT:    urhadd z0.h, p0/m, z0.h, z1.h
+; SVE2-NEXT:    ret
+entry:
+  %s0s = zext <vscale x 8 x i16> %s0 to <vscale x 8 x i32>
+  %s1s = zext <vscale x 8 x i16> %s1 to <vscale x 8 x i32>
+  %add = add nuw nsw <vscale x 8 x i32> %s0s, shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 1, i32 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer)
+  %add2 = add nuw nsw <vscale x 8 x i32> %add, %s1s
+  %s = lshr <vscale x 8 x i32> %add2, shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 1, i32 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer)
+  %result = trunc <vscale x 8 x i32> %s to <vscale x 8 x i16>
+  ret <vscale x 8 x i16> %result
+}
+
+define <vscale x 4 x i8> @rhadds_v4i8(<vscale x 4 x i8> %s0, <vscale x 4 x i8> %s1) {
+; CHECK-LABEL: rhadds_v4i8:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    mov z2.s, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    sxtb z0.s, p0/m, z0.s
+; CHECK-NEXT:    sxtb z1.s, p0/m, z1.s
+; CHECK-NEXT:    eor z0.d, z0.d, z2.d
+; CHECK-NEXT:    sub z0.s, z1.s, z0.s
+; CHECK-NEXT:    asr z0.s, z0.s, #1
+; CHECK-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 4 x i8> %s0 to <vscale x 4 x i16>
+  %s1s = sext <vscale x 4 x i8> %s1 to <vscale x 4 x i16>
+  %add = add <vscale x 4 x i16> %s0s, shufflevector (<vscale x 4 x i16> insertelement (<vscale x 4 x i16> poison, i16 1, i32 0), <vscale x 4 x i16> poison, <vscale x 4 x i32> zeroinitializer)
+  %add2 = add <vscale x 4 x i16> %add, %s1s
+  %s = ashr <vscale x 4 x i16> %add2, shufflevector (<vscale x 4 x i16> insertelement (<vscale x 4 x i16> poison, i16 1, i32 0), <vscale x 4 x i16> poison, <vscale x 4 x i32> zeroinitializer)
+  %result = trunc <vscale x 4 x i16> %s to <vscale x 4 x i8>
+  ret <vscale x 4 x i8> %result
+}
+
+define <vscale x 4 x i8> @rhadds_v4i8_lsh(<vscale x 4 x i8> %s0, <vscale x 4 x i8> %s1) {
+; CHECK-LABEL: rhadds_v4i8_lsh:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    mov z2.s, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    sxtb z0.s, p0/m, z0.s
+; CHECK-NEXT:    sxtb z1.s, p0/m, z1.s
+; CHECK-NEXT:    eor z0.d, z0.d, z2.d
+; CHECK-NEXT:    sub z0.s, z1.s, z0.s
+; CHECK-NEXT:    and z0.s, z0.s, #0xffff
+; CHECK-NEXT:    lsr z0.s, z0.s, #1
+; CHECK-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 4 x i8> %s0 to <vscale x 4 x i16>
+  %s1s = sext <vscale x 4 x i8> %s1 to <vscale x 4 x i16>
+  %add = add <vscale x 4 x i16> %s0s, shufflevector (<vscale x 4 x i16> insertelement (<vscale x 4 x i16> poison, i16 1, i32 0), <vscale x 4 x i16> poison, <vscale x 4 x i32> zeroinitializer)
+  %add2 = add <vscale x 4 x i16> %add, %s1s
+  %s = lshr <vscale x 4 x i16> %add2, shufflevector (<vscale x 4 x i16> insertelement (<vscale x 4 x i16> poison, i16 1, i32 0), <vscale x 4 x i16> poison, <vscale x 4 x i32> zeroinitializer)
+  %result = trunc <vscale x 4 x i16> %s to <vscale x 4 x i8>
+  ret <vscale x 4 x i8> %result
+}
+
+define <vscale x 4 x i8> @rhaddu_v4i8(<vscale x 4 x i8> %s0, <vscale x 4 x i8> %s1) {
+; CHECK-LABEL: rhaddu_v4i8:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    mov z2.s, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    and z0.s, z0.s, #0xff
+; CHECK-NEXT:    and z1.s, z1.s, #0xff
+; CHECK-NEXT:    eor z0.d, z0.d, z2.d
+; CHECK-NEXT:    sub z0.s, z1.s, z0.s
+; CHECK-NEXT:    lsr z0.s, z0.s, #1
+; CHECK-NEXT:    ret
+entry:
+  %s0s = zext <vscale x 4 x i8> %s0 to <vscale x 4 x i16>
+  %s1s = zext <vscale x 4 x i8> %s1 to <vscale x 4 x i16>
+  %add = add nuw nsw <vscale x 4 x i16> %s0s, shufflevector (<vscale x 4 x i16> insertelement (<vscale x 4 x i16> poison, i16 1, i32 0), <vscale x 4 x i16> poison, <vscale x 4 x i32> zeroinitializer)
+  %add2 = add nuw nsw <vscale x 4 x i16> %add, %s1s
+  %s = lshr <vscale x 4 x i16> %add2, shufflevector (<vscale x 4 x i16> insertelement (<vscale x 4 x i16> poison, i16 1, i32 0), <vscale x 4 x i16> poison, <vscale x 4 x i32> zeroinitializer)
+  %result = trunc <vscale x 4 x i16> %s to <vscale x 4 x i8>
+  ret <vscale x 4 x i8> %result
+}
+
+define <vscale x 8 x i8> @rhadds_v8i8(<vscale x 8 x i8> %s0, <vscale x 8 x i8> %s1) {
+; CHECK-LABEL: rhadds_v8i8:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    mov z2.h, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    sxtb z0.h, p0/m, z0.h
+; CHECK-NEXT:    sxtb z1.h, p0/m, z1.h
+; CHECK-NEXT:    eor z0.d, z0.d, z2.d
+; CHECK-NEXT:    sub z0.h, z1.h, z0.h
+; CHECK-NEXT:    asr z0.h, z0.h, #1
+; CHECK-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 8 x i8> %s0 to <vscale x 8 x i16>
+  %s1s = sext <vscale x 8 x i8> %s1 to <vscale x 8 x i16>
+  %add = add <vscale x 8 x i16> %s0s, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 1, i32 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
+  %add2 = add <vscale x 8 x i16> %add, %s1s
+  %s = ashr <vscale x 8 x i16> %add2, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 1, i32 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
+  %result = trunc <vscale x 8 x i16> %s to <vscale x 8 x i8>
+  ret <vscale x 8 x i8> %result
+}
+
+define <vscale x 8 x i8> @rhadds_v8i8_lsh(<vscale x 8 x i8> %s0, <vscale x 8 x i8> %s1) {
+; CHECK-LABEL: rhadds_v8i8_lsh:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    mov z2.h, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    sxtb z0.h, p0/m, z0.h
+; CHECK-NEXT:    sxtb z1.h, p0/m, z1.h
+; CHECK-NEXT:    eor z0.d, z0.d, z2.d
+; CHECK-NEXT:    sub z0.h, z1.h, z0.h
+; CHECK-NEXT:    lsr z0.h, z0.h, #1
+; CHECK-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 8 x i8> %s0 to <vscale x 8 x i16>
+  %s1s = sext <vscale x 8 x i8> %s1 to <vscale x 8 x i16>
+  %add = add <vscale x 8 x i16> %s0s, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 1, i32 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
+  %add2 = add <vscale x 8 x i16> %add, %s1s
+  %s = lshr <vscale x 8 x i16> %add2, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 1, i32 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
+  %result = trunc <vscale x 8 x i16> %s to <vscale x 8 x i8>
+  ret <vscale x 8 x i8> %result
+}
+
+define <vscale x 8 x i8> @rhaddu_v8i8(<vscale x 8 x i8> %s0, <vscale x 8 x i8> %s1) {
+; SVE-LABEL: rhaddu_v8i8:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    mov z2.h, #-1 // =0xffffffffffffffff
+; SVE-NEXT:    and z0.h, z0.h, #0xff
+; SVE-NEXT:    and z1.h, z1.h, #0xff
+; SVE-NEXT:    eor z0.d, z0.d, z2.d
+; SVE-NEXT:    sub z0.h, z1.h, z0.h
+; SVE-NEXT:    lsr z0.h, z0.h, #1
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: rhaddu_v8i8:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.h
+; SVE2-NEXT:    and z0.h, z0.h, #0xff
+; SVE2-NEXT:    and z1.h, z1.h, #0xff
+; SVE2-NEXT:    urhadd z0.h, p0/m, z0.h, z1.h
+; SVE2-NEXT:    ret
+entry:
+  %s0s = zext <vscale x 8 x i8> %s0 to <vscale x 8 x i16>
+  %s1s = zext <vscale x 8 x i8> %s1 to <vscale x 8 x i16>
+  %add = add nuw nsw <vscale x 8 x i16> %s0s, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 1, i32 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
+  %add2 = add nuw nsw <vscale x 8 x i16> %add, %s1s
+  %s = lshr <vscale x 8 x i16> %add2, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 1, i32 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
+  %result = trunc <vscale x 8 x i16> %s to <vscale x 8 x i8>
+  ret <vscale x 8 x i8> %result
+}
+
+define <vscale x 16 x i8> @rhadds_v16i8(<vscale x 16 x i8> %s0, <vscale x 16 x i8> %s1) {
+; SVE-LABEL: rhadds_v16i8:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    asr z2.b, z1.b, #1
+; SVE-NEXT:    asr z3.b, z0.b, #1
+; SVE-NEXT:    orr z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.b, z3.b, z2.b
+; SVE-NEXT:    and z0.b, z0.b, #0x1
+; SVE-NEXT:    add z0.b, z1.b, z0.b
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: rhadds_v16i8:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.b
+; SVE2-NEXT:    srhadd z0.b, p0/m, z0.b, z1.b
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 16 x i8> %s0 to <vscale x 16 x i16>
+  %s1s = sext <vscale x 16 x i8> %s1 to <vscale x 16 x i16>
+  %add = add <vscale x 16 x i16> %s0s, shufflevector (<vscale x 16 x i16> insertelement (<vscale x 16 x i16> poison, i16 1, i32 0), <vscale x 16 x i16> poison, <vscale x 16 x i32> zeroinitializer)
+  %add2 = add <vscale x 16 x i16> %add, %s1s
+  %s = ashr <vscale x 16 x i16> %add2, shufflevector (<vscale x 16 x i16> insertelement (<vscale x 16 x i16> poison, i16 1, i32 0), <vscale x 16 x i16> poison, <vscale x 16 x i32> zeroinitializer)
+  %result = trunc <vscale x 16 x i16> %s to <vscale x 16 x i8>
+  ret <vscale x 16 x i8> %result
+}
+
+define <vscale x 16 x i8> @rhadds_v16i8_lsh(<vscale x 16 x i8> %s0, <vscale x 16 x i8> %s1) {
+; SVE-LABEL: rhadds_v16i8_lsh:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    asr z2.b, z1.b, #1
+; SVE-NEXT:    asr z3.b, z0.b, #1
+; SVE-NEXT:    orr z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.b, z3.b, z2.b
+; SVE-NEXT:    and z0.b, z0.b, #0x1
+; SVE-NEXT:    add z0.b, z1.b, z0.b
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: rhadds_v16i8_lsh:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.b
+; SVE2-NEXT:    srhadd z0.b, p0/m, z0.b, z1.b
+; SVE2-NEXT:    ret
+entry:
+  %s0s = sext <vscale x 16 x i8> %s0 to <vscale x 16 x i16>
+  %s1s = sext <vscale x 16 x i8> %s1 to <vscale x 16 x i16>
+  %add = add <vscale x 16 x i16> %s0s, shufflevector (<vscale x 16 x i16> insertelement (<vscale x 16 x i16> poison, i16 1, i32 0), <vscale x 16 x i16> poison, <vscale x 16 x i32> zeroinitializer)
+  %add2 = add <vscale x 16 x i16> %add, %s1s
+  %s = lshr <vscale x 16 x i16> %add2, shufflevector (<vscale x 16 x i16> insertelement (<vscale x 16 x i16> poison, i16 1, i32 0), <vscale x 16 x i16> poison, <vscale x 16 x i32> zeroinitializer)
+  %result = trunc <vscale x 16 x i16> %s to <vscale x 16 x i8>
+  ret <vscale x 16 x i8> %result
+}
+
+define <vscale x 16 x i8> @rhaddu_v16i8(<vscale x 16 x i8> %s0, <vscale x 16 x i8> %s1) {
+; SVE-LABEL: rhaddu_v16i8:
+; SVE:       // %bb.0: // %entry
+; SVE-NEXT:    lsr z2.b, z1.b, #1
+; SVE-NEXT:    lsr z3.b, z0.b, #1
+; SVE-NEXT:    orr z0.d, z0.d, z1.d
+; SVE-NEXT:    add z1.b, z3.b, z2.b
+; SVE-NEXT:    and z0.b, z0.b, #0x1
+; SVE-NEXT:    add z0.b, z1.b, z0.b
+; SVE-NEXT:    ret
+;
+; SVE2-LABEL: rhaddu_v16i8:
+; SVE2:       // %bb.0: // %entry
+; SVE2-NEXT:    ptrue p0.b
+; SVE2-NEXT:    urhadd z0.b, p0/m, z0.b, z1.b
+; SVE2-NEXT:    ret
+entry:
+  %s0s = zext <vscale x 16 x i8> %s0 to <vscale x 16 x i16>
+  %s1s = zext <vscale x 16 x i8> %s1 to <vscale x 16 x i16>
+  %add = add nuw nsw <vscale x 16 x i16> %s0s, shufflevector (<vscale x 16 x i16> insertelement (<vscale x 16 x i16> poison, i16 1, i32 0), <vscale x 16 x i16> poison, <vscale x 16 x i32> zeroinitializer)
+  %add2 = add nuw nsw <vscale x 16 x i16> %add, %s1s
+  %s = lshr <vscale x 16 x i16> %add2, shufflevector (<vscale x 16 x i16> insertelement (<vscale x 16 x i16> poison, i16 1, i32 0), <vscale x 16 x i16> poison, <vscale x 16 x i32> zeroinitializer)
+  %result = trunc <vscale x 16 x i16> %s to <vscale x 16 x i8>
+  ret <vscale x 16 x i8> %result
+}
diff --git a/llvm/test/CodeGen/AArch64/sve-int-arith.ll b/llvm/test/CodeGen/AArch64/sve-int-arith.ll
index 2f9062d192cfe..c3be583cbb0ae 100644
--- a/llvm/test/CodeGen/AArch64/sve-int-arith.ll
+++ b/llvm/test/CodeGen/AArch64/sve-int-arith.ll
@@ -1,5 +1,5 @@
 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve < %s | FileCheck %s
+; RUN: llc -mtriple=aarch64 -mattr=+sve < %s | FileCheck %s
 
 define <vscale x 2 x i64> @add_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b) {
 ; CHECK-LABEL: add_i64:
@@ -337,18 +337,94 @@ define <vscale x 16 x i8> @uqsub_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b
   ret <vscale x 16 x i8> %res
 }
 
-define <vscale x 16 x i8> @mla_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b, <vscale x 16 x i8> %c) {
-; CHECK-LABEL: mla_i8:
+define <vscale x 16 x i8> @mad_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b, <vscale x 16 x i8> %c) {
+; CHECK-LABEL: mad_i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.b
-; CHECK-NEXT:    mla z2.b, p0/m, z0.b, z1.b
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    mad z0.b, p0/m, z1.b, z2.b
 ; CHECK-NEXT:    ret
   %prod = mul <vscale x 16 x i8> %a, %b
   %res = add <vscale x 16 x i8> %c, %prod
   ret <vscale x 16 x i8> %res
 }
 
+define <vscale x 8 x i16> @mad_i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b, <vscale x 8 x i16> %c) {
+; CHECK-LABEL: mad_i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    mad z0.h, p0/m, z1.h, z2.h
+; CHECK-NEXT:    ret
+  %prod = mul <vscale x 8 x i16> %a, %b
+  %res = add <vscale x 8 x i16> %c, %prod
+  ret <vscale x 8 x i16> %res
+}
+
+define <vscale x 4 x i32> @mad_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b, <vscale x 4 x i32> %c) {
+; CHECK-LABEL: mad_i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    mad z0.s, p0/m, z1.s, z2.s
+; CHECK-NEXT:    ret
+  %prod = mul <vscale x 4 x i32> %a, %b
+  %res = add <vscale x 4 x i32> %c, %prod
+  ret <vscale x 4 x i32> %res
+}
+
+define <vscale x 2 x i64> @mad_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b, <vscale x 2 x i64> %c) {
+; CHECK-LABEL: mad_i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    mad z0.d, p0/m, z1.d, z2.d
+; CHECK-NEXT:    ret
+  %prod = mul <vscale x 2 x i64> %a, %b
+  %res = add <vscale x 2 x i64> %c, %prod
+  ret <vscale x 2 x i64> %res
+}
+
+define <vscale x 16 x i8> @mla_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b, <vscale x 16 x i8> %c) {
+; CHECK-LABEL: mla_i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    mla z0.b, p0/m, z1.b, z2.b
+; CHECK-NEXT:    ret
+  %prod = mul <vscale x 16 x i8> %b, %c
+  %res = add <vscale x 16 x i8> %a, %prod
+  ret <vscale x 16 x i8> %res
+}
+
+define <vscale x 8 x i16> @mla_i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b, <vscale x 8 x i16> %c) {
+; CHECK-LABEL: mla_i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    mla z0.h, p0/m, z1.h, z2.h
+; CHECK-NEXT:    ret
+  %prod = mul <vscale x 8 x i16> %b, %c
+  %res = add <vscale x 8 x i16> %a, %prod
+  ret <vscale x 8 x i16> %res
+}
+
+define <vscale x 4 x i32> @mla_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b, <vscale x 4 x i32> %c) {
+; CHECK-LABEL: mla_i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    mla z0.s, p0/m, z1.s, z2.s
+; CHECK-NEXT:    ret
+  %prod = mul <vscale x 4 x i32> %b, %c
+  %res = add <vscale x 4 x i32> %a, %prod
+  ret <vscale x 4 x i32> %res
+}
+
+define <vscale x 2 x i64> @mla_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b, <vscale x 2 x i64> %c) {
+; CHECK-LABEL: mla_i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    mla z0.d, p0/m, z1.d, z2.d
+; CHECK-NEXT:    ret
+  %prod = mul <vscale x 2 x i64> %b, %c
+  %res = add <vscale x 2 x i64> %a, %prod
+  ret <vscale x 2 x i64> %res
+}
+
 define <vscale x 16 x i8> @mla_i8_multiuse(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b, <vscale x 16 x i8> %c, <vscale x 16 x i8>* %p) {
 ; CHECK-LABEL: mla_i8_multiuse:
 ; CHECK:       // %bb.0:
@@ -363,18 +439,311 @@ define <vscale x 16 x i8> @mla_i8_multiuse(<vscale x 16 x i8> %a, <vscale x 16 x
   ret <vscale x 16 x i8> %res
 }
 
-define <vscale x 16 x i8> @mls_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b, <vscale x 16 x i8> %c) {
-; CHECK-LABEL: mls_i8:
+define <vscale x 16 x i8> @msb_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b, <vscale x 16 x i8> %c) {
+; CHECK-LABEL: msb_i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.b
-; CHECK-NEXT:    mls z2.b, p0/m, z0.b, z1.b
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    msb z0.b, p0/m, z1.b, z2.b
 ; CHECK-NEXT:    ret
   %prod = mul <vscale x 16 x i8> %a, %b
   %res = sub <vscale x 16 x i8> %c, %prod
   ret <vscale x 16 x i8> %res
 }
 
+define <vscale x 8 x i16> @msb_i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b, <vscale x 8 x i16> %c) {
+; CHECK-LABEL: msb_i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    msb z0.h, p0/m, z1.h, z2.h
+; CHECK-NEXT:    ret
+  %prod = mul <vscale x 8 x i16> %a, %b
+  %res = sub <vscale x 8 x i16> %c, %prod
+  ret <vscale x 8 x i16> %res
+}
+
+define <vscale x 4 x i32> @msb_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b, <vscale x 4 x i32> %c) {
+; CHECK-LABEL: msb_i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    msb z0.s, p0/m, z1.s, z2.s
+; CHECK-NEXT:    ret
+  %prod = mul <vscale x 4 x i32> %a, %b
+  %res = sub <vscale x 4 x i32> %c, %prod
+  ret <vscale x 4 x i32> %res
+}
+
+define <vscale x 2 x i64> @msb_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b, <vscale x 2 x i64> %c) {
+; CHECK-LABEL: msb_i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    msb z0.d, p0/m, z1.d, z2.d
+; CHECK-NEXT:    ret
+  %prod = mul <vscale x 2 x i64> %a, %b
+  %res = sub <vscale x 2 x i64> %c, %prod
+  ret <vscale x 2 x i64> %res
+}
+
+define <vscale x 16 x i8> @mls_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b, <vscale x 16 x i8> %c) {
+; CHECK-LABEL: mls_i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    mls z0.b, p0/m, z1.b, z2.b
+; CHECK-NEXT:    ret
+  %prod = mul <vscale x 16 x i8> %b, %c
+  %res = sub <vscale x 16 x i8> %a, %prod
+  ret <vscale x 16 x i8> %res
+}
+
+define <vscale x 8 x i16> @mls_i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b, <vscale x 8 x i16> %c) {
+; CHECK-LABEL: mls_i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    mls z0.h, p0/m, z1.h, z2.h
+; CHECK-NEXT:    ret
+  %prod = mul <vscale x 8 x i16> %b, %c
+  %res = sub <vscale x 8 x i16> %a, %prod
+  ret <vscale x 8 x i16> %res
+}
+
+define <vscale x 4 x i32> @mls_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b, <vscale x 4 x i32> %c) {
+; CHECK-LABEL: mls_i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    mls z0.s, p0/m, z1.s, z2.s
+; CHECK-NEXT:    ret
+  %prod = mul <vscale x 4 x i32> %b, %c
+  %res = sub <vscale x 4 x i32> %a, %prod
+  ret <vscale x 4 x i32> %res
+}
+
+define <vscale x 2 x i64> @mls_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b, <vscale x 2 x i64> %c) {
+; CHECK-LABEL: mls_i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    mls z0.d, p0/m, z1.d, z2.d
+; CHECK-NEXT:    ret
+  %prod = mul <vscale x 2 x i64> %b, %c
+  %res = sub <vscale x 2 x i64> %a, %prod
+  ret <vscale x 2 x i64> %res
+}
+
+; Test cases below have one of the add/sub operands as constant splat
+
+ define <vscale x 2 x i64> @muladd_i64_positiveAddend(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b)
+; CHECK-LABEL: muladd_i64_positiveAddend:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    mov z2.d, #0xffffffff
+; CHECK-NEXT:    mad z0.d, p0/m, z1.d, z2.d
+; CHECK-NEXT:    ret
+{
+  %1 = mul <vscale x 2 x i64> %a, %b
+  %2 = add <vscale x 2 x i64> %1, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 4294967295, i64 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+  ret <vscale x 2 x i64> %2
+}
+
+define <vscale x 2 x i64> @muladd_i64_negativeAddend(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b)
+; CHECK-LABEL: muladd_i64_negativeAddend:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    mov z2.d, #0xffffffff00000001
+; CHECK-NEXT:    mad z0.d, p0/m, z1.d, z2.d
+; CHECK-NEXT:    ret
+{
+  %1 = mul <vscale x 2 x i64> %a, %b
+  %2 = add <vscale x 2 x i64> %1, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 -4294967295, i64 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+  ret <vscale x 2 x i64> %2
+}
+
+
+define <vscale x 4 x i32> @muladd_i32_positiveAddend(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b)
+; CHECK-LABEL: muladd_i32_positiveAddend:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    mov z2.s, #0x10000
+; CHECK-NEXT:    mad z0.s, p0/m, z1.s, z2.s
+; CHECK-NEXT:    ret
+{
+  %1 = mul <vscale x 4 x i32> %a, %b
+  %2 = add <vscale x 4 x i32> %1, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 65536, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
+  ret <vscale x 4 x i32> %2
+}
+
+define <vscale x 4 x i32> @muladd_i32_negativeAddend(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b)
+; CHECK-LABEL: muladd_i32_negativeAddend:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    mov z2.s, #0xffff0000
+; CHECK-NEXT:    mad z0.s, p0/m, z1.s, z2.s
+; CHECK-NEXT:    ret
+{
+  %1 = mul <vscale x 4 x i32> %a, %b
+  %2 = add <vscale x 4 x i32> %1, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 -65536, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
+  ret <vscale x 4 x i32> %2
+}
+
+define <vscale x 8 x i16> @muladd_i16_positiveAddend(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b)
+; CHECK-LABEL: muladd_i16_positiveAddend:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    mul z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    add z0.h, z0.h, #255 // =0xff
+; CHECK-NEXT:    ret
+{
+  %1 = mul <vscale x 8 x i16> %a, %b
+  %2 = add <vscale x 8 x i16> %1, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 255, i16 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
+  ret <vscale x 8 x i16> %2
+}
+
+define <vscale x 8 x i16> @muladd_i16_negativeAddend(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b)
+; CHECK-LABEL: muladd_i16_negativeAddend:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    mov z2.h, #-255 // =0xffffffffffffff01
+; CHECK-NEXT:    mad z0.h, p0/m, z1.h, z2.h
+; CHECK-NEXT:    ret
+{
+  %1 = mul <vscale x 8 x i16> %a, %b
+  %2 = add <vscale x 8 x i16> %1, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 -255, i16 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
+  ret <vscale x 8 x i16> %2
+}
+
+define <vscale x 16 x i8> @muladd_i8_positiveAddend(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b)
+; CHECK-LABEL: muladd_i8_positiveAddend:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    mul z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    add z0.b, z0.b, #15 // =0xf
+; CHECK-NEXT:    ret
+{
+  %1 = mul <vscale x 16 x i8> %a, %b
+  %2 = add <vscale x 16 x i8> %1, shufflevector (<vscale x 16 x i8> insertelement (<vscale x 16 x i8> poison, i8 15, i8 0), <vscale x 16 x i8> poison, <vscale x 16 x i32> zeroinitializer)
+  ret <vscale x 16 x i8> %2
+}
+
+define <vscale x 16 x i8> @muladd_i8_negativeAddend(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b)
+; CHECK-LABEL: muladd_i8_negativeAddend:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    mul z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    add z0.b, z0.b, #241 // =0xf1
+; CHECK-NEXT:    ret
+{
+  %1 = mul <vscale x 16 x i8> %a, %b
+  %2 = add <vscale x 16 x i8> %1, shufflevector (<vscale x 16 x i8> insertelement (<vscale x 16 x i8> poison, i8 -15, i8 0), <vscale x 16 x i8> poison, <vscale x 16 x i32> zeroinitializer)
+  ret <vscale x 16 x i8> %2
+}
+
+define <vscale x 2 x i64> @mulsub_i64_positiveAddend(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b)
+; CHECK-LABEL: mulsub_i64_positiveAddend:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    mul z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    mov z1.d, #0xffffffff
+; CHECK-NEXT:    sub z0.d, z0.d, z1.d
+; CHECK-NEXT:    ret
+{
+  %1 = mul <vscale x 2 x i64> %a, %b
+  %2 = sub <vscale x 2 x i64> %1, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 4294967295, i64 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+  ret <vscale x 2 x i64> %2
+}
+
+define <vscale x 2 x i64> @mulsub_i64_negativeAddend(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b)
+; CHECK-LABEL: mulsub_i64_negativeAddend:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    mul z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    mov z1.d, #0xffffffff00000001
+; CHECK-NEXT:    sub z0.d, z0.d, z1.d
+; CHECK-NEXT:    ret
+{
+  %1 = mul <vscale x 2 x i64> %a, %b
+  %2 = sub <vscale x 2 x i64> %1, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 -4294967295, i64 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+  ret <vscale x 2 x i64> %2
+}
+
+
+define <vscale x 4 x i32> @mulsub_i32_positiveAddend(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b)
+; CHECK-LABEL: mulsub_i32_positiveAddend:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    mul z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    mov z1.s, #0x10000
+; CHECK-NEXT:    sub z0.s, z0.s, z1.s
+; CHECK-NEXT:    ret
+{
+  %1 = mul <vscale x 4 x i32> %a, %b
+  %2 = sub <vscale x 4 x i32> %1, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 65536, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
+  ret <vscale x 4 x i32> %2
+}
+
+define <vscale x 4 x i32> @mulsub_i32_negativeAddend(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b)
+; CHECK-LABEL: mulsub_i32_negativeAddend:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    mul z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    mov z1.s, #0xffff0000
+; CHECK-NEXT:    sub z0.s, z0.s, z1.s
+; CHECK-NEXT:    ret
+{
+  %1 = mul <vscale x 4 x i32> %a, %b
+  %2 = sub <vscale x 4 x i32> %1, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 -65536, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
+  ret <vscale x 4 x i32> %2
+}
+
+define <vscale x 8 x i16> @mulsub_i16_positiveAddend(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b)
+; CHECK-LABEL: mulsub_i16_positiveAddend:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    mul z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    sub z0.h, z0.h, #255 // =0xff
+; CHECK-NEXT:    ret
+{
+  %1 = mul <vscale x 8 x i16> %a, %b
+  %2 = sub <vscale x 8 x i16> %1, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 255, i16 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
+  ret <vscale x 8 x i16> %2
+}
+
+define <vscale x 8 x i16> @mulsub_i16_negativeAddend(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b)
+; CHECK-LABEL: mulsub_i16_negativeAddend:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    mul z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    mov z1.h, #-255 // =0xffffffffffffff01
+; CHECK-NEXT:    sub z0.h, z0.h, z1.h
+; CHECK-NEXT:    ret
+{
+  %1 = mul <vscale x 8 x i16> %a, %b
+  %2 = sub <vscale x 8 x i16> %1, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 -255, i16 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
+  ret <vscale x 8 x i16> %2
+}
+
+define <vscale x 16 x i8> @mulsub_i8_positiveAddend(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b)
+; CHECK-LABEL: mulsub_i8_positiveAddend:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    mul z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    sub z0.b, z0.b, #15 // =0xf
+; CHECK-NEXT:    ret
+{
+  %1 = mul <vscale x 16 x i8> %a, %b
+  %2 = sub <vscale x 16 x i8> %1, shufflevector (<vscale x 16 x i8> insertelement (<vscale x 16 x i8> poison, i8 15, i8 0), <vscale x 16 x i8> poison, <vscale x 16 x i32> zeroinitializer)
+  ret <vscale x 16 x i8> %2
+}
+
+define <vscale x 16 x i8> @mulsub_i8_negativeAddend(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b)
+; CHECK-LABEL: mulsub_i8_negativeAddend:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    mul z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    sub z0.b, z0.b, #241 // =0xf1
+; CHECK-NEXT:    ret
+{
+  %1 = mul <vscale x 16 x i8> %a, %b
+  %2 = sub <vscale x 16 x i8> %1, shufflevector (<vscale x 16 x i8> insertelement (<vscale x 16 x i8> poison, i8 -15, i8 0), <vscale x 16 x i8> poison, <vscale x 16 x i32> zeroinitializer)
+  ret <vscale x 16 x i8> %2
+}
+
 declare <vscale x 16 x i8> @llvm.sadd.sat.nxv16i8(<vscale x 16 x i8>, <vscale x 16 x i8>)
 declare <vscale x 8 x i16> @llvm.sadd.sat.nxv8i16(<vscale x 8 x i16>, <vscale x 8 x i16>)
 declare <vscale x 4 x i32> @llvm.sadd.sat.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>)
diff --git a/llvm/test/CodeGen/AArch64/sve-intrinsics-int-compares.ll b/llvm/test/CodeGen/AArch64/sve-intrinsics-int-compares.ll
index be1eecee8507e..a995823fe446c 100644
--- a/llvm/test/CodeGen/AArch64/sve-intrinsics-int-compares.ll
+++ b/llvm/test/CodeGen/AArch64/sve-intrinsics-int-compares.ll
@@ -1037,10 +1037,11 @@ define <vscale x 2 x i1> @cmpne_ir_d(<vscale x 2 x i64> %a, <vscale x 2 x i64> %
 
 define <vscale x 1 x i1> @cmpne_ir_q(<vscale x 1 x i64> %a, <vscale x 1 x i64> %b) {
 ; CHECK-LABEL: cmpne_ir_q:
-; CHECK: ptrue p0.d
-; CHECK-NEXT: cmpne p0.d, p0/z, z0.d, z1.d
-; CHECK-NEXT: punpklo p0.h, p0.b
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, z1.d
+; CHECK-NEXT:    punpklo p0.h, p0.b
+; CHECK-NEXT:    ret
   %out = icmp ne <vscale x 1 x i64> %a, %b
   ret <vscale x 1 x i1> %out
 }
@@ -1216,6 +1217,66 @@ define <vscale x 8 x i1> @predicated_icmp_ult_imm(<vscale x 8 x i1> %a, <vscale
   ret <vscale x 8 x i1> %and
 }
 
+%svboolx2 = type { <vscale x 4 x i1>, <vscale x 4 x i1> }
+
+define %svboolx2 @and_of_multiuse_icmp_sle(<vscale x 4 x i1> %a, <vscale x 4 x i32> %b, <vscale x 4 x i32> %c) {
+; CHECK-LABEL: and_of_multiuse_icmp_sle:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p1.s
+; CHECK-NEXT:    cmpge p1.s, p1/z, z1.s, z0.s
+; CHECK-NEXT:    and p0.b, p0/z, p0.b, p1.b
+; CHECK-NEXT:    ret
+  %cmp = icmp sle <vscale x 4 x i32> %b, %c
+  %and = and <vscale x 4 x i1> %a, %cmp
+  %ins.1 = insertvalue %svboolx2 poison, <vscale x 4 x i1> %and, 0
+  %ins.2 = insertvalue %svboolx2 %ins.1, <vscale x 4 x i1> %cmp, 1
+  ret %svboolx2 %ins.2
+}
+
+define %svboolx2 @and_of_multiuse_icmp_sle_imm(<vscale x 4 x i1> %a, <vscale x 4 x i32> %b) {
+; CHECK-LABEL: and_of_multiuse_icmp_sle_imm:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p1.s
+; CHECK-NEXT:    cmple p1.s, p1/z, z0.s, #1
+; CHECK-NEXT:    and p0.b, p0/z, p0.b, p1.b
+; CHECK-NEXT:    ret
+  %imm = shufflevector <vscale x 4 x i32> insertelement (<vscale x 4 x i32> undef, i32 1, i64 0), <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
+  %cmp = icmp sle <vscale x 4 x i32> %b, %imm
+  %and = and <vscale x 4 x i1> %a, %cmp
+  %ins.1 = insertvalue %svboolx2 poison, <vscale x 4 x i1> %and, 0
+  %ins.2 = insertvalue %svboolx2 %ins.1, <vscale x 4 x i1> %cmp, 1
+  ret %svboolx2 %ins.2
+}
+
+define %svboolx2 @and_of_multiuse_icmp_ugt(<vscale x 4 x i1> %a, <vscale x 4 x i32> %b, <vscale x 4 x i32> %c) {
+; CHECK-LABEL: and_of_multiuse_icmp_ugt:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p1.s
+; CHECK-NEXT:    cmphi p1.s, p1/z, z0.s, z1.s
+; CHECK-NEXT:    and p0.b, p0/z, p0.b, p1.b
+; CHECK-NEXT:    ret
+  %cmp = icmp ugt <vscale x 4 x i32> %b, %c
+  %and = and <vscale x 4 x i1> %a, %cmp
+  %ins.1 = insertvalue %svboolx2 poison, <vscale x 4 x i1> %and, 0
+  %ins.2 = insertvalue %svboolx2 %ins.1, <vscale x 4 x i1> %cmp, 1
+  ret %svboolx2 %ins.2
+}
+
+define %svboolx2 @and_of_multiuse_icmp_ugt_imm(<vscale x 4 x i1> %a, <vscale x 4 x i32> %b) {
+; CHECK-LABEL: and_of_multiuse_icmp_ugt_imm:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p1.s
+; CHECK-NEXT:    cmphi p1.s, p1/z, z0.s, #1
+; CHECK-NEXT:    and p0.b, p0/z, p0.b, p1.b
+; CHECK-NEXT:    ret
+  %imm = shufflevector <vscale x 4 x i32> insertelement (<vscale x 4 x i32> undef, i32 1, i64 0), <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
+  %cmp = icmp ugt <vscale x 4 x i32> %b, %imm
+  %and = and <vscale x 4 x i1> %a, %cmp
+  %ins.1 = insertvalue %svboolx2 poison, <vscale x 4 x i1> %and, 0
+  %ins.2 = insertvalue %svboolx2 %ins.1, <vscale x 4 x i1> %cmp, 1
+  ret %svboolx2 %ins.2
+}
+
 declare <vscale x 16 x i1> @llvm.aarch64.sve.cmpeq.nxv16i8(<vscale x 16 x i1>, <vscale x 16 x i8>, <vscale x 16 x i8>)
 declare <vscale x 8 x i1> @llvm.aarch64.sve.cmpeq.nxv8i16(<vscale x 8 x i1>, <vscale x 8 x i16>, <vscale x 8 x i16>)
 declare <vscale x 4 x i1> @llvm.aarch64.sve.cmpeq.nxv4i32(<vscale x 4 x i1>, <vscale x 4 x i32>, <vscale x 4 x i32>)
diff --git a/llvm/test/CodeGen/AArch64/sve-pseudos-expand-undef.mir b/llvm/test/CodeGen/AArch64/sve-pseudos-expand-undef.mir
index c81363f6133b7..1524fdd0336cb 100644
--- a/llvm/test/CodeGen/AArch64/sve-pseudos-expand-undef.mir
+++ b/llvm/test/CodeGen/AArch64/sve-pseudos-expand-undef.mir
@@ -20,3 +20,23 @@ body:             |
     RET_ReallyLR
 
 ...
+
+# CHECK: {{.*}} MSB_ZPmZZ_B {{.*}}
+---
+name: expand_mls_to_msb
+body:             |
+  bb.0:
+    renamable $p0 = PTRUE_B 31
+    renamable $z0 = MLS_ZPZZZ_UNDEF_B killed renamable $p0, killed renamable $z2, killed renamable $z0, killed renamable $z1
+    RET_ReallyLR implicit $z0
+...
+
+# CHECK: {{.*}} MAD_ZPmZZ_B {{.*}}
+---
+name: expand_mla_to_mad
+body:             |
+  bb.0:
+    renamable $p0 = PTRUE_B 31
+    renamable $z0 = MLA_ZPZZZ_UNDEF_B killed renamable $p0, killed renamable $z2, killed renamable $z0, killed renamable $z1
+    RET_ReallyLR implicit $z0
+...
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-int-rem.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-int-rem.ll
index 910e18dec1a94..31c0a24c166d3 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-int-rem.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-int-rem.ll
@@ -109,10 +109,10 @@ define <16 x i8> @srem_v16i8(<16 x i8> %op1, <16 x i8> %op2) #0 {
 define void @srem_v32i8(ptr %a, ptr %b) #0 {
 ; CHECK-LABEL: srem_v32i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ldp q1, q0, [x0]
+; CHECK-NEXT:    ldp q2, q0, [x0]
 ; CHECK-NEXT:    ptrue p0.s, vl4
 ; CHECK-NEXT:    ptrue p1.h, vl4
-; CHECK-NEXT:    ldp q3, q2, [x1]
+; CHECK-NEXT:    ldp q3, q1, [x1]
 ; CHECK-NEXT:    mov z5.d, z0.d
 ; CHECK-NEXT:    sunpklo z7.h, z0.b
 ; CHECK-NEXT:    ext z5.b, z5.b, z0.b, #8
@@ -120,9 +120,9 @@ define void @srem_v32i8(ptr %a, ptr %b) #0 {
 ; CHECK-NEXT:    sunpklo z18.s, z5.h
 ; CHECK-NEXT:    ext z5.b, z5.b, z5.b, #8
 ; CHECK-NEXT:    sunpklo z5.s, z5.h
-; CHECK-NEXT:    mov z4.d, z2.d
-; CHECK-NEXT:    sunpklo z6.h, z2.b
-; CHECK-NEXT:    ext z4.b, z4.b, z2.b, #8
+; CHECK-NEXT:    mov z4.d, z1.d
+; CHECK-NEXT:    sunpklo z6.h, z1.b
+; CHECK-NEXT:    ext z4.b, z4.b, z1.b, #8
 ; CHECK-NEXT:    sunpklo z16.s, z6.h
 ; CHECK-NEXT:    sunpklo z4.h, z4.b
 ; CHECK-NEXT:    ext z6.b, z6.b, z6.b, #8
@@ -139,9 +139,9 @@ define void @srem_v32i8(ptr %a, ptr %b) #0 {
 ; CHECK-NEXT:    splice z17.h, p1, z17.h, z4.h
 ; CHECK-NEXT:    sunpklo z4.s, z7.h
 ; CHECK-NEXT:    mov z6.d, z3.d
-; CHECK-NEXT:    mov z7.d, z1.d
+; CHECK-NEXT:    mov z7.d, z2.d
 ; CHECK-NEXT:    ext z6.b, z6.b, z3.b, #8
-; CHECK-NEXT:    ext z7.b, z7.b, z1.b, #8
+; CHECK-NEXT:    ext z7.b, z7.b, z2.b, #8
 ; CHECK-NEXT:    sdivr z16.s, p0/m, z16.s, z18.s
 ; CHECK-NEXT:    sunpklo z6.h, z6.b
 ; CHECK-NEXT:    sunpklo z7.h, z7.b
@@ -161,7 +161,7 @@ define void @srem_v32i8(ptr %a, ptr %b) #0 {
 ; CHECK-NEXT:    splice z5.h, p1, z5.h, z4.h
 ; CHECK-NEXT:    splice z7.h, p1, z7.h, z6.h
 ; CHECK-NEXT:    sunpklo z4.h, z3.b
-; CHECK-NEXT:    sunpklo z6.h, z1.b
+; CHECK-NEXT:    sunpklo z6.h, z2.b
 ; CHECK-NEXT:    sunpklo z16.s, z4.h
 ; CHECK-NEXT:    sunpklo z18.s, z6.h
 ; CHECK-NEXT:    ext z4.b, z4.b, z4.b, #8
@@ -181,9 +181,9 @@ define void @srem_v32i8(ptr %a, ptr %b) #0 {
 ; CHECK-NEXT:    ptrue p1.b, vl16
 ; CHECK-NEXT:    splice z7.b, p0, z7.b, z4.b
 ; CHECK-NEXT:    splice z5.b, p0, z5.b, z6.b
-; CHECK-NEXT:    mls z1.b, p1/m, z7.b, z3.b
-; CHECK-NEXT:    mls z0.b, p1/m, z5.b, z2.b
-; CHECK-NEXT:    stp q1, q0, [x0]
+; CHECK-NEXT:    mls z2.b, p1/m, z7.b, z3.b
+; CHECK-NEXT:    mls z0.b, p1/m, z5.b, z1.b
+; CHECK-NEXT:    stp q2, q0, [x0]
 ; CHECK-NEXT:    ret
   %op1 = load <32 x i8>, ptr %a
   %op2 = load <32 x i8>, ptr %b
@@ -492,10 +492,10 @@ define <16 x i8> @urem_v16i8(<16 x i8> %op1, <16 x i8> %op2) #0 {
 define void @urem_v32i8(ptr %a, ptr %b) #0 {
 ; CHECK-LABEL: urem_v32i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ldp q1, q0, [x0]
+; CHECK-NEXT:    ldp q2, q0, [x0]
 ; CHECK-NEXT:    ptrue p0.s, vl4
 ; CHECK-NEXT:    ptrue p1.h, vl4
-; CHECK-NEXT:    ldp q3, q2, [x1]
+; CHECK-NEXT:    ldp q3, q1, [x1]
 ; CHECK-NEXT:    mov z5.d, z0.d
 ; CHECK-NEXT:    uunpklo z7.h, z0.b
 ; CHECK-NEXT:    ext z5.b, z5.b, z0.b, #8
@@ -503,9 +503,9 @@ define void @urem_v32i8(ptr %a, ptr %b) #0 {
 ; CHECK-NEXT:    uunpklo z18.s, z5.h
 ; CHECK-NEXT:    ext z5.b, z5.b, z5.b, #8
 ; CHECK-NEXT:    uunpklo z5.s, z5.h
-; CHECK-NEXT:    mov z4.d, z2.d
-; CHECK-NEXT:    uunpklo z6.h, z2.b
-; CHECK-NEXT:    ext z4.b, z4.b, z2.b, #8
+; CHECK-NEXT:    mov z4.d, z1.d
+; CHECK-NEXT:    uunpklo z6.h, z1.b
+; CHECK-NEXT:    ext z4.b, z4.b, z1.b, #8
 ; CHECK-NEXT:    uunpklo z16.s, z6.h
 ; CHECK-NEXT:    uunpklo z4.h, z4.b
 ; CHECK-NEXT:    ext z6.b, z6.b, z6.b, #8
@@ -522,9 +522,9 @@ define void @urem_v32i8(ptr %a, ptr %b) #0 {
 ; CHECK-NEXT:    splice z17.h, p1, z17.h, z4.h
 ; CHECK-NEXT:    uunpklo z4.s, z7.h
 ; CHECK-NEXT:    mov z6.d, z3.d
-; CHECK-NEXT:    mov z7.d, z1.d
+; CHECK-NEXT:    mov z7.d, z2.d
 ; CHECK-NEXT:    ext z6.b, z6.b, z3.b, #8
-; CHECK-NEXT:    ext z7.b, z7.b, z1.b, #8
+; CHECK-NEXT:    ext z7.b, z7.b, z2.b, #8
 ; CHECK-NEXT:    udivr z16.s, p0/m, z16.s, z18.s
 ; CHECK-NEXT:    uunpklo z6.h, z6.b
 ; CHECK-NEXT:    uunpklo z7.h, z7.b
@@ -544,7 +544,7 @@ define void @urem_v32i8(ptr %a, ptr %b) #0 {
 ; CHECK-NEXT:    splice z5.h, p1, z5.h, z4.h
 ; CHECK-NEXT:    splice z7.h, p1, z7.h, z6.h
 ; CHECK-NEXT:    uunpklo z4.h, z3.b
-; CHECK-NEXT:    uunpklo z6.h, z1.b
+; CHECK-NEXT:    uunpklo z6.h, z2.b
 ; CHECK-NEXT:    uunpklo z16.s, z4.h
 ; CHECK-NEXT:    uunpklo z18.s, z6.h
 ; CHECK-NEXT:    ext z4.b, z4.b, z4.b, #8
@@ -564,9 +564,9 @@ define void @urem_v32i8(ptr %a, ptr %b) #0 {
 ; CHECK-NEXT:    ptrue p1.b, vl16
 ; CHECK-NEXT:    splice z7.b, p0, z7.b, z4.b
 ; CHECK-NEXT:    splice z5.b, p0, z5.b, z6.b
-; CHECK-NEXT:    mls z1.b, p1/m, z7.b, z3.b
-; CHECK-NEXT:    mls z0.b, p1/m, z5.b, z2.b
-; CHECK-NEXT:    stp q1, q0, [x0]
+; CHECK-NEXT:    mls z2.b, p1/m, z7.b, z3.b
+; CHECK-NEXT:    mls z0.b, p1/m, z5.b, z1.b
+; CHECK-NEXT:    stp q2, q0, [x0]
 ; CHECK-NEXT:    ret
   %op1 = load <32 x i8>, ptr %a
   %op2 = load <32 x i8>, ptr %b
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-ld2-alloca.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-ld2-alloca.ll
index 14df062df15d7..c96b88c0a33e3 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-ld2-alloca.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-ld2-alloca.ll
@@ -8,23 +8,25 @@ declare void @def(ptr)
 define void @alloc_v4i8(ptr %st_ptr) #0 {
 ; CHECK-LABEL: alloc_v4i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    sub sp, sp, #32
-; CHECK-NEXT:    stp x30, x19, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    sub sp, sp, #48
+; CHECK-NEXT:    stp x20, x19, [sp, #32] // 16-byte Folded Spill
 ; CHECK-NEXT:    mov x19, x0
-; CHECK-NEXT:    add x0, sp, #12
+; CHECK-NEXT:    add x0, sp, #28
+; CHECK-NEXT:    str x30, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEXT:    add x20, sp, #28
 ; CHECK-NEXT:    bl def
-; CHECK-NEXT:    add x8, sp, #12
 ; CHECK-NEXT:    ptrue p0.b, vl2
-; CHECK-NEXT:    ld2b { z0.b, z1.b }, p0/z, [x8]
+; CHECK-NEXT:    ld2b { z0.b, z1.b }, p0/z, [x20]
 ; CHECK-NEXT:    ptrue p0.s, vl2
+; CHECK-NEXT:    ldr x30, [sp, #16] // 8-byte Folded Reload
 ; CHECK-NEXT:    mov z2.b, z0.b[1]
 ; CHECK-NEXT:    fmov w8, s0
 ; CHECK-NEXT:    fmov w9, s2
-; CHECK-NEXT:    stp w8, w9, [sp]
-; CHECK-NEXT:    ldr d0, [sp]
+; CHECK-NEXT:    stp w8, w9, [sp, #8]
+; CHECK-NEXT:    ldr d0, [sp, #8]
 ; CHECK-NEXT:    st1b { z0.s }, p0, [x19]
-; CHECK-NEXT:    ldp x30, x19, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT:    add sp, sp, #32
+; CHECK-NEXT:    ldp x20, x19, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT:    add sp, sp, #48
 ; CHECK-NEXT:    ret
   %alloc = alloca [4 x i8]
   call void @def(ptr %alloc)
@@ -38,13 +40,14 @@ define void @alloc_v6i8(ptr %st_ptr) #0 {
 ; CHECK-LABEL: alloc_v6i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    sub sp, sp, #48
-; CHECK-NEXT:    stp x30, x19, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT:    stp x20, x19, [sp, #32] // 16-byte Folded Spill
 ; CHECK-NEXT:    mov x19, x0
 ; CHECK-NEXT:    add x0, sp, #24
+; CHECK-NEXT:    str x30, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEXT:    add x20, sp, #24
 ; CHECK-NEXT:    bl def
-; CHECK-NEXT:    add x8, sp, #24
 ; CHECK-NEXT:    ptrue p0.b, vl3
-; CHECK-NEXT:    ld2b { z0.b, z1.b }, p0/z, [x8]
+; CHECK-NEXT:    ld2b { z0.b, z1.b }, p0/z, [x20]
 ; CHECK-NEXT:    ptrue p0.h, vl4
 ; CHECK-NEXT:    fmov w8, s1
 ; CHECK-NEXT:    mov z2.b, z1.b[3]
@@ -52,18 +55,19 @@ define void @alloc_v6i8(ptr %st_ptr) #0 {
 ; CHECK-NEXT:    mov z0.b, z1.b[1]
 ; CHECK-NEXT:    fmov w9, s2
 ; CHECK-NEXT:    fmov w10, s3
-; CHECK-NEXT:    strh w8, [sp, #8]
+; CHECK-NEXT:    strh w8, [sp]
 ; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    strh w9, [sp, #14]
-; CHECK-NEXT:    strh w10, [sp, #12]
-; CHECK-NEXT:    strh w8, [sp, #10]
-; CHECK-NEXT:    add x8, sp, #20
-; CHECK-NEXT:    ldr d0, [sp, #8]
+; CHECK-NEXT:    strh w9, [sp, #6]
+; CHECK-NEXT:    strh w10, [sp, #4]
+; CHECK-NEXT:    strh w8, [sp, #2]
+; CHECK-NEXT:    add x8, sp, #12
+; CHECK-NEXT:    ldr d0, [sp]
 ; CHECK-NEXT:    st1b { z0.h }, p0, [x8]
-; CHECK-NEXT:    ldrh w8, [sp, #20]
+; CHECK-NEXT:    ldrh w8, [sp, #12]
 ; CHECK-NEXT:    strb w10, [x19, #2]
+; CHECK-NEXT:    ldr x30, [sp, #16] // 8-byte Folded Reload
 ; CHECK-NEXT:    strh w8, [x19]
-; CHECK-NEXT:    ldp x30, x19, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp x20, x19, [sp, #32] // 16-byte Folded Reload
 ; CHECK-NEXT:    add sp, sp, #48
 ; CHECK-NEXT:    ret
   %alloc = alloca [6 x i8]
@@ -135,7 +139,7 @@ define void @alloc_v8f64(ptr %st_ptr) #0 {
 ; CHECK-NEXT:    bl def
 ; CHECK-NEXT:    mov x8, #4
 ; CHECK-NEXT:    ptrue p0.d, vl2
-; CHECK-NEXT:    ld2d { z0.d, z1.d }, p0/z, [sp]
+; CHECK-NEXT:    ld2d { z0.d, z1.d }, p0/z, [x20]
 ; CHECK-NEXT:    ld2d { z2.d, z3.d }, p0/z, [x20, x8, lsl #3]
 ; CHECK-NEXT:    ldr x30, [sp, #64] // 8-byte Folded Reload
 ; CHECK-NEXT:    stp q0, q2, [x19]
diff --git a/llvm/test/CodeGen/AArch64/sve-vscale.ll b/llvm/test/CodeGen/AArch64/sve-vscale.ll
index aa24a111036cf..3f2f5e6e3136d 100644
--- a/llvm/test/CodeGen/AArch64/sve-vscale.ll
+++ b/llvm/test/CodeGen/AArch64/sve-vscale.ll
@@ -46,7 +46,10 @@ define i64 @rdvl_i64() nounwind {
 ; CHECK:       rdvl x0, #1
 ; CHECK-NEXT:  ret
 define i32 @rdvl_const() nounwind {
-  ret i32 mul nsw (i32 ptrtoint (<vscale x 1 x i8>* getelementptr (<vscale x 1 x i8>, <vscale x 1 x i8>* null, i64 1) to i32), i32 16)
+  %vscale.ptr = getelementptr <vscale x 1 x i8>, ptr null, i64 1
+  %vscale.int = ptrtoint ptr %vscale.ptr to i32
+  %vscale.scaled = mul nsw i32 %vscale.int, 16
+  ret i32 %vscale.scaled
 }
 
 define i32 @vscale_1() nounwind {
diff --git a/llvm/test/CodeGen/AArch64/sve2-hadd.ll b/llvm/test/CodeGen/AArch64/sve2-hadd.ll
deleted file mode 100644
index 9363412ff06ba..0000000000000
--- a/llvm/test/CodeGen/AArch64/sve2-hadd.ll
+++ /dev/null
@@ -1,627 +0,0 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc < %s -mtriple aarch64-none-eabi -mattr=+sve2 -o - | FileCheck %s
-
-define <vscale x 2 x i64> @hadds_v2i64(<vscale x 2 x i64> %s0, <vscale x 2 x i64> %s1) {
-; CHECK-LABEL: hadds_v2i64:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    shadd z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT:    ret
-entry:
-  %s0s = sext <vscale x 2 x i64> %s0 to <vscale x 2 x i128>
-  %s1s = sext <vscale x 2 x i64> %s1 to <vscale x 2 x i128>
-  %m = add nsw <vscale x 2 x i128> %s0s, %s1s
-  %s = lshr <vscale x 2 x i128> %m, shufflevector (<vscale x 2 x i128> insertelement (<vscale x 2 x i128> poison, i128 1, i32 0), <vscale x 2 x i128> poison, <vscale x 2 x i32> zeroinitializer)
-  %s2 = trunc <vscale x 2 x i128> %s to <vscale x 2 x i64>
-  ret <vscale x 2 x i64> %s2
-}
-
-define <vscale x 2 x i64> @haddu_v2i64(<vscale x 2 x i64> %s0, <vscale x 2 x i64> %s1) {
-; CHECK-LABEL: haddu_v2i64:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    uhadd z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT:    ret
-entry:
-  %s0s = zext <vscale x 2 x i64> %s0 to <vscale x 2 x i128>
-  %s1s = zext <vscale x 2 x i64> %s1 to <vscale x 2 x i128>
-  %m = add nuw nsw <vscale x 2 x i128> %s0s, %s1s
-  %s = lshr <vscale x 2 x i128> %m, shufflevector (<vscale x 2 x i128> insertelement (<vscale x 2 x i128> poison, i128 1, i32 0), <vscale x 2 x i128> poison, <vscale x 2 x i32> zeroinitializer)
-  %s2 = trunc <vscale x 2 x i128> %s to <vscale x 2 x i64>
-  ret <vscale x 2 x i64> %s2
-}
-
-define <vscale x 2 x i32> @hadds_v2i32(<vscale x 2 x i32> %s0, <vscale x 2 x i32> %s1) {
-; CHECK-LABEL: hadds_v2i32:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    sxtw z0.d, p0/m, z0.d
-; CHECK-NEXT:    adr z0.d, [z0.d, z1.d, sxtw]
-; CHECK-NEXT:    lsr z0.d, z0.d, #1
-; CHECK-NEXT:    ret
-entry:
-  %s0s = sext <vscale x 2 x i32> %s0 to <vscale x 2 x i64>
-  %s1s = sext <vscale x 2 x i32> %s1 to <vscale x 2 x i64>
-  %m = add nsw <vscale x 2 x i64> %s0s, %s1s
-  %s = lshr <vscale x 2 x i64> %m, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 1, i32 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
-  %s2 = trunc <vscale x 2 x i64> %s to <vscale x 2 x i32>
-  ret <vscale x 2 x i32> %s2
-}
-
-define <vscale x 2 x i32> @haddu_v2i32(<vscale x 2 x i32> %s0, <vscale x 2 x i32> %s1) {
-; CHECK-LABEL: haddu_v2i32:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    and z0.d, z0.d, #0xffffffff
-; CHECK-NEXT:    and z1.d, z1.d, #0xffffffff
-; CHECK-NEXT:    uhadd z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT:    ret
-entry:
-  %s0s = zext <vscale x 2 x i32> %s0 to <vscale x 2 x i64>
-  %s1s = zext <vscale x 2 x i32> %s1 to <vscale x 2 x i64>
-  %m = add nuw nsw <vscale x 2 x i64> %s0s, %s1s
-  %s = lshr <vscale x 2 x i64> %m, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 1, i32 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
-  %s2 = trunc <vscale x 2 x i64> %s to <vscale x 2 x i32>
-  ret <vscale x 2 x i32> %s2
-}
-
-define <vscale x 4 x i32> @hadds_v4i32(<vscale x 4 x i32> %s0, <vscale x 4 x i32> %s1) {
-; CHECK-LABEL: hadds_v4i32:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    shadd z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT:    ret
-entry:
-  %s0s = sext <vscale x 4 x i32> %s0 to <vscale x 4 x i64>
-  %s1s = sext <vscale x 4 x i32> %s1 to <vscale x 4 x i64>
-  %m = add nsw <vscale x 4 x i64> %s0s, %s1s
-  %s = lshr <vscale x 4 x i64> %m, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 1, i32 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer)
-  %s2 = trunc <vscale x 4 x i64> %s to <vscale x 4 x i32>
-  ret <vscale x 4 x i32> %s2
-}
-
-define <vscale x 4 x i32> @haddu_v4i32(<vscale x 4 x i32> %s0, <vscale x 4 x i32> %s1) {
-; CHECK-LABEL: haddu_v4i32:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    uhadd z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT:    ret
-entry:
-  %s0s = zext <vscale x 4 x i32> %s0 to <vscale x 4 x i64>
-  %s1s = zext <vscale x 4 x i32> %s1 to <vscale x 4 x i64>
-  %m = add nuw nsw <vscale x 4 x i64> %s0s, %s1s
-  %s = lshr <vscale x 4 x i64> %m, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 1, i32 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer)
-  %s2 = trunc <vscale x 4 x i64> %s to <vscale x 4 x i32>
-  ret <vscale x 4 x i32> %s2
-}
-
-define <vscale x 2 x i16> @hadds_v2i16(<vscale x 2 x i16> %s0, <vscale x 2 x i16> %s1) {
-; CHECK-LABEL: hadds_v2i16:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    sxth z0.d, p0/m, z0.d
-; CHECK-NEXT:    sxth z1.d, p0/m, z1.d
-; CHECK-NEXT:    add z0.d, z0.d, z1.d
-; CHECK-NEXT:    and z0.d, z0.d, #0xffffffff
-; CHECK-NEXT:    lsr z0.d, z0.d, #1
-; CHECK-NEXT:    ret
-entry:
-  %s0s = sext <vscale x 2 x i16> %s0 to <vscale x 2 x i32>
-  %s1s = sext <vscale x 2 x i16> %s1 to <vscale x 2 x i32>
-  %m = add nsw <vscale x 2 x i32> %s0s, %s1s
-  %s = lshr <vscale x 2 x i32> %m, shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> poison, i32 1, i32 0), <vscale x 2 x i32> poison, <vscale x 2 x i32> zeroinitializer)
-  %s2 = trunc <vscale x 2 x i32> %s to <vscale x 2 x i16>
-  ret <vscale x 2 x i16> %s2
-}
-
-define <vscale x 2 x i16> @haddu_v2i16(<vscale x 2 x i16> %s0, <vscale x 2 x i16> %s1) {
-; CHECK-LABEL: haddu_v2i16:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    and z0.d, z0.d, #0xffff
-; CHECK-NEXT:    and z1.d, z1.d, #0xffff
-; CHECK-NEXT:    uhadd z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT:    ret
-entry:
-  %s0s = zext <vscale x 2 x i16> %s0 to <vscale x 2 x i32>
-  %s1s = zext <vscale x 2 x i16> %s1 to <vscale x 2 x i32>
-  %m = add nuw nsw <vscale x 2 x i32> %s0s, %s1s
-  %s = lshr <vscale x 2 x i32> %m, shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> poison, i32 1, i32 0), <vscale x 2 x i32> poison, <vscale x 2 x i32> zeroinitializer)
-  %s2 = trunc <vscale x 2 x i32> %s to <vscale x 2 x i16>
-  ret <vscale x 2 x i16> %s2
-}
-
-define <vscale x 4 x i16> @hadds_v4i16(<vscale x 4 x i16> %s0, <vscale x 4 x i16> %s1) {
-; CHECK-LABEL: hadds_v4i16:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    sxth z0.s, p0/m, z0.s
-; CHECK-NEXT:    sxth z1.s, p0/m, z1.s
-; CHECK-NEXT:    add z0.s, z0.s, z1.s
-; CHECK-NEXT:    lsr z0.s, z0.s, #1
-; CHECK-NEXT:    ret
-entry:
-  %s0s = sext <vscale x 4 x i16> %s0 to <vscale x 4 x i32>
-  %s1s = sext <vscale x 4 x i16> %s1 to <vscale x 4 x i32>
-  %m = add nsw <vscale x 4 x i32> %s0s, %s1s
-  %s = lshr <vscale x 4 x i32> %m, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 1, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
-  %s2 = trunc <vscale x 4 x i32> %s to <vscale x 4 x i16>
-  ret <vscale x 4 x i16> %s2
-}
-
-define <vscale x 4 x i16> @haddu_v4i16(<vscale x 4 x i16> %s0, <vscale x 4 x i16> %s1) {
-; CHECK-LABEL: haddu_v4i16:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    and z0.s, z0.s, #0xffff
-; CHECK-NEXT:    and z1.s, z1.s, #0xffff
-; CHECK-NEXT:    uhadd z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT:    ret
-entry:
-  %s0s = zext <vscale x 4 x i16> %s0 to <vscale x 4 x i32>
-  %s1s = zext <vscale x 4 x i16> %s1 to <vscale x 4 x i32>
-  %m = add nuw nsw <vscale x 4 x i32> %s0s, %s1s
-  %s = lshr <vscale x 4 x i32> %m, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 1, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
-  %s2 = trunc <vscale x 4 x i32> %s to <vscale x 4 x i16>
-  ret <vscale x 4 x i16> %s2
-}
-
-define <vscale x 8 x i16> @hadds_v8i16(<vscale x 8 x i16> %s0, <vscale x 8 x i16> %s1) {
-; CHECK-LABEL: hadds_v8i16:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.h
-; CHECK-NEXT:    shadd z0.h, p0/m, z0.h, z1.h
-; CHECK-NEXT:    ret
-entry:
-  %s0s = sext <vscale x 8 x i16> %s0 to <vscale x 8 x i32>
-  %s1s = sext <vscale x 8 x i16> %s1 to <vscale x 8 x i32>
-  %m = add nsw <vscale x 8 x i32> %s0s, %s1s
-  %s = lshr <vscale x 8 x i32> %m, shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 1, i32 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer)
-  %s2 = trunc <vscale x 8 x i32> %s to <vscale x 8 x i16>
-  ret <vscale x 8 x i16> %s2
-}
-
-define <vscale x 8 x i16> @haddu_v8i16(<vscale x 8 x i16> %s0, <vscale x 8 x i16> %s1) {
-; CHECK-LABEL: haddu_v8i16:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.h
-; CHECK-NEXT:    uhadd z0.h, p0/m, z0.h, z1.h
-; CHECK-NEXT:    ret
-entry:
-  %s0s = zext <vscale x 8 x i16> %s0 to <vscale x 8 x i32>
-  %s1s = zext <vscale x 8 x i16> %s1 to <vscale x 8 x i32>
-  %m = add nuw nsw <vscale x 8 x i32> %s0s, %s1s
-  %s = lshr <vscale x 8 x i32> %m, shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 1, i32 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer)
-  %s2 = trunc <vscale x 8 x i32> %s to <vscale x 8 x i16>
-  ret <vscale x 8 x i16> %s2
-}
-
-define <vscale x 4 x i8> @hadds_v4i8(<vscale x 4 x i8> %s0, <vscale x 4 x i8> %s1) {
-; CHECK-LABEL: hadds_v4i8:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    sxtb z0.s, p0/m, z0.s
-; CHECK-NEXT:    sxtb z1.s, p0/m, z1.s
-; CHECK-NEXT:    add z0.s, z0.s, z1.s
-; CHECK-NEXT:    and z0.s, z0.s, #0xffff
-; CHECK-NEXT:    lsr z0.s, z0.s, #1
-; CHECK-NEXT:    ret
-entry:
-  %s0s = sext <vscale x 4 x i8> %s0 to <vscale x 4 x i16>
-  %s1s = sext <vscale x 4 x i8> %s1 to <vscale x 4 x i16>
-  %m = add nsw <vscale x 4 x i16> %s0s, %s1s
-  %s = lshr <vscale x 4 x i16> %m, shufflevector (<vscale x 4 x i16> insertelement (<vscale x 4 x i16> poison, i16 1, i32 0), <vscale x 4 x i16> poison, <vscale x 4 x i32> zeroinitializer)
-  %s2 = trunc <vscale x 4 x i16> %s to <vscale x 4 x i8>
-  ret <vscale x 4 x i8> %s2
-}
-
-define <vscale x 4 x i8> @haddu_v4i8(<vscale x 4 x i8> %s0, <vscale x 4 x i8> %s1) {
-; CHECK-LABEL: haddu_v4i8:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    and z0.s, z0.s, #0xff
-; CHECK-NEXT:    and z1.s, z1.s, #0xff
-; CHECK-NEXT:    uhadd z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT:    ret
-entry:
-  %s0s = zext <vscale x 4 x i8> %s0 to <vscale x 4 x i16>
-  %s1s = zext <vscale x 4 x i8> %s1 to <vscale x 4 x i16>
-  %m = add nuw nsw <vscale x 4 x i16> %s0s, %s1s
-  %s = lshr <vscale x 4 x i16> %m, shufflevector (<vscale x 4 x i16> insertelement (<vscale x 4 x i16> poison, i16 1, i32 0), <vscale x 4 x i16> poison, <vscale x 4 x i32> zeroinitializer)
-  %s2 = trunc <vscale x 4 x i16> %s to <vscale x 4 x i8>
-  ret <vscale x 4 x i8> %s2
-}
-
-define <vscale x 8 x i8> @hadds_v8i8(<vscale x 8 x i8> %s0, <vscale x 8 x i8> %s1) {
-; CHECK-LABEL: hadds_v8i8:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.h
-; CHECK-NEXT:    sxtb z0.h, p0/m, z0.h
-; CHECK-NEXT:    sxtb z1.h, p0/m, z1.h
-; CHECK-NEXT:    add z0.h, z0.h, z1.h
-; CHECK-NEXT:    lsr z0.h, z0.h, #1
-; CHECK-NEXT:    ret
-entry:
-  %s0s = sext <vscale x 8 x i8> %s0 to <vscale x 8 x i16>
-  %s1s = sext <vscale x 8 x i8> %s1 to <vscale x 8 x i16>
-  %m = add nsw <vscale x 8 x i16> %s0s, %s1s
-  %s = lshr <vscale x 8 x i16> %m, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 1, i32 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
-  %s2 = trunc <vscale x 8 x i16> %s to <vscale x 8 x i8>
-  ret <vscale x 8 x i8> %s2
-}
-
-define <vscale x 8 x i8> @haddu_v8i8(<vscale x 8 x i8> %s0, <vscale x 8 x i8> %s1) {
-; CHECK-LABEL: haddu_v8i8:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.h
-; CHECK-NEXT:    and z0.h, z0.h, #0xff
-; CHECK-NEXT:    and z1.h, z1.h, #0xff
-; CHECK-NEXT:    uhadd z0.h, p0/m, z0.h, z1.h
-; CHECK-NEXT:    ret
-entry:
-  %s0s = zext <vscale x 8 x i8> %s0 to <vscale x 8 x i16>
-  %s1s = zext <vscale x 8 x i8> %s1 to <vscale x 8 x i16>
-  %m = add nuw nsw <vscale x 8 x i16> %s0s, %s1s
-  %s = lshr <vscale x 8 x i16> %m, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 1, i32 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
-  %s2 = trunc <vscale x 8 x i16> %s to <vscale x 8 x i8>
-  ret <vscale x 8 x i8> %s2
-}
-
-define <vscale x 16 x i8> @hadds_v16i8(<vscale x 16 x i8> %s0, <vscale x 16 x i8> %s1) {
-; CHECK-LABEL: hadds_v16i8:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.b
-; CHECK-NEXT:    shadd z0.b, p0/m, z0.b, z1.b
-; CHECK-NEXT:    ret
-entry:
-  %s0s = sext <vscale x 16 x i8> %s0 to <vscale x 16 x i16>
-  %s1s = sext <vscale x 16 x i8> %s1 to <vscale x 16 x i16>
-  %m = add nsw <vscale x 16 x i16> %s0s, %s1s
-  %s = lshr <vscale x 16 x i16> %m, shufflevector (<vscale x 16 x i16> insertelement (<vscale x 16 x i16> poison, i16 1, i32 0), <vscale x 16 x i16> poison, <vscale x 16 x i32> zeroinitializer)
-  %s2 = trunc <vscale x 16 x i16> %s to <vscale x 16 x i8>
-  ret <vscale x 16 x i8> %s2
-}
-
-define <vscale x 16 x i8> @haddu_v16i8(<vscale x 16 x i8> %s0, <vscale x 16 x i8> %s1) {
-; CHECK-LABEL: haddu_v16i8:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.b
-; CHECK-NEXT:    uhadd z0.b, p0/m, z0.b, z1.b
-; CHECK-NEXT:    ret
-entry:
-  %s0s = zext <vscale x 16 x i8> %s0 to <vscale x 16 x i16>
-  %s1s = zext <vscale x 16 x i8> %s1 to <vscale x 16 x i16>
-  %m = add nuw nsw <vscale x 16 x i16> %s0s, %s1s
-  %s = lshr <vscale x 16 x i16> %m, shufflevector (<vscale x 16 x i16> insertelement (<vscale x 16 x i16> poison, i16 1, i32 0), <vscale x 16 x i16> poison, <vscale x 16 x i32> zeroinitializer)
-  %s2 = trunc <vscale x 16 x i16> %s to <vscale x 16 x i8>
-  ret <vscale x 16 x i8> %s2
-}
-
-define <vscale x 2 x i64> @rhadds_v2i64(<vscale x 2 x i64> %s0, <vscale x 2 x i64> %s1) {
-; CHECK-LABEL: rhadds_v2i64:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    srhadd z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT:    ret
-entry:
-  %s0s = sext <vscale x 2 x i64> %s0 to <vscale x 2 x i128>
-  %s1s = sext <vscale x 2 x i64> %s1 to <vscale x 2 x i128>
-  %add = add <vscale x 2 x i128> %s0s, shufflevector (<vscale x 2 x i128> insertelement (<vscale x 2 x i128> poison, i128 1, i32 0), <vscale x 2 x i128> poison, <vscale x 2 x i32> zeroinitializer)
-  %add2 = add <vscale x 2 x i128> %add, %s1s
-  %s = lshr <vscale x 2 x i128> %add2, shufflevector (<vscale x 2 x i128> insertelement (<vscale x 2 x i128> poison, i128 1, i32 0), <vscale x 2 x i128> poison, <vscale x 2 x i32> zeroinitializer)
-  %result = trunc <vscale x 2 x i128> %s to <vscale x 2 x i64>
-  ret <vscale x 2 x i64> %result
-}
-
-define <vscale x 2 x i64> @rhaddu_v2i64(<vscale x 2 x i64> %s0, <vscale x 2 x i64> %s1) {
-; CHECK-LABEL: rhaddu_v2i64:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    urhadd z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT:    ret
-entry:
-  %s0s = zext <vscale x 2 x i64> %s0 to <vscale x 2 x i128>
-  %s1s = zext <vscale x 2 x i64> %s1 to <vscale x 2 x i128>
-  %add = add nuw nsw <vscale x 2 x i128> %s0s, shufflevector (<vscale x 2 x i128> insertelement (<vscale x 2 x i128> poison, i128 1, i32 0), <vscale x 2 x i128> poison, <vscale x 2 x i32> zeroinitializer)
-  %add2 = add nuw nsw <vscale x 2 x i128> %add, %s1s
-  %s = lshr <vscale x 2 x i128> %add2, shufflevector (<vscale x 2 x i128> insertelement (<vscale x 2 x i128> poison, i128 1, i32 0), <vscale x 2 x i128> poison, <vscale x 2 x i32> zeroinitializer)
-  %result = trunc <vscale x 2 x i128> %s to <vscale x 2 x i64>
-  ret <vscale x 2 x i64> %result
-}
-
-define <vscale x 2 x i32> @rhadds_v2i32(<vscale x 2 x i32> %s0, <vscale x 2 x i32> %s1) {
-; CHECK-LABEL: rhadds_v2i32:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    mov z2.d, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    sxtw z0.d, p0/m, z0.d
-; CHECK-NEXT:    sxtw z1.d, p0/m, z1.d
-; CHECK-NEXT:    eor z0.d, z0.d, z2.d
-; CHECK-NEXT:    sub z0.d, z1.d, z0.d
-; CHECK-NEXT:    lsr z0.d, z0.d, #1
-; CHECK-NEXT:    ret
-entry:
-  %s0s = sext <vscale x 2 x i32> %s0 to <vscale x 2 x i64>
-  %s1s = sext <vscale x 2 x i32> %s1 to <vscale x 2 x i64>
-  %add = add <vscale x 2 x i64> %s0s, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 1, i32 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
-  %add2 = add <vscale x 2 x i64> %add, %s1s
-  %s = lshr <vscale x 2 x i64> %add2, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 1, i32 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
-  %result = trunc <vscale x 2 x i64> %s to <vscale x 2 x i32>
-  ret <vscale x 2 x i32> %result
-}
-
-define <vscale x 2 x i32> @rhaddu_v2i32(<vscale x 2 x i32> %s0, <vscale x 2 x i32> %s1) {
-; CHECK-LABEL: rhaddu_v2i32:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    and z0.d, z0.d, #0xffffffff
-; CHECK-NEXT:    and z1.d, z1.d, #0xffffffff
-; CHECK-NEXT:    urhadd z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT:    ret
-entry:
-  %s0s = zext <vscale x 2 x i32> %s0 to <vscale x 2 x i64>
-  %s1s = zext <vscale x 2 x i32> %s1 to <vscale x 2 x i64>
-  %add = add nuw nsw <vscale x 2 x i64> %s0s, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 1, i32 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
-  %add2 = add nuw nsw <vscale x 2 x i64> %add, %s1s
-  %s = lshr <vscale x 2 x i64> %add2, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 1, i32 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
-  %result = trunc <vscale x 2 x i64> %s to <vscale x 2 x i32>
-  ret <vscale x 2 x i32> %result
-}
-
-define <vscale x 4 x i32> @rhadds_v4i32(<vscale x 4 x i32> %s0, <vscale x 4 x i32> %s1) {
-; CHECK-LABEL: rhadds_v4i32:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    srhadd z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT:    ret
-entry:
-  %s0s = sext <vscale x 4 x i32> %s0 to <vscale x 4 x i64>
-  %s1s = sext <vscale x 4 x i32> %s1 to <vscale x 4 x i64>
-  %add = add <vscale x 4 x i64> %s0s, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 1, i32 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer)
-  %add2 = add <vscale x 4 x i64> %add, %s1s
-  %s = lshr <vscale x 4 x i64> %add2, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 1, i32 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer)
-  %result = trunc <vscale x 4 x i64> %s to <vscale x 4 x i32>
-  ret <vscale x 4 x i32> %result
-}
-
-define <vscale x 4 x i32> @rhaddu_v4i32(<vscale x 4 x i32> %s0, <vscale x 4 x i32> %s1) {
-; CHECK-LABEL: rhaddu_v4i32:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    urhadd z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT:    ret
-entry:
-  %s0s = zext <vscale x 4 x i32> %s0 to <vscale x 4 x i64>
-  %s1s = zext <vscale x 4 x i32> %s1 to <vscale x 4 x i64>
-  %add = add nuw nsw <vscale x 4 x i64> %s0s, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 1, i32 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer)
-  %add2 = add nuw nsw <vscale x 4 x i64> %add, %s1s
-  %s = lshr <vscale x 4 x i64> %add2, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 1, i32 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer)
-  %result = trunc <vscale x 4 x i64> %s to <vscale x 4 x i32>
-  ret <vscale x 4 x i32> %result
-}
-
-define <vscale x 2 x i16> @rhadds_v2i16(<vscale x 2 x i16> %s0, <vscale x 2 x i16> %s1) {
-; CHECK-LABEL: rhadds_v2i16:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    mov z2.d, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    sxth z0.d, p0/m, z0.d
-; CHECK-NEXT:    sxth z1.d, p0/m, z1.d
-; CHECK-NEXT:    eor z0.d, z0.d, z2.d
-; CHECK-NEXT:    sub z0.d, z1.d, z0.d
-; CHECK-NEXT:    and z0.d, z0.d, #0xffffffff
-; CHECK-NEXT:    lsr z0.d, z0.d, #1
-; CHECK-NEXT:    ret
-entry:
-  %s0s = sext <vscale x 2 x i16> %s0 to <vscale x 2 x i32>
-  %s1s = sext <vscale x 2 x i16> %s1 to <vscale x 2 x i32>
-  %add = add <vscale x 2 x i32> %s0s, shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> poison, i32 1, i32 0), <vscale x 2 x i32> poison, <vscale x 2 x i32> zeroinitializer)
-  %add2 = add <vscale x 2 x i32> %add, %s1s
-  %s = lshr <vscale x 2 x i32> %add2, shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> poison, i32 1, i32 0), <vscale x 2 x i32> poison, <vscale x 2 x i32> zeroinitializer)
-  %result = trunc <vscale x 2 x i32> %s to <vscale x 2 x i16>
-  ret <vscale x 2 x i16> %result
-}
-
-define <vscale x 2 x i16> @rhaddu_v2i16(<vscale x 2 x i16> %s0, <vscale x 2 x i16> %s1) {
-; CHECK-LABEL: rhaddu_v2i16:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    mov z2.d, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    and z0.d, z0.d, #0xffff
-; CHECK-NEXT:    and z1.d, z1.d, #0xffff
-; CHECK-NEXT:    eor z0.d, z0.d, z2.d
-; CHECK-NEXT:    sub z0.d, z1.d, z0.d
-; CHECK-NEXT:    lsr z0.d, z0.d, #1
-; CHECK-NEXT:    ret
-entry:
-  %s0s = zext <vscale x 2 x i16> %s0 to <vscale x 2 x i32>
-  %s1s = zext <vscale x 2 x i16> %s1 to <vscale x 2 x i32>
-  %add = add nuw nsw <vscale x 2 x i32> %s0s, shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> poison, i32 1, i32 0), <vscale x 2 x i32> poison, <vscale x 2 x i32> zeroinitializer)
-  %add2 = add nuw nsw <vscale x 2 x i32> %add, %s1s
-  %s = lshr <vscale x 2 x i32> %add2, shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> poison, i32 1, i32 0), <vscale x 2 x i32> poison, <vscale x 2 x i32> zeroinitializer)
-  %result = trunc <vscale x 2 x i32> %s to <vscale x 2 x i16>
-  ret <vscale x 2 x i16> %result
-}
-
-define <vscale x 4 x i16> @rhadds_v4i16(<vscale x 4 x i16> %s0, <vscale x 4 x i16> %s1) {
-; CHECK-LABEL: rhadds_v4i16:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    mov z2.s, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    sxth z0.s, p0/m, z0.s
-; CHECK-NEXT:    sxth z1.s, p0/m, z1.s
-; CHECK-NEXT:    eor z0.d, z0.d, z2.d
-; CHECK-NEXT:    sub z0.s, z1.s, z0.s
-; CHECK-NEXT:    lsr z0.s, z0.s, #1
-; CHECK-NEXT:    ret
-entry:
-  %s0s = sext <vscale x 4 x i16> %s0 to <vscale x 4 x i32>
-  %s1s = sext <vscale x 4 x i16> %s1 to <vscale x 4 x i32>
-  %add = add <vscale x 4 x i32> %s0s, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 1, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
-  %add2 = add <vscale x 4 x i32> %add, %s1s
-  %s = lshr <vscale x 4 x i32> %add2, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 1, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
-  %result = trunc <vscale x 4 x i32> %s to <vscale x 4 x i16>
-  ret <vscale x 4 x i16> %result
-}
-
-define <vscale x 4 x i16> @rhaddu_v4i16(<vscale x 4 x i16> %s0, <vscale x 4 x i16> %s1) {
-; CHECK-LABEL: rhaddu_v4i16:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    and z0.s, z0.s, #0xffff
-; CHECK-NEXT:    and z1.s, z1.s, #0xffff
-; CHECK-NEXT:    urhadd z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT:    ret
-entry:
-  %s0s = zext <vscale x 4 x i16> %s0 to <vscale x 4 x i32>
-  %s1s = zext <vscale x 4 x i16> %s1 to <vscale x 4 x i32>
-  %add = add nuw nsw <vscale x 4 x i32> %s0s, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 1, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
-  %add2 = add nuw nsw <vscale x 4 x i32> %add, %s1s
-  %s = lshr <vscale x 4 x i32> %add2, shufflevector (<vscale x 4 x i32> insertelement (<vscale x 4 x i32> poison, i32 1, i32 0), <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer)
-  %result = trunc <vscale x 4 x i32> %s to <vscale x 4 x i16>
-  ret <vscale x 4 x i16> %result
-}
-
-define <vscale x 8 x i16> @rhadds_v8i16(<vscale x 8 x i16> %s0, <vscale x 8 x i16> %s1) {
-; CHECK-LABEL: rhadds_v8i16:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.h
-; CHECK-NEXT:    srhadd z0.h, p0/m, z0.h, z1.h
-; CHECK-NEXT:    ret
-entry:
-  %s0s = sext <vscale x 8 x i16> %s0 to <vscale x 8 x i32>
-  %s1s = sext <vscale x 8 x i16> %s1 to <vscale x 8 x i32>
-  %add = add <vscale x 8 x i32> %s0s, shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 1, i32 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer)
-  %add2 = add <vscale x 8 x i32> %add, %s1s
-  %s = lshr <vscale x 8 x i32> %add2, shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 1, i32 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer)
-  %result = trunc <vscale x 8 x i32> %s to <vscale x 8 x i16>
-  ret <vscale x 8 x i16> %result
-}
-
-define <vscale x 8 x i16> @rhaddu_v8i16(<vscale x 8 x i16> %s0, <vscale x 8 x i16> %s1) {
-; CHECK-LABEL: rhaddu_v8i16:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.h
-; CHECK-NEXT:    urhadd z0.h, p0/m, z0.h, z1.h
-; CHECK-NEXT:    ret
-entry:
-  %s0s = zext <vscale x 8 x i16> %s0 to <vscale x 8 x i32>
-  %s1s = zext <vscale x 8 x i16> %s1 to <vscale x 8 x i32>
-  %add = add nuw nsw <vscale x 8 x i32> %s0s, shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 1, i32 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer)
-  %add2 = add nuw nsw <vscale x 8 x i32> %add, %s1s
-  %s = lshr <vscale x 8 x i32> %add2, shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 1, i32 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer)
-  %result = trunc <vscale x 8 x i32> %s to <vscale x 8 x i16>
-  ret <vscale x 8 x i16> %result
-}
-
-define <vscale x 4 x i8> @rhadds_v4i8(<vscale x 4 x i8> %s0, <vscale x 4 x i8> %s1) {
-; CHECK-LABEL: rhadds_v4i8:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    mov z2.s, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    sxtb z0.s, p0/m, z0.s
-; CHECK-NEXT:    sxtb z1.s, p0/m, z1.s
-; CHECK-NEXT:    eor z0.d, z0.d, z2.d
-; CHECK-NEXT:    sub z0.s, z1.s, z0.s
-; CHECK-NEXT:    and z0.s, z0.s, #0xffff
-; CHECK-NEXT:    lsr z0.s, z0.s, #1
-; CHECK-NEXT:    ret
-entry:
-  %s0s = sext <vscale x 4 x i8> %s0 to <vscale x 4 x i16>
-  %s1s = sext <vscale x 4 x i8> %s1 to <vscale x 4 x i16>
-  %add = add <vscale x 4 x i16> %s0s, shufflevector (<vscale x 4 x i16> insertelement (<vscale x 4 x i16> poison, i16 1, i32 0), <vscale x 4 x i16> poison, <vscale x 4 x i32> zeroinitializer)
-  %add2 = add <vscale x 4 x i16> %add, %s1s
-  %s = lshr <vscale x 4 x i16> %add2, shufflevector (<vscale x 4 x i16> insertelement (<vscale x 4 x i16> poison, i16 1, i32 0), <vscale x 4 x i16> poison, <vscale x 4 x i32> zeroinitializer)
-  %result = trunc <vscale x 4 x i16> %s to <vscale x 4 x i8>
-  ret <vscale x 4 x i8> %result
-}
-
-define <vscale x 4 x i8> @rhaddu_v4i8(<vscale x 4 x i8> %s0, <vscale x 4 x i8> %s1) {
-; CHECK-LABEL: rhaddu_v4i8:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    mov z2.s, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    and z0.s, z0.s, #0xff
-; CHECK-NEXT:    and z1.s, z1.s, #0xff
-; CHECK-NEXT:    eor z0.d, z0.d, z2.d
-; CHECK-NEXT:    sub z0.s, z1.s, z0.s
-; CHECK-NEXT:    lsr z0.s, z0.s, #1
-; CHECK-NEXT:    ret
-entry:
-  %s0s = zext <vscale x 4 x i8> %s0 to <vscale x 4 x i16>
-  %s1s = zext <vscale x 4 x i8> %s1 to <vscale x 4 x i16>
-  %add = add nuw nsw <vscale x 4 x i16> %s0s, shufflevector (<vscale x 4 x i16> insertelement (<vscale x 4 x i16> poison, i16 1, i32 0), <vscale x 4 x i16> poison, <vscale x 4 x i32> zeroinitializer)
-  %add2 = add nuw nsw <vscale x 4 x i16> %add, %s1s
-  %s = lshr <vscale x 4 x i16> %add2, shufflevector (<vscale x 4 x i16> insertelement (<vscale x 4 x i16> poison, i16 1, i32 0), <vscale x 4 x i16> poison, <vscale x 4 x i32> zeroinitializer)
-  %result = trunc <vscale x 4 x i16> %s to <vscale x 4 x i8>
-  ret <vscale x 4 x i8> %result
-}
-
-define <vscale x 8 x i8> @rhadds_v8i8(<vscale x 8 x i8> %s0, <vscale x 8 x i8> %s1) {
-; CHECK-LABEL: rhadds_v8i8:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.h
-; CHECK-NEXT:    mov z2.h, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    sxtb z0.h, p0/m, z0.h
-; CHECK-NEXT:    sxtb z1.h, p0/m, z1.h
-; CHECK-NEXT:    eor z0.d, z0.d, z2.d
-; CHECK-NEXT:    sub z0.h, z1.h, z0.h
-; CHECK-NEXT:    lsr z0.h, z0.h, #1
-; CHECK-NEXT:    ret
-entry:
-  %s0s = sext <vscale x 8 x i8> %s0 to <vscale x 8 x i16>
-  %s1s = sext <vscale x 8 x i8> %s1 to <vscale x 8 x i16>
-  %add = add <vscale x 8 x i16> %s0s, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 1, i32 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
-  %add2 = add <vscale x 8 x i16> %add, %s1s
-  %s = lshr <vscale x 8 x i16> %add2, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 1, i32 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
-  %result = trunc <vscale x 8 x i16> %s to <vscale x 8 x i8>
-  ret <vscale x 8 x i8> %result
-}
-
-define <vscale x 8 x i8> @rhaddu_v8i8(<vscale x 8 x i8> %s0, <vscale x 8 x i8> %s1) {
-; CHECK-LABEL: rhaddu_v8i8:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.h
-; CHECK-NEXT:    and z0.h, z0.h, #0xff
-; CHECK-NEXT:    and z1.h, z1.h, #0xff
-; CHECK-NEXT:    urhadd z0.h, p0/m, z0.h, z1.h
-; CHECK-NEXT:    ret
-entry:
-  %s0s = zext <vscale x 8 x i8> %s0 to <vscale x 8 x i16>
-  %s1s = zext <vscale x 8 x i8> %s1 to <vscale x 8 x i16>
-  %add = add nuw nsw <vscale x 8 x i16> %s0s, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 1, i32 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
-  %add2 = add nuw nsw <vscale x 8 x i16> %add, %s1s
-  %s = lshr <vscale x 8 x i16> %add2, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 1, i32 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
-  %result = trunc <vscale x 8 x i16> %s to <vscale x 8 x i8>
-  ret <vscale x 8 x i8> %result
-}
-
-define <vscale x 16 x i8> @rhadds_v16i8(<vscale x 16 x i8> %s0, <vscale x 16 x i8> %s1) {
-; CHECK-LABEL: rhadds_v16i8:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.b
-; CHECK-NEXT:    srhadd z0.b, p0/m, z0.b, z1.b
-; CHECK-NEXT:    ret
-entry:
-  %s0s = sext <vscale x 16 x i8> %s0 to <vscale x 16 x i16>
-  %s1s = sext <vscale x 16 x i8> %s1 to <vscale x 16 x i16>
-  %add = add <vscale x 16 x i16> %s0s, shufflevector (<vscale x 16 x i16> insertelement (<vscale x 16 x i16> poison, i16 1, i32 0), <vscale x 16 x i16> poison, <vscale x 16 x i32> zeroinitializer)
-  %add2 = add <vscale x 16 x i16> %add, %s1s
-  %s = lshr <vscale x 16 x i16> %add2, shufflevector (<vscale x 16 x i16> insertelement (<vscale x 16 x i16> poison, i16 1, i32 0), <vscale x 16 x i16> poison, <vscale x 16 x i32> zeroinitializer)
-  %result = trunc <vscale x 16 x i16> %s to <vscale x 16 x i8>
-  ret <vscale x 16 x i8> %result
-}
-
-define <vscale x 16 x i8> @rhaddu_v16i8(<vscale x 16 x i8> %s0, <vscale x 16 x i8> %s1) {
-; CHECK-LABEL: rhaddu_v16i8:
-; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ptrue p0.b
-; CHECK-NEXT:    urhadd z0.b, p0/m, z0.b, z1.b
-; CHECK-NEXT:    ret
-entry:
-  %s0s = zext <vscale x 16 x i8> %s0 to <vscale x 16 x i16>
-  %s1s = zext <vscale x 16 x i8> %s1 to <vscale x 16 x i16>
-  %add = add nuw nsw <vscale x 16 x i16> %s0s, shufflevector (<vscale x 16 x i16> insertelement (<vscale x 16 x i16> poison, i16 1, i32 0), <vscale x 16 x i16> poison, <vscale x 16 x i32> zeroinitializer)
-  %add2 = add nuw nsw <vscale x 16 x i16> %add, %s1s
-  %s = lshr <vscale x 16 x i16> %add2, shufflevector (<vscale x 16 x i16> insertelement (<vscale x 16 x i16> poison, i16 1, i32 0), <vscale x 16 x i16> poison, <vscale x 16 x i32> zeroinitializer)
-  %result = trunc <vscale x 16 x i16> %s to <vscale x 16 x i8>
-  ret <vscale x 16 x i8> %result
-}
diff --git a/llvm/test/CodeGen/AArch64/vecreduce-add-legalization.ll b/llvm/test/CodeGen/AArch64/vecreduce-add-legalization.ll
index 94aea27b38e1b..4d847fb96cbb7 100644
--- a/llvm/test/CodeGen/AArch64/vecreduce-add-legalization.ll
+++ b/llvm/test/CodeGen/AArch64/vecreduce-add-legalization.ll
@@ -99,13 +99,9 @@ define i8 @test_v3i8(<3 x i8> %a) nounwind {
 define i8 @test_v9i8(<9 x i8> %a) nounwind {
 ; CHECK-LABEL: test_v9i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov v0.b[9], wzr
-; CHECK-NEXT:    mov v0.b[10], wzr
-; CHECK-NEXT:    mov v0.b[11], wzr
-; CHECK-NEXT:    mov v0.b[12], wzr
-; CHECK-NEXT:    mov v0.b[13], wzr
-; CHECK-NEXT:    mov v0.b[14], wzr
-; CHECK-NEXT:    mov v0.b[15], wzr
+; CHECK-NEXT:    adrp x8, .LCPI8_0
+; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI8_0]
+; CHECK-NEXT:    and v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    addv b0, v0.16b
 ; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
diff --git a/llvm/test/CodeGen/AArch64/vecreduce-fadd.ll b/llvm/test/CodeGen/AArch64/vecreduce-fadd.ll
index c2c436ca41a6f..55b140b28bd49 100644
--- a/llvm/test/CodeGen/AArch64/vecreduce-fadd.ll
+++ b/llvm/test/CodeGen/AArch64/vecreduce-fadd.ll
@@ -236,7 +236,7 @@ define float @add_S_init_42(<4 x float> %bin.rdx)  {
   ret float %r
 }
 
-; FIXME: The faddp.4s in the loop should not use v0.4s as second operand,
+; The faddp.4s in the loop should not use v0.4s as second operand,
 ; because this introduces an unnecessary cross-iteration dependency.
 define float @fadd_reduction_v4f32_in_loop(ptr %ptr.start) {
 ; CHECK-LABEL: fadd_reduction_v4f32_in_loop:
@@ -273,7 +273,7 @@ exit:
   ret float %red.next
 }
 
-; FIXME: The faddp.4h in the loop should not use v0.4h as second operand,
+; The faddp.4h in the loop should not use v0.4h as second operand,
 ; because this introduces an unnecessary cross-iteration dependency.
 define half @fadd_reduction_v4f16_in_loop(ptr %ptr.start) {
 ; FULLFP16-LABEL: fadd_reduction_v4f16_in_loop:
@@ -342,7 +342,7 @@ exit:
   ret half %red.next
 }
 
-; FIXME: The faddp.8h in the loop should not use v0.8h as second operand,
+; The faddp.8h in the loop should not use v0.8h as second operand,
 ; because this introduces an unnecessary cross-iteration dependency.
 define half @fadd_reduction_v8f16_in_loop(ptr %ptr.start) {
 ; FULLFP16-LABEL: fadd_reduction_v8f16_in_loop:
diff --git a/llvm/test/CodeGen/AArch64/vecreduce-umax-legalization.ll b/llvm/test/CodeGen/AArch64/vecreduce-umax-legalization.ll
index b80d6f922e571..a14f12a407376 100644
--- a/llvm/test/CodeGen/AArch64/vecreduce-umax-legalization.ll
+++ b/llvm/test/CodeGen/AArch64/vecreduce-umax-legalization.ll
@@ -114,13 +114,9 @@ define i8 @test_v3i8(<3 x i8> %a) nounwind {
 define i8 @test_v9i8(<9 x i8> %a) nounwind {
 ; CHECK-LABEL: test_v9i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov v0.b[9], wzr
-; CHECK-NEXT:    mov v0.b[10], wzr
-; CHECK-NEXT:    mov v0.b[11], wzr
-; CHECK-NEXT:    mov v0.b[12], wzr
-; CHECK-NEXT:    mov v0.b[13], wzr
-; CHECK-NEXT:    mov v0.b[14], wzr
-; CHECK-NEXT:    mov v0.b[15], wzr
+; CHECK-NEXT:    adrp x8, .LCPI9_0
+; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI9_0]
+; CHECK-NEXT:    and v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    umaxv b0, v0.16b
 ; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/call-outgoing-stack-args.ll b/llvm/test/CodeGen/AMDGPU/GlobalISel/call-outgoing-stack-args.ll
index 28a7b1a62a708..ff4515e371434 100644
--- a/llvm/test/CodeGen/AMDGPU/GlobalISel/call-outgoing-stack-args.ll
+++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/call-outgoing-stack-args.ll
@@ -33,20 +33,24 @@ define amdgpu_kernel void @kernel_caller_stack() {
 ;
 ; FLATSCR-LABEL: kernel_caller_stack:
 ; FLATSCR:       ; %bb.0:
-; FLATSCR-NEXT:    s_add_u32 flat_scratch_lo, s0, s3
 ; FLATSCR-NEXT:    s_mov_b32 s32, 0
+; FLATSCR-NEXT:    s_add_u32 flat_scratch_lo, s0, s3
 ; FLATSCR-NEXT:    s_addc_u32 flat_scratch_hi, s1, 0
+; FLATSCR-NEXT:    s_add_u32 s0, s32, 4
 ; FLATSCR-NEXT:    v_mov_b32_e32 v0, 9
-; FLATSCR-NEXT:    scratch_store_dword off, v0, s32 offset:4
+; FLATSCR-NEXT:    scratch_store_dword off, v0, s0
+; FLATSCR-NEXT:    s_add_u32 s0, s32, 8
 ; FLATSCR-NEXT:    v_mov_b32_e32 v0, 10
-; FLATSCR-NEXT:    scratch_store_dword off, v0, s32 offset:8
+; FLATSCR-NEXT:    scratch_store_dword off, v0, s0
+; FLATSCR-NEXT:    s_add_u32 s0, s32, 12
 ; FLATSCR-NEXT:    v_mov_b32_e32 v0, 11
-; FLATSCR-NEXT:    scratch_store_dword off, v0, s32 offset:12
+; FLATSCR-NEXT:    scratch_store_dword off, v0, s0
+; FLATSCR-NEXT:    s_add_u32 s2, s32, 16
 ; FLATSCR-NEXT:    v_mov_b32_e32 v0, 12
 ; FLATSCR-NEXT:    s_getpc_b64 s[0:1]
 ; FLATSCR-NEXT:    s_add_u32 s0, s0, external_void_func_v16i32_v16i32_v4i32@rel32@lo+4
 ; FLATSCR-NEXT:    s_addc_u32 s1, s1, external_void_func_v16i32_v16i32_v4i32@rel32@hi+12
-; FLATSCR-NEXT:    scratch_store_dword off, v0, s32 offset:16
+; FLATSCR-NEXT:    scratch_store_dword off, v0, s2
 ; FLATSCR-NEXT:    s_swappc_b64 s[30:31], s[0:1]
 ; FLATSCR-NEXT:    s_endpgm
   call void @external_void_func_v16i32_v16i32_v4i32(<16 x i32> undef, <16 x i32> undef, <4 x i32> <i32 9, i32 10, i32 11, i32 12>)
@@ -155,42 +159,40 @@ define amdgpu_kernel void @kernel_caller_byval() {
 ; FLATSCR-NEXT:    v_mov_b32_e32 v0, 0
 ; FLATSCR-NEXT:    s_addc_u32 flat_scratch_hi, s1, 0
 ; FLATSCR-NEXT:    v_mov_b32_e32 v1, 0
+; FLATSCR-NEXT:    s_mov_b32 s8, 0
+; FLATSCR-NEXT:    s_mov_b32 s10, 0
+; FLATSCR-NEXT:    s_mov_b32 s9, 0
 ; FLATSCR-NEXT:    s_mov_b32 vcc_lo, 0
 ; FLATSCR-NEXT:    s_mov_b32 vcc_hi, 0
 ; FLATSCR-NEXT:    s_mov_b32 s11, 0
-; FLATSCR-NEXT:    s_mov_b32 s10, 0
-; FLATSCR-NEXT:    s_mov_b32 s9, 0
-; FLATSCR-NEXT:    s_mov_b32 s8, 0
 ; FLATSCR-NEXT:    s_mov_b32 s7, 0
-; FLATSCR-NEXT:    s_mov_b32 s6, 0
 ; FLATSCR-NEXT:    s_mov_b32 s5, 0
+; FLATSCR-NEXT:    s_mov_b32 s3, 0
 ; FLATSCR-NEXT:    s_mov_b32 s1, 0
 ; FLATSCR-NEXT:    s_mov_b32 s0, 0
-; FLATSCR-NEXT:    s_mov_b32 s4, 0
-; FLATSCR-NEXT:    s_mov_b32 s3, 0
 ; FLATSCR-NEXT:    s_mov_b32 s2, 0
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], vcc_lo offset:8
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], vcc_hi offset:16
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s11 offset:24
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s10 offset:32
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s9 offset:40
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s8 offset:48
+; FLATSCR-NEXT:    s_mov_b32 s4, 0
+; FLATSCR-NEXT:    s_mov_b32 s6, 0
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s8 offset:8
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s10 offset:16
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s9 offset:24
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], vcc_lo offset:32
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], vcc_hi offset:40
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s11 offset:48
 ; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s7 offset:56
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s6 offset:64
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s5 offset:72
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s5 offset:64
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s3 offset:72
 ; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s1 offset:80
 ; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s0 offset:88
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s4 offset:96
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s3 offset:104
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s2 offset:112
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], vcc_lo offset:120
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], vcc_hi offset:128
-; FLATSCR-NEXT:    s_mov_b32 s40, 0
-; FLATSCR-NEXT:    scratch_load_dwordx2 v[0:1], off, s40 offset:8
-; FLATSCR-NEXT:    s_mov_b32 s39, 0
-; FLATSCR-NEXT:    scratch_load_dwordx2 v[2:3], off, s39 offset:16
-; FLATSCR-NEXT:    s_mov_b32 s38, 0
-; FLATSCR-NEXT:    scratch_load_dwordx2 v[4:5], off, s38 offset:24
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s2 offset:96
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s4 offset:104
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s6 offset:112
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s8 offset:120
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s10 offset:128
+; FLATSCR-NEXT:    scratch_load_dwordx2 v[0:1], off, s9 offset:8
+; FLATSCR-NEXT:    s_nop 0
+; FLATSCR-NEXT:    scratch_load_dwordx2 v[2:3], off, vcc_lo offset:16
+; FLATSCR-NEXT:    scratch_load_dwordx2 v[4:5], off, vcc_hi offset:24
 ; FLATSCR-NEXT:    s_mov_b32 s37, 0
 ; FLATSCR-NEXT:    scratch_load_dwordx2 v[6:7], off, s37 offset:32
 ; FLATSCR-NEXT:    s_mov_b32 s36, 0
@@ -205,22 +207,29 @@ define amdgpu_kernel void @kernel_caller_byval() {
 ; FLATSCR-NEXT:    s_getpc_b64 s[0:1]
 ; FLATSCR-NEXT:    s_add_u32 s0, s0, external_void_func_byval@rel32@lo+4
 ; FLATSCR-NEXT:    s_addc_u32 s1, s1, external_void_func_byval@rel32@hi+12
+; FLATSCR-NEXT:    s_add_u32 s2, s32, 8
+; FLATSCR-NEXT:    s_add_u32 s3, s32, 16
+; FLATSCR-NEXT:    s_add_u32 s4, s32, 24
+; FLATSCR-NEXT:    s_add_u32 s5, s32, 32
+; FLATSCR-NEXT:    s_add_u32 s6, s32, 40
+; FLATSCR-NEXT:    s_add_u32 s7, s32, 48
+; FLATSCR-NEXT:    s_add_u32 s8, s32, 56
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(7)
 ; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s32
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(7)
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[2:3], s32 offset:8
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[2:3], s2
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(7)
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[4:5], s32 offset:16
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[4:5], s3
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(7)
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[6:7], s32 offset:24
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[6:7], s4
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(7)
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[8:9], s32 offset:32
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[8:9], s5
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(7)
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[10:11], s32 offset:40
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[10:11], s6
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(7)
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[12:13], s32 offset:48
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[12:13], s7
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(7)
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[14:15], s32 offset:56
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[14:15], s8
 ; FLATSCR-NEXT:    s_swappc_b64 s[30:31], s[0:1]
 ; FLATSCR-NEXT:    s_endpgm
   %alloca = alloca [16 x i32], align 4, addrspace(5)
@@ -277,17 +286,21 @@ define void @func_caller_stack() {
 ; FLATSCR-NEXT:    scratch_store_dword off, v41, s33 offset:4 ; 4-byte Folded Spill
 ; FLATSCR-NEXT:    s_mov_b64 exec, s[2:3]
 ; FLATSCR-NEXT:    s_add_i32 s32, s32, 16
+; FLATSCR-NEXT:    v_writelane_b32 v41, s0, 0
+; FLATSCR-NEXT:    s_add_u32 s0, s32, 4
 ; FLATSCR-NEXT:    v_mov_b32_e32 v0, 9
-; FLATSCR-NEXT:    scratch_store_dword off, v0, s32 offset:4
+; FLATSCR-NEXT:    scratch_store_dword off, v0, s0
+; FLATSCR-NEXT:    s_add_u32 s0, s32, 8
 ; FLATSCR-NEXT:    v_mov_b32_e32 v0, 10
-; FLATSCR-NEXT:    scratch_store_dword off, v0, s32 offset:8
+; FLATSCR-NEXT:    scratch_store_dword off, v0, s0
+; FLATSCR-NEXT:    s_add_u32 s0, s32, 12
 ; FLATSCR-NEXT:    v_mov_b32_e32 v0, 11
 ; FLATSCR-NEXT:    v_writelane_b32 v40, s30, 0
-; FLATSCR-NEXT:    scratch_store_dword off, v0, s32 offset:12
+; FLATSCR-NEXT:    scratch_store_dword off, v0, s0
+; FLATSCR-NEXT:    s_add_u32 s0, s32, 16
 ; FLATSCR-NEXT:    v_mov_b32_e32 v0, 12
-; FLATSCR-NEXT:    v_writelane_b32 v41, s0, 0
 ; FLATSCR-NEXT:    v_writelane_b32 v40, s31, 1
-; FLATSCR-NEXT:    scratch_store_dword off, v0, s32 offset:16
+; FLATSCR-NEXT:    scratch_store_dword off, v0, s0
 ; FLATSCR-NEXT:    s_getpc_b64 s[0:1]
 ; FLATSCR-NEXT:    s_add_u32 s0, s0, external_void_func_v16i32_v16i32_v4i32@rel32@lo+4
 ; FLATSCR-NEXT:    s_addc_u32 s1, s1, external_void_func_v16i32_v16i32_v4i32@rel32@hi+12
@@ -403,35 +416,49 @@ define void @func_caller_byval(ptr addrspace(5) %argptr) {
 ; FLATSCR-NEXT:    s_mov_b64 exec, s[2:3]
 ; FLATSCR-NEXT:    scratch_load_dwordx2 v[1:2], v0, off
 ; FLATSCR-NEXT:    s_add_i32 s32, s32, 16
-; FLATSCR-NEXT:    v_writelane_b32 v40, s30, 0
+; FLATSCR-NEXT:    v_add_u32_e32 v3, 8, v0
 ; FLATSCR-NEXT:    v_writelane_b32 v41, s0, 0
+; FLATSCR-NEXT:    s_add_u32 s0, s32, 8
+; FLATSCR-NEXT:    v_writelane_b32 v40, s30, 0
+; FLATSCR-NEXT:    s_add_u32 s2, s32, 56
 ; FLATSCR-NEXT:    v_writelane_b32 v40, s31, 1
-; FLATSCR-NEXT:    s_getpc_b64 s[0:1]
-; FLATSCR-NEXT:    s_add_u32 s0, s0, external_void_func_byval@rel32@lo+4
-; FLATSCR-NEXT:    s_addc_u32 s1, s1, external_void_func_byval@rel32@hi+12
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
 ; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[1:2], s32
-; FLATSCR-NEXT:    scratch_load_dwordx2 v[1:2], v0, off offset:8
+; FLATSCR-NEXT:    scratch_load_dwordx2 v[1:2], v3, off
+; FLATSCR-NEXT:    v_add_u32_e32 v3, 16, v0
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[1:2], s32 offset:8
-; FLATSCR-NEXT:    scratch_load_dwordx2 v[1:2], v0, off offset:16
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[1:2], s0
+; FLATSCR-NEXT:    scratch_load_dwordx2 v[1:2], v3, off
+; FLATSCR-NEXT:    s_add_u32 s0, s32, 16
+; FLATSCR-NEXT:    v_add_u32_e32 v3, 24, v0
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[1:2], s32 offset:16
-; FLATSCR-NEXT:    scratch_load_dwordx2 v[1:2], v0, off offset:24
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[1:2], s0
+; FLATSCR-NEXT:    scratch_load_dwordx2 v[1:2], v3, off
+; FLATSCR-NEXT:    s_add_u32 s0, s32, 24
+; FLATSCR-NEXT:    v_add_u32_e32 v3, 32, v0
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[1:2], s32 offset:24
-; FLATSCR-NEXT:    scratch_load_dwordx2 v[1:2], v0, off offset:32
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[1:2], s0
+; FLATSCR-NEXT:    scratch_load_dwordx2 v[1:2], v3, off
+; FLATSCR-NEXT:    s_add_u32 s0, s32, 32
+; FLATSCR-NEXT:    v_add_u32_e32 v3, 40, v0
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[1:2], s32 offset:32
-; FLATSCR-NEXT:    scratch_load_dwordx2 v[1:2], v0, off offset:40
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[1:2], s0
+; FLATSCR-NEXT:    scratch_load_dwordx2 v[1:2], v3, off
+; FLATSCR-NEXT:    s_add_u32 s0, s32, 40
+; FLATSCR-NEXT:    v_add_u32_e32 v3, 48, v0
+; FLATSCR-NEXT:    v_add_u32_e32 v0, 56, v0
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[1:2], s32 offset:40
-; FLATSCR-NEXT:    scratch_load_dwordx2 v[1:2], v0, off offset:48
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[1:2], s0
+; FLATSCR-NEXT:    scratch_load_dwordx2 v[1:2], v3, off
+; FLATSCR-NEXT:    s_add_u32 s0, s32, 48
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[1:2], s32 offset:48
-; FLATSCR-NEXT:    scratch_load_dwordx2 v[0:1], v0, off offset:56
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[1:2], s0
+; FLATSCR-NEXT:    scratch_load_dwordx2 v[0:1], v0, off
+; FLATSCR-NEXT:    s_getpc_b64 s[0:1]
+; FLATSCR-NEXT:    s_add_u32 s0, s0, external_void_func_byval@rel32@lo+4
+; FLATSCR-NEXT:    s_addc_u32 s1, s1, external_void_func_byval@rel32@hi+12
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s32 offset:56
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s2
 ; FLATSCR-NEXT:    s_swappc_b64 s[30:31], s[0:1]
 ; FLATSCR-NEXT:    v_readlane_b32 s31, v40, 1
 ; FLATSCR-NEXT:    v_readlane_b32 s30, v40, 0
diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.ll b/llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.ll
index 8e558b539fa72..e639fce9d690e 100644
--- a/llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.ll
+++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.ll
@@ -3061,7 +3061,7 @@ define amdgpu_kernel void @dyn_extract_v5f64_s_s(ptr addrspace(1) %out, i32 %sel
 ; GPRIDX-NEXT:     gds_segment_byte_size = 0
 ; GPRIDX-NEXT:     kernarg_segment_byte_size = 12
 ; GPRIDX-NEXT:     workgroup_fbarrier_count = 0
-; GPRIDX-NEXT:     wavefront_sgpr_count = 9
+; GPRIDX-NEXT:     wavefront_sgpr_count = 13
 ; GPRIDX-NEXT:     workitem_vgpr_count = 3
 ; GPRIDX-NEXT:     reserved_vgpr_first = 0
 ; GPRIDX-NEXT:     reserved_vgpr_count = 0
@@ -3913,7 +3913,7 @@ define amdgpu_kernel void @dyn_extract_v4f32_s_s_s(ptr addrspace(1) %out, i32 %s
 ; GPRIDX-NEXT:     kernel_code_entry_byte_offset = 256
 ; GPRIDX-NEXT:     kernel_code_prefetch_byte_size = 0
 ; GPRIDX-NEXT:     granulated_workitem_vgpr_count = 0
-; GPRIDX-NEXT:     granulated_wavefront_sgpr_count = 0
+; GPRIDX-NEXT:     granulated_wavefront_sgpr_count = 1
 ; GPRIDX-NEXT:     priority = 0
 ; GPRIDX-NEXT:     float_mode = 240
 ; GPRIDX-NEXT:     priv = 0
@@ -3956,7 +3956,7 @@ define amdgpu_kernel void @dyn_extract_v4f32_s_s_s(ptr addrspace(1) %out, i32 %s
 ; GPRIDX-NEXT:     gds_segment_byte_size = 0
 ; GPRIDX-NEXT:     kernarg_segment_byte_size = 12
 ; GPRIDX-NEXT:     workgroup_fbarrier_count = 0
-; GPRIDX-NEXT:     wavefront_sgpr_count = 6
+; GPRIDX-NEXT:     wavefront_sgpr_count = 10
 ; GPRIDX-NEXT:     workitem_vgpr_count = 2
 ; GPRIDX-NEXT:     reserved_vgpr_first = 0
 ; GPRIDX-NEXT:     reserved_vgpr_count = 0
@@ -4259,7 +4259,7 @@ define amdgpu_kernel void @dyn_extract_v4f64_s_s_s(ptr addrspace(1) %out, i32 %s
 ; GPRIDX-NEXT:     kernel_code_entry_byte_offset = 256
 ; GPRIDX-NEXT:     kernel_code_prefetch_byte_size = 0
 ; GPRIDX-NEXT:     granulated_workitem_vgpr_count = 0
-; GPRIDX-NEXT:     granulated_wavefront_sgpr_count = 0
+; GPRIDX-NEXT:     granulated_wavefront_sgpr_count = 1
 ; GPRIDX-NEXT:     priority = 0
 ; GPRIDX-NEXT:     float_mode = 240
 ; GPRIDX-NEXT:     priv = 0
@@ -4302,7 +4302,7 @@ define amdgpu_kernel void @dyn_extract_v4f64_s_s_s(ptr addrspace(1) %out, i32 %s
 ; GPRIDX-NEXT:     gds_segment_byte_size = 0
 ; GPRIDX-NEXT:     kernarg_segment_byte_size = 12
 ; GPRIDX-NEXT:     workgroup_fbarrier_count = 0
-; GPRIDX-NEXT:     wavefront_sgpr_count = 7
+; GPRIDX-NEXT:     wavefront_sgpr_count = 11
 ; GPRIDX-NEXT:     workitem_vgpr_count = 3
 ; GPRIDX-NEXT:     reserved_vgpr_first = 0
 ; GPRIDX-NEXT:     reserved_vgpr_count = 0
diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/flat-scratch.ll b/llvm/test/CodeGen/AMDGPU/GlobalISel/flat-scratch.ll
index 1369b02d87fe1..4130e6bc932bf 100644
--- a/llvm/test/CodeGen/AMDGPU/GlobalISel/flat-scratch.ll
+++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/flat-scratch.ll
@@ -50,9 +50,9 @@ define amdgpu_kernel void @store_load_sindex_kernel(i32 %idx) {
 ; GFX940-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-NEXT:    s_lshl_b32 s1, s0, 2
 ; GFX940-NEXT:    s_and_b32 s0, s0, 15
-; GFX940-NEXT:    v_mov_b32_e32 v1, s1
+; GFX940-NEXT:    s_add_i32 s1, s1, 4
 ; GFX940-NEXT:    s_lshl_b32 s0, s0, 2
-; GFX940-NEXT:    scratch_store_dword v1, v0, off offset:4 sc0 sc1
+; GFX940-NEXT:    scratch_store_dword off, v0, s1 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX940-NEXT:    scratch_load_dword v0, v0, off offset:4 sc0 sc1
@@ -62,17 +62,16 @@ define amdgpu_kernel void @store_load_sindex_kernel(i32 %idx) {
 ; GFX11-LABEL: store_load_sindex_kernel:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    s_load_b32 s0, s[0:1], 0x24
-; GFX11-NEXT:    v_mov_b32_e32 v1, 15
 ; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-NEXT:    s_lshl_b32 s1, s0, 2
-; GFX11-NEXT:    s_and_b32 s0, s0, 15
-; GFX11-NEXT:    v_mov_b32_e32 v0, s1
+; GFX11-NEXT:    s_and_b32 s1, s0, 15
 ; GFX11-NEXT:    s_lshl_b32 s0, s0, 2
+; GFX11-NEXT:    s_lshl_b32 s1, s1, 2
 ; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11-NEXT:    v_mov_b32_e32 v2, s0
-; GFX11-NEXT:    scratch_store_b32 v0, v1, off offset:4 dlc
+; GFX11-NEXT:    v_dual_mov_b32 v0, 15 :: v_dual_mov_b32 v1, s1
+; GFX11-NEXT:    s_add_i32 s0, s0, 4
+; GFX11-NEXT:    scratch_store_b32 off, v0, s0 dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_load_b32 v0, v2, off offset:4 glc dlc
+; GFX11-NEXT:    scratch_load_b32 v0, v1, off offset:4 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_endpgm
 bb:
@@ -90,15 +89,17 @@ define amdgpu_kernel void @store_load_vindex_kernel() {
 ; GFX9:       ; %bb.0: ; %bb
 ; GFX9-NEXT:    s_add_u32 flat_scratch_lo, s0, s3
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v1, 2, v0
-; GFX9-NEXT:    v_sub_u32_e32 v0, 0, v0
 ; GFX9-NEXT:    s_addc_u32 flat_scratch_hi, s1, 0
 ; GFX9-NEXT:    v_add_u32_e32 v1, 4, v1
-; GFX9-NEXT:    v_mov_b32_e32 v2, 15
-; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX9-NEXT:    scratch_store_dword v1, v2, off
+; GFX9-NEXT:    v_mov_b32_e32 v3, 15
+; GFX9-NEXT:    v_sub_u32_e32 v0, 0, v0
+; GFX9-NEXT:    v_mov_b32_e32 v2, 4
+; GFX9-NEXT:    scratch_store_dword v1, v3, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_add_u32_e32 v0, 4, v0
-; GFX9-NEXT:    scratch_load_dword v0, v0, off offset:124 glc
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX9-NEXT:    v_mov_b32_e32 v1, 0x7c
+; GFX9-NEXT:    v_add3_u32 v0, v2, v0, v1
+; GFX9-NEXT:    scratch_load_dword v0, v0, off glc
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_endpgm
 ;
@@ -110,37 +111,42 @@ define amdgpu_kernel void @store_load_vindex_kernel() {
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
 ; GFX10-NEXT:    v_sub_nc_u32_e32 v1, 0, v0
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX10-NEXT:    v_mov_b32_e32 v2, 15
+; GFX10-NEXT:    v_mov_b32_e32 v2, 0x7c
+; GFX10-NEXT:    v_mov_b32_e32 v3, 15
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v1, 2, v1
 ; GFX10-NEXT:    v_add_nc_u32_e32 v0, 4, v0
-; GFX10-NEXT:    v_add_nc_u32_e32 v1, 4, v1
-; GFX10-NEXT:    scratch_store_dword v0, v2, off
+; GFX10-NEXT:    v_add3_u32 v1, 4, v1, v2
+; GFX10-NEXT:    scratch_store_dword v0, v3, off
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-NEXT:    scratch_load_dword v0, v1, off offset:124 glc dlc
+; GFX10-NEXT:    scratch_load_dword v0, v1, off glc dlc
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX940-LABEL: store_load_vindex_kernel:
 ; GFX940:       ; %bb.0: ; %bb
 ; GFX940-NEXT:    v_lshlrev_b32_e32 v1, 2, v0
-; GFX940-NEXT:    v_mov_b32_e32 v2, 15
+; GFX940-NEXT:    v_mov_b32_e32 v3, 15
 ; GFX940-NEXT:    v_sub_u32_e32 v0, 0, v0
-; GFX940-NEXT:    scratch_store_dword v1, v2, off offset:4 sc0 sc1
+; GFX940-NEXT:    v_mov_b32_e32 v2, 4
+; GFX940-NEXT:    scratch_store_dword v1, v3, off offset:4 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX940-NEXT:    scratch_load_dword v0, v0, off offset:128 sc0 sc1
+; GFX940-NEXT:    v_mov_b32_e32 v1, 0x7c
+; GFX940-NEXT:    v_add3_u32 v0, v2, v0, v1
+; GFX940-NEXT:    scratch_load_dword v0, v0, off sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: store_load_vindex_kernel:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    v_sub_nc_u32_e32 v1, 0, v0
-; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX11-NEXT:    v_dual_mov_b32 v3, 15 :: v_dual_lshlrev_b32 v0, 2, v0
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2)
-; GFX11-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_lshlrev_b32 v1, 2, v1
-; GFX11-NEXT:    scratch_store_b32 v0, v2, off offset:4 dlc
+; GFX11-NEXT:    v_dual_mov_b32 v2, 0x7c :: v_dual_lshlrev_b32 v1, 2, v1
+; GFX11-NEXT:    scratch_store_b32 v0, v3, off offset:4 dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_load_b32 v0, v1, off offset:128 glc dlc
+; GFX11-NEXT:    v_add3_u32 v1, 4, v1, v2
+; GFX11-NEXT:    scratch_load_b32 v0, v1, off glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_endpgm
 bb:
@@ -191,9 +197,10 @@ define void @store_load_vindex_foo(i32 %idx) {
 ; GFX940:       ; %bb.0: ; %bb
 ; GFX940-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX940-NEXT:    v_lshlrev_b32_e32 v1, 2, v0
+; GFX940-NEXT:    v_add_u32_e32 v1, s32, v1
 ; GFX940-NEXT:    v_mov_b32_e32 v2, 15
 ; GFX940-NEXT:    v_and_b32_e32 v0, 15, v0
-; GFX940-NEXT:    scratch_store_dword v1, v2, s32 sc0 sc1
+; GFX940-NEXT:    scratch_store_dword v1, v2, off sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX940-NEXT:    scratch_load_dword v0, v0, s32 sc0 sc1
@@ -204,13 +211,14 @@ define void @store_load_vindex_foo(i32 %idx) {
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_and_b32 v1, 15, v0
+; GFX11-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_lshlrev_b32 v1, 2, v0
+; GFX11-NEXT:    v_and_b32_e32 v0, 15, v0
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_add_nc_u32_e32 v1, s32, v1
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2)
-; GFX11-NEXT:    v_lshlrev_b32_e32 v1, 2, v1
-; GFX11-NEXT:    scratch_store_b32 v0, v2, s32 dlc
+; GFX11-NEXT:    scratch_store_b32 v1, v2, off dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_load_b32 v0, v1, s32 glc dlc
+; GFX11-NEXT:    scratch_load_b32 v0, v0, s32 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_setpc_b64 s[30:31]
 bb:
@@ -227,8 +235,9 @@ define void @private_ptr_foo(ptr addrspace(5) nocapture %arg) {
 ; GFX9-LABEL: private_ptr_foo:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0x41200000
-; GFX9-NEXT:    scratch_store_dword v0, v1, off offset:4
+; GFX9-NEXT:    scratch_store_dword v0, v1, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
 ;
@@ -236,16 +245,18 @@ define void @private_ptr_foo(ptr addrspace(5) nocapture %arg) {
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, 4, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v1, 0x41200000
-; GFX10-NEXT:    scratch_store_dword v0, v1, off offset:4
+; GFX10-NEXT:    scratch_store_dword v0, v1, off
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX940-LABEL: private_ptr_foo:
 ; GFX940:       ; %bb.0:
 ; GFX940-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX940-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX940-NEXT:    v_mov_b32_e32 v1, 0x41200000
-; GFX940-NEXT:    scratch_store_dword v0, v1, off offset:4
+; GFX940-NEXT:    scratch_store_dword v0, v1, off
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    s_setpc_b64 s[30:31]
 ;
@@ -253,8 +264,8 @@ define void @private_ptr_foo(ptr addrspace(5) nocapture %arg) {
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    v_mov_b32_e32 v1, 0x41200000
-; GFX11-NEXT:    scratch_store_b32 v0, v1, off offset:4
+; GFX11-NEXT:    v_dual_mov_b32 v1, 0x41200000 :: v_dual_add_nc_u32 v0, 4, v0
+; GFX11-NEXT:    scratch_store_b32 v0, v1, off
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    s_setpc_b64 s[30:31]
   %gep = getelementptr inbounds float, ptr addrspace(5) %arg, i32 1
@@ -314,9 +325,9 @@ define amdgpu_kernel void @store_load_sindex_small_offset_kernel(i32 %idx) {
 ; GFX940-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-NEXT:    s_lshl_b32 s1, s0, 2
 ; GFX940-NEXT:    s_and_b32 s0, s0, 15
-; GFX940-NEXT:    v_mov_b32_e32 v1, s1
+; GFX940-NEXT:    s_addk_i32 s1, 0x104
 ; GFX940-NEXT:    s_lshl_b32 s0, s0, 2
-; GFX940-NEXT:    scratch_store_dword v1, v0, off offset:260 sc0 sc1
+; GFX940-NEXT:    scratch_store_dword off, v0, s1 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX940-NEXT:    scratch_load_dword v0, v0, off offset:260 sc0 sc1
@@ -326,19 +337,17 @@ define amdgpu_kernel void @store_load_sindex_small_offset_kernel(i32 %idx) {
 ; GFX11-LABEL: store_load_sindex_small_offset_kernel:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    s_load_b32 s0, s[0:1], 0x24
-; GFX11-NEXT:    scratch_load_b32 v3, off, off offset:4 glc dlc
-; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_mov_b32_e32 v1, 15
-; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-NEXT:    s_lshl_b32 s1, s0, 2
-; GFX11-NEXT:    s_and_b32 s0, s0, 15
-; GFX11-NEXT:    v_mov_b32_e32 v0, s1
+; GFX11-NEXT:    scratch_load_b32 v2, off, off offset:4 glc dlc
+; GFX11-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX11-NEXT:    s_and_b32 s1, s0, 15
 ; GFX11-NEXT:    s_lshl_b32 s0, s0, 2
+; GFX11-NEXT:    s_lshl_b32 s1, s1, 2
 ; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11-NEXT:    v_mov_b32_e32 v2, s0
-; GFX11-NEXT:    scratch_store_b32 v0, v1, off offset:260 dlc
+; GFX11-NEXT:    v_dual_mov_b32 v0, 15 :: v_dual_mov_b32 v1, s1
+; GFX11-NEXT:    s_addk_i32 s0, 0x104
+; GFX11-NEXT:    scratch_store_b32 off, v0, s0 dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_load_b32 v0, v2, off offset:260 glc dlc
+; GFX11-NEXT:    scratch_load_b32 v0, v1, off offset:260 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_endpgm
 bb:
@@ -363,14 +372,16 @@ define amdgpu_kernel void @store_load_vindex_small_offset_kernel() {
 ; GFX9-NEXT:    scratch_load_dword v1, off, vcc_hi offset:4 glc
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v1, 2, v0
-; GFX9-NEXT:    v_sub_u32_e32 v0, 0, v0
 ; GFX9-NEXT:    v_add_u32_e32 v1, 0x104, v1
-; GFX9-NEXT:    v_mov_b32_e32 v2, 15
-; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX9-NEXT:    scratch_store_dword v1, v2, off
+; GFX9-NEXT:    v_mov_b32_e32 v3, 15
+; GFX9-NEXT:    v_sub_u32_e32 v0, 0, v0
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0x104
+; GFX9-NEXT:    scratch_store_dword v1, v3, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_add_u32_e32 v0, 0x104, v0
-; GFX9-NEXT:    scratch_load_dword v0, v0, off offset:124 glc
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX9-NEXT:    v_mov_b32_e32 v1, 0x7c
+; GFX9-NEXT:    v_add3_u32 v0, v2, v0, v1
+; GFX9-NEXT:    scratch_load_dword v0, v0, off glc
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_endpgm
 ;
@@ -382,15 +393,16 @@ define amdgpu_kernel void @store_load_vindex_small_offset_kernel() {
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
 ; GFX10-NEXT:    v_sub_nc_u32_e32 v1, 0, v0
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX10-NEXT:    v_mov_b32_e32 v2, 15
-; GFX10-NEXT:    scratch_load_dword v3, off, off offset:4 glc dlc
-; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_mov_b32_e32 v2, 0x7c
+; GFX10-NEXT:    v_mov_b32_e32 v3, 15
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v1, 2, v1
 ; GFX10-NEXT:    v_add_nc_u32_e32 v0, 0x104, v0
-; GFX10-NEXT:    v_add_nc_u32_e32 v1, 0x104, v1
-; GFX10-NEXT:    scratch_store_dword v0, v2, off
+; GFX10-NEXT:    v_add3_u32 v1, 0x104, v1, v2
+; GFX10-NEXT:    scratch_load_dword v2, off, off offset:4 glc dlc
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    scratch_store_dword v0, v3, off
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-NEXT:    scratch_load_dword v0, v1, off offset:124 glc dlc
+; GFX10-NEXT:    scratch_load_dword v0, v1, off glc dlc
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    s_endpgm
 ;
@@ -399,26 +411,30 @@ define amdgpu_kernel void @store_load_vindex_small_offset_kernel() {
 ; GFX940-NEXT:    scratch_load_dword v1, off, off offset:4 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_lshlrev_b32_e32 v1, 2, v0
-; GFX940-NEXT:    v_mov_b32_e32 v2, 15
+; GFX940-NEXT:    v_mov_b32_e32 v3, 15
 ; GFX940-NEXT:    v_sub_u32_e32 v0, 0, v0
-; GFX940-NEXT:    scratch_store_dword v1, v2, off offset:260 sc0 sc1
+; GFX940-NEXT:    v_mov_b32_e32 v2, 0x104
+; GFX940-NEXT:    scratch_store_dword v1, v3, off offset:260 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX940-NEXT:    scratch_load_dword v0, v0, off offset:384 sc0 sc1
+; GFX940-NEXT:    v_mov_b32_e32 v1, 0x7c
+; GFX940-NEXT:    v_add3_u32 v0, v2, v0, v1
+; GFX940-NEXT:    scratch_load_dword v0, v0, off sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: store_load_vindex_small_offset_kernel:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    v_sub_nc_u32_e32 v1, 0, v0
-; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX11-NEXT:    v_mov_b32_e32 v2, 15
-; GFX11-NEXT:    scratch_load_b32 v3, off, off offset:4 glc dlc
+; GFX11-NEXT:    v_dual_mov_b32 v3, 15 :: v_dual_lshlrev_b32 v0, 2, v0
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX11-NEXT:    v_dual_mov_b32 v2, 0x7c :: v_dual_lshlrev_b32 v1, 2, v1
+; GFX11-NEXT:    v_add3_u32 v1, 0x104, v1, v2
+; GFX11-NEXT:    scratch_load_b32 v2, off, off offset:4 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_lshlrev_b32_e32 v1, 2, v1
-; GFX11-NEXT:    scratch_store_b32 v0, v2, off offset:260 dlc
+; GFX11-NEXT:    scratch_store_b32 v0, v3, off offset:260 dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_load_b32 v0, v1, off offset:384 glc dlc
+; GFX11-NEXT:    scratch_load_b32 v0, v1, off glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_endpgm
 bb:
@@ -482,9 +498,11 @@ define void @store_load_vindex_small_offset_foo(i32 %idx) {
 ; GFX940-NEXT:    scratch_load_dword v1, off, s32 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_lshlrev_b32_e32 v1, 2, v0
+; GFX940-NEXT:    s_add_i32 vcc_hi, s32, 0x100
+; GFX940-NEXT:    v_add_u32_e32 v1, vcc_hi, v1
 ; GFX940-NEXT:    v_mov_b32_e32 v2, 15
 ; GFX940-NEXT:    v_and_b32_e32 v0, 15, v0
-; GFX940-NEXT:    scratch_store_dword v1, v2, s32 offset:256 sc0 sc1
+; GFX940-NEXT:    scratch_store_dword v1, v2, off sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX940-NEXT:    scratch_load_dword v0, v0, s32 offset:256 sc0 sc1
@@ -495,14 +513,16 @@ define void @store_load_vindex_small_offset_foo(i32 %idx) {
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_and_b32 v1, 15, v0
-; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX11-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_lshlrev_b32 v1, 2, v0
+; GFX11-NEXT:    v_and_b32_e32 v0, 15, v0
+; GFX11-NEXT:    s_add_i32 vcc_lo, s32, 0x100
 ; GFX11-NEXT:    scratch_load_b32 v3, off, s32 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_lshlrev_b32_e32 v1, 2, v1
-; GFX11-NEXT:    scratch_store_b32 v0, v2, s32 offset:256 dlc
+; GFX11-NEXT:    v_add_nc_u32_e32 v1, vcc_lo, v1
+; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX11-NEXT:    scratch_store_b32 v1, v2, off dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_load_b32 v0, v1, s32 offset:256 glc dlc
+; GFX11-NEXT:    scratch_load_b32 v0, v0, s32 offset:256 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_setpc_b64 s[30:31]
 bb:
@@ -567,14 +587,13 @@ define amdgpu_kernel void @store_load_sindex_large_offset_kernel(i32 %idx) {
 ; GFX940-NEXT:    scratch_load_dword v0, off, off offset:4 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_mov_b32_e32 v0, 15
-; GFX940-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX940-NEXT:    s_movk_i32 vcc_hi, 0x4004
 ; GFX940-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-NEXT:    s_lshl_b32 s1, s0, 2
 ; GFX940-NEXT:    s_and_b32 s0, s0, 15
-; GFX940-NEXT:    v_mov_b32_e32 v1, s1
+; GFX940-NEXT:    s_addk_i32 s1, 0x4004
 ; GFX940-NEXT:    s_lshl_b32 s0, s0, 2
-; GFX940-NEXT:    scratch_store_dword v1, v0, vcc_lo sc0 sc1
+; GFX940-NEXT:    scratch_store_dword off, v0, s1 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX940-NEXT:    scratch_load_dword v0, v0, vcc_hi sc0 sc1
@@ -585,20 +604,17 @@ define amdgpu_kernel void @store_load_sindex_large_offset_kernel(i32 %idx) {
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    s_load_b32 s0, s[0:1], 0x24
 ; GFX11-NEXT:    s_movk_i32 vcc_lo, 0x4004
-; GFX11-NEXT:    scratch_load_b32 v3, off, off offset:4 glc dlc
-; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_mov_b32_e32 v1, 15
-; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-NEXT:    s_lshl_b32 s1, s0, 2
-; GFX11-NEXT:    s_and_b32 s0, s0, 15
-; GFX11-NEXT:    v_mov_b32_e32 v0, s1
+; GFX11-NEXT:    scratch_load_b32 v2, off, off offset:4 glc dlc
+; GFX11-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX11-NEXT:    s_and_b32 s1, s0, 15
 ; GFX11-NEXT:    s_lshl_b32 s0, s0, 2
+; GFX11-NEXT:    s_lshl_b32 s1, s1, 2
 ; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11-NEXT:    v_mov_b32_e32 v2, s0
-; GFX11-NEXT:    s_movk_i32 s0, 0x4004
-; GFX11-NEXT:    scratch_store_b32 v0, v1, s0 dlc
+; GFX11-NEXT:    v_dual_mov_b32 v0, 15 :: v_dual_mov_b32 v1, s1
+; GFX11-NEXT:    s_addk_i32 s0, 0x4004
+; GFX11-NEXT:    scratch_store_b32 off, v0, s0 dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_load_b32 v0, v2, vcc_lo glc dlc
+; GFX11-NEXT:    scratch_load_b32 v0, v1, vcc_lo glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_endpgm
 bb:
@@ -623,14 +639,16 @@ define amdgpu_kernel void @store_load_vindex_large_offset_kernel() {
 ; GFX9-NEXT:    scratch_load_dword v1, off, vcc_hi offset:4 glc
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v1, 2, v0
-; GFX9-NEXT:    v_sub_u32_e32 v0, 0, v0
 ; GFX9-NEXT:    v_add_u32_e32 v1, 0x4004, v1
-; GFX9-NEXT:    v_mov_b32_e32 v2, 15
-; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX9-NEXT:    scratch_store_dword v1, v2, off
+; GFX9-NEXT:    v_mov_b32_e32 v3, 15
+; GFX9-NEXT:    v_sub_u32_e32 v0, 0, v0
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0x4004
+; GFX9-NEXT:    scratch_store_dword v1, v3, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_add_u32_e32 v0, 0x4004, v0
-; GFX9-NEXT:    scratch_load_dword v0, v0, off offset:124 glc
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX9-NEXT:    v_mov_b32_e32 v1, 0x7c
+; GFX9-NEXT:    v_add3_u32 v0, v2, v0, v1
+; GFX9-NEXT:    scratch_load_dword v0, v0, off glc
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_endpgm
 ;
@@ -642,15 +660,16 @@ define amdgpu_kernel void @store_load_vindex_large_offset_kernel() {
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
 ; GFX10-NEXT:    v_sub_nc_u32_e32 v1, 0, v0
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX10-NEXT:    v_mov_b32_e32 v2, 15
-; GFX10-NEXT:    scratch_load_dword v3, off, off offset:4 glc dlc
-; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_mov_b32_e32 v2, 0x7c
+; GFX10-NEXT:    v_mov_b32_e32 v3, 15
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v1, 2, v1
 ; GFX10-NEXT:    v_add_nc_u32_e32 v0, 0x4004, v0
-; GFX10-NEXT:    v_add_nc_u32_e32 v1, 0x4004, v1
-; GFX10-NEXT:    scratch_store_dword v0, v2, off
+; GFX10-NEXT:    v_add3_u32 v1, 0x4004, v1, v2
+; GFX10-NEXT:    scratch_load_dword v2, off, off offset:4 glc dlc
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    scratch_store_dword v0, v3, off
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-NEXT:    scratch_load_dword v0, v1, off offset:124 glc dlc
+; GFX10-NEXT:    scratch_load_dword v0, v1, off glc dlc
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    s_endpgm
 ;
@@ -659,30 +678,32 @@ define amdgpu_kernel void @store_load_vindex_large_offset_kernel() {
 ; GFX940-NEXT:    scratch_load_dword v1, off, off offset:4 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_lshlrev_b32_e32 v1, 2, v0
-; GFX940-NEXT:    v_mov_b32_e32 v2, 15
-; GFX940-NEXT:    s_movk_i32 vcc_lo, 0x4004
+; GFX940-NEXT:    v_mov_b32_e32 v3, 15
+; GFX940-NEXT:    s_movk_i32 vcc_hi, 0x4004
 ; GFX940-NEXT:    v_sub_u32_e32 v0, 0, v0
-; GFX940-NEXT:    scratch_store_dword v1, v2, vcc_lo sc0 sc1
+; GFX940-NEXT:    v_mov_b32_e32 v2, 0x4004
+; GFX940-NEXT:    scratch_store_dword v1, v3, vcc_hi sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX940-NEXT:    s_movk_i32 vcc_hi, 0x4004
-; GFX940-NEXT:    scratch_load_dword v0, v0, vcc_hi offset:124 sc0 sc1
+; GFX940-NEXT:    v_mov_b32_e32 v1, 0x7c
+; GFX940-NEXT:    v_add3_u32 v0, v2, v0, v1
+; GFX940-NEXT:    scratch_load_dword v0, v0, off sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: store_load_vindex_large_offset_kernel:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    v_sub_nc_u32_e32 v1, 0, v0
-; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX11-NEXT:    v_mov_b32_e32 v2, 15
-; GFX11-NEXT:    s_movk_i32 s0, 0x4004
+; GFX11-NEXT:    v_dual_mov_b32 v3, 15 :: v_dual_lshlrev_b32 v0, 2, v0
 ; GFX11-NEXT:    s_movk_i32 vcc_lo, 0x4004
-; GFX11-NEXT:    v_lshlrev_b32_e32 v1, 2, v1
-; GFX11-NEXT:    scratch_load_b32 v3, off, off offset:4 glc dlc
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX11-NEXT:    v_dual_mov_b32 v2, 0x7c :: v_dual_lshlrev_b32 v1, 2, v1
+; GFX11-NEXT:    v_add3_u32 v1, 0x4004, v1, v2
+; GFX11-NEXT:    scratch_load_b32 v2, off, off offset:4 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    scratch_store_b32 v0, v2, s0 dlc
+; GFX11-NEXT:    scratch_store_b32 v0, v3, vcc_lo dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_load_b32 v0, v1, vcc_lo offset:124 glc dlc
+; GFX11-NEXT:    scratch_load_b32 v0, v1, off glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_endpgm
 bb:
@@ -746,10 +767,11 @@ define void @store_load_vindex_large_offset_foo(i32 %idx) {
 ; GFX940-NEXT:    scratch_load_dword v1, off, s32 offset:4 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_lshlrev_b32_e32 v1, 2, v0
-; GFX940-NEXT:    v_mov_b32_e32 v2, 15
 ; GFX940-NEXT:    s_add_i32 vcc_lo, s32, 0x4004
+; GFX940-NEXT:    v_add_u32_e32 v1, vcc_lo, v1
+; GFX940-NEXT:    v_mov_b32_e32 v2, 15
 ; GFX940-NEXT:    v_and_b32_e32 v0, 15, v0
-; GFX940-NEXT:    scratch_store_dword v1, v2, vcc_lo sc0 sc1
+; GFX940-NEXT:    scratch_store_dword v1, v2, off sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX940-NEXT:    s_add_i32 vcc_hi, s32, 0x4004
@@ -761,17 +783,18 @@ define void @store_load_vindex_large_offset_foo(i32 %idx) {
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_and_b32 v1, 15, v0
-; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX11-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_lshlrev_b32 v1, 2, v0
+; GFX11-NEXT:    v_and_b32_e32 v0, 15, v0
 ; GFX11-NEXT:    s_add_i32 s0, s32, 0x4004
 ; GFX11-NEXT:    s_add_i32 vcc_lo, s32, 0x4004
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2)
-; GFX11-NEXT:    v_lshlrev_b32_e32 v1, 2, v1
+; GFX11-NEXT:    v_add_nc_u32_e32 v1, s0, v1
 ; GFX11-NEXT:    scratch_load_b32 v3, off, s32 offset:4 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    scratch_store_b32 v0, v2, s0 dlc
+; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX11-NEXT:    scratch_store_b32 v1, v2, off dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_load_b32 v0, v1, vcc_lo glc dlc
+; GFX11-NEXT:    scratch_load_b32 v0, v0, vcc_lo glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_setpc_b64 s[30:31]
 bb:
@@ -935,14 +958,16 @@ define amdgpu_kernel void @store_load_vidx_sidx_offset(i32 %sidx) {
 ; GFX9:       ; %bb.0: ; %bb
 ; GFX9-NEXT:    s_load_dword s0, s[0:1], 0x24
 ; GFX9-NEXT:    s_add_u32 flat_scratch_lo, s2, s5
+; GFX9-NEXT:    v_mov_b32_e32 v1, 4
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0x400
 ; GFX9-NEXT:    s_addc_u32 flat_scratch_hi, s3, 0
-; GFX9-NEXT:    v_mov_b32_e32 v1, 15
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_add_lshl_u32 v0, s0, v0, 2
-; GFX9-NEXT:    v_add_u32_e32 v0, 4, v0
-; GFX9-NEXT:    scratch_store_dword v0, v1, off offset:1024
+; GFX9-NEXT:    v_add3_u32 v0, v1, v0, v2
+; GFX9-NEXT:    v_mov_b32_e32 v1, 15
+; GFX9-NEXT:    scratch_store_dword v0, v1, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    scratch_load_dword v0, v0, off offset:1024 glc
+; GFX9-NEXT:    scratch_load_dword v0, v0, off glc
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_endpgm
 ;
@@ -953,37 +978,44 @@ define amdgpu_kernel void @store_load_vidx_sidx_offset(i32 %sidx) {
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s2
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s3
 ; GFX10-NEXT:    s_load_dword s0, s[0:1], 0x24
-; GFX10-NEXT:    v_mov_b32_e32 v1, 15
+; GFX10-NEXT:    v_mov_b32_e32 v1, 0x400
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-NEXT:    v_add_lshl_u32 v0, s0, v0, 2
-; GFX10-NEXT:    v_add_nc_u32_e32 v0, 4, v0
-; GFX10-NEXT:    scratch_store_dword v0, v1, off offset:1024
+; GFX10-NEXT:    v_add3_u32 v0, 4, v0, v1
+; GFX10-NEXT:    v_mov_b32_e32 v1, 15
+; GFX10-NEXT:    scratch_store_dword v0, v1, off
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-NEXT:    scratch_load_dword v0, v0, off offset:1024 glc dlc
+; GFX10-NEXT:    scratch_load_dword v0, v0, off glc dlc
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX940-LABEL: store_load_vidx_sidx_offset:
 ; GFX940:       ; %bb.0: ; %bb
 ; GFX940-NEXT:    s_load_dword s0, s[0:1], 0x24
-; GFX940-NEXT:    v_mov_b32_e32 v1, 15
+; GFX940-NEXT:    v_mov_b32_e32 v1, 4
+; GFX940-NEXT:    v_mov_b32_e32 v2, 0x400
 ; GFX940-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-NEXT:    v_add_lshl_u32 v0, s0, v0, 2
-; GFX940-NEXT:    scratch_store_dword v0, v1, off offset:1028 sc0 sc1
+; GFX940-NEXT:    v_add3_u32 v0, v1, v0, v2
+; GFX940-NEXT:    v_mov_b32_e32 v1, 15
+; GFX940-NEXT:    scratch_store_dword v0, v1, off sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-NEXT:    scratch_load_dword v0, v0, off offset:1028 sc0 sc1
+; GFX940-NEXT:    scratch_load_dword v0, v0, off sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: store_load_vidx_sidx_offset:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    s_load_b32 s0, s[0:1], 0x24
-; GFX11-NEXT:    v_mov_b32_e32 v1, 15
+; GFX11-NEXT:    v_mov_b32_e32 v1, 0x400
 ; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-NEXT:    v_add_lshl_u32 v0, s0, v0, 2
-; GFX11-NEXT:    scratch_store_b32 v0, v1, off offset:1028 dlc
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_add3_u32 v0, 4, v0, v1
+; GFX11-NEXT:    v_mov_b32_e32 v1, 15
+; GFX11-NEXT:    scratch_store_b32 v0, v1, off dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_load_b32 v0, v0, off offset:1028 glc dlc
+; GFX11-NEXT:    scratch_load_b32 v0, v0, off glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_endpgm
 bb:
diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-atomicrmw-add-flat.mir b/llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-atomicrmw-add-flat.mir
index 782cc5553881e..78071022fc05c 100644
--- a/llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-atomicrmw-add-flat.mir
+++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-atomicrmw-add-flat.mir
@@ -110,8 +110,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: [[FLAT_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = FLAT_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX7-NEXT: $vgpr0 = COPY [[FLAT_ATOMIC_ADD_RTN]]
     ; GFX9-LABEL: name: flat_atomicrmw_add_s32_offset2047
@@ -134,8 +134,8 @@ body:             |
     ; GFX10-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX10-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX10-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX10-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX10-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX10-NEXT: [[FLAT_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = FLAT_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX10-NEXT: $vgpr0 = COPY [[FLAT_ATOMIC_ADD_RTN]]
     ; GFX11-LABEL: name: flat_atomicrmw_add_s32_offset2047
@@ -176,8 +176,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: FLAT_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX9-LABEL: name: flat_atomicrmw_add_s32_offset2047_nortn
     ; GFX9: liveins: $vgpr0_vgpr1, $vgpr2
@@ -198,8 +198,8 @@ body:             |
     ; GFX10-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX10-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX10-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX10-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX10-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX10-NEXT: FLAT_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX11-LABEL: name: flat_atomicrmw_add_s32_offset2047_nortn
     ; GFX11: liveins: $vgpr0_vgpr1, $vgpr2
@@ -237,8 +237,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: [[FLAT_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = FLAT_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX7-NEXT: $vgpr0 = COPY [[FLAT_ATOMIC_ADD_RTN]]
     ; GFX9-LABEL: name: flat_atomicrmw_add_s32_offset2048
@@ -261,8 +261,8 @@ body:             |
     ; GFX10-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX10-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX10-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX10-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX10-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX10-NEXT: [[FLAT_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = FLAT_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX10-NEXT: $vgpr0 = COPY [[FLAT_ATOMIC_ADD_RTN]]
     ; GFX11-LABEL: name: flat_atomicrmw_add_s32_offset2048
@@ -303,8 +303,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: FLAT_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX9-LABEL: name: flat_atomicrmw_add_s32_offset2048_nortn
     ; GFX9: liveins: $vgpr0_vgpr1, $vgpr2
@@ -325,8 +325,8 @@ body:             |
     ; GFX10-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX10-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX10-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX10-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX10-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX10-NEXT: FLAT_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX11-LABEL: name: flat_atomicrmw_add_s32_offset2048_nortn
     ; GFX11: liveins: $vgpr0_vgpr1, $vgpr2
@@ -364,8 +364,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: [[FLAT_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = FLAT_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX7-NEXT: $vgpr0 = COPY [[FLAT_ATOMIC_ADD_RTN]]
     ; GFX9-LABEL: name: flat_atomicrmw_add_s32_offset4095
@@ -388,8 +388,8 @@ body:             |
     ; GFX10-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX10-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX10-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX10-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX10-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX10-NEXT: [[FLAT_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = FLAT_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX10-NEXT: $vgpr0 = COPY [[FLAT_ATOMIC_ADD_RTN]]
     ; GFX11-LABEL: name: flat_atomicrmw_add_s32_offset4095
@@ -430,8 +430,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: FLAT_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX9-LABEL: name: flat_atomicrmw_add_s32_offset4095_nortn
     ; GFX9: liveins: $vgpr0_vgpr1, $vgpr2
@@ -452,8 +452,8 @@ body:             |
     ; GFX10-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX10-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX10-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX10-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX10-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX10-NEXT: FLAT_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX11-LABEL: name: flat_atomicrmw_add_s32_offset4095_nortn
     ; GFX11: liveins: $vgpr0_vgpr1, $vgpr2
@@ -491,8 +491,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: [[FLAT_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = FLAT_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX7-NEXT: $vgpr0 = COPY [[FLAT_ATOMIC_ADD_RTN]]
     ; GFX9-LABEL: name: flat_atomicrmw_add_s32_offset4097
@@ -508,8 +508,8 @@ body:             |
     ; GFX9-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX9-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX9-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX9-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX9-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX9-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX9-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX9-NEXT: [[FLAT_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = FLAT_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX9-NEXT: $vgpr0 = COPY [[FLAT_ATOMIC_ADD_RTN]]
     ; GFX10-LABEL: name: flat_atomicrmw_add_s32_offset4097
@@ -525,8 +525,8 @@ body:             |
     ; GFX10-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX10-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX10-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX10-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX10-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX10-NEXT: [[FLAT_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = FLAT_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX10-NEXT: $vgpr0 = COPY [[FLAT_ATOMIC_ADD_RTN]]
     ; GFX11-LABEL: name: flat_atomicrmw_add_s32_offset4097
@@ -542,8 +542,8 @@ body:             |
     ; GFX11-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX11-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX11-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX11-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX11-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX11-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX11-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX11-NEXT: [[FLAT_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = FLAT_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX11-NEXT: $vgpr0 = COPY [[FLAT_ATOMIC_ADD_RTN]]
     %0:vgpr(p0) = COPY $vgpr0_vgpr1
@@ -577,8 +577,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: FLAT_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX9-LABEL: name: flat_atomicrmw_add_s32_offset4097_nortn
     ; GFX9: liveins: $vgpr0_vgpr1, $vgpr2
@@ -593,8 +593,8 @@ body:             |
     ; GFX9-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX9-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX9-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX9-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX9-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX9-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX9-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX9-NEXT: FLAT_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX10-LABEL: name: flat_atomicrmw_add_s32_offset4097_nortn
     ; GFX10: liveins: $vgpr0_vgpr1, $vgpr2
@@ -609,8 +609,8 @@ body:             |
     ; GFX10-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX10-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX10-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX10-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX10-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX10-NEXT: FLAT_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     ; GFX11-LABEL: name: flat_atomicrmw_add_s32_offset4097_nortn
     ; GFX11: liveins: $vgpr0_vgpr1, $vgpr2
@@ -625,8 +625,8 @@ body:             |
     ; GFX11-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX11-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX11-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX11-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX11-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX11-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX11-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX11-NEXT: FLAT_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32))
     %0:vgpr(p0) = COPY $vgpr0_vgpr1
     %1:vgpr(s32) = COPY $vgpr2
@@ -741,8 +741,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: [[FLAT_ATOMIC_ADD_X2_RTN:%[0-9]+]]:vreg_64 = FLAT_ATOMIC_ADD_X2_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec, implicit $flat_scr :: (load store seq_cst (s64))
     ; GFX7-NEXT: $vgpr0_vgpr1 = COPY [[FLAT_ATOMIC_ADD_X2_RTN]]
     ; GFX9-LABEL: name: flat_atomicrmw_add_s64_offset4095
@@ -765,8 +765,8 @@ body:             |
     ; GFX10-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX10-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX10-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX10-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX10-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX10-NEXT: [[FLAT_ATOMIC_ADD_X2_RTN:%[0-9]+]]:vreg_64 = FLAT_ATOMIC_ADD_X2_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec, implicit $flat_scr :: (load store seq_cst (s64))
     ; GFX10-NEXT: $vgpr0_vgpr1 = COPY [[FLAT_ATOMIC_ADD_X2_RTN]]
     ; GFX11-LABEL: name: flat_atomicrmw_add_s64_offset4095
@@ -807,8 +807,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: FLAT_ATOMIC_ADD_X2 [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec, implicit $flat_scr :: (load store seq_cst (s64))
     ; GFX9-LABEL: name: flat_atomicrmw_add_s64_offset4095_nortn
     ; GFX9: liveins: $vgpr0_vgpr1, $vgpr2_vgpr3
@@ -829,8 +829,8 @@ body:             |
     ; GFX10-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX10-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX10-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX10-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX10-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX10-NEXT: FLAT_ATOMIC_ADD_X2 [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec, implicit $flat_scr :: (load store seq_cst (s64))
     ; GFX11-LABEL: name: flat_atomicrmw_add_s64_offset4095_nortn
     ; GFX11: liveins: $vgpr0_vgpr1, $vgpr2_vgpr3
diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-atomicrmw-add-global.mir b/llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-atomicrmw-add-global.mir
index 930d2789f2ea9..d3eaee283d644 100644
--- a/llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-atomicrmw-add-global.mir
+++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-atomicrmw-add-global.mir
@@ -146,8 +146,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: [[FLAT_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = FLAT_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32), addrspace 1)
     ; GFX7-NEXT: $vgpr0 = COPY [[FLAT_ATOMIC_ADD_RTN]]
     ; GFX9-LABEL: name: global_atomicrmw_add_s32_offset2047
@@ -213,8 +213,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: FLAT_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32), addrspace 1)
     ; GFX9-LABEL: name: global_atomicrmw_add_s32_offset2047_nortn
     ; GFX9: liveins: $vgpr0_vgpr1, $vgpr2
@@ -276,8 +276,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: [[FLAT_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = FLAT_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32), addrspace 1)
     ; GFX7-NEXT: $vgpr0 = COPY [[FLAT_ATOMIC_ADD_RTN]]
     ; GFX9-LABEL: name: global_atomicrmw_add_s32_offset2048
@@ -300,8 +300,8 @@ body:             |
     ; GFX10-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX10-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX10-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX10-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX10-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX10-NEXT: [[GLOBAL_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = GLOBAL_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec :: (load store seq_cst (s32), addrspace 1)
     ; GFX10-NEXT: $vgpr0 = COPY [[GLOBAL_ATOMIC_ADD_RTN]]
     ; GFX11-LABEL: name: global_atomicrmw_add_s32_offset2048
@@ -353,8 +353,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: FLAT_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32), addrspace 1)
     ; GFX9-LABEL: name: global_atomicrmw_add_s32_offset2048_nortn
     ; GFX9: liveins: $vgpr0_vgpr1, $vgpr2
@@ -375,8 +375,8 @@ body:             |
     ; GFX10-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX10-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX10-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX10-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX10-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX10-NEXT: GLOBAL_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec :: (load store seq_cst (s32), addrspace 1)
     ; GFX11-LABEL: name: global_atomicrmw_add_s32_offset2048_nortn
     ; GFX11: liveins: $vgpr0_vgpr1, $vgpr2
@@ -426,8 +426,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: [[FLAT_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = FLAT_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32), addrspace 1)
     ; GFX7-NEXT: $vgpr0 = COPY [[FLAT_ATOMIC_ADD_RTN]]
     ; GFX9-LABEL: name: global_atomicrmw_add_s32_offset4095
@@ -450,8 +450,8 @@ body:             |
     ; GFX10-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX10-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX10-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX10-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX10-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX10-NEXT: [[GLOBAL_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = GLOBAL_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec :: (load store seq_cst (s32), addrspace 1)
     ; GFX10-NEXT: $vgpr0 = COPY [[GLOBAL_ATOMIC_ADD_RTN]]
     ; GFX11-LABEL: name: global_atomicrmw_add_s32_offset4095
@@ -503,8 +503,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: FLAT_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32), addrspace 1)
     ; GFX9-LABEL: name: global_atomicrmw_add_s32_offset4095_nortn
     ; GFX9: liveins: $vgpr0_vgpr1, $vgpr2
@@ -525,8 +525,8 @@ body:             |
     ; GFX10-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX10-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX10-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX10-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX10-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX10-NEXT: GLOBAL_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec :: (load store seq_cst (s32), addrspace 1)
     ; GFX11-LABEL: name: global_atomicrmw_add_s32_offset4095_nortn
     ; GFX11: liveins: $vgpr0_vgpr1, $vgpr2
@@ -577,8 +577,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: [[FLAT_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = FLAT_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32), addrspace 1)
     ; GFX7-NEXT: $vgpr0 = COPY [[FLAT_ATOMIC_ADD_RTN]]
     ; GFX9-LABEL: name: global_atomicrmw_add_s32_offset4097
@@ -594,8 +594,8 @@ body:             |
     ; GFX9-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX9-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX9-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX9-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX9-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX9-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX9-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX9-NEXT: [[GLOBAL_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = GLOBAL_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec :: (load store seq_cst (s32), addrspace 1)
     ; GFX9-NEXT: $vgpr0 = COPY [[GLOBAL_ATOMIC_ADD_RTN]]
     ; GFX10-LABEL: name: global_atomicrmw_add_s32_offset4097
@@ -611,8 +611,8 @@ body:             |
     ; GFX10-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX10-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX10-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX10-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX10-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX10-NEXT: [[GLOBAL_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = GLOBAL_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec :: (load store seq_cst (s32), addrspace 1)
     ; GFX10-NEXT: $vgpr0 = COPY [[GLOBAL_ATOMIC_ADD_RTN]]
     ; GFX11-LABEL: name: global_atomicrmw_add_s32_offset4097
@@ -628,8 +628,8 @@ body:             |
     ; GFX11-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX11-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX11-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX11-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX11-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX11-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX11-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX11-NEXT: [[GLOBAL_ATOMIC_ADD_RTN:%[0-9]+]]:vgpr_32 = GLOBAL_ATOMIC_ADD_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec :: (load store seq_cst (s32), addrspace 1)
     ; GFX11-NEXT: $vgpr0 = COPY [[GLOBAL_ATOMIC_ADD_RTN]]
     %0:vgpr(p1) = COPY $vgpr0_vgpr1
@@ -675,8 +675,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: FLAT_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec, implicit $flat_scr :: (load store seq_cst (s32), addrspace 1)
     ; GFX9-LABEL: name: global_atomicrmw_add_s32_offset4097_nortn
     ; GFX9: liveins: $vgpr0_vgpr1, $vgpr2
@@ -691,8 +691,8 @@ body:             |
     ; GFX9-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX9-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX9-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX9-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX9-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX9-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX9-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX9-NEXT: GLOBAL_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec :: (load store seq_cst (s32), addrspace 1)
     ; GFX10-LABEL: name: global_atomicrmw_add_s32_offset4097_nortn
     ; GFX10: liveins: $vgpr0_vgpr1, $vgpr2
@@ -707,8 +707,8 @@ body:             |
     ; GFX10-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX10-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX10-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX10-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX10-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX10-NEXT: GLOBAL_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec :: (load store seq_cst (s32), addrspace 1)
     ; GFX11-LABEL: name: global_atomicrmw_add_s32_offset4097_nortn
     ; GFX11: liveins: $vgpr0_vgpr1, $vgpr2
@@ -723,8 +723,8 @@ body:             |
     ; GFX11-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX11-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX11-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX11-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX11-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX11-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX11-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX11-NEXT: GLOBAL_ATOMIC_ADD [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec :: (load store seq_cst (s32), addrspace 1)
     %0:vgpr(p1) = COPY $vgpr0_vgpr1
     %1:vgpr(s32) = COPY $vgpr2
@@ -874,8 +874,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: [[FLAT_ATOMIC_ADD_X2_RTN:%[0-9]+]]:vreg_64 = FLAT_ATOMIC_ADD_X2_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec, implicit $flat_scr :: (load store seq_cst (s64), addrspace 1)
     ; GFX7-NEXT: $vgpr0_vgpr1 = COPY [[FLAT_ATOMIC_ADD_X2_RTN]]
     ; GFX9-LABEL: name: global_atomicrmw_add_s64_offset4095
@@ -898,8 +898,8 @@ body:             |
     ; GFX10-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX10-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX10-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX10-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX10-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX10-NEXT: [[GLOBAL_ATOMIC_ADD_X2_RTN:%[0-9]+]]:vreg_64 = GLOBAL_ATOMIC_ADD_X2_RTN [[REG_SEQUENCE1]], [[COPY1]], 0, 1, implicit $exec :: (load store seq_cst (s64), addrspace 1)
     ; GFX10-NEXT: $vgpr0_vgpr1 = COPY [[GLOBAL_ATOMIC_ADD_X2_RTN]]
     ; GFX11-LABEL: name: global_atomicrmw_add_s64_offset4095
@@ -951,8 +951,8 @@ body:             |
     ; GFX7-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX7-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX7-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX7-NEXT: %10:vgpr_32, dead %12:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX7-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX7-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX7-NEXT: FLAT_ATOMIC_ADD_X2 [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec, implicit $flat_scr :: (load store seq_cst (s64), addrspace 1)
     ; GFX9-LABEL: name: global_atomicrmw_add_s64_offset4095_nortn
     ; GFX9: liveins: $vgpr0_vgpr1, $vgpr2_vgpr3
@@ -973,8 +973,8 @@ body:             |
     ; GFX10-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY [[COPY]].sub1
     ; GFX10-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY [[REG_SEQUENCE]].sub1
     ; GFX10-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY2]], [[COPY3]], 0, implicit $exec
-    ; GFX10-NEXT: %10:vgpr_32, dead %12:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %10, %subreg.sub1
+    ; GFX10-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY4]], [[COPY5]], killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; GFX10-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; GFX10-NEXT: GLOBAL_ATOMIC_ADD_X2 [[REG_SEQUENCE1]], [[COPY1]], 0, 0, implicit $exec :: (load store seq_cst (s64), addrspace 1)
     ; GFX11-LABEL: name: global_atomicrmw_add_s64_offset4095_nortn
     ; GFX11: liveins: $vgpr0_vgpr1, $vgpr2_vgpr3
diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-load-private.mir b/llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-load-private.mir
index 2fe38a64c8d86..4bf6b6a623d82 100644
--- a/llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-load-private.mir
+++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-load-private.mir
@@ -270,7 +270,9 @@ body: |
     ; GFX11: liveins: $vgpr0
     ; GFX11-NEXT: {{  $}}
     ; GFX11-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
-    ; GFX11-NEXT: [[SCRATCH_LOAD_UBYTE:%[0-9]+]]:vgpr_32 = SCRATCH_LOAD_UBYTE [[COPY]], 2047, 0, implicit $exec, implicit $flat_scr :: (load (s8), addrspace 5)
+    ; GFX11-NEXT: [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 2047, implicit $exec
+    ; GFX11-NEXT: [[V_ADD_U32_e64_:%[0-9]+]]:vgpr_32 = V_ADD_U32_e64 [[COPY]], [[V_MOV_B32_e32_]], 0, implicit $exec
+    ; GFX11-NEXT: [[SCRATCH_LOAD_UBYTE:%[0-9]+]]:vgpr_32 = SCRATCH_LOAD_UBYTE [[V_ADD_U32_e64_]], 0, 0, implicit $exec, implicit $flat_scr :: (load (s8), addrspace 5)
     ; GFX11-NEXT: $vgpr0 = COPY [[SCRATCH_LOAD_UBYTE]]
     %0:vgpr(p5) = COPY $vgpr0
     %1:vgpr(s32) = G_CONSTANT i32 2047
@@ -361,7 +363,9 @@ body: |
     ; GFX11: liveins: $vgpr0
     ; GFX11-NEXT: {{  $}}
     ; GFX11-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
-    ; GFX11-NEXT: [[SCRATCH_LOAD_UBYTE:%[0-9]+]]:vgpr_32 = SCRATCH_LOAD_UBYTE [[COPY]], 2048, 0, implicit $exec, implicit $flat_scr :: (load (s8), addrspace 5)
+    ; GFX11-NEXT: [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 2048, implicit $exec
+    ; GFX11-NEXT: [[V_ADD_U32_e64_:%[0-9]+]]:vgpr_32 = V_ADD_U32_e64 [[COPY]], [[V_MOV_B32_e32_]], 0, implicit $exec
+    ; GFX11-NEXT: [[SCRATCH_LOAD_UBYTE:%[0-9]+]]:vgpr_32 = SCRATCH_LOAD_UBYTE [[V_ADD_U32_e64_]], 0, 0, implicit $exec, implicit $flat_scr :: (load (s8), addrspace 5)
     ; GFX11-NEXT: $vgpr0 = COPY [[SCRATCH_LOAD_UBYTE]]
     %0:vgpr(p5) = COPY $vgpr0
     %1:vgpr(s32) = G_CONSTANT i32 2048
@@ -405,7 +409,9 @@ body: |
     ; GFX11: liveins: $vgpr0
     ; GFX11-NEXT: {{  $}}
     ; GFX11-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
-    ; GFX11-NEXT: [[SCRATCH_LOAD_UBYTE:%[0-9]+]]:vgpr_32 = SCRATCH_LOAD_UBYTE [[COPY]], -2047, 0, implicit $exec, implicit $flat_scr :: (load (s8), addrspace 5)
+    ; GFX11-NEXT: [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 -2047, implicit $exec
+    ; GFX11-NEXT: [[V_ADD_U32_e64_:%[0-9]+]]:vgpr_32 = V_ADD_U32_e64 [[COPY]], [[V_MOV_B32_e32_]], 0, implicit $exec
+    ; GFX11-NEXT: [[SCRATCH_LOAD_UBYTE:%[0-9]+]]:vgpr_32 = SCRATCH_LOAD_UBYTE [[V_ADD_U32_e64_]], 0, 0, implicit $exec, implicit $flat_scr :: (load (s8), addrspace 5)
     ; GFX11-NEXT: $vgpr0 = COPY [[SCRATCH_LOAD_UBYTE]]
     %0:vgpr(p5) = COPY $vgpr0
     %1:vgpr(s32) = G_CONSTANT i32 -2047
@@ -449,7 +455,9 @@ body: |
     ; GFX11: liveins: $vgpr0
     ; GFX11-NEXT: {{  $}}
     ; GFX11-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
-    ; GFX11-NEXT: [[SCRATCH_LOAD_UBYTE:%[0-9]+]]:vgpr_32 = SCRATCH_LOAD_UBYTE [[COPY]], -2048, 0, implicit $exec, implicit $flat_scr :: (load (s8), addrspace 5)
+    ; GFX11-NEXT: [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 -2048, implicit $exec
+    ; GFX11-NEXT: [[V_ADD_U32_e64_:%[0-9]+]]:vgpr_32 = V_ADD_U32_e64 [[COPY]], [[V_MOV_B32_e32_]], 0, implicit $exec
+    ; GFX11-NEXT: [[SCRATCH_LOAD_UBYTE:%[0-9]+]]:vgpr_32 = SCRATCH_LOAD_UBYTE [[V_ADD_U32_e64_]], 0, 0, implicit $exec, implicit $flat_scr :: (load (s8), addrspace 5)
     ; GFX11-NEXT: $vgpr0 = COPY [[SCRATCH_LOAD_UBYTE]]
     %0:vgpr(p5) = COPY $vgpr0
     %1:vgpr(s32) = G_CONSTANT i32 -2048
@@ -491,7 +499,9 @@ body: |
     ; GFX11: liveins: $vgpr0
     ; GFX11-NEXT: {{  $}}
     ; GFX11-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
-    ; GFX11-NEXT: [[SCRATCH_LOAD_UBYTE:%[0-9]+]]:vgpr_32 = SCRATCH_LOAD_UBYTE [[COPY]], 4095, 0, implicit $exec, implicit $flat_scr :: (load (s8), addrspace 5)
+    ; GFX11-NEXT: [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 4095, implicit $exec
+    ; GFX11-NEXT: [[V_ADD_U32_e64_:%[0-9]+]]:vgpr_32 = V_ADD_U32_e64 [[COPY]], [[V_MOV_B32_e32_]], 0, implicit $exec
+    ; GFX11-NEXT: [[SCRATCH_LOAD_UBYTE:%[0-9]+]]:vgpr_32 = SCRATCH_LOAD_UBYTE [[V_ADD_U32_e64_]], 0, 0, implicit $exec, implicit $flat_scr :: (load (s8), addrspace 5)
     ; GFX11-NEXT: $vgpr0 = COPY [[SCRATCH_LOAD_UBYTE]]
     %0:vgpr(p5) = COPY $vgpr0
     %1:vgpr(s32) = G_CONSTANT i32 4095
@@ -581,7 +591,9 @@ body: |
     ; GFX11: liveins: $vgpr0
     ; GFX11-NEXT: {{  $}}
     ; GFX11-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
-    ; GFX11-NEXT: [[SCRATCH_LOAD_UBYTE:%[0-9]+]]:vgpr_32 = SCRATCH_LOAD_UBYTE [[COPY]], -4095, 0, implicit $exec, implicit $flat_scr :: (load (s8), addrspace 5)
+    ; GFX11-NEXT: [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 -4095, implicit $exec
+    ; GFX11-NEXT: [[V_ADD_U32_e64_:%[0-9]+]]:vgpr_32 = V_ADD_U32_e64 [[COPY]], [[V_MOV_B32_e32_]], 0, implicit $exec
+    ; GFX11-NEXT: [[SCRATCH_LOAD_UBYTE:%[0-9]+]]:vgpr_32 = SCRATCH_LOAD_UBYTE [[V_ADD_U32_e64_]], 0, 0, implicit $exec, implicit $flat_scr :: (load (s8), addrspace 5)
     ; GFX11-NEXT: $vgpr0 = COPY [[SCRATCH_LOAD_UBYTE]]
     %0:vgpr(p5) = COPY $vgpr0
     %1:vgpr(s32) = G_CONSTANT i32 -4095
@@ -625,7 +637,9 @@ body: |
     ; GFX11: liveins: $vgpr0
     ; GFX11-NEXT: {{  $}}
     ; GFX11-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
-    ; GFX11-NEXT: [[SCRATCH_LOAD_UBYTE:%[0-9]+]]:vgpr_32 = SCRATCH_LOAD_UBYTE [[COPY]], -4096, 0, implicit $exec, implicit $flat_scr :: (load (s8), addrspace 5)
+    ; GFX11-NEXT: [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 -4096, implicit $exec
+    ; GFX11-NEXT: [[V_ADD_U32_e64_:%[0-9]+]]:vgpr_32 = V_ADD_U32_e64 [[COPY]], [[V_MOV_B32_e32_]], 0, implicit $exec
+    ; GFX11-NEXT: [[SCRATCH_LOAD_UBYTE:%[0-9]+]]:vgpr_32 = SCRATCH_LOAD_UBYTE [[V_ADD_U32_e64_]], 0, 0, implicit $exec, implicit $flat_scr :: (load (s8), addrspace 5)
     ; GFX11-NEXT: $vgpr0 = COPY [[SCRATCH_LOAD_UBYTE]]
     %0:vgpr(p5) = COPY $vgpr0
     %1:vgpr(s32) = G_CONSTANT i32 -4096
diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-store-private.mir b/llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-store-private.mir
index 9ed5369b1a4d9..854739a594751 100644
--- a/llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-store-private.mir
+++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-store-private.mir
@@ -750,7 +750,10 @@ body: |
     ; GFX11-NEXT: {{  $}}
     ; GFX11-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; GFX11-NEXT: [[V_LSHRREV_B32_e64_:%[0-9]+]]:vgpr_32 = V_LSHRREV_B32_e64 5, $sgpr32, implicit $exec
-    ; GFX11-NEXT: SCRATCH_STORE_DWORD [[COPY]], [[V_LSHRREV_B32_e64_]], 4095, 0, implicit $exec, implicit $flat_scr :: (store (s32), addrspace 5)
+    ; GFX11-NEXT: [[S_MOV_B32_:%[0-9]+]]:sreg_32 = S_MOV_B32 4095
+    ; GFX11-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY [[S_MOV_B32_]]
+    ; GFX11-NEXT: [[V_ADD_U32_e64_:%[0-9]+]]:vgpr_32 = V_ADD_U32_e64 [[V_LSHRREV_B32_e64_]], [[COPY1]], 0, implicit $exec
+    ; GFX11-NEXT: SCRATCH_STORE_DWORD [[COPY]], [[V_ADD_U32_e64_]], 0, 0, implicit $exec, implicit $flat_scr :: (store (s32), addrspace 5)
     %0:vgpr(s32) = COPY $vgpr0
     %1:vgpr(p5) = G_AMDGPU_WAVE_ADDRESS $sgpr32
     %2:sgpr(s32) = G_CONSTANT i32 4095
@@ -793,7 +796,9 @@ body: |
     ; GFX11-NEXT: {{  $}}
     ; GFX11-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; GFX11-NEXT: [[V_LSHRREV_B32_e64_:%[0-9]+]]:vgpr_32 = V_LSHRREV_B32_e64 5, $sgpr32, implicit $exec
-    ; GFX11-NEXT: SCRATCH_STORE_DWORD [[COPY]], [[V_LSHRREV_B32_e64_]], 4095, 0, implicit $exec, implicit $flat_scr :: (store (s32), addrspace 5)
+    ; GFX11-NEXT: [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 4095, implicit $exec
+    ; GFX11-NEXT: [[V_ADD_U32_e64_:%[0-9]+]]:vgpr_32 = V_ADD_U32_e64 [[V_LSHRREV_B32_e64_]], [[V_MOV_B32_e32_]], 0, implicit $exec
+    ; GFX11-NEXT: SCRATCH_STORE_DWORD [[COPY]], [[V_ADD_U32_e64_]], 0, 0, implicit $exec, implicit $flat_scr :: (store (s32), addrspace 5)
     %0:vgpr(s32) = COPY $vgpr0
     %1:vgpr(p5) = G_AMDGPU_WAVE_ADDRESS $sgpr32
     %2:vgpr(s32) = G_CONSTANT i32 4095
diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/irtranslator-constantexpr.ll b/llvm/test/CodeGen/AMDGPU/GlobalISel/irtranslator-constantexpr.ll
index 6a0975f7fa72e..9f96b1d0d7192 100644
--- a/llvm/test/CodeGen/AMDGPU/GlobalISel/irtranslator-constantexpr.ll
+++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/irtranslator-constantexpr.ll
@@ -20,43 +20,6 @@ define i32 @test() {
   ret i32 bitcast (<1 x i32> <i32 extractelement (<1 x i32> bitcast (i32 zext (i1 icmp eq (ptr @var, ptr inttoptr (i32 -1 to ptr)) to i32) to <1 x i32>), i64 0)> to i32)
 }
 
-@gint = external addrspace(1) global i8, align 4
-
-; Technically we should be able to fold away the compare to true, but
-; currently constexpr doesn't understand null in non-0 address spaces.
-define amdgpu_kernel void @constantexpr_select_0() {
-  ; CHECK-LABEL: name: constantexpr_select_0
-  ; CHECK: bb.1 (%ir-block.0):
-  ; CHECK-NEXT:   [[GV:%[0-9]+]]:_(p1) = G_GLOBAL_VALUE @gint
-  ; CHECK-NEXT:   [[C:%[0-9]+]]:_(p1) = G_CONSTANT i64 0
-  ; CHECK-NEXT:   [[ICMP:%[0-9]+]]:_(s1) = G_ICMP intpred(eq), [[GV]](p1), [[C]]
-  ; CHECK-NEXT:   [[C1:%[0-9]+]]:_(s32) = G_CONSTANT i32 1
-  ; CHECK-NEXT:   [[C2:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
-  ; CHECK-NEXT:   [[SELECT:%[0-9]+]]:_(s32) = G_SELECT [[ICMP]](s1), [[C1]], [[C2]]
-  ; CHECK-NEXT:   [[DEF:%[0-9]+]]:_(p1) = G_IMPLICIT_DEF
-  ; CHECK-NEXT:   G_STORE [[SELECT]](s32), [[DEF]](p1) :: (store (s32) into `ptr addrspace(1) undef`, addrspace 1)
-  ; CHECK-NEXT:   S_ENDPGM 0
-  store i32 select (i1 icmp eq (ptr addrspace(1) @gint, ptr addrspace(1) null), i32 1, i32 0), ptr addrspace(1) undef, align 4
-  ret void
-}
-
-define amdgpu_kernel void @constantexpr_select_1() {
-  ; CHECK-LABEL: name: constantexpr_select_1
-  ; CHECK: bb.1 (%ir-block.0):
-  ; CHECK-NEXT:   [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 1024
-  ; CHECK-NEXT:   [[INTTOPTR:%[0-9]+]]:_(p1) = G_INTTOPTR [[C]](s64)
-  ; CHECK-NEXT:   [[GV:%[0-9]+]]:_(p1) = G_GLOBAL_VALUE @gint
-  ; CHECK-NEXT:   [[ICMP:%[0-9]+]]:_(s1) = G_ICMP intpred(eq), [[INTTOPTR]](p1), [[GV]]
-  ; CHECK-NEXT:   [[C1:%[0-9]+]]:_(s32) = G_CONSTANT i32 1
-  ; CHECK-NEXT:   [[C2:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
-  ; CHECK-NEXT:   [[SELECT:%[0-9]+]]:_(s32) = G_SELECT [[ICMP]](s1), [[C1]], [[C2]]
-  ; CHECK-NEXT:   [[DEF:%[0-9]+]]:_(p1) = G_IMPLICIT_DEF
-  ; CHECK-NEXT:   G_STORE [[SELECT]](s32), [[DEF]](p1) :: (store (s32) into `ptr addrspace(1) undef`, addrspace 1)
-  ; CHECK-NEXT:   S_ENDPGM 0
-  store i32 select (i1 icmp eq (ptr addrspace(1) @gint, ptr addrspace(1) inttoptr (i64 1024 to ptr addrspace(1))), i32 1, i32 0), ptr addrspace(1) undef, align 4
-  ret void
-}
-
 @a = external global [2 x i32], align 4
 
 define i32 @test_fcmp_constexpr() {
diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/v_bfe_i32.ll b/llvm/test/CodeGen/AMDGPU/GlobalISel/v_bfe_i32.ll
new file mode 100644
index 0000000000000..a07c760576350
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/v_bfe_i32.ll
@@ -0,0 +1,25 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=amdgcn-amd-amdhsa --global-isel -verify-machineinstrs < %s | FileCheck --check-prefix=PREGFX9 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa --global-isel -mcpu=hawaii -verify-machineinstrs < %s | FileCheck --check-prefix=PREGFX9 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa --global-isel -mcpu=fiji -verify-machineinstrs < %s | FileCheck --check-prefix=PREGFX9 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa --global-isel -mcpu=gfx90a -verify-machineinstrs < %s | FileCheck --check-prefix=PREGFX9 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa --global-isel -mcpu=gfx1030 -verify-machineinstrs < %s | FileCheck --check-prefix=GFX10PLUS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa --global-isel -mcpu=gfx1100 -verify-machineinstrs < %s | FileCheck --check-prefix=GFX10PLUS %s
+
+define i32 @check_v_bfe(i16 %a) {
+; PREGFX9-LABEL: check_v_bfe:
+; PREGFX9:       ; %bb.0: ; %entry
+; PREGFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; PREGFX9-NEXT:    v_bfe_i32 v0, v0, 0, 16
+; PREGFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX10PLUS-LABEL: check_v_bfe:
+; GFX10PLUS:       ; %bb.0: ; %entry
+; GFX10PLUS-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10PLUS-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10PLUS-NEXT:    v_bfe_i32 v0, v0, 0, 16
+; GFX10PLUS-NEXT:    s_setpc_b64 s[30:31]
+entry:
+  %res = sext i16 %a to i32
+  ret i32 %res
+}
diff --git a/llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-idiv.ll b/llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-idiv.ll
index 92f5d0a2d4057..cd38bf181fe08 100644
--- a/llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-idiv.ll
+++ b/llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-idiv.ll
@@ -4706,7 +4706,7 @@ define amdgpu_kernel void @udiv_v3i15(ptr addrspace(1) %out, <3 x i15> %x, <3 x
 ; GFX6-NEXT:    v_and_b32_e32 v3, 0x7fff, v4
 ; GFX6-NEXT:    v_lshl_b64 v[0:1], v[0:1], 30
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v3, 15, v3
-; GFX6-NEXT:    v_or_b32_e32 v2, v2, v3
+; GFX6-NEXT:    v_or_b32_e32 v2, v3, v2
 ; GFX6-NEXT:    v_or_b32_e32 v0, v2, v0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_waitcnt expcnt(0)
@@ -4901,7 +4901,7 @@ define amdgpu_kernel void @urem_v3i15(ptr addrspace(1) %out, <3 x i15> %x, <3 x
 ; GFX6-NEXT:    v_lshl_b64 v[0:1], v[0:1], 30
 ; GFX6-NEXT:    v_and_b32_e32 v2, 0x7fff, v6
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v3, 15, v3
-; GFX6-NEXT:    v_or_b32_e32 v2, v2, v3
+; GFX6-NEXT:    v_or_b32_e32 v2, v3, v2
 ; GFX6-NEXT:    v_or_b32_e32 v0, v2, v0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_waitcnt expcnt(0)
@@ -5116,7 +5116,7 @@ define amdgpu_kernel void @sdiv_v3i15(ptr addrspace(1) %out, <3 x i15> %x, <3 x
 ; GFX6-NEXT:    v_and_b32_e32 v3, 0x7fff, v4
 ; GFX6-NEXT:    v_lshl_b64 v[0:1], v[0:1], 30
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v3, 15, v3
-; GFX6-NEXT:    v_or_b32_e32 v2, v2, v3
+; GFX6-NEXT:    v_or_b32_e32 v2, v3, v2
 ; GFX6-NEXT:    v_or_b32_e32 v0, v2, v0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_waitcnt expcnt(0)
@@ -5353,7 +5353,7 @@ define amdgpu_kernel void @srem_v3i15(ptr addrspace(1) %out, <3 x i15> %x, <3 x
 ; GFX6-NEXT:    v_lshl_b64 v[0:1], v[0:1], 30
 ; GFX6-NEXT:    v_and_b32_e32 v3, 0x7fff, v4
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v2, 15, v2
-; GFX6-NEXT:    v_or_b32_e32 v2, v3, v2
+; GFX6-NEXT:    v_or_b32_e32 v2, v2, v3
 ; GFX6-NEXT:    v_or_b32_e32 v0, v2, v0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_waitcnt expcnt(0)
diff --git a/llvm/test/CodeGen/AMDGPU/amdpal-callable.ll b/llvm/test/CodeGen/AMDGPU/amdpal-callable.ll
index 14db2ab9c419c..824adbecfc3ae 100644
--- a/llvm/test/CodeGen/AMDGPU/amdpal-callable.ll
+++ b/llvm/test/CodeGen/AMDGPU/amdpal-callable.ll
@@ -1,6 +1,6 @@
-; RUN: llc -mtriple=amdgcn--amdpal -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,SDAG,GFX8 -enable-var-scope %s
-; RUN: llc -mtriple=amdgcn--amdpal -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,SDAG,GFX9 -enable-var-scope %s
-; RUN: llc -global-isel -mtriple=amdgcn--amdpal -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GISEL,GFX9 -enable-var-scope %s
+; RUN: llc -mtriple=amdgcn--amdpal -mattr=-xnack -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,SDAG,GFX8 -enable-var-scope %s
+; RUN: llc -mtriple=amdgcn--amdpal -mcpu=gfx900 -mattr=-xnack -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,SDAG,GFX9 -enable-var-scope %s
+; RUN: llc -global-isel -mtriple=amdgcn--amdpal -mattr=-xnack -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GISEL,GFX9 -enable-var-scope %s
 
 declare amdgpu_gfx float @extern_func(float) #0
 declare amdgpu_gfx float @extern_func_many_args(<64 x float>) #0
diff --git a/llvm/test/CodeGen/AMDGPU/chain-hi-to-lo.ll b/llvm/test/CodeGen/AMDGPU/chain-hi-to-lo.ll
index c3684e24dc2aa..35794c9737860 100644
--- a/llvm/test/CodeGen/AMDGPU/chain-hi-to-lo.ll
+++ b/llvm/test/CodeGen/AMDGPU/chain-hi-to-lo.ll
@@ -734,11 +734,11 @@ define <2 x i16> @chain_hi_to_lo_private_other_dep(ptr addrspace(5) %ptr) {
 ; FLATSCR:       ; %bb.0: ; %bb
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; FLATSCR-NEXT:    scratch_load_short_d16_hi v1, v0, off
+; FLATSCR-NEXT:    v_add_u32_e32 v2, 2, v0
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
-; FLATSCR-NEXT:    v_pk_sub_u16 v1, v1, -12 op_sel_hi:[1,0]
-; FLATSCR-NEXT:    scratch_load_short_d16 v1, v0, off offset:2
+; FLATSCR-NEXT:    v_pk_sub_u16 v0, v1, -12 op_sel_hi:[1,0]
+; FLATSCR-NEXT:    scratch_load_short_d16 v0, v2, off
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
-; FLATSCR-NEXT:    v_mov_b32_e32 v0, v1
 ; FLATSCR-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX10_DEFAULT-LABEL: chain_hi_to_lo_private_other_dep:
@@ -758,11 +758,11 @@ define <2 x i16> @chain_hi_to_lo_private_other_dep(ptr addrspace(5) %ptr) {
 ; FLATSCR_GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; FLATSCR_GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; FLATSCR_GFX10-NEXT:    scratch_load_short_d16_hi v1, v0, off
+; FLATSCR_GFX10-NEXT:    v_add_nc_u32_e32 v2, 2, v0
 ; FLATSCR_GFX10-NEXT:    s_waitcnt vmcnt(0)
-; FLATSCR_GFX10-NEXT:    v_pk_sub_u16 v1, v1, -12 op_sel_hi:[1,0]
-; FLATSCR_GFX10-NEXT:    scratch_load_short_d16 v1, v0, off offset:2
+; FLATSCR_GFX10-NEXT:    v_pk_sub_u16 v0, v1, -12 op_sel_hi:[1,0]
+; FLATSCR_GFX10-NEXT:    scratch_load_short_d16 v0, v2, off
 ; FLATSCR_GFX10-NEXT:    s_waitcnt vmcnt(0)
-; FLATSCR_GFX10-NEXT:    v_mov_b32_e32 v0, v1
 ; FLATSCR_GFX10-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX11-LABEL: chain_hi_to_lo_private_other_dep:
@@ -770,11 +770,11 @@ define <2 x i16> @chain_hi_to_lo_private_other_dep(ptr addrspace(5) %ptr) {
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    scratch_load_d16_hi_b16 v1, v0, off
+; GFX11-NEXT:    v_add_nc_u32_e32 v2, 2, v0
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_pk_sub_u16 v1, v1, -12 op_sel_hi:[1,0]
-; GFX11-NEXT:    scratch_load_d16_b16 v1, v0, off offset:2
+; GFX11-NEXT:    v_pk_sub_u16 v0, v1, -12 op_sel_hi:[1,0]
+; GFX11-NEXT:    scratch_load_d16_b16 v0, v2, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_mov_b32_e32 v0, v1
 ; GFX11-NEXT:    s_setpc_b64 s[30:31]
 bb:
   %gep_lo = getelementptr inbounds i16, ptr addrspace(5) %ptr, i64 1
diff --git a/llvm/test/CodeGen/AMDGPU/combine-add-zext-xor.ll b/llvm/test/CodeGen/AMDGPU/combine-add-zext-xor.ll
index eaede6b6c4a6a..7c17a3dbdbfd0 100644
--- a/llvm/test/CodeGen/AMDGPU/combine-add-zext-xor.ll
+++ b/llvm/test/CodeGen/AMDGPU/combine-add-zext-xor.ll
@@ -173,54 +173,59 @@ define i32 @combine_add_zext_or() {
 ; GFX1010:       ; %bb.0: ; %.entry
 ; GFX1010-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX1010-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1010-NEXT:    v_mov_b32_e32 v1, 0
+; GFX1010-NEXT:    s_mov_b32 s4, 0
 ; GFX1010-NEXT:    s_branch .LBB2_2
 ; GFX1010-NEXT:  .LBB2_1: ; %bb9
 ; GFX1010-NEXT:    ; in Loop: Header=BB2_2 Depth=1
-; GFX1010-NEXT:    v_cmp_lt_i32_e32 vcc_lo, 0xfffffbe6, v1
-; GFX1010-NEXT:    s_or_b32 s4, s4, vcc_lo
-; GFX1010-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s4
-; GFX1010-NEXT:    v_add_nc_u32_e32 v1, v1, v0
+; GFX1010-NEXT:    s_cmpk_gt_i32 s4, 0xfbe6
+; GFX1010-NEXT:    s_cselect_b32 s6, -1, 0
+; GFX1010-NEXT:    s_add_i32 s4, s4, 1
+; GFX1010-NEXT:    s_and_b32 vcc_lo, exec_lo, s6
 ; GFX1010-NEXT:    s_cbranch_vccz .LBB2_4
 ; GFX1010-NEXT:  .LBB2_2: ; %.a
 ; GFX1010-NEXT:    ; =>This Inner Loop Header: Depth=1
-; GFX1010-NEXT:    ; implicit-def: $sgpr4
+; GFX1010-NEXT:    ; implicit-def: $sgpr5
 ; GFX1010-NEXT:    s_cbranch_scc1 .LBB2_1
 ; GFX1010-NEXT:  ; %bb.3: ; %bb
 ; GFX1010-NEXT:    ; in Loop: Header=BB2_2 Depth=1
-; GFX1010-NEXT:    buffer_load_dword v0, v1, s[4:7], 64 offen glc
+; GFX1010-NEXT:    v_mov_b32_e32 v0, s4
+; GFX1010-NEXT:    buffer_load_dword v0, v0, s[4:7], 64 offen glc
 ; GFX1010-NEXT:    s_waitcnt vmcnt(0)
-; GFX1010-NEXT:    v_cmp_eq_u32_e64 s4, 0, v0
+; GFX1010-NEXT:    v_cmp_eq_u32_e64 s5, 0, v0
 ; GFX1010-NEXT:    s_branch .LBB2_1
 ; GFX1010-NEXT:  .LBB2_4: ; %.exit
+; GFX1010-NEXT:    s_or_b32 s4, s5, s6
+; GFX1010-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s4
 ; GFX1010-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX1100-LABEL: combine_add_zext_or:
 ; GFX1100:       ; %bb.0: ; %.entry
 ; GFX1100-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX1100-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1100-NEXT:    v_mov_b32_e32 v1, 0
+; GFX1100-NEXT:    s_mov_b32 s0, 0
 ; GFX1100-NEXT:    s_branch .LBB2_2
 ; GFX1100-NEXT:  .LBB2_1: ; %bb9
 ; GFX1100-NEXT:    ; in Loop: Header=BB2_2 Depth=1
-; GFX1100-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX1100-NEXT:    v_cmp_lt_i32_e32 vcc_lo, 0xfffffbe6, v1
-; GFX1100-NEXT:    s_or_b32 s0, s0, vcc_lo
-; GFX1100-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1100-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
-; GFX1100-NEXT:    v_add_nc_u32_e32 v1, v1, v0
+; GFX1100-NEXT:    s_cmpk_gt_i32 s0, 0xfbe6
+; GFX1100-NEXT:    s_cselect_b32 s2, -1, 0
+; GFX1100-NEXT:    s_add_i32 s0, s0, 1
+; GFX1100-NEXT:    s_and_b32 vcc_lo, exec_lo, s2
 ; GFX1100-NEXT:    s_cbranch_vccz .LBB2_4
 ; GFX1100-NEXT:  .LBB2_2: ; %.a
 ; GFX1100-NEXT:    ; =>This Inner Loop Header: Depth=1
-; GFX1100-NEXT:    ; implicit-def: $sgpr0
+; GFX1100-NEXT:    ; implicit-def: $sgpr1
 ; GFX1100-NEXT:    s_cbranch_scc1 .LBB2_1
 ; GFX1100-NEXT:  ; %bb.3: ; %bb
 ; GFX1100-NEXT:    ; in Loop: Header=BB2_2 Depth=1
-; GFX1100-NEXT:    buffer_load_b32 v0, v1, s[0:3], 64 offen glc
+; GFX1100-NEXT:    v_mov_b32_e32 v0, s0
+; GFX1100-NEXT:    buffer_load_b32 v0, v0, s[0:3], 64 offen glc
 ; GFX1100-NEXT:    s_waitcnt vmcnt(0)
-; GFX1100-NEXT:    v_cmp_eq_u32_e64 s0, 0, v0
+; GFX1100-NEXT:    v_cmp_eq_u32_e64 s1, 0, v0
 ; GFX1100-NEXT:    s_branch .LBB2_1
 ; GFX1100-NEXT:  .LBB2_4: ; %.exit
+; GFX1100-NEXT:    s_or_b32 s0, s1, s2
+; GFX1100-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1100-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
 ; GFX1100-NEXT:    s_setpc_b64 s[30:31]
 .entry:
   br label %.a
@@ -254,54 +259,59 @@ define i32 @combine_sub_zext_or() {
 ; GFX1010:       ; %bb.0: ; %.entry
 ; GFX1010-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX1010-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1010-NEXT:    v_mov_b32_e32 v1, 0
+; GFX1010-NEXT:    s_mov_b32 s4, 0
 ; GFX1010-NEXT:    s_branch .LBB3_2
 ; GFX1010-NEXT:  .LBB3_1: ; %bb9
 ; GFX1010-NEXT:    ; in Loop: Header=BB3_2 Depth=1
-; GFX1010-NEXT:    v_cmp_lt_i32_e32 vcc_lo, 0xfffffbe6, v1
-; GFX1010-NEXT:    s_or_b32 s4, s4, vcc_lo
-; GFX1010-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s4
-; GFX1010-NEXT:    v_sub_nc_u32_e32 v1, v1, v0
+; GFX1010-NEXT:    s_cmpk_gt_i32 s4, 0xfbe6
+; GFX1010-NEXT:    s_cselect_b32 s6, -1, 0
+; GFX1010-NEXT:    s_add_i32 s4, s4, -1
+; GFX1010-NEXT:    s_and_b32 vcc_lo, exec_lo, s6
 ; GFX1010-NEXT:    s_cbranch_vccz .LBB3_4
 ; GFX1010-NEXT:  .LBB3_2: ; %.a
 ; GFX1010-NEXT:    ; =>This Inner Loop Header: Depth=1
-; GFX1010-NEXT:    ; implicit-def: $sgpr4
+; GFX1010-NEXT:    ; implicit-def: $sgpr5
 ; GFX1010-NEXT:    s_cbranch_scc1 .LBB3_1
 ; GFX1010-NEXT:  ; %bb.3: ; %bb
 ; GFX1010-NEXT:    ; in Loop: Header=BB3_2 Depth=1
-; GFX1010-NEXT:    buffer_load_dword v0, v1, s[4:7], 64 offen glc
+; GFX1010-NEXT:    v_mov_b32_e32 v0, s4
+; GFX1010-NEXT:    buffer_load_dword v0, v0, s[4:7], 64 offen glc
 ; GFX1010-NEXT:    s_waitcnt vmcnt(0)
-; GFX1010-NEXT:    v_cmp_eq_u32_e64 s4, 0, v0
+; GFX1010-NEXT:    v_cmp_eq_u32_e64 s5, 0, v0
 ; GFX1010-NEXT:    s_branch .LBB3_1
 ; GFX1010-NEXT:  .LBB3_4: ; %.exit
+; GFX1010-NEXT:    s_or_b32 s4, s5, s6
+; GFX1010-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s4
 ; GFX1010-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX1100-LABEL: combine_sub_zext_or:
 ; GFX1100:       ; %bb.0: ; %.entry
 ; GFX1100-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX1100-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1100-NEXT:    v_mov_b32_e32 v1, 0
+; GFX1100-NEXT:    s_mov_b32 s0, 0
 ; GFX1100-NEXT:    s_branch .LBB3_2
 ; GFX1100-NEXT:  .LBB3_1: ; %bb9
 ; GFX1100-NEXT:    ; in Loop: Header=BB3_2 Depth=1
-; GFX1100-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX1100-NEXT:    v_cmp_lt_i32_e32 vcc_lo, 0xfffffbe6, v1
-; GFX1100-NEXT:    s_or_b32 s0, s0, vcc_lo
-; GFX1100-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1100-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
-; GFX1100-NEXT:    v_sub_nc_u32_e32 v1, v1, v0
+; GFX1100-NEXT:    s_cmpk_gt_i32 s0, 0xfbe6
+; GFX1100-NEXT:    s_cselect_b32 s2, -1, 0
+; GFX1100-NEXT:    s_add_i32 s0, s0, -1
+; GFX1100-NEXT:    s_and_b32 vcc_lo, exec_lo, s2
 ; GFX1100-NEXT:    s_cbranch_vccz .LBB3_4
 ; GFX1100-NEXT:  .LBB3_2: ; %.a
 ; GFX1100-NEXT:    ; =>This Inner Loop Header: Depth=1
-; GFX1100-NEXT:    ; implicit-def: $sgpr0
+; GFX1100-NEXT:    ; implicit-def: $sgpr1
 ; GFX1100-NEXT:    s_cbranch_scc1 .LBB3_1
 ; GFX1100-NEXT:  ; %bb.3: ; %bb
 ; GFX1100-NEXT:    ; in Loop: Header=BB3_2 Depth=1
-; GFX1100-NEXT:    buffer_load_b32 v0, v1, s[0:3], 64 offen glc
+; GFX1100-NEXT:    v_mov_b32_e32 v0, s0
+; GFX1100-NEXT:    buffer_load_b32 v0, v0, s[0:3], 64 offen glc
 ; GFX1100-NEXT:    s_waitcnt vmcnt(0)
-; GFX1100-NEXT:    v_cmp_eq_u32_e64 s0, 0, v0
+; GFX1100-NEXT:    v_cmp_eq_u32_e64 s1, 0, v0
 ; GFX1100-NEXT:    s_branch .LBB3_1
 ; GFX1100-NEXT:  .LBB3_4: ; %.exit
+; GFX1100-NEXT:    s_or_b32 s0, s1, s2
+; GFX1100-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1100-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
 ; GFX1100-NEXT:    s_setpc_b64 s[30:31]
 .entry:
   br label %.a
@@ -496,5 +506,3 @@ declare i32 @llvm.amdgcn.raw.buffer.load.i32(<4 x i32>, i32, i32, i32 immarg) #0
 
 attributes #0 = { nounwind readonly willreturn }
 
-;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
-; COMMON: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/ctpop16.ll b/llvm/test/CodeGen/AMDGPU/ctpop16.ll
index 1b2bca5cdc00d..0acd24dc5e3a0 100644
--- a/llvm/test/CodeGen/AMDGPU/ctpop16.ll
+++ b/llvm/test/CodeGen/AMDGPU/ctpop16.ll
@@ -457,54 +457,50 @@ define amdgpu_kernel void @v_ctpop_v4i16(ptr addrspace(1) noalias %out, ptr addr
 ;
 ; EG-LABEL: v_ctpop_v4i16:
 ; EG:       ; %bb.0:
-; EG-NEXT:    ALU 2, @8, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    ALU 3, @8, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    TEX 0 @6
-; EG-NEXT:    ALU 42, @11, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    ALU 37, @12, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T8.XY, T0.X, 1
 ; EG-NEXT:    CF_END
 ; EG-NEXT:    PAD
 ; EG-NEXT:    Fetch clause starting at 6:
-; EG-NEXT:     VTX_READ_64 T0.XY, T0.X, 0, #1
+; EG-NEXT:     VTX_READ_64 T8.XY, T0.X, 0, #1
 ; EG-NEXT:    ALU clause starting at 8:
-; EG-NEXT:     LSHL * T0.W, T0.X, literal.x,
+; EG-NEXT:     MOV T0.Y, T4.X,
+; EG-NEXT:     LSHL * T0.W, T0.X, literal.x, BS:VEC_120/SCL_212
 ; EG-NEXT:    3(4.203895e-45), 0(0.000000e+00)
 ; EG-NEXT:     ADD_INT * T0.X, KC0[2].Z, PV.W,
-; EG-NEXT:    ALU clause starting at 11:
-; EG-NEXT:     MOV T2.X, T0.X,
-; EG-NEXT:     MOV * T3.X, T0.Y,
-; EG-NEXT:     MOV T0.X, T4.X,
-; EG-NEXT:     MOV * T0.Y, PV.X,
-; EG-NEXT:     AND_INT * T0.W, PV.Y, literal.x,
+; EG-NEXT:    ALU clause starting at 12:
+; EG-NEXT:     AND_INT * T0.W, T8.X, literal.x,
 ; EG-NEXT:    65535(9.183409e-41), 0(0.000000e+00)
 ; EG-NEXT:     BCNT_INT T0.W, PV.W,
-; EG-NEXT:     AND_INT * T1.W, T0.X, literal.x,
+; EG-NEXT:     AND_INT * T1.W, T0.Y, literal.x,
 ; EG-NEXT:    -65536(nan), 0(0.000000e+00)
 ; EG-NEXT:     OR_INT * T0.W, PS, PV.W,
-; EG-NEXT:     MOV T0.X, T3.X,
 ; EG-NEXT:     MOV * T4.X, PV.W,
-; EG-NEXT:     MOV T0.Z, PS,
-; EG-NEXT:     LSHR * T0.W, T0.Y, literal.x,
+; EG-NEXT:     MOV T0.X, PV.X,
+; EG-NEXT:     LSHR * T0.W, T8.X, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
 ; EG-NEXT:     BCNT_INT T0.W, PV.W,
-; EG-NEXT:     AND_INT * T1.W, PV.Z, literal.x,
+; EG-NEXT:     AND_INT * T1.W, PV.X, literal.x,
 ; EG-NEXT:    65535(9.183409e-41), 0(0.000000e+00)
 ; EG-NEXT:     LSHL * T0.W, PV.W, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
 ; EG-NEXT:     OR_INT * T0.W, T1.W, PV.W,
 ; EG-NEXT:     MOV T4.X, PV.W,
-; EG-NEXT:     MOV T0.Y, T5.X,
-; EG-NEXT:     AND_INT * T0.W, T0.X, literal.x, BS:VEC_120/SCL_212
+; EG-NEXT:     MOV * T0.X, T5.X,
+; EG-NEXT:     AND_INT * T0.W, T8.Y, literal.x,
 ; EG-NEXT:    65535(9.183409e-41), 0(0.000000e+00)
 ; EG-NEXT:     BCNT_INT T0.W, PV.W,
-; EG-NEXT:     AND_INT * T1.W, PV.Y, literal.x,
+; EG-NEXT:     AND_INT * T1.W, T0.X, literal.x,
 ; EG-NEXT:    -65536(nan), 0(0.000000e+00)
 ; EG-NEXT:     OR_INT * T0.W, PS, PV.W,
 ; EG-NEXT:     MOV * T5.X, PV.W,
-; EG-NEXT:     MOV T0.Y, PV.X,
-; EG-NEXT:     LSHR * T0.W, T0.X, literal.x,
+; EG-NEXT:     MOV T0.X, PV.X,
+; EG-NEXT:     LSHR * T0.W, T8.Y, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
 ; EG-NEXT:     BCNT_INT T0.W, PV.W,
-; EG-NEXT:     AND_INT * T1.W, PV.Y, literal.x,
+; EG-NEXT:     AND_INT * T1.W, PV.X, literal.x,
 ; EG-NEXT:    65535(9.183409e-41), 0(0.000000e+00)
 ; EG-NEXT:     LSHL * T0.W, PV.W, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
diff --git a/llvm/test/CodeGen/AMDGPU/divergence-at-use.ll b/llvm/test/CodeGen/AMDGPU/divergence-at-use.ll
index 201691ad968b7..0c3470bebb0ff 100644
--- a/llvm/test/CodeGen/AMDGPU/divergence-at-use.ll
+++ b/llvm/test/CodeGen/AMDGPU/divergence-at-use.ll
@@ -1,12 +1,12 @@
 ; RUN: llc -march=amdgcn -mcpu=gfx900 -amdgpu-atomic-optimizations=true < %s | FileCheck %s
-; RUN: llc -march=amdgcn -mcpu=gfx900 -amdgpu-atomic-optimizations=true < %s -use-gpu-divergence-analysis | FileCheck %s
 
 @local = addrspace(3) global i32 undef
 
-define amdgpu_kernel void @reducible(i32 %x) {
+define amdgpu_kernel void @reducible() {
 ; CHECK-LABEL: reducible:
 ; CHECK-NOT: dpp
 entry:
+  %x = call i32 @llvm.amdgcn.workitem.id.x()
   br label %loop
 loop:
   %i = phi i32 [ 0, %entry ], [ %i1, %loop ]
@@ -18,3 +18,27 @@ exit:
   %old = atomicrmw add ptr addrspace(3) %gep, i32 %x acq_rel
   ret void
 }
+
+define amdgpu_kernel void @def_in_nested_cycle() {
+; CHECK-LABEL: def_in_nested_cycle:
+; CHECK-NOT: dpp
+entry:
+  %x = call i32 @llvm.amdgcn.workitem.id.x()
+  br label %loop
+loop:
+  %i = phi i32 [ 0, %entry ], [ 0, %innerloop ], [ %i1, %loop ]
+  %cond = icmp ult i32 %i, %x
+  %i1 = add i32 %i, 1
+  br i1 %cond, label %innerloop, label %loop
+innerloop:
+  %i.inner = phi i32 [ 0, %loop ], [ %i1.inner, %innerloop ]
+  %gep = getelementptr i32, ptr addrspace(3) @local, i32 %i
+  %i1.inner = add i32 %i, 1
+  %cond.inner = icmp ult i32 %i, %x
+  br i1 %cond, label %innerloop, label %loop
+exit:
+  %old = atomicrmw add ptr addrspace(3) %gep, i32 %x acq_rel
+  ret void
+}
+
+declare i32 @llvm.amdgcn.workitem.id.x()
diff --git a/llvm/test/CodeGen/AMDGPU/fast-unaligned-load-store.private.ll b/llvm/test/CodeGen/AMDGPU/fast-unaligned-load-store.private.ll
index 8184764aa352b..596daadd277ba 100644
--- a/llvm/test/CodeGen/AMDGPU/fast-unaligned-load-store.private.ll
+++ b/llvm/test/CodeGen/AMDGPU/fast-unaligned-load-store.private.ll
@@ -46,10 +46,11 @@ define i32 @private_load_2xi16_align2(ptr addrspace(5) %p) #0 {
 ; GFX9-FLASTSCR-LABEL: private_load_2xi16_align2:
 ; GFX9-FLASTSCR:       ; %bb.0:
 ; GFX9-FLASTSCR-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX9-FLASTSCR-NEXT:    scratch_load_ushort v1, v0, off
-; GFX9-FLASTSCR-NEXT:    scratch_load_ushort v2, v0, off offset:2
+; GFX9-FLASTSCR-NEXT:    v_add_u32_e32 v1, 2, v0
+; GFX9-FLASTSCR-NEXT:    scratch_load_ushort v2, v0, off
+; GFX9-FLASTSCR-NEXT:    scratch_load_ushort v3, v1, off
 ; GFX9-FLASTSCR-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-FLASTSCR-NEXT:    v_lshl_or_b32 v0, v2, 16, v1
+; GFX9-FLASTSCR-NEXT:    v_lshl_or_b32 v0, v3, 16, v2
 ; GFX9-FLASTSCR-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX10-LABEL: private_load_2xi16_align2:
@@ -67,33 +68,36 @@ define i32 @private_load_2xi16_align2(ptr addrspace(5) %p) #0 {
 ; GFX10-FLASTSCR:       ; %bb.0:
 ; GFX10-FLASTSCR-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-FLASTSCR-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-FLASTSCR-NEXT:    v_add_nc_u32_e32 v1, 2, v0
 ; GFX10-FLASTSCR-NEXT:    s_clause 0x1
-; GFX10-FLASTSCR-NEXT:    scratch_load_ushort v1, v0, off
-; GFX10-FLASTSCR-NEXT:    scratch_load_ushort v2, v0, off offset:2
+; GFX10-FLASTSCR-NEXT:    scratch_load_ushort v2, v0, off
+; GFX10-FLASTSCR-NEXT:    scratch_load_ushort v3, v1, off
 ; GFX10-FLASTSCR-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-FLASTSCR-NEXT:    v_lshl_or_b32 v0, v2, 16, v1
+; GFX10-FLASTSCR-NEXT:    v_lshl_or_b32 v0, v3, 16, v2
 ; GFX10-FLASTSCR-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX11-LABEL: private_load_2xi16_align2:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11-NEXT:    v_add_nc_u32_e32 v1, 2, v0
 ; GFX11-NEXT:    s_clause 0x1
-; GFX11-NEXT:    scratch_load_u16 v1, v0, off
-; GFX11-NEXT:    scratch_load_u16 v0, v0, off offset:2
+; GFX11-NEXT:    scratch_load_u16 v0, v0, off
+; GFX11-NEXT:    scratch_load_u16 v1, v1, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_lshl_or_b32 v0, v0, 16, v1
+; GFX11-NEXT:    v_lshl_or_b32 v0, v1, 16, v0
 ; GFX11-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX11-FLASTSCR-LABEL: private_load_2xi16_align2:
 ; GFX11-FLASTSCR:       ; %bb.0:
 ; GFX11-FLASTSCR-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-FLASTSCR-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11-FLASTSCR-NEXT:    v_add_nc_u32_e32 v1, 2, v0
 ; GFX11-FLASTSCR-NEXT:    s_clause 0x1
-; GFX11-FLASTSCR-NEXT:    scratch_load_u16 v1, v0, off
-; GFX11-FLASTSCR-NEXT:    scratch_load_u16 v0, v0, off offset:2
+; GFX11-FLASTSCR-NEXT:    scratch_load_u16 v0, v0, off
+; GFX11-FLASTSCR-NEXT:    scratch_load_u16 v1, v1, off
 ; GFX11-FLASTSCR-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-FLASTSCR-NEXT:    v_lshl_or_b32 v0, v0, 16, v1
+; GFX11-FLASTSCR-NEXT:    v_lshl_or_b32 v0, v1, 16, v0
 ; GFX11-FLASTSCR-NEXT:    s_setpc_b64 s[30:31]
   %gep.p = getelementptr i16, ptr addrspace(5) %p, i64 1
   %p.0 = load i16, ptr addrspace(5) %p, align 2
@@ -142,10 +146,11 @@ define void @private_store_2xi16_align2(ptr addrspace(5) %p, ptr addrspace(5) %r
 ; GFX9-FLASTSCR-LABEL: private_store_2xi16_align2:
 ; GFX9-FLASTSCR:       ; %bb.0:
 ; GFX9-FLASTSCR-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX9-FLASTSCR-NEXT:    v_mov_b32_e32 v0, 1
-; GFX9-FLASTSCR-NEXT:    scratch_store_short v1, v0, off
-; GFX9-FLASTSCR-NEXT:    v_mov_b32_e32 v0, 2
-; GFX9-FLASTSCR-NEXT:    scratch_store_short v1, v0, off offset:2
+; GFX9-FLASTSCR-NEXT:    v_mov_b32_e32 v2, 1
+; GFX9-FLASTSCR-NEXT:    v_add_u32_e32 v0, 2, v1
+; GFX9-FLASTSCR-NEXT:    scratch_store_short v1, v2, off
+; GFX9-FLASTSCR-NEXT:    v_mov_b32_e32 v1, 2
+; GFX9-FLASTSCR-NEXT:    scratch_store_short v0, v1, off
 ; GFX9-FLASTSCR-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-FLASTSCR-NEXT:    s_setpc_b64 s[30:31]
 ;
@@ -165,9 +170,10 @@ define void @private_store_2xi16_align2(ptr addrspace(5) %p, ptr addrspace(5) %r
 ; GFX10-FLASTSCR-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-FLASTSCR-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-FLASTSCR-NEXT:    v_mov_b32_e32 v0, 1
-; GFX10-FLASTSCR-NEXT:    v_mov_b32_e32 v2, 2
+; GFX10-FLASTSCR-NEXT:    v_add_nc_u32_e32 v2, 2, v1
+; GFX10-FLASTSCR-NEXT:    v_mov_b32_e32 v3, 2
 ; GFX10-FLASTSCR-NEXT:    scratch_store_short v1, v0, off
-; GFX10-FLASTSCR-NEXT:    scratch_store_short v1, v2, off offset:2
+; GFX10-FLASTSCR-NEXT:    scratch_store_short v2, v3, off
 ; GFX10-FLASTSCR-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-FLASTSCR-NEXT:    s_setpc_b64 s[30:31]
 ;
@@ -175,11 +181,11 @@ define void @private_store_2xi16_align2(ptr addrspace(5) %p, ptr addrspace(5) %r
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    v_mov_b32_e32 v0, 1
-; GFX11-NEXT:    v_mov_b32_e32 v2, 2
+; GFX11-NEXT:    v_dual_mov_b32 v0, 1 :: v_dual_mov_b32 v3, 2
+; GFX11-NEXT:    v_add_nc_u32_e32 v2, 2, v1
 ; GFX11-NEXT:    s_clause 0x1
 ; GFX11-NEXT:    scratch_store_b16 v1, v0, off
-; GFX11-NEXT:    scratch_store_b16 v1, v2, off offset:2
+; GFX11-NEXT:    scratch_store_b16 v2, v3, off
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    s_setpc_b64 s[30:31]
 ;
@@ -187,11 +193,11 @@ define void @private_store_2xi16_align2(ptr addrspace(5) %p, ptr addrspace(5) %r
 ; GFX11-FLASTSCR:       ; %bb.0:
 ; GFX11-FLASTSCR-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-FLASTSCR-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-FLASTSCR-NEXT:    v_mov_b32_e32 v0, 1
-; GFX11-FLASTSCR-NEXT:    v_mov_b32_e32 v2, 2
+; GFX11-FLASTSCR-NEXT:    v_dual_mov_b32 v0, 1 :: v_dual_mov_b32 v3, 2
+; GFX11-FLASTSCR-NEXT:    v_add_nc_u32_e32 v2, 2, v1
 ; GFX11-FLASTSCR-NEXT:    s_clause 0x1
 ; GFX11-FLASTSCR-NEXT:    scratch_store_b16 v1, v0, off
-; GFX11-FLASTSCR-NEXT:    scratch_store_b16 v1, v2, off offset:2
+; GFX11-FLASTSCR-NEXT:    scratch_store_b16 v2, v3, off
 ; GFX11-FLASTSCR-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-FLASTSCR-NEXT:    s_setpc_b64 s[30:31]
   %gep.r = getelementptr i16, ptr addrspace(5) %r, i64 1
diff --git a/llvm/test/CodeGen/AMDGPU/fcopysign.f16.ll b/llvm/test/CodeGen/AMDGPU/fcopysign.f16.ll
index 396ae55313edb..1103c18b72b89 100644
--- a/llvm/test/CodeGen/AMDGPU/fcopysign.f16.ll
+++ b/llvm/test/CodeGen/AMDGPU/fcopysign.f16.ll
@@ -872,22 +872,24 @@ define amdgpu_kernel void @v_copysign_out_f64_mag_f64_sign_f16(ptr addrspace(1)
 ; SI-NEXT:    s_mov_b32 s11, 0xf000
 ; SI-NEXT:    s_mov_b32 s14, 0
 ; SI-NEXT:    s_mov_b32 s15, s11
+; SI-NEXT:    v_mov_b32_e32 v1, 0
+; SI-NEXT:    s_mov_b64 s[2:3], s[14:15]
+; SI-NEXT:    v_lshlrev_b32_e32 v2, 1, v0
+; SI-NEXT:    v_mov_b32_e32 v3, v1
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    buffer_load_ushort v2, v[2:3], s[0:3], 0 addr64
 ; SI-NEXT:    s_mov_b64 s[12:13], s[6:7]
-; SI-NEXT:    v_lshlrev_b32_e32 v1, 3, v0
-; SI-NEXT:    v_mov_b32_e32 v2, 0
-; SI-NEXT:    buffer_load_dwordx2 v[3:4], v[1:2], s[12:15], 0 addr64
-; SI-NEXT:    s_mov_b64 s[2:3], s[14:15]
-; SI-NEXT:    v_lshlrev_b32_e32 v1, 1, v0
-; SI-NEXT:    buffer_load_ushort v0, v[1:2], s[0:3], 0 addr64
+; SI-NEXT:    v_lshlrev_b32_e32 v0, 3, v0
+; SI-NEXT:    buffer_load_dwordx2 v[0:1], v[0:1], s[12:15], 0 addr64
 ; SI-NEXT:    s_brev_b32 s0, -2
 ; SI-NEXT:    s_mov_b32 s10, -1
 ; SI-NEXT:    s_mov_b32 s8, s4
 ; SI-NEXT:    s_mov_b32 s9, s5
+; SI-NEXT:    s_waitcnt vmcnt(1)
+; SI-NEXT:    v_cvt_f32_f16_e32 v2, v2
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    v_cvt_f32_f16_e32 v0, v0
-; SI-NEXT:    v_bfi_b32 v4, s0, v4, v0
-; SI-NEXT:    buffer_store_dwordx2 v[3:4], off, s[8:11], 0
+; SI-NEXT:    v_bfi_b32 v1, s0, v1, v2
+; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[8:11], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: v_copysign_out_f64_mag_f64_sign_f16:
diff --git a/llvm/test/CodeGen/AMDGPU/fcopysign.f64.ll b/llvm/test/CodeGen/AMDGPU/fcopysign.f64.ll
index 97dd018b677c2..62ffd8818e41b 100644
--- a/llvm/test/CodeGen/AMDGPU/fcopysign.f64.ll
+++ b/llvm/test/CodeGen/AMDGPU/fcopysign.f64.ll
@@ -279,28 +279,26 @@ define amdgpu_kernel void @s_test_copysign_f64_0_mag(ptr addrspace(1) %out, doub
 ; SI-LABEL: s_test_copysign_f64_0_mag:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; SI-NEXT:    s_waitcnt lgkmcnt(0)
-; SI-NEXT:    s_brev_b32 s2, -2
 ; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_mov_b32 s6, -1
-; SI-NEXT:    v_mov_b32_e32 v0, s3
+; SI-NEXT:    v_mov_b32_e32 v0, 0
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_mov_b32 s4, s0
+; SI-NEXT:    s_and_b32 s0, s3, 0x80000000
 ; SI-NEXT:    s_mov_b32 s5, s1
-; SI-NEXT:    v_bfi_b32 v1, s2, 0, v0
-; SI-NEXT:    v_mov_b32_e32 v0, 0
+; SI-NEXT:    v_mov_b32_e32 v1, s0
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: s_test_copysign_f64_0_mag:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; VI-NEXT:    v_mov_b32_e32 v2, 0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    s_brev_b32 s2, -2
-; VI-NEXT:    v_mov_b32_e32 v2, s3
 ; VI-NEXT:    v_mov_b32_e32 v0, s0
+; VI-NEXT:    s_and_b32 s0, s3, 0x80000000
 ; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_bfi_b32 v3, s2, 0, v2
-; VI-NEXT:    v_mov_b32_e32 v2, 0
+; VI-NEXT:    v_mov_b32_e32 v3, s0
 ; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
 ; VI-NEXT:    s_endpgm
   %result = call double @llvm.copysign.f64(double 0.0, double %sign)
@@ -312,30 +310,28 @@ define amdgpu_kernel void @s_test_copysign_f64_1_mag(ptr addrspace(1) %out, doub
 ; SI-LABEL: s_test_copysign_f64_1_mag:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; SI-NEXT:    s_waitcnt lgkmcnt(0)
-; SI-NEXT:    s_brev_b32 s2, -2
-; SI-NEXT:    v_mov_b32_e32 v0, 0x3ff00000
 ; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_mov_b32 s6, -1
-; SI-NEXT:    v_mov_b32_e32 v1, s3
+; SI-NEXT:    v_mov_b32_e32 v0, 0
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_mov_b32 s4, s0
+; SI-NEXT:    s_and_b32 s0, s3, 0x80000000
+; SI-NEXT:    s_or_b32 s0, s0, 0x3ff00000
 ; SI-NEXT:    s_mov_b32 s5, s1
-; SI-NEXT:    v_bfi_b32 v1, s2, v0, v1
-; SI-NEXT:    v_mov_b32_e32 v0, 0
+; SI-NEXT:    v_mov_b32_e32 v1, s0
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: s_test_copysign_f64_1_mag:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; VI-NEXT:    v_mov_b32_e32 v2, 0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    s_brev_b32 s2, -2
-; VI-NEXT:    v_mov_b32_e32 v2, 0x3ff00000
-; VI-NEXT:    v_mov_b32_e32 v3, s3
 ; VI-NEXT:    v_mov_b32_e32 v0, s0
+; VI-NEXT:    s_and_b32 s0, s3, 0x80000000
+; VI-NEXT:    s_or_b32 s0, s0, 0x3ff00000
 ; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_bfi_b32 v3, s2, v2, v3
-; VI-NEXT:    v_mov_b32_e32 v2, 0
+; VI-NEXT:    v_mov_b32_e32 v3, s0
 ; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
 ; VI-NEXT:    s_endpgm
   %result = call double @llvm.copysign.f64(double 1.0, double %sign)
@@ -347,30 +343,28 @@ define amdgpu_kernel void @s_test_copysign_f64_10_mag(ptr addrspace(1) %out, dou
 ; SI-LABEL: s_test_copysign_f64_10_mag:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; SI-NEXT:    s_waitcnt lgkmcnt(0)
-; SI-NEXT:    s_brev_b32 s2, -2
-; SI-NEXT:    v_mov_b32_e32 v0, 0x40240000
 ; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_mov_b32 s6, -1
-; SI-NEXT:    v_mov_b32_e32 v1, s3
+; SI-NEXT:    v_mov_b32_e32 v0, 0
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_mov_b32 s4, s0
+; SI-NEXT:    s_and_b32 s0, s3, 0x80000000
+; SI-NEXT:    s_or_b32 s0, s0, 0x40240000
 ; SI-NEXT:    s_mov_b32 s5, s1
-; SI-NEXT:    v_bfi_b32 v1, s2, v0, v1
-; SI-NEXT:    v_mov_b32_e32 v0, 0
+; SI-NEXT:    v_mov_b32_e32 v1, s0
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: s_test_copysign_f64_10_mag:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; VI-NEXT:    v_mov_b32_e32 v2, 0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    s_brev_b32 s2, -2
-; VI-NEXT:    v_mov_b32_e32 v2, 0x40240000
-; VI-NEXT:    v_mov_b32_e32 v3, s3
 ; VI-NEXT:    v_mov_b32_e32 v0, s0
+; VI-NEXT:    s_and_b32 s0, s3, 0x80000000
+; VI-NEXT:    s_or_b32 s0, s0, 0x40240000
 ; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_bfi_b32 v3, s2, v2, v3
-; VI-NEXT:    v_mov_b32_e32 v2, 0
+; VI-NEXT:    v_mov_b32_e32 v3, s0
 ; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
 ; VI-NEXT:    s_endpgm
   %result = call double @llvm.copysign.f64(double 10.0, double %sign)
@@ -382,30 +376,28 @@ define amdgpu_kernel void @s_test_copysign_f64_neg1_mag(ptr addrspace(1) %out, d
 ; SI-LABEL: s_test_copysign_f64_neg1_mag:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; SI-NEXT:    s_waitcnt lgkmcnt(0)
-; SI-NEXT:    s_brev_b32 s2, -2
-; SI-NEXT:    v_mov_b32_e32 v0, 0xbff00000
 ; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_mov_b32 s6, -1
-; SI-NEXT:    v_mov_b32_e32 v1, s3
+; SI-NEXT:    v_mov_b32_e32 v0, 0
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_mov_b32 s4, s0
+; SI-NEXT:    s_and_b32 s0, s3, 0x80000000
+; SI-NEXT:    s_or_b32 s0, s0, 0x3ff00000
 ; SI-NEXT:    s_mov_b32 s5, s1
-; SI-NEXT:    v_bfi_b32 v1, s2, v0, v1
-; SI-NEXT:    v_mov_b32_e32 v0, 0
+; SI-NEXT:    v_mov_b32_e32 v1, s0
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: s_test_copysign_f64_neg1_mag:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; VI-NEXT:    v_mov_b32_e32 v2, 0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    s_brev_b32 s2, -2
-; VI-NEXT:    v_mov_b32_e32 v2, 0xbff00000
-; VI-NEXT:    v_mov_b32_e32 v3, s3
 ; VI-NEXT:    v_mov_b32_e32 v0, s0
+; VI-NEXT:    s_and_b32 s0, s3, 0x80000000
+; VI-NEXT:    s_or_b32 s0, s0, 0x3ff00000
 ; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_bfi_b32 v3, s2, v2, v3
-; VI-NEXT:    v_mov_b32_e32 v2, 0
+; VI-NEXT:    v_mov_b32_e32 v3, s0
 ; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
 ; VI-NEXT:    s_endpgm
   %result = call double @llvm.copysign.f64(double -1.0, double %sign)
@@ -417,30 +409,28 @@ define amdgpu_kernel void @s_test_copysign_f64_neg10_mag(ptr addrspace(1) %out,
 ; SI-LABEL: s_test_copysign_f64_neg10_mag:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; SI-NEXT:    s_waitcnt lgkmcnt(0)
-; SI-NEXT:    s_brev_b32 s2, -2
-; SI-NEXT:    v_mov_b32_e32 v0, 0xc0240000
 ; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_mov_b32 s6, -1
-; SI-NEXT:    v_mov_b32_e32 v1, s3
+; SI-NEXT:    v_mov_b32_e32 v0, 0
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_mov_b32 s4, s0
+; SI-NEXT:    s_and_b32 s0, s3, 0x80000000
+; SI-NEXT:    s_or_b32 s0, s0, 0x40240000
 ; SI-NEXT:    s_mov_b32 s5, s1
-; SI-NEXT:    v_bfi_b32 v1, s2, v0, v1
-; SI-NEXT:    v_mov_b32_e32 v0, 0
+; SI-NEXT:    v_mov_b32_e32 v1, s0
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: s_test_copysign_f64_neg10_mag:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; VI-NEXT:    v_mov_b32_e32 v2, 0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    s_brev_b32 s2, -2
-; VI-NEXT:    v_mov_b32_e32 v2, 0xc0240000
-; VI-NEXT:    v_mov_b32_e32 v3, s3
 ; VI-NEXT:    v_mov_b32_e32 v0, s0
+; VI-NEXT:    s_and_b32 s0, s3, 0x80000000
+; VI-NEXT:    s_or_b32 s0, s0, 0x40240000
 ; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_bfi_b32 v3, s2, v2, v3
-; VI-NEXT:    v_mov_b32_e32 v2, 0
+; VI-NEXT:    v_mov_b32_e32 v3, s0
 ; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
 ; VI-NEXT:    s_endpgm
   %result = call double @llvm.copysign.f64(double -10.0, double %sign)
@@ -462,9 +452,9 @@ define amdgpu_kernel void @s_test_copysign_v2f64(ptr addrspace(1) %out, <2 x dou
 ; SI-NEXT:    v_bfi_b32 v3, s8, v0, v1
 ; SI-NEXT:    v_mov_b32_e32 v0, s5
 ; SI-NEXT:    v_mov_b32_e32 v1, s9
-; SI-NEXT:    v_mov_b32_e32 v2, s6
 ; SI-NEXT:    v_bfi_b32 v1, s8, v0, v1
 ; SI-NEXT:    v_mov_b32_e32 v0, s4
+; SI-NEXT:    v_mov_b32_e32 v2, s6
 ; SI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
@@ -476,13 +466,13 @@ define amdgpu_kernel void @s_test_copysign_v2f64(ptr addrspace(1) %out, <2 x dou
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    v_mov_b32_e32 v0, s7
 ; VI-NEXT:    v_mov_b32_e32 v1, s11
+; VI-NEXT:    v_mov_b32_e32 v2, s5
 ; VI-NEXT:    v_bfi_b32 v3, s2, v0, v1
-; VI-NEXT:    v_mov_b32_e32 v0, s5
-; VI-NEXT:    v_mov_b32_e32 v1, s9
+; VI-NEXT:    v_mov_b32_e32 v0, s9
 ; VI-NEXT:    v_mov_b32_e32 v5, s1
-; VI-NEXT:    v_mov_b32_e32 v2, s6
-; VI-NEXT:    v_bfi_b32 v1, s2, v0, v1
+; VI-NEXT:    v_bfi_b32 v1, s2, v2, v0
 ; VI-NEXT:    v_mov_b32_e32 v0, s4
+; VI-NEXT:    v_mov_b32_e32 v2, s6
 ; VI-NEXT:    v_mov_b32_e32 v4, s0
 ; VI-NEXT:    flat_store_dwordx4 v[4:5], v[0:3]
 ; VI-NEXT:    s_endpgm
@@ -502,16 +492,16 @@ define amdgpu_kernel void @s_test_copysign_v3f64(ptr addrspace(1) %out, <3 x dou
 ; SI-NEXT:    s_mov_b32 s2, -1
 ; SI-NEXT:    v_mov_b32_e32 v0, s7
 ; SI-NEXT:    v_mov_b32_e32 v1, s15
-; SI-NEXT:    v_mov_b32_e32 v4, s9
-; SI-NEXT:    v_mov_b32_e32 v5, s17
 ; SI-NEXT:    v_bfi_b32 v3, s10, v0, v1
 ; SI-NEXT:    v_mov_b32_e32 v0, s5
 ; SI-NEXT:    v_mov_b32_e32 v1, s13
-; SI-NEXT:    v_bfi_b32 v5, s10, v4, v5
-; SI-NEXT:    v_mov_b32_e32 v4, s8
-; SI-NEXT:    v_mov_b32_e32 v2, s6
 ; SI-NEXT:    v_bfi_b32 v1, s10, v0, v1
+; SI-NEXT:    v_mov_b32_e32 v0, s9
+; SI-NEXT:    v_mov_b32_e32 v2, s17
+; SI-NEXT:    v_bfi_b32 v5, s10, v0, v2
+; SI-NEXT:    v_mov_b32_e32 v4, s8
 ; SI-NEXT:    v_mov_b32_e32 v0, s4
+; SI-NEXT:    v_mov_b32_e32 v2, s6
 ; SI-NEXT:    buffer_store_dwordx2 v[4:5], off, s[0:3], 0 offset:16
 ; SI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
@@ -524,13 +514,13 @@ define amdgpu_kernel void @s_test_copysign_v3f64(ptr addrspace(1) %out, <3 x dou
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    v_mov_b32_e32 v0, s7
 ; VI-NEXT:    v_mov_b32_e32 v1, s15
+; VI-NEXT:    v_mov_b32_e32 v2, s5
 ; VI-NEXT:    v_bfi_b32 v3, s2, v0, v1
-; VI-NEXT:    v_mov_b32_e32 v0, s5
-; VI-NEXT:    v_mov_b32_e32 v1, s13
-; VI-NEXT:    v_mov_b32_e32 v4, s9
-; VI-NEXT:    v_mov_b32_e32 v5, s17
-; VI-NEXT:    v_bfi_b32 v1, s2, v0, v1
-; VI-NEXT:    v_bfi_b32 v5, s2, v4, v5
+; VI-NEXT:    v_mov_b32_e32 v0, s13
+; VI-NEXT:    v_bfi_b32 v1, s2, v2, v0
+; VI-NEXT:    v_mov_b32_e32 v0, s9
+; VI-NEXT:    v_mov_b32_e32 v2, s17
+; VI-NEXT:    v_bfi_b32 v5, s2, v0, v2
 ; VI-NEXT:    s_add_u32 s2, s0, 16
 ; VI-NEXT:    s_addc_u32 s3, s1, 0
 ; VI-NEXT:    v_mov_b32_e32 v7, s3
@@ -538,8 +528,8 @@ define amdgpu_kernel void @s_test_copysign_v3f64(ptr addrspace(1) %out, <3 x dou
 ; VI-NEXT:    v_mov_b32_e32 v6, s2
 ; VI-NEXT:    flat_store_dwordx2 v[6:7], v[4:5]
 ; VI-NEXT:    v_mov_b32_e32 v5, s1
-; VI-NEXT:    v_mov_b32_e32 v2, s6
 ; VI-NEXT:    v_mov_b32_e32 v0, s4
+; VI-NEXT:    v_mov_b32_e32 v2, s6
 ; VI-NEXT:    v_mov_b32_e32 v4, s0
 ; VI-NEXT:    flat_store_dwordx4 v[4:5], v[0:3]
 ; VI-NEXT:    s_endpgm
@@ -552,27 +542,28 @@ define amdgpu_kernel void @s_test_copysign_v4f64(ptr addrspace(1) %out, <4 x dou
 ; SI-LABEL: s_test_copysign_v4f64:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_load_dwordx16 s[4:19], s[0:1], 0x11
-; SI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_brev_b32 s12, -2
+; SI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
 ; SI-NEXT:    s_mov_b32 s3, 0xf000
 ; SI-NEXT:    s_mov_b32 s2, -1
-; SI-NEXT:    v_mov_b32_e32 v4, s11
-; SI-NEXT:    v_mov_b32_e32 v5, s19
 ; SI-NEXT:    v_mov_b32_e32 v0, s7
 ; SI-NEXT:    v_mov_b32_e32 v1, s15
-; SI-NEXT:    v_bfi_b32 v7, s12, v4, v5
-; SI-NEXT:    v_mov_b32_e32 v4, s9
-; SI-NEXT:    v_mov_b32_e32 v5, s17
 ; SI-NEXT:    v_bfi_b32 v3, s12, v0, v1
 ; SI-NEXT:    v_mov_b32_e32 v0, s5
 ; SI-NEXT:    v_mov_b32_e32 v1, s13
-; SI-NEXT:    v_mov_b32_e32 v6, s10
-; SI-NEXT:    v_bfi_b32 v5, s12, v4, v5
-; SI-NEXT:    v_mov_b32_e32 v4, s8
-; SI-NEXT:    v_mov_b32_e32 v2, s6
 ; SI-NEXT:    v_bfi_b32 v1, s12, v0, v1
+; SI-NEXT:    v_mov_b32_e32 v0, s11
+; SI-NEXT:    v_mov_b32_e32 v2, s19
+; SI-NEXT:    v_bfi_b32 v7, s12, v0, v2
+; SI-NEXT:    v_mov_b32_e32 v0, s9
+; SI-NEXT:    v_mov_b32_e32 v2, s17
+; SI-NEXT:    v_bfi_b32 v5, s12, v0, v2
+; SI-NEXT:    v_mov_b32_e32 v4, s8
+; SI-NEXT:    v_mov_b32_e32 v6, s10
 ; SI-NEXT:    v_mov_b32_e32 v0, s4
+; SI-NEXT:    v_mov_b32_e32 v2, s6
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    buffer_store_dwordx4 v[4:7], off, s[0:3], 0 offset:16
 ; SI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
@@ -585,26 +576,26 @@ define amdgpu_kernel void @s_test_copysign_v4f64(ptr addrspace(1) %out, <4 x dou
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    v_mov_b32_e32 v0, s7
 ; VI-NEXT:    v_mov_b32_e32 v1, s15
-; VI-NEXT:    v_mov_b32_e32 v4, s11
-; VI-NEXT:    v_mov_b32_e32 v5, s19
+; VI-NEXT:    v_mov_b32_e32 v2, s5
 ; VI-NEXT:    v_bfi_b32 v3, s2, v0, v1
-; VI-NEXT:    v_mov_b32_e32 v0, s5
-; VI-NEXT:    v_mov_b32_e32 v1, s13
-; VI-NEXT:    v_bfi_b32 v7, s2, v4, v5
-; VI-NEXT:    v_mov_b32_e32 v4, s9
-; VI-NEXT:    v_mov_b32_e32 v5, s17
-; VI-NEXT:    v_bfi_b32 v1, s2, v0, v1
-; VI-NEXT:    v_bfi_b32 v5, s2, v4, v5
+; VI-NEXT:    v_mov_b32_e32 v0, s13
+; VI-NEXT:    v_bfi_b32 v1, s2, v2, v0
+; VI-NEXT:    v_mov_b32_e32 v0, s11
+; VI-NEXT:    v_mov_b32_e32 v2, s19
+; VI-NEXT:    v_bfi_b32 v7, s2, v0, v2
+; VI-NEXT:    v_mov_b32_e32 v0, s9
+; VI-NEXT:    v_mov_b32_e32 v2, s17
+; VI-NEXT:    v_bfi_b32 v5, s2, v0, v2
 ; VI-NEXT:    s_add_u32 s2, s0, 16
 ; VI-NEXT:    s_addc_u32 s3, s1, 0
 ; VI-NEXT:    v_mov_b32_e32 v9, s3
-; VI-NEXT:    v_mov_b32_e32 v6, s10
 ; VI-NEXT:    v_mov_b32_e32 v4, s8
+; VI-NEXT:    v_mov_b32_e32 v6, s10
 ; VI-NEXT:    v_mov_b32_e32 v8, s2
 ; VI-NEXT:    flat_store_dwordx4 v[8:9], v[4:7]
-; VI-NEXT:    v_mov_b32_e32 v2, s6
-; VI-NEXT:    v_mov_b32_e32 v5, s1
 ; VI-NEXT:    v_mov_b32_e32 v0, s4
+; VI-NEXT:    v_mov_b32_e32 v5, s1
+; VI-NEXT:    v_mov_b32_e32 v2, s6
 ; VI-NEXT:    v_mov_b32_e32 v4, s0
 ; VI-NEXT:    flat_store_dwordx4 v[4:5], v[0:3]
 ; VI-NEXT:    s_endpgm
diff --git a/llvm/test/CodeGen/AMDGPU/flat-scratch-i8-i16.ll b/llvm/test/CodeGen/AMDGPU/flat-scratch-i8-i16.ll
index 499ee9acbbf34..30b310a501f77 100644
--- a/llvm/test/CodeGen/AMDGPU/flat-scratch-i8-i16.ll
+++ b/llvm/test/CodeGen/AMDGPU/flat-scratch-i8-i16.ll
@@ -11,14 +11,16 @@ define amdgpu_ps void @test_scratch_load_i8_zext_v(ptr addrspace(5) %in, ptr %ou
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
-; GFX10-NEXT:    scratch_load_ubyte v0, v0, off offset:1
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, 1, v0
+; GFX10-NEXT:    scratch_load_ubyte v0, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v0
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_load_i8_zext_v:
 ; GFX11:       ; %bb.0:
-; GFX11-NEXT:    scratch_load_u8 v0, v0, off offset:1
+; GFX11-NEXT:    v_add_nc_u32_e32 v0, 1, v0
+; GFX11-NEXT:    scratch_load_u8 v0, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v0
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -37,14 +39,16 @@ define amdgpu_ps void @test_scratch_load_i8_sext_v(ptr addrspace(5) %in, ptr %ou
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
-; GFX10-NEXT:    scratch_load_sbyte v0, v0, off offset:1
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, 1, v0
+; GFX10-NEXT:    scratch_load_sbyte v0, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v0
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_load_i8_sext_v:
 ; GFX11:       ; %bb.0:
-; GFX11-NEXT:    scratch_load_i8 v0, v0, off offset:1
+; GFX11-NEXT:    v_add_nc_u32_e32 v0, 1, v0
+; GFX11-NEXT:    scratch_load_i8 v0, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v0
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -63,14 +67,16 @@ define amdgpu_ps void @test_scratch_load_i16_zext_v(ptr addrspace(5) %in, ptr %o
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
-; GFX10-NEXT:    scratch_load_ushort v0, v0, off offset:2
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, 2, v0
+; GFX10-NEXT:    scratch_load_ushort v0, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v0
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_load_i16_zext_v:
 ; GFX11:       ; %bb.0:
-; GFX11-NEXT:    scratch_load_u16 v0, v0, off offset:2
+; GFX11-NEXT:    v_add_nc_u32_e32 v0, 2, v0
+; GFX11-NEXT:    scratch_load_u16 v0, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v0
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -89,14 +95,16 @@ define amdgpu_ps void @test_scratch_load_i16_sext_v(ptr addrspace(5) %in, ptr %o
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
-; GFX10-NEXT:    scratch_load_sshort v0, v0, off offset:2
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, 2, v0
+; GFX10-NEXT:    scratch_load_sshort v0, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v0
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_load_i16_sext_v:
 ; GFX11:       ; %bb.0:
-; GFX11-NEXT:    scratch_load_i16 v0, v0, off offset:2
+; GFX11-NEXT:    v_add_nc_u32_e32 v0, 2, v0
+; GFX11-NEXT:    scratch_load_i16 v0, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v0
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -115,16 +123,17 @@ define amdgpu_ps void @test_scratch_load_i8_zext_to_d16_lo_v(ptr addrspace(5) %i
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, 1, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v3, 0xffff0000
-; GFX10-NEXT:    scratch_load_ubyte_d16 v3, v0, off offset:1
+; GFX10-NEXT:    scratch_load_ubyte_d16 v3, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v3
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_load_i8_zext_to_d16_lo_v:
 ; GFX11:       ; %bb.0: ; %bb
-; GFX11-NEXT:    v_mov_b32_e32 v3, 0xffff0000
-; GFX11-NEXT:    scratch_load_d16_u8 v3, v0, off offset:1
+; GFX11-NEXT:    v_dual_mov_b32 v3, 0xffff0000 :: v_dual_add_nc_u32 v0, 1, v0
+; GFX11-NEXT:    scratch_load_d16_u8 v3, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v3
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -145,16 +154,17 @@ define amdgpu_ps void @test_scratch_load_i8_sext_to_d16_lo_v(ptr addrspace(5) %i
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, 1, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v3, 0xffff0000
-; GFX10-NEXT:    scratch_load_sbyte_d16 v3, v0, off offset:1
+; GFX10-NEXT:    scratch_load_sbyte_d16 v3, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v3
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_load_i8_sext_to_d16_lo_v:
 ; GFX11:       ; %bb.0: ; %bb
-; GFX11-NEXT:    v_mov_b32_e32 v3, 0xffff0000
-; GFX11-NEXT:    scratch_load_d16_i8 v3, v0, off offset:1
+; GFX11-NEXT:    v_dual_mov_b32 v3, 0xffff0000 :: v_dual_add_nc_u32 v0, 1, v0
+; GFX11-NEXT:    scratch_load_d16_i8 v3, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v3
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -175,16 +185,17 @@ define amdgpu_ps void @test_scratch_load_i16_to_d16_lo_v(ptr addrspace(5) %in, p
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, 2, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v3, 0xffff0000
-; GFX10-NEXT:    scratch_load_short_d16 v3, v0, off offset:2
+; GFX10-NEXT:    scratch_load_short_d16 v3, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v3
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_load_i16_to_d16_lo_v:
 ; GFX11:       ; %bb.0: ; %bb
-; GFX11-NEXT:    v_mov_b32_e32 v3, 0xffff0000
-; GFX11-NEXT:    scratch_load_d16_b16 v3, v0, off offset:2
+; GFX11-NEXT:    v_dual_mov_b32 v3, 0xffff0000 :: v_dual_add_nc_u32 v0, 2, v0
+; GFX11-NEXT:    scratch_load_d16_b16 v3, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v3
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -205,16 +216,17 @@ define amdgpu_ps void @test_scratch_load_i8_zext_to_d16_hi_v(ptr addrspace(5) %i
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, 1, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v3, -1
-; GFX10-NEXT:    scratch_load_ubyte_d16_hi v3, v0, off offset:1
+; GFX10-NEXT:    scratch_load_ubyte_d16_hi v3, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v3
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_load_i8_zext_to_d16_hi_v:
 ; GFX11:       ; %bb.0: ; %bb
-; GFX11-NEXT:    v_mov_b32_e32 v3, -1
-; GFX11-NEXT:    scratch_load_d16_hi_u8 v3, v0, off offset:1
+; GFX11-NEXT:    v_dual_mov_b32 v3, -1 :: v_dual_add_nc_u32 v0, 1, v0
+; GFX11-NEXT:    scratch_load_d16_hi_u8 v3, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v3
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -235,16 +247,17 @@ define amdgpu_ps void @test_scratch_load_i8_sext_to_d16_hi_v(ptr addrspace(5) %i
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, 1, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v3, -1
-; GFX10-NEXT:    scratch_load_sbyte_d16_hi v3, v0, off offset:1
+; GFX10-NEXT:    scratch_load_sbyte_d16_hi v3, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v3
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_load_i8_sext_to_d16_hi_v:
 ; GFX11:       ; %bb.0: ; %bb
-; GFX11-NEXT:    v_mov_b32_e32 v3, -1
-; GFX11-NEXT:    scratch_load_d16_hi_i8 v3, v0, off offset:1
+; GFX11-NEXT:    v_dual_mov_b32 v3, -1 :: v_dual_add_nc_u32 v0, 1, v0
+; GFX11-NEXT:    scratch_load_d16_hi_i8 v3, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v3
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -265,16 +278,17 @@ define amdgpu_ps void @test_scratch_load_i16_to_d16_hi_v(ptr addrspace(5) %in, p
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, 2, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v3, -1
-; GFX10-NEXT:    scratch_load_short_d16_hi v3, v0, off offset:2
+; GFX10-NEXT:    scratch_load_short_d16_hi v3, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v3
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_load_i16_to_d16_hi_v:
 ; GFX11:       ; %bb.0: ; %bb
-; GFX11-NEXT:    v_mov_b32_e32 v3, -1
-; GFX11-NEXT:    scratch_load_d16_hi_b16 v3, v0, off offset:2
+; GFX11-NEXT:    v_dual_mov_b32 v3, -1 :: v_dual_add_nc_u32 v0, 2, v0
+; GFX11-NEXT:    scratch_load_d16_hi_b16 v3, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v3
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -295,15 +309,17 @@ define amdgpu_ps void @test_scratch_store_b8_from_d16_hi_v(ptr %in, ptr addrspac
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
 ; GFX10-NEXT:    flat_load_dword v0, v[0:1]
+; GFX10-NEXT:    v_add_nc_u32_e32 v1, 4, v2
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
-; GFX10-NEXT:    scratch_store_byte_d16_hi v2, v0, off offset:4
+; GFX10-NEXT:    scratch_store_byte_d16_hi v1, v0, off
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_store_b8_from_d16_hi_v:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    flat_load_b32 v0, v[0:1]
+; GFX11-NEXT:    v_add_nc_u32_e32 v1, 4, v2
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
-; GFX11-NEXT:    scratch_store_d16_hi_b8 v2, v0, off offset:4
+; GFX11-NEXT:    scratch_store_d16_hi_b8 v1, v0, off
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
 bb:
@@ -322,15 +338,17 @@ define amdgpu_ps void @test_scratch_store_b16_from_d16_hi_v(ptr %in, ptr addrspa
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
 ; GFX10-NEXT:    flat_load_dword v0, v[0:1]
+; GFX10-NEXT:    v_add_nc_u32_e32 v1, 2, v2
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
-; GFX10-NEXT:    scratch_store_short_d16_hi v2, v0, off offset:2
+; GFX10-NEXT:    scratch_store_short_d16_hi v1, v0, off
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_store_b16_from_d16_hi_v:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    flat_load_b32 v0, v[0:1]
+; GFX11-NEXT:    v_add_nc_u32_e32 v1, 2, v2
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
-; GFX11-NEXT:    scratch_store_d16_hi_b16 v2, v0, off offset:2
+; GFX11-NEXT:    scratch_store_d16_hi_b16 v1, v0, off
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
 bb:
@@ -353,14 +371,16 @@ define amdgpu_ps void @test_scratch_load_i8_zext_s(ptr addrspace(5) inreg %in, p
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
-; GFX10-NEXT:    scratch_load_ubyte v2, off, s2 offset:1
+; GFX10-NEXT:    s_add_i32 s2, s2, 1
+; GFX10-NEXT:    scratch_load_ubyte v2, off, s2
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[0:1], v2
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_load_i8_zext_s:
 ; GFX11:       ; %bb.0:
-; GFX11-NEXT:    scratch_load_u8 v2, off, s0 offset:1
+; GFX11-NEXT:    s_add_i32 s0, s0, 1
+; GFX11-NEXT:    scratch_load_u8 v2, off, s0
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[0:1], v2
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -379,14 +399,16 @@ define amdgpu_ps void @test_scratch_load_i8_sext_s(ptr addrspace(5) inreg %in, p
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
-; GFX10-NEXT:    scratch_load_sbyte v2, off, s2 offset:1
+; GFX10-NEXT:    s_add_i32 s2, s2, 1
+; GFX10-NEXT:    scratch_load_sbyte v2, off, s2
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[0:1], v2
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_load_i8_sext_s:
 ; GFX11:       ; %bb.0:
-; GFX11-NEXT:    scratch_load_i8 v2, off, s0 offset:1
+; GFX11-NEXT:    s_add_i32 s0, s0, 1
+; GFX11-NEXT:    scratch_load_i8 v2, off, s0
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[0:1], v2
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -405,14 +427,16 @@ define amdgpu_ps void @test_scratch_load_i16_zext_s(ptr addrspace(5) inreg %in,
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
-; GFX10-NEXT:    scratch_load_ushort v2, off, s2 offset:2
+; GFX10-NEXT:    s_add_i32 s2, s2, 2
+; GFX10-NEXT:    scratch_load_ushort v2, off, s2
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[0:1], v2
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_load_i16_zext_s:
 ; GFX11:       ; %bb.0:
-; GFX11-NEXT:    scratch_load_u16 v2, off, s0 offset:2
+; GFX11-NEXT:    s_add_i32 s0, s0, 2
+; GFX11-NEXT:    scratch_load_u16 v2, off, s0
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[0:1], v2
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -431,14 +455,16 @@ define amdgpu_ps void @test_scratch_load_i16_sext_s(ptr addrspace(5) inreg %in,
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
-; GFX10-NEXT:    scratch_load_sshort v2, off, s2 offset:2
+; GFX10-NEXT:    s_add_i32 s2, s2, 2
+; GFX10-NEXT:    scratch_load_sshort v2, off, s2
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[0:1], v2
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_load_i16_sext_s:
 ; GFX11:       ; %bb.0:
-; GFX11-NEXT:    scratch_load_i16 v2, off, s0 offset:2
+; GFX11-NEXT:    s_add_i32 s0, s0, 2
+; GFX11-NEXT:    scratch_load_i16 v2, off, s0
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[0:1], v2
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -458,7 +484,8 @@ define amdgpu_ps void @test_scratch_load_i8_zext_to_d16_lo_s(ptr addrspace(5) in
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
 ; GFX10-NEXT:    v_mov_b32_e32 v2, 0xffff0000
-; GFX10-NEXT:    scratch_load_ubyte_d16 v2, off, s2 offset:1
+; GFX10-NEXT:    s_add_i32 s2, s2, 1
+; GFX10-NEXT:    scratch_load_ubyte_d16 v2, off, s2
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[0:1], v2
 ; GFX10-NEXT:    s_endpgm
@@ -466,7 +493,8 @@ define amdgpu_ps void @test_scratch_load_i8_zext_to_d16_lo_s(ptr addrspace(5) in
 ; GFX11-LABEL: test_scratch_load_i8_zext_to_d16_lo_s:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    v_mov_b32_e32 v2, 0xffff0000
-; GFX11-NEXT:    scratch_load_d16_u8 v2, off, s0 offset:1
+; GFX11-NEXT:    s_add_i32 s0, s0, 1
+; GFX11-NEXT:    scratch_load_d16_u8 v2, off, s0
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[0:1], v2
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -488,7 +516,8 @@ define amdgpu_ps void @test_scratch_load_i8_sext_to_d16_lo_s(ptr addrspace(5) in
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
 ; GFX10-NEXT:    v_mov_b32_e32 v2, 0xffff0000
-; GFX10-NEXT:    scratch_load_sbyte_d16 v2, off, s2 offset:1
+; GFX10-NEXT:    s_add_i32 s2, s2, 1
+; GFX10-NEXT:    scratch_load_sbyte_d16 v2, off, s2
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[0:1], v2
 ; GFX10-NEXT:    s_endpgm
@@ -496,7 +525,8 @@ define amdgpu_ps void @test_scratch_load_i8_sext_to_d16_lo_s(ptr addrspace(5) in
 ; GFX11-LABEL: test_scratch_load_i8_sext_to_d16_lo_s:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    v_mov_b32_e32 v2, 0xffff0000
-; GFX11-NEXT:    scratch_load_d16_i8 v2, off, s0 offset:1
+; GFX11-NEXT:    s_add_i32 s0, s0, 1
+; GFX11-NEXT:    scratch_load_d16_i8 v2, off, s0
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[0:1], v2
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -518,7 +548,8 @@ define amdgpu_ps void @test_scratch_load_i16_to_d16_lo_s(ptr addrspace(5) inreg
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
 ; GFX10-NEXT:    v_mov_b32_e32 v2, 0xffff0000
-; GFX10-NEXT:    scratch_load_short_d16 v2, off, s2 offset:2
+; GFX10-NEXT:    s_add_i32 s2, s2, 2
+; GFX10-NEXT:    scratch_load_short_d16 v2, off, s2
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[0:1], v2
 ; GFX10-NEXT:    s_endpgm
@@ -526,7 +557,8 @@ define amdgpu_ps void @test_scratch_load_i16_to_d16_lo_s(ptr addrspace(5) inreg
 ; GFX11-LABEL: test_scratch_load_i16_to_d16_lo_s:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    v_mov_b32_e32 v2, 0xffff0000
-; GFX11-NEXT:    scratch_load_d16_b16 v2, off, s0 offset:2
+; GFX11-NEXT:    s_add_i32 s0, s0, 2
+; GFX11-NEXT:    scratch_load_d16_b16 v2, off, s0
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[0:1], v2
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -548,7 +580,8 @@ define amdgpu_ps void @test_scratch_load_i8_zext_to_d16_hi_s(ptr addrspace(5) in
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
 ; GFX10-NEXT:    v_mov_b32_e32 v2, -1
-; GFX10-NEXT:    scratch_load_ubyte_d16_hi v2, off, s2 offset:1
+; GFX10-NEXT:    s_add_i32 s2, s2, 1
+; GFX10-NEXT:    scratch_load_ubyte_d16_hi v2, off, s2
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[0:1], v2
 ; GFX10-NEXT:    s_endpgm
@@ -556,7 +589,8 @@ define amdgpu_ps void @test_scratch_load_i8_zext_to_d16_hi_s(ptr addrspace(5) in
 ; GFX11-LABEL: test_scratch_load_i8_zext_to_d16_hi_s:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    v_mov_b32_e32 v2, -1
-; GFX11-NEXT:    scratch_load_d16_hi_u8 v2, off, s0 offset:1
+; GFX11-NEXT:    s_add_i32 s0, s0, 1
+; GFX11-NEXT:    scratch_load_d16_hi_u8 v2, off, s0
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[0:1], v2
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -578,7 +612,8 @@ define amdgpu_ps void @test_scratch_load_i8_sext_to_d16_hi_s(ptr addrspace(5) in
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
 ; GFX10-NEXT:    v_mov_b32_e32 v2, -1
-; GFX10-NEXT:    scratch_load_sbyte_d16_hi v2, off, s2 offset:1
+; GFX10-NEXT:    s_add_i32 s2, s2, 1
+; GFX10-NEXT:    scratch_load_sbyte_d16_hi v2, off, s2
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[0:1], v2
 ; GFX10-NEXT:    s_endpgm
@@ -586,7 +621,8 @@ define amdgpu_ps void @test_scratch_load_i8_sext_to_d16_hi_s(ptr addrspace(5) in
 ; GFX11-LABEL: test_scratch_load_i8_sext_to_d16_hi_s:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    v_mov_b32_e32 v2, -1
-; GFX11-NEXT:    scratch_load_d16_hi_i8 v2, off, s0 offset:1
+; GFX11-NEXT:    s_add_i32 s0, s0, 1
+; GFX11-NEXT:    scratch_load_d16_hi_i8 v2, off, s0
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[0:1], v2
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -608,7 +644,8 @@ define amdgpu_ps void @test_scratch_load_i16_to_d16_hi_s(ptr addrspace(5) inreg
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
 ; GFX10-NEXT:    v_mov_b32_e32 v2, -1
-; GFX10-NEXT:    scratch_load_short_d16_hi v2, off, s2 offset:2
+; GFX10-NEXT:    s_add_i32 s2, s2, 2
+; GFX10-NEXT:    scratch_load_short_d16_hi v2, off, s2
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[0:1], v2
 ; GFX10-NEXT:    s_endpgm
@@ -616,7 +653,8 @@ define amdgpu_ps void @test_scratch_load_i16_to_d16_hi_s(ptr addrspace(5) inreg
 ; GFX11-LABEL: test_scratch_load_i16_to_d16_hi_s:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    v_mov_b32_e32 v2, -1
-; GFX11-NEXT:    scratch_load_d16_hi_b16 v2, off, s0 offset:2
+; GFX11-NEXT:    s_add_i32 s0, s0, 2
+; GFX11-NEXT:    scratch_load_d16_hi_b16 v2, off, s0
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[0:1], v2
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -637,15 +675,17 @@ define amdgpu_ps void @test_scratch_store_b8_from_d16_hi_s(ptr %in, ptr addrspac
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
 ; GFX10-NEXT:    flat_load_dword v0, v[0:1]
+; GFX10-NEXT:    s_add_i32 s2, s2, 4
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
-; GFX10-NEXT:    scratch_store_byte_d16_hi off, v0, s2 offset:4
+; GFX10-NEXT:    scratch_store_byte_d16_hi off, v0, s2
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_store_b8_from_d16_hi_s:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    flat_load_b32 v0, v[0:1]
+; GFX11-NEXT:    s_add_i32 s0, s0, 4
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
-; GFX11-NEXT:    scratch_store_d16_hi_b8 off, v0, s0 offset:4
+; GFX11-NEXT:    scratch_store_d16_hi_b8 off, v0, s0
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
 bb:
@@ -664,15 +704,17 @@ define amdgpu_ps void @test_scratch_store_b16_from_d16_hi_s(ptr %in, ptr addrspa
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
 ; GFX10-NEXT:    flat_load_dword v0, v[0:1]
+; GFX10-NEXT:    s_add_i32 s2, s2, 2
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
-; GFX10-NEXT:    scratch_store_short_d16_hi off, v0, s2 offset:2
+; GFX10-NEXT:    scratch_store_short_d16_hi off, v0, s2
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_store_b16_from_d16_hi_s:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    flat_load_b32 v0, v[0:1]
+; GFX11-NEXT:    s_add_i32 s0, s0, 2
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
-; GFX11-NEXT:    scratch_store_d16_hi_b16 off, v0, s0 offset:2
+; GFX11-NEXT:    scratch_store_d16_hi_b16 off, v0, s0
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
 bb:
@@ -695,8 +737,9 @@ define amdgpu_ps void @test_scratch_load_i8_zext_svs(ptr addrspace(5) inreg %in,
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
-; GFX10-NEXT:    v_lshl_add_u32 v0, v0, 2, s2
-; GFX10-NEXT:    scratch_load_ubyte v0, v0, off offset:1
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX10-NEXT:    v_add3_u32 v0, s2, v0, 1
+; GFX10-NEXT:    scratch_load_ubyte v0, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v0
 ; GFX10-NEXT:    s_endpgm
@@ -704,7 +747,9 @@ define amdgpu_ps void @test_scratch_load_i8_zext_svs(ptr addrspace(5) inreg %in,
 ; GFX11-LABEL: test_scratch_load_i8_zext_svs:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX11-NEXT:    scratch_load_u8 v0, v0, s0 offset:1
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_add3_u32 v0, s0, v0, 1
+; GFX11-NEXT:    scratch_load_u8 v0, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v0
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -725,8 +770,9 @@ define amdgpu_ps void @test_scratch_load_i8_sext_svs(ptr addrspace(5) inreg %in,
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
-; GFX10-NEXT:    v_lshl_add_u32 v0, v0, 2, s2
-; GFX10-NEXT:    scratch_load_sbyte v0, v0, off offset:1
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX10-NEXT:    v_add3_u32 v0, s2, v0, 1
+; GFX10-NEXT:    scratch_load_sbyte v0, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v0
 ; GFX10-NEXT:    s_endpgm
@@ -734,7 +780,9 @@ define amdgpu_ps void @test_scratch_load_i8_sext_svs(ptr addrspace(5) inreg %in,
 ; GFX11-LABEL: test_scratch_load_i8_sext_svs:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX11-NEXT:    scratch_load_i8 v0, v0, s0 offset:1
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_add3_u32 v0, s0, v0, 1
+; GFX11-NEXT:    scratch_load_i8 v0, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v0
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -755,8 +803,9 @@ define amdgpu_ps void @test_scratch_load_i16_zext_svs(ptr addrspace(5) inreg %in
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
-; GFX10-NEXT:    v_lshl_add_u32 v0, v0, 2, s2
-; GFX10-NEXT:    scratch_load_ushort v0, v0, off offset:2
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX10-NEXT:    v_add3_u32 v0, s2, v0, 2
+; GFX10-NEXT:    scratch_load_ushort v0, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v0
 ; GFX10-NEXT:    s_endpgm
@@ -764,7 +813,9 @@ define amdgpu_ps void @test_scratch_load_i16_zext_svs(ptr addrspace(5) inreg %in
 ; GFX11-LABEL: test_scratch_load_i16_zext_svs:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX11-NEXT:    scratch_load_u16 v0, v0, s0 offset:2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_add3_u32 v0, s0, v0, 2
+; GFX11-NEXT:    scratch_load_u16 v0, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v0
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -785,8 +836,9 @@ define amdgpu_ps void @test_scratch_load_i16_sext_svs(ptr addrspace(5) inreg %in
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
-; GFX10-NEXT:    v_lshl_add_u32 v0, v0, 2, s2
-; GFX10-NEXT:    scratch_load_sshort v0, v0, off offset:2
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX10-NEXT:    v_add3_u32 v0, s2, v0, 2
+; GFX10-NEXT:    scratch_load_sshort v0, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v0
 ; GFX10-NEXT:    s_endpgm
@@ -794,7 +846,9 @@ define amdgpu_ps void @test_scratch_load_i16_sext_svs(ptr addrspace(5) inreg %in
 ; GFX11-LABEL: test_scratch_load_i16_sext_svs:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX11-NEXT:    scratch_load_i16 v0, v0, s0 offset:2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_add3_u32 v0, s0, v0, 2
+; GFX11-NEXT:    scratch_load_i16 v0, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v0
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -815,9 +869,10 @@ define amdgpu_ps void @test_scratch_load_i8_zext_to_d16_lo_svs(ptr addrspace(5)
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
-; GFX10-NEXT:    v_lshl_add_u32 v0, v0, 2, s2
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v3, 0xffff0000
-; GFX10-NEXT:    scratch_load_ubyte_d16 v3, v0, off offset:1
+; GFX10-NEXT:    v_add3_u32 v0, s2, v0, 1
+; GFX10-NEXT:    scratch_load_ubyte_d16 v3, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v3
 ; GFX10-NEXT:    s_endpgm
@@ -825,7 +880,9 @@ define amdgpu_ps void @test_scratch_load_i8_zext_to_d16_lo_svs(ptr addrspace(5)
 ; GFX11-LABEL: test_scratch_load_i8_zext_to_d16_lo_svs:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    v_dual_mov_b32 v3, 0xffff0000 :: v_dual_lshlrev_b32 v0, 2, v0
-; GFX11-NEXT:    scratch_load_d16_u8 v3, v0, s0 offset:1
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_add3_u32 v0, s0, v0, 1
+; GFX11-NEXT:    scratch_load_d16_u8 v3, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v3
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -848,9 +905,10 @@ define amdgpu_ps void @test_scratch_load_i8_sext_to_d16_lo_svs(ptr addrspace(5)
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
-; GFX10-NEXT:    v_lshl_add_u32 v0, v0, 2, s2
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v3, 0xffff0000
-; GFX10-NEXT:    scratch_load_sbyte_d16 v3, v0, off offset:1
+; GFX10-NEXT:    v_add3_u32 v0, s2, v0, 1
+; GFX10-NEXT:    scratch_load_sbyte_d16 v3, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v3
 ; GFX10-NEXT:    s_endpgm
@@ -858,7 +916,9 @@ define amdgpu_ps void @test_scratch_load_i8_sext_to_d16_lo_svs(ptr addrspace(5)
 ; GFX11-LABEL: test_scratch_load_i8_sext_to_d16_lo_svs:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    v_dual_mov_b32 v3, 0xffff0000 :: v_dual_lshlrev_b32 v0, 2, v0
-; GFX11-NEXT:    scratch_load_d16_i8 v3, v0, s0 offset:1
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_add3_u32 v0, s0, v0, 1
+; GFX11-NEXT:    scratch_load_d16_i8 v3, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v3
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -881,9 +941,10 @@ define amdgpu_ps void @test_scratch_load_i16_to_d16_lo_svs(ptr addrspace(5) inre
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
-; GFX10-NEXT:    v_lshl_add_u32 v0, v0, 2, s2
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v3, 0xffff0000
-; GFX10-NEXT:    scratch_load_short_d16 v3, v0, off offset:2
+; GFX10-NEXT:    v_add3_u32 v0, s2, v0, 2
+; GFX10-NEXT:    scratch_load_short_d16 v3, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v3
 ; GFX10-NEXT:    s_endpgm
@@ -891,7 +952,9 @@ define amdgpu_ps void @test_scratch_load_i16_to_d16_lo_svs(ptr addrspace(5) inre
 ; GFX11-LABEL: test_scratch_load_i16_to_d16_lo_svs:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    v_dual_mov_b32 v3, 0xffff0000 :: v_dual_lshlrev_b32 v0, 2, v0
-; GFX11-NEXT:    scratch_load_d16_b16 v3, v0, s0 offset:2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_add3_u32 v0, s0, v0, 2
+; GFX11-NEXT:    scratch_load_d16_b16 v3, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v3
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -914,9 +977,10 @@ define amdgpu_ps void @test_scratch_load_i8_zext_to_d16_hi_svs(ptr addrspace(5)
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
-; GFX10-NEXT:    v_lshl_add_u32 v0, v0, 2, s2
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v3, -1
-; GFX10-NEXT:    scratch_load_ubyte_d16_hi v3, v0, off offset:1
+; GFX10-NEXT:    v_add3_u32 v0, s2, v0, 1
+; GFX10-NEXT:    scratch_load_ubyte_d16_hi v3, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v3
 ; GFX10-NEXT:    s_endpgm
@@ -924,7 +988,9 @@ define amdgpu_ps void @test_scratch_load_i8_zext_to_d16_hi_svs(ptr addrspace(5)
 ; GFX11-LABEL: test_scratch_load_i8_zext_to_d16_hi_svs:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    v_dual_mov_b32 v3, -1 :: v_dual_lshlrev_b32 v0, 2, v0
-; GFX11-NEXT:    scratch_load_d16_hi_u8 v3, v0, s0 offset:1
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_add3_u32 v0, s0, v0, 1
+; GFX11-NEXT:    scratch_load_d16_hi_u8 v3, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v3
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -947,9 +1013,10 @@ define amdgpu_ps void @test_scratch_load_i8_sext_to_d16_hi_svs(ptr addrspace(5)
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
-; GFX10-NEXT:    v_lshl_add_u32 v0, v0, 2, s2
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v3, -1
-; GFX10-NEXT:    scratch_load_sbyte_d16_hi v3, v0, off offset:1
+; GFX10-NEXT:    v_add3_u32 v0, s2, v0, 1
+; GFX10-NEXT:    scratch_load_sbyte_d16_hi v3, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v3
 ; GFX10-NEXT:    s_endpgm
@@ -957,7 +1024,9 @@ define amdgpu_ps void @test_scratch_load_i8_sext_to_d16_hi_svs(ptr addrspace(5)
 ; GFX11-LABEL: test_scratch_load_i8_sext_to_d16_hi_svs:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    v_dual_mov_b32 v3, -1 :: v_dual_lshlrev_b32 v0, 2, v0
-; GFX11-NEXT:    scratch_load_d16_hi_i8 v3, v0, s0 offset:1
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_add3_u32 v0, s0, v0, 1
+; GFX11-NEXT:    scratch_load_d16_hi_i8 v3, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v3
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -980,9 +1049,10 @@ define amdgpu_ps void @test_scratch_load_i16_to_d16_hi_svs(ptr addrspace(5) inre
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
-; GFX10-NEXT:    v_lshl_add_u32 v0, v0, 2, s2
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v3, -1
-; GFX10-NEXT:    scratch_load_short_d16_hi v3, v0, off offset:2
+; GFX10-NEXT:    v_add3_u32 v0, s2, v0, 2
+; GFX10-NEXT:    scratch_load_short_d16_hi v3, v0, off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    flat_store_dword v[1:2], v3
 ; GFX10-NEXT:    s_endpgm
@@ -990,7 +1060,9 @@ define amdgpu_ps void @test_scratch_load_i16_to_d16_hi_svs(ptr addrspace(5) inre
 ; GFX11-LABEL: test_scratch_load_i16_to_d16_hi_svs:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    v_dual_mov_b32 v3, -1 :: v_dual_lshlrev_b32 v0, 2, v0
-; GFX11-NEXT:    scratch_load_d16_hi_b16 v3, v0, s0 offset:2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_add3_u32 v0, s0, v0, 2
+; GFX11-NEXT:    scratch_load_d16_hi_b16 v3, v0, off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    flat_store_b32 v[1:2], v3
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -1013,17 +1085,20 @@ define amdgpu_ps void @test_scratch_store_b8_from_d16_hi_svs(ptr %in, ptr addrsp
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
 ; GFX10-NEXT:    flat_load_dword v0, v[0:1]
-; GFX10-NEXT:    v_lshl_add_u32 v1, v2, 2, s2
+; GFX10-NEXT:    v_lshlrev_b32_e32 v1, 2, v2
+; GFX10-NEXT:    v_add3_u32 v1, s2, v1, 4
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
-; GFX10-NEXT:    scratch_store_byte_d16_hi v1, v0, off offset:4
+; GFX10-NEXT:    scratch_store_byte_d16_hi v1, v0, off
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_store_b8_from_d16_hi_svs:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    flat_load_b32 v0, v[0:1]
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v1, 2, v2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_add3_u32 v1, s0, v1, 4
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
-; GFX11-NEXT:    scratch_store_d16_hi_b8 v1, v0, s0 offset:4
+; GFX11-NEXT:    scratch_store_d16_hi_b8 v1, v0, off
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
 bb:
@@ -1044,17 +1119,20 @@ define amdgpu_ps void @test_scratch_store_b16_from_d16_hi_svs(ptr %in, ptr addrs
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
 ; GFX10-NEXT:    flat_load_dword v0, v[0:1]
-; GFX10-NEXT:    v_lshl_add_u32 v1, v2, 2, s2
+; GFX10-NEXT:    v_lshlrev_b32_e32 v1, 2, v2
+; GFX10-NEXT:    v_add3_u32 v1, s2, v1, 2
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
-; GFX10-NEXT:    scratch_store_short_d16_hi v1, v0, off offset:2
+; GFX10-NEXT:    scratch_store_short_d16_hi v1, v0, off
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: test_scratch_store_b16_from_d16_hi_svs:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    flat_load_b32 v0, v[0:1]
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v1, 2, v2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_add3_u32 v1, s0, v1, 2
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
-; GFX11-NEXT:    scratch_store_d16_hi_b16 v1, v0, s0 offset:2
+; GFX11-NEXT:    scratch_store_d16_hi_b16 v1, v0, off
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
 bb:
diff --git a/llvm/test/CodeGen/AMDGPU/flat-scratch-svs.ll b/llvm/test/CodeGen/AMDGPU/flat-scratch-svs.ll
index 4d162e26ebd05..74ba38cddff59 100644
--- a/llvm/test/CodeGen/AMDGPU/flat-scratch-svs.ll
+++ b/llvm/test/CodeGen/AMDGPU/flat-scratch-svs.ll
@@ -13,16 +13,21 @@ define amdgpu_kernel void @soff1_voff1(i32 %soff) {
 ; GFX940-SDAG-LABEL: soff1_voff1:
 ; GFX940-SDAG:       ; %bb.0: ; %bb
 ; GFX940-SDAG-NEXT:    s_load_dword s0, s[0:1], 0x24
-; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 1
-; GFX940-SDAG-NEXT:    v_mov_b32_e32 v2, 2
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 4
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v2, 1
 ; GFX940-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX940-SDAG-NEXT:    s_add_i32 s0, s0, 4
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, s0 offset:1 sc0 sc1
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v1, s0, v1
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v0, v1, v0
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v1, 1, v0
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v3, 2, v0
+; GFX940-SDAG-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v2, s0 offset:2 sc0 sc1
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 2
+; GFX940-SDAG-NEXT:    scratch_store_byte v3, v1, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 4
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, s0 offset:4 sc0 sc1
+; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-SDAG-NEXT:    s_endpgm
 ;
@@ -31,15 +36,18 @@ define amdgpu_kernel void @soff1_voff1(i32 %soff) {
 ; GFX940-GISEL-NEXT:    s_load_dword s0, s[0:1], 0x24
 ; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 4
 ; GFX940-GISEL-NEXT:    v_mov_b32_e32 v2, 1
-; GFX940-GISEL-NEXT:    v_mov_b32_e32 v3, 2
 ; GFX940-GISEL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-GISEL-NEXT:    v_add3_u32 v0, v1, s0, v0
-; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 4
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v2, off offset:1 sc0 sc1
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v1, 1, v0
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v3, 2, v0
+; GFX940-GISEL-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v3, off offset:2 sc0 sc1
+; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 2
+; GFX940-GISEL-NEXT:    scratch_store_byte v3, v1, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off offset:4 sc0 sc1
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v0, 4, v0
+; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 4
+; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-GISEL-NEXT:    s_endpgm
 ;
@@ -47,14 +55,17 @@ define amdgpu_kernel void @soff1_voff1(i32 %soff) {
 ; GFX11-SDAG:       ; %bb.0: ; %bb
 ; GFX11-SDAG-NEXT:    s_load_b32 s0, s[0:1], 0x24
 ; GFX11-SDAG-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_mov_b32 v2, 2
-; GFX11-SDAG-NEXT:    v_mov_b32_e32 v3, 4
 ; GFX11-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-SDAG-NEXT:    v_add3_u32 v0, 4, s0, v0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v1, off offset:1 dlc
+; GFX11-SDAG-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-SDAG-NEXT:    v_dual_mov_b32 v3, 4 :: v_dual_add_nc_u32 v4, 1, v0
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v5, 2, v0
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX11-SDAG-NEXT:    scratch_store_b8 v4, v1, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v2, off offset:2 dlc
+; GFX11-SDAG-NEXT:    scratch_store_b8 v5, v2, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v3, off offset:4 dlc
+; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v3, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-SDAG-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-SDAG-NEXT:    s_endpgm
@@ -63,14 +74,17 @@ define amdgpu_kernel void @soff1_voff1(i32 %soff) {
 ; GFX11-GISEL:       ; %bb.0: ; %bb
 ; GFX11-GISEL-NEXT:    s_load_b32 s0, s[0:1], 0x24
 ; GFX11-GISEL-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_mov_b32 v2, 2
-; GFX11-GISEL-NEXT:    v_mov_b32_e32 v3, 4
 ; GFX11-GISEL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-GISEL-NEXT:    v_add3_u32 v0, 4, s0, v0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v1, off offset:1 dlc
+; GFX11-GISEL-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-GISEL-NEXT:    v_dual_mov_b32 v3, 4 :: v_dual_add_nc_u32 v4, 1, v0
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v5, 2, v0
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX11-GISEL-NEXT:    scratch_store_b8 v4, v1, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v2, off offset:2 dlc
+; GFX11-GISEL-NEXT:    scratch_store_b8 v5, v2, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v3, off offset:4 dlc
+; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v3, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-GISEL-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-GISEL-NEXT:    s_endpgm
@@ -94,17 +108,21 @@ define amdgpu_kernel void @soff1_voff2(i32 %soff) {
 ; GFX940-SDAG-LABEL: soff1_voff2:
 ; GFX940-SDAG:       ; %bb.0: ; %bb
 ; GFX940-SDAG-NEXT:    s_load_dword s0, s[0:1], 0x24
-; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 1
-; GFX940-SDAG-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
-; GFX940-SDAG-NEXT:    v_mov_b32_e32 v2, 2
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 4
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v2, 1
 ; GFX940-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX940-SDAG-NEXT:    s_add_i32 s0, s0, 4
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, s0 offset:1 sc0 sc1
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v1, s0, v1
+; GFX940-SDAG-NEXT:    v_lshl_add_u32 v0, v0, 1, v1
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v1, 1, v0
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v3, 2, v0
+; GFX940-SDAG-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v2, s0 offset:2 sc0 sc1
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 2
+; GFX940-SDAG-NEXT:    scratch_store_byte v3, v1, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 4
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, s0 offset:4 sc0 sc1
+; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-SDAG-NEXT:    s_endpgm
 ;
@@ -116,13 +134,16 @@ define amdgpu_kernel void @soff1_voff2(i32 %soff) {
 ; GFX940-GISEL-NEXT:    v_mov_b32_e32 v2, 1
 ; GFX940-GISEL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-GISEL-NEXT:    v_add3_u32 v0, v1, s0, v0
-; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 2
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v2, off offset:1 sc0 sc1
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v1, 1, v0
+; GFX940-GISEL-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off offset:2 sc0 sc1
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v1, 2, v0
+; GFX940-GISEL-NEXT:    v_mov_b32_e32 v2, 2
+; GFX940-GISEL-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 4
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off offset:4 sc0 sc1
+; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-GISEL-NEXT:    s_endpgm
 ;
@@ -132,13 +153,16 @@ define amdgpu_kernel void @soff1_voff2(i32 %soff) {
 ; GFX11-SDAG-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_lshlrev_b32 v0, 1, v0
 ; GFX11-SDAG-NEXT:    v_dual_mov_b32 v2, 2 :: v_dual_mov_b32 v3, 4
 ; GFX11-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-SDAG-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11-SDAG-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
 ; GFX11-SDAG-NEXT:    v_add3_u32 v0, 4, s0, v0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v1, off offset:1 dlc
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v4, 1, v0
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v5, 2, v0
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX11-SDAG-NEXT:    scratch_store_b8 v4, v1, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v2, off offset:2 dlc
+; GFX11-SDAG-NEXT:    scratch_store_b8 v5, v2, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v3, off offset:4 dlc
+; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v3, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-SDAG-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-SDAG-NEXT:    s_endpgm
@@ -149,13 +173,16 @@ define amdgpu_kernel void @soff1_voff2(i32 %soff) {
 ; GFX11-GISEL-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_lshlrev_b32 v0, 1, v0
 ; GFX11-GISEL-NEXT:    v_dual_mov_b32 v2, 2 :: v_dual_mov_b32 v3, 4
 ; GFX11-GISEL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-GISEL-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11-GISEL-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
 ; GFX11-GISEL-NEXT:    v_add3_u32 v0, 4, s0, v0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v1, off offset:1 dlc
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v4, 1, v0
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v5, 2, v0
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX11-GISEL-NEXT:    scratch_store_b8 v4, v1, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v2, off offset:2 dlc
+; GFX11-GISEL-NEXT:    scratch_store_b8 v5, v2, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v3, off offset:4 dlc
+; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v3, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-GISEL-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-GISEL-NEXT:    s_endpgm
@@ -179,17 +206,21 @@ define amdgpu_kernel void @soff1_voff4(i32 %soff) {
 ; GFX940-SDAG-LABEL: soff1_voff4:
 ; GFX940-SDAG:       ; %bb.0: ; %bb
 ; GFX940-SDAG-NEXT:    s_load_dword s0, s[0:1], 0x24
-; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 1
-; GFX940-SDAG-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX940-SDAG-NEXT:    v_mov_b32_e32 v2, 2
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 4
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v2, 1
 ; GFX940-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX940-SDAG-NEXT:    s_add_i32 s0, s0, 4
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, s0 offset:1 sc0 sc1
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v1, s0, v1
+; GFX940-SDAG-NEXT:    v_lshl_add_u32 v0, v0, 2, v1
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v1, 1, v0
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v3, 2, v0
+; GFX940-SDAG-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v2, s0 offset:2 sc0 sc1
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 2
+; GFX940-SDAG-NEXT:    scratch_store_byte v3, v1, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 4
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, s0 offset:4 sc0 sc1
+; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-SDAG-NEXT:    s_endpgm
 ;
@@ -201,13 +232,16 @@ define amdgpu_kernel void @soff1_voff4(i32 %soff) {
 ; GFX940-GISEL-NEXT:    v_mov_b32_e32 v2, 1
 ; GFX940-GISEL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-GISEL-NEXT:    v_add3_u32 v0, v1, s0, v0
-; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 2
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v2, off offset:1 sc0 sc1
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v1, 1, v0
+; GFX940-GISEL-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off offset:2 sc0 sc1
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v1, 2, v0
+; GFX940-GISEL-NEXT:    v_mov_b32_e32 v2, 2
+; GFX940-GISEL-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 4
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off offset:4 sc0 sc1
+; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-GISEL-NEXT:    s_endpgm
 ;
@@ -217,12 +251,16 @@ define amdgpu_kernel void @soff1_voff4(i32 %soff) {
 ; GFX11-SDAG-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_lshlrev_b32 v0, 2, v0
 ; GFX11-SDAG-NEXT:    v_dual_mov_b32 v2, 2 :: v_dual_mov_b32 v3, 4
 ; GFX11-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-SDAG-NEXT:    s_add_i32 s0, s0, 4
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v1, s0 offset:1 dlc
+; GFX11-SDAG-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX11-SDAG-NEXT:    v_add3_u32 v0, 4, s0, v0
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v4, 1, v0
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v5, 2, v0
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX11-SDAG-NEXT:    scratch_store_b8 v4, v1, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v2, s0 offset:2 dlc
+; GFX11-SDAG-NEXT:    scratch_store_b8 v5, v2, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v3, s0 offset:4 dlc
+; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v3, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-SDAG-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-SDAG-NEXT:    s_endpgm
@@ -233,13 +271,16 @@ define amdgpu_kernel void @soff1_voff4(i32 %soff) {
 ; GFX11-GISEL-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_lshlrev_b32 v0, 2, v0
 ; GFX11-GISEL-NEXT:    v_dual_mov_b32 v2, 2 :: v_dual_mov_b32 v3, 4
 ; GFX11-GISEL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-GISEL-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11-GISEL-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
 ; GFX11-GISEL-NEXT:    v_add3_u32 v0, 4, s0, v0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v1, off offset:1 dlc
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v4, 1, v0
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v5, 2, v0
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX11-GISEL-NEXT:    scratch_store_b8 v4, v1, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v2, off offset:2 dlc
+; GFX11-GISEL-NEXT:    scratch_store_b8 v5, v2, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v3, off offset:4 dlc
+; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v3, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-GISEL-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-GISEL-NEXT:    s_endpgm
@@ -263,17 +304,22 @@ define amdgpu_kernel void @soff2_voff1(i32 %soff) {
 ; GFX940-SDAG-LABEL: soff2_voff1:
 ; GFX940-SDAG:       ; %bb.0: ; %bb
 ; GFX940-SDAG-NEXT:    s_load_dword s0, s[0:1], 0x24
-; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 1
-; GFX940-SDAG-NEXT:    v_mov_b32_e32 v2, 2
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 4
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v2, 1
 ; GFX940-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-SDAG-NEXT:    s_lshl_b32 s0, s0, 1
-; GFX940-SDAG-NEXT:    s_add_i32 s0, s0, 4
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, s0 offset:1 sc0 sc1
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v1, s0, v1
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v0, v1, v0
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v1, 1, v0
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v3, 2, v0
+; GFX940-SDAG-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v2, s0 offset:2 sc0 sc1
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 2
+; GFX940-SDAG-NEXT:    scratch_store_byte v3, v1, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 4
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, s0 offset:4 sc0 sc1
+; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-SDAG-NEXT:    s_endpgm
 ;
@@ -282,16 +328,19 @@ define amdgpu_kernel void @soff2_voff1(i32 %soff) {
 ; GFX940-GISEL-NEXT:    s_load_dword s0, s[0:1], 0x24
 ; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 4
 ; GFX940-GISEL-NEXT:    v_mov_b32_e32 v2, 1
-; GFX940-GISEL-NEXT:    v_mov_b32_e32 v3, 2
 ; GFX940-GISEL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-GISEL-NEXT:    s_lshl_b32 s0, s0, 1
 ; GFX940-GISEL-NEXT:    v_add3_u32 v0, v1, s0, v0
-; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 4
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v2, off offset:1 sc0 sc1
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v1, 1, v0
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v3, 2, v0
+; GFX940-GISEL-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v3, off offset:2 sc0 sc1
+; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 2
+; GFX940-GISEL-NEXT:    scratch_store_byte v3, v1, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off offset:4 sc0 sc1
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v0, 4, v0
+; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 4
+; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-GISEL-NEXT:    s_endpgm
 ;
@@ -299,16 +348,18 @@ define amdgpu_kernel void @soff2_voff1(i32 %soff) {
 ; GFX11-SDAG:       ; %bb.0: ; %bb
 ; GFX11-SDAG-NEXT:    s_load_b32 s0, s[0:1], 0x24
 ; GFX11-SDAG-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_mov_b32 v2, 2
-; GFX11-SDAG-NEXT:    v_mov_b32_e32 v3, 4
 ; GFX11-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-SDAG-NEXT:    s_lshl_b32 s0, s0, 1
-; GFX11-SDAG-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
+; GFX11-SDAG-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(VALU_DEP_1)
 ; GFX11-SDAG-NEXT:    v_add3_u32 v0, 4, s0, v0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v1, off offset:1 dlc
+; GFX11-SDAG-NEXT:    v_dual_mov_b32 v3, 4 :: v_dual_add_nc_u32 v4, 1, v0
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v5, 2, v0
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX11-SDAG-NEXT:    scratch_store_b8 v4, v1, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v2, off offset:2 dlc
+; GFX11-SDAG-NEXT:    scratch_store_b8 v5, v2, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v3, off offset:4 dlc
+; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v3, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-SDAG-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-SDAG-NEXT:    s_endpgm
@@ -317,16 +368,18 @@ define amdgpu_kernel void @soff2_voff1(i32 %soff) {
 ; GFX11-GISEL:       ; %bb.0: ; %bb
 ; GFX11-GISEL-NEXT:    s_load_b32 s0, s[0:1], 0x24
 ; GFX11-GISEL-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_mov_b32 v2, 2
-; GFX11-GISEL-NEXT:    v_mov_b32_e32 v3, 4
 ; GFX11-GISEL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-GISEL-NEXT:    s_lshl_b32 s0, s0, 1
-; GFX11-GISEL-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
+; GFX11-GISEL-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(VALU_DEP_1)
 ; GFX11-GISEL-NEXT:    v_add3_u32 v0, 4, s0, v0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v1, off offset:1 dlc
+; GFX11-GISEL-NEXT:    v_dual_mov_b32 v3, 4 :: v_dual_add_nc_u32 v4, 1, v0
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v5, 2, v0
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX11-GISEL-NEXT:    scratch_store_b8 v4, v1, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v2, off offset:2 dlc
+; GFX11-GISEL-NEXT:    scratch_store_b8 v5, v2, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v3, off offset:4 dlc
+; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v3, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-GISEL-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-GISEL-NEXT:    s_endpgm
@@ -350,18 +403,22 @@ define amdgpu_kernel void @soff2_voff2(i32 %soff) {
 ; GFX940-SDAG-LABEL: soff2_voff2:
 ; GFX940-SDAG:       ; %bb.0: ; %bb
 ; GFX940-SDAG-NEXT:    s_load_dword s0, s[0:1], 0x24
-; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 1
-; GFX940-SDAG-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
-; GFX940-SDAG-NEXT:    v_mov_b32_e32 v2, 2
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 4
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v2, 1
 ; GFX940-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-SDAG-NEXT:    s_lshl_b32 s0, s0, 1
-; GFX940-SDAG-NEXT:    s_add_i32 s0, s0, 4
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, s0 offset:1 sc0 sc1
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v1, s0, v1
+; GFX940-SDAG-NEXT:    v_lshl_add_u32 v0, v0, 1, v1
+; GFX940-SDAG-NEXT:    v_or_b32_e32 v1, 1, v0
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v3, 2, v0
+; GFX940-SDAG-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v2, s0 offset:2 sc0 sc1
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 2
+; GFX940-SDAG-NEXT:    scratch_store_byte v3, v1, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 4
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, s0 offset:4 sc0 sc1
+; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-SDAG-NEXT:    s_endpgm
 ;
@@ -374,13 +431,16 @@ define amdgpu_kernel void @soff2_voff2(i32 %soff) {
 ; GFX940-GISEL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-GISEL-NEXT:    s_lshl_b32 s0, s0, 1
 ; GFX940-GISEL-NEXT:    v_add3_u32 v0, v1, s0, v0
-; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 2
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v2, off offset:1 sc0 sc1
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v1, 1, v0
+; GFX940-GISEL-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off offset:2 sc0 sc1
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v1, 2, v0
+; GFX940-GISEL-NEXT:    v_mov_b32_e32 v2, 2
+; GFX940-GISEL-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 4
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off offset:4 sc0 sc1
+; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-GISEL-NEXT:    s_endpgm
 ;
@@ -393,11 +453,15 @@ define amdgpu_kernel void @soff2_voff2(i32 %soff) {
 ; GFX11-SDAG-NEXT:    s_lshl_b32 s0, s0, 1
 ; GFX11-SDAG-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instid1(SALU_CYCLE_1)
 ; GFX11-SDAG-NEXT:    v_add3_u32 v0, 4, s0, v0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v1, off offset:1 dlc
+; GFX11-SDAG-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-SDAG-NEXT:    v_or_b32_e32 v4, 1, v0
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v5, 2, v0
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX11-SDAG-NEXT:    scratch_store_b8 v4, v1, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v2, off offset:2 dlc
+; GFX11-SDAG-NEXT:    scratch_store_b8 v5, v2, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v3, off offset:4 dlc
+; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v3, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-SDAG-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-SDAG-NEXT:    s_endpgm
@@ -411,11 +475,15 @@ define amdgpu_kernel void @soff2_voff2(i32 %soff) {
 ; GFX11-GISEL-NEXT:    s_lshl_b32 s0, s0, 1
 ; GFX11-GISEL-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instid1(SALU_CYCLE_1)
 ; GFX11-GISEL-NEXT:    v_add3_u32 v0, 4, s0, v0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v1, off offset:1 dlc
+; GFX11-GISEL-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v4, 1, v0
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v5, 2, v0
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX11-GISEL-NEXT:    scratch_store_b8 v4, v1, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v2, off offset:2 dlc
+; GFX11-GISEL-NEXT:    scratch_store_b8 v5, v2, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v3, off offset:4 dlc
+; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v3, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-GISEL-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-GISEL-NEXT:    s_endpgm
@@ -439,18 +507,22 @@ define amdgpu_kernel void @soff2_voff4(i32 %soff) {
 ; GFX940-SDAG-LABEL: soff2_voff4:
 ; GFX940-SDAG:       ; %bb.0: ; %bb
 ; GFX940-SDAG-NEXT:    s_load_dword s0, s[0:1], 0x24
-; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 1
-; GFX940-SDAG-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX940-SDAG-NEXT:    v_mov_b32_e32 v2, 2
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 4
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v2, 1
 ; GFX940-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-SDAG-NEXT:    s_lshl_b32 s0, s0, 1
-; GFX940-SDAG-NEXT:    s_add_i32 s0, s0, 4
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, s0 offset:1 sc0 sc1
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v1, s0, v1
+; GFX940-SDAG-NEXT:    v_lshl_add_u32 v0, v0, 2, v1
+; GFX940-SDAG-NEXT:    v_or_b32_e32 v1, 1, v0
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v3, 2, v0
+; GFX940-SDAG-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v2, s0 offset:2 sc0 sc1
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 2
+; GFX940-SDAG-NEXT:    scratch_store_byte v3, v1, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 4
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, s0 offset:4 sc0 sc1
+; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-SDAG-NEXT:    s_endpgm
 ;
@@ -463,30 +535,37 @@ define amdgpu_kernel void @soff2_voff4(i32 %soff) {
 ; GFX940-GISEL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-GISEL-NEXT:    s_lshl_b32 s0, s0, 1
 ; GFX940-GISEL-NEXT:    v_add3_u32 v0, v1, s0, v0
-; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 2
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v2, off offset:1 sc0 sc1
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v1, 1, v0
+; GFX940-GISEL-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off offset:2 sc0 sc1
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v1, 2, v0
+; GFX940-GISEL-NEXT:    v_mov_b32_e32 v2, 2
+; GFX940-GISEL-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 4
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off offset:4 sc0 sc1
+; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-GISEL-NEXT:    s_endpgm
 ;
 ; GFX11-SDAG-LABEL: soff2_voff4:
 ; GFX11-SDAG:       ; %bb.0: ; %bb
 ; GFX11-SDAG-NEXT:    s_load_b32 s0, s[0:1], 0x24
-; GFX11-SDAG-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_mov_b32 v2, 2
-; GFX11-SDAG-NEXT:    v_dual_mov_b32 v3, 4 :: v_dual_lshlrev_b32 v0, 2, v0
+; GFX11-SDAG-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_lshlrev_b32 v0, 2, v0
+; GFX11-SDAG-NEXT:    v_dual_mov_b32 v2, 2 :: v_dual_mov_b32 v3, 4
 ; GFX11-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-SDAG-NEXT:    s_lshl_b32 s0, s0, 1
-; GFX11-SDAG-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11-SDAG-NEXT:    s_add_i32 s0, s0, 4
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v1, s0 offset:1 dlc
+; GFX11-SDAG-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instid1(SALU_CYCLE_1)
+; GFX11-SDAG-NEXT:    v_add3_u32 v0, 4, s0, v0
+; GFX11-SDAG-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-SDAG-NEXT:    v_or_b32_e32 v4, 1, v0
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v5, 2, v0
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX11-SDAG-NEXT:    scratch_store_b8 v4, v1, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v2, s0 offset:2 dlc
+; GFX11-SDAG-NEXT:    scratch_store_b8 v5, v2, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v3, s0 offset:4 dlc
+; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v3, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-SDAG-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-SDAG-NEXT:    s_endpgm
@@ -500,11 +579,15 @@ define amdgpu_kernel void @soff2_voff4(i32 %soff) {
 ; GFX11-GISEL-NEXT:    s_lshl_b32 s0, s0, 1
 ; GFX11-GISEL-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instid1(SALU_CYCLE_1)
 ; GFX11-GISEL-NEXT:    v_add3_u32 v0, 4, s0, v0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v1, off offset:1 dlc
+; GFX11-GISEL-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v4, 1, v0
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v5, 2, v0
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX11-GISEL-NEXT:    scratch_store_b8 v4, v1, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v2, off offset:2 dlc
+; GFX11-GISEL-NEXT:    scratch_store_b8 v5, v2, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v3, off offset:4 dlc
+; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v3, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-GISEL-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-GISEL-NEXT:    s_endpgm
@@ -528,17 +611,22 @@ define amdgpu_kernel void @soff4_voff1(i32 %soff) {
 ; GFX940-SDAG-LABEL: soff4_voff1:
 ; GFX940-SDAG:       ; %bb.0: ; %bb
 ; GFX940-SDAG-NEXT:    s_load_dword s0, s[0:1], 0x24
-; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 1
-; GFX940-SDAG-NEXT:    v_mov_b32_e32 v2, 2
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 4
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v2, 1
 ; GFX940-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-SDAG-NEXT:    s_lshl_b32 s0, s0, 2
-; GFX940-SDAG-NEXT:    s_add_i32 s0, s0, 4
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, s0 offset:1 sc0 sc1
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v1, s0, v1
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v0, v1, v0
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v1, 1, v0
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v3, 2, v0
+; GFX940-SDAG-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v2, s0 offset:2 sc0 sc1
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 2
+; GFX940-SDAG-NEXT:    scratch_store_byte v3, v1, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 4
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, s0 offset:4 sc0 sc1
+; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-SDAG-NEXT:    s_endpgm
 ;
@@ -547,34 +635,38 @@ define amdgpu_kernel void @soff4_voff1(i32 %soff) {
 ; GFX940-GISEL-NEXT:    s_load_dword s0, s[0:1], 0x24
 ; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 4
 ; GFX940-GISEL-NEXT:    v_mov_b32_e32 v2, 1
-; GFX940-GISEL-NEXT:    v_mov_b32_e32 v3, 2
 ; GFX940-GISEL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-GISEL-NEXT:    s_lshl_b32 s0, s0, 2
 ; GFX940-GISEL-NEXT:    v_add3_u32 v0, v1, s0, v0
-; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 4
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v2, off offset:1 sc0 sc1
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v1, 1, v0
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v3, 2, v0
+; GFX940-GISEL-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v3, off offset:2 sc0 sc1
+; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 2
+; GFX940-GISEL-NEXT:    scratch_store_byte v3, v1, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off offset:4 sc0 sc1
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v0, 4, v0
+; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 4
+; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-GISEL-NEXT:    s_endpgm
 ;
 ; GFX11-SDAG-LABEL: soff4_voff1:
 ; GFX11-SDAG:       ; %bb.0: ; %bb
 ; GFX11-SDAG-NEXT:    s_load_b32 s0, s[0:1], 0x24
-; GFX11-SDAG-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_mov_b32 v4, 4
-; GFX11-SDAG-NEXT:    v_mov_b32_e32 v3, 2
+; GFX11-SDAG-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_mov_b32 v2, 2
 ; GFX11-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-SDAG-NEXT:    s_lshl_b32 s0, s0, 2
-; GFX11-SDAG-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11-SDAG-NEXT:    v_add3_u32 v2, 4, s0, v0
-; GFX11-SDAG-NEXT:    s_add_i32 s0, s0, 4
-; GFX11-SDAG-NEXT:    scratch_store_b8 v2, v1, off offset:1 dlc
+; GFX11-SDAG-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX11-SDAG-NEXT:    v_add3_u32 v0, 4, s0, v0
+; GFX11-SDAG-NEXT:    v_dual_mov_b32 v3, 4 :: v_dual_add_nc_u32 v4, 1, v0
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v5, 2, v0
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX11-SDAG-NEXT:    scratch_store_b8 v4, v1, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v2, v3, off offset:2 dlc
+; GFX11-SDAG-NEXT:    scratch_store_b8 v5, v2, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v4, s0 offset:4 dlc
+; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v3, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-SDAG-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-SDAG-NEXT:    s_endpgm
@@ -583,16 +675,18 @@ define amdgpu_kernel void @soff4_voff1(i32 %soff) {
 ; GFX11-GISEL:       ; %bb.0: ; %bb
 ; GFX11-GISEL-NEXT:    s_load_b32 s0, s[0:1], 0x24
 ; GFX11-GISEL-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_mov_b32 v2, 2
-; GFX11-GISEL-NEXT:    v_mov_b32_e32 v3, 4
 ; GFX11-GISEL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-GISEL-NEXT:    s_lshl_b32 s0, s0, 2
-; GFX11-GISEL-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
+; GFX11-GISEL-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(VALU_DEP_1)
 ; GFX11-GISEL-NEXT:    v_add3_u32 v0, 4, s0, v0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v1, off offset:1 dlc
+; GFX11-GISEL-NEXT:    v_dual_mov_b32 v3, 4 :: v_dual_add_nc_u32 v4, 1, v0
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v5, 2, v0
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX11-GISEL-NEXT:    scratch_store_b8 v4, v1, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v2, off offset:2 dlc
+; GFX11-GISEL-NEXT:    scratch_store_b8 v5, v2, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v3, off offset:4 dlc
+; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v3, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-GISEL-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-GISEL-NEXT:    s_endpgm
@@ -616,18 +710,22 @@ define amdgpu_kernel void @soff4_voff2(i32 %soff) {
 ; GFX940-SDAG-LABEL: soff4_voff2:
 ; GFX940-SDAG:       ; %bb.0: ; %bb
 ; GFX940-SDAG-NEXT:    s_load_dword s0, s[0:1], 0x24
-; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 1
-; GFX940-SDAG-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
-; GFX940-SDAG-NEXT:    v_mov_b32_e32 v2, 2
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 4
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v2, 1
 ; GFX940-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-SDAG-NEXT:    s_lshl_b32 s0, s0, 2
-; GFX940-SDAG-NEXT:    s_add_i32 s0, s0, 4
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, s0 offset:1 sc0 sc1
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v1, s0, v1
+; GFX940-SDAG-NEXT:    v_lshl_add_u32 v0, v0, 1, v1
+; GFX940-SDAG-NEXT:    v_or_b32_e32 v1, 1, v0
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v3, 2, v0
+; GFX940-SDAG-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v2, s0 offset:2 sc0 sc1
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 2
+; GFX940-SDAG-NEXT:    scratch_store_byte v3, v1, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 4
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, s0 offset:4 sc0 sc1
+; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-SDAG-NEXT:    s_endpgm
 ;
@@ -640,13 +738,16 @@ define amdgpu_kernel void @soff4_voff2(i32 %soff) {
 ; GFX940-GISEL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-GISEL-NEXT:    s_lshl_b32 s0, s0, 2
 ; GFX940-GISEL-NEXT:    v_add3_u32 v0, v1, s0, v0
-; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 2
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v2, off offset:1 sc0 sc1
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v1, 1, v0
+; GFX940-GISEL-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off offset:2 sc0 sc1
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v1, 2, v0
+; GFX940-GISEL-NEXT:    v_mov_b32_e32 v2, 2
+; GFX940-GISEL-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 4
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off offset:4 sc0 sc1
+; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-GISEL-NEXT:    s_endpgm
 ;
@@ -654,18 +755,20 @@ define amdgpu_kernel void @soff4_voff2(i32 %soff) {
 ; GFX11-SDAG:       ; %bb.0: ; %bb
 ; GFX11-SDAG-NEXT:    s_load_b32 s0, s[0:1], 0x24
 ; GFX11-SDAG-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_lshlrev_b32 v0, 1, v0
-; GFX11-SDAG-NEXT:    v_mov_b32_e32 v2, 2
-; GFX11-SDAG-NEXT:    v_mov_b32_e32 v4, 4
+; GFX11-SDAG-NEXT:    v_dual_mov_b32 v2, 2 :: v_dual_mov_b32 v3, 4
 ; GFX11-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-SDAG-NEXT:    s_lshl_b32 s0, s0, 2
-; GFX11-SDAG-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11-SDAG-NEXT:    v_add3_u32 v3, 4, s0, v0
-; GFX11-SDAG-NEXT:    s_add_i32 s0, s0, 4
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v1, s0 offset:1 dlc
+; GFX11-SDAG-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instid1(SALU_CYCLE_1)
+; GFX11-SDAG-NEXT:    v_add3_u32 v0, 4, s0, v0
+; GFX11-SDAG-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-SDAG-NEXT:    v_or_b32_e32 v4, 1, v0
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v5, 2, v0
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX11-SDAG-NEXT:    scratch_store_b8 v4, v1, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v3, v2, off offset:2 dlc
+; GFX11-SDAG-NEXT:    scratch_store_b8 v5, v2, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v4, s0 offset:4 dlc
+; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v3, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-SDAG-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-SDAG-NEXT:    s_endpgm
@@ -679,11 +782,15 @@ define amdgpu_kernel void @soff4_voff2(i32 %soff) {
 ; GFX11-GISEL-NEXT:    s_lshl_b32 s0, s0, 2
 ; GFX11-GISEL-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instid1(SALU_CYCLE_1)
 ; GFX11-GISEL-NEXT:    v_add3_u32 v0, 4, s0, v0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v1, off offset:1 dlc
+; GFX11-GISEL-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v4, 1, v0
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v5, 2, v0
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX11-GISEL-NEXT:    scratch_store_b8 v4, v1, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v2, off offset:2 dlc
+; GFX11-GISEL-NEXT:    scratch_store_b8 v5, v2, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v3, off offset:4 dlc
+; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v3, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-GISEL-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-GISEL-NEXT:    s_endpgm
@@ -707,18 +814,22 @@ define amdgpu_kernel void @soff4_voff4(i32 %soff) {
 ; GFX940-SDAG-LABEL: soff4_voff4:
 ; GFX940-SDAG:       ; %bb.0: ; %bb
 ; GFX940-SDAG-NEXT:    s_load_dword s0, s[0:1], 0x24
-; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 1
-; GFX940-SDAG-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX940-SDAG-NEXT:    v_mov_b32_e32 v2, 2
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 4
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v2, 1
 ; GFX940-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-SDAG-NEXT:    s_lshl_b32 s0, s0, 2
-; GFX940-SDAG-NEXT:    s_add_i32 s0, s0, 4
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, s0 offset:1 sc0 sc1
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v1, s0, v1
+; GFX940-SDAG-NEXT:    v_lshl_add_u32 v0, v0, 2, v1
+; GFX940-SDAG-NEXT:    v_or_b32_e32 v1, 1, v0
+; GFX940-SDAG-NEXT:    v_or_b32_e32 v3, 2, v0
+; GFX940-SDAG-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v2, s0 offset:2 sc0 sc1
+; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 2
+; GFX940-SDAG-NEXT:    scratch_store_byte v3, v1, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-SDAG-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX940-SDAG-NEXT:    v_mov_b32_e32 v1, 4
-; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, s0 offset:4 sc0 sc1
+; GFX940-SDAG-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-SDAG-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-SDAG-NEXT:    s_endpgm
 ;
@@ -731,30 +842,37 @@ define amdgpu_kernel void @soff4_voff4(i32 %soff) {
 ; GFX940-GISEL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-GISEL-NEXT:    s_lshl_b32 s0, s0, 2
 ; GFX940-GISEL-NEXT:    v_add3_u32 v0, v1, s0, v0
-; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 2
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v2, off offset:1 sc0 sc1
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v1, 1, v0
+; GFX940-GISEL-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off offset:2 sc0 sc1
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v1, 2, v0
+; GFX940-GISEL-NEXT:    v_mov_b32_e32 v2, 2
+; GFX940-GISEL-NEXT:    scratch_store_byte v1, v2, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-GISEL-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX940-GISEL-NEXT:    v_mov_b32_e32 v1, 4
-; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off offset:4 sc0 sc1
+; GFX940-GISEL-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-GISEL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-GISEL-NEXT:    s_endpgm
 ;
 ; GFX11-SDAG-LABEL: soff4_voff4:
 ; GFX11-SDAG:       ; %bb.0: ; %bb
 ; GFX11-SDAG-NEXT:    s_load_b32 s0, s[0:1], 0x24
-; GFX11-SDAG-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_mov_b32 v2, 2
-; GFX11-SDAG-NEXT:    v_dual_mov_b32 v3, 4 :: v_dual_lshlrev_b32 v0, 2, v0
+; GFX11-SDAG-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_lshlrev_b32 v0, 2, v0
+; GFX11-SDAG-NEXT:    v_dual_mov_b32 v2, 2 :: v_dual_mov_b32 v3, 4
 ; GFX11-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-SDAG-NEXT:    s_lshl_b32 s0, s0, 2
-; GFX11-SDAG-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11-SDAG-NEXT:    s_add_i32 s0, s0, 4
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v1, s0 offset:1 dlc
+; GFX11-SDAG-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instid1(SALU_CYCLE_1)
+; GFX11-SDAG-NEXT:    v_add3_u32 v0, 4, s0, v0
+; GFX11-SDAG-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-SDAG-NEXT:    v_or_b32_e32 v4, 1, v0
+; GFX11-SDAG-NEXT:    v_or_b32_e32 v5, 2, v0
+; GFX11-SDAG-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX11-SDAG-NEXT:    scratch_store_b8 v4, v1, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v2, s0 offset:2 dlc
+; GFX11-SDAG-NEXT:    scratch_store_b8 v5, v2, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v3, s0 offset:4 dlc
+; GFX11-SDAG-NEXT:    scratch_store_b8 v0, v3, off dlc
 ; GFX11-SDAG-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-SDAG-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-SDAG-NEXT:    s_endpgm
@@ -768,11 +886,15 @@ define amdgpu_kernel void @soff4_voff4(i32 %soff) {
 ; GFX11-GISEL-NEXT:    s_lshl_b32 s0, s0, 2
 ; GFX11-GISEL-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instid1(SALU_CYCLE_1)
 ; GFX11-GISEL-NEXT:    v_add3_u32 v0, 4, s0, v0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v1, off offset:1 dlc
+; GFX11-GISEL-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v4, 1, v0
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v5, 2, v0
+; GFX11-GISEL-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX11-GISEL-NEXT:    scratch_store_b8 v4, v1, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v2, off offset:2 dlc
+; GFX11-GISEL-NEXT:    scratch_store_b8 v5, v2, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v3, off offset:4 dlc
+; GFX11-GISEL-NEXT:    scratch_store_b8 v0, v3, off dlc
 ; GFX11-GISEL-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-GISEL-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-GISEL-NEXT:    s_endpgm
diff --git a/llvm/test/CodeGen/AMDGPU/flat-scratch.ll b/llvm/test/CodeGen/AMDGPU/flat-scratch.ll
index 0cc45f28fc501..68be23b403eb1 100644
--- a/llvm/test/CodeGen/AMDGPU/flat-scratch.ll
+++ b/llvm/test/CodeGen/AMDGPU/flat-scratch.ll
@@ -638,10 +638,11 @@ define amdgpu_kernel void @store_load_vindex_kernel() {
 ; GFX9-NEXT:    s_addc_u32 flat_scratch_hi, s1, 0
 ; GFX9-NEXT:    v_add_u32_e32 v1, 4, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 15
+; GFX9-NEXT:    v_sub_u32_e32 v0, 4, v0
 ; GFX9-NEXT:    scratch_store_dword v1, v2, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_sub_u32_e32 v0, 4, v0
-; GFX9-NEXT:    scratch_load_dword v0, v0, off offset:124 glc
+; GFX9-NEXT:    v_add_u32_e32 v0, 0x7c, v0
+; GFX9-NEXT:    scratch_load_dword v0, v0, off glc
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_endpgm
 ;
@@ -653,22 +654,24 @@ define amdgpu_kernel void @store_load_vindex_kernel() {
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v2, 15
-; GFX10-NEXT:    v_add_nc_u32_e32 v1, 4, v0
-; GFX10-NEXT:    v_sub_nc_u32_e32 v0, 4, v0
-; GFX10-NEXT:    scratch_store_dword v1, v2, off
+; GFX10-NEXT:    v_sub_nc_u32_e32 v1, 4, v0
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX10-NEXT:    v_add_nc_u32_e32 v1, 0x7c, v1
+; GFX10-NEXT:    scratch_store_dword v0, v2, off
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-NEXT:    scratch_load_dword v0, v0, off offset:124 glc dlc
+; GFX10-NEXT:    scratch_load_dword v0, v1, off glc dlc
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: store_load_vindex_kernel:
 ; GFX11:       ; %bb.0: ; %bb
-; GFX11-NEXT:    v_dual_mov_b32 v1, 15 :: v_dual_lshlrev_b32 v0, 2, v0
-; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11-NEXT:    v_sub_nc_u32_e32 v2, 4, v0
-; GFX11-NEXT:    scratch_store_b32 v0, v1, off offset:4 dlc
+; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX11-NEXT:    v_sub_nc_u32_e32 v1, 4, v0
+; GFX11-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_add_nc_u32 v1, 0x7c, v1
+; GFX11-NEXT:    scratch_store_b32 v0, v2, off offset:4 dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_load_b32 v0, v2, off offset:124 glc dlc
+; GFX11-NEXT:    scratch_load_b32 v0, v1, off glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_endpgm
 ;
@@ -687,7 +690,8 @@ define amdgpu_kernel void @store_load_vindex_kernel() {
 ; GFX9-PAL-NEXT:    s_addc_u32 flat_scratch_hi, s3, 0
 ; GFX9-PAL-NEXT:    scratch_store_dword v1, v2, off
 ; GFX9-PAL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-PAL-NEXT:    scratch_load_dword v0, v0, off offset:124 glc
+; GFX9-PAL-NEXT:    v_add_u32_e32 v0, 0x7c, v0
+; GFX9-PAL-NEXT:    scratch_load_dword v0, v0, off glc
 ; GFX9-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-PAL-NEXT:    s_endpgm
 ;
@@ -698,7 +702,8 @@ define amdgpu_kernel void @store_load_vindex_kernel() {
 ; GFX940-NEXT:    scratch_store_dword v0, v1, off offset:4 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_sub_u32_e32 v0, 4, v0
-; GFX940-NEXT:    scratch_load_dword v0, v0, off offset:124 sc0 sc1
+; GFX940-NEXT:    v_add_u32_e32 v0, 0x7c, v0
+; GFX940-NEXT:    scratch_load_dword v0, v0, off sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    s_endpgm
 ;
@@ -715,22 +720,24 @@ define amdgpu_kernel void @store_load_vindex_kernel() {
 ; GFX10-PAL-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s3
 ; GFX10-PAL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX10-PAL-NEXT:    v_mov_b32_e32 v2, 15
-; GFX10-PAL-NEXT:    v_add_nc_u32_e32 v1, 4, v0
-; GFX10-PAL-NEXT:    v_sub_nc_u32_e32 v0, 4, v0
-; GFX10-PAL-NEXT:    scratch_store_dword v1, v2, off
+; GFX10-PAL-NEXT:    v_sub_nc_u32_e32 v1, 4, v0
+; GFX10-PAL-NEXT:    v_add_nc_u32_e32 v0, 4, v0
+; GFX10-PAL-NEXT:    v_add_nc_u32_e32 v1, 0x7c, v1
+; GFX10-PAL-NEXT:    scratch_store_dword v0, v2, off
 ; GFX10-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-PAL-NEXT:    scratch_load_dword v0, v0, off offset:124 glc dlc
+; GFX10-PAL-NEXT:    scratch_load_dword v0, v1, off glc dlc
 ; GFX10-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-PAL-NEXT:    s_endpgm
 ;
 ; GFX11-PAL-LABEL: store_load_vindex_kernel:
 ; GFX11-PAL:       ; %bb.0: ; %bb
-; GFX11-PAL-NEXT:    v_dual_mov_b32 v1, 15 :: v_dual_lshlrev_b32 v0, 2, v0
-; GFX11-PAL-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11-PAL-NEXT:    v_sub_nc_u32_e32 v2, 4, v0
-; GFX11-PAL-NEXT:    scratch_store_b32 v0, v1, off offset:4 dlc
+; GFX11-PAL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX11-PAL-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX11-PAL-NEXT:    v_sub_nc_u32_e32 v1, 4, v0
+; GFX11-PAL-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_add_nc_u32 v1, 0x7c, v1
+; GFX11-PAL-NEXT:    scratch_store_b32 v0, v2, off offset:4 dlc
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-PAL-NEXT:    scratch_load_b32 v0, v2, off offset:124 glc dlc
+; GFX11-PAL-NEXT:    scratch_load_b32 v0, v1, off glc dlc
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-PAL-NEXT:    s_endpgm
 ; GCN-LABEL: store_load_vindex_kernel:
@@ -789,10 +796,10 @@ define void @store_load_vindex_foo(i32 %idx) {
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_and_b32 v1, 15, v0
-; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX11-NEXT:    v_lshl_add_u32 v0, v0, 2, s32
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2)
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v1, 2, v1
-; GFX11-NEXT:    scratch_store_b32 v0, v2, s32 dlc
+; GFX11-NEXT:    scratch_store_b32 v0, v2, off dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    scratch_load_b32 v0, v1, s32 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
@@ -815,10 +822,11 @@ define void @store_load_vindex_foo(i32 %idx) {
 ; GFX940-LABEL: store_load_vindex_foo:
 ; GFX940:       ; %bb.0: ; %bb
 ; GFX940-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX940-NEXT:    v_lshlrev_b32_e32 v1, 2, v0
+; GFX940-NEXT:    v_mov_b32_e32 v1, s32
+; GFX940-NEXT:    v_lshl_add_u32 v1, v0, 2, v1
 ; GFX940-NEXT:    v_mov_b32_e32 v2, 15
 ; GFX940-NEXT:    v_and_b32_e32 v0, 15, v0
-; GFX940-NEXT:    scratch_store_dword v1, v2, s32 sc0 sc1
+; GFX940-NEXT:    scratch_store_dword v1, v2, off sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX940-NEXT:    scratch_load_dword v0, v0, s32 sc0 sc1
@@ -844,10 +852,10 @@ define void @store_load_vindex_foo(i32 %idx) {
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-PAL-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_and_b32 v1, 15, v0
-; GFX11-PAL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX11-PAL-NEXT:    v_lshl_add_u32 v0, v0, 2, s32
 ; GFX11-PAL-NEXT:    s_delay_alu instid0(VALU_DEP_2)
 ; GFX11-PAL-NEXT:    v_lshlrev_b32_e32 v1, 2, v1
-; GFX11-PAL-NEXT:    scratch_store_b32 v0, v2, s32 dlc
+; GFX11-PAL-NEXT:    scratch_store_b32 v0, v2, off dlc
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-PAL-NEXT:    scratch_load_b32 v0, v1, s32 glc dlc
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0)
@@ -878,8 +886,9 @@ define void @private_ptr_foo(ptr addrspace(5) nocapture %arg) {
 ; GFX9-LABEL: private_ptr_foo:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0x41200000
-; GFX9-NEXT:    scratch_store_dword v0, v1, off offset:4
+; GFX9-NEXT:    scratch_store_dword v0, v1, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
 ;
@@ -887,8 +896,9 @@ define void @private_ptr_foo(ptr addrspace(5) nocapture %arg) {
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, 4, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v1, 0x41200000
-; GFX10-NEXT:    scratch_store_dword v0, v1, off offset:4
+; GFX10-NEXT:    scratch_store_dword v0, v1, off
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
 ;
@@ -896,24 +906,26 @@ define void @private_ptr_foo(ptr addrspace(5) nocapture %arg) {
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    v_mov_b32_e32 v1, 0x41200000
-; GFX11-NEXT:    scratch_store_b32 v0, v1, off offset:4
+; GFX11-NEXT:    v_dual_mov_b32 v1, 0x41200000 :: v_dual_add_nc_u32 v0, 4, v0
+; GFX11-NEXT:    scratch_store_b32 v0, v1, off
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-PAL-LABEL: private_ptr_foo:
 ; GFX9-PAL:       ; %bb.0:
 ; GFX9-PAL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-PAL-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX9-PAL-NEXT:    v_mov_b32_e32 v1, 0x41200000
-; GFX9-PAL-NEXT:    scratch_store_dword v0, v1, off offset:4
+; GFX9-PAL-NEXT:    scratch_store_dword v0, v1, off
 ; GFX9-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-PAL-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX940-LABEL: private_ptr_foo:
 ; GFX940:       ; %bb.0:
 ; GFX940-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX940-NEXT:    v_add_u32_e32 v0, 4, v0
 ; GFX940-NEXT:    v_mov_b32_e32 v1, 0x41200000
-; GFX940-NEXT:    scratch_store_dword v0, v1, off offset:4
+; GFX940-NEXT:    scratch_store_dword v0, v1, off
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    s_setpc_b64 s[30:31]
 ;
@@ -921,8 +933,9 @@ define void @private_ptr_foo(ptr addrspace(5) nocapture %arg) {
 ; GFX10-PAL:       ; %bb.0:
 ; GFX10-PAL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-PAL-NEXT:    v_add_nc_u32_e32 v0, 4, v0
 ; GFX10-PAL-NEXT:    v_mov_b32_e32 v1, 0x41200000
-; GFX10-PAL-NEXT:    scratch_store_dword v0, v1, off offset:4
+; GFX10-PAL-NEXT:    scratch_store_dword v0, v1, off
 ; GFX10-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-PAL-NEXT:    s_setpc_b64 s[30:31]
 ;
@@ -930,8 +943,8 @@ define void @private_ptr_foo(ptr addrspace(5) nocapture %arg) {
 ; GFX11-PAL:       ; %bb.0:
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-PAL-NEXT:    v_mov_b32_e32 v1, 0x41200000
-; GFX11-PAL-NEXT:    scratch_store_b32 v0, v1, off offset:4
+; GFX11-PAL-NEXT:    v_dual_mov_b32 v1, 0x41200000 :: v_dual_add_nc_u32 v0, 4, v0
+; GFX11-PAL-NEXT:    scratch_store_b32 v0, v1, off
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-PAL-NEXT:    s_setpc_b64 s[30:31]
 ; GCN-LABEL: private_ptr_foo:
@@ -1683,10 +1696,11 @@ define amdgpu_kernel void @store_load_vindex_small_offset_kernel() {
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX9-NEXT:    v_add_u32_e32 v1, 0x104, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 15
+; GFX9-NEXT:    v_sub_u32_e32 v0, 0x104, v0
 ; GFX9-NEXT:    scratch_store_dword v1, v2, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_sub_u32_e32 v0, 0x104, v0
-; GFX9-NEXT:    scratch_load_dword v0, v0, off offset:124 glc
+; GFX9-NEXT:    v_add_u32_e32 v0, 0x7c, v0
+; GFX9-NEXT:    scratch_load_dword v0, v0, off glc
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_endpgm
 ;
@@ -1700,23 +1714,26 @@ define amdgpu_kernel void @store_load_vindex_small_offset_kernel() {
 ; GFX10-NEXT:    v_mov_b32_e32 v2, 15
 ; GFX10-NEXT:    scratch_load_dword v3, off, off offset:4 glc dlc
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_add_nc_u32_e32 v1, 0x104, v0
-; GFX10-NEXT:    v_sub_nc_u32_e32 v0, 0x104, v0
-; GFX10-NEXT:    scratch_store_dword v1, v2, off
+; GFX10-NEXT:    v_sub_nc_u32_e32 v1, 0x104, v0
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, 0x104, v0
+; GFX10-NEXT:    v_add_nc_u32_e32 v1, 0x7c, v1
+; GFX10-NEXT:    scratch_store_dword v0, v2, off
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-NEXT:    scratch_load_dword v0, v0, off offset:124 glc dlc
+; GFX10-NEXT:    scratch_load_dword v0, v1, off glc dlc
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: store_load_vindex_small_offset_kernel:
 ; GFX11:       ; %bb.0: ; %bb
-; GFX11-NEXT:    v_dual_mov_b32 v1, 15 :: v_dual_lshlrev_b32 v0, 2, v0
+; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX11-NEXT:    scratch_load_b32 v3, off, off offset:4 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_sub_nc_u32_e32 v2, 0x104, v0
-; GFX11-NEXT:    scratch_store_b32 v0, v1, off offset:260 dlc
+; GFX11-NEXT:    v_sub_nc_u32_e32 v1, 0x104, v0
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_add_nc_u32 v1, 0x7c, v1
+; GFX11-NEXT:    scratch_store_b32 v0, v2, off offset:260 dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_load_b32 v0, v2, off offset:124 glc dlc
+; GFX11-NEXT:    scratch_load_b32 v0, v1, off glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_endpgm
 ;
@@ -1735,10 +1752,11 @@ define amdgpu_kernel void @store_load_vindex_small_offset_kernel() {
 ; GFX9-PAL-NEXT:    scratch_load_dword v1, off, vcc_hi offset:4 glc
 ; GFX9-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-PAL-NEXT:    v_add_u32_e32 v1, 0x104, v0
+; GFX9-PAL-NEXT:    v_sub_u32_e32 v0, 0x104, v0
 ; GFX9-PAL-NEXT:    scratch_store_dword v1, v2, off
 ; GFX9-PAL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-PAL-NEXT:    v_sub_u32_e32 v0, 0x104, v0
-; GFX9-PAL-NEXT:    scratch_load_dword v0, v0, off offset:124 glc
+; GFX9-PAL-NEXT:    v_add_u32_e32 v0, 0x7c, v0
+; GFX9-PAL-NEXT:    scratch_load_dword v0, v0, off glc
 ; GFX9-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-PAL-NEXT:    s_endpgm
 ;
@@ -1751,7 +1769,8 @@ define amdgpu_kernel void @store_load_vindex_small_offset_kernel() {
 ; GFX940-NEXT:    scratch_store_dword v0, v1, off offset:260 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_sub_u32_e32 v0, 0x104, v0
-; GFX940-NEXT:    scratch_load_dword v0, v0, off offset:124 sc0 sc1
+; GFX940-NEXT:    v_add_u32_e32 v0, 0x7c, v0
+; GFX940-NEXT:    scratch_load_dword v0, v0, off sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    s_endpgm
 ;
@@ -1771,11 +1790,12 @@ define amdgpu_kernel void @store_load_vindex_small_offset_kernel() {
 ; GFX1010-PAL-NEXT:    s_mov_b32 vcc_lo, 0
 ; GFX1010-PAL-NEXT:    scratch_load_dword v3, off, vcc_lo offset:4 glc dlc
 ; GFX1010-PAL-NEXT:    s_waitcnt vmcnt(0)
-; GFX1010-PAL-NEXT:    v_add_nc_u32_e32 v1, 0x104, v0
-; GFX1010-PAL-NEXT:    v_sub_nc_u32_e32 v0, 0x104, v0
-; GFX1010-PAL-NEXT:    scratch_store_dword v1, v2, off
+; GFX1010-PAL-NEXT:    v_sub_nc_u32_e32 v1, 0x104, v0
+; GFX1010-PAL-NEXT:    v_add_nc_u32_e32 v0, 0x104, v0
+; GFX1010-PAL-NEXT:    v_add_nc_u32_e32 v1, 0x7c, v1
+; GFX1010-PAL-NEXT:    scratch_store_dword v0, v2, off
 ; GFX1010-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1010-PAL-NEXT:    scratch_load_dword v0, v0, off offset:124 glc dlc
+; GFX1010-PAL-NEXT:    scratch_load_dword v0, v1, off glc dlc
 ; GFX1010-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX1010-PAL-NEXT:    s_endpgm
 ;
@@ -1794,23 +1814,26 @@ define amdgpu_kernel void @store_load_vindex_small_offset_kernel() {
 ; GFX1030-PAL-NEXT:    v_mov_b32_e32 v2, 15
 ; GFX1030-PAL-NEXT:    scratch_load_dword v3, off, off offset:4 glc dlc
 ; GFX1030-PAL-NEXT:    s_waitcnt vmcnt(0)
-; GFX1030-PAL-NEXT:    v_add_nc_u32_e32 v1, 0x104, v0
-; GFX1030-PAL-NEXT:    v_sub_nc_u32_e32 v0, 0x104, v0
-; GFX1030-PAL-NEXT:    scratch_store_dword v1, v2, off
+; GFX1030-PAL-NEXT:    v_sub_nc_u32_e32 v1, 0x104, v0
+; GFX1030-PAL-NEXT:    v_add_nc_u32_e32 v0, 0x104, v0
+; GFX1030-PAL-NEXT:    v_add_nc_u32_e32 v1, 0x7c, v1
+; GFX1030-PAL-NEXT:    scratch_store_dword v0, v2, off
 ; GFX1030-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1030-PAL-NEXT:    scratch_load_dword v0, v0, off offset:124 glc dlc
+; GFX1030-PAL-NEXT:    scratch_load_dword v0, v1, off glc dlc
 ; GFX1030-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX1030-PAL-NEXT:    s_endpgm
 ;
 ; GFX11-PAL-LABEL: store_load_vindex_small_offset_kernel:
 ; GFX11-PAL:       ; %bb.0: ; %bb
-; GFX11-PAL-NEXT:    v_dual_mov_b32 v1, 15 :: v_dual_lshlrev_b32 v0, 2, v0
+; GFX11-PAL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX11-PAL-NEXT:    scratch_load_b32 v3, off, off offset:4 glc dlc
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-PAL-NEXT:    v_sub_nc_u32_e32 v2, 0x104, v0
-; GFX11-PAL-NEXT:    scratch_store_b32 v0, v1, off offset:260 dlc
+; GFX11-PAL-NEXT:    v_sub_nc_u32_e32 v1, 0x104, v0
+; GFX11-PAL-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-PAL-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_add_nc_u32 v1, 0x7c, v1
+; GFX11-PAL-NEXT:    scratch_store_b32 v0, v2, off offset:260 dlc
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-PAL-NEXT:    scratch_load_b32 v0, v2, off offset:124 glc dlc
+; GFX11-PAL-NEXT:    scratch_load_b32 v0, v1, off glc dlc
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-PAL-NEXT:    s_endpgm
 bb:
@@ -1869,11 +1892,12 @@ define void @store_load_vindex_small_offset_foo(i32 %idx) {
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_and_b32 v1, 15, v0
-; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX11-NEXT:    s_add_i32 vcc_lo, s32, 0x100
 ; GFX11-NEXT:    scratch_load_b32 v3, off, s32 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    v_lshl_add_u32 v0, v0, 2, vcc_lo
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v1, 2, v1
-; GFX11-NEXT:    scratch_store_b32 v0, v2, s32 offset:256 dlc
+; GFX11-NEXT:    scratch_store_b32 v0, v2, off dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    scratch_load_b32 v0, v1, s32 offset:256 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
@@ -1901,10 +1925,12 @@ define void @store_load_vindex_small_offset_foo(i32 %idx) {
 ; GFX940-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX940-NEXT:    scratch_load_dword v1, off, s32 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-NEXT:    v_lshlrev_b32_e32 v1, 2, v0
+; GFX940-NEXT:    s_add_i32 vcc_hi, s32, 0x100
+; GFX940-NEXT:    v_mov_b32_e32 v1, vcc_hi
+; GFX940-NEXT:    v_lshl_add_u32 v1, v0, 2, v1
 ; GFX940-NEXT:    v_mov_b32_e32 v2, 15
 ; GFX940-NEXT:    v_and_b32_e32 v0, 15, v0
-; GFX940-NEXT:    scratch_store_dword v1, v2, s32 offset:256 sc0 sc1
+; GFX940-NEXT:    scratch_store_dword v1, v2, off sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX940-NEXT:    scratch_load_dword v0, v0, s32 offset:256 sc0 sc1
@@ -1934,11 +1960,12 @@ define void @store_load_vindex_small_offset_foo(i32 %idx) {
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-PAL-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_and_b32 v1, 15, v0
-; GFX11-PAL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX11-PAL-NEXT:    s_add_i32 vcc_lo, s32, 0x100
 ; GFX11-PAL-NEXT:    scratch_load_b32 v3, off, s32 glc dlc
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-PAL-NEXT:    v_lshl_add_u32 v0, v0, 2, vcc_lo
 ; GFX11-PAL-NEXT:    v_lshlrev_b32_e32 v1, 2, v1
-; GFX11-PAL-NEXT:    scratch_store_b32 v0, v2, s32 offset:256 dlc
+; GFX11-PAL-NEXT:    scratch_store_b32 v0, v2, off dlc
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-PAL-NEXT:    scratch_load_b32 v0, v1, s32 offset:256 glc dlc
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0)
@@ -2739,10 +2766,11 @@ define amdgpu_kernel void @store_load_vindex_large_offset_kernel() {
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX9-NEXT:    v_add_u32_e32 v1, 0x4004, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 15
+; GFX9-NEXT:    v_sub_u32_e32 v0, 0x4004, v0
 ; GFX9-NEXT:    scratch_store_dword v1, v2, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_sub_u32_e32 v0, 0x4004, v0
-; GFX9-NEXT:    scratch_load_dword v0, v0, off offset:124 glc
+; GFX9-NEXT:    v_add_u32_e32 v0, 0x7c, v0
+; GFX9-NEXT:    scratch_load_dword v0, v0, off glc
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_endpgm
 ;
@@ -2756,24 +2784,27 @@ define amdgpu_kernel void @store_load_vindex_large_offset_kernel() {
 ; GFX10-NEXT:    v_mov_b32_e32 v2, 15
 ; GFX10-NEXT:    scratch_load_dword v3, off, off offset:4 glc dlc
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_add_nc_u32_e32 v1, 0x4004, v0
-; GFX10-NEXT:    v_sub_nc_u32_e32 v0, 0x4004, v0
-; GFX10-NEXT:    scratch_store_dword v1, v2, off
+; GFX10-NEXT:    v_sub_nc_u32_e32 v1, 0x4004, v0
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, 0x4004, v0
+; GFX10-NEXT:    v_add_nc_u32_e32 v1, 0x7c, v1
+; GFX10-NEXT:    scratch_store_dword v0, v2, off
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-NEXT:    scratch_load_dword v0, v0, off offset:124 glc dlc
+; GFX10-NEXT:    scratch_load_dword v0, v1, off glc dlc
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: store_load_vindex_large_offset_kernel:
 ; GFX11:       ; %bb.0: ; %bb
-; GFX11-NEXT:    v_dual_mov_b32 v1, 15 :: v_dual_lshlrev_b32 v0, 2, v0
+; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX11-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX11-NEXT:    scratch_load_b32 v3, off, off offset:4 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_sub_nc_u32_e32 v2, 0x4004, v0
-; GFX11-NEXT:    scratch_store_b32 v0, v1, vcc_lo dlc
+; GFX11-NEXT:    v_sub_nc_u32_e32 v1, 0x4004, v0
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_add_nc_u32 v1, 0x7c, v1
+; GFX11-NEXT:    scratch_store_b32 v0, v2, vcc_lo dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_load_b32 v0, v2, off offset:124 glc dlc
+; GFX11-NEXT:    scratch_load_b32 v0, v1, off glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_endpgm
 ;
@@ -2792,10 +2823,11 @@ define amdgpu_kernel void @store_load_vindex_large_offset_kernel() {
 ; GFX9-PAL-NEXT:    scratch_load_dword v1, off, vcc_hi offset:4 glc
 ; GFX9-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-PAL-NEXT:    v_add_u32_e32 v1, 0x4004, v0
+; GFX9-PAL-NEXT:    v_sub_u32_e32 v0, 0x4004, v0
 ; GFX9-PAL-NEXT:    scratch_store_dword v1, v2, off
 ; GFX9-PAL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-PAL-NEXT:    v_sub_u32_e32 v0, 0x4004, v0
-; GFX9-PAL-NEXT:    scratch_load_dword v0, v0, off offset:124 glc
+; GFX9-PAL-NEXT:    v_add_u32_e32 v0, 0x7c, v0
+; GFX9-PAL-NEXT:    scratch_load_dword v0, v0, off glc
 ; GFX9-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-PAL-NEXT:    s_endpgm
 ;
@@ -2809,7 +2841,8 @@ define amdgpu_kernel void @store_load_vindex_large_offset_kernel() {
 ; GFX940-NEXT:    scratch_store_dword v0, v1, vcc_hi sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_sub_u32_e32 v0, 0x4004, v0
-; GFX940-NEXT:    scratch_load_dword v0, v0, off offset:124 sc0 sc1
+; GFX940-NEXT:    v_add_u32_e32 v0, 0x7c, v0
+; GFX940-NEXT:    scratch_load_dword v0, v0, off sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    s_endpgm
 ;
@@ -2829,11 +2862,12 @@ define amdgpu_kernel void @store_load_vindex_large_offset_kernel() {
 ; GFX1010-PAL-NEXT:    s_mov_b32 vcc_lo, 0
 ; GFX1010-PAL-NEXT:    scratch_load_dword v3, off, vcc_lo offset:4 glc dlc
 ; GFX1010-PAL-NEXT:    s_waitcnt vmcnt(0)
-; GFX1010-PAL-NEXT:    v_add_nc_u32_e32 v1, 0x4004, v0
-; GFX1010-PAL-NEXT:    v_sub_nc_u32_e32 v0, 0x4004, v0
-; GFX1010-PAL-NEXT:    scratch_store_dword v1, v2, off
+; GFX1010-PAL-NEXT:    v_sub_nc_u32_e32 v1, 0x4004, v0
+; GFX1010-PAL-NEXT:    v_add_nc_u32_e32 v0, 0x4004, v0
+; GFX1010-PAL-NEXT:    v_add_nc_u32_e32 v1, 0x7c, v1
+; GFX1010-PAL-NEXT:    scratch_store_dword v0, v2, off
 ; GFX1010-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1010-PAL-NEXT:    scratch_load_dword v0, v0, off offset:124 glc dlc
+; GFX1010-PAL-NEXT:    scratch_load_dword v0, v1, off glc dlc
 ; GFX1010-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX1010-PAL-NEXT:    s_endpgm
 ;
@@ -2852,24 +2886,27 @@ define amdgpu_kernel void @store_load_vindex_large_offset_kernel() {
 ; GFX1030-PAL-NEXT:    v_mov_b32_e32 v2, 15
 ; GFX1030-PAL-NEXT:    scratch_load_dword v3, off, off offset:4 glc dlc
 ; GFX1030-PAL-NEXT:    s_waitcnt vmcnt(0)
-; GFX1030-PAL-NEXT:    v_add_nc_u32_e32 v1, 0x4004, v0
-; GFX1030-PAL-NEXT:    v_sub_nc_u32_e32 v0, 0x4004, v0
-; GFX1030-PAL-NEXT:    scratch_store_dword v1, v2, off
+; GFX1030-PAL-NEXT:    v_sub_nc_u32_e32 v1, 0x4004, v0
+; GFX1030-PAL-NEXT:    v_add_nc_u32_e32 v0, 0x4004, v0
+; GFX1030-PAL-NEXT:    v_add_nc_u32_e32 v1, 0x7c, v1
+; GFX1030-PAL-NEXT:    scratch_store_dword v0, v2, off
 ; GFX1030-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1030-PAL-NEXT:    scratch_load_dword v0, v0, off offset:124 glc dlc
+; GFX1030-PAL-NEXT:    scratch_load_dword v0, v1, off glc dlc
 ; GFX1030-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX1030-PAL-NEXT:    s_endpgm
 ;
 ; GFX11-PAL-LABEL: store_load_vindex_large_offset_kernel:
 ; GFX11-PAL:       ; %bb.0: ; %bb
-; GFX11-PAL-NEXT:    v_dual_mov_b32 v1, 15 :: v_dual_lshlrev_b32 v0, 2, v0
+; GFX11-PAL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX11-PAL-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX11-PAL-NEXT:    scratch_load_b32 v3, off, off offset:4 glc dlc
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-PAL-NEXT:    v_sub_nc_u32_e32 v2, 0x4004, v0
-; GFX11-PAL-NEXT:    scratch_store_b32 v0, v1, vcc_lo dlc
+; GFX11-PAL-NEXT:    v_sub_nc_u32_e32 v1, 0x4004, v0
+; GFX11-PAL-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-PAL-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_add_nc_u32 v1, 0x7c, v1
+; GFX11-PAL-NEXT:    scratch_store_b32 v0, v2, vcc_lo dlc
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-PAL-NEXT:    scratch_load_b32 v0, v2, off offset:124 glc dlc
+; GFX11-PAL-NEXT:    scratch_load_b32 v0, v1, off glc dlc
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-PAL-NEXT:    s_endpgm
 bb:
@@ -2928,14 +2965,14 @@ define void @store_load_vindex_large_offset_foo(i32 %idx) {
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_and_b32 v1, 15, v0
-; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX11-NEXT:    s_add_i32 s0, s32, 0x4004
 ; GFX11-NEXT:    s_add_i32 vcc_lo, s32, 0x4004
+; GFX11-NEXT:    v_lshl_add_u32 v0, v0, 2, s0
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2)
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v1, 2, v1
 ; GFX11-NEXT:    scratch_load_b32 v3, off, s32 offset:4 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    scratch_store_b32 v0, v2, s0 dlc
+; GFX11-NEXT:    scratch_store_b32 v0, v2, off dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    scratch_load_b32 v0, v1, vcc_lo glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
@@ -2963,11 +3000,12 @@ define void @store_load_vindex_large_offset_foo(i32 %idx) {
 ; GFX940-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX940-NEXT:    scratch_load_dword v1, off, s32 offset:4 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-NEXT:    v_lshlrev_b32_e32 v1, 2, v0
-; GFX940-NEXT:    v_mov_b32_e32 v2, 15
 ; GFX940-NEXT:    s_add_i32 vcc_lo, s32, 0x4004
+; GFX940-NEXT:    v_mov_b32_e32 v1, vcc_lo
+; GFX940-NEXT:    v_lshl_add_u32 v1, v0, 2, v1
+; GFX940-NEXT:    v_mov_b32_e32 v2, 15
 ; GFX940-NEXT:    v_and_b32_e32 v0, 15, v0
-; GFX940-NEXT:    scratch_store_dword v1, v2, vcc_lo sc0 sc1
+; GFX940-NEXT:    scratch_store_dword v1, v2, off sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX940-NEXT:    s_add_i32 vcc_hi, s32, 0x4004
@@ -2998,14 +3036,14 @@ define void @store_load_vindex_large_offset_foo(i32 %idx) {
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-PAL-NEXT:    v_dual_mov_b32 v2, 15 :: v_dual_and_b32 v1, 15, v0
-; GFX11-PAL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX11-PAL-NEXT:    s_add_i32 s0, s32, 0x4004
 ; GFX11-PAL-NEXT:    s_add_i32 vcc_lo, s32, 0x4004
+; GFX11-PAL-NEXT:    v_lshl_add_u32 v0, v0, 2, s0
 ; GFX11-PAL-NEXT:    s_delay_alu instid0(VALU_DEP_2)
 ; GFX11-PAL-NEXT:    v_lshlrev_b32_e32 v1, 2, v1
 ; GFX11-PAL-NEXT:    scratch_load_b32 v3, off, s32 offset:4 glc dlc
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-PAL-NEXT:    scratch_store_b32 v0, v2, s0 dlc
+; GFX11-PAL-NEXT:    scratch_store_b32 v0, v2, off dlc
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-PAL-NEXT:    scratch_load_b32 v0, v1, vcc_lo glc dlc
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0)
@@ -3077,13 +3115,15 @@ define amdgpu_kernel void @store_load_large_imm_offset_kernel() {
 ;
 ; GFX11-LABEL: store_load_large_imm_offset_kernel:
 ; GFX11:       ; %bb.0: ; %bb
-; GFX11-NEXT:    v_dual_mov_b32 v0, 13 :: v_dual_mov_b32 v1, 0x3000
-; GFX11-NEXT:    v_mov_b32_e32 v2, 15
+; GFX11-NEXT:    v_dual_mov_b32 v0, 13 :: v_dual_mov_b32 v1, 15
+; GFX11-NEXT:    s_movk_i32 s0, 0x3000
+; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
+; GFX11-NEXT:    s_add_i32 s0, s0, 4
 ; GFX11-NEXT:    scratch_store_b32 off, v0, off offset:4 dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_store_b32 v1, v2, off offset:3716 dlc
+; GFX11-NEXT:    scratch_store_b32 off, v1, s0 offset:3712 dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_load_b32 v0, v1, off offset:3716 glc dlc
+; GFX11-NEXT:    scratch_load_b32 v0, off, s0 offset:3712 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_endpgm
 ;
@@ -3112,13 +3152,14 @@ define amdgpu_kernel void @store_load_large_imm_offset_kernel() {
 ; GFX940-LABEL: store_load_large_imm_offset_kernel:
 ; GFX940:       ; %bb.0: ; %bb
 ; GFX940-NEXT:    v_mov_b32_e32 v0, 13
+; GFX940-NEXT:    s_movk_i32 s0, 0x3000
 ; GFX940-NEXT:    scratch_store_dword off, v0, off offset:4 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-NEXT:    v_mov_b32_e32 v0, 0x3000
-; GFX940-NEXT:    v_mov_b32_e32 v1, 15
-; GFX940-NEXT:    scratch_store_dword v0, v1, off offset:3716 sc0 sc1
+; GFX940-NEXT:    s_add_i32 s0, s0, 4
+; GFX940-NEXT:    v_mov_b32_e32 v0, 15
+; GFX940-NEXT:    scratch_store_dword off, v0, s0 offset:3712 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-NEXT:    scratch_load_dword v0, v0, off offset:3716 sc0 sc1
+; GFX940-NEXT:    scratch_load_dword v0, off, s0 offset:3712 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    s_endpgm
 ;
@@ -3171,13 +3212,15 @@ define amdgpu_kernel void @store_load_large_imm_offset_kernel() {
 ;
 ; GFX11-PAL-LABEL: store_load_large_imm_offset_kernel:
 ; GFX11-PAL:       ; %bb.0: ; %bb
-; GFX11-PAL-NEXT:    v_dual_mov_b32 v0, 13 :: v_dual_mov_b32 v1, 0x3000
-; GFX11-PAL-NEXT:    v_mov_b32_e32 v2, 15
+; GFX11-PAL-NEXT:    v_dual_mov_b32 v0, 13 :: v_dual_mov_b32 v1, 15
+; GFX11-PAL-NEXT:    s_movk_i32 s0, 0x3000
+; GFX11-PAL-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
+; GFX11-PAL-NEXT:    s_add_i32 s0, s0, 4
 ; GFX11-PAL-NEXT:    scratch_store_b32 off, v0, off offset:4 dlc
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-PAL-NEXT:    scratch_store_b32 v1, v2, off offset:3716 dlc
+; GFX11-PAL-NEXT:    scratch_store_b32 off, v1, s0 offset:3712 dlc
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-PAL-NEXT:    scratch_load_b32 v0, v1, off offset:3716 glc dlc
+; GFX11-PAL-NEXT:    scratch_load_b32 v0, off, s0 offset:3712 glc dlc
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-PAL-NEXT:    s_endpgm
 bb:
@@ -3229,13 +3272,16 @@ define void @store_load_large_imm_offset_foo() {
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    v_dual_mov_b32 v0, 13 :: v_dual_mov_b32 v1, 0x3000
-; GFX11-NEXT:    v_mov_b32_e32 v2, 15
+; GFX11-NEXT:    v_dual_mov_b32 v0, 13 :: v_dual_mov_b32 v1, 15
+; GFX11-NEXT:    s_movk_i32 s0, 0x3000
+; GFX11-NEXT:    s_add_i32 vcc_lo, s32, 4
+; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
+; GFX11-NEXT:    s_add_i32 s0, s0, vcc_lo
 ; GFX11-NEXT:    scratch_store_b32 off, v0, s32 offset:4 dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_store_b32 v1, v2, s32 offset:3716 dlc
+; GFX11-NEXT:    scratch_store_b32 off, v1, s0 offset:3712 dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_load_b32 v0, v1, s32 offset:3716 glc dlc
+; GFX11-NEXT:    scratch_load_b32 v0, off, s0 offset:3712 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_setpc_b64 s[30:31]
 ;
@@ -3259,13 +3305,15 @@ define void @store_load_large_imm_offset_foo() {
 ; GFX940:       ; %bb.0: ; %bb
 ; GFX940-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX940-NEXT:    v_mov_b32_e32 v0, 13
+; GFX940-NEXT:    s_movk_i32 s0, 0x3000
+; GFX940-NEXT:    s_add_i32 vcc_hi, s32, 4
 ; GFX940-NEXT:    scratch_store_dword off, v0, s32 offset:4 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-NEXT:    v_mov_b32_e32 v0, 0x3000
-; GFX940-NEXT:    v_mov_b32_e32 v1, 15
-; GFX940-NEXT:    scratch_store_dword v0, v1, s32 offset:3716 sc0 sc1
+; GFX940-NEXT:    s_add_i32 s0, s0, vcc_hi
+; GFX940-NEXT:    v_mov_b32_e32 v0, 15
+; GFX940-NEXT:    scratch_store_dword off, v0, s0 offset:3712 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-NEXT:    scratch_load_dword v0, v0, s32 offset:3716 sc0 sc1
+; GFX940-NEXT:    scratch_load_dword v0, off, s0 offset:3712 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    s_setpc_b64 s[30:31]
 ;
@@ -3290,13 +3338,16 @@ define void @store_load_large_imm_offset_foo() {
 ; GFX11-PAL:       ; %bb.0: ; %bb
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-PAL-NEXT:    v_dual_mov_b32 v0, 13 :: v_dual_mov_b32 v1, 0x3000
-; GFX11-PAL-NEXT:    v_mov_b32_e32 v2, 15
+; GFX11-PAL-NEXT:    v_dual_mov_b32 v0, 13 :: v_dual_mov_b32 v1, 15
+; GFX11-PAL-NEXT:    s_movk_i32 s0, 0x3000
+; GFX11-PAL-NEXT:    s_add_i32 vcc_lo, s32, 4
+; GFX11-PAL-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
+; GFX11-PAL-NEXT:    s_add_i32 s0, s0, vcc_lo
 ; GFX11-PAL-NEXT:    scratch_store_b32 off, v0, s32 offset:4 dlc
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-PAL-NEXT:    scratch_store_b32 v1, v2, s32 offset:3716 dlc
+; GFX11-PAL-NEXT:    scratch_store_b32 off, v1, s0 offset:3712 dlc
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-PAL-NEXT:    scratch_load_b32 v0, v1, s32 offset:3716 glc dlc
+; GFX11-PAL-NEXT:    scratch_load_b32 v0, off, s0 offset:3712 glc dlc
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-PAL-NEXT:    s_setpc_b64 s[30:31]
 ; GCN-LABEL: store_load_large_imm_offset_foo:
@@ -3327,16 +3378,17 @@ define amdgpu_kernel void @store_load_vidx_sidx_offset(i32 %sidx) {
 ; GFX9-LABEL: store_load_vidx_sidx_offset:
 ; GFX9:       ; %bb.0: ; %bb
 ; GFX9-NEXT:    s_load_dword s0, s[0:1], 0x24
-; GFX9-NEXT:    s_add_u32 flat_scratch_lo, s2, s5
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 4
+; GFX9-NEXT:    s_add_u32 flat_scratch_lo, s2, s5
 ; GFX9-NEXT:    s_addc_u32 flat_scratch_hi, s3, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_add_u32_e32 v0, s0, v0
 ; GFX9-NEXT:    v_lshl_add_u32 v0, v0, 2, v1
+; GFX9-NEXT:    v_add_u32_e32 v0, 0x400, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 15
-; GFX9-NEXT:    scratch_store_dword v0, v1, off offset:1024
+; GFX9-NEXT:    scratch_store_dword v0, v1, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    scratch_load_dword v0, v0, off offset:1024 glc
+; GFX9-NEXT:    scratch_load_dword v0, v0, off glc
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_endpgm
 ;
@@ -3347,25 +3399,29 @@ define amdgpu_kernel void @store_load_vidx_sidx_offset(i32 %sidx) {
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s2
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s3
 ; GFX10-NEXT:    s_load_dword s0, s[0:1], 0x24
-; GFX10-NEXT:    v_mov_b32_e32 v1, 15
+; GFX10-NEXT:    v_mov_b32_e32 v1, 4
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10-NEXT:    v_add_nc_u32_e32 v0, s0, v0
-; GFX10-NEXT:    v_lshl_add_u32 v0, v0, 2, 4
-; GFX10-NEXT:    scratch_store_dword v0, v1, off offset:1024
+; GFX10-NEXT:    v_add_lshl_u32 v0, s0, v0, 2
+; GFX10-NEXT:    v_add3_u32 v0, v1, v0, 0x400
+; GFX10-NEXT:    v_mov_b32_e32 v1, 15
+; GFX10-NEXT:    scratch_store_dword v0, v1, off
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-NEXT:    scratch_load_dword v0, v0, off offset:1024 glc dlc
+; GFX10-NEXT:    scratch_load_dword v0, v0, off glc dlc
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: store_load_vidx_sidx_offset:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    s_load_b32 s0, s[0:1], 0x24
-; GFX11-NEXT:    v_mov_b32_e32 v1, 15
+; GFX11-NEXT:    v_mov_b32_e32 v1, 4
 ; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-NEXT:    v_add_lshl_u32 v0, s0, v0, 2
-; GFX11-NEXT:    scratch_store_b32 v0, v1, off offset:1028 dlc
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_add3_u32 v0, v1, v0, 0x400
+; GFX11-NEXT:    v_mov_b32_e32 v1, 15
+; GFX11-NEXT:    scratch_store_b32 v0, v1, off dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_load_b32 v0, v0, off offset:1028 glc dlc
+; GFX11-NEXT:    scratch_load_b32 v0, v0, off glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_endpgm
 ;
@@ -3380,24 +3436,28 @@ define amdgpu_kernel void @store_load_vidx_sidx_offset(i32 %sidx) {
 ; GFX9-PAL-NEXT:    s_and_b32 s5, s5, 0xffff
 ; GFX9-PAL-NEXT:    s_add_u32 flat_scratch_lo, s4, s3
 ; GFX9-PAL-NEXT:    v_add_u32_e32 v0, s0, v0
-; GFX9-PAL-NEXT:    s_addc_u32 flat_scratch_hi, s5, 0
 ; GFX9-PAL-NEXT:    v_lshl_add_u32 v0, v0, 2, v1
+; GFX9-PAL-NEXT:    s_addc_u32 flat_scratch_hi, s5, 0
+; GFX9-PAL-NEXT:    v_add_u32_e32 v0, 0x400, v0
 ; GFX9-PAL-NEXT:    v_mov_b32_e32 v1, 15
-; GFX9-PAL-NEXT:    scratch_store_dword v0, v1, off offset:1024
+; GFX9-PAL-NEXT:    scratch_store_dword v0, v1, off
 ; GFX9-PAL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-PAL-NEXT:    scratch_load_dword v0, v0, off offset:1024 glc
+; GFX9-PAL-NEXT:    scratch_load_dword v0, v0, off glc
 ; GFX9-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-PAL-NEXT:    s_endpgm
 ;
 ; GFX940-LABEL: store_load_vidx_sidx_offset:
 ; GFX940:       ; %bb.0: ; %bb
 ; GFX940-NEXT:    s_load_dword s0, s[0:1], 0x24
-; GFX940-NEXT:    v_mov_b32_e32 v1, 15
+; GFX940-NEXT:    v_mov_b32_e32 v1, 4
 ; GFX940-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX940-NEXT:    v_add_lshl_u32 v0, s0, v0, 2
-; GFX940-NEXT:    scratch_store_dword v0, v1, off offset:1028 sc0 sc1
+; GFX940-NEXT:    v_add_u32_e32 v0, s0, v0
+; GFX940-NEXT:    v_lshl_add_u32 v0, v0, 2, v1
+; GFX940-NEXT:    v_add_u32_e32 v0, 0x400, v0
+; GFX940-NEXT:    v_mov_b32_e32 v1, 15
+; GFX940-NEXT:    scratch_store_dword v0, v1, off sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-NEXT:    scratch_load_dword v0, v0, off offset:1028 sc0 sc1
+; GFX940-NEXT:    scratch_load_dword v0, v0, off sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    s_endpgm
 ;
@@ -3413,25 +3473,29 @@ define amdgpu_kernel void @store_load_vidx_sidx_offset(i32 %sidx) {
 ; GFX10-PAL-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s4
 ; GFX10-PAL-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s5
 ; GFX10-PAL-NEXT:    s_load_dword s0, s[0:1], 0x0
-; GFX10-PAL-NEXT:    v_mov_b32_e32 v1, 15
+; GFX10-PAL-NEXT:    v_mov_b32_e32 v1, 4
 ; GFX10-PAL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10-PAL-NEXT:    v_add_nc_u32_e32 v0, s0, v0
-; GFX10-PAL-NEXT:    v_lshl_add_u32 v0, v0, 2, 4
-; GFX10-PAL-NEXT:    scratch_store_dword v0, v1, off offset:1024
+; GFX10-PAL-NEXT:    v_add_lshl_u32 v0, s0, v0, 2
+; GFX10-PAL-NEXT:    v_add3_u32 v0, v1, v0, 0x400
+; GFX10-PAL-NEXT:    v_mov_b32_e32 v1, 15
+; GFX10-PAL-NEXT:    scratch_store_dword v0, v1, off
 ; GFX10-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-PAL-NEXT:    scratch_load_dword v0, v0, off offset:1024 glc dlc
+; GFX10-PAL-NEXT:    scratch_load_dword v0, v0, off glc dlc
 ; GFX10-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-PAL-NEXT:    s_endpgm
 ;
 ; GFX11-PAL-LABEL: store_load_vidx_sidx_offset:
 ; GFX11-PAL:       ; %bb.0: ; %bb
 ; GFX11-PAL-NEXT:    s_load_b32 s0, s[0:1], 0x0
-; GFX11-PAL-NEXT:    v_mov_b32_e32 v1, 15
+; GFX11-PAL-NEXT:    v_mov_b32_e32 v1, 4
 ; GFX11-PAL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-PAL-NEXT:    v_add_lshl_u32 v0, s0, v0, 2
-; GFX11-PAL-NEXT:    scratch_store_b32 v0, v1, off offset:1028 dlc
+; GFX11-PAL-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-PAL-NEXT:    v_add3_u32 v0, v1, v0, 0x400
+; GFX11-PAL-NEXT:    v_mov_b32_e32 v1, 15
+; GFX11-PAL-NEXT:    scratch_store_b32 v0, v1, off dlc
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-PAL-NEXT:    scratch_load_b32 v0, v0, off offset:1028 glc dlc
+; GFX11-PAL-NEXT:    scratch_load_b32 v0, v0, off glc dlc
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-PAL-NEXT:    s_endpgm
 ; GCN-LABEL: store_load_vidx_sidx_offset:
@@ -3874,10 +3938,11 @@ define void @store_load_i32_negative_unaligned(ptr addrspace(5) nocapture %arg)
 ; GFX10:       ; %bb.0: ; %bb
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, -1, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v1, 1
-; GFX10-NEXT:    scratch_store_byte v0, v1, off offset:-1
+; GFX10-NEXT:    scratch_store_byte v0, v1, off
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-NEXT:    scratch_load_ubyte v0, v0, off offset:-1 glc dlc
+; GFX10-NEXT:    scratch_load_ubyte v0, v0, off glc dlc
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
 ;
@@ -3885,10 +3950,10 @@ define void @store_load_i32_negative_unaligned(ptr addrspace(5) nocapture %arg)
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    v_mov_b32_e32 v1, 1
-; GFX11-NEXT:    scratch_store_b8 v0, v1, off offset:-1 dlc
+; GFX11-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_add_nc_u32 v0, -1, v0
+; GFX11-NEXT:    scratch_store_b8 v0, v1, off dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_load_u8 v0, v0, off offset:-1 glc dlc
+; GFX11-NEXT:    scratch_load_u8 v0, v0, off glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_setpc_b64 s[30:31]
 ;
@@ -3914,37 +3979,26 @@ define void @store_load_i32_negative_unaligned(ptr addrspace(5) nocapture %arg)
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    s_setpc_b64 s[30:31]
 ;
-; GFX1010-PAL-LABEL: store_load_i32_negative_unaligned:
-; GFX1010-PAL:       ; %bb.0: ; %bb
-; GFX1010-PAL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX1010-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1010-PAL-NEXT:    v_add_nc_u32_e32 v0, -1, v0
-; GFX1010-PAL-NEXT:    v_mov_b32_e32 v1, 1
-; GFX1010-PAL-NEXT:    scratch_store_byte v0, v1, off
-; GFX1010-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1010-PAL-NEXT:    scratch_load_ubyte v0, v0, off glc dlc
-; GFX1010-PAL-NEXT:    s_waitcnt vmcnt(0)
-; GFX1010-PAL-NEXT:    s_setpc_b64 s[30:31]
-;
-; GFX1030-PAL-LABEL: store_load_i32_negative_unaligned:
-; GFX1030-PAL:       ; %bb.0: ; %bb
-; GFX1030-PAL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX1030-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1030-PAL-NEXT:    v_mov_b32_e32 v1, 1
-; GFX1030-PAL-NEXT:    scratch_store_byte v0, v1, off offset:-1
-; GFX1030-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1030-PAL-NEXT:    scratch_load_ubyte v0, v0, off offset:-1 glc dlc
-; GFX1030-PAL-NEXT:    s_waitcnt vmcnt(0)
-; GFX1030-PAL-NEXT:    s_setpc_b64 s[30:31]
+; GFX10-PAL-LABEL: store_load_i32_negative_unaligned:
+; GFX10-PAL:       ; %bb.0: ; %bb
+; GFX10-PAL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-PAL-NEXT:    v_add_nc_u32_e32 v0, -1, v0
+; GFX10-PAL-NEXT:    v_mov_b32_e32 v1, 1
+; GFX10-PAL-NEXT:    scratch_store_byte v0, v1, off
+; GFX10-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-PAL-NEXT:    scratch_load_ubyte v0, v0, off glc dlc
+; GFX10-PAL-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-PAL-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX11-PAL-LABEL: store_load_i32_negative_unaligned:
 ; GFX11-PAL:       ; %bb.0: ; %bb
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-PAL-NEXT:    v_mov_b32_e32 v1, 1
-; GFX11-PAL-NEXT:    scratch_store_b8 v0, v1, off offset:-1 dlc
+; GFX11-PAL-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_add_nc_u32 v0, -1, v0
+; GFX11-PAL-NEXT:    scratch_store_b8 v0, v1, off dlc
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-PAL-NEXT:    scratch_load_u8 v0, v0, off offset:-1 glc dlc
+; GFX11-PAL-NEXT:    scratch_load_u8 v0, v0, off glc dlc
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-PAL-NEXT:    s_setpc_b64 s[30:31]
 bb:
@@ -3970,11 +4024,11 @@ define void @store_load_i32_large_negative_unaligned(ptr addrspace(5) nocapture
 ; GFX10:       ; %bb.0: ; %bb
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-NEXT:    v_add_nc_u32_e32 v0, 0xfffff000, v0
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, 0xffffef7f, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v1, 1
-; GFX10-NEXT:    scratch_store_byte v0, v1, off offset:-129
+; GFX10-NEXT:    scratch_store_byte v0, v1, off
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-NEXT:    scratch_load_ubyte v0, v0, off offset:-129 glc dlc
+; GFX10-NEXT:    scratch_load_ubyte v0, v0, off glc dlc
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
 ;
@@ -3982,10 +4036,10 @@ define void @store_load_i32_large_negative_unaligned(ptr addrspace(5) nocapture
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_add_nc_u32 v0, 0xfffff000, v0
-; GFX11-NEXT:    scratch_store_b8 v0, v1, off offset:-129 dlc
+; GFX11-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_add_nc_u32 v0, 0xffffef7f, v0
+; GFX11-NEXT:    scratch_store_b8 v0, v1, off dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    scratch_load_u8 v0, v0, off offset:-129 glc dlc
+; GFX11-NEXT:    scratch_load_u8 v0, v0, off glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_setpc_b64 s[30:31]
 ;
@@ -4003,46 +4057,34 @@ define void @store_load_i32_large_negative_unaligned(ptr addrspace(5) nocapture
 ; GFX940-LABEL: store_load_i32_large_negative_unaligned:
 ; GFX940:       ; %bb.0: ; %bb
 ; GFX940-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX940-NEXT:    s_movk_i32 s0, 0xef7f
+; GFX940-NEXT:    v_add_u32_e32 v0, 0xffffef7f, v0
 ; GFX940-NEXT:    v_mov_b32_e32 v1, 1
-; GFX940-NEXT:    scratch_store_byte v0, v1, s0 sc0 sc1
+; GFX940-NEXT:    scratch_store_byte v0, v1, off sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
-; GFX940-NEXT:    scratch_load_ubyte v0, v0, s0 sc0 sc1
+; GFX940-NEXT:    scratch_load_ubyte v0, v0, off sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    s_setpc_b64 s[30:31]
 ;
-; GFX1010-PAL-LABEL: store_load_i32_large_negative_unaligned:
-; GFX1010-PAL:       ; %bb.0: ; %bb
-; GFX1010-PAL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX1010-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1010-PAL-NEXT:    v_add_nc_u32_e32 v0, 0xffffefff, v0
-; GFX1010-PAL-NEXT:    v_mov_b32_e32 v1, 1
-; GFX1010-PAL-NEXT:    scratch_store_byte v0, v1, off offset:-128
-; GFX1010-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1010-PAL-NEXT:    scratch_load_ubyte v0, v0, off offset:-128 glc dlc
-; GFX1010-PAL-NEXT:    s_waitcnt vmcnt(0)
-; GFX1010-PAL-NEXT:    s_setpc_b64 s[30:31]
-;
-; GFX1030-PAL-LABEL: store_load_i32_large_negative_unaligned:
-; GFX1030-PAL:       ; %bb.0: ; %bb
-; GFX1030-PAL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX1030-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1030-PAL-NEXT:    v_add_nc_u32_e32 v0, 0xfffff000, v0
-; GFX1030-PAL-NEXT:    v_mov_b32_e32 v1, 1
-; GFX1030-PAL-NEXT:    scratch_store_byte v0, v1, off offset:-129
-; GFX1030-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1030-PAL-NEXT:    scratch_load_ubyte v0, v0, off offset:-129 glc dlc
-; GFX1030-PAL-NEXT:    s_waitcnt vmcnt(0)
-; GFX1030-PAL-NEXT:    s_setpc_b64 s[30:31]
+; GFX10-PAL-LABEL: store_load_i32_large_negative_unaligned:
+; GFX10-PAL:       ; %bb.0: ; %bb
+; GFX10-PAL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-PAL-NEXT:    v_add_nc_u32_e32 v0, 0xffffef7f, v0
+; GFX10-PAL-NEXT:    v_mov_b32_e32 v1, 1
+; GFX10-PAL-NEXT:    scratch_store_byte v0, v1, off
+; GFX10-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-PAL-NEXT:    scratch_load_ubyte v0, v0, off glc dlc
+; GFX10-PAL-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-PAL-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX11-PAL-LABEL: store_load_i32_large_negative_unaligned:
 ; GFX11-PAL:       ; %bb.0: ; %bb
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-PAL-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_add_nc_u32 v0, 0xfffff000, v0
-; GFX11-PAL-NEXT:    scratch_store_b8 v0, v1, off offset:-129 dlc
+; GFX11-PAL-NEXT:    v_dual_mov_b32 v1, 1 :: v_dual_add_nc_u32 v0, 0xffffef7f, v0
+; GFX11-PAL-NEXT:    scratch_store_b8 v0, v1, off dlc
 ; GFX11-PAL-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-PAL-NEXT:    scratch_load_u8 v0, v0, off offset:-129 glc dlc
+; GFX11-PAL-NEXT:    scratch_load_u8 v0, v0, off glc dlc
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-PAL-NEXT:    s_setpc_b64 s[30:31]
 bb:
diff --git a/llvm/test/CodeGen/AMDGPU/fneg.ll b/llvm/test/CodeGen/AMDGPU/fneg.ll
index 7d7162165f9ed..760d9e96a6090 100644
--- a/llvm/test/CodeGen/AMDGPU/fneg.ll
+++ b/llvm/test/CodeGen/AMDGPU/fneg.ll
@@ -218,13 +218,8 @@ define half @v_fneg_i16_fp_use(i16 %in) {
   ret half %fadd
 }
 
-; FIXME: This is terrible
 ; FUNC-LABEL: {{^}}s_fneg_v2i16:
-; SI: s_and_b32 s5, s4, 0xffff0000
-; SI: s_xor_b32 s4, s4, 0x8000
-; SI: s_and_b32 s4, s4, 0xffff
-; SI: s_or_b32 s4, s4, s5
-; SI: s_add_i32 s4, s4, 0x80000000
+; SI: s_xor_b32 s4, s4, 0x80008000
 
 ; VI: s_lshr_b32 s5, s4, 16
 ; VI: s_xor_b32 s4, s4, 0x8000
diff --git a/llvm/test/CodeGen/AMDGPU/fp-min-max-flat-atomics-f64.ll b/llvm/test/CodeGen/AMDGPU/fp-min-max-flat-atomics-f64.ll
new file mode 100644
index 0000000000000..10617a6e966fb
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/fp-min-max-flat-atomics-f64.ll
@@ -0,0 +1,51 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -global-isel=0 -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs | FileCheck %s -check-prefixes=GFX10,GFX10-SDAG
+; RUN: llc < %s -global-isel=1 -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs | FileCheck %s -check-prefixes=GFX10,GFX10-GISEL
+
+declare double @llvm.amdgcn.flat.atomic.fmin.f64.p1.f64(ptr %ptr, double %data)
+declare double @llvm.amdgcn.flat.atomic.fmax.f64.p1.f64(ptr %ptr, double %data)
+
+define amdgpu_cs void @flat_atomic_fmin_f64_noret(ptr %ptr, double %data) {
+; GFX10-LABEL: flat_atomic_fmin_f64_noret:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    flat_atomic_fmin_x2 v[0:1], v[2:3]
+; GFX10-NEXT:    s_endpgm
+  %ret = call double @llvm.amdgcn.flat.atomic.fmin.f64.p1.f64(ptr %ptr, double %data)
+  ret void
+}
+
+define amdgpu_cs void @flat_atomic_fmax_f64_noret(ptr %ptr, double %data) {
+; GFX10-LABEL: flat_atomic_fmax_f64_noret:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    flat_atomic_fmax_x2 v[0:1], v[2:3]
+; GFX10-NEXT:    s_endpgm
+  %ret = call double @llvm.amdgcn.flat.atomic.fmax.f64.p1.f64(ptr %ptr, double %data)
+  ret void
+}
+
+define amdgpu_cs void @flat_atomic_fmin_f64_rtn(ptr %ptr, double %data, ptr %out) {
+; GFX10-LABEL: flat_atomic_fmin_f64_rtn:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    flat_atomic_fmin_x2 v[0:1], v[0:1], v[2:3] glc
+; GFX10-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX10-NEXT:    flat_store_dwordx2 v[4:5], v[0:1]
+; GFX10-NEXT:    s_endpgm
+  %ret = call double @llvm.amdgcn.flat.atomic.fmin.f64.p1.f64(ptr %ptr, double %data)
+  store double %ret, ptr %out
+  ret void
+}
+
+define amdgpu_cs void @flat_atomic_fmax_f64_rtn(ptr %ptr, double %data, ptr %out) {
+; GFX10-LABEL: flat_atomic_fmax_f64_rtn:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    flat_atomic_fmax_x2 v[0:1], v[0:1], v[2:3] glc
+; GFX10-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX10-NEXT:    flat_store_dwordx2 v[4:5], v[0:1]
+; GFX10-NEXT:    s_endpgm
+  %ret = call double @llvm.amdgcn.flat.atomic.fmax.f64.p1.f64(ptr %ptr, double %data)
+  store double %ret, ptr %out
+  ret void
+}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX10-GISEL: {{.*}}
+; GFX10-SDAG: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/fp-min-max-flat-atomics.ll b/llvm/test/CodeGen/AMDGPU/fp-min-max-flat-atomics.ll
new file mode 100644
index 0000000000000..b50bcd9da6ac7
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/fp-min-max-flat-atomics.ll
@@ -0,0 +1,89 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -global-isel=0 -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs | FileCheck %s -check-prefixes=GFX10,GFX10-SDAG
+; RUN: llc < %s -global-isel=0 -march=amdgcn -mcpu=gfx1100 -verify-machineinstrs | FileCheck %s -check-prefixes=GFX11,GFX11-SDAG
+; RUN: llc < %s -global-isel=1 -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs | FileCheck %s -check-prefixes=GFX10,GFX10-GISEL
+; RUN: llc < %s -global-isel=1 -march=amdgcn -mcpu=gfx1100 -verify-machineinstrs | FileCheck %s -check-prefixes=GFX11,GFX11-GISEL
+
+declare float @llvm.amdgcn.flat.atomic.fmin.f32.p1.f32(ptr %ptr, float %data)
+declare float @llvm.amdgcn.flat.atomic.fmax.f32.p1.f32(ptr %ptr, float %data)
+
+define amdgpu_cs void @flat_atomic_fmin_f32_noret(ptr %ptr, float %data) {
+; GFX10-LABEL: flat_atomic_fmin_f32_noret:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    flat_atomic_fmin v[0:1], v2
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: flat_atomic_fmin_f32_noret:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    flat_atomic_min_f32 v[0:1], v2
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+  %ret = call float @llvm.amdgcn.flat.atomic.fmin.f32.p1.f32(ptr %ptr, float %data)
+  ret void
+}
+
+define amdgpu_cs void @flat_atomic_fmax_f32_noret(ptr %ptr, float %data) {
+; GFX10-LABEL: flat_atomic_fmax_f32_noret:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    flat_atomic_fmax v[0:1], v2
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: flat_atomic_fmax_f32_noret:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    flat_atomic_max_f32 v[0:1], v2
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+  %ret = call float @llvm.amdgcn.flat.atomic.fmax.f32.p1.f32(ptr %ptr, float %data)
+  ret void
+}
+
+define amdgpu_cs float @flat_atomic_fmin_f32_rtn(ptr %ptr, float %data, ptr %out) {
+; GFX10-LABEL: flat_atomic_fmin_f32_rtn:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    flat_atomic_fmin v0, v[0:1], v2 glc
+; GFX10-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX10-NEXT:    flat_store_dword v[3:4], v0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: flat_atomic_fmin_f32_rtn:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    flat_atomic_min_f32 v0, v[0:1], v2 glc
+; GFX11-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX11-NEXT:    flat_store_b32 v[3:4], v0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11-NEXT:    ; return to shader part epilog
+  %ret = call float @llvm.amdgcn.flat.atomic.fmin.f32.p1.f32(ptr %ptr, float %data)
+  store float %ret, ptr %out
+  ret float %ret
+}
+
+define amdgpu_cs float @flat_atomic_fmax_f32_rtn(ptr %ptr, float %data, ptr %out) {
+; GFX10-LABEL: flat_atomic_fmax_f32_rtn:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    flat_atomic_fmax v0, v[0:1], v2 glc
+; GFX10-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX10-NEXT:    flat_store_dword v[3:4], v0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: flat_atomic_fmax_f32_rtn:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    flat_atomic_max_f32 v0, v[0:1], v2 glc
+; GFX11-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX11-NEXT:    flat_store_b32 v[3:4], v0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11-NEXT:    ; return to shader part epilog
+  %ret = call float @llvm.amdgcn.flat.atomic.fmax.f32.p1.f32(ptr %ptr, float %data)
+  store float %ret, ptr %out
+  ret float %ret
+}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX10-GISEL: {{.*}}
+; GFX10-SDAG: {{.*}}
+; GFX11-GISEL: {{.*}}
+; GFX11-SDAG: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/fp-min-max-global-atomics-f64.ll b/llvm/test/CodeGen/AMDGPU/fp-min-max-global-atomics-f64.ll
new file mode 100644
index 0000000000000..5a861498bbbcd
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/fp-min-max-global-atomics-f64.ll
@@ -0,0 +1,51 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -global-isel=0 -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs | FileCheck %s -check-prefixes=GFX10,GFX10-SDAG
+; RUN: llc < %s -global-isel=1 -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs | FileCheck %s -check-prefixes=GFX10,GFX10-GISEL
+
+declare double @llvm.amdgcn.global.atomic.fmin.f64.p1.f64(ptr addrspace(1) %ptr, double %data)
+declare double @llvm.amdgcn.global.atomic.fmax.f64.p1.f64(ptr addrspace(1) %ptr, double %data)
+
+define amdgpu_cs void @global_atomic_fmin_f64_noret(ptr addrspace(1) %ptr, double %data) {
+; GFX10-LABEL: global_atomic_fmin_f64_noret:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    global_atomic_fmin_x2 v[0:1], v[2:3], off
+; GFX10-NEXT:    s_endpgm
+  %ret = call double @llvm.amdgcn.global.atomic.fmin.f64.p1.f64(ptr addrspace(1) %ptr, double %data)
+  ret void
+}
+
+define amdgpu_cs void @global_atomic_fmax_f64_noret(ptr addrspace(1) %ptr, double %data) {
+; GFX10-LABEL: global_atomic_fmax_f64_noret:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    global_atomic_fmax_x2 v[0:1], v[2:3], off
+; GFX10-NEXT:    s_endpgm
+  %ret = call double @llvm.amdgcn.global.atomic.fmax.f64.p1.f64(ptr addrspace(1) %ptr, double %data)
+  ret void
+}
+
+define amdgpu_cs void @global_atomic_fmin_f64_rtn(ptr addrspace(1) %ptr, double %data, ptr addrspace(1) %out) {
+; GFX10-LABEL: global_atomic_fmin_f64_rtn:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    global_atomic_fmin_x2 v[0:1], v[0:1], v[2:3], off glc
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    global_store_dwordx2 v[4:5], v[0:1], off
+; GFX10-NEXT:    s_endpgm
+  %ret = call double @llvm.amdgcn.global.atomic.fmin.f64.p1.f64(ptr addrspace(1) %ptr, double %data)
+  store double %ret, ptr addrspace(1) %out
+  ret void
+}
+
+define amdgpu_cs void @global_atomic_fmax_f64_rtn(ptr addrspace(1) %ptr, double %data, ptr addrspace(1) %out) {
+; GFX10-LABEL: global_atomic_fmax_f64_rtn:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    global_atomic_fmax_x2 v[0:1], v[0:1], v[2:3], off glc
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    global_store_dwordx2 v[4:5], v[0:1], off
+; GFX10-NEXT:    s_endpgm
+  %ret = call double @llvm.amdgcn.global.atomic.fmax.f64.p1.f64(ptr addrspace(1) %ptr, double %data)
+  store double %ret, ptr addrspace(1) %out
+  ret void
+}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX10-GISEL: {{.*}}
+; GFX10-SDAG: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/fp-min-max-global-atomics-gfx10.ll b/llvm/test/CodeGen/AMDGPU/fp-min-max-global-atomics-gfx10.ll
deleted file mode 100644
index c0ca697833b08..0000000000000
--- a/llvm/test/CodeGen/AMDGPU/fp-min-max-global-atomics-gfx10.ll
+++ /dev/null
@@ -1,197 +0,0 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc < %s -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs | FileCheck %s -check-prefix=GFX10
-
-; RUN: llc < %s -global-isel -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs | FileCheck %s -check-prefix=G_GFX10
-
-declare double @llvm.amdgcn.global.atomic.fmin.f64.p1.f64(ptr addrspace(1) %ptr, double %data)
-declare double @llvm.amdgcn.global.atomic.fmax.f64.p1.f64(ptr addrspace(1) %ptr, double %data)
-declare float @llvm.amdgcn.global.atomic.fmin.f32.p1.f32(ptr addrspace(1) %ptr, float %data)
-declare float @llvm.amdgcn.global.atomic.fmax.f32.p1.f32(ptr addrspace(1) %ptr, float %data)
-
-define amdgpu_kernel void @global_atomic_fmin_f32_noret(ptr addrspace(1) %ptr, float %data) {
-; GFX10-LABEL: global_atomic_fmin_f32_noret:
-; GFX10:       ; %bb.0: ; %main_body
-; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
-; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
-; GFX10-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10-NEXT:    v_mov_b32_e32 v1, s4
-; GFX10-NEXT:    global_atomic_fmin v0, v1, s[2:3]
-; GFX10-NEXT:    s_endpgm
-;
-; G_GFX10-LABEL: global_atomic_fmin_f32_noret:
-; G_GFX10:       ; %bb.0: ; %main_body
-; G_GFX10-NEXT:    s_clause 0x1
-; G_GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
-; G_GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
-; G_GFX10-NEXT:    v_mov_b32_e32 v1, 0
-; G_GFX10-NEXT:    s_waitcnt lgkmcnt(0)
-; G_GFX10-NEXT:    v_mov_b32_e32 v0, s4
-; G_GFX10-NEXT:    global_atomic_fmin v1, v0, s[2:3]
-; G_GFX10-NEXT:    s_endpgm
-main_body:
-  %ret = call float @llvm.amdgcn.global.atomic.fmin.f32.p1.f32(ptr addrspace(1) %ptr, float %data)
-  ret void
-}
-
-define amdgpu_kernel void @global_atomic_fmax_f32_noret(ptr addrspace(1) %ptr, float %data) {
-; GFX10-LABEL: global_atomic_fmax_f32_noret:
-; GFX10:       ; %bb.0: ; %main_body
-; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
-; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
-; GFX10-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10-NEXT:    v_mov_b32_e32 v1, s4
-; GFX10-NEXT:    global_atomic_fmax v0, v1, s[2:3]
-; GFX10-NEXT:    s_endpgm
-;
-; G_GFX10-LABEL: global_atomic_fmax_f32_noret:
-; G_GFX10:       ; %bb.0: ; %main_body
-; G_GFX10-NEXT:    s_clause 0x1
-; G_GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
-; G_GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
-; G_GFX10-NEXT:    v_mov_b32_e32 v1, 0
-; G_GFX10-NEXT:    s_waitcnt lgkmcnt(0)
-; G_GFX10-NEXT:    v_mov_b32_e32 v0, s4
-; G_GFX10-NEXT:    global_atomic_fmax v1, v0, s[2:3]
-; G_GFX10-NEXT:    s_endpgm
-main_body:
-  %ret = call float @llvm.amdgcn.global.atomic.fmax.f32.p1.f32(ptr addrspace(1) %ptr, float %data)
-  ret void
-}
-
-define float @global_atomic_fmax_f32_rtn(ptr addrspace(1) %ptr, float %data) {
-; GFX10-LABEL: global_atomic_fmax_f32_rtn:
-; GFX10:       ; %bb.0: ; %main_body
-; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-NEXT:    global_atomic_fmax v0, v[0:1], v2, off glc
-; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    s_setpc_b64 s[30:31]
-;
-; G_GFX10-LABEL: global_atomic_fmax_f32_rtn:
-; G_GFX10:       ; %bb.0: ; %main_body
-; G_GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; G_GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; G_GFX10-NEXT:    global_atomic_fmax v0, v[0:1], v2, off glc
-; G_GFX10-NEXT:    s_waitcnt vmcnt(0)
-; G_GFX10-NEXT:    s_setpc_b64 s[30:31]
-main_body:
-  %ret = call float @llvm.amdgcn.global.atomic.fmax.f32.p1.f32(ptr addrspace(1) %ptr, float %data)
-  ret float %ret
-}
-
-define float @global_atomic_fmin_f32_rtn(ptr addrspace(1) %ptr, float %data) {
-; GFX10-LABEL: global_atomic_fmin_f32_rtn:
-; GFX10:       ; %bb.0: ; %main_body
-; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-NEXT:    global_atomic_fmin v0, v[0:1], v2, off glc
-; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    s_setpc_b64 s[30:31]
-;
-; G_GFX10-LABEL: global_atomic_fmin_f32_rtn:
-; G_GFX10:       ; %bb.0: ; %main_body
-; G_GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; G_GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; G_GFX10-NEXT:    global_atomic_fmin v0, v[0:1], v2, off glc
-; G_GFX10-NEXT:    s_waitcnt vmcnt(0)
-; G_GFX10-NEXT:    s_setpc_b64 s[30:31]
-main_body:
-  %ret = call float @llvm.amdgcn.global.atomic.fmin.f32.p1.f32(ptr addrspace(1) %ptr, float %data)
-  ret float %ret
-}
-
-define amdgpu_kernel void @global_atomic_fmin_f64_noret(ptr addrspace(1) %ptr, double %data) {
-; GFX10-LABEL: global_atomic_fmin_f64_noret:
-; GFX10:       ; %bb.0: ; %main_body
-; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; GFX10-NEXT:    v_mov_b32_e32 v2, 0
-; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10-NEXT:    v_mov_b32_e32 v0, s2
-; GFX10-NEXT:    v_mov_b32_e32 v1, s3
-; GFX10-NEXT:    global_atomic_fmin_x2 v2, v[0:1], s[0:1]
-; GFX10-NEXT:    s_endpgm
-;
-; G_GFX10-LABEL: global_atomic_fmin_f64_noret:
-; G_GFX10:       ; %bb.0: ; %main_body
-; G_GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; G_GFX10-NEXT:    v_mov_b32_e32 v2, 0
-; G_GFX10-NEXT:    s_waitcnt lgkmcnt(0)
-; G_GFX10-NEXT:    v_mov_b32_e32 v0, s2
-; G_GFX10-NEXT:    v_mov_b32_e32 v1, s3
-; G_GFX10-NEXT:    global_atomic_fmin_x2 v2, v[0:1], s[0:1]
-; G_GFX10-NEXT:    s_endpgm
-main_body:
-  %ret = call double @llvm.amdgcn.global.atomic.fmin.f64.p1.f64(ptr addrspace(1) %ptr, double %data)
-  ret void
-}
-
-define amdgpu_kernel void @global_atomic_fmax_f64_noret(ptr addrspace(1) %ptr, double %data) {
-; GFX10-LABEL: global_atomic_fmax_f64_noret:
-; GFX10:       ; %bb.0: ; %main_body
-; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; GFX10-NEXT:    v_mov_b32_e32 v2, 0
-; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10-NEXT:    v_mov_b32_e32 v0, s2
-; GFX10-NEXT:    v_mov_b32_e32 v1, s3
-; GFX10-NEXT:    global_atomic_fmax_x2 v2, v[0:1], s[0:1]
-; GFX10-NEXT:    s_endpgm
-;
-; G_GFX10-LABEL: global_atomic_fmax_f64_noret:
-; G_GFX10:       ; %bb.0: ; %main_body
-; G_GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; G_GFX10-NEXT:    v_mov_b32_e32 v2, 0
-; G_GFX10-NEXT:    s_waitcnt lgkmcnt(0)
-; G_GFX10-NEXT:    v_mov_b32_e32 v0, s2
-; G_GFX10-NEXT:    v_mov_b32_e32 v1, s3
-; G_GFX10-NEXT:    global_atomic_fmax_x2 v2, v[0:1], s[0:1]
-; G_GFX10-NEXT:    s_endpgm
-main_body:
-  %ret = call double @llvm.amdgcn.global.atomic.fmax.f64.p1.f64(ptr addrspace(1) %ptr, double %data)
-  ret void
-}
-
-define double @global_atomic_fmax_f64_rtn(ptr addrspace(1) %ptr, double %data) {
-; GFX10-LABEL: global_atomic_fmax_f64_rtn:
-; GFX10:       ; %bb.0: ; %main_body
-; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-NEXT:    global_atomic_fmax_x2 v[0:1], v[0:1], v[2:3], off glc
-; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    s_setpc_b64 s[30:31]
-;
-; G_GFX10-LABEL: global_atomic_fmax_f64_rtn:
-; G_GFX10:       ; %bb.0: ; %main_body
-; G_GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; G_GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; G_GFX10-NEXT:    global_atomic_fmax_x2 v[0:1], v[0:1], v[2:3], off glc
-; G_GFX10-NEXT:    s_waitcnt vmcnt(0)
-; G_GFX10-NEXT:    s_setpc_b64 s[30:31]
-main_body:
-  %ret = call double @llvm.amdgcn.global.atomic.fmax.f64.p1.f64(ptr addrspace(1) %ptr, double %data)
-  ret double %ret
-}
-
-define double @global_atomic_fmin_f64_rtn(ptr addrspace(1) %ptr, double %data) {
-; GFX10-LABEL: global_atomic_fmin_f64_rtn:
-; GFX10:       ; %bb.0: ; %main_body
-; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-NEXT:    global_atomic_fmin_x2 v[0:1], v[0:1], v[2:3], off glc
-; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    s_setpc_b64 s[30:31]
-;
-; G_GFX10-LABEL: global_atomic_fmin_f64_rtn:
-; G_GFX10:       ; %bb.0: ; %main_body
-; G_GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; G_GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; G_GFX10-NEXT:    global_atomic_fmin_x2 v[0:1], v[0:1], v[2:3], off glc
-; G_GFX10-NEXT:    s_waitcnt vmcnt(0)
-; G_GFX10-NEXT:    s_setpc_b64 s[30:31]
-main_body:
-  %ret = call double @llvm.amdgcn.global.atomic.fmin.f64.p1.f64(ptr addrspace(1) %ptr, double %data)
-  ret double %ret
-}
diff --git a/llvm/test/CodeGen/AMDGPU/fp-min-max-global-atomics.ll b/llvm/test/CodeGen/AMDGPU/fp-min-max-global-atomics.ll
new file mode 100644
index 0000000000000..ac78fb2c1d2fb
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/fp-min-max-global-atomics.ll
@@ -0,0 +1,83 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -global-isel=0 -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs | FileCheck %s -check-prefixes=GFX10,GFX10-SDAG
+; RUN: llc < %s -global-isel=0 -march=amdgcn -mcpu=gfx1100 -verify-machineinstrs | FileCheck %s -check-prefixes=GFX11,GFX11-SDAG
+; RUN: llc < %s -global-isel=1 -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs | FileCheck %s -check-prefixes=GFX10,GFX10-GISEL
+; RUN: llc < %s -global-isel=1 -march=amdgcn -mcpu=gfx1100 -verify-machineinstrs | FileCheck %s -check-prefixes=GFX11,GFX11-GISEL
+
+declare float @llvm.amdgcn.global.atomic.fmin.f32.p1.f32(ptr addrspace(1) %ptr, float %data)
+declare float @llvm.amdgcn.global.atomic.fmax.f32.p1.f32(ptr addrspace(1) %ptr, float %data)
+
+define amdgpu_cs void @global_atomic_fmin_f32_noret(ptr addrspace(1) %ptr, float %data) {
+; GFX10-LABEL: global_atomic_fmin_f32_noret:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    global_atomic_fmin v[0:1], v2, off
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: global_atomic_fmin_f32_noret:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    global_atomic_min_f32 v[0:1], v2, off
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+  %ret = call float @llvm.amdgcn.global.atomic.fmin.f32.p1.f32(ptr addrspace(1) %ptr, float %data)
+  ret void
+}
+
+define amdgpu_cs void @global_atomic_fmax_f32_noret(ptr addrspace(1) %ptr, float %data) {
+; GFX10-LABEL: global_atomic_fmax_f32_noret:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    global_atomic_fmax v[0:1], v2, off
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: global_atomic_fmax_f32_noret:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    global_atomic_max_f32 v[0:1], v2, off
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+  %ret = call float @llvm.amdgcn.global.atomic.fmax.f32.p1.f32(ptr addrspace(1) %ptr, float %data)
+  ret void
+}
+
+define amdgpu_cs void @global_atomic_fmax_f32_rtn(ptr addrspace(1) %ptr, float %data, ptr addrspace(1) %out) {
+; GFX10-LABEL: global_atomic_fmax_f32_rtn:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    global_atomic_fmax v0, v[0:1], v2, off glc
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    global_store_dword v[3:4], v0, off
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: global_atomic_fmax_f32_rtn:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    global_atomic_max_f32 v0, v[0:1], v2, off glc
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    global_store_b32 v[3:4], v0, off
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+  %ret = call float @llvm.amdgcn.global.atomic.fmax.f32.p1.f32(ptr addrspace(1) %ptr, float %data)
+  store float %ret, ptr addrspace(1) %out
+  ret void
+}
+
+define amdgpu_cs void @global_atomic_fmin_f32_rtn(ptr addrspace(1) %ptr, float %data, ptr addrspace(1) %out) {
+; GFX10-LABEL: global_atomic_fmin_f32_rtn:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    global_atomic_fmin v0, v[0:1], v2, off glc
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    global_store_dword v[3:4], v0, off
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: global_atomic_fmin_f32_rtn:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    global_atomic_min_f32 v0, v[0:1], v2, off glc
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    global_store_b32 v[3:4], v0, off
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+  %ret = call float @llvm.amdgcn.global.atomic.fmin.f32.p1.f32(ptr addrspace(1) %ptr, float %data)
+  store float %ret, ptr addrspace(1) %out
+  ret void
+}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX10-GISEL: {{.*}}
+; GFX10-SDAG: {{.*}}
+; GFX11-GISEL: {{.*}}
+; GFX11-SDAG: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/gfx-callable-argument-types.ll b/llvm/test/CodeGen/AMDGPU/gfx-callable-argument-types.ll
index 8da0dc3c0e0ee..7f481b2a134e2 100644
--- a/llvm/test/CodeGen/AMDGPU/gfx-callable-argument-types.ll
+++ b/llvm/test/CodeGen/AMDGPU/gfx-callable-argument-types.ll
@@ -13845,6 +13845,8 @@ define amdgpu_gfx void @test_call_external_void_func_v32i32_inreg() #0 {
 ; GFX11-NEXT:    v_writelane_b32 v41, s0, 0
 ; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x0
 ; GFX11-NEXT:    s_add_i32 s32, s32, 16
+; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
+; GFX11-NEXT:    s_add_i32 s2, s32, 16
 ; GFX11-NEXT:    v_writelane_b32 v40, s5, 1
 ; GFX11-NEXT:    v_writelane_b32 v40, s6, 2
 ; GFX11-NEXT:    v_writelane_b32 v40, s7, 3
@@ -13883,8 +13885,7 @@ define amdgpu_gfx void @test_call_external_void_func_v32i32_inreg() #0 {
 ; GFX11-NEXT:    v_writelane_b32 v40, s25, 21
 ; GFX11-NEXT:    s_mov_b32 s24, s40
 ; GFX11-NEXT:    s_mov_b32 s25, s41
-; GFX11-NEXT:    s_clause 0x1
-; GFX11-NEXT:    scratch_store_b64 off, v[4:5], s32 offset:16
+; GFX11-NEXT:    scratch_store_b64 off, v[4:5], s2
 ; GFX11-NEXT:    scratch_store_b128 off, v[0:3], s32
 ; GFX11-NEXT:    v_writelane_b32 v40, s26, 22
 ; GFX11-NEXT:    s_mov_b32 s26, s42
@@ -13952,6 +13953,7 @@ define amdgpu_gfx void @test_call_external_void_func_v32i32_inreg() #0 {
 ; GFX10-SCRATCH-NEXT:    v_writelane_b32 v41, s0, 0
 ; GFX10-SCRATCH-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x0
 ; GFX10-SCRATCH-NEXT:    s_add_i32 s32, s32, 16
+; GFX10-SCRATCH-NEXT:    s_add_i32 s2, s32, 16
 ; GFX10-SCRATCH-NEXT:    v_writelane_b32 v40, s5, 1
 ; GFX10-SCRATCH-NEXT:    v_writelane_b32 v40, s6, 2
 ; GFX10-SCRATCH-NEXT:    v_writelane_b32 v40, s7, 3
@@ -13993,7 +13995,7 @@ define amdgpu_gfx void @test_call_external_void_func_v32i32_inreg() #0 {
 ; GFX10-SCRATCH-NEXT:    s_mov_b32 s23, s39
 ; GFX10-SCRATCH-NEXT:    s_mov_b32 s24, s40
 ; GFX10-SCRATCH-NEXT:    s_mov_b32 s25, s41
-; GFX10-SCRATCH-NEXT:    scratch_store_dwordx2 off, v[4:5], s32 offset:16
+; GFX10-SCRATCH-NEXT:    scratch_store_dwordx2 off, v[4:5], s2
 ; GFX10-SCRATCH-NEXT:    scratch_store_dwordx4 off, v[0:3], s32
 ; GFX10-SCRATCH-NEXT:    v_writelane_b32 v40, s26, 22
 ; GFX10-SCRATCH-NEXT:    s_mov_b32 s26, s42
@@ -14321,6 +14323,7 @@ define amdgpu_gfx void @test_call_external_void_func_v32i32_i32_inreg(i32) #0 {
 ; GFX11-NEXT:    s_getpc_b64 s[0:1]
 ; GFX11-NEXT:    s_add_u32 s0, s0, external_void_func_v32i32_i32_inreg@rel32@lo+4
 ; GFX11-NEXT:    s_addc_u32 s1, s1, external_void_func_v32i32_i32_inreg@rel32@hi+12
+; GFX11-NEXT:    s_add_i32 s3, s32, 16
 ; GFX11-NEXT:    v_writelane_b32 v40, s20, 16
 ; GFX11-NEXT:    v_writelane_b32 v40, s21, 17
 ; GFX11-NEXT:    v_writelane_b32 v40, s22, 18
@@ -14331,19 +14334,19 @@ define amdgpu_gfx void @test_call_external_void_func_v32i32_i32_inreg(i32) #0 {
 ; GFX11-NEXT:    v_dual_mov_b32 v0, s46 :: v_dual_mov_b32 v3, s49
 ; GFX11-NEXT:    v_writelane_b32 v40, s24, 20
 ; GFX11-NEXT:    v_mov_b32_e32 v2, s48
+; GFX11-NEXT:    s_add_i32 s2, s32, 24
 ; GFX11-NEXT:    s_mov_b32 s20, s36
 ; GFX11-NEXT:    s_mov_b32 s21, s37
-; GFX11-NEXT:    s_mov_b32 s22, s38
 ; GFX11-NEXT:    v_writelane_b32 v40, s25, 21
+; GFX11-NEXT:    s_mov_b32 s22, s38
 ; GFX11-NEXT:    s_mov_b32 s23, s39
 ; GFX11-NEXT:    s_mov_b32 s24, s40
 ; GFX11-NEXT:    s_mov_b32 s25, s41
-; GFX11-NEXT:    s_clause 0x2
-; GFX11-NEXT:    scratch_store_b32 off, v6, s32 offset:24
-; GFX11-NEXT:    scratch_store_b64 off, v[4:5], s32 offset:16
-; GFX11-NEXT:    scratch_store_b128 off, v[0:3], s32
 ; GFX11-NEXT:    v_writelane_b32 v40, s26, 22
 ; GFX11-NEXT:    s_mov_b32 s26, s42
+; GFX11-NEXT:    scratch_store_b32 off, v6, s2
+; GFX11-NEXT:    scratch_store_b64 off, v[4:5], s3
+; GFX11-NEXT:    scratch_store_b128 off, v[0:3], s32
 ; GFX11-NEXT:    v_writelane_b32 v40, s27, 23
 ; GFX11-NEXT:    s_mov_b32 s27, s43
 ; GFX11-NEXT:    v_writelane_b32 v40, s28, 24
@@ -14433,11 +14436,13 @@ define amdgpu_gfx void @test_call_external_void_func_v32i32_i32_inreg(i32) #0 {
 ; GFX10-SCRATCH-NEXT:    s_getpc_b64 s[0:1]
 ; GFX10-SCRATCH-NEXT:    s_add_u32 s0, s0, external_void_func_v32i32_i32_inreg@rel32@lo+4
 ; GFX10-SCRATCH-NEXT:    s_addc_u32 s1, s1, external_void_func_v32i32_i32_inreg@rel32@hi+12
+; GFX10-SCRATCH-NEXT:    s_add_i32 s3, s32, 16
 ; GFX10-SCRATCH-NEXT:    v_writelane_b32 v40, s20, 16
 ; GFX10-SCRATCH-NEXT:    v_writelane_b32 v40, s21, 17
 ; GFX10-SCRATCH-NEXT:    v_writelane_b32 v40, s22, 18
 ; GFX10-SCRATCH-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v6, s2
+; GFX10-SCRATCH-NEXT:    s_add_i32 s2, s32, 24
 ; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v4, s50
 ; GFX10-SCRATCH-NEXT:    v_writelane_b32 v40, s23, 19
 ; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v5, s51
@@ -14453,8 +14458,8 @@ define amdgpu_gfx void @test_call_external_void_func_v32i32_i32_inreg(i32) #0 {
 ; GFX10-SCRATCH-NEXT:    s_mov_b32 s23, s39
 ; GFX10-SCRATCH-NEXT:    s_mov_b32 s24, s40
 ; GFX10-SCRATCH-NEXT:    s_mov_b32 s25, s41
-; GFX10-SCRATCH-NEXT:    scratch_store_dword off, v6, s32 offset:24
-; GFX10-SCRATCH-NEXT:    scratch_store_dwordx2 off, v[4:5], s32 offset:16
+; GFX10-SCRATCH-NEXT:    scratch_store_dword off, v6, s2
+; GFX10-SCRATCH-NEXT:    scratch_store_dwordx2 off, v[4:5], s3
 ; GFX10-SCRATCH-NEXT:    scratch_store_dwordx4 off, v[0:3], s32
 ; GFX10-SCRATCH-NEXT:    v_writelane_b32 v40, s26, 22
 ; GFX10-SCRATCH-NEXT:    s_mov_b32 s26, s42
@@ -15115,15 +15120,16 @@ define amdgpu_gfx void @stack_8xv5i32() #0 {
 ; GFX11-NEXT:    scratch_store_b32 off, v40, s33
 ; GFX11-NEXT:    scratch_store_b32 off, v41, s33 offset:4
 ; GFX11-NEXT:    s_mov_b32 exec_lo, s1
-; GFX11-NEXT:    v_dual_mov_b32 v0, 12 :: v_dual_mov_b32 v1, 13
-; GFX11-NEXT:    v_dual_mov_b32 v2, 14 :: v_dual_mov_b32 v3, 15
-; GFX11-NEXT:    v_dual_mov_b32 v4, 8 :: v_dual_mov_b32 v5, 9
-; GFX11-NEXT:    v_dual_mov_b32 v6, 10 :: v_dual_mov_b32 v7, 11
+; GFX11-NEXT:    v_dual_mov_b32 v0, 8 :: v_dual_mov_b32 v1, 9
+; GFX11-NEXT:    v_dual_mov_b32 v2, 10 :: v_dual_mov_b32 v3, 11
+; GFX11-NEXT:    v_dual_mov_b32 v4, 12 :: v_dual_mov_b32 v5, 13
+; GFX11-NEXT:    v_dual_mov_b32 v6, 14 :: v_dual_mov_b32 v7, 15
 ; GFX11-NEXT:    s_add_i32 s32, s32, 16
+; GFX11-NEXT:    v_writelane_b32 v41, s0, 0
+; GFX11-NEXT:    s_add_i32 s0, s32, 16
 ; GFX11-NEXT:    v_writelane_b32 v40, s30, 0
-; GFX11-NEXT:    s_clause 0x1
-; GFX11-NEXT:    scratch_store_b128 off, v[0:3], s32 offset:16
-; GFX11-NEXT:    scratch_store_b128 off, v[4:7], s32
+; GFX11-NEXT:    scratch_store_b128 off, v[0:3], s32
+; GFX11-NEXT:    scratch_store_b128 off, v[4:7], s0
 ; GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, 0
 ; GFX11-NEXT:    v_dual_mov_b32 v2, 0 :: v_dual_mov_b32 v3, 0
 ; GFX11-NEXT:    v_dual_mov_b32 v4, 0 :: v_dual_mov_b32 v5, 1
@@ -15140,7 +15146,6 @@ define amdgpu_gfx void @stack_8xv5i32() #0 {
 ; GFX11-NEXT:    v_dual_mov_b32 v26, 5 :: v_dual_mov_b32 v27, 5
 ; GFX11-NEXT:    v_dual_mov_b32 v28, 5 :: v_dual_mov_b32 v29, 5
 ; GFX11-NEXT:    v_dual_mov_b32 v30, 6 :: v_dual_mov_b32 v31, 7
-; GFX11-NEXT:    v_writelane_b32 v41, s0, 0
 ; GFX11-NEXT:    v_writelane_b32 v40, s31, 1
 ; GFX11-NEXT:    s_getpc_b64 s[0:1]
 ; GFX11-NEXT:    s_add_u32 s0, s0, external_void_func_8xv5i32@rel32@lo+4
@@ -15171,18 +15176,20 @@ define amdgpu_gfx void @stack_8xv5i32() #0 {
 ; GFX10-SCRATCH-NEXT:    scratch_store_dword off, v41, s33 offset:4 ; 4-byte Folded Spill
 ; GFX10-SCRATCH-NEXT:    s_waitcnt_depctr 0xffe3
 ; GFX10-SCRATCH-NEXT:    s_mov_b32 exec_lo, s1
-; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v0, 12
-; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v1, 13
-; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v2, 14
-; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v3, 15
-; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v4, 8
-; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v5, 9
-; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v6, 10
-; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v7, 11
+; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v0, 8
+; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v1, 9
+; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v2, 10
+; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v3, 11
+; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v4, 12
+; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v5, 13
+; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v6, 14
+; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v7, 15
 ; GFX10-SCRATCH-NEXT:    s_add_i32 s32, s32, 16
+; GFX10-SCRATCH-NEXT:    v_writelane_b32 v41, s0, 0
+; GFX10-SCRATCH-NEXT:    s_add_i32 s0, s32, 16
 ; GFX10-SCRATCH-NEXT:    v_writelane_b32 v40, s30, 0
-; GFX10-SCRATCH-NEXT:    scratch_store_dwordx4 off, v[0:3], s32 offset:16
-; GFX10-SCRATCH-NEXT:    scratch_store_dwordx4 off, v[4:7], s32
+; GFX10-SCRATCH-NEXT:    scratch_store_dwordx4 off, v[0:3], s32
+; GFX10-SCRATCH-NEXT:    scratch_store_dwordx4 off, v[4:7], s0
 ; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v2, 0
@@ -15215,7 +15222,6 @@ define amdgpu_gfx void @stack_8xv5i32() #0 {
 ; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v29, 5
 ; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v30, 6
 ; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v31, 7
-; GFX10-SCRATCH-NEXT:    v_writelane_b32 v41, s0, 0
 ; GFX10-SCRATCH-NEXT:    v_writelane_b32 v40, s31, 1
 ; GFX10-SCRATCH-NEXT:    s_getpc_b64 s[0:1]
 ; GFX10-SCRATCH-NEXT:    s_add_u32 s0, s0, external_void_func_8xv5i32@rel32@lo+4
@@ -15417,19 +15423,20 @@ define amdgpu_gfx void @stack_8xv5f32() #0 {
 ; GFX11-NEXT:    scratch_store_b32 off, v40, s33
 ; GFX11-NEXT:    scratch_store_b32 off, v41, s33 offset:4
 ; GFX11-NEXT:    s_mov_b32 exec_lo, s1
-; GFX11-NEXT:    v_mov_b32_e32 v0, 0x41400000
-; GFX11-NEXT:    v_mov_b32_e32 v1, 0x41500000
-; GFX11-NEXT:    v_mov_b32_e32 v2, 0x41600000
-; GFX11-NEXT:    v_mov_b32_e32 v3, 0x41700000
-; GFX11-NEXT:    v_mov_b32_e32 v4, 0x41000000
-; GFX11-NEXT:    v_mov_b32_e32 v5, 0x41100000
-; GFX11-NEXT:    v_mov_b32_e32 v6, 0x41200000
-; GFX11-NEXT:    v_mov_b32_e32 v7, 0x41300000
+; GFX11-NEXT:    v_mov_b32_e32 v0, 0x41000000
+; GFX11-NEXT:    v_mov_b32_e32 v1, 0x41100000
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0x41200000
+; GFX11-NEXT:    v_mov_b32_e32 v3, 0x41300000
+; GFX11-NEXT:    v_mov_b32_e32 v4, 0x41400000
+; GFX11-NEXT:    v_mov_b32_e32 v5, 0x41500000
+; GFX11-NEXT:    v_mov_b32_e32 v6, 0x41600000
+; GFX11-NEXT:    v_mov_b32_e32 v7, 0x41700000
 ; GFX11-NEXT:    s_add_i32 s32, s32, 16
+; GFX11-NEXT:    v_writelane_b32 v41, s0, 0
+; GFX11-NEXT:    s_add_i32 s0, s32, 16
 ; GFX11-NEXT:    v_writelane_b32 v40, s30, 0
-; GFX11-NEXT:    s_clause 0x1
-; GFX11-NEXT:    scratch_store_b128 off, v[0:3], s32 offset:16
-; GFX11-NEXT:    scratch_store_b128 off, v[4:7], s32
+; GFX11-NEXT:    scratch_store_b128 off, v[0:3], s32
+; GFX11-NEXT:    scratch_store_b128 off, v[4:7], s0
 ; GFX11-NEXT:    v_mov_b32_e32 v6, 1.0
 ; GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, 0
 ; GFX11-NEXT:    v_dual_mov_b32 v2, 0 :: v_dual_mov_b32 v3, 0
@@ -15448,7 +15455,6 @@ define amdgpu_gfx void @stack_8xv5f32() #0 {
 ; GFX11-NEXT:    v_mov_b32_e32 v29, 0x40a00000
 ; GFX11-NEXT:    v_mov_b32_e32 v30, 0x40c00000
 ; GFX11-NEXT:    v_mov_b32_e32 v31, 0x40e00000
-; GFX11-NEXT:    v_writelane_b32 v41, s0, 0
 ; GFX11-NEXT:    v_writelane_b32 v40, s31, 1
 ; GFX11-NEXT:    s_getpc_b64 s[0:1]
 ; GFX11-NEXT:    s_add_u32 s0, s0, external_void_func_8xv5f32@rel32@lo+4
@@ -15479,18 +15485,20 @@ define amdgpu_gfx void @stack_8xv5f32() #0 {
 ; GFX10-SCRATCH-NEXT:    scratch_store_dword off, v41, s33 offset:4 ; 4-byte Folded Spill
 ; GFX10-SCRATCH-NEXT:    s_waitcnt_depctr 0xffe3
 ; GFX10-SCRATCH-NEXT:    s_mov_b32 exec_lo, s1
-; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v0, 0x41400000
-; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v1, 0x41500000
-; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v2, 0x41600000
-; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v3, 0x41700000
-; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v4, 0x41000000
-; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v5, 0x41100000
-; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v6, 0x41200000
-; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v7, 0x41300000
+; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v0, 0x41000000
+; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v1, 0x41100000
+; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v2, 0x41200000
+; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v3, 0x41300000
+; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v4, 0x41400000
+; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v5, 0x41500000
+; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v6, 0x41600000
+; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v7, 0x41700000
 ; GFX10-SCRATCH-NEXT:    s_add_i32 s32, s32, 16
+; GFX10-SCRATCH-NEXT:    v_writelane_b32 v41, s0, 0
+; GFX10-SCRATCH-NEXT:    s_add_i32 s0, s32, 16
 ; GFX10-SCRATCH-NEXT:    v_writelane_b32 v40, s30, 0
-; GFX10-SCRATCH-NEXT:    scratch_store_dwordx4 off, v[0:3], s32 offset:16
-; GFX10-SCRATCH-NEXT:    scratch_store_dwordx4 off, v[4:7], s32
+; GFX10-SCRATCH-NEXT:    scratch_store_dwordx4 off, v[0:3], s32
+; GFX10-SCRATCH-NEXT:    scratch_store_dwordx4 off, v[4:7], s0
 ; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v2, 0
@@ -15523,7 +15531,6 @@ define amdgpu_gfx void @stack_8xv5f32() #0 {
 ; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v29, 0x40a00000
 ; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v30, 0x40c00000
 ; GFX10-SCRATCH-NEXT:    v_mov_b32_e32 v31, 0x40e00000
-; GFX10-SCRATCH-NEXT:    v_writelane_b32 v41, s0, 0
 ; GFX10-SCRATCH-NEXT:    v_writelane_b32 v40, s31, 1
 ; GFX10-SCRATCH-NEXT:    s_getpc_b64 s[0:1]
 ; GFX10-SCRATCH-NEXT:    s_add_u32 s0, s0, external_void_func_8xv5f32@rel32@lo+4
diff --git a/llvm/test/CodeGen/AMDGPU/gfx-callable-return-types.ll b/llvm/test/CodeGen/AMDGPU/gfx-callable-return-types.ll
index a67a44971b647..ea4702bf0f27d 100644
--- a/llvm/test/CodeGen/AMDGPU/gfx-callable-return-types.ll
+++ b/llvm/test/CodeGen/AMDGPU/gfx-callable-return-types.ll
@@ -1497,137 +1497,263 @@ define amdgpu_gfx <512 x i32> @return_512xi32() #0 {
 ; GFX11-NEXT:    s_mov_b32 s2, s0
 ; GFX11-NEXT:    v_dual_mov_b32 v4, s3 :: v_dual_mov_b32 v3, s2
 ; GFX11-NEXT:    v_dual_mov_b32 v2, s1 :: v_dual_mov_b32 v1, s0
-; GFX11-NEXT:    s_clause 0x3e
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:2032
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:2016
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:2000
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1984
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1968
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1952
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1936
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1920
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1904
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1888
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1872
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1856
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1840
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1824
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1808
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1792
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1776
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1760
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1744
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1728
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1712
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1696
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1680
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1664
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1648
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1632
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1616
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1600
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1584
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1568
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1552
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1536
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1520
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1504
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1488
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1472
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1456
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1440
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1424
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1408
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1392
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1376
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1360
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1344
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1328
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1312
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1296
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1280
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1264
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1248
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1232
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1216
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1200
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1184
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1168
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1152
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1136
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1120
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1104
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1088
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1072
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1056
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1040
-; GFX11-NEXT:    s_clause 0x3e
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1024
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:1008
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:992
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:976
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:960
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:944
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:928
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:912
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:896
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:880
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:864
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:848
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:832
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:816
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:800
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:784
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:768
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:752
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:736
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:720
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:704
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:688
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:672
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:656
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:640
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:624
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:608
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:592
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:576
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:560
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:544
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:528
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:512
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:496
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:480
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:464
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:448
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:432
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:416
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:400
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:384
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:368
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:352
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:336
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:320
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:304
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:288
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:272
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:256
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:240
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:224
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:208
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:192
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:176
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:160
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:144
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:128
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:112
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:96
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:80
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:64
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:48
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:32
-; GFX11-NEXT:    s_clause 0x1
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off offset:16
-; GFX11-NEXT:    scratch_store_b128 v0, v[1:4], off
+; GFX11-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x7f0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x7e0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s0
+; GFX11-NEXT:    s_add_i32 s3, s0, 0x7d0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s3
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x7c0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x7b0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x7a0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x790
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x780
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x770
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x760
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x750
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x740
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x730
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x720
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x710
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x700
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x6f0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x6e0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x6d0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x6c0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x6b0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x6a0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x690
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x680
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x670
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x660
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x650
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x640
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x630
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x620
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x610
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x600
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x5f0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x5e0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x5d0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x5c0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x5b0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x5a0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x590
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x580
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x570
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x560
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x550
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x540
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x530
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x520
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x510
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x500
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x4f0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x4e0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x4d0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x4c0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x4b0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x4a0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x490
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x480
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x470
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x460
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x450
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x440
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x430
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x420
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x410
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x400
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x3f0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x3e0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x3d0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x3c0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x3b0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x3a0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x390
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x380
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x370
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x360
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x350
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x340
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x330
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x320
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x310
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x300
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x2f0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x2e0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x2d0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x2c0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x2b0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x2a0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x290
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x280
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x270
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x260
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x250
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x240
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x230
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x220
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x210
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x200
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x1f0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x1e0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x1d0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x1c0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x1b0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x1a0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x190
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x180
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x170
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x160
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x150
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x140
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x130
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x120
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x110
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x100
+; GFX11-NEXT:    s_add_i32 s2, s0, 0xf0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0xe0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0xd0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0xc0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0xb0
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0xa0
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x90
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x80
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x70
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 0x60
+; GFX11-NEXT:    s_add_i32 s2, s0, 0x50
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 64
+; GFX11-NEXT:    s_add_i32 s2, s0, 48
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s2
+; GFX11-NEXT:    s_add_i32 s1, s0, 32
+; GFX11-NEXT:    s_add_i32 s0, s0, 16
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s1
+; GFX11-NEXT:    scratch_store_b128 off, v[1:4], s0
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    s_setpc_b64 s[30:31]
 entry:
diff --git a/llvm/test/CodeGen/AMDGPU/idemponent-atomics.ll b/llvm/test/CodeGen/AMDGPU/idemponent-atomics.ll
new file mode 100644
index 0000000000000..fbb36f1d7ec8f
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/idemponent-atomics.ll
@@ -0,0 +1,168 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx940 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX940 %s
+; RUN: opt -mtriple=amdgcn-amd-amdhsa -S -atomic-expand < %s | FileCheck --check-prefix=OPT %s
+
+define i32 @global_agent_monotonic_idempotent_or(ptr addrspace(1) %in) {
+; GFX940-LABEL: global_agent_monotonic_idempotent_or:
+; GFX940:       ; %bb.0: ; %entry
+; GFX940-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX940-NEXT:    global_load_dword v0, v[0:1], off sc1
+; GFX940-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-NEXT:    s_setpc_b64 s[30:31]
+; OPT-LABEL: @global_agent_monotonic_idempotent_or(
+; OPT-NEXT:  entry:
+; OPT-NEXT:    [[VAL:%.*]] = load atomic i32, ptr addrspace(1) [[IN:%.*]] syncscope("agent-one-as") monotonic, align 4
+; OPT-NEXT:    ret i32 [[VAL]]
+;
+entry:
+  %val = atomicrmw or ptr addrspace(1) %in, i32 0 syncscope("agent-one-as") monotonic, align 4
+  ret i32 %val
+}
+
+define i32 @global_agent_acquire_idempotent_or(ptr addrspace(1) %in) {
+; GFX940-LABEL: global_agent_acquire_idempotent_or:
+; GFX940:       ; %bb.0: ; %entry
+; GFX940-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX940-NEXT:    global_load_dword v0, v[0:1], off sc1
+; GFX940-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-NEXT:    buffer_inv sc1
+; GFX940-NEXT:    s_setpc_b64 s[30:31]
+; OPT-LABEL: @global_agent_acquire_idempotent_or(
+; OPT-NEXT:  entry:
+; OPT-NEXT:    [[VAL:%.*]] = load atomic i32, ptr addrspace(1) [[IN:%.*]] syncscope("agent-one-as") acquire, align 4
+; OPT-NEXT:    ret i32 [[VAL]]
+;
+entry:
+  %val = atomicrmw or ptr addrspace(1) %in, i32 0 syncscope("agent-one-as") acquire, align 4
+  ret i32 %val
+}
+
+define i32 @global_agent_release_idempotent_or(ptr addrspace(1) %in) {
+; GFX940-LABEL: global_agent_release_idempotent_or:
+; GFX940:       ; %bb.0: ; %entry
+; GFX940-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX940-NEXT:    v_mov_b32_e32 v2, 0
+; GFX940-NEXT:    buffer_wbl2 sc1
+; GFX940-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-NEXT:    global_atomic_or v0, v[0:1], v2, off sc0
+; GFX940-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-NEXT:    s_setpc_b64 s[30:31]
+; OPT-LABEL: @global_agent_release_idempotent_or(
+; OPT-NEXT:  entry:
+; OPT-NEXT:    [[VAL:%.*]] = atomicrmw or ptr addrspace(1) [[IN:%.*]], i32 0 syncscope("agent-one-as") release, align 4
+; OPT-NEXT:    ret i32 [[VAL]]
+;
+entry:
+  %val = atomicrmw or ptr addrspace(1) %in, i32 0 syncscope("agent-one-as") release, align 4
+  ret i32 %val
+}
+
+define i32 @global_agent_acquire_release_idempotent_or(ptr addrspace(1) %in) {
+; GFX940-LABEL: global_agent_acquire_release_idempotent_or:
+; GFX940:       ; %bb.0: ; %entry
+; GFX940-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX940-NEXT:    v_mov_b32_e32 v2, 0
+; GFX940-NEXT:    buffer_wbl2 sc1
+; GFX940-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-NEXT:    global_atomic_or v0, v[0:1], v2, off sc0
+; GFX940-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-NEXT:    buffer_inv sc1
+; GFX940-NEXT:    s_setpc_b64 s[30:31]
+; OPT-LABEL: @global_agent_acquire_release_idempotent_or(
+; OPT-NEXT:  entry:
+; OPT-NEXT:    [[VAL:%.*]] = atomicrmw or ptr addrspace(1) [[IN:%.*]], i32 0 syncscope("agent-one-as") acq_rel, align 4
+; OPT-NEXT:    ret i32 [[VAL]]
+;
+entry:
+  %val = atomicrmw or ptr addrspace(1) %in, i32 0 syncscope("agent-one-as") acq_rel, align 4
+  ret i32 %val
+}
+
+define i32 @global_agent_seq_cst_idempotent_or(ptr addrspace(1) %in) {
+; GFX940-LABEL: global_agent_seq_cst_idempotent_or:
+; GFX940:       ; %bb.0: ; %entry
+; GFX940-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX940-NEXT:    v_mov_b32_e32 v2, 0
+; GFX940-NEXT:    buffer_wbl2 sc1
+; GFX940-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-NEXT:    global_atomic_or v0, v[0:1], v2, off sc0
+; GFX940-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-NEXT:    buffer_inv sc1
+; GFX940-NEXT:    s_setpc_b64 s[30:31]
+; OPT-LABEL: @global_agent_seq_cst_idempotent_or(
+; OPT-NEXT:  entry:
+; OPT-NEXT:    [[VAL:%.*]] = atomicrmw or ptr addrspace(1) [[IN:%.*]], i32 0 syncscope("agent-one-as") seq_cst, align 4
+; OPT-NEXT:    ret i32 [[VAL]]
+;
+entry:
+  %val = atomicrmw or ptr addrspace(1) %in, i32 0 syncscope("agent-one-as") seq_cst, align 4
+  ret i32 %val
+}
+
+define i32 @global_agent_monotonic_idempotent_add(ptr addrspace(1) %in) {
+; GFX940-LABEL: global_agent_monotonic_idempotent_add:
+; GFX940:       ; %bb.0: ; %entry
+; GFX940-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX940-NEXT:    global_load_dword v0, v[0:1], off sc0
+; GFX940-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-NEXT:    s_setpc_b64 s[30:31]
+; OPT-LABEL: @global_agent_monotonic_idempotent_add(
+; OPT-NEXT:  entry:
+; OPT-NEXT:    [[VAL:%.*]] = load atomic i32, ptr addrspace(1) [[IN:%.*]] syncscope("workgroup") monotonic, align 4
+; OPT-NEXT:    ret i32 [[VAL]]
+;
+entry:
+  %val = atomicrmw add ptr addrspace(1) %in, i32 0 syncscope("workgroup") monotonic, align 4
+  ret i32 %val
+}
+
+define i32 @global_agent_monotonic_idempotent_sub(ptr addrspace(1) %in) {
+; GFX940-LABEL: global_agent_monotonic_idempotent_sub:
+; GFX940:       ; %bb.0: ; %entry
+; GFX940-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX940-NEXT:    global_load_dword v0, v[0:1], off
+; GFX940-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-NEXT:    s_setpc_b64 s[30:31]
+; OPT-LABEL: @global_agent_monotonic_idempotent_sub(
+; OPT-NEXT:  entry:
+; OPT-NEXT:    [[VAL:%.*]] = load atomic i32, ptr addrspace(1) [[IN:%.*]] syncscope("wavefront") monotonic, align 4
+; OPT-NEXT:    ret i32 [[VAL]]
+;
+entry:
+  %val = atomicrmw sub ptr addrspace(1) %in, i32 0 syncscope("wavefront") monotonic, align 4
+  ret i32 %val
+}
+
+define i32 @global_system_monotonic_idempotent_xor(ptr addrspace(1) %in) {
+; GFX940-LABEL: global_system_monotonic_idempotent_xor:
+; GFX940:       ; %bb.0: ; %entry
+; GFX940-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX940-NEXT:    global_load_dword v0, v[0:1], off sc0 sc1
+; GFX940-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-NEXT:    s_setpc_b64 s[30:31]
+; OPT-LABEL: @global_system_monotonic_idempotent_xor(
+; OPT-NEXT:  entry:
+; OPT-NEXT:    [[VAL:%.*]] = load atomic i32, ptr addrspace(1) [[IN:%.*]] monotonic, align 4
+; OPT-NEXT:    ret i32 [[VAL]]
+;
+entry:
+  %val = atomicrmw xor ptr addrspace(1) %in, i32 0 monotonic, align 4
+  ret i32 %val
+}
+
+define i32 @global_agent_monotonic_idempotent_and(ptr addrspace(1) %in) {
+; GFX940-LABEL: global_agent_monotonic_idempotent_and:
+; GFX940:       ; %bb.0: ; %entry
+; GFX940-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX940-NEXT:    global_load_dword v0, v[0:1], off
+; GFX940-NEXT:    s_waitcnt vmcnt(0)
+; GFX940-NEXT:    s_setpc_b64 s[30:31]
+; OPT-LABEL: @global_agent_monotonic_idempotent_and(
+; OPT-NEXT:  entry:
+; OPT-NEXT:    [[VAL:%.*]] = load atomic i32, ptr addrspace(1) [[IN:%.*]] syncscope("singlethread") monotonic, align 4
+; OPT-NEXT:    ret i32 [[VAL]]
+;
+entry:
+  %val = atomicrmw and ptr addrspace(1) %in, i32 -1 syncscope("singlethread") monotonic, align 4
+  ret i32 %val
+}
diff --git a/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll b/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll
index dcc1f0eddf08c..6582a6a1c441e 100644
--- a/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll
+++ b/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll
@@ -1,16 +1,15 @@
-; When EXPENSIVE_CHECKS are enabled, the machine verifier appears between each
-; pass. Ignore it with 'grep -v'.
+; UNSUPPORTED: expensive_checks
 ; RUN: llc -O0 -mtriple=amdgcn--amdhsa -disable-verify -debug-pass=Structure < %s 2>&1 \
-; RUN:   | grep -v 'Verify generated machine code' | FileCheck -match-full-lines -strict-whitespace -check-prefix=GCN-O0 %s
+; RUN:   | FileCheck -match-full-lines -strict-whitespace -check-prefix=GCN-O0 %s
 ; RUN: llc -O1 -mtriple=amdgcn--amdhsa -disable-verify -debug-pass=Structure < %s 2>&1 \
-; RUN:   | grep -v 'Verify generated machine code' | FileCheck -match-full-lines -strict-whitespace -check-prefix=GCN-O1 %s
+; RUN:   | FileCheck -match-full-lines -strict-whitespace -check-prefix=GCN-O1 %s
 ; RUN: llc -O1 -mtriple=amdgcn--amdhsa -disable-verify -amdgpu-scalar-ir-passes -amdgpu-sdwa-peephole \
 ; RUN:   -amdgpu-load-store-vectorizer -amdgpu-enable-pre-ra-optimizations -debug-pass=Structure < %s 2>&1 \
-; RUN:   | grep -v 'Verify generated machine code' | FileCheck -match-full-lines -strict-whitespace -check-prefix=GCN-O1-OPTS %s
+; RUN:   | FileCheck -match-full-lines -strict-whitespace -check-prefix=GCN-O1-OPTS %s
 ; RUN: llc -O2 -mtriple=amdgcn--amdhsa -disable-verify -debug-pass=Structure < %s 2>&1 \
-; RUN:   | grep -v 'Verify generated machine code' | FileCheck -match-full-lines -strict-whitespace -check-prefix=GCN-O2 %s
+; RUN:   | FileCheck -match-full-lines -strict-whitespace -check-prefix=GCN-O2 %s
 ; RUN: llc -O3 -mtriple=amdgcn--amdhsa -disable-verify -debug-pass=Structure < %s 2>&1 \
-; RUN:   | grep -v 'Verify generated machine code' | FileCheck -match-full-lines -strict-whitespace -check-prefix=GCN-O3 %s
+; RUN:   | FileCheck -match-full-lines -strict-whitespace -check-prefix=GCN-O3 %s
 
 ; REQUIRES: asserts
 
@@ -87,8 +86,8 @@
 ; GCN-O0-NEXT:        AMDGPU Annotate Uniform Values
 ; GCN-O0-NEXT:        Natural Loop Information
 ; GCN-O0-NEXT:        SI annotate control flow
-; GCN-O0-NEXT:        Post-Dominator Tree Construction
-; GCN-O0-NEXT:        Legacy Divergence Analysis
+; GCN-O0-NEXT:        Cycle Info Analysis
+; GCN-O0-NEXT:        Uniformity Analysis
 ; GCN-O0-NEXT:        AMDGPU Rewrite Undef for PHI
 ; GCN-O0-NEXT:        LCSSA Verifier
 ; GCN-O0-NEXT:        Loop-Closed SSA Form Pass
@@ -98,9 +97,8 @@
 ; GCN-O0-NEXT:        Safe Stack instrumentation pass
 ; GCN-O0-NEXT:        Insert stack protectors
 ; GCN-O0-NEXT:        Dominator Tree Construction
-; GCN-O0-NEXT:        Post-Dominator Tree Construction
-; GCN-O0-NEXT:        Natural Loop Information
-; GCN-O0-NEXT:        Legacy Divergence Analysis
+; GCN-O0-NEXT:        Cycle Info Analysis
+; GCN-O0-NEXT:        Uniformity Analysis
 ; GCN-O0-NEXT:        Assignment Tracking Analysis
 ; GCN-O0-NEXT:        AMDGPU DAG->DAG Pattern Instruction Selection
 ; GCN-O0-NEXT:        MachineDominator Tree Construction
@@ -200,11 +198,11 @@
 ; GCN-O1-NEXT:      AMDGPU Promote Alloca
 ; GCN-O1-NEXT:      Dominator Tree Construction
 ; GCN-O1-NEXT:      SROA
-; GCN-O1-NEXT:      Post-Dominator Tree Construction
-; GCN-O1-NEXT:      Natural Loop Information
-; GCN-O1-NEXT:      Legacy Divergence Analysis
+; GCN-O1-NEXT:      Cycle Info Analysis
+; GCN-O1-NEXT:      Uniformity Analysis
 ; GCN-O1-NEXT:      AMDGPU IR optimizations
 ; GCN-O1-NEXT:      Basic Alias Analysis (stateless AA impl)
+; GCN-O1-NEXT:      Natural Loop Information
 ; GCN-O1-NEXT:      Canonicalize natural loops
 ; GCN-O1-NEXT:      Scalar Evolution Analysis
 ; GCN-O1-NEXT:      Loop Pass Manager
@@ -253,15 +251,14 @@
 ; GCN-O1-NEXT:        Function Alias Analysis Results
 ; GCN-O1-NEXT:        Flatten the CFG
 ; GCN-O1-NEXT:        Dominator Tree Construction
-; GCN-O1-NEXT:        Post-Dominator Tree Construction
-; GCN-O1-NEXT:        Natural Loop Information
-; GCN-O1-NEXT:        Legacy Divergence Analysis
+; GCN-O1-NEXT:        Cycle Info Analysis
+; GCN-O1-NEXT:        Uniformity Analysis
 ; GCN-O1-NEXT:        AMDGPU IR late optimizations
 ; GCN-O1-NEXT:        Basic Alias Analysis (stateless AA impl)
 ; GCN-O1-NEXT:        Function Alias Analysis Results
+; GCN-O1-NEXT:        Natural Loop Information
 ; GCN-O1-NEXT:        Code sinking
-; GCN-O1-NEXT:        Cycle Info Analysis
-; GCN-O1-NEXT:        Uniformity Analysis
+; GCN-O1-NEXT:        Post-Dominator Tree Construction
 ; GCN-O1-NEXT:        Unify divergent function exit nodes
 ; GCN-O1-NEXT:        Lazy Value Information Analysis
 ; GCN-O1-NEXT:        Lower SwitchInst's to branches
@@ -282,8 +279,8 @@
 ; GCN-O1-NEXT:        AMDGPU Annotate Uniform Values
 ; GCN-O1-NEXT:        Natural Loop Information
 ; GCN-O1-NEXT:        SI annotate control flow
-; GCN-O1-NEXT:        Post-Dominator Tree Construction
-; GCN-O1-NEXT:        Legacy Divergence Analysis
+; GCN-O1-NEXT:        Cycle Info Analysis
+; GCN-O1-NEXT:        Uniformity Analysis
 ; GCN-O1-NEXT:        AMDGPU Rewrite Undef for PHI
 ; GCN-O1-NEXT:        LCSSA Verifier
 ; GCN-O1-NEXT:        Loop-Closed SSA Form Pass
@@ -293,11 +290,12 @@
 ; GCN-O1-NEXT:        Safe Stack instrumentation pass
 ; GCN-O1-NEXT:        Insert stack protectors
 ; GCN-O1-NEXT:        Dominator Tree Construction
-; GCN-O1-NEXT:        Post-Dominator Tree Construction
-; GCN-O1-NEXT:        Natural Loop Information
-; GCN-O1-NEXT:        Legacy Divergence Analysis
+; GCN-O1-NEXT:        Cycle Info Analysis
+; GCN-O1-NEXT:        Uniformity Analysis
 ; GCN-O1-NEXT:        Basic Alias Analysis (stateless AA impl)
 ; GCN-O1-NEXT:        Function Alias Analysis Results
+; GCN-O1-NEXT:        Natural Loop Information
+; GCN-O1-NEXT:        Post-Dominator Tree Construction
 ; GCN-O1-NEXT:        Branch Probability Analysis
 ; GCN-O1-NEXT:        Assignment Tracking Analysis
 ; GCN-O1-NEXT:        Lazy Branch Probability Analysis
@@ -494,8 +492,8 @@
 ; GCN-O1-OPTS-NEXT:      Scalar Evolution Analysis
 ; GCN-O1-OPTS-NEXT:      Nary reassociation
 ; GCN-O1-OPTS-NEXT:      Early CSE
-; GCN-O1-OPTS-NEXT:      Post-Dominator Tree Construction
-; GCN-O1-OPTS-NEXT:      Legacy Divergence Analysis
+; GCN-O1-OPTS-NEXT:      Cycle Info Analysis
+; GCN-O1-OPTS-NEXT:      Uniformity Analysis
 ; GCN-O1-OPTS-NEXT:      AMDGPU IR optimizations
 ; GCN-O1-OPTS-NEXT:      Canonicalize natural loops
 ; GCN-O1-OPTS-NEXT:      Scalar Evolution Analysis
@@ -552,15 +550,14 @@
 ; GCN-O1-OPTS-NEXT:        Function Alias Analysis Results
 ; GCN-O1-OPTS-NEXT:        Flatten the CFG
 ; GCN-O1-OPTS-NEXT:        Dominator Tree Construction
-; GCN-O1-OPTS-NEXT:        Post-Dominator Tree Construction
-; GCN-O1-OPTS-NEXT:        Natural Loop Information
-; GCN-O1-OPTS-NEXT:        Legacy Divergence Analysis
+; GCN-O1-OPTS-NEXT:        Cycle Info Analysis
+; GCN-O1-OPTS-NEXT:        Uniformity Analysis
 ; GCN-O1-OPTS-NEXT:        AMDGPU IR late optimizations
 ; GCN-O1-OPTS-NEXT:        Basic Alias Analysis (stateless AA impl)
 ; GCN-O1-OPTS-NEXT:        Function Alias Analysis Results
+; GCN-O1-OPTS-NEXT:        Natural Loop Information
 ; GCN-O1-OPTS-NEXT:        Code sinking
-; GCN-O1-OPTS-NEXT:        Cycle Info Analysis
-; GCN-O1-OPTS-NEXT:        Uniformity Analysis
+; GCN-O1-OPTS-NEXT:        Post-Dominator Tree Construction
 ; GCN-O1-OPTS-NEXT:        Unify divergent function exit nodes
 ; GCN-O1-OPTS-NEXT:        Lazy Value Information Analysis
 ; GCN-O1-OPTS-NEXT:        Lower SwitchInst's to branches
@@ -581,8 +578,8 @@
 ; GCN-O1-OPTS-NEXT:        AMDGPU Annotate Uniform Values
 ; GCN-O1-OPTS-NEXT:        Natural Loop Information
 ; GCN-O1-OPTS-NEXT:        SI annotate control flow
-; GCN-O1-OPTS-NEXT:        Post-Dominator Tree Construction
-; GCN-O1-OPTS-NEXT:        Legacy Divergence Analysis
+; GCN-O1-OPTS-NEXT:        Cycle Info Analysis
+; GCN-O1-OPTS-NEXT:        Uniformity Analysis
 ; GCN-O1-OPTS-NEXT:        AMDGPU Rewrite Undef for PHI
 ; GCN-O1-OPTS-NEXT:        LCSSA Verifier
 ; GCN-O1-OPTS-NEXT:        Loop-Closed SSA Form Pass
@@ -592,11 +589,12 @@
 ; GCN-O1-OPTS-NEXT:        Safe Stack instrumentation pass
 ; GCN-O1-OPTS-NEXT:        Insert stack protectors
 ; GCN-O1-OPTS-NEXT:        Dominator Tree Construction
-; GCN-O1-OPTS-NEXT:        Post-Dominator Tree Construction
-; GCN-O1-OPTS-NEXT:        Natural Loop Information
-; GCN-O1-OPTS-NEXT:        Legacy Divergence Analysis
+; GCN-O1-OPTS-NEXT:        Cycle Info Analysis
+; GCN-O1-OPTS-NEXT:        Uniformity Analysis
 ; GCN-O1-OPTS-NEXT:        Basic Alias Analysis (stateless AA impl)
 ; GCN-O1-OPTS-NEXT:        Function Alias Analysis Results
+; GCN-O1-OPTS-NEXT:        Natural Loop Information
+; GCN-O1-OPTS-NEXT:        Post-Dominator Tree Construction
 ; GCN-O1-OPTS-NEXT:        Branch Probability Analysis
 ; GCN-O1-OPTS-NEXT:        Assignment Tracking Analysis
 ; GCN-O1-OPTS-NEXT:        Lazy Branch Probability Analysis
@@ -801,8 +799,8 @@
 ; GCN-O2-NEXT:      Scalar Evolution Analysis
 ; GCN-O2-NEXT:      Nary reassociation
 ; GCN-O2-NEXT:      Early CSE
-; GCN-O2-NEXT:      Post-Dominator Tree Construction
-; GCN-O2-NEXT:      Legacy Divergence Analysis
+; GCN-O2-NEXT:      Cycle Info Analysis
+; GCN-O2-NEXT:      Uniformity Analysis
 ; GCN-O2-NEXT:      AMDGPU IR optimizations
 ; GCN-O2-NEXT:      Canonicalize natural loops
 ; GCN-O2-NEXT:      Scalar Evolution Analysis
@@ -859,15 +857,14 @@
 ; GCN-O2-NEXT:        Function Alias Analysis Results
 ; GCN-O2-NEXT:        Flatten the CFG
 ; GCN-O2-NEXT:        Dominator Tree Construction
-; GCN-O2-NEXT:        Post-Dominator Tree Construction
-; GCN-O2-NEXT:        Natural Loop Information
-; GCN-O2-NEXT:        Legacy Divergence Analysis
+; GCN-O2-NEXT:        Cycle Info Analysis
+; GCN-O2-NEXT:        Uniformity Analysis
 ; GCN-O2-NEXT:        AMDGPU IR late optimizations
 ; GCN-O2-NEXT:        Basic Alias Analysis (stateless AA impl)
 ; GCN-O2-NEXT:        Function Alias Analysis Results
+; GCN-O2-NEXT:        Natural Loop Information
 ; GCN-O2-NEXT:        Code sinking
-; GCN-O2-NEXT:        Cycle Info Analysis
-; GCN-O2-NEXT:        Uniformity Analysis
+; GCN-O2-NEXT:        Post-Dominator Tree Construction
 ; GCN-O2-NEXT:        Unify divergent function exit nodes
 ; GCN-O2-NEXT:        Lazy Value Information Analysis
 ; GCN-O2-NEXT:        Lower SwitchInst's to branches
@@ -888,8 +885,8 @@
 ; GCN-O2-NEXT:        AMDGPU Annotate Uniform Values
 ; GCN-O2-NEXT:        Natural Loop Information
 ; GCN-O2-NEXT:        SI annotate control flow
-; GCN-O2-NEXT:        Post-Dominator Tree Construction
-; GCN-O2-NEXT:        Legacy Divergence Analysis
+; GCN-O2-NEXT:        Cycle Info Analysis
+; GCN-O2-NEXT:        Uniformity Analysis
 ; GCN-O2-NEXT:        AMDGPU Rewrite Undef for PHI
 ; GCN-O2-NEXT:        LCSSA Verifier
 ; GCN-O2-NEXT:        Loop-Closed SSA Form Pass
@@ -900,11 +897,12 @@
 ; GCN-O2-NEXT:        Safe Stack instrumentation pass
 ; GCN-O2-NEXT:        Insert stack protectors
 ; GCN-O2-NEXT:        Dominator Tree Construction
-; GCN-O2-NEXT:        Post-Dominator Tree Construction
-; GCN-O2-NEXT:        Natural Loop Information
-; GCN-O2-NEXT:        Legacy Divergence Analysis
+; GCN-O2-NEXT:        Cycle Info Analysis
+; GCN-O2-NEXT:        Uniformity Analysis
 ; GCN-O2-NEXT:        Basic Alias Analysis (stateless AA impl)
 ; GCN-O2-NEXT:        Function Alias Analysis Results
+; GCN-O2-NEXT:        Natural Loop Information
+; GCN-O2-NEXT:        Post-Dominator Tree Construction
 ; GCN-O2-NEXT:        Branch Probability Analysis
 ; GCN-O2-NEXT:        Assignment Tracking Analysis
 ; GCN-O2-NEXT:        Lazy Branch Probability Analysis
@@ -1114,8 +1112,8 @@
 ; GCN-O3-NEXT:      Scalar Evolution Analysis
 ; GCN-O3-NEXT:      Nary reassociation
 ; GCN-O3-NEXT:      Early CSE
-; GCN-O3-NEXT:      Post-Dominator Tree Construction
-; GCN-O3-NEXT:      Legacy Divergence Analysis
+; GCN-O3-NEXT:      Cycle Info Analysis
+; GCN-O3-NEXT:      Uniformity Analysis
 ; GCN-O3-NEXT:      AMDGPU IR optimizations
 ; GCN-O3-NEXT:      Basic Alias Analysis (stateless AA impl)
 ; GCN-O3-NEXT:      Canonicalize natural loops
@@ -1179,15 +1177,14 @@
 ; GCN-O3-NEXT:        Function Alias Analysis Results
 ; GCN-O3-NEXT:        Flatten the CFG
 ; GCN-O3-NEXT:        Dominator Tree Construction
-; GCN-O3-NEXT:        Post-Dominator Tree Construction
-; GCN-O3-NEXT:        Natural Loop Information
-; GCN-O3-NEXT:        Legacy Divergence Analysis
+; GCN-O3-NEXT:        Cycle Info Analysis
+; GCN-O3-NEXT:        Uniformity Analysis
 ; GCN-O3-NEXT:        AMDGPU IR late optimizations
 ; GCN-O3-NEXT:        Basic Alias Analysis (stateless AA impl)
 ; GCN-O3-NEXT:        Function Alias Analysis Results
+; GCN-O3-NEXT:        Natural Loop Information
 ; GCN-O3-NEXT:        Code sinking
-; GCN-O3-NEXT:        Cycle Info Analysis
-; GCN-O3-NEXT:        Uniformity Analysis
+; GCN-O3-NEXT:        Post-Dominator Tree Construction
 ; GCN-O3-NEXT:        Unify divergent function exit nodes
 ; GCN-O3-NEXT:        Lazy Value Information Analysis
 ; GCN-O3-NEXT:        Lower SwitchInst's to branches
@@ -1208,8 +1205,8 @@
 ; GCN-O3-NEXT:        AMDGPU Annotate Uniform Values
 ; GCN-O3-NEXT:        Natural Loop Information
 ; GCN-O3-NEXT:        SI annotate control flow
-; GCN-O3-NEXT:        Post-Dominator Tree Construction
-; GCN-O3-NEXT:        Legacy Divergence Analysis
+; GCN-O3-NEXT:        Cycle Info Analysis
+; GCN-O3-NEXT:        Uniformity Analysis
 ; GCN-O3-NEXT:        AMDGPU Rewrite Undef for PHI
 ; GCN-O3-NEXT:        LCSSA Verifier
 ; GCN-O3-NEXT:        Loop-Closed SSA Form Pass
@@ -1220,11 +1217,12 @@
 ; GCN-O3-NEXT:        Safe Stack instrumentation pass
 ; GCN-O3-NEXT:        Insert stack protectors
 ; GCN-O3-NEXT:        Dominator Tree Construction
-; GCN-O3-NEXT:        Post-Dominator Tree Construction
-; GCN-O3-NEXT:        Natural Loop Information
-; GCN-O3-NEXT:        Legacy Divergence Analysis
+; GCN-O3-NEXT:        Cycle Info Analysis
+; GCN-O3-NEXT:        Uniformity Analysis
 ; GCN-O3-NEXT:        Basic Alias Analysis (stateless AA impl)
 ; GCN-O3-NEXT:        Function Alias Analysis Results
+; GCN-O3-NEXT:        Natural Loop Information
+; GCN-O3-NEXT:        Post-Dominator Tree Construction
 ; GCN-O3-NEXT:        Branch Probability Analysis
 ; GCN-O3-NEXT:        Assignment Tracking Analysis
 ; GCN-O3-NEXT:        Lazy Branch Probability Analysis
diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.s.buffer.load.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.s.buffer.load.ll
index 1042290ce8362..b787aa364f1cb 100644
--- a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.s.buffer.load.ll
+++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.s.buffer.load.ll
@@ -1,13 +1,35 @@
-; RUN: llc < %s -march=amdgcn -mcpu=tahiti -verify-machineinstrs | FileCheck %s -check-prefixes=GCN,SI,SICI
-; RUN: llc < %s -march=amdgcn -mcpu=hawaii -verify-machineinstrs | FileCheck %s -check-prefixes=GCN,CI,SICI
-; RUN: llc < %s -march=amdgcn -mcpu=tonga -verify-machineinstrs | FileCheck %s -check-prefixes=GCN,VI
-
-;GCN-LABEL: {{^}}s_buffer_load_imm:
-;GCN-NOT: s_waitcnt;
-;SI: s_buffer_load_dword s{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0x1
-;CI: s_buffer_load_dword s{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0x1
-;VI: s_buffer_load_dword s{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0x4
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -march=amdgcn -mcpu=tahiti -verify-machineinstrs | FileCheck %s -check-prefixes=GFX678910,GFX6789,GFX678,GFX67,GFX6
+; RUN: llc < %s -march=amdgcn -mcpu=hawaii -verify-machineinstrs | FileCheck %s -check-prefixes=GFX678910,GFX6789,GFX78910,GFX678,GFX789,GFX67,GFX78,GFX7
+; RUN: llc < %s -march=amdgcn -mcpu=tonga -verify-machineinstrs | FileCheck %s -check-prefixes=GFX678910,GFX6789,GFX78910,GFX678,GFX789,GFX8910,GFX78,GFX89,GFX8
+; RUN: llc < %s -march=amdgcn -mcpu=gfx900 -verify-machineinstrs | FileCheck %s -check-prefixes=GFX678910,GFX6789,GFX78910,GFX789,GFX8910,GFX89,GFX910,GFX9
+; RUN: llc < %s -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs | FileCheck %s -check-prefixes=GFX678910,GFX78910,GFX8910,GFX910,GFX10
+; RUN: llc < %s -march=amdgcn -mcpu=gfx1100 -amdgpu-enable-vopd=0 -verify-machineinstrs | FileCheck %s -check-prefixes=GFX11
+
 define amdgpu_ps void @s_buffer_load_imm(<4 x i32> inreg %desc) {
+; GFX67-LABEL: s_buffer_load_imm:
+; GFX67:       ; %bb.0: ; %main_body
+; GFX67-NEXT:    s_buffer_load_dword s0, s[0:3], 0x1
+; GFX67-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX67-NEXT:    v_mov_b32_e32 v0, s0
+; GFX67-NEXT:    exp mrt0 v0, v0, v0, v0 done vm
+; GFX67-NEXT:    s_endpgm
+;
+; GFX8910-LABEL: s_buffer_load_imm:
+; GFX8910:       ; %bb.0: ; %main_body
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], 0x4
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    v_mov_b32_e32 v0, s0
+; GFX8910-NEXT:    exp mrt0 v0, v0, v0, v0 done vm
+; GFX8910-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_buffer_load_imm:
+; GFX11:       ; %bb.0: ; %main_body
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], 0x4
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s0
+; GFX11-NEXT:    exp mrt0 v0, v0, v0, v0 done
+; GFX11-NEXT:    s_endpgm
 main_body:
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 4, i32 0)
   %bitcast = bitcast i32 %load to float
@@ -15,10 +37,22 @@ main_body:
   ret void
 }
 
-;GCN-LABEL: {{^}}s_buffer_load_index:
-;GCN-NOT: s_waitcnt;
-;GCN: s_buffer_load_dword s{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], s{{[0-9]+}}
 define amdgpu_ps void @s_buffer_load_index(<4 x i32> inreg %desc, i32 inreg %index) {
+; GFX678910-LABEL: s_buffer_load_index:
+; GFX678910:       ; %bb.0: ; %main_body
+; GFX678910-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX678910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX678910-NEXT:    v_mov_b32_e32 v0, s0
+; GFX678910-NEXT:    exp mrt0 v0, v0, v0, v0 done vm
+; GFX678910-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_buffer_load_index:
+; GFX11:       ; %bb.0: ; %main_body
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], s4
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s0
+; GFX11-NEXT:    exp mrt0 v0, v0, v0, v0 done
+; GFX11-NEXT:    s_endpgm
 main_body:
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 %index, i32 0)
   %bitcast = bitcast i32 %load to float
@@ -26,10 +60,20 @@ main_body:
   ret void
 }
 
-;GCN-LABEL: {{^}}s_buffer_load_index_divergent:
-;GCN-NOT: s_waitcnt;
-;GCN: buffer_load_dword v{{[0-9]+}}, v{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0 offen
 define amdgpu_ps void @s_buffer_load_index_divergent(<4 x i32> inreg %desc, i32 %index) {
+; GFX678910-LABEL: s_buffer_load_index_divergent:
+; GFX678910:       ; %bb.0: ; %main_body
+; GFX678910-NEXT:    buffer_load_dword v0, v0, s[0:3], 0 offen
+; GFX678910-NEXT:    s_waitcnt vmcnt(0)
+; GFX678910-NEXT:    exp mrt0 v0, v0, v0, v0 done vm
+; GFX678910-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_buffer_load_index_divergent:
+; GFX11:       ; %bb.0: ; %main_body
+; GFX11-NEXT:    buffer_load_b32 v0, v0, s[0:3], 0 offen
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    exp mrt0 v0, v0, v0, v0 done
+; GFX11-NEXT:    s_endpgm
 main_body:
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 %index, i32 0)
   %bitcast = bitcast i32 %load to float
@@ -37,12 +81,33 @@ main_body:
   ret void
 }
 
-;GCN-LABEL: {{^}}s_buffer_loadx2_imm:
-;GCN-NOT: s_waitcnt;
-;SI: s_buffer_load_dwordx2 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0x10
-;CI: s_buffer_load_dwordx2 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0x10
-;VI: s_buffer_load_dwordx2 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0x40
 define amdgpu_ps void @s_buffer_loadx2_imm(<4 x i32> inreg %desc) {
+; GFX67-LABEL: s_buffer_loadx2_imm:
+; GFX67:       ; %bb.0: ; %main_body
+; GFX67-NEXT:    s_buffer_load_dwordx2 s[0:1], s[0:3], 0x10
+; GFX67-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX67-NEXT:    v_mov_b32_e32 v0, s0
+; GFX67-NEXT:    v_mov_b32_e32 v1, s1
+; GFX67-NEXT:    exp mrt0 v0, v1, v0, v0 done vm
+; GFX67-NEXT:    s_endpgm
+;
+; GFX8910-LABEL: s_buffer_loadx2_imm:
+; GFX8910:       ; %bb.0: ; %main_body
+; GFX8910-NEXT:    s_buffer_load_dwordx2 s[0:1], s[0:3], 0x40
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    v_mov_b32_e32 v0, s0
+; GFX8910-NEXT:    v_mov_b32_e32 v1, s1
+; GFX8910-NEXT:    exp mrt0 v0, v1, v0, v0 done vm
+; GFX8910-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_buffer_loadx2_imm:
+; GFX11:       ; %bb.0: ; %main_body
+; GFX11-NEXT:    s_buffer_load_b64 s[0:1], s[0:3], 0x40
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s0
+; GFX11-NEXT:    v_mov_b32_e32 v1, s1
+; GFX11-NEXT:    exp mrt0 v0, v1, v0, v0 done
+; GFX11-NEXT:    s_endpgm
 main_body:
   %load = call <2 x i32> @llvm.amdgcn.s.buffer.load.v2i32(<4 x i32> %desc, i32 64, i32 0)
   %bitcast = bitcast <2 x i32> %load to <2 x float>
@@ -52,10 +117,24 @@ main_body:
   ret void
 }
 
-;GCN-LABEL: {{^}}s_buffer_loadx2_index:
-;GCN-NOT: s_waitcnt;
-;GCN: s_buffer_load_dwordx2 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], s{{[0-9]+}}
 define amdgpu_ps void @s_buffer_loadx2_index(<4 x i32> inreg %desc, i32 inreg %index) {
+; GFX678910-LABEL: s_buffer_loadx2_index:
+; GFX678910:       ; %bb.0: ; %main_body
+; GFX678910-NEXT:    s_buffer_load_dwordx2 s[0:1], s[0:3], s4
+; GFX678910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX678910-NEXT:    v_mov_b32_e32 v0, s0
+; GFX678910-NEXT:    v_mov_b32_e32 v1, s1
+; GFX678910-NEXT:    exp mrt0 v0, v1, v0, v0 done vm
+; GFX678910-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_buffer_loadx2_index:
+; GFX11:       ; %bb.0: ; %main_body
+; GFX11-NEXT:    s_buffer_load_b64 s[0:1], s[0:3], s4
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s0
+; GFX11-NEXT:    v_mov_b32_e32 v1, s1
+; GFX11-NEXT:    exp mrt0 v0, v1, v0, v0 done
+; GFX11-NEXT:    s_endpgm
 main_body:
   %load = call <2 x i32> @llvm.amdgcn.s.buffer.load.v2i32(<4 x i32> %desc, i32 %index, i32 0)
   %bitcast = bitcast <2 x i32> %load to <2 x float>
@@ -65,10 +144,20 @@ main_body:
   ret void
 }
 
-;GCN-LABEL: {{^}}s_buffer_loadx2_index_divergent:
-;GCN-NOT: s_waitcnt;
-;GCN: buffer_load_dwordx2 v[{{[0-9]+:[0-9]+}}], v{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0 offen
 define amdgpu_ps void @s_buffer_loadx2_index_divergent(<4 x i32> inreg %desc, i32 %index) {
+; GFX678910-LABEL: s_buffer_loadx2_index_divergent:
+; GFX678910:       ; %bb.0: ; %main_body
+; GFX678910-NEXT:    buffer_load_dwordx2 v[0:1], v0, s[0:3], 0 offen
+; GFX678910-NEXT:    s_waitcnt vmcnt(0)
+; GFX678910-NEXT:    exp mrt0 v0, v1, v0, v0 done vm
+; GFX678910-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_buffer_loadx2_index_divergent:
+; GFX11:       ; %bb.0: ; %main_body
+; GFX11-NEXT:    buffer_load_b64 v[0:1], v0, s[0:3], 0 offen
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    exp mrt0 v0, v1, v0, v0 done
+; GFX11-NEXT:    s_endpgm
 main_body:
   %load = call <2 x i32> @llvm.amdgcn.s.buffer.load.v2i32(<4 x i32> %desc, i32 %index, i32 0)
   %bitcast = bitcast <2 x i32> %load to <2 x float>
@@ -78,12 +167,36 @@ main_body:
   ret void
 }
 
-;GCN-LABEL: {{^}}s_buffer_loadx3_imm:
-;GCN-NOT: s_waitcnt;
-;SI: s_buffer_load_dwordx4 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0x10
-;CI: s_buffer_load_dwordx4 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0x10
-;VI: s_buffer_load_dwordx4 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0x40
 define amdgpu_ps void @s_buffer_loadx3_imm(<4 x i32> inreg %desc) {
+; GFX67-LABEL: s_buffer_loadx3_imm:
+; GFX67:       ; %bb.0: ; %main_body
+; GFX67-NEXT:    s_buffer_load_dwordx4 s[0:3], s[0:3], 0x10
+; GFX67-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX67-NEXT:    v_mov_b32_e32 v0, s0
+; GFX67-NEXT:    v_mov_b32_e32 v1, s1
+; GFX67-NEXT:    v_mov_b32_e32 v2, s2
+; GFX67-NEXT:    exp mrt0 v0, v1, v2, v0 done vm
+; GFX67-NEXT:    s_endpgm
+;
+; GFX8910-LABEL: s_buffer_loadx3_imm:
+; GFX8910:       ; %bb.0: ; %main_body
+; GFX8910-NEXT:    s_buffer_load_dwordx4 s[0:3], s[0:3], 0x40
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    v_mov_b32_e32 v0, s0
+; GFX8910-NEXT:    v_mov_b32_e32 v1, s1
+; GFX8910-NEXT:    v_mov_b32_e32 v2, s2
+; GFX8910-NEXT:    exp mrt0 v0, v1, v2, v0 done vm
+; GFX8910-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_buffer_loadx3_imm:
+; GFX11:       ; %bb.0: ; %main_body
+; GFX11-NEXT:    s_buffer_load_b128 s[0:3], s[0:3], 0x40
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s0
+; GFX11-NEXT:    v_mov_b32_e32 v1, s1
+; GFX11-NEXT:    v_mov_b32_e32 v2, s2
+; GFX11-NEXT:    exp mrt0 v0, v1, v2, v0 done
+; GFX11-NEXT:    s_endpgm
 main_body:
   %load = call <3 x i32> @llvm.amdgcn.s.buffer.load.v3i32(<4 x i32> %desc, i32 64, i32 0)
   %bitcast = bitcast <3 x i32> %load to <3 x float>
@@ -94,10 +207,26 @@ main_body:
   ret void
 }
 
-;GCN-LABEL: {{^}}s_buffer_loadx3_index:
-;GCN-NOT: s_waitcnt;
-;GCN: s_buffer_load_dwordx4 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], s{{[0-9]+}}
 define amdgpu_ps void @s_buffer_loadx3_index(<4 x i32> inreg %desc, i32 inreg %index) {
+; GFX678910-LABEL: s_buffer_loadx3_index:
+; GFX678910:       ; %bb.0: ; %main_body
+; GFX678910-NEXT:    s_buffer_load_dwordx4 s[0:3], s[0:3], s4
+; GFX678910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX678910-NEXT:    v_mov_b32_e32 v0, s0
+; GFX678910-NEXT:    v_mov_b32_e32 v1, s1
+; GFX678910-NEXT:    v_mov_b32_e32 v2, s2
+; GFX678910-NEXT:    exp mrt0 v0, v1, v2, v0 done vm
+; GFX678910-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_buffer_loadx3_index:
+; GFX11:       ; %bb.0: ; %main_body
+; GFX11-NEXT:    s_buffer_load_b128 s[0:3], s[0:3], s4
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s0
+; GFX11-NEXT:    v_mov_b32_e32 v1, s1
+; GFX11-NEXT:    v_mov_b32_e32 v2, s2
+; GFX11-NEXT:    exp mrt0 v0, v1, v2, v0 done
+; GFX11-NEXT:    s_endpgm
 main_body:
   %load = call <3 x i32> @llvm.amdgcn.s.buffer.load.v3i32(<4 x i32> %desc, i32 %index, i32 0)
   %bitcast = bitcast <3 x i32> %load to <3 x float>
@@ -108,12 +237,27 @@ main_body:
   ret void
 }
 
-;GCN-LABEL: {{^}}s_buffer_loadx3_index_divergent:
-;GCN-NOT: s_waitcnt;
-;SI: buffer_load_dwordx4 v[{{[0-9]+:[0-9]+}}], v{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0 offen
-;CI: buffer_load_dwordx3 v[{{[0-9]+:[0-9]+}}], v{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0 offen
-;VI: buffer_load_dwordx3 v[{{[0-9]+:[0-9]+}}], v{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0 offen
 define amdgpu_ps void @s_buffer_loadx3_index_divergent(<4 x i32> inreg %desc, i32 %index) {
+; GFX6-LABEL: s_buffer_loadx3_index_divergent:
+; GFX6:       ; %bb.0: ; %main_body
+; GFX6-NEXT:    buffer_load_dwordx4 v[0:3], v0, s[0:3], 0 offen
+; GFX6-NEXT:    s_waitcnt vmcnt(0)
+; GFX6-NEXT:    exp mrt0 v0, v1, v2, v0 done vm
+; GFX6-NEXT:    s_endpgm
+;
+; GFX78910-LABEL: s_buffer_loadx3_index_divergent:
+; GFX78910:       ; %bb.0: ; %main_body
+; GFX78910-NEXT:    buffer_load_dwordx3 v[0:2], v0, s[0:3], 0 offen
+; GFX78910-NEXT:    s_waitcnt vmcnt(0)
+; GFX78910-NEXT:    exp mrt0 v0, v1, v2, v0 done vm
+; GFX78910-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_buffer_loadx3_index_divergent:
+; GFX11:       ; %bb.0: ; %main_body
+; GFX11-NEXT:    buffer_load_b96 v[0:2], v0, s[0:3], 0 offen
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    exp mrt0 v0, v1, v2, v0 done
+; GFX11-NEXT:    s_endpgm
 main_body:
   %load = call <3 x i32> @llvm.amdgcn.s.buffer.load.v3i32(<4 x i32> %desc, i32 %index, i32 0)
   %bitcast = bitcast <3 x i32> %load to <3 x float>
@@ -124,12 +268,39 @@ main_body:
   ret void
 }
 
-;GCN-LABEL: {{^}}s_buffer_loadx4_imm:
-;GCN-NOT: s_waitcnt;
-;SI: s_buffer_load_dwordx4 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0x32
-;CI: s_buffer_load_dwordx4 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0x32
-;VI: s_buffer_load_dwordx4 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0xc8
 define amdgpu_ps void @s_buffer_loadx4_imm(<4 x i32> inreg %desc) {
+; GFX67-LABEL: s_buffer_loadx4_imm:
+; GFX67:       ; %bb.0: ; %main_body
+; GFX67-NEXT:    s_buffer_load_dwordx4 s[0:3], s[0:3], 0x32
+; GFX67-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX67-NEXT:    v_mov_b32_e32 v0, s0
+; GFX67-NEXT:    v_mov_b32_e32 v1, s1
+; GFX67-NEXT:    v_mov_b32_e32 v2, s2
+; GFX67-NEXT:    v_mov_b32_e32 v3, s3
+; GFX67-NEXT:    exp mrt0 v0, v1, v2, v3 done vm
+; GFX67-NEXT:    s_endpgm
+;
+; GFX8910-LABEL: s_buffer_loadx4_imm:
+; GFX8910:       ; %bb.0: ; %main_body
+; GFX8910-NEXT:    s_buffer_load_dwordx4 s[0:3], s[0:3], 0xc8
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    v_mov_b32_e32 v0, s0
+; GFX8910-NEXT:    v_mov_b32_e32 v1, s1
+; GFX8910-NEXT:    v_mov_b32_e32 v2, s2
+; GFX8910-NEXT:    v_mov_b32_e32 v3, s3
+; GFX8910-NEXT:    exp mrt0 v0, v1, v2, v3 done vm
+; GFX8910-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_buffer_loadx4_imm:
+; GFX11:       ; %bb.0: ; %main_body
+; GFX11-NEXT:    s_buffer_load_b128 s[0:3], s[0:3], 0xc8
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s0
+; GFX11-NEXT:    v_mov_b32_e32 v1, s1
+; GFX11-NEXT:    v_mov_b32_e32 v2, s2
+; GFX11-NEXT:    v_mov_b32_e32 v3, s3
+; GFX11-NEXT:    exp mrt0 v0, v1, v2, v3 done
+; GFX11-NEXT:    s_endpgm
 main_body:
   %load = call <4 x i32> @llvm.amdgcn.s.buffer.load.v4i32(<4 x i32> %desc, i32 200, i32 0)
   %bitcast = bitcast <4 x i32> %load to <4 x float>
@@ -141,10 +312,28 @@ main_body:
   ret void
 }
 
-;GCN-LABEL: {{^}}s_buffer_loadx4_index:
-;GCN-NOT: s_waitcnt;
-;GCN: buffer_load_dwordx4 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], s{{[0-9]+}}
 define amdgpu_ps void @s_buffer_loadx4_index(<4 x i32> inreg %desc, i32 inreg %index) {
+; GFX678910-LABEL: s_buffer_loadx4_index:
+; GFX678910:       ; %bb.0: ; %main_body
+; GFX678910-NEXT:    s_buffer_load_dwordx4 s[0:3], s[0:3], s4
+; GFX678910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX678910-NEXT:    v_mov_b32_e32 v0, s0
+; GFX678910-NEXT:    v_mov_b32_e32 v1, s1
+; GFX678910-NEXT:    v_mov_b32_e32 v2, s2
+; GFX678910-NEXT:    v_mov_b32_e32 v3, s3
+; GFX678910-NEXT:    exp mrt0 v0, v1, v2, v3 done vm
+; GFX678910-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_buffer_loadx4_index:
+; GFX11:       ; %bb.0: ; %main_body
+; GFX11-NEXT:    s_buffer_load_b128 s[0:3], s[0:3], s4
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s0
+; GFX11-NEXT:    v_mov_b32_e32 v1, s1
+; GFX11-NEXT:    v_mov_b32_e32 v2, s2
+; GFX11-NEXT:    v_mov_b32_e32 v3, s3
+; GFX11-NEXT:    exp mrt0 v0, v1, v2, v3 done
+; GFX11-NEXT:    s_endpgm
 main_body:
   %load = call <4 x i32> @llvm.amdgcn.s.buffer.load.v4i32(<4 x i32> %desc, i32 %index, i32 0)
   %bitcast = bitcast <4 x i32> %load to <4 x float>
@@ -156,10 +345,20 @@ main_body:
   ret void
 }
 
-;GCN-LABEL: {{^}}s_buffer_loadx4_index_divergent:
-;GCN-NOT: s_waitcnt;
-;GCN: buffer_load_dwordx4 v[{{[0-9]+:[0-9]+}}], v{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0 offen
 define amdgpu_ps void @s_buffer_loadx4_index_divergent(<4 x i32> inreg %desc, i32 %index) {
+; GFX678910-LABEL: s_buffer_loadx4_index_divergent:
+; GFX678910:       ; %bb.0: ; %main_body
+; GFX678910-NEXT:    buffer_load_dwordx4 v[0:3], v0, s[0:3], 0 offen
+; GFX678910-NEXT:    s_waitcnt vmcnt(0)
+; GFX678910-NEXT:    exp mrt0 v0, v1, v2, v3 done vm
+; GFX678910-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_buffer_loadx4_index_divergent:
+; GFX11:       ; %bb.0: ; %main_body
+; GFX11-NEXT:    buffer_load_b128 v[0:3], v0, s[0:3], 0 offen
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    exp mrt0 v0, v1, v2, v3 done
+; GFX11-NEXT:    s_endpgm
 main_body:
   %load = call <4 x i32> @llvm.amdgcn.s.buffer.load.v4i32(<4 x i32> %desc, i32 %index, i32 0)
   %bitcast = bitcast <4 x i32> %load to <4 x float>
@@ -171,12 +370,33 @@ main_body:
   ret void
 }
 
-;GCN-LABEL: {{^}}s_buffer_load_imm_mergex2:
-;GCN-NOT: s_waitcnt;
-;SI: s_buffer_load_dwordx2 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0x1
-;CI: s_buffer_load_dwordx2 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0x1
-;VI: s_buffer_load_dwordx2 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0x4
 define amdgpu_ps void @s_buffer_load_imm_mergex2(<4 x i32> inreg %desc) {
+; GFX67-LABEL: s_buffer_load_imm_mergex2:
+; GFX67:       ; %bb.0: ; %main_body
+; GFX67-NEXT:    s_buffer_load_dwordx2 s[0:1], s[0:3], 0x1
+; GFX67-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX67-NEXT:    v_mov_b32_e32 v0, s0
+; GFX67-NEXT:    v_mov_b32_e32 v1, s1
+; GFX67-NEXT:    exp mrt0 v0, v1, v0, v0 done vm
+; GFX67-NEXT:    s_endpgm
+;
+; GFX8910-LABEL: s_buffer_load_imm_mergex2:
+; GFX8910:       ; %bb.0: ; %main_body
+; GFX8910-NEXT:    s_buffer_load_dwordx2 s[0:1], s[0:3], 0x4
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    v_mov_b32_e32 v0, s0
+; GFX8910-NEXT:    v_mov_b32_e32 v1, s1
+; GFX8910-NEXT:    exp mrt0 v0, v1, v0, v0 done vm
+; GFX8910-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_buffer_load_imm_mergex2:
+; GFX11:       ; %bb.0: ; %main_body
+; GFX11-NEXT:    s_buffer_load_b64 s[0:1], s[0:3], 0x4
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s0
+; GFX11-NEXT:    v_mov_b32_e32 v1, s1
+; GFX11-NEXT:    exp mrt0 v0, v1, v0, v0 done
+; GFX11-NEXT:    s_endpgm
 main_body:
   %load0 = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 4, i32 0)
   %load1 = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 8, i32 0)
@@ -186,12 +406,39 @@ main_body:
   ret void
 }
 
-;GCN-LABEL: {{^}}s_buffer_load_imm_mergex4:
-;GCN-NOT: s_waitcnt;
-;SI: s_buffer_load_dwordx4 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0x2
-;CI: s_buffer_load_dwordx4 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0x2
-;VI: s_buffer_load_dwordx4 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0x8
 define amdgpu_ps void @s_buffer_load_imm_mergex4(<4 x i32> inreg %desc) {
+; GFX67-LABEL: s_buffer_load_imm_mergex4:
+; GFX67:       ; %bb.0: ; %main_body
+; GFX67-NEXT:    s_buffer_load_dwordx4 s[0:3], s[0:3], 0x2
+; GFX67-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX67-NEXT:    v_mov_b32_e32 v0, s0
+; GFX67-NEXT:    v_mov_b32_e32 v1, s1
+; GFX67-NEXT:    v_mov_b32_e32 v2, s2
+; GFX67-NEXT:    v_mov_b32_e32 v3, s3
+; GFX67-NEXT:    exp mrt0 v0, v1, v2, v3 done vm
+; GFX67-NEXT:    s_endpgm
+;
+; GFX8910-LABEL: s_buffer_load_imm_mergex4:
+; GFX8910:       ; %bb.0: ; %main_body
+; GFX8910-NEXT:    s_buffer_load_dwordx4 s[0:3], s[0:3], 0x8
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    v_mov_b32_e32 v0, s0
+; GFX8910-NEXT:    v_mov_b32_e32 v1, s1
+; GFX8910-NEXT:    v_mov_b32_e32 v2, s2
+; GFX8910-NEXT:    v_mov_b32_e32 v3, s3
+; GFX8910-NEXT:    exp mrt0 v0, v1, v2, v3 done vm
+; GFX8910-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_buffer_load_imm_mergex4:
+; GFX11:       ; %bb.0: ; %main_body
+; GFX11-NEXT:    s_buffer_load_b128 s[0:3], s[0:3], 0x8
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s0
+; GFX11-NEXT:    v_mov_b32_e32 v1, s1
+; GFX11-NEXT:    v_mov_b32_e32 v2, s2
+; GFX11-NEXT:    v_mov_b32_e32 v3, s3
+; GFX11-NEXT:    exp mrt0 v0, v1, v2, v3 done
+; GFX11-NEXT:    s_endpgm
 main_body:
   %load0 = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 8, i32 0)
   %load1 = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 12, i32 0)
@@ -207,11 +454,106 @@ main_body:
 
 @gv = external addrspace(1) global i32
 
-;GCN-LABEL: {{^}}s_buffer_load_index_across_bb:
-;GCN-NOT: s_waitcnt;
-;GCN: v_or_b32
-;GCN: buffer_load_dword v{{[0-9]+}}, v{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0 offen
 define amdgpu_ps void @s_buffer_load_index_across_bb(<4 x i32> inreg %desc, i32 %index) {
+; GFX6-LABEL: s_buffer_load_index_across_bb:
+; GFX6:       ; %bb.0: ; %main_body
+; GFX6-NEXT:    s_getpc_b64 s[4:5]
+; GFX6-NEXT:    s_add_u32 s4, s4, gv@gotpcrel32@lo+4
+; GFX6-NEXT:    s_addc_u32 s5, s5, gv@gotpcrel32@hi+12
+; GFX6-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x0
+; GFX6-NEXT:    v_lshlrev_b32_e32 v0, 4, v0
+; GFX6-NEXT:    s_mov_b32 s7, 0xf000
+; GFX6-NEXT:    s_mov_b32 s6, -1
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
+; GFX6-NEXT:    s_waitcnt expcnt(0)
+; GFX6-NEXT:    v_or_b32_e32 v0, 8, v0
+; GFX6-NEXT:    buffer_load_dword v0, v0, s[0:3], 0 offen
+; GFX6-NEXT:    s_waitcnt vmcnt(0)
+; GFX6-NEXT:    exp mrt0 v0, v0, v0, v0 done vm
+; GFX6-NEXT:    s_endpgm
+;
+; GFX7-LABEL: s_buffer_load_index_across_bb:
+; GFX7:       ; %bb.0: ; %main_body
+; GFX7-NEXT:    s_getpc_b64 s[4:5]
+; GFX7-NEXT:    s_add_u32 s4, s4, gv@gotpcrel32@lo+4
+; GFX7-NEXT:    s_addc_u32 s5, s5, gv@gotpcrel32@hi+12
+; GFX7-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x0
+; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 4, v0
+; GFX7-NEXT:    s_mov_b32 s7, 0xf000
+; GFX7-NEXT:    s_mov_b32 s6, -1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    buffer_store_dword v0, off, s[4:7], 0
+; GFX7-NEXT:    v_or_b32_e32 v0, 8, v0
+; GFX7-NEXT:    buffer_load_dword v0, v0, s[0:3], 0 offen
+; GFX7-NEXT:    s_waitcnt vmcnt(0)
+; GFX7-NEXT:    exp mrt0 v0, v0, v0, v0 done vm
+; GFX7-NEXT:    s_endpgm
+;
+; GFX8-LABEL: s_buffer_load_index_across_bb:
+; GFX8:       ; %bb.0: ; %main_body
+; GFX8-NEXT:    s_getpc_b64 s[4:5]
+; GFX8-NEXT:    s_add_u32 s4, s4, gv@gotpcrel32@lo+4
+; GFX8-NEXT:    s_addc_u32 s5, s5, gv@gotpcrel32@hi+12
+; GFX8-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x0
+; GFX8-NEXT:    v_lshlrev_b32_e32 v0, 4, v0
+; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8-NEXT:    v_mov_b32_e32 v1, s4
+; GFX8-NEXT:    v_mov_b32_e32 v2, s5
+; GFX8-NEXT:    flat_store_dword v[1:2], v0
+; GFX8-NEXT:    v_or_b32_e32 v0, 8, v0
+; GFX8-NEXT:    buffer_load_dword v0, v0, s[0:3], 0 offen
+; GFX8-NEXT:    s_waitcnt vmcnt(0)
+; GFX8-NEXT:    exp mrt0 v0, v0, v0, v0 done vm
+; GFX8-NEXT:    s_endpgm
+;
+; GFX9-LABEL: s_buffer_load_index_across_bb:
+; GFX9:       ; %bb.0: ; %main_body
+; GFX9-NEXT:    s_getpc_b64 s[4:5]
+; GFX9-NEXT:    s_add_u32 s4, s4, gv@gotpcrel32@lo+4
+; GFX9-NEXT:    s_addc_u32 s5, s5, gv@gotpcrel32@hi+12
+; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x0
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v1, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_store_dword v1, v0, s[4:5]
+; GFX9-NEXT:    v_or_b32_e32 v0, 8, v0
+; GFX9-NEXT:    buffer_load_dword v0, v0, s[0:3], 0 offen
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    exp mrt0 v0, v0, v0, v0 done vm
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: s_buffer_load_index_across_bb:
+; GFX10:       ; %bb.0: ; %main_body
+; GFX10-NEXT:    s_getpc_b64 s[4:5]
+; GFX10-NEXT:    s_add_u32 s4, s4, gv@gotpcrel32@lo+4
+; GFX10-NEXT:    s_addc_u32 s5, s5, gv@gotpcrel32@hi+12
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 4, v0
+; GFX10-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x0
+; GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    global_store_dword v1, v0, s[4:5]
+; GFX10-NEXT:    v_or_b32_e32 v0, 8, v0
+; GFX10-NEXT:    buffer_load_dword v0, v0, s[0:3], 0 offen
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    exp mrt0 v0, v0, v0, v0 done vm
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_buffer_load_index_across_bb:
+; GFX11:       ; %bb.0: ; %main_body
+; GFX11-NEXT:    s_getpc_b64 s[4:5]
+; GFX11-NEXT:    s_add_u32 s4, s4, gv@gotpcrel32@lo+4
+; GFX11-NEXT:    s_addc_u32 s5, s5, gv@gotpcrel32@hi+12
+; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 4, v0
+; GFX11-NEXT:    s_load_b64 s[4:5], s[4:5], 0x0
+; GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    global_store_b32 v1, v0, s[4:5]
+; GFX11-NEXT:    v_or_b32_e32 v0, 8, v0
+; GFX11-NEXT:    buffer_load_b32 v0, v0, s[0:3], 0 offen
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    exp mrt0 v0, v0, v0, v0 done
+; GFX11-NEXT:    s_endpgm
 main_body:
   %tmp = shl i32 %index, 4
   store i32 %tmp, ptr addrspace(1) @gv
@@ -225,10 +567,22 @@ bb1:                                              ; preds = %main_body
   ret void
 }
 
-;GCN-LABEL: {{^}}s_buffer_load_index_across_bb_merged:
-;GCN-NOT: s_waitcnt;
-;GCN: buffer_load_dwordx2 v[{{[0-9]+:[0-9]+}}], v{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0 offen offset:8
 define amdgpu_ps void @s_buffer_load_index_across_bb_merged(<4 x i32> inreg %desc, i32 %index) {
+; GFX678910-LABEL: s_buffer_load_index_across_bb_merged:
+; GFX678910:       ; %bb.0: ; %main_body
+; GFX678910-NEXT:    v_lshlrev_b32_e32 v0, 4, v0
+; GFX678910-NEXT:    buffer_load_dwordx2 v[0:1], v0, s[0:3], 0 offen offset:8
+; GFX678910-NEXT:    s_waitcnt vmcnt(0)
+; GFX678910-NEXT:    exp mrt0 v0, v1, v0, v0 done vm
+; GFX678910-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_buffer_load_index_across_bb_merged:
+; GFX11:       ; %bb.0: ; %main_body
+; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 4, v0
+; GFX11-NEXT:    buffer_load_b64 v[0:1], v0, s[0:3], 0 offen offset:8
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    exp mrt0 v0, v1, v0, v0 done
+; GFX11-NEXT:    s_endpgm
 main_body:
   %tmp = shl i32 %index, 4
   br label %bb1
@@ -244,211 +598,565 @@ bb1:                                              ; preds = %main_body
   ret void
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_neg1:
-; GCN: s_mov_b32 [[K:s[0-9]+]], -1{{$}}
-; GCN: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_neg1(<4 x i32> inreg %desc) {
+; GFX6-LABEL: s_buffer_load_imm_neg1:
+; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_mov_b32 s4, -1
+; GFX6-NEXT:    s_nop 3
+; GFX6-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    ; return to shader part epilog
+;
+; GFX78910-LABEL: s_buffer_load_imm_neg1:
+; GFX78910:       ; %bb.0:
+; GFX78910-NEXT:    s_mov_b32 s4, -1
+; GFX78910-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX78910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX78910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_neg1:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_mov_b32 s4, -1
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], s4
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 -1, i32 0)
   ret i32 %load
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_neg4:
-; SI: s_mov_b32 [[K:s[0-9]+]], -4{{$}}
-; SI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
-
-; CI: s_buffer_load_dword s0, s[0:3], 0x3fffffff{{$}}
-
-; VI: s_mov_b32 [[K:s[0-9]+]], -4{{$}}
-; VI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_neg4(<4 x i32> inreg %desc) {
+; GFX6-LABEL: s_buffer_load_imm_neg4:
+; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_mov_b32 s4, -4
+; GFX6-NEXT:    s_nop 3
+; GFX6-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: s_buffer_load_imm_neg4:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_buffer_load_dword s0, s[0:3], 0x3fffffff
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    ; return to shader part epilog
+;
+; GFX8910-LABEL: s_buffer_load_imm_neg4:
+; GFX8910:       ; %bb.0:
+; GFX8910-NEXT:    s_mov_b32 s4, -4
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_neg4:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_mov_b32 s4, -4
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], s4
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 -4, i32 0)
   ret i32 %load
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_neg8:
-; SI: s_mov_b32 [[K:s[0-9]+]], -8{{$}}
-; SI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
-
-; CI: s_buffer_load_dword s0, s[0:3], 0x3ffffffe{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_neg8(<4 x i32> inreg %desc) {
+; GFX6-LABEL: s_buffer_load_imm_neg8:
+; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_mov_b32 s4, -8
+; GFX6-NEXT:    s_nop 3
+; GFX6-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: s_buffer_load_imm_neg8:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_buffer_load_dword s0, s[0:3], 0x3ffffffe
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    ; return to shader part epilog
+;
+; GFX8910-LABEL: s_buffer_load_imm_neg8:
+; GFX8910:       ; %bb.0:
+; GFX8910-NEXT:    s_mov_b32 s4, -8
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_neg8:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_mov_b32 s4, -8
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], s4
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 -8, i32 0)
   ret i32 %load
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_bit31:
-; SI: s_brev_b32 [[K:s[0-9]+]], 1{{$}}
-; SI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
-
-; CI: s_buffer_load_dword s0, s[0:3], 0x20000000{{$}}
-
-; VI: s_brev_b32 [[K:s[0-9]+]], 1{{$}}
-; VI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_bit31(<4 x i32> inreg %desc) {
+; GFX6-LABEL: s_buffer_load_imm_bit31:
+; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_brev_b32 s4, 1
+; GFX6-NEXT:    s_nop 3
+; GFX6-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: s_buffer_load_imm_bit31:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_buffer_load_dword s0, s[0:3], 0x20000000
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    ; return to shader part epilog
+;
+; GFX8910-LABEL: s_buffer_load_imm_bit31:
+; GFX8910:       ; %bb.0:
+; GFX8910-NEXT:    s_brev_b32 s4, 1
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_bit31:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_brev_b32 s4, 1
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], s4
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 -2147483648, i32 0)
   ret i32 %load
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_bit30:
-; SI: s_mov_b32 [[K:s[0-9]+]], 2.0{{$}}
-; SI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
-
-; CI: s_buffer_load_dword s0, s[0:3], 0x10000000{{$}}
-
-; VI: s_mov_b32 [[K:s[0-9]+]], 2.0{{$}}
-; VI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_bit30(<4 x i32> inreg %desc) {
+; GFX6-LABEL: s_buffer_load_imm_bit30:
+; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_mov_b32 s4, 2.0
+; GFX6-NEXT:    s_nop 3
+; GFX6-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: s_buffer_load_imm_bit30:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_buffer_load_dword s0, s[0:3], 0x10000000
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    ; return to shader part epilog
+;
+; GFX8910-LABEL: s_buffer_load_imm_bit30:
+; GFX8910:       ; %bb.0:
+; GFX8910-NEXT:    s_mov_b32 s4, 2.0
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_bit30:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_mov_b32 s4, 2.0
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], s4
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 1073741824, i32 0)
   ret i32 %load
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_bit29:
-; SI: s_brev_b32 [[K:s[0-9]+]], 4{{$}}
-; SI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
-
-; CI: s_buffer_load_dword s0, s[0:3], 0x8000000{{$}}
-
-; VI: s_brev_b32 [[K:s[0-9]+]], 4{{$}}
-; VI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_bit29(<4 x i32> inreg %desc) {
+; GFX6-LABEL: s_buffer_load_imm_bit29:
+; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_brev_b32 s4, 4
+; GFX6-NEXT:    s_nop 3
+; GFX6-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: s_buffer_load_imm_bit29:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_buffer_load_dword s0, s[0:3], 0x8000000
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    ; return to shader part epilog
+;
+; GFX8910-LABEL: s_buffer_load_imm_bit29:
+; GFX8910:       ; %bb.0:
+; GFX8910-NEXT:    s_brev_b32 s4, 4
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_bit29:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_brev_b32 s4, 4
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], s4
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 536870912, i32 0)
   ret i32 %load
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_bit21:
-; SI: s_mov_b32 [[K:s[0-9]+]], 0x200000{{$}}
-; SI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
-
-; CI: s_buffer_load_dword s0, s[0:3], 0x80000{{$}}
-
-; VI: s_mov_b32 [[K:s[0-9]+]], 0x200000{{$}}
-; VI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_bit21(<4 x i32> inreg %desc) {
+; GFX6-LABEL: s_buffer_load_imm_bit21:
+; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_mov_b32 s4, 0x200000
+; GFX6-NEXT:    s_nop 3
+; GFX6-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: s_buffer_load_imm_bit21:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_buffer_load_dword s0, s[0:3], 0x80000
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    ; return to shader part epilog
+;
+; GFX8910-LABEL: s_buffer_load_imm_bit21:
+; GFX8910:       ; %bb.0:
+; GFX8910-NEXT:    s_mov_b32 s4, 0x200000
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_bit21:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_mov_b32 s4, 0x200000
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], s4
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 2097152, i32 0)
   ret i32 %load
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_bit20:
-; SI: s_mov_b32 [[K:s[0-9]+]], 0x100000{{$}}
-; SI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
-
-; CI: s_buffer_load_dword s0, s[0:3], 0x40000{{$}}
-
-; VI: s_mov_b32 [[K:s[0-9]+]], 0x100000{{$}}
-; VI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_bit20(<4 x i32> inreg %desc) {
+; GFX6-LABEL: s_buffer_load_imm_bit20:
+; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_mov_b32 s4, 0x100000
+; GFX6-NEXT:    s_nop 3
+; GFX6-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: s_buffer_load_imm_bit20:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_buffer_load_dword s0, s[0:3], 0x40000
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    ; return to shader part epilog
+;
+; GFX8910-LABEL: s_buffer_load_imm_bit20:
+; GFX8910:       ; %bb.0:
+; GFX8910-NEXT:    s_mov_b32 s4, 0x100000
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_bit20:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_mov_b32 s4, 0x100000
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], s4
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 1048576, i32 0)
   ret i32 %load
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_neg_bit20:
-; SI: s_mov_b32 [[K:s[0-9]+]], 0xfff00000{{$}}
-; SI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
-
-; CI: s_buffer_load_dword s0, s[0:3], 0x3ffc0000{{$}}
-
-; VI: s_mov_b32 [[K:s[0-9]+]], 0xfff00000{{$}}
-; VI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_neg_bit20(<4 x i32> inreg %desc) {
+; GFX6-LABEL: s_buffer_load_imm_neg_bit20:
+; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_mov_b32 s4, 0xfff00000
+; GFX6-NEXT:    s_nop 3
+; GFX6-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: s_buffer_load_imm_neg_bit20:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_buffer_load_dword s0, s[0:3], 0x3ffc0000
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    ; return to shader part epilog
+;
+; GFX8910-LABEL: s_buffer_load_imm_neg_bit20:
+; GFX8910:       ; %bb.0:
+; GFX8910-NEXT:    s_mov_b32 s4, 0xfff00000
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_neg_bit20:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_mov_b32 s4, 0xfff00000
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], s4
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32  -1048576, i32 0)
   ret i32 %load
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_bit19:
-; SI: s_mov_b32 [[K:s[0-9]+]], 0x80000{{$}}
-; SI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
-
-; CI: s_buffer_load_dword s0, s[0:3], 0x20000{{$}}
-
-; VI: s_buffer_load_dword s0, s[0:3], 0x80000{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_bit19(<4 x i32> inreg %desc) {
+; GFX6-LABEL: s_buffer_load_imm_bit19:
+; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_mov_b32 s4, 0x80000
+; GFX6-NEXT:    s_nop 3
+; GFX6-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: s_buffer_load_imm_bit19:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_buffer_load_dword s0, s[0:3], 0x20000
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    ; return to shader part epilog
+;
+; GFX8910-LABEL: s_buffer_load_imm_bit19:
+; GFX8910:       ; %bb.0:
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], 0x80000
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_bit19:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], 0x80000
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 524288, i32 0)
   ret i32 %load
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_neg_bit19:
-; SI: s_mov_b32 [[K:s[0-9]+]], 0xfff80000{{$}}
-; SI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
-
-; CI: s_buffer_load_dword s0, s[0:3], 0x3ffe0000{{$}}
-
-; VI: s_mov_b32 [[K:s[0-9]+]], 0xfff80000{{$}}
-; VI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_neg_bit19(<4 x i32> inreg %desc) {
+; GFX6-LABEL: s_buffer_load_imm_neg_bit19:
+; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_mov_b32 s4, 0xfff80000
+; GFX6-NEXT:    s_nop 3
+; GFX6-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: s_buffer_load_imm_neg_bit19:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_buffer_load_dword s0, s[0:3], 0x3ffe0000
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    ; return to shader part epilog
+;
+; GFX8910-LABEL: s_buffer_load_imm_neg_bit19:
+; GFX8910:       ; %bb.0:
+; GFX8910-NEXT:    s_mov_b32 s4, 0xfff80000
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_neg_bit19:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_mov_b32 s4, 0xfff80000
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], s4
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 -524288, i32 0)
   ret i32 %load
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_255:
-; SICI: s_movk_i32 [[K:s[0-9]+]], 0xff{{$}}
-; SICI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
-
-; VI: s_buffer_load_dword s0, s[0:3], 0xff{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_255(<4 x i32> inreg %desc) {
+; GFX6-LABEL: s_buffer_load_imm_255:
+; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_movk_i32 s4, 0xff
+; GFX6-NEXT:    s_nop 3
+; GFX6-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: s_buffer_load_imm_255:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_movk_i32 s4, 0xff
+; GFX7-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    ; return to shader part epilog
+;
+; GFX8910-LABEL: s_buffer_load_imm_255:
+; GFX8910:       ; %bb.0:
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], 0xff
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_255:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], 0xff
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 255, i32 0)
   ret i32 %load
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_256:
-; SICI: s_buffer_load_dword s0, s[0:3], 0x40{{$}}
-; VI: s_buffer_load_dword s0, s[0:3], 0x100{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_256(<4 x i32> inreg %desc) {
+; GFX67-LABEL: s_buffer_load_imm_256:
+; GFX67:       ; %bb.0:
+; GFX67-NEXT:    s_buffer_load_dword s0, s[0:3], 0x40
+; GFX67-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX67-NEXT:    ; return to shader part epilog
+;
+; GFX8910-LABEL: s_buffer_load_imm_256:
+; GFX8910:       ; %bb.0:
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], 0x100
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_256:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], 0x100
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 256, i32 0)
   ret i32 %load
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_1016:
-; SICI: s_buffer_load_dword s0, s[0:3], 0xfe{{$}}
-; VI: s_buffer_load_dword s0, s[0:3], 0x3f8{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_1016(<4 x i32> inreg %desc) {
+; GFX67-LABEL: s_buffer_load_imm_1016:
+; GFX67:       ; %bb.0:
+; GFX67-NEXT:    s_buffer_load_dword s0, s[0:3], 0xfe
+; GFX67-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX67-NEXT:    ; return to shader part epilog
+;
+; GFX8910-LABEL: s_buffer_load_imm_1016:
+; GFX8910:       ; %bb.0:
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], 0x3f8
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_1016:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], 0x3f8
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 1016, i32 0)
   ret i32 %load
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_1020:
-; SICI: s_buffer_load_dword s0, s[0:3], 0xff{{$}}
-; VI: s_buffer_load_dword s0, s[0:3], 0x3fc{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_1020(<4 x i32> inreg %desc) {
+; GFX67-LABEL: s_buffer_load_imm_1020:
+; GFX67:       ; %bb.0:
+; GFX67-NEXT:    s_buffer_load_dword s0, s[0:3], 0xff
+; GFX67-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX67-NEXT:    ; return to shader part epilog
+;
+; GFX8910-LABEL: s_buffer_load_imm_1020:
+; GFX8910:       ; %bb.0:
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], 0x3fc
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_1020:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], 0x3fc
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 1020, i32 0)
   ret i32 %load
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_1021:
-; SICI: s_movk_i32 [[K:s[0-9]+]], 0x3fd{{$}}
-; SICI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_1021(<4 x i32> inreg %desc) {
+; GFX6-LABEL: s_buffer_load_imm_1021:
+; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_movk_i32 s4, 0x3fd
+; GFX6-NEXT:    s_nop 3
+; GFX6-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: s_buffer_load_imm_1021:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_movk_i32 s4, 0x3fd
+; GFX7-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    ; return to shader part epilog
+;
+; GFX8910-LABEL: s_buffer_load_imm_1021:
+; GFX8910:       ; %bb.0:
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], 0x3fd
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_1021:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], 0x3fd
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 1021, i32 0)
   ret i32 %load
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_1024:
-; SI: s_movk_i32 [[K:s[0-9]+]], 0x400{{$}}
-; SI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
-
-; CI: s_buffer_load_dword s0, s[0:3], 0x100{{$}}
-
-; VI: s_buffer_load_dword s0, s[0:3], 0x400{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_1024(<4 x i32> inreg %desc) {
+; GFX6-LABEL: s_buffer_load_imm_1024:
+; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_movk_i32 s4, 0x400
+; GFX6-NEXT:    s_nop 3
+; GFX6-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: s_buffer_load_imm_1024:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_buffer_load_dword s0, s[0:3], 0x100
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    ; return to shader part epilog
+;
+; GFX8910-LABEL: s_buffer_load_imm_1024:
+; GFX8910:       ; %bb.0:
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], 0x400
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_1024:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], 0x400
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 1024, i32 0)
   ret i32 %load
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_1025:
-; SICI: s_movk_i32 [[K:s[0-9]+]], 0x401{{$}}
-; SICI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
-
-; VI: s_buffer_load_dword s0, s[0:3], 0x401{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_1025(<4 x i32> inreg %desc) {
+; GFX6-LABEL: s_buffer_load_imm_1025:
+; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_movk_i32 s4, 0x401
+; GFX6-NEXT:    s_nop 3
+; GFX6-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: s_buffer_load_imm_1025:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_movk_i32 s4, 0x401
+; GFX7-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    ; return to shader part epilog
+;
+; GFX8910-LABEL: s_buffer_load_imm_1025:
+; GFX8910:       ; %bb.0:
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], 0x401
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_1025:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], 0x401
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 1025, i32 0)
   ret i32 %load
 }
 
-; GCN-LABEL: {{^}}s_buffer_load_imm_1028:
-; SI: s_movk_i32 [[K:s[0-9]+]], 0x400{{$}}
-; SI: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
-
-; CI: s_buffer_load_dword s0, s[0:3], 0x100{{$}}
-; VI: s_buffer_load_dword s0, s[0:3], 0x400{{$}}
 define amdgpu_ps i32 @s_buffer_load_imm_1028(<4 x i32> inreg %desc) {
+; GFX6-LABEL: s_buffer_load_imm_1028:
+; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_movk_i32 s4, 0x400
+; GFX6-NEXT:    s_nop 3
+; GFX6-NEXT:    s_buffer_load_dword s0, s[0:3], s4
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: s_buffer_load_imm_1028:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_buffer_load_dword s0, s[0:3], 0x100
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    ; return to shader part epilog
+;
+; GFX8910-LABEL: s_buffer_load_imm_1028:
+; GFX8910:       ; %bb.0:
+; GFX8910-NEXT:    s_buffer_load_dword s0, s[0:3], 0x400
+; GFX8910-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8910-NEXT:    ; return to shader part epilog
+;
+; GFX11-LABEL: s_buffer_load_imm_1028:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_buffer_load_b32 s0, s[0:3], 0x400
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    ; return to shader part epilog
   %load = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 1024, i32 0)
   ret i32 %load
 }
@@ -458,3 +1166,11 @@ declare i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32>, i32, i32)
 declare <2 x i32> @llvm.amdgcn.s.buffer.load.v2i32(<4 x i32>, i32, i32)
 declare <3 x i32> @llvm.amdgcn.s.buffer.load.v3i32(<4 x i32>, i32, i32)
 declare <4 x i32> @llvm.amdgcn.s.buffer.load.v4i32(<4 x i32>, i32, i32)
+
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX678: {{.*}}
+; GFX6789: {{.*}}
+; GFX78: {{.*}}
+; GFX789: {{.*}}
+; GFX89: {{.*}}
+; GFX910: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/llvm.is.fpclass.f16.ll b/llvm/test/CodeGen/AMDGPU/llvm.is.fpclass.f16.ll
index f4072960f7cfe..35090ec6cef7f 100644
--- a/llvm/test/CodeGen/AMDGPU/llvm.is.fpclass.f16.ll
+++ b/llvm/test/CodeGen/AMDGPU/llvm.is.fpclass.f16.ll
@@ -1758,16 +1758,16 @@ define i1 @not_isnormal_f16(half %x) {
 ; GFX7GLISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
 ; GFX7GLISEL-NEXT:    v_bfe_u32 v1, v0, 0, 16
-; GFX7GLISEL-NEXT:    s_movk_i32 s6, 0x7c00
+; GFX7GLISEL-NEXT:    v_subrev_i32_e64 v0, s[4:5], 1, v0
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v2, 0x3ff
 ; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
-; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s6, v1
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e64 s[4:5], v0, v2
+; GFX7GLISEL-NEXT:    s_movk_i32 s6, 0x7c00
 ; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
-; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, s6, v1
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e32 vcc, s6, v1
 ; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
-; GFX7GLISEL-NEXT:    v_subrev_i32_e32 v0, vcc, 1, v0
-; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0xffff, v0
-; GFX7GLISEL-NEXT:    v_mov_b32_e32 v1, 0x3ff
-; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, v0, v1
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, s6, v1
 ; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
 ; GFX7GLISEL-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
 ; GFX7GLISEL-NEXT:    s_setpc_b64 s[30:31]
@@ -1827,19 +1827,19 @@ define i1 @not_is_plus_normal_f16(half %x) {
 ; GFX7GLISEL:       ; %bb.0:
 ; GFX7GLISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX7GLISEL-NEXT:    v_and_b32_e32 v1, 0x7fff, v0
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0xffff, v0
 ; GFX7GLISEL-NEXT:    v_bfe_u32 v2, v1, 0, 16
-; GFX7GLISEL-NEXT:    s_movk_i32 s8, 0x7c00
-; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
-; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e64 s[6:7], s8, v2
+; GFX7GLISEL-NEXT:    v_cmp_ne_u32_e64 s[4:5], v0, v2
+; GFX7GLISEL-NEXT:    v_subrev_i32_e64 v0, s[6:7], 1, v1
 ; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v3, 0x3ff
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e64 s[6:7], v0, v3
+; GFX7GLISEL-NEXT:    s_movk_i32 s8, 0x7c00
 ; GFX7GLISEL-NEXT:    s_or_b64 s[6:7], vcc, s[6:7]
-; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, s8, v2
-; GFX7GLISEL-NEXT:    v_cmp_ne_u32_e64 s[4:5], v0, v2
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e32 vcc, s8, v2
 ; GFX7GLISEL-NEXT:    s_or_b64 s[6:7], s[6:7], vcc
-; GFX7GLISEL-NEXT:    v_subrev_i32_e32 v0, vcc, 1, v1
-; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0xffff, v0
-; GFX7GLISEL-NEXT:    v_mov_b32_e32 v2, 0x3ff
-; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, v0, v2
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, s8, v2
 ; GFX7GLISEL-NEXT:    s_or_b64 s[6:7], s[6:7], vcc
 ; GFX7GLISEL-NEXT:    v_subrev_i32_e32 v0, vcc, 0x400, v1
 ; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0xffff, v0
@@ -1905,19 +1905,19 @@ define i1 @not_is_neg_normal_f16(half %x) {
 ; GFX7GLISEL:       ; %bb.0:
 ; GFX7GLISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX7GLISEL-NEXT:    v_and_b32_e32 v1, 0x7fff, v0
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0xffff, v0
 ; GFX7GLISEL-NEXT:    v_bfe_u32 v2, v1, 0, 16
-; GFX7GLISEL-NEXT:    s_movk_i32 s8, 0x7c00
-; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
-; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e64 s[6:7], s8, v2
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], v0, v2
+; GFX7GLISEL-NEXT:    v_subrev_i32_e64 v0, s[6:7], 1, v1
 ; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v3, 0x3ff
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e64 s[6:7], v0, v3
+; GFX7GLISEL-NEXT:    s_movk_i32 s8, 0x7c00
 ; GFX7GLISEL-NEXT:    s_or_b64 s[6:7], vcc, s[6:7]
-; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, s8, v2
-; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], v0, v2
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e32 vcc, s8, v2
 ; GFX7GLISEL-NEXT:    s_or_b64 s[6:7], s[6:7], vcc
-; GFX7GLISEL-NEXT:    v_subrev_i32_e32 v0, vcc, 1, v1
-; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0xffff, v0
-; GFX7GLISEL-NEXT:    v_mov_b32_e32 v2, 0x3ff
-; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, v0, v2
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, s8, v2
 ; GFX7GLISEL-NEXT:    s_or_b64 s[6:7], s[6:7], vcc
 ; GFX7GLISEL-NEXT:    v_subrev_i32_e32 v0, vcc, 0x400, v1
 ; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0xffff, v0
@@ -2155,15 +2155,15 @@ define i1 @not_iszero_f16(half %x) {
 ; GFX7GLISEL:       ; %bb.0:
 ; GFX7GLISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
-; GFX7GLISEL-NEXT:    v_bfe_u32 v1, v0, 0, 16
-; GFX7GLISEL-NEXT:    s_movk_i32 s4, 0x7c00
-; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e32 vcc, s4, v1
-; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e64 s[4:5], s4, v1
-; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
 ; GFX7GLISEL-NEXT:    v_subrev_i32_e32 v1, vcc, 1, v0
 ; GFX7GLISEL-NEXT:    v_and_b32_e32 v1, 0xffff, v1
 ; GFX7GLISEL-NEXT:    v_mov_b32_e32 v2, 0x3ff
 ; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, v1, v2
+; GFX7GLISEL-NEXT:    v_bfe_u32 v1, v0, 0, 16
+; GFX7GLISEL-NEXT:    s_movk_i32 s6, 0x7c00
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s6, v1
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, s6, v1
 ; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
 ; GFX7GLISEL-NEXT:    v_subrev_i32_e32 v0, vcc, 0x400, v0
 ; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0xffff, v0
@@ -2464,7 +2464,952 @@ define i1 @not_isnegative_f16(half %x) {
   ret i1 %class
 }
 
-declare i1 @llvm.is.fpclass.f16(half, i32)
-declare <2 x i1> @llvm.is.fpclass.v2f16(<2 x half>, i32)
-declare <3 x i1> @llvm.is.fpclass.v3f16(<3 x half>, i32)
-declare <4 x i1> @llvm.is.fpclass.v4f16(<4 x half>, i32)
+define i1 @iszero_or_nan_f16(half %x) {
+; GFX7SELDAG-LABEL: iszero_or_nan_f16:
+; GFX7SELDAG:       ; %bb.0: ; %entry
+; GFX7SELDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7SELDAG-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x7c00
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7SELDAG-NEXT:    v_cmp_lt_i32_e32 vcc, s4, v0
+; GFX7SELDAG-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v0
+; GFX7SELDAG-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7SELDAG-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7SELDAG-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7GLISEL-LABEL: iszero_or_nan_f16:
+; GFX7GLISEL:       ; %bb.0: ; %entry
+; GFX7GLISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7GLISEL-NEXT:    v_bfe_u32 v0, v0, 0, 16
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v1, 0x7c00
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX7GLISEL-NEXT:    v_cmp_gt_u32_e64 s[4:5], v0, v1
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
+; GFX7GLISEL-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7GLISEL-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX8CHECK-LABEL: iszero_or_nan_f16:
+; GFX8CHECK:       ; %bb.0: ; %entry
+; GFX8CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX8CHECK-NEXT:    v_mov_b32_e32 v1, 0x63
+; GFX8CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX8CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX8CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9CHECK-LABEL: iszero_or_nan_f16:
+; GFX9CHECK:       ; %bb.0: ; %entry
+; GFX9CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9CHECK-NEXT:    v_mov_b32_e32 v1, 0x63
+; GFX9CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX9CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX9CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX10CHECK-LABEL: iszero_or_nan_f16:
+; GFX10CHECK:       ; %bb.0: ; %entry
+; GFX10CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10CHECK-NEXT:    v_cmp_class_f16_e64 s4, v0, 0x63
+; GFX10CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s4
+; GFX10CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11CHECK-LABEL: iszero_or_nan_f16:
+; GFX11CHECK:       ; %bb.0: ; %entry
+; GFX11CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11CHECK-NEXT:    v_cmp_class_f16_e64 s0, v0, 0x63
+; GFX11CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
+; GFX11CHECK-NEXT:    s_setpc_b64 s[30:31]
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f16(half %x, i32 99)  ; 0x60|0x3 = "zero|nan"
+  ret i1 %0
+}
+
+define i1 @iszero_or_nan_f_daz(half %x) #0 {
+; GFX7SELDAG-LABEL: iszero_or_nan_f_daz:
+; GFX7SELDAG:       ; %bb.0: ; %entry
+; GFX7SELDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7SELDAG-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x7c00
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7SELDAG-NEXT:    v_cmp_lt_i32_e32 vcc, s4, v0
+; GFX7SELDAG-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v0
+; GFX7SELDAG-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7SELDAG-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7SELDAG-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7GLISEL-LABEL: iszero_or_nan_f_daz:
+; GFX7GLISEL:       ; %bb.0: ; %entry
+; GFX7GLISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7GLISEL-NEXT:    v_bfe_u32 v0, v0, 0, 16
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v1, 0x7c00
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX7GLISEL-NEXT:    v_cmp_gt_u32_e64 s[4:5], v0, v1
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
+; GFX7GLISEL-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7GLISEL-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX8CHECK-LABEL: iszero_or_nan_f_daz:
+; GFX8CHECK:       ; %bb.0: ; %entry
+; GFX8CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX8CHECK-NEXT:    v_mov_b32_e32 v1, 0x63
+; GFX8CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX8CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX8CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9CHECK-LABEL: iszero_or_nan_f_daz:
+; GFX9CHECK:       ; %bb.0: ; %entry
+; GFX9CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9CHECK-NEXT:    v_mov_b32_e32 v1, 0x63
+; GFX9CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX9CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX9CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX10CHECK-LABEL: iszero_or_nan_f_daz:
+; GFX10CHECK:       ; %bb.0: ; %entry
+; GFX10CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10CHECK-NEXT:    v_cmp_class_f16_e64 s4, v0, 0x63
+; GFX10CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s4
+; GFX10CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11CHECK-LABEL: iszero_or_nan_f_daz:
+; GFX11CHECK:       ; %bb.0: ; %entry
+; GFX11CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11CHECK-NEXT:    v_cmp_class_f16_e64 s0, v0, 0x63
+; GFX11CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
+; GFX11CHECK-NEXT:    s_setpc_b64 s[30:31]
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f16(half %x, i32 99)  ; 0x60|0x3 = "zero|nan"
+  ret i1 %0
+}
+
+define i1 @iszero_or_nan_f_maybe_daz(half %x) #1 {
+; GFX7SELDAG-LABEL: iszero_or_nan_f_maybe_daz:
+; GFX7SELDAG:       ; %bb.0: ; %entry
+; GFX7SELDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7SELDAG-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x7c00
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7SELDAG-NEXT:    v_cmp_lt_i32_e32 vcc, s4, v0
+; GFX7SELDAG-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v0
+; GFX7SELDAG-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7SELDAG-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7SELDAG-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7GLISEL-LABEL: iszero_or_nan_f_maybe_daz:
+; GFX7GLISEL:       ; %bb.0: ; %entry
+; GFX7GLISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7GLISEL-NEXT:    v_bfe_u32 v0, v0, 0, 16
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v1, 0x7c00
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX7GLISEL-NEXT:    v_cmp_gt_u32_e64 s[4:5], v0, v1
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
+; GFX7GLISEL-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7GLISEL-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX8CHECK-LABEL: iszero_or_nan_f_maybe_daz:
+; GFX8CHECK:       ; %bb.0: ; %entry
+; GFX8CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX8CHECK-NEXT:    v_mov_b32_e32 v1, 0x63
+; GFX8CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX8CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX8CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9CHECK-LABEL: iszero_or_nan_f_maybe_daz:
+; GFX9CHECK:       ; %bb.0: ; %entry
+; GFX9CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9CHECK-NEXT:    v_mov_b32_e32 v1, 0x63
+; GFX9CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX9CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX9CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX10CHECK-LABEL: iszero_or_nan_f_maybe_daz:
+; GFX10CHECK:       ; %bb.0: ; %entry
+; GFX10CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10CHECK-NEXT:    v_cmp_class_f16_e64 s4, v0, 0x63
+; GFX10CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s4
+; GFX10CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11CHECK-LABEL: iszero_or_nan_f_maybe_daz:
+; GFX11CHECK:       ; %bb.0: ; %entry
+; GFX11CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11CHECK-NEXT:    v_cmp_class_f16_e64 s0, v0, 0x63
+; GFX11CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
+; GFX11CHECK-NEXT:    s_setpc_b64 s[30:31]
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f16(half %x, i32 99)  ; 0x60|0x3 = "zero|nan"
+  ret i1 %0
+}
+
+define i1 @not_iszero_or_nan_f16(half %x) {
+; GFX7SELDAG-LABEL: not_iszero_or_nan_f16:
+; GFX7SELDAG:       ; %bb.0: ; %entry
+; GFX7SELDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7SELDAG-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x7c00
+; GFX7SELDAG-NEXT:    s_movk_i32 s6, 0x7800
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7SELDAG-NEXT:    v_cmp_eq_u32_e32 vcc, s4, v0
+; GFX7SELDAG-NEXT:    v_add_i32_e64 v1, s[4:5], -1, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x3ff
+; GFX7SELDAG-NEXT:    v_cmp_gt_u32_e64 s[4:5], s4, v1
+; GFX7SELDAG-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7SELDAG-NEXT:    v_add_i32_e32 v0, vcc, 0xfffffc00, v0
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX7SELDAG-NEXT:    v_cmp_gt_u32_e32 vcc, s6, v0
+; GFX7SELDAG-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7SELDAG-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7SELDAG-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7GLISEL-LABEL: not_iszero_or_nan_f16:
+; GFX7GLISEL:       ; %bb.0: ; %entry
+; GFX7GLISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7GLISEL-NEXT:    v_subrev_i32_e32 v1, vcc, 1, v0
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v2, 0x3ff
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, v1, v2
+; GFX7GLISEL-NEXT:    v_bfe_u32 v1, v0, 0, 16
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v2, 0x7c00
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], v1, v2
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
+; GFX7GLISEL-NEXT:    v_subrev_i32_e32 v0, vcc, 0x400, v0
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v1, 0x7800
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, v0, v1
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7GLISEL-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7GLISEL-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX8CHECK-LABEL: not_iszero_or_nan_f16:
+; GFX8CHECK:       ; %bb.0: ; %entry
+; GFX8CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX8CHECK-NEXT:    v_mov_b32_e32 v1, 0x39c
+; GFX8CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX8CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX8CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9CHECK-LABEL: not_iszero_or_nan_f16:
+; GFX9CHECK:       ; %bb.0: ; %entry
+; GFX9CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9CHECK-NEXT:    v_mov_b32_e32 v1, 0x39c
+; GFX9CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX9CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX9CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX10CHECK-LABEL: not_iszero_or_nan_f16:
+; GFX10CHECK:       ; %bb.0: ; %entry
+; GFX10CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10CHECK-NEXT:    v_cmp_class_f16_e64 s4, v0, 0x39c
+; GFX10CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s4
+; GFX10CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11CHECK-LABEL: not_iszero_or_nan_f16:
+; GFX11CHECK:       ; %bb.0: ; %entry
+; GFX11CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11CHECK-NEXT:    v_cmp_class_f16_e64 s0, v0, 0x39c
+; GFX11CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
+; GFX11CHECK-NEXT:    s_setpc_b64 s[30:31]
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f16(half %x, i32 924)  ; ~0x60 = "~(zero|nan)"
+  ret i1 %0
+}
+
+define i1 @not_iszero_or_nan_f_daz(half %x) #0 {
+; GFX7SELDAG-LABEL: not_iszero_or_nan_f_daz:
+; GFX7SELDAG:       ; %bb.0: ; %entry
+; GFX7SELDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7SELDAG-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x7c00
+; GFX7SELDAG-NEXT:    s_movk_i32 s6, 0x7800
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7SELDAG-NEXT:    v_cmp_eq_u32_e32 vcc, s4, v0
+; GFX7SELDAG-NEXT:    v_add_i32_e64 v1, s[4:5], -1, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x3ff
+; GFX7SELDAG-NEXT:    v_cmp_gt_u32_e64 s[4:5], s4, v1
+; GFX7SELDAG-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7SELDAG-NEXT:    v_add_i32_e32 v0, vcc, 0xfffffc00, v0
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX7SELDAG-NEXT:    v_cmp_gt_u32_e32 vcc, s6, v0
+; GFX7SELDAG-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7SELDAG-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7SELDAG-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7GLISEL-LABEL: not_iszero_or_nan_f_daz:
+; GFX7GLISEL:       ; %bb.0: ; %entry
+; GFX7GLISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7GLISEL-NEXT:    v_subrev_i32_e32 v1, vcc, 1, v0
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v2, 0x3ff
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, v1, v2
+; GFX7GLISEL-NEXT:    v_bfe_u32 v1, v0, 0, 16
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v2, 0x7c00
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], v1, v2
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
+; GFX7GLISEL-NEXT:    v_subrev_i32_e32 v0, vcc, 0x400, v0
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v1, 0x7800
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, v0, v1
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7GLISEL-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7GLISEL-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX8CHECK-LABEL: not_iszero_or_nan_f_daz:
+; GFX8CHECK:       ; %bb.0: ; %entry
+; GFX8CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX8CHECK-NEXT:    v_mov_b32_e32 v1, 0x39c
+; GFX8CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX8CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX8CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9CHECK-LABEL: not_iszero_or_nan_f_daz:
+; GFX9CHECK:       ; %bb.0: ; %entry
+; GFX9CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9CHECK-NEXT:    v_mov_b32_e32 v1, 0x39c
+; GFX9CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX9CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX9CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX10CHECK-LABEL: not_iszero_or_nan_f_daz:
+; GFX10CHECK:       ; %bb.0: ; %entry
+; GFX10CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10CHECK-NEXT:    v_cmp_class_f16_e64 s4, v0, 0x39c
+; GFX10CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s4
+; GFX10CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11CHECK-LABEL: not_iszero_or_nan_f_daz:
+; GFX11CHECK:       ; %bb.0: ; %entry
+; GFX11CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11CHECK-NEXT:    v_cmp_class_f16_e64 s0, v0, 0x39c
+; GFX11CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
+; GFX11CHECK-NEXT:    s_setpc_b64 s[30:31]
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f16(half %x, i32 924)  ; ~(0x60|0x3) = "~(zero|nan)"
+  ret i1 %0
+}
+
+define i1 @not_iszero_or_nan_f_maybe_daz(half %x) #1 {
+; GFX7SELDAG-LABEL: not_iszero_or_nan_f_maybe_daz:
+; GFX7SELDAG:       ; %bb.0: ; %entry
+; GFX7SELDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7SELDAG-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x7c00
+; GFX7SELDAG-NEXT:    s_movk_i32 s6, 0x7800
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7SELDAG-NEXT:    v_cmp_eq_u32_e32 vcc, s4, v0
+; GFX7SELDAG-NEXT:    v_add_i32_e64 v1, s[4:5], -1, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x3ff
+; GFX7SELDAG-NEXT:    v_cmp_gt_u32_e64 s[4:5], s4, v1
+; GFX7SELDAG-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7SELDAG-NEXT:    v_add_i32_e32 v0, vcc, 0xfffffc00, v0
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX7SELDAG-NEXT:    v_cmp_gt_u32_e32 vcc, s6, v0
+; GFX7SELDAG-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7SELDAG-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7SELDAG-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7GLISEL-LABEL: not_iszero_or_nan_f_maybe_daz:
+; GFX7GLISEL:       ; %bb.0: ; %entry
+; GFX7GLISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7GLISEL-NEXT:    v_subrev_i32_e32 v1, vcc, 1, v0
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v2, 0x3ff
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, v1, v2
+; GFX7GLISEL-NEXT:    v_bfe_u32 v1, v0, 0, 16
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v2, 0x7c00
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], v1, v2
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
+; GFX7GLISEL-NEXT:    v_subrev_i32_e32 v0, vcc, 0x400, v0
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v1, 0x7800
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, v0, v1
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7GLISEL-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7GLISEL-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX8CHECK-LABEL: not_iszero_or_nan_f_maybe_daz:
+; GFX8CHECK:       ; %bb.0: ; %entry
+; GFX8CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX8CHECK-NEXT:    v_mov_b32_e32 v1, 0x39c
+; GFX8CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX8CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX8CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9CHECK-LABEL: not_iszero_or_nan_f_maybe_daz:
+; GFX9CHECK:       ; %bb.0: ; %entry
+; GFX9CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9CHECK-NEXT:    v_mov_b32_e32 v1, 0x39c
+; GFX9CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX9CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX9CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX10CHECK-LABEL: not_iszero_or_nan_f_maybe_daz:
+; GFX10CHECK:       ; %bb.0: ; %entry
+; GFX10CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10CHECK-NEXT:    v_cmp_class_f16_e64 s4, v0, 0x39c
+; GFX10CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s4
+; GFX10CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11CHECK-LABEL: not_iszero_or_nan_f_maybe_daz:
+; GFX11CHECK:       ; %bb.0: ; %entry
+; GFX11CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11CHECK-NEXT:    v_cmp_class_f16_e64 s0, v0, 0x39c
+; GFX11CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
+; GFX11CHECK-NEXT:    s_setpc_b64 s[30:31]
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f16(half %x, i32 924)  ; ~(0x60|0x3) = "~(zero|nan)"
+  ret i1 %0
+}
+
+define i1 @iszero_or_qnan_f16(half %x) {
+; GFX7SELDAG-LABEL: iszero_or_qnan_f16:
+; GFX7SELDAG:       ; %bb.0: ; %entry
+; GFX7SELDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7SELDAG-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x7dff
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7SELDAG-NEXT:    v_cmp_lt_i32_e32 vcc, s4, v0
+; GFX7SELDAG-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v0
+; GFX7SELDAG-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7SELDAG-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7SELDAG-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7GLISEL-LABEL: iszero_or_qnan_f16:
+; GFX7GLISEL:       ; %bb.0: ; %entry
+; GFX7GLISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7GLISEL-NEXT:    v_bfe_u32 v0, v0, 0, 16
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v1, 0x7e00
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX7GLISEL-NEXT:    v_cmp_ge_u32_e64 s[4:5], v0, v1
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
+; GFX7GLISEL-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7GLISEL-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX8CHECK-LABEL: iszero_or_qnan_f16:
+; GFX8CHECK:       ; %bb.0: ; %entry
+; GFX8CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX8CHECK-NEXT:    v_mov_b32_e32 v1, 0x62
+; GFX8CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX8CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX8CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9CHECK-LABEL: iszero_or_qnan_f16:
+; GFX9CHECK:       ; %bb.0: ; %entry
+; GFX9CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9CHECK-NEXT:    v_mov_b32_e32 v1, 0x62
+; GFX9CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX9CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX9CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX10CHECK-LABEL: iszero_or_qnan_f16:
+; GFX10CHECK:       ; %bb.0: ; %entry
+; GFX10CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10CHECK-NEXT:    v_cmp_class_f16_e64 s4, v0, 0x62
+; GFX10CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s4
+; GFX10CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11CHECK-LABEL: iszero_or_qnan_f16:
+; GFX11CHECK:       ; %bb.0: ; %entry
+; GFX11CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11CHECK-NEXT:    v_cmp_class_f16_e64 s0, v0, 0x62
+; GFX11CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
+; GFX11CHECK-NEXT:    s_setpc_b64 s[30:31]
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f16(half %x, i32 98)  ; 0x60|0x2 = "zero|qnan"
+  ret i1 %0
+}
+
+define i1 @iszero_or_snan_f16(half %x) {
+; GFX7SELDAG-LABEL: iszero_or_snan_f16:
+; GFX7SELDAG:       ; %bb.0: ; %entry
+; GFX7SELDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7SELDAG-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x7e00
+; GFX7SELDAG-NEXT:    s_movk_i32 s5, 0x7c00
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7SELDAG-NEXT:    v_cmp_gt_i32_e32 vcc, s4, v0
+; GFX7SELDAG-NEXT:    v_cmp_lt_i32_e64 s[4:5], s5, v0
+; GFX7SELDAG-NEXT:    s_and_b64 s[4:5], s[4:5], vcc
+; GFX7SELDAG-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX7SELDAG-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
+; GFX7SELDAG-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7SELDAG-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7GLISEL-LABEL: iszero_or_snan_f16:
+; GFX7GLISEL:       ; %bb.0: ; %entry
+; GFX7GLISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7GLISEL-NEXT:    v_bfe_u32 v0, v0, 0, 16
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v1, 0x7c00
+; GFX7GLISEL-NEXT:    v_cmp_gt_u32_e64 s[4:5], v0, v1
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v1, 0x7e00
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e64 s[6:7], v0, v1
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX7GLISEL-NEXT:    s_and_b64 s[4:5], s[4:5], s[6:7]
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
+; GFX7GLISEL-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7GLISEL-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX8CHECK-LABEL: iszero_or_snan_f16:
+; GFX8CHECK:       ; %bb.0: ; %entry
+; GFX8CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX8CHECK-NEXT:    v_mov_b32_e32 v1, 0x61
+; GFX8CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX8CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX8CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9CHECK-LABEL: iszero_or_snan_f16:
+; GFX9CHECK:       ; %bb.0: ; %entry
+; GFX9CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9CHECK-NEXT:    v_mov_b32_e32 v1, 0x61
+; GFX9CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX9CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX9CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX10CHECK-LABEL: iszero_or_snan_f16:
+; GFX10CHECK:       ; %bb.0: ; %entry
+; GFX10CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10CHECK-NEXT:    v_cmp_class_f16_e64 s4, v0, 0x61
+; GFX10CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s4
+; GFX10CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11CHECK-LABEL: iszero_or_snan_f16:
+; GFX11CHECK:       ; %bb.0: ; %entry
+; GFX11CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11CHECK-NEXT:    v_cmp_class_f16_e64 s0, v0, 0x61
+; GFX11CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
+; GFX11CHECK-NEXT:    s_setpc_b64 s[30:31]
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f16(half %x, i32 97)  ; 0x60|0x1 = "zero|snan"
+  ret i1 %0
+}
+
+define i1 @not_iszero_or_qnan_f16(half %x) {
+; GFX7SELDAG-LABEL: not_iszero_or_qnan_f16:
+; GFX7SELDAG:       ; %bb.0: ; %entry
+; GFX7SELDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7SELDAG-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x7e00
+; GFX7SELDAG-NEXT:    s_movk_i32 s8, 0x7c00
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7SELDAG-NEXT:    v_cmp_gt_i32_e32 vcc, s4, v0
+; GFX7SELDAG-NEXT:    v_cmp_lt_i32_e64 s[4:5], s8, v0
+; GFX7SELDAG-NEXT:    s_and_b64 s[6:7], s[4:5], vcc
+; GFX7SELDAG-NEXT:    v_add_i32_e64 v1, s[4:5], -1, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x3ff
+; GFX7SELDAG-NEXT:    v_cmp_eq_u32_e32 vcc, s8, v0
+; GFX7SELDAG-NEXT:    v_cmp_gt_u32_e64 s[4:5], s4, v1
+; GFX7SELDAG-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7SELDAG-NEXT:    v_add_i32_e32 v0, vcc, 0xfffffc00, v0
+; GFX7SELDAG-NEXT:    s_or_b64 s[4:5], s[4:5], s[6:7]
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s6, 0x7800
+; GFX7SELDAG-NEXT:    v_cmp_gt_u32_e32 vcc, s6, v0
+; GFX7SELDAG-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7SELDAG-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7SELDAG-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7GLISEL-LABEL: not_iszero_or_qnan_f16:
+; GFX7GLISEL:       ; %bb.0: ; %entry
+; GFX7GLISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7GLISEL-NEXT:    v_subrev_i32_e32 v1, vcc, 1, v0
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v2, 0x3ff
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, v1, v2
+; GFX7GLISEL-NEXT:    v_bfe_u32 v1, v0, 0, 16
+; GFX7GLISEL-NEXT:    s_movk_i32 s8, 0x7c00
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s8, v1
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v2, 0x7e00
+; GFX7GLISEL-NEXT:    s_or_b64 s[6:7], vcc, s[4:5]
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, s8, v1
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e64 s[4:5], v1, v2
+; GFX7GLISEL-NEXT:    s_and_b64 s[4:5], vcc, s[4:5]
+; GFX7GLISEL-NEXT:    v_subrev_i32_e32 v0, vcc, 0x400, v0
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v1, 0x7800
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], s[6:7], s[4:5]
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, v0, v1
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7GLISEL-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7GLISEL-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX8CHECK-LABEL: not_iszero_or_qnan_f16:
+; GFX8CHECK:       ; %bb.0: ; %entry
+; GFX8CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX8CHECK-NEXT:    v_mov_b32_e32 v1, 0x39d
+; GFX8CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX8CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX8CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9CHECK-LABEL: not_iszero_or_qnan_f16:
+; GFX9CHECK:       ; %bb.0: ; %entry
+; GFX9CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9CHECK-NEXT:    v_mov_b32_e32 v1, 0x39d
+; GFX9CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX9CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX9CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX10CHECK-LABEL: not_iszero_or_qnan_f16:
+; GFX10CHECK:       ; %bb.0: ; %entry
+; GFX10CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10CHECK-NEXT:    v_cmp_class_f16_e64 s4, v0, 0x39d
+; GFX10CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s4
+; GFX10CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11CHECK-LABEL: not_iszero_or_qnan_f16:
+; GFX11CHECK:       ; %bb.0: ; %entry
+; GFX11CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11CHECK-NEXT:    v_cmp_class_f16_e64 s0, v0, 0x39d
+; GFX11CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
+; GFX11CHECK-NEXT:    s_setpc_b64 s[30:31]
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f16(half %x, i32 925)  ; ~(0x60|0x2) = "~(zero|qnan)"
+  ret i1 %0
+}
+
+define i1 @not_iszero_or_snan_f16(half %x) {
+; GFX7SELDAG-LABEL: not_iszero_or_snan_f16:
+; GFX7SELDAG:       ; %bb.0: ; %entry
+; GFX7SELDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7SELDAG-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x7c00
+; GFX7SELDAG-NEXT:    s_movk_i32 s6, 0x7dff
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7SELDAG-NEXT:    v_cmp_eq_u32_e32 vcc, s4, v0
+; GFX7SELDAG-NEXT:    v_add_i32_e64 v1, s[4:5], -1, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x3ff
+; GFX7SELDAG-NEXT:    v_cmp_gt_u32_e64 s[4:5], s4, v1
+; GFX7SELDAG-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7SELDAG-NEXT:    v_cmp_lt_i32_e32 vcc, s6, v0
+; GFX7SELDAG-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7SELDAG-NEXT:    v_add_i32_e32 v0, vcc, 0xfffffc00, v0
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s6, 0x7800
+; GFX7SELDAG-NEXT:    v_cmp_gt_u32_e32 vcc, s6, v0
+; GFX7SELDAG-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7SELDAG-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7SELDAG-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7GLISEL-LABEL: not_iszero_or_snan_f16:
+; GFX7GLISEL:       ; %bb.0: ; %entry
+; GFX7GLISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7GLISEL-NEXT:    v_subrev_i32_e32 v1, vcc, 1, v0
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v2, 0x3ff
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, v1, v2
+; GFX7GLISEL-NEXT:    v_bfe_u32 v1, v0, 0, 16
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v2, 0x7c00
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], v1, v2
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v2, 0x7e00
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
+; GFX7GLISEL-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v2
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7GLISEL-NEXT:    v_subrev_i32_e32 v0, vcc, 0x400, v0
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v1, 0x7800
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, v0, v1
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX7GLISEL-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7GLISEL-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX8CHECK-LABEL: not_iszero_or_snan_f16:
+; GFX8CHECK:       ; %bb.0: ; %entry
+; GFX8CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX8CHECK-NEXT:    v_mov_b32_e32 v1, 0x39e
+; GFX8CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX8CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX8CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9CHECK-LABEL: not_iszero_or_snan_f16:
+; GFX9CHECK:       ; %bb.0: ; %entry
+; GFX9CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9CHECK-NEXT:    v_mov_b32_e32 v1, 0x39e
+; GFX9CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX9CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX9CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX10CHECK-LABEL: not_iszero_or_snan_f16:
+; GFX10CHECK:       ; %bb.0: ; %entry
+; GFX10CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10CHECK-NEXT:    v_cmp_class_f16_e64 s4, v0, 0x39e
+; GFX10CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s4
+; GFX10CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11CHECK-LABEL: not_iszero_or_snan_f16:
+; GFX11CHECK:       ; %bb.0: ; %entry
+; GFX11CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11CHECK-NEXT:    v_cmp_class_f16_e64 s0, v0, 0x39e
+; GFX11CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
+; GFX11CHECK-NEXT:    s_setpc_b64 s[30:31]
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f16(half %x, i32 926)  ; ~(0x60|0x1) = "~(zero|snan)"
+  ret i1 %0
+}
+
+define i1 @isinf_or_nan_f16(half %x) {
+; GFX7SELDAG-LABEL: isinf_or_nan_f16:
+; GFX7SELDAG:       ; %bb.0: ; %entry
+; GFX7SELDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7SELDAG-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x7bff
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7SELDAG-NEXT:    v_cmp_lt_i32_e32 vcc, s4, v0
+; GFX7SELDAG-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX7SELDAG-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7GLISEL-LABEL: isinf_or_nan_f16:
+; GFX7GLISEL:       ; %bb.0: ; %entry
+; GFX7GLISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7GLISEL-NEXT:    v_bfe_u32 v0, v0, 0, 16
+; GFX7GLISEL-NEXT:    s_movk_i32 s4, 0x7c00
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e32 vcc, s4, v0
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e64 s[4:5], s4, v0
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
+; GFX7GLISEL-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7GLISEL-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX8CHECK-LABEL: isinf_or_nan_f16:
+; GFX8CHECK:       ; %bb.0: ; %entry
+; GFX8CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX8CHECK-NEXT:    v_mov_b32_e32 v1, 0x207
+; GFX8CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX8CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX8CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9CHECK-LABEL: isinf_or_nan_f16:
+; GFX9CHECK:       ; %bb.0: ; %entry
+; GFX9CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9CHECK-NEXT:    v_mov_b32_e32 v1, 0x207
+; GFX9CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX9CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX9CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX10CHECK-LABEL: isinf_or_nan_f16:
+; GFX10CHECK:       ; %bb.0: ; %entry
+; GFX10CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10CHECK-NEXT:    v_cmp_class_f16_e64 s4, v0, 0x207
+; GFX10CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s4
+; GFX10CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11CHECK-LABEL: isinf_or_nan_f16:
+; GFX11CHECK:       ; %bb.0: ; %entry
+; GFX11CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11CHECK-NEXT:    v_cmp_class_f16_e64 s0, v0, 0x207
+; GFX11CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
+; GFX11CHECK-NEXT:    s_setpc_b64 s[30:31]
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f16(half %x, i32 519)  ; 0x204|0x3 = "inf|nan"
+  ret i1 %0
+}
+
+define i1 @not_isinf_or_nan_f16(half %x) {
+; GFX7SELDAG-LABEL: not_isinf_or_nan_f16:
+; GFX7SELDAG:       ; %bb.0: ; %entry
+; GFX7SELDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7SELDAG-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x7c00
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7SELDAG-NEXT:    v_cmp_gt_i32_e32 vcc, s4, v0
+; GFX7SELDAG-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX7SELDAG-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7GLISEL-LABEL: not_isinf_or_nan_f16:
+; GFX7GLISEL:       ; %bb.0: ; %entry
+; GFX7GLISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7GLISEL-NEXT:    v_bfe_u32 v0, v0, 0, 16
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v1, 0x7c00
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e32 vcc, v0, v1
+; GFX7GLISEL-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX7GLISEL-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX8CHECK-LABEL: not_isinf_or_nan_f16:
+; GFX8CHECK:       ; %bb.0: ; %entry
+; GFX8CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX8CHECK-NEXT:    v_mov_b32_e32 v1, 0x1f8
+; GFX8CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX8CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX8CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9CHECK-LABEL: not_isinf_or_nan_f16:
+; GFX9CHECK:       ; %bb.0: ; %entry
+; GFX9CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9CHECK-NEXT:    v_mov_b32_e32 v1, 0x1f8
+; GFX9CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX9CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX9CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX10CHECK-LABEL: not_isinf_or_nan_f16:
+; GFX10CHECK:       ; %bb.0: ; %entry
+; GFX10CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10CHECK-NEXT:    v_cmp_class_f16_e64 s4, v0, 0x1f8
+; GFX10CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s4
+; GFX10CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11CHECK-LABEL: not_isinf_or_nan_f16:
+; GFX11CHECK:       ; %bb.0: ; %entry
+; GFX11CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11CHECK-NEXT:    v_cmp_class_f16_e64 s0, v0, 0x1f8
+; GFX11CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
+; GFX11CHECK-NEXT:    s_setpc_b64 s[30:31]
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f16(half %x, i32 504)  ; ~(0x204|0x3) = "~(inf|nan)"
+  ret i1 %0
+}
+
+define i1 @isfinite_or_nan_f(half %x) {
+; GFX7SELDAG-LABEL: isfinite_or_nan_f:
+; GFX7SELDAG:       ; %bb.0: ; %entry
+; GFX7SELDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7SELDAG-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x7c00
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7SELDAG-NEXT:    v_cmp_ne_u32_e32 vcc, s4, v0
+; GFX7SELDAG-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX7SELDAG-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7GLISEL-LABEL: isfinite_or_nan_f:
+; GFX7GLISEL:       ; %bb.0: ; %entry
+; GFX7GLISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7GLISEL-NEXT:    v_bfe_u32 v0, v0, 0, 16
+; GFX7GLISEL-NEXT:    s_movk_i32 s4, 0x7c00
+; GFX7GLISEL-NEXT:    v_cmp_gt_u32_e32 vcc, s4, v0
+; GFX7GLISEL-NEXT:    v_cmp_lt_u32_e64 s[4:5], s4, v0
+; GFX7GLISEL-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
+; GFX7GLISEL-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
+; GFX7GLISEL-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX8CHECK-LABEL: isfinite_or_nan_f:
+; GFX8CHECK:       ; %bb.0: ; %entry
+; GFX8CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX8CHECK-NEXT:    v_mov_b32_e32 v1, 0x1fb
+; GFX8CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX8CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX8CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9CHECK-LABEL: isfinite_or_nan_f:
+; GFX9CHECK:       ; %bb.0: ; %entry
+; GFX9CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9CHECK-NEXT:    v_mov_b32_e32 v1, 0x1fb
+; GFX9CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX9CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX9CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX10CHECK-LABEL: isfinite_or_nan_f:
+; GFX10CHECK:       ; %bb.0: ; %entry
+; GFX10CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10CHECK-NEXT:    v_cmp_class_f16_e64 s4, v0, 0x1fb
+; GFX10CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s4
+; GFX10CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11CHECK-LABEL: isfinite_or_nan_f:
+; GFX11CHECK:       ; %bb.0: ; %entry
+; GFX11CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11CHECK-NEXT:    v_cmp_class_f16_e64 s0, v0, 0x1fb
+; GFX11CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
+; GFX11CHECK-NEXT:    s_setpc_b64 s[30:31]
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f16(half %x, i32 507)  ; 0x1f8|0x3 = "finite|nan"
+  ret i1 %0
+}
+
+define i1 @not_isfinite_or_nan_f(half %x) {
+; GFX7SELDAG-LABEL: not_isfinite_or_nan_f:
+; GFX7SELDAG:       ; %bb.0: ; %entry
+; GFX7SELDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7SELDAG-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; GFX7SELDAG-NEXT:    s_movk_i32 s4, 0x7c00
+; GFX7SELDAG-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7SELDAG-NEXT:    v_cmp_eq_u32_e32 vcc, s4, v0
+; GFX7SELDAG-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX7SELDAG-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7GLISEL-LABEL: not_isfinite_or_nan_f:
+; GFX7GLISEL:       ; %bb.0: ; %entry
+; GFX7GLISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7GLISEL-NEXT:    v_and_b32_e32 v0, 0x7fff, v0
+; GFX7GLISEL-NEXT:    v_bfe_u32 v0, v0, 0, 16
+; GFX7GLISEL-NEXT:    v_mov_b32_e32 v1, 0x7c00
+; GFX7GLISEL-NEXT:    v_cmp_eq_u32_e32 vcc, v0, v1
+; GFX7GLISEL-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX7GLISEL-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX8CHECK-LABEL: not_isfinite_or_nan_f:
+; GFX8CHECK:       ; %bb.0: ; %entry
+; GFX8CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX8CHECK-NEXT:    v_mov_b32_e32 v1, 0x204
+; GFX8CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX8CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX8CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9CHECK-LABEL: not_isfinite_or_nan_f:
+; GFX9CHECK:       ; %bb.0: ; %entry
+; GFX9CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9CHECK-NEXT:    v_mov_b32_e32 v1, 0x204
+; GFX9CHECK-NEXT:    v_cmp_class_f16_e32 vcc, v0, v1
+; GFX9CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
+; GFX9CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX10CHECK-LABEL: not_isfinite_or_nan_f:
+; GFX10CHECK:       ; %bb.0: ; %entry
+; GFX10CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10CHECK-NEXT:    v_cmp_class_f16_e64 s4, v0, 0x204
+; GFX10CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s4
+; GFX10CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11CHECK-LABEL: not_isfinite_or_nan_f:
+; GFX11CHECK:       ; %bb.0: ; %entry
+; GFX11CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11CHECK-NEXT:    v_cmp_class_f16_e64 s0, v0, 0x204
+; GFX11CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
+; GFX11CHECK-NEXT:    s_setpc_b64 s[30:31]
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f16(half %x, i32 516)  ; ~(0x1f8|0x3) = "~(finite|nan)"
+  ret i1 %0
+}
+
+declare i1 @llvm.is.fpclass.f16(half, i32)
+declare <2 x i1> @llvm.is.fpclass.v2f16(<2 x half>, i32)
+declare <3 x i1> @llvm.is.fpclass.v3f16(<3 x half>, i32)
+declare <4 x i1> @llvm.is.fpclass.v4f16(<4 x half>, i32)
+
+; Assume DAZ
+attributes #0 = { "denormal-fp-math"="ieee,preserve-sign" }
+
+; Maybe daz
+attributes #1 = { "denormal-fp-math"="ieee,dynamic" }
diff --git a/llvm/test/CodeGen/AMDGPU/llvm.round.f64.ll b/llvm/test/CodeGen/AMDGPU/llvm.round.f64.ll
index 1aed1de7cf08b..6b013175ff61b 100644
--- a/llvm/test/CodeGen/AMDGPU/llvm.round.f64.ll
+++ b/llvm/test/CodeGen/AMDGPU/llvm.round.f64.ll
@@ -7,33 +7,32 @@ define amdgpu_kernel void @round_f64(ptr addrspace(1) %out, double %x) #0 {
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
 ; SI-NEXT:    s_mov_b32 s6, -1
-; SI-NEXT:    s_mov_b32 s9, 0xfffff
-; SI-NEXT:    s_mov_b32 s8, s6
-; SI-NEXT:    v_mov_b32_e32 v2, 0x3ff00000
+; SI-NEXT:    s_mov_b32 s5, 0xfffff
+; SI-NEXT:    s_mov_b32 s4, s6
+; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
-; SI-NEXT:    s_mov_b32 s4, s0
-; SI-NEXT:    s_bfe_u32 s0, s3, 0xb0014
-; SI-NEXT:    s_addk_i32 s0, 0xfc01
-; SI-NEXT:    s_lshr_b64 s[8:9], s[8:9], s0
-; SI-NEXT:    s_andn2_b64 s[8:9], s[2:3], s[8:9]
-; SI-NEXT:    s_and_b32 s5, s3, 0x80000000
-; SI-NEXT:    s_cmp_lt_i32 s0, 0
-; SI-NEXT:    s_cselect_b32 s8, 0, s8
-; SI-NEXT:    s_cselect_b32 s5, s5, s9
-; SI-NEXT:    s_cmp_gt_i32 s0, 51
-; SI-NEXT:    s_cselect_b32 s8, s2, s8
+; SI-NEXT:    s_bfe_u32 s8, s3, 0xb0014
+; SI-NEXT:    s_addk_i32 s8, 0xfc01
+; SI-NEXT:    s_lshr_b64 s[4:5], s[4:5], s8
+; SI-NEXT:    s_andn2_b64 s[4:5], s[2:3], s[4:5]
+; SI-NEXT:    s_and_b32 s10, s3, 0x80000000
+; SI-NEXT:    s_cmp_lt_i32 s8, 0
+; SI-NEXT:    s_cselect_b32 s4, 0, s4
+; SI-NEXT:    s_cselect_b32 s5, s10, s5
+; SI-NEXT:    s_cmp_gt_i32 s8, 51
+; SI-NEXT:    s_cselect_b32 s8, s2, s4
 ; SI-NEXT:    s_cselect_b32 s9, s3, s5
 ; SI-NEXT:    v_mov_b32_e32 v0, s8
 ; SI-NEXT:    v_mov_b32_e32 v1, s9
 ; SI-NEXT:    v_add_f64 v[0:1], s[2:3], -v[0:1]
-; SI-NEXT:    s_brev_b32 s0, -2
-; SI-NEXT:    v_mov_b32_e32 v3, s3
-; SI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[0:1]|, 0.5
-; SI-NEXT:    v_bfi_b32 v2, s0, v2, v3
-; SI-NEXT:    v_cndmask_b32_e32 v1, 0, v2, vcc
+; SI-NEXT:    s_mov_b32 s4, s0
+; SI-NEXT:    v_cmp_ge_f64_e64 s[2:3], |v[0:1]|, 0.5
+; SI-NEXT:    s_or_b32 s0, s10, 0x3ff00000
+; SI-NEXT:    s_and_b64 s[2:3], s[2:3], exec
+; SI-NEXT:    s_cselect_b32 s0, s0, 0
 ; SI-NEXT:    v_mov_b32_e32 v0, 0
+; SI-NEXT:    v_mov_b32_e32 v1, s0
 ; SI-NEXT:    v_add_f64 v[0:1], s[8:9], v[0:1]
-; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_mov_b32 s5, s1
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; SI-NEXT:    s_endpgm
@@ -41,20 +40,19 @@ define amdgpu_kernel void @round_f64(ptr addrspace(1) %out, double %x) #0 {
 ; CI-LABEL: round_f64:
 ; CI:       ; %bb.0:
 ; CI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; CI-NEXT:    s_brev_b32 s5, -2
-; CI-NEXT:    v_mov_b32_e32 v4, 0x3ff00000
+; CI-NEXT:    s_mov_b32 s8, 0
 ; CI-NEXT:    s_mov_b32 s7, 0xf000
 ; CI-NEXT:    s_mov_b32 s6, -1
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    v_trunc_f64_e32 v[0:1], s[2:3]
-; CI-NEXT:    v_mov_b32_e32 v5, s3
-; CI-NEXT:    v_add_f64 v[2:3], s[2:3], -v[0:1]
-; CI-NEXT:    v_bfi_b32 v4, s5, v4, v5
-; CI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[2:3]|, 0.5
-; CI-NEXT:    v_mov_b32_e32 v2, 0
-; CI-NEXT:    v_cndmask_b32_e32 v3, 0, v4, vcc
-; CI-NEXT:    v_add_f64 v[0:1], v[0:1], v[2:3]
 ; CI-NEXT:    s_mov_b32 s4, s0
+; CI-NEXT:    v_add_f64 v[2:3], s[2:3], -v[0:1]
+; CI-NEXT:    s_and_b32 s0, s3, 0x80000000
+; CI-NEXT:    v_cmp_ge_f64_e64 s[2:3], |v[2:3]|, 0.5
+; CI-NEXT:    s_or_b32 s0, s0, 0x3ff00000
+; CI-NEXT:    s_and_b64 s[2:3], s[2:3], exec
+; CI-NEXT:    s_cselect_b32 s9, s0, 0
+; CI-NEXT:    v_add_f64 v[0:1], v[0:1], s[8:9]
 ; CI-NEXT:    s_mov_b32 s5, s1
 ; CI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; CI-NEXT:    s_endpgm
@@ -77,8 +75,6 @@ define amdgpu_kernel void @v_round_f64(ptr addrspace(1) %out, ptr addrspace(1) %
 ; SI-NEXT:    s_movk_i32 s4, 0xfc01
 ; SI-NEXT:    s_mov_b32 s2, -1
 ; SI-NEXT:    s_mov_b32 s3, 0xfffff
-; SI-NEXT:    s_brev_b32 s5, -2
-; SI-NEXT:    v_mov_b32_e32 v8, 0x3ff00000
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_bfe_u32 v4, v3, 20, 11
 ; SI-NEXT:    v_add_i32_e32 v6, vcc, s4, v4
@@ -94,13 +90,13 @@ define amdgpu_kernel void @v_round_f64(ptr addrspace(1) %out, ptr addrspace(1) %
 ; SI-NEXT:    v_cmp_lt_i32_e32 vcc, 51, v6
 ; SI-NEXT:    v_cndmask_b32_e32 v5, v5, v3, vcc
 ; SI-NEXT:    v_cndmask_b32_e32 v4, v4, v2, vcc
-; SI-NEXT:    v_add_f64 v[6:7], v[2:3], -v[4:5]
-; SI-NEXT:    v_bfi_b32 v2, s5, v8, v3
-; SI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[6:7]|, 0.5
-; SI-NEXT:    s_mov_b64 s[2:3], s[6:7]
-; SI-NEXT:    v_cndmask_b32_e32 v3, 0, v2, vcc
-; SI-NEXT:    v_mov_b32_e32 v2, 0
+; SI-NEXT:    v_add_f64 v[2:3], v[2:3], -v[4:5]
+; SI-NEXT:    v_or_b32_e32 v6, 0x3ff00000, v7
+; SI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[2:3]|, 0.5
+; SI-NEXT:    v_mov_b32_e32 v2, v1
+; SI-NEXT:    v_cndmask_b32_e32 v3, 0, v6, vcc
 ; SI-NEXT:    v_add_f64 v[2:3], v[4:5], v[2:3]
+; SI-NEXT:    s_mov_b64 s[2:3], s[6:7]
 ; SI-NEXT:    buffer_store_dwordx2 v[2:3], v[0:1], s[0:3], 0 addr64
 ; SI-NEXT:    s_endpgm
 ;
@@ -114,16 +110,15 @@ define amdgpu_kernel void @v_round_f64(ptr addrspace(1) %out, ptr addrspace(1) %
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    s_mov_b64 s[4:5], s[2:3]
 ; CI-NEXT:    buffer_load_dwordx2 v[2:3], v[0:1], s[4:7], 0 addr64
-; CI-NEXT:    s_brev_b32 s2, -2
-; CI-NEXT:    v_mov_b32_e32 v8, 0x3ff00000
+; CI-NEXT:    s_mov_b64 s[2:3], s[6:7]
 ; CI-NEXT:    s_waitcnt vmcnt(0)
 ; CI-NEXT:    v_trunc_f64_e32 v[4:5], v[2:3]
-; CI-NEXT:    v_add_f64 v[6:7], v[2:3], -v[4:5]
-; CI-NEXT:    v_bfi_b32 v2, s2, v8, v3
-; CI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[6:7]|, 0.5
-; CI-NEXT:    s_mov_b64 s[2:3], s[6:7]
-; CI-NEXT:    v_cndmask_b32_e32 v3, 0, v2, vcc
-; CI-NEXT:    v_mov_b32_e32 v2, 0
+; CI-NEXT:    v_and_b32_e32 v6, 0x80000000, v3
+; CI-NEXT:    v_add_f64 v[2:3], v[2:3], -v[4:5]
+; CI-NEXT:    v_or_b32_e32 v6, 0x3ff00000, v6
+; CI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[2:3]|, 0.5
+; CI-NEXT:    v_mov_b32_e32 v2, v1
+; CI-NEXT:    v_cndmask_b32_e32 v3, 0, v6, vcc
 ; CI-NEXT:    v_add_f64 v[2:3], v[4:5], v[2:3]
 ; CI-NEXT:    buffer_store_dwordx2 v[2:3], v[0:1], s[0:3], 0 addr64
 ; CI-NEXT:    s_endpgm
@@ -140,16 +135,16 @@ define amdgpu_kernel void @round_v2f64(ptr addrspace(1) %out, <2 x double> %in)
 ; SI-LABEL: round_v2f64:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0xd
-; SI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
 ; SI-NEXT:    s_mov_b32 s2, -1
 ; SI-NEXT:    s_mov_b32 s9, 0xfffff
 ; SI-NEXT:    s_mov_b32 s8, s2
+; SI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_bfe_u32 s3, s7, 0xb0014
 ; SI-NEXT:    s_addk_i32 s3, 0xfc01
 ; SI-NEXT:    s_lshr_b64 s[10:11], s[8:9], s3
-; SI-NEXT:    s_andn2_b64 s[10:11], s[6:7], s[10:11]
 ; SI-NEXT:    s_and_b32 s12, s7, 0x80000000
+; SI-NEXT:    s_andn2_b64 s[10:11], s[6:7], s[10:11]
 ; SI-NEXT:    s_cmp_lt_i32 s3, 0
 ; SI-NEXT:    s_cselect_b32 s10, 0, s10
 ; SI-NEXT:    s_cselect_b32 s11, s12, s11
@@ -159,33 +154,32 @@ define amdgpu_kernel void @round_v2f64(ptr addrspace(1) %out, <2 x double> %in)
 ; SI-NEXT:    v_mov_b32_e32 v0, s10
 ; SI-NEXT:    v_mov_b32_e32 v1, s11
 ; SI-NEXT:    v_add_f64 v[0:1], s[6:7], -v[0:1]
-; SI-NEXT:    s_brev_b32 s3, -2
-; SI-NEXT:    v_mov_b32_e32 v4, 0x3ff00000
-; SI-NEXT:    v_mov_b32_e32 v2, s7
-; SI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[0:1]|, 0.5
-; SI-NEXT:    v_bfi_b32 v2, s3, v4, v2
-; SI-NEXT:    v_cndmask_b32_e32 v1, 0, v2, vcc
+; SI-NEXT:    s_or_b32 s3, s12, 0x3ff00000
+; SI-NEXT:    v_cmp_ge_f64_e64 s[6:7], |v[0:1]|, 0.5
 ; SI-NEXT:    v_mov_b32_e32 v0, 0
-; SI-NEXT:    s_bfe_u32 s6, s5, 0xb0014
-; SI-NEXT:    v_add_f64 v[2:3], s[10:11], v[0:1]
-; SI-NEXT:    s_add_i32 s10, s6, 0xfffffc01
-; SI-NEXT:    s_lshr_b64 s[6:7], s[8:9], s10
+; SI-NEXT:    s_and_b64 s[6:7], s[6:7], exec
+; SI-NEXT:    s_cselect_b32 s3, s3, 0
+; SI-NEXT:    v_mov_b32_e32 v1, s3
+; SI-NEXT:    s_bfe_u32 s3, s5, 0xb0014
+; SI-NEXT:    s_addk_i32 s3, 0xfc01
+; SI-NEXT:    s_lshr_b64 s[6:7], s[8:9], s3
 ; SI-NEXT:    s_andn2_b64 s[6:7], s[4:5], s[6:7]
 ; SI-NEXT:    s_and_b32 s8, s5, 0x80000000
-; SI-NEXT:    s_cmp_lt_i32 s10, 0
+; SI-NEXT:    s_cmp_lt_i32 s3, 0
 ; SI-NEXT:    s_cselect_b32 s6, 0, s6
 ; SI-NEXT:    s_cselect_b32 s7, s8, s7
-; SI-NEXT:    s_cmp_gt_i32 s10, 51
+; SI-NEXT:    s_cmp_gt_i32 s3, 51
 ; SI-NEXT:    s_cselect_b32 s6, s4, s6
 ; SI-NEXT:    s_cselect_b32 s7, s5, s7
-; SI-NEXT:    v_mov_b32_e32 v0, s6
-; SI-NEXT:    v_mov_b32_e32 v1, s7
-; SI-NEXT:    v_add_f64 v[0:1], s[4:5], -v[0:1]
-; SI-NEXT:    v_mov_b32_e32 v5, s5
-; SI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[0:1]|, 0.5
-; SI-NEXT:    v_bfi_b32 v4, s3, v4, v5
-; SI-NEXT:    v_cndmask_b32_e32 v1, 0, v4, vcc
-; SI-NEXT:    v_mov_b32_e32 v0, 0
+; SI-NEXT:    v_mov_b32_e32 v2, s6
+; SI-NEXT:    v_mov_b32_e32 v3, s7
+; SI-NEXT:    v_add_f64 v[4:5], s[4:5], -v[2:3]
+; SI-NEXT:    s_or_b32 s3, s8, 0x3ff00000
+; SI-NEXT:    v_cmp_ge_f64_e64 s[4:5], |v[4:5]|, 0.5
+; SI-NEXT:    v_add_f64 v[2:3], s[10:11], v[0:1]
+; SI-NEXT:    s_and_b64 s[4:5], s[4:5], exec
+; SI-NEXT:    s_cselect_b32 s3, s3, 0
+; SI-NEXT:    v_mov_b32_e32 v1, s3
 ; SI-NEXT:    v_add_f64 v[0:1], s[6:7], v[0:1]
 ; SI-NEXT:    s_mov_b32 s3, 0xf000
 ; SI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
@@ -195,26 +189,25 @@ define amdgpu_kernel void @round_v2f64(ptr addrspace(1) %out, <2 x double> %in)
 ; CI:       ; %bb.0:
 ; CI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0xd
 ; CI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; CI-NEXT:    s_brev_b32 s2, -2
-; CI-NEXT:    v_mov_b32_e32 v6, 0x3ff00000
+; CI-NEXT:    s_mov_b32 s8, 0
 ; CI-NEXT:    s_mov_b32 s3, 0xf000
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    v_trunc_f64_e32 v[0:1], s[6:7]
-; CI-NEXT:    v_mov_b32_e32 v4, s7
-; CI-NEXT:    v_add_f64 v[2:3], s[6:7], -v[0:1]
-; CI-NEXT:    v_bfi_b32 v4, s2, v6, v4
-; CI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[2:3]|, 0.5
-; CI-NEXT:    v_mov_b32_e32 v2, 0
-; CI-NEXT:    v_cndmask_b32_e32 v3, 0, v4, vcc
 ; CI-NEXT:    v_trunc_f64_e32 v[4:5], s[4:5]
-; CI-NEXT:    v_add_f64 v[2:3], v[0:1], v[2:3]
-; CI-NEXT:    v_add_f64 v[0:1], s[4:5], -v[4:5]
-; CI-NEXT:    v_mov_b32_e32 v7, s5
-; CI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[0:1]|, 0.5
-; CI-NEXT:    v_bfi_b32 v6, s2, v6, v7
-; CI-NEXT:    v_cndmask_b32_e32 v1, 0, v6, vcc
-; CI-NEXT:    v_mov_b32_e32 v0, 0
-; CI-NEXT:    v_add_f64 v[0:1], v[4:5], v[0:1]
+; CI-NEXT:    v_add_f64 v[2:3], s[6:7], -v[0:1]
+; CI-NEXT:    s_and_b32 s2, s7, 0x80000000
+; CI-NEXT:    v_cmp_ge_f64_e64 s[6:7], |v[2:3]|, 0.5
+; CI-NEXT:    s_or_b32 s2, s2, 0x3ff00000
+; CI-NEXT:    v_add_f64 v[6:7], s[4:5], -v[4:5]
+; CI-NEXT:    s_and_b64 s[6:7], s[6:7], exec
+; CI-NEXT:    s_cselect_b32 s9, s2, 0
+; CI-NEXT:    v_cmp_ge_f64_e64 s[6:7], |v[6:7]|, 0.5
+; CI-NEXT:    s_and_b32 s2, s5, 0x80000000
+; CI-NEXT:    s_or_b32 s2, s2, 0x3ff00000
+; CI-NEXT:    s_and_b64 s[4:5], s[6:7], exec
+; CI-NEXT:    v_add_f64 v[2:3], v[0:1], s[8:9]
+; CI-NEXT:    s_cselect_b32 s9, s2, 0
+; CI-NEXT:    v_add_f64 v[0:1], v[4:5], s[8:9]
 ; CI-NEXT:    s_mov_b32 s2, -1
 ; CI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; CI-NEXT:    s_endpgm
@@ -229,92 +222,92 @@ define amdgpu_kernel void @round_v4f64(ptr addrspace(1) %out, <4 x double> %in)
 ; SI-NEXT:    s_load_dwordx8 s[4:11], s[0:1], 0x11
 ; SI-NEXT:    s_mov_b32 s2, -1
 ; SI-NEXT:    s_mov_b32 s13, 0xfffff
-; SI-NEXT:    v_mov_b32_e32 v8, 0x3ff00000
-; SI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; SI-NEXT:    s_waitcnt lgkmcnt(0)
-; SI-NEXT:    s_bfe_u32 s12, s7, 0xb0014
-; SI-NEXT:    s_add_i32 s16, s12, 0xfffffc01
 ; SI-NEXT:    s_mov_b32 s12, s2
-; SI-NEXT:    s_lshr_b64 s[14:15], s[12:13], s16
+; SI-NEXT:    v_mov_b32_e32 v4, 0
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    s_bfe_u32 s3, s7, 0xb0014
+; SI-NEXT:    s_addk_i32 s3, 0xfc01
+; SI-NEXT:    s_lshr_b64 s[14:15], s[12:13], s3
+; SI-NEXT:    s_and_b32 s16, s7, 0x80000000
 ; SI-NEXT:    s_andn2_b64 s[14:15], s[6:7], s[14:15]
-; SI-NEXT:    s_and_b32 s17, s7, 0x80000000
-; SI-NEXT:    s_cmp_lt_i32 s16, 0
+; SI-NEXT:    s_cmp_lt_i32 s3, 0
 ; SI-NEXT:    s_cselect_b32 s14, 0, s14
-; SI-NEXT:    s_cselect_b32 s15, s17, s15
-; SI-NEXT:    s_cmp_gt_i32 s16, 51
+; SI-NEXT:    s_cselect_b32 s15, s16, s15
+; SI-NEXT:    s_cmp_gt_i32 s3, 51
 ; SI-NEXT:    s_cselect_b32 s14, s6, s14
 ; SI-NEXT:    s_cselect_b32 s15, s7, s15
 ; SI-NEXT:    v_mov_b32_e32 v0, s14
 ; SI-NEXT:    v_mov_b32_e32 v1, s15
 ; SI-NEXT:    v_add_f64 v[0:1], s[6:7], -v[0:1]
-; SI-NEXT:    s_brev_b32 s16, -2
-; SI-NEXT:    v_mov_b32_e32 v2, s7
-; SI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[0:1]|, 0.5
-; SI-NEXT:    v_bfi_b32 v2, s16, v8, v2
-; SI-NEXT:    v_cndmask_b32_e32 v1, 0, v2, vcc
-; SI-NEXT:    v_mov_b32_e32 v0, 0
-; SI-NEXT:    s_bfe_u32 s6, s5, 0xb0014
-; SI-NEXT:    v_add_f64 v[2:3], s[14:15], v[0:1]
-; SI-NEXT:    s_add_i32 s14, s6, 0xfffffc01
-; SI-NEXT:    s_lshr_b64 s[6:7], s[12:13], s14
+; SI-NEXT:    s_or_b32 s3, s16, 0x3ff00000
+; SI-NEXT:    v_cmp_ge_f64_e64 s[6:7], |v[0:1]|, 0.5
+; SI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; SI-NEXT:    s_and_b64 s[6:7], s[6:7], exec
+; SI-NEXT:    s_cselect_b32 s3, s3, 0
+; SI-NEXT:    v_mov_b32_e32 v5, s3
+; SI-NEXT:    s_bfe_u32 s3, s5, 0xb0014
+; SI-NEXT:    s_addk_i32 s3, 0xfc01
+; SI-NEXT:    s_lshr_b64 s[6:7], s[12:13], s3
 ; SI-NEXT:    s_andn2_b64 s[6:7], s[4:5], s[6:7]
-; SI-NEXT:    s_and_b32 s15, s5, 0x80000000
-; SI-NEXT:    s_cmp_lt_i32 s14, 0
+; SI-NEXT:    s_and_b32 s16, s5, 0x80000000
+; SI-NEXT:    s_cmp_lt_i32 s3, 0
 ; SI-NEXT:    s_cselect_b32 s6, 0, s6
-; SI-NEXT:    s_cselect_b32 s7, s15, s7
-; SI-NEXT:    s_cmp_gt_i32 s14, 51
+; SI-NEXT:    s_cselect_b32 s7, s16, s7
+; SI-NEXT:    s_cmp_gt_i32 s3, 51
 ; SI-NEXT:    s_cselect_b32 s6, s4, s6
 ; SI-NEXT:    s_cselect_b32 s7, s5, s7
 ; SI-NEXT:    v_mov_b32_e32 v0, s6
 ; SI-NEXT:    v_mov_b32_e32 v1, s7
 ; SI-NEXT:    v_add_f64 v[0:1], s[4:5], -v[0:1]
-; SI-NEXT:    s_bfe_u32 s4, s11, 0xb0014
-; SI-NEXT:    s_add_i32 s14, s4, 0xfffffc01
-; SI-NEXT:    v_mov_b32_e32 v4, s5
-; SI-NEXT:    s_lshr_b64 s[4:5], s[12:13], s14
+; SI-NEXT:    s_or_b32 s3, s16, 0x3ff00000
+; SI-NEXT:    v_cmp_ge_f64_e64 s[4:5], |v[0:1]|, 0.5
+; SI-NEXT:    v_add_f64 v[2:3], s[14:15], v[4:5]
+; SI-NEXT:    s_and_b64 s[4:5], s[4:5], exec
+; SI-NEXT:    s_cselect_b32 s3, s3, 0
+; SI-NEXT:    v_mov_b32_e32 v5, s3
+; SI-NEXT:    s_bfe_u32 s3, s11, 0xb0014
+; SI-NEXT:    s_addk_i32 s3, 0xfc01
+; SI-NEXT:    s_lshr_b64 s[4:5], s[12:13], s3
 ; SI-NEXT:    s_andn2_b64 s[4:5], s[10:11], s[4:5]
-; SI-NEXT:    s_and_b32 s15, s11, 0x80000000
-; SI-NEXT:    s_cmp_lt_i32 s14, 0
-; SI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[0:1]|, 0.5
+; SI-NEXT:    s_and_b32 s14, s11, 0x80000000
+; SI-NEXT:    s_cmp_lt_i32 s3, 0
 ; SI-NEXT:    s_cselect_b32 s4, 0, s4
-; SI-NEXT:    s_cselect_b32 s5, s15, s5
-; SI-NEXT:    s_cmp_gt_i32 s14, 51
-; SI-NEXT:    v_bfi_b32 v4, s16, v8, v4
+; SI-NEXT:    s_cselect_b32 s5, s14, s5
+; SI-NEXT:    s_cmp_gt_i32 s3, 51
 ; SI-NEXT:    s_cselect_b32 s4, s10, s4
-; SI-NEXT:    v_cndmask_b32_e32 v1, 0, v4, vcc
 ; SI-NEXT:    s_cselect_b32 s5, s11, s5
-; SI-NEXT:    v_mov_b32_e32 v4, s4
-; SI-NEXT:    v_mov_b32_e32 v5, s5
-; SI-NEXT:    v_add_f64 v[4:5], s[10:11], -v[4:5]
-; SI-NEXT:    v_mov_b32_e32 v6, s11
-; SI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[4:5]|, 0.5
-; SI-NEXT:    v_bfi_b32 v6, s16, v8, v6
-; SI-NEXT:    v_cndmask_b32_e32 v5, 0, v6, vcc
-; SI-NEXT:    v_mov_b32_e32 v4, 0
+; SI-NEXT:    v_mov_b32_e32 v0, s4
+; SI-NEXT:    v_mov_b32_e32 v1, s5
+; SI-NEXT:    v_add_f64 v[6:7], s[10:11], -v[0:1]
+; SI-NEXT:    v_add_f64 v[0:1], s[6:7], v[4:5]
+; SI-NEXT:    v_cmp_ge_f64_e64 s[6:7], |v[6:7]|, 0.5
+; SI-NEXT:    s_or_b32 s3, s14, 0x3ff00000
+; SI-NEXT:    s_and_b64 s[6:7], s[6:7], exec
+; SI-NEXT:    s_cselect_b32 s3, s3, 0
+; SI-NEXT:    v_mov_b32_e32 v5, s3
+; SI-NEXT:    s_bfe_u32 s3, s9, 0xb0014
+; SI-NEXT:    s_addk_i32 s3, 0xfc01
+; SI-NEXT:    s_lshr_b64 s[6:7], s[12:13], s3
+; SI-NEXT:    s_andn2_b64 s[6:7], s[8:9], s[6:7]
+; SI-NEXT:    s_and_b32 s10, s9, 0x80000000
+; SI-NEXT:    s_cmp_lt_i32 s3, 0
+; SI-NEXT:    s_cselect_b32 s6, 0, s6
+; SI-NEXT:    s_cselect_b32 s7, s10, s7
+; SI-NEXT:    s_cmp_gt_i32 s3, 51
+; SI-NEXT:    s_cselect_b32 s6, s8, s6
+; SI-NEXT:    s_cselect_b32 s7, s9, s7
+; SI-NEXT:    v_mov_b32_e32 v6, s6
+; SI-NEXT:    v_mov_b32_e32 v7, s7
+; SI-NEXT:    v_add_f64 v[8:9], s[8:9], -v[6:7]
 ; SI-NEXT:    v_add_f64 v[6:7], s[4:5], v[4:5]
-; SI-NEXT:    s_bfe_u32 s4, s9, 0xb0014
-; SI-NEXT:    s_add_i32 s10, s4, 0xfffffc01
-; SI-NEXT:    s_lshr_b64 s[4:5], s[12:13], s10
-; SI-NEXT:    s_andn2_b64 s[4:5], s[8:9], s[4:5]
-; SI-NEXT:    s_and_b32 s11, s9, 0x80000000
-; SI-NEXT:    s_cmp_lt_i32 s10, 0
-; SI-NEXT:    s_cselect_b32 s4, 0, s4
-; SI-NEXT:    s_cselect_b32 s5, s11, s5
-; SI-NEXT:    s_cmp_gt_i32 s10, 51
-; SI-NEXT:    s_cselect_b32 s4, s8, s4
-; SI-NEXT:    s_cselect_b32 s5, s9, s5
-; SI-NEXT:    v_mov_b32_e32 v4, s4
-; SI-NEXT:    v_mov_b32_e32 v5, s5
-; SI-NEXT:    v_add_f64 v[4:5], s[8:9], -v[4:5]
-; SI-NEXT:    v_mov_b32_e32 v9, s9
-; SI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[4:5]|, 0.5
-; SI-NEXT:    v_bfi_b32 v8, s16, v8, v9
-; SI-NEXT:    v_cndmask_b32_e32 v5, 0, v8, vcc
-; SI-NEXT:    v_mov_b32_e32 v4, 0
-; SI-NEXT:    v_mov_b32_e32 v0, 0
-; SI-NEXT:    v_add_f64 v[4:5], s[4:5], v[4:5]
+; SI-NEXT:    v_cmp_ge_f64_e64 s[4:5], |v[8:9]|, 0.5
+; SI-NEXT:    s_or_b32 s3, s10, 0x3ff00000
+; SI-NEXT:    s_and_b64 s[4:5], s[4:5], exec
+; SI-NEXT:    s_cselect_b32 s3, s3, 0
+; SI-NEXT:    v_mov_b32_e32 v5, s3
+; SI-NEXT:    v_add_f64 v[4:5], s[6:7], v[4:5]
 ; SI-NEXT:    s_mov_b32 s3, 0xf000
-; SI-NEXT:    v_add_f64 v[0:1], s[6:7], v[0:1]
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    buffer_store_dwordx4 v[4:7], off, s[0:3], 0 offset:16
 ; SI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
@@ -322,44 +315,43 @@ define amdgpu_kernel void @round_v4f64(ptr addrspace(1) %out, <4 x double> %in)
 ; CI-LABEL: round_v4f64:
 ; CI:       ; %bb.0:
 ; CI-NEXT:    s_load_dwordx8 s[4:11], s[0:1], 0x11
-; CI-NEXT:    s_brev_b32 s12, -2
-; CI-NEXT:    v_mov_b32_e32 v12, 0x3ff00000
+; CI-NEXT:    s_mov_b32 s12, 0
 ; CI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
 ; CI-NEXT:    s_mov_b32 s3, 0xf000
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    v_trunc_f64_e32 v[0:1], s[6:7]
-; CI-NEXT:    v_mov_b32_e32 v4, s7
+; CI-NEXT:    v_trunc_f64_e32 v[4:5], s[4:5]
 ; CI-NEXT:    v_add_f64 v[2:3], s[6:7], -v[0:1]
-; CI-NEXT:    v_bfi_b32 v4, s12, v12, v4
-; CI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[2:3]|, 0.5
-; CI-NEXT:    v_trunc_f64_e32 v[8:9], s[4:5]
-; CI-NEXT:    v_cndmask_b32_e32 v3, 0, v4, vcc
-; CI-NEXT:    v_mov_b32_e32 v2, 0
-; CI-NEXT:    v_add_f64 v[2:3], v[0:1], v[2:3]
-; CI-NEXT:    v_add_f64 v[0:1], s[4:5], -v[8:9]
-; CI-NEXT:    v_mov_b32_e32 v4, s5
-; CI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[0:1]|, 0.5
-; CI-NEXT:    v_bfi_b32 v4, s12, v12, v4
-; CI-NEXT:    v_cndmask_b32_e32 v1, 0, v4, vcc
-; CI-NEXT:    v_trunc_f64_e32 v[4:5], s[10:11]
-; CI-NEXT:    v_mov_b32_e32 v10, s11
-; CI-NEXT:    v_add_f64 v[6:7], s[10:11], -v[4:5]
-; CI-NEXT:    v_bfi_b32 v10, s12, v12, v10
-; CI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[6:7]|, 0.5
-; CI-NEXT:    v_mov_b32_e32 v6, 0
-; CI-NEXT:    v_cndmask_b32_e32 v7, 0, v10, vcc
-; CI-NEXT:    v_trunc_f64_e32 v[10:11], s[8:9]
-; CI-NEXT:    v_add_f64 v[6:7], v[4:5], v[6:7]
-; CI-NEXT:    v_add_f64 v[4:5], s[8:9], -v[10:11]
-; CI-NEXT:    v_mov_b32_e32 v13, s9
-; CI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[4:5]|, 0.5
-; CI-NEXT:    v_bfi_b32 v12, s12, v12, v13
-; CI-NEXT:    v_cndmask_b32_e32 v5, 0, v12, vcc
-; CI-NEXT:    v_mov_b32_e32 v4, 0
-; CI-NEXT:    v_mov_b32_e32 v0, 0
-; CI-NEXT:    v_add_f64 v[4:5], v[10:11], v[4:5]
+; CI-NEXT:    s_and_b32 s2, s7, 0x80000000
+; CI-NEXT:    v_cmp_ge_f64_e64 s[6:7], |v[2:3]|, 0.5
+; CI-NEXT:    s_or_b32 s2, s2, 0x3ff00000
+; CI-NEXT:    v_add_f64 v[6:7], s[4:5], -v[4:5]
+; CI-NEXT:    s_and_b64 s[6:7], s[6:7], exec
+; CI-NEXT:    s_cselect_b32 s13, s2, 0
+; CI-NEXT:    v_cmp_ge_f64_e64 s[6:7], |v[6:7]|, 0.5
+; CI-NEXT:    s_and_b32 s2, s5, 0x80000000
+; CI-NEXT:    s_or_b32 s2, s2, 0x3ff00000
+; CI-NEXT:    v_trunc_f64_e32 v[6:7], s[10:11]
+; CI-NEXT:    s_and_b64 s[4:5], s[6:7], exec
+; CI-NEXT:    v_add_f64 v[2:3], v[0:1], s[12:13]
+; CI-NEXT:    s_cselect_b32 s13, s2, 0
+; CI-NEXT:    v_add_f64 v[8:9], s[10:11], -v[6:7]
+; CI-NEXT:    v_add_f64 v[0:1], v[4:5], s[12:13]
+; CI-NEXT:    v_trunc_f64_e32 v[4:5], s[8:9]
+; CI-NEXT:    v_cmp_ge_f64_e64 s[4:5], |v[8:9]|, 0.5
+; CI-NEXT:    s_and_b32 s2, s11, 0x80000000
+; CI-NEXT:    s_or_b32 s2, s2, 0x3ff00000
+; CI-NEXT:    v_add_f64 v[8:9], s[8:9], -v[4:5]
+; CI-NEXT:    s_and_b64 s[4:5], s[4:5], exec
+; CI-NEXT:    s_cselect_b32 s13, s2, 0
+; CI-NEXT:    v_cmp_ge_f64_e64 s[4:5], |v[8:9]|, 0.5
+; CI-NEXT:    s_and_b32 s2, s9, 0x80000000
+; CI-NEXT:    s_or_b32 s2, s2, 0x3ff00000
+; CI-NEXT:    s_and_b64 s[4:5], s[4:5], exec
+; CI-NEXT:    v_add_f64 v[6:7], v[6:7], s[12:13]
+; CI-NEXT:    s_cselect_b32 s13, s2, 0
+; CI-NEXT:    v_add_f64 v[4:5], v[4:5], s[12:13]
 ; CI-NEXT:    s_mov_b32 s2, -1
-; CI-NEXT:    v_add_f64 v[0:1], v[8:9], v[0:1]
 ; CI-NEXT:    buffer_store_dwordx4 v[4:7], off, s[0:3], 0 offset:16
 ; CI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; CI-NEXT:    s_endpgm
@@ -374,174 +366,174 @@ define amdgpu_kernel void @round_v8f64(ptr addrspace(1) %out, <8 x double> %in)
 ; SI-NEXT:    s_load_dwordx16 s[4:19], s[0:1], 0x19
 ; SI-NEXT:    s_mov_b32 s2, -1
 ; SI-NEXT:    s_mov_b32 s21, 0xfffff
-; SI-NEXT:    v_mov_b32_e32 v8, 0x3ff00000
-; SI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; SI-NEXT:    s_waitcnt lgkmcnt(0)
-; SI-NEXT:    s_bfe_u32 s20, s7, 0xb0014
-; SI-NEXT:    s_add_i32 s24, s20, 0xfffffc01
 ; SI-NEXT:    s_mov_b32 s20, s2
-; SI-NEXT:    s_lshr_b64 s[22:23], s[20:21], s24
+; SI-NEXT:    v_mov_b32_e32 v8, 0
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    s_bfe_u32 s3, s7, 0xb0014
+; SI-NEXT:    s_addk_i32 s3, 0xfc01
+; SI-NEXT:    s_lshr_b64 s[22:23], s[20:21], s3
+; SI-NEXT:    s_and_b32 s24, s7, 0x80000000
 ; SI-NEXT:    s_andn2_b64 s[22:23], s[6:7], s[22:23]
-; SI-NEXT:    s_and_b32 s25, s7, 0x80000000
-; SI-NEXT:    s_cmp_lt_i32 s24, 0
+; SI-NEXT:    s_cmp_lt_i32 s3, 0
 ; SI-NEXT:    s_cselect_b32 s22, 0, s22
-; SI-NEXT:    s_cselect_b32 s23, s25, s23
-; SI-NEXT:    s_cmp_gt_i32 s24, 51
+; SI-NEXT:    s_cselect_b32 s23, s24, s23
+; SI-NEXT:    s_cmp_gt_i32 s3, 51
 ; SI-NEXT:    s_cselect_b32 s22, s6, s22
 ; SI-NEXT:    s_cselect_b32 s23, s7, s23
 ; SI-NEXT:    v_mov_b32_e32 v0, s22
 ; SI-NEXT:    v_mov_b32_e32 v1, s23
 ; SI-NEXT:    v_add_f64 v[0:1], s[6:7], -v[0:1]
-; SI-NEXT:    s_brev_b32 s6, -2
-; SI-NEXT:    v_mov_b32_e32 v2, s7
-; SI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[0:1]|, 0.5
-; SI-NEXT:    v_bfi_b32 v2, s6, v8, v2
-; SI-NEXT:    s_bfe_u32 s7, s5, 0xb0014
-; SI-NEXT:    v_cndmask_b32_e32 v1, 0, v2, vcc
-; SI-NEXT:    v_mov_b32_e32 v0, 0
-; SI-NEXT:    s_addk_i32 s7, 0xfc01
-; SI-NEXT:    v_add_f64 v[2:3], s[22:23], v[0:1]
-; SI-NEXT:    s_lshr_b64 s[22:23], s[20:21], s7
-; SI-NEXT:    s_andn2_b64 s[22:23], s[4:5], s[22:23]
+; SI-NEXT:    s_or_b32 s3, s24, 0x3ff00000
+; SI-NEXT:    v_cmp_ge_f64_e64 s[6:7], |v[0:1]|, 0.5
+; SI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; SI-NEXT:    s_and_b64 s[6:7], s[6:7], exec
+; SI-NEXT:    s_cselect_b32 s3, s3, 0
+; SI-NEXT:    v_mov_b32_e32 v9, s3
+; SI-NEXT:    s_bfe_u32 s3, s5, 0xb0014
+; SI-NEXT:    s_addk_i32 s3, 0xfc01
+; SI-NEXT:    s_lshr_b64 s[6:7], s[20:21], s3
+; SI-NEXT:    s_andn2_b64 s[6:7], s[4:5], s[6:7]
 ; SI-NEXT:    s_and_b32 s24, s5, 0x80000000
-; SI-NEXT:    s_cmp_lt_i32 s7, 0
-; SI-NEXT:    s_cselect_b32 s22, 0, s22
-; SI-NEXT:    s_cselect_b32 s23, s24, s23
-; SI-NEXT:    s_cmp_gt_i32 s7, 51
-; SI-NEXT:    s_cselect_b32 s22, s4, s22
-; SI-NEXT:    s_cselect_b32 s23, s5, s23
-; SI-NEXT:    v_mov_b32_e32 v0, s22
-; SI-NEXT:    v_mov_b32_e32 v1, s23
+; SI-NEXT:    s_cmp_lt_i32 s3, 0
+; SI-NEXT:    s_cselect_b32 s6, 0, s6
+; SI-NEXT:    s_cselect_b32 s7, s24, s7
+; SI-NEXT:    s_cmp_gt_i32 s3, 51
+; SI-NEXT:    s_cselect_b32 s6, s4, s6
+; SI-NEXT:    s_cselect_b32 s7, s5, s7
+; SI-NEXT:    v_mov_b32_e32 v0, s6
+; SI-NEXT:    v_mov_b32_e32 v1, s7
 ; SI-NEXT:    v_add_f64 v[0:1], s[4:5], -v[0:1]
-; SI-NEXT:    v_mov_b32_e32 v4, s5
-; SI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[0:1]|, 0.5
-; SI-NEXT:    s_bfe_u32 s4, s11, 0xb0014
-; SI-NEXT:    v_bfi_b32 v4, s6, v8, v4
-; SI-NEXT:    s_add_i32 s7, s4, 0xfffffc01
-; SI-NEXT:    v_cndmask_b32_e32 v1, 0, v4, vcc
-; SI-NEXT:    v_mov_b32_e32 v0, 0
-; SI-NEXT:    s_lshr_b64 s[4:5], s[20:21], s7
-; SI-NEXT:    v_add_f64 v[0:1], s[22:23], v[0:1]
+; SI-NEXT:    s_or_b32 s3, s24, 0x3ff00000
+; SI-NEXT:    v_cmp_ge_f64_e64 s[4:5], |v[0:1]|, 0.5
+; SI-NEXT:    v_add_f64 v[2:3], s[22:23], v[8:9]
+; SI-NEXT:    s_and_b64 s[4:5], s[4:5], exec
+; SI-NEXT:    s_cselect_b32 s3, s3, 0
+; SI-NEXT:    v_mov_b32_e32 v9, s3
+; SI-NEXT:    s_bfe_u32 s3, s11, 0xb0014
+; SI-NEXT:    s_addk_i32 s3, 0xfc01
+; SI-NEXT:    s_lshr_b64 s[4:5], s[20:21], s3
 ; SI-NEXT:    s_andn2_b64 s[4:5], s[10:11], s[4:5]
 ; SI-NEXT:    s_and_b32 s22, s11, 0x80000000
-; SI-NEXT:    s_cmp_lt_i32 s7, 0
+; SI-NEXT:    s_cmp_lt_i32 s3, 0
 ; SI-NEXT:    s_cselect_b32 s4, 0, s4
 ; SI-NEXT:    s_cselect_b32 s5, s22, s5
-; SI-NEXT:    s_cmp_gt_i32 s7, 51
+; SI-NEXT:    s_cmp_gt_i32 s3, 51
 ; SI-NEXT:    s_cselect_b32 s4, s10, s4
 ; SI-NEXT:    s_cselect_b32 s5, s11, s5
-; SI-NEXT:    v_mov_b32_e32 v4, s4
-; SI-NEXT:    v_mov_b32_e32 v5, s5
-; SI-NEXT:    v_add_f64 v[4:5], s[10:11], -v[4:5]
-; SI-NEXT:    v_mov_b32_e32 v6, s11
-; SI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[4:5]|, 0.5
-; SI-NEXT:    v_bfi_b32 v6, s6, v8, v6
-; SI-NEXT:    v_cndmask_b32_e32 v5, 0, v6, vcc
-; SI-NEXT:    v_mov_b32_e32 v4, 0
-; SI-NEXT:    v_add_f64 v[6:7], s[4:5], v[4:5]
-; SI-NEXT:    s_bfe_u32 s4, s9, 0xb0014
-; SI-NEXT:    s_add_i32 s7, s4, 0xfffffc01
-; SI-NEXT:    s_lshr_b64 s[4:5], s[20:21], s7
-; SI-NEXT:    s_andn2_b64 s[4:5], s[8:9], s[4:5]
+; SI-NEXT:    v_mov_b32_e32 v0, s4
+; SI-NEXT:    v_mov_b32_e32 v1, s5
+; SI-NEXT:    v_add_f64 v[4:5], s[10:11], -v[0:1]
+; SI-NEXT:    v_add_f64 v[0:1], s[6:7], v[8:9]
+; SI-NEXT:    v_cmp_ge_f64_e64 s[6:7], |v[4:5]|, 0.5
+; SI-NEXT:    s_or_b32 s3, s22, 0x3ff00000
+; SI-NEXT:    s_and_b64 s[6:7], s[6:7], exec
+; SI-NEXT:    s_cselect_b32 s3, s3, 0
+; SI-NEXT:    v_mov_b32_e32 v9, s3
+; SI-NEXT:    s_bfe_u32 s3, s9, 0xb0014
+; SI-NEXT:    s_addk_i32 s3, 0xfc01
+; SI-NEXT:    s_lshr_b64 s[6:7], s[20:21], s3
+; SI-NEXT:    s_andn2_b64 s[6:7], s[8:9], s[6:7]
 ; SI-NEXT:    s_and_b32 s10, s9, 0x80000000
-; SI-NEXT:    s_cmp_lt_i32 s7, 0
-; SI-NEXT:    s_cselect_b32 s4, 0, s4
-; SI-NEXT:    s_cselect_b32 s5, s10, s5
-; SI-NEXT:    s_cmp_gt_i32 s7, 51
-; SI-NEXT:    s_cselect_b32 s4, s8, s4
-; SI-NEXT:    s_cselect_b32 s5, s9, s5
-; SI-NEXT:    v_mov_b32_e32 v4, s4
-; SI-NEXT:    v_mov_b32_e32 v5, s5
+; SI-NEXT:    s_cmp_lt_i32 s3, 0
+; SI-NEXT:    s_cselect_b32 s6, 0, s6
+; SI-NEXT:    s_cselect_b32 s7, s10, s7
+; SI-NEXT:    s_cmp_gt_i32 s3, 51
+; SI-NEXT:    s_cselect_b32 s6, s8, s6
+; SI-NEXT:    s_cselect_b32 s7, s9, s7
+; SI-NEXT:    v_mov_b32_e32 v4, s6
+; SI-NEXT:    v_mov_b32_e32 v5, s7
 ; SI-NEXT:    v_add_f64 v[4:5], s[8:9], -v[4:5]
-; SI-NEXT:    v_mov_b32_e32 v9, s9
-; SI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[4:5]|, 0.5
-; SI-NEXT:    v_bfi_b32 v9, s6, v8, v9
-; SI-NEXT:    v_cndmask_b32_e32 v5, 0, v9, vcc
-; SI-NEXT:    v_mov_b32_e32 v4, 0
-; SI-NEXT:    v_add_f64 v[4:5], s[4:5], v[4:5]
-; SI-NEXT:    s_bfe_u32 s4, s15, 0xb0014
-; SI-NEXT:    s_add_i32 s7, s4, 0xfffffc01
-; SI-NEXT:    s_lshr_b64 s[4:5], s[20:21], s7
+; SI-NEXT:    v_add_f64 v[6:7], s[4:5], v[8:9]
+; SI-NEXT:    v_cmp_ge_f64_e64 s[4:5], |v[4:5]|, 0.5
+; SI-NEXT:    s_or_b32 s3, s10, 0x3ff00000
+; SI-NEXT:    s_and_b64 s[4:5], s[4:5], exec
+; SI-NEXT:    s_cselect_b32 s3, s3, 0
+; SI-NEXT:    v_mov_b32_e32 v9, s3
+; SI-NEXT:    s_bfe_u32 s3, s15, 0xb0014
+; SI-NEXT:    s_addk_i32 s3, 0xfc01
+; SI-NEXT:    s_lshr_b64 s[4:5], s[20:21], s3
 ; SI-NEXT:    s_andn2_b64 s[4:5], s[14:15], s[4:5]
 ; SI-NEXT:    s_and_b32 s8, s15, 0x80000000
-; SI-NEXT:    s_cmp_lt_i32 s7, 0
+; SI-NEXT:    s_cmp_lt_i32 s3, 0
 ; SI-NEXT:    s_cselect_b32 s4, 0, s4
 ; SI-NEXT:    s_cselect_b32 s5, s8, s5
-; SI-NEXT:    s_cmp_gt_i32 s7, 51
-; SI-NEXT:    s_cselect_b32 s5, s15, s5
+; SI-NEXT:    s_cmp_gt_i32 s3, 51
 ; SI-NEXT:    s_cselect_b32 s4, s14, s4
-; SI-NEXT:    v_mov_b32_e32 v10, s5
-; SI-NEXT:    v_mov_b32_e32 v9, s4
-; SI-NEXT:    v_add_f64 v[9:10], s[14:15], -v[9:10]
-; SI-NEXT:    v_mov_b32_e32 v11, s15
-; SI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[9:10]|, 0.5
-; SI-NEXT:    v_bfi_b32 v11, s6, v8, v11
-; SI-NEXT:    v_cndmask_b32_e32 v10, 0, v11, vcc
-; SI-NEXT:    v_mov_b32_e32 v9, 0
-; SI-NEXT:    v_add_f64 v[10:11], s[4:5], v[9:10]
-; SI-NEXT:    s_bfe_u32 s4, s13, 0xb0014
-; SI-NEXT:    s_add_i32 s7, s4, 0xfffffc01
-; SI-NEXT:    s_lshr_b64 s[4:5], s[20:21], s7
-; SI-NEXT:    s_andn2_b64 s[4:5], s[12:13], s[4:5]
+; SI-NEXT:    s_cselect_b32 s5, s15, s5
+; SI-NEXT:    v_mov_b32_e32 v4, s4
+; SI-NEXT:    v_mov_b32_e32 v5, s5
+; SI-NEXT:    v_add_f64 v[10:11], s[14:15], -v[4:5]
+; SI-NEXT:    v_add_f64 v[4:5], s[6:7], v[8:9]
+; SI-NEXT:    v_cmp_ge_f64_e64 s[6:7], |v[10:11]|, 0.5
+; SI-NEXT:    s_or_b32 s3, s8, 0x3ff00000
+; SI-NEXT:    s_and_b64 s[6:7], s[6:7], exec
+; SI-NEXT:    s_cselect_b32 s3, s3, 0
+; SI-NEXT:    v_mov_b32_e32 v9, s3
+; SI-NEXT:    s_bfe_u32 s3, s13, 0xb0014
+; SI-NEXT:    s_addk_i32 s3, 0xfc01
+; SI-NEXT:    s_lshr_b64 s[6:7], s[20:21], s3
+; SI-NEXT:    s_andn2_b64 s[6:7], s[12:13], s[6:7]
 ; SI-NEXT:    s_and_b32 s8, s13, 0x80000000
-; SI-NEXT:    s_cmp_lt_i32 s7, 0
+; SI-NEXT:    s_cmp_lt_i32 s3, 0
+; SI-NEXT:    s_cselect_b32 s6, 0, s6
+; SI-NEXT:    s_cselect_b32 s7, s8, s7
+; SI-NEXT:    s_cmp_gt_i32 s3, 51
+; SI-NEXT:    s_cselect_b32 s7, s13, s7
+; SI-NEXT:    s_cselect_b32 s6, s12, s6
+; SI-NEXT:    v_mov_b32_e32 v11, s7
+; SI-NEXT:    v_mov_b32_e32 v10, s6
+; SI-NEXT:    v_add_f64 v[10:11], s[12:13], -v[10:11]
+; SI-NEXT:    v_add_f64 v[12:13], s[4:5], v[8:9]
+; SI-NEXT:    v_cmp_ge_f64_e64 s[4:5], |v[10:11]|, 0.5
+; SI-NEXT:    s_or_b32 s3, s8, 0x3ff00000
+; SI-NEXT:    s_and_b64 s[4:5], s[4:5], exec
+; SI-NEXT:    s_cselect_b32 s3, s3, 0
+; SI-NEXT:    v_mov_b32_e32 v9, s3
+; SI-NEXT:    s_bfe_u32 s3, s19, 0xb0014
+; SI-NEXT:    s_addk_i32 s3, 0xfc01
+; SI-NEXT:    s_lshr_b64 s[4:5], s[20:21], s3
+; SI-NEXT:    s_andn2_b64 s[4:5], s[18:19], s[4:5]
+; SI-NEXT:    s_and_b32 s8, s19, 0x80000000
+; SI-NEXT:    s_cmp_lt_i32 s3, 0
 ; SI-NEXT:    s_cselect_b32 s4, 0, s4
 ; SI-NEXT:    s_cselect_b32 s5, s8, s5
-; SI-NEXT:    s_cmp_gt_i32 s7, 51
-; SI-NEXT:    s_cselect_b32 s5, s13, s5
-; SI-NEXT:    s_cselect_b32 s4, s12, s4
-; SI-NEXT:    s_bfe_u32 s7, s19, 0xb0014
-; SI-NEXT:    s_addk_i32 s7, 0xfc01
-; SI-NEXT:    s_lshr_b64 s[8:9], s[20:21], s7
-; SI-NEXT:    v_mov_b32_e32 v13, s5
-; SI-NEXT:    s_andn2_b64 s[8:9], s[18:19], s[8:9]
-; SI-NEXT:    s_and_b32 s10, s19, 0x80000000
-; SI-NEXT:    v_mov_b32_e32 v12, s4
-; SI-NEXT:    s_cmp_lt_i32 s7, 0
-; SI-NEXT:    v_add_f64 v[12:13], s[12:13], -v[12:13]
-; SI-NEXT:    s_cselect_b32 s8, 0, s8
-; SI-NEXT:    s_cselect_b32 s9, s10, s9
-; SI-NEXT:    s_cmp_gt_i32 s7, 51
-; SI-NEXT:    s_cselect_b32 s9, s19, s9
-; SI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[12:13]|, 0.5
-; SI-NEXT:    s_cselect_b32 s8, s18, s8
-; SI-NEXT:    v_mov_b32_e32 v13, s9
-; SI-NEXT:    v_mov_b32_e32 v12, s8
-; SI-NEXT:    v_mov_b32_e32 v9, s13
-; SI-NEXT:    v_add_f64 v[12:13], s[18:19], -v[12:13]
-; SI-NEXT:    v_bfi_b32 v9, s6, v8, v9
-; SI-NEXT:    v_cndmask_b32_e32 v17, 0, v9, vcc
-; SI-NEXT:    v_mov_b32_e32 v9, s19
-; SI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[12:13]|, 0.5
-; SI-NEXT:    v_bfi_b32 v9, s6, v8, v9
-; SI-NEXT:    s_bfe_u32 s7, s17, 0xb0014
-; SI-NEXT:    v_cndmask_b32_e32 v13, 0, v9, vcc
-; SI-NEXT:    v_mov_b32_e32 v12, 0
-; SI-NEXT:    s_addk_i32 s7, 0xfc01
-; SI-NEXT:    v_add_f64 v[14:15], s[8:9], v[12:13]
-; SI-NEXT:    s_lshr_b64 s[8:9], s[20:21], s7
-; SI-NEXT:    s_andn2_b64 s[8:9], s[16:17], s[8:9]
-; SI-NEXT:    s_and_b32 s10, s17, 0x80000000
-; SI-NEXT:    s_cmp_lt_i32 s7, 0
-; SI-NEXT:    s_cselect_b32 s8, 0, s8
-; SI-NEXT:    s_cselect_b32 s9, s10, s9
-; SI-NEXT:    s_cmp_gt_i32 s7, 51
-; SI-NEXT:    s_cselect_b32 s9, s17, s9
-; SI-NEXT:    s_cselect_b32 s8, s16, s8
-; SI-NEXT:    v_mov_b32_e32 v13, s9
-; SI-NEXT:    v_mov_b32_e32 v12, s8
-; SI-NEXT:    v_add_f64 v[12:13], s[16:17], -v[12:13]
-; SI-NEXT:    v_mov_b32_e32 v9, s17
-; SI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[12:13]|, 0.5
-; SI-NEXT:    v_bfi_b32 v8, s6, v8, v9
-; SI-NEXT:    v_cndmask_b32_e32 v9, 0, v8, vcc
-; SI-NEXT:    v_mov_b32_e32 v8, 0
-; SI-NEXT:    v_mov_b32_e32 v16, 0
-; SI-NEXT:    v_add_f64 v[12:13], s[8:9], v[8:9]
+; SI-NEXT:    s_cmp_gt_i32 s3, 51
+; SI-NEXT:    s_cselect_b32 s5, s19, s5
+; SI-NEXT:    s_cselect_b32 s4, s18, s4
+; SI-NEXT:    v_mov_b32_e32 v11, s5
+; SI-NEXT:    v_mov_b32_e32 v10, s4
+; SI-NEXT:    v_add_f64 v[14:15], s[18:19], -v[10:11]
+; SI-NEXT:    v_add_f64 v[10:11], s[6:7], v[8:9]
+; SI-NEXT:    v_cmp_ge_f64_e64 s[6:7], |v[14:15]|, 0.5
+; SI-NEXT:    s_or_b32 s3, s8, 0x3ff00000
+; SI-NEXT:    s_and_b64 s[6:7], s[6:7], exec
+; SI-NEXT:    s_cselect_b32 s3, s3, 0
+; SI-NEXT:    v_mov_b32_e32 v9, s3
+; SI-NEXT:    s_bfe_u32 s3, s17, 0xb0014
+; SI-NEXT:    s_addk_i32 s3, 0xfc01
+; SI-NEXT:    s_lshr_b64 s[6:7], s[20:21], s3
+; SI-NEXT:    s_andn2_b64 s[6:7], s[16:17], s[6:7]
+; SI-NEXT:    s_and_b32 s8, s17, 0x80000000
+; SI-NEXT:    s_cmp_lt_i32 s3, 0
+; SI-NEXT:    s_cselect_b32 s6, 0, s6
+; SI-NEXT:    s_cselect_b32 s7, s8, s7
+; SI-NEXT:    s_cmp_gt_i32 s3, 51
+; SI-NEXT:    s_cselect_b32 s7, s17, s7
+; SI-NEXT:    s_cselect_b32 s6, s16, s6
+; SI-NEXT:    v_mov_b32_e32 v15, s7
+; SI-NEXT:    v_mov_b32_e32 v14, s6
+; SI-NEXT:    v_add_f64 v[14:15], s[16:17], -v[14:15]
+; SI-NEXT:    v_add_f64 v[16:17], s[4:5], v[8:9]
+; SI-NEXT:    v_cmp_ge_f64_e64 s[4:5], |v[14:15]|, 0.5
+; SI-NEXT:    s_or_b32 s3, s8, 0x3ff00000
+; SI-NEXT:    s_and_b64 s[4:5], s[4:5], exec
+; SI-NEXT:    s_cselect_b32 s3, s3, 0
+; SI-NEXT:    v_mov_b32_e32 v9, s3
+; SI-NEXT:    v_add_f64 v[14:15], s[6:7], v[8:9]
 ; SI-NEXT:    s_mov_b32 s3, 0xf000
-; SI-NEXT:    v_add_f64 v[8:9], s[4:5], v[16:17]
-; SI-NEXT:    buffer_store_dwordx4 v[12:15], off, s[0:3], 0 offset:48
-; SI-NEXT:    buffer_store_dwordx4 v[8:11], off, s[0:3], 0 offset:32
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    buffer_store_dwordx4 v[14:17], off, s[0:3], 0 offset:48
+; SI-NEXT:    buffer_store_dwordx4 v[10:13], off, s[0:3], 0 offset:32
 ; SI-NEXT:    buffer_store_dwordx4 v[4:7], off, s[0:3], 0 offset:16
 ; SI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
@@ -549,76 +541,75 @@ define amdgpu_kernel void @round_v8f64(ptr addrspace(1) %out, <8 x double> %in)
 ; CI-LABEL: round_v8f64:
 ; CI:       ; %bb.0:
 ; CI-NEXT:    s_load_dwordx16 s[4:19], s[0:1], 0x19
-; CI-NEXT:    s_brev_b32 s20, -2
-; CI-NEXT:    v_mov_b32_e32 v20, 0x3ff00000
+; CI-NEXT:    s_mov_b32 s20, 0
 ; CI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
 ; CI-NEXT:    s_mov_b32 s3, 0xf000
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    v_trunc_f64_e32 v[0:1], s[6:7]
-; CI-NEXT:    v_mov_b32_e32 v4, s7
-; CI-NEXT:    v_add_f64 v[2:3], s[6:7], -v[0:1]
-; CI-NEXT:    v_bfi_b32 v4, s20, v20, v4
-; CI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[2:3]|, 0.5
-; CI-NEXT:    v_mov_b32_e32 v2, 0
-; CI-NEXT:    v_cndmask_b32_e32 v3, 0, v4, vcc
 ; CI-NEXT:    v_trunc_f64_e32 v[4:5], s[4:5]
-; CI-NEXT:    v_add_f64 v[2:3], v[0:1], v[2:3]
-; CI-NEXT:    v_add_f64 v[0:1], s[4:5], -v[4:5]
-; CI-NEXT:    v_mov_b32_e32 v6, s5
-; CI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[0:1]|, 0.5
-; CI-NEXT:    v_bfi_b32 v6, s20, v20, v6
-; CI-NEXT:    v_cndmask_b32_e32 v1, 0, v6, vcc
+; CI-NEXT:    v_add_f64 v[2:3], s[6:7], -v[0:1]
+; CI-NEXT:    s_and_b32 s2, s7, 0x80000000
+; CI-NEXT:    v_cmp_ge_f64_e64 s[6:7], |v[2:3]|, 0.5
+; CI-NEXT:    s_or_b32 s2, s2, 0x3ff00000
+; CI-NEXT:    v_add_f64 v[6:7], s[4:5], -v[4:5]
+; CI-NEXT:    s_and_b64 s[6:7], s[6:7], exec
+; CI-NEXT:    s_cselect_b32 s21, s2, 0
+; CI-NEXT:    v_cmp_ge_f64_e64 s[6:7], |v[6:7]|, 0.5
+; CI-NEXT:    s_and_b32 s2, s5, 0x80000000
+; CI-NEXT:    s_or_b32 s2, s2, 0x3ff00000
 ; CI-NEXT:    v_trunc_f64_e32 v[6:7], s[10:11]
-; CI-NEXT:    v_mov_b32_e32 v0, 0
-; CI-NEXT:    v_add_f64 v[0:1], v[4:5], v[0:1]
-; CI-NEXT:    v_add_f64 v[4:5], s[10:11], -v[6:7]
-; CI-NEXT:    v_mov_b32_e32 v8, s11
-; CI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[4:5]|, 0.5
-; CI-NEXT:    v_bfi_b32 v8, s20, v20, v8
-; CI-NEXT:    v_cndmask_b32_e32 v5, 0, v8, vcc
-; CI-NEXT:    v_trunc_f64_e32 v[8:9], s[8:9]
-; CI-NEXT:    v_mov_b32_e32 v4, 0
-; CI-NEXT:    v_add_f64 v[6:7], v[6:7], v[4:5]
-; CI-NEXT:    v_add_f64 v[4:5], s[8:9], -v[8:9]
-; CI-NEXT:    v_mov_b32_e32 v10, s9
-; CI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[4:5]|, 0.5
-; CI-NEXT:    v_bfi_b32 v10, s20, v20, v10
-; CI-NEXT:    v_cndmask_b32_e32 v5, 0, v10, vcc
-; CI-NEXT:    v_trunc_f64_e32 v[10:11], s[14:15]
-; CI-NEXT:    v_mov_b32_e32 v4, 0
-; CI-NEXT:    v_add_f64 v[4:5], v[8:9], v[4:5]
-; CI-NEXT:    v_add_f64 v[8:9], s[14:15], -v[10:11]
-; CI-NEXT:    v_mov_b32_e32 v12, s15
-; CI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[8:9]|, 0.5
-; CI-NEXT:    v_bfi_b32 v12, s20, v20, v12
-; CI-NEXT:    v_trunc_f64_e32 v[16:17], s[12:13]
-; CI-NEXT:    v_cndmask_b32_e32 v9, 0, v12, vcc
-; CI-NEXT:    v_mov_b32_e32 v8, 0
-; CI-NEXT:    v_add_f64 v[10:11], v[10:11], v[8:9]
-; CI-NEXT:    v_add_f64 v[8:9], s[12:13], -v[16:17]
-; CI-NEXT:    v_mov_b32_e32 v12, s13
-; CI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[8:9]|, 0.5
-; CI-NEXT:    v_bfi_b32 v12, s20, v20, v12
-; CI-NEXT:    v_cndmask_b32_e32 v9, 0, v12, vcc
-; CI-NEXT:    v_trunc_f64_e32 v[12:13], s[18:19]
-; CI-NEXT:    v_mov_b32_e32 v18, s19
-; CI-NEXT:    v_add_f64 v[14:15], s[18:19], -v[12:13]
-; CI-NEXT:    v_bfi_b32 v18, s20, v20, v18
-; CI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[14:15]|, 0.5
-; CI-NEXT:    v_mov_b32_e32 v14, 0
-; CI-NEXT:    v_cndmask_b32_e32 v15, 0, v18, vcc
-; CI-NEXT:    v_trunc_f64_e32 v[18:19], s[16:17]
-; CI-NEXT:    v_add_f64 v[14:15], v[12:13], v[14:15]
-; CI-NEXT:    v_add_f64 v[12:13], s[16:17], -v[18:19]
-; CI-NEXT:    v_mov_b32_e32 v21, s17
-; CI-NEXT:    v_cmp_ge_f64_e64 vcc, |v[12:13]|, 0.5
-; CI-NEXT:    v_bfi_b32 v20, s20, v20, v21
-; CI-NEXT:    v_cndmask_b32_e32 v13, 0, v20, vcc
-; CI-NEXT:    v_mov_b32_e32 v12, 0
-; CI-NEXT:    v_mov_b32_e32 v8, 0
-; CI-NEXT:    v_add_f64 v[12:13], v[18:19], v[12:13]
+; CI-NEXT:    s_and_b64 s[4:5], s[6:7], exec
+; CI-NEXT:    v_add_f64 v[2:3], v[0:1], s[20:21]
+; CI-NEXT:    s_cselect_b32 s21, s2, 0
+; CI-NEXT:    v_add_f64 v[8:9], s[10:11], -v[6:7]
+; CI-NEXT:    v_add_f64 v[0:1], v[4:5], s[20:21]
+; CI-NEXT:    v_trunc_f64_e32 v[4:5], s[8:9]
+; CI-NEXT:    v_cmp_ge_f64_e64 s[4:5], |v[8:9]|, 0.5
+; CI-NEXT:    s_and_b32 s2, s11, 0x80000000
+; CI-NEXT:    s_or_b32 s2, s2, 0x3ff00000
+; CI-NEXT:    v_add_f64 v[8:9], s[8:9], -v[4:5]
+; CI-NEXT:    s_and_b64 s[4:5], s[4:5], exec
+; CI-NEXT:    v_cmp_ge_f64_e64 s[4:5], |v[8:9]|, 0.5
+; CI-NEXT:    v_trunc_f64_e32 v[8:9], s[14:15]
+; CI-NEXT:    s_cselect_b32 s21, s2, 0
+; CI-NEXT:    s_and_b32 s2, s9, 0x80000000
+; CI-NEXT:    s_or_b32 s2, s2, 0x3ff00000
+; CI-NEXT:    v_add_f64 v[10:11], s[14:15], -v[8:9]
+; CI-NEXT:    s_and_b64 s[4:5], s[4:5], exec
+; CI-NEXT:    v_trunc_f64_e32 v[12:13], s[12:13]
+; CI-NEXT:    v_add_f64 v[6:7], v[6:7], s[20:21]
+; CI-NEXT:    s_cselect_b32 s21, s2, 0
+; CI-NEXT:    v_cmp_ge_f64_e64 s[4:5], |v[10:11]|, 0.5
+; CI-NEXT:    s_and_b32 s2, s15, 0x80000000
+; CI-NEXT:    s_or_b32 s2, s2, 0x3ff00000
+; CI-NEXT:    v_add_f64 v[14:15], s[12:13], -v[12:13]
+; CI-NEXT:    s_and_b64 s[4:5], s[4:5], exec
+; CI-NEXT:    v_add_f64 v[4:5], v[4:5], s[20:21]
+; CI-NEXT:    s_cselect_b32 s21, s2, 0
+; CI-NEXT:    v_cmp_ge_f64_e64 s[4:5], |v[14:15]|, 0.5
+; CI-NEXT:    s_and_b32 s2, s13, 0x80000000
+; CI-NEXT:    s_or_b32 s2, s2, 0x3ff00000
+; CI-NEXT:    v_trunc_f64_e32 v[14:15], s[18:19]
+; CI-NEXT:    s_and_b64 s[4:5], s[4:5], exec
+; CI-NEXT:    v_add_f64 v[10:11], v[8:9], s[20:21]
+; CI-NEXT:    s_cselect_b32 s21, s2, 0
+; CI-NEXT:    v_add_f64 v[16:17], s[18:19], -v[14:15]
+; CI-NEXT:    v_add_f64 v[8:9], v[12:13], s[20:21]
+; CI-NEXT:    v_trunc_f64_e32 v[12:13], s[16:17]
+; CI-NEXT:    v_cmp_ge_f64_e64 s[4:5], |v[16:17]|, 0.5
+; CI-NEXT:    s_and_b32 s2, s19, 0x80000000
+; CI-NEXT:    s_or_b32 s2, s2, 0x3ff00000
+; CI-NEXT:    v_add_f64 v[16:17], s[16:17], -v[12:13]
+; CI-NEXT:    s_and_b64 s[4:5], s[4:5], exec
+; CI-NEXT:    s_cselect_b32 s21, s2, 0
+; CI-NEXT:    v_cmp_ge_f64_e64 s[4:5], |v[16:17]|, 0.5
+; CI-NEXT:    s_and_b32 s2, s17, 0x80000000
+; CI-NEXT:    s_or_b32 s2, s2, 0x3ff00000
+; CI-NEXT:    s_and_b64 s[4:5], s[4:5], exec
+; CI-NEXT:    v_add_f64 v[14:15], v[14:15], s[20:21]
+; CI-NEXT:    s_cselect_b32 s21, s2, 0
+; CI-NEXT:    v_add_f64 v[12:13], v[12:13], s[20:21]
 ; CI-NEXT:    s_mov_b32 s2, -1
-; CI-NEXT:    v_add_f64 v[8:9], v[16:17], v[8:9]
 ; CI-NEXT:    buffer_store_dwordx4 v[12:15], off, s[0:3], 0 offset:48
 ; CI-NEXT:    buffer_store_dwordx4 v[8:11], off, s[0:3], 0 offset:32
 ; CI-NEXT:    buffer_store_dwordx4 v[4:7], off, s[0:3], 0 offset:16
diff --git a/llvm/test/CodeGen/AMDGPU/load-constant-i16.ll b/llvm/test/CodeGen/AMDGPU/load-constant-i16.ll
index adfe7c49dd6fa..885a8007e3b5f 100644
--- a/llvm/test/CodeGen/AMDGPU/load-constant-i16.ll
+++ b/llvm/test/CodeGen/AMDGPU/load-constant-i16.ll
@@ -1249,7 +1249,7 @@ define amdgpu_kernel void @constant_zextload_v4i16_to_v4i32(ptr addrspace(1) %ou
 ; EG:       ; %bb.0:
 ; EG-NEXT:    ALU 0, @8, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    TEX 0 @6
-; EG-NEXT:    ALU 12, @9, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    ALU 8, @9, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T5.XYZW, T6.X, 1
 ; EG-NEXT:    CF_END
 ; EG-NEXT:    PAD
@@ -1258,17 +1258,13 @@ define amdgpu_kernel void @constant_zextload_v4i16_to_v4i32(ptr addrspace(1) %ou
 ; EG-NEXT:    ALU clause starting at 8:
 ; EG-NEXT:     MOV * T5.X, KC0[2].Z,
 ; EG-NEXT:    ALU clause starting at 9:
-; EG-NEXT:     MOV T2.X, T5.X,
-; EG-NEXT:     MOV * T3.X, T5.Y,
-; EG-NEXT:     MOV T0.Y, PV.X,
-; EG-NEXT:     MOV * T0.Z, PS,
-; EG-NEXT:     LSHR * T5.W, PV.Z, literal.x,
+; EG-NEXT:     LSHR * T5.W, T5.Y, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; EG-NEXT:     AND_INT * T5.Z, T0.Z, literal.x,
+; EG-NEXT:     AND_INT * T5.Z, T5.Y, literal.x,
 ; EG-NEXT:    65535(9.183409e-41), 0(0.000000e+00)
-; EG-NEXT:     LSHR * T5.Y, T0.Y, literal.x,
+; EG-NEXT:     LSHR * T5.Y, T5.X, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; EG-NEXT:     AND_INT T5.X, T0.Y, literal.x,
+; EG-NEXT:     AND_INT T5.X, T5.X, literal.x,
 ; EG-NEXT:     LSHR * T6.X, KC0[2].Y, literal.y,
 ; EG-NEXT:    65535(9.183409e-41), 2(2.802597e-45)
   %load = load <4 x i16>, ptr addrspace(4) %in
@@ -1342,8 +1338,8 @@ define amdgpu_kernel void @constant_sextload_v4i16_to_v4i32(ptr addrspace(1) %ou
 ; EG:       ; %bb.0:
 ; EG-NEXT:    ALU 0, @8, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    TEX 0 @6
-; EG-NEXT:    ALU 14, @9, KC0[CB0:0-32], KC1[]
-; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T5.XYZW, T6.X, 1
+; EG-NEXT:    ALU 10, @9, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T6.XYZW, T5.X, 1
 ; EG-NEXT:    CF_END
 ; EG-NEXT:    PAD
 ; EG-NEXT:    Fetch clause starting at 6:
@@ -1351,20 +1347,16 @@ define amdgpu_kernel void @constant_sextload_v4i16_to_v4i32(ptr addrspace(1) %ou
 ; EG-NEXT:    ALU clause starting at 8:
 ; EG-NEXT:     MOV * T5.X, KC0[2].Z,
 ; EG-NEXT:    ALU clause starting at 9:
-; EG-NEXT:     MOV T2.X, T5.X,
-; EG-NEXT:     MOV * T3.X, T5.Y,
-; EG-NEXT:     MOV T0.Y, PV.X,
-; EG-NEXT:     MOV * T0.Z, PS,
-; EG-NEXT:     BFE_INT * T5.Z, PV.Z, 0.0, literal.x,
+; EG-NEXT:     BFE_INT * T6.Z, T5.Y, 0.0, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; EG-NEXT:     BFE_INT T5.X, T0.Y, 0.0, literal.x,
-; EG-NEXT:     LSHR * T0.W, T0.Z, literal.x,
+; EG-NEXT:     BFE_INT T6.X, T5.X, 0.0, literal.x,
+; EG-NEXT:     LSHR * T0.W, T5.Y, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; EG-NEXT:     BFE_INT T5.W, PV.W, 0.0, literal.x,
-; EG-NEXT:     LSHR * T0.W, T0.Y, literal.x,
+; EG-NEXT:     BFE_INT T6.W, PV.W, 0.0, literal.x,
+; EG-NEXT:     LSHR * T0.W, T5.X, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; EG-NEXT:     LSHR T6.X, KC0[2].Y, literal.x,
-; EG-NEXT:     BFE_INT * T5.Y, PS, 0.0, literal.y,
+; EG-NEXT:     LSHR T5.X, KC0[2].Y, literal.x,
+; EG-NEXT:     BFE_INT * T6.Y, PS, 0.0, literal.y,
 ; EG-NEXT:    2(2.802597e-45), 16(2.242078e-44)
   %load = load <4 x i16>, ptr addrspace(4) %in
   %ext = sext <4 x i16> %load to <4 x i32>
@@ -4879,29 +4871,25 @@ define amdgpu_kernel void @constant_zextload_v4i16_to_v4i64(ptr addrspace(1) %ou
 ; EG:       ; %bb.0:
 ; EG-NEXT:    ALU 0, @8, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    TEX 0 @6
-; EG-NEXT:    ALU 18, @9, KC0[CB0:0-32], KC1[]
-; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T5.XYZW, T8.X, 0
-; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T6.XYZW, T7.X, 1
+; EG-NEXT:    ALU 14, @9, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T6.XYZW, T8.X, 0
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T5.XYZW, T7.X, 1
 ; EG-NEXT:    CF_END
 ; EG-NEXT:    Fetch clause starting at 6:
 ; EG-NEXT:     VTX_READ_64 T5.XY, T5.X, 0, #1
 ; EG-NEXT:    ALU clause starting at 8:
 ; EG-NEXT:     MOV * T5.X, KC0[2].Z,
 ; EG-NEXT:    ALU clause starting at 9:
-; EG-NEXT:     MOV T2.X, T5.X,
-; EG-NEXT:     MOV * T3.X, T5.Y,
-; EG-NEXT:     MOV T0.Y, PV.X,
-; EG-NEXT:     MOV * T0.Z, PS,
-; EG-NEXT:     LSHR * T5.Z, PV.Z, literal.x,
+; EG-NEXT:     LSHR * T6.Z, T5.Y, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; EG-NEXT:     AND_INT T5.X, T0.Z, literal.x,
-; EG-NEXT:     MOV T5.Y, 0.0,
-; EG-NEXT:     LSHR T6.Z, T0.Y, literal.y,
-; EG-NEXT:     AND_INT * T6.X, T0.Y, literal.x,
-; EG-NEXT:    65535(9.183409e-41), 16(2.242078e-44)
+; EG-NEXT:     AND_INT T6.X, T5.Y, literal.x,
 ; EG-NEXT:     MOV T6.Y, 0.0,
-; EG-NEXT:     MOV T5.W, 0.0,
-; EG-NEXT:     MOV * T6.W, 0.0,
+; EG-NEXT:     LSHR T5.Z, T5.X, literal.y,
+; EG-NEXT:     AND_INT * T5.X, T5.X, literal.x,
+; EG-NEXT:    65535(9.183409e-41), 16(2.242078e-44)
+; EG-NEXT:     MOV T5.Y, 0.0,
+; EG-NEXT:     MOV T6.W, 0.0,
+; EG-NEXT:     MOV * T5.W, 0.0,
 ; EG-NEXT:     LSHR T7.X, KC0[2].Y, literal.x,
 ; EG-NEXT:     ADD_INT * T0.W, KC0[2].Y, literal.y,
 ; EG-NEXT:    2(2.802597e-45), 16(2.242078e-44)
@@ -5003,7 +4991,7 @@ define amdgpu_kernel void @constant_sextload_v4i16_to_v4i64(ptr addrspace(1) %ou
 ; EG:       ; %bb.0:
 ; EG-NEXT:    ALU 0, @8, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    TEX 0 @6
-; EG-NEXT:    ALU 20, @9, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    ALU 16, @9, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T7.XYZW, T8.X, 0
 ; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T5.XYZW, T6.X, 1
 ; EG-NEXT:    CF_END
@@ -5012,21 +5000,17 @@ define amdgpu_kernel void @constant_sextload_v4i16_to_v4i64(ptr addrspace(1) %ou
 ; EG-NEXT:    ALU clause starting at 8:
 ; EG-NEXT:     MOV * T5.X, KC0[2].Z,
 ; EG-NEXT:    ALU clause starting at 9:
-; EG-NEXT:     MOV T2.X, T5.X,
-; EG-NEXT:     MOV * T3.X, T5.Y,
-; EG-NEXT:     MOV T0.Y, PS,
-; EG-NEXT:     MOV * T0.Z, PV.X,
-; EG-NEXT:     ASHR * T5.W, PV.Z, literal.x,
+; EG-NEXT:     ASHR * T5.W, T5.X, literal.x,
 ; EG-NEXT:    31(4.344025e-44), 0(0.000000e+00)
 ; EG-NEXT:     LSHR T6.X, KC0[2].Y, literal.x,
-; EG-NEXT:     ASHR T5.Z, T0.Z, literal.y,
-; EG-NEXT:     ASHR * T7.W, T0.Y, literal.z,
+; EG-NEXT:     ASHR T5.Z, T5.X, literal.y,
+; EG-NEXT:     ASHR * T7.W, T5.Y, literal.z,
 ; EG-NEXT:    2(2.802597e-45), 16(2.242078e-44)
 ; EG-NEXT:    31(4.344025e-44), 0(0.000000e+00)
-; EG-NEXT:     BFE_INT T5.X, T0.Z, 0.0, literal.x,
-; EG-NEXT:     ASHR * T7.Z, T0.Y, literal.x,
+; EG-NEXT:     BFE_INT T5.X, T5.X, 0.0, literal.x,
+; EG-NEXT:     ASHR * T7.Z, T5.Y, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; EG-NEXT:     BFE_INT T7.X, T0.Y, 0.0, literal.x,
+; EG-NEXT:     BFE_INT T7.X, T5.Y, 0.0, literal.x,
 ; EG-NEXT:     ASHR T5.Y, PV.X, literal.y,
 ; EG-NEXT:     ADD_INT * T0.W, KC0[2].Y, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 31(4.344025e-44)
@@ -5129,27 +5113,27 @@ define amdgpu_kernel void @constant_zextload_v8i16_to_v8i64(ptr addrspace(1) %ou
 ; GCN-NOHSA-VI-NEXT:    s_mov_b32 s3, 0xf000
 ; GCN-NOHSA-VI-NEXT:    s_mov_b32 s2, -1
 ; GCN-NOHSA-VI-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NOHSA-VI-NEXT:    s_and_b32 s11, s7, 0xffff
-; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s7, s7, 16
-; GCN-NOHSA-VI-NEXT:    s_and_b32 s10, s6, 0xffff
-; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s6, s6, 16
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s11
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s7
-; GCN-NOHSA-VI-NEXT:    s_and_b32 s9, s5, 0xffff
-; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s5, s5, 16
+; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s11, s7, 16
+; GCN-NOHSA-VI-NEXT:    s_and_b32 s7, s7, 0xffff
+; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s10, s6, 16
+; GCN-NOHSA-VI-NEXT:    s_and_b32 s6, s6, 0xffff
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s7
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s11
+; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s9, s5, 16
+; GCN-NOHSA-VI-NEXT:    s_and_b32 s5, s5, 0xffff
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0 offset:48
-; GCN-NOHSA-VI-NEXT:    s_and_b32 s8, s4, 0xffff
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s10
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s6
-; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s4, s4, 16
+; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s8, s4, 16
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s6
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s10
+; GCN-NOHSA-VI-NEXT:    s_and_b32 s4, s4, 0xffff
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0 offset:32
 ; GCN-NOHSA-VI-NEXT:    s_nop 0
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s9
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s5
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s5
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s9
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0 offset:16
 ; GCN-NOHSA-VI-NEXT:    s_nop 0
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s8
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s4
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s8
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; GCN-NOHSA-VI-NEXT:    s_endpgm
 ;
@@ -5555,47 +5539,47 @@ define amdgpu_kernel void @constant_zextload_v16i16_to_v16i64(ptr addrspace(1) %
 ; GCN-NOHSA-VI-NEXT:    s_mov_b32 s3, 0xf000
 ; GCN-NOHSA-VI-NEXT:    s_mov_b32 s2, -1
 ; GCN-NOHSA-VI-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NOHSA-VI-NEXT:    s_and_b32 s19, s9, 0xffff
-; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s9, s9, 16
-; GCN-NOHSA-VI-NEXT:    s_and_b32 s18, s8, 0xffff
-; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s8, s8, 16
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s19
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s9
-; GCN-NOHSA-VI-NEXT:    s_and_b32 s17, s11, 0xffff
-; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s11, s11, 16
+; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s19, s9, 16
+; GCN-NOHSA-VI-NEXT:    s_and_b32 s9, s9, 0xffff
+; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s18, s8, 16
+; GCN-NOHSA-VI-NEXT:    s_and_b32 s8, s8, 0xffff
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s9
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s19
+; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s17, s11, 16
+; GCN-NOHSA-VI-NEXT:    s_and_b32 s11, s11, 0xffff
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0 offset:80
-; GCN-NOHSA-VI-NEXT:    s_and_b32 s16, s10, 0xffff
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s18
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s8
-; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s10, s10, 16
+; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s16, s10, 16
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s8
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s18
+; GCN-NOHSA-VI-NEXT:    s_and_b32 s10, s10, 0xffff
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0 offset:64
-; GCN-NOHSA-VI-NEXT:    s_and_b32 s15, s7, 0xffff
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s17
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s11
-; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s7, s7, 16
+; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s15, s7, 16
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s11
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s17
+; GCN-NOHSA-VI-NEXT:    s_and_b32 s7, s7, 0xffff
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0 offset:112
-; GCN-NOHSA-VI-NEXT:    s_and_b32 s14, s6, 0xffff
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s16
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s10
-; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s6, s6, 16
+; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s14, s6, 16
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s10
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s16
+; GCN-NOHSA-VI-NEXT:    s_and_b32 s6, s6, 0xffff
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0 offset:96
-; GCN-NOHSA-VI-NEXT:    s_and_b32 s13, s5, 0xffff
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s15
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s7
-; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s5, s5, 16
+; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s13, s5, 16
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s7
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s15
+; GCN-NOHSA-VI-NEXT:    s_and_b32 s5, s5, 0xffff
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0 offset:48
-; GCN-NOHSA-VI-NEXT:    s_and_b32 s12, s4, 0xffff
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s14
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s6
-; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s4, s4, 16
+; GCN-NOHSA-VI-NEXT:    s_lshr_b32 s12, s4, 16
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s6
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s14
+; GCN-NOHSA-VI-NEXT:    s_and_b32 s4, s4, 0xffff
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0 offset:32
 ; GCN-NOHSA-VI-NEXT:    s_nop 0
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s13
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s5
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s5
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s13
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0 offset:16
 ; GCN-NOHSA-VI-NEXT:    s_nop 0
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s12
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s4
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v2, s12
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; GCN-NOHSA-VI-NEXT:    s_endpgm
 ;
diff --git a/llvm/test/CodeGen/AMDGPU/load-global-i16.ll b/llvm/test/CodeGen/AMDGPU/load-global-i16.ll
index 227c2f50ca218..8a7cdf36accb7 100644
--- a/llvm/test/CodeGen/AMDGPU/load-global-i16.ll
+++ b/llvm/test/CodeGen/AMDGPU/load-global-i16.ll
@@ -1636,7 +1636,7 @@ define amdgpu_kernel void @global_zextload_v4i16_to_v4i32(ptr addrspace(1) %out,
 ; EG:       ; %bb.0:
 ; EG-NEXT:    ALU 0, @8, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    TEX 0 @6
-; EG-NEXT:    ALU 12, @9, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    ALU 8, @9, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T5.XYZW, T6.X, 1
 ; EG-NEXT:    CF_END
 ; EG-NEXT:    PAD
@@ -1645,17 +1645,13 @@ define amdgpu_kernel void @global_zextload_v4i16_to_v4i32(ptr addrspace(1) %out,
 ; EG-NEXT:    ALU clause starting at 8:
 ; EG-NEXT:     MOV * T5.X, KC0[2].Z,
 ; EG-NEXT:    ALU clause starting at 9:
-; EG-NEXT:     MOV T2.X, T5.X,
-; EG-NEXT:     MOV * T3.X, T5.Y,
-; EG-NEXT:     MOV T0.Y, PV.X,
-; EG-NEXT:     MOV * T0.Z, PS,
-; EG-NEXT:     LSHR * T5.W, PV.Z, literal.x,
+; EG-NEXT:     LSHR * T5.W, T5.Y, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; EG-NEXT:     AND_INT * T5.Z, T0.Z, literal.x,
+; EG-NEXT:     AND_INT * T5.Z, T5.Y, literal.x,
 ; EG-NEXT:    65535(9.183409e-41), 0(0.000000e+00)
-; EG-NEXT:     LSHR * T5.Y, T0.Y, literal.x,
+; EG-NEXT:     LSHR * T5.Y, T5.X, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; EG-NEXT:     AND_INT T5.X, T0.Y, literal.x,
+; EG-NEXT:     AND_INT T5.X, T5.X, literal.x,
 ; EG-NEXT:     LSHR * T6.X, KC0[2].Y, literal.y,
 ; EG-NEXT:    65535(9.183409e-41), 2(2.802597e-45)
 ;
@@ -1663,7 +1659,7 @@ define amdgpu_kernel void @global_zextload_v4i16_to_v4i32(ptr addrspace(1) %out,
 ; CM:       ; %bb.0:
 ; CM-NEXT:    ALU 0, @8, KC0[CB0:0-32], KC1[]
 ; CM-NEXT:    TEX 0 @6
-; CM-NEXT:    ALU 13, @9, KC0[CB0:0-32], KC1[]
+; CM-NEXT:    ALU 9, @9, KC0[CB0:0-32], KC1[]
 ; CM-NEXT:    MEM_RAT_CACHELESS STORE_DWORD T5, T6.X
 ; CM-NEXT:    CF_END
 ; CM-NEXT:    PAD
@@ -1672,17 +1668,13 @@ define amdgpu_kernel void @global_zextload_v4i16_to_v4i32(ptr addrspace(1) %out,
 ; CM-NEXT:    ALU clause starting at 8:
 ; CM-NEXT:     MOV * T5.X, KC0[2].Z,
 ; CM-NEXT:    ALU clause starting at 9:
-; CM-NEXT:     MOV * T2.X, T5.X,
-; CM-NEXT:     MOV T3.X, T5.Y,
-; CM-NEXT:     MOV * T0.Y, PV.X,
-; CM-NEXT:     MOV * T0.Z, PV.X,
-; CM-NEXT:     LSHR * T5.W, PV.Z, literal.x,
+; CM-NEXT:     LSHR * T5.W, T5.Y, literal.x,
 ; CM-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; CM-NEXT:     AND_INT * T5.Z, T0.Z, literal.x,
+; CM-NEXT:     AND_INT * T5.Z, T5.Y, literal.x,
 ; CM-NEXT:    65535(9.183409e-41), 0(0.000000e+00)
-; CM-NEXT:     LSHR * T5.Y, T0.Y, literal.x,
+; CM-NEXT:     LSHR * T5.Y, T5.X, literal.x,
 ; CM-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; CM-NEXT:     AND_INT * T5.X, T0.Y, literal.x,
+; CM-NEXT:     AND_INT * T5.X, T5.X, literal.x,
 ; CM-NEXT:    65535(9.183409e-41), 0(0.000000e+00)
 ; CM-NEXT:     LSHR * T6.X, KC0[2].Y, literal.x,
 ; CM-NEXT:    2(2.802597e-45), 0(0.000000e+00)
@@ -1760,8 +1752,8 @@ define amdgpu_kernel void @global_sextload_v4i16_to_v4i32(ptr addrspace(1) %out,
 ; EG:       ; %bb.0:
 ; EG-NEXT:    ALU 0, @8, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    TEX 0 @6
-; EG-NEXT:    ALU 14, @9, KC0[CB0:0-32], KC1[]
-; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T5.XYZW, T6.X, 1
+; EG-NEXT:    ALU 10, @9, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T6.XYZW, T5.X, 1
 ; EG-NEXT:    CF_END
 ; EG-NEXT:    PAD
 ; EG-NEXT:    Fetch clause starting at 6:
@@ -1769,28 +1761,24 @@ define amdgpu_kernel void @global_sextload_v4i16_to_v4i32(ptr addrspace(1) %out,
 ; EG-NEXT:    ALU clause starting at 8:
 ; EG-NEXT:     MOV * T5.X, KC0[2].Z,
 ; EG-NEXT:    ALU clause starting at 9:
-; EG-NEXT:     MOV T2.X, T5.X,
-; EG-NEXT:     MOV * T3.X, T5.Y,
-; EG-NEXT:     MOV T0.Y, PV.X,
-; EG-NEXT:     MOV * T0.Z, PS,
-; EG-NEXT:     BFE_INT * T5.Z, PV.Z, 0.0, literal.x,
+; EG-NEXT:     BFE_INT * T6.Z, T5.Y, 0.0, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; EG-NEXT:     BFE_INT T5.X, T0.Y, 0.0, literal.x,
-; EG-NEXT:     LSHR * T0.W, T0.Z, literal.x,
+; EG-NEXT:     BFE_INT T6.X, T5.X, 0.0, literal.x,
+; EG-NEXT:     LSHR * T0.W, T5.Y, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; EG-NEXT:     BFE_INT T5.W, PV.W, 0.0, literal.x,
-; EG-NEXT:     LSHR * T0.W, T0.Y, literal.x,
+; EG-NEXT:     BFE_INT T6.W, PV.W, 0.0, literal.x,
+; EG-NEXT:     LSHR * T0.W, T5.X, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; EG-NEXT:     LSHR T6.X, KC0[2].Y, literal.x,
-; EG-NEXT:     BFE_INT * T5.Y, PS, 0.0, literal.y,
+; EG-NEXT:     LSHR T5.X, KC0[2].Y, literal.x,
+; EG-NEXT:     BFE_INT * T6.Y, PS, 0.0, literal.y,
 ; EG-NEXT:    2(2.802597e-45), 16(2.242078e-44)
 ;
 ; CM-LABEL: global_sextload_v4i16_to_v4i32:
 ; CM:       ; %bb.0:
 ; CM-NEXT:    ALU 0, @8, KC0[CB0:0-32], KC1[]
 ; CM-NEXT:    TEX 0 @6
-; CM-NEXT:    ALU 14, @9, KC0[CB0:0-32], KC1[]
-; CM-NEXT:    MEM_RAT_CACHELESS STORE_DWORD T5, T6.X
+; CM-NEXT:    ALU 10, @9, KC0[CB0:0-32], KC1[]
+; CM-NEXT:    MEM_RAT_CACHELESS STORE_DWORD T6, T5.X
 ; CM-NEXT:    CF_END
 ; CM-NEXT:    PAD
 ; CM-NEXT:    Fetch clause starting at 6:
@@ -1798,20 +1786,16 @@ define amdgpu_kernel void @global_sextload_v4i16_to_v4i32(ptr addrspace(1) %out,
 ; CM-NEXT:    ALU clause starting at 8:
 ; CM-NEXT:     MOV * T5.X, KC0[2].Z,
 ; CM-NEXT:    ALU clause starting at 9:
-; CM-NEXT:     MOV * T2.X, T5.X,
-; CM-NEXT:     MOV T3.X, T5.Y,
-; CM-NEXT:     MOV * T0.Y, PV.X,
-; CM-NEXT:     MOV * T0.Z, PV.X,
-; CM-NEXT:     BFE_INT * T5.Z, PV.Z, 0.0, literal.x,
+; CM-NEXT:     BFE_INT * T6.Z, T5.Y, 0.0, literal.x,
 ; CM-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; CM-NEXT:     BFE_INT T5.X, T0.Y, 0.0, literal.x,
-; CM-NEXT:     LSHR * T0.W, T0.Z, literal.x,
+; CM-NEXT:     BFE_INT T6.X, T5.X, 0.0, literal.x,
+; CM-NEXT:     LSHR * T0.W, T5.Y, literal.x,
 ; CM-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; CM-NEXT:     LSHR T0.Z, T0.Y, literal.x,
-; CM-NEXT:     BFE_INT * T5.W, PV.W, 0.0, literal.x,
+; CM-NEXT:     LSHR T0.Z, T5.X, literal.x,
+; CM-NEXT:     BFE_INT * T6.W, PV.W, 0.0, literal.x,
 ; CM-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; CM-NEXT:     LSHR T6.X, KC0[2].Y, literal.x,
-; CM-NEXT:     BFE_INT * T5.Y, PV.Z, 0.0, literal.y,
+; CM-NEXT:     LSHR T5.X, KC0[2].Y, literal.x,
+; CM-NEXT:     BFE_INT * T6.Y, PV.Z, 0.0, literal.y,
 ; CM-NEXT:    2(2.802597e-45), 16(2.242078e-44)
   %load = load <4 x i16>, ptr addrspace(1) %in
   %ext = sext <4 x i16> %load to <4 x i32>
@@ -5788,29 +5772,25 @@ define amdgpu_kernel void @global_zextload_v4i16_to_v4i64(ptr addrspace(1) %out,
 ; EG:       ; %bb.0:
 ; EG-NEXT:    ALU 0, @8, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    TEX 0 @6
-; EG-NEXT:    ALU 18, @9, KC0[CB0:0-32], KC1[]
-; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T5.XYZW, T8.X, 0
-; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T6.XYZW, T7.X, 1
+; EG-NEXT:    ALU 14, @9, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T6.XYZW, T8.X, 0
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T5.XYZW, T7.X, 1
 ; EG-NEXT:    CF_END
 ; EG-NEXT:    Fetch clause starting at 6:
 ; EG-NEXT:     VTX_READ_64 T5.XY, T5.X, 0, #1
 ; EG-NEXT:    ALU clause starting at 8:
 ; EG-NEXT:     MOV * T5.X, KC0[2].Z,
 ; EG-NEXT:    ALU clause starting at 9:
-; EG-NEXT:     MOV T2.X, T5.X,
-; EG-NEXT:     MOV * T3.X, T5.Y,
-; EG-NEXT:     MOV T0.Y, PV.X,
-; EG-NEXT:     MOV * T0.Z, PS,
-; EG-NEXT:     LSHR * T5.Z, PV.Z, literal.x,
+; EG-NEXT:     LSHR * T6.Z, T5.Y, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; EG-NEXT:     AND_INT T5.X, T0.Z, literal.x,
-; EG-NEXT:     MOV T5.Y, 0.0,
-; EG-NEXT:     LSHR T6.Z, T0.Y, literal.y,
-; EG-NEXT:     AND_INT * T6.X, T0.Y, literal.x,
-; EG-NEXT:    65535(9.183409e-41), 16(2.242078e-44)
+; EG-NEXT:     AND_INT T6.X, T5.Y, literal.x,
 ; EG-NEXT:     MOV T6.Y, 0.0,
-; EG-NEXT:     MOV T5.W, 0.0,
-; EG-NEXT:     MOV * T6.W, 0.0,
+; EG-NEXT:     LSHR T5.Z, T5.X, literal.y,
+; EG-NEXT:     AND_INT * T5.X, T5.X, literal.x,
+; EG-NEXT:    65535(9.183409e-41), 16(2.242078e-44)
+; EG-NEXT:     MOV T5.Y, 0.0,
+; EG-NEXT:     MOV T6.W, 0.0,
+; EG-NEXT:     MOV * T5.W, 0.0,
 ; EG-NEXT:     LSHR T7.X, KC0[2].Y, literal.x,
 ; EG-NEXT:     ADD_INT * T0.W, KC0[2].Y, literal.y,
 ; EG-NEXT:    2(2.802597e-45), 16(2.242078e-44)
@@ -5821,30 +5801,26 @@ define amdgpu_kernel void @global_zextload_v4i16_to_v4i64(ptr addrspace(1) %out,
 ; CM:       ; %bb.0:
 ; CM-NEXT:    ALU 0, @8, KC0[CB0:0-32], KC1[]
 ; CM-NEXT:    TEX 0 @6
-; CM-NEXT:    ALU 20, @9, KC0[CB0:0-32], KC1[]
-; CM-NEXT:    MEM_RAT_CACHELESS STORE_DWORD T5, T8.X
-; CM-NEXT:    MEM_RAT_CACHELESS STORE_DWORD T6, T7.X
+; CM-NEXT:    ALU 16, @9, KC0[CB0:0-32], KC1[]
+; CM-NEXT:    MEM_RAT_CACHELESS STORE_DWORD T6, T8.X
+; CM-NEXT:    MEM_RAT_CACHELESS STORE_DWORD T5, T7.X
 ; CM-NEXT:    CF_END
 ; CM-NEXT:    Fetch clause starting at 6:
 ; CM-NEXT:     VTX_READ_64 T5.XY, T5.X, 0, #1
 ; CM-NEXT:    ALU clause starting at 8:
 ; CM-NEXT:     MOV * T5.X, KC0[2].Z,
 ; CM-NEXT:    ALU clause starting at 9:
-; CM-NEXT:     MOV * T2.X, T5.X,
-; CM-NEXT:     MOV * T3.X, T5.Y,
-; CM-NEXT:     MOV T0.Y, PV.X,
-; CM-NEXT:     MOV * T0.Z, T2.X,
-; CM-NEXT:     LSHR * T5.Z, PV.Z, literal.x,
+; CM-NEXT:     LSHR * T6.Z, T5.X, literal.x,
 ; CM-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; CM-NEXT:     AND_INT T5.X, T0.Z, literal.x,
-; CM-NEXT:     MOV T5.Y, 0.0,
-; CM-NEXT:     LSHR * T6.Z, T0.Y, literal.y,
-; CM-NEXT:    65535(9.183409e-41), 16(2.242078e-44)
-; CM-NEXT:     AND_INT T6.X, T0.Y, literal.x,
+; CM-NEXT:     AND_INT T6.X, T5.X, literal.x,
 ; CM-NEXT:     MOV T6.Y, 0.0,
-; CM-NEXT:     MOV * T5.W, 0.0,
-; CM-NEXT:    65535(9.183409e-41), 0(0.000000e+00)
+; CM-NEXT:     LSHR * T5.Z, T5.Y, literal.y,
+; CM-NEXT:    65535(9.183409e-41), 16(2.242078e-44)
+; CM-NEXT:     AND_INT T5.X, T5.Y, literal.x,
+; CM-NEXT:     MOV T5.Y, 0.0,
 ; CM-NEXT:     MOV * T6.W, 0.0,
+; CM-NEXT:    65535(9.183409e-41), 0(0.000000e+00)
+; CM-NEXT:     MOV * T5.W, 0.0,
 ; CM-NEXT:     ADD_INT * T0.W, KC0[2].Y, literal.x,
 ; CM-NEXT:    16(2.242078e-44), 0(0.000000e+00)
 ; CM-NEXT:     LSHR * T7.X, PV.W, literal.x,
@@ -5945,7 +5921,7 @@ define amdgpu_kernel void @global_sextload_v4i16_to_v4i64(ptr addrspace(1) %out,
 ; EG:       ; %bb.0:
 ; EG-NEXT:    ALU 0, @8, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    TEX 0 @6
-; EG-NEXT:    ALU 20, @9, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    ALU 16, @9, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T7.XYZW, T8.X, 0
 ; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T5.XYZW, T6.X, 1
 ; EG-NEXT:    CF_END
@@ -5954,21 +5930,17 @@ define amdgpu_kernel void @global_sextload_v4i16_to_v4i64(ptr addrspace(1) %out,
 ; EG-NEXT:    ALU clause starting at 8:
 ; EG-NEXT:     MOV * T5.X, KC0[2].Z,
 ; EG-NEXT:    ALU clause starting at 9:
-; EG-NEXT:     MOV T2.X, T5.X,
-; EG-NEXT:     MOV * T3.X, T5.Y,
-; EG-NEXT:     MOV T0.Y, PS,
-; EG-NEXT:     MOV * T0.Z, PV.X,
-; EG-NEXT:     ASHR * T5.W, PV.Z, literal.x,
+; EG-NEXT:     ASHR * T5.W, T5.X, literal.x,
 ; EG-NEXT:    31(4.344025e-44), 0(0.000000e+00)
 ; EG-NEXT:     LSHR T6.X, KC0[2].Y, literal.x,
-; EG-NEXT:     ASHR T5.Z, T0.Z, literal.y,
-; EG-NEXT:     ASHR * T7.W, T0.Y, literal.z,
+; EG-NEXT:     ASHR T5.Z, T5.X, literal.y,
+; EG-NEXT:     ASHR * T7.W, T5.Y, literal.z,
 ; EG-NEXT:    2(2.802597e-45), 16(2.242078e-44)
 ; EG-NEXT:    31(4.344025e-44), 0(0.000000e+00)
-; EG-NEXT:     BFE_INT T5.X, T0.Z, 0.0, literal.x,
-; EG-NEXT:     ASHR * T7.Z, T0.Y, literal.x,
+; EG-NEXT:     BFE_INT T5.X, T5.X, 0.0, literal.x,
+; EG-NEXT:     ASHR * T7.Z, T5.Y, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; EG-NEXT:     BFE_INT T7.X, T0.Y, 0.0, literal.x,
+; EG-NEXT:     BFE_INT T7.X, T5.Y, 0.0, literal.x,
 ; EG-NEXT:     ASHR T5.Y, PV.X, literal.y,
 ; EG-NEXT:     ADD_INT * T0.W, KC0[2].Y, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 31(4.344025e-44)
@@ -5980,35 +5952,31 @@ define amdgpu_kernel void @global_sextload_v4i16_to_v4i64(ptr addrspace(1) %out,
 ; CM:       ; %bb.0:
 ; CM-NEXT:    ALU 0, @8, KC0[CB0:0-32], KC1[]
 ; CM-NEXT:    TEX 0 @6
-; CM-NEXT:    ALU 20, @9, KC0[CB0:0-32], KC1[]
-; CM-NEXT:    MEM_RAT_CACHELESS STORE_DWORD T7, T8.X
-; CM-NEXT:    MEM_RAT_CACHELESS STORE_DWORD T5, T6.X
+; CM-NEXT:    ALU 16, @9, KC0[CB0:0-32], KC1[]
+; CM-NEXT:    MEM_RAT_CACHELESS STORE_DWORD T5, T8.X
+; CM-NEXT:    MEM_RAT_CACHELESS STORE_DWORD T6, T7.X
 ; CM-NEXT:    CF_END
 ; CM-NEXT:    Fetch clause starting at 6:
 ; CM-NEXT:     VTX_READ_64 T5.XY, T5.X, 0, #1
 ; CM-NEXT:    ALU clause starting at 8:
 ; CM-NEXT:     MOV * T5.X, KC0[2].Z,
 ; CM-NEXT:    ALU clause starting at 9:
-; CM-NEXT:     MOV * T2.X, T5.X,
-; CM-NEXT:     MOV T3.X, T5.Y,
-; CM-NEXT:     MOV * T0.Y, PV.X,
-; CM-NEXT:     MOV * T0.Z, PV.X,
-; CM-NEXT:     ADD_INT T1.Z, KC0[2].Y, literal.x,
-; CM-NEXT:     ASHR * T5.W, PV.Z, literal.y,
+; CM-NEXT:     ADD_INT T0.Z, KC0[2].Y, literal.x,
+; CM-NEXT:     ASHR * T6.W, T5.Y, literal.y,
 ; CM-NEXT:    16(2.242078e-44), 31(4.344025e-44)
-; CM-NEXT:     LSHR T6.X, PV.Z, literal.x,
-; CM-NEXT:     ASHR T5.Z, T0.Z, literal.y,
-; CM-NEXT:     ASHR * T7.W, T0.Y, literal.z,
+; CM-NEXT:     LSHR T7.X, PV.Z, literal.x,
+; CM-NEXT:     ASHR T6.Z, T5.Y, literal.y,
+; CM-NEXT:     ASHR * T5.W, T5.X, literal.z,
 ; CM-NEXT:    2(2.802597e-45), 16(2.242078e-44)
 ; CM-NEXT:    31(4.344025e-44), 0(0.000000e+00)
-; CM-NEXT:     BFE_INT T5.X, T0.Z, 0.0, literal.x,
-; CM-NEXT:     ASHR * T7.Z, T0.Y, literal.x,
+; CM-NEXT:     BFE_INT T6.X, T5.Y, 0.0, literal.x,
+; CM-NEXT:     ASHR * T5.Z, T5.X, literal.x,
 ; CM-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; CM-NEXT:     BFE_INT T7.X, T0.Y, 0.0, literal.x,
-; CM-NEXT:     ASHR * T5.Y, PV.X, literal.y,
+; CM-NEXT:     BFE_INT T5.X, T5.X, 0.0, literal.x,
+; CM-NEXT:     ASHR * T6.Y, PV.X, literal.y,
 ; CM-NEXT:    16(2.242078e-44), 31(4.344025e-44)
 ; CM-NEXT:     LSHR T8.X, KC0[2].Y, literal.x,
-; CM-NEXT:     ASHR * T7.Y, PV.X, literal.y,
+; CM-NEXT:     ASHR * T5.Y, PV.X, literal.y,
 ; CM-NEXT:    2(2.802597e-45), 31(4.344025e-44)
   %load = load <4 x i16>, ptr addrspace(1) %in
   %ext = sext <4 x i16> %load to <4 x i64>
@@ -6113,20 +6081,20 @@ define amdgpu_kernel void @global_zextload_v8i16_to_v8i64(ptr addrspace(1) %out,
 ; GCN-NOHSA-VI-NEXT:    s_mov_b32 s0, s4
 ; GCN-NOHSA-VI-NEXT:    s_mov_b32 s1, s5
 ; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v7, 0
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v5, 0
 ; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v11, 0
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v9, 0
 ; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v15, 0
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v13, v17
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v9, v17
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v5, v17
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v13, 0
 ; GCN-NOHSA-VI-NEXT:    s_waitcnt vmcnt(0)
-; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v16, 0xffff, v3
 ; GCN-NOHSA-VI-NEXT:    v_lshrrev_b32_e32 v18, 16, v3
-; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v4, 0xffff, v0
+; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v16, 0xffff, v3
 ; GCN-NOHSA-VI-NEXT:    v_lshrrev_b32_e32 v6, 16, v0
-; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v8, 0xffff, v1
+; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v4, 0xffff, v0
 ; GCN-NOHSA-VI-NEXT:    v_lshrrev_b32_e32 v10, 16, v1
-; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v12, 0xffff, v2
+; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v8, 0xffff, v1
 ; GCN-NOHSA-VI-NEXT:    v_lshrrev_b32_e32 v14, 16, v2
+; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v12, 0xffff, v2
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[16:19], off, s[0:3], 0 offset:48
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[12:15], off, s[0:3], 0 offset:32
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[8:11], off, s[0:3], 0 offset:16
@@ -6616,49 +6584,48 @@ define amdgpu_kernel void @global_zextload_v16i16_to_v16i64(ptr addrspace(1) %ou
 ; GCN-NOHSA-VI-NEXT:    s_mov_b32 s9, s7
 ; GCN-NOHSA-VI-NEXT:    buffer_load_dwordx4 v[0:3], off, s[8:11], 0
 ; GCN-NOHSA-VI-NEXT:    buffer_load_dwordx4 v[4:7], off, s[8:11], 0 offset:16
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v30, 0
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v28, 0
 ; GCN-NOHSA-VI-NEXT:    s_mov_b32 s0, s4
 ; GCN-NOHSA-VI-NEXT:    s_mov_b32 s1, s5
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v31, 0
 ; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v11, 0
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v9, 0
 ; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v15, 0
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v13, 0
 ; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v19, 0
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v17, 0
 ; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v23, 0
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v27, 0
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v21, 0
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v25, 0
 ; GCN-NOHSA-VI-NEXT:    s_waitcnt vmcnt(1)
-; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v16, 0xffff, v3
+; GCN-NOHSA-VI-NEXT:    v_lshrrev_b32_e32 v10, 16, v0
 ; GCN-NOHSA-VI-NEXT:    s_waitcnt vmcnt(0)
-; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v28, 0xffff, v4
-; GCN-NOHSA-VI-NEXT:    v_lshrrev_b32_e32 v30, 16, v4
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v4, 0
-; GCN-NOHSA-VI-NEXT:    v_lshrrev_b32_e32 v18, 16, v3
-; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v20, 0xffff, v6
 ; GCN-NOHSA-VI-NEXT:    v_lshrrev_b32_e32 v22, 16, v6
-; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v3, 0xffff, v5
-; GCN-NOHSA-VI-NEXT:    v_lshrrev_b32_e32 v5, 16, v5
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v6, 0
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v29, v4
+; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v20, 0xffff, v6
+; GCN-NOHSA-VI-NEXT:    v_lshrrev_b32_e32 v26, 16, v7
+; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v24, 0xffff, v7
+; GCN-NOHSA-VI-NEXT:    v_lshrrev_b32_e32 v6, 16, v4
+; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v4, 0xffff, v4
+; GCN-NOHSA-VI-NEXT:    v_lshrrev_b32_e32 v29, 16, v5
+; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v27, 0xffff, v5
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v7, 0
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v5, 0
 ; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v8, 0xffff, v0
-; GCN-NOHSA-VI-NEXT:    v_lshrrev_b32_e32 v10, 16, v0
-; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v12, 0xffff, v1
 ; GCN-NOHSA-VI-NEXT:    v_lshrrev_b32_e32 v14, 16, v1
-; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v0, 0xffff, v2
-; GCN-NOHSA-VI-NEXT:    v_lshrrev_b32_e32 v2, 16, v2
-; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v24, 0xffff, v7
-; GCN-NOHSA-VI-NEXT:    v_lshrrev_b32_e32 v26, 16, v7
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v25, v4
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v21, v4
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v17, v4
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v1, v4
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v13, v4
-; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v9, v4
-; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[3:6], off, s[0:3], 0 offset:80
-; GCN-NOHSA-VI-NEXT:    s_nop 0
+; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v12, 0xffff, v1
+; GCN-NOHSA-VI-NEXT:    v_lshrrev_b32_e32 v18, 16, v2
+; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v16, 0xffff, v2
+; GCN-NOHSA-VI-NEXT:    v_lshrrev_b32_e32 v2, 16, v3
+; GCN-NOHSA-VI-NEXT:    v_and_b32_e32 v0, 0xffff, v3
+; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[27:30], off, s[0:3], 0 offset:80
 ; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v3, 0
-; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[28:31], off, s[0:3], 0 offset:64
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v1, 0
+; GCN-NOHSA-VI-NEXT:    v_mov_b32_e32 v27, 0
+; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[4:7], off, s[0:3], 0 offset:64
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[24:27], off, s[0:3], 0 offset:112
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[20:23], off, s[0:3], 0 offset:96
-; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[16:19], off, s[0:3], 0 offset:48
-; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0 offset:32
+; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0 offset:48
+; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[16:19], off, s[0:3], 0 offset:32
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[12:15], off, s[0:3], 0 offset:16
 ; GCN-NOHSA-VI-NEXT:    buffer_store_dwordx4 v[8:11], off, s[0:3], 0
 ; GCN-NOHSA-VI-NEXT:    s_endpgm
diff --git a/llvm/test/CodeGen/AMDGPU/lower-kernel-and-module-lds.ll b/llvm/test/CodeGen/AMDGPU/lower-kernel-and-module-lds.ll
index 5aa5fcac08a9b..0c47b439706ba 100644
--- a/llvm/test/CodeGen/AMDGPU/lower-kernel-and-module-lds.ll
+++ b/llvm/test/CodeGen/AMDGPU/lower-kernel-and-module-lds.ll
@@ -14,19 +14,19 @@
 ; CHECK: %llvm.amdgcn.kernel.k3.lds.t = type { [4 x i8] }
 
 ;.
-; CHECK: @llvm.amdgcn.module.lds = internal addrspace(3) global %llvm.amdgcn.module.lds.t undef, align 8
+; CHECK: @llvm.amdgcn.module.lds = internal addrspace(3) global %llvm.amdgcn.module.lds.t undef, align 8, !absolute_symbol !0
 ; CHECK: @llvm.compiler.used = appending global [1 x ptr] [ptr addrspacecast (ptr addrspace(3) @llvm.amdgcn.module.lds to ptr)], section "llvm.metadata"
-; CHECK: @llvm.amdgcn.kernel.k0.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k0.lds.t undef, align 16
-; CHECK: @llvm.amdgcn.kernel.k1.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k1.lds.t undef, align 16
-; CHECK: @llvm.amdgcn.kernel.k2.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k2.lds.t undef, align 2
-; CHECK: @llvm.amdgcn.kernel.k3.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k3.lds.t undef, align 4
+; CHECK: @llvm.amdgcn.kernel.k0.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k0.lds.t undef, align 16, !absolute_symbol !0
+; CHECK: @llvm.amdgcn.kernel.k1.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k1.lds.t undef, align 16, !absolute_symbol !0
+; CHECK: @llvm.amdgcn.kernel.k2.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k2.lds.t undef, align 2, !absolute_symbol !0
+; CHECK: @llvm.amdgcn.kernel.k3.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k3.lds.t undef, align 4, !absolute_symbol !0
 ;.
 define amdgpu_kernel void @k0() #0 {
 ; CHECK-LABEL: @k0(
-; CHECK-NEXT: store i8 1, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, i32 0, i32 3), align 2, !alias.scope !0, !noalias !3
-; CHECK-NEXT: store i8 2, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, i32 0, i32 2), align 4, !alias.scope !7, !noalias !8
-; CHECK-NEXT: store i8 4, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, i32 0, i32 1), align 16, !alias.scope !9, !noalias !10
-; CHECK-NEXT: store i8 16, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, align 16, !alias.scope !11, !noalias !12
+; CHECK-NEXT: store i8 1, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, i32 0, i32 3), align 2, !alias.scope !1, !noalias !4
+; CHECK-NEXT: store i8 2, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, i32 0, i32 2), align 4, !alias.scope !8, !noalias !9
+; CHECK-NEXT: store i8 4, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, i32 0, i32 1), align 16, !alias.scope !10, !noalias !11
+; CHECK-NEXT: store i8 16, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, align 16, !alias.scope !12, !noalias !13
 ; CHECK-NEXT:    ret void
   store i8 1, ptr addrspace(3) @lds.size.1.align.1, align 1
 
@@ -41,9 +41,9 @@ define amdgpu_kernel void @k0() #0 {
 
 define amdgpu_kernel void @k1() #0 {
 ; CHECK-LABEL: @k1(
-; CHECK-NEXT: store i8 2, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k1.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k1.lds, i32 0, i32 2), align 4, !alias.scope !13, !noalias !16
-; CHECK-NEXT: store i8 4, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k1.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k1.lds, i32 0, i32 1), align 16, !alias.scope !19, !noalias !20
-; CHECK-NEXT: store i8 16, ptr addrspace(3) @llvm.amdgcn.kernel.k1.lds, align 16, !alias.scope !21, !noalias !22
+; CHECK-NEXT: store i8 2, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k1.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k1.lds, i32 0, i32 2), align 4, !alias.scope !14, !noalias !17
+; CHECK-NEXT: store i8 4, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k1.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k1.lds, i32 0, i32 1), align 16, !alias.scope !20, !noalias !21
+; CHECK-NEXT: store i8 16, ptr addrspace(3) @llvm.amdgcn.kernel.k1.lds, align 16, !alias.scope !22, !noalias !23
 ; CHECK-NEXT:    ret void
 ;
   store i8 2, ptr addrspace(3) @lds.size.2.align.2, align 2
@@ -83,8 +83,8 @@ define amdgpu_kernel void @calls_f0() {
 
 define void @f0() {
 ; CHECK-LABEL: define void @f0(
-; CHECK-NEXT: store i8 1, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.module.lds.t, ptr addrspace(3) @llvm.amdgcn.module.lds, i32 0, i32 1), align 8, !noalias !23
-; CHECK-NEXT: store i8 8, ptr addrspace(3) @llvm.amdgcn.module.lds, align 8, !noalias !23
+; CHECK-NEXT: store i8 1, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.module.lds.t, ptr addrspace(3) @llvm.amdgcn.module.lds, i32 0, i32 1), align 8, !noalias !24
+; CHECK-NEXT: store i8 8, ptr addrspace(3) @llvm.amdgcn.module.lds, align 8, !noalias !24
 ; CHECK-NEXT: ret void
   store i8 1, ptr addrspace(3) @lds.size.1.align.1, align 1
 
@@ -95,3 +95,5 @@ define void @f0() {
 
 attributes #0 = { "amdgpu-elide-module-lds" }
 ; CHECK: attributes #0 = { "amdgpu-elide-module-lds" }
+
+; CHECK: !0 = !{i64 0, i64 1}
diff --git a/llvm/test/CodeGen/AMDGPU/lower-kernel-lds.ll b/llvm/test/CodeGen/AMDGPU/lower-kernel-lds.ll
index da1923e1c1127..e9be71e8f4831 100644
--- a/llvm/test/CodeGen/AMDGPU/lower-kernel-lds.ll
+++ b/llvm/test/CodeGen/AMDGPU/lower-kernel-lds.ll
@@ -11,15 +11,15 @@
 
 ;.
 ; CHECK: @lds.k2 = addrspace(3) global [1 x i8] undef, align 1
-; CHECK: @llvm.amdgcn.kernel.k0.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k0.lds.t undef, align 16
-; CHECK: @llvm.amdgcn.kernel.k1.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k1.lds.t undef, align 16
+; CHECK: @llvm.amdgcn.kernel.k0.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k0.lds.t undef, align 16, !absolute_symbol !0
+; CHECK: @llvm.amdgcn.kernel.k1.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k1.lds.t undef, align 16, !absolute_symbol !0
 ;.
 define amdgpu_kernel void @k0() {
 ; CHECK-LABEL: @k0(
-; CHECK-NEXT: store i8 1, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, i32 0, i32 3), align 2, !alias.scope !0, !noalias !3
-; CHECK-NEXT: store i8 2, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, i32 0, i32 2), align 4, !alias.scope !7, !noalias !8
-; CHECK-NEXT: store i8 4, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, i32 0, i32 1), align 16, !alias.scope !9, !noalias !10
-; CHECK-NEXT: store i8 16, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, align 16, !alias.scope !11, !noalias !12
+; CHECK-NEXT: store i8 1, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, i32 0, i32 3), align 2, !alias.scope !1, !noalias !4
+; CHECK-NEXT: store i8 2, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, i32 0, i32 2), align 4, !alias.scope !8, !noalias !9
+; CHECK-NEXT: store i8 4, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, i32 0, i32 1), align 16, !alias.scope !10, !noalias !11
+; CHECK-NEXT: store i8 16, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, align 16, !alias.scope !12, !noalias !13
 ; CHECK-NEXT: ret void
   store i8 1, ptr addrspace(3) @lds.size.1.align.1, align 1
 
@@ -34,9 +34,9 @@ define amdgpu_kernel void @k0() {
 
 define amdgpu_kernel void @k1() {
 ; CHECK-LABEL: @k1(
-; CHECK-NEXT: store i8 2, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k1.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k1.lds, i32 0, i32 2), align 4, !alias.scope !13, !noalias !16
-; CHECK-NEXT: store i8 4, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k1.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k1.lds, i32 0, i32 1), align 16, !alias.scope !19, !noalias !20
-; CHECK-NEXT: store i8 16, ptr addrspace(3) @llvm.amdgcn.kernel.k1.lds, align 16, !alias.scope !21, !noalias !22
+; CHECK-NEXT: store i8 2, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k1.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k1.lds, i32 0, i32 2), align 4, !alias.scope !14, !noalias !17
+; CHECK-NEXT: store i8 4, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k1.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k1.lds, i32 0, i32 1), align 16, !alias.scope !20, !noalias !21
+; CHECK-NEXT: store i8 16, ptr addrspace(3) @llvm.amdgcn.kernel.k1.lds, align 16, !alias.scope !22, !noalias !23
 ; CHECK-NEXT: ret void
 ;
   store i8 2, ptr addrspace(3) @lds.size.2.align.2, align 2
@@ -61,3 +61,5 @@ define amdgpu_ps void @k2() {
 
   ret void
 }
+
+; CHECK: !0 = !{i64 0, i64 1}
diff --git a/llvm/test/CodeGen/AMDGPU/lower-lds-struct-aa-memcpy.ll b/llvm/test/CodeGen/AMDGPU/lower-lds-struct-aa-memcpy.ll
index 35a4d46b3a5ae..72ba840c15f17 100644
--- a/llvm/test/CodeGen/AMDGPU/lower-lds-struct-aa-memcpy.ll
+++ b/llvm/test/CodeGen/AMDGPU/lower-lds-struct-aa-memcpy.ll
@@ -13,7 +13,7 @@ $_f2 = comdat any
 @_f2 = linkonce_odr hidden local_unnamed_addr addrspace(3) global %vec_type undef, comdat, align 1
 
 ;.
-; CHECK: @[[LLVM_AMDGCN_KERNEL_TEST_LDS:[a-zA-Z0-9_$"\\.-]+]] = internal addrspace(3) global [[LLVM_AMDGCN_KERNEL_TEST_LDS_T:%.*]] undef, align 4
+; CHECK: @[[LLVM_AMDGCN_KERNEL_TEST_LDS:[a-zA-Z0-9_$"\\.-]+]] = internal addrspace(3) global [[LLVM_AMDGCN_KERNEL_TEST_LDS_T:%.*]] undef, align 4, !absolute_symbol !0
 ;.
 define protected amdgpu_kernel void @test(ptr addrspace(1) nocapture %ptr.coerce) local_unnamed_addr #0 {
 ; GCN-LABEL: test:
@@ -33,13 +33,13 @@ define protected amdgpu_kernel void @test(ptr addrspace(1) nocapture %ptr.coerce
 ; GCN-NEXT:    s_endpgm
 ; CHECK-LABEL: @test(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    store i8 3, ptr addrspace(3) @llvm.amdgcn.kernel.test.lds, align 4, !alias.scope !0, !noalias !3
-; CHECK-NEXT:    tail call void @llvm.memcpy.p3.p3.i64(ptr addrspace(3) noundef align 1 dereferenceable(3) getelementptr inbounds (%llvm.amdgcn.kernel.test.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.test.lds, i32 0, i32 2), ptr addrspace(3) noundef align 1 dereferenceable(3) @llvm.amdgcn.kernel.test.lds, i64 3, i1 false), !alias.scope !5, !noalias !6
-; CHECK-NEXT:    [[TMP4:%.*]] = load i8, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.test.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.test.lds, i32 0, i32 2), align 4, !alias.scope !3, !noalias !0
+; CHECK-NEXT:    store i8 3, ptr addrspace(3) @llvm.amdgcn.kernel.test.lds, align 4, !alias.scope !1, !noalias !4
+; CHECK-NEXT:    tail call void @llvm.memcpy.p3.p3.i64(ptr addrspace(3) noundef align 1 dereferenceable(3) getelementptr inbounds (%llvm.amdgcn.kernel.test.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.test.lds, i32 0, i32 2), ptr addrspace(3) noundef align 1 dereferenceable(3) @llvm.amdgcn.kernel.test.lds, i64 3, i1 false), !alias.scope !6, !noalias !7
+; CHECK-NEXT:    [[TMP4:%.*]] = load i8, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.test.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.test.lds, i32 0, i32 2), align 4, !alias.scope !4, !noalias !1
 ; CHECK-NEXT:    [[CMP_I_I:%.*]] = icmp eq i8 [[TMP4]], 3
-; CHECK-NEXT:    store i8 2, ptr addrspace(3) @llvm.amdgcn.kernel.test.lds, align 4, !alias.scope !0, !noalias !3
-; CHECK-NEXT: tail call void @llvm.memcpy.p3.p3.i64(ptr addrspace(3) noundef align 1 dereferenceable(3) getelementptr inbounds (%llvm.amdgcn.kernel.test.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.test.lds, i32 0, i32 2), ptr addrspace(3) noundef align 1 dereferenceable(3) @llvm.amdgcn.kernel.test.lds, i64 3, i1 false), !alias.scope !5, !noalias !6
-; CHECK-NEXT:    [[TMP9:%.*]] = load i8, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.test.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.test.lds, i32 0, i32 2), align 4, !alias.scope !3, !noalias !0
+; CHECK-NEXT:    store i8 2, ptr addrspace(3) @llvm.amdgcn.kernel.test.lds, align 4, !alias.scope !1, !noalias !4
+; CHECK-NEXT: tail call void @llvm.memcpy.p3.p3.i64(ptr addrspace(3) noundef align 1 dereferenceable(3) getelementptr inbounds (%llvm.amdgcn.kernel.test.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.test.lds, i32 0, i32 2), ptr addrspace(3) noundef align 1 dereferenceable(3) @llvm.amdgcn.kernel.test.lds, i64 3, i1 false), !alias.scope !6, !noalias !7
+; CHECK-NEXT:    [[TMP9:%.*]] = load i8, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.test.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.test.lds, i32 0, i32 2), align 4, !alias.scope !4, !noalias !1
 ; CHECK-NEXT:    [[CMP_I_I19:%.*]] = icmp eq i8 [[TMP9]], 2
 ; CHECK-NEXT:    [[TMP10:%.*]] = and i1 [[CMP_I_I19]], [[CMP_I_I]]
 ; CHECK-NEXT:    [[FROMBOOL8:%.*]] = zext i1 [[TMP10]] to i8
@@ -66,11 +66,12 @@ declare void @llvm.memcpy.p3.p3.i64(ptr addrspace(3) noalias nocapture writeonly
 ;.
 ; CHECK: attributes #[[ATTR0:[0-9]+]] = { nocallback nofree nounwind willreturn memory(argmem: readwrite) }
 ;.
-; CHECK: [[META0:![0-9]+]] = !{!1}
-; CHECK: [[META1:![0-9]+]] = distinct !{!1, !2}
-; CHECK: [[META2:![0-9]+]] = distinct !{!2}
-; CHECK: [[META3:![0-9]+]] = !{!4}
-; CHECK: [[META4:![0-9]+]] = distinct !{!4, !2}
-; CHECK: [[META5:![0-9]+]] = !{!4, !1}
-; CHECK: [[META6:![0-9]+]] = !{}
+; CHECK: [[META0:![0-9]+]] = !{i64 0, i64 1}
+; CHECK: [[META1:![0-9]+]] = !{!2}
+; CHECK: [[META2:![0-9]+]] = distinct !{!2, !3}
+; CHECK: [[META3:![0-9]+]] = distinct !{!3}
+; CHECK: [[META4:![0-9]+]] = !{!5}
+; CHECK: [[META5:![0-9]+]] = distinct !{!5, !3}
+; CHECK: [[META6:![0-9]+]] = !{!5, !2}
+; CHECK: [[META7:![0-9]+]] = !{}
 ;.
diff --git a/llvm/test/CodeGen/AMDGPU/lower-lds-struct-aa-merge.ll b/llvm/test/CodeGen/AMDGPU/lower-lds-struct-aa-merge.ll
index 8ed0fb7f66877..10064664aa991 100644
--- a/llvm/test/CodeGen/AMDGPU/lower-lds-struct-aa-merge.ll
+++ b/llvm/test/CodeGen/AMDGPU/lower-lds-struct-aa-merge.ll
@@ -5,10 +5,10 @@
 @b = internal unnamed_addr addrspace(3) global [64 x i32] undef, align 4
 
 ; CHECK-LABEL: @no_clobber_ds_load_stores_x2_preexisting_aa
-; CHECK: store i32 1, ptr addrspace(3) @llvm.amdgcn.kernel.no_clobber_ds_load_stores_x2_preexisting_aa.lds, align 16, !tbaa !0, !noalias !5
-; CHECK: %val.a = load i32, ptr addrspace(3) %gep.a, align 4, !tbaa !0, !noalias !5
-; CHECK: store i32 2, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.no_clobber_ds_load_stores_x2_preexisting_aa.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.no_clobber_ds_load_stores_x2_preexisting_aa.lds, i32 0, i32 1), align 16, !tbaa !0, !noalias !5
-; CHECK: %val.b = load i32, ptr addrspace(3) %gep.b, align 4, !tbaa !0, !noalias !5
+; CHECK: store i32 1, ptr addrspace(3) @llvm.amdgcn.kernel.no_clobber_ds_load_stores_x2_preexisting_aa.lds, align 16, !tbaa !1, !noalias !6
+; CHECK: %val.a = load i32, ptr addrspace(3) %gep.a, align 4, !tbaa !1, !noalias !6
+; CHECK: store i32 2, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.no_clobber_ds_load_stores_x2_preexisting_aa.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.no_clobber_ds_load_stores_x2_preexisting_aa.lds, i32 0, i32 1), align 16, !tbaa !1, !noalias !6
+; CHECK: %val.b = load i32, ptr addrspace(3) %gep.b, align 4, !tbaa !1, !noalias !6
 
 define amdgpu_kernel void @no_clobber_ds_load_stores_x2_preexisting_aa(ptr addrspace(1) %arg, i32 %i) {
 bb:
@@ -34,9 +34,10 @@ bb:
 !8 = !{!"omnipotent char", !9, i64 0}
 !9 = !{!"Simple C++ TBAA"}
 
-; CHECK:!0 = !{!1, !2, i64 0}
-; CHECK:!1 = !{!"no_clobber_ds_load_stores_x2_preexisting_aa", !2, i64 0}
-; CHECK:!2 = !{!"int", !3, i64 0}
-; CHECK:!3 = !{!"omnipotent char", !4, i64 0}
-; CHECK:!4 = !{!"Simple C++ TBAA"}
-; CHECK:!5 = !{}
+; CHECK:!0 = !{i64 0, i64 1}
+; CHECK:!1 = !{!2, !3, i64 0}
+; CHECK:!2 = !{!"no_clobber_ds_load_stores_x2_preexisting_aa", !3, i64 0}
+; CHECK:!3 = !{!"int", !4, i64 0}
+; CHECK:!4 = !{!"omnipotent char", !5, i64 0}
+; CHECK:!5 = !{!"Simple C++ TBAA"}
+; CHECK:!6 = !{}
diff --git a/llvm/test/CodeGen/AMDGPU/lower-lds-struct-aa.ll b/llvm/test/CodeGen/AMDGPU/lower-lds-struct-aa.ll
index 983516c3cabdc..d7697c903721b 100644
--- a/llvm/test/CodeGen/AMDGPU/lower-lds-struct-aa.ll
+++ b/llvm/test/CodeGen/AMDGPU/lower-lds-struct-aa.ll
@@ -17,10 +17,10 @@
 ; GCN: ds_read_b32
 
 ; CHECK-LABEL: @no_clobber_ds_load_stores_x2
-; CHECK: store i32 1, ptr addrspace(3) @llvm.amdgcn.kernel.no_clobber_ds_load_stores_x2.lds, align 16, !alias.scope !0, !noalias !3
-; CHECK: %val.a = load i32, ptr addrspace(3) %gep.a, align 4, !alias.scope !0, !noalias !3
-; CHECK: store i32 2, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.no_clobber_ds_load_stores_x2.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.no_clobber_ds_load_stores_x2.lds, i32 0, i32 1), align 16, !alias.scope !3, !noalias !0
-; CHECK: %val.b = load i32, ptr addrspace(3) %gep.b, align 4, !alias.scope !3, !noalias !0
+; CHECK: store i32 1, ptr addrspace(3) @llvm.amdgcn.kernel.no_clobber_ds_load_stores_x2.lds, align 16, !alias.scope !1, !noalias !4
+; CHECK: %val.a = load i32, ptr addrspace(3) %gep.a, align 4, !alias.scope !1, !noalias !4
+; CHECK: store i32 2, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.no_clobber_ds_load_stores_x2.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.no_clobber_ds_load_stores_x2.lds, i32 0, i32 1), align 16, !alias.scope !4, !noalias !1
+; CHECK: %val.b = load i32, ptr addrspace(3) %gep.b, align 4, !alias.scope !4, !noalias !1
 
 define amdgpu_kernel void @no_clobber_ds_load_stores_x2(ptr addrspace(1) %arg, i32 %i) {
 bb:
@@ -44,13 +44,13 @@ bb:
 ; GCN-DAG: ds_read_b32
 
 ; CHECK-LABEL: @no_clobber_ds_load_stores_x3
-; CHECK: store i32 1, ptr addrspace(3) @llvm.amdgcn.kernel.no_clobber_ds_load_stores_x3.lds, align 16, !alias.scope !5, !noalias !8
+; CHECK: store i32 1, ptr addrspace(3) @llvm.amdgcn.kernel.no_clobber_ds_load_stores_x3.lds, align 16, !alias.scope !6, !noalias !9
 ; CHECK: %gep.a = getelementptr inbounds [64 x i32], ptr addrspace(3) @llvm.amdgcn.kernel.no_clobber_ds_load_stores_x3.lds, i32 0, i32 %i
-; CHECK: %val.a = load i32, ptr addrspace(3) %gep.a, align 4, !alias.scope !5, !noalias !8
-; CHECK: store i32 2, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.no_clobber_ds_load_stores_x3.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.no_clobber_ds_load_stores_x3.lds, i32 0, i32 1), align 16, !alias.scope !11, !noalias !12
-; CHECK: %val.b = load i32, ptr addrspace(3) %gep.b, align 4, !alias.scope !11, !noalias !12
-; CHECK: store i32 3, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.no_clobber_ds_load_stores_x3.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.no_clobber_ds_load_stores_x3.lds, i32 0, i32 2), align 16, !alias.scope !13, !noalias !14
-; CHECK: %val.c = load i32, ptr addrspace(3) %gep.c, align 4, !alias.scope !13, !noalias !14
+; CHECK: %val.a = load i32, ptr addrspace(3) %gep.a, align 4, !alias.scope !6, !noalias !9
+; CHECK: store i32 2, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.no_clobber_ds_load_stores_x3.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.no_clobber_ds_load_stores_x3.lds, i32 0, i32 1), align 16, !alias.scope !12, !noalias !13
+; CHECK: %val.b = load i32, ptr addrspace(3) %gep.b, align 4, !alias.scope !12, !noalias !13
+; CHECK: store i32 3, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.no_clobber_ds_load_stores_x3.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.no_clobber_ds_load_stores_x3.lds, i32 0, i32 2), align 16, !alias.scope !14, !noalias !15
+; CHECK: %val.c = load i32, ptr addrspace(3) %gep.c, align 4, !alias.scope !14, !noalias !15
 
 define amdgpu_kernel void @no_clobber_ds_load_stores_x3(ptr addrspace(1) %arg, i32 %i) {
 bb:
@@ -69,18 +69,19 @@ bb:
   ret void
 }
 
-; CHECK: !0 = !{!1}
-; CHECK: !1 = distinct !{!1, !2}
-; CHECK: !2 = distinct !{!2}
-; CHECK: !3 = !{!4}
-; CHECK: !4 = distinct !{!4, !2}
-; CHECK: !5 = !{!6}
-; CHECK: !6 = distinct !{!6, !7}
-; CHECK: !7 = distinct !{!7}
-; CHECK: !8 = !{!9, !10}
-; CHECK: !9 = distinct !{!9, !7}
-; CHECK: !10 = distinct !{!10, !7}
-; CHECK: !11 = !{!9}
-; CHECK: !12 = !{!6, !10}
-; CHECK: !13 = !{!10}
-; CHECK: !14 = !{!6, !9}
+; CHECK: !0 = !{i64 0, i64 1}
+; CHECK: !1 = !{!2}
+; CHECK: !2 = distinct !{!2, !3}
+; CHECK: !3 = distinct !{!3}
+; CHECK: !4 = !{!5}
+; CHECK: !5 = distinct !{!5, !3}
+; CHECK: !6 = !{!7}
+; CHECK: !7 = distinct !{!7, !8}
+; CHECK: !8 = distinct !{!8}
+; CHECK: !9 = !{!10, !11}
+; CHECK: !10 = distinct !{!10, !8}
+; CHECK: !11 = distinct !{!11, !8}
+; CHECK: !12 = !{!10}
+; CHECK: !13 = !{!7, !11}
+; CHECK: !14 = !{!11}
+; CHECK: !15 = !{!7, !10}
diff --git a/llvm/test/CodeGen/AMDGPU/lower-module-lds-single-var-unambiguous.ll b/llvm/test/CodeGen/AMDGPU/lower-module-lds-single-var-unambiguous.ll
index c380bb1c9b926..b3a0f94d4cde7 100644
--- a/llvm/test/CodeGen/AMDGPU/lower-module-lds-single-var-unambiguous.ll
+++ b/llvm/test/CodeGen/AMDGPU/lower-module-lds-single-var-unambiguous.ll
@@ -28,9 +28,9 @@ define amdgpu_kernel void @k0() {
 @f0.lds = addrspace(3) global i16 undef
 define void @f0() {
 ; MODULE-LABEL: @f0(
-; MODULE-NEXT:    [[LD:%.*]] = load i16, ptr addrspace(3) getelementptr inbounds ([[LLVM_AMDGCN_MODULE_LDS_T:%.*]], ptr addrspace(3) @llvm.amdgcn.module.lds, i32 0, i32 1), align 4, !alias.scope !0, !noalias !3
+; MODULE-NEXT:    [[LD:%.*]] = load i16, ptr addrspace(3) getelementptr inbounds ([[LLVM_AMDGCN_MODULE_LDS_T:%.*]], ptr addrspace(3) @llvm.amdgcn.module.lds, i32 0, i32 1), align 4, !alias.scope !1, !noalias !4
 ; MODULE-NEXT:    [[MUL:%.*]] = mul i16 [[LD]], 3
-; MODULE-NEXT:    store i16 [[MUL]], ptr addrspace(3) getelementptr inbounds ([[LLVM_AMDGCN_MODULE_LDS_T]], ptr addrspace(3) @llvm.amdgcn.module.lds, i32 0, i32 1), align 4, !alias.scope !0, !noalias !3
+; MODULE-NEXT:    store i16 [[MUL]], ptr addrspace(3) getelementptr inbounds ([[LLVM_AMDGCN_MODULE_LDS_T]], ptr addrspace(3) @llvm.amdgcn.module.lds, i32 0, i32 1), align 4, !alias.scope !1, !noalias !4
 ; MODULE-NEXT:    ret void
 ;
 ; TABLE-LABEL: @f0(
@@ -60,7 +60,7 @@ define void @f0() {
 
 define amdgpu_kernel void @k_f0() {
 ; MODULE-LABEL: @k_f0(
-; MODULE-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.module.lds) ]
+; MODULE-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.module.lds) ], !alias.scope !5, !noalias !1
 ; MODULE-NEXT:    call void @f0()
 ; MODULE-NEXT:    ret void
 ;
@@ -82,9 +82,9 @@ define amdgpu_kernel void @k_f0() {
 @both.lds = addrspace(3) global i32 undef
 define void @f_both() {
 ; MODULE-LABEL: @f_both(
-; MODULE-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(3) @llvm.amdgcn.module.lds, align 4, !alias.scope !4, !noalias !3
+; MODULE-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(3) @llvm.amdgcn.module.lds, align 4, !alias.scope !5, !noalias !4
 ; MODULE-NEXT:    [[MUL:%.*]] = mul i32 [[LD]], 4
-; MODULE-NEXT:    store i32 [[MUL]], ptr addrspace(3) @llvm.amdgcn.module.lds, align 4, !alias.scope !4, !noalias !3
+; MODULE-NEXT:    store i32 [[MUL]], ptr addrspace(3) @llvm.amdgcn.module.lds, align 4, !alias.scope !5, !noalias !4
 ; MODULE-NEXT:    ret void
 ;
 ; TABLE-LABEL: @f_both(
@@ -115,9 +115,9 @@ define void @f_both() {
 define amdgpu_kernel void @k0_both() {
 ; MODULE-LABEL: @k0_both(
 ; MODULE-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.module.lds) ]
-; MODULE-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(3) @llvm.amdgcn.module.lds, align 4, !alias.scope !4, !noalias !0
+; MODULE-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(3) @llvm.amdgcn.module.lds, align 4, !alias.scope !5, !noalias !1
 ; MODULE-NEXT:    [[MUL:%.*]] = mul i32 [[LD]], 5
-; MODULE-NEXT:    store i32 [[MUL]], ptr addrspace(3) @llvm.amdgcn.module.lds, align 4, !alias.scope !4, !noalias !0
+; MODULE-NEXT:    store i32 [[MUL]], ptr addrspace(3) @llvm.amdgcn.module.lds, align 4, !alias.scope !5, !noalias !1
 ; MODULE-NEXT:    call void @f_both()
 ; MODULE-NEXT:    ret void
 ;
diff --git a/llvm/test/CodeGen/AMDGPU/lower-module-lds-via-hybrid.ll b/llvm/test/CodeGen/AMDGPU/lower-module-lds-via-hybrid.ll
index cdead7560b295..75211d595273b 100644
--- a/llvm/test/CodeGen/AMDGPU/lower-module-lds-via-hybrid.ll
+++ b/llvm/test/CodeGen/AMDGPU/lower-module-lds-via-hybrid.ll
@@ -11,12 +11,12 @@
 @v3 = addrspace(3) global i8 undef
 @unused = addrspace(3) global i16 undef
 
-; OPT: @llvm.amdgcn.module.lds = internal addrspace(3) global %llvm.amdgcn.module.lds.t undef, align 16
+; OPT: @llvm.amdgcn.module.lds = internal addrspace(3) global %llvm.amdgcn.module.lds.t undef, align 16, !absolute_symbol !0
 ; OPT: @llvm.compiler.used = appending global [1 x ptr] [ptr addrspacecast (ptr addrspace(3) @llvm.amdgcn.module.lds to ptr)], section "llvm.metadata"
-; OPT: @llvm.amdgcn.kernel.kernel_no_table.lds = internal addrspace(3) global %llvm.amdgcn.kernel.kernel_no_table.lds.t undef, align 8
-; OPT: @llvm.amdgcn.kernel.k01.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k01.lds.t undef, align 4
-; OPT: @llvm.amdgcn.kernel.k23.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k23.lds.t undef, align 8
-; OPT: @llvm.amdgcn.kernel.k123.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k123.lds.t undef, align 8
+; OPT: @llvm.amdgcn.kernel.kernel_no_table.lds = internal addrspace(3) global %llvm.amdgcn.kernel.kernel_no_table.lds.t undef, align 8, !absolute_symbol !0
+; OPT: @llvm.amdgcn.kernel.k01.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k01.lds.t undef, align 4, !absolute_symbol !1
+; OPT: @llvm.amdgcn.kernel.k23.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k23.lds.t undef, align 8, !absolute_symbol !0
+; OPT: @llvm.amdgcn.kernel.k123.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k123.lds.t undef, align 8, !absolute_symbol !2
 ; OPT{LITERAL}: @llvm.amdgcn.lds.offset.table = internal addrspace(4) constant [2 x [1 x i32]] [[1 x i32] [i32 ptrtoint (ptr addrspace(3) @llvm.amdgcn.kernel.k123.lds to i32)], [1 x i32] [i32 ptrtoint (ptr addrspace(3) @llvm.amdgcn.kernel.k23.lds to i32)]]
 
 ;.
@@ -234,9 +234,9 @@ define amdgpu_kernel void @k123() {
 ; OPT-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.kernel.k123.lds) ]
 ; OPT-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.module.lds) ]
 ; OPT-NEXT:    call void @f1()
-; OPT-NEXT:    %ld = load i8, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k123.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k123.lds, i32 0, i32 1), align 8, !alias.scope !2, !noalias !5
+; OPT-NEXT:    %ld = load i8, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k123.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k123.lds, i32 0, i32 1), align 8, !alias.scope !5, !noalias !8
 ; OPT-NEXT:    %mul = mul i8 %ld, 8
-; OPT-NEXT:    store i8 %mul, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k123.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k123.lds, i32 0, i32 1), align 8, !alias.scope !2, !noalias !5
+; OPT-NEXT:    store i8 %mul, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k123.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k123.lds, i32 0, i32 1), align 8, !alias.scope !5, !noalias !8
 ; OPT-NEXT:    call void @f2()
 ; OPT-NEXT:    ret void
 ;
@@ -289,13 +289,16 @@ define amdgpu_kernel void @k123() {
 ; OPT: attributes #1 = { nocallback nofree nosync nounwind willreturn memory(none) }
 ; OPT: attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memory(none) }
 ;.
-; OPT: !0 = !{i32 1}
-; OPT: !1 = !{i32 0}
-; OPT: !2 = !{!3}
-; OPT: !3 = distinct !{!3, !4}
-; OPT: !4 = distinct !{!4}
+; OPT: !0 = !{i64 0, i64 1}
+; OPT: !1 = !{i64 4, i64 5}
+; OPT: !2 = !{i64 8, i64 9}
+; OPT: !3 = !{i32 1}
+; OPT: !4 = !{i32 0}
 ; OPT: !5 = !{!6}
-; OPT: !6 = distinct !{!6, !4}
+; OPT: !6 = distinct !{!6, !7}
+; OPT: !7 = distinct !{!7}
+; OPT: !8 = !{!9}
+; OPT: !9 = distinct !{!9, !7}
 ;.
 
 ; Table size length number-kernels * number-variables * sizeof(uint16_t)
diff --git a/llvm/test/CodeGen/AMDGPU/lower-module-lds-via-table.ll b/llvm/test/CodeGen/AMDGPU/lower-module-lds-via-table.ll
index 8b289f12fce48..5b7da296f2e94 100644
--- a/llvm/test/CodeGen/AMDGPU/lower-module-lds-via-table.ll
+++ b/llvm/test/CodeGen/AMDGPU/lower-module-lds-via-table.ll
@@ -15,10 +15,10 @@
 ; OPT: %llvm.amdgcn.kernel.k23.lds.t = type { i64, i8 }
 ; OPT: %llvm.amdgcn.kernel.k123.lds.t = type { i16, i8, [5 x i8], i64 }
 
-; OPT: @llvm.amdgcn.kernel.kernel_no_table.lds = internal addrspace(3) global %llvm.amdgcn.kernel.kernel_no_table.lds.t undef, align 8
-; OPT: @llvm.amdgcn.kernel.k01.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k01.lds.t undef, align 16
-; OPT: @llvm.amdgcn.kernel.k23.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k23.lds.t undef, align 8
-; OPT: @llvm.amdgcn.kernel.k123.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k123.lds.t undef, align 16
+; OPT: @llvm.amdgcn.kernel.kernel_no_table.lds = internal addrspace(3) global %llvm.amdgcn.kernel.kernel_no_table.lds.t undef, align 8, !absolute_symbol !0
+; OPT: @llvm.amdgcn.kernel.k01.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k01.lds.t undef, align 16, !absolute_symbol !0
+; OPT: @llvm.amdgcn.kernel.k23.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k23.lds.t undef, align 8, !absolute_symbol !0
+; OPT: @llvm.amdgcn.kernel.k123.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k123.lds.t undef, align 16, !absolute_symbol !0
 
 ; Salient parts of the IR lookup table check:
 ; It has (top level) size 3 as there are 3 kernels that call functions which use lds
@@ -218,7 +218,7 @@ define amdgpu_kernel void @kernel_no_table() {
 
 ; Access two variables, will allocate those two
 define amdgpu_kernel void @k01() {
-; OPT-LABEL: @k01() !llvm.amdgcn.lds.kernel.id !0 {
+; OPT-LABEL: @k01() !llvm.amdgcn.lds.kernel.id !1 {
 ; OPT-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.kernel.k01.lds) ]
 ; OPT-NEXT:    call void @f0()
 ; OPT-NEXT:    call void @f1()
@@ -256,7 +256,7 @@ define amdgpu_kernel void @k01() {
 }
 
 define amdgpu_kernel void @k23() {
-; OPT-LABEL: @k23() !llvm.amdgcn.lds.kernel.id !1 {
+; OPT-LABEL: @k23() !llvm.amdgcn.lds.kernel.id !2 {
 ; OPT-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.kernel.k23.lds) ]
 ; OPT-NEXT:    call void @f2()
 ; OPT-NEXT:    call void @f3()
@@ -295,12 +295,12 @@ define amdgpu_kernel void @k23() {
 
 ; Access and allocate three variables
 define amdgpu_kernel void @k123() {
-; OPT-LABEL: @k123() !llvm.amdgcn.lds.kernel.id !2 {
+; OPT-LABEL: @k123() !llvm.amdgcn.lds.kernel.id !3 {
 ; OPT-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.kernel.k123.lds) ]
 ; OPT-NEXT:    call void @f1()
-; OPT-NEXT:    [[LD:%.*]] = load i8, ptr addrspace(3) getelementptr inbounds ([[LLVM_AMDGCN_KERNEL_K123_LDS_T:%.*]], ptr addrspace(3) @llvm.amdgcn.kernel.k123.lds, i32 0, i32 1), align 2, !alias.scope !3, !noalias !6
+; OPT-NEXT:    [[LD:%.*]] = load i8, ptr addrspace(3) getelementptr inbounds ([[LLVM_AMDGCN_KERNEL_K123_LDS_T:%.*]], ptr addrspace(3) @llvm.amdgcn.kernel.k123.lds, i32 0, i32 1), align 2, !alias.scope !4, !noalias !7
 ; OPT-NEXT:    [[MUL:%.*]] = mul i8 [[LD]], 8
-; OPT-NEXT:    store i8 [[MUL]], ptr addrspace(3) getelementptr inbounds ([[LLVM_AMDGCN_KERNEL_K123_LDS_T]], ptr addrspace(3) @llvm.amdgcn.kernel.k123.lds, i32 0, i32 1), align 2, !alias.scope !3, !noalias !6
+; OPT-NEXT:    store i8 [[MUL]], ptr addrspace(3) getelementptr inbounds ([[LLVM_AMDGCN_KERNEL_K123_LDS_T]], ptr addrspace(3) @llvm.amdgcn.kernel.k123.lds, i32 0, i32 1), align 2, !alias.scope !4, !noalias !7
 ; OPT-NEXT:    call void @f2()
 ; OPT-NEXT:    ret void
 ;
@@ -346,9 +346,10 @@ define amdgpu_kernel void @k123() {
 
 ; OPT: declare i32 @llvm.amdgcn.lds.kernel.id()
 
-!0 = !{i32 0}
-!1 = !{i32 2}
-!2 = !{i32 1}
+!0 = !{i64 0, i64 1}
+!1 = !{i32 0}
+!2 = !{i32 2}
+!3 = !{i32 1}
 
 
 ; Table size length number-kernels * number-variables * sizeof(uint16_t)
diff --git a/llvm/test/CodeGen/AMDGPU/lower-work-group-id-intrinsics.ll b/llvm/test/CodeGen/AMDGPU/lower-work-group-id-intrinsics.ll
new file mode 100644
index 0000000000000..83ea07ab4d924
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/lower-work-group-id-intrinsics.ll
@@ -0,0 +1,39 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=amdgcn-amd-amdpal -mcpu=gfx900 -mattr=+architected-sgprs --verify-machineinstrs < %s | FileCheck -check-prefix=GFX9-SDAG %s
+; RUN: llc -mtriple=amdgcn-amd-amdpal -mcpu=gfx900 -mattr=+architected-sgprs -global-isel --verify-machineinstrs < %s | FileCheck -check-prefix=GFX9-GISEL %s
+define amdgpu_cs void @_amdgpu_cs_main() {
+; GFX9-SDAG-LABEL: _amdgpu_cs_main:
+; GFX9-SDAG:       ; %bb.0: ; %.entry
+; GFX9-SDAG-NEXT:    s_lshr_b32 s2, ttmp7, 16
+; GFX9-SDAG-NEXT:    s_and_b32 s1, ttmp7, 0xffff
+; GFX9-SDAG-NEXT:    v_mov_b32_e32 v0, ttmp9
+; GFX9-SDAG-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-SDAG-NEXT:    v_mov_b32_e32 v2, s2
+; GFX9-SDAG-NEXT:    buffer_store_dwordx3 v[0:2], off, s[0:3], 0
+; GFX9-SDAG-NEXT:    s_endpgm
+;
+; GFX9-GISEL-LABEL: _amdgpu_cs_main:
+; GFX9-GISEL:       ; %bb.0: ; %.entry
+; GFX9-GISEL-NEXT:    s_mov_b32 s0, ttmp9
+; GFX9-GISEL-NEXT:    s_and_b32 s1, ttmp7, 0xffff
+; GFX9-GISEL-NEXT:    s_lshr_b32 s2, ttmp7, 16
+; GFX9-GISEL-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-GISEL-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-GISEL-NEXT:    v_mov_b32_e32 v2, s2
+; GFX9-GISEL-NEXT:    buffer_store_dwordx3 v[0:2], off, s[0:3], 0
+; GFX9-GISEL-NEXT:    s_endpgm
+.entry:
+  %idx = call i32 @llvm.amdgcn.workgroup.id.x()
+  %idy = call i32 @llvm.amdgcn.workgroup.id.y()
+  %idz = call i32 @llvm.amdgcn.workgroup.id.z()
+  %ielemx = insertelement <3 x i32> undef, i32 %idx, i64 0
+  %ielemy = insertelement <3 x i32> %ielemx, i32 %idy, i64 1
+  %ielemz = insertelement <3 x i32> %ielemy, i32 %idz, i64 2
+  call void @llvm.amdgcn.raw.buffer.store.v3i32(<3 x i32> %ielemz, <4 x i32> undef, i32 0, i32 0, i32 0)
+  ret void
+}
+
+declare i32 @llvm.amdgcn.workgroup.id.x()
+declare i32 @llvm.amdgcn.workgroup.id.y()
+declare i32 @llvm.amdgcn.workgroup.id.z()
+declare void @llvm.amdgcn.raw.buffer.store.v3i32(<3 x i32>, <4 x i32>, i32, i32, i32 immarg)
diff --git a/llvm/test/CodeGen/AMDGPU/mad_uint24.ll b/llvm/test/CodeGen/AMDGPU/mad_uint24.ll
index b9143150280a8..ef26280ffc9e0 100644
--- a/llvm/test/CodeGen/AMDGPU/mad_uint24.ll
+++ b/llvm/test/CodeGen/AMDGPU/mad_uint24.ll
@@ -158,9 +158,11 @@ bb18:                                             ; preds = %bb4
 ; EG: BFE_INT {{[* ]*}}T{{[0-9]\.[XYZW]}}, PV.[[MAD_CHAN]], 0.0, literal.x
 ; EG: 8
 ; SI: v_mad_u32_u24 [[MAD:v[0-9]]], {{[sv][0-9], [sv][0-9]}}
+; SI: v_bfe_i32 [[EXT:v[0-9]]], [[MAD]], 0, 16
+; SI: v_med3_i32 v{{[0-9]}}, [[EXT]],
 ; VI: v_mad_u16 [[MAD:v[0-9]]], {{[sv][0-9], [sv][0-9]}}
-; GCN: v_bfe_i32 [[EXT:v[0-9]]], [[MAD]], 0, 16
-; GCN: v_med3_i32 v{{[0-9]}}, [[EXT]],
+; VI: v_max_i16_e32 [[MAX:v[0-9]]], 0xff80, [[MAD]]
+; VI: v_min_i16_e32 {{v[0-9]}}, 0x7f, [[MAX]]
 define amdgpu_kernel void @i8_mad_sat_16(ptr addrspace(1) %out, ptr addrspace(1) %in0, ptr addrspace(1) %in1, ptr addrspace(1) %in2, ptr addrspace(5) %idx) {
 entry:
   %retval.0.i = load i64, ptr addrspace(5) %idx
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-fence.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-fence.ll
index 98406ab613eae..01428a968cad0 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-fence.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-fence.ll
@@ -1335,14 +1335,12 @@ define amdgpu_kernel void @agent_acquire_fence() {
 ;
 ; GFX940-NOTTGSPLIT-LABEL: agent_acquire_fence:
 ; GFX940-NOTTGSPLIT:       ; %bb.0: ; %entry
-; GFX940-NOTTGSPLIT-NEXT:    buffer_wbl2 sc1
 ; GFX940-NOTTGSPLIT-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX940-NOTTGSPLIT-NEXT:    buffer_inv sc1
 ; GFX940-NOTTGSPLIT-NEXT:    s_endpgm
 ;
 ; GFX940-TGSPLIT-LABEL: agent_acquire_fence:
 ; GFX940-TGSPLIT:       ; %bb.0: ; %entry
-; GFX940-TGSPLIT-NEXT:    buffer_wbl2 sc1
 ; GFX940-TGSPLIT-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX940-TGSPLIT-NEXT:    buffer_inv sc1
 ; GFX940-TGSPLIT-NEXT:    s_endpgm
@@ -1641,14 +1639,12 @@ define amdgpu_kernel void @agent_one_as_acquire_fence() {
 ;
 ; GFX940-NOTTGSPLIT-LABEL: agent_one_as_acquire_fence:
 ; GFX940-NOTTGSPLIT:       ; %bb.0: ; %entry
-; GFX940-NOTTGSPLIT-NEXT:    buffer_wbl2 sc1
 ; GFX940-NOTTGSPLIT-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NOTTGSPLIT-NEXT:    buffer_inv sc1
 ; GFX940-NOTTGSPLIT-NEXT:    s_endpgm
 ;
 ; GFX940-TGSPLIT-LABEL: agent_one_as_acquire_fence:
 ; GFX940-TGSPLIT:       ; %bb.0: ; %entry
-; GFX940-TGSPLIT-NEXT:    buffer_wbl2 sc1
 ; GFX940-TGSPLIT-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-TGSPLIT-NEXT:    buffer_inv sc1
 ; GFX940-TGSPLIT-NEXT:    s_endpgm
@@ -1935,7 +1931,6 @@ define amdgpu_kernel void @system_acquire_fence() {
 ;
 ; GFX90A-NOTTGSPLIT-LABEL: system_acquire_fence:
 ; GFX90A-NOTTGSPLIT:       ; %bb.0: ; %entry
-; GFX90A-NOTTGSPLIT-NEXT:    buffer_wbl2
 ; GFX90A-NOTTGSPLIT-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX90A-NOTTGSPLIT-NEXT:    buffer_invl2
 ; GFX90A-NOTTGSPLIT-NEXT:    buffer_wbinvl1_vol
@@ -1943,7 +1938,6 @@ define amdgpu_kernel void @system_acquire_fence() {
 ;
 ; GFX90A-TGSPLIT-LABEL: system_acquire_fence:
 ; GFX90A-TGSPLIT:       ; %bb.0: ; %entry
-; GFX90A-TGSPLIT-NEXT:    buffer_wbl2
 ; GFX90A-TGSPLIT-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX90A-TGSPLIT-NEXT:    buffer_invl2
 ; GFX90A-TGSPLIT-NEXT:    buffer_wbinvl1_vol
@@ -1951,14 +1945,12 @@ define amdgpu_kernel void @system_acquire_fence() {
 ;
 ; GFX940-NOTTGSPLIT-LABEL: system_acquire_fence:
 ; GFX940-NOTTGSPLIT:       ; %bb.0: ; %entry
-; GFX940-NOTTGSPLIT-NEXT:    buffer_wbl2 sc0 sc1
 ; GFX940-NOTTGSPLIT-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX940-NOTTGSPLIT-NEXT:    buffer_inv sc0 sc1
 ; GFX940-NOTTGSPLIT-NEXT:    s_endpgm
 ;
 ; GFX940-TGSPLIT-LABEL: system_acquire_fence:
 ; GFX940-TGSPLIT:       ; %bb.0: ; %entry
-; GFX940-TGSPLIT-NEXT:    buffer_wbl2 sc0 sc1
 ; GFX940-TGSPLIT-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX940-TGSPLIT-NEXT:    buffer_inv sc0 sc1
 ; GFX940-TGSPLIT-NEXT:    s_endpgm
@@ -2255,7 +2247,6 @@ define amdgpu_kernel void @system_one_as_acquire_fence() {
 ;
 ; GFX90A-NOTTGSPLIT-LABEL: system_one_as_acquire_fence:
 ; GFX90A-NOTTGSPLIT:       ; %bb.0: ; %entry
-; GFX90A-NOTTGSPLIT-NEXT:    buffer_wbl2
 ; GFX90A-NOTTGSPLIT-NEXT:    s_waitcnt vmcnt(0)
 ; GFX90A-NOTTGSPLIT-NEXT:    buffer_invl2
 ; GFX90A-NOTTGSPLIT-NEXT:    buffer_wbinvl1_vol
@@ -2263,7 +2254,6 @@ define amdgpu_kernel void @system_one_as_acquire_fence() {
 ;
 ; GFX90A-TGSPLIT-LABEL: system_one_as_acquire_fence:
 ; GFX90A-TGSPLIT:       ; %bb.0: ; %entry
-; GFX90A-TGSPLIT-NEXT:    buffer_wbl2
 ; GFX90A-TGSPLIT-NEXT:    s_waitcnt vmcnt(0)
 ; GFX90A-TGSPLIT-NEXT:    buffer_invl2
 ; GFX90A-TGSPLIT-NEXT:    buffer_wbinvl1_vol
@@ -2271,14 +2261,12 @@ define amdgpu_kernel void @system_one_as_acquire_fence() {
 ;
 ; GFX940-NOTTGSPLIT-LABEL: system_one_as_acquire_fence:
 ; GFX940-NOTTGSPLIT:       ; %bb.0: ; %entry
-; GFX940-NOTTGSPLIT-NEXT:    buffer_wbl2 sc0 sc1
 ; GFX940-NOTTGSPLIT-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NOTTGSPLIT-NEXT:    buffer_inv sc0 sc1
 ; GFX940-NOTTGSPLIT-NEXT:    s_endpgm
 ;
 ; GFX940-TGSPLIT-LABEL: system_one_as_acquire_fence:
 ; GFX940-TGSPLIT:       ; %bb.0: ; %entry
-; GFX940-TGSPLIT-NEXT:    buffer_wbl2 sc0 sc1
 ; GFX940-TGSPLIT-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-TGSPLIT-NEXT:    buffer_inv sc0 sc1
 ; GFX940-TGSPLIT-NEXT:    s_endpgm
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll
index 8cbaf57318911..20c1594f44fa5 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll
@@ -310,10 +310,10 @@ define amdgpu_kernel void @private_nontemporal_load_1(
 ; GFX940-NOTTGSPLIT:       ; %bb.0: ; %entry
 ; GFX940-NOTTGSPLIT-NEXT:    s_load_dword s4, s[0:1], 0x0
 ; GFX940-NOTTGSPLIT-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x8
-; GFX940-NOTTGSPLIT-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX940-NOTTGSPLIT-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX940-NOTTGSPLIT-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX940-NOTTGSPLIT-NEXT:    scratch_load_dword v0, v0, s4 nt
+; GFX940-NOTTGSPLIT-NEXT:    v_lshl_add_u32 v0, v0, 2, s4
+; GFX940-NOTTGSPLIT-NEXT:    scratch_load_dword v0, v0, off nt
 ; GFX940-NOTTGSPLIT-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NOTTGSPLIT-NEXT:    global_store_dword v1, v0, s[2:3]
 ; GFX940-NOTTGSPLIT-NEXT:    s_endpgm
@@ -322,21 +322,23 @@ define amdgpu_kernel void @private_nontemporal_load_1(
 ; GFX940-TGSPLIT:       ; %bb.0: ; %entry
 ; GFX940-TGSPLIT-NEXT:    s_load_dword s4, s[0:1], 0x0
 ; GFX940-TGSPLIT-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x8
-; GFX940-TGSPLIT-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX940-TGSPLIT-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX940-TGSPLIT-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX940-TGSPLIT-NEXT:    scratch_load_dword v0, v0, s4 nt
+; GFX940-TGSPLIT-NEXT:    v_lshl_add_u32 v0, v0, 2, s4
+; GFX940-TGSPLIT-NEXT:    scratch_load_dword v0, v0, off nt
 ; GFX940-TGSPLIT-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-TGSPLIT-NEXT:    global_store_dword v1, v0, s[2:3]
 ; GFX940-TGSPLIT-NEXT:    s_endpgm
 ;
 ; GFX11-WGP-LABEL: private_nontemporal_load_1:
 ; GFX11-WGP:       ; %bb.0: ; %entry
+; GFX11-WGP-NEXT:    s_clause 0x1
 ; GFX11-WGP-NEXT:    s_load_b32 s2, s[0:1], 0x0
-; GFX11-WGP-NEXT:    v_dual_mov_b32 v1, 0 :: v_dual_lshlrev_b32 v0, 2, v0
 ; GFX11-WGP-NEXT:    s_load_b64 s[0:1], s[0:1], 0x8
+; GFX11-WGP-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX11-WGP-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-WGP-NEXT:    scratch_load_b32 v0, v0, s2 slc dlc
+; GFX11-WGP-NEXT:    v_lshl_add_u32 v0, v0, 2, s2
+; GFX11-WGP-NEXT:    scratch_load_b32 v0, v0, off slc dlc
 ; GFX11-WGP-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-WGP-NEXT:    global_store_b32 v1, v0, s[0:1]
 ; GFX11-WGP-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -344,11 +346,13 @@ define amdgpu_kernel void @private_nontemporal_load_1(
 ;
 ; GFX11-CU-LABEL: private_nontemporal_load_1:
 ; GFX11-CU:       ; %bb.0: ; %entry
+; GFX11-CU-NEXT:    s_clause 0x1
 ; GFX11-CU-NEXT:    s_load_b32 s2, s[0:1], 0x0
-; GFX11-CU-NEXT:    v_dual_mov_b32 v1, 0 :: v_dual_lshlrev_b32 v0, 2, v0
 ; GFX11-CU-NEXT:    s_load_b64 s[0:1], s[0:1], 0x8
+; GFX11-CU-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX11-CU-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-CU-NEXT:    scratch_load_b32 v0, v0, s2 slc dlc
+; GFX11-CU-NEXT:    v_lshl_add_u32 v0, v0, 2, s2
+; GFX11-CU-NEXT:    scratch_load_b32 v0, v0, off slc dlc
 ; GFX11-CU-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-CU-NEXT:    global_store_b32 v1, v0, s[0:1]
 ; GFX11-CU-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -657,24 +661,24 @@ define amdgpu_kernel void @private_nontemporal_store_1(
 ; GFX940-NOTTGSPLIT:       ; %bb.0: ; %entry
 ; GFX940-NOTTGSPLIT-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x0
 ; GFX940-NOTTGSPLIT-NEXT:    s_load_dword s4, s[0:1], 0x8
-; GFX940-NOTTGSPLIT-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX940-NOTTGSPLIT-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX940-NOTTGSPLIT-NEXT:    v_lshl_add_u32 v0, v0, 2, s4
 ; GFX940-NOTTGSPLIT-NEXT:    s_load_dword s0, s[2:3], 0x0
 ; GFX940-NOTTGSPLIT-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-NOTTGSPLIT-NEXT:    v_mov_b32_e32 v1, s0
-; GFX940-NOTTGSPLIT-NEXT:    scratch_store_dword v0, v1, s4 nt
+; GFX940-NOTTGSPLIT-NEXT:    scratch_store_dword v0, v1, off nt
 ; GFX940-NOTTGSPLIT-NEXT:    s_endpgm
 ;
 ; GFX940-TGSPLIT-LABEL: private_nontemporal_store_1:
 ; GFX940-TGSPLIT:       ; %bb.0: ; %entry
 ; GFX940-TGSPLIT-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x0
 ; GFX940-TGSPLIT-NEXT:    s_load_dword s4, s[0:1], 0x8
-; GFX940-TGSPLIT-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX940-TGSPLIT-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX940-TGSPLIT-NEXT:    v_lshl_add_u32 v0, v0, 2, s4
 ; GFX940-TGSPLIT-NEXT:    s_load_dword s0, s[2:3], 0x0
 ; GFX940-TGSPLIT-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-TGSPLIT-NEXT:    v_mov_b32_e32 v1, s0
-; GFX940-TGSPLIT-NEXT:    scratch_store_dword v0, v1, s4 nt
+; GFX940-TGSPLIT-NEXT:    scratch_store_dword v0, v1, off nt
 ; GFX940-TGSPLIT-NEXT:    s_endpgm
 ;
 ; GFX11-WGP-LABEL: private_nontemporal_store_1:
@@ -684,9 +688,10 @@ define amdgpu_kernel void @private_nontemporal_store_1(
 ; GFX11-WGP-NEXT:    s_load_b32 s0, s[0:1], 0x8
 ; GFX11-WGP-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-WGP-NEXT:    s_load_b32 s1, s[2:3], 0x0
+; GFX11-WGP-NEXT:    v_lshl_add_u32 v0, v0, 2, s0
 ; GFX11-WGP-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-WGP-NEXT:    v_dual_mov_b32 v1, s1 :: v_dual_lshlrev_b32 v0, 2, v0
-; GFX11-WGP-NEXT:    scratch_store_b32 v0, v1, s0 glc slc dlc
+; GFX11-WGP-NEXT:    v_mov_b32_e32 v1, s1
+; GFX11-WGP-NEXT:    scratch_store_b32 v0, v1, off glc slc dlc
 ; GFX11-WGP-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-WGP-NEXT:    s_endpgm
 ;
@@ -697,9 +702,10 @@ define amdgpu_kernel void @private_nontemporal_store_1(
 ; GFX11-CU-NEXT:    s_load_b32 s0, s[0:1], 0x8
 ; GFX11-CU-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-CU-NEXT:    s_load_b32 s1, s[2:3], 0x0
+; GFX11-CU-NEXT:    v_lshl_add_u32 v0, v0, 2, s0
 ; GFX11-CU-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-CU-NEXT:    v_dual_mov_b32 v1, s1 :: v_dual_lshlrev_b32 v0, 2, v0
-; GFX11-CU-NEXT:    scratch_store_b32 v0, v1, s0 glc slc dlc
+; GFX11-CU-NEXT:    v_mov_b32_e32 v1, s1
+; GFX11-CU-NEXT:    scratch_store_b32 v0, v1, off glc slc dlc
 ; GFX11-CU-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-CU-NEXT:    s_endpgm
     ptr addrspace(1) %in, ptr addrspace(5) %out) {
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll
index fea679a9aed84..424b8ec764f82 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll
@@ -222,11 +222,13 @@ define amdgpu_kernel void @private_volatile_load_1(
 ;
 ; GFX11-WGP-LABEL: private_volatile_load_1:
 ; GFX11-WGP:       ; %bb.0: ; %entry
+; GFX11-WGP-NEXT:    s_clause 0x1
 ; GFX11-WGP-NEXT:    s_load_b32 s2, s[0:1], 0x0
-; GFX11-WGP-NEXT:    v_dual_mov_b32 v1, 0 :: v_dual_lshlrev_b32 v0, 2, v0
 ; GFX11-WGP-NEXT:    s_load_b64 s[0:1], s[0:1], 0x8
+; GFX11-WGP-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX11-WGP-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-WGP-NEXT:    scratch_load_b32 v0, v0, s2 glc dlc
+; GFX11-WGP-NEXT:    v_lshl_add_u32 v0, v0, 2, s2
+; GFX11-WGP-NEXT:    scratch_load_b32 v0, v0, off glc dlc
 ; GFX11-WGP-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-WGP-NEXT:    global_store_b32 v1, v0, s[0:1]
 ; GFX11-WGP-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -234,11 +236,13 @@ define amdgpu_kernel void @private_volatile_load_1(
 ;
 ; GFX11-CU-LABEL: private_volatile_load_1:
 ; GFX11-CU:       ; %bb.0: ; %entry
+; GFX11-CU-NEXT:    s_clause 0x1
 ; GFX11-CU-NEXT:    s_load_b32 s2, s[0:1], 0x0
-; GFX11-CU-NEXT:    v_dual_mov_b32 v1, 0 :: v_dual_lshlrev_b32 v0, 2, v0
 ; GFX11-CU-NEXT:    s_load_b64 s[0:1], s[0:1], 0x8
+; GFX11-CU-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX11-CU-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-CU-NEXT:    scratch_load_b32 v0, v0, s2 glc dlc
+; GFX11-CU-NEXT:    v_lshl_add_u32 v0, v0, 2, s2
+; GFX11-CU-NEXT:    scratch_load_b32 v0, v0, off glc dlc
 ; GFX11-CU-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-CU-NEXT:    global_store_b32 v1, v0, s[0:1]
 ; GFX11-CU-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
@@ -480,9 +484,10 @@ define amdgpu_kernel void @private_volatile_store_1(
 ; GFX11-WGP-NEXT:    s_load_b32 s0, s[0:1], 0x8
 ; GFX11-WGP-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-WGP-NEXT:    s_load_b32 s1, s[2:3], 0x0
+; GFX11-WGP-NEXT:    v_lshl_add_u32 v0, v0, 2, s0
 ; GFX11-WGP-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-WGP-NEXT:    v_dual_mov_b32 v1, s1 :: v_dual_lshlrev_b32 v0, 2, v0
-; GFX11-WGP-NEXT:    scratch_store_b32 v0, v1, s0 dlc
+; GFX11-WGP-NEXT:    v_mov_b32_e32 v1, s1
+; GFX11-WGP-NEXT:    scratch_store_b32 v0, v1, off dlc
 ; GFX11-WGP-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-WGP-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-WGP-NEXT:    s_endpgm
@@ -494,9 +499,10 @@ define amdgpu_kernel void @private_volatile_store_1(
 ; GFX11-CU-NEXT:    s_load_b32 s0, s[0:1], 0x8
 ; GFX11-CU-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-CU-NEXT:    s_load_b32 s1, s[2:3], 0x0
+; GFX11-CU-NEXT:    v_lshl_add_u32 v0, v0, 2, s0
 ; GFX11-CU-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-CU-NEXT:    v_dual_mov_b32 v1, s1 :: v_dual_lshlrev_b32 v0, 2, v0
-; GFX11-CU-NEXT:    scratch_store_b32 v0, v1, s0 dlc
+; GFX11-CU-NEXT:    v_mov_b32_e32 v1, s1
+; GFX11-CU-NEXT:    scratch_store_b32 v0, v1, off dlc
 ; GFX11-CU-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-CU-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-CU-NEXT:    s_endpgm
diff --git a/llvm/test/CodeGen/AMDGPU/memory_clause.ll b/llvm/test/CodeGen/AMDGPU/memory_clause.ll
index 52dd4afcdca5f..6a1c717a471b4 100644
--- a/llvm/test/CodeGen/AMDGPU/memory_clause.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory_clause.ll
@@ -209,20 +209,26 @@ define void @mubuf_clause(ptr addrspace(5) noalias nocapture readonly %arg, ptr
 ; GCN-SCRATCH-NEXT:    v_lshlrev_b32_e32 v2, 4, v31
 ; GCN-SCRATCH-NEXT:    v_and_b32_e32 v18, 0x3ff0, v2
 ; GCN-SCRATCH-NEXT:    v_add_nc_u32_e32 v0, v0, v18
+; GCN-SCRATCH-NEXT:    v_add_nc_u32_e32 v6, 16, v0
+; GCN-SCRATCH-NEXT:    v_add_nc_u32_e32 v10, 32, v0
+; GCN-SCRATCH-NEXT:    v_add_nc_u32_e32 v14, 48, v0
 ; GCN-SCRATCH-NEXT:    s_clause 0x3
 ; GCN-SCRATCH-NEXT:    scratch_load_dwordx4 v[2:5], v0, off
-; GCN-SCRATCH-NEXT:    scratch_load_dwordx4 v[6:9], v0, off offset:16
-; GCN-SCRATCH-NEXT:    scratch_load_dwordx4 v[10:13], v0, off offset:32
-; GCN-SCRATCH-NEXT:    scratch_load_dwordx4 v[14:17], v0, off offset:48
+; GCN-SCRATCH-NEXT:    scratch_load_dwordx4 v[6:9], v6, off
+; GCN-SCRATCH-NEXT:    scratch_load_dwordx4 v[10:13], v10, off
+; GCN-SCRATCH-NEXT:    scratch_load_dwordx4 v[14:17], v14, off
 ; GCN-SCRATCH-NEXT:    v_add_nc_u32_e32 v0, v1, v18
+; GCN-SCRATCH-NEXT:    v_add_nc_u32_e32 v1, 16, v0
+; GCN-SCRATCH-NEXT:    v_add_nc_u32_e32 v18, 32, v0
+; GCN-SCRATCH-NEXT:    v_add_nc_u32_e32 v19, 48, v0
 ; GCN-SCRATCH-NEXT:    s_waitcnt vmcnt(3)
 ; GCN-SCRATCH-NEXT:    scratch_store_dwordx4 v0, v[2:5], off
 ; GCN-SCRATCH-NEXT:    s_waitcnt vmcnt(2)
-; GCN-SCRATCH-NEXT:    scratch_store_dwordx4 v0, v[6:9], off offset:16
+; GCN-SCRATCH-NEXT:    scratch_store_dwordx4 v1, v[6:9], off
 ; GCN-SCRATCH-NEXT:    s_waitcnt vmcnt(1)
-; GCN-SCRATCH-NEXT:    scratch_store_dwordx4 v0, v[10:13], off offset:32
+; GCN-SCRATCH-NEXT:    scratch_store_dwordx4 v18, v[10:13], off
 ; GCN-SCRATCH-NEXT:    s_waitcnt vmcnt(0)
-; GCN-SCRATCH-NEXT:    scratch_store_dwordx4 v0, v[14:17], off offset:48
+; GCN-SCRATCH-NEXT:    scratch_store_dwordx4 v19, v[14:17], off
 ; GCN-SCRATCH-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GCN-SCRATCH-NEXT:    s_setpc_b64 s[30:31]
 bb:
diff --git a/llvm/test/CodeGen/AMDGPU/mul.ll b/llvm/test/CodeGen/AMDGPU/mul.ll
index 85dd59a0c4217..648f44cad56f2 100644
--- a/llvm/test/CodeGen/AMDGPU/mul.ll
+++ b/llvm/test/CodeGen/AMDGPU/mul.ll
@@ -1,20 +1,128 @@
-; RUN:  llc -amdgpu-scalarize-global-loads=false  -march=amdgcn -mcpu=verde -verify-machineinstrs < %s | FileCheck -allow-deprecated-dag-overlap -check-prefixes=GCN,SI,FUNC %s
-; RUN:  llc -amdgpu-scalarize-global-loads=false  -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -allow-deprecated-dag-overlap -check-prefixes=GCN,VI,FUNC %s
-; RUN:  llc -amdgpu-scalarize-global-loads=false  -march=amdgcn -mcpu=gfx900 -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -allow-deprecated-dag-overlap -check-prefixes=FUNC,GFX9PLUS %s
-; RUN:  llc -amdgpu-scalarize-global-loads=false  -march=amdgcn -mcpu=gfx1010 -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -allow-deprecated-dag-overlap -check-prefixes=FUNC,GFX9PLUS %s
-; RUN:  llc -amdgpu-scalarize-global-loads=false  -march=amdgcn -mcpu=gfx1100 -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -allow-deprecated-dag-overlap -check-prefixes=FUNC,GFX9PLUS %s
-; RUN:  llc -amdgpu-scalarize-global-loads=false  -march=r600 -mcpu=redwood < %s | FileCheck -allow-deprecated-dag-overlap -check-prefixes=EG,FUNC %s
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN:  llc -amdgpu-scalarize-global-loads=false  -march=amdgcn -mcpu=verde -verify-machineinstrs < %s | FileCheck -check-prefixes=SI %s
+; RUN:  llc -amdgpu-scalarize-global-loads=false  -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -check-prefixes=VI %s
+; RUN:  llc -amdgpu-scalarize-global-loads=false  -march=amdgcn -mcpu=gfx900 -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX9 %s
+; RUN:  llc -amdgpu-scalarize-global-loads=false  -march=amdgcn -mcpu=gfx1010 -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX10 %s
+; RUN:  llc -amdgpu-scalarize-global-loads=false  -march=amdgcn -mcpu=gfx1100 -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX11 %s
+; RUN:  llc -amdgpu-scalarize-global-loads=false  -march=r600 -mcpu=redwood < %s | FileCheck -check-prefixes=EG %s
 
 ; mul24 and mad24 are affected
 
-; FUNC-LABEL: {{^}}test_mul_v2i32:
-; EG: MULLO_INT {{\*? *}}T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}
-; EG: MULLO_INT {{\*? *}}T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}
-
-; GCN: v_mul_lo_u32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
-; GCN: v_mul_lo_u32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
-
 define amdgpu_kernel void @test_mul_v2i32(ptr addrspace(1) %out, ptr addrspace(1) %in) {
+; SI-LABEL: test_mul_v2i32:
+; SI:       ; %bb.0: ; %entry
+; SI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; SI-NEXT:    s_mov_b32 s7, 0xf000
+; SI-NEXT:    s_mov_b32 s6, -1
+; SI-NEXT:    s_mov_b32 s10, s6
+; SI-NEXT:    s_mov_b32 s11, s7
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    s_mov_b32 s8, s2
+; SI-NEXT:    s_mov_b32 s9, s3
+; SI-NEXT:    buffer_load_dwordx4 v[0:3], off, s[8:11], 0
+; SI-NEXT:    s_mov_b32 s4, s0
+; SI-NEXT:    s_mov_b32 s5, s1
+; SI-NEXT:    s_waitcnt vmcnt(0)
+; SI-NEXT:    v_mul_lo_u32 v1, v1, v3
+; SI-NEXT:    v_mul_lo_u32 v0, v0, v2
+; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; SI-NEXT:    s_endpgm
+;
+; VI-LABEL: test_mul_v2i32:
+; VI:       ; %bb.0: ; %entry
+; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; VI-NEXT:    s_mov_b32 s7, 0xf000
+; VI-NEXT:    s_mov_b32 s6, -1
+; VI-NEXT:    s_mov_b32 s10, s6
+; VI-NEXT:    s_mov_b32 s11, s7
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_mov_b32 s8, s2
+; VI-NEXT:    s_mov_b32 s9, s3
+; VI-NEXT:    buffer_load_dwordx4 v[0:3], off, s[8:11], 0
+; VI-NEXT:    s_mov_b32 s4, s0
+; VI-NEXT:    s_mov_b32 s5, s1
+; VI-NEXT:    s_waitcnt vmcnt(0)
+; VI-NEXT:    v_mul_lo_u32 v1, v1, v3
+; VI-NEXT:    v_mul_lo_u32 v0, v0, v2
+; VI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: test_mul_v2i32:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX9-NEXT:    s_mov_b32 s7, 0xf000
+; GFX9-NEXT:    s_mov_b32 s6, -1
+; GFX9-NEXT:    s_mov_b32 s10, s6
+; GFX9-NEXT:    s_mov_b32 s11, s7
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mov_b32 s8, s2
+; GFX9-NEXT:    s_mov_b32 s9, s3
+; GFX9-NEXT:    buffer_load_dwordx4 v[0:3], off, s[8:11], 0
+; GFX9-NEXT:    s_mov_b32 s4, s0
+; GFX9-NEXT:    s_mov_b32 s5, s1
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_mul_lo_u32 v1, v1, v3
+; GFX9-NEXT:    v_mul_lo_u32 v0, v0, v2
+; GFX9-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: test_mul_v2i32:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX10-NEXT:    s_mov_b32 s6, -1
+; GFX10-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX10-NEXT:    s_mov_b32 s10, s6
+; GFX10-NEXT:    s_mov_b32 s11, s7
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mov_b32 s8, s2
+; GFX10-NEXT:    s_mov_b32 s9, s3
+; GFX10-NEXT:    s_mov_b32 s4, s0
+; GFX10-NEXT:    buffer_load_dwordx4 v[0:3], off, s[8:11], 0
+; GFX10-NEXT:    s_mov_b32 s5, s1
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_mul_lo_u32 v1, v1, v3
+; GFX10-NEXT:    v_mul_lo_u32 v0, v0, v2
+; GFX10-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: test_mul_v2i32:
+; GFX11:       ; %bb.0: ; %entry
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    s_mov_b32 s6, -1
+; GFX11-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX11-NEXT:    s_mov_b32 s10, s6
+; GFX11-NEXT:    s_mov_b32 s11, s7
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_mov_b32 s8, s2
+; GFX11-NEXT:    s_mov_b32 s9, s3
+; GFX11-NEXT:    s_mov_b32 s4, s0
+; GFX11-NEXT:    buffer_load_b128 v[0:3], off, s[8:11], 0
+; GFX11-NEXT:    s_mov_b32 s5, s1
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    v_mul_lo_u32 v1, v1, v3
+; GFX11-NEXT:    v_mul_lo_u32 v0, v0, v2
+; GFX11-NEXT:    buffer_store_b64 v[0:1], off, s[4:7], 0
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; EG-LABEL: test_mul_v2i32:
+; EG:       ; %bb.0: ; %entry
+; EG-NEXT:    ALU 0, @8, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    TEX 0 @6
+; EG-NEXT:    ALU 3, @9, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T0.XY, T1.X, 1
+; EG-NEXT:    CF_END
+; EG-NEXT:    PAD
+; EG-NEXT:    Fetch clause starting at 6:
+; EG-NEXT:     VTX_READ_128 T0.XYZW, T0.X, 0, #1
+; EG-NEXT:    ALU clause starting at 8:
+; EG-NEXT:     MOV * T0.X, KC0[2].Z,
+; EG-NEXT:    ALU clause starting at 9:
+; EG-NEXT:     MULLO_INT * T0.Y, T0.Y, T0.W,
+; EG-NEXT:     LSHR T1.X, KC0[2].Y, literal.x,
+; EG-NEXT:     MULLO_INT * T0.X, T0.X, T0.Z,
+; EG-NEXT:    2(2.802597e-45), 0(0.000000e+00)
+entry:
   %b_ptr = getelementptr <2 x i32>, ptr addrspace(1) %in, i32 1
   %a = load <2 x i32>, ptr addrspace(1) %in
   %b = load <2 x i32>, ptr addrspace(1) %b_ptr
@@ -23,18 +131,141 @@ define amdgpu_kernel void @test_mul_v2i32(ptr addrspace(1) %out, ptr addrspace(1
   ret void
 }
 
-; FUNC-LABEL: {{^}}v_mul_v4i32:
-; EG: MULLO_INT {{\*? *}}T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}
-; EG: MULLO_INT {{\*? *}}T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}
-; EG: MULLO_INT {{\*? *}}T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}
-; EG: MULLO_INT {{\*? *}}T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}
-
-; GCN: v_mul_lo_u32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
-; GCN: v_mul_lo_u32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
-; GCN: v_mul_lo_u32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
-; GCN: v_mul_lo_u32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
-
 define amdgpu_kernel void @v_mul_v4i32(ptr addrspace(1) %out, ptr addrspace(1) %in) {
+; SI-LABEL: v_mul_v4i32:
+; SI:       ; %bb.0: ; %entry
+; SI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; SI-NEXT:    s_mov_b32 s7, 0xf000
+; SI-NEXT:    s_mov_b32 s6, -1
+; SI-NEXT:    s_mov_b32 s10, s6
+; SI-NEXT:    s_mov_b32 s11, s7
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    s_mov_b32 s8, s2
+; SI-NEXT:    s_mov_b32 s9, s3
+; SI-NEXT:    buffer_load_dwordx4 v[0:3], off, s[8:11], 0
+; SI-NEXT:    buffer_load_dwordx4 v[4:7], off, s[8:11], 0 offset:16
+; SI-NEXT:    s_mov_b32 s4, s0
+; SI-NEXT:    s_mov_b32 s5, s1
+; SI-NEXT:    s_waitcnt vmcnt(0)
+; SI-NEXT:    v_mul_lo_u32 v3, v3, v7
+; SI-NEXT:    v_mul_lo_u32 v2, v2, v6
+; SI-NEXT:    v_mul_lo_u32 v1, v1, v5
+; SI-NEXT:    v_mul_lo_u32 v0, v0, v4
+; SI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
+; SI-NEXT:    s_endpgm
+;
+; VI-LABEL: v_mul_v4i32:
+; VI:       ; %bb.0: ; %entry
+; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; VI-NEXT:    s_mov_b32 s7, 0xf000
+; VI-NEXT:    s_mov_b32 s6, -1
+; VI-NEXT:    s_mov_b32 s10, s6
+; VI-NEXT:    s_mov_b32 s11, s7
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_mov_b32 s8, s2
+; VI-NEXT:    s_mov_b32 s9, s3
+; VI-NEXT:    buffer_load_dwordx4 v[0:3], off, s[8:11], 0
+; VI-NEXT:    buffer_load_dwordx4 v[4:7], off, s[8:11], 0 offset:16
+; VI-NEXT:    s_mov_b32 s4, s0
+; VI-NEXT:    s_mov_b32 s5, s1
+; VI-NEXT:    s_waitcnt vmcnt(0)
+; VI-NEXT:    v_mul_lo_u32 v3, v3, v7
+; VI-NEXT:    v_mul_lo_u32 v2, v2, v6
+; VI-NEXT:    v_mul_lo_u32 v1, v1, v5
+; VI-NEXT:    v_mul_lo_u32 v0, v0, v4
+; VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
+; VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_mul_v4i32:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX9-NEXT:    s_mov_b32 s7, 0xf000
+; GFX9-NEXT:    s_mov_b32 s6, -1
+; GFX9-NEXT:    s_mov_b32 s10, s6
+; GFX9-NEXT:    s_mov_b32 s11, s7
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mov_b32 s8, s2
+; GFX9-NEXT:    s_mov_b32 s9, s3
+; GFX9-NEXT:    buffer_load_dwordx4 v[0:3], off, s[8:11], 0
+; GFX9-NEXT:    buffer_load_dwordx4 v[4:7], off, s[8:11], 0 offset:16
+; GFX9-NEXT:    s_mov_b32 s4, s0
+; GFX9-NEXT:    s_mov_b32 s5, s1
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_mul_lo_u32 v3, v3, v7
+; GFX9-NEXT:    v_mul_lo_u32 v2, v2, v6
+; GFX9-NEXT:    v_mul_lo_u32 v1, v1, v5
+; GFX9-NEXT:    v_mul_lo_u32 v0, v0, v4
+; GFX9-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: v_mul_v4i32:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX10-NEXT:    s_mov_b32 s6, -1
+; GFX10-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX10-NEXT:    s_mov_b32 s10, s6
+; GFX10-NEXT:    s_mov_b32 s11, s7
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mov_b32 s8, s2
+; GFX10-NEXT:    s_mov_b32 s9, s3
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    buffer_load_dwordx4 v[0:3], off, s[8:11], 0
+; GFX10-NEXT:    buffer_load_dwordx4 v[4:7], off, s[8:11], 0 offset:16
+; GFX10-NEXT:    s_mov_b32 s4, s0
+; GFX10-NEXT:    s_mov_b32 s5, s1
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_mul_lo_u32 v3, v3, v7
+; GFX10-NEXT:    v_mul_lo_u32 v2, v2, v6
+; GFX10-NEXT:    v_mul_lo_u32 v1, v1, v5
+; GFX10-NEXT:    v_mul_lo_u32 v0, v0, v4
+; GFX10-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: v_mul_v4i32:
+; GFX11:       ; %bb.0: ; %entry
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    s_mov_b32 s6, -1
+; GFX11-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX11-NEXT:    s_mov_b32 s10, s6
+; GFX11-NEXT:    s_mov_b32 s11, s7
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_mov_b32 s8, s2
+; GFX11-NEXT:    s_mov_b32 s9, s3
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    buffer_load_b128 v[0:3], off, s[8:11], 0
+; GFX11-NEXT:    buffer_load_b128 v[4:7], off, s[8:11], 0 offset:16
+; GFX11-NEXT:    s_mov_b32 s4, s0
+; GFX11-NEXT:    s_mov_b32 s5, s1
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    v_mul_lo_u32 v3, v3, v7
+; GFX11-NEXT:    v_mul_lo_u32 v2, v2, v6
+; GFX11-NEXT:    v_mul_lo_u32 v1, v1, v5
+; GFX11-NEXT:    v_mul_lo_u32 v0, v0, v4
+; GFX11-NEXT:    buffer_store_b128 v[0:3], off, s[4:7], 0
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; EG-LABEL: v_mul_v4i32:
+; EG:       ; %bb.0: ; %entry
+; EG-NEXT:    ALU 0, @10, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    TEX 1 @6
+; EG-NEXT:    ALU 5, @11, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T0.XYZW, T2.X, 1
+; EG-NEXT:    CF_END
+; EG-NEXT:    PAD
+; EG-NEXT:    Fetch clause starting at 6:
+; EG-NEXT:     VTX_READ_128 T1.XYZW, T0.X, 16, #1
+; EG-NEXT:     VTX_READ_128 T0.XYZW, T0.X, 0, #1
+; EG-NEXT:    ALU clause starting at 10:
+; EG-NEXT:     MOV * T0.X, KC0[2].Z,
+; EG-NEXT:    ALU clause starting at 11:
+; EG-NEXT:     MULLO_INT * T0.W, T0.W, T1.W,
+; EG-NEXT:     MULLO_INT * T0.Z, T0.Z, T1.Z,
+; EG-NEXT:     MULLO_INT * T0.Y, T0.Y, T1.Y,
+; EG-NEXT:     LSHR T2.X, KC0[2].Y, literal.x,
+; EG-NEXT:     MULLO_INT * T0.X, T0.X, T1.X,
+; EG-NEXT:    2(2.802597e-45), 0(0.000000e+00)
+entry:
   %b_ptr = getelementptr <4 x i32>, ptr addrspace(1) %in, i32 1
   %a = load <4 x i32>, ptr addrspace(1) %in
   %b = load <4 x i32>, ptr addrspace(1) %b_ptr
@@ -43,24 +274,230 @@ define amdgpu_kernel void @v_mul_v4i32(ptr addrspace(1) %out, ptr addrspace(1) %
   ret void
 }
 
-; FUNC-LABEL: {{^}}s_trunc_i64_mul_to_i32:
-; GCN: s_load_dword
-; GCN: s_load_dword
-; GCN: s_mul_i32
-; GCN: buffer_store_dword
 define amdgpu_kernel void @s_trunc_i64_mul_to_i32(ptr addrspace(1) %out, i64 %a, i64 %b) {
+; SI-LABEL: s_trunc_i64_mul_to_i32:
+; SI:       ; %bb.0: ; %entry
+; SI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    s_load_dword s7, s[0:1], 0xd
+; SI-NEXT:    s_mov_b32 s3, 0xf000
+; SI-NEXT:    s_mov_b32 s2, -1
+; SI-NEXT:    s_mov_b32 s0, s4
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    s_mul_i32 s4, s7, s6
+; SI-NEXT:    s_mov_b32 s1, s5
+; SI-NEXT:    v_mov_b32_e32 v0, s4
+; SI-NEXT:    buffer_store_dword v0, off, s[0:3], 0
+; SI-NEXT:    s_endpgm
+;
+; VI-LABEL: s_trunc_i64_mul_to_i32:
+; VI:       ; %bb.0: ; %entry
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_load_dword s7, s[0:1], 0x34
+; VI-NEXT:    s_mov_b32 s3, 0xf000
+; VI-NEXT:    s_mov_b32 s2, -1
+; VI-NEXT:    s_mov_b32 s0, s4
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_mul_i32 s4, s7, s6
+; VI-NEXT:    s_mov_b32 s1, s5
+; VI-NEXT:    v_mov_b32_e32 v0, s4
+; VI-NEXT:    buffer_store_dword v0, off, s[0:3], 0
+; VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: s_trunc_i64_mul_to_i32:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_load_dword s7, s[0:1], 0x34
+; GFX9-NEXT:    ; kill: killed $sgpr0_sgpr1
+; GFX9-NEXT:    s_mov_b32 s3, 0xf000
+; GFX9-NEXT:    s_mov_b32 s2, -1
+; GFX9-NEXT:    s_mov_b32 s0, s4
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mul_i32 s4, s7, s6
+; GFX9-NEXT:    s_mov_b32 s1, s5
+; GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GFX9-NEXT:    buffer_store_dword v0, off, s[0:3], 0
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: s_trunc_i64_mul_to_i32:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dword s2, s[0:1], 0x34
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX10-NEXT:    s_mul_i32 s0, s2, s6
+; GFX10-NEXT:    s_mov_b32 s6, -1
+; GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GFX10-NEXT:    buffer_store_dword v0, off, s[4:7], 0
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_trunc_i64_mul_to_i32:
+; GFX11:       ; %bb.0: ; %entry
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b128 s[4:7], s[0:1], 0x24
+; GFX11-NEXT:    s_load_b32 s0, s[0:1], 0x34
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX11-NEXT:    s_mul_i32 s0, s0, s6
+; GFX11-NEXT:    s_mov_b32 s6, -1
+; GFX11-NEXT:    v_mov_b32_e32 v0, s0
+; GFX11-NEXT:    buffer_store_b32 v0, off, s[4:7], 0
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; EG-LABEL: s_trunc_i64_mul_to_i32:
+; EG:       ; %bb.0: ; %entry
+; EG-NEXT:    ALU 2, @4, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T1.X, T0.X, 1
+; EG-NEXT:    CF_END
+; EG-NEXT:    PAD
+; EG-NEXT:    ALU clause starting at 4:
+; EG-NEXT:     LSHR * T0.X, KC0[2].Y, literal.x,
+; EG-NEXT:    2(2.802597e-45), 0(0.000000e+00)
+; EG-NEXT:     MULLO_INT * T1.X, KC0[3].Y, KC0[2].W,
+entry:
   %mul = mul i64 %b, %a
   %trunc = trunc i64 %mul to i32
   store i32 %trunc, ptr addrspace(1) %out, align 8
   ret void
 }
 
-; FUNC-LABEL: {{^}}v_trunc_i64_mul_to_i32:
-; GCN: s_load_dword
-; GCN: s_load_dword
-; GCN: v_mul_lo_u32
-; GCN: buffer_store_dword
 define amdgpu_kernel void @v_trunc_i64_mul_to_i32(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) nounwind {
+; SI-LABEL: v_trunc_i64_mul_to_i32:
+; SI:       ; %bb.0: ; %entry
+; SI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; SI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0xd
+; SI-NEXT:    s_mov_b32 s3, 0xf000
+; SI-NEXT:    s_mov_b32 s2, -1
+; SI-NEXT:    s_mov_b32 s14, s2
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    s_mov_b32 s12, s6
+; SI-NEXT:    s_mov_b32 s13, s7
+; SI-NEXT:    s_mov_b32 s15, s3
+; SI-NEXT:    s_mov_b32 s10, s2
+; SI-NEXT:    s_mov_b32 s11, s3
+; SI-NEXT:    buffer_load_dword v0, off, s[12:15], 0
+; SI-NEXT:    buffer_load_dword v1, off, s[8:11], 0
+; SI-NEXT:    s_mov_b32 s0, s4
+; SI-NEXT:    s_mov_b32 s1, s5
+; SI-NEXT:    s_waitcnt vmcnt(0)
+; SI-NEXT:    v_mul_lo_u32 v0, v1, v0
+; SI-NEXT:    buffer_store_dword v0, off, s[0:3], 0
+; SI-NEXT:    s_endpgm
+;
+; VI-LABEL: v_trunc_i64_mul_to_i32:
+; VI:       ; %bb.0: ; %entry
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; VI-NEXT:    s_mov_b32 s3, 0xf000
+; VI-NEXT:    s_mov_b32 s2, -1
+; VI-NEXT:    s_mov_b32 s14, s2
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_mov_b32 s12, s6
+; VI-NEXT:    s_mov_b32 s13, s7
+; VI-NEXT:    s_mov_b32 s15, s3
+; VI-NEXT:    s_mov_b32 s10, s2
+; VI-NEXT:    s_mov_b32 s11, s3
+; VI-NEXT:    buffer_load_dword v0, off, s[12:15], 0
+; VI-NEXT:    buffer_load_dword v1, off, s[8:11], 0
+; VI-NEXT:    s_mov_b32 s0, s4
+; VI-NEXT:    s_mov_b32 s1, s5
+; VI-NEXT:    s_waitcnt vmcnt(0)
+; VI-NEXT:    v_mul_lo_u32 v0, v1, v0
+; VI-NEXT:    buffer_store_dword v0, off, s[0:3], 0
+; VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_trunc_i64_mul_to_i32:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; GFX9-NEXT:    s_mov_b32 s3, 0xf000
+; GFX9-NEXT:    s_mov_b32 s2, -1
+; GFX9-NEXT:    s_mov_b32 s14, s2
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mov_b32 s12, s6
+; GFX9-NEXT:    s_mov_b32 s13, s7
+; GFX9-NEXT:    s_mov_b32 s15, s3
+; GFX9-NEXT:    s_mov_b32 s10, s2
+; GFX9-NEXT:    s_mov_b32 s11, s3
+; GFX9-NEXT:    buffer_load_dword v0, off, s[12:15], 0
+; GFX9-NEXT:    buffer_load_dword v1, off, s[8:11], 0
+; GFX9-NEXT:    s_mov_b32 s0, s4
+; GFX9-NEXT:    s_mov_b32 s1, s5
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_mul_lo_u32 v0, v1, v0
+; GFX9-NEXT:    buffer_store_dword v0, off, s[0:3], 0
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: v_trunc_i64_mul_to_i32:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; GFX10-NEXT:    s_mov_b32 s2, -1
+; GFX10-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX10-NEXT:    s_mov_b32 s14, s2
+; GFX10-NEXT:    s_mov_b32 s15, s3
+; GFX10-NEXT:    s_mov_b32 s10, s2
+; GFX10-NEXT:    s_mov_b32 s11, s3
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mov_b32 s12, s6
+; GFX10-NEXT:    s_mov_b32 s13, s7
+; GFX10-NEXT:    buffer_load_dword v0, off, s[12:15], 0
+; GFX10-NEXT:    buffer_load_dword v1, off, s[8:11], 0
+; GFX10-NEXT:    s_mov_b32 s0, s4
+; GFX10-NEXT:    s_mov_b32 s1, s5
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_mul_lo_u32 v0, v1, v0
+; GFX10-NEXT:    buffer_store_dword v0, off, s[0:3], 0
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: v_trunc_i64_mul_to_i32:
+; GFX11:       ; %bb.0: ; %entry
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b128 s[4:7], s[0:1], 0x24
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x34
+; GFX11-NEXT:    s_mov_b32 s10, -1
+; GFX11-NEXT:    s_mov_b32 s11, 0x31016000
+; GFX11-NEXT:    s_mov_b32 s14, s10
+; GFX11-NEXT:    s_mov_b32 s15, s11
+; GFX11-NEXT:    s_mov_b32 s2, s10
+; GFX11-NEXT:    s_mov_b32 s3, s11
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_mov_b32 s12, s6
+; GFX11-NEXT:    s_mov_b32 s13, s7
+; GFX11-NEXT:    buffer_load_b32 v0, off, s[12:15], 0
+; GFX11-NEXT:    buffer_load_b32 v1, off, s[0:3], 0
+; GFX11-NEXT:    s_mov_b32 s8, s4
+; GFX11-NEXT:    s_mov_b32 s9, s5
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    v_mul_lo_u32 v0, v1, v0
+; GFX11-NEXT:    buffer_store_b32 v0, off, s[8:11], 0
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; EG-LABEL: v_trunc_i64_mul_to_i32:
+; EG:       ; %bb.0: ; %entry
+; EG-NEXT:    ALU 1, @10, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    TEX 1 @6
+; EG-NEXT:    ALU 2, @12, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T0.X, T2.X, 1
+; EG-NEXT:    CF_END
+; EG-NEXT:    PAD
+; EG-NEXT:    Fetch clause starting at 6:
+; EG-NEXT:     VTX_READ_32 T1.X, T1.X, 0, #1
+; EG-NEXT:     VTX_READ_32 T0.X, T0.X, 0, #1
+; EG-NEXT:    ALU clause starting at 10:
+; EG-NEXT:     MOV T0.X, KC0[2].Z,
+; EG-NEXT:     MOV * T1.X, KC0[2].W,
+; EG-NEXT:    ALU clause starting at 12:
+; EG-NEXT:     LSHR T2.X, KC0[2].Y, literal.x,
+; EG-NEXT:     MULLO_INT * T0.X, T1.X, T0.X,
+; EG-NEXT:    2(2.802597e-45), 0(0.000000e+00)
+entry:
   %a = load i64, ptr addrspace(1) %aptr, align 8
   %b = load i64, ptr addrspace(1) %bptr, align 8
   %mul = mul i64 %b, %a
@@ -71,13 +508,92 @@ define amdgpu_kernel void @v_trunc_i64_mul_to_i32(ptr addrspace(1) %out, ptr add
 
 ; This 64-bit multiply should just use MUL_HI and MUL_LO, since the top
 ; 32-bits of both arguments are sign bits.
-; FUNC-LABEL: {{^}}mul64_sext_c:
-; EG-DAG: MULLO_INT
-; EG-DAG: MULHI_INT
-; SI-DAG: s_mulk_i32
-; SI-DAG: v_mul_hi_i32
-; VI: v_mad_i64_i32
+
 define amdgpu_kernel void @mul64_sext_c(ptr addrspace(1) %out, i32 %in) {
+; SI-LABEL: mul64_sext_c:
+; SI:       ; %bb.0: ; %entry
+; SI-NEXT:    s_load_dword s4, s[0:1], 0xb
+; SI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; SI-NEXT:    v_mov_b32_e32 v0, 0x50
+; SI-NEXT:    s_mov_b32 s3, 0xf000
+; SI-NEXT:    s_mov_b32 s2, -1
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    v_mul_hi_i32 v1, s4, v0
+; SI-NEXT:    s_mulk_i32 s4, 0x50
+; SI-NEXT:    v_mov_b32_e32 v0, s4
+; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; SI-NEXT:    s_endpgm
+;
+; VI-LABEL: mul64_sext_c:
+; VI:       ; %bb.0: ; %entry
+; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; VI-NEXT:    v_mov_b32_e32 v0, 0x50
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_mad_i64_i32 v[0:1], s[2:3], s2, v0, 0
+; VI-NEXT:    s_mov_b32 s3, 0xf000
+; VI-NEXT:    s_mov_b32 s2, -1
+; VI-NEXT:    s_nop 2
+; VI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: mul64_sext_c:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
+; GFX9-NEXT:    s_mov_b32 s7, 0xf000
+; GFX9-NEXT:    s_mov_b32 s6, -1
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mul_hi_i32 s0, s2, 0x50
+; GFX9-NEXT:    s_mulk_i32 s2, 0x50
+; GFX9-NEXT:    v_mov_b32_e32 v0, s2
+; GFX9-NEXT:    v_mov_b32_e32 v1, s0
+; GFX9-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: mul64_sext_c:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GFX10-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
+; GFX10-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX10-NEXT:    s_mov_b32 s6, -1
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mul_i32 s0, s2, 0x50
+; GFX10-NEXT:    s_mul_hi_i32 s1, s2, 0x50
+; GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GFX10-NEXT:    v_mov_b32_e32 v1, s1
+; GFX10-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: mul64_sext_c:
+; GFX11:       ; %bb.0: ; %entry
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_mul_i32 s3, s2, 0x50
+; GFX11-NEXT:    s_mul_hi_i32 s2, s2, 0x50
+; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
+; GFX11-NEXT:    v_dual_mov_b32 v0, s3 :: v_dual_mov_b32 v1, s2
+; GFX11-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX11-NEXT:    s_mov_b32 s2, -1
+; GFX11-NEXT:    buffer_store_b64 v[0:1], off, s[0:3], 0
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; EG-LABEL: mul64_sext_c:
+; EG:       ; %bb.0: ; %entry
+; EG-NEXT:    ALU 4, @4, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T0.XY, T1.X, 1
+; EG-NEXT:    CF_END
+; EG-NEXT:    PAD
+; EG-NEXT:    ALU clause starting at 4:
+; EG-NEXT:     MULHI_INT * T0.Y, KC0[2].Z, literal.x,
+; EG-NEXT:    80(1.121039e-43), 0(0.000000e+00)
+; EG-NEXT:     LSHR T1.X, KC0[2].Y, literal.x,
+; EG-NEXT:     MULLO_INT * T0.X, KC0[2].Z, literal.y,
+; EG-NEXT:    2(2.802597e-45), 80(1.121039e-43)
 entry:
   %0 = sext i32 %in to i64
   %1 = mul i64 %0, 80
@@ -85,14 +601,124 @@ entry:
   ret void
 }
 
-; FUNC-LABEL: {{^}}v_mul64_sext_c:
-; EG-DAG: MULLO_INT
-; EG-DAG: MULHI_INT
-; SI-DAG: v_mul_lo_u32
-; SI-DAG: v_mul_hi_i32
-; VI: v_mad_i64_i32
-; GCN: s_endpgm
 define amdgpu_kernel void @v_mul64_sext_c(ptr addrspace(1) %out, ptr addrspace(1) %in) {
+; SI-LABEL: v_mul64_sext_c:
+; SI:       ; %bb.0: ; %entry
+; SI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; SI-NEXT:    s_mov_b32 s7, 0xf000
+; SI-NEXT:    s_mov_b32 s6, -1
+; SI-NEXT:    s_mov_b32 s10, s6
+; SI-NEXT:    s_mov_b32 s11, s7
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    s_mov_b32 s8, s2
+; SI-NEXT:    s_mov_b32 s9, s3
+; SI-NEXT:    buffer_load_dword v0, off, s[8:11], 0
+; SI-NEXT:    s_movk_i32 s2, 0x50
+; SI-NEXT:    s_mov_b32 s4, s0
+; SI-NEXT:    s_mov_b32 s5, s1
+; SI-NEXT:    s_waitcnt vmcnt(0)
+; SI-NEXT:    v_mul_hi_i32 v1, v0, s2
+; SI-NEXT:    v_mul_lo_u32 v0, v0, s2
+; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; SI-NEXT:    s_endpgm
+;
+; VI-LABEL: v_mul64_sext_c:
+; VI:       ; %bb.0: ; %entry
+; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; VI-NEXT:    s_mov_b32 s7, 0xf000
+; VI-NEXT:    s_mov_b32 s6, -1
+; VI-NEXT:    s_mov_b32 s10, s6
+; VI-NEXT:    s_mov_b32 s11, s7
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_mov_b32 s8, s2
+; VI-NEXT:    s_mov_b32 s9, s3
+; VI-NEXT:    buffer_load_dword v0, off, s[8:11], 0
+; VI-NEXT:    s_movk_i32 s2, 0x50
+; VI-NEXT:    s_mov_b32 s4, s0
+; VI-NEXT:    s_mov_b32 s5, s1
+; VI-NEXT:    s_waitcnt vmcnt(0)
+; VI-NEXT:    v_mad_i64_i32 v[0:1], s[2:3], v0, s2, 0
+; VI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_mul64_sext_c:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX9-NEXT:    s_mov_b32 s7, 0xf000
+; GFX9-NEXT:    s_mov_b32 s6, -1
+; GFX9-NEXT:    s_mov_b32 s10, s6
+; GFX9-NEXT:    s_mov_b32 s11, s7
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mov_b32 s8, s2
+; GFX9-NEXT:    s_mov_b32 s9, s3
+; GFX9-NEXT:    buffer_load_dword v0, off, s[8:11], 0
+; GFX9-NEXT:    s_movk_i32 s2, 0x50
+; GFX9-NEXT:    s_mov_b32 s4, s0
+; GFX9-NEXT:    s_mov_b32 s5, s1
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_mul_hi_i32 v1, v0, s2
+; GFX9-NEXT:    v_mul_lo_u32 v0, v0, s2
+; GFX9-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: v_mul64_sext_c:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX10-NEXT:    s_mov_b32 s6, -1
+; GFX10-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX10-NEXT:    s_mov_b32 s10, s6
+; GFX10-NEXT:    s_mov_b32 s11, s7
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mov_b32 s8, s2
+; GFX10-NEXT:    s_mov_b32 s9, s3
+; GFX10-NEXT:    s_mov_b32 s4, s0
+; GFX10-NEXT:    buffer_load_dword v0, off, s[8:11], 0
+; GFX10-NEXT:    s_mov_b32 s5, s1
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_mul_hi_i32 v1, 0x50, v0
+; GFX10-NEXT:    v_mul_lo_u32 v0, 0x50, v0
+; GFX10-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: v_mul64_sext_c:
+; GFX11:       ; %bb.0: ; %entry
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    s_mov_b32 s6, -1
+; GFX11-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX11-NEXT:    s_mov_b32 s10, s6
+; GFX11-NEXT:    s_mov_b32 s11, s7
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_mov_b32 s8, s2
+; GFX11-NEXT:    s_mov_b32 s9, s3
+; GFX11-NEXT:    s_mov_b32 s4, s0
+; GFX11-NEXT:    buffer_load_b32 v0, off, s[8:11], 0
+; GFX11-NEXT:    s_mov_b32 s5, s1
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    v_mul_hi_i32 v1, 0x50, v0
+; GFX11-NEXT:    v_mul_lo_u32 v0, 0x50, v0
+; GFX11-NEXT:    buffer_store_b64 v[0:1], off, s[4:7], 0
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; EG-LABEL: v_mul64_sext_c:
+; EG:       ; %bb.0: ; %entry
+; EG-NEXT:    ALU 0, @8, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    TEX 0 @6
+; EG-NEXT:    ALU 4, @9, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T0.XY, T1.X, 1
+; EG-NEXT:    CF_END
+; EG-NEXT:    PAD
+; EG-NEXT:    Fetch clause starting at 6:
+; EG-NEXT:     VTX_READ_32 T0.X, T0.X, 0, #1
+; EG-NEXT:    ALU clause starting at 8:
+; EG-NEXT:     MOV * T0.X, KC0[2].Z,
+; EG-NEXT:    ALU clause starting at 9:
+; EG-NEXT:     MULHI_INT * T0.Y, T0.X, literal.x,
+; EG-NEXT:    80(1.121039e-43), 0(0.000000e+00)
+; EG-NEXT:     LSHR T1.X, KC0[2].Y, literal.x,
+; EG-NEXT:     MULLO_INT * T0.X, T0.X, literal.y,
+; EG-NEXT:    2(2.802597e-45), 80(1.121039e-43)
+entry:
   %val = load i32, ptr addrspace(1) %in, align 4
   %ext = sext i32 %val to i64
   %mul = mul i64 %ext, 80
@@ -100,12 +726,121 @@ define amdgpu_kernel void @v_mul64_sext_c(ptr addrspace(1) %out, ptr addrspace(1
   ret void
 }
 
-; FUNC-LABEL: {{^}}v_mul64_sext_inline_imm:
-; SI-DAG: v_mul_lo_u32 v{{[0-9]+}}, v{{[0-9]+}}, 9
-; SI-DAG: v_mul_hi_i32 v{{[0-9]+}}, v{{[0-9]+}}, 9
-; VI: v_mad_i64_i32 v[{{[0-9]+}}:{{[0-9]+}}], s[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}, 9, 0
-; GCN: s_endpgm
 define amdgpu_kernel void @v_mul64_sext_inline_imm(ptr addrspace(1) %out, ptr addrspace(1) %in) {
+; SI-LABEL: v_mul64_sext_inline_imm:
+; SI:       ; %bb.0: ; %entry
+; SI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; SI-NEXT:    s_mov_b32 s7, 0xf000
+; SI-NEXT:    s_mov_b32 s6, -1
+; SI-NEXT:    s_mov_b32 s10, s6
+; SI-NEXT:    s_mov_b32 s11, s7
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    s_mov_b32 s8, s2
+; SI-NEXT:    s_mov_b32 s9, s3
+; SI-NEXT:    buffer_load_dword v0, off, s[8:11], 0
+; SI-NEXT:    s_mov_b32 s4, s0
+; SI-NEXT:    s_mov_b32 s5, s1
+; SI-NEXT:    s_waitcnt vmcnt(0)
+; SI-NEXT:    v_mul_hi_i32 v1, v0, 9
+; SI-NEXT:    v_mul_lo_u32 v0, v0, 9
+; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; SI-NEXT:    s_endpgm
+;
+; VI-LABEL: v_mul64_sext_inline_imm:
+; VI:       ; %bb.0: ; %entry
+; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; VI-NEXT:    s_mov_b32 s7, 0xf000
+; VI-NEXT:    s_mov_b32 s6, -1
+; VI-NEXT:    s_mov_b32 s10, s6
+; VI-NEXT:    s_mov_b32 s11, s7
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_mov_b32 s8, s2
+; VI-NEXT:    s_mov_b32 s9, s3
+; VI-NEXT:    buffer_load_dword v0, off, s[8:11], 0
+; VI-NEXT:    s_mov_b32 s4, s0
+; VI-NEXT:    s_mov_b32 s5, s1
+; VI-NEXT:    s_waitcnt vmcnt(0)
+; VI-NEXT:    v_mad_i64_i32 v[0:1], s[2:3], v0, 9, 0
+; VI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_mul64_sext_inline_imm:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX9-NEXT:    s_mov_b32 s7, 0xf000
+; GFX9-NEXT:    s_mov_b32 s6, -1
+; GFX9-NEXT:    s_mov_b32 s10, s6
+; GFX9-NEXT:    s_mov_b32 s11, s7
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mov_b32 s8, s2
+; GFX9-NEXT:    s_mov_b32 s9, s3
+; GFX9-NEXT:    buffer_load_dword v0, off, s[8:11], 0
+; GFX9-NEXT:    s_mov_b32 s4, s0
+; GFX9-NEXT:    s_mov_b32 s5, s1
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_mul_hi_i32 v1, v0, 9
+; GFX9-NEXT:    v_mul_lo_u32 v0, v0, 9
+; GFX9-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: v_mul64_sext_inline_imm:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX10-NEXT:    s_mov_b32 s6, -1
+; GFX10-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX10-NEXT:    s_mov_b32 s10, s6
+; GFX10-NEXT:    s_mov_b32 s11, s7
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mov_b32 s8, s2
+; GFX10-NEXT:    s_mov_b32 s9, s3
+; GFX10-NEXT:    s_mov_b32 s4, s0
+; GFX10-NEXT:    buffer_load_dword v0, off, s[8:11], 0
+; GFX10-NEXT:    s_mov_b32 s5, s1
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_mul_hi_i32 v1, v0, 9
+; GFX10-NEXT:    v_mul_lo_u32 v0, v0, 9
+; GFX10-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: v_mul64_sext_inline_imm:
+; GFX11:       ; %bb.0: ; %entry
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    s_mov_b32 s6, -1
+; GFX11-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX11-NEXT:    s_mov_b32 s10, s6
+; GFX11-NEXT:    s_mov_b32 s11, s7
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_mov_b32 s8, s2
+; GFX11-NEXT:    s_mov_b32 s9, s3
+; GFX11-NEXT:    s_mov_b32 s4, s0
+; GFX11-NEXT:    buffer_load_b32 v0, off, s[8:11], 0
+; GFX11-NEXT:    s_mov_b32 s5, s1
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    v_mul_hi_i32 v1, v0, 9
+; GFX11-NEXT:    v_mul_lo_u32 v0, v0, 9
+; GFX11-NEXT:    buffer_store_b64 v[0:1], off, s[4:7], 0
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; EG-LABEL: v_mul64_sext_inline_imm:
+; EG:       ; %bb.0: ; %entry
+; EG-NEXT:    ALU 0, @8, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    TEX 0 @6
+; EG-NEXT:    ALU 4, @9, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T0.XY, T1.X, 1
+; EG-NEXT:    CF_END
+; EG-NEXT:    PAD
+; EG-NEXT:    Fetch clause starting at 6:
+; EG-NEXT:     VTX_READ_32 T0.X, T0.X, 0, #1
+; EG-NEXT:    ALU clause starting at 8:
+; EG-NEXT:     MOV * T0.X, KC0[2].Z,
+; EG-NEXT:    ALU clause starting at 9:
+; EG-NEXT:     MULHI_INT * T0.Y, T0.X, literal.x,
+; EG-NEXT:    9(1.261169e-44), 0(0.000000e+00)
+; EG-NEXT:     LSHR T1.X, KC0[2].Y, literal.x,
+; EG-NEXT:     MULLO_INT * T0.X, T0.X, literal.y,
+; EG-NEXT:    2(2.802597e-45), 9(1.261169e-44)
+entry:
   %val = load i32, ptr addrspace(1) %in, align 4
   %ext = sext i32 %val to i64
   %mul = mul i64 %ext, 9
@@ -113,22 +848,200 @@ define amdgpu_kernel void @v_mul64_sext_inline_imm(ptr addrspace(1) %out, ptr ad
   ret void
 }
 
-; FUNC-LABEL: {{^}}s_mul_i32:
-; GCN: s_load_dword [[SRC0:s[0-9]+]],
-; GCN: s_load_dword [[SRC1:s[0-9]+]],
-; GCN: s_mul_i32 [[SRESULT:s[0-9]+]], [[SRC0]], [[SRC1]]
-; GCN: v_mov_b32_e32 [[VRESULT:v[0-9]+]], [[SRESULT]]
-; GCN: buffer_store_dword [[VRESULT]],
-; GCN: s_endpgm
 define amdgpu_kernel void @s_mul_i32(ptr addrspace(1) %out, [8 x i32], i32 %a, [8 x i32], i32 %b) nounwind {
+; SI-LABEL: s_mul_i32:
+; SI:       ; %bb.0: ; %entry
+; SI-NEXT:    s_load_dword s4, s[0:1], 0x13
+; SI-NEXT:    s_load_dword s5, s[0:1], 0x1c
+; SI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; SI-NEXT:    s_mov_b32 s3, 0xf000
+; SI-NEXT:    s_mov_b32 s2, -1
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    s_mul_i32 s4, s4, s5
+; SI-NEXT:    v_mov_b32_e32 v0, s4
+; SI-NEXT:    buffer_store_dword v0, off, s[0:3], 0
+; SI-NEXT:    s_endpgm
+;
+; VI-LABEL: s_mul_i32:
+; VI:       ; %bb.0: ; %entry
+; VI-NEXT:    s_load_dword s4, s[0:1], 0x4c
+; VI-NEXT:    s_load_dword s5, s[0:1], 0x70
+; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; VI-NEXT:    s_mov_b32 s3, 0xf000
+; VI-NEXT:    s_mov_b32 s2, -1
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_mul_i32 s4, s4, s5
+; VI-NEXT:    v_mov_b32_e32 v0, s4
+; VI-NEXT:    buffer_store_dword v0, off, s[0:3], 0
+; VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: s_mul_i32:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dword s2, s[0:1], 0x4c
+; GFX9-NEXT:    s_load_dword s3, s[0:1], 0x70
+; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
+; GFX9-NEXT:    s_mov_b32 s7, 0xf000
+; GFX9-NEXT:    s_mov_b32 s6, -1
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mul_i32 s0, s2, s3
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    buffer_store_dword v0, off, s[4:7], 0
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: s_mul_i32:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x2
+; GFX10-NEXT:    s_load_dword s2, s[0:1], 0x4c
+; GFX10-NEXT:    s_load_dword s3, s[0:1], 0x70
+; GFX10-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
+; GFX10-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX10-NEXT:    s_mov_b32 s6, -1
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mul_i32 s0, s2, s3
+; GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GFX10-NEXT:    buffer_store_dword v0, off, s[4:7], 0
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_mul_i32:
+; GFX11:       ; %bb.0: ; %entry
+; GFX11-NEXT:    s_clause 0x2
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x4c
+; GFX11-NEXT:    s_load_b32 s3, s[0:1], 0x70
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_mul_i32 s2, s2, s3
+; GFX11-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    s_mov_b32 s2, -1
+; GFX11-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; EG-LABEL: s_mul_i32:
+; EG:       ; %bb.0: ; %entry
+; EG-NEXT:    ALU 2, @4, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T1.X, T0.X, 1
+; EG-NEXT:    CF_END
+; EG-NEXT:    PAD
+; EG-NEXT:    ALU clause starting at 4:
+; EG-NEXT:     LSHR * T0.X, KC0[2].Y, literal.x,
+; EG-NEXT:    2(2.802597e-45), 0(0.000000e+00)
+; EG-NEXT:     MULLO_INT * T1.X, KC0[4].Z, KC0[6].W,
+entry:
   %mul = mul i32 %a, %b
   store i32 %mul, ptr addrspace(1) %out, align 4
   ret void
 }
 
-; FUNC-LABEL: {{^}}v_mul_i32:
-; GCN: v_mul_lo_u32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
 define amdgpu_kernel void @v_mul_i32(ptr addrspace(1) %out, ptr addrspace(1) %in) {
+; SI-LABEL: v_mul_i32:
+; SI:       ; %bb.0: ; %entry
+; SI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; SI-NEXT:    s_mov_b32 s7, 0xf000
+; SI-NEXT:    s_mov_b32 s6, -1
+; SI-NEXT:    s_mov_b32 s10, s6
+; SI-NEXT:    s_mov_b32 s11, s7
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    s_mov_b32 s8, s2
+; SI-NEXT:    s_mov_b32 s9, s3
+; SI-NEXT:    buffer_load_dwordx2 v[0:1], off, s[8:11], 0
+; SI-NEXT:    s_mov_b32 s4, s0
+; SI-NEXT:    s_mov_b32 s5, s1
+; SI-NEXT:    s_waitcnt vmcnt(0)
+; SI-NEXT:    v_mul_lo_u32 v0, v0, v1
+; SI-NEXT:    buffer_store_dword v0, off, s[4:7], 0
+; SI-NEXT:    s_endpgm
+;
+; VI-LABEL: v_mul_i32:
+; VI:       ; %bb.0: ; %entry
+; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; VI-NEXT:    s_mov_b32 s7, 0xf000
+; VI-NEXT:    s_mov_b32 s6, -1
+; VI-NEXT:    s_mov_b32 s10, s6
+; VI-NEXT:    s_mov_b32 s11, s7
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_mov_b32 s8, s2
+; VI-NEXT:    s_mov_b32 s9, s3
+; VI-NEXT:    buffer_load_dwordx2 v[0:1], off, s[8:11], 0
+; VI-NEXT:    s_mov_b32 s4, s0
+; VI-NEXT:    s_mov_b32 s5, s1
+; VI-NEXT:    s_waitcnt vmcnt(0)
+; VI-NEXT:    v_mul_lo_u32 v0, v0, v1
+; VI-NEXT:    buffer_store_dword v0, off, s[4:7], 0
+; VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_mul_i32:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX9-NEXT:    s_mov_b32 s7, 0xf000
+; GFX9-NEXT:    s_mov_b32 s6, -1
+; GFX9-NEXT:    s_mov_b32 s10, s6
+; GFX9-NEXT:    s_mov_b32 s11, s7
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mov_b32 s8, s2
+; GFX9-NEXT:    s_mov_b32 s9, s3
+; GFX9-NEXT:    buffer_load_dwordx2 v[0:1], off, s[8:11], 0
+; GFX9-NEXT:    s_mov_b32 s4, s0
+; GFX9-NEXT:    s_mov_b32 s5, s1
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_mul_lo_u32 v0, v0, v1
+; GFX9-NEXT:    buffer_store_dword v0, off, s[4:7], 0
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: v_mul_i32:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX10-NEXT:    s_mov_b32 s6, -1
+; GFX10-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX10-NEXT:    s_mov_b32 s10, s6
+; GFX10-NEXT:    s_mov_b32 s11, s7
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mov_b32 s8, s2
+; GFX10-NEXT:    s_mov_b32 s9, s3
+; GFX10-NEXT:    s_mov_b32 s4, s0
+; GFX10-NEXT:    buffer_load_dwordx2 v[0:1], off, s[8:11], 0
+; GFX10-NEXT:    s_mov_b32 s5, s1
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_mul_lo_u32 v0, v0, v1
+; GFX10-NEXT:    buffer_store_dword v0, off, s[4:7], 0
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: v_mul_i32:
+; GFX11:       ; %bb.0: ; %entry
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    s_mov_b32 s6, -1
+; GFX11-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX11-NEXT:    s_mov_b32 s10, s6
+; GFX11-NEXT:    s_mov_b32 s11, s7
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_mov_b32 s8, s2
+; GFX11-NEXT:    s_mov_b32 s9, s3
+; GFX11-NEXT:    s_mov_b32 s4, s0
+; GFX11-NEXT:    buffer_load_b64 v[0:1], off, s[8:11], 0
+; GFX11-NEXT:    s_mov_b32 s5, s1
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    v_mul_lo_u32 v0, v0, v1
+; GFX11-NEXT:    buffer_store_b32 v0, off, s[4:7], 0
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; EG-LABEL: v_mul_i32:
+; EG:       ; %bb.0: ; %entry
+; EG-NEXT:    ALU 0, @8, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    TEX 0 @6
+; EG-NEXT:    ALU 2, @9, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T0.X, T1.X, 1
+; EG-NEXT:    CF_END
+; EG-NEXT:    PAD
+; EG-NEXT:    Fetch clause starting at 6:
+; EG-NEXT:     VTX_READ_64 T0.XY, T0.X, 0, #1
+; EG-NEXT:    ALU clause starting at 8:
+; EG-NEXT:     MOV * T0.X, KC0[2].Z,
+; EG-NEXT:    ALU clause starting at 9:
+; EG-NEXT:     LSHR T1.X, KC0[2].Y, literal.x,
+; EG-NEXT:     MULLO_INT * T0.X, T0.X, T0.Y,
+; EG-NEXT:    2(2.802597e-45), 0(0.000000e+00)
+entry:
   %b_ptr = getelementptr i32, ptr addrspace(1) %in, i32 1
   %a = load i32, ptr addrspace(1) %in
   %b = load i32, ptr addrspace(1) %b_ptr
@@ -144,21 +1057,292 @@ define amdgpu_kernel void @v_mul_i32(ptr addrspace(1) %out, ptr addrspace(1) %in
 ; so this test just uses FUNC-LABEL to make sure the compiler does not
 ; crash with a 'failed to select' error.
 
-; FUNC-LABEL: {{^}}s_mul_i64:
-; GFX9PLUS-DAG: s_mul_i32
-; GFX9PLUS-DAG: s_mul_hi_u32
-; GFX9PLUS-DAG: s_mul_i32
-; GFX9PLUS-DAG: s_mul_i32
-; GFX9PLUS: s_endpgm
 define amdgpu_kernel void @s_mul_i64(ptr addrspace(1) %out, i64 %a, i64 %b) nounwind {
+; SI-LABEL: s_mul_i64:
+; SI:       ; %bb.0: ; %entry
+; SI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; SI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0xd
+; SI-NEXT:    s_mov_b32 s3, 0xf000
+; SI-NEXT:    s_mov_b32 s2, -1
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    s_mov_b32 s0, s4
+; SI-NEXT:    v_mov_b32_e32 v0, s8
+; SI-NEXT:    v_mul_hi_u32 v0, s6, v0
+; SI-NEXT:    s_mul_i32 s4, s6, s9
+; SI-NEXT:    s_mov_b32 s1, s5
+; SI-NEXT:    v_add_i32_e32 v0, vcc, s4, v0
+; SI-NEXT:    s_mul_i32 s4, s7, s8
+; SI-NEXT:    v_add_i32_e32 v1, vcc, s4, v0
+; SI-NEXT:    s_mul_i32 s4, s6, s8
+; SI-NEXT:    v_mov_b32_e32 v0, s4
+; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; SI-NEXT:    s_endpgm
+;
+; VI-LABEL: s_mul_i64:
+; VI:       ; %bb.0: ; %entry
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; VI-NEXT:    s_mov_b32 s3, 0xf000
+; VI-NEXT:    s_mov_b32 s2, -1
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_mov_b32 s0, s4
+; VI-NEXT:    v_mov_b32_e32 v0, s8
+; VI-NEXT:    v_mad_u64_u32 v[0:1], s[10:11], s6, v0, 0
+; VI-NEXT:    s_mul_i32 s4, s6, s9
+; VI-NEXT:    s_mov_b32 s1, s5
+; VI-NEXT:    v_add_u32_e32 v1, vcc, s4, v1
+; VI-NEXT:    s_mul_i32 s4, s7, s8
+; VI-NEXT:    v_add_u32_e32 v1, vcc, s4, v1
+; VI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: s_mul_i64:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; GFX9-NEXT:    s_mov_b32 s3, 0xf000
+; GFX9-NEXT:    s_mov_b32 s2, -1
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mov_b32 s0, s4
+; GFX9-NEXT:    s_mov_b32 s1, s5
+; GFX9-NEXT:    s_mul_i32 s4, s6, s9
+; GFX9-NEXT:    s_mul_hi_u32 s5, s6, s8
+; GFX9-NEXT:    s_add_i32 s4, s5, s4
+; GFX9-NEXT:    s_mul_i32 s5, s7, s8
+; GFX9-NEXT:    s_add_i32 s4, s4, s5
+; GFX9-NEXT:    s_mul_i32 s5, s6, s8
+; GFX9-NEXT:    v_mov_b32_e32 v0, s5
+; GFX9-NEXT:    v_mov_b32_e32 v1, s4
+; GFX9-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: s_mul_i64:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mul_i32 s0, s6, s3
+; GFX10-NEXT:    s_mul_hi_u32 s1, s6, s2
+; GFX10-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX10-NEXT:    s_add_i32 s0, s1, s0
+; GFX10-NEXT:    s_mul_i32 s1, s7, s2
+; GFX10-NEXT:    s_mul_i32 s2, s6, s2
+; GFX10-NEXT:    s_add_i32 s0, s0, s1
+; GFX10-NEXT:    v_mov_b32_e32 v0, s2
+; GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; GFX10-NEXT:    s_mov_b32 s2, -1
+; GFX10-NEXT:    s_mov_b32 s0, s4
+; GFX10-NEXT:    s_mov_b32 s1, s5
+; GFX10-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_mul_i64:
+; GFX11:       ; %bb.0: ; %entry
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b128 s[4:7], s[0:1], 0x24
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x34
+; GFX11-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_mul_i32 s1, s6, s1
+; GFX11-NEXT:    s_mul_hi_u32 s2, s6, s0
+; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(SALU_CYCLE_1)
+; GFX11-NEXT:    s_add_i32 s1, s2, s1
+; GFX11-NEXT:    s_mul_i32 s2, s7, s0
+; GFX11-NEXT:    s_mul_i32 s0, s6, s0
+; GFX11-NEXT:    s_add_i32 s1, s1, s2
+; GFX11-NEXT:    v_dual_mov_b32 v0, s0 :: v_dual_mov_b32 v1, s1
+; GFX11-NEXT:    s_mov_b32 s2, -1
+; GFX11-NEXT:    s_mov_b32 s0, s4
+; GFX11-NEXT:    s_mov_b32 s1, s5
+; GFX11-NEXT:    buffer_store_b64 v[0:1], off, s[0:3], 0
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; EG-LABEL: s_mul_i64:
+; EG:       ; %bb.0: ; %entry
+; EG-NEXT:    ALU 7, @4, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T0.XY, T1.X, 1
+; EG-NEXT:    CF_END
+; EG-NEXT:    PAD
+; EG-NEXT:    ALU clause starting at 4:
+; EG-NEXT:     MULHI * T0.X, KC0[2].W, KC0[3].Y,
+; EG-NEXT:     MULLO_INT * T0.Y, KC0[2].W, KC0[3].Z,
+; EG-NEXT:     ADD_INT T0.W, T0.X, PS,
+; EG-NEXT:     MULLO_INT * T0.X, KC0[3].X, KC0[3].Y,
+; EG-NEXT:     ADD_INT * T0.Y, PV.W, PS,
+; EG-NEXT:     LSHR * T1.X, KC0[2].Y, literal.x,
+; EG-NEXT:    2(2.802597e-45), 0(0.000000e+00)
+; EG-NEXT:     MULLO_INT * T0.X, KC0[2].W, KC0[3].Y,
+entry:
   %mul = mul i64 %a, %b
   store i64 %mul, ptr addrspace(1) %out, align 8
   ret void
 }
 
-; FUNC-LABEL: {{^}}v_mul_i64:
-; GCN: v_mul_lo_u32
 define amdgpu_kernel void @v_mul_i64(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
+; SI-LABEL: v_mul_i64:
+; SI:       ; %bb.0: ; %entry
+; SI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; SI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0xd
+; SI-NEXT:    s_mov_b32 s3, 0xf000
+; SI-NEXT:    s_mov_b32 s2, -1
+; SI-NEXT:    s_mov_b32 s10, s2
+; SI-NEXT:    s_mov_b32 s11, s3
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    s_mov_b32 s12, s6
+; SI-NEXT:    s_mov_b32 s13, s7
+; SI-NEXT:    s_mov_b32 s14, s2
+; SI-NEXT:    s_mov_b32 s15, s3
+; SI-NEXT:    buffer_load_dwordx2 v[0:1], off, s[8:11], 0
+; SI-NEXT:    buffer_load_dwordx2 v[2:3], off, s[12:15], 0
+; SI-NEXT:    s_mov_b32 s0, s4
+; SI-NEXT:    s_mov_b32 s1, s5
+; SI-NEXT:    s_waitcnt vmcnt(0)
+; SI-NEXT:    v_mul_lo_u32 v1, v2, v1
+; SI-NEXT:    v_mul_hi_u32 v4, v2, v0
+; SI-NEXT:    v_mul_lo_u32 v3, v3, v0
+; SI-NEXT:    v_mul_lo_u32 v0, v2, v0
+; SI-NEXT:    v_add_i32_e32 v1, vcc, v4, v1
+; SI-NEXT:    v_add_i32_e32 v1, vcc, v3, v1
+; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; SI-NEXT:    s_endpgm
+;
+; VI-LABEL: v_mul_i64:
+; VI:       ; %bb.0: ; %entry
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; VI-NEXT:    s_mov_b32 s3, 0xf000
+; VI-NEXT:    s_mov_b32 s2, -1
+; VI-NEXT:    s_mov_b32 s10, s2
+; VI-NEXT:    s_mov_b32 s11, s3
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_mov_b32 s12, s6
+; VI-NEXT:    s_mov_b32 s13, s7
+; VI-NEXT:    s_mov_b32 s14, s2
+; VI-NEXT:    s_mov_b32 s15, s3
+; VI-NEXT:    buffer_load_dwordx2 v[0:1], off, s[8:11], 0
+; VI-NEXT:    buffer_load_dwordx2 v[2:3], off, s[12:15], 0
+; VI-NEXT:    s_mov_b32 s0, s4
+; VI-NEXT:    s_mov_b32 s1, s5
+; VI-NEXT:    s_waitcnt vmcnt(0)
+; VI-NEXT:    v_mul_lo_u32 v4, v2, v1
+; VI-NEXT:    v_mad_u64_u32 v[1:2], s[6:7], v2, v0, 0
+; VI-NEXT:    v_mul_lo_u32 v0, v3, v0
+; VI-NEXT:    v_add_u32_e32 v2, vcc, v4, v2
+; VI-NEXT:    v_add_u32_e32 v2, vcc, v0, v2
+; VI-NEXT:    buffer_store_dwordx2 v[1:2], off, s[0:3], 0
+; VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_mul_i64:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; GFX9-NEXT:    s_mov_b32 s3, 0xf000
+; GFX9-NEXT:    s_mov_b32 s2, -1
+; GFX9-NEXT:    s_mov_b32 s10, s2
+; GFX9-NEXT:    s_mov_b32 s11, s3
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mov_b32 s12, s6
+; GFX9-NEXT:    s_mov_b32 s13, s7
+; GFX9-NEXT:    s_mov_b32 s14, s2
+; GFX9-NEXT:    s_mov_b32 s15, s3
+; GFX9-NEXT:    buffer_load_dwordx2 v[0:1], off, s[8:11], 0
+; GFX9-NEXT:    buffer_load_dwordx2 v[2:3], off, s[12:15], 0
+; GFX9-NEXT:    s_mov_b32 s0, s4
+; GFX9-NEXT:    s_mov_b32 s1, s5
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_mul_lo_u32 v1, v2, v1
+; GFX9-NEXT:    v_mul_hi_u32 v4, v2, v0
+; GFX9-NEXT:    v_mul_lo_u32 v3, v3, v0
+; GFX9-NEXT:    v_mul_lo_u32 v0, v2, v0
+; GFX9-NEXT:    v_add_u32_e32 v1, v4, v1
+; GFX9-NEXT:    v_add_u32_e32 v1, v1, v3
+; GFX9-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: v_mul_i64:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; GFX10-NEXT:    s_mov_b32 s2, -1
+; GFX10-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX10-NEXT:    s_mov_b32 s10, s2
+; GFX10-NEXT:    s_mov_b32 s11, s3
+; GFX10-NEXT:    s_mov_b32 s14, s2
+; GFX10-NEXT:    s_mov_b32 s15, s3
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mov_b32 s12, s6
+; GFX10-NEXT:    s_mov_b32 s13, s7
+; GFX10-NEXT:    buffer_load_dwordx2 v[0:1], off, s[8:11], 0
+; GFX10-NEXT:    buffer_load_dwordx2 v[2:3], off, s[12:15], 0
+; GFX10-NEXT:    s_mov_b32 s0, s4
+; GFX10-NEXT:    s_mov_b32 s1, s5
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_mul_lo_u32 v1, v2, v1
+; GFX10-NEXT:    v_mul_hi_u32 v4, v2, v0
+; GFX10-NEXT:    v_mul_lo_u32 v3, v3, v0
+; GFX10-NEXT:    v_mul_lo_u32 v0, v2, v0
+; GFX10-NEXT:    v_add_nc_u32_e32 v1, v4, v1
+; GFX10-NEXT:    v_add_nc_u32_e32 v1, v1, v3
+; GFX10-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: v_mul_i64:
+; GFX11:       ; %bb.0: ; %entry
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b128 s[4:7], s[0:1], 0x24
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x34
+; GFX11-NEXT:    s_mov_b32 s10, -1
+; GFX11-NEXT:    s_mov_b32 s11, 0x31016000
+; GFX11-NEXT:    s_mov_b32 s2, s10
+; GFX11-NEXT:    s_mov_b32 s3, s11
+; GFX11-NEXT:    s_mov_b32 s14, s10
+; GFX11-NEXT:    s_mov_b32 s15, s11
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_mov_b32 s12, s6
+; GFX11-NEXT:    s_mov_b32 s13, s7
+; GFX11-NEXT:    buffer_load_b64 v[0:1], off, s[0:3], 0
+; GFX11-NEXT:    buffer_load_b64 v[2:3], off, s[12:15], 0
+; GFX11-NEXT:    s_mov_b32 s8, s4
+; GFX11-NEXT:    s_mov_b32 s9, s5
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    v_mul_lo_u32 v1, v2, v1
+; GFX11-NEXT:    v_mul_hi_u32 v4, v2, v0
+; GFX11-NEXT:    v_mul_lo_u32 v3, v3, v0
+; GFX11-NEXT:    v_mul_lo_u32 v0, v2, v0
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_3) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX11-NEXT:    v_add_nc_u32_e32 v1, v4, v1
+; GFX11-NEXT:    v_add_nc_u32_e32 v1, v1, v3
+; GFX11-NEXT:    buffer_store_b64 v[0:1], off, s[8:11], 0
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; EG-LABEL: v_mul_i64:
+; EG:       ; %bb.0: ; %entry
+; EG-NEXT:    ALU 1, @10, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    TEX 1 @6
+; EG-NEXT:    ALU 7, @12, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T0.XY, T2.X, 1
+; EG-NEXT:    CF_END
+; EG-NEXT:    PAD
+; EG-NEXT:    Fetch clause starting at 6:
+; EG-NEXT:     VTX_READ_64 T1.XY, T1.X, 0, #1
+; EG-NEXT:     VTX_READ_64 T0.XY, T0.X, 0, #1
+; EG-NEXT:    ALU clause starting at 10:
+; EG-NEXT:     MOV T0.X, KC0[2].Z,
+; EG-NEXT:     MOV * T1.X, KC0[2].W,
+; EG-NEXT:    ALU clause starting at 12:
+; EG-NEXT:     MULHI * T0.Z, T0.X, T1.X,
+; EG-NEXT:     MULLO_INT * T0.W, T0.X, T1.Y,
+; EG-NEXT:     ADD_INT T0.W, T0.Z, PS,
+; EG-NEXT:     MULLO_INT * T0.Y, T0.Y, T1.X,
+; EG-NEXT:     ADD_INT * T0.Y, PV.W, PS,
+; EG-NEXT:     LSHR T2.X, KC0[2].Y, literal.x,
+; EG-NEXT:     MULLO_INT * T0.X, T0.X, T1.X,
+; EG-NEXT:    2(2.802597e-45), 0(0.000000e+00)
+entry:
   %a = load i64, ptr addrspace(1) %aptr, align 8
   %b = load i64, ptr addrspace(1) %bptr, align 8
   %mul = mul i64 %a, %b
@@ -166,9 +1350,219 @@ define amdgpu_kernel void @v_mul_i64(ptr addrspace(1) %out, ptr addrspace(1) %ap
   ret void
 }
 
-; FUNC-LABEL: {{^}}mul32_in_branch:
-; GCN: s_mul_i32
 define amdgpu_kernel void @mul32_in_branch(ptr addrspace(1) %out, ptr addrspace(1) %in, i32 %a, i32 %b, i32 %c) {
+; SI-LABEL: mul32_in_branch:
+; SI:       ; %bb.0: ; %entry
+; SI-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0xd
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    s_cmp_lg_u32 s2, 0
+; SI-NEXT:    s_cbranch_scc0 .LBB11_2
+; SI-NEXT:  ; %bb.1: ; %else
+; SI-NEXT:    s_mul_i32 s6, s2, s3
+; SI-NEXT:    s_mov_b64 s[4:5], 0
+; SI-NEXT:    s_branch .LBB11_3
+; SI-NEXT:  .LBB11_2:
+; SI-NEXT:    s_mov_b64 s[4:5], -1
+; SI-NEXT:    ; implicit-def: $sgpr6
+; SI-NEXT:  .LBB11_3: ; %Flow
+; SI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; SI-NEXT:    s_andn2_b64 vcc, exec, s[4:5]
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    s_mov_b64 vcc, vcc
+; SI-NEXT:    s_cbranch_vccnz .LBB11_5
+; SI-NEXT:  ; %bb.4: ; %if
+; SI-NEXT:    s_mov_b32 s7, 0xf000
+; SI-NEXT:    s_mov_b32 s6, -1
+; SI-NEXT:    s_mov_b32 s4, s2
+; SI-NEXT:    s_mov_b32 s5, s3
+; SI-NEXT:    buffer_load_dword v0, off, s[4:7], 0
+; SI-NEXT:    s_branch .LBB11_6
+; SI-NEXT:  .LBB11_5:
+; SI-NEXT:    v_mov_b32_e32 v0, s6
+; SI-NEXT:  .LBB11_6: ; %endif
+; SI-NEXT:    s_mov_b32 s3, 0xf000
+; SI-NEXT:    s_mov_b32 s2, -1
+; SI-NEXT:    s_waitcnt vmcnt(0)
+; SI-NEXT:    buffer_store_dword v0, off, s[0:3], 0
+; SI-NEXT:    s_endpgm
+;
+; VI-LABEL: mul32_in_branch:
+; VI:       ; %bb.0: ; %entry
+; VI-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_cmp_lg_u32 s2, 0
+; VI-NEXT:    s_cbranch_scc0 .LBB11_2
+; VI-NEXT:  ; %bb.1: ; %else
+; VI-NEXT:    s_mul_i32 s6, s2, s3
+; VI-NEXT:    s_mov_b64 s[4:5], 0
+; VI-NEXT:    s_branch .LBB11_3
+; VI-NEXT:  .LBB11_2:
+; VI-NEXT:    s_mov_b64 s[4:5], -1
+; VI-NEXT:    ; implicit-def: $sgpr6
+; VI-NEXT:  .LBB11_3: ; %Flow
+; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; VI-NEXT:    s_andn2_b64 vcc, exec, s[4:5]
+; VI-NEXT:    s_cbranch_vccnz .LBB11_5
+; VI-NEXT:  ; %bb.4: ; %if
+; VI-NEXT:    s_mov_b32 s7, 0xf000
+; VI-NEXT:    s_mov_b32 s6, -1
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_mov_b32 s4, s2
+; VI-NEXT:    s_mov_b32 s5, s3
+; VI-NEXT:    buffer_load_dword v0, off, s[4:7], 0
+; VI-NEXT:    s_branch .LBB11_6
+; VI-NEXT:  .LBB11_5:
+; VI-NEXT:    v_mov_b32_e32 v0, s6
+; VI-NEXT:  .LBB11_6: ; %endif
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_mov_b32 s3, 0xf000
+; VI-NEXT:    s_mov_b32 s2, -1
+; VI-NEXT:    s_waitcnt vmcnt(0)
+; VI-NEXT:    buffer_store_dword v0, off, s[0:3], 0
+; VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: mul32_in_branch:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_cmp_lg_u32 s2, 0
+; GFX9-NEXT:    s_cbranch_scc0 .LBB11_2
+; GFX9-NEXT:  ; %bb.1: ; %else
+; GFX9-NEXT:    s_mul_i32 s6, s2, s3
+; GFX9-NEXT:    s_mov_b64 s[4:5], 0
+; GFX9-NEXT:    s_branch .LBB11_3
+; GFX9-NEXT:  .LBB11_2:
+; GFX9-NEXT:    s_mov_b64 s[4:5], -1
+; GFX9-NEXT:    ; implicit-def: $sgpr6
+; GFX9-NEXT:  .LBB11_3: ; %Flow
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX9-NEXT:    s_andn2_b64 vcc, exec, s[4:5]
+; GFX9-NEXT:    s_cbranch_vccnz .LBB11_5
+; GFX9-NEXT:  ; %bb.4: ; %if
+; GFX9-NEXT:    s_mov_b32 s7, 0xf000
+; GFX9-NEXT:    s_mov_b32 s6, -1
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mov_b32 s4, s2
+; GFX9-NEXT:    s_mov_b32 s5, s3
+; GFX9-NEXT:    buffer_load_dword v0, off, s[4:7], 0
+; GFX9-NEXT:    s_branch .LBB11_6
+; GFX9-NEXT:  .LBB11_5:
+; GFX9-NEXT:    v_mov_b32_e32 v0, s6
+; GFX9-NEXT:  .LBB11_6: ; %endif
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mov_b32 s3, 0xf000
+; GFX9-NEXT:    s_mov_b32 s2, -1
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    buffer_store_dword v0, off, s[0:3], 0
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: mul32_in_branch:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX10-NEXT:    s_mov_b32 s4, 0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_cmp_lg_u32 s2, 0
+; GFX10-NEXT:    s_cbranch_scc0 .LBB11_2
+; GFX10-NEXT:  ; %bb.1: ; %else
+; GFX10-NEXT:    s_mul_i32 s5, s2, s3
+; GFX10-NEXT:    s_branch .LBB11_3
+; GFX10-NEXT:  .LBB11_2:
+; GFX10-NEXT:    s_mov_b32 s4, -1
+; GFX10-NEXT:    ; implicit-def: $sgpr5
+; GFX10-NEXT:  .LBB11_3: ; %Flow
+; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX10-NEXT:    s_andn2_b32 vcc_lo, exec_lo, s4
+; GFX10-NEXT:    s_cbranch_vccnz .LBB11_5
+; GFX10-NEXT:  ; %bb.4: ; %if
+; GFX10-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX10-NEXT:    s_mov_b32 s6, -1
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mov_b32 s4, s2
+; GFX10-NEXT:    s_mov_b32 s5, s3
+; GFX10-NEXT:    buffer_load_dword v0, off, s[4:7], 0
+; GFX10-NEXT:    s_branch .LBB11_6
+; GFX10-NEXT:  .LBB11_5:
+; GFX10-NEXT:    v_mov_b32_e32 v0, s5
+; GFX10-NEXT:  .LBB11_6: ; %endif
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX10-NEXT:    s_mov_b32 s2, -1
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    buffer_store_dword v0, off, s[0:3], 0
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: mul32_in_branch:
+; GFX11:       ; %bb.0: ; %entry
+; GFX11-NEXT:    s_load_b64 s[2:3], s[0:1], 0x34
+; GFX11-NEXT:    s_mov_b32 s4, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_cmp_lg_u32 s2, 0
+; GFX11-NEXT:    s_cbranch_scc0 .LBB11_2
+; GFX11-NEXT:  ; %bb.1: ; %else
+; GFX11-NEXT:    s_mul_i32 s5, s2, s3
+; GFX11-NEXT:    s_branch .LBB11_3
+; GFX11-NEXT:  .LBB11_2:
+; GFX11-NEXT:    s_mov_b32 s4, -1
+; GFX11-NEXT:    ; implicit-def: $sgpr5
+; GFX11-NEXT:  .LBB11_3: ; %Flow
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    s_and_not1_b32 vcc_lo, exec_lo, s4
+; GFX11-NEXT:    s_cbranch_vccnz .LBB11_5
+; GFX11-NEXT:  ; %bb.4: ; %if
+; GFX11-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX11-NEXT:    s_mov_b32 s6, -1
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_mov_b32 s4, s2
+; GFX11-NEXT:    s_mov_b32 s5, s3
+; GFX11-NEXT:    buffer_load_b32 v0, off, s[4:7], 0
+; GFX11-NEXT:    s_branch .LBB11_6
+; GFX11-NEXT:  .LBB11_5:
+; GFX11-NEXT:    v_mov_b32_e32 v0, s5
+; GFX11-NEXT:  .LBB11_6: ; %endif
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX11-NEXT:    s_mov_b32 s2, -1
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; EG-LABEL: mul32_in_branch:
+; EG:       ; %bb.0: ; %entry
+; EG-NEXT:    ALU_PUSH_BEFORE 3, @14, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    JUMP @3 POP:1
+; EG-NEXT:    ALU_POP_AFTER 4, @18, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    ALU_PUSH_BEFORE 2, @23, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    JUMP @8 POP:1
+; EG-NEXT:    ALU 0, @26, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    TEX 0 @12
+; EG-NEXT:    POP @8 POP:1
+; EG-NEXT:    ALU 1, @27, KC0[], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T0.X, T1.X, 1
+; EG-NEXT:    CF_END
+; EG-NEXT:    PAD
+; EG-NEXT:    Fetch clause starting at 12:
+; EG-NEXT:     VTX_READ_32 T0.X, T0.X, 0, #1
+; EG-NEXT:    ALU clause starting at 14:
+; EG-NEXT:     MOV T1.W, literal.x,
+; EG-NEXT:     SETNE_INT * T0.W, KC0[2].W, 0.0,
+; EG-NEXT:    1(1.401298e-45), 0(0.000000e+00)
+; EG-NEXT:     PRED_SETNE_INT * ExecMask,PredicateBit (MASKED), PS, 0.0,
+; EG-NEXT:    ALU clause starting at 18:
+; EG-NEXT:     MOV T0.W, KC0[2].W,
+; EG-NEXT:     MOV * T2.W, KC0[3].X,
+; EG-NEXT:     MOV T1.W, literal.x,
+; EG-NEXT:     MULLO_INT * T0.X, PV.W, PS,
+; EG-NEXT:    0(0.000000e+00), 0(0.000000e+00)
+; EG-NEXT:    ALU clause starting at 23:
+; EG-NEXT:     MOV T0.W, KC0[2].Y,
+; EG-NEXT:     SETE_INT * T1.W, T1.W, 0.0,
+; EG-NEXT:     PRED_SETE_INT * ExecMask,PredicateBit (MASKED), PS, 0.0,
+; EG-NEXT:    ALU clause starting at 26:
+; EG-NEXT:     MOV * T0.X, KC0[2].Z,
+; EG-NEXT:    ALU clause starting at 27:
+; EG-NEXT:     LSHR * T1.X, T0.W, literal.x,
+; EG-NEXT:    2(2.802597e-45), 0(0.000000e+00)
 entry:
   %0 = icmp eq i32 %a, 0
   br i1 %0, label %if, label %else
@@ -187,12 +1581,226 @@ endif:
   ret void
 }
 
-; FUNC-LABEL: {{^}}mul64_in_branch:
-; SI-DAG: s_mul_i32
-; SI-DAG: v_mul_hi_u32
-; VI: v_mad_u64_u32
-; GCN: s_endpgm
 define amdgpu_kernel void @mul64_in_branch(ptr addrspace(1) %out, ptr addrspace(1) %in, i64 %a, i64 %b, i64 %c) {
+; SI-LABEL: mul64_in_branch:
+; SI:       ; %bb.0: ; %entry
+; SI-NEXT:    s_load_dwordx8 s[0:7], s[0:1], 0x9
+; SI-NEXT:    s_mov_b64 s[8:9], 0
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    v_cmp_ne_u64_e64 s[10:11], s[4:5], 0
+; SI-NEXT:    s_and_b64 vcc, exec, s[10:11]
+; SI-NEXT:    s_cbranch_vccz .LBB12_4
+; SI-NEXT:  ; %bb.1: ; %else
+; SI-NEXT:    v_mov_b32_e32 v0, s6
+; SI-NEXT:    v_mul_hi_u32 v0, s4, v0
+; SI-NEXT:    s_mul_i32 s7, s4, s7
+; SI-NEXT:    s_mul_i32 s5, s5, s6
+; SI-NEXT:    s_mul_i32 s4, s4, s6
+; SI-NEXT:    v_add_i32_e32 v0, vcc, s7, v0
+; SI-NEXT:    v_add_i32_e32 v1, vcc, s5, v0
+; SI-NEXT:    v_mov_b32_e32 v0, s4
+; SI-NEXT:    s_andn2_b64 vcc, exec, s[8:9]
+; SI-NEXT:    s_cbranch_vccnz .LBB12_3
+; SI-NEXT:  .LBB12_2: ; %if
+; SI-NEXT:    s_mov_b32 s7, 0xf000
+; SI-NEXT:    s_mov_b32 s6, -1
+; SI-NEXT:    s_mov_b32 s4, s2
+; SI-NEXT:    s_mov_b32 s5, s3
+; SI-NEXT:    buffer_load_dwordx2 v[0:1], off, s[4:7], 0
+; SI-NEXT:  .LBB12_3: ; %endif
+; SI-NEXT:    s_mov_b32 s3, 0xf000
+; SI-NEXT:    s_mov_b32 s2, -1
+; SI-NEXT:    s_waitcnt vmcnt(0)
+; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; SI-NEXT:    s_endpgm
+; SI-NEXT:  .LBB12_4:
+; SI-NEXT:    ; implicit-def: $vgpr0_vgpr1
+; SI-NEXT:    s_branch .LBB12_2
+;
+; VI-LABEL: mul64_in_branch:
+; VI:       ; %bb.0: ; %entry
+; VI-NEXT:    s_load_dwordx8 s[0:7], s[0:1], 0x24
+; VI-NEXT:    s_mov_b64 s[8:9], 0
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_cmp_lg_u64 s[4:5], 0
+; VI-NEXT:    s_cbranch_scc0 .LBB12_4
+; VI-NEXT:  ; %bb.1: ; %else
+; VI-NEXT:    v_mov_b32_e32 v0, s6
+; VI-NEXT:    v_mad_u64_u32 v[0:1], s[10:11], s4, v0, 0
+; VI-NEXT:    s_mul_i32 s4, s4, s7
+; VI-NEXT:    v_add_u32_e32 v1, vcc, s4, v1
+; VI-NEXT:    s_mul_i32 s4, s5, s6
+; VI-NEXT:    v_add_u32_e32 v1, vcc, s4, v1
+; VI-NEXT:    s_andn2_b64 vcc, exec, s[8:9]
+; VI-NEXT:    s_cbranch_vccnz .LBB12_3
+; VI-NEXT:  .LBB12_2: ; %if
+; VI-NEXT:    s_mov_b32 s7, 0xf000
+; VI-NEXT:    s_mov_b32 s6, -1
+; VI-NEXT:    s_mov_b32 s4, s2
+; VI-NEXT:    s_mov_b32 s5, s3
+; VI-NEXT:    buffer_load_dwordx2 v[0:1], off, s[4:7], 0
+; VI-NEXT:  .LBB12_3: ; %endif
+; VI-NEXT:    s_mov_b32 s3, 0xf000
+; VI-NEXT:    s_mov_b32 s2, -1
+; VI-NEXT:    s_waitcnt vmcnt(0)
+; VI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; VI-NEXT:    s_endpgm
+; VI-NEXT:  .LBB12_4:
+; VI-NEXT:    ; implicit-def: $vgpr0_vgpr1
+; VI-NEXT:    s_branch .LBB12_2
+;
+; GFX9-LABEL: mul64_in_branch:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx8 s[0:7], s[0:1], 0x24
+; GFX9-NEXT:    s_mov_b64 s[8:9], 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_cmp_lg_u64 s[4:5], 0
+; GFX9-NEXT:    s_cbranch_scc0 .LBB12_3
+; GFX9-NEXT:  ; %bb.1: ; %else
+; GFX9-NEXT:    s_mul_i32 s7, s4, s7
+; GFX9-NEXT:    s_mul_hi_u32 s10, s4, s6
+; GFX9-NEXT:    s_add_i32 s7, s10, s7
+; GFX9-NEXT:    s_mul_i32 s5, s5, s6
+; GFX9-NEXT:    s_add_i32 s5, s7, s5
+; GFX9-NEXT:    s_mul_i32 s4, s4, s6
+; GFX9-NEXT:    s_andn2_b64 vcc, exec, s[8:9]
+; GFX9-NEXT:    s_cbranch_vccnz .LBB12_4
+; GFX9-NEXT:  .LBB12_2: ; %if
+; GFX9-NEXT:    s_mov_b32 s7, 0xf000
+; GFX9-NEXT:    s_mov_b32 s6, -1
+; GFX9-NEXT:    s_mov_b32 s4, s2
+; GFX9-NEXT:    s_mov_b32 s5, s3
+; GFX9-NEXT:    buffer_load_dwordx2 v[0:1], off, s[4:7], 0
+; GFX9-NEXT:    s_branch .LBB12_5
+; GFX9-NEXT:  .LBB12_3:
+; GFX9-NEXT:    ; implicit-def: $sgpr4_sgpr5
+; GFX9-NEXT:    s_branch .LBB12_2
+; GFX9-NEXT:  .LBB12_4:
+; GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GFX9-NEXT:    v_mov_b32_e32 v1, s5
+; GFX9-NEXT:  .LBB12_5: ; %endif
+; GFX9-NEXT:    s_mov_b32 s3, 0xf000
+; GFX9-NEXT:    s_mov_b32 s2, -1
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: mul64_in_branch:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_load_dwordx8 s[0:7], s[0:1], 0x24
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_cmp_lg_u64 s[4:5], 0
+; GFX10-NEXT:    s_cbranch_scc0 .LBB12_3
+; GFX10-NEXT:  ; %bb.1: ; %else
+; GFX10-NEXT:    s_mul_i32 s7, s4, s7
+; GFX10-NEXT:    s_mul_hi_u32 s8, s4, s6
+; GFX10-NEXT:    s_mul_i32 s5, s5, s6
+; GFX10-NEXT:    s_add_i32 s7, s8, s7
+; GFX10-NEXT:    s_mul_i32 s4, s4, s6
+; GFX10-NEXT:    s_add_i32 s5, s7, s5
+; GFX10-NEXT:    s_mov_b32 s6, 0
+; GFX10-NEXT:    s_cbranch_execnz .LBB12_4
+; GFX10-NEXT:  .LBB12_2: ; %if
+; GFX10-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX10-NEXT:    s_mov_b32 s6, -1
+; GFX10-NEXT:    s_mov_b32 s4, s2
+; GFX10-NEXT:    s_mov_b32 s5, s3
+; GFX10-NEXT:    buffer_load_dwordx2 v[0:1], off, s[4:7], 0
+; GFX10-NEXT:    s_branch .LBB12_5
+; GFX10-NEXT:  .LBB12_3:
+; GFX10-NEXT:    s_mov_b32 s6, -1
+; GFX10-NEXT:    ; implicit-def: $sgpr4_sgpr5
+; GFX10-NEXT:    s_branch .LBB12_2
+; GFX10-NEXT:  .LBB12_4:
+; GFX10-NEXT:    v_mov_b32_e32 v0, s4
+; GFX10-NEXT:    v_mov_b32_e32 v1, s5
+; GFX10-NEXT:  .LBB12_5: ; %endif
+; GFX10-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX10-NEXT:    s_mov_b32 s2, -1
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: mul64_in_branch:
+; GFX11:       ; %bb.0: ; %entry
+; GFX11-NEXT:    s_load_b256 s[0:7], s[0:1], 0x24
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_cmp_lg_u64 s[4:5], 0
+; GFX11-NEXT:    s_cbranch_scc0 .LBB12_3
+; GFX11-NEXT:  ; %bb.1: ; %else
+; GFX11-NEXT:    s_mul_i32 s7, s4, s7
+; GFX11-NEXT:    s_mul_hi_u32 s8, s4, s6
+; GFX11-NEXT:    s_mul_i32 s5, s5, s6
+; GFX11-NEXT:    s_add_i32 s7, s8, s7
+; GFX11-NEXT:    s_mul_i32 s4, s4, s6
+; GFX11-NEXT:    s_add_i32 s5, s7, s5
+; GFX11-NEXT:    s_mov_b32 s6, 0
+; GFX11-NEXT:    s_cbranch_execnz .LBB12_4
+; GFX11-NEXT:  .LBB12_2: ; %if
+; GFX11-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX11-NEXT:    s_mov_b32 s6, -1
+; GFX11-NEXT:    s_mov_b32 s4, s2
+; GFX11-NEXT:    s_mov_b32 s5, s3
+; GFX11-NEXT:    buffer_load_b64 v[0:1], off, s[4:7], 0
+; GFX11-NEXT:    s_branch .LBB12_5
+; GFX11-NEXT:  .LBB12_3:
+; GFX11-NEXT:    s_mov_b32 s6, -1
+; GFX11-NEXT:    ; implicit-def: $sgpr4_sgpr5
+; GFX11-NEXT:    s_branch .LBB12_2
+; GFX11-NEXT:  .LBB12_4:
+; GFX11-NEXT:    v_dual_mov_b32 v0, s4 :: v_dual_mov_b32 v1, s5
+; GFX11-NEXT:  .LBB12_5: ; %endif
+; GFX11-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX11-NEXT:    s_mov_b32 s2, -1
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    buffer_store_b64 v[0:1], off, s[0:3], 0
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; EG-LABEL: mul64_in_branch:
+; EG:       ; %bb.0: ; %entry
+; EG-NEXT:    ALU_PUSH_BEFORE 4, @14, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    JUMP @3 POP:1
+; EG-NEXT:    ALU_POP_AFTER 11, @19, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    ALU_PUSH_BEFORE 2, @31, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    JUMP @8 POP:1
+; EG-NEXT:    ALU 0, @34, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    TEX 0 @12
+; EG-NEXT:    POP @8 POP:1
+; EG-NEXT:    ALU 1, @35, KC0[], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T0.XY, T1.X, 1
+; EG-NEXT:    CF_END
+; EG-NEXT:    PAD
+; EG-NEXT:    Fetch clause starting at 12:
+; EG-NEXT:     VTX_READ_64 T0.XY, T0.X, 0, #1
+; EG-NEXT:    ALU clause starting at 14:
+; EG-NEXT:     OR_INT T0.W, KC0[2].W, KC0[3].X,
+; EG-NEXT:     MOV * T1.W, literal.x,
+; EG-NEXT:    1(1.401298e-45), 0(0.000000e+00)
+; EG-NEXT:     SETNE_INT * T0.W, PV.W, 0.0,
+; EG-NEXT:     PRED_SETNE_INT * ExecMask,PredicateBit (MASKED), PV.W, 0.0,
+; EG-NEXT:    ALU clause starting at 19:
+; EG-NEXT:     MOV T0.W, KC0[2].W,
+; EG-NEXT:     MOV * T1.W, KC0[3].Z,
+; EG-NEXT:     MOV T2.W, KC0[3].Y,
+; EG-NEXT:     MULLO_INT * T0.X, PV.W, PS,
+; EG-NEXT:     MOV T1.W, KC0[3].X,
+; EG-NEXT:     MULHI * T0.Y, T0.W, PV.W,
+; EG-NEXT:     ADD_INT T3.W, PS, T0.X,
+; EG-NEXT:     MULLO_INT * T0.X, PV.W, T2.W,
+; EG-NEXT:     ADD_INT T0.Y, PV.W, PS,
+; EG-NEXT:     MOV T1.W, literal.x,
+; EG-NEXT:     MULLO_INT * T0.X, T0.W, T2.W,
+; EG-NEXT:    0(0.000000e+00), 0(0.000000e+00)
+; EG-NEXT:    ALU clause starting at 31:
+; EG-NEXT:     MOV T0.W, KC0[2].Y,
+; EG-NEXT:     SETE_INT * T1.W, T1.W, 0.0,
+; EG-NEXT:     PRED_SETE_INT * ExecMask,PredicateBit (MASKED), PS, 0.0,
+; EG-NEXT:    ALU clause starting at 34:
+; EG-NEXT:     MOV * T0.X, KC0[2].Z,
+; EG-NEXT:    ALU clause starting at 35:
+; EG-NEXT:     LSHR * T1.X, T0.W, literal.x,
+; EG-NEXT:    2(2.802597e-45), 0(0.000000e+00)
 entry:
   %0 = icmp eq i64 %a, 0
   br i1 %0, label %if, label %else
@@ -211,79 +1819,556 @@ endif:
   ret void
 }
 
-; FIXME: Load dwordx4
-; FUNC-LABEL: {{^}}s_mul_i128:
-; GCN: s_load_dwordx4
-; GCN: s_load_dwordx4
-
-; SI: v_mul_hi_u32
-; SI: v_mul_hi_u32
-; SI: s_mul_i32
-; SI: v_mul_hi_u32
-; SI: s_mul_i32
-; SI: s_mul_i32
-
-; SI-DAG: s_mul_i32
-; SI-DAG: v_mul_hi_u32
-; SI-DAG: v_mul_hi_u32
-; SI-DAG: s_mul_i32
-; SI-DAG: s_mul_i32
-; SI-DAG: v_mul_hi_u32
-
-; VI-DAG: v_mad_u64_u32
-; VI-DAG: v_mad_u64_u32
-; VI-DAG: v_mad_u64_u32
-; VI-DAG: v_mad_u64_u32
-; VI-DAG: v_mad_u64_u32
-; VI-DAG: v_mad_u64_u32
-; VI-DAG: s_mul_i32
-; VI-DAG: s_mul_i32
-; VI-DAG: s_mul_i32
-; VI-DAG: s_mul_i32
-
-
-; GCN: buffer_store_dwordx4
 define amdgpu_kernel void @s_mul_i128(ptr addrspace(1) %out, [8 x i32], i128 %a, [8 x i32], i128 %b) nounwind #0 {
+; SI-LABEL: s_mul_i128:
+; SI:       ; %bb.0: ; %entry
+; SI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x13
+; SI-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x1f
+; SI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; SI-NEXT:    s_mov_b32 s3, 0xf000
+; SI-NEXT:    s_mov_b32 s2, -1
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    v_mov_b32_e32 v0, s6
+; SI-NEXT:    v_mul_hi_u32 v0, s8, v0
+; SI-NEXT:    v_mov_b32_e32 v1, s4
+; SI-NEXT:    v_mul_hi_u32 v1, s10, v1
+; SI-NEXT:    s_mul_i32 s7, s8, s7
+; SI-NEXT:    v_add_i32_e32 v0, vcc, s7, v0
+; SI-NEXT:    s_mul_i32 s7, s10, s5
+; SI-NEXT:    s_mul_i32 s12, s9, s6
+; SI-NEXT:    s_mul_i32 s6, s8, s6
+; SI-NEXT:    v_add_i32_e32 v1, vcc, s7, v1
+; SI-NEXT:    s_mul_i32 s7, s11, s4
+; SI-NEXT:    v_add_i32_e32 v0, vcc, s12, v0
+; SI-NEXT:    v_add_i32_e32 v1, vcc, s7, v1
+; SI-NEXT:    s_mul_i32 s7, s10, s4
+; SI-NEXT:    v_mov_b32_e32 v2, s6
+; SI-NEXT:    v_add_i32_e32 v2, vcc, s7, v2
+; SI-NEXT:    v_addc_u32_e32 v0, vcc, v1, v0, vcc
+; SI-NEXT:    v_mov_b32_e32 v1, s8
+; SI-NEXT:    v_mul_hi_u32 v5, s4, v1
+; SI-NEXT:    v_mul_hi_u32 v1, s5, v1
+; SI-NEXT:    v_mov_b32_e32 v3, s9
+; SI-NEXT:    v_mul_hi_u32 v4, s4, v3
+; SI-NEXT:    s_mul_i32 s7, s5, s8
+; SI-NEXT:    v_add_i32_e32 v5, vcc, s7, v5
+; SI-NEXT:    s_mul_i32 s6, s4, s9
+; SI-NEXT:    v_addc_u32_e32 v6, vcc, 0, v1, vcc
+; SI-NEXT:    v_add_i32_e32 v1, vcc, s6, v5
+; SI-NEXT:    v_mul_hi_u32 v3, s5, v3
+; SI-NEXT:    v_addc_u32_e32 v4, vcc, 0, v4, vcc
+; SI-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
+; SI-NEXT:    s_mul_i32 s5, s5, s9
+; SI-NEXT:    v_addc_u32_e64 v5, s[6:7], 0, 0, vcc
+; SI-NEXT:    v_add_i32_e32 v4, vcc, s5, v4
+; SI-NEXT:    v_addc_u32_e32 v3, vcc, v3, v5, vcc
+; SI-NEXT:    v_add_i32_e32 v2, vcc, v4, v2
+; SI-NEXT:    s_mul_i32 s4, s4, s8
+; SI-NEXT:    v_addc_u32_e32 v3, vcc, v3, v0, vcc
+; SI-NEXT:    v_mov_b32_e32 v0, s4
+; SI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
+; SI-NEXT:    s_endpgm
+;
+; VI-LABEL: s_mul_i128:
+; VI:       ; %bb.0: ; %entry
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x4c
+; VI-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x7c
+; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; VI-NEXT:    v_mov_b32_e32 v5, 0
+; VI-NEXT:    s_mov_b32 s3, 0xf000
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_mov_b32_e32 v0, s6
+; VI-NEXT:    v_mad_u64_u32 v[2:3], s[12:13], s8, v0, 0
+; VI-NEXT:    s_mul_i32 s7, s8, s7
+; VI-NEXT:    v_mov_b32_e32 v6, s8
+; VI-NEXT:    v_add_u32_e32 v3, vcc, s7, v3
+; VI-NEXT:    s_mul_i32 s12, s9, s6
+; VI-NEXT:    v_mad_u64_u32 v[0:1], s[6:7], s4, v6, 0
+; VI-NEXT:    v_add_u32_e32 v3, vcc, s12, v3
+; VI-NEXT:    v_mov_b32_e32 v4, v1
+; VI-NEXT:    v_mad_u64_u32 v[6:7], s[6:7], s5, v6, v[4:5]
+; VI-NEXT:    v_mov_b32_e32 v8, s4
+; VI-NEXT:    v_mad_u64_u32 v[1:2], s[6:7], s10, v8, v[2:3]
+; VI-NEXT:    v_mov_b32_e32 v3, v7
+; VI-NEXT:    v_mov_b32_e32 v7, v5
+; VI-NEXT:    v_mov_b32_e32 v8, s9
+; VI-NEXT:    v_mad_u64_u32 v[4:5], s[6:7], s4, v8, v[6:7]
+; VI-NEXT:    s_mul_i32 s8, s11, s4
+; VI-NEXT:    v_add_u32_e32 v6, vcc, s8, v2
+; VI-NEXT:    v_mov_b32_e32 v2, v5
+; VI-NEXT:    v_add_u32_e32 v2, vcc, v3, v2
+; VI-NEXT:    v_addc_u32_e64 v3, s[6:7], 0, 0, vcc
+; VI-NEXT:    s_mul_i32 s8, s10, s5
+; VI-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], s5, v8, v[2:3]
+; VI-NEXT:    v_add_u32_e32 v5, vcc, s8, v6
+; VI-NEXT:    v_add_u32_e32 v2, vcc, v2, v1
+; VI-NEXT:    s_mov_b32 s2, -1
+; VI-NEXT:    v_addc_u32_e32 v3, vcc, v3, v5, vcc
+; VI-NEXT:    v_mov_b32_e32 v1, v4
+; VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
+; VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: s_mul_i128:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x4c
+; GFX9-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x7c
+; GFX9-NEXT:    s_load_dwordx2 s[12:13], s[0:1], 0x24
+; GFX9-NEXT:    s_mov_b32 s15, 0xf000
+; GFX9-NEXT:    s_mov_b32 s14, -1
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mul_i32 s0, s8, s7
+; GFX9-NEXT:    s_mul_hi_u32 s1, s8, s6
+; GFX9-NEXT:    s_mul_i32 s2, s10, s5
+; GFX9-NEXT:    s_mul_hi_u32 s3, s10, s4
+; GFX9-NEXT:    s_add_i32 s0, s1, s0
+; GFX9-NEXT:    s_mul_i32 s1, s9, s6
+; GFX9-NEXT:    s_add_i32 s2, s3, s2
+; GFX9-NEXT:    s_mul_i32 s3, s11, s4
+; GFX9-NEXT:    s_add_i32 s0, s0, s1
+; GFX9-NEXT:    s_mul_i32 s1, s8, s6
+; GFX9-NEXT:    s_add_i32 s2, s2, s3
+; GFX9-NEXT:    s_mul_i32 s3, s10, s4
+; GFX9-NEXT:    s_add_u32 s3, s3, s1
+; GFX9-NEXT:    s_addc_u32 s2, s2, s0
+; GFX9-NEXT:    s_mul_i32 s10, s5, s8
+; GFX9-NEXT:    s_mul_hi_u32 s11, s4, s8
+; GFX9-NEXT:    s_mul_hi_u32 s7, s5, s8
+; GFX9-NEXT:    s_add_u32 s10, s10, s11
+; GFX9-NEXT:    s_mul_i32 s1, s4, s9
+; GFX9-NEXT:    s_addc_u32 s7, s7, 0
+; GFX9-NEXT:    s_mul_hi_u32 s6, s4, s9
+; GFX9-NEXT:    s_add_u32 s1, s1, s10
+; GFX9-NEXT:    s_addc_u32 s6, s6, 0
+; GFX9-NEXT:    s_add_u32 s6, s7, s6
+; GFX9-NEXT:    s_addc_u32 s7, 0, 0
+; GFX9-NEXT:    s_mul_hi_u32 s10, s5, s9
+; GFX9-NEXT:    s_mul_i32 s5, s5, s9
+; GFX9-NEXT:    s_add_u32 s5, s5, s6
+; GFX9-NEXT:    s_addc_u32 s6, s10, s7
+; GFX9-NEXT:    s_mov_b32 s0, 0
+; GFX9-NEXT:    s_add_u32 s5, s5, s3
+; GFX9-NEXT:    s_addc_u32 s6, s6, s2
+; GFX9-NEXT:    s_mul_i32 s2, s4, s8
+; GFX9-NEXT:    s_mov_b32 s3, s0
+; GFX9-NEXT:    s_or_b64 s[0:1], s[2:3], s[0:1]
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    v_mov_b32_e32 v2, s5
+; GFX9-NEXT:    v_mov_b32_e32 v3, s6
+; GFX9-NEXT:    buffer_store_dwordx4 v[0:3], off, s[12:15], 0
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: s_mul_i128:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x4c
+; GFX10-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x7c
+; GFX10-NEXT:    s_mov_b32 s2, 0
+; GFX10-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GFX10-NEXT:    s_mov_b32 s13, s2
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mul_i32 s3, s8, s7
+; GFX10-NEXT:    s_mul_hi_u32 s7, s8, s6
+; GFX10-NEXT:    s_mul_i32 s14, s10, s5
+; GFX10-NEXT:    s_mul_hi_u32 s15, s10, s4
+; GFX10-NEXT:    s_mul_i32 s12, s9, s6
+; GFX10-NEXT:    s_mul_i32 s11, s11, s4
+; GFX10-NEXT:    s_add_i32 s3, s7, s3
+; GFX10-NEXT:    s_add_i32 s7, s15, s14
+; GFX10-NEXT:    s_mul_i32 s6, s8, s6
+; GFX10-NEXT:    s_mul_i32 s10, s10, s4
+; GFX10-NEXT:    s_add_i32 s3, s3, s12
+; GFX10-NEXT:    s_add_i32 s7, s7, s11
+; GFX10-NEXT:    s_mul_i32 s19, s5, s8
+; GFX10-NEXT:    s_mul_hi_u32 s20, s4, s8
+; GFX10-NEXT:    s_add_u32 s6, s10, s6
+; GFX10-NEXT:    s_mul_hi_u32 s18, s5, s8
+; GFX10-NEXT:    s_addc_u32 s7, s7, s3
+; GFX10-NEXT:    s_mul_i32 s17, s4, s9
+; GFX10-NEXT:    s_add_u32 s3, s19, s20
+; GFX10-NEXT:    s_mul_hi_u32 s16, s4, s9
+; GFX10-NEXT:    s_mul_hi_u32 s21, s5, s9
+; GFX10-NEXT:    s_mul_i32 s5, s5, s9
+; GFX10-NEXT:    s_addc_u32 s9, s18, 0
+; GFX10-NEXT:    s_add_u32 s3, s17, s3
+; GFX10-NEXT:    s_addc_u32 s10, s16, 0
+; GFX10-NEXT:    s_mul_i32 s12, s4, s8
+; GFX10-NEXT:    s_add_u32 s4, s9, s10
+; GFX10-NEXT:    s_addc_u32 s8, 0, 0
+; GFX10-NEXT:    s_add_u32 s4, s5, s4
+; GFX10-NEXT:    s_addc_u32 s5, s21, s8
+; GFX10-NEXT:    s_add_u32 s4, s4, s6
+; GFX10-NEXT:    s_addc_u32 s5, s5, s7
+; GFX10-NEXT:    s_or_b64 s[2:3], s[12:13], s[2:3]
+; GFX10-NEXT:    v_mov_b32_e32 v2, s4
+; GFX10-NEXT:    v_mov_b32_e32 v0, s2
+; GFX10-NEXT:    v_mov_b32_e32 v1, s3
+; GFX10-NEXT:    v_mov_b32_e32 v3, s5
+; GFX10-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX10-NEXT:    s_mov_b32 s2, -1
+; GFX10-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: s_mul_i128:
+; GFX11:       ; %bb.0: ; %entry
+; GFX11-NEXT:    s_clause 0x2
+; GFX11-NEXT:    s_load_b128 s[4:7], s[0:1], 0x4c
+; GFX11-NEXT:    s_load_b128 s[8:11], s[0:1], 0x7c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    s_mov_b32 s2, 0
+; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
+; GFX11-NEXT:    s_mov_b32 s13, s2
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_mul_i32 s3, s8, s7
+; GFX11-NEXT:    s_mul_hi_u32 s7, s8, s6
+; GFX11-NEXT:    s_mul_i32 s14, s10, s5
+; GFX11-NEXT:    s_mul_hi_u32 s15, s10, s4
+; GFX11-NEXT:    s_mul_i32 s12, s9, s6
+; GFX11-NEXT:    s_mul_i32 s11, s11, s4
+; GFX11-NEXT:    s_add_i32 s3, s7, s3
+; GFX11-NEXT:    s_add_i32 s7, s15, s14
+; GFX11-NEXT:    s_mul_i32 s6, s8, s6
+; GFX11-NEXT:    s_mul_i32 s10, s10, s4
+; GFX11-NEXT:    s_add_i32 s3, s3, s12
+; GFX11-NEXT:    s_add_i32 s7, s7, s11
+; GFX11-NEXT:    s_mul_i32 s19, s5, s8
+; GFX11-NEXT:    s_mul_hi_u32 s20, s4, s8
+; GFX11-NEXT:    s_add_u32 s6, s10, s6
+; GFX11-NEXT:    s_mul_hi_u32 s18, s5, s8
+; GFX11-NEXT:    s_addc_u32 s7, s7, s3
+; GFX11-NEXT:    s_mul_i32 s17, s4, s9
+; GFX11-NEXT:    s_add_u32 s3, s19, s20
+; GFX11-NEXT:    s_mul_hi_u32 s16, s4, s9
+; GFX11-NEXT:    s_mul_hi_u32 s21, s5, s9
+; GFX11-NEXT:    s_mul_i32 s5, s5, s9
+; GFX11-NEXT:    s_addc_u32 s9, s18, 0
+; GFX11-NEXT:    s_add_u32 s3, s17, s3
+; GFX11-NEXT:    s_addc_u32 s10, s16, 0
+; GFX11-NEXT:    s_mul_i32 s12, s4, s8
+; GFX11-NEXT:    s_add_u32 s4, s9, s10
+; GFX11-NEXT:    s_addc_u32 s8, 0, 0
+; GFX11-NEXT:    s_add_u32 s4, s5, s4
+; GFX11-NEXT:    s_addc_u32 s5, s21, s8
+; GFX11-NEXT:    s_add_u32 s4, s4, s6
+; GFX11-NEXT:    s_addc_u32 s5, s5, s7
+; GFX11-NEXT:    s_or_b64 s[2:3], s[12:13], s[2:3]
+; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
+; GFX11-NEXT:    v_dual_mov_b32 v2, s4 :: v_dual_mov_b32 v1, s3
+; GFX11-NEXT:    v_dual_mov_b32 v0, s2 :: v_dual_mov_b32 v3, s5
+; GFX11-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX11-NEXT:    s_mov_b32 s2, -1
+; GFX11-NEXT:    buffer_store_b128 v[0:3], off, s[0:3], 0
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; EG-LABEL: s_mul_i128:
+; EG:       ; %bb.0: ; %entry
+; EG-NEXT:    ALU 41, @4, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T0.XYZW, T1.X, 1
+; EG-NEXT:    CF_END
+; EG-NEXT:    PAD
+; EG-NEXT:    ALU clause starting at 4:
+; EG-NEXT:     MULLO_INT * T0.X, KC0[5].X, KC0[8].X,
+; EG-NEXT:     MULHI * T0.Y, KC0[5].X, KC0[8].X,
+; EG-NEXT:     MULLO_INT * T0.Z, KC0[8].Y, KC0[4].W,
+; EG-NEXT:     MULLO_INT * T0.W, KC0[8].X, KC0[5].Y,
+; EG-NEXT:     MULHI * T1.X, KC0[5].X, KC0[7].W,
+; EG-NEXT:     MULHI * T1.Y, KC0[4].W, KC0[8].X,
+; EG-NEXT:     MULHI * T1.Z, KC0[8].Y, KC0[4].W,
+; EG-NEXT:     MULLO_INT * T1.W, KC0[8].Y, KC0[5].X,
+; EG-NEXT:     MULHI * T2.X, KC0[7].W, KC0[5].Y,
+; EG-NEXT:     MULLO_INT * T2.Y, KC0[5].X, KC0[7].W,
+; EG-NEXT:     MULHI * T2.Z, KC0[4].W, KC0[7].W,
+; EG-NEXT:     ADD_INT T2.W, T2.Y, PS,
+; EG-NEXT:     MULLO_INT * T3.X, KC0[4].W, KC0[8].X,
+; EG-NEXT:     ADDC_UINT T2.Z, T2.Y, T2.Z,
+; EG-NEXT:     ADDC_UINT T3.W, PS, PV.W,
+; EG-NEXT:     MULLO_INT * T2.Y, KC0[7].W, KC0[5].Z,
+; EG-NEXT:     ADD_INT T2.X, T2.X, PS,
+; EG-NEXT:     ADD_INT T2.Y, T1.Z, T1.W,
+; EG-NEXT:     ADD_INT T1.Z, T1.Y, PV.W,
+; EG-NEXT:     ADD_INT T1.W, T1.X, PV.Z, BS:VEC_120/SCL_212
+; EG-NEXT:     MULLO_INT * T1.X, KC0[8].Z, KC0[4].W,
+; EG-NEXT:     ADD_INT T4.X, PV.W, PV.Z,
+; EG-NEXT:     ADDC_UINT T1.Y, PV.W, PV.Z,
+; EG-NEXT:     ADD_INT T1.Z, PV.Y, PS,
+; EG-NEXT:     ADD_INT T0.W, PV.X, T0.W,
+; EG-NEXT:     MULLO_INT * T1.X, KC0[7].W, KC0[5].Y,
+; EG-NEXT:     ADD_INT T2.Y, PV.Z, PV.W,
+; EG-NEXT:     ADDC_UINT T1.Z, T0.Z, PS,
+; EG-NEXT:     ADD_INT T0.W, T0.Y, PV.Y,
+; EG-NEXT:     ADDC_UINT * T1.W, T0.X, PV.X,
+; EG-NEXT:     ADD_INT T0.Y, T0.X, T4.X,
+; EG-NEXT:     ADD_INT T0.Z, T0.Z, T1.X, BS:VEC_021/SCL_122
+; EG-NEXT:     ADD_INT T0.W, PV.W, PS,
+; EG-NEXT:     ADD_INT * T1.W, PV.Y, PV.Z,
+; EG-NEXT:     ADD_INT T0.W, PV.W, PS,
+; EG-NEXT:     ADDC_UINT * T1.W, PV.Y, PV.Z,
+; EG-NEXT:     ADD_INT * T0.W, PV.W, PS,
+; EG-NEXT:     ADD_INT * T0.Z, T0.Y, T0.Z,
+; EG-NEXT:     ADD_INT * T0.Y, T3.X, T2.W,
+; EG-NEXT:     LSHR * T1.X, KC0[2].Y, literal.x,
+; EG-NEXT:    2(2.802597e-45), 0(0.000000e+00)
+; EG-NEXT:     MULLO_INT * T0.X, KC0[4].W, KC0[7].W,
+entry:
   %mul = mul i128 %a, %b
   store i128 %mul, ptr addrspace(1) %out
   ret void
 }
 
-; FUNC-LABEL: {{^}}v_mul_i128:
-; GCN: {{buffer|flat}}_load_dwordx4
-; GCN: {{buffer|flat}}_load_dwordx4
-
-; SI-DAG: v_mul_lo_u32
-; SI-DAG: v_mul_hi_u32
-; SI-DAG: v_mul_hi_u32
-; SI-DAG: v_mul_lo_u32
-; SI-DAG: v_mul_hi_u32
-; SI-DAG: v_mul_hi_u32
-; SI-DAG: v_mul_lo_u32
-; SI-DAG: v_mul_lo_u32
-; SI-DAG: v_add_i32_e32
-
-; SI-DAG: v_mul_hi_u32
-; SI-DAG: v_mul_lo_u32
-; SI-DAG: v_mul_hi_u32
-; SI-DAG: v_mul_lo_u32
-; SI-DAG: v_mul_lo_u32
-; SI-DAG: v_mul_lo_u32
-; SI-DAG: v_mul_lo_u32
-; SI-DAG: v_mul_lo_u32
-
-; VI-DAG: v_mad_u64_u32
-; VI-DAG: v_mad_u64_u32
-; VI-DAG: v_mad_u64_u32
-; VI-DAG: v_mad_u64_u32
-; VI-DAG: v_mad_u64_u32
-; VI-DAG: v_mad_u64_u32
-; VI-DAG: v_mul_lo_u32
-; VI-DAG: v_mul_lo_u32
-; VI-DAG: v_mul_lo_u32
-
-; GCN: {{buffer|flat}}_store_dwordx4
 define amdgpu_kernel void @v_mul_i128(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) #0 {
+; SI-LABEL: v_mul_i128:
+; SI:       ; %bb.0: ; %entry
+; SI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0xb
+; SI-NEXT:    s_mov_b32 s7, 0xf000
+; SI-NEXT:    s_mov_b32 s6, 0
+; SI-NEXT:    v_lshlrev_b32_e32 v8, 4, v0
+; SI-NEXT:    v_mov_b32_e32 v9, 0
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    s_mov_b64 s[4:5], s[0:1]
+; SI-NEXT:    s_mov_b64 s[0:1], s[2:3]
+; SI-NEXT:    s_mov_b64 s[2:3], s[6:7]
+; SI-NEXT:    buffer_load_dwordx4 v[0:3], v[8:9], s[4:7], 0 addr64
+; SI-NEXT:    buffer_load_dwordx4 v[4:7], v[8:9], s[0:3], 0 addr64
+; SI-NEXT:    s_waitcnt vmcnt(0)
+; SI-NEXT:    v_mul_lo_u32 v3, v4, v3
+; SI-NEXT:    v_mul_hi_u32 v10, v4, v2
+; SI-NEXT:    v_mul_lo_u32 v12, v6, v1
+; SI-NEXT:    v_mul_hi_u32 v13, v6, v0
+; SI-NEXT:    v_mul_lo_u32 v17, v1, v4
+; SI-NEXT:    v_mul_hi_u32 v18, v0, v4
+; SI-NEXT:    v_mul_lo_u32 v11, v5, v2
+; SI-NEXT:    v_mul_lo_u32 v7, v7, v0
+; SI-NEXT:    v_mul_hi_u32 v16, v1, v4
+; SI-NEXT:    v_mul_lo_u32 v15, v0, v5
+; SI-NEXT:    v_mul_hi_u32 v14, v0, v5
+; SI-NEXT:    v_mul_hi_u32 v19, v1, v5
+; SI-NEXT:    v_mul_lo_u32 v5, v1, v5
+; SI-NEXT:    v_add_i32_e32 v1, vcc, v10, v3
+; SI-NEXT:    v_add_i32_e32 v3, vcc, v13, v12
+; SI-NEXT:    v_mul_lo_u32 v2, v4, v2
+; SI-NEXT:    v_mul_lo_u32 v6, v6, v0
+; SI-NEXT:    v_mul_lo_u32 v0, v0, v4
+; SI-NEXT:    v_add_i32_e32 v4, vcc, v17, v18
+; SI-NEXT:    v_addc_u32_e32 v10, vcc, 0, v16, vcc
+; SI-NEXT:    v_add_i32_e32 v11, vcc, v1, v11
+; SI-NEXT:    v_add_i32_e32 v3, vcc, v3, v7
+; SI-NEXT:    v_add_i32_e32 v1, vcc, v15, v4
+; SI-NEXT:    v_addc_u32_e32 v4, vcc, 0, v14, vcc
+; SI-NEXT:    v_add_i32_e32 v2, vcc, v6, v2
+; SI-NEXT:    v_addc_u32_e32 v3, vcc, v3, v11, vcc
+; SI-NEXT:    v_add_i32_e32 v4, vcc, v10, v4
+; SI-NEXT:    v_addc_u32_e64 v6, s[4:5], 0, 0, vcc
+; SI-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
+; SI-NEXT:    v_addc_u32_e32 v5, vcc, v19, v6, vcc
+; SI-NEXT:    v_add_i32_e32 v2, vcc, v4, v2
+; SI-NEXT:    v_addc_u32_e32 v3, vcc, v5, v3, vcc
+; SI-NEXT:    buffer_store_dwordx4 v[0:3], v[8:9], s[0:3], 0 addr64
+; SI-NEXT:    s_endpgm
+;
+; VI-LABEL: v_mul_i128:
+; VI:       ; %bb.0: ; %entry
+; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x2c
+; VI-NEXT:    v_lshlrev_b32_e32 v2, 4, v0
+; VI-NEXT:    v_mov_b32_e32 v11, 0
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_mov_b32_e32 v1, s1
+; VI-NEXT:    v_add_u32_e32 v0, vcc, s0, v2
+; VI-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; VI-NEXT:    v_mov_b32_e32 v3, s3
+; VI-NEXT:    v_add_u32_e32 v8, vcc, s2, v2
+; VI-NEXT:    v_addc_u32_e32 v9, vcc, 0, v3, vcc
+; VI-NEXT:    flat_load_dwordx4 v[0:3], v[0:1]
+; VI-NEXT:    flat_load_dwordx4 v[4:7], v[8:9]
+; VI-NEXT:    s_waitcnt vmcnt(0)
+; VI-NEXT:    v_mul_lo_u32 v10, v4, v3
+; VI-NEXT:    v_mad_u64_u32 v[12:13], s[0:1], v4, v2, 0
+; VI-NEXT:    v_mul_lo_u32 v14, v5, v2
+; VI-NEXT:    v_mad_u64_u32 v[2:3], s[0:1], v0, v4, 0
+; VI-NEXT:    v_mul_lo_u32 v15, v7, v0
+; VI-NEXT:    v_add_u32_e32 v7, vcc, v13, v10
+; VI-NEXT:    v_mov_b32_e32 v10, v3
+; VI-NEXT:    v_mad_u64_u32 v[3:4], s[0:1], v1, v4, v[10:11]
+; VI-NEXT:    v_add_u32_e32 v13, vcc, v7, v14
+; VI-NEXT:    v_mov_b32_e32 v7, v4
+; VI-NEXT:    v_mov_b32_e32 v4, v11
+; VI-NEXT:    v_mad_u64_u32 v[12:13], s[0:1], v6, v0, v[12:13]
+; VI-NEXT:    v_mad_u64_u32 v[3:4], s[0:1], v0, v5, v[3:4]
+; VI-NEXT:    v_add_u32_e32 v11, vcc, v15, v13
+; VI-NEXT:    v_mov_b32_e32 v0, v4
+; VI-NEXT:    v_mul_lo_u32 v10, v6, v1
+; VI-NEXT:    v_add_u32_e32 v6, vcc, v7, v0
+; VI-NEXT:    v_addc_u32_e64 v7, s[0:1], 0, 0, vcc
+; VI-NEXT:    v_mad_u64_u32 v[0:1], s[0:1], v1, v5, v[6:7]
+; VI-NEXT:    v_add_u32_e32 v5, vcc, v10, v11
+; VI-NEXT:    v_add_u32_e32 v4, vcc, v0, v12
+; VI-NEXT:    v_addc_u32_e32 v5, vcc, v1, v5, vcc
+; VI-NEXT:    flat_store_dwordx4 v[8:9], v[2:5]
+; VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_mul_i128:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x2c
+; GFX9-NEXT:    v_lshlrev_b32_e32 v13, 4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v10, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dwordx4 v[0:3], v13, s[0:1]
+; GFX9-NEXT:    global_load_dwordx4 v[4:7], v13, s[2:3]
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_mad_u64_u32 v[8:9], s[0:1], v0, v4, 0
+; GFX9-NEXT:    v_mul_lo_u32 v14, v5, v2
+; GFX9-NEXT:    v_mul_lo_u32 v15, v4, v3
+; GFX9-NEXT:    v_mad_u64_u32 v[11:12], s[0:1], v1, v4, v[9:10]
+; GFX9-NEXT:    v_mad_u64_u32 v[2:3], s[0:1], v4, v2, 0
+; GFX9-NEXT:    v_mov_b32_e32 v4, v12
+; GFX9-NEXT:    v_mov_b32_e32 v12, v10
+; GFX9-NEXT:    v_mad_u64_u32 v[9:10], s[0:1], v0, v5, v[11:12]
+; GFX9-NEXT:    v_add3_u32 v3, v3, v15, v14
+; GFX9-NEXT:    v_mul_lo_u32 v17, v7, v0
+; GFX9-NEXT:    v_mad_u64_u32 v[2:3], s[0:1], v6, v0, v[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v0, v10
+; GFX9-NEXT:    v_mul_lo_u32 v16, v6, v1
+; GFX9-NEXT:    v_add_co_u32_e32 v6, vcc, v4, v0
+; GFX9-NEXT:    v_addc_co_u32_e64 v7, s[0:1], 0, 0, vcc
+; GFX9-NEXT:    v_mad_u64_u32 v[0:1], s[0:1], v1, v5, v[6:7]
+; GFX9-NEXT:    v_add3_u32 v3, v17, v3, v16
+; GFX9-NEXT:    v_add_co_u32_e32 v10, vcc, v0, v2
+; GFX9-NEXT:    v_addc_co_u32_e32 v11, vcc, v1, v3, vcc
+; GFX9-NEXT:    global_store_dwordx4 v13, v[8:11], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: v_mul_i128:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x2c
+; GFX10-NEXT:    v_lshlrev_b32_e32 v14, 4, v0
+; GFX10-NEXT:    v_mov_b32_e32 v10, 0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    global_load_dwordx4 v[0:3], v14, s[0:1]
+; GFX10-NEXT:    global_load_dwordx4 v[4:7], v14, s[2:3]
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_mad_u64_u32 v[8:9], s0, v0, v4, 0
+; GFX10-NEXT:    v_mul_lo_u32 v7, v7, v0
+; GFX10-NEXT:    v_mad_u64_u32 v[11:12], s0, v1, v4, v[9:10]
+; GFX10-NEXT:    v_mov_b32_e32 v9, v12
+; GFX10-NEXT:    v_mov_b32_e32 v12, v10
+; GFX10-NEXT:    v_mul_lo_u32 v10, v5, v2
+; GFX10-NEXT:    v_mad_u64_u32 v[12:13], s0, v0, v5, v[11:12]
+; GFX10-NEXT:    v_mul_lo_u32 v11, v4, v3
+; GFX10-NEXT:    v_mad_u64_u32 v[2:3], s0, v4, v2, 0
+; GFX10-NEXT:    v_mov_b32_e32 v4, v13
+; GFX10-NEXT:    v_mul_lo_u32 v13, v6, v1
+; GFX10-NEXT:    v_add3_u32 v3, v3, v11, v10
+; GFX10-NEXT:    v_add_co_u32 v9, s0, v9, v4
+; GFX10-NEXT:    v_add_co_ci_u32_e64 v10, s0, 0, 0, s0
+; GFX10-NEXT:    v_mad_u64_u32 v[2:3], s0, v6, v0, v[2:3]
+; GFX10-NEXT:    v_mad_u64_u32 v[0:1], s0, v1, v5, v[9:10]
+; GFX10-NEXT:    v_mov_b32_e32 v9, v12
+; GFX10-NEXT:    v_add3_u32 v3, v7, v3, v13
+; GFX10-NEXT:    v_add_co_u32 v10, vcc_lo, v0, v2
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v11, vcc_lo, v1, v3, vcc_lo
+; GFX10-NEXT:    global_store_dwordx4 v14, v[8:11], s[2:3]
+; GFX10-NEXT:    s_endpgm
+;
+; GFX11-LABEL: v_mul_i128:
+; GFX11:       ; %bb.0: ; %entry
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x2c
+; GFX11-NEXT:    v_lshlrev_b32_e32 v16, 4, v0
+; GFX11-NEXT:    v_mov_b32_e32 v10, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    global_load_b128 v[0:3], v16, s[0:1]
+; GFX11-NEXT:    global_load_b128 v[4:7], v16, s[2:3]
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    v_mad_u64_u32 v[8:9], null, v0, v4, 0
+; GFX11-NEXT:    v_mul_lo_u32 v15, v5, v2
+; GFX11-NEXT:    v_mul_lo_u32 v3, v4, v3
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_3) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX11-NEXT:    v_mad_u64_u32 v[11:12], null, v1, v4, v[9:10]
+; GFX11-NEXT:    v_dual_mov_b32 v9, v12 :: v_dual_mov_b32 v12, v10
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_3) | instid1(VALU_DEP_4)
+; GFX11-NEXT:    v_mad_u64_u32 v[13:14], null, v0, v5, v[11:12]
+; GFX11-NEXT:    v_mad_u64_u32 v[10:11], null, v4, v2, 0
+; GFX11-NEXT:    v_mul_lo_u32 v4, v6, v1
+; GFX11-NEXT:    v_mul_lo_u32 v12, v7, v0
+; GFX11-NEXT:    v_mov_b32_e32 v2, v14
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_4) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_add3_u32 v11, v11, v3, v15
+; GFX11-NEXT:    v_add_co_u32 v2, s0, v9, v2
+; GFX11-NEXT:    v_mov_b32_e32 v9, v13
+; GFX11-NEXT:    v_add_co_ci_u32_e64 v3, null, 0, 0, s0
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_4) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mad_u64_u32 v[14:15], null, v6, v0, v[10:11]
+; GFX11-NEXT:    v_mad_u64_u32 v[6:7], null, v1, v5, v[2:3]
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_add3_u32 v0, v12, v15, v4
+; GFX11-NEXT:    v_add_co_u32 v10, vcc_lo, v6, v14
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v11, vcc_lo, v7, v0, vcc_lo
+; GFX11-NEXT:    global_store_b128 v16, v[8:11], s[2:3]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; EG-LABEL: v_mul_i128:
+; EG:       ; %bb.0: ; %entry
+; EG-NEXT:    ALU 3, @10, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    TEX 1 @6
+; EG-NEXT:    ALU 41, @14, KC0[], KC1[]
+; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T0.XYZW, T1.X, 1
+; EG-NEXT:    CF_END
+; EG-NEXT:    PAD
+; EG-NEXT:    Fetch clause starting at 6:
+; EG-NEXT:     VTX_READ_128 T2.XYZW, T1.X, 0, #1
+; EG-NEXT:     VTX_READ_128 T0.XYZW, T0.X, 0, #1
+; EG-NEXT:    ALU clause starting at 10:
+; EG-NEXT:     LSHL * T0.W, T0.X, literal.x,
+; EG-NEXT:    4(5.605194e-45), 0(0.000000e+00)
+; EG-NEXT:     ADD_INT T0.X, KC0[2].Z, PV.W,
+; EG-NEXT:     ADD_INT * T1.X, KC0[2].W, PV.W,
+; EG-NEXT:    ALU clause starting at 14:
+; EG-NEXT:     MULLO_INT * T1.Y, T0.Y, T2.Y,
+; EG-NEXT:     MULHI * T1.Z, T0.Y, T2.Y,
+; EG-NEXT:     MULLO_INT * T1.W, T2.Z, T0.X,
+; EG-NEXT:     MULLO_INT * T3.X, T2.Y, T0.Z,
+; EG-NEXT:     MULHI * T3.Y, T0.Y, T2.X,
+; EG-NEXT:     MULHI * T3.Z, T0.X, T2.Y,
+; EG-NEXT:     MULHI * T3.W, T2.Z, T0.X,
+; EG-NEXT:     MULLO_INT * T2.Z, T2.Z, T0.Y,
+; EG-NEXT:     MULHI * T4.X, T2.X, T0.Z,
+; EG-NEXT:     MULLO_INT * T0.Y, T0.Y, T2.X,
+; EG-NEXT:     MULHI * T4.Y, T0.X, T2.X,
+; EG-NEXT:     ADD_INT T4.W, T0.Y, PS,
+; EG-NEXT:     MULLO_INT * T2.Y, T0.X, T2.Y,
+; EG-NEXT:     ADDC_UINT T4.Z, T0.Y, T4.Y,
+; EG-NEXT:     ADDC_UINT T5.W, PS, PV.W,
+; EG-NEXT:     MULLO_INT * T0.Y, T2.X, T0.W,
+; EG-NEXT:     ADD_INT T4.X, T4.X, PS,
+; EG-NEXT:     ADD_INT T0.Y, T3.W, T2.Z,
+; EG-NEXT:     ADD_INT T2.Z, T3.Z, PV.W,
+; EG-NEXT:     ADD_INT T0.W, T3.Y, PV.Z,
+; EG-NEXT:     MULLO_INT * T2.W, T2.W, T0.X,
+; EG-NEXT:     ADD_INT T5.X, PV.W, PV.Z,
+; EG-NEXT:     ADDC_UINT T3.Y, PV.W, PV.Z,
+; EG-NEXT:     ADD_INT T2.Z, PV.Y, PS,
+; EG-NEXT:     ADD_INT T0.W, PV.X, T3.X,
+; EG-NEXT:     MULLO_INT * T0.Y, T2.X, T0.Z,
+; EG-NEXT:     ADD_INT T4.Y, PV.Z, PV.W,
+; EG-NEXT:     ADDC_UINT T0.Z, T1.W, PS,
+; EG-NEXT:     ADD_INT T0.W, T1.Z, PV.Y,
+; EG-NEXT:     ADDC_UINT * T2.W, T1.Y, PV.X,
+; EG-NEXT:     ADD_INT T1.Y, T1.Y, T5.X,
+; EG-NEXT:     ADD_INT T1.Z, T1.W, T0.Y,
+; EG-NEXT:     ADD_INT T0.W, PV.W, PS,
+; EG-NEXT:     ADD_INT * T1.W, PV.Y, PV.Z,
+; EG-NEXT:     ADD_INT T0.W, PV.W, PS,
+; EG-NEXT:     ADDC_UINT * T1.W, PV.Y, PV.Z,
+; EG-NEXT:     ADD_INT * T0.W, PV.W, PS,
+; EG-NEXT:     ADD_INT * T0.Z, T1.Y, T1.Z,
+; EG-NEXT:     ADD_INT * T0.Y, T2.Y, T4.W,
+; EG-NEXT:     LSHR T1.X, T1.X, literal.x,
+; EG-NEXT:     MULLO_INT * T0.X, T0.X, T2.X,
+; EG-NEXT:    2(2.802597e-45), 0(0.000000e+00)
+entry:
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %gep.a = getelementptr inbounds i128, ptr addrspace(1) %aptr, i32 %tid
   %gep.b = getelementptr inbounds i128, ptr addrspace(1) %bptr, i32 %tid
diff --git a/llvm/test/CodeGen/AMDGPU/occupancy-levels.ll b/llvm/test/CodeGen/AMDGPU/occupancy-levels.ll
index 3527329d1ee3b..5f3509c2517f7 100644
--- a/llvm/test/CodeGen/AMDGPU/occupancy-levels.ll
+++ b/llvm/test/CodeGen/AMDGPU/occupancy-levels.ll
@@ -1,6 +1,6 @@
-; RUN: llc -march=amdgcn -mcpu=gfx900 < %s | FileCheck --check-prefixes=GCN,GFX9 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1010 < %s | FileCheck --check-prefixes=GCN,GFX10,GFX10W32,GFX1010,GFX1010W32 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize64 < %s | FileCheck --check-prefixes=GCN,GFX10,GFX10W64,GFX1010,GFX1010W64 %s
+; RUN: llc -march=amdgcn -mcpu=gfx900 -mattr=-xnack < %s | FileCheck --check-prefixes=GCN,GFX9 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-xnack < %s | FileCheck --check-prefixes=GCN,GFX10,GFX10W32,GFX1010,GFX1010W32 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-xnack -mattr=+wavefrontsize64 < %s | FileCheck --check-prefixes=GCN,GFX10,GFX10W64,GFX1010,GFX1010W64 %s
 ; RUN: llc -march=amdgcn -mcpu=gfx1030 < %s | FileCheck --check-prefixes=GCN,GFX10,GFX10W32,GFX1030,GFX1030W32 %s
 ; RUN: llc -march=amdgcn -mcpu=gfx1030 -mattr=+wavefrontsize64 < %s | FileCheck --check-prefixes=GCN,GFX10,GFX10W64,GFX1030,GFX1030W64 %s
 ; RUN: llc -march=amdgcn -mcpu=gfx1100 < %s | FileCheck --check-prefixes=GCN,GFX1100,GFX1100W32 %s
diff --git a/llvm/test/CodeGen/AMDGPU/promote-constOffset-to-imm.ll b/llvm/test/CodeGen/AMDGPU/promote-constOffset-to-imm.ll
index 8fd73ed68b24e..be742480a59bd 100644
--- a/llvm/test/CodeGen/AMDGPU/promote-constOffset-to-imm.ll
+++ b/llvm/test/CodeGen/AMDGPU/promote-constOffset-to-imm.ll
@@ -112,53 +112,53 @@ define amdgpu_kernel void @clmem_read_simplified(ptr addrspace(1)  %buffer) {
 ; GFX900-NEXT:    s_swappc_b64 s[30:31], s[4:5]
 ; GFX900-NEXT:    v_and_b32_e32 v1, 0xff, v0
 ; GFX900-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
+; GFX900-NEXT:    v_and_b32_e32 v18, 0xffff8000, v0
 ; GFX900-NEXT:    v_mov_b32_e32 v2, 0
-; GFX900-NEXT:    v_and_b32_e32 v0, 0xffff8000, v0
-; GFX900-NEXT:    v_mov_b32_e32 v4, s35
-; GFX900-NEXT:    v_add_co_u32_e32 v3, vcc, s34, v0
+; GFX900-NEXT:    v_mov_b32_e32 v0, s35
+; GFX900-NEXT:    v_add_co_u32_e32 v3, vcc, s34, v18
+; GFX900-NEXT:    v_addc_co_u32_e32 v4, vcc, 0, v0, vcc
 ; GFX900-NEXT:    v_lshlrev_b64 v[0:1], 3, v[1:2]
-; GFX900-NEXT:    v_addc_co_u32_e32 v4, vcc, 0, v4, vcc
+; GFX900-NEXT:    s_movk_i32 s1, 0x2000
 ; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v3, v0
 ; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v4, v1, vcc
-; GFX900-NEXT:    s_movk_i32 s1, 0x2000
-; GFX900-NEXT:    global_load_dwordx2 v[5:6], v[0:1], off
-; GFX900-NEXT:    global_load_dwordx2 v[7:8], v[0:1], off offset:2048
-; GFX900-NEXT:    v_add_co_u32_e32 v9, vcc, s1, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v10, vcc, 0, v1, vcc
-; GFX900-NEXT:    global_load_dwordx2 v[11:12], v[9:10], off offset:-4096
+; GFX900-NEXT:    global_load_dwordx2 v[2:3], v[0:1], off
+; GFX900-NEXT:    global_load_dwordx2 v[4:5], v[0:1], off offset:2048
+; GFX900-NEXT:    v_add_co_u32_e32 v6, vcc, s1, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v7, vcc, 0, v1, vcc
+; GFX900-NEXT:    global_load_dwordx2 v[8:9], v[6:7], off offset:-4096
 ; GFX900-NEXT:    s_movk_i32 s0, 0x1000
-; GFX900-NEXT:    v_add_co_u32_e32 v13, vcc, s0, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v14, vcc, 0, v1, vcc
-; GFX900-NEXT:    global_load_dwordx2 v[15:16], v[13:14], off offset:2048
-; GFX900-NEXT:    global_load_dwordx2 v[17:18], v[9:10], off
-; GFX900-NEXT:    global_load_dwordx2 v[19:20], v[9:10], off offset:2048
+; GFX900-NEXT:    v_add_co_u32_e32 v10, vcc, s0, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v11, vcc, 0, v1, vcc
+; GFX900-NEXT:    global_load_dwordx2 v[12:13], v[10:11], off offset:2048
+; GFX900-NEXT:    global_load_dwordx2 v[14:15], v[6:7], off
+; GFX900-NEXT:    global_load_dwordx2 v[16:17], v[6:7], off offset:2048
 ; GFX900-NEXT:    s_movk_i32 s0, 0x3000
 ; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, s0, v0
 ; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
-; GFX900-NEXT:    global_load_dwordx2 v[9:10], v[0:1], off
-; GFX900-NEXT:    global_load_dwordx2 v[13:14], v[0:1], off offset:2048
+; GFX900-NEXT:    global_load_dwordx2 v[6:7], v[0:1], off
+; GFX900-NEXT:    global_load_dwordx2 v[10:11], v[0:1], off offset:2048
 ; GFX900-NEXT:    s_waitcnt vmcnt(6)
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v7, v5
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v8, v6, vcc
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v4, v2
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v5, v3, vcc
 ; GFX900-NEXT:    s_waitcnt vmcnt(5)
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v11, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v12, v1, vcc
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v8, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v9, v1, vcc
 ; GFX900-NEXT:    s_waitcnt vmcnt(4)
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v15, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v16, v1, vcc
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v12, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v13, v1, vcc
 ; GFX900-NEXT:    s_waitcnt vmcnt(3)
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v17, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v18, v1, vcc
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v14, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v15, v1, vcc
 ; GFX900-NEXT:    s_waitcnt vmcnt(2)
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v19, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v20, v1, vcc
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v16, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v17, v1, vcc
 ; GFX900-NEXT:    s_waitcnt vmcnt(1)
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v9, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v10, v1, vcc
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v6, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v7, v1, vcc
 ; GFX900-NEXT:    s_waitcnt vmcnt(0)
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v13, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v14, v1, vcc
-; GFX900-NEXT:    global_store_dwordx2 v[3:4], v[0:1], off
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v10, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v11, v1, vcc
+; GFX900-NEXT:    global_store_dwordx2 v18, v[0:1], s[34:35]
 ; GFX900-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: clmem_read_simplified:
@@ -184,54 +184,54 @@ define amdgpu_kernel void @clmem_read_simplified(ptr addrspace(1)  %buffer) {
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v2, 7, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX10-NEXT:    v_and_b32_e32 v0, 0xff, v0
-; GFX10-NEXT:    v_and_b32_e32 v2, 0xffff8000, v2
+; GFX10-NEXT:    v_and_b32_e32 v20, 0xffff8000, v2
 ; GFX10-NEXT:    v_lshlrev_b64 v[0:1], 3, v[0:1]
-; GFX10-NEXT:    v_add_co_u32 v2, s0, s34, v2
+; GFX10-NEXT:    v_add_co_u32 v2, s0, s34, v20
 ; GFX10-NEXT:    v_add_co_ci_u32_e64 v3, s0, s35, 0, s0
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v2, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v4, vcc_lo, v0, 0x1000
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, 0, v1, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v10, vcc_lo, v0, 0x2000
+; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, v0, 0x1000
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, 0, v1, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v8, vcc_lo, v0, 0x2000
 ; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    global_load_dwordx2 v[6:7], v[0:1], off
-; GFX10-NEXT:    global_load_dwordx2 v[8:9], v[4:5], off offset:-2048
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v11, vcc_lo, 0, v1, vcc_lo
+; GFX10-NEXT:    global_load_dwordx2 v[4:5], v[0:1], off
+; GFX10-NEXT:    global_load_dwordx2 v[6:7], v[2:3], off offset:-2048
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v9, vcc_lo, 0, v1, vcc_lo
 ; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    global_load_dwordx2 v[12:13], v[4:5], off
-; GFX10-NEXT:    global_load_dwordx2 v[14:15], v[10:11], off offset:-2048
-; GFX10-NEXT:    v_add_co_u32 v4, vcc_lo, v0, 0x3000
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, 0, v1, vcc_lo
+; GFX10-NEXT:    global_load_dwordx2 v[10:11], v[2:3], off
+; GFX10-NEXT:    global_load_dwordx2 v[12:13], v[8:9], off offset:-2048
+; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, v0, 0x3000
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, 0, v1, vcc_lo
 ; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    global_load_dwordx2 v[16:17], v[10:11], off
-; GFX10-NEXT:    global_load_dwordx2 v[18:19], v[4:5], off offset:-2048
+; GFX10-NEXT:    global_load_dwordx2 v[14:15], v[8:9], off
+; GFX10-NEXT:    global_load_dwordx2 v[16:17], v[2:3], off offset:-2048
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, 0x3800, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, 0, v1, vcc_lo
 ; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    global_load_dwordx2 v[10:11], v[4:5], off
-; GFX10-NEXT:    global_load_dwordx2 v[20:21], v[0:1], off
+; GFX10-NEXT:    global_load_dwordx2 v[8:9], v[2:3], off
+; GFX10-NEXT:    global_load_dwordx2 v[18:19], v[0:1], off
 ; GFX10-NEXT:    s_waitcnt vmcnt(6)
-; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v8, v6
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v9, v7, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v6, v4
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v7, v5, vcc_lo
 ; GFX10-NEXT:    s_waitcnt vmcnt(5)
+; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v10, v0
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v11, v1, vcc_lo
+; GFX10-NEXT:    s_waitcnt vmcnt(4)
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v12, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v13, v1, vcc_lo
-; GFX10-NEXT:    s_waitcnt vmcnt(4)
+; GFX10-NEXT:    s_waitcnt vmcnt(3)
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v14, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v15, v1, vcc_lo
-; GFX10-NEXT:    s_waitcnt vmcnt(3)
+; GFX10-NEXT:    s_waitcnt vmcnt(2)
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v16, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v17, v1, vcc_lo
-; GFX10-NEXT:    s_waitcnt vmcnt(2)
-; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v18, v0
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v19, v1, vcc_lo
 ; GFX10-NEXT:    s_waitcnt vmcnt(1)
-; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v10, v0
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v11, v1, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v8, v0
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v9, v1, vcc_lo
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v20, v0
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v21, v1, vcc_lo
-; GFX10-NEXT:    global_store_dwordx2 v[2:3], v[0:1], off
+; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v18, v0
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v19, v1, vcc_lo
+; GFX10-NEXT:    global_store_dwordx2 v20, v[0:1], s[34:35]
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX90A-LABEL: clmem_read_simplified:
@@ -256,53 +256,53 @@ define amdgpu_kernel void @clmem_read_simplified(ptr addrspace(1)  %buffer) {
 ; GFX90A-NEXT:    s_swappc_b64 s[30:31], s[4:5]
 ; GFX90A-NEXT:    v_and_b32_e32 v2, 0xff, v0
 ; GFX90A-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
-; GFX90A-NEXT:    v_and_b32_e32 v0, 0xffff8000, v0
+; GFX90A-NEXT:    v_and_b32_e32 v18, 0xffff8000, v0
 ; GFX90A-NEXT:    v_mov_b32_e32 v3, 0
-; GFX90A-NEXT:    v_mov_b32_e32 v1, s35
-; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, s34, v0
-; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
-; GFX90A-NEXT:    v_lshlrev_b64 v[2:3], 3, v[2:3]
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v0, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v1, v3, vcc
+; GFX90A-NEXT:    v_mov_b32_e32 v0, s35
+; GFX90A-NEXT:    v_add_co_u32_e32 v4, vcc, s34, v18
+; GFX90A-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v0, vcc
+; GFX90A-NEXT:    v_lshlrev_b64 v[0:1], 3, v[2:3]
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v4, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v5, v1, vcc
 ; GFX90A-NEXT:    s_movk_i32 s1, 0x2000
-; GFX90A-NEXT:    global_load_dwordx2 v[4:5], v[2:3], off
-; GFX90A-NEXT:    global_load_dwordx2 v[6:7], v[2:3], off offset:2048
-; GFX90A-NEXT:    v_add_co_u32_e32 v8, vcc, s1, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v9, vcc, 0, v3, vcc
-; GFX90A-NEXT:    global_load_dwordx2 v[10:11], v[8:9], off offset:-4096
+; GFX90A-NEXT:    global_load_dwordx2 v[2:3], v[0:1], off
+; GFX90A-NEXT:    global_load_dwordx2 v[4:5], v[0:1], off offset:2048
+; GFX90A-NEXT:    v_add_co_u32_e32 v6, vcc, s1, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v7, vcc, 0, v1, vcc
+; GFX90A-NEXT:    global_load_dwordx2 v[8:9], v[6:7], off offset:-4096
 ; GFX90A-NEXT:    s_movk_i32 s0, 0x1000
-; GFX90A-NEXT:    v_add_co_u32_e32 v12, vcc, s0, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v13, vcc, 0, v3, vcc
-; GFX90A-NEXT:    global_load_dwordx2 v[14:15], v[12:13], off offset:2048
-; GFX90A-NEXT:    global_load_dwordx2 v[16:17], v[8:9], off
-; GFX90A-NEXT:    global_load_dwordx2 v[18:19], v[8:9], off offset:2048
+; GFX90A-NEXT:    v_add_co_u32_e32 v10, vcc, s0, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v11, vcc, 0, v1, vcc
+; GFX90A-NEXT:    global_load_dwordx2 v[12:13], v[10:11], off offset:2048
+; GFX90A-NEXT:    global_load_dwordx2 v[14:15], v[6:7], off
+; GFX90A-NEXT:    global_load_dwordx2 v[16:17], v[6:7], off offset:2048
 ; GFX90A-NEXT:    s_movk_i32 s0, 0x3000
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, s0, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, 0, v3, vcc
-; GFX90A-NEXT:    global_load_dwordx2 v[8:9], v[2:3], off
-; GFX90A-NEXT:    global_load_dwordx2 v[12:13], v[2:3], off offset:2048
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, s0, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
+; GFX90A-NEXT:    global_load_dwordx2 v[6:7], v[0:1], off
+; GFX90A-NEXT:    global_load_dwordx2 v[10:11], v[0:1], off offset:2048
 ; GFX90A-NEXT:    s_waitcnt vmcnt(6)
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v6, v4
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v7, v5, vcc
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v4, v2
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v5, v3, vcc
 ; GFX90A-NEXT:    s_waitcnt vmcnt(5)
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v10, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v11, v3, vcc
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v8, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v9, v1, vcc
 ; GFX90A-NEXT:    s_waitcnt vmcnt(4)
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v14, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v15, v3, vcc
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v12, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v13, v1, vcc
 ; GFX90A-NEXT:    s_waitcnt vmcnt(3)
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v16, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v17, v3, vcc
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v14, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v15, v1, vcc
 ; GFX90A-NEXT:    s_waitcnt vmcnt(2)
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v18, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v19, v3, vcc
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v16, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v17, v1, vcc
 ; GFX90A-NEXT:    s_waitcnt vmcnt(1)
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v8, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v9, v3, vcc
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v6, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v7, v1, vcc
 ; GFX90A-NEXT:    s_waitcnt vmcnt(0)
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v12, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v13, v3, vcc
-; GFX90A-NEXT:    global_store_dwordx2 v[0:1], v[2:3], off
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v10, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v11, v1, vcc
+; GFX90A-NEXT:    global_store_dwordx2 v18, v[0:1], s[34:35]
 ; GFX90A-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: clmem_read_simplified:
@@ -319,61 +319,61 @@ define amdgpu_kernel void @clmem_read_simplified(ptr addrspace(1)  %buffer) {
 ; GFX11-NEXT:    v_dual_mov_b32 v1, 0 :: v_dual_lshlrev_b32 v2, 7, v0
 ; GFX11-NEXT:    v_and_b32_e32 v0, 0xff, v0
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_and_b32_e32 v2, 0xffff8000, v2
+; GFX11-NEXT:    v_and_b32_e32 v16, 0xffff8000, v2
 ; GFX11-NEXT:    v_lshlrev_b64 v[0:1], 3, v[0:1]
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11-NEXT:    v_add_co_u32 v2, s0, s34, v2
+; GFX11-NEXT:    v_add_co_u32 v2, s0, s34, v16
 ; GFX11-NEXT:    v_add_co_ci_u32_e64 v3, null, s35, 0, s0
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
 ; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, v2, v0
 ; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
 ; GFX11-NEXT:    s_clause 0x1
-; GFX11-NEXT:    global_load_b64 v[4:5], v[0:1], off
-; GFX11-NEXT:    global_load_b64 v[6:7], v[0:1], off offset:2048
-; GFX11-NEXT:    v_add_co_u32 v8, vcc_lo, v0, 0x2000
+; GFX11-NEXT:    global_load_b64 v[2:3], v[0:1], off
+; GFX11-NEXT:    global_load_b64 v[4:5], v[0:1], off offset:2048
+; GFX11-NEXT:    v_add_co_u32 v6, vcc_lo, v0, 0x2000
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v7, vcc_lo, 0, v1, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v8, vcc_lo, 0x1000, v0
 ; GFX11-NEXT:    v_add_co_ci_u32_e32 v9, vcc_lo, 0, v1, vcc_lo
-; GFX11-NEXT:    v_add_co_u32 v10, vcc_lo, 0x1000, v0
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v11, vcc_lo, 0, v1, vcc_lo
 ; GFX11-NEXT:    s_clause 0x1
-; GFX11-NEXT:    global_load_b64 v[12:13], v[8:9], off offset:-4096
-; GFX11-NEXT:    global_load_b64 v[10:11], v[10:11], off offset:2048
-; GFX11-NEXT:    v_add_co_u32 v14, vcc_lo, 0x2000, v0
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v15, vcc_lo, 0, v1, vcc_lo
-; GFX11-NEXT:    global_load_b64 v[8:9], v[8:9], off
+; GFX11-NEXT:    global_load_b64 v[10:11], v[6:7], off offset:-4096
+; GFX11-NEXT:    global_load_b64 v[8:9], v[8:9], off offset:2048
+; GFX11-NEXT:    v_add_co_u32 v12, vcc_lo, 0x2000, v0
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v13, vcc_lo, 0, v1, vcc_lo
+; GFX11-NEXT:    global_load_b64 v[6:7], v[6:7], off
 ; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, 0x3000, v0
 ; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, 0, v1, vcc_lo
 ; GFX11-NEXT:    s_clause 0x2
-; GFX11-NEXT:    global_load_b64 v[14:15], v[14:15], off offset:2048
-; GFX11-NEXT:    global_load_b64 v[16:17], v[0:1], off
+; GFX11-NEXT:    global_load_b64 v[12:13], v[12:13], off offset:2048
+; GFX11-NEXT:    global_load_b64 v[14:15], v[0:1], off
 ; GFX11-NEXT:    global_load_b64 v[0:1], v[0:1], off offset:2048
 ; GFX11-NEXT:    s_waitcnt vmcnt(6)
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v6, v4
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v7, v5, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, v4, v2
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v5, v3, vcc_lo
 ; GFX11-NEXT:    s_waitcnt vmcnt(5)
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v12, v4
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v13, v5, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, v10, v2
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v11, v3, vcc_lo
 ; GFX11-NEXT:    s_waitcnt vmcnt(4)
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v10, v4
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v11, v5, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, v8, v2
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v9, v3, vcc_lo
 ; GFX11-NEXT:    s_waitcnt vmcnt(3)
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v8, v4
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v9, v5, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, v6, v2
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v7, v3, vcc_lo
 ; GFX11-NEXT:    s_waitcnt vmcnt(2)
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v14, v4
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v15, v5, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, v12, v2
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v13, v3, vcc_lo
 ; GFX11-NEXT:    s_waitcnt vmcnt(1)
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v16, v4
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v17, v5, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, v14, v2
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v15, v3, vcc_lo
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, v0, v4
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v1, v5, vcc_lo
-; GFX11-NEXT:    global_store_b64 v[2:3], v[0:1], off
+; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, v0, v2
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v1, v3, vcc_lo
+; GFX11-NEXT:    global_store_b64 v16, v[0:1], s[34:35]
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
 entry:
@@ -440,7 +440,7 @@ define hidden amdgpu_kernel void @clmem_read(ptr addrspace(1)  %buffer) {
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v0, 17, v0
 ; GFX8-NEXT:    v_lshlrev_b64 v[1:2], 3, v[1:2]
 ; GFX8-NEXT:    v_and_b32_e32 v0, 0xfe000000, v0
-; GFX8-NEXT:    v_or_b32_e32 v1, v0, v1
+; GFX8-NEXT:    v_or_b32_e32 v1, v1, v0
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s35
 ; GFX8-NEXT:    v_add_u32_e32 v1, vcc, s34, v1
 ; GFX8-NEXT:    v_addc_u32_e32 v2, vcc, v2, v3, vcc
@@ -568,7 +568,7 @@ define hidden amdgpu_kernel void @clmem_read(ptr addrspace(1)  %buffer) {
 ; GFX900-NEXT:    v_lshlrev_b32_e32 v0, 17, v0
 ; GFX900-NEXT:    v_lshlrev_b64 v[1:2], 3, v[1:2]
 ; GFX900-NEXT:    v_and_b32_e32 v0, 0xfe000000, v0
-; GFX900-NEXT:    v_or_b32_e32 v1, v0, v1
+; GFX900-NEXT:    v_or_b32_e32 v1, v1, v0
 ; GFX900-NEXT:    v_mov_b32_e32 v3, s35
 ; GFX900-NEXT:    v_add_co_u32_e32 v1, vcc, s34, v1
 ; GFX900-NEXT:    v_addc_co_u32_e32 v2, vcc, v2, v3, vcc
@@ -689,7 +689,7 @@ define hidden amdgpu_kernel void @clmem_read(ptr addrspace(1)  %buffer) {
 ; GFX10-NEXT:    s_movk_i32 s1, 0x7f
 ; GFX10-NEXT:    v_lshlrev_b64 v[1:2], 3, v[1:2]
 ; GFX10-NEXT:    v_and_b32_e32 v0, 0xfe000000, v0
-; GFX10-NEXT:    v_or_b32_e32 v1, v0, v1
+; GFX10-NEXT:    v_or_b32_e32 v1, v1, v0
 ; GFX10-NEXT:    v_add_co_u32 v1, vcc_lo, v1, s34
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v2, vcc_lo, s35, v2, vcc_lo
 ; GFX10-NEXT:    v_add_co_u32 v1, vcc_lo, 0x5000, v1
@@ -800,7 +800,7 @@ define hidden amdgpu_kernel void @clmem_read(ptr addrspace(1)  %buffer) {
 ; GFX90A-NEXT:    v_lshlrev_b32_e32 v0, 17, v0
 ; GFX90A-NEXT:    v_and_b32_e32 v0, 0xfe000000, v0
 ; GFX90A-NEXT:    v_lshlrev_b64 v[2:3], 3, v[2:3]
-; GFX90A-NEXT:    v_or_b32_e32 v1, v0, v2
+; GFX90A-NEXT:    v_or_b32_e32 v1, v2, v0
 ; GFX90A-NEXT:    v_mov_b32_e32 v2, s35
 ; GFX90A-NEXT:    v_add_co_u32_e32 v1, vcc, s34, v1
 ; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v3, v2, vcc
@@ -910,7 +910,7 @@ define hidden amdgpu_kernel void @clmem_read(ptr addrspace(1)  %buffer) {
 ; GFX11-NEXT:    v_lshlrev_b64 v[1:2], 3, v[1:2]
 ; GFX11-NEXT:    v_and_b32_e32 v0, 0xfe000000, v0
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11-NEXT:    v_or_b32_e32 v1, v0, v1
+; GFX11-NEXT:    v_or_b32_e32 v1, v1, v0
 ; GFX11-NEXT:    v_add_co_u32 v1, vcc_lo, v1, s34
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_4) | instskip(NEXT) | instid1(VALU_DEP_2)
 ; GFX11-NEXT:    v_add_co_ci_u32_e32 v2, vcc_lo, s35, v2, vcc_lo
@@ -1222,40 +1222,40 @@ define amdgpu_kernel void @Address32(ptr addrspace(1) %buffer) {
 ; GFX900-NEXT:    s_swappc_b64 s[30:31], s[4:5]
 ; GFX900-NEXT:    v_and_b32_e32 v1, 0xff, v0
 ; GFX900-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
+; GFX900-NEXT:    v_and_b32_e32 v4, 0xffff8000, v0
 ; GFX900-NEXT:    v_mov_b32_e32 v2, 0
-; GFX900-NEXT:    v_and_b32_e32 v0, 0xffff8000, v0
-; GFX900-NEXT:    v_mov_b32_e32 v4, s35
-; GFX900-NEXT:    v_add_co_u32_e32 v3, vcc, s34, v0
+; GFX900-NEXT:    v_mov_b32_e32 v0, s35
+; GFX900-NEXT:    v_add_co_u32_e32 v3, vcc, s34, v4
+; GFX900-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v0, vcc
 ; GFX900-NEXT:    v_lshlrev_b64 v[0:1], 2, v[1:2]
-; GFX900-NEXT:    v_addc_co_u32_e32 v4, vcc, 0, v4, vcc
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v3, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v4, v1, vcc
 ; GFX900-NEXT:    s_movk_i32 s0, 0x1000
-; GFX900-NEXT:    v_add_co_u32_e32 v5, vcc, s0, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v6, vcc, 0, v1, vcc
-; GFX900-NEXT:    global_load_dword v2, v[0:1], off
-; GFX900-NEXT:    global_load_dword v7, v[0:1], off offset:1024
-; GFX900-NEXT:    global_load_dword v8, v[0:1], off offset:2048
-; GFX900-NEXT:    global_load_dword v9, v[0:1], off offset:3072
-; GFX900-NEXT:    global_load_dword v10, v[5:6], off
-; GFX900-NEXT:    global_load_dword v11, v[5:6], off offset:1024
-; GFX900-NEXT:    global_load_dword v12, v[5:6], off offset:2048
-; GFX900-NEXT:    global_load_dword v13, v[5:6], off offset:3072
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, 0x2000, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v3, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v5, v1, vcc
+; GFX900-NEXT:    v_add_co_u32_e32 v2, vcc, s0, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v3, vcc, 0, v1, vcc
 ; GFX900-NEXT:    global_load_dword v5, v[0:1], off
 ; GFX900-NEXT:    global_load_dword v6, v[0:1], off offset:1024
+; GFX900-NEXT:    global_load_dword v7, v[0:1], off offset:2048
+; GFX900-NEXT:    global_load_dword v8, v[0:1], off offset:3072
+; GFX900-NEXT:    global_load_dword v9, v[2:3], off
+; GFX900-NEXT:    global_load_dword v10, v[2:3], off offset:1024
+; GFX900-NEXT:    global_load_dword v11, v[2:3], off offset:2048
+; GFX900-NEXT:    global_load_dword v12, v[2:3], off offset:3072
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, 0x2000, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
+; GFX900-NEXT:    global_load_dword v2, v[0:1], off
+; GFX900-NEXT:    global_load_dword v3, v[0:1], off offset:1024
 ; GFX900-NEXT:    s_waitcnt vmcnt(8)
-; GFX900-NEXT:    v_add_u32_e32 v0, v7, v2
+; GFX900-NEXT:    v_add_u32_e32 v0, v6, v5
 ; GFX900-NEXT:    s_waitcnt vmcnt(6)
-; GFX900-NEXT:    v_add3_u32 v0, v8, v0, v9
+; GFX900-NEXT:    v_add3_u32 v0, v7, v0, v8
 ; GFX900-NEXT:    s_waitcnt vmcnt(4)
-; GFX900-NEXT:    v_add3_u32 v0, v10, v0, v11
+; GFX900-NEXT:    v_add3_u32 v0, v9, v0, v10
 ; GFX900-NEXT:    s_waitcnt vmcnt(2)
-; GFX900-NEXT:    v_add3_u32 v0, v12, v0, v13
+; GFX900-NEXT:    v_add3_u32 v0, v11, v0, v12
 ; GFX900-NEXT:    s_waitcnt vmcnt(0)
-; GFX900-NEXT:    v_add3_u32 v0, v5, v0, v6
-; GFX900-NEXT:    global_store_dword v[3:4], v0, off
+; GFX900-NEXT:    v_add3_u32 v0, v2, v0, v3
+; GFX900-NEXT:    global_store_dword v4, v0, s[34:35]
 ; GFX900-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: Address32:
@@ -1281,48 +1281,48 @@ define amdgpu_kernel void @Address32(ptr addrspace(1) %buffer) {
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v2, 7, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX10-NEXT:    v_and_b32_e32 v0, 0xff, v0
-; GFX10-NEXT:    v_and_b32_e32 v2, 0xffff8000, v2
+; GFX10-NEXT:    v_and_b32_e32 v8, 0xffff8000, v2
 ; GFX10-NEXT:    v_lshlrev_b64 v[0:1], 2, v[0:1]
-; GFX10-NEXT:    v_add_co_u32 v2, s0, s34, v2
+; GFX10-NEXT:    v_add_co_u32 v2, s0, s34, v8
 ; GFX10-NEXT:    v_add_co_ci_u32_e64 v3, s0, s35, 0, s0
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v2, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v4, vcc_lo, 0x800, v0
+; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, 0x800, v0
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, 0, v1, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v4, vcc_lo, v0, 0x1000
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, 0, v1, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v6, vcc_lo, v0, 0x1000
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v7, vcc_lo, 0, v1, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v8, vcc_lo, 0x1000, v0
+; GFX10-NEXT:    v_add_co_u32 v6, vcc_lo, 0x1000, v0
 ; GFX10-NEXT:    s_clause 0x4
-; GFX10-NEXT:    global_load_dword v10, v[0:1], off
-; GFX10-NEXT:    global_load_dword v11, v[0:1], off offset:1024
-; GFX10-NEXT:    global_load_dword v12, v[4:5], off offset:1024
-; GFX10-NEXT:    global_load_dword v13, v[6:7], off offset:-2048
-; GFX10-NEXT:    global_load_dword v14, v[6:7], off
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v9, vcc_lo, 0, v1, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v4, vcc_lo, 0x1800, v0
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, 0, v1, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v6, vcc_lo, v0, 0x2000
+; GFX10-NEXT:    global_load_dword v9, v[0:1], off
+; GFX10-NEXT:    global_load_dword v10, v[0:1], off offset:1024
+; GFX10-NEXT:    global_load_dword v11, v[2:3], off offset:1024
+; GFX10-NEXT:    global_load_dword v12, v[4:5], off offset:-2048
+; GFX10-NEXT:    global_load_dword v13, v[4:5], off
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v7, vcc_lo, 0, v1, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, 0x1800, v0
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, 0, v1, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v4, vcc_lo, v0, 0x2000
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, 0, v1, vcc_lo
 ; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    global_load_dword v15, v[8:9], off offset:1024
-; GFX10-NEXT:    global_load_dword v16, v[4:5], off offset:1024
+; GFX10-NEXT:    global_load_dword v14, v[6:7], off offset:1024
+; GFX10-NEXT:    global_load_dword v15, v[2:3], off offset:1024
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, 0x2000, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, 0, v1, vcc_lo
 ; GFX10-NEXT:    s_clause 0x2
-; GFX10-NEXT:    global_load_dword v4, v[6:7], off offset:-2048
-; GFX10-NEXT:    global_load_dword v5, v[6:7], off
-; GFX10-NEXT:    global_load_dword v8, v[0:1], off offset:1024
+; GFX10-NEXT:    global_load_dword v2, v[4:5], off offset:-2048
+; GFX10-NEXT:    global_load_dword v3, v[4:5], off
+; GFX10-NEXT:    global_load_dword v6, v[0:1], off offset:1024
 ; GFX10-NEXT:    s_waitcnt vmcnt(8)
-; GFX10-NEXT:    v_add_nc_u32_e32 v0, v11, v10
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, v10, v9
 ; GFX10-NEXT:    s_waitcnt vmcnt(6)
-; GFX10-NEXT:    v_add3_u32 v0, v13, v0, v12
+; GFX10-NEXT:    v_add3_u32 v0, v12, v0, v11
 ; GFX10-NEXT:    s_waitcnt vmcnt(4)
-; GFX10-NEXT:    v_add3_u32 v0, v14, v0, v15
+; GFX10-NEXT:    v_add3_u32 v0, v13, v0, v14
 ; GFX10-NEXT:    s_waitcnt vmcnt(2)
-; GFX10-NEXT:    v_add3_u32 v0, v4, v0, v16
+; GFX10-NEXT:    v_add3_u32 v0, v2, v0, v15
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_add3_u32 v0, v5, v0, v8
-; GFX10-NEXT:    global_store_dword v[2:3], v0, off
+; GFX10-NEXT:    v_add3_u32 v0, v3, v0, v6
+; GFX10-NEXT:    global_store_dword v8, v0, s[34:35]
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX90A-LABEL: Address32:
@@ -1347,40 +1347,40 @@ define amdgpu_kernel void @Address32(ptr addrspace(1) %buffer) {
 ; GFX90A-NEXT:    s_swappc_b64 s[30:31], s[4:5]
 ; GFX90A-NEXT:    v_and_b32_e32 v2, 0xff, v0
 ; GFX90A-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
-; GFX90A-NEXT:    v_and_b32_e32 v0, 0xffff8000, v0
+; GFX90A-NEXT:    v_and_b32_e32 v4, 0xffff8000, v0
 ; GFX90A-NEXT:    v_mov_b32_e32 v3, 0
-; GFX90A-NEXT:    v_mov_b32_e32 v1, s35
-; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, s34, v0
-; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
-; GFX90A-NEXT:    v_lshlrev_b64 v[2:3], 2, v[2:3]
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v0, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v1, v3, vcc
+; GFX90A-NEXT:    v_mov_b32_e32 v0, s35
+; GFX90A-NEXT:    v_add_co_u32_e32 v5, vcc, s34, v4
+; GFX90A-NEXT:    v_addc_co_u32_e32 v6, vcc, 0, v0, vcc
+; GFX90A-NEXT:    v_lshlrev_b64 v[0:1], 2, v[2:3]
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v5, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v6, v1, vcc
 ; GFX90A-NEXT:    s_movk_i32 s0, 0x1000
-; GFX90A-NEXT:    v_add_co_u32_e32 v4, vcc, s0, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v3, vcc
-; GFX90A-NEXT:    global_load_dword v6, v[2:3], off
-; GFX90A-NEXT:    global_load_dword v7, v[2:3], off offset:1024
-; GFX90A-NEXT:    global_load_dword v8, v[2:3], off offset:2048
-; GFX90A-NEXT:    global_load_dword v9, v[2:3], off offset:3072
-; GFX90A-NEXT:    global_load_dword v10, v[4:5], off
-; GFX90A-NEXT:    global_load_dword v11, v[4:5], off offset:1024
-; GFX90A-NEXT:    global_load_dword v12, v[4:5], off offset:2048
-; GFX90A-NEXT:    global_load_dword v13, v[4:5], off offset:3072
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, 0x2000, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, 0, v3, vcc
-; GFX90A-NEXT:    global_load_dword v4, v[2:3], off
-; GFX90A-NEXT:    global_load_dword v5, v[2:3], off offset:1024
+; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, s0, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, 0, v1, vcc
+; GFX90A-NEXT:    global_load_dword v5, v[0:1], off
+; GFX90A-NEXT:    global_load_dword v6, v[0:1], off offset:1024
+; GFX90A-NEXT:    global_load_dword v7, v[0:1], off offset:2048
+; GFX90A-NEXT:    global_load_dword v8, v[0:1], off offset:3072
+; GFX90A-NEXT:    global_load_dword v9, v[2:3], off
+; GFX90A-NEXT:    global_load_dword v10, v[2:3], off offset:1024
+; GFX90A-NEXT:    global_load_dword v11, v[2:3], off offset:2048
+; GFX90A-NEXT:    global_load_dword v12, v[2:3], off offset:3072
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, 0x2000, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
+; GFX90A-NEXT:    global_load_dword v2, v[0:1], off
+; GFX90A-NEXT:    global_load_dword v3, v[0:1], off offset:1024
 ; GFX90A-NEXT:    s_waitcnt vmcnt(8)
-; GFX90A-NEXT:    v_add_u32_e32 v2, v7, v6
+; GFX90A-NEXT:    v_add_u32_e32 v0, v6, v5
 ; GFX90A-NEXT:    s_waitcnt vmcnt(6)
-; GFX90A-NEXT:    v_add3_u32 v2, v8, v2, v9
+; GFX90A-NEXT:    v_add3_u32 v0, v7, v0, v8
 ; GFX90A-NEXT:    s_waitcnt vmcnt(4)
-; GFX90A-NEXT:    v_add3_u32 v2, v10, v2, v11
+; GFX90A-NEXT:    v_add3_u32 v0, v9, v0, v10
 ; GFX90A-NEXT:    s_waitcnt vmcnt(2)
-; GFX90A-NEXT:    v_add3_u32 v2, v12, v2, v13
+; GFX90A-NEXT:    v_add3_u32 v0, v11, v0, v12
 ; GFX90A-NEXT:    s_waitcnt vmcnt(0)
-; GFX90A-NEXT:    v_add3_u32 v2, v4, v2, v5
-; GFX90A-NEXT:    global_store_dword v[0:1], v2, off
+; GFX90A-NEXT:    v_add3_u32 v0, v2, v0, v3
+; GFX90A-NEXT:    global_store_dword v4, v0, s[34:35]
 ; GFX90A-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: Address32:
@@ -1397,46 +1397,46 @@ define amdgpu_kernel void @Address32(ptr addrspace(1) %buffer) {
 ; GFX11-NEXT:    v_dual_mov_b32 v1, 0 :: v_dual_lshlrev_b32 v2, 7, v0
 ; GFX11-NEXT:    v_and_b32_e32 v0, 0xff, v0
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_and_b32_e32 v2, 0xffff8000, v2
+; GFX11-NEXT:    v_and_b32_e32 v6, 0xffff8000, v2
 ; GFX11-NEXT:    v_lshlrev_b64 v[0:1], 2, v[0:1]
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11-NEXT:    v_add_co_u32 v2, s0, s34, v2
+; GFX11-NEXT:    v_add_co_u32 v2, s0, s34, v6
 ; GFX11-NEXT:    v_add_co_ci_u32_e64 v3, null, s35, 0, s0
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
 ; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, v2, v0
 ; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
 ; GFX11-NEXT:    s_clause 0x1
-; GFX11-NEXT:    global_load_b32 v8, v[0:1], off
-; GFX11-NEXT:    global_load_b32 v9, v[0:1], off offset:1024
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, 0x1000, v0
+; GFX11-NEXT:    global_load_b32 v7, v[0:1], off
+; GFX11-NEXT:    global_load_b32 v8, v[0:1], off offset:1024
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, 0x1000, v0
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, 0, v1, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v0, 0x2000
 ; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, 0, v1, vcc_lo
-; GFX11-NEXT:    v_add_co_u32 v6, vcc_lo, v0, 0x2000
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v7, vcc_lo, 0, v1, vcc_lo
 ; GFX11-NEXT:    s_clause 0x5
-; GFX11-NEXT:    global_load_b32 v10, v[0:1], off offset:2048
-; GFX11-NEXT:    global_load_b32 v11, v[0:1], off offset:3072
-; GFX11-NEXT:    global_load_b32 v12, v[6:7], off offset:-4096
-; GFX11-NEXT:    global_load_b32 v13, v[4:5], off offset:1024
-; GFX11-NEXT:    global_load_b32 v14, v[4:5], off offset:2048
-; GFX11-NEXT:    global_load_b32 v4, v[4:5], off offset:3072
+; GFX11-NEXT:    global_load_b32 v9, v[0:1], off offset:2048
+; GFX11-NEXT:    global_load_b32 v10, v[0:1], off offset:3072
+; GFX11-NEXT:    global_load_b32 v11, v[4:5], off offset:-4096
+; GFX11-NEXT:    global_load_b32 v12, v[2:3], off offset:1024
+; GFX11-NEXT:    global_load_b32 v13, v[2:3], off offset:2048
+; GFX11-NEXT:    global_load_b32 v2, v[2:3], off offset:3072
 ; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, 0x2000, v0
 ; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, 0, v1, vcc_lo
 ; GFX11-NEXT:    s_clause 0x1
-; GFX11-NEXT:    global_load_b32 v5, v[6:7], off
+; GFX11-NEXT:    global_load_b32 v3, v[4:5], off
 ; GFX11-NEXT:    global_load_b32 v0, v[0:1], off offset:1024
 ; GFX11-NEXT:    s_waitcnt vmcnt(8)
-; GFX11-NEXT:    v_add_nc_u32_e32 v1, v9, v8
+; GFX11-NEXT:    v_add_nc_u32_e32 v1, v8, v7
 ; GFX11-NEXT:    s_waitcnt vmcnt(6)
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
-; GFX11-NEXT:    v_add3_u32 v1, v10, v1, v11
+; GFX11-NEXT:    v_add3_u32 v1, v9, v1, v10
 ; GFX11-NEXT:    s_waitcnt vmcnt(4)
-; GFX11-NEXT:    v_add3_u32 v1, v12, v1, v13
+; GFX11-NEXT:    v_add3_u32 v1, v11, v1, v12
 ; GFX11-NEXT:    s_waitcnt vmcnt(2)
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
-; GFX11-NEXT:    v_add3_u32 v1, v14, v1, v4
+; GFX11-NEXT:    v_add3_u32 v1, v13, v1, v2
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_add3_u32 v0, v5, v1, v0
-; GFX11-NEXT:    global_store_b32 v[2:3], v0, off
+; GFX11-NEXT:    v_add3_u32 v0, v3, v1, v0
+; GFX11-NEXT:    global_store_b32 v6, v0, s[34:35]
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
 entry:
@@ -1566,32 +1566,32 @@ define amdgpu_kernel void @Offset64(ptr addrspace(1)  %buffer) {
 ; GFX900-NEXT:    s_swappc_b64 s[30:31], s[4:5]
 ; GFX900-NEXT:    v_and_b32_e32 v1, 0xff, v0
 ; GFX900-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
+; GFX900-NEXT:    v_and_b32_e32 v12, 0xffff8000, v0
 ; GFX900-NEXT:    v_mov_b32_e32 v2, 0
-; GFX900-NEXT:    v_and_b32_e32 v0, 0xffff8000, v0
-; GFX900-NEXT:    v_mov_b32_e32 v4, s35
-; GFX900-NEXT:    v_add_co_u32_e32 v3, vcc, s34, v0
+; GFX900-NEXT:    v_mov_b32_e32 v0, s35
+; GFX900-NEXT:    v_add_co_u32_e32 v3, vcc, s34, v12
+; GFX900-NEXT:    v_addc_co_u32_e32 v4, vcc, 0, v0, vcc
 ; GFX900-NEXT:    v_lshlrev_b64 v[0:1], 3, v[1:2]
-; GFX900-NEXT:    v_addc_co_u32_e32 v4, vcc, 0, v4, vcc
+; GFX900-NEXT:    s_movk_i32 s0, 0xf000
 ; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v3, v0
 ; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v4, v1, vcc
-; GFX900-NEXT:    v_add_co_u32_e32 v7, vcc, 0, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v8, vcc, 1, v1, vcc
-; GFX900-NEXT:    global_load_dwordx2 v[5:6], v[0:1], off
-; GFX900-NEXT:    global_load_dwordx2 v[9:10], v[7:8], off offset:-4096
-; GFX900-NEXT:    s_movk_i32 s0, 0xf000
+; GFX900-NEXT:    v_add_co_u32_e32 v4, vcc, 0, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v5, vcc, 1, v1, vcc
+; GFX900-NEXT:    global_load_dwordx2 v[2:3], v[0:1], off
+; GFX900-NEXT:    global_load_dwordx2 v[6:7], v[4:5], off offset:-4096
 ; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, s0, v0
 ; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
-; GFX900-NEXT:    global_load_dwordx2 v[11:12], v[7:8], off
-; GFX900-NEXT:    global_load_dwordx2 v[13:14], v[0:1], off offset:2048
+; GFX900-NEXT:    global_load_dwordx2 v[8:9], v[4:5], off
+; GFX900-NEXT:    global_load_dwordx2 v[10:11], v[0:1], off offset:2048
 ; GFX900-NEXT:    s_waitcnt vmcnt(2)
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v9, v5
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v10, v6, vcc
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v6, v2
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v7, v3, vcc
 ; GFX900-NEXT:    s_waitcnt vmcnt(0)
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v13, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v14, v1, vcc
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v11, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v12, v1, vcc
-; GFX900-NEXT:    global_store_dwordx2 v[3:4], v[0:1], off
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v10, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v11, v1, vcc
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v8, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v9, v1, vcc
+; GFX900-NEXT:    global_store_dwordx2 v12, v[0:1], s[34:35]
 ; GFX900-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: Offset64:
@@ -1617,32 +1617,32 @@ define amdgpu_kernel void @Offset64(ptr addrspace(1)  %buffer) {
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v2, 7, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX10-NEXT:    v_and_b32_e32 v0, 0xff, v0
-; GFX10-NEXT:    v_and_b32_e32 v2, 0xffff8000, v2
+; GFX10-NEXT:    v_and_b32_e32 v12, 0xffff8000, v2
 ; GFX10-NEXT:    v_lshlrev_b64 v[0:1], 3, v[0:1]
-; GFX10-NEXT:    v_add_co_u32 v2, s0, s34, v2
+; GFX10-NEXT:    v_add_co_u32 v2, s0, s34, v12
 ; GFX10-NEXT:    v_add_co_ci_u32_e64 v3, s0, s35, 0, s0
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v2, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v4, vcc_lo, v0, 0xfffff800
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, 0, v1, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, v0, 0xfffff800
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, 0, v1, vcc_lo
 ; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    global_load_dwordx2 v[6:7], v[0:1], off
-; GFX10-NEXT:    global_load_dwordx2 v[8:9], v[4:5], off offset:-2048
+; GFX10-NEXT:    global_load_dwordx2 v[4:5], v[0:1], off
+; GFX10-NEXT:    global_load_dwordx2 v[6:7], v[2:3], off offset:-2048
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, 0, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, 1, v1, vcc_lo
 ; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    global_load_dwordx2 v[10:11], v[4:5], off
-; GFX10-NEXT:    global_load_dwordx2 v[12:13], v[0:1], off
+; GFX10-NEXT:    global_load_dwordx2 v[8:9], v[2:3], off
+; GFX10-NEXT:    global_load_dwordx2 v[10:11], v[0:1], off
 ; GFX10-NEXT:    s_waitcnt vmcnt(2)
-; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v8, v6
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v9, v7, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v6, v4
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v7, v5, vcc_lo
 ; GFX10-NEXT:    s_waitcnt vmcnt(1)
+; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v8, v0
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v9, v1, vcc_lo
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v10, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v11, v1, vcc_lo
-; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v12, v0
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v13, v1, vcc_lo
-; GFX10-NEXT:    global_store_dwordx2 v[2:3], v[0:1], off
+; GFX10-NEXT:    global_store_dwordx2 v12, v[0:1], s[34:35]
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX90A-LABEL: Offset64:
@@ -1667,32 +1667,32 @@ define amdgpu_kernel void @Offset64(ptr addrspace(1)  %buffer) {
 ; GFX90A-NEXT:    s_swappc_b64 s[30:31], s[4:5]
 ; GFX90A-NEXT:    v_and_b32_e32 v2, 0xff, v0
 ; GFX90A-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
-; GFX90A-NEXT:    v_and_b32_e32 v0, 0xffff8000, v0
+; GFX90A-NEXT:    v_and_b32_e32 v12, 0xffff8000, v0
 ; GFX90A-NEXT:    v_mov_b32_e32 v3, 0
-; GFX90A-NEXT:    v_mov_b32_e32 v1, s35
-; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, s34, v0
-; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
-; GFX90A-NEXT:    v_lshlrev_b64 v[2:3], 3, v[2:3]
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v0, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v1, v3, vcc
-; GFX90A-NEXT:    v_add_co_u32_e32 v6, vcc, 0, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v7, vcc, 1, v3, vcc
-; GFX90A-NEXT:    global_load_dwordx2 v[4:5], v[2:3], off
-; GFX90A-NEXT:    global_load_dwordx2 v[8:9], v[6:7], off offset:-4096
+; GFX90A-NEXT:    v_mov_b32_e32 v0, s35
+; GFX90A-NEXT:    v_add_co_u32_e32 v4, vcc, s34, v12
+; GFX90A-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v0, vcc
+; GFX90A-NEXT:    v_lshlrev_b64 v[0:1], 3, v[2:3]
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v4, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v5, v1, vcc
+; GFX90A-NEXT:    v_add_co_u32_e32 v4, vcc, 0, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v5, vcc, 1, v1, vcc
+; GFX90A-NEXT:    global_load_dwordx2 v[2:3], v[0:1], off
+; GFX90A-NEXT:    global_load_dwordx2 v[6:7], v[4:5], off offset:-4096
 ; GFX90A-NEXT:    s_movk_i32 s0, 0xf000
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, s0, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, 0, v3, vcc
-; GFX90A-NEXT:    global_load_dwordx2 v[10:11], v[6:7], off
-; GFX90A-NEXT:    global_load_dwordx2 v[12:13], v[2:3], off offset:2048
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, s0, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
+; GFX90A-NEXT:    global_load_dwordx2 v[8:9], v[4:5], off
+; GFX90A-NEXT:    global_load_dwordx2 v[10:11], v[0:1], off offset:2048
 ; GFX90A-NEXT:    s_waitcnt vmcnt(2)
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v8, v4
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v9, v5, vcc
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v6, v2
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v7, v3, vcc
 ; GFX90A-NEXT:    s_waitcnt vmcnt(0)
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v12, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v13, v3, vcc
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v10, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v11, v3, vcc
-; GFX90A-NEXT:    global_store_dwordx2 v[0:1], v[2:3], off
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v10, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v11, v1, vcc
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v8, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v9, v1, vcc
+; GFX90A-NEXT:    global_store_dwordx2 v12, v[0:1], s[34:35]
 ; GFX90A-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: Offset64:
@@ -1709,35 +1709,35 @@ define amdgpu_kernel void @Offset64(ptr addrspace(1)  %buffer) {
 ; GFX11-NEXT:    v_dual_mov_b32 v1, 0 :: v_dual_lshlrev_b32 v2, 7, v0
 ; GFX11-NEXT:    v_and_b32_e32 v0, 0xff, v0
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_and_b32_e32 v2, 0xffff8000, v2
+; GFX11-NEXT:    v_and_b32_e32 v8, 0xffff8000, v2
 ; GFX11-NEXT:    v_lshlrev_b64 v[0:1], 3, v[0:1]
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11-NEXT:    v_add_co_u32 v2, s0, s34, v2
+; GFX11-NEXT:    v_add_co_u32 v2, s0, s34, v8
 ; GFX11-NEXT:    v_add_co_ci_u32_e64 v3, null, s35, 0, s0
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
 ; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, v2, v0
 ; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v6, vcc_lo, v0, 0
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v7, vcc_lo, 1, v1, vcc_lo
-; GFX11-NEXT:    global_load_b64 v[4:5], v[0:1], off
+; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v0, 0
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, 1, v1, vcc_lo
+; GFX11-NEXT:    global_load_b64 v[2:3], v[0:1], off
 ; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, 0xfffff000, v0
 ; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, 0, v1, vcc_lo
 ; GFX11-NEXT:    s_clause 0x2
-; GFX11-NEXT:    global_load_b64 v[8:9], v[6:7], off offset:-4096
-; GFX11-NEXT:    global_load_b64 v[6:7], v[6:7], off
+; GFX11-NEXT:    global_load_b64 v[6:7], v[4:5], off offset:-4096
+; GFX11-NEXT:    global_load_b64 v[4:5], v[4:5], off
 ; GFX11-NEXT:    global_load_b64 v[0:1], v[0:1], off offset:2048
 ; GFX11-NEXT:    s_waitcnt vmcnt(2)
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v8, v4
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v9, v5, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, v6, v2
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v7, v3, vcc_lo
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, v0, v4
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v1, v5, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, v0, v2
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v1, v3, vcc_lo
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, v6, v0
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v7, v1, vcc_lo
-; GFX11-NEXT:    global_store_b64 v[2:3], v[0:1], off
+; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, v4, v0
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v5, v1, vcc_lo
+; GFX11-NEXT:    global_store_b64 v8, v[0:1], s[34:35]
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
 entry:
@@ -1843,27 +1843,27 @@ define amdgpu_kernel void @p32Offset64(ptr addrspace(1)  %buffer) {
 ; GFX900-NEXT:    s_swappc_b64 s[30:31], s[4:5]
 ; GFX900-NEXT:    v_and_b32_e32 v1, 0xff, v0
 ; GFX900-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
+; GFX900-NEXT:    v_and_b32_e32 v6, 0xffff8000, v0
 ; GFX900-NEXT:    v_mov_b32_e32 v2, 0
-; GFX900-NEXT:    v_and_b32_e32 v0, 0xffff8000, v0
-; GFX900-NEXT:    v_mov_b32_e32 v4, s35
-; GFX900-NEXT:    v_add_co_u32_e32 v3, vcc, s34, v0
+; GFX900-NEXT:    v_mov_b32_e32 v0, s35
+; GFX900-NEXT:    v_add_co_u32_e32 v3, vcc, s34, v6
+; GFX900-NEXT:    v_addc_co_u32_e32 v4, vcc, 0, v0, vcc
 ; GFX900-NEXT:    v_lshlrev_b64 v[0:1], 2, v[1:2]
-; GFX900-NEXT:    v_addc_co_u32_e32 v4, vcc, 0, v4, vcc
 ; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v3, v0
 ; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v4, v1, vcc
-; GFX900-NEXT:    v_add_co_u32_e32 v5, vcc, 0x7ffff000, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v6, vcc, 0, v1, vcc
-; GFX900-NEXT:    v_add_co_u32_e32 v7, vcc, 0x80000000, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v8, vcc, 0, v1, vcc
-; GFX900-NEXT:    global_load_dword v2, v[0:1], off
-; GFX900-NEXT:    global_load_dword v9, v[5:6], off offset:2048
-; GFX900-NEXT:    global_load_dword v10, v[5:6], off offset:3072
-; GFX900-NEXT:    global_load_dword v11, v[7:8], off
+; GFX900-NEXT:    v_add_co_u32_e32 v2, vcc, 0x7ffff000, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v3, vcc, 0, v1, vcc
+; GFX900-NEXT:    v_add_co_u32_e32 v4, vcc, 0x80000000, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v1, vcc
+; GFX900-NEXT:    global_load_dword v7, v[0:1], off
+; GFX900-NEXT:    global_load_dword v8, v[2:3], off offset:2048
+; GFX900-NEXT:    global_load_dword v9, v[2:3], off offset:3072
+; GFX900-NEXT:    global_load_dword v10, v[4:5], off
 ; GFX900-NEXT:    s_waitcnt vmcnt(2)
-; GFX900-NEXT:    v_add_u32_e32 v0, v9, v2
+; GFX900-NEXT:    v_add_u32_e32 v0, v8, v7
 ; GFX900-NEXT:    s_waitcnt vmcnt(0)
-; GFX900-NEXT:    v_add3_u32 v0, v10, v0, v11
-; GFX900-NEXT:    global_store_dword v[3:4], v0, off
+; GFX900-NEXT:    v_add3_u32 v0, v9, v0, v10
+; GFX900-NEXT:    global_store_dword v6, v0, s[34:35]
 ; GFX900-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: p32Offset64:
@@ -1889,26 +1889,26 @@ define amdgpu_kernel void @p32Offset64(ptr addrspace(1)  %buffer) {
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v2, 7, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX10-NEXT:    v_and_b32_e32 v0, 0xff, v0
-; GFX10-NEXT:    v_and_b32_e32 v2, 0xffff8000, v2
+; GFX10-NEXT:    v_and_b32_e32 v4, 0xffff8000, v2
 ; GFX10-NEXT:    v_lshlrev_b64 v[0:1], 2, v[0:1]
-; GFX10-NEXT:    v_add_co_u32 v2, s0, s34, v2
+; GFX10-NEXT:    v_add_co_u32 v2, s0, s34, v4
 ; GFX10-NEXT:    v_add_co_ci_u32_e64 v3, s0, s35, 0, s0
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v2, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v4, vcc_lo, v0, 0x80000000
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, 0, v1, vcc_lo
-; GFX10-NEXT:    global_load_dword v6, v[0:1], off
+; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, v0, 0x80000000
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, 0, v1, vcc_lo
+; GFX10-NEXT:    global_load_dword v5, v[0:1], off
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, 0x7ffff800, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, 0, v1, vcc_lo
 ; GFX10-NEXT:    s_clause 0x2
-; GFX10-NEXT:    global_load_dword v7, v[4:5], off offset:-2048
-; GFX10-NEXT:    global_load_dword v8, v[4:5], off
-; GFX10-NEXT:    global_load_dword v9, v[0:1], off offset:1024
+; GFX10-NEXT:    global_load_dword v6, v[2:3], off offset:-2048
+; GFX10-NEXT:    global_load_dword v7, v[2:3], off
+; GFX10-NEXT:    global_load_dword v8, v[0:1], off offset:1024
 ; GFX10-NEXT:    s_waitcnt vmcnt(2)
-; GFX10-NEXT:    v_add_nc_u32_e32 v0, v7, v6
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, v6, v5
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_add3_u32 v0, v9, v0, v8
-; GFX10-NEXT:    global_store_dword v[2:3], v0, off
+; GFX10-NEXT:    v_add3_u32 v0, v8, v0, v7
+; GFX10-NEXT:    global_store_dword v4, v0, s[34:35]
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX90A-LABEL: p32Offset64:
@@ -1933,27 +1933,27 @@ define amdgpu_kernel void @p32Offset64(ptr addrspace(1)  %buffer) {
 ; GFX90A-NEXT:    s_swappc_b64 s[30:31], s[4:5]
 ; GFX90A-NEXT:    v_and_b32_e32 v2, 0xff, v0
 ; GFX90A-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
-; GFX90A-NEXT:    v_and_b32_e32 v0, 0xffff8000, v0
+; GFX90A-NEXT:    v_and_b32_e32 v6, 0xffff8000, v0
 ; GFX90A-NEXT:    v_mov_b32_e32 v3, 0
-; GFX90A-NEXT:    v_mov_b32_e32 v1, s35
-; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, s34, v0
-; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
-; GFX90A-NEXT:    v_lshlrev_b64 v[2:3], 2, v[2:3]
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v0, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v1, v3, vcc
-; GFX90A-NEXT:    v_add_co_u32_e32 v4, vcc, 0x7ffff000, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v3, vcc
-; GFX90A-NEXT:    v_add_co_u32_e32 v6, vcc, 0x80000000, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v7, vcc, 0, v3, vcc
-; GFX90A-NEXT:    global_load_dword v8, v[2:3], off
-; GFX90A-NEXT:    global_load_dword v9, v[4:5], off offset:2048
-; GFX90A-NEXT:    global_load_dword v10, v[4:5], off offset:3072
-; GFX90A-NEXT:    global_load_dword v11, v[6:7], off
+; GFX90A-NEXT:    v_mov_b32_e32 v0, s35
+; GFX90A-NEXT:    v_add_co_u32_e32 v4, vcc, s34, v6
+; GFX90A-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v0, vcc
+; GFX90A-NEXT:    v_lshlrev_b64 v[0:1], 2, v[2:3]
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v4, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v5, v1, vcc
+; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, 0x7ffff000, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, 0, v1, vcc
+; GFX90A-NEXT:    v_add_co_u32_e32 v4, vcc, 0x80000000, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v1, vcc
+; GFX90A-NEXT:    global_load_dword v7, v[0:1], off
+; GFX90A-NEXT:    global_load_dword v8, v[2:3], off offset:2048
+; GFX90A-NEXT:    global_load_dword v9, v[2:3], off offset:3072
+; GFX90A-NEXT:    global_load_dword v10, v[4:5], off
 ; GFX90A-NEXT:    s_waitcnt vmcnt(2)
-; GFX90A-NEXT:    v_add_u32_e32 v2, v9, v8
+; GFX90A-NEXT:    v_add_u32_e32 v0, v8, v7
 ; GFX90A-NEXT:    s_waitcnt vmcnt(0)
-; GFX90A-NEXT:    v_add3_u32 v2, v10, v2, v11
-; GFX90A-NEXT:    global_store_dword v[0:1], v2, off
+; GFX90A-NEXT:    v_add3_u32 v0, v9, v0, v10
+; GFX90A-NEXT:    global_store_dword v6, v0, s[34:35]
 ; GFX90A-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: p32Offset64:
@@ -1970,30 +1970,30 @@ define amdgpu_kernel void @p32Offset64(ptr addrspace(1)  %buffer) {
 ; GFX11-NEXT:    v_dual_mov_b32 v1, 0 :: v_dual_lshlrev_b32 v2, 7, v0
 ; GFX11-NEXT:    v_and_b32_e32 v0, 0xff, v0
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_and_b32_e32 v2, 0xffff8000, v2
+; GFX11-NEXT:    v_and_b32_e32 v6, 0xffff8000, v2
 ; GFX11-NEXT:    v_lshlrev_b64 v[0:1], 2, v[0:1]
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11-NEXT:    v_add_co_u32 v2, s0, s34, v2
+; GFX11-NEXT:    v_add_co_u32 v2, s0, s34, v6
 ; GFX11-NEXT:    v_add_co_ci_u32_e64 v3, null, s35, 0, s0
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
 ; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, v2, v0
 ; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, 0x7ffff000, v0
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, 0x7ffff000, v0
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, 0, v1, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, 0x80000000, v0
 ; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, 0, v1, vcc_lo
-; GFX11-NEXT:    v_add_co_u32 v6, vcc_lo, 0x80000000, v0
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v7, vcc_lo, 0, v1, vcc_lo
 ; GFX11-NEXT:    s_clause 0x3
 ; GFX11-NEXT:    global_load_b32 v0, v[0:1], off
-; GFX11-NEXT:    global_load_b32 v1, v[4:5], off offset:2048
-; GFX11-NEXT:    global_load_b32 v4, v[4:5], off offset:3072
-; GFX11-NEXT:    global_load_b32 v5, v[6:7], off
+; GFX11-NEXT:    global_load_b32 v1, v[2:3], off offset:2048
+; GFX11-NEXT:    global_load_b32 v2, v[2:3], off offset:3072
+; GFX11-NEXT:    global_load_b32 v3, v[4:5], off
 ; GFX11-NEXT:    s_waitcnt vmcnt(2)
 ; GFX11-NEXT:    v_add_nc_u32_e32 v0, v1, v0
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11-NEXT:    v_add3_u32 v0, v4, v0, v5
-; GFX11-NEXT:    global_store_b32 v[2:3], v0, off
+; GFX11-NEXT:    v_add3_u32 v0, v2, v0, v3
+; GFX11-NEXT:    global_store_b32 v6, v0, s[34:35]
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
 entry:
@@ -2053,28 +2053,22 @@ define amdgpu_kernel void @DiffBase(ptr addrspace(1) %buffer1,
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s39
 ; GFX8-NEXT:    v_add_u32_e32 v12, vcc, s38, v2
 ; GFX8-NEXT:    v_addc_u32_e32 v13, vcc, 0, v3, vcc
-; GFX8-NEXT:    s_movk_i32 s0, 0x1000
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, s0, v0
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, 0x1000, v0
 ; GFX8-NEXT:    v_addc_u32_e32 v3, vcc, 0, v1, vcc
-; GFX8-NEXT:    s_movk_i32 s0, 0x1800
-; GFX8-NEXT:    v_add_u32_e32 v4, vcc, s0, v0
+; GFX8-NEXT:    v_add_u32_e32 v4, vcc, 0x1800, v0
 ; GFX8-NEXT:    v_addc_u32_e32 v5, vcc, 0, v1, vcc
-; GFX8-NEXT:    s_movk_i32 s0, 0x2000
-; GFX8-NEXT:    v_add_u32_e32 v6, vcc, s0, v0
+; GFX8-NEXT:    v_add_u32_e32 v6, vcc, 0x2000, v0
 ; GFX8-NEXT:    v_addc_u32_e32 v7, vcc, 0, v1, vcc
-; GFX8-NEXT:    s_movk_i32 s0, 0x2800
-; GFX8-NEXT:    v_add_u32_e32 v8, vcc, s0, v12
+; GFX8-NEXT:    v_add_u32_e32 v8, vcc, 0x2800, v12
 ; GFX8-NEXT:    flat_load_dwordx2 v[2:3], v[2:3]
 ; GFX8-NEXT:    flat_load_dwordx2 v[4:5], v[4:5]
 ; GFX8-NEXT:    flat_load_dwordx2 v[6:7], v[6:7]
 ; GFX8-NEXT:    v_addc_u32_e32 v9, vcc, 0, v13, vcc
-; GFX8-NEXT:    s_movk_i32 s0, 0x3000
-; GFX8-NEXT:    v_add_u32_e32 v10, vcc, s0, v12
+; GFX8-NEXT:    v_add_u32_e32 v10, vcc, 0x3000, v12
 ; GFX8-NEXT:    v_addc_u32_e32 v11, vcc, 0, v13, vcc
-; GFX8-NEXT:    s_movk_i32 s0, 0x3800
 ; GFX8-NEXT:    flat_load_dwordx2 v[8:9], v[8:9]
 ; GFX8-NEXT:    flat_load_dwordx2 v[10:11], v[10:11]
-; GFX8-NEXT:    v_add_u32_e32 v12, vcc, s0, v12
+; GFX8-NEXT:    v_add_u32_e32 v12, vcc, 0x3800, v12
 ; GFX8-NEXT:    v_addc_u32_e32 v13, vcc, 0, v13, vcc
 ; GFX8-NEXT:    flat_load_dwordx2 v[12:13], v[12:13]
 ; GFX8-NEXT:    s_waitcnt vmcnt(4)
@@ -2115,42 +2109,42 @@ define amdgpu_kernel void @DiffBase(ptr addrspace(1) %buffer1,
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_swappc_b64 s[30:31], s[4:5]
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
-; GFX9-NEXT:    v_and_b32_e32 v2, 0xffff8000, v0
-; GFX9-NEXT:    v_mov_b32_e32 v1, s37
-; GFX9-NEXT:    v_add_co_u32_e32 v0, vcc, s36, v2
-; GFX9-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
-; GFX9-NEXT:    v_mov_b32_e32 v3, s39
-; GFX9-NEXT:    v_add_co_u32_e32 v12, vcc, s38, v2
-; GFX9-NEXT:    v_addc_co_u32_e32 v13, vcc, 0, v3, vcc
-; GFX9-NEXT:    v_add_co_u32_e32 v2, vcc, 0x1000, v0
-; GFX9-NEXT:    v_addc_co_u32_e32 v3, vcc, 0, v1, vcc
-; GFX9-NEXT:    v_add_co_u32_e32 v4, vcc, 0x2000, v0
-; GFX9-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v1, vcc
-; GFX9-NEXT:    global_load_dwordx2 v[6:7], v[2:3], off
-; GFX9-NEXT:    global_load_dwordx2 v[8:9], v[2:3], off offset:2048
-; GFX9-NEXT:    global_load_dwordx2 v[10:11], v[4:5], off
-; GFX9-NEXT:    v_add_co_u32_e32 v2, vcc, 0x2000, v12
-; GFX9-NEXT:    v_addc_co_u32_e32 v3, vcc, 0, v13, vcc
-; GFX9-NEXT:    v_add_co_u32_e32 v4, vcc, 0x3000, v12
-; GFX9-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v13, vcc
-; GFX9-NEXT:    global_load_dwordx2 v[12:13], v[2:3], off offset:2048
-; GFX9-NEXT:    global_load_dwordx2 v[14:15], v[4:5], off
-; GFX9-NEXT:    global_load_dwordx2 v[16:17], v[4:5], off offset:2048
+; GFX9-NEXT:    v_and_b32_e32 v16, 0xffff8000, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s37
+; GFX9-NEXT:    v_add_co_u32_e32 v2, vcc, s36, v16
+; GFX9-NEXT:    v_addc_co_u32_e32 v3, vcc, 0, v0, vcc
+; GFX9-NEXT:    v_mov_b32_e32 v0, s39
+; GFX9-NEXT:    v_add_co_u32_e32 v10, vcc, s38, v16
+; GFX9-NEXT:    v_addc_co_u32_e32 v11, vcc, 0, v0, vcc
+; GFX9-NEXT:    v_add_co_u32_e32 v0, vcc, 0x1000, v2
+; GFX9-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v3, vcc
+; GFX9-NEXT:    v_add_co_u32_e32 v2, vcc, 0x2000, v2
+; GFX9-NEXT:    v_addc_co_u32_e32 v3, vcc, 0, v3, vcc
+; GFX9-NEXT:    global_load_dwordx2 v[4:5], v[0:1], off
+; GFX9-NEXT:    global_load_dwordx2 v[6:7], v[0:1], off offset:2048
+; GFX9-NEXT:    global_load_dwordx2 v[8:9], v[2:3], off
+; GFX9-NEXT:    v_add_co_u32_e32 v0, vcc, 0x2000, v10
+; GFX9-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v11, vcc
+; GFX9-NEXT:    v_add_co_u32_e32 v2, vcc, 0x3000, v10
+; GFX9-NEXT:    v_addc_co_u32_e32 v3, vcc, 0, v11, vcc
+; GFX9-NEXT:    global_load_dwordx2 v[10:11], v[0:1], off offset:2048
+; GFX9-NEXT:    global_load_dwordx2 v[12:13], v[2:3], off
+; GFX9-NEXT:    global_load_dwordx2 v[14:15], v[2:3], off offset:2048
 ; GFX9-NEXT:    s_waitcnt vmcnt(4)
-; GFX9-NEXT:    v_add_co_u32_e32 v2, vcc, v8, v6
-; GFX9-NEXT:    v_addc_co_u32_e32 v3, vcc, v9, v7, vcc
+; GFX9-NEXT:    v_add_co_u32_e32 v0, vcc, v6, v4
+; GFX9-NEXT:    v_addc_co_u32_e32 v1, vcc, v7, v5, vcc
 ; GFX9-NEXT:    s_waitcnt vmcnt(3)
-; GFX9-NEXT:    v_add_co_u32_e32 v2, vcc, v10, v2
-; GFX9-NEXT:    v_addc_co_u32_e32 v3, vcc, v11, v3, vcc
+; GFX9-NEXT:    v_add_co_u32_e32 v0, vcc, v8, v0
+; GFX9-NEXT:    v_addc_co_u32_e32 v1, vcc, v9, v1, vcc
 ; GFX9-NEXT:    s_waitcnt vmcnt(1)
-; GFX9-NEXT:    v_add_co_u32_e32 v4, vcc, v14, v12
-; GFX9-NEXT:    v_addc_co_u32_e32 v5, vcc, v15, v13, vcc
+; GFX9-NEXT:    v_add_co_u32_e32 v2, vcc, v12, v10
+; GFX9-NEXT:    v_addc_co_u32_e32 v3, vcc, v13, v11, vcc
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_add_co_u32_e32 v4, vcc, v16, v4
-; GFX9-NEXT:    v_addc_co_u32_e32 v5, vcc, v17, v5, vcc
-; GFX9-NEXT:    v_add_co_u32_e32 v2, vcc, v2, v4
-; GFX9-NEXT:    v_addc_co_u32_e32 v3, vcc, v3, v5, vcc
-; GFX9-NEXT:    global_store_dwordx2 v[0:1], v[2:3], off
+; GFX9-NEXT:    v_add_co_u32_e32 v2, vcc, v14, v2
+; GFX9-NEXT:    v_addc_co_u32_e32 v3, vcc, v15, v3, vcc
+; GFX9-NEXT:    v_add_co_u32_e32 v0, vcc, v0, v2
+; GFX9-NEXT:    v_addc_co_u32_e32 v1, vcc, v1, v3, vcc
+; GFX9-NEXT:    global_store_dwordx2 v16, v[0:1], s[36:37]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: DiffBase:
@@ -2174,42 +2168,42 @@ define amdgpu_kernel void @DiffBase(ptr addrspace(1) %buffer1,
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-NEXT:    s_swappc_b64 s[30:31], s[4:5]
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
-; GFX10-NEXT:    v_and_b32_e32 v2, 0xffff8000, v0
-; GFX10-NEXT:    v_add_co_u32 v0, s0, s36, v2
-; GFX10-NEXT:    v_add_co_ci_u32_e64 v1, s0, s37, 0, s0
-; GFX10-NEXT:    v_add_co_u32 v14, s0, s38, v2
-; GFX10-NEXT:    v_add_co_ci_u32_e64 v15, s0, s39, 0, s0
-; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, v0, 0x1800
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, 0, v1, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v4, vcc_lo, v14, 0x3000
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, 0, v15, vcc_lo
+; GFX10-NEXT:    v_and_b32_e32 v16, 0xffff8000, v0
+; GFX10-NEXT:    v_add_co_u32 v8, s0, s36, v16
+; GFX10-NEXT:    v_add_co_ci_u32_e64 v9, s0, s37, 0, s0
+; GFX10-NEXT:    v_add_co_u32 v12, s0, s38, v16
+; GFX10-NEXT:    v_add_co_ci_u32_e64 v13, s0, s39, 0, s0
+; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v8, 0x1800
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, 0, v9, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, v12, 0x3000
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, 0, v13, vcc_lo
 ; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    global_load_dwordx2 v[6:7], v[2:3], off offset:-2048
-; GFX10-NEXT:    global_load_dwordx2 v[8:9], v[2:3], off
-; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, 0x2000, v0
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, 0, v1, vcc_lo
+; GFX10-NEXT:    global_load_dwordx2 v[4:5], v[0:1], off offset:-2048
+; GFX10-NEXT:    global_load_dwordx2 v[6:7], v[0:1], off
+; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, 0x2000, v8
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, 0, v9, vcc_lo
 ; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    global_load_dwordx2 v[10:11], v[4:5], off offset:-2048
-; GFX10-NEXT:    global_load_dwordx2 v[12:13], v[4:5], off
-; GFX10-NEXT:    v_add_co_u32 v4, vcc_lo, 0x3800, v14
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, 0, v15, vcc_lo
+; GFX10-NEXT:    global_load_dwordx2 v[8:9], v[2:3], off offset:-2048
+; GFX10-NEXT:    global_load_dwordx2 v[10:11], v[2:3], off
+; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, 0x3800, v12
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, 0, v13, vcc_lo
+; GFX10-NEXT:    global_load_dwordx2 v[12:13], v[0:1], off
 ; GFX10-NEXT:    global_load_dwordx2 v[14:15], v[2:3], off
-; GFX10-NEXT:    global_load_dwordx2 v[16:17], v[4:5], off
 ; GFX10-NEXT:    s_waitcnt vmcnt(4)
-; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, v8, v6
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v9, v7, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v6, v4
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v7, v5, vcc_lo
 ; GFX10-NEXT:    s_waitcnt vmcnt(2)
-; GFX10-NEXT:    v_add_co_u32 v4, vcc_lo, v12, v10
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v13, v11, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, v10, v8
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v11, v9, vcc_lo
 ; GFX10-NEXT:    s_waitcnt vmcnt(1)
+; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v12, v0
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v13, v1, vcc_lo
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, v14, v2
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v15, v3, vcc_lo
-; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_add_co_u32 v4, vcc_lo, v16, v4
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v17, v5, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, v2, v4
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v3, v5, vcc_lo
-; GFX10-NEXT:    global_store_dwordx2 v[0:1], v[2:3], off
+; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v0, v2
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v1, v3, vcc_lo
+; GFX10-NEXT:    global_store_dwordx2 v16, v[0:1], s[36:37]
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: DiffBase:
@@ -2225,44 +2219,44 @@ define amdgpu_kernel void @DiffBase(ptr addrspace(1) %buffer1,
 ; GFX11-NEXT:    s_swappc_b64 s[30:31], s[2:3]
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11-NEXT:    v_and_b32_e32 v2, 0xffff8000, v0
-; GFX11-NEXT:    v_add_co_u32 v0, s0, s36, v2
+; GFX11-NEXT:    v_and_b32_e32 v12, 0xffff8000, v0
+; GFX11-NEXT:    v_add_co_u32 v2, s0, s36, v12
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_3)
-; GFX11-NEXT:    v_add_co_ci_u32_e64 v1, null, s37, 0, s0
-; GFX11-NEXT:    v_add_co_u32 v10, s0, s38, v2
-; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, 0x1000, v0
+; GFX11-NEXT:    v_add_co_ci_u32_e64 v3, null, s37, 0, s0
+; GFX11-NEXT:    v_add_co_u32 v8, s0, s38, v12
+; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, 0x1000, v2
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_3) | instskip(SKIP_4) | instid1(VALU_DEP_4)
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, 0, v1, vcc_lo
-; GFX11-NEXT:    v_add_co_ci_u32_e64 v11, null, s39, 0, s0
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v0, 0x2000
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, 0, v1, vcc_lo
-; GFX11-NEXT:    v_add_co_u32 v6, vcc_lo, 0x2000, v10
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v7, vcc_lo, 0, v11, vcc_lo
-; GFX11-NEXT:    v_add_co_u32 v10, vcc_lo, 0x3000, v10
-; GFX11-NEXT:    global_load_b64 v[8:9], v[4:5], off offset:-4096
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v11, vcc_lo, 0, v11, vcc_lo
-; GFX11-NEXT:    global_load_b64 v[2:3], v[2:3], off offset:2048
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, 0, v3, vcc_lo
+; GFX11-NEXT:    v_add_co_ci_u32_e64 v9, null, s39, 0, s0
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, v2, 0x2000
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, 0, v3, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, 0x2000, v8
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, 0, v9, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v8, vcc_lo, 0x3000, v8
+; GFX11-NEXT:    global_load_b64 v[6:7], v[2:3], off offset:-4096
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v9, vcc_lo, 0, v9, vcc_lo
+; GFX11-NEXT:    global_load_b64 v[0:1], v[0:1], off offset:2048
 ; GFX11-NEXT:    s_clause 0x1
-; GFX11-NEXT:    global_load_b64 v[6:7], v[6:7], off offset:2048
-; GFX11-NEXT:    global_load_b64 v[12:13], v[10:11], off
-; GFX11-NEXT:    global_load_b64 v[4:5], v[4:5], off
-; GFX11-NEXT:    global_load_b64 v[10:11], v[10:11], off offset:2048
+; GFX11-NEXT:    global_load_b64 v[4:5], v[4:5], off offset:2048
+; GFX11-NEXT:    global_load_b64 v[10:11], v[8:9], off
+; GFX11-NEXT:    global_load_b64 v[2:3], v[2:3], off
+; GFX11-NEXT:    global_load_b64 v[8:9], v[8:9], off offset:2048
 ; GFX11-NEXT:    s_waitcnt vmcnt(4)
-; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, v2, v8
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v3, v9, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, v0, v6
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v1, v7, vcc_lo
 ; GFX11-NEXT:    s_waitcnt vmcnt(2)
-; GFX11-NEXT:    v_add_co_u32 v6, vcc_lo, v12, v6
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v7, vcc_lo, v13, v7, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v10, v4
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v11, v5, vcc_lo
 ; GFX11-NEXT:    s_waitcnt vmcnt(1)
-; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, v4, v2
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v5, v3, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, v2, v0
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v10, v6
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v11, v7, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, v8, v4
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v9, v5, vcc_lo
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, v2, v4
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v3, v5, vcc_lo
-; GFX11-NEXT:    global_store_b64 v[0:1], v[2:3], off
+; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, v0, v2
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v1, v3, vcc_lo
+; GFX11-NEXT:    global_store_b64 v12, v[0:1], s[36:37]
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
                                     ptr addrspace(1) %buffer2) {
@@ -2406,52 +2400,52 @@ define amdgpu_kernel void @ReverseOrder(ptr addrspace(1) %buffer) {
 ; GFX900-NEXT:    s_swappc_b64 s[30:31], s[4:5]
 ; GFX900-NEXT:    v_and_b32_e32 v1, 0xff, v0
 ; GFX900-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
+; GFX900-NEXT:    v_and_b32_e32 v22, 0xffff8000, v0
 ; GFX900-NEXT:    v_mov_b32_e32 v2, 0
-; GFX900-NEXT:    v_and_b32_e32 v0, 0xffff8000, v0
-; GFX900-NEXT:    v_mov_b32_e32 v4, s35
-; GFX900-NEXT:    v_add_co_u32_e32 v3, vcc, s34, v0
+; GFX900-NEXT:    v_mov_b32_e32 v0, s35
+; GFX900-NEXT:    v_add_co_u32_e32 v3, vcc, s34, v22
+; GFX900-NEXT:    v_addc_co_u32_e32 v4, vcc, 0, v0, vcc
 ; GFX900-NEXT:    v_lshlrev_b64 v[0:1], 3, v[1:2]
-; GFX900-NEXT:    v_addc_co_u32_e32 v4, vcc, 0, v4, vcc
+; GFX900-NEXT:    s_movk_i32 s0, 0x3000
 ; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v3, v0
 ; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v4, v1, vcc
-; GFX900-NEXT:    s_movk_i32 s0, 0x3000
-; GFX900-NEXT:    v_add_co_u32_e32 v7, vcc, s0, v0
-; GFX900-NEXT:    global_load_dwordx2 v[5:6], v[0:1], off
-; GFX900-NEXT:    v_addc_co_u32_e32 v8, vcc, 0, v1, vcc
-; GFX900-NEXT:    global_load_dwordx2 v[9:10], v[7:8], off offset:2048
-; GFX900-NEXT:    global_load_dwordx2 v[11:12], v[7:8], off
+; GFX900-NEXT:    v_add_co_u32_e32 v4, vcc, s0, v0
+; GFX900-NEXT:    global_load_dwordx2 v[2:3], v[0:1], off
+; GFX900-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v1, vcc
+; GFX900-NEXT:    global_load_dwordx2 v[6:7], v[4:5], off offset:2048
+; GFX900-NEXT:    global_load_dwordx2 v[8:9], v[4:5], off
 ; GFX900-NEXT:    s_movk_i32 s0, 0x2000
-; GFX900-NEXT:    v_add_co_u32_e32 v7, vcc, s0, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v8, vcc, 0, v1, vcc
-; GFX900-NEXT:    global_load_dwordx2 v[13:14], v[7:8], off offset:2048
+; GFX900-NEXT:    v_add_co_u32_e32 v4, vcc, s0, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v1, vcc
+; GFX900-NEXT:    global_load_dwordx2 v[10:11], v[4:5], off offset:2048
 ; GFX900-NEXT:    s_movk_i32 s0, 0x1000
-; GFX900-NEXT:    v_add_co_u32_e32 v15, vcc, s0, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v16, vcc, 0, v1, vcc
-; GFX900-NEXT:    global_load_dwordx2 v[17:18], v[15:16], off
-; GFX900-NEXT:    global_load_dwordx2 v[19:20], v[7:8], off
-; GFX900-NEXT:    global_load_dwordx2 v[21:22], v[15:16], off offset:2048
-; GFX900-NEXT:    global_load_dwordx2 v[23:24], v[0:1], off offset:2048
+; GFX900-NEXT:    v_add_co_u32_e32 v12, vcc, s0, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v13, vcc, 0, v1, vcc
+; GFX900-NEXT:    global_load_dwordx2 v[14:15], v[12:13], off
+; GFX900-NEXT:    global_load_dwordx2 v[16:17], v[4:5], off
+; GFX900-NEXT:    global_load_dwordx2 v[18:19], v[12:13], off offset:2048
+; GFX900-NEXT:    global_load_dwordx2 v[20:21], v[0:1], off offset:2048
 ; GFX900-NEXT:    s_waitcnt vmcnt(6)
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v9, v5
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v10, v6, vcc
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v6, v2
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v7, v3, vcc
 ; GFX900-NEXT:    s_waitcnt vmcnt(5)
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v11, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v12, v1, vcc
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v8, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v9, v1, vcc
 ; GFX900-NEXT:    s_waitcnt vmcnt(4)
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v13, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v14, v1, vcc
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v10, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v11, v1, vcc
 ; GFX900-NEXT:    s_waitcnt vmcnt(2)
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v19, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v20, v1, vcc
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v16, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v17, v1, vcc
 ; GFX900-NEXT:    s_waitcnt vmcnt(1)
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v21, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v22, v1, vcc
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v17, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v18, v1, vcc
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v18, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v19, v1, vcc
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v14, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v15, v1, vcc
 ; GFX900-NEXT:    s_waitcnt vmcnt(0)
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v23, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v24, v1, vcc
-; GFX900-NEXT:    global_store_dwordx2 v[3:4], v[0:1], off
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v20, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v21, v1, vcc
+; GFX900-NEXT:    global_store_dwordx2 v22, v[0:1], s[34:35]
 ; GFX900-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: ReverseOrder:
@@ -2477,58 +2471,58 @@ define amdgpu_kernel void @ReverseOrder(ptr addrspace(1) %buffer) {
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v2, 7, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX10-NEXT:    v_and_b32_e32 v0, 0xff, v0
-; GFX10-NEXT:    v_and_b32_e32 v2, 0xffff8000, v2
+; GFX10-NEXT:    v_and_b32_e32 v20, 0xffff8000, v2
 ; GFX10-NEXT:    v_lshlrev_b64 v[0:1], 3, v[0:1]
-; GFX10-NEXT:    v_add_co_u32 v2, s0, s34, v2
+; GFX10-NEXT:    v_add_co_u32 v2, s0, s34, v20
 ; GFX10-NEXT:    v_add_co_ci_u32_e64 v3, s0, s35, 0, s0
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v2, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v4, vcc_lo, 0x3800, v0
+; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, 0x3800, v0
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, 0, v1, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v4, vcc_lo, 0x3000, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, 0, v1, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v6, vcc_lo, 0x3000, v0
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v7, vcc_lo, 0, v1, vcc_lo
 ; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    global_load_dwordx2 v[8:9], v[0:1], off
-; GFX10-NEXT:    global_load_dwordx2 v[10:11], v[4:5], off
-; GFX10-NEXT:    v_add_co_u32 v4, vcc_lo, 0x2800, v0
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, 0, v1, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v12, vcc_lo, 0x2000, v0
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v13, vcc_lo, 0, v1, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v14, vcc_lo, 0x1800, v0
+; GFX10-NEXT:    global_load_dwordx2 v[6:7], v[0:1], off
+; GFX10-NEXT:    global_load_dwordx2 v[8:9], v[2:3], off
+; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, 0x2800, v0
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, 0, v1, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v10, vcc_lo, 0x2000, v0
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v11, vcc_lo, 0, v1, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v12, vcc_lo, 0x1800, v0
 ; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    global_load_dwordx2 v[6:7], v[6:7], off
-; GFX10-NEXT:    global_load_dwordx2 v[12:13], v[12:13], off
+; GFX10-NEXT:    global_load_dwordx2 v[4:5], v[4:5], off
+; GFX10-NEXT:    global_load_dwordx2 v[10:11], v[10:11], off
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v13, vcc_lo, 0, v1, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v14, vcc_lo, 0x1000, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v15, vcc_lo, 0, v1, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v16, vcc_lo, 0x1000, v0
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v17, vcc_lo, 0, v1, vcc_lo
 ; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    global_load_dwordx2 v[14:15], v[14:15], off
-; GFX10-NEXT:    global_load_dwordx2 v[4:5], v[4:5], off
+; GFX10-NEXT:    global_load_dwordx2 v[12:13], v[12:13], off
+; GFX10-NEXT:    global_load_dwordx2 v[2:3], v[2:3], off
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, 0x800, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, 0, v1, vcc_lo
 ; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    global_load_dwordx2 v[18:19], v[16:17], off
-; GFX10-NEXT:    global_load_dwordx2 v[20:21], v[0:1], off
+; GFX10-NEXT:    global_load_dwordx2 v[16:17], v[14:15], off
+; GFX10-NEXT:    global_load_dwordx2 v[18:19], v[0:1], off
 ; GFX10-NEXT:    s_waitcnt vmcnt(6)
-; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v10, v8
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v11, v9, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v8, v6
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v9, v7, vcc_lo
 ; GFX10-NEXT:    s_waitcnt vmcnt(5)
-; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v6, v0
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v7, v1, vcc_lo
-; GFX10-NEXT:    s_waitcnt vmcnt(2)
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v4, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v5, v1, vcc_lo
+; GFX10-NEXT:    s_waitcnt vmcnt(2)
+; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v2, v0
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v10, v0
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v11, v1, vcc_lo
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v12, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v13, v1, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v14, v0
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v15, v1, vcc_lo
 ; GFX10-NEXT:    s_waitcnt vmcnt(1)
+; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v16, v0
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v17, v1, vcc_lo
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v18, v0
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v19, v1, vcc_lo
-; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v20, v0
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v21, v1, vcc_lo
-; GFX10-NEXT:    global_store_dwordx2 v[2:3], v[0:1], off
+; GFX10-NEXT:    global_store_dwordx2 v20, v[0:1], s[34:35]
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX90A-LABEL: ReverseOrder:
@@ -2553,52 +2547,52 @@ define amdgpu_kernel void @ReverseOrder(ptr addrspace(1) %buffer) {
 ; GFX90A-NEXT:    s_swappc_b64 s[30:31], s[4:5]
 ; GFX90A-NEXT:    v_and_b32_e32 v2, 0xff, v0
 ; GFX90A-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
-; GFX90A-NEXT:    v_and_b32_e32 v0, 0xffff8000, v0
+; GFX90A-NEXT:    v_and_b32_e32 v22, 0xffff8000, v0
 ; GFX90A-NEXT:    v_mov_b32_e32 v3, 0
-; GFX90A-NEXT:    v_mov_b32_e32 v1, s35
-; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, s34, v0
-; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
-; GFX90A-NEXT:    v_lshlrev_b64 v[2:3], 3, v[2:3]
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v0, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v1, v3, vcc
+; GFX90A-NEXT:    v_mov_b32_e32 v0, s35
+; GFX90A-NEXT:    v_add_co_u32_e32 v4, vcc, s34, v22
+; GFX90A-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v0, vcc
+; GFX90A-NEXT:    v_lshlrev_b64 v[0:1], 3, v[2:3]
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v4, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v5, v1, vcc
 ; GFX90A-NEXT:    s_movk_i32 s0, 0x3000
-; GFX90A-NEXT:    v_add_co_u32_e32 v6, vcc, s0, v2
-; GFX90A-NEXT:    global_load_dwordx2 v[4:5], v[2:3], off
-; GFX90A-NEXT:    v_addc_co_u32_e32 v7, vcc, 0, v3, vcc
-; GFX90A-NEXT:    global_load_dwordx2 v[8:9], v[6:7], off offset:2048
-; GFX90A-NEXT:    global_load_dwordx2 v[10:11], v[6:7], off
+; GFX90A-NEXT:    v_add_co_u32_e32 v4, vcc, s0, v0
+; GFX90A-NEXT:    global_load_dwordx2 v[2:3], v[0:1], off
+; GFX90A-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v1, vcc
+; GFX90A-NEXT:    global_load_dwordx2 v[6:7], v[4:5], off offset:2048
+; GFX90A-NEXT:    global_load_dwordx2 v[8:9], v[4:5], off
 ; GFX90A-NEXT:    s_movk_i32 s0, 0x2000
-; GFX90A-NEXT:    v_add_co_u32_e32 v6, vcc, s0, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v7, vcc, 0, v3, vcc
-; GFX90A-NEXT:    global_load_dwordx2 v[12:13], v[6:7], off offset:2048
+; GFX90A-NEXT:    v_add_co_u32_e32 v4, vcc, s0, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v1, vcc
+; GFX90A-NEXT:    global_load_dwordx2 v[10:11], v[4:5], off offset:2048
 ; GFX90A-NEXT:    s_movk_i32 s0, 0x1000
-; GFX90A-NEXT:    v_add_co_u32_e32 v14, vcc, s0, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v15, vcc, 0, v3, vcc
-; GFX90A-NEXT:    global_load_dwordx2 v[16:17], v[14:15], off
-; GFX90A-NEXT:    global_load_dwordx2 v[18:19], v[6:7], off
-; GFX90A-NEXT:    global_load_dwordx2 v[20:21], v[14:15], off offset:2048
-; GFX90A-NEXT:    global_load_dwordx2 v[22:23], v[2:3], off offset:2048
+; GFX90A-NEXT:    v_add_co_u32_e32 v12, vcc, s0, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v13, vcc, 0, v1, vcc
+; GFX90A-NEXT:    global_load_dwordx2 v[14:15], v[12:13], off
+; GFX90A-NEXT:    global_load_dwordx2 v[16:17], v[4:5], off
+; GFX90A-NEXT:    global_load_dwordx2 v[18:19], v[12:13], off offset:2048
+; GFX90A-NEXT:    global_load_dwordx2 v[20:21], v[0:1], off offset:2048
 ; GFX90A-NEXT:    s_waitcnt vmcnt(6)
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v8, v4
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v9, v5, vcc
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v6, v2
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v7, v3, vcc
 ; GFX90A-NEXT:    s_waitcnt vmcnt(5)
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v10, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v11, v3, vcc
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v8, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v9, v1, vcc
 ; GFX90A-NEXT:    s_waitcnt vmcnt(4)
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v12, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v13, v3, vcc
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v10, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v11, v1, vcc
 ; GFX90A-NEXT:    s_waitcnt vmcnt(2)
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v18, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v19, v3, vcc
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v16, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v17, v1, vcc
 ; GFX90A-NEXT:    s_waitcnt vmcnt(1)
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v20, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v21, v3, vcc
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v16, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v17, v3, vcc
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v18, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v19, v1, vcc
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v14, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v15, v1, vcc
 ; GFX90A-NEXT:    s_waitcnt vmcnt(0)
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v22, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v23, v3, vcc
-; GFX90A-NEXT:    global_store_dwordx2 v[0:1], v[2:3], off
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v20, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v21, v1, vcc
+; GFX90A-NEXT:    global_store_dwordx2 v22, v[0:1], s[34:35]
 ; GFX90A-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: ReverseOrder:
@@ -2615,58 +2609,58 @@ define amdgpu_kernel void @ReverseOrder(ptr addrspace(1) %buffer) {
 ; GFX11-NEXT:    v_dual_mov_b32 v1, 0 :: v_dual_lshlrev_b32 v2, 7, v0
 ; GFX11-NEXT:    v_and_b32_e32 v0, 0xff, v0
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_and_b32_e32 v2, 0xffff8000, v2
+; GFX11-NEXT:    v_and_b32_e32 v16, 0xffff8000, v2
 ; GFX11-NEXT:    v_lshlrev_b64 v[0:1], 3, v[0:1]
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11-NEXT:    v_add_co_u32 v2, s0, s34, v2
+; GFX11-NEXT:    v_add_co_u32 v2, s0, s34, v16
 ; GFX11-NEXT:    v_add_co_ci_u32_e64 v3, null, s35, 0, s0
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
 ; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, v2, v0
 ; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, 0x3000, v0
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, 0, v1, vcc_lo
-; GFX11-NEXT:    v_add_co_u32 v10, vcc_lo, 0x2000, v0
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, 0x3000, v0
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, 0, v1, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v8, vcc_lo, 0x2000, v0
 ; GFX11-NEXT:    s_clause 0x2
-; GFX11-NEXT:    global_load_b64 v[6:7], v[0:1], off
-; GFX11-NEXT:    global_load_b64 v[8:9], v[4:5], off offset:2048
-; GFX11-NEXT:    global_load_b64 v[4:5], v[4:5], off
+; GFX11-NEXT:    global_load_b64 v[4:5], v[0:1], off
+; GFX11-NEXT:    global_load_b64 v[6:7], v[2:3], off offset:2048
+; GFX11-NEXT:    global_load_b64 v[2:3], v[2:3], off
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v9, vcc_lo, 0, v1, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v10, vcc_lo, 0x1000, v0
 ; GFX11-NEXT:    v_add_co_ci_u32_e32 v11, vcc_lo, 0, v1, vcc_lo
-; GFX11-NEXT:    v_add_co_u32 v12, vcc_lo, 0x1000, v0
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v13, vcc_lo, 0, v1, vcc_lo
 ; GFX11-NEXT:    s_clause 0x4
-; GFX11-NEXT:    global_load_b64 v[14:15], v[10:11], off offset:2048
-; GFX11-NEXT:    global_load_b64 v[16:17], v[12:13], off
-; GFX11-NEXT:    global_load_b64 v[10:11], v[10:11], off
-; GFX11-NEXT:    global_load_b64 v[12:13], v[12:13], off offset:2048
+; GFX11-NEXT:    global_load_b64 v[12:13], v[8:9], off offset:2048
+; GFX11-NEXT:    global_load_b64 v[14:15], v[10:11], off
+; GFX11-NEXT:    global_load_b64 v[8:9], v[8:9], off
+; GFX11-NEXT:    global_load_b64 v[10:11], v[10:11], off offset:2048
 ; GFX11-NEXT:    global_load_b64 v[0:1], v[0:1], off offset:2048
 ; GFX11-NEXT:    s_waitcnt vmcnt(6)
-; GFX11-NEXT:    v_add_co_u32 v6, vcc_lo, v8, v6
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v7, vcc_lo, v9, v7, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v6, v4
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v7, v5, vcc_lo
 ; GFX11-NEXT:    s_waitcnt vmcnt(5)
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v4, v6
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v5, v7, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, v2, v4
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v3, v5, vcc_lo
 ; GFX11-NEXT:    s_waitcnt vmcnt(4)
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v14, v4
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v15, v5, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, v12, v2
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v13, v3, vcc_lo
 ; GFX11-NEXT:    s_waitcnt vmcnt(2)
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v10, v4
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v11, v5, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, v8, v2
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v9, v3, vcc_lo
 ; GFX11-NEXT:    s_waitcnt vmcnt(1)
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v12, v4
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v13, v5, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, v10, v2
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v11, v3, vcc_lo
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v16, v4
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v17, v5, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, v14, v2
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v15, v3, vcc_lo
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, v0, v4
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v1, v5, vcc_lo
-; GFX11-NEXT:    global_store_b64 v[2:3], v[0:1], off
+; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, v0, v2
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v1, v3, vcc_lo
+; GFX11-NEXT:    global_store_b64 v16, v[0:1], s[34:35]
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
 entry:
@@ -2777,25 +2771,25 @@ define hidden amdgpu_kernel void @negativeoffset(ptr addrspace(1) nocapture %buf
 ; GFX900-NEXT:    s_swappc_b64 s[30:31], s[4:5]
 ; GFX900-NEXT:    v_and_b32_e32 v1, 0xff, v0
 ; GFX900-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
+; GFX900-NEXT:    v_and_b32_e32 v8, 0xffff8000, v0
 ; GFX900-NEXT:    v_mov_b32_e32 v2, 0
-; GFX900-NEXT:    v_and_b32_e32 v0, 0xffff8000, v0
-; GFX900-NEXT:    v_mov_b32_e32 v4, s35
-; GFX900-NEXT:    v_add_co_u32_e32 v3, vcc, s34, v0
+; GFX900-NEXT:    v_mov_b32_e32 v0, s35
+; GFX900-NEXT:    v_add_co_u32_e32 v3, vcc, s34, v8
+; GFX900-NEXT:    v_addc_co_u32_e32 v4, vcc, 0, v0, vcc
 ; GFX900-NEXT:    v_lshlrev_b64 v[0:1], 3, v[1:2]
-; GFX900-NEXT:    v_addc_co_u32_e32 v4, vcc, 0, v4, vcc
-; GFX900-NEXT:    v_add_co_u32_e32 v2, vcc, v3, v0
-; GFX900-NEXT:    v_addc_co_u32_e32 v6, vcc, v4, v1, vcc
 ; GFX900-NEXT:    s_movk_i32 s0, 0x1000
+; GFX900-NEXT:    v_add_co_u32_e32 v2, vcc, v3, v0
+; GFX900-NEXT:    v_addc_co_u32_e32 v3, vcc, v4, v1, vcc
 ; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, s0, v2
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, -1, v6, vcc
-; GFX900-NEXT:    v_add_co_u32_e32 v5, vcc, 0, v2
-; GFX900-NEXT:    v_addc_co_u32_e32 v6, vcc, -1, v6, vcc
-; GFX900-NEXT:    global_load_dwordx2 v[7:8], v[0:1], off offset:-2048
-; GFX900-NEXT:    global_load_dwordx2 v[9:10], v[5:6], off
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, -1, v3, vcc
+; GFX900-NEXT:    v_add_co_u32_e32 v2, vcc, 0, v2
+; GFX900-NEXT:    v_addc_co_u32_e32 v3, vcc, -1, v3, vcc
+; GFX900-NEXT:    global_load_dwordx2 v[4:5], v[0:1], off offset:-2048
+; GFX900-NEXT:    global_load_dwordx2 v[6:7], v[2:3], off
 ; GFX900-NEXT:    s_waitcnt vmcnt(0)
-; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v9, v7
-; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v10, v8, vcc
-; GFX900-NEXT:    global_store_dwordx2 v[3:4], v[0:1], off
+; GFX900-NEXT:    v_add_co_u32_e32 v0, vcc, v6, v4
+; GFX900-NEXT:    v_addc_co_u32_e32 v1, vcc, v7, v5, vcc
+; GFX900-NEXT:    global_store_dwordx2 v8, v[0:1], s[34:35]
 ; GFX900-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: negativeoffset:
@@ -2821,23 +2815,23 @@ define hidden amdgpu_kernel void @negativeoffset(ptr addrspace(1) nocapture %buf
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v2, 7, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX10-NEXT:    v_and_b32_e32 v0, 0xff, v0
-; GFX10-NEXT:    v_and_b32_e32 v2, 0xffff8000, v2
+; GFX10-NEXT:    v_and_b32_e32 v8, 0xffff8000, v2
 ; GFX10-NEXT:    v_lshlrev_b64 v[0:1], 3, v[0:1]
-; GFX10-NEXT:    v_add_co_u32 v2, s0, s34, v2
+; GFX10-NEXT:    v_add_co_u32 v2, s0, s34, v8
 ; GFX10-NEXT:    v_add_co_ci_u32_e64 v3, s0, s35, 0, s0
-; GFX10-NEXT:    v_add_co_u32 v4, vcc_lo, v2, v0
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v3, v1, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, 0x800, v4
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, -1, v5, vcc_lo
-; GFX10-NEXT:    v_add_co_u32 v4, vcc_lo, 0, v4
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, -1, v5, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, v2, v0
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v3, v1, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, 0x800, v2
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, -1, v3, vcc_lo
+; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, 0, v2
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, -1, v3, vcc_lo
 ; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    global_load_dwordx2 v[6:7], v[0:1], off
-; GFX10-NEXT:    global_load_dwordx2 v[8:9], v[4:5], off
+; GFX10-NEXT:    global_load_dwordx2 v[4:5], v[0:1], off
+; GFX10-NEXT:    global_load_dwordx2 v[6:7], v[2:3], off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v8, v6
-; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v9, v7, vcc_lo
-; GFX10-NEXT:    global_store_dwordx2 v[2:3], v[0:1], off
+; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v6, v4
+; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v7, v5, vcc_lo
+; GFX10-NEXT:    global_store_dwordx2 v8, v[0:1], s[34:35]
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX90A-LABEL: negativeoffset:
@@ -2862,25 +2856,25 @@ define hidden amdgpu_kernel void @negativeoffset(ptr addrspace(1) nocapture %buf
 ; GFX90A-NEXT:    s_swappc_b64 s[30:31], s[4:5]
 ; GFX90A-NEXT:    v_and_b32_e32 v2, 0xff, v0
 ; GFX90A-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
-; GFX90A-NEXT:    v_and_b32_e32 v0, 0xffff8000, v0
+; GFX90A-NEXT:    v_and_b32_e32 v8, 0xffff8000, v0
 ; GFX90A-NEXT:    v_mov_b32_e32 v3, 0
-; GFX90A-NEXT:    v_mov_b32_e32 v1, s35
-; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, s34, v0
-; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
-; GFX90A-NEXT:    v_lshlrev_b64 v[2:3], 3, v[2:3]
-; GFX90A-NEXT:    v_add_co_u32_e32 v4, vcc, v0, v2
-; GFX90A-NEXT:    v_addc_co_u32_e32 v5, vcc, v1, v3, vcc
+; GFX90A-NEXT:    v_mov_b32_e32 v0, s35
+; GFX90A-NEXT:    v_add_co_u32_e32 v4, vcc, s34, v8
+; GFX90A-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v0, vcc
+; GFX90A-NEXT:    v_lshlrev_b64 v[0:1], 3, v[2:3]
+; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v4, v0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v5, v1, vcc
 ; GFX90A-NEXT:    s_movk_i32 s0, 0x1000
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, s0, v4
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, -1, v5, vcc
-; GFX90A-NEXT:    v_add_co_u32_e32 v4, vcc, 0, v4
-; GFX90A-NEXT:    v_addc_co_u32_e32 v5, vcc, -1, v5, vcc
-; GFX90A-NEXT:    global_load_dwordx2 v[6:7], v[2:3], off offset:-2048
-; GFX90A-NEXT:    global_load_dwordx2 v[8:9], v[4:5], off
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, s0, v2
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, -1, v3, vcc
+; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, 0, v2
+; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, -1, v3, vcc
+; GFX90A-NEXT:    global_load_dwordx2 v[4:5], v[0:1], off offset:-2048
+; GFX90A-NEXT:    global_load_dwordx2 v[6:7], v[2:3], off
 ; GFX90A-NEXT:    s_waitcnt vmcnt(0)
-; GFX90A-NEXT:    v_add_co_u32_e32 v2, vcc, v8, v6
-; GFX90A-NEXT:    v_addc_co_u32_e32 v3, vcc, v9, v7, vcc
-; GFX90A-NEXT:    global_store_dwordx2 v[0:1], v[2:3], off
+; GFX90A-NEXT:    v_add_co_u32_e32 v0, vcc, v6, v4
+; GFX90A-NEXT:    v_addc_co_u32_e32 v1, vcc, v7, v5, vcc
+; GFX90A-NEXT:    global_store_dwordx2 v8, v[0:1], s[34:35]
 ; GFX90A-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: negativeoffset:
@@ -2897,26 +2891,26 @@ define hidden amdgpu_kernel void @negativeoffset(ptr addrspace(1) nocapture %buf
 ; GFX11-NEXT:    v_dual_mov_b32 v1, 0 :: v_dual_lshlrev_b32 v2, 7, v0
 ; GFX11-NEXT:    v_and_b32_e32 v0, 0xff, v0
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_and_b32_e32 v2, 0xffff8000, v2
+; GFX11-NEXT:    v_and_b32_e32 v4, 0xffff8000, v2
 ; GFX11-NEXT:    v_lshlrev_b64 v[0:1], 3, v[0:1]
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11-NEXT:    v_add_co_u32 v2, s0, s34, v2
+; GFX11-NEXT:    v_add_co_u32 v2, s0, s34, v4
 ; GFX11-NEXT:    v_add_co_ci_u32_e64 v3, null, s35, 0, s0
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, v2, v0
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, v3, v1, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, v2, v0
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, v3, v1, vcc_lo
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, 0x1000, v4
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, -1, v5, vcc_lo
-; GFX11-NEXT:    v_add_co_u32 v4, vcc_lo, 0, v4
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v5, vcc_lo, -1, v5, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, 0x1000, v2
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, -1, v3, vcc_lo
+; GFX11-NEXT:    v_add_co_u32 v2, vcc_lo, 0, v2
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, -1, v3, vcc_lo
 ; GFX11-NEXT:    s_clause 0x1
 ; GFX11-NEXT:    global_load_b64 v[0:1], v[0:1], off offset:-2048
-; GFX11-NEXT:    global_load_b64 v[4:5], v[4:5], off
+; GFX11-NEXT:    global_load_b64 v[2:3], v[2:3], off
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, v4, v0
-; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v5, v1, vcc_lo
-; GFX11-NEXT:    global_store_b64 v[2:3], v[0:1], off
+; GFX11-NEXT:    v_add_co_u32 v0, vcc_lo, v2, v0
+; GFX11-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
+; GFX11-NEXT:    global_store_b64 v4, v[0:1], s[34:35]
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
 entry:
diff --git a/llvm/test/CodeGen/AMDGPU/resource-optimization-remarks.ll b/llvm/test/CodeGen/AMDGPU/resource-optimization-remarks.ll
index 50a1d48b71304..2616b04332419 100644
--- a/llvm/test/CodeGen/AMDGPU/resource-optimization-remarks.ll
+++ b/llvm/test/CodeGen/AMDGPU/resource-optimization-remarks.ll
@@ -2,7 +2,7 @@
 ; RUN: FileCheck -check-prefix=REMARK %s < %t
 
 ; STDERR: remark: foo.cl:27:0: Function Name: test_kernel
-; STDERR-NEXT: remark: foo.cl:27:0:     SGPRs: 24
+; STDERR-NEXT: remark: foo.cl:27:0:     SGPRs: 28
 ; STDERR-NEXT: remark: foo.cl:27:0:     VGPRs: 9
 ; STDERR-NEXT: remark: foo.cl:27:0:     AGPRs: 43
 ; STDERR-NEXT: remark: foo.cl:27:0:     ScratchSize [bytes/lane]: 0
@@ -27,7 +27,7 @@
 ; REMARK-NEXT: Function:        test_kernel
 ; REMARK-NEXT: Args:
 ; REMARK-NEXT:   - String:          '    SGPRs: '
-; REMARK-NEXT:   - NumSGPR:         '24'
+; REMARK-NEXT:   - NumSGPR:         '28'
 ; REMARK-NEXT: ...
 ; REMARK-NEXT: --- !Analysis
 ; REMARK-NEXT: Pass:            kernel-resource-usage
@@ -120,7 +120,7 @@ define void @test_func() !dbg !6 {
 }
 
 ; STDERR: remark: foo.cl:8:0: Function Name: empty_kernel
-; STDERR-NEXT: remark: foo.cl:8:0:     SGPRs: 0
+; STDERR-NEXT: remark: foo.cl:8:0:     SGPRs: 4
 ; STDERR-NEXT: remark: foo.cl:8:0:     VGPRs: 0
 ; STDERR-NEXT: remark: foo.cl:8:0:     AGPRs: 0
 ; STDERR-NEXT: remark: foo.cl:8:0:     ScratchSize [bytes/lane]: 0
diff --git a/llvm/test/CodeGen/AMDGPU/saddsat.ll b/llvm/test/CodeGen/AMDGPU/saddsat.ll
index 6f6ff8124ffdf..7d6b547cff7c9 100644
--- a/llvm/test/CodeGen/AMDGPU/saddsat.ll
+++ b/llvm/test/CodeGen/AMDGPU/saddsat.ll
@@ -12,8 +12,9 @@ define i8 @v_saddsat_i8(i8 %lhs, i8 %rhs) {
 ; GFX6-NEXT:    v_bfe_i32 v1, v1, 0, 8
 ; GFX6-NEXT:    v_bfe_i32 v0, v0, 0, 8
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
-; GFX6-NEXT:    v_min_i32_e32 v0, 0x7f, v0
-; GFX6-NEXT:    v_max_i32_e32 v0, 0xffffff80, v0
+; GFX6-NEXT:    s_movk_i32 s4, 0xff80
+; GFX6-NEXT:    v_mov_b32_e32 v1, 0x7f
+; GFX6-NEXT:    v_med3_i32 v0, v0, s4, v1
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_saddsat_i8:
@@ -53,8 +54,9 @@ define i16 @v_saddsat_i16(i16 %lhs, i16 %rhs) {
 ; GFX6-NEXT:    v_bfe_i32 v1, v1, 0, 16
 ; GFX6-NEXT:    v_bfe_i32 v0, v0, 0, 16
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
-; GFX6-NEXT:    v_min_i32_e32 v0, 0x7fff, v0
-; GFX6-NEXT:    v_max_i32_e32 v0, 0xffff8000, v0
+; GFX6-NEXT:    s_movk_i32 s4, 0x8000
+; GFX6-NEXT:    v_mov_b32_e32 v1, 0x7fff
+; GFX6-NEXT:    v_med3_i32 v0, v0, s4, v1
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_saddsat_i16:
@@ -135,14 +137,14 @@ define <2 x i16> @v_saddsat_v2i16(<2 x i16> %lhs, <2 x i16> %rhs) {
 ; GFX6-NEXT:    v_bfe_i32 v3, v3, 0, 16
 ; GFX6-NEXT:    v_bfe_i32 v1, v1, 0, 16
 ; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
+; GFX6-NEXT:    s_movk_i32 s4, 0x8000
+; GFX6-NEXT:    v_mov_b32_e32 v3, 0x7fff
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
-; GFX6-NEXT:    v_min_i32_e32 v1, 0x7fff, v1
-; GFX6-NEXT:    v_min_i32_e32 v0, 0x7fff, v0
-; GFX6-NEXT:    v_max_i32_e32 v1, 0xffff8000, v1
-; GFX6-NEXT:    v_max_i32_e32 v0, 0xffff8000, v0
-; GFX6-NEXT:    v_lshlrev_b32_e32 v3, 16, v1
+; GFX6-NEXT:    v_med3_i32 v1, v1, s4, v3
+; GFX6-NEXT:    v_med3_i32 v0, v0, s4, v3
+; GFX6-NEXT:    v_lshlrev_b32_e32 v4, 16, v1
 ; GFX6-NEXT:    v_and_b32_e32 v0, 0xffff, v0
-; GFX6-NEXT:    v_or_b32_e32 v0, v0, v3
+; GFX6-NEXT:    v_or_b32_e32 v0, v0, v4
 ; GFX6-NEXT:    v_and_b32_e32 v1, 0xffff, v1
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
@@ -196,16 +198,15 @@ define <3 x i16> @v_saddsat_v3i16(<3 x i16> %lhs, <3 x i16> %rhs) {
 ; GFX6-NEXT:    v_bfe_i32 v5, v5, 0, 16
 ; GFX6-NEXT:    v_bfe_i32 v2, v2, 0, 16
 ; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v1, v4
+; GFX6-NEXT:    s_movk_i32 s4, 0x8000
+; GFX6-NEXT:    v_mov_b32_e32 v4, 0x7fff
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v3
-; GFX6-NEXT:    v_min_i32_e32 v1, 0x7fff, v1
-; GFX6-NEXT:    v_min_i32_e32 v0, 0x7fff, v0
+; GFX6-NEXT:    v_med3_i32 v1, v1, s4, v4
+; GFX6-NEXT:    v_med3_i32 v0, v0, s4, v4
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
-; GFX6-NEXT:    v_max_i32_e32 v1, 0xffff8000, v1
-; GFX6-NEXT:    v_max_i32_e32 v0, 0xffff8000, v0
-; GFX6-NEXT:    v_min_i32_e32 v2, 0x7fff, v2
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
 ; GFX6-NEXT:    v_and_b32_e32 v0, 0xffff, v0
-; GFX6-NEXT:    v_max_i32_e32 v3, 0xffff8000, v2
+; GFX6-NEXT:    v_med3_i32 v3, v2, s4, v4
 ; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX6-NEXT:    v_or_b32_e32 v2, 0xffff0000, v3
 ; GFX6-NEXT:    v_alignbit_b32 v1, v3, v1, 16
@@ -268,11 +269,11 @@ define <2 x float> @v_saddsat_v4i16(<4 x i16> %lhs, <4 x i16> %rhs) {
 ; GFX6-NEXT:    v_bfe_i32 v5, v5, 0, 16
 ; GFX6-NEXT:    v_bfe_i32 v1, v1, 0, 16
 ; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v1, v5
+; GFX6-NEXT:    s_movk_i32 s4, 0x8000
+; GFX6-NEXT:    v_mov_b32_e32 v5, 0x7fff
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v4
-; GFX6-NEXT:    v_min_i32_e32 v1, 0x7fff, v1
-; GFX6-NEXT:    v_min_i32_e32 v0, 0x7fff, v0
-; GFX6-NEXT:    v_max_i32_e32 v1, 0xffff8000, v1
-; GFX6-NEXT:    v_max_i32_e32 v0, 0xffff8000, v0
+; GFX6-NEXT:    v_med3_i32 v1, v1, s4, v5
+; GFX6-NEXT:    v_med3_i32 v0, v0, s4, v5
 ; GFX6-NEXT:    v_bfe_i32 v6, v6, 0, 16
 ; GFX6-NEXT:    v_bfe_i32 v2, v2, 0, 16
 ; GFX6-NEXT:    v_bfe_i32 v7, v7, 0, 16
@@ -282,10 +283,8 @@ define <2 x float> @v_saddsat_v4i16(<4 x i16> %lhs, <4 x i16> %rhs) {
 ; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v3, v7
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v6
-; GFX6-NEXT:    v_min_i32_e32 v1, 0x7fff, v1
-; GFX6-NEXT:    v_min_i32_e32 v2, 0x7fff, v2
-; GFX6-NEXT:    v_max_i32_e32 v1, 0xffff8000, v1
-; GFX6-NEXT:    v_max_i32_e32 v2, 0xffff8000, v2
+; GFX6-NEXT:    v_med3_i32 v1, v1, s4, v5
+; GFX6-NEXT:    v_med3_i32 v2, v2, s4, v5
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
 ; GFX6-NEXT:    v_and_b32_e32 v2, 0xffff, v2
 ; GFX6-NEXT:    v_or_b32_e32 v1, v2, v1
diff --git a/llvm/test/CodeGen/AMDGPU/scheduler-rp-calc-one-successor-two-predecessors-bug.ll b/llvm/test/CodeGen/AMDGPU/scheduler-rp-calc-one-successor-two-predecessors-bug.ll
new file mode 100644
index 0000000000000..a57a6decc2ba0
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/scheduler-rp-calc-one-successor-two-predecessors-bug.ll
@@ -0,0 +1,85 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
+; RUN: llc -march=amdgcn -mcpu=gfx900 < %s | FileCheck -check-prefix=GFX900 %s
+
+declare void @llvm.amdgcn.kill(i1)
+declare <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 immarg, float, float, <8 x i32>, <4 x i32>, i1 immarg, i32 immarg, i32 immarg)
+declare <2 x half> @llvm.amdgcn.cvt.pkrtz(float, float)
+declare void @llvm.amdgcn.exp.compr.v2f16(i32 immarg, i32 immarg, <2 x half>, <2 x half>, i1 immarg, i1 immarg)
+
+define amdgpu_ps void @_amdgpu_ps_main(float %arg) {
+; GFX900-LABEL: _amdgpu_ps_main:
+; GFX900:       ; %bb.0: ; %bb
+; GFX900-NEXT:    s_mov_b64 s[4:5], exec
+; GFX900-NEXT:    s_wqm_b64 exec, exec
+; GFX900-NEXT:    v_mov_b32_e32 v1, v0
+; GFX900-NEXT:    s_mov_b32 s0, 0
+; GFX900-NEXT:    v_cmp_ngt_f32_e32 vcc, 0, v1
+; GFX900-NEXT:    ; implicit-def: $vgpr0
+; GFX900-NEXT:    ; implicit-def: $sgpr2
+; GFX900-NEXT:    s_and_saveexec_b64 s[6:7], vcc
+; GFX900-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; GFX900-NEXT:    s_cbranch_execz .LBB0_2
+; GFX900-NEXT:  ; %bb.1: ; %bb1
+; GFX900-NEXT:    v_mov_b32_e32 v0, 0
+; GFX900-NEXT:    s_mov_b32 s1, s0
+; GFX900-NEXT:    s_mov_b32 s2, s0
+; GFX900-NEXT:    s_mov_b32 s3, s0
+; GFX900-NEXT:    s_mov_b32 s8, s0
+; GFX900-NEXT:    s_mov_b32 s9, s0
+; GFX900-NEXT:    s_mov_b32 s10, s0
+; GFX900-NEXT:    s_mov_b32 s11, s0
+; GFX900-NEXT:    s_mov_b32 s12, s0
+; GFX900-NEXT:    s_mov_b32 s13, s0
+; GFX900-NEXT:    s_mov_b32 s14, s0
+; GFX900-NEXT:    s_mov_b32 s15, s0
+; GFX900-NEXT:    image_sample v[0:1], v[0:1], s[8:15], s[0:3] dmask:0x3
+; GFX900-NEXT:    s_mov_b32 s2, 1.0
+; GFX900-NEXT:  .LBB0_2: ; %Flow
+; GFX900-NEXT:    s_or_saveexec_b64 s[0:1], s[6:7]
+; GFX900-NEXT:    s_and_b64 exec, exec, s[4:5]
+; GFX900-NEXT:    s_and_b64 s[0:1], exec, s[0:1]
+; GFX900-NEXT:    v_mov_b32_e32 v2, s2
+; GFX900-NEXT:    s_xor_b64 exec, exec, s[0:1]
+; GFX900-NEXT:    s_cbranch_execz .LBB0_5
+; GFX900-NEXT:  ; %bb.3: ; %bb5
+; GFX900-NEXT:    s_andn2_b64 s[4:5], s[4:5], exec
+; GFX900-NEXT:    s_cbranch_scc0 .LBB0_6
+; GFX900-NEXT:  ; %bb.4: ; %bb5
+; GFX900-NEXT:    s_mov_b64 exec, 0
+; GFX900-NEXT:    s_waitcnt vmcnt(0)
+; GFX900-NEXT:    v_mov_b32_e32 v1, 0
+; GFX900-NEXT:    v_mov_b32_e32 v2, 0
+; GFX900-NEXT:  .LBB0_5: ; %bb6
+; GFX900-NEXT:    s_or_b64 exec, exec, s[0:1]
+; GFX900-NEXT:    s_waitcnt vmcnt(0)
+; GFX900-NEXT:    v_cvt_pkrtz_f16_f32 v1, 0, v1
+; GFX900-NEXT:    v_cvt_pkrtz_f16_f32 v0, v2, v0
+; GFX900-NEXT:    exp mrt0 off, off, off, off compr
+; GFX900-NEXT:    s_endpgm
+; GFX900-NEXT:  .LBB0_6:
+; GFX900-NEXT:    s_mov_b64 exec, 0
+; GFX900-NEXT:    exp null off, off, off, off done vm
+; GFX900-NEXT:    s_endpgm
+bb:
+  %i = fcmp olt float %arg, 0.000000e+00
+  br i1 %i, label %bb5, label %bb1
+
+bb1:
+  %i2 = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float %arg, <8 x i32> zeroinitializer, <4 x i32> zeroinitializer, i1 false, i32 0, i32 0)
+  %i3 = extractelement <4 x float> %i2, i64 1
+  %i4 = extractelement <4 x float> %i2, i64 0
+  br label %bb6
+
+bb5:
+  call void @llvm.amdgcn.kill(i1 false)
+  br label %bb6
+
+bb6:
+  %i7 = phi float [ 0.000000e+00, %bb5 ], [ %i3, %bb1 ]
+  %i8 = phi float [ 0.000000e+00, %bb5 ], [ 1.000000e+00, %bb1 ]
+  %i9 = phi float [ undef, %bb5 ], [ %i4, %bb1 ]
+  %i10 = call <2 x half> @llvm.amdgcn.cvt.pkrtz(float 0.000000e+00, float %i7)
+  %i11 = call <2 x half> @llvm.amdgcn.cvt.pkrtz(float %i8, float %i9)
+  call void @llvm.amdgcn.exp.compr.v2f16(i32 0, i32 0, <2 x half> %i10, <2 x half> %i11, i1 false, i1 false)
+  ret void
+}
diff --git a/llvm/test/CodeGen/AMDGPU/sdwa-peephole.ll b/llvm/test/CodeGen/AMDGPU/sdwa-peephole.ll
index 256fa6b2a6a75..d5a1d8f649adf 100644
--- a/llvm/test/CodeGen/AMDGPU/sdwa-peephole.ll
+++ b/llvm/test/CodeGen/AMDGPU/sdwa-peephole.ll
@@ -1,18 +1,60 @@
-; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=fiji -amdgpu-sdwa-peephole=0 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=NOSDWA,GCN %s
-; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=fiji -amdgpu-sdwa-peephole -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=VI,GFX89,SDWA,GCN %s
-; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=gfx900 -amdgpu-sdwa-peephole -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX9,GFX9_10,SDWA,GCN %s
-; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=gfx1010 -amdgpu-sdwa-peephole -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX9_10,SDWA,GCN %s
-
-; GCN-LABEL: {{^}}add_shr_i32:
-; NOSDWA: v_lshrrev_b32_e32 v[[DST:[0-9]+]], 16, v{{[0-9]+}}
-; NOSDWA: v_add_u32_e32 v{{[0-9]+}}, vcc, v{{[0-9]+}}, v[[DST]]
-; NOSDWA-NOT: v_add_{{(_co)?}}_u32_sdwa
-
-; VI: v_add_u32_sdwa v{{[0-9]+}}, vcc, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9: v_add_u32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX10: v_add_nc_u32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
+; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=fiji -amdgpu-sdwa-peephole=0 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,NOSDWA %s
+; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=fiji -amdgpu-sdwa-peephole -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,SDWA,GFX89 %s
+; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=gfx900 -amdgpu-sdwa-peephole -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,SDWA,GFX9_10,GFX9 %s
+; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=gfx1010 -amdgpu-sdwa-peephole -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,SDWA,GFX9_10,GFX10 %s
 
 define amdgpu_kernel void @add_shr_i32(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 {
+; NOSDWA-LABEL: add_shr_i32:
+; NOSDWA:       ; %bb.0:
+; NOSDWA-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s2
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s3
+; NOSDWA-NEXT:    flat_load_dword v2, v[0:1]
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s0
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s1
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
+; NOSDWA-NEXT:    v_add_u32_e32 v2, vcc, v2, v3
+; NOSDWA-NEXT:    flat_store_dword v[0:1], v2
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: add_shr_i32:
+; GFX89:       ; %bb.0:
+; GFX89-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v0, s2
+; GFX89-NEXT:    v_mov_b32_e32 v1, s3
+; GFX89-NEXT:    flat_load_dword v2, v[0:1]
+; GFX89-NEXT:    v_mov_b32_e32 v0, s0
+; GFX89-NEXT:    v_mov_b32_e32 v1, s1
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_add_u32_sdwa v2, vcc, v2, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX89-NEXT:    flat_store_dword v[0:1], v2
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: add_shr_i32:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dword v1, v0, s[2:3]
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_add_u32_sdwa v1, v1, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-NEXT:    global_store_dword v0, v1, s[0:1]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: add_shr_i32:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    global_load_dword v1, v0, s[2:3]
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_add_nc_u32_sdwa v1, v1, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX10-NEXT:    global_store_dword v0, v1, s[0:1]
+; GFX10-NEXT:    s_endpgm
   %a = load i32, ptr addrspace(1) %in, align 4
   %shr = lshr i32 %a, 16
   %add = add i32 %a, %shr
@@ -20,15 +62,57 @@ define amdgpu_kernel void @add_shr_i32(ptr addrspace(1) %out, ptr addrspace(1) %
   ret void
 }
 
-; GCN-LABEL: {{^}}sub_shr_i32:
-; NOSDWA: v_lshrrev_b32_e32 v[[DST:[0-9]+]], 16, v{{[0-9]+}}
-; NOSDWA: v_subrev_u32_e32 v{{[0-9]+}}, vcc, v{{[0-9]+}}, v[[DST]]
-; NOSDWA-NOT: v_subrev_{{(_co)?}}_u32_sdwa
-
-; VI: v_subrev_u32_sdwa v{{[0-9]+}}, vcc, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9: v_sub_u32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX10: v_sub_nc_u32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
 define amdgpu_kernel void @sub_shr_i32(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 {
+; NOSDWA-LABEL: sub_shr_i32:
+; NOSDWA:       ; %bb.0:
+; NOSDWA-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s2
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s3
+; NOSDWA-NEXT:    flat_load_dword v2, v[0:1]
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s0
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s1
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
+; NOSDWA-NEXT:    v_subrev_u32_e32 v2, vcc, v2, v3
+; NOSDWA-NEXT:    flat_store_dword v[0:1], v2
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: sub_shr_i32:
+; GFX89:       ; %bb.0:
+; GFX89-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v0, s2
+; GFX89-NEXT:    v_mov_b32_e32 v1, s3
+; GFX89-NEXT:    flat_load_dword v2, v[0:1]
+; GFX89-NEXT:    v_mov_b32_e32 v0, s0
+; GFX89-NEXT:    v_mov_b32_e32 v1, s1
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_subrev_u32_sdwa v2, vcc, v2, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX89-NEXT:    flat_store_dword v[0:1], v2
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: sub_shr_i32:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dword v1, v0, s[2:3]
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_sub_u32_sdwa v1, v1, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-NEXT:    global_store_dword v0, v1, s[0:1]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: sub_shr_i32:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    global_load_dword v1, v0, s[2:3]
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_sub_nc_u32_sdwa v1, v1, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-NEXT:    global_store_dword v0, v1, s[0:1]
+; GFX10-NEXT:    s_endpgm
   %a = load i32, ptr addrspace(1) %in, align 4
   %shr = lshr i32 %a, 16
   %sub = sub i32 %shr, %a
@@ -36,15 +120,81 @@ define amdgpu_kernel void @sub_shr_i32(ptr addrspace(1) %out, ptr addrspace(1) %
   ret void
 }
 
-; GCN-LABEL: {{^}}mul_shr_i32:
-; NOSDWA: v_lshrrev_b32_e32 v[[DST0:[0-9]+]], 16, v{{[0-9]+}}
-; NOSDWA: v_lshrrev_b32_e32 v[[DST1:[0-9]+]], 16, v{{[0-9]+}}
-; NOSDWA: v_mul_u32_u24_e32 v{{[0-9]+}}, v[[DST0]], v[[DST1]]
-; NOSDWA-NOT: v_mul_u32_u24_sdwa
-
-; SDWA: v_mul_u32_u24_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
-
 define amdgpu_kernel void @mul_shr_i32(ptr addrspace(1) %out, ptr addrspace(1) %in1, ptr addrspace(1) %in2) #0 {
+; NOSDWA-LABEL: mul_shr_i32:
+; NOSDWA:       ; %bb.0:
+; NOSDWA-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; NOSDWA-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v2, 2, v0
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s7
+; NOSDWA-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; NOSDWA-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; NOSDWA-NEXT:    v_mov_b32_e32 v3, s1
+; NOSDWA-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
+; NOSDWA-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
+; NOSDWA-NEXT:    flat_load_dword v4, v[0:1]
+; NOSDWA-NEXT:    flat_load_dword v2, v[2:3]
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s4
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s5
+; NOSDWA-NEXT:    s_waitcnt vmcnt(1)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v3, 16, v4
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v2, 16, v2
+; NOSDWA-NEXT:    v_mul_u32_u24_e32 v2, v3, v2
+; NOSDWA-NEXT:    flat_store_dword v[0:1], v2
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: mul_shr_i32:
+; GFX89:       ; %bb.0:
+; GFX89-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX89-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX89-NEXT:    v_lshlrev_b32_e32 v2, 2, v0
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v1, s7
+; GFX89-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; GFX89-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GFX89-NEXT:    v_mov_b32_e32 v3, s1
+; GFX89-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
+; GFX89-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
+; GFX89-NEXT:    flat_load_dword v4, v[0:1]
+; GFX89-NEXT:    flat_load_dword v2, v[2:3]
+; GFX89-NEXT:    v_mov_b32_e32 v0, s4
+; GFX89-NEXT:    v_mov_b32_e32 v1, s5
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_mul_u32_u24_sdwa v2, v4, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    flat_store_dword v[0:1], v2
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: mul_shr_i32:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dword v1, v0, s[6:7]
+; GFX9-NEXT:    global_load_dword v2, v0, s[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_mul_u32_u24_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX9-NEXT:    global_store_dword v0, v1, s[4:5]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: mul_shr_i32:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    global_load_dword v1, v0, s[6:7]
+; GFX10-NEXT:    global_load_dword v2, v0, s[2:3]
+; GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_mul_u32_u24_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX10-NEXT:    global_store_dword v0, v1, s[4:5]
+; GFX10-NEXT:    s_endpgm
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gep1 = getelementptr i32, ptr addrspace(1) %in1, i32 %idx
   %gep2 = getelementptr i32, ptr addrspace(1) %in2, i32 %idx
@@ -57,14 +207,78 @@ define amdgpu_kernel void @mul_shr_i32(ptr addrspace(1) %out, ptr addrspace(1) %
   ret void
 }
 
-; GCN-LABEL: {{^}}mul_i16:
-; NOSDWA: v_mul_lo_u16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA-NOT: v_mul_u32_u24_sdwa
-; GFX89: v_mul_lo_u16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; GFX10: v_mul_lo_u16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; SDWA-NOT: v_mul_u32_u24_sdwa
-
 define amdgpu_kernel void @mul_i16(ptr addrspace(1) %out, ptr addrspace(1) %ina, ptr addrspace(1) %inb) #0 {
+; NOSDWA-LABEL: mul_i16:
+; NOSDWA:       ; %bb.0: ; %entry
+; NOSDWA-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; NOSDWA-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v2, 1, v0
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s7
+; NOSDWA-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; NOSDWA-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; NOSDWA-NEXT:    v_mov_b32_e32 v3, s1
+; NOSDWA-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
+; NOSDWA-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
+; NOSDWA-NEXT:    flat_load_ushort v4, v[0:1]
+; NOSDWA-NEXT:    flat_load_ushort v2, v[2:3]
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s4
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s5
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v2, v4, v2
+; NOSDWA-NEXT:    flat_store_short v[0:1], v2
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: mul_i16:
+; GFX89:       ; %bb.0: ; %entry
+; GFX89-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX89-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX89-NEXT:    v_lshlrev_b32_e32 v2, 1, v0
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v1, s7
+; GFX89-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; GFX89-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GFX89-NEXT:    v_mov_b32_e32 v3, s1
+; GFX89-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
+; GFX89-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
+; GFX89-NEXT:    flat_load_ushort v4, v[0:1]
+; GFX89-NEXT:    flat_load_ushort v2, v[2:3]
+; GFX89-NEXT:    v_mov_b32_e32 v0, s4
+; GFX89-NEXT:    v_mov_b32_e32 v1, s5
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_mul_lo_u16_e32 v2, v4, v2
+; GFX89-NEXT:    flat_store_short v[0:1], v2
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: mul_i16:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_ushort v1, v0, s[6:7]
+; GFX9-NEXT:    global_load_ushort v2, v0, s[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_mul_lo_u16_e32 v1, v1, v2
+; GFX9-NEXT:    global_store_short v0, v1, s[4:5]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: mul_i16:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    global_load_ushort v1, v0, s[6:7]
+; GFX10-NEXT:    global_load_ushort v2, v0, s[2:3]
+; GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_mul_lo_u16 v1, v1, v2
+; GFX10-NEXT:    global_store_short v0, v1, s[4:5]
+; GFX10-NEXT:    s_endpgm
 entry:
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gepa = getelementptr i16, ptr addrspace(1) %ina, i32 %idx
@@ -76,21 +290,85 @@ entry:
   ret void
 }
 
-; GCN-LABEL: {{^}}mul_v2i16:
-; NOSDWA: v_lshrrev_b32_e32 v[[DST0:[0-9]+]], 16, v{{[0-9]+}}
-; NOSDWA: v_lshrrev_b32_e32 v[[DST1:[0-9]+]], 16, v{{[0-9]+}}
-; NOSDWA: v_mul_lo_u16_e32 v[[DST_MUL:[0-9]+]], v[[DST1]], v[[DST0]]
-; NOSDWA: v_lshlrev_b32_e32 v[[DST_SHL:[0-9]+]], 16, v[[DST_MUL]]
-; NOSDWA: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v[[DST_SHL]]
-; NOSDWA-NOT: v_mul_u32_u24_sdwa
-
-; VI-DAG: v_mul_lo_u16_e32 v[[DST_MUL_LO:[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}}
-; VI-DAG: v_mul_lo_u16_sdwa v[[DST_MUL_HI:[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
-; VI: v_or_b32_e32 v{{[0-9]+}}, v[[DST_MUL_LO]], v[[DST_MUL_HI]]
-
-; GFX9_10: v_pk_mul_lo_u16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-
 define amdgpu_kernel void @mul_v2i16(ptr addrspace(1) %out, ptr addrspace(1) %ina, ptr addrspace(1) %inb) #0 {
+; NOSDWA-LABEL: mul_v2i16:
+; NOSDWA:       ; %bb.0: ; %entry
+; NOSDWA-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; NOSDWA-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v2, 2, v0
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s7
+; NOSDWA-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; NOSDWA-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; NOSDWA-NEXT:    v_mov_b32_e32 v3, s1
+; NOSDWA-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
+; NOSDWA-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
+; NOSDWA-NEXT:    flat_load_dword v4, v[0:1]
+; NOSDWA-NEXT:    flat_load_dword v2, v[2:3]
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s4
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s5
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v3, v4, v2
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v2, 16, v2
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v4, 16, v4
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v2, v4, v2
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
+; NOSDWA-NEXT:    v_or_b32_e32 v2, v3, v2
+; NOSDWA-NEXT:    flat_store_dword v[0:1], v2
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: mul_v2i16:
+; GFX89:       ; %bb.0: ; %entry
+; GFX89-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX89-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX89-NEXT:    v_lshlrev_b32_e32 v2, 2, v0
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v1, s7
+; GFX89-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; GFX89-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GFX89-NEXT:    v_mov_b32_e32 v3, s1
+; GFX89-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
+; GFX89-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
+; GFX89-NEXT:    flat_load_dword v4, v[0:1]
+; GFX89-NEXT:    flat_load_dword v2, v[2:3]
+; GFX89-NEXT:    v_mov_b32_e32 v0, s4
+; GFX89-NEXT:    v_mov_b32_e32 v1, s5
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_mul_lo_u16_e32 v3, v4, v2
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v2, v4, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_or_b32_e32 v2, v3, v2
+; GFX89-NEXT:    flat_store_dword v[0:1], v2
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: mul_v2i16:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dword v1, v0, s[6:7]
+; GFX9-NEXT:    global_load_dword v2, v0, s[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_pk_mul_lo_u16 v1, v1, v2
+; GFX9-NEXT:    global_store_dword v0, v1, s[4:5]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: mul_v2i16:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    global_load_dword v1, v0, s[6:7]
+; GFX10-NEXT:    global_load_dword v2, v0, s[2:3]
+; GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_pk_mul_lo_u16 v1, v1, v2
+; GFX10-NEXT:    global_store_dword v0, v1, s[4:5]
+; GFX10-NEXT:    s_endpgm
 entry:
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gepa = getelementptr <2 x i16>, ptr addrspace(1) %ina, i32 %idx
@@ -102,25 +380,96 @@ entry:
   ret void
 }
 
-; GCN-LABEL: {{^}}mul_v4i16:
-; NOSDWA: v_lshrrev_b32_e32 v{{[0-9]+}}, 16, v{{[0-9]+}}
-; NOSDWA: v_lshrrev_b32_e32 v{{[0-9]+}}, 16, v{{[0-9]+}}
-; NOSDWA: v_mul_lo_u16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA: v_lshlrev_b32_e32 v{{[0-9]+}}, 16, v{{[0-9]+}}
-; NOSDWA: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA-NOT: v_mul_u32_u24_sdwa
-
-; VI-DAG: v_mul_lo_u16_e32 v[[DST_MUL0:[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}}
-; VI-DAG: v_mul_lo_u16_sdwa v[[DST_MUL1:[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
-; VI-DAG: v_mul_lo_u16_e32 v[[DST_MUL2:[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}}
-; VI-DAG: v_mul_lo_u16_sdwa v[[DST_MUL3:[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
-; VI-DAG: v_or_b32_e32 v{{[0-9]+}}, v[[DST_MUL2]], v[[DST_MUL3]]
-; VI-DAG: v_or_b32_e32 v{{[0-9]+}}, v[[DST_MUL0]], v[[DST_MUL1]]
-
-; GFX9_10-DAG: v_pk_mul_lo_u16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; GFX9_10-DAG: v_pk_mul_lo_u16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-
 define amdgpu_kernel void @mul_v4i16(ptr addrspace(1) %out, ptr addrspace(1) %ina, ptr addrspace(1) %inb) #0 {
+; NOSDWA-LABEL: mul_v4i16:
+; NOSDWA:       ; %bb.0: ; %entry
+; NOSDWA-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; NOSDWA-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s7
+; NOSDWA-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; NOSDWA-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; NOSDWA-NEXT:    v_mov_b32_e32 v3, s1
+; NOSDWA-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
+; NOSDWA-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
+; NOSDWA-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
+; NOSDWA-NEXT:    flat_load_dwordx2 v[2:3], v[2:3]
+; NOSDWA-NEXT:    v_mov_b32_e32 v4, s4
+; NOSDWA-NEXT:    v_mov_b32_e32 v5, s5
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v6, v1, v3
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v3, 16, v3
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v1, 16, v1
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v7, v0, v2
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v2, 16, v2
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v0, 16, v0
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v1, v1, v3
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v0, v0, v2
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v0, 16, v0
+; NOSDWA-NEXT:    v_or_b32_e32 v1, v6, v1
+; NOSDWA-NEXT:    v_or_b32_e32 v0, v7, v0
+; NOSDWA-NEXT:    flat_store_dwordx2 v[4:5], v[0:1]
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: mul_v4i16:
+; GFX89:       ; %bb.0: ; %entry
+; GFX89-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX89-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX89-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v1, s7
+; GFX89-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; GFX89-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GFX89-NEXT:    v_mov_b32_e32 v3, s1
+; GFX89-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
+; GFX89-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
+; GFX89-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
+; GFX89-NEXT:    flat_load_dwordx2 v[2:3], v[2:3]
+; GFX89-NEXT:    v_mov_b32_e32 v4, s4
+; GFX89-NEXT:    v_mov_b32_e32 v5, s5
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_mul_lo_u16_e32 v6, v1, v3
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v1, v1, v3 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_mul_lo_u16_e32 v3, v0, v2
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v0, v0, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_or_b32_e32 v1, v6, v1
+; GFX89-NEXT:    v_or_b32_e32 v0, v3, v0
+; GFX89-NEXT:    flat_store_dwordx2 v[4:5], v[0:1]
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: mul_v4i16:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX9-NEXT:    v_lshlrev_b32_e32 v4, 3, v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dwordx2 v[0:1], v4, s[6:7]
+; GFX9-NEXT:    global_load_dwordx2 v[2:3], v4, s[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v4, 0
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_pk_mul_lo_u16 v1, v1, v3
+; GFX9-NEXT:    v_pk_mul_lo_u16 v0, v0, v2
+; GFX9-NEXT:    global_store_dwordx2 v4, v[0:1], s[4:5]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: mul_v4i16:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX10-NEXT:    v_lshlrev_b32_e32 v4, 3, v0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    global_load_dwordx2 v[0:1], v4, s[6:7]
+; GFX10-NEXT:    global_load_dwordx2 v[2:3], v4, s[2:3]
+; GFX10-NEXT:    v_mov_b32_e32 v4, 0
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_pk_mul_lo_u16 v1, v1, v3
+; GFX10-NEXT:    v_pk_mul_lo_u16 v0, v0, v2
+; GFX10-NEXT:    global_store_dwordx2 v4, v[0:1], s[4:5]
+; GFX10-NEXT:    s_endpgm
 entry:
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gepa = getelementptr <4 x i16>, ptr addrspace(1) %ina, i32 %idx
@@ -132,33 +481,118 @@ entry:
   ret void
 }
 
-; GCN-LABEL: {{^}}mul_v8i16:
-; NOSDWA: v_lshrrev_b32_e32 v{{[0-9]+}}, 16, v{{[0-9]+}}
-; NOSDWA: v_lshrrev_b32_e32 v{{[0-9]+}}, 16, v{{[0-9]+}}
-; NOSDWA: v_mul_lo_u16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA: v_lshlrev_b32_e32 v{{[0-9]+}}, 16, v{{[0-9]+}}
-; NOSDWA: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA-NOT: v_mul_u32_u24_sdwa
-
-; VI-DAG: v_mul_lo_u16_e32 v[[DST_MUL0:[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}}
-; VI-DAG: v_mul_lo_u16_sdwa v[[DST_MUL1:[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
-; VI-DAG: v_mul_lo_u16_e32 v[[DST_MUL2:[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}}
-; VI-DAG: v_mul_lo_u16_sdwa v[[DST_MUL3:[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
-; VI-DAG: v_mul_lo_u16_e32 v[[DST_MUL4:[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}}
-; VI-DAG: v_mul_lo_u16_sdwa v[[DST_MUL5:[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
-; VI-DAG: v_mul_lo_u16_e32 v[[DST_MUL6:[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}}
-; VI-DAG: v_mul_lo_u16_sdwa v[[DST_MUL7:[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
-; VI-DAG: v_or_b32_e32 v{{[0-9]+}}, v[[DST_MUL6]], v[[DST_MUL7]]
-; VI-DAG: v_or_b32_e32 v{{[0-9]+}}, v[[DST_MUL4]], v[[DST_MUL5]]
-; VI-DAG: v_or_b32_e32 v{{[0-9]+}}, v[[DST_MUL2]], v[[DST_MUL3]]
-; VI-DAG: v_or_b32_e32 v{{[0-9]+}}, v[[DST_MUL0]], v[[DST_MUL1]]
-
-; GFX9_10-DAG: v_pk_mul_lo_u16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; GFX9_10-DAG: v_pk_mul_lo_u16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; GFX9_10-DAG: v_pk_mul_lo_u16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; GFX9_10-DAG: v_pk_mul_lo_u16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-
 define amdgpu_kernel void @mul_v8i16(ptr addrspace(1) %out, ptr addrspace(1) %ina, ptr addrspace(1) %inb) #0 {
+; NOSDWA-LABEL: mul_v8i16:
+; NOSDWA:       ; %bb.0: ; %entry
+; NOSDWA-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; NOSDWA-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v2, 4, v0
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s7
+; NOSDWA-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; NOSDWA-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; NOSDWA-NEXT:    v_mov_b32_e32 v3, s1
+; NOSDWA-NEXT:    v_add_u32_e32 v4, vcc, s0, v2
+; NOSDWA-NEXT:    v_addc_u32_e32 v5, vcc, 0, v3, vcc
+; NOSDWA-NEXT:    flat_load_dwordx4 v[0:3], v[0:1]
+; NOSDWA-NEXT:    flat_load_dwordx4 v[4:7], v[4:5]
+; NOSDWA-NEXT:    v_mov_b32_e32 v8, s4
+; NOSDWA-NEXT:    v_mov_b32_e32 v9, s5
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v10, v3, v7
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v7, 16, v7
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v3, 16, v3
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v11, v2, v6
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v6, 16, v6
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v2, 16, v2
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v12, v1, v5
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v5, 16, v5
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v1, 16, v1
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v13, v0, v4
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v4, 16, v4
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v0, 16, v0
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v3, v3, v7
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v2, v2, v6
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v1, v1, v5
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v0, v0, v4
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v0, 16, v0
+; NOSDWA-NEXT:    v_or_b32_e32 v3, v10, v3
+; NOSDWA-NEXT:    v_or_b32_e32 v2, v11, v2
+; NOSDWA-NEXT:    v_or_b32_e32 v1, v12, v1
+; NOSDWA-NEXT:    v_or_b32_e32 v0, v13, v0
+; NOSDWA-NEXT:    flat_store_dwordx4 v[8:9], v[0:3]
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: mul_v8i16:
+; GFX89:       ; %bb.0: ; %entry
+; GFX89-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX89-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX89-NEXT:    v_lshlrev_b32_e32 v2, 4, v0
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v1, s7
+; GFX89-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; GFX89-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GFX89-NEXT:    v_mov_b32_e32 v3, s1
+; GFX89-NEXT:    v_add_u32_e32 v4, vcc, s0, v2
+; GFX89-NEXT:    v_addc_u32_e32 v5, vcc, 0, v3, vcc
+; GFX89-NEXT:    flat_load_dwordx4 v[0:3], v[0:1]
+; GFX89-NEXT:    flat_load_dwordx4 v[4:7], v[4:5]
+; GFX89-NEXT:    v_mov_b32_e32 v8, s4
+; GFX89-NEXT:    v_mov_b32_e32 v9, s5
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_mul_lo_u16_e32 v10, v3, v7
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v3, v3, v7 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_mul_lo_u16_e32 v7, v2, v6
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v2, v2, v6 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_mul_lo_u16_e32 v6, v1, v5
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v1, v1, v5 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_mul_lo_u16_e32 v5, v0, v4
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v0, v0, v4 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_or_b32_e32 v3, v10, v3
+; GFX89-NEXT:    v_or_b32_e32 v2, v7, v2
+; GFX89-NEXT:    v_or_b32_e32 v1, v6, v1
+; GFX89-NEXT:    v_or_b32_e32 v0, v5, v0
+; GFX89-NEXT:    flat_store_dwordx4 v[8:9], v[0:3]
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: mul_v8i16:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX9-NEXT:    v_lshlrev_b32_e32 v8, 4, v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dwordx4 v[0:3], v8, s[6:7]
+; GFX9-NEXT:    global_load_dwordx4 v[4:7], v8, s[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v8, 0
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_pk_mul_lo_u16 v3, v3, v7
+; GFX9-NEXT:    v_pk_mul_lo_u16 v2, v2, v6
+; GFX9-NEXT:    v_pk_mul_lo_u16 v1, v1, v5
+; GFX9-NEXT:    v_pk_mul_lo_u16 v0, v0, v4
+; GFX9-NEXT:    global_store_dwordx4 v8, v[0:3], s[4:5]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: mul_v8i16:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX10-NEXT:    v_lshlrev_b32_e32 v8, 4, v0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    global_load_dwordx4 v[0:3], v8, s[6:7]
+; GFX10-NEXT:    global_load_dwordx4 v[4:7], v8, s[2:3]
+; GFX10-NEXT:    v_mov_b32_e32 v8, 0
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_pk_mul_lo_u16 v3, v3, v7
+; GFX10-NEXT:    v_pk_mul_lo_u16 v2, v2, v6
+; GFX10-NEXT:    v_pk_mul_lo_u16 v1, v1, v5
+; GFX10-NEXT:    v_pk_mul_lo_u16 v0, v0, v4
+; GFX10-NEXT:    global_store_dwordx4 v8, v[0:3], s[4:5]
+; GFX10-NEXT:    s_endpgm
 entry:
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gepa = getelementptr <8 x i16>, ptr addrspace(1) %ina, i32 %idx
@@ -170,13 +604,70 @@ entry:
   ret void
 }
 
-; GCN-LABEL: {{^}}mul_half:
-; NOSDWA: v_mul_f16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA-NOT: v_mul_f16_sdwa
-; SDWA: v_mul_f16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; SDWA-NOT: v_mul_f16_sdwa
-
 define amdgpu_kernel void @mul_half(ptr addrspace(1) %out, ptr addrspace(1) %ina, ptr addrspace(1) %inb) #0 {
+; NOSDWA-LABEL: mul_half:
+; NOSDWA:       ; %bb.0: ; %entry
+; NOSDWA-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; NOSDWA-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s6
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s7
+; NOSDWA-NEXT:    v_mov_b32_e32 v2, s0
+; NOSDWA-NEXT:    v_mov_b32_e32 v3, s1
+; NOSDWA-NEXT:    flat_load_ushort v4, v[0:1]
+; NOSDWA-NEXT:    flat_load_ushort v2, v[2:3]
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s4
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s5
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_mul_f16_e32 v2, v4, v2
+; NOSDWA-NEXT:    flat_store_short v[0:1], v2
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: mul_half:
+; GFX89:       ; %bb.0: ; %entry
+; GFX89-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX89-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v0, s6
+; GFX89-NEXT:    v_mov_b32_e32 v1, s7
+; GFX89-NEXT:    v_mov_b32_e32 v2, s0
+; GFX89-NEXT:    v_mov_b32_e32 v3, s1
+; GFX89-NEXT:    flat_load_ushort v4, v[0:1]
+; GFX89-NEXT:    flat_load_ushort v2, v[2:3]
+; GFX89-NEXT:    v_mov_b32_e32 v0, s4
+; GFX89-NEXT:    v_mov_b32_e32 v1, s5
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_mul_f16_e32 v2, v4, v2
+; GFX89-NEXT:    flat_store_short v[0:1], v2
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: mul_half:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_ushort v1, v0, s[6:7]
+; GFX9-NEXT:    global_load_ushort v2, v0, s[2:3]
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_mul_f16_e32 v1, v1, v2
+; GFX9-NEXT:    global_store_short v0, v1, s[4:5]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: mul_half:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    global_load_ushort v1, v0, s[6:7]
+; GFX10-NEXT:    global_load_ushort v2, v0, s[2:3]
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_mul_f16_e32 v1, v1, v2
+; GFX10-NEXT:    global_store_short v0, v1, s[4:5]
+; GFX10-NEXT:    s_endpgm
 entry:
   %a = load half, ptr addrspace(1) %ina, align 4
   %b = load half, ptr addrspace(1) %inb, align 4
@@ -185,21 +676,78 @@ entry:
   ret void
 }
 
-; GCN-LABEL: {{^}}mul_v2half:
-; NOSDWA: v_lshrrev_b32_e32 v[[DST1:[0-9]+]], 16, v{{[0-9]+}}
-; NOSDWA: v_lshrrev_b32_e32 v[[DST0:[0-9]+]], 16, v{{[0-9]+}}
-; NOSDWA: v_mul_f16_e32 v[[DST_MUL:[0-9]+]], v[[DST0]], v[[DST1]]
-; NOSDWA: v_lshlrev_b32_e32 v[[DST_SHL:[0-9]+]], 16, v[[DST_MUL]]
-; NOSDWA: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v[[DST_SHL]]
-; NOSDWA-NOT: v_mul_f16_sdwa
-
-; VI-DAG: v_mul_f16_sdwa v[[DST_MUL_HI:[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
-; VI-DAG: v_mul_f16_e32 v[[DST_MUL_LO:[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}}
-; VI: v_or_b32_e32 v{{[0-9]+}}, v[[DST_MUL_LO]], v[[DST_MUL_HI]]
-
-; GFX9_10: v_pk_mul_f16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-
 define amdgpu_kernel void @mul_v2half(ptr addrspace(1) %out, ptr addrspace(1) %ina, ptr addrspace(1) %inb) #0 {
+; NOSDWA-LABEL: mul_v2half:
+; NOSDWA:       ; %bb.0: ; %entry
+; NOSDWA-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; NOSDWA-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s6
+; NOSDWA-NEXT:    v_mov_b32_e32 v2, s0
+; NOSDWA-NEXT:    v_mov_b32_e32 v3, s1
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s7
+; NOSDWA-NEXT:    flat_load_dword v2, v[2:3]
+; NOSDWA-NEXT:    flat_load_dword v3, v[0:1]
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s4
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s5
+; NOSDWA-NEXT:    s_waitcnt vmcnt(1)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v4, 16, v2
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v5, 16, v3
+; NOSDWA-NEXT:    v_mul_f16_e32 v4, v5, v4
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
+; NOSDWA-NEXT:    v_mul_f16_e32 v2, v3, v2
+; NOSDWA-NEXT:    v_or_b32_e32 v2, v2, v4
+; NOSDWA-NEXT:    flat_store_dword v[0:1], v2
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: mul_v2half:
+; GFX89:       ; %bb.0: ; %entry
+; GFX89-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX89-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v0, s6
+; GFX89-NEXT:    v_mov_b32_e32 v1, s7
+; GFX89-NEXT:    v_mov_b32_e32 v2, s0
+; GFX89-NEXT:    v_mov_b32_e32 v3, s1
+; GFX89-NEXT:    flat_load_dword v4, v[0:1]
+; GFX89-NEXT:    flat_load_dword v2, v[2:3]
+; GFX89-NEXT:    v_mov_b32_e32 v0, s4
+; GFX89-NEXT:    v_mov_b32_e32 v1, s5
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_mul_f16_sdwa v3, v4, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_mul_f16_e32 v2, v4, v2
+; GFX89-NEXT:    v_or_b32_e32 v2, v2, v3
+; GFX89-NEXT:    flat_store_dword v[0:1], v2
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: mul_v2half:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dword v1, v0, s[6:7]
+; GFX9-NEXT:    global_load_dword v2, v0, s[2:3]
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_pk_mul_f16 v1, v1, v2
+; GFX9-NEXT:    global_store_dword v0, v1, s[4:5]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: mul_v2half:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    global_load_dword v1, v0, s[6:7]
+; GFX10-NEXT:    global_load_dword v2, v0, s[2:3]
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_pk_mul_f16 v1, v1, v2
+; GFX10-NEXT:    global_store_dword v0, v1, s[4:5]
+; GFX10-NEXT:    s_endpgm
 entry:
   %a = load <2 x half>, ptr addrspace(1) %ina, align 4
   %b = load <2 x half>, ptr addrspace(1) %inb, align 4
@@ -208,23 +756,89 @@ entry:
   ret void
 }
 
-; GCN-LABEL: {{^}}mul_v4half:
-; NOSDWA: v_lshrrev_b32_e32 v{{[0-9]+}}, 16, v{{[0-9]+}}
-; NOSDWA: v_lshrrev_b32_e32 v{{[0-9]+}}, 16, v{{[0-9]+}}
-; NOSDWA: v_mul_f16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA: v_lshlrev_b32_e32 v{{[0-9]+}}, 16, v{{[0-9]+}}
-; NOSDWA: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA-NOT: v_mul_f16_sdwa
-
-; VI-DAG: v_mul_f16_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
-; VI-DAG: v_mul_f16_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
-; VI-DAG: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; VI-DAG: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-
-; GFX9_10-DAG: v_pk_mul_f16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; GFX9_10-DAG: v_pk_mul_f16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-
 define amdgpu_kernel void @mul_v4half(ptr addrspace(1) %out, ptr addrspace(1) %ina, ptr addrspace(1) %inb) #0 {
+; NOSDWA-LABEL: mul_v4half:
+; NOSDWA:       ; %bb.0: ; %entry
+; NOSDWA-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; NOSDWA-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s6
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s7
+; NOSDWA-NEXT:    v_mov_b32_e32 v2, s0
+; NOSDWA-NEXT:    v_mov_b32_e32 v3, s1
+; NOSDWA-NEXT:    flat_load_dwordx2 v[2:3], v[2:3]
+; NOSDWA-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
+; NOSDWA-NEXT:    v_mov_b32_e32 v4, s4
+; NOSDWA-NEXT:    v_mov_b32_e32 v5, s5
+; NOSDWA-NEXT:    s_waitcnt vmcnt(1)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v6, 16, v3
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v7, 16, v1
+; NOSDWA-NEXT:    v_mul_f16_e32 v1, v1, v3
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v8, 16, v0
+; NOSDWA-NEXT:    v_mul_f16_e32 v0, v0, v2
+; NOSDWA-NEXT:    v_mul_f16_e32 v2, v7, v6
+; NOSDWA-NEXT:    v_mul_f16_e32 v3, v8, v3
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
+; NOSDWA-NEXT:    v_or_b32_e32 v1, v1, v2
+; NOSDWA-NEXT:    v_or_b32_e32 v0, v0, v3
+; NOSDWA-NEXT:    flat_store_dwordx2 v[4:5], v[0:1]
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: mul_v4half:
+; GFX89:       ; %bb.0: ; %entry
+; GFX89-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX89-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v0, s6
+; GFX89-NEXT:    v_mov_b32_e32 v1, s7
+; GFX89-NEXT:    v_mov_b32_e32 v2, s0
+; GFX89-NEXT:    v_mov_b32_e32 v3, s1
+; GFX89-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
+; GFX89-NEXT:    flat_load_dwordx2 v[2:3], v[2:3]
+; GFX89-NEXT:    v_mov_b32_e32 v4, s4
+; GFX89-NEXT:    v_mov_b32_e32 v5, s5
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_mul_f16_sdwa v6, v1, v3 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_mul_f16_e32 v1, v1, v3
+; GFX89-NEXT:    v_mul_f16_sdwa v3, v0, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_mul_f16_e32 v0, v0, v2
+; GFX89-NEXT:    v_or_b32_e32 v1, v1, v6
+; GFX89-NEXT:    v_or_b32_e32 v0, v0, v3
+; GFX89-NEXT:    flat_store_dwordx2 v[4:5], v[0:1]
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: mul_v4half:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX9-NEXT:    v_mov_b32_e32 v4, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dwordx2 v[0:1], v4, s[6:7]
+; GFX9-NEXT:    global_load_dwordx2 v[2:3], v4, s[2:3]
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_pk_mul_f16 v1, v1, v3
+; GFX9-NEXT:    v_pk_mul_f16 v0, v0, v2
+; GFX9-NEXT:    global_store_dwordx2 v4, v[0:1], s[4:5]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: mul_v4half:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX10-NEXT:    v_mov_b32_e32 v4, 0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    global_load_dwordx2 v[0:1], v4, s[6:7]
+; GFX10-NEXT:    global_load_dwordx2 v[2:3], v4, s[2:3]
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_pk_mul_f16 v1, v1, v3
+; GFX10-NEXT:    v_pk_mul_f16 v0, v0, v2
+; GFX10-NEXT:    global_store_dwordx2 v4, v[0:1], s[4:5]
+; GFX10-NEXT:    s_endpgm
 entry:
   %a = load <4 x half>, ptr addrspace(1) %ina, align 4
   %b = load <4 x half>, ptr addrspace(1) %inb, align 4
@@ -233,29 +847,111 @@ entry:
   ret void
 }
 
-; GCN-LABEL: {{^}}mul_v8half:
-; NOSDWA: v_lshrrev_b32_e32 v{{[0-9]+}}, 16, v{{[0-9]+}}
-; NOSDWA: v_lshrrev_b32_e32 v{{[0-9]+}}, 16, v{{[0-9]+}}
-; NOSDWA: v_mul_f16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA: v_lshlrev_b32_e32 v{{[0-9]+}}, 16, v{{[0-9]+}}
-; NOSDWA: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA-NOT: v_mul_f16_sdwa
-
-; VI-DAG: v_mul_f16_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
-; VI-DAG: v_mul_f16_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
-; VI-DAG: v_mul_f16_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
-; VI-DAG: v_mul_f16_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
-; VI-DAG: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; VI-DAG: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; VI-DAG: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; VI-DAG: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-
-; GFX9_10-DAG: v_pk_mul_f16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; GFX9_10-DAG: v_pk_mul_f16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; GFX9_10-DAG: v_pk_mul_f16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; GFX9_10-DAG: v_pk_mul_f16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-
 define amdgpu_kernel void @mul_v8half(ptr addrspace(1) %out, ptr addrspace(1) %ina, ptr addrspace(1) %inb) #0 {
+; NOSDWA-LABEL: mul_v8half:
+; NOSDWA:       ; %bb.0: ; %entry
+; NOSDWA-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; NOSDWA-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v4, s6
+; NOSDWA-NEXT:    v_mov_b32_e32 v5, s7
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s0
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s1
+; NOSDWA-NEXT:    flat_load_dwordx4 v[0:3], v[0:1]
+; NOSDWA-NEXT:    flat_load_dwordx4 v[4:7], v[4:5]
+; NOSDWA-NEXT:    v_mov_b32_e32 v8, s4
+; NOSDWA-NEXT:    v_mov_b32_e32 v9, s5
+; NOSDWA-NEXT:    s_waitcnt vmcnt(1)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v10, 16, v3
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v11, 16, v7
+; NOSDWA-NEXT:    v_mul_f16_e32 v3, v7, v3
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v7, 16, v2
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v12, 16, v6
+; NOSDWA-NEXT:    v_mul_f16_e32 v2, v6, v2
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v6, 16, v1
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v13, 16, v5
+; NOSDWA-NEXT:    v_mul_f16_e32 v1, v5, v1
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v5, 16, v0
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v14, 16, v4
+; NOSDWA-NEXT:    v_mul_f16_e32 v0, v4, v0
+; NOSDWA-NEXT:    v_mul_f16_e32 v4, v11, v10
+; NOSDWA-NEXT:    v_mul_f16_e32 v7, v12, v7
+; NOSDWA-NEXT:    v_mul_f16_e32 v6, v13, v6
+; NOSDWA-NEXT:    v_mul_f16_e32 v5, v14, v5
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v7, 16, v7
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v6, 16, v6
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
+; NOSDWA-NEXT:    v_or_b32_e32 v3, v3, v4
+; NOSDWA-NEXT:    v_or_b32_e32 v2, v2, v7
+; NOSDWA-NEXT:    v_or_b32_e32 v1, v1, v6
+; NOSDWA-NEXT:    v_or_b32_e32 v0, v0, v5
+; NOSDWA-NEXT:    flat_store_dwordx4 v[8:9], v[0:3]
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: mul_v8half:
+; GFX89:       ; %bb.0: ; %entry
+; GFX89-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX89-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v0, s6
+; GFX89-NEXT:    v_mov_b32_e32 v1, s7
+; GFX89-NEXT:    v_mov_b32_e32 v4, s0
+; GFX89-NEXT:    v_mov_b32_e32 v5, s1
+; GFX89-NEXT:    flat_load_dwordx4 v[0:3], v[0:1]
+; GFX89-NEXT:    flat_load_dwordx4 v[4:7], v[4:5]
+; GFX89-NEXT:    v_mov_b32_e32 v8, s4
+; GFX89-NEXT:    v_mov_b32_e32 v9, s5
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_mul_f16_sdwa v10, v3, v7 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_mul_f16_e32 v3, v3, v7
+; GFX89-NEXT:    v_mul_f16_sdwa v7, v2, v6 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_mul_f16_e32 v2, v2, v6
+; GFX89-NEXT:    v_mul_f16_sdwa v6, v1, v5 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_mul_f16_e32 v1, v1, v5
+; GFX89-NEXT:    v_mul_f16_sdwa v5, v0, v4 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_mul_f16_e32 v0, v0, v4
+; GFX89-NEXT:    v_or_b32_e32 v3, v3, v10
+; GFX89-NEXT:    v_or_b32_e32 v2, v2, v7
+; GFX89-NEXT:    v_or_b32_e32 v1, v1, v6
+; GFX89-NEXT:    v_or_b32_e32 v0, v0, v5
+; GFX89-NEXT:    flat_store_dwordx4 v[8:9], v[0:3]
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: mul_v8half:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX9-NEXT:    v_mov_b32_e32 v8, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dwordx4 v[0:3], v8, s[6:7]
+; GFX9-NEXT:    global_load_dwordx4 v[4:7], v8, s[2:3]
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_pk_mul_f16 v3, v3, v7
+; GFX9-NEXT:    v_pk_mul_f16 v2, v2, v6
+; GFX9-NEXT:    v_pk_mul_f16 v1, v1, v5
+; GFX9-NEXT:    v_pk_mul_f16 v0, v0, v4
+; GFX9-NEXT:    global_store_dwordx4 v8, v[0:3], s[4:5]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: mul_v8half:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX10-NEXT:    v_mov_b32_e32 v8, 0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    global_load_dwordx4 v[0:3], v8, s[6:7]
+; GFX10-NEXT:    global_load_dwordx4 v[4:7], v8, s[2:3]
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_pk_mul_f16 v3, v3, v7
+; GFX10-NEXT:    v_pk_mul_f16 v2, v2, v6
+; GFX10-NEXT:    v_pk_mul_f16 v1, v1, v5
+; GFX10-NEXT:    v_pk_mul_f16 v0, v0, v4
+; GFX10-NEXT:    global_store_dwordx4 v8, v[0:3], s[4:5]
+; GFX10-NEXT:    s_endpgm
 entry:
   %a = load <8 x half>, ptr addrspace(1) %ina, align 4
   %b = load <8 x half>, ptr addrspace(1) %inb, align 4
@@ -264,14 +960,74 @@ entry:
   ret void
 }
 
-; GCN-LABEL: {{^}}mul_i8:
-; NOSDWA: v_mul_lo_u16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA-NOT: v_mul_u32_u24_sdwa
-; GFX89: v_mul_lo_u16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; GFX10: v_mul_lo_u16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; SDWA-NOT: v_mul_u32_u24_sdwa
-
 define amdgpu_kernel void @mul_i8(ptr addrspace(1) %out, ptr addrspace(1) %ina, ptr addrspace(1) %inb) #0 {
+; NOSDWA-LABEL: mul_i8:
+; NOSDWA:       ; %bb.0: ; %entry
+; NOSDWA-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; NOSDWA-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v2, s7
+; NOSDWA-NEXT:    v_add_u32_e32 v1, vcc, s6, v0
+; NOSDWA-NEXT:    v_addc_u32_e32 v2, vcc, 0, v2, vcc
+; NOSDWA-NEXT:    v_mov_b32_e32 v4, s1
+; NOSDWA-NEXT:    v_add_u32_e32 v3, vcc, s0, v0
+; NOSDWA-NEXT:    v_addc_u32_e32 v4, vcc, 0, v4, vcc
+; NOSDWA-NEXT:    flat_load_ubyte v2, v[1:2]
+; NOSDWA-NEXT:    flat_load_ubyte v3, v[3:4]
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s4
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s5
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v2, v2, v3
+; NOSDWA-NEXT:    flat_store_byte v[0:1], v2
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: mul_i8:
+; GFX89:       ; %bb.0: ; %entry
+; GFX89-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX89-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v2, s7
+; GFX89-NEXT:    v_add_u32_e32 v1, vcc, s6, v0
+; GFX89-NEXT:    v_addc_u32_e32 v2, vcc, 0, v2, vcc
+; GFX89-NEXT:    v_mov_b32_e32 v4, s1
+; GFX89-NEXT:    v_add_u32_e32 v3, vcc, s0, v0
+; GFX89-NEXT:    v_addc_u32_e32 v4, vcc, 0, v4, vcc
+; GFX89-NEXT:    flat_load_ubyte v2, v[1:2]
+; GFX89-NEXT:    flat_load_ubyte v3, v[3:4]
+; GFX89-NEXT:    v_mov_b32_e32 v0, s4
+; GFX89-NEXT:    v_mov_b32_e32 v1, s5
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_mul_lo_u16_e32 v2, v2, v3
+; GFX89-NEXT:    flat_store_byte v[0:1], v2
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: mul_i8:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_ubyte v1, v0, s[6:7]
+; GFX9-NEXT:    global_load_ubyte v2, v0, s[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_mul_lo_u16_e32 v1, v1, v2
+; GFX9-NEXT:    global_store_byte v0, v1, s[4:5]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: mul_i8:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    global_load_ubyte v1, v0, s[6:7]
+; GFX10-NEXT:    global_load_ubyte v2, v0, s[2:3]
+; GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_mul_lo_u16 v1, v1, v2
+; GFX10-NEXT:    global_store_byte v0, v1, s[4:5]
+; GFX10-NEXT:    s_endpgm
 entry:
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gepa = getelementptr i8, ptr addrspace(1) %ina, i32 %idx
@@ -283,27 +1039,95 @@ entry:
   ret void
 }
 
-; GCN-LABEL: {{^}}mul_v2i8:
-; NOSDWA: v_lshrrev_b16_e32 v{{[0-9]+}}, 8, v{{[0-9]+}}
-; NOSDWA: v_lshrrev_b16_e32 v{{[0-9]+}}, 8, v{{[0-9]+}}
-; NOSDWA: v_mul_lo_u16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA: v_lshlrev_b16_e32 v{{[0-9]+}}, 8, v{{[0-9]+}}
-; NOSDWA: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA-NOT: v_mul_u32_u24_sdwa
-
-; VI: v_mul_lo_u16_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
-
-; GFX9-DAG: v_mul_lo_u16_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
-; GFX9-DAG: v_mul_lo_u16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-
-; GFX10-DAG: v_mul_lo_u16
-; GFX10-DAG: v_mul_lo_u16
-
-; GFX9: v_or_b32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-
-; GFX10: v_lshlrev_b16 v{{[0-9]+}}, 8, v
-; GFX10: v_or_b32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 define amdgpu_kernel void @mul_v2i8(ptr addrspace(1) %out, ptr addrspace(1) %ina, ptr addrspace(1) %inb) #0 {
+; NOSDWA-LABEL: mul_v2i8:
+; NOSDWA:       ; %bb.0: ; %entry
+; NOSDWA-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; NOSDWA-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v2, 1, v0
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s7
+; NOSDWA-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; NOSDWA-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; NOSDWA-NEXT:    v_mov_b32_e32 v3, s1
+; NOSDWA-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
+; NOSDWA-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
+; NOSDWA-NEXT:    flat_load_ushort v4, v[0:1]
+; NOSDWA-NEXT:    flat_load_ushort v2, v[2:3]
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s4
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s5
+; NOSDWA-NEXT:    s_waitcnt vmcnt(1)
+; NOSDWA-NEXT:    v_lshrrev_b16_e32 v3, 8, v4
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_lshrrev_b16_e32 v5, 8, v2
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v2, v4, v2
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v3, v3, v5
+; NOSDWA-NEXT:    v_and_b32_e32 v2, 0xff, v2
+; NOSDWA-NEXT:    v_lshlrev_b16_e32 v3, 8, v3
+; NOSDWA-NEXT:    v_or_b32_e32 v2, v2, v3
+; NOSDWA-NEXT:    flat_store_short v[0:1], v2
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: mul_v2i8:
+; GFX89:       ; %bb.0: ; %entry
+; GFX89-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX89-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX89-NEXT:    v_lshlrev_b32_e32 v2, 1, v0
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v1, s7
+; GFX89-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; GFX89-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GFX89-NEXT:    v_mov_b32_e32 v3, s1
+; GFX89-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
+; GFX89-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
+; GFX89-NEXT:    flat_load_ushort v4, v[0:1]
+; GFX89-NEXT:    flat_load_ushort v2, v[2:3]
+; GFX89-NEXT:    v_mov_b32_e32 v0, s4
+; GFX89-NEXT:    v_mov_b32_e32 v1, s5
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_mul_lo_u16_e32 v3, v4, v2
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v2, v4, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
+; GFX89-NEXT:    v_or_b32_sdwa v2, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX89-NEXT:    flat_store_short v[0:1], v2
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: mul_v2i8:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_ushort v1, v0, s[6:7]
+; GFX9-NEXT:    global_load_ushort v2, v0, s[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_mul_lo_u16_e32 v3, v1, v2
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v1, v1, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
+; GFX9-NEXT:    v_or_b32_sdwa v1, v3, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    global_store_short v0, v1, s[4:5]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: mul_v2i8:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    global_load_ushort v1, v0, s[6:7]
+; GFX10-NEXT:    global_load_ushort v2, v0, s[2:3]
+; GFX10-NEXT:    s_waitcnt vmcnt(1)
+; GFX10-NEXT:    v_lshrrev_b16 v0, 8, v1
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_lshrrev_b16 v3, 8, v2
+; GFX10-NEXT:    v_mul_lo_u16 v1, v1, v2
+; GFX10-NEXT:    v_mov_b32_e32 v2, 0
+; GFX10-NEXT:    v_mul_lo_u16 v0, v0, v3
+; GFX10-NEXT:    v_lshlrev_b16 v0, 8, v0
+; GFX10-NEXT:    v_or_b32_sdwa v0, v1, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-NEXT:    global_store_short v2, v0, s[4:5]
+; GFX10-NEXT:    s_endpgm
 entry:
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gepa = getelementptr <2 x i8>, ptr addrspace(1) %ina, i32 %idx
@@ -315,28 +1139,124 @@ entry:
   ret void
 }
 
-; GCN-LABEL: {{^}}mul_v4i8:
-; NOSDWA: v_lshrrev_b16_e32 v{{[0-9]+}}, 8, v{{[0-9]+}}
-; NOSDWA: v_lshrrev_b16_e32 v{{[0-9]+}}, 8, v{{[0-9]+}}
-; NOSDWA: v_mul_lo_u16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA: v_lshlrev_b16_e32 v{{[0-9]+}}, 8, v{{[0-9]+}}
-; NOSDWA: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA-NOT: v_mul_u32_u24_sdwa
-
-; VI-DAG: v_mul_lo_u16_sdwa
-; VI-DAG: v_mul_lo_u16_sdwa
-; VI-DAG: v_mul_lo_u16_sdwa
-
-; GFX9-DAG: v_mul_lo_u16_sdwa
-; GFX9-DAG: v_mul_lo_u16_sdwa
-; GFX9-DAG: v_mul_lo_u16_sdwa
-
-; GFX10-DAG: v_mul_lo_u16
-; GFX10-DAG: v_mul_lo_u16
-; GFX10-DAG: v_mul_lo_u16
-; GFX10-DAG: v_mul_lo_u16
-
 define amdgpu_kernel void @mul_v4i8(ptr addrspace(1) %out, ptr addrspace(1) %ina, ptr addrspace(1) %inb) #0 {
+; NOSDWA-LABEL: mul_v4i8:
+; NOSDWA:       ; %bb.0: ; %entry
+; NOSDWA-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; NOSDWA-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v2, 2, v0
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s7
+; NOSDWA-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; NOSDWA-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; NOSDWA-NEXT:    v_mov_b32_e32 v3, s1
+; NOSDWA-NEXT:    flat_load_dword v4, v[0:1]
+; NOSDWA-NEXT:    v_add_u32_e32 v0, vcc, s0, v2
+; NOSDWA-NEXT:    v_addc_u32_e32 v1, vcc, 0, v3, vcc
+; NOSDWA-NEXT:    flat_load_dword v2, v[0:1]
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s4
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s5
+; NOSDWA-NEXT:    s_waitcnt vmcnt(1)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v3, 16, v4
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v5, 24, v4
+; NOSDWA-NEXT:    v_lshrrev_b16_e32 v6, 8, v4
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v7, 16, v2
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v8, 24, v2
+; NOSDWA-NEXT:    v_lshrrev_b16_e32 v9, 8, v2
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v2, v4, v2
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v4, v6, v9
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v5, v5, v8
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v3, v3, v7
+; NOSDWA-NEXT:    v_and_b32_e32 v2, 0xff, v2
+; NOSDWA-NEXT:    v_lshlrev_b16_e32 v4, 8, v4
+; NOSDWA-NEXT:    v_lshlrev_b16_e32 v5, 8, v5
+; NOSDWA-NEXT:    v_and_b32_e32 v3, 0xff, v3
+; NOSDWA-NEXT:    v_or_b32_e32 v2, v2, v4
+; NOSDWA-NEXT:    v_or_b32_e32 v3, v3, v5
+; NOSDWA-NEXT:    v_and_b32_e32 v2, 0xffff, v2
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
+; NOSDWA-NEXT:    v_or_b32_e32 v2, v2, v3
+; NOSDWA-NEXT:    flat_store_dword v[0:1], v2
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: mul_v4i8:
+; GFX89:       ; %bb.0: ; %entry
+; GFX89-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX89-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX89-NEXT:    v_lshlrev_b32_e32 v2, 2, v0
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v1, s7
+; GFX89-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; GFX89-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GFX89-NEXT:    v_mov_b32_e32 v3, s1
+; GFX89-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
+; GFX89-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
+; GFX89-NEXT:    flat_load_dword v4, v[0:1]
+; GFX89-NEXT:    flat_load_dword v2, v[2:3]
+; GFX89-NEXT:    v_mov_b32_e32 v0, s4
+; GFX89-NEXT:    v_mov_b32_e32 v1, s5
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_mul_lo_u16_e32 v3, v4, v2
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v5, v4, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v6, v4, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:BYTE_3
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v2, v4, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_or_b32_sdwa v3, v3, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX89-NEXT:    v_or_b32_sdwa v2, v2, v6 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX89-NEXT:    v_or_b32_sdwa v2, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX89-NEXT:    flat_store_dword v[0:1], v2
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: mul_v4i8:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dword v1, v0, s[6:7]
+; GFX9-NEXT:    global_load_dword v2, v0, s[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_mul_lo_u16_e32 v3, v1, v2
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v4, v1, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v5, v1, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:BYTE_3
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX9-NEXT:    v_or_b32_sdwa v2, v3, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v1, v1, v5 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v1, v2, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX9-NEXT:    global_store_dword v0, v1, s[4:5]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: mul_v4i8:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    global_load_dword v1, v0, s[6:7]
+; GFX10-NEXT:    global_load_dword v2, v0, s[2:3]
+; GFX10-NEXT:    s_waitcnt vmcnt(1)
+; GFX10-NEXT:    v_lshrrev_b32_e32 v0, 24, v1
+; GFX10-NEXT:    v_lshrrev_b16 v3, 8, v1
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_lshrrev_b16 v4, 8, v2
+; GFX10-NEXT:    v_lshrrev_b32_e32 v5, 24, v2
+; GFX10-NEXT:    v_lshrrev_b32_e32 v6, 16, v1
+; GFX10-NEXT:    v_lshrrev_b32_e32 v7, 16, v2
+; GFX10-NEXT:    v_mul_lo_u16 v1, v1, v2
+; GFX10-NEXT:    v_mul_lo_u16 v3, v3, v4
+; GFX10-NEXT:    v_mul_lo_u16 v0, v0, v5
+; GFX10-NEXT:    v_mul_lo_u16 v2, v6, v7
+; GFX10-NEXT:    v_lshlrev_b16 v3, 8, v3
+; GFX10-NEXT:    v_lshlrev_b16 v0, 8, v0
+; GFX10-NEXT:    v_or_b32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-NEXT:    v_or_b32_sdwa v0, v2, v0 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-NEXT:    v_mov_b32_e32 v2, 0
+; GFX10-NEXT:    v_or_b32_sdwa v0, v1, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX10-NEXT:    global_store_dword v2, v0, s[4:5]
+; GFX10-NEXT:    s_endpgm
 entry:
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gepa = getelementptr <4 x i8>, ptr addrspace(1) %ina, i32 %idx
@@ -348,38 +1268,172 @@ entry:
   ret void
 }
 
-; GCN-LABEL: {{^}}mul_v8i8:
-; NOSDWA: v_lshrrev_b16_e32 v{{[0-9]+}}, 8, v{{[0-9]+}}
-; NOSDWA: v_lshrrev_b16_e32 v{{[0-9]+}}, 8, v{{[0-9]+}}
-; NOSDWA: v_mul_lo_u16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA: v_lshlrev_b16_e32 v{{[0-9]+}}, 8, v{{[0-9]+}}
-; NOSDWA: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA-NOT: v_mul_u32_u24_sdwa
-
-; VI-DAG: v_mul_lo_u16_sdwa
-; VI-DAG: v_mul_lo_u16_sdwa
-; VI-DAG: v_mul_lo_u16_sdwa
-; VI-DAG: v_mul_lo_u16_sdwa
-; VI-DAG: v_mul_lo_u16_sdwa
-; VI-DAG: v_mul_lo_u16_sdwa
-
-; GFX9-DAG: v_mul_lo_u16_sdwa
-; GFX9-DAG: v_mul_lo_u16_sdwa
-; GFX9-DAG: v_mul_lo_u16_sdwa
-; GFX9-DAG: v_mul_lo_u16_sdwa
-; GFX9-DAG: v_mul_lo_u16_sdwa
-; GFX9-DAG: v_mul_lo_u16_sdwa
-
-; GFX10-DAG: v_mul_lo_u16
-; GFX10-DAG: v_mul_lo_u16
-; GFX10-DAG: v_mul_lo_u16
-; GFX10-DAG: v_mul_lo_u16
-; GFX10-DAG: v_mul_lo_u16
-; GFX10-DAG: v_mul_lo_u16
-; GFX10-DAG: v_mul_lo_u16
-; GFX10-DAG: v_mul_lo_u16
-
 define amdgpu_kernel void @mul_v8i8(ptr addrspace(1) %out, ptr addrspace(1) %ina, ptr addrspace(1) %inb) #0 {
+; NOSDWA-LABEL: mul_v8i8:
+; NOSDWA:       ; %bb.0: ; %entry
+; NOSDWA-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; NOSDWA-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s7
+; NOSDWA-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; NOSDWA-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; NOSDWA-NEXT:    v_mov_b32_e32 v3, s1
+; NOSDWA-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
+; NOSDWA-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
+; NOSDWA-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
+; NOSDWA-NEXT:    flat_load_dwordx2 v[2:3], v[2:3]
+; NOSDWA-NEXT:    v_mov_b32_e32 v4, s4
+; NOSDWA-NEXT:    v_mov_b32_e32 v5, s5
+; NOSDWA-NEXT:    s_waitcnt vmcnt(1)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v6, 16, v0
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v7, 24, v0
+; NOSDWA-NEXT:    v_lshrrev_b16_e32 v8, 8, v0
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v9, 16, v1
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v10, 24, v1
+; NOSDWA-NEXT:    v_lshrrev_b16_e32 v11, 8, v1
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v12, 16, v2
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v13, 24, v2
+; NOSDWA-NEXT:    v_lshrrev_b16_e32 v14, 8, v2
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v15, 16, v3
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v16, 24, v3
+; NOSDWA-NEXT:    v_lshrrev_b16_e32 v17, 8, v3
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v1, v1, v3
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v0, v0, v2
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v2, v11, v17
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v3, v10, v16
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v9, v9, v15
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v8, v8, v14
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v7, v7, v13
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v6, v6, v12
+; NOSDWA-NEXT:    v_and_b32_e32 v1, 0xff, v1
+; NOSDWA-NEXT:    v_and_b32_e32 v0, 0xff, v0
+; NOSDWA-NEXT:    v_lshlrev_b16_e32 v2, 8, v2
+; NOSDWA-NEXT:    v_lshlrev_b16_e32 v3, 8, v3
+; NOSDWA-NEXT:    v_and_b32_e32 v9, 0xff, v9
+; NOSDWA-NEXT:    v_lshlrev_b16_e32 v8, 8, v8
+; NOSDWA-NEXT:    v_lshlrev_b16_e32 v7, 8, v7
+; NOSDWA-NEXT:    v_and_b32_e32 v6, 0xff, v6
+; NOSDWA-NEXT:    v_or_b32_e32 v1, v1, v2
+; NOSDWA-NEXT:    v_or_b32_e32 v2, v9, v3
+; NOSDWA-NEXT:    v_or_b32_e32 v0, v0, v8
+; NOSDWA-NEXT:    v_or_b32_e32 v3, v6, v7
+; NOSDWA-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
+; NOSDWA-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
+; NOSDWA-NEXT:    v_or_b32_e32 v1, v1, v2
+; NOSDWA-NEXT:    v_or_b32_e32 v0, v0, v3
+; NOSDWA-NEXT:    flat_store_dwordx2 v[4:5], v[0:1]
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: mul_v8i8:
+; GFX89:       ; %bb.0: ; %entry
+; GFX89-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX89-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX89-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v1, s7
+; GFX89-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; GFX89-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GFX89-NEXT:    v_mov_b32_e32 v3, s1
+; GFX89-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
+; GFX89-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
+; GFX89-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
+; GFX89-NEXT:    flat_load_dwordx2 v[2:3], v[2:3]
+; GFX89-NEXT:    v_mov_b32_e32 v4, s4
+; GFX89-NEXT:    v_mov_b32_e32 v5, s5
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_mul_lo_u16_e32 v6, v1, v3
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v7, v1, v3 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v8, v1, v3 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:BYTE_3
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_mul_lo_u16_e32 v3, v0, v2
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v9, v0, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v10, v0, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:BYTE_3
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v0, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_or_b32_sdwa v2, v6, v7 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX89-NEXT:    v_or_b32_sdwa v1, v1, v8 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX89-NEXT:    v_or_b32_sdwa v3, v3, v9 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX89-NEXT:    v_or_b32_sdwa v0, v0, v10 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX89-NEXT:    v_or_b32_sdwa v1, v2, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX89-NEXT:    v_or_b32_sdwa v0, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX89-NEXT:    flat_store_dwordx2 v[4:5], v[0:1]
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: mul_v8i8:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX9-NEXT:    v_lshlrev_b32_e32 v4, 3, v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dwordx2 v[0:1], v4, s[6:7]
+; GFX9-NEXT:    global_load_dwordx2 v[2:3], v4, s[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v4, 0
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_mul_lo_u16_e32 v5, v1, v3
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v6, v1, v3 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v7, v1, v3 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:BYTE_3
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX9-NEXT:    v_mul_lo_u16_e32 v3, v0, v2
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v8, v0, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v9, v0, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:BYTE_3
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v0, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX9-NEXT:    v_or_b32_sdwa v2, v5, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v1, v1, v7 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v3, v3, v8 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v0, v0, v9 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v1, v2, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v0, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX9-NEXT:    global_store_dwordx2 v4, v[0:1], s[4:5]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: mul_v8i8:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX10-NEXT:    v_lshlrev_b32_e32 v4, 3, v0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    global_load_dwordx2 v[0:1], v4, s[6:7]
+; GFX10-NEXT:    global_load_dwordx2 v[2:3], v4, s[2:3]
+; GFX10-NEXT:    s_waitcnt vmcnt(1)
+; GFX10-NEXT:    v_lshrrev_b32_e32 v5, 24, v0
+; GFX10-NEXT:    v_lshrrev_b16 v6, 8, v0
+; GFX10-NEXT:    v_lshrrev_b32_e32 v8, 24, v1
+; GFX10-NEXT:    v_lshrrev_b16 v9, 8, v1
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_lshrrev_b32_e32 v11, 24, v2
+; GFX10-NEXT:    v_lshrrev_b16 v12, 8, v2
+; GFX10-NEXT:    v_lshrrev_b16 v13, 8, v3
+; GFX10-NEXT:    v_lshrrev_b32_e32 v14, 24, v3
+; GFX10-NEXT:    v_lshrrev_b32_e32 v4, 16, v0
+; GFX10-NEXT:    v_lshrrev_b32_e32 v7, 16, v1
+; GFX10-NEXT:    v_lshrrev_b32_e32 v10, 16, v2
+; GFX10-NEXT:    v_lshrrev_b32_e32 v15, 16, v3
+; GFX10-NEXT:    v_mul_lo_u16 v1, v1, v3
+; GFX10-NEXT:    v_mul_lo_u16 v0, v0, v2
+; GFX10-NEXT:    v_mul_lo_u16 v2, v9, v13
+; GFX10-NEXT:    v_mul_lo_u16 v3, v8, v14
+; GFX10-NEXT:    v_mul_lo_u16 v6, v6, v12
+; GFX10-NEXT:    v_mul_lo_u16 v5, v5, v11
+; GFX10-NEXT:    v_mul_lo_u16 v7, v7, v15
+; GFX10-NEXT:    v_mul_lo_u16 v4, v4, v10
+; GFX10-NEXT:    v_lshlrev_b16 v2, 8, v2
+; GFX10-NEXT:    v_lshlrev_b16 v3, 8, v3
+; GFX10-NEXT:    v_lshlrev_b16 v6, 8, v6
+; GFX10-NEXT:    v_lshlrev_b16 v5, 8, v5
+; GFX10-NEXT:    v_or_b32_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-NEXT:    v_or_b32_sdwa v2, v7, v3 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-NEXT:    v_or_b32_sdwa v0, v0, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-NEXT:    v_or_b32_sdwa v3, v4, v5 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-NEXT:    v_mov_b32_e32 v4, 0
+; GFX10-NEXT:    v_or_b32_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX10-NEXT:    v_or_b32_sdwa v0, v0, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX10-NEXT:    global_store_dwordx2 v4, v[0:1], s[4:5]
+; GFX10-NEXT:    s_endpgm
 entry:
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gepa = getelementptr <8 x i8>, ptr addrspace(1) %ina, i32 %idx
@@ -391,17 +1445,68 @@ entry:
   ret void
 }
 
-; GCN-LABEL: {{^}}sitofp_v2i16_to_v2f16:
-; NOSDWA-DAG: v_cvt_f16_i16_e32 v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA-DAG: v_lshrrev_b32_e32 v{{[0-9]+}}, 16, v{{[0-9]+}}
-; NOSDWA-DAG: v_cvt_f16_i16_e32 v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA-NOT: v_cvt_f16_i16_sdwa
-
-; SDWA-DAG: v_cvt_f16_i16_e32 v{{[0-9]+}}, v{{[0-9]+}}
-; SDWA-DAG: v_cvt_f16_i16_sdwa v{{[0-9]+}}, v{{[0-9]+}} dst_sel:{{(WORD_1|DWORD)?}} dst_unused:UNUSED_PAD src0_sel:WORD_1
-
 ; FIXME: Should be able to avoid or
 define amdgpu_kernel void @sitofp_v2i16_to_v2f16(
+; NOSDWA-LABEL: sitofp_v2i16_to_v2f16:
+; NOSDWA:       ; %bb.0: ; %entry
+; NOSDWA-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s2
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s3
+; NOSDWA-NEXT:    flat_load_dword v2, v[0:1]
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s0
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s1
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
+; NOSDWA-NEXT:    v_cvt_f16_i16_e32 v3, v3
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
+; NOSDWA-NEXT:    v_cvt_f16_i16_e32 v2, v2
+; NOSDWA-NEXT:    v_or_b32_e32 v2, v2, v3
+; NOSDWA-NEXT:    flat_store_dword v[0:1], v2
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: sitofp_v2i16_to_v2f16:
+; GFX89:       ; %bb.0: ; %entry
+; GFX89-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v0, s2
+; GFX89-NEXT:    v_mov_b32_e32 v1, s3
+; GFX89-NEXT:    flat_load_dword v2, v[0:1]
+; GFX89-NEXT:    v_mov_b32_e32 v0, s0
+; GFX89-NEXT:    v_mov_b32_e32 v1, s1
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_cvt_f16_i16_sdwa v3, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1
+; GFX89-NEXT:    v_cvt_f16_i16_e32 v2, v2
+; GFX89-NEXT:    v_or_b32_e32 v2, v2, v3
+; GFX89-NEXT:    flat_store_dword v[0:1], v2
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: sitofp_v2i16_to_v2f16:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dword v1, v0, s[2:3]
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_cvt_f16_i16_e32 v2, v1
+; GFX9-NEXT:    v_cvt_f16_i16_sdwa v1, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1
+; GFX9-NEXT:    v_lshl_or_b32 v1, v1, 16, v2
+; GFX9-NEXT:    global_store_dword v0, v1, s[0:1]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: sitofp_v2i16_to_v2f16:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    global_load_dword v1, v0, s[2:3]
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_cvt_f16_i16_e32 v2, v1
+; GFX10-NEXT:    v_cvt_f16_i16_sdwa v1, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1
+; GFX10-NEXT:    v_and_b32_e32 v2, 0xffff, v2
+; GFX10-NEXT:    v_lshl_or_b32 v1, v1, 16, v2
+; GFX10-NEXT:    global_store_dword v0, v1, s[0:1]
+; GFX10-NEXT:    s_endpgm
     ptr addrspace(1) %r,
     ptr addrspace(1) %a) #0 {
 entry:
@@ -411,22 +1516,83 @@ entry:
   ret void
 }
 
-
-; GCN-LABEL: {{^}}mac_v2half:
-; NOSDWA: v_lshrrev_b32_e32 v[[DST1:[0-9]+]], 16, v{{[0-9]+}}
-; NOSDWA: v_lshrrev_b32_e32 v[[DST0:[0-9]+]], 16, v{{[0-9]+}}
-; NOSDWA: v_mac_f16_e32 v[[DST_MAC:[0-9]+]], v[[DST0]], v[[DST1]]
-; NOSDWA: v_lshlrev_b32_e32 v[[DST_SHL:[0-9]+]], 16, v[[DST_MAC]]
-; NOSDWA: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v[[DST_SHL]]
-; NOSDWA-NOT: v_mac_f16_sdwa
-
-; VI: v_mac_f16_sdwa v[[DST_MAC:[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
-; VI: v_lshlrev_b32_e32 v[[DST_SHL:[0-9]+]], 16, v[[DST_MAC]]
-
-; GFX9_10: v_pk_mul_f16 v[[DST_MUL:[0-9]+]], v{{[0-9]+}}, v[[SRC:[0-9]+]]
-; GFX9_10: v_pk_add_f16 v{{[0-9]+}}, v[[DST_MUL]], v[[SRC]]
-
 define amdgpu_kernel void @mac_v2half(ptr addrspace(1) %out, ptr addrspace(1) %ina, ptr addrspace(1) %inb) #0 {
+; NOSDWA-LABEL: mac_v2half:
+; NOSDWA:       ; %bb.0: ; %entry
+; NOSDWA-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; NOSDWA-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s6
+; NOSDWA-NEXT:    v_mov_b32_e32 v2, s0
+; NOSDWA-NEXT:    v_mov_b32_e32 v3, s1
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s7
+; NOSDWA-NEXT:    flat_load_dword v2, v[2:3]
+; NOSDWA-NEXT:    flat_load_dword v3, v[0:1]
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s4
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s5
+; NOSDWA-NEXT:    s_waitcnt vmcnt(1)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v4, 16, v2
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v5, 16, v3
+; NOSDWA-NEXT:    v_mac_f16_e32 v4, v5, v4
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
+; NOSDWA-NEXT:    v_mac_f16_e32 v2, v3, v2
+; NOSDWA-NEXT:    v_or_b32_e32 v2, v2, v4
+; NOSDWA-NEXT:    flat_store_dword v[0:1], v2
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: mac_v2half:
+; GFX89:       ; %bb.0: ; %entry
+; GFX89-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX89-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v0, s6
+; GFX89-NEXT:    v_mov_b32_e32 v2, s0
+; GFX89-NEXT:    v_mov_b32_e32 v3, s1
+; GFX89-NEXT:    v_mov_b32_e32 v1, s7
+; GFX89-NEXT:    flat_load_dword v2, v[2:3]
+; GFX89-NEXT:    flat_load_dword v3, v[0:1]
+; GFX89-NEXT:    v_mov_b32_e32 v0, s4
+; GFX89-NEXT:    v_mov_b32_e32 v1, s5
+; GFX89-NEXT:    s_waitcnt vmcnt(1)
+; GFX89-NEXT:    v_lshrrev_b32_e32 v4, 16, v2
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_mac_f16_sdwa v4, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
+; GFX89-NEXT:    v_mac_f16_e32 v2, v3, v2
+; GFX89-NEXT:    v_or_b32_e32 v2, v2, v4
+; GFX89-NEXT:    flat_store_dword v[0:1], v2
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: mac_v2half:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dword v1, v0, s[6:7]
+; GFX9-NEXT:    global_load_dword v2, v0, s[2:3]
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_pk_mul_f16 v1, v1, v2
+; GFX9-NEXT:    v_pk_add_f16 v1, v1, v2
+; GFX9-NEXT:    global_store_dword v0, v1, s[4:5]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: mac_v2half:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    global_load_dword v1, v0, s[6:7]
+; GFX10-NEXT:    global_load_dword v2, v0, s[2:3]
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_pk_mul_f16 v1, v1, v2
+; GFX10-NEXT:    v_pk_add_f16 v1, v1, v2
+; GFX10-NEXT:    global_store_dword v0, v1, s[4:5]
+; GFX10-NEXT:    s_endpgm
 entry:
   %a = load <2 x half>, ptr addrspace(1) %ina, align 4
   %b = load <2 x half>, ptr addrspace(1) %inb, align 4
@@ -436,18 +1602,70 @@ entry:
   ret void
 }
 
-; GCN-LABEL: {{^}}immediate_mul_v2i16:
-; NOSDWA-NOT: v_mul_u32_u24_sdwa
-; VI-DAG: v_mov_b32_e32 v[[M321:[0-9]+]], 0x141
-; VI-DAG: v_mul_lo_u16_e32 v{{[0-9]+}}, 0x7b, v{{[0-9]+}}
-; VI-DAG: v_mul_lo_u16_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v[[M321]] dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-
-; GFX9: s_mov_b32 s[[IMM:[0-9]+]], 0x141007b
-; GFX9: v_pk_mul_lo_u16 v{{[0-9]+}}, v{{[0-9]+}}, s[[IMM]]
-
-; GFX10: v_pk_mul_lo_u16 v{{[0-9]+}}, 0x141007b, v{{[0-9]+}}
-
 define amdgpu_kernel void @immediate_mul_v2i16(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 {
+; NOSDWA-LABEL: immediate_mul_v2i16:
+; NOSDWA:       ; %bb.0: ; %entry
+; NOSDWA-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s3
+; NOSDWA-NEXT:    v_add_u32_e32 v0, vcc, s2, v0
+; NOSDWA-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; NOSDWA-NEXT:    flat_load_dword v2, v[0:1]
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s0
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s1
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v3, 0x7b, v2
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v2, 16, v2
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v2, 0x141, v2
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
+; NOSDWA-NEXT:    v_or_b32_e32 v2, v3, v2
+; NOSDWA-NEXT:    flat_store_dword v[0:1], v2
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: immediate_mul_v2i16:
+; GFX89:       ; %bb.0: ; %entry
+; GFX89-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX89-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX89-NEXT:    v_mov_b32_e32 v3, 0x141
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v1, s3
+; GFX89-NEXT:    v_add_u32_e32 v0, vcc, s2, v0
+; GFX89-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GFX89-NEXT:    flat_load_dword v2, v[0:1]
+; GFX89-NEXT:    v_mov_b32_e32 v0, s0
+; GFX89-NEXT:    v_mov_b32_e32 v1, s1
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_mul_lo_u16_e32 v4, 0x7b, v2
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v2, v2, v3 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX89-NEXT:    v_or_b32_e32 v2, v4, v2
+; GFX89-NEXT:    flat_store_dword v[0:1], v2
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: immediate_mul_v2i16:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX9-NEXT:    v_mov_b32_e32 v1, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dword v0, v0, s[2:3]
+; GFX9-NEXT:    s_mov_b32 s2, 0x141007b
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_pk_mul_lo_u16 v0, v0, s2
+; GFX9-NEXT:    global_store_dword v1, v0, s[0:1]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: immediate_mul_v2i16:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    global_load_dword v0, v0, s[2:3]
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_pk_mul_lo_u16 v0, 0x141007b, v0
+; GFX10-NEXT:    global_store_dword v1, v0, s[0:1]
+; GFX10-NEXT:    s_endpgm
 entry:
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gep = getelementptr <2 x i16>, ptr addrspace(1) %in, i32 %idx
@@ -458,20 +1676,94 @@ entry:
 }
 
 ; Double use of same src - should not convert it
-; GCN-LABEL: {{^}}mulmul_v2i16:
-; NOSDWA: v_lshrrev_b32_e32 v{{[0-9]+}}, 16, v{{[0-9]+}}
-; NOSDWA: v_lshrrev_b32_e32 v{{[0-9]+}}, 16, v{{[0-9]+}}
-; NOSDWA: v_mul_lo_u16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA: v_lshlrev_b32_e32 v{{[0-9]+}}, 16, v{{[0-9]+}}
-; NOSDWA: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA-NOT: v_mul_u32_u24_sdwa
-
-; VI: v_mul_lo_u16_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-
-; GFX9_10: v_pk_mul_lo_u16 v[[DST1:[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}}
-; GFX9_10: v_pk_mul_lo_u16 v{{[0-9]+}}, v[[DST1]], v{{[0-9]+}}
-
 define amdgpu_kernel void @mulmul_v2i16(ptr addrspace(1) %out, ptr addrspace(1) %ina, ptr addrspace(1) %inb) #0 {
+; NOSDWA-LABEL: mulmul_v2i16:
+; NOSDWA:       ; %bb.0: ; %entry
+; NOSDWA-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; NOSDWA-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v2, 2, v0
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s7
+; NOSDWA-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; NOSDWA-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; NOSDWA-NEXT:    v_mov_b32_e32 v3, s1
+; NOSDWA-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
+; NOSDWA-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
+; NOSDWA-NEXT:    flat_load_dword v2, v[2:3]
+; NOSDWA-NEXT:    flat_load_dword v3, v[0:1]
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s4
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s5
+; NOSDWA-NEXT:    s_waitcnt vmcnt(1)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v4, 16, v2
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v5, 16, v3
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v3, v3, v2
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v5, v5, v4
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v2, v3, v2
+; NOSDWA-NEXT:    v_mul_lo_u16_e32 v3, v5, v4
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
+; NOSDWA-NEXT:    v_or_b32_e32 v2, v2, v3
+; NOSDWA-NEXT:    flat_store_dword v[0:1], v2
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: mulmul_v2i16:
+; GFX89:       ; %bb.0: ; %entry
+; GFX89-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX89-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX89-NEXT:    v_lshlrev_b32_e32 v2, 2, v0
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v1, s7
+; GFX89-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; GFX89-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GFX89-NEXT:    v_mov_b32_e32 v3, s1
+; GFX89-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
+; GFX89-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
+; GFX89-NEXT:    flat_load_dword v2, v[2:3]
+; GFX89-NEXT:    flat_load_dword v3, v[0:1]
+; GFX89-NEXT:    v_mov_b32_e32 v0, s4
+; GFX89-NEXT:    v_mov_b32_e32 v1, s5
+; GFX89-NEXT:    s_waitcnt vmcnt(1)
+; GFX89-NEXT:    v_lshrrev_b32_e32 v4, 16, v2
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_mul_lo_u16_e32 v5, v3, v2
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v3, v3, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX89-NEXT:    v_mul_lo_u16_e32 v2, v5, v2
+; GFX89-NEXT:    v_mul_lo_u16_sdwa v3, v3, v4 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX89-NEXT:    v_or_b32_e32 v2, v2, v3
+; GFX89-NEXT:    flat_store_dword v[0:1], v2
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: mulmul_v2i16:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dword v1, v0, s[6:7]
+; GFX9-NEXT:    global_load_dword v2, v0, s[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_pk_mul_lo_u16 v1, v1, v2
+; GFX9-NEXT:    v_pk_mul_lo_u16 v1, v1, v2
+; GFX9-NEXT:    global_store_dword v0, v1, s[4:5]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: mulmul_v2i16:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    global_load_dword v1, v0, s[6:7]
+; GFX10-NEXT:    global_load_dword v2, v0, s[2:3]
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_pk_mul_lo_u16 v0, v1, v2
+; GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GFX10-NEXT:    v_pk_mul_lo_u16 v0, v0, v2
+; GFX10-NEXT:    global_store_dword v1, v0, s[4:5]
+; GFX10-NEXT:    s_endpgm
 entry:
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gepa = getelementptr <2 x i16>, ptr addrspace(1) %ina, i32 %idx
@@ -484,16 +1776,79 @@ entry:
   ret void
 }
 
-; GCN-LABEL: {{^}}add_bb_v2i16:
-; NOSDWA-NOT: v_add_{{(_co)?}}_u32_sdwa
-
-; VI:      v_add_u32_sdwa v{{[0-9]+}}, vcc, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
-; VI-NEXT: v_add_u32_e32 v{{[0-9]+}}, vcc, v{{[0-9]+}}, v{{[0-9]+}}
-; VI-NEXT: v_or_b32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-
-; GFX9_10: v_pk_add_u16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-
 define amdgpu_kernel void @add_bb_v2i16(ptr addrspace(1) %out, ptr addrspace(1) %ina, ptr addrspace(1) %inb) #0 {
+; NOSDWA-LABEL: add_bb_v2i16:
+; NOSDWA:       ; %bb.0: ; %entry
+; NOSDWA-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; NOSDWA-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s6
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s7
+; NOSDWA-NEXT:    v_mov_b32_e32 v2, s0
+; NOSDWA-NEXT:    v_mov_b32_e32 v3, s1
+; NOSDWA-NEXT:    flat_load_dword v1, v[0:1]
+; NOSDWA-NEXT:    flat_load_dword v2, v[2:3]
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s4
+; NOSDWA-NEXT:    s_waitcnt vmcnt(1)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v3, 16, v1
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v4, 16, v2
+; NOSDWA-NEXT:    v_add_u32_e32 v1, vcc, v1, v2
+; NOSDWA-NEXT:    v_add_u32_e32 v2, vcc, v3, v4
+; NOSDWA-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
+; NOSDWA-NEXT:    v_or_b32_e32 v2, v1, v2
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s5
+; NOSDWA-NEXT:    flat_store_dword v[0:1], v2
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: add_bb_v2i16:
+; GFX89:       ; %bb.0: ; %entry
+; GFX89-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX89-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v0, s6
+; GFX89-NEXT:    v_mov_b32_e32 v1, s7
+; GFX89-NEXT:    v_mov_b32_e32 v2, s0
+; GFX89-NEXT:    v_mov_b32_e32 v3, s1
+; GFX89-NEXT:    flat_load_dword v1, v[0:1]
+; GFX89-NEXT:    flat_load_dword v2, v[2:3]
+; GFX89-NEXT:    v_mov_b32_e32 v0, s4
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_add_u32_sdwa v3, vcc, v1, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GFX89-NEXT:    v_add_u32_e32 v1, vcc, v1, v2
+; GFX89-NEXT:    v_or_b32_sdwa v2, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX89-NEXT:    v_mov_b32_e32 v1, s5
+; GFX89-NEXT:    flat_store_dword v[0:1], v2
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: add_bb_v2i16:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dword v1, v0, s[6:7]
+; GFX9-NEXT:    global_load_dword v2, v0, s[2:3]
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_pk_add_u16 v1, v1, v2
+; GFX9-NEXT:    global_store_dword v0, v1, s[4:5]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: add_bb_v2i16:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_clause 0x1
+; GFX10-NEXT:    global_load_dword v1, v0, s[6:7]
+; GFX10-NEXT:    global_load_dword v2, v0, s[2:3]
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_pk_add_u16 v1, v1, v2
+; GFX10-NEXT:    global_store_dword v0, v1, s[4:5]
+; GFX10-NEXT:    s_endpgm
 entry:
   %a = load <2 x i16>, ptr addrspace(1) %ina, align 4
   %b = load <2 x i16>, ptr addrspace(1) %inb, align 4
@@ -506,39 +1861,125 @@ store_label:
   ret void
 }
 
-
 ; Check that "pulling out" SDWA operands works correctly.
-; GCN-LABEL: {{^}}pulled_out_test:
-; NOSDWA-DAG: v_and_b32_e32 v{{[0-9]+}}, 0xff, v{{[0-9]+}}
-; NOSDWA-DAG: v_lshlrev_b16_e32 v{{[0-9]+}}, 8, v{{[0-9]+}}
-; NOSDWA-DAG: v_and_b32_e32 v{{[0-9]+}}, 0xff, v{{[0-9]+}}
-; NOSDWA-DAG: v_lshlrev_b16_e32 v{{[0-9]+}}, 8, v{{[0-9]+}}
-; NOSDWA: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; NOSDWA-NOT: v_and_b32_sdwa
-; NOSDWA-NOT: v_or_b32_sdwa
-
-; VI-DAG: v_and_b32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX9-DAG: v_and_b32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, s{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX10-DAG: v_and_b32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX89-DAG: v_lshlrev_b16_e32 v{{[0-9]+}}, 8, v{{[0-9]+}}
-;
-; GFX10-DAG: v_lshrrev_b32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-;
-; VI-DAG: v_and_b32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX9-DAG: v_and_b32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, s{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX10-DAG: v_and_b32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX89-DAG: v_lshlrev_b16_e32 v{{[0-9]+}}, 8, v{{[0-9]+}}
+define amdgpu_kernel void @pulled_out_test(ptr addrspace(1) %sourceA, ptr addrspace(1) %destValues) #0 {
+; NOSDWA-LABEL: pulled_out_test:
+; NOSDWA:       ; %bb.0: ; %entry
+; NOSDWA-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
+; NOSDWA-NEXT:    v_mov_b32_e32 v0, s0
+; NOSDWA-NEXT:    v_mov_b32_e32 v1, s1
+; NOSDWA-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
+; NOSDWA-NEXT:    v_mov_b32_e32 v2, s2
+; NOSDWA-NEXT:    v_mov_b32_e32 v3, s3
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    v_lshrrev_b64 v[4:5], 24, v[0:1]
+; NOSDWA-NEXT:    v_and_b32_e32 v6, 0xff, v0
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v7, 8, v0
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v0, 16, v0
+; NOSDWA-NEXT:    v_and_b32_e32 v5, 0xff, v1
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v8, 8, v1
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v9, 24, v1
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v1, 16, v1
+; NOSDWA-NEXT:    v_lshlrev_b16_e32 v7, 8, v7
+; NOSDWA-NEXT:    v_and_b32_e32 v0, 0xff, v0
+; NOSDWA-NEXT:    v_lshlrev_b16_e32 v8, 8, v8
+; NOSDWA-NEXT:    v_lshlrev_b16_e32 v9, 8, v9
+; NOSDWA-NEXT:    v_and_b32_e32 v1, 0xff, v1
+; NOSDWA-NEXT:    v_lshlrev_b16_e32 v4, 8, v4
+; NOSDWA-NEXT:    v_or_b32_e32 v6, v6, v7
+; NOSDWA-NEXT:    v_or_b32_e32 v5, v5, v8
+; NOSDWA-NEXT:    v_or_b32_e32 v1, v1, v9
+; NOSDWA-NEXT:    v_or_b32_e32 v0, v0, v4
+; NOSDWA-NEXT:    v_and_b32_e32 v6, 0xffff, v6
+; NOSDWA-NEXT:    v_and_b32_e32 v4, 0xffff, v5
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v0, 16, v0
+; NOSDWA-NEXT:    v_or_b32_e32 v0, v6, v0
+; NOSDWA-NEXT:    v_or_b32_e32 v1, v4, v1
+; NOSDWA-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; NOSDWA-NEXT:    s_endpgm
 ;
-; GFX10-DAG: v_lshrrev_b32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX89-LABEL: pulled_out_test:
+; GFX89:       ; %bb.0: ; %entry
+; GFX89-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX89-NEXT:    v_mov_b32_e32 v6, 8
+; GFX89-NEXT:    v_mov_b32_e32 v7, 0xff
+; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX89-NEXT:    v_mov_b32_e32 v0, s0
+; GFX89-NEXT:    v_mov_b32_e32 v1, s1
+; GFX89-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
+; GFX89-NEXT:    v_mov_b32_e32 v2, s2
+; GFX89-NEXT:    v_mov_b32_e32 v3, s3
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    v_lshrrev_b64 v[4:5], 24, v[0:1]
+; GFX89-NEXT:    v_lshrrev_b32_sdwa v8, v6, v0 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX89-NEXT:    v_lshrrev_b32_sdwa v6, v6, v1 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX89-NEXT:    v_lshrrev_b32_e32 v9, 24, v1
+; GFX89-NEXT:    v_and_b32_sdwa v5, v0, v7 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX89-NEXT:    v_and_b32_sdwa v7, v1, v7 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX89-NEXT:    v_or_b32_sdwa v1, v1, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX89-NEXT:    v_lshlrev_b16_e32 v6, 8, v9
+; GFX89-NEXT:    v_lshlrev_b16_e32 v4, 8, v4
+; GFX89-NEXT:    v_or_b32_sdwa v0, v0, v8 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX89-NEXT:    v_or_b32_sdwa v6, v7, v6 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX89-NEXT:    v_or_b32_sdwa v4, v5, v4 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX89-NEXT:    v_or_b32_sdwa v0, v0, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX89-NEXT:    v_or_b32_sdwa v1, v1, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX89-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GFX89-NEXT:    s_endpgm
 ;
-; GFX89: v_or_b32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-LABEL: pulled_out_test:
+; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v4, 0
+; GFX9-NEXT:    v_mov_b32_e32 v5, 8
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dwordx2 v[0:1], v4, s[0:1]
+; GFX9-NEXT:    s_movk_i32 s0, 0xff
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_lshrrev_b64 v[2:3], 24, v[0:1]
+; GFX9-NEXT:    v_lshrrev_b32_sdwa v6, v5, v0 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_lshrrev_b32_sdwa v5, v5, v1 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_lshrrev_b32_e32 v7, 24, v1
+; GFX9-NEXT:    v_and_b32_sdwa v3, v0, s0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_sdwa v8, v1, s0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_lshlrev_b16_e32 v5, 8, v7
+; GFX9-NEXT:    v_lshlrev_b16_e32 v2, 8, v2
+; GFX9-NEXT:    v_or_b32_sdwa v0, v0, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v5, v8, v5 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v2, v3, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v0, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX9-NEXT:    global_store_dwordx2 v4, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
 ;
-; GFX10: v_or_b32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX10: v_or_b32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX10: v_or_b32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX10: v_or_b32_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-
-define amdgpu_kernel void @pulled_out_test(ptr addrspace(1) %sourceA, ptr addrspace(1) %destValues) #0 {
+; GFX10-LABEL: pulled_out_test:
+; GFX10:       ; %bb.0: ; %entry
+; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX10-NEXT:    v_mov_b32_e32 v4, 0
+; GFX10-NEXT:    v_mov_b32_e32 v5, 8
+; GFX10-NEXT:    v_mov_b32_e32 v6, 0xff
+; GFX10-NEXT:    v_mov_b32_e32 v7, 24
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    global_load_dwordx2 v[0:1], v4, s[0:1]
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_lshrrev_b64 v[2:3], 24, v[0:1]
+; GFX10-NEXT:    v_lshrrev_b32_sdwa v3, v5, v0 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX10-NEXT:    v_and_b32_sdwa v8, v0, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-NEXT:    v_lshrrev_b32_sdwa v5, v5, v1 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX10-NEXT:    v_lshrrev_b32_sdwa v7, v7, v1 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX10-NEXT:    v_and_b32_sdwa v6, v1, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-NEXT:    v_lshlrev_b16 v2, 8, v2
+; GFX10-NEXT:    v_or_b32_sdwa v0, v0, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-NEXT:    v_or_b32_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-NEXT:    v_or_b32_sdwa v3, v6, v7 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX10-NEXT:    v_or_b32_sdwa v2, v8, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX10-NEXT:    v_or_b32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX10-NEXT:    v_or_b32_sdwa v0, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX10-NEXT:    global_store_dwordx2 v4, v[0:1], s[2:3]
+; GFX10-NEXT:    s_endpgm
 entry:
   %idxprom = ashr exact i64 15, 32
   %arrayidx = getelementptr inbounds <8 x i8>, ptr addrspace(1) %sourceA, i64 %idxprom
@@ -571,15 +2012,74 @@ entry:
   ret void
 }
 
-; GCN-LABEL: {{^}}sdwa_crash_inlineasm_def:
-; GCN: s_mov_b32 s{{[0-9]+}}, 0xffff
-; GCN: v_and_b32_e32 v{{[0-9]+}}, s{{[0-9]+}}, v{{[0-9]+}}
-;
 ; TODO: Why is the constant not peepholed into the v_or_b32_e32?
-;
-; NOSDWA: v_or_b32_e32 v{{[0-9]+}}, 0x10000,
-; SDWA: v_or_b32_e32 v{{[0-9]+}}, 0x10000,
 define amdgpu_kernel void @sdwa_crash_inlineasm_def() #0 {
+; NOSDWA-LABEL: sdwa_crash_inlineasm_def:
+; NOSDWA:       ; %bb.0: ; %bb
+; NOSDWA-NEXT:    s_mov_b32 s0, 0xffff
+; NOSDWA-NEXT:    ;;#ASMSTART
+; NOSDWA-NEXT:    v_and_b32_e32 v0, s0, v0
+; NOSDWA-NEXT:    ;;#ASMEND
+; NOSDWA-NEXT:    v_or_b32_e32 v0, 0x10000, v0
+; NOSDWA-NEXT:    s_and_b64 vcc, exec, -1
+; NOSDWA-NEXT:  .LBB21_1: ; %bb1
+; NOSDWA-NEXT:    ; =>This Inner Loop Header: Depth=1
+; NOSDWA-NEXT:    flat_store_dwordx2 v[0:1], v[0:1]
+; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
+; NOSDWA-NEXT:    s_mov_b64 vcc, vcc
+; NOSDWA-NEXT:    s_cbranch_vccnz .LBB21_1
+; NOSDWA-NEXT:  ; %bb.2: ; %DummyReturnBlock
+; NOSDWA-NEXT:    s_endpgm
+;
+; GFX89-LABEL: sdwa_crash_inlineasm_def:
+; GFX89:       ; %bb.0: ; %bb
+; GFX89-NEXT:    s_mov_b32 s0, 0xffff
+; GFX89-NEXT:    ;;#ASMSTART
+; GFX89-NEXT:    v_and_b32_e32 v0, s0, v0
+; GFX89-NEXT:    ;;#ASMEND
+; GFX89-NEXT:    v_or_b32_e32 v0, 0x10000, v0
+; GFX89-NEXT:    s_and_b64 vcc, exec, -1
+; GFX89-NEXT:  .LBB21_1: ; %bb1
+; GFX89-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX89-NEXT:    flat_store_dwordx2 v[0:1], v[0:1]
+; GFX89-NEXT:    s_waitcnt vmcnt(0)
+; GFX89-NEXT:    s_mov_b64 vcc, vcc
+; GFX89-NEXT:    s_cbranch_vccnz .LBB21_1
+; GFX89-NEXT:  ; %bb.2: ; %DummyReturnBlock
+; GFX89-NEXT:    s_endpgm
+;
+; GFX9-LABEL: sdwa_crash_inlineasm_def:
+; GFX9:       ; %bb.0: ; %bb
+; GFX9-NEXT:    s_mov_b32 s0, 0xffff
+; GFX9-NEXT:    ;;#ASMSTART
+; GFX9-NEXT:    v_and_b32_e32 v0, s0, v0
+; GFX9-NEXT:    ;;#ASMEND
+; GFX9-NEXT:    v_or_b32_e32 v0, 0x10000, v0
+; GFX9-NEXT:    s_and_b64 vcc, exec, -1
+; GFX9-NEXT:  .LBB21_1: ; %bb1
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    global_store_dwordx2 v[0:1], v[0:1], off
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    s_mov_b64 vcc, vcc
+; GFX9-NEXT:    s_cbranch_vccnz .LBB21_1
+; GFX9-NEXT:  ; %bb.2: ; %DummyReturnBlock
+; GFX9-NEXT:    s_endpgm
+;
+; GFX10-LABEL: sdwa_crash_inlineasm_def:
+; GFX10:       ; %bb.0: ; %bb
+; GFX10-NEXT:    s_mov_b32 s0, 0xffff
+; GFX10-NEXT:    ;;#ASMSTART
+; GFX10-NEXT:    v_and_b32_e32 v0, s0, v0
+; GFX10-NEXT:    ;;#ASMEND
+; GFX10-NEXT:    v_or_b32_e32 v0, 0x10000, v0
+; GFX10-NEXT:    s_mov_b32 vcc_lo, exec_lo
+; GFX10-NEXT:  .LBB21_1: ; %bb1
+; GFX10-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX10-NEXT:    global_store_dwordx2 v[0:1], v[0:1], off
+; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-NEXT:    s_cbranch_vccnz .LBB21_1
+; GFX10-NEXT:  ; %bb.2: ; %DummyReturnBlock
+; GFX10-NEXT:    s_endpgm
 bb:
   br label %bb1
 
@@ -602,3 +2102,7 @@ bb11:                                             ; preds = %bb10, %bb2
 declare i32 @llvm.amdgcn.workitem.id.x()
 
 attributes #0 = { "denormal-fp-math"="preserve-sign,preserve-sign" }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GCN: {{.*}}
+; GFX9_10: {{.*}}
+; SDWA: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/shl.ll b/llvm/test/CodeGen/AMDGPU/shl.ll
index 8f99ab780ca93..20c9544f73bd2 100644
--- a/llvm/test/CodeGen/AMDGPU/shl.ll
+++ b/llvm/test/CodeGen/AMDGPU/shl.ll
@@ -681,51 +681,43 @@ define amdgpu_kernel void @shl_v4i16(ptr addrspace(1) %out, ptr addrspace(1) %in
 ;
 ; EG-LABEL: shl_v4i16:
 ; EG:       ; %bb.0:
-; EG-NEXT:    ALU 2, @8, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    ALU 3, @8, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    TEX 0 @6
-; EG-NEXT:    ALU 51, @11, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    ALU 42, @12, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T10.XY, T0.X, 1
 ; EG-NEXT:    CF_END
 ; EG-NEXT:    PAD
 ; EG-NEXT:    Fetch clause starting at 6:
 ; EG-NEXT:     VTX_READ_128 T10.XYZW, T0.X, 0, #1
 ; EG-NEXT:    ALU clause starting at 8:
-; EG-NEXT:     LSHL * T0.W, T0.X, literal.x,
+; EG-NEXT:     MOV T0.Y, T6.X,
+; EG-NEXT:     LSHL * T0.W, T0.X, literal.x, BS:VEC_120/SCL_212
 ; EG-NEXT:    3(4.203895e-45), 0(0.000000e+00)
 ; EG-NEXT:     ADD_INT * T0.X, KC0[2].Z, PV.W,
-; EG-NEXT:    ALU clause starting at 11:
-; EG-NEXT:     MOV T4.X, T10.X,
-; EG-NEXT:     MOV * T5.X, T10.Y,
-; EG-NEXT:     MOV T0.X, PV.X,
-; EG-NEXT:     MOV T0.Y, PS,
-; EG-NEXT:     MOV * T2.X, T10.Z,
-; EG-NEXT:     MOV T3.X, T10.W,
-; EG-NEXT:     MOV * T0.Z, T6.X,
-; EG-NEXT:     MOV * T1.Y, T2.X,
-; EG-NEXT:     AND_INT * T1.W, PV.Y, literal.x,
+; EG-NEXT:    ALU clause starting at 12:
+; EG-NEXT:     AND_INT * T1.W, T10.Z, literal.x,
 ; EG-NEXT:    65535(9.183409e-41), 0(0.000000e+00)
-; EG-NEXT:     LSHL * T1.W, T0.X, PV.W,
+; EG-NEXT:     LSHL * T1.W, T10.X, PV.W,
 ; EG-NEXT:     AND_INT T1.W, PV.W, literal.x,
-; EG-NEXT:     AND_INT * T2.W, T0.Z, literal.y,
+; EG-NEXT:     AND_INT * T2.W, T0.Y, literal.y,
 ; EG-NEXT:    65535(9.183409e-41), -65536(nan)
 ; EG-NEXT:     OR_INT * T1.W, PS, PV.W,
-; EG-NEXT:     MOV * T0.Z, T3.X,
-; EG-NEXT:     MOV * T6.X, T1.W,
-; EG-NEXT:     MOV T1.Z, PV.X,
-; EG-NEXT:     LSHR T1.W, T1.Y, literal.x,
-; EG-NEXT:     LSHR * T2.W, T0.X, literal.x,
+; EG-NEXT:     MOV * T6.X, PV.W,
+; EG-NEXT:     MOV T0.X, PV.X,
+; EG-NEXT:     LSHR T1.W, T10.Z, literal.x,
+; EG-NEXT:     LSHR * T2.W, T10.X, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
 ; EG-NEXT:     LSHL T1.W, PS, PV.W,
-; EG-NEXT:     AND_INT * T2.W, PV.Z, literal.x,
+; EG-NEXT:     AND_INT * T2.W, PV.X, literal.x,
 ; EG-NEXT:    65535(9.183409e-41), 0(0.000000e+00)
 ; EG-NEXT:     LSHL * T1.W, PV.W, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
 ; EG-NEXT:     OR_INT * T1.W, T2.W, PV.W,
 ; EG-NEXT:     MOV T6.X, PV.W,
 ; EG-NEXT:     MOV * T0.X, T7.X,
-; EG-NEXT:     AND_INT * T1.W, T0.Z, literal.x,
+; EG-NEXT:     AND_INT * T1.W, T10.W, literal.x,
 ; EG-NEXT:    65535(9.183409e-41), 0(0.000000e+00)
-; EG-NEXT:     LSHL T1.W, T0.Y, PV.W,
+; EG-NEXT:     LSHL T1.W, T10.Y, PV.W,
 ; EG-NEXT:     AND_INT * T2.W, T0.X, literal.x,
 ; EG-NEXT:    -65536(nan), 0(0.000000e+00)
 ; EG-NEXT:     AND_INT * T1.W, PV.W, literal.x,
@@ -733,8 +725,8 @@ define amdgpu_kernel void @shl_v4i16(ptr addrspace(1) %out, ptr addrspace(1) %in
 ; EG-NEXT:     OR_INT * T1.W, T2.W, PV.W,
 ; EG-NEXT:     MOV * T7.X, PV.W,
 ; EG-NEXT:     MOV T0.X, PV.X,
-; EG-NEXT:     LSHR T1.W, T0.Z, literal.x,
-; EG-NEXT:     LSHR * T2.W, T0.Y, literal.x,
+; EG-NEXT:     LSHR T1.W, T10.W, literal.x,
+; EG-NEXT:     LSHR * T2.W, T10.Y, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
 ; EG-NEXT:     LSHL * T1.W, PS, PV.W,
 ; EG-NEXT:     AND_INT T0.Z, T0.X, literal.x,
diff --git a/llvm/test/CodeGen/AMDGPU/smed3.ll b/llvm/test/CodeGen/AMDGPU/smed3.ll
index d9e21e5a83c99..a3c6ca1c8673d 100644
--- a/llvm/test/CodeGen/AMDGPU/smed3.ll
+++ b/llvm/test/CodeGen/AMDGPU/smed3.ll
@@ -1,5 +1,5 @@
-; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -check-prefix=GCN -check-prefix=SICIVI -check-prefix=SI %s
-; RUN: llc -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -check-prefix=GCN -check-prefix=SICIVI -check-prefix=VI %s
+; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -check-prefix=GCN -check-prefix=SI %s
+; RUN: llc -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -check-prefix=GCN -check-prefix=VI %s
 ; RUN: llc -march=amdgcn -mcpu=gfx900 -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -check-prefix=GCN -check-prefix=GFX9 %s
 
 declare i32 @llvm.amdgcn.workitem.id.x() #0
@@ -94,7 +94,9 @@ declare i64 @llvm.smax.i64(i64, i64)
 declare i64 @llvm.smin.i64(i64, i64)
 
 ; GCN-LABEL: {{^}}v_test_smed3_r_i_i_i16:
-; SICIVI: v_med3_i32 v{{[0-9]+}}, v{{[0-9]+}}, 12, 17
+; SI: v_med3_i32 v{{[0-9]+}}, v{{[0-9]+}}, 12, 17
+; VI: v_max_i16_e32 [[MAX:v[0-9]]], 12, {{v[0-9]}}
+; VI: v_min_i16_e32 {{v[0-9]}}, 17, [[MAX]]
 ; GFX9: v_med3_i16 v{{[0-9]+}}, v{{[0-9]+}}, 12, 17
 define amdgpu_kernel void @v_test_smed3_r_i_i_i16(ptr addrspace(1) %out, ptr addrspace(1) %aptr) #1 {
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
diff --git a/llvm/test/CodeGen/AMDGPU/sra.ll b/llvm/test/CodeGen/AMDGPU/sra.ll
index cac917902922e..7097f58004855 100644
--- a/llvm/test/CodeGen/AMDGPU/sra.ll
+++ b/llvm/test/CodeGen/AMDGPU/sra.ll
@@ -323,52 +323,43 @@ define amdgpu_kernel void @ashr_v4i16(ptr addrspace(1) %out, ptr addrspace(1) %i
 ;
 ; EG-LABEL: ashr_v4i16:
 ; EG:       ; %bb.0:
-; EG-NEXT:    ALU 0, @8, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    ALU 1, @8, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    TEX 0 @6
-; EG-NEXT:    ALU 58, @9, KC0[CB0:0-32], KC1[]
+; EG-NEXT:    ALU 48, @10, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    MEM_RAT_CACHELESS STORE_RAW T10.XY, T9.X, 1
 ; EG-NEXT:    CF_END
 ; EG-NEXT:    PAD
 ; EG-NEXT:    Fetch clause starting at 6:
 ; EG-NEXT:     VTX_READ_128 T9.XYZW, T9.X, 0, #1
 ; EG-NEXT:    ALU clause starting at 8:
+; EG-NEXT:     MOV * T0.Y, T6.X,
 ; EG-NEXT:     MOV * T9.X, KC0[2].Z,
-; EG-NEXT:    ALU clause starting at 9:
-; EG-NEXT:     MOV T4.X, T9.X,
-; EG-NEXT:     MOV * T5.X, T9.Y,
-; EG-NEXT:     MOV T0.Y, PV.X,
-; EG-NEXT:     MOV * T0.Z, PS,
-; EG-NEXT:     MOV T2.X, T9.Z,
-; EG-NEXT:     MOV * T3.X, T9.W,
-; EG-NEXT:     MOV * T0.W, T6.X,
-; EG-NEXT:     MOV T1.Y, T2.X,
-; EG-NEXT:     BFE_INT * T1.W, T0.Y, 0.0, literal.x,
-; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; EG-NEXT:     AND_INT * T2.W, PV.Y, literal.x,
-; EG-NEXT:    65535(9.183409e-41), 0(0.000000e+00)
-; EG-NEXT:     ASHR * T1.W, T1.W, PV.W,
-; EG-NEXT:     AND_INT T1.W, PV.W, literal.x,
-; EG-NEXT:     AND_INT * T0.W, T0.W, literal.y,
+; EG-NEXT:    ALU clause starting at 10:
+; EG-NEXT:     BFE_INT T0.W, T9.X, 0.0, literal.x,
+; EG-NEXT:     AND_INT * T1.W, T9.Z, literal.y,
+; EG-NEXT:    16(2.242078e-44), 65535(9.183409e-41)
+; EG-NEXT:     ASHR * T0.W, PV.W, PS,
+; EG-NEXT:     AND_INT T0.W, PV.W, literal.x,
+; EG-NEXT:     AND_INT * T1.W, T0.Y, literal.y,
 ; EG-NEXT:    65535(9.183409e-41), -65536(nan)
 ; EG-NEXT:     OR_INT * T0.W, PS, PV.W,
-; EG-NEXT:     MOV * T1.Z, T3.X,
-; EG-NEXT:     MOV * T6.X, T0.W,
-; EG-NEXT:     MOV T0.W, PV.X,
-; EG-NEXT:     LSHR * T1.W, T0.Y, literal.x,
+; EG-NEXT:     MOV * T6.X, PV.W,
+; EG-NEXT:     MOV T0.Y, PV.X,
+; EG-NEXT:     LSHR * T0.W, T9.X, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; EG-NEXT:     BFE_INT T1.W, PS, 0.0, literal.x,
-; EG-NEXT:     LSHR * T2.W, T1.Y, literal.x,
+; EG-NEXT:     BFE_INT T0.W, PV.W, 0.0, literal.x,
+; EG-NEXT:     LSHR * T1.W, T9.Z, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; EG-NEXT:     ASHR T1.W, PV.W, PS,
-; EG-NEXT:     AND_INT * T0.W, T0.W, literal.x,
+; EG-NEXT:     ASHR T0.W, PV.W, PS,
+; EG-NEXT:     AND_INT * T1.W, T0.Y, literal.x,
 ; EG-NEXT:    65535(9.183409e-41), 0(0.000000e+00)
-; EG-NEXT:     LSHL * T1.W, PV.W, literal.x,
+; EG-NEXT:     LSHL * T0.W, PV.W, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
-; EG-NEXT:     OR_INT * T0.W, T0.W, PV.W,
+; EG-NEXT:     OR_INT * T0.W, T1.W, PV.W,
 ; EG-NEXT:     MOV T6.X, PV.W,
 ; EG-NEXT:     MOV T0.Y, T7.X,
-; EG-NEXT:     BFE_INT T0.W, T0.Z, 0.0, literal.x,
-; EG-NEXT:     AND_INT * T1.W, T1.Z, literal.y,
+; EG-NEXT:     BFE_INT T0.W, T9.Y, 0.0, literal.x,
+; EG-NEXT:     AND_INT * T1.W, T9.W, literal.y,
 ; EG-NEXT:    16(2.242078e-44), 65535(9.183409e-41)
 ; EG-NEXT:     ASHR T0.W, PV.W, PS,
 ; EG-NEXT:     AND_INT * T1.W, PV.Y, literal.x,
@@ -378,10 +369,10 @@ define amdgpu_kernel void @ashr_v4i16(ptr addrspace(1) %out, ptr addrspace(1) %i
 ; EG-NEXT:     OR_INT * T0.W, T1.W, PV.W,
 ; EG-NEXT:     MOV * T7.X, PV.W,
 ; EG-NEXT:     MOV T0.Y, PV.X,
-; EG-NEXT:     LSHR * T0.W, T0.Z, literal.x,
+; EG-NEXT:     LSHR * T0.W, T9.Y, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
 ; EG-NEXT:     BFE_INT T0.W, PV.W, 0.0, literal.x,
-; EG-NEXT:     LSHR * T1.W, T1.Z, literal.x,
+; EG-NEXT:     LSHR * T1.W, T9.W, literal.x,
 ; EG-NEXT:    16(2.242078e-44), 0(0.000000e+00)
 ; EG-NEXT:     ASHR T0.W, PV.W, PS,
 ; EG-NEXT:     AND_INT * T1.W, T0.Y, literal.x,
diff --git a/llvm/test/CodeGen/AMDGPU/ssubsat.ll b/llvm/test/CodeGen/AMDGPU/ssubsat.ll
index 13deecbc78857..ad9ecd24555f9 100644
--- a/llvm/test/CodeGen/AMDGPU/ssubsat.ll
+++ b/llvm/test/CodeGen/AMDGPU/ssubsat.ll
@@ -12,8 +12,9 @@ define i8 @v_ssubsat_i8(i8 %lhs, i8 %rhs) {
 ; GFX6-NEXT:    v_bfe_i32 v1, v1, 0, 8
 ; GFX6-NEXT:    v_bfe_i32 v0, v0, 0, 8
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, v0, v1
-; GFX6-NEXT:    v_min_i32_e32 v0, 0x7f, v0
-; GFX6-NEXT:    v_max_i32_e32 v0, 0xffffff80, v0
+; GFX6-NEXT:    s_movk_i32 s4, 0xff80
+; GFX6-NEXT:    v_mov_b32_e32 v1, 0x7f
+; GFX6-NEXT:    v_med3_i32 v0, v0, s4, v1
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_ssubsat_i8:
@@ -53,8 +54,9 @@ define i16 @v_ssubsat_i16(i16 %lhs, i16 %rhs) {
 ; GFX6-NEXT:    v_bfe_i32 v1, v1, 0, 16
 ; GFX6-NEXT:    v_bfe_i32 v0, v0, 0, 16
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, v0, v1
-; GFX6-NEXT:    v_min_i32_e32 v0, 0x7fff, v0
-; GFX6-NEXT:    v_max_i32_e32 v0, 0xffff8000, v0
+; GFX6-NEXT:    s_movk_i32 s4, 0x8000
+; GFX6-NEXT:    v_mov_b32_e32 v1, 0x7fff
+; GFX6-NEXT:    v_med3_i32 v0, v0, s4, v1
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_ssubsat_i16:
@@ -135,14 +137,14 @@ define <2 x i16> @v_ssubsat_v2i16(<2 x i16> %lhs, <2 x i16> %rhs) {
 ; GFX6-NEXT:    v_bfe_i32 v3, v3, 0, 16
 ; GFX6-NEXT:    v_bfe_i32 v1, v1, 0, 16
 ; GFX6-NEXT:    v_sub_i32_e32 v1, vcc, v1, v3
+; GFX6-NEXT:    s_movk_i32 s4, 0x8000
+; GFX6-NEXT:    v_mov_b32_e32 v3, 0x7fff
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
-; GFX6-NEXT:    v_min_i32_e32 v1, 0x7fff, v1
-; GFX6-NEXT:    v_min_i32_e32 v0, 0x7fff, v0
-; GFX6-NEXT:    v_max_i32_e32 v1, 0xffff8000, v1
-; GFX6-NEXT:    v_max_i32_e32 v0, 0xffff8000, v0
-; GFX6-NEXT:    v_lshlrev_b32_e32 v3, 16, v1
+; GFX6-NEXT:    v_med3_i32 v1, v1, s4, v3
+; GFX6-NEXT:    v_med3_i32 v0, v0, s4, v3
+; GFX6-NEXT:    v_lshlrev_b32_e32 v4, 16, v1
 ; GFX6-NEXT:    v_and_b32_e32 v0, 0xffff, v0
-; GFX6-NEXT:    v_or_b32_e32 v0, v0, v3
+; GFX6-NEXT:    v_or_b32_e32 v0, v0, v4
 ; GFX6-NEXT:    v_and_b32_e32 v1, 0xffff, v1
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
@@ -196,16 +198,15 @@ define <3 x i16> @v_ssubsat_v3i16(<3 x i16> %lhs, <3 x i16> %rhs) {
 ; GFX6-NEXT:    v_bfe_i32 v5, v5, 0, 16
 ; GFX6-NEXT:    v_bfe_i32 v2, v2, 0, 16
 ; GFX6-NEXT:    v_sub_i32_e32 v1, vcc, v1, v4
+; GFX6-NEXT:    s_movk_i32 s4, 0x8000
+; GFX6-NEXT:    v_mov_b32_e32 v4, 0x7fff
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, v0, v3
-; GFX6-NEXT:    v_min_i32_e32 v1, 0x7fff, v1
-; GFX6-NEXT:    v_min_i32_e32 v0, 0x7fff, v0
+; GFX6-NEXT:    v_med3_i32 v1, v1, s4, v4
+; GFX6-NEXT:    v_med3_i32 v0, v0, s4, v4
 ; GFX6-NEXT:    v_sub_i32_e32 v2, vcc, v2, v5
-; GFX6-NEXT:    v_max_i32_e32 v1, 0xffff8000, v1
-; GFX6-NEXT:    v_max_i32_e32 v0, 0xffff8000, v0
-; GFX6-NEXT:    v_min_i32_e32 v2, 0x7fff, v2
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
 ; GFX6-NEXT:    v_and_b32_e32 v0, 0xffff, v0
-; GFX6-NEXT:    v_max_i32_e32 v3, 0xffff8000, v2
+; GFX6-NEXT:    v_med3_i32 v3, v2, s4, v4
 ; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX6-NEXT:    v_and_b32_e32 v2, 0xffff, v3
 ; GFX6-NEXT:    v_alignbit_b32 v1, v3, v1, 16
@@ -268,11 +269,11 @@ define <2 x float> @v_ssubsat_v4i16(<4 x i16> %lhs, <4 x i16> %rhs) {
 ; GFX6-NEXT:    v_bfe_i32 v5, v5, 0, 16
 ; GFX6-NEXT:    v_bfe_i32 v1, v1, 0, 16
 ; GFX6-NEXT:    v_sub_i32_e32 v1, vcc, v1, v5
+; GFX6-NEXT:    s_movk_i32 s4, 0x8000
+; GFX6-NEXT:    v_mov_b32_e32 v5, 0x7fff
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, v0, v4
-; GFX6-NEXT:    v_min_i32_e32 v1, 0x7fff, v1
-; GFX6-NEXT:    v_min_i32_e32 v0, 0x7fff, v0
-; GFX6-NEXT:    v_max_i32_e32 v1, 0xffff8000, v1
-; GFX6-NEXT:    v_max_i32_e32 v0, 0xffff8000, v0
+; GFX6-NEXT:    v_med3_i32 v1, v1, s4, v5
+; GFX6-NEXT:    v_med3_i32 v0, v0, s4, v5
 ; GFX6-NEXT:    v_bfe_i32 v6, v6, 0, 16
 ; GFX6-NEXT:    v_bfe_i32 v2, v2, 0, 16
 ; GFX6-NEXT:    v_bfe_i32 v7, v7, 0, 16
@@ -282,10 +283,8 @@ define <2 x float> @v_ssubsat_v4i16(<4 x i16> %lhs, <4 x i16> %rhs) {
 ; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX6-NEXT:    v_sub_i32_e32 v1, vcc, v3, v7
 ; GFX6-NEXT:    v_sub_i32_e32 v2, vcc, v2, v6
-; GFX6-NEXT:    v_min_i32_e32 v1, 0x7fff, v1
-; GFX6-NEXT:    v_min_i32_e32 v2, 0x7fff, v2
-; GFX6-NEXT:    v_max_i32_e32 v1, 0xffff8000, v1
-; GFX6-NEXT:    v_max_i32_e32 v2, 0xffff8000, v2
+; GFX6-NEXT:    v_med3_i32 v1, v1, s4, v5
+; GFX6-NEXT:    v_med3_i32 v2, v2, s4, v5
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
 ; GFX6-NEXT:    v_and_b32_e32 v2, 0xffff, v2
 ; GFX6-NEXT:    v_or_b32_e32 v1, v2, v1
diff --git a/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-any.ll b/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-any.ll
new file mode 100644
index 0000000000000..2d2d64910c4fb
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-any.ll
@@ -0,0 +1,27 @@
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx90a < %s | FileCheck --check-prefixes=ASM %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx90a --filetype=obj < %s | llvm-objdump -s -j .rodata - | FileCheck --check-prefixes=OBJ %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx90a --filetype=obj < %s | llvm-readelf --notes - | FileCheck --check-prefixes=ELF %s
+
+; TODO: Update to check for granulated sgpr count directive once one is added.
+
+define amdgpu_kernel void @kern() {
+; ASM-LABEL: kern:
+; ASM: .amdhsa_next_free_sgpr 5
+; ASM: .amdhsa_reserve_xnack_mask 1
+
+; Verify that an extra SGPR block is reserved with XNACK "any" tid setting.
+; OBJ: Contents of section .rodata:
+; OBJ-NEXT: 0000 00000000 00000000 00000000 00000000  ................
+; OBJ-NEXT: 0010 00000000 00000000 00000000 00000000  ................
+; OBJ-NEXT: 0020 00000000 00000000 00000000 00000000  ................
+; OBJ-NEXT: 0030 4000af00 88000000 01000000 00000000  @...............
+
+; ELF: AMDGPU Metadata
+; ELF: .sgpr_count:     9
+entry:
+  tail call void asm sideeffect "", "~{s[0:4]}"()
+  ret void
+}
+
+!llvm.module.flags = !{!0}
+!0 = !{i32 1, !"amdgpu_code_object_version", i32 400}
diff --git a/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-off.ll b/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-off.ll
new file mode 100644
index 0000000000000..e676f4f8de74d
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-off.ll
@@ -0,0 +1,27 @@
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx90a -mattr=-xnack < %s | FileCheck --check-prefixes=ASM %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx90a -mattr=-xnack --filetype=obj < %s | llvm-objdump -s -j .rodata - | FileCheck --check-prefixes=OBJ %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx90a -mattr=-xnack --filetype=obj < %s | llvm-readelf --notes - | FileCheck --check-prefixes=ELF %s
+
+; TODO: Update to check for granulated sgpr count directive once one is added.
+
+define amdgpu_kernel void @kern() {
+; ASM-LABEL: kern:
+; ASM: .amdhsa_next_free_sgpr 5
+; ASM: .amdhsa_reserve_xnack_mask 0
+
+; Verify that an extra SGPR block is not reserved with XNACK "off" tid setting.
+; OBJ: Contents of section .rodata:
+; OBJ-NEXT: 0000 00000000 00000000 00000000 00000000  ................
+; OBJ-NEXT: 0010 00000000 00000000 00000000 00000000  ................
+; OBJ-NEXT: 0020 00000000 00000000 00000000 00000000  ................
+; OBJ-NEXT: 0030 0000af00 88000000 01000000 00000000  ................
+
+; ELF: AMDGPU Metadata
+; ELF: .sgpr_count:     5
+entry:
+  tail call void asm sideeffect "", "~{s[0:4]}"()
+  ret void
+}
+
+!llvm.module.flags = !{!0}
+!0 = !{i32 1, !"amdgpu_code_object_version", i32 400}
diff --git a/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-on.ll b/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-on.ll
new file mode 100644
index 0000000000000..705bedf450975
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-on.ll
@@ -0,0 +1,27 @@
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx90a -mattr=+xnack < %s | FileCheck --check-prefixes=ASM %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx90a -mattr=+xnack --filetype=obj < %s | llvm-objdump -s -j .rodata - | FileCheck --check-prefixes=OBJ %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx90a -mattr=+xnack --filetype=obj < %s | llvm-readelf --notes - | FileCheck --check-prefixes=ELF %s
+
+; TODO: Update to check for granulated sgpr count directive once one is added.
+
+define amdgpu_kernel void @kern() {
+; ASM-LABEL: kern:
+; ASM: .amdhsa_next_free_sgpr 5
+; ASM: .amdhsa_reserve_xnack_mask 1
+
+; Verify that an extra SGPR block is reserved with XNACK "on" tid setting.
+; OBJ: Contents of section .rodata:
+; OBJ-NEXT: 0000 00000000 00000000 00000000 00000000  ................
+; OBJ-NEXT: 0010 00000000 00000000 00000000 00000000  ................
+; OBJ-NEXT: 0020 00000000 00000000 00000000 00000000  ................
+; OBJ-NEXT: 0030 4000af00 88000000 01000000 00000000  @...............
+
+; ELF: AMDGPU Metadata
+; ELF: .sgpr_count:     9
+entry:
+  tail call void asm sideeffect "", "~{s[0:4]}"()
+  ret void
+}
+
+!llvm.module.flags = !{!0}
+!0 = !{i32 1, !"amdgpu_code_object_version", i32 400}
diff --git a/llvm/test/CodeGen/AMDGPU/trap-abis.ll b/llvm/test/CodeGen/AMDGPU/trap-abis.ll
index 8cdd8ad002c69..c9987ac7831a4 100644
--- a/llvm/test/CodeGen/AMDGPU/trap-abis.ll
+++ b/llvm/test/CodeGen/AMDGPU/trap-abis.ll
@@ -17,7 +17,75 @@ declare void @llvm.debugtrap() #1
 
 define amdgpu_kernel void @trap(ptr addrspace(1) nocapture readonly %arg0) {
 ; NOHSA-TRAP-GFX900-V2-LABEL: trap:
-; NOHSA-TRAP-GFX900-V2:       ; %bb.0:
+; NOHSA-TRAP-GFX900-V2:         .amd_kernel_code_t
+; NOHSA-TRAP-GFX900-V2-NEXT:     amd_code_version_major = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     amd_code_version_minor = 2
+; NOHSA-TRAP-GFX900-V2-NEXT:     amd_machine_kind = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     amd_machine_version_major = 9
+; NOHSA-TRAP-GFX900-V2-NEXT:     amd_machine_version_minor = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     amd_machine_version_stepping = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     kernel_code_entry_byte_offset = 256
+; NOHSA-TRAP-GFX900-V2-NEXT:     kernel_code_prefetch_byte_size = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     granulated_workitem_vgpr_count = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     granulated_wavefront_sgpr_count = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     priority = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     float_mode = 240
+; NOHSA-TRAP-GFX900-V2-NEXT:     priv = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_dx10_clamp = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     debug_mode = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_ieee_mode = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_wgp_mode = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_mem_ordered = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_fwd_progress = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_private_segment_wave_byte_offset = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     user_sgpr_count = 4
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_trap_handler = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_workgroup_id_x = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_workgroup_id_y = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_workgroup_id_z = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_workgroup_info = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_vgpr_workitem_id = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_exception_msb = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     granulated_lds_size = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_exception = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_private_segment_buffer = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_dispatch_ptr = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_queue_ptr = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_kernarg_segment_ptr = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_dispatch_id = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_flat_scratch_init = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_private_segment_size = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_grid_workgroup_count_x = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_grid_workgroup_count_y = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_grid_workgroup_count_z = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_wavefront_size32 = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_ordered_append_gds = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     private_element_size = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     is_ptr64 = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     is_dynamic_callstack = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     is_debug_enabled = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     is_xnack_enabled = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     workitem_private_segment_byte_size = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     workgroup_group_segment_byte_size = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     gds_segment_byte_size = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     kernarg_segment_byte_size = 44
+; NOHSA-TRAP-GFX900-V2-NEXT:     workgroup_fbarrier_count = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     wavefront_sgpr_count = 8
+; NOHSA-TRAP-GFX900-V2-NEXT:     workitem_vgpr_count = 2
+; NOHSA-TRAP-GFX900-V2-NEXT:     reserved_vgpr_first = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     reserved_vgpr_count = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     reserved_sgpr_first = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     reserved_sgpr_count = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     debug_wavefront_private_segment_offset_sgpr = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     debug_private_segment_buffer_sgpr = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     kernarg_segment_alignment = 4
+; NOHSA-TRAP-GFX900-V2-NEXT:     group_segment_alignment = 4
+; NOHSA-TRAP-GFX900-V2-NEXT:     private_segment_alignment = 4
+; NOHSA-TRAP-GFX900-V2-NEXT:     wavefront_size = 6
+; NOHSA-TRAP-GFX900-V2-NEXT:     call_convention = -1
+; NOHSA-TRAP-GFX900-V2-NEXT:     runtime_loader_kernel_symbol = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:    .end_amd_kernel_code_t
+; NOHSA-TRAP-GFX900-V2-NEXT:  ; %bb.0:
 ; NOHSA-TRAP-GFX900-V2-NEXT:    s_load_dwordx2 s[0:1], s[2:3], 0x24
 ; NOHSA-TRAP-GFX900-V2-NEXT:    v_mov_b32_e32 v0, 0
 ; NOHSA-TRAP-GFX900-V2-NEXT:    v_mov_b32_e32 v1, 1
@@ -161,7 +229,7 @@ define amdgpu_kernel void @trap(ptr addrspace(1) nocapture readonly %arg0) {
 ; HSA-TRAP-GFX900-V2-NEXT:     kernel_code_entry_byte_offset = 256
 ; HSA-TRAP-GFX900-V2-NEXT:     kernel_code_prefetch_byte_size = 0
 ; HSA-TRAP-GFX900-V2-NEXT:     granulated_workitem_vgpr_count = 0
-; HSA-TRAP-GFX900-V2-NEXT:     granulated_wavefront_sgpr_count = 0
+; HSA-TRAP-GFX900-V2-NEXT:     granulated_wavefront_sgpr_count = 1
 ; HSA-TRAP-GFX900-V2-NEXT:     priority = 0
 ; HSA-TRAP-GFX900-V2-NEXT:     float_mode = 240
 ; HSA-TRAP-GFX900-V2-NEXT:     priv = 0
@@ -204,7 +272,7 @@ define amdgpu_kernel void @trap(ptr addrspace(1) nocapture readonly %arg0) {
 ; HSA-TRAP-GFX900-V2-NEXT:     gds_segment_byte_size = 0
 ; HSA-TRAP-GFX900-V2-NEXT:     kernarg_segment_byte_size = 8
 ; HSA-TRAP-GFX900-V2-NEXT:     workgroup_fbarrier_count = 0
-; HSA-TRAP-GFX900-V2-NEXT:     wavefront_sgpr_count = 8
+; HSA-TRAP-GFX900-V2-NEXT:     wavefront_sgpr_count = 12
 ; HSA-TRAP-GFX900-V2-NEXT:     workitem_vgpr_count = 2
 ; HSA-TRAP-GFX900-V2-NEXT:     reserved_vgpr_first = 0
 ; HSA-TRAP-GFX900-V2-NEXT:     reserved_vgpr_count = 0
@@ -261,7 +329,7 @@ define amdgpu_kernel void @trap(ptr addrspace(1) nocapture readonly %arg0) {
 ; HSA-NOTRAP-GFX900-V2-NEXT:     kernel_code_entry_byte_offset = 256
 ; HSA-NOTRAP-GFX900-V2-NEXT:     kernel_code_prefetch_byte_size = 0
 ; HSA-NOTRAP-GFX900-V2-NEXT:     granulated_workitem_vgpr_count = 0
-; HSA-NOTRAP-GFX900-V2-NEXT:     granulated_wavefront_sgpr_count = 0
+; HSA-NOTRAP-GFX900-V2-NEXT:     granulated_wavefront_sgpr_count = 1
 ; HSA-NOTRAP-GFX900-V2-NEXT:     priority = 0
 ; HSA-NOTRAP-GFX900-V2-NEXT:     float_mode = 240
 ; HSA-NOTRAP-GFX900-V2-NEXT:     priv = 0
@@ -304,7 +372,7 @@ define amdgpu_kernel void @trap(ptr addrspace(1) nocapture readonly %arg0) {
 ; HSA-NOTRAP-GFX900-V2-NEXT:     gds_segment_byte_size = 0
 ; HSA-NOTRAP-GFX900-V2-NEXT:     kernarg_segment_byte_size = 8
 ; HSA-NOTRAP-GFX900-V2-NEXT:     workgroup_fbarrier_count = 0
-; HSA-NOTRAP-GFX900-V2-NEXT:     wavefront_sgpr_count = 8
+; HSA-NOTRAP-GFX900-V2-NEXT:     wavefront_sgpr_count = 12
 ; HSA-NOTRAP-GFX900-V2-NEXT:     workitem_vgpr_count = 2
 ; HSA-NOTRAP-GFX900-V2-NEXT:     reserved_vgpr_first = 0
 ; HSA-NOTRAP-GFX900-V2-NEXT:     reserved_vgpr_count = 0
@@ -356,7 +424,75 @@ define amdgpu_kernel void @trap(ptr addrspace(1) nocapture readonly %arg0) {
 
 define amdgpu_kernel void @non_entry_trap(ptr addrspace(1) nocapture readonly %arg0) local_unnamed_addr {
 ; NOHSA-TRAP-GFX900-V2-LABEL: non_entry_trap:
-; NOHSA-TRAP-GFX900-V2:       ; %bb.0: ; %entry
+; NOHSA-TRAP-GFX900-V2:         .amd_kernel_code_t
+; NOHSA-TRAP-GFX900-V2-NEXT:     amd_code_version_major = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     amd_code_version_minor = 2
+; NOHSA-TRAP-GFX900-V2-NEXT:     amd_machine_kind = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     amd_machine_version_major = 9
+; NOHSA-TRAP-GFX900-V2-NEXT:     amd_machine_version_minor = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     amd_machine_version_stepping = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     kernel_code_entry_byte_offset = 256
+; NOHSA-TRAP-GFX900-V2-NEXT:     kernel_code_prefetch_byte_size = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     granulated_workitem_vgpr_count = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     granulated_wavefront_sgpr_count = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     priority = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     float_mode = 240
+; NOHSA-TRAP-GFX900-V2-NEXT:     priv = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_dx10_clamp = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     debug_mode = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_ieee_mode = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_wgp_mode = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_mem_ordered = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_fwd_progress = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_private_segment_wave_byte_offset = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     user_sgpr_count = 4
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_trap_handler = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_workgroup_id_x = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_workgroup_id_y = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_workgroup_id_z = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_workgroup_info = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_vgpr_workitem_id = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_exception_msb = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     granulated_lds_size = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_exception = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_private_segment_buffer = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_dispatch_ptr = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_queue_ptr = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_kernarg_segment_ptr = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_dispatch_id = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_flat_scratch_init = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_private_segment_size = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_grid_workgroup_count_x = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_grid_workgroup_count_y = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_grid_workgroup_count_z = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_wavefront_size32 = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_ordered_append_gds = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     private_element_size = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     is_ptr64 = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     is_dynamic_callstack = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     is_debug_enabled = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     is_xnack_enabled = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     workitem_private_segment_byte_size = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     workgroup_group_segment_byte_size = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     gds_segment_byte_size = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     kernarg_segment_byte_size = 44
+; NOHSA-TRAP-GFX900-V2-NEXT:     workgroup_fbarrier_count = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     wavefront_sgpr_count = 8
+; NOHSA-TRAP-GFX900-V2-NEXT:     workitem_vgpr_count = 2
+; NOHSA-TRAP-GFX900-V2-NEXT:     reserved_vgpr_first = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     reserved_vgpr_count = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     reserved_sgpr_first = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     reserved_sgpr_count = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     debug_wavefront_private_segment_offset_sgpr = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     debug_private_segment_buffer_sgpr = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     kernarg_segment_alignment = 4
+; NOHSA-TRAP-GFX900-V2-NEXT:     group_segment_alignment = 4
+; NOHSA-TRAP-GFX900-V2-NEXT:     private_segment_alignment = 4
+; NOHSA-TRAP-GFX900-V2-NEXT:     wavefront_size = 6
+; NOHSA-TRAP-GFX900-V2-NEXT:     call_convention = -1
+; NOHSA-TRAP-GFX900-V2-NEXT:     runtime_loader_kernel_symbol = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:    .end_amd_kernel_code_t
+; NOHSA-TRAP-GFX900-V2-NEXT:  ; %bb.0: ; %entry
 ; NOHSA-TRAP-GFX900-V2-NEXT:    s_load_dwordx2 s[0:1], s[2:3], 0x24
 ; NOHSA-TRAP-GFX900-V2-NEXT:    v_mov_b32_e32 v0, 0
 ; NOHSA-TRAP-GFX900-V2-NEXT:    s_waitcnt lgkmcnt(0)
@@ -591,7 +727,7 @@ define amdgpu_kernel void @non_entry_trap(ptr addrspace(1) nocapture readonly %a
 ; HSA-TRAP-GFX900-V2-NEXT:     gds_segment_byte_size = 0
 ; HSA-TRAP-GFX900-V2-NEXT:     kernarg_segment_byte_size = 8
 ; HSA-TRAP-GFX900-V2-NEXT:     workgroup_fbarrier_count = 0
-; HSA-TRAP-GFX900-V2-NEXT:     wavefront_sgpr_count = 10
+; HSA-TRAP-GFX900-V2-NEXT:     wavefront_sgpr_count = 12
 ; HSA-TRAP-GFX900-V2-NEXT:     workitem_vgpr_count = 2
 ; HSA-TRAP-GFX900-V2-NEXT:     reserved_vgpr_first = 0
 ; HSA-TRAP-GFX900-V2-NEXT:     reserved_vgpr_count = 0
@@ -712,7 +848,7 @@ define amdgpu_kernel void @non_entry_trap(ptr addrspace(1) nocapture readonly %a
 ; HSA-NOTRAP-GFX900-V2-NEXT:     gds_segment_byte_size = 0
 ; HSA-NOTRAP-GFX900-V2-NEXT:     kernarg_segment_byte_size = 8
 ; HSA-NOTRAP-GFX900-V2-NEXT:     workgroup_fbarrier_count = 0
-; HSA-NOTRAP-GFX900-V2-NEXT:     wavefront_sgpr_count = 10
+; HSA-NOTRAP-GFX900-V2-NEXT:     wavefront_sgpr_count = 12
 ; HSA-NOTRAP-GFX900-V2-NEXT:     workitem_vgpr_count = 2
 ; HSA-NOTRAP-GFX900-V2-NEXT:     reserved_vgpr_first = 0
 ; HSA-NOTRAP-GFX900-V2-NEXT:     reserved_vgpr_count = 0
@@ -792,7 +928,75 @@ ret:
 
 define amdgpu_kernel void @debugtrap(ptr addrspace(1) nocapture readonly %arg0) {
 ; NOHSA-TRAP-GFX900-V2-LABEL: debugtrap:
-; NOHSA-TRAP-GFX900-V2:       ; %bb.0:
+; NOHSA-TRAP-GFX900-V2:         .amd_kernel_code_t
+; NOHSA-TRAP-GFX900-V2-NEXT:     amd_code_version_major = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     amd_code_version_minor = 2
+; NOHSA-TRAP-GFX900-V2-NEXT:     amd_machine_kind = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     amd_machine_version_major = 9
+; NOHSA-TRAP-GFX900-V2-NEXT:     amd_machine_version_minor = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     amd_machine_version_stepping = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     kernel_code_entry_byte_offset = 256
+; NOHSA-TRAP-GFX900-V2-NEXT:     kernel_code_prefetch_byte_size = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     granulated_workitem_vgpr_count = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     granulated_wavefront_sgpr_count = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     priority = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     float_mode = 240
+; NOHSA-TRAP-GFX900-V2-NEXT:     priv = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_dx10_clamp = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     debug_mode = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_ieee_mode = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_wgp_mode = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_mem_ordered = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_fwd_progress = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_private_segment_wave_byte_offset = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     user_sgpr_count = 2
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_trap_handler = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_workgroup_id_x = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_workgroup_id_y = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_workgroup_id_z = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_workgroup_info = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_vgpr_workitem_id = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_exception_msb = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     granulated_lds_size = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_exception = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_private_segment_buffer = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_dispatch_ptr = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_queue_ptr = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_kernarg_segment_ptr = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_dispatch_id = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_flat_scratch_init = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_private_segment_size = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_grid_workgroup_count_x = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_grid_workgroup_count_y = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_sgpr_grid_workgroup_count_z = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_wavefront_size32 = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     enable_ordered_append_gds = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     private_element_size = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     is_ptr64 = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     is_dynamic_callstack = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     is_debug_enabled = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     is_xnack_enabled = 1
+; NOHSA-TRAP-GFX900-V2-NEXT:     workitem_private_segment_byte_size = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     workgroup_group_segment_byte_size = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     gds_segment_byte_size = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     kernarg_segment_byte_size = 44
+; NOHSA-TRAP-GFX900-V2-NEXT:     workgroup_fbarrier_count = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     wavefront_sgpr_count = 6
+; NOHSA-TRAP-GFX900-V2-NEXT:     workitem_vgpr_count = 3
+; NOHSA-TRAP-GFX900-V2-NEXT:     reserved_vgpr_first = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     reserved_vgpr_count = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     reserved_sgpr_first = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     reserved_sgpr_count = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     debug_wavefront_private_segment_offset_sgpr = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     debug_private_segment_buffer_sgpr = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:     kernarg_segment_alignment = 4
+; NOHSA-TRAP-GFX900-V2-NEXT:     group_segment_alignment = 4
+; NOHSA-TRAP-GFX900-V2-NEXT:     private_segment_alignment = 4
+; NOHSA-TRAP-GFX900-V2-NEXT:     wavefront_size = 6
+; NOHSA-TRAP-GFX900-V2-NEXT:     call_convention = -1
+; NOHSA-TRAP-GFX900-V2-NEXT:     runtime_loader_kernel_symbol = 0
+; NOHSA-TRAP-GFX900-V2-NEXT:    .end_amd_kernel_code_t
+; NOHSA-TRAP-GFX900-V2-NEXT:  ; %bb.0:
 ; NOHSA-TRAP-GFX900-V2-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; NOHSA-TRAP-GFX900-V2-NEXT:    v_mov_b32_e32 v0, 0
 ; NOHSA-TRAP-GFX900-V2-NEXT:    v_mov_b32_e32 v1, 1
@@ -954,7 +1158,7 @@ define amdgpu_kernel void @debugtrap(ptr addrspace(1) nocapture readonly %arg0)
 ; HSA-TRAP-GFX900-V2-NEXT:     kernel_code_entry_byte_offset = 256
 ; HSA-TRAP-GFX900-V2-NEXT:     kernel_code_prefetch_byte_size = 0
 ; HSA-TRAP-GFX900-V2-NEXT:     granulated_workitem_vgpr_count = 0
-; HSA-TRAP-GFX900-V2-NEXT:     granulated_wavefront_sgpr_count = 0
+; HSA-TRAP-GFX900-V2-NEXT:     granulated_wavefront_sgpr_count = 1
 ; HSA-TRAP-GFX900-V2-NEXT:     priority = 0
 ; HSA-TRAP-GFX900-V2-NEXT:     float_mode = 240
 ; HSA-TRAP-GFX900-V2-NEXT:     priv = 0
@@ -997,7 +1201,7 @@ define amdgpu_kernel void @debugtrap(ptr addrspace(1) nocapture readonly %arg0)
 ; HSA-TRAP-GFX900-V2-NEXT:     gds_segment_byte_size = 0
 ; HSA-TRAP-GFX900-V2-NEXT:     kernarg_segment_byte_size = 8
 ; HSA-TRAP-GFX900-V2-NEXT:     workgroup_fbarrier_count = 0
-; HSA-TRAP-GFX900-V2-NEXT:     wavefront_sgpr_count = 6
+; HSA-TRAP-GFX900-V2-NEXT:     wavefront_sgpr_count = 10
 ; HSA-TRAP-GFX900-V2-NEXT:     workitem_vgpr_count = 3
 ; HSA-TRAP-GFX900-V2-NEXT:     reserved_vgpr_first = 0
 ; HSA-TRAP-GFX900-V2-NEXT:     reserved_vgpr_count = 0
@@ -1064,7 +1268,7 @@ define amdgpu_kernel void @debugtrap(ptr addrspace(1) nocapture readonly %arg0)
 ; HSA-NOTRAP-GFX900-V2-NEXT:     kernel_code_entry_byte_offset = 256
 ; HSA-NOTRAP-GFX900-V2-NEXT:     kernel_code_prefetch_byte_size = 0
 ; HSA-NOTRAP-GFX900-V2-NEXT:     granulated_workitem_vgpr_count = 0
-; HSA-NOTRAP-GFX900-V2-NEXT:     granulated_wavefront_sgpr_count = 0
+; HSA-NOTRAP-GFX900-V2-NEXT:     granulated_wavefront_sgpr_count = 1
 ; HSA-NOTRAP-GFX900-V2-NEXT:     priority = 0
 ; HSA-NOTRAP-GFX900-V2-NEXT:     float_mode = 240
 ; HSA-NOTRAP-GFX900-V2-NEXT:     priv = 0
@@ -1107,7 +1311,7 @@ define amdgpu_kernel void @debugtrap(ptr addrspace(1) nocapture readonly %arg0)
 ; HSA-NOTRAP-GFX900-V2-NEXT:     gds_segment_byte_size = 0
 ; HSA-NOTRAP-GFX900-V2-NEXT:     kernarg_segment_byte_size = 8
 ; HSA-NOTRAP-GFX900-V2-NEXT:     workgroup_fbarrier_count = 0
-; HSA-NOTRAP-GFX900-V2-NEXT:     wavefront_sgpr_count = 6
+; HSA-NOTRAP-GFX900-V2-NEXT:     wavefront_sgpr_count = 10
 ; HSA-NOTRAP-GFX900-V2-NEXT:     workitem_vgpr_count = 3
 ; HSA-NOTRAP-GFX900-V2-NEXT:     reserved_vgpr_first = 0
 ; HSA-NOTRAP-GFX900-V2-NEXT:     reserved_vgpr_count = 0
diff --git a/llvm/test/CodeGen/AMDGPU/umed3.ll b/llvm/test/CodeGen/AMDGPU/umed3.ll
index 13b47c0d7f8f9..ced86b98d779e 100644
--- a/llvm/test/CodeGen/AMDGPU/umed3.ll
+++ b/llvm/test/CodeGen/AMDGPU/umed3.ll
@@ -1,5 +1,5 @@
-; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -check-prefix=GCN -check-prefix=SICIVI -check-prefix=SI %s
-; RUN: llc -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -check-prefix=GCN -check-prefix=SICIVI -check-prefix=VI %s
+; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -check-prefix=GCN  -check-prefix=SI %s
+; RUN: llc -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -check-prefix=GCN -check-prefix=VI %s
 ; RUN: llc -march=amdgcn -mcpu=gfx900 -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -check-prefix=GCN -check-prefix=GFX9 %s
 
 declare i32 @llvm.amdgcn.workitem.id.x() #0
@@ -81,7 +81,9 @@ define amdgpu_kernel void @v_test_umed3_r_i_i_i64(ptr addrspace(1) %out, ptr add
 }
 
 ; GCN-LABEL: {{^}}v_test_umed3_r_i_i_i16:
-; SICIVI: v_med3_u32 v{{[0-9]+}}, v{{[0-9]+}}, 12, 17
+; SI: v_med3_u32 v{{[0-9]+}}, v{{[0-9]+}}, 12, 17
+; VI: v_max_u16_e32 [[MAX:v[0-9]]], 12, {{v[0-9]}}
+; VI: v_min_u16_e32 {{v[0-9]}}, 17, [[MAX]]
 ; GFX9: v_med3_u16 v{{[0-9]+}}, v{{[0-9]+}}, 12, 17
 define amdgpu_kernel void @v_test_umed3_r_i_i_i16(ptr addrspace(1) %out, ptr addrspace(1) %aptr) #1 {
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
diff --git a/llvm/test/CodeGen/AMDGPU/v_sat_pk_u8_i16.ll b/llvm/test/CodeGen/AMDGPU/v_sat_pk_u8_i16.ll
new file mode 100644
index 0000000000000..32eac94e0218b
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/v_sat_pk_u8_i16.ll
@@ -0,0 +1,492 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -march=amdgcn -mcpu=fiji -verify-machineinstrs < %s | FileCheck -check-prefixes=SDAG-VI %s
+; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX9,SDAG-GFX9 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1101 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX11,SDAG-GFX11 %s
+
+; RUN: llc -march=amdgcn -mcpu=fiji -verify-machineinstrs -global-isel < %s | FileCheck -check-prefixes=GISEL-VI %s
+; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs -global-isel < %s | FileCheck -check-prefixes=GFX9,GISEL-GFX9 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1101 -verify-machineinstrs -global-isel < %s | FileCheck -check-prefixes=GFX11,GISEL-GFX11 %s
+
+; <GFX9 has no V_SAT_PK, GFX9+ has V_SAT_PK, GFX11 has V_SAT_PK with t16
+
+declare i16 @llvm.smin.i16(i16, i16)
+declare i16 @llvm.smax.i16(i16, i16)
+
+declare <2 x i16> @llvm.smin.v2i16(<2 x i16>, <2 x i16>)
+declare <2 x i16> @llvm.smax.v2i16(<2 x i16>, <2 x i16>)
+
+define <2 x i16> @basic_smax_smin(i16 %src0, i16 %src1) {
+; SDAG-VI-LABEL: basic_smax_smin:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; SDAG-VI-NEXT:    v_max_i16_e32 v0, 0, v0
+; SDAG-VI-NEXT:    v_max_i16_e32 v1, 0, v1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, 0xff
+; SDAG-VI-NEXT:    v_min_i16_sdwa v1, v1, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; SDAG-VI-NEXT:    v_min_i16_e32 v0, 0xff, v0
+; SDAG-VI-NEXT:    v_or_b32_e32 v0, v0, v1
+; SDAG-VI-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9-LABEL: basic_smax_smin:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX9-NEXT:    v_lshl_or_b32 v0, v1, 16, v0
+; GFX9-NEXT:    v_sat_pk_u8_i16_e32 v0, v0
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11-LABEL: basic_smax_smin:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX11-NEXT:    v_lshl_or_b32 v0, v1, 16, v0
+; GFX11-NEXT:    v_sat_pk_u8_i16_e32 v0, v0
+; GFX11-NEXT:    s_setpc_b64 s[30:31]
+;
+; GISEL-VI-LABEL: basic_smax_smin:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GISEL-VI-NEXT:    v_max_i16_e32 v0, 0, v0
+; GISEL-VI-NEXT:    v_max_i16_e32 v1, 0, v1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, 0xff
+; GISEL-VI-NEXT:    v_min_i16_e32 v0, 0xff, v0
+; GISEL-VI-NEXT:    v_min_i16_sdwa v1, v1, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GISEL-VI-NEXT:    v_or_b32_e32 v0, v0, v1
+; GISEL-VI-NEXT:    s_setpc_b64 s[30:31]
+  %src0.max = call i16 @llvm.smax.i16(i16 %src0, i16 0)
+  %src0.clamp = call i16 @llvm.smin.i16(i16 %src0.max, i16 255)
+  %src1.max = call i16 @llvm.smax.i16(i16 %src1, i16 0)
+  %src1.clamp = call i16 @llvm.smin.i16(i16 %src1.max, i16 255)
+  %insert.0 = insertelement <2 x i16> undef, i16 %src0.clamp, i32 0
+  %vec = insertelement <2 x i16> %insert.0, i16 %src1.clamp, i32 1
+  ret <2 x i16> %vec
+}
+
+; Check that we don't emit a VALU instruction for SGPR inputs.
+define amdgpu_kernel void @basic_smax_smin_sgpr(ptr addrspace(1) %out, i32 inreg %src0ext, i32 inreg %src1ext) {
+; SDAG-VI-LABEL: basic_smax_smin_sgpr:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0xff
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_max_i16_e64 v1, s2, 0
+; SDAG-VI-NEXT:    v_max_i16_e64 v2, s3, 0
+; SDAG-VI-NEXT:    v_min_i16_sdwa v0, v2, v0 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; SDAG-VI-NEXT:    v_min_i16_e32 v1, 0xff, v1
+; SDAG-VI-NEXT:    v_or_b32_e32 v2, v1, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    flat_store_dword v[0:1], v2
+; SDAG-VI-NEXT:    s_endpgm
+;
+; SDAG-GFX9-LABEL: basic_smax_smin_sgpr:
+; SDAG-GFX9:       ; %bb.0:
+; SDAG-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v1, 0xff
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX9-NEXT:    v_med3_i16 v2, s2, 0, v1
+; SDAG-GFX9-NEXT:    v_med3_i16 v1, s3, 0, v1
+; SDAG-GFX9-NEXT:    v_and_b32_e32 v2, 0xffff, v2
+; SDAG-GFX9-NEXT:    v_lshl_or_b32 v1, v1, 16, v2
+; SDAG-GFX9-NEXT:    global_store_dword v0, v1, s[0:1]
+; SDAG-GFX9-NEXT:    s_endpgm
+;
+; SDAG-GFX11-LABEL: basic_smax_smin_sgpr:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_med3_i16 v0, s2, 0, 0xff
+; SDAG-GFX11-NEXT:    v_med3_i16 v1, s3, 0, 0xff
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; SDAG-GFX11-NEXT:    v_lshl_or_b32 v0, v1, 16, v0
+; SDAG-GFX11-NEXT:    global_store_b32 v2, v0, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: basic_smax_smin_sgpr:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_sext_i32_i16 s4, 0
+; GISEL-VI-NEXT:    s_sext_i32_i16 s5, 0xff
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    s_sext_i32_i16 s3, s3
+; GISEL-VI-NEXT:    s_sext_i32_i16 s2, s2
+; GISEL-VI-NEXT:    s_max_i32 s3, s3, s4
+; GISEL-VI-NEXT:    s_max_i32 s2, s2, s4
+; GISEL-VI-NEXT:    s_sext_i32_i16 s3, s3
+; GISEL-VI-NEXT:    s_sext_i32_i16 s2, s2
+; GISEL-VI-NEXT:    s_min_i32 s3, s3, s5
+; GISEL-VI-NEXT:    s_min_i32 s2, s2, s5
+; GISEL-VI-NEXT:    s_bfe_u32 s3, s3, 0x100000
+; GISEL-VI-NEXT:    s_bfe_u32 s2, s2, 0x100000
+; GISEL-VI-NEXT:    s_lshl_b32 s3, s3, 16
+; GISEL-VI-NEXT:    s_or_b32 s2, s2, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-VI-NEXT:    flat_store_dword v[0:1], v2
+; GISEL-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX9-LABEL: basic_smax_smin_sgpr:
+; GISEL-GFX9:       ; %bb.0:
+; GISEL-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX9-NEXT:    s_sext_i32_i16 s4, 0
+; GISEL-GFX9-NEXT:    s_sext_i32_i16 s5, 0xff
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX9-NEXT:    s_sext_i32_i16 s2, s2
+; GISEL-GFX9-NEXT:    s_sext_i32_i16 s3, s3
+; GISEL-GFX9-NEXT:    s_max_i32 s2, s2, s4
+; GISEL-GFX9-NEXT:    s_max_i32 s3, s3, s4
+; GISEL-GFX9-NEXT:    s_sext_i32_i16 s2, s2
+; GISEL-GFX9-NEXT:    s_sext_i32_i16 s3, s3
+; GISEL-GFX9-NEXT:    s_min_i32 s2, s2, s5
+; GISEL-GFX9-NEXT:    s_min_i32 s3, s3, s5
+; GISEL-GFX9-NEXT:    s_pack_ll_b32_b16 s2, s2, s3
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX9-NEXT:    global_store_dword v1, v0, s[0:1]
+; GISEL-GFX9-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: basic_smax_smin_sgpr:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    s_sext_i32_i16 s4, 0
+; GISEL-GFX11-NEXT:    s_sext_i32_i16 s5, 0xff
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    s_sext_i32_i16 s2, s2
+; GISEL-GFX11-NEXT:    s_sext_i32_i16 s3, s3
+; GISEL-GFX11-NEXT:    s_max_i32 s2, s2, s4
+; GISEL-GFX11-NEXT:    s_max_i32 s3, s3, s4
+; GISEL-GFX11-NEXT:    s_sext_i32_i16 s2, s2
+; GISEL-GFX11-NEXT:    s_sext_i32_i16 s3, s3
+; GISEL-GFX11-NEXT:    s_min_i32 s2, s2, s5
+; GISEL-GFX11-NEXT:    s_min_i32 s3, s3, s5
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GISEL-GFX11-NEXT:    s_pack_ll_b32_b16 s2, s2, s3
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+  %src0 = trunc i32 %src0ext to i16
+  %src1 = trunc i32 %src1ext to i16
+  %src0.max = call i16 @llvm.smax.i16(i16 %src0, i16 0)
+  %src0.clamp = call i16 @llvm.smin.i16(i16 %src0.max, i16 255)
+  %src1.max = call i16 @llvm.smax.i16(i16 %src1, i16 0)
+  %src1.clamp = call i16 @llvm.smin.i16(i16 %src1.max, i16 255)
+  %insert.0 = insertelement <2 x i16> undef, i16 %src0.clamp, i32 0
+  %vec = insertelement <2 x i16> %insert.0, i16 %src1.clamp, i32 1
+  store <2 x i16> %vec, ptr addrspace(1) %out
+  ret void
+}
+
+define <2 x i16> @basic_smin_smax(i16 %src0, i16 %src1) {
+; SDAG-VI-LABEL: basic_smin_smax:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; SDAG-VI-NEXT:    v_min_i16_e32 v0, 0xff, v0
+; SDAG-VI-NEXT:    v_min_i16_e32 v1, 0xff, v1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, 0
+; SDAG-VI-NEXT:    v_max_i16_sdwa v1, v1, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; SDAG-VI-NEXT:    v_max_i16_e32 v0, 0, v0
+; SDAG-VI-NEXT:    v_or_b32_e32 v0, v0, v1
+; SDAG-VI-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9-LABEL: basic_smin_smax:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX9-NEXT:    v_lshl_or_b32 v0, v1, 16, v0
+; GFX9-NEXT:    v_sat_pk_u8_i16_e32 v0, v0
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11-LABEL: basic_smin_smax:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX11-NEXT:    v_lshl_or_b32 v0, v1, 16, v0
+; GFX11-NEXT:    v_sat_pk_u8_i16_e32 v0, v0
+; GFX11-NEXT:    s_setpc_b64 s[30:31]
+;
+; GISEL-VI-LABEL: basic_smin_smax:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GISEL-VI-NEXT:    v_min_i16_e32 v0, 0xff, v0
+; GISEL-VI-NEXT:    v_min_i16_e32 v1, 0xff, v1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-VI-NEXT:    v_max_i16_e32 v0, 0, v0
+; GISEL-VI-NEXT:    v_max_i16_sdwa v1, v1, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GISEL-VI-NEXT:    v_or_b32_e32 v0, v0, v1
+; GISEL-VI-NEXT:    s_setpc_b64 s[30:31]
+  %src0.min = call i16 @llvm.smin.i16(i16 %src0, i16 255)
+  %src0.clamp = call i16 @llvm.smax.i16(i16 %src0.min, i16 0)
+  %src1.min = call i16 @llvm.smin.i16(i16 %src1, i16 255)
+  %src1.clamp = call i16 @llvm.smax.i16(i16 %src1.min, i16 0)
+  %insert.0 = insertelement <2 x i16> undef, i16 %src0.clamp, i32 0
+  %vec = insertelement <2 x i16> %insert.0, i16 %src1.clamp, i32 1
+  ret <2 x i16> %vec
+}
+
+define <2 x i16> @basic_smin_smax_combined(i16 %src0, i16 %src1) {
+; SDAG-VI-LABEL: basic_smin_smax_combined:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; SDAG-VI-NEXT:    v_min_i16_e32 v0, 0xff, v0
+; SDAG-VI-NEXT:    v_max_i16_e32 v1, 0, v1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, 0xff
+; SDAG-VI-NEXT:    v_min_i16_sdwa v1, v1, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; SDAG-VI-NEXT:    v_max_i16_e32 v0, 0, v0
+; SDAG-VI-NEXT:    v_or_b32_e32 v0, v0, v1
+; SDAG-VI-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9-LABEL: basic_smin_smax_combined:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX9-NEXT:    v_lshl_or_b32 v0, v1, 16, v0
+; GFX9-NEXT:    v_sat_pk_u8_i16_e32 v0, v0
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11-LABEL: basic_smin_smax_combined:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX11-NEXT:    v_lshl_or_b32 v0, v1, 16, v0
+; GFX11-NEXT:    v_sat_pk_u8_i16_e32 v0, v0
+; GFX11-NEXT:    s_setpc_b64 s[30:31]
+;
+; GISEL-VI-LABEL: basic_smin_smax_combined:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GISEL-VI-NEXT:    v_min_i16_e32 v0, 0xff, v0
+; GISEL-VI-NEXT:    v_max_i16_e32 v1, 0, v1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, 0xff
+; GISEL-VI-NEXT:    v_max_i16_e32 v0, 0, v0
+; GISEL-VI-NEXT:    v_min_i16_sdwa v1, v1, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GISEL-VI-NEXT:    v_or_b32_e32 v0, v0, v1
+; GISEL-VI-NEXT:    s_setpc_b64 s[30:31]
+  %src0.min = call i16 @llvm.smin.i16(i16 %src0, i16 255)
+  %src0.clamp = call i16 @llvm.smax.i16(i16 %src0.min, i16 0)
+  %src1.max = call i16 @llvm.smax.i16(i16 %src1, i16 0)
+  %src1.clamp = call i16 @llvm.smin.i16(i16 %src1.max, i16 255)
+  %insert.0 = insertelement <2 x i16> undef, i16 %src0.clamp, i32 0
+  %vec = insertelement <2 x i16> %insert.0, i16 %src1.clamp, i32 1
+  ret <2 x i16> %vec
+}
+
+define <2 x i16> @vec_smax_smin(<2 x i16> %src) {
+; SDAG-VI-LABEL: vec_smax_smin:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-VI-NEXT:    v_max_i16_sdwa v1, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; SDAG-VI-NEXT:    v_max_i16_e32 v0, 0, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, 0xff
+; SDAG-VI-NEXT:    v_min_i16_e32 v0, 0xff, v0
+; SDAG-VI-NEXT:    v_min_i16_sdwa v1, v1, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; SDAG-VI-NEXT:    v_or_b32_e32 v0, v0, v1
+; SDAG-VI-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9-LABEL: vec_smax_smin:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    v_sat_pk_u8_i16_e32 v0, v0
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11-LABEL: vec_smax_smin:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11-NEXT:    v_sat_pk_u8_i16_e32 v0, v0
+; GFX11-NEXT:    s_setpc_b64 s[30:31]
+;
+; GISEL-VI-LABEL: vec_smax_smin:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-VI-NEXT:    v_max_i16_e32 v1, 0, v0
+; GISEL-VI-NEXT:    v_max_i16_sdwa v0, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, 0xff
+; GISEL-VI-NEXT:    v_min_i16_e32 v1, 0xff, v1
+; GISEL-VI-NEXT:    v_min_i16_sdwa v0, v0, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GISEL-VI-NEXT:    v_or_b32_e32 v0, v1, v0
+; GISEL-VI-NEXT:    s_setpc_b64 s[30:31]
+  %src.max = call <2 x i16> @llvm.smax.v2i16(<2 x i16> %src, <2 x i16> <i16 0, i16 0>)
+  %src.clamp = call <2 x i16> @llvm.smin.v2i16(<2 x i16> %src.max, <2 x i16> <i16 255, i16 255>)
+  ret <2 x i16> %src.clamp
+}
+
+; Check that we don't emit a VALU instruction for SGPR inputs.
+define amdgpu_kernel void @vec_smax_smin_sgpr(ptr addrspace(1) %out, <2 x i16> inreg %src) {
+; SDAG-VI-LABEL: vec_smax_smin_sgpr:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0xff
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    s_lshr_b32 s3, s2, 16
+; SDAG-VI-NEXT:    v_max_i16_e64 v1, s2, 0
+; SDAG-VI-NEXT:    v_max_i16_e64 v2, s3, 0
+; SDAG-VI-NEXT:    v_min_i16_e32 v1, 0xff, v1
+; SDAG-VI-NEXT:    v_min_i16_sdwa v0, v2, v0 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; SDAG-VI-NEXT:    v_or_b32_e32 v2, v1, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    flat_store_dword v[0:1], v2
+; SDAG-VI-NEXT:    s_endpgm
+;
+; SDAG-GFX9-LABEL: vec_smax_smin_sgpr:
+; SDAG-GFX9:       ; %bb.0:
+; SDAG-GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX9-NEXT:    s_movk_i32 s0, 0xff
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX9-NEXT:    v_pk_max_i16 v1, s4, 0
+; SDAG-GFX9-NEXT:    v_pk_min_i16 v1, v1, s0 op_sel_hi:[1,0]
+; SDAG-GFX9-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX9-NEXT:    s_endpgm
+;
+; SDAG-GFX11-LABEL: vec_smax_smin_sgpr:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_pk_max_i16 v0, s2, 0
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_pk_min_i16 v0, 0xff, v0 op_sel_hi:[0,1]
+; SDAG-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: vec_smax_smin_sgpr:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_sext_i32_i16 s3, 0
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    s_lshr_b32 s4, s2, 16
+; GISEL-VI-NEXT:    s_sext_i32_i16 s2, s2
+; GISEL-VI-NEXT:    s_sext_i32_i16 s4, s4
+; GISEL-VI-NEXT:    s_max_i32 s2, s2, s3
+; GISEL-VI-NEXT:    s_max_i32 s3, s4, s3
+; GISEL-VI-NEXT:    s_sext_i32_i16 s4, 0xff
+; GISEL-VI-NEXT:    s_sext_i32_i16 s3, s3
+; GISEL-VI-NEXT:    s_sext_i32_i16 s2, s2
+; GISEL-VI-NEXT:    s_min_i32 s3, s3, s4
+; GISEL-VI-NEXT:    s_min_i32 s2, s2, s4
+; GISEL-VI-NEXT:    s_bfe_u32 s3, s3, 0x100000
+; GISEL-VI-NEXT:    s_bfe_u32 s2, s2, 0x100000
+; GISEL-VI-NEXT:    s_lshl_b32 s3, s3, 16
+; GISEL-VI-NEXT:    s_or_b32 s2, s2, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-VI-NEXT:    flat_store_dword v[0:1], v2
+; GISEL-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX9-LABEL: vec_smax_smin_sgpr:
+; GISEL-GFX9:       ; %bb.0:
+; GISEL-GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX9-NEXT:    s_sext_i32_i16 s0, 0
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX9-NEXT:    s_sext_i32_i16 s1, s4
+; GISEL-GFX9-NEXT:    s_ashr_i32 s4, s4, 16
+; GISEL-GFX9-NEXT:    s_max_i32 s0, s1, s0
+; GISEL-GFX9-NEXT:    s_max_i32 s1, s4, 0
+; GISEL-GFX9-NEXT:    s_pack_ll_b32_b16 s0, s0, s1
+; GISEL-GFX9-NEXT:    s_sext_i32_i16 s1, s0
+; GISEL-GFX9-NEXT:    s_ashr_i32 s0, s0, 16
+; GISEL-GFX9-NEXT:    s_sext_i32_i16 s4, 0xff00ff
+; GISEL-GFX9-NEXT:    s_min_i32 s1, s1, s4
+; GISEL-GFX9-NEXT:    s_min_i32 s0, s0, 0xff
+; GISEL-GFX9-NEXT:    s_pack_ll_b32_b16 s0, s1, s0
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX9-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX9-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: vec_smax_smin_sgpr:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    s_sext_i32_i16 s3, 0
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    s_sext_i32_i16 s4, s2
+; GISEL-GFX11-NEXT:    s_ashr_i32 s2, s2, 16
+; GISEL-GFX11-NEXT:    s_max_i32 s3, s4, s3
+; GISEL-GFX11-NEXT:    s_max_i32 s2, s2, 0
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
+; GISEL-GFX11-NEXT:    s_pack_ll_b32_b16 s2, s3, s2
+; GISEL-GFX11-NEXT:    s_sext_i32_i16 s3, 0xff00ff
+; GISEL-GFX11-NEXT:    s_sext_i32_i16 s4, s2
+; GISEL-GFX11-NEXT:    s_ashr_i32 s2, s2, 16
+; GISEL-GFX11-NEXT:    s_min_i32 s3, s4, s3
+; GISEL-GFX11-NEXT:    s_min_i32 s2, s2, 0xff
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GISEL-GFX11-NEXT:    s_pack_ll_b32_b16 s2, s3, s2
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+  %src.max = call <2 x i16> @llvm.smax.v2i16(<2 x i16> %src, <2 x i16> <i16 0, i16 0>)
+  %src.clamp = call <2 x i16> @llvm.smin.v2i16(<2 x i16> %src.max, <2 x i16> <i16 255, i16 255>)
+  store <2 x i16> %src.clamp, ptr addrspace(1) %out
+  ret void
+}
+
+define <2 x i16> @vec_smin_smax(<2 x i16> %src) {
+; SDAG-VI-LABEL: vec_smin_smax:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, 0xff
+; SDAG-VI-NEXT:    v_min_i16_sdwa v1, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; SDAG-VI-NEXT:    v_min_i16_e32 v0, 0xff, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, 0
+; SDAG-VI-NEXT:    v_max_i16_e32 v0, 0, v0
+; SDAG-VI-NEXT:    v_max_i16_sdwa v1, v1, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; SDAG-VI-NEXT:    v_or_b32_e32 v0, v0, v1
+; SDAG-VI-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9-LABEL: vec_smin_smax:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    v_sat_pk_u8_i16_e32 v0, v0
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX11-LABEL: vec_smin_smax:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX11-NEXT:    v_sat_pk_u8_i16_e32 v0, v0
+; GFX11-NEXT:    s_setpc_b64 s[30:31]
+;
+; GISEL-VI-LABEL: vec_smin_smax:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, 0xff
+; GISEL-VI-NEXT:    v_min_i16_e32 v1, 0xff, v0
+; GISEL-VI-NEXT:    v_min_i16_sdwa v0, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-VI-NEXT:    v_max_i16_e32 v1, 0, v1
+; GISEL-VI-NEXT:    v_max_i16_sdwa v0, v0, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GISEL-VI-NEXT:    v_or_b32_e32 v0, v1, v0
+; GISEL-VI-NEXT:    s_setpc_b64 s[30:31]
+  %src.min = call <2 x i16> @llvm.smin.v2i16(<2 x i16> %src, <2 x i16> <i16 255, i16 255>)
+  %src.clamp = call <2 x i16> @llvm.smax.v2i16(<2 x i16> %src.min, <2 x i16> <i16 0, i16 0>)
+  ret <2 x i16> %src.clamp
+}
diff --git a/llvm/test/CodeGen/AMDGPU/verify-image-partial-nsa.mir b/llvm/test/CodeGen/AMDGPU/verify-image.mir
similarity index 79%
rename from llvm/test/CodeGen/AMDGPU/verify-image-partial-nsa.mir
rename to llvm/test/CodeGen/AMDGPU/verify-image.mir
index df275390349a6..62de9ab05918c 100644
--- a/llvm/test/CodeGen/AMDGPU/verify-image-partial-nsa.mir
+++ b/llvm/test/CodeGen/AMDGPU/verify-image.mir
@@ -1,7 +1,7 @@
-# RUN: not --crash llc -march=amdgcn -mcpu=gfx11 -run-pass=machineverifier -o /dev/null %s 2>&1 | FileCheck -check-prefix=GFX11-ERR %s
+# RUN: not --crash llc -march=amdgcn -mcpu=gfx1100 -run-pass=machineverifier -o /dev/null %s 2>&1 | FileCheck -check-prefix=GFX11-ERR %s
 
 ---
-name: image_sample_d_v1v9_nsa_partial
+name: image_verify
 body:             |
   bb.0:
     liveins: $sgpr0_sgpr1_sgpr2_sgpr3_sgpr4_sgpr5_sgpr6_sgpr7, $sgpr8_sgpr9_sgpr10_sgpr11, $vgpr0, $vgpr1, $vgpr2, $vgpr3, $vgpr4, $vgpr5, $vgpr6, $vgpr7, $vgpr8, $vgpr9
@@ -23,6 +23,11 @@ body:             |
     ; GFX11-ERR: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9 is not a VReg_160 register.
 
     renamable $vgpr11 = IMAGE_SAMPLE_D_V1_V9_nsa_gfx11 renamable $vgpr1, renamable $vgpr0, renamable $vgpr2, renamable $vgpr3, renamable $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9, renamable $sgpr0_sgpr1_sgpr2_sgpr3_sgpr4_sgpr5_sgpr6_sgpr7, renamable $sgpr8_sgpr9_sgpr10_sgpr11, 1, 2, 0, 0, 0, 0, 0, 0, 0, implicit $exec :: (dereferenceable load (s128), addrspace 7)
+
+    ; GFX11-ERR: *** Bad machine code: Image instruction returns too many registers for dst register class ***
+    ; GFX11-ERR: - instruction: renamable $vgpr12 = IMAGE_SAMPLE_V1_V1_gfx11 renamable $vgpr0, renamable $sgpr0_sgpr1_sgpr2_sgpr3_sgpr4_sgpr5_sgpr6_sgpr7, renamable $sgpr8_sgpr9_sgpr10_sgpr11, 7, 0, 0, 0, 0, 0, 0, 0, -1, implicit $exec :: (dereferenceable load (s128), addrspace 7)
+
+    renamable $vgpr12 = IMAGE_SAMPLE_V1_V1_gfx11 renamable $vgpr0, renamable $sgpr0_sgpr1_sgpr2_sgpr3_sgpr4_sgpr5_sgpr6_sgpr7, renamable $sgpr8_sgpr9_sgpr10_sgpr11, 7, 0, 0, 0, 0, 0, 0, 0, -1, implicit $exec :: (dereferenceable load (s128), addrspace 7)
 ...
 
 # GFX11-ERR-NOT: *** Bad machine code
diff --git a/llvm/test/CodeGen/AMDGPU/wave32.ll b/llvm/test/CodeGen/AMDGPU/wave32.ll
index fe968a380f0d6..9d06212894a2a 100644
--- a/llvm/test/CodeGen/AMDGPU/wave32.ll
+++ b/llvm/test/CodeGen/AMDGPU/wave32.ll
@@ -1,15 +1,34 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
 ; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -verify-machineinstrs -simplifycfg-require-and-preserve-domtree=1 < %s | FileCheck -check-prefixes=GCN,GFX1032 %s
 ; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs -simplifycfg-require-and-preserve-domtree=1 < %s | FileCheck -check-prefixes=GCN,GFX1064 %s
 ; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -amdgpu-early-ifcvt=1 -verify-machineinstrs -simplifycfg-require-and-preserve-domtree=1 < %s | FileCheck -check-prefixes=GCN,GFX1032 %s
 ; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -amdgpu-early-ifcvt=1 -verify-machineinstrs -simplifycfg-require-and-preserve-domtree=1 < %s | FileCheck -check-prefixes=GCN,GFX1064 %s
 ; RUN: llc -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs -simplifycfg-require-and-preserve-domtree=1 < %s | FileCheck -check-prefixes=GCN,GFX1032,GFX10DEFWAVE %s
 
-; GCN-LABEL: {{^}}test_vopc_i32:
-; GFX1032: v_cmp_lt_i32_e32 vcc_lo, 0, v{{[0-9]+}}
-; GFX1032: v_cndmask_b32_e64 v{{[0-9]+}}, 2, 1, vcc_lo
-; GFX1064: v_cmp_lt_i32_e32 vcc, 0, v{{[0-9]+}}
-; GFX1064: v_cndmask_b32_e64 v{{[0-9]+}}, 2, 1, vcc{{$}}
 define amdgpu_kernel void @test_vopc_i32(ptr addrspace(1) %arg) {
+; GFX1032-LABEL: test_vopc_i32:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GFX1032-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    global_load_dword v1, v0, s[0:1]
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_cmp_lt_i32_e32 vcc_lo, 0, v1
+; GFX1032-NEXT:    v_cndmask_b32_e64 v1, 2, 1, vcc_lo
+; GFX1032-NEXT:    global_store_dword v0, v1, s[0:1]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_vopc_i32:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GFX1064-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    global_load_dword v1, v0, s[0:1]
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_cmp_lt_i32_e32 vcc, 0, v1
+; GFX1064-NEXT:    v_cndmask_b32_e64 v1, 2, 1, vcc
+; GFX1064-NEXT:    global_store_dword v0, v1, s[0:1]
+; GFX1064-NEXT:    s_endpgm
   %lid = tail call i32 @llvm.amdgcn.workitem.id.x()
   %gep = getelementptr inbounds i32, ptr addrspace(1) %arg, i32 %lid
   %load = load i32, ptr addrspace(1) %gep, align 4
@@ -19,12 +38,30 @@ define amdgpu_kernel void @test_vopc_i32(ptr addrspace(1) %arg) {
   ret void
 }
 
-; GCN-LABEL: {{^}}test_vopc_f32:
-; GFX1032: v_cmp_nge_f32_e32 vcc_lo, 0, v{{[0-9]+}}
-; GFX1032: v_cndmask_b32_e64 v{{[0-9]+}}, 2.0, 1.0, vcc_lo
-; GFX1064: v_cmp_nge_f32_e32 vcc, 0, v{{[0-9]+}}
-; GFX1064: v_cndmask_b32_e64 v{{[0-9]+}}, 2.0, 1.0, vcc{{$}}
 define amdgpu_kernel void @test_vopc_f32(ptr addrspace(1) %arg) {
+; GFX1032-LABEL: test_vopc_f32:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GFX1032-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    global_load_dword v1, v0, s[0:1]
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_cmp_nge_f32_e32 vcc_lo, 0, v1
+; GFX1032-NEXT:    v_cndmask_b32_e64 v1, 2.0, 1.0, vcc_lo
+; GFX1032-NEXT:    global_store_dword v0, v1, s[0:1]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_vopc_f32:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GFX1064-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    global_load_dword v1, v0, s[0:1]
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_cmp_nge_f32_e32 vcc, 0, v1
+; GFX1064-NEXT:    v_cndmask_b32_e64 v1, 2.0, 1.0, vcc
+; GFX1064-NEXT:    global_store_dword v0, v1, s[0:1]
+; GFX1064-NEXT:    s_endpgm
   %lid = tail call i32 @llvm.amdgcn.workitem.id.x()
   %gep = getelementptr inbounds float, ptr addrspace(1) %arg, i32 %lid
   %load = load float, ptr addrspace(1) %gep, align 4
@@ -34,21 +71,59 @@ define amdgpu_kernel void @test_vopc_f32(ptr addrspace(1) %arg) {
   ret void
 }
 
-; GCN-LABEL: {{^}}test_vopc_vcmp:
-; GFX1032: v_cmp_nle_f32_e32 vcc_lo, 0, v{{[0-9]+}}
-; GFX1064: v_cmp_nle_f32_e32 vcc, 0, v{{[0-9]+}}
 define amdgpu_ps void @test_vopc_vcmp(float %x) {
+; GFX1032-LABEL: test_vopc_vcmp:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    v_cmp_nle_f32_e32 vcc_lo, 0, v0
+; GFX1032-NEXT:    s_andn2_b32 exec_lo, exec_lo, vcc_lo
+; GFX1032-NEXT:    s_cbranch_scc0 .LBB2_1
+; GFX1032-NEXT:    s_endpgm
+; GFX1032-NEXT:  .LBB2_1:
+; GFX1032-NEXT:    s_mov_b32 exec_lo, 0
+; GFX1032-NEXT:    exp null off, off, off, off done vm
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_vopc_vcmp:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    v_cmp_nle_f32_e32 vcc, 0, v0
+; GFX1064-NEXT:    s_andn2_b64 exec, exec, vcc
+; GFX1064-NEXT:    s_cbranch_scc0 .LBB2_1
+; GFX1064-NEXT:    s_endpgm
+; GFX1064-NEXT:  .LBB2_1:
+; GFX1064-NEXT:    s_mov_b64 exec, 0
+; GFX1064-NEXT:    exp null off, off, off, off done vm
+; GFX1064-NEXT:    s_endpgm
   %cmp = fcmp oge float %x, 0.0
   call void @llvm.amdgcn.kill(i1 %cmp)
   ret void
 }
 
-; GCN-LABEL: {{^}}test_vopc_2xf16:
-; GFX1032: v_cmp_le_f16_sdwa [[SC:vcc_lo|s[0-9]+]], {{[vs][0-9]+}}, v{{[0-9]+}} src0_sel:WORD_1 src1_sel:DWORD
-; GFX1032: v_cndmask_b32_e32 v{{[0-9]+}}, 0x3c003c00, v{{[0-9]+}}, [[SC]]
-; GFX1064: v_cmp_le_f16_sdwa [[SC:vcc|s\[[0-9:]+\]]], {{[vs][0-9]+}}, v{{[0-9]+}} src0_sel:WORD_1 src1_sel:DWORD
-; GFX1064: v_cndmask_b32_e32 v{{[0-9]+}}, 0x3c003c00, v{{[0-9]+}}, [[SC]]
 define amdgpu_kernel void @test_vopc_2xf16(ptr addrspace(1) %arg) {
+; GFX1032-LABEL: test_vopc_2xf16:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GFX1032-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX1032-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    global_load_dword v1, v0, s[0:1]
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_cmp_le_f16_sdwa vcc_lo, v1, v2 src0_sel:WORD_1 src1_sel:DWORD
+; GFX1032-NEXT:    v_cndmask_b32_e32 v1, 0x3c003c00, v1, vcc_lo
+; GFX1032-NEXT:    global_store_dword v0, v1, s[0:1]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_vopc_2xf16:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GFX1064-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX1064-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    global_load_dword v1, v0, s[0:1]
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_cmp_le_f16_sdwa vcc, v1, v2 src0_sel:WORD_1 src1_sel:DWORD
+; GFX1064-NEXT:    v_cndmask_b32_e32 v1, 0x3c003c00, v1, vcc
+; GFX1064-NEXT:    global_store_dword v0, v1, s[0:1]
+; GFX1064-NEXT:    s_endpgm
   %lid = tail call i32 @llvm.amdgcn.workitem.id.x()
   %gep = getelementptr inbounds <2 x half>, ptr addrspace(1) %arg, i32 %lid
   %load = load <2 x half>, ptr addrspace(1) %gep, align 4
@@ -59,12 +134,30 @@ define amdgpu_kernel void @test_vopc_2xf16(ptr addrspace(1) %arg) {
   ret void
 }
 
-; GCN-LABEL: {{^}}test_vopc_class:
-; GFX1032: v_cmp_class_f32_e64 [[C:vcc_lo|s[0-9:]+]], s{{[0-9]+}}, 0x204
-; GFX1032: v_cndmask_b32_e64 v{{[0-9]+}}, 0, 1, [[C]]
-; GFX1064: v_cmp_class_f32_e64 [[C:vcc|s\[[0-9:]+\]]], s{{[0-9]+}}, 0x204
-; GFX1064: v_cndmask_b32_e64 v{{[0-9]+}}, 0, 1, [[C]]{{$}}
 define amdgpu_kernel void @test_vopc_class(ptr addrspace(1) %out, float %x) #0 {
+; GFX1032-LABEL: test_vopc_class:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_clause 0x1
+; GFX1032-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX1032-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_cmp_class_f32_e64 s0, s4, 0x204
+; GFX1032-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s0
+; GFX1032-NEXT:    global_store_dword v0, v1, s[2:3]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_vopc_class:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_clause 0x1
+; GFX1064-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX1064-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_cmp_class_f32_e64 s[0:1], s4, 0x204
+; GFX1064-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s[0:1]
+; GFX1064-NEXT:    global_store_dword v0, v1, s[2:3]
+; GFX1064-NEXT:    s_endpgm
   %fabs = tail call float @llvm.fabs.f32(float %x)
   %cmp = fcmp oeq float %fabs, 0x7FF0000000000000
   %ext = zext i1 %cmp to i32
@@ -72,29 +165,66 @@ define amdgpu_kernel void @test_vopc_class(ptr addrspace(1) %out, float %x) #0 {
   ret void
 }
 
-; GCN-LABEL: {{^}}test_vcmp_vcnd_f16:
-; GFX1032: v_cmp_neq_f16_e64 [[C:vcc_lo|s\[[0-9:]+\]]], 0x7c00, s{{[0-9]+}}
-; GFX1032: v_cndmask_b32_e32 v{{[0-9]+}}, 0x3c00, v{{[0-9]+}}, [[C]]
-
-; GFX1064: v_cmp_neq_f16_e64 [[C:vcc|s\[[0-9:]+\]]], 0x7c00, s{{[0-9]+}}
-; GFX1064: v_cndmask_b32_e32 v{{[0-9]+}}, 0x3c00, v{{[0-9]+}}, [[C]]{{$}}
 define amdgpu_kernel void @test_vcmp_vcnd_f16(ptr addrspace(1) %out, half %x) #0 {
+; GFX1032-LABEL: test_vcmp_vcnd_f16:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_clause 0x1
+; GFX1032-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX1032-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1032-NEXT:    v_mov_b32_e32 v1, 0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_mov_b32_e32 v0, s4
+; GFX1032-NEXT:    v_cmp_neq_f16_e64 vcc_lo, 0x7c00, s4
+; GFX1032-NEXT:    v_cndmask_b32_e32 v0, 0x3c00, v0, vcc_lo
+; GFX1032-NEXT:    global_store_short v1, v0, s[2:3]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_vcmp_vcnd_f16:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_clause 0x1
+; GFX1064-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX1064-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1064-NEXT:    v_mov_b32_e32 v1, 0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_mov_b32_e32 v0, s4
+; GFX1064-NEXT:    v_cmp_neq_f16_e64 vcc, 0x7c00, s4
+; GFX1064-NEXT:    v_cndmask_b32_e32 v0, 0x3c00, v0, vcc
+; GFX1064-NEXT:    global_store_short v1, v0, s[2:3]
+; GFX1064-NEXT:    s_endpgm
   %cmp = fcmp oeq half %x, 0x7FF0000000000000
   %sel = select i1 %cmp, half 1.0, half %x
   store half %sel, ptr addrspace(1) %out, align 2
   ret void
 }
 
-; GCN-LABEL: {{^}}test_vop3_cmp_f32_sop_and:
-; GFX1032: v_cmp_nge_f32_e32 vcc_lo, 0, v{{[0-9]+}}
-; GFX1032: v_cmp_nle_f32_e64 [[C2:s[0-9]+]], 1.0, v{{[0-9]+}}
-; GFX1032: s_and_b32 [[AND:s[0-9]+]], vcc_lo, [[C2]]
-; GFX1032: v_cndmask_b32_e64 v{{[0-9]+}}, 2.0, 1.0, [[AND]]
-; GFX1064: v_cmp_nge_f32_e32 vcc, 0, v{{[0-9]+}}
-; GFX1064: v_cmp_nle_f32_e64 [[C2:s\[[0-9:]+\]]], 1.0, v{{[0-9]+}}
-; GFX1064: s_and_b64 [[AND:s\[[0-9:]+\]]], vcc, [[C2]]
-; GFX1064: v_cndmask_b32_e64 v{{[0-9]+}}, 2.0, 1.0, [[AND]]
 define amdgpu_kernel void @test_vop3_cmp_f32_sop_and(ptr addrspace(1) %arg) {
+; GFX1032-LABEL: test_vop3_cmp_f32_sop_and:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1032-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    global_load_dword v1, v0, s[2:3]
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_cmp_nge_f32_e32 vcc_lo, 0, v1
+; GFX1032-NEXT:    v_cmp_nle_f32_e64 s0, 1.0, v1
+; GFX1032-NEXT:    s_and_b32 s0, vcc_lo, s0
+; GFX1032-NEXT:    v_cndmask_b32_e64 v1, 2.0, 1.0, s0
+; GFX1032-NEXT:    global_store_dword v0, v1, s[2:3]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_vop3_cmp_f32_sop_and:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1064-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    global_load_dword v1, v0, s[2:3]
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_cmp_nge_f32_e32 vcc, 0, v1
+; GFX1064-NEXT:    v_cmp_nle_f32_e64 s[0:1], 1.0, v1
+; GFX1064-NEXT:    s_and_b64 s[0:1], vcc, s[0:1]
+; GFX1064-NEXT:    v_cndmask_b32_e64 v1, 2.0, 1.0, s[0:1]
+; GFX1064-NEXT:    global_store_dword v0, v1, s[2:3]
+; GFX1064-NEXT:    s_endpgm
   %lid = tail call i32 @llvm.amdgcn.workitem.id.x()
   %gep = getelementptr inbounds float, ptr addrspace(1) %arg, i32 %lid
   %load = load float, ptr addrspace(1) %gep, align 4
@@ -106,16 +236,34 @@ define amdgpu_kernel void @test_vop3_cmp_f32_sop_and(ptr addrspace(1) %arg) {
   ret void
 }
 
-; GCN-LABEL: {{^}}test_vop3_cmp_i32_sop_xor:
-; GFX1032: v_cmp_lt_i32_e32 vcc_lo, 0, v{{[0-9]+}}
-; GFX1032: v_cmp_gt_i32_e64 [[C2:s[0-9]+]], 1, v{{[0-9]+}}
-; GFX1032: s_xor_b32 [[AND:s[0-9]+]], vcc_lo, [[C2]]
-; GFX1032: v_cndmask_b32_e64 v{{[0-9]+}}, 2, 1, [[AND]]
-; GFX1064: v_cmp_lt_i32_e32 vcc, 0, v{{[0-9]+}}
-; GFX1064: v_cmp_gt_i32_e64 [[C2:s\[[0-9:]+\]]], 1, v{{[0-9]+}}
-; GFX1064: s_xor_b64 [[AND:s\[[0-9:]+\]]], vcc, [[C2]]
-; GFX1064: v_cndmask_b32_e64 v{{[0-9]+}}, 2, 1, [[AND]]
 define amdgpu_kernel void @test_vop3_cmp_i32_sop_xor(ptr addrspace(1) %arg) {
+; GFX1032-LABEL: test_vop3_cmp_i32_sop_xor:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1032-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    global_load_dword v1, v0, s[2:3]
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_cmp_lt_i32_e32 vcc_lo, 0, v1
+; GFX1032-NEXT:    v_cmp_gt_i32_e64 s0, 1, v1
+; GFX1032-NEXT:    s_xor_b32 s0, vcc_lo, s0
+; GFX1032-NEXT:    v_cndmask_b32_e64 v1, 2, 1, s0
+; GFX1032-NEXT:    global_store_dword v0, v1, s[2:3]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_vop3_cmp_i32_sop_xor:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1064-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    global_load_dword v1, v0, s[2:3]
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_cmp_lt_i32_e32 vcc, 0, v1
+; GFX1064-NEXT:    v_cmp_gt_i32_e64 s[0:1], 1, v1
+; GFX1064-NEXT:    s_xor_b64 s[0:1], vcc, s[0:1]
+; GFX1064-NEXT:    v_cndmask_b32_e64 v1, 2, 1, s[0:1]
+; GFX1064-NEXT:    global_store_dword v0, v1, s[2:3]
+; GFX1064-NEXT:    s_endpgm
   %lid = tail call i32 @llvm.amdgcn.workitem.id.x()
   %gep = getelementptr inbounds i32, ptr addrspace(1) %arg, i32 %lid
   %load = load i32, ptr addrspace(1) %gep, align 4
@@ -127,16 +275,34 @@ define amdgpu_kernel void @test_vop3_cmp_i32_sop_xor(ptr addrspace(1) %arg) {
   ret void
 }
 
-; GCN-LABEL: {{^}}test_vop3_cmp_u32_sop_or:
-; GFX1032: v_cmp_lt_u32_e32 vcc_lo, 3, v{{[0-9]+}}
-; GFX1032: v_cmp_gt_u32_e64 [[C2:s[0-9]+]], 2, v{{[0-9]+}}
-; GFX1032: s_or_b32 [[AND:s[0-9]+]], vcc_lo, [[C2]]
-; GFX1032: v_cndmask_b32_e64 v{{[0-9]+}}, 2, 1, [[AND]]
-; GFX1064: v_cmp_lt_u32_e32 vcc, 3, v{{[0-9]+}}
-; GFX1064: v_cmp_gt_u32_e64 [[C2:s\[[0-9:]+\]]], 2, v{{[0-9]+}}
-; GFX1064: s_or_b64 [[AND:s\[[0-9:]+\]]], vcc, [[C2]]
-; GFX1064: v_cndmask_b32_e64 v{{[0-9]+}}, 2, 1, [[AND]]
 define amdgpu_kernel void @test_vop3_cmp_u32_sop_or(ptr addrspace(1) %arg) {
+; GFX1032-LABEL: test_vop3_cmp_u32_sop_or:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1032-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    global_load_dword v1, v0, s[2:3]
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_cmp_lt_u32_e32 vcc_lo, 3, v1
+; GFX1032-NEXT:    v_cmp_gt_u32_e64 s0, 2, v1
+; GFX1032-NEXT:    s_or_b32 s0, vcc_lo, s0
+; GFX1032-NEXT:    v_cndmask_b32_e64 v1, 2, 1, s0
+; GFX1032-NEXT:    global_store_dword v0, v1, s[2:3]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_vop3_cmp_u32_sop_or:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1064-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    global_load_dword v1, v0, s[2:3]
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_cmp_lt_u32_e32 vcc, 3, v1
+; GFX1064-NEXT:    v_cmp_gt_u32_e64 s[0:1], 2, v1
+; GFX1064-NEXT:    s_or_b64 s[0:1], vcc, s[0:1]
+; GFX1064-NEXT:    v_cndmask_b32_e64 v1, 2, 1, s[0:1]
+; GFX1064-NEXT:    global_store_dword v0, v1, s[2:3]
+; GFX1064-NEXT:    s_endpgm
   %lid = tail call i32 @llvm.amdgcn.workitem.id.x()
   %gep = getelementptr inbounds i32, ptr addrspace(1) %arg, i32 %lid
   %load = load i32, ptr addrspace(1) %gep, align 4
@@ -148,11 +314,32 @@ define amdgpu_kernel void @test_vop3_cmp_u32_sop_or(ptr addrspace(1) %arg) {
   ret void
 }
 
-; GCN-LABEL: {{^}}test_mask_if:
-; GFX1032: s_and_saveexec_b32 s{{[0-9]+}}, vcc_lo
-; GFX1064: s_and_saveexec_b64 s[{{[0-9:]+}}], vcc{{$}}
-; GCN: s_cbranch_execz
 define amdgpu_kernel void @test_mask_if(ptr addrspace(1) %arg) #0 {
+; GFX1032-LABEL: test_mask_if:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    v_cmp_lt_u32_e32 vcc_lo, 10, v0
+; GFX1032-NEXT:    s_and_saveexec_b32 s2, vcc_lo
+; GFX1032-NEXT:    s_cbranch_execz .LBB9_2
+; GFX1032-NEXT:  ; %bb.1: ; %if
+; GFX1032-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    global_store_dword v0, v0, s[0:1]
+; GFX1032-NEXT:  .LBB9_2: ; %endif
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_mask_if:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    v_cmp_lt_u32_e32 vcc, 10, v0
+; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], vcc
+; GFX1064-NEXT:    s_cbranch_execz .LBB9_2
+; GFX1064-NEXT:  ; %bb.1: ; %if
+; GFX1064-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    global_store_dword v0, v0, s[0:1]
+; GFX1064-NEXT:  .LBB9_2: ; %endif
+; GFX1064-NEXT:    s_endpgm
   %lid = tail call i32 @llvm.amdgcn.workitem.id.x()
   %cmp = icmp ugt i32 %lid, 10
   br i1 %cmp, label %if, label %endif
@@ -165,31 +352,130 @@ endif:
   ret void
 }
 
-; GCN-LABEL: {{^}}test_loop_with_if:
-; GFX1032: s_or_b32 s{{[0-9]+}}, vcc_lo, s{{[0-9]+}}
-; GFX1032: s_andn2_b32 exec_lo, exec_lo, s{{[0-9]+}}
-; GFX1064: s_or_b64 s[{{[0-9:]+}}], vcc, s[{{[0-9:]+}}]
-; GFX1064: s_andn2_b64 exec, exec, s[{{[0-9:]+}}]
-; GCN:     s_cbranch_execz
-; GCN:   .LBB{{.*}}:
-; GFX1032: s_and_saveexec_b32 s{{[0-9]+}}, vcc_lo
-; GFX1064: s_and_saveexec_b64 s[{{[0-9:]+}}], vcc{{$}}
-; GCN:     s_cbranch_execz
-; GCN:   ; %bb.{{[0-9]+}}:
-; GCN:   .LBB{{.*}}:
-; GFX1032: s_xor_b32 s{{[0-9]+}}, exec_lo, s{{[0-9]+}}
-; GFX1064: s_xor_b64 s[{{[0-9:]+}}], exec, s[{{[0-9:]+}}]
-; GCN:   ; %bb.{{[0-9]+}}:
-; GCN:   ; %bb.{{[0-9]+}}:
-; GFX1032: s_or_b32 exec_lo, exec_lo, s{{[0-9]+}}
-; GFX1032: s_and_saveexec_b32 s{{[0-9]+}}, s{{[0-9]+}}
-; GFX1064: s_or_b64 exec, exec, s[{{[0-9:]+}}]
-; GFX1064: s_and_saveexec_b64 s[{{[0-9:]+}}], s[{{[0-9:]+}}]{{$}}
-; GCN:     s_cbranch_execz .LBB
-; GCN:   ; %bb.{{[0-9]+}}:
-; GCN:   .LBB{{.*}}:
-; GCN:     s_endpgm
 define amdgpu_kernel void @test_loop_with_if(ptr addrspace(1) %arg) #0 {
+; GFX1032-LABEL: test_loop_with_if:
+; GFX1032:       ; %bb.0: ; %bb
+; GFX1032-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GFX1032-NEXT:    v_mov_b32_e32 v1, 0
+; GFX1032-NEXT:    s_mov_b32 s2, 0
+; GFX1032-NEXT:    ; implicit-def: $vgpr2_vgpr3
+; GFX1032-NEXT:    s_branch .LBB10_2
+; GFX1032-NEXT:  .LBB10_1: ; %bb13
+; GFX1032-NEXT:    ; in Loop: Header=BB10_2 Depth=1
+; GFX1032-NEXT:    s_waitcnt_depctr 0xffe3
+; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s4
+; GFX1032-NEXT:    v_cmp_lt_i32_e32 vcc_lo, 0xfe, v4
+; GFX1032-NEXT:    v_add_nc_u32_e32 v1, 1, v4
+; GFX1032-NEXT:    s_or_b32 s2, vcc_lo, s2
+; GFX1032-NEXT:    s_andn2_b32 exec_lo, exec_lo, s2
+; GFX1032-NEXT:    s_cbranch_execz .LBB10_8
+; GFX1032-NEXT:  .LBB10_2: ; %bb2
+; GFX1032-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1032-NEXT:    v_cmp_ge_i32_e64 s4, v1, v0
+; GFX1032-NEXT:    v_cmp_lt_i32_e32 vcc_lo, v1, v0
+; GFX1032-NEXT:    s_mov_b32 s3, 0
+; GFX1032-NEXT:    s_and_saveexec_b32 s5, vcc_lo
+; GFX1032-NEXT:    s_cbranch_execz .LBB10_4
+; GFX1032-NEXT:  ; %bb.3: ; %bb5
+; GFX1032-NEXT:    ; in Loop: Header=BB10_2 Depth=1
+; GFX1032-NEXT:    v_ashrrev_i32_e32 v2, 31, v1
+; GFX1032-NEXT:    s_andn2_b32 s4, s4, exec_lo
+; GFX1032-NEXT:    s_mov_b32 s3, exec_lo
+; GFX1032-NEXT:    v_lshlrev_b64 v[2:3], 2, v[1:2]
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_add_co_u32 v2, vcc_lo, s0, v2
+; GFX1032-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, s1, v3, vcc_lo
+; GFX1032-NEXT:    global_load_dword v4, v[2:3], off
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_cmp_gt_i32_e32 vcc_lo, 11, v4
+; GFX1032-NEXT:    s_and_b32 s6, vcc_lo, exec_lo
+; GFX1032-NEXT:    s_or_b32 s4, s4, s6
+; GFX1032-NEXT:  .LBB10_4: ; %Flow
+; GFX1032-NEXT:    ; in Loop: Header=BB10_2 Depth=1
+; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s5
+; GFX1032-NEXT:    ; implicit-def: $vgpr4
+; GFX1032-NEXT:    s_and_saveexec_b32 s5, s4
+; GFX1032-NEXT:    s_xor_b32 s4, exec_lo, s5
+; GFX1032-NEXT:  ; %bb.5: ; %bb11
+; GFX1032-NEXT:    ; in Loop: Header=BB10_2 Depth=1
+; GFX1032-NEXT:    v_lshrrev_b32_e32 v4, 31, v1
+; GFX1032-NEXT:    s_andn2_b32 s3, s3, exec_lo
+; GFX1032-NEXT:    v_add_nc_u32_e32 v4, v1, v4
+; GFX1032-NEXT:    v_ashrrev_i32_e32 v4, 1, v4
+; GFX1032-NEXT:  ; %bb.6: ; %Flow1
+; GFX1032-NEXT:    ; in Loop: Header=BB10_2 Depth=1
+; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s4
+; GFX1032-NEXT:    s_and_saveexec_b32 s4, s3
+; GFX1032-NEXT:    s_cbranch_execz .LBB10_1
+; GFX1032-NEXT:  ; %bb.7: ; %bb10
+; GFX1032-NEXT:    ; in Loop: Header=BB10_2 Depth=1
+; GFX1032-NEXT:    v_mov_b32_e32 v4, v1
+; GFX1032-NEXT:    global_store_dword v[2:3], v0, off
+; GFX1032-NEXT:    s_branch .LBB10_1
+; GFX1032-NEXT:  .LBB10_8: ; %bb1
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_loop_with_if:
+; GFX1064:       ; %bb.0: ; %bb
+; GFX1064-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GFX1064-NEXT:    v_mov_b32_e32 v1, 0
+; GFX1064-NEXT:    s_mov_b64 s[2:3], 0
+; GFX1064-NEXT:    ; implicit-def: $vgpr2_vgpr3
+; GFX1064-NEXT:    s_branch .LBB10_2
+; GFX1064-NEXT:  .LBB10_1: ; %bb13
+; GFX1064-NEXT:    ; in Loop: Header=BB10_2 Depth=1
+; GFX1064-NEXT:    s_waitcnt_depctr 0xffe3
+; GFX1064-NEXT:    s_or_b64 exec, exec, s[6:7]
+; GFX1064-NEXT:    v_cmp_lt_i32_e32 vcc, 0xfe, v4
+; GFX1064-NEXT:    v_add_nc_u32_e32 v1, 1, v4
+; GFX1064-NEXT:    s_or_b64 s[2:3], vcc, s[2:3]
+; GFX1064-NEXT:    s_andn2_b64 exec, exec, s[2:3]
+; GFX1064-NEXT:    s_cbranch_execz .LBB10_8
+; GFX1064-NEXT:  .LBB10_2: ; %bb2
+; GFX1064-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1064-NEXT:    v_cmp_ge_i32_e64 s[6:7], v1, v0
+; GFX1064-NEXT:    v_cmp_lt_i32_e32 vcc, v1, v0
+; GFX1064-NEXT:    s_mov_b64 s[4:5], 0
+; GFX1064-NEXT:    s_and_saveexec_b64 s[8:9], vcc
+; GFX1064-NEXT:    s_cbranch_execz .LBB10_4
+; GFX1064-NEXT:  ; %bb.3: ; %bb5
+; GFX1064-NEXT:    ; in Loop: Header=BB10_2 Depth=1
+; GFX1064-NEXT:    v_ashrrev_i32_e32 v2, 31, v1
+; GFX1064-NEXT:    s_andn2_b64 s[6:7], s[6:7], exec
+; GFX1064-NEXT:    s_mov_b64 s[4:5], exec
+; GFX1064-NEXT:    v_lshlrev_b64 v[2:3], 2, v[1:2]
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_add_co_u32 v2, vcc, s0, v2
+; GFX1064-NEXT:    v_add_co_ci_u32_e32 v3, vcc, s1, v3, vcc
+; GFX1064-NEXT:    global_load_dword v4, v[2:3], off
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_cmp_gt_i32_e32 vcc, 11, v4
+; GFX1064-NEXT:    s_and_b64 s[10:11], vcc, exec
+; GFX1064-NEXT:    s_or_b64 s[6:7], s[6:7], s[10:11]
+; GFX1064-NEXT:  .LBB10_4: ; %Flow
+; GFX1064-NEXT:    ; in Loop: Header=BB10_2 Depth=1
+; GFX1064-NEXT:    s_or_b64 exec, exec, s[8:9]
+; GFX1064-NEXT:    ; implicit-def: $vgpr4
+; GFX1064-NEXT:    s_and_saveexec_b64 s[8:9], s[6:7]
+; GFX1064-NEXT:    s_xor_b64 s[6:7], exec, s[8:9]
+; GFX1064-NEXT:  ; %bb.5: ; %bb11
+; GFX1064-NEXT:    ; in Loop: Header=BB10_2 Depth=1
+; GFX1064-NEXT:    v_lshrrev_b32_e32 v4, 31, v1
+; GFX1064-NEXT:    s_andn2_b64 s[4:5], s[4:5], exec
+; GFX1064-NEXT:    v_add_nc_u32_e32 v4, v1, v4
+; GFX1064-NEXT:    v_ashrrev_i32_e32 v4, 1, v4
+; GFX1064-NEXT:  ; %bb.6: ; %Flow1
+; GFX1064-NEXT:    ; in Loop: Header=BB10_2 Depth=1
+; GFX1064-NEXT:    s_or_b64 exec, exec, s[6:7]
+; GFX1064-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
+; GFX1064-NEXT:    s_cbranch_execz .LBB10_1
+; GFX1064-NEXT:  ; %bb.7: ; %bb10
+; GFX1064-NEXT:    ; in Loop: Header=BB10_2 Depth=1
+; GFX1064-NEXT:    v_mov_b32_e32 v4, v1
+; GFX1064-NEXT:    global_store_dword v[2:3], v0, off
+; GFX1064-NEXT:    s_branch .LBB10_1
+; GFX1064-NEXT:  .LBB10_8: ; %bb1
+; GFX1064-NEXT:    s_endpgm
 bb:
   %tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
   br label %bb2
@@ -224,38 +510,96 @@ bb13:
   br i1 %tmp16, label %bb2, label %bb1
 }
 
-; GCN-LABEL: {{^}}test_loop_with_if_else_break:
-; GFX1032: s_and_saveexec_b32 s{{[0-9]+}}, vcc_lo
-; GFX1064: s_and_saveexec_b64 s[{{[0-9:]+}}], vcc{{$}}
-; GCN:     s_cbranch_execz
-; GCN:   ; %bb.{{[0-9]+}}: ; %.preheader
-; GCN:   .LBB{{.*}}:
-
-; GCN:     global_store_dword
-; GFX1032: s_or_b32 [[MASK0:s[0-9]+]], [[MASK0]], vcc_lo
-; GFX1064: s_or_b64 [[MASK0:s\[[0-9:]+\]]], [[MASK0]], vcc
-; GFX1032: s_andn2_b32 [[MASK1:s[0-9]+]], [[MASK1]], exec_lo
-; GFX1064: s_andn2_b64 [[MASK1:s\[[0-9:]+\]]], [[MASK1]], exec
-; GFX1032: s_and_b32 [[MASK0]], [[MASK0]], exec_lo
-; GFX1064: s_and_b64 [[MASK0]], [[MASK0]], exec
-; GFX1032: s_or_b32 [[MASK1]], [[MASK1]], [[MASK0]]
-; GFX1064: s_or_b64 [[MASK1]], [[MASK1]], [[MASK0]]
-; GCN:   .LBB{{.*}}: ; %Flow
-; GFX1032: s_and_b32 [[TMP0:s[0-9]+]], exec_lo, [[MASK1]]
-; GFX1064: s_and_b64 [[TMP0:s\[[0-9:]+\]]], exec, [[MASK1]]
-; GFX1032: s_or_b32  [[ACC:s[0-9]+]], [[TMP0]], [[ACC]]
-; GFX1064: s_or_b64  [[ACC:s\[[0-9:]+\]]], [[TMP0]], [[ACC]]
-; GFX1032: s_andn2_b32 exec_lo, exec_lo, [[ACC]]
-; GFX1064: s_andn2_b64 exec, exec, [[ACC]]
-; GCN:     s_cbranch_execz
-; GCN:   .LBB{{.*}}:
-
-; GFX1032-DAG: s_or_b32 [[MASK1]], [[MASK1]], exec_lo
-; GFX1064-DAG: s_or_b64 [[MASK1]], [[MASK1]], exec
-; GCN-DAG: global_load_dword [[LOAD:v[0-9]+]]
-; GFX1032: v_cmp_gt_i32_e32 vcc_lo, 11, [[LOAD]]
-; GFX1064: v_cmp_gt_i32_e32 vcc, 11, [[LOAD]]
+
+
 define amdgpu_kernel void @test_loop_with_if_else_break(ptr addrspace(1) %arg) #0 {
+; GFX1032-LABEL: test_loop_with_if_else_break:
+; GFX1032:       ; %bb.0: ; %bb
+; GFX1032-NEXT:    v_cmp_ne_u32_e32 vcc_lo, 0, v0
+; GFX1032-NEXT:    s_mov_b32 s2, 0
+; GFX1032-NEXT:    s_and_saveexec_b32 s3, vcc_lo
+; GFX1032-NEXT:    s_cbranch_execz .LBB11_6
+; GFX1032-NEXT:  ; %bb.1: ; %.preheader
+; GFX1032-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GFX1032-NEXT:    v_min_u32_e32 v1, 0x100, v0
+; GFX1032-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1032-NEXT:    s_mov_b32 s4, 0
+; GFX1032-NEXT:    ; implicit-def: $sgpr3
+; GFX1032-NEXT:    s_branch .LBB11_4
+; GFX1032-NEXT:  .LBB11_2: ; %bb8
+; GFX1032-NEXT:    ; in Loop: Header=BB11_4 Depth=1
+; GFX1032-NEXT:    s_add_i32 s4, s4, 1
+; GFX1032-NEXT:    global_store_dword v2, v0, s[0:1]
+; GFX1032-NEXT:    v_cmp_ge_u32_e32 vcc_lo, s4, v1
+; GFX1032-NEXT:    s_add_u32 s0, s0, 4
+; GFX1032-NEXT:    s_addc_u32 s1, s1, 0
+; GFX1032-NEXT:    s_andn2_b32 s3, s3, exec_lo
+; GFX1032-NEXT:    s_and_b32 s5, vcc_lo, exec_lo
+; GFX1032-NEXT:    s_or_b32 s3, s3, s5
+; GFX1032-NEXT:  .LBB11_3: ; %Flow
+; GFX1032-NEXT:    ; in Loop: Header=BB11_4 Depth=1
+; GFX1032-NEXT:    s_and_b32 s5, exec_lo, s3
+; GFX1032-NEXT:    s_or_b32 s2, s5, s2
+; GFX1032-NEXT:    s_andn2_b32 exec_lo, exec_lo, s2
+; GFX1032-NEXT:    s_cbranch_execz .LBB11_6
+; GFX1032-NEXT:  .LBB11_4: ; %bb2
+; GFX1032-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    global_load_dword v3, v2, s[0:1]
+; GFX1032-NEXT:    s_or_b32 s3, s3, exec_lo
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_cmp_gt_i32_e32 vcc_lo, 11, v3
+; GFX1032-NEXT:    s_cbranch_vccz .LBB11_2
+; GFX1032-NEXT:  ; %bb.5: ; in Loop: Header=BB11_4 Depth=1
+; GFX1032-NEXT:    ; implicit-def: $sgpr4
+; GFX1032-NEXT:    ; implicit-def: $sgpr0_sgpr1
+; GFX1032-NEXT:    s_branch .LBB11_3
+; GFX1032-NEXT:  .LBB11_6: ; %.loopexit
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_loop_with_if_else_break:
+; GFX1064:       ; %bb.0: ; %bb
+; GFX1064-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v0
+; GFX1064-NEXT:    s_mov_b32 s6, 0
+; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], vcc
+; GFX1064-NEXT:    s_cbranch_execz .LBB11_6
+; GFX1064-NEXT:  ; %bb.1: ; %.preheader
+; GFX1064-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GFX1064-NEXT:    v_min_u32_e32 v1, 0x100, v0
+; GFX1064-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1064-NEXT:    s_mov_b64 s[2:3], 0
+; GFX1064-NEXT:    ; implicit-def: $sgpr4_sgpr5
+; GFX1064-NEXT:    s_branch .LBB11_4
+; GFX1064-NEXT:  .LBB11_2: ; %bb8
+; GFX1064-NEXT:    ; in Loop: Header=BB11_4 Depth=1
+; GFX1064-NEXT:    s_add_i32 s6, s6, 1
+; GFX1064-NEXT:    global_store_dword v2, v0, s[0:1]
+; GFX1064-NEXT:    v_cmp_ge_u32_e32 vcc, s6, v1
+; GFX1064-NEXT:    s_add_u32 s0, s0, 4
+; GFX1064-NEXT:    s_addc_u32 s1, s1, 0
+; GFX1064-NEXT:    s_andn2_b64 s[4:5], s[4:5], exec
+; GFX1064-NEXT:    s_and_b64 s[8:9], vcc, exec
+; GFX1064-NEXT:    s_or_b64 s[4:5], s[4:5], s[8:9]
+; GFX1064-NEXT:  .LBB11_3: ; %Flow
+; GFX1064-NEXT:    ; in Loop: Header=BB11_4 Depth=1
+; GFX1064-NEXT:    s_and_b64 s[8:9], exec, s[4:5]
+; GFX1064-NEXT:    s_or_b64 s[2:3], s[8:9], s[2:3]
+; GFX1064-NEXT:    s_andn2_b64 exec, exec, s[2:3]
+; GFX1064-NEXT:    s_cbranch_execz .LBB11_6
+; GFX1064-NEXT:  .LBB11_4: ; %bb2
+; GFX1064-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    global_load_dword v3, v2, s[0:1]
+; GFX1064-NEXT:    s_or_b64 s[4:5], s[4:5], exec
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_cmp_gt_i32_e32 vcc, 11, v3
+; GFX1064-NEXT:    s_cbranch_vccz .LBB11_2
+; GFX1064-NEXT:  ; %bb.5: ; in Loop: Header=BB11_4 Depth=1
+; GFX1064-NEXT:    ; implicit-def: $sgpr6
+; GFX1064-NEXT:    ; implicit-def: $sgpr0_sgpr1
+; GFX1064-NEXT:    s_branch .LBB11_3
+; GFX1064-NEXT:  .LBB11_6: ; %.loopexit
+; GFX1064-NEXT:    s_endpgm
 bb:
   %tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
   %tmp1 = icmp eq i32 %tmp, 0
@@ -284,12 +628,30 @@ bb8:
   ret void
 }
 
-; GCN-LABEL: {{^}}test_addc_vop2b:
-; GFX1032: v_add_co_u32 v{{[0-9]+}}, vcc_lo, v{{[0-9]+}}, s{{[0-9]+}}
-; GFX1032: v_add_co_ci_u32_e32 v{{[0-9]+}}, vcc_lo, s{{[0-9]+}}, v{{[0-9]+}}, vcc_lo
-; GFX1064: v_add_co_u32 v{{[0-9]+}}, vcc, v{{[0-9]+}}, s{{[0-9]+}}
-; GFX1064: v_add_co_ci_u32_e32 v{{[0-9]+}}, vcc, s{{[0-9]+}}, v{{[0-9]+}}, vcc{{$}}
 define amdgpu_kernel void @test_addc_vop2b(ptr addrspace(1) %arg, i64 %arg1) #0 {
+; GFX1032-LABEL: test_addc_vop2b:
+; GFX1032:       ; %bb.0: ; %bb
+; GFX1032-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX1032-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    global_load_dwordx2 v[0:1], v2, s[0:1]
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_add_co_u32 v0, vcc_lo, v0, s2
+; GFX1032-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, s3, v1, vcc_lo
+; GFX1032-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_addc_vop2b:
+; GFX1064:       ; %bb.0: ; %bb
+; GFX1064-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX1064-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    global_load_dwordx2 v[0:1], v2, s[0:1]
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_add_co_u32 v0, vcc, v0, s2
+; GFX1064-NEXT:    v_add_co_ci_u32_e32 v1, vcc, s3, v1, vcc
+; GFX1064-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GFX1064-NEXT:    s_endpgm
 bb:
   %tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
   %tmp3 = getelementptr inbounds i64, ptr addrspace(1) %arg, i32 %tmp
@@ -299,12 +661,30 @@ bb:
   ret void
 }
 
-; GCN-LABEL: {{^}}test_subbrev_vop2b:
-; GFX1032: v_sub_co_u32 v{{[0-9]+}}, [[A0:s[0-9]+|vcc_lo]], v{{[0-9]+}}, s{{[0-9]+}}{{$}}
-; GFX1032: v_subrev_co_ci_u32_e32 v{{[0-9]+}}, vcc_lo, {{[vs][0-9]+}}, {{[vs][0-9]+}}, [[A0]]{{$}}
-; GFX1064: v_sub_co_u32 v{{[0-9]+}}, [[A0:s\[[0-9:]+\]|vcc]], v{{[0-9]+}}, s{{[0-9]+}}{{$}}
-; GFX1064: v_subrev_co_ci_u32_e32 v{{[0-9]+}}, vcc, {{[vs][0-9]+}}, {{[vs][0-9]+}}, [[A0]]{{$}}
 define amdgpu_kernel void @test_subbrev_vop2b(ptr addrspace(1) %arg, i64 %arg1) #0 {
+; GFX1032-LABEL: test_subbrev_vop2b:
+; GFX1032:       ; %bb.0: ; %bb
+; GFX1032-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX1032-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    global_load_dwordx2 v[0:1], v2, s[0:1]
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_sub_co_u32 v0, vcc_lo, v0, s2
+; GFX1032-NEXT:    v_subrev_co_ci_u32_e32 v1, vcc_lo, s3, v1, vcc_lo
+; GFX1032-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_subbrev_vop2b:
+; GFX1064:       ; %bb.0: ; %bb
+; GFX1064-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX1064-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    global_load_dwordx2 v[0:1], v2, s[0:1]
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_sub_co_u32 v0, vcc, v0, s2
+; GFX1064-NEXT:    v_subrev_co_ci_u32_e32 v1, vcc, s3, v1, vcc
+; GFX1064-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GFX1064-NEXT:    s_endpgm
 bb:
   %tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
   %tmp3 = getelementptr inbounds i64, ptr addrspace(1) %arg, i32 %tmp
@@ -314,12 +694,30 @@ bb:
   ret void
 }
 
-; GCN-LABEL: {{^}}test_subb_vop2b:
-; GFX1032: v_sub_co_u32 v{{[0-9]+}}, [[A0:s[0-9]+|vcc_lo]], s{{[0-9]+}}, v{{[0-9]+}}{{$}}
-; GFX1032: v_sub_co_ci_u32_e32 v{{[0-9]+}}, vcc_lo, {{[vs][0-9]+}}, v{{[0-9]+}}, [[A0]]{{$}}
-; GFX1064: v_sub_co_u32 v{{[0-9]+}}, [[A0:s\[[0-9:]+\]|vcc]], s{{[0-9]+}}, v{{[0-9]+}}{{$}}
-; GFX1064: v_sub_co_ci_u32_e32 v{{[0-9]+}}, vcc, {{[vs][0-9]+}}, v{{[0-9]+}}, [[A0]]{{$}}
 define amdgpu_kernel void @test_subb_vop2b(ptr addrspace(1) %arg, i64 %arg1) #0 {
+; GFX1032-LABEL: test_subb_vop2b:
+; GFX1032:       ; %bb.0: ; %bb
+; GFX1032-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX1032-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    global_load_dwordx2 v[0:1], v2, s[0:1]
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_sub_co_u32 v0, vcc_lo, s2, v0
+; GFX1032-NEXT:    v_sub_co_ci_u32_e32 v1, vcc_lo, s3, v1, vcc_lo
+; GFX1032-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_subb_vop2b:
+; GFX1064:       ; %bb.0: ; %bb
+; GFX1064-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX1064-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    global_load_dwordx2 v[0:1], v2, s[0:1]
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_sub_co_u32 v0, vcc, s2, v0
+; GFX1064-NEXT:    v_sub_co_ci_u32_e32 v1, vcc, s3, v1, vcc
+; GFX1064-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GFX1064-NEXT:    s_endpgm
 bb:
   %tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
   %tmp3 = getelementptr inbounds i64, ptr addrspace(1) %arg, i32 %tmp
@@ -329,17 +727,329 @@ bb:
   ret void
 }
 
-; GCN-LABEL: {{^}}test_udiv64:
-; GCN: s_add_u32 s{{[0-9]+}}, s{{[0-9]+}}, s{{[0-9]+}}
-; GCN: s_addc_u32 s{{[0-9]+}}, 0, s{{[0-9]+}}
-; GCN: s_add_u32 s{{[0-9]+}}, s{{[0-9]+}}, s{{[0-9]+}}
-; GCN: s_addc_u32 s{{[0-9]+}}, s{{[0-9]+}}, s{{[0-9]+}}
-; GCN: s_addc_u32 s{{[0-9]+}}, s{{[0-9]+}}, 0
-; GCN: s_add_u32 s{{[0-9]+}}, s{{[0-9]+}}, s{{[0-9]+}}
-; GCN: s_add_u32 s{{[0-9]+}}, s{{[0-9]+}}, s{{[0-9]+}}
-; GCN: s_addc_u32 s{{[0-9]+}}, 0, s{{[0-9]+}}
-; GCN: s_addc_u32 s{{[0-9]+}}, s{{[0-9]+}}, s{{[0-9]+}}
 define amdgpu_kernel void @test_udiv64(ptr addrspace(1) %arg) #0 {
+; GFX1032-LABEL: test_udiv64:
+; GFX1032:       ; %bb.0: ; %bb
+; GFX1032-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    s_load_dwordx4 s[4:7], s[2:3], 0x0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    s_or_b64 s[8:9], s[6:7], s[4:5]
+; GFX1032-NEXT:    s_mov_b32 s8, 0
+; GFX1032-NEXT:    s_cmp_lg_u64 s[8:9], 0
+; GFX1032-NEXT:    s_cbranch_scc0 .LBB15_4
+; GFX1032-NEXT:  ; %bb.1:
+; GFX1032-NEXT:    v_cvt_f32_u32_e32 v0, s4
+; GFX1032-NEXT:    v_cvt_f32_u32_e32 v1, s5
+; GFX1032-NEXT:    s_sub_u32 s9, 0, s4
+; GFX1032-NEXT:    s_subb_u32 s10, 0, s5
+; GFX1032-NEXT:    v_mac_f32_e32 v0, 0x4f800000, v1
+; GFX1032-NEXT:    v_rcp_f32_e32 v0, v0
+; GFX1032-NEXT:    v_mul_f32_e32 v0, 0x5f7ffffc, v0
+; GFX1032-NEXT:    v_mul_f32_e32 v1, 0x2f800000, v0
+; GFX1032-NEXT:    v_trunc_f32_e32 v1, v1
+; GFX1032-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v1
+; GFX1032-NEXT:    v_cvt_u32_f32_e32 v1, v1
+; GFX1032-NEXT:    v_cvt_u32_f32_e32 v0, v0
+; GFX1032-NEXT:    v_readfirstlane_b32 s0, v1
+; GFX1032-NEXT:    v_readfirstlane_b32 s1, v0
+; GFX1032-NEXT:    s_mul_i32 s11, s9, s0
+; GFX1032-NEXT:    s_mul_hi_u32 s13, s9, s1
+; GFX1032-NEXT:    s_mul_i32 s12, s10, s1
+; GFX1032-NEXT:    s_add_i32 s11, s13, s11
+; GFX1032-NEXT:    s_mul_i32 s14, s9, s1
+; GFX1032-NEXT:    s_add_i32 s11, s11, s12
+; GFX1032-NEXT:    s_mul_hi_u32 s13, s1, s14
+; GFX1032-NEXT:    s_mul_hi_u32 s15, s0, s14
+; GFX1032-NEXT:    s_mul_i32 s12, s0, s14
+; GFX1032-NEXT:    s_mul_hi_u32 s14, s1, s11
+; GFX1032-NEXT:    s_mul_i32 s1, s1, s11
+; GFX1032-NEXT:    s_mul_hi_u32 s16, s0, s11
+; GFX1032-NEXT:    s_add_u32 s1, s13, s1
+; GFX1032-NEXT:    s_addc_u32 s13, 0, s14
+; GFX1032-NEXT:    s_add_u32 s1, s1, s12
+; GFX1032-NEXT:    s_mul_i32 s11, s0, s11
+; GFX1032-NEXT:    s_addc_u32 s1, s13, s15
+; GFX1032-NEXT:    s_addc_u32 s12, s16, 0
+; GFX1032-NEXT:    s_add_u32 s1, s1, s11
+; GFX1032-NEXT:    s_addc_u32 s11, 0, s12
+; GFX1032-NEXT:    v_add_co_u32 v0, s1, v0, s1
+; GFX1032-NEXT:    s_cmp_lg_u32 s1, 0
+; GFX1032-NEXT:    s_addc_u32 s0, s0, s11
+; GFX1032-NEXT:    v_readfirstlane_b32 s1, v0
+; GFX1032-NEXT:    s_mul_i32 s11, s9, s0
+; GFX1032-NEXT:    s_mul_hi_u32 s12, s9, s1
+; GFX1032-NEXT:    s_mul_i32 s10, s10, s1
+; GFX1032-NEXT:    s_add_i32 s11, s12, s11
+; GFX1032-NEXT:    s_mul_i32 s9, s9, s1
+; GFX1032-NEXT:    s_add_i32 s11, s11, s10
+; GFX1032-NEXT:    s_mul_hi_u32 s12, s0, s9
+; GFX1032-NEXT:    s_mul_i32 s13, s0, s9
+; GFX1032-NEXT:    s_mul_hi_u32 s9, s1, s9
+; GFX1032-NEXT:    s_mul_hi_u32 s14, s1, s11
+; GFX1032-NEXT:    s_mul_i32 s1, s1, s11
+; GFX1032-NEXT:    s_mul_hi_u32 s10, s0, s11
+; GFX1032-NEXT:    s_add_u32 s1, s9, s1
+; GFX1032-NEXT:    s_addc_u32 s9, 0, s14
+; GFX1032-NEXT:    s_add_u32 s1, s1, s13
+; GFX1032-NEXT:    s_mul_i32 s11, s0, s11
+; GFX1032-NEXT:    s_addc_u32 s1, s9, s12
+; GFX1032-NEXT:    s_addc_u32 s9, s10, 0
+; GFX1032-NEXT:    s_add_u32 s1, s1, s11
+; GFX1032-NEXT:    s_addc_u32 s9, 0, s9
+; GFX1032-NEXT:    v_add_co_u32 v0, s1, v0, s1
+; GFX1032-NEXT:    s_cmp_lg_u32 s1, 0
+; GFX1032-NEXT:    s_addc_u32 s0, s0, s9
+; GFX1032-NEXT:    v_readfirstlane_b32 s1, v0
+; GFX1032-NEXT:    s_mul_i32 s10, s6, s0
+; GFX1032-NEXT:    s_mul_hi_u32 s9, s6, s0
+; GFX1032-NEXT:    s_mul_hi_u32 s11, s7, s0
+; GFX1032-NEXT:    s_mul_i32 s0, s7, s0
+; GFX1032-NEXT:    s_mul_hi_u32 s12, s6, s1
+; GFX1032-NEXT:    s_mul_hi_u32 s13, s7, s1
+; GFX1032-NEXT:    s_mul_i32 s1, s7, s1
+; GFX1032-NEXT:    s_add_u32 s10, s12, s10
+; GFX1032-NEXT:    s_addc_u32 s9, 0, s9
+; GFX1032-NEXT:    s_add_u32 s1, s10, s1
+; GFX1032-NEXT:    s_addc_u32 s1, s9, s13
+; GFX1032-NEXT:    s_addc_u32 s9, s11, 0
+; GFX1032-NEXT:    s_add_u32 s1, s1, s0
+; GFX1032-NEXT:    s_addc_u32 s9, 0, s9
+; GFX1032-NEXT:    s_mul_hi_u32 s0, s4, s1
+; GFX1032-NEXT:    s_mul_i32 s11, s4, s9
+; GFX1032-NEXT:    s_mul_i32 s12, s4, s1
+; GFX1032-NEXT:    s_add_i32 s0, s0, s11
+; GFX1032-NEXT:    v_sub_co_u32 v0, s11, s6, s12
+; GFX1032-NEXT:    s_mul_i32 s10, s5, s1
+; GFX1032-NEXT:    s_add_i32 s0, s0, s10
+; GFX1032-NEXT:    v_sub_co_u32 v1, s12, v0, s4
+; GFX1032-NEXT:    s_sub_i32 s10, s7, s0
+; GFX1032-NEXT:    s_cmp_lg_u32 s11, 0
+; GFX1032-NEXT:    s_subb_u32 s10, s10, s5
+; GFX1032-NEXT:    s_cmp_lg_u32 s12, 0
+; GFX1032-NEXT:    v_cmp_le_u32_e32 vcc_lo, s4, v1
+; GFX1032-NEXT:    s_subb_u32 s10, s10, 0
+; GFX1032-NEXT:    s_cmp_ge_u32 s10, s5
+; GFX1032-NEXT:    v_cndmask_b32_e64 v1, 0, -1, vcc_lo
+; GFX1032-NEXT:    s_cselect_b32 s12, -1, 0
+; GFX1032-NEXT:    s_cmp_eq_u32 s10, s5
+; GFX1032-NEXT:    s_cselect_b32 vcc_lo, -1, 0
+; GFX1032-NEXT:    s_add_u32 s10, s1, 1
+; GFX1032-NEXT:    v_cndmask_b32_e32 v1, s12, v1, vcc_lo
+; GFX1032-NEXT:    s_addc_u32 s12, s9, 0
+; GFX1032-NEXT:    s_add_u32 s13, s1, 2
+; GFX1032-NEXT:    s_addc_u32 s14, s9, 0
+; GFX1032-NEXT:    s_cmp_lg_u32 s11, 0
+; GFX1032-NEXT:    v_cmp_le_u32_e32 vcc_lo, s4, v0
+; GFX1032-NEXT:    s_subb_u32 s0, s7, s0
+; GFX1032-NEXT:    v_mov_b32_e32 v2, s13
+; GFX1032-NEXT:    s_cmp_ge_u32 s0, s5
+; GFX1032-NEXT:    v_cndmask_b32_e64 v0, 0, -1, vcc_lo
+; GFX1032-NEXT:    s_cselect_b32 s7, -1, 0
+; GFX1032-NEXT:    s_cmp_eq_u32 s0, s5
+; GFX1032-NEXT:    v_cmp_ne_u32_e32 vcc_lo, 0, v1
+; GFX1032-NEXT:    s_cselect_b32 s0, -1, 0
+; GFX1032-NEXT:    v_mov_b32_e32 v1, s14
+; GFX1032-NEXT:    v_cndmask_b32_e64 v0, s7, v0, s0
+; GFX1032-NEXT:    v_cndmask_b32_e32 v2, s10, v2, vcc_lo
+; GFX1032-NEXT:    v_cndmask_b32_e32 v1, s12, v1, vcc_lo
+; GFX1032-NEXT:    v_cmp_ne_u32_e32 vcc_lo, 0, v0
+; GFX1032-NEXT:    v_cndmask_b32_e32 v1, s9, v1, vcc_lo
+; GFX1032-NEXT:    v_cndmask_b32_e32 v0, s1, v2, vcc_lo
+; GFX1032-NEXT:    s_andn2_b32 vcc_lo, exec_lo, s8
+; GFX1032-NEXT:    s_cbranch_vccnz .LBB15_3
+; GFX1032-NEXT:  .LBB15_2:
+; GFX1032-NEXT:    v_cvt_f32_u32_e32 v0, s4
+; GFX1032-NEXT:    s_sub_i32 s1, 0, s4
+; GFX1032-NEXT:    v_rcp_iflag_f32_e32 v0, v0
+; GFX1032-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
+; GFX1032-NEXT:    v_cvt_u32_f32_e32 v0, v0
+; GFX1032-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX1032-NEXT:    s_mul_i32 s1, s1, s0
+; GFX1032-NEXT:    s_mul_hi_u32 s1, s0, s1
+; GFX1032-NEXT:    s_add_i32 s0, s0, s1
+; GFX1032-NEXT:    s_mul_hi_u32 s0, s6, s0
+; GFX1032-NEXT:    s_mul_i32 s1, s0, s4
+; GFX1032-NEXT:    s_add_i32 s5, s0, 1
+; GFX1032-NEXT:    s_sub_i32 s1, s6, s1
+; GFX1032-NEXT:    s_sub_i32 s6, s1, s4
+; GFX1032-NEXT:    s_cmp_ge_u32 s1, s4
+; GFX1032-NEXT:    s_cselect_b32 s0, s5, s0
+; GFX1032-NEXT:    s_cselect_b32 s1, s6, s1
+; GFX1032-NEXT:    s_add_i32 s5, s0, 1
+; GFX1032-NEXT:    s_cmp_ge_u32 s1, s4
+; GFX1032-NEXT:    s_mov_b32 s1, 0
+; GFX1032-NEXT:    s_cselect_b32 s0, s5, s0
+; GFX1032-NEXT:    v_mov_b32_e32 v0, s0
+; GFX1032-NEXT:    v_mov_b32_e32 v1, s1
+; GFX1032-NEXT:  .LBB15_3:
+; GFX1032-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1032-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3] offset:16
+; GFX1032-NEXT:    s_endpgm
+; GFX1032-NEXT:  .LBB15_4:
+; GFX1032-NEXT:    ; implicit-def: $vgpr0_vgpr1
+; GFX1032-NEXT:    s_branch .LBB15_2
+;
+; GFX1064-LABEL: test_udiv64:
+; GFX1064:       ; %bb.0: ; %bb
+; GFX1064-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    s_load_dwordx4 s[4:7], s[2:3], 0x0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    s_or_b64 s[0:1], s[6:7], s[4:5]
+; GFX1064-NEXT:    s_mov_b32 s0, 0
+; GFX1064-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX1064-NEXT:    s_cbranch_scc0 .LBB15_4
+; GFX1064-NEXT:  ; %bb.1:
+; GFX1064-NEXT:    v_cvt_f32_u32_e32 v0, s4
+; GFX1064-NEXT:    v_cvt_f32_u32_e32 v1, s5
+; GFX1064-NEXT:    s_sub_u32 s9, 0, s4
+; GFX1064-NEXT:    s_subb_u32 s10, 0, s5
+; GFX1064-NEXT:    v_mac_f32_e32 v0, 0x4f800000, v1
+; GFX1064-NEXT:    v_rcp_f32_e32 v0, v0
+; GFX1064-NEXT:    v_mul_f32_e32 v0, 0x5f7ffffc, v0
+; GFX1064-NEXT:    v_mul_f32_e32 v1, 0x2f800000, v0
+; GFX1064-NEXT:    v_trunc_f32_e32 v1, v1
+; GFX1064-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v1
+; GFX1064-NEXT:    v_cvt_u32_f32_e32 v1, v1
+; GFX1064-NEXT:    v_cvt_u32_f32_e32 v0, v0
+; GFX1064-NEXT:    v_readfirstlane_b32 s8, v1
+; GFX1064-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX1064-NEXT:    s_mul_i32 s1, s9, s8
+; GFX1064-NEXT:    s_mul_hi_u32 s12, s9, s0
+; GFX1064-NEXT:    s_mul_i32 s11, s10, s0
+; GFX1064-NEXT:    s_add_i32 s1, s12, s1
+; GFX1064-NEXT:    s_mul_i32 s13, s9, s0
+; GFX1064-NEXT:    s_add_i32 s1, s1, s11
+; GFX1064-NEXT:    s_mul_hi_u32 s12, s0, s13
+; GFX1064-NEXT:    s_mul_hi_u32 s14, s8, s13
+; GFX1064-NEXT:    s_mul_i32 s11, s8, s13
+; GFX1064-NEXT:    s_mul_hi_u32 s13, s0, s1
+; GFX1064-NEXT:    s_mul_i32 s0, s0, s1
+; GFX1064-NEXT:    s_mul_hi_u32 s15, s8, s1
+; GFX1064-NEXT:    s_add_u32 s0, s12, s0
+; GFX1064-NEXT:    s_addc_u32 s12, 0, s13
+; GFX1064-NEXT:    s_add_u32 s0, s0, s11
+; GFX1064-NEXT:    s_mul_i32 s1, s8, s1
+; GFX1064-NEXT:    s_addc_u32 s0, s12, s14
+; GFX1064-NEXT:    s_addc_u32 s11, s15, 0
+; GFX1064-NEXT:    s_add_u32 s0, s0, s1
+; GFX1064-NEXT:    s_addc_u32 s11, 0, s11
+; GFX1064-NEXT:    v_add_co_u32 v0, s[0:1], v0, s0
+; GFX1064-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX1064-NEXT:    s_addc_u32 s8, s8, s11
+; GFX1064-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX1064-NEXT:    s_mul_i32 s1, s9, s8
+; GFX1064-NEXT:    s_mul_hi_u32 s11, s9, s0
+; GFX1064-NEXT:    s_mul_i32 s10, s10, s0
+; GFX1064-NEXT:    s_add_i32 s1, s11, s1
+; GFX1064-NEXT:    s_mul_i32 s9, s9, s0
+; GFX1064-NEXT:    s_add_i32 s1, s1, s10
+; GFX1064-NEXT:    s_mul_hi_u32 s11, s8, s9
+; GFX1064-NEXT:    s_mul_i32 s12, s8, s9
+; GFX1064-NEXT:    s_mul_hi_u32 s9, s0, s9
+; GFX1064-NEXT:    s_mul_hi_u32 s13, s0, s1
+; GFX1064-NEXT:    s_mul_i32 s0, s0, s1
+; GFX1064-NEXT:    s_mul_hi_u32 s10, s8, s1
+; GFX1064-NEXT:    s_add_u32 s0, s9, s0
+; GFX1064-NEXT:    s_addc_u32 s9, 0, s13
+; GFX1064-NEXT:    s_add_u32 s0, s0, s12
+; GFX1064-NEXT:    s_mul_i32 s1, s8, s1
+; GFX1064-NEXT:    s_addc_u32 s0, s9, s11
+; GFX1064-NEXT:    s_addc_u32 s9, s10, 0
+; GFX1064-NEXT:    s_add_u32 s0, s0, s1
+; GFX1064-NEXT:    s_addc_u32 s9, 0, s9
+; GFX1064-NEXT:    v_add_co_u32 v0, s[0:1], v0, s0
+; GFX1064-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX1064-NEXT:    s_addc_u32 s0, s8, s9
+; GFX1064-NEXT:    v_readfirstlane_b32 s1, v0
+; GFX1064-NEXT:    s_mul_i32 s9, s6, s0
+; GFX1064-NEXT:    s_mul_hi_u32 s8, s6, s0
+; GFX1064-NEXT:    s_mul_hi_u32 s10, s7, s0
+; GFX1064-NEXT:    s_mul_i32 s0, s7, s0
+; GFX1064-NEXT:    s_mul_hi_u32 s11, s6, s1
+; GFX1064-NEXT:    s_mul_hi_u32 s12, s7, s1
+; GFX1064-NEXT:    s_mul_i32 s1, s7, s1
+; GFX1064-NEXT:    s_add_u32 s9, s11, s9
+; GFX1064-NEXT:    s_addc_u32 s8, 0, s8
+; GFX1064-NEXT:    s_add_u32 s1, s9, s1
+; GFX1064-NEXT:    s_addc_u32 s1, s8, s12
+; GFX1064-NEXT:    s_addc_u32 s8, s10, 0
+; GFX1064-NEXT:    s_add_u32 s10, s1, s0
+; GFX1064-NEXT:    s_addc_u32 s11, 0, s8
+; GFX1064-NEXT:    s_mul_hi_u32 s0, s4, s10
+; GFX1064-NEXT:    s_mul_i32 s1, s4, s11
+; GFX1064-NEXT:    s_mul_i32 s9, s4, s10
+; GFX1064-NEXT:    s_add_i32 s12, s0, s1
+; GFX1064-NEXT:    v_sub_co_u32 v0, s[0:1], s6, s9
+; GFX1064-NEXT:    s_mul_i32 s8, s5, s10
+; GFX1064-NEXT:    s_add_i32 s12, s12, s8
+; GFX1064-NEXT:    v_sub_co_u32 v1, s[8:9], v0, s4
+; GFX1064-NEXT:    s_sub_i32 s13, s7, s12
+; GFX1064-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX1064-NEXT:    s_subb_u32 s13, s13, s5
+; GFX1064-NEXT:    s_cmp_lg_u64 s[8:9], 0
+; GFX1064-NEXT:    v_cmp_le_u32_e32 vcc, s4, v1
+; GFX1064-NEXT:    s_subb_u32 s8, s13, 0
+; GFX1064-NEXT:    s_cmp_ge_u32 s8, s5
+; GFX1064-NEXT:    v_cndmask_b32_e64 v1, 0, -1, vcc
+; GFX1064-NEXT:    s_cselect_b32 s9, -1, 0
+; GFX1064-NEXT:    s_cmp_eq_u32 s8, s5
+; GFX1064-NEXT:    s_cselect_b64 vcc, -1, 0
+; GFX1064-NEXT:    s_add_u32 s8, s10, 1
+; GFX1064-NEXT:    v_cndmask_b32_e32 v1, s9, v1, vcc
+; GFX1064-NEXT:    s_addc_u32 s9, s11, 0
+; GFX1064-NEXT:    s_add_u32 s13, s10, 2
+; GFX1064-NEXT:    s_addc_u32 s14, s11, 0
+; GFX1064-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX1064-NEXT:    v_cmp_le_u32_e32 vcc, s4, v0
+; GFX1064-NEXT:    s_subb_u32 s0, s7, s12
+; GFX1064-NEXT:    v_mov_b32_e32 v2, s13
+; GFX1064-NEXT:    s_cmp_ge_u32 s0, s5
+; GFX1064-NEXT:    v_cndmask_b32_e64 v0, 0, -1, vcc
+; GFX1064-NEXT:    s_cselect_b32 s7, -1, 0
+; GFX1064-NEXT:    s_cmp_eq_u32 s0, s5
+; GFX1064-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v1
+; GFX1064-NEXT:    s_cselect_b64 s[0:1], -1, 0
+; GFX1064-NEXT:    v_mov_b32_e32 v1, s14
+; GFX1064-NEXT:    v_cndmask_b32_e64 v0, s7, v0, s[0:1]
+; GFX1064-NEXT:    v_cndmask_b32_e32 v2, s8, v2, vcc
+; GFX1064-NEXT:    v_cndmask_b32_e32 v1, s9, v1, vcc
+; GFX1064-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v0
+; GFX1064-NEXT:    v_cndmask_b32_e32 v1, s11, v1, vcc
+; GFX1064-NEXT:    v_cndmask_b32_e32 v0, s10, v2, vcc
+; GFX1064-NEXT:    s_cbranch_execnz .LBB15_3
+; GFX1064-NEXT:  .LBB15_2:
+; GFX1064-NEXT:    v_cvt_f32_u32_e32 v0, s4
+; GFX1064-NEXT:    s_sub_i32 s1, 0, s4
+; GFX1064-NEXT:    v_rcp_iflag_f32_e32 v0, v0
+; GFX1064-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
+; GFX1064-NEXT:    v_cvt_u32_f32_e32 v0, v0
+; GFX1064-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX1064-NEXT:    s_mul_i32 s1, s1, s0
+; GFX1064-NEXT:    s_mul_hi_u32 s1, s0, s1
+; GFX1064-NEXT:    s_add_i32 s0, s0, s1
+; GFX1064-NEXT:    s_mul_hi_u32 s0, s6, s0
+; GFX1064-NEXT:    s_mul_i32 s1, s0, s4
+; GFX1064-NEXT:    s_add_i32 s5, s0, 1
+; GFX1064-NEXT:    s_sub_i32 s1, s6, s1
+; GFX1064-NEXT:    s_sub_i32 s6, s1, s4
+; GFX1064-NEXT:    s_cmp_ge_u32 s1, s4
+; GFX1064-NEXT:    s_cselect_b32 s0, s5, s0
+; GFX1064-NEXT:    s_cselect_b32 s1, s6, s1
+; GFX1064-NEXT:    s_add_i32 s5, s0, 1
+; GFX1064-NEXT:    s_cmp_ge_u32 s1, s4
+; GFX1064-NEXT:    s_mov_b32 s1, 0
+; GFX1064-NEXT:    s_cselect_b32 s0, s5, s0
+; GFX1064-NEXT:    v_mov_b32_e32 v0, s0
+; GFX1064-NEXT:    v_mov_b32_e32 v1, s1
+; GFX1064-NEXT:  .LBB15_3:
+; GFX1064-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1064-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3] offset:16
+; GFX1064-NEXT:    s_endpgm
+; GFX1064-NEXT:  .LBB15_4:
+; GFX1064-NEXT:    ; implicit-def: $vgpr0_vgpr1
+; GFX1064-NEXT:    s_branch .LBB15_2
 bb:
   %tmp = getelementptr inbounds i64, ptr addrspace(1) %arg, i64 1
   %tmp1 = load i64, ptr addrspace(1) %tmp, align 8
@@ -350,10 +1060,34 @@ bb:
   ret void
 }
 
-; GCN-LABEL: {{^}}test_div_scale_f32:
-; GFX1032: v_div_scale_f32 v{{[0-9]+}}, s{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; GFX1064: v_div_scale_f32 v{{[0-9]+}}, s[{{[0-9:]+}}], v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
 define amdgpu_kernel void @test_div_scale_f32(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 {
+; GFX1032-LABEL: test_div_scale_f32:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX1032-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    global_load_dword v1, v0, s[2:3] glc dlc
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    global_load_dword v2, v0, s[2:3] offset:4 glc dlc
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    v_div_scale_f32 v1, s2, v2, v2, v1
+; GFX1032-NEXT:    global_store_dword v0, v1, s[0:1]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_div_scale_f32:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX1064-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    global_load_dword v1, v0, s[2:3] glc dlc
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    global_load_dword v2, v0, s[2:3] offset:4 glc dlc
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    v_div_scale_f32 v1, s[2:3], v2, v2, v1
+; GFX1064-NEXT:    global_store_dword v0, v1, s[0:1]
+; GFX1064-NEXT:    s_endpgm
   %tid = call i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
   %gep.0 = getelementptr float, ptr addrspace(1) %in, i32 %tid
   %gep.1 = getelementptr float, ptr addrspace(1) %gep.0, i32 1
@@ -367,10 +1101,36 @@ define amdgpu_kernel void @test_div_scale_f32(ptr addrspace(1) %out, ptr addrspa
   ret void
 }
 
-; GCN-LABEL: {{^}}test_div_scale_f64:
-; GFX1032: v_div_scale_f64 v[{{[0-9:]+}}], s{{[0-9]+}}, v[{{[0-9:]+}}], v[{{[0-9:]+}}], v[{{[0-9:]+}}]
-; GFX1064: v_div_scale_f64 v[{{[0-9:]+}}], s[{{[0-9:]+}}], v[{{[0-9:]+}}], v[{{[0-9:]+}}], v[{{[0-9:]+}}]
 define amdgpu_kernel void @test_div_scale_f64(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %in) #0 {
+; GFX1032-LABEL: test_div_scale_f64:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX1032-NEXT:    v_lshlrev_b32_e32 v4, 3, v0
+; GFX1032-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    global_load_dwordx2 v[0:1], v4, s[2:3] glc dlc
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    global_load_dwordx2 v[2:3], v4, s[2:3] offset:8 glc dlc
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_div_scale_f64 v[0:1], s2, v[0:1], v[2:3], v[0:1]
+; GFX1032-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1032-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_div_scale_f64:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX1064-NEXT:    v_lshlrev_b32_e32 v4, 3, v0
+; GFX1064-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    global_load_dwordx2 v[0:1], v4, s[2:3] glc dlc
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    global_load_dwordx2 v[2:3], v4, s[2:3] offset:8 glc dlc
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_div_scale_f64 v[0:1], s[2:3], v[0:1], v[2:3], v[0:1]
+; GFX1064-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1064-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GFX1064-NEXT:    s_endpgm
   %tid = call i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
   %gep.0 = getelementptr double, ptr addrspace(1) %in, i32 %tid
   %gep.1 = getelementptr double, ptr addrspace(1) %gep.0, i32 1
@@ -384,10 +1144,20 @@ define amdgpu_kernel void @test_div_scale_f64(ptr addrspace(1) %out, ptr addrspa
   ret void
 }
 
-; GCN-LABEL: {{^}}test_mad_i64_i32:
-; GFX1032: v_mad_i64_i32 v[{{[0-9:]+}}], s{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, v[{{[0-9:]+}}]
-; GFX1064: v_mad_i64_i32 v[{{[0-9:]+}}], s[{{[0-9:]+}}], v{{[0-9]+}}, v{{[0-9]+}}, v[{{[0-9:]+}}]
 define i64 @test_mad_i64_i32(i32 %arg0, i32 %arg1, i64 %arg2) #0 {
+; GFX1032-LABEL: test_mad_i64_i32:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX1032-NEXT:    v_mad_i64_i32 v[0:1], s4, v0, v1, v[2:3]
+; GFX1032-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX1064-LABEL: test_mad_i64_i32:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX1064-NEXT:    v_mad_i64_i32 v[0:1], s[4:5], v0, v1, v[2:3]
+; GFX1064-NEXT:    s_setpc_b64 s[30:31]
   %sext0 = sext i32 %arg0 to i64
   %sext1 = sext i32 %arg1 to i64
   %mul = mul i64 %sext0, %sext1
@@ -395,10 +1165,20 @@ define i64 @test_mad_i64_i32(i32 %arg0, i32 %arg1, i64 %arg2) #0 {
   ret i64 %mad
 }
 
-; GCN-LABEL: {{^}}test_mad_u64_u32:
-; GFX1032: v_mad_u64_u32 v[{{[0-9:]+}}], s{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, v[{{[0-9:]+}}]
-; GFX1064: v_mad_u64_u32 v[{{[0-9:]+}}], s[{{[0-9:]+}}], v{{[0-9]+}}, v{{[0-9]+}}, v[{{[0-9:]+}}]
 define i64 @test_mad_u64_u32(i32 %arg0, i32 %arg1, i64 %arg2) #0 {
+; GFX1032-LABEL: test_mad_u64_u32:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX1032-NEXT:    v_mad_u64_u32 v[0:1], s4, v0, v1, v[2:3]
+; GFX1032-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX1064-LABEL: test_mad_u64_u32:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX1064-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], v0, v1, v[2:3]
+; GFX1064-NEXT:    s_setpc_b64 s[30:31]
   %sext0 = zext i32 %arg0 to i64
   %sext1 = zext i32 %arg1 to i64
   %mul = mul i64 %sext0, %sext1
@@ -406,42 +1186,138 @@ define i64 @test_mad_u64_u32(i32 %arg0, i32 %arg1, i64 %arg2) #0 {
   ret i64 %mad
 }
 
-; GCN-LABEL: {{^}}test_div_fmas_f32:
-; GCN:     s_bitcmp1_b32 s{{[0-9]+}}, 0
-; GFX1032: s_cselect_b32 vcc_lo, -1, 0
-; GFX1064: s_cselect_b64 vcc, -1, 0
-; GCN:     v_div_fmas_f32 v{{[0-9]+}}, {{[vs][0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
 define amdgpu_kernel void @test_div_fmas_f32(ptr addrspace(1) %out, float %a, float %b, float %c, i1 %d) nounwind {
+; GFX1032-LABEL: test_div_fmas_f32:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_clause 0x1
+; GFX1032-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x2c
+; GFX1032-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1032-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_mov_b32_e32 v0, s5
+; GFX1032-NEXT:    v_mov_b32_e32 v1, s6
+; GFX1032-NEXT:    s_bitcmp1_b32 s7, 0
+; GFX1032-NEXT:    s_cselect_b32 vcc_lo, -1, 0
+; GFX1032-NEXT:    v_div_fmas_f32 v0, s4, v0, v1
+; GFX1032-NEXT:    global_store_dword v2, v0, s[2:3]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_div_fmas_f32:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_clause 0x1
+; GFX1064-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x2c
+; GFX1064-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1064-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_mov_b32_e32 v0, s5
+; GFX1064-NEXT:    v_mov_b32_e32 v1, s6
+; GFX1064-NEXT:    s_bitcmp1_b32 s7, 0
+; GFX1064-NEXT:    s_cselect_b64 vcc, -1, 0
+; GFX1064-NEXT:    v_div_fmas_f32 v0, s4, v0, v1
+; GFX1064-NEXT:    global_store_dword v2, v0, s[2:3]
+; GFX1064-NEXT:    s_endpgm
   %result = call float @llvm.amdgcn.div.fmas.f32(float %a, float %b, float %c, i1 %d) nounwind readnone
   store float %result, ptr addrspace(1) %out, align 4
   ret void
 }
 
-; GCN-LABEL: {{^}}test_div_fmas_f64:
-; GCN:     s_bitcmp1_b32 s{{[0-9]+}}, 0
-; GFX1032: s_cselect_b32 vcc_lo, -1, 0
-; GFX1064: s_cselect_b64 vcc, -1, 0
-; GCN-DAG: v_div_fmas_f64 v[{{[0-9:]+}}], {{[vs]}}[{{[0-9:]+}}], v[{{[0-9:]+}}], v[{{[0-9:]+}}]
 define amdgpu_kernel void @test_div_fmas_f64(ptr addrspace(1) %out, double %a, double %b, double %c, i1 %d) nounwind {
+; GFX1032-LABEL: test_div_fmas_f64:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_clause 0x1
+; GFX1032-NEXT:    s_load_dwordx8 s[4:11], s[0:1], 0x24
+; GFX1032-NEXT:    s_load_dword s2, s[0:1], 0x44
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_mov_b32_e32 v0, s8
+; GFX1032-NEXT:    v_mov_b32_e32 v1, s9
+; GFX1032-NEXT:    v_mov_b32_e32 v2, s10
+; GFX1032-NEXT:    v_mov_b32_e32 v3, s11
+; GFX1032-NEXT:    s_bitcmp1_b32 s2, 0
+; GFX1032-NEXT:    s_cselect_b32 vcc_lo, -1, 0
+; GFX1032-NEXT:    v_div_fmas_f64 v[0:1], s[6:7], v[0:1], v[2:3]
+; GFX1032-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1032-NEXT:    global_store_dwordx2 v2, v[0:1], s[4:5]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_div_fmas_f64:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_clause 0x1
+; GFX1064-NEXT:    s_load_dwordx8 s[4:11], s[0:1], 0x24
+; GFX1064-NEXT:    s_load_dword s2, s[0:1], 0x44
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_mov_b32_e32 v0, s8
+; GFX1064-NEXT:    v_mov_b32_e32 v1, s9
+; GFX1064-NEXT:    v_mov_b32_e32 v2, s10
+; GFX1064-NEXT:    v_mov_b32_e32 v3, s11
+; GFX1064-NEXT:    s_bitcmp1_b32 s2, 0
+; GFX1064-NEXT:    s_cselect_b64 vcc, -1, 0
+; GFX1064-NEXT:    v_div_fmas_f64 v[0:1], s[6:7], v[0:1], v[2:3]
+; GFX1064-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1064-NEXT:    global_store_dwordx2 v2, v[0:1], s[4:5]
+; GFX1064-NEXT:    s_endpgm
   %result = call double @llvm.amdgcn.div.fmas.f64(double %a, double %b, double %c, i1 %d) nounwind readnone
   store double %result, ptr addrspace(1) %out, align 8
   ret void
 }
 
-; GCN-LABEL: {{^}}test_div_fmas_f32_i1_phi_vcc:
-; GFX1032: s_mov_b32 [[VCC:vcc_lo]], 0{{$}}
-; GFX1064: s_mov_b64 [[VCC:vcc]], 0{{$}}
-; GFX1032: s_and_saveexec_b32 [[SAVE:s[0-9]+]], s{{[0-9]+}}{{$}}
-; GFX1064: s_and_saveexec_b64 [[SAVE:s\[[0-9]+:[0-9]+\]]], s[{{[0-9:]+}}]{{$}}
 
-; GCN: load_dword [[LOAD:v[0-9]+]]
-; GCN: v_cmp_ne_u32_e32 [[VCC]], 0, [[LOAD]]
 
-; GCN: .LBB{{[0-9_]+}}:
-; GFX1032: s_or_b32 exec_lo, exec_lo, [[SAVE]]
-; GFX1064: s_or_b64 exec, exec, [[SAVE]]
-; GCN: v_div_fmas_f32 {{v[0-9]+}}, {{v[0-9]+}}, {{v[0-9]+}}, {{v[0-9]+}}
 define amdgpu_kernel void @test_div_fmas_f32_i1_phi_vcc(ptr addrspace(1) %out, ptr addrspace(1) %in, ptr addrspace(1) %dummy) #0 {
+; GFX1032-LABEL: test_div_fmas_f32_i1_phi_vcc:
+; GFX1032:       ; %bb.0: ; %entry
+; GFX1032-NEXT:    s_clause 0x1
+; GFX1032-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX1032-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX1032-NEXT:    v_lshlrev_b32_e32 v1, 2, v0
+; GFX1032-NEXT:    s_mov_b32 null, 0
+; GFX1032-NEXT:    v_cmp_eq_u32_e64 s0, 0, v0
+; GFX1032-NEXT:    s_mov_b32 vcc_lo, 0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    global_load_dwordx3 v[1:3], v1, s[6:7]
+; GFX1032-NEXT:    s_and_saveexec_b32 s1, s0
+; GFX1032-NEXT:    s_cbranch_execz .LBB22_2
+; GFX1032-NEXT:  ; %bb.1: ; %bb
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    global_load_dword v0, v0, s[2:3] glc dlc
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_cmp_ne_u32_e32 vcc_lo, 0, v0
+; GFX1032-NEXT:    s_and_b32 vcc_lo, vcc_lo, exec_lo
+; GFX1032-NEXT:  .LBB22_2: ; %exit
+; GFX1032-NEXT:    s_waitcnt_depctr 0xffe3
+; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s1
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_div_fmas_f32 v1, v1, v2, v3
+; GFX1032-NEXT:    global_store_dword v0, v1, s[4:5] offset:8
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_div_fmas_f32_i1_phi_vcc:
+; GFX1064:       ; %bb.0: ; %entry
+; GFX1064-NEXT:    s_clause 0x1
+; GFX1064-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX1064-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; GFX1064-NEXT:    v_lshlrev_b32_e32 v1, 2, v0
+; GFX1064-NEXT:    s_mov_b32 null, 0
+; GFX1064-NEXT:    v_cmp_eq_u32_e64 s[0:1], 0, v0
+; GFX1064-NEXT:    s_mov_b64 vcc, 0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    global_load_dwordx3 v[1:3], v1, s[6:7]
+; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], s[0:1]
+; GFX1064-NEXT:    s_cbranch_execz .LBB22_2
+; GFX1064-NEXT:  ; %bb.1: ; %bb
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    global_load_dword v0, v0, s[8:9] glc dlc
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v0
+; GFX1064-NEXT:    s_and_b64 vcc, vcc, exec
+; GFX1064-NEXT:  .LBB22_2: ; %exit
+; GFX1064-NEXT:    s_waitcnt_depctr 0xffe3
+; GFX1064-NEXT:    s_or_b64 exec, exec, s[2:3]
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_div_fmas_f32 v1, v1, v2, v3
+; GFX1064-NEXT:    global_store_dword v0, v1, s[4:5] offset:8
+; GFX1064-NEXT:    s_endpgm
 entry:
   %tid = call i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
   %gep.out = getelementptr float, ptr addrspace(1) %out, i32 2
@@ -468,27 +1344,91 @@ exit:
   ret void
 }
 
-; GCN-LABEL: {{^}}fdiv_f32:
-; GFX1032: v_div_scale_f32 v{{[0-9]+}}, s{{[0-9]+}}, s{{[0-9]+}}, s{{[0-9]+}}, s{{[0-9]+}}
-; GFX1064: v_div_scale_f32 v{{[0-9]+}}, s{{\[[0-9]+:[0-9]+\]}}, s{{[0-9]+}}, s{{[0-9]+}}, s{{[0-9]+}}
-; GCN: v_rcp_f32_e32 v{{[0-9]+}}, v{{[0-9]+}}
-; GFX1032: v_div_scale_f32 v{{[0-9]+}}, vcc_lo, s{{[0-9]+}}, s{{[0-9]+}}, s{{[0-9]+}}
-; GFX1064: v_div_scale_f32 v{{[0-9]+}}, vcc, s{{[0-9]+}}, s{{[0-9]+}}, s{{[0-9]+}}
 
-; GCN-NOT: vcc
-; GCN: v_div_fmas_f32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
 define amdgpu_kernel void @fdiv_f32(ptr addrspace(1) %out, float %a, float %b) #0 {
+; GFX1032-LABEL: fdiv_f32:
+; GFX1032:       ; %bb.0: ; %entry
+; GFX1032-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_div_scale_f32 v0, s4, s3, s3, s2
+; GFX1032-NEXT:    v_rcp_f32_e32 v1, v0
+; GFX1032-NEXT:    v_fma_f32 v2, -v0, v1, 1.0
+; GFX1032-NEXT:    v_fmac_f32_e32 v1, v2, v1
+; GFX1032-NEXT:    v_div_scale_f32 v2, vcc_lo, s2, s3, s2
+; GFX1032-NEXT:    v_mul_f32_e32 v3, v2, v1
+; GFX1032-NEXT:    v_fma_f32 v4, -v0, v3, v2
+; GFX1032-NEXT:    v_fmac_f32_e32 v3, v4, v1
+; GFX1032-NEXT:    v_fma_f32 v0, -v0, v3, v2
+; GFX1032-NEXT:    v_div_fmas_f32 v0, v0, v1, v3
+; GFX1032-NEXT:    v_mov_b32_e32 v1, 0
+; GFX1032-NEXT:    v_div_fixup_f32 v0, v0, s3, s2
+; GFX1032-NEXT:    global_store_dword v1, v0, s[0:1]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: fdiv_f32:
+; GFX1064:       ; %bb.0: ; %entry
+; GFX1064-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_div_scale_f32 v0, s[4:5], s3, s3, s2
+; GFX1064-NEXT:    v_rcp_f32_e32 v1, v0
+; GFX1064-NEXT:    v_fma_f32 v2, -v0, v1, 1.0
+; GFX1064-NEXT:    v_fmac_f32_e32 v1, v2, v1
+; GFX1064-NEXT:    v_div_scale_f32 v2, vcc, s2, s3, s2
+; GFX1064-NEXT:    v_mul_f32_e32 v3, v2, v1
+; GFX1064-NEXT:    v_fma_f32 v4, -v0, v3, v2
+; GFX1064-NEXT:    v_fmac_f32_e32 v3, v4, v1
+; GFX1064-NEXT:    v_fma_f32 v0, -v0, v3, v2
+; GFX1064-NEXT:    v_div_fmas_f32 v0, v0, v1, v3
+; GFX1064-NEXT:    v_mov_b32_e32 v1, 0
+; GFX1064-NEXT:    v_div_fixup_f32 v0, v0, s3, s2
+; GFX1064-NEXT:    global_store_dword v1, v0, s[0:1]
+; GFX1064-NEXT:    s_endpgm
 entry:
   %fdiv = fdiv float %a, %b
   store float %fdiv, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}test_br_cc_f16:
-; GFX1032:  v_cmp_nlt_f16_e32 vcc_lo,
-; GFX1064:  v_cmp_nlt_f16_e32 vcc,
-; GCN-NEXT: s_cbranch_vccnz
 define amdgpu_kernel void @test_br_cc_f16(
+; GFX1032-LABEL: test_br_cc_f16:
+; GFX1032:       ; %bb.0: ; %entry
+; GFX1032-NEXT:    s_clause 0x1
+; GFX1032-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX1032-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    s_clause 0x1
+; GFX1032-NEXT:    global_load_ushort v1, v0, s[6:7]
+; GFX1032-NEXT:    global_load_ushort v2, v0, s[2:3]
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_cmp_nlt_f16_e32 vcc_lo, v1, v2
+; GFX1032-NEXT:    s_cbranch_vccnz .LBB24_2
+; GFX1032-NEXT:  ; %bb.1: ; %one
+; GFX1032-NEXT:    global_store_short v0, v1, s[4:5]
+; GFX1032-NEXT:    s_endpgm
+; GFX1032-NEXT:  .LBB24_2: ; %two
+; GFX1032-NEXT:    global_store_short v0, v2, s[4:5]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_br_cc_f16:
+; GFX1064:       ; %bb.0: ; %entry
+; GFX1064-NEXT:    s_clause 0x1
+; GFX1064-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX1064-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    s_clause 0x1
+; GFX1064-NEXT:    global_load_ushort v1, v0, s[6:7]
+; GFX1064-NEXT:    global_load_ushort v2, v0, s[2:3]
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_cmp_nlt_f16_e32 vcc, v1, v2
+; GFX1064-NEXT:    s_cbranch_vccnz .LBB24_2
+; GFX1064-NEXT:  ; %bb.1: ; %one
+; GFX1064-NEXT:    global_store_short v0, v1, s[4:5]
+; GFX1064-NEXT:    s_endpgm
+; GFX1064-NEXT:  .LBB24_2: ; %two
+; GFX1064-NEXT:    global_store_short v0, v2, s[4:5]
+; GFX1064-NEXT:    s_endpgm
     ptr addrspace(1) %r,
     ptr addrspace(1) %a,
     ptr addrspace(1) %b) {
@@ -507,10 +1447,21 @@ two:
   ret void
 }
 
-; GCN-LABEL: {{^}}test_brcc_i1:
-; GCN:      s_bitcmp0_b32 s{{[0-9]+}}, 0
-; GCN-NEXT: s_cbranch_scc1
 define amdgpu_kernel void @test_brcc_i1(ptr addrspace(1) noalias %out, ptr addrspace(1) noalias %in, i1 %val) #0 {
+; GCN-LABEL: test_brcc_i1:
+; GCN:       ; %bb.0:
+; GCN-NEXT:    s_load_dword s2, s[0:1], 0x34
+; GCN-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN-NEXT:    s_bitcmp0_b32 s2, 0
+; GCN-NEXT:    s_cbranch_scc1 .LBB25_2
+; GCN-NEXT:  ; %bb.1: ; %store
+; GCN-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GCN-NEXT:    v_mov_b32_e32 v0, 0
+; GCN-NEXT:    v_mov_b32_e32 v1, 0xde
+; GCN-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN-NEXT:    global_store_dword v0, v1, s[0:1]
+; GCN-NEXT:  .LBB25_2: ; %end
+; GCN-NEXT:    s_endpgm
   %cmp0 = icmp ne i1 %val, 0
   br i1 %cmp0, label %store, label %end
 
@@ -522,21 +1473,46 @@ end:
   ret void
 }
 
-; GCN-LABEL: {{^}}test_preserve_condition_undef_flag:
-; GFX1032-DAG: v_cmp_nlt_f32_e64 s{{[0-9]+}}, s{{[0-9]+}}, 1.0
-; GFX1032-DAG: v_cmp_ngt_f32_e64 s{{[0-9]+}}, s{{[0-9]+}}, 0
-; GFX1032: v_cmp_nlt_f32_e64 s{{[0-9]+}}, s{{[0-9]+}}, 1.0
-; GFX1032: s_or_b32 [[OR1:s[0-9]+]], s{{[0-9]+}}, s{{[0-9]+}}
-; GFX1032: s_or_b32 [[OR2:s[0-9]+]], [[OR1]], s{{[0-9]+}}
-; GFX1032: s_and_b32 vcc_lo, exec_lo, [[OR2]]
-; GFX1064-DAG: v_cmp_nlt_f32_e64 s[{{[0-9:]+}}], s{{[0-9]+}}, 1.0
-; GFX1064-DAG: v_cmp_ngt_f32_e64 s[{{[0-9:]+}}], s{{[0-9]+}}, 0
-; GFX1064: v_cmp_nlt_f32_e64 s[{{[0-9:]+}}], s{{[0-9]+}}, 1.0
-; GFX1064: s_or_b64 [[OR1:s\[[0-9:]+\]]], s[{{[0-9:]+}}], s[{{[0-9:]+}}]
-; GFX1064: s_or_b64 [[OR2:s\[[0-9:]+\]]], [[OR1]], s[{{[0-9:]+}}]
-; GFX1064: s_and_b64 vcc, exec, [[OR2]]
-; GCN:     s_cbranch_vccnz
 define amdgpu_kernel void @test_preserve_condition_undef_flag(float %arg, i32 %arg1, float %arg2) #0 {
+; GFX1032-LABEL: test_preserve_condition_undef_flag:
+; GFX1032:       ; %bb.0: ; %bb0
+; GFX1032-NEXT:    s_clause 0x1
+; GFX1032-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GFX1032-NEXT:    s_load_dword s3, s[0:1], 0x24
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_cmp_ngt_f32_e64 s0, s2, 0
+; GFX1032-NEXT:    v_cmp_nlt_f32_e64 s1, s2, 1.0
+; GFX1032-NEXT:    v_cmp_nlt_f32_e64 s2, s3, 1.0
+; GFX1032-NEXT:    s_or_b32 s0, s0, s1
+; GFX1032-NEXT:    s_or_b32 s0, s0, s2
+; GFX1032-NEXT:    s_and_b32 vcc_lo, exec_lo, s0
+; GFX1032-NEXT:    s_cbranch_vccnz .LBB26_2
+; GFX1032-NEXT:  ; %bb.1: ; %bb1
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    global_store_dword v[0:1], v0, off
+; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX1032-NEXT:  .LBB26_2: ; %bb2
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_preserve_condition_undef_flag:
+; GFX1064:       ; %bb.0: ; %bb0
+; GFX1064-NEXT:    s_clause 0x1
+; GFX1064-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GFX1064-NEXT:    s_load_dword s4, s[0:1], 0x24
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_cmp_ngt_f32_e64 s[0:1], s2, 0
+; GFX1064-NEXT:    v_cmp_nlt_f32_e64 s[2:3], s2, 1.0
+; GFX1064-NEXT:    v_cmp_nlt_f32_e64 s[4:5], s4, 1.0
+; GFX1064-NEXT:    s_or_b64 s[0:1], s[0:1], s[2:3]
+; GFX1064-NEXT:    s_or_b64 s[0:1], s[0:1], s[4:5]
+; GFX1064-NEXT:    s_and_b64 vcc, exec, s[0:1]
+; GFX1064-NEXT:    s_cbranch_vccnz .LBB26_2
+; GFX1064-NEXT:  ; %bb.1: ; %bb1
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    global_store_dword v[0:1], v0, off
+; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX1064-NEXT:  .LBB26_2: ; %bb2
+; GFX1064-NEXT:    s_endpgm
 bb0:
   %tmp = icmp sgt i32 %arg1, 4
   %undef = call i1 @llvm.amdgcn.class.f32(float undef, i32 undef)
@@ -556,12 +1532,82 @@ bb2:
   ret void
 }
 
-; GCN-LABEL: {{^}}test_invert_true_phi_cond_break_loop:
-; GFX1032: s_xor_b32 s{{[0-9]+}}, s{{[0-9]+}}, -1
-; GFX1032: s_or_b32 s{{[0-9]+}}, s{{[0-9]+}}, s{{[0-9]+}}
-; GFX1064: s_xor_b64 s[{{[0-9:]+}}], s[{{[0-9:]+}}], -1
-; GFX1064: s_or_b64 s[{{[0-9:]+}}], s[{{[0-9:]+}}], s[{{[0-9:]+}}]
 define amdgpu_kernel void @test_invert_true_phi_cond_break_loop(i32 %arg) #0 {
+; GFX1032-LABEL: test_invert_true_phi_cond_break_loop:
+; GFX1032:       ; %bb.0: ; %bb
+; GFX1032-NEXT:    s_load_dword s0, s[0:1], 0x24
+; GFX1032-NEXT:    ; implicit-def: $sgpr1
+; GFX1032-NEXT:    ; implicit-def: $sgpr2
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_subrev_nc_u32_e32 v0, s0, v0
+; GFX1032-NEXT:    s_mov_b32 s0, 0
+; GFX1032-NEXT:    s_branch .LBB27_2
+; GFX1032-NEXT:  .LBB27_1: ; %Flow
+; GFX1032-NEXT:    ; in Loop: Header=BB27_2 Depth=1
+; GFX1032-NEXT:    s_xor_b32 s3, s1, -1
+; GFX1032-NEXT:    s_add_i32 s2, s2, 1
+; GFX1032-NEXT:    s_and_b32 s3, exec_lo, s3
+; GFX1032-NEXT:    s_or_b32 s0, s3, s0
+; GFX1032-NEXT:    s_andn2_b32 exec_lo, exec_lo, s0
+; GFX1032-NEXT:    s_cbranch_execz .LBB27_4
+; GFX1032-NEXT:  .LBB27_2: ; %bb1
+; GFX1032-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1032-NEXT:    s_or_b32 s1, s1, exec_lo
+; GFX1032-NEXT:    s_cmp_gt_i32 s2, -1
+; GFX1032-NEXT:    s_cbranch_scc1 .LBB27_1
+; GFX1032-NEXT:  ; %bb.3: ; %bb4
+; GFX1032-NEXT:    ; in Loop: Header=BB27_2 Depth=1
+; GFX1032-NEXT:    global_load_dword v1, v[0:1], off glc dlc
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    s_andn2_b32 s1, s1, exec_lo
+; GFX1032-NEXT:    v_cmp_ge_i32_e32 vcc_lo, v0, v1
+; GFX1032-NEXT:    s_and_b32 s3, vcc_lo, exec_lo
+; GFX1032-NEXT:    s_or_b32 s1, s1, s3
+; GFX1032-NEXT:    s_branch .LBB27_1
+; GFX1032-NEXT:  .LBB27_4: ; %bb9
+; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s0
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 7
+; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX1032-NEXT:    ds_write_b32 v0, v0
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_invert_true_phi_cond_break_loop:
+; GFX1064:       ; %bb.0: ; %bb
+; GFX1064-NEXT:    s_load_dword s0, s[0:1], 0x24
+; GFX1064-NEXT:    ; implicit-def: $sgpr2_sgpr3
+; GFX1064-NEXT:    ; implicit-def: $sgpr4
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_subrev_nc_u32_e32 v0, s0, v0
+; GFX1064-NEXT:    s_mov_b64 s[0:1], 0
+; GFX1064-NEXT:    s_branch .LBB27_2
+; GFX1064-NEXT:  .LBB27_1: ; %Flow
+; GFX1064-NEXT:    ; in Loop: Header=BB27_2 Depth=1
+; GFX1064-NEXT:    s_xor_b64 s[6:7], s[2:3], -1
+; GFX1064-NEXT:    s_add_i32 s4, s4, 1
+; GFX1064-NEXT:    s_and_b64 s[6:7], exec, s[6:7]
+; GFX1064-NEXT:    s_or_b64 s[0:1], s[6:7], s[0:1]
+; GFX1064-NEXT:    s_andn2_b64 exec, exec, s[0:1]
+; GFX1064-NEXT:    s_cbranch_execz .LBB27_4
+; GFX1064-NEXT:  .LBB27_2: ; %bb1
+; GFX1064-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1064-NEXT:    s_or_b64 s[2:3], s[2:3], exec
+; GFX1064-NEXT:    s_cmp_gt_i32 s4, -1
+; GFX1064-NEXT:    s_cbranch_scc1 .LBB27_1
+; GFX1064-NEXT:  ; %bb.3: ; %bb4
+; GFX1064-NEXT:    ; in Loop: Header=BB27_2 Depth=1
+; GFX1064-NEXT:    global_load_dword v1, v[0:1], off glc dlc
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    s_andn2_b64 s[2:3], s[2:3], exec
+; GFX1064-NEXT:    v_cmp_ge_i32_e32 vcc, v0, v1
+; GFX1064-NEXT:    s_and_b64 s[6:7], vcc, exec
+; GFX1064-NEXT:    s_or_b64 s[2:3], s[2:3], s[6:7]
+; GFX1064-NEXT:    s_branch .LBB27_1
+; GFX1064-NEXT:  .LBB27_4: ; %bb9
+; GFX1064-NEXT:    s_or_b64 exec, exec, s[0:1]
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 7
+; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX1064-NEXT:    ds_write_b32 v0, v0
+; GFX1064-NEXT:    s_endpgm
 bb:
   %id = call i32 @llvm.amdgcn.workitem.id.x()
   %tmp = sub i32 %id, %arg
@@ -588,20 +1634,36 @@ bb9:                                              ; preds = %Flow
   ret void
 }
 
-; GCN-LABEL: {{^}}test_movrels_extract_neg_offset_vgpr:
-; GFX1032: v_cmp_eq_u32_e32 vcc_lo, 1, v{{[0-9]+}}
-; GFX1032: v_cndmask_b32_e64 v{{[0-9]+}}, 0, 1, vcc_lo
-; GFX1032: v_cmp_ne_u32_e32 vcc_lo, 2, v{{[0-9]+}}
-; GFX1032: v_cndmask_b32_e32 v{{[0-9]+}}, 2, v{{[0-9]+}}, vcc_lo
-; GFX1032: v_cmp_ne_u32_e32 vcc_lo, 3, v{{[0-9]+}}
-; GFX1032: v_cndmask_b32_e32 v{{[0-9]+}}, 3, v{{[0-9]+}}, vcc_lo
-; GFX1064: v_cmp_eq_u32_e32 vcc, 1, v{{[0-9]+}}
-; GFX1064: v_cndmask_b32_e64 v{{[0-9]+}}, 0, 1, vcc
-; GFX1064: v_cmp_ne_u32_e32 vcc, 2, v{{[0-9]+}}
-; GFX1064: v_cndmask_b32_e32 v{{[0-9]+}}, 2, v{{[0-9]+}}, vcc
-; GFX1064: v_cmp_ne_u32_e32 vcc, 3, v{{[0-9]+}}
-; GFX1064: v_cndmask_b32_e32 v{{[0-9]+}}, 3, v{{[0-9]+}}, vcc
 define amdgpu_kernel void @test_movrels_extract_neg_offset_vgpr(ptr addrspace(1) %out) #0 {
+; GFX1032-LABEL: test_movrels_extract_neg_offset_vgpr:
+; GFX1032:       ; %bb.0: ; %entry
+; GFX1032-NEXT:    v_add_nc_u32_e32 v0, 0xfffffe00, v0
+; GFX1032-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GFX1032-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 1, v0
+; GFX1032-NEXT:    v_cndmask_b32_e64 v1, 0, 1, vcc_lo
+; GFX1032-NEXT:    v_cmp_ne_u32_e32 vcc_lo, 2, v0
+; GFX1032-NEXT:    v_cndmask_b32_e32 v1, 2, v1, vcc_lo
+; GFX1032-NEXT:    v_cmp_ne_u32_e32 vcc_lo, 3, v0
+; GFX1032-NEXT:    v_cndmask_b32_e32 v0, 3, v1, vcc_lo
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    global_store_dword v2, v0, s[0:1]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_movrels_extract_neg_offset_vgpr:
+; GFX1064:       ; %bb.0: ; %entry
+; GFX1064-NEXT:    v_add_nc_u32_e32 v0, 0xfffffe00, v0
+; GFX1064-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GFX1064-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v0
+; GFX1064-NEXT:    v_cndmask_b32_e64 v1, 0, 1, vcc
+; GFX1064-NEXT:    v_cmp_ne_u32_e32 vcc, 2, v0
+; GFX1064-NEXT:    v_cndmask_b32_e32 v1, 2, v1, vcc
+; GFX1064-NEXT:    v_cmp_ne_u32_e32 vcc, 3, v0
+; GFX1064-NEXT:    v_cndmask_b32_e32 v0, 3, v1, vcc
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    global_store_dword v2, v0, s[0:1]
+; GFX1064-NEXT:    s_endpgm
 entry:
   %id = call i32 @llvm.amdgcn.workitem.id.x() #1
   %index = add i32 %id, -512
@@ -610,54 +1672,128 @@ entry:
   ret void
 }
 
-; GCN-LABEL: {{^}}test_set_inactive:
-; GFX1032: s_not_b32 exec_lo, exec_lo
-; GFX1032: v_mov_b32_e32 {{v[0-9]+}}, 42
-; GFX1032: s_not_b32 exec_lo, exec_lo
-; GFX1064: s_not_b64 exec, exec{{$}}
-; GFX1064: v_mov_b32_e32 {{v[0-9]+}}, 42
-; GFX1064: s_not_b64 exec, exec{{$}}
 define amdgpu_kernel void @test_set_inactive(ptr addrspace(1) %out, i32 %in) #0 {
+; GFX1032-LABEL: test_set_inactive:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_clause 0x1
+; GFX1032-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX1032-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1032-NEXT:    v_mov_b32_e32 v1, 0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_mov_b32_e32 v0, s4
+; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 42
+; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
+; GFX1032-NEXT:    global_store_dword v1, v0, s[2:3]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_set_inactive:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_clause 0x1
+; GFX1064-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX1064-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1064-NEXT:    v_mov_b32_e32 v1, 0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_mov_b32_e32 v0, s4
+; GFX1064-NEXT:    s_not_b64 exec, exec
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 42
+; GFX1064-NEXT:    s_not_b64 exec, exec
+; GFX1064-NEXT:    global_store_dword v1, v0, s[2:3]
+; GFX1064-NEXT:    s_endpgm
   %tmp = call i32 @llvm.amdgcn.set.inactive.i32(i32 %in, i32 42)
   store i32 %tmp, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}test_set_inactive_64:
-; GFX1032: s_not_b32 exec_lo, exec_lo
-; GFX1032: v_mov_b32_e32 {{v[0-9]+}}, 0
-; GFX1032: v_mov_b32_e32 {{v[0-9]+}}, 0
-; GFX1032: s_not_b32 exec_lo, exec_lo
-; GFX1064: s_not_b64 exec, exec{{$}}
-; GFX1064: v_mov_b32_e32 {{v[0-9]+}}, 0
-; GFX1064: v_mov_b32_e32 {{v[0-9]+}}, 0
-; GFX1064: s_not_b64 exec, exec{{$}}
 define amdgpu_kernel void @test_set_inactive_64(ptr addrspace(1) %out, i64 %in) #0 {
+; GFX1032-LABEL: test_set_inactive_64:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX1032-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_mov_b32_e32 v0, s2
+; GFX1032-NEXT:    v_mov_b32_e32 v1, s3
+; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    v_mov_b32_e32 v1, 0
+; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
+; GFX1032-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_set_inactive_64:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX1064-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_mov_b32_e32 v0, s2
+; GFX1064-NEXT:    v_mov_b32_e32 v1, s3
+; GFX1064-NEXT:    s_not_b64 exec, exec
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    v_mov_b32_e32 v1, 0
+; GFX1064-NEXT:    s_not_b64 exec, exec
+; GFX1064-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GFX1064-NEXT:    s_endpgm
   %tmp = call i64 @llvm.amdgcn.set.inactive.i64(i64 %in, i64 0)
   store i64 %tmp, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}test_kill_i1_terminator_float:
-; GFX1032: s_mov_b32 exec_lo, 0
-; GFX1064: s_mov_b64 exec, 0
 define amdgpu_ps void @test_kill_i1_terminator_float() #0 {
+; GFX1032-LABEL: test_kill_i1_terminator_float:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_andn2_b32 exec_lo, exec_lo, exec_lo
+; GFX1032-NEXT:    s_cbranch_scc0 .LBB31_1
+; GFX1032-NEXT:    s_endpgm
+; GFX1032-NEXT:  .LBB31_1:
+; GFX1032-NEXT:    s_mov_b32 exec_lo, 0
+; GFX1032-NEXT:    exp null off, off, off, off done vm
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_kill_i1_terminator_float:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_andn2_b64 exec, exec, exec
+; GFX1064-NEXT:    s_cbranch_scc0 .LBB31_1
+; GFX1064-NEXT:    s_endpgm
+; GFX1064-NEXT:  .LBB31_1:
+; GFX1064-NEXT:    s_mov_b64 exec, 0
+; GFX1064-NEXT:    exp null off, off, off, off done vm
+; GFX1064-NEXT:    s_endpgm
   call void @llvm.amdgcn.kill(i1 false)
   ret void
 }
 
-; GCN-LABEL: {{^}}test_kill_i1_terminator_i1:
-; GFX1032: s_mov_b32 [[LIVE:s[0-9]+]], exec_lo
-; GFX1032: s_or_b32 [[OR:s[0-9]+]],
-; GFX1032: s_xor_b32 [[KILL:s[0-9]+]], [[OR]], exec_lo
-; GFX1032: s_andn2_b32 [[MASK:s[0-9]+]], [[LIVE]], [[KILL]]
-; GFX1032: s_and_b32 exec_lo, exec_lo, [[MASK]]
-; GFX1064: s_mov_b64 [[LIVE:s\[[0-9:]+\]]], exec
-; GFX1064: s_or_b64 [[OR:s\[[0-9:]+\]]],
-; GFX1064: s_xor_b64 [[KILL:s\[[0-9:]+\]]], [[OR]], exec
-; GFX1064: s_andn2_b64 [[MASK:s\[[0-9:]+\]]], [[LIVE]], [[KILL]]
-; GFX1064: s_and_b64 exec, exec, [[MASK]]
 define amdgpu_gs void @test_kill_i1_terminator_i1(i32 %a, i32 %b, i32 %c, i32 %d) #0 {
+; GFX1032-LABEL: test_kill_i1_terminator_i1:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    v_cmp_lt_i32_e32 vcc_lo, v0, v1
+; GFX1032-NEXT:    v_cmp_lt_i32_e64 s0, v2, v3
+; GFX1032-NEXT:    s_mov_b32 s1, exec_lo
+; GFX1032-NEXT:    s_or_b32 s0, vcc_lo, s0
+; GFX1032-NEXT:    s_xor_b32 s0, s0, exec_lo
+; GFX1032-NEXT:    s_andn2_b32 s1, s1, s0
+; GFX1032-NEXT:    s_and_b32 exec_lo, exec_lo, s1
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    exp mrt0 off, off, off, off
+; GFX1032-NEXT:    s_endpgm
+; GFX1032-NEXT:  ; %bb.1:
+; GFX1032-NEXT:    s_mov_b32 exec_lo, 0
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_kill_i1_terminator_i1:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    v_cmp_lt_i32_e32 vcc, v0, v1
+; GFX1064-NEXT:    v_cmp_lt_i32_e64 s[0:1], v2, v3
+; GFX1064-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1064-NEXT:    s_or_b64 s[0:1], vcc, s[0:1]
+; GFX1064-NEXT:    s_xor_b64 s[0:1], s[0:1], exec
+; GFX1064-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[0:1]
+; GFX1064-NEXT:    s_and_b64 exec, exec, s[2:3]
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    exp mrt0 off, off, off, off
+; GFX1064-NEXT:    s_endpgm
+; GFX1064-NEXT:  ; %bb.1:
+; GFX1064-NEXT:    s_mov_b64 exec, 0
+; GFX1064-NEXT:    s_endpgm
   %c1 = icmp slt i32 %a, %b
   %c2 = icmp slt i32 %c, %d
   %x = or i1 %c1, %c2
@@ -666,11 +1802,70 @@ define amdgpu_gs void @test_kill_i1_terminator_i1(i32 %a, i32 %b, i32 %c, i32 %d
   ret void
 }
 
-; GCN-LABEL: {{^}}test_loop_vcc:
-; GFX1032: v_cmp_lt_f32_e32 vcc_lo,
-; GFX1064: v_cmp_lt_f32_e32 vcc,
-; GCN: s_cbranch_vccz
 define amdgpu_ps <4 x float> @test_loop_vcc(<4 x float> %in) #0 {
+; GFX1032-LABEL: test_loop_vcc:
+; GFX1032:       ; %bb.0: ; %entry
+; GFX1032-NEXT:    s_mov_b32 s0, exec_lo
+; GFX1032-NEXT:    s_wqm_b32 exec_lo, exec_lo
+; GFX1032-NEXT:    v_mov_b32_e32 v8, 0
+; GFX1032-NEXT:    s_branch .LBB33_2
+; GFX1032-NEXT:  .LBB33_1: ; %body
+; GFX1032-NEXT:    ; in Loop: Header=BB33_2 Depth=1
+; GFX1032-NEXT:    image_sample v[0:3], v4, s[0:7], s[0:3] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX1032-NEXT:    v_add_f32_e32 v8, 2.0, v8
+; GFX1032-NEXT:    s_cbranch_execz .LBB33_4
+; GFX1032-NEXT:  .LBB33_2: ; %loop
+; GFX1032-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1032-NEXT:    v_cmp_lt_f32_e32 vcc_lo, 0x40e00000, v8
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_mov_b32_e32 v7, v3
+; GFX1032-NEXT:    v_mov_b32_e32 v6, v2
+; GFX1032-NEXT:    v_mov_b32_e32 v5, v1
+; GFX1032-NEXT:    v_mov_b32_e32 v4, v0
+; GFX1032-NEXT:    s_cbranch_vccz .LBB33_1
+; GFX1032-NEXT:  ; %bb.3:
+; GFX1032-NEXT:    ; implicit-def: $vgpr0_vgpr1_vgpr2_vgpr3
+; GFX1032-NEXT:    ; implicit-def: $vgpr8
+; GFX1032-NEXT:  .LBB33_4: ; %break
+; GFX1032-NEXT:    s_and_b32 exec_lo, exec_lo, s0
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_mov_b32_e32 v0, v4
+; GFX1032-NEXT:    v_mov_b32_e32 v1, v5
+; GFX1032-NEXT:    v_mov_b32_e32 v2, v6
+; GFX1032-NEXT:    v_mov_b32_e32 v3, v7
+; GFX1032-NEXT:    ; return to shader part epilog
+;
+; GFX1064-LABEL: test_loop_vcc:
+; GFX1064:       ; %bb.0: ; %entry
+; GFX1064-NEXT:    s_mov_b64 s[0:1], exec
+; GFX1064-NEXT:    s_wqm_b64 exec, exec
+; GFX1064-NEXT:    v_mov_b32_e32 v8, 0
+; GFX1064-NEXT:    s_branch .LBB33_2
+; GFX1064-NEXT:  .LBB33_1: ; %body
+; GFX1064-NEXT:    ; in Loop: Header=BB33_2 Depth=1
+; GFX1064-NEXT:    image_sample v[0:3], v4, s[0:7], s[0:3] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX1064-NEXT:    v_add_f32_e32 v8, 2.0, v8
+; GFX1064-NEXT:    s_cbranch_execz .LBB33_4
+; GFX1064-NEXT:  .LBB33_2: ; %loop
+; GFX1064-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1064-NEXT:    v_cmp_lt_f32_e32 vcc, 0x40e00000, v8
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_mov_b32_e32 v7, v3
+; GFX1064-NEXT:    v_mov_b32_e32 v6, v2
+; GFX1064-NEXT:    v_mov_b32_e32 v5, v1
+; GFX1064-NEXT:    v_mov_b32_e32 v4, v0
+; GFX1064-NEXT:    s_cbranch_vccz .LBB33_1
+; GFX1064-NEXT:  ; %bb.3:
+; GFX1064-NEXT:    ; implicit-def: $vgpr0_vgpr1_vgpr2_vgpr3
+; GFX1064-NEXT:    ; implicit-def: $vgpr8
+; GFX1064-NEXT:  .LBB33_4: ; %break
+; GFX1064-NEXT:    s_and_b64 exec, exec, s[0:1]
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_mov_b32_e32 v0, v4
+; GFX1064-NEXT:    v_mov_b32_e32 v1, v5
+; GFX1064-NEXT:    v_mov_b32_e32 v2, v6
+; GFX1064-NEXT:    v_mov_b32_e32 v3, v7
+; GFX1064-NEXT:    ; return to shader part epilog
 entry:
   br label %loop
 
@@ -690,31 +1885,72 @@ break:
   ret <4 x float> %c.iv
 }
 
-; NOTE: llvm.amdgcn.wwm is deprecated, use llvm.amdgcn.strict.wwm instead.
-; GCN-LABEL: {{^}}test_wwm1:
-; GFX1032: s_or_saveexec_b32 [[SAVE:s[0-9]+]], -1
-; GFX1032: s_mov_b32 exec_lo, [[SAVE]]
-; GFX1064: s_or_saveexec_b64 [[SAVE:s\[[0-9]+:[0-9]+\]]], -1
-; GFX1064: s_mov_b64 exec, [[SAVE]]
 define amdgpu_ps float @test_wwm1(i32 inreg %idx0, i32 inreg %idx1, float %src0, float %src1) {
+; GFX1032-LABEL: test_wwm1:
+; GFX1032:       ; %bb.0: ; %main_body
+; GFX1032-NEXT:    s_or_saveexec_b32 s0, -1
+; GFX1032-NEXT:    v_mov_b32_e32 v2, v0
+; GFX1032-NEXT:    v_add_f32_e32 v2, v2, v1
+; GFX1032-NEXT:    s_mov_b32 exec_lo, s0
+; GFX1032-NEXT:    v_mov_b32_e32 v0, v2
+; GFX1032-NEXT:    ; return to shader part epilog
+;
+; GFX1064-LABEL: test_wwm1:
+; GFX1064:       ; %bb.0: ; %main_body
+; GFX1064-NEXT:    s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT:    v_mov_b32_e32 v2, v0
+; GFX1064-NEXT:    v_add_f32_e32 v2, v2, v1
+; GFX1064-NEXT:    s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT:    v_mov_b32_e32 v0, v2
+; GFX1064-NEXT:    ; return to shader part epilog
 main_body:
   %out = fadd float %src0, %src1
   %out.0 = call float @llvm.amdgcn.wwm.f32(float %out)
   ret float %out.0
 }
 
-; GCN-LABEL: {{^}}test_wwm2:
-; GFX1032: v_cmp_gt_u32_e32 vcc_lo, 16, v{{[0-9]+}}
-; GFX1032: s_and_saveexec_b32 [[SAVE1:s[0-9]+]], vcc_lo
-; GFX1032: s_or_saveexec_b32 [[SAVE2:s[0-9]+]], -1
-; GFX1032: s_mov_b32 exec_lo, [[SAVE2]]
-; GFX1032: s_or_b32 exec_lo, exec_lo, [[SAVE1]]
-; GFX1064: v_cmp_gt_u32_e32 vcc, 16, v{{[0-9]+}}
-; GFX1064: s_and_saveexec_b64 [[SAVE1:s\[[0-9:]+\]]], vcc{{$}}
-; GFX1064: s_or_saveexec_b64 [[SAVE2:s\[[0-9:]+\]]], -1
-; GFX1064: s_mov_b64 exec, [[SAVE2]]
-; GFX1064: s_or_b64 exec, exec, [[SAVE1]]
 define amdgpu_ps float @test_wwm2(i32 inreg %idx) {
+; GFX1032-LABEL: test_wwm2:
+; GFX1032:       ; %bb.0: ; %main_body
+; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v0, -1, 0
+; GFX1032-NEXT:    v_mbcnt_hi_u32_b32 v0, -1, v0
+; GFX1032-NEXT:    v_cmp_gt_u32_e32 vcc_lo, 16, v0
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    s_and_saveexec_b32 s1, vcc_lo
+; GFX1032-NEXT:    s_cbranch_execz .LBB35_2
+; GFX1032-NEXT:  ; %bb.1: ; %if
+; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
+; GFX1032-NEXT:    v_mov_b32_e32 v1, s0
+; GFX1032-NEXT:    buffer_load_dword v1, v1, s[0:3], 0 idxen
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_add_f32_e32 v2, v1, v1
+; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
+; GFX1032-NEXT:    v_mov_b32_e32 v0, v2
+; GFX1032-NEXT:    v_add_f32_e32 v0, v1, v0
+; GFX1032-NEXT:  .LBB35_2: ; %endif
+; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s1
+; GFX1032-NEXT:    ; return to shader part epilog
+;
+; GFX1064-LABEL: test_wwm2:
+; GFX1064:       ; %bb.0: ; %main_body
+; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v0, -1, 0
+; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v0, -1, v0
+; GFX1064-NEXT:    v_cmp_gt_u32_e32 vcc, 16, v0
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], vcc
+; GFX1064-NEXT:    s_cbranch_execz .LBB35_2
+; GFX1064-NEXT:  ; %bb.1: ; %if
+; GFX1064-NEXT:    s_or_saveexec_b64 s[4:5], -1
+; GFX1064-NEXT:    v_mov_b32_e32 v1, s0
+; GFX1064-NEXT:    buffer_load_dword v1, v1, s[0:3], 0 idxen
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_add_f32_e32 v2, v1, v1
+; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]
+; GFX1064-NEXT:    v_mov_b32_e32 v0, v2
+; GFX1064-NEXT:    v_add_f32_e32 v0, v1, v0
+; GFX1064-NEXT:  .LBB35_2: ; %endif
+; GFX1064-NEXT:    s_or_b64 exec, exec, s[2:3]
+; GFX1064-NEXT:    ; return to shader part epilog
 main_body:
   ; use mbcnt to make sure the branch is divergent
   %lo = call i32 @llvm.amdgcn.mbcnt.lo(i32 -1, i32 0)
@@ -734,30 +1970,72 @@ endif:
   ret float %out.2
 }
 
-; GCN-LABEL: {{^}}test_strict_wwm1:
-; GFX1032: s_or_saveexec_b32 [[SAVE:s[0-9]+]], -1
-; GFX1032: s_mov_b32 exec_lo, [[SAVE]]
-; GFX1064: s_or_saveexec_b64 [[SAVE:s\[[0-9]+:[0-9]+\]]], -1
-; GFX1064: s_mov_b64 exec, [[SAVE]]
 define amdgpu_ps float @test_strict_wwm1(i32 inreg %idx0, i32 inreg %idx1, float %src0, float %src1) {
+; GFX1032-LABEL: test_strict_wwm1:
+; GFX1032:       ; %bb.0: ; %main_body
+; GFX1032-NEXT:    s_or_saveexec_b32 s0, -1
+; GFX1032-NEXT:    v_mov_b32_e32 v2, v0
+; GFX1032-NEXT:    v_add_f32_e32 v2, v2, v1
+; GFX1032-NEXT:    s_mov_b32 exec_lo, s0
+; GFX1032-NEXT:    v_mov_b32_e32 v0, v2
+; GFX1032-NEXT:    ; return to shader part epilog
+;
+; GFX1064-LABEL: test_strict_wwm1:
+; GFX1064:       ; %bb.0: ; %main_body
+; GFX1064-NEXT:    s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT:    v_mov_b32_e32 v2, v0
+; GFX1064-NEXT:    v_add_f32_e32 v2, v2, v1
+; GFX1064-NEXT:    s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT:    v_mov_b32_e32 v0, v2
+; GFX1064-NEXT:    ; return to shader part epilog
 main_body:
   %out = fadd float %src0, %src1
   %out.0 = call float @llvm.amdgcn.strict.wwm.f32(float %out)
   ret float %out.0
 }
 
-; GCN-LABEL: {{^}}test_strict_wwm2:
-; GFX1032: v_cmp_gt_u32_e32 vcc_lo, 16, v{{[0-9]+}}
-; GFX1032: s_and_saveexec_b32 [[SAVE1:s[0-9]+]], vcc_lo
-; GFX1032: s_or_saveexec_b32 [[SAVE2:s[0-9]+]], -1
-; GFX1032: s_mov_b32 exec_lo, [[SAVE2]]
-; GFX1032: s_or_b32 exec_lo, exec_lo, [[SAVE1]]
-; GFX1064: v_cmp_gt_u32_e32 vcc, 16, v{{[0-9]+}}
-; GFX1064: s_and_saveexec_b64 [[SAVE1:s\[[0-9:]+\]]], vcc{{$}}
-; GFX1064: s_or_saveexec_b64 [[SAVE2:s\[[0-9:]+\]]], -1
-; GFX1064: s_mov_b64 exec, [[SAVE2]]
-; GFX1064: s_or_b64 exec, exec, [[SAVE1]]
 define amdgpu_ps float @test_strict_wwm2(i32 inreg %idx) {
+; GFX1032-LABEL: test_strict_wwm2:
+; GFX1032:       ; %bb.0: ; %main_body
+; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v0, -1, 0
+; GFX1032-NEXT:    v_mbcnt_hi_u32_b32 v0, -1, v0
+; GFX1032-NEXT:    v_cmp_gt_u32_e32 vcc_lo, 16, v0
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    s_and_saveexec_b32 s1, vcc_lo
+; GFX1032-NEXT:    s_cbranch_execz .LBB37_2
+; GFX1032-NEXT:  ; %bb.1: ; %if
+; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
+; GFX1032-NEXT:    v_mov_b32_e32 v1, s0
+; GFX1032-NEXT:    buffer_load_dword v1, v1, s[0:3], 0 idxen
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_add_f32_e32 v2, v1, v1
+; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
+; GFX1032-NEXT:    v_mov_b32_e32 v0, v2
+; GFX1032-NEXT:    v_add_f32_e32 v0, v1, v0
+; GFX1032-NEXT:  .LBB37_2: ; %endif
+; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s1
+; GFX1032-NEXT:    ; return to shader part epilog
+;
+; GFX1064-LABEL: test_strict_wwm2:
+; GFX1064:       ; %bb.0: ; %main_body
+; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v0, -1, 0
+; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v0, -1, v0
+; GFX1064-NEXT:    v_cmp_gt_u32_e32 vcc, 16, v0
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], vcc
+; GFX1064-NEXT:    s_cbranch_execz .LBB37_2
+; GFX1064-NEXT:  ; %bb.1: ; %if
+; GFX1064-NEXT:    s_or_saveexec_b64 s[4:5], -1
+; GFX1064-NEXT:    v_mov_b32_e32 v1, s0
+; GFX1064-NEXT:    buffer_load_dword v1, v1, s[0:3], 0 idxen
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_add_f32_e32 v2, v1, v1
+; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]
+; GFX1064-NEXT:    v_mov_b32_e32 v0, v2
+; GFX1064-NEXT:    v_add_f32_e32 v0, v1, v0
+; GFX1064-NEXT:  .LBB37_2: ; %endif
+; GFX1064-NEXT:    s_or_b64 exec, exec, s[2:3]
+; GFX1064-NEXT:    ; return to shader part epilog
 main_body:
   ; use mbcnt to make sure the branch is divergent
   %lo = call i32 @llvm.amdgcn.mbcnt.lo(i32 -1, i32 0)
@@ -778,14 +2056,34 @@ endif:
 }
 
 
-; GCN-LABEL: {{^}}test_wqm1:
-; GFX1032: s_mov_b32 [[ORIG:s[0-9]+]], exec_lo
-; GFX1032: s_wqm_b32 exec_lo, exec_lo
-; GFX1032: s_and_b32 exec_lo, exec_lo, [[ORIG]]
-; GFX1064: s_mov_b64 [[ORIG:s\[[0-9]+:[0-9]+\]]], exec{{$}}
-; GFX1064: s_wqm_b64 exec, exec{{$}}
-; GFX1064: s_and_b64 exec, exec, [[ORIG]]
 define amdgpu_ps <4 x float> @test_wqm1(i32 inreg, i32 inreg, i32 inreg, i32 inreg %m0, <8 x i32> inreg %rsrc, <4 x i32> inreg %sampler, <2 x float> %pos) #0 {
+; GFX1032-LABEL: test_wqm1:
+; GFX1032:       ; %bb.0: ; %main_body
+; GFX1032-NEXT:    s_mov_b32 s0, exec_lo
+; GFX1032-NEXT:    s_wqm_b32 exec_lo, exec_lo
+; GFX1032-NEXT:    s_mov_b32 m0, s3
+; GFX1032-NEXT:    v_interp_p1_f32_e32 v2, v0, attr0.x
+; GFX1032-NEXT:    v_interp_p1_f32_e32 v3, v0, attr0.y
+; GFX1032-NEXT:    v_interp_p2_f32_e32 v2, v1, attr0.x
+; GFX1032-NEXT:    v_interp_p2_f32_e32 v3, v1, attr0.y
+; GFX1032-NEXT:    s_and_b32 exec_lo, exec_lo, s0
+; GFX1032-NEXT:    image_sample v[0:3], v[2:3], s[4:11], s[12:15] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    ; return to shader part epilog
+;
+; GFX1064-LABEL: test_wqm1:
+; GFX1064:       ; %bb.0: ; %main_body
+; GFX1064-NEXT:    s_mov_b64 s[0:1], exec
+; GFX1064-NEXT:    s_wqm_b64 exec, exec
+; GFX1064-NEXT:    s_mov_b32 m0, s3
+; GFX1064-NEXT:    v_interp_p1_f32_e32 v2, v0, attr0.x
+; GFX1064-NEXT:    v_interp_p1_f32_e32 v3, v0, attr0.y
+; GFX1064-NEXT:    v_interp_p2_f32_e32 v2, v1, attr0.x
+; GFX1064-NEXT:    v_interp_p2_f32_e32 v3, v1, attr0.y
+; GFX1064-NEXT:    s_and_b64 exec, exec, s[0:1]
+; GFX1064-NEXT:    image_sample v[0:3], v[2:3], s[4:11], s[12:15] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    ; return to shader part epilog
 main_body:
   %inst23 = extractelement <2 x float> %pos, i32 0
   %inst24 = extractelement <2 x float> %pos, i32 1
@@ -797,12 +2095,36 @@ main_body:
   ret <4 x float> %tex
 }
 
-; GCN-LABEL: {{^}}test_wqm2:
-; GFX1032: s_wqm_b32 exec_lo, exec_lo
-; GFX1032: s_and_b32 exec_lo, exec_lo, s{{[0-9]+}}
-; GFX1064: s_wqm_b64 exec, exec{{$}}
-; GFX1064: s_and_b64 exec, exec, s[{{[0-9:]+}}]
 define amdgpu_ps float @test_wqm2(i32 inreg %idx0, i32 inreg %idx1) #0 {
+; GFX1032-LABEL: test_wqm2:
+; GFX1032:       ; %bb.0: ; %main_body
+; GFX1032-NEXT:    s_mov_b32 s2, exec_lo
+; GFX1032-NEXT:    s_wqm_b32 exec_lo, exec_lo
+; GFX1032-NEXT:    v_mov_b32_e32 v0, s0
+; GFX1032-NEXT:    v_mov_b32_e32 v1, s1
+; GFX1032-NEXT:    s_clause 0x1
+; GFX1032-NEXT:    buffer_load_dword v2, v0, s[0:3], 0 idxen
+; GFX1032-NEXT:    buffer_load_dword v3, v1, s[0:3], 0 idxen
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    v_add_f32_e32 v0, v2, v3
+; GFX1032-NEXT:    ; kill: def $vgpr0 killed $vgpr0 killed $exec
+; GFX1032-NEXT:    s_and_b32 exec_lo, exec_lo, s2
+; GFX1032-NEXT:    ; return to shader part epilog
+;
+; GFX1064-LABEL: test_wqm2:
+; GFX1064:       ; %bb.0: ; %main_body
+; GFX1064-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1064-NEXT:    s_wqm_b64 exec, exec
+; GFX1064-NEXT:    v_mov_b32_e32 v0, s0
+; GFX1064-NEXT:    v_mov_b32_e32 v1, s1
+; GFX1064-NEXT:    s_clause 0x1
+; GFX1064-NEXT:    buffer_load_dword v2, v0, s[0:3], 0 idxen
+; GFX1064-NEXT:    buffer_load_dword v3, v1, s[0:3], 0 idxen
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    v_add_f32_e32 v0, v2, v3
+; GFX1064-NEXT:    ; kill: def $vgpr0 killed $vgpr0 killed $exec
+; GFX1064-NEXT:    s_and_b64 exec, exec, s[2:3]
+; GFX1064-NEXT:    ; return to shader part epilog
 main_body:
   %src0 = call float @llvm.amdgcn.struct.buffer.load.f32(<4 x i32> undef, i32 %idx0, i32 0, i32 0, i32 0)
   %src1 = call float @llvm.amdgcn.struct.buffer.load.f32(<4 x i32> undef, i32 %idx1, i32 0, i32 0, i32 0)
@@ -813,74 +2135,154 @@ main_body:
   ret float %out.2
 }
 
-; GCN-LABEL: {{^}}test_intr_fcmp_i64:
-; GFX1032-DAG: v_mov_b32_e32 v[[V_HI:[0-9]+]], 0{{$}}
-; GFX1032-DAG: v_cmp_eq_f32_e64 s[[C_LO:[0-9]+]], {{s[0-9]+}}, |{{[vs][0-9]+}}|
-; GFX1032-DAG: v_mov_b32_e32 v[[V_LO:[0-9]+]], s[[C_LO]]
-; GFX1064:     v_cmp_eq_f32_e64 s[[[C_LO:[0-9]+]]:[[C_HI:[0-9]+]]], {{s[0-9]+}}, |{{[vs][0-9]+}}|
-; GFX1064-DAG: v_mov_b32_e32 v[[V_LO:[0-9]+]], s[[C_LO]]
-; GFX1064-DAG: v_mov_b32_e32 v[[V_HI:[0-9]+]], s[[C_HI]]
-; GCN:         store_dwordx2 v{{[0-9]+}}, v[[[V_LO]]:[[V_HI]]], s
 define amdgpu_kernel void @test_intr_fcmp_i64(ptr addrspace(1) %out, float %src, float %a) {
+; GFX1032-LABEL: test_intr_fcmp_i64:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX1032-NEXT:    v_mov_b32_e32 v1, 0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_cmp_eq_f32_e64 s2, s2, |s3|
+; GFX1032-NEXT:    v_mov_b32_e32 v0, s2
+; GFX1032-NEXT:    global_store_dwordx2 v1, v[0:1], s[0:1]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_intr_fcmp_i64:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX1064-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_cmp_eq_f32_e64 s[2:3], s2, |s3|
+; GFX1064-NEXT:    v_mov_b32_e32 v0, s2
+; GFX1064-NEXT:    v_mov_b32_e32 v1, s3
+; GFX1064-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GFX1064-NEXT:    s_endpgm
   %temp = call float @llvm.fabs.f32(float %a)
   %result = call i64 @llvm.amdgcn.fcmp.i64.f32(float %src, float %temp, i32 1)
   store i64 %result, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}test_intr_icmp_i64:
-; GFX1032-DAG: v_mov_b32_e32 v[[V_HI:[0-9]+]], 0{{$}}
-; GFX1032-DAG: v_cmp_eq_u32_e64 [[C_LO:vcc_lo|s[0-9]+]], 0x64, {{s[0-9]+}}
-; GFX1032-DAG: v_mov_b32_e32 v[[V_LO:[0-9]+]], [[C_LO]]
-; GFX1064:     v_cmp_eq_u32_e64 s[[[C_LO:[0-9]+]]:[[C_HI:[0-9]+]]], 0x64, {{s[0-9]+}}
-; GFX1064-DAG: v_mov_b32_e32 v[[V_LO:[0-9]+]], s[[C_LO]]
-; GFX1064-DAG: v_mov_b32_e32 v[[V_HI:[0-9]+]], s[[C_HI]]
-; GCN:         store_dwordx2 v{{[0-9]+}}, v[[[V_LO]]:[[V_HI]]], s
 define amdgpu_kernel void @test_intr_icmp_i64(ptr addrspace(1) %out, i32 %src) {
+; GFX1032-LABEL: test_intr_icmp_i64:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_clause 0x1
+; GFX1032-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX1032-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1032-NEXT:    v_mov_b32_e32 v1, 0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_cmp_eq_u32_e64 s0, 0x64, s4
+; GFX1032-NEXT:    v_mov_b32_e32 v0, s0
+; GFX1032-NEXT:    global_store_dwordx2 v1, v[0:1], s[2:3]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_intr_icmp_i64:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_clause 0x1
+; GFX1064-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX1064-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1064-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_cmp_eq_u32_e64 s[0:1], 0x64, s4
+; GFX1064-NEXT:    v_mov_b32_e32 v0, s0
+; GFX1064-NEXT:    v_mov_b32_e32 v1, s1
+; GFX1064-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX1064-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %src, i32 100, i32 32)
   store i64 %result, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}test_intr_fcmp_i32:
-; GFX1032-DAG: v_cmp_eq_f32_e64 s[[C_LO:[0-9]+]], {{s[0-9]+}}, |{{[vs][0-9]+}}|
-; GFX1032-DAG: v_mov_b32_e32 v[[V_LO:[0-9]+]], s[[C_LO]]
-; GFX1064:     v_cmp_eq_f32_e64 s[[[C_LO:[0-9]+]]:[[C_HI:[0-9]+]]], {{s[0-9]+}}, |{{[vs][0-9]+}}|
-; GFX1064-DAG: v_mov_b32_e32 v[[V_LO:[0-9]+]], s[[C_LO]]
-; GCN:         store_dword v{{[0-9]+}}, v[[V_LO]], s
 define amdgpu_kernel void @test_intr_fcmp_i32(ptr addrspace(1) %out, float %src, float %a) {
+; GFX1032-LABEL: test_intr_fcmp_i32:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_cmp_eq_f32_e64 s2, s2, |s3|
+; GFX1032-NEXT:    v_mov_b32_e32 v1, s2
+; GFX1032-NEXT:    global_store_dword v0, v1, s[0:1]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_intr_fcmp_i32:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_cmp_eq_f32_e64 s[2:3], s2, |s3|
+; GFX1064-NEXT:    v_mov_b32_e32 v1, s2
+; GFX1064-NEXT:    global_store_dword v0, v1, s[0:1]
+; GFX1064-NEXT:    s_endpgm
   %temp = call float @llvm.fabs.f32(float %a)
   %result = call i32 @llvm.amdgcn.fcmp.i32.f32(float %src, float %temp, i32 1)
   store i32 %result, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}test_intr_icmp_i32:
-; GFX1032-DAG: v_cmp_eq_u32_e64 s[[C_LO:[0-9]+]], 0x64, {{s[0-9]+}}
-; GFX1032-DAG: v_mov_b32_e32 v[[V_LO:[0-9]+]], s[[C_LO]]{{$}}
-; GFX1064:     v_cmp_eq_u32_e64 s[[[C_LO:[0-9]+]]:{{[0-9]+}}], 0x64, {{s[0-9]+}}
-; GFX1064-DAG: v_mov_b32_e32 v[[V_LO:[0-9]+]], s[[C_LO]]{{$}}
-; GCN:         store_dword v{{[0-9]+}}, v[[V_LO]], s
 define amdgpu_kernel void @test_intr_icmp_i32(ptr addrspace(1) %out, i32 %src) {
+; GFX1032-LABEL: test_intr_icmp_i32:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_clause 0x1
+; GFX1032-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX1032-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_cmp_eq_u32_e64 s0, 0x64, s4
+; GFX1032-NEXT:    v_mov_b32_e32 v1, s0
+; GFX1032-NEXT:    global_store_dword v0, v1, s[2:3]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_intr_icmp_i32:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_clause 0x1
+; GFX1064-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX1064-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_cmp_eq_u32_e64 s[0:1], 0x64, s4
+; GFX1064-NEXT:    v_mov_b32_e32 v1, s0
+; GFX1064-NEXT:    global_store_dword v0, v1, s[2:3]
+; GFX1064-NEXT:    s_endpgm
   %result = call i32 @llvm.amdgcn.icmp.i32.i32(i32 %src, i32 100, i32 32)
   store i32 %result, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}test_wqm_vote:
-; GFX1032: v_cmp_neq_f32_e32 vcc_lo, 0
-; GFX1032: s_mov_b32 [[LIVE:s[0-9]+]], exec_lo
-; GFX1032: s_wqm_b32 [[WQM:s[0-9]+]], vcc_lo
-; GFX1032: s_xor_b32 [[KILL:s[0-9]+]], [[WQM]], exec_lo
-; GFX1032: s_andn2_b32 [[MASK:s[0-9]+]], [[LIVE]], [[KILL]]
-; GFX1032: s_and_b32 exec_lo, exec_lo, [[MASK]]
-; GFX1064: v_cmp_neq_f32_e32 vcc, 0
-; GFX1064: s_mov_b64 [[LIVE:s\[[0-9:]+\]]], exec
-; GFX1064: s_wqm_b64 [[WQM:s\[[0-9:]+\]]], vcc
-; GFX1064: s_xor_b64 [[KILL:s\[[0-9:]+\]]], [[WQM]], exec
-; GFX1064: s_andn2_b64 [[MASK:s\[[0-9:]+\]]], [[LIVE]], [[KILL]]
-; GFX1064: s_and_b64 exec, exec, [[MASK]]
 define amdgpu_ps void @test_wqm_vote(float %a) {
+; GFX1032-LABEL: test_wqm_vote:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    v_cmp_neq_f32_e32 vcc_lo, 0, v0
+; GFX1032-NEXT:    s_mov_b32 s0, exec_lo
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    s_wqm_b32 s1, vcc_lo
+; GFX1032-NEXT:    s_xor_b32 s1, s1, exec_lo
+; GFX1032-NEXT:    s_andn2_b32 s0, s0, s1
+; GFX1032-NEXT:    s_cbranch_scc0 .LBB44_2
+; GFX1032-NEXT:  ; %bb.1:
+; GFX1032-NEXT:    s_and_b32 exec_lo, exec_lo, s0
+; GFX1032-NEXT:    exp mrt0 off, off, off, off
+; GFX1032-NEXT:    s_endpgm
+; GFX1032-NEXT:  .LBB44_2:
+; GFX1032-NEXT:    s_mov_b32 exec_lo, 0
+; GFX1032-NEXT:    exp null off, off, off, off done vm
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_wqm_vote:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    v_cmp_neq_f32_e32 vcc, 0, v0
+; GFX1064-NEXT:    s_mov_b64 s[0:1], exec
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    s_wqm_b64 s[2:3], vcc
+; GFX1064-NEXT:    s_xor_b64 s[2:3], s[2:3], exec
+; GFX1064-NEXT:    s_andn2_b64 s[0:1], s[0:1], s[2:3]
+; GFX1064-NEXT:    s_cbranch_scc0 .LBB44_2
+; GFX1064-NEXT:  ; %bb.1:
+; GFX1064-NEXT:    s_and_b64 exec, exec, s[0:1]
+; GFX1064-NEXT:    exp mrt0 off, off, off, off
+; GFX1064-NEXT:    s_endpgm
+; GFX1064-NEXT:  .LBB44_2:
+; GFX1064-NEXT:    s_mov_b64 exec, 0
+; GFX1064-NEXT:    exp null off, off, off, off done vm
+; GFX1064-NEXT:    s_endpgm
   %c1 = fcmp une float %a, 0.0
   %c2 = call i1 @llvm.amdgcn.wqm.vote(i1 %c1)
   call void @llvm.amdgcn.kill(i1 %c2)
@@ -888,10 +2290,36 @@ define amdgpu_ps void @test_wqm_vote(float %a) {
   ret void
 }
 
-; GCN-LABEL: {{^}}test_branch_true:
-; GFX1032: s_mov_b32 vcc_lo, exec_lo
-; GFX1064: s_mov_b64 vcc, exec
 define amdgpu_kernel void @test_branch_true() #2 {
+; GFX1032-LABEL: test_branch_true:
+; GFX1032:       ; %bb.0: ; %entry
+; GFX1032-NEXT:    s_mov_b32 vcc_lo, exec_lo
+; GFX1032-NEXT:    s_cbranch_execnz .LBB45_2
+; GFX1032-NEXT:  ; %bb.1: ; %for.body.lr.ph
+; GFX1032-NEXT:    s_branch .LBB45_3
+; GFX1032-NEXT:  .LBB45_2: ; %Flow
+; GFX1032-NEXT:    s_branch .LBB45_5
+; GFX1032-NEXT:  .LBB45_3: ; %for.body
+; GFX1032-NEXT:    s_mov_b32 vcc_lo, 0
+; GFX1032-NEXT:  ; %bb.4: ; %for.end.loopexit
+; GFX1032-NEXT:    s_branch .LBB45_2
+; GFX1032-NEXT:  .LBB45_5: ; %for.end
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_branch_true:
+; GFX1064:       ; %bb.0: ; %entry
+; GFX1064-NEXT:    s_mov_b64 vcc, exec
+; GFX1064-NEXT:    s_cbranch_execnz .LBB45_2
+; GFX1064-NEXT:  ; %bb.1: ; %for.body.lr.ph
+; GFX1064-NEXT:    s_branch .LBB45_3
+; GFX1064-NEXT:  .LBB45_2: ; %Flow
+; GFX1064-NEXT:    s_branch .LBB45_5
+; GFX1064-NEXT:  .LBB45_3: ; %for.body
+; GFX1064-NEXT:    s_mov_b64 vcc, 0
+; GFX1064-NEXT:  ; %bb.4: ; %for.end.loopexit
+; GFX1064-NEXT:    s_branch .LBB45_2
+; GFX1064-NEXT:  .LBB45_5: ; %for.end
+; GFX1064-NEXT:    s_endpgm
 entry:
   br i1 true, label %for.end, label %for.body.lr.ph
 
@@ -905,23 +2333,64 @@ for.end:                                          ; preds = %for.body, %entry
   ret void
 }
 
-; GCN-LABEL: {{^}}test_ps_live:
-; GFX1032: s_mov_b32 [[C:s[0-9]+]], exec_lo
-; GFX1064: s_mov_b64 [[C:s\[[0-9:]+\]]], exec{{$}}
-; GCN: v_cndmask_b32_e64 v{{[0-9]+}}, 0, 1, [[C]]
 define amdgpu_ps float @test_ps_live() #0 {
+; GFX1032-LABEL: test_ps_live:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_mov_b32 s0, exec_lo
+; GFX1032-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
+; GFX1032-NEXT:    ; return to shader part epilog
+;
+; GFX1064-LABEL: test_ps_live:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_mov_b64 s[0:1], exec
+; GFX1064-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[0:1]
+; GFX1064-NEXT:    ; return to shader part epilog
   %live = call i1 @llvm.amdgcn.ps.live()
   %live.32 = zext i1 %live to i32
   %r = bitcast i32 %live.32 to float
   ret float %r
 }
 
-; GCN-LABEL: {{^}}test_vccnz_ifcvt_triangle64:
-; GFX1032: v_cmp_neq_f64_e64 [[C:s[0-9]+]], s[{{[0-9:]+}}], 1.0
-; GFX1032: s_and_b32 vcc_lo, exec_lo, [[C]]
-; GFX1064: v_cmp_neq_f64_e64 [[C:s\[[0-9:]+\]]], s[{{[0-9:]+}}], 1.0
-; GFX1064: s_and_b64 vcc, exec, [[C]]
 define amdgpu_kernel void @test_vccnz_ifcvt_triangle64(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 {
+; GFX1032-LABEL: test_vccnz_ifcvt_triangle64:
+; GFX1032:       ; %bb.0: ; %entry
+; GFX1032-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    s_load_dwordx2 s[2:3], s[2:3], 0x0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_cmp_neq_f64_e64 s4, s[2:3], 1.0
+; GFX1032-NEXT:    s_and_b32 vcc_lo, exec_lo, s4
+; GFX1032-NEXT:    s_cbranch_vccnz .LBB47_2
+; GFX1032-NEXT:  ; %bb.1: ; %if
+; GFX1032-NEXT:    v_add_f64 v[0:1], s[2:3], s[2:3]
+; GFX1032-NEXT:    s_branch .LBB47_3
+; GFX1032-NEXT:  .LBB47_2:
+; GFX1032-NEXT:    v_mov_b32_e32 v0, s2
+; GFX1032-NEXT:    v_mov_b32_e32 v1, s3
+; GFX1032-NEXT:  .LBB47_3: ; %endif
+; GFX1032-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1032-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: test_vccnz_ifcvt_triangle64:
+; GFX1064:       ; %bb.0: ; %entry
+; GFX1064-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    s_load_dwordx2 s[2:3], s[2:3], 0x0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_cmp_neq_f64_e64 s[4:5], s[2:3], 1.0
+; GFX1064-NEXT:    s_and_b64 vcc, exec, s[4:5]
+; GFX1064-NEXT:    s_cbranch_vccnz .LBB47_2
+; GFX1064-NEXT:  ; %bb.1: ; %if
+; GFX1064-NEXT:    v_add_f64 v[0:1], s[2:3], s[2:3]
+; GFX1064-NEXT:    s_branch .LBB47_3
+; GFX1064-NEXT:  .LBB47_2:
+; GFX1064-NEXT:    v_mov_b32_e32 v0, s2
+; GFX1064-NEXT:    v_mov_b32_e32 v1, s3
+; GFX1064-NEXT:  .LBB47_3: ; %endif
+; GFX1064-NEXT:    v_mov_b32_e32 v2, 0
+; GFX1064-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GFX1064-NEXT:    s_endpgm
 entry:
   %v = load double, ptr addrspace(1) %in
   %cc = fcmp oeq double %v, 1.000000e+00
@@ -937,10 +2406,21 @@ endif:
   ret void
 }
 
-; GCN-LABEL: {{^}}test_vgprblocks_w32_attr:
-; Test that the wave size can be overridden in function attributes and that the block size is correct as a result
-; GFX10DEFWAVE: ; VGPRBlocks: 1
 define amdgpu_gs float @test_vgprblocks_w32_attr(float %a, float %b, float %c, float %d, float %e,
+; GCN-LABEL: test_vgprblocks_w32_attr:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v1
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v2
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v3
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v4
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v5
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v6
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v7
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v8
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v9
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v10
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v11
+; GCN-NEXT:    ; return to shader part epilog
                                         float %f, float %g, float %h, float %i, float %j, float %k, float %l) #3 {
 main_body:
   %s = fadd float %a, %b
@@ -957,10 +2437,21 @@ main_body:
   ret float %s.10
 }
 
-; GCN-LABEL: {{^}}test_vgprblocks_w64_attr:
-; Test that the wave size can be overridden in function attributes and that the block size is correct as a result
-; GFX10DEFWAVE: ; VGPRBlocks: 2
 define amdgpu_gs float @test_vgprblocks_w64_attr(float %a, float %b, float %c, float %d, float %e,
+; GCN-LABEL: test_vgprblocks_w64_attr:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v1
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v2
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v3
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v4
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v5
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v6
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v7
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v8
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v9
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v10
+; GCN-NEXT:    v_add_f32_e32 v0, v0, v11
+; GCN-NEXT:    ; return to shader part epilog
                                         float %f, float %g, float %h, float %i, float %j, float %k, float %l) #4 {
 main_body:
   %s = fadd float %a, %b
@@ -977,10 +2468,80 @@ main_body:
   ret float %s.10
 }
 
-; GCN-LABEL: {{^}}icmp64:
-; GFX1032: v_cmp_eq_u32_e32 vcc_lo, 0, v
-; GFX1064: v_cmp_eq_u32_e32 vcc, 0, v
 define amdgpu_kernel void @icmp64(i32 %n, i32 %s) {
+; GFX1032-LABEL: icmp64:
+; GFX1032:       ; %bb.0: ; %entry
+; GFX1032-NEXT:    s_load_dword s0, s[0:1], 0x28
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_cvt_f32_u32_e32 v1, s0
+; GFX1032-NEXT:    s_sub_i32 s1, 0, s0
+; GFX1032-NEXT:    v_rcp_iflag_f32_e32 v1, v1
+; GFX1032-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v1
+; GFX1032-NEXT:    v_cvt_u32_f32_e32 v1, v1
+; GFX1032-NEXT:    v_mul_lo_u32 v2, s1, v1
+; GFX1032-NEXT:    s_ff1_i32_b32 s1, 0x80000000
+; GFX1032-NEXT:    s_add_i32 s1, s1, 32
+; GFX1032-NEXT:    v_mul_hi_u32 v2, v1, v2
+; GFX1032-NEXT:    v_add_nc_u32_e32 v1, v1, v2
+; GFX1032-NEXT:    v_mul_hi_u32 v1, v0, v1
+; GFX1032-NEXT:    v_mul_lo_u32 v1, v1, s0
+; GFX1032-NEXT:    v_sub_nc_u32_e32 v0, v0, v1
+; GFX1032-NEXT:    v_subrev_nc_u32_e32 v1, s0, v0
+; GFX1032-NEXT:    v_cmp_le_u32_e32 vcc_lo, s0, v0
+; GFX1032-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc_lo
+; GFX1032-NEXT:    v_subrev_nc_u32_e32 v1, s0, v0
+; GFX1032-NEXT:    v_cmp_le_u32_e32 vcc_lo, s0, v0
+; GFX1032-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc_lo
+; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
+; GFX1032-NEXT:    v_alignbit_b32 v0, 0, vcc_lo, 1
+; GFX1032-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX1032-NEXT:    s_ff1_i32_b32 s0, s0
+; GFX1032-NEXT:    s_min_u32 s0, s0, s1
+; GFX1032-NEXT:    s_cmp_gt_u32 s0, 9
+; GFX1032-NEXT:    s_cselect_b32 s0, -1, 0
+; GFX1032-NEXT:    s_and_b32 s0, vcc_lo, s0
+; GFX1032-NEXT:    s_and_saveexec_b32 s1, s0
+; GFX1032-NEXT:  ; %bb.1: ; %if.then
+; GFX1032-NEXT:    ; divergent unreachable
+; GFX1032-NEXT:  ; %bb.2: ; %UnifiedReturnBlock
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: icmp64:
+; GFX1064:       ; %bb.0: ; %entry
+; GFX1064-NEXT:    s_load_dword s0, s[0:1], 0x28
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_cvt_f32_u32_e32 v1, s0
+; GFX1064-NEXT:    s_sub_i32 s1, 0, s0
+; GFX1064-NEXT:    v_rcp_iflag_f32_e32 v1, v1
+; GFX1064-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v1
+; GFX1064-NEXT:    v_cvt_u32_f32_e32 v1, v1
+; GFX1064-NEXT:    v_mul_lo_u32 v2, s1, v1
+; GFX1064-NEXT:    v_mul_hi_u32 v2, v1, v2
+; GFX1064-NEXT:    v_add_nc_u32_e32 v1, v1, v2
+; GFX1064-NEXT:    v_mul_hi_u32 v1, v0, v1
+; GFX1064-NEXT:    v_mul_lo_u32 v1, v1, s0
+; GFX1064-NEXT:    v_sub_nc_u32_e32 v0, v0, v1
+; GFX1064-NEXT:    v_subrev_nc_u32_e32 v1, s0, v0
+; GFX1064-NEXT:    v_cmp_le_u32_e32 vcc, s0, v0
+; GFX1064-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
+; GFX1064-NEXT:    v_subrev_nc_u32_e32 v1, s0, v0
+; GFX1064-NEXT:    v_cmp_le_u32_e32 vcc, s0, v0
+; GFX1064-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
+; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX1064-NEXT:    s_lshr_b64 s[0:1], vcc, 1
+; GFX1064-NEXT:    s_bitset1_b32 s1, 31
+; GFX1064-NEXT:    s_ff1_i32_b32 s0, s0
+; GFX1064-NEXT:    s_ff1_i32_b32 s1, s1
+; GFX1064-NEXT:    s_add_i32 s1, s1, 32
+; GFX1064-NEXT:    s_min_u32 s0, s0, s1
+; GFX1064-NEXT:    s_cmp_gt_u32 s0, 9
+; GFX1064-NEXT:    s_cselect_b64 s[0:1], -1, 0
+; GFX1064-NEXT:    s_and_b64 s[0:1], vcc, s[0:1]
+; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], s[0:1]
+; GFX1064-NEXT:  ; %bb.1: ; %if.then
+; GFX1064-NEXT:    ; divergent unreachable
+; GFX1064-NEXT:  ; %bb.2: ; %UnifiedReturnBlock
+; GFX1064-NEXT:    s_endpgm
 entry:
   %id = tail call i32 @llvm.amdgcn.workitem.id.x()
   %mul4 = mul nsw i32 %s, %n
@@ -1008,10 +2569,76 @@ if.end2:                                          ; preds = %if.end
   ret void
 }
 
-; GCN-LABEL: {{^}}fcmp64:
-; GFX1032: v_cmp_eq_f32_e32 vcc_lo, 0, v
-; GFX1064: v_cmp_eq_f32_e32 vcc, 0, v
 define amdgpu_kernel void @fcmp64(float %n, float %s) {
+; GFX1032-LABEL: fcmp64:
+; GFX1032:       ; %bb.0: ; %entry
+; GFX1032-NEXT:    s_load_dword s0, s[0:1], 0x28
+; GFX1032-NEXT:    v_cvt_f32_u32_e32 v0, v0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_div_scale_f32 v1, s1, s0, s0, v0
+; GFX1032-NEXT:    v_div_scale_f32 v4, vcc_lo, v0, s0, v0
+; GFX1032-NEXT:    s_ff1_i32_b32 s1, 0x80000000
+; GFX1032-NEXT:    v_rcp_f32_e32 v2, v1
+; GFX1032-NEXT:    s_add_i32 s1, s1, 32
+; GFX1032-NEXT:    v_fma_f32 v3, -v1, v2, 1.0
+; GFX1032-NEXT:    v_fmac_f32_e32 v2, v3, v2
+; GFX1032-NEXT:    v_mul_f32_e32 v3, v4, v2
+; GFX1032-NEXT:    v_fma_f32 v5, -v1, v3, v4
+; GFX1032-NEXT:    v_fmac_f32_e32 v3, v5, v2
+; GFX1032-NEXT:    v_fma_f32 v1, -v1, v3, v4
+; GFX1032-NEXT:    v_div_fmas_f32 v1, v1, v2, v3
+; GFX1032-NEXT:    v_div_fixup_f32 v1, v1, s0, v0
+; GFX1032-NEXT:    v_trunc_f32_e32 v1, v1
+; GFX1032-NEXT:    v_fma_f32 v0, -v1, s0, v0
+; GFX1032-NEXT:    v_cmp_eq_f32_e32 vcc_lo, 0, v0
+; GFX1032-NEXT:    v_alignbit_b32 v1, 0, vcc_lo, 1
+; GFX1032-NEXT:    v_cmp_nlg_f32_e32 vcc_lo, 0, v0
+; GFX1032-NEXT:    v_readfirstlane_b32 s0, v1
+; GFX1032-NEXT:    s_ff1_i32_b32 s0, s0
+; GFX1032-NEXT:    s_min_u32 s0, s0, s1
+; GFX1032-NEXT:    s_cmp_gt_u32 s0, 9
+; GFX1032-NEXT:    s_cselect_b32 s0, -1, 0
+; GFX1032-NEXT:    s_and_b32 s0, vcc_lo, s0
+; GFX1032-NEXT:    s_and_saveexec_b32 s1, s0
+; GFX1032-NEXT:  ; %bb.1: ; %if.then
+; GFX1032-NEXT:    ; divergent unreachable
+; GFX1032-NEXT:  ; %bb.2: ; %UnifiedReturnBlock
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: fcmp64:
+; GFX1064:       ; %bb.0: ; %entry
+; GFX1064-NEXT:    s_load_dword s2, s[0:1], 0x28
+; GFX1064-NEXT:    v_cvt_f32_u32_e32 v0, v0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_div_scale_f32 v1, s[0:1], s2, s2, v0
+; GFX1064-NEXT:    v_div_scale_f32 v4, vcc, v0, s2, v0
+; GFX1064-NEXT:    v_rcp_f32_e32 v2, v1
+; GFX1064-NEXT:    v_fma_f32 v3, -v1, v2, 1.0
+; GFX1064-NEXT:    v_fmac_f32_e32 v2, v3, v2
+; GFX1064-NEXT:    v_mul_f32_e32 v3, v4, v2
+; GFX1064-NEXT:    v_fma_f32 v5, -v1, v3, v4
+; GFX1064-NEXT:    v_fmac_f32_e32 v3, v5, v2
+; GFX1064-NEXT:    v_fma_f32 v1, -v1, v3, v4
+; GFX1064-NEXT:    v_div_fmas_f32 v1, v1, v2, v3
+; GFX1064-NEXT:    v_div_fixup_f32 v1, v1, s2, v0
+; GFX1064-NEXT:    v_trunc_f32_e32 v1, v1
+; GFX1064-NEXT:    v_fma_f32 v0, -v1, s2, v0
+; GFX1064-NEXT:    v_cmp_eq_f32_e32 vcc, 0, v0
+; GFX1064-NEXT:    s_lshr_b64 s[0:1], vcc, 1
+; GFX1064-NEXT:    v_cmp_nlg_f32_e32 vcc, 0, v0
+; GFX1064-NEXT:    s_bitset1_b32 s1, 31
+; GFX1064-NEXT:    s_ff1_i32_b32 s0, s0
+; GFX1064-NEXT:    s_ff1_i32_b32 s1, s1
+; GFX1064-NEXT:    s_add_i32 s1, s1, 32
+; GFX1064-NEXT:    s_min_u32 s0, s0, s1
+; GFX1064-NEXT:    s_cmp_gt_u32 s0, 9
+; GFX1064-NEXT:    s_cselect_b64 s[0:1], -1, 0
+; GFX1064-NEXT:    s_and_b64 s[0:1], vcc, s[0:1]
+; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], s[0:1]
+; GFX1064-NEXT:  ; %bb.1: ; %if.then
+; GFX1064-NEXT:    ; divergent unreachable
+; GFX1064-NEXT:  ; %bb.2: ; %UnifiedReturnBlock
+; GFX1064-NEXT:    s_endpgm
 entry:
   %id = tail call i32 @llvm.amdgcn.workitem.id.x()
   %id.f = uitofp i32 %id to float
@@ -1040,10 +2667,74 @@ if.end2:                                          ; preds = %if.end
   ret void
 }
 
-; GCN-LABEL: {{^}}icmp32:
-; GFX1032: v_cmp_eq_u32_e32 vcc_lo, 0, v
-; GFX1064: v_cmp_eq_u32_e32 vcc, 0, v
 define amdgpu_kernel void @icmp32(i32 %n, i32 %s) {
+; GFX1032-LABEL: icmp32:
+; GFX1032:       ; %bb.0: ; %entry
+; GFX1032-NEXT:    s_load_dword s0, s[0:1], 0x28
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_cvt_f32_u32_e32 v1, s0
+; GFX1032-NEXT:    s_sub_i32 s1, 0, s0
+; GFX1032-NEXT:    v_rcp_iflag_f32_e32 v1, v1
+; GFX1032-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v1
+; GFX1032-NEXT:    v_cvt_u32_f32_e32 v1, v1
+; GFX1032-NEXT:    v_mul_lo_u32 v2, s1, v1
+; GFX1032-NEXT:    v_mul_hi_u32 v2, v1, v2
+; GFX1032-NEXT:    v_add_nc_u32_e32 v1, v1, v2
+; GFX1032-NEXT:    v_mul_hi_u32 v1, v0, v1
+; GFX1032-NEXT:    v_mul_lo_u32 v1, v1, s0
+; GFX1032-NEXT:    v_sub_nc_u32_e32 v0, v0, v1
+; GFX1032-NEXT:    v_subrev_nc_u32_e32 v1, s0, v0
+; GFX1032-NEXT:    v_cmp_le_u32_e32 vcc_lo, s0, v0
+; GFX1032-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc_lo
+; GFX1032-NEXT:    v_subrev_nc_u32_e32 v1, s0, v0
+; GFX1032-NEXT:    v_cmp_le_u32_e32 vcc_lo, s0, v0
+; GFX1032-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc_lo
+; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
+; GFX1032-NEXT:    s_lshr_b32 s0, vcc_lo, 1
+; GFX1032-NEXT:    s_bitset1_b32 s0, 31
+; GFX1032-NEXT:    s_ff1_i32_b32 s0, s0
+; GFX1032-NEXT:    s_cmp_gt_u32 s0, 9
+; GFX1032-NEXT:    s_cselect_b32 s0, -1, 0
+; GFX1032-NEXT:    s_and_b32 s0, vcc_lo, s0
+; GFX1032-NEXT:    s_and_saveexec_b32 s1, s0
+; GFX1032-NEXT:  ; %bb.1: ; %if.then
+; GFX1032-NEXT:    ; divergent unreachable
+; GFX1032-NEXT:  ; %bb.2: ; %UnifiedReturnBlock
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: icmp32:
+; GFX1064:       ; %bb.0: ; %entry
+; GFX1064-NEXT:    s_load_dword s0, s[0:1], 0x28
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_cvt_f32_u32_e32 v1, s0
+; GFX1064-NEXT:    s_sub_i32 s1, 0, s0
+; GFX1064-NEXT:    v_rcp_iflag_f32_e32 v1, v1
+; GFX1064-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v1
+; GFX1064-NEXT:    v_cvt_u32_f32_e32 v1, v1
+; GFX1064-NEXT:    v_mul_lo_u32 v2, s1, v1
+; GFX1064-NEXT:    v_mul_hi_u32 v2, v1, v2
+; GFX1064-NEXT:    v_add_nc_u32_e32 v1, v1, v2
+; GFX1064-NEXT:    v_mul_hi_u32 v1, v0, v1
+; GFX1064-NEXT:    v_mul_lo_u32 v1, v1, s0
+; GFX1064-NEXT:    v_sub_nc_u32_e32 v0, v0, v1
+; GFX1064-NEXT:    v_subrev_nc_u32_e32 v1, s0, v0
+; GFX1064-NEXT:    v_cmp_le_u32_e32 vcc, s0, v0
+; GFX1064-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
+; GFX1064-NEXT:    v_subrev_nc_u32_e32 v1, s0, v0
+; GFX1064-NEXT:    v_cmp_le_u32_e32 vcc, s0, v0
+; GFX1064-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
+; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX1064-NEXT:    s_lshr_b32 s0, vcc_lo, 1
+; GFX1064-NEXT:    s_bitset1_b32 s0, 31
+; GFX1064-NEXT:    s_ff1_i32_b32 s0, s0
+; GFX1064-NEXT:    s_cmp_gt_u32 s0, 9
+; GFX1064-NEXT:    s_cselect_b64 s[0:1], -1, 0
+; GFX1064-NEXT:    s_and_b64 s[0:1], vcc, s[0:1]
+; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], s[0:1]
+; GFX1064-NEXT:  ; %bb.1: ; %if.then
+; GFX1064-NEXT:    ; divergent unreachable
+; GFX1064-NEXT:  ; %bb.2: ; %UnifiedReturnBlock
+; GFX1064-NEXT:    s_endpgm
 entry:
   %id = tail call i32 @llvm.amdgcn.workitem.id.x()
   %mul4 = mul nsw i32 %s, %n
@@ -1070,10 +2761,70 @@ if.end2:                                          ; preds = %if.end
   ret void
 }
 
-; GCN-LABEL: {{^}}fcmp32:
-; GFX1032: v_cmp_eq_f32_e32 vcc_lo, 0, v
-; GFX1064: v_cmp_eq_f32_e32 vcc, 0, v
 define amdgpu_kernel void @fcmp32(float %n, float %s) {
+; GFX1032-LABEL: fcmp32:
+; GFX1032:       ; %bb.0: ; %entry
+; GFX1032-NEXT:    s_load_dword s0, s[0:1], 0x28
+; GFX1032-NEXT:    v_cvt_f32_u32_e32 v0, v0
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    v_div_scale_f32 v1, s1, s0, s0, v0
+; GFX1032-NEXT:    v_rcp_f32_e32 v2, v1
+; GFX1032-NEXT:    v_fma_f32 v3, -v1, v2, 1.0
+; GFX1032-NEXT:    v_fmac_f32_e32 v2, v3, v2
+; GFX1032-NEXT:    v_div_scale_f32 v3, vcc_lo, v0, s0, v0
+; GFX1032-NEXT:    v_mul_f32_e32 v4, v3, v2
+; GFX1032-NEXT:    v_fma_f32 v5, -v1, v4, v3
+; GFX1032-NEXT:    v_fmac_f32_e32 v4, v5, v2
+; GFX1032-NEXT:    v_fma_f32 v1, -v1, v4, v3
+; GFX1032-NEXT:    v_div_fmas_f32 v1, v1, v2, v4
+; GFX1032-NEXT:    v_div_fixup_f32 v1, v1, s0, v0
+; GFX1032-NEXT:    v_trunc_f32_e32 v1, v1
+; GFX1032-NEXT:    v_fma_f32 v0, -v1, s0, v0
+; GFX1032-NEXT:    v_cmp_eq_f32_e32 vcc_lo, 0, v0
+; GFX1032-NEXT:    s_lshr_b32 s0, vcc_lo, 1
+; GFX1032-NEXT:    v_cmp_nlg_f32_e32 vcc_lo, 0, v0
+; GFX1032-NEXT:    s_bitset1_b32 s0, 31
+; GFX1032-NEXT:    s_ff1_i32_b32 s0, s0
+; GFX1032-NEXT:    s_cmp_gt_u32 s0, 9
+; GFX1032-NEXT:    s_cselect_b32 s0, -1, 0
+; GFX1032-NEXT:    s_and_b32 s0, vcc_lo, s0
+; GFX1032-NEXT:    s_and_saveexec_b32 s1, s0
+; GFX1032-NEXT:  ; %bb.1: ; %if.then
+; GFX1032-NEXT:    ; divergent unreachable
+; GFX1032-NEXT:  ; %bb.2: ; %UnifiedReturnBlock
+; GFX1032-NEXT:    s_endpgm
+;
+; GFX1064-LABEL: fcmp32:
+; GFX1064:       ; %bb.0: ; %entry
+; GFX1064-NEXT:    s_load_dword s2, s[0:1], 0x28
+; GFX1064-NEXT:    v_cvt_f32_u32_e32 v0, v0
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    v_div_scale_f32 v1, s[0:1], s2, s2, v0
+; GFX1064-NEXT:    v_rcp_f32_e32 v2, v1
+; GFX1064-NEXT:    v_fma_f32 v3, -v1, v2, 1.0
+; GFX1064-NEXT:    v_fmac_f32_e32 v2, v3, v2
+; GFX1064-NEXT:    v_div_scale_f32 v3, vcc, v0, s2, v0
+; GFX1064-NEXT:    v_mul_f32_e32 v4, v3, v2
+; GFX1064-NEXT:    v_fma_f32 v5, -v1, v4, v3
+; GFX1064-NEXT:    v_fmac_f32_e32 v4, v5, v2
+; GFX1064-NEXT:    v_fma_f32 v1, -v1, v4, v3
+; GFX1064-NEXT:    v_div_fmas_f32 v1, v1, v2, v4
+; GFX1064-NEXT:    v_div_fixup_f32 v1, v1, s2, v0
+; GFX1064-NEXT:    v_trunc_f32_e32 v1, v1
+; GFX1064-NEXT:    v_fma_f32 v0, -v1, s2, v0
+; GFX1064-NEXT:    v_cmp_eq_f32_e32 vcc, 0, v0
+; GFX1064-NEXT:    s_lshr_b32 s0, vcc_lo, 1
+; GFX1064-NEXT:    v_cmp_nlg_f32_e32 vcc, 0, v0
+; GFX1064-NEXT:    s_bitset1_b32 s0, 31
+; GFX1064-NEXT:    s_ff1_i32_b32 s0, s0
+; GFX1064-NEXT:    s_cmp_gt_u32 s0, 9
+; GFX1064-NEXT:    s_cselect_b64 s[0:1], -1, 0
+; GFX1064-NEXT:    s_and_b64 s[0:1], vcc, s[0:1]
+; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], s[0:1]
+; GFX1064-NEXT:  ; %bb.1: ; %if.then
+; GFX1064-NEXT:    ; divergent unreachable
+; GFX1064-NEXT:  ; %bb.2: ; %UnifiedReturnBlock
+; GFX1064-NEXT:    s_endpgm
 entry:
   %id = tail call i32 @llvm.amdgcn.workitem.id.x()
   %id.f = uitofp i32 %id to float
@@ -1103,47 +2854,76 @@ if.end2:                                          ; preds = %if.end
 
 declare void @external_void_func_void() #1
 
-; Test save/restore of VGPR needed for SGPR spilling.
-
-; GCN-LABEL: {{^}}callee_no_stack_with_call:
-; GCN: s_waitcnt
-; GCN-NEXT: s_waitcnt_vscnt
-; GCN-NEXT: s_mov_b32 [[FP_SCRATCH_COPY:s[0-9]+]], s33
-; GCN-NEXT: s_mov_b32 s33, s32
-; GFX1064-NEXT: s_or_saveexec_b64 [[COPY_EXEC0:s\[[0-9]+:[0-9]+\]]], -1{{$}}
-; GFX1032-NEXT: s_or_saveexec_b32 [[COPY_EXEC0:s[0-9]+]], -1{{$}}
-; GCN-NEXT: buffer_store_dword v40, off, s[0:3], s33 ; 4-byte Folded Spill
-; GCN-NEXT: buffer_store_dword v41, off, s[0:3], s33 offset:4 ; 4-byte Folded Spill
-; GCN-NEXT: s_waitcnt_depctr 0xffe3
-; GFX1064-NEXT: s_mov_b64 exec, [[COPY_EXEC0]]
-; GFX1032-NEXT: s_mov_b32 exec_lo, [[COPY_EXEC0]]
-
-; GFX1064: s_addk_i32 s32, 0x400
-; GFX1032: s_addk_i32 s32, 0x200
-; GCN-NEXT: v_writelane_b32 v41, [[FP_SCRATCH_COPY]], 0
-
-; GCN-DAG: v_writelane_b32 v40, s30, 0
-
-; GCN-DAG: v_writelane_b32 v40, s31, 1
-; GCN: s_swappc_b64
-; GCN-DAG: v_readlane_b32 s30, v40, 0
-; GCN-DAG: v_readlane_b32 s31, v40, 1
-
-; GCN: v_readlane_b32 [[FP_SCRATCH_COPY:s[0-9]+]], v41, 0
-; GFX1064: s_or_saveexec_b64 [[COPY_EXEC1:s\[[0-9]+:[0-9]+\]]], -1{{$}}
-; GFX1032: s_or_saveexec_b32 [[COPY_EXEC1:s[0-9]]], -1{{$}}
-; GCN-NEXT: s_clause 0x1
-; GCN-NEXT: buffer_load_dword v40, off, s[0:3], s33
-; GCN-NEXT: buffer_load_dword v41, off, s[0:3], s33 offset:4
-; GCN-NEXT: s_waitcnt_depctr 0xffe3
-; GFX1064-NEXT: s_mov_b64 exec, [[COPY_EXEC1]]
-; GFX1032-NEXT: s_mov_b32 exec_lo, [[COPY_EXEC1]]
-; GFX1064: s_addk_i32 s32, 0xfc00
-; GFX1032: s_addk_i32 s32, 0xfe00
-; GCN-NEXT: s_mov_b32 s33, [[FP_SCRATCH_COPY]]
-; GCN-NEXT: s_waitcnt vmcnt(0)
-; GCN-NEXT: s_setpc_b64
 define void @callee_no_stack_with_call() #1 {
+; GFX1032-LABEL: callee_no_stack_with_call:
+; GFX1032:       ; %bb.0:
+; GFX1032-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX1032-NEXT:    s_mov_b32 s16, s33
+; GFX1032-NEXT:    s_mov_b32 s33, s32
+; GFX1032-NEXT:    s_or_saveexec_b32 s17, -1
+; GFX1032-NEXT:    buffer_store_dword v40, off, s[0:3], s33 ; 4-byte Folded Spill
+; GFX1032-NEXT:    buffer_store_dword v41, off, s[0:3], s33 offset:4 ; 4-byte Folded Spill
+; GFX1032-NEXT:    s_waitcnt_depctr 0xffe3
+; GFX1032-NEXT:    s_mov_b32 exec_lo, s17
+; GFX1032-NEXT:    s_addk_i32 s32, 0x200
+; GFX1032-NEXT:    v_writelane_b32 v41, s16, 0
+; GFX1032-NEXT:    s_getpc_b64 s[16:17]
+; GFX1032-NEXT:    s_add_u32 s16, s16, external_void_func_void@gotpcrel32@lo+4
+; GFX1032-NEXT:    s_addc_u32 s17, s17, external_void_func_void@gotpcrel32@hi+12
+; GFX1032-NEXT:    v_writelane_b32 v40, s30, 0
+; GFX1032-NEXT:    s_load_dwordx2 s[16:17], s[16:17], 0x0
+; GFX1032-NEXT:    v_writelane_b32 v40, s31, 1
+; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT:    s_swappc_b64 s[30:31], s[16:17]
+; GFX1032-NEXT:    v_readlane_b32 s31, v40, 1
+; GFX1032-NEXT:    v_readlane_b32 s30, v40, 0
+; GFX1032-NEXT:    v_readlane_b32 s4, v41, 0
+; GFX1032-NEXT:    s_or_saveexec_b32 s5, -1
+; GFX1032-NEXT:    s_clause 0x1
+; GFX1032-NEXT:    buffer_load_dword v40, off, s[0:3], s33
+; GFX1032-NEXT:    buffer_load_dword v41, off, s[0:3], s33 offset:4
+; GFX1032-NEXT:    s_waitcnt_depctr 0xffe3
+; GFX1032-NEXT:    s_mov_b32 exec_lo, s5
+; GFX1032-NEXT:    s_addk_i32 s32, 0xfe00
+; GFX1032-NEXT:    s_mov_b32 s33, s4
+; GFX1032-NEXT:    s_waitcnt vmcnt(0)
+; GFX1032-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX1064-LABEL: callee_no_stack_with_call:
+; GFX1064:       ; %bb.0:
+; GFX1064-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX1064-NEXT:    s_mov_b32 s16, s33
+; GFX1064-NEXT:    s_mov_b32 s33, s32
+; GFX1064-NEXT:    s_or_saveexec_b64 s[18:19], -1
+; GFX1064-NEXT:    buffer_store_dword v40, off, s[0:3], s33 ; 4-byte Folded Spill
+; GFX1064-NEXT:    buffer_store_dword v41, off, s[0:3], s33 offset:4 ; 4-byte Folded Spill
+; GFX1064-NEXT:    s_waitcnt_depctr 0xffe3
+; GFX1064-NEXT:    s_mov_b64 exec, s[18:19]
+; GFX1064-NEXT:    s_addk_i32 s32, 0x400
+; GFX1064-NEXT:    v_writelane_b32 v41, s16, 0
+; GFX1064-NEXT:    s_getpc_b64 s[16:17]
+; GFX1064-NEXT:    s_add_u32 s16, s16, external_void_func_void@gotpcrel32@lo+4
+; GFX1064-NEXT:    s_addc_u32 s17, s17, external_void_func_void@gotpcrel32@hi+12
+; GFX1064-NEXT:    v_writelane_b32 v40, s30, 0
+; GFX1064-NEXT:    s_load_dwordx2 s[16:17], s[16:17], 0x0
+; GFX1064-NEXT:    v_writelane_b32 v40, s31, 1
+; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT:    s_swappc_b64 s[30:31], s[16:17]
+; GFX1064-NEXT:    v_readlane_b32 s31, v40, 1
+; GFX1064-NEXT:    v_readlane_b32 s30, v40, 0
+; GFX1064-NEXT:    v_readlane_b32 s4, v41, 0
+; GFX1064-NEXT:    s_or_saveexec_b64 s[6:7], -1
+; GFX1064-NEXT:    s_clause 0x1
+; GFX1064-NEXT:    buffer_load_dword v40, off, s[0:3], s33
+; GFX1064-NEXT:    buffer_load_dword v41, off, s[0:3], s33 offset:4
+; GFX1064-NEXT:    s_waitcnt_depctr 0xffe3
+; GFX1064-NEXT:    s_mov_b64 exec, s[6:7]
+; GFX1064-NEXT:    s_addk_i32 s32, 0xfc00
+; GFX1064-NEXT:    s_mov_b32 s33, s4
+; GFX1064-NEXT:    s_waitcnt vmcnt(0)
+; GFX1064-NEXT:    s_setpc_b64 s[30:31]
   call void @external_void_func_void()
   ret void
 }
@@ -1185,3 +2965,5 @@ attributes #2 = { nounwind readnone optnone noinline }
 attributes #3 = { "target-features"="+wavefrontsize32" }
 attributes #4 = { "target-features"="+wavefrontsize64" }
 attributes #5 = { inaccessiblememonly nounwind }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX10DEFWAVE: {{.*}}
diff --git a/llvm/test/CodeGen/ARM/aes-erratum-fix.ll b/llvm/test/CodeGen/ARM/aes-erratum-fix.ll
index afd9a929f75ad..d1e083b0ee883 100644
--- a/llvm/test/CodeGen/ARM/aes-erratum-fix.ll
+++ b/llvm/test/CodeGen/ARM/aes-erratum-fix.ll
@@ -1353,335 +1353,210 @@ define arm_aapcs_vfpcc void @aese_setf16_via_val(half %0, <16 x i8> %1, <16 x i8
 define arm_aapcs_vfpcc void @aese_setf16_cond_via_ptr(i1 zeroext %0, half* %1, <16 x i8> %2, <16 x i8>* %3) nounwind {
 ; CHECK-FIX-NOSCHED-LABEL: aese_setf16_cond_via_ptr:
 ; CHECK-FIX-NOSCHED:       @ %bb.0:
-; CHECK-FIX-NOSCHED-NEXT:    .save {r4, r5, r6, r7, r8, lr}
-; CHECK-FIX-NOSCHED-NEXT:    push {r4, r5, r6, r7, r8, lr}
+; CHECK-FIX-NOSCHED-NEXT:    .save {r4, r5, r6, r7, r8, r9, r10, r11, lr}
+; CHECK-FIX-NOSCHED-NEXT:    push {r4, r5, r6, r7, r8, r9, r10, r11, lr}
+; CHECK-FIX-NOSCHED-NEXT:    .pad #24
+; CHECK-FIX-NOSCHED-NEXT:    sub sp, sp, #24
 ; CHECK-FIX-NOSCHED-NEXT:    cmp r0, #0
 ; CHECK-FIX-NOSCHED-NEXT:    beq .LBB36_3
 ; CHECK-FIX-NOSCHED-NEXT:  @ %bb.1:
 ; CHECK-FIX-NOSCHED-NEXT:    vld1.64 {d16, d17}, [r2]
-; CHECK-FIX-NOSCHED-NEXT:    vorr q9, q8, q8
-; CHECK-FIX-NOSCHED-NEXT:    vmov lr, r12, d17
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r3, d16[1]
-; CHECK-FIX-NOSCHED-NEXT:    vld1.16 {d18[0]}, [r1:16]
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r4, d18[0]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s6, lr
-; CHECK-FIX-NOSCHED-NEXT:    lsr r5, r3, #16
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s8, s6
-; CHECK-FIX-NOSCHED-NEXT:    vmov s6, r3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s12, s6
-; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r12, #16
-; CHECK-FIX-NOSCHED-NEXT:    lsr lr, lr, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s4, r12
-; CHECK-FIX-NOSCHED-NEXT:    vmov s10, lr
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s4, s4
-; CHECK-FIX-NOSCHED-NEXT:    vmov s14, r5
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s10, s10
-; CHECK-FIX-NOSCHED-NEXT:    vmov s6, r4
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s14, s14
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s5, s6
-; CHECK-FIX-NOSCHED-NEXT:    vmov s6, r3
-; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r4, #16
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s6, s6
-; CHECK-FIX-NOSCHED-NEXT:    vmov s7, r3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s7, s7
+; CHECK-FIX-NOSCHED-NEXT:    vmov r7, r6, d17
+; CHECK-FIX-NOSCHED-NEXT:    vld1.16 {d16[0]}, [r1:16]
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r5, d16[0]
+; CHECK-FIX-NOSCHED-NEXT:    uxth r4, r3
+; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r3, #16
+; CHECK-FIX-NOSCHED-NEXT:    str r4, [sp, #8] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    uxth r4, r7
+; CHECK-FIX-NOSCHED-NEXT:    str r4, [sp, #12] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    uxth r4, r6
+; CHECK-FIX-NOSCHED-NEXT:    lsr r6, r6, #16
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #4] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    str r4, [sp, #20] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    lsr r7, r7, #16
+; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r5, #16
+; CHECK-FIX-NOSCHED-NEXT:    str r6, [sp, #16] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    uxth r10, r5
 ; CHECK-FIX-NOSCHED-NEXT:    cmp r0, #0
 ; CHECK-FIX-NOSCHED-NEXT:    bne .LBB36_4
 ; CHECK-FIX-NOSCHED-NEXT:  .LBB36_2:
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, r1, d0
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, r7, d1
-; CHECK-FIX-NOSCHED-NEXT:    vmov s1, r1
-; CHECK-FIX-NOSCHED-NEXT:    lsr r1, r1, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s0, r7
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s9, s1
-; CHECK-FIX-NOSCHED-NEXT:    vmov s2, r3
-; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r3, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s1, r0
-; CHECK-FIX-NOSCHED-NEXT:    lsr r7, r7, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s3, r3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s11, s1
-; CHECK-FIX-NOSCHED-NEXT:    vmov s1, r7
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s0, s0
-; CHECK-FIX-NOSCHED-NEXT:    vmov s13, r1
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s2, s2
-; CHECK-FIX-NOSCHED-NEXT:    lsr r0, r0, #16
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s1, s1
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s3, s3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s13, s13
+; CHECK-FIX-NOSCHED-NEXT:    vmov r4, r6, d1
+; CHECK-FIX-NOSCHED-NEXT:    vmov r0, r3, d0
+; CHECK-FIX-NOSCHED-NEXT:    lsr r5, r4, #16
+; CHECK-FIX-NOSCHED-NEXT:    lsr r1, r6, #16
+; CHECK-FIX-NOSCHED-NEXT:    uxth r11, r6
+; CHECK-FIX-NOSCHED-NEXT:    lsr lr, r0, #16
+; CHECK-FIX-NOSCHED-NEXT:    lsr r12, r3, #16
+; CHECK-FIX-NOSCHED-NEXT:    uxth r9, r4
+; CHECK-FIX-NOSCHED-NEXT:    uxth r6, r3
 ; CHECK-FIX-NOSCHED-NEXT:    b .LBB36_5
 ; CHECK-FIX-NOSCHED-NEXT:  .LBB36_3:
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r5, [r2, #10]
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r12, [r2, #6]
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r6, [r2, #2]
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r7, [r2, #14]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s8, r5
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r2, #14]
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #16] @ 4-byte Spill
 ; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r2, #12]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s12, r12
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r4, [r2, #8]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s5, r6
-; CHECK-FIX-NOSCHED-NEXT:    ldrh lr, [r2, #4]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s4, r7
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r8, [r2]
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s6, s4
-; CHECK-FIX-NOSCHED-NEXT:    vmov s4, r3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s10, s8
-; CHECK-FIX-NOSCHED-NEXT:    vmov s8, r4
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s14, s12
-; CHECK-FIX-NOSCHED-NEXT:    vmov s12, lr
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s7, s5
-; CHECK-FIX-NOSCHED-NEXT:    vmov s5, r8
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s4, s4
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s8, s8
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s12, s12
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s5, s5
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #20] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r2, #8]
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #12] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r2, #6]
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r7, [r2, #10]
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r10, [r2]
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #4] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r2, #4]
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #8] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r2, #2]
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp] @ 4-byte Spill
 ; CHECK-FIX-NOSCHED-NEXT:    cmp r0, #0
 ; CHECK-FIX-NOSCHED-NEXT:    beq .LBB36_2
 ; CHECK-FIX-NOSCHED-NEXT:  .LBB36_4:
-; CHECK-FIX-NOSCHED-NEXT:    vorr q8, q0, q0
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r3, d0[1]
-; CHECK-FIX-NOSCHED-NEXT:    vld1.16 {d16[0]}, [r1:16]
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, r1, d1
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r7, d16[0]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s1, r3
-; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r3, #16
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s9, s1
-; CHECK-FIX-NOSCHED-NEXT:    vmov s13, r3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s13, s13
-; CHECK-FIX-NOSCHED-NEXT:    vmov s0, r1
-; CHECK-FIX-NOSCHED-NEXT:    lsr r1, r1, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s2, r0
-; CHECK-FIX-NOSCHED-NEXT:    lsr r0, r0, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s3, r0
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s0, s0
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s2, s2
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s3, s3
-; CHECK-FIX-NOSCHED-NEXT:    vmov s1, r7
-; CHECK-FIX-NOSCHED-NEXT:    lsr r0, r7, #16
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s11, s1
-; CHECK-FIX-NOSCHED-NEXT:    vmov s1, r1
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s1, s1
+; CHECK-FIX-NOSCHED-NEXT:    vmov r5, r3, d1
+; CHECK-FIX-NOSCHED-NEXT:    mov r4, r7
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r7, d0[1]
+; CHECK-FIX-NOSCHED-NEXT:    vld1.16 {d0[0]}, [r1:16]
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r0, d0[0]
+; CHECK-FIX-NOSCHED-NEXT:    uxth r9, r5
+; CHECK-FIX-NOSCHED-NEXT:    uxth r11, r3
+; CHECK-FIX-NOSCHED-NEXT:    uxth r6, r7
+; CHECK-FIX-NOSCHED-NEXT:    lsr r12, r7, #16
+; CHECK-FIX-NOSCHED-NEXT:    lsr r1, r3, #16
+; CHECK-FIX-NOSCHED-NEXT:    lsr r5, r5, #16
+; CHECK-FIX-NOSCHED-NEXT:    mov r7, r4
+; CHECK-FIX-NOSCHED-NEXT:    lsr lr, r0, #16
 ; CHECK-FIX-NOSCHED-NEXT:  .LBB36_5:
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s5, s5
-; CHECK-FIX-NOSCHED-NEXT:    vmov s15, r0
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, s5
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s5, s7
-; CHECK-FIX-NOSCHED-NEXT:    vmov r1, s5
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s15, s15
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s5, s15
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s11, s11
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s5
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s12, s12
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s5, s9
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s8, s8
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s2, s2
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s0, s0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r0, r1, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r1, s11
+; CHECK-FIX-NOSCHED-NEXT:    uxth r8, r0
+; CHECK-FIX-NOSCHED-NEXT:    ldr r0, [sp] @ 4-byte Reload
+; CHECK-FIX-NOSCHED-NEXT:    ldr r3, [sp, #4] @ 4-byte Reload
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r10, r0, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d16[0], r0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r1, r3, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r1, s12
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s12, s14
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r8, lr, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d18[0], r0
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, s12
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s12, s13
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s12
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r1, r0, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r1, s5
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r6, r12, lsl #16
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d18[1], r0
+; CHECK-FIX-NOSCHED-NEXT:    ldr r0, [sp, #8] @ 4-byte Reload
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r0, r3, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d16[1], r0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r1, r1, r3, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d18[1], r1
-; CHECK-FIX-NOSCHED-NEXT:    vmov r1, s8
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s8, s10
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s8
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r1, r3, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r1, s2
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s2, s3
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s2
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d17[0], r0
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s2, s4
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r1, r3, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r1, s0
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s0, s1
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r9, r5, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d19[0], r0
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, s0
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s0, s6
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r1, r0, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r1, s2
+; CHECK-FIX-NOSCHED-NEXT:    ldr r0, [sp, #12] @ 4-byte Reload
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r0, r7, lsl #16
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d17[0], r0
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r11, r1, lsl #16
+; CHECK-FIX-NOSCHED-NEXT:    ldr r1, [sp, #16] @ 4-byte Reload
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d19[1], r0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r1, r3, lsl #16
+; CHECK-FIX-NOSCHED-NEXT:    ldr r0, [sp, #20] @ 4-byte Reload
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r0, r1, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d17[1], r0
 ; CHECK-FIX-NOSCHED-NEXT:    aese.8 q8, q9
 ; CHECK-FIX-NOSCHED-NEXT:    aesmc.8 q8, q8
 ; CHECK-FIX-NOSCHED-NEXT:    vst1.64 {d16, d17}, [r2]
-; CHECK-FIX-NOSCHED-NEXT:    pop {r4, r5, r6, r7, r8, pc}
+; CHECK-FIX-NOSCHED-NEXT:    add sp, sp, #24
+; CHECK-FIX-NOSCHED-NEXT:    pop {r4, r5, r6, r7, r8, r9, r10, r11, pc}
 ;
 ; CHECK-CORTEX-FIX-LABEL: aese_setf16_cond_via_ptr:
 ; CHECK-CORTEX-FIX:       @ %bb.0:
-; CHECK-CORTEX-FIX-NEXT:    .save {r4, r5, r6, r7, r8, lr}
-; CHECK-CORTEX-FIX-NEXT:    push {r4, r5, r6, r7, r8, lr}
-; CHECK-CORTEX-FIX-NEXT:    .vsave {d8, d9}
-; CHECK-CORTEX-FIX-NEXT:    vpush {d8, d9}
+; CHECK-CORTEX-FIX-NEXT:    .save {r4, r5, r6, r7, r8, r9, r10, r11, lr}
+; CHECK-CORTEX-FIX-NEXT:    push {r4, r5, r6, r7, r8, r9, r10, r11, lr}
+; CHECK-CORTEX-FIX-NEXT:    .pad #24
+; CHECK-CORTEX-FIX-NEXT:    sub sp, sp, #24
 ; CHECK-CORTEX-FIX-NEXT:    cmp r0, #0
 ; CHECK-CORTEX-FIX-NEXT:    beq .LBB36_3
 ; CHECK-CORTEX-FIX-NEXT:  @ %bb.1:
 ; CHECK-CORTEX-FIX-NEXT:    vld1.64 {d16, d17}, [r2]
 ; CHECK-CORTEX-FIX-NEXT:    vorr q9, q8, q8
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 r3, d16[1]
-; CHECK-CORTEX-FIX-NEXT:    vmov r5, r6, d17
 ; CHECK-CORTEX-FIX-NEXT:    vld1.16 {d18[0]}, [r1:16]
-; CHECK-CORTEX-FIX-NEXT:    lsr r7, r5, #16
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 r3, d18[0]
+; CHECK-CORTEX-FIX-NEXT:    uxth r7, r3
+; CHECK-CORTEX-FIX-NEXT:    lsr r3, r3, #16
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #16] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 r3, d16[1]
+; CHECK-CORTEX-FIX-NEXT:    str r7, [sp, #20] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    uxth r7, r3
+; CHECK-CORTEX-FIX-NEXT:    lsr r3, r3, #16
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #4] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    vmov r3, r6, d17
+; CHECK-CORTEX-FIX-NEXT:    str r7, [sp, #8] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    uxth r7, r3
+; CHECK-CORTEX-FIX-NEXT:    lsr r3, r3, #16
+; CHECK-CORTEX-FIX-NEXT:    uxth r11, r6
 ; CHECK-CORTEX-FIX-NEXT:    lsr r4, r6, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s4, r6
-; CHECK-CORTEX-FIX-NEXT:    vmov s6, r5
-; CHECK-CORTEX-FIX-NEXT:    lsr r8, r3, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s8, r3
-; CHECK-CORTEX-FIX-NEXT:    vmov s12, r4
-; CHECK-CORTEX-FIX-NEXT:    vmov s5, r7
-; CHECK-CORTEX-FIX-NEXT:    vmov s9, r8
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s10, s4
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s4, s6
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 lr, d18[0]
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s6, s5
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s7, s8
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s14, s12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s5, s9
-; CHECK-CORTEX-FIX-NEXT:    lsr r12, lr, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s11, lr
-; CHECK-CORTEX-FIX-NEXT:    vmov s13, r12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s12, s11
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s8, s13
+; CHECK-CORTEX-FIX-NEXT:    str r7, [sp, #12] @ 4-byte Spill
 ; CHECK-CORTEX-FIX-NEXT:    cmp r0, #0
 ; CHECK-CORTEX-FIX-NEXT:    bne .LBB36_4
 ; CHECK-CORTEX-FIX-NEXT:  .LBB36_2:
-; CHECK-CORTEX-FIX-NEXT:    vmov r6, r5, d1
-; CHECK-CORTEX-FIX-NEXT:    vmov r0, r1, d0
-; CHECK-CORTEX-FIX-NEXT:    lsr r7, r1, #16
-; CHECK-CORTEX-FIX-NEXT:    lsr r4, r6, #16
-; CHECK-CORTEX-FIX-NEXT:    lsr r3, r5, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s2, r6
-; CHECK-CORTEX-FIX-NEXT:    vmov s0, r5
-; CHECK-CORTEX-FIX-NEXT:    vmov s3, r1
-; CHECK-CORTEX-FIX-NEXT:    vmov s9, r0
-; CHECK-CORTEX-FIX-NEXT:    lsr r12, r0, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s13, r3
-; CHECK-CORTEX-FIX-NEXT:    vmov s15, r4
-; CHECK-CORTEX-FIX-NEXT:    vmov s16, r7
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s1, s0
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s11, s3
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s3, s9
-; CHECK-CORTEX-FIX-NEXT:    vmov s0, r12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s2, s2
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s9, s15
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s13, s13
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s15, s16
+; CHECK-CORTEX-FIX-NEXT:    vmov r1, r7, d0
+; CHECK-CORTEX-FIX-NEXT:    uxth r0, r1
+; CHECK-CORTEX-FIX-NEXT:    uxth r6, r7
+; CHECK-CORTEX-FIX-NEXT:    lsr r12, r7, #16
+; CHECK-CORTEX-FIX-NEXT:    lsr r9, r1, #16
+; CHECK-CORTEX-FIX-NEXT:    str r0, [sp] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    mov r0, r3
+; CHECK-CORTEX-FIX-NEXT:    vmov r7, r3, d1
+; CHECK-CORTEX-FIX-NEXT:    uxth r10, r7
+; CHECK-CORTEX-FIX-NEXT:    lsr r5, r7, #16
+; CHECK-CORTEX-FIX-NEXT:    uxth lr, r3
+; CHECK-CORTEX-FIX-NEXT:    lsr r8, r3, #16
+; CHECK-CORTEX-FIX-NEXT:    mov r3, r0
 ; CHECK-CORTEX-FIX-NEXT:    b .LBB36_5
 ; CHECK-CORTEX-FIX-NEXT:  .LBB36_3:
-; CHECK-CORTEX-FIX-NEXT:    ldrh r12, [r2]
-; CHECK-CORTEX-FIX-NEXT:    ldrh lr, [r2, #2]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r8, [r2, #4]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r5, [r2, #6]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r4, [r2, #8]
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r2]
+; CHECK-CORTEX-FIX-NEXT:    ldrh r11, [r2, #12]
+; CHECK-CORTEX-FIX-NEXT:    ldrh r4, [r2, #14]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #20] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r2, #2]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #16] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r2, #4]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #8] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r2, #6]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #4] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r2, #8]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #12] @ 4-byte Spill
 ; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r2, #10]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r7, [r2, #12]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r6, [r2, #14]
-; CHECK-CORTEX-FIX-NEXT:    vmov s5, r5
-; CHECK-CORTEX-FIX-NEXT:    vmov s7, r8
-; CHECK-CORTEX-FIX-NEXT:    vmov s4, r6
-; CHECK-CORTEX-FIX-NEXT:    vmov s6, r7
-; CHECK-CORTEX-FIX-NEXT:    vmov s8, r3
-; CHECK-CORTEX-FIX-NEXT:    vmov s12, r4
-; CHECK-CORTEX-FIX-NEXT:    vmov s9, lr
-; CHECK-CORTEX-FIX-NEXT:    vmov s11, r12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s14, s4
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s10, s6
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s6, s8
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s4, s12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s5, s5
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s7, s7
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s8, s9
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s12, s11
 ; CHECK-CORTEX-FIX-NEXT:    cmp r0, #0
 ; CHECK-CORTEX-FIX-NEXT:    beq .LBB36_2
 ; CHECK-CORTEX-FIX-NEXT:  .LBB36_4:
 ; CHECK-CORTEX-FIX-NEXT:    vorr q8, q0, q0
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 r3, d0[1]
-; CHECK-CORTEX-FIX-NEXT:    vmov r6, r5, d1
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 r5, d0[1]
 ; CHECK-CORTEX-FIX-NEXT:    vld1.16 {d16[0]}, [r1:16]
-; CHECK-CORTEX-FIX-NEXT:    lsr r4, r6, #16
-; CHECK-CORTEX-FIX-NEXT:    lsr r1, r5, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s2, r6
-; CHECK-CORTEX-FIX-NEXT:    vmov s0, r5
-; CHECK-CORTEX-FIX-NEXT:    lsr r7, r3, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s3, r3
-; CHECK-CORTEX-FIX-NEXT:    vmov s9, r1
-; CHECK-CORTEX-FIX-NEXT:    vmov s15, r4
-; CHECK-CORTEX-FIX-NEXT:    vmov s16, r7
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s1, s0
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s2, s2
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 r0, d16[0]
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s13, s9
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s9, s15
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s11, s3
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s15, s16
-; CHECK-CORTEX-FIX-NEXT:    vmov s18, r0
-; CHECK-CORTEX-FIX-NEXT:    lsr r12, r0, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s0, r12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s3, s18
+; CHECK-CORTEX-FIX-NEXT:    uxth r6, r5
+; CHECK-CORTEX-FIX-NEXT:    lsr r12, r5, #16
+; CHECK-CORTEX-FIX-NEXT:    vmov r5, r7, d1
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 r1, d16[0]
+; CHECK-CORTEX-FIX-NEXT:    uxth r10, r5
+; CHECK-CORTEX-FIX-NEXT:    lsr r5, r5, #16
+; CHECK-CORTEX-FIX-NEXT:    uxth lr, r7
+; CHECK-CORTEX-FIX-NEXT:    lsr r8, r7, #16
+; CHECK-CORTEX-FIX-NEXT:    uxth r0, r1
+; CHECK-CORTEX-FIX-NEXT:    lsr r9, r1, #16
+; CHECK-CORTEX-FIX-NEXT:    str r0, [sp] @ 4-byte Spill
 ; CHECK-CORTEX-FIX-NEXT:  .LBB36_5:
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s10, s10
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s14, s14
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s7, s7
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s6, s6
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s4, s4
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s8, s8
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s12, s12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s0, s0
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s2, s2
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s9, s9
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s3, s3
-; CHECK-CORTEX-FIX-NEXT:    vmov r0, s10
-; CHECK-CORTEX-FIX-NEXT:    vmov r1, s14
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s10, s5
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s14, s1
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s1, s13
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s5, s11
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s11, s15
-; CHECK-CORTEX-FIX-NEXT:    vmov r5, s6
-; CHECK-CORTEX-FIX-NEXT:    vmov r4, s8
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s0, s0
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r12, r0, r1, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r1, s7
-; CHECK-CORTEX-FIX-NEXT:    vmov r3, s10
-; CHECK-CORTEX-FIX-NEXT:    vmov r7, s1
-; CHECK-CORTEX-FIX-NEXT:    vmov r6, s11
-; CHECK-CORTEX-FIX-NEXT:    vmov r0, s9
-; CHECK-CORTEX-FIX-NEXT:    pkhbt lr, r1, r3, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r3, s14
-; CHECK-CORTEX-FIX-NEXT:    vmov r1, s0
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r3, r3, r7, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r7, s5
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r7, r7, r6, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r6, s4
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r6, r6, r5, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r5, s12
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r5, r5, r4, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r4, s2
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d18[0], r5
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d19[0], r6
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d18[1], lr
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d19[1], r12
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r0, r4, r0, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r4, s3
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r1, r4, r1, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d16[0], r1
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d17[0], r0
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d16[1], r7
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d17[1], r3
+; CHECK-CORTEX-FIX-NEXT:    ldr r0, [sp, #8] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    ldr r1, [sp, #4] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r11, r11, r4, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    ldr r4, [sp, #16] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r6, r6, r12, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r5, r10, r5, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r7, r0, r1, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    ldr r1, [sp, #12] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r0, lr, r8, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r1, r1, r3, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    ldr r3, [sp, #20] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r4, r3, r4, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    ldr r3, [sp] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d18[0], r4
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d19[0], r1
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d18[1], r7
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d19[1], r11
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r3, r3, r9, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d16[0], r3
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d17[0], r5
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d16[1], r6
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d17[1], r0
 ; CHECK-CORTEX-FIX-NEXT:    aese.8 q9, q8
 ; CHECK-CORTEX-FIX-NEXT:    aesmc.8 q8, q9
 ; CHECK-CORTEX-FIX-NEXT:    vst1.64 {d16, d17}, [r2]
-; CHECK-CORTEX-FIX-NEXT:    vpop {d8, d9}
-; CHECK-CORTEX-FIX-NEXT:    pop {r4, r5, r6, r7, r8, pc}
+; CHECK-CORTEX-FIX-NEXT:    add sp, sp, #24
+; CHECK-CORTEX-FIX-NEXT:    pop {r4, r5, r6, r7, r8, r9, r10, r11, pc}
   br i1 %0, label %5, label %12
 
 5:
@@ -1727,341 +1602,213 @@ define arm_aapcs_vfpcc void @aese_setf16_cond_via_ptr(i1 zeroext %0, half* %1, <
 define arm_aapcs_vfpcc void @aese_setf16_cond_via_val(i1 zeroext %0, half %1, <16 x i8> %2, <16 x i8>* %3) nounwind {
 ; CHECK-FIX-NOSCHED-LABEL: aese_setf16_cond_via_val:
 ; CHECK-FIX-NOSCHED:       @ %bb.0:
-; CHECK-FIX-NOSCHED-NEXT:    .save {r4, r5, r6, r7, r11, lr}
-; CHECK-FIX-NOSCHED-NEXT:    push {r4, r5, r6, r7, r11, lr}
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s9, s0
+; CHECK-FIX-NOSCHED-NEXT:    .save {r4, r5, r6, r7, r8, r9, r10, r11, lr}
+; CHECK-FIX-NOSCHED-NEXT:    push {r4, r5, r6, r7, r8, r9, r10, r11, lr}
+; CHECK-FIX-NOSCHED-NEXT:    .pad #24
+; CHECK-FIX-NOSCHED-NEXT:    sub sp, sp, #24
+; CHECK-FIX-NOSCHED-NEXT:    vmov r12, s0
 ; CHECK-FIX-NOSCHED-NEXT:    cmp r0, #0
-; CHECK-FIX-NOSCHED-NEXT:    beq .LBB37_3
+; CHECK-FIX-NOSCHED-NEXT:    beq .LBB37_2
 ; CHECK-FIX-NOSCHED-NEXT:  @ %bb.1:
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s0, s9
 ; CHECK-FIX-NOSCHED-NEXT:    vld1.64 {d16, d17}, [r1]
-; CHECK-FIX-NOSCHED-NEXT:    vmov r2, s0
-; CHECK-FIX-NOSCHED-NEXT:    vmov lr, r12, d17
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r3, d16[1]
-; CHECK-FIX-NOSCHED-NEXT:    vmov.16 d16[0], r2
-; CHECK-FIX-NOSCHED-NEXT:    vmov s2, lr
-; CHECK-FIX-NOSCHED-NEXT:    lsr lr, lr, #16
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s8, s2
-; CHECK-FIX-NOSCHED-NEXT:    vmov s2, r3
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r2, d16[0]
-; CHECK-FIX-NOSCHED-NEXT:    lsr r4, r3, #16
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s12, s2
-; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r12, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s0, r12
-; CHECK-FIX-NOSCHED-NEXT:    vmov s10, lr
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s0, s0
-; CHECK-FIX-NOSCHED-NEXT:    vmov s14, r4
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s10, s10
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s14, s14
-; CHECK-FIX-NOSCHED-NEXT:    vmov s2, r2
-; CHECK-FIX-NOSCHED-NEXT:    lsr r2, r2, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s3, r2
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s1, s2
-; CHECK-FIX-NOSCHED-NEXT:    vmov s2, r3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s3, s3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s2, s2
-; CHECK-FIX-NOSCHED-NEXT:    cmp r0, #0
-; CHECK-FIX-NOSCHED-NEXT:    bne .LBB37_4
-; CHECK-FIX-NOSCHED-NEXT:  .LBB37_2:
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, r2, d2
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, r7, d3
-; CHECK-FIX-NOSCHED-NEXT:    vmov s5, r2
-; CHECK-FIX-NOSCHED-NEXT:    lsr r2, r2, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s4, r7
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s9, s5
-; CHECK-FIX-NOSCHED-NEXT:    vmov s6, r3
+; CHECK-FIX-NOSCHED-NEXT:    vmov r7, r6, d17
+; CHECK-FIX-NOSCHED-NEXT:    vmov.16 d16[0], r12
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r5, d16[0]
+; CHECK-FIX-NOSCHED-NEXT:    uxth r4, r7
+; CHECK-FIX-NOSCHED-NEXT:    uxth r2, r3
 ; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r3, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s5, r0
+; CHECK-FIX-NOSCHED-NEXT:    str r4, [sp, #12] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    uxth r4, r6
+; CHECK-FIX-NOSCHED-NEXT:    lsr r6, r6, #16
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #8] @ 4-byte Spill
 ; CHECK-FIX-NOSCHED-NEXT:    lsr r7, r7, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s7, r3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s11, s5
-; CHECK-FIX-NOSCHED-NEXT:    vmov s5, r7
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s4, s4
-; CHECK-FIX-NOSCHED-NEXT:    vmov s13, r2
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s6, s6
-; CHECK-FIX-NOSCHED-NEXT:    lsr r0, r0, #16
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s5, s5
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s7, s7
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s13, s13
-; CHECK-FIX-NOSCHED-NEXT:    b .LBB37_5
+; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r5, #16
+; CHECK-FIX-NOSCHED-NEXT:    str r4, [sp, #20] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #4] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    uxth r3, r5
+; CHECK-FIX-NOSCHED-NEXT:    str r6, [sp, #16] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    b .LBB37_3
+; CHECK-FIX-NOSCHED-NEXT:  .LBB37_2:
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r1, #14]
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #16] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r1, #12]
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #20] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r1, #8]
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #12] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r2, [r1, #6]
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r1, #2]
+; CHECK-FIX-NOSCHED-NEXT:    str r2, [sp, #8] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #4] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r7, [r1, #10]
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r2, [r1, #4]
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r1]
 ; CHECK-FIX-NOSCHED-NEXT:  .LBB37_3:
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r1, #10]
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r12, [r1, #6]
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r5, [r1, #2]
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r7, [r1, #14]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s8, r3
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r2, [r1, #12]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s12, r12
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r4, [r1, #8]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s1, r5
-; CHECK-FIX-NOSCHED-NEXT:    ldrh lr, [r1, #4]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s0, r7
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r6, [r1]
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s2, s0
-; CHECK-FIX-NOSCHED-NEXT:    vmov s0, r2
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s10, s8
-; CHECK-FIX-NOSCHED-NEXT:    vmov s8, r4
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s14, s12
-; CHECK-FIX-NOSCHED-NEXT:    vmov s12, lr
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s3, s1
-; CHECK-FIX-NOSCHED-NEXT:    vmov s1, r6
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s0, s0
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s8, s8
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s12, s12
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s1, s1
 ; CHECK-FIX-NOSCHED-NEXT:    cmp r0, #0
-; CHECK-FIX-NOSCHED-NEXT:    beq .LBB37_2
-; CHECK-FIX-NOSCHED-NEXT:  .LBB37_4:
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s9, s9
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, r2, d3
-; CHECK-FIX-NOSCHED-NEXT:    vmov r7, s9
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r3, d2[1]
-; CHECK-FIX-NOSCHED-NEXT:    vmov.16 d2[0], r7
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r7, d2[0]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s5, r3
-; CHECK-FIX-NOSCHED-NEXT:    vmov s4, r2
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s9, s5
-; CHECK-FIX-NOSCHED-NEXT:    vmov s6, r0
-; CHECK-FIX-NOSCHED-NEXT:    lsr r0, r0, #16
-; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r3, #16
-; CHECK-FIX-NOSCHED-NEXT:    lsr r2, r2, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s7, r0
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s4, s4
-; CHECK-FIX-NOSCHED-NEXT:    vmov s13, r3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s6, s6
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s7, s7
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s13, s13
-; CHECK-FIX-NOSCHED-NEXT:    vmov s5, r7
-; CHECK-FIX-NOSCHED-NEXT:    lsr r0, r7, #16
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s11, s5
-; CHECK-FIX-NOSCHED-NEXT:    vmov s5, r2
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s5, s5
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    beq .LBB37_5
+; CHECK-FIX-NOSCHED-NEXT:  @ %bb.4:
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r6, d2[1]
+; CHECK-FIX-NOSCHED-NEXT:    mov r3, r2
+; CHECK-FIX-NOSCHED-NEXT:    mov r2, r7
+; CHECK-FIX-NOSCHED-NEXT:    vmov r4, r7, d3
+; CHECK-FIX-NOSCHED-NEXT:    vmov.16 d2[0], r12
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r0, d2[0]
+; CHECK-FIX-NOSCHED-NEXT:    uxth r5, r6
+; CHECK-FIX-NOSCHED-NEXT:    lsr r12, r6, #16
+; CHECK-FIX-NOSCHED-NEXT:    uxth r10, r4
+; CHECK-FIX-NOSCHED-NEXT:    uxth r11, r7
+; CHECK-FIX-NOSCHED-NEXT:    lsr r9, r7, #16
+; CHECK-FIX-NOSCHED-NEXT:    mov r7, r2
+; CHECK-FIX-NOSCHED-NEXT:    mov r2, r3
+; CHECK-FIX-NOSCHED-NEXT:    lsr r4, r4, #16
+; CHECK-FIX-NOSCHED-NEXT:    lsr lr, r0, #16
+; CHECK-FIX-NOSCHED-NEXT:    b .LBB37_6
 ; CHECK-FIX-NOSCHED-NEXT:  .LBB37_5:
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s1, s1
-; CHECK-FIX-NOSCHED-NEXT:    vmov s15, r0
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, s1
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s1, s3
-; CHECK-FIX-NOSCHED-NEXT:    vmov r2, s1
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s15, s15
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s1, s15
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s11, s11
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s1
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s12, s12
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s1, s9
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s8, s8
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s6, s6
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s4, s4
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s0, s0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r0, r2, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r2, s11
+; CHECK-FIX-NOSCHED-NEXT:    vmov r3, r6, d3
+; CHECK-FIX-NOSCHED-NEXT:    vmov r0, r5, d2
+; CHECK-FIX-NOSCHED-NEXT:    lsr r4, r3, #16
+; CHECK-FIX-NOSCHED-NEXT:    lsr r9, r6, #16
+; CHECK-FIX-NOSCHED-NEXT:    lsr r12, r5, #16
+; CHECK-FIX-NOSCHED-NEXT:    lsr lr, r0, #16
+; CHECK-FIX-NOSCHED-NEXT:    uxth r11, r6
+; CHECK-FIX-NOSCHED-NEXT:    uxth r10, r3
+; CHECK-FIX-NOSCHED-NEXT:    uxth r5, r5
+; CHECK-FIX-NOSCHED-NEXT:  .LBB37_6:
+; CHECK-FIX-NOSCHED-NEXT:    uxth r8, r0
+; CHECK-FIX-NOSCHED-NEXT:    ldr r0, [sp, #4] @ 4-byte Reload
+; CHECK-FIX-NOSCHED-NEXT:    ldr r3, [sp] @ 4-byte Reload
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r3, r0, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d16[0], r0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r2, r3, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r2, s12
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s12, s14
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r8, lr, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d18[0], r0
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, s12
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s12, s13
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s12
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r5, r12, lsl #16
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d18[1], r0
+; CHECK-FIX-NOSCHED-NEXT:    ldr r0, [sp, #8] @ 4-byte Reload
 ; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r2, r0, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r2, s1
+; CHECK-FIX-NOSCHED-NEXT:    ldr r2, [sp, #16] @ 4-byte Reload
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d16[1], r0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r2, r2, r3, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d18[1], r2
-; CHECK-FIX-NOSCHED-NEXT:    vmov r2, s8
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s8, s10
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s8
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r2, r3, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r2, s6
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s6, s7
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s6
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d17[0], r0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r2, r3, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r2, s4
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s4, s5
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r10, r4, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d19[0], r0
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, s4
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r2, r0, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r2, s0
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s0, s2
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s0
+; CHECK-FIX-NOSCHED-NEXT:    ldr r0, [sp, #12] @ 4-byte Reload
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r0, r7, lsl #16
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d17[0], r0
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r11, r9, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d19[1], r0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r2, r3, lsl #16
+; CHECK-FIX-NOSCHED-NEXT:    ldr r0, [sp, #20] @ 4-byte Reload
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r0, r2, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d17[1], r0
 ; CHECK-FIX-NOSCHED-NEXT:    aese.8 q8, q9
 ; CHECK-FIX-NOSCHED-NEXT:    aesmc.8 q8, q8
 ; CHECK-FIX-NOSCHED-NEXT:    vst1.64 {d16, d17}, [r1]
-; CHECK-FIX-NOSCHED-NEXT:    pop {r4, r5, r6, r7, r11, pc}
+; CHECK-FIX-NOSCHED-NEXT:    add sp, sp, #24
+; CHECK-FIX-NOSCHED-NEXT:    pop {r4, r5, r6, r7, r8, r9, r10, r11, pc}
 ;
 ; CHECK-CORTEX-FIX-LABEL: aese_setf16_cond_via_val:
 ; CHECK-CORTEX-FIX:       @ %bb.0:
-; CHECK-CORTEX-FIX-NEXT:    .save {r4, r5, r6, r7, r11, lr}
-; CHECK-CORTEX-FIX-NEXT:    push {r4, r5, r6, r7, r11, lr}
-; CHECK-CORTEX-FIX-NEXT:    .vsave {d8, d9}
-; CHECK-CORTEX-FIX-NEXT:    vpush {d8, d9}
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s9, s0
+; CHECK-CORTEX-FIX-NEXT:    .save {r4, r5, r6, r7, r8, r9, r10, r11, lr}
+; CHECK-CORTEX-FIX-NEXT:    push {r4, r5, r6, r7, r8, r9, r10, r11, lr}
+; CHECK-CORTEX-FIX-NEXT:    .pad #28
+; CHECK-CORTEX-FIX-NEXT:    sub sp, sp, #28
+; CHECK-CORTEX-FIX-NEXT:    vmov r2, s0
 ; CHECK-CORTEX-FIX-NEXT:    cmp r0, #0
-; CHECK-CORTEX-FIX-NEXT:    beq .LBB37_3
+; CHECK-CORTEX-FIX-NEXT:    beq .LBB37_2
 ; CHECK-CORTEX-FIX-NEXT:  @ %bb.1:
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s0, s9
 ; CHECK-CORTEX-FIX-NEXT:    vld1.64 {d16, d17}, [r1]
-; CHECK-CORTEX-FIX-NEXT:    vmov r2, s0
 ; CHECK-CORTEX-FIX-NEXT:    vmov.32 r3, d16[1]
 ; CHECK-CORTEX-FIX-NEXT:    vmov.16 d16[0], r2
-; CHECK-CORTEX-FIX-NEXT:    vmov r4, r5, d17
-; CHECK-CORTEX-FIX-NEXT:    lsr lr, r3, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s8, r3
-; CHECK-CORTEX-FIX-NEXT:    vmov s11, lr
-; CHECK-CORTEX-FIX-NEXT:    lsr r6, r4, #16
-; CHECK-CORTEX-FIX-NEXT:    lsr r7, r5, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s0, r5
-; CHECK-CORTEX-FIX-NEXT:    vmov s2, r4
-; CHECK-CORTEX-FIX-NEXT:    vmov s12, r7
-; CHECK-CORTEX-FIX-NEXT:    vmov s1, r6
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s3, s8
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 r2, d16[0]
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s10, s0
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s0, s2
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s2, s1
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s14, s12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s1, s11
-; CHECK-CORTEX-FIX-NEXT:    lsr r12, r2, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s13, r2
-; CHECK-CORTEX-FIX-NEXT:    vmov s15, r12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s12, s13
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s8, s15
-; CHECK-CORTEX-FIX-NEXT:    cmp r0, #0
-; CHECK-CORTEX-FIX-NEXT:    bne .LBB37_4
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 r7, d16[0]
+; CHECK-CORTEX-FIX-NEXT:    uxth r6, r7
+; CHECK-CORTEX-FIX-NEXT:    lsr r7, r7, #16
+; CHECK-CORTEX-FIX-NEXT:    str r7, [sp, #20] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    uxth r7, r3
+; CHECK-CORTEX-FIX-NEXT:    lsr r3, r3, #16
+; CHECK-CORTEX-FIX-NEXT:    str r6, [sp, #24] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    str r7, [sp, #8] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #4] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    vmov r3, r7, d17
+; CHECK-CORTEX-FIX-NEXT:    uxth r6, r3
+; CHECK-CORTEX-FIX-NEXT:    lsr r3, r3, #16
+; CHECK-CORTEX-FIX-NEXT:    uxth r11, r7
+; CHECK-CORTEX-FIX-NEXT:    lsr r7, r7, #16
+; CHECK-CORTEX-FIX-NEXT:    str r6, [sp, #16] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    b .LBB37_3
 ; CHECK-CORTEX-FIX-NEXT:  .LBB37_2:
-; CHECK-CORTEX-FIX-NEXT:    vmov r6, r5, d3
-; CHECK-CORTEX-FIX-NEXT:    vmov r0, r2, d2
-; CHECK-CORTEX-FIX-NEXT:    lsr r7, r2, #16
-; CHECK-CORTEX-FIX-NEXT:    lsr r4, r6, #16
-; CHECK-CORTEX-FIX-NEXT:    lsr r3, r5, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s6, r6
-; CHECK-CORTEX-FIX-NEXT:    vmov s4, r5
-; CHECK-CORTEX-FIX-NEXT:    vmov s7, r2
-; CHECK-CORTEX-FIX-NEXT:    vmov s9, r0
-; CHECK-CORTEX-FIX-NEXT:    lsr r12, r0, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s13, r3
-; CHECK-CORTEX-FIX-NEXT:    vmov s15, r4
-; CHECK-CORTEX-FIX-NEXT:    vmov s16, r7
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s5, s4
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s11, s7
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s7, s9
-; CHECK-CORTEX-FIX-NEXT:    vmov s4, r12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s6, s6
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s9, s15
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s13, s13
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s15, s16
-; CHECK-CORTEX-FIX-NEXT:    b .LBB37_5
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r1]
+; CHECK-CORTEX-FIX-NEXT:    ldrh r11, [r1, #12]
+; CHECK-CORTEX-FIX-NEXT:    ldrh r7, [r1, #14]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #24] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r1, #2]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #20] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r1, #4]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #8] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r1, #6]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #4] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r1, #8]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #16] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r1, #10]
 ; CHECK-CORTEX-FIX-NEXT:  .LBB37_3:
-; CHECK-CORTEX-FIX-NEXT:    ldrh r12, [r1]
-; CHECK-CORTEX-FIX-NEXT:    ldrh lr, [r1, #2]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r7, [r1, #4]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r6, [r1, #6]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r5, [r1, #8]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r4, [r1, #10]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r2, [r1, #12]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r1, #14]
-; CHECK-CORTEX-FIX-NEXT:    vmov s1, r6
-; CHECK-CORTEX-FIX-NEXT:    vmov s3, r7
-; CHECK-CORTEX-FIX-NEXT:    vmov s0, r3
-; CHECK-CORTEX-FIX-NEXT:    vmov s2, r2
-; CHECK-CORTEX-FIX-NEXT:    vmov s8, r4
-; CHECK-CORTEX-FIX-NEXT:    vmov s12, r5
-; CHECK-CORTEX-FIX-NEXT:    vmov s11, lr
-; CHECK-CORTEX-FIX-NEXT:    vmov s13, r12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s14, s0
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s10, s2
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s2, s8
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s0, s12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s1, s1
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s3, s3
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s8, s11
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s12, s13
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #12] @ 4-byte Spill
 ; CHECK-CORTEX-FIX-NEXT:    cmp r0, #0
-; CHECK-CORTEX-FIX-NEXT:    beq .LBB37_2
-; CHECK-CORTEX-FIX-NEXT:  .LBB37_4:
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s9, s9
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 r2, d2[1]
-; CHECK-CORTEX-FIX-NEXT:    vmov r0, s9
-; CHECK-CORTEX-FIX-NEXT:    lsr r7, r2, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s16, r7
-; CHECK-CORTEX-FIX-NEXT:    vmov.16 d2[0], r0
-; CHECK-CORTEX-FIX-NEXT:    vmov r6, r5, d3
-; CHECK-CORTEX-FIX-NEXT:    vmov s7, r2
-; CHECK-CORTEX-FIX-NEXT:    lsr r4, r6, #16
-; CHECK-CORTEX-FIX-NEXT:    lsr r3, r5, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s6, r6
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s11, s7
-; CHECK-CORTEX-FIX-NEXT:    vmov s9, r3
-; CHECK-CORTEX-FIX-NEXT:    vmov s15, r4
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 r0, d2[0]
-; CHECK-CORTEX-FIX-NEXT:    vmov s4, r5
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s6, s6
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s13, s9
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s9, s15
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s15, s16
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s5, s4
-; CHECK-CORTEX-FIX-NEXT:    vmov s18, r0
-; CHECK-CORTEX-FIX-NEXT:    lsr r12, r0, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s4, r12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s7, s18
+; CHECK-CORTEX-FIX-NEXT:    beq .LBB37_5
+; CHECK-CORTEX-FIX-NEXT:  @ %bb.4:
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 r3, d2[1]
+; CHECK-CORTEX-FIX-NEXT:    vmov.16 d2[0], r2
+; CHECK-CORTEX-FIX-NEXT:    vmov r4, r6, d3
+; CHECK-CORTEX-FIX-NEXT:    uxth r10, r4
+; CHECK-CORTEX-FIX-NEXT:    lsr r4, r4, #16
+; CHECK-CORTEX-FIX-NEXT:    uxth lr, r6
+; CHECK-CORTEX-FIX-NEXT:    lsr r8, r6, #16
+; CHECK-CORTEX-FIX-NEXT:    uxth r5, r3
+; CHECK-CORTEX-FIX-NEXT:    lsr r12, r3, #16
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 r2, d2[0]
+; CHECK-CORTEX-FIX-NEXT:    uxth r0, r2
+; CHECK-CORTEX-FIX-NEXT:    lsr r9, r2, #16
+; CHECK-CORTEX-FIX-NEXT:    str r0, [sp] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    b .LBB37_6
 ; CHECK-CORTEX-FIX-NEXT:  .LBB37_5:
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s10, s10
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s14, s14
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s3, s3
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s2, s2
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s0, s0
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s8, s8
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s12, s12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s4, s4
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s6, s6
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s9, s9
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s7, s7
-; CHECK-CORTEX-FIX-NEXT:    vmov r0, s10
-; CHECK-CORTEX-FIX-NEXT:    vmov r2, s14
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s10, s1
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s14, s5
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s1, s13
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s5, s11
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s11, s15
-; CHECK-CORTEX-FIX-NEXT:    vmov r5, s2
-; CHECK-CORTEX-FIX-NEXT:    vmov r4, s8
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r12, r0, r2, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r2, s3
-; CHECK-CORTEX-FIX-NEXT:    vmov r3, s10
-; CHECK-CORTEX-FIX-NEXT:    vmov r7, s1
-; CHECK-CORTEX-FIX-NEXT:    vmov r6, s11
-; CHECK-CORTEX-FIX-NEXT:    vmov r0, s9
-; CHECK-CORTEX-FIX-NEXT:    pkhbt lr, r2, r3, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r3, s14
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r3, r3, r7, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r7, s5
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r7, r7, r6, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r6, s0
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s0, s4
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r6, r6, r5, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r5, s12
-; CHECK-CORTEX-FIX-NEXT:    vmov r2, s0
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r5, r5, r4, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r4, s6
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d18[0], r5
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d19[0], r6
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d18[1], lr
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d19[1], r12
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r0, r4, r0, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r4, s7
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r2, r4, r2, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d16[0], r2
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d17[0], r0
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d16[1], r7
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d17[1], r3
+; CHECK-CORTEX-FIX-NEXT:    vmov r2, r3, d2
+; CHECK-CORTEX-FIX-NEXT:    uxth r0, r2
+; CHECK-CORTEX-FIX-NEXT:    lsr r9, r2, #16
+; CHECK-CORTEX-FIX-NEXT:    uxth r5, r3
+; CHECK-CORTEX-FIX-NEXT:    lsr r12, r3, #16
+; CHECK-CORTEX-FIX-NEXT:    str r0, [sp] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    mov r0, r7
+; CHECK-CORTEX-FIX-NEXT:    vmov r6, r7, d3
+; CHECK-CORTEX-FIX-NEXT:    uxth r10, r6
+; CHECK-CORTEX-FIX-NEXT:    lsr r4, r6, #16
+; CHECK-CORTEX-FIX-NEXT:    uxth lr, r7
+; CHECK-CORTEX-FIX-NEXT:    lsr r8, r7, #16
+; CHECK-CORTEX-FIX-NEXT:    mov r7, r0
+; CHECK-CORTEX-FIX-NEXT:  .LBB37_6:
+; CHECK-CORTEX-FIX-NEXT:    ldr r0, [sp, #8] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    ldr r2, [sp, #4] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r11, r11, r7, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    ldr r3, [sp, #12] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    ldr r6, [sp, #20] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r5, r5, r12, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r4, r10, r4, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r7, r0, r2, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    ldr r2, [sp, #16] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r0, lr, r8, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r2, r2, r3, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    ldr r3, [sp, #24] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r3, r3, r6, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    ldr r6, [sp] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d18[0], r3
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d19[0], r2
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d18[1], r7
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d19[1], r11
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r6, r6, r9, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d16[0], r6
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d17[0], r4
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d16[1], r5
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d17[1], r0
 ; CHECK-CORTEX-FIX-NEXT:    aese.8 q9, q8
 ; CHECK-CORTEX-FIX-NEXT:    aesmc.8 q8, q9
 ; CHECK-CORTEX-FIX-NEXT:    vst1.64 {d16, d17}, [r1]
-; CHECK-CORTEX-FIX-NEXT:    vpop {d8, d9}
-; CHECK-CORTEX-FIX-NEXT:    pop {r4, r5, r6, r7, r11, pc}
+; CHECK-CORTEX-FIX-NEXT:    add sp, sp, #28
+; CHECK-CORTEX-FIX-NEXT:    pop {r4, r5, r6, r7, r8, r9, r10, r11, pc}
   br i1 %0, label %5, label %11
 
 5:
@@ -2159,10 +1906,7 @@ define arm_aapcs_vfpcc void @aese_setf16_loop_via_val(i32 %0, half %1, <16 x i8>
 ; CHECK-FIX-NEXT:    cmp r0, #0
 ; CHECK-FIX-NEXT:    bxeq lr
 ; CHECK-FIX-NEXT:  .LBB39_1:
-; CHECK-FIX-NEXT:    vcvtb.f32.f16 s0, s0
-; CHECK-FIX-NEXT:    vcvtb.f16.f32 s0, s0
 ; CHECK-FIX-NEXT:    vmov r2, s0
-; CHECK-FIX-NEXT:    uxth r2, r2
 ; CHECK-FIX-NEXT:    vmov.16 d2[0], r2
 ; CHECK-FIX-NEXT:  .LBB39_2: @ =>This Inner Loop Header: Depth=1
 ; CHECK-FIX-NEXT:    vld1.64 {d16, d17}, [r1]
@@ -3821,335 +3565,210 @@ define arm_aapcs_vfpcc void @aesd_setf16_via_val(half %0, <16 x i8> %1, <16 x i8
 define arm_aapcs_vfpcc void @aesd_setf16_cond_via_ptr(i1 zeroext %0, half* %1, <16 x i8> %2, <16 x i8>* %3) nounwind {
 ; CHECK-FIX-NOSCHED-LABEL: aesd_setf16_cond_via_ptr:
 ; CHECK-FIX-NOSCHED:       @ %bb.0:
-; CHECK-FIX-NOSCHED-NEXT:    .save {r4, r5, r6, r7, r8, lr}
-; CHECK-FIX-NOSCHED-NEXT:    push {r4, r5, r6, r7, r8, lr}
+; CHECK-FIX-NOSCHED-NEXT:    .save {r4, r5, r6, r7, r8, r9, r10, r11, lr}
+; CHECK-FIX-NOSCHED-NEXT:    push {r4, r5, r6, r7, r8, r9, r10, r11, lr}
+; CHECK-FIX-NOSCHED-NEXT:    .pad #24
+; CHECK-FIX-NOSCHED-NEXT:    sub sp, sp, #24
 ; CHECK-FIX-NOSCHED-NEXT:    cmp r0, #0
 ; CHECK-FIX-NOSCHED-NEXT:    beq .LBB82_3
 ; CHECK-FIX-NOSCHED-NEXT:  @ %bb.1:
 ; CHECK-FIX-NOSCHED-NEXT:    vld1.64 {d16, d17}, [r2]
-; CHECK-FIX-NOSCHED-NEXT:    vorr q9, q8, q8
-; CHECK-FIX-NOSCHED-NEXT:    vmov lr, r12, d17
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r3, d16[1]
-; CHECK-FIX-NOSCHED-NEXT:    vld1.16 {d18[0]}, [r1:16]
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r4, d18[0]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s6, lr
-; CHECK-FIX-NOSCHED-NEXT:    lsr r5, r3, #16
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s8, s6
-; CHECK-FIX-NOSCHED-NEXT:    vmov s6, r3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s12, s6
-; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r12, #16
-; CHECK-FIX-NOSCHED-NEXT:    lsr lr, lr, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s4, r12
-; CHECK-FIX-NOSCHED-NEXT:    vmov s10, lr
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s4, s4
-; CHECK-FIX-NOSCHED-NEXT:    vmov s14, r5
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s10, s10
-; CHECK-FIX-NOSCHED-NEXT:    vmov s6, r4
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s14, s14
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s5, s6
-; CHECK-FIX-NOSCHED-NEXT:    vmov s6, r3
-; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r4, #16
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s6, s6
-; CHECK-FIX-NOSCHED-NEXT:    vmov s7, r3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s7, s7
+; CHECK-FIX-NOSCHED-NEXT:    vmov r7, r6, d17
+; CHECK-FIX-NOSCHED-NEXT:    vld1.16 {d16[0]}, [r1:16]
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r5, d16[0]
+; CHECK-FIX-NOSCHED-NEXT:    uxth r4, r3
+; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r3, #16
+; CHECK-FIX-NOSCHED-NEXT:    str r4, [sp, #8] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    uxth r4, r7
+; CHECK-FIX-NOSCHED-NEXT:    str r4, [sp, #12] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    uxth r4, r6
+; CHECK-FIX-NOSCHED-NEXT:    lsr r6, r6, #16
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #4] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    str r4, [sp, #20] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    lsr r7, r7, #16
+; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r5, #16
+; CHECK-FIX-NOSCHED-NEXT:    str r6, [sp, #16] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    uxth r10, r5
 ; CHECK-FIX-NOSCHED-NEXT:    cmp r0, #0
 ; CHECK-FIX-NOSCHED-NEXT:    bne .LBB82_4
 ; CHECK-FIX-NOSCHED-NEXT:  .LBB82_2:
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, r1, d0
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, r7, d1
-; CHECK-FIX-NOSCHED-NEXT:    vmov s1, r1
-; CHECK-FIX-NOSCHED-NEXT:    lsr r1, r1, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s0, r7
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s9, s1
-; CHECK-FIX-NOSCHED-NEXT:    vmov s2, r3
-; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r3, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s1, r0
-; CHECK-FIX-NOSCHED-NEXT:    lsr r7, r7, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s3, r3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s11, s1
-; CHECK-FIX-NOSCHED-NEXT:    vmov s1, r7
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s0, s0
-; CHECK-FIX-NOSCHED-NEXT:    vmov s13, r1
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s2, s2
-; CHECK-FIX-NOSCHED-NEXT:    lsr r0, r0, #16
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s1, s1
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s3, s3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s13, s13
+; CHECK-FIX-NOSCHED-NEXT:    vmov r4, r6, d1
+; CHECK-FIX-NOSCHED-NEXT:    vmov r0, r3, d0
+; CHECK-FIX-NOSCHED-NEXT:    lsr r5, r4, #16
+; CHECK-FIX-NOSCHED-NEXT:    lsr r1, r6, #16
+; CHECK-FIX-NOSCHED-NEXT:    uxth r11, r6
+; CHECK-FIX-NOSCHED-NEXT:    lsr lr, r0, #16
+; CHECK-FIX-NOSCHED-NEXT:    lsr r12, r3, #16
+; CHECK-FIX-NOSCHED-NEXT:    uxth r9, r4
+; CHECK-FIX-NOSCHED-NEXT:    uxth r6, r3
 ; CHECK-FIX-NOSCHED-NEXT:    b .LBB82_5
 ; CHECK-FIX-NOSCHED-NEXT:  .LBB82_3:
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r5, [r2, #10]
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r12, [r2, #6]
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r6, [r2, #2]
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r7, [r2, #14]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s8, r5
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r2, #14]
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #16] @ 4-byte Spill
 ; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r2, #12]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s12, r12
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r4, [r2, #8]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s5, r6
-; CHECK-FIX-NOSCHED-NEXT:    ldrh lr, [r2, #4]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s4, r7
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r8, [r2]
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s6, s4
-; CHECK-FIX-NOSCHED-NEXT:    vmov s4, r3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s10, s8
-; CHECK-FIX-NOSCHED-NEXT:    vmov s8, r4
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s14, s12
-; CHECK-FIX-NOSCHED-NEXT:    vmov s12, lr
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s7, s5
-; CHECK-FIX-NOSCHED-NEXT:    vmov s5, r8
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s4, s4
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s8, s8
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s12, s12
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s5, s5
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #20] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r2, #8]
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #12] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r2, #6]
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r7, [r2, #10]
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r10, [r2]
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #4] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r2, #4]
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #8] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r2, #2]
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp] @ 4-byte Spill
 ; CHECK-FIX-NOSCHED-NEXT:    cmp r0, #0
 ; CHECK-FIX-NOSCHED-NEXT:    beq .LBB82_2
 ; CHECK-FIX-NOSCHED-NEXT:  .LBB82_4:
-; CHECK-FIX-NOSCHED-NEXT:    vorr q8, q0, q0
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r3, d0[1]
-; CHECK-FIX-NOSCHED-NEXT:    vld1.16 {d16[0]}, [r1:16]
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, r1, d1
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r7, d16[0]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s1, r3
-; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r3, #16
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s9, s1
-; CHECK-FIX-NOSCHED-NEXT:    vmov s13, r3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s13, s13
-; CHECK-FIX-NOSCHED-NEXT:    vmov s0, r1
-; CHECK-FIX-NOSCHED-NEXT:    lsr r1, r1, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s2, r0
-; CHECK-FIX-NOSCHED-NEXT:    lsr r0, r0, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s3, r0
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s0, s0
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s2, s2
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s3, s3
-; CHECK-FIX-NOSCHED-NEXT:    vmov s1, r7
-; CHECK-FIX-NOSCHED-NEXT:    lsr r0, r7, #16
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s11, s1
-; CHECK-FIX-NOSCHED-NEXT:    vmov s1, r1
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s1, s1
+; CHECK-FIX-NOSCHED-NEXT:    vmov r5, r3, d1
+; CHECK-FIX-NOSCHED-NEXT:    mov r4, r7
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r7, d0[1]
+; CHECK-FIX-NOSCHED-NEXT:    vld1.16 {d0[0]}, [r1:16]
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r0, d0[0]
+; CHECK-FIX-NOSCHED-NEXT:    uxth r9, r5
+; CHECK-FIX-NOSCHED-NEXT:    uxth r11, r3
+; CHECK-FIX-NOSCHED-NEXT:    uxth r6, r7
+; CHECK-FIX-NOSCHED-NEXT:    lsr r12, r7, #16
+; CHECK-FIX-NOSCHED-NEXT:    lsr r1, r3, #16
+; CHECK-FIX-NOSCHED-NEXT:    lsr r5, r5, #16
+; CHECK-FIX-NOSCHED-NEXT:    mov r7, r4
+; CHECK-FIX-NOSCHED-NEXT:    lsr lr, r0, #16
 ; CHECK-FIX-NOSCHED-NEXT:  .LBB82_5:
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s5, s5
-; CHECK-FIX-NOSCHED-NEXT:    vmov s15, r0
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, s5
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s5, s7
-; CHECK-FIX-NOSCHED-NEXT:    vmov r1, s5
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s15, s15
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s5, s15
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s11, s11
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s5
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s12, s12
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s5, s9
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s8, s8
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s2, s2
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s0, s0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r0, r1, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r1, s11
+; CHECK-FIX-NOSCHED-NEXT:    uxth r8, r0
+; CHECK-FIX-NOSCHED-NEXT:    ldr r0, [sp] @ 4-byte Reload
+; CHECK-FIX-NOSCHED-NEXT:    ldr r3, [sp, #4] @ 4-byte Reload
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r10, r0, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d16[0], r0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r1, r3, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r1, s12
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s12, s14
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r8, lr, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d18[0], r0
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, s12
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s12, s13
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s12
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r1, r0, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r1, s5
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r6, r12, lsl #16
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d18[1], r0
+; CHECK-FIX-NOSCHED-NEXT:    ldr r0, [sp, #8] @ 4-byte Reload
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r0, r3, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d16[1], r0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r1, r1, r3, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d18[1], r1
-; CHECK-FIX-NOSCHED-NEXT:    vmov r1, s8
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s8, s10
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s8
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r1, r3, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r1, s2
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s2, s3
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s2
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d17[0], r0
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s2, s4
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r1, r3, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r1, s0
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s0, s1
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r9, r5, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d19[0], r0
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, s0
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s0, s6
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r1, r0, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r1, s2
+; CHECK-FIX-NOSCHED-NEXT:    ldr r0, [sp, #12] @ 4-byte Reload
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r0, r7, lsl #16
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d17[0], r0
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r11, r1, lsl #16
+; CHECK-FIX-NOSCHED-NEXT:    ldr r1, [sp, #16] @ 4-byte Reload
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d19[1], r0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r1, r3, lsl #16
+; CHECK-FIX-NOSCHED-NEXT:    ldr r0, [sp, #20] @ 4-byte Reload
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r0, r1, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d17[1], r0
 ; CHECK-FIX-NOSCHED-NEXT:    aesd.8 q8, q9
 ; CHECK-FIX-NOSCHED-NEXT:    aesimc.8 q8, q8
 ; CHECK-FIX-NOSCHED-NEXT:    vst1.64 {d16, d17}, [r2]
-; CHECK-FIX-NOSCHED-NEXT:    pop {r4, r5, r6, r7, r8, pc}
+; CHECK-FIX-NOSCHED-NEXT:    add sp, sp, #24
+; CHECK-FIX-NOSCHED-NEXT:    pop {r4, r5, r6, r7, r8, r9, r10, r11, pc}
 ;
 ; CHECK-CORTEX-FIX-LABEL: aesd_setf16_cond_via_ptr:
 ; CHECK-CORTEX-FIX:       @ %bb.0:
-; CHECK-CORTEX-FIX-NEXT:    .save {r4, r5, r6, r7, r8, lr}
-; CHECK-CORTEX-FIX-NEXT:    push {r4, r5, r6, r7, r8, lr}
-; CHECK-CORTEX-FIX-NEXT:    .vsave {d8, d9}
-; CHECK-CORTEX-FIX-NEXT:    vpush {d8, d9}
+; CHECK-CORTEX-FIX-NEXT:    .save {r4, r5, r6, r7, r8, r9, r10, r11, lr}
+; CHECK-CORTEX-FIX-NEXT:    push {r4, r5, r6, r7, r8, r9, r10, r11, lr}
+; CHECK-CORTEX-FIX-NEXT:    .pad #24
+; CHECK-CORTEX-FIX-NEXT:    sub sp, sp, #24
 ; CHECK-CORTEX-FIX-NEXT:    cmp r0, #0
 ; CHECK-CORTEX-FIX-NEXT:    beq .LBB82_3
 ; CHECK-CORTEX-FIX-NEXT:  @ %bb.1:
 ; CHECK-CORTEX-FIX-NEXT:    vld1.64 {d16, d17}, [r2]
 ; CHECK-CORTEX-FIX-NEXT:    vorr q9, q8, q8
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 r3, d16[1]
-; CHECK-CORTEX-FIX-NEXT:    vmov r5, r6, d17
 ; CHECK-CORTEX-FIX-NEXT:    vld1.16 {d18[0]}, [r1:16]
-; CHECK-CORTEX-FIX-NEXT:    lsr r7, r5, #16
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 r3, d18[0]
+; CHECK-CORTEX-FIX-NEXT:    uxth r7, r3
+; CHECK-CORTEX-FIX-NEXT:    lsr r3, r3, #16
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #16] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 r3, d16[1]
+; CHECK-CORTEX-FIX-NEXT:    str r7, [sp, #20] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    uxth r7, r3
+; CHECK-CORTEX-FIX-NEXT:    lsr r3, r3, #16
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #4] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    vmov r3, r6, d17
+; CHECK-CORTEX-FIX-NEXT:    str r7, [sp, #8] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    uxth r7, r3
+; CHECK-CORTEX-FIX-NEXT:    lsr r3, r3, #16
+; CHECK-CORTEX-FIX-NEXT:    uxth r11, r6
 ; CHECK-CORTEX-FIX-NEXT:    lsr r4, r6, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s4, r6
-; CHECK-CORTEX-FIX-NEXT:    vmov s6, r5
-; CHECK-CORTEX-FIX-NEXT:    lsr r8, r3, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s8, r3
-; CHECK-CORTEX-FIX-NEXT:    vmov s12, r4
-; CHECK-CORTEX-FIX-NEXT:    vmov s5, r7
-; CHECK-CORTEX-FIX-NEXT:    vmov s9, r8
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s10, s4
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s4, s6
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 lr, d18[0]
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s6, s5
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s7, s8
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s14, s12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s5, s9
-; CHECK-CORTEX-FIX-NEXT:    lsr r12, lr, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s11, lr
-; CHECK-CORTEX-FIX-NEXT:    vmov s13, r12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s12, s11
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s8, s13
+; CHECK-CORTEX-FIX-NEXT:    str r7, [sp, #12] @ 4-byte Spill
 ; CHECK-CORTEX-FIX-NEXT:    cmp r0, #0
 ; CHECK-CORTEX-FIX-NEXT:    bne .LBB82_4
 ; CHECK-CORTEX-FIX-NEXT:  .LBB82_2:
-; CHECK-CORTEX-FIX-NEXT:    vmov r6, r5, d1
-; CHECK-CORTEX-FIX-NEXT:    vmov r0, r1, d0
-; CHECK-CORTEX-FIX-NEXT:    lsr r7, r1, #16
-; CHECK-CORTEX-FIX-NEXT:    lsr r4, r6, #16
-; CHECK-CORTEX-FIX-NEXT:    lsr r3, r5, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s2, r6
-; CHECK-CORTEX-FIX-NEXT:    vmov s0, r5
-; CHECK-CORTEX-FIX-NEXT:    vmov s3, r1
-; CHECK-CORTEX-FIX-NEXT:    vmov s9, r0
-; CHECK-CORTEX-FIX-NEXT:    lsr r12, r0, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s13, r3
-; CHECK-CORTEX-FIX-NEXT:    vmov s15, r4
-; CHECK-CORTEX-FIX-NEXT:    vmov s16, r7
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s1, s0
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s11, s3
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s3, s9
-; CHECK-CORTEX-FIX-NEXT:    vmov s0, r12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s2, s2
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s9, s15
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s13, s13
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s15, s16
+; CHECK-CORTEX-FIX-NEXT:    vmov r1, r7, d0
+; CHECK-CORTEX-FIX-NEXT:    uxth r0, r1
+; CHECK-CORTEX-FIX-NEXT:    uxth r6, r7
+; CHECK-CORTEX-FIX-NEXT:    lsr r12, r7, #16
+; CHECK-CORTEX-FIX-NEXT:    lsr r9, r1, #16
+; CHECK-CORTEX-FIX-NEXT:    str r0, [sp] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    mov r0, r3
+; CHECK-CORTEX-FIX-NEXT:    vmov r7, r3, d1
+; CHECK-CORTEX-FIX-NEXT:    uxth r10, r7
+; CHECK-CORTEX-FIX-NEXT:    lsr r5, r7, #16
+; CHECK-CORTEX-FIX-NEXT:    uxth lr, r3
+; CHECK-CORTEX-FIX-NEXT:    lsr r8, r3, #16
+; CHECK-CORTEX-FIX-NEXT:    mov r3, r0
 ; CHECK-CORTEX-FIX-NEXT:    b .LBB82_5
 ; CHECK-CORTEX-FIX-NEXT:  .LBB82_3:
-; CHECK-CORTEX-FIX-NEXT:    ldrh r12, [r2]
-; CHECK-CORTEX-FIX-NEXT:    ldrh lr, [r2, #2]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r8, [r2, #4]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r5, [r2, #6]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r4, [r2, #8]
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r2]
+; CHECK-CORTEX-FIX-NEXT:    ldrh r11, [r2, #12]
+; CHECK-CORTEX-FIX-NEXT:    ldrh r4, [r2, #14]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #20] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r2, #2]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #16] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r2, #4]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #8] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r2, #6]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #4] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r2, #8]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #12] @ 4-byte Spill
 ; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r2, #10]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r7, [r2, #12]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r6, [r2, #14]
-; CHECK-CORTEX-FIX-NEXT:    vmov s5, r5
-; CHECK-CORTEX-FIX-NEXT:    vmov s7, r8
-; CHECK-CORTEX-FIX-NEXT:    vmov s4, r6
-; CHECK-CORTEX-FIX-NEXT:    vmov s6, r7
-; CHECK-CORTEX-FIX-NEXT:    vmov s8, r3
-; CHECK-CORTEX-FIX-NEXT:    vmov s12, r4
-; CHECK-CORTEX-FIX-NEXT:    vmov s9, lr
-; CHECK-CORTEX-FIX-NEXT:    vmov s11, r12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s14, s4
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s10, s6
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s6, s8
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s4, s12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s5, s5
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s7, s7
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s8, s9
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s12, s11
 ; CHECK-CORTEX-FIX-NEXT:    cmp r0, #0
 ; CHECK-CORTEX-FIX-NEXT:    beq .LBB82_2
 ; CHECK-CORTEX-FIX-NEXT:  .LBB82_4:
 ; CHECK-CORTEX-FIX-NEXT:    vorr q8, q0, q0
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 r3, d0[1]
-; CHECK-CORTEX-FIX-NEXT:    vmov r6, r5, d1
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 r5, d0[1]
 ; CHECK-CORTEX-FIX-NEXT:    vld1.16 {d16[0]}, [r1:16]
-; CHECK-CORTEX-FIX-NEXT:    lsr r4, r6, #16
-; CHECK-CORTEX-FIX-NEXT:    lsr r1, r5, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s2, r6
-; CHECK-CORTEX-FIX-NEXT:    vmov s0, r5
-; CHECK-CORTEX-FIX-NEXT:    lsr r7, r3, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s3, r3
-; CHECK-CORTEX-FIX-NEXT:    vmov s9, r1
-; CHECK-CORTEX-FIX-NEXT:    vmov s15, r4
-; CHECK-CORTEX-FIX-NEXT:    vmov s16, r7
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s1, s0
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s2, s2
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 r0, d16[0]
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s13, s9
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s9, s15
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s11, s3
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s15, s16
-; CHECK-CORTEX-FIX-NEXT:    vmov s18, r0
-; CHECK-CORTEX-FIX-NEXT:    lsr r12, r0, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s0, r12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s3, s18
+; CHECK-CORTEX-FIX-NEXT:    uxth r6, r5
+; CHECK-CORTEX-FIX-NEXT:    lsr r12, r5, #16
+; CHECK-CORTEX-FIX-NEXT:    vmov r5, r7, d1
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 r1, d16[0]
+; CHECK-CORTEX-FIX-NEXT:    uxth r10, r5
+; CHECK-CORTEX-FIX-NEXT:    lsr r5, r5, #16
+; CHECK-CORTEX-FIX-NEXT:    uxth lr, r7
+; CHECK-CORTEX-FIX-NEXT:    lsr r8, r7, #16
+; CHECK-CORTEX-FIX-NEXT:    uxth r0, r1
+; CHECK-CORTEX-FIX-NEXT:    lsr r9, r1, #16
+; CHECK-CORTEX-FIX-NEXT:    str r0, [sp] @ 4-byte Spill
 ; CHECK-CORTEX-FIX-NEXT:  .LBB82_5:
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s10, s10
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s14, s14
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s7, s7
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s6, s6
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s4, s4
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s8, s8
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s12, s12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s0, s0
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s2, s2
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s9, s9
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s3, s3
-; CHECK-CORTEX-FIX-NEXT:    vmov r0, s10
-; CHECK-CORTEX-FIX-NEXT:    vmov r1, s14
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s10, s5
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s14, s1
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s1, s13
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s5, s11
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s11, s15
-; CHECK-CORTEX-FIX-NEXT:    vmov r5, s6
-; CHECK-CORTEX-FIX-NEXT:    vmov r4, s8
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s0, s0
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r12, r0, r1, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r1, s7
-; CHECK-CORTEX-FIX-NEXT:    vmov r3, s10
-; CHECK-CORTEX-FIX-NEXT:    vmov r7, s1
-; CHECK-CORTEX-FIX-NEXT:    vmov r6, s11
-; CHECK-CORTEX-FIX-NEXT:    vmov r0, s9
-; CHECK-CORTEX-FIX-NEXT:    pkhbt lr, r1, r3, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r3, s14
-; CHECK-CORTEX-FIX-NEXT:    vmov r1, s0
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r3, r3, r7, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r7, s5
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r7, r7, r6, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r6, s4
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r6, r6, r5, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r5, s12
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r5, r5, r4, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r4, s2
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d18[0], r5
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d19[0], r6
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d18[1], lr
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d19[1], r12
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r0, r4, r0, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r4, s3
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r1, r4, r1, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d16[0], r1
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d17[0], r0
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d16[1], r7
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d17[1], r3
+; CHECK-CORTEX-FIX-NEXT:    ldr r0, [sp, #8] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    ldr r1, [sp, #4] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r11, r11, r4, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    ldr r4, [sp, #16] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r6, r6, r12, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r5, r10, r5, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r7, r0, r1, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    ldr r1, [sp, #12] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r0, lr, r8, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r1, r1, r3, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    ldr r3, [sp, #20] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r4, r3, r4, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    ldr r3, [sp] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d18[0], r4
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d19[0], r1
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d18[1], r7
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d19[1], r11
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r3, r3, r9, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d16[0], r3
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d17[0], r5
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d16[1], r6
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d17[1], r0
 ; CHECK-CORTEX-FIX-NEXT:    aesd.8 q9, q8
 ; CHECK-CORTEX-FIX-NEXT:    aesimc.8 q8, q9
 ; CHECK-CORTEX-FIX-NEXT:    vst1.64 {d16, d17}, [r2]
-; CHECK-CORTEX-FIX-NEXT:    vpop {d8, d9}
-; CHECK-CORTEX-FIX-NEXT:    pop {r4, r5, r6, r7, r8, pc}
+; CHECK-CORTEX-FIX-NEXT:    add sp, sp, #24
+; CHECK-CORTEX-FIX-NEXT:    pop {r4, r5, r6, r7, r8, r9, r10, r11, pc}
   br i1 %0, label %5, label %12
 
 5:
@@ -4195,341 +3814,213 @@ define arm_aapcs_vfpcc void @aesd_setf16_cond_via_ptr(i1 zeroext %0, half* %1, <
 define arm_aapcs_vfpcc void @aesd_setf16_cond_via_val(i1 zeroext %0, half %1, <16 x i8> %2, <16 x i8>* %3) nounwind {
 ; CHECK-FIX-NOSCHED-LABEL: aesd_setf16_cond_via_val:
 ; CHECK-FIX-NOSCHED:       @ %bb.0:
-; CHECK-FIX-NOSCHED-NEXT:    .save {r4, r5, r6, r7, r11, lr}
-; CHECK-FIX-NOSCHED-NEXT:    push {r4, r5, r6, r7, r11, lr}
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s9, s0
+; CHECK-FIX-NOSCHED-NEXT:    .save {r4, r5, r6, r7, r8, r9, r10, r11, lr}
+; CHECK-FIX-NOSCHED-NEXT:    push {r4, r5, r6, r7, r8, r9, r10, r11, lr}
+; CHECK-FIX-NOSCHED-NEXT:    .pad #24
+; CHECK-FIX-NOSCHED-NEXT:    sub sp, sp, #24
+; CHECK-FIX-NOSCHED-NEXT:    vmov r12, s0
 ; CHECK-FIX-NOSCHED-NEXT:    cmp r0, #0
-; CHECK-FIX-NOSCHED-NEXT:    beq .LBB83_3
+; CHECK-FIX-NOSCHED-NEXT:    beq .LBB83_2
 ; CHECK-FIX-NOSCHED-NEXT:  @ %bb.1:
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s0, s9
 ; CHECK-FIX-NOSCHED-NEXT:    vld1.64 {d16, d17}, [r1]
-; CHECK-FIX-NOSCHED-NEXT:    vmov r2, s0
-; CHECK-FIX-NOSCHED-NEXT:    vmov lr, r12, d17
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r3, d16[1]
-; CHECK-FIX-NOSCHED-NEXT:    vmov.16 d16[0], r2
-; CHECK-FIX-NOSCHED-NEXT:    vmov s2, lr
-; CHECK-FIX-NOSCHED-NEXT:    lsr lr, lr, #16
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s8, s2
-; CHECK-FIX-NOSCHED-NEXT:    vmov s2, r3
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r2, d16[0]
-; CHECK-FIX-NOSCHED-NEXT:    lsr r4, r3, #16
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s12, s2
-; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r12, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s0, r12
-; CHECK-FIX-NOSCHED-NEXT:    vmov s10, lr
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s0, s0
-; CHECK-FIX-NOSCHED-NEXT:    vmov s14, r4
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s10, s10
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s14, s14
-; CHECK-FIX-NOSCHED-NEXT:    vmov s2, r2
-; CHECK-FIX-NOSCHED-NEXT:    lsr r2, r2, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s3, r2
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s1, s2
-; CHECK-FIX-NOSCHED-NEXT:    vmov s2, r3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s3, s3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s2, s2
-; CHECK-FIX-NOSCHED-NEXT:    cmp r0, #0
-; CHECK-FIX-NOSCHED-NEXT:    bne .LBB83_4
-; CHECK-FIX-NOSCHED-NEXT:  .LBB83_2:
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, r2, d2
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, r7, d3
-; CHECK-FIX-NOSCHED-NEXT:    vmov s5, r2
-; CHECK-FIX-NOSCHED-NEXT:    lsr r2, r2, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s4, r7
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s9, s5
-; CHECK-FIX-NOSCHED-NEXT:    vmov s6, r3
+; CHECK-FIX-NOSCHED-NEXT:    vmov r7, r6, d17
+; CHECK-FIX-NOSCHED-NEXT:    vmov.16 d16[0], r12
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r5, d16[0]
+; CHECK-FIX-NOSCHED-NEXT:    uxth r4, r7
+; CHECK-FIX-NOSCHED-NEXT:    uxth r2, r3
 ; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r3, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s5, r0
+; CHECK-FIX-NOSCHED-NEXT:    str r4, [sp, #12] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    uxth r4, r6
+; CHECK-FIX-NOSCHED-NEXT:    lsr r6, r6, #16
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #8] @ 4-byte Spill
 ; CHECK-FIX-NOSCHED-NEXT:    lsr r7, r7, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s7, r3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s11, s5
-; CHECK-FIX-NOSCHED-NEXT:    vmov s5, r7
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s4, s4
-; CHECK-FIX-NOSCHED-NEXT:    vmov s13, r2
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s6, s6
-; CHECK-FIX-NOSCHED-NEXT:    lsr r0, r0, #16
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s5, s5
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s7, s7
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s13, s13
-; CHECK-FIX-NOSCHED-NEXT:    b .LBB83_5
+; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r5, #16
+; CHECK-FIX-NOSCHED-NEXT:    str r4, [sp, #20] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #4] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    uxth r3, r5
+; CHECK-FIX-NOSCHED-NEXT:    str r6, [sp, #16] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    b .LBB83_3
+; CHECK-FIX-NOSCHED-NEXT:  .LBB83_2:
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r1, #14]
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #16] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r1, #12]
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #20] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r1, #8]
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #12] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r2, [r1, #6]
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r1, #2]
+; CHECK-FIX-NOSCHED-NEXT:    str r2, [sp, #8] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp, #4] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r7, [r1, #10]
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r2, [r1, #4]
+; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r1]
 ; CHECK-FIX-NOSCHED-NEXT:  .LBB83_3:
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r3, [r1, #10]
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r12, [r1, #6]
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r5, [r1, #2]
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r7, [r1, #14]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s8, r3
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r2, [r1, #12]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s12, r12
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r4, [r1, #8]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s1, r5
-; CHECK-FIX-NOSCHED-NEXT:    ldrh lr, [r1, #4]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s0, r7
-; CHECK-FIX-NOSCHED-NEXT:    ldrh r6, [r1]
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s2, s0
-; CHECK-FIX-NOSCHED-NEXT:    vmov s0, r2
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s10, s8
-; CHECK-FIX-NOSCHED-NEXT:    vmov s8, r4
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s14, s12
-; CHECK-FIX-NOSCHED-NEXT:    vmov s12, lr
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s3, s1
-; CHECK-FIX-NOSCHED-NEXT:    vmov s1, r6
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s0, s0
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s8, s8
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s12, s12
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s1, s1
 ; CHECK-FIX-NOSCHED-NEXT:    cmp r0, #0
-; CHECK-FIX-NOSCHED-NEXT:    beq .LBB83_2
-; CHECK-FIX-NOSCHED-NEXT:  .LBB83_4:
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s9, s9
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, r2, d3
-; CHECK-FIX-NOSCHED-NEXT:    vmov r7, s9
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r3, d2[1]
-; CHECK-FIX-NOSCHED-NEXT:    vmov.16 d2[0], r7
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r7, d2[0]
-; CHECK-FIX-NOSCHED-NEXT:    vmov s5, r3
-; CHECK-FIX-NOSCHED-NEXT:    vmov s4, r2
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s9, s5
-; CHECK-FIX-NOSCHED-NEXT:    vmov s6, r0
-; CHECK-FIX-NOSCHED-NEXT:    lsr r0, r0, #16
-; CHECK-FIX-NOSCHED-NEXT:    lsr r3, r3, #16
-; CHECK-FIX-NOSCHED-NEXT:    lsr r2, r2, #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov s7, r0
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s4, s4
-; CHECK-FIX-NOSCHED-NEXT:    vmov s13, r3
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s6, s6
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s7, s7
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s13, s13
-; CHECK-FIX-NOSCHED-NEXT:    vmov s5, r7
-; CHECK-FIX-NOSCHED-NEXT:    lsr r0, r7, #16
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s11, s5
-; CHECK-FIX-NOSCHED-NEXT:    vmov s5, r2
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s5, s5
+; CHECK-FIX-NOSCHED-NEXT:    str r3, [sp] @ 4-byte Spill
+; CHECK-FIX-NOSCHED-NEXT:    beq .LBB83_5
+; CHECK-FIX-NOSCHED-NEXT:  @ %bb.4:
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r6, d2[1]
+; CHECK-FIX-NOSCHED-NEXT:    mov r3, r2
+; CHECK-FIX-NOSCHED-NEXT:    mov r2, r7
+; CHECK-FIX-NOSCHED-NEXT:    vmov r4, r7, d3
+; CHECK-FIX-NOSCHED-NEXT:    vmov.16 d2[0], r12
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 r0, d2[0]
+; CHECK-FIX-NOSCHED-NEXT:    uxth r5, r6
+; CHECK-FIX-NOSCHED-NEXT:    lsr r12, r6, #16
+; CHECK-FIX-NOSCHED-NEXT:    uxth r10, r4
+; CHECK-FIX-NOSCHED-NEXT:    uxth r11, r7
+; CHECK-FIX-NOSCHED-NEXT:    lsr r9, r7, #16
+; CHECK-FIX-NOSCHED-NEXT:    mov r7, r2
+; CHECK-FIX-NOSCHED-NEXT:    mov r2, r3
+; CHECK-FIX-NOSCHED-NEXT:    lsr r4, r4, #16
+; CHECK-FIX-NOSCHED-NEXT:    lsr lr, r0, #16
+; CHECK-FIX-NOSCHED-NEXT:    b .LBB83_6
 ; CHECK-FIX-NOSCHED-NEXT:  .LBB83_5:
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s1, s1
-; CHECK-FIX-NOSCHED-NEXT:    vmov s15, r0
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, s1
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s1, s3
-; CHECK-FIX-NOSCHED-NEXT:    vmov r2, s1
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f32.f16 s15, s15
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s1, s15
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s11, s11
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s1
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s12, s12
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s1, s9
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s8, s8
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s6, s6
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s4, s4
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s0, s0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r0, r2, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r2, s11
+; CHECK-FIX-NOSCHED-NEXT:    vmov r3, r6, d3
+; CHECK-FIX-NOSCHED-NEXT:    vmov r0, r5, d2
+; CHECK-FIX-NOSCHED-NEXT:    lsr r4, r3, #16
+; CHECK-FIX-NOSCHED-NEXT:    lsr r9, r6, #16
+; CHECK-FIX-NOSCHED-NEXT:    lsr r12, r5, #16
+; CHECK-FIX-NOSCHED-NEXT:    lsr lr, r0, #16
+; CHECK-FIX-NOSCHED-NEXT:    uxth r11, r6
+; CHECK-FIX-NOSCHED-NEXT:    uxth r10, r3
+; CHECK-FIX-NOSCHED-NEXT:    uxth r5, r5
+; CHECK-FIX-NOSCHED-NEXT:  .LBB83_6:
+; CHECK-FIX-NOSCHED-NEXT:    uxth r8, r0
+; CHECK-FIX-NOSCHED-NEXT:    ldr r0, [sp, #4] @ 4-byte Reload
+; CHECK-FIX-NOSCHED-NEXT:    ldr r3, [sp] @ 4-byte Reload
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r3, r0, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d16[0], r0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r2, r3, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r2, s12
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s12, s14
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r8, lr, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d18[0], r0
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, s12
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s12, s13
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s12
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r5, r12, lsl #16
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d18[1], r0
+; CHECK-FIX-NOSCHED-NEXT:    ldr r0, [sp, #8] @ 4-byte Reload
 ; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r2, r0, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r2, s1
+; CHECK-FIX-NOSCHED-NEXT:    ldr r2, [sp, #16] @ 4-byte Reload
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d16[1], r0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r2, r2, r3, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d18[1], r2
-; CHECK-FIX-NOSCHED-NEXT:    vmov r2, s8
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s8, s10
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s8
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r2, r3, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r2, s6
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s6, s7
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s6
-; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d17[0], r0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r2, r3, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r2, s4
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s4, s5
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r10, r4, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d19[0], r0
-; CHECK-FIX-NOSCHED-NEXT:    vmov r0, s4
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r2, r0, lsl #16
-; CHECK-FIX-NOSCHED-NEXT:    vmov r2, s0
-; CHECK-FIX-NOSCHED-NEXT:    vcvtb.f16.f32 s0, s2
-; CHECK-FIX-NOSCHED-NEXT:    vmov r3, s0
+; CHECK-FIX-NOSCHED-NEXT:    ldr r0, [sp, #12] @ 4-byte Reload
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r0, r7, lsl #16
+; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d17[0], r0
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r11, r9, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d19[1], r0
-; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r2, r3, lsl #16
+; CHECK-FIX-NOSCHED-NEXT:    ldr r0, [sp, #20] @ 4-byte Reload
+; CHECK-FIX-NOSCHED-NEXT:    pkhbt r0, r0, r2, lsl #16
 ; CHECK-FIX-NOSCHED-NEXT:    vmov.32 d17[1], r0
 ; CHECK-FIX-NOSCHED-NEXT:    aesd.8 q8, q9
 ; CHECK-FIX-NOSCHED-NEXT:    aesimc.8 q8, q8
 ; CHECK-FIX-NOSCHED-NEXT:    vst1.64 {d16, d17}, [r1]
-; CHECK-FIX-NOSCHED-NEXT:    pop {r4, r5, r6, r7, r11, pc}
+; CHECK-FIX-NOSCHED-NEXT:    add sp, sp, #24
+; CHECK-FIX-NOSCHED-NEXT:    pop {r4, r5, r6, r7, r8, r9, r10, r11, pc}
 ;
 ; CHECK-CORTEX-FIX-LABEL: aesd_setf16_cond_via_val:
 ; CHECK-CORTEX-FIX:       @ %bb.0:
-; CHECK-CORTEX-FIX-NEXT:    .save {r4, r5, r6, r7, r11, lr}
-; CHECK-CORTEX-FIX-NEXT:    push {r4, r5, r6, r7, r11, lr}
-; CHECK-CORTEX-FIX-NEXT:    .vsave {d8, d9}
-; CHECK-CORTEX-FIX-NEXT:    vpush {d8, d9}
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s9, s0
+; CHECK-CORTEX-FIX-NEXT:    .save {r4, r5, r6, r7, r8, r9, r10, r11, lr}
+; CHECK-CORTEX-FIX-NEXT:    push {r4, r5, r6, r7, r8, r9, r10, r11, lr}
+; CHECK-CORTEX-FIX-NEXT:    .pad #28
+; CHECK-CORTEX-FIX-NEXT:    sub sp, sp, #28
+; CHECK-CORTEX-FIX-NEXT:    vmov r2, s0
 ; CHECK-CORTEX-FIX-NEXT:    cmp r0, #0
-; CHECK-CORTEX-FIX-NEXT:    beq .LBB83_3
+; CHECK-CORTEX-FIX-NEXT:    beq .LBB83_2
 ; CHECK-CORTEX-FIX-NEXT:  @ %bb.1:
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s0, s9
 ; CHECK-CORTEX-FIX-NEXT:    vld1.64 {d16, d17}, [r1]
-; CHECK-CORTEX-FIX-NEXT:    vmov r2, s0
 ; CHECK-CORTEX-FIX-NEXT:    vmov.32 r3, d16[1]
 ; CHECK-CORTEX-FIX-NEXT:    vmov.16 d16[0], r2
-; CHECK-CORTEX-FIX-NEXT:    vmov r4, r5, d17
-; CHECK-CORTEX-FIX-NEXT:    lsr lr, r3, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s8, r3
-; CHECK-CORTEX-FIX-NEXT:    vmov s11, lr
-; CHECK-CORTEX-FIX-NEXT:    lsr r6, r4, #16
-; CHECK-CORTEX-FIX-NEXT:    lsr r7, r5, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s0, r5
-; CHECK-CORTEX-FIX-NEXT:    vmov s2, r4
-; CHECK-CORTEX-FIX-NEXT:    vmov s12, r7
-; CHECK-CORTEX-FIX-NEXT:    vmov s1, r6
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s3, s8
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 r2, d16[0]
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s10, s0
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s0, s2
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s2, s1
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s14, s12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s1, s11
-; CHECK-CORTEX-FIX-NEXT:    lsr r12, r2, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s13, r2
-; CHECK-CORTEX-FIX-NEXT:    vmov s15, r12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s12, s13
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s8, s15
-; CHECK-CORTEX-FIX-NEXT:    cmp r0, #0
-; CHECK-CORTEX-FIX-NEXT:    bne .LBB83_4
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 r7, d16[0]
+; CHECK-CORTEX-FIX-NEXT:    uxth r6, r7
+; CHECK-CORTEX-FIX-NEXT:    lsr r7, r7, #16
+; CHECK-CORTEX-FIX-NEXT:    str r7, [sp, #20] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    uxth r7, r3
+; CHECK-CORTEX-FIX-NEXT:    lsr r3, r3, #16
+; CHECK-CORTEX-FIX-NEXT:    str r6, [sp, #24] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    str r7, [sp, #8] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #4] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    vmov r3, r7, d17
+; CHECK-CORTEX-FIX-NEXT:    uxth r6, r3
+; CHECK-CORTEX-FIX-NEXT:    lsr r3, r3, #16
+; CHECK-CORTEX-FIX-NEXT:    uxth r11, r7
+; CHECK-CORTEX-FIX-NEXT:    lsr r7, r7, #16
+; CHECK-CORTEX-FIX-NEXT:    str r6, [sp, #16] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    b .LBB83_3
 ; CHECK-CORTEX-FIX-NEXT:  .LBB83_2:
-; CHECK-CORTEX-FIX-NEXT:    vmov r6, r5, d3
-; CHECK-CORTEX-FIX-NEXT:    vmov r0, r2, d2
-; CHECK-CORTEX-FIX-NEXT:    lsr r7, r2, #16
-; CHECK-CORTEX-FIX-NEXT:    lsr r4, r6, #16
-; CHECK-CORTEX-FIX-NEXT:    lsr r3, r5, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s6, r6
-; CHECK-CORTEX-FIX-NEXT:    vmov s4, r5
-; CHECK-CORTEX-FIX-NEXT:    vmov s7, r2
-; CHECK-CORTEX-FIX-NEXT:    vmov s9, r0
-; CHECK-CORTEX-FIX-NEXT:    lsr r12, r0, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s13, r3
-; CHECK-CORTEX-FIX-NEXT:    vmov s15, r4
-; CHECK-CORTEX-FIX-NEXT:    vmov s16, r7
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s5, s4
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s11, s7
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s7, s9
-; CHECK-CORTEX-FIX-NEXT:    vmov s4, r12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s6, s6
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s9, s15
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s13, s13
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s15, s16
-; CHECK-CORTEX-FIX-NEXT:    b .LBB83_5
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r1]
+; CHECK-CORTEX-FIX-NEXT:    ldrh r11, [r1, #12]
+; CHECK-CORTEX-FIX-NEXT:    ldrh r7, [r1, #14]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #24] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r1, #2]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #20] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r1, #4]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #8] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r1, #6]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #4] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r1, #8]
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #16] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r1, #10]
 ; CHECK-CORTEX-FIX-NEXT:  .LBB83_3:
-; CHECK-CORTEX-FIX-NEXT:    ldrh r12, [r1]
-; CHECK-CORTEX-FIX-NEXT:    ldrh lr, [r1, #2]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r7, [r1, #4]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r6, [r1, #6]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r5, [r1, #8]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r4, [r1, #10]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r2, [r1, #12]
-; CHECK-CORTEX-FIX-NEXT:    ldrh r3, [r1, #14]
-; CHECK-CORTEX-FIX-NEXT:    vmov s1, r6
-; CHECK-CORTEX-FIX-NEXT:    vmov s3, r7
-; CHECK-CORTEX-FIX-NEXT:    vmov s0, r3
-; CHECK-CORTEX-FIX-NEXT:    vmov s2, r2
-; CHECK-CORTEX-FIX-NEXT:    vmov s8, r4
-; CHECK-CORTEX-FIX-NEXT:    vmov s12, r5
-; CHECK-CORTEX-FIX-NEXT:    vmov s11, lr
-; CHECK-CORTEX-FIX-NEXT:    vmov s13, r12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s14, s0
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s10, s2
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s2, s8
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s0, s12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s1, s1
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s3, s3
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s8, s11
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s12, s13
+; CHECK-CORTEX-FIX-NEXT:    str r3, [sp, #12] @ 4-byte Spill
 ; CHECK-CORTEX-FIX-NEXT:    cmp r0, #0
-; CHECK-CORTEX-FIX-NEXT:    beq .LBB83_2
-; CHECK-CORTEX-FIX-NEXT:  .LBB83_4:
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s9, s9
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 r2, d2[1]
-; CHECK-CORTEX-FIX-NEXT:    vmov r0, s9
-; CHECK-CORTEX-FIX-NEXT:    lsr r7, r2, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s16, r7
-; CHECK-CORTEX-FIX-NEXT:    vmov.16 d2[0], r0
-; CHECK-CORTEX-FIX-NEXT:    vmov r6, r5, d3
-; CHECK-CORTEX-FIX-NEXT:    vmov s7, r2
-; CHECK-CORTEX-FIX-NEXT:    lsr r4, r6, #16
-; CHECK-CORTEX-FIX-NEXT:    lsr r3, r5, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s6, r6
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s11, s7
-; CHECK-CORTEX-FIX-NEXT:    vmov s9, r3
-; CHECK-CORTEX-FIX-NEXT:    vmov s15, r4
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 r0, d2[0]
-; CHECK-CORTEX-FIX-NEXT:    vmov s4, r5
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s6, s6
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s13, s9
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s9, s15
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s15, s16
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s5, s4
-; CHECK-CORTEX-FIX-NEXT:    vmov s18, r0
-; CHECK-CORTEX-FIX-NEXT:    lsr r12, r0, #16
-; CHECK-CORTEX-FIX-NEXT:    vmov s4, r12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s7, s18
+; CHECK-CORTEX-FIX-NEXT:    beq .LBB83_5
+; CHECK-CORTEX-FIX-NEXT:  @ %bb.4:
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 r3, d2[1]
+; CHECK-CORTEX-FIX-NEXT:    vmov.16 d2[0], r2
+; CHECK-CORTEX-FIX-NEXT:    vmov r4, r6, d3
+; CHECK-CORTEX-FIX-NEXT:    uxth r10, r4
+; CHECK-CORTEX-FIX-NEXT:    lsr r4, r4, #16
+; CHECK-CORTEX-FIX-NEXT:    uxth lr, r6
+; CHECK-CORTEX-FIX-NEXT:    lsr r8, r6, #16
+; CHECK-CORTEX-FIX-NEXT:    uxth r5, r3
+; CHECK-CORTEX-FIX-NEXT:    lsr r12, r3, #16
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 r2, d2[0]
+; CHECK-CORTEX-FIX-NEXT:    uxth r0, r2
+; CHECK-CORTEX-FIX-NEXT:    lsr r9, r2, #16
+; CHECK-CORTEX-FIX-NEXT:    str r0, [sp] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    b .LBB83_6
 ; CHECK-CORTEX-FIX-NEXT:  .LBB83_5:
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s10, s10
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s14, s14
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s3, s3
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s2, s2
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s0, s0
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s8, s8
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s12, s12
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f32.f16 s4, s4
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s6, s6
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s9, s9
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s7, s7
-; CHECK-CORTEX-FIX-NEXT:    vmov r0, s10
-; CHECK-CORTEX-FIX-NEXT:    vmov r2, s14
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s10, s1
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s14, s5
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s1, s13
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s5, s11
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s11, s15
-; CHECK-CORTEX-FIX-NEXT:    vmov r5, s2
-; CHECK-CORTEX-FIX-NEXT:    vmov r4, s8
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r12, r0, r2, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r2, s3
-; CHECK-CORTEX-FIX-NEXT:    vmov r3, s10
-; CHECK-CORTEX-FIX-NEXT:    vmov r7, s1
-; CHECK-CORTEX-FIX-NEXT:    vmov r6, s11
-; CHECK-CORTEX-FIX-NEXT:    vmov r0, s9
-; CHECK-CORTEX-FIX-NEXT:    pkhbt lr, r2, r3, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r3, s14
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r3, r3, r7, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r7, s5
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r7, r7, r6, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r6, s0
-; CHECK-CORTEX-FIX-NEXT:    vcvtb.f16.f32 s0, s4
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r6, r6, r5, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r5, s12
-; CHECK-CORTEX-FIX-NEXT:    vmov r2, s0
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r5, r5, r4, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r4, s6
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d18[0], r5
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d19[0], r6
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d18[1], lr
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d19[1], r12
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r0, r4, r0, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov r4, s7
-; CHECK-CORTEX-FIX-NEXT:    pkhbt r2, r4, r2, lsl #16
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d16[0], r2
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d17[0], r0
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d16[1], r7
-; CHECK-CORTEX-FIX-NEXT:    vmov.32 d17[1], r3
+; CHECK-CORTEX-FIX-NEXT:    vmov r2, r3, d2
+; CHECK-CORTEX-FIX-NEXT:    uxth r0, r2
+; CHECK-CORTEX-FIX-NEXT:    lsr r9, r2, #16
+; CHECK-CORTEX-FIX-NEXT:    uxth r5, r3
+; CHECK-CORTEX-FIX-NEXT:    lsr r12, r3, #16
+; CHECK-CORTEX-FIX-NEXT:    str r0, [sp] @ 4-byte Spill
+; CHECK-CORTEX-FIX-NEXT:    mov r0, r7
+; CHECK-CORTEX-FIX-NEXT:    vmov r6, r7, d3
+; CHECK-CORTEX-FIX-NEXT:    uxth r10, r6
+; CHECK-CORTEX-FIX-NEXT:    lsr r4, r6, #16
+; CHECK-CORTEX-FIX-NEXT:    uxth lr, r7
+; CHECK-CORTEX-FIX-NEXT:    lsr r8, r7, #16
+; CHECK-CORTEX-FIX-NEXT:    mov r7, r0
+; CHECK-CORTEX-FIX-NEXT:  .LBB83_6:
+; CHECK-CORTEX-FIX-NEXT:    ldr r0, [sp, #8] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    ldr r2, [sp, #4] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r11, r11, r7, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    ldr r3, [sp, #12] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    ldr r6, [sp, #20] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r5, r5, r12, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r4, r10, r4, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r7, r0, r2, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    ldr r2, [sp, #16] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r0, lr, r8, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r2, r2, r3, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    ldr r3, [sp, #24] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r3, r3, r6, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    ldr r6, [sp] @ 4-byte Reload
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d18[0], r3
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d19[0], r2
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d18[1], r7
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d19[1], r11
+; CHECK-CORTEX-FIX-NEXT:    pkhbt r6, r6, r9, lsl #16
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d16[0], r6
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d17[0], r4
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d16[1], r5
+; CHECK-CORTEX-FIX-NEXT:    vmov.32 d17[1], r0
 ; CHECK-CORTEX-FIX-NEXT:    aesd.8 q9, q8
 ; CHECK-CORTEX-FIX-NEXT:    aesimc.8 q8, q9
 ; CHECK-CORTEX-FIX-NEXT:    vst1.64 {d16, d17}, [r1]
-; CHECK-CORTEX-FIX-NEXT:    vpop {d8, d9}
-; CHECK-CORTEX-FIX-NEXT:    pop {r4, r5, r6, r7, r11, pc}
+; CHECK-CORTEX-FIX-NEXT:    add sp, sp, #28
+; CHECK-CORTEX-FIX-NEXT:    pop {r4, r5, r6, r7, r8, r9, r10, r11, pc}
   br i1 %0, label %5, label %11
 
 5:
@@ -4627,10 +4118,7 @@ define arm_aapcs_vfpcc void @aesd_setf16_loop_via_val(i32 %0, half %1, <16 x i8>
 ; CHECK-FIX-NEXT:    cmp r0, #0
 ; CHECK-FIX-NEXT:    bxeq lr
 ; CHECK-FIX-NEXT:  .LBB85_1:
-; CHECK-FIX-NEXT:    vcvtb.f32.f16 s0, s0
-; CHECK-FIX-NEXT:    vcvtb.f16.f32 s0, s0
 ; CHECK-FIX-NEXT:    vmov r2, s0
-; CHECK-FIX-NEXT:    uxth r2, r2
 ; CHECK-FIX-NEXT:    vmov.16 d2[0], r2
 ; CHECK-FIX-NEXT:  .LBB85_2: @ =>This Inner Loop Header: Depth=1
 ; CHECK-FIX-NEXT:    vld1.64 {d16, d17}, [r1]
diff --git a/llvm/test/CodeGen/ARM/ctors_dtors.ll b/llvm/test/CodeGen/ARM/ctors_dtors.ll
index b85d5c91abc49..066c250ca1448 100644
--- a/llvm/test/CodeGen/ARM/ctors_dtors.ll
+++ b/llvm/test/CodeGen/ARM/ctors_dtors.ll
@@ -1,5 +1,4 @@
 ; RUN: llc < %s -mtriple=arm-apple-darwin  | FileCheck %s -check-prefix=DARWIN
-; RUN: llc < %s -mtriple=arm-apple-darwin -lower-global-dtors-via-cxa-atexit=false  | FileCheck %s -check-prefix=DARWIN-OLD
 ; RUN: llc < %s -mtriple=arm-linux-gnu -target-abi=apcs  | FileCheck %s -check-prefix=ELF
 ; RUN: llc < %s -mtriple=arm-linux-gnueabi | FileCheck %s -check-prefix=GNUEABI
 
@@ -8,9 +7,6 @@
 ; DARWIN: .section	__DATA,__mod_init_func,mod_init_funcs
 ; DARWIN-NOT: __mod_term_func
 
-; DARWIN-OLD: .section	__DATA,__mod_init_func,mod_init_funcs
-; DARWIN-OLD: .section	__DATA,__mod_term_func,mod_term_funcs
-
 ; ELF: .section .ctors,"aw",%progbits
 ; ELF: .section .dtors,"aw",%progbits
 
diff --git a/llvm/test/CodeGen/ARM/fp16-promote.ll b/llvm/test/CodeGen/ARM/fp16-promote.ll
index 15e7caa51083f..ab8282c5aa120 100644
--- a/llvm/test/CodeGen/ARM/fp16-promote.ll
+++ b/llvm/test/CodeGen/ARM/fp16-promote.ll
@@ -223,18 +223,12 @@ else:
 
 declare i1 @test_dummy(ptr %p) #0
 ; CHECK-ALL-LABEL: test_phi:
-; CHECK-FP16: vcvtb.f32.f16
-; CHECK-FP16: [[LOOP:.LBB[1-9_]+]]:
-; CHECK-FP16: vcvtb.f32.f16
+; CHECK-FP16: [[LOOP:.LBB[0-9_]+]]:
 ; CHECK-FP16: bl      test_dummy
 ; CHECK-FP16: bne     [[LOOP]]
-; CHECK-FP16: vcvtb.f16.f32
-; CHECK-LIBCALL-VFP: bl __aeabi_h2f
-; CHECK-LIBCALL: [[LOOP:.LBB[1-9_]+]]:
-; CHECK-LIBCALL-VFP: bl __aeabi_h2f
+; CHECK-LIBCALL: [[LOOP:.LBB[0-9_]+]]:
 ; CHECK-LIBCALL: bl test_dummy
 ; CHECK-LIBCALL: bne     [[LOOP]]
-; CHECK-LIBCALL-VFP: bl __aeabi_f2h
 define void @test_phi(ptr %p) #0 {
 entry:
   %a = load half, ptr %p
diff --git a/llvm/test/CodeGen/ARM/fp16-return-pr60510.ll b/llvm/test/CodeGen/ARM/fp16-return-pr60510.ll
new file mode 100644
index 0000000000000..a1c89dcdda177
--- /dev/null
+++ b/llvm/test/CodeGen/ARM/fp16-return-pr60510.ll
@@ -0,0 +1,212 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --filter-out "(\.v?save|push|pop)"
+
+; No FP16/BF16
+; RUN: llc -mtriple=arm-none-eabi -float-abi=soft -mattr=+armv8.2-a,+fp-armv8,-fullfp16,-bf16,-neon %s -o - | FileCheck %s --check-prefixes=NO-FP16-SOFT
+; RUN: llc -mtriple=thumb-none-eabi -float-abi=soft -mattr=+armv8.2-a,+fp-armv8,-fullfp16,-bf16,-neon %s -o - | FileCheck %s --check-prefixes=NO-FP16-SOFT
+; RUN: llc -mtriple=arm-none-eabi -mattr=+armv8.2-a,+fp-armv8,-fullfp16,-bf16,-neon %s -o - | FileCheck %s --check-prefixes=NO-FP16-SOFT
+; RUN: llc -mtriple=thumb-none-eabi -mattr=+armv8.2-a,+fp-armv8,-fullfp16,-bf16,-neon %s -o - | FileCheck %s --check-prefixes=NO-FP16-SOFT
+; RUN: llc -mtriple=arm-none-eabihf -mattr=+armv8.2-a,+fp-armv8,-fullfp16,-bf16,-neon %s -o - | FileCheck %s --check-prefixes=NO-FP16-HARD
+; RUN: llc -mtriple=thumb-none-eabihf -mattr=+armv8.2-a,+fp-armv8,-fullfp16,-bf16,-neon %s -o - | FileCheck %s --check-prefixes=NO-FP16-HARD
+
+; FP16/BF16
+; RUN: llc -mtriple=arm-none-eabi -float-abi=soft -mattr=+armv8.2-a,+fp-armv8,+fullfp16,+bf16,-neon %s -o - | FileCheck %s --check-prefixes=FP16-SOFT
+; RUN: llc -mtriple=thumb-none-eabi -float-abi=soft -mattr=+armv8.2-a,+fp-armv8,+fullfp16,+bf16,-neon %s -o - | FileCheck %s --check-prefixes=FP16-SOFT
+; RUN: llc -mtriple=arm-none-eabi -mattr=+armv8.2-a,+fp-armv8,+fullfp16,+bf16,-neon %s -o - | FileCheck %s --check-prefixes=FP16-SOFT
+; RUN: llc -mtriple=thumb-none-eabi -mattr=+armv8.2-a,+fp-armv8,+fullfp16,+bf16,-neon %s -o - | FileCheck %s --check-prefixes=FP16-SOFT
+; RUN: llc -mtriple=arm-none-eabihf -mattr=+armv8.2-a,+fp-armv8,+fullfp16,+bf16,-neon %s -o - | FileCheck %s --check-prefixes=FP16-HARD
+; RUN: llc -mtriple=thumb-none-eabihf -mattr=+armv8.2-a,+fp-armv8,+fullfp16,+bf16,-neon %s -o - | FileCheck %s --check-prefixes=FP16-HARD
+
+; PR60510 showed a bug where the return from `*_inner` was getting "lost" by an
+; optimisation, and a garbage value was being left in `s0`.
+
+declare dso_local float @other(float) nounwind
+declare dso_local void @fp16_sink(half) nounwind
+declare dso_local void @bf16_sink(bfloat) nounwind
+
+declare dso_local bfloat @fp16_inner() nounwind
+declare dso_local bfloat @bf_inner() nounwind
+
+define half @fp16_out_call_oneuse(float %arg) nounwind {
+; NO-FP16-SOFT-LABEL: fp16_out_call_oneuse:
+; NO-FP16-SOFT:  @ %bb.0:
+; NO-FP16-SOFT:    mov r4, r0
+; NO-FP16-SOFT:    bl fp16_inner
+; NO-FP16-SOFT:    mov r5, r0
+; NO-FP16-SOFT:    mov r0, r4
+; NO-FP16-SOFT:    bl other
+; NO-FP16-SOFT:    mov r0, r5
+;
+; NO-FP16-HARD-LABEL: fp16_out_call_oneuse:
+; NO-FP16-HARD:  @ %bb.0:
+; NO-FP16-HARD:    vmov.f32 s16, s0
+; NO-FP16-HARD:    bl fp16_inner
+; NO-FP16-HARD:    vmov.f32 s18, s0
+; NO-FP16-HARD:    vmov.f32 s0, s16
+; NO-FP16-HARD:    bl other
+; NO-FP16-HARD:    vmov.f32 s0, s18
+;
+; FP16-SOFT-LABEL: fp16_out_call_oneuse:
+; FP16-SOFT:  @ %bb.0:
+; FP16-SOFT:    mov r4, r0
+; FP16-SOFT:    bl fp16_inner
+; FP16-SOFT:    mov r5, r0
+; FP16-SOFT:    mov r0, r4
+; FP16-SOFT:    bl other
+; FP16-SOFT:    vmov.f16 s0, r5
+; FP16-SOFT:    vmov r0, s0
+;
+; FP16-HARD-LABEL: fp16_out_call_oneuse:
+; FP16-HARD:  @ %bb.0:
+; FP16-HARD:    vmov.f32 s16, s0
+; FP16-HARD:    bl fp16_inner
+; FP16-HARD:    vmov.f32 s18, s0
+; FP16-HARD:    vmov.f32 s0, s16
+; FP16-HARD:    bl other
+; FP16-HARD:    vmov.f32 s0, s18
+  %call = call half @fp16_inner()
+  %call1 = call float @other(float %arg)
+  ret half %call
+}
+
+define half @fp16_out_call_multiuse(float %arg) nounwind {
+; NO-FP16-SOFT-LABEL: fp16_out_call_multiuse:
+; NO-FP16-SOFT:  @ %bb.0:
+; NO-FP16-SOFT:    mov r4, r0
+; NO-FP16-SOFT:    bl fp16_inner
+; NO-FP16-SOFT:    mov r5, r0
+; NO-FP16-SOFT:    mov r0, r4
+; NO-FP16-SOFT:    bl other
+; NO-FP16-SOFT:    mov r0, r5
+; NO-FP16-SOFT:    bl fp16_sink
+; NO-FP16-SOFT:    mov r0, r5
+;
+; NO-FP16-HARD-LABEL: fp16_out_call_multiuse:
+; NO-FP16-HARD:  @ %bb.0:
+; NO-FP16-HARD:    vmov.f32 s16, s0
+; NO-FP16-HARD:    bl fp16_inner
+; NO-FP16-HARD:    vmov.f32 s18, s0
+; NO-FP16-HARD:    vmov.f32 s0, s16
+; NO-FP16-HARD:    bl other
+; NO-FP16-HARD:    vmov.f32 s0, s18
+; NO-FP16-HARD:    bl fp16_sink
+; NO-FP16-HARD:    vmov.f32 s0, s18
+;
+; FP16-SOFT-LABEL: fp16_out_call_multiuse:
+; FP16-SOFT:  @ %bb.0:
+; FP16-SOFT:    mov r4, r0
+; FP16-SOFT:    bl fp16_inner
+; FP16-SOFT:    mov r5, r0
+; FP16-SOFT:    mov r0, r4
+; FP16-SOFT:    bl other
+; FP16-SOFT:    vmov.f16 s16, r5
+; FP16-SOFT:    vmov.f16 r0, s16
+; FP16-SOFT:    bl fp16_sink
+; FP16-SOFT:    vmov r0, s16
+;
+; FP16-HARD-LABEL: fp16_out_call_multiuse:
+; FP16-HARD:  @ %bb.0:
+; FP16-HARD:    vmov.f32 s16, s0
+; FP16-HARD:    bl fp16_inner
+; FP16-HARD:    vmov.f32 s18, s0
+; FP16-HARD:    vmov.f32 s0, s16
+; FP16-HARD:    bl other
+; FP16-HARD:    vmov.f16 r0, s18
+; FP16-HARD:    vmov s0, r0
+; FP16-HARD:    bl fp16_sink
+; FP16-HARD:    vmov.f32 s0, s18
+  %call = call half @fp16_inner()
+  %call1 = call float @other(float %arg)
+  call void @fp16_sink(half %call)
+  ret half %call
+}
+
+define bfloat @bf_out_call_oneuse(float %arg) nounwind {
+; NO-FP16-SOFT-LABEL: bf_out_call_oneuse:
+; NO-FP16-SOFT:  @ %bb.0:
+; NO-FP16-SOFT:    mov r4, r0
+; NO-FP16-SOFT:    bl bf_inner
+; NO-FP16-SOFT:    mov r5, r0
+; NO-FP16-SOFT:    mov r0, r4
+; NO-FP16-SOFT:    bl other
+; NO-FP16-SOFT:    mov r0, r5
+;
+; NO-FP16-HARD-LABEL: bf_out_call_oneuse:
+; NO-FP16-HARD:  @ %bb.0:
+; NO-FP16-HARD:    vmov.f32 s16, s0
+; NO-FP16-HARD:    bl bf_inner
+; NO-FP16-HARD:    vmov.f32 s18, s0
+; NO-FP16-HARD:    vmov.f32 s0, s16
+; NO-FP16-HARD:    bl other
+; NO-FP16-HARD:    vmov.f32 s0, s18
+;
+; FP16-SOFT-LABEL: bf_out_call_oneuse:
+; FP16-SOFT:  @ %bb.0:
+; FP16-SOFT:    mov r4, r0
+; FP16-SOFT:    bl bf_inner
+; FP16-SOFT:    mov r5, r0
+; FP16-SOFT:    mov r0, r4
+; FP16-SOFT:    bl other
+; FP16-SOFT:    mov r0, r5
+;
+; FP16-HARD-LABEL: bf_out_call_oneuse:
+; FP16-HARD:  @ %bb.0:
+; FP16-HARD:    vmov.f32 s16, s0
+; FP16-HARD:    bl bf_inner
+; FP16-HARD:    vmov.f32 s18, s0
+; FP16-HARD:    vmov.f32 s0, s16
+; FP16-HARD:    bl other
+; FP16-HARD:    vmov.f32 s0, s18
+  %call = call bfloat @bf_inner()
+  %call1 = call float @other(float %arg)
+  ret bfloat %call
+}
+
+define bfloat @bf_out_call_multiuse(float %arg) nounwind {
+; NO-FP16-SOFT-LABEL: bf_out_call_multiuse:
+; NO-FP16-SOFT:  @ %bb.0:
+; NO-FP16-SOFT:    mov r4, r0
+; NO-FP16-SOFT:    bl bf_inner
+; NO-FP16-SOFT:    mov r5, r0
+; NO-FP16-SOFT:    mov r0, r4
+; NO-FP16-SOFT:    bl other
+; NO-FP16-SOFT:    mov r0, r5
+; NO-FP16-SOFT:    bl bf16_sink
+; NO-FP16-SOFT:    mov r0, r5
+;
+; NO-FP16-HARD-LABEL: bf_out_call_multiuse:
+; NO-FP16-HARD:  @ %bb.0:
+; NO-FP16-HARD:    vmov.f32 s16, s0
+; NO-FP16-HARD:    bl bf_inner
+; NO-FP16-HARD:    vmov.f32 s18, s0
+; NO-FP16-HARD:    vmov.f32 s0, s16
+; NO-FP16-HARD:    bl other
+; NO-FP16-HARD:    vmov.f32 s0, s18
+; NO-FP16-HARD:    bl bf16_sink
+; NO-FP16-HARD:    vmov.f32 s0, s18
+;
+; FP16-SOFT-LABEL: bf_out_call_multiuse:
+; FP16-SOFT:  @ %bb.0:
+; FP16-SOFT:    mov r4, r0
+; FP16-SOFT:    bl bf_inner
+; FP16-SOFT:    mov r5, r0
+; FP16-SOFT:    mov r0, r4
+; FP16-SOFT:    bl other
+; FP16-SOFT:    mov r0, r5
+; FP16-SOFT:    bl bf16_sink
+; FP16-SOFT:    mov r0, r5
+;
+; FP16-HARD-LABEL: bf_out_call_multiuse:
+; FP16-HARD:  @ %bb.0:
+; FP16-HARD:    vmov.f32 s16, s0
+; FP16-HARD:    bl bf_inner
+; FP16-HARD:    vmov.f32 s18, s0
+; FP16-HARD:    vmov.f32 s0, s16
+; FP16-HARD:    bl other
+; FP16-HARD:    vmov.f32 s0, s18
+; FP16-HARD:    bl bf16_sink
+; FP16-HARD:    vmov.f32 s0, s18
+  %call = call bfloat @bf_inner()
+  %call1 = call float @other(float %arg)
+  call void @bf16_sink(bfloat %call)
+  ret bfloat %call
+}
diff --git a/llvm/test/CodeGen/ARM/inlineasm-fp-half.ll b/llvm/test/CodeGen/ARM/inlineasm-fp-half.ll
index 6da247795a3f6..9840e3f63c5ac 100644
--- a/llvm/test/CodeGen/ARM/inlineasm-fp-half.ll
+++ b/llvm/test/CodeGen/ARM/inlineasm-fp-half.ll
@@ -1,60 +1,173 @@
 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=arm -mattr=+armv8.2-a,+fp-armv8,+fullfp16,+bf16,-neon %s -o - | FileCheck %s
-; RUN: llc -mtriple=thumb -mattr=+armv8.2-a,+fp-armv8,+fullfp16,+bf16,-neon %s -o - | FileCheck %s
+; No FP16/BF16
+; RUN: llc -mtriple=arm-none-eabi -mattr=+armv8.2-a,+fp-armv8,-fullfp16,-bf16,-neon %s -o - | FileCheck %s --check-prefixes=NO-FP16-SOFTFP
+; RUN: llc -mtriple=thumb-none-eabi -mattr=+armv8.2-a,+fp-armv8,-fullfp16,-bf16,-neon %s -o - | FileCheck %s --check-prefixes=NO-FP16-SOFTFP
+; RUN: llc -mtriple=arm-none-eabihf -mattr=+armv8.2-a,+fp-armv8,-fullfp16,-bf16,-neon %s -o - | FileCheck %s --check-prefixes=NO-FP16-HARD
+; RUN: llc -mtriple=thumb-none-eabihf -mattr=+armv8.2-a,+fp-armv8,-fullfp16,-bf16,-neon %s -o - | FileCheck %s --check-prefixes=NO-FP16-HARD
 
+; With FP16, Without BF16
+; RUN: llc -mtriple=arm-none-eabi -mattr=+armv8.2-a,+fp-armv8,+fullfp16,-bf16,-neon %s -o - | FileCheck %s --check-prefixes=FP16-SOFTFP
+; RUN: llc -mtriple=thumb-none-eabi -mattr=+armv8.2-a,+fp-armv8,+fullfp16,-bf16,-neon %s -o - | FileCheck %s --check-prefixes=FP16-SOFTFP
+; RUN: llc -mtriple=arm-none-eabihf -mattr=+armv8.2-a,+fp-armv8,+fullfp16,-bf16,-neon %s -o - | FileCheck %s --check-prefixes=FP16-HARD
+; RUN: llc -mtriple=thumb-none-eabihf -mattr=+armv8.2-a,+fp-armv8,+fullfp16,-bf16,-neon %s -o - | FileCheck %s --check-prefixes=FP16-HARD
 
-define arm_aapcscc half @f_t(half %x) nounwind {
-; CHECK-LABEL: f_t:
-; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r0
-; CHECK-NEXT:    @APP
-; CHECK-NEXT:    vsqrt.f16 s0, s0
-; CHECK-NEXT:    @NO_APP
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    bx lr
+; With FP16/BF16
+; RUN: llc -mtriple=arm-none-eabi -mattr=+armv8.2-a,+fp-armv8,+fullfp16,+bf16,-neon %s -o - | FileCheck %s --check-prefixes=FP16-SOFTFP,BF16-SOFTFP
+; RUN: llc -mtriple=thumb-none-eabi -mattr=+armv8.2-a,+fp-armv8,+fullfp16,+bf16,-neon %s -o - | FileCheck %s --check-prefixes=FP16-SOFTFP,BF16-SOFTFP
+; RUN: llc -mtriple=arm-none-eabihf -mattr=+armv8.2-a,+fp-armv8,+fullfp16,+bf16,-neon %s -o - | FileCheck %s --check-prefixes=FP16-HARD,BF16-HARD
+; RUN: llc -mtriple=thumb-none-eabihf -mattr=+armv8.2-a,+fp-armv8,+fullfp16,+bf16,-neon %s -o - | FileCheck %s --check-prefixes=FP16-HARD,BF16-HARD
+
+; This test ensures that we can use `w` and `t` constraints to allocate
+; S-registers for 16-bit FP inputs and outputs for inline assembly, with either
+; the softfp or hard float ABIs. (With the soft abi, no S-regs are available).
+
+define half @half_t(half %x) nounwind {
+; NO-FP16-SOFTFP-LABEL: half_t:
+; NO-FP16-SOFTFP:       @ %bb.0: @ %entry
+; NO-FP16-SOFTFP-NEXT:    vmov s0, r0
+; NO-FP16-SOFTFP-NEXT:    @APP
+; NO-FP16-SOFTFP-NEXT:    vmov.f32 s0, s0
+; NO-FP16-SOFTFP-NEXT:    @NO_APP
+; NO-FP16-SOFTFP-NEXT:    vmov r0, s0
+; NO-FP16-SOFTFP-NEXT:    bx lr
+;
+; NO-FP16-HARD-LABEL: half_t:
+; NO-FP16-HARD:       @ %bb.0: @ %entry
+; NO-FP16-HARD-NEXT:    @APP
+; NO-FP16-HARD-NEXT:    vmov.f32 s0, s0
+; NO-FP16-HARD-NEXT:    @NO_APP
+; NO-FP16-HARD-NEXT:    bx lr
+;
+; FP16-SOFTFP-LABEL: half_t:
+; FP16-SOFTFP:       @ %bb.0: @ %entry
+; FP16-SOFTFP-NEXT:    vmov.f16 s0, r0
+; FP16-SOFTFP-NEXT:    @APP
+; FP16-SOFTFP-NEXT:    vmov.f32 s0, s0
+; FP16-SOFTFP-NEXT:    @NO_APP
+; FP16-SOFTFP-NEXT:    vmov r0, s0
+; FP16-SOFTFP-NEXT:    bx lr
+;
+; FP16-HARD-LABEL: half_t:
+; FP16-HARD:       @ %bb.0: @ %entry
+; FP16-HARD-NEXT:    @APP
+; FP16-HARD-NEXT:    vmov.f32 s0, s0
+; FP16-HARD-NEXT:    @NO_APP
+; FP16-HARD-NEXT:    bx lr
 entry:
-  %0 = tail call half asm "vsqrt.f16 $0, $1", "=t,t"(half %x)
+  %0 = tail call half asm "vmov $0, $1", "=t,t"(half %x)
   ret half %0
 }
 
-define arm_aapcscc half @f_w(half %x) nounwind {
-; CHECK-LABEL: f_w:
-; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r0
-; CHECK-NEXT:    @APP
-; CHECK-NEXT:    vsqrt.f16 s0, s0
-; CHECK-NEXT:    @NO_APP
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    bx lr
+define half @half_w(half %x) nounwind {
+; NO-FP16-SOFTFP-LABEL: half_w:
+; NO-FP16-SOFTFP:       @ %bb.0: @ %entry
+; NO-FP16-SOFTFP-NEXT:    vmov s0, r0
+; NO-FP16-SOFTFP-NEXT:    @APP
+; NO-FP16-SOFTFP-NEXT:    vmov.f32 s0, s0
+; NO-FP16-SOFTFP-NEXT:    @NO_APP
+; NO-FP16-SOFTFP-NEXT:    vmov r0, s0
+; NO-FP16-SOFTFP-NEXT:    bx lr
+;
+; NO-FP16-HARD-LABEL: half_w:
+; NO-FP16-HARD:       @ %bb.0: @ %entry
+; NO-FP16-HARD-NEXT:    @APP
+; NO-FP16-HARD-NEXT:    vmov.f32 s0, s0
+; NO-FP16-HARD-NEXT:    @NO_APP
+; NO-FP16-HARD-NEXT:    bx lr
+;
+; FP16-SOFTFP-LABEL: half_w:
+; FP16-SOFTFP:       @ %bb.0: @ %entry
+; FP16-SOFTFP-NEXT:    vmov.f16 s0, r0
+; FP16-SOFTFP-NEXT:    @APP
+; FP16-SOFTFP-NEXT:    vmov.f32 s0, s0
+; FP16-SOFTFP-NEXT:    @NO_APP
+; FP16-SOFTFP-NEXT:    vmov r0, s0
+; FP16-SOFTFP-NEXT:    bx lr
+;
+; FP16-HARD-LABEL: half_w:
+; FP16-HARD:       @ %bb.0: @ %entry
+; FP16-HARD-NEXT:    @APP
+; FP16-HARD-NEXT:    vmov.f32 s0, s0
+; FP16-HARD-NEXT:    @NO_APP
+; FP16-HARD-NEXT:    bx lr
 entry:
-  %0 = tail call half asm "vsqrt.f16 $0, $1", "=w,w"(half %x)
+  %0 = tail call half asm "vmov $0, $1", "=w,w"(half %x)
   ret half %0
 }
 
-define arm_aapcscc bfloat @h_t(bfloat %x) nounwind {
-; CHECK-LABEL: h_t:
-; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r0
-; CHECK-NEXT:    @APP
-; CHECK-NEXT:    vmov.f32 s0, s0
-; CHECK-NEXT:    @NO_APP
-; CHECK-NEXT:    vmov.f16 r0, s0
-; CHECK-NEXT:    bx lr
+define bfloat @bf16_t(bfloat %x) nounwind {
+; NO-FP16-SOFTFP-LABEL: bf16_t:
+; NO-FP16-SOFTFP:       @ %bb.0: @ %entry
+; NO-FP16-SOFTFP-NEXT:    vmov s0, r0
+; NO-FP16-SOFTFP-NEXT:    @APP
+; NO-FP16-SOFTFP-NEXT:    vmov.f32 s0, s0
+; NO-FP16-SOFTFP-NEXT:    @NO_APP
+; NO-FP16-SOFTFP-NEXT:    vmov r0, s0
+; NO-FP16-SOFTFP-NEXT:    bx lr
+;
+; NO-FP16-HARD-LABEL: bf16_t:
+; NO-FP16-HARD:       @ %bb.0: @ %entry
+; NO-FP16-HARD-NEXT:    @APP
+; NO-FP16-HARD-NEXT:    vmov.f32 s0, s0
+; NO-FP16-HARD-NEXT:    @NO_APP
+; NO-FP16-HARD-NEXT:    bx lr
+;
+; FP16-SOFTFP-LABEL: bf16_t:
+; FP16-SOFTFP:       @ %bb.0: @ %entry
+; FP16-SOFTFP-NEXT:    vmov s0, r0
+; FP16-SOFTFP-NEXT:    @APP
+; FP16-SOFTFP-NEXT:    vmov.f32 s0, s0
+; FP16-SOFTFP-NEXT:    @NO_APP
+; FP16-SOFTFP-NEXT:    vmov r0, s0
+; FP16-SOFTFP-NEXT:    bx lr
+;
+; FP16-HARD-LABEL: bf16_t:
+; FP16-HARD:       @ %bb.0: @ %entry
+; FP16-HARD-NEXT:    @APP
+; FP16-HARD-NEXT:    vmov.f32 s0, s0
+; FP16-HARD-NEXT:    @NO_APP
+; FP16-HARD-NEXT:    bx lr
 entry:
-  %0 = tail call bfloat asm "vmov.f32 $0, $1", "=t,t"(bfloat %x)
+  %0 = tail call bfloat asm "vmov $0, $1", "=t,t"(bfloat %x)
   ret bfloat %0
 }
 
-define arm_aapcscc bfloat @h_w(bfloat %x) nounwind {
-; CHECK-LABEL: h_w:
-; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r0
-; CHECK-NEXT:    @APP
-; CHECK-NEXT:    vmov.f32 s0, s0
-; CHECK-NEXT:    @NO_APP
-; CHECK-NEXT:    vmov.f16 r0, s0
-; CHECK-NEXT:    bx lr
+define bfloat @bf16_w(bfloat %x) nounwind {
+; NO-FP16-SOFTFP-LABEL: bf16_w:
+; NO-FP16-SOFTFP:       @ %bb.0: @ %entry
+; NO-FP16-SOFTFP-NEXT:    vmov s0, r0
+; NO-FP16-SOFTFP-NEXT:    @APP
+; NO-FP16-SOFTFP-NEXT:    vmov.f32 s0, s0
+; NO-FP16-SOFTFP-NEXT:    @NO_APP
+; NO-FP16-SOFTFP-NEXT:    vmov r0, s0
+; NO-FP16-SOFTFP-NEXT:    bx lr
+;
+; NO-FP16-HARD-LABEL: bf16_w:
+; NO-FP16-HARD:       @ %bb.0: @ %entry
+; NO-FP16-HARD-NEXT:    @APP
+; NO-FP16-HARD-NEXT:    vmov.f32 s0, s0
+; NO-FP16-HARD-NEXT:    @NO_APP
+; NO-FP16-HARD-NEXT:    bx lr
+;
+; FP16-SOFTFP-LABEL: bf16_w:
+; FP16-SOFTFP:       @ %bb.0: @ %entry
+; FP16-SOFTFP-NEXT:    vmov s0, r0
+; FP16-SOFTFP-NEXT:    @APP
+; FP16-SOFTFP-NEXT:    vmov.f32 s0, s0
+; FP16-SOFTFP-NEXT:    @NO_APP
+; FP16-SOFTFP-NEXT:    vmov r0, s0
+; FP16-SOFTFP-NEXT:    bx lr
+;
+; FP16-HARD-LABEL: bf16_w:
+; FP16-HARD:       @ %bb.0: @ %entry
+; FP16-HARD-NEXT:    @APP
+; FP16-HARD-NEXT:    vmov.f32 s0, s0
+; FP16-HARD-NEXT:    @NO_APP
+; FP16-HARD-NEXT:    bx lr
 entry:
-  %0 = tail call bfloat asm "vmov.f32 $0, $1", "=w,w"(bfloat %x)
+  %0 = tail call bfloat asm "vmov $0, $1", "=w,w"(bfloat %x)
   ret bfloat %0
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; BF16-HARD: {{.*}}
+; BF16-SOFTFP: {{.*}}
diff --git a/llvm/test/CodeGen/ARM/pr47454.ll b/llvm/test/CodeGen/ARM/pr47454.ll
index 6624c94d2e9c5..95f0ac75bd4d2 100644
--- a/llvm/test/CodeGen/ARM/pr47454.ll
+++ b/llvm/test/CodeGen/ARM/pr47454.ll
@@ -10,29 +10,15 @@ define internal fastcc void @main() {
 ; CHECK:       @ %bb.0: @ %Entry
 ; CHECK-NEXT:    push {r11, lr}
 ; CHECK-NEXT:    mov r11, sp
-; CHECK-NEXT:    sub sp, sp, #16
+; CHECK-NEXT:    sub sp, sp, #8
 ; CHECK-NEXT:    mov r0, #31744
 ; CHECK-NEXT:    strh r0, [r11, #-2]
 ; CHECK-NEXT:    ldrh r0, [r11, #-2]
-; CHECK-NEXT:    bl __gnu_h2f_ieee
 ; CHECK-NEXT:    vmov s0, r0
-; CHECK-NEXT:    vstr s0, [sp, #4] @ 4-byte Spill
+; CHECK-NEXT:    vstr s0, [sp] @ 4-byte Spill
 ; CHECK-NEXT:    bl getConstant
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    bl __gnu_h2f_ieee
-; CHECK-NEXT:    vmov s0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    bl __gnu_f2h_ieee
-; CHECK-NEXT:    vldr s0, [sp, #4] @ 4-byte Reload
-; CHECK-NEXT:    str r0, [sp, #8] @ 4-byte Spill
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    bl __gnu_f2h_ieee
-; CHECK-NEXT:    mov r1, r0
-; CHECK-NEXT:    ldr r0, [sp, #8] @ 4-byte Reload
-; CHECK-NEXT:    uxth r1, r1
-; CHECK-NEXT:    vmov s0, r1
-; CHECK-NEXT:    uxth r0, r0
-; CHECK-NEXT:    vmov s1, r0
+; CHECK-NEXT:    vmov.f32 s1, s0
+; CHECK-NEXT:    vldr s0, [sp] @ 4-byte Reload
 ; CHECK-NEXT:    bl isEqual
 ; CHECK-NEXT:    mov sp, r11
 ; CHECK-NEXT:    pop {r11, pc}
diff --git a/llvm/test/CodeGen/BPF/BTF/pruning-dup-ptr-struct.ll b/llvm/test/CodeGen/BPF/BTF/pruning-dup-ptr-struct.ll
new file mode 100644
index 0000000000000..1d9993727a6f6
--- /dev/null
+++ b/llvm/test/CodeGen/BPF/BTF/pruning-dup-ptr-struct.ll
@@ -0,0 +1,113 @@
+; RUN: llc -march=bpfel -o - %s | FileCheck %s
+; RUN: llc -march=bpfeb -o - %s | FileCheck %s
+; Source:
+;   struct t1 {
+;     int a;
+;   };
+;   struct t2 {
+;     struct t1 *p1;
+;     struct t1 *p2;
+;     int b;
+;   };
+;   int foo(struct t2 *arg) {
+;     return arg->b;
+;   }
+; Compilation flags:
+;   clang -target bpf -O2 -g -S -emit-llvm t.c
+
+%struct.t2 = type { ptr, ptr, i32 }
+
+; Function Attrs: argmemonly mustprogress nofree norecurse nosync nounwind readonly willreturn
+define dso_local i32 @foo(ptr nocapture noundef readonly %arg) local_unnamed_addr #0 !dbg !7 {
+entry:
+  call void @llvm.dbg.value(metadata ptr %arg, metadata !22, metadata !DIExpression()), !dbg !23
+  %b = getelementptr inbounds %struct.t2, ptr %arg, i64 0, i32 2, !dbg !24
+  %0 = load i32, ptr %b, align 8, !dbg !24, !tbaa !25
+  ret i32 %0, !dbg !31
+}
+
+; CHECK:             .long   0                               # BTF_KIND_PTR(id = 1)
+; CHECK-NEXT:        .long   33554432                        # 0x2000000
+; CHECK-NEXT:        .long   2
+; CHECK-NEXT:        .long   1                               # BTF_KIND_STRUCT(id = 2)
+; CHECK-NEXT:        .long   67108867                        # 0x4000003
+; CHECK-NEXT:        .long   24
+; CHECK-NEXT:        .long   4
+; CHECK-NEXT:        .long   3
+; CHECK-NEXT:        .long   0                               # 0x0
+; CHECK-NEXT:        .long   7
+; CHECK-NEXT:        .long   3
+; CHECK-NEXT:        .long   64                              # 0x40
+; CHECK-NEXT:        .long   10
+; CHECK-NEXT:        .long   4
+; CHECK-NEXT:        .long   128                             # 0x80
+; CHECK-NEXT:        .long   0                               # BTF_KIND_PTR(id = 3)
+; CHECK-NEXT:        .long   33554432                        # 0x2000000
+; CHECK-NEXT:        .long   7
+; CHECK-NEXT:        .long   12                              # BTF_KIND_INT(id = 4)
+; CHECK-NEXT:        .long   16777216                        # 0x1000000
+; CHECK-NEXT:        .long   4
+; CHECK-NEXT:        .long   16777248                        # 0x1000020
+; CHECK-NEXT:        .long   0                               # BTF_KIND_FUNC_PROTO(id = 5)
+; CHECK-NEXT:        .long   218103809                       # 0xd000001
+; CHECK-NEXT:        .long   4
+; CHECK-NEXT:        .long   16
+; CHECK-NEXT:        .long   1
+; CHECK-NEXT:        .long   20                              # BTF_KIND_FUNC(id = 6)
+; CHECK-NEXT:        .long   201326593                       # 0xc000001
+; CHECK-NEXT:        .long   5
+; CHECK-NEXT:        .long   60                              # BTF_KIND_FWD(id = 7)
+; CHECK-NEXT:        .long   117440512                       # 0x7000000
+; CHECK-NEXT:        .long   0
+
+; CHECK:             .ascii  "t2"                            # string offset=1
+; CHECK:             .ascii  "p1"                            # string offset=4
+; CHECK:             .ascii  "p2"                            # string offset=7
+; CHECK:             .byte   98                              # string offset=10
+; CHECK:             .ascii  "int"                           # string offset=12
+; CHECK:             .ascii  "arg"                           # string offset=16
+; CHECK:             .ascii  "foo"                           # string offset=20
+; CHECK:             .ascii  "t1"                            # string offset=60
+
+; Function Attrs: nocallback nofree nosync nounwind readnone speculatable willreturn
+declare void @llvm.dbg.value(metadata, metadata, metadata) #1
+
+attributes #0 = { argmemonly mustprogress nofree norecurse nosync nounwind readonly willreturn "frame-pointer"="all" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" }
+attributes #1 = { nocallback nofree nosync nounwind readnone speculatable willreturn }
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!2, !3, !4, !5}
+!llvm.ident = !{!6}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 15.0.7 (https://github.com/llvm/llvm-project.git 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!1 = !DIFile(filename: "t1.c", directory: "/tmp/home/yhs/tmp3/tests", checksumkind: CSK_MD5, checksum: "9a79ff24a6244249e556afd85288af94")
+!2 = !{i32 7, !"Dwarf Version", i32 5}
+!3 = !{i32 2, !"Debug Info Version", i32 3}
+!4 = !{i32 1, !"wchar_size", i32 4}
+!5 = !{i32 7, !"frame-pointer", i32 2}
+!6 = !{!"clang version 15.0.7 (https://github.com/llvm/llvm-project.git 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a)"}
+!7 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 9, type: !8, scopeLine: 9, flags: DIFlagPrototyped | DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !21)
+!8 = !DISubroutineType(types: !9)
+!9 = !{!10, !11}
+!10 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!11 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !12, size: 64)
+!12 = distinct !DICompositeType(tag: DW_TAG_structure_type, name: "t2", file: !1, line: 4, size: 192, elements: !13)
+!13 = !{!14, !19, !20}
+!14 = !DIDerivedType(tag: DW_TAG_member, name: "p1", scope: !12, file: !1, line: 5, baseType: !15, size: 64)
+!15 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !16, size: 64)
+!16 = distinct !DICompositeType(tag: DW_TAG_structure_type, name: "t1", file: !1, line: 1, size: 32, elements: !17)
+!17 = !{!18}
+!18 = !DIDerivedType(tag: DW_TAG_member, name: "a", scope: !16, file: !1, line: 2, baseType: !10, size: 32)
+!19 = !DIDerivedType(tag: DW_TAG_member, name: "p2", scope: !12, file: !1, line: 6, baseType: !15, size: 64, offset: 64)
+!20 = !DIDerivedType(tag: DW_TAG_member, name: "b", scope: !12, file: !1, line: 7, baseType: !10, size: 32, offset: 128)
+!21 = !{!22}
+!22 = !DILocalVariable(name: "arg", arg: 1, scope: !7, file: !1, line: 9, type: !11)
+!23 = !DILocation(line: 0, scope: !7)
+!24 = !DILocation(line: 10, column: 15, scope: !7)
+!25 = !{!26, !30, i64 16}
+!26 = !{!"t2", !27, i64 0, !27, i64 8, !30, i64 16}
+!27 = !{!"any pointer", !28, i64 0}
+!28 = !{!"omnipotent char", !29, i64 0}
+!29 = !{!"Simple C/C++ TBAA"}
+!30 = !{!"int", !28, i64 0}
+!31 = !DILocation(line: 10, column: 3, scope: !7)
diff --git a/llvm/test/CodeGen/Generic/pr33094.ll b/llvm/test/CodeGen/Generic/pr33094.ll
index f1cf67b0ae63b..cd98ec3b139ba 100644
--- a/llvm/test/CodeGen/Generic/pr33094.ll
+++ b/llvm/test/CodeGen/Generic/pr33094.ll
@@ -12,8 +12,8 @@
 @B_Inst = global %B zeroinitializer
 
 define i64 @foo() {
-  %e = extractvalue %Tuple select (i1 icmp eq
-                        (ptr @A_Inst, ptr @B_Inst),
-                        %Tuple { i64 33 }, %Tuple { i64 42 }), 0
+  %s = select i1 icmp eq (ptr @A_Inst, ptr @B_Inst),
+       %Tuple { i64 33 }, %Tuple { i64 42 }
+  %e = extractvalue %Tuple %s, 0
   ret i64 %e
 }
diff --git a/llvm/test/CodeGen/Hexagon/autohvx/vector-align-order.ll b/llvm/test/CodeGen/Hexagon/autohvx/vector-align-order.ll
new file mode 100644
index 0000000000000..f8bf0e9957ab4
--- /dev/null
+++ b/llvm/test/CodeGen/Hexagon/autohvx/vector-align-order.ll
@@ -0,0 +1,65 @@
+; RUN: llc -march=hexagon < %s | FileCheck %s
+
+; Check that this doesn't crash.
+; CHECK: vmem
+
+target datalayout = "e-m:e-p:32:32:32-a:0-n16:32-i64:64:64-i32:32:32-i16:16:16-i1:8:8-f32:32:32-f64:64:64-v32:32:32-v64:64:64-v512:512:512-v1024:1024:1024-v2048:2048:2048"
+target triple = "hexagon"
+
+@g0 = external dllexport local_unnamed_addr global ptr, align 4
+@g1 = external dllexport local_unnamed_addr global ptr, align 4
+
+define dso_local fastcc void @f0(ptr %a0, i32 %a1) unnamed_addr #0 {
+b0:
+  %v0 = load ptr, ptr @g1, align 4
+  %v1 = tail call ptr %v0(i32 16, i32 %a1, i64 2400, i32 0, i32 32)
+  br i1 poison, label %b2, label %b1
+
+b1:                                               ; preds = %b0
+  %v2 = load <64 x i8>, ptr poison, align 64
+  %v3 = zext <64 x i8> %v2 to <64 x i32>
+  %v4 = load <128 x i8>, ptr poison, align 64
+  %v5 = zext <128 x i8> %v4 to <128 x i32>
+  %v6 = load <128 x i8>, ptr poison, align 64
+  %v7 = zext <128 x i8> %v6 to <128 x i32>
+  %v8 = getelementptr i8, ptr %a0, i32 576
+  %v9 = getelementptr i8, ptr %v8, i32 -64
+  %v10 = load <64 x i8>, ptr %v9, align 64
+  %v11 = sext <64 x i8> %v10 to <64 x i32>
+  %v12 = add nsw <64 x i32> %v11, <i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128>
+  %v13 = mul nuw nsw <64 x i32> %v12, %v3
+  %v14 = extractelement <64 x i32> %v13, i32 0
+  store i32 %v14, ptr %v1, align 8, !tbaa !0
+  %v15 = getelementptr i8, ptr %v8, i32 384
+  %v16 = load <128 x i8>, ptr %v15, align 64, !tbaa !3
+  %v17 = sext <128 x i8> %v16 to <128 x i32>
+  %v18 = add nsw <128 x i32> %v17, <i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128>
+  %v19 = mul nuw nsw <128 x i32> %v18, %v5
+  %v20 = getelementptr i8, ptr %v8, i32 256
+  %v21 = load <128 x i8>, ptr %v20, align 64, !tbaa !3
+  %v22 = sext <128 x i8> %v21 to <128 x i32>
+  %v23 = add nsw <128 x i32> %v22, <i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128, i32 128>
+  %v24 = mul nuw nsw <128 x i32> %v23, %v7
+  %v25 = add nsw <128 x i32> %v19, %v24
+  %v26 = extractelement <128 x i32> %v25, i32 0
+  %v27 = insertelement <64 x i32> <i32 poison, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0>, i32 %v26, i64 0
+  %v28 = extractelement <64 x i32> %v27, i32 0
+  %v29 = getelementptr i8, ptr %v1, i32 4
+  store i32 %v28, ptr %v29, align 4
+  %v30 = load ptr, ptr @g0, align 4
+  %v31 = call i32 %v30(ptr nonnull @f1, ptr nonnull poison, i32 0)
+  unreachable
+
+b2:                                               ; preds = %b0
+  ret void
+}
+
+declare dso_local i32 @f1(i32, ptr, ptr) #0
+
+attributes #0 = { "target-features"="+hvxv68,+hvx-length128b,+hvx-qfloat,-hvx-ieee-fp" }
+
+!0 = !{!1, !1, i64 0}
+!1 = !{!"0x555e886c7bc0", !2, i64 0}
+!2 = !{!"tvm-tbaa"}
+!3 = !{!4, !4, i64 0}
+!4 = !{!"0x555e88cb2ce0", !2, i64 0}
diff --git a/llvm/test/CodeGen/Hexagon/hexagon_vector_loop_carried_reuse.ll b/llvm/test/CodeGen/Hexagon/hexagon_vector_loop_carried_reuse.ll
index d7b18ae078ff0..0771fda02cfb7 100644
--- a/llvm/test/CodeGen/Hexagon/hexagon_vector_loop_carried_reuse.ll
+++ b/llvm/test/CodeGen/Hexagon/hexagon_vector_loop_carried_reuse.ll
@@ -1,4 +1,4 @@
-; RUN: opt < %s -hexagon-vlcr -adce -S | FileCheck %s
+; RUN: opt < %s -hexagon-vlcr | opt -passes=adce -S | FileCheck %s
 
 ; CHECK: %.hexagon.vlcr = tail call <32 x i32> @llvm.hexagon.V6.vmaxub.128B
 ; ModuleID = 'hexagon_vector_loop_carried_reuse.c'
diff --git a/llvm/test/CodeGen/Hexagon/hexagon_vector_loop_carried_reuse_commutative.ll b/llvm/test/CodeGen/Hexagon/hexagon_vector_loop_carried_reuse_commutative.ll
index b280830d403a5..25afb9f1a137d 100644
--- a/llvm/test/CodeGen/Hexagon/hexagon_vector_loop_carried_reuse_commutative.ll
+++ b/llvm/test/CodeGen/Hexagon/hexagon_vector_loop_carried_reuse_commutative.ll
@@ -1,4 +1,4 @@
-; RUN: opt -march=hexagon < %s -hexagon-vlcr -adce -S | FileCheck %s
+; RUN: opt < %s -march=hexagon -hexagon-vlcr | opt -passes=adce -S | FileCheck %s
 
 ; CHECK: %v32.hexagon.vlcr = tail call <32 x i32> @llvm.hexagon.V6.vmaxub.128B
 
diff --git a/llvm/test/CodeGen/Hexagon/hexagon_vector_loop_carried_reuse_constant.ll b/llvm/test/CodeGen/Hexagon/hexagon_vector_loop_carried_reuse_constant.ll
index c64404f977754..53973423732cd 100644
--- a/llvm/test/CodeGen/Hexagon/hexagon_vector_loop_carried_reuse_constant.ll
+++ b/llvm/test/CodeGen/Hexagon/hexagon_vector_loop_carried_reuse_constant.ll
@@ -1,4 +1,4 @@
-; RUN: opt < %s -hexagon-vlcr -adce -S | FileCheck %s
+; RUN: opt < %s -hexagon-vlcr | opt -passes=adce -S | FileCheck %s
 
 ; CHECK-NOT: %.hexagon.vlcr
 ; ModuleID = 'hexagon_vector_loop_carried_reuse.c'
diff --git a/llvm/test/CodeGen/LoongArch/bitreverse.ll b/llvm/test/CodeGen/LoongArch/bitreverse.ll
index 8cc731eaa7a54..259d8565c6842 100644
--- a/llvm/test/CodeGen/LoongArch/bitreverse.ll
+++ b/llvm/test/CodeGen/LoongArch/bitreverse.ll
@@ -111,11 +111,9 @@ define i24 @test_bitreverse_i24(i24 %a) nounwind {
 define i48 @test_bitreverse_i48(i48 %a) nounwind {
 ; LA32-LABEL: test_bitreverse_i48:
 ; LA32:       # %bb.0:
-; LA32-NEXT:    bitrev.w $a1, $a1
-; LA32-NEXT:    srli.w $a1, $a1, 16
 ; LA32-NEXT:    bitrev.w $a2, $a0
-; LA32-NEXT:    slli.w $a0, $a2, 16
-; LA32-NEXT:    or $a0, $a1, $a0
+; LA32-NEXT:    bitrev.w $a0, $a1
+; LA32-NEXT:    bytepick.w $a0, $a0, $a2, 2
 ; LA32-NEXT:    srli.w $a1, $a2, 16
 ; LA32-NEXT:    ret
 ;
diff --git a/llvm/test/CodeGen/LoongArch/bswap.ll b/llvm/test/CodeGen/LoongArch/bswap.ll
index 1ef73b4f1c0bf..eb9107302ef68 100644
--- a/llvm/test/CodeGen/LoongArch/bswap.ll
+++ b/llvm/test/CodeGen/LoongArch/bswap.ll
@@ -63,13 +63,11 @@ define i64 @test_bswap_i64(i64 %a) nounwind {
 define i48 @test_bswap_i48(i48 %a) nounwind {
 ; LA32-LABEL: test_bswap_i48:
 ; LA32:       # %bb.0:
-; LA32-NEXT:    revb.2h $a1, $a1
-; LA32-NEXT:    rotri.w $a1, $a1, 16
-; LA32-NEXT:    srli.w $a1, $a1, 16
 ; LA32-NEXT:    revb.2h $a0, $a0
 ; LA32-NEXT:    rotri.w $a2, $a0, 16
-; LA32-NEXT:    slli.w $a0, $a2, 16
-; LA32-NEXT:    or $a0, $a1, $a0
+; LA32-NEXT:    revb.2h $a0, $a1
+; LA32-NEXT:    rotri.w $a0, $a0, 16
+; LA32-NEXT:    bytepick.w $a0, $a0, $a2, 2
 ; LA32-NEXT:    srli.w $a1, $a2, 16
 ; LA32-NEXT:    ret
 ;
@@ -91,28 +89,22 @@ define i80 @test_bswap_i80(i80 %a) nounwind {
 ; LA32-NEXT:    ld.w $a3, $a1, 4
 ; LA32-NEXT:    revb.2h $a3, $a3
 ; LA32-NEXT:    rotri.w $a3, $a3, 16
-; LA32-NEXT:    srli.w $a4, $a3, 16
-; LA32-NEXT:    slli.w $a5, $a2, 16
-; LA32-NEXT:    or $a4, $a5, $a4
-; LA32-NEXT:    srli.w $a2, $a2, 16
-; LA32-NEXT:    st.h $a2, $a0, 8
+; LA32-NEXT:    bytepick.w $a4, $a3, $a2, 2
 ; LA32-NEXT:    st.w $a4, $a0, 4
-; LA32-NEXT:    slli.w $a2, $a3, 16
 ; LA32-NEXT:    ld.w $a1, $a1, 8
 ; LA32-NEXT:    revb.2h $a1, $a1
 ; LA32-NEXT:    rotri.w $a1, $a1, 16
-; LA32-NEXT:    srli.w $a1, $a1, 16
-; LA32-NEXT:    or $a1, $a1, $a2
+; LA32-NEXT:    bytepick.w $a1, $a1, $a3, 2
 ; LA32-NEXT:    st.w $a1, $a0, 0
+; LA32-NEXT:    srli.w $a1, $a2, 16
+; LA32-NEXT:    st.h $a1, $a0, 8
 ; LA32-NEXT:    ret
 ;
 ; LA64-LABEL: test_bswap_i80:
 ; LA64:       # %bb.0:
-; LA64-NEXT:    revb.d $a1, $a1
-; LA64-NEXT:    srli.d $a1, $a1, 48
 ; LA64-NEXT:    revb.d $a2, $a0
-; LA64-NEXT:    slli.d $a0, $a2, 16
-; LA64-NEXT:    or $a0, $a1, $a0
+; LA64-NEXT:    revb.d $a0, $a1
+; LA64-NEXT:    bytepick.d $a0, $a0, $a2, 2
 ; LA64-NEXT:    srli.d $a1, $a2, 48
 ; LA64-NEXT:    ret
   %tmp = call i80 @llvm.bswap.i80(i80 %a)
diff --git a/llvm/test/CodeGen/LoongArch/bytepick.ll b/llvm/test/CodeGen/LoongArch/bytepick.ll
new file mode 100644
index 0000000000000..86148b374e70d
--- /dev/null
+++ b/llvm/test/CodeGen/LoongArch/bytepick.ll
@@ -0,0 +1,258 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc --mtriple=loongarch32 --verify-machineinstrs < %s \
+; RUN:   | FileCheck %s --check-prefix=LA32
+; RUN: llc --mtriple=loongarch64 --verify-machineinstrs < %s \
+; RUN:   | FileCheck %s --check-prefix=LA64
+
+;; a=00112233 b=44556677
+;; expected 11223344
+define i32 @pick_i32_1(i32 %a, i32 %b) {
+; LA32-LABEL: pick_i32_1:
+; LA32:       # %bb.0:
+; LA32-NEXT:    bytepick.w $a0, $a1, $a0, 1
+; LA32-NEXT:    ret
+;
+; LA64-LABEL: pick_i32_1:
+; LA64:       # %bb.0:
+; LA64-NEXT:    slli.d $a0, $a0, 8
+; LA64-NEXT:    bstrpick.d $a1, $a1, 31, 24
+; LA64-NEXT:    or $a0, $a1, $a0
+; LA64-NEXT:    ret
+  %1 = lshr i32 %b, 24
+  %2 = shl i32 %a, 8
+  %3 = or i32 %1, %2
+  ret i32 %3
+}
+
+;; a=00112233 b=44556677
+;; expected 11223344
+define signext i32 @pick_i32_1_sext(i32 %a, i32 %b) {
+; LA32-LABEL: pick_i32_1_sext:
+; LA32:       # %bb.0:
+; LA32-NEXT:    bytepick.w $a0, $a1, $a0, 1
+; LA32-NEXT:    ret
+;
+; LA64-LABEL: pick_i32_1_sext:
+; LA64:       # %bb.0:
+; LA64-NEXT:    bytepick.w $a0, $a1, $a0, 1
+; LA64-NEXT:    ret
+  %1 = lshr i32 %b, 24
+  %2 = shl i32 %a, 8
+  %3 = or i32 %1, %2
+  ret i32 %3
+}
+
+;; a=00112233 b=44556677
+;; expected 22334455
+define i32 @pick_i32_2(i32 %a, i32 %b) {
+; LA32-LABEL: pick_i32_2:
+; LA32:       # %bb.0:
+; LA32-NEXT:    bytepick.w $a0, $a1, $a0, 2
+; LA32-NEXT:    ret
+;
+; LA64-LABEL: pick_i32_2:
+; LA64:       # %bb.0:
+; LA64-NEXT:    slli.d $a0, $a0, 16
+; LA64-NEXT:    bstrpick.d $a1, $a1, 31, 16
+; LA64-NEXT:    or $a0, $a1, $a0
+; LA64-NEXT:    ret
+  %1 = lshr i32 %b, 16
+  %2 = shl i32 %a, 16
+  %3 = or i32 %1, %2
+  ret i32 %3
+}
+
+;; a=00112233 b=44556677
+;; expected 22334455
+define signext i32 @pick_i32_2_sext(i32 %a, i32 %b) {
+; LA32-LABEL: pick_i32_2_sext:
+; LA32:       # %bb.0:
+; LA32-NEXT:    bytepick.w $a0, $a1, $a0, 2
+; LA32-NEXT:    ret
+;
+; LA64-LABEL: pick_i32_2_sext:
+; LA64:       # %bb.0:
+; LA64-NEXT:    bytepick.w $a0, $a1, $a0, 2
+; LA64-NEXT:    ret
+  %1 = lshr i32 %b, 16
+  %2 = shl i32 %a, 16
+  %3 = or i32 %1, %2
+  ret i32 %3
+}
+
+;; a=00112233 b=44556677
+;; expected 33445566
+define i32 @pick_i32_3(i32 %a, i32 %b) {
+; LA32-LABEL: pick_i32_3:
+; LA32:       # %bb.0:
+; LA32-NEXT:    bytepick.w $a0, $a1, $a0, 3
+; LA32-NEXT:    ret
+;
+; LA64-LABEL: pick_i32_3:
+; LA64:       # %bb.0:
+; LA64-NEXT:    slli.d $a0, $a0, 24
+; LA64-NEXT:    bstrpick.d $a1, $a1, 31, 8
+; LA64-NEXT:    or $a0, $a1, $a0
+; LA64-NEXT:    ret
+  %1 = lshr i32 %b, 8
+  %2 = shl i32 %a, 24
+  %3 = or i32 %1, %2
+  ret i32 %3
+}
+
+;; a=00112233 b=44556677
+;; expected 33445566
+define signext i32 @pick_i32_3_sext(i32 %a, i32 %b) {
+; LA32-LABEL: pick_i32_3_sext:
+; LA32:       # %bb.0:
+; LA32-NEXT:    bytepick.w $a0, $a1, $a0, 3
+; LA32-NEXT:    ret
+;
+; LA64-LABEL: pick_i32_3_sext:
+; LA64:       # %bb.0:
+; LA64-NEXT:    bytepick.w $a0, $a1, $a0, 3
+; LA64-NEXT:    ret
+  %1 = lshr i32 %b, 8
+  %2 = shl i32 %a, 24
+  %3 = or i32 %1, %2
+  ret i32 %3
+}
+
+;; a=0011223344556677 b=8899aabbccddeeff
+;; expected 1122334455667788
+define i64 @pick_i64_1(i64 %a, i64 %b) {
+; LA32-LABEL: pick_i64_1:
+; LA32:       # %bb.0:
+; LA32-NEXT:    bytepick.w $a2, $a3, $a0, 1
+; LA32-NEXT:    bytepick.w $a1, $a0, $a1, 1
+; LA32-NEXT:    move $a0, $a2
+; LA32-NEXT:    ret
+;
+; LA64-LABEL: pick_i64_1:
+; LA64:       # %bb.0:
+; LA64-NEXT:    bytepick.d $a0, $a1, $a0, 1
+; LA64-NEXT:    ret
+  %1 = lshr i64 %b, 56
+  %2 = shl i64 %a, 8
+  %3 = or i64 %1, %2
+  ret i64 %3
+}
+
+;; a=0011223344556677 b=8899aabbccddeeff
+;; expected 2233445566778899
+define i64 @pick_i64_2(i64 %a, i64 %b) {
+; LA32-LABEL: pick_i64_2:
+; LA32:       # %bb.0:
+; LA32-NEXT:    bytepick.w $a2, $a3, $a0, 2
+; LA32-NEXT:    bytepick.w $a1, $a0, $a1, 2
+; LA32-NEXT:    move $a0, $a2
+; LA32-NEXT:    ret
+;
+; LA64-LABEL: pick_i64_2:
+; LA64:       # %bb.0:
+; LA64-NEXT:    bytepick.d $a0, $a1, $a0, 2
+; LA64-NEXT:    ret
+  %1 = lshr i64 %b, 48
+  %2 = shl i64 %a, 16
+  %3 = or i64 %1, %2
+  ret i64 %3
+}
+
+;; a=0011223344556677 b=8899aabbccddeeff
+;; expected 33445566778899aa
+define i64 @pick_i64_3(i64 %a, i64 %b) {
+; LA32-LABEL: pick_i64_3:
+; LA32:       # %bb.0:
+; LA32-NEXT:    bytepick.w $a2, $a3, $a0, 3
+; LA32-NEXT:    bytepick.w $a1, $a0, $a1, 3
+; LA32-NEXT:    move $a0, $a2
+; LA32-NEXT:    ret
+;
+; LA64-LABEL: pick_i64_3:
+; LA64:       # %bb.0:
+; LA64-NEXT:    bytepick.d $a0, $a1, $a0, 3
+; LA64-NEXT:    ret
+  %1 = lshr i64 %b, 40
+  %2 = shl i64 %a, 24
+  %3 = or i64 %1, %2
+  ret i64 %3
+}
+
+;; a=0011223344556677 b=8899aabbccddeeff
+;; expected 445566778899aabb
+define i64 @pick_i64_4(i64 %a, i64 %b) {
+; LA32-LABEL: pick_i64_4:
+; LA32:       # %bb.0:
+; LA32-NEXT:    move $a1, $a0
+; LA32-NEXT:    move $a0, $a3
+; LA32-NEXT:    ret
+;
+; LA64-LABEL: pick_i64_4:
+; LA64:       # %bb.0:
+; LA64-NEXT:    bytepick.d $a0, $a1, $a0, 4
+; LA64-NEXT:    ret
+  %1 = lshr i64 %b, 32
+  %2 = shl i64 %a, 32
+  %3 = or i64 %1, %2
+  ret i64 %3
+}
+
+;; a=0011223344556677 b=8899aabbccddeeff
+;; expected 5566778899aabbcc
+define i64 @pick_i64_5(i64 %a, i64 %b) {
+; LA32-LABEL: pick_i64_5:
+; LA32:       # %bb.0:
+; LA32-NEXT:    bytepick.w $a2, $a2, $a3, 1
+; LA32-NEXT:    bytepick.w $a1, $a3, $a0, 1
+; LA32-NEXT:    move $a0, $a2
+; LA32-NEXT:    ret
+;
+; LA64-LABEL: pick_i64_5:
+; LA64:       # %bb.0:
+; LA64-NEXT:    bytepick.d $a0, $a1, $a0, 5
+; LA64-NEXT:    ret
+  %1 = lshr i64 %b, 24
+  %2 = shl i64 %a,40
+  %3 = or i64 %1, %2
+  ret i64 %3
+}
+
+;; a=0011223344556677 b=8899aabbccddeeff
+;; expected 66778899aabbccdd
+define i64 @pick_i64_6(i64 %a, i64 %b) {
+; LA32-LABEL: pick_i64_6:
+; LA32:       # %bb.0:
+; LA32-NEXT:    bytepick.w $a2, $a2, $a3, 2
+; LA32-NEXT:    bytepick.w $a1, $a3, $a0, 2
+; LA32-NEXT:    move $a0, $a2
+; LA32-NEXT:    ret
+;
+; LA64-LABEL: pick_i64_6:
+; LA64:       # %bb.0:
+; LA64-NEXT:    bytepick.d $a0, $a1, $a0, 6
+; LA64-NEXT:    ret
+  %1 = lshr i64 %b, 16
+  %2 = shl i64 %a, 48
+  %3 = or i64 %1, %2
+  ret i64 %3
+}
+
+;; a=0011223344556677 b=8899aabbccddeeff
+;; expected 778899aabbccddee
+define i64 @pick_i64_7(i64 %a, i64 %b) {
+; LA32-LABEL: pick_i64_7:
+; LA32:       # %bb.0:
+; LA32-NEXT:    bytepick.w $a2, $a2, $a3, 3
+; LA32-NEXT:    bytepick.w $a1, $a3, $a0, 3
+; LA32-NEXT:    move $a0, $a2
+; LA32-NEXT:    ret
+;
+; LA64-LABEL: pick_i64_7:
+; LA64:       # %bb.0:
+; LA64-NEXT:    bytepick.d $a0, $a1, $a0, 7
+; LA64-NEXT:    ret
+  %1 = lshr i64 %b, 8
+  %2 = shl i64 %a, 56
+  %3 = or i64 %1, %2
+  ret i64 %3
+}
diff --git a/llvm/test/CodeGen/LoongArch/patchable-function-entry.ll b/llvm/test/CodeGen/LoongArch/patchable-function-entry.ll
new file mode 100644
index 0000000000000..12d4bfb50a198
--- /dev/null
+++ b/llvm/test/CodeGen/LoongArch/patchable-function-entry.ll
@@ -0,0 +1,63 @@
+;; Test the function attribute "patchable-function-entry".
+;; Adapted from the RISCV test case.
+; RUN: llc --mtriple=loongarch32 < %s | FileCheck %s --check-prefixes=CHECK,LA32
+; RUN: llc --mtriple=loongarch64 < %s | FileCheck %s --check-prefixes=CHECK,LA64
+
+define void @f0() "patchable-function-entry"="0" {
+; CHECK-LABEL: f0:
+; CHECK-NEXT:  .Lfunc_begin0:
+; CHECK-NOT:     nop
+; CHECK:         ret
+; CHECK-NOT:   .section __patchable_function_entries
+  ret void
+}
+
+define void @f1() "patchable-function-entry"="1" {
+; CHECK-LABEL: f1:
+; CHECK-NEXT: .Lfunc_begin1:
+; CHECK:         nop
+; CHECK-NEXT:    ret
+; CHECK:       .section __patchable_function_entries,"awo",@progbits,f1{{$}}
+; LA32:        .p2align 2
+; LA32-NEXT:   .word .Lfunc_begin1
+; LA64:        .p2align 3
+; LA64-NEXT:   .dword .Lfunc_begin1
+  ret void
+}
+
+$f5 = comdat any
+define void @f5() "patchable-function-entry"="5" comdat {
+; CHECK-LABEL:   f5:
+; CHECK-NEXT:    .Lfunc_begin2:
+; CHECK-COUNT-5:   nop
+; CHECK-NEXT:      ret
+; CHECK:         .section __patchable_function_entries,"aGwo",@progbits,f5,comdat,f5{{$}}
+; LA32:          .p2align 2
+; LA32-NEXT:     .word .Lfunc_begin2
+; LA64:          .p2align 3
+; LA64-NEXT:     .dword .Lfunc_begin2
+  ret void
+}
+
+;; -fpatchable-function-entry=3,2
+;; "patchable-function-prefix" emits data before the function entry label.
+define void @f3_2() "patchable-function-entry"="1" "patchable-function-prefix"="2" {
+; CHECK-LABEL:   .type f3_2,@function
+; CHECK-NEXT:    .Ltmp0: # @f3_2
+; CHECK-COUNT-2:   nop
+; CHECK-NEXT:    f3_2:
+; CHECK:         # %bb.0:
+; CHECK-NEXT:      nop
+; LA32-NEXT:       addi.w $sp, $sp, -16
+; LA64-NEXT:       addi.d $sp, $sp, -16
+;; .size does not include the prefix.
+; CHECK:      .Lfunc_end3:
+; CHECK-NEXT: .size f3_2, .Lfunc_end3-f3_2
+; CHECK:      .section __patchable_function_entries,"awo",@progbits,f3_2{{$}}
+; LA32:       .p2align 2
+; LA32-NEXT:  .word .Ltmp0
+; LA64:       .p2align 3
+; LA64-NEXT:  .dword .Ltmp0
+  %frame = alloca i8, i32 16
+  ret void
+}
diff --git a/llvm/test/CodeGen/M68k/inline-asm.ll b/llvm/test/CodeGen/M68k/inline-asm.ll
index 6f9260f23cff6..72cdfb78e82d3 100644
--- a/llvm/test/CodeGen/M68k/inline-asm.ll
+++ b/llvm/test/CodeGen/M68k/inline-asm.ll
@@ -1,6 +1,8 @@
 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=m68k < %s -o - | FileCheck %s
 
+@g = internal global i32 10, align 4
+
 ; This function is primarily testing constant constraints that can NOT
 ; be easily checked by Clang. For example, 'K' and 'M' are both
 ; constraints for values that are outside certain numerical range.
@@ -120,3 +122,33 @@ entry:
   ret void
 }
 
+define void @memory_constraints() {
+; CHECK-LABEL: memory_constraints:
+; CHECK:         .cfi_startproc
+; CHECK-NEXT:  ; %bb.0: ; %entry
+; CHECK-NEXT:    suba.l #4, %sp
+; CHECK-NEXT:    .cfi_def_cfa_offset -8
+; CHECK-NEXT:    ;APP
+; CHECK-NEXT:    move.l (0,%sp), %d1
+; CHECK-NEXT:    ;NO_APP
+; CHECK-NEXT:    ;APP
+; CHECK-NEXT:    move.l (g,%pc), %d2
+; CHECK-NEXT:    ;NO_APP
+; CHECK-NEXT:    lea (0,%sp), %a0
+; CHECK-NEXT:    ;APP
+; CHECK-NEXT:    move.l (%a0), %d3
+; CHECK-NEXT:    ;NO_APP
+; CHECK-NEXT:    ;APP
+; CHECK-NEXT:    move.l (0,%sp), %d4
+; CHECK-NEXT:    ;NO_APP
+; CHECK-NEXT:    adda.l #4, %sp
+; CHECK-NEXT:    rts
+entry:
+  %x = alloca i32, align 4
+  call void asm sideeffect "move.l $0, %d1", "*m"(ptr elementtype(i32) %x)
+  call void asm sideeffect "move.l $0, %d2", "*m"(ptr elementtype(i32) @g)
+  call void asm sideeffect "move.l $0, %d3", "*Q"(ptr elementtype(i32) %x)
+  call void asm sideeffect "move.l $0, %d4", "*U"(ptr elementtype(i32) %x)
+  ret void
+}
+
diff --git a/llvm/test/CodeGen/Mips/o32_cc_byval.ll b/llvm/test/CodeGen/Mips/o32_cc_byval.ll
index 94c327b5a4665..de6b4dd2ab71d 100644
--- a/llvm/test/CodeGen/Mips/o32_cc_byval.ll
+++ b/llvm/test/CodeGen/Mips/o32_cc_byval.ll
@@ -69,8 +69,8 @@ define void @f1() nounwind {
 ; CHECK-NEXT:    sw $1, 16($sp)
 ; CHECK-NEXT:    lw $7, 4($18)
 ; CHECK-NEXT:    lw $6, %lo(f1.s1)($17)
-; CHECK-NEXT:    lbu $5, 40($sp)
 ; CHECK-NEXT:    lw $25, %call16(callee3)($16)
+; CHECK-NEXT:    addiu $5, $zero, 11
 ; CHECK-NEXT:    jalr $25
 ; CHECK-NEXT:    move $gp, $16
 ; CHECK-NEXT:    lw $16, 48($sp) # 4-byte Folded Reload
@@ -234,6 +234,7 @@ define void @f5(i64 %a0, ptr nocapture byval(%struct.S4) %a1) nounwind {
 ; CHECK-NEXT:    addiu $sp, $sp, -32
 ; CHECK-NEXT:    sw $ra, 28($sp) # 4-byte Folded Spill
 ; CHECK-NEXT:    addu $gp, $2, $25
+; CHECK-NEXT:    move $1, $6
 ; CHECK-NEXT:    sw $7, 44($sp)
 ; CHECK-NEXT:    sw $6, 40($sp)
 ; CHECK-NEXT:    sw $5, 20($sp)
@@ -243,7 +244,7 @@ define void @f5(i64 %a0, ptr nocapture byval(%struct.S4) %a1) nounwind {
 ; CHECK-NEXT:    lw $5, 44($sp)
 ; CHECK-NEXT:    lw $25, %call16(f6)($gp)
 ; CHECK-NEXT:    jalr $25
-; CHECK-NEXT:    lw $4, 40($sp)
+; CHECK-NEXT:    move $4, $1
 ; CHECK-NEXT:    lw $ra, 28($sp) # 4-byte Folded Reload
 ; CHECK-NEXT:    jr $ra
 ; CHECK-NEXT:    addiu $sp, $sp, 32
diff --git a/llvm/test/CodeGen/NVPTX/anonymous-fn-param.ll b/llvm/test/CodeGen/NVPTX/anonymous-fn-param.ll
new file mode 100644
index 0000000000000..27e08c9d75639
--- /dev/null
+++ b/llvm/test/CodeGen/NVPTX/anonymous-fn-param.ll
@@ -0,0 +1,25 @@
+; RUN: llc < %s -march=nvptx64 -mcpu=sm_20 | FileCheck %s
+; RUN: %if ptxas %{ llc < %s -march=nvptx64 -mcpu=sm_20 | %ptxas-verify %}
+
+; Check that parameter names we generate in the function signature and the name
+; we use when we refer to the parameter in the function body do match.
+
+; CHECK:      .func (.param .b32 func_retval0) __unnamed_1(
+; CHECK-NEXT: .param .b32 __unnamed_1_param_0
+; CHECK:      ld.param.u32 {{%r[0-9]+}}, [__unnamed_1_param_0];
+
+define internal i32 @0(i32 %a) {
+entry:
+  %r = add i32 %a, 1
+  ret i32 %r
+}
+
+; CHECK:      .func (.param .b32 func_retval0) __unnamed_2(
+; CHECK-NEXT: .param .b32 __unnamed_2_param_0
+; CHECK:      ld.param.u32 {{%r[0-9]+}}, [__unnamed_2_param_0];
+
+define internal i32 @1(i32 %a) {
+entry:
+  %r = add i32 %a, 1
+  ret i32 %r
+}
diff --git a/llvm/test/CodeGen/NVPTX/empty-type.ll b/llvm/test/CodeGen/NVPTX/empty-type.ll
new file mode 100644
index 0000000000000..e5c02f99681c2
--- /dev/null
+++ b/llvm/test/CodeGen/NVPTX/empty-type.ll
@@ -0,0 +1,10 @@
+; RUN: not --crash llc < %s -march=nvptx -mcpu=sm_20 2>&1 | FileCheck %s
+
+%struct.A = type { [0 x float] }
+%struct.B = type { i32, i32 }
+
+; CHECK: ERROR: Empty parameter types are not supported
+define void @kernel(%struct.A %a, %struct.B %b) {
+entry:
+  ret void
+}
diff --git a/llvm/test/CodeGen/NVPTX/ldu-ldg.ll b/llvm/test/CodeGen/NVPTX/ldu-ldg.ll
index d40eb7a32027d..c152f835afe07 100644
--- a/llvm/test/CodeGen/NVPTX/ldu-ldg.ll
+++ b/llvm/test/CodeGen/NVPTX/ldu-ldg.ll
@@ -3,7 +3,13 @@
 
 
 declare i8 @llvm.nvvm.ldu.global.i.i8.p1(ptr addrspace(1) %ptr, i32 %align)
+declare i16 @llvm.nvvm.ldu.global.i.i16.p1(ptr addrspace(1) %ptr, i32 %align)
 declare i32 @llvm.nvvm.ldu.global.i.i32.p1(ptr addrspace(1) %ptr, i32 %align)
+declare i64 @llvm.nvvm.ldu.global.i.i64.p1(ptr addrspace(1) %ptr, i32 %align)
+declare float @llvm.nvvm.ldu.global.f.f32.p1(ptr addrspace(1) %ptr, i32 %align)
+declare double @llvm.nvvm.ldu.global.f.f64.p1(ptr addrspace(1) %ptr, i32 %align)
+declare half @llvm.nvvm.ldu.global.f.f16.p1(ptr addrspace(1) %ptr, i32 %align)
+declare <2 x half> @llvm.nvvm.ldu.global.f.v2f16.p1(ptr addrspace(1) %ptr, i32 %align)
 
 declare i8 @llvm.nvvm.ldg.global.i.i8.p1(ptr addrspace(1) %ptr, i32 %align)
 declare i16 @llvm.nvvm.ldg.global.i.i16.p1(ptr addrspace(1) %ptr, i32 %align)
@@ -14,72 +20,114 @@ declare double @llvm.nvvm.ldg.global.f.f64.p1(ptr addrspace(1) %ptr, i32 %align)
 declare half @llvm.nvvm.ldg.global.f.f16.p1(ptr addrspace(1) %ptr, i32 %align)
 declare <2 x half> @llvm.nvvm.ldg.global.f.v2f16.p1(ptr addrspace(1) %ptr, i32 %align)
 
-; CHECK: test_ldu_i8
+; CHECK-LABEL: test_ldu_i8
 define i8 @test_ldu_i8(ptr addrspace(1) %ptr) {
-; ldu.global.u8
+  ; CHECK: ldu.global.u8
   %val = tail call i8 @llvm.nvvm.ldu.global.i.i8.p1(ptr addrspace(1) %ptr, i32 4)
   ret i8 %val
 }
 
-; CHECK: test_ldu_i32
+; CHECK-LABEL: test_ldu_i16
+define i16 @test_ldu_i16(ptr addrspace(1) %ptr) {
+  ; CHECK: ldu.global.u16
+  %val = tail call i16 @llvm.nvvm.ldu.global.i.i16.p1(ptr addrspace(1) %ptr, i32 2)
+  ret i16 %val
+}
+
+; CHECK-LABEL: test_ldu_i32
 define i32 @test_ldu_i32(ptr addrspace(1) %ptr) {
-; ldu.global.u32
+  ; CHECK: ldu.global.u32
   %val = tail call i32 @llvm.nvvm.ldu.global.i.i32.p1(ptr addrspace(1) %ptr, i32 4)
   ret i32 %val
 }
 
-; CHECK: test_ldg_i8
+; CHECK-LABEL: test_ldu_i64
+define i64 @test_ldu_i64(ptr addrspace(1) %ptr) {
+  ; CHECK: ldu.global.u64
+  %val = tail call i64 @llvm.nvvm.ldu.global.i.i64.p1(ptr addrspace(1) %ptr, i32 8)
+  ret i64 %val
+}
+
+; CHECK-LABEL: test_ldu_f32
+define float @test_ldu_f32(ptr addrspace(1) %ptr) {
+  ; CHECK: ldu.global.f32
+  %val = tail call float @llvm.nvvm.ldu.global.f.f32.p1(ptr addrspace(1) %ptr, i32 4)
+  ret float %val
+}
+
+; CHECK-LABEL: test_ldu_f64
+define double @test_ldu_f64(ptr addrspace(1) %ptr) {
+  ; CHECK: ldu.global.f64
+  %val = tail call double @llvm.nvvm.ldu.global.f.f64.p1(ptr addrspace(1) %ptr, i32 8)
+  ret double %val
+}
+
+; CHECK-LABEL: test_ldu_f16
+define half @test_ldu_f16(ptr addrspace(1) %ptr) {
+  ; CHECK: ldu.global.b16
+  %val = tail call half @llvm.nvvm.ldu.global.f.f16.p1(ptr addrspace(1) %ptr, i32 2)
+  ret half %val
+}
+
+; CHECK-LABEL: test_ldu_v2f16
+define <2 x half> @test_ldu_v2f16(ptr addrspace(1) %ptr) {
+  ; CHECK: ldu.global.b32
+  %val = tail call <2 x half> @llvm.nvvm.ldu.global.f.v2f16.p1(ptr addrspace(1) %ptr, i32 4)
+  ret <2 x half> %val
+}
+
+; CHECK-LABEL: test_ldg_i8
 define i8 @test_ldg_i8(ptr addrspace(1) %ptr) {
-; ld.global.nc.u8
+  ; CHECK: ld.global.nc.u8
   %val = tail call i8 @llvm.nvvm.ldg.global.i.i8.p1(ptr addrspace(1) %ptr, i32 4)
   ret i8 %val
 }
 
-; CHECK: test_ldg_i16
+; CHECK-LABEL: test_ldg_i16
 define i16 @test_ldg_i16(ptr addrspace(1) %ptr) {
-; ld.global.nc.u16
-  %val = tail call i16 @llvm.nvvm.ldg.global.i.i16.p1(ptr addrspace(1) %ptr, i32 4)
+  ; CHECK: ld.global.nc.u16
+  %val = tail call i16 @llvm.nvvm.ldg.global.i.i16.p1(ptr addrspace(1) %ptr, i32 2)
   ret i16 %val
 }
 
-; CHECK: test_ldg_i32
+; CHECK-LABEL: test_ldg_i32
 define i32 @test_ldg_i32(ptr addrspace(1) %ptr) {
-; ld.global.nc.u32
+  ; CHECK: ld.global.nc.u32
   %val = tail call i32 @llvm.nvvm.ldg.global.i.i32.p1(ptr addrspace(1) %ptr, i32 4)
   ret i32 %val
 }
 
-; CHECK: test_ldg_i64
+; CHECK-LABEL: test_ldg_i64
 define i64 @test_ldg_i64(ptr addrspace(1) %ptr) {
-; ld.global.nc.u64
+  ; CHECK: ld.global.nc.u64
   %val = tail call i64 @llvm.nvvm.ldg.global.i.i64.p1(ptr addrspace(1) %ptr, i32 8)
   ret i64 %val
 }
 
-; CHECK: test_ldg_f32
+; CHECK-LABEL: test_ldg_f32
 define float @test_ldg_f32(ptr addrspace(1) %ptr) {
-; ld.global.nc.u64
+  ; CHECK: ld.global.nc.f32
   %val = tail call float @llvm.nvvm.ldg.global.f.f32.p1(ptr addrspace(1) %ptr, i32 4)
   ret float %val
 }
 
-; CHECK: test_ldg_f64
+; CHECK-LABEL: test_ldg_f64
 define double @test_ldg_f64(ptr addrspace(1) %ptr) {
-; ld.global.nc.u64
+  ; CHECK: ld.global.nc.f64
   %val = tail call double @llvm.nvvm.ldg.global.f.f64.p1(ptr addrspace(1) %ptr, i32 8)
   ret double %val
 }
 
-; CHECK: test_ldg_f16
+; CHECK-LABEL: test_ldg_f16
 define half @test_ldg_f16(ptr addrspace(1) %ptr) {
-; ld.global.nc.b16
-  %val = tail call half @llvm.nvvm.ldg.global.f.f16.p1(ptr addrspace(1) %ptr, i32 4)
+  ; CHECK: ld.global.nc.b16
+  %val = tail call half @llvm.nvvm.ldg.global.f.f16.p1(ptr addrspace(1) %ptr, i32 2)
   ret half %val
 }
 
-; CHECK: test_ldg_v2f16
+; CHECK-LABEL: test_ldg_v2f16
 define <2 x half> @test_ldg_v2f16(ptr addrspace(1) %ptr) {
-; ld.global.nc.b32
+  ; CHECK: ld.global.nc.b32
   %val = tail call <2 x half> @llvm.nvvm.ldg.global.f.v2f16.p1(ptr addrspace(1) %ptr, i32 4)
   ret <2 x half> %val
 }
diff --git a/llvm/test/CodeGen/PowerPC/aix-cc-byval.ll b/llvm/test/CodeGen/PowerPC/aix-cc-byval.ll
index 508dd633d3750..5e7a1bc81916e 100644
--- a/llvm/test/CodeGen/PowerPC/aix-cc-byval.ll
+++ b/llvm/test/CodeGen/PowerPC/aix-cc-byval.ll
@@ -958,33 +958,33 @@ declare i32 @test_byval_homogeneous_float_struct(ptr byval(%struct.F) align 4)
 ; CHECK-LABEL: name: call_test_byval_homogeneous_float_struct{{.*}}
 
 ; 32BIT:       ADJCALLSTACKDOWN 56, 0, implicit-def dead $r1, implicit $r1
-; 32BIT-DAG:   renamable $r3 = LWZ 0, %stack.0.s :: (load (s32) from %stack.0.s, align 8)
 ; 32BIT-DAG:   renamable $r4 = LWZ 4, %stack.0.s :: (load (s32) from %stack.0.s + 4)
 ; 32BIT-DAG:   renamable $r5 = LWZ 8, %stack.0.s :: (load (s32) from %stack.0.s + 8, align 8)
+; 32BIT-DAG:   $r3 = LI 0
 ; 32BIT-NEXT:  BL_NOP <mcsymbol .test_byval_homogeneous_float_struct[PR]>, csr_aix32, implicit-def dead $lr, implicit $rm, implicit $r3, implicit $r4, implicit $r5, implicit $r2, implicit-def $r1, implicit-def $r3
 ; 32BIT-NEXT:  ADJCALLSTACKUP 56, 0, implicit-def dead $r1, implicit $r1
 
 ; CHECKASM-LABEL: .call_test_byval_homogeneous_float_struct:
 
 ; ASM32:       stwu 1, -80(1)
-; ASM32-DAG:   lwz 3, 64(1)
 ; ASM32-DAG:   lwz 4, 68(1)
 ; ASM32-DAG:   lwz 5, 72(1)
+; ASM32-DAG:   stw 3, 64(1)
 ; ASM32-NEXT:  bl .test_byval_homogeneous_float_struct[PR]
 ; ASM32-NEXT:  nop
 
 ; The DAG block permits some invalid inputs for the benefit of allowing more valid orderings.
 ; 64BIT:       ADJCALLSTACKDOWN 112, 0, implicit-def dead $r1, implicit $r1
-; 64BIT-DAG:   renamable $x3 = LD 0, %stack.0.s :: (load (s64) from %stack.0.s)
-; 64BIT-DAG:   renamable $x4 = LWZ8 8, %stack.0.s :: (load (s32) from %stack.0.s + 8, align 8)
-; 64BIT-DAG:   renamable $x4 = RLDICR killed renamable $x4, 32, 31
+; 64BIT:       renamable $x3 = LWZ8 8, %stack.0.s :: (load (s32) from %stack.0.s + 8, align 8)
+; 64BIT-NEXT:  renamable $x4 = RLDICR killed renamable $x3, 32, 31
+; 64BIT-NEXT:  $x3 = LI8 0
 ; 64BIT-NEXT:  BL8_NOP <mcsymbol .test_byval_homogeneous_float_struct[PR]>, csr_ppc64, implicit-def dead $lr8, implicit $rm, implicit $x3, implicit $x4, implicit $x2, implicit-def $r1, implicit-def $x3
 ; 64BIT-NEXT:  ADJCALLSTACKUP 112, 0, implicit-def dead $r1, implicit $r1
 
 ; The DAG block permits some invalid inputs for the benefit of allowing more valid orderings.
 ; ASM64:       stdu 1, -128(1)
-; ASM64-DAG:   ld 3, 112(1)
-; ASM64-DAG:   lwz 4, 120(1)
-; ASM64-DAG:   sldi 4, 4, 32
+; ASM64:       lwz 3, 120(1)
+; ASM64-NEXT:  sldi 4, 3, 32
+; ASM64-NEXT:  li 3, 0
 ; ASM64-NEXT:  bl .test_byval_homogeneous_float_struct[PR]
 ; ASM64-NEXT:  nop
diff --git a/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable-clobber-register.ll b/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable-clobber-register.ll
index 95e9077a48edd..2384ca3a74ace 100644
--- a/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable-clobber-register.ll
+++ b/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable-clobber-register.ll
@@ -46,7 +46,7 @@ entry:
 ; COMMON-NEXT:                                        # -IsFloatingPointOperationLogOrAbortEnabled
 ; COMMON-NEXT:  .byte   0x40                            # -IsInterruptHandler, +IsFunctionNamePresent, -IsAllocaUsed
 ; COMMON-NEXT:                                        # OnConditionDirective = 0, -IsCRSaved, -IsLRSaved
-; COMMON-NEXT:  .byte   0x85                            # +IsBackChainStored, -IsFixup, NumOfFPRsSaved = 5
+; COMMON-NEXT:  .byte   0x05                            # -IsBackChainStored, -IsFixup, NumOfFPRsSaved = 5
 ; COMMON-NEXT:  .byte   0x04                            # -HasExtensionTable, -HasVectorInfo, NumOfGPRsSaved = 4
 ; COMMON-NEXT:  .byte   0x00                            # NumberOfFixedParms = 0
 ; COMMON-NEXT:  .byte   0x01                            # NumberOfFPParms = 0, +HasParmsOnStack
@@ -67,7 +67,7 @@ entry:
 ; COMMON-NEXT:                                         # -IsFloatingPointOperationLogOrAbortEnabled
 ; COMMON-NEXT:  .byte   0x40                            # -IsInterruptHandler, +IsFunctionNamePresent, -IsAllocaUsed
 ; COMMON-NEXT:                                         # OnConditionDirective = 0, -IsCRSaved, -IsLRSaved
-; COMMON-NEXT:  .byte   0x80                            # +IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
+; COMMON-NEXT:  .byte   0x00                            # -IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
 ; COMMON-NEXT:  .byte   0xc0                            # +HasExtensionTable, +HasVectorInfo, NumOfGPRsSaved = 0
 ; COMMON-NEXT:  .byte   0x00                            # NumberOfFixedParms = 0
 ; COMMON-NEXT:  .byte   0x01                            # NumberOfFPParms = 0, +HasParmsOnStack
diff --git a/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable-redzone-boundary.mir b/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable-redzone-boundary.mir
new file mode 100644
index 0000000000000..3d4b5a73a8225
--- /dev/null
+++ b/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable-redzone-boundary.mir
@@ -0,0 +1,60 @@
+## Test when the frame size is equal to or large than redzone boundary size(288).
+## The behaviour of IsBackChainStored bit of traceback table.
+# RUN: llc  -mtriple powerpc64-ibm-aix-xcoff -o -\
+# RUN:   %s | FileCheck %s
+
+---
+name:            test
+tracksRegLiveness: true
+stack:
+  - { id: 0, size: 288, alignment: 16 }
+body: |
+  bb.0:
+    BLR8 implicit $lr8, implicit $rm
+
+---
+name:            overredzone
+tracksRegLiveness: true
+stack:
+  - { id: 0, size: 296, alignment: 16 }
+body: |
+  bb.0:
+    BLR8 implicit $lr8, implicit $rm
+
+
+  ; CHECK:              .vbyte  4, 0x00000000                   # Traceback table begin
+  ; CHECK-NEXT:         .byte   0x00                            # Version = 0
+  ; CHECK-NEXT:         .byte   0x09                            # Language = CPlusPlus
+  ; CHECK-NEXT:         .byte   0x20                            # -IsGlobaLinkage, -IsOutOfLineEpilogOrPrologue
+  ; CHECK-NEXT:                                         # +HasTraceBackTableOffset, -IsInternalProcedure
+  ; CHECK-NEXT:                                         # -HasControlledStorage, -IsTOCless
+  ; CHECK-NEXT:                                         # -IsFloatingPointPresent
+  ; CHECK-NEXT:                                         # -IsFloatingPointOperationLogOrAbortEnabled
+  ; CHECK-NEXT:         .byte   0x40                            # -IsInterruptHandler, +IsFunctionNamePresent, -IsAllocaUsed
+  ; CHECK-NEXT:                                         # OnConditionDirective = 0, -IsCRSaved, -IsLRSaved
+  ; CHECK-NEXT:         .byte   0x00                            # -IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
+  ; CHECK-NEXT:         .byte   0x00                            # -HasExtensionTable, -HasVectorInfo, NumOfGPRsSaved = 0
+  ; CHECK-NEXT:         .byte   0x00                            # NumberOfFixedParms = 0
+  ; CHECK-NEXT:         .byte   0x01                            # NumberOfFPParms = 0, +HasParmsOnStack
+  ; CHECK-NEXT:         .vbyte  4, L..test0-.test               # Function size
+  ; CHECK-NEXT:         .vbyte  2, 0x0004                       # Function name len = 4
+  ; CHECK-NEXT:         .byte   "test"                          # Function Name
+
+  ; CHECK:              .vbyte  4, 0x00000000                   # Traceback table begin
+  ; CHECK-NEXT:         .byte   0x00                            # Version = 0
+  ; CHECK-NEXT:         .byte   0x09                            # Language = CPlusPlus
+  ; CHECK-NEXT:         .byte   0x20                            # -IsGlobaLinkage, -IsOutOfLineEpilogOrPrologue
+  ; CHECK-NEXT:                                         # +HasTraceBackTableOffset, -IsInternalProcedure
+  ; CHECK-NEXT:                                         # -HasControlledStorage, -IsTOCless
+  ; CHECK-NEXT:                                         # -IsFloatingPointPresent
+  ; CHECK-NEXT:                                         # -IsFloatingPointOperationLogOrAbortEnabled
+  ; CHECK-NEXT:         .byte   0x40                            # -IsInterruptHandler, +IsFunctionNamePresent, -IsAllocaUsed
+  ; CHECK-NEXT:                                         # OnConditionDirective = 0, -IsCRSaved, -IsLRSaved
+  ; CHECK-NEXT:         .byte   0x80                            # +IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
+  ; CHECK-NEXT:         .byte   0x00                            # -HasExtensionTable, -HasVectorInfo, NumOfGPRsSaved = 0
+  ; CHECK-NEXT:         .byte   0x00                            # NumberOfFixedParms = 0
+  ; CHECK-NEXT:         .byte   0x01                            # NumberOfFPParms = 0, +HasParmsOnStack
+  ; CHECK-NEXT:         .vbyte  4, L..overredzone0-.overredzone # Function size
+  ; CHECK-NEXT:         .vbyte  2, 0x000b                       # Function name len = 11
+  ; CHECK-NEXT:         .byte   "overredzone"                   # Function Name
+...
diff --git a/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable-vectorinfo.ll b/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable-vectorinfo.ll
index 6410cf37ac576..83e413ab408bb 100644
--- a/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable-vectorinfo.ll
+++ b/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable-vectorinfo.ll
@@ -89,7 +89,7 @@ declare <4 x float> @llvm.fabs.v4f32(<4 x float>) #1
 ; COMMON-NEXT:                                         # -IsFloatingPointOperationLogOrAbortEnabled
 ; COMMON-NEXT:  .byte   0x40                            # -IsInterruptHandler, +IsFunctionNamePresent, -IsAllocaUsed
 ; COMMON-NEXT:                                         # OnConditionDirective = 0, -IsCRSaved, -IsLRSaved
-; COMMON-NEXT:  .byte   0x80                            # +IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
+; COMMON-NEXT:  .byte   0x00                            # -IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
 ; COMMON-NEXT:  .byte   0x40                            # -HasExtensionTable, +HasVectorInfo, NumOfGPRsSaved = 0
 ; COMMON-NEXT:  .byte   0x02                            # NumberOfFixedParms = 2
 ; COMMON-NEXT:  .byte   0x05                            # NumberOfFPParms = 2, +HasParmsOnStack
@@ -114,7 +114,7 @@ declare <4 x float> @llvm.fabs.v4f32(<4 x float>) #1
 ; COMMON-NEXT:                                         # -IsFloatingPointOperationLogOrAbortEnabled
 ; COMMON-NEXT:  .byte   0x40                            # -IsInterruptHandler, +IsFunctionNamePresent, -IsAllocaUsed
 ; COMMON-NEXT:                                         # OnConditionDirective = 0, -IsCRSaved, -IsLRSaved
-; COMMON-NEXT:  .byte   0x80                            # +IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
+; COMMON-NEXT:  .byte   0x00                            # -IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
 ; COMMON-NEXT:  .byte   0x40                            # -HasExtensionTable, +HasVectorInfo, NumOfGPRsSaved = 0
 ; COMMON-NEXT:  .byte   0x01                            # NumberOfFixedParms = 1
 ; COMMON-NEXT:  .byte   0x01                            # NumberOfFPParms = 0, +HasParmsOnStack
diff --git a/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable-vectorinfo_hasvarg.ll b/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable-vectorinfo_hasvarg.ll
index 4a2ad89a423ae..8c0a58929742c 100644
--- a/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable-vectorinfo_hasvarg.ll
+++ b/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable-vectorinfo_hasvarg.ll
@@ -22,7 +22,7 @@ entry:
 ;CHECK-ASM-NEXT:                                         # -IsFloatingPointOperationLogOrAbortEnabled
 ;CHECK-ASM-NEXT:        .byte   0x40                            # -IsInterruptHandler, +IsFunctionNamePresent, -IsAllocaUsed
 ;CHECK-ASM-NEXT:                                         # OnConditionDirective = 0, -IsCRSaved, -IsLRSaved
-;CHECK-ASM-NEXT:        .byte   0x80                            # +IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
+;CHECK-ASM-NEXT:        .byte   0x00                            # -IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
 ;CHECK-ASM-NEXT:        .byte   0x40                            # -HasExtensionTable, +HasVectorInfo, NumOfGPRsSaved = 0
 ;CHECK-ASM-NEXT:        .byte   0x00                            # NumberOfFixedParms = 0
 ;CHECK-ASM-NEXT:        .byte   0x01                            # NumberOfFPParms = 0, +HasParmsOnStack
diff --git a/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable.ll b/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable.ll
index cbcd51f9dd9b4..ce97f37ad2b10 100644
--- a/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable.ll
+++ b/llvm/test/CodeGen/PowerPC/aix-emit-tracebacktable.ll
@@ -145,7 +145,7 @@ entry:
 ; COMMON-NEXT:                                        # -IsFloatingPointOperationLogOrAbortEnabled
 ; COMMON-NEXT:  .byte   0x40                            # -IsInterruptHandler, +IsFunctionNamePresent, -IsAllocaUsed
 ; COMMON-NEXT:                                        # OnConditionDirective = 0, -IsCRSaved, -IsLRSaved
-; COMMON-NEXT:  .byte   0x80                            # +IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
+; COMMON-NEXT:  .byte   0x00                            # -IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
 ; COMMON-NEXT:  .byte   0x00                            # -HasExtensionTable, -HasVectorInfo, NumOfGPRsSaved = 0
 ; COMMON-NEXT:  .byte   0x05                            # NumberOfFixedParms = 5
 ; COMMON-NEXT:  .byte   0x05                            # NumberOfFPParms = 2, +HasParmsOnStack
@@ -197,7 +197,7 @@ entry:
 ; COMMON-NEXT:                                        # -IsFloatingPointOperationLogOrAbortEnabled
 ; COMMON-NEXT:  .byte   0x40                            # -IsInterruptHandler, +IsFunctionNamePresent, -IsAllocaUsed
 ; COMMON-NEXT:                                        # OnConditionDirective = 0, -IsCRSaved, -IsLRSaved
-; COMMON-NEXT:  .byte   0x80                            # +IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
+; COMMON-NEXT:  .byte   0x00                            # -IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
 ; COMMON-NEXT:  .byte   0x00                            # -HasExtensionTable, -HasVectorInfo, NumOfGPRsSaved = 0
 ; COMMON-NEXT:  .byte   0x05                            # NumberOfFixedParms = 5
 ; COMMON-NEXT:  .byte   0x05                            # NumberOfFPParms = 2, +HasParmsOnStack
@@ -224,7 +224,7 @@ entry:
 ; COMMON-NEXT:                                        # -IsFloatingPointOperationLogOrAbortEnabled
 ; COMMON-NEXT:  .byte   0x40                            # -IsInterruptHandler, +IsFunctionNamePresent, -IsAllocaUsed
 ; COMMON-NEXT:                                        # OnConditionDirective = 0, -IsCRSaved, -IsLRSaved
-; COMMON-NEXT:  .byte   0x80                            # +IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
+; COMMON-NEXT:  .byte   0x00                            # -IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
 ; COMMON-NEXT:  .byte   0x00                            # -HasExtensionTable, -HasVectorInfo, NumOfGPRsSaved = 0
 ; COMMON-NEXT:  .byte   0x07                            # NumberOfFixedParms = 7
 ; COMMON-NEXT:  .byte   0x1b                            # NumberOfFPParms = 13, +HasParmsOnStack
diff --git a/llvm/test/CodeGen/PowerPC/byval-lhs.ll b/llvm/test/CodeGen/PowerPC/byval-lhs.ll
index 80de18133dd4d..aef374a5edbb7 100644
--- a/llvm/test/CodeGen/PowerPC/byval-lhs.ll
+++ b/llvm/test/CodeGen/PowerPC/byval-lhs.ll
@@ -17,7 +17,6 @@ define void @bar1(i64 %a) nounwind {
 ; LE-NEXT:    stdu r1, -48(r1)
 ; LE-NEXT:    std r0, 64(r1)
 ; LE-NEXT:    std r3, 40(r1)
-; LE-NEXT:    ld r3, 40(r1)
 ; LE-NEXT:    bl f0
 ; LE-NEXT:    nop
 ; LE-NEXT:    addi r1, r1, 48
@@ -31,7 +30,6 @@ define void @bar1(i64 %a) nounwind {
 ; AIX-NEXT:    stdu r1, -128(r1)
 ; AIX-NEXT:    std r0, 144(r1)
 ; AIX-NEXT:    std r3, 120(r1)
-; AIX-NEXT:    ld r3, 120(r1)
 ; AIX-NEXT:    bl .f0[PR]
 ; AIX-NEXT:    nop
 ; AIX-NEXT:    addi r1, r1, 128
@@ -49,11 +47,10 @@ define void @bar2(i64 %a) nounwind {
 ; LE:       # %bb.0:
 ; LE-NEXT:    mflr r0
 ; LE-NEXT:    stdu r1, -48(r1)
+; LE-NEXT:    mr r4, r3
 ; LE-NEXT:    std r0, 64(r1)
 ; LE-NEXT:    std r3, 32(r1)
 ; LE-NEXT:    std r3, 40(r1)
-; LE-NEXT:    ld r4, 40(r1)
-; LE-NEXT:    ld r3, 32(r1)
 ; LE-NEXT:    bl f1
 ; LE-NEXT:    nop
 ; LE-NEXT:    addi r1, r1, 48
@@ -65,11 +62,10 @@ define void @bar2(i64 %a) nounwind {
 ; AIX:       # %bb.0:
 ; AIX-NEXT:    mflr r0
 ; AIX-NEXT:    stdu r1, -128(r1)
+; AIX-NEXT:    mr r4, r3
 ; AIX-NEXT:    std r0, 144(r1)
 ; AIX-NEXT:    std r3, 112(r1)
 ; AIX-NEXT:    std r3, 120(r1)
-; AIX-NEXT:    ld r4, 120(r1)
-; AIX-NEXT:    ld r3, 112(r1)
 ; AIX-NEXT:    bl .f1[PR]
 ; AIX-NEXT:    nop
 ; AIX-NEXT:    addi r1, r1, 128
diff --git a/llvm/test/CodeGen/PowerPC/ext-bool-trunc-repl.ll b/llvm/test/CodeGen/PowerPC/ext-bool-trunc-repl.ll
index b12414e159a7e..a64354222fa75 100644
--- a/llvm/test/CodeGen/PowerPC/ext-bool-trunc-repl.ll
+++ b/llvm/test/CodeGen/PowerPC/ext-bool-trunc-repl.ll
@@ -11,26 +11,35 @@ define void @fn2() #0 {
 
   br i1 undef, label %1, label %10
 
-; <label>:1:                                      ; preds = %0
+1:                                                ; preds = %0
   br i1 undef, label %3, label %2
 
-; <label>:2:                                      ; preds = %2, %1
+2:                                                ; preds = %2, %1
   br i1 undef, label %3, label %2
 
-; <label>:3:                                      ; preds = %2, %1
+3:                                                ; preds = %2, %1
   br i1 undef, label %8, label %4
 
-; <label>:4:                                      ; preds = %4, %3
+4:                                                ; preds = %4, %3
   %5 = phi i64 [ %6, %4 ], [ undef, %3 ]
-  %6 = and i64 %5, and (i64 and (i64 and (i64 and (i64 and (i64 and (i64 and (i64 sext (i32 select (i1 icmp slt (i16 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i16), i16 0), i32 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i32), i32 lshr (i32 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i32), i32 6)) to i64), i64 sext (i32 select (i1 icmp slt (i16 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i16), i16 0), i32 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i32), i32 lshr (i32 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i32), i32 6)) to i64)), i64 sext (i32 select (i1 icmp slt (i16 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i16), i16 0), i32 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i32), i32 lshr (i32 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i32), i32 6)) to i64)), i64 sext (i32 select (i1 icmp slt (i16 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i16), i16 0), i32 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i32), i32 lshr (i32 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i32), i32 6)) to i64)), i64 sext (i32 select (i1 icmp slt (i16 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i16), i16 0), i32 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i32), i32 lshr (i32 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i32), i32 6)) to i64)), i64 sext (i32 select (i1 icmp slt (i16 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i16), i16 0), i32 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i32), i32 lshr (i32 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i32), i32 6)) to i64)), i64 sext (i32 select (i1 icmp slt (i16 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i16), i16 0), i32 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i32), i32 lshr (i32 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i32), i32 6)) to i64)), i64 sext (i32 select (i1 icmp slt (i16 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i16), i16 0), i32 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i32), i32 lshr (i32 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i32), i32 6)) to i64))
+  %constexpr = select i1 icmp slt (i16 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i16), i16 0), i32 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i32), i32 lshr (i32 zext (i1 icmp eq (ptr getelementptr inbounds ([2 x i32], ptr @d, i64 0, i64 1), ptr @c) to i32), i32 6)
+  %constexpr1 = sext i32 %constexpr to i64
+  %constexpr2 = and i64 %constexpr1, %constexpr1
+  %constexpr3 = and i64 %constexpr2, %constexpr1
+  %constexpr4 = and i64 %constexpr3, %constexpr1
+  %constexpr5 = and i64 %constexpr4, %constexpr1
+  %constexpr6 = and i64 %constexpr5, %constexpr1
+  %constexpr7 = and i64 %constexpr6, %constexpr1
+  %constexpr8 = and i64 %constexpr7, %constexpr1
+  %6 = and i64 %5, %constexpr8
   %7 = icmp slt i32 undef, 6
   br i1 %7, label %4, label %8
 
-; <label>:8:                                      ; preds = %4, %3
+8:                                                ; preds = %4, %3
   %9 = phi i64 [ undef, %3 ], [ %6, %4 ]
   br label %10
 
-; <label>:10:                                     ; preds = %8, %0
+10:                                               ; preds = %8, %0
   ret void
 }
 
diff --git a/llvm/test/CodeGen/PowerPC/loop-instr-form-non-inc.ll b/llvm/test/CodeGen/PowerPC/loop-instr-form-non-inc.ll
new file mode 100644
index 0000000000000..93697ec06be2c
--- /dev/null
+++ b/llvm/test/CodeGen/PowerPC/loop-instr-form-non-inc.ll
@@ -0,0 +1,39 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -verify-machineinstrs -mtriple=powerpc-unknown-linux-gnu \
+; RUN:   -mcpu=pwr9 < %s | FileCheck %s
+
+define dso_local void @test_no_inc(i32 signext %a) local_unnamed_addr nounwind align 2 {
+; CHECK-LABEL: test_no_inc:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    srawi 4, 3, 31
+; CHECK-NEXT:    cmpwi 3, 0
+; CHECK-NEXT:    li 6, 1
+; CHECK-NEXT:    li 7, 0
+; CHECK-NEXT:    andc 4, 3, 4
+; CHECK-NEXT:    addi 5, 4, 1
+; CHECK-NEXT:    .p2align 5
+; CHECK-NEXT:  .LBB0_1: # %for.cond
+; CHECK-NEXT:    #
+; CHECK-NEXT:    add 8, 3, 6
+; CHECK-NEXT:    stb 7, 0(5)
+; CHECK-NEXT:    add 5, 5, 4
+; CHECK-NEXT:    iselgt 6, 8, 6
+; CHECK-NEXT:    b .LBB0_1
+entry:
+  %cmp10 = icmp sgt i32 %a, 0
+  br label %for.cond
+
+for.cond:                                         ; preds = %for.cond.cleanup, %entry
+  %g.0 = phi i32 [ 1, %entry ], [ %g.1.lcssa, %for.cond.cleanup ]
+  br i1 %cmp10, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:                               ; preds = %for.cond
+  %0 = add i32 %a, %g.0
+  br label %for.cond.cleanup
+
+for.cond.cleanup:                                 ; preds = %for.body.preheader, %for.cond
+  %g.1.lcssa = phi i32 [ %g.0, %for.cond ], [ %0, %for.body.preheader ]
+  %arrayidx5 = getelementptr inbounds i8, ptr null, i32 %g.1.lcssa
+  store i8 0, ptr %arrayidx5, align 1
+  br label %for.cond
+}
diff --git a/llvm/test/CodeGen/PowerPC/lsr-postinc-pos.ll b/llvm/test/CodeGen/PowerPC/lsr-postinc-pos.ll
index f9ef6f6998fd6..83763f5ef76aa 100644
--- a/llvm/test/CodeGen/PowerPC/lsr-postinc-pos.ll
+++ b/llvm/test/CodeGen/PowerPC/lsr-postinc-pos.ll
@@ -4,8 +4,8 @@
 ; scevgep needs to be inserted in %bb so that it is dominated by %t.
 
 ; CHECK: %t = load ptr, ptr %inp
-; CHECK: %uglygep = getelementptr i8, ptr %t, i32 %lsr.iv.next
-; CHECK: %c1 = icmp ult ptr %uglygep, %inp2
+; CHECK: %scevgep = getelementptr i8, ptr %t, i32 %lsr.iv.next
+; CHECK: %c1 = icmp ult ptr %scevgep, %inp2
 
 target datalayout = "E-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:32:64-v64:64:64-v128:128:128-a0:0:64-f128:64:128-n32"
 target triple = "powerpc-unknown-linux-gnu"
diff --git a/llvm/test/CodeGen/PowerPC/ppc64-P9-vabsd.ll b/llvm/test/CodeGen/PowerPC/ppc64-P9-vabsd.ll
index 9e9271ed7c5d7..342a9044b9bcc 100644
--- a/llvm/test/CodeGen/PowerPC/ppc64-P9-vabsd.ll
+++ b/llvm/test/CodeGen/PowerPC/ppc64-P9-vabsd.ll
@@ -1834,13 +1834,20 @@ define <16 x i8> @absd_int8_ule(<16 x i8>, <16 x i8>) {
 ; Tests for ABDS icmp + sub + select sequence
 
 define <4 x i32> @absd_int32_sgt(<4 x i32>, <4 x i32>) {
-; CHECK-LABEL: absd_int32_sgt:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vcmpgtsw v4, v2, v3
-; CHECK-NEXT:    vsubuwm v5, v2, v3
-; CHECK-NEXT:    vsubuwm v2, v3, v2
-; CHECK-NEXT:    xxsel v2, v2, v5, v4
-; CHECK-NEXT:    blr
+; CHECK-PWR9-LABEL: absd_int32_sgt:
+; CHECK-PWR9:       # %bb.0:
+; CHECK-PWR9-NEXT:    xvnegsp v3, v3
+; CHECK-PWR9-NEXT:    xvnegsp v2, v2
+; CHECK-PWR9-NEXT:    vabsduw v2, v2, v3
+; CHECK-PWR9-NEXT:    blr
+;
+; CHECK-PWR78-LABEL: absd_int32_sgt:
+; CHECK-PWR78:       # %bb.0:
+; CHECK-PWR78-NEXT:    vcmpgtsw v4, v2, v3
+; CHECK-PWR78-NEXT:    vsubuwm v5, v2, v3
+; CHECK-PWR78-NEXT:    vsubuwm v2, v3, v2
+; CHECK-PWR78-NEXT:    xxsel v2, v2, v5, v4
+; CHECK-PWR78-NEXT:    blr
   %3 = icmp sgt <4 x i32> %0, %1
   %4 = sub <4 x i32> %0, %1
   %5 = sub <4 x i32> %1, %0
@@ -1849,14 +1856,21 @@ define <4 x i32> @absd_int32_sgt(<4 x i32>, <4 x i32>) {
 }
 
 define <4 x i32> @absd_int32_sge(<4 x i32>, <4 x i32>) {
-; CHECK-LABEL: absd_int32_sge:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vcmpgtsw v4, v3, v2
-; CHECK-NEXT:    xxlnor vs0, v4, v4
-; CHECK-NEXT:    vsubuwm v4, v2, v3
-; CHECK-NEXT:    vsubuwm v2, v3, v2
-; CHECK-NEXT:    xxsel v2, v2, v4, vs0
-; CHECK-NEXT:    blr
+; CHECK-PWR9-LABEL: absd_int32_sge:
+; CHECK-PWR9:       # %bb.0:
+; CHECK-PWR9-NEXT:    xvnegsp v3, v3
+; CHECK-PWR9-NEXT:    xvnegsp v2, v2
+; CHECK-PWR9-NEXT:    vabsduw v2, v2, v3
+; CHECK-PWR9-NEXT:    blr
+;
+; CHECK-PWR78-LABEL: absd_int32_sge:
+; CHECK-PWR78:       # %bb.0:
+; CHECK-PWR78-NEXT:    vcmpgtsw v4, v3, v2
+; CHECK-PWR78-NEXT:    xxlnor vs0, v4, v4
+; CHECK-PWR78-NEXT:    vsubuwm v4, v2, v3
+; CHECK-PWR78-NEXT:    vsubuwm v2, v3, v2
+; CHECK-PWR78-NEXT:    xxsel v2, v2, v4, vs0
+; CHECK-PWR78-NEXT:    blr
   %3 = icmp sge <4 x i32> %0, %1
   %4 = sub <4 x i32> %0, %1
   %5 = sub <4 x i32> %1, %0
@@ -1865,13 +1879,20 @@ define <4 x i32> @absd_int32_sge(<4 x i32>, <4 x i32>) {
 }
 
 define <4 x i32> @absd_int32_slt(<4 x i32>, <4 x i32>) {
-; CHECK-LABEL: absd_int32_slt:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vcmpgtsw v4, v3, v2
-; CHECK-NEXT:    vsubuwm v5, v2, v3
-; CHECK-NEXT:    vsubuwm v2, v3, v2
-; CHECK-NEXT:    xxsel v2, v5, v2, v4
-; CHECK-NEXT:    blr
+; CHECK-PWR9-LABEL: absd_int32_slt:
+; CHECK-PWR9:       # %bb.0:
+; CHECK-PWR9-NEXT:    xvnegsp v3, v3
+; CHECK-PWR9-NEXT:    xvnegsp v2, v2
+; CHECK-PWR9-NEXT:    vabsduw v2, v2, v3
+; CHECK-PWR9-NEXT:    blr
+;
+; CHECK-PWR78-LABEL: absd_int32_slt:
+; CHECK-PWR78:       # %bb.0:
+; CHECK-PWR78-NEXT:    vcmpgtsw v4, v3, v2
+; CHECK-PWR78-NEXT:    vsubuwm v5, v2, v3
+; CHECK-PWR78-NEXT:    vsubuwm v2, v3, v2
+; CHECK-PWR78-NEXT:    xxsel v2, v5, v2, v4
+; CHECK-PWR78-NEXT:    blr
   %3 = icmp slt <4 x i32> %0, %1
   %4 = sub <4 x i32> %0, %1
   %5 = sub <4 x i32> %1, %0
@@ -1880,14 +1901,21 @@ define <4 x i32> @absd_int32_slt(<4 x i32>, <4 x i32>) {
 }
 
 define <4 x i32> @absd_int32_sle(<4 x i32>, <4 x i32>) {
-; CHECK-LABEL: absd_int32_sle:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vcmpgtsw v4, v2, v3
-; CHECK-NEXT:    xxlnor vs0, v4, v4
-; CHECK-NEXT:    vsubuwm v4, v2, v3
-; CHECK-NEXT:    vsubuwm v2, v3, v2
-; CHECK-NEXT:    xxsel v2, v4, v2, vs0
-; CHECK-NEXT:    blr
+; CHECK-PWR9-LABEL: absd_int32_sle:
+; CHECK-PWR9:       # %bb.0:
+; CHECK-PWR9-NEXT:    xvnegsp v3, v3
+; CHECK-PWR9-NEXT:    xvnegsp v2, v2
+; CHECK-PWR9-NEXT:    vabsduw v2, v2, v3
+; CHECK-PWR9-NEXT:    blr
+;
+; CHECK-PWR78-LABEL: absd_int32_sle:
+; CHECK-PWR78:       # %bb.0:
+; CHECK-PWR78-NEXT:    vcmpgtsw v4, v2, v3
+; CHECK-PWR78-NEXT:    xxlnor vs0, v4, v4
+; CHECK-PWR78-NEXT:    vsubuwm v4, v2, v3
+; CHECK-PWR78-NEXT:    vsubuwm v2, v3, v2
+; CHECK-PWR78-NEXT:    xxsel v2, v4, v2, vs0
+; CHECK-PWR78-NEXT:    blr
   %3 = icmp sle <4 x i32> %0, %1
   %4 = sub <4 x i32> %0, %1
   %5 = sub <4 x i32> %1, %0
diff --git a/llvm/test/CodeGen/PowerPC/ppc64-byval-larger-struct.ll b/llvm/test/CodeGen/PowerPC/ppc64-byval-larger-struct.ll
index 429b8774f6ec3..39b06619063df 100644
--- a/llvm/test/CodeGen/PowerPC/ppc64-byval-larger-struct.ll
+++ b/llvm/test/CodeGen/PowerPC/ppc64-byval-larger-struct.ll
@@ -184,7 +184,6 @@ define signext i8 @caller_9_callee_9(ptr nocapture readonly byval([9 x i8]) %dat
 ; P8LE-NEXT:    stdx r3, 0, r5
 ; P8LE-NEXT:    stb r4, 79(r1)
 ; P8LE-NEXT:    lbz r4, 56(r1)
-; P8LE-NEXT:    ld r3, 48(r1)
 ; P8LE-NEXT:    bl callee_9
 ; P8LE-NEXT:    nop
 ; P8LE-NEXT:    li r3, 0
@@ -204,7 +203,6 @@ define signext i8 @caller_9_callee_9(ptr nocapture readonly byval([9 x i8]) %dat
 ; P9LE-NEXT:    std r3, 48(r1)
 ; P9LE-NEXT:    stdx r3, 0, r4
 ; P9LE-NEXT:    lbz r4, 56(r1)
-; P9LE-NEXT:    ld r3, 48(r1)
 ; P9LE-NEXT:    stb r5, 79(r1)
 ; P9LE-NEXT:    bl callee_9
 ; P9LE-NEXT:    nop
@@ -225,7 +223,6 @@ define signext i8 @caller_9_callee_9(ptr nocapture readonly byval([9 x i8]) %dat
 ; P10LE-NEXT:    lbz r5, 56(r1)
 ; P10LE-NEXT:    stdx r3, 0, r4
 ; P10LE-NEXT:    lbz r4, 56(r1)
-; P10LE-NEXT:    ld r3, 48(r1)
 ; P10LE-NEXT:    stb r5, 79(r1)
 ; P10LE-NEXT:    bl callee_9@notoc
 ; P10LE-NEXT:    li r3, 0
@@ -246,7 +243,6 @@ define signext i8 @caller_9_callee_9(ptr nocapture readonly byval([9 x i8]) %dat
 ; P8BE-NEXT:    stdx r3, 0, r5
 ; P8BE-NEXT:    stb r4, 143(r1)
 ; P8BE-NEXT:    lbz r4, 200(r1)
-; P8BE-NEXT:    ld r3, 192(r1)
 ; P8BE-NEXT:    bl callee_9
 ; P8BE-NEXT:    nop
 ; P8BE-NEXT:    li r3, 0
@@ -266,7 +262,6 @@ define signext i8 @caller_9_callee_9(ptr nocapture readonly byval([9 x i8]) %dat
 ; P9BE-NEXT:    std r3, 192(r1)
 ; P9BE-NEXT:    stdx r3, 0, r4
 ; P9BE-NEXT:    lbz r4, 200(r1)
-; P9BE-NEXT:    ld r3, 192(r1)
 ; P9BE-NEXT:    stb r5, 143(r1)
 ; P9BE-NEXT:    bl callee_9
 ; P9BE-NEXT:    nop
@@ -287,7 +282,6 @@ define signext i8 @caller_9_callee_9(ptr nocapture readonly byval([9 x i8]) %dat
 ; P10BE-NEXT:    lbz r5, 200(r1)
 ; P10BE-NEXT:    stdx r3, 0, r4
 ; P10BE-NEXT:    lbz r4, 200(r1)
-; P10BE-NEXT:    ld r3, 192(r1)
 ; P10BE-NEXT:    stb r5, 143(r1)
 ; P10BE-NEXT:    bl callee_9
 ; P10BE-NEXT:    nop
diff --git a/llvm/test/CodeGen/PowerPC/pr24636.ll b/llvm/test/CodeGen/PowerPC/pr24636.ll
index d423e6a053f00..bb13a7aa6c2d6 100644
--- a/llvm/test/CodeGen/PowerPC/pr24636.ll
+++ b/llvm/test/CodeGen/PowerPC/pr24636.ll
@@ -2,7 +2,7 @@
 target datalayout = "e-m:e-i64:64-n32:64"
 target triple = "powerpc64le-unknown-linux-gnu"
 
-@c = external global i32, align 4
+@c = external unnamed_addr global i32, align 4
 @b = external global [1 x i32], align 4
 
 ; Function Attrs: nounwind
@@ -25,7 +25,16 @@ define void @fn2() #0 align 4 {
 
 .lr.ph.split.split:                               ; preds = %.lr.ph.split.split, %.lr.ph.split
   %1 = phi i32 [ %2, %.lr.ph.split.split ], [ undef, %.lr.ph.split ]
-  %2 = and i32 %1, and (i32 and (i32 and (i32 and (i32 and (i32 and (i32 and (i32 zext (i1 select (i1 icmp eq (ptr @c, ptr @b), i1 true, i1 false) to i32), i32 zext (i1 select (i1 icmp eq (ptr @c, ptr @b), i1 true, i1 false) to i32)), i32 zext (i1 select (i1 icmp eq (ptr @c, ptr @b), i1 true, i1 false) to i32)), i32 zext (i1 select (i1 icmp eq (ptr @c, ptr @b), i1 true, i1 false) to i32)), i32 zext (i1 select (i1 icmp eq (ptr @c, ptr @b), i1 true, i1 false) to i32)), i32 zext (i1 select (i1 icmp eq (ptr @c, ptr @b), i1 true, i1 false) to i32)), i32 zext (i1 select (i1 icmp eq (ptr @c, ptr @b), i1 true, i1 false) to i32)), i32 zext (i1 select (i1 icmp eq (ptr @c, ptr @b), i1 true, i1 false) to i32))
+  %constexpr = select i1 icmp eq (ptr @c, ptr @b), i1 true, i1 false
+  %constexpr1 = zext i1 %constexpr to i32
+  %constexpr2 = and i32 %constexpr1, %constexpr1
+  %constexpr3 = and i32 %constexpr2, %constexpr1
+  %constexpr4 = and i32 %constexpr3, %constexpr1
+  %constexpr5 = and i32 %constexpr4, %constexpr1
+  %constexpr6 = and i32 %constexpr5, %constexpr1
+  %constexpr7 = and i32 %constexpr6, %constexpr1
+  %constexpr8 = and i32 %constexpr7, %constexpr1
+  %2 = and i32 %1, %constexpr8
   %3 = icmp slt i32 undef, 4
   br i1 %3, label %.lr.ph.split.split, label %._crit_edge
 
@@ -33,7 +42,7 @@ define void @fn2() #0 align 4 {
   %.lcssa = phi i32 [ undef, %.lr.ph.split ], [ %2, %.lr.ph.split.split ]
   br label %4
 
-; <label>:4                                       ; preds = %._crit_edge, %0
+4:                                                ; preds = %._crit_edge, %0
   ret void
 }
 
diff --git a/llvm/test/CodeGen/PowerPC/sms-cpy-1.ll b/llvm/test/CodeGen/PowerPC/sms-cpy-1.ll
index 5b8742b6f61af..33dfd2268bd2b 100644
--- a/llvm/test/CodeGen/PowerPC/sms-cpy-1.ll
+++ b/llvm/test/CodeGen/PowerPC/sms-cpy-1.ll
@@ -13,18 +13,16 @@ define void @print_res() nounwind {
 ; CHECK-NEXT:    lwz 3, 0(3)
 ; CHECK-NEXT:    addi 3, 3, -1
 ; CHECK-NEXT:    clrldi 4, 3, 32
-; CHECK-NEXT:    cmplwi 3, 1
-; CHECK-NEXT:    li 3, 1
-; CHECK-NEXT:    iselgt 3, 4, 3
-; CHECK-NEXT:    li 4, 2
-; CHECK-NEXT:    addi 3, 3, -1
-; CHECK-NEXT:    cmpldi 3, 2
-; CHECK-NEXT:    isellt 3, 3, 4
+; CHECK-NEXT:    cmplwi 3, 3
+; CHECK-NEXT:    li 3, 3
+; CHECK-NEXT:    isellt 3, 4, 3
+; CHECK-NEXT:    li 4, 1
+; CHECK-NEXT:    cmpldi 3, 1
+; CHECK-NEXT:    iselgt 3, 3, 4
 ; CHECK-NEXT:    li 4, 0
-; CHECK-NEXT:    addi 3, 3, 1
 ; CHECK-NEXT:    li 5, 0
-; CHECK-NEXT:    li 7, -1
 ; CHECK-NEXT:    mtctr 3
+; CHECK-NEXT:    li 7, -1
 ; CHECK-NEXT:    lbz 5, 0(5)
 ; CHECK-NEXT:    li 3, 1
 ; CHECK-NEXT:    bdz .LBB0_6
diff --git a/llvm/test/CodeGen/PowerPC/vector-sum-sat-bit-side-effect.ll b/llvm/test/CodeGen/PowerPC/vector-sum-sat-bit-side-effect.ll
new file mode 100644
index 0000000000000..e371d7c70f3e3
--- /dev/null
+++ b/llvm/test/CodeGen/PowerPC/vector-sum-sat-bit-side-effect.ll
@@ -0,0 +1,140 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu \
+; RUN:   -mcpu=pwr8 -ppc-asm-full-reg-names -ppc-vsr-nums-as-vr < %s | FileCheck %s
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64-aix-xcoff \
+; RUN:   -mcpu=pwr8 -ppc-asm-full-reg-names -ppc-vsr-nums-as-vr < %s | FileCheck %s
+; RUN: llc -verify-machineinstrs -mtriple=powerpc-aix-xcoff \
+; RUN:   -mcpu=pwr8 -ppc-asm-full-reg-names -ppc-vsr-nums-as-vr < %s | FileCheck %s
+
+define void @test1(<16 x i8> %0) {
+; CHECK-LABEL: test1:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    blr
+entry:
+  %1 = tail call <4 x i32> @llvm.ppc.altivec.vsum4sbs(<16 x i8> %0, <4 x i32> zeroinitializer)
+  ret void
+}
+
+define void @test2(<8 x i16> %0) {
+; CHECK-LABEL: test2:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    blr
+entry:
+  %1 = tail call <4 x i32> @llvm.ppc.altivec.vsum4shs(<8 x i16> %0, <4 x i32> zeroinitializer)
+  ret void
+}
+
+define void @test3(<16 x i8> %0) {
+; CHECK-LABEL: test3:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    blr
+entry:
+  %1 = tail call <4 x i32> @llvm.ppc.altivec.vsum4ubs(<16 x i8> %0, <4 x i32> zeroinitializer)
+  ret void
+}
+
+define void @test4(<16 x i8> %0) {
+; CHECK-LABEL: test4:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vspltisw v3, 1
+; CHECK-NEXT:    vsum4sbs v2, v2, v3
+; CHECK-NEXT:    blr
+entry:
+  %1 = tail call <4 x i32> @llvm.ppc.altivec.vsum4sbs(<16 x i8> %0, <4 x i32> <i32 1, i32 1, i32 1, i32 1>)
+  ret void
+}
+
+define void @test5(<8 x i16> %0) {
+; CHECK-LABEL: test5:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vspltisw v3, 1
+; CHECK-NEXT:    vsum4shs v2, v2, v3
+; CHECK-NEXT:    blr
+entry:
+  %1 = tail call <4 x i32> @llvm.ppc.altivec.vsum4shs(<8 x i16> %0, <4 x i32> <i32 1, i32 1, i32 1, i32 1>)
+  ret void
+}
+
+define void @test6(<16 x i8> %0) {
+; CHECK-LABEL: test6:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vspltisw v3, 1
+; CHECK-NEXT:    vsum4ubs v2, v2, v3
+; CHECK-NEXT:    blr
+entry:
+  %1 = tail call <4 x i32> @llvm.ppc.altivec.vsum4ubs(<16 x i8> %0, <4 x i32> <i32 1, i32 1, i32 1, i32 1>)
+  ret void
+}
+
+define <4 x i32> @test7(<16 x i8> %0) {
+; CHECK-LABEL: test7:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    xxlxor v3, v3, v3
+; CHECK-NEXT:    vsum4sbs v2, v2, v3
+; CHECK-NEXT:    blr
+entry:
+  %1 = tail call <4 x i32> @llvm.ppc.altivec.vsum4sbs(<16 x i8> %0, <4 x i32> zeroinitializer)
+  ret <4 x i32> %1
+}
+
+define <4 x i32> @test8(<8 x i16> %0) {
+; CHECK-LABEL: test8:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    xxlxor v3, v3, v3
+; CHECK-NEXT:    vsum4shs v2, v2, v3
+; CHECK-NEXT:    blr
+entry:
+  %1 = tail call <4 x i32> @llvm.ppc.altivec.vsum4shs(<8 x i16> %0, <4 x i32> zeroinitializer)
+  ret <4 x i32> %1
+}
+
+define <4 x i32> @test9(<16 x i8> %0) {
+; CHECK-LABEL: test9:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    xxlxor v3, v3, v3
+; CHECK-NEXT:    vsum4ubs v2, v2, v3
+; CHECK-NEXT:    blr
+entry:
+  %1 = tail call <4 x i32> @llvm.ppc.altivec.vsum4ubs(<16 x i8> %0, <4 x i32> zeroinitializer)
+  ret <4 x i32> %1
+}
+
+define <4 x i32> @test10(<16 x i8> %0, <16 x i8> %1) {
+; CHECK-LABEL: test10:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    xxlxor v3, v3, v3
+; CHECK-NEXT:    vsum4sbs v2, v2, v3
+; CHECK-NEXT:    blr
+entry:
+  %2 = tail call <4 x i32> @llvm.ppc.altivec.vsum4sbs(<16 x i8> %0, <4 x i32> zeroinitializer)
+  %3 = tail call <4 x i32> @llvm.ppc.altivec.vsum4sbs(<16 x i8> %1, <4 x i32> zeroinitializer)
+  ret <4 x i32> %2
+}
+
+define <4 x i32> @test11(<8 x i16> %0, <8 x i16> %1) {
+; CHECK-LABEL: test11:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    xxlxor v3, v3, v3
+; CHECK-NEXT:    vsum4shs v2, v2, v3
+; CHECK-NEXT:    blr
+entry:
+  %2 = tail call <4 x i32> @llvm.ppc.altivec.vsum4shs(<8 x i16> %0, <4 x i32> zeroinitializer)
+  %3 = tail call <4 x i32> @llvm.ppc.altivec.vsum4shs(<8 x i16> %1, <4 x i32> zeroinitializer)
+  ret <4 x i32> %2
+}
+
+define <4 x i32> @test12(<16 x i8> %0, <16 x i8> %1) {
+; CHECK-LABEL: test12:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    xxlxor v3, v3, v3
+; CHECK-NEXT:    vsum4ubs v2, v2, v3
+; CHECK-NEXT:    blr
+entry:
+  %2 = tail call <4 x i32> @llvm.ppc.altivec.vsum4ubs(<16 x i8> %0, <4 x i32> zeroinitializer)
+  %3 = tail call <4 x i32> @llvm.ppc.altivec.vsum4ubs(<16 x i8> %1, <4 x i32> zeroinitializer)
+  ret <4 x i32> %2
+}
+
+declare <4 x i32> @llvm.ppc.altivec.vsum4sbs(<16 x i8>, <4 x i32>)
+declare <4 x i32> @llvm.ppc.altivec.vsum4shs(<8 x i16>, <4 x i32>)
+declare <4 x i32> @llvm.ppc.altivec.vsum4ubs(<16 x i8>, <4 x i32>)
diff --git a/llvm/test/CodeGen/RISCV/O3-pipeline.ll b/llvm/test/CodeGen/RISCV/O3-pipeline.ll
index 3cc231bd1fcf6..0ad08074c104e 100644
--- a/llvm/test/CodeGen/RISCV/O3-pipeline.ll
+++ b/llvm/test/CodeGen/RISCV/O3-pipeline.ll
@@ -166,6 +166,7 @@
 ; CHECK-NEXT:       Implement the 'patchable-function' attribute
 ; CHECK-NEXT:       Branch relaxation pass
 ; CHECK-NEXT:       RISCV Make Compressible
+; CHECK-NEXT:       Machine Copy Propagation Pass
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
diff --git a/llvm/test/CodeGen/RISCV/addimm-mulimm.ll b/llvm/test/CodeGen/RISCV/addimm-mulimm.ll
index 4454af837004a..d1bc480455dd3 100644
--- a/llvm/test/CodeGen/RISCV/addimm-mulimm.ll
+++ b/llvm/test/CodeGen/RISCV/addimm-mulimm.ll
@@ -18,7 +18,7 @@ define i32 @add_mul_combine_accept_a1(i32 %x) {
 ; RV64IMB-LABEL: add_mul_combine_accept_a1:
 ; RV64IMB:       # %bb.0:
 ; RV64IMB-NEXT:    li a1, 29
-; RV64IMB-NEXT:    mulw a0, a0, a1
+; RV64IMB-NEXT:    mul a0, a0, a1
 ; RV64IMB-NEXT:    addiw a0, a0, 1073
 ; RV64IMB-NEXT:    ret
   %tmp0 = add i32 %x, 37
@@ -37,7 +37,7 @@ define signext i32 @add_mul_combine_accept_a2(i32 signext %x) {
 ; RV64IMB-LABEL: add_mul_combine_accept_a2:
 ; RV64IMB:       # %bb.0:
 ; RV64IMB-NEXT:    li a1, 29
-; RV64IMB-NEXT:    mulw a0, a0, a1
+; RV64IMB-NEXT:    mul a0, a0, a1
 ; RV64IMB-NEXT:    addiw a0, a0, 1073
 ; RV64IMB-NEXT:    ret
   %tmp0 = add i32 %x, 37
@@ -82,7 +82,7 @@ define i32 @add_mul_combine_accept_b1(i32 %x) {
 ; RV64IMB-LABEL: add_mul_combine_accept_b1:
 ; RV64IMB:       # %bb.0:
 ; RV64IMB-NEXT:    li a1, 23
-; RV64IMB-NEXT:    mulw a0, a0, a1
+; RV64IMB-NEXT:    mul a0, a0, a1
 ; RV64IMB-NEXT:    lui a1, 50
 ; RV64IMB-NEXT:    addiw a1, a1, 1119
 ; RV64IMB-NEXT:    addw a0, a0, a1
@@ -105,7 +105,7 @@ define signext i32 @add_mul_combine_accept_b2(i32 signext %x) {
 ; RV64IMB-LABEL: add_mul_combine_accept_b2:
 ; RV64IMB:       # %bb.0:
 ; RV64IMB-NEXT:    li a1, 23
-; RV64IMB-NEXT:    mulw a0, a0, a1
+; RV64IMB-NEXT:    mul a0, a0, a1
 ; RV64IMB-NEXT:    lui a1, 50
 ; RV64IMB-NEXT:    addiw a1, a1, 1119
 ; RV64IMB-NEXT:    addw a0, a0, a1
@@ -416,7 +416,7 @@ define i32 @add_mul_combine_reject_f1(i32 %x) {
 ; RV64IMB:       # %bb.0:
 ; RV64IMB-NEXT:    addiw a0, a0, 1972
 ; RV64IMB-NEXT:    li a1, 29
-; RV64IMB-NEXT:    mulw a0, a0, a1
+; RV64IMB-NEXT:    mul a0, a0, a1
 ; RV64IMB-NEXT:    addiw a0, a0, 11
 ; RV64IMB-NEXT:    ret
   %tmp0 = mul i32 %x, 29
@@ -437,7 +437,7 @@ define signext i32 @add_mul_combine_reject_f2(i32 signext %x) {
 ; RV64IMB:       # %bb.0:
 ; RV64IMB-NEXT:    addiw a0, a0, 1972
 ; RV64IMB-NEXT:    li a1, 29
-; RV64IMB-NEXT:    mulw a0, a0, a1
+; RV64IMB-NEXT:    mul a0, a0, a1
 ; RV64IMB-NEXT:    addiw a0, a0, 11
 ; RV64IMB-NEXT:    ret
   %tmp0 = mul i32 %x, 29
@@ -584,7 +584,7 @@ define i32 @mul3000_add8990_a(i32 %x) {
 ; RV64IMB-NEXT:    addiw a0, a0, 3
 ; RV64IMB-NEXT:    lui a1, 1
 ; RV64IMB-NEXT:    addiw a1, a1, -1096
-; RV64IMB-NEXT:    mulw a0, a0, a1
+; RV64IMB-NEXT:    mul a0, a0, a1
 ; RV64IMB-NEXT:    addiw a0, a0, -10
 ; RV64IMB-NEXT:    ret
   %tmp0 = mul i32 %x, 3000
@@ -607,7 +607,7 @@ define signext i32 @mul3000_add8990_b(i32 signext %x) {
 ; RV64IMB-NEXT:    addiw a0, a0, 3
 ; RV64IMB-NEXT:    lui a1, 1
 ; RV64IMB-NEXT:    addiw a1, a1, -1096
-; RV64IMB-NEXT:    mulw a0, a0, a1
+; RV64IMB-NEXT:    mul a0, a0, a1
 ; RV64IMB-NEXT:    addiw a0, a0, -10
 ; RV64IMB-NEXT:    ret
   %tmp0 = mul i32 %x, 3000
@@ -659,7 +659,7 @@ define i32 @mul3000_sub8990_a(i32 %x) {
 ; RV64IMB-NEXT:    addiw a0, a0, -3
 ; RV64IMB-NEXT:    lui a1, 1
 ; RV64IMB-NEXT:    addiw a1, a1, -1096
-; RV64IMB-NEXT:    mulw a0, a0, a1
+; RV64IMB-NEXT:    mul a0, a0, a1
 ; RV64IMB-NEXT:    addiw a0, a0, 10
 ; RV64IMB-NEXT:    ret
   %tmp0 = mul i32 %x, 3000
@@ -682,7 +682,7 @@ define signext i32 @mul3000_sub8990_b(i32 signext %x) {
 ; RV64IMB-NEXT:    addiw a0, a0, -3
 ; RV64IMB-NEXT:    lui a1, 1
 ; RV64IMB-NEXT:    addiw a1, a1, -1096
-; RV64IMB-NEXT:    mulw a0, a0, a1
+; RV64IMB-NEXT:    mul a0, a0, a1
 ; RV64IMB-NEXT:    addiw a0, a0, 10
 ; RV64IMB-NEXT:    ret
   %tmp0 = mul i32 %x, 3000
@@ -735,7 +735,7 @@ define i32 @mulneg3000_add8990_a(i32 %x) {
 ; RV64IMB-NEXT:    addiw a0, a0, -3
 ; RV64IMB-NEXT:    lui a1, 1048575
 ; RV64IMB-NEXT:    addiw a1, a1, 1096
-; RV64IMB-NEXT:    mulw a0, a0, a1
+; RV64IMB-NEXT:    mul a0, a0, a1
 ; RV64IMB-NEXT:    addiw a0, a0, -10
 ; RV64IMB-NEXT:    ret
   %tmp0 = mul i32 %x, -3000
@@ -758,7 +758,7 @@ define signext i32 @mulneg3000_add8990_b(i32 signext %x) {
 ; RV64IMB-NEXT:    addiw a0, a0, -3
 ; RV64IMB-NEXT:    lui a1, 1048575
 ; RV64IMB-NEXT:    addiw a1, a1, 1096
-; RV64IMB-NEXT:    mulw a0, a0, a1
+; RV64IMB-NEXT:    mul a0, a0, a1
 ; RV64IMB-NEXT:    addiw a0, a0, -10
 ; RV64IMB-NEXT:    ret
   %tmp0 = mul i32 %x, -3000
@@ -811,7 +811,7 @@ define i32 @mulneg3000_sub8990_a(i32 %x) {
 ; RV64IMB-NEXT:    addiw a0, a0, 3
 ; RV64IMB-NEXT:    lui a1, 1048575
 ; RV64IMB-NEXT:    addiw a1, a1, 1096
-; RV64IMB-NEXT:    mulw a0, a0, a1
+; RV64IMB-NEXT:    mul a0, a0, a1
 ; RV64IMB-NEXT:    addiw a0, a0, 10
 ; RV64IMB-NEXT:    ret
   %tmp0 = mul i32 %x, -3000
@@ -834,7 +834,7 @@ define signext i32 @mulneg3000_sub8990_b(i32 signext %x) {
 ; RV64IMB-NEXT:    addiw a0, a0, 3
 ; RV64IMB-NEXT:    lui a1, 1048575
 ; RV64IMB-NEXT:    addiw a1, a1, 1096
-; RV64IMB-NEXT:    mulw a0, a0, a1
+; RV64IMB-NEXT:    mul a0, a0, a1
 ; RV64IMB-NEXT:    addiw a0, a0, 10
 ; RV64IMB-NEXT:    ret
   %tmp0 = mul i32 %x, -3000
diff --git a/llvm/test/CodeGen/RISCV/bitextract-mac.ll b/llvm/test/CodeGen/RISCV/bitextract-mac.ll
index f0abc372fb057..ce1e0c4711ffb 100644
--- a/llvm/test/CodeGen/RISCV/bitextract-mac.ll
+++ b/llvm/test/CodeGen/RISCV/bitextract-mac.ll
@@ -74,29 +74,29 @@ define i32 @f(i32 %A, i32 %B, i32 %C) {
 ;
 ; RV64I-LABEL: f:
 ; RV64I:       # %bb.0: # %entry
-; RV64I-NEXT:    mulw a0, a1, a0
+; RV64I-NEXT:    mul a0, a1, a0
 ; RV64I-NEXT:    slli a1, a0, 58
 ; RV64I-NEXT:    srli a1, a1, 60
 ; RV64I-NEXT:    slli a0, a0, 52
 ; RV64I-NEXT:    srli a0, a0, 57
-; RV64I-NEXT:    mulw a0, a1, a0
+; RV64I-NEXT:    mul a0, a1, a0
 ; RV64I-NEXT:    addw a0, a0, a2
 ; RV64I-NEXT:    ret
 ;
 ; RV64ZBB-LABEL: f:
 ; RV64ZBB:       # %bb.0: # %entry
-; RV64ZBB-NEXT:    mulw a0, a1, a0
+; RV64ZBB-NEXT:    mul a0, a1, a0
 ; RV64ZBB-NEXT:    slli a1, a0, 58
 ; RV64ZBB-NEXT:    srli a1, a1, 60
 ; RV64ZBB-NEXT:    slli a0, a0, 52
 ; RV64ZBB-NEXT:    srli a0, a0, 57
-; RV64ZBB-NEXT:    mulw a0, a1, a0
+; RV64ZBB-NEXT:    mul a0, a1, a0
 ; RV64ZBB-NEXT:    addw a0, a0, a2
 ; RV64ZBB-NEXT:    ret
 ;
 ; RV64XTHEADMAC-LABEL: f:
 ; RV64XTHEADMAC:       # %bb.0: # %entry
-; RV64XTHEADMAC-NEXT:    mulw a0, a1, a0
+; RV64XTHEADMAC-NEXT:    mul a0, a1, a0
 ; RV64XTHEADMAC-NEXT:    slli a1, a0, 58
 ; RV64XTHEADMAC-NEXT:    srli a1, a1, 60
 ; RV64XTHEADMAC-NEXT:    slli a0, a0, 52
@@ -110,7 +110,7 @@ define i32 @f(i32 %A, i32 %B, i32 %C) {
 ; RV64XTHEADBB-NEXT:    mul a0, a1, a0
 ; RV64XTHEADBB-NEXT:    th.extu a1, a0, 5, 2
 ; RV64XTHEADBB-NEXT:    th.extu a0, a0, 11, 5
-; RV64XTHEADBB-NEXT:    mulw a0, a1, a0
+; RV64XTHEADBB-NEXT:    mul a0, a1, a0
 ; RV64XTHEADBB-NEXT:    addw a0, a0, a2
 ; RV64XTHEADBB-NEXT:    ret
 ;
diff --git a/llvm/test/CodeGen/RISCV/calling-conv-vector-on-stack.ll b/llvm/test/CodeGen/RISCV/calling-conv-vector-on-stack.ll
new file mode 100644
index 0000000000000..3e2af11365297
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/calling-conv-vector-on-stack.ll
@@ -0,0 +1,47 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv64 -mattr=+v -verify-machineinstrs < %s 2>&1 | FileCheck %s
+
+; CHECK-NOT: warning: Invalid size request on a scalable vector
+
+define void @bar() nounwind {
+; CHECK-LABEL: bar:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    addi sp, sp, -96
+; CHECK-NEXT:    sd ra, 88(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 80(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s1, 72(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 96
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    slli a0, a0, 3
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    andi sp, sp, -64
+; CHECK-NEXT:    mv s1, sp
+; CHECK-NEXT:    addi sp, sp, -16
+; CHECK-NEXT:    vsetvli a0, zero, e32, m8, ta, ma
+; CHECK-NEXT:    vmv.v.i v8, 0
+; CHECK-NEXT:    addi a0, s1, 64
+; CHECK-NEXT:    vs8r.v v8, (a0)
+; CHECK-NEXT:    sd a0, 0(sp)
+; CHECK-NEXT:    li a0, 0
+; CHECK-NEXT:    li a1, 0
+; CHECK-NEXT:    li a2, 0
+; CHECK-NEXT:    li a3, 0
+; CHECK-NEXT:    li a4, 0
+; CHECK-NEXT:    li a5, 0
+; CHECK-NEXT:    li a6, 0
+; CHECK-NEXT:    li a7, 0
+; CHECK-NEXT:    vmv.v.i v16, 0
+; CHECK-NEXT:    call foo@plt
+; CHECK-NEXT:    addi sp, sp, 16
+; CHECK-NEXT:    addi sp, s0, -96
+; CHECK-NEXT:    ld ra, 88(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s0, 80(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s1, 72(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 96
+; CHECK-NEXT:    ret
+entry:
+  tail call void @foo(i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, <vscale x 16 x float> zeroinitializer, <vscale x 16 x float> zeroinitializer, <vscale x 16 x float> zeroinitializer)
+  ret void
+}
+
+declare void @foo(i32, i32, i32, i32, i32, i32, i32, i32, <vscale x 16 x float>, <vscale x 16 x float>, <vscale x 16 x float>) nounwind
diff --git a/llvm/test/CodeGen/RISCV/copyprop.ll b/llvm/test/CodeGen/RISCV/copyprop.ll
new file mode 100644
index 0000000000000..5726a35352788
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/copyprop.ll
@@ -0,0 +1,64 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -O3 -mtriple=riscv64 -riscv-enable-copy-propagation=false | FileCheck %s --check-prefix=NOPROP
+; RUN: llc < %s -O3 -mtriple=riscv64 -riscv-enable-copy-propagation=true | FileCheck %s --check-prefix=PROP
+
+define void @copyprop_after_mbp(i32 %v, i32* %a, i32* %b, i32* %c, i32* %d) {
+; NOPROP-LABEL: copyprop_after_mbp:
+; NOPROP:       # %bb.0:
+; NOPROP-NEXT:    sext.w a0, a0
+; NOPROP-NEXT:    li a5, 10
+; NOPROP-NEXT:    bne a0, a5, .LBB0_2
+; NOPROP-NEXT:  # %bb.1: # %bb.0
+; NOPROP-NEXT:    li a0, 15
+; NOPROP-NEXT:    sw a0, 0(a2)
+; NOPROP-NEXT:    li a0, 1
+; NOPROP-NEXT:    sw a0, 0(a1)
+; NOPROP-NEXT:    li a0, 12
+; NOPROP-NEXT:    sw a0, 0(a4)
+; NOPROP-NEXT:    ret
+; NOPROP-NEXT:  .LBB0_2: # %bb.1
+; NOPROP-NEXT:    li a0, 0
+; NOPROP-NEXT:    li a2, 25
+; NOPROP-NEXT:    sw a2, 0(a3)
+; NOPROP-NEXT:    sw a0, 0(a1)
+; NOPROP-NEXT:    li a0, 12
+; NOPROP-NEXT:    sw a0, 0(a4)
+; NOPROP-NEXT:    ret
+;
+; PROP-LABEL: copyprop_after_mbp:
+; PROP:       # %bb.0:
+; PROP-NEXT:    sext.w a0, a0
+; PROP-NEXT:    li a5, 10
+; PROP-NEXT:    bne a0, a5, .LBB0_2
+; PROP-NEXT:  # %bb.1: # %bb.0
+; PROP-NEXT:    li a0, 15
+; PROP-NEXT:    sw a0, 0(a2)
+; PROP-NEXT:    li a0, 1
+; PROP-NEXT:    sw a0, 0(a1)
+; PROP-NEXT:    li a0, 12
+; PROP-NEXT:    sw a0, 0(a4)
+; PROP-NEXT:    ret
+; PROP-NEXT:  .LBB0_2: # %bb.1
+; PROP-NEXT:    li a2, 25
+; PROP-NEXT:    sw a2, 0(a3)
+; PROP-NEXT:    sw zero, 0(a1)
+; PROP-NEXT:    li a0, 12
+; PROP-NEXT:    sw a0, 0(a4)
+; PROP-NEXT:    ret
+  %1 = icmp eq i32 %v, 10
+  br i1 %1, label %bb.0, label %bb.1
+
+bb.0:
+  store i32 15, i32* %b, align 4
+  br label %bb.2
+
+bb.1:
+  store i32 25, i32* %c, align 4
+  br label %bb.2
+
+bb.2:
+  %2 = phi i32 [ 1, %bb.0 ], [ 0, %bb.1 ]
+  store i32 %2, i32* %a, align 4
+  store i32 12, i32* %d, align 4
+  ret void
+}
diff --git a/llvm/test/CodeGen/RISCV/ctlz-cttz-ctpop.ll b/llvm/test/CodeGen/RISCV/ctlz-cttz-ctpop.ll
index f251c9808f05c..ac1b94d1c9146 100644
--- a/llvm/test/CodeGen/RISCV/ctlz-cttz-ctpop.ll
+++ b/llvm/test/CodeGen/RISCV/ctlz-cttz-ctpop.ll
@@ -309,7 +309,7 @@ define i32 @test_cttz_i32(i32 %a) nounwind {
 ; RV64M-NEXT:    and a0, a0, a1
 ; RV64M-NEXT:    lui a1, 30667
 ; RV64M-NEXT:    addiw a1, a1, 1329
-; RV64M-NEXT:    mulw a0, a0, a1
+; RV64M-NEXT:    mul a0, a0, a1
 ; RV64M-NEXT:    srliw a0, a0, 27
 ; RV64M-NEXT:    lui a1, %hi(.LCPI2_0)
 ; RV64M-NEXT:    addi a1, a1, %lo(.LCPI2_0)
@@ -754,7 +754,7 @@ define i32 @test_cttz_i32_zero_undef(i32 %a) nounwind {
 ; RV64M-NEXT:    and a0, a0, a1
 ; RV64M-NEXT:    lui a1, 30667
 ; RV64M-NEXT:    addiw a1, a1, 1329
-; RV64M-NEXT:    mulw a0, a0, a1
+; RV64M-NEXT:    mul a0, a0, a1
 ; RV64M-NEXT:    srliw a0, a0, 27
 ; RV64M-NEXT:    lui a1, %hi(.LCPI6_0)
 ; RV64M-NEXT:    addi a1, a1, %lo(.LCPI6_0)
@@ -1319,7 +1319,7 @@ define i32 @test_ctlz_i32(i32 %a) nounwind {
 ; RV64M-NEXT:    and a0, a0, a1
 ; RV64M-NEXT:    lui a1, 4112
 ; RV64M-NEXT:    addiw a1, a1, 257
-; RV64M-NEXT:    mulw a0, a0, a1
+; RV64M-NEXT:    mul a0, a0, a1
 ; RV64M-NEXT:    srliw a0, a0, 24
 ; RV64M-NEXT:    ret
 ; RV64M-NEXT:  .LBB10_2:
@@ -1957,7 +1957,7 @@ define i32 @test_ctlz_i32_zero_undef(i32 %a) nounwind {
 ; RV64M-NEXT:    and a0, a0, a1
 ; RV64M-NEXT:    lui a1, 4112
 ; RV64M-NEXT:    addiw a1, a1, 257
-; RV64M-NEXT:    mulw a0, a0, a1
+; RV64M-NEXT:    mul a0, a0, a1
 ; RV64M-NEXT:    srliw a0, a0, 24
 ; RV64M-NEXT:    ret
 ;
@@ -2530,7 +2530,7 @@ define i32 @test_ctpop_i32(i32 %a) nounwind {
 ; RV64M-NEXT:    and a0, a0, a1
 ; RV64M-NEXT:    lui a1, 4112
 ; RV64M-NEXT:    addiw a1, a1, 257
-; RV64M-NEXT:    mulw a0, a0, a1
+; RV64M-NEXT:    mul a0, a0, a1
 ; RV64M-NEXT:    srliw a0, a0, 24
 ; RV64M-NEXT:    ret
 ;
diff --git a/llvm/test/CodeGen/RISCV/double-convert-strict.ll b/llvm/test/CodeGen/RISCV/double-convert-strict.ll
index 557cc1c7f1704..b353d425098c5 100644
--- a/llvm/test/CodeGen/RISCV/double-convert-strict.ll
+++ b/llvm/test/CodeGen/RISCV/double-convert-strict.ll
@@ -134,10 +134,8 @@ define i32 @fcvt_wu_d_multiple_use(double %x, ptr %y) nounwind {
 ; CHECKIFD-LABEL: fcvt_wu_d_multiple_use:
 ; CHECKIFD:       # %bb.0:
 ; CHECKIFD-NEXT:    fcvt.wu.d a0, fa0, rtz
-; CHECKIFD-NEXT:    bnez a0, .LBB4_2
-; CHECKIFD-NEXT:  # %bb.1:
-; CHECKIFD-NEXT:    li a0, 1
-; CHECKIFD-NEXT:  .LBB4_2:
+; CHECKIFD-NEXT:    seqz a1, a0
+; CHECKIFD-NEXT:    add a0, a0, a1
 ; CHECKIFD-NEXT:    ret
 ;
 ; RV32I-LABEL: fcvt_wu_d_multiple_use:
@@ -145,10 +143,8 @@ define i32 @fcvt_wu_d_multiple_use(double %x, ptr %y) nounwind {
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    call __fixunsdfsi@plt
-; RV32I-NEXT:    bnez a0, .LBB4_2
-; RV32I-NEXT:  # %bb.1:
-; RV32I-NEXT:    li a0, 1
-; RV32I-NEXT:  .LBB4_2:
+; RV32I-NEXT:    seqz a1, a0
+; RV32I-NEXT:    add a0, a0, a1
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
@@ -158,10 +154,8 @@ define i32 @fcvt_wu_d_multiple_use(double %x, ptr %y) nounwind {
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    call __fixunsdfsi@plt
-; RV64I-NEXT:    bnez a0, .LBB4_2
-; RV64I-NEXT:  # %bb.1:
-; RV64I-NEXT:    li a0, 1
-; RV64I-NEXT:  .LBB4_2:
+; RV64I-NEXT:    seqz a1, a0
+; RV64I-NEXT:    add a0, a0, a1
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/double-convert.ll b/llvm/test/CodeGen/RISCV/double-convert.ll
index f65f55e59fb3c..e84be956b567a 100644
--- a/llvm/test/CodeGen/RISCV/double-convert.ll
+++ b/llvm/test/CodeGen/RISCV/double-convert.ll
@@ -236,10 +236,8 @@ define i32 @fcvt_wu_d_multiple_use(double %x, ptr %y) nounwind {
 ; CHECKIFD-LABEL: fcvt_wu_d_multiple_use:
 ; CHECKIFD:       # %bb.0:
 ; CHECKIFD-NEXT:    fcvt.wu.d a0, fa0, rtz
-; CHECKIFD-NEXT:    bnez a0, .LBB5_2
-; CHECKIFD-NEXT:  # %bb.1:
-; CHECKIFD-NEXT:    li a0, 1
-; CHECKIFD-NEXT:  .LBB5_2:
+; CHECKIFD-NEXT:    seqz a1, a0
+; CHECKIFD-NEXT:    add a0, a0, a1
 ; CHECKIFD-NEXT:    ret
 ;
 ; RV32I-LABEL: fcvt_wu_d_multiple_use:
@@ -247,10 +245,8 @@ define i32 @fcvt_wu_d_multiple_use(double %x, ptr %y) nounwind {
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    call __fixunsdfsi@plt
-; RV32I-NEXT:    bnez a0, .LBB5_2
-; RV32I-NEXT:  # %bb.1:
-; RV32I-NEXT:    li a0, 1
-; RV32I-NEXT:  .LBB5_2:
+; RV32I-NEXT:    seqz a1, a0
+; RV32I-NEXT:    add a0, a0, a1
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
@@ -260,10 +256,8 @@ define i32 @fcvt_wu_d_multiple_use(double %x, ptr %y) nounwind {
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    call __fixunsdfsi@plt
-; RV64I-NEXT:    bnez a0, .LBB5_2
-; RV64I-NEXT:  # %bb.1:
-; RV64I-NEXT:    li a0, 1
-; RV64I-NEXT:  .LBB5_2:
+; RV64I-NEXT:    seqz a1, a0
+; RV64I-NEXT:    add a0, a0, a1
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/double-zfa.ll b/llvm/test/CodeGen/RISCV/double-zfa.ll
index a4550cde2d67c..9dcb0c93562c3 100644
--- a/llvm/test/CodeGen/RISCV/double-zfa.ll
+++ b/llvm/test/CodeGen/RISCV/double-zfa.ll
@@ -4,6 +4,135 @@
 ; RUN: llc -mtriple=riscv64 -target-abi lp64d -mattr=+experimental-zfa,+d < %s \
 ; RUN:     | FileCheck --check-prefixes=CHECK,RV64DZFA %s
 
+define double @loadfpimm1() {
+; CHECK-LABEL: loadfpimm1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.d fa0, 0.0625
+; CHECK-NEXT:    ret
+  ret double 0.0625
+}
+
+define double @loadfpimm2() {
+; CHECK-LABEL: loadfpimm2:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.d fa0, 0.75
+; CHECK-NEXT:    ret
+  ret double 0.75
+}
+
+define double @loadfpimm3() {
+; CHECK-LABEL: loadfpimm3:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.d fa0, 1.25
+; CHECK-NEXT:    ret
+  ret double 1.25
+}
+
+define double @loadfpimm4() {
+; CHECK-LABEL: loadfpimm4:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.d fa0, 3.0
+; CHECK-NEXT:    ret
+  ret double 3.0
+}
+
+define double @loadfpimm5() {
+; CHECK-LABEL: loadfpimm5:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.d fa0, 256.0
+; CHECK-NEXT:    ret
+  ret double 256.0
+}
+
+define double @loadfpimm6() {
+; CHECK-LABEL: loadfpimm6:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.d fa0, inf
+; CHECK-NEXT:    ret
+  ret double 0x7FF0000000000000
+}
+
+define double @loadfpimm7() {
+; CHECK-LABEL: loadfpimm7:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.d fa0, nan
+; CHECK-NEXT:    ret
+  ret double 0x7FF8000000000000
+}
+
+define double @loadfpimm8() {
+; CHECK-LABEL: loadfpimm8:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.d fa0, min
+; CHECK-NEXT:    ret
+  ret double 0x0010000000000000
+}
+
+define double @loadfpimm9() {
+; CHECK-LABEL: loadfpimm9:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    lui a0, %hi(.LCPI8_0)
+; CHECK-NEXT:    fld fa0, %lo(.LCPI8_0)(a0)
+; CHECK-NEXT:    ret
+  ret double 255.0
+}
+
+; Negative test. This is 1 * 2^256.
+define double @loadfpimm10() {
+; CHECK-LABEL: loadfpimm10:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    lui a0, %hi(.LCPI9_0)
+; CHECK-NEXT:    fld fa0, %lo(.LCPI9_0)(a0)
+; CHECK-NEXT:    ret
+  ret double 0x1000000000000000
+}
+
+; Negative test. This is a qnan with payload of 1.
+define double @loadfpimm11() {
+; CHECK-LABEL: loadfpimm11:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    lui a0, %hi(.LCPI10_0)
+; CHECK-NEXT:    fld fa0, %lo(.LCPI10_0)(a0)
+; CHECK-NEXT:    ret
+  ret double 0x7ff8000000000001
+}
+
+; Negative test. This is an snan with payload of 1.
+define double @loadfpimm12() {
+; CHECK-LABEL: loadfpimm12:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    lui a0, %hi(.LCPI11_0)
+; CHECK-NEXT:    fld fa0, %lo(.LCPI11_0)(a0)
+; CHECK-NEXT:    ret
+  ret double 0x7ff0000000000001
+}
+
+; Negative test. This is the smallest denormal.
+define double @loadfpimm13() {
+; RV32IDZFA-LABEL: loadfpimm13:
+; RV32IDZFA:       # %bb.0:
+; RV32IDZFA-NEXT:    lui a0, %hi(.LCPI12_0)
+; RV32IDZFA-NEXT:    fld fa0, %lo(.LCPI12_0)(a0)
+; RV32IDZFA-NEXT:    ret
+;
+; RV64DZFA-LABEL: loadfpimm13:
+; RV64DZFA:       # %bb.0:
+; RV64DZFA-NEXT:    li a0, 1
+; RV64DZFA-NEXT:    fmv.d.x fa0, a0
+; RV64DZFA-NEXT:    ret
+  ret double 0x0000000000000001
+}
+
+; Negative test. This is 2^-1023, a denormal.
+define double @loadfpimm15() {
+; CHECK-LABEL: loadfpimm15:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    lui a0, %hi(.LCPI13_0)
+; CHECK-NEXT:    fld fa0, %lo(.LCPI13_0)(a0)
+; CHECK-NEXT:    ret
+  ret double 0x0008000000000000
+}
+
 declare double @llvm.minimum.f64(double, double)
 
 define double @fminm_d(double %a, double %b) nounwind {
diff --git a/llvm/test/CodeGen/RISCV/fli-licm.ll b/llvm/test/CodeGen/RISCV/fli-licm.ll
new file mode 100644
index 0000000000000..93bb934c1cb0d
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/fli-licm.ll
@@ -0,0 +1,70 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
+; RUN: llc < %s -mtriple=riscv32 -target-abi=ilp32f -mattr=+experimental-zfa \
+; RUN:   | FileCheck %s --check-prefix=RV32
+; RUN: llc < %s -mtriple=riscv64 -target-abi=lp64f -mattr=+experimental-zfa \
+; RUN:   | FileCheck %s --check-prefix=RV64
+
+; The purpose of this test is to check that an FLI instruction that
+; materializes an immediate is not MachineLICM'd out of a loop.
+
+%struct.Node = type { ptr, i8* }
+
+define void @process_nodes(ptr %0) nounwind {
+; RV32-LABEL: process_nodes:
+; RV32:       # %bb.0: # %entry
+; RV32-NEXT:    addi sp, sp, -16
+; RV32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; RV32-NEXT:    sw s0, 8(sp) # 4-byte Folded Spill
+; RV32-NEXT:    beqz a0, .LBB0_3
+; RV32-NEXT:  # %bb.1: # %loop.preheader
+; RV32-NEXT:    mv s0, a0
+; RV32-NEXT:  .LBB0_2: # %loop
+; RV32-NEXT:    # =>This Inner Loop Header: Depth=1
+; RV32-NEXT:    fli.s fa0, 1.0
+; RV32-NEXT:    mv a0, s0
+; RV32-NEXT:    call do_it@plt
+; RV32-NEXT:    lw s0, 0(s0)
+; RV32-NEXT:    bnez s0, .LBB0_2
+; RV32-NEXT:  .LBB0_3: # %exit
+; RV32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; RV32-NEXT:    lw s0, 8(sp) # 4-byte Folded Reload
+; RV32-NEXT:    addi sp, sp, 16
+; RV32-NEXT:    ret
+;
+; RV64-LABEL: process_nodes:
+; RV64:       # %bb.0: # %entry
+; RV64-NEXT:    addi sp, sp, -16
+; RV64-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
+; RV64-NEXT:    sd s0, 0(sp) # 8-byte Folded Spill
+; RV64-NEXT:    beqz a0, .LBB0_3
+; RV64-NEXT:  # %bb.1: # %loop.preheader
+; RV64-NEXT:    mv s0, a0
+; RV64-NEXT:  .LBB0_2: # %loop
+; RV64-NEXT:    # =>This Inner Loop Header: Depth=1
+; RV64-NEXT:    fli.s fa0, 1.0
+; RV64-NEXT:    mv a0, s0
+; RV64-NEXT:    call do_it@plt
+; RV64-NEXT:    ld s0, 0(s0)
+; RV64-NEXT:    bnez s0, .LBB0_2
+; RV64-NEXT:  .LBB0_3: # %exit
+; RV64-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
+; RV64-NEXT:    ld s0, 0(sp) # 8-byte Folded Reload
+; RV64-NEXT:    addi sp, sp, 16
+; RV64-NEXT:    ret
+entry:
+  %1 = icmp eq ptr %0, null
+  br i1 %1, label %exit, label %loop
+
+loop:
+  %2 = phi %struct.Node* [ %4, %loop ], [ %0, %entry ]
+  tail call void @do_it(float 1.000000e+00, ptr nonnull %2)
+  %3 = getelementptr inbounds %struct.Node, ptr %2, i64 0, i32 0
+  %4 = load ptr, ptr %3, align 8
+  %5 = icmp eq ptr %4, null
+  br i1 %5, label %exit, label %loop
+
+exit:
+  ret void
+}
+
+declare void @do_it(float, ptr)
diff --git a/llvm/test/CodeGen/RISCV/float-convert-strict.ll b/llvm/test/CodeGen/RISCV/float-convert-strict.ll
index b38f03c60f584..9bf675f3a6814 100644
--- a/llvm/test/CodeGen/RISCV/float-convert-strict.ll
+++ b/llvm/test/CodeGen/RISCV/float-convert-strict.ll
@@ -76,10 +76,8 @@ define i32 @fcvt_wu_s_multiple_use(float %x, ptr %y) nounwind {
 ; CHECKIF-LABEL: fcvt_wu_s_multiple_use:
 ; CHECKIF:       # %bb.0:
 ; CHECKIF-NEXT:    fcvt.wu.s a0, fa0, rtz
-; CHECKIF-NEXT:    bnez a0, .LBB2_2
-; CHECKIF-NEXT:  # %bb.1:
-; CHECKIF-NEXT:    li a0, 1
-; CHECKIF-NEXT:  .LBB2_2:
+; CHECKIF-NEXT:    seqz a1, a0
+; CHECKIF-NEXT:    add a0, a0, a1
 ; CHECKIF-NEXT:    ret
 ;
 ; RV32I-LABEL: fcvt_wu_s_multiple_use:
@@ -87,10 +85,8 @@ define i32 @fcvt_wu_s_multiple_use(float %x, ptr %y) nounwind {
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    call __fixunssfsi@plt
-; RV32I-NEXT:    bnez a0, .LBB2_2
-; RV32I-NEXT:  # %bb.1:
-; RV32I-NEXT:    li a0, 1
-; RV32I-NEXT:  .LBB2_2:
+; RV32I-NEXT:    seqz a1, a0
+; RV32I-NEXT:    add a0, a0, a1
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
@@ -100,10 +96,8 @@ define i32 @fcvt_wu_s_multiple_use(float %x, ptr %y) nounwind {
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    call __fixunssfsi@plt
-; RV64I-NEXT:    bnez a0, .LBB2_2
-; RV64I-NEXT:  # %bb.1:
-; RV64I-NEXT:    li a0, 1
-; RV64I-NEXT:  .LBB2_2:
+; RV64I-NEXT:    seqz a1, a0
+; RV64I-NEXT:    add a0, a0, a1
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/float-convert.ll b/llvm/test/CodeGen/RISCV/float-convert.ll
index 22d59fccc9e77..e0ce301dad3a7 100644
--- a/llvm/test/CodeGen/RISCV/float-convert.ll
+++ b/llvm/test/CodeGen/RISCV/float-convert.ll
@@ -167,10 +167,8 @@ define i32 @fcvt_wu_s_multiple_use(float %x, ptr %y) nounwind {
 ; CHECKIF-LABEL: fcvt_wu_s_multiple_use:
 ; CHECKIF:       # %bb.0:
 ; CHECKIF-NEXT:    fcvt.wu.s a0, fa0, rtz
-; CHECKIF-NEXT:    bnez a0, .LBB3_2
-; CHECKIF-NEXT:  # %bb.1:
-; CHECKIF-NEXT:    li a0, 1
-; CHECKIF-NEXT:  .LBB3_2:
+; CHECKIF-NEXT:    seqz a1, a0
+; CHECKIF-NEXT:    add a0, a0, a1
 ; CHECKIF-NEXT:    ret
 ;
 ; RV32I-LABEL: fcvt_wu_s_multiple_use:
@@ -178,10 +176,8 @@ define i32 @fcvt_wu_s_multiple_use(float %x, ptr %y) nounwind {
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    call __fixunssfsi@plt
-; RV32I-NEXT:    bnez a0, .LBB3_2
-; RV32I-NEXT:  # %bb.1:
-; RV32I-NEXT:    li a0, 1
-; RV32I-NEXT:  .LBB3_2:
+; RV32I-NEXT:    seqz a1, a0
+; RV32I-NEXT:    add a0, a0, a1
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
@@ -191,10 +187,8 @@ define i32 @fcvt_wu_s_multiple_use(float %x, ptr %y) nounwind {
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    call __fixunssfsi@plt
-; RV64I-NEXT:    bnez a0, .LBB3_2
-; RV64I-NEXT:  # %bb.1:
-; RV64I-NEXT:    li a0, 1
-; RV64I-NEXT:  .LBB3_2:
+; RV64I-NEXT:    seqz a1, a0
+; RV64I-NEXT:    add a0, a0, a1
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/float-zfa.ll b/llvm/test/CodeGen/RISCV/float-zfa.ll
index 2b47399e75278..e56ab8c9107fa 100644
--- a/llvm/test/CodeGen/RISCV/float-zfa.ll
+++ b/llvm/test/CodeGen/RISCV/float-zfa.ll
@@ -4,6 +4,89 @@
 ; RUN: llc -mtriple=riscv64 -target-abi lp64f -mattr=+experimental-zfa < %s \
 ; RUN:     | FileCheck %s
 
+define float @loadfpimm1() {
+; CHECK-LABEL: loadfpimm1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.s fa0, 0.0625
+; CHECK-NEXT:    ret
+  ret float 0.0625
+}
+
+define float @loadfpimm2() {
+; CHECK-LABEL: loadfpimm2:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.s fa0, 0.75
+; CHECK-NEXT:    ret
+  ret float 0.75
+}
+
+define float @loadfpimm3() {
+; CHECK-LABEL: loadfpimm3:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.s fa0, 1.25
+; CHECK-NEXT:    ret
+  ret float 1.25
+}
+
+define float @loadfpimm4() {
+; CHECK-LABEL: loadfpimm4:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.s fa0, 3.0
+; CHECK-NEXT:    ret
+  ret float 3.0
+}
+
+define float @loadfpimm5() {
+; CHECK-LABEL: loadfpimm5:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.s fa0, 256.0
+; CHECK-NEXT:    ret
+  ret float 256.0
+}
+
+define float @loadfpimm6() {
+; CHECK-LABEL: loadfpimm6:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.s fa0, inf
+; CHECK-NEXT:    ret
+  ret float 0x7FF0000000000000
+}
+
+define float @loadfpimm7() {
+; CHECK-LABEL: loadfpimm7:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.s fa0, nan
+; CHECK-NEXT:    ret
+  ret float 0x7FF8000000000000
+}
+
+define float @loadfpimm8() {
+; CHECK-LABEL: loadfpimm8:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.s fa0, min
+; CHECK-NEXT:    ret
+  ret float 0x3810000000000000
+}
+
+define float @loadfpimm9() {
+; CHECK-LABEL: loadfpimm9:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    lui a0, 276464
+; CHECK-NEXT:    fmv.w.x fa0, a0
+; CHECK-NEXT:    ret
+  ret float 255.0
+}
+
+; This is the f16 minimum value. Make sure we don't use fli.s.
+define float @loadfpimm10() {
+; CHECK-LABEL: loadfpimm10:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    lui a0, 231424
+; CHECK-NEXT:    fmv.w.x fa0, a0
+; CHECK-NEXT:    ret
+  ret float 0.00006103515625
+}
+
 declare float @llvm.minimum.f32(float, float)
 
 define float @fminm_s(float %a, float %b) nounwind {
@@ -144,3 +227,16 @@ define i32 @fcmp_ueq_q(float %a, float %b) nounwind strictfp {
   %2 = zext i1 %1 to i32
   ret i32 %2
 }
+
+declare void @foo(float, float)
+
+; Make sure we use two fli instructions instead of copying.
+define void @fli_remat() {
+; CHECK-LABEL: fli_remat:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.s fa0, 1.0
+; CHECK-NEXT:    fli.s fa1, 1.0
+; CHECK-NEXT:    tail foo@plt
+  tail call void @foo(float 1.000000e+00, float 1.000000e+00)
+  ret void
+}
diff --git a/llvm/test/CodeGen/RISCV/forced-atomics.ll b/llvm/test/CodeGen/RISCV/forced-atomics.ll
index 1bc88567d9ecc..7c92a74dc9517 100644
--- a/llvm/test/CodeGen/RISCV/forced-atomics.ll
+++ b/llvm/test/CodeGen/RISCV/forced-atomics.ll
@@ -1169,9 +1169,10 @@ define i32 @rmw32_umax_seq_cst(ptr %p) nounwind {
 ; RV32-NO-ATOMIC-NEXT:    sw s0, 8(sp) # 4-byte Folded Spill
 ; RV32-NO-ATOMIC-NEXT:    mv s0, a0
 ; RV32-NO-ATOMIC-NEXT:    lw a1, 0(a0)
-; RV32-NO-ATOMIC-NEXT:    j .LBB25_2
 ; RV32-NO-ATOMIC-NEXT:  .LBB25_1: # %atomicrmw.start
-; RV32-NO-ATOMIC-NEXT:    # in Loop: Header=BB25_2 Depth=1
+; RV32-NO-ATOMIC-NEXT:    # =>This Inner Loop Header: Depth=1
+; RV32-NO-ATOMIC-NEXT:    seqz a2, a1
+; RV32-NO-ATOMIC-NEXT:    add a2, a1, a2
 ; RV32-NO-ATOMIC-NEXT:    sw a1, 4(sp)
 ; RV32-NO-ATOMIC-NEXT:    addi a1, sp, 4
 ; RV32-NO-ATOMIC-NEXT:    li a3, 5
@@ -1179,16 +1180,8 @@ define i32 @rmw32_umax_seq_cst(ptr %p) nounwind {
 ; RV32-NO-ATOMIC-NEXT:    mv a0, s0
 ; RV32-NO-ATOMIC-NEXT:    call __atomic_compare_exchange_4@plt
 ; RV32-NO-ATOMIC-NEXT:    lw a1, 4(sp)
-; RV32-NO-ATOMIC-NEXT:    bnez a0, .LBB25_4
-; RV32-NO-ATOMIC-NEXT:  .LBB25_2: # %atomicrmw.start
-; RV32-NO-ATOMIC-NEXT:    # =>This Inner Loop Header: Depth=1
-; RV32-NO-ATOMIC-NEXT:    mv a2, a1
-; RV32-NO-ATOMIC-NEXT:    bnez a1, .LBB25_1
-; RV32-NO-ATOMIC-NEXT:  # %bb.3: # %atomicrmw.start
-; RV32-NO-ATOMIC-NEXT:    # in Loop: Header=BB25_2 Depth=1
-; RV32-NO-ATOMIC-NEXT:    li a2, 1
-; RV32-NO-ATOMIC-NEXT:    j .LBB25_1
-; RV32-NO-ATOMIC-NEXT:  .LBB25_4: # %atomicrmw.end
+; RV32-NO-ATOMIC-NEXT:    beqz a0, .LBB25_1
+; RV32-NO-ATOMIC-NEXT:  # %bb.2: # %atomicrmw.end
 ; RV32-NO-ATOMIC-NEXT:    mv a0, a1
 ; RV32-NO-ATOMIC-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32-NO-ATOMIC-NEXT:    lw s0, 8(sp) # 4-byte Folded Reload
@@ -2740,9 +2733,10 @@ define i64 @rmw64_umax_seq_cst(ptr %p) nounwind {
 ; RV64-NO-ATOMIC-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
 ; RV64-NO-ATOMIC-NEXT:    mv s0, a0
 ; RV64-NO-ATOMIC-NEXT:    ld a1, 0(a0)
-; RV64-NO-ATOMIC-NEXT:    j .LBB51_2
 ; RV64-NO-ATOMIC-NEXT:  .LBB51_1: # %atomicrmw.start
-; RV64-NO-ATOMIC-NEXT:    # in Loop: Header=BB51_2 Depth=1
+; RV64-NO-ATOMIC-NEXT:    # =>This Inner Loop Header: Depth=1
+; RV64-NO-ATOMIC-NEXT:    seqz a2, a1
+; RV64-NO-ATOMIC-NEXT:    add a2, a1, a2
 ; RV64-NO-ATOMIC-NEXT:    sd a1, 8(sp)
 ; RV64-NO-ATOMIC-NEXT:    addi a1, sp, 8
 ; RV64-NO-ATOMIC-NEXT:    li a3, 5
@@ -2750,16 +2744,8 @@ define i64 @rmw64_umax_seq_cst(ptr %p) nounwind {
 ; RV64-NO-ATOMIC-NEXT:    mv a0, s0
 ; RV64-NO-ATOMIC-NEXT:    call __atomic_compare_exchange_8@plt
 ; RV64-NO-ATOMIC-NEXT:    ld a1, 8(sp)
-; RV64-NO-ATOMIC-NEXT:    bnez a0, .LBB51_4
-; RV64-NO-ATOMIC-NEXT:  .LBB51_2: # %atomicrmw.start
-; RV64-NO-ATOMIC-NEXT:    # =>This Inner Loop Header: Depth=1
-; RV64-NO-ATOMIC-NEXT:    mv a2, a1
-; RV64-NO-ATOMIC-NEXT:    bnez a1, .LBB51_1
-; RV64-NO-ATOMIC-NEXT:  # %bb.3: # %atomicrmw.start
-; RV64-NO-ATOMIC-NEXT:    # in Loop: Header=BB51_2 Depth=1
-; RV64-NO-ATOMIC-NEXT:    li a2, 1
-; RV64-NO-ATOMIC-NEXT:    j .LBB51_1
-; RV64-NO-ATOMIC-NEXT:  .LBB51_4: # %atomicrmw.end
+; RV64-NO-ATOMIC-NEXT:    beqz a0, .LBB51_1
+; RV64-NO-ATOMIC-NEXT:  # %bb.2: # %atomicrmw.end
 ; RV64-NO-ATOMIC-NEXT:    mv a0, a1
 ; RV64-NO-ATOMIC-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
 ; RV64-NO-ATOMIC-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
diff --git a/llvm/test/CodeGen/RISCV/fpclamptosat.ll b/llvm/test/CodeGen/RISCV/fpclamptosat.ll
index 7eb7e14353329..588c629537893 100644
--- a/llvm/test/CodeGen/RISCV/fpclamptosat.ll
+++ b/llvm/test/CodeGen/RISCV/fpclamptosat.ll
@@ -92,7 +92,7 @@ define i32 @stest_f64i32(double %x) {
 ; RV64IFD-NEXT:    ret
 ; RV64IFD-NEXT:  .LBB0_3: # %entry
 ; RV64IFD-NEXT:    mv a0, a2
-; RV64IFD-NEXT:    blt a1, a0, .LBB0_2
+; RV64IFD-NEXT:    blt a1, a2, .LBB0_2
 ; RV64IFD-NEXT:  .LBB0_4: # %entry
 ; RV64IFD-NEXT:    lui a0, 524288
 ; RV64IFD-NEXT:    ret
@@ -273,7 +273,7 @@ define i32 @stest_f32i32(float %x) {
 ; RV64-NEXT:    ret
 ; RV64-NEXT:  .LBB3_3: # %entry
 ; RV64-NEXT:    mv a0, a2
-; RV64-NEXT:    blt a1, a0, .LBB3_2
+; RV64-NEXT:    blt a1, a2, .LBB3_2
 ; RV64-NEXT:  .LBB3_4: # %entry
 ; RV64-NEXT:    lui a0, 524288
 ; RV64-NEXT:    ret
@@ -2043,7 +2043,7 @@ define i32 @stest_f64i32_mm(double %x) {
 ; RV64IFD-NEXT:    ret
 ; RV64IFD-NEXT:  .LBB27_3: # %entry
 ; RV64IFD-NEXT:    mv a0, a2
-; RV64IFD-NEXT:    blt a1, a0, .LBB27_2
+; RV64IFD-NEXT:    blt a1, a2, .LBB27_2
 ; RV64IFD-NEXT:  .LBB27_4: # %entry
 ; RV64IFD-NEXT:    lui a0, 524288
 ; RV64IFD-NEXT:    ret
@@ -2211,7 +2211,7 @@ define i32 @stest_f32i32_mm(float %x) {
 ; RV64-NEXT:    ret
 ; RV64-NEXT:  .LBB30_3: # %entry
 ; RV64-NEXT:    mv a0, a2
-; RV64-NEXT:    blt a1, a0, .LBB30_2
+; RV64-NEXT:    blt a1, a2, .LBB30_2
 ; RV64-NEXT:  .LBB30_4: # %entry
 ; RV64-NEXT:    lui a0, 524288
 ; RV64-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/fpclamptosat_vec.ll b/llvm/test/CodeGen/RISCV/fpclamptosat_vec.ll
index 77faf67002eac..de3bbf360a360 100644
--- a/llvm/test/CodeGen/RISCV/fpclamptosat_vec.ll
+++ b/llvm/test/CodeGen/RISCV/fpclamptosat_vec.ll
@@ -27,7 +27,7 @@ define <2 x i32> @stest_f64i32(<2 x double> %x) {
 ; CHECK-NOV-NEXT:    blt a0, a3, .LBB0_2
 ; CHECK-NOV-NEXT:  .LBB0_6: # %entry
 ; CHECK-NOV-NEXT:    mv a0, a3
-; CHECK-NOV-NEXT:    blt a2, a0, .LBB0_3
+; CHECK-NOV-NEXT:    blt a2, a3, .LBB0_3
 ; CHECK-NOV-NEXT:  .LBB0_7: # %entry
 ; CHECK-NOV-NEXT:    lui a0, 524288
 ; CHECK-NOV-NEXT:    blt a2, a1, .LBB0_4
@@ -182,7 +182,7 @@ define <4 x i32> @stest_f32i32(<4 x float> %x) {
 ; CHECK-NOV-NEXT:    blt a5, a6, .LBB3_4
 ; CHECK-NOV-NEXT:  .LBB3_13: # %entry
 ; CHECK-NOV-NEXT:    mv a5, a6
-; CHECK-NOV-NEXT:    blt a3, a5, .LBB3_5
+; CHECK-NOV-NEXT:    blt a3, a6, .LBB3_5
 ; CHECK-NOV-NEXT:  .LBB3_14: # %entry
 ; CHECK-NOV-NEXT:    lui a5, 524288
 ; CHECK-NOV-NEXT:    blt a3, a4, .LBB3_6
@@ -431,7 +431,7 @@ define <4 x i32> @stest_f16i32(<4 x half> %x) {
 ; CHECK-NOV-NEXT:    blt a3, a4, .LBB6_4
 ; CHECK-NOV-NEXT:  .LBB6_13: # %entry
 ; CHECK-NOV-NEXT:    mv a3, a4
-; CHECK-NOV-NEXT:    blt a1, a3, .LBB6_5
+; CHECK-NOV-NEXT:    blt a1, a4, .LBB6_5
 ; CHECK-NOV-NEXT:  .LBB6_14: # %entry
 ; CHECK-NOV-NEXT:    lui a3, 524288
 ; CHECK-NOV-NEXT:    blt a1, a2, .LBB6_6
@@ -476,24 +476,24 @@ define <4 x i32> @stest_f16i32(<4 x half> %x) {
 ; CHECK-V-NEXT:    sd a0, 8(sp)
 ; CHECK-V-NEXT:    addi a0, sp, 24
 ; CHECK-V-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-V-NEXT:    vle64.v v8, (a0)
-; CHECK-V-NEXT:    mv a0, sp
 ; CHECK-V-NEXT:    vle64.v v10, (a0)
+; CHECK-V-NEXT:    mv a0, sp
+; CHECK-V-NEXT:    vle64.v v8, (a0)
 ; CHECK-V-NEXT:    vsetivli zero, 2, e64, m2, tu, ma
-; CHECK-V-NEXT:    vslideup.vi v10, v8, 1
+; CHECK-V-NEXT:    vslideup.vi v8, v10, 1
 ; CHECK-V-NEXT:    addi a0, sp, 16
 ; CHECK-V-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-V-NEXT:    vle64.v v8, (a0)
+; CHECK-V-NEXT:    vle64.v v10, (a0)
 ; CHECK-V-NEXT:    vsetivli zero, 3, e64, m2, tu, ma
-; CHECK-V-NEXT:    vslideup.vi v10, v8, 2
+; CHECK-V-NEXT:    vslideup.vi v8, v10, 2
 ; CHECK-V-NEXT:    addi a0, sp, 8
 ; CHECK-V-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-V-NEXT:    vle64.v v8, (a0)
+; CHECK-V-NEXT:    vle64.v v10, (a0)
 ; CHECK-V-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
-; CHECK-V-NEXT:    vslideup.vi v10, v8, 3
+; CHECK-V-NEXT:    vslideup.vi v8, v10, 3
 ; CHECK-V-NEXT:    lui a0, 524288
 ; CHECK-V-NEXT:    addiw a1, a0, -1
-; CHECK-V-NEXT:    vmin.vx v8, v10, a1
+; CHECK-V-NEXT:    vmin.vx v8, v8, a1
 ; CHECK-V-NEXT:    vmax.vx v10, v8, a0
 ; CHECK-V-NEXT:    vsetvli zero, zero, e32, m1, ta, ma
 ; CHECK-V-NEXT:    vnsrl.wi v8, v10, 0
@@ -626,24 +626,24 @@ define <4 x i32> @utesth_f16i32(<4 x half> %x) {
 ; CHECK-V-NEXT:    sd a0, 8(sp)
 ; CHECK-V-NEXT:    addi a0, sp, 24
 ; CHECK-V-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-V-NEXT:    vle64.v v8, (a0)
-; CHECK-V-NEXT:    mv a0, sp
 ; CHECK-V-NEXT:    vle64.v v10, (a0)
+; CHECK-V-NEXT:    mv a0, sp
+; CHECK-V-NEXT:    vle64.v v8, (a0)
 ; CHECK-V-NEXT:    vsetivli zero, 2, e64, m2, tu, ma
-; CHECK-V-NEXT:    vslideup.vi v10, v8, 1
+; CHECK-V-NEXT:    vslideup.vi v8, v10, 1
 ; CHECK-V-NEXT:    addi a0, sp, 16
 ; CHECK-V-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-V-NEXT:    vle64.v v8, (a0)
+; CHECK-V-NEXT:    vle64.v v10, (a0)
 ; CHECK-V-NEXT:    vsetivli zero, 3, e64, m2, tu, ma
-; CHECK-V-NEXT:    vslideup.vi v10, v8, 2
+; CHECK-V-NEXT:    vslideup.vi v8, v10, 2
 ; CHECK-V-NEXT:    addi a0, sp, 8
 ; CHECK-V-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-V-NEXT:    vle64.v v8, (a0)
+; CHECK-V-NEXT:    vle64.v v10, (a0)
 ; CHECK-V-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
-; CHECK-V-NEXT:    vslideup.vi v10, v8, 3
+; CHECK-V-NEXT:    vslideup.vi v8, v10, 3
 ; CHECK-V-NEXT:    li a0, -1
 ; CHECK-V-NEXT:    srli a0, a0, 32
-; CHECK-V-NEXT:    vminu.vx v10, v10, a0
+; CHECK-V-NEXT:    vminu.vx v10, v8, a0
 ; CHECK-V-NEXT:    vsetvli zero, zero, e32, m1, ta, ma
 ; CHECK-V-NEXT:    vnsrl.wi v8, v10, 0
 ; CHECK-V-NEXT:    ld ra, 56(sp) # 8-byte Folded Reload
@@ -785,24 +785,24 @@ define <4 x i32> @ustest_f16i32(<4 x half> %x) {
 ; CHECK-V-NEXT:    sd a0, 8(sp)
 ; CHECK-V-NEXT:    addi a0, sp, 24
 ; CHECK-V-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-V-NEXT:    vle64.v v8, (a0)
-; CHECK-V-NEXT:    mv a0, sp
 ; CHECK-V-NEXT:    vle64.v v10, (a0)
+; CHECK-V-NEXT:    mv a0, sp
+; CHECK-V-NEXT:    vle64.v v8, (a0)
 ; CHECK-V-NEXT:    vsetivli zero, 2, e64, m2, tu, ma
-; CHECK-V-NEXT:    vslideup.vi v10, v8, 1
+; CHECK-V-NEXT:    vslideup.vi v8, v10, 1
 ; CHECK-V-NEXT:    addi a0, sp, 16
 ; CHECK-V-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-V-NEXT:    vle64.v v8, (a0)
+; CHECK-V-NEXT:    vle64.v v10, (a0)
 ; CHECK-V-NEXT:    vsetivli zero, 3, e64, m2, tu, ma
-; CHECK-V-NEXT:    vslideup.vi v10, v8, 2
+; CHECK-V-NEXT:    vslideup.vi v8, v10, 2
 ; CHECK-V-NEXT:    addi a0, sp, 8
 ; CHECK-V-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-V-NEXT:    vle64.v v8, (a0)
+; CHECK-V-NEXT:    vle64.v v10, (a0)
 ; CHECK-V-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
-; CHECK-V-NEXT:    vslideup.vi v10, v8, 3
+; CHECK-V-NEXT:    vslideup.vi v8, v10, 3
 ; CHECK-V-NEXT:    li a0, -1
 ; CHECK-V-NEXT:    srli a0, a0, 32
-; CHECK-V-NEXT:    vmin.vx v8, v10, a0
+; CHECK-V-NEXT:    vmin.vx v8, v8, a0
 ; CHECK-V-NEXT:    vmax.vx v10, v8, zero
 ; CHECK-V-NEXT:    vsetvli zero, zero, e32, m1, ta, ma
 ; CHECK-V-NEXT:    vnsrl.wi v8, v10, 0
@@ -3324,7 +3324,7 @@ define <2 x i32> @stest_f64i32_mm(<2 x double> %x) {
 ; CHECK-NOV-NEXT:    blt a0, a3, .LBB27_2
 ; CHECK-NOV-NEXT:  .LBB27_6: # %entry
 ; CHECK-NOV-NEXT:    mv a0, a3
-; CHECK-NOV-NEXT:    blt a2, a0, .LBB27_3
+; CHECK-NOV-NEXT:    blt a2, a3, .LBB27_3
 ; CHECK-NOV-NEXT:  .LBB27_7: # %entry
 ; CHECK-NOV-NEXT:    lui a0, 524288
 ; CHECK-NOV-NEXT:    blt a2, a1, .LBB27_4
@@ -3474,7 +3474,7 @@ define <4 x i32> @stest_f32i32_mm(<4 x float> %x) {
 ; CHECK-NOV-NEXT:    blt a5, a6, .LBB30_4
 ; CHECK-NOV-NEXT:  .LBB30_13: # %entry
 ; CHECK-NOV-NEXT:    mv a5, a6
-; CHECK-NOV-NEXT:    blt a3, a5, .LBB30_5
+; CHECK-NOV-NEXT:    blt a3, a6, .LBB30_5
 ; CHECK-NOV-NEXT:  .LBB30_14: # %entry
 ; CHECK-NOV-NEXT:    lui a5, 524288
 ; CHECK-NOV-NEXT:    blt a3, a4, .LBB30_6
@@ -3718,7 +3718,7 @@ define <4 x i32> @stest_f16i32_mm(<4 x half> %x) {
 ; CHECK-NOV-NEXT:    blt a3, a4, .LBB33_4
 ; CHECK-NOV-NEXT:  .LBB33_13: # %entry
 ; CHECK-NOV-NEXT:    mv a3, a4
-; CHECK-NOV-NEXT:    blt a1, a3, .LBB33_5
+; CHECK-NOV-NEXT:    blt a1, a4, .LBB33_5
 ; CHECK-NOV-NEXT:  .LBB33_14: # %entry
 ; CHECK-NOV-NEXT:    lui a3, 524288
 ; CHECK-NOV-NEXT:    blt a1, a2, .LBB33_6
@@ -3763,24 +3763,24 @@ define <4 x i32> @stest_f16i32_mm(<4 x half> %x) {
 ; CHECK-V-NEXT:    sd a0, 8(sp)
 ; CHECK-V-NEXT:    addi a0, sp, 24
 ; CHECK-V-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-V-NEXT:    vle64.v v8, (a0)
-; CHECK-V-NEXT:    mv a0, sp
 ; CHECK-V-NEXT:    vle64.v v10, (a0)
+; CHECK-V-NEXT:    mv a0, sp
+; CHECK-V-NEXT:    vle64.v v8, (a0)
 ; CHECK-V-NEXT:    vsetivli zero, 2, e64, m2, tu, ma
-; CHECK-V-NEXT:    vslideup.vi v10, v8, 1
+; CHECK-V-NEXT:    vslideup.vi v8, v10, 1
 ; CHECK-V-NEXT:    addi a0, sp, 16
 ; CHECK-V-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-V-NEXT:    vle64.v v8, (a0)
+; CHECK-V-NEXT:    vle64.v v10, (a0)
 ; CHECK-V-NEXT:    vsetivli zero, 3, e64, m2, tu, ma
-; CHECK-V-NEXT:    vslideup.vi v10, v8, 2
+; CHECK-V-NEXT:    vslideup.vi v8, v10, 2
 ; CHECK-V-NEXT:    addi a0, sp, 8
 ; CHECK-V-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-V-NEXT:    vle64.v v8, (a0)
+; CHECK-V-NEXT:    vle64.v v10, (a0)
 ; CHECK-V-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
-; CHECK-V-NEXT:    vslideup.vi v10, v8, 3
+; CHECK-V-NEXT:    vslideup.vi v8, v10, 3
 ; CHECK-V-NEXT:    lui a0, 524288
 ; CHECK-V-NEXT:    addiw a1, a0, -1
-; CHECK-V-NEXT:    vmin.vx v8, v10, a1
+; CHECK-V-NEXT:    vmin.vx v8, v8, a1
 ; CHECK-V-NEXT:    vmax.vx v10, v8, a0
 ; CHECK-V-NEXT:    vsetvli zero, zero, e32, m1, ta, ma
 ; CHECK-V-NEXT:    vnsrl.wi v8, v10, 0
@@ -3911,24 +3911,24 @@ define <4 x i32> @utesth_f16i32_mm(<4 x half> %x) {
 ; CHECK-V-NEXT:    sd a0, 8(sp)
 ; CHECK-V-NEXT:    addi a0, sp, 24
 ; CHECK-V-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-V-NEXT:    vle64.v v8, (a0)
-; CHECK-V-NEXT:    mv a0, sp
 ; CHECK-V-NEXT:    vle64.v v10, (a0)
+; CHECK-V-NEXT:    mv a0, sp
+; CHECK-V-NEXT:    vle64.v v8, (a0)
 ; CHECK-V-NEXT:    vsetivli zero, 2, e64, m2, tu, ma
-; CHECK-V-NEXT:    vslideup.vi v10, v8, 1
+; CHECK-V-NEXT:    vslideup.vi v8, v10, 1
 ; CHECK-V-NEXT:    addi a0, sp, 16
 ; CHECK-V-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-V-NEXT:    vle64.v v8, (a0)
+; CHECK-V-NEXT:    vle64.v v10, (a0)
 ; CHECK-V-NEXT:    vsetivli zero, 3, e64, m2, tu, ma
-; CHECK-V-NEXT:    vslideup.vi v10, v8, 2
+; CHECK-V-NEXT:    vslideup.vi v8, v10, 2
 ; CHECK-V-NEXT:    addi a0, sp, 8
 ; CHECK-V-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-V-NEXT:    vle64.v v8, (a0)
+; CHECK-V-NEXT:    vle64.v v10, (a0)
 ; CHECK-V-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
-; CHECK-V-NEXT:    vslideup.vi v10, v8, 3
+; CHECK-V-NEXT:    vslideup.vi v8, v10, 3
 ; CHECK-V-NEXT:    li a0, -1
 ; CHECK-V-NEXT:    srli a0, a0, 32
-; CHECK-V-NEXT:    vminu.vx v10, v10, a0
+; CHECK-V-NEXT:    vminu.vx v10, v8, a0
 ; CHECK-V-NEXT:    vsetvli zero, zero, e32, m1, ta, ma
 ; CHECK-V-NEXT:    vnsrl.wi v8, v10, 0
 ; CHECK-V-NEXT:    ld ra, 56(sp) # 8-byte Folded Reload
@@ -4069,24 +4069,24 @@ define <4 x i32> @ustest_f16i32_mm(<4 x half> %x) {
 ; CHECK-V-NEXT:    sd a0, 8(sp)
 ; CHECK-V-NEXT:    addi a0, sp, 24
 ; CHECK-V-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-V-NEXT:    vle64.v v8, (a0)
-; CHECK-V-NEXT:    mv a0, sp
 ; CHECK-V-NEXT:    vle64.v v10, (a0)
+; CHECK-V-NEXT:    mv a0, sp
+; CHECK-V-NEXT:    vle64.v v8, (a0)
 ; CHECK-V-NEXT:    vsetivli zero, 2, e64, m2, tu, ma
-; CHECK-V-NEXT:    vslideup.vi v10, v8, 1
+; CHECK-V-NEXT:    vslideup.vi v8, v10, 1
 ; CHECK-V-NEXT:    addi a0, sp, 16
 ; CHECK-V-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-V-NEXT:    vle64.v v8, (a0)
+; CHECK-V-NEXT:    vle64.v v10, (a0)
 ; CHECK-V-NEXT:    vsetivli zero, 3, e64, m2, tu, ma
-; CHECK-V-NEXT:    vslideup.vi v10, v8, 2
+; CHECK-V-NEXT:    vslideup.vi v8, v10, 2
 ; CHECK-V-NEXT:    addi a0, sp, 8
 ; CHECK-V-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-V-NEXT:    vle64.v v8, (a0)
+; CHECK-V-NEXT:    vle64.v v10, (a0)
 ; CHECK-V-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
-; CHECK-V-NEXT:    vslideup.vi v10, v8, 3
+; CHECK-V-NEXT:    vslideup.vi v8, v10, 3
 ; CHECK-V-NEXT:    li a0, -1
 ; CHECK-V-NEXT:    srli a0, a0, 32
-; CHECK-V-NEXT:    vmin.vx v8, v10, a0
+; CHECK-V-NEXT:    vmin.vx v8, v8, a0
 ; CHECK-V-NEXT:    vmax.vx v10, v8, zero
 ; CHECK-V-NEXT:    vsetvli zero, zero, e32, m1, ta, ma
 ; CHECK-V-NEXT:    vnsrl.wi v8, v10, 0
diff --git a/llvm/test/CodeGen/RISCV/half-convert-strict.ll b/llvm/test/CodeGen/RISCV/half-convert-strict.ll
index 2fe03a0110218..296db8d1beef5 100644
--- a/llvm/test/CodeGen/RISCV/half-convert-strict.ll
+++ b/llvm/test/CodeGen/RISCV/half-convert-strict.ll
@@ -173,48 +173,38 @@ define i32 @fcvt_wu_h_multiple_use(half %x, ptr %y) {
 ; CHECKIZFH-LABEL: fcvt_wu_h_multiple_use:
 ; CHECKIZFH:       # %bb.0:
 ; CHECKIZFH-NEXT:    fcvt.wu.h a0, fa0, rtz
-; CHECKIZFH-NEXT:    bnez a0, .LBB4_2
-; CHECKIZFH-NEXT:  # %bb.1:
-; CHECKIZFH-NEXT:    li a0, 1
-; CHECKIZFH-NEXT:  .LBB4_2:
+; CHECKIZFH-NEXT:    seqz a1, a0
+; CHECKIZFH-NEXT:    add a0, a0, a1
 ; CHECKIZFH-NEXT:    ret
 ;
 ; RV32IDZFH-LABEL: fcvt_wu_h_multiple_use:
 ; RV32IDZFH:       # %bb.0:
 ; RV32IDZFH-NEXT:    fcvt.wu.h a0, fa0, rtz
-; RV32IDZFH-NEXT:    bnez a0, .LBB4_2
-; RV32IDZFH-NEXT:  # %bb.1:
-; RV32IDZFH-NEXT:    li a0, 1
-; RV32IDZFH-NEXT:  .LBB4_2:
+; RV32IDZFH-NEXT:    seqz a1, a0
+; RV32IDZFH-NEXT:    add a0, a0, a1
 ; RV32IDZFH-NEXT:    ret
 ;
 ; RV64IDZFH-LABEL: fcvt_wu_h_multiple_use:
 ; RV64IDZFH:       # %bb.0:
 ; RV64IDZFH-NEXT:    fcvt.wu.h a0, fa0, rtz
-; RV64IDZFH-NEXT:    bnez a0, .LBB4_2
-; RV64IDZFH-NEXT:  # %bb.1:
-; RV64IDZFH-NEXT:    li a0, 1
-; RV64IDZFH-NEXT:  .LBB4_2:
+; RV64IDZFH-NEXT:    seqz a1, a0
+; RV64IDZFH-NEXT:    add a0, a0, a1
 ; RV64IDZFH-NEXT:    ret
 ;
 ; CHECK32-IZFHMIN-LABEL: fcvt_wu_h_multiple_use:
 ; CHECK32-IZFHMIN:       # %bb.0:
 ; CHECK32-IZFHMIN-NEXT:    fcvt.s.h ft0, fa0
 ; CHECK32-IZFHMIN-NEXT:    fcvt.wu.s a0, ft0, rtz
-; CHECK32-IZFHMIN-NEXT:    bnez a0, .LBB4_2
-; CHECK32-IZFHMIN-NEXT:  # %bb.1:
-; CHECK32-IZFHMIN-NEXT:    li a0, 1
-; CHECK32-IZFHMIN-NEXT:  .LBB4_2:
+; CHECK32-IZFHMIN-NEXT:    seqz a1, a0
+; CHECK32-IZFHMIN-NEXT:    add a0, a0, a1
 ; CHECK32-IZFHMIN-NEXT:    ret
 ;
 ; CHECK64-IZFHMIN-LABEL: fcvt_wu_h_multiple_use:
 ; CHECK64-IZFHMIN:       # %bb.0:
 ; CHECK64-IZFHMIN-NEXT:    fcvt.s.h ft0, fa0
 ; CHECK64-IZFHMIN-NEXT:    fcvt.wu.s a0, ft0, rtz
-; CHECK64-IZFHMIN-NEXT:    bnez a0, .LBB4_2
-; CHECK64-IZFHMIN-NEXT:  # %bb.1:
-; CHECK64-IZFHMIN-NEXT:    li a0, 1
-; CHECK64-IZFHMIN-NEXT:  .LBB4_2:
+; CHECK64-IZFHMIN-NEXT:    seqz a1, a0
+; CHECK64-IZFHMIN-NEXT:    add a0, a0, a1
 ; CHECK64-IZFHMIN-NEXT:    ret
   %a = call i32 @llvm.experimental.constrained.fptoui.i32.f16(half %x, metadata !"fpexcept.strict") strictfp
   %b = icmp eq i32 %a, 0
diff --git a/llvm/test/CodeGen/RISCV/half-convert.ll b/llvm/test/CodeGen/RISCV/half-convert.ll
index fcba34a732e55..e28f0cc454839 100644
--- a/llvm/test/CodeGen/RISCV/half-convert.ll
+++ b/llvm/test/CodeGen/RISCV/half-convert.ll
@@ -735,28 +735,22 @@ define i32 @fcvt_wu_h_multiple_use(half %x, ptr %y) nounwind {
 ; CHECKIZFH-LABEL: fcvt_wu_h_multiple_use:
 ; CHECKIZFH:       # %bb.0:
 ; CHECKIZFH-NEXT:    fcvt.wu.h a0, fa0, rtz
-; CHECKIZFH-NEXT:    bnez a0, .LBB7_2
-; CHECKIZFH-NEXT:  # %bb.1:
-; CHECKIZFH-NEXT:    li a0, 1
-; CHECKIZFH-NEXT:  .LBB7_2:
+; CHECKIZFH-NEXT:    seqz a1, a0
+; CHECKIZFH-NEXT:    add a0, a0, a1
 ; CHECKIZFH-NEXT:    ret
 ;
 ; RV32IDZFH-LABEL: fcvt_wu_h_multiple_use:
 ; RV32IDZFH:       # %bb.0:
 ; RV32IDZFH-NEXT:    fcvt.wu.h a0, fa0, rtz
-; RV32IDZFH-NEXT:    bnez a0, .LBB7_2
-; RV32IDZFH-NEXT:  # %bb.1:
-; RV32IDZFH-NEXT:    li a0, 1
-; RV32IDZFH-NEXT:  .LBB7_2:
+; RV32IDZFH-NEXT:    seqz a1, a0
+; RV32IDZFH-NEXT:    add a0, a0, a1
 ; RV32IDZFH-NEXT:    ret
 ;
 ; RV64IDZFH-LABEL: fcvt_wu_h_multiple_use:
 ; RV64IDZFH:       # %bb.0:
 ; RV64IDZFH-NEXT:    fcvt.wu.h a0, fa0, rtz
-; RV64IDZFH-NEXT:    bnez a0, .LBB7_2
-; RV64IDZFH-NEXT:  # %bb.1:
-; RV64IDZFH-NEXT:    li a0, 1
-; RV64IDZFH-NEXT:  .LBB7_2:
+; RV64IDZFH-NEXT:    seqz a1, a0
+; RV64IDZFH-NEXT:    add a0, a0, a1
 ; RV64IDZFH-NEXT:    ret
 ;
 ; RV32I-LABEL: fcvt_wu_h_multiple_use:
@@ -767,10 +761,8 @@ define i32 @fcvt_wu_h_multiple_use(half %x, ptr %y) nounwind {
 ; RV32I-NEXT:    srli a0, a0, 16
 ; RV32I-NEXT:    call __extendhfsf2@plt
 ; RV32I-NEXT:    call __fixunssfsi@plt
-; RV32I-NEXT:    bnez a0, .LBB7_2
-; RV32I-NEXT:  # %bb.1:
-; RV32I-NEXT:    li a0, 1
-; RV32I-NEXT:  .LBB7_2:
+; RV32I-NEXT:    seqz a1, a0
+; RV32I-NEXT:    add a0, a0, a1
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
@@ -783,10 +775,8 @@ define i32 @fcvt_wu_h_multiple_use(half %x, ptr %y) nounwind {
 ; RV64I-NEXT:    srli a0, a0, 48
 ; RV64I-NEXT:    call __extendhfsf2@plt
 ; RV64I-NEXT:    call __fixunssfdi@plt
-; RV64I-NEXT:    bnez a0, .LBB7_2
-; RV64I-NEXT:  # %bb.1:
-; RV64I-NEXT:    li a0, 1
-; RV64I-NEXT:  .LBB7_2:
+; RV64I-NEXT:    seqz a1, a0
+; RV64I-NEXT:    add a0, a0, a1
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
@@ -795,20 +785,16 @@ define i32 @fcvt_wu_h_multiple_use(half %x, ptr %y) nounwind {
 ; CHECK32-IZFHMIN:       # %bb.0:
 ; CHECK32-IZFHMIN-NEXT:    fcvt.s.h ft0, fa0
 ; CHECK32-IZFHMIN-NEXT:    fcvt.wu.s a0, ft0, rtz
-; CHECK32-IZFHMIN-NEXT:    bnez a0, .LBB7_2
-; CHECK32-IZFHMIN-NEXT:  # %bb.1:
-; CHECK32-IZFHMIN-NEXT:    li a0, 1
-; CHECK32-IZFHMIN-NEXT:  .LBB7_2:
+; CHECK32-IZFHMIN-NEXT:    seqz a1, a0
+; CHECK32-IZFHMIN-NEXT:    add a0, a0, a1
 ; CHECK32-IZFHMIN-NEXT:    ret
 ;
 ; CHECK64-IZFHMIN-LABEL: fcvt_wu_h_multiple_use:
 ; CHECK64-IZFHMIN:       # %bb.0:
 ; CHECK64-IZFHMIN-NEXT:    fcvt.s.h ft0, fa0
 ; CHECK64-IZFHMIN-NEXT:    fcvt.wu.s a0, ft0, rtz
-; CHECK64-IZFHMIN-NEXT:    bnez a0, .LBB7_2
-; CHECK64-IZFHMIN-NEXT:  # %bb.1:
-; CHECK64-IZFHMIN-NEXT:    li a0, 1
-; CHECK64-IZFHMIN-NEXT:  .LBB7_2:
+; CHECK64-IZFHMIN-NEXT:    seqz a1, a0
+; CHECK64-IZFHMIN-NEXT:    add a0, a0, a1
 ; CHECK64-IZFHMIN-NEXT:    ret
   %a = fptoui half %x to i32
   %b = icmp eq i32 %a, 0
diff --git a/llvm/test/CodeGen/RISCV/half-zfa-fli.ll b/llvm/test/CodeGen/RISCV/half-zfa-fli.ll
new file mode 100644
index 0000000000000..f82ae5c00d159
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/half-zfa-fli.ll
@@ -0,0 +1,182 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 -target-abi ilp32f -mattr=+experimental-zfa,+zfh < %s \
+; RUN:     | FileCheck %s
+; RUN: llc -mtriple=riscv64 -target-abi lp64f -mattr=+experimental-zfa,+zfh < %s \
+; RUN:     | FileCheck %s
+; RUN: llc -mtriple=riscv32 -target-abi ilp32f -mattr=+experimental-zfa,+zfhmin < %s \
+; RUN:     | FileCheck %s --check-prefix=ZFHMIN
+; RUN: llc -mtriple=riscv64 -target-abi lp64f -mattr=+experimental-zfa,+zfhmin < %s \
+; RUN:     | FileCheck %s --check-prefix=ZFHMIN
+
+define half @loadfpimm1() {
+; CHECK-LABEL: loadfpimm1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.h fa0, 0.0625
+; CHECK-NEXT:    ret
+;
+; ZFHMIN-LABEL: loadfpimm1:
+; ZFHMIN:       # %bb.0:
+; ZFHMIN-NEXT:    lui a0, %hi(.LCPI0_0)
+; ZFHMIN-NEXT:    flh fa0, %lo(.LCPI0_0)(a0)
+; ZFHMIN-NEXT:    ret
+  ret half 0.0625
+}
+
+define half @loadfpimm2() {
+; CHECK-LABEL: loadfpimm2:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.h fa0, 0.75
+; CHECK-NEXT:    ret
+;
+; ZFHMIN-LABEL: loadfpimm2:
+; ZFHMIN:       # %bb.0:
+; ZFHMIN-NEXT:    lui a0, %hi(.LCPI1_0)
+; ZFHMIN-NEXT:    flh fa0, %lo(.LCPI1_0)(a0)
+; ZFHMIN-NEXT:    ret
+  ret half 0.75
+}
+
+define half @loadfpimm3() {
+; CHECK-LABEL: loadfpimm3:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.h fa0, 1.25
+; CHECK-NEXT:    ret
+;
+; ZFHMIN-LABEL: loadfpimm3:
+; ZFHMIN:       # %bb.0:
+; ZFHMIN-NEXT:    lui a0, %hi(.LCPI2_0)
+; ZFHMIN-NEXT:    flh fa0, %lo(.LCPI2_0)(a0)
+; ZFHMIN-NEXT:    ret
+  ret half 1.25
+}
+
+define half @loadfpimm4() {
+; CHECK-LABEL: loadfpimm4:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.h fa0, 3.0
+; CHECK-NEXT:    ret
+;
+; ZFHMIN-LABEL: loadfpimm4:
+; ZFHMIN:       # %bb.0:
+; ZFHMIN-NEXT:    lui a0, %hi(.LCPI3_0)
+; ZFHMIN-NEXT:    flh fa0, %lo(.LCPI3_0)(a0)
+; ZFHMIN-NEXT:    ret
+  ret half 3.0
+}
+
+define half @loadfpimm5() {
+; CHECK-LABEL: loadfpimm5:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.h fa0, 256.0
+; CHECK-NEXT:    ret
+;
+; ZFHMIN-LABEL: loadfpimm5:
+; ZFHMIN:       # %bb.0:
+; ZFHMIN-NEXT:    lui a0, %hi(.LCPI4_0)
+; ZFHMIN-NEXT:    flh fa0, %lo(.LCPI4_0)(a0)
+; ZFHMIN-NEXT:    ret
+  ret half 256.0
+}
+
+define half @loadfpimm6() {
+; CHECK-LABEL: loadfpimm6:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.h fa0, inf
+; CHECK-NEXT:    ret
+;
+; ZFHMIN-LABEL: loadfpimm6:
+; ZFHMIN:       # %bb.0:
+; ZFHMIN-NEXT:    lui a0, %hi(.LCPI5_0)
+; ZFHMIN-NEXT:    flh fa0, %lo(.LCPI5_0)(a0)
+; ZFHMIN-NEXT:    ret
+  ret half 0xH7C00
+}
+
+define half @loadfpimm7() {
+; CHECK-LABEL: loadfpimm7:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.h fa0, nan
+; CHECK-NEXT:    ret
+;
+; ZFHMIN-LABEL: loadfpimm7:
+; ZFHMIN:       # %bb.0:
+; ZFHMIN-NEXT:    lui a0, %hi(.LCPI6_0)
+; ZFHMIN-NEXT:    flh fa0, %lo(.LCPI6_0)(a0)
+; ZFHMIN-NEXT:    ret
+  ret half 0xH7E00
+}
+
+define half @loadfpimm8() {
+; CHECK-LABEL: loadfpimm8:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.h fa0, min
+; CHECK-NEXT:    ret
+;
+; ZFHMIN-LABEL: loadfpimm8:
+; ZFHMIN:       # %bb.0:
+; ZFHMIN-NEXT:    li a0, 1024
+; ZFHMIN-NEXT:    fmv.h.x fa0, a0
+; ZFHMIN-NEXT:    ret
+  ret half 0xH0400
+}
+
+define half @loadfpimm9() {
+; CHECK-LABEL: loadfpimm9:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    lui a0, %hi(.LCPI8_0)
+; CHECK-NEXT:    flh fa0, %lo(.LCPI8_0)(a0)
+; CHECK-NEXT:    ret
+;
+; ZFHMIN-LABEL: loadfpimm9:
+; ZFHMIN:       # %bb.0:
+; ZFHMIN-NEXT:    lui a0, %hi(.LCPI8_0)
+; ZFHMIN-NEXT:    flh fa0, %lo(.LCPI8_0)(a0)
+; ZFHMIN-NEXT:    ret
+  ret half 255.0
+}
+
+; This is 1 * 2^-16
+define half @loadfpimm10() {
+; CHECK-LABEL: loadfpimm10:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.h fa0, 1.52587890625e-05
+; CHECK-NEXT:    ret
+;
+; ZFHMIN-LABEL: loadfpimm10:
+; ZFHMIN:       # %bb.0:
+; ZFHMIN-NEXT:    li a0, 256
+; ZFHMIN-NEXT:    fmv.h.x fa0, a0
+; ZFHMIN-NEXT:    ret
+  ret half 0xH0100
+}
+
+; This is 1 * 2^-15
+define half @loadfpimm11() {
+; CHECK-LABEL: loadfpimm11:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.h fa0, 3.0517578125e-05
+; CHECK-NEXT:    ret
+;
+; ZFHMIN-LABEL: loadfpimm11:
+; ZFHMIN:       # %bb.0:
+; ZFHMIN-NEXT:    li a0, 512
+; ZFHMIN-NEXT:    fmv.h.x fa0, a0
+; ZFHMIN-NEXT:    ret
+  ret half 0xH0200
+}
+
+; Negative test. This is an snan with payload of 1.
+define half @loadfpimm12() {
+; CHECK-LABEL: loadfpimm12:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    lui a0, %hi(.LCPI11_0)
+; CHECK-NEXT:    flh fa0, %lo(.LCPI11_0)(a0)
+; CHECK-NEXT:    ret
+;
+; ZFHMIN-LABEL: loadfpimm12:
+; ZFHMIN:       # %bb.0:
+; ZFHMIN-NEXT:    lui a0, %hi(.LCPI11_0)
+; ZFHMIN-NEXT:    flh fa0, %lo(.LCPI11_0)(a0)
+; ZFHMIN-NEXT:    ret
+  ret half 0xH7c01
+}
diff --git a/llvm/test/CodeGen/RISCV/inline-asm.ll b/llvm/test/CodeGen/RISCV/inline-asm.ll
index 4dfbb62c0eb88..22d382517ccdf 100644
--- a/llvm/test/CodeGen/RISCV/inline-asm.ll
+++ b/llvm/test/CodeGen/RISCV/inline-asm.ll
@@ -82,6 +82,27 @@ define i32 @constraint_m2(ptr %a) nounwind {
   ret i32 %1
 }
 
+define i32 @constraint_m_with_offset(ptr %a) nounwind {
+; RV32I-LABEL: constraint_m_with_offset:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, a0, 4
+; RV32I-NEXT:    #APP
+; RV32I-NEXT:    lw a0, 0(a0)
+; RV32I-NEXT:    #NO_APP
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: constraint_m_with_offset:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, a0, 4
+; RV64I-NEXT:    #APP
+; RV64I-NEXT:    lw a0, 0(a0)
+; RV64I-NEXT:    #NO_APP
+; RV64I-NEXT:    ret
+  %1 = getelementptr i32, ptr %a, i32 1
+  %2 = tail call i32 asm "lw $0, $1", "=r,*m"(ptr elementtype(i32) %1)
+  ret i32 %2
+}
+
 define void @constraint_I() nounwind {
 ; RV32I-LABEL: constraint_I:
 ; RV32I:       # %bb.0:
@@ -175,6 +196,27 @@ define void @constraint_A(ptr %a) nounwind {
   ret void
 }
 
+define i32 @constraint_A_with_offset(ptr %a) nounwind {
+; RV32I-LABEL: constraint_A_with_offset:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, a0, 4
+; RV32I-NEXT:    #APP
+; RV32I-NEXT:    lw a0, 0(a0)
+; RV32I-NEXT:    #NO_APP
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: constraint_A_with_offset:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, a0, 4
+; RV64I-NEXT:    #APP
+; RV64I-NEXT:    lw a0, 0(a0)
+; RV64I-NEXT:    #NO_APP
+; RV64I-NEXT:    ret
+  %1 = getelementptr i32, ptr %a, i32 1
+  %2 = tail call i32 asm "lw $0, $1", "=r,*A"(ptr elementtype(i32) %1)
+  ret i32 %2
+}
+
 define i32 @modifier_z_zero(i32 %a) nounwind {
 ; RV32I-LABEL: modifier_z_zero:
 ; RV32I:       # %bb.0:
diff --git a/llvm/test/CodeGen/RISCV/machine-combiner.ll b/llvm/test/CodeGen/RISCV/machine-combiner.ll
index 1cb57ff307310..197abcfb0e1e1 100644
--- a/llvm/test/CodeGen/RISCV/machine-combiner.ll
+++ b/llvm/test/CodeGen/RISCV/machine-combiner.ll
@@ -713,8 +713,8 @@ define i16 @test_reassoc_mul_i16(i16 %a0, i16 %a1, i16 %a2, i16 %a3) {
 define i32 @test_reassoc_mul_i32(i32 %a0, i32 %a1, i32 %a2, i32 %a3) {
 ; CHECK-LABEL: test_reassoc_mul_i32:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    mulw a0, a0, a1
-; CHECK-NEXT:    mulw a1, a2, a3
+; CHECK-NEXT:    mul a0, a0, a1
+; CHECK-NEXT:    mul a1, a2, a3
 ; CHECK-NEXT:    mulw a0, a0, a1
 ; CHECK-NEXT:    ret
   %t0 = mul i32 %a0, %a1
diff --git a/llvm/test/CodeGen/RISCV/machine-outliner-and-machine-copy-propagation.ll b/llvm/test/CodeGen/RISCV/machine-outliner-and-machine-copy-propagation.ll
new file mode 100644
index 0000000000000..45c582a613308
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/machine-outliner-and-machine-copy-propagation.ll
@@ -0,0 +1,198 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --include-generated-funcs --version 2
+; RUN: llc -mtriple=riscv64 -riscv-enable-copy-propagation -enable-machine-outliner -verify-machineinstrs < %s \
+; RUN:   | FileCheck %s -check-prefix=RV64I
+
+; The outlined function produced by this test case contains a register to
+; register copy, which is at risk of being removed by MachineCopyPropagation
+; if it can't see that the copy is used. At the time of writing this test
+; case, MCP will remove the copy if it is run after the machine outliner.
+
+define signext i32 @nge(i32 signext %a, i32 signext %b) nounwind {
+entry:
+  %cmp = icmp sge i32 %a, %b
+  %conv.neg = sext i1 %cmp to i32
+  ret i32 %conv.neg
+}
+
+define signext i32 @ngt(i32 signext %a, i32 signext %b) nounwind {
+entry:
+  %cmp = icmp sgt i32 %a, %b
+  %conv.neg = sext i1 %cmp to i32
+  ret i32 %conv.neg
+}
+
+define signext i32 @nle(i32 signext %a, i32 signext %b) nounwind {
+entry:
+  %cmp = icmp sle i32 %a, %b
+  %conv.neg = sext i1 %cmp to i32
+  ret i32 %conv.neg
+}
+
+define signext i32 @nlt(i32 signext %a, i32 signext %b) nounwind {
+entry:
+  %cmp = icmp slt i32 %a, %b
+  %conv.neg = sext i1 %cmp to i32
+  ret i32 %conv.neg
+}
+
+define signext i32 @main() nounwind {
+entry:
+  %call = tail call signext i32 @nge(i32 signext -2147483648, i32 signext 2147483647)
+  %cmp.not = icmp eq i32 %call, 0
+  br i1 %cmp.not, label %if.end, label %if.then
+
+if.then:
+  tail call void @abort()
+  unreachable
+
+if.end:
+  %call1 = tail call signext i32 @nge(i32 signext 2147483647, i32 signext -2147483648)
+  %cmp2.not = icmp eq i32 %call1, -1
+  br i1 %cmp2.not, label %if.end4, label %if.then3
+
+if.then3:
+  tail call void @abort()
+  unreachable
+
+if.end4:
+  %call5 = tail call signext i32 @ngt(i32 signext -2147483648, i32 signext 2147483647)
+  %cmp6.not = icmp eq i32 %call5, 0
+  br i1 %cmp6.not, label %if.end8, label %if.then7
+
+if.then7:
+  tail call void @abort()
+  unreachable
+
+if.end8:
+  %call9 = tail call signext i32 @ngt(i32 signext 2147483647, i32 signext -2147483648)
+  %cmp10.not = icmp eq i32 %call9, -1
+  br i1 %cmp10.not, label %if.end12, label %if.then11
+
+if.then11:
+  tail call void @abort()
+  unreachable
+
+if.end12:
+  %call13 = tail call signext i32 @nle(i32 signext -2147483648, i32 signext 2147483647)
+  %cmp14.not = icmp eq i32 %call13, -1
+  br i1 %cmp14.not, label %if.end16, label %if.then15
+
+if.then15:
+  tail call void @abort()
+  unreachable
+
+if.end16:
+  %call17 = tail call signext i32 @nle(i32 signext 2147483647, i32 signext -2147483648)
+  %cmp18.not = icmp eq i32 %call17, 0
+  br i1 %cmp18.not, label %if.end20, label %if.then19
+
+if.then19:
+  tail call void @abort()
+  unreachable
+
+if.end20:
+  %call21 = tail call signext i32 @nlt(i32 signext -2147483648, i32 signext 2147483647)
+  %cmp22.not = icmp eq i32 %call21, -1
+  br i1 %cmp22.not, label %if.end24, label %if.then23
+
+if.then23:
+  tail call void @abort()
+  unreachable
+
+if.end24:
+  %call25 = tail call signext i32 @nlt(i32 signext 2147483647, i32 signext -2147483648)
+  %cmp26.not = icmp eq i32 %call25, 0
+  br i1 %cmp26.not, label %if.end28, label %if.then27
+
+if.then27:
+  tail call void @abort()
+  unreachable
+
+if.end28:
+  tail call void @exit(i32 signext 0)
+  unreachable
+}
+
+declare void @abort() noreturn
+declare void @exit(i32 signext) noreturn
+
+; RV64I-LABEL: nge:
+; RV64I:       # %bb.0: # %entry
+; RV64I-NEXT:    slt a0, a0, a1
+; RV64I-NEXT:    addi a0, a0, -1
+; RV64I-NEXT:    ret
+;
+; RV64I-LABEL: ngt:
+; RV64I:       # %bb.0: # %entry
+; RV64I-NEXT:    slt a0, a1, a0
+; RV64I-NEXT:    neg a0, a0
+; RV64I-NEXT:    ret
+;
+; RV64I-LABEL: nle:
+; RV64I:       # %bb.0: # %entry
+; RV64I-NEXT:    slt a0, a1, a0
+; RV64I-NEXT:    addi a0, a0, -1
+; RV64I-NEXT:    ret
+;
+; RV64I-LABEL: nlt:
+; RV64I:       # %bb.0: # %entry
+; RV64I-NEXT:    slt a0, a0, a1
+; RV64I-NEXT:    neg a0, a0
+; RV64I-NEXT:    ret
+;
+; RV64I-LABEL: main:
+; RV64I:       # %bb.0: # %entry
+; RV64I-NEXT:    addi sp, sp, -32
+; RV64I-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
+; RV64I-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
+; RV64I-NEXT:    sd s1, 8(sp) # 8-byte Folded Spill
+; RV64I-NEXT:    call t0, OUTLINED_FUNCTION_0
+; RV64I-NEXT:    call nge@plt
+; RV64I-NEXT:    bnez a0, .LBB4_9
+; RV64I-NEXT:  # %bb.1: # %if.end
+; RV64I-NEXT:    lui a1, 524288
+; RV64I-NEXT:    mv a0, s0
+; RV64I-NEXT:    call nge@plt
+; RV64I-NEXT:    li a1, -1
+; RV64I-NEXT:    bne a0, a1, .LBB4_9
+; RV64I-NEXT:  # %bb.2: # %if.end4
+; RV64I-NEXT:    call t0, OUTLINED_FUNCTION_0
+; RV64I-NEXT:    call ngt@plt
+; RV64I-NEXT:    bnez a0, .LBB4_9
+; RV64I-NEXT:  # %bb.3: # %if.end8
+; RV64I-NEXT:    lui a1, 524288
+; RV64I-NEXT:    mv a0, s0
+; RV64I-NEXT:    call ngt@plt
+; RV64I-NEXT:    li s1, -1
+; RV64I-NEXT:    bne a0, s1, .LBB4_9
+; RV64I-NEXT:  # %bb.4: # %if.end12
+; RV64I-NEXT:    call t0, OUTLINED_FUNCTION_0
+; RV64I-NEXT:    call nle@plt
+; RV64I-NEXT:    bne a0, s1, .LBB4_9
+; RV64I-NEXT:  # %bb.5: # %if.end16
+; RV64I-NEXT:    lui a1, 524288
+; RV64I-NEXT:    mv a0, s0
+; RV64I-NEXT:    call nle@plt
+; RV64I-NEXT:    bnez a0, .LBB4_9
+; RV64I-NEXT:  # %bb.6: # %if.end20
+; RV64I-NEXT:    call t0, OUTLINED_FUNCTION_0
+; RV64I-NEXT:    call nlt@plt
+; RV64I-NEXT:    li a1, -1
+; RV64I-NEXT:    bne a0, a1, .LBB4_9
+; RV64I-NEXT:  # %bb.7: # %if.end24
+; RV64I-NEXT:    lui a1, 524288
+; RV64I-NEXT:    mv a0, s0
+; RV64I-NEXT:    call nlt@plt
+; RV64I-NEXT:    bnez a0, .LBB4_9
+; RV64I-NEXT:  # %bb.8: # %if.end28
+; RV64I-NEXT:    call exit@plt
+; RV64I-NEXT:  .LBB4_9: # %if.then
+; RV64I-NEXT:    call abort@plt
+;
+; RV64I-LABEL: OUTLINED_FUNCTION_0:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    lui s0, 524288
+; RV64I-NEXT:    addiw s0, s0, -1
+; RV64I-NEXT:    lui a0, 524288
+; RV64I-NEXT:    mv a1, s0
+; RV64I-NEXT:    jr t0
diff --git a/llvm/test/CodeGen/RISCV/machine-outliner-throw.ll b/llvm/test/CodeGen/RISCV/machine-outliner-throw.ll
index d5d263a05133c..03419932d030a 100644
--- a/llvm/test/CodeGen/RISCV/machine-outliner-throw.ll
+++ b/llvm/test/CodeGen/RISCV/machine-outliner-throw.ll
@@ -12,7 +12,7 @@ define i32 @func1(i32 %x) #0 {
 ; CHECK-NEXT:    sd s0, 0(sp) # 8-byte Folded Spill
 ; CHECK-NEXT:    .cfi_offset ra, -8
 ; CHECK-NEXT:    .cfi_offset s0, -16
-; CHECK-NEXT:    mulw a0, a0, a0
+; CHECK-NEXT:    mul a0, a0, a0
 ; CHECK-NEXT:    addiw s0, a0, 1
 ; CHECK-NEXT:    li a0, 4
 ; CHECK-NEXT:    call __cxa_allocate_exception@plt
@@ -39,7 +39,7 @@ define i32 @func2(i32 %x) #0 {
 ; CHECK-NEXT:    sd s0, 0(sp) # 8-byte Folded Spill
 ; CHECK-NEXT:    .cfi_offset ra, -8
 ; CHECK-NEXT:    .cfi_offset s0, -16
-; CHECK-NEXT:    mulw a0, a0, a0
+; CHECK-NEXT:    mul a0, a0, a0
 ; CHECK-NEXT:    addiw s0, a0, 1
 ; CHECK-NEXT:    li a0, 4
 ; CHECK-NEXT:    call __cxa_allocate_exception@plt
diff --git a/llvm/test/CodeGen/RISCV/min-max.ll b/llvm/test/CodeGen/RISCV/min-max.ll
index a7ac4a7e2a0cb..4a1039ab8543c 100644
--- a/llvm/test/CodeGen/RISCV/min-max.ll
+++ b/llvm/test/CodeGen/RISCV/min-max.ll
@@ -719,3 +719,51 @@ define i64 @smin_i64_negone(i64 %a) {
   %c = call i64 @llvm.smin.i64(i64 %a, i64 -1)
   ret i64 %c
 }
+
+define i64 @umax_i64_one(i64 %a, i64 %b) {
+; RV32I-LABEL: umax_i64_one:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    mv a2, a0
+; RV32I-NEXT:    beqz a1, .LBB28_3
+; RV32I-NEXT:  # %bb.1:
+; RV32I-NEXT:    beqz a1, .LBB28_4
+; RV32I-NEXT:  .LBB28_2:
+; RV32I-NEXT:    ret
+; RV32I-NEXT:  .LBB28_3:
+; RV32I-NEXT:    li a0, 1
+; RV32I-NEXT:    bnez a1, .LBB28_2
+; RV32I-NEXT:  .LBB28_4:
+; RV32I-NEXT:    seqz a0, a2
+; RV32I-NEXT:    add a0, a2, a0
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: umax_i64_one:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    seqz a1, a0
+; RV64I-NEXT:    add a0, a0, a1
+; RV64I-NEXT:    ret
+;
+; RV32ZBB-LABEL: umax_i64_one:
+; RV32ZBB:       # %bb.0:
+; RV32ZBB-NEXT:    mv a2, a0
+; RV32ZBB-NEXT:    li a3, 1
+; RV32ZBB-NEXT:    beqz a1, .LBB28_3
+; RV32ZBB-NEXT:  # %bb.1:
+; RV32ZBB-NEXT:    beqz a1, .LBB28_4
+; RV32ZBB-NEXT:  .LBB28_2:
+; RV32ZBB-NEXT:    ret
+; RV32ZBB-NEXT:  .LBB28_3:
+; RV32ZBB-NEXT:    li a0, 1
+; RV32ZBB-NEXT:    bnez a1, .LBB28_2
+; RV32ZBB-NEXT:  .LBB28_4:
+; RV32ZBB-NEXT:    maxu a0, a2, a3
+; RV32ZBB-NEXT:    ret
+;
+; RV64ZBB-LABEL: umax_i64_one:
+; RV64ZBB:       # %bb.0:
+; RV64ZBB-NEXT:    li a1, 1
+; RV64ZBB-NEXT:    maxu a0, a0, a1
+; RV64ZBB-NEXT:    ret
+  %c = call i64 @llvm.umax.i64(i64 %a, i64 1)
+  ret i64 %c
+}
diff --git a/llvm/test/CodeGen/RISCV/overflow-intrinsics.ll b/llvm/test/CodeGen/RISCV/overflow-intrinsics.ll
index fea0199c8cfec..543c17f748a45 100644
--- a/llvm/test/CodeGen/RISCV/overflow-intrinsics.ll
+++ b/llvm/test/CodeGen/RISCV/overflow-intrinsics.ll
@@ -543,21 +543,17 @@ define i1 @uaddo_i64_increment(i64 %x, ptr %p) {
 define i1 @uaddo_i8_increment_noncanonical_1(i8 %x, ptr %p) {
 ; RV32-LABEL: uaddo_i8_increment_noncanonical_1:
 ; RV32:       # %bb.0:
-; RV32-NEXT:    andi a0, a0, 255
 ; RV32-NEXT:    addi a2, a0, 1
 ; RV32-NEXT:    andi a0, a2, 255
-; RV32-NEXT:    xor a0, a0, a2
-; RV32-NEXT:    snez a0, a0
+; RV32-NEXT:    seqz a0, a0
 ; RV32-NEXT:    sb a2, 0(a1)
 ; RV32-NEXT:    ret
 ;
 ; RV64-LABEL: uaddo_i8_increment_noncanonical_1:
 ; RV64:       # %bb.0:
-; RV64-NEXT:    andi a0, a0, 255
-; RV64-NEXT:    addi a2, a0, 1
+; RV64-NEXT:    addiw a2, a0, 1
 ; RV64-NEXT:    andi a0, a2, 255
-; RV64-NEXT:    xor a0, a0, a2
-; RV64-NEXT:    snez a0, a0
+; RV64-NEXT:    seqz a0, a0
 ; RV64-NEXT:    sb a2, 0(a1)
 ; RV64-NEXT:    ret
   %a = add i8 1, %x        ; commute
@@ -589,26 +585,20 @@ define i1 @uaddo_i32_increment_noncanonical_2(i32 %x, ptr %p) {
 define i1 @uaddo_i16_increment_noncanonical_3(i16 %x, ptr %p) {
 ; RV32-LABEL: uaddo_i16_increment_noncanonical_3:
 ; RV32:       # %bb.0:
-; RV32-NEXT:    lui a2, 16
-; RV32-NEXT:    addi a2, a2, -1
-; RV32-NEXT:    and a0, a0, a2
-; RV32-NEXT:    addi a3, a0, 1
-; RV32-NEXT:    and a2, a3, a2
-; RV32-NEXT:    xor a2, a2, a3
-; RV32-NEXT:    snez a0, a2
-; RV32-NEXT:    sh a3, 0(a1)
+; RV32-NEXT:    addi a2, a0, 1
+; RV32-NEXT:    slli a0, a2, 16
+; RV32-NEXT:    srli a0, a0, 16
+; RV32-NEXT:    seqz a0, a0
+; RV32-NEXT:    sh a2, 0(a1)
 ; RV32-NEXT:    ret
 ;
 ; RV64-LABEL: uaddo_i16_increment_noncanonical_3:
 ; RV64:       # %bb.0:
-; RV64-NEXT:    lui a2, 16
-; RV64-NEXT:    addiw a2, a2, -1
-; RV64-NEXT:    and a0, a0, a2
-; RV64-NEXT:    addi a3, a0, 1
-; RV64-NEXT:    and a2, a3, a2
-; RV64-NEXT:    xor a2, a2, a3
-; RV64-NEXT:    snez a0, a2
-; RV64-NEXT:    sh a3, 0(a1)
+; RV64-NEXT:    addiw a2, a0, 1
+; RV64-NEXT:    slli a0, a2, 48
+; RV64-NEXT:    srli a0, a0, 48
+; RV64-NEXT:    seqz a0, a0
+; RV64-NEXT:    sh a2, 0(a1)
 ; RV64-NEXT:    ret
   %a = add i16 1, %x        ; commute
   %ov = icmp eq i16 0, %a   ; commute
@@ -1281,3 +1271,39 @@ exit:
   ret void
 }
 
+define i16 @overflow_not_used(i16 %a, i16 %b, ptr %res) {
+; RV32-LABEL: overflow_not_used:
+; RV32:       # %bb.0:
+; RV32-NEXT:    lui a3, 16
+; RV32-NEXT:    addi a3, a3, -1
+; RV32-NEXT:    and a4, a1, a3
+; RV32-NEXT:    add a0, a1, a0
+; RV32-NEXT:    and a3, a0, a3
+; RV32-NEXT:    bltu a3, a4, .LBB37_2
+; RV32-NEXT:  # %bb.1:
+; RV32-NEXT:    li a1, 42
+; RV32-NEXT:  .LBB37_2:
+; RV32-NEXT:    sh a0, 0(a2)
+; RV32-NEXT:    mv a0, a1
+; RV32-NEXT:    ret
+;
+; RV64-LABEL: overflow_not_used:
+; RV64:       # %bb.0:
+; RV64-NEXT:    lui a3, 16
+; RV64-NEXT:    addiw a3, a3, -1
+; RV64-NEXT:    and a4, a1, a3
+; RV64-NEXT:    add a0, a1, a0
+; RV64-NEXT:    and a3, a0, a3
+; RV64-NEXT:    bltu a3, a4, .LBB37_2
+; RV64-NEXT:  # %bb.1:
+; RV64-NEXT:    li a1, 42
+; RV64-NEXT:  .LBB37_2:
+; RV64-NEXT:    sh a0, 0(a2)
+; RV64-NEXT:    mv a0, a1
+; RV64-NEXT:    ret
+  %add = add i16 %b, %a
+  %cmp = icmp ult i16 %add, %b
+  %Q = select i1 %cmp, i16 %b, i16 42
+  store i16 %add, ptr %res
+  ret i16 %Q
+}
diff --git a/llvm/test/CodeGen/RISCV/reserved-regs.ll b/llvm/test/CodeGen/RISCV/reserved-regs.ll
index da549c0ad3033..68c6a6558eccc 100644
--- a/llvm/test/CodeGen/RISCV/reserved-regs.ll
+++ b/llvm/test/CodeGen/RISCV/reserved-regs.ll
@@ -58,6 +58,7 @@
 ; RUN: llc -mtriple=riscv64 -mattr=+reserve-x31 -verify-machineinstrs < %s | FileCheck %s -check-prefix=X31
 
 ; RUN: llc -mtriple=riscv64-fuchsia -verify-machineinstrs < %s | FileCheck %s -check-prefix=X18
+; RUN: llc -mtriple=riscv64-linux-android -verify-machineinstrs < %s | FileCheck %s -check-prefix=X18
 
 ; This program is free to use all registers, but needs a stack pointer for
 ; spill values, so do not test for reserving the stack pointer.
diff --git a/llvm/test/CodeGen/RISCV/rv64i-demanded-bits.ll b/llvm/test/CodeGen/RISCV/rv64i-demanded-bits.ll
index 8427c64b86edd..f957d25f5c490 100644
--- a/llvm/test/CodeGen/RISCV/rv64i-demanded-bits.ll
+++ b/llvm/test/CodeGen/RISCV/rv64i-demanded-bits.ll
@@ -8,9 +8,9 @@
 define i32 @foo(i32 %x, i32 %y, i32 %z) {
 ; CHECK-LABEL: foo:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    mulw a0, a0, a0
+; CHECK-NEXT:    mul a0, a0, a0
 ; CHECK-NEXT:    addiw a0, a0, 1
-; CHECK-NEXT:    mulw a0, a0, a0
+; CHECK-NEXT:    mul a0, a0, a0
 ; CHECK-NEXT:    add a0, a0, a2
 ; CHECK-NEXT:    addiw a0, a0, 1
 ; CHECK-NEXT:    sllw a0, a0, a1
diff --git a/llvm/test/CodeGen/RISCV/rv64i-w-insts-legalization.ll b/llvm/test/CodeGen/RISCV/rv64i-w-insts-legalization.ll
index c616fbcd37425..eea04ae03f8d6 100644
--- a/llvm/test/CodeGen/RISCV/rv64i-w-insts-legalization.ll
+++ b/llvm/test/CodeGen/RISCV/rv64i-w-insts-legalization.ll
@@ -9,7 +9,7 @@ define signext i32 @addw(i32 signext %s, i32 signext %n, i32 signext %k) nounwin
 ; CHECK-NEXT:    not a2, a0
 ; CHECK-NEXT:    add a2, a2, a1
 ; CHECK-NEXT:    addiw a3, a0, 1
-; CHECK-NEXT:    mulw a3, a2, a3
+; CHECK-NEXT:    mul a3, a2, a3
 ; CHECK-NEXT:    subw a1, a1, a0
 ; CHECK-NEXT:    addiw a1, a1, -2
 ; CHECK-NEXT:    slli a1, a1, 32
@@ -54,7 +54,7 @@ define signext i32 @subw(i32 signext %s, i32 signext %n, i32 signext %k) nounwin
 ; CHECK-NEXT:  # %bb.1: # %for.body.preheader
 ; CHECK-NEXT:    not a2, a0
 ; CHECK-NEXT:    add a3, a2, a1
-; CHECK-NEXT:    mulw a2, a3, a2
+; CHECK-NEXT:    mul a2, a3, a2
 ; CHECK-NEXT:    subw a1, a1, a0
 ; CHECK-NEXT:    addiw a1, a1, -2
 ; CHECK-NEXT:    slli a1, a1, 32
diff --git a/llvm/test/CodeGen/RISCV/rv64m-exhaustive-w-insts.ll b/llvm/test/CodeGen/RISCV/rv64m-exhaustive-w-insts.ll
index 7da8cc82d32ac..5732ac9aa4109 100644
--- a/llvm/test/CodeGen/RISCV/rv64m-exhaustive-w-insts.ll
+++ b/llvm/test/CodeGen/RISCV/rv64m-exhaustive-w-insts.ll
@@ -171,7 +171,7 @@ define signext i32 @sext_mulw_zext_zext(i32 zeroext %a, i32 zeroext %b) nounwind
 define zeroext i32 @zext_mulw_aext_aext(i32 %a, i32 %b) nounwind {
 ; RV64IM-LABEL: zext_mulw_aext_aext:
 ; RV64IM:       # %bb.0:
-; RV64IM-NEXT:    mulw a0, a0, a1
+; RV64IM-NEXT:    mul a0, a0, a1
 ; RV64IM-NEXT:    slli a0, a0, 32
 ; RV64IM-NEXT:    srli a0, a0, 32
 ; RV64IM-NEXT:    ret
@@ -182,7 +182,7 @@ define zeroext i32 @zext_mulw_aext_aext(i32 %a, i32 %b) nounwind {
 define zeroext i32 @zext_mulw_aext_sext(i32 %a, i32 signext %b) nounwind {
 ; RV64IM-LABEL: zext_mulw_aext_sext:
 ; RV64IM:       # %bb.0:
-; RV64IM-NEXT:    mulw a0, a0, a1
+; RV64IM-NEXT:    mul a0, a0, a1
 ; RV64IM-NEXT:    slli a0, a0, 32
 ; RV64IM-NEXT:    srli a0, a0, 32
 ; RV64IM-NEXT:    ret
@@ -193,7 +193,7 @@ define zeroext i32 @zext_mulw_aext_sext(i32 %a, i32 signext %b) nounwind {
 define zeroext i32 @zext_mulw_aext_zext(i32 %a, i32 zeroext %b) nounwind {
 ; RV64IM-LABEL: zext_mulw_aext_zext:
 ; RV64IM:       # %bb.0:
-; RV64IM-NEXT:    mulw a0, a0, a1
+; RV64IM-NEXT:    mul a0, a0, a1
 ; RV64IM-NEXT:    slli a0, a0, 32
 ; RV64IM-NEXT:    srli a0, a0, 32
 ; RV64IM-NEXT:    ret
@@ -204,7 +204,7 @@ define zeroext i32 @zext_mulw_aext_zext(i32 %a, i32 zeroext %b) nounwind {
 define zeroext i32 @zext_mulw_sext_aext(i32 signext %a, i32 %b) nounwind {
 ; RV64IM-LABEL: zext_mulw_sext_aext:
 ; RV64IM:       # %bb.0:
-; RV64IM-NEXT:    mulw a0, a0, a1
+; RV64IM-NEXT:    mul a0, a0, a1
 ; RV64IM-NEXT:    slli a0, a0, 32
 ; RV64IM-NEXT:    srli a0, a0, 32
 ; RV64IM-NEXT:    ret
@@ -215,7 +215,7 @@ define zeroext i32 @zext_mulw_sext_aext(i32 signext %a, i32 %b) nounwind {
 define zeroext i32 @zext_mulw_sext_sext(i32 signext %a, i32 signext %b) nounwind {
 ; RV64IM-LABEL: zext_mulw_sext_sext:
 ; RV64IM:       # %bb.0:
-; RV64IM-NEXT:    mulw a0, a0, a1
+; RV64IM-NEXT:    mul a0, a0, a1
 ; RV64IM-NEXT:    slli a0, a0, 32
 ; RV64IM-NEXT:    srli a0, a0, 32
 ; RV64IM-NEXT:    ret
@@ -226,7 +226,7 @@ define zeroext i32 @zext_mulw_sext_sext(i32 signext %a, i32 signext %b) nounwind
 define zeroext i32 @zext_mulw_sext_zext(i32 signext %a, i32 zeroext %b) nounwind {
 ; RV64IM-LABEL: zext_mulw_sext_zext:
 ; RV64IM:       # %bb.0:
-; RV64IM-NEXT:    mulw a0, a0, a1
+; RV64IM-NEXT:    mul a0, a0, a1
 ; RV64IM-NEXT:    slli a0, a0, 32
 ; RV64IM-NEXT:    srli a0, a0, 32
 ; RV64IM-NEXT:    ret
@@ -237,7 +237,7 @@ define zeroext i32 @zext_mulw_sext_zext(i32 signext %a, i32 zeroext %b) nounwind
 define zeroext i32 @zext_mulw_zext_aext(i32 zeroext %a, i32 %b) nounwind {
 ; RV64IM-LABEL: zext_mulw_zext_aext:
 ; RV64IM:       # %bb.0:
-; RV64IM-NEXT:    mulw a0, a0, a1
+; RV64IM-NEXT:    mul a0, a0, a1
 ; RV64IM-NEXT:    slli a0, a0, 32
 ; RV64IM-NEXT:    srli a0, a0, 32
 ; RV64IM-NEXT:    ret
@@ -248,7 +248,7 @@ define zeroext i32 @zext_mulw_zext_aext(i32 zeroext %a, i32 %b) nounwind {
 define zeroext i32 @zext_mulw_zext_sext(i32 zeroext %a, i32 signext %b) nounwind {
 ; RV64IM-LABEL: zext_mulw_zext_sext:
 ; RV64IM:       # %bb.0:
-; RV64IM-NEXT:    mulw a0, a0, a1
+; RV64IM-NEXT:    mul a0, a0, a1
 ; RV64IM-NEXT:    slli a0, a0, 32
 ; RV64IM-NEXT:    srli a0, a0, 32
 ; RV64IM-NEXT:    ret
@@ -259,7 +259,7 @@ define zeroext i32 @zext_mulw_zext_sext(i32 zeroext %a, i32 signext %b) nounwind
 define zeroext i32 @zext_mulw_zext_zext(i32 zeroext %a, i32 zeroext %b) nounwind {
 ; RV64IM-LABEL: zext_mulw_zext_zext:
 ; RV64IM:       # %bb.0:
-; RV64IM-NEXT:    mulw a0, a0, a1
+; RV64IM-NEXT:    mul a0, a0, a1
 ; RV64IM-NEXT:    slli a0, a0, 32
 ; RV64IM-NEXT:    srli a0, a0, 32
 ; RV64IM-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/rvv/extract-subvector.ll b/llvm/test/CodeGen/RISCV/rvv/extract-subvector.ll
index 53995fe70704d..8dfb472f1f805 100644
--- a/llvm/test/CodeGen/RISCV/rvv/extract-subvector.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/extract-subvector.ll
@@ -5,7 +5,6 @@
 define <vscale x 4 x i32> @extract_nxv8i32_nxv4i32_0(<vscale x 8 x i32> %vec) {
 ; CHECK-LABEL: extract_nxv8i32_nxv4i32_0:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m4
 ; CHECK-NEXT:    ret
   %c = call <vscale x 4 x i32> @llvm.vector.extract.nxv4i32.nxv8i32(<vscale x 8 x i32> %vec, i64 0)
   ret <vscale x 4 x i32> %c
@@ -23,7 +22,6 @@ define <vscale x 4 x i32> @extract_nxv8i32_nxv4i32_4(<vscale x 8 x i32> %vec) {
 define <vscale x 2 x i32> @extract_nxv8i32_nxv2i32_0(<vscale x 8 x i32> %vec) {
 ; CHECK-LABEL: extract_nxv8i32_nxv2i32_0:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8m4
 ; CHECK-NEXT:    ret
   %c = call <vscale x 2 x i32> @llvm.vector.extract.nxv2i32.nxv8i32(<vscale x 8 x i32> %vec, i64 0)
   ret <vscale x 2 x i32> %c
@@ -59,7 +57,6 @@ define <vscale x 2 x i32> @extract_nxv8i32_nxv2i32_6(<vscale x 8 x i32> %vec) {
 define <vscale x 8 x i32> @extract_nxv16i32_nxv8i32_0(<vscale x 16 x i32> %vec) {
 ; CHECK-LABEL: extract_nxv16i32_nxv8i32_0:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m8
 ; CHECK-NEXT:    ret
   %c = call <vscale x 8 x i32> @llvm.vector.extract.nxv8i32.nxv16i32(<vscale x 16 x i32> %vec, i64 0)
   ret <vscale x 8 x i32> %c
@@ -77,7 +74,6 @@ define <vscale x 8 x i32> @extract_nxv16i32_nxv8i32_8(<vscale x 16 x i32> %vec)
 define <vscale x 4 x i32> @extract_nxv16i32_nxv4i32_0(<vscale x 16 x i32> %vec) {
 ; CHECK-LABEL: extract_nxv16i32_nxv4i32_0:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m8
 ; CHECK-NEXT:    ret
   %c = call <vscale x 4 x i32> @llvm.vector.extract.nxv4i32.nxv16i32(<vscale x 16 x i32> %vec, i64 0)
   ret <vscale x 4 x i32> %c
@@ -113,7 +109,6 @@ define <vscale x 4 x i32> @extract_nxv16i32_nxv4i32_12(<vscale x 16 x i32> %vec)
 define <vscale x 2 x i32> @extract_nxv16i32_nxv2i32_0(<vscale x 16 x i32> %vec) {
 ; CHECK-LABEL: extract_nxv16i32_nxv2i32_0:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8m8
 ; CHECK-NEXT:    ret
   %c = call <vscale x 2 x i32> @llvm.vector.extract.nxv2i32.nxv16i32(<vscale x 16 x i32> %vec, i64 0)
   ret <vscale x 2 x i32> %c
@@ -185,7 +180,6 @@ define <vscale x 2 x i32> @extract_nxv16i32_nxv2i32_14(<vscale x 16 x i32> %vec)
 define <vscale x 1 x i32> @extract_nxv16i32_nxv1i32_0(<vscale x 16 x i32> %vec) {
 ; CHECK-LABEL: extract_nxv16i32_nxv1i32_0:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8m8
 ; CHECK-NEXT:    ret
   %c = call <vscale x 1 x i32> @llvm.vector.extract.nxv1i32.nxv16i32(<vscale x 16 x i32> %vec, i64 0)
   ret <vscale x 1 x i32> %c
@@ -247,7 +241,6 @@ define <vscale x 1 x i32> @extract_nxv2i32_nxv1i32_0(<vscale x 2 x i32> %vec) {
 define <vscale x 2 x i8> @extract_nxv32i8_nxv2i8_0(<vscale x 32 x i8> %vec) {
 ; CHECK-LABEL: extract_nxv32i8_nxv2i8_0:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8m4
 ; CHECK-NEXT:    ret
   %c = call <vscale x 2 x i8> @llvm.vector.extract.nxv2i8.nxv32i8(<vscale x 32 x i8> %vec, i64 0)
   ret <vscale x 2 x i8> %c
@@ -344,7 +337,6 @@ define <vscale x 1 x i8> @extract_nxv4i8_nxv1i8_3(<vscale x 4 x i8> %vec) {
 define <vscale x 2 x half> @extract_nxv2f16_nxv16f16_0(<vscale x 16 x half> %vec) {
 ; CHECK-LABEL: extract_nxv2f16_nxv16f16_0:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8m4
 ; CHECK-NEXT:    ret
   %c = call <vscale x 2 x half> @llvm.vector.extract.nxv2f16.nxv16f16(<vscale x 16 x half> %vec, i64 0)
   ret <vscale x 2 x half> %c
@@ -467,7 +459,6 @@ define <vscale x 16 x i1> @extract_nxv16i1_nxv32i1_16(<vscale x 32 x i1> %x) {
 define <vscale x 6 x half> @extract_nxv6f16_nxv12f16_0(<vscale x 12 x half> %in) {
 ; CHECK-LABEL: extract_nxv6f16_nxv12f16_0:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m4
 ; CHECK-NEXT:    ret
   %res = call <vscale x 6 x half> @llvm.vector.extract.nxv6f16.nxv12f16(<vscale x 12 x half> %in, i64 0)
   ret <vscale x 6 x half> %res
@@ -479,14 +470,13 @@ define <vscale x 6 x half> @extract_nxv6f16_nxv12f16_6(<vscale x 12 x half> %in)
 ; CHECK-NEXT:    csrr a0, vlenb
 ; CHECK-NEXT:    srli a0, a0, 2
 ; CHECK-NEXT:    vsetvli a1, zero, e16, m1, ta, ma
-; CHECK-NEXT:    vslidedown.vx v14, v10, a0
-; CHECK-NEXT:    vslidedown.vx v12, v9, a0
+; CHECK-NEXT:    vslidedown.vx v11, v10, a0
+; CHECK-NEXT:    vslidedown.vx v8, v9, a0
 ; CHECK-NEXT:    vsetvli zero, a0, e16, m1, tu, ma
-; CHECK-NEXT:    vslideup.vi v13, v14, 0
+; CHECK-NEXT:    vslideup.vi v9, v11, 0
 ; CHECK-NEXT:    add a1, a0, a0
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, tu, ma
-; CHECK-NEXT:    vslideup.vx v12, v10, a0
-; CHECK-NEXT:    vmv2r.v v8, v12
+; CHECK-NEXT:    vslideup.vx v8, v10, a0
 ; CHECK-NEXT:    ret
   %res = call <vscale x 6 x half> @llvm.vector.extract.nxv6f16.nxv12f16(<vscale x 12 x half> %in, i64 6)
   ret <vscale x 6 x half> %res
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vector-fptrunc-vp.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vector-fptrunc-vp.ll
index 04d394a02188a..a31197366b9a7 100644
--- a/llvm/test/CodeGen/RISCV/rvv/fixed-vector-fptrunc-vp.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vector-fptrunc-vp.ll
@@ -94,15 +94,7 @@ declare <32 x float> @llvm.vp.fptrunc.v32f64.v32f32(<32 x double>, <32 x i1>, i3
 define <32 x float> @vfptrunc_v32f32_v32f64(<32 x double> %a, <32 x i1> %m, i32 zeroext %vl) {
 ; CHECK-LABEL: vfptrunc_v32f32_v32f64:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    addi sp, sp, -16
-; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    csrr a1, vlenb
-; CHECK-NEXT:    slli a1, a1, 3
-; CHECK-NEXT:    sub sp, sp, a1
-; CHECK-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x08, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 8 * vlenb
-; CHECK-NEXT:    vmv1r.v v24, v0
-; CHECK-NEXT:    addi a1, sp, 16
-; CHECK-NEXT:    vs8r.v v8, (a1) # Unknown-size Folded Spill
+; CHECK-NEXT:    vmv1r.v v28, v0
 ; CHECK-NEXT:    vsetivli zero, 2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vslidedown.vi v0, v0, 2
 ; CHECK-NEXT:    addi a1, a0, -16
@@ -111,24 +103,18 @@ define <32 x float> @vfptrunc_v32f32_v32f64(<32 x double> %a, <32 x i1> %m, i32
 ; CHECK-NEXT:    and a1, a2, a1
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    li a1, 16
-; CHECK-NEXT:    vfncvt.f.f.w v8, v16, v0.t
+; CHECK-NEXT:    vfncvt.f.f.w v24, v16, v0.t
 ; CHECK-NEXT:    bltu a0, a1, .LBB7_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    li a0, 16
 ; CHECK-NEXT:  .LBB7_2:
 ; CHECK-NEXT:    vsetvli zero, a0, e32, m4, ta, ma
-; CHECK-NEXT:    vmv1r.v v0, v24
-; CHECK-NEXT:    addi a0, sp, 16
-; CHECK-NEXT:    vl8r.v v24, (a0) # Unknown-size Folded Reload
-; CHECK-NEXT:    vfncvt.f.f.w v16, v24, v0.t
+; CHECK-NEXT:    vmv1r.v v0, v28
+; CHECK-NEXT:    vfncvt.f.f.w v16, v8, v0.t
 ; CHECK-NEXT:    li a0, 32
 ; CHECK-NEXT:    vsetvli zero, a0, e32, m8, tu, ma
-; CHECK-NEXT:    vslideup.vi v16, v8, 16
+; CHECK-NEXT:    vslideup.vi v16, v24, 16
 ; CHECK-NEXT:    vmv8r.v v8, v16
-; CHECK-NEXT:    csrr a0, vlenb
-; CHECK-NEXT:    slli a0, a0, 3
-; CHECK-NEXT:    add sp, sp, a0
-; CHECK-NEXT:    addi sp, sp, 16
 ; CHECK-NEXT:    ret
   %v = call <32 x float> @llvm.vp.fptrunc.v32f64.v32f32(<32 x double> %a, <32 x i1> %m, i32 %vl)
   ret <32 x float> %v
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vector-segN-load.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vector-segN-load.ll
index 15d65a2a6aaa2..03b98412ce1b2 100644
--- a/llvm/test/CodeGen/RISCV/rvv/fixed-vector-segN-load.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vector-segN-load.ll
@@ -7,7 +7,6 @@ define <8 x i8> @load_factor2(ptr %ptr) {
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetivli zero, 8, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
     %1 = call { <8 x i8>, <8 x i8> } @llvm.riscv.seg2.load.v8i8.p0.i64(ptr %ptr, i64 8)
     %2 = extractvalue { <8 x i8>, <8 x i8> } %1, 0
@@ -20,7 +19,6 @@ define <8 x i8> @load_factor3(ptr %ptr) {
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetivli zero, 8, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg3e8.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v6_v7_v8
 ; CHECK-NEXT:    ret
     %1 = call { <8 x i8>, <8 x i8>, <8 x i8> } @llvm.riscv.seg3.load.v8i8.p0.i64(ptr %ptr, i64 8)
     %2 = extractvalue { <8 x i8>, <8 x i8>, <8 x i8> } %1, 0
@@ -34,7 +32,6 @@ define <8 x i8> @load_factor4(ptr %ptr) {
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetivli zero, 8, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg4e8.v v5, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v5_v6_v7_v8
 ; CHECK-NEXT:    ret
     %1 = call { <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8> } @llvm.riscv.seg4.load.v8i8.p0.i64(ptr %ptr, i64 8)
     %2 = extractvalue { <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8> } %1, 0
@@ -49,7 +46,6 @@ define <8 x i8> @load_factor5(ptr %ptr) {
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetivli zero, 8, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg5e8.v v4, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v4_v5_v6_v7_v8
 ; CHECK-NEXT:    ret
     %1 = call { <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8> } @llvm.riscv.seg5.load.v8i8.p0.i64(ptr %ptr, i64 8)
     %2 = extractvalue { <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8> } %1, 0
@@ -65,7 +61,6 @@ define <8 x i8> @load_factor6(ptr %ptr) {
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetivli zero, 8, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg6e8.v v3, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v3_v4_v5_v6_v7_v8
 ; CHECK-NEXT:    ret
     %1 = call { <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8> } @llvm.riscv.seg6.load.v8i8.p0.i64(ptr %ptr, i64 8)
     %2 = extractvalue { <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8> } %1, 0
@@ -82,7 +77,6 @@ define <8 x i8> @load_factor7(ptr %ptr) {
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetivli zero, 8, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg7e8.v v2, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v2_v3_v4_v5_v6_v7_v8
 ; CHECK-NEXT:    ret
     %1 = call { <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8> } @llvm.riscv.seg7.load.v8i8.p0.i64(ptr %ptr, i64 8)
     %2 = extractvalue { <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8> } %1, 0
@@ -100,7 +94,6 @@ define <8 x i8> @load_factor8(ptr %ptr) {
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetivli zero, 8, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg8e8.v v1, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v1_v2_v3_v4_v5_v6_v7_v8
 ; CHECK-NEXT:    ret
     %1 = call { <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8> } @llvm.riscv.seg8.load.v8i8.p0.i64(ptr %ptr, i64 8)
     %2 = extractvalue { <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8> } %1, 0
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vector-segN-store.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vector-segN-store.ll
new file mode 100644
index 0000000000000..0b4184ad7474f
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vector-segN-store.ll
@@ -0,0 +1,83 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: sed 's/iXLen/i32/g' %s | llc -mtriple=riscv32 -mattr=+v -verify-machineinstrs | FileCheck %s -check-prefixes=CHECK,RV32
+; RUN: sed 's/iXLen/i64/g' %s |llc -mtriple=riscv64 -mattr=+v -verify-machineinstrs | FileCheck %s -check-prefixes=CHECK,RV64
+
+declare void @llvm.riscv.seg2.store.v8i8.p0.iXLen(<8 x i8>, <8 x i8>, ptr, iXLen)
+define void @store_factor2(<8 x i8> %v0, <8 x i8> %v1, ptr %ptr) {
+; CHECK-LABEL: store_factor2:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e8, mf2, ta, ma
+; CHECK-NEXT:    vsseg2e8.v v8, (a0)
+; CHECK-NEXT:    ret
+    call void @llvm.riscv.seg2.store.v8i8.p0.iXLen(<8 x i8> %v0, <8 x i8> %v1, ptr %ptr, iXLen 8)
+    ret void
+}
+
+declare void @llvm.riscv.seg3.store.v8i8.p0.iXLen(<8 x i8>, <8 x i8>, <8 x i8>, ptr, iXLen)
+define void @store_factor3(<8 x i8> %v0, <8 x i8> %v1, <8 x i8> %v2, ptr %ptr) {
+; CHECK-LABEL: store_factor3:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e8, mf2, ta, ma
+; CHECK-NEXT:    vsseg3e8.v v8, (a0)
+; CHECK-NEXT:    ret
+    call void @llvm.riscv.seg3.store.v8i8.p0.iXLen(<8 x i8> %v0, <8 x i8> %v1, <8 x i8> %v2, ptr %ptr, iXLen 8)
+    ret void
+}
+
+declare void @llvm.riscv.seg4.store.v8i8.p0.iXLen(<8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, ptr, iXLen)
+define void @store_factor4(<8 x i8> %v0, <8 x i8> %v1, <8 x i8> %v2, <8 x i8> %v3, ptr %ptr) {
+; CHECK-LABEL: store_factor4:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e8, mf2, ta, ma
+; CHECK-NEXT:    vsseg4e8.v v8, (a0)
+; CHECK-NEXT:    ret
+    call void @llvm.riscv.seg4.store.v8i8.p0.iXLen(<8 x i8> %v0, <8 x i8> %v1, <8 x i8> %v2, <8 x i8> %v3, ptr %ptr, iXLen 8)
+    ret void
+}
+
+declare void @llvm.riscv.seg5.store.v8i8.p0.iXLen(<8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, ptr, iXLen)
+define void @store_factor5(<8 x i8> %v0, <8 x i8> %v1, <8 x i8> %v2, <8 x i8> %v3, <8 x i8> %v4, ptr %ptr) {
+; CHECK-LABEL: store_factor5:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e8, mf2, ta, ma
+; CHECK-NEXT:    vsseg5e8.v v8, (a0)
+; CHECK-NEXT:    ret
+    call void @llvm.riscv.seg5.store.v8i8.p0.iXLen(<8 x i8> %v0, <8 x i8> %v1, <8 x i8> %v2, <8 x i8> %v3, <8 x i8> %v4, ptr %ptr, iXLen 8)
+    ret void
+}
+
+declare void @llvm.riscv.seg6.store.v8i8.p0.iXLen(<8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, ptr, iXLen)
+define void @store_factor6(<8 x i8> %v0, <8 x i8> %v1, <8 x i8> %v2, <8 x i8> %v3, <8 x i8> %v4, <8 x i8> %v5, ptr %ptr) {
+; CHECK-LABEL: store_factor6:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e8, mf2, ta, ma
+; CHECK-NEXT:    vsseg6e8.v v8, (a0)
+; CHECK-NEXT:    ret
+    call void @llvm.riscv.seg6.store.v8i8.p0.iXLen(<8 x i8> %v0, <8 x i8> %v1, <8 x i8> %v2, <8 x i8> %v3, <8 x i8> %v4, <8 x i8> %v5, ptr %ptr, iXLen 8)
+    ret void
+}
+
+declare void @llvm.riscv.seg7.store.v8i8.p0.iXLen(<8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8x i8>, ptr, iXLen)
+define void @store_factor7(<8 x i8> %v0, <8 x i8> %v1, <8 x i8> %v2, <8 x i8> %v3, <8 x i8> %v4, <8 x i8> %v5, <8 x i8> %v6, ptr %ptr) {
+; CHECK-LABEL: store_factor7:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e8, mf2, ta, ma
+; CHECK-NEXT:    vsseg7e8.v v8, (a0)
+; CHECK-NEXT:    ret
+    call void @llvm.riscv.seg7.store.v8i8.p0.iXLen(<8 x i8> %v0, <8 x i8> %v1, <8 x i8> %v2, <8 x i8> %v3, <8 x i8> %v4, <8 x i8> %v5, <8 x i8> %v6, ptr %ptr, iXLen 8)
+    ret void
+}
+
+declare void @llvm.riscv.seg8.store.v8i8.p0.iXLen(<8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, <8 x i8>, ptr, iXLen)
+define void @store_factor8(<8 x i8> %v0, <8 x i8> %v1, <8 x i8> %v2, <8 x i8> %v3, <8 x i8> %v4, <8 x i8> %v5, <8 x i8> %v6, <8 x i8> %v7, ptr %ptr) {
+; CHECK-LABEL: store_factor8:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e8, mf2, ta, ma
+; CHECK-NEXT:    vsseg8e8.v v8, (a0)
+; CHECK-NEXT:    ret
+    call void @llvm.riscv.seg8.store.v8i8.p0.iXLen(<8 x i8> %v0, <8 x i8> %v1, <8 x i8> %v2, <8 x i8> %v3, <8 x i8> %v4, <8 x i8> %v5, <8 x i8> %v6, <8 x i8> %v7, ptr %ptr, iXLen 8)
+    ret void
+}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; RV32: {{.*}}
+; RV64: {{.*}}
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vector-trunc-vp.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vector-trunc-vp.ll
index e5f83f8ad0aef..e697cb728b4b9 100644
--- a/llvm/test/CodeGen/RISCV/rvv/fixed-vector-trunc-vp.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vector-trunc-vp.ll
@@ -53,15 +53,7 @@ declare <128 x i7> @llvm.vp.trunc.v128i7.v128i16(<128 x i16>, <128 x i1>, i32)
 define <128 x i7> @vtrunc_v128i7_v128i16(<128 x i16> %a, <128 x i1> %m, i32 zeroext %vl) {
 ; CHECK-LABEL: vtrunc_v128i7_v128i16:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    addi sp, sp, -16
-; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    csrr a1, vlenb
-; CHECK-NEXT:    slli a1, a1, 3
-; CHECK-NEXT:    sub sp, sp, a1
-; CHECK-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x08, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 8 * vlenb
-; CHECK-NEXT:    vmv1r.v v24, v0
-; CHECK-NEXT:    addi a1, sp, 16
-; CHECK-NEXT:    vs8r.v v8, (a1) # Unknown-size Folded Spill
+; CHECK-NEXT:    vmv1r.v v28, v0
 ; CHECK-NEXT:    vsetivli zero, 8, e8, m1, ta, ma
 ; CHECK-NEXT:    vslidedown.vi v0, v0, 8
 ; CHECK-NEXT:    addi a1, a0, -64
@@ -70,24 +62,18 @@ define <128 x i7> @vtrunc_v128i7_v128i16(<128 x i16> %a, <128 x i1> %m, i32 zero
 ; CHECK-NEXT:    and a1, a2, a1
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
 ; CHECK-NEXT:    li a1, 64
-; CHECK-NEXT:    vnsrl.wi v8, v16, 0, v0.t
+; CHECK-NEXT:    vnsrl.wi v24, v16, 0, v0.t
 ; CHECK-NEXT:    bltu a0, a1, .LBB4_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    li a0, 64
 ; CHECK-NEXT:  .LBB4_2:
 ; CHECK-NEXT:    vsetvli zero, a0, e8, m4, ta, ma
-; CHECK-NEXT:    vmv1r.v v0, v24
-; CHECK-NEXT:    addi a0, sp, 16
-; CHECK-NEXT:    vl8r.v v24, (a0) # Unknown-size Folded Reload
-; CHECK-NEXT:    vnsrl.wi v16, v24, 0, v0.t
+; CHECK-NEXT:    vmv1r.v v0, v28
+; CHECK-NEXT:    vnsrl.wi v16, v8, 0, v0.t
 ; CHECK-NEXT:    li a0, 128
 ; CHECK-NEXT:    vsetvli zero, a0, e8, m8, tu, ma
-; CHECK-NEXT:    vslideup.vx v16, v8, a1
+; CHECK-NEXT:    vslideup.vx v16, v24, a1
 ; CHECK-NEXT:    vmv8r.v v8, v16
-; CHECK-NEXT:    csrr a0, vlenb
-; CHECK-NEXT:    slli a0, a0, 3
-; CHECK-NEXT:    add sp, sp, a0
-; CHECK-NEXT:    addi sp, sp, 16
 ; CHECK-NEXT:    ret
   %v = call <128 x i7> @llvm.vp.trunc.v128i7.v128i16(<128 x i16> %a, <128 x i1> %m, i32 %vl)
   ret <128 x i7> %v
@@ -247,13 +233,13 @@ define <128 x i32> @vtrunc_v128i32_v128i64(<128 x i64> %a, <128 x i1> %m, i32 ze
 ; CHECK-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x38, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 56 * vlenb
 ; CHECK-NEXT:    vmv1r.v v1, v0
 ; CHECK-NEXT:    csrr a2, vlenb
-; CHECK-NEXT:    li a3, 24
-; CHECK-NEXT:    mul a2, a2, a3
+; CHECK-NEXT:    slli a2, a2, 4
 ; CHECK-NEXT:    add a2, sp, a2
 ; CHECK-NEXT:    addi a2, a2, 16
 ; CHECK-NEXT:    vs8r.v v16, (a2) # Unknown-size Folded Spill
 ; CHECK-NEXT:    csrr a2, vlenb
-; CHECK-NEXT:    slli a2, a2, 5
+; CHECK-NEXT:    li a3, 24
+; CHECK-NEXT:    mul a2, a2, a3
 ; CHECK-NEXT:    add a2, sp, a2
 ; CHECK-NEXT:    addi a2, a2, 16
 ; CHECK-NEXT:    vs8r.v v8, (a2) # Unknown-size Folded Spill
@@ -290,7 +276,7 @@ define <128 x i32> @vtrunc_v128i32_v128i64(<128 x i64> %a, <128 x i1> %m, i32 ze
 ; CHECK-NEXT:    vsetvli zero, a5, e32, m4, ta, ma
 ; CHECK-NEXT:    vnsrl.wi v16, v8, 0, v0.t
 ; CHECK-NEXT:    csrr a2, vlenb
-; CHECK-NEXT:    slli a2, a2, 4
+; CHECK-NEXT:    slli a2, a2, 3
 ; CHECK-NEXT:    add a2, sp, a2
 ; CHECK-NEXT:    addi a2, a2, 16
 ; CHECK-NEXT:    vs8r.v v16, (a2) # Unknown-size Folded Spill
@@ -326,7 +312,7 @@ define <128 x i32> @vtrunc_v128i32_v128i64(<128 x i64> %a, <128 x i1> %m, i32 ze
 ; CHECK-NEXT:  .LBB16_4:
 ; CHECK-NEXT:    li a3, 32
 ; CHECK-NEXT:    vsetivli zero, 16, e64, m8, ta, ma
-; CHECK-NEXT:    vle64.v v16, (a1)
+; CHECK-NEXT:    vle64.v v24, (a1)
 ; CHECK-NEXT:    addi a5, a7, -32
 ; CHECK-NEXT:    sltu a6, a7, a5
 ; CHECK-NEXT:    addi a6, a6, -1
@@ -337,12 +323,9 @@ define <128 x i32> @vtrunc_v128i32_v128i64(<128 x i64> %a, <128 x i1> %m, i32 ze
 ; CHECK-NEXT:    and a6, t0, a6
 ; CHECK-NEXT:    vsetvli zero, a6, e32, m4, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v4
-; CHECK-NEXT:    vnsrl.wi v24, v8, 0, v0.t
-; CHECK-NEXT:    csrr a6, vlenb
-; CHECK-NEXT:    slli a6, a6, 3
-; CHECK-NEXT:    add a6, sp, a6
-; CHECK-NEXT:    addi a6, a6, 16
-; CHECK-NEXT:    vs8r.v v24, (a6) # Unknown-size Folded Spill
+; CHECK-NEXT:    vnsrl.wi v16, v8, 0, v0.t
+; CHECK-NEXT:    addi a6, sp, 16
+; CHECK-NEXT:    vs8r.v v16, (a6) # Unknown-size Folded Spill
 ; CHECK-NEXT:    bltu a5, a2, .LBB16_6
 ; CHECK-NEXT:  # %bb.5:
 ; CHECK-NEXT:    li a5, 16
@@ -351,7 +334,7 @@ define <128 x i32> @vtrunc_v128i32_v128i64(<128 x i64> %a, <128 x i1> %m, i32 ze
 ; CHECK-NEXT:    addi a1, a1, 256
 ; CHECK-NEXT:    vsetvli zero, a5, e32, m4, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v2
-; CHECK-NEXT:    vnsrl.wi v8, v16, 0, v0.t
+; CHECK-NEXT:    vnsrl.wi v8, v24, 0, v0.t
 ; CHECK-NEXT:    csrr a5, vlenb
 ; CHECK-NEXT:    li t0, 40
 ; CHECK-NEXT:    mul a5, a5, t0
@@ -363,10 +346,10 @@ define <128 x i32> @vtrunc_v128i32_v128i64(<128 x i64> %a, <128 x i1> %m, i32 ze
 ; CHECK-NEXT:    li a4, 32
 ; CHECK-NEXT:  .LBB16_8:
 ; CHECK-NEXT:    vsetivli zero, 2, e8, mf4, ta, ma
-; CHECK-NEXT:    vslidedown.vi v4, v3, 2
+; CHECK-NEXT:    vslidedown.vi v20, v3, 2
 ; CHECK-NEXT:    vsetivli zero, 16, e64, m8, ta, ma
-; CHECK-NEXT:    vle64.v v16, (a6)
-; CHECK-NEXT:    vle64.v v24, (a1)
+; CHECK-NEXT:    vle64.v v24, (a6)
+; CHECK-NEXT:    vle64.v v8, (a1)
 ; CHECK-NEXT:    mv a1, a4
 ; CHECK-NEXT:    bltu a4, a2, .LBB16_10
 ; CHECK-NEXT:  # %bb.9:
@@ -376,16 +359,19 @@ define <128 x i32> @vtrunc_v128i32_v128i64(<128 x i64> %a, <128 x i1> %m, i32 ze
 ; CHECK-NEXT:    vslidedown.vi v2, v1, 2
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v3
-; CHECK-NEXT:    vnsrl.wi v8, v24, 0, v0.t
+; CHECK-NEXT:    vnsrl.wi v16, v8, 0, v0.t
+; CHECK-NEXT:    csrr a1, vlenb
+; CHECK-NEXT:    slli a1, a1, 5
+; CHECK-NEXT:    add a1, sp, a1
+; CHECK-NEXT:    addi a1, a1, 16
+; CHECK-NEXT:    vs8r.v v16, (a1) # Unknown-size Folded Spill
 ; CHECK-NEXT:    addi a1, a4, -16
 ; CHECK-NEXT:    sltu a4, a4, a1
 ; CHECK-NEXT:    addi a4, a4, -1
 ; CHECK-NEXT:    and a1, a4, a1
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
-; CHECK-NEXT:    vmv1r.v v0, v4
-; CHECK-NEXT:    vnsrl.wi v24, v16, 0, v0.t
-; CHECK-NEXT:    addi a1, sp, 16
-; CHECK-NEXT:    vs8r.v v24, (a1) # Unknown-size Folded Spill
+; CHECK-NEXT:    vmv1r.v v0, v20
+; CHECK-NEXT:    vnsrl.wi v8, v24, 0, v0.t
 ; CHECK-NEXT:    bltu a7, a3, .LBB16_12
 ; CHECK-NEXT:  # %bb.11:
 ; CHECK-NEXT:    li a7, 32
@@ -398,7 +384,7 @@ define <128 x i32> @vtrunc_v128i32_v128i64(<128 x i64> %a, <128 x i1> %m, i32 ze
 ; CHECK-NEXT:    addi a1, a1, 16
 ; CHECK-NEXT:    vl8r.v v16, (a1) # Unknown-size Folded Reload
 ; CHECK-NEXT:    csrr a1, vlenb
-; CHECK-NEXT:    slli a1, a1, 4
+; CHECK-NEXT:    slli a1, a1, 3
 ; CHECK-NEXT:    add a1, sp, a1
 ; CHECK-NEXT:    addi a1, a1, 16
 ; CHECK-NEXT:    vl8r.v v24, (a1) # Unknown-size Folded Reload
@@ -415,10 +401,7 @@ define <128 x i32> @vtrunc_v128i32_v128i64(<128 x i64> %a, <128 x i1> %m, i32 ze
 ; CHECK-NEXT:    add a1, sp, a1
 ; CHECK-NEXT:    addi a1, a1, 16
 ; CHECK-NEXT:    vl8r.v v16, (a1) # Unknown-size Folded Reload
-; CHECK-NEXT:    csrr a1, vlenb
-; CHECK-NEXT:    slli a1, a1, 3
-; CHECK-NEXT:    add a1, sp, a1
-; CHECK-NEXT:    addi a1, a1, 16
+; CHECK-NEXT:    addi a1, sp, 16
 ; CHECK-NEXT:    vl8r.v v24, (a1) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vslideup.vi v16, v24, 16
 ; CHECK-NEXT:    csrr a1, vlenb
@@ -427,9 +410,17 @@ define <128 x i32> @vtrunc_v128i32_v128i64(<128 x i64> %a, <128 x i1> %m, i32 ze
 ; CHECK-NEXT:    add a1, sp, a1
 ; CHECK-NEXT:    addi a1, a1, 16
 ; CHECK-NEXT:    vs8r.v v16, (a1) # Unknown-size Folded Spill
-; CHECK-NEXT:    addi a1, sp, 16
+; CHECK-NEXT:    csrr a1, vlenb
+; CHECK-NEXT:    slli a1, a1, 5
+; CHECK-NEXT:    add a1, sp, a1
+; CHECK-NEXT:    addi a1, a1, 16
 ; CHECK-NEXT:    vl8r.v v16, (a1) # Unknown-size Folded Reload
-; CHECK-NEXT:    vslideup.vi v8, v16, 16
+; CHECK-NEXT:    vslideup.vi v16, v8, 16
+; CHECK-NEXT:    csrr a1, vlenb
+; CHECK-NEXT:    slli a1, a1, 5
+; CHECK-NEXT:    add a1, sp, a1
+; CHECK-NEXT:    addi a1, a1, 16
+; CHECK-NEXT:    vs8r.v v16, (a1) # Unknown-size Folded Spill
 ; CHECK-NEXT:    addi a1, a7, -16
 ; CHECK-NEXT:    sltu a4, a7, a1
 ; CHECK-NEXT:    addi a4, a4, -1
@@ -437,17 +428,11 @@ define <128 x i32> @vtrunc_v128i32_v128i64(<128 x i64> %a, <128 x i1> %m, i32 ze
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v2
 ; CHECK-NEXT:    csrr a1, vlenb
-; CHECK-NEXT:    li a4, 24
-; CHECK-NEXT:    mul a1, a1, a4
-; CHECK-NEXT:    add a1, sp, a1
-; CHECK-NEXT:    addi a1, a1, 16
-; CHECK-NEXT:    vl8r.v v16, (a1) # Unknown-size Folded Reload
-; CHECK-NEXT:    vnsrl.wi v24, v16, 0, v0.t
-; CHECK-NEXT:    csrr a1, vlenb
 ; CHECK-NEXT:    slli a1, a1, 4
 ; CHECK-NEXT:    add a1, sp, a1
 ; CHECK-NEXT:    addi a1, a1, 16
-; CHECK-NEXT:    vs8r.v v24, (a1) # Unknown-size Folded Spill
+; CHECK-NEXT:    vl8r.v v16, (a1) # Unknown-size Folded Reload
+; CHECK-NEXT:    vnsrl.wi v8, v16, 0, v0.t
 ; CHECK-NEXT:    bltu a7, a2, .LBB16_14
 ; CHECK-NEXT:  # %bb.13:
 ; CHECK-NEXT:    li a7, 16
@@ -455,20 +440,21 @@ define <128 x i32> @vtrunc_v128i32_v128i64(<128 x i64> %a, <128 x i1> %m, i32 ze
 ; CHECK-NEXT:    vsetvli zero, a7, e32, m4, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1
 ; CHECK-NEXT:    csrr a1, vlenb
-; CHECK-NEXT:    slli a1, a1, 5
+; CHECK-NEXT:    li a2, 24
+; CHECK-NEXT:    mul a1, a1, a2
 ; CHECK-NEXT:    add a1, sp, a1
 ; CHECK-NEXT:    addi a1, a1, 16
-; CHECK-NEXT:    vl8r.v v16, (a1) # Unknown-size Folded Reload
-; CHECK-NEXT:    vnsrl.wi v24, v16, 0, v0.t
+; CHECK-NEXT:    vl8r.v v24, (a1) # Unknown-size Folded Reload
+; CHECK-NEXT:    vnsrl.wi v16, v24, 0, v0.t
 ; CHECK-NEXT:    vsetvli zero, a3, e32, m8, tu, ma
-; CHECK-NEXT:    csrr a1, vlenb
-; CHECK-NEXT:    slli a1, a1, 4
-; CHECK-NEXT:    add a1, sp, a1
-; CHECK-NEXT:    addi a1, a1, 16
-; CHECK-NEXT:    vl8r.v v16, (a1) # Unknown-size Folded Reload
-; CHECK-NEXT:    vslideup.vi v24, v16, 16
-; CHECK-NEXT:    vse32.v v24, (a0)
+; CHECK-NEXT:    vslideup.vi v16, v8, 16
+; CHECK-NEXT:    vse32.v v16, (a0)
 ; CHECK-NEXT:    addi a1, a0, 256
+; CHECK-NEXT:    csrr a2, vlenb
+; CHECK-NEXT:    slli a2, a2, 5
+; CHECK-NEXT:    add a2, sp, a2
+; CHECK-NEXT:    addi a2, a2, 16
+; CHECK-NEXT:    vl8r.v v8, (a2) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vse32.v v8, (a1)
 ; CHECK-NEXT:    addi a1, a0, 128
 ; CHECK-NEXT:    csrr a2, vlenb
@@ -501,15 +487,7 @@ declare <32 x i32> @llvm.vp.trunc.v32i32.v32i64(<32 x i64>, <32 x i1>, i32)
 define <32 x i32> @vtrunc_v32i32_v32i64(<32 x i64> %a, <32 x i1> %m, i32 zeroext %vl) {
 ; CHECK-LABEL: vtrunc_v32i32_v32i64:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    addi sp, sp, -16
-; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    csrr a1, vlenb
-; CHECK-NEXT:    slli a1, a1, 3
-; CHECK-NEXT:    sub sp, sp, a1
-; CHECK-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x08, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 8 * vlenb
-; CHECK-NEXT:    vmv1r.v v24, v0
-; CHECK-NEXT:    addi a1, sp, 16
-; CHECK-NEXT:    vs8r.v v8, (a1) # Unknown-size Folded Spill
+; CHECK-NEXT:    vmv1r.v v28, v0
 ; CHECK-NEXT:    vsetivli zero, 2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vslidedown.vi v0, v0, 2
 ; CHECK-NEXT:    addi a1, a0, -16
@@ -518,24 +496,18 @@ define <32 x i32> @vtrunc_v32i32_v32i64(<32 x i64> %a, <32 x i1> %m, i32 zeroext
 ; CHECK-NEXT:    and a1, a2, a1
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    li a1, 16
-; CHECK-NEXT:    vnsrl.wi v8, v16, 0, v0.t
+; CHECK-NEXT:    vnsrl.wi v24, v16, 0, v0.t
 ; CHECK-NEXT:    bltu a0, a1, .LBB17_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    li a0, 16
 ; CHECK-NEXT:  .LBB17_2:
 ; CHECK-NEXT:    vsetvli zero, a0, e32, m4, ta, ma
-; CHECK-NEXT:    vmv1r.v v0, v24
-; CHECK-NEXT:    addi a0, sp, 16
-; CHECK-NEXT:    vl8r.v v24, (a0) # Unknown-size Folded Reload
-; CHECK-NEXT:    vnsrl.wi v16, v24, 0, v0.t
+; CHECK-NEXT:    vmv1r.v v0, v28
+; CHECK-NEXT:    vnsrl.wi v16, v8, 0, v0.t
 ; CHECK-NEXT:    li a0, 32
 ; CHECK-NEXT:    vsetvli zero, a0, e32, m8, tu, ma
-; CHECK-NEXT:    vslideup.vi v16, v8, 16
+; CHECK-NEXT:    vslideup.vi v16, v24, 16
 ; CHECK-NEXT:    vmv8r.v v8, v16
-; CHECK-NEXT:    csrr a0, vlenb
-; CHECK-NEXT:    slli a0, a0, 3
-; CHECK-NEXT:    add sp, sp, a0
-; CHECK-NEXT:    addi sp, sp, 16
 ; CHECK-NEXT:    ret
   %v = call <32 x i32> @llvm.vp.trunc.v32i32.v32i64(<32 x i64> %a, <32 x i1> %m, i32 %vl)
   ret <32 x i32> %v
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-calling-conv.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-calling-conv.ll
index b6d83b7d883ae..a95517e22efa2 100644
--- a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-calling-conv.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-calling-conv.ll
@@ -1188,17 +1188,15 @@ define <32 x i32> @vector_arg_via_stack(i32 %0, i32 %1, i32 %2, i32 %3, i32 %4,
 ; LMULMAX8-LABEL: vector_arg_via_stack:
 ; LMULMAX8:       # %bb.0:
 ; LMULMAX8-NEXT:    li a0, 32
-; LMULMAX8-NEXT:    mv a1, sp
 ; LMULMAX8-NEXT:    vsetvli zero, a0, e32, m8, ta, ma
-; LMULMAX8-NEXT:    vle32.v v16, (a1)
+; LMULMAX8-NEXT:    vle32.v v16, (sp)
 ; LMULMAX8-NEXT:    vadd.vv v8, v8, v16
 ; LMULMAX8-NEXT:    ret
 ;
 ; LMULMAX4-LABEL: vector_arg_via_stack:
 ; LMULMAX4:       # %bb.0:
 ; LMULMAX4-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
-; LMULMAX4-NEXT:    mv a0, sp
-; LMULMAX4-NEXT:    vle32.v v16, (a0)
+; LMULMAX4-NEXT:    vle32.v v16, (sp)
 ; LMULMAX4-NEXT:    addi a0, sp, 64
 ; LMULMAX4-NEXT:    vle32.v v20, (a0)
 ; LMULMAX4-NEXT:    vadd.vv v8, v8, v16
@@ -1210,8 +1208,7 @@ define <32 x i32> @vector_arg_via_stack(i32 %0, i32 %1, i32 %2, i32 %3, i32 %4,
 ; LMULMAX2-NEXT:    vsetivli zero, 8, e32, m2, ta, ma
 ; LMULMAX2-NEXT:    addi a0, sp, 64
 ; LMULMAX2-NEXT:    vle32.v v16, (a0)
-; LMULMAX2-NEXT:    mv a0, sp
-; LMULMAX2-NEXT:    vle32.v v18, (a0)
+; LMULMAX2-NEXT:    vle32.v v18, (sp)
 ; LMULMAX2-NEXT:    addi a0, sp, 32
 ; LMULMAX2-NEXT:    vle32.v v20, (a0)
 ; LMULMAX2-NEXT:    addi a0, sp, 96
@@ -1235,8 +1232,7 @@ define <32 x i32> @vector_arg_via_stack(i32 %0, i32 %1, i32 %2, i32 %3, i32 %4,
 ; LMULMAX1-NEXT:    vle32.v v19, (a0)
 ; LMULMAX1-NEXT:    addi a0, sp, 32
 ; LMULMAX1-NEXT:    vle32.v v20, (a0)
-; LMULMAX1-NEXT:    mv a0, sp
-; LMULMAX1-NEXT:    vle32.v v21, (a0)
+; LMULMAX1-NEXT:    vle32.v v21, (sp)
 ; LMULMAX1-NEXT:    addi a0, sp, 16
 ; LMULMAX1-NEXT:    vle32.v v22, (a0)
 ; LMULMAX1-NEXT:    addi a0, sp, 48
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-fp-interleave.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-fp-interleave.ll
index e97a710ba4129..54d267b282d1b 100644
--- a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-fp-interleave.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-fp-interleave.ll
@@ -39,14 +39,13 @@ define <4 x double> @interleave_v2f64(<2 x double> %x, <2 x double> %y) {
 ; RV32-V128-LABEL: interleave_v2f64:
 ; RV32-V128:       # %bb.0:
 ; RV32-V128-NEXT:    vmv1r.v v12, v9
-; RV32-V128-NEXT:    # kill: def $v8 killed $v8 def $v8m2
 ; RV32-V128-NEXT:    vsetivli zero, 4, e16, mf2, ta, ma
-; RV32-V128-NEXT:    vid.v v10
-; RV32-V128-NEXT:    vsrl.vi v14, v10, 1
+; RV32-V128-NEXT:    vid.v v9
+; RV32-V128-NEXT:    vsrl.vi v14, v9, 1
 ; RV32-V128-NEXT:    vsetvli zero, zero, e64, m2, ta, mu
+; RV32-V128-NEXT:    vrgatherei16.vv v10, v8, v14
 ; RV32-V128-NEXT:    li a0, 10
 ; RV32-V128-NEXT:    vmv.s.x v0, a0
-; RV32-V128-NEXT:    vrgatherei16.vv v10, v8, v14
 ; RV32-V128-NEXT:    vrgatherei16.vv v10, v12, v14, v0.t
 ; RV32-V128-NEXT:    vmv.v.v v8, v10
 ; RV32-V128-NEXT:    ret
@@ -54,13 +53,12 @@ define <4 x double> @interleave_v2f64(<2 x double> %x, <2 x double> %y) {
 ; RV64-V128-LABEL: interleave_v2f64:
 ; RV64-V128:       # %bb.0:
 ; RV64-V128-NEXT:    vmv1r.v v12, v9
-; RV64-V128-NEXT:    # kill: def $v8 killed $v8 def $v8m2
 ; RV64-V128-NEXT:    vsetivli zero, 4, e64, m2, ta, mu
 ; RV64-V128-NEXT:    vid.v v10
 ; RV64-V128-NEXT:    vsrl.vi v14, v10, 1
+; RV64-V128-NEXT:    vrgather.vv v10, v8, v14
 ; RV64-V128-NEXT:    li a0, 10
 ; RV64-V128-NEXT:    vmv.s.x v0, a0
-; RV64-V128-NEXT:    vrgather.vv v10, v8, v14
 ; RV64-V128-NEXT:    vrgather.vv v10, v12, v14, v0.t
 ; RV64-V128-NEXT:    vmv.v.v v8, v10
 ; RV64-V128-NEXT:    ret
@@ -266,9 +264,9 @@ define <64 x float> @interleave_v32f32(<32 x float> %x, <32 x float> %y) {
 ; RV32-V128-NEXT:    vsetvli zero, a1, e32, m8, ta, ma
 ; RV32-V128-NEXT:    vle32.v v0, (a0)
 ; RV32-V128-NEXT:    vmv8r.v v24, v8
-; RV32-V128-NEXT:    addi a0, sp, 16
-; RV32-V128-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; RV32-V128-NEXT:    vrgather.vv v8, v24, v0
+; RV32-V128-NEXT:    addi a0, sp, 16
+; RV32-V128-NEXT:    vs8r.v v24, (a0) # Unknown-size Folded Spill
 ; RV32-V128-NEXT:    lui a0, %hi(.LCPI10_1)
 ; RV32-V128-NEXT:    addi a0, a0, %lo(.LCPI10_1)
 ; RV32-V128-NEXT:    vle32.v v24, (a0)
@@ -317,9 +315,9 @@ define <64 x float> @interleave_v32f32(<32 x float> %x, <32 x float> %y) {
 ; RV64-V128-NEXT:    vsetvli zero, a1, e32, m8, ta, ma
 ; RV64-V128-NEXT:    vle32.v v0, (a0)
 ; RV64-V128-NEXT:    vmv8r.v v24, v8
-; RV64-V128-NEXT:    addi a0, sp, 16
-; RV64-V128-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; RV64-V128-NEXT:    vrgather.vv v8, v24, v0
+; RV64-V128-NEXT:    addi a0, sp, 16
+; RV64-V128-NEXT:    vs8r.v v24, (a0) # Unknown-size Folded Spill
 ; RV64-V128-NEXT:    lui a0, %hi(.LCPI10_1)
 ; RV64-V128-NEXT:    addi a0, a0, %lo(.LCPI10_1)
 ; RV64-V128-NEXT:    vle32.v v24, (a0)
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-insert-subvector.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-insert-subvector.ll
index 3d1b849b7a388..7a662952caf4c 100644
--- a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-insert-subvector.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-insert-subvector.ll
@@ -59,14 +59,14 @@ define <vscale x 8 x i32> @insert_nxv8i32_v8i32_0(<vscale x 8 x i32> %vec, ptr %
 ;
 ; LMULMAX1-LABEL: insert_nxv8i32_v8i32_0:
 ; LMULMAX1:       # %bb.0:
+; LMULMAX1-NEXT:    addi a1, a0, 16
 ; LMULMAX1-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
-; LMULMAX1-NEXT:    vle32.v v12, (a0)
-; LMULMAX1-NEXT:    addi a0, a0, 16
+; LMULMAX1-NEXT:    vle32.v v12, (a1)
 ; LMULMAX1-NEXT:    vle32.v v16, (a0)
 ; LMULMAX1-NEXT:    vsetivli zero, 4, e32, m4, tu, ma
-; LMULMAX1-NEXT:    vslideup.vi v8, v12, 0
+; LMULMAX1-NEXT:    vslideup.vi v8, v16, 0
 ; LMULMAX1-NEXT:    vsetivli zero, 8, e32, m4, tu, ma
-; LMULMAX1-NEXT:    vslideup.vi v8, v16, 4
+; LMULMAX1-NEXT:    vslideup.vi v8, v12, 4
 ; LMULMAX1-NEXT:    ret
   %sv = load <8 x i32>, ptr %svp
   %v = call <vscale x 8 x i32> @llvm.vector.insert.v8i32.nxv8i32(<vscale x 8 x i32> %vec, <8 x i32> %sv, i64 0)
@@ -84,14 +84,14 @@ define <vscale x 8 x i32> @insert_nxv8i32_v8i32_8(<vscale x 8 x i32> %vec, ptr %
 ;
 ; LMULMAX1-LABEL: insert_nxv8i32_v8i32_8:
 ; LMULMAX1:       # %bb.0:
+; LMULMAX1-NEXT:    addi a1, a0, 16
 ; LMULMAX1-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
-; LMULMAX1-NEXT:    vle32.v v12, (a0)
-; LMULMAX1-NEXT:    addi a0, a0, 16
+; LMULMAX1-NEXT:    vle32.v v12, (a1)
 ; LMULMAX1-NEXT:    vle32.v v16, (a0)
 ; LMULMAX1-NEXT:    vsetivli zero, 12, e32, m4, tu, ma
-; LMULMAX1-NEXT:    vslideup.vi v8, v12, 8
+; LMULMAX1-NEXT:    vslideup.vi v8, v16, 8
 ; LMULMAX1-NEXT:    vsetivli zero, 16, e32, m4, tu, ma
-; LMULMAX1-NEXT:    vslideup.vi v8, v16, 12
+; LMULMAX1-NEXT:    vslideup.vi v8, v12, 12
 ; LMULMAX1-NEXT:    ret
   %sv = load <8 x i32>, ptr %svp
   %v = call <vscale x 8 x i32> @llvm.vector.insert.v8i32.nxv8i32(<vscale x 8 x i32> %vec, <8 x i32> %sv, i64 8)
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-int-exttrunc.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-int-exttrunc.ll
index 1208510ad8e2d..e007f84d01c93 100644
--- a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-int-exttrunc.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-int-exttrunc.ll
@@ -92,16 +92,16 @@ define void @sext_v32i8_v32i32(ptr %x, ptr %z) {
 ; LMULMAX2-NEXT:    vsetivli zero, 16, e8, m2, ta, ma
 ; LMULMAX2-NEXT:    vslidedown.vi v10, v8, 16
 ; LMULMAX2-NEXT:    vsetivli zero, 8, e8, m1, ta, ma
-; LMULMAX2-NEXT:    vslidedown.vi v14, v10, 8
+; LMULMAX2-NEXT:    vslidedown.vi v9, v10, 8
 ; LMULMAX2-NEXT:    vsetivli zero, 8, e32, m2, ta, ma
-; LMULMAX2-NEXT:    vsext.vf4 v16, v14
-; LMULMAX2-NEXT:    vsext.vf4 v14, v8
+; LMULMAX2-NEXT:    vsext.vf4 v14, v9
+; LMULMAX2-NEXT:    vsext.vf4 v16, v8
 ; LMULMAX2-NEXT:    vsext.vf4 v8, v10
 ; LMULMAX2-NEXT:    addi a0, a1, 64
 ; LMULMAX2-NEXT:    vse32.v v8, (a0)
-; LMULMAX2-NEXT:    vse32.v v14, (a1)
+; LMULMAX2-NEXT:    vse32.v v16, (a1)
 ; LMULMAX2-NEXT:    addi a0, a1, 96
-; LMULMAX2-NEXT:    vse32.v v16, (a0)
+; LMULMAX2-NEXT:    vse32.v v14, (a0)
 ; LMULMAX2-NEXT:    addi a0, a1, 32
 ; LMULMAX2-NEXT:    vse32.v v12, (a0)
 ; LMULMAX2-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-int-interleave.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-int-interleave.ll
index 2e97c5114483b..57312808ca9a2 100644
--- a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-int-interleave.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-int-interleave.ll
@@ -52,14 +52,13 @@ define <4 x i64> @interleave_v2i64(<2 x i64> %x, <2 x i64> %y) {
 ; RV32-V128-LABEL: interleave_v2i64:
 ; RV32-V128:       # %bb.0:
 ; RV32-V128-NEXT:    vmv1r.v v12, v9
-; RV32-V128-NEXT:    # kill: def $v8 killed $v8 def $v8m2
 ; RV32-V128-NEXT:    vsetivli zero, 4, e16, mf2, ta, ma
-; RV32-V128-NEXT:    vid.v v10
-; RV32-V128-NEXT:    vsrl.vi v14, v10, 1
+; RV32-V128-NEXT:    vid.v v9
+; RV32-V128-NEXT:    vsrl.vi v14, v9, 1
 ; RV32-V128-NEXT:    vsetvli zero, zero, e64, m2, ta, mu
+; RV32-V128-NEXT:    vrgatherei16.vv v10, v8, v14
 ; RV32-V128-NEXT:    li a0, 10
 ; RV32-V128-NEXT:    vmv.s.x v0, a0
-; RV32-V128-NEXT:    vrgatherei16.vv v10, v8, v14
 ; RV32-V128-NEXT:    vrgatherei16.vv v10, v12, v14, v0.t
 ; RV32-V128-NEXT:    vmv.v.v v8, v10
 ; RV32-V128-NEXT:    ret
@@ -67,13 +66,12 @@ define <4 x i64> @interleave_v2i64(<2 x i64> %x, <2 x i64> %y) {
 ; RV64-V128-LABEL: interleave_v2i64:
 ; RV64-V128:       # %bb.0:
 ; RV64-V128-NEXT:    vmv1r.v v12, v9
-; RV64-V128-NEXT:    # kill: def $v8 killed $v8 def $v8m2
 ; RV64-V128-NEXT:    vsetivli zero, 4, e64, m2, ta, mu
 ; RV64-V128-NEXT:    vid.v v10
 ; RV64-V128-NEXT:    vsrl.vi v14, v10, 1
+; RV64-V128-NEXT:    vrgather.vv v10, v8, v14
 ; RV64-V128-NEXT:    li a0, 10
 ; RV64-V128-NEXT:    vmv.s.x v0, a0
-; RV64-V128-NEXT:    vrgather.vv v10, v8, v14
 ; RV64-V128-NEXT:    vrgather.vv v10, v12, v14, v0.t
 ; RV64-V128-NEXT:    vmv.v.v v8, v10
 ; RV64-V128-NEXT:    ret
@@ -370,9 +368,9 @@ define <64 x i32> @interleave_v32i32(<32 x i32> %x, <32 x i32> %y) {
 ; RV32-V128-NEXT:    vsetvli zero, a1, e32, m8, ta, ma
 ; RV32-V128-NEXT:    vle32.v v0, (a0)
 ; RV32-V128-NEXT:    vmv8r.v v24, v8
-; RV32-V128-NEXT:    addi a0, sp, 16
-; RV32-V128-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; RV32-V128-NEXT:    vrgather.vv v8, v24, v0
+; RV32-V128-NEXT:    addi a0, sp, 16
+; RV32-V128-NEXT:    vs8r.v v24, (a0) # Unknown-size Folded Spill
 ; RV32-V128-NEXT:    lui a0, %hi(.LCPI15_1)
 ; RV32-V128-NEXT:    addi a0, a0, %lo(.LCPI15_1)
 ; RV32-V128-NEXT:    vle32.v v24, (a0)
@@ -421,9 +419,9 @@ define <64 x i32> @interleave_v32i32(<32 x i32> %x, <32 x i32> %y) {
 ; RV64-V128-NEXT:    vsetvli zero, a1, e32, m8, ta, ma
 ; RV64-V128-NEXT:    vle32.v v0, (a0)
 ; RV64-V128-NEXT:    vmv8r.v v24, v8
-; RV64-V128-NEXT:    addi a0, sp, 16
-; RV64-V128-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; RV64-V128-NEXT:    vrgather.vv v8, v24, v0
+; RV64-V128-NEXT:    addi a0, sp, 16
+; RV64-V128-NEXT:    vs8r.v v24, (a0) # Unknown-size Folded Spill
 ; RV64-V128-NEXT:    lui a0, %hi(.LCPI15_1)
 ; RV64-V128-NEXT:    addi a0, a0, %lo(.LCPI15_1)
 ; RV64-V128-NEXT:    vle32.v v24, (a0)
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-int-shuffles.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-int-shuffles.ll
index 9ce01467c83ba..7b28ce7ad654c 100644
--- a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-int-shuffles.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-int-shuffles.ll
@@ -635,7 +635,6 @@ define <4 x i16> @shuffle_shuffle_vslidedown(<16 x i16> %0) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetivli zero, 16, e16, m2, ta, ma
 ; CHECK-NEXT:    vslidedown.vi v8, v8, 5
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8m2
 ; CHECK-NEXT:    ret
 entry:
   %1 = shufflevector <16 x i16> %0, <16 x i16> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-masked-gather.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-masked-gather.ll
index be781a983460c..b531df25cad7b 100644
--- a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-masked-gather.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-masked-gather.ll
@@ -12719,8 +12719,8 @@ define <32 x i8> @mgather_baseidx_v32i8(ptr %base, <32 x i8> %idxs, <32 x i1> %m
 ; RV64ZVE32F-NEXT:    bgez a2, .LBB98_23
 ; RV64ZVE32F-NEXT:  # %bb.22: # %cond.load37
 ; RV64ZVE32F-NEXT:    vsetivli zero, 1, e8, mf4, ta, ma
-; RV64ZVE32F-NEXT:    vslidedown.vi v13, v12, 1
-; RV64ZVE32F-NEXT:    vmv.x.s a2, v13
+; RV64ZVE32F-NEXT:    vslidedown.vi v9, v12, 1
+; RV64ZVE32F-NEXT:    vmv.x.s a2, v9
 ; RV64ZVE32F-NEXT:    add a2, a0, a2
 ; RV64ZVE32F-NEXT:    lb a2, 0(a2)
 ; RV64ZVE32F-NEXT:    li a3, 32
@@ -12731,7 +12731,7 @@ define <32 x i8> @mgather_baseidx_v32i8(ptr %base, <32 x i8> %idxs, <32 x i1> %m
 ; RV64ZVE32F-NEXT:  .LBB98_23: # %else38
 ; RV64ZVE32F-NEXT:    vsetivli zero, 2, e8, mf4, ta, ma
 ; RV64ZVE32F-NEXT:    slli a2, a1, 49
-; RV64ZVE32F-NEXT:    vslidedown.vi v12, v12, 2
+; RV64ZVE32F-NEXT:    vslidedown.vi v9, v12, 2
 ; RV64ZVE32F-NEXT:    bltz a2, .LBB98_55
 ; RV64ZVE32F-NEXT:  # %bb.24: # %else41
 ; RV64ZVE32F-NEXT:    slli a2, a1, 48
@@ -12744,8 +12744,8 @@ define <32 x i8> @mgather_baseidx_v32i8(ptr %base, <32 x i8> %idxs, <32 x i1> %m
 ; RV64ZVE32F-NEXT:    bgez a2, .LBB98_28
 ; RV64ZVE32F-NEXT:  .LBB98_27: # %cond.load49
 ; RV64ZVE32F-NEXT:    vsetivli zero, 1, e8, mf4, ta, ma
-; RV64ZVE32F-NEXT:    vslidedown.vi v12, v8, 1
-; RV64ZVE32F-NEXT:    vmv.x.s a2, v12
+; RV64ZVE32F-NEXT:    vslidedown.vi v9, v8, 1
+; RV64ZVE32F-NEXT:    vmv.x.s a2, v9
 ; RV64ZVE32F-NEXT:    add a2, a0, a2
 ; RV64ZVE32F-NEXT:    lb a2, 0(a2)
 ; RV64ZVE32F-NEXT:    li a3, 32
@@ -12756,10 +12756,10 @@ define <32 x i8> @mgather_baseidx_v32i8(ptr %base, <32 x i8> %idxs, <32 x i1> %m
 ; RV64ZVE32F-NEXT:  .LBB98_28: # %else50
 ; RV64ZVE32F-NEXT:    vsetivli zero, 2, e8, mf4, ta, ma
 ; RV64ZVE32F-NEXT:    slli a2, a1, 45
-; RV64ZVE32F-NEXT:    vslidedown.vi v13, v8, 2
+; RV64ZVE32F-NEXT:    vslidedown.vi v12, v8, 2
 ; RV64ZVE32F-NEXT:    bgez a2, .LBB98_30
 ; RV64ZVE32F-NEXT:  # %bb.29: # %cond.load52
-; RV64ZVE32F-NEXT:    vmv.x.s a2, v13
+; RV64ZVE32F-NEXT:    vmv.x.s a2, v12
 ; RV64ZVE32F-NEXT:    add a2, a0, a2
 ; RV64ZVE32F-NEXT:    lb a2, 0(a2)
 ; RV64ZVE32F-NEXT:    li a3, 32
@@ -12770,7 +12770,7 @@ define <32 x i8> @mgather_baseidx_v32i8(ptr %base, <32 x i8> %idxs, <32 x i1> %m
 ; RV64ZVE32F-NEXT:  .LBB98_30: # %else53
 ; RV64ZVE32F-NEXT:    vsetivli zero, 4, e8, mf2, ta, ma
 ; RV64ZVE32F-NEXT:    slli a2, a1, 44
-; RV64ZVE32F-NEXT:    vslidedown.vi v12, v8, 4
+; RV64ZVE32F-NEXT:    vslidedown.vi v9, v8, 4
 ; RV64ZVE32F-NEXT:    bltz a2, .LBB98_58
 ; RV64ZVE32F-NEXT:  # %bb.31: # %else56
 ; RV64ZVE32F-NEXT:    slli a2, a1, 43
@@ -12780,21 +12780,21 @@ define <32 x i8> @mgather_baseidx_v32i8(ptr %base, <32 x i8> %idxs, <32 x i1> %m
 ; RV64ZVE32F-NEXT:    bgez a2, .LBB98_34
 ; RV64ZVE32F-NEXT:  .LBB98_33: # %cond.load61
 ; RV64ZVE32F-NEXT:    vsetivli zero, 1, e8, mf4, ta, ma
-; RV64ZVE32F-NEXT:    vslidedown.vi v13, v12, 1
-; RV64ZVE32F-NEXT:    vmv.x.s a2, v13
+; RV64ZVE32F-NEXT:    vslidedown.vi v12, v9, 1
+; RV64ZVE32F-NEXT:    vmv.x.s a2, v12
 ; RV64ZVE32F-NEXT:    add a2, a0, a2
 ; RV64ZVE32F-NEXT:    lb a2, 0(a2)
 ; RV64ZVE32F-NEXT:    li a3, 32
 ; RV64ZVE32F-NEXT:    vsetvli zero, a3, e8, m1, ta, ma
-; RV64ZVE32F-NEXT:    vmv.s.x v14, a2
+; RV64ZVE32F-NEXT:    vmv.s.x v12, a2
 ; RV64ZVE32F-NEXT:    vsetivli zero, 22, e8, m2, tu, ma
-; RV64ZVE32F-NEXT:    vslideup.vi v10, v14, 21
+; RV64ZVE32F-NEXT:    vslideup.vi v10, v12, 21
 ; RV64ZVE32F-NEXT:  .LBB98_34: # %else62
 ; RV64ZVE32F-NEXT:    vsetivli zero, 8, e8, m1, ta, ma
 ; RV64ZVE32F-NEXT:    vslidedown.vi v8, v8, 8
 ; RV64ZVE32F-NEXT:    vsetivli zero, 2, e8, mf4, ta, ma
 ; RV64ZVE32F-NEXT:    slli a2, a1, 41
-; RV64ZVE32F-NEXT:    vslidedown.vi v9, v12, 2
+; RV64ZVE32F-NEXT:    vslidedown.vi v9, v9, 2
 ; RV64ZVE32F-NEXT:    bltz a2, .LBB98_60
 ; RV64ZVE32F-NEXT:  # %bb.35: # %else65
 ; RV64ZVE32F-NEXT:    slli a2, a1, 40
@@ -12948,20 +12948,20 @@ define <32 x i8> @mgather_baseidx_v32i8(ptr %base, <32 x i8> %idxs, <32 x i1> %m
 ; RV64ZVE32F-NEXT:    bnez a2, .LBB98_14
 ; RV64ZVE32F-NEXT:    j .LBB98_15
 ; RV64ZVE32F-NEXT:  .LBB98_55: # %cond.load40
-; RV64ZVE32F-NEXT:    vmv.x.s a2, v12
+; RV64ZVE32F-NEXT:    vmv.x.s a2, v9
 ; RV64ZVE32F-NEXT:    add a2, a0, a2
 ; RV64ZVE32F-NEXT:    lb a2, 0(a2)
 ; RV64ZVE32F-NEXT:    li a3, 32
 ; RV64ZVE32F-NEXT:    vsetvli zero, a3, e8, m1, ta, ma
-; RV64ZVE32F-NEXT:    vmv.s.x v14, a2
+; RV64ZVE32F-NEXT:    vmv.s.x v12, a2
 ; RV64ZVE32F-NEXT:    vsetivli zero, 15, e8, m2, tu, ma
-; RV64ZVE32F-NEXT:    vslideup.vi v10, v14, 14
+; RV64ZVE32F-NEXT:    vslideup.vi v10, v12, 14
 ; RV64ZVE32F-NEXT:    slli a2, a1, 48
 ; RV64ZVE32F-NEXT:    bgez a2, .LBB98_25
 ; RV64ZVE32F-NEXT:  .LBB98_56: # %cond.load43
 ; RV64ZVE32F-NEXT:    vsetivli zero, 1, e8, mf4, ta, ma
-; RV64ZVE32F-NEXT:    vslidedown.vi v12, v12, 1
-; RV64ZVE32F-NEXT:    vmv.x.s a2, v12
+; RV64ZVE32F-NEXT:    vslidedown.vi v9, v9, 1
+; RV64ZVE32F-NEXT:    vmv.x.s a2, v9
 ; RV64ZVE32F-NEXT:    add a2, a0, a2
 ; RV64ZVE32F-NEXT:    lb a2, 0(a2)
 ; RV64ZVE32F-NEXT:    li a3, 32
@@ -12986,27 +12986,27 @@ define <32 x i8> @mgather_baseidx_v32i8(ptr %base, <32 x i8> %idxs, <32 x i1> %m
 ; RV64ZVE32F-NEXT:    j .LBB98_28
 ; RV64ZVE32F-NEXT:  .LBB98_58: # %cond.load55
 ; RV64ZVE32F-NEXT:    vsetivli zero, 1, e8, mf4, ta, ma
-; RV64ZVE32F-NEXT:    vslidedown.vi v13, v13, 1
-; RV64ZVE32F-NEXT:    vmv.x.s a2, v13
+; RV64ZVE32F-NEXT:    vslidedown.vi v12, v12, 1
+; RV64ZVE32F-NEXT:    vmv.x.s a2, v12
 ; RV64ZVE32F-NEXT:    add a2, a0, a2
 ; RV64ZVE32F-NEXT:    lb a2, 0(a2)
 ; RV64ZVE32F-NEXT:    li a3, 32
 ; RV64ZVE32F-NEXT:    vsetvli zero, a3, e8, m1, ta, ma
-; RV64ZVE32F-NEXT:    vmv.s.x v14, a2
+; RV64ZVE32F-NEXT:    vmv.s.x v12, a2
 ; RV64ZVE32F-NEXT:    vsetivli zero, 20, e8, m2, tu, ma
-; RV64ZVE32F-NEXT:    vslideup.vi v10, v14, 19
+; RV64ZVE32F-NEXT:    vslideup.vi v10, v12, 19
 ; RV64ZVE32F-NEXT:    slli a2, a1, 43
 ; RV64ZVE32F-NEXT:    bgez a2, .LBB98_32
 ; RV64ZVE32F-NEXT:  .LBB98_59: # %cond.load58
 ; RV64ZVE32F-NEXT:    vsetivli zero, 0, e8, mf4, ta, ma
-; RV64ZVE32F-NEXT:    vmv.x.s a2, v12
+; RV64ZVE32F-NEXT:    vmv.x.s a2, v9
 ; RV64ZVE32F-NEXT:    add a2, a0, a2
 ; RV64ZVE32F-NEXT:    lb a2, 0(a2)
 ; RV64ZVE32F-NEXT:    li a3, 32
 ; RV64ZVE32F-NEXT:    vsetvli zero, a3, e8, m1, ta, ma
-; RV64ZVE32F-NEXT:    vmv.s.x v14, a2
+; RV64ZVE32F-NEXT:    vmv.s.x v12, a2
 ; RV64ZVE32F-NEXT:    vsetivli zero, 21, e8, m2, tu, ma
-; RV64ZVE32F-NEXT:    vslideup.vi v10, v14, 20
+; RV64ZVE32F-NEXT:    vslideup.vi v10, v12, 20
 ; RV64ZVE32F-NEXT:    slli a2, a1, 42
 ; RV64ZVE32F-NEXT:    bltz a2, .LBB98_33
 ; RV64ZVE32F-NEXT:    j .LBB98_34
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-masked-scatter.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-masked-scatter.ll
index aecbd1fbc5f15..65fe75047526d 100644
--- a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-masked-scatter.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-masked-scatter.ll
@@ -11089,8 +11089,8 @@ define void @mscatter_baseidx_v32i8(<32 x i8> %val, ptr %base, <32 x i8> %idxs,
 ; RV64ZVE32F-NEXT:    bgez a2, .LBB92_23
 ; RV64ZVE32F-NEXT:  # %bb.22: # %cond.store25
 ; RV64ZVE32F-NEXT:    vsetivli zero, 1, e8, mf4, ta, ma
-; RV64ZVE32F-NEXT:    vslidedown.vi v13, v12, 1
-; RV64ZVE32F-NEXT:    vmv.x.s a2, v13
+; RV64ZVE32F-NEXT:    vslidedown.vi v11, v12, 1
+; RV64ZVE32F-NEXT:    vmv.x.s a2, v11
 ; RV64ZVE32F-NEXT:    add a2, a0, a2
 ; RV64ZVE32F-NEXT:    vsetivli zero, 1, e8, m2, ta, ma
 ; RV64ZVE32F-NEXT:    vslidedown.vi v14, v8, 13
@@ -11098,7 +11098,7 @@ define void @mscatter_baseidx_v32i8(<32 x i8> %val, ptr %base, <32 x i8> %idxs,
 ; RV64ZVE32F-NEXT:  .LBB92_23: # %else26
 ; RV64ZVE32F-NEXT:    vsetivli zero, 2, e8, mf4, ta, ma
 ; RV64ZVE32F-NEXT:    slli a2, a1, 49
-; RV64ZVE32F-NEXT:    vslidedown.vi v12, v12, 2
+; RV64ZVE32F-NEXT:    vslidedown.vi v11, v12, 2
 ; RV64ZVE32F-NEXT:    bltz a2, .LBB92_55
 ; RV64ZVE32F-NEXT:  # %bb.24: # %else28
 ; RV64ZVE32F-NEXT:    slli a2, a1, 48
@@ -11111,8 +11111,8 @@ define void @mscatter_baseidx_v32i8(<32 x i8> %val, ptr %base, <32 x i8> %idxs,
 ; RV64ZVE32F-NEXT:    bgez a2, .LBB92_28
 ; RV64ZVE32F-NEXT:  .LBB92_27: # %cond.store33
 ; RV64ZVE32F-NEXT:    vsetivli zero, 1, e8, mf4, ta, ma
-; RV64ZVE32F-NEXT:    vslidedown.vi v12, v10, 1
-; RV64ZVE32F-NEXT:    vmv.x.s a2, v12
+; RV64ZVE32F-NEXT:    vslidedown.vi v11, v10, 1
+; RV64ZVE32F-NEXT:    vmv.x.s a2, v11
 ; RV64ZVE32F-NEXT:    add a2, a0, a2
 ; RV64ZVE32F-NEXT:    vsetivli zero, 1, e8, m2, ta, ma
 ; RV64ZVE32F-NEXT:    vslidedown.vi v12, v8, 17
@@ -11120,10 +11120,10 @@ define void @mscatter_baseidx_v32i8(<32 x i8> %val, ptr %base, <32 x i8> %idxs,
 ; RV64ZVE32F-NEXT:  .LBB92_28: # %else34
 ; RV64ZVE32F-NEXT:    vsetivli zero, 2, e8, mf4, ta, ma
 ; RV64ZVE32F-NEXT:    slli a2, a1, 45
-; RV64ZVE32F-NEXT:    vslidedown.vi v13, v10, 2
+; RV64ZVE32F-NEXT:    vslidedown.vi v12, v10, 2
 ; RV64ZVE32F-NEXT:    bgez a2, .LBB92_30
 ; RV64ZVE32F-NEXT:  # %bb.29: # %cond.store35
-; RV64ZVE32F-NEXT:    vmv.x.s a2, v13
+; RV64ZVE32F-NEXT:    vmv.x.s a2, v12
 ; RV64ZVE32F-NEXT:    add a2, a0, a2
 ; RV64ZVE32F-NEXT:    vsetivli zero, 1, e8, m2, ta, ma
 ; RV64ZVE32F-NEXT:    vslidedown.vi v14, v8, 18
@@ -11131,7 +11131,7 @@ define void @mscatter_baseidx_v32i8(<32 x i8> %val, ptr %base, <32 x i8> %idxs,
 ; RV64ZVE32F-NEXT:  .LBB92_30: # %else36
 ; RV64ZVE32F-NEXT:    vsetivli zero, 4, e8, mf2, ta, ma
 ; RV64ZVE32F-NEXT:    slli a2, a1, 44
-; RV64ZVE32F-NEXT:    vslidedown.vi v12, v10, 4
+; RV64ZVE32F-NEXT:    vslidedown.vi v11, v10, 4
 ; RV64ZVE32F-NEXT:    bltz a2, .LBB92_58
 ; RV64ZVE32F-NEXT:  # %bb.31: # %else38
 ; RV64ZVE32F-NEXT:    slli a2, a1, 43
@@ -11141,18 +11141,18 @@ define void @mscatter_baseidx_v32i8(<32 x i8> %val, ptr %base, <32 x i8> %idxs,
 ; RV64ZVE32F-NEXT:    bgez a2, .LBB92_34
 ; RV64ZVE32F-NEXT:  .LBB92_33: # %cond.store41
 ; RV64ZVE32F-NEXT:    vsetivli zero, 1, e8, mf4, ta, ma
-; RV64ZVE32F-NEXT:    vslidedown.vi v13, v12, 1
-; RV64ZVE32F-NEXT:    vmv.x.s a2, v13
+; RV64ZVE32F-NEXT:    vslidedown.vi v12, v11, 1
+; RV64ZVE32F-NEXT:    vmv.x.s a2, v12
 ; RV64ZVE32F-NEXT:    add a2, a0, a2
 ; RV64ZVE32F-NEXT:    vsetivli zero, 1, e8, m2, ta, ma
-; RV64ZVE32F-NEXT:    vslidedown.vi v14, v8, 21
-; RV64ZVE32F-NEXT:    vse8.v v14, (a2)
+; RV64ZVE32F-NEXT:    vslidedown.vi v12, v8, 21
+; RV64ZVE32F-NEXT:    vse8.v v12, (a2)
 ; RV64ZVE32F-NEXT:  .LBB92_34: # %else42
 ; RV64ZVE32F-NEXT:    vsetivli zero, 8, e8, m1, ta, ma
 ; RV64ZVE32F-NEXT:    vslidedown.vi v10, v10, 8
 ; RV64ZVE32F-NEXT:    vsetivli zero, 2, e8, mf4, ta, ma
 ; RV64ZVE32F-NEXT:    slli a2, a1, 41
-; RV64ZVE32F-NEXT:    vslidedown.vi v11, v12, 2
+; RV64ZVE32F-NEXT:    vslidedown.vi v11, v11, 2
 ; RV64ZVE32F-NEXT:    bltz a2, .LBB92_60
 ; RV64ZVE32F-NEXT:  # %bb.35: # %else44
 ; RV64ZVE32F-NEXT:    slli a2, a1, 40
@@ -11275,17 +11275,17 @@ define void @mscatter_baseidx_v32i8(<32 x i8> %val, ptr %base, <32 x i8> %idxs,
 ; RV64ZVE32F-NEXT:    bnez a2, .LBB92_14
 ; RV64ZVE32F-NEXT:    j .LBB92_15
 ; RV64ZVE32F-NEXT:  .LBB92_55: # %cond.store27
-; RV64ZVE32F-NEXT:    vmv.x.s a2, v12
+; RV64ZVE32F-NEXT:    vmv.x.s a2, v11
 ; RV64ZVE32F-NEXT:    add a2, a0, a2
 ; RV64ZVE32F-NEXT:    vsetivli zero, 1, e8, m2, ta, ma
-; RV64ZVE32F-NEXT:    vslidedown.vi v14, v8, 14
-; RV64ZVE32F-NEXT:    vse8.v v14, (a2)
+; RV64ZVE32F-NEXT:    vslidedown.vi v12, v8, 14
+; RV64ZVE32F-NEXT:    vse8.v v12, (a2)
 ; RV64ZVE32F-NEXT:    slli a2, a1, 48
 ; RV64ZVE32F-NEXT:    bgez a2, .LBB92_25
 ; RV64ZVE32F-NEXT:  .LBB92_56: # %cond.store29
 ; RV64ZVE32F-NEXT:    vsetivli zero, 1, e8, mf4, ta, ma
-; RV64ZVE32F-NEXT:    vslidedown.vi v12, v12, 1
-; RV64ZVE32F-NEXT:    vmv.x.s a2, v12
+; RV64ZVE32F-NEXT:    vslidedown.vi v11, v11, 1
+; RV64ZVE32F-NEXT:    vmv.x.s a2, v11
 ; RV64ZVE32F-NEXT:    add a2, a0, a2
 ; RV64ZVE32F-NEXT:    vsetivli zero, 1, e8, m2, ta, ma
 ; RV64ZVE32F-NEXT:    vslidedown.vi v12, v8, 15
@@ -11304,21 +11304,21 @@ define void @mscatter_baseidx_v32i8(<32 x i8> %val, ptr %base, <32 x i8> %idxs,
 ; RV64ZVE32F-NEXT:    j .LBB92_28
 ; RV64ZVE32F-NEXT:  .LBB92_58: # %cond.store37
 ; RV64ZVE32F-NEXT:    vsetivli zero, 1, e8, mf4, ta, ma
-; RV64ZVE32F-NEXT:    vslidedown.vi v13, v13, 1
-; RV64ZVE32F-NEXT:    vmv.x.s a2, v13
+; RV64ZVE32F-NEXT:    vslidedown.vi v12, v12, 1
+; RV64ZVE32F-NEXT:    vmv.x.s a2, v12
 ; RV64ZVE32F-NEXT:    add a2, a0, a2
 ; RV64ZVE32F-NEXT:    vsetivli zero, 1, e8, m2, ta, ma
-; RV64ZVE32F-NEXT:    vslidedown.vi v14, v8, 19
-; RV64ZVE32F-NEXT:    vse8.v v14, (a2)
+; RV64ZVE32F-NEXT:    vslidedown.vi v12, v8, 19
+; RV64ZVE32F-NEXT:    vse8.v v12, (a2)
 ; RV64ZVE32F-NEXT:    slli a2, a1, 43
 ; RV64ZVE32F-NEXT:    bgez a2, .LBB92_32
 ; RV64ZVE32F-NEXT:  .LBB92_59: # %cond.store39
 ; RV64ZVE32F-NEXT:    vsetivli zero, 0, e8, mf4, ta, ma
-; RV64ZVE32F-NEXT:    vmv.x.s a2, v12
+; RV64ZVE32F-NEXT:    vmv.x.s a2, v11
 ; RV64ZVE32F-NEXT:    add a2, a0, a2
 ; RV64ZVE32F-NEXT:    vsetivli zero, 1, e8, m2, ta, ma
-; RV64ZVE32F-NEXT:    vslidedown.vi v14, v8, 20
-; RV64ZVE32F-NEXT:    vse8.v v14, (a2)
+; RV64ZVE32F-NEXT:    vslidedown.vi v12, v8, 20
+; RV64ZVE32F-NEXT:    vse8.v v12, (a2)
 ; RV64ZVE32F-NEXT:    slli a2, a1, 42
 ; RV64ZVE32F-NEXT:    bltz a2, .LBB92_33
 ; RV64ZVE32F-NEXT:    j .LBB92_34
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-reduction-fp.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-reduction-fp.ll
index 8bad561fadfba..3feb976fc0e18 100644
--- a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-reduction-fp.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-reduction-fp.ll
@@ -693,11 +693,11 @@ define float @vreduce_fwadd_v64f32(ptr %x, float %s) {
 ; CHECK-NEXT:    vsetvli zero, a0, e16, m8, ta, ma
 ; CHECK-NEXT:    vslidedown.vx v16, v8, a0
 ; CHECK-NEXT:    vsetvli zero, a0, e32, m1, ta, ma
-; CHECK-NEXT:    vfmv.s.f v24, fa0
+; CHECK-NEXT:    vfmv.s.f v12, fa0
 ; CHECK-NEXT:    vsetvli zero, a0, e16, m4, ta, ma
-; CHECK-NEXT:    vfwadd.vv v0, v8, v16
+; CHECK-NEXT:    vfwadd.vv v24, v8, v16
 ; CHECK-NEXT:    vsetvli zero, zero, e32, m8, ta, ma
-; CHECK-NEXT:    vfredusum.vs v8, v0, v24
+; CHECK-NEXT:    vfredusum.vs v8, v24, v12
 ; CHECK-NEXT:    vfmv.f.s fa0, v8
 ; CHECK-NEXT:    ret
   %v = load <64 x half>, ptr %x
@@ -716,9 +716,9 @@ define float @vreduce_ord_fwadd_v64f32(ptr %x, float %s) {
 ; CHECK-NEXT:    vsetvli zero, a0, e16, m8, ta, ma
 ; CHECK-NEXT:    vslidedown.vx v16, v8, a0
 ; CHECK-NEXT:    vsetvli zero, a0, e32, m1, ta, ma
-; CHECK-NEXT:    vfmv.s.f v24, fa0
+; CHECK-NEXT:    vfmv.s.f v12, fa0
 ; CHECK-NEXT:    vsetvli zero, a0, e16, m4, ta, ma
-; CHECK-NEXT:    vfwredosum.vs v8, v8, v24
+; CHECK-NEXT:    vfwredosum.vs v8, v8, v12
 ; CHECK-NEXT:    vfwredosum.vs v8, v16, v8
 ; CHECK-NEXT:    vsetivli zero, 0, e32, m1, ta, ma
 ; CHECK-NEXT:    vfmv.f.s fa0, v8
@@ -1119,9 +1119,9 @@ define double @vreduce_ord_fwadd_v32f64(ptr %x, double %s) {
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m8, ta, ma
 ; CHECK-NEXT:    vslidedown.vi v16, v8, 16
 ; CHECK-NEXT:    vsetivli zero, 16, e64, m1, ta, ma
-; CHECK-NEXT:    vfmv.s.f v24, fa0
+; CHECK-NEXT:    vfmv.s.f v12, fa0
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
-; CHECK-NEXT:    vfwredosum.vs v8, v8, v24
+; CHECK-NEXT:    vfwredosum.vs v8, v8, v12
 ; CHECK-NEXT:    vfwredosum.vs v8, v16, v8
 ; CHECK-NEXT:    vsetivli zero, 0, e64, m1, ta, ma
 ; CHECK-NEXT:    vfmv.f.s fa0, v8
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-reduction-int.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-reduction-int.ll
index 45e91c50447b2..22b7841e9924c 100644
--- a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-reduction-int.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-reduction-int.ll
@@ -535,11 +535,11 @@ define i16 @vwreduce_add_v128i16(ptr %x) {
 ; CHECK-NEXT:    vsetvli zero, a0, e8, m8, ta, ma
 ; CHECK-NEXT:    vslidedown.vx v16, v8, a0
 ; CHECK-NEXT:    vsetvli zero, a0, e16, m1, ta, ma
-; CHECK-NEXT:    vmv.s.x v24, zero
+; CHECK-NEXT:    vmv.s.x v12, zero
 ; CHECK-NEXT:    vsetvli zero, a0, e8, m4, ta, ma
-; CHECK-NEXT:    vwadd.vv v0, v8, v16
+; CHECK-NEXT:    vwadd.vv v24, v8, v16
 ; CHECK-NEXT:    vsetvli zero, zero, e16, m8, ta, ma
-; CHECK-NEXT:    vredsum.vs v8, v0, v24
+; CHECK-NEXT:    vredsum.vs v8, v24, v12
 ; CHECK-NEXT:    vmv.x.s a0, v8
 ; CHECK-NEXT:    ret
   %v = load <128 x i8>, ptr %x
@@ -558,11 +558,11 @@ define i16 @vwreduce_uadd_v128i16(ptr %x) {
 ; CHECK-NEXT:    vsetvli zero, a0, e8, m8, ta, ma
 ; CHECK-NEXT:    vslidedown.vx v16, v8, a0
 ; CHECK-NEXT:    vsetvli zero, a0, e16, m1, ta, ma
-; CHECK-NEXT:    vmv.s.x v24, zero
+; CHECK-NEXT:    vmv.s.x v12, zero
 ; CHECK-NEXT:    vsetvli zero, a0, e8, m4, ta, ma
-; CHECK-NEXT:    vwaddu.vv v0, v8, v16
+; CHECK-NEXT:    vwaddu.vv v24, v8, v16
 ; CHECK-NEXT:    vsetvli zero, zero, e16, m8, ta, ma
-; CHECK-NEXT:    vredsum.vs v8, v0, v24
+; CHECK-NEXT:    vredsum.vs v8, v24, v12
 ; CHECK-NEXT:    vmv.x.s a0, v8
 ; CHECK-NEXT:    ret
   %v = load <128 x i8>, ptr %x
@@ -902,11 +902,11 @@ define i32 @vwreduce_add_v64i32(ptr %x) {
 ; CHECK-NEXT:    vsetvli zero, a0, e16, m8, ta, ma
 ; CHECK-NEXT:    vslidedown.vx v16, v8, a0
 ; CHECK-NEXT:    vsetvli zero, a0, e32, m1, ta, ma
-; CHECK-NEXT:    vmv.s.x v24, zero
+; CHECK-NEXT:    vmv.s.x v12, zero
 ; CHECK-NEXT:    vsetvli zero, a0, e16, m4, ta, ma
-; CHECK-NEXT:    vwadd.vv v0, v8, v16
+; CHECK-NEXT:    vwadd.vv v24, v8, v16
 ; CHECK-NEXT:    vsetvli zero, zero, e32, m8, ta, ma
-; CHECK-NEXT:    vredsum.vs v8, v0, v24
+; CHECK-NEXT:    vredsum.vs v8, v24, v12
 ; CHECK-NEXT:    vmv.x.s a0, v8
 ; CHECK-NEXT:    ret
   %v = load <64 x i16>, ptr %x
@@ -925,11 +925,11 @@ define i32 @vwreduce_uadd_v64i32(ptr %x) {
 ; CHECK-NEXT:    vsetvli zero, a0, e16, m8, ta, ma
 ; CHECK-NEXT:    vslidedown.vx v16, v8, a0
 ; CHECK-NEXT:    vsetvli zero, a0, e32, m1, ta, ma
-; CHECK-NEXT:    vmv.s.x v24, zero
+; CHECK-NEXT:    vmv.s.x v12, zero
 ; CHECK-NEXT:    vsetvli zero, a0, e16, m4, ta, ma
-; CHECK-NEXT:    vwaddu.vv v0, v8, v16
+; CHECK-NEXT:    vwaddu.vv v24, v8, v16
 ; CHECK-NEXT:    vsetvli zero, zero, e32, m8, ta, ma
-; CHECK-NEXT:    vredsum.vs v8, v0, v24
+; CHECK-NEXT:    vredsum.vs v8, v24, v12
 ; CHECK-NEXT:    vmv.x.s a0, v8
 ; CHECK-NEXT:    ret
   %v = load <64 x i16>, ptr %x
@@ -1556,45 +1556,24 @@ define i64 @vwreduce_add_v64i64(ptr %x) {
 ; RV32-NEXT:    addi sp, sp, -16
 ; RV32-NEXT:    .cfi_def_cfa_offset 16
 ; RV32-NEXT:    csrr a1, vlenb
-; RV32-NEXT:    slli a1, a1, 5
+; RV32-NEXT:    slli a1, a1, 4
 ; RV32-NEXT:    sub sp, sp, a1
-; RV32-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x20, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 32 * vlenb
+; RV32-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x10, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 16 * vlenb
 ; RV32-NEXT:    addi a1, a0, 128
 ; RV32-NEXT:    li a2, 32
 ; RV32-NEXT:    vsetvli zero, a2, e32, m8, ta, ma
 ; RV32-NEXT:    vle32.v v8, (a0)
-; RV32-NEXT:    addi a0, sp, 16
-; RV32-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; RV32-NEXT:    vle32.v v16, (a1)
 ; RV32-NEXT:    vsetivli zero, 16, e32, m8, ta, ma
 ; RV32-NEXT:    vslidedown.vi v24, v8, 16
-; RV32-NEXT:    csrr a0, vlenb
-; RV32-NEXT:    li a1, 24
-; RV32-NEXT:    mul a0, a0, a1
-; RV32-NEXT:    add a0, sp, a0
-; RV32-NEXT:    addi a0, a0, 16
-; RV32-NEXT:    vs8r.v v24, (a0) # Unknown-size Folded Spill
-; RV32-NEXT:    vslidedown.vi v24, v16, 16
-; RV32-NEXT:    csrr a0, vlenb
-; RV32-NEXT:    slli a0, a0, 3
-; RV32-NEXT:    add a0, sp, a0
-; RV32-NEXT:    addi a0, a0, 16
-; RV32-NEXT:    vs8r.v v24, (a0) # Unknown-size Folded Spill
+; RV32-NEXT:    addi a0, sp, 16
+; RV32-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
+; RV32-NEXT:    vslidedown.vi v0, v16, 16
 ; RV32-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
-; RV32-NEXT:    csrr a0, vlenb
-; RV32-NEXT:    li a1, 24
-; RV32-NEXT:    mul a0, a0, a1
-; RV32-NEXT:    add a0, sp, a0
-; RV32-NEXT:    addi a0, a0, 16
-; RV32-NEXT:    vl8r.v v24, (a0) # Unknown-size Folded Reload
-; RV32-NEXT:    csrr a0, vlenb
-; RV32-NEXT:    slli a0, a0, 3
-; RV32-NEXT:    add a0, sp, a0
-; RV32-NEXT:    addi a0, a0, 16
-; RV32-NEXT:    vl8r.v v8, (a0) # Unknown-size Folded Reload
+; RV32-NEXT:    vmv4r.v v8, v0
 ; RV32-NEXT:    vwadd.vv v0, v24, v8
 ; RV32-NEXT:    csrr a0, vlenb
-; RV32-NEXT:    slli a0, a0, 4
+; RV32-NEXT:    slli a0, a0, 3
 ; RV32-NEXT:    add a0, sp, a0
 ; RV32-NEXT:    addi a0, a0, 16
 ; RV32-NEXT:    vs8r.v v0, (a0) # Unknown-size Folded Spill
@@ -1603,7 +1582,7 @@ define i64 @vwreduce_add_v64i64(ptr %x) {
 ; RV32-NEXT:    vwadd.vv v0, v8, v16
 ; RV32-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; RV32-NEXT:    csrr a0, vlenb
-; RV32-NEXT:    slli a0, a0, 4
+; RV32-NEXT:    slli a0, a0, 3
 ; RV32-NEXT:    add a0, sp, a0
 ; RV32-NEXT:    addi a0, a0, 16
 ; RV32-NEXT:    vl8r.v v8, (a0) # Unknown-size Folded Reload
@@ -1615,7 +1594,7 @@ define i64 @vwreduce_add_v64i64(ptr %x) {
 ; RV32-NEXT:    vsrl.vx v8, v8, a2
 ; RV32-NEXT:    vmv.x.s a1, v8
 ; RV32-NEXT:    csrr a2, vlenb
-; RV32-NEXT:    slli a2, a2, 5
+; RV32-NEXT:    slli a2, a2, 4
 ; RV32-NEXT:    add sp, sp, a2
 ; RV32-NEXT:    addi sp, sp, 16
 ; RV32-NEXT:    ret
@@ -1625,45 +1604,24 @@ define i64 @vwreduce_add_v64i64(ptr %x) {
 ; RV64-NEXT:    addi sp, sp, -16
 ; RV64-NEXT:    .cfi_def_cfa_offset 16
 ; RV64-NEXT:    csrr a1, vlenb
-; RV64-NEXT:    slli a1, a1, 5
+; RV64-NEXT:    slli a1, a1, 4
 ; RV64-NEXT:    sub sp, sp, a1
-; RV64-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x20, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 32 * vlenb
+; RV64-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x10, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 16 * vlenb
 ; RV64-NEXT:    addi a1, a0, 128
 ; RV64-NEXT:    li a2, 32
 ; RV64-NEXT:    vsetvli zero, a2, e32, m8, ta, ma
 ; RV64-NEXT:    vle32.v v8, (a0)
-; RV64-NEXT:    addi a0, sp, 16
-; RV64-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; RV64-NEXT:    vle32.v v16, (a1)
 ; RV64-NEXT:    vsetivli zero, 16, e32, m8, ta, ma
 ; RV64-NEXT:    vslidedown.vi v24, v8, 16
-; RV64-NEXT:    csrr a0, vlenb
-; RV64-NEXT:    li a1, 24
-; RV64-NEXT:    mul a0, a0, a1
-; RV64-NEXT:    add a0, sp, a0
-; RV64-NEXT:    addi a0, a0, 16
-; RV64-NEXT:    vs8r.v v24, (a0) # Unknown-size Folded Spill
-; RV64-NEXT:    vslidedown.vi v24, v16, 16
-; RV64-NEXT:    csrr a0, vlenb
-; RV64-NEXT:    slli a0, a0, 3
-; RV64-NEXT:    add a0, sp, a0
-; RV64-NEXT:    addi a0, a0, 16
-; RV64-NEXT:    vs8r.v v24, (a0) # Unknown-size Folded Spill
+; RV64-NEXT:    addi a0, sp, 16
+; RV64-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
+; RV64-NEXT:    vslidedown.vi v0, v16, 16
 ; RV64-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
-; RV64-NEXT:    csrr a0, vlenb
-; RV64-NEXT:    li a1, 24
-; RV64-NEXT:    mul a0, a0, a1
-; RV64-NEXT:    add a0, sp, a0
-; RV64-NEXT:    addi a0, a0, 16
-; RV64-NEXT:    vl8r.v v24, (a0) # Unknown-size Folded Reload
-; RV64-NEXT:    csrr a0, vlenb
-; RV64-NEXT:    slli a0, a0, 3
-; RV64-NEXT:    add a0, sp, a0
-; RV64-NEXT:    addi a0, a0, 16
-; RV64-NEXT:    vl8r.v v8, (a0) # Unknown-size Folded Reload
+; RV64-NEXT:    vmv4r.v v8, v0
 ; RV64-NEXT:    vwadd.vv v0, v24, v8
 ; RV64-NEXT:    csrr a0, vlenb
-; RV64-NEXT:    slli a0, a0, 4
+; RV64-NEXT:    slli a0, a0, 3
 ; RV64-NEXT:    add a0, sp, a0
 ; RV64-NEXT:    addi a0, a0, 16
 ; RV64-NEXT:    vs8r.v v0, (a0) # Unknown-size Folded Spill
@@ -1672,7 +1630,7 @@ define i64 @vwreduce_add_v64i64(ptr %x) {
 ; RV64-NEXT:    vwadd.vv v0, v8, v16
 ; RV64-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; RV64-NEXT:    csrr a0, vlenb
-; RV64-NEXT:    slli a0, a0, 4
+; RV64-NEXT:    slli a0, a0, 3
 ; RV64-NEXT:    add a0, sp, a0
 ; RV64-NEXT:    addi a0, a0, 16
 ; RV64-NEXT:    vl8r.v v8, (a0) # Unknown-size Folded Reload
@@ -1681,7 +1639,7 @@ define i64 @vwreduce_add_v64i64(ptr %x) {
 ; RV64-NEXT:    vredsum.vs v8, v8, v16
 ; RV64-NEXT:    vmv.x.s a0, v8
 ; RV64-NEXT:    csrr a1, vlenb
-; RV64-NEXT:    slli a1, a1, 5
+; RV64-NEXT:    slli a1, a1, 4
 ; RV64-NEXT:    add sp, sp, a1
 ; RV64-NEXT:    addi sp, sp, 16
 ; RV64-NEXT:    ret
@@ -1697,45 +1655,24 @@ define i64 @vwreduce_uadd_v64i64(ptr %x) {
 ; RV32-NEXT:    addi sp, sp, -16
 ; RV32-NEXT:    .cfi_def_cfa_offset 16
 ; RV32-NEXT:    csrr a1, vlenb
-; RV32-NEXT:    slli a1, a1, 5
+; RV32-NEXT:    slli a1, a1, 4
 ; RV32-NEXT:    sub sp, sp, a1
-; RV32-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x20, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 32 * vlenb
+; RV32-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x10, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 16 * vlenb
 ; RV32-NEXT:    addi a1, a0, 128
 ; RV32-NEXT:    li a2, 32
 ; RV32-NEXT:    vsetvli zero, a2, e32, m8, ta, ma
 ; RV32-NEXT:    vle32.v v8, (a0)
-; RV32-NEXT:    addi a0, sp, 16
-; RV32-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; RV32-NEXT:    vle32.v v16, (a1)
 ; RV32-NEXT:    vsetivli zero, 16, e32, m8, ta, ma
 ; RV32-NEXT:    vslidedown.vi v24, v8, 16
-; RV32-NEXT:    csrr a0, vlenb
-; RV32-NEXT:    li a1, 24
-; RV32-NEXT:    mul a0, a0, a1
-; RV32-NEXT:    add a0, sp, a0
-; RV32-NEXT:    addi a0, a0, 16
-; RV32-NEXT:    vs8r.v v24, (a0) # Unknown-size Folded Spill
-; RV32-NEXT:    vslidedown.vi v24, v16, 16
-; RV32-NEXT:    csrr a0, vlenb
-; RV32-NEXT:    slli a0, a0, 3
-; RV32-NEXT:    add a0, sp, a0
-; RV32-NEXT:    addi a0, a0, 16
-; RV32-NEXT:    vs8r.v v24, (a0) # Unknown-size Folded Spill
+; RV32-NEXT:    addi a0, sp, 16
+; RV32-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
+; RV32-NEXT:    vslidedown.vi v0, v16, 16
 ; RV32-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
-; RV32-NEXT:    csrr a0, vlenb
-; RV32-NEXT:    li a1, 24
-; RV32-NEXT:    mul a0, a0, a1
-; RV32-NEXT:    add a0, sp, a0
-; RV32-NEXT:    addi a0, a0, 16
-; RV32-NEXT:    vl8r.v v24, (a0) # Unknown-size Folded Reload
-; RV32-NEXT:    csrr a0, vlenb
-; RV32-NEXT:    slli a0, a0, 3
-; RV32-NEXT:    add a0, sp, a0
-; RV32-NEXT:    addi a0, a0, 16
-; RV32-NEXT:    vl8r.v v8, (a0) # Unknown-size Folded Reload
+; RV32-NEXT:    vmv4r.v v8, v0
 ; RV32-NEXT:    vwaddu.vv v0, v24, v8
 ; RV32-NEXT:    csrr a0, vlenb
-; RV32-NEXT:    slli a0, a0, 4
+; RV32-NEXT:    slli a0, a0, 3
 ; RV32-NEXT:    add a0, sp, a0
 ; RV32-NEXT:    addi a0, a0, 16
 ; RV32-NEXT:    vs8r.v v0, (a0) # Unknown-size Folded Spill
@@ -1744,7 +1681,7 @@ define i64 @vwreduce_uadd_v64i64(ptr %x) {
 ; RV32-NEXT:    vwaddu.vv v0, v8, v16
 ; RV32-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; RV32-NEXT:    csrr a0, vlenb
-; RV32-NEXT:    slli a0, a0, 4
+; RV32-NEXT:    slli a0, a0, 3
 ; RV32-NEXT:    add a0, sp, a0
 ; RV32-NEXT:    addi a0, a0, 16
 ; RV32-NEXT:    vl8r.v v8, (a0) # Unknown-size Folded Reload
@@ -1756,7 +1693,7 @@ define i64 @vwreduce_uadd_v64i64(ptr %x) {
 ; RV32-NEXT:    vsrl.vx v8, v8, a2
 ; RV32-NEXT:    vmv.x.s a1, v8
 ; RV32-NEXT:    csrr a2, vlenb
-; RV32-NEXT:    slli a2, a2, 5
+; RV32-NEXT:    slli a2, a2, 4
 ; RV32-NEXT:    add sp, sp, a2
 ; RV32-NEXT:    addi sp, sp, 16
 ; RV32-NEXT:    ret
@@ -1766,45 +1703,24 @@ define i64 @vwreduce_uadd_v64i64(ptr %x) {
 ; RV64-NEXT:    addi sp, sp, -16
 ; RV64-NEXT:    .cfi_def_cfa_offset 16
 ; RV64-NEXT:    csrr a1, vlenb
-; RV64-NEXT:    slli a1, a1, 5
+; RV64-NEXT:    slli a1, a1, 4
 ; RV64-NEXT:    sub sp, sp, a1
-; RV64-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x20, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 32 * vlenb
+; RV64-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x10, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 16 * vlenb
 ; RV64-NEXT:    addi a1, a0, 128
 ; RV64-NEXT:    li a2, 32
 ; RV64-NEXT:    vsetvli zero, a2, e32, m8, ta, ma
 ; RV64-NEXT:    vle32.v v8, (a0)
-; RV64-NEXT:    addi a0, sp, 16
-; RV64-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; RV64-NEXT:    vle32.v v16, (a1)
 ; RV64-NEXT:    vsetivli zero, 16, e32, m8, ta, ma
 ; RV64-NEXT:    vslidedown.vi v24, v8, 16
-; RV64-NEXT:    csrr a0, vlenb
-; RV64-NEXT:    li a1, 24
-; RV64-NEXT:    mul a0, a0, a1
-; RV64-NEXT:    add a0, sp, a0
-; RV64-NEXT:    addi a0, a0, 16
-; RV64-NEXT:    vs8r.v v24, (a0) # Unknown-size Folded Spill
-; RV64-NEXT:    vslidedown.vi v24, v16, 16
-; RV64-NEXT:    csrr a0, vlenb
-; RV64-NEXT:    slli a0, a0, 3
-; RV64-NEXT:    add a0, sp, a0
-; RV64-NEXT:    addi a0, a0, 16
-; RV64-NEXT:    vs8r.v v24, (a0) # Unknown-size Folded Spill
+; RV64-NEXT:    addi a0, sp, 16
+; RV64-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
+; RV64-NEXT:    vslidedown.vi v0, v16, 16
 ; RV64-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
-; RV64-NEXT:    csrr a0, vlenb
-; RV64-NEXT:    li a1, 24
-; RV64-NEXT:    mul a0, a0, a1
-; RV64-NEXT:    add a0, sp, a0
-; RV64-NEXT:    addi a0, a0, 16
-; RV64-NEXT:    vl8r.v v24, (a0) # Unknown-size Folded Reload
-; RV64-NEXT:    csrr a0, vlenb
-; RV64-NEXT:    slli a0, a0, 3
-; RV64-NEXT:    add a0, sp, a0
-; RV64-NEXT:    addi a0, a0, 16
-; RV64-NEXT:    vl8r.v v8, (a0) # Unknown-size Folded Reload
+; RV64-NEXT:    vmv4r.v v8, v0
 ; RV64-NEXT:    vwaddu.vv v0, v24, v8
 ; RV64-NEXT:    csrr a0, vlenb
-; RV64-NEXT:    slli a0, a0, 4
+; RV64-NEXT:    slli a0, a0, 3
 ; RV64-NEXT:    add a0, sp, a0
 ; RV64-NEXT:    addi a0, a0, 16
 ; RV64-NEXT:    vs8r.v v0, (a0) # Unknown-size Folded Spill
@@ -1813,7 +1729,7 @@ define i64 @vwreduce_uadd_v64i64(ptr %x) {
 ; RV64-NEXT:    vwaddu.vv v0, v8, v16
 ; RV64-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; RV64-NEXT:    csrr a0, vlenb
-; RV64-NEXT:    slli a0, a0, 4
+; RV64-NEXT:    slli a0, a0, 3
 ; RV64-NEXT:    add a0, sp, a0
 ; RV64-NEXT:    addi a0, a0, 16
 ; RV64-NEXT:    vl8r.v v8, (a0) # Unknown-size Folded Reload
@@ -1822,7 +1738,7 @@ define i64 @vwreduce_uadd_v64i64(ptr %x) {
 ; RV64-NEXT:    vredsum.vs v8, v8, v16
 ; RV64-NEXT:    vmv.x.s a0, v8
 ; RV64-NEXT:    csrr a1, vlenb
-; RV64-NEXT:    slli a1, a1, 5
+; RV64-NEXT:    slli a1, a1, 4
 ; RV64-NEXT:    add sp, sp, a1
 ; RV64-NEXT:    addi sp, sp, 16
 ; RV64-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfadd-constrained-sdnode.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfadd-constrained-sdnode.ll
new file mode 100644
index 0000000000000..f7b51e8a7aea2
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfadd-constrained-sdnode.ll
@@ -0,0 +1,295 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=ilp32d -riscv-v-vector-bits-min=128\
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=riscv64 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=lp64d -riscv-v-vector-bits-min=128\
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+
+declare <2 x half> @llvm.experimental.constrained.fadd.v2f16(<2 x half>, <2 x half>, metadata, metadata)
+define <2 x half> @vfadd_vv_v2f16(<2 x half> %va, <2 x half> %vb) {
+; CHECK-LABEL: vfadd_vv_v2f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 2, e16, mf4, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <2 x half> @llvm.experimental.constrained.fadd.v2f16(<2 x half> %va, <2 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x half> %vc
+}
+
+define <2 x half> @vfadd_vf_v2f16(<2 x half> %va, half %b) {
+; CHECK-LABEL: vfadd_vf_v2f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 2, e16, mf4, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <2 x half> poison, half %b, i32 0
+  %splat = shufflevector <2 x half> %head, <2 x half> poison, <2 x i32> zeroinitializer
+  %vc = call <2 x half> @llvm.experimental.constrained.fadd.v2f16(<2 x half> %va, <2 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x half> %vc
+}
+
+declare <4 x half> @llvm.experimental.constrained.fadd.v4f16(<4 x half>, <4 x half>, metadata, metadata)
+define <4 x half> @vfadd_vv_v4f16(<4 x half> %va, <4 x half> %vb) {
+; CHECK-LABEL: vfadd_vv_v4f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 4, e16, mf2, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <4 x half> @llvm.experimental.constrained.fadd.v4f16(<4 x half> %va, <4 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x half> %vc
+}
+
+define <4 x half> @vfadd_vf_v4f16(<4 x half> %va, half %b) {
+; CHECK-LABEL: vfadd_vf_v4f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 4, e16, mf2, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <4 x half> poison, half %b, i32 0
+  %splat = shufflevector <4 x half> %head, <4 x half> poison, <4 x i32> zeroinitializer
+  %vc = call <4 x half> @llvm.experimental.constrained.fadd.v4f16(<4 x half> %va, <4 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x half> %vc
+}
+
+declare <8 x half> @llvm.experimental.constrained.fadd.v8f16(<8 x half>, <8 x half>, metadata, metadata)
+define <8 x half> @vfadd_vv_v8f16(<8 x half> %va, <8 x half> %vb) {
+; CHECK-LABEL: vfadd_vv_v8f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 8, e16, m1, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <8 x half> @llvm.experimental.constrained.fadd.v8f16(<8 x half> %va, <8 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x half> %vc
+}
+
+define <8 x half> @vfadd_vf_v8f16(<8 x half> %va, half %b) {
+; CHECK-LABEL: vfadd_vf_v8f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e16, m1, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <8 x half> poison, half %b, i32 0
+  %splat = shufflevector <8 x half> %head, <8 x half> poison, <8 x i32> zeroinitializer
+  %vc = call <8 x half> @llvm.experimental.constrained.fadd.v8f16(<8 x half> %va, <8 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x half> %vc
+}
+
+declare <16 x half> @llvm.experimental.constrained.fadd.v16f16(<16 x half>, <16 x half>, metadata, metadata)
+define <16 x half> @vfadd_vv_v16f16(<16 x half> %va, <16 x half> %vb) {
+; CHECK-LABEL: vfadd_vv_v16f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 16, e16, m2, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <16 x half> @llvm.experimental.constrained.fadd.v16f16(<16 x half> %va, <16 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <16 x half> %vc
+}
+
+define <16 x half> @vfadd_vf_v16f16(<16 x half> %va, half %b) {
+; CHECK-LABEL: vfadd_vf_v16f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 16, e16, m2, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <16 x half> poison, half %b, i32 0
+  %splat = shufflevector <16 x half> %head, <16 x half> poison, <16 x i32> zeroinitializer
+  %vc = call <16 x half> @llvm.experimental.constrained.fadd.v16f16(<16 x half> %va, <16 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <16 x half> %vc
+}
+
+declare <32 x half> @llvm.experimental.constrained.fadd.v32f16(<32 x half>, <32 x half>, metadata, metadata)
+define <32 x half> @vfadd_vv_v32f16(<32 x half> %va, <32 x half> %vb) {
+; CHECK-LABEL: vfadd_vv_v32f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    li a0, 32
+; CHECK-NEXT:    vsetvli zero, a0, e16, m4, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <32 x half> @llvm.experimental.constrained.fadd.v32f16(<32 x half> %va, <32 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <32 x half> %vc
+}
+
+define <32 x half> @vfadd_vf_v32f16(<32 x half> %va, half %b) {
+; CHECK-LABEL: vfadd_vf_v32f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    li a0, 32
+; CHECK-NEXT:    vsetvli zero, a0, e16, m4, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <32 x half> poison, half %b, i32 0
+  %splat = shufflevector <32 x half> %head, <32 x half> poison, <32 x i32> zeroinitializer
+  %vc = call <32 x half> @llvm.experimental.constrained.fadd.v32f16(<32 x half> %va, <32 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <32 x half> %vc
+}
+
+declare <2 x float> @llvm.experimental.constrained.fadd.v2f32(<2 x float>, <2 x float>, metadata, metadata)
+define <2 x float> @vfadd_vv_v2f32(<2 x float> %va, <2 x float> %vb) {
+; CHECK-LABEL: vfadd_vv_v2f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 2, e32, mf2, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <2 x float> @llvm.experimental.constrained.fadd.v2f32(<2 x float> %va, <2 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x float> %vc
+}
+
+define <2 x float> @vfadd_vf_v2f32(<2 x float> %va, float %b) {
+; CHECK-LABEL: vfadd_vf_v2f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 2, e32, mf2, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <2 x float> poison, float %b, i32 0
+  %splat = shufflevector <2 x float> %head, <2 x float> poison, <2 x i32> zeroinitializer
+  %vc = call <2 x float> @llvm.experimental.constrained.fadd.v2f32(<2 x float> %va, <2 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x float> %vc
+}
+
+declare <4 x float> @llvm.experimental.constrained.fadd.v4f32(<4 x float>, <4 x float>, metadata, metadata)
+define <4 x float> @vfadd_vv_v4f32(<4 x float> %va, <4 x float> %vb) {
+; CHECK-LABEL: vfadd_vv_v4f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <4 x float> @llvm.experimental.constrained.fadd.v4f32(<4 x float> %va, <4 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x float> %vc
+}
+
+define <4 x float> @vfadd_vf_v4f32(<4 x float> %va, float %b) {
+; CHECK-LABEL: vfadd_vf_v4f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <4 x float> poison, float %b, i32 0
+  %splat = shufflevector <4 x float> %head, <4 x float> poison, <4 x i32> zeroinitializer
+  %vc = call <4 x float> @llvm.experimental.constrained.fadd.v4f32(<4 x float> %va, <4 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x float> %vc
+}
+
+declare <8 x float> @llvm.experimental.constrained.fadd.v8f32(<8 x float>, <8 x float>, metadata, metadata)
+define <8 x float> @vfadd_vv_v8f32(<8 x float> %va, <8 x float> %vb) {
+; CHECK-LABEL: vfadd_vv_v8f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 8, e32, m2, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <8 x float> @llvm.experimental.constrained.fadd.v8f32(<8 x float> %va, <8 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x float> %vc
+}
+
+define <8 x float> @vfadd_vf_v8f32(<8 x float> %va, float %b) {
+; CHECK-LABEL: vfadd_vf_v8f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e32, m2, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <8 x float> poison, float %b, i32 0
+  %splat = shufflevector <8 x float> %head, <8 x float> poison, <8 x i32> zeroinitializer
+  %vc = call <8 x float> @llvm.experimental.constrained.fadd.v8f32(<8 x float> %va, <8 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x float> %vc
+}
+
+declare <16 x float> @llvm.experimental.constrained.fadd.v16f32(<16 x float>, <16 x float>, metadata, metadata)
+define <16 x float> @vfadd_vv_v16f32(<16 x float> %va, <16 x float> %vb) {
+; CHECK-LABEL: vfadd_vv_v16f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <16 x float> @llvm.experimental.constrained.fadd.v16f32(<16 x float> %va, <16 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <16 x float> %vc
+}
+
+define <16 x float> @vfadd_vf_v16f32(<16 x float> %va, float %b) {
+; CHECK-LABEL: vfadd_vf_v16f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <16 x float> poison, float %b, i32 0
+  %splat = shufflevector <16 x float> %head, <16 x float> poison, <16 x i32> zeroinitializer
+  %vc = call <16 x float> @llvm.experimental.constrained.fadd.v16f32(<16 x float> %va, <16 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <16 x float> %vc
+}
+
+declare <2 x double> @llvm.experimental.constrained.fadd.v2f64(<2 x double>, <2 x double>, metadata, metadata)
+define <2 x double> @vfadd_vv_v2f64(<2 x double> %va, <2 x double> %vb) {
+; CHECK-LABEL: vfadd_vv_v2f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <2 x double> @llvm.experimental.constrained.fadd.v2f64(<2 x double> %va, <2 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x double> %vc
+}
+
+define <2 x double> @vfadd_vf_v2f64(<2 x double> %va, double %b) {
+; CHECK-LABEL: vfadd_vf_v2f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <2 x double> poison, double %b, i32 0
+  %splat = shufflevector <2 x double> %head, <2 x double> poison, <2 x i32> zeroinitializer
+  %vc = call <2 x double> @llvm.experimental.constrained.fadd.v2f64(<2 x double> %va, <2 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x double> %vc
+}
+
+declare <4 x double> @llvm.experimental.constrained.fadd.v4f64(<4 x double>, <4 x double>, metadata, metadata)
+define <4 x double> @vfadd_vv_v4f64(<4 x double> %va, <4 x double> %vb) {
+; CHECK-LABEL: vfadd_vv_v4f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <4 x double> @llvm.experimental.constrained.fadd.v4f64(<4 x double> %va, <4 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x double> %vc
+}
+
+define <4 x double> @vfadd_vf_v4f64(<4 x double> %va, double %b) {
+; CHECK-LABEL: vfadd_vf_v4f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <4 x double> poison, double %b, i32 0
+  %splat = shufflevector <4 x double> %head, <4 x double> poison, <4 x i32> zeroinitializer
+  %vc = call <4 x double> @llvm.experimental.constrained.fadd.v4f64(<4 x double> %va, <4 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x double> %vc
+}
+
+declare <8 x double> @llvm.experimental.constrained.fadd.v8f64(<8 x double>, <8 x double>, metadata, metadata)
+define <8 x double> @vfadd_vv_v8f64(<8 x double> %va, <8 x double> %vb) {
+; CHECK-LABEL: vfadd_vv_v8f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <8 x double> @llvm.experimental.constrained.fadd.v8f64(<8 x double> %va, <8 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x double> %vc
+}
+
+define <8 x double> @vfadd_vf_v8f64(<8 x double> %va, double %b) {
+; CHECK-LABEL: vfadd_vf_v8f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <8 x double> poison, double %b, i32 0
+  %splat = shufflevector <8 x double> %head, <8 x double> poison, <8 x i32> zeroinitializer
+  %vc = call <8 x double> @llvm.experimental.constrained.fadd.v8f64(<8 x double> %va, <8 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x double> %vc
+}
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfdiv-constrained-sdnode.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfdiv-constrained-sdnode.ll
new file mode 100644
index 0000000000000..900386b621ec5
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfdiv-constrained-sdnode.ll
@@ -0,0 +1,331 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=ilp32d \
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=riscv64 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=lp64d \
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+
+declare <2 x half> @llvm.experimental.constrained.fdiv.v2f16(<2 x half>, <2 x half>, metadata, metadata)
+define <2 x half> @vfdiv_vv_v2f16(<2 x half> %va, <2 x half> %vb) {
+; CHECK-LABEL: vfdiv_vv_v2f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 2, e16, mf4, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <2 x half> @llvm.experimental.constrained.fdiv.v2f16(<2 x half> %va, <2 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x half> %vc
+}
+
+define <2 x half> @vfdiv_vf_v2f16(<2 x half> %va, half %b) {
+; CHECK-LABEL: vfdiv_vf_v2f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 2, e16, mf4, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <2 x half> poison, half %b, i32 0
+  %splat = shufflevector <2 x half> %head, <2 x half> poison, <2 x i32> zeroinitializer
+  %vc = call <2 x half> @llvm.experimental.constrained.fdiv.v2f16(<2 x half> %va, <2 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x half> %vc
+}
+
+declare <4 x half> @llvm.experimental.constrained.fdiv.v4f16(<4 x half>, <4 x half>, metadata, metadata)
+define <4 x half> @vfdiv_vv_v4f16(<4 x half> %va, <4 x half> %vb) {
+; CHECK-LABEL: vfdiv_vv_v4f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 4, e16, mf2, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <4 x half> @llvm.experimental.constrained.fdiv.v4f16(<4 x half> %va, <4 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x half> %vc
+}
+
+define <4 x half> @vfdiv_vf_v4f16(<4 x half> %va, half %b) {
+; CHECK-LABEL: vfdiv_vf_v4f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 4, e16, mf2, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <4 x half> poison, half %b, i32 0
+  %splat = shufflevector <4 x half> %head, <4 x half> poison, <4 x i32> zeroinitializer
+  %vc = call <4 x half> @llvm.experimental.constrained.fdiv.v4f16(<4 x half> %va, <4 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x half> %vc
+}
+
+declare <8 x half> @llvm.experimental.constrained.fdiv.v8f16(<8 x half>, <8 x half>, metadata, metadata)
+define <8 x half> @vfdiv_vv_v8f16(<8 x half> %va, <8 x half> %vb) {
+; CHECK-LABEL: vfdiv_vv_v8f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 8, e16, m1, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <8 x half> @llvm.experimental.constrained.fdiv.v8f16(<8 x half> %va, <8 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x half> %vc
+}
+
+define <8 x half> @vfdiv_vf_v8f16(<8 x half> %va, half %b) {
+; CHECK-LABEL: vfdiv_vf_v8f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e16, m1, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <8 x half> poison, half %b, i32 0
+  %splat = shufflevector <8 x half> %head, <8 x half> poison, <8 x i32> zeroinitializer
+  %vc = call <8 x half> @llvm.experimental.constrained.fdiv.v8f16(<8 x half> %va, <8 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x half> %vc
+}
+
+define <8 x half> @vfdiv_fv_v8f16(<8 x half> %va, half %b) {
+; CHECK-LABEL: vfdiv_fv_v8f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e16, m1, ta, ma
+; CHECK-NEXT:    vfrdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <8 x half> poison, half %b, i32 0
+  %splat = shufflevector <8 x half> %head, <8 x half> poison, <8 x i32> zeroinitializer
+  %vc = call <8 x half> @llvm.experimental.constrained.fdiv.v8f16(<8 x half> %splat, <8 x half> %va, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x half> %vc
+}
+
+declare <16 x half> @llvm.experimental.constrained.fdiv.v16f16(<16 x half>, <16 x half>, metadata, metadata)
+define <16 x half> @vfdiv_vv_v16f16(<16 x half> %va, <16 x half> %vb) {
+; CHECK-LABEL: vfdiv_vv_v16f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 16, e16, m2, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <16 x half> @llvm.experimental.constrained.fdiv.v16f16(<16 x half> %va, <16 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <16 x half> %vc
+}
+
+define <16 x half> @vfdiv_vf_v16f16(<16 x half> %va, half %b) {
+; CHECK-LABEL: vfdiv_vf_v16f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 16, e16, m2, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <16 x half> poison, half %b, i32 0
+  %splat = shufflevector <16 x half> %head, <16 x half> poison, <16 x i32> zeroinitializer
+  %vc = call <16 x half> @llvm.experimental.constrained.fdiv.v16f16(<16 x half> %va, <16 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <16 x half> %vc
+}
+
+declare <32 x half> @llvm.experimental.constrained.fdiv.v32f16(<32 x half>, <32 x half>, metadata, metadata)
+define <32 x half> @vfdiv_vv_v32f16(<32 x half> %va, <32 x half> %vb) {
+; CHECK-LABEL: vfdiv_vv_v32f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    li a0, 32
+; CHECK-NEXT:    vsetvli zero, a0, e16, m4, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <32 x half> @llvm.experimental.constrained.fdiv.v32f16(<32 x half> %va, <32 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <32 x half> %vc
+}
+
+define <32 x half> @vfdiv_vf_v32f16(<32 x half> %va, half %b) {
+; CHECK-LABEL: vfdiv_vf_v32f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    li a0, 32
+; CHECK-NEXT:    vsetvli zero, a0, e16, m4, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <32 x half> poison, half %b, i32 0
+  %splat = shufflevector <32 x half> %head, <32 x half> poison, <32 x i32> zeroinitializer
+  %vc = call <32 x half> @llvm.experimental.constrained.fdiv.v32f16(<32 x half> %va, <32 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <32 x half> %vc
+}
+
+declare <2 x float> @llvm.experimental.constrained.fdiv.v2f32(<2 x float>, <2 x float>, metadata, metadata)
+define <2 x float> @vfdiv_vv_v2f32(<2 x float> %va, <2 x float> %vb) {
+; CHECK-LABEL: vfdiv_vv_v2f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 2, e32, mf2, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <2 x float> @llvm.experimental.constrained.fdiv.v2f32(<2 x float> %va, <2 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x float> %vc
+}
+
+define <2 x float> @vfdiv_vf_v2f32(<2 x float> %va, float %b) {
+; CHECK-LABEL: vfdiv_vf_v2f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 2, e32, mf2, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <2 x float> poison, float %b, i32 0
+  %splat = shufflevector <2 x float> %head, <2 x float> poison, <2 x i32> zeroinitializer
+  %vc = call <2 x float> @llvm.experimental.constrained.fdiv.v2f32(<2 x float> %va, <2 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x float> %vc
+}
+
+declare <4 x float> @llvm.experimental.constrained.fdiv.v4f32(<4 x float>, <4 x float>, metadata, metadata)
+define <4 x float> @vfdiv_vv_v4f32(<4 x float> %va, <4 x float> %vb) {
+; CHECK-LABEL: vfdiv_vv_v4f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <4 x float> @llvm.experimental.constrained.fdiv.v4f32(<4 x float> %va, <4 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x float> %vc
+}
+
+define <4 x float> @vfdiv_vf_v4f32(<4 x float> %va, float %b) {
+; CHECK-LABEL: vfdiv_vf_v4f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <4 x float> poison, float %b, i32 0
+  %splat = shufflevector <4 x float> %head, <4 x float> poison, <4 x i32> zeroinitializer
+  %vc = call <4 x float> @llvm.experimental.constrained.fdiv.v4f32(<4 x float> %va, <4 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x float> %vc
+}
+
+declare <8 x float> @llvm.experimental.constrained.fdiv.v8f32(<8 x float>, <8 x float>, metadata, metadata)
+define <8 x float> @vfdiv_vv_v8f32(<8 x float> %va, <8 x float> %vb) {
+; CHECK-LABEL: vfdiv_vv_v8f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 8, e32, m2, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <8 x float> @llvm.experimental.constrained.fdiv.v8f32(<8 x float> %va, <8 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x float> %vc
+}
+
+define <8 x float> @vfdiv_vf_v8f32(<8 x float> %va, float %b) {
+; CHECK-LABEL: vfdiv_vf_v8f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e32, m2, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <8 x float> poison, float %b, i32 0
+  %splat = shufflevector <8 x float> %head, <8 x float> poison, <8 x i32> zeroinitializer
+  %vc = call <8 x float> @llvm.experimental.constrained.fdiv.v8f32(<8 x float> %va, <8 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x float> %vc
+}
+
+define <8 x float> @vfdiv_fv_v8f32(<8 x float> %va, float %b) {
+; CHECK-LABEL: vfdiv_fv_v8f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e32, m2, ta, ma
+; CHECK-NEXT:    vfrdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <8 x float> poison, float %b, i32 0
+  %splat = shufflevector <8 x float> %head, <8 x float> poison, <8 x i32> zeroinitializer
+  %vc = call <8 x float> @llvm.experimental.constrained.fdiv.v8f32(<8 x float> %splat, <8 x float> %va, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x float> %vc
+}
+
+declare <16 x float> @llvm.experimental.constrained.fdiv.v16f32(<16 x float>, <16 x float>, metadata, metadata)
+define <16 x float> @vfdiv_vv_v16f32(<16 x float> %va, <16 x float> %vb) {
+; CHECK-LABEL: vfdiv_vv_v16f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <16 x float> @llvm.experimental.constrained.fdiv.v16f32(<16 x float> %va, <16 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <16 x float> %vc
+}
+
+define <16 x float> @vfdiv_vf_v16f32(<16 x float> %va, float %b) {
+; CHECK-LABEL: vfdiv_vf_v16f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <16 x float> poison, float %b, i32 0
+  %splat = shufflevector <16 x float> %head, <16 x float> poison, <16 x i32> zeroinitializer
+  %vc = call <16 x float> @llvm.experimental.constrained.fdiv.v16f32(<16 x float> %va, <16 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <16 x float> %vc
+}
+
+declare <2 x double> @llvm.experimental.constrained.fdiv.v2f64(<2 x double>, <2 x double>, metadata, metadata)
+define <2 x double> @vfdiv_vv_v2f64(<2 x double> %va, <2 x double> %vb) {
+; CHECK-LABEL: vfdiv_vv_v2f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <2 x double> @llvm.experimental.constrained.fdiv.v2f64(<2 x double> %va, <2 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x double> %vc
+}
+
+define <2 x double> @vfdiv_vf_v2f64(<2 x double> %va, double %b) {
+; CHECK-LABEL: vfdiv_vf_v2f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <2 x double> poison, double %b, i32 0
+  %splat = shufflevector <2 x double> %head, <2 x double> poison, <2 x i32> zeroinitializer
+  %vc = call <2 x double> @llvm.experimental.constrained.fdiv.v2f64(<2 x double> %va, <2 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x double> %vc
+}
+
+declare <4 x double> @llvm.experimental.constrained.fdiv.v4f64(<4 x double>, <4 x double>, metadata, metadata)
+define <4 x double> @vfdiv_vv_v4f64(<4 x double> %va, <4 x double> %vb) {
+; CHECK-LABEL: vfdiv_vv_v4f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <4 x double> @llvm.experimental.constrained.fdiv.v4f64(<4 x double> %va, <4 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x double> %vc
+}
+
+define <4 x double> @vfdiv_vf_v4f64(<4 x double> %va, double %b) {
+; CHECK-LABEL: vfdiv_vf_v4f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <4 x double> poison, double %b, i32 0
+  %splat = shufflevector <4 x double> %head, <4 x double> poison, <4 x i32> zeroinitializer
+  %vc = call <4 x double> @llvm.experimental.constrained.fdiv.v4f64(<4 x double> %va, <4 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x double> %vc
+}
+
+declare <8 x double> @llvm.experimental.constrained.fdiv.v8f64(<8 x double>, <8 x double>, metadata, metadata)
+define <8 x double> @vfdiv_vv_v8f64(<8 x double> %va, <8 x double> %vb) {
+; CHECK-LABEL: vfdiv_vv_v8f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <8 x double> @llvm.experimental.constrained.fdiv.v8f64(<8 x double> %va, <8 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x double> %vc
+}
+
+define <8 x double> @vfdiv_vf_v8f64(<8 x double> %va, double %b) {
+; CHECK-LABEL: vfdiv_vf_v8f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <8 x double> poison, double %b, i32 0
+  %splat = shufflevector <8 x double> %head, <8 x double> poison, <8 x i32> zeroinitializer
+  %vc = call <8 x double> @llvm.experimental.constrained.fdiv.v8f64(<8 x double> %va, <8 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x double> %vc
+}
+
+define <8 x double> @vfdiv_fv_v8f64(<8 x double> %va, double %b) {
+; CHECK-LABEL: vfdiv_fv_v8f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
+; CHECK-NEXT:    vfrdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <8 x double> poison, double %b, i32 0
+  %splat = shufflevector <8 x double> %head, <8 x double> poison, <8 x i32> zeroinitializer
+  %vc = call <8 x double> @llvm.experimental.constrained.fdiv.v8f64(<8 x double> %splat, <8 x double> %va, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x double> %vc
+}
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfmul-constrained-sdnode.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfmul-constrained-sdnode.ll
new file mode 100644
index 0000000000000..13ae8831391ae
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfmul-constrained-sdnode.ll
@@ -0,0 +1,367 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=ilp32d \
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=riscv64 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=lp64d \
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+
+declare <1 x half> @llvm.experimental.constrained.fmul.v1f16(<1 x half>, <1 x half>, metadata, metadata)
+define <1 x half> @vfmul_vv_v1f16(<1 x half> %va, <1 x half> %vb) {
+; CHECK-LABEL: vfmul_vv_v1f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 1, e16, mf4, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <1 x half> @llvm.experimental.constrained.fmul.v1f16(<1 x half> %va, <1 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <1 x half> %vc
+}
+
+define <1 x half> @vfmul_vf_v1f16(<1 x half> %va, half %b) {
+; CHECK-LABEL: vfmul_vf_v1f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 1, e16, mf4, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <1 x half> poison, half %b, i32 0
+  %splat = shufflevector <1 x half> %head, <1 x half> poison, <1 x i32> zeroinitializer
+  %vc = call <1 x half> @llvm.experimental.constrained.fmul.v1f16(<1 x half> %va, <1 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <1 x half> %vc
+}
+
+declare <2 x half> @llvm.experimental.constrained.fmul.v2f16(<2 x half>, <2 x half>, metadata, metadata)
+define <2 x half> @vfmul_vv_v2f16(<2 x half> %va, <2 x half> %vb) {
+; CHECK-LABEL: vfmul_vv_v2f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 2, e16, mf4, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <2 x half> @llvm.experimental.constrained.fmul.v2f16(<2 x half> %va, <2 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x half> %vc
+}
+
+define <2 x half> @vfmul_vf_v2f16(<2 x half> %va, half %b) {
+; CHECK-LABEL: vfmul_vf_v2f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 2, e16, mf4, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <2 x half> poison, half %b, i32 0
+  %splat = shufflevector <2 x half> %head, <2 x half> poison, <2 x i32> zeroinitializer
+  %vc = call <2 x half> @llvm.experimental.constrained.fmul.v2f16(<2 x half> %va, <2 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x half> %vc
+}
+
+declare <4 x half> @llvm.experimental.constrained.fmul.v4f16(<4 x half>, <4 x half>, metadata, metadata)
+define <4 x half> @vfmul_vv_v4f16(<4 x half> %va, <4 x half> %vb) {
+; CHECK-LABEL: vfmul_vv_v4f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 4, e16, mf2, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <4 x half> @llvm.experimental.constrained.fmul.v4f16(<4 x half> %va, <4 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x half> %vc
+}
+
+define <4 x half> @vfmul_vf_v4f16(<4 x half> %va, half %b) {
+; CHECK-LABEL: vfmul_vf_v4f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 4, e16, mf2, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <4 x half> poison, half %b, i32 0
+  %splat = shufflevector <4 x half> %head, <4 x half> poison, <4 x i32> zeroinitializer
+  %vc = call <4 x half> @llvm.experimental.constrained.fmul.v4f16(<4 x half> %va, <4 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x half> %vc
+}
+
+declare <8 x half> @llvm.experimental.constrained.fmul.v8f16(<8 x half>, <8 x half>, metadata, metadata)
+define <8 x half> @vfmul_vv_v8f16(<8 x half> %va, <8 x half> %vb) {
+; CHECK-LABEL: vfmul_vv_v8f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 8, e16, m1, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <8 x half> @llvm.experimental.constrained.fmul.v8f16(<8 x half> %va, <8 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x half> %vc
+}
+
+define <8 x half> @vfmul_vf_v8f16(<8 x half> %va, half %b) {
+; CHECK-LABEL: vfmul_vf_v8f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e16, m1, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <8 x half> poison, half %b, i32 0
+  %splat = shufflevector <8 x half> %head, <8 x half> poison, <8 x i32> zeroinitializer
+  %vc = call <8 x half> @llvm.experimental.constrained.fmul.v8f16(<8 x half> %va, <8 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x half> %vc
+}
+
+declare <16 x half> @llvm.experimental.constrained.fmul.v16f16(<16 x half>, <16 x half>, metadata, metadata)
+define <16 x half> @vfmul_vv_v16f16(<16 x half> %va, <16 x half> %vb) {
+; CHECK-LABEL: vfmul_vv_v16f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 16, e16, m2, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <16 x half> @llvm.experimental.constrained.fmul.v16f16(<16 x half> %va, <16 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <16 x half> %vc
+}
+
+define <16 x half> @vfmul_vf_v16f16(<16 x half> %va, half %b) {
+; CHECK-LABEL: vfmul_vf_v16f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 16, e16, m2, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <16 x half> poison, half %b, i32 0
+  %splat = shufflevector <16 x half> %head, <16 x half> poison, <16 x i32> zeroinitializer
+  %vc = call <16 x half> @llvm.experimental.constrained.fmul.v16f16(<16 x half> %va, <16 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <16 x half> %vc
+}
+
+declare <32 x half> @llvm.experimental.constrained.fmul.v32f16(<32 x half>, <32 x half>, metadata, metadata)
+define <32 x half> @vfmul_vv_v32f16(<32 x half> %va, <32 x half> %vb) {
+; CHECK-LABEL: vfmul_vv_v32f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    li a0, 32
+; CHECK-NEXT:    vsetvli zero, a0, e16, m4, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <32 x half> @llvm.experimental.constrained.fmul.v32f16(<32 x half> %va, <32 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <32 x half> %vc
+}
+
+define <32 x half> @vfmul_vf_v32f16(<32 x half> %va, half %b) {
+; CHECK-LABEL: vfmul_vf_v32f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    li a0, 32
+; CHECK-NEXT:    vsetvli zero, a0, e16, m4, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <32 x half> poison, half %b, i32 0
+  %splat = shufflevector <32 x half> %head, <32 x half> poison, <32 x i32> zeroinitializer
+  %vc = call <32 x half> @llvm.experimental.constrained.fmul.v32f16(<32 x half> %va, <32 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <32 x half> %vc
+}
+
+declare <1 x float> @llvm.experimental.constrained.fmul.v1f32(<1 x float>, <1 x float>, metadata, metadata)
+define <1 x float> @vfmul_vv_v1f32(<1 x float> %va, <1 x float> %vb) {
+; CHECK-LABEL: vfmul_vv_v1f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 1, e32, mf2, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <1 x float> @llvm.experimental.constrained.fmul.v1f32(<1 x float> %va, <1 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <1 x float> %vc
+}
+
+define <1 x float> @vfmul_vf_v1f32(<1 x float> %va, float %b) {
+; CHECK-LABEL: vfmul_vf_v1f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 1, e32, mf2, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <1 x float> poison, float %b, i32 0
+  %splat = shufflevector <1 x float> %head, <1 x float> poison, <1 x i32> zeroinitializer
+  %vc = call <1 x float> @llvm.experimental.constrained.fmul.v1f32(<1 x float> %va, <1 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <1 x float> %vc
+}
+
+declare <2 x float> @llvm.experimental.constrained.fmul.v2f32(<2 x float>, <2 x float>, metadata, metadata)
+define <2 x float> @vfmul_vv_v2f32(<2 x float> %va, <2 x float> %vb) {
+; CHECK-LABEL: vfmul_vv_v2f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 2, e32, mf2, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <2 x float> @llvm.experimental.constrained.fmul.v2f32(<2 x float> %va, <2 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x float> %vc
+}
+
+define <2 x float> @vfmul_vf_v2f32(<2 x float> %va, float %b) {
+; CHECK-LABEL: vfmul_vf_v2f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 2, e32, mf2, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <2 x float> poison, float %b, i32 0
+  %splat = shufflevector <2 x float> %head, <2 x float> poison, <2 x i32> zeroinitializer
+  %vc = call <2 x float> @llvm.experimental.constrained.fmul.v2f32(<2 x float> %va, <2 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x float> %vc
+}
+
+declare <4 x float> @llvm.experimental.constrained.fmul.v4f32(<4 x float>, <4 x float>, metadata, metadata)
+define <4 x float> @vfmul_vv_v4f32(<4 x float> %va, <4 x float> %vb) {
+; CHECK-LABEL: vfmul_vv_v4f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <4 x float> @llvm.experimental.constrained.fmul.v4f32(<4 x float> %va, <4 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x float> %vc
+}
+
+define <4 x float> @vfmul_vf_v4f32(<4 x float> %va, float %b) {
+; CHECK-LABEL: vfmul_vf_v4f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <4 x float> poison, float %b, i32 0
+  %splat = shufflevector <4 x float> %head, <4 x float> poison, <4 x i32> zeroinitializer
+  %vc = call <4 x float> @llvm.experimental.constrained.fmul.v4f32(<4 x float> %va, <4 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x float> %vc
+}
+
+declare <8 x float> @llvm.experimental.constrained.fmul.v8f32(<8 x float>, <8 x float>, metadata, metadata)
+define <8 x float> @vfmul_vv_v8f32(<8 x float> %va, <8 x float> %vb) {
+; CHECK-LABEL: vfmul_vv_v8f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 8, e32, m2, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <8 x float> @llvm.experimental.constrained.fmul.v8f32(<8 x float> %va, <8 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x float> %vc
+}
+
+define <8 x float> @vfmul_vf_v8f32(<8 x float> %va, float %b) {
+; CHECK-LABEL: vfmul_vf_v8f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e32, m2, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <8 x float> poison, float %b, i32 0
+  %splat = shufflevector <8 x float> %head, <8 x float> poison, <8 x i32> zeroinitializer
+  %vc = call <8 x float> @llvm.experimental.constrained.fmul.v8f32(<8 x float> %va, <8 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x float> %vc
+}
+
+declare <16 x float> @llvm.experimental.constrained.fmul.v16f32(<16 x float>, <16 x float>, metadata, metadata)
+define <16 x float> @vfmul_vv_v16f32(<16 x float> %va, <16 x float> %vb) {
+; CHECK-LABEL: vfmul_vv_v16f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <16 x float> @llvm.experimental.constrained.fmul.v16f32(<16 x float> %va, <16 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <16 x float> %vc
+}
+
+define <16 x float> @vfmul_vf_v16f32(<16 x float> %va, float %b) {
+; CHECK-LABEL: vfmul_vf_v16f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <16 x float> poison, float %b, i32 0
+  %splat = shufflevector <16 x float> %head, <16 x float> poison, <16 x i32> zeroinitializer
+  %vc = call <16 x float> @llvm.experimental.constrained.fmul.v16f32(<16 x float> %va, <16 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <16 x float> %vc
+}
+
+declare <1 x double> @llvm.experimental.constrained.fmul.v1f64(<1 x double>, <1 x double>, metadata, metadata)
+define <1 x double> @vfmul_vv_v1f64(<1 x double> %va, <1 x double> %vb) {
+; CHECK-LABEL: vfmul_vv_v1f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <1 x double> @llvm.experimental.constrained.fmul.v1f64(<1 x double> %va, <1 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <1 x double> %vc
+}
+
+define <1 x double> @vfmul_vf_v1f64(<1 x double> %va, double %b) {
+; CHECK-LABEL: vfmul_vf_v1f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <1 x double> poison, double %b, i32 0
+  %splat = shufflevector <1 x double> %head, <1 x double> poison, <1 x i32> zeroinitializer
+  %vc = call <1 x double> @llvm.experimental.constrained.fmul.v1f64(<1 x double> %va, <1 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <1 x double> %vc
+}
+
+declare <2 x double> @llvm.experimental.constrained.fmul.v2f64(<2 x double>, <2 x double>, metadata, metadata)
+define <2 x double> @vfmul_vv_v2f64(<2 x double> %va, <2 x double> %vb) {
+; CHECK-LABEL: vfmul_vv_v2f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <2 x double> @llvm.experimental.constrained.fmul.v2f64(<2 x double> %va, <2 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x double> %vc
+}
+
+define <2 x double> @vfmul_vf_v2f64(<2 x double> %va, double %b) {
+; CHECK-LABEL: vfmul_vf_v2f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <2 x double> poison, double %b, i32 0
+  %splat = shufflevector <2 x double> %head, <2 x double> poison, <2 x i32> zeroinitializer
+  %vc = call <2 x double> @llvm.experimental.constrained.fmul.v2f64(<2 x double> %va, <2 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x double> %vc
+}
+
+declare <4 x double> @llvm.experimental.constrained.fmul.v4f64(<4 x double>, <4 x double>, metadata, metadata)
+define <4 x double> @vfmul_vv_v4f64(<4 x double> %va, <4 x double> %vb) {
+; CHECK-LABEL: vfmul_vv_v4f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <4 x double> @llvm.experimental.constrained.fmul.v4f64(<4 x double> %va, <4 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x double> %vc
+}
+
+define <4 x double> @vfmul_vf_v4f64(<4 x double> %va, double %b) {
+; CHECK-LABEL: vfmul_vf_v4f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <4 x double> poison, double %b, i32 0
+  %splat = shufflevector <4 x double> %head, <4 x double> poison, <4 x i32> zeroinitializer
+  %vc = call <4 x double> @llvm.experimental.constrained.fmul.v4f64(<4 x double> %va, <4 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x double> %vc
+}
+
+declare <8 x double> @llvm.experimental.constrained.fmul.v8f64(<8 x double>, <8 x double>, metadata, metadata)
+define <8 x double> @vfmul_vv_v8f64(<8 x double> %va, <8 x double> %vb) {
+; CHECK-LABEL: vfmul_vv_v8f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <8 x double> @llvm.experimental.constrained.fmul.v8f64(<8 x double> %va, <8 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x double> %vc
+}
+
+define <8 x double> @vfmul_vf_v8f64(<8 x double> %va, double %b) {
+; CHECK-LABEL: vfmul_vf_v8f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <8 x double> poison, double %b, i32 0
+  %splat = shufflevector <8 x double> %head, <8 x double> poison, <8 x i32> zeroinitializer
+  %vc = call <8 x double> @llvm.experimental.constrained.fmul.v8f64(<8 x double> %va, <8 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x double> %vc
+}
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfpext-constrained-sdnode.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfpext-constrained-sdnode.ll
new file mode 100644
index 0000000000000..270146b99f867
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfpext-constrained-sdnode.ll
@@ -0,0 +1,116 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=ilp32d -riscv-v-vector-bits-min=128\
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=riscv64 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=lp64d -riscv-v-vector-bits-min=128\
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+
+declare <2 x float> @llvm.experimental.constrained.fpext.v2f32.v2f16(<2 x half>, metadata)
+define <2 x float> @vfpext_v2f16_v2f32(<2 x half> %va) {
+; CHECK-LABEL: vfpext_v2f16_v2f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 2, e16, mf4, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v9, v8
+; CHECK-NEXT:    vmv1r.v v8, v9
+; CHECK-NEXT:    ret
+  %evec = call <2 x float> @llvm.experimental.constrained.fpext.v2f32.v2f16(<2 x half> %va, metadata !"fpexcept.strict")
+  ret <2 x float> %evec
+}
+
+declare <2 x double> @llvm.experimental.constrained.fpext.v2f64.v2f16(<2 x half>, metadata)
+define <2 x double> @vfpext_v2f16_v2f64(<2 x half> %va) {
+; CHECK-LABEL: vfpext_v2f16_v2f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 2, e16, mf4, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v9, v8
+; CHECK-NEXT:    vsetvli zero, zero, e32, mf2, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v8, v9
+; CHECK-NEXT:    ret
+  %evec = call <2 x double> @llvm.experimental.constrained.fpext.v2f64.v2f16(<2 x half> %va, metadata !"fpexcept.strict")
+  ret <2 x double> %evec
+}
+
+declare <4 x float> @llvm.experimental.constrained.fpext.v4f32.v4f16(<4 x half>, metadata)
+define <4 x float> @vfpext_v4f16_v4f32(<4 x half> %va) {
+; CHECK-LABEL: vfpext_v4f16_v4f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 4, e16, mf2, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v9, v8
+; CHECK-NEXT:    vmv1r.v v8, v9
+; CHECK-NEXT:    ret
+  %evec = call <4 x float> @llvm.experimental.constrained.fpext.v4f32.v4f16(<4 x half> %va, metadata !"fpexcept.strict")
+  ret <4 x float> %evec
+}
+
+declare <4 x double> @llvm.experimental.constrained.fpext.v4f64.v4f16(<4 x half>, metadata)
+define <4 x double> @vfpext_v4f16_v4f64(<4 x half> %va) {
+; CHECK-LABEL: vfpext_v4f16_v4f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 4, e16, mf2, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v10, v8
+; CHECK-NEXT:    vsetvli zero, zero, e32, m1, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v8, v10
+; CHECK-NEXT:    ret
+  %evec = call <4 x double> @llvm.experimental.constrained.fpext.v4f64.v4f16(<4 x half> %va, metadata !"fpexcept.strict")
+  ret <4 x double> %evec
+}
+
+declare <8 x float> @llvm.experimental.constrained.fpext.v8f32.v8f16(<8 x half>, metadata)
+define <8 x float> @vfpext_v8f16_v8f32(<8 x half> %va) {
+; CHECK-LABEL: vfpext_v8f16_v8f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e16, m1, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v10, v8
+; CHECK-NEXT:    vmv2r.v v8, v10
+; CHECK-NEXT:    ret
+  %evec = call <8 x float> @llvm.experimental.constrained.fpext.v8f32.v8f16(<8 x half> %va, metadata !"fpexcept.strict")
+  ret <8 x float> %evec
+}
+
+declare <8 x double> @llvm.experimental.constrained.fpext.v8f64.v8f16(<8 x half>, metadata)
+define <8 x double> @vfpext_v8f16_v8f64(<8 x half> %va) {
+; CHECK-LABEL: vfpext_v8f16_v8f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e16, m1, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v12, v8
+; CHECK-NEXT:    vsetvli zero, zero, e32, m2, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v8, v12
+; CHECK-NEXT:    ret
+  %evec = call <8 x double> @llvm.experimental.constrained.fpext.v8f64.v8f16(<8 x half> %va, metadata !"fpexcept.strict")
+  ret <8 x double> %evec
+}
+
+declare <2 x double> @llvm.experimental.constrained.fpext.v2f64.v2f32(<2 x float>, metadata)
+define <2 x double> @vfpext_v2f32_v2f64(<2 x float> %va) {
+; CHECK-LABEL: vfpext_v2f32_v2f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 2, e32, mf2, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v9, v8
+; CHECK-NEXT:    vmv1r.v v8, v9
+; CHECK-NEXT:    ret
+  %evec = call <2 x double> @llvm.experimental.constrained.fpext.v2f64.v2f32(<2 x float> %va, metadata !"fpexcept.strict")
+  ret <2 x double> %evec
+}
+
+declare <4 x double> @llvm.experimental.constrained.fpext.v4f64.v4f32(<4 x float>, metadata)
+define <4 x double> @vfpext_v4f32_v4f64(<4 x float> %va) {
+; CHECK-LABEL: vfpext_v4f32_v4f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v10, v8
+; CHECK-NEXT:    vmv2r.v v8, v10
+; CHECK-NEXT:    ret
+  %evec = call <4 x double> @llvm.experimental.constrained.fpext.v4f64.v4f32(<4 x float> %va, metadata !"fpexcept.strict")
+  ret <4 x double> %evec
+}
+
+declare <8 x double> @llvm.experimental.constrained.fpext.v8f64.v8f32(<8 x float>, metadata)
+define <8 x double> @vfpext_v8f32_v8f64(<8 x float> %va) {
+; CHECK-LABEL: vfpext_v8f32_v8f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e32, m2, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v12, v8
+; CHECK-NEXT:    vmv4r.v v8, v12
+; CHECK-NEXT:    ret
+  %evec = call <8 x double> @llvm.experimental.constrained.fpext.v8f64.v8f32(<8 x float> %va, metadata !"fpexcept.strict")
+  ret <8 x double> %evec
+}
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfsub-constrained-sdnode.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfsub-constrained-sdnode.ll
new file mode 100644
index 0000000000000..50e6e2dc51154
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfsub-constrained-sdnode.ll
@@ -0,0 +1,331 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=ilp32d \
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=riscv64 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=lp64d \
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+
+declare <2 x half> @llvm.experimental.constrained.fsub.v2f16(<2 x half>, <2 x half>, metadata, metadata)
+define <2 x half> @vfsub_vv_v2f16(<2 x half> %va, <2 x half> %vb) {
+; CHECK-LABEL: vfsub_vv_v2f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 2, e16, mf4, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <2 x half> @llvm.experimental.constrained.fsub.v2f16(<2 x half> %va, <2 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x half> %vc
+}
+
+define <2 x half> @vfsub_vf_v2f16(<2 x half> %va, half %b) {
+; CHECK-LABEL: vfsub_vf_v2f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 2, e16, mf4, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <2 x half> poison, half %b, i32 0
+  %splat = shufflevector <2 x half> %head, <2 x half> poison, <2 x i32> zeroinitializer
+  %vc = call <2 x half> @llvm.experimental.constrained.fsub.v2f16(<2 x half> %va, <2 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x half> %vc
+}
+
+declare <4 x half> @llvm.experimental.constrained.fsub.v4f16(<4 x half>, <4 x half>, metadata, metadata)
+define <4 x half> @vfsub_vv_v4f16(<4 x half> %va, <4 x half> %vb) {
+; CHECK-LABEL: vfsub_vv_v4f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 4, e16, mf2, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <4 x half> @llvm.experimental.constrained.fsub.v4f16(<4 x half> %va, <4 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x half> %vc
+}
+
+define <4 x half> @vfsub_vf_v4f16(<4 x half> %va, half %b) {
+; CHECK-LABEL: vfsub_vf_v4f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 4, e16, mf2, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <4 x half> poison, half %b, i32 0
+  %splat = shufflevector <4 x half> %head, <4 x half> poison, <4 x i32> zeroinitializer
+  %vc = call <4 x half> @llvm.experimental.constrained.fsub.v4f16(<4 x half> %va, <4 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x half> %vc
+}
+
+declare <8 x half> @llvm.experimental.constrained.fsub.v8f16(<8 x half>, <8 x half>, metadata, metadata)
+define <8 x half> @vfsub_vv_v8f16(<8 x half> %va, <8 x half> %vb) {
+; CHECK-LABEL: vfsub_vv_v8f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 8, e16, m1, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <8 x half> @llvm.experimental.constrained.fsub.v8f16(<8 x half> %va, <8 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x half> %vc
+}
+
+define <8 x half> @vfsub_vf_v8f16(<8 x half> %va, half %b) {
+; CHECK-LABEL: vfsub_vf_v8f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e16, m1, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <8 x half> poison, half %b, i32 0
+  %splat = shufflevector <8 x half> %head, <8 x half> poison, <8 x i32> zeroinitializer
+  %vc = call <8 x half> @llvm.experimental.constrained.fsub.v8f16(<8 x half> %va, <8 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x half> %vc
+}
+
+define <8 x half> @vfsub_fv_v8f16(<8 x half> %va, half %b) {
+; CHECK-LABEL: vfsub_fv_v8f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e16, m1, ta, ma
+; CHECK-NEXT:    vfrsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <8 x half> poison, half %b, i32 0
+  %splat = shufflevector <8 x half> %head, <8 x half> poison, <8 x i32> zeroinitializer
+  %vc = call <8 x half> @llvm.experimental.constrained.fsub.v8f16(<8 x half> %splat, <8 x half> %va, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x half> %vc
+}
+
+declare <16 x half> @llvm.experimental.constrained.fsub.v16f16(<16 x half>, <16 x half>, metadata, metadata)
+define <16 x half> @vfsub_vv_v16f16(<16 x half> %va, <16 x half> %vb) {
+; CHECK-LABEL: vfsub_vv_v16f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 16, e16, m2, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <16 x half> @llvm.experimental.constrained.fsub.v16f16(<16 x half> %va, <16 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <16 x half> %vc
+}
+
+define <16 x half> @vfsub_vf_v16f16(<16 x half> %va, half %b) {
+; CHECK-LABEL: vfsub_vf_v16f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 16, e16, m2, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <16 x half> poison, half %b, i32 0
+  %splat = shufflevector <16 x half> %head, <16 x half> poison, <16 x i32> zeroinitializer
+  %vc = call <16 x half> @llvm.experimental.constrained.fsub.v16f16(<16 x half> %va, <16 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <16 x half> %vc
+}
+
+declare <32 x half> @llvm.experimental.constrained.fsub.v32f16(<32 x half>, <32 x half>, metadata, metadata)
+define <32 x half> @vfsub_vv_v32f16(<32 x half> %va, <32 x half> %vb) {
+; CHECK-LABEL: vfsub_vv_v32f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    li a0, 32
+; CHECK-NEXT:    vsetvli zero, a0, e16, m4, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <32 x half> @llvm.experimental.constrained.fsub.v32f16(<32 x half> %va, <32 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <32 x half> %vc
+}
+
+define <32 x half> @vfsub_vf_v32f16(<32 x half> %va, half %b) {
+; CHECK-LABEL: vfsub_vf_v32f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    li a0, 32
+; CHECK-NEXT:    vsetvli zero, a0, e16, m4, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <32 x half> poison, half %b, i32 0
+  %splat = shufflevector <32 x half> %head, <32 x half> poison, <32 x i32> zeroinitializer
+  %vc = call <32 x half> @llvm.experimental.constrained.fsub.v32f16(<32 x half> %va, <32 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <32 x half> %vc
+}
+
+declare <2 x float> @llvm.experimental.constrained.fsub.v2f32(<2 x float>, <2 x float>, metadata, metadata)
+define <2 x float> @vfsub_vv_v2f32(<2 x float> %va, <2 x float> %vb) {
+; CHECK-LABEL: vfsub_vv_v2f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 2, e32, mf2, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <2 x float> @llvm.experimental.constrained.fsub.v2f32(<2 x float> %va, <2 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x float> %vc
+}
+
+define <2 x float> @vfsub_vf_v2f32(<2 x float> %va, float %b) {
+; CHECK-LABEL: vfsub_vf_v2f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 2, e32, mf2, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <2 x float> poison, float %b, i32 0
+  %splat = shufflevector <2 x float> %head, <2 x float> poison, <2 x i32> zeroinitializer
+  %vc = call <2 x float> @llvm.experimental.constrained.fsub.v2f32(<2 x float> %va, <2 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x float> %vc
+}
+
+declare <4 x float> @llvm.experimental.constrained.fsub.v4f32(<4 x float>, <4 x float>, metadata, metadata)
+define <4 x float> @vfsub_vv_v4f32(<4 x float> %va, <4 x float> %vb) {
+; CHECK-LABEL: vfsub_vv_v4f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <4 x float> @llvm.experimental.constrained.fsub.v4f32(<4 x float> %va, <4 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x float> %vc
+}
+
+define <4 x float> @vfsub_vf_v4f32(<4 x float> %va, float %b) {
+; CHECK-LABEL: vfsub_vf_v4f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <4 x float> poison, float %b, i32 0
+  %splat = shufflevector <4 x float> %head, <4 x float> poison, <4 x i32> zeroinitializer
+  %vc = call <4 x float> @llvm.experimental.constrained.fsub.v4f32(<4 x float> %va, <4 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x float> %vc
+}
+
+declare <8 x float> @llvm.experimental.constrained.fsub.v8f32(<8 x float>, <8 x float>, metadata, metadata)
+define <8 x float> @vfsub_vv_v8f32(<8 x float> %va, <8 x float> %vb) {
+; CHECK-LABEL: vfsub_vv_v8f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 8, e32, m2, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <8 x float> @llvm.experimental.constrained.fsub.v8f32(<8 x float> %va, <8 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x float> %vc
+}
+
+define <8 x float> @vfsub_vf_v8f32(<8 x float> %va, float %b) {
+; CHECK-LABEL: vfsub_vf_v8f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e32, m2, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <8 x float> poison, float %b, i32 0
+  %splat = shufflevector <8 x float> %head, <8 x float> poison, <8 x i32> zeroinitializer
+  %vc = call <8 x float> @llvm.experimental.constrained.fsub.v8f32(<8 x float> %va, <8 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x float> %vc
+}
+
+define <8 x float> @vfsub_fv_v8f32(<8 x float> %va, float %b) {
+; CHECK-LABEL: vfsub_fv_v8f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e32, m2, ta, ma
+; CHECK-NEXT:    vfrsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <8 x float> poison, float %b, i32 0
+  %splat = shufflevector <8 x float> %head, <8 x float> poison, <8 x i32> zeroinitializer
+  %vc = call <8 x float> @llvm.experimental.constrained.fsub.v8f32(<8 x float> %splat, <8 x float> %va, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x float> %vc
+}
+
+declare <16 x float> @llvm.experimental.constrained.fsub.v16f32(<16 x float>, <16 x float>, metadata, metadata)
+define <16 x float> @vfsub_vv_v16f32(<16 x float> %va, <16 x float> %vb) {
+; CHECK-LABEL: vfsub_vv_v16f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <16 x float> @llvm.experimental.constrained.fsub.v16f32(<16 x float> %va, <16 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <16 x float> %vc
+}
+
+define <16 x float> @vfsub_vf_v16f32(<16 x float> %va, float %b) {
+; CHECK-LABEL: vfsub_vf_v16f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <16 x float> poison, float %b, i32 0
+  %splat = shufflevector <16 x float> %head, <16 x float> poison, <16 x i32> zeroinitializer
+  %vc = call <16 x float> @llvm.experimental.constrained.fsub.v16f32(<16 x float> %va, <16 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <16 x float> %vc
+}
+
+declare <2 x double> @llvm.experimental.constrained.fsub.v2f64(<2 x double>, <2 x double>, metadata, metadata)
+define <2 x double> @vfsub_vv_v2f64(<2 x double> %va, <2 x double> %vb) {
+; CHECK-LABEL: vfsub_vv_v2f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <2 x double> @llvm.experimental.constrained.fsub.v2f64(<2 x double> %va, <2 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x double> %vc
+}
+
+define <2 x double> @vfsub_vf_v2f64(<2 x double> %va, double %b) {
+; CHECK-LABEL: vfsub_vf_v2f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <2 x double> poison, double %b, i32 0
+  %splat = shufflevector <2 x double> %head, <2 x double> poison, <2 x i32> zeroinitializer
+  %vc = call <2 x double> @llvm.experimental.constrained.fsub.v2f64(<2 x double> %va, <2 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <2 x double> %vc
+}
+
+declare <4 x double> @llvm.experimental.constrained.fsub.v4f64(<4 x double>, <4 x double>, metadata, metadata)
+define <4 x double> @vfsub_vv_v4f64(<4 x double> %va, <4 x double> %vb) {
+; CHECK-LABEL: vfsub_vv_v4f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <4 x double> @llvm.experimental.constrained.fsub.v4f64(<4 x double> %va, <4 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x double> %vc
+}
+
+define <4 x double> @vfsub_vf_v4f64(<4 x double> %va, double %b) {
+; CHECK-LABEL: vfsub_vf_v4f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <4 x double> poison, double %b, i32 0
+  %splat = shufflevector <4 x double> %head, <4 x double> poison, <4 x i32> zeroinitializer
+  %vc = call <4 x double> @llvm.experimental.constrained.fsub.v4f64(<4 x double> %va, <4 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <4 x double> %vc
+}
+
+declare <8 x double> @llvm.experimental.constrained.fsub.v8f64(<8 x double>, <8 x double>, metadata, metadata)
+define <8 x double> @vfsub_vv_v8f64(<8 x double> %va, <8 x double> %vb) {
+; CHECK-LABEL: vfsub_vv_v8f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <8 x double> @llvm.experimental.constrained.fsub.v8f64(<8 x double> %va, <8 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x double> %vc
+}
+
+define <8 x double> @vfsub_vf_v8f64(<8 x double> %va, double %b) {
+; CHECK-LABEL: vfsub_vf_v8f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <8 x double> poison, double %b, i32 0
+  %splat = shufflevector <8 x double> %head, <8 x double> poison, <8 x i32> zeroinitializer
+  %vc = call <8 x double> @llvm.experimental.constrained.fsub.v8f64(<8 x double> %va, <8 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x double> %vc
+}
+
+define <8 x double> @vfsub_fv_v8f64(<8 x double> %va, double %b) {
+; CHECK-LABEL: vfsub_fv_v8f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
+; CHECK-NEXT:    vfrsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <8 x double> poison, double %b, i32 0
+  %splat = shufflevector <8 x double> %head, <8 x double> poison, <8 x i32> zeroinitializer
+  %vc = call <8 x double> @llvm.experimental.constrained.fsub.v8f64(<8 x double> %splat, <8 x double> %va, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <8 x double> %vc
+}
diff --git a/llvm/test/CodeGen/RISCV/rvv/insert-subvector.ll b/llvm/test/CodeGen/RISCV/rvv/insert-subvector.ll
index 12d7409de82c0..03f1a7e6a932e 100644
--- a/llvm/test/CodeGen/RISCV/rvv/insert-subvector.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/insert-subvector.ll
@@ -367,7 +367,6 @@ define <vscale x 32 x half> @insert_nxv32f16_nxv2f16_26(<vscale x 32 x half> %ve
 define <vscale x 32 x half> @insert_nxv32f16_undef_nxv1f16_0(<vscale x 1 x half> %subvec) {
 ; CHECK-LABEL: insert_nxv32f16_undef_nxv1f16_0:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8m8
 ; CHECK-NEXT:    ret
   %v = call <vscale x 32 x half> @llvm.vector.insert.nxv1f16.nxv32f16(<vscale x 32 x half> undef, <vscale x 1 x half> %subvec, i64 0)
   ret <vscale x 32 x half> %v
@@ -381,8 +380,7 @@ define <vscale x 32 x half> @insert_nxv32f16_undef_nxv1f16_26(<vscale x 1 x half
 ; CHECK-NEXT:    srli a0, a0, 2
 ; CHECK-NEXT:    add a1, a0, a1
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vslideup.vx v22, v8, a0
-; CHECK-NEXT:    vmv8r.v v8, v16
+; CHECK-NEXT:    vslideup.vx v14, v8, a0
 ; CHECK-NEXT:    ret
   %v = call <vscale x 32 x half> @llvm.vector.insert.nxv1f16.nxv32f16(<vscale x 32 x half> undef, <vscale x 1 x half> %subvec, i64 26)
   ret <vscale x 32 x half> %v
diff --git a/llvm/test/CodeGen/RISCV/rvv/rv32-spill-zvlsseg.ll b/llvm/test/CodeGen/RISCV/rvv/rv32-spill-zvlsseg.ll
index c88dfabc34eb3..b22f483ce90ca 100644
--- a/llvm/test/CodeGen/RISCV/rvv/rv32-spill-zvlsseg.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/rv32-spill-zvlsseg.ll
@@ -45,7 +45,6 @@ define <vscale x 1 x i32> @spill_zvlsseg_nxv1i32(ptr %base, i32 %vl) nounwind {
 ; SPILL-O2-NEXT:    vl1r.v v7, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl1r.v v8, (a0) # Unknown-size Folded Reload
-; SPILL-O2-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    add sp, sp, a0
@@ -100,7 +99,6 @@ define <vscale x 2 x i32> @spill_zvlsseg_nxv2i32(ptr %base, i32 %vl) nounwind {
 ; SPILL-O2-NEXT:    vl1r.v v7, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl1r.v v8, (a0) # Unknown-size Folded Reload
-; SPILL-O2-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    add sp, sp, a0
@@ -158,7 +156,6 @@ define <vscale x 4 x i32> @spill_zvlsseg_nxv4i32(ptr %base, i32 %vl) nounwind {
 ; SPILL-O2-NEXT:    vl2r.v v6, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl2r.v v8, (a0) # Unknown-size Folded Reload
-; SPILL-O2-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 2
 ; SPILL-O2-NEXT:    add sp, sp, a0
@@ -216,7 +213,6 @@ define <vscale x 8 x i32> @spill_zvlsseg_nxv8i32(ptr %base, i32 %vl) nounwind {
 ; SPILL-O2-NEXT:    vl4r.v v4, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl4r.v v8, (a0) # Unknown-size Folded Reload
-; SPILL-O2-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 3
 ; SPILL-O2-NEXT:    add sp, sp, a0
@@ -279,7 +275,6 @@ define <vscale x 4 x i32> @spill_zvlsseg3_nxv4i32(ptr %base, i32 %vl) nounwind {
 ; SPILL-O2-NEXT:    vl2r.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl2r.v v10, (a0) # Unknown-size Folded Reload
-; SPILL-O2-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    li a1, 6
 ; SPILL-O2-NEXT:    mul a0, a0, a1
diff --git a/llvm/test/CodeGen/RISCV/rvv/rv64-spill-zvlsseg.ll b/llvm/test/CodeGen/RISCV/rvv/rv64-spill-zvlsseg.ll
index 37f87fecc23fa..32e0060322d36 100644
--- a/llvm/test/CodeGen/RISCV/rvv/rv64-spill-zvlsseg.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/rv64-spill-zvlsseg.ll
@@ -45,7 +45,6 @@ define <vscale x 1 x i32> @spill_zvlsseg_nxv1i32(ptr %base, i64 %vl) nounwind {
 ; SPILL-O2-NEXT:    vl1r.v v7, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl1r.v v8, (a0) # Unknown-size Folded Reload
-; SPILL-O2-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    add sp, sp, a0
@@ -100,7 +99,6 @@ define <vscale x 2 x i32> @spill_zvlsseg_nxv2i32(ptr %base, i64 %vl) nounwind {
 ; SPILL-O2-NEXT:    vl1r.v v7, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl1r.v v8, (a0) # Unknown-size Folded Reload
-; SPILL-O2-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    add sp, sp, a0
@@ -158,7 +156,6 @@ define <vscale x 4 x i32> @spill_zvlsseg_nxv4i32(ptr %base, i64 %vl) nounwind {
 ; SPILL-O2-NEXT:    vl2r.v v6, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl2r.v v8, (a0) # Unknown-size Folded Reload
-; SPILL-O2-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 2
 ; SPILL-O2-NEXT:    add sp, sp, a0
@@ -216,7 +213,6 @@ define <vscale x 8 x i32> @spill_zvlsseg_nxv8i32(ptr %base, i64 %vl) nounwind {
 ; SPILL-O2-NEXT:    vl4r.v v4, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl4r.v v8, (a0) # Unknown-size Folded Reload
-; SPILL-O2-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 3
 ; SPILL-O2-NEXT:    add sp, sp, a0
@@ -279,7 +275,6 @@ define <vscale x 4 x i32> @spill_zvlsseg3_nxv4i32(ptr %base, i64 %vl) nounwind {
 ; SPILL-O2-NEXT:    vl2r.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl2r.v v10, (a0) # Unknown-size Folded Reload
-; SPILL-O2-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    li a1, 6
 ; SPILL-O2-NEXT:    mul a0, a0, a1
diff --git a/llvm/test/CodeGen/RISCV/rvv/rvv-peephole-vmerge-masked-vops.ll b/llvm/test/CodeGen/RISCV/rvv/rvv-peephole-vmerge-masked-vops.ll
new file mode 100644
index 0000000000000..11a752f01a542
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/rvv/rvv-peephole-vmerge-masked-vops.ll
@@ -0,0 +1,95 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=riscv64 -mattr=+v | FileCheck %s
+
+declare <vscale x 2 x i32> @llvm.riscv.vmerge.nxv2i32.nxv2i32(<vscale x 2 x i32>, <vscale x 2 x i32>, <vscale x 2 x i32>, <vscale x 2 x i1>, i64);
+declare <vscale x 2 x float> @llvm.riscv.vmerge.nxv2f32.nxv2f32(<vscale x 2 x float>, <vscale x 2 x float>, <vscale x 2 x float>, <vscale x 2 x i1>, i64);
+
+define <vscale x 2 x i32> @vpmerge_vadd(<vscale x 2 x i32> %passthru, <vscale x 2 x i32> %x, <vscale x 2 x i32> %y, <vscale x 2 x i1> %m, i64 %vl) {
+; CHECK-LABEL: vpmerge_vadd:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a0, e32, m1, tu, mu
+; CHECK-NEXT:    vadd.vv v8, v9, v10, v0.t
+; CHECK-NEXT:    ret
+  %a = call <vscale x 2 x i32> @llvm.riscv.vadd.mask.nxv2i32.nxv2i32(<vscale x 2 x i32> %passthru, <vscale x 2 x i32> %x, <vscale x 2 x i32> %y, <vscale x 2 x i1> %m, i64 %vl, i64 1)
+  %splat = insertelement <vscale x 2 x i1> poison, i1 -1, i32 0
+  %mask = shufflevector <vscale x 2 x i1> %splat, <vscale x 2 x i1> poison, <vscale x 2 x i32> zeroinitializer
+  %b = call <vscale x 2 x i32> @llvm.riscv.vmerge.nxv2i32.nxv2i32(<vscale x 2 x i32> %passthru, <vscale x 2 x i32> %passthru, <vscale x 2 x i32> %a, <vscale x 2 x i1> %mask, i64 %vl)
+  ret <vscale x 2 x i32> %b
+}
+declare <vscale x 2 x i32> @llvm.riscv.vadd.mask.nxv2i32.nxv2i32(<vscale x 2 x i32>, <vscale x 2 x i32>, <vscale x 2 x i32>, <vscale x 2 x i1>, i64, i64)
+
+define <vscale x 2 x i32> @vpmerge_vsub(<vscale x 2 x i32> %passthru, <vscale x 2 x i32> %x, <vscale x 2 x i32> %y, <vscale x 2 x i1> %m, i64 %vl) {
+; CHECK-LABEL: vpmerge_vsub:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a0, e32, m1, tu, mu
+; CHECK-NEXT:    vsub.vv v8, v9, v10, v0.t
+; CHECK-NEXT:    ret
+  %a = call <vscale x 2 x i32> @llvm.riscv.vsub.mask.nxv2i32.nxv2i32(<vscale x 2 x i32> %passthru, <vscale x 2 x i32> %x, <vscale x 2 x i32> %y, <vscale x 2 x i1> %m, i64 %vl, i64 1)
+  %splat = insertelement <vscale x 2 x i1> poison, i1 -1, i32 0
+  %mask = shufflevector <vscale x 2 x i1> %splat, <vscale x 2 x i1> poison, <vscale x 2 x i32> zeroinitializer
+  %b = call <vscale x 2 x i32> @llvm.riscv.vmerge.nxv2i32.nxv2i32(<vscale x 2 x i32> %passthru, <vscale x 2 x i32> %passthru, <vscale x 2 x i32> %a, <vscale x 2 x i1> %mask, i64 %vl)
+  ret <vscale x 2 x i32> %b
+}
+declare <vscale x 2 x i32> @llvm.riscv.vsub.mask.nxv2i32.nxv2i32(<vscale x 2 x i32>, <vscale x 2 x i32>, <vscale x 2 x i32>, <vscale x 2 x i1>, i64, i64)
+
+define <vscale x 2 x float> @vpmerge_vfadd(<vscale x 2 x float> %passthru, <vscale x 2 x float> %x, <vscale x 2 x float> %y, <vscale x 2 x i1> %m, i64 %vl) {
+; CHECK-LABEL: vpmerge_vfadd:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a0, e32, m1, tu, mu
+; CHECK-NEXT:    vfadd.vv v8, v9, v10, v0.t
+; CHECK-NEXT:    ret
+  %a = call <vscale x 2 x float> @llvm.riscv.vfadd.mask.nxv2f32.nxv2f32(<vscale x 2 x float> %passthru, <vscale x 2 x float> %x, <vscale x 2 x float> %y, <vscale x 2 x i1> %m, i64 %vl, i64 1)
+  %splat = insertelement <vscale x 2 x i1> poison, i1 -1, i32 0
+  %mask = shufflevector <vscale x 2 x i1> %splat, <vscale x 2 x i1> poison, <vscale x 2 x i32> zeroinitializer
+  %b = call <vscale x 2 x float> @llvm.riscv.vmerge.nxv2f32.nxv2f32(<vscale x 2 x float> %passthru, <vscale x 2 x float> %passthru, <vscale x 2 x float> %a, <vscale x 2 x i1> %mask, i64 %vl)
+
+  ret <vscale x 2 x float> %b
+}
+declare <vscale x 2 x float> @llvm.riscv.vfadd.mask.nxv2f32.nxv2f32(<vscale x 2 x float>, <vscale x 2 x float>, <vscale x 2 x float>, <vscale x 2 x i1>, i64, i64)
+
+define <vscale x 2 x float> @vpmerge_vfsub(<vscale x 2 x float> %passthru, <vscale x 2 x float> %x, <vscale x 2 x float> %y, <vscale x 2 x i1> %m, i64 %vl) {
+; CHECK-LABEL: vpmerge_vfsub:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a0, e32, m1, tu, mu
+; CHECK-NEXT:    vfsub.vv v8, v9, v10, v0.t
+; CHECK-NEXT:    ret
+  %a = call <vscale x 2 x float> @llvm.riscv.vfsub.mask.nxv2f32.nxv2f32(<vscale x 2 x float> %passthru, <vscale x 2 x float> %x, <vscale x 2 x float> %y, <vscale x 2 x i1> %m, i64 %vl, i64 1)
+  %splat = insertelement <vscale x 2 x i1> poison, i1 -1, i32 0
+  %mask = shufflevector <vscale x 2 x i1> %splat, <vscale x 2 x i1> poison, <vscale x 2 x i32> zeroinitializer
+  %b = call <vscale x 2 x float> @llvm.riscv.vmerge.nxv2f32.nxv2f32(<vscale x 2 x float> %passthru, <vscale x 2 x float> %passthru, <vscale x 2 x float> %a, <vscale x 2 x i1> %mask, i64 %vl)
+  ret <vscale x 2 x float> %b
+}
+declare <vscale x 2 x float> @llvm.riscv.vfsub.mask.nxv2f32.nxv2f32(<vscale x 2 x float>, <vscale x 2 x float>, <vscale x 2 x float>, <vscale x 2 x i1>, i64, i64)
+
+define <vscale x 2 x i32> @vpmerge_vwadd(<vscale x 2 x i32> %passthru, <vscale x 2 x i16> %x, <vscale x 2 x i16> %y, <vscale x 2 x i1> %m, i64 %vl) {
+; CHECK-LABEL: vpmerge_vwadd:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a0, e16, mf2, tu, mu
+; CHECK-NEXT:    vwadd.vv v8, v9, v10, v0.t
+; CHECK-NEXT:    ret
+  %a = call <vscale x 2 x i32> @llvm.riscv.vwadd.mask.nxv2i32.nxv2i16.nxv2i16(<vscale x 2 x i32> %passthru, <vscale x 2 x i16> %x, <vscale x 2 x i16> %y, <vscale x 2 x i1> %m, i64 %vl, i64 1)
+  %splat = insertelement <vscale x 2 x i1> poison, i1 -1, i32 0
+  %mask = shufflevector <vscale x 2 x i1> %splat, <vscale x 2 x i1> poison, <vscale x 2 x i32> zeroinitializer
+  %b = call <vscale x 2 x i32> @llvm.riscv.vmerge.nxv2i32.nxv2i32(<vscale x 2 x i32> %passthru, <vscale x 2 x i32> %passthru, <vscale x 2 x i32> %a, <vscale x 2 x i1> %mask, i64 %vl)
+  ret <vscale x 2 x i32> %b
+}
+declare <vscale x 2 x i32> @llvm.riscv.vwadd.mask.nxv2i32.nxv2i16.nxv2i16(<vscale x 2 x i32>, <vscale x 2 x i16>, <vscale x 2 x i16>, <vscale x 2 x i1>, i64, i64)
+
+define <vscale x 2 x i32> @vpmerge_vle(<vscale x 2 x i32> %passthru, <vscale x 2 x i32>* %p, <vscale x 2 x i1> %m, i64 %vl) {
+; CHECK-LABEL: vpmerge_vle:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e32, m1, tu, mu
+; CHECK-NEXT:    vle32.v v8, (a0), v0.t
+; CHECK-NEXT:    ret
+  %a = call <vscale x 2 x i32> @llvm.riscv.vle.mask.nxv2i32(
+    <vscale x 2 x i32> %passthru,
+    <vscale x 2 x i32>* %p,
+    <vscale x 2 x i1> %m,
+    i64 %vl, i64 1)
+  %splat = insertelement <vscale x 2 x i1> poison, i1 -1, i32 0
+  %mask = shufflevector <vscale x 2 x i1> %splat, <vscale x 2 x i1> poison, <vscale x 2 x i32> zeroinitializer
+  %b = call <vscale x 2 x i32> @llvm.riscv.vmerge.nxv2i32.nxv2i32(<vscale x 2 x i32> %passthru, <vscale x 2 x i32> %passthru, <vscale x 2 x i32> %a, <vscale x 2 x i1> %mask, i64 %vl)
+  ret <vscale x 2 x i32> %b
+}
+declare <vscale x 2 x i32> @llvm.riscv.vle.mask.nxv2i32(<vscale x 2 x i32>, <vscale x 2 x i32>*, <vscale x 2 x i1>, i64, i64)
+
diff --git a/llvm/test/CodeGen/RISCV/rvv/sink-splat-operands.ll b/llvm/test/CodeGen/RISCV/rvv/sink-splat-operands.ll
index 01be29994cbe6..583e29b82c675 100644
--- a/llvm/test/CodeGen/RISCV/rvv/sink-splat-operands.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/sink-splat-operands.ll
@@ -275,7 +275,7 @@ define void @sink_splat_mul_scalable(ptr nocapture %a, i32 signext %x) {
 ; CHECK-NEXT:  .LBB7_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    lw a2, 0(a0)
-; CHECK-NEXT:    mulw a2, a2, a1
+; CHECK-NEXT:    mul a2, a2, a1
 ; CHECK-NEXT:    sw a2, 0(a0)
 ; CHECK-NEXT:    addi a3, a3, 1
 ; CHECK-NEXT:    addi a0, a0, 4
diff --git a/llvm/test/CodeGen/RISCV/rvv/vector-deinterleave-fixed.ll b/llvm/test/CodeGen/RISCV/rvv/vector-deinterleave-fixed.ll
index 81675626bb055..efcfd326f129d 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vector-deinterleave-fixed.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vector-deinterleave-fixed.ll
@@ -298,10 +298,9 @@ define {<2 x i64>, <2 x i64>} @vector_deinterleave_v2i64_v4i64(<4 x i64> %vec) {
 ; CHECK-NEXT:    vsetivli zero, 2, e64, m1, ta, mu
 ; CHECK-NEXT:    vrgather.vi v10, v8, 0
 ; CHECK-NEXT:    vrgather.vi v10, v12, 0, v0.t
-; CHECK-NEXT:    vrgather.vi v11, v8, 1
-; CHECK-NEXT:    vrgather.vi v11, v12, 1, v0.t
+; CHECK-NEXT:    vrgather.vi v9, v8, 1
+; CHECK-NEXT:    vrgather.vi v9, v12, 1, v0.t
 ; CHECK-NEXT:    vmv.v.v v8, v10
-; CHECK-NEXT:    vmv.v.v v9, v11
 ; CHECK-NEXT:    ret
 %retval = call {<2 x i64>, <2 x i64>} @llvm.experimental.vector.deinterleave2.v4i64(<4 x i64> %vec)
 ret {<2 x i64>, <2 x i64>} %retval
@@ -388,10 +387,9 @@ define {<2 x double>, <2 x double>} @vector_deinterleave_v2f64_v4f64(<4 x double
 ; CHECK-NEXT:    vsetivli zero, 2, e64, m1, ta, mu
 ; CHECK-NEXT:    vrgather.vi v10, v8, 0
 ; CHECK-NEXT:    vrgather.vi v10, v12, 0, v0.t
-; CHECK-NEXT:    vrgather.vi v11, v8, 1
-; CHECK-NEXT:    vrgather.vi v11, v12, 1, v0.t
+; CHECK-NEXT:    vrgather.vi v9, v8, 1
+; CHECK-NEXT:    vrgather.vi v9, v12, 1, v0.t
 ; CHECK-NEXT:    vmv.v.v v8, v10
-; CHECK-NEXT:    vmv.v.v v9, v11
 ; CHECK-NEXT:    ret
 %retval = call {<2 x double>, <2 x double>} @llvm.experimental.vector.deinterleave2.v4f64(<4 x double> %vec)
 ret {<2 x double>, <2 x double>} %retval
diff --git a/llvm/test/CodeGen/RISCV/rvv/vector-interleave-fixed.ll b/llvm/test/CodeGen/RISCV/rvv/vector-interleave-fixed.ll
index 38d5196a82c75..914782e9839fd 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vector-interleave-fixed.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vector-interleave-fixed.ll
@@ -284,7 +284,6 @@ define <4 x i64> @vector_interleave_v4i64_v2i64(<2 x i64> %a, <2 x i64> %b) {
 ; RV32-LABEL: vector_interleave_v4i64_v2i64:
 ; RV32:       # %bb.0:
 ; RV32-NEXT:    vmv1r.v v10, v9
-; RV32-NEXT:    # kill: def $v8 killed $v8 def $v8m2
 ; RV32-NEXT:    vsetivli zero, 8, e32, m2, ta, ma
 ; RV32-NEXT:    vmv.v.i v12, 0
 ; RV32-NEXT:    vsetivli zero, 2, e64, m2, tu, ma
@@ -301,7 +300,6 @@ define <4 x i64> @vector_interleave_v4i64_v2i64(<2 x i64> %a, <2 x i64> %b) {
 ; RV64-LABEL: vector_interleave_v4i64_v2i64:
 ; RV64:       # %bb.0:
 ; RV64-NEXT:    vmv1r.v v10, v9
-; RV64-NEXT:    # kill: def $v8 killed $v8 def $v8m2
 ; RV64-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
 ; RV64-NEXT:    vmv.v.i v12, 0
 ; RV64-NEXT:    vsetivli zero, 2, e64, m2, tu, ma
@@ -394,7 +392,6 @@ define <4 x double> @vector_interleave_v4f64_v2f64(<2 x double> %a, <2 x double>
 ; RV32-LABEL: vector_interleave_v4f64_v2f64:
 ; RV32:       # %bb.0:
 ; RV32-NEXT:    vmv1r.v v10, v9
-; RV32-NEXT:    # kill: def $v8 killed $v8 def $v8m2
 ; RV32-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
 ; RV32-NEXT:    vmv.v.i v12, 0
 ; RV32-NEXT:    vsetivli zero, 2, e64, m2, tu, ma
@@ -411,7 +408,6 @@ define <4 x double> @vector_interleave_v4f64_v2f64(<2 x double> %a, <2 x double>
 ; RV64-LABEL: vector_interleave_v4f64_v2f64:
 ; RV64:       # %bb.0:
 ; RV64-NEXT:    vmv1r.v v10, v9
-; RV64-NEXT:    # kill: def $v8 killed $v8 def $v8m2
 ; RV64-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
 ; RV64-NEXT:    vmv.v.i v12, 0
 ; RV64-NEXT:    vsetivli zero, 2, e64, m2, tu, ma
diff --git a/llvm/test/CodeGen/RISCV/rvv/vector-interleave.ll b/llvm/test/CodeGen/RISCV/rvv/vector-interleave.ll
index fb70a1ced5172..f6e6de24af91f 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vector-interleave.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vector-interleave.ll
@@ -60,7 +60,6 @@ define <vscale x 8 x i32> @vector_interleave_nxv8i32_nxv4i32(<vscale x 4 x i32>
 define <vscale x 4 x i64> @vector_interleave_nxv4i64_nxv2i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b) {
 ; CHECK-LABEL: vector_interleave_nxv4i64_nxv2i64:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $v10m2 killed $v10m2 killed $v8m4 def $v8m4
 ; CHECK-NEXT:    csrr a0, vlenb
 ; CHECK-NEXT:    srli a0, a0, 2
 ; CHECK-NEXT:    vsetvli a1, zero, e16, m1, ta, mu
@@ -70,7 +69,6 @@ define <vscale x 4 x i64> @vector_interleave_nxv4i64_nxv2i64(<vscale x 2 x i64>
 ; CHECK-NEXT:    vsrl.vi v16, v12, 1
 ; CHECK-NEXT:    vadd.vx v16, v16, a0, v0.t
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m4, ta, ma
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m4 def $v8m4
 ; CHECK-NEXT:    vrgatherei16.vv v12, v8, v16, v0.t
 ; CHECK-NEXT:    vmv.v.v v8, v12
 ; CHECK-NEXT:    ret
@@ -160,7 +158,6 @@ define <vscale x 8 x float> @vector_interleave_nxv8f32_nxv4f32(<vscale x 4 x flo
 define <vscale x 4 x double> @vector_interleave_nxv4f64_nxv2f64(<vscale x 2 x double> %a, <vscale x 2 x double> %b) {
 ; CHECK-LABEL: vector_interleave_nxv4f64_nxv2f64:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $v10m2 killed $v10m2 killed $v8m4 def $v8m4
 ; CHECK-NEXT:    csrr a0, vlenb
 ; CHECK-NEXT:    srli a0, a0, 2
 ; CHECK-NEXT:    vsetvli a1, zero, e16, m1, ta, mu
@@ -170,7 +167,6 @@ define <vscale x 4 x double> @vector_interleave_nxv4f64_nxv2f64(<vscale x 2 x do
 ; CHECK-NEXT:    vsrl.vi v16, v12, 1
 ; CHECK-NEXT:    vadd.vx v16, v16, a0, v0.t
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m4, ta, ma
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m4 def $v8m4
 ; CHECK-NEXT:    vrgatherei16.vv v12, v8, v16, v0.t
 ; CHECK-NEXT:    vmv.v.v v8, v12
 ; CHECK-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/rvv/vfadd-constrained-sdnode.ll b/llvm/test/CodeGen/RISCV/rvv/vfadd-constrained-sdnode.ll
new file mode 100644
index 0000000000000..313a58efe902b
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/rvv/vfadd-constrained-sdnode.ll
@@ -0,0 +1,365 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=ilp32d \
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=riscv64 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=lp64d \
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+
+declare <vscale x 1 x half> @llvm.experimental.constrained.fadd.nxv1f16(<vscale x 1 x half>, <vscale x 1 x half>, metadata, metadata)
+define <vscale x 1 x half> @vfadd_vv_nxv1f16(<vscale x 1 x half> %va, <vscale x 1 x half> %vb) {
+; CHECK-LABEL: vfadd_vv_nxv1f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf4, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 1 x half> @llvm.experimental.constrained.fadd.nxv1f16(<vscale x 1 x half> %va, <vscale x 1 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x half> %vc
+}
+
+define <vscale x 1 x half> @vfadd_vf_nxv1f16(<vscale x 1 x half> %va, half %b) {
+; CHECK-LABEL: vfadd_vf_nxv1f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf4, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 1 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 1 x half> %head, <vscale x 1 x half> poison, <vscale x 1 x i32> zeroinitializer
+  %vc = call <vscale x 1 x half> @llvm.experimental.constrained.fadd.nxv1f16(<vscale x 1 x half> %va, <vscale x 1 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x half> %vc
+}
+
+declare <vscale x 2 x half> @llvm.experimental.constrained.fadd.nxv2f16(<vscale x 2 x half>, <vscale x 2 x half>, metadata, metadata)
+define <vscale x 2 x half> @vfadd_vv_nxv2f16(<vscale x 2 x half> %va, <vscale x 2 x half> %vb) {
+; CHECK-LABEL: vfadd_vv_nxv2f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf2, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 2 x half> @llvm.experimental.constrained.fadd.nxv2f16(<vscale x 2 x half> %va, <vscale x 2 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x half> %vc
+}
+
+define <vscale x 2 x half> @vfadd_vf_nxv2f16(<vscale x 2 x half> %va, half %b) {
+; CHECK-LABEL: vfadd_vf_nxv2f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf2, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 2 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 2 x half> %head, <vscale x 2 x half> poison, <vscale x 2 x i32> zeroinitializer
+  %vc = call <vscale x 2 x half> @llvm.experimental.constrained.fadd.nxv2f16(<vscale x 2 x half> %va, <vscale x 2 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x half> %vc
+}
+
+declare <vscale x 4 x half> @llvm.experimental.constrained.fadd.nxv4f16(<vscale x 4 x half>, <vscale x 4 x half>, metadata, metadata)
+define <vscale x 4 x half> @vfadd_vv_nxv4f16(<vscale x 4 x half> %va, <vscale x 4 x half> %vb) {
+; CHECK-LABEL: vfadd_vv_nxv4f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, m1, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 4 x half> @llvm.experimental.constrained.fadd.nxv4f16(<vscale x 4 x half> %va, <vscale x 4 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x half> %vc
+}
+
+define <vscale x 4 x half> @vfadd_vf_nxv4f16(<vscale x 4 x half> %va, half %b) {
+; CHECK-LABEL: vfadd_vf_nxv4f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m1, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 4 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 4 x half> %head, <vscale x 4 x half> poison, <vscale x 4 x i32> zeroinitializer
+  %vc = call <vscale x 4 x half> @llvm.experimental.constrained.fadd.nxv4f16(<vscale x 4 x half> %va, <vscale x 4 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x half> %vc
+}
+
+declare <vscale x 8 x half> @llvm.experimental.constrained.fadd.nxv8f16(<vscale x 8 x half>, <vscale x 8 x half>, metadata, metadata)
+define <vscale x 8 x half> @vfadd_vv_nxv8f16(<vscale x 8 x half> %va, <vscale x 8 x half> %vb) {
+; CHECK-LABEL: vfadd_vv_nxv8f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, m2, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 8 x half> @llvm.experimental.constrained.fadd.nxv8f16(<vscale x 8 x half> %va, <vscale x 8 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x half> %vc
+}
+
+define <vscale x 8 x half> @vfadd_vf_nxv8f16(<vscale x 8 x half> %va, half %b) {
+; CHECK-LABEL: vfadd_vf_nxv8f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m2, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 8 x half> %head, <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
+  %vc = call <vscale x 8 x half> @llvm.experimental.constrained.fadd.nxv8f16(<vscale x 8 x half> %va, <vscale x 8 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x half> %vc
+}
+
+declare <vscale x 16 x half> @llvm.experimental.constrained.fadd.nxv16f16(<vscale x 16 x half>, <vscale x 16 x half>, metadata, metadata)
+define <vscale x 16 x half> @vfadd_vv_nxv16f16(<vscale x 16 x half> %va, <vscale x 16 x half> %vb) {
+; CHECK-LABEL: vfadd_vv_nxv16f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, m4, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 16 x half> @llvm.experimental.constrained.fadd.nxv16f16(<vscale x 16 x half> %va, <vscale x 16 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 16 x half> %vc
+}
+
+define <vscale x 16 x half> @vfadd_vf_nxv16f16(<vscale x 16 x half> %va, half %b) {
+; CHECK-LABEL: vfadd_vf_nxv16f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m4, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 16 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 16 x half> %head, <vscale x 16 x half> poison, <vscale x 16 x i32> zeroinitializer
+  %vc = call <vscale x 16 x half> @llvm.experimental.constrained.fadd.nxv16f16(<vscale x 16 x half> %va, <vscale x 16 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 16 x half> %vc
+}
+
+declare <vscale x 32 x half> @llvm.experimental.constrained.fadd.nxv32f16(<vscale x 32 x half>, <vscale x 32 x half>, metadata, metadata)
+define <vscale x 32 x half> @vfadd_vv_nxv32f16(<vscale x 32 x half> %va, <vscale x 32 x half> %vb) {
+; CHECK-LABEL: vfadd_vv_nxv32f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, m8, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v16
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 32 x half> @llvm.experimental.constrained.fadd.nxv32f16(<vscale x 32 x half> %va, <vscale x 32 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 32 x half> %vc
+}
+
+define <vscale x 32 x half> @vfadd_vf_nxv32f16(<vscale x 32 x half> %va, half %b) {
+; CHECK-LABEL: vfadd_vf_nxv32f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m8, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 32 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 32 x half> %head, <vscale x 32 x half> poison, <vscale x 32 x i32> zeroinitializer
+  %vc = call <vscale x 32 x half> @llvm.experimental.constrained.fadd.nxv32f16(<vscale x 32 x half> %va, <vscale x 32 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 32 x half> %vc
+}
+
+declare <vscale x 1 x float> @llvm.experimental.constrained.fadd.nxv1f32(<vscale x 1 x float>, <vscale x 1 x float>, metadata, metadata)
+define <vscale x 1 x float> @vfadd_vv_nxv1f32(<vscale x 1 x float> %va, <vscale x 1 x float> %vb) {
+; CHECK-LABEL: vfadd_vv_nxv1f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, mf2, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 1 x float> @llvm.experimental.constrained.fadd.nxv1f32(<vscale x 1 x float> %va, <vscale x 1 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x float> %vc
+}
+
+define <vscale x 1 x float> @vfadd_vf_nxv1f32(<vscale x 1 x float> %va, float %b) {
+; CHECK-LABEL: vfadd_vf_nxv1f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, mf2, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 1 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 1 x float> %head, <vscale x 1 x float> poison, <vscale x 1 x i32> zeroinitializer
+  %vc = call <vscale x 1 x float> @llvm.experimental.constrained.fadd.nxv1f32(<vscale x 1 x float> %va, <vscale x 1 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x float> %vc
+}
+
+declare <vscale x 2 x float> @llvm.experimental.constrained.fadd.nxv2f32(<vscale x 2 x float>, <vscale x 2 x float>, metadata, metadata)
+define <vscale x 2 x float> @vfadd_vv_nxv2f32(<vscale x 2 x float> %va, <vscale x 2 x float> %vb) {
+; CHECK-LABEL: vfadd_vv_nxv2f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, m1, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 2 x float> @llvm.experimental.constrained.fadd.nxv2f32(<vscale x 2 x float> %va, <vscale x 2 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x float> %vc
+}
+
+define <vscale x 2 x float> @vfadd_vf_nxv2f32(<vscale x 2 x float> %va, float %b) {
+; CHECK-LABEL: vfadd_vf_nxv2f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m1, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 2 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 2 x float> %head, <vscale x 2 x float> poison, <vscale x 2 x i32> zeroinitializer
+  %vc = call <vscale x 2 x float> @llvm.experimental.constrained.fadd.nxv2f32(<vscale x 2 x float> %va, <vscale x 2 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x float> %vc
+}
+
+declare <vscale x 4 x float> @llvm.experimental.constrained.fadd.nxv4f32(<vscale x 4 x float>, <vscale x 4 x float>, metadata, metadata)
+define <vscale x 4 x float> @vfadd_vv_nxv4f32(<vscale x 4 x float> %va, <vscale x 4 x float> %vb) {
+; CHECK-LABEL: vfadd_vv_nxv4f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, m2, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 4 x float> @llvm.experimental.constrained.fadd.nxv4f32(<vscale x 4 x float> %va, <vscale x 4 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x float> %vc
+}
+
+define <vscale x 4 x float> @vfadd_vf_nxv4f32(<vscale x 4 x float> %va, float %b) {
+; CHECK-LABEL: vfadd_vf_nxv4f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m2, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 4 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 4 x float> %head, <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
+  %vc = call <vscale x 4 x float> @llvm.experimental.constrained.fadd.nxv4f32(<vscale x 4 x float> %va, <vscale x 4 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x float> %vc
+}
+
+declare <vscale x 8 x float> @llvm.experimental.constrained.fadd.nxv8f32(<vscale x 8 x float>, <vscale x 8 x float>, metadata, metadata)
+define <vscale x 8 x float> @vfadd_vv_nxv8f32(<vscale x 8 x float> %va, <vscale x 8 x float> %vb) {
+; CHECK-LABEL: vfadd_vv_nxv8f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, m4, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 8 x float> @llvm.experimental.constrained.fadd.nxv8f32(<vscale x 8 x float> %va, <vscale x 8 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x float> %vc
+}
+
+define <vscale x 8 x float> @vfadd_vf_nxv8f32(<vscale x 8 x float> %va, float %b) {
+; CHECK-LABEL: vfadd_vf_nxv8f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m4, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 8 x float> %head, <vscale x 8 x float> poison, <vscale x 8 x i32> zeroinitializer
+  %vc = call <vscale x 8 x float> @llvm.experimental.constrained.fadd.nxv8f32(<vscale x 8 x float> %va, <vscale x 8 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x float> %vc
+}
+
+declare <vscale x 16 x float> @llvm.experimental.constrained.fadd.nxv16f32(<vscale x 16 x float>, <vscale x 16 x float>, metadata, metadata)
+define <vscale x 16 x float> @vfadd_vv_nxv16f32(<vscale x 16 x float> %va, <vscale x 16 x float> %vb) {
+; CHECK-LABEL: vfadd_vv_nxv16f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, m8, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v16
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 16 x float> @llvm.experimental.constrained.fadd.nxv16f32(<vscale x 16 x float> %va, <vscale x 16 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 16 x float> %vc
+}
+
+define <vscale x 16 x float> @vfadd_vf_nxv16f32(<vscale x 16 x float> %va, float %b) {
+; CHECK-LABEL: vfadd_vf_nxv16f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m8, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 16 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 16 x float> %head, <vscale x 16 x float> poison, <vscale x 16 x i32> zeroinitializer
+  %vc = call <vscale x 16 x float> @llvm.experimental.constrained.fadd.nxv16f32(<vscale x 16 x float> %va, <vscale x 16 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 16 x float> %vc
+}
+
+declare <vscale x 1 x double> @llvm.experimental.constrained.fadd.nxv1f64(<vscale x 1 x double>, <vscale x 1 x double>, metadata, metadata)
+define <vscale x 1 x double> @vfadd_vv_nxv1f64(<vscale x 1 x double> %va, <vscale x 1 x double> %vb) {
+; CHECK-LABEL: vfadd_vv_nxv1f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e64, m1, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 1 x double> @llvm.experimental.constrained.fadd.nxv1f64(<vscale x 1 x double> %va, <vscale x 1 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x double> %vc
+}
+
+define <vscale x 1 x double> @vfadd_vf_nxv1f64(<vscale x 1 x double> %va, double %b) {
+; CHECK-LABEL: vfadd_vf_nxv1f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e64, m1, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 1 x double> poison, double %b, i32 0
+  %splat = shufflevector <vscale x 1 x double> %head, <vscale x 1 x double> poison, <vscale x 1 x i32> zeroinitializer
+  %vc = call <vscale x 1 x double> @llvm.experimental.constrained.fadd.nxv1f64(<vscale x 1 x double> %va, <vscale x 1 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x double> %vc
+}
+
+declare <vscale x 2 x double> @llvm.experimental.constrained.fadd.nxv2f64(<vscale x 2 x double>, <vscale x 2 x double>, metadata, metadata)
+define <vscale x 2 x double> @vfadd_vv_nxv2f64(<vscale x 2 x double> %va, <vscale x 2 x double> %vb) {
+; CHECK-LABEL: vfadd_vv_nxv2f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e64, m2, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 2 x double> @llvm.experimental.constrained.fadd.nxv2f64(<vscale x 2 x double> %va, <vscale x 2 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x double> %vc
+}
+
+define <vscale x 2 x double> @vfadd_vf_nxv2f64(<vscale x 2 x double> %va, double %b) {
+; CHECK-LABEL: vfadd_vf_nxv2f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e64, m2, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 2 x double> poison, double %b, i32 0
+  %splat = shufflevector <vscale x 2 x double> %head, <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
+  %vc = call <vscale x 2 x double> @llvm.experimental.constrained.fadd.nxv2f64(<vscale x 2 x double> %va, <vscale x 2 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x double> %vc
+}
+
+declare <vscale x 4 x double> @llvm.experimental.constrained.fadd.nxv4f64(<vscale x 4 x double>, <vscale x 4 x double>, metadata, metadata)
+define <vscale x 4 x double> @vfadd_vv_nxv4f64(<vscale x 4 x double> %va, <vscale x 4 x double> %vb) {
+; CHECK-LABEL: vfadd_vv_nxv4f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e64, m4, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 4 x double> @llvm.experimental.constrained.fadd.nxv4f64(<vscale x 4 x double> %va, <vscale x 4 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x double> %vc
+}
+
+define <vscale x 4 x double> @vfadd_vf_nxv4f64(<vscale x 4 x double> %va, double %b) {
+; CHECK-LABEL: vfadd_vf_nxv4f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e64, m4, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 4 x double> poison, double %b, i32 0
+  %splat = shufflevector <vscale x 4 x double> %head, <vscale x 4 x double> poison, <vscale x 4 x i32> zeroinitializer
+  %vc = call <vscale x 4 x double> @llvm.experimental.constrained.fadd.nxv4f64(<vscale x 4 x double> %va, <vscale x 4 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x double> %vc
+}
+
+declare <vscale x 8 x double> @llvm.experimental.constrained.fadd.nxv8f64(<vscale x 8 x double>, <vscale x 8 x double>, metadata, metadata)
+define <vscale x 8 x double> @vfadd_vv_nxv8f64(<vscale x 8 x double> %va, <vscale x 8 x double> %vb) {
+; CHECK-LABEL: vfadd_vv_nxv8f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e64, m8, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v8, v16
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 8 x double> @llvm.experimental.constrained.fadd.nxv8f64(<vscale x 8 x double> %va, <vscale x 8 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x double> %vc
+}
+
+define <vscale x 8 x double> @vfadd_vf_nxv8f64(<vscale x 8 x double> %va, double %b) {
+; CHECK-LABEL: vfadd_vf_nxv8f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e64, m8, ta, ma
+; CHECK-NEXT:    vfadd.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x double> poison, double %b, i32 0
+  %splat = shufflevector <vscale x 8 x double> %head, <vscale x 8 x double> poison, <vscale x 8 x i32> zeroinitializer
+  %vc = call <vscale x 8 x double> @llvm.experimental.constrained.fadd.nxv8f64(<vscale x 8 x double> %va, <vscale x 8 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x double> %vc
+}
diff --git a/llvm/test/CodeGen/RISCV/rvv/vfdiv-constrained-sdnode.ll b/llvm/test/CodeGen/RISCV/rvv/vfdiv-constrained-sdnode.ll
new file mode 100644
index 0000000000000..7c10ca993011d
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/rvv/vfdiv-constrained-sdnode.ll
@@ -0,0 +1,401 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=ilp32d \
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=riscv64 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=lp64d \
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+
+declare <vscale x 1 x half> @llvm.experimental.constrained.fdiv.nxv1f16(<vscale x 1 x half>, <vscale x 1 x half>, metadata, metadata)
+define <vscale x 1 x half> @vfdiv_vv_nxv1f16(<vscale x 1 x half> %va, <vscale x 1 x half> %vb) {
+; CHECK-LABEL: vfdiv_vv_nxv1f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf4, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 1 x half> @llvm.experimental.constrained.fdiv.nxv1f16(<vscale x 1 x half> %va, <vscale x 1 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x half> %vc
+}
+
+define <vscale x 1 x half> @vfdiv_vf_nxv1f16(<vscale x 1 x half> %va, half %b) {
+; CHECK-LABEL: vfdiv_vf_nxv1f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf4, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 1 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 1 x half> %head, <vscale x 1 x half> poison, <vscale x 1 x i32> zeroinitializer
+  %vc = call <vscale x 1 x half> @llvm.experimental.constrained.fdiv.nxv1f16(<vscale x 1 x half> %va, <vscale x 1 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x half> %vc
+}
+
+declare <vscale x 2 x half> @llvm.experimental.constrained.fdiv.nxv2f16(<vscale x 2 x half>, <vscale x 2 x half>, metadata, metadata)
+define <vscale x 2 x half> @vfdiv_vv_nxv2f16(<vscale x 2 x half> %va, <vscale x 2 x half> %vb) {
+; CHECK-LABEL: vfdiv_vv_nxv2f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf2, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 2 x half> @llvm.experimental.constrained.fdiv.nxv2f16(<vscale x 2 x half> %va, <vscale x 2 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x half> %vc
+}
+
+define <vscale x 2 x half> @vfdiv_vf_nxv2f16(<vscale x 2 x half> %va, half %b) {
+; CHECK-LABEL: vfdiv_vf_nxv2f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf2, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 2 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 2 x half> %head, <vscale x 2 x half> poison, <vscale x 2 x i32> zeroinitializer
+  %vc = call <vscale x 2 x half> @llvm.experimental.constrained.fdiv.nxv2f16(<vscale x 2 x half> %va, <vscale x 2 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x half> %vc
+}
+
+declare <vscale x 4 x half> @llvm.experimental.constrained.fdiv.nxv4f16(<vscale x 4 x half>, <vscale x 4 x half>, metadata, metadata)
+define <vscale x 4 x half> @vfdiv_vv_nxv4f16(<vscale x 4 x half> %va, <vscale x 4 x half> %vb) {
+; CHECK-LABEL: vfdiv_vv_nxv4f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, m1, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 4 x half> @llvm.experimental.constrained.fdiv.nxv4f16(<vscale x 4 x half> %va, <vscale x 4 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x half> %vc
+}
+
+define <vscale x 4 x half> @vfdiv_vf_nxv4f16(<vscale x 4 x half> %va, half %b) {
+; CHECK-LABEL: vfdiv_vf_nxv4f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m1, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 4 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 4 x half> %head, <vscale x 4 x half> poison, <vscale x 4 x i32> zeroinitializer
+  %vc = call <vscale x 4 x half> @llvm.experimental.constrained.fdiv.nxv4f16(<vscale x 4 x half> %va, <vscale x 4 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x half> %vc
+}
+
+declare <vscale x 8 x half> @llvm.experimental.constrained.fdiv.nxv8f16(<vscale x 8 x half>, <vscale x 8 x half>, metadata, metadata)
+define <vscale x 8 x half> @vfdiv_vv_nxv8f16(<vscale x 8 x half> %va, <vscale x 8 x half> %vb) {
+; CHECK-LABEL: vfdiv_vv_nxv8f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, m2, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 8 x half> @llvm.experimental.constrained.fdiv.nxv8f16(<vscale x 8 x half> %va, <vscale x 8 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x half> %vc
+}
+
+define <vscale x 8 x half> @vfdiv_vf_nxv8f16(<vscale x 8 x half> %va, half %b) {
+; CHECK-LABEL: vfdiv_vf_nxv8f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m2, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 8 x half> %head, <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
+  %vc = call <vscale x 8 x half> @llvm.experimental.constrained.fdiv.nxv8f16(<vscale x 8 x half> %va, <vscale x 8 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x half> %vc
+}
+
+define <vscale x 8 x half> @vfdiv_fv_nxv8f16(<vscale x 8 x half> %va, half %b) {
+; CHECK-LABEL: vfdiv_fv_nxv8f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m2, ta, ma
+; CHECK-NEXT:    vfrdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 8 x half> %head, <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
+  %vc = call <vscale x 8 x half> @llvm.experimental.constrained.fdiv.nxv8f16(<vscale x 8 x half> %splat, <vscale x 8 x half> %va, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x half> %vc
+}
+
+declare <vscale x 16 x half> @llvm.experimental.constrained.fdiv.nxv16f16(<vscale x 16 x half>, <vscale x 16 x half>, metadata, metadata)
+define <vscale x 16 x half> @vfdiv_vv_nxv16f16(<vscale x 16 x half> %va, <vscale x 16 x half> %vb) {
+; CHECK-LABEL: vfdiv_vv_nxv16f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, m4, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 16 x half> @llvm.experimental.constrained.fdiv.nxv16f16(<vscale x 16 x half> %va, <vscale x 16 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 16 x half> %vc
+}
+
+define <vscale x 16 x half> @vfdiv_vf_nxv16f16(<vscale x 16 x half> %va, half %b) {
+; CHECK-LABEL: vfdiv_vf_nxv16f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m4, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 16 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 16 x half> %head, <vscale x 16 x half> poison, <vscale x 16 x i32> zeroinitializer
+  %vc = call <vscale x 16 x half> @llvm.experimental.constrained.fdiv.nxv16f16(<vscale x 16 x half> %va, <vscale x 16 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 16 x half> %vc
+}
+
+declare <vscale x 32 x half> @llvm.experimental.constrained.fdiv.nxv32f16(<vscale x 32 x half>, <vscale x 32 x half>, metadata, metadata)
+define <vscale x 32 x half> @vfdiv_vv_nxv32f16(<vscale x 32 x half> %va, <vscale x 32 x half> %vb) {
+; CHECK-LABEL: vfdiv_vv_nxv32f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, m8, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v16
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 32 x half> @llvm.experimental.constrained.fdiv.nxv32f16(<vscale x 32 x half> %va, <vscale x 32 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 32 x half> %vc
+}
+
+define <vscale x 32 x half> @vfdiv_vf_nxv32f16(<vscale x 32 x half> %va, half %b) {
+; CHECK-LABEL: vfdiv_vf_nxv32f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m8, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 32 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 32 x half> %head, <vscale x 32 x half> poison, <vscale x 32 x i32> zeroinitializer
+  %vc = call <vscale x 32 x half> @llvm.experimental.constrained.fdiv.nxv32f16(<vscale x 32 x half> %va, <vscale x 32 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 32 x half> %vc
+}
+
+declare <vscale x 1 x float> @llvm.experimental.constrained.fdiv.nxv1f32(<vscale x 1 x float>, <vscale x 1 x float>, metadata, metadata)
+define <vscale x 1 x float> @vfdiv_vv_nxv1f32(<vscale x 1 x float> %va, <vscale x 1 x float> %vb) {
+; CHECK-LABEL: vfdiv_vv_nxv1f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, mf2, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 1 x float> @llvm.experimental.constrained.fdiv.nxv1f32(<vscale x 1 x float> %va, <vscale x 1 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x float> %vc
+}
+
+define <vscale x 1 x float> @vfdiv_vf_nxv1f32(<vscale x 1 x float> %va, float %b) {
+; CHECK-LABEL: vfdiv_vf_nxv1f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, mf2, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 1 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 1 x float> %head, <vscale x 1 x float> poison, <vscale x 1 x i32> zeroinitializer
+  %vc = call <vscale x 1 x float> @llvm.experimental.constrained.fdiv.nxv1f32(<vscale x 1 x float> %va, <vscale x 1 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x float> %vc
+}
+
+declare <vscale x 2 x float> @llvm.experimental.constrained.fdiv.nxv2f32(<vscale x 2 x float>, <vscale x 2 x float>, metadata, metadata)
+define <vscale x 2 x float> @vfdiv_vv_nxv2f32(<vscale x 2 x float> %va, <vscale x 2 x float> %vb) {
+; CHECK-LABEL: vfdiv_vv_nxv2f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, m1, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 2 x float> @llvm.experimental.constrained.fdiv.nxv2f32(<vscale x 2 x float> %va, <vscale x 2 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x float> %vc
+}
+
+define <vscale x 2 x float> @vfdiv_vf_nxv2f32(<vscale x 2 x float> %va, float %b) {
+; CHECK-LABEL: vfdiv_vf_nxv2f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m1, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 2 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 2 x float> %head, <vscale x 2 x float> poison, <vscale x 2 x i32> zeroinitializer
+  %vc = call <vscale x 2 x float> @llvm.experimental.constrained.fdiv.nxv2f32(<vscale x 2 x float> %va, <vscale x 2 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x float> %vc
+}
+
+declare <vscale x 4 x float> @llvm.experimental.constrained.fdiv.nxv4f32(<vscale x 4 x float>, <vscale x 4 x float>, metadata, metadata)
+define <vscale x 4 x float> @vfdiv_vv_nxv4f32(<vscale x 4 x float> %va, <vscale x 4 x float> %vb) {
+; CHECK-LABEL: vfdiv_vv_nxv4f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, m2, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 4 x float> @llvm.experimental.constrained.fdiv.nxv4f32(<vscale x 4 x float> %va, <vscale x 4 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x float> %vc
+}
+
+define <vscale x 4 x float> @vfdiv_vf_nxv4f32(<vscale x 4 x float> %va, float %b) {
+; CHECK-LABEL: vfdiv_vf_nxv4f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m2, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 4 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 4 x float> %head, <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
+  %vc = call <vscale x 4 x float> @llvm.experimental.constrained.fdiv.nxv4f32(<vscale x 4 x float> %va, <vscale x 4 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x float> %vc
+}
+
+declare <vscale x 8 x float> @llvm.experimental.constrained.fdiv.nxv8f32(<vscale x 8 x float>, <vscale x 8 x float>, metadata, metadata)
+define <vscale x 8 x float> @vfdiv_vv_nxv8f32(<vscale x 8 x float> %va, <vscale x 8 x float> %vb) {
+; CHECK-LABEL: vfdiv_vv_nxv8f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, m4, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 8 x float> @llvm.experimental.constrained.fdiv.nxv8f32(<vscale x 8 x float> %va, <vscale x 8 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x float> %vc
+}
+
+define <vscale x 8 x float> @vfdiv_vf_nxv8f32(<vscale x 8 x float> %va, float %b) {
+; CHECK-LABEL: vfdiv_vf_nxv8f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m4, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 8 x float> %head, <vscale x 8 x float> poison, <vscale x 8 x i32> zeroinitializer
+  %vc = call <vscale x 8 x float> @llvm.experimental.constrained.fdiv.nxv8f32(<vscale x 8 x float> %va, <vscale x 8 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x float> %vc
+}
+
+define <vscale x 8 x float> @vfdiv_fv_nxv8f32(<vscale x 8 x float> %va, float %b) {
+; CHECK-LABEL: vfdiv_fv_nxv8f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m4, ta, ma
+; CHECK-NEXT:    vfrdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 8 x float> %head, <vscale x 8 x float> poison, <vscale x 8 x i32> zeroinitializer
+  %vc = call <vscale x 8 x float> @llvm.experimental.constrained.fdiv.nxv8f32(<vscale x 8 x float> %splat, <vscale x 8 x float> %va, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x float> %vc
+}
+
+declare <vscale x 16 x float> @llvm.experimental.constrained.fdiv.nxv16f32(<vscale x 16 x float>, <vscale x 16 x float>, metadata, metadata)
+define <vscale x 16 x float> @vfdiv_vv_nxv16f32(<vscale x 16 x float> %va, <vscale x 16 x float> %vb) {
+; CHECK-LABEL: vfdiv_vv_nxv16f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, m8, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v16
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 16 x float> @llvm.experimental.constrained.fdiv.nxv16f32(<vscale x 16 x float> %va, <vscale x 16 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 16 x float> %vc
+}
+
+define <vscale x 16 x float> @vfdiv_vf_nxv16f32(<vscale x 16 x float> %va, float %b) {
+; CHECK-LABEL: vfdiv_vf_nxv16f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m8, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 16 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 16 x float> %head, <vscale x 16 x float> poison, <vscale x 16 x i32> zeroinitializer
+  %vc = call <vscale x 16 x float> @llvm.experimental.constrained.fdiv.nxv16f32(<vscale x 16 x float> %va, <vscale x 16 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 16 x float> %vc
+}
+
+declare <vscale x 1 x double> @llvm.experimental.constrained.fdiv.nxv1f64(<vscale x 1 x double>, <vscale x 1 x double>, metadata, metadata)
+define <vscale x 1 x double> @vfdiv_vv_nxv1f64(<vscale x 1 x double> %va, <vscale x 1 x double> %vb) {
+; CHECK-LABEL: vfdiv_vv_nxv1f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e64, m1, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 1 x double> @llvm.experimental.constrained.fdiv.nxv1f64(<vscale x 1 x double> %va, <vscale x 1 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x double> %vc
+}
+
+define <vscale x 1 x double> @vfdiv_vf_nxv1f64(<vscale x 1 x double> %va, double %b) {
+; CHECK-LABEL: vfdiv_vf_nxv1f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e64, m1, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 1 x double> poison, double %b, i32 0
+  %splat = shufflevector <vscale x 1 x double> %head, <vscale x 1 x double> poison, <vscale x 1 x i32> zeroinitializer
+  %vc = call <vscale x 1 x double> @llvm.experimental.constrained.fdiv.nxv1f64(<vscale x 1 x double> %va, <vscale x 1 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x double> %vc
+}
+
+declare <vscale x 2 x double> @llvm.experimental.constrained.fdiv.nxv2f64(<vscale x 2 x double>, <vscale x 2 x double>, metadata, metadata)
+define <vscale x 2 x double> @vfdiv_vv_nxv2f64(<vscale x 2 x double> %va, <vscale x 2 x double> %vb) {
+; CHECK-LABEL: vfdiv_vv_nxv2f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e64, m2, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 2 x double> @llvm.experimental.constrained.fdiv.nxv2f64(<vscale x 2 x double> %va, <vscale x 2 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x double> %vc
+}
+
+define <vscale x 2 x double> @vfdiv_vf_nxv2f64(<vscale x 2 x double> %va, double %b) {
+; CHECK-LABEL: vfdiv_vf_nxv2f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e64, m2, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 2 x double> poison, double %b, i32 0
+  %splat = shufflevector <vscale x 2 x double> %head, <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
+  %vc = call <vscale x 2 x double> @llvm.experimental.constrained.fdiv.nxv2f64(<vscale x 2 x double> %va, <vscale x 2 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x double> %vc
+}
+
+declare <vscale x 4 x double> @llvm.experimental.constrained.fdiv.nxv4f64(<vscale x 4 x double>, <vscale x 4 x double>, metadata, metadata)
+define <vscale x 4 x double> @vfdiv_vv_nxv4f64(<vscale x 4 x double> %va, <vscale x 4 x double> %vb) {
+; CHECK-LABEL: vfdiv_vv_nxv4f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e64, m4, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 4 x double> @llvm.experimental.constrained.fdiv.nxv4f64(<vscale x 4 x double> %va, <vscale x 4 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x double> %vc
+}
+
+define <vscale x 4 x double> @vfdiv_vf_nxv4f64(<vscale x 4 x double> %va, double %b) {
+; CHECK-LABEL: vfdiv_vf_nxv4f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e64, m4, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 4 x double> poison, double %b, i32 0
+  %splat = shufflevector <vscale x 4 x double> %head, <vscale x 4 x double> poison, <vscale x 4 x i32> zeroinitializer
+  %vc = call <vscale x 4 x double> @llvm.experimental.constrained.fdiv.nxv4f64(<vscale x 4 x double> %va, <vscale x 4 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x double> %vc
+}
+
+declare <vscale x 8 x double> @llvm.experimental.constrained.fdiv.nxv8f64(<vscale x 8 x double>, <vscale x 8 x double>, metadata, metadata)
+define <vscale x 8 x double> @vfdiv_vv_nxv8f64(<vscale x 8 x double> %va, <vscale x 8 x double> %vb) {
+; CHECK-LABEL: vfdiv_vv_nxv8f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e64, m8, ta, ma
+; CHECK-NEXT:    vfdiv.vv v8, v8, v16
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 8 x double> @llvm.experimental.constrained.fdiv.nxv8f64(<vscale x 8 x double> %va, <vscale x 8 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x double> %vc
+}
+
+define <vscale x 8 x double> @vfdiv_vf_nxv8f64(<vscale x 8 x double> %va, double %b) {
+; CHECK-LABEL: vfdiv_vf_nxv8f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e64, m8, ta, ma
+; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x double> poison, double %b, i32 0
+  %splat = shufflevector <vscale x 8 x double> %head, <vscale x 8 x double> poison, <vscale x 8 x i32> zeroinitializer
+  %vc = call <vscale x 8 x double> @llvm.experimental.constrained.fdiv.nxv8f64(<vscale x 8 x double> %va, <vscale x 8 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x double> %vc
+}
+
+define <vscale x 8 x double> @vfdiv_fv_nxv8f64(<vscale x 8 x double> %va, double %b) {
+; CHECK-LABEL: vfdiv_fv_nxv8f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e64, m8, ta, ma
+; CHECK-NEXT:    vfrdiv.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x double> poison, double %b, i32 0
+  %splat = shufflevector <vscale x 8 x double> %head, <vscale x 8 x double> poison, <vscale x 8 x i32> zeroinitializer
+  %vc = call <vscale x 8 x double> @llvm.experimental.constrained.fdiv.nxv8f64(<vscale x 8 x double> %splat, <vscale x 8 x double> %va, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x double> %vc
+}
diff --git a/llvm/test/CodeGen/RISCV/rvv/vfmul-constrained-sdnode.ll b/llvm/test/CodeGen/RISCV/rvv/vfmul-constrained-sdnode.ll
new file mode 100644
index 0000000000000..1c8a34b4eff00
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/rvv/vfmul-constrained-sdnode.ll
@@ -0,0 +1,365 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=ilp32d \
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=riscv64 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=lp64d \
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+
+declare <vscale x 1 x half> @llvm.experimental.constrained.fmul.nxv1f16(<vscale x 1 x half>, <vscale x 1 x half>, metadata, metadata)
+define <vscale x 1 x half> @vfmul_vv_nxv1f16(<vscale x 1 x half> %va, <vscale x 1 x half> %vb) {
+; CHECK-LABEL: vfmul_vv_nxv1f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf4, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 1 x half> @llvm.experimental.constrained.fmul.nxv1f16(<vscale x 1 x half> %va, <vscale x 1 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x half> %vc
+}
+
+define <vscale x 1 x half> @vfmul_vf_nxv1f16(<vscale x 1 x half> %va, half %b) {
+; CHECK-LABEL: vfmul_vf_nxv1f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf4, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 1 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 1 x half> %head, <vscale x 1 x half> poison, <vscale x 1 x i32> zeroinitializer
+  %vc = call <vscale x 1 x half> @llvm.experimental.constrained.fmul.nxv1f16(<vscale x 1 x half> %va, <vscale x 1 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x half> %vc
+}
+
+declare <vscale x 2 x half> @llvm.experimental.constrained.fmul.nxv2f16(<vscale x 2 x half>, <vscale x 2 x half>, metadata, metadata)
+define <vscale x 2 x half> @vfmul_vv_nxv2f16(<vscale x 2 x half> %va, <vscale x 2 x half> %vb) {
+; CHECK-LABEL: vfmul_vv_nxv2f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf2, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 2 x half> @llvm.experimental.constrained.fmul.nxv2f16(<vscale x 2 x half> %va, <vscale x 2 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x half> %vc
+}
+
+define <vscale x 2 x half> @vfmul_vf_nxv2f16(<vscale x 2 x half> %va, half %b) {
+; CHECK-LABEL: vfmul_vf_nxv2f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf2, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 2 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 2 x half> %head, <vscale x 2 x half> poison, <vscale x 2 x i32> zeroinitializer
+  %vc = call <vscale x 2 x half> @llvm.experimental.constrained.fmul.nxv2f16(<vscale x 2 x half> %va, <vscale x 2 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x half> %vc
+}
+
+declare <vscale x 4 x half> @llvm.experimental.constrained.fmul.nxv4f16(<vscale x 4 x half>, <vscale x 4 x half>, metadata, metadata)
+define <vscale x 4 x half> @vfmul_vv_nxv4f16(<vscale x 4 x half> %va, <vscale x 4 x half> %vb) {
+; CHECK-LABEL: vfmul_vv_nxv4f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, m1, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 4 x half> @llvm.experimental.constrained.fmul.nxv4f16(<vscale x 4 x half> %va, <vscale x 4 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x half> %vc
+}
+
+define <vscale x 4 x half> @vfmul_vf_nxv4f16(<vscale x 4 x half> %va, half %b) {
+; CHECK-LABEL: vfmul_vf_nxv4f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m1, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 4 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 4 x half> %head, <vscale x 4 x half> poison, <vscale x 4 x i32> zeroinitializer
+  %vc = call <vscale x 4 x half> @llvm.experimental.constrained.fmul.nxv4f16(<vscale x 4 x half> %va, <vscale x 4 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x half> %vc
+}
+
+declare <vscale x 8 x half> @llvm.experimental.constrained.fmul.nxv8f16(<vscale x 8 x half>, <vscale x 8 x half>, metadata, metadata)
+define <vscale x 8 x half> @vfmul_vv_nxv8f16(<vscale x 8 x half> %va, <vscale x 8 x half> %vb) {
+; CHECK-LABEL: vfmul_vv_nxv8f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, m2, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 8 x half> @llvm.experimental.constrained.fmul.nxv8f16(<vscale x 8 x half> %va, <vscale x 8 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x half> %vc
+}
+
+define <vscale x 8 x half> @vfmul_vf_nxv8f16(<vscale x 8 x half> %va, half %b) {
+; CHECK-LABEL: vfmul_vf_nxv8f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m2, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 8 x half> %head, <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
+  %vc = call <vscale x 8 x half> @llvm.experimental.constrained.fmul.nxv8f16(<vscale x 8 x half> %va, <vscale x 8 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x half> %vc
+}
+
+declare <vscale x 16 x half> @llvm.experimental.constrained.fmul.nxv16f16(<vscale x 16 x half>, <vscale x 16 x half>, metadata, metadata)
+define <vscale x 16 x half> @vfmul_vv_nxv16f16(<vscale x 16 x half> %va, <vscale x 16 x half> %vb) {
+; CHECK-LABEL: vfmul_vv_nxv16f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, m4, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 16 x half> @llvm.experimental.constrained.fmul.nxv16f16(<vscale x 16 x half> %va, <vscale x 16 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 16 x half> %vc
+}
+
+define <vscale x 16 x half> @vfmul_vf_nxv16f16(<vscale x 16 x half> %va, half %b) {
+; CHECK-LABEL: vfmul_vf_nxv16f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m4, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 16 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 16 x half> %head, <vscale x 16 x half> poison, <vscale x 16 x i32> zeroinitializer
+  %vc = call <vscale x 16 x half> @llvm.experimental.constrained.fmul.nxv16f16(<vscale x 16 x half> %va, <vscale x 16 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 16 x half> %vc
+}
+
+declare <vscale x 32 x half> @llvm.experimental.constrained.fmul.nxv32f16(<vscale x 32 x half>, <vscale x 32 x half>, metadata, metadata)
+define <vscale x 32 x half> @vfmul_vv_nxv32f16(<vscale x 32 x half> %va, <vscale x 32 x half> %vb) {
+; CHECK-LABEL: vfmul_vv_nxv32f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, m8, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v16
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 32 x half> @llvm.experimental.constrained.fmul.nxv32f16(<vscale x 32 x half> %va, <vscale x 32 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 32 x half> %vc
+}
+
+define <vscale x 32 x half> @vfmul_vf_nxv32f16(<vscale x 32 x half> %va, half %b) {
+; CHECK-LABEL: vfmul_vf_nxv32f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m8, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 32 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 32 x half> %head, <vscale x 32 x half> poison, <vscale x 32 x i32> zeroinitializer
+  %vc = call <vscale x 32 x half> @llvm.experimental.constrained.fmul.nxv32f16(<vscale x 32 x half> %va, <vscale x 32 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 32 x half> %vc
+}
+
+declare <vscale x 1 x float> @llvm.experimental.constrained.fmul.nxv1f32(<vscale x 1 x float>, <vscale x 1 x float>, metadata, metadata)
+define <vscale x 1 x float> @vfmul_vv_nxv1f32(<vscale x 1 x float> %va, <vscale x 1 x float> %vb) {
+; CHECK-LABEL: vfmul_vv_nxv1f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, mf2, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 1 x float> @llvm.experimental.constrained.fmul.nxv1f32(<vscale x 1 x float> %va, <vscale x 1 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x float> %vc
+}
+
+define <vscale x 1 x float> @vfmul_vf_nxv1f32(<vscale x 1 x float> %va, float %b) {
+; CHECK-LABEL: vfmul_vf_nxv1f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, mf2, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 1 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 1 x float> %head, <vscale x 1 x float> poison, <vscale x 1 x i32> zeroinitializer
+  %vc = call <vscale x 1 x float> @llvm.experimental.constrained.fmul.nxv1f32(<vscale x 1 x float> %va, <vscale x 1 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x float> %vc
+}
+
+declare <vscale x 2 x float> @llvm.experimental.constrained.fmul.nxv2f32(<vscale x 2 x float>, <vscale x 2 x float>, metadata, metadata)
+define <vscale x 2 x float> @vfmul_vv_nxv2f32(<vscale x 2 x float> %va, <vscale x 2 x float> %vb) {
+; CHECK-LABEL: vfmul_vv_nxv2f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, m1, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 2 x float> @llvm.experimental.constrained.fmul.nxv2f32(<vscale x 2 x float> %va, <vscale x 2 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x float> %vc
+}
+
+define <vscale x 2 x float> @vfmul_vf_nxv2f32(<vscale x 2 x float> %va, float %b) {
+; CHECK-LABEL: vfmul_vf_nxv2f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m1, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 2 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 2 x float> %head, <vscale x 2 x float> poison, <vscale x 2 x i32> zeroinitializer
+  %vc = call <vscale x 2 x float> @llvm.experimental.constrained.fmul.nxv2f32(<vscale x 2 x float> %va, <vscale x 2 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x float> %vc
+}
+
+declare <vscale x 4 x float> @llvm.experimental.constrained.fmul.nxv4f32(<vscale x 4 x float>, <vscale x 4 x float>, metadata, metadata)
+define <vscale x 4 x float> @vfmul_vv_nxv4f32(<vscale x 4 x float> %va, <vscale x 4 x float> %vb) {
+; CHECK-LABEL: vfmul_vv_nxv4f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, m2, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 4 x float> @llvm.experimental.constrained.fmul.nxv4f32(<vscale x 4 x float> %va, <vscale x 4 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x float> %vc
+}
+
+define <vscale x 4 x float> @vfmul_vf_nxv4f32(<vscale x 4 x float> %va, float %b) {
+; CHECK-LABEL: vfmul_vf_nxv4f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m2, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 4 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 4 x float> %head, <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
+  %vc = call <vscale x 4 x float> @llvm.experimental.constrained.fmul.nxv4f32(<vscale x 4 x float> %va, <vscale x 4 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x float> %vc
+}
+
+declare <vscale x 8 x float> @llvm.experimental.constrained.fmul.nxv8f32(<vscale x 8 x float>, <vscale x 8 x float>, metadata, metadata)
+define <vscale x 8 x float> @vfmul_vv_nxv8f32(<vscale x 8 x float> %va, <vscale x 8 x float> %vb) {
+; CHECK-LABEL: vfmul_vv_nxv8f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, m4, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 8 x float> @llvm.experimental.constrained.fmul.nxv8f32(<vscale x 8 x float> %va, <vscale x 8 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x float> %vc
+}
+
+define <vscale x 8 x float> @vfmul_vf_nxv8f32(<vscale x 8 x float> %va, float %b) {
+; CHECK-LABEL: vfmul_vf_nxv8f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m4, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 8 x float> %head, <vscale x 8 x float> poison, <vscale x 8 x i32> zeroinitializer
+  %vc = call <vscale x 8 x float> @llvm.experimental.constrained.fmul.nxv8f32(<vscale x 8 x float> %va, <vscale x 8 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x float> %vc
+}
+
+declare <vscale x 16 x float> @llvm.experimental.constrained.fmul.nxv16f32(<vscale x 16 x float>, <vscale x 16 x float>, metadata, metadata)
+define <vscale x 16 x float> @vfmul_vv_nxv16f32(<vscale x 16 x float> %va, <vscale x 16 x float> %vb) {
+; CHECK-LABEL: vfmul_vv_nxv16f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, m8, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v16
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 16 x float> @llvm.experimental.constrained.fmul.nxv16f32(<vscale x 16 x float> %va, <vscale x 16 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 16 x float> %vc
+}
+
+define <vscale x 16 x float> @vfmul_vf_nxv16f32(<vscale x 16 x float> %va, float %b) {
+; CHECK-LABEL: vfmul_vf_nxv16f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m8, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 16 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 16 x float> %head, <vscale x 16 x float> poison, <vscale x 16 x i32> zeroinitializer
+  %vc = call <vscale x 16 x float> @llvm.experimental.constrained.fmul.nxv16f32(<vscale x 16 x float> %va, <vscale x 16 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 16 x float> %vc
+}
+
+declare <vscale x 1 x double> @llvm.experimental.constrained.fmul.nxv1f64(<vscale x 1 x double>, <vscale x 1 x double>, metadata, metadata)
+define <vscale x 1 x double> @vfmul_vv_nxv1f64(<vscale x 1 x double> %va, <vscale x 1 x double> %vb) {
+; CHECK-LABEL: vfmul_vv_nxv1f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e64, m1, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 1 x double> @llvm.experimental.constrained.fmul.nxv1f64(<vscale x 1 x double> %va, <vscale x 1 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x double> %vc
+}
+
+define <vscale x 1 x double> @vfmul_vf_nxv1f64(<vscale x 1 x double> %va, double %b) {
+; CHECK-LABEL: vfmul_vf_nxv1f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e64, m1, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 1 x double> poison, double %b, i32 0
+  %splat = shufflevector <vscale x 1 x double> %head, <vscale x 1 x double> poison, <vscale x 1 x i32> zeroinitializer
+  %vc = call <vscale x 1 x double> @llvm.experimental.constrained.fmul.nxv1f64(<vscale x 1 x double> %va, <vscale x 1 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x double> %vc
+}
+
+declare <vscale x 2 x double> @llvm.experimental.constrained.fmul.nxv2f64(<vscale x 2 x double>, <vscale x 2 x double>, metadata, metadata)
+define <vscale x 2 x double> @vfmul_vv_nxv2f64(<vscale x 2 x double> %va, <vscale x 2 x double> %vb) {
+; CHECK-LABEL: vfmul_vv_nxv2f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e64, m2, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 2 x double> @llvm.experimental.constrained.fmul.nxv2f64(<vscale x 2 x double> %va, <vscale x 2 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x double> %vc
+}
+
+define <vscale x 2 x double> @vfmul_vf_nxv2f64(<vscale x 2 x double> %va, double %b) {
+; CHECK-LABEL: vfmul_vf_nxv2f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e64, m2, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 2 x double> poison, double %b, i32 0
+  %splat = shufflevector <vscale x 2 x double> %head, <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
+  %vc = call <vscale x 2 x double> @llvm.experimental.constrained.fmul.nxv2f64(<vscale x 2 x double> %va, <vscale x 2 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x double> %vc
+}
+
+declare <vscale x 4 x double> @llvm.experimental.constrained.fmul.nxv4f64(<vscale x 4 x double>, <vscale x 4 x double>, metadata, metadata)
+define <vscale x 4 x double> @vfmul_vv_nxv4f64(<vscale x 4 x double> %va, <vscale x 4 x double> %vb) {
+; CHECK-LABEL: vfmul_vv_nxv4f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e64, m4, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 4 x double> @llvm.experimental.constrained.fmul.nxv4f64(<vscale x 4 x double> %va, <vscale x 4 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x double> %vc
+}
+
+define <vscale x 4 x double> @vfmul_vf_nxv4f64(<vscale x 4 x double> %va, double %b) {
+; CHECK-LABEL: vfmul_vf_nxv4f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e64, m4, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 4 x double> poison, double %b, i32 0
+  %splat = shufflevector <vscale x 4 x double> %head, <vscale x 4 x double> poison, <vscale x 4 x i32> zeroinitializer
+  %vc = call <vscale x 4 x double> @llvm.experimental.constrained.fmul.nxv4f64(<vscale x 4 x double> %va, <vscale x 4 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x double> %vc
+}
+
+declare <vscale x 8 x double> @llvm.experimental.constrained.fmul.nxv8f64(<vscale x 8 x double>, <vscale x 8 x double>, metadata, metadata)
+define <vscale x 8 x double> @vfmul_vv_nxv8f64(<vscale x 8 x double> %va, <vscale x 8 x double> %vb) {
+; CHECK-LABEL: vfmul_vv_nxv8f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e64, m8, ta, ma
+; CHECK-NEXT:    vfmul.vv v8, v8, v16
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 8 x double> @llvm.experimental.constrained.fmul.nxv8f64(<vscale x 8 x double> %va, <vscale x 8 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x double> %vc
+}
+
+define <vscale x 8 x double> @vfmul_vf_nxv8f64(<vscale x 8 x double> %va, double %b) {
+; CHECK-LABEL: vfmul_vf_nxv8f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e64, m8, ta, ma
+; CHECK-NEXT:    vfmul.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x double> poison, double %b, i32 0
+  %splat = shufflevector <vscale x 8 x double> %head, <vscale x 8 x double> poison, <vscale x 8 x i32> zeroinitializer
+  %vc = call <vscale x 8 x double> @llvm.experimental.constrained.fmul.nxv8f64(<vscale x 8 x double> %va, <vscale x 8 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x double> %vc
+}
diff --git a/llvm/test/CodeGen/RISCV/rvv/vfpext-constrained-sdnode.ll b/llvm/test/CodeGen/RISCV/rvv/vfpext-constrained-sdnode.ll
new file mode 100644
index 0000000000000..8e836b9656897
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/rvv/vfpext-constrained-sdnode.ll
@@ -0,0 +1,153 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=ilp32d \
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=riscv64 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=lp64d \
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+
+declare <vscale x 1 x float> @llvm.experimental.constrained.fpext.nxv1f32.nxv1f16(<vscale x 1 x half>, metadata)
+define <vscale x 1 x float> @vfpext_nxv1f16_nxv1f32(<vscale x 1 x half> %va) {
+; CHECK-LABEL: vfpext_nxv1f16_nxv1f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf4, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v9, v8
+; CHECK-NEXT:    vmv1r.v v8, v9
+; CHECK-NEXT:    ret
+  %evec = call <vscale x 1 x float> @llvm.experimental.constrained.fpext.nxv1f32.nxv1f16(<vscale x 1 x half> %va, metadata !"fpexcept.strict")
+  ret <vscale x 1 x float> %evec
+}
+
+declare <vscale x 1 x double> @llvm.experimental.constrained.fpext.nxv1f64.nxv1f16(<vscale x 1 x half>, metadata)
+define <vscale x 1 x double> @vfpext_nxv1f16_nxv1f64(<vscale x 1 x half> %va) {
+; CHECK-LABEL: vfpext_nxv1f16_nxv1f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf4, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v9, v8
+; CHECK-NEXT:    vsetvli zero, zero, e32, mf2, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v8, v9
+; CHECK-NEXT:    ret
+  %evec = call <vscale x 1 x double> @llvm.experimental.constrained.fpext.nxv1f64.nxv1f16(<vscale x 1 x half> %va, metadata !"fpexcept.strict")
+  ret <vscale x 1 x double> %evec
+}
+
+declare <vscale x 2 x float> @llvm.experimental.constrained.fpext.nxv2f32.nxv2f16(<vscale x 2 x half>, metadata)
+define <vscale x 2 x float> @vfpext_nxv2f16_nxv2f32(<vscale x 2 x half> %va) {
+; CHECK-LABEL: vfpext_nxv2f16_nxv2f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf2, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v9, v8
+; CHECK-NEXT:    vmv1r.v v8, v9
+; CHECK-NEXT:    ret
+  %evec = call <vscale x 2 x float> @llvm.experimental.constrained.fpext.nxv2f32.nxv2f16(<vscale x 2 x half> %va, metadata !"fpexcept.strict")
+  ret <vscale x 2 x float> %evec
+}
+
+declare <vscale x 2 x double> @llvm.experimental.constrained.fpext.nxv2f64.nxv2f16(<vscale x 2 x half>, metadata)
+define <vscale x 2 x double> @vfpext_nxv2f16_nxv2f64(<vscale x 2 x half> %va) {
+; CHECK-LABEL: vfpext_nxv2f16_nxv2f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf2, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v10, v8
+; CHECK-NEXT:    vsetvli zero, zero, e32, m1, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v8, v10
+; CHECK-NEXT:    ret
+  %evec = call <vscale x 2 x double> @llvm.experimental.constrained.fpext.nxv2f64.nxv2f16(<vscale x 2 x half> %va, metadata !"fpexcept.strict")
+  ret <vscale x 2 x double> %evec
+}
+
+declare <vscale x 4 x float> @llvm.experimental.constrained.fpext.nxv4f32.nxv4f16(<vscale x 4 x half>, metadata)
+define <vscale x 4 x float> @vfpext_nxv4f16_nxv4f32(<vscale x 4 x half> %va) {
+; CHECK-LABEL: vfpext_nxv4f16_nxv4f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m1, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v10, v8
+; CHECK-NEXT:    vmv2r.v v8, v10
+; CHECK-NEXT:    ret
+  %evec = call <vscale x 4 x float> @llvm.experimental.constrained.fpext.nxv4f32.nxv4f16(<vscale x 4 x half> %va, metadata !"fpexcept.strict")
+  ret <vscale x 4 x float> %evec
+}
+
+declare <vscale x 4 x double> @llvm.experimental.constrained.fpext.nxv4f64.nxv4f16(<vscale x 4 x half>, metadata)
+define <vscale x 4 x double> @vfpext_nxv4f16_nxv4f64(<vscale x 4 x half> %va) {
+; CHECK-LABEL: vfpext_nxv4f16_nxv4f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m1, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v12, v8
+; CHECK-NEXT:    vsetvli zero, zero, e32, m2, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v8, v12
+; CHECK-NEXT:    ret
+  %evec = call <vscale x 4 x double> @llvm.experimental.constrained.fpext.nxv4f64.nxv4f16(<vscale x 4 x half> %va, metadata !"fpexcept.strict")
+  ret <vscale x 4 x double> %evec
+}
+
+declare <vscale x 8 x float> @llvm.experimental.constrained.fpext.nxv8f32.nxv8f16(<vscale x 8 x half>, metadata)
+define <vscale x 8 x float> @vfpext_nxv8f16_nxv8f32(<vscale x 8 x half> %va) {
+; CHECK-LABEL: vfpext_nxv8f16_nxv8f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m2, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v12, v8
+; CHECK-NEXT:    vmv4r.v v8, v12
+; CHECK-NEXT:    ret
+  %evec = call <vscale x 8 x float> @llvm.experimental.constrained.fpext.nxv8f32.nxv8f16(<vscale x 8 x half> %va, metadata !"fpexcept.strict")
+  ret <vscale x 8 x float> %evec
+}
+
+declare <vscale x 8 x double> @llvm.experimental.constrained.fpext.nxv8f64.nxv8f16(<vscale x 8 x half>, metadata)
+define <vscale x 8 x double> @vfpext_nxv8f16_nxv8f64(<vscale x 8 x half> %va) {
+; CHECK-LABEL: vfpext_nxv8f16_nxv8f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m2, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v16, v8
+; CHECK-NEXT:    vsetvli zero, zero, e32, m4, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v8, v16
+; CHECK-NEXT:    ret
+  %evec = call <vscale x 8 x double> @llvm.experimental.constrained.fpext.nxv8f64.nxv8f16(<vscale x 8 x half> %va, metadata !"fpexcept.strict")
+  ret <vscale x 8 x double> %evec
+}
+
+declare <vscale x 1 x double> @llvm.experimental.constrained.fpext.nxv1f64.nxv1f32(<vscale x 1 x float>, metadata)
+define <vscale x 1 x double> @vfpext_nxv1f32_nxv1f64(<vscale x 1 x float> %va) {
+; CHECK-LABEL: vfpext_nxv1f32_nxv1f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, mf2, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v9, v8
+; CHECK-NEXT:    vmv1r.v v8, v9
+; CHECK-NEXT:    ret
+  %evec = call <vscale x 1 x double> @llvm.experimental.constrained.fpext.nxv1f64.nxv1f32(<vscale x 1 x float> %va, metadata !"fpexcept.strict")
+  ret <vscale x 1 x double> %evec
+}
+
+declare <vscale x 2 x double> @llvm.experimental.constrained.fpext.nxv2f64.nxv2f32(<vscale x 2 x float>, metadata)
+define <vscale x 2 x double> @vfpext_nxv2f32_nxv2f64(<vscale x 2 x float> %va) {
+; CHECK-LABEL: vfpext_nxv2f32_nxv2f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m1, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v10, v8
+; CHECK-NEXT:    vmv2r.v v8, v10
+; CHECK-NEXT:    ret
+  %evec = call <vscale x 2 x double> @llvm.experimental.constrained.fpext.nxv2f64.nxv2f32(<vscale x 2 x float> %va, metadata !"fpexcept.strict")
+  ret <vscale x 2 x double> %evec
+}
+
+declare <vscale x 4 x double> @llvm.experimental.constrained.fpext.nxv4f64.nxv4f32(<vscale x 4 x float>, metadata)
+define <vscale x 4 x double> @vfpext_nxv4f32_nxv4f64(<vscale x 4 x float> %va) {
+; CHECK-LABEL: vfpext_nxv4f32_nxv4f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m2, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v12, v8
+; CHECK-NEXT:    vmv4r.v v8, v12
+; CHECK-NEXT:    ret
+  %evec = call <vscale x 4 x double> @llvm.experimental.constrained.fpext.nxv4f64.nxv4f32(<vscale x 4 x float> %va, metadata !"fpexcept.strict")
+  ret <vscale x 4 x double> %evec
+}
+
+declare <vscale x 8 x double> @llvm.experimental.constrained.fpext.nxv8f64.nxv8f32(<vscale x 8 x float>, metadata)
+define <vscale x 8 x double> @vfpext_nxv8f32_nxv8f64(<vscale x 8 x float> %va) {
+; CHECK-LABEL: vfpext_nxv8f32_nxv8f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m4, ta, ma
+; CHECK-NEXT:    vfwcvt.f.f.v v16, v8
+; CHECK-NEXT:    vmv8r.v v8, v16
+; CHECK-NEXT:    ret
+  %evec = call <vscale x 8 x double> @llvm.experimental.constrained.fpext.nxv8f64.nxv8f32(<vscale x 8 x float> %va, metadata !"fpexcept.strict")
+  ret <vscale x 8 x double> %evec
+}
diff --git a/llvm/test/CodeGen/RISCV/rvv/vfptosi-vp.ll b/llvm/test/CodeGen/RISCV/rvv/vfptosi-vp.ll
index f51945a8617b2..c8859a358fcae 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vfptosi-vp.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vfptosi-vp.ll
@@ -314,15 +314,7 @@ declare <vscale x 32 x i16> @llvm.vp.fptosi.nxv32i16.nxv32f32(<vscale x 32 x flo
 define <vscale x 32 x i16> @vfptosi_nxv32i16_nxv32f32(<vscale x 32 x float> %va, <vscale x 32 x i1> %m, i32 zeroext %evl) {
 ; CHECK-LABEL: vfptosi_nxv32i16_nxv32f32:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    addi sp, sp, -16
-; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    csrr a1, vlenb
-; CHECK-NEXT:    slli a1, a1, 3
-; CHECK-NEXT:    sub sp, sp, a1
-; CHECK-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x08, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 8 * vlenb
 ; CHECK-NEXT:    vmv1r.v v24, v0
-; CHECK-NEXT:    addi a1, sp, 16
-; CHECK-NEXT:    vs8r.v v8, (a1) # Unknown-size Folded Spill
 ; CHECK-NEXT:    csrr a1, vlenb
 ; CHECK-NEXT:    srli a2, a1, 2
 ; CHECK-NEXT:    vsetvli a3, zero, e8, mf2, ta, ma
@@ -333,20 +325,15 @@ define <vscale x 32 x i16> @vfptosi_nxv32i16_nxv32f32(<vscale x 32 x float> %va,
 ; CHECK-NEXT:    addi a3, a3, -1
 ; CHECK-NEXT:    and a2, a3, a2
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m4, ta, ma
-; CHECK-NEXT:    vfncvt.rtz.x.f.w v12, v16, v0.t
+; CHECK-NEXT:    vfncvt.rtz.x.f.w v28, v16, v0.t
 ; CHECK-NEXT:    bltu a0, a1, .LBB25_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a0, a1
 ; CHECK-NEXT:  .LBB25_2:
 ; CHECK-NEXT:    vsetvli zero, a0, e16, m4, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v24
-; CHECK-NEXT:    addi a0, sp, 16
-; CHECK-NEXT:    vl8r.v v16, (a0) # Unknown-size Folded Reload
-; CHECK-NEXT:    vfncvt.rtz.x.f.w v8, v16, v0.t
-; CHECK-NEXT:    csrr a0, vlenb
-; CHECK-NEXT:    slli a0, a0, 3
-; CHECK-NEXT:    add sp, sp, a0
-; CHECK-NEXT:    addi sp, sp, 16
+; CHECK-NEXT:    vfncvt.rtz.x.f.w v24, v8, v0.t
+; CHECK-NEXT:    vmv8r.v v8, v24
 ; CHECK-NEXT:    ret
   %v = call <vscale x 32 x i16> @llvm.vp.fptosi.nxv32i16.nxv32f32(<vscale x 32 x float> %va, <vscale x 32 x i1> %m, i32 %evl)
   ret <vscale x 32 x i16> %v
diff --git a/llvm/test/CodeGen/RISCV/rvv/vfptoui-vp.ll b/llvm/test/CodeGen/RISCV/rvv/vfptoui-vp.ll
index cf8ace03d576e..9ef1dd3634e9f 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vfptoui-vp.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vfptoui-vp.ll
@@ -314,15 +314,7 @@ declare <vscale x 32 x i16> @llvm.vp.fptoui.nxv32i16.nxv32f32(<vscale x 32 x flo
 define <vscale x 32 x i16> @vfptoui_nxv32i16_nxv32f32(<vscale x 32 x float> %va, <vscale x 32 x i1> %m, i32 zeroext %evl) {
 ; CHECK-LABEL: vfptoui_nxv32i16_nxv32f32:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    addi sp, sp, -16
-; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    csrr a1, vlenb
-; CHECK-NEXT:    slli a1, a1, 3
-; CHECK-NEXT:    sub sp, sp, a1
-; CHECK-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x08, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 8 * vlenb
 ; CHECK-NEXT:    vmv1r.v v24, v0
-; CHECK-NEXT:    addi a1, sp, 16
-; CHECK-NEXT:    vs8r.v v8, (a1) # Unknown-size Folded Spill
 ; CHECK-NEXT:    csrr a1, vlenb
 ; CHECK-NEXT:    srli a2, a1, 2
 ; CHECK-NEXT:    vsetvli a3, zero, e8, mf2, ta, ma
@@ -333,20 +325,15 @@ define <vscale x 32 x i16> @vfptoui_nxv32i16_nxv32f32(<vscale x 32 x float> %va,
 ; CHECK-NEXT:    addi a3, a3, -1
 ; CHECK-NEXT:    and a2, a3, a2
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m4, ta, ma
-; CHECK-NEXT:    vfncvt.rtz.xu.f.w v12, v16, v0.t
+; CHECK-NEXT:    vfncvt.rtz.xu.f.w v28, v16, v0.t
 ; CHECK-NEXT:    bltu a0, a1, .LBB25_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a0, a1
 ; CHECK-NEXT:  .LBB25_2:
 ; CHECK-NEXT:    vsetvli zero, a0, e16, m4, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v24
-; CHECK-NEXT:    addi a0, sp, 16
-; CHECK-NEXT:    vl8r.v v16, (a0) # Unknown-size Folded Reload
-; CHECK-NEXT:    vfncvt.rtz.xu.f.w v8, v16, v0.t
-; CHECK-NEXT:    csrr a0, vlenb
-; CHECK-NEXT:    slli a0, a0, 3
-; CHECK-NEXT:    add sp, sp, a0
-; CHECK-NEXT:    addi sp, sp, 16
+; CHECK-NEXT:    vfncvt.rtz.xu.f.w v24, v8, v0.t
+; CHECK-NEXT:    vmv8r.v v8, v24
 ; CHECK-NEXT:    ret
   %v = call <vscale x 32 x i16> @llvm.vp.fptoui.nxv32i16.nxv32f32(<vscale x 32 x float> %va, <vscale x 32 x i1> %m, i32 %evl)
   ret <vscale x 32 x i16> %v
diff --git a/llvm/test/CodeGen/RISCV/rvv/vfptrunc-vp.ll b/llvm/test/CodeGen/RISCV/rvv/vfptrunc-vp.ll
index ec2eeaa8327b6..86b9a6c694ea2 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vfptrunc-vp.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vfptrunc-vp.ll
@@ -140,29 +140,25 @@ define <vscale x 32 x float> @vfptrunc_nxv32f32_nxv32f64(<vscale x 32 x double>
 ; CHECK-NEXT:    addi sp, sp, -16
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    csrr a1, vlenb
-; CHECK-NEXT:    li a3, 24
-; CHECK-NEXT:    mul a1, a1, a3
+; CHECK-NEXT:    slli a1, a1, 4
 ; CHECK-NEXT:    sub sp, sp, a1
-; CHECK-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x18, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 24 * vlenb
+; CHECK-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x10, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 16 * vlenb
 ; CHECK-NEXT:    vmv1r.v v1, v0
-; CHECK-NEXT:    csrr a1, vlenb
-; CHECK-NEXT:    slli a1, a1, 3
-; CHECK-NEXT:    add a1, sp, a1
-; CHECK-NEXT:    addi a1, a1, 16
+; CHECK-NEXT:    addi a1, sp, 16
 ; CHECK-NEXT:    vs8r.v v16, (a1) # Unknown-size Folded Spill
 ; CHECK-NEXT:    csrr a1, vlenb
-; CHECK-NEXT:    slli a1, a1, 4
+; CHECK-NEXT:    slli a1, a1, 3
 ; CHECK-NEXT:    add a1, sp, a1
 ; CHECK-NEXT:    addi a1, a1, 16
 ; CHECK-NEXT:    vs8r.v v8, (a1) # Unknown-size Folded Spill
 ; CHECK-NEXT:    csrr a1, vlenb
 ; CHECK-NEXT:    srli a3, a1, 2
 ; CHECK-NEXT:    vsetvli a4, zero, e8, mf2, ta, ma
-; CHECK-NEXT:    vslidedown.vx v25, v0, a3
+; CHECK-NEXT:    vslidedown.vx v16, v0, a3
 ; CHECK-NEXT:    vsetvli a3, zero, e8, mf4, ta, ma
 ; CHECK-NEXT:    slli a3, a1, 3
 ; CHECK-NEXT:    add a3, a0, a3
-; CHECK-NEXT:    vl8re64.v v8, (a3)
+; CHECK-NEXT:    vl8re64.v v24, (a3)
 ; CHECK-NEXT:    slli a3, a1, 1
 ; CHECK-NEXT:    sub a4, a2, a3
 ; CHECK-NEXT:    sltu a5, a2, a4
@@ -173,22 +169,18 @@ define <vscale x 32 x float> @vfptrunc_nxv32f32_nxv32f64(<vscale x 32 x double>
 ; CHECK-NEXT:    addi a6, a6, -1
 ; CHECK-NEXT:    and a6, a6, a5
 ; CHECK-NEXT:    srli a5, a1, 3
-; CHECK-NEXT:    vl8re64.v v16, (a0)
-; CHECK-NEXT:    addi a0, sp, 16
-; CHECK-NEXT:    vs8r.v v16, (a0) # Unknown-size Folded Spill
-; CHECK-NEXT:    vslidedown.vx v0, v25, a5
+; CHECK-NEXT:    vl8re64.v v8, (a0)
+; CHECK-NEXT:    vslidedown.vx v0, v16, a5
 ; CHECK-NEXT:    vsetvli zero, a6, e32, m4, ta, ma
-; CHECK-NEXT:    vfncvt.f.f.w v20, v8, v0.t
+; CHECK-NEXT:    vfncvt.f.f.w v20, v24, v0.t
 ; CHECK-NEXT:    bltu a4, a1, .LBB8_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a4, a1
 ; CHECK-NEXT:  .LBB8_2:
 ; CHECK-NEXT:    vsetvli a0, zero, e8, mf4, ta, ma
-; CHECK-NEXT:    vslidedown.vx v26, v1, a5
+; CHECK-NEXT:    vslidedown.vx v2, v1, a5
 ; CHECK-NEXT:    vsetvli zero, a4, e32, m4, ta, ma
-; CHECK-NEXT:    vmv1r.v v0, v25
-; CHECK-NEXT:    addi a0, sp, 16
-; CHECK-NEXT:    vl8r.v v8, (a0) # Unknown-size Folded Reload
+; CHECK-NEXT:    vmv1r.v v0, v16
 ; CHECK-NEXT:    vfncvt.f.f.w v16, v8, v0.t
 ; CHECK-NEXT:    bltu a2, a3, .LBB8_4
 ; CHECK-NEXT:  # %bb.3:
@@ -199,13 +191,10 @@ define <vscale x 32 x float> @vfptrunc_nxv32f32_nxv32f64(<vscale x 32 x double>
 ; CHECK-NEXT:    addi a3, a3, -1
 ; CHECK-NEXT:    and a0, a3, a0
 ; CHECK-NEXT:    vsetvli zero, a0, e32, m4, ta, ma
-; CHECK-NEXT:    vmv1r.v v0, v26
-; CHECK-NEXT:    csrr a0, vlenb
-; CHECK-NEXT:    slli a0, a0, 3
-; CHECK-NEXT:    add a0, sp, a0
-; CHECK-NEXT:    addi a0, a0, 16
-; CHECK-NEXT:    vl8r.v v24, (a0) # Unknown-size Folded Reload
-; CHECK-NEXT:    vfncvt.f.f.w v12, v24, v0.t
+; CHECK-NEXT:    vmv1r.v v0, v2
+; CHECK-NEXT:    addi a0, sp, 16
+; CHECK-NEXT:    vl8r.v v8, (a0) # Unknown-size Folded Reload
+; CHECK-NEXT:    vfncvt.f.f.w v28, v8, v0.t
 ; CHECK-NEXT:    bltu a2, a1, .LBB8_6
 ; CHECK-NEXT:  # %bb.5:
 ; CHECK-NEXT:    mv a2, a1
@@ -213,14 +202,14 @@ define <vscale x 32 x float> @vfptrunc_nxv32f32_nxv32f64(<vscale x 32 x double>
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m4, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1
 ; CHECK-NEXT:    csrr a0, vlenb
-; CHECK-NEXT:    slli a0, a0, 4
+; CHECK-NEXT:    slli a0, a0, 3
 ; CHECK-NEXT:    add a0, sp, a0
 ; CHECK-NEXT:    addi a0, a0, 16
-; CHECK-NEXT:    vl8r.v v24, (a0) # Unknown-size Folded Reload
-; CHECK-NEXT:    vfncvt.f.f.w v8, v24, v0.t
+; CHECK-NEXT:    vl8r.v v8, (a0) # Unknown-size Folded Reload
+; CHECK-NEXT:    vfncvt.f.f.w v24, v8, v0.t
+; CHECK-NEXT:    vmv8r.v v8, v24
 ; CHECK-NEXT:    csrr a0, vlenb
-; CHECK-NEXT:    li a1, 24
-; CHECK-NEXT:    mul a0, a0, a1
+; CHECK-NEXT:    slli a0, a0, 4
 ; CHECK-NEXT:    add sp, sp, a0
 ; CHECK-NEXT:    addi sp, sp, 16
 ; CHECK-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/rvv/vfsub-constrained-sdnode.ll b/llvm/test/CodeGen/RISCV/rvv/vfsub-constrained-sdnode.ll
new file mode 100644
index 0000000000000..f4e24a7b9442c
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/rvv/vfsub-constrained-sdnode.ll
@@ -0,0 +1,401 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=ilp32d \
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=riscv64 -mattr=+d,+zfh,+experimental-zvfh,+v -target-abi=lp64d \
+; RUN:     -verify-machineinstrs < %s | FileCheck %s
+
+declare <vscale x 1 x half> @llvm.experimental.constrained.fsub.nxv1f16(<vscale x 1 x half>, <vscale x 1 x half>, metadata, metadata)
+define <vscale x 1 x half> @vfsub_vv_nxv1f16(<vscale x 1 x half> %va, <vscale x 1 x half> %vb) {
+; CHECK-LABEL: vfsub_vv_nxv1f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf4, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 1 x half> @llvm.experimental.constrained.fsub.nxv1f16(<vscale x 1 x half> %va, <vscale x 1 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x half> %vc
+}
+
+define <vscale x 1 x half> @vfsub_vf_nxv1f16(<vscale x 1 x half> %va, half %b) {
+; CHECK-LABEL: vfsub_vf_nxv1f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf4, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 1 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 1 x half> %head, <vscale x 1 x half> poison, <vscale x 1 x i32> zeroinitializer
+  %vc = call <vscale x 1 x half> @llvm.experimental.constrained.fsub.nxv1f16(<vscale x 1 x half> %va, <vscale x 1 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x half> %vc
+}
+
+declare <vscale x 2 x half> @llvm.experimental.constrained.fsub.nxv2f16(<vscale x 2 x half>, <vscale x 2 x half>, metadata, metadata)
+define <vscale x 2 x half> @vfsub_vv_nxv2f16(<vscale x 2 x half> %va, <vscale x 2 x half> %vb) {
+; CHECK-LABEL: vfsub_vv_nxv2f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf2, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 2 x half> @llvm.experimental.constrained.fsub.nxv2f16(<vscale x 2 x half> %va, <vscale x 2 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x half> %vc
+}
+
+define <vscale x 2 x half> @vfsub_vf_nxv2f16(<vscale x 2 x half> %va, half %b) {
+; CHECK-LABEL: vfsub_vf_nxv2f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, mf2, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 2 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 2 x half> %head, <vscale x 2 x half> poison, <vscale x 2 x i32> zeroinitializer
+  %vc = call <vscale x 2 x half> @llvm.experimental.constrained.fsub.nxv2f16(<vscale x 2 x half> %va, <vscale x 2 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x half> %vc
+}
+
+declare <vscale x 4 x half> @llvm.experimental.constrained.fsub.nxv4f16(<vscale x 4 x half>, <vscale x 4 x half>, metadata, metadata)
+define <vscale x 4 x half> @vfsub_vv_nxv4f16(<vscale x 4 x half> %va, <vscale x 4 x half> %vb) {
+; CHECK-LABEL: vfsub_vv_nxv4f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, m1, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 4 x half> @llvm.experimental.constrained.fsub.nxv4f16(<vscale x 4 x half> %va, <vscale x 4 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x half> %vc
+}
+
+define <vscale x 4 x half> @vfsub_vf_nxv4f16(<vscale x 4 x half> %va, half %b) {
+; CHECK-LABEL: vfsub_vf_nxv4f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m1, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 4 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 4 x half> %head, <vscale x 4 x half> poison, <vscale x 4 x i32> zeroinitializer
+  %vc = call <vscale x 4 x half> @llvm.experimental.constrained.fsub.nxv4f16(<vscale x 4 x half> %va, <vscale x 4 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x half> %vc
+}
+
+declare <vscale x 8 x half> @llvm.experimental.constrained.fsub.nxv8f16(<vscale x 8 x half>, <vscale x 8 x half>, metadata, metadata)
+define <vscale x 8 x half> @vfsub_vv_nxv8f16(<vscale x 8 x half> %va, <vscale x 8 x half> %vb) {
+; CHECK-LABEL: vfsub_vv_nxv8f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, m2, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 8 x half> @llvm.experimental.constrained.fsub.nxv8f16(<vscale x 8 x half> %va, <vscale x 8 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x half> %vc
+}
+
+define <vscale x 8 x half> @vfsub_vf_nxv8f16(<vscale x 8 x half> %va, half %b) {
+; CHECK-LABEL: vfsub_vf_nxv8f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m2, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 8 x half> %head, <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
+  %vc = call <vscale x 8 x half> @llvm.experimental.constrained.fsub.nxv8f16(<vscale x 8 x half> %va, <vscale x 8 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x half> %vc
+}
+
+define <vscale x 8 x half> @vfsub_fv_nxv8f16(<vscale x 8 x half> %va, half %b) {
+; CHECK-LABEL: vfsub_fv_nxv8f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m2, ta, ma
+; CHECK-NEXT:    vfrsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 8 x half> %head, <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
+  %vc = call <vscale x 8 x half> @llvm.experimental.constrained.fsub.nxv8f16(<vscale x 8 x half> %splat, <vscale x 8 x half> %va, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x half> %vc
+}
+
+declare <vscale x 16 x half> @llvm.experimental.constrained.fsub.nxv16f16(<vscale x 16 x half>, <vscale x 16 x half>, metadata, metadata)
+define <vscale x 16 x half> @vfsub_vv_nxv16f16(<vscale x 16 x half> %va, <vscale x 16 x half> %vb) {
+; CHECK-LABEL: vfsub_vv_nxv16f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, m4, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 16 x half> @llvm.experimental.constrained.fsub.nxv16f16(<vscale x 16 x half> %va, <vscale x 16 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 16 x half> %vc
+}
+
+define <vscale x 16 x half> @vfsub_vf_nxv16f16(<vscale x 16 x half> %va, half %b) {
+; CHECK-LABEL: vfsub_vf_nxv16f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m4, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 16 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 16 x half> %head, <vscale x 16 x half> poison, <vscale x 16 x i32> zeroinitializer
+  %vc = call <vscale x 16 x half> @llvm.experimental.constrained.fsub.nxv16f16(<vscale x 16 x half> %va, <vscale x 16 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 16 x half> %vc
+}
+
+declare <vscale x 32 x half> @llvm.experimental.constrained.fsub.nxv32f16(<vscale x 32 x half>, <vscale x 32 x half>, metadata, metadata)
+define <vscale x 32 x half> @vfsub_vv_nxv32f16(<vscale x 32 x half> %va, <vscale x 32 x half> %vb) {
+; CHECK-LABEL: vfsub_vv_nxv32f16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e16, m8, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v16
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 32 x half> @llvm.experimental.constrained.fsub.nxv32f16(<vscale x 32 x half> %va, <vscale x 32 x half> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 32 x half> %vc
+}
+
+define <vscale x 32 x half> @vfsub_vf_nxv32f16(<vscale x 32 x half> %va, half %b) {
+; CHECK-LABEL: vfsub_vf_nxv32f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e16, m8, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 32 x half> poison, half %b, i32 0
+  %splat = shufflevector <vscale x 32 x half> %head, <vscale x 32 x half> poison, <vscale x 32 x i32> zeroinitializer
+  %vc = call <vscale x 32 x half> @llvm.experimental.constrained.fsub.nxv32f16(<vscale x 32 x half> %va, <vscale x 32 x half> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 32 x half> %vc
+}
+
+declare <vscale x 1 x float> @llvm.experimental.constrained.fsub.nxv1f32(<vscale x 1 x float>, <vscale x 1 x float>, metadata, metadata)
+define <vscale x 1 x float> @vfsub_vv_nxv1f32(<vscale x 1 x float> %va, <vscale x 1 x float> %vb) {
+; CHECK-LABEL: vfsub_vv_nxv1f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, mf2, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 1 x float> @llvm.experimental.constrained.fsub.nxv1f32(<vscale x 1 x float> %va, <vscale x 1 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x float> %vc
+}
+
+define <vscale x 1 x float> @vfsub_vf_nxv1f32(<vscale x 1 x float> %va, float %b) {
+; CHECK-LABEL: vfsub_vf_nxv1f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, mf2, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 1 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 1 x float> %head, <vscale x 1 x float> poison, <vscale x 1 x i32> zeroinitializer
+  %vc = call <vscale x 1 x float> @llvm.experimental.constrained.fsub.nxv1f32(<vscale x 1 x float> %va, <vscale x 1 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x float> %vc
+}
+
+declare <vscale x 2 x float> @llvm.experimental.constrained.fsub.nxv2f32(<vscale x 2 x float>, <vscale x 2 x float>, metadata, metadata)
+define <vscale x 2 x float> @vfsub_vv_nxv2f32(<vscale x 2 x float> %va, <vscale x 2 x float> %vb) {
+; CHECK-LABEL: vfsub_vv_nxv2f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, m1, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 2 x float> @llvm.experimental.constrained.fsub.nxv2f32(<vscale x 2 x float> %va, <vscale x 2 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x float> %vc
+}
+
+define <vscale x 2 x float> @vfsub_vf_nxv2f32(<vscale x 2 x float> %va, float %b) {
+; CHECK-LABEL: vfsub_vf_nxv2f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m1, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 2 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 2 x float> %head, <vscale x 2 x float> poison, <vscale x 2 x i32> zeroinitializer
+  %vc = call <vscale x 2 x float> @llvm.experimental.constrained.fsub.nxv2f32(<vscale x 2 x float> %va, <vscale x 2 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x float> %vc
+}
+
+declare <vscale x 4 x float> @llvm.experimental.constrained.fsub.nxv4f32(<vscale x 4 x float>, <vscale x 4 x float>, metadata, metadata)
+define <vscale x 4 x float> @vfsub_vv_nxv4f32(<vscale x 4 x float> %va, <vscale x 4 x float> %vb) {
+; CHECK-LABEL: vfsub_vv_nxv4f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, m2, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 4 x float> @llvm.experimental.constrained.fsub.nxv4f32(<vscale x 4 x float> %va, <vscale x 4 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x float> %vc
+}
+
+define <vscale x 4 x float> @vfsub_vf_nxv4f32(<vscale x 4 x float> %va, float %b) {
+; CHECK-LABEL: vfsub_vf_nxv4f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m2, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 4 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 4 x float> %head, <vscale x 4 x float> poison, <vscale x 4 x i32> zeroinitializer
+  %vc = call <vscale x 4 x float> @llvm.experimental.constrained.fsub.nxv4f32(<vscale x 4 x float> %va, <vscale x 4 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x float> %vc
+}
+
+declare <vscale x 8 x float> @llvm.experimental.constrained.fsub.nxv8f32(<vscale x 8 x float>, <vscale x 8 x float>, metadata, metadata)
+define <vscale x 8 x float> @vfsub_vv_nxv8f32(<vscale x 8 x float> %va, <vscale x 8 x float> %vb) {
+; CHECK-LABEL: vfsub_vv_nxv8f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, m4, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 8 x float> @llvm.experimental.constrained.fsub.nxv8f32(<vscale x 8 x float> %va, <vscale x 8 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x float> %vc
+}
+
+define <vscale x 8 x float> @vfsub_vf_nxv8f32(<vscale x 8 x float> %va, float %b) {
+; CHECK-LABEL: vfsub_vf_nxv8f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m4, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 8 x float> %head, <vscale x 8 x float> poison, <vscale x 8 x i32> zeroinitializer
+  %vc = call <vscale x 8 x float> @llvm.experimental.constrained.fsub.nxv8f32(<vscale x 8 x float> %va, <vscale x 8 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x float> %vc
+}
+
+define <vscale x 8 x float> @vfsub_fv_nxv8f32(<vscale x 8 x float> %va, float %b) {
+; CHECK-LABEL: vfsub_fv_nxv8f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m4, ta, ma
+; CHECK-NEXT:    vfrsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 8 x float> %head, <vscale x 8 x float> poison, <vscale x 8 x i32> zeroinitializer
+  %vc = call <vscale x 8 x float> @llvm.experimental.constrained.fsub.nxv8f32(<vscale x 8 x float> %splat, <vscale x 8 x float> %va, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x float> %vc
+}
+
+declare <vscale x 16 x float> @llvm.experimental.constrained.fsub.nxv16f32(<vscale x 16 x float>, <vscale x 16 x float>, metadata, metadata)
+define <vscale x 16 x float> @vfsub_vv_nxv16f32(<vscale x 16 x float> %va, <vscale x 16 x float> %vb) {
+; CHECK-LABEL: vfsub_vv_nxv16f32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e32, m8, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v16
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 16 x float> @llvm.experimental.constrained.fsub.nxv16f32(<vscale x 16 x float> %va, <vscale x 16 x float> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 16 x float> %vc
+}
+
+define <vscale x 16 x float> @vfsub_vf_nxv16f32(<vscale x 16 x float> %va, float %b) {
+; CHECK-LABEL: vfsub_vf_nxv16f32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m8, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 16 x float> poison, float %b, i32 0
+  %splat = shufflevector <vscale x 16 x float> %head, <vscale x 16 x float> poison, <vscale x 16 x i32> zeroinitializer
+  %vc = call <vscale x 16 x float> @llvm.experimental.constrained.fsub.nxv16f32(<vscale x 16 x float> %va, <vscale x 16 x float> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 16 x float> %vc
+}
+
+declare <vscale x 1 x double> @llvm.experimental.constrained.fsub.nxv1f64(<vscale x 1 x double>, <vscale x 1 x double>, metadata, metadata)
+define <vscale x 1 x double> @vfsub_vv_nxv1f64(<vscale x 1 x double> %va, <vscale x 1 x double> %vb) {
+; CHECK-LABEL: vfsub_vv_nxv1f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e64, m1, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v9
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 1 x double> @llvm.experimental.constrained.fsub.nxv1f64(<vscale x 1 x double> %va, <vscale x 1 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x double> %vc
+}
+
+define <vscale x 1 x double> @vfsub_vf_nxv1f64(<vscale x 1 x double> %va, double %b) {
+; CHECK-LABEL: vfsub_vf_nxv1f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e64, m1, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 1 x double> poison, double %b, i32 0
+  %splat = shufflevector <vscale x 1 x double> %head, <vscale x 1 x double> poison, <vscale x 1 x i32> zeroinitializer
+  %vc = call <vscale x 1 x double> @llvm.experimental.constrained.fsub.nxv1f64(<vscale x 1 x double> %va, <vscale x 1 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 1 x double> %vc
+}
+
+declare <vscale x 2 x double> @llvm.experimental.constrained.fsub.nxv2f64(<vscale x 2 x double>, <vscale x 2 x double>, metadata, metadata)
+define <vscale x 2 x double> @vfsub_vv_nxv2f64(<vscale x 2 x double> %va, <vscale x 2 x double> %vb) {
+; CHECK-LABEL: vfsub_vv_nxv2f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e64, m2, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v10
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 2 x double> @llvm.experimental.constrained.fsub.nxv2f64(<vscale x 2 x double> %va, <vscale x 2 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x double> %vc
+}
+
+define <vscale x 2 x double> @vfsub_vf_nxv2f64(<vscale x 2 x double> %va, double %b) {
+; CHECK-LABEL: vfsub_vf_nxv2f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e64, m2, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 2 x double> poison, double %b, i32 0
+  %splat = shufflevector <vscale x 2 x double> %head, <vscale x 2 x double> poison, <vscale x 2 x i32> zeroinitializer
+  %vc = call <vscale x 2 x double> @llvm.experimental.constrained.fsub.nxv2f64(<vscale x 2 x double> %va, <vscale x 2 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 2 x double> %vc
+}
+
+declare <vscale x 4 x double> @llvm.experimental.constrained.fsub.nxv4f64(<vscale x 4 x double>, <vscale x 4 x double>, metadata, metadata)
+define <vscale x 4 x double> @vfsub_vv_nxv4f64(<vscale x 4 x double> %va, <vscale x 4 x double> %vb) {
+; CHECK-LABEL: vfsub_vv_nxv4f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e64, m4, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v12
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 4 x double> @llvm.experimental.constrained.fsub.nxv4f64(<vscale x 4 x double> %va, <vscale x 4 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x double> %vc
+}
+
+define <vscale x 4 x double> @vfsub_vf_nxv4f64(<vscale x 4 x double> %va, double %b) {
+; CHECK-LABEL: vfsub_vf_nxv4f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e64, m4, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 4 x double> poison, double %b, i32 0
+  %splat = shufflevector <vscale x 4 x double> %head, <vscale x 4 x double> poison, <vscale x 4 x i32> zeroinitializer
+  %vc = call <vscale x 4 x double> @llvm.experimental.constrained.fsub.nxv4f64(<vscale x 4 x double> %va, <vscale x 4 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 4 x double> %vc
+}
+
+declare <vscale x 8 x double> @llvm.experimental.constrained.fsub.nxv8f64(<vscale x 8 x double>, <vscale x 8 x double>, metadata, metadata)
+define <vscale x 8 x double> @vfsub_vv_nxv8f64(<vscale x 8 x double> %va, <vscale x 8 x double> %vb) {
+; CHECK-LABEL: vfsub_vv_nxv8f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vsetvli a0, zero, e64, m8, ta, ma
+; CHECK-NEXT:    vfsub.vv v8, v8, v16
+; CHECK-NEXT:    ret
+entry:
+  %vc = call <vscale x 8 x double> @llvm.experimental.constrained.fsub.nxv8f64(<vscale x 8 x double> %va, <vscale x 8 x double> %vb, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x double> %vc
+}
+
+define <vscale x 8 x double> @vfsub_vf_nxv8f64(<vscale x 8 x double> %va, double %b) {
+; CHECK-LABEL: vfsub_vf_nxv8f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e64, m8, ta, ma
+; CHECK-NEXT:    vfsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x double> poison, double %b, i32 0
+  %splat = shufflevector <vscale x 8 x double> %head, <vscale x 8 x double> poison, <vscale x 8 x i32> zeroinitializer
+  %vc = call <vscale x 8 x double> @llvm.experimental.constrained.fsub.nxv8f64(<vscale x 8 x double> %va, <vscale x 8 x double> %splat, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x double> %vc
+}
+
+define <vscale x 8 x double> @vfsub_fv_nxv8f64(<vscale x 8 x double> %va, double %b) {
+; CHECK-LABEL: vfsub_fv_nxv8f64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e64, m8, ta, ma
+; CHECK-NEXT:    vfrsub.vf v8, v8, fa0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x double> poison, double %b, i32 0
+  %splat = shufflevector <vscale x 8 x double> %head, <vscale x 8 x double> poison, <vscale x 8 x i32> zeroinitializer
+  %vc = call <vscale x 8 x double> @llvm.experimental.constrained.fsub.nxv8f64(<vscale x 8 x double> %splat, <vscale x 8 x double> %va, metadata !"round.dynamic", metadata !"fpexcept.ignore")
+  ret <vscale x 8 x double> %vc
+}
diff --git a/llvm/test/CodeGen/RISCV/rvv/vloxseg-rv32.ll b/llvm/test/CodeGen/RISCV/rvv/vloxseg-rv32.ll
index 8a5ddcf7eef71..ec4d954ed5fc2 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vloxseg-rv32.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vloxseg-rv32.ll
@@ -24,7 +24,6 @@ define <vscale x 16 x i16> @test_vloxseg2_mask_nxv16i16_nxv16i16(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vloxseg2.mask.nxv16i16.nxv16i16(<vscale x 16 x i16> %val,<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -54,7 +53,6 @@ define <vscale x 16 x i16> @test_vloxseg2_mask_nxv16i16_nxv16i8(<vscale x 16 x i
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vloxseg2.mask.nxv16i16.nxv16i8(<vscale x 16 x i16> %val,<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -84,7 +82,6 @@ define <vscale x 16 x i16> @test_vloxseg2_mask_nxv16i16_nxv16i32(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v4, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vloxseg2.mask.nxv16i16.nxv16i32(<vscale x 16 x i16> %val,<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -114,7 +111,6 @@ define <vscale x 1 x i8> @test_vloxseg2_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vloxseg2.mask.nxv1i8.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -144,7 +140,6 @@ define <vscale x 1 x i8> @test_vloxseg2_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vloxseg2.mask.nxv1i8.nxv1i32(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -174,7 +169,6 @@ define <vscale x 1 x i8> @test_vloxseg2_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vloxseg2.mask.nxv1i8.nxv1i16(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -201,12 +195,11 @@ entry:
 define <vscale x 1 x i8> @test_vloxseg3_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vloxseg3.mask.nxv1i8.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -233,12 +226,11 @@ entry:
 define <vscale x 1 x i8> @test_vloxseg3_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vloxseg3.mask.nxv1i8.nxv1i32(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -265,12 +257,11 @@ entry:
 define <vscale x 1 x i8> @test_vloxseg3_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vloxseg3.mask.nxv1i8.nxv1i16(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -298,9 +289,9 @@ define <vscale x 1 x i8> @test_vloxseg4_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -331,9 +322,9 @@ define <vscale x 1 x i8> @test_vloxseg4_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -364,9 +355,9 @@ define <vscale x 1 x i8> @test_vloxseg4_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -397,10 +388,10 @@ define <vscale x 1 x i8> @test_vloxseg5_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -431,10 +422,10 @@ define <vscale x 1 x i8> @test_vloxseg5_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -465,10 +456,10 @@ define <vscale x 1 x i8> @test_vloxseg5_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -499,11 +490,11 @@ define <vscale x 1 x i8> @test_vloxseg6_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -534,11 +525,11 @@ define <vscale x 1 x i8> @test_vloxseg6_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -569,11 +560,11 @@ define <vscale x 1 x i8> @test_vloxseg6_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -604,12 +595,12 @@ define <vscale x 1 x i8> @test_vloxseg7_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -640,12 +631,12 @@ define <vscale x 1 x i8> @test_vloxseg7_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -676,12 +667,12 @@ define <vscale x 1 x i8> @test_vloxseg7_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -712,13 +703,13 @@ define <vscale x 1 x i8> @test_vloxseg8_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -749,13 +740,13 @@ define <vscale x 1 x i8> @test_vloxseg8_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -786,13 +777,13 @@ define <vscale x 1 x i8> @test_vloxseg8_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -825,7 +816,6 @@ define <vscale x 16 x i8> @test_vloxseg2_mask_nxv16i8_nxv16i16(<vscale x 16 x i8
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vloxseg2.mask.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -855,7 +845,6 @@ define <vscale x 16 x i8> @test_vloxseg2_mask_nxv16i8_nxv16i8(<vscale x 16 x i8>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vloxseg2.mask.nxv16i8.nxv16i8(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -885,7 +874,6 @@ define <vscale x 16 x i8> @test_vloxseg2_mask_nxv16i8_nxv16i32(<vscale x 16 x i8
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vloxseg2.mask.nxv16i8.nxv16i32(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -913,10 +901,9 @@ define <vscale x 16 x i8> @test_vloxseg3_mask_nxv16i8_nxv16i16(<vscale x 16 x i8
 ; CHECK-LABEL: test_vloxseg3_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vloxseg3ei16.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vloxseg3.mask.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -943,12 +930,11 @@ entry:
 define <vscale x 16 x i8> @test_vloxseg3_mask_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, i32 %vl, <vscale x 16 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vloxseg3.mask.nxv16i8.nxv16i8(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -976,10 +962,9 @@ define <vscale x 16 x i8> @test_vloxseg3_mask_nxv16i8_nxv16i32(<vscale x 16 x i8
 ; CHECK-LABEL: test_vloxseg3_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vloxseg3ei32.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vloxseg3.mask.nxv16i8.nxv16i32(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -1006,13 +991,12 @@ entry:
 define <vscale x 16 x i8> @test_vloxseg4_mask_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, i32 %vl, <vscale x 16 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
-; CHECK-NEXT:    vloxseg4ei16.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v18
+; CHECK-NEXT:    vloxseg4ei16.v v6, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vloxseg4.mask.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -1040,9 +1024,9 @@ define <vscale x 16 x i8> @test_vloxseg4_mask_nxv16i8_nxv16i8(<vscale x 16 x i8>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -1073,11 +1057,10 @@ define <vscale x 16 x i8> @test_vloxseg4_mask_nxv16i8_nxv16i32(<vscale x 16 x i8
 ; CHECK-LABEL: test_vloxseg4_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vloxseg4.mask.nxv16i8.nxv16i32(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -1107,7 +1090,6 @@ define <vscale x 2 x i32> @test_vloxseg2_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vloxseg2.mask.nxv2i32.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -1137,7 +1119,6 @@ define <vscale x 2 x i32> @test_vloxseg2_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vloxseg2.mask.nxv2i32.nxv2i8(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -1167,7 +1148,6 @@ define <vscale x 2 x i32> @test_vloxseg2_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vloxseg2.mask.nxv2i32.nxv2i16(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -1194,12 +1174,11 @@ entry:
 define <vscale x 2 x i32> @test_vloxseg3_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vloxseg3.mask.nxv2i32.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -1226,12 +1205,11 @@ entry:
 define <vscale x 2 x i32> @test_vloxseg3_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vloxseg3.mask.nxv2i32.nxv2i8(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -1258,12 +1236,11 @@ entry:
 define <vscale x 2 x i32> @test_vloxseg3_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vloxseg3.mask.nxv2i32.nxv2i16(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -1291,9 +1268,9 @@ define <vscale x 2 x i32> @test_vloxseg4_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1324,9 +1301,9 @@ define <vscale x 2 x i32> @test_vloxseg4_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1357,9 +1334,9 @@ define <vscale x 2 x i32> @test_vloxseg4_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1390,10 +1367,10 @@ define <vscale x 2 x i32> @test_vloxseg5_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1424,10 +1401,10 @@ define <vscale x 2 x i32> @test_vloxseg5_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1458,10 +1435,10 @@ define <vscale x 2 x i32> @test_vloxseg5_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1492,11 +1469,11 @@ define <vscale x 2 x i32> @test_vloxseg6_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1527,11 +1504,11 @@ define <vscale x 2 x i32> @test_vloxseg6_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1562,11 +1539,11 @@ define <vscale x 2 x i32> @test_vloxseg6_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1597,12 +1574,12 @@ define <vscale x 2 x i32> @test_vloxseg7_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1633,12 +1610,12 @@ define <vscale x 2 x i32> @test_vloxseg7_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1669,12 +1646,12 @@ define <vscale x 2 x i32> @test_vloxseg7_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1705,13 +1682,13 @@ define <vscale x 2 x i32> @test_vloxseg8_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1742,13 +1719,13 @@ define <vscale x 2 x i32> @test_vloxseg8_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1779,13 +1756,13 @@ define <vscale x 2 x i32> @test_vloxseg8_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1818,7 +1795,6 @@ define <vscale x 4 x i16> @test_vloxseg2_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg2.mask.nxv4i16.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -1848,7 +1824,6 @@ define <vscale x 4 x i16> @test_vloxseg2_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg2.mask.nxv4i16.nxv4i8(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -1878,7 +1853,6 @@ define <vscale x 4 x i16> @test_vloxseg2_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg2.mask.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -1905,12 +1879,11 @@ entry:
 define <vscale x 4 x i16> @test_vloxseg3_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg3.mask.nxv4i16.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -1937,12 +1910,11 @@ entry:
 define <vscale x 4 x i16> @test_vloxseg3_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg3.mask.nxv4i16.nxv4i8(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -1970,10 +1942,9 @@ define <vscale x 4 x i16> @test_vloxseg3_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg3.mask.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -2001,9 +1972,9 @@ define <vscale x 4 x i16> @test_vloxseg4_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2034,9 +2005,9 @@ define <vscale x 4 x i16> @test_vloxseg4_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2066,13 +2037,12 @@ entry:
 define <vscale x 4 x i16> @test_vloxseg4_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vloxseg4ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vloxseg4ei32.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg4.mask.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -2100,10 +2070,10 @@ define <vscale x 4 x i16> @test_vloxseg5_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2134,10 +2104,10 @@ define <vscale x 4 x i16> @test_vloxseg5_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2168,10 +2138,10 @@ define <vscale x 4 x i16> @test_vloxseg5_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -2202,11 +2172,11 @@ define <vscale x 4 x i16> @test_vloxseg6_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2237,11 +2207,11 @@ define <vscale x 4 x i16> @test_vloxseg6_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2272,11 +2242,11 @@ define <vscale x 4 x i16> @test_vloxseg6_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -2307,12 +2277,12 @@ define <vscale x 4 x i16> @test_vloxseg7_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2343,12 +2313,12 @@ define <vscale x 4 x i16> @test_vloxseg7_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2379,12 +2349,12 @@ define <vscale x 4 x i16> @test_vloxseg7_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -2415,13 +2385,13 @@ define <vscale x 4 x i16> @test_vloxseg8_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2452,13 +2422,13 @@ define <vscale x 4 x i16> @test_vloxseg8_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2489,13 +2459,13 @@ define <vscale x 4 x i16> @test_vloxseg8_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -2528,7 +2498,6 @@ define <vscale x 1 x i32> @test_vloxseg2_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vloxseg2.mask.nxv1i32.nxv1i8(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2558,7 +2527,6 @@ define <vscale x 1 x i32> @test_vloxseg2_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vloxseg2.mask.nxv1i32.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2588,7 +2556,6 @@ define <vscale x 1 x i32> @test_vloxseg2_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vloxseg2.mask.nxv1i32.nxv1i16(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2615,12 +2582,11 @@ entry:
 define <vscale x 1 x i32> @test_vloxseg3_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vloxseg3.mask.nxv1i32.nxv1i8(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2647,12 +2613,11 @@ entry:
 define <vscale x 1 x i32> @test_vloxseg3_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vloxseg3.mask.nxv1i32.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2679,12 +2644,11 @@ entry:
 define <vscale x 1 x i32> @test_vloxseg3_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vloxseg3.mask.nxv1i32.nxv1i16(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2712,9 +2676,9 @@ define <vscale x 1 x i32> @test_vloxseg4_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2745,9 +2709,9 @@ define <vscale x 1 x i32> @test_vloxseg4_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2778,9 +2742,9 @@ define <vscale x 1 x i32> @test_vloxseg4_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2811,10 +2775,10 @@ define <vscale x 1 x i32> @test_vloxseg5_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2845,10 +2809,10 @@ define <vscale x 1 x i32> @test_vloxseg5_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2879,10 +2843,10 @@ define <vscale x 1 x i32> @test_vloxseg5_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2913,11 +2877,11 @@ define <vscale x 1 x i32> @test_vloxseg6_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2948,11 +2912,11 @@ define <vscale x 1 x i32> @test_vloxseg6_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2983,11 +2947,11 @@ define <vscale x 1 x i32> @test_vloxseg6_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3018,12 +2982,12 @@ define <vscale x 1 x i32> @test_vloxseg7_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3054,12 +3018,12 @@ define <vscale x 1 x i32> @test_vloxseg7_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3090,12 +3054,12 @@ define <vscale x 1 x i32> @test_vloxseg7_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3126,13 +3090,13 @@ define <vscale x 1 x i32> @test_vloxseg8_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3163,13 +3127,13 @@ define <vscale x 1 x i32> @test_vloxseg8_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3200,13 +3164,13 @@ define <vscale x 1 x i32> @test_vloxseg8_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3239,7 +3203,6 @@ define <vscale x 8 x i16> @test_vloxseg2_mask_nxv8i16_nxv8i16(<vscale x 8 x i16>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vloxseg2.mask.nxv8i16.nxv8i16(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3269,7 +3232,6 @@ define <vscale x 8 x i16> @test_vloxseg2_mask_nxv8i16_nxv8i8(<vscale x 8 x i16>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vloxseg2.mask.nxv8i16.nxv8i8(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3299,7 +3261,6 @@ define <vscale x 8 x i16> @test_vloxseg2_mask_nxv8i16_nxv8i32(<vscale x 8 x i16>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vloxseg2.mask.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3326,12 +3287,11 @@ entry:
 define <vscale x 8 x i16> @test_vloxseg3_mask_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vloxseg3.mask.nxv8i16.nxv8i16(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3358,12 +3318,11 @@ entry:
 define <vscale x 8 x i16> @test_vloxseg3_mask_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, i32 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vloxseg3.mask.nxv8i16.nxv8i8(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3391,10 +3350,9 @@ define <vscale x 8 x i16> @test_vloxseg3_mask_nxv8i16_nxv8i32(<vscale x 8 x i16>
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg3ei32.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vloxseg3.mask.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3422,9 +3380,9 @@ define <vscale x 8 x i16> @test_vloxseg4_mask_nxv8i16_nxv8i16(<vscale x 8 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -3455,9 +3413,9 @@ define <vscale x 8 x i16> @test_vloxseg4_mask_nxv8i16_nxv8i8(<vscale x 8 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -3487,13 +3445,12 @@ entry:
 define <vscale x 8 x i16> @test_vloxseg4_mask_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vloxseg4ei32.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v18
+; CHECK-NEXT:    vloxseg4ei32.v v6, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vloxseg4.mask.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3523,7 +3480,6 @@ define <vscale x 8 x i8> @test_vloxseg2_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg2.mask.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3553,7 +3509,6 @@ define <vscale x 8 x i8> @test_vloxseg2_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg2.mask.nxv8i8.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3583,7 +3538,6 @@ define <vscale x 8 x i8> @test_vloxseg2_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg2.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3611,10 +3565,9 @@ define <vscale x 8 x i8> @test_vloxseg3_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg3.mask.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3641,12 +3594,11 @@ entry:
 define <vscale x 8 x i8> @test_vloxseg3_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, i32 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg3.mask.nxv8i8.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3674,10 +3626,9 @@ define <vscale x 8 x i8> @test_vloxseg3_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg3.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3704,13 +3655,12 @@ entry:
 define <vscale x 8 x i8> @test_vloxseg4_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
-; CHECK-NEXT:    vloxseg4ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vloxseg4ei16.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg4.mask.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3738,9 +3688,9 @@ define <vscale x 8 x i8> @test_vloxseg4_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3771,11 +3721,10 @@ define <vscale x 8 x i8> @test_vloxseg4_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg4.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3803,10 +3752,10 @@ define <vscale x 8 x i8> @test_vloxseg5_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -3837,10 +3786,10 @@ define <vscale x 8 x i8> @test_vloxseg5_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vloxseg5_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3871,12 +3820,11 @@ define <vscale x 8 x i8> @test_vloxseg5_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg5.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3904,11 +3852,11 @@ define <vscale x 8 x i8> @test_vloxseg6_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg6_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -3939,11 +3887,11 @@ define <vscale x 8 x i8> @test_vloxseg6_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vloxseg6_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3973,15 +3921,14 @@ entry:
 define <vscale x 8 x i8> @test_vloxseg6_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg6_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
-; CHECK-NEXT:    vloxseg6ei32.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v17
+; CHECK-NEXT:    vloxseg6ei32.v v7, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg6.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -4009,12 +3956,12 @@ define <vscale x 8 x i8> @test_vloxseg7_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -4045,12 +3992,12 @@ define <vscale x 8 x i8> @test_vloxseg7_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vloxseg7_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4081,12 +4028,12 @@ define <vscale x 8 x i8> @test_vloxseg7_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -4117,13 +4064,13 @@ define <vscale x 8 x i8> @test_vloxseg8_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -4154,13 +4101,13 @@ define <vscale x 8 x i8> @test_vloxseg8_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vloxseg8_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4191,13 +4138,13 @@ define <vscale x 8 x i8> @test_vloxseg8_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -4230,7 +4177,6 @@ define <vscale x 8 x i32> @test_vloxseg2_mask_nxv8i32_nxv8i16(<vscale x 8 x i32>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vloxseg2.mask.nxv8i32.nxv8i16(<vscale x 8 x i32> %val,<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -4260,7 +4206,6 @@ define <vscale x 8 x i32> @test_vloxseg2_mask_nxv8i32_nxv8i8(<vscale x 8 x i32>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vloxseg2.mask.nxv8i32.nxv8i8(<vscale x 8 x i32> %val,<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -4290,7 +4235,6 @@ define <vscale x 8 x i32> @test_vloxseg2_mask_nxv8i32_nxv8i32(<vscale x 8 x i32>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vloxseg2.mask.nxv8i32.nxv8i32(<vscale x 8 x i32> %val,<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -4320,7 +4264,6 @@ define <vscale x 4 x i8> @test_vloxseg2_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg2.mask.nxv4i8.nxv4i16(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4350,7 +4293,6 @@ define <vscale x 4 x i8> @test_vloxseg2_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg2.mask.nxv4i8.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4380,7 +4322,6 @@ define <vscale x 4 x i8> @test_vloxseg2_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg2.mask.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4407,12 +4348,11 @@ entry:
 define <vscale x 4 x i8> @test_vloxseg3_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg3.mask.nxv4i8.nxv4i16(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4439,12 +4379,11 @@ entry:
 define <vscale x 4 x i8> @test_vloxseg3_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg3.mask.nxv4i8.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4472,10 +4411,9 @@ define <vscale x 4 x i8> @test_vloxseg3_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg3.mask.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4503,9 +4441,9 @@ define <vscale x 4 x i8> @test_vloxseg4_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4536,9 +4474,9 @@ define <vscale x 4 x i8> @test_vloxseg4_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4568,13 +4506,12 @@ entry:
 define <vscale x 4 x i8> @test_vloxseg4_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
-; CHECK-NEXT:    vloxseg4ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vloxseg4ei32.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg4.mask.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4602,10 +4539,10 @@ define <vscale x 4 x i8> @test_vloxseg5_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4636,10 +4573,10 @@ define <vscale x 4 x i8> @test_vloxseg5_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4670,10 +4607,10 @@ define <vscale x 4 x i8> @test_vloxseg5_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -4704,11 +4641,11 @@ define <vscale x 4 x i8> @test_vloxseg6_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4739,11 +4676,11 @@ define <vscale x 4 x i8> @test_vloxseg6_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4774,11 +4711,11 @@ define <vscale x 4 x i8> @test_vloxseg6_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -4809,12 +4746,12 @@ define <vscale x 4 x i8> @test_vloxseg7_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4845,12 +4782,12 @@ define <vscale x 4 x i8> @test_vloxseg7_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4881,12 +4818,12 @@ define <vscale x 4 x i8> @test_vloxseg7_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -4917,13 +4854,13 @@ define <vscale x 4 x i8> @test_vloxseg8_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4954,13 +4891,13 @@ define <vscale x 4 x i8> @test_vloxseg8_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4991,13 +4928,13 @@ define <vscale x 4 x i8> @test_vloxseg8_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -5030,7 +4967,6 @@ define <vscale x 1 x i16> @test_vloxseg2_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vloxseg2.mask.nxv1i16.nxv1i8(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -5060,7 +4996,6 @@ define <vscale x 1 x i16> @test_vloxseg2_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vloxseg2.mask.nxv1i16.nxv1i32(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -5090,7 +5025,6 @@ define <vscale x 1 x i16> @test_vloxseg2_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vloxseg2.mask.nxv1i16.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -5117,12 +5051,11 @@ entry:
 define <vscale x 1 x i16> @test_vloxseg3_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vloxseg3.mask.nxv1i16.nxv1i8(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -5149,12 +5082,11 @@ entry:
 define <vscale x 1 x i16> @test_vloxseg3_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vloxseg3.mask.nxv1i16.nxv1i32(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -5181,12 +5113,11 @@ entry:
 define <vscale x 1 x i16> @test_vloxseg3_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vloxseg3.mask.nxv1i16.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -5214,9 +5145,9 @@ define <vscale x 1 x i16> @test_vloxseg4_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5247,9 +5178,9 @@ define <vscale x 1 x i16> @test_vloxseg4_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5280,9 +5211,9 @@ define <vscale x 1 x i16> @test_vloxseg4_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5313,10 +5244,10 @@ define <vscale x 1 x i16> @test_vloxseg5_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5347,10 +5278,10 @@ define <vscale x 1 x i16> @test_vloxseg5_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5381,10 +5312,10 @@ define <vscale x 1 x i16> @test_vloxseg5_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5415,11 +5346,11 @@ define <vscale x 1 x i16> @test_vloxseg6_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5450,11 +5381,11 @@ define <vscale x 1 x i16> @test_vloxseg6_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5485,11 +5416,11 @@ define <vscale x 1 x i16> @test_vloxseg6_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5520,12 +5451,12 @@ define <vscale x 1 x i16> @test_vloxseg7_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5556,12 +5487,12 @@ define <vscale x 1 x i16> @test_vloxseg7_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5592,12 +5523,12 @@ define <vscale x 1 x i16> @test_vloxseg7_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5628,13 +5559,13 @@ define <vscale x 1 x i16> @test_vloxseg8_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5665,13 +5596,13 @@ define <vscale x 1 x i16> @test_vloxseg8_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5702,13 +5633,13 @@ define <vscale x 1 x i16> @test_vloxseg8_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5741,7 +5672,6 @@ define <vscale x 32 x i8> @test_vloxseg2_mask_nxv32i8_nxv32i16(<vscale x 32 x i8
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v4, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>} @llvm.riscv.vloxseg2.mask.nxv32i8.nxv32i16(<vscale x 32 x i8> %val,<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i16> %index, <vscale x 32 x i1> %mask, i32 %vl, i32 1)
@@ -5771,7 +5701,6 @@ define <vscale x 32 x i8> @test_vloxseg2_mask_nxv32i8_nxv32i8(<vscale x 32 x i8>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>} @llvm.riscv.vloxseg2.mask.nxv32i8.nxv32i8(<vscale x 32 x i8> %val,<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i8> %index, <vscale x 32 x i1> %mask, i32 %vl, i32 1)
@@ -5801,7 +5730,6 @@ define <vscale x 2 x i8> @test_vloxseg2_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vloxseg2.mask.nxv2i8.nxv2i32(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -5831,7 +5759,6 @@ define <vscale x 2 x i8> @test_vloxseg2_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vloxseg2.mask.nxv2i8.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -5861,7 +5788,6 @@ define <vscale x 2 x i8> @test_vloxseg2_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vloxseg2.mask.nxv2i8.nxv2i16(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -5888,12 +5814,11 @@ entry:
 define <vscale x 2 x i8> @test_vloxseg3_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vloxseg3.mask.nxv2i8.nxv2i32(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -5920,12 +5845,11 @@ entry:
 define <vscale x 2 x i8> @test_vloxseg3_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vloxseg3.mask.nxv2i8.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -5952,12 +5876,11 @@ entry:
 define <vscale x 2 x i8> @test_vloxseg3_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vloxseg3.mask.nxv2i8.nxv2i16(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -5985,9 +5908,9 @@ define <vscale x 2 x i8> @test_vloxseg4_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6018,9 +5941,9 @@ define <vscale x 2 x i8> @test_vloxseg4_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6051,9 +5974,9 @@ define <vscale x 2 x i8> @test_vloxseg4_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6084,10 +6007,10 @@ define <vscale x 2 x i8> @test_vloxseg5_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6118,10 +6041,10 @@ define <vscale x 2 x i8> @test_vloxseg5_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6152,10 +6075,10 @@ define <vscale x 2 x i8> @test_vloxseg5_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6186,11 +6109,11 @@ define <vscale x 2 x i8> @test_vloxseg6_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6221,11 +6144,11 @@ define <vscale x 2 x i8> @test_vloxseg6_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6256,11 +6179,11 @@ define <vscale x 2 x i8> @test_vloxseg6_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6291,12 +6214,12 @@ define <vscale x 2 x i8> @test_vloxseg7_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6327,12 +6250,12 @@ define <vscale x 2 x i8> @test_vloxseg7_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6363,12 +6286,12 @@ define <vscale x 2 x i8> @test_vloxseg7_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6399,13 +6322,13 @@ define <vscale x 2 x i8> @test_vloxseg8_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6436,13 +6359,13 @@ define <vscale x 2 x i8> @test_vloxseg8_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6473,13 +6396,13 @@ define <vscale x 2 x i8> @test_vloxseg8_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6512,7 +6435,6 @@ define <vscale x 2 x i16> @test_vloxseg2_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vloxseg2.mask.nxv2i16.nxv2i32(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -6542,7 +6464,6 @@ define <vscale x 2 x i16> @test_vloxseg2_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vloxseg2.mask.nxv2i16.nxv2i8(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -6572,7 +6493,6 @@ define <vscale x 2 x i16> @test_vloxseg2_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vloxseg2.mask.nxv2i16.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -6599,12 +6519,11 @@ entry:
 define <vscale x 2 x i16> @test_vloxseg3_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vloxseg3.mask.nxv2i16.nxv2i32(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -6631,12 +6550,11 @@ entry:
 define <vscale x 2 x i16> @test_vloxseg3_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vloxseg3.mask.nxv2i16.nxv2i8(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -6663,12 +6581,11 @@ entry:
 define <vscale x 2 x i16> @test_vloxseg3_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vloxseg3.mask.nxv2i16.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -6696,9 +6613,9 @@ define <vscale x 2 x i16> @test_vloxseg4_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6729,9 +6646,9 @@ define <vscale x 2 x i16> @test_vloxseg4_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6762,9 +6679,9 @@ define <vscale x 2 x i16> @test_vloxseg4_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6795,10 +6712,10 @@ define <vscale x 2 x i16> @test_vloxseg5_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6829,10 +6746,10 @@ define <vscale x 2 x i16> @test_vloxseg5_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6863,10 +6780,10 @@ define <vscale x 2 x i16> @test_vloxseg5_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6897,11 +6814,11 @@ define <vscale x 2 x i16> @test_vloxseg6_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6932,11 +6849,11 @@ define <vscale x 2 x i16> @test_vloxseg6_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6967,11 +6884,11 @@ define <vscale x 2 x i16> @test_vloxseg6_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7002,12 +6919,12 @@ define <vscale x 2 x i16> @test_vloxseg7_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7038,12 +6955,12 @@ define <vscale x 2 x i16> @test_vloxseg7_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7074,12 +6991,12 @@ define <vscale x 2 x i16> @test_vloxseg7_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7110,13 +7027,13 @@ define <vscale x 2 x i16> @test_vloxseg8_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7147,13 +7064,13 @@ define <vscale x 2 x i16> @test_vloxseg8_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7184,13 +7101,13 @@ define <vscale x 2 x i16> @test_vloxseg8_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7223,7 +7140,6 @@ define <vscale x 4 x i32> @test_vloxseg2_mask_nxv4i32_nxv4i16(<vscale x 4 x i32>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vloxseg2.mask.nxv4i32.nxv4i16(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -7253,7 +7169,6 @@ define <vscale x 4 x i32> @test_vloxseg2_mask_nxv4i32_nxv4i8(<vscale x 4 x i32>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vloxseg2.mask.nxv4i32.nxv4i8(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -7283,7 +7198,6 @@ define <vscale x 4 x i32> @test_vloxseg2_mask_nxv4i32_nxv4i32(<vscale x 4 x i32>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vloxseg2.mask.nxv4i32.nxv4i32(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -7310,12 +7224,11 @@ entry:
 define <vscale x 4 x i32> @test_vloxseg3_mask_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vloxseg3.mask.nxv4i32.nxv4i16(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -7342,12 +7255,11 @@ entry:
 define <vscale x 4 x i32> @test_vloxseg3_mask_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vloxseg3.mask.nxv4i32.nxv4i8(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -7374,12 +7286,11 @@ entry:
 define <vscale x 4 x i32> @test_vloxseg3_mask_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei32.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vloxseg3.mask.nxv4i32.nxv4i32(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -7407,9 +7318,9 @@ define <vscale x 4 x i32> @test_vloxseg4_mask_nxv4i32_nxv4i16(<vscale x 4 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -7440,9 +7351,9 @@ define <vscale x 4 x i32> @test_vloxseg4_mask_nxv4i32_nxv4i8(<vscale x 4 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -7473,9 +7384,9 @@ define <vscale x 4 x i32> @test_vloxseg4_mask_nxv4i32_nxv4i32(<vscale x 4 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -7508,7 +7419,6 @@ define <vscale x 16 x half> @test_vloxseg2_mask_nxv16f16_nxv16i16(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vloxseg2.mask.nxv16f16.nxv16i16(<vscale x 16 x half> %val,<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -7538,7 +7448,6 @@ define <vscale x 16 x half> @test_vloxseg2_mask_nxv16f16_nxv16i8(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vloxseg2.mask.nxv16f16.nxv16i8(<vscale x 16 x half> %val,<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -7568,7 +7477,6 @@ define <vscale x 16 x half> @test_vloxseg2_mask_nxv16f16_nxv16i32(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v4, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vloxseg2.mask.nxv16f16.nxv16i32(<vscale x 16 x half> %val,<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -7598,7 +7506,6 @@ define <vscale x 4 x double> @test_vloxseg2_mask_nxv4f64_nxv4i16(<vscale x 4 x d
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vloxseg2.mask.nxv4f64.nxv4i16(<vscale x 4 x double> %val,<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -7628,7 +7535,6 @@ define <vscale x 4 x double> @test_vloxseg2_mask_nxv4f64_nxv4i8(<vscale x 4 x do
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vloxseg2.mask.nxv4f64.nxv4i8(<vscale x 4 x double> %val,<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -7658,7 +7564,6 @@ define <vscale x 4 x double> @test_vloxseg2_mask_nxv4f64_nxv4i32(<vscale x 4 x d
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vloxseg2.mask.nxv4f64.nxv4i32(<vscale x 4 x double> %val,<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -7688,7 +7593,6 @@ define <vscale x 1 x double> @test_vloxseg2_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vloxseg2.mask.nxv1f64.nxv1i8(<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -7718,7 +7622,6 @@ define <vscale x 1 x double> @test_vloxseg2_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vloxseg2.mask.nxv1f64.nxv1i32(<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -7748,7 +7651,6 @@ define <vscale x 1 x double> @test_vloxseg2_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vloxseg2.mask.nxv1f64.nxv1i16(<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -7775,12 +7677,11 @@ entry:
 define <vscale x 1 x double> @test_vloxseg3_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vloxseg3.mask.nxv1f64.nxv1i8(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -7807,12 +7708,11 @@ entry:
 define <vscale x 1 x double> @test_vloxseg3_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vloxseg3.mask.nxv1f64.nxv1i32(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -7839,12 +7739,11 @@ entry:
 define <vscale x 1 x double> @test_vloxseg3_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vloxseg3.mask.nxv1f64.nxv1i16(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -7872,9 +7771,9 @@ define <vscale x 1 x double> @test_vloxseg4_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7905,9 +7804,9 @@ define <vscale x 1 x double> @test_vloxseg4_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7938,9 +7837,9 @@ define <vscale x 1 x double> @test_vloxseg4_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7971,10 +7870,10 @@ define <vscale x 1 x double> @test_vloxseg5_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8005,10 +7904,10 @@ define <vscale x 1 x double> @test_vloxseg5_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8039,10 +7938,10 @@ define <vscale x 1 x double> @test_vloxseg5_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8073,11 +7972,11 @@ define <vscale x 1 x double> @test_vloxseg6_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8108,11 +8007,11 @@ define <vscale x 1 x double> @test_vloxseg6_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8143,11 +8042,11 @@ define <vscale x 1 x double> @test_vloxseg6_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8178,12 +8077,12 @@ define <vscale x 1 x double> @test_vloxseg7_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8214,12 +8113,12 @@ define <vscale x 1 x double> @test_vloxseg7_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8250,12 +8149,12 @@ define <vscale x 1 x double> @test_vloxseg7_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8286,13 +8185,13 @@ define <vscale x 1 x double> @test_vloxseg8_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8323,13 +8222,13 @@ define <vscale x 1 x double> @test_vloxseg8_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8360,13 +8259,13 @@ define <vscale x 1 x double> @test_vloxseg8_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8399,7 +8298,6 @@ define <vscale x 2 x float> @test_vloxseg2_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vloxseg2.mask.nxv2f32.nxv2i32(<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -8429,7 +8327,6 @@ define <vscale x 2 x float> @test_vloxseg2_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vloxseg2.mask.nxv2f32.nxv2i8(<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -8459,7 +8356,6 @@ define <vscale x 2 x float> @test_vloxseg2_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vloxseg2.mask.nxv2f32.nxv2i16(<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -8486,12 +8382,11 @@ entry:
 define <vscale x 2 x float> @test_vloxseg3_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vloxseg3.mask.nxv2f32.nxv2i32(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -8518,12 +8413,11 @@ entry:
 define <vscale x 2 x float> @test_vloxseg3_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vloxseg3.mask.nxv2f32.nxv2i8(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -8550,12 +8444,11 @@ entry:
 define <vscale x 2 x float> @test_vloxseg3_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vloxseg3.mask.nxv2f32.nxv2i16(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -8583,9 +8476,9 @@ define <vscale x 2 x float> @test_vloxseg4_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8616,9 +8509,9 @@ define <vscale x 2 x float> @test_vloxseg4_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8649,9 +8542,9 @@ define <vscale x 2 x float> @test_vloxseg4_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8682,10 +8575,10 @@ define <vscale x 2 x float> @test_vloxseg5_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8716,10 +8609,10 @@ define <vscale x 2 x float> @test_vloxseg5_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8750,10 +8643,10 @@ define <vscale x 2 x float> @test_vloxseg5_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8784,11 +8677,11 @@ define <vscale x 2 x float> @test_vloxseg6_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8819,11 +8712,11 @@ define <vscale x 2 x float> @test_vloxseg6_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8854,11 +8747,11 @@ define <vscale x 2 x float> @test_vloxseg6_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8889,12 +8782,12 @@ define <vscale x 2 x float> @test_vloxseg7_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8925,12 +8818,12 @@ define <vscale x 2 x float> @test_vloxseg7_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8961,12 +8854,12 @@ define <vscale x 2 x float> @test_vloxseg7_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8997,13 +8890,13 @@ define <vscale x 2 x float> @test_vloxseg8_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9034,13 +8927,13 @@ define <vscale x 2 x float> @test_vloxseg8_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9071,13 +8964,13 @@ define <vscale x 2 x float> @test_vloxseg8_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9110,7 +9003,6 @@ define <vscale x 1 x half> @test_vloxseg2_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vloxseg2.mask.nxv1f16.nxv1i8(<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9140,7 +9032,6 @@ define <vscale x 1 x half> @test_vloxseg2_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vloxseg2.mask.nxv1f16.nxv1i32(<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9170,7 +9061,6 @@ define <vscale x 1 x half> @test_vloxseg2_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vloxseg2.mask.nxv1f16.nxv1i16(<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9197,12 +9087,11 @@ entry:
 define <vscale x 1 x half> @test_vloxseg3_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vloxseg3.mask.nxv1f16.nxv1i8(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9229,12 +9118,11 @@ entry:
 define <vscale x 1 x half> @test_vloxseg3_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vloxseg3.mask.nxv1f16.nxv1i32(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9261,12 +9149,11 @@ entry:
 define <vscale x 1 x half> @test_vloxseg3_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vloxseg3.mask.nxv1f16.nxv1i16(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9294,9 +9181,9 @@ define <vscale x 1 x half> @test_vloxseg4_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9327,9 +9214,9 @@ define <vscale x 1 x half> @test_vloxseg4_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9360,9 +9247,9 @@ define <vscale x 1 x half> @test_vloxseg4_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9393,10 +9280,10 @@ define <vscale x 1 x half> @test_vloxseg5_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9427,10 +9314,10 @@ define <vscale x 1 x half> @test_vloxseg5_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9461,10 +9348,10 @@ define <vscale x 1 x half> @test_vloxseg5_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9495,11 +9382,11 @@ define <vscale x 1 x half> @test_vloxseg6_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9530,11 +9417,11 @@ define <vscale x 1 x half> @test_vloxseg6_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9565,11 +9452,11 @@ define <vscale x 1 x half> @test_vloxseg6_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9600,12 +9487,12 @@ define <vscale x 1 x half> @test_vloxseg7_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9636,12 +9523,12 @@ define <vscale x 1 x half> @test_vloxseg7_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9672,12 +9559,12 @@ define <vscale x 1 x half> @test_vloxseg7_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9708,13 +9595,13 @@ define <vscale x 1 x half> @test_vloxseg8_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9745,13 +9632,13 @@ define <vscale x 1 x half> @test_vloxseg8_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9782,13 +9669,13 @@ define <vscale x 1 x half> @test_vloxseg8_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9821,7 +9708,6 @@ define <vscale x 1 x float> @test_vloxseg2_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vloxseg2.mask.nxv1f32.nxv1i8(<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9851,7 +9737,6 @@ define <vscale x 1 x float> @test_vloxseg2_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vloxseg2.mask.nxv1f32.nxv1i32(<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9881,7 +9766,6 @@ define <vscale x 1 x float> @test_vloxseg2_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vloxseg2.mask.nxv1f32.nxv1i16(<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9908,12 +9792,11 @@ entry:
 define <vscale x 1 x float> @test_vloxseg3_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vloxseg3.mask.nxv1f32.nxv1i8(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9940,12 +9823,11 @@ entry:
 define <vscale x 1 x float> @test_vloxseg3_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vloxseg3.mask.nxv1f32.nxv1i32(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9972,12 +9854,11 @@ entry:
 define <vscale x 1 x float> @test_vloxseg3_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vloxseg3.mask.nxv1f32.nxv1i16(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -10005,9 +9886,9 @@ define <vscale x 1 x float> @test_vloxseg4_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10038,9 +9919,9 @@ define <vscale x 1 x float> @test_vloxseg4_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10071,9 +9952,9 @@ define <vscale x 1 x float> @test_vloxseg4_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10104,10 +9985,10 @@ define <vscale x 1 x float> @test_vloxseg5_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10138,10 +10019,10 @@ define <vscale x 1 x float> @test_vloxseg5_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10172,10 +10053,10 @@ define <vscale x 1 x float> @test_vloxseg5_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10206,11 +10087,11 @@ define <vscale x 1 x float> @test_vloxseg6_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10241,11 +10122,11 @@ define <vscale x 1 x float> @test_vloxseg6_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10276,11 +10157,11 @@ define <vscale x 1 x float> @test_vloxseg6_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10311,12 +10192,12 @@ define <vscale x 1 x float> @test_vloxseg7_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10347,12 +10228,12 @@ define <vscale x 1 x float> @test_vloxseg7_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10383,12 +10264,12 @@ define <vscale x 1 x float> @test_vloxseg7_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10419,13 +10300,13 @@ define <vscale x 1 x float> @test_vloxseg8_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10456,13 +10337,13 @@ define <vscale x 1 x float> @test_vloxseg8_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10493,13 +10374,13 @@ define <vscale x 1 x float> @test_vloxseg8_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10532,7 +10413,6 @@ define <vscale x 8 x half> @test_vloxseg2_mask_nxv8f16_nxv8i16(<vscale x 8 x hal
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vloxseg2.mask.nxv8f16.nxv8i16(<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10562,7 +10442,6 @@ define <vscale x 8 x half> @test_vloxseg2_mask_nxv8f16_nxv8i8(<vscale x 8 x half
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vloxseg2.mask.nxv8f16.nxv8i8(<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10592,7 +10471,6 @@ define <vscale x 8 x half> @test_vloxseg2_mask_nxv8f16_nxv8i32(<vscale x 8 x hal
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vloxseg2.mask.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10619,12 +10497,11 @@ entry:
 define <vscale x 8 x half> @test_vloxseg3_mask_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vloxseg3.mask.nxv8f16.nxv8i16(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10651,12 +10528,11 @@ entry:
 define <vscale x 8 x half> @test_vloxseg3_mask_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, i32 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vloxseg3.mask.nxv8f16.nxv8i8(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10684,10 +10560,9 @@ define <vscale x 8 x half> @test_vloxseg3_mask_nxv8f16_nxv8i32(<vscale x 8 x hal
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg3ei32.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vloxseg3.mask.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10715,9 +10590,9 @@ define <vscale x 8 x half> @test_vloxseg4_mask_nxv8f16_nxv8i16(<vscale x 8 x hal
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -10748,9 +10623,9 @@ define <vscale x 8 x half> @test_vloxseg4_mask_nxv8f16_nxv8i8(<vscale x 8 x half
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -10780,13 +10655,12 @@ entry:
 define <vscale x 8 x half> @test_vloxseg4_mask_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vloxseg4ei32.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v18
+; CHECK-NEXT:    vloxseg4ei32.v v6, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vloxseg4.mask.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10816,7 +10690,6 @@ define <vscale x 8 x float> @test_vloxseg2_mask_nxv8f32_nxv8i16(<vscale x 8 x fl
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vloxseg2.mask.nxv8f32.nxv8i16(<vscale x 8 x float> %val,<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10846,7 +10719,6 @@ define <vscale x 8 x float> @test_vloxseg2_mask_nxv8f32_nxv8i8(<vscale x 8 x flo
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vloxseg2.mask.nxv8f32.nxv8i8(<vscale x 8 x float> %val,<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10876,7 +10748,6 @@ define <vscale x 8 x float> @test_vloxseg2_mask_nxv8f32_nxv8i32(<vscale x 8 x fl
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vloxseg2.mask.nxv8f32.nxv8i32(<vscale x 8 x float> %val,<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10906,7 +10777,6 @@ define <vscale x 2 x double> @test_vloxseg2_mask_nxv2f64_nxv2i32(<vscale x 2 x d
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vloxseg2.mask.nxv2f64.nxv2i32(<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -10936,7 +10806,6 @@ define <vscale x 2 x double> @test_vloxseg2_mask_nxv2f64_nxv2i8(<vscale x 2 x do
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vloxseg2.mask.nxv2f64.nxv2i8(<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -10966,7 +10835,6 @@ define <vscale x 2 x double> @test_vloxseg2_mask_nxv2f64_nxv2i16(<vscale x 2 x d
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vloxseg2.mask.nxv2f64.nxv2i16(<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -10993,12 +10861,11 @@ entry:
 define <vscale x 2 x double> @test_vloxseg3_mask_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei32.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vloxseg3.mask.nxv2f64.nxv2i32(<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -11025,12 +10892,11 @@ entry:
 define <vscale x 2 x double> @test_vloxseg3_mask_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vloxseg3.mask.nxv2f64.nxv2i8(<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -11057,12 +10923,11 @@ entry:
 define <vscale x 2 x double> @test_vloxseg3_mask_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vloxseg3.mask.nxv2f64.nxv2i16(<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -11090,9 +10955,9 @@ define <vscale x 2 x double> @test_vloxseg4_mask_nxv2f64_nxv2i32(<vscale x 2 x d
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -11123,9 +10988,9 @@ define <vscale x 2 x double> @test_vloxseg4_mask_nxv2f64_nxv2i8(<vscale x 2 x do
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -11156,9 +11021,9 @@ define <vscale x 2 x double> @test_vloxseg4_mask_nxv2f64_nxv2i16(<vscale x 2 x d
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -11191,7 +11056,6 @@ define <vscale x 4 x half> @test_vloxseg2_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg2.mask.nxv4f16.nxv4i16(<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -11221,7 +11085,6 @@ define <vscale x 4 x half> @test_vloxseg2_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg2.mask.nxv4f16.nxv4i8(<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -11251,7 +11114,6 @@ define <vscale x 4 x half> @test_vloxseg2_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg2.mask.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -11278,12 +11140,11 @@ entry:
 define <vscale x 4 x half> @test_vloxseg3_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg3.mask.nxv4f16.nxv4i16(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -11310,12 +11171,11 @@ entry:
 define <vscale x 4 x half> @test_vloxseg3_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg3.mask.nxv4f16.nxv4i8(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -11343,10 +11203,9 @@ define <vscale x 4 x half> @test_vloxseg3_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg3.mask.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -11374,9 +11233,9 @@ define <vscale x 4 x half> @test_vloxseg4_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11407,9 +11266,9 @@ define <vscale x 4 x half> @test_vloxseg4_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11439,13 +11298,12 @@ entry:
 define <vscale x 4 x half> @test_vloxseg4_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vloxseg4ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vloxseg4ei32.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg4.mask.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -11473,10 +11331,10 @@ define <vscale x 4 x half> @test_vloxseg5_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11507,10 +11365,10 @@ define <vscale x 4 x half> @test_vloxseg5_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11541,10 +11399,10 @@ define <vscale x 4 x half> @test_vloxseg5_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -11575,11 +11433,11 @@ define <vscale x 4 x half> @test_vloxseg6_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11610,11 +11468,11 @@ define <vscale x 4 x half> @test_vloxseg6_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11645,11 +11503,11 @@ define <vscale x 4 x half> @test_vloxseg6_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -11680,12 +11538,12 @@ define <vscale x 4 x half> @test_vloxseg7_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11716,12 +11574,12 @@ define <vscale x 4 x half> @test_vloxseg7_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11752,12 +11610,12 @@ define <vscale x 4 x half> @test_vloxseg7_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -11788,13 +11646,13 @@ define <vscale x 4 x half> @test_vloxseg8_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11825,13 +11683,13 @@ define <vscale x 4 x half> @test_vloxseg8_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11862,13 +11720,13 @@ define <vscale x 4 x half> @test_vloxseg8_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -11901,7 +11759,6 @@ define <vscale x 2 x half> @test_vloxseg2_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vloxseg2.mask.nxv2f16.nxv2i32(<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -11931,7 +11788,6 @@ define <vscale x 2 x half> @test_vloxseg2_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vloxseg2.mask.nxv2f16.nxv2i8(<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -11961,7 +11817,6 @@ define <vscale x 2 x half> @test_vloxseg2_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vloxseg2.mask.nxv2f16.nxv2i16(<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -11988,12 +11843,11 @@ entry:
 define <vscale x 2 x half> @test_vloxseg3_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vloxseg3.mask.nxv2f16.nxv2i32(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -12020,12 +11874,11 @@ entry:
 define <vscale x 2 x half> @test_vloxseg3_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vloxseg3.mask.nxv2f16.nxv2i8(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -12052,12 +11905,11 @@ entry:
 define <vscale x 2 x half> @test_vloxseg3_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vloxseg3.mask.nxv2f16.nxv2i16(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -12085,9 +11937,9 @@ define <vscale x 2 x half> @test_vloxseg4_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12118,9 +11970,9 @@ define <vscale x 2 x half> @test_vloxseg4_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12151,9 +12003,9 @@ define <vscale x 2 x half> @test_vloxseg4_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12184,10 +12036,10 @@ define <vscale x 2 x half> @test_vloxseg5_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12218,10 +12070,10 @@ define <vscale x 2 x half> @test_vloxseg5_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12252,10 +12104,10 @@ define <vscale x 2 x half> @test_vloxseg5_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12286,11 +12138,11 @@ define <vscale x 2 x half> @test_vloxseg6_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12321,11 +12173,11 @@ define <vscale x 2 x half> @test_vloxseg6_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12356,11 +12208,11 @@ define <vscale x 2 x half> @test_vloxseg6_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12391,12 +12243,12 @@ define <vscale x 2 x half> @test_vloxseg7_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12427,12 +12279,12 @@ define <vscale x 2 x half> @test_vloxseg7_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12463,12 +12315,12 @@ define <vscale x 2 x half> @test_vloxseg7_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12499,13 +12351,13 @@ define <vscale x 2 x half> @test_vloxseg8_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12536,13 +12388,13 @@ define <vscale x 2 x half> @test_vloxseg8_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12573,13 +12425,13 @@ define <vscale x 2 x half> @test_vloxseg8_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12612,7 +12464,6 @@ define <vscale x 4 x float> @test_vloxseg2_mask_nxv4f32_nxv4i16(<vscale x 4 x fl
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vloxseg2.mask.nxv4f32.nxv4i16(<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -12642,7 +12493,6 @@ define <vscale x 4 x float> @test_vloxseg2_mask_nxv4f32_nxv4i8(<vscale x 4 x flo
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vloxseg2.mask.nxv4f32.nxv4i8(<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -12672,7 +12522,6 @@ define <vscale x 4 x float> @test_vloxseg2_mask_nxv4f32_nxv4i32(<vscale x 4 x fl
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vloxseg2.mask.nxv4f32.nxv4i32(<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -12699,12 +12548,11 @@ entry:
 define <vscale x 4 x float> @test_vloxseg3_mask_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vloxseg3.mask.nxv4f32.nxv4i16(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -12731,12 +12579,11 @@ entry:
 define <vscale x 4 x float> @test_vloxseg3_mask_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vloxseg3.mask.nxv4f32.nxv4i8(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -12763,12 +12610,11 @@ entry:
 define <vscale x 4 x float> @test_vloxseg3_mask_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei32.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vloxseg3.mask.nxv4f32.nxv4i32(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -12796,9 +12642,9 @@ define <vscale x 4 x float> @test_vloxseg4_mask_nxv4f32_nxv4i16(<vscale x 4 x fl
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -12829,9 +12675,9 @@ define <vscale x 4 x float> @test_vloxseg4_mask_nxv4f32_nxv4i8(<vscale x 4 x flo
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -12862,9 +12708,9 @@ define <vscale x 4 x float> @test_vloxseg4_mask_nxv4f32_nxv4i32(<vscale x 4 x fl
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
diff --git a/llvm/test/CodeGen/RISCV/rvv/vloxseg-rv64.ll b/llvm/test/CodeGen/RISCV/rvv/vloxseg-rv64.ll
index b79a0fb709022..ad2af75acef4f 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vloxseg-rv64.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vloxseg-rv64.ll
@@ -24,7 +24,6 @@ define <vscale x 16 x i16> @test_vloxseg2_mask_nxv16i16_nxv16i16(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vloxseg2.mask.nxv16i16.nxv16i16(<vscale x 16 x i16> %val,<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -54,7 +53,6 @@ define <vscale x 16 x i16> @test_vloxseg2_mask_nxv16i16_nxv16i8(<vscale x 16 x i
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vloxseg2.mask.nxv16i16.nxv16i8(<vscale x 16 x i16> %val,<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -84,7 +82,6 @@ define <vscale x 16 x i16> @test_vloxseg2_mask_nxv16i16_nxv16i32(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v4, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vloxseg2.mask.nxv16i16.nxv16i32(<vscale x 16 x i16> %val,<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -114,7 +111,6 @@ define <vscale x 4 x i32> @test_vloxseg2_mask_nxv4i32_nxv4i32(<vscale x 4 x i32>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vloxseg2.mask.nxv4i32.nxv4i32(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -144,7 +140,6 @@ define <vscale x 4 x i32> @test_vloxseg2_mask_nxv4i32_nxv4i8(<vscale x 4 x i32>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vloxseg2.mask.nxv4i32.nxv4i8(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -174,7 +169,6 @@ define <vscale x 4 x i32> @test_vloxseg2_mask_nxv4i32_nxv4i64(<vscale x 4 x i32>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vloxseg2.mask.nxv4i32.nxv4i64(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -204,7 +198,6 @@ define <vscale x 4 x i32> @test_vloxseg2_mask_nxv4i32_nxv4i16(<vscale x 4 x i32>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vloxseg2.mask.nxv4i32.nxv4i16(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -231,12 +224,11 @@ entry:
 define <vscale x 4 x i32> @test_vloxseg3_mask_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei32.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vloxseg3.mask.nxv4i32.nxv4i32(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -263,12 +255,11 @@ entry:
 define <vscale x 4 x i32> @test_vloxseg3_mask_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vloxseg3.mask.nxv4i32.nxv4i8(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -296,10 +287,9 @@ define <vscale x 4 x i32> @test_vloxseg3_mask_nxv4i32_nxv4i64(<vscale x 4 x i32>
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg3ei64.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vloxseg3.mask.nxv4i32.nxv4i64(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -326,12 +316,11 @@ entry:
 define <vscale x 4 x i32> @test_vloxseg3_mask_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vloxseg3.mask.nxv4i32.nxv4i16(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -359,9 +348,9 @@ define <vscale x 4 x i32> @test_vloxseg4_mask_nxv4i32_nxv4i32(<vscale x 4 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -392,9 +381,9 @@ define <vscale x 4 x i32> @test_vloxseg4_mask_nxv4i32_nxv4i8(<vscale x 4 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -424,13 +413,12 @@ entry:
 define <vscale x 4 x i32> @test_vloxseg4_mask_nxv4i32_nxv4i64(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vloxseg4ei64.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v18
+; CHECK-NEXT:    vloxseg4ei64.v v6, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vloxseg4.mask.nxv4i32.nxv4i64(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -458,9 +446,9 @@ define <vscale x 4 x i32> @test_vloxseg4_mask_nxv4i32_nxv4i16(<vscale x 4 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -493,7 +481,6 @@ define <vscale x 16 x i8> @test_vloxseg2_mask_nxv16i8_nxv16i16(<vscale x 16 x i8
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vloxseg2.mask.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -523,7 +510,6 @@ define <vscale x 16 x i8> @test_vloxseg2_mask_nxv16i8_nxv16i8(<vscale x 16 x i8>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vloxseg2.mask.nxv16i8.nxv16i8(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -553,7 +539,6 @@ define <vscale x 16 x i8> @test_vloxseg2_mask_nxv16i8_nxv16i32(<vscale x 16 x i8
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vloxseg2.mask.nxv16i8.nxv16i32(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -581,10 +566,9 @@ define <vscale x 16 x i8> @test_vloxseg3_mask_nxv16i8_nxv16i16(<vscale x 16 x i8
 ; CHECK-LABEL: test_vloxseg3_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vloxseg3ei16.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vloxseg3.mask.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -611,12 +595,11 @@ entry:
 define <vscale x 16 x i8> @test_vloxseg3_mask_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, i64 %vl, <vscale x 16 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vloxseg3.mask.nxv16i8.nxv16i8(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -644,10 +627,9 @@ define <vscale x 16 x i8> @test_vloxseg3_mask_nxv16i8_nxv16i32(<vscale x 16 x i8
 ; CHECK-LABEL: test_vloxseg3_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vloxseg3ei32.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vloxseg3.mask.nxv16i8.nxv16i32(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -674,13 +656,12 @@ entry:
 define <vscale x 16 x i8> @test_vloxseg4_mask_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, i64 %vl, <vscale x 16 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
-; CHECK-NEXT:    vloxseg4ei16.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v18
+; CHECK-NEXT:    vloxseg4ei16.v v6, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vloxseg4.mask.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -708,9 +689,9 @@ define <vscale x 16 x i8> @test_vloxseg4_mask_nxv16i8_nxv16i8(<vscale x 16 x i8>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -741,11 +722,10 @@ define <vscale x 16 x i8> @test_vloxseg4_mask_nxv16i8_nxv16i32(<vscale x 16 x i8
 ; CHECK-LABEL: test_vloxseg4_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vloxseg4.mask.nxv16i8.nxv16i32(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -775,7 +755,6 @@ define <vscale x 1 x i64> @test_vloxseg2_mask_nxv1i64_nxv1i64(<vscale x 1 x i64>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vloxseg2.mask.nxv1i64.nxv1i64(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -805,7 +784,6 @@ define <vscale x 1 x i64> @test_vloxseg2_mask_nxv1i64_nxv1i32(<vscale x 1 x i64>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vloxseg2.mask.nxv1i64.nxv1i32(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -835,7 +813,6 @@ define <vscale x 1 x i64> @test_vloxseg2_mask_nxv1i64_nxv1i16(<vscale x 1 x i64>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vloxseg2.mask.nxv1i64.nxv1i16(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -865,7 +842,6 @@ define <vscale x 1 x i64> @test_vloxseg2_mask_nxv1i64_nxv1i8(<vscale x 1 x i64>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vloxseg2.mask.nxv1i64.nxv1i8(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -892,12 +868,11 @@ entry:
 define <vscale x 1 x i64> @test_vloxseg3_mask_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei64.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei64.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vloxseg3.mask.nxv1i64.nxv1i64(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -924,12 +899,11 @@ entry:
 define <vscale x 1 x i64> @test_vloxseg3_mask_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vloxseg3.mask.nxv1i64.nxv1i32(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -956,12 +930,11 @@ entry:
 define <vscale x 1 x i64> @test_vloxseg3_mask_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vloxseg3.mask.nxv1i64.nxv1i16(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -988,12 +961,11 @@ entry:
 define <vscale x 1 x i64> @test_vloxseg3_mask_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vloxseg3.mask.nxv1i64.nxv1i8(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1021,9 +993,9 @@ define <vscale x 1 x i64> @test_vloxseg4_mask_nxv1i64_nxv1i64(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1054,9 +1026,9 @@ define <vscale x 1 x i64> @test_vloxseg4_mask_nxv1i64_nxv1i32(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1087,9 +1059,9 @@ define <vscale x 1 x i64> @test_vloxseg4_mask_nxv1i64_nxv1i16(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1120,9 +1092,9 @@ define <vscale x 1 x i64> @test_vloxseg4_mask_nxv1i64_nxv1i8(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1153,10 +1125,10 @@ define <vscale x 1 x i64> @test_vloxseg5_mask_nxv1i64_nxv1i64(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1187,10 +1159,10 @@ define <vscale x 1 x i64> @test_vloxseg5_mask_nxv1i64_nxv1i32(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1221,10 +1193,10 @@ define <vscale x 1 x i64> @test_vloxseg5_mask_nxv1i64_nxv1i16(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1255,10 +1227,10 @@ define <vscale x 1 x i64> @test_vloxseg5_mask_nxv1i64_nxv1i8(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1289,11 +1261,11 @@ define <vscale x 1 x i64> @test_vloxseg6_mask_nxv1i64_nxv1i64(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1324,11 +1296,11 @@ define <vscale x 1 x i64> @test_vloxseg6_mask_nxv1i64_nxv1i32(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1359,11 +1331,11 @@ define <vscale x 1 x i64> @test_vloxseg6_mask_nxv1i64_nxv1i16(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1394,11 +1366,11 @@ define <vscale x 1 x i64> @test_vloxseg6_mask_nxv1i64_nxv1i8(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1429,12 +1401,12 @@ define <vscale x 1 x i64> @test_vloxseg7_mask_nxv1i64_nxv1i64(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1465,12 +1437,12 @@ define <vscale x 1 x i64> @test_vloxseg7_mask_nxv1i64_nxv1i32(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1501,12 +1473,12 @@ define <vscale x 1 x i64> @test_vloxseg7_mask_nxv1i64_nxv1i16(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1537,12 +1509,12 @@ define <vscale x 1 x i64> @test_vloxseg7_mask_nxv1i64_nxv1i8(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1573,13 +1545,13 @@ define <vscale x 1 x i64> @test_vloxseg8_mask_nxv1i64_nxv1i64(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1610,13 +1582,13 @@ define <vscale x 1 x i64> @test_vloxseg8_mask_nxv1i64_nxv1i32(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1647,13 +1619,13 @@ define <vscale x 1 x i64> @test_vloxseg8_mask_nxv1i64_nxv1i16(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1684,13 +1656,13 @@ define <vscale x 1 x i64> @test_vloxseg8_mask_nxv1i64_nxv1i8(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1723,7 +1695,6 @@ define <vscale x 1 x i32> @test_vloxseg2_mask_nxv1i32_nxv1i64(<vscale x 1 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vloxseg2.mask.nxv1i32.nxv1i64(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1753,7 +1724,6 @@ define <vscale x 1 x i32> @test_vloxseg2_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vloxseg2.mask.nxv1i32.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1783,7 +1753,6 @@ define <vscale x 1 x i32> @test_vloxseg2_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vloxseg2.mask.nxv1i32.nxv1i16(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1813,7 +1782,6 @@ define <vscale x 1 x i32> @test_vloxseg2_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vloxseg2.mask.nxv1i32.nxv1i8(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1840,12 +1808,11 @@ entry:
 define <vscale x 1 x i32> @test_vloxseg3_mask_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei64.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei64.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vloxseg3.mask.nxv1i32.nxv1i64(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1872,12 +1839,11 @@ entry:
 define <vscale x 1 x i32> @test_vloxseg3_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vloxseg3.mask.nxv1i32.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1904,12 +1870,11 @@ entry:
 define <vscale x 1 x i32> @test_vloxseg3_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vloxseg3.mask.nxv1i32.nxv1i16(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1936,12 +1901,11 @@ entry:
 define <vscale x 1 x i32> @test_vloxseg3_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vloxseg3.mask.nxv1i32.nxv1i8(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1969,9 +1933,9 @@ define <vscale x 1 x i32> @test_vloxseg4_mask_nxv1i32_nxv1i64(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2002,9 +1966,9 @@ define <vscale x 1 x i32> @test_vloxseg4_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2035,9 +1999,9 @@ define <vscale x 1 x i32> @test_vloxseg4_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2068,9 +2032,9 @@ define <vscale x 1 x i32> @test_vloxseg4_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2101,10 +2065,10 @@ define <vscale x 1 x i32> @test_vloxseg5_mask_nxv1i32_nxv1i64(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2135,10 +2099,10 @@ define <vscale x 1 x i32> @test_vloxseg5_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2169,10 +2133,10 @@ define <vscale x 1 x i32> @test_vloxseg5_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2203,10 +2167,10 @@ define <vscale x 1 x i32> @test_vloxseg5_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2237,11 +2201,11 @@ define <vscale x 1 x i32> @test_vloxseg6_mask_nxv1i32_nxv1i64(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2272,11 +2236,11 @@ define <vscale x 1 x i32> @test_vloxseg6_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2307,11 +2271,11 @@ define <vscale x 1 x i32> @test_vloxseg6_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2342,11 +2306,11 @@ define <vscale x 1 x i32> @test_vloxseg6_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2377,12 +2341,12 @@ define <vscale x 1 x i32> @test_vloxseg7_mask_nxv1i32_nxv1i64(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2413,12 +2377,12 @@ define <vscale x 1 x i32> @test_vloxseg7_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2449,12 +2413,12 @@ define <vscale x 1 x i32> @test_vloxseg7_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2485,12 +2449,12 @@ define <vscale x 1 x i32> @test_vloxseg7_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2521,13 +2485,13 @@ define <vscale x 1 x i32> @test_vloxseg8_mask_nxv1i32_nxv1i64(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2558,13 +2522,13 @@ define <vscale x 1 x i32> @test_vloxseg8_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2595,13 +2559,13 @@ define <vscale x 1 x i32> @test_vloxseg8_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2632,13 +2596,13 @@ define <vscale x 1 x i32> @test_vloxseg8_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2671,7 +2635,6 @@ define <vscale x 8 x i16> @test_vloxseg2_mask_nxv8i16_nxv8i16(<vscale x 8 x i16>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vloxseg2.mask.nxv8i16.nxv8i16(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2701,7 +2664,6 @@ define <vscale x 8 x i16> @test_vloxseg2_mask_nxv8i16_nxv8i8(<vscale x 8 x i16>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vloxseg2.mask.nxv8i16.nxv8i8(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2731,7 +2693,6 @@ define <vscale x 8 x i16> @test_vloxseg2_mask_nxv8i16_nxv8i64(<vscale x 8 x i16>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vloxseg2.mask.nxv8i16.nxv8i64(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2761,7 +2722,6 @@ define <vscale x 8 x i16> @test_vloxseg2_mask_nxv8i16_nxv8i32(<vscale x 8 x i16>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vloxseg2.mask.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2788,12 +2748,11 @@ entry:
 define <vscale x 8 x i16> @test_vloxseg3_mask_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vloxseg3.mask.nxv8i16.nxv8i16(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2820,12 +2779,11 @@ entry:
 define <vscale x 8 x i16> @test_vloxseg3_mask_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, i64 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vloxseg3.mask.nxv8i16.nxv8i8(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2853,10 +2811,9 @@ define <vscale x 8 x i16> @test_vloxseg3_mask_nxv8i16_nxv8i64(<vscale x 8 x i16>
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8i16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg3ei64.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vloxseg3.mask.nxv8i16.nxv8i64(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2884,10 +2841,9 @@ define <vscale x 8 x i16> @test_vloxseg3_mask_nxv8i16_nxv8i32(<vscale x 8 x i16>
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg3ei32.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vloxseg3.mask.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2915,9 +2871,9 @@ define <vscale x 8 x i16> @test_vloxseg4_mask_nxv8i16_nxv8i16(<vscale x 8 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -2948,9 +2904,9 @@ define <vscale x 8 x i16> @test_vloxseg4_mask_nxv8i16_nxv8i8(<vscale x 8 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -2981,11 +2937,10 @@ define <vscale x 8 x i16> @test_vloxseg4_mask_nxv8i16_nxv8i64(<vscale x 8 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8i16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei64.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vloxseg4.mask.nxv8i16.nxv8i64(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -3012,13 +2967,12 @@ entry:
 define <vscale x 8 x i16> @test_vloxseg4_mask_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vloxseg4ei32.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v18
+; CHECK-NEXT:    vloxseg4ei32.v v6, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vloxseg4.mask.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -3048,7 +3002,6 @@ define <vscale x 4 x i8> @test_vloxseg2_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg2.mask.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3078,7 +3031,6 @@ define <vscale x 4 x i8> @test_vloxseg2_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg2.mask.nxv4i8.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3108,7 +3060,6 @@ define <vscale x 4 x i8> @test_vloxseg2_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg2.mask.nxv4i8.nxv4i64(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3138,7 +3089,6 @@ define <vscale x 4 x i8> @test_vloxseg2_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg2.mask.nxv4i8.nxv4i16(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3166,10 +3116,9 @@ define <vscale x 4 x i8> @test_vloxseg3_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg3.mask.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3196,12 +3145,11 @@ entry:
 define <vscale x 4 x i8> @test_vloxseg3_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg3.mask.nxv4i8.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3229,10 +3177,9 @@ define <vscale x 4 x i8> @test_vloxseg3_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg3ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg3.mask.nxv4i8.nxv4i64(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3259,12 +3206,11 @@ entry:
 define <vscale x 4 x i8> @test_vloxseg3_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg3.mask.nxv4i8.nxv4i16(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3291,13 +3237,12 @@ entry:
 define <vscale x 4 x i8> @test_vloxseg4_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
-; CHECK-NEXT:    vloxseg4ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vloxseg4ei32.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg4.mask.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3325,9 +3270,9 @@ define <vscale x 4 x i8> @test_vloxseg4_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3358,11 +3303,10 @@ define <vscale x 4 x i8> @test_vloxseg4_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg4.mask.nxv4i8.nxv4i64(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3390,9 +3334,9 @@ define <vscale x 4 x i8> @test_vloxseg4_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3423,10 +3367,10 @@ define <vscale x 4 x i8> @test_vloxseg5_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -3457,10 +3401,10 @@ define <vscale x 4 x i8> @test_vloxseg5_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3491,12 +3435,11 @@ define <vscale x 4 x i8> @test_vloxseg5_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg5.mask.nxv4i8.nxv4i64(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3524,10 +3467,10 @@ define <vscale x 4 x i8> @test_vloxseg5_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3558,11 +3501,11 @@ define <vscale x 4 x i8> @test_vloxseg6_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -3593,11 +3536,11 @@ define <vscale x 4 x i8> @test_vloxseg6_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3627,15 +3570,14 @@ entry:
 define <vscale x 4 x i8> @test_vloxseg6_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
-; CHECK-NEXT:    vloxseg6ei64.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v17
+; CHECK-NEXT:    vloxseg6ei64.v v7, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vloxseg6.mask.nxv4i8.nxv4i64(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3663,11 +3605,11 @@ define <vscale x 4 x i8> @test_vloxseg6_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3698,12 +3640,12 @@ define <vscale x 4 x i8> @test_vloxseg7_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -3734,12 +3676,12 @@ define <vscale x 4 x i8> @test_vloxseg7_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3770,12 +3712,12 @@ define <vscale x 4 x i8> @test_vloxseg7_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -3806,12 +3748,12 @@ define <vscale x 4 x i8> @test_vloxseg7_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3842,13 +3784,13 @@ define <vscale x 4 x i8> @test_vloxseg8_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -3879,13 +3821,13 @@ define <vscale x 4 x i8> @test_vloxseg8_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3916,13 +3858,13 @@ define <vscale x 4 x i8> @test_vloxseg8_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -3953,13 +3895,13 @@ define <vscale x 4 x i8> @test_vloxseg8_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3992,7 +3934,6 @@ define <vscale x 1 x i16> @test_vloxseg2_mask_nxv1i16_nxv1i64(<vscale x 1 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vloxseg2.mask.nxv1i16.nxv1i64(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4022,7 +3963,6 @@ define <vscale x 1 x i16> @test_vloxseg2_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vloxseg2.mask.nxv1i16.nxv1i32(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4052,7 +3992,6 @@ define <vscale x 1 x i16> @test_vloxseg2_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vloxseg2.mask.nxv1i16.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4082,7 +4021,6 @@ define <vscale x 1 x i16> @test_vloxseg2_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vloxseg2.mask.nxv1i16.nxv1i8(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4109,12 +4047,11 @@ entry:
 define <vscale x 1 x i16> @test_vloxseg3_mask_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei64.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei64.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vloxseg3.mask.nxv1i16.nxv1i64(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4141,12 +4078,11 @@ entry:
 define <vscale x 1 x i16> @test_vloxseg3_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vloxseg3.mask.nxv1i16.nxv1i32(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4173,12 +4109,11 @@ entry:
 define <vscale x 1 x i16> @test_vloxseg3_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vloxseg3.mask.nxv1i16.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4205,12 +4140,11 @@ entry:
 define <vscale x 1 x i16> @test_vloxseg3_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vloxseg3.mask.nxv1i16.nxv1i8(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4238,9 +4172,9 @@ define <vscale x 1 x i16> @test_vloxseg4_mask_nxv1i16_nxv1i64(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4271,9 +4205,9 @@ define <vscale x 1 x i16> @test_vloxseg4_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4304,9 +4238,9 @@ define <vscale x 1 x i16> @test_vloxseg4_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4337,9 +4271,9 @@ define <vscale x 1 x i16> @test_vloxseg4_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4370,10 +4304,10 @@ define <vscale x 1 x i16> @test_vloxseg5_mask_nxv1i16_nxv1i64(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4404,10 +4338,10 @@ define <vscale x 1 x i16> @test_vloxseg5_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4438,10 +4372,10 @@ define <vscale x 1 x i16> @test_vloxseg5_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4472,10 +4406,10 @@ define <vscale x 1 x i16> @test_vloxseg5_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4506,11 +4440,11 @@ define <vscale x 1 x i16> @test_vloxseg6_mask_nxv1i16_nxv1i64(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4541,11 +4475,11 @@ define <vscale x 1 x i16> @test_vloxseg6_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4576,11 +4510,11 @@ define <vscale x 1 x i16> @test_vloxseg6_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4611,11 +4545,11 @@ define <vscale x 1 x i16> @test_vloxseg6_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4646,12 +4580,12 @@ define <vscale x 1 x i16> @test_vloxseg7_mask_nxv1i16_nxv1i64(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4682,12 +4616,12 @@ define <vscale x 1 x i16> @test_vloxseg7_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4718,12 +4652,12 @@ define <vscale x 1 x i16> @test_vloxseg7_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4754,12 +4688,12 @@ define <vscale x 1 x i16> @test_vloxseg7_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4790,13 +4724,13 @@ define <vscale x 1 x i16> @test_vloxseg8_mask_nxv1i16_nxv1i64(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4827,13 +4761,13 @@ define <vscale x 1 x i16> @test_vloxseg8_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4864,13 +4798,13 @@ define <vscale x 1 x i16> @test_vloxseg8_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4901,13 +4835,13 @@ define <vscale x 1 x i16> @test_vloxseg8_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4940,7 +4874,6 @@ define <vscale x 2 x i32> @test_vloxseg2_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vloxseg2.mask.nxv2i32.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -4970,7 +4903,6 @@ define <vscale x 2 x i32> @test_vloxseg2_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vloxseg2.mask.nxv2i32.nxv2i8(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5000,7 +4932,6 @@ define <vscale x 2 x i32> @test_vloxseg2_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vloxseg2.mask.nxv2i32.nxv2i16(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5030,7 +4961,6 @@ define <vscale x 2 x i32> @test_vloxseg2_mask_nxv2i32_nxv2i64(<vscale x 2 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vloxseg2.mask.nxv2i32.nxv2i64(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5057,12 +4987,11 @@ entry:
 define <vscale x 2 x i32> @test_vloxseg3_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vloxseg3.mask.nxv2i32.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5089,12 +5018,11 @@ entry:
 define <vscale x 2 x i32> @test_vloxseg3_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vloxseg3.mask.nxv2i32.nxv2i8(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5121,12 +5049,11 @@ entry:
 define <vscale x 2 x i32> @test_vloxseg3_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vloxseg3.mask.nxv2i32.nxv2i16(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5154,10 +5081,9 @@ define <vscale x 2 x i32> @test_vloxseg3_mask_nxv2i32_nxv2i64(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg3ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vloxseg3.mask.nxv2i32.nxv2i64(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5185,9 +5111,9 @@ define <vscale x 2 x i32> @test_vloxseg4_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5218,9 +5144,9 @@ define <vscale x 2 x i32> @test_vloxseg4_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5251,9 +5177,9 @@ define <vscale x 2 x i32> @test_vloxseg4_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5283,13 +5209,12 @@ entry:
 define <vscale x 2 x i32> @test_vloxseg4_mask_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vloxseg4ei64.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vloxseg4ei64.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vloxseg4.mask.nxv2i32.nxv2i64(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5317,10 +5242,10 @@ define <vscale x 2 x i32> @test_vloxseg5_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5351,10 +5276,10 @@ define <vscale x 2 x i32> @test_vloxseg5_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5385,10 +5310,10 @@ define <vscale x 2 x i32> @test_vloxseg5_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5419,10 +5344,10 @@ define <vscale x 2 x i32> @test_vloxseg5_mask_nxv2i32_nxv2i64(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -5453,11 +5378,11 @@ define <vscale x 2 x i32> @test_vloxseg6_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5488,11 +5413,11 @@ define <vscale x 2 x i32> @test_vloxseg6_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5523,11 +5448,11 @@ define <vscale x 2 x i32> @test_vloxseg6_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5558,11 +5483,11 @@ define <vscale x 2 x i32> @test_vloxseg6_mask_nxv2i32_nxv2i64(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -5593,12 +5518,12 @@ define <vscale x 2 x i32> @test_vloxseg7_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5629,12 +5554,12 @@ define <vscale x 2 x i32> @test_vloxseg7_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5665,12 +5590,12 @@ define <vscale x 2 x i32> @test_vloxseg7_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5701,12 +5626,12 @@ define <vscale x 2 x i32> @test_vloxseg7_mask_nxv2i32_nxv2i64(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -5737,13 +5662,13 @@ define <vscale x 2 x i32> @test_vloxseg8_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5774,13 +5699,13 @@ define <vscale x 2 x i32> @test_vloxseg8_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5811,13 +5736,13 @@ define <vscale x 2 x i32> @test_vloxseg8_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5848,13 +5773,13 @@ define <vscale x 2 x i32> @test_vloxseg8_mask_nxv2i32_nxv2i64(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -5887,7 +5812,6 @@ define <vscale x 8 x i8> @test_vloxseg2_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg2.mask.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -5917,7 +5841,6 @@ define <vscale x 8 x i8> @test_vloxseg2_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg2.mask.nxv8i8.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -5947,7 +5870,6 @@ define <vscale x 8 x i8> @test_vloxseg2_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v7, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg2.mask.nxv8i8.nxv8i64(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -5977,7 +5899,6 @@ define <vscale x 8 x i8> @test_vloxseg2_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg2.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6005,10 +5926,9 @@ define <vscale x 8 x i8> @test_vloxseg3_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg3.mask.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6035,12 +5955,11 @@ entry:
 define <vscale x 8 x i8> @test_vloxseg3_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, i64 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg3.mask.nxv8i8.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6068,10 +5987,9 @@ define <vscale x 8 x i8> @test_vloxseg3_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg3ei64.v v7, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg3.mask.nxv8i8.nxv8i64(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6099,10 +6017,9 @@ define <vscale x 8 x i8> @test_vloxseg3_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg3.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6129,13 +6046,12 @@ entry:
 define <vscale x 8 x i8> @test_vloxseg4_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
-; CHECK-NEXT:    vloxseg4ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vloxseg4ei16.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg4.mask.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6163,9 +6079,9 @@ define <vscale x 8 x i8> @test_vloxseg4_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6196,11 +6112,10 @@ define <vscale x 8 x i8> @test_vloxseg4_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei64.v v7, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg4.mask.nxv8i8.nxv8i64(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6228,11 +6143,10 @@ define <vscale x 8 x i8> @test_vloxseg4_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg4.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6260,10 +6174,10 @@ define <vscale x 8 x i8> @test_vloxseg5_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -6294,10 +6208,10 @@ define <vscale x 8 x i8> @test_vloxseg5_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vloxseg5_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6328,12 +6242,11 @@ define <vscale x 8 x i8> @test_vloxseg5_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei64.v v7, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg5.mask.nxv8i8.nxv8i64(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6361,12 +6274,11 @@ define <vscale x 8 x i8> @test_vloxseg5_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg5.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6394,11 +6306,11 @@ define <vscale x 8 x i8> @test_vloxseg6_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg6_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -6429,11 +6341,11 @@ define <vscale x 8 x i8> @test_vloxseg6_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vloxseg6_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6464,13 +6376,12 @@ define <vscale x 8 x i8> @test_vloxseg6_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg6_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei64.v v7, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg6.mask.nxv8i8.nxv8i64(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6497,15 +6408,14 @@ entry:
 define <vscale x 8 x i8> @test_vloxseg6_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg6_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
-; CHECK-NEXT:    vloxseg6ei32.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v17
+; CHECK-NEXT:    vloxseg6ei32.v v7, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg6.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6533,12 +6443,12 @@ define <vscale x 8 x i8> @test_vloxseg7_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -6569,12 +6479,12 @@ define <vscale x 8 x i8> @test_vloxseg7_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vloxseg7_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6605,14 +6515,13 @@ define <vscale x 8 x i8> @test_vloxseg7_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei64.v v7, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg7.mask.nxv8i8.nxv8i64(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6640,12 +6549,12 @@ define <vscale x 8 x i8> @test_vloxseg7_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -6676,13 +6585,13 @@ define <vscale x 8 x i8> @test_vloxseg8_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -6713,13 +6622,13 @@ define <vscale x 8 x i8> @test_vloxseg8_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vloxseg8_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6750,15 +6659,14 @@ define <vscale x 8 x i8> @test_vloxseg8_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei64.v v7, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vloxseg8.mask.nxv8i8.nxv8i64(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6786,13 +6694,13 @@ define <vscale x 8 x i8> @test_vloxseg8_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -6825,7 +6733,6 @@ define <vscale x 4 x i64> @test_vloxseg2_mask_nxv4i64_nxv4i32(<vscale x 4 x i64>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i64>,<vscale x 4 x i64>} @llvm.riscv.vloxseg2.mask.nxv4i64.nxv4i32(<vscale x 4 x i64> %val,<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -6855,7 +6762,6 @@ define <vscale x 4 x i64> @test_vloxseg2_mask_nxv4i64_nxv4i8(<vscale x 4 x i64>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i64>,<vscale x 4 x i64>} @llvm.riscv.vloxseg2.mask.nxv4i64.nxv4i8(<vscale x 4 x i64> %val,<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -6885,7 +6791,6 @@ define <vscale x 4 x i64> @test_vloxseg2_mask_nxv4i64_nxv4i64(<vscale x 4 x i64>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i64>,<vscale x 4 x i64>} @llvm.riscv.vloxseg2.mask.nxv4i64.nxv4i64(<vscale x 4 x i64> %val,<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -6915,7 +6820,6 @@ define <vscale x 4 x i64> @test_vloxseg2_mask_nxv4i64_nxv4i16(<vscale x 4 x i64>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i64>,<vscale x 4 x i64>} @llvm.riscv.vloxseg2.mask.nxv4i64.nxv4i16(<vscale x 4 x i64> %val,<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -6945,7 +6849,6 @@ define <vscale x 4 x i16> @test_vloxseg2_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg2.mask.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -6975,7 +6878,6 @@ define <vscale x 4 x i16> @test_vloxseg2_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg2.mask.nxv4i16.nxv4i8(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7005,7 +6907,6 @@ define <vscale x 4 x i16> @test_vloxseg2_mask_nxv4i16_nxv4i64(<vscale x 4 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg2.mask.nxv4i16.nxv4i64(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7035,7 +6936,6 @@ define <vscale x 4 x i16> @test_vloxseg2_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg2.mask.nxv4i16.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7063,10 +6963,9 @@ define <vscale x 4 x i16> @test_vloxseg3_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg3.mask.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7093,12 +6992,11 @@ entry:
 define <vscale x 4 x i16> @test_vloxseg3_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg3.mask.nxv4i16.nxv4i8(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7126,10 +7024,9 @@ define <vscale x 4 x i16> @test_vloxseg3_mask_nxv4i16_nxv4i64(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg3ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg3.mask.nxv4i16.nxv4i64(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7156,12 +7053,11 @@ entry:
 define <vscale x 4 x i16> @test_vloxseg3_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg3.mask.nxv4i16.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7188,13 +7084,12 @@ entry:
 define <vscale x 4 x i16> @test_vloxseg4_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vloxseg4ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vloxseg4ei32.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg4.mask.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7222,9 +7117,9 @@ define <vscale x 4 x i16> @test_vloxseg4_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7255,11 +7150,10 @@ define <vscale x 4 x i16> @test_vloxseg4_mask_nxv4i16_nxv4i64(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg4.mask.nxv4i16.nxv4i64(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7287,9 +7181,9 @@ define <vscale x 4 x i16> @test_vloxseg4_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7320,10 +7214,10 @@ define <vscale x 4 x i16> @test_vloxseg5_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -7354,10 +7248,10 @@ define <vscale x 4 x i16> @test_vloxseg5_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7388,12 +7282,11 @@ define <vscale x 4 x i16> @test_vloxseg5_mask_nxv4i16_nxv4i64(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg5.mask.nxv4i16.nxv4i64(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7421,10 +7314,10 @@ define <vscale x 4 x i16> @test_vloxseg5_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7455,11 +7348,11 @@ define <vscale x 4 x i16> @test_vloxseg6_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -7490,11 +7383,11 @@ define <vscale x 4 x i16> @test_vloxseg6_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7524,15 +7417,14 @@ entry:
 define <vscale x 4 x i16> @test_vloxseg6_mask_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vloxseg6ei64.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v17
+; CHECK-NEXT:    vloxseg6ei64.v v7, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vloxseg6.mask.nxv4i16.nxv4i64(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7560,11 +7452,11 @@ define <vscale x 4 x i16> @test_vloxseg6_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7595,12 +7487,12 @@ define <vscale x 4 x i16> @test_vloxseg7_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -7631,12 +7523,12 @@ define <vscale x 4 x i16> @test_vloxseg7_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7667,12 +7559,12 @@ define <vscale x 4 x i16> @test_vloxseg7_mask_nxv4i16_nxv4i64(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -7703,12 +7595,12 @@ define <vscale x 4 x i16> @test_vloxseg7_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7739,13 +7631,13 @@ define <vscale x 4 x i16> @test_vloxseg8_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -7776,13 +7668,13 @@ define <vscale x 4 x i16> @test_vloxseg8_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7813,13 +7705,13 @@ define <vscale x 4 x i16> @test_vloxseg8_mask_nxv4i16_nxv4i64(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -7850,13 +7742,13 @@ define <vscale x 4 x i16> @test_vloxseg8_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7889,7 +7781,6 @@ define <vscale x 1 x i8> @test_vloxseg2_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vloxseg2.mask.nxv1i8.nxv1i64(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -7919,7 +7810,6 @@ define <vscale x 1 x i8> @test_vloxseg2_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vloxseg2.mask.nxv1i8.nxv1i32(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -7949,7 +7839,6 @@ define <vscale x 1 x i8> @test_vloxseg2_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vloxseg2.mask.nxv1i8.nxv1i16(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -7979,7 +7868,6 @@ define <vscale x 1 x i8> @test_vloxseg2_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vloxseg2.mask.nxv1i8.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -8006,12 +7894,11 @@ entry:
 define <vscale x 1 x i8> @test_vloxseg3_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
-; CHECK-NEXT:    vloxseg3ei64.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei64.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vloxseg3.mask.nxv1i8.nxv1i64(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -8038,12 +7925,11 @@ entry:
 define <vscale x 1 x i8> @test_vloxseg3_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vloxseg3.mask.nxv1i8.nxv1i32(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -8070,12 +7956,11 @@ entry:
 define <vscale x 1 x i8> @test_vloxseg3_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vloxseg3.mask.nxv1i8.nxv1i16(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -8102,12 +7987,11 @@ entry:
 define <vscale x 1 x i8> @test_vloxseg3_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vloxseg3.mask.nxv1i8.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -8135,9 +8019,9 @@ define <vscale x 1 x i8> @test_vloxseg4_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8168,9 +8052,9 @@ define <vscale x 1 x i8> @test_vloxseg4_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8201,9 +8085,9 @@ define <vscale x 1 x i8> @test_vloxseg4_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8234,9 +8118,9 @@ define <vscale x 1 x i8> @test_vloxseg4_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8267,10 +8151,10 @@ define <vscale x 1 x i8> @test_vloxseg5_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8301,10 +8185,10 @@ define <vscale x 1 x i8> @test_vloxseg5_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8335,10 +8219,10 @@ define <vscale x 1 x i8> @test_vloxseg5_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8369,10 +8253,10 @@ define <vscale x 1 x i8> @test_vloxseg5_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8403,11 +8287,11 @@ define <vscale x 1 x i8> @test_vloxseg6_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8438,11 +8322,11 @@ define <vscale x 1 x i8> @test_vloxseg6_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8473,11 +8357,11 @@ define <vscale x 1 x i8> @test_vloxseg6_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8508,11 +8392,11 @@ define <vscale x 1 x i8> @test_vloxseg6_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8543,12 +8427,12 @@ define <vscale x 1 x i8> @test_vloxseg7_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8579,12 +8463,12 @@ define <vscale x 1 x i8> @test_vloxseg7_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8615,12 +8499,12 @@ define <vscale x 1 x i8> @test_vloxseg7_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8651,12 +8535,12 @@ define <vscale x 1 x i8> @test_vloxseg7_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8687,13 +8571,13 @@ define <vscale x 1 x i8> @test_vloxseg8_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8724,13 +8608,13 @@ define <vscale x 1 x i8> @test_vloxseg8_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8761,14 +8645,14 @@ define <vscale x 1 x i8> @test_vloxseg8_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
-; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
 ; CHECK-NEXT:    ret
@@ -8798,13 +8682,13 @@ define <vscale x 1 x i8> @test_vloxseg8_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8837,7 +8721,6 @@ define <vscale x 2 x i8> @test_vloxseg2_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vloxseg2.mask.nxv2i8.nxv2i32(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -8867,7 +8750,6 @@ define <vscale x 2 x i8> @test_vloxseg2_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vloxseg2.mask.nxv2i8.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -8897,7 +8779,6 @@ define <vscale x 2 x i8> @test_vloxseg2_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vloxseg2.mask.nxv2i8.nxv2i16(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -8927,7 +8808,6 @@ define <vscale x 2 x i8> @test_vloxseg2_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vloxseg2.mask.nxv2i8.nxv2i64(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -8954,12 +8834,11 @@ entry:
 define <vscale x 2 x i8> @test_vloxseg3_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vloxseg3.mask.nxv2i8.nxv2i32(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -8986,12 +8865,11 @@ entry:
 define <vscale x 2 x i8> @test_vloxseg3_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vloxseg3.mask.nxv2i8.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -9018,12 +8896,11 @@ entry:
 define <vscale x 2 x i8> @test_vloxseg3_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vloxseg3.mask.nxv2i8.nxv2i16(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -9051,10 +8928,9 @@ define <vscale x 2 x i8> @test_vloxseg3_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg3ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vloxseg3.mask.nxv2i8.nxv2i64(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -9082,9 +8958,9 @@ define <vscale x 2 x i8> @test_vloxseg4_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9115,9 +8991,9 @@ define <vscale x 2 x i8> @test_vloxseg4_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9148,9 +9024,9 @@ define <vscale x 2 x i8> @test_vloxseg4_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9180,13 +9056,12 @@ entry:
 define <vscale x 2 x i8> @test_vloxseg4_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
-; CHECK-NEXT:    vloxseg4ei64.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vloxseg4ei64.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vloxseg4.mask.nxv2i8.nxv2i64(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -9214,10 +9089,10 @@ define <vscale x 2 x i8> @test_vloxseg5_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9248,10 +9123,10 @@ define <vscale x 2 x i8> @test_vloxseg5_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9282,10 +9157,10 @@ define <vscale x 2 x i8> @test_vloxseg5_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9316,10 +9191,10 @@ define <vscale x 2 x i8> @test_vloxseg5_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -9350,11 +9225,11 @@ define <vscale x 2 x i8> @test_vloxseg6_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9385,11 +9260,11 @@ define <vscale x 2 x i8> @test_vloxseg6_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9420,11 +9295,11 @@ define <vscale x 2 x i8> @test_vloxseg6_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9455,11 +9330,11 @@ define <vscale x 2 x i8> @test_vloxseg6_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -9490,12 +9365,12 @@ define <vscale x 2 x i8> @test_vloxseg7_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9526,12 +9401,12 @@ define <vscale x 2 x i8> @test_vloxseg7_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9562,12 +9437,12 @@ define <vscale x 2 x i8> @test_vloxseg7_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9598,12 +9473,12 @@ define <vscale x 2 x i8> @test_vloxseg7_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -9634,13 +9509,13 @@ define <vscale x 2 x i8> @test_vloxseg8_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9671,13 +9546,13 @@ define <vscale x 2 x i8> @test_vloxseg8_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9708,13 +9583,13 @@ define <vscale x 2 x i8> @test_vloxseg8_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9745,13 +9620,13 @@ define <vscale x 2 x i8> @test_vloxseg8_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -9784,7 +9659,6 @@ define <vscale x 8 x i32> @test_vloxseg2_mask_nxv8i32_nxv8i16(<vscale x 8 x i32>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vloxseg2.mask.nxv8i32.nxv8i16(<vscale x 8 x i32> %val,<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -9814,7 +9688,6 @@ define <vscale x 8 x i32> @test_vloxseg2_mask_nxv8i32_nxv8i8(<vscale x 8 x i32>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vloxseg2.mask.nxv8i32.nxv8i8(<vscale x 8 x i32> %val,<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -9844,7 +9717,6 @@ define <vscale x 8 x i32> @test_vloxseg2_mask_nxv8i32_nxv8i64(<vscale x 8 x i32>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v4, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vloxseg2.mask.nxv8i32.nxv8i64(<vscale x 8 x i32> %val,<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -9874,7 +9746,6 @@ define <vscale x 8 x i32> @test_vloxseg2_mask_nxv8i32_nxv8i32(<vscale x 8 x i32>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vloxseg2.mask.nxv8i32.nxv8i32(<vscale x 8 x i32> %val,<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -9904,7 +9775,6 @@ define <vscale x 32 x i8> @test_vloxseg2_mask_nxv32i8_nxv32i16(<vscale x 32 x i8
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v4, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>} @llvm.riscv.vloxseg2.mask.nxv32i8.nxv32i16(<vscale x 32 x i8> %val,<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i16> %index, <vscale x 32 x i1> %mask, i64 %vl, i64 1)
@@ -9934,7 +9804,6 @@ define <vscale x 32 x i8> @test_vloxseg2_mask_nxv32i8_nxv32i8(<vscale x 32 x i8>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>} @llvm.riscv.vloxseg2.mask.nxv32i8.nxv32i8(<vscale x 32 x i8> %val,<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i8> %index, <vscale x 32 x i1> %mask, i64 %vl, i64 1)
@@ -9964,7 +9833,6 @@ define <vscale x 2 x i16> @test_vloxseg2_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vloxseg2.mask.nxv2i16.nxv2i32(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -9994,7 +9862,6 @@ define <vscale x 2 x i16> @test_vloxseg2_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vloxseg2.mask.nxv2i16.nxv2i8(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10024,7 +9891,6 @@ define <vscale x 2 x i16> @test_vloxseg2_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vloxseg2.mask.nxv2i16.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10054,7 +9920,6 @@ define <vscale x 2 x i16> @test_vloxseg2_mask_nxv2i16_nxv2i64(<vscale x 2 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vloxseg2.mask.nxv2i16.nxv2i64(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10081,12 +9946,11 @@ entry:
 define <vscale x 2 x i16> @test_vloxseg3_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vloxseg3.mask.nxv2i16.nxv2i32(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10113,12 +9977,11 @@ entry:
 define <vscale x 2 x i16> @test_vloxseg3_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vloxseg3.mask.nxv2i16.nxv2i8(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10145,12 +10008,11 @@ entry:
 define <vscale x 2 x i16> @test_vloxseg3_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vloxseg3.mask.nxv2i16.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10178,10 +10040,9 @@ define <vscale x 2 x i16> @test_vloxseg3_mask_nxv2i16_nxv2i64(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg3ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vloxseg3.mask.nxv2i16.nxv2i64(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10209,9 +10070,9 @@ define <vscale x 2 x i16> @test_vloxseg4_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10242,9 +10103,9 @@ define <vscale x 2 x i16> @test_vloxseg4_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10275,9 +10136,9 @@ define <vscale x 2 x i16> @test_vloxseg4_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10307,13 +10168,12 @@ entry:
 define <vscale x 2 x i16> @test_vloxseg4_mask_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vloxseg4ei64.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vloxseg4ei64.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vloxseg4.mask.nxv2i16.nxv2i64(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10341,10 +10201,10 @@ define <vscale x 2 x i16> @test_vloxseg5_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10375,10 +10235,10 @@ define <vscale x 2 x i16> @test_vloxseg5_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10409,10 +10269,10 @@ define <vscale x 2 x i16> @test_vloxseg5_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10443,10 +10303,10 @@ define <vscale x 2 x i16> @test_vloxseg5_mask_nxv2i16_nxv2i64(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -10477,11 +10337,11 @@ define <vscale x 2 x i16> @test_vloxseg6_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10512,11 +10372,11 @@ define <vscale x 2 x i16> @test_vloxseg6_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10547,11 +10407,11 @@ define <vscale x 2 x i16> @test_vloxseg6_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10582,11 +10442,11 @@ define <vscale x 2 x i16> @test_vloxseg6_mask_nxv2i16_nxv2i64(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -10617,12 +10477,12 @@ define <vscale x 2 x i16> @test_vloxseg7_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10653,12 +10513,12 @@ define <vscale x 2 x i16> @test_vloxseg7_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10689,12 +10549,12 @@ define <vscale x 2 x i16> @test_vloxseg7_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10725,12 +10585,12 @@ define <vscale x 2 x i16> @test_vloxseg7_mask_nxv2i16_nxv2i64(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -10761,13 +10621,13 @@ define <vscale x 2 x i16> @test_vloxseg8_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10798,13 +10658,13 @@ define <vscale x 2 x i16> @test_vloxseg8_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10835,13 +10695,13 @@ define <vscale x 2 x i16> @test_vloxseg8_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10872,13 +10732,13 @@ define <vscale x 2 x i16> @test_vloxseg8_mask_nxv2i16_nxv2i64(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -10911,7 +10771,6 @@ define <vscale x 2 x i64> @test_vloxseg2_mask_nxv2i64_nxv2i32(<vscale x 2 x i64>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vloxseg2.mask.nxv2i64.nxv2i32(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10941,7 +10800,6 @@ define <vscale x 2 x i64> @test_vloxseg2_mask_nxv2i64_nxv2i8(<vscale x 2 x i64>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vloxseg2.mask.nxv2i64.nxv2i8(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10971,7 +10829,6 @@ define <vscale x 2 x i64> @test_vloxseg2_mask_nxv2i64_nxv2i16(<vscale x 2 x i64>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vloxseg2.mask.nxv2i64.nxv2i16(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -11001,7 +10858,6 @@ define <vscale x 2 x i64> @test_vloxseg2_mask_nxv2i64_nxv2i64(<vscale x 2 x i64>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vloxseg2.mask.nxv2i64.nxv2i64(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -11028,12 +10884,11 @@ entry:
 define <vscale x 2 x i64> @test_vloxseg3_mask_nxv2i64_nxv2i32(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei32.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vloxseg3.mask.nxv2i64.nxv2i32(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -11060,12 +10915,11 @@ entry:
 define <vscale x 2 x i64> @test_vloxseg3_mask_nxv2i64_nxv2i8(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vloxseg3.mask.nxv2i64.nxv2i8(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -11092,12 +10946,11 @@ entry:
 define <vscale x 2 x i64> @test_vloxseg3_mask_nxv2i64_nxv2i16(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vloxseg3.mask.nxv2i64.nxv2i16(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -11124,12 +10977,11 @@ entry:
 define <vscale x 2 x i64> @test_vloxseg3_mask_nxv2i64_nxv2i64(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2i64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei64.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei64.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vloxseg3.mask.nxv2i64.nxv2i64(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -11157,9 +11009,9 @@ define <vscale x 2 x i64> @test_vloxseg4_mask_nxv2i64_nxv2i32(<vscale x 2 x i64>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -11190,9 +11042,9 @@ define <vscale x 2 x i64> @test_vloxseg4_mask_nxv2i64_nxv2i8(<vscale x 2 x i64>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -11223,9 +11075,9 @@ define <vscale x 2 x i64> @test_vloxseg4_mask_nxv2i64_nxv2i16(<vscale x 2 x i64>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -11256,9 +11108,9 @@ define <vscale x 2 x i64> @test_vloxseg4_mask_nxv2i64_nxv2i64(<vscale x 2 x i64>
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2i64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -11291,7 +11143,6 @@ define <vscale x 16 x half> @test_vloxseg2_mask_nxv16f16_nxv16i16(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vloxseg2.mask.nxv16f16.nxv16i16(<vscale x 16 x half> %val,<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -11321,7 +11172,6 @@ define <vscale x 16 x half> @test_vloxseg2_mask_nxv16f16_nxv16i8(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vloxseg2.mask.nxv16f16.nxv16i8(<vscale x 16 x half> %val,<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -11351,7 +11201,6 @@ define <vscale x 16 x half> @test_vloxseg2_mask_nxv16f16_nxv16i32(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v4, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vloxseg2.mask.nxv16f16.nxv16i32(<vscale x 16 x half> %val,<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -11381,7 +11230,6 @@ define <vscale x 4 x double> @test_vloxseg2_mask_nxv4f64_nxv4i32(<vscale x 4 x d
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vloxseg2.mask.nxv4f64.nxv4i32(<vscale x 4 x double> %val,<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -11411,7 +11259,6 @@ define <vscale x 4 x double> @test_vloxseg2_mask_nxv4f64_nxv4i8(<vscale x 4 x do
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vloxseg2.mask.nxv4f64.nxv4i8(<vscale x 4 x double> %val,<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -11441,7 +11288,6 @@ define <vscale x 4 x double> @test_vloxseg2_mask_nxv4f64_nxv4i64(<vscale x 4 x d
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vloxseg2.mask.nxv4f64.nxv4i64(<vscale x 4 x double> %val,<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -11471,7 +11317,6 @@ define <vscale x 4 x double> @test_vloxseg2_mask_nxv4f64_nxv4i16(<vscale x 4 x d
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vloxseg2.mask.nxv4f64.nxv4i16(<vscale x 4 x double> %val,<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -11501,7 +11346,6 @@ define <vscale x 1 x double> @test_vloxseg2_mask_nxv1f64_nxv1i64(<vscale x 1 x d
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vloxseg2.mask.nxv1f64.nxv1i64(<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -11531,7 +11375,6 @@ define <vscale x 1 x double> @test_vloxseg2_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vloxseg2.mask.nxv1f64.nxv1i32(<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -11561,7 +11404,6 @@ define <vscale x 1 x double> @test_vloxseg2_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vloxseg2.mask.nxv1f64.nxv1i16(<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -11591,7 +11433,6 @@ define <vscale x 1 x double> @test_vloxseg2_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vloxseg2.mask.nxv1f64.nxv1i8(<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -11618,12 +11459,11 @@ entry:
 define <vscale x 1 x double> @test_vloxseg3_mask_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei64.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei64.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vloxseg3.mask.nxv1f64.nxv1i64(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -11650,12 +11490,11 @@ entry:
 define <vscale x 1 x double> @test_vloxseg3_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vloxseg3.mask.nxv1f64.nxv1i32(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -11682,12 +11521,11 @@ entry:
 define <vscale x 1 x double> @test_vloxseg3_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vloxseg3.mask.nxv1f64.nxv1i16(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -11714,12 +11552,11 @@ entry:
 define <vscale x 1 x double> @test_vloxseg3_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vloxseg3.mask.nxv1f64.nxv1i8(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -11747,9 +11584,9 @@ define <vscale x 1 x double> @test_vloxseg4_mask_nxv1f64_nxv1i64(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11780,9 +11617,9 @@ define <vscale x 1 x double> @test_vloxseg4_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11813,9 +11650,9 @@ define <vscale x 1 x double> @test_vloxseg4_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11846,9 +11683,9 @@ define <vscale x 1 x double> @test_vloxseg4_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11879,10 +11716,10 @@ define <vscale x 1 x double> @test_vloxseg5_mask_nxv1f64_nxv1i64(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11913,10 +11750,10 @@ define <vscale x 1 x double> @test_vloxseg5_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11947,10 +11784,10 @@ define <vscale x 1 x double> @test_vloxseg5_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11981,10 +11818,10 @@ define <vscale x 1 x double> @test_vloxseg5_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12015,11 +11852,11 @@ define <vscale x 1 x double> @test_vloxseg6_mask_nxv1f64_nxv1i64(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12050,11 +11887,11 @@ define <vscale x 1 x double> @test_vloxseg6_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12085,11 +11922,11 @@ define <vscale x 1 x double> @test_vloxseg6_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12120,11 +11957,11 @@ define <vscale x 1 x double> @test_vloxseg6_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12155,12 +11992,12 @@ define <vscale x 1 x double> @test_vloxseg7_mask_nxv1f64_nxv1i64(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12191,12 +12028,12 @@ define <vscale x 1 x double> @test_vloxseg7_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12227,12 +12064,12 @@ define <vscale x 1 x double> @test_vloxseg7_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12263,12 +12100,12 @@ define <vscale x 1 x double> @test_vloxseg7_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12299,13 +12136,13 @@ define <vscale x 1 x double> @test_vloxseg8_mask_nxv1f64_nxv1i64(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12336,13 +12173,13 @@ define <vscale x 1 x double> @test_vloxseg8_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12373,13 +12210,13 @@ define <vscale x 1 x double> @test_vloxseg8_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12410,13 +12247,13 @@ define <vscale x 1 x double> @test_vloxseg8_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12449,7 +12286,6 @@ define <vscale x 2 x float> @test_vloxseg2_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vloxseg2.mask.nxv2f32.nxv2i32(<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -12479,7 +12315,6 @@ define <vscale x 2 x float> @test_vloxseg2_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vloxseg2.mask.nxv2f32.nxv2i8(<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -12509,7 +12344,6 @@ define <vscale x 2 x float> @test_vloxseg2_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vloxseg2.mask.nxv2f32.nxv2i16(<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -12539,7 +12373,6 @@ define <vscale x 2 x float> @test_vloxseg2_mask_nxv2f32_nxv2i64(<vscale x 2 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vloxseg2.mask.nxv2f32.nxv2i64(<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -12566,12 +12399,11 @@ entry:
 define <vscale x 2 x float> @test_vloxseg3_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vloxseg3.mask.nxv2f32.nxv2i32(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -12598,12 +12430,11 @@ entry:
 define <vscale x 2 x float> @test_vloxseg3_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vloxseg3.mask.nxv2f32.nxv2i8(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -12630,12 +12461,11 @@ entry:
 define <vscale x 2 x float> @test_vloxseg3_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vloxseg3.mask.nxv2f32.nxv2i16(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -12663,10 +12493,9 @@ define <vscale x 2 x float> @test_vloxseg3_mask_nxv2f32_nxv2i64(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg3ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vloxseg3.mask.nxv2f32.nxv2i64(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -12694,9 +12523,9 @@ define <vscale x 2 x float> @test_vloxseg4_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12727,9 +12556,9 @@ define <vscale x 2 x float> @test_vloxseg4_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12760,9 +12589,9 @@ define <vscale x 2 x float> @test_vloxseg4_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12792,13 +12621,12 @@ entry:
 define <vscale x 2 x float> @test_vloxseg4_mask_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vloxseg4ei64.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vloxseg4ei64.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vloxseg4.mask.nxv2f32.nxv2i64(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -12826,10 +12654,10 @@ define <vscale x 2 x float> @test_vloxseg5_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12860,10 +12688,10 @@ define <vscale x 2 x float> @test_vloxseg5_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12894,10 +12722,10 @@ define <vscale x 2 x float> @test_vloxseg5_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12928,10 +12756,10 @@ define <vscale x 2 x float> @test_vloxseg5_mask_nxv2f32_nxv2i64(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -12962,11 +12790,11 @@ define <vscale x 2 x float> @test_vloxseg6_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12997,11 +12825,11 @@ define <vscale x 2 x float> @test_vloxseg6_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13032,11 +12860,11 @@ define <vscale x 2 x float> @test_vloxseg6_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13067,11 +12895,11 @@ define <vscale x 2 x float> @test_vloxseg6_mask_nxv2f32_nxv2i64(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -13102,12 +12930,12 @@ define <vscale x 2 x float> @test_vloxseg7_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13138,12 +12966,12 @@ define <vscale x 2 x float> @test_vloxseg7_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13174,12 +13002,12 @@ define <vscale x 2 x float> @test_vloxseg7_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13210,12 +13038,12 @@ define <vscale x 2 x float> @test_vloxseg7_mask_nxv2f32_nxv2i64(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -13246,13 +13074,13 @@ define <vscale x 2 x float> @test_vloxseg8_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13283,13 +13111,13 @@ define <vscale x 2 x float> @test_vloxseg8_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13320,13 +13148,13 @@ define <vscale x 2 x float> @test_vloxseg8_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13357,13 +13185,13 @@ define <vscale x 2 x float> @test_vloxseg8_mask_nxv2f32_nxv2i64(<vscale x 2 x fl
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -13396,7 +13224,6 @@ define <vscale x 1 x half> @test_vloxseg2_mask_nxv1f16_nxv1i64(<vscale x 1 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vloxseg2.mask.nxv1f16.nxv1i64(<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -13426,7 +13253,6 @@ define <vscale x 1 x half> @test_vloxseg2_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vloxseg2.mask.nxv1f16.nxv1i32(<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -13456,7 +13282,6 @@ define <vscale x 1 x half> @test_vloxseg2_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vloxseg2.mask.nxv1f16.nxv1i16(<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -13486,7 +13311,6 @@ define <vscale x 1 x half> @test_vloxseg2_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vloxseg2.mask.nxv1f16.nxv1i8(<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -13513,12 +13337,11 @@ entry:
 define <vscale x 1 x half> @test_vloxseg3_mask_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei64.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei64.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vloxseg3.mask.nxv1f16.nxv1i64(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -13545,12 +13368,11 @@ entry:
 define <vscale x 1 x half> @test_vloxseg3_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vloxseg3.mask.nxv1f16.nxv1i32(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -13577,12 +13399,11 @@ entry:
 define <vscale x 1 x half> @test_vloxseg3_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vloxseg3.mask.nxv1f16.nxv1i16(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -13609,12 +13430,11 @@ entry:
 define <vscale x 1 x half> @test_vloxseg3_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vloxseg3.mask.nxv1f16.nxv1i8(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -13642,9 +13462,9 @@ define <vscale x 1 x half> @test_vloxseg4_mask_nxv1f16_nxv1i64(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13675,9 +13495,9 @@ define <vscale x 1 x half> @test_vloxseg4_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13708,9 +13528,9 @@ define <vscale x 1 x half> @test_vloxseg4_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13741,9 +13561,9 @@ define <vscale x 1 x half> @test_vloxseg4_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13774,10 +13594,10 @@ define <vscale x 1 x half> @test_vloxseg5_mask_nxv1f16_nxv1i64(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13808,10 +13628,10 @@ define <vscale x 1 x half> @test_vloxseg5_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13842,10 +13662,10 @@ define <vscale x 1 x half> @test_vloxseg5_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13876,10 +13696,10 @@ define <vscale x 1 x half> @test_vloxseg5_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13910,11 +13730,11 @@ define <vscale x 1 x half> @test_vloxseg6_mask_nxv1f16_nxv1i64(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13945,11 +13765,11 @@ define <vscale x 1 x half> @test_vloxseg6_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13980,11 +13800,11 @@ define <vscale x 1 x half> @test_vloxseg6_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14015,11 +13835,11 @@ define <vscale x 1 x half> @test_vloxseg6_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14050,12 +13870,12 @@ define <vscale x 1 x half> @test_vloxseg7_mask_nxv1f16_nxv1i64(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14086,12 +13906,12 @@ define <vscale x 1 x half> @test_vloxseg7_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14122,12 +13942,12 @@ define <vscale x 1 x half> @test_vloxseg7_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14158,12 +13978,12 @@ define <vscale x 1 x half> @test_vloxseg7_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14194,13 +14014,13 @@ define <vscale x 1 x half> @test_vloxseg8_mask_nxv1f16_nxv1i64(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14231,13 +14051,13 @@ define <vscale x 1 x half> @test_vloxseg8_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14268,13 +14088,13 @@ define <vscale x 1 x half> @test_vloxseg8_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14305,13 +14125,13 @@ define <vscale x 1 x half> @test_vloxseg8_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14344,7 +14164,6 @@ define <vscale x 1 x float> @test_vloxseg2_mask_nxv1f32_nxv1i64(<vscale x 1 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vloxseg2.mask.nxv1f32.nxv1i64(<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -14374,7 +14193,6 @@ define <vscale x 1 x float> @test_vloxseg2_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vloxseg2.mask.nxv1f32.nxv1i32(<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -14404,7 +14222,6 @@ define <vscale x 1 x float> @test_vloxseg2_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vloxseg2.mask.nxv1f32.nxv1i16(<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -14434,7 +14251,6 @@ define <vscale x 1 x float> @test_vloxseg2_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vloxseg2.mask.nxv1f32.nxv1i8(<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -14461,12 +14277,11 @@ entry:
 define <vscale x 1 x float> @test_vloxseg3_mask_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei64.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei64.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vloxseg3.mask.nxv1f32.nxv1i64(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -14493,12 +14308,11 @@ entry:
 define <vscale x 1 x float> @test_vloxseg3_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vloxseg3.mask.nxv1f32.nxv1i32(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -14525,12 +14339,11 @@ entry:
 define <vscale x 1 x float> @test_vloxseg3_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vloxseg3.mask.nxv1f32.nxv1i16(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -14557,12 +14370,11 @@ entry:
 define <vscale x 1 x float> @test_vloxseg3_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vloxseg3.mask.nxv1f32.nxv1i8(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -14590,9 +14402,9 @@ define <vscale x 1 x float> @test_vloxseg4_mask_nxv1f32_nxv1i64(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14623,9 +14435,9 @@ define <vscale x 1 x float> @test_vloxseg4_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14656,9 +14468,9 @@ define <vscale x 1 x float> @test_vloxseg4_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14689,9 +14501,9 @@ define <vscale x 1 x float> @test_vloxseg4_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vloxseg4_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14722,10 +14534,10 @@ define <vscale x 1 x float> @test_vloxseg5_mask_nxv1f32_nxv1i64(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14756,10 +14568,10 @@ define <vscale x 1 x float> @test_vloxseg5_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14790,10 +14602,10 @@ define <vscale x 1 x float> @test_vloxseg5_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14824,10 +14636,10 @@ define <vscale x 1 x float> @test_vloxseg5_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vloxseg5_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14858,11 +14670,11 @@ define <vscale x 1 x float> @test_vloxseg6_mask_nxv1f32_nxv1i64(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14893,11 +14705,11 @@ define <vscale x 1 x float> @test_vloxseg6_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14928,11 +14740,11 @@ define <vscale x 1 x float> @test_vloxseg6_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14963,11 +14775,11 @@ define <vscale x 1 x float> @test_vloxseg6_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vloxseg6_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14998,12 +14810,12 @@ define <vscale x 1 x float> @test_vloxseg7_mask_nxv1f32_nxv1i64(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -15034,12 +14846,12 @@ define <vscale x 1 x float> @test_vloxseg7_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -15070,12 +14882,12 @@ define <vscale x 1 x float> @test_vloxseg7_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -15106,12 +14918,12 @@ define <vscale x 1 x float> @test_vloxseg7_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vloxseg7_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -15142,13 +14954,13 @@ define <vscale x 1 x float> @test_vloxseg8_mask_nxv1f32_nxv1i64(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -15179,13 +14991,13 @@ define <vscale x 1 x float> @test_vloxseg8_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -15216,13 +15028,13 @@ define <vscale x 1 x float> @test_vloxseg8_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -15253,13 +15065,13 @@ define <vscale x 1 x float> @test_vloxseg8_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vloxseg8_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -15292,7 +15104,6 @@ define <vscale x 8 x half> @test_vloxseg2_mask_nxv8f16_nxv8i16(<vscale x 8 x hal
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vloxseg2.mask.nxv8f16.nxv8i16(<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15322,7 +15133,6 @@ define <vscale x 8 x half> @test_vloxseg2_mask_nxv8f16_nxv8i8(<vscale x 8 x half
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vloxseg2.mask.nxv8f16.nxv8i8(<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15352,7 +15162,6 @@ define <vscale x 8 x half> @test_vloxseg2_mask_nxv8f16_nxv8i64(<vscale x 8 x hal
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vloxseg2.mask.nxv8f16.nxv8i64(<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15382,7 +15191,6 @@ define <vscale x 8 x half> @test_vloxseg2_mask_nxv8f16_nxv8i32(<vscale x 8 x hal
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vloxseg2.mask.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15409,12 +15217,11 @@ entry:
 define <vscale x 8 x half> @test_vloxseg3_mask_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vloxseg3.mask.nxv8f16.nxv8i16(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15441,12 +15248,11 @@ entry:
 define <vscale x 8 x half> @test_vloxseg3_mask_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, i64 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vloxseg3.mask.nxv8f16.nxv8i8(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15474,10 +15280,9 @@ define <vscale x 8 x half> @test_vloxseg3_mask_nxv8f16_nxv8i64(<vscale x 8 x hal
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8f16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg3ei64.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vloxseg3.mask.nxv8f16.nxv8i64(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15505,10 +15310,9 @@ define <vscale x 8 x half> @test_vloxseg3_mask_nxv8f16_nxv8i32(<vscale x 8 x hal
 ; CHECK-LABEL: test_vloxseg3_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg3ei32.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vloxseg3.mask.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15536,9 +15340,9 @@ define <vscale x 8 x half> @test_vloxseg4_mask_nxv8f16_nxv8i16(<vscale x 8 x hal
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -15569,9 +15373,9 @@ define <vscale x 8 x half> @test_vloxseg4_mask_nxv8f16_nxv8i8(<vscale x 8 x half
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -15602,11 +15406,10 @@ define <vscale x 8 x half> @test_vloxseg4_mask_nxv8f16_nxv8i64(<vscale x 8 x hal
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8f16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei64.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vloxseg4.mask.nxv8f16.nxv8i64(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15633,13 +15436,12 @@ entry:
 define <vscale x 8 x half> @test_vloxseg4_mask_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vloxseg4ei32.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v18
+; CHECK-NEXT:    vloxseg4ei32.v v6, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vloxseg4.mask.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15669,7 +15471,6 @@ define <vscale x 8 x float> @test_vloxseg2_mask_nxv8f32_nxv8i16(<vscale x 8 x fl
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vloxseg2.mask.nxv8f32.nxv8i16(<vscale x 8 x float> %val,<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15699,7 +15500,6 @@ define <vscale x 8 x float> @test_vloxseg2_mask_nxv8f32_nxv8i8(<vscale x 8 x flo
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vloxseg2.mask.nxv8f32.nxv8i8(<vscale x 8 x float> %val,<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15729,7 +15529,6 @@ define <vscale x 8 x float> @test_vloxseg2_mask_nxv8f32_nxv8i64(<vscale x 8 x fl
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v4, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vloxseg2.mask.nxv8f32.nxv8i64(<vscale x 8 x float> %val,<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15759,7 +15558,6 @@ define <vscale x 8 x float> @test_vloxseg2_mask_nxv8f32_nxv8i32(<vscale x 8 x fl
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vloxseg2.mask.nxv8f32.nxv8i32(<vscale x 8 x float> %val,<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15789,7 +15587,6 @@ define <vscale x 2 x double> @test_vloxseg2_mask_nxv2f64_nxv2i32(<vscale x 2 x d
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vloxseg2.mask.nxv2f64.nxv2i32(<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -15819,7 +15616,6 @@ define <vscale x 2 x double> @test_vloxseg2_mask_nxv2f64_nxv2i8(<vscale x 2 x do
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vloxseg2.mask.nxv2f64.nxv2i8(<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -15849,7 +15645,6 @@ define <vscale x 2 x double> @test_vloxseg2_mask_nxv2f64_nxv2i16(<vscale x 2 x d
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vloxseg2.mask.nxv2f64.nxv2i16(<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -15879,7 +15674,6 @@ define <vscale x 2 x double> @test_vloxseg2_mask_nxv2f64_nxv2i64(<vscale x 2 x d
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vloxseg2.mask.nxv2f64.nxv2i64(<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -15906,12 +15700,11 @@ entry:
 define <vscale x 2 x double> @test_vloxseg3_mask_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei32.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vloxseg3.mask.nxv2f64.nxv2i32(<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -15938,12 +15731,11 @@ entry:
 define <vscale x 2 x double> @test_vloxseg3_mask_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vloxseg3.mask.nxv2f64.nxv2i8(<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -15970,12 +15762,11 @@ entry:
 define <vscale x 2 x double> @test_vloxseg3_mask_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vloxseg3.mask.nxv2f64.nxv2i16(<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -16002,12 +15793,11 @@ entry:
 define <vscale x 2 x double> @test_vloxseg3_mask_nxv2f64_nxv2i64(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei64.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei64.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vloxseg3.mask.nxv2f64.nxv2i64(<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -16035,9 +15825,9 @@ define <vscale x 2 x double> @test_vloxseg4_mask_nxv2f64_nxv2i32(<vscale x 2 x d
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -16068,9 +15858,9 @@ define <vscale x 2 x double> @test_vloxseg4_mask_nxv2f64_nxv2i8(<vscale x 2 x do
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -16101,9 +15891,9 @@ define <vscale x 2 x double> @test_vloxseg4_mask_nxv2f64_nxv2i16(<vscale x 2 x d
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -16134,9 +15924,9 @@ define <vscale x 2 x double> @test_vloxseg4_mask_nxv2f64_nxv2i64(<vscale x 2 x d
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -16169,7 +15959,6 @@ define <vscale x 4 x half> @test_vloxseg2_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg2.mask.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16199,7 +15988,6 @@ define <vscale x 4 x half> @test_vloxseg2_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg2.mask.nxv4f16.nxv4i8(<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16229,7 +16017,6 @@ define <vscale x 4 x half> @test_vloxseg2_mask_nxv4f16_nxv4i64(<vscale x 4 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg2.mask.nxv4f16.nxv4i64(<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16259,7 +16046,6 @@ define <vscale x 4 x half> @test_vloxseg2_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg2.mask.nxv4f16.nxv4i16(<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16287,10 +16073,9 @@ define <vscale x 4 x half> @test_vloxseg3_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg3.mask.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16317,12 +16102,11 @@ entry:
 define <vscale x 4 x half> @test_vloxseg3_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg3.mask.nxv4f16.nxv4i8(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16350,10 +16134,9 @@ define <vscale x 4 x half> @test_vloxseg3_mask_nxv4f16_nxv4i64(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg3ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg3.mask.nxv4f16.nxv4i64(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16380,12 +16163,11 @@ entry:
 define <vscale x 4 x half> @test_vloxseg3_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg3.mask.nxv4f16.nxv4i16(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16412,13 +16194,12 @@ entry:
 define <vscale x 4 x half> @test_vloxseg4_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vloxseg4ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vloxseg4ei32.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg4.mask.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16446,9 +16227,9 @@ define <vscale x 4 x half> @test_vloxseg4_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -16479,11 +16260,10 @@ define <vscale x 4 x half> @test_vloxseg4_mask_nxv4f16_nxv4i64(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg4.mask.nxv4f16.nxv4i64(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16511,9 +16291,9 @@ define <vscale x 4 x half> @test_vloxseg4_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -16544,10 +16324,10 @@ define <vscale x 4 x half> @test_vloxseg5_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -16578,10 +16358,10 @@ define <vscale x 4 x half> @test_vloxseg5_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -16612,12 +16392,11 @@ define <vscale x 4 x half> @test_vloxseg5_mask_nxv4f16_nxv4i64(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg5.mask.nxv4f16.nxv4i64(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16645,10 +16424,10 @@ define <vscale x 4 x half> @test_vloxseg5_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg5_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -16679,11 +16458,11 @@ define <vscale x 4 x half> @test_vloxseg6_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -16714,11 +16493,11 @@ define <vscale x 4 x half> @test_vloxseg6_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -16748,15 +16527,14 @@ entry:
 define <vscale x 4 x half> @test_vloxseg6_mask_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vloxseg6ei64.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v17
+; CHECK-NEXT:    vloxseg6ei64.v v7, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vloxseg6.mask.nxv4f16.nxv4i64(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16784,11 +16562,11 @@ define <vscale x 4 x half> @test_vloxseg6_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg6_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -16819,12 +16597,12 @@ define <vscale x 4 x half> @test_vloxseg7_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -16855,12 +16633,12 @@ define <vscale x 4 x half> @test_vloxseg7_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -16891,12 +16669,12 @@ define <vscale x 4 x half> @test_vloxseg7_mask_nxv4f16_nxv4i64(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -16927,12 +16705,12 @@ define <vscale x 4 x half> @test_vloxseg7_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg7_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -16963,13 +16741,13 @@ define <vscale x 4 x half> @test_vloxseg8_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -17000,13 +16778,13 @@ define <vscale x 4 x half> @test_vloxseg8_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17037,13 +16815,13 @@ define <vscale x 4 x half> @test_vloxseg8_mask_nxv4f16_nxv4i64(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -17074,13 +16852,13 @@ define <vscale x 4 x half> @test_vloxseg8_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vloxseg8_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17113,7 +16891,6 @@ define <vscale x 2 x half> @test_vloxseg2_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vloxseg2.mask.nxv2f16.nxv2i32(<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -17143,7 +16920,6 @@ define <vscale x 2 x half> @test_vloxseg2_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vloxseg2.mask.nxv2f16.nxv2i8(<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -17173,7 +16949,6 @@ define <vscale x 2 x half> @test_vloxseg2_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vloxseg2.mask.nxv2f16.nxv2i16(<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -17203,7 +16978,6 @@ define <vscale x 2 x half> @test_vloxseg2_mask_nxv2f16_nxv2i64(<vscale x 2 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vloxseg2.mask.nxv2f16.nxv2i64(<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -17230,12 +17004,11 @@ entry:
 define <vscale x 2 x half> @test_vloxseg3_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vloxseg3.mask.nxv2f16.nxv2i32(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -17262,12 +17035,11 @@ entry:
 define <vscale x 2 x half> @test_vloxseg3_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vloxseg3.mask.nxv2f16.nxv2i8(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -17294,12 +17066,11 @@ entry:
 define <vscale x 2 x half> @test_vloxseg3_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vloxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vloxseg3.mask.nxv2f16.nxv2i16(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -17327,10 +17098,9 @@ define <vscale x 2 x half> @test_vloxseg3_mask_nxv2f16_nxv2i64(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg3_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg3ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vloxseg3.mask.nxv2f16.nxv2i64(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -17358,9 +17128,9 @@ define <vscale x 2 x half> @test_vloxseg4_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17391,9 +17161,9 @@ define <vscale x 2 x half> @test_vloxseg4_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17424,9 +17194,9 @@ define <vscale x 2 x half> @test_vloxseg4_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17456,13 +17226,12 @@ entry:
 define <vscale x 2 x half> @test_vloxseg4_mask_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vloxseg4ei64.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vloxseg4ei64.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vloxseg4.mask.nxv2f16.nxv2i64(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -17490,10 +17259,10 @@ define <vscale x 2 x half> @test_vloxseg5_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17524,10 +17293,10 @@ define <vscale x 2 x half> @test_vloxseg5_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17558,10 +17327,10 @@ define <vscale x 2 x half> @test_vloxseg5_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17592,10 +17361,10 @@ define <vscale x 2 x half> @test_vloxseg5_mask_nxv2f16_nxv2i64(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg5_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -17626,11 +17395,11 @@ define <vscale x 2 x half> @test_vloxseg6_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17661,11 +17430,11 @@ define <vscale x 2 x half> @test_vloxseg6_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17696,11 +17465,11 @@ define <vscale x 2 x half> @test_vloxseg6_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17731,11 +17500,11 @@ define <vscale x 2 x half> @test_vloxseg6_mask_nxv2f16_nxv2i64(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg6_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -17766,12 +17535,12 @@ define <vscale x 2 x half> @test_vloxseg7_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17802,12 +17571,12 @@ define <vscale x 2 x half> @test_vloxseg7_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17838,12 +17607,12 @@ define <vscale x 2 x half> @test_vloxseg7_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17874,12 +17643,12 @@ define <vscale x 2 x half> @test_vloxseg7_mask_nxv2f16_nxv2i64(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg7_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -17910,13 +17679,13 @@ define <vscale x 2 x half> @test_vloxseg8_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17947,13 +17716,13 @@ define <vscale x 2 x half> @test_vloxseg8_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17984,13 +17753,13 @@ define <vscale x 2 x half> @test_vloxseg8_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -18021,13 +17790,13 @@ define <vscale x 2 x half> @test_vloxseg8_mask_nxv2f16_nxv2i64(<vscale x 2 x hal
 ; CHECK-LABEL: test_vloxseg8_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vloxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -18060,7 +17829,6 @@ define <vscale x 4 x float> @test_vloxseg2_mask_nxv4f32_nxv4i32(<vscale x 4 x fl
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei32.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vloxseg2.mask.nxv4f32.nxv4i32(<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -18090,7 +17858,6 @@ define <vscale x 4 x float> @test_vloxseg2_mask_nxv4f32_nxv4i8(<vscale x 4 x flo
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vloxseg2.mask.nxv4f32.nxv4i8(<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -18120,7 +17887,6 @@ define <vscale x 4 x float> @test_vloxseg2_mask_nxv4f32_nxv4i64(<vscale x 4 x fl
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei64.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vloxseg2.mask.nxv4f32.nxv4i64(<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -18150,7 +17916,6 @@ define <vscale x 4 x float> @test_vloxseg2_mask_nxv4f32_nxv4i16(<vscale x 4 x fl
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vloxseg2.mask.nxv4f32.nxv4i16(<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -18177,12 +17942,11 @@ entry:
 define <vscale x 4 x float> @test_vloxseg3_mask_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei32.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vloxseg3.mask.nxv4f32.nxv4i32(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -18209,12 +17973,11 @@ entry:
 define <vscale x 4 x float> @test_vloxseg3_mask_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vloxseg3.mask.nxv4f32.nxv4i8(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -18242,10 +18005,9 @@ define <vscale x 4 x float> @test_vloxseg3_mask_nxv4f32_nxv4i64(<vscale x 4 x fl
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4f32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg3ei64.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vloxseg3.mask.nxv4f32.nxv4i64(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -18272,12 +18034,11 @@ entry:
 define <vscale x 4 x float> @test_vloxseg3_mask_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg3_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vloxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vloxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vloxseg3.mask.nxv4f32.nxv4i16(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -18305,9 +18066,9 @@ define <vscale x 4 x float> @test_vloxseg4_mask_nxv4f32_nxv4i32(<vscale x 4 x fl
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -18338,9 +18099,9 @@ define <vscale x 4 x float> @test_vloxseg4_mask_nxv4f32_nxv4i8(<vscale x 4 x flo
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -18370,13 +18131,12 @@ entry:
 define <vscale x 4 x float> @test_vloxseg4_mask_nxv4f32_nxv4i64(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4f32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vloxseg4ei64.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v18
+; CHECK-NEXT:    vloxseg4ei64.v v6, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vloxseg4.mask.nxv4f32.nxv4i64(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -18404,9 +18164,9 @@ define <vscale x 4 x float> @test_vloxseg4_mask_nxv4f32_nxv4i16(<vscale x 4 x fl
 ; CHECK-LABEL: test_vloxseg4_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vloxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
diff --git a/llvm/test/CodeGen/RISCV/rvv/vlseg-rv32.ll b/llvm/test/CodeGen/RISCV/rvv/vlseg-rv32.ll
index 452b5be5c998f..98cba968170b2 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vlseg-rv32.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vlseg-rv32.ll
@@ -10,7 +10,6 @@ define <vscale x 16 x i16> @test_vlseg2_nxv16i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v4, (a0)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vlseg2.nxv16i16(<vscale x 16 x i16> undef, <vscale x 16 x i16> undef, ptr %base, i32 %vl)
@@ -25,7 +24,6 @@ define <vscale x 16 x i16> @test_vlseg2_mask_nxv16i16(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vlseg2e16.v v4, (a0)
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlseg2e16.v v4, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vlseg2.nxv16i16(<vscale x 16 x i16> undef, <vscale x 16 x i16> undef, ptr %base, i32 %vl)
@@ -43,7 +41,6 @@ define <vscale x 1 x i8> @test_vlseg2_nxv1i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg2.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -58,7 +55,6 @@ define <vscale x 1 x i8> @test_vlseg2_mask_nxv1i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg2.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -76,7 +72,6 @@ define <vscale x 1 x i8> @test_vlseg3_nxv1i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlseg3e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg3.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -92,7 +87,6 @@ define <vscale x 1 x i8> @test_vlseg3_mask_nxv1i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg3.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -110,7 +104,6 @@ define <vscale x 1 x i8> @test_vlseg4_nxv1i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlseg4e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg4.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -127,7 +120,6 @@ define <vscale x 1 x i8> @test_vlseg4_mask_nxv1i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg4.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -145,7 +137,6 @@ define <vscale x 1 x i8> @test_vlseg5_nxv1i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlseg5e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg5.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -163,7 +154,6 @@ define <vscale x 1 x i8> @test_vlseg5_mask_nxv1i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg5.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -181,7 +171,6 @@ define <vscale x 1 x i8> @test_vlseg6_nxv1i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlseg6e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg6.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -200,7 +189,6 @@ define <vscale x 1 x i8> @test_vlseg6_mask_nxv1i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg6.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -218,7 +206,6 @@ define <vscale x 1 x i8> @test_vlseg7_nxv1i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlseg7e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg7.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -238,7 +225,6 @@ define <vscale x 1 x i8> @test_vlseg7_mask_nxv1i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg7.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -256,7 +242,6 @@ define <vscale x 1 x i8> @test_vlseg8_nxv1i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlseg8e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg8.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -277,7 +262,6 @@ define <vscale x 1 x i8> @test_vlseg8_mask_nxv1i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg8.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -295,7 +279,6 @@ define <vscale x 16 x i8> @test_vlseg2_nxv16i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vlseg2e8.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlseg2.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i32 %vl)
@@ -310,7 +293,6 @@ define <vscale x 16 x i8> @test_vlseg2_mask_nxv16i8(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e8.v v6, (a0)
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlseg2e8.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlseg2.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i32 %vl)
@@ -328,7 +310,6 @@ define <vscale x 16 x i8> @test_vlseg3_nxv16i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vlseg3e8.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlseg3.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i32 %vl)
@@ -344,7 +325,6 @@ define <vscale x 16 x i8> @test_vlseg3_mask_nxv16i8(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlseg3e8.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlseg3.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i32 %vl)
@@ -362,7 +342,6 @@ define <vscale x 16 x i8> @test_vlseg4_nxv16i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vlseg4e8.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlseg4.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i32 %vl)
@@ -379,7 +358,6 @@ define <vscale x 16 x i8> @test_vlseg4_mask_nxv16i8(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlseg4e8.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlseg4.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i32 %vl)
@@ -397,7 +375,6 @@ define <vscale x 2 x i32> @test_vlseg2_nxv2i32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg2.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -412,7 +389,6 @@ define <vscale x 2 x i32> @test_vlseg2_mask_nxv2i32(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg2.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -430,7 +406,6 @@ define <vscale x 2 x i32> @test_vlseg3_nxv2i32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg3e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg3.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -446,7 +421,6 @@ define <vscale x 2 x i32> @test_vlseg3_mask_nxv2i32(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg3.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -464,7 +438,6 @@ define <vscale x 2 x i32> @test_vlseg4_nxv2i32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg4e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg4.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -481,7 +454,6 @@ define <vscale x 2 x i32> @test_vlseg4_mask_nxv2i32(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg4.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -499,7 +471,6 @@ define <vscale x 2 x i32> @test_vlseg5_nxv2i32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg5e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg5.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -517,7 +488,6 @@ define <vscale x 2 x i32> @test_vlseg5_mask_nxv2i32(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg5.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -535,7 +505,6 @@ define <vscale x 2 x i32> @test_vlseg6_nxv2i32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg6e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg6.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -554,7 +523,6 @@ define <vscale x 2 x i32> @test_vlseg6_mask_nxv2i32(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg6.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -572,7 +540,6 @@ define <vscale x 2 x i32> @test_vlseg7_nxv2i32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg7e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg7.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -592,7 +559,6 @@ define <vscale x 2 x i32> @test_vlseg7_mask_nxv2i32(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg7.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -610,7 +576,6 @@ define <vscale x 2 x i32> @test_vlseg8_nxv2i32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg8e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg8.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -631,7 +596,6 @@ define <vscale x 2 x i32> @test_vlseg8_mask_nxv2i32(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg8.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -649,7 +613,6 @@ define <vscale x 4 x i16> @test_vlseg2_nxv4i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg2.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -664,7 +627,6 @@ define <vscale x 4 x i16> @test_vlseg2_mask_nxv4i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg2.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -682,7 +644,6 @@ define <vscale x 4 x i16> @test_vlseg3_nxv4i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg3.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -698,7 +659,6 @@ define <vscale x 4 x i16> @test_vlseg3_mask_nxv4i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg3.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -716,7 +676,6 @@ define <vscale x 4 x i16> @test_vlseg4_nxv4i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg4.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -733,7 +692,6 @@ define <vscale x 4 x i16> @test_vlseg4_mask_nxv4i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg4.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -751,7 +709,6 @@ define <vscale x 4 x i16> @test_vlseg5_nxv4i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg5.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -769,7 +726,6 @@ define <vscale x 4 x i16> @test_vlseg5_mask_nxv4i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg5.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -787,7 +743,6 @@ define <vscale x 4 x i16> @test_vlseg6_nxv4i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg6.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -806,7 +761,6 @@ define <vscale x 4 x i16> @test_vlseg6_mask_nxv4i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg6.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -824,7 +778,6 @@ define <vscale x 4 x i16> @test_vlseg7_nxv4i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg7.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -844,7 +797,6 @@ define <vscale x 4 x i16> @test_vlseg7_mask_nxv4i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg7.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -862,7 +814,6 @@ define <vscale x 4 x i16> @test_vlseg8_nxv4i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg8.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -883,7 +834,6 @@ define <vscale x 4 x i16> @test_vlseg8_mask_nxv4i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg8.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -901,7 +851,6 @@ define <vscale x 1 x i32> @test_vlseg2_nxv1i32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg2.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -916,7 +865,6 @@ define <vscale x 1 x i32> @test_vlseg2_mask_nxv1i32(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg2.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -934,7 +882,6 @@ define <vscale x 1 x i32> @test_vlseg3_nxv1i32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg3e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg3.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -950,7 +897,6 @@ define <vscale x 1 x i32> @test_vlseg3_mask_nxv1i32(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg3.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -968,7 +914,6 @@ define <vscale x 1 x i32> @test_vlseg4_nxv1i32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg4e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg4.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -985,7 +930,6 @@ define <vscale x 1 x i32> @test_vlseg4_mask_nxv1i32(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg4.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -1003,7 +947,6 @@ define <vscale x 1 x i32> @test_vlseg5_nxv1i32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg5e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg5.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -1021,7 +964,6 @@ define <vscale x 1 x i32> @test_vlseg5_mask_nxv1i32(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg5.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -1039,7 +981,6 @@ define <vscale x 1 x i32> @test_vlseg6_nxv1i32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg6e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg6.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -1058,7 +999,6 @@ define <vscale x 1 x i32> @test_vlseg6_mask_nxv1i32(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg6.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -1076,7 +1016,6 @@ define <vscale x 1 x i32> @test_vlseg7_nxv1i32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg7e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg7.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -1096,7 +1035,6 @@ define <vscale x 1 x i32> @test_vlseg7_mask_nxv1i32(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg7.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -1114,7 +1052,6 @@ define <vscale x 1 x i32> @test_vlseg8_nxv1i32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg8e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg8.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -1135,7 +1072,6 @@ define <vscale x 1 x i32> @test_vlseg8_mask_nxv1i32(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg8.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -1153,7 +1089,6 @@ define <vscale x 8 x i16> @test_vlseg2_nxv8i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlseg2.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i32 %vl)
@@ -1168,7 +1103,6 @@ define <vscale x 8 x i16> @test_vlseg2_mask_nxv8i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e16.v v6, (a0)
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlseg2e16.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlseg2.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i32 %vl)
@@ -1186,7 +1120,6 @@ define <vscale x 8 x i16> @test_vlseg3_nxv8i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vlseg3e16.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlseg3.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i32 %vl)
@@ -1202,7 +1135,6 @@ define <vscale x 8 x i16> @test_vlseg3_mask_nxv8i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlseg3e16.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlseg3.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i32 %vl)
@@ -1220,7 +1152,6 @@ define <vscale x 8 x i16> @test_vlseg4_nxv8i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vlseg4e16.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlseg4.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i32 %vl)
@@ -1237,7 +1168,6 @@ define <vscale x 8 x i16> @test_vlseg4_mask_nxv8i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlseg4e16.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlseg4.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i32 %vl)
@@ -1255,7 +1185,6 @@ define <vscale x 8 x i8> @test_vlseg2_nxv8i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg2.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1270,7 +1199,6 @@ define <vscale x 8 x i8> @test_vlseg2_mask_nxv8i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg2.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1288,7 +1216,6 @@ define <vscale x 8 x i8> @test_vlseg3_nxv8i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vlseg3e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg3.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1304,7 +1231,6 @@ define <vscale x 8 x i8> @test_vlseg3_mask_nxv8i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg3.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1322,7 +1248,6 @@ define <vscale x 8 x i8> @test_vlseg4_nxv8i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vlseg4e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg4.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1339,7 +1264,6 @@ define <vscale x 8 x i8> @test_vlseg4_mask_nxv8i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg4.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1357,7 +1281,6 @@ define <vscale x 8 x i8> @test_vlseg5_nxv8i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vlseg5e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg5.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1375,7 +1298,6 @@ define <vscale x 8 x i8> @test_vlseg5_mask_nxv8i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg5.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1393,7 +1315,6 @@ define <vscale x 8 x i8> @test_vlseg6_nxv8i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vlseg6e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg6.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1412,7 +1333,6 @@ define <vscale x 8 x i8> @test_vlseg6_mask_nxv8i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg6.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1430,7 +1350,6 @@ define <vscale x 8 x i8> @test_vlseg7_nxv8i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vlseg7e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg7.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1450,7 +1369,6 @@ define <vscale x 8 x i8> @test_vlseg7_mask_nxv8i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg7.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1468,7 +1386,6 @@ define <vscale x 8 x i8> @test_vlseg8_nxv8i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vlseg8e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg8.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1489,7 +1406,6 @@ define <vscale x 8 x i8> @test_vlseg8_mask_nxv8i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg8.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1507,7 +1423,6 @@ define <vscale x 8 x i32> @test_vlseg2_nxv8i32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vlseg2e32.v v4, (a0)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vlseg2.nxv8i32(<vscale x 8 x i32> undef, <vscale x 8 x i32> undef, ptr %base, i32 %vl)
@@ -1522,7 +1437,6 @@ define <vscale x 8 x i32> @test_vlseg2_mask_nxv8i32(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e32.v v4, (a0)
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlseg2e32.v v4, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vlseg2.nxv8i32(<vscale x 8 x i32> undef, <vscale x 8 x i32> undef, ptr %base, i32 %vl)
@@ -1540,7 +1454,6 @@ define <vscale x 4 x i8> @test_vlseg2_nxv4i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg2.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1555,7 +1468,6 @@ define <vscale x 4 x i8> @test_vlseg2_mask_nxv4i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg2.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1573,7 +1485,6 @@ define <vscale x 4 x i8> @test_vlseg3_nxv4i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg3e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg3.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1589,7 +1500,6 @@ define <vscale x 4 x i8> @test_vlseg3_mask_nxv4i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg3.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1607,7 +1517,6 @@ define <vscale x 4 x i8> @test_vlseg4_nxv4i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg4e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg4.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1624,7 +1533,6 @@ define <vscale x 4 x i8> @test_vlseg4_mask_nxv4i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg4.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1642,7 +1550,6 @@ define <vscale x 4 x i8> @test_vlseg5_nxv4i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg5e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg5.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1660,7 +1567,6 @@ define <vscale x 4 x i8> @test_vlseg5_mask_nxv4i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg5.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1678,7 +1584,6 @@ define <vscale x 4 x i8> @test_vlseg6_nxv4i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg6e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg6.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1697,7 +1602,6 @@ define <vscale x 4 x i8> @test_vlseg6_mask_nxv4i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg6.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1715,7 +1619,6 @@ define <vscale x 4 x i8> @test_vlseg7_nxv4i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg7e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg7.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1735,7 +1638,6 @@ define <vscale x 4 x i8> @test_vlseg7_mask_nxv4i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg7.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1753,7 +1655,6 @@ define <vscale x 4 x i8> @test_vlseg8_nxv4i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg8e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg8.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1774,7 +1675,6 @@ define <vscale x 4 x i8> @test_vlseg8_mask_nxv4i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg8.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1792,7 +1692,6 @@ define <vscale x 1 x i16> @test_vlseg2_nxv1i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg2.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -1807,7 +1706,6 @@ define <vscale x 1 x i16> @test_vlseg2_mask_nxv1i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg2.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -1825,7 +1723,6 @@ define <vscale x 1 x i16> @test_vlseg3_nxv1i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg3.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -1841,7 +1738,6 @@ define <vscale x 1 x i16> @test_vlseg3_mask_nxv1i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg3.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -1859,7 +1755,6 @@ define <vscale x 1 x i16> @test_vlseg4_nxv1i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg4.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -1876,7 +1771,6 @@ define <vscale x 1 x i16> @test_vlseg4_mask_nxv1i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg4.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -1894,7 +1788,6 @@ define <vscale x 1 x i16> @test_vlseg5_nxv1i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg5.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -1912,7 +1805,6 @@ define <vscale x 1 x i16> @test_vlseg5_mask_nxv1i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg5.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -1930,7 +1822,6 @@ define <vscale x 1 x i16> @test_vlseg6_nxv1i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg6.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -1949,7 +1840,6 @@ define <vscale x 1 x i16> @test_vlseg6_mask_nxv1i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg6.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -1967,7 +1857,6 @@ define <vscale x 1 x i16> @test_vlseg7_nxv1i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg7.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -1987,7 +1876,6 @@ define <vscale x 1 x i16> @test_vlseg7_mask_nxv1i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg7.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -2005,7 +1893,6 @@ define <vscale x 1 x i16> @test_vlseg8_nxv1i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg8.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -2026,7 +1913,6 @@ define <vscale x 1 x i16> @test_vlseg8_mask_nxv1i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg8.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -2044,7 +1930,6 @@ define <vscale x 32 x i8> @test_vlseg2_nxv32i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
 ; CHECK-NEXT:    vlseg2e8.v v4, (a0)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>} @llvm.riscv.vlseg2.nxv32i8(<vscale x 32 x i8> undef, <vscale x 32 x i8> undef, ptr %base, i32 %vl)
@@ -2059,7 +1944,6 @@ define <vscale x 32 x i8> @test_vlseg2_mask_nxv32i8(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e8.v v4, (a0)
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlseg2e8.v v4, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>} @llvm.riscv.vlseg2.nxv32i8(<vscale x 32 x i8> undef, <vscale x 32 x i8> undef, ptr %base, i32 %vl)
@@ -2077,7 +1961,6 @@ define <vscale x 2 x i8> @test_vlseg2_nxv2i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg2.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2092,7 +1975,6 @@ define <vscale x 2 x i8> @test_vlseg2_mask_nxv2i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg2.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2110,7 +1992,6 @@ define <vscale x 2 x i8> @test_vlseg3_nxv2i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlseg3e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg3.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2126,7 +2007,6 @@ define <vscale x 2 x i8> @test_vlseg3_mask_nxv2i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg3.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2144,7 +2024,6 @@ define <vscale x 2 x i8> @test_vlseg4_nxv2i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlseg4e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg4.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2161,7 +2040,6 @@ define <vscale x 2 x i8> @test_vlseg4_mask_nxv2i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg4.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2179,7 +2057,6 @@ define <vscale x 2 x i8> @test_vlseg5_nxv2i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlseg5e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg5.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2197,7 +2074,6 @@ define <vscale x 2 x i8> @test_vlseg5_mask_nxv2i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg5.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2215,7 +2091,6 @@ define <vscale x 2 x i8> @test_vlseg6_nxv2i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlseg6e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg6.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2234,7 +2109,6 @@ define <vscale x 2 x i8> @test_vlseg6_mask_nxv2i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg6.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2252,7 +2126,6 @@ define <vscale x 2 x i8> @test_vlseg7_nxv2i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlseg7e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg7.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2272,7 +2145,6 @@ define <vscale x 2 x i8> @test_vlseg7_mask_nxv2i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg7.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2290,7 +2162,6 @@ define <vscale x 2 x i8> @test_vlseg8_nxv2i8(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlseg8e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg8.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2311,7 +2182,6 @@ define <vscale x 2 x i8> @test_vlseg8_mask_nxv2i8(ptr %base, i32 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg8.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2329,7 +2199,6 @@ define <vscale x 2 x i16> @test_vlseg2_nxv2i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg2.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2344,7 +2213,6 @@ define <vscale x 2 x i16> @test_vlseg2_mask_nxv2i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg2.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2362,7 +2230,6 @@ define <vscale x 2 x i16> @test_vlseg3_nxv2i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg3.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2378,7 +2245,6 @@ define <vscale x 2 x i16> @test_vlseg3_mask_nxv2i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg3.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2396,7 +2262,6 @@ define <vscale x 2 x i16> @test_vlseg4_nxv2i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg4.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2413,7 +2278,6 @@ define <vscale x 2 x i16> @test_vlseg4_mask_nxv2i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg4.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2431,7 +2295,6 @@ define <vscale x 2 x i16> @test_vlseg5_nxv2i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg5.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2449,7 +2312,6 @@ define <vscale x 2 x i16> @test_vlseg5_mask_nxv2i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg5.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2467,7 +2329,6 @@ define <vscale x 2 x i16> @test_vlseg6_nxv2i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg6.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2486,7 +2347,6 @@ define <vscale x 2 x i16> @test_vlseg6_mask_nxv2i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg6.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2504,7 +2364,6 @@ define <vscale x 2 x i16> @test_vlseg7_nxv2i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg7.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2524,7 +2383,6 @@ define <vscale x 2 x i16> @test_vlseg7_mask_nxv2i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg7.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2542,7 +2400,6 @@ define <vscale x 2 x i16> @test_vlseg8_nxv2i16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg8.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2563,7 +2420,6 @@ define <vscale x 2 x i16> @test_vlseg8_mask_nxv2i16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg8.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2581,7 +2437,6 @@ define <vscale x 4 x i32> @test_vlseg2_nxv4i32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vlseg2e32.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlseg2.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i32 %vl)
@@ -2596,7 +2451,6 @@ define <vscale x 4 x i32> @test_vlseg2_mask_nxv4i32(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e32.v v6, (a0)
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlseg2e32.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlseg2.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i32 %vl)
@@ -2614,7 +2468,6 @@ define <vscale x 4 x i32> @test_vlseg3_nxv4i32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vlseg3e32.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlseg3.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i32 %vl)
@@ -2630,7 +2483,6 @@ define <vscale x 4 x i32> @test_vlseg3_mask_nxv4i32(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlseg3e32.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlseg3.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i32 %vl)
@@ -2648,7 +2500,6 @@ define <vscale x 4 x i32> @test_vlseg4_nxv4i32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vlseg4e32.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlseg4.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i32 %vl)
@@ -2665,7 +2516,6 @@ define <vscale x 4 x i32> @test_vlseg4_mask_nxv4i32(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlseg4e32.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlseg4.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i32 %vl)
@@ -2683,7 +2533,6 @@ define <vscale x 16 x half> @test_vlseg2_nxv16f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v4, (a0)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vlseg2.nxv16f16(<vscale x 16 x half> undef, <vscale x 16 x half> undef, ptr %base, i32 %vl)
@@ -2698,7 +2547,6 @@ define <vscale x 16 x half> @test_vlseg2_mask_nxv16f16(ptr %base, i32 %vl, <vsca
 ; CHECK-NEXT:    vlseg2e16.v v4, (a0)
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlseg2e16.v v4, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vlseg2.nxv16f16(<vscale x 16 x half> undef, <vscale x 16 x half> undef, ptr %base, i32 %vl)
@@ -2716,7 +2564,6 @@ define <vscale x 4 x double> @test_vlseg2_nxv4f64(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
 ; CHECK-NEXT:    vlseg2e64.v v4, (a0)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vlseg2.nxv4f64(<vscale x 4 x double> undef, <vscale x 4 x double> undef, ptr %base, i32 %vl)
@@ -2731,7 +2578,6 @@ define <vscale x 4 x double> @test_vlseg2_mask_nxv4f64(ptr %base, i32 %vl, <vsca
 ; CHECK-NEXT:    vlseg2e64.v v4, (a0)
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlseg2e64.v v4, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vlseg2.nxv4f64(<vscale x 4 x double> undef, <vscale x 4 x double> undef, ptr %base, i32 %vl)
@@ -2749,7 +2595,6 @@ define <vscale x 1 x double> @test_vlseg2_nxv1f64(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg2e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg2.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -2764,7 +2609,6 @@ define <vscale x 1 x double> @test_vlseg2_mask_nxv1f64(ptr %base, i32 %vl, <vsca
 ; CHECK-NEXT:    vlseg2e64.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg2.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -2782,7 +2626,6 @@ define <vscale x 1 x double> @test_vlseg3_nxv1f64(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg3e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg3.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -2798,7 +2641,6 @@ define <vscale x 1 x double> @test_vlseg3_mask_nxv1f64(ptr %base, i32 %vl, <vsca
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg3.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -2816,7 +2658,6 @@ define <vscale x 1 x double> @test_vlseg4_nxv1f64(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg4e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg4.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -2833,7 +2674,6 @@ define <vscale x 1 x double> @test_vlseg4_mask_nxv1f64(ptr %base, i32 %vl, <vsca
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg4.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -2851,7 +2691,6 @@ define <vscale x 1 x double> @test_vlseg5_nxv1f64(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg5e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg5.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -2869,7 +2708,6 @@ define <vscale x 1 x double> @test_vlseg5_mask_nxv1f64(ptr %base, i32 %vl, <vsca
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg5.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -2887,7 +2725,6 @@ define <vscale x 1 x double> @test_vlseg6_nxv1f64(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg6e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg6.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -2906,7 +2743,6 @@ define <vscale x 1 x double> @test_vlseg6_mask_nxv1f64(ptr %base, i32 %vl, <vsca
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg6.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -2924,7 +2760,6 @@ define <vscale x 1 x double> @test_vlseg7_nxv1f64(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg7e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg7.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -2944,7 +2779,6 @@ define <vscale x 1 x double> @test_vlseg7_mask_nxv1f64(ptr %base, i32 %vl, <vsca
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg7.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -2962,7 +2796,6 @@ define <vscale x 1 x double> @test_vlseg8_nxv1f64(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg8e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg8.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -2983,7 +2816,6 @@ define <vscale x 1 x double> @test_vlseg8_mask_nxv1f64(ptr %base, i32 %vl, <vsca
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg8.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -3001,7 +2833,6 @@ define <vscale x 2 x float> @test_vlseg2_nxv2f32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg2.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3016,7 +2847,6 @@ define <vscale x 2 x float> @test_vlseg2_mask_nxv2f32(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg2.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3034,7 +2864,6 @@ define <vscale x 2 x float> @test_vlseg3_nxv2f32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg3e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg3.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3050,7 +2879,6 @@ define <vscale x 2 x float> @test_vlseg3_mask_nxv2f32(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg3.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3068,7 +2896,6 @@ define <vscale x 2 x float> @test_vlseg4_nxv2f32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg4e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg4.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3085,7 +2912,6 @@ define <vscale x 2 x float> @test_vlseg4_mask_nxv2f32(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg4.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3103,7 +2929,6 @@ define <vscale x 2 x float> @test_vlseg5_nxv2f32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg5e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg5.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3121,7 +2946,6 @@ define <vscale x 2 x float> @test_vlseg5_mask_nxv2f32(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg5.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3139,7 +2963,6 @@ define <vscale x 2 x float> @test_vlseg6_nxv2f32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg6e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg6.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3158,7 +2981,6 @@ define <vscale x 2 x float> @test_vlseg6_mask_nxv2f32(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg6.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3176,7 +2998,6 @@ define <vscale x 2 x float> @test_vlseg7_nxv2f32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg7e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg7.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3196,7 +3017,6 @@ define <vscale x 2 x float> @test_vlseg7_mask_nxv2f32(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg7.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3214,7 +3034,6 @@ define <vscale x 2 x float> @test_vlseg8_nxv2f32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg8e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg8.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3235,7 +3054,6 @@ define <vscale x 2 x float> @test_vlseg8_mask_nxv2f32(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg8.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3253,7 +3071,6 @@ define <vscale x 1 x half> @test_vlseg2_nxv1f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg2.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3268,7 +3085,6 @@ define <vscale x 1 x half> @test_vlseg2_mask_nxv1f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg2.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3286,7 +3102,6 @@ define <vscale x 1 x half> @test_vlseg3_nxv1f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg3.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3302,7 +3117,6 @@ define <vscale x 1 x half> @test_vlseg3_mask_nxv1f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg3.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3320,7 +3134,6 @@ define <vscale x 1 x half> @test_vlseg4_nxv1f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg4.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3337,7 +3150,6 @@ define <vscale x 1 x half> @test_vlseg4_mask_nxv1f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg4.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3355,7 +3167,6 @@ define <vscale x 1 x half> @test_vlseg5_nxv1f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg5.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3373,7 +3184,6 @@ define <vscale x 1 x half> @test_vlseg5_mask_nxv1f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg5.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3391,7 +3201,6 @@ define <vscale x 1 x half> @test_vlseg6_nxv1f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg6.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3410,7 +3219,6 @@ define <vscale x 1 x half> @test_vlseg6_mask_nxv1f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg6.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3428,7 +3236,6 @@ define <vscale x 1 x half> @test_vlseg7_nxv1f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg7.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3448,7 +3255,6 @@ define <vscale x 1 x half> @test_vlseg7_mask_nxv1f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg7.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3466,7 +3272,6 @@ define <vscale x 1 x half> @test_vlseg8_nxv1f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg8.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3487,7 +3292,6 @@ define <vscale x 1 x half> @test_vlseg8_mask_nxv1f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg8.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3505,7 +3309,6 @@ define <vscale x 1 x float> @test_vlseg2_nxv1f32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg2.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -3520,7 +3323,6 @@ define <vscale x 1 x float> @test_vlseg2_mask_nxv1f32(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg2.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -3538,7 +3340,6 @@ define <vscale x 1 x float> @test_vlseg3_nxv1f32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg3e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg3.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -3554,7 +3355,6 @@ define <vscale x 1 x float> @test_vlseg3_mask_nxv1f32(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg3.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -3572,7 +3372,6 @@ define <vscale x 1 x float> @test_vlseg4_nxv1f32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg4e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg4.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -3589,7 +3388,6 @@ define <vscale x 1 x float> @test_vlseg4_mask_nxv1f32(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg4.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -3607,7 +3405,6 @@ define <vscale x 1 x float> @test_vlseg5_nxv1f32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg5e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg5.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -3625,7 +3422,6 @@ define <vscale x 1 x float> @test_vlseg5_mask_nxv1f32(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg5.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -3643,7 +3439,6 @@ define <vscale x 1 x float> @test_vlseg6_nxv1f32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg6e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg6.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -3662,7 +3457,6 @@ define <vscale x 1 x float> @test_vlseg6_mask_nxv1f32(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg6.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -3680,7 +3474,6 @@ define <vscale x 1 x float> @test_vlseg7_nxv1f32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg7e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg7.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -3700,7 +3493,6 @@ define <vscale x 1 x float> @test_vlseg7_mask_nxv1f32(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg7.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -3718,7 +3510,6 @@ define <vscale x 1 x float> @test_vlseg8_nxv1f32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg8e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg8.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -3739,7 +3530,6 @@ define <vscale x 1 x float> @test_vlseg8_mask_nxv1f32(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg8.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -3757,7 +3547,6 @@ define <vscale x 8 x half> @test_vlseg2_nxv8f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlseg2.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i32 %vl)
@@ -3772,7 +3561,6 @@ define <vscale x 8 x half> @test_vlseg2_mask_nxv8f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e16.v v6, (a0)
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlseg2e16.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlseg2.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i32 %vl)
@@ -3790,7 +3578,6 @@ define <vscale x 8 x half> @test_vlseg3_nxv8f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vlseg3e16.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlseg3.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i32 %vl)
@@ -3806,7 +3593,6 @@ define <vscale x 8 x half> @test_vlseg3_mask_nxv8f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlseg3e16.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlseg3.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i32 %vl)
@@ -3824,7 +3610,6 @@ define <vscale x 8 x half> @test_vlseg4_nxv8f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vlseg4e16.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlseg4.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i32 %vl)
@@ -3841,7 +3626,6 @@ define <vscale x 8 x half> @test_vlseg4_mask_nxv8f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlseg4e16.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlseg4.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i32 %vl)
@@ -3859,7 +3643,6 @@ define <vscale x 8 x float> @test_vlseg2_nxv8f32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vlseg2e32.v v4, (a0)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vlseg2.nxv8f32(<vscale x 8 x float> undef, <vscale x 8 x float> undef, ptr %base, i32 %vl)
@@ -3874,7 +3657,6 @@ define <vscale x 8 x float> @test_vlseg2_mask_nxv8f32(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vlseg2e32.v v4, (a0)
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlseg2e32.v v4, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vlseg2.nxv8f32(<vscale x 8 x float> undef, <vscale x 8 x float> undef, ptr %base, i32 %vl)
@@ -3892,7 +3674,6 @@ define <vscale x 2 x double> @test_vlseg2_nxv2f64(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vlseg2e64.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlseg2.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i32 %vl)
@@ -3907,7 +3688,6 @@ define <vscale x 2 x double> @test_vlseg2_mask_nxv2f64(ptr %base, i32 %vl, <vsca
 ; CHECK-NEXT:    vlseg2e64.v v6, (a0)
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlseg2e64.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlseg2.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i32 %vl)
@@ -3925,7 +3705,6 @@ define <vscale x 2 x double> @test_vlseg3_nxv2f64(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vlseg3e64.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlseg3.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i32 %vl)
@@ -3941,7 +3720,6 @@ define <vscale x 2 x double> @test_vlseg3_mask_nxv2f64(ptr %base, i32 %vl, <vsca
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlseg3e64.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlseg3.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i32 %vl)
@@ -3959,7 +3737,6 @@ define <vscale x 2 x double> @test_vlseg4_nxv2f64(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vlseg4e64.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlseg4.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i32 %vl)
@@ -3976,7 +3753,6 @@ define <vscale x 2 x double> @test_vlseg4_mask_nxv2f64(ptr %base, i32 %vl, <vsca
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlseg4e64.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlseg4.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i32 %vl)
@@ -3994,7 +3770,6 @@ define <vscale x 4 x half> @test_vlseg2_nxv4f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg2.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4009,7 +3784,6 @@ define <vscale x 4 x half> @test_vlseg2_mask_nxv4f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg2.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4027,7 +3801,6 @@ define <vscale x 4 x half> @test_vlseg3_nxv4f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg3.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4043,7 +3816,6 @@ define <vscale x 4 x half> @test_vlseg3_mask_nxv4f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg3.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4061,7 +3833,6 @@ define <vscale x 4 x half> @test_vlseg4_nxv4f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg4.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4078,7 +3849,6 @@ define <vscale x 4 x half> @test_vlseg4_mask_nxv4f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg4.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4096,7 +3866,6 @@ define <vscale x 4 x half> @test_vlseg5_nxv4f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg5.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4114,7 +3883,6 @@ define <vscale x 4 x half> @test_vlseg5_mask_nxv4f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg5.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4132,7 +3900,6 @@ define <vscale x 4 x half> @test_vlseg6_nxv4f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg6.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4151,7 +3918,6 @@ define <vscale x 4 x half> @test_vlseg6_mask_nxv4f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg6.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4169,7 +3935,6 @@ define <vscale x 4 x half> @test_vlseg7_nxv4f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg7.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4189,7 +3954,6 @@ define <vscale x 4 x half> @test_vlseg7_mask_nxv4f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg7.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4207,7 +3971,6 @@ define <vscale x 4 x half> @test_vlseg8_nxv4f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg8.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4228,7 +3991,6 @@ define <vscale x 4 x half> @test_vlseg8_mask_nxv4f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg8.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4246,7 +4008,6 @@ define <vscale x 2 x half> @test_vlseg2_nxv2f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg2.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -4261,7 +4022,6 @@ define <vscale x 2 x half> @test_vlseg2_mask_nxv2f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg2.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -4279,7 +4039,6 @@ define <vscale x 2 x half> @test_vlseg3_nxv2f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg3.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -4295,7 +4054,6 @@ define <vscale x 2 x half> @test_vlseg3_mask_nxv2f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg3.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -4313,7 +4071,6 @@ define <vscale x 2 x half> @test_vlseg4_nxv2f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg4.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -4330,7 +4087,6 @@ define <vscale x 2 x half> @test_vlseg4_mask_nxv2f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg4.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -4348,7 +4104,6 @@ define <vscale x 2 x half> @test_vlseg5_nxv2f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg5.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -4366,7 +4121,6 @@ define <vscale x 2 x half> @test_vlseg5_mask_nxv2f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg5.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -4384,7 +4138,6 @@ define <vscale x 2 x half> @test_vlseg6_nxv2f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg6.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -4403,7 +4156,6 @@ define <vscale x 2 x half> @test_vlseg6_mask_nxv2f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg6.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -4421,7 +4173,6 @@ define <vscale x 2 x half> @test_vlseg7_nxv2f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg7.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -4441,7 +4192,6 @@ define <vscale x 2 x half> @test_vlseg7_mask_nxv2f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg7.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -4459,7 +4209,6 @@ define <vscale x 2 x half> @test_vlseg8_nxv2f16(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg8.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -4480,7 +4229,6 @@ define <vscale x 2 x half> @test_vlseg8_mask_nxv2f16(ptr %base, i32 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg8.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -4498,7 +4246,6 @@ define <vscale x 4 x float> @test_vlseg2_nxv4f32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vlseg2e32.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlseg2.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i32 %vl)
@@ -4513,7 +4260,6 @@ define <vscale x 4 x float> @test_vlseg2_mask_nxv4f32(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vlseg2e32.v v6, (a0)
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlseg2e32.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlseg2.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i32 %vl)
@@ -4531,7 +4277,6 @@ define <vscale x 4 x float> @test_vlseg3_nxv4f32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vlseg3e32.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlseg3.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i32 %vl)
@@ -4547,7 +4292,6 @@ define <vscale x 4 x float> @test_vlseg3_mask_nxv4f32(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlseg3e32.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlseg3.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i32 %vl)
@@ -4565,7 +4309,6 @@ define <vscale x 4 x float> @test_vlseg4_nxv4f32(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vlseg4e32.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlseg4.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i32 %vl)
@@ -4582,7 +4325,6 @@ define <vscale x 4 x float> @test_vlseg4_mask_nxv4f32(ptr %base, i32 %vl, <vscal
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlseg4e32.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlseg4.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i32 %vl)
diff --git a/llvm/test/CodeGen/RISCV/rvv/vlseg-rv64.ll b/llvm/test/CodeGen/RISCV/rvv/vlseg-rv64.ll
index 12604322bbbcf..33672ce647a2f 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vlseg-rv64.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vlseg-rv64.ll
@@ -10,7 +10,6 @@ define <vscale x 16 x i16> @test_vlseg2_nxv16i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v4, (a0)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vlseg2.nxv16i16(<vscale x 16 x i16> undef, <vscale x 16 x i16> undef, ptr %base, i64 %vl)
@@ -25,7 +24,6 @@ define <vscale x 16 x i16> @test_vlseg2_mask_nxv16i16(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vlseg2e16.v v4, (a0)
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlseg2e16.v v4, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vlseg2.nxv16i16(<vscale x 16 x i16> undef, <vscale x 16 x i16> undef, ptr %base, i64 %vl)
@@ -43,7 +41,6 @@ define <vscale x 4 x i32> @test_vlseg2_nxv4i32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vlseg2e32.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlseg2.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i64 %vl)
@@ -58,7 +55,6 @@ define <vscale x 4 x i32> @test_vlseg2_mask_nxv4i32(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e32.v v6, (a0)
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlseg2e32.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlseg2.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i64 %vl)
@@ -76,7 +72,6 @@ define <vscale x 4 x i32> @test_vlseg3_nxv4i32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vlseg3e32.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlseg3.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i64 %vl)
@@ -92,7 +87,6 @@ define <vscale x 4 x i32> @test_vlseg3_mask_nxv4i32(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlseg3e32.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlseg3.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i64 %vl)
@@ -110,7 +104,6 @@ define <vscale x 4 x i32> @test_vlseg4_nxv4i32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vlseg4e32.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlseg4.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i64 %vl)
@@ -127,7 +120,6 @@ define <vscale x 4 x i32> @test_vlseg4_mask_nxv4i32(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlseg4e32.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlseg4.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i64 %vl)
@@ -145,7 +137,6 @@ define <vscale x 16 x i8> @test_vlseg2_nxv16i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vlseg2e8.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlseg2.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i64 %vl)
@@ -160,7 +151,6 @@ define <vscale x 16 x i8> @test_vlseg2_mask_nxv16i8(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e8.v v6, (a0)
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlseg2e8.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlseg2.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i64 %vl)
@@ -178,7 +168,6 @@ define <vscale x 16 x i8> @test_vlseg3_nxv16i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vlseg3e8.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlseg3.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i64 %vl)
@@ -194,7 +183,6 @@ define <vscale x 16 x i8> @test_vlseg3_mask_nxv16i8(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlseg3e8.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlseg3.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i64 %vl)
@@ -212,7 +200,6 @@ define <vscale x 16 x i8> @test_vlseg4_nxv16i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vlseg4e8.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlseg4.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i64 %vl)
@@ -229,7 +216,6 @@ define <vscale x 16 x i8> @test_vlseg4_mask_nxv16i8(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlseg4e8.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlseg4.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i64 %vl)
@@ -247,7 +233,6 @@ define <vscale x 1 x i64> @test_vlseg2_nxv1i64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg2e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlseg2.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -262,7 +247,6 @@ define <vscale x 1 x i64> @test_vlseg2_mask_nxv1i64(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e64.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlseg2.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -280,7 +264,6 @@ define <vscale x 1 x i64> @test_vlseg3_nxv1i64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg3e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlseg3.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -296,7 +279,6 @@ define <vscale x 1 x i64> @test_vlseg3_mask_nxv1i64(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlseg3.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -314,7 +296,6 @@ define <vscale x 1 x i64> @test_vlseg4_nxv1i64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg4e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlseg4.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -331,7 +312,6 @@ define <vscale x 1 x i64> @test_vlseg4_mask_nxv1i64(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlseg4.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -349,7 +329,6 @@ define <vscale x 1 x i64> @test_vlseg5_nxv1i64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg5e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlseg5.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -367,7 +346,6 @@ define <vscale x 1 x i64> @test_vlseg5_mask_nxv1i64(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlseg5.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -385,7 +363,6 @@ define <vscale x 1 x i64> @test_vlseg6_nxv1i64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg6e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlseg6.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -404,7 +381,6 @@ define <vscale x 1 x i64> @test_vlseg6_mask_nxv1i64(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlseg6.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -422,7 +398,6 @@ define <vscale x 1 x i64> @test_vlseg7_nxv1i64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg7e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlseg7.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -442,7 +417,6 @@ define <vscale x 1 x i64> @test_vlseg7_mask_nxv1i64(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlseg7.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -460,7 +434,6 @@ define <vscale x 1 x i64> @test_vlseg8_nxv1i64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg8e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlseg8.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef ,<vscale x 1 x i64> undef ,<vscale x 1 x i64> undef, <vscale x 1 x i64> undef ,<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -481,7 +454,6 @@ define <vscale x 1 x i64> @test_vlseg8_mask_nxv1i64(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlseg8.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef ,<vscale x 1 x i64> undef ,<vscale x 1 x i64> undef, <vscale x 1 x i64> undef ,<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -499,7 +471,6 @@ define <vscale x 1 x i32> @test_vlseg2_nxv1i32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg2.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -514,7 +485,6 @@ define <vscale x 1 x i32> @test_vlseg2_mask_nxv1i32(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg2.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -532,7 +502,6 @@ define <vscale x 1 x i32> @test_vlseg3_nxv1i32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg3e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg3.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -548,7 +517,6 @@ define <vscale x 1 x i32> @test_vlseg3_mask_nxv1i32(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg3.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -566,7 +534,6 @@ define <vscale x 1 x i32> @test_vlseg4_nxv1i32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg4e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg4.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -583,7 +550,6 @@ define <vscale x 1 x i32> @test_vlseg4_mask_nxv1i32(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg4.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -601,7 +567,6 @@ define <vscale x 1 x i32> @test_vlseg5_nxv1i32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg5e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg5.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -619,7 +584,6 @@ define <vscale x 1 x i32> @test_vlseg5_mask_nxv1i32(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg5.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -637,7 +601,6 @@ define <vscale x 1 x i32> @test_vlseg6_nxv1i32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg6e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg6.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -656,7 +619,6 @@ define <vscale x 1 x i32> @test_vlseg6_mask_nxv1i32(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg6.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -674,7 +636,6 @@ define <vscale x 1 x i32> @test_vlseg7_nxv1i32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg7e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg7.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -694,7 +655,6 @@ define <vscale x 1 x i32> @test_vlseg7_mask_nxv1i32(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg7.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -712,7 +672,6 @@ define <vscale x 1 x i32> @test_vlseg8_nxv1i32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg8e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg8.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -733,7 +692,6 @@ define <vscale x 1 x i32> @test_vlseg8_mask_nxv1i32(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlseg8.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -751,7 +709,6 @@ define <vscale x 8 x i16> @test_vlseg2_nxv8i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlseg2.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i64 %vl)
@@ -766,7 +723,6 @@ define <vscale x 8 x i16> @test_vlseg2_mask_nxv8i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e16.v v6, (a0)
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlseg2e16.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlseg2.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i64 %vl)
@@ -784,7 +740,6 @@ define <vscale x 8 x i16> @test_vlseg3_nxv8i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vlseg3e16.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlseg3.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i64 %vl)
@@ -800,7 +755,6 @@ define <vscale x 8 x i16> @test_vlseg3_mask_nxv8i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlseg3e16.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlseg3.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i64 %vl)
@@ -818,7 +772,6 @@ define <vscale x 8 x i16> @test_vlseg4_nxv8i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vlseg4e16.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlseg4.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i64 %vl)
@@ -835,7 +788,6 @@ define <vscale x 8 x i16> @test_vlseg4_mask_nxv8i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlseg4e16.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlseg4.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i64 %vl)
@@ -853,7 +805,6 @@ define <vscale x 4 x i8> @test_vlseg2_nxv4i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg2.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -868,7 +819,6 @@ define <vscale x 4 x i8> @test_vlseg2_mask_nxv4i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg2.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -886,7 +836,6 @@ define <vscale x 4 x i8> @test_vlseg3_nxv4i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg3e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg3.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -902,7 +851,6 @@ define <vscale x 4 x i8> @test_vlseg3_mask_nxv4i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg3.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -920,7 +868,6 @@ define <vscale x 4 x i8> @test_vlseg4_nxv4i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg4e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg4.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -937,7 +884,6 @@ define <vscale x 4 x i8> @test_vlseg4_mask_nxv4i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg4.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -955,7 +901,6 @@ define <vscale x 4 x i8> @test_vlseg5_nxv4i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg5e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg5.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -973,7 +918,6 @@ define <vscale x 4 x i8> @test_vlseg5_mask_nxv4i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg5.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -991,7 +935,6 @@ define <vscale x 4 x i8> @test_vlseg6_nxv4i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg6e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg6.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -1010,7 +953,6 @@ define <vscale x 4 x i8> @test_vlseg6_mask_nxv4i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg6.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -1028,7 +970,6 @@ define <vscale x 4 x i8> @test_vlseg7_nxv4i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg7e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg7.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -1048,7 +989,6 @@ define <vscale x 4 x i8> @test_vlseg7_mask_nxv4i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg7.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -1066,7 +1006,6 @@ define <vscale x 4 x i8> @test_vlseg8_nxv4i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlseg8e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg8.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -1087,7 +1026,6 @@ define <vscale x 4 x i8> @test_vlseg8_mask_nxv4i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlseg8.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -1105,7 +1043,6 @@ define <vscale x 1 x i16> @test_vlseg2_nxv1i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg2.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1120,7 +1057,6 @@ define <vscale x 1 x i16> @test_vlseg2_mask_nxv1i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg2.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1138,7 +1074,6 @@ define <vscale x 1 x i16> @test_vlseg3_nxv1i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg3.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1154,7 +1089,6 @@ define <vscale x 1 x i16> @test_vlseg3_mask_nxv1i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg3.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1172,7 +1106,6 @@ define <vscale x 1 x i16> @test_vlseg4_nxv1i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg4.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1189,7 +1122,6 @@ define <vscale x 1 x i16> @test_vlseg4_mask_nxv1i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg4.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1207,7 +1139,6 @@ define <vscale x 1 x i16> @test_vlseg5_nxv1i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg5.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1225,7 +1156,6 @@ define <vscale x 1 x i16> @test_vlseg5_mask_nxv1i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg5.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1243,7 +1173,6 @@ define <vscale x 1 x i16> @test_vlseg6_nxv1i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg6.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1262,7 +1191,6 @@ define <vscale x 1 x i16> @test_vlseg6_mask_nxv1i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg6.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1280,7 +1208,6 @@ define <vscale x 1 x i16> @test_vlseg7_nxv1i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg7.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1300,7 +1227,6 @@ define <vscale x 1 x i16> @test_vlseg7_mask_nxv1i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg7.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1318,7 +1244,6 @@ define <vscale x 1 x i16> @test_vlseg8_nxv1i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg8.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1339,7 +1264,6 @@ define <vscale x 1 x i16> @test_vlseg8_mask_nxv1i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlseg8.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1357,7 +1281,6 @@ define <vscale x 2 x i32> @test_vlseg2_nxv2i32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg2.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1372,7 +1295,6 @@ define <vscale x 2 x i32> @test_vlseg2_mask_nxv2i32(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg2.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1390,7 +1312,6 @@ define <vscale x 2 x i32> @test_vlseg3_nxv2i32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg3e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg3.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1406,7 +1327,6 @@ define <vscale x 2 x i32> @test_vlseg3_mask_nxv2i32(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg3.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1424,7 +1344,6 @@ define <vscale x 2 x i32> @test_vlseg4_nxv2i32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg4e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg4.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1441,7 +1360,6 @@ define <vscale x 2 x i32> @test_vlseg4_mask_nxv2i32(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg4.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1459,7 +1377,6 @@ define <vscale x 2 x i32> @test_vlseg5_nxv2i32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg5e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg5.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1477,7 +1394,6 @@ define <vscale x 2 x i32> @test_vlseg5_mask_nxv2i32(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg5.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1495,7 +1411,6 @@ define <vscale x 2 x i32> @test_vlseg6_nxv2i32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg6e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg6.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1514,7 +1429,6 @@ define <vscale x 2 x i32> @test_vlseg6_mask_nxv2i32(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg6.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1532,7 +1446,6 @@ define <vscale x 2 x i32> @test_vlseg7_nxv2i32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg7e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg7.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1552,7 +1465,6 @@ define <vscale x 2 x i32> @test_vlseg7_mask_nxv2i32(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg7.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1570,7 +1482,6 @@ define <vscale x 2 x i32> @test_vlseg8_nxv2i32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg8e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg8.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1591,7 +1502,6 @@ define <vscale x 2 x i32> @test_vlseg8_mask_nxv2i32(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlseg8.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1609,7 +1519,6 @@ define <vscale x 8 x i8> @test_vlseg2_nxv8i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg2.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -1624,7 +1533,6 @@ define <vscale x 8 x i8> @test_vlseg2_mask_nxv8i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg2.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -1642,7 +1550,6 @@ define <vscale x 8 x i8> @test_vlseg3_nxv8i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vlseg3e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg3.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -1658,7 +1565,6 @@ define <vscale x 8 x i8> @test_vlseg3_mask_nxv8i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg3.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -1676,7 +1582,6 @@ define <vscale x 8 x i8> @test_vlseg4_nxv8i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vlseg4e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg4.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -1693,7 +1598,6 @@ define <vscale x 8 x i8> @test_vlseg4_mask_nxv8i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg4.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -1711,7 +1615,6 @@ define <vscale x 8 x i8> @test_vlseg5_nxv8i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vlseg5e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg5.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -1729,7 +1632,6 @@ define <vscale x 8 x i8> @test_vlseg5_mask_nxv8i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg5.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -1747,7 +1649,6 @@ define <vscale x 8 x i8> @test_vlseg6_nxv8i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vlseg6e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg6.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -1766,7 +1667,6 @@ define <vscale x 8 x i8> @test_vlseg6_mask_nxv8i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg6.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -1784,7 +1684,6 @@ define <vscale x 8 x i8> @test_vlseg7_nxv8i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vlseg7e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg7.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -1804,7 +1703,6 @@ define <vscale x 8 x i8> @test_vlseg7_mask_nxv8i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg7.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -1822,7 +1720,6 @@ define <vscale x 8 x i8> @test_vlseg8_nxv8i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vlseg8e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg8.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -1843,7 +1740,6 @@ define <vscale x 8 x i8> @test_vlseg8_mask_nxv8i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlseg8.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -1861,7 +1757,6 @@ define <vscale x 4 x i64> @test_vlseg2_nxv4i64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
 ; CHECK-NEXT:    vlseg2e64.v v4, (a0)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i64>,<vscale x 4 x i64>} @llvm.riscv.vlseg2.nxv4i64(<vscale x 4 x i64> undef, <vscale x 4 x i64> undef, ptr %base, i64 %vl)
@@ -1876,7 +1771,6 @@ define <vscale x 4 x i64> @test_vlseg2_mask_nxv4i64(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e64.v v4, (a0)
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlseg2e64.v v4, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i64>,<vscale x 4 x i64>} @llvm.riscv.vlseg2.nxv4i64(<vscale x 4 x i64> undef, <vscale x 4 x i64> undef, ptr %base, i64 %vl)
@@ -1894,7 +1788,6 @@ define <vscale x 4 x i16> @test_vlseg2_nxv4i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg2.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -1909,7 +1802,6 @@ define <vscale x 4 x i16> @test_vlseg2_mask_nxv4i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg2.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -1927,7 +1819,6 @@ define <vscale x 4 x i16> @test_vlseg3_nxv4i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg3.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -1943,7 +1834,6 @@ define <vscale x 4 x i16> @test_vlseg3_mask_nxv4i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg3.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -1961,7 +1851,6 @@ define <vscale x 4 x i16> @test_vlseg4_nxv4i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg4.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -1978,7 +1867,6 @@ define <vscale x 4 x i16> @test_vlseg4_mask_nxv4i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg4.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -1996,7 +1884,6 @@ define <vscale x 4 x i16> @test_vlseg5_nxv4i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg5.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -2014,7 +1901,6 @@ define <vscale x 4 x i16> @test_vlseg5_mask_nxv4i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg5.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -2032,7 +1918,6 @@ define <vscale x 4 x i16> @test_vlseg6_nxv4i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg6.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -2051,7 +1936,6 @@ define <vscale x 4 x i16> @test_vlseg6_mask_nxv4i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg6.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -2069,7 +1953,6 @@ define <vscale x 4 x i16> @test_vlseg7_nxv4i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg7.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -2089,7 +1972,6 @@ define <vscale x 4 x i16> @test_vlseg7_mask_nxv4i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg7.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -2107,7 +1989,6 @@ define <vscale x 4 x i16> @test_vlseg8_nxv4i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg8.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -2128,7 +2009,6 @@ define <vscale x 4 x i16> @test_vlseg8_mask_nxv4i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlseg8.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -2146,7 +2026,6 @@ define <vscale x 1 x i8> @test_vlseg2_nxv1i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg2.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2161,7 +2040,6 @@ define <vscale x 1 x i8> @test_vlseg2_mask_nxv1i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg2.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2179,7 +2057,6 @@ define <vscale x 1 x i8> @test_vlseg3_nxv1i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlseg3e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg3.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2195,7 +2072,6 @@ define <vscale x 1 x i8> @test_vlseg3_mask_nxv1i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg3.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2213,7 +2089,6 @@ define <vscale x 1 x i8> @test_vlseg4_nxv1i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlseg4e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg4.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2230,7 +2105,6 @@ define <vscale x 1 x i8> @test_vlseg4_mask_nxv1i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg4.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2248,7 +2122,6 @@ define <vscale x 1 x i8> @test_vlseg5_nxv1i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlseg5e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg5.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2266,7 +2139,6 @@ define <vscale x 1 x i8> @test_vlseg5_mask_nxv1i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg5.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2284,7 +2156,6 @@ define <vscale x 1 x i8> @test_vlseg6_nxv1i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlseg6e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg6.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2303,7 +2174,6 @@ define <vscale x 1 x i8> @test_vlseg6_mask_nxv1i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg6.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2321,7 +2191,6 @@ define <vscale x 1 x i8> @test_vlseg7_nxv1i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlseg7e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg7.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2341,7 +2210,6 @@ define <vscale x 1 x i8> @test_vlseg7_mask_nxv1i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg7.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2359,7 +2227,6 @@ define <vscale x 1 x i8> @test_vlseg8_nxv1i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlseg8e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg8.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2380,7 +2247,6 @@ define <vscale x 1 x i8> @test_vlseg8_mask_nxv1i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlseg8.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2398,7 +2264,6 @@ define <vscale x 2 x i8> @test_vlseg2_nxv2i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg2.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2413,7 +2278,6 @@ define <vscale x 2 x i8> @test_vlseg2_mask_nxv2i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg2.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2431,7 +2295,6 @@ define <vscale x 2 x i8> @test_vlseg3_nxv2i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlseg3e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg3.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2447,7 +2310,6 @@ define <vscale x 2 x i8> @test_vlseg3_mask_nxv2i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg3.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2465,7 +2327,6 @@ define <vscale x 2 x i8> @test_vlseg4_nxv2i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlseg4e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg4.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2482,7 +2343,6 @@ define <vscale x 2 x i8> @test_vlseg4_mask_nxv2i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg4.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2500,7 +2360,6 @@ define <vscale x 2 x i8> @test_vlseg5_nxv2i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlseg5e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg5.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2518,7 +2377,6 @@ define <vscale x 2 x i8> @test_vlseg5_mask_nxv2i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg5.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2536,7 +2394,6 @@ define <vscale x 2 x i8> @test_vlseg6_nxv2i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlseg6e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg6.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2555,7 +2412,6 @@ define <vscale x 2 x i8> @test_vlseg6_mask_nxv2i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg6.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2573,7 +2429,6 @@ define <vscale x 2 x i8> @test_vlseg7_nxv2i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlseg7e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg7.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2593,7 +2448,6 @@ define <vscale x 2 x i8> @test_vlseg7_mask_nxv2i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg7.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2611,7 +2465,6 @@ define <vscale x 2 x i8> @test_vlseg8_nxv2i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlseg8e8.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg8.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2632,7 +2485,6 @@ define <vscale x 2 x i8> @test_vlseg8_mask_nxv2i8(ptr %base, i64 %vl, <vscale x
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e8.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlseg8.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2650,7 +2502,6 @@ define <vscale x 8 x i32> @test_vlseg2_nxv8i32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vlseg2e32.v v4, (a0)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vlseg2.nxv8i32(<vscale x 8 x i32> undef, <vscale x 8 x i32> undef, ptr %base, i64 %vl)
@@ -2665,7 +2516,6 @@ define <vscale x 8 x i32> @test_vlseg2_mask_nxv8i32(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e32.v v4, (a0)
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlseg2e32.v v4, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vlseg2.nxv8i32(<vscale x 8 x i32> undef, <vscale x 8 x i32> undef, ptr %base, i64 %vl)
@@ -2683,7 +2533,6 @@ define <vscale x 32 x i8> @test_vlseg2_nxv32i8(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
 ; CHECK-NEXT:    vlseg2e8.v v4, (a0)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>} @llvm.riscv.vlseg2.nxv32i8(<vscale x 32 x i8> undef, <vscale x 32 x i8> undef, ptr %base, i64 %vl)
@@ -2698,7 +2547,6 @@ define <vscale x 32 x i8> @test_vlseg2_mask_nxv32i8(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e8.v v4, (a0)
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlseg2e8.v v4, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>} @llvm.riscv.vlseg2.nxv32i8(<vscale x 32 x i8> undef, <vscale x 32 x i8> undef, ptr %base, i64 %vl)
@@ -2716,7 +2564,6 @@ define <vscale x 2 x i16> @test_vlseg2_nxv2i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg2.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -2731,7 +2578,6 @@ define <vscale x 2 x i16> @test_vlseg2_mask_nxv2i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg2.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -2749,7 +2595,6 @@ define <vscale x 2 x i16> @test_vlseg3_nxv2i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg3.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -2765,7 +2610,6 @@ define <vscale x 2 x i16> @test_vlseg3_mask_nxv2i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg3.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -2783,7 +2627,6 @@ define <vscale x 2 x i16> @test_vlseg4_nxv2i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg4.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -2800,7 +2643,6 @@ define <vscale x 2 x i16> @test_vlseg4_mask_nxv2i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg4.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -2818,7 +2660,6 @@ define <vscale x 2 x i16> @test_vlseg5_nxv2i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg5.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -2836,7 +2677,6 @@ define <vscale x 2 x i16> @test_vlseg5_mask_nxv2i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg5.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -2854,7 +2694,6 @@ define <vscale x 2 x i16> @test_vlseg6_nxv2i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg6.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -2873,7 +2712,6 @@ define <vscale x 2 x i16> @test_vlseg6_mask_nxv2i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg6.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -2891,7 +2729,6 @@ define <vscale x 2 x i16> @test_vlseg7_nxv2i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg7.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -2911,7 +2748,6 @@ define <vscale x 2 x i16> @test_vlseg7_mask_nxv2i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg7.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -2929,7 +2765,6 @@ define <vscale x 2 x i16> @test_vlseg8_nxv2i16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg8.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -2950,7 +2785,6 @@ define <vscale x 2 x i16> @test_vlseg8_mask_nxv2i16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlseg8.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -2968,7 +2802,6 @@ define <vscale x 2 x i64> @test_vlseg2_nxv2i64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vlseg2e64.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vlseg2.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i64> undef, ptr %base, i64 %vl)
@@ -2983,7 +2816,6 @@ define <vscale x 2 x i64> @test_vlseg2_mask_nxv2i64(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e64.v v6, (a0)
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlseg2e64.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vlseg2.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i64> undef, ptr %base, i64 %vl)
@@ -3001,7 +2833,6 @@ define <vscale x 2 x i64> @test_vlseg3_nxv2i64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vlseg3e64.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vlseg3.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, ptr %base, i64 %vl)
@@ -3017,7 +2848,6 @@ define <vscale x 2 x i64> @test_vlseg3_mask_nxv2i64(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlseg3e64.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vlseg3.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, ptr %base, i64 %vl)
@@ -3035,7 +2865,6 @@ define <vscale x 2 x i64> @test_vlseg4_nxv2i64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vlseg4e64.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vlseg4.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, ptr %base, i64 %vl)
@@ -3052,7 +2881,6 @@ define <vscale x 2 x i64> @test_vlseg4_mask_nxv2i64(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlseg4e64.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vlseg4.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, ptr %base, i64 %vl)
@@ -3070,7 +2898,6 @@ define <vscale x 16 x half> @test_vlseg2_nxv16f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v4, (a0)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vlseg2.nxv16f16(<vscale x 16 x half> undef, <vscale x 16 x half> undef, ptr %base, i64 %vl)
@@ -3085,7 +2912,6 @@ define <vscale x 16 x half> @test_vlseg2_mask_nxv16f16(ptr %base, i64 %vl, <vsca
 ; CHECK-NEXT:    vlseg2e16.v v4, (a0)
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlseg2e16.v v4, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vlseg2.nxv16f16(<vscale x 16 x half> undef, <vscale x 16 x half> undef, ptr %base, i64 %vl)
@@ -3103,7 +2929,6 @@ define <vscale x 4 x double> @test_vlseg2_nxv4f64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
 ; CHECK-NEXT:    vlseg2e64.v v4, (a0)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vlseg2.nxv4f64(<vscale x 4 x double> undef, <vscale x 4 x double> undef, ptr %base, i64 %vl)
@@ -3118,7 +2943,6 @@ define <vscale x 4 x double> @test_vlseg2_mask_nxv4f64(ptr %base, i64 %vl, <vsca
 ; CHECK-NEXT:    vlseg2e64.v v4, (a0)
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlseg2e64.v v4, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vlseg2.nxv4f64(<vscale x 4 x double> undef, <vscale x 4 x double> undef, ptr %base, i64 %vl)
@@ -3136,7 +2960,6 @@ define <vscale x 1 x double> @test_vlseg2_nxv1f64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg2e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg2.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3151,7 +2974,6 @@ define <vscale x 1 x double> @test_vlseg2_mask_nxv1f64(ptr %base, i64 %vl, <vsca
 ; CHECK-NEXT:    vlseg2e64.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg2.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3169,7 +2991,6 @@ define <vscale x 1 x double> @test_vlseg3_nxv1f64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg3e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg3.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3185,7 +3006,6 @@ define <vscale x 1 x double> @test_vlseg3_mask_nxv1f64(ptr %base, i64 %vl, <vsca
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg3.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3203,7 +3023,6 @@ define <vscale x 1 x double> @test_vlseg4_nxv1f64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg4e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg4.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3220,7 +3039,6 @@ define <vscale x 1 x double> @test_vlseg4_mask_nxv1f64(ptr %base, i64 %vl, <vsca
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg4.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3238,7 +3056,6 @@ define <vscale x 1 x double> @test_vlseg5_nxv1f64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg5e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg5.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3256,7 +3073,6 @@ define <vscale x 1 x double> @test_vlseg5_mask_nxv1f64(ptr %base, i64 %vl, <vsca
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg5.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3274,7 +3090,6 @@ define <vscale x 1 x double> @test_vlseg6_nxv1f64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg6e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg6.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3293,7 +3108,6 @@ define <vscale x 1 x double> @test_vlseg6_mask_nxv1f64(ptr %base, i64 %vl, <vsca
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg6.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3311,7 +3125,6 @@ define <vscale x 1 x double> @test_vlseg7_nxv1f64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg7e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg7.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3331,7 +3144,6 @@ define <vscale x 1 x double> @test_vlseg7_mask_nxv1f64(ptr %base, i64 %vl, <vsca
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg7.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3349,7 +3161,6 @@ define <vscale x 1 x double> @test_vlseg8_nxv1f64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vlseg8e64.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg8.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3370,7 +3181,6 @@ define <vscale x 1 x double> @test_vlseg8_mask_nxv1f64(ptr %base, i64 %vl, <vsca
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e64.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlseg8.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3388,7 +3198,6 @@ define <vscale x 2 x float> @test_vlseg2_nxv2f32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg2.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -3403,7 +3212,6 @@ define <vscale x 2 x float> @test_vlseg2_mask_nxv2f32(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg2.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -3421,7 +3229,6 @@ define <vscale x 2 x float> @test_vlseg3_nxv2f32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg3e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg3.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -3437,7 +3244,6 @@ define <vscale x 2 x float> @test_vlseg3_mask_nxv2f32(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg3.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -3455,7 +3261,6 @@ define <vscale x 2 x float> @test_vlseg4_nxv2f32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg4e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg4.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -3472,7 +3277,6 @@ define <vscale x 2 x float> @test_vlseg4_mask_nxv2f32(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg4.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -3490,7 +3294,6 @@ define <vscale x 2 x float> @test_vlseg5_nxv2f32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg5e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg5.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -3508,7 +3311,6 @@ define <vscale x 2 x float> @test_vlseg5_mask_nxv2f32(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg5.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -3526,7 +3328,6 @@ define <vscale x 2 x float> @test_vlseg6_nxv2f32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg6e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg6.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -3545,7 +3346,6 @@ define <vscale x 2 x float> @test_vlseg6_mask_nxv2f32(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg6.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -3563,7 +3363,6 @@ define <vscale x 2 x float> @test_vlseg7_nxv2f32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg7e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg7.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -3583,7 +3382,6 @@ define <vscale x 2 x float> @test_vlseg7_mask_nxv2f32(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg7.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -3601,7 +3399,6 @@ define <vscale x 2 x float> @test_vlseg8_nxv2f32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vlseg8e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg8.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -3622,7 +3419,6 @@ define <vscale x 2 x float> @test_vlseg8_mask_nxv2f32(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlseg8.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -3640,7 +3436,6 @@ define <vscale x 1 x half> @test_vlseg2_nxv1f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg2.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -3655,7 +3450,6 @@ define <vscale x 1 x half> @test_vlseg2_mask_nxv1f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg2.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -3673,7 +3467,6 @@ define <vscale x 1 x half> @test_vlseg3_nxv1f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg3.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -3689,7 +3482,6 @@ define <vscale x 1 x half> @test_vlseg3_mask_nxv1f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg3.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -3707,7 +3499,6 @@ define <vscale x 1 x half> @test_vlseg4_nxv1f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg4.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -3724,7 +3515,6 @@ define <vscale x 1 x half> @test_vlseg4_mask_nxv1f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg4.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -3742,7 +3532,6 @@ define <vscale x 1 x half> @test_vlseg5_nxv1f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg5.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -3760,7 +3549,6 @@ define <vscale x 1 x half> @test_vlseg5_mask_nxv1f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg5.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -3778,7 +3566,6 @@ define <vscale x 1 x half> @test_vlseg6_nxv1f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg6.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -3797,7 +3584,6 @@ define <vscale x 1 x half> @test_vlseg6_mask_nxv1f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg6.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -3815,7 +3601,6 @@ define <vscale x 1 x half> @test_vlseg7_nxv1f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg7.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -3835,7 +3620,6 @@ define <vscale x 1 x half> @test_vlseg7_mask_nxv1f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg7.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -3853,7 +3637,6 @@ define <vscale x 1 x half> @test_vlseg8_nxv1f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg8.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -3874,7 +3657,6 @@ define <vscale x 1 x half> @test_vlseg8_mask_nxv1f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlseg8.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -3892,7 +3674,6 @@ define <vscale x 1 x float> @test_vlseg2_nxv1f32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg2.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -3907,7 +3688,6 @@ define <vscale x 1 x float> @test_vlseg2_mask_nxv1f32(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg2.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -3925,7 +3705,6 @@ define <vscale x 1 x float> @test_vlseg3_nxv1f32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg3e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg3.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -3941,7 +3720,6 @@ define <vscale x 1 x float> @test_vlseg3_mask_nxv1f32(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg3.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -3959,7 +3737,6 @@ define <vscale x 1 x float> @test_vlseg4_nxv1f32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg4e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg4.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -3976,7 +3753,6 @@ define <vscale x 1 x float> @test_vlseg4_mask_nxv1f32(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg4.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -3994,7 +3770,6 @@ define <vscale x 1 x float> @test_vlseg5_nxv1f32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg5e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg5.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -4012,7 +3787,6 @@ define <vscale x 1 x float> @test_vlseg5_mask_nxv1f32(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg5.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -4030,7 +3804,6 @@ define <vscale x 1 x float> @test_vlseg6_nxv1f32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg6e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg6.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -4049,7 +3822,6 @@ define <vscale x 1 x float> @test_vlseg6_mask_nxv1f32(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg6.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -4067,7 +3839,6 @@ define <vscale x 1 x float> @test_vlseg7_nxv1f32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg7e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg7.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -4087,7 +3858,6 @@ define <vscale x 1 x float> @test_vlseg7_mask_nxv1f32(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg7.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -4105,7 +3875,6 @@ define <vscale x 1 x float> @test_vlseg8_nxv1f32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlseg8e32.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg8.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -4126,7 +3895,6 @@ define <vscale x 1 x float> @test_vlseg8_mask_nxv1f32(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e32.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlseg8.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -4144,7 +3912,6 @@ define <vscale x 8 x half> @test_vlseg2_nxv8f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlseg2.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i64 %vl)
@@ -4159,7 +3926,6 @@ define <vscale x 8 x half> @test_vlseg2_mask_nxv8f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e16.v v6, (a0)
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlseg2e16.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlseg2.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i64 %vl)
@@ -4177,7 +3943,6 @@ define <vscale x 8 x half> @test_vlseg3_nxv8f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vlseg3e16.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlseg3.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i64 %vl)
@@ -4193,7 +3958,6 @@ define <vscale x 8 x half> @test_vlseg3_mask_nxv8f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlseg3e16.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlseg3.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i64 %vl)
@@ -4211,7 +3975,6 @@ define <vscale x 8 x half> @test_vlseg4_nxv8f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vlseg4e16.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlseg4.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i64 %vl)
@@ -4228,7 +3991,6 @@ define <vscale x 8 x half> @test_vlseg4_mask_nxv8f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlseg4e16.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlseg4.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i64 %vl)
@@ -4246,7 +4008,6 @@ define <vscale x 8 x float> @test_vlseg2_nxv8f32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vlseg2e32.v v4, (a0)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vlseg2.nxv8f32(<vscale x 8 x float> undef, <vscale x 8 x float> undef, ptr %base, i64 %vl)
@@ -4261,7 +4022,6 @@ define <vscale x 8 x float> @test_vlseg2_mask_nxv8f32(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vlseg2e32.v v4, (a0)
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlseg2e32.v v4, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vlseg2.nxv8f32(<vscale x 8 x float> undef, <vscale x 8 x float> undef, ptr %base, i64 %vl)
@@ -4279,7 +4039,6 @@ define <vscale x 2 x double> @test_vlseg2_nxv2f64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vlseg2e64.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlseg2.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i64 %vl)
@@ -4294,7 +4053,6 @@ define <vscale x 2 x double> @test_vlseg2_mask_nxv2f64(ptr %base, i64 %vl, <vsca
 ; CHECK-NEXT:    vlseg2e64.v v6, (a0)
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlseg2e64.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlseg2.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i64 %vl)
@@ -4312,7 +4070,6 @@ define <vscale x 2 x double> @test_vlseg3_nxv2f64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vlseg3e64.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlseg3.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i64 %vl)
@@ -4328,7 +4085,6 @@ define <vscale x 2 x double> @test_vlseg3_mask_nxv2f64(ptr %base, i64 %vl, <vsca
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlseg3e64.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlseg3.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i64 %vl)
@@ -4346,7 +4102,6 @@ define <vscale x 2 x double> @test_vlseg4_nxv2f64(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vlseg4e64.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlseg4.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i64 %vl)
@@ -4363,7 +4118,6 @@ define <vscale x 2 x double> @test_vlseg4_mask_nxv2f64(ptr %base, i64 %vl, <vsca
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlseg4e64.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlseg4.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i64 %vl)
@@ -4381,7 +4135,6 @@ define <vscale x 4 x half> @test_vlseg2_nxv4f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg2.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -4396,7 +4149,6 @@ define <vscale x 4 x half> @test_vlseg2_mask_nxv4f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg2.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -4414,7 +4166,6 @@ define <vscale x 4 x half> @test_vlseg3_nxv4f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg3.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -4430,7 +4181,6 @@ define <vscale x 4 x half> @test_vlseg3_mask_nxv4f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg3.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -4448,7 +4198,6 @@ define <vscale x 4 x half> @test_vlseg4_nxv4f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg4.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -4465,7 +4214,6 @@ define <vscale x 4 x half> @test_vlseg4_mask_nxv4f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg4.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -4483,7 +4231,6 @@ define <vscale x 4 x half> @test_vlseg5_nxv4f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg5.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -4501,7 +4248,6 @@ define <vscale x 4 x half> @test_vlseg5_mask_nxv4f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg5.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -4519,7 +4265,6 @@ define <vscale x 4 x half> @test_vlseg6_nxv4f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg6.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -4538,7 +4283,6 @@ define <vscale x 4 x half> @test_vlseg6_mask_nxv4f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg6.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -4556,7 +4300,6 @@ define <vscale x 4 x half> @test_vlseg7_nxv4f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg7.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -4576,7 +4319,6 @@ define <vscale x 4 x half> @test_vlseg7_mask_nxv4f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg7.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -4594,7 +4336,6 @@ define <vscale x 4 x half> @test_vlseg8_nxv4f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg8.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -4615,7 +4356,6 @@ define <vscale x 4 x half> @test_vlseg8_mask_nxv4f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlseg8.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -4633,7 +4373,6 @@ define <vscale x 2 x half> @test_vlseg2_nxv2f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg2.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -4648,7 +4387,6 @@ define <vscale x 2 x half> @test_vlseg2_mask_nxv2f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0)
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlseg2e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg2.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -4666,7 +4404,6 @@ define <vscale x 2 x half> @test_vlseg3_nxv2f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg3.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -4682,7 +4419,6 @@ define <vscale x 2 x half> @test_vlseg3_mask_nxv2f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlseg3e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg3.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -4700,7 +4436,6 @@ define <vscale x 2 x half> @test_vlseg4_nxv2f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg4.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -4717,7 +4452,6 @@ define <vscale x 2 x half> @test_vlseg4_mask_nxv2f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlseg4e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg4.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -4735,7 +4469,6 @@ define <vscale x 2 x half> @test_vlseg5_nxv2f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg5.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -4753,7 +4486,6 @@ define <vscale x 2 x half> @test_vlseg5_mask_nxv2f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlseg5e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg5.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -4771,7 +4503,6 @@ define <vscale x 2 x half> @test_vlseg6_nxv2f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg6.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -4790,7 +4521,6 @@ define <vscale x 2 x half> @test_vlseg6_mask_nxv2f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlseg6e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg6.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -4808,7 +4538,6 @@ define <vscale x 2 x half> @test_vlseg7_nxv2f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg7.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -4828,7 +4557,6 @@ define <vscale x 2 x half> @test_vlseg7_mask_nxv2f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlseg7e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg7.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -4846,7 +4574,6 @@ define <vscale x 2 x half> @test_vlseg8_nxv2f16(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg8.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -4867,7 +4594,6 @@ define <vscale x 2 x half> @test_vlseg8_mask_nxv2f16(ptr %base, i64 %vl, <vscale
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlseg8e16.v v7, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlseg8.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -4885,7 +4611,6 @@ define <vscale x 4 x float> @test_vlseg2_nxv4f32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vlseg2e32.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlseg2.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i64 %vl)
@@ -4900,7 +4625,6 @@ define <vscale x 4 x float> @test_vlseg2_mask_nxv4f32(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vlseg2e32.v v6, (a0)
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlseg2e32.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlseg2.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i64 %vl)
@@ -4918,7 +4642,6 @@ define <vscale x 4 x float> @test_vlseg3_nxv4f32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vlseg3e32.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlseg3.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i64 %vl)
@@ -4934,7 +4657,6 @@ define <vscale x 4 x float> @test_vlseg3_mask_nxv4f32(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlseg3e32.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlseg3.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i64 %vl)
@@ -4952,7 +4674,6 @@ define <vscale x 4 x float> @test_vlseg4_nxv4f32(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vlseg4e32.v v6, (a0)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlseg4.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i64 %vl)
@@ -4969,7 +4690,6 @@ define <vscale x 4 x float> @test_vlseg4_mask_nxv4f32(ptr %base, i64 %vl, <vscal
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlseg4e32.v v6, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlseg4.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i64 %vl)
diff --git a/llvm/test/CodeGen/RISCV/rvv/vlsegff-rv32-dead.ll b/llvm/test/CodeGen/RISCV/rvv/vlsegff-rv32-dead.ll
index 1312fc3f76149..e39d0755da5e3 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vlsegff-rv32-dead.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vlsegff-rv32-dead.ll
@@ -23,7 +23,6 @@ entry:
 define void @test_vlseg2ff_mask_dead_value(<vscale x 16 x i16> %val, ptr %base, i32 %vl, <vscale x 16 x i1> %mask, ptr %outvl) {
 ; CHECK-LABEL: test_vlseg2ff_mask_dead_value:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vlseg2e16ff.v v8, (a0), v0.t
@@ -42,7 +41,6 @@ define <vscale x 16 x i16> @test_vlseg2ff_dead_vl(ptr %base, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vlseg2e16ff.v v4, (a0)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>, i32} @llvm.riscv.vlseg2ff.nxv16i16(<vscale x 16 x i16> undef, <vscale x 16 x i16> undef, ptr %base, i32 %vl)
@@ -56,7 +54,6 @@ define <vscale x 16 x i16> @test_vlseg2ff_mask_dead_vl(<vscale x 16 x i16> %val,
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vlseg2e16ff.v v4, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>, i32} @llvm.riscv.vlseg2ff.mask.nxv16i16(<vscale x 16 x i16> %val,<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -78,7 +75,6 @@ entry:
 define void @test_vlseg2ff_mask_dead_all(<vscale x 16 x i16> %val, ptr %base, i32 %vl, <vscale x 16 x i1> %mask) {
 ; CHECK-LABEL: test_vlseg2ff_mask_dead_all:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vlseg2e16ff.v v8, (a0), v0.t
diff --git a/llvm/test/CodeGen/RISCV/rvv/vlsegff-rv32.ll b/llvm/test/CodeGen/RISCV/rvv/vlsegff-rv32.ll
index 0059e9ad222ab..3ce08eae9c9ee 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vlsegff-rv32.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vlsegff-rv32.ll
@@ -12,7 +12,6 @@ define <vscale x 16 x i16> @test_vlseg2ff_nxv16i16(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg2e16ff.v v4, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>, i32} @llvm.riscv.vlseg2ff.nxv16i16(<vscale x 16 x i16> undef, <vscale x 16 x i16> undef, ptr %base, i32 %vl)
@@ -30,7 +29,6 @@ define <vscale x 16 x i16> @test_vlseg2ff_mask_nxv16i16(<vscale x 16 x i16> %val
 ; CHECK-NEXT:    vlseg2e16ff.v v4, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>, i32} @llvm.riscv.vlseg2ff.mask.nxv16i16(<vscale x 16 x i16> %val,<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -50,7 +48,6 @@ define <vscale x 1 x i8> @test_vlseg2ff_nxv1i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg2e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>, i32} @llvm.riscv.vlseg2ff.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -68,7 +65,6 @@ define <vscale x 1 x i8> @test_vlseg2ff_mask_nxv1i8(<vscale x 1 x i8> %val, ptr
 ; CHECK-NEXT:    vlseg2e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>, i32} @llvm.riscv.vlseg2ff.mask.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -88,7 +84,6 @@ define <vscale x 1 x i8> @test_vlseg3ff_nxv1i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg3e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i32} @llvm.riscv.vlseg3ff.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -102,12 +97,11 @@ define <vscale x 1 x i8> @test_vlseg3ff_mask_nxv1i8(<vscale x 1 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vlseg3e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i32} @llvm.riscv.vlseg3ff.mask.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -127,7 +121,6 @@ define <vscale x 1 x i8> @test_vlseg4ff_nxv1i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg4e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i32} @llvm.riscv.vlseg4ff.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -141,13 +134,12 @@ define <vscale x 1 x i8> @test_vlseg4ff_mask_nxv1i8(<vscale x 1 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vlseg4e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i32} @llvm.riscv.vlseg4ff.mask.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -167,7 +159,6 @@ define <vscale x 1 x i8> @test_vlseg5ff_nxv1i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg5e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i32} @llvm.riscv.vlseg5ff.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -181,14 +172,13 @@ define <vscale x 1 x i8> @test_vlseg5ff_mask_nxv1i8(<vscale x 1 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vlseg5e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i32} @llvm.riscv.vlseg5ff.mask.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -208,7 +198,6 @@ define <vscale x 1 x i8> @test_vlseg6ff_nxv1i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg6e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i32} @llvm.riscv.vlseg6ff.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -222,15 +211,14 @@ define <vscale x 1 x i8> @test_vlseg6ff_mask_nxv1i8(<vscale x 1 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vlseg6e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i32} @llvm.riscv.vlseg6ff.mask.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -250,7 +238,6 @@ define <vscale x 1 x i8> @test_vlseg7ff_nxv1i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg7e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i32} @llvm.riscv.vlseg7ff.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -264,16 +251,15 @@ define <vscale x 1 x i8> @test_vlseg7ff_mask_nxv1i8(<vscale x 1 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vlseg7e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i32} @llvm.riscv.vlseg7ff.mask.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -293,7 +279,6 @@ define <vscale x 1 x i8> @test_vlseg8ff_nxv1i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg8e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i32} @llvm.riscv.vlseg8ff.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %vl)
@@ -307,17 +292,16 @@ define <vscale x 1 x i8> @test_vlseg8ff_mask_nxv1i8(<vscale x 1 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vlseg8e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i32} @llvm.riscv.vlseg8ff.mask.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -337,7 +321,6 @@ define <vscale x 16 x i8> @test_vlseg2ff_nxv16i8(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e8ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>, i32} @llvm.riscv.vlseg2ff.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i32 %vl)
@@ -355,7 +338,6 @@ define <vscale x 16 x i8> @test_vlseg2ff_mask_nxv16i8(<vscale x 16 x i8> %val, p
 ; CHECK-NEXT:    vlseg2e8ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>, i32} @llvm.riscv.vlseg2ff.mask.nxv16i8(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -375,7 +357,6 @@ define <vscale x 16 x i8> @test_vlseg3ff_nxv16i8(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg3e8ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>, i32} @llvm.riscv.vlseg3ff.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i32 %vl)
@@ -389,12 +370,11 @@ define <vscale x 16 x i8> @test_vlseg3ff_mask_nxv16i8(<vscale x 16 x i8> %val, p
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vlseg3e8ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>, i32} @llvm.riscv.vlseg3ff.mask.nxv16i8(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -414,7 +394,6 @@ define <vscale x 16 x i8> @test_vlseg4ff_nxv16i8(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg4e8ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>, i32} @llvm.riscv.vlseg4ff.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i32 %vl)
@@ -428,13 +407,12 @@ define <vscale x 16 x i8> @test_vlseg4ff_mask_nxv16i8(<vscale x 16 x i8> %val, p
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vlseg4e8ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>, i32} @llvm.riscv.vlseg4ff.mask.nxv16i8(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -454,7 +432,6 @@ define <vscale x 2 x i32> @test_vlseg2ff_nxv2i32(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>, i32} @llvm.riscv.vlseg2ff.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -472,7 +449,6 @@ define <vscale x 2 x i32> @test_vlseg2ff_mask_nxv2i32(<vscale x 2 x i32> %val, p
 ; CHECK-NEXT:    vlseg2e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>, i32} @llvm.riscv.vlseg2ff.mask.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -492,7 +468,6 @@ define <vscale x 2 x i32> @test_vlseg3ff_nxv2i32(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg3e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i32} @llvm.riscv.vlseg3ff.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -506,12 +481,11 @@ define <vscale x 2 x i32> @test_vlseg3ff_mask_nxv2i32(<vscale x 2 x i32> %val, p
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg3e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i32} @llvm.riscv.vlseg3ff.mask.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -531,7 +505,6 @@ define <vscale x 2 x i32> @test_vlseg4ff_nxv2i32(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg4e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i32} @llvm.riscv.vlseg4ff.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -545,13 +518,12 @@ define <vscale x 2 x i32> @test_vlseg4ff_mask_nxv2i32(<vscale x 2 x i32> %val, p
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg4e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i32} @llvm.riscv.vlseg4ff.mask.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -571,7 +543,6 @@ define <vscale x 2 x i32> @test_vlseg5ff_nxv2i32(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg5e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i32} @llvm.riscv.vlseg5ff.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -585,14 +556,13 @@ define <vscale x 2 x i32> @test_vlseg5ff_mask_nxv2i32(<vscale x 2 x i32> %val, p
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg5e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i32} @llvm.riscv.vlseg5ff.mask.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -612,7 +582,6 @@ define <vscale x 2 x i32> @test_vlseg6ff_nxv2i32(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg6e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i32} @llvm.riscv.vlseg6ff.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -626,15 +595,14 @@ define <vscale x 2 x i32> @test_vlseg6ff_mask_nxv2i32(<vscale x 2 x i32> %val, p
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg6e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i32} @llvm.riscv.vlseg6ff.mask.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -654,7 +622,6 @@ define <vscale x 2 x i32> @test_vlseg7ff_nxv2i32(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg7e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i32} @llvm.riscv.vlseg7ff.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -668,16 +635,15 @@ define <vscale x 2 x i32> @test_vlseg7ff_mask_nxv2i32(<vscale x 2 x i32> %val, p
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg7e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i32} @llvm.riscv.vlseg7ff.mask.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -697,7 +663,6 @@ define <vscale x 2 x i32> @test_vlseg8ff_nxv2i32(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg8e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i32} @llvm.riscv.vlseg8ff.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %vl)
@@ -711,17 +676,16 @@ define <vscale x 2 x i32> @test_vlseg8ff_mask_nxv2i32(<vscale x 2 x i32> %val, p
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg8e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i32} @llvm.riscv.vlseg8ff.mask.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -741,7 +705,6 @@ define <vscale x 4 x i16> @test_vlseg2ff_nxv4i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>, i32} @llvm.riscv.vlseg2ff.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -759,7 +722,6 @@ define <vscale x 4 x i16> @test_vlseg2ff_mask_nxv4i16(<vscale x 4 x i16> %val, p
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>, i32} @llvm.riscv.vlseg2ff.mask.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -779,7 +741,6 @@ define <vscale x 4 x i16> @test_vlseg3ff_nxv4i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i32} @llvm.riscv.vlseg3ff.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -793,12 +754,11 @@ define <vscale x 4 x i16> @test_vlseg3ff_mask_nxv4i16(<vscale x 4 x i16> %val, p
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i32} @llvm.riscv.vlseg3ff.mask.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -818,7 +778,6 @@ define <vscale x 4 x i16> @test_vlseg4ff_nxv4i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i32} @llvm.riscv.vlseg4ff.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -832,13 +791,12 @@ define <vscale x 4 x i16> @test_vlseg4ff_mask_nxv4i16(<vscale x 4 x i16> %val, p
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i32} @llvm.riscv.vlseg4ff.mask.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -858,7 +816,6 @@ define <vscale x 4 x i16> @test_vlseg5ff_nxv4i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i32} @llvm.riscv.vlseg5ff.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -872,14 +829,13 @@ define <vscale x 4 x i16> @test_vlseg5ff_mask_nxv4i16(<vscale x 4 x i16> %val, p
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i32} @llvm.riscv.vlseg5ff.mask.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -899,7 +855,6 @@ define <vscale x 4 x i16> @test_vlseg6ff_nxv4i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i32} @llvm.riscv.vlseg6ff.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -913,15 +868,14 @@ define <vscale x 4 x i16> @test_vlseg6ff_mask_nxv4i16(<vscale x 4 x i16> %val, p
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i32} @llvm.riscv.vlseg6ff.mask.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -941,7 +895,6 @@ define <vscale x 4 x i16> @test_vlseg7ff_nxv4i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i32} @llvm.riscv.vlseg7ff.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -955,16 +908,15 @@ define <vscale x 4 x i16> @test_vlseg7ff_mask_nxv4i16(<vscale x 4 x i16> %val, p
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i32} @llvm.riscv.vlseg7ff.mask.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -984,7 +936,6 @@ define <vscale x 4 x i16> @test_vlseg8ff_nxv4i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i32} @llvm.riscv.vlseg8ff.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %vl)
@@ -998,17 +949,16 @@ define <vscale x 4 x i16> @test_vlseg8ff_mask_nxv4i16(<vscale x 4 x i16> %val, p
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i32} @llvm.riscv.vlseg8ff.mask.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -1028,7 +978,6 @@ define <vscale x 1 x i32> @test_vlseg2ff_nxv1i32(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>, i32} @llvm.riscv.vlseg2ff.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -1046,7 +995,6 @@ define <vscale x 1 x i32> @test_vlseg2ff_mask_nxv1i32(<vscale x 1 x i32> %val, p
 ; CHECK-NEXT:    vlseg2e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>, i32} @llvm.riscv.vlseg2ff.mask.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -1066,7 +1014,6 @@ define <vscale x 1 x i32> @test_vlseg3ff_nxv1i32(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg3e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i32} @llvm.riscv.vlseg3ff.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -1080,12 +1027,11 @@ define <vscale x 1 x i32> @test_vlseg3ff_mask_nxv1i32(<vscale x 1 x i32> %val, p
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg3e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i32} @llvm.riscv.vlseg3ff.mask.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -1105,7 +1051,6 @@ define <vscale x 1 x i32> @test_vlseg4ff_nxv1i32(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg4e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i32} @llvm.riscv.vlseg4ff.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -1119,13 +1064,12 @@ define <vscale x 1 x i32> @test_vlseg4ff_mask_nxv1i32(<vscale x 1 x i32> %val, p
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg4e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i32} @llvm.riscv.vlseg4ff.mask.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -1145,7 +1089,6 @@ define <vscale x 1 x i32> @test_vlseg5ff_nxv1i32(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg5e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i32} @llvm.riscv.vlseg5ff.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -1159,14 +1102,13 @@ define <vscale x 1 x i32> @test_vlseg5ff_mask_nxv1i32(<vscale x 1 x i32> %val, p
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg5e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i32} @llvm.riscv.vlseg5ff.mask.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -1186,7 +1128,6 @@ define <vscale x 1 x i32> @test_vlseg6ff_nxv1i32(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg6e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i32} @llvm.riscv.vlseg6ff.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -1200,15 +1141,14 @@ define <vscale x 1 x i32> @test_vlseg6ff_mask_nxv1i32(<vscale x 1 x i32> %val, p
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg6e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i32} @llvm.riscv.vlseg6ff.mask.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -1228,7 +1168,6 @@ define <vscale x 1 x i32> @test_vlseg7ff_nxv1i32(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg7e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i32} @llvm.riscv.vlseg7ff.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -1242,16 +1181,15 @@ define <vscale x 1 x i32> @test_vlseg7ff_mask_nxv1i32(<vscale x 1 x i32> %val, p
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg7e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i32} @llvm.riscv.vlseg7ff.mask.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -1271,7 +1209,6 @@ define <vscale x 1 x i32> @test_vlseg8ff_nxv1i32(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg8e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i32} @llvm.riscv.vlseg8ff.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %vl)
@@ -1285,17 +1222,16 @@ define <vscale x 1 x i32> @test_vlseg8ff_mask_nxv1i32(<vscale x 1 x i32> %val, p
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg8e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i32} @llvm.riscv.vlseg8ff.mask.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -1315,7 +1251,6 @@ define <vscale x 8 x i16> @test_vlseg2ff_nxv8i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e16ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>, i32} @llvm.riscv.vlseg2ff.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i32 %vl)
@@ -1333,7 +1268,6 @@ define <vscale x 8 x i16> @test_vlseg2ff_mask_nxv8i16(<vscale x 8 x i16> %val, p
 ; CHECK-NEXT:    vlseg2e16ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>, i32} @llvm.riscv.vlseg2ff.mask.nxv8i16(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -1353,7 +1287,6 @@ define <vscale x 8 x i16> @test_vlseg3ff_nxv8i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg3e16ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>, i32} @llvm.riscv.vlseg3ff.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i32 %vl)
@@ -1367,12 +1300,11 @@ define <vscale x 8 x i16> @test_vlseg3ff_mask_nxv8i16(<vscale x 8 x i16> %val, p
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vlseg3e16ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>, i32} @llvm.riscv.vlseg3ff.mask.nxv8i16(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -1392,7 +1324,6 @@ define <vscale x 8 x i16> @test_vlseg4ff_nxv8i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg4e16ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>, i32} @llvm.riscv.vlseg4ff.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i32 %vl)
@@ -1406,13 +1337,12 @@ define <vscale x 8 x i16> @test_vlseg4ff_mask_nxv8i16(<vscale x 8 x i16> %val, p
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vlseg4e16ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>, i32} @llvm.riscv.vlseg4ff.mask.nxv8i16(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -1432,7 +1362,6 @@ define <vscale x 8 x i8> @test_vlseg2ff_nxv8i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg2e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>, i32} @llvm.riscv.vlseg2ff.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1450,7 +1379,6 @@ define <vscale x 8 x i8> @test_vlseg2ff_mask_nxv8i8(<vscale x 8 x i8> %val, ptr
 ; CHECK-NEXT:    vlseg2e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>, i32} @llvm.riscv.vlseg2ff.mask.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -1470,7 +1398,6 @@ define <vscale x 8 x i8> @test_vlseg3ff_nxv8i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg3e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i32} @llvm.riscv.vlseg3ff.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1484,12 +1411,11 @@ define <vscale x 8 x i8> @test_vlseg3ff_mask_nxv8i8(<vscale x 8 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vlseg3e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i32} @llvm.riscv.vlseg3ff.mask.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -1509,7 +1435,6 @@ define <vscale x 8 x i8> @test_vlseg4ff_nxv8i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg4e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i32} @llvm.riscv.vlseg4ff.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1523,13 +1448,12 @@ define <vscale x 8 x i8> @test_vlseg4ff_mask_nxv8i8(<vscale x 8 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vlseg4e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i32} @llvm.riscv.vlseg4ff.mask.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -1549,7 +1473,6 @@ define <vscale x 8 x i8> @test_vlseg5ff_nxv8i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg5e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i32} @llvm.riscv.vlseg5ff.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1563,14 +1486,13 @@ define <vscale x 8 x i8> @test_vlseg5ff_mask_nxv8i8(<vscale x 8 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vlseg5e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i32} @llvm.riscv.vlseg5ff.mask.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -1590,7 +1512,6 @@ define <vscale x 8 x i8> @test_vlseg6ff_nxv8i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg6e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i32} @llvm.riscv.vlseg6ff.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1604,15 +1525,14 @@ define <vscale x 8 x i8> @test_vlseg6ff_mask_nxv8i8(<vscale x 8 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vlseg6e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i32} @llvm.riscv.vlseg6ff.mask.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -1632,7 +1552,6 @@ define <vscale x 8 x i8> @test_vlseg7ff_nxv8i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg7e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i32} @llvm.riscv.vlseg7ff.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1646,16 +1565,15 @@ define <vscale x 8 x i8> @test_vlseg7ff_mask_nxv8i8(<vscale x 8 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vlseg7e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i32} @llvm.riscv.vlseg7ff.mask.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -1675,7 +1593,6 @@ define <vscale x 8 x i8> @test_vlseg8ff_nxv8i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg8e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i32} @llvm.riscv.vlseg8ff.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %vl)
@@ -1689,17 +1606,16 @@ define <vscale x 8 x i8> @test_vlseg8ff_mask_nxv8i8(<vscale x 8 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vlseg8e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i32} @llvm.riscv.vlseg8ff.mask.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -1719,7 +1635,6 @@ define <vscale x 8 x i32> @test_vlseg2ff_nxv8i32(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e32ff.v v4, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>, i32} @llvm.riscv.vlseg2ff.nxv8i32(<vscale x 8 x i32> undef, <vscale x 8 x i32> undef, ptr %base, i32 %vl)
@@ -1737,7 +1652,6 @@ define <vscale x 8 x i32> @test_vlseg2ff_mask_nxv8i32(<vscale x 8 x i32> %val, p
 ; CHECK-NEXT:    vlseg2e32ff.v v4, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>, i32} @llvm.riscv.vlseg2ff.mask.nxv8i32(<vscale x 8 x i32> %val,<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -1757,7 +1671,6 @@ define <vscale x 4 x i8> @test_vlseg2ff_nxv4i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg2e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>, i32} @llvm.riscv.vlseg2ff.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1775,7 +1688,6 @@ define <vscale x 4 x i8> @test_vlseg2ff_mask_nxv4i8(<vscale x 4 x i8> %val, ptr
 ; CHECK-NEXT:    vlseg2e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>, i32} @llvm.riscv.vlseg2ff.mask.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -1795,7 +1707,6 @@ define <vscale x 4 x i8> @test_vlseg3ff_nxv4i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg3e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i32} @llvm.riscv.vlseg3ff.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1809,12 +1720,11 @@ define <vscale x 4 x i8> @test_vlseg3ff_mask_nxv4i8(<vscale x 4 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vlseg3e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i32} @llvm.riscv.vlseg3ff.mask.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -1834,7 +1744,6 @@ define <vscale x 4 x i8> @test_vlseg4ff_nxv4i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg4e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i32} @llvm.riscv.vlseg4ff.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1848,13 +1757,12 @@ define <vscale x 4 x i8> @test_vlseg4ff_mask_nxv4i8(<vscale x 4 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vlseg4e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i32} @llvm.riscv.vlseg4ff.mask.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -1874,7 +1782,6 @@ define <vscale x 4 x i8> @test_vlseg5ff_nxv4i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg5e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i32} @llvm.riscv.vlseg5ff.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1888,14 +1795,13 @@ define <vscale x 4 x i8> @test_vlseg5ff_mask_nxv4i8(<vscale x 4 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vlseg5e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i32} @llvm.riscv.vlseg5ff.mask.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -1915,7 +1821,6 @@ define <vscale x 4 x i8> @test_vlseg6ff_nxv4i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg6e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i32} @llvm.riscv.vlseg6ff.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1929,15 +1834,14 @@ define <vscale x 4 x i8> @test_vlseg6ff_mask_nxv4i8(<vscale x 4 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vlseg6e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i32} @llvm.riscv.vlseg6ff.mask.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -1957,7 +1861,6 @@ define <vscale x 4 x i8> @test_vlseg7ff_nxv4i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg7e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i32} @llvm.riscv.vlseg7ff.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -1971,16 +1874,15 @@ define <vscale x 4 x i8> @test_vlseg7ff_mask_nxv4i8(<vscale x 4 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vlseg7e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i32} @llvm.riscv.vlseg7ff.mask.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -2000,7 +1902,6 @@ define <vscale x 4 x i8> @test_vlseg8ff_nxv4i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg8e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i32} @llvm.riscv.vlseg8ff.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %vl)
@@ -2014,17 +1915,16 @@ define <vscale x 4 x i8> @test_vlseg8ff_mask_nxv4i8(<vscale x 4 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vlseg8e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i32} @llvm.riscv.vlseg8ff.mask.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -2044,7 +1944,6 @@ define <vscale x 1 x i16> @test_vlseg2ff_nxv1i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>, i32} @llvm.riscv.vlseg2ff.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -2062,7 +1961,6 @@ define <vscale x 1 x i16> @test_vlseg2ff_mask_nxv1i16(<vscale x 1 x i16> %val, p
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>, i32} @llvm.riscv.vlseg2ff.mask.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2082,7 +1980,6 @@ define <vscale x 1 x i16> @test_vlseg3ff_nxv1i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i32} @llvm.riscv.vlseg3ff.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -2096,12 +1993,11 @@ define <vscale x 1 x i16> @test_vlseg3ff_mask_nxv1i16(<vscale x 1 x i16> %val, p
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i32} @llvm.riscv.vlseg3ff.mask.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2121,7 +2017,6 @@ define <vscale x 1 x i16> @test_vlseg4ff_nxv1i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i32} @llvm.riscv.vlseg4ff.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -2135,13 +2030,12 @@ define <vscale x 1 x i16> @test_vlseg4ff_mask_nxv1i16(<vscale x 1 x i16> %val, p
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i32} @llvm.riscv.vlseg4ff.mask.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2161,7 +2055,6 @@ define <vscale x 1 x i16> @test_vlseg5ff_nxv1i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i32} @llvm.riscv.vlseg5ff.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -2175,14 +2068,13 @@ define <vscale x 1 x i16> @test_vlseg5ff_mask_nxv1i16(<vscale x 1 x i16> %val, p
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i32} @llvm.riscv.vlseg5ff.mask.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2202,7 +2094,6 @@ define <vscale x 1 x i16> @test_vlseg6ff_nxv1i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i32} @llvm.riscv.vlseg6ff.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -2216,15 +2107,14 @@ define <vscale x 1 x i16> @test_vlseg6ff_mask_nxv1i16(<vscale x 1 x i16> %val, p
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i32} @llvm.riscv.vlseg6ff.mask.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2244,7 +2134,6 @@ define <vscale x 1 x i16> @test_vlseg7ff_nxv1i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i32} @llvm.riscv.vlseg7ff.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -2258,16 +2147,15 @@ define <vscale x 1 x i16> @test_vlseg7ff_mask_nxv1i16(<vscale x 1 x i16> %val, p
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i32} @llvm.riscv.vlseg7ff.mask.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2287,7 +2175,6 @@ define <vscale x 1 x i16> @test_vlseg8ff_nxv1i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i32} @llvm.riscv.vlseg8ff.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %vl)
@@ -2301,17 +2188,16 @@ define <vscale x 1 x i16> @test_vlseg8ff_mask_nxv1i16(<vscale x 1 x i16> %val, p
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i32} @llvm.riscv.vlseg8ff.mask.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2331,7 +2217,6 @@ define <vscale x 32 x i8> @test_vlseg2ff_nxv32i8(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e8ff.v v4, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>, i32} @llvm.riscv.vlseg2ff.nxv32i8(<vscale x 32 x i8> undef, <vscale x 32 x i8> undef, ptr %base, i32 %vl)
@@ -2349,7 +2234,6 @@ define <vscale x 32 x i8> @test_vlseg2ff_mask_nxv32i8(<vscale x 32 x i8> %val, p
 ; CHECK-NEXT:    vlseg2e8ff.v v4, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>, i32} @llvm.riscv.vlseg2ff.mask.nxv32i8(<vscale x 32 x i8> %val,<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i1> %mask, i32 %vl, i32 1)
@@ -2369,7 +2253,6 @@ define <vscale x 2 x i8> @test_vlseg2ff_nxv2i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg2e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>, i32} @llvm.riscv.vlseg2ff.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2387,7 +2270,6 @@ define <vscale x 2 x i8> @test_vlseg2ff_mask_nxv2i8(<vscale x 2 x i8> %val, ptr
 ; CHECK-NEXT:    vlseg2e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>, i32} @llvm.riscv.vlseg2ff.mask.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -2407,7 +2289,6 @@ define <vscale x 2 x i8> @test_vlseg3ff_nxv2i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg3e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i32} @llvm.riscv.vlseg3ff.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2421,12 +2302,11 @@ define <vscale x 2 x i8> @test_vlseg3ff_mask_nxv2i8(<vscale x 2 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vlseg3e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i32} @llvm.riscv.vlseg3ff.mask.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -2446,7 +2326,6 @@ define <vscale x 2 x i8> @test_vlseg4ff_nxv2i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg4e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i32} @llvm.riscv.vlseg4ff.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2460,13 +2339,12 @@ define <vscale x 2 x i8> @test_vlseg4ff_mask_nxv2i8(<vscale x 2 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vlseg4e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i32} @llvm.riscv.vlseg4ff.mask.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -2486,7 +2364,6 @@ define <vscale x 2 x i8> @test_vlseg5ff_nxv2i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg5e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i32} @llvm.riscv.vlseg5ff.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2500,14 +2377,13 @@ define <vscale x 2 x i8> @test_vlseg5ff_mask_nxv2i8(<vscale x 2 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vlseg5e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i32} @llvm.riscv.vlseg5ff.mask.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -2527,7 +2403,6 @@ define <vscale x 2 x i8> @test_vlseg6ff_nxv2i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg6e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i32} @llvm.riscv.vlseg6ff.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2541,15 +2416,14 @@ define <vscale x 2 x i8> @test_vlseg6ff_mask_nxv2i8(<vscale x 2 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vlseg6e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i32} @llvm.riscv.vlseg6ff.mask.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -2569,7 +2443,6 @@ define <vscale x 2 x i8> @test_vlseg7ff_nxv2i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg7e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i32} @llvm.riscv.vlseg7ff.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2583,16 +2456,15 @@ define <vscale x 2 x i8> @test_vlseg7ff_mask_nxv2i8(<vscale x 2 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vlseg7e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i32} @llvm.riscv.vlseg7ff.mask.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -2612,7 +2484,6 @@ define <vscale x 2 x i8> @test_vlseg8ff_nxv2i8(ptr %base, i32 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg8e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i32} @llvm.riscv.vlseg8ff.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %vl)
@@ -2626,17 +2497,16 @@ define <vscale x 2 x i8> @test_vlseg8ff_mask_nxv2i8(<vscale x 2 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vlseg8e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i32} @llvm.riscv.vlseg8ff.mask.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -2656,7 +2526,6 @@ define <vscale x 2 x i16> @test_vlseg2ff_nxv2i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>, i32} @llvm.riscv.vlseg2ff.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2674,7 +2543,6 @@ define <vscale x 2 x i16> @test_vlseg2ff_mask_nxv2i16(<vscale x 2 x i16> %val, p
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>, i32} @llvm.riscv.vlseg2ff.mask.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -2694,7 +2562,6 @@ define <vscale x 2 x i16> @test_vlseg3ff_nxv2i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i32} @llvm.riscv.vlseg3ff.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2708,12 +2575,11 @@ define <vscale x 2 x i16> @test_vlseg3ff_mask_nxv2i16(<vscale x 2 x i16> %val, p
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i32} @llvm.riscv.vlseg3ff.mask.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -2733,7 +2599,6 @@ define <vscale x 2 x i16> @test_vlseg4ff_nxv2i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i32} @llvm.riscv.vlseg4ff.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2747,13 +2612,12 @@ define <vscale x 2 x i16> @test_vlseg4ff_mask_nxv2i16(<vscale x 2 x i16> %val, p
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i32} @llvm.riscv.vlseg4ff.mask.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -2773,7 +2637,6 @@ define <vscale x 2 x i16> @test_vlseg5ff_nxv2i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i32} @llvm.riscv.vlseg5ff.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2787,14 +2650,13 @@ define <vscale x 2 x i16> @test_vlseg5ff_mask_nxv2i16(<vscale x 2 x i16> %val, p
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i32} @llvm.riscv.vlseg5ff.mask.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -2814,7 +2676,6 @@ define <vscale x 2 x i16> @test_vlseg6ff_nxv2i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i32} @llvm.riscv.vlseg6ff.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2828,15 +2689,14 @@ define <vscale x 2 x i16> @test_vlseg6ff_mask_nxv2i16(<vscale x 2 x i16> %val, p
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i32} @llvm.riscv.vlseg6ff.mask.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -2856,7 +2716,6 @@ define <vscale x 2 x i16> @test_vlseg7ff_nxv2i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i32} @llvm.riscv.vlseg7ff.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2870,16 +2729,15 @@ define <vscale x 2 x i16> @test_vlseg7ff_mask_nxv2i16(<vscale x 2 x i16> %val, p
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i32} @llvm.riscv.vlseg7ff.mask.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -2899,7 +2757,6 @@ define <vscale x 2 x i16> @test_vlseg8ff_nxv2i16(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i32} @llvm.riscv.vlseg8ff.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %vl)
@@ -2913,17 +2770,16 @@ define <vscale x 2 x i16> @test_vlseg8ff_mask_nxv2i16(<vscale x 2 x i16> %val, p
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i32} @llvm.riscv.vlseg8ff.mask.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -2943,7 +2799,6 @@ define <vscale x 4 x i32> @test_vlseg2ff_nxv4i32(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e32ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>, i32} @llvm.riscv.vlseg2ff.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i32 %vl)
@@ -2961,7 +2816,6 @@ define <vscale x 4 x i32> @test_vlseg2ff_mask_nxv4i32(<vscale x 4 x i32> %val, p
 ; CHECK-NEXT:    vlseg2e32ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>, i32} @llvm.riscv.vlseg2ff.mask.nxv4i32(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -2981,7 +2835,6 @@ define <vscale x 4 x i32> @test_vlseg3ff_nxv4i32(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg3e32ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>, i32} @llvm.riscv.vlseg3ff.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i32 %vl)
@@ -2995,12 +2848,11 @@ define <vscale x 4 x i32> @test_vlseg3ff_mask_nxv4i32(<vscale x 4 x i32> %val, p
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vlseg3e32ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>, i32} @llvm.riscv.vlseg3ff.mask.nxv4i32(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -3020,7 +2872,6 @@ define <vscale x 4 x i32> @test_vlseg4ff_nxv4i32(ptr %base, i32 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg4e32ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>, i32} @llvm.riscv.vlseg4ff.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i32 %vl)
@@ -3034,13 +2885,12 @@ define <vscale x 4 x i32> @test_vlseg4ff_mask_nxv4i32(<vscale x 4 x i32> %val, p
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vlseg4e32ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>, i32} @llvm.riscv.vlseg4ff.mask.nxv4i32(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -3060,7 +2910,6 @@ define <vscale x 16 x half> @test_vlseg2ff_nxv16f16(ptr %base, i32 %vl, ptr %out
 ; CHECK-NEXT:    vlseg2e16ff.v v4, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>, i32} @llvm.riscv.vlseg2ff.nxv16f16(<vscale x 16 x half> undef, <vscale x 16 x half> undef, ptr %base, i32 %vl)
@@ -3078,7 +2927,6 @@ define <vscale x 16 x half> @test_vlseg2ff_mask_nxv16f16(<vscale x 16 x half> %v
 ; CHECK-NEXT:    vlseg2e16ff.v v4, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>, i32} @llvm.riscv.vlseg2ff.mask.nxv16f16(<vscale x 16 x half> %val,<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -3098,7 +2946,6 @@ define <vscale x 4 x double> @test_vlseg2ff_nxv4f64(ptr %base, i32 %vl, ptr %out
 ; CHECK-NEXT:    vlseg2e64ff.v v4, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>, i32} @llvm.riscv.vlseg2ff.nxv4f64(<vscale x 4 x double> undef, <vscale x 4 x double> undef, ptr %base, i32 %vl)
@@ -3116,7 +2963,6 @@ define <vscale x 4 x double> @test_vlseg2ff_mask_nxv4f64(<vscale x 4 x double> %
 ; CHECK-NEXT:    vlseg2e64ff.v v4, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>, i32} @llvm.riscv.vlseg2ff.mask.nxv4f64(<vscale x 4 x double> %val,<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -3136,7 +2982,6 @@ define <vscale x 1 x double> @test_vlseg2ff_nxv1f64(ptr %base, i32 %vl, ptr %out
 ; CHECK-NEXT:    vlseg2e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>, i32} @llvm.riscv.vlseg2ff.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -3154,7 +2999,6 @@ define <vscale x 1 x double> @test_vlseg2ff_mask_nxv1f64(<vscale x 1 x double> %
 ; CHECK-NEXT:    vlseg2e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>, i32} @llvm.riscv.vlseg2ff.mask.nxv1f64(<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -3174,7 +3018,6 @@ define <vscale x 1 x double> @test_vlseg3ff_nxv1f64(ptr %base, i32 %vl, ptr %out
 ; CHECK-NEXT:    vlseg3e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i32} @llvm.riscv.vlseg3ff.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -3188,12 +3031,11 @@ define <vscale x 1 x double> @test_vlseg3ff_mask_nxv1f64(<vscale x 1 x double> %
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vlseg3e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i32} @llvm.riscv.vlseg3ff.mask.nxv1f64(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -3213,7 +3055,6 @@ define <vscale x 1 x double> @test_vlseg4ff_nxv1f64(ptr %base, i32 %vl, ptr %out
 ; CHECK-NEXT:    vlseg4e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i32} @llvm.riscv.vlseg4ff.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -3227,13 +3068,12 @@ define <vscale x 1 x double> @test_vlseg4ff_mask_nxv1f64(<vscale x 1 x double> %
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vlseg4e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i32} @llvm.riscv.vlseg4ff.mask.nxv1f64(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -3253,7 +3093,6 @@ define <vscale x 1 x double> @test_vlseg5ff_nxv1f64(ptr %base, i32 %vl, ptr %out
 ; CHECK-NEXT:    vlseg5e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i32} @llvm.riscv.vlseg5ff.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -3267,14 +3106,13 @@ define <vscale x 1 x double> @test_vlseg5ff_mask_nxv1f64(<vscale x 1 x double> %
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vlseg5e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i32} @llvm.riscv.vlseg5ff.mask.nxv1f64(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -3294,7 +3132,6 @@ define <vscale x 1 x double> @test_vlseg6ff_nxv1f64(ptr %base, i32 %vl, ptr %out
 ; CHECK-NEXT:    vlseg6e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i32} @llvm.riscv.vlseg6ff.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -3308,15 +3145,14 @@ define <vscale x 1 x double> @test_vlseg6ff_mask_nxv1f64(<vscale x 1 x double> %
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vlseg6e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i32} @llvm.riscv.vlseg6ff.mask.nxv1f64(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -3336,7 +3172,6 @@ define <vscale x 1 x double> @test_vlseg7ff_nxv1f64(ptr %base, i32 %vl, ptr %out
 ; CHECK-NEXT:    vlseg7e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i32} @llvm.riscv.vlseg7ff.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -3350,16 +3185,15 @@ define <vscale x 1 x double> @test_vlseg7ff_mask_nxv1f64(<vscale x 1 x double> %
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vlseg7e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i32} @llvm.riscv.vlseg7ff.mask.nxv1f64(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -3379,7 +3213,6 @@ define <vscale x 1 x double> @test_vlseg8ff_nxv1f64(ptr %base, i32 %vl, ptr %out
 ; CHECK-NEXT:    vlseg8e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i32} @llvm.riscv.vlseg8ff.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %vl)
@@ -3393,17 +3226,16 @@ define <vscale x 1 x double> @test_vlseg8ff_mask_nxv1f64(<vscale x 1 x double> %
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vlseg8e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i32} @llvm.riscv.vlseg8ff.mask.nxv1f64(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -3423,7 +3255,6 @@ define <vscale x 2 x float> @test_vlseg2ff_nxv2f32(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg2e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>, i32} @llvm.riscv.vlseg2ff.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3441,7 +3272,6 @@ define <vscale x 2 x float> @test_vlseg2ff_mask_nxv2f32(<vscale x 2 x float> %va
 ; CHECK-NEXT:    vlseg2e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>, i32} @llvm.riscv.vlseg2ff.mask.nxv2f32(<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -3461,7 +3291,6 @@ define <vscale x 2 x float> @test_vlseg3ff_nxv2f32(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg3e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i32} @llvm.riscv.vlseg3ff.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3475,12 +3304,11 @@ define <vscale x 2 x float> @test_vlseg3ff_mask_nxv2f32(<vscale x 2 x float> %va
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg3e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i32} @llvm.riscv.vlseg3ff.mask.nxv2f32(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -3500,7 +3328,6 @@ define <vscale x 2 x float> @test_vlseg4ff_nxv2f32(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg4e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i32} @llvm.riscv.vlseg4ff.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3514,13 +3341,12 @@ define <vscale x 2 x float> @test_vlseg4ff_mask_nxv2f32(<vscale x 2 x float> %va
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg4e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i32} @llvm.riscv.vlseg4ff.mask.nxv2f32(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -3540,7 +3366,6 @@ define <vscale x 2 x float> @test_vlseg5ff_nxv2f32(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg5e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i32} @llvm.riscv.vlseg5ff.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3554,14 +3379,13 @@ define <vscale x 2 x float> @test_vlseg5ff_mask_nxv2f32(<vscale x 2 x float> %va
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg5e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i32} @llvm.riscv.vlseg5ff.mask.nxv2f32(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -3581,7 +3405,6 @@ define <vscale x 2 x float> @test_vlseg6ff_nxv2f32(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg6e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i32} @llvm.riscv.vlseg6ff.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3595,15 +3418,14 @@ define <vscale x 2 x float> @test_vlseg6ff_mask_nxv2f32(<vscale x 2 x float> %va
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg6e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i32} @llvm.riscv.vlseg6ff.mask.nxv2f32(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -3623,7 +3445,6 @@ define <vscale x 2 x float> @test_vlseg7ff_nxv2f32(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg7e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i32} @llvm.riscv.vlseg7ff.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3637,16 +3458,15 @@ define <vscale x 2 x float> @test_vlseg7ff_mask_nxv2f32(<vscale x 2 x float> %va
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg7e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i32} @llvm.riscv.vlseg7ff.mask.nxv2f32(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -3666,7 +3486,6 @@ define <vscale x 2 x float> @test_vlseg8ff_nxv2f32(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg8e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i32} @llvm.riscv.vlseg8ff.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %vl)
@@ -3680,17 +3499,16 @@ define <vscale x 2 x float> @test_vlseg8ff_mask_nxv2f32(<vscale x 2 x float> %va
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg8e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i32} @llvm.riscv.vlseg8ff.mask.nxv2f32(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -3710,7 +3528,6 @@ define <vscale x 1 x half> @test_vlseg2ff_nxv1f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>, i32} @llvm.riscv.vlseg2ff.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3728,7 +3545,6 @@ define <vscale x 1 x half> @test_vlseg2ff_mask_nxv1f16(<vscale x 1 x half> %val,
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>, i32} @llvm.riscv.vlseg2ff.mask.nxv1f16(<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -3748,7 +3564,6 @@ define <vscale x 1 x half> @test_vlseg3ff_nxv1f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i32} @llvm.riscv.vlseg3ff.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3762,12 +3577,11 @@ define <vscale x 1 x half> @test_vlseg3ff_mask_nxv1f16(<vscale x 1 x half> %val,
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i32} @llvm.riscv.vlseg3ff.mask.nxv1f16(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -3787,7 +3601,6 @@ define <vscale x 1 x half> @test_vlseg4ff_nxv1f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i32} @llvm.riscv.vlseg4ff.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3801,13 +3614,12 @@ define <vscale x 1 x half> @test_vlseg4ff_mask_nxv1f16(<vscale x 1 x half> %val,
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i32} @llvm.riscv.vlseg4ff.mask.nxv1f16(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -3827,7 +3639,6 @@ define <vscale x 1 x half> @test_vlseg5ff_nxv1f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i32} @llvm.riscv.vlseg5ff.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3841,14 +3652,13 @@ define <vscale x 1 x half> @test_vlseg5ff_mask_nxv1f16(<vscale x 1 x half> %val,
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i32} @llvm.riscv.vlseg5ff.mask.nxv1f16(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -3868,7 +3678,6 @@ define <vscale x 1 x half> @test_vlseg6ff_nxv1f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i32} @llvm.riscv.vlseg6ff.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3882,15 +3691,14 @@ define <vscale x 1 x half> @test_vlseg6ff_mask_nxv1f16(<vscale x 1 x half> %val,
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i32} @llvm.riscv.vlseg6ff.mask.nxv1f16(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -3910,7 +3718,6 @@ define <vscale x 1 x half> @test_vlseg7ff_nxv1f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i32} @llvm.riscv.vlseg7ff.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3924,16 +3731,15 @@ define <vscale x 1 x half> @test_vlseg7ff_mask_nxv1f16(<vscale x 1 x half> %val,
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i32} @llvm.riscv.vlseg7ff.mask.nxv1f16(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -3953,7 +3759,6 @@ define <vscale x 1 x half> @test_vlseg8ff_nxv1f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i32} @llvm.riscv.vlseg8ff.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %vl)
@@ -3967,17 +3772,16 @@ define <vscale x 1 x half> @test_vlseg8ff_mask_nxv1f16(<vscale x 1 x half> %val,
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i32} @llvm.riscv.vlseg8ff.mask.nxv1f16(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -3997,7 +3801,6 @@ define <vscale x 1 x float> @test_vlseg2ff_nxv1f32(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg2e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>, i32} @llvm.riscv.vlseg2ff.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -4015,7 +3818,6 @@ define <vscale x 1 x float> @test_vlseg2ff_mask_nxv1f32(<vscale x 1 x float> %va
 ; CHECK-NEXT:    vlseg2e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>, i32} @llvm.riscv.vlseg2ff.mask.nxv1f32(<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -4035,7 +3837,6 @@ define <vscale x 1 x float> @test_vlseg3ff_nxv1f32(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg3e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i32} @llvm.riscv.vlseg3ff.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -4049,12 +3850,11 @@ define <vscale x 1 x float> @test_vlseg3ff_mask_nxv1f32(<vscale x 1 x float> %va
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg3e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i32} @llvm.riscv.vlseg3ff.mask.nxv1f32(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -4074,7 +3874,6 @@ define <vscale x 1 x float> @test_vlseg4ff_nxv1f32(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg4e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i32} @llvm.riscv.vlseg4ff.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -4088,13 +3887,12 @@ define <vscale x 1 x float> @test_vlseg4ff_mask_nxv1f32(<vscale x 1 x float> %va
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg4e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i32} @llvm.riscv.vlseg4ff.mask.nxv1f32(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -4114,7 +3912,6 @@ define <vscale x 1 x float> @test_vlseg5ff_nxv1f32(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg5e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i32} @llvm.riscv.vlseg5ff.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -4128,14 +3925,13 @@ define <vscale x 1 x float> @test_vlseg5ff_mask_nxv1f32(<vscale x 1 x float> %va
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg5e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i32} @llvm.riscv.vlseg5ff.mask.nxv1f32(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -4155,7 +3951,6 @@ define <vscale x 1 x float> @test_vlseg6ff_nxv1f32(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg6e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i32} @llvm.riscv.vlseg6ff.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -4169,15 +3964,14 @@ define <vscale x 1 x float> @test_vlseg6ff_mask_nxv1f32(<vscale x 1 x float> %va
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg6e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i32} @llvm.riscv.vlseg6ff.mask.nxv1f32(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -4197,7 +3991,6 @@ define <vscale x 1 x float> @test_vlseg7ff_nxv1f32(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg7e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i32} @llvm.riscv.vlseg7ff.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -4211,16 +4004,15 @@ define <vscale x 1 x float> @test_vlseg7ff_mask_nxv1f32(<vscale x 1 x float> %va
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg7e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i32} @llvm.riscv.vlseg7ff.mask.nxv1f32(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -4240,7 +4032,6 @@ define <vscale x 1 x float> @test_vlseg8ff_nxv1f32(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg8e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i32} @llvm.riscv.vlseg8ff.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %vl)
@@ -4254,17 +4045,16 @@ define <vscale x 1 x float> @test_vlseg8ff_mask_nxv1f32(<vscale x 1 x float> %va
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg8e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i32} @llvm.riscv.vlseg8ff.mask.nxv1f32(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -4284,7 +4074,6 @@ define <vscale x 8 x half> @test_vlseg2ff_nxv8f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg2e16ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>, i32} @llvm.riscv.vlseg2ff.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i32 %vl)
@@ -4302,7 +4091,6 @@ define <vscale x 8 x half> @test_vlseg2ff_mask_nxv8f16(<vscale x 8 x half> %val,
 ; CHECK-NEXT:    vlseg2e16ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>, i32} @llvm.riscv.vlseg2ff.mask.nxv8f16(<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -4322,7 +4110,6 @@ define <vscale x 8 x half> @test_vlseg3ff_nxv8f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg3e16ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>, i32} @llvm.riscv.vlseg3ff.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i32 %vl)
@@ -4336,12 +4123,11 @@ define <vscale x 8 x half> @test_vlseg3ff_mask_nxv8f16(<vscale x 8 x half> %val,
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vlseg3e16ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>, i32} @llvm.riscv.vlseg3ff.mask.nxv8f16(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -4361,7 +4147,6 @@ define <vscale x 8 x half> @test_vlseg4ff_nxv8f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg4e16ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>, i32} @llvm.riscv.vlseg4ff.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i32 %vl)
@@ -4375,13 +4160,12 @@ define <vscale x 8 x half> @test_vlseg4ff_mask_nxv8f16(<vscale x 8 x half> %val,
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vlseg4e16ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>, i32} @llvm.riscv.vlseg4ff.mask.nxv8f16(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -4401,7 +4185,6 @@ define <vscale x 8 x float> @test_vlseg2ff_nxv8f32(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg2e32ff.v v4, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>, i32} @llvm.riscv.vlseg2ff.nxv8f32(<vscale x 8 x float> undef, <vscale x 8 x float> undef, ptr %base, i32 %vl)
@@ -4419,7 +4202,6 @@ define <vscale x 8 x float> @test_vlseg2ff_mask_nxv8f32(<vscale x 8 x float> %va
 ; CHECK-NEXT:    vlseg2e32ff.v v4, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>, i32} @llvm.riscv.vlseg2ff.mask.nxv8f32(<vscale x 8 x float> %val,<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -4439,7 +4221,6 @@ define <vscale x 2 x double> @test_vlseg2ff_nxv2f64(ptr %base, i32 %vl, ptr %out
 ; CHECK-NEXT:    vlseg2e64ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>, i32} @llvm.riscv.vlseg2ff.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i32 %vl)
@@ -4457,7 +4238,6 @@ define <vscale x 2 x double> @test_vlseg2ff_mask_nxv2f64(<vscale x 2 x double> %
 ; CHECK-NEXT:    vlseg2e64ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>, i32} @llvm.riscv.vlseg2ff.mask.nxv2f64(<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -4477,7 +4257,6 @@ define <vscale x 2 x double> @test_vlseg3ff_nxv2f64(ptr %base, i32 %vl, ptr %out
 ; CHECK-NEXT:    vlseg3e64ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>, i32} @llvm.riscv.vlseg3ff.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i32 %vl)
@@ -4491,12 +4270,11 @@ define <vscale x 2 x double> @test_vlseg3ff_mask_nxv2f64(<vscale x 2 x double> %
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vlseg3e64ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>, i32} @llvm.riscv.vlseg3ff.mask.nxv2f64(<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -4516,7 +4294,6 @@ define <vscale x 2 x double> @test_vlseg4ff_nxv2f64(ptr %base, i32 %vl, ptr %out
 ; CHECK-NEXT:    vlseg4e64ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>, i32} @llvm.riscv.vlseg4ff.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i32 %vl)
@@ -4530,13 +4307,12 @@ define <vscale x 2 x double> @test_vlseg4ff_mask_nxv2f64(<vscale x 2 x double> %
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vlseg4e64ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>, i32} @llvm.riscv.vlseg4ff.mask.nxv2f64(<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -4556,7 +4332,6 @@ define <vscale x 4 x half> @test_vlseg2ff_nxv4f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>, i32} @llvm.riscv.vlseg2ff.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4574,7 +4349,6 @@ define <vscale x 4 x half> @test_vlseg2ff_mask_nxv4f16(<vscale x 4 x half> %val,
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>, i32} @llvm.riscv.vlseg2ff.mask.nxv4f16(<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4594,7 +4368,6 @@ define <vscale x 4 x half> @test_vlseg3ff_nxv4f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i32} @llvm.riscv.vlseg3ff.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4608,12 +4381,11 @@ define <vscale x 4 x half> @test_vlseg3ff_mask_nxv4f16(<vscale x 4 x half> %val,
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i32} @llvm.riscv.vlseg3ff.mask.nxv4f16(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4633,7 +4405,6 @@ define <vscale x 4 x half> @test_vlseg4ff_nxv4f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i32} @llvm.riscv.vlseg4ff.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4647,13 +4418,12 @@ define <vscale x 4 x half> @test_vlseg4ff_mask_nxv4f16(<vscale x 4 x half> %val,
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i32} @llvm.riscv.vlseg4ff.mask.nxv4f16(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4673,7 +4443,6 @@ define <vscale x 4 x half> @test_vlseg5ff_nxv4f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i32} @llvm.riscv.vlseg5ff.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4687,14 +4456,13 @@ define <vscale x 4 x half> @test_vlseg5ff_mask_nxv4f16(<vscale x 4 x half> %val,
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i32} @llvm.riscv.vlseg5ff.mask.nxv4f16(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4714,7 +4482,6 @@ define <vscale x 4 x half> @test_vlseg6ff_nxv4f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i32} @llvm.riscv.vlseg6ff.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4728,15 +4495,14 @@ define <vscale x 4 x half> @test_vlseg6ff_mask_nxv4f16(<vscale x 4 x half> %val,
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i32} @llvm.riscv.vlseg6ff.mask.nxv4f16(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4756,7 +4522,6 @@ define <vscale x 4 x half> @test_vlseg7ff_nxv4f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i32} @llvm.riscv.vlseg7ff.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4770,16 +4535,15 @@ define <vscale x 4 x half> @test_vlseg7ff_mask_nxv4f16(<vscale x 4 x half> %val,
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i32} @llvm.riscv.vlseg7ff.mask.nxv4f16(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4799,7 +4563,6 @@ define <vscale x 4 x half> @test_vlseg8ff_nxv4f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i32} @llvm.riscv.vlseg8ff.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %vl)
@@ -4813,17 +4576,16 @@ define <vscale x 4 x half> @test_vlseg8ff_mask_nxv4f16(<vscale x 4 x half> %val,
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i32} @llvm.riscv.vlseg8ff.mask.nxv4f16(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4843,7 +4605,6 @@ define <vscale x 2 x half> @test_vlseg2ff_nxv2f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>, i32} @llvm.riscv.vlseg2ff.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -4861,7 +4622,6 @@ define <vscale x 2 x half> @test_vlseg2ff_mask_nxv2f16(<vscale x 2 x half> %val,
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>, i32} @llvm.riscv.vlseg2ff.mask.nxv2f16(<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -4881,7 +4641,6 @@ define <vscale x 2 x half> @test_vlseg3ff_nxv2f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i32} @llvm.riscv.vlseg3ff.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -4895,12 +4654,11 @@ define <vscale x 2 x half> @test_vlseg3ff_mask_nxv2f16(<vscale x 2 x half> %val,
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i32} @llvm.riscv.vlseg3ff.mask.nxv2f16(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -4920,7 +4678,6 @@ define <vscale x 2 x half> @test_vlseg4ff_nxv2f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i32} @llvm.riscv.vlseg4ff.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -4934,13 +4691,12 @@ define <vscale x 2 x half> @test_vlseg4ff_mask_nxv2f16(<vscale x 2 x half> %val,
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i32} @llvm.riscv.vlseg4ff.mask.nxv2f16(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -4960,7 +4716,6 @@ define <vscale x 2 x half> @test_vlseg5ff_nxv2f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i32} @llvm.riscv.vlseg5ff.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -4974,14 +4729,13 @@ define <vscale x 2 x half> @test_vlseg5ff_mask_nxv2f16(<vscale x 2 x half> %val,
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i32} @llvm.riscv.vlseg5ff.mask.nxv2f16(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -5001,7 +4755,6 @@ define <vscale x 2 x half> @test_vlseg6ff_nxv2f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i32} @llvm.riscv.vlseg6ff.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -5015,15 +4768,14 @@ define <vscale x 2 x half> @test_vlseg6ff_mask_nxv2f16(<vscale x 2 x half> %val,
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i32} @llvm.riscv.vlseg6ff.mask.nxv2f16(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -5043,7 +4795,6 @@ define <vscale x 2 x half> @test_vlseg7ff_nxv2f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i32} @llvm.riscv.vlseg7ff.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -5057,16 +4808,15 @@ define <vscale x 2 x half> @test_vlseg7ff_mask_nxv2f16(<vscale x 2 x half> %val,
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i32} @llvm.riscv.vlseg7ff.mask.nxv2f16(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -5086,7 +4836,6 @@ define <vscale x 2 x half> @test_vlseg8ff_nxv2f16(ptr %base, i32 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i32} @llvm.riscv.vlseg8ff.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %vl)
@@ -5100,17 +4849,16 @@ define <vscale x 2 x half> @test_vlseg8ff_mask_nxv2f16(<vscale x 2 x half> %val,
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i32} @llvm.riscv.vlseg8ff.mask.nxv2f16(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -5130,7 +4878,6 @@ define <vscale x 4 x float> @test_vlseg2ff_nxv4f32(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg2e32ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>, i32} @llvm.riscv.vlseg2ff.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i32 %vl)
@@ -5148,7 +4895,6 @@ define <vscale x 4 x float> @test_vlseg2ff_mask_nxv4f32(<vscale x 4 x float> %va
 ; CHECK-NEXT:    vlseg2e32ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>, i32} @llvm.riscv.vlseg2ff.mask.nxv4f32(<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -5168,7 +4914,6 @@ define <vscale x 4 x float> @test_vlseg3ff_nxv4f32(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg3e32ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>, i32} @llvm.riscv.vlseg3ff.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i32 %vl)
@@ -5182,12 +4927,11 @@ define <vscale x 4 x float> @test_vlseg3ff_mask_nxv4f32(<vscale x 4 x float> %va
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vlseg3e32ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>, i32} @llvm.riscv.vlseg3ff.mask.nxv4f32(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -5207,7 +4951,6 @@ define <vscale x 4 x float> @test_vlseg4ff_nxv4f32(ptr %base, i32 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg4e32ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>, i32} @llvm.riscv.vlseg4ff.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i32 %vl)
@@ -5221,13 +4964,12 @@ define <vscale x 4 x float> @test_vlseg4ff_mask_nxv4f32(<vscale x 4 x float> %va
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vlseg4e32ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sw a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>, i32} @llvm.riscv.vlseg4ff.mask.nxv4f32(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
diff --git a/llvm/test/CodeGen/RISCV/rvv/vlsegff-rv64-dead.ll b/llvm/test/CodeGen/RISCV/rvv/vlsegff-rv64-dead.ll
index 2930a38d02805..3cd44f2c3bef4 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vlsegff-rv64-dead.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vlsegff-rv64-dead.ll
@@ -23,7 +23,6 @@ entry:
 define void @test_vlseg2ff_mask_dead_value(<vscale x 16 x i16> %val, ptr %base, i64 %vl, <vscale x 16 x i1> %mask, ptr %outvl) {
 ; CHECK-LABEL: test_vlseg2ff_mask_dead_value:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vlseg2e16ff.v v8, (a0), v0.t
@@ -42,7 +41,6 @@ define <vscale x 16 x i16> @test_vlseg2ff_dead_vl(ptr %base, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vlseg2e16ff.v v4, (a0)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>, i64} @llvm.riscv.vlseg2ff.nxv16i16(<vscale x 16 x i16> undef, <vscale x 16 x i16> undef, ptr %base, i64 %vl)
@@ -56,7 +54,6 @@ define <vscale x 16 x i16> @test_vlseg2ff_mask_dead_vl(<vscale x 16 x i16> %val,
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vlseg2e16ff.v v4, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>, i64} @llvm.riscv.vlseg2ff.mask.nxv16i16(<vscale x 16 x i16> %val,<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -78,7 +75,6 @@ entry:
 define void @test_vlseg2ff_mask_dead_all(<vscale x 16 x i16> %val, ptr %base, i64 %vl, <vscale x 16 x i1> %mask) {
 ; CHECK-LABEL: test_vlseg2ff_mask_dead_all:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vlseg2e16ff.v v8, (a0), v0.t
diff --git a/llvm/test/CodeGen/RISCV/rvv/vlsegff-rv64.ll b/llvm/test/CodeGen/RISCV/rvv/vlsegff-rv64.ll
index ff81faeaf4186..126975b1a60fa 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vlsegff-rv64.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vlsegff-rv64.ll
@@ -12,7 +12,6 @@ define <vscale x 16 x i16> @test_vlseg2ff_nxv16i16(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg2e16ff.v v4, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>, i64} @llvm.riscv.vlseg2ff.nxv16i16(<vscale x 16 x i16> undef, <vscale x 16 x i16> undef, ptr %base, i64 %vl)
@@ -30,7 +29,6 @@ define <vscale x 16 x i16> @test_vlseg2ff_mask_nxv16i16(<vscale x 16 x i16> %val
 ; CHECK-NEXT:    vlseg2e16ff.v v4, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>, i64} @llvm.riscv.vlseg2ff.mask.nxv16i16(<vscale x 16 x i16> %val,<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -50,7 +48,6 @@ define <vscale x 4 x i32> @test_vlseg2ff_nxv4i32(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e32ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>, i64} @llvm.riscv.vlseg2ff.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i64 %vl)
@@ -68,7 +65,6 @@ define <vscale x 4 x i32> @test_vlseg2ff_mask_nxv4i32(<vscale x 4 x i32> %val, p
 ; CHECK-NEXT:    vlseg2e32ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>, i64} @llvm.riscv.vlseg2ff.mask.nxv4i32(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -88,7 +84,6 @@ define <vscale x 4 x i32> @test_vlseg3ff_nxv4i32(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg3e32ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>, i64} @llvm.riscv.vlseg3ff.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i64 %vl)
@@ -102,12 +97,11 @@ define <vscale x 4 x i32> @test_vlseg3ff_mask_nxv4i32(<vscale x 4 x i32> %val, p
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vlseg3e32ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>, i64} @llvm.riscv.vlseg3ff.mask.nxv4i32(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -127,7 +121,6 @@ define <vscale x 4 x i32> @test_vlseg4ff_nxv4i32(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg4e32ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>, i64} @llvm.riscv.vlseg4ff.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i64 %vl)
@@ -141,13 +134,12 @@ define <vscale x 4 x i32> @test_vlseg4ff_mask_nxv4i32(<vscale x 4 x i32> %val, p
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vlseg4e32ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>, i64} @llvm.riscv.vlseg4ff.mask.nxv4i32(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -167,7 +159,6 @@ define <vscale x 16 x i8> @test_vlseg2ff_nxv16i8(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e8ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>, i64} @llvm.riscv.vlseg2ff.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i64 %vl)
@@ -185,7 +176,6 @@ define <vscale x 16 x i8> @test_vlseg2ff_mask_nxv16i8(<vscale x 16 x i8> %val, p
 ; CHECK-NEXT:    vlseg2e8ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>, i64} @llvm.riscv.vlseg2ff.mask.nxv16i8(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -205,7 +195,6 @@ define <vscale x 16 x i8> @test_vlseg3ff_nxv16i8(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg3e8ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>, i64} @llvm.riscv.vlseg3ff.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i64 %vl)
@@ -219,12 +208,11 @@ define <vscale x 16 x i8> @test_vlseg3ff_mask_nxv16i8(<vscale x 16 x i8> %val, p
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vlseg3e8ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>, i64} @llvm.riscv.vlseg3ff.mask.nxv16i8(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -244,7 +232,6 @@ define <vscale x 16 x i8> @test_vlseg4ff_nxv16i8(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg4e8ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>, i64} @llvm.riscv.vlseg4ff.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i64 %vl)
@@ -258,13 +245,12 @@ define <vscale x 16 x i8> @test_vlseg4ff_mask_nxv16i8(<vscale x 16 x i8> %val, p
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vlseg4e8ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>, i64} @llvm.riscv.vlseg4ff.mask.nxv16i8(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -284,7 +270,6 @@ define <vscale x 1 x i64> @test_vlseg2ff_nxv1i64(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>, i64} @llvm.riscv.vlseg2ff.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -302,7 +287,6 @@ define <vscale x 1 x i64> @test_vlseg2ff_mask_nxv1i64(<vscale x 1 x i64> %val, p
 ; CHECK-NEXT:    vlseg2e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>, i64} @llvm.riscv.vlseg2ff.mask.nxv1i64(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -322,7 +306,6 @@ define <vscale x 1 x i64> @test_vlseg3ff_nxv1i64(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg3e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>, i64} @llvm.riscv.vlseg3ff.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -336,12 +319,11 @@ define <vscale x 1 x i64> @test_vlseg3ff_mask_nxv1i64(<vscale x 1 x i64> %val, p
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vlseg3e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>, i64} @llvm.riscv.vlseg3ff.mask.nxv1i64(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -361,7 +343,6 @@ define <vscale x 1 x i64> @test_vlseg4ff_nxv1i64(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg4e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>, i64} @llvm.riscv.vlseg4ff.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -375,13 +356,12 @@ define <vscale x 1 x i64> @test_vlseg4ff_mask_nxv1i64(<vscale x 1 x i64> %val, p
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vlseg4e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>, i64} @llvm.riscv.vlseg4ff.mask.nxv1i64(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -401,7 +381,6 @@ define <vscale x 1 x i64> @test_vlseg5ff_nxv1i64(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg5e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>, i64} @llvm.riscv.vlseg5ff.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -415,14 +394,13 @@ define <vscale x 1 x i64> @test_vlseg5ff_mask_nxv1i64(<vscale x 1 x i64> %val, p
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vlseg5e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>, i64} @llvm.riscv.vlseg5ff.mask.nxv1i64(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -442,7 +420,6 @@ define <vscale x 1 x i64> @test_vlseg6ff_nxv1i64(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg6e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>, i64} @llvm.riscv.vlseg6ff.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -456,15 +433,14 @@ define <vscale x 1 x i64> @test_vlseg6ff_mask_nxv1i64(<vscale x 1 x i64> %val, p
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vlseg6e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>, i64} @llvm.riscv.vlseg6ff.mask.nxv1i64(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -484,7 +460,6 @@ define <vscale x 1 x i64> @test_vlseg7ff_nxv1i64(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg7e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>, i64} @llvm.riscv.vlseg7ff.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -498,16 +473,15 @@ define <vscale x 1 x i64> @test_vlseg7ff_mask_nxv1i64(<vscale x 1 x i64> %val, p
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vlseg7e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>, i64} @llvm.riscv.vlseg7ff.mask.nxv1i64(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -527,7 +501,6 @@ define <vscale x 1 x i64> @test_vlseg8ff_nxv1i64(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg8e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>, i64} @llvm.riscv.vlseg8ff.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef ,<vscale x 1 x i64> undef ,<vscale x 1 x i64> undef, <vscale x 1 x i64> undef ,<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %vl)
@@ -541,17 +514,16 @@ define <vscale x 1 x i64> @test_vlseg8ff_mask_nxv1i64(<vscale x 1 x i64> %val, p
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vlseg8e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>, i64} @llvm.riscv.vlseg8ff.mask.nxv1i64(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -571,7 +543,6 @@ define <vscale x 1 x i32> @test_vlseg2ff_nxv1i32(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>, i64} @llvm.riscv.vlseg2ff.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -589,7 +560,6 @@ define <vscale x 1 x i32> @test_vlseg2ff_mask_nxv1i32(<vscale x 1 x i32> %val, p
 ; CHECK-NEXT:    vlseg2e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>, i64} @llvm.riscv.vlseg2ff.mask.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -609,7 +579,6 @@ define <vscale x 1 x i32> @test_vlseg3ff_nxv1i32(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg3e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i64} @llvm.riscv.vlseg3ff.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -623,12 +592,11 @@ define <vscale x 1 x i32> @test_vlseg3ff_mask_nxv1i32(<vscale x 1 x i32> %val, p
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg3e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i64} @llvm.riscv.vlseg3ff.mask.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -648,7 +616,6 @@ define <vscale x 1 x i32> @test_vlseg4ff_nxv1i32(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg4e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i64} @llvm.riscv.vlseg4ff.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -662,13 +629,12 @@ define <vscale x 1 x i32> @test_vlseg4ff_mask_nxv1i32(<vscale x 1 x i32> %val, p
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg4e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i64} @llvm.riscv.vlseg4ff.mask.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -688,7 +654,6 @@ define <vscale x 1 x i32> @test_vlseg5ff_nxv1i32(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg5e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i64} @llvm.riscv.vlseg5ff.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -702,14 +667,13 @@ define <vscale x 1 x i32> @test_vlseg5ff_mask_nxv1i32(<vscale x 1 x i32> %val, p
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg5e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i64} @llvm.riscv.vlseg5ff.mask.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -729,7 +693,6 @@ define <vscale x 1 x i32> @test_vlseg6ff_nxv1i32(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg6e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i64} @llvm.riscv.vlseg6ff.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -743,15 +706,14 @@ define <vscale x 1 x i32> @test_vlseg6ff_mask_nxv1i32(<vscale x 1 x i32> %val, p
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg6e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i64} @llvm.riscv.vlseg6ff.mask.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -771,7 +733,6 @@ define <vscale x 1 x i32> @test_vlseg7ff_nxv1i32(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg7e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i64} @llvm.riscv.vlseg7ff.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -785,16 +746,15 @@ define <vscale x 1 x i32> @test_vlseg7ff_mask_nxv1i32(<vscale x 1 x i32> %val, p
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg7e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i64} @llvm.riscv.vlseg7ff.mask.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -814,7 +774,6 @@ define <vscale x 1 x i32> @test_vlseg8ff_nxv1i32(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg8e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i64} @llvm.riscv.vlseg8ff.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %vl)
@@ -828,17 +787,16 @@ define <vscale x 1 x i32> @test_vlseg8ff_mask_nxv1i32(<vscale x 1 x i32> %val, p
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg8e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>, i64} @llvm.riscv.vlseg8ff.mask.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -858,7 +816,6 @@ define <vscale x 8 x i16> @test_vlseg2ff_nxv8i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e16ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>, i64} @llvm.riscv.vlseg2ff.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i64 %vl)
@@ -876,7 +833,6 @@ define <vscale x 8 x i16> @test_vlseg2ff_mask_nxv8i16(<vscale x 8 x i16> %val, p
 ; CHECK-NEXT:    vlseg2e16ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>, i64} @llvm.riscv.vlseg2ff.mask.nxv8i16(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -896,7 +852,6 @@ define <vscale x 8 x i16> @test_vlseg3ff_nxv8i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg3e16ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>, i64} @llvm.riscv.vlseg3ff.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i64 %vl)
@@ -910,12 +865,11 @@ define <vscale x 8 x i16> @test_vlseg3ff_mask_nxv8i16(<vscale x 8 x i16> %val, p
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vlseg3e16ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>, i64} @llvm.riscv.vlseg3ff.mask.nxv8i16(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -935,7 +889,6 @@ define <vscale x 8 x i16> @test_vlseg4ff_nxv8i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg4e16ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>, i64} @llvm.riscv.vlseg4ff.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i64 %vl)
@@ -949,13 +902,12 @@ define <vscale x 8 x i16> @test_vlseg4ff_mask_nxv8i16(<vscale x 8 x i16> %val, p
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vlseg4e16ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>, i64} @llvm.riscv.vlseg4ff.mask.nxv8i16(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -975,7 +927,6 @@ define <vscale x 4 x i8> @test_vlseg2ff_nxv4i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg2e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>, i64} @llvm.riscv.vlseg2ff.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -993,7 +944,6 @@ define <vscale x 4 x i8> @test_vlseg2ff_mask_nxv4i8(<vscale x 4 x i8> %val, ptr
 ; CHECK-NEXT:    vlseg2e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>, i64} @llvm.riscv.vlseg2ff.mask.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -1013,7 +963,6 @@ define <vscale x 4 x i8> @test_vlseg3ff_nxv4i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg3e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i64} @llvm.riscv.vlseg3ff.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -1027,12 +976,11 @@ define <vscale x 4 x i8> @test_vlseg3ff_mask_nxv4i8(<vscale x 4 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vlseg3e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i64} @llvm.riscv.vlseg3ff.mask.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -1052,7 +1000,6 @@ define <vscale x 4 x i8> @test_vlseg4ff_nxv4i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg4e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i64} @llvm.riscv.vlseg4ff.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -1066,13 +1013,12 @@ define <vscale x 4 x i8> @test_vlseg4ff_mask_nxv4i8(<vscale x 4 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vlseg4e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i64} @llvm.riscv.vlseg4ff.mask.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -1092,7 +1038,6 @@ define <vscale x 4 x i8> @test_vlseg5ff_nxv4i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg5e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i64} @llvm.riscv.vlseg5ff.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -1106,14 +1051,13 @@ define <vscale x 4 x i8> @test_vlseg5ff_mask_nxv4i8(<vscale x 4 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vlseg5e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i64} @llvm.riscv.vlseg5ff.mask.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -1133,7 +1077,6 @@ define <vscale x 4 x i8> @test_vlseg6ff_nxv4i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg6e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i64} @llvm.riscv.vlseg6ff.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -1147,15 +1090,14 @@ define <vscale x 4 x i8> @test_vlseg6ff_mask_nxv4i8(<vscale x 4 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vlseg6e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i64} @llvm.riscv.vlseg6ff.mask.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -1175,7 +1117,6 @@ define <vscale x 4 x i8> @test_vlseg7ff_nxv4i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg7e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i64} @llvm.riscv.vlseg7ff.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -1189,16 +1130,15 @@ define <vscale x 4 x i8> @test_vlseg7ff_mask_nxv4i8(<vscale x 4 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vlseg7e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i64} @llvm.riscv.vlseg7ff.mask.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -1218,7 +1158,6 @@ define <vscale x 4 x i8> @test_vlseg8ff_nxv4i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg8e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i64} @llvm.riscv.vlseg8ff.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %vl)
@@ -1232,17 +1171,16 @@ define <vscale x 4 x i8> @test_vlseg8ff_mask_nxv4i8(<vscale x 4 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vlseg8e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>, i64} @llvm.riscv.vlseg8ff.mask.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -1262,7 +1200,6 @@ define <vscale x 1 x i16> @test_vlseg2ff_nxv1i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>, i64} @llvm.riscv.vlseg2ff.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1280,7 +1217,6 @@ define <vscale x 1 x i16> @test_vlseg2ff_mask_nxv1i16(<vscale x 1 x i16> %val, p
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>, i64} @llvm.riscv.vlseg2ff.mask.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1300,7 +1236,6 @@ define <vscale x 1 x i16> @test_vlseg3ff_nxv1i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i64} @llvm.riscv.vlseg3ff.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1314,12 +1249,11 @@ define <vscale x 1 x i16> @test_vlseg3ff_mask_nxv1i16(<vscale x 1 x i16> %val, p
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i64} @llvm.riscv.vlseg3ff.mask.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1339,7 +1273,6 @@ define <vscale x 1 x i16> @test_vlseg4ff_nxv1i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i64} @llvm.riscv.vlseg4ff.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1353,13 +1286,12 @@ define <vscale x 1 x i16> @test_vlseg4ff_mask_nxv1i16(<vscale x 1 x i16> %val, p
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i64} @llvm.riscv.vlseg4ff.mask.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1379,7 +1311,6 @@ define <vscale x 1 x i16> @test_vlseg5ff_nxv1i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i64} @llvm.riscv.vlseg5ff.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1393,14 +1324,13 @@ define <vscale x 1 x i16> @test_vlseg5ff_mask_nxv1i16(<vscale x 1 x i16> %val, p
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i64} @llvm.riscv.vlseg5ff.mask.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1420,7 +1350,6 @@ define <vscale x 1 x i16> @test_vlseg6ff_nxv1i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i64} @llvm.riscv.vlseg6ff.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1434,15 +1363,14 @@ define <vscale x 1 x i16> @test_vlseg6ff_mask_nxv1i16(<vscale x 1 x i16> %val, p
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i64} @llvm.riscv.vlseg6ff.mask.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1462,7 +1390,6 @@ define <vscale x 1 x i16> @test_vlseg7ff_nxv1i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i64} @llvm.riscv.vlseg7ff.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1476,16 +1403,15 @@ define <vscale x 1 x i16> @test_vlseg7ff_mask_nxv1i16(<vscale x 1 x i16> %val, p
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i64} @llvm.riscv.vlseg7ff.mask.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1505,7 +1431,6 @@ define <vscale x 1 x i16> @test_vlseg8ff_nxv1i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i64} @llvm.riscv.vlseg8ff.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %vl)
@@ -1519,17 +1444,16 @@ define <vscale x 1 x i16> @test_vlseg8ff_mask_nxv1i16(<vscale x 1 x i16> %val, p
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>, i64} @llvm.riscv.vlseg8ff.mask.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1549,7 +1473,6 @@ define <vscale x 2 x i32> @test_vlseg2ff_nxv2i32(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>, i64} @llvm.riscv.vlseg2ff.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1567,7 +1490,6 @@ define <vscale x 2 x i32> @test_vlseg2ff_mask_nxv2i32(<vscale x 2 x i32> %val, p
 ; CHECK-NEXT:    vlseg2e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>, i64} @llvm.riscv.vlseg2ff.mask.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -1587,7 +1509,6 @@ define <vscale x 2 x i32> @test_vlseg3ff_nxv2i32(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg3e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i64} @llvm.riscv.vlseg3ff.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1601,12 +1522,11 @@ define <vscale x 2 x i32> @test_vlseg3ff_mask_nxv2i32(<vscale x 2 x i32> %val, p
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg3e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i64} @llvm.riscv.vlseg3ff.mask.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -1626,7 +1546,6 @@ define <vscale x 2 x i32> @test_vlseg4ff_nxv2i32(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg4e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i64} @llvm.riscv.vlseg4ff.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1640,13 +1559,12 @@ define <vscale x 2 x i32> @test_vlseg4ff_mask_nxv2i32(<vscale x 2 x i32> %val, p
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg4e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i64} @llvm.riscv.vlseg4ff.mask.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -1666,7 +1584,6 @@ define <vscale x 2 x i32> @test_vlseg5ff_nxv2i32(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg5e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i64} @llvm.riscv.vlseg5ff.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1680,14 +1597,13 @@ define <vscale x 2 x i32> @test_vlseg5ff_mask_nxv2i32(<vscale x 2 x i32> %val, p
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg5e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i64} @llvm.riscv.vlseg5ff.mask.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -1707,7 +1623,6 @@ define <vscale x 2 x i32> @test_vlseg6ff_nxv2i32(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg6e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i64} @llvm.riscv.vlseg6ff.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1721,15 +1636,14 @@ define <vscale x 2 x i32> @test_vlseg6ff_mask_nxv2i32(<vscale x 2 x i32> %val, p
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg6e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i64} @llvm.riscv.vlseg6ff.mask.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -1749,7 +1663,6 @@ define <vscale x 2 x i32> @test_vlseg7ff_nxv2i32(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg7e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i64} @llvm.riscv.vlseg7ff.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1763,16 +1676,15 @@ define <vscale x 2 x i32> @test_vlseg7ff_mask_nxv2i32(<vscale x 2 x i32> %val, p
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg7e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i64} @llvm.riscv.vlseg7ff.mask.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -1792,7 +1704,6 @@ define <vscale x 2 x i32> @test_vlseg8ff_nxv2i32(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg8e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i64} @llvm.riscv.vlseg8ff.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %vl)
@@ -1806,17 +1717,16 @@ define <vscale x 2 x i32> @test_vlseg8ff_mask_nxv2i32(<vscale x 2 x i32> %val, p
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg8e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>, i64} @llvm.riscv.vlseg8ff.mask.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -1836,7 +1746,6 @@ define <vscale x 8 x i8> @test_vlseg2ff_nxv8i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg2e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>, i64} @llvm.riscv.vlseg2ff.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -1854,7 +1763,6 @@ define <vscale x 8 x i8> @test_vlseg2ff_mask_nxv8i8(<vscale x 8 x i8> %val, ptr
 ; CHECK-NEXT:    vlseg2e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>, i64} @llvm.riscv.vlseg2ff.mask.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -1874,7 +1782,6 @@ define <vscale x 8 x i8> @test_vlseg3ff_nxv8i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg3e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i64} @llvm.riscv.vlseg3ff.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -1888,12 +1795,11 @@ define <vscale x 8 x i8> @test_vlseg3ff_mask_nxv8i8(<vscale x 8 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vlseg3e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i64} @llvm.riscv.vlseg3ff.mask.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -1913,7 +1819,6 @@ define <vscale x 8 x i8> @test_vlseg4ff_nxv8i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg4e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i64} @llvm.riscv.vlseg4ff.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -1927,13 +1832,12 @@ define <vscale x 8 x i8> @test_vlseg4ff_mask_nxv8i8(<vscale x 8 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vlseg4e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i64} @llvm.riscv.vlseg4ff.mask.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -1953,7 +1857,6 @@ define <vscale x 8 x i8> @test_vlseg5ff_nxv8i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg5e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i64} @llvm.riscv.vlseg5ff.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -1967,14 +1870,13 @@ define <vscale x 8 x i8> @test_vlseg5ff_mask_nxv8i8(<vscale x 8 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vlseg5e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i64} @llvm.riscv.vlseg5ff.mask.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -1994,7 +1896,6 @@ define <vscale x 8 x i8> @test_vlseg6ff_nxv8i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg6e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i64} @llvm.riscv.vlseg6ff.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -2008,15 +1909,14 @@ define <vscale x 8 x i8> @test_vlseg6ff_mask_nxv8i8(<vscale x 8 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vlseg6e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i64} @llvm.riscv.vlseg6ff.mask.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2036,7 +1936,6 @@ define <vscale x 8 x i8> @test_vlseg7ff_nxv8i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg7e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i64} @llvm.riscv.vlseg7ff.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -2050,16 +1949,15 @@ define <vscale x 8 x i8> @test_vlseg7ff_mask_nxv8i8(<vscale x 8 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vlseg7e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i64} @llvm.riscv.vlseg7ff.mask.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2079,7 +1977,6 @@ define <vscale x 8 x i8> @test_vlseg8ff_nxv8i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg8e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i64} @llvm.riscv.vlseg8ff.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %vl)
@@ -2093,17 +1990,16 @@ define <vscale x 8 x i8> @test_vlseg8ff_mask_nxv8i8(<vscale x 8 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vlseg8e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>, i64} @llvm.riscv.vlseg8ff.mask.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2123,7 +2019,6 @@ define <vscale x 4 x i64> @test_vlseg2ff_nxv4i64(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e64ff.v v4, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i64>,<vscale x 4 x i64>, i64} @llvm.riscv.vlseg2ff.nxv4i64(<vscale x 4 x i64> undef, <vscale x 4 x i64> undef, ptr %base, i64 %vl)
@@ -2141,7 +2036,6 @@ define <vscale x 4 x i64> @test_vlseg2ff_mask_nxv4i64(<vscale x 4 x i64> %val, p
 ; CHECK-NEXT:    vlseg2e64ff.v v4, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i64>,<vscale x 4 x i64>, i64} @llvm.riscv.vlseg2ff.mask.nxv4i64(<vscale x 4 x i64> %val,<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -2161,7 +2055,6 @@ define <vscale x 4 x i16> @test_vlseg2ff_nxv4i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>, i64} @llvm.riscv.vlseg2ff.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -2179,7 +2072,6 @@ define <vscale x 4 x i16> @test_vlseg2ff_mask_nxv4i16(<vscale x 4 x i16> %val, p
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>, i64} @llvm.riscv.vlseg2ff.mask.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -2199,7 +2091,6 @@ define <vscale x 4 x i16> @test_vlseg3ff_nxv4i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i64} @llvm.riscv.vlseg3ff.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -2213,12 +2104,11 @@ define <vscale x 4 x i16> @test_vlseg3ff_mask_nxv4i16(<vscale x 4 x i16> %val, p
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i64} @llvm.riscv.vlseg3ff.mask.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -2238,7 +2128,6 @@ define <vscale x 4 x i16> @test_vlseg4ff_nxv4i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i64} @llvm.riscv.vlseg4ff.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -2252,13 +2141,12 @@ define <vscale x 4 x i16> @test_vlseg4ff_mask_nxv4i16(<vscale x 4 x i16> %val, p
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i64} @llvm.riscv.vlseg4ff.mask.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -2278,7 +2166,6 @@ define <vscale x 4 x i16> @test_vlseg5ff_nxv4i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i64} @llvm.riscv.vlseg5ff.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -2292,14 +2179,13 @@ define <vscale x 4 x i16> @test_vlseg5ff_mask_nxv4i16(<vscale x 4 x i16> %val, p
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i64} @llvm.riscv.vlseg5ff.mask.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -2319,7 +2205,6 @@ define <vscale x 4 x i16> @test_vlseg6ff_nxv4i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i64} @llvm.riscv.vlseg6ff.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -2333,15 +2218,14 @@ define <vscale x 4 x i16> @test_vlseg6ff_mask_nxv4i16(<vscale x 4 x i16> %val, p
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i64} @llvm.riscv.vlseg6ff.mask.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -2361,7 +2245,6 @@ define <vscale x 4 x i16> @test_vlseg7ff_nxv4i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i64} @llvm.riscv.vlseg7ff.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -2375,16 +2258,15 @@ define <vscale x 4 x i16> @test_vlseg7ff_mask_nxv4i16(<vscale x 4 x i16> %val, p
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i64} @llvm.riscv.vlseg7ff.mask.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -2404,7 +2286,6 @@ define <vscale x 4 x i16> @test_vlseg8ff_nxv4i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i64} @llvm.riscv.vlseg8ff.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %vl)
@@ -2418,17 +2299,16 @@ define <vscale x 4 x i16> @test_vlseg8ff_mask_nxv4i16(<vscale x 4 x i16> %val, p
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>, i64} @llvm.riscv.vlseg8ff.mask.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -2448,7 +2328,6 @@ define <vscale x 1 x i8> @test_vlseg2ff_nxv1i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg2e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>, i64} @llvm.riscv.vlseg2ff.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2466,7 +2345,6 @@ define <vscale x 1 x i8> @test_vlseg2ff_mask_nxv1i8(<vscale x 1 x i8> %val, ptr
 ; CHECK-NEXT:    vlseg2e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>, i64} @llvm.riscv.vlseg2ff.mask.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -2486,7 +2364,6 @@ define <vscale x 1 x i8> @test_vlseg3ff_nxv1i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg3e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i64} @llvm.riscv.vlseg3ff.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2500,12 +2377,11 @@ define <vscale x 1 x i8> @test_vlseg3ff_mask_nxv1i8(<vscale x 1 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vlseg3e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i64} @llvm.riscv.vlseg3ff.mask.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -2525,7 +2401,6 @@ define <vscale x 1 x i8> @test_vlseg4ff_nxv1i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg4e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i64} @llvm.riscv.vlseg4ff.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2539,13 +2414,12 @@ define <vscale x 1 x i8> @test_vlseg4ff_mask_nxv1i8(<vscale x 1 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vlseg4e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i64} @llvm.riscv.vlseg4ff.mask.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -2565,7 +2439,6 @@ define <vscale x 1 x i8> @test_vlseg5ff_nxv1i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg5e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i64} @llvm.riscv.vlseg5ff.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2579,14 +2452,13 @@ define <vscale x 1 x i8> @test_vlseg5ff_mask_nxv1i8(<vscale x 1 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vlseg5e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i64} @llvm.riscv.vlseg5ff.mask.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -2606,7 +2478,6 @@ define <vscale x 1 x i8> @test_vlseg6ff_nxv1i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg6e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i64} @llvm.riscv.vlseg6ff.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2620,15 +2491,14 @@ define <vscale x 1 x i8> @test_vlseg6ff_mask_nxv1i8(<vscale x 1 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vlseg6e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i64} @llvm.riscv.vlseg6ff.mask.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -2648,7 +2518,6 @@ define <vscale x 1 x i8> @test_vlseg7ff_nxv1i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg7e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i64} @llvm.riscv.vlseg7ff.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2662,16 +2531,15 @@ define <vscale x 1 x i8> @test_vlseg7ff_mask_nxv1i8(<vscale x 1 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vlseg7e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i64} @llvm.riscv.vlseg7ff.mask.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -2691,7 +2559,6 @@ define <vscale x 1 x i8> @test_vlseg8ff_nxv1i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg8e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i64} @llvm.riscv.vlseg8ff.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %vl)
@@ -2705,17 +2572,16 @@ define <vscale x 1 x i8> @test_vlseg8ff_mask_nxv1i8(<vscale x 1 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vlseg8e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>, i64} @llvm.riscv.vlseg8ff.mask.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -2735,7 +2601,6 @@ define <vscale x 2 x i8> @test_vlseg2ff_nxv2i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg2e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>, i64} @llvm.riscv.vlseg2ff.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2753,7 +2618,6 @@ define <vscale x 2 x i8> @test_vlseg2ff_mask_nxv2i8(<vscale x 2 x i8> %val, ptr
 ; CHECK-NEXT:    vlseg2e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>, i64} @llvm.riscv.vlseg2ff.mask.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -2773,7 +2637,6 @@ define <vscale x 2 x i8> @test_vlseg3ff_nxv2i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg3e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i64} @llvm.riscv.vlseg3ff.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2787,12 +2650,11 @@ define <vscale x 2 x i8> @test_vlseg3ff_mask_nxv2i8(<vscale x 2 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vlseg3e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i64} @llvm.riscv.vlseg3ff.mask.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -2812,7 +2674,6 @@ define <vscale x 2 x i8> @test_vlseg4ff_nxv2i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg4e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i64} @llvm.riscv.vlseg4ff.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2826,13 +2687,12 @@ define <vscale x 2 x i8> @test_vlseg4ff_mask_nxv2i8(<vscale x 2 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vlseg4e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i64} @llvm.riscv.vlseg4ff.mask.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -2852,7 +2712,6 @@ define <vscale x 2 x i8> @test_vlseg5ff_nxv2i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg5e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i64} @llvm.riscv.vlseg5ff.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2866,14 +2725,13 @@ define <vscale x 2 x i8> @test_vlseg5ff_mask_nxv2i8(<vscale x 2 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vlseg5e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i64} @llvm.riscv.vlseg5ff.mask.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -2893,7 +2751,6 @@ define <vscale x 2 x i8> @test_vlseg6ff_nxv2i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg6e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i64} @llvm.riscv.vlseg6ff.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2907,15 +2764,14 @@ define <vscale x 2 x i8> @test_vlseg6ff_mask_nxv2i8(<vscale x 2 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vlseg6e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i64} @llvm.riscv.vlseg6ff.mask.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -2935,7 +2791,6 @@ define <vscale x 2 x i8> @test_vlseg7ff_nxv2i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg7e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i64} @llvm.riscv.vlseg7ff.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2949,16 +2804,15 @@ define <vscale x 2 x i8> @test_vlseg7ff_mask_nxv2i8(<vscale x 2 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vlseg7e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i64} @llvm.riscv.vlseg7ff.mask.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -2978,7 +2832,6 @@ define <vscale x 2 x i8> @test_vlseg8ff_nxv2i8(ptr %base, i64 %vl, ptr %outvl) {
 ; CHECK-NEXT:    vlseg8e8ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i64} @llvm.riscv.vlseg8ff.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %vl)
@@ -2992,17 +2845,16 @@ define <vscale x 2 x i8> @test_vlseg8ff_mask_nxv2i8(<vscale x 2 x i8> %val, ptr
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vlseg8e8ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>, i64} @llvm.riscv.vlseg8ff.mask.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -3022,7 +2874,6 @@ define <vscale x 8 x i32> @test_vlseg2ff_nxv8i32(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e32ff.v v4, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>, i64} @llvm.riscv.vlseg2ff.nxv8i32(<vscale x 8 x i32> undef, <vscale x 8 x i32> undef, ptr %base, i64 %vl)
@@ -3040,7 +2891,6 @@ define <vscale x 8 x i32> @test_vlseg2ff_mask_nxv8i32(<vscale x 8 x i32> %val, p
 ; CHECK-NEXT:    vlseg2e32ff.v v4, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>, i64} @llvm.riscv.vlseg2ff.mask.nxv8i32(<vscale x 8 x i32> %val,<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -3060,7 +2910,6 @@ define <vscale x 32 x i8> @test_vlseg2ff_nxv32i8(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e8ff.v v4, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>, i64} @llvm.riscv.vlseg2ff.nxv32i8(<vscale x 32 x i8> undef, <vscale x 32 x i8> undef, ptr %base, i64 %vl)
@@ -3078,7 +2927,6 @@ define <vscale x 32 x i8> @test_vlseg2ff_mask_nxv32i8(<vscale x 32 x i8> %val, p
 ; CHECK-NEXT:    vlseg2e8ff.v v4, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>, i64} @llvm.riscv.vlseg2ff.mask.nxv32i8(<vscale x 32 x i8> %val,<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i1> %mask, i64 %vl, i64 1)
@@ -3098,7 +2946,6 @@ define <vscale x 2 x i16> @test_vlseg2ff_nxv2i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>, i64} @llvm.riscv.vlseg2ff.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -3116,7 +2963,6 @@ define <vscale x 2 x i16> @test_vlseg2ff_mask_nxv2i16(<vscale x 2 x i16> %val, p
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>, i64} @llvm.riscv.vlseg2ff.mask.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -3136,7 +2982,6 @@ define <vscale x 2 x i16> @test_vlseg3ff_nxv2i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i64} @llvm.riscv.vlseg3ff.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -3150,12 +2995,11 @@ define <vscale x 2 x i16> @test_vlseg3ff_mask_nxv2i16(<vscale x 2 x i16> %val, p
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i64} @llvm.riscv.vlseg3ff.mask.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -3175,7 +3019,6 @@ define <vscale x 2 x i16> @test_vlseg4ff_nxv2i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i64} @llvm.riscv.vlseg4ff.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -3189,13 +3032,12 @@ define <vscale x 2 x i16> @test_vlseg4ff_mask_nxv2i16(<vscale x 2 x i16> %val, p
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i64} @llvm.riscv.vlseg4ff.mask.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -3215,7 +3057,6 @@ define <vscale x 2 x i16> @test_vlseg5ff_nxv2i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i64} @llvm.riscv.vlseg5ff.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -3229,14 +3070,13 @@ define <vscale x 2 x i16> @test_vlseg5ff_mask_nxv2i16(<vscale x 2 x i16> %val, p
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i64} @llvm.riscv.vlseg5ff.mask.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -3256,7 +3096,6 @@ define <vscale x 2 x i16> @test_vlseg6ff_nxv2i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i64} @llvm.riscv.vlseg6ff.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -3270,15 +3109,14 @@ define <vscale x 2 x i16> @test_vlseg6ff_mask_nxv2i16(<vscale x 2 x i16> %val, p
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i64} @llvm.riscv.vlseg6ff.mask.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -3298,7 +3136,6 @@ define <vscale x 2 x i16> @test_vlseg7ff_nxv2i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i64} @llvm.riscv.vlseg7ff.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -3312,16 +3149,15 @@ define <vscale x 2 x i16> @test_vlseg7ff_mask_nxv2i16(<vscale x 2 x i16> %val, p
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i64} @llvm.riscv.vlseg7ff.mask.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -3341,7 +3177,6 @@ define <vscale x 2 x i16> @test_vlseg8ff_nxv2i16(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i64} @llvm.riscv.vlseg8ff.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %vl)
@@ -3355,17 +3190,16 @@ define <vscale x 2 x i16> @test_vlseg8ff_mask_nxv2i16(<vscale x 2 x i16> %val, p
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>, i64} @llvm.riscv.vlseg8ff.mask.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -3385,7 +3219,6 @@ define <vscale x 2 x i64> @test_vlseg2ff_nxv2i64(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg2e64ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>, i64} @llvm.riscv.vlseg2ff.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i64> undef, ptr %base, i64 %vl)
@@ -3403,7 +3236,6 @@ define <vscale x 2 x i64> @test_vlseg2ff_mask_nxv2i64(<vscale x 2 x i64> %val, p
 ; CHECK-NEXT:    vlseg2e64ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>, i64} @llvm.riscv.vlseg2ff.mask.nxv2i64(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -3423,7 +3255,6 @@ define <vscale x 2 x i64> @test_vlseg3ff_nxv2i64(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg3e64ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>, i64} @llvm.riscv.vlseg3ff.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, ptr %base, i64 %vl)
@@ -3437,12 +3268,11 @@ define <vscale x 2 x i64> @test_vlseg3ff_mask_nxv2i64(<vscale x 2 x i64> %val, p
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vlseg3e64ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>, i64} @llvm.riscv.vlseg3ff.mask.nxv2i64(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -3462,7 +3292,6 @@ define <vscale x 2 x i64> @test_vlseg4ff_nxv2i64(ptr %base, i64 %vl, ptr %outvl)
 ; CHECK-NEXT:    vlseg4e64ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>, i64} @llvm.riscv.vlseg4ff.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, ptr %base, i64 %vl)
@@ -3476,13 +3305,12 @@ define <vscale x 2 x i64> @test_vlseg4ff_mask_nxv2i64(<vscale x 2 x i64> %val, p
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vlseg4e64ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>, i64} @llvm.riscv.vlseg4ff.mask.nxv2i64(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val,<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -3502,7 +3330,6 @@ define <vscale x 16 x half> @test_vlseg2ff_nxv16f16(ptr %base, i64 %vl, ptr %out
 ; CHECK-NEXT:    vlseg2e16ff.v v4, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>, i64} @llvm.riscv.vlseg2ff.nxv16f16(<vscale x 16 x half> undef, <vscale x 16 x half> undef, ptr %base, i64 %vl)
@@ -3520,7 +3347,6 @@ define <vscale x 16 x half> @test_vlseg2ff_mask_nxv16f16(<vscale x 16 x half> %v
 ; CHECK-NEXT:    vlseg2e16ff.v v4, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>, i64} @llvm.riscv.vlseg2ff.mask.nxv16f16(<vscale x 16 x half> %val,<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -3540,7 +3366,6 @@ define <vscale x 4 x double> @test_vlseg2ff_nxv4f64(ptr %base, i64 %vl, ptr %out
 ; CHECK-NEXT:    vlseg2e64ff.v v4, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>, i64} @llvm.riscv.vlseg2ff.nxv4f64(<vscale x 4 x double> undef, <vscale x 4 x double> undef, ptr %base, i64 %vl)
@@ -3558,7 +3383,6 @@ define <vscale x 4 x double> @test_vlseg2ff_mask_nxv4f64(<vscale x 4 x double> %
 ; CHECK-NEXT:    vlseg2e64ff.v v4, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>, i64} @llvm.riscv.vlseg2ff.mask.nxv4f64(<vscale x 4 x double> %val,<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3578,7 +3402,6 @@ define <vscale x 1 x double> @test_vlseg2ff_nxv1f64(ptr %base, i64 %vl, ptr %out
 ; CHECK-NEXT:    vlseg2e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>, i64} @llvm.riscv.vlseg2ff.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3596,7 +3419,6 @@ define <vscale x 1 x double> @test_vlseg2ff_mask_nxv1f64(<vscale x 1 x double> %
 ; CHECK-NEXT:    vlseg2e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>, i64} @llvm.riscv.vlseg2ff.mask.nxv1f64(<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -3616,7 +3438,6 @@ define <vscale x 1 x double> @test_vlseg3ff_nxv1f64(ptr %base, i64 %vl, ptr %out
 ; CHECK-NEXT:    vlseg3e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i64} @llvm.riscv.vlseg3ff.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3630,12 +3451,11 @@ define <vscale x 1 x double> @test_vlseg3ff_mask_nxv1f64(<vscale x 1 x double> %
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vlseg3e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i64} @llvm.riscv.vlseg3ff.mask.nxv1f64(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -3655,7 +3475,6 @@ define <vscale x 1 x double> @test_vlseg4ff_nxv1f64(ptr %base, i64 %vl, ptr %out
 ; CHECK-NEXT:    vlseg4e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i64} @llvm.riscv.vlseg4ff.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3669,13 +3488,12 @@ define <vscale x 1 x double> @test_vlseg4ff_mask_nxv1f64(<vscale x 1 x double> %
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vlseg4e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i64} @llvm.riscv.vlseg4ff.mask.nxv1f64(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -3695,7 +3513,6 @@ define <vscale x 1 x double> @test_vlseg5ff_nxv1f64(ptr %base, i64 %vl, ptr %out
 ; CHECK-NEXT:    vlseg5e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i64} @llvm.riscv.vlseg5ff.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3709,14 +3526,13 @@ define <vscale x 1 x double> @test_vlseg5ff_mask_nxv1f64(<vscale x 1 x double> %
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vlseg5e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i64} @llvm.riscv.vlseg5ff.mask.nxv1f64(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -3736,7 +3552,6 @@ define <vscale x 1 x double> @test_vlseg6ff_nxv1f64(ptr %base, i64 %vl, ptr %out
 ; CHECK-NEXT:    vlseg6e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i64} @llvm.riscv.vlseg6ff.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3750,15 +3565,14 @@ define <vscale x 1 x double> @test_vlseg6ff_mask_nxv1f64(<vscale x 1 x double> %
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vlseg6e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i64} @llvm.riscv.vlseg6ff.mask.nxv1f64(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -3778,7 +3592,6 @@ define <vscale x 1 x double> @test_vlseg7ff_nxv1f64(ptr %base, i64 %vl, ptr %out
 ; CHECK-NEXT:    vlseg7e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i64} @llvm.riscv.vlseg7ff.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3792,16 +3605,15 @@ define <vscale x 1 x double> @test_vlseg7ff_mask_nxv1f64(<vscale x 1 x double> %
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vlseg7e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i64} @llvm.riscv.vlseg7ff.mask.nxv1f64(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -3821,7 +3633,6 @@ define <vscale x 1 x double> @test_vlseg8ff_nxv1f64(ptr %base, i64 %vl, ptr %out
 ; CHECK-NEXT:    vlseg8e64ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i64} @llvm.riscv.vlseg8ff.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %vl)
@@ -3835,17 +3646,16 @@ define <vscale x 1 x double> @test_vlseg8ff_mask_nxv1f64(<vscale x 1 x double> %
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vlseg8e64ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>, i64} @llvm.riscv.vlseg8ff.mask.nxv1f64(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -3865,7 +3675,6 @@ define <vscale x 2 x float> @test_vlseg2ff_nxv2f32(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg2e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>, i64} @llvm.riscv.vlseg2ff.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -3883,7 +3692,6 @@ define <vscale x 2 x float> @test_vlseg2ff_mask_nxv2f32(<vscale x 2 x float> %va
 ; CHECK-NEXT:    vlseg2e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>, i64} @llvm.riscv.vlseg2ff.mask.nxv2f32(<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -3903,7 +3711,6 @@ define <vscale x 2 x float> @test_vlseg3ff_nxv2f32(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg3e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i64} @llvm.riscv.vlseg3ff.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -3917,12 +3724,11 @@ define <vscale x 2 x float> @test_vlseg3ff_mask_nxv2f32(<vscale x 2 x float> %va
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg3e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i64} @llvm.riscv.vlseg3ff.mask.nxv2f32(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -3942,7 +3748,6 @@ define <vscale x 2 x float> @test_vlseg4ff_nxv2f32(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg4e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i64} @llvm.riscv.vlseg4ff.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -3956,13 +3761,12 @@ define <vscale x 2 x float> @test_vlseg4ff_mask_nxv2f32(<vscale x 2 x float> %va
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg4e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i64} @llvm.riscv.vlseg4ff.mask.nxv2f32(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -3982,7 +3786,6 @@ define <vscale x 2 x float> @test_vlseg5ff_nxv2f32(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg5e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i64} @llvm.riscv.vlseg5ff.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -3996,14 +3799,13 @@ define <vscale x 2 x float> @test_vlseg5ff_mask_nxv2f32(<vscale x 2 x float> %va
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg5e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i64} @llvm.riscv.vlseg5ff.mask.nxv2f32(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -4023,7 +3825,6 @@ define <vscale x 2 x float> @test_vlseg6ff_nxv2f32(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg6e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i64} @llvm.riscv.vlseg6ff.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -4037,15 +3838,14 @@ define <vscale x 2 x float> @test_vlseg6ff_mask_nxv2f32(<vscale x 2 x float> %va
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg6e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i64} @llvm.riscv.vlseg6ff.mask.nxv2f32(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -4065,7 +3865,6 @@ define <vscale x 2 x float> @test_vlseg7ff_nxv2f32(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg7e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i64} @llvm.riscv.vlseg7ff.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -4079,16 +3878,15 @@ define <vscale x 2 x float> @test_vlseg7ff_mask_nxv2f32(<vscale x 2 x float> %va
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg7e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i64} @llvm.riscv.vlseg7ff.mask.nxv2f32(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -4108,7 +3906,6 @@ define <vscale x 2 x float> @test_vlseg8ff_nxv2f32(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg8e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i64} @llvm.riscv.vlseg8ff.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %vl)
@@ -4122,17 +3919,16 @@ define <vscale x 2 x float> @test_vlseg8ff_mask_nxv2f32(<vscale x 2 x float> %va
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vlseg8e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>, i64} @llvm.riscv.vlseg8ff.mask.nxv2f32(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -4152,7 +3948,6 @@ define <vscale x 1 x half> @test_vlseg2ff_nxv1f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>, i64} @llvm.riscv.vlseg2ff.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -4170,7 +3965,6 @@ define <vscale x 1 x half> @test_vlseg2ff_mask_nxv1f16(<vscale x 1 x half> %val,
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>, i64} @llvm.riscv.vlseg2ff.mask.nxv1f16(<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4190,7 +3984,6 @@ define <vscale x 1 x half> @test_vlseg3ff_nxv1f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i64} @llvm.riscv.vlseg3ff.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -4204,12 +3997,11 @@ define <vscale x 1 x half> @test_vlseg3ff_mask_nxv1f16(<vscale x 1 x half> %val,
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i64} @llvm.riscv.vlseg3ff.mask.nxv1f16(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4229,7 +4021,6 @@ define <vscale x 1 x half> @test_vlseg4ff_nxv1f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i64} @llvm.riscv.vlseg4ff.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -4243,13 +4034,12 @@ define <vscale x 1 x half> @test_vlseg4ff_mask_nxv1f16(<vscale x 1 x half> %val,
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i64} @llvm.riscv.vlseg4ff.mask.nxv1f16(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4269,7 +4059,6 @@ define <vscale x 1 x half> @test_vlseg5ff_nxv1f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i64} @llvm.riscv.vlseg5ff.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -4283,14 +4072,13 @@ define <vscale x 1 x half> @test_vlseg5ff_mask_nxv1f16(<vscale x 1 x half> %val,
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i64} @llvm.riscv.vlseg5ff.mask.nxv1f16(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4310,7 +4098,6 @@ define <vscale x 1 x half> @test_vlseg6ff_nxv1f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i64} @llvm.riscv.vlseg6ff.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -4324,15 +4111,14 @@ define <vscale x 1 x half> @test_vlseg6ff_mask_nxv1f16(<vscale x 1 x half> %val,
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i64} @llvm.riscv.vlseg6ff.mask.nxv1f16(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4352,7 +4138,6 @@ define <vscale x 1 x half> @test_vlseg7ff_nxv1f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i64} @llvm.riscv.vlseg7ff.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -4366,16 +4151,15 @@ define <vscale x 1 x half> @test_vlseg7ff_mask_nxv1f16(<vscale x 1 x half> %val,
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i64} @llvm.riscv.vlseg7ff.mask.nxv1f16(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4395,7 +4179,6 @@ define <vscale x 1 x half> @test_vlseg8ff_nxv1f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i64} @llvm.riscv.vlseg8ff.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %vl)
@@ -4409,17 +4192,16 @@ define <vscale x 1 x half> @test_vlseg8ff_mask_nxv1f16(<vscale x 1 x half> %val,
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>, i64} @llvm.riscv.vlseg8ff.mask.nxv1f16(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4439,7 +4221,6 @@ define <vscale x 1 x float> @test_vlseg2ff_nxv1f32(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg2e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>, i64} @llvm.riscv.vlseg2ff.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -4457,7 +4238,6 @@ define <vscale x 1 x float> @test_vlseg2ff_mask_nxv1f32(<vscale x 1 x float> %va
 ; CHECK-NEXT:    vlseg2e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>, i64} @llvm.riscv.vlseg2ff.mask.nxv1f32(<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4477,7 +4257,6 @@ define <vscale x 1 x float> @test_vlseg3ff_nxv1f32(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg3e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i64} @llvm.riscv.vlseg3ff.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -4491,12 +4270,11 @@ define <vscale x 1 x float> @test_vlseg3ff_mask_nxv1f32(<vscale x 1 x float> %va
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg3e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i64} @llvm.riscv.vlseg3ff.mask.nxv1f32(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4516,7 +4294,6 @@ define <vscale x 1 x float> @test_vlseg4ff_nxv1f32(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg4e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i64} @llvm.riscv.vlseg4ff.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -4530,13 +4307,12 @@ define <vscale x 1 x float> @test_vlseg4ff_mask_nxv1f32(<vscale x 1 x float> %va
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg4e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i64} @llvm.riscv.vlseg4ff.mask.nxv1f32(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4556,7 +4332,6 @@ define <vscale x 1 x float> @test_vlseg5ff_nxv1f32(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg5e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i64} @llvm.riscv.vlseg5ff.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -4570,14 +4345,13 @@ define <vscale x 1 x float> @test_vlseg5ff_mask_nxv1f32(<vscale x 1 x float> %va
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg5e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i64} @llvm.riscv.vlseg5ff.mask.nxv1f32(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4597,7 +4371,6 @@ define <vscale x 1 x float> @test_vlseg6ff_nxv1f32(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg6e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i64} @llvm.riscv.vlseg6ff.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -4611,15 +4384,14 @@ define <vscale x 1 x float> @test_vlseg6ff_mask_nxv1f32(<vscale x 1 x float> %va
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg6e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i64} @llvm.riscv.vlseg6ff.mask.nxv1f32(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4639,7 +4411,6 @@ define <vscale x 1 x float> @test_vlseg7ff_nxv1f32(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg7e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i64} @llvm.riscv.vlseg7ff.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -4653,16 +4424,15 @@ define <vscale x 1 x float> @test_vlseg7ff_mask_nxv1f32(<vscale x 1 x float> %va
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg7e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i64} @llvm.riscv.vlseg7ff.mask.nxv1f32(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4682,7 +4452,6 @@ define <vscale x 1 x float> @test_vlseg8ff_nxv1f32(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg8e32ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i64} @llvm.riscv.vlseg8ff.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %vl)
@@ -4696,17 +4465,16 @@ define <vscale x 1 x float> @test_vlseg8ff_mask_nxv1f32(<vscale x 1 x float> %va
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vlseg8e32ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>, i64} @llvm.riscv.vlseg8ff.mask.nxv1f32(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4726,7 +4494,6 @@ define <vscale x 8 x half> @test_vlseg2ff_nxv8f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg2e16ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>, i64} @llvm.riscv.vlseg2ff.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i64 %vl)
@@ -4744,7 +4511,6 @@ define <vscale x 8 x half> @test_vlseg2ff_mask_nxv8f16(<vscale x 8 x half> %val,
 ; CHECK-NEXT:    vlseg2e16ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>, i64} @llvm.riscv.vlseg2ff.mask.nxv8f16(<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -4764,7 +4530,6 @@ define <vscale x 8 x half> @test_vlseg3ff_nxv8f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg3e16ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>, i64} @llvm.riscv.vlseg3ff.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i64 %vl)
@@ -4778,12 +4543,11 @@ define <vscale x 8 x half> @test_vlseg3ff_mask_nxv8f16(<vscale x 8 x half> %val,
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vlseg3e16ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>, i64} @llvm.riscv.vlseg3ff.mask.nxv8f16(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -4803,7 +4567,6 @@ define <vscale x 8 x half> @test_vlseg4ff_nxv8f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg4e16ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>, i64} @llvm.riscv.vlseg4ff.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i64 %vl)
@@ -4817,13 +4580,12 @@ define <vscale x 8 x half> @test_vlseg4ff_mask_nxv8f16(<vscale x 8 x half> %val,
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vlseg4e16ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>, i64} @llvm.riscv.vlseg4ff.mask.nxv8f16(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -4843,7 +4605,6 @@ define <vscale x 8 x float> @test_vlseg2ff_nxv8f32(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg2e32ff.v v4, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>, i64} @llvm.riscv.vlseg2ff.nxv8f32(<vscale x 8 x float> undef, <vscale x 8 x float> undef, ptr %base, i64 %vl)
@@ -4861,7 +4622,6 @@ define <vscale x 8 x float> @test_vlseg2ff_mask_nxv8f32(<vscale x 8 x float> %va
 ; CHECK-NEXT:    vlseg2e32ff.v v4, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>, i64} @llvm.riscv.vlseg2ff.mask.nxv8f32(<vscale x 8 x float> %val,<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -4881,7 +4641,6 @@ define <vscale x 2 x double> @test_vlseg2ff_nxv2f64(ptr %base, i64 %vl, ptr %out
 ; CHECK-NEXT:    vlseg2e64ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>, i64} @llvm.riscv.vlseg2ff.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i64 %vl)
@@ -4899,7 +4658,6 @@ define <vscale x 2 x double> @test_vlseg2ff_mask_nxv2f64(<vscale x 2 x double> %
 ; CHECK-NEXT:    vlseg2e64ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>, i64} @llvm.riscv.vlseg2ff.mask.nxv2f64(<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -4919,7 +4677,6 @@ define <vscale x 2 x double> @test_vlseg3ff_nxv2f64(ptr %base, i64 %vl, ptr %out
 ; CHECK-NEXT:    vlseg3e64ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>, i64} @llvm.riscv.vlseg3ff.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i64 %vl)
@@ -4933,12 +4690,11 @@ define <vscale x 2 x double> @test_vlseg3ff_mask_nxv2f64(<vscale x 2 x double> %
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vlseg3e64ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>, i64} @llvm.riscv.vlseg3ff.mask.nxv2f64(<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -4958,7 +4714,6 @@ define <vscale x 2 x double> @test_vlseg4ff_nxv2f64(ptr %base, i64 %vl, ptr %out
 ; CHECK-NEXT:    vlseg4e64ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>, i64} @llvm.riscv.vlseg4ff.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i64 %vl)
@@ -4972,13 +4727,12 @@ define <vscale x 2 x double> @test_vlseg4ff_mask_nxv2f64(<vscale x 2 x double> %
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vlseg4e64ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>, i64} @llvm.riscv.vlseg4ff.mask.nxv2f64(<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -4998,7 +4752,6 @@ define <vscale x 4 x half> @test_vlseg2ff_nxv4f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>, i64} @llvm.riscv.vlseg2ff.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -5016,7 +4769,6 @@ define <vscale x 4 x half> @test_vlseg2ff_mask_nxv4f16(<vscale x 4 x half> %val,
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>, i64} @llvm.riscv.vlseg2ff.mask.nxv4f16(<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -5036,7 +4788,6 @@ define <vscale x 4 x half> @test_vlseg3ff_nxv4f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i64} @llvm.riscv.vlseg3ff.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -5050,12 +4801,11 @@ define <vscale x 4 x half> @test_vlseg3ff_mask_nxv4f16(<vscale x 4 x half> %val,
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i64} @llvm.riscv.vlseg3ff.mask.nxv4f16(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -5075,7 +4825,6 @@ define <vscale x 4 x half> @test_vlseg4ff_nxv4f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i64} @llvm.riscv.vlseg4ff.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -5089,13 +4838,12 @@ define <vscale x 4 x half> @test_vlseg4ff_mask_nxv4f16(<vscale x 4 x half> %val,
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i64} @llvm.riscv.vlseg4ff.mask.nxv4f16(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -5115,7 +4863,6 @@ define <vscale x 4 x half> @test_vlseg5ff_nxv4f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i64} @llvm.riscv.vlseg5ff.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -5129,14 +4876,13 @@ define <vscale x 4 x half> @test_vlseg5ff_mask_nxv4f16(<vscale x 4 x half> %val,
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i64} @llvm.riscv.vlseg5ff.mask.nxv4f16(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -5156,7 +4902,6 @@ define <vscale x 4 x half> @test_vlseg6ff_nxv4f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i64} @llvm.riscv.vlseg6ff.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -5170,15 +4915,14 @@ define <vscale x 4 x half> @test_vlseg6ff_mask_nxv4f16(<vscale x 4 x half> %val,
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i64} @llvm.riscv.vlseg6ff.mask.nxv4f16(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -5198,7 +4942,6 @@ define <vscale x 4 x half> @test_vlseg7ff_nxv4f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i64} @llvm.riscv.vlseg7ff.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -5212,16 +4955,15 @@ define <vscale x 4 x half> @test_vlseg7ff_mask_nxv4f16(<vscale x 4 x half> %val,
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i64} @llvm.riscv.vlseg7ff.mask.nxv4f16(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -5241,7 +4983,6 @@ define <vscale x 4 x half> @test_vlseg8ff_nxv4f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i64} @llvm.riscv.vlseg8ff.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %vl)
@@ -5255,17 +4996,16 @@ define <vscale x 4 x half> @test_vlseg8ff_mask_nxv4f16(<vscale x 4 x half> %val,
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>, i64} @llvm.riscv.vlseg8ff.mask.nxv4f16(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -5285,7 +5025,6 @@ define <vscale x 2 x half> @test_vlseg2ff_nxv2f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>, i64} @llvm.riscv.vlseg2ff.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -5303,7 +5042,6 @@ define <vscale x 2 x half> @test_vlseg2ff_mask_nxv2f16(<vscale x 2 x half> %val,
 ; CHECK-NEXT:    vlseg2e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>, i64} @llvm.riscv.vlseg2ff.mask.nxv2f16(<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5323,7 +5061,6 @@ define <vscale x 2 x half> @test_vlseg3ff_nxv2f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i64} @llvm.riscv.vlseg3ff.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -5337,12 +5074,11 @@ define <vscale x 2 x half> @test_vlseg3ff_mask_nxv2f16(<vscale x 2 x half> %val,
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg3e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i64} @llvm.riscv.vlseg3ff.mask.nxv2f16(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5362,7 +5098,6 @@ define <vscale x 2 x half> @test_vlseg4ff_nxv2f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i64} @llvm.riscv.vlseg4ff.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -5376,13 +5111,12 @@ define <vscale x 2 x half> @test_vlseg4ff_mask_nxv2f16(<vscale x 2 x half> %val,
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg4e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i64} @llvm.riscv.vlseg4ff.mask.nxv2f16(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5402,7 +5136,6 @@ define <vscale x 2 x half> @test_vlseg5ff_nxv2f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i64} @llvm.riscv.vlseg5ff.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -5416,14 +5149,13 @@ define <vscale x 2 x half> @test_vlseg5ff_mask_nxv2f16(<vscale x 2 x half> %val,
 ; CHECK-LABEL: test_vlseg5ff_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg5e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i64} @llvm.riscv.vlseg5ff.mask.nxv2f16(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5443,7 +5175,6 @@ define <vscale x 2 x half> @test_vlseg6ff_nxv2f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i64} @llvm.riscv.vlseg6ff.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -5457,15 +5188,14 @@ define <vscale x 2 x half> @test_vlseg6ff_mask_nxv2f16(<vscale x 2 x half> %val,
 ; CHECK-LABEL: test_vlseg6ff_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg6e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i64} @llvm.riscv.vlseg6ff.mask.nxv2f16(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5485,7 +5215,6 @@ define <vscale x 2 x half> @test_vlseg7ff_nxv2f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i64} @llvm.riscv.vlseg7ff.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -5499,16 +5228,15 @@ define <vscale x 2 x half> @test_vlseg7ff_mask_nxv2f16(<vscale x 2 x half> %val,
 ; CHECK-LABEL: test_vlseg7ff_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg7e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i64} @llvm.riscv.vlseg7ff.mask.nxv2f16(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5528,7 +5256,6 @@ define <vscale x 2 x half> @test_vlseg8ff_nxv2f16(ptr %base, i64 %vl, ptr %outvl
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i64} @llvm.riscv.vlseg8ff.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %vl)
@@ -5542,17 +5269,16 @@ define <vscale x 2 x half> @test_vlseg8ff_mask_nxv2f16(<vscale x 2 x half> %val,
 ; CHECK-LABEL: test_vlseg8ff_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vlseg8e16ff.v v7, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>, i64} @llvm.riscv.vlseg8ff.mask.nxv2f16(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5572,7 +5298,6 @@ define <vscale x 4 x float> @test_vlseg2ff_nxv4f32(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg2e32ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>, i64} @llvm.riscv.vlseg2ff.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i64 %vl)
@@ -5590,7 +5315,6 @@ define <vscale x 4 x float> @test_vlseg2ff_mask_nxv4f32(<vscale x 4 x float> %va
 ; CHECK-NEXT:    vlseg2e32ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>, i64} @llvm.riscv.vlseg2ff.mask.nxv4f32(<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -5610,7 +5334,6 @@ define <vscale x 4 x float> @test_vlseg3ff_nxv4f32(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg3e32ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>, i64} @llvm.riscv.vlseg3ff.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i64 %vl)
@@ -5624,12 +5347,11 @@ define <vscale x 4 x float> @test_vlseg3ff_mask_nxv4f32(<vscale x 4 x float> %va
 ; CHECK-LABEL: test_vlseg3ff_mask_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vlseg3e32ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>, i64} @llvm.riscv.vlseg3ff.mask.nxv4f32(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -5649,7 +5371,6 @@ define <vscale x 4 x float> @test_vlseg4ff_nxv4f32(ptr %base, i64 %vl, ptr %outv
 ; CHECK-NEXT:    vlseg4e32ff.v v6, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>, i64} @llvm.riscv.vlseg4ff.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i64 %vl)
@@ -5663,13 +5384,12 @@ define <vscale x 4 x float> @test_vlseg4ff_mask_nxv4f32(<vscale x 4 x float> %va
 ; CHECK-LABEL: test_vlseg4ff_mask_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vlseg4e32ff.v v6, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a2)
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>, i64} @llvm.riscv.vlseg4ff.mask.nxv4f32(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
diff --git a/llvm/test/CodeGen/RISCV/rvv/vlsseg-rv32.ll b/llvm/test/CodeGen/RISCV/rvv/vlsseg-rv32.ll
index 775f07e4a205a..2773c8c319eb9 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vlsseg-rv32.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vlsseg-rv32.ll
@@ -10,7 +10,6 @@ define <vscale x 16 x i16> @test_vlsseg2_nxv16i16(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m4, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v4, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vlsseg2.nxv16i16(<vscale x 16 x i16> undef, <vscale x 16 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -25,7 +24,6 @@ define <vscale x 16 x i16> @test_vlsseg2_mask_nxv16i16(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vlsseg2e16.v v4, (a0), a1
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlsseg2e16.v v4, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vlsseg2.nxv16i16(<vscale x 16 x i16> undef, <vscale x 16 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -43,7 +41,6 @@ define <vscale x 1 x i8> @test_vlsseg2_nxv1i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg2.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -58,7 +55,6 @@ define <vscale x 1 x i8> @test_vlsseg2_mask_nxv1i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg2.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -76,7 +72,6 @@ define <vscale x 1 x i8> @test_vlsseg3_nxv1i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlsseg3e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg3.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -92,7 +87,6 @@ define <vscale x 1 x i8> @test_vlsseg3_mask_nxv1i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg3.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -110,7 +104,6 @@ define <vscale x 1 x i8> @test_vlsseg4_nxv1i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlsseg4e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg4.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -127,7 +120,6 @@ define <vscale x 1 x i8> @test_vlsseg4_mask_nxv1i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg4.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -145,7 +137,6 @@ define <vscale x 1 x i8> @test_vlsseg5_nxv1i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlsseg5e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg5.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -163,7 +154,6 @@ define <vscale x 1 x i8> @test_vlsseg5_mask_nxv1i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg5.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -181,7 +171,6 @@ define <vscale x 1 x i8> @test_vlsseg6_nxv1i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlsseg6e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg6.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -200,7 +189,6 @@ define <vscale x 1 x i8> @test_vlsseg6_mask_nxv1i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg6.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -218,7 +206,6 @@ define <vscale x 1 x i8> @test_vlsseg7_nxv1i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlsseg7e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg7.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -238,7 +225,6 @@ define <vscale x 1 x i8> @test_vlsseg7_mask_nxv1i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg7.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -256,7 +242,6 @@ define <vscale x 1 x i8> @test_vlsseg8_nxv1i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlsseg8e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg8.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -277,7 +262,6 @@ define <vscale x 1 x i8> @test_vlsseg8_mask_nxv1i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg8.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -295,7 +279,6 @@ define <vscale x 16 x i8> @test_vlsseg2_nxv16i8(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
 ; CHECK-NEXT:    vlsseg2e8.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlsseg2.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -310,7 +293,6 @@ define <vscale x 16 x i8> @test_vlsseg2_mask_nxv16i8(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vlsseg2e8.v v6, (a0), a1
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlsseg2e8.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlsseg2.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -328,7 +310,6 @@ define <vscale x 16 x i8> @test_vlsseg3_nxv16i8(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
 ; CHECK-NEXT:    vlsseg3e8.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlsseg3.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -344,7 +325,6 @@ define <vscale x 16 x i8> @test_vlsseg3_mask_nxv16i8(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlsseg3e8.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlsseg3.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -362,7 +342,6 @@ define <vscale x 16 x i8> @test_vlsseg4_nxv16i8(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
 ; CHECK-NEXT:    vlsseg4e8.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlsseg4.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -379,7 +358,6 @@ define <vscale x 16 x i8> @test_vlsseg4_mask_nxv16i8(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlsseg4e8.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlsseg4.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -397,7 +375,6 @@ define <vscale x 2 x i32> @test_vlsseg2_nxv2i32(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg2.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -412,7 +389,6 @@ define <vscale x 2 x i32> @test_vlsseg2_mask_nxv2i32(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg2.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -430,7 +406,6 @@ define <vscale x 2 x i32> @test_vlsseg3_nxv2i32(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg3e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg3.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -446,7 +421,6 @@ define <vscale x 2 x i32> @test_vlsseg3_mask_nxv2i32(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg3.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -464,7 +438,6 @@ define <vscale x 2 x i32> @test_vlsseg4_nxv2i32(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg4e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg4.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -481,7 +454,6 @@ define <vscale x 2 x i32> @test_vlsseg4_mask_nxv2i32(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg4.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -499,7 +471,6 @@ define <vscale x 2 x i32> @test_vlsseg5_nxv2i32(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg5e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg5.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -517,7 +488,6 @@ define <vscale x 2 x i32> @test_vlsseg5_mask_nxv2i32(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg5.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -535,7 +505,6 @@ define <vscale x 2 x i32> @test_vlsseg6_nxv2i32(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg6e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg6.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -554,7 +523,6 @@ define <vscale x 2 x i32> @test_vlsseg6_mask_nxv2i32(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg6.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -572,7 +540,6 @@ define <vscale x 2 x i32> @test_vlsseg7_nxv2i32(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg7e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg7.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -592,7 +559,6 @@ define <vscale x 2 x i32> @test_vlsseg7_mask_nxv2i32(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg7.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -610,7 +576,6 @@ define <vscale x 2 x i32> @test_vlsseg8_nxv2i32(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg8e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg8.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -631,7 +596,6 @@ define <vscale x 2 x i32> @test_vlsseg8_mask_nxv2i32(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg8.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -649,7 +613,6 @@ define <vscale x 4 x i16> @test_vlsseg2_nxv4i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg2.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -664,7 +627,6 @@ define <vscale x 4 x i16> @test_vlsseg2_mask_nxv4i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg2.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -682,7 +644,6 @@ define <vscale x 4 x i16> @test_vlsseg3_nxv4i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg3.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -698,7 +659,6 @@ define <vscale x 4 x i16> @test_vlsseg3_mask_nxv4i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg3.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -716,7 +676,6 @@ define <vscale x 4 x i16> @test_vlsseg4_nxv4i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg4.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -733,7 +692,6 @@ define <vscale x 4 x i16> @test_vlsseg4_mask_nxv4i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg4.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -751,7 +709,6 @@ define <vscale x 4 x i16> @test_vlsseg5_nxv4i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg5.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -769,7 +726,6 @@ define <vscale x 4 x i16> @test_vlsseg5_mask_nxv4i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg5.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -787,7 +743,6 @@ define <vscale x 4 x i16> @test_vlsseg6_nxv4i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg6.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -806,7 +761,6 @@ define <vscale x 4 x i16> @test_vlsseg6_mask_nxv4i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg6.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -824,7 +778,6 @@ define <vscale x 4 x i16> @test_vlsseg7_nxv4i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg7.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -844,7 +797,6 @@ define <vscale x 4 x i16> @test_vlsseg7_mask_nxv4i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg7.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -862,7 +814,6 @@ define <vscale x 4 x i16> @test_vlsseg8_nxv4i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg8.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -883,7 +834,6 @@ define <vscale x 4 x i16> @test_vlsseg8_mask_nxv4i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg8.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -901,7 +851,6 @@ define <vscale x 1 x i32> @test_vlsseg2_nxv1i32(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg2.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -916,7 +865,6 @@ define <vscale x 1 x i32> @test_vlsseg2_mask_nxv1i32(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg2.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -934,7 +882,6 @@ define <vscale x 1 x i32> @test_vlsseg3_nxv1i32(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg3e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg3.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -950,7 +897,6 @@ define <vscale x 1 x i32> @test_vlsseg3_mask_nxv1i32(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg3.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -968,7 +914,6 @@ define <vscale x 1 x i32> @test_vlsseg4_nxv1i32(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg4e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg4.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -985,7 +930,6 @@ define <vscale x 1 x i32> @test_vlsseg4_mask_nxv1i32(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg4.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1003,7 +947,6 @@ define <vscale x 1 x i32> @test_vlsseg5_nxv1i32(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg5e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg5.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1021,7 +964,6 @@ define <vscale x 1 x i32> @test_vlsseg5_mask_nxv1i32(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg5.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1039,7 +981,6 @@ define <vscale x 1 x i32> @test_vlsseg6_nxv1i32(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg6e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg6.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1058,7 +999,6 @@ define <vscale x 1 x i32> @test_vlsseg6_mask_nxv1i32(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg6.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1076,7 +1016,6 @@ define <vscale x 1 x i32> @test_vlsseg7_nxv1i32(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg7e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg7.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1096,7 +1035,6 @@ define <vscale x 1 x i32> @test_vlsseg7_mask_nxv1i32(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg7.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1114,7 +1052,6 @@ define <vscale x 1 x i32> @test_vlsseg8_nxv1i32(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg8e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg8.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1135,7 +1072,6 @@ define <vscale x 1 x i32> @test_vlsseg8_mask_nxv1i32(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg8.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1153,7 +1089,6 @@ define <vscale x 8 x i16> @test_vlsseg2_nxv8i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlsseg2.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1168,7 +1103,6 @@ define <vscale x 8 x i16> @test_vlsseg2_mask_nxv8i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vlsseg2e16.v v6, (a0), a1
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlsseg2e16.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlsseg2.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1186,7 +1120,6 @@ define <vscale x 8 x i16> @test_vlsseg3_nxv8i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vlsseg3e16.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlsseg3.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1202,7 +1135,6 @@ define <vscale x 8 x i16> @test_vlsseg3_mask_nxv8i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlsseg3e16.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlsseg3.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1220,7 +1152,6 @@ define <vscale x 8 x i16> @test_vlsseg4_nxv8i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vlsseg4e16.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlsseg4.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1237,7 +1168,6 @@ define <vscale x 8 x i16> @test_vlsseg4_mask_nxv8i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlsseg4e16.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlsseg4.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1255,7 +1185,6 @@ define <vscale x 8 x i8> @test_vlsseg2_nxv8i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg2.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1270,7 +1199,6 @@ define <vscale x 8 x i8> @test_vlsseg2_mask_nxv8i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg2.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1288,7 +1216,6 @@ define <vscale x 8 x i8> @test_vlsseg3_nxv8i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
 ; CHECK-NEXT:    vlsseg3e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg3.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1304,7 +1231,6 @@ define <vscale x 8 x i8> @test_vlsseg3_mask_nxv8i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg3.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1322,7 +1248,6 @@ define <vscale x 8 x i8> @test_vlsseg4_nxv8i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
 ; CHECK-NEXT:    vlsseg4e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg4.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1339,7 +1264,6 @@ define <vscale x 8 x i8> @test_vlsseg4_mask_nxv8i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg4.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1357,7 +1281,6 @@ define <vscale x 8 x i8> @test_vlsseg5_nxv8i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
 ; CHECK-NEXT:    vlsseg5e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg5.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1375,7 +1298,6 @@ define <vscale x 8 x i8> @test_vlsseg5_mask_nxv8i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg5.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1393,7 +1315,6 @@ define <vscale x 8 x i8> @test_vlsseg6_nxv8i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
 ; CHECK-NEXT:    vlsseg6e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg6.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1412,7 +1333,6 @@ define <vscale x 8 x i8> @test_vlsseg6_mask_nxv8i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg6.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1430,7 +1350,6 @@ define <vscale x 8 x i8> @test_vlsseg7_nxv8i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
 ; CHECK-NEXT:    vlsseg7e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg7.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1450,7 +1369,6 @@ define <vscale x 8 x i8> @test_vlsseg7_mask_nxv8i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg7.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1468,7 +1386,6 @@ define <vscale x 8 x i8> @test_vlsseg8_nxv8i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
 ; CHECK-NEXT:    vlsseg8e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg8.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1489,7 +1406,6 @@ define <vscale x 8 x i8> @test_vlsseg8_mask_nxv8i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg8.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1507,7 +1423,6 @@ define <vscale x 8 x i32> @test_vlsseg2_nxv8i32(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m4, ta, ma
 ; CHECK-NEXT:    vlsseg2e32.v v4, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vlsseg2.nxv8i32(<vscale x 8 x i32> undef, <vscale x 8 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1522,7 +1437,6 @@ define <vscale x 8 x i32> @test_vlsseg2_mask_nxv8i32(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vlsseg2e32.v v4, (a0), a1
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlsseg2e32.v v4, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vlsseg2.nxv8i32(<vscale x 8 x i32> undef, <vscale x 8 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1540,7 +1454,6 @@ define <vscale x 4 x i8> @test_vlsseg2_nxv4i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg2.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1555,7 +1468,6 @@ define <vscale x 4 x i8> @test_vlsseg2_mask_nxv4i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg2.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1573,7 +1485,6 @@ define <vscale x 4 x i8> @test_vlsseg3_nxv4i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg3e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg3.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1589,7 +1500,6 @@ define <vscale x 4 x i8> @test_vlsseg3_mask_nxv4i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg3.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1607,7 +1517,6 @@ define <vscale x 4 x i8> @test_vlsseg4_nxv4i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg4e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg4.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1624,7 +1533,6 @@ define <vscale x 4 x i8> @test_vlsseg4_mask_nxv4i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg4.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1642,7 +1550,6 @@ define <vscale x 4 x i8> @test_vlsseg5_nxv4i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg5e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg5.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1660,7 +1567,6 @@ define <vscale x 4 x i8> @test_vlsseg5_mask_nxv4i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg5.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1678,7 +1584,6 @@ define <vscale x 4 x i8> @test_vlsseg6_nxv4i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg6e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg6.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1697,7 +1602,6 @@ define <vscale x 4 x i8> @test_vlsseg6_mask_nxv4i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg6.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1715,7 +1619,6 @@ define <vscale x 4 x i8> @test_vlsseg7_nxv4i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg7e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg7.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1735,7 +1638,6 @@ define <vscale x 4 x i8> @test_vlsseg7_mask_nxv4i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg7.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1753,7 +1655,6 @@ define <vscale x 4 x i8> @test_vlsseg8_nxv4i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg8e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg8.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1774,7 +1675,6 @@ define <vscale x 4 x i8> @test_vlsseg8_mask_nxv4i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg8.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1792,7 +1692,6 @@ define <vscale x 1 x i16> @test_vlsseg2_nxv1i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg2.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1807,7 +1706,6 @@ define <vscale x 1 x i16> @test_vlsseg2_mask_nxv1i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg2.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1825,7 +1723,6 @@ define <vscale x 1 x i16> @test_vlsseg3_nxv1i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg3.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1841,7 +1738,6 @@ define <vscale x 1 x i16> @test_vlsseg3_mask_nxv1i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg3.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1859,7 +1755,6 @@ define <vscale x 1 x i16> @test_vlsseg4_nxv1i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg4.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1876,7 +1771,6 @@ define <vscale x 1 x i16> @test_vlsseg4_mask_nxv1i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg4.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1894,7 +1788,6 @@ define <vscale x 1 x i16> @test_vlsseg5_nxv1i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg5.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1912,7 +1805,6 @@ define <vscale x 1 x i16> @test_vlsseg5_mask_nxv1i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg5.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1930,7 +1822,6 @@ define <vscale x 1 x i16> @test_vlsseg6_nxv1i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg6.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1949,7 +1840,6 @@ define <vscale x 1 x i16> @test_vlsseg6_mask_nxv1i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg6.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1967,7 +1857,6 @@ define <vscale x 1 x i16> @test_vlsseg7_nxv1i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg7.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -1987,7 +1876,6 @@ define <vscale x 1 x i16> @test_vlsseg7_mask_nxv1i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg7.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2005,7 +1893,6 @@ define <vscale x 1 x i16> @test_vlsseg8_nxv1i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg8.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2026,7 +1913,6 @@ define <vscale x 1 x i16> @test_vlsseg8_mask_nxv1i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg8.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2044,7 +1930,6 @@ define <vscale x 32 x i8> @test_vlsseg2_nxv32i8(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m4, ta, ma
 ; CHECK-NEXT:    vlsseg2e8.v v4, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>} @llvm.riscv.vlsseg2.nxv32i8(<vscale x 32 x i8> undef, <vscale x 32 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2059,7 +1944,6 @@ define <vscale x 32 x i8> @test_vlsseg2_mask_nxv32i8(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vlsseg2e8.v v4, (a0), a1
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlsseg2e8.v v4, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>} @llvm.riscv.vlsseg2.nxv32i8(<vscale x 32 x i8> undef, <vscale x 32 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2077,7 +1961,6 @@ define <vscale x 2 x i8> @test_vlsseg2_nxv2i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg2.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2092,7 +1975,6 @@ define <vscale x 2 x i8> @test_vlsseg2_mask_nxv2i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg2.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2110,7 +1992,6 @@ define <vscale x 2 x i8> @test_vlsseg3_nxv2i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg3e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg3.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2126,7 +2007,6 @@ define <vscale x 2 x i8> @test_vlsseg3_mask_nxv2i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg3.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2144,7 +2024,6 @@ define <vscale x 2 x i8> @test_vlsseg4_nxv2i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg4e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg4.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2161,7 +2040,6 @@ define <vscale x 2 x i8> @test_vlsseg4_mask_nxv2i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg4.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2179,7 +2057,6 @@ define <vscale x 2 x i8> @test_vlsseg5_nxv2i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg5e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg5.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2197,7 +2074,6 @@ define <vscale x 2 x i8> @test_vlsseg5_mask_nxv2i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg5.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2215,7 +2091,6 @@ define <vscale x 2 x i8> @test_vlsseg6_nxv2i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg6e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg6.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2234,7 +2109,6 @@ define <vscale x 2 x i8> @test_vlsseg6_mask_nxv2i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg6.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2252,7 +2126,6 @@ define <vscale x 2 x i8> @test_vlsseg7_nxv2i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg7e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg7.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2272,7 +2145,6 @@ define <vscale x 2 x i8> @test_vlsseg7_mask_nxv2i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg7.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2290,7 +2162,6 @@ define <vscale x 2 x i8> @test_vlsseg8_nxv2i8(ptr %base, i32 %offset, i32 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg8e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg8.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2311,7 +2182,6 @@ define <vscale x 2 x i8> @test_vlsseg8_mask_nxv2i8(ptr %base, i32 %offset, i32 %
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg8.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2329,7 +2199,6 @@ define <vscale x 2 x i16> @test_vlsseg2_nxv2i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg2.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2344,7 +2213,6 @@ define <vscale x 2 x i16> @test_vlsseg2_mask_nxv2i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg2.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2362,7 +2230,6 @@ define <vscale x 2 x i16> @test_vlsseg3_nxv2i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg3.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2378,7 +2245,6 @@ define <vscale x 2 x i16> @test_vlsseg3_mask_nxv2i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg3.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2396,7 +2262,6 @@ define <vscale x 2 x i16> @test_vlsseg4_nxv2i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg4.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2413,7 +2278,6 @@ define <vscale x 2 x i16> @test_vlsseg4_mask_nxv2i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg4.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2431,7 +2295,6 @@ define <vscale x 2 x i16> @test_vlsseg5_nxv2i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg5.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2449,7 +2312,6 @@ define <vscale x 2 x i16> @test_vlsseg5_mask_nxv2i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg5.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2467,7 +2329,6 @@ define <vscale x 2 x i16> @test_vlsseg6_nxv2i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg6.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2486,7 +2347,6 @@ define <vscale x 2 x i16> @test_vlsseg6_mask_nxv2i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg6.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2504,7 +2364,6 @@ define <vscale x 2 x i16> @test_vlsseg7_nxv2i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg7.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2524,7 +2383,6 @@ define <vscale x 2 x i16> @test_vlsseg7_mask_nxv2i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg7.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2542,7 +2400,6 @@ define <vscale x 2 x i16> @test_vlsseg8_nxv2i16(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg8.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2563,7 +2420,6 @@ define <vscale x 2 x i16> @test_vlsseg8_mask_nxv2i16(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg8.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2581,7 +2437,6 @@ define <vscale x 4 x i32> @test_vlsseg2_nxv4i32(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vlsseg2e32.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlsseg2.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2596,7 +2451,6 @@ define <vscale x 4 x i32> @test_vlsseg2_mask_nxv4i32(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vlsseg2e32.v v6, (a0), a1
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlsseg2e32.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlsseg2.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2614,7 +2468,6 @@ define <vscale x 4 x i32> @test_vlsseg3_nxv4i32(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vlsseg3e32.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlsseg3.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2630,7 +2483,6 @@ define <vscale x 4 x i32> @test_vlsseg3_mask_nxv4i32(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlsseg3e32.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlsseg3.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2648,7 +2500,6 @@ define <vscale x 4 x i32> @test_vlsseg4_nxv4i32(ptr %base, i32 %offset, i32 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vlsseg4e32.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlsseg4.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2665,7 +2516,6 @@ define <vscale x 4 x i32> @test_vlsseg4_mask_nxv4i32(ptr %base, i32 %offset, i32
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlsseg4e32.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlsseg4.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2683,7 +2533,6 @@ define <vscale x 16 x half> @test_vlsseg2_nxv16f16(ptr %base, i32 %offset, i32 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m4, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v4, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vlsseg2.nxv16f16(<vscale x 16 x half> undef, <vscale x 16 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2698,7 +2547,6 @@ define <vscale x 16 x half> @test_vlsseg2_mask_nxv16f16(ptr %base, i32 %offset,
 ; CHECK-NEXT:    vlsseg2e16.v v4, (a0), a1
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlsseg2e16.v v4, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vlsseg2.nxv16f16(<vscale x 16 x half> undef, <vscale x 16 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2716,7 +2564,6 @@ define <vscale x 4 x double> @test_vlsseg2_nxv4f64(ptr %base, i32 %offset, i32 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m4, ta, ma
 ; CHECK-NEXT:    vlsseg2e64.v v4, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vlsseg2.nxv4f64(<vscale x 4 x double> undef, <vscale x 4 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2731,7 +2578,6 @@ define <vscale x 4 x double> @test_vlsseg2_mask_nxv4f64(ptr %base, i32 %offset,
 ; CHECK-NEXT:    vlsseg2e64.v v4, (a0), a1
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlsseg2e64.v v4, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vlsseg2.nxv4f64(<vscale x 4 x double> undef, <vscale x 4 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2749,7 +2595,6 @@ define <vscale x 1 x double> @test_vlsseg2_nxv1f64(ptr %base, i32 %offset, i32 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg2e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg2.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2764,7 +2609,6 @@ define <vscale x 1 x double> @test_vlsseg2_mask_nxv1f64(ptr %base, i32 %offset,
 ; CHECK-NEXT:    vlsseg2e64.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg2.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2782,7 +2626,6 @@ define <vscale x 1 x double> @test_vlsseg3_nxv1f64(ptr %base, i32 %offset, i32 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg3e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg3.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2798,7 +2641,6 @@ define <vscale x 1 x double> @test_vlsseg3_mask_nxv1f64(ptr %base, i32 %offset,
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg3.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2816,7 +2658,6 @@ define <vscale x 1 x double> @test_vlsseg4_nxv1f64(ptr %base, i32 %offset, i32 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg4e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg4.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2833,7 +2674,6 @@ define <vscale x 1 x double> @test_vlsseg4_mask_nxv1f64(ptr %base, i32 %offset,
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg4.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2851,7 +2691,6 @@ define <vscale x 1 x double> @test_vlsseg5_nxv1f64(ptr %base, i32 %offset, i32 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg5e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg5.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2869,7 +2708,6 @@ define <vscale x 1 x double> @test_vlsseg5_mask_nxv1f64(ptr %base, i32 %offset,
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg5.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2887,7 +2725,6 @@ define <vscale x 1 x double> @test_vlsseg6_nxv1f64(ptr %base, i32 %offset, i32 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg6e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg6.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2906,7 +2743,6 @@ define <vscale x 1 x double> @test_vlsseg6_mask_nxv1f64(ptr %base, i32 %offset,
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg6.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2924,7 +2760,6 @@ define <vscale x 1 x double> @test_vlsseg7_nxv1f64(ptr %base, i32 %offset, i32 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg7e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg7.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2944,7 +2779,6 @@ define <vscale x 1 x double> @test_vlsseg7_mask_nxv1f64(ptr %base, i32 %offset,
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg7.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2962,7 +2796,6 @@ define <vscale x 1 x double> @test_vlsseg8_nxv1f64(ptr %base, i32 %offset, i32 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg8e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg8.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -2983,7 +2816,6 @@ define <vscale x 1 x double> @test_vlsseg8_mask_nxv1f64(ptr %base, i32 %offset,
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg8.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3001,7 +2833,6 @@ define <vscale x 2 x float> @test_vlsseg2_nxv2f32(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg2.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3016,7 +2847,6 @@ define <vscale x 2 x float> @test_vlsseg2_mask_nxv2f32(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg2.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3034,7 +2864,6 @@ define <vscale x 2 x float> @test_vlsseg3_nxv2f32(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg3e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg3.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3050,7 +2879,6 @@ define <vscale x 2 x float> @test_vlsseg3_mask_nxv2f32(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg3.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3068,7 +2896,6 @@ define <vscale x 2 x float> @test_vlsseg4_nxv2f32(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg4e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg4.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3085,7 +2912,6 @@ define <vscale x 2 x float> @test_vlsseg4_mask_nxv2f32(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg4.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3103,7 +2929,6 @@ define <vscale x 2 x float> @test_vlsseg5_nxv2f32(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg5e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg5.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3121,7 +2946,6 @@ define <vscale x 2 x float> @test_vlsseg5_mask_nxv2f32(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg5.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3139,7 +2963,6 @@ define <vscale x 2 x float> @test_vlsseg6_nxv2f32(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg6e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg6.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3158,7 +2981,6 @@ define <vscale x 2 x float> @test_vlsseg6_mask_nxv2f32(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg6.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3176,7 +2998,6 @@ define <vscale x 2 x float> @test_vlsseg7_nxv2f32(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg7e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg7.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3196,7 +3017,6 @@ define <vscale x 2 x float> @test_vlsseg7_mask_nxv2f32(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg7.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3214,7 +3034,6 @@ define <vscale x 2 x float> @test_vlsseg8_nxv2f32(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg8e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg8.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3235,7 +3054,6 @@ define <vscale x 2 x float> @test_vlsseg8_mask_nxv2f32(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg8.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3253,7 +3071,6 @@ define <vscale x 1 x half> @test_vlsseg2_nxv1f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg2.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3268,7 +3085,6 @@ define <vscale x 1 x half> @test_vlsseg2_mask_nxv1f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg2.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3286,7 +3102,6 @@ define <vscale x 1 x half> @test_vlsseg3_nxv1f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg3.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3302,7 +3117,6 @@ define <vscale x 1 x half> @test_vlsseg3_mask_nxv1f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg3.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3320,7 +3134,6 @@ define <vscale x 1 x half> @test_vlsseg4_nxv1f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg4.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3337,7 +3150,6 @@ define <vscale x 1 x half> @test_vlsseg4_mask_nxv1f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg4.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3355,7 +3167,6 @@ define <vscale x 1 x half> @test_vlsseg5_nxv1f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg5.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3373,7 +3184,6 @@ define <vscale x 1 x half> @test_vlsseg5_mask_nxv1f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg5.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3391,7 +3201,6 @@ define <vscale x 1 x half> @test_vlsseg6_nxv1f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg6.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3410,7 +3219,6 @@ define <vscale x 1 x half> @test_vlsseg6_mask_nxv1f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg6.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3428,7 +3236,6 @@ define <vscale x 1 x half> @test_vlsseg7_nxv1f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg7.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3448,7 +3255,6 @@ define <vscale x 1 x half> @test_vlsseg7_mask_nxv1f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg7.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3466,7 +3272,6 @@ define <vscale x 1 x half> @test_vlsseg8_nxv1f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg8.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3487,7 +3292,6 @@ define <vscale x 1 x half> @test_vlsseg8_mask_nxv1f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg8.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3505,7 +3309,6 @@ define <vscale x 1 x float> @test_vlsseg2_nxv1f32(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg2.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3520,7 +3323,6 @@ define <vscale x 1 x float> @test_vlsseg2_mask_nxv1f32(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg2.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3538,7 +3340,6 @@ define <vscale x 1 x float> @test_vlsseg3_nxv1f32(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg3e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg3.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3554,7 +3355,6 @@ define <vscale x 1 x float> @test_vlsseg3_mask_nxv1f32(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg3.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3572,7 +3372,6 @@ define <vscale x 1 x float> @test_vlsseg4_nxv1f32(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg4e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg4.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3589,7 +3388,6 @@ define <vscale x 1 x float> @test_vlsseg4_mask_nxv1f32(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg4.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3607,7 +3405,6 @@ define <vscale x 1 x float> @test_vlsseg5_nxv1f32(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg5e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg5.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3625,7 +3422,6 @@ define <vscale x 1 x float> @test_vlsseg5_mask_nxv1f32(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg5.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3643,7 +3439,6 @@ define <vscale x 1 x float> @test_vlsseg6_nxv1f32(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg6e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg6.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3662,7 +3457,6 @@ define <vscale x 1 x float> @test_vlsseg6_mask_nxv1f32(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg6.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3680,7 +3474,6 @@ define <vscale x 1 x float> @test_vlsseg7_nxv1f32(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg7e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg7.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3700,7 +3493,6 @@ define <vscale x 1 x float> @test_vlsseg7_mask_nxv1f32(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg7.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3718,7 +3510,6 @@ define <vscale x 1 x float> @test_vlsseg8_nxv1f32(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg8e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg8.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3739,7 +3530,6 @@ define <vscale x 1 x float> @test_vlsseg8_mask_nxv1f32(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg8.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3757,7 +3547,6 @@ define <vscale x 8 x half> @test_vlsseg2_nxv8f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlsseg2.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3772,7 +3561,6 @@ define <vscale x 8 x half> @test_vlsseg2_mask_nxv8f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vlsseg2e16.v v6, (a0), a1
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlsseg2e16.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlsseg2.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3790,7 +3578,6 @@ define <vscale x 8 x half> @test_vlsseg3_nxv8f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vlsseg3e16.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlsseg3.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3806,7 +3593,6 @@ define <vscale x 8 x half> @test_vlsseg3_mask_nxv8f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlsseg3e16.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlsseg3.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3824,7 +3610,6 @@ define <vscale x 8 x half> @test_vlsseg4_nxv8f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vlsseg4e16.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlsseg4.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3841,7 +3626,6 @@ define <vscale x 8 x half> @test_vlsseg4_mask_nxv8f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlsseg4e16.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlsseg4.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3859,7 +3643,6 @@ define <vscale x 8 x float> @test_vlsseg2_nxv8f32(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m4, ta, ma
 ; CHECK-NEXT:    vlsseg2e32.v v4, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vlsseg2.nxv8f32(<vscale x 8 x float> undef, <vscale x 8 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3874,7 +3657,6 @@ define <vscale x 8 x float> @test_vlsseg2_mask_nxv8f32(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vlsseg2e32.v v4, (a0), a1
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlsseg2e32.v v4, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vlsseg2.nxv8f32(<vscale x 8 x float> undef, <vscale x 8 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3892,7 +3674,6 @@ define <vscale x 2 x double> @test_vlsseg2_nxv2f64(ptr %base, i32 %offset, i32 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
 ; CHECK-NEXT:    vlsseg2e64.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlsseg2.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3907,7 +3688,6 @@ define <vscale x 2 x double> @test_vlsseg2_mask_nxv2f64(ptr %base, i32 %offset,
 ; CHECK-NEXT:    vlsseg2e64.v v6, (a0), a1
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlsseg2e64.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlsseg2.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3925,7 +3705,6 @@ define <vscale x 2 x double> @test_vlsseg3_nxv2f64(ptr %base, i32 %offset, i32 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
 ; CHECK-NEXT:    vlsseg3e64.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlsseg3.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3941,7 +3720,6 @@ define <vscale x 2 x double> @test_vlsseg3_mask_nxv2f64(ptr %base, i32 %offset,
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlsseg3e64.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlsseg3.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3959,7 +3737,6 @@ define <vscale x 2 x double> @test_vlsseg4_nxv2f64(ptr %base, i32 %offset, i32 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
 ; CHECK-NEXT:    vlsseg4e64.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlsseg4.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3976,7 +3753,6 @@ define <vscale x 2 x double> @test_vlsseg4_mask_nxv2f64(ptr %base, i32 %offset,
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlsseg4e64.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlsseg4.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i32 %offset, i32 %vl)
@@ -3994,7 +3770,6 @@ define <vscale x 4 x half> @test_vlsseg2_nxv4f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg2.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4009,7 +3784,6 @@ define <vscale x 4 x half> @test_vlsseg2_mask_nxv4f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg2.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4027,7 +3801,6 @@ define <vscale x 4 x half> @test_vlsseg3_nxv4f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg3.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4043,7 +3816,6 @@ define <vscale x 4 x half> @test_vlsseg3_mask_nxv4f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg3.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4061,7 +3833,6 @@ define <vscale x 4 x half> @test_vlsseg4_nxv4f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg4.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4078,7 +3849,6 @@ define <vscale x 4 x half> @test_vlsseg4_mask_nxv4f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg4.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4096,7 +3866,6 @@ define <vscale x 4 x half> @test_vlsseg5_nxv4f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg5.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4114,7 +3883,6 @@ define <vscale x 4 x half> @test_vlsseg5_mask_nxv4f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg5.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4132,7 +3900,6 @@ define <vscale x 4 x half> @test_vlsseg6_nxv4f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg6.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4151,7 +3918,6 @@ define <vscale x 4 x half> @test_vlsseg6_mask_nxv4f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg6.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4169,7 +3935,6 @@ define <vscale x 4 x half> @test_vlsseg7_nxv4f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg7.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4189,7 +3954,6 @@ define <vscale x 4 x half> @test_vlsseg7_mask_nxv4f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg7.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4207,7 +3971,6 @@ define <vscale x 4 x half> @test_vlsseg8_nxv4f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg8.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4228,7 +3991,6 @@ define <vscale x 4 x half> @test_vlsseg8_mask_nxv4f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg8.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4246,7 +4008,6 @@ define <vscale x 2 x half> @test_vlsseg2_nxv2f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg2.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4261,7 +4022,6 @@ define <vscale x 2 x half> @test_vlsseg2_mask_nxv2f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg2.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4279,7 +4039,6 @@ define <vscale x 2 x half> @test_vlsseg3_nxv2f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg3.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4295,7 +4054,6 @@ define <vscale x 2 x half> @test_vlsseg3_mask_nxv2f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg3.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4313,7 +4071,6 @@ define <vscale x 2 x half> @test_vlsseg4_nxv2f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg4.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4330,7 +4087,6 @@ define <vscale x 2 x half> @test_vlsseg4_mask_nxv2f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg4.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4348,7 +4104,6 @@ define <vscale x 2 x half> @test_vlsseg5_nxv2f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg5.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4366,7 +4121,6 @@ define <vscale x 2 x half> @test_vlsseg5_mask_nxv2f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg5.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4384,7 +4138,6 @@ define <vscale x 2 x half> @test_vlsseg6_nxv2f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg6.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4403,7 +4156,6 @@ define <vscale x 2 x half> @test_vlsseg6_mask_nxv2f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg6.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4421,7 +4173,6 @@ define <vscale x 2 x half> @test_vlsseg7_nxv2f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg7.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4441,7 +4192,6 @@ define <vscale x 2 x half> @test_vlsseg7_mask_nxv2f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg7.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4459,7 +4209,6 @@ define <vscale x 2 x half> @test_vlsseg8_nxv2f16(ptr %base, i32 %offset, i32 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg8.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4480,7 +4229,6 @@ define <vscale x 2 x half> @test_vlsseg8_mask_nxv2f16(ptr %base, i32 %offset, i3
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg8.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4498,7 +4246,6 @@ define <vscale x 4 x float> @test_vlsseg2_nxv4f32(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vlsseg2e32.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlsseg2.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4513,7 +4260,6 @@ define <vscale x 4 x float> @test_vlsseg2_mask_nxv4f32(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vlsseg2e32.v v6, (a0), a1
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlsseg2e32.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlsseg2.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4531,7 +4277,6 @@ define <vscale x 4 x float> @test_vlsseg3_nxv4f32(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vlsseg3e32.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlsseg3.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4547,7 +4292,6 @@ define <vscale x 4 x float> @test_vlsseg3_mask_nxv4f32(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlsseg3e32.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlsseg3.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4565,7 +4309,6 @@ define <vscale x 4 x float> @test_vlsseg4_nxv4f32(ptr %base, i32 %offset, i32 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vlsseg4e32.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlsseg4.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i32 %offset, i32 %vl)
@@ -4582,7 +4325,6 @@ define <vscale x 4 x float> @test_vlsseg4_mask_nxv4f32(ptr %base, i32 %offset, i
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlsseg4e32.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlsseg4.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i32 %offset, i32 %vl)
diff --git a/llvm/test/CodeGen/RISCV/rvv/vlsseg-rv64.ll b/llvm/test/CodeGen/RISCV/rvv/vlsseg-rv64.ll
index e621cbc802864..7d78be45e165e 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vlsseg-rv64.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vlsseg-rv64.ll
@@ -10,7 +10,6 @@ define <vscale x 16 x i16> @test_vlsseg2_nxv16i16(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m4, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v4, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vlsseg2.nxv16i16(<vscale x 16 x i16> undef, <vscale x 16 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -25,7 +24,6 @@ define <vscale x 16 x i16> @test_vlsseg2_mask_nxv16i16(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vlsseg2e16.v v4, (a0), a1
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlsseg2e16.v v4, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vlsseg2.nxv16i16(<vscale x 16 x i16> undef, <vscale x 16 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -43,7 +41,6 @@ define <vscale x 4 x i32> @test_vlsseg2_nxv4i32(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vlsseg2e32.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlsseg2.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -58,7 +55,6 @@ define <vscale x 4 x i32> @test_vlsseg2_mask_nxv4i32(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vlsseg2e32.v v6, (a0), a1
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlsseg2e32.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlsseg2.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -76,7 +72,6 @@ define <vscale x 4 x i32> @test_vlsseg3_nxv4i32(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vlsseg3e32.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlsseg3.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -92,7 +87,6 @@ define <vscale x 4 x i32> @test_vlsseg3_mask_nxv4i32(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlsseg3e32.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlsseg3.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -110,7 +104,6 @@ define <vscale x 4 x i32> @test_vlsseg4_nxv4i32(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vlsseg4e32.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlsseg4.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -127,7 +120,6 @@ define <vscale x 4 x i32> @test_vlsseg4_mask_nxv4i32(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlsseg4e32.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vlsseg4.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, <vscale x 4 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -145,7 +137,6 @@ define <vscale x 16 x i8> @test_vlsseg2_nxv16i8(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
 ; CHECK-NEXT:    vlsseg2e8.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlsseg2.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -160,7 +151,6 @@ define <vscale x 16 x i8> @test_vlsseg2_mask_nxv16i8(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vlsseg2e8.v v6, (a0), a1
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlsseg2e8.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlsseg2.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -178,7 +168,6 @@ define <vscale x 16 x i8> @test_vlsseg3_nxv16i8(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
 ; CHECK-NEXT:    vlsseg3e8.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlsseg3.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -194,7 +183,6 @@ define <vscale x 16 x i8> @test_vlsseg3_mask_nxv16i8(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlsseg3e8.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlsseg3.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -212,7 +200,6 @@ define <vscale x 16 x i8> @test_vlsseg4_nxv16i8(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
 ; CHECK-NEXT:    vlsseg4e8.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlsseg4.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -229,7 +216,6 @@ define <vscale x 16 x i8> @test_vlsseg4_mask_nxv16i8(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlsseg4e8.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vlsseg4.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, <vscale x 16 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -247,7 +233,6 @@ define <vscale x 1 x i64> @test_vlsseg2_nxv1i64(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg2e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlsseg2.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -262,7 +247,6 @@ define <vscale x 1 x i64> @test_vlsseg2_mask_nxv1i64(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vlsseg2e64.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlsseg2.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -280,7 +264,6 @@ define <vscale x 1 x i64> @test_vlsseg3_nxv1i64(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg3e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlsseg3.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -296,7 +279,6 @@ define <vscale x 1 x i64> @test_vlsseg3_mask_nxv1i64(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlsseg3.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -314,7 +296,6 @@ define <vscale x 1 x i64> @test_vlsseg4_nxv1i64(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg4e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlsseg4.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -331,7 +312,6 @@ define <vscale x 1 x i64> @test_vlsseg4_mask_nxv1i64(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlsseg4.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -349,7 +329,6 @@ define <vscale x 1 x i64> @test_vlsseg5_nxv1i64(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg5e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlsseg5.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -367,7 +346,6 @@ define <vscale x 1 x i64> @test_vlsseg5_mask_nxv1i64(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlsseg5.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -385,7 +363,6 @@ define <vscale x 1 x i64> @test_vlsseg6_nxv1i64(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg6e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlsseg6.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -404,7 +381,6 @@ define <vscale x 1 x i64> @test_vlsseg6_mask_nxv1i64(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlsseg6.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -422,7 +398,6 @@ define <vscale x 1 x i64> @test_vlsseg7_nxv1i64(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg7e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlsseg7.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -442,7 +417,6 @@ define <vscale x 1 x i64> @test_vlsseg7_mask_nxv1i64(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlsseg7.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -460,7 +434,6 @@ define <vscale x 1 x i64> @test_vlsseg8_nxv1i64(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg8e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlsseg8.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef ,<vscale x 1 x i64> undef ,<vscale x 1 x i64> undef, <vscale x 1 x i64> undef ,<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -481,7 +454,6 @@ define <vscale x 1 x i64> @test_vlsseg8_mask_nxv1i64(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vlsseg8.nxv1i64(<vscale x 1 x i64> undef, <vscale x 1 x i64> undef ,<vscale x 1 x i64> undef ,<vscale x 1 x i64> undef, <vscale x 1 x i64> undef ,<vscale x 1 x i64> undef, <vscale x 1 x i64> undef, <vscale x 1 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -499,7 +471,6 @@ define <vscale x 1 x i32> @test_vlsseg2_nxv1i32(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg2.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -514,7 +485,6 @@ define <vscale x 1 x i32> @test_vlsseg2_mask_nxv1i32(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg2.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -532,7 +502,6 @@ define <vscale x 1 x i32> @test_vlsseg3_nxv1i32(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg3e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg3.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -548,7 +517,6 @@ define <vscale x 1 x i32> @test_vlsseg3_mask_nxv1i32(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg3.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -566,7 +534,6 @@ define <vscale x 1 x i32> @test_vlsseg4_nxv1i32(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg4e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg4.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -583,7 +550,6 @@ define <vscale x 1 x i32> @test_vlsseg4_mask_nxv1i32(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg4.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -601,7 +567,6 @@ define <vscale x 1 x i32> @test_vlsseg5_nxv1i32(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg5e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg5.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -619,7 +584,6 @@ define <vscale x 1 x i32> @test_vlsseg5_mask_nxv1i32(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg5.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -637,7 +601,6 @@ define <vscale x 1 x i32> @test_vlsseg6_nxv1i32(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg6e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg6.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -656,7 +619,6 @@ define <vscale x 1 x i32> @test_vlsseg6_mask_nxv1i32(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg6.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -674,7 +636,6 @@ define <vscale x 1 x i32> @test_vlsseg7_nxv1i32(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg7e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg7.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -694,7 +655,6 @@ define <vscale x 1 x i32> @test_vlsseg7_mask_nxv1i32(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg7.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -712,7 +672,6 @@ define <vscale x 1 x i32> @test_vlsseg8_nxv1i32(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg8e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg8.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -733,7 +692,6 @@ define <vscale x 1 x i32> @test_vlsseg8_mask_nxv1i32(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vlsseg8.nxv1i32(<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef ,<vscale x 1 x i32> undef, <vscale x 1 x i32> undef, <vscale x 1 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -751,7 +709,6 @@ define <vscale x 8 x i16> @test_vlsseg2_nxv8i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlsseg2.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -766,7 +723,6 @@ define <vscale x 8 x i16> @test_vlsseg2_mask_nxv8i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vlsseg2e16.v v6, (a0), a1
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlsseg2e16.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlsseg2.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -784,7 +740,6 @@ define <vscale x 8 x i16> @test_vlsseg3_nxv8i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vlsseg3e16.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlsseg3.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -800,7 +755,6 @@ define <vscale x 8 x i16> @test_vlsseg3_mask_nxv8i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlsseg3e16.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlsseg3.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -818,7 +772,6 @@ define <vscale x 8 x i16> @test_vlsseg4_nxv8i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vlsseg4e16.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlsseg4.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -835,7 +788,6 @@ define <vscale x 8 x i16> @test_vlsseg4_mask_nxv8i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlsseg4e16.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vlsseg4.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, <vscale x 8 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -853,7 +805,6 @@ define <vscale x 4 x i8> @test_vlsseg2_nxv4i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg2.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -868,7 +819,6 @@ define <vscale x 4 x i8> @test_vlsseg2_mask_nxv4i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg2.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -886,7 +836,6 @@ define <vscale x 4 x i8> @test_vlsseg3_nxv4i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg3e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg3.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -902,7 +851,6 @@ define <vscale x 4 x i8> @test_vlsseg3_mask_nxv4i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg3.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -920,7 +868,6 @@ define <vscale x 4 x i8> @test_vlsseg4_nxv4i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg4e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg4.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -937,7 +884,6 @@ define <vscale x 4 x i8> @test_vlsseg4_mask_nxv4i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg4.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -955,7 +901,6 @@ define <vscale x 4 x i8> @test_vlsseg5_nxv4i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg5e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg5.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -973,7 +918,6 @@ define <vscale x 4 x i8> @test_vlsseg5_mask_nxv4i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg5.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -991,7 +935,6 @@ define <vscale x 4 x i8> @test_vlsseg6_nxv4i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg6e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg6.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1010,7 +953,6 @@ define <vscale x 4 x i8> @test_vlsseg6_mask_nxv4i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg6.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1028,7 +970,6 @@ define <vscale x 4 x i8> @test_vlsseg7_nxv4i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg7e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg7.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1048,7 +989,6 @@ define <vscale x 4 x i8> @test_vlsseg7_mask_nxv4i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg7.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1066,7 +1006,6 @@ define <vscale x 4 x i8> @test_vlsseg8_nxv4i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg8e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg8.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1087,7 +1026,6 @@ define <vscale x 4 x i8> @test_vlsseg8_mask_nxv4i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vlsseg8.nxv4i8(<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef ,<vscale x 4 x i8> undef, <vscale x 4 x i8> undef, <vscale x 4 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1105,7 +1043,6 @@ define <vscale x 1 x i16> @test_vlsseg2_nxv1i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg2.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1120,7 +1057,6 @@ define <vscale x 1 x i16> @test_vlsseg2_mask_nxv1i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg2.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1138,7 +1074,6 @@ define <vscale x 1 x i16> @test_vlsseg3_nxv1i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg3.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1154,7 +1089,6 @@ define <vscale x 1 x i16> @test_vlsseg3_mask_nxv1i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg3.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1172,7 +1106,6 @@ define <vscale x 1 x i16> @test_vlsseg4_nxv1i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg4.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1189,7 +1122,6 @@ define <vscale x 1 x i16> @test_vlsseg4_mask_nxv1i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg4.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1207,7 +1139,6 @@ define <vscale x 1 x i16> @test_vlsseg5_nxv1i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg5.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1225,7 +1156,6 @@ define <vscale x 1 x i16> @test_vlsseg5_mask_nxv1i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg5.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1243,7 +1173,6 @@ define <vscale x 1 x i16> @test_vlsseg6_nxv1i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg6.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1262,7 +1191,6 @@ define <vscale x 1 x i16> @test_vlsseg6_mask_nxv1i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg6.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1280,7 +1208,6 @@ define <vscale x 1 x i16> @test_vlsseg7_nxv1i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg7.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1300,7 +1227,6 @@ define <vscale x 1 x i16> @test_vlsseg7_mask_nxv1i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg7.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1318,7 +1244,6 @@ define <vscale x 1 x i16> @test_vlsseg8_nxv1i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg8.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1339,7 +1264,6 @@ define <vscale x 1 x i16> @test_vlsseg8_mask_nxv1i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vlsseg8.nxv1i16(<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef ,<vscale x 1 x i16> undef, <vscale x 1 x i16> undef, <vscale x 1 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1357,7 +1281,6 @@ define <vscale x 2 x i32> @test_vlsseg2_nxv2i32(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg2.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1372,7 +1295,6 @@ define <vscale x 2 x i32> @test_vlsseg2_mask_nxv2i32(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg2.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1390,7 +1312,6 @@ define <vscale x 2 x i32> @test_vlsseg3_nxv2i32(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg3e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg3.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1406,7 +1327,6 @@ define <vscale x 2 x i32> @test_vlsseg3_mask_nxv2i32(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg3.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1424,7 +1344,6 @@ define <vscale x 2 x i32> @test_vlsseg4_nxv2i32(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg4e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg4.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1441,7 +1360,6 @@ define <vscale x 2 x i32> @test_vlsseg4_mask_nxv2i32(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg4.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1459,7 +1377,6 @@ define <vscale x 2 x i32> @test_vlsseg5_nxv2i32(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg5e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg5.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1477,7 +1394,6 @@ define <vscale x 2 x i32> @test_vlsseg5_mask_nxv2i32(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg5.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1495,7 +1411,6 @@ define <vscale x 2 x i32> @test_vlsseg6_nxv2i32(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg6e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg6.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1514,7 +1429,6 @@ define <vscale x 2 x i32> @test_vlsseg6_mask_nxv2i32(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg6.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1532,7 +1446,6 @@ define <vscale x 2 x i32> @test_vlsseg7_nxv2i32(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg7e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg7.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1552,7 +1465,6 @@ define <vscale x 2 x i32> @test_vlsseg7_mask_nxv2i32(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg7.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1570,7 +1482,6 @@ define <vscale x 2 x i32> @test_vlsseg8_nxv2i32(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg8e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg8.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1591,7 +1502,6 @@ define <vscale x 2 x i32> @test_vlsseg8_mask_nxv2i32(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vlsseg8.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef ,<vscale x 2 x i32> undef, <vscale x 2 x i32> undef, <vscale x 2 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1609,7 +1519,6 @@ define <vscale x 8 x i8> @test_vlsseg2_nxv8i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg2.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1624,7 +1533,6 @@ define <vscale x 8 x i8> @test_vlsseg2_mask_nxv8i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg2.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1642,7 +1550,6 @@ define <vscale x 8 x i8> @test_vlsseg3_nxv8i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
 ; CHECK-NEXT:    vlsseg3e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg3.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1658,7 +1565,6 @@ define <vscale x 8 x i8> @test_vlsseg3_mask_nxv8i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg3.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1676,7 +1582,6 @@ define <vscale x 8 x i8> @test_vlsseg4_nxv8i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
 ; CHECK-NEXT:    vlsseg4e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg4.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1693,7 +1598,6 @@ define <vscale x 8 x i8> @test_vlsseg4_mask_nxv8i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg4.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1711,7 +1615,6 @@ define <vscale x 8 x i8> @test_vlsseg5_nxv8i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
 ; CHECK-NEXT:    vlsseg5e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg5.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1729,7 +1632,6 @@ define <vscale x 8 x i8> @test_vlsseg5_mask_nxv8i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg5.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1747,7 +1649,6 @@ define <vscale x 8 x i8> @test_vlsseg6_nxv8i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
 ; CHECK-NEXT:    vlsseg6e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg6.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1766,7 +1667,6 @@ define <vscale x 8 x i8> @test_vlsseg6_mask_nxv8i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg6.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1784,7 +1684,6 @@ define <vscale x 8 x i8> @test_vlsseg7_nxv8i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
 ; CHECK-NEXT:    vlsseg7e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg7.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1804,7 +1703,6 @@ define <vscale x 8 x i8> @test_vlsseg7_mask_nxv8i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg7.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1822,7 +1720,6 @@ define <vscale x 8 x i8> @test_vlsseg8_nxv8i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
 ; CHECK-NEXT:    vlsseg8e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg8.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1843,7 +1740,6 @@ define <vscale x 8 x i8> @test_vlsseg8_mask_nxv8i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vlsseg8.nxv8i8(<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef ,<vscale x 8 x i8> undef, <vscale x 8 x i8> undef, <vscale x 8 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1861,7 +1757,6 @@ define <vscale x 4 x i64> @test_vlsseg2_nxv4i64(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m4, ta, ma
 ; CHECK-NEXT:    vlsseg2e64.v v4, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i64>,<vscale x 4 x i64>} @llvm.riscv.vlsseg2.nxv4i64(<vscale x 4 x i64> undef, <vscale x 4 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1876,7 +1771,6 @@ define <vscale x 4 x i64> @test_vlsseg2_mask_nxv4i64(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vlsseg2e64.v v4, (a0), a1
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlsseg2e64.v v4, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i64>,<vscale x 4 x i64>} @llvm.riscv.vlsseg2.nxv4i64(<vscale x 4 x i64> undef, <vscale x 4 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1894,7 +1788,6 @@ define <vscale x 4 x i16> @test_vlsseg2_nxv4i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg2.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1909,7 +1802,6 @@ define <vscale x 4 x i16> @test_vlsseg2_mask_nxv4i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg2.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1927,7 +1819,6 @@ define <vscale x 4 x i16> @test_vlsseg3_nxv4i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg3.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1943,7 +1834,6 @@ define <vscale x 4 x i16> @test_vlsseg3_mask_nxv4i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg3.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1961,7 +1851,6 @@ define <vscale x 4 x i16> @test_vlsseg4_nxv4i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg4.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1978,7 +1867,6 @@ define <vscale x 4 x i16> @test_vlsseg4_mask_nxv4i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg4.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -1996,7 +1884,6 @@ define <vscale x 4 x i16> @test_vlsseg5_nxv4i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg5.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2014,7 +1901,6 @@ define <vscale x 4 x i16> @test_vlsseg5_mask_nxv4i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg5.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2032,7 +1918,6 @@ define <vscale x 4 x i16> @test_vlsseg6_nxv4i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg6.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2051,7 +1936,6 @@ define <vscale x 4 x i16> @test_vlsseg6_mask_nxv4i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg6.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2069,7 +1953,6 @@ define <vscale x 4 x i16> @test_vlsseg7_nxv4i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg7.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2089,7 +1972,6 @@ define <vscale x 4 x i16> @test_vlsseg7_mask_nxv4i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg7.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2107,7 +1989,6 @@ define <vscale x 4 x i16> @test_vlsseg8_nxv4i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg8.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2128,7 +2009,6 @@ define <vscale x 4 x i16> @test_vlsseg8_mask_nxv4i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vlsseg8.nxv4i16(<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef ,<vscale x 4 x i16> undef, <vscale x 4 x i16> undef, <vscale x 4 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2146,7 +2026,6 @@ define <vscale x 1 x i8> @test_vlsseg2_nxv1i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg2.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2161,7 +2040,6 @@ define <vscale x 1 x i8> @test_vlsseg2_mask_nxv1i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg2.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2179,7 +2057,6 @@ define <vscale x 1 x i8> @test_vlsseg3_nxv1i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlsseg3e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg3.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2195,7 +2072,6 @@ define <vscale x 1 x i8> @test_vlsseg3_mask_nxv1i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg3.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2213,7 +2089,6 @@ define <vscale x 1 x i8> @test_vlsseg4_nxv1i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlsseg4e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg4.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2230,7 +2105,6 @@ define <vscale x 1 x i8> @test_vlsseg4_mask_nxv1i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg4.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2248,7 +2122,6 @@ define <vscale x 1 x i8> @test_vlsseg5_nxv1i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlsseg5e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg5.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2266,7 +2139,6 @@ define <vscale x 1 x i8> @test_vlsseg5_mask_nxv1i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg5.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2284,7 +2156,6 @@ define <vscale x 1 x i8> @test_vlsseg6_nxv1i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlsseg6e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg6.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2303,7 +2174,6 @@ define <vscale x 1 x i8> @test_vlsseg6_mask_nxv1i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg6.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2321,7 +2191,6 @@ define <vscale x 1 x i8> @test_vlsseg7_nxv1i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlsseg7e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg7.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2341,7 +2210,6 @@ define <vscale x 1 x i8> @test_vlsseg7_mask_nxv1i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg7.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2359,7 +2227,6 @@ define <vscale x 1 x i8> @test_vlsseg8_nxv1i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
 ; CHECK-NEXT:    vlsseg8e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg8.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2380,7 +2247,6 @@ define <vscale x 1 x i8> @test_vlsseg8_mask_nxv1i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vlsseg8.nxv1i8(<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef ,<vscale x 1 x i8> undef, <vscale x 1 x i8> undef, <vscale x 1 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2398,7 +2264,6 @@ define <vscale x 2 x i8> @test_vlsseg2_nxv2i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg2.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2413,7 +2278,6 @@ define <vscale x 2 x i8> @test_vlsseg2_mask_nxv2i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg2.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2431,7 +2295,6 @@ define <vscale x 2 x i8> @test_vlsseg3_nxv2i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg3e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg3.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2447,7 +2310,6 @@ define <vscale x 2 x i8> @test_vlsseg3_mask_nxv2i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg3.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2465,7 +2327,6 @@ define <vscale x 2 x i8> @test_vlsseg4_nxv2i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg4e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg4.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2482,7 +2343,6 @@ define <vscale x 2 x i8> @test_vlsseg4_mask_nxv2i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg4.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2500,7 +2360,6 @@ define <vscale x 2 x i8> @test_vlsseg5_nxv2i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg5e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg5.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2518,7 +2377,6 @@ define <vscale x 2 x i8> @test_vlsseg5_mask_nxv2i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg5.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2536,7 +2394,6 @@ define <vscale x 2 x i8> @test_vlsseg6_nxv2i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg6e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg6.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2555,7 +2412,6 @@ define <vscale x 2 x i8> @test_vlsseg6_mask_nxv2i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg6.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2573,7 +2429,6 @@ define <vscale x 2 x i8> @test_vlsseg7_nxv2i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg7e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg7.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2593,7 +2448,6 @@ define <vscale x 2 x i8> @test_vlsseg7_mask_nxv2i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg7.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2611,7 +2465,6 @@ define <vscale x 2 x i8> @test_vlsseg8_nxv2i8(ptr %base, i64 %offset, i64 %vl) {
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg8e8.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg8.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2632,7 +2485,6 @@ define <vscale x 2 x i8> @test_vlsseg8_mask_nxv2i8(ptr %base, i64 %offset, i64 %
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e8.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vlsseg8.nxv2i8(<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef ,<vscale x 2 x i8> undef, <vscale x 2 x i8> undef, <vscale x 2 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2650,7 +2502,6 @@ define <vscale x 8 x i32> @test_vlsseg2_nxv8i32(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m4, ta, ma
 ; CHECK-NEXT:    vlsseg2e32.v v4, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vlsseg2.nxv8i32(<vscale x 8 x i32> undef, <vscale x 8 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2665,7 +2516,6 @@ define <vscale x 8 x i32> @test_vlsseg2_mask_nxv8i32(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vlsseg2e32.v v4, (a0), a1
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlsseg2e32.v v4, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vlsseg2.nxv8i32(<vscale x 8 x i32> undef, <vscale x 8 x i32> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2683,7 +2533,6 @@ define <vscale x 32 x i8> @test_vlsseg2_nxv32i8(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m4, ta, ma
 ; CHECK-NEXT:    vlsseg2e8.v v4, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>} @llvm.riscv.vlsseg2.nxv32i8(<vscale x 32 x i8> undef, <vscale x 32 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2698,7 +2547,6 @@ define <vscale x 32 x i8> @test_vlsseg2_mask_nxv32i8(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vlsseg2e8.v v4, (a0), a1
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlsseg2e8.v v4, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>} @llvm.riscv.vlsseg2.nxv32i8(<vscale x 32 x i8> undef, <vscale x 32 x i8> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2716,7 +2564,6 @@ define <vscale x 2 x i16> @test_vlsseg2_nxv2i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg2.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2731,7 +2578,6 @@ define <vscale x 2 x i16> @test_vlsseg2_mask_nxv2i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg2.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2749,7 +2595,6 @@ define <vscale x 2 x i16> @test_vlsseg3_nxv2i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg3.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2765,7 +2610,6 @@ define <vscale x 2 x i16> @test_vlsseg3_mask_nxv2i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg3.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2783,7 +2627,6 @@ define <vscale x 2 x i16> @test_vlsseg4_nxv2i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg4.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2800,7 +2643,6 @@ define <vscale x 2 x i16> @test_vlsseg4_mask_nxv2i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg4.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2818,7 +2660,6 @@ define <vscale x 2 x i16> @test_vlsseg5_nxv2i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg5.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2836,7 +2677,6 @@ define <vscale x 2 x i16> @test_vlsseg5_mask_nxv2i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg5.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2854,7 +2694,6 @@ define <vscale x 2 x i16> @test_vlsseg6_nxv2i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg6.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2873,7 +2712,6 @@ define <vscale x 2 x i16> @test_vlsseg6_mask_nxv2i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg6.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2891,7 +2729,6 @@ define <vscale x 2 x i16> @test_vlsseg7_nxv2i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg7.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2911,7 +2748,6 @@ define <vscale x 2 x i16> @test_vlsseg7_mask_nxv2i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg7.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2929,7 +2765,6 @@ define <vscale x 2 x i16> @test_vlsseg8_nxv2i16(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg8.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2950,7 +2785,6 @@ define <vscale x 2 x i16> @test_vlsseg8_mask_nxv2i16(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vlsseg8.nxv2i16(<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef ,<vscale x 2 x i16> undef, <vscale x 2 x i16> undef, <vscale x 2 x i16> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2968,7 +2802,6 @@ define <vscale x 2 x i64> @test_vlsseg2_nxv2i64(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
 ; CHECK-NEXT:    vlsseg2e64.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vlsseg2.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -2983,7 +2816,6 @@ define <vscale x 2 x i64> @test_vlsseg2_mask_nxv2i64(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vlsseg2e64.v v6, (a0), a1
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlsseg2e64.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vlsseg2.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3001,7 +2833,6 @@ define <vscale x 2 x i64> @test_vlsseg3_nxv2i64(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
 ; CHECK-NEXT:    vlsseg3e64.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vlsseg3.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3017,7 +2848,6 @@ define <vscale x 2 x i64> @test_vlsseg3_mask_nxv2i64(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlsseg3e64.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vlsseg3.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3035,7 +2865,6 @@ define <vscale x 2 x i64> @test_vlsseg4_nxv2i64(ptr %base, i64 %offset, i64 %vl)
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
 ; CHECK-NEXT:    vlsseg4e64.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vlsseg4.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3052,7 +2881,6 @@ define <vscale x 2 x i64> @test_vlsseg4_mask_nxv2i64(ptr %base, i64 %offset, i64
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlsseg4e64.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vlsseg4.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, <vscale x 2 x i64> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3070,7 +2898,6 @@ define <vscale x 16 x half> @test_vlsseg2_nxv16f16(ptr %base, i64 %offset, i64 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m4, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v4, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vlsseg2.nxv16f16(<vscale x 16 x half> undef, <vscale x 16 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3085,7 +2912,6 @@ define <vscale x 16 x half> @test_vlsseg2_mask_nxv16f16(ptr %base, i64 %offset,
 ; CHECK-NEXT:    vlsseg2e16.v v4, (a0), a1
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlsseg2e16.v v4, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vlsseg2.nxv16f16(<vscale x 16 x half> undef, <vscale x 16 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3103,7 +2929,6 @@ define <vscale x 4 x double> @test_vlsseg2_nxv4f64(ptr %base, i64 %offset, i64 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m4, ta, ma
 ; CHECK-NEXT:    vlsseg2e64.v v4, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vlsseg2.nxv4f64(<vscale x 4 x double> undef, <vscale x 4 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3118,7 +2943,6 @@ define <vscale x 4 x double> @test_vlsseg2_mask_nxv4f64(ptr %base, i64 %offset,
 ; CHECK-NEXT:    vlsseg2e64.v v4, (a0), a1
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlsseg2e64.v v4, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vlsseg2.nxv4f64(<vscale x 4 x double> undef, <vscale x 4 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3136,7 +2960,6 @@ define <vscale x 1 x double> @test_vlsseg2_nxv1f64(ptr %base, i64 %offset, i64 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg2e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg2.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3151,7 +2974,6 @@ define <vscale x 1 x double> @test_vlsseg2_mask_nxv1f64(ptr %base, i64 %offset,
 ; CHECK-NEXT:    vlsseg2e64.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg2.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3169,7 +2991,6 @@ define <vscale x 1 x double> @test_vlsseg3_nxv1f64(ptr %base, i64 %offset, i64 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg3e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg3.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3185,7 +3006,6 @@ define <vscale x 1 x double> @test_vlsseg3_mask_nxv1f64(ptr %base, i64 %offset,
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg3.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3203,7 +3023,6 @@ define <vscale x 1 x double> @test_vlsseg4_nxv1f64(ptr %base, i64 %offset, i64 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg4e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg4.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3220,7 +3039,6 @@ define <vscale x 1 x double> @test_vlsseg4_mask_nxv1f64(ptr %base, i64 %offset,
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg4.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3238,7 +3056,6 @@ define <vscale x 1 x double> @test_vlsseg5_nxv1f64(ptr %base, i64 %offset, i64 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg5e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg5.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3256,7 +3073,6 @@ define <vscale x 1 x double> @test_vlsseg5_mask_nxv1f64(ptr %base, i64 %offset,
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg5.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3274,7 +3090,6 @@ define <vscale x 1 x double> @test_vlsseg6_nxv1f64(ptr %base, i64 %offset, i64 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg6e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg6.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3293,7 +3108,6 @@ define <vscale x 1 x double> @test_vlsseg6_mask_nxv1f64(ptr %base, i64 %offset,
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg6.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3311,7 +3125,6 @@ define <vscale x 1 x double> @test_vlsseg7_nxv1f64(ptr %base, i64 %offset, i64 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg7e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg7.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3331,7 +3144,6 @@ define <vscale x 1 x double> @test_vlsseg7_mask_nxv1f64(ptr %base, i64 %offset,
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg7.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3349,7 +3161,6 @@ define <vscale x 1 x double> @test_vlsseg8_nxv1f64(ptr %base, i64 %offset, i64 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vlsseg8e64.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg8.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3370,7 +3181,6 @@ define <vscale x 1 x double> @test_vlsseg8_mask_nxv1f64(ptr %base, i64 %offset,
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e64.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vlsseg8.nxv1f64(<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef ,<vscale x 1 x double> undef, <vscale x 1 x double> undef, <vscale x 1 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3388,7 +3198,6 @@ define <vscale x 2 x float> @test_vlsseg2_nxv2f32(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg2.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3403,7 +3212,6 @@ define <vscale x 2 x float> @test_vlsseg2_mask_nxv2f32(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg2.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3421,7 +3229,6 @@ define <vscale x 2 x float> @test_vlsseg3_nxv2f32(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg3e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg3.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3437,7 +3244,6 @@ define <vscale x 2 x float> @test_vlsseg3_mask_nxv2f32(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg3.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3455,7 +3261,6 @@ define <vscale x 2 x float> @test_vlsseg4_nxv2f32(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg4e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg4.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3472,7 +3277,6 @@ define <vscale x 2 x float> @test_vlsseg4_mask_nxv2f32(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg4.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3490,7 +3294,6 @@ define <vscale x 2 x float> @test_vlsseg5_nxv2f32(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg5e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg5.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3508,7 +3311,6 @@ define <vscale x 2 x float> @test_vlsseg5_mask_nxv2f32(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg5.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3526,7 +3328,6 @@ define <vscale x 2 x float> @test_vlsseg6_nxv2f32(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg6e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg6.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3545,7 +3346,6 @@ define <vscale x 2 x float> @test_vlsseg6_mask_nxv2f32(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg6.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3563,7 +3363,6 @@ define <vscale x 2 x float> @test_vlsseg7_nxv2f32(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg7e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg7.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3583,7 +3382,6 @@ define <vscale x 2 x float> @test_vlsseg7_mask_nxv2f32(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg7.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3601,7 +3399,6 @@ define <vscale x 2 x float> @test_vlsseg8_nxv2f32(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vlsseg8e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg8.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3622,7 +3419,6 @@ define <vscale x 2 x float> @test_vlsseg8_mask_nxv2f32(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vlsseg8.nxv2f32(<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef ,<vscale x 2 x float> undef, <vscale x 2 x float> undef, <vscale x 2 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3640,7 +3436,6 @@ define <vscale x 1 x half> @test_vlsseg2_nxv1f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg2.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3655,7 +3450,6 @@ define <vscale x 1 x half> @test_vlsseg2_mask_nxv1f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg2.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3673,7 +3467,6 @@ define <vscale x 1 x half> @test_vlsseg3_nxv1f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg3.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3689,7 +3482,6 @@ define <vscale x 1 x half> @test_vlsseg3_mask_nxv1f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg3.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3707,7 +3499,6 @@ define <vscale x 1 x half> @test_vlsseg4_nxv1f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg4.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3724,7 +3515,6 @@ define <vscale x 1 x half> @test_vlsseg4_mask_nxv1f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg4.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3742,7 +3532,6 @@ define <vscale x 1 x half> @test_vlsseg5_nxv1f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg5.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3760,7 +3549,6 @@ define <vscale x 1 x half> @test_vlsseg5_mask_nxv1f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg5.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3778,7 +3566,6 @@ define <vscale x 1 x half> @test_vlsseg6_nxv1f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg6.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3797,7 +3584,6 @@ define <vscale x 1 x half> @test_vlsseg6_mask_nxv1f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg6.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3815,7 +3601,6 @@ define <vscale x 1 x half> @test_vlsseg7_nxv1f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg7.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3835,7 +3620,6 @@ define <vscale x 1 x half> @test_vlsseg7_mask_nxv1f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg7.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3853,7 +3637,6 @@ define <vscale x 1 x half> @test_vlsseg8_nxv1f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg8.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3874,7 +3657,6 @@ define <vscale x 1 x half> @test_vlsseg8_mask_nxv1f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vlsseg8.nxv1f16(<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef ,<vscale x 1 x half> undef, <vscale x 1 x half> undef, <vscale x 1 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3892,7 +3674,6 @@ define <vscale x 1 x float> @test_vlsseg2_nxv1f32(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg2.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3907,7 +3688,6 @@ define <vscale x 1 x float> @test_vlsseg2_mask_nxv1f32(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg2.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3925,7 +3705,6 @@ define <vscale x 1 x float> @test_vlsseg3_nxv1f32(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg3e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg3.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3941,7 +3720,6 @@ define <vscale x 1 x float> @test_vlsseg3_mask_nxv1f32(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg3.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3959,7 +3737,6 @@ define <vscale x 1 x float> @test_vlsseg4_nxv1f32(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg4e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg4.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3976,7 +3753,6 @@ define <vscale x 1 x float> @test_vlsseg4_mask_nxv1f32(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg4.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -3994,7 +3770,6 @@ define <vscale x 1 x float> @test_vlsseg5_nxv1f32(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg5e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg5.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4012,7 +3787,6 @@ define <vscale x 1 x float> @test_vlsseg5_mask_nxv1f32(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg5.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4030,7 +3804,6 @@ define <vscale x 1 x float> @test_vlsseg6_nxv1f32(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg6e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg6.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4049,7 +3822,6 @@ define <vscale x 1 x float> @test_vlsseg6_mask_nxv1f32(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg6.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4067,7 +3839,6 @@ define <vscale x 1 x float> @test_vlsseg7_nxv1f32(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg7e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg7.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4087,7 +3858,6 @@ define <vscale x 1 x float> @test_vlsseg7_mask_nxv1f32(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg7.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4105,7 +3875,6 @@ define <vscale x 1 x float> @test_vlsseg8_nxv1f32(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg8e32.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg8.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4126,7 +3895,6 @@ define <vscale x 1 x float> @test_vlsseg8_mask_nxv1f32(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e32.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vlsseg8.nxv1f32(<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef ,<vscale x 1 x float> undef, <vscale x 1 x float> undef, <vscale x 1 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4144,7 +3912,6 @@ define <vscale x 8 x half> @test_vlsseg2_nxv8f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlsseg2.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4159,7 +3926,6 @@ define <vscale x 8 x half> @test_vlsseg2_mask_nxv8f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vlsseg2e16.v v6, (a0), a1
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlsseg2e16.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlsseg2.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4177,7 +3943,6 @@ define <vscale x 8 x half> @test_vlsseg3_nxv8f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vlsseg3e16.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlsseg3.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4193,7 +3958,6 @@ define <vscale x 8 x half> @test_vlsseg3_mask_nxv8f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlsseg3e16.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlsseg3.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4211,7 +3975,6 @@ define <vscale x 8 x half> @test_vlsseg4_nxv8f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vlsseg4e16.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlsseg4.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4228,7 +3991,6 @@ define <vscale x 8 x half> @test_vlsseg4_mask_nxv8f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlsseg4e16.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vlsseg4.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, <vscale x 8 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4246,7 +4008,6 @@ define <vscale x 8 x float> @test_vlsseg2_nxv8f32(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m4, ta, ma
 ; CHECK-NEXT:    vlsseg2e32.v v4, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vlsseg2.nxv8f32(<vscale x 8 x float> undef, <vscale x 8 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4261,7 +4022,6 @@ define <vscale x 8 x float> @test_vlsseg2_mask_nxv8f32(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vlsseg2e32.v v4, (a0), a1
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlsseg2e32.v v4, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vlsseg2.nxv8f32(<vscale x 8 x float> undef, <vscale x 8 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4279,7 +4039,6 @@ define <vscale x 2 x double> @test_vlsseg2_nxv2f64(ptr %base, i64 %offset, i64 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
 ; CHECK-NEXT:    vlsseg2e64.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlsseg2.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4294,7 +4053,6 @@ define <vscale x 2 x double> @test_vlsseg2_mask_nxv2f64(ptr %base, i64 %offset,
 ; CHECK-NEXT:    vlsseg2e64.v v6, (a0), a1
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlsseg2e64.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlsseg2.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4312,7 +4070,6 @@ define <vscale x 2 x double> @test_vlsseg3_nxv2f64(ptr %base, i64 %offset, i64 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
 ; CHECK-NEXT:    vlsseg3e64.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlsseg3.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4328,7 +4085,6 @@ define <vscale x 2 x double> @test_vlsseg3_mask_nxv2f64(ptr %base, i64 %offset,
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlsseg3e64.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlsseg3.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4346,7 +4102,6 @@ define <vscale x 2 x double> @test_vlsseg4_nxv2f64(ptr %base, i64 %offset, i64 %
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
 ; CHECK-NEXT:    vlsseg4e64.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlsseg4.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4363,7 +4118,6 @@ define <vscale x 2 x double> @test_vlsseg4_mask_nxv2f64(ptr %base, i64 %offset,
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlsseg4e64.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vlsseg4.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, <vscale x 2 x double> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4381,7 +4135,6 @@ define <vscale x 4 x half> @test_vlsseg2_nxv4f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg2.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4396,7 +4149,6 @@ define <vscale x 4 x half> @test_vlsseg2_mask_nxv4f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg2.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4414,7 +4166,6 @@ define <vscale x 4 x half> @test_vlsseg3_nxv4f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg3.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4430,7 +4181,6 @@ define <vscale x 4 x half> @test_vlsseg3_mask_nxv4f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg3.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4448,7 +4198,6 @@ define <vscale x 4 x half> @test_vlsseg4_nxv4f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg4.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4465,7 +4214,6 @@ define <vscale x 4 x half> @test_vlsseg4_mask_nxv4f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg4.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4483,7 +4231,6 @@ define <vscale x 4 x half> @test_vlsseg5_nxv4f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg5.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4501,7 +4248,6 @@ define <vscale x 4 x half> @test_vlsseg5_mask_nxv4f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg5.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4519,7 +4265,6 @@ define <vscale x 4 x half> @test_vlsseg6_nxv4f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg6.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4538,7 +4283,6 @@ define <vscale x 4 x half> @test_vlsseg6_mask_nxv4f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg6.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4556,7 +4300,6 @@ define <vscale x 4 x half> @test_vlsseg7_nxv4f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg7.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4576,7 +4319,6 @@ define <vscale x 4 x half> @test_vlsseg7_mask_nxv4f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg7.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4594,7 +4336,6 @@ define <vscale x 4 x half> @test_vlsseg8_nxv4f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg8.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4615,7 +4356,6 @@ define <vscale x 4 x half> @test_vlsseg8_mask_nxv4f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vlsseg8.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef ,<vscale x 4 x half> undef, <vscale x 4 x half> undef, <vscale x 4 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4633,7 +4373,6 @@ define <vscale x 2 x half> @test_vlsseg2_nxv2f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg2.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4648,7 +4387,6 @@ define <vscale x 2 x half> @test_vlsseg2_mask_nxv2f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vlsseg2e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg2.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4666,7 +4404,6 @@ define <vscale x 2 x half> @test_vlsseg3_nxv2f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg3.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4682,7 +4419,6 @@ define <vscale x 2 x half> @test_vlsseg3_mask_nxv2f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv1r.v v8, v7
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vlsseg3e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg3.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4700,7 +4436,6 @@ define <vscale x 2 x half> @test_vlsseg4_nxv2f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg4.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4717,7 +4452,6 @@ define <vscale x 2 x half> @test_vlsseg4_mask_nxv2f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv1r.v v9, v7
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vlsseg4e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg4.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4735,7 +4469,6 @@ define <vscale x 2 x half> @test_vlsseg5_nxv2f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg5.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4753,7 +4486,6 @@ define <vscale x 2 x half> @test_vlsseg5_mask_nxv2f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv1r.v v10, v7
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vlsseg5e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg5.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4771,7 +4503,6 @@ define <vscale x 2 x half> @test_vlsseg6_nxv2f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg6.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4790,7 +4521,6 @@ define <vscale x 2 x half> @test_vlsseg6_mask_nxv2f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv1r.v v11, v7
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vlsseg6e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg6.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4808,7 +4538,6 @@ define <vscale x 2 x half> @test_vlsseg7_nxv2f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg7.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4828,7 +4557,6 @@ define <vscale x 2 x half> @test_vlsseg7_mask_nxv2f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv1r.v v12, v7
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vlsseg7e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg7.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4846,7 +4574,6 @@ define <vscale x 2 x half> @test_vlsseg8_nxv2f16(ptr %base, i64 %offset, i64 %vl
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg8.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4867,7 +4594,6 @@ define <vscale x 2 x half> @test_vlsseg8_mask_nxv2f16(ptr %base, i64 %offset, i6
 ; CHECK-NEXT:    vmv1r.v v13, v7
 ; CHECK-NEXT:    vmv1r.v v14, v7
 ; CHECK-NEXT:    vlsseg8e16.v v7, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vlsseg8.nxv2f16(<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef ,<vscale x 2 x half> undef, <vscale x 2 x half> undef, <vscale x 2 x half> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4885,7 +4611,6 @@ define <vscale x 4 x float> @test_vlsseg2_nxv4f32(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vlsseg2e32.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlsseg2.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4900,7 +4625,6 @@ define <vscale x 4 x float> @test_vlsseg2_mask_nxv4f32(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vlsseg2e32.v v6, (a0), a1
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vlsseg2e32.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlsseg2.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4918,7 +4642,6 @@ define <vscale x 4 x float> @test_vlsseg3_nxv4f32(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vlsseg3e32.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlsseg3.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4934,7 +4657,6 @@ define <vscale x 4 x float> @test_vlsseg3_mask_nxv4f32(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vmv2r.v v8, v6
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vlsseg3e32.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlsseg3.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4952,7 +4674,6 @@ define <vscale x 4 x float> @test_vlsseg4_nxv4f32(ptr %base, i64 %offset, i64 %v
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vlsseg4e32.v v6, (a0), a1
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlsseg4.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i64 %offset, i64 %vl)
@@ -4969,7 +4690,6 @@ define <vscale x 4 x float> @test_vlsseg4_mask_nxv4f32(ptr %base, i64 %offset, i
 ; CHECK-NEXT:    vmv2r.v v10, v6
 ; CHECK-NEXT:    vmv2r.v v12, v6
 ; CHECK-NEXT:    vlsseg4e32.v v6, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vlsseg4.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, <vscale x 4 x float> undef, ptr %base, i64 %offset, i64 %vl)
diff --git a/llvm/test/CodeGen/RISCV/rvv/vluxseg-rv32.ll b/llvm/test/CodeGen/RISCV/rvv/vluxseg-rv32.ll
index 5fdf0facb6f76..5d094264d7d45 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vluxseg-rv32.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vluxseg-rv32.ll
@@ -24,7 +24,6 @@ define <vscale x 16 x i16> @test_vluxseg2_mask_nxv16i16_nxv16i16(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vluxseg2.mask.nxv16i16.nxv16i16(<vscale x 16 x i16> %val,<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -54,7 +53,6 @@ define <vscale x 16 x i16> @test_vluxseg2_mask_nxv16i16_nxv16i8(<vscale x 16 x i
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vluxseg2.mask.nxv16i16.nxv16i8(<vscale x 16 x i16> %val,<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -84,7 +82,6 @@ define <vscale x 16 x i16> @test_vluxseg2_mask_nxv16i16_nxv16i32(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v4, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vluxseg2.mask.nxv16i16.nxv16i32(<vscale x 16 x i16> %val,<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -114,7 +111,6 @@ define <vscale x 1 x i8> @test_vluxseg2_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vluxseg2.mask.nxv1i8.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -144,7 +140,6 @@ define <vscale x 1 x i8> @test_vluxseg2_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vluxseg2.mask.nxv1i8.nxv1i32(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -174,7 +169,6 @@ define <vscale x 1 x i8> @test_vluxseg2_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vluxseg2.mask.nxv1i8.nxv1i16(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -201,12 +195,11 @@ entry:
 define <vscale x 1 x i8> @test_vluxseg3_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vluxseg3.mask.nxv1i8.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -233,12 +226,11 @@ entry:
 define <vscale x 1 x i8> @test_vluxseg3_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vluxseg3.mask.nxv1i8.nxv1i32(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -265,12 +257,11 @@ entry:
 define <vscale x 1 x i8> @test_vluxseg3_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vluxseg3.mask.nxv1i8.nxv1i16(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -298,9 +289,9 @@ define <vscale x 1 x i8> @test_vluxseg4_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -331,9 +322,9 @@ define <vscale x 1 x i8> @test_vluxseg4_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -364,9 +355,9 @@ define <vscale x 1 x i8> @test_vluxseg4_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -397,10 +388,10 @@ define <vscale x 1 x i8> @test_vluxseg5_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -431,10 +422,10 @@ define <vscale x 1 x i8> @test_vluxseg5_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -465,10 +456,10 @@ define <vscale x 1 x i8> @test_vluxseg5_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -499,11 +490,11 @@ define <vscale x 1 x i8> @test_vluxseg6_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -534,11 +525,11 @@ define <vscale x 1 x i8> @test_vluxseg6_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -569,11 +560,11 @@ define <vscale x 1 x i8> @test_vluxseg6_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -604,12 +595,12 @@ define <vscale x 1 x i8> @test_vluxseg7_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -640,12 +631,12 @@ define <vscale x 1 x i8> @test_vluxseg7_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -676,12 +667,12 @@ define <vscale x 1 x i8> @test_vluxseg7_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -712,13 +703,13 @@ define <vscale x 1 x i8> @test_vluxseg8_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -749,13 +740,13 @@ define <vscale x 1 x i8> @test_vluxseg8_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -786,13 +777,13 @@ define <vscale x 1 x i8> @test_vluxseg8_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -825,7 +816,6 @@ define <vscale x 16 x i8> @test_vluxseg2_mask_nxv16i8_nxv16i16(<vscale x 16 x i8
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vluxseg2.mask.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -855,7 +845,6 @@ define <vscale x 16 x i8> @test_vluxseg2_mask_nxv16i8_nxv16i8(<vscale x 16 x i8>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vluxseg2.mask.nxv16i8.nxv16i8(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -885,7 +874,6 @@ define <vscale x 16 x i8> @test_vluxseg2_mask_nxv16i8_nxv16i32(<vscale x 16 x i8
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vluxseg2.mask.nxv16i8.nxv16i32(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -913,10 +901,9 @@ define <vscale x 16 x i8> @test_vluxseg3_mask_nxv16i8_nxv16i16(<vscale x 16 x i8
 ; CHECK-LABEL: test_vluxseg3_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vluxseg3ei16.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vluxseg3.mask.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -943,12 +930,11 @@ entry:
 define <vscale x 16 x i8> @test_vluxseg3_mask_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, i32 %vl, <vscale x 16 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vluxseg3.mask.nxv16i8.nxv16i8(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -976,10 +962,9 @@ define <vscale x 16 x i8> @test_vluxseg3_mask_nxv16i8_nxv16i32(<vscale x 16 x i8
 ; CHECK-LABEL: test_vluxseg3_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vluxseg3ei32.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vluxseg3.mask.nxv16i8.nxv16i32(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -1006,13 +991,12 @@ entry:
 define <vscale x 16 x i8> @test_vluxseg4_mask_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, i32 %vl, <vscale x 16 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
-; CHECK-NEXT:    vluxseg4ei16.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v18
+; CHECK-NEXT:    vluxseg4ei16.v v6, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vluxseg4.mask.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -1040,9 +1024,9 @@ define <vscale x 16 x i8> @test_vluxseg4_mask_nxv16i8_nxv16i8(<vscale x 16 x i8>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -1073,11 +1057,10 @@ define <vscale x 16 x i8> @test_vluxseg4_mask_nxv16i8_nxv16i32(<vscale x 16 x i8
 ; CHECK-LABEL: test_vluxseg4_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vluxseg4.mask.nxv16i8.nxv16i32(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -1107,7 +1090,6 @@ define <vscale x 2 x i32> @test_vluxseg2_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vluxseg2.mask.nxv2i32.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -1137,7 +1119,6 @@ define <vscale x 2 x i32> @test_vluxseg2_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vluxseg2.mask.nxv2i32.nxv2i8(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -1167,7 +1148,6 @@ define <vscale x 2 x i32> @test_vluxseg2_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vluxseg2.mask.nxv2i32.nxv2i16(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -1194,12 +1174,11 @@ entry:
 define <vscale x 2 x i32> @test_vluxseg3_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vluxseg3.mask.nxv2i32.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -1226,12 +1205,11 @@ entry:
 define <vscale x 2 x i32> @test_vluxseg3_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vluxseg3.mask.nxv2i32.nxv2i8(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -1258,12 +1236,11 @@ entry:
 define <vscale x 2 x i32> @test_vluxseg3_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vluxseg3.mask.nxv2i32.nxv2i16(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -1291,9 +1268,9 @@ define <vscale x 2 x i32> @test_vluxseg4_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1324,9 +1301,9 @@ define <vscale x 2 x i32> @test_vluxseg4_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1357,9 +1334,9 @@ define <vscale x 2 x i32> @test_vluxseg4_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1390,10 +1367,10 @@ define <vscale x 2 x i32> @test_vluxseg5_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1424,10 +1401,10 @@ define <vscale x 2 x i32> @test_vluxseg5_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1458,10 +1435,10 @@ define <vscale x 2 x i32> @test_vluxseg5_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1492,11 +1469,11 @@ define <vscale x 2 x i32> @test_vluxseg6_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1527,11 +1504,11 @@ define <vscale x 2 x i32> @test_vluxseg6_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1562,11 +1539,11 @@ define <vscale x 2 x i32> @test_vluxseg6_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1597,12 +1574,12 @@ define <vscale x 2 x i32> @test_vluxseg7_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1633,12 +1610,12 @@ define <vscale x 2 x i32> @test_vluxseg7_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1669,12 +1646,12 @@ define <vscale x 2 x i32> @test_vluxseg7_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1705,13 +1682,13 @@ define <vscale x 2 x i32> @test_vluxseg8_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1742,13 +1719,13 @@ define <vscale x 2 x i32> @test_vluxseg8_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1779,13 +1756,13 @@ define <vscale x 2 x i32> @test_vluxseg8_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1818,7 +1795,6 @@ define <vscale x 4 x i16> @test_vluxseg2_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg2.mask.nxv4i16.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -1848,7 +1824,6 @@ define <vscale x 4 x i16> @test_vluxseg2_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg2.mask.nxv4i16.nxv4i8(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -1878,7 +1853,6 @@ define <vscale x 4 x i16> @test_vluxseg2_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg2.mask.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -1905,12 +1879,11 @@ entry:
 define <vscale x 4 x i16> @test_vluxseg3_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg3.mask.nxv4i16.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -1937,12 +1910,11 @@ entry:
 define <vscale x 4 x i16> @test_vluxseg3_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg3.mask.nxv4i16.nxv4i8(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -1970,10 +1942,9 @@ define <vscale x 4 x i16> @test_vluxseg3_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg3.mask.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -2001,9 +1972,9 @@ define <vscale x 4 x i16> @test_vluxseg4_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2034,9 +2005,9 @@ define <vscale x 4 x i16> @test_vluxseg4_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2066,13 +2037,12 @@ entry:
 define <vscale x 4 x i16> @test_vluxseg4_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vluxseg4ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vluxseg4ei32.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg4.mask.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -2100,10 +2070,10 @@ define <vscale x 4 x i16> @test_vluxseg5_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2134,10 +2104,10 @@ define <vscale x 4 x i16> @test_vluxseg5_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2168,10 +2138,10 @@ define <vscale x 4 x i16> @test_vluxseg5_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -2202,11 +2172,11 @@ define <vscale x 4 x i16> @test_vluxseg6_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2237,11 +2207,11 @@ define <vscale x 4 x i16> @test_vluxseg6_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2272,11 +2242,11 @@ define <vscale x 4 x i16> @test_vluxseg6_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -2307,12 +2277,12 @@ define <vscale x 4 x i16> @test_vluxseg7_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2343,12 +2313,12 @@ define <vscale x 4 x i16> @test_vluxseg7_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2379,12 +2349,12 @@ define <vscale x 4 x i16> @test_vluxseg7_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -2415,13 +2385,13 @@ define <vscale x 4 x i16> @test_vluxseg8_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2452,13 +2422,13 @@ define <vscale x 4 x i16> @test_vluxseg8_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2489,13 +2459,13 @@ define <vscale x 4 x i16> @test_vluxseg8_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -2528,7 +2498,6 @@ define <vscale x 1 x i32> @test_vluxseg2_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vluxseg2.mask.nxv1i32.nxv1i8(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2558,7 +2527,6 @@ define <vscale x 1 x i32> @test_vluxseg2_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vluxseg2.mask.nxv1i32.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2588,7 +2556,6 @@ define <vscale x 1 x i32> @test_vluxseg2_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vluxseg2.mask.nxv1i32.nxv1i16(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2615,12 +2582,11 @@ entry:
 define <vscale x 1 x i32> @test_vluxseg3_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vluxseg3.mask.nxv1i32.nxv1i8(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2647,12 +2613,11 @@ entry:
 define <vscale x 1 x i32> @test_vluxseg3_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vluxseg3.mask.nxv1i32.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2679,12 +2644,11 @@ entry:
 define <vscale x 1 x i32> @test_vluxseg3_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vluxseg3.mask.nxv1i32.nxv1i16(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -2712,9 +2676,9 @@ define <vscale x 1 x i32> @test_vluxseg4_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2745,9 +2709,9 @@ define <vscale x 1 x i32> @test_vluxseg4_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2778,9 +2742,9 @@ define <vscale x 1 x i32> @test_vluxseg4_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2811,10 +2775,10 @@ define <vscale x 1 x i32> @test_vluxseg5_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2845,10 +2809,10 @@ define <vscale x 1 x i32> @test_vluxseg5_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2879,10 +2843,10 @@ define <vscale x 1 x i32> @test_vluxseg5_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2913,11 +2877,11 @@ define <vscale x 1 x i32> @test_vluxseg6_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2948,11 +2912,11 @@ define <vscale x 1 x i32> @test_vluxseg6_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2983,11 +2947,11 @@ define <vscale x 1 x i32> @test_vluxseg6_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3018,12 +2982,12 @@ define <vscale x 1 x i32> @test_vluxseg7_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3054,12 +3018,12 @@ define <vscale x 1 x i32> @test_vluxseg7_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3090,12 +3054,12 @@ define <vscale x 1 x i32> @test_vluxseg7_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3126,13 +3090,13 @@ define <vscale x 1 x i32> @test_vluxseg8_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3163,13 +3127,13 @@ define <vscale x 1 x i32> @test_vluxseg8_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3200,13 +3164,13 @@ define <vscale x 1 x i32> @test_vluxseg8_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3239,7 +3203,6 @@ define <vscale x 8 x i16> @test_vluxseg2_mask_nxv8i16_nxv8i16(<vscale x 8 x i16>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vluxseg2.mask.nxv8i16.nxv8i16(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3269,7 +3232,6 @@ define <vscale x 8 x i16> @test_vluxseg2_mask_nxv8i16_nxv8i8(<vscale x 8 x i16>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vluxseg2.mask.nxv8i16.nxv8i8(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3299,7 +3261,6 @@ define <vscale x 8 x i16> @test_vluxseg2_mask_nxv8i16_nxv8i32(<vscale x 8 x i16>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vluxseg2.mask.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3326,12 +3287,11 @@ entry:
 define <vscale x 8 x i16> @test_vluxseg3_mask_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vluxseg3.mask.nxv8i16.nxv8i16(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3358,12 +3318,11 @@ entry:
 define <vscale x 8 x i16> @test_vluxseg3_mask_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, i32 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vluxseg3.mask.nxv8i16.nxv8i8(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3391,10 +3350,9 @@ define <vscale x 8 x i16> @test_vluxseg3_mask_nxv8i16_nxv8i32(<vscale x 8 x i16>
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg3ei32.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vluxseg3.mask.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3422,9 +3380,9 @@ define <vscale x 8 x i16> @test_vluxseg4_mask_nxv8i16_nxv8i16(<vscale x 8 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -3455,9 +3413,9 @@ define <vscale x 8 x i16> @test_vluxseg4_mask_nxv8i16_nxv8i8(<vscale x 8 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -3487,13 +3445,12 @@ entry:
 define <vscale x 8 x i16> @test_vluxseg4_mask_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vluxseg4ei32.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v18
+; CHECK-NEXT:    vluxseg4ei32.v v6, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vluxseg4.mask.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3523,7 +3480,6 @@ define <vscale x 8 x i8> @test_vluxseg2_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg2.mask.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3553,7 +3509,6 @@ define <vscale x 8 x i8> @test_vluxseg2_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg2.mask.nxv8i8.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3583,7 +3538,6 @@ define <vscale x 8 x i8> @test_vluxseg2_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg2.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3611,10 +3565,9 @@ define <vscale x 8 x i8> @test_vluxseg3_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg3.mask.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3641,12 +3594,11 @@ entry:
 define <vscale x 8 x i8> @test_vluxseg3_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, i32 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg3.mask.nxv8i8.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3674,10 +3626,9 @@ define <vscale x 8 x i8> @test_vluxseg3_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg3.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3704,13 +3655,12 @@ entry:
 define <vscale x 8 x i8> @test_vluxseg4_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
-; CHECK-NEXT:    vluxseg4ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vluxseg4ei16.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg4.mask.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3738,9 +3688,9 @@ define <vscale x 8 x i8> @test_vluxseg4_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3771,11 +3721,10 @@ define <vscale x 8 x i8> @test_vluxseg4_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg4.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3803,10 +3752,10 @@ define <vscale x 8 x i8> @test_vluxseg5_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -3837,10 +3786,10 @@ define <vscale x 8 x i8> @test_vluxseg5_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vluxseg5_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3871,12 +3820,11 @@ define <vscale x 8 x i8> @test_vluxseg5_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg5.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -3904,11 +3852,11 @@ define <vscale x 8 x i8> @test_vluxseg6_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg6_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -3939,11 +3887,11 @@ define <vscale x 8 x i8> @test_vluxseg6_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vluxseg6_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3973,15 +3921,14 @@ entry:
 define <vscale x 8 x i8> @test_vluxseg6_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg6_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
-; CHECK-NEXT:    vluxseg6ei32.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v17
+; CHECK-NEXT:    vluxseg6ei32.v v7, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg6.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -4009,12 +3956,12 @@ define <vscale x 8 x i8> @test_vluxseg7_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -4045,12 +3992,12 @@ define <vscale x 8 x i8> @test_vluxseg7_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vluxseg7_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4081,12 +4028,12 @@ define <vscale x 8 x i8> @test_vluxseg7_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -4117,13 +4064,13 @@ define <vscale x 8 x i8> @test_vluxseg8_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -4154,13 +4101,13 @@ define <vscale x 8 x i8> @test_vluxseg8_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vluxseg8_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4191,13 +4138,13 @@ define <vscale x 8 x i8> @test_vluxseg8_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -4230,7 +4177,6 @@ define <vscale x 8 x i32> @test_vluxseg2_mask_nxv8i32_nxv8i16(<vscale x 8 x i32>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vluxseg2.mask.nxv8i32.nxv8i16(<vscale x 8 x i32> %val,<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -4260,7 +4206,6 @@ define <vscale x 8 x i32> @test_vluxseg2_mask_nxv8i32_nxv8i8(<vscale x 8 x i32>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vluxseg2.mask.nxv8i32.nxv8i8(<vscale x 8 x i32> %val,<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -4290,7 +4235,6 @@ define <vscale x 8 x i32> @test_vluxseg2_mask_nxv8i32_nxv8i32(<vscale x 8 x i32>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vluxseg2.mask.nxv8i32.nxv8i32(<vscale x 8 x i32> %val,<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -4320,7 +4264,6 @@ define <vscale x 4 x i8> @test_vluxseg2_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg2.mask.nxv4i8.nxv4i16(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4350,7 +4293,6 @@ define <vscale x 4 x i8> @test_vluxseg2_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg2.mask.nxv4i8.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4380,7 +4322,6 @@ define <vscale x 4 x i8> @test_vluxseg2_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg2.mask.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4407,12 +4348,11 @@ entry:
 define <vscale x 4 x i8> @test_vluxseg3_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg3.mask.nxv4i8.nxv4i16(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4439,12 +4379,11 @@ entry:
 define <vscale x 4 x i8> @test_vluxseg3_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg3.mask.nxv4i8.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4472,10 +4411,9 @@ define <vscale x 4 x i8> @test_vluxseg3_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg3.mask.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4503,9 +4441,9 @@ define <vscale x 4 x i8> @test_vluxseg4_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4536,9 +4474,9 @@ define <vscale x 4 x i8> @test_vluxseg4_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4568,13 +4506,12 @@ entry:
 define <vscale x 4 x i8> @test_vluxseg4_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
-; CHECK-NEXT:    vluxseg4ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vluxseg4ei32.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg4.mask.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -4602,10 +4539,10 @@ define <vscale x 4 x i8> @test_vluxseg5_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4636,10 +4573,10 @@ define <vscale x 4 x i8> @test_vluxseg5_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4670,10 +4607,10 @@ define <vscale x 4 x i8> @test_vluxseg5_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -4704,11 +4641,11 @@ define <vscale x 4 x i8> @test_vluxseg6_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4739,11 +4676,11 @@ define <vscale x 4 x i8> @test_vluxseg6_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4774,11 +4711,11 @@ define <vscale x 4 x i8> @test_vluxseg6_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -4809,12 +4746,12 @@ define <vscale x 4 x i8> @test_vluxseg7_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4845,12 +4782,12 @@ define <vscale x 4 x i8> @test_vluxseg7_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4881,12 +4818,12 @@ define <vscale x 4 x i8> @test_vluxseg7_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -4917,13 +4854,13 @@ define <vscale x 4 x i8> @test_vluxseg8_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4954,13 +4891,13 @@ define <vscale x 4 x i8> @test_vluxseg8_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4991,13 +4928,13 @@ define <vscale x 4 x i8> @test_vluxseg8_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -5030,7 +4967,6 @@ define <vscale x 1 x i16> @test_vluxseg2_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vluxseg2.mask.nxv1i16.nxv1i8(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -5060,7 +4996,6 @@ define <vscale x 1 x i16> @test_vluxseg2_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vluxseg2.mask.nxv1i16.nxv1i32(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -5090,7 +5025,6 @@ define <vscale x 1 x i16> @test_vluxseg2_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vluxseg2.mask.nxv1i16.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -5117,12 +5051,11 @@ entry:
 define <vscale x 1 x i16> @test_vluxseg3_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vluxseg3.mask.nxv1i16.nxv1i8(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -5149,12 +5082,11 @@ entry:
 define <vscale x 1 x i16> @test_vluxseg3_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vluxseg3.mask.nxv1i16.nxv1i32(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -5181,12 +5113,11 @@ entry:
 define <vscale x 1 x i16> @test_vluxseg3_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vluxseg3.mask.nxv1i16.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -5214,9 +5145,9 @@ define <vscale x 1 x i16> @test_vluxseg4_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5247,9 +5178,9 @@ define <vscale x 1 x i16> @test_vluxseg4_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5280,9 +5211,9 @@ define <vscale x 1 x i16> @test_vluxseg4_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5313,10 +5244,10 @@ define <vscale x 1 x i16> @test_vluxseg5_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5347,10 +5278,10 @@ define <vscale x 1 x i16> @test_vluxseg5_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5381,10 +5312,10 @@ define <vscale x 1 x i16> @test_vluxseg5_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5415,11 +5346,11 @@ define <vscale x 1 x i16> @test_vluxseg6_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5450,11 +5381,11 @@ define <vscale x 1 x i16> @test_vluxseg6_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5485,11 +5416,11 @@ define <vscale x 1 x i16> @test_vluxseg6_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5520,12 +5451,12 @@ define <vscale x 1 x i16> @test_vluxseg7_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5556,12 +5487,12 @@ define <vscale x 1 x i16> @test_vluxseg7_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5592,12 +5523,12 @@ define <vscale x 1 x i16> @test_vluxseg7_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5628,13 +5559,13 @@ define <vscale x 1 x i16> @test_vluxseg8_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5665,13 +5596,13 @@ define <vscale x 1 x i16> @test_vluxseg8_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5702,13 +5633,13 @@ define <vscale x 1 x i16> @test_vluxseg8_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5741,7 +5672,6 @@ define <vscale x 32 x i8> @test_vluxseg2_mask_nxv32i8_nxv32i16(<vscale x 32 x i8
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v4, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>} @llvm.riscv.vluxseg2.mask.nxv32i8.nxv32i16(<vscale x 32 x i8> %val,<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i16> %index, <vscale x 32 x i1> %mask, i32 %vl, i32 1)
@@ -5771,7 +5701,6 @@ define <vscale x 32 x i8> @test_vluxseg2_mask_nxv32i8_nxv32i8(<vscale x 32 x i8>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>} @llvm.riscv.vluxseg2.mask.nxv32i8.nxv32i8(<vscale x 32 x i8> %val,<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i8> %index, <vscale x 32 x i1> %mask, i32 %vl, i32 1)
@@ -5801,7 +5730,6 @@ define <vscale x 2 x i8> @test_vluxseg2_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vluxseg2.mask.nxv2i8.nxv2i32(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -5831,7 +5759,6 @@ define <vscale x 2 x i8> @test_vluxseg2_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vluxseg2.mask.nxv2i8.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -5861,7 +5788,6 @@ define <vscale x 2 x i8> @test_vluxseg2_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vluxseg2.mask.nxv2i8.nxv2i16(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -5888,12 +5814,11 @@ entry:
 define <vscale x 2 x i8> @test_vluxseg3_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vluxseg3.mask.nxv2i8.nxv2i32(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -5920,12 +5845,11 @@ entry:
 define <vscale x 2 x i8> @test_vluxseg3_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vluxseg3.mask.nxv2i8.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -5952,12 +5876,11 @@ entry:
 define <vscale x 2 x i8> @test_vluxseg3_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vluxseg3.mask.nxv2i8.nxv2i16(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -5985,9 +5908,9 @@ define <vscale x 2 x i8> @test_vluxseg4_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6018,9 +5941,9 @@ define <vscale x 2 x i8> @test_vluxseg4_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6051,9 +5974,9 @@ define <vscale x 2 x i8> @test_vluxseg4_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6084,10 +6007,10 @@ define <vscale x 2 x i8> @test_vluxseg5_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6118,10 +6041,10 @@ define <vscale x 2 x i8> @test_vluxseg5_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6152,10 +6075,10 @@ define <vscale x 2 x i8> @test_vluxseg5_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6186,11 +6109,11 @@ define <vscale x 2 x i8> @test_vluxseg6_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6221,11 +6144,11 @@ define <vscale x 2 x i8> @test_vluxseg6_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6256,11 +6179,11 @@ define <vscale x 2 x i8> @test_vluxseg6_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6291,12 +6214,12 @@ define <vscale x 2 x i8> @test_vluxseg7_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6327,12 +6250,12 @@ define <vscale x 2 x i8> @test_vluxseg7_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6363,12 +6286,12 @@ define <vscale x 2 x i8> @test_vluxseg7_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6399,13 +6322,13 @@ define <vscale x 2 x i8> @test_vluxseg8_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6436,13 +6359,13 @@ define <vscale x 2 x i8> @test_vluxseg8_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6473,13 +6396,13 @@ define <vscale x 2 x i8> @test_vluxseg8_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6512,7 +6435,6 @@ define <vscale x 2 x i16> @test_vluxseg2_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vluxseg2.mask.nxv2i16.nxv2i32(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -6542,7 +6464,6 @@ define <vscale x 2 x i16> @test_vluxseg2_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vluxseg2.mask.nxv2i16.nxv2i8(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -6572,7 +6493,6 @@ define <vscale x 2 x i16> @test_vluxseg2_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vluxseg2.mask.nxv2i16.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -6599,12 +6519,11 @@ entry:
 define <vscale x 2 x i16> @test_vluxseg3_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vluxseg3.mask.nxv2i16.nxv2i32(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -6631,12 +6550,11 @@ entry:
 define <vscale x 2 x i16> @test_vluxseg3_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vluxseg3.mask.nxv2i16.nxv2i8(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -6663,12 +6581,11 @@ entry:
 define <vscale x 2 x i16> @test_vluxseg3_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vluxseg3.mask.nxv2i16.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -6696,9 +6613,9 @@ define <vscale x 2 x i16> @test_vluxseg4_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6729,9 +6646,9 @@ define <vscale x 2 x i16> @test_vluxseg4_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6762,9 +6679,9 @@ define <vscale x 2 x i16> @test_vluxseg4_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6795,10 +6712,10 @@ define <vscale x 2 x i16> @test_vluxseg5_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6829,10 +6746,10 @@ define <vscale x 2 x i16> @test_vluxseg5_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6863,10 +6780,10 @@ define <vscale x 2 x i16> @test_vluxseg5_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6897,11 +6814,11 @@ define <vscale x 2 x i16> @test_vluxseg6_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6932,11 +6849,11 @@ define <vscale x 2 x i16> @test_vluxseg6_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6967,11 +6884,11 @@ define <vscale x 2 x i16> @test_vluxseg6_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7002,12 +6919,12 @@ define <vscale x 2 x i16> @test_vluxseg7_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7038,12 +6955,12 @@ define <vscale x 2 x i16> @test_vluxseg7_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7074,12 +6991,12 @@ define <vscale x 2 x i16> @test_vluxseg7_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7110,13 +7027,13 @@ define <vscale x 2 x i16> @test_vluxseg8_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7147,13 +7064,13 @@ define <vscale x 2 x i16> @test_vluxseg8_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7184,13 +7101,13 @@ define <vscale x 2 x i16> @test_vluxseg8_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7223,7 +7140,6 @@ define <vscale x 4 x i32> @test_vluxseg2_mask_nxv4i32_nxv4i16(<vscale x 4 x i32>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vluxseg2.mask.nxv4i32.nxv4i16(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -7253,7 +7169,6 @@ define <vscale x 4 x i32> @test_vluxseg2_mask_nxv4i32_nxv4i8(<vscale x 4 x i32>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vluxseg2.mask.nxv4i32.nxv4i8(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -7283,7 +7198,6 @@ define <vscale x 4 x i32> @test_vluxseg2_mask_nxv4i32_nxv4i32(<vscale x 4 x i32>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vluxseg2.mask.nxv4i32.nxv4i32(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -7310,12 +7224,11 @@ entry:
 define <vscale x 4 x i32> @test_vluxseg3_mask_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vluxseg3.mask.nxv4i32.nxv4i16(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -7342,12 +7255,11 @@ entry:
 define <vscale x 4 x i32> @test_vluxseg3_mask_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vluxseg3.mask.nxv4i32.nxv4i8(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -7374,12 +7286,11 @@ entry:
 define <vscale x 4 x i32> @test_vluxseg3_mask_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei32.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vluxseg3.mask.nxv4i32.nxv4i32(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -7407,9 +7318,9 @@ define <vscale x 4 x i32> @test_vluxseg4_mask_nxv4i32_nxv4i16(<vscale x 4 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -7440,9 +7351,9 @@ define <vscale x 4 x i32> @test_vluxseg4_mask_nxv4i32_nxv4i8(<vscale x 4 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -7473,9 +7384,9 @@ define <vscale x 4 x i32> @test_vluxseg4_mask_nxv4i32_nxv4i32(<vscale x 4 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -7508,7 +7419,6 @@ define <vscale x 16 x half> @test_vluxseg2_mask_nxv16f16_nxv16i16(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vluxseg2.mask.nxv16f16.nxv16i16(<vscale x 16 x half> %val,<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -7538,7 +7448,6 @@ define <vscale x 16 x half> @test_vluxseg2_mask_nxv16f16_nxv16i8(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vluxseg2.mask.nxv16f16.nxv16i8(<vscale x 16 x half> %val,<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -7568,7 +7477,6 @@ define <vscale x 16 x half> @test_vluxseg2_mask_nxv16f16_nxv16i32(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v4, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vluxseg2.mask.nxv16f16.nxv16i32(<vscale x 16 x half> %val,<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl, i32 1)
@@ -7598,7 +7506,6 @@ define <vscale x 4 x double> @test_vluxseg2_mask_nxv4f64_nxv4i16(<vscale x 4 x d
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vluxseg2.mask.nxv4f64.nxv4i16(<vscale x 4 x double> %val,<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -7628,7 +7535,6 @@ define <vscale x 4 x double> @test_vluxseg2_mask_nxv4f64_nxv4i8(<vscale x 4 x do
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vluxseg2.mask.nxv4f64.nxv4i8(<vscale x 4 x double> %val,<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -7658,7 +7564,6 @@ define <vscale x 4 x double> @test_vluxseg2_mask_nxv4f64_nxv4i32(<vscale x 4 x d
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vluxseg2.mask.nxv4f64.nxv4i32(<vscale x 4 x double> %val,<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -7688,7 +7593,6 @@ define <vscale x 1 x double> @test_vluxseg2_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vluxseg2.mask.nxv1f64.nxv1i8(<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -7718,7 +7622,6 @@ define <vscale x 1 x double> @test_vluxseg2_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vluxseg2.mask.nxv1f64.nxv1i32(<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -7748,7 +7651,6 @@ define <vscale x 1 x double> @test_vluxseg2_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vluxseg2.mask.nxv1f64.nxv1i16(<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -7775,12 +7677,11 @@ entry:
 define <vscale x 1 x double> @test_vluxseg3_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vluxseg3.mask.nxv1f64.nxv1i8(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -7807,12 +7708,11 @@ entry:
 define <vscale x 1 x double> @test_vluxseg3_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vluxseg3.mask.nxv1f64.nxv1i32(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -7839,12 +7739,11 @@ entry:
 define <vscale x 1 x double> @test_vluxseg3_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vluxseg3.mask.nxv1f64.nxv1i16(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -7872,9 +7771,9 @@ define <vscale x 1 x double> @test_vluxseg4_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7905,9 +7804,9 @@ define <vscale x 1 x double> @test_vluxseg4_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7938,9 +7837,9 @@ define <vscale x 1 x double> @test_vluxseg4_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7971,10 +7870,10 @@ define <vscale x 1 x double> @test_vluxseg5_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8005,10 +7904,10 @@ define <vscale x 1 x double> @test_vluxseg5_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8039,10 +7938,10 @@ define <vscale x 1 x double> @test_vluxseg5_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8073,11 +7972,11 @@ define <vscale x 1 x double> @test_vluxseg6_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8108,11 +8007,11 @@ define <vscale x 1 x double> @test_vluxseg6_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8143,11 +8042,11 @@ define <vscale x 1 x double> @test_vluxseg6_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8178,12 +8077,12 @@ define <vscale x 1 x double> @test_vluxseg7_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8214,12 +8113,12 @@ define <vscale x 1 x double> @test_vluxseg7_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8250,12 +8149,12 @@ define <vscale x 1 x double> @test_vluxseg7_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8286,13 +8185,13 @@ define <vscale x 1 x double> @test_vluxseg8_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8323,13 +8222,13 @@ define <vscale x 1 x double> @test_vluxseg8_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8360,13 +8259,13 @@ define <vscale x 1 x double> @test_vluxseg8_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8399,7 +8298,6 @@ define <vscale x 2 x float> @test_vluxseg2_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vluxseg2.mask.nxv2f32.nxv2i32(<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -8429,7 +8327,6 @@ define <vscale x 2 x float> @test_vluxseg2_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vluxseg2.mask.nxv2f32.nxv2i8(<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -8459,7 +8356,6 @@ define <vscale x 2 x float> @test_vluxseg2_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vluxseg2.mask.nxv2f32.nxv2i16(<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -8486,12 +8382,11 @@ entry:
 define <vscale x 2 x float> @test_vluxseg3_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vluxseg3.mask.nxv2f32.nxv2i32(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -8518,12 +8413,11 @@ entry:
 define <vscale x 2 x float> @test_vluxseg3_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vluxseg3.mask.nxv2f32.nxv2i8(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -8550,12 +8444,11 @@ entry:
 define <vscale x 2 x float> @test_vluxseg3_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vluxseg3.mask.nxv2f32.nxv2i16(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -8583,9 +8476,9 @@ define <vscale x 2 x float> @test_vluxseg4_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8616,9 +8509,9 @@ define <vscale x 2 x float> @test_vluxseg4_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8649,9 +8542,9 @@ define <vscale x 2 x float> @test_vluxseg4_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8682,10 +8575,10 @@ define <vscale x 2 x float> @test_vluxseg5_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8716,10 +8609,10 @@ define <vscale x 2 x float> @test_vluxseg5_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8750,10 +8643,10 @@ define <vscale x 2 x float> @test_vluxseg5_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8784,11 +8677,11 @@ define <vscale x 2 x float> @test_vluxseg6_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8819,11 +8712,11 @@ define <vscale x 2 x float> @test_vluxseg6_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8854,11 +8747,11 @@ define <vscale x 2 x float> @test_vluxseg6_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8889,12 +8782,12 @@ define <vscale x 2 x float> @test_vluxseg7_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8925,12 +8818,12 @@ define <vscale x 2 x float> @test_vluxseg7_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8961,12 +8854,12 @@ define <vscale x 2 x float> @test_vluxseg7_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8997,13 +8890,13 @@ define <vscale x 2 x float> @test_vluxseg8_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9034,13 +8927,13 @@ define <vscale x 2 x float> @test_vluxseg8_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9071,13 +8964,13 @@ define <vscale x 2 x float> @test_vluxseg8_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9110,7 +9003,6 @@ define <vscale x 1 x half> @test_vluxseg2_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vluxseg2.mask.nxv1f16.nxv1i8(<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9140,7 +9032,6 @@ define <vscale x 1 x half> @test_vluxseg2_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vluxseg2.mask.nxv1f16.nxv1i32(<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9170,7 +9061,6 @@ define <vscale x 1 x half> @test_vluxseg2_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vluxseg2.mask.nxv1f16.nxv1i16(<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9197,12 +9087,11 @@ entry:
 define <vscale x 1 x half> @test_vluxseg3_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vluxseg3.mask.nxv1f16.nxv1i8(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9229,12 +9118,11 @@ entry:
 define <vscale x 1 x half> @test_vluxseg3_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vluxseg3.mask.nxv1f16.nxv1i32(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9261,12 +9149,11 @@ entry:
 define <vscale x 1 x half> @test_vluxseg3_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vluxseg3.mask.nxv1f16.nxv1i16(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9294,9 +9181,9 @@ define <vscale x 1 x half> @test_vluxseg4_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9327,9 +9214,9 @@ define <vscale x 1 x half> @test_vluxseg4_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9360,9 +9247,9 @@ define <vscale x 1 x half> @test_vluxseg4_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9393,10 +9280,10 @@ define <vscale x 1 x half> @test_vluxseg5_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9427,10 +9314,10 @@ define <vscale x 1 x half> @test_vluxseg5_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9461,10 +9348,10 @@ define <vscale x 1 x half> @test_vluxseg5_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9495,11 +9382,11 @@ define <vscale x 1 x half> @test_vluxseg6_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9530,11 +9417,11 @@ define <vscale x 1 x half> @test_vluxseg6_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9565,11 +9452,11 @@ define <vscale x 1 x half> @test_vluxseg6_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9600,12 +9487,12 @@ define <vscale x 1 x half> @test_vluxseg7_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9636,12 +9523,12 @@ define <vscale x 1 x half> @test_vluxseg7_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9672,12 +9559,12 @@ define <vscale x 1 x half> @test_vluxseg7_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9708,13 +9595,13 @@ define <vscale x 1 x half> @test_vluxseg8_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9745,13 +9632,13 @@ define <vscale x 1 x half> @test_vluxseg8_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9782,13 +9669,13 @@ define <vscale x 1 x half> @test_vluxseg8_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9821,7 +9708,6 @@ define <vscale x 1 x float> @test_vluxseg2_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vluxseg2.mask.nxv1f32.nxv1i8(<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9851,7 +9737,6 @@ define <vscale x 1 x float> @test_vluxseg2_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vluxseg2.mask.nxv1f32.nxv1i32(<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9881,7 +9766,6 @@ define <vscale x 1 x float> @test_vluxseg2_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vluxseg2.mask.nxv1f32.nxv1i16(<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9908,12 +9792,11 @@ entry:
 define <vscale x 1 x float> @test_vluxseg3_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vluxseg3.mask.nxv1f32.nxv1i8(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9940,12 +9823,11 @@ entry:
 define <vscale x 1 x float> @test_vluxseg3_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vluxseg3.mask.nxv1f32.nxv1i32(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -9972,12 +9854,11 @@ entry:
 define <vscale x 1 x float> @test_vluxseg3_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vluxseg3.mask.nxv1f32.nxv1i16(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl, i32 1)
@@ -10005,9 +9886,9 @@ define <vscale x 1 x float> @test_vluxseg4_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10038,9 +9919,9 @@ define <vscale x 1 x float> @test_vluxseg4_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10071,9 +9952,9 @@ define <vscale x 1 x float> @test_vluxseg4_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10104,10 +9985,10 @@ define <vscale x 1 x float> @test_vluxseg5_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10138,10 +10019,10 @@ define <vscale x 1 x float> @test_vluxseg5_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10172,10 +10053,10 @@ define <vscale x 1 x float> @test_vluxseg5_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10206,11 +10087,11 @@ define <vscale x 1 x float> @test_vluxseg6_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10241,11 +10122,11 @@ define <vscale x 1 x float> @test_vluxseg6_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10276,11 +10157,11 @@ define <vscale x 1 x float> @test_vluxseg6_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10311,12 +10192,12 @@ define <vscale x 1 x float> @test_vluxseg7_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10347,12 +10228,12 @@ define <vscale x 1 x float> @test_vluxseg7_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10383,12 +10264,12 @@ define <vscale x 1 x float> @test_vluxseg7_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10419,13 +10300,13 @@ define <vscale x 1 x float> @test_vluxseg8_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10456,13 +10337,13 @@ define <vscale x 1 x float> @test_vluxseg8_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10493,13 +10374,13 @@ define <vscale x 1 x float> @test_vluxseg8_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10532,7 +10413,6 @@ define <vscale x 8 x half> @test_vluxseg2_mask_nxv8f16_nxv8i16(<vscale x 8 x hal
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vluxseg2.mask.nxv8f16.nxv8i16(<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10562,7 +10442,6 @@ define <vscale x 8 x half> @test_vluxseg2_mask_nxv8f16_nxv8i8(<vscale x 8 x half
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vluxseg2.mask.nxv8f16.nxv8i8(<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10592,7 +10471,6 @@ define <vscale x 8 x half> @test_vluxseg2_mask_nxv8f16_nxv8i32(<vscale x 8 x hal
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vluxseg2.mask.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10619,12 +10497,11 @@ entry:
 define <vscale x 8 x half> @test_vluxseg3_mask_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vluxseg3.mask.nxv8f16.nxv8i16(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10651,12 +10528,11 @@ entry:
 define <vscale x 8 x half> @test_vluxseg3_mask_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, i32 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vluxseg3.mask.nxv8f16.nxv8i8(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10684,10 +10560,9 @@ define <vscale x 8 x half> @test_vluxseg3_mask_nxv8f16_nxv8i32(<vscale x 8 x hal
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg3ei32.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vluxseg3.mask.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10715,9 +10590,9 @@ define <vscale x 8 x half> @test_vluxseg4_mask_nxv8f16_nxv8i16(<vscale x 8 x hal
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -10748,9 +10623,9 @@ define <vscale x 8 x half> @test_vluxseg4_mask_nxv8f16_nxv8i8(<vscale x 8 x half
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -10780,13 +10655,12 @@ entry:
 define <vscale x 8 x half> @test_vluxseg4_mask_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vluxseg4ei32.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v18
+; CHECK-NEXT:    vluxseg4ei32.v v6, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vluxseg4.mask.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10816,7 +10690,6 @@ define <vscale x 8 x float> @test_vluxseg2_mask_nxv8f32_nxv8i16(<vscale x 8 x fl
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vluxseg2.mask.nxv8f32.nxv8i16(<vscale x 8 x float> %val,<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10846,7 +10719,6 @@ define <vscale x 8 x float> @test_vluxseg2_mask_nxv8f32_nxv8i8(<vscale x 8 x flo
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vluxseg2.mask.nxv8f32.nxv8i8(<vscale x 8 x float> %val,<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10876,7 +10748,6 @@ define <vscale x 8 x float> @test_vluxseg2_mask_nxv8f32_nxv8i32(<vscale x 8 x fl
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vluxseg2.mask.nxv8f32.nxv8i32(<vscale x 8 x float> %val,<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl, i32 1)
@@ -10906,7 +10777,6 @@ define <vscale x 2 x double> @test_vluxseg2_mask_nxv2f64_nxv2i32(<vscale x 2 x d
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vluxseg2.mask.nxv2f64.nxv2i32(<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -10936,7 +10806,6 @@ define <vscale x 2 x double> @test_vluxseg2_mask_nxv2f64_nxv2i8(<vscale x 2 x do
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vluxseg2.mask.nxv2f64.nxv2i8(<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -10966,7 +10835,6 @@ define <vscale x 2 x double> @test_vluxseg2_mask_nxv2f64_nxv2i16(<vscale x 2 x d
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vluxseg2.mask.nxv2f64.nxv2i16(<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -10993,12 +10861,11 @@ entry:
 define <vscale x 2 x double> @test_vluxseg3_mask_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei32.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vluxseg3.mask.nxv2f64.nxv2i32(<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -11025,12 +10892,11 @@ entry:
 define <vscale x 2 x double> @test_vluxseg3_mask_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vluxseg3.mask.nxv2f64.nxv2i8(<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -11057,12 +10923,11 @@ entry:
 define <vscale x 2 x double> @test_vluxseg3_mask_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vluxseg3.mask.nxv2f64.nxv2i16(<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -11090,9 +10955,9 @@ define <vscale x 2 x double> @test_vluxseg4_mask_nxv2f64_nxv2i32(<vscale x 2 x d
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -11123,9 +10988,9 @@ define <vscale x 2 x double> @test_vluxseg4_mask_nxv2f64_nxv2i8(<vscale x 2 x do
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -11156,9 +11021,9 @@ define <vscale x 2 x double> @test_vluxseg4_mask_nxv2f64_nxv2i16(<vscale x 2 x d
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -11191,7 +11056,6 @@ define <vscale x 4 x half> @test_vluxseg2_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg2.mask.nxv4f16.nxv4i16(<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -11221,7 +11085,6 @@ define <vscale x 4 x half> @test_vluxseg2_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg2.mask.nxv4f16.nxv4i8(<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -11251,7 +11114,6 @@ define <vscale x 4 x half> @test_vluxseg2_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg2.mask.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -11278,12 +11140,11 @@ entry:
 define <vscale x 4 x half> @test_vluxseg3_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg3.mask.nxv4f16.nxv4i16(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -11310,12 +11171,11 @@ entry:
 define <vscale x 4 x half> @test_vluxseg3_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg3.mask.nxv4f16.nxv4i8(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -11343,10 +11203,9 @@ define <vscale x 4 x half> @test_vluxseg3_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg3.mask.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -11374,9 +11233,9 @@ define <vscale x 4 x half> @test_vluxseg4_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11407,9 +11266,9 @@ define <vscale x 4 x half> @test_vluxseg4_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11439,13 +11298,12 @@ entry:
 define <vscale x 4 x half> @test_vluxseg4_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vluxseg4ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vluxseg4ei32.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg4.mask.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -11473,10 +11331,10 @@ define <vscale x 4 x half> @test_vluxseg5_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11507,10 +11365,10 @@ define <vscale x 4 x half> @test_vluxseg5_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11541,10 +11399,10 @@ define <vscale x 4 x half> @test_vluxseg5_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -11575,11 +11433,11 @@ define <vscale x 4 x half> @test_vluxseg6_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11610,11 +11468,11 @@ define <vscale x 4 x half> @test_vluxseg6_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11645,11 +11503,11 @@ define <vscale x 4 x half> @test_vluxseg6_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -11680,12 +11538,12 @@ define <vscale x 4 x half> @test_vluxseg7_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11716,12 +11574,12 @@ define <vscale x 4 x half> @test_vluxseg7_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11752,12 +11610,12 @@ define <vscale x 4 x half> @test_vluxseg7_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -11788,13 +11646,13 @@ define <vscale x 4 x half> @test_vluxseg8_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11825,13 +11683,13 @@ define <vscale x 4 x half> @test_vluxseg8_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11862,13 +11720,13 @@ define <vscale x 4 x half> @test_vluxseg8_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -11901,7 +11759,6 @@ define <vscale x 2 x half> @test_vluxseg2_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vluxseg2.mask.nxv2f16.nxv2i32(<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -11931,7 +11788,6 @@ define <vscale x 2 x half> @test_vluxseg2_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vluxseg2.mask.nxv2f16.nxv2i8(<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -11961,7 +11817,6 @@ define <vscale x 2 x half> @test_vluxseg2_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vluxseg2.mask.nxv2f16.nxv2i16(<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -11988,12 +11843,11 @@ entry:
 define <vscale x 2 x half> @test_vluxseg3_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vluxseg3.mask.nxv2f16.nxv2i32(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -12020,12 +11874,11 @@ entry:
 define <vscale x 2 x half> @test_vluxseg3_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vluxseg3.mask.nxv2f16.nxv2i8(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -12052,12 +11905,11 @@ entry:
 define <vscale x 2 x half> @test_vluxseg3_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vluxseg3.mask.nxv2f16.nxv2i16(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl, i32 1)
@@ -12085,9 +11937,9 @@ define <vscale x 2 x half> @test_vluxseg4_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12118,9 +11970,9 @@ define <vscale x 2 x half> @test_vluxseg4_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12151,9 +12003,9 @@ define <vscale x 2 x half> @test_vluxseg4_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12184,10 +12036,10 @@ define <vscale x 2 x half> @test_vluxseg5_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12218,10 +12070,10 @@ define <vscale x 2 x half> @test_vluxseg5_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12252,10 +12104,10 @@ define <vscale x 2 x half> @test_vluxseg5_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12286,11 +12138,11 @@ define <vscale x 2 x half> @test_vluxseg6_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12321,11 +12173,11 @@ define <vscale x 2 x half> @test_vluxseg6_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12356,11 +12208,11 @@ define <vscale x 2 x half> @test_vluxseg6_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12391,12 +12243,12 @@ define <vscale x 2 x half> @test_vluxseg7_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12427,12 +12279,12 @@ define <vscale x 2 x half> @test_vluxseg7_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12463,12 +12315,12 @@ define <vscale x 2 x half> @test_vluxseg7_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12499,13 +12351,13 @@ define <vscale x 2 x half> @test_vluxseg8_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12536,13 +12388,13 @@ define <vscale x 2 x half> @test_vluxseg8_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12573,13 +12425,13 @@ define <vscale x 2 x half> @test_vluxseg8_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12612,7 +12464,6 @@ define <vscale x 4 x float> @test_vluxseg2_mask_nxv4f32_nxv4i16(<vscale x 4 x fl
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vluxseg2.mask.nxv4f32.nxv4i16(<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -12642,7 +12493,6 @@ define <vscale x 4 x float> @test_vluxseg2_mask_nxv4f32_nxv4i8(<vscale x 4 x flo
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vluxseg2.mask.nxv4f32.nxv4i8(<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -12672,7 +12522,6 @@ define <vscale x 4 x float> @test_vluxseg2_mask_nxv4f32_nxv4i32(<vscale x 4 x fl
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vluxseg2.mask.nxv4f32.nxv4i32(<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -12699,12 +12548,11 @@ entry:
 define <vscale x 4 x float> @test_vluxseg3_mask_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vluxseg3.mask.nxv4f32.nxv4i16(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -12731,12 +12579,11 @@ entry:
 define <vscale x 4 x float> @test_vluxseg3_mask_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vluxseg3.mask.nxv4f32.nxv4i8(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -12763,12 +12610,11 @@ entry:
 define <vscale x 4 x float> @test_vluxseg3_mask_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei32.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vluxseg3.mask.nxv4f32.nxv4i32(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl, i32 1)
@@ -12796,9 +12642,9 @@ define <vscale x 4 x float> @test_vluxseg4_mask_nxv4f32_nxv4i16(<vscale x 4 x fl
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -12829,9 +12675,9 @@ define <vscale x 4 x float> @test_vluxseg4_mask_nxv4f32_nxv4i8(<vscale x 4 x flo
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -12862,9 +12708,9 @@ define <vscale x 4 x float> @test_vluxseg4_mask_nxv4f32_nxv4i32(<vscale x 4 x fl
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
diff --git a/llvm/test/CodeGen/RISCV/rvv/vluxseg-rv64.ll b/llvm/test/CodeGen/RISCV/rvv/vluxseg-rv64.ll
index 0072cfef8ae92..c8a419b672bff 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vluxseg-rv64.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vluxseg-rv64.ll
@@ -24,7 +24,6 @@ define <vscale x 16 x i16> @test_vluxseg2_mask_nxv16i16_nxv16i16(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vluxseg2.mask.nxv16i16.nxv16i16(<vscale x 16 x i16> %val,<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -54,7 +53,6 @@ define <vscale x 16 x i16> @test_vluxseg2_mask_nxv16i16_nxv16i8(<vscale x 16 x i
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vluxseg2.mask.nxv16i16.nxv16i8(<vscale x 16 x i16> %val,<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -84,7 +82,6 @@ define <vscale x 16 x i16> @test_vluxseg2_mask_nxv16i16_nxv16i32(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v4, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vluxseg2.mask.nxv16i16.nxv16i32(<vscale x 16 x i16> %val,<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -114,7 +111,6 @@ define <vscale x 4 x i32> @test_vluxseg2_mask_nxv4i32_nxv4i32(<vscale x 4 x i32>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vluxseg2.mask.nxv4i32.nxv4i32(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -144,7 +140,6 @@ define <vscale x 4 x i32> @test_vluxseg2_mask_nxv4i32_nxv4i8(<vscale x 4 x i32>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vluxseg2.mask.nxv4i32.nxv4i8(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -174,7 +169,6 @@ define <vscale x 4 x i32> @test_vluxseg2_mask_nxv4i32_nxv4i64(<vscale x 4 x i32>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vluxseg2.mask.nxv4i32.nxv4i64(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -204,7 +198,6 @@ define <vscale x 4 x i32> @test_vluxseg2_mask_nxv4i32_nxv4i16(<vscale x 4 x i32>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vluxseg2.mask.nxv4i32.nxv4i16(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -231,12 +224,11 @@ entry:
 define <vscale x 4 x i32> @test_vluxseg3_mask_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei32.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vluxseg3.mask.nxv4i32.nxv4i32(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -263,12 +255,11 @@ entry:
 define <vscale x 4 x i32> @test_vluxseg3_mask_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vluxseg3.mask.nxv4i32.nxv4i8(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -296,10 +287,9 @@ define <vscale x 4 x i32> @test_vluxseg3_mask_nxv4i32_nxv4i64(<vscale x 4 x i32>
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg3ei64.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vluxseg3.mask.nxv4i32.nxv4i64(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -326,12 +316,11 @@ entry:
 define <vscale x 4 x i32> @test_vluxseg3_mask_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vluxseg3.mask.nxv4i32.nxv4i16(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -359,9 +348,9 @@ define <vscale x 4 x i32> @test_vluxseg4_mask_nxv4i32_nxv4i32(<vscale x 4 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -392,9 +381,9 @@ define <vscale x 4 x i32> @test_vluxseg4_mask_nxv4i32_nxv4i8(<vscale x 4 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -424,13 +413,12 @@ entry:
 define <vscale x 4 x i32> @test_vluxseg4_mask_nxv4i32_nxv4i64(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vluxseg4ei64.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v18
+; CHECK-NEXT:    vluxseg4ei64.v v6, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>,<vscale x 4 x i32>} @llvm.riscv.vluxseg4.mask.nxv4i32.nxv4i64(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -458,9 +446,9 @@ define <vscale x 4 x i32> @test_vluxseg4_mask_nxv4i32_nxv4i16(<vscale x 4 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -493,7 +481,6 @@ define <vscale x 16 x i8> @test_vluxseg2_mask_nxv16i8_nxv16i16(<vscale x 16 x i8
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vluxseg2.mask.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -523,7 +510,6 @@ define <vscale x 16 x i8> @test_vluxseg2_mask_nxv16i8_nxv16i8(<vscale x 16 x i8>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vluxseg2.mask.nxv16i8.nxv16i8(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -553,7 +539,6 @@ define <vscale x 16 x i8> @test_vluxseg2_mask_nxv16i8_nxv16i32(<vscale x 16 x i8
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vluxseg2.mask.nxv16i8.nxv16i32(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -581,10 +566,9 @@ define <vscale x 16 x i8> @test_vluxseg3_mask_nxv16i8_nxv16i16(<vscale x 16 x i8
 ; CHECK-LABEL: test_vluxseg3_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vluxseg3ei16.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vluxseg3.mask.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -611,12 +595,11 @@ entry:
 define <vscale x 16 x i8> @test_vluxseg3_mask_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, i64 %vl, <vscale x 16 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vluxseg3.mask.nxv16i8.nxv16i8(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -644,10 +627,9 @@ define <vscale x 16 x i8> @test_vluxseg3_mask_nxv16i8_nxv16i32(<vscale x 16 x i8
 ; CHECK-LABEL: test_vluxseg3_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vluxseg3ei32.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vluxseg3.mask.nxv16i8.nxv16i32(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -674,13 +656,12 @@ entry:
 define <vscale x 16 x i8> @test_vluxseg4_mask_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, i64 %vl, <vscale x 16 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
-; CHECK-NEXT:    vluxseg4ei16.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v18
+; CHECK-NEXT:    vluxseg4ei16.v v6, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vluxseg4.mask.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -708,9 +689,9 @@ define <vscale x 16 x i8> @test_vluxseg4_mask_nxv16i8_nxv16i8(<vscale x 16 x i8>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -741,11 +722,10 @@ define <vscale x 16 x i8> @test_vluxseg4_mask_nxv16i8_nxv16i32(<vscale x 16 x i8
 ; CHECK-LABEL: test_vluxseg4_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>,<vscale x 16 x i8>} @llvm.riscv.vluxseg4.mask.nxv16i8.nxv16i32(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -775,7 +755,6 @@ define <vscale x 1 x i64> @test_vluxseg2_mask_nxv1i64_nxv1i64(<vscale x 1 x i64>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vluxseg2.mask.nxv1i64.nxv1i64(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -805,7 +784,6 @@ define <vscale x 1 x i64> @test_vluxseg2_mask_nxv1i64_nxv1i32(<vscale x 1 x i64>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vluxseg2.mask.nxv1i64.nxv1i32(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -835,7 +813,6 @@ define <vscale x 1 x i64> @test_vluxseg2_mask_nxv1i64_nxv1i16(<vscale x 1 x i64>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vluxseg2.mask.nxv1i64.nxv1i16(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -865,7 +842,6 @@ define <vscale x 1 x i64> @test_vluxseg2_mask_nxv1i64_nxv1i8(<vscale x 1 x i64>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vluxseg2.mask.nxv1i64.nxv1i8(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -892,12 +868,11 @@ entry:
 define <vscale x 1 x i64> @test_vluxseg3_mask_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei64.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei64.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vluxseg3.mask.nxv1i64.nxv1i64(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -924,12 +899,11 @@ entry:
 define <vscale x 1 x i64> @test_vluxseg3_mask_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vluxseg3.mask.nxv1i64.nxv1i32(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -956,12 +930,11 @@ entry:
 define <vscale x 1 x i64> @test_vluxseg3_mask_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vluxseg3.mask.nxv1i64.nxv1i16(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -988,12 +961,11 @@ entry:
 define <vscale x 1 x i64> @test_vluxseg3_mask_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i64>,<vscale x 1 x i64>,<vscale x 1 x i64>} @llvm.riscv.vluxseg3.mask.nxv1i64.nxv1i8(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1021,9 +993,9 @@ define <vscale x 1 x i64> @test_vluxseg4_mask_nxv1i64_nxv1i64(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1054,9 +1026,9 @@ define <vscale x 1 x i64> @test_vluxseg4_mask_nxv1i64_nxv1i32(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1087,9 +1059,9 @@ define <vscale x 1 x i64> @test_vluxseg4_mask_nxv1i64_nxv1i16(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1120,9 +1092,9 @@ define <vscale x 1 x i64> @test_vluxseg4_mask_nxv1i64_nxv1i8(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1153,10 +1125,10 @@ define <vscale x 1 x i64> @test_vluxseg5_mask_nxv1i64_nxv1i64(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1187,10 +1159,10 @@ define <vscale x 1 x i64> @test_vluxseg5_mask_nxv1i64_nxv1i32(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1221,10 +1193,10 @@ define <vscale x 1 x i64> @test_vluxseg5_mask_nxv1i64_nxv1i16(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1255,10 +1227,10 @@ define <vscale x 1 x i64> @test_vluxseg5_mask_nxv1i64_nxv1i8(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1289,11 +1261,11 @@ define <vscale x 1 x i64> @test_vluxseg6_mask_nxv1i64_nxv1i64(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1324,11 +1296,11 @@ define <vscale x 1 x i64> @test_vluxseg6_mask_nxv1i64_nxv1i32(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1359,11 +1331,11 @@ define <vscale x 1 x i64> @test_vluxseg6_mask_nxv1i64_nxv1i16(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1394,11 +1366,11 @@ define <vscale x 1 x i64> @test_vluxseg6_mask_nxv1i64_nxv1i8(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1429,12 +1401,12 @@ define <vscale x 1 x i64> @test_vluxseg7_mask_nxv1i64_nxv1i64(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1465,12 +1437,12 @@ define <vscale x 1 x i64> @test_vluxseg7_mask_nxv1i64_nxv1i32(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1501,12 +1473,12 @@ define <vscale x 1 x i64> @test_vluxseg7_mask_nxv1i64_nxv1i16(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1537,12 +1509,12 @@ define <vscale x 1 x i64> @test_vluxseg7_mask_nxv1i64_nxv1i8(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1573,13 +1545,13 @@ define <vscale x 1 x i64> @test_vluxseg8_mask_nxv1i64_nxv1i64(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1610,13 +1582,13 @@ define <vscale x 1 x i64> @test_vluxseg8_mask_nxv1i64_nxv1i32(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1647,13 +1619,13 @@ define <vscale x 1 x i64> @test_vluxseg8_mask_nxv1i64_nxv1i16(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1684,13 +1656,13 @@ define <vscale x 1 x i64> @test_vluxseg8_mask_nxv1i64_nxv1i8(<vscale x 1 x i64>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -1723,7 +1695,6 @@ define <vscale x 1 x i32> @test_vluxseg2_mask_nxv1i32_nxv1i64(<vscale x 1 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vluxseg2.mask.nxv1i32.nxv1i64(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1753,7 +1724,6 @@ define <vscale x 1 x i32> @test_vluxseg2_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vluxseg2.mask.nxv1i32.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1783,7 +1753,6 @@ define <vscale x 1 x i32> @test_vluxseg2_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vluxseg2.mask.nxv1i32.nxv1i16(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1813,7 +1782,6 @@ define <vscale x 1 x i32> @test_vluxseg2_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vluxseg2.mask.nxv1i32.nxv1i8(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1840,12 +1808,11 @@ entry:
 define <vscale x 1 x i32> @test_vluxseg3_mask_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei64.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei64.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vluxseg3.mask.nxv1i32.nxv1i64(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1872,12 +1839,11 @@ entry:
 define <vscale x 1 x i32> @test_vluxseg3_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vluxseg3.mask.nxv1i32.nxv1i32(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1904,12 +1870,11 @@ entry:
 define <vscale x 1 x i32> @test_vluxseg3_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vluxseg3.mask.nxv1i32.nxv1i16(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1936,12 +1901,11 @@ entry:
 define <vscale x 1 x i32> @test_vluxseg3_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i32>,<vscale x 1 x i32>,<vscale x 1 x i32>} @llvm.riscv.vluxseg3.mask.nxv1i32.nxv1i8(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -1969,9 +1933,9 @@ define <vscale x 1 x i32> @test_vluxseg4_mask_nxv1i32_nxv1i64(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2002,9 +1966,9 @@ define <vscale x 1 x i32> @test_vluxseg4_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2035,9 +1999,9 @@ define <vscale x 1 x i32> @test_vluxseg4_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2068,9 +2032,9 @@ define <vscale x 1 x i32> @test_vluxseg4_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2101,10 +2065,10 @@ define <vscale x 1 x i32> @test_vluxseg5_mask_nxv1i32_nxv1i64(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2135,10 +2099,10 @@ define <vscale x 1 x i32> @test_vluxseg5_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2169,10 +2133,10 @@ define <vscale x 1 x i32> @test_vluxseg5_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2203,10 +2167,10 @@ define <vscale x 1 x i32> @test_vluxseg5_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2237,11 +2201,11 @@ define <vscale x 1 x i32> @test_vluxseg6_mask_nxv1i32_nxv1i64(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2272,11 +2236,11 @@ define <vscale x 1 x i32> @test_vluxseg6_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2307,11 +2271,11 @@ define <vscale x 1 x i32> @test_vluxseg6_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2342,11 +2306,11 @@ define <vscale x 1 x i32> @test_vluxseg6_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2377,12 +2341,12 @@ define <vscale x 1 x i32> @test_vluxseg7_mask_nxv1i32_nxv1i64(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2413,12 +2377,12 @@ define <vscale x 1 x i32> @test_vluxseg7_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2449,12 +2413,12 @@ define <vscale x 1 x i32> @test_vluxseg7_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2485,12 +2449,12 @@ define <vscale x 1 x i32> @test_vluxseg7_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2521,13 +2485,13 @@ define <vscale x 1 x i32> @test_vluxseg8_mask_nxv1i32_nxv1i64(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2558,13 +2522,13 @@ define <vscale x 1 x i32> @test_vluxseg8_mask_nxv1i32_nxv1i32(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2595,13 +2559,13 @@ define <vscale x 1 x i32> @test_vluxseg8_mask_nxv1i32_nxv1i16(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2632,13 +2596,13 @@ define <vscale x 1 x i32> @test_vluxseg8_mask_nxv1i32_nxv1i8(<vscale x 1 x i32>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -2671,7 +2635,6 @@ define <vscale x 8 x i16> @test_vluxseg2_mask_nxv8i16_nxv8i16(<vscale x 8 x i16>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vluxseg2.mask.nxv8i16.nxv8i16(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2701,7 +2664,6 @@ define <vscale x 8 x i16> @test_vluxseg2_mask_nxv8i16_nxv8i8(<vscale x 8 x i16>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vluxseg2.mask.nxv8i16.nxv8i8(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2731,7 +2693,6 @@ define <vscale x 8 x i16> @test_vluxseg2_mask_nxv8i16_nxv8i64(<vscale x 8 x i16>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vluxseg2.mask.nxv8i16.nxv8i64(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2761,7 +2722,6 @@ define <vscale x 8 x i16> @test_vluxseg2_mask_nxv8i16_nxv8i32(<vscale x 8 x i16>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vluxseg2.mask.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2788,12 +2748,11 @@ entry:
 define <vscale x 8 x i16> @test_vluxseg3_mask_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vluxseg3.mask.nxv8i16.nxv8i16(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2820,12 +2779,11 @@ entry:
 define <vscale x 8 x i16> @test_vluxseg3_mask_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, i64 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vluxseg3.mask.nxv8i16.nxv8i8(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2853,10 +2811,9 @@ define <vscale x 8 x i16> @test_vluxseg3_mask_nxv8i16_nxv8i64(<vscale x 8 x i16>
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8i16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg3ei64.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vluxseg3.mask.nxv8i16.nxv8i64(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2884,10 +2841,9 @@ define <vscale x 8 x i16> @test_vluxseg3_mask_nxv8i16_nxv8i32(<vscale x 8 x i16>
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg3ei32.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vluxseg3.mask.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -2915,9 +2871,9 @@ define <vscale x 8 x i16> @test_vluxseg4_mask_nxv8i16_nxv8i16(<vscale x 8 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -2948,9 +2904,9 @@ define <vscale x 8 x i16> @test_vluxseg4_mask_nxv8i16_nxv8i8(<vscale x 8 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -2981,11 +2937,10 @@ define <vscale x 8 x i16> @test_vluxseg4_mask_nxv8i16_nxv8i64(<vscale x 8 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8i16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei64.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vluxseg4.mask.nxv8i16.nxv8i64(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -3012,13 +2967,12 @@ entry:
 define <vscale x 8 x i16> @test_vluxseg4_mask_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vluxseg4ei32.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v18
+; CHECK-NEXT:    vluxseg4ei32.v v6, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>,<vscale x 8 x i16>} @llvm.riscv.vluxseg4.mask.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -3048,7 +3002,6 @@ define <vscale x 4 x i8> @test_vluxseg2_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg2.mask.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3078,7 +3031,6 @@ define <vscale x 4 x i8> @test_vluxseg2_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg2.mask.nxv4i8.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3108,7 +3060,6 @@ define <vscale x 4 x i8> @test_vluxseg2_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg2.mask.nxv4i8.nxv4i64(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3138,7 +3089,6 @@ define <vscale x 4 x i8> @test_vluxseg2_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg2.mask.nxv4i8.nxv4i16(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3166,10 +3116,9 @@ define <vscale x 4 x i8> @test_vluxseg3_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg3.mask.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3196,12 +3145,11 @@ entry:
 define <vscale x 4 x i8> @test_vluxseg3_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg3.mask.nxv4i8.nxv4i8(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3229,10 +3177,9 @@ define <vscale x 4 x i8> @test_vluxseg3_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg3ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg3.mask.nxv4i8.nxv4i64(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3259,12 +3206,11 @@ entry:
 define <vscale x 4 x i8> @test_vluxseg3_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg3.mask.nxv4i8.nxv4i16(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3291,13 +3237,12 @@ entry:
 define <vscale x 4 x i8> @test_vluxseg4_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
-; CHECK-NEXT:    vluxseg4ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vluxseg4ei32.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg4.mask.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3325,9 +3270,9 @@ define <vscale x 4 x i8> @test_vluxseg4_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3358,11 +3303,10 @@ define <vscale x 4 x i8> @test_vluxseg4_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg4.mask.nxv4i8.nxv4i64(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3390,9 +3334,9 @@ define <vscale x 4 x i8> @test_vluxseg4_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3423,10 +3367,10 @@ define <vscale x 4 x i8> @test_vluxseg5_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -3457,10 +3401,10 @@ define <vscale x 4 x i8> @test_vluxseg5_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3491,12 +3435,11 @@ define <vscale x 4 x i8> @test_vluxseg5_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg5.mask.nxv4i8.nxv4i64(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3524,10 +3467,10 @@ define <vscale x 4 x i8> @test_vluxseg5_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3558,11 +3501,11 @@ define <vscale x 4 x i8> @test_vluxseg6_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -3593,11 +3536,11 @@ define <vscale x 4 x i8> @test_vluxseg6_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3627,15 +3570,14 @@ entry:
 define <vscale x 4 x i8> @test_vluxseg6_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
-; CHECK-NEXT:    vluxseg6ei64.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v17
+; CHECK-NEXT:    vluxseg6ei64.v v7, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>,<vscale x 4 x i8>} @llvm.riscv.vluxseg6.mask.nxv4i8.nxv4i64(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -3663,11 +3605,11 @@ define <vscale x 4 x i8> @test_vluxseg6_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3698,12 +3640,12 @@ define <vscale x 4 x i8> @test_vluxseg7_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -3734,12 +3676,12 @@ define <vscale x 4 x i8> @test_vluxseg7_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3770,12 +3712,12 @@ define <vscale x 4 x i8> @test_vluxseg7_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -3806,12 +3748,12 @@ define <vscale x 4 x i8> @test_vluxseg7_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3842,13 +3784,13 @@ define <vscale x 4 x i8> @test_vluxseg8_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -3879,13 +3821,13 @@ define <vscale x 4 x i8> @test_vluxseg8_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %va
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3916,13 +3858,13 @@ define <vscale x 4 x i8> @test_vluxseg8_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -3953,13 +3895,13 @@ define <vscale x 4 x i8> @test_vluxseg8_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -3992,7 +3934,6 @@ define <vscale x 1 x i16> @test_vluxseg2_mask_nxv1i16_nxv1i64(<vscale x 1 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vluxseg2.mask.nxv1i16.nxv1i64(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4022,7 +3963,6 @@ define <vscale x 1 x i16> @test_vluxseg2_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vluxseg2.mask.nxv1i16.nxv1i32(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4052,7 +3992,6 @@ define <vscale x 1 x i16> @test_vluxseg2_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vluxseg2.mask.nxv1i16.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4082,7 +4021,6 @@ define <vscale x 1 x i16> @test_vluxseg2_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vluxseg2.mask.nxv1i16.nxv1i8(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4109,12 +4047,11 @@ entry:
 define <vscale x 1 x i16> @test_vluxseg3_mask_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei64.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei64.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vluxseg3.mask.nxv1i16.nxv1i64(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4141,12 +4078,11 @@ entry:
 define <vscale x 1 x i16> @test_vluxseg3_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vluxseg3.mask.nxv1i16.nxv1i32(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4173,12 +4109,11 @@ entry:
 define <vscale x 1 x i16> @test_vluxseg3_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vluxseg3.mask.nxv1i16.nxv1i16(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4205,12 +4140,11 @@ entry:
 define <vscale x 1 x i16> @test_vluxseg3_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i16>,<vscale x 1 x i16>,<vscale x 1 x i16>} @llvm.riscv.vluxseg3.mask.nxv1i16.nxv1i8(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -4238,9 +4172,9 @@ define <vscale x 1 x i16> @test_vluxseg4_mask_nxv1i16_nxv1i64(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4271,9 +4205,9 @@ define <vscale x 1 x i16> @test_vluxseg4_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4304,9 +4238,9 @@ define <vscale x 1 x i16> @test_vluxseg4_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4337,9 +4271,9 @@ define <vscale x 1 x i16> @test_vluxseg4_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4370,10 +4304,10 @@ define <vscale x 1 x i16> @test_vluxseg5_mask_nxv1i16_nxv1i64(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4404,10 +4338,10 @@ define <vscale x 1 x i16> @test_vluxseg5_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4438,10 +4372,10 @@ define <vscale x 1 x i16> @test_vluxseg5_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4472,10 +4406,10 @@ define <vscale x 1 x i16> @test_vluxseg5_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4506,11 +4440,11 @@ define <vscale x 1 x i16> @test_vluxseg6_mask_nxv1i16_nxv1i64(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4541,11 +4475,11 @@ define <vscale x 1 x i16> @test_vluxseg6_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4576,11 +4510,11 @@ define <vscale x 1 x i16> @test_vluxseg6_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4611,11 +4545,11 @@ define <vscale x 1 x i16> @test_vluxseg6_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4646,12 +4580,12 @@ define <vscale x 1 x i16> @test_vluxseg7_mask_nxv1i16_nxv1i64(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4682,12 +4616,12 @@ define <vscale x 1 x i16> @test_vluxseg7_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4718,12 +4652,12 @@ define <vscale x 1 x i16> @test_vluxseg7_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4754,12 +4688,12 @@ define <vscale x 1 x i16> @test_vluxseg7_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4790,13 +4724,13 @@ define <vscale x 1 x i16> @test_vluxseg8_mask_nxv1i16_nxv1i64(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4827,13 +4761,13 @@ define <vscale x 1 x i16> @test_vluxseg8_mask_nxv1i16_nxv1i32(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4864,13 +4798,13 @@ define <vscale x 1 x i16> @test_vluxseg8_mask_nxv1i16_nxv1i16(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4901,13 +4835,13 @@ define <vscale x 1 x i16> @test_vluxseg8_mask_nxv1i16_nxv1i8(<vscale x 1 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -4940,7 +4874,6 @@ define <vscale x 2 x i32> @test_vluxseg2_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vluxseg2.mask.nxv2i32.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -4970,7 +4903,6 @@ define <vscale x 2 x i32> @test_vluxseg2_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vluxseg2.mask.nxv2i32.nxv2i8(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5000,7 +4932,6 @@ define <vscale x 2 x i32> @test_vluxseg2_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vluxseg2.mask.nxv2i32.nxv2i16(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5030,7 +4961,6 @@ define <vscale x 2 x i32> @test_vluxseg2_mask_nxv2i32_nxv2i64(<vscale x 2 x i32>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vluxseg2.mask.nxv2i32.nxv2i64(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5057,12 +4987,11 @@ entry:
 define <vscale x 2 x i32> @test_vluxseg3_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vluxseg3.mask.nxv2i32.nxv2i32(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5089,12 +5018,11 @@ entry:
 define <vscale x 2 x i32> @test_vluxseg3_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vluxseg3.mask.nxv2i32.nxv2i8(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5121,12 +5049,11 @@ entry:
 define <vscale x 2 x i32> @test_vluxseg3_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vluxseg3.mask.nxv2i32.nxv2i16(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5154,10 +5081,9 @@ define <vscale x 2 x i32> @test_vluxseg3_mask_nxv2i32_nxv2i64(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg3ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vluxseg3.mask.nxv2i32.nxv2i64(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5185,9 +5111,9 @@ define <vscale x 2 x i32> @test_vluxseg4_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5218,9 +5144,9 @@ define <vscale x 2 x i32> @test_vluxseg4_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5251,9 +5177,9 @@ define <vscale x 2 x i32> @test_vluxseg4_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5283,13 +5209,12 @@ entry:
 define <vscale x 2 x i32> @test_vluxseg4_mask_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vluxseg4ei64.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vluxseg4ei64.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>,<vscale x 2 x i32>} @llvm.riscv.vluxseg4.mask.nxv2i32.nxv2i64(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -5317,10 +5242,10 @@ define <vscale x 2 x i32> @test_vluxseg5_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5351,10 +5276,10 @@ define <vscale x 2 x i32> @test_vluxseg5_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5385,10 +5310,10 @@ define <vscale x 2 x i32> @test_vluxseg5_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5419,10 +5344,10 @@ define <vscale x 2 x i32> @test_vluxseg5_mask_nxv2i32_nxv2i64(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -5453,11 +5378,11 @@ define <vscale x 2 x i32> @test_vluxseg6_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5488,11 +5413,11 @@ define <vscale x 2 x i32> @test_vluxseg6_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5523,11 +5448,11 @@ define <vscale x 2 x i32> @test_vluxseg6_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5558,11 +5483,11 @@ define <vscale x 2 x i32> @test_vluxseg6_mask_nxv2i32_nxv2i64(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -5593,12 +5518,12 @@ define <vscale x 2 x i32> @test_vluxseg7_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5629,12 +5554,12 @@ define <vscale x 2 x i32> @test_vluxseg7_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5665,12 +5590,12 @@ define <vscale x 2 x i32> @test_vluxseg7_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5701,12 +5626,12 @@ define <vscale x 2 x i32> @test_vluxseg7_mask_nxv2i32_nxv2i64(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -5737,13 +5662,13 @@ define <vscale x 2 x i32> @test_vluxseg8_mask_nxv2i32_nxv2i32(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5774,13 +5699,13 @@ define <vscale x 2 x i32> @test_vluxseg8_mask_nxv2i32_nxv2i8(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5811,13 +5736,13 @@ define <vscale x 2 x i32> @test_vluxseg8_mask_nxv2i32_nxv2i16(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -5848,13 +5773,13 @@ define <vscale x 2 x i32> @test_vluxseg8_mask_nxv2i32_nxv2i64(<vscale x 2 x i32>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -5887,7 +5812,6 @@ define <vscale x 8 x i8> @test_vluxseg2_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg2.mask.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -5917,7 +5841,6 @@ define <vscale x 8 x i8> @test_vluxseg2_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg2.mask.nxv8i8.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -5947,7 +5870,6 @@ define <vscale x 8 x i8> @test_vluxseg2_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v7, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg2.mask.nxv8i8.nxv8i64(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -5977,7 +5899,6 @@ define <vscale x 8 x i8> @test_vluxseg2_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg2.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6005,10 +5926,9 @@ define <vscale x 8 x i8> @test_vluxseg3_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg3.mask.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6035,12 +5955,11 @@ entry:
 define <vscale x 8 x i8> @test_vluxseg3_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, i64 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg3.mask.nxv8i8.nxv8i8(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6068,10 +5987,9 @@ define <vscale x 8 x i8> @test_vluxseg3_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg3ei64.v v7, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg3.mask.nxv8i8.nxv8i64(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6099,10 +6017,9 @@ define <vscale x 8 x i8> @test_vluxseg3_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg3.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6129,13 +6046,12 @@ entry:
 define <vscale x 8 x i8> @test_vluxseg4_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
-; CHECK-NEXT:    vluxseg4ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vluxseg4ei16.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg4.mask.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6163,9 +6079,9 @@ define <vscale x 8 x i8> @test_vluxseg4_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6196,11 +6112,10 @@ define <vscale x 8 x i8> @test_vluxseg4_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei64.v v7, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg4.mask.nxv8i8.nxv8i64(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6228,11 +6143,10 @@ define <vscale x 8 x i8> @test_vluxseg4_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg4.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6260,10 +6174,10 @@ define <vscale x 8 x i8> @test_vluxseg5_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -6294,10 +6208,10 @@ define <vscale x 8 x i8> @test_vluxseg5_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vluxseg5_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6328,12 +6242,11 @@ define <vscale x 8 x i8> @test_vluxseg5_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei64.v v7, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg5.mask.nxv8i8.nxv8i64(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6361,12 +6274,11 @@ define <vscale x 8 x i8> @test_vluxseg5_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg5.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6394,11 +6306,11 @@ define <vscale x 8 x i8> @test_vluxseg6_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg6_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -6429,11 +6341,11 @@ define <vscale x 8 x i8> @test_vluxseg6_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vluxseg6_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6464,13 +6376,12 @@ define <vscale x 8 x i8> @test_vluxseg6_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg6_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei64.v v7, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg6.mask.nxv8i8.nxv8i64(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6497,15 +6408,14 @@ entry:
 define <vscale x 8 x i8> @test_vluxseg6_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg6_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
-; CHECK-NEXT:    vluxseg6ei32.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v17
+; CHECK-NEXT:    vluxseg6ei32.v v7, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg6.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6533,12 +6443,12 @@ define <vscale x 8 x i8> @test_vluxseg7_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -6569,12 +6479,12 @@ define <vscale x 8 x i8> @test_vluxseg7_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vluxseg7_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6605,14 +6515,13 @@ define <vscale x 8 x i8> @test_vluxseg7_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei64.v v7, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg7.mask.nxv8i8.nxv8i64(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6640,12 +6549,12 @@ define <vscale x 8 x i8> @test_vluxseg7_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -6676,13 +6585,13 @@ define <vscale x 8 x i8> @test_vluxseg8_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -6713,13 +6622,13 @@ define <vscale x 8 x i8> @test_vluxseg8_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %va
 ; CHECK-LABEL: test_vluxseg8_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -6750,15 +6659,14 @@ define <vscale x 8 x i8> @test_vluxseg8_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
-; CHECK-NEXT:    vmv1r.v v12, v7
-; CHECK-NEXT:    vmv1r.v v13, v7
-; CHECK-NEXT:    vmv1r.v v14, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei64.v v7, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>,<vscale x 8 x i8>} @llvm.riscv.vluxseg8.mask.nxv8i8.nxv8i64(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -6786,13 +6694,13 @@ define <vscale x 8 x i8> @test_vluxseg8_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -6825,7 +6733,6 @@ define <vscale x 4 x i64> @test_vluxseg2_mask_nxv4i64_nxv4i32(<vscale x 4 x i64>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i64>,<vscale x 4 x i64>} @llvm.riscv.vluxseg2.mask.nxv4i64.nxv4i32(<vscale x 4 x i64> %val,<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -6855,7 +6762,6 @@ define <vscale x 4 x i64> @test_vluxseg2_mask_nxv4i64_nxv4i8(<vscale x 4 x i64>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i64>,<vscale x 4 x i64>} @llvm.riscv.vluxseg2.mask.nxv4i64.nxv4i8(<vscale x 4 x i64> %val,<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -6885,7 +6791,6 @@ define <vscale x 4 x i64> @test_vluxseg2_mask_nxv4i64_nxv4i64(<vscale x 4 x i64>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i64>,<vscale x 4 x i64>} @llvm.riscv.vluxseg2.mask.nxv4i64.nxv4i64(<vscale x 4 x i64> %val,<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -6915,7 +6820,6 @@ define <vscale x 4 x i64> @test_vluxseg2_mask_nxv4i64_nxv4i16(<vscale x 4 x i64>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i64>,<vscale x 4 x i64>} @llvm.riscv.vluxseg2.mask.nxv4i64.nxv4i16(<vscale x 4 x i64> %val,<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -6945,7 +6849,6 @@ define <vscale x 4 x i16> @test_vluxseg2_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg2.mask.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -6975,7 +6878,6 @@ define <vscale x 4 x i16> @test_vluxseg2_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg2.mask.nxv4i16.nxv4i8(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7005,7 +6907,6 @@ define <vscale x 4 x i16> @test_vluxseg2_mask_nxv4i16_nxv4i64(<vscale x 4 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg2.mask.nxv4i16.nxv4i64(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7035,7 +6936,6 @@ define <vscale x 4 x i16> @test_vluxseg2_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg2.mask.nxv4i16.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7063,10 +6963,9 @@ define <vscale x 4 x i16> @test_vluxseg3_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg3.mask.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7093,12 +6992,11 @@ entry:
 define <vscale x 4 x i16> @test_vluxseg3_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg3.mask.nxv4i16.nxv4i8(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7126,10 +7024,9 @@ define <vscale x 4 x i16> @test_vluxseg3_mask_nxv4i16_nxv4i64(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg3ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg3.mask.nxv4i16.nxv4i64(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7156,12 +7053,11 @@ entry:
 define <vscale x 4 x i16> @test_vluxseg3_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg3.mask.nxv4i16.nxv4i16(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7188,13 +7084,12 @@ entry:
 define <vscale x 4 x i16> @test_vluxseg4_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vluxseg4ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vluxseg4ei32.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg4.mask.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7222,9 +7117,9 @@ define <vscale x 4 x i16> @test_vluxseg4_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7255,11 +7150,10 @@ define <vscale x 4 x i16> @test_vluxseg4_mask_nxv4i16_nxv4i64(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg4.mask.nxv4i16.nxv4i64(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7287,9 +7181,9 @@ define <vscale x 4 x i16> @test_vluxseg4_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7320,10 +7214,10 @@ define <vscale x 4 x i16> @test_vluxseg5_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -7354,10 +7248,10 @@ define <vscale x 4 x i16> @test_vluxseg5_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7388,12 +7282,11 @@ define <vscale x 4 x i16> @test_vluxseg5_mask_nxv4i16_nxv4i64(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg5.mask.nxv4i16.nxv4i64(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7421,10 +7314,10 @@ define <vscale x 4 x i16> @test_vluxseg5_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7455,11 +7348,11 @@ define <vscale x 4 x i16> @test_vluxseg6_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -7490,11 +7383,11 @@ define <vscale x 4 x i16> @test_vluxseg6_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7524,15 +7417,14 @@ entry:
 define <vscale x 4 x i16> @test_vluxseg6_mask_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vluxseg6ei64.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v17
+; CHECK-NEXT:    vluxseg6ei64.v v7, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>,<vscale x 4 x i16>} @llvm.riscv.vluxseg6.mask.nxv4i16.nxv4i64(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -7560,11 +7452,11 @@ define <vscale x 4 x i16> @test_vluxseg6_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7595,12 +7487,12 @@ define <vscale x 4 x i16> @test_vluxseg7_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -7631,12 +7523,12 @@ define <vscale x 4 x i16> @test_vluxseg7_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7667,12 +7559,12 @@ define <vscale x 4 x i16> @test_vluxseg7_mask_nxv4i16_nxv4i64(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -7703,12 +7595,12 @@ define <vscale x 4 x i16> @test_vluxseg7_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7739,13 +7631,13 @@ define <vscale x 4 x i16> @test_vluxseg8_mask_nxv4i16_nxv4i32(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -7776,13 +7668,13 @@ define <vscale x 4 x i16> @test_vluxseg8_mask_nxv4i16_nxv4i8(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7813,13 +7705,13 @@ define <vscale x 4 x i16> @test_vluxseg8_mask_nxv4i16_nxv4i64(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -7850,13 +7742,13 @@ define <vscale x 4 x i16> @test_vluxseg8_mask_nxv4i16_nxv4i16(<vscale x 4 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -7889,7 +7781,6 @@ define <vscale x 1 x i8> @test_vluxseg2_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vluxseg2.mask.nxv1i8.nxv1i64(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -7919,7 +7810,6 @@ define <vscale x 1 x i8> @test_vluxseg2_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vluxseg2.mask.nxv1i8.nxv1i32(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -7949,7 +7839,6 @@ define <vscale x 1 x i8> @test_vluxseg2_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vluxseg2.mask.nxv1i8.nxv1i16(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -7979,7 +7868,6 @@ define <vscale x 1 x i8> @test_vluxseg2_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vluxseg2.mask.nxv1i8.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -8006,12 +7894,11 @@ entry:
 define <vscale x 1 x i8> @test_vluxseg3_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
-; CHECK-NEXT:    vluxseg3ei64.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei64.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vluxseg3.mask.nxv1i8.nxv1i64(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -8038,12 +7925,11 @@ entry:
 define <vscale x 1 x i8> @test_vluxseg3_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vluxseg3.mask.nxv1i8.nxv1i32(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -8070,12 +7956,11 @@ entry:
 define <vscale x 1 x i8> @test_vluxseg3_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vluxseg3.mask.nxv1i8.nxv1i16(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -8102,12 +7987,11 @@ entry:
 define <vscale x 1 x i8> @test_vluxseg3_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x i8>,<vscale x 1 x i8>,<vscale x 1 x i8>} @llvm.riscv.vluxseg3.mask.nxv1i8.nxv1i8(<vscale x 1 x i8> %val,<vscale x 1 x i8> %val,<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -8135,9 +8019,9 @@ define <vscale x 1 x i8> @test_vluxseg4_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8168,9 +8052,9 @@ define <vscale x 1 x i8> @test_vluxseg4_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8201,9 +8085,9 @@ define <vscale x 1 x i8> @test_vluxseg4_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8234,9 +8118,9 @@ define <vscale x 1 x i8> @test_vluxseg4_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8267,10 +8151,10 @@ define <vscale x 1 x i8> @test_vluxseg5_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8301,10 +8185,10 @@ define <vscale x 1 x i8> @test_vluxseg5_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8335,10 +8219,10 @@ define <vscale x 1 x i8> @test_vluxseg5_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8369,10 +8253,10 @@ define <vscale x 1 x i8> @test_vluxseg5_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8403,11 +8287,11 @@ define <vscale x 1 x i8> @test_vluxseg6_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8438,11 +8322,11 @@ define <vscale x 1 x i8> @test_vluxseg6_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8473,11 +8357,11 @@ define <vscale x 1 x i8> @test_vluxseg6_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8508,11 +8392,11 @@ define <vscale x 1 x i8> @test_vluxseg6_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8543,12 +8427,12 @@ define <vscale x 1 x i8> @test_vluxseg7_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8579,12 +8463,12 @@ define <vscale x 1 x i8> @test_vluxseg7_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8615,12 +8499,12 @@ define <vscale x 1 x i8> @test_vluxseg7_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8651,12 +8535,12 @@ define <vscale x 1 x i8> @test_vluxseg7_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8687,13 +8571,13 @@ define <vscale x 1 x i8> @test_vluxseg8_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8724,13 +8608,13 @@ define <vscale x 1 x i8> @test_vluxseg8_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8761,14 +8645,14 @@ define <vscale x 1 x i8> @test_vluxseg8_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
-; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
 ; CHECK-NEXT:    ret
@@ -8798,13 +8682,13 @@ define <vscale x 1 x i8> @test_vluxseg8_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %va
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -8837,7 +8721,6 @@ define <vscale x 2 x i8> @test_vluxseg2_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vluxseg2.mask.nxv2i8.nxv2i32(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -8867,7 +8750,6 @@ define <vscale x 2 x i8> @test_vluxseg2_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vluxseg2.mask.nxv2i8.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -8897,7 +8779,6 @@ define <vscale x 2 x i8> @test_vluxseg2_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vluxseg2.mask.nxv2i8.nxv2i16(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -8927,7 +8808,6 @@ define <vscale x 2 x i8> @test_vluxseg2_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %v
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vluxseg2.mask.nxv2i8.nxv2i64(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -8954,12 +8834,11 @@ entry:
 define <vscale x 2 x i8> @test_vluxseg3_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vluxseg3.mask.nxv2i8.nxv2i32(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -8986,12 +8865,11 @@ entry:
 define <vscale x 2 x i8> @test_vluxseg3_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vluxseg3.mask.nxv2i8.nxv2i8(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -9018,12 +8896,11 @@ entry:
 define <vscale x 2 x i8> @test_vluxseg3_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vluxseg3.mask.nxv2i8.nxv2i16(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -9051,10 +8928,9 @@ define <vscale x 2 x i8> @test_vluxseg3_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg3ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vluxseg3.mask.nxv2i8.nxv2i64(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -9082,9 +8958,9 @@ define <vscale x 2 x i8> @test_vluxseg4_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9115,9 +8991,9 @@ define <vscale x 2 x i8> @test_vluxseg4_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9148,9 +9024,9 @@ define <vscale x 2 x i8> @test_vluxseg4_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9180,13 +9056,12 @@ entry:
 define <vscale x 2 x i8> @test_vluxseg4_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
-; CHECK-NEXT:    vluxseg4ei64.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vluxseg4ei64.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>,<vscale x 2 x i8>} @llvm.riscv.vluxseg4.mask.nxv2i8.nxv2i64(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -9214,10 +9089,10 @@ define <vscale x 2 x i8> @test_vluxseg5_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9248,10 +9123,10 @@ define <vscale x 2 x i8> @test_vluxseg5_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9282,10 +9157,10 @@ define <vscale x 2 x i8> @test_vluxseg5_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9316,10 +9191,10 @@ define <vscale x 2 x i8> @test_vluxseg5_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -9350,11 +9225,11 @@ define <vscale x 2 x i8> @test_vluxseg6_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9385,11 +9260,11 @@ define <vscale x 2 x i8> @test_vluxseg6_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9420,11 +9295,11 @@ define <vscale x 2 x i8> @test_vluxseg6_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9455,11 +9330,11 @@ define <vscale x 2 x i8> @test_vluxseg6_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -9490,12 +9365,12 @@ define <vscale x 2 x i8> @test_vluxseg7_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9526,12 +9401,12 @@ define <vscale x 2 x i8> @test_vluxseg7_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9562,12 +9437,12 @@ define <vscale x 2 x i8> @test_vluxseg7_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9598,12 +9473,12 @@ define <vscale x 2 x i8> @test_vluxseg7_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -9634,13 +9509,13 @@ define <vscale x 2 x i8> @test_vluxseg8_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9671,13 +9546,13 @@ define <vscale x 2 x i8> @test_vluxseg8_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %va
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9708,13 +9583,13 @@ define <vscale x 2 x i8> @test_vluxseg8_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -9745,13 +9620,13 @@ define <vscale x 2 x i8> @test_vluxseg8_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %v
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -9784,7 +9659,6 @@ define <vscale x 8 x i32> @test_vluxseg2_mask_nxv8i32_nxv8i16(<vscale x 8 x i32>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vluxseg2.mask.nxv8i32.nxv8i16(<vscale x 8 x i32> %val,<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -9814,7 +9688,6 @@ define <vscale x 8 x i32> @test_vluxseg2_mask_nxv8i32_nxv8i8(<vscale x 8 x i32>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vluxseg2.mask.nxv8i32.nxv8i8(<vscale x 8 x i32> %val,<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -9844,7 +9717,6 @@ define <vscale x 8 x i32> @test_vluxseg2_mask_nxv8i32_nxv8i64(<vscale x 8 x i32>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v4, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vluxseg2.mask.nxv8i32.nxv8i64(<vscale x 8 x i32> %val,<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -9874,7 +9746,6 @@ define <vscale x 8 x i32> @test_vluxseg2_mask_nxv8i32_nxv8i32(<vscale x 8 x i32>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x i32>,<vscale x 8 x i32>} @llvm.riscv.vluxseg2.mask.nxv8i32.nxv8i32(<vscale x 8 x i32> %val,<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -9904,7 +9775,6 @@ define <vscale x 32 x i8> @test_vluxseg2_mask_nxv32i8_nxv32i16(<vscale x 32 x i8
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v4, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>} @llvm.riscv.vluxseg2.mask.nxv32i8.nxv32i16(<vscale x 32 x i8> %val,<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i16> %index, <vscale x 32 x i1> %mask, i64 %vl, i64 1)
@@ -9934,7 +9804,6 @@ define <vscale x 32 x i8> @test_vluxseg2_mask_nxv32i8_nxv32i8(<vscale x 32 x i8>
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 32 x i8>,<vscale x 32 x i8>} @llvm.riscv.vluxseg2.mask.nxv32i8.nxv32i8(<vscale x 32 x i8> %val,<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i8> %index, <vscale x 32 x i1> %mask, i64 %vl, i64 1)
@@ -9964,7 +9833,6 @@ define <vscale x 2 x i16> @test_vluxseg2_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vluxseg2.mask.nxv2i16.nxv2i32(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -9994,7 +9862,6 @@ define <vscale x 2 x i16> @test_vluxseg2_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vluxseg2.mask.nxv2i16.nxv2i8(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10024,7 +9891,6 @@ define <vscale x 2 x i16> @test_vluxseg2_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vluxseg2.mask.nxv2i16.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10054,7 +9920,6 @@ define <vscale x 2 x i16> @test_vluxseg2_mask_nxv2i16_nxv2i64(<vscale x 2 x i16>
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vluxseg2.mask.nxv2i16.nxv2i64(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10081,12 +9946,11 @@ entry:
 define <vscale x 2 x i16> @test_vluxseg3_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vluxseg3.mask.nxv2i16.nxv2i32(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10113,12 +9977,11 @@ entry:
 define <vscale x 2 x i16> @test_vluxseg3_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vluxseg3.mask.nxv2i16.nxv2i8(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10145,12 +10008,11 @@ entry:
 define <vscale x 2 x i16> @test_vluxseg3_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vluxseg3.mask.nxv2i16.nxv2i16(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10178,10 +10040,9 @@ define <vscale x 2 x i16> @test_vluxseg3_mask_nxv2i16_nxv2i64(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg3ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vluxseg3.mask.nxv2i16.nxv2i64(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10209,9 +10070,9 @@ define <vscale x 2 x i16> @test_vluxseg4_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10242,9 +10103,9 @@ define <vscale x 2 x i16> @test_vluxseg4_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10275,9 +10136,9 @@ define <vscale x 2 x i16> @test_vluxseg4_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10307,13 +10168,12 @@ entry:
 define <vscale x 2 x i16> @test_vluxseg4_mask_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vluxseg4ei64.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vluxseg4ei64.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>,<vscale x 2 x i16>} @llvm.riscv.vluxseg4.mask.nxv2i16.nxv2i64(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10341,10 +10201,10 @@ define <vscale x 2 x i16> @test_vluxseg5_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10375,10 +10235,10 @@ define <vscale x 2 x i16> @test_vluxseg5_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10409,10 +10269,10 @@ define <vscale x 2 x i16> @test_vluxseg5_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10443,10 +10303,10 @@ define <vscale x 2 x i16> @test_vluxseg5_mask_nxv2i16_nxv2i64(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -10477,11 +10337,11 @@ define <vscale x 2 x i16> @test_vluxseg6_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10512,11 +10372,11 @@ define <vscale x 2 x i16> @test_vluxseg6_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10547,11 +10407,11 @@ define <vscale x 2 x i16> @test_vluxseg6_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10582,11 +10442,11 @@ define <vscale x 2 x i16> @test_vluxseg6_mask_nxv2i16_nxv2i64(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -10617,12 +10477,12 @@ define <vscale x 2 x i16> @test_vluxseg7_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10653,12 +10513,12 @@ define <vscale x 2 x i16> @test_vluxseg7_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10689,12 +10549,12 @@ define <vscale x 2 x i16> @test_vluxseg7_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10725,12 +10585,12 @@ define <vscale x 2 x i16> @test_vluxseg7_mask_nxv2i16_nxv2i64(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -10761,13 +10621,13 @@ define <vscale x 2 x i16> @test_vluxseg8_mask_nxv2i16_nxv2i32(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10798,13 +10658,13 @@ define <vscale x 2 x i16> @test_vluxseg8_mask_nxv2i16_nxv2i8(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10835,13 +10695,13 @@ define <vscale x 2 x i16> @test_vluxseg8_mask_nxv2i16_nxv2i16(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -10872,13 +10732,13 @@ define <vscale x 2 x i16> @test_vluxseg8_mask_nxv2i16_nxv2i64(<vscale x 2 x i16>
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -10911,7 +10771,6 @@ define <vscale x 2 x i64> @test_vluxseg2_mask_nxv2i64_nxv2i32(<vscale x 2 x i64>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vluxseg2.mask.nxv2i64.nxv2i32(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10941,7 +10800,6 @@ define <vscale x 2 x i64> @test_vluxseg2_mask_nxv2i64_nxv2i8(<vscale x 2 x i64>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vluxseg2.mask.nxv2i64.nxv2i8(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -10971,7 +10829,6 @@ define <vscale x 2 x i64> @test_vluxseg2_mask_nxv2i64_nxv2i16(<vscale x 2 x i64>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vluxseg2.mask.nxv2i64.nxv2i16(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -11001,7 +10858,6 @@ define <vscale x 2 x i64> @test_vluxseg2_mask_nxv2i64_nxv2i64(<vscale x 2 x i64>
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vluxseg2.mask.nxv2i64.nxv2i64(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -11028,12 +10884,11 @@ entry:
 define <vscale x 2 x i64> @test_vluxseg3_mask_nxv2i64_nxv2i32(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei32.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vluxseg3.mask.nxv2i64.nxv2i32(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -11060,12 +10915,11 @@ entry:
 define <vscale x 2 x i64> @test_vluxseg3_mask_nxv2i64_nxv2i8(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vluxseg3.mask.nxv2i64.nxv2i8(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -11092,12 +10946,11 @@ entry:
 define <vscale x 2 x i64> @test_vluxseg3_mask_nxv2i64_nxv2i16(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vluxseg3.mask.nxv2i64.nxv2i16(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -11124,12 +10977,11 @@ entry:
 define <vscale x 2 x i64> @test_vluxseg3_mask_nxv2i64_nxv2i64(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2i64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei64.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei64.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x i64>,<vscale x 2 x i64>,<vscale x 2 x i64>} @llvm.riscv.vluxseg3.mask.nxv2i64.nxv2i64(<vscale x 2 x i64> %val,<vscale x 2 x i64> %val,<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -11157,9 +11009,9 @@ define <vscale x 2 x i64> @test_vluxseg4_mask_nxv2i64_nxv2i32(<vscale x 2 x i64>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -11190,9 +11042,9 @@ define <vscale x 2 x i64> @test_vluxseg4_mask_nxv2i64_nxv2i8(<vscale x 2 x i64>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -11223,9 +11075,9 @@ define <vscale x 2 x i64> @test_vluxseg4_mask_nxv2i64_nxv2i16(<vscale x 2 x i64>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -11256,9 +11108,9 @@ define <vscale x 2 x i64> @test_vluxseg4_mask_nxv2i64_nxv2i64(<vscale x 2 x i64>
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2i64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -11291,7 +11143,6 @@ define <vscale x 16 x half> @test_vluxseg2_mask_nxv16f16_nxv16i16(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vluxseg2.mask.nxv16f16.nxv16i16(<vscale x 16 x half> %val,<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -11321,7 +11172,6 @@ define <vscale x 16 x half> @test_vluxseg2_mask_nxv16f16_nxv16i8(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vluxseg2.mask.nxv16f16.nxv16i8(<vscale x 16 x half> %val,<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -11351,7 +11201,6 @@ define <vscale x 16 x half> @test_vluxseg2_mask_nxv16f16_nxv16i32(<vscale x 16 x
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v4, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x half>,<vscale x 16 x half>} @llvm.riscv.vluxseg2.mask.nxv16f16.nxv16i32(<vscale x 16 x half> %val,<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl, i64 1)
@@ -11381,7 +11230,6 @@ define <vscale x 4 x double> @test_vluxseg2_mask_nxv4f64_nxv4i32(<vscale x 4 x d
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vluxseg2.mask.nxv4f64.nxv4i32(<vscale x 4 x double> %val,<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -11411,7 +11259,6 @@ define <vscale x 4 x double> @test_vluxseg2_mask_nxv4f64_nxv4i8(<vscale x 4 x do
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vluxseg2.mask.nxv4f64.nxv4i8(<vscale x 4 x double> %val,<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -11441,7 +11288,6 @@ define <vscale x 4 x double> @test_vluxseg2_mask_nxv4f64_nxv4i64(<vscale x 4 x d
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vluxseg2.mask.nxv4f64.nxv4i64(<vscale x 4 x double> %val,<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -11471,7 +11317,6 @@ define <vscale x 4 x double> @test_vluxseg2_mask_nxv4f64_nxv4i16(<vscale x 4 x d
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x double>,<vscale x 4 x double>} @llvm.riscv.vluxseg2.mask.nxv4f64.nxv4i16(<vscale x 4 x double> %val,<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -11501,7 +11346,6 @@ define <vscale x 1 x double> @test_vluxseg2_mask_nxv1f64_nxv1i64(<vscale x 1 x d
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vluxseg2.mask.nxv1f64.nxv1i64(<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -11531,7 +11375,6 @@ define <vscale x 1 x double> @test_vluxseg2_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vluxseg2.mask.nxv1f64.nxv1i32(<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -11561,7 +11404,6 @@ define <vscale x 1 x double> @test_vluxseg2_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vluxseg2.mask.nxv1f64.nxv1i16(<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -11591,7 +11433,6 @@ define <vscale x 1 x double> @test_vluxseg2_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vluxseg2.mask.nxv1f64.nxv1i8(<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -11618,12 +11459,11 @@ entry:
 define <vscale x 1 x double> @test_vluxseg3_mask_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei64.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei64.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vluxseg3.mask.nxv1f64.nxv1i64(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -11650,12 +11490,11 @@ entry:
 define <vscale x 1 x double> @test_vluxseg3_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vluxseg3.mask.nxv1f64.nxv1i32(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -11682,12 +11521,11 @@ entry:
 define <vscale x 1 x double> @test_vluxseg3_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vluxseg3.mask.nxv1f64.nxv1i16(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -11714,12 +11552,11 @@ entry:
 define <vscale x 1 x double> @test_vluxseg3_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x double>,<vscale x 1 x double>,<vscale x 1 x double>} @llvm.riscv.vluxseg3.mask.nxv1f64.nxv1i8(<vscale x 1 x double> %val,<vscale x 1 x double> %val,<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -11747,9 +11584,9 @@ define <vscale x 1 x double> @test_vluxseg4_mask_nxv1f64_nxv1i64(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11780,9 +11617,9 @@ define <vscale x 1 x double> @test_vluxseg4_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11813,9 +11650,9 @@ define <vscale x 1 x double> @test_vluxseg4_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11846,9 +11683,9 @@ define <vscale x 1 x double> @test_vluxseg4_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11879,10 +11716,10 @@ define <vscale x 1 x double> @test_vluxseg5_mask_nxv1f64_nxv1i64(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11913,10 +11750,10 @@ define <vscale x 1 x double> @test_vluxseg5_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11947,10 +11784,10 @@ define <vscale x 1 x double> @test_vluxseg5_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -11981,10 +11818,10 @@ define <vscale x 1 x double> @test_vluxseg5_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12015,11 +11852,11 @@ define <vscale x 1 x double> @test_vluxseg6_mask_nxv1f64_nxv1i64(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12050,11 +11887,11 @@ define <vscale x 1 x double> @test_vluxseg6_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12085,11 +11922,11 @@ define <vscale x 1 x double> @test_vluxseg6_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12120,11 +11957,11 @@ define <vscale x 1 x double> @test_vluxseg6_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12155,12 +11992,12 @@ define <vscale x 1 x double> @test_vluxseg7_mask_nxv1f64_nxv1i64(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12191,12 +12028,12 @@ define <vscale x 1 x double> @test_vluxseg7_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12227,12 +12064,12 @@ define <vscale x 1 x double> @test_vluxseg7_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12263,12 +12100,12 @@ define <vscale x 1 x double> @test_vluxseg7_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12299,13 +12136,13 @@ define <vscale x 1 x double> @test_vluxseg8_mask_nxv1f64_nxv1i64(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12336,13 +12173,13 @@ define <vscale x 1 x double> @test_vluxseg8_mask_nxv1f64_nxv1i32(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12373,13 +12210,13 @@ define <vscale x 1 x double> @test_vluxseg8_mask_nxv1f64_nxv1i16(<vscale x 1 x d
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12410,13 +12247,13 @@ define <vscale x 1 x double> @test_vluxseg8_mask_nxv1f64_nxv1i8(<vscale x 1 x do
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12449,7 +12286,6 @@ define <vscale x 2 x float> @test_vluxseg2_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vluxseg2.mask.nxv2f32.nxv2i32(<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -12479,7 +12315,6 @@ define <vscale x 2 x float> @test_vluxseg2_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vluxseg2.mask.nxv2f32.nxv2i8(<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -12509,7 +12344,6 @@ define <vscale x 2 x float> @test_vluxseg2_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vluxseg2.mask.nxv2f32.nxv2i16(<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -12539,7 +12373,6 @@ define <vscale x 2 x float> @test_vluxseg2_mask_nxv2f32_nxv2i64(<vscale x 2 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vluxseg2.mask.nxv2f32.nxv2i64(<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -12566,12 +12399,11 @@ entry:
 define <vscale x 2 x float> @test_vluxseg3_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vluxseg3.mask.nxv2f32.nxv2i32(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -12598,12 +12430,11 @@ entry:
 define <vscale x 2 x float> @test_vluxseg3_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vluxseg3.mask.nxv2f32.nxv2i8(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -12630,12 +12461,11 @@ entry:
 define <vscale x 2 x float> @test_vluxseg3_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vluxseg3.mask.nxv2f32.nxv2i16(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -12663,10 +12493,9 @@ define <vscale x 2 x float> @test_vluxseg3_mask_nxv2f32_nxv2i64(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg3ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vluxseg3.mask.nxv2f32.nxv2i64(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -12694,9 +12523,9 @@ define <vscale x 2 x float> @test_vluxseg4_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12727,9 +12556,9 @@ define <vscale x 2 x float> @test_vluxseg4_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12760,9 +12589,9 @@ define <vscale x 2 x float> @test_vluxseg4_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12792,13 +12621,12 @@ entry:
 define <vscale x 2 x float> @test_vluxseg4_mask_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
-; CHECK-NEXT:    vluxseg4ei64.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vluxseg4ei64.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>,<vscale x 2 x float>} @llvm.riscv.vluxseg4.mask.nxv2f32.nxv2i64(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -12826,10 +12654,10 @@ define <vscale x 2 x float> @test_vluxseg5_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12860,10 +12688,10 @@ define <vscale x 2 x float> @test_vluxseg5_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12894,10 +12722,10 @@ define <vscale x 2 x float> @test_vluxseg5_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12928,10 +12756,10 @@ define <vscale x 2 x float> @test_vluxseg5_mask_nxv2f32_nxv2i64(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -12962,11 +12790,11 @@ define <vscale x 2 x float> @test_vluxseg6_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -12997,11 +12825,11 @@ define <vscale x 2 x float> @test_vluxseg6_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13032,11 +12860,11 @@ define <vscale x 2 x float> @test_vluxseg6_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13067,11 +12895,11 @@ define <vscale x 2 x float> @test_vluxseg6_mask_nxv2f32_nxv2i64(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -13102,12 +12930,12 @@ define <vscale x 2 x float> @test_vluxseg7_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13138,12 +12966,12 @@ define <vscale x 2 x float> @test_vluxseg7_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13174,12 +13002,12 @@ define <vscale x 2 x float> @test_vluxseg7_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13210,12 +13038,12 @@ define <vscale x 2 x float> @test_vluxseg7_mask_nxv2f32_nxv2i64(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -13246,13 +13074,13 @@ define <vscale x 2 x float> @test_vluxseg8_mask_nxv2f32_nxv2i32(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13283,13 +13111,13 @@ define <vscale x 2 x float> @test_vluxseg8_mask_nxv2f32_nxv2i8(<vscale x 2 x flo
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13320,13 +13148,13 @@ define <vscale x 2 x float> @test_vluxseg8_mask_nxv2f32_nxv2i16(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13357,13 +13185,13 @@ define <vscale x 2 x float> @test_vluxseg8_mask_nxv2f32_nxv2i64(<vscale x 2 x fl
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -13396,7 +13224,6 @@ define <vscale x 1 x half> @test_vluxseg2_mask_nxv1f16_nxv1i64(<vscale x 1 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vluxseg2.mask.nxv1f16.nxv1i64(<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -13426,7 +13253,6 @@ define <vscale x 1 x half> @test_vluxseg2_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vluxseg2.mask.nxv1f16.nxv1i32(<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -13456,7 +13282,6 @@ define <vscale x 1 x half> @test_vluxseg2_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vluxseg2.mask.nxv1f16.nxv1i16(<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -13486,7 +13311,6 @@ define <vscale x 1 x half> @test_vluxseg2_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vluxseg2.mask.nxv1f16.nxv1i8(<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -13513,12 +13337,11 @@ entry:
 define <vscale x 1 x half> @test_vluxseg3_mask_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei64.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei64.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vluxseg3.mask.nxv1f16.nxv1i64(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -13545,12 +13368,11 @@ entry:
 define <vscale x 1 x half> @test_vluxseg3_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vluxseg3.mask.nxv1f16.nxv1i32(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -13577,12 +13399,11 @@ entry:
 define <vscale x 1 x half> @test_vluxseg3_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vluxseg3.mask.nxv1f16.nxv1i16(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -13609,12 +13430,11 @@ entry:
 define <vscale x 1 x half> @test_vluxseg3_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x half>,<vscale x 1 x half>,<vscale x 1 x half>} @llvm.riscv.vluxseg3.mask.nxv1f16.nxv1i8(<vscale x 1 x half> %val,<vscale x 1 x half> %val,<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -13642,9 +13462,9 @@ define <vscale x 1 x half> @test_vluxseg4_mask_nxv1f16_nxv1i64(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13675,9 +13495,9 @@ define <vscale x 1 x half> @test_vluxseg4_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13708,9 +13528,9 @@ define <vscale x 1 x half> @test_vluxseg4_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13741,9 +13561,9 @@ define <vscale x 1 x half> @test_vluxseg4_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13774,10 +13594,10 @@ define <vscale x 1 x half> @test_vluxseg5_mask_nxv1f16_nxv1i64(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13808,10 +13628,10 @@ define <vscale x 1 x half> @test_vluxseg5_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13842,10 +13662,10 @@ define <vscale x 1 x half> @test_vluxseg5_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13876,10 +13696,10 @@ define <vscale x 1 x half> @test_vluxseg5_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13910,11 +13730,11 @@ define <vscale x 1 x half> @test_vluxseg6_mask_nxv1f16_nxv1i64(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13945,11 +13765,11 @@ define <vscale x 1 x half> @test_vluxseg6_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -13980,11 +13800,11 @@ define <vscale x 1 x half> @test_vluxseg6_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14015,11 +13835,11 @@ define <vscale x 1 x half> @test_vluxseg6_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14050,12 +13870,12 @@ define <vscale x 1 x half> @test_vluxseg7_mask_nxv1f16_nxv1i64(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14086,12 +13906,12 @@ define <vscale x 1 x half> @test_vluxseg7_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14122,12 +13942,12 @@ define <vscale x 1 x half> @test_vluxseg7_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14158,12 +13978,12 @@ define <vscale x 1 x half> @test_vluxseg7_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14194,13 +14014,13 @@ define <vscale x 1 x half> @test_vluxseg8_mask_nxv1f16_nxv1i64(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14231,13 +14051,13 @@ define <vscale x 1 x half> @test_vluxseg8_mask_nxv1f16_nxv1i32(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14268,13 +14088,13 @@ define <vscale x 1 x half> @test_vluxseg8_mask_nxv1f16_nxv1i16(<vscale x 1 x hal
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14305,13 +14125,13 @@ define <vscale x 1 x half> @test_vluxseg8_mask_nxv1f16_nxv1i8(<vscale x 1 x half
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14344,7 +14164,6 @@ define <vscale x 1 x float> @test_vluxseg2_mask_nxv1f32_nxv1i64(<vscale x 1 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vluxseg2.mask.nxv1f32.nxv1i64(<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -14374,7 +14193,6 @@ define <vscale x 1 x float> @test_vluxseg2_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vluxseg2.mask.nxv1f32.nxv1i32(<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -14404,7 +14222,6 @@ define <vscale x 1 x float> @test_vluxseg2_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vluxseg2.mask.nxv1f32.nxv1i16(<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -14434,7 +14251,6 @@ define <vscale x 1 x float> @test_vluxseg2_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vluxseg2.mask.nxv1f32.nxv1i8(<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -14461,12 +14277,11 @@ entry:
 define <vscale x 1 x float> @test_vluxseg3_mask_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei64.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei64.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vluxseg3.mask.nxv1f32.nxv1i64(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -14493,12 +14308,11 @@ entry:
 define <vscale x 1 x float> @test_vluxseg3_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vluxseg3.mask.nxv1f32.nxv1i32(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -14525,12 +14339,11 @@ entry:
 define <vscale x 1 x float> @test_vluxseg3_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vluxseg3.mask.nxv1f32.nxv1i16(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -14557,12 +14370,11 @@ entry:
 define <vscale x 1 x float> @test_vluxseg3_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl, <vscale x 1 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 1 x float>,<vscale x 1 x float>,<vscale x 1 x float>} @llvm.riscv.vluxseg3.mask.nxv1f32.nxv1i8(<vscale x 1 x float> %val,<vscale x 1 x float> %val,<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl, i64 1)
@@ -14590,9 +14402,9 @@ define <vscale x 1 x float> @test_vluxseg4_mask_nxv1f32_nxv1i64(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14623,9 +14435,9 @@ define <vscale x 1 x float> @test_vluxseg4_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14656,9 +14468,9 @@ define <vscale x 1 x float> @test_vluxseg4_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14689,9 +14501,9 @@ define <vscale x 1 x float> @test_vluxseg4_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vluxseg4_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14722,10 +14534,10 @@ define <vscale x 1 x float> @test_vluxseg5_mask_nxv1f32_nxv1i64(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14756,10 +14568,10 @@ define <vscale x 1 x float> @test_vluxseg5_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14790,10 +14602,10 @@ define <vscale x 1 x float> @test_vluxseg5_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14824,10 +14636,10 @@ define <vscale x 1 x float> @test_vluxseg5_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vluxseg5_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14858,11 +14670,11 @@ define <vscale x 1 x float> @test_vluxseg6_mask_nxv1f32_nxv1i64(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14893,11 +14705,11 @@ define <vscale x 1 x float> @test_vluxseg6_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14928,11 +14740,11 @@ define <vscale x 1 x float> @test_vluxseg6_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14963,11 +14775,11 @@ define <vscale x 1 x float> @test_vluxseg6_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vluxseg6_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -14998,12 +14810,12 @@ define <vscale x 1 x float> @test_vluxseg7_mask_nxv1f32_nxv1i64(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -15034,12 +14846,12 @@ define <vscale x 1 x float> @test_vluxseg7_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -15070,12 +14882,12 @@ define <vscale x 1 x float> @test_vluxseg7_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -15106,12 +14918,12 @@ define <vscale x 1 x float> @test_vluxseg7_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vluxseg7_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -15142,13 +14954,13 @@ define <vscale x 1 x float> @test_vluxseg8_mask_nxv1f32_nxv1i64(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -15179,13 +14991,13 @@ define <vscale x 1 x float> @test_vluxseg8_mask_nxv1f32_nxv1i32(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -15216,13 +15028,13 @@ define <vscale x 1 x float> @test_vluxseg8_mask_nxv1f32_nxv1i16(<vscale x 1 x fl
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -15253,13 +15065,13 @@ define <vscale x 1 x float> @test_vluxseg8_mask_nxv1f32_nxv1i8(<vscale x 1 x flo
 ; CHECK-LABEL: test_vluxseg8_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -15292,7 +15104,6 @@ define <vscale x 8 x half> @test_vluxseg2_mask_nxv8f16_nxv8i16(<vscale x 8 x hal
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vluxseg2.mask.nxv8f16.nxv8i16(<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15322,7 +15133,6 @@ define <vscale x 8 x half> @test_vluxseg2_mask_nxv8f16_nxv8i8(<vscale x 8 x half
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vluxseg2.mask.nxv8f16.nxv8i8(<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15352,7 +15162,6 @@ define <vscale x 8 x half> @test_vluxseg2_mask_nxv8f16_nxv8i64(<vscale x 8 x hal
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vluxseg2.mask.nxv8f16.nxv8i64(<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15382,7 +15191,6 @@ define <vscale x 8 x half> @test_vluxseg2_mask_nxv8f16_nxv8i32(<vscale x 8 x hal
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vluxseg2.mask.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15409,12 +15217,11 @@ entry:
 define <vscale x 8 x half> @test_vluxseg3_mask_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vluxseg3.mask.nxv8f16.nxv8i16(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15441,12 +15248,11 @@ entry:
 define <vscale x 8 x half> @test_vluxseg3_mask_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, i64 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vluxseg3.mask.nxv8f16.nxv8i8(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15474,10 +15280,9 @@ define <vscale x 8 x half> @test_vluxseg3_mask_nxv8f16_nxv8i64(<vscale x 8 x hal
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8f16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg3ei64.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vluxseg3.mask.nxv8f16.nxv8i64(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15505,10 +15310,9 @@ define <vscale x 8 x half> @test_vluxseg3_mask_nxv8f16_nxv8i32(<vscale x 8 x hal
 ; CHECK-LABEL: test_vluxseg3_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg3ei32.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vluxseg3.mask.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15536,9 +15340,9 @@ define <vscale x 8 x half> @test_vluxseg4_mask_nxv8f16_nxv8i16(<vscale x 8 x hal
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -15569,9 +15373,9 @@ define <vscale x 8 x half> @test_vluxseg4_mask_nxv8f16_nxv8i8(<vscale x 8 x half
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -15602,11 +15406,10 @@ define <vscale x 8 x half> @test_vluxseg4_mask_nxv8f16_nxv8i64(<vscale x 8 x hal
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8f16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
-; CHECK-NEXT:    vmv2r.v v12, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei64.v v6, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2_v12m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vluxseg4.mask.nxv8f16.nxv8i64(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15633,13 +15436,12 @@ entry:
 define <vscale x 8 x half> @test_vluxseg4_mask_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl, <vscale x 8 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
-; CHECK-NEXT:    vluxseg4ei32.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v18
+; CHECK-NEXT:    vluxseg4ei32.v v6, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>,<vscale x 8 x half>} @llvm.riscv.vluxseg4.mask.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15669,7 +15471,6 @@ define <vscale x 8 x float> @test_vluxseg2_mask_nxv8f32_nxv8i16(<vscale x 8 x fl
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vluxseg2.mask.nxv8f32.nxv8i16(<vscale x 8 x float> %val,<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15699,7 +15500,6 @@ define <vscale x 8 x float> @test_vluxseg2_mask_nxv8f32_nxv8i8(<vscale x 8 x flo
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vluxseg2.mask.nxv8f32.nxv8i8(<vscale x 8 x float> %val,<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15729,7 +15529,6 @@ define <vscale x 8 x float> @test_vluxseg2_mask_nxv8f32_nxv8i64(<vscale x 8 x fl
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v4, (a0), v16, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vluxseg2.mask.nxv8f32.nxv8i64(<vscale x 8 x float> %val,<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15759,7 +15558,6 @@ define <vscale x 8 x float> @test_vluxseg2_mask_nxv8f32_nxv8i32(<vscale x 8 x fl
 ; CHECK-NEXT:    vmv4r.v v4, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v4, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 8 x float>,<vscale x 8 x float>} @llvm.riscv.vluxseg2.mask.nxv8f32.nxv8i32(<vscale x 8 x float> %val,<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl, i64 1)
@@ -15789,7 +15587,6 @@ define <vscale x 2 x double> @test_vluxseg2_mask_nxv2f64_nxv2i32(<vscale x 2 x d
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vluxseg2.mask.nxv2f64.nxv2i32(<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -15819,7 +15616,6 @@ define <vscale x 2 x double> @test_vluxseg2_mask_nxv2f64_nxv2i8(<vscale x 2 x do
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vluxseg2.mask.nxv2f64.nxv2i8(<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -15849,7 +15645,6 @@ define <vscale x 2 x double> @test_vluxseg2_mask_nxv2f64_nxv2i16(<vscale x 2 x d
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vluxseg2.mask.nxv2f64.nxv2i16(<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -15879,7 +15674,6 @@ define <vscale x 2 x double> @test_vluxseg2_mask_nxv2f64_nxv2i64(<vscale x 2 x d
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vluxseg2.mask.nxv2f64.nxv2i64(<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -15906,12 +15700,11 @@ entry:
 define <vscale x 2 x double> @test_vluxseg3_mask_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei32.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vluxseg3.mask.nxv2f64.nxv2i32(<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -15938,12 +15731,11 @@ entry:
 define <vscale x 2 x double> @test_vluxseg3_mask_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vluxseg3.mask.nxv2f64.nxv2i8(<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -15970,12 +15762,11 @@ entry:
 define <vscale x 2 x double> @test_vluxseg3_mask_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vluxseg3.mask.nxv2f64.nxv2i16(<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -16002,12 +15793,11 @@ entry:
 define <vscale x 2 x double> @test_vluxseg3_mask_nxv2f64_nxv2i64(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei64.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei64.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x double>,<vscale x 2 x double>,<vscale x 2 x double>} @llvm.riscv.vluxseg3.mask.nxv2f64.nxv2i64(<vscale x 2 x double> %val,<vscale x 2 x double> %val,<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -16035,9 +15825,9 @@ define <vscale x 2 x double> @test_vluxseg4_mask_nxv2f64_nxv2i32(<vscale x 2 x d
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -16068,9 +15858,9 @@ define <vscale x 2 x double> @test_vluxseg4_mask_nxv2f64_nxv2i8(<vscale x 2 x do
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -16101,9 +15891,9 @@ define <vscale x 2 x double> @test_vluxseg4_mask_nxv2f64_nxv2i16(<vscale x 2 x d
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -16134,9 +15924,9 @@ define <vscale x 2 x double> @test_vluxseg4_mask_nxv2f64_nxv2i64(<vscale x 2 x d
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -16169,7 +15959,6 @@ define <vscale x 4 x half> @test_vluxseg2_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg2.mask.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16199,7 +15988,6 @@ define <vscale x 4 x half> @test_vluxseg2_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg2.mask.nxv4f16.nxv4i8(<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16229,7 +16017,6 @@ define <vscale x 4 x half> @test_vluxseg2_mask_nxv4f16_nxv4i64(<vscale x 4 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg2.mask.nxv4f16.nxv4i64(<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16259,7 +16046,6 @@ define <vscale x 4 x half> @test_vluxseg2_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg2.mask.nxv4f16.nxv4i16(<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16287,10 +16073,9 @@ define <vscale x 4 x half> @test_vluxseg3_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg3.mask.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16317,12 +16102,11 @@ entry:
 define <vscale x 4 x half> @test_vluxseg3_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg3.mask.nxv4f16.nxv4i8(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16350,10 +16134,9 @@ define <vscale x 4 x half> @test_vluxseg3_mask_nxv4f16_nxv4i64(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg3ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg3.mask.nxv4f16.nxv4i64(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16380,12 +16163,11 @@ entry:
 define <vscale x 4 x half> @test_vluxseg3_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg3.mask.nxv4f16.nxv4i16(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16412,13 +16194,12 @@ entry:
 define <vscale x 4 x half> @test_vluxseg4_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vluxseg4ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vluxseg4ei32.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg4.mask.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16446,9 +16227,9 @@ define <vscale x 4 x half> @test_vluxseg4_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -16479,11 +16260,10 @@ define <vscale x 4 x half> @test_vluxseg4_mask_nxv4f16_nxv4i64(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg4.mask.nxv4f16.nxv4i64(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16511,9 +16291,9 @@ define <vscale x 4 x half> @test_vluxseg4_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -16544,10 +16324,10 @@ define <vscale x 4 x half> @test_vluxseg5_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -16578,10 +16358,10 @@ define <vscale x 4 x half> @test_vluxseg5_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -16612,12 +16392,11 @@ define <vscale x 4 x half> @test_vluxseg5_mask_nxv4f16_nxv4i64(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
-; CHECK-NEXT:    vmv1r.v v10, v7
-; CHECK-NEXT:    vmv1r.v v11, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei64.v v7, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9_v10_v11
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg5.mask.nxv4f16.nxv4i64(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16645,10 +16424,10 @@ define <vscale x 4 x half> @test_vluxseg5_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg5_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -16679,11 +16458,11 @@ define <vscale x 4 x half> @test_vluxseg6_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -16714,11 +16493,11 @@ define <vscale x 4 x half> @test_vluxseg6_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -16748,15 +16527,14 @@ entry:
 define <vscale x 4 x half> @test_vluxseg6_mask_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
-; CHECK-NEXT:    vluxseg6ei64.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v17
+; CHECK-NEXT:    vluxseg6ei64.v v7, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>,<vscale x 4 x half>} @llvm.riscv.vluxseg6.mask.nxv4f16.nxv4i64(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -16784,11 +16562,11 @@ define <vscale x 4 x half> @test_vluxseg6_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg6_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -16819,12 +16597,12 @@ define <vscale x 4 x half> @test_vluxseg7_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -16855,12 +16633,12 @@ define <vscale x 4 x half> @test_vluxseg7_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -16891,12 +16669,12 @@ define <vscale x 4 x half> @test_vluxseg7_mask_nxv4f16_nxv4i64(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -16927,12 +16705,12 @@ define <vscale x 4 x half> @test_vluxseg7_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg7_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -16963,13 +16741,13 @@ define <vscale x 4 x half> @test_vluxseg8_mask_nxv4f16_nxv4i32(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -17000,13 +16778,13 @@ define <vscale x 4 x half> @test_vluxseg8_mask_nxv4f16_nxv4i8(<vscale x 4 x half
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17037,13 +16815,13 @@ define <vscale x 4 x half> @test_vluxseg8_mask_nxv4f16_nxv4i64(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v17
@@ -17074,13 +16852,13 @@ define <vscale x 4 x half> @test_vluxseg8_mask_nxv4f16_nxv4i16(<vscale x 4 x hal
 ; CHECK-LABEL: test_vluxseg8_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17113,7 +16891,6 @@ define <vscale x 2 x half> @test_vluxseg2_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vluxseg2.mask.nxv2f16.nxv2i32(<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -17143,7 +16920,6 @@ define <vscale x 2 x half> @test_vluxseg2_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vluxseg2.mask.nxv2f16.nxv2i8(<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -17173,7 +16949,6 @@ define <vscale x 2 x half> @test_vluxseg2_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v7, (a0), v9, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vluxseg2.mask.nxv2f16.nxv2i16(<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -17203,7 +16978,6 @@ define <vscale x 2 x half> @test_vluxseg2_mask_nxv2f16_nxv2i64(<vscale x 2 x hal
 ; CHECK-NEXT:    vmv1r.v v7, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vluxseg2.mask.nxv2f16.nxv2i64(<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -17230,12 +17004,11 @@ entry:
 define <vscale x 2 x half> @test_vluxseg3_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei32.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vluxseg3.mask.nxv2f16.nxv2i32(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -17262,12 +17035,11 @@ entry:
 define <vscale x 2 x half> @test_vluxseg3_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei8.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vluxseg3.mask.nxv2f16.nxv2i8(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -17294,12 +17066,11 @@ entry:
 define <vscale x 2 x half> @test_vluxseg3_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v10, v9
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v11
+; CHECK-NEXT:    vluxseg3ei16.v v7, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vluxseg3.mask.nxv2f16.nxv2i16(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -17327,10 +17098,9 @@ define <vscale x 2 x half> @test_vluxseg3_mask_nxv2f16_nxv2i64(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg3_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v7, v8
-; CHECK-NEXT:    vmv1r.v v9, v7
+; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg3ei64.v v7, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8_v9
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vluxseg3.mask.nxv2f16.nxv2i64(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -17358,9 +17128,9 @@ define <vscale x 2 x half> @test_vluxseg4_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17391,9 +17161,9 @@ define <vscale x 2 x half> @test_vluxseg4_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17424,9 +17194,9 @@ define <vscale x 2 x half> @test_vluxseg4_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17456,13 +17226,12 @@ entry:
 define <vscale x 2 x half> @test_vluxseg4_mask_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl, <vscale x 2 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v7, v8
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
-; CHECK-NEXT:    vluxseg4ei64.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv1r.v v8, v13
+; CHECK-NEXT:    vluxseg4ei64.v v7, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>,<vscale x 2 x half>} @llvm.riscv.vluxseg4.mask.nxv2f16.nxv2i64(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl, i64 1)
@@ -17490,10 +17259,10 @@ define <vscale x 2 x half> @test_vluxseg5_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17524,10 +17293,10 @@ define <vscale x 2 x half> @test_vluxseg5_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17558,10 +17327,10 @@ define <vscale x 2 x half> @test_vluxseg5_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17592,10 +17361,10 @@ define <vscale x 2 x half> @test_vluxseg5_mask_nxv2f16_nxv2i64(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg5_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -17626,11 +17395,11 @@ define <vscale x 2 x half> @test_vluxseg6_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17661,11 +17430,11 @@ define <vscale x 2 x half> @test_vluxseg6_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17696,11 +17465,11 @@ define <vscale x 2 x half> @test_vluxseg6_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17731,11 +17500,11 @@ define <vscale x 2 x half> @test_vluxseg6_mask_nxv2f16_nxv2i64(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg6_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -17766,12 +17535,12 @@ define <vscale x 2 x half> @test_vluxseg7_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17802,12 +17571,12 @@ define <vscale x 2 x half> @test_vluxseg7_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17838,12 +17607,12 @@ define <vscale x 2 x half> @test_vluxseg7_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17874,12 +17643,12 @@ define <vscale x 2 x half> @test_vluxseg7_mask_nxv2f16_nxv2i64(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg7_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -17910,13 +17679,13 @@ define <vscale x 2 x half> @test_vluxseg8_mask_nxv2f16_nxv2i32(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17947,13 +17716,13 @@ define <vscale x 2 x half> @test_vluxseg8_mask_nxv2f16_nxv2i8(<vscale x 2 x half
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -17984,13 +17753,13 @@ define <vscale x 2 x half> @test_vluxseg8_mask_nxv2f16_nxv2i16(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v11
@@ -18021,13 +17790,13 @@ define <vscale x 2 x half> @test_vluxseg8_mask_nxv2f16_nxv2i64(<vscale x 2 x hal
 ; CHECK-LABEL: test_vluxseg8_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, mu
 ; CHECK-NEXT:    vluxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv1r.v v8, v13
@@ -18060,7 +17829,6 @@ define <vscale x 4 x float> @test_vluxseg2_mask_nxv4f32_nxv4i32(<vscale x 4 x fl
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei32.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vluxseg2.mask.nxv4f32.nxv4i32(<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -18090,7 +17858,6 @@ define <vscale x 4 x float> @test_vluxseg2_mask_nxv4f32_nxv4i8(<vscale x 4 x flo
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei8.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vluxseg2.mask.nxv4f32.nxv4i8(<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -18120,7 +17887,6 @@ define <vscale x 4 x float> @test_vluxseg2_mask_nxv4f32_nxv4i64(<vscale x 4 x fl
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei64.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vluxseg2.mask.nxv4f32.nxv4i64(<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -18150,7 +17916,6 @@ define <vscale x 4 x float> @test_vluxseg2_mask_nxv4f32_nxv4i16(<vscale x 4 x fl
 ; CHECK-NEXT:    vmv2r.v v6, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg2ei16.v v6, (a0), v10, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vluxseg2.mask.nxv4f32.nxv4i16(<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -18177,12 +17942,11 @@ entry:
 define <vscale x 4 x float> @test_vluxseg3_mask_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei32.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei32.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vluxseg3.mask.nxv4f32.nxv4i32(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -18209,12 +17973,11 @@ entry:
 define <vscale x 4 x float> @test_vluxseg3_mask_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei8.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei8.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vluxseg3.mask.nxv4f32.nxv4i8(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -18242,10 +18005,9 @@ define <vscale x 4 x float> @test_vluxseg3_mask_nxv4f32_nxv4i64(<vscale x 4 x fl
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4f32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v6, v8
-; CHECK-NEXT:    vmv2r.v v10, v6
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg3ei64.v v6, (a0), v12, v0.t
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vluxseg3.mask.nxv4f32.nxv4i64(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -18272,12 +18034,11 @@ entry:
 define <vscale x 4 x float> @test_vluxseg3_mask_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg3_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vluxseg3ei16.v v12, (a0), v10, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v14
+; CHECK-NEXT:    vluxseg3ei16.v v6, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vluxseg3.mask.nxv4f32.nxv4i16(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -18305,9 +18066,9 @@ define <vscale x 4 x float> @test_vluxseg4_mask_nxv4f32_nxv4i32(<vscale x 4 x fl
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -18338,9 +18099,9 @@ define <vscale x 4 x float> @test_vluxseg4_mask_nxv4f32_nxv4i8(<vscale x 4 x flo
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
@@ -18370,13 +18131,12 @@ entry:
 define <vscale x 4 x float> @test_vluxseg4_mask_nxv4f32_nxv4i64(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl, <vscale x 4 x i1> %mask) {
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4f32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v6, v8
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
-; CHECK-NEXT:    vluxseg4ei64.v v16, (a0), v12, v0.t
-; CHECK-NEXT:    vmv2r.v v8, v18
+; CHECK-NEXT:    vluxseg4ei64.v v6, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>,<vscale x 4 x float>} @llvm.riscv.vluxseg4.mask.nxv4f32.nxv4i64(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl, i64 1)
@@ -18404,9 +18164,9 @@ define <vscale x 4 x float> @test_vluxseg4_mask_nxv4f32_nxv4i16(<vscale x 4 x fl
 ; CHECK-LABEL: test_vluxseg4_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, mu
 ; CHECK-NEXT:    vluxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    vmv2r.v v8, v14
diff --git a/llvm/test/CodeGen/RISCV/rvv/vpgather-sdnode.ll b/llvm/test/CodeGen/RISCV/rvv/vpgather-sdnode.ll
index 199fc6fa62432..aa8850f8210e8 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vpgather-sdnode.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vpgather-sdnode.ll
@@ -267,19 +267,18 @@ define <vscale x 32 x i8> @vpgather_baseidx_nxv32i8(ptr %base, <vscale x 32 x i8
 ; RV32-NEXT:    vsetvli a5, zero, e8, mf2, ta, ma
 ; RV32-NEXT:    vslidedown.vx v0, v0, a3
 ; RV32-NEXT:    vsetvli a3, zero, e32, m8, ta, ma
-; RV32-NEXT:    vsext.vf4 v24, v10
+; RV32-NEXT:    vsext.vf4 v16, v10
 ; RV32-NEXT:    vsetvli zero, a4, e8, m2, ta, ma
-; RV32-NEXT:    vluxei32.v v18, (a0), v24, v0.t
+; RV32-NEXT:    vluxei32.v v10, (a0), v16, v0.t
 ; RV32-NEXT:    bltu a1, a2, .LBB12_2
 ; RV32-NEXT:  # %bb.1:
 ; RV32-NEXT:    mv a1, a2
 ; RV32-NEXT:  .LBB12_2:
 ; RV32-NEXT:    vsetvli a2, zero, e32, m8, ta, ma
-; RV32-NEXT:    vsext.vf4 v24, v8
+; RV32-NEXT:    vsext.vf4 v16, v8
 ; RV32-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; RV32-NEXT:    vmv1r.v v0, v12
-; RV32-NEXT:    vluxei32.v v16, (a0), v24, v0.t
-; RV32-NEXT:    vmv4r.v v8, v16
+; RV32-NEXT:    vluxei32.v v8, (a0), v16, v0.t
 ; RV32-NEXT:    ret
 ;
 ; RV64-LABEL: vpgather_baseidx_nxv32i8:
@@ -290,7 +289,7 @@ define <vscale x 32 x i8> @vpgather_baseidx_nxv32i8(ptr %base, <vscale x 32 x i8
 ; RV64-NEXT:    sltu a5, a1, a3
 ; RV64-NEXT:    addi a5, a5, -1
 ; RV64-NEXT:    and a3, a5, a3
-; RV64-NEXT:    vmv1r.v v17, v0
+; RV64-NEXT:    vmv1r.v v13, v0
 ; RV64-NEXT:    mv a5, a3
 ; RV64-NEXT:    bltu a3, a2, .LBB12_2
 ; RV64-NEXT:  # %bb.1:
@@ -298,12 +297,12 @@ define <vscale x 32 x i8> @vpgather_baseidx_nxv32i8(ptr %base, <vscale x 32 x i8
 ; RV64-NEXT:  .LBB12_2:
 ; RV64-NEXT:    srli a6, a2, 2
 ; RV64-NEXT:    vsetvli a7, zero, e8, mf2, ta, ma
-; RV64-NEXT:    vslidedown.vx v16, v17, a6
+; RV64-NEXT:    vslidedown.vx v12, v13, a6
 ; RV64-NEXT:    vsetvli a6, zero, e64, m8, ta, ma
-; RV64-NEXT:    vsext.vf8 v24, v10
+; RV64-NEXT:    vsext.vf8 v16, v10
 ; RV64-NEXT:    vsetvli zero, a5, e8, m1, ta, ma
-; RV64-NEXT:    vmv1r.v v0, v16
-; RV64-NEXT:    vluxei64.v v14, (a0), v24, v0.t
+; RV64-NEXT:    vmv1r.v v0, v12
+; RV64-NEXT:    vluxei64.v v10, (a0), v16, v0.t
 ; RV64-NEXT:    bltu a1, a4, .LBB12_4
 ; RV64-NEXT:  # %bb.3:
 ; RV64-NEXT:    mv a1, a4
@@ -314,31 +313,30 @@ define <vscale x 32 x i8> @vpgather_baseidx_nxv32i8(ptr %base, <vscale x 32 x i8
 ; RV64-NEXT:    and a5, a5, a4
 ; RV64-NEXT:    srli a4, a2, 3
 ; RV64-NEXT:    vsetvli a6, zero, e8, mf4, ta, ma
-; RV64-NEXT:    vslidedown.vx v0, v17, a4
+; RV64-NEXT:    vslidedown.vx v0, v13, a4
 ; RV64-NEXT:    vsetvli a6, zero, e64, m8, ta, ma
-; RV64-NEXT:    vsext.vf8 v24, v9
+; RV64-NEXT:    vsext.vf8 v16, v9
 ; RV64-NEXT:    vsetvli zero, a5, e8, m1, ta, ma
-; RV64-NEXT:    vluxei64.v v13, (a0), v24, v0.t
+; RV64-NEXT:    vluxei64.v v9, (a0), v16, v0.t
 ; RV64-NEXT:    bltu a1, a2, .LBB12_6
 ; RV64-NEXT:  # %bb.5:
 ; RV64-NEXT:    mv a1, a2
 ; RV64-NEXT:  .LBB12_6:
 ; RV64-NEXT:    vsetvli a5, zero, e64, m8, ta, ma
-; RV64-NEXT:    vsext.vf8 v24, v8
+; RV64-NEXT:    vsext.vf8 v16, v8
 ; RV64-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
-; RV64-NEXT:    vmv1r.v v0, v17
-; RV64-NEXT:    vluxei64.v v12, (a0), v24, v0.t
+; RV64-NEXT:    vmv1r.v v0, v13
+; RV64-NEXT:    vluxei64.v v8, (a0), v16, v0.t
 ; RV64-NEXT:    sub a1, a3, a2
 ; RV64-NEXT:    sltu a2, a3, a1
 ; RV64-NEXT:    addi a2, a2, -1
 ; RV64-NEXT:    and a1, a2, a1
 ; RV64-NEXT:    vsetvli a2, zero, e8, mf4, ta, ma
-; RV64-NEXT:    vslidedown.vx v0, v16, a4
+; RV64-NEXT:    vslidedown.vx v0, v12, a4
 ; RV64-NEXT:    vsetvli a2, zero, e64, m8, ta, ma
 ; RV64-NEXT:    vsext.vf8 v16, v11
 ; RV64-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
-; RV64-NEXT:    vluxei64.v v15, (a0), v16, v0.t
-; RV64-NEXT:    vmv4r.v v8, v12
+; RV64-NEXT:    vluxei64.v v11, (a0), v16, v0.t
 ; RV64-NEXT:    ret
   %ptrs = getelementptr inbounds i8, ptr %base, <vscale x 32 x i8> %idxs
   %v = call <vscale x 32 x i8> @llvm.vp.gather.nxv32i8.nxv32p0(<vscale x 32 x ptr> %ptrs, <vscale x 32 x i1> %m, i32 %evl)
diff --git a/llvm/test/CodeGen/RISCV/rvv/vrgatherei16-subreg-liveness.ll b/llvm/test/CodeGen/RISCV/rvv/vrgatherei16-subreg-liveness.ll
index 09d926787d522..0c0a3dc9675b1 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vrgatherei16-subreg-liveness.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vrgatherei16-subreg-liveness.ll
@@ -12,7 +12,6 @@
 define internal void @foo(<vscale x 1 x i16> %v15, <vscale x 1 x i16> %0, <vscale x 1 x i16> %vs12.i.i.i, <vscale x 1 x i16> %1, <vscale x 8 x i8> %v37) {
 ; NOSUBREG-LABEL: foo:
 ; NOSUBREG:       # %bb.0: # %loopIR.preheader.i.i
-; NOSUBREG-NEXT:    # kill: def $v10 killed $v10 def $v10m2
 ; NOSUBREG-NEXT:    vsetvli a0, zero, e16, m2, ta, ma
 ; NOSUBREG-NEXT:    vmv.v.i v14, 0
 ; NOSUBREG-NEXT:    vsetvli zero, zero, e8, m1, ta, ma
diff --git a/llvm/test/CodeGen/RISCV/rvv/vsetvli-insert-crossbb.ll b/llvm/test/CodeGen/RISCV/rvv/vsetvli-insert-crossbb.ll
index f1b84da0351a9..c0fa9042871c5 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vsetvli-insert-crossbb.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vsetvli-insert-crossbb.ll
@@ -91,11 +91,13 @@ define <vscale x 1 x double> @test3(i64 %avl, i8 zeroext %cond, <vscale x 1 x do
 ; CHECK-NEXT:    vsetvli a0, a0, e64, m1, ta, ma
 ; CHECK-NEXT:    vfadd.vv v9, v8, v9
 ; CHECK-NEXT:    vfmul.vv v8, v9, v8
+; CHECK-NEXT:    # implicit-def: $x10
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB2_2: # %if.else
 ; CHECK-NEXT:    vsetvli a0, a0, e64, m1, ta, ma
 ; CHECK-NEXT:    vfsub.vv v9, v8, v9
 ; CHECK-NEXT:    vfmul.vv v8, v9, v8
+; CHECK-NEXT:    # implicit-def: $x10
 ; CHECK-NEXT:    ret
 entry:
   %tobool = icmp eq i8 %cond, 0
diff --git a/llvm/test/CodeGen/RISCV/rvv/vsitofp-vp.ll b/llvm/test/CodeGen/RISCV/rvv/vsitofp-vp.ll
index 10e41682034e2..0cfb36037221c 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vsitofp-vp.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vsitofp-vp.ll
@@ -306,15 +306,7 @@ declare <vscale x 32 x half> @llvm.vp.sitofp.nxv32f16.nxv32i32(<vscale x 32 x i3
 define <vscale x 32 x half> @vsitofp_nxv32f16_nxv32i32(<vscale x 32 x i32> %va, <vscale x 32 x i1> %m, i32 zeroext %evl) {
 ; CHECK-LABEL: vsitofp_nxv32f16_nxv32i32:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    addi sp, sp, -16
-; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    csrr a1, vlenb
-; CHECK-NEXT:    slli a1, a1, 3
-; CHECK-NEXT:    sub sp, sp, a1
-; CHECK-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x08, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 8 * vlenb
 ; CHECK-NEXT:    vmv1r.v v24, v0
-; CHECK-NEXT:    addi a1, sp, 16
-; CHECK-NEXT:    vs8r.v v8, (a1) # Unknown-size Folded Spill
 ; CHECK-NEXT:    csrr a1, vlenb
 ; CHECK-NEXT:    srli a2, a1, 2
 ; CHECK-NEXT:    vsetvli a3, zero, e8, mf2, ta, ma
@@ -325,20 +317,15 @@ define <vscale x 32 x half> @vsitofp_nxv32f16_nxv32i32(<vscale x 32 x i32> %va,
 ; CHECK-NEXT:    addi a3, a3, -1
 ; CHECK-NEXT:    and a2, a3, a2
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m4, ta, ma
-; CHECK-NEXT:    vfncvt.f.x.w v12, v16, v0.t
+; CHECK-NEXT:    vfncvt.f.x.w v28, v16, v0.t
 ; CHECK-NEXT:    bltu a0, a1, .LBB25_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a0, a1
 ; CHECK-NEXT:  .LBB25_2:
 ; CHECK-NEXT:    vsetvli zero, a0, e16, m4, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v24
-; CHECK-NEXT:    addi a0, sp, 16
-; CHECK-NEXT:    vl8r.v v16, (a0) # Unknown-size Folded Reload
-; CHECK-NEXT:    vfncvt.f.x.w v8, v16, v0.t
-; CHECK-NEXT:    csrr a0, vlenb
-; CHECK-NEXT:    slli a0, a0, 3
-; CHECK-NEXT:    add sp, sp, a0
-; CHECK-NEXT:    addi sp, sp, 16
+; CHECK-NEXT:    vfncvt.f.x.w v24, v8, v0.t
+; CHECK-NEXT:    vmv8r.v v8, v24
 ; CHECK-NEXT:    ret
   %v = call <vscale x 32 x half> @llvm.vp.sitofp.nxv32f16.nxv32i32(<vscale x 32 x i32> %va, <vscale x 32 x i1> %m, i32 %evl)
   ret <vscale x 32 x half> %v
diff --git a/llvm/test/CodeGen/RISCV/rvv/vsoxseg-rv32.ll b/llvm/test/CodeGen/RISCV/rvv/vsoxseg-rv32.ll
index da16846c7d673..076196c7bb665 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vsoxseg-rv32.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vsoxseg-rv32.ll
@@ -8,7 +8,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16i16.nxv16i16(<vscale x 16 x i16>,<vs
 define void @test_vsoxseg2_nxv16i16_nxv16i16(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16i16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -22,7 +21,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16i16_nxv16i16(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16i16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -39,7 +37,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16i16.nxv16i8(<vscale x 16 x i16>,<vsc
 define void @test_vsoxseg2_nxv16i16_nxv16i8(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16i16_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -53,7 +50,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16i16_nxv16i8(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16i16_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -70,7 +66,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16i16.nxv16i32(<vscale x 16 x i16>,<vs
 define void @test_vsoxseg2_nxv16i16_nxv16i32(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16i16_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v16
@@ -83,7 +78,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16i16_nxv16i32(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16i16_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v16, v0.t
@@ -99,7 +93,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i8.nxv1i8(<vscale x 1 x i8>,<vscale x
 define void @test_vsoxseg2_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -113,7 +106,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -130,7 +122,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i8.nxv1i32(<vscale x 1 x i8>,<vscale
 define void @test_vsoxseg2_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -144,7 +135,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -161,7 +151,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i8.nxv1i16(<vscale x 1 x i8>,<vscale
 define void @test_vsoxseg2_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -175,7 +164,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -193,8 +181,8 @@ define void @test_vsoxseg3_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg3_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -207,8 +195,8 @@ define void @test_vsoxseg3_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -224,8 +212,8 @@ define void @test_vsoxseg3_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg3_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -238,8 +226,8 @@ define void @test_vsoxseg3_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -255,8 +243,8 @@ define void @test_vsoxseg3_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg3_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -269,8 +257,8 @@ define void @test_vsoxseg3_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -286,9 +274,9 @@ define void @test_vsoxseg4_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg4_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -301,9 +289,9 @@ define void @test_vsoxseg4_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -319,9 +307,9 @@ define void @test_vsoxseg4_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg4_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -334,9 +322,9 @@ define void @test_vsoxseg4_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -352,9 +340,9 @@ define void @test_vsoxseg4_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg4_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -367,9 +355,9 @@ define void @test_vsoxseg4_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -385,10 +373,10 @@ define void @test_vsoxseg5_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg5_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -401,10 +389,10 @@ define void @test_vsoxseg5_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -420,10 +408,10 @@ define void @test_vsoxseg5_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg5_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -436,10 +424,10 @@ define void @test_vsoxseg5_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -455,10 +443,10 @@ define void @test_vsoxseg5_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg5_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -471,10 +459,10 @@ define void @test_vsoxseg5_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -490,11 +478,11 @@ define void @test_vsoxseg6_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg6_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -507,11 +495,11 @@ define void @test_vsoxseg6_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -527,11 +515,11 @@ define void @test_vsoxseg6_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -544,11 +532,11 @@ define void @test_vsoxseg6_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -564,11 +552,11 @@ define void @test_vsoxseg6_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -581,11 +569,11 @@ define void @test_vsoxseg6_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -601,12 +589,12 @@ define void @test_vsoxseg7_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg7_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -619,12 +607,12 @@ define void @test_vsoxseg7_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -640,12 +628,12 @@ define void @test_vsoxseg7_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -658,12 +646,12 @@ define void @test_vsoxseg7_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -679,12 +667,12 @@ define void @test_vsoxseg7_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -697,12 +685,12 @@ define void @test_vsoxseg7_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -718,13 +706,13 @@ define void @test_vsoxseg8_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg8_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -737,13 +725,13 @@ define void @test_vsoxseg8_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -759,13 +747,13 @@ define void @test_vsoxseg8_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -778,13 +766,13 @@ define void @test_vsoxseg8_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -800,13 +788,13 @@ define void @test_vsoxseg8_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -819,13 +807,13 @@ define void @test_vsoxseg8_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -840,7 +828,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16i8.nxv16i16(<vscale x 16 x i8>,<vsca
 define void @test_vsoxseg2_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei16.v v8, (a0), v12
@@ -853,7 +840,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei16.v v8, (a0), v12, v0.t
@@ -869,7 +855,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16i8.nxv16i8(<vscale x 16 x i8>,<vscal
 define void @test_vsoxseg2_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -883,7 +868,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -900,7 +884,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16i8.nxv16i32(<vscale x 16 x i8>,<vsca
 define void @test_vsoxseg2_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v16
@@ -913,7 +896,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v16, v0.t
@@ -929,11 +911,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv16i8.nxv16i16(<vscale x 16 x i8>,<vsca
 define void @test_vsoxseg3_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei16.v v16, (a0), v12
+; CHECK-NEXT:    vsoxseg3ei16.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, i32 %vl)
@@ -943,11 +925,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei16.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsoxseg3ei16.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl)
@@ -961,8 +943,8 @@ define void @test_vsoxseg3_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -975,8 +957,8 @@ define void @test_vsoxseg3_mask_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -991,7 +973,6 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv16i8.nxv16i32(<vscale x 16 x i8>,<vsca
 define void @test_vsoxseg3_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -1005,7 +986,6 @@ entry:
 define void @test_vsoxseg3_mask_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -1023,9 +1003,9 @@ define void @test_vsoxseg4_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -1038,9 +1018,9 @@ define void @test_vsoxseg4_mask_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -1056,9 +1036,9 @@ define void @test_vsoxseg4_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -1071,9 +1051,9 @@ define void @test_vsoxseg4_mask_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -1088,7 +1068,6 @@ declare void @llvm.riscv.vsoxseg4.mask.nxv16i8.nxv16i32(<vscale x 16 x i8>,<vsca
 define void @test_vsoxseg4_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -1103,7 +1082,6 @@ entry:
 define void @test_vsoxseg4_mask_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -1121,7 +1099,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i32.nxv2i32(<vscale x 2 x i32>,<vscal
 define void @test_vsoxseg2_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -1135,7 +1112,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -1152,7 +1128,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i32.nxv2i8(<vscale x 2 x i32>,<vscale
 define void @test_vsoxseg2_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -1166,7 +1141,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -1183,7 +1157,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i32.nxv2i16(<vscale x 2 x i32>,<vscal
 define void @test_vsoxseg2_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -1197,7 +1170,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -1215,8 +1187,8 @@ define void @test_vsoxseg3_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1229,8 +1201,8 @@ define void @test_vsoxseg3_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1246,8 +1218,8 @@ define void @test_vsoxseg3_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg3_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1260,8 +1232,8 @@ define void @test_vsoxseg3_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1277,8 +1249,8 @@ define void @test_vsoxseg3_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1291,8 +1263,8 @@ define void @test_vsoxseg3_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1308,9 +1280,9 @@ define void @test_vsoxseg4_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1323,9 +1295,9 @@ define void @test_vsoxseg4_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1341,9 +1313,9 @@ define void @test_vsoxseg4_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg4_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1356,9 +1328,9 @@ define void @test_vsoxseg4_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1374,9 +1346,9 @@ define void @test_vsoxseg4_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1389,9 +1361,9 @@ define void @test_vsoxseg4_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1407,10 +1379,10 @@ define void @test_vsoxseg5_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1423,10 +1395,10 @@ define void @test_vsoxseg5_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1442,10 +1414,10 @@ define void @test_vsoxseg5_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg5_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1458,10 +1430,10 @@ define void @test_vsoxseg5_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1477,10 +1449,10 @@ define void @test_vsoxseg5_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1493,10 +1465,10 @@ define void @test_vsoxseg5_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1512,11 +1484,11 @@ define void @test_vsoxseg6_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1529,11 +1501,11 @@ define void @test_vsoxseg6_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1549,11 +1521,11 @@ define void @test_vsoxseg6_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg6_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1566,11 +1538,11 @@ define void @test_vsoxseg6_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1586,11 +1558,11 @@ define void @test_vsoxseg6_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1603,11 +1575,11 @@ define void @test_vsoxseg6_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1623,12 +1595,12 @@ define void @test_vsoxseg7_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1641,12 +1613,12 @@ define void @test_vsoxseg7_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1662,12 +1634,12 @@ define void @test_vsoxseg7_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg7_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1680,12 +1652,12 @@ define void @test_vsoxseg7_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1701,12 +1673,12 @@ define void @test_vsoxseg7_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1719,12 +1691,12 @@ define void @test_vsoxseg7_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1740,13 +1712,13 @@ define void @test_vsoxseg8_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1759,13 +1731,13 @@ define void @test_vsoxseg8_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1781,13 +1753,13 @@ define void @test_vsoxseg8_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg8_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1800,13 +1772,13 @@ define void @test_vsoxseg8_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1822,13 +1794,13 @@ define void @test_vsoxseg8_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1841,13 +1813,13 @@ define void @test_vsoxseg8_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1862,7 +1834,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i16.nxv4i16(<vscale x 4 x i16>,<vscal
 define void @test_vsoxseg2_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -1876,7 +1847,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -1893,7 +1863,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i16.nxv4i8(<vscale x 4 x i16>,<vscale
 define void @test_vsoxseg2_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -1907,7 +1876,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -1924,7 +1892,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i16.nxv4i32(<vscale x 4 x i16>,<vscal
 define void @test_vsoxseg2_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v10
@@ -1937,7 +1904,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v10, v0.t
@@ -1954,8 +1920,8 @@ define void @test_vsoxseg3_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1968,8 +1934,8 @@ define void @test_vsoxseg3_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1985,8 +1951,8 @@ define void @test_vsoxseg3_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg3_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1999,8 +1965,8 @@ define void @test_vsoxseg3_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2015,11 +1981,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv4i16.nxv4i32(<vscale x 4 x i16>,<vscal
 define void @test_vsoxseg3_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl)
@@ -2029,11 +1995,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl)
@@ -2047,9 +2013,9 @@ define void @test_vsoxseg4_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2062,9 +2028,9 @@ define void @test_vsoxseg4_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2080,9 +2046,9 @@ define void @test_vsoxseg4_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg4_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2095,9 +2061,9 @@ define void @test_vsoxseg4_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2113,9 +2079,9 @@ define void @test_vsoxseg4_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -2128,9 +2094,9 @@ define void @test_vsoxseg4_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -2146,10 +2112,10 @@ define void @test_vsoxseg5_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2162,10 +2128,10 @@ define void @test_vsoxseg5_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2181,10 +2147,10 @@ define void @test_vsoxseg5_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg5_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2197,10 +2163,10 @@ define void @test_vsoxseg5_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2216,10 +2182,10 @@ define void @test_vsoxseg5_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -2232,10 +2198,10 @@ define void @test_vsoxseg5_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -2251,11 +2217,11 @@ define void @test_vsoxseg6_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2268,11 +2234,11 @@ define void @test_vsoxseg6_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2288,11 +2254,11 @@ define void @test_vsoxseg6_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg6_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2305,11 +2271,11 @@ define void @test_vsoxseg6_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2325,11 +2291,11 @@ define void @test_vsoxseg6_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -2342,11 +2308,11 @@ define void @test_vsoxseg6_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -2362,12 +2328,12 @@ define void @test_vsoxseg7_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2380,12 +2346,12 @@ define void @test_vsoxseg7_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2401,12 +2367,12 @@ define void @test_vsoxseg7_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg7_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2419,12 +2385,12 @@ define void @test_vsoxseg7_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2440,12 +2406,12 @@ define void @test_vsoxseg7_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -2458,12 +2424,12 @@ define void @test_vsoxseg7_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -2479,13 +2445,13 @@ define void @test_vsoxseg8_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2498,13 +2464,13 @@ define void @test_vsoxseg8_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2520,13 +2486,13 @@ define void @test_vsoxseg8_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg8_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2539,13 +2505,13 @@ define void @test_vsoxseg8_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2561,13 +2527,13 @@ define void @test_vsoxseg8_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -2580,13 +2546,13 @@ define void @test_vsoxseg8_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -2601,7 +2567,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i32.nxv1i8(<vscale x 1 x i32>,<vscale
 define void @test_vsoxseg2_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -2615,7 +2580,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -2632,7 +2596,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i32.nxv1i32(<vscale x 1 x i32>,<vscal
 define void @test_vsoxseg2_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -2646,7 +2609,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -2663,7 +2625,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i32.nxv1i16(<vscale x 1 x i32>,<vscal
 define void @test_vsoxseg2_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -2677,7 +2638,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -2695,8 +2655,8 @@ define void @test_vsoxseg3_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg3_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2709,8 +2669,8 @@ define void @test_vsoxseg3_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2726,8 +2686,8 @@ define void @test_vsoxseg3_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2740,8 +2700,8 @@ define void @test_vsoxseg3_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2757,8 +2717,8 @@ define void @test_vsoxseg3_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2771,8 +2731,8 @@ define void @test_vsoxseg3_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2788,9 +2748,9 @@ define void @test_vsoxseg4_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg4_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2803,9 +2763,9 @@ define void @test_vsoxseg4_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2821,9 +2781,9 @@ define void @test_vsoxseg4_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2836,9 +2796,9 @@ define void @test_vsoxseg4_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2854,9 +2814,9 @@ define void @test_vsoxseg4_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2869,9 +2829,9 @@ define void @test_vsoxseg4_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2887,10 +2847,10 @@ define void @test_vsoxseg5_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg5_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2903,10 +2863,10 @@ define void @test_vsoxseg5_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2922,10 +2882,10 @@ define void @test_vsoxseg5_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2938,10 +2898,10 @@ define void @test_vsoxseg5_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2957,10 +2917,10 @@ define void @test_vsoxseg5_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2973,10 +2933,10 @@ define void @test_vsoxseg5_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2992,11 +2952,11 @@ define void @test_vsoxseg6_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg6_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3009,11 +2969,11 @@ define void @test_vsoxseg6_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3029,11 +2989,11 @@ define void @test_vsoxseg6_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3046,11 +3006,11 @@ define void @test_vsoxseg6_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3066,11 +3026,11 @@ define void @test_vsoxseg6_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3083,11 +3043,11 @@ define void @test_vsoxseg6_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3103,12 +3063,12 @@ define void @test_vsoxseg7_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg7_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3121,12 +3081,12 @@ define void @test_vsoxseg7_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3142,12 +3102,12 @@ define void @test_vsoxseg7_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3160,12 +3120,12 @@ define void @test_vsoxseg7_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3181,12 +3141,12 @@ define void @test_vsoxseg7_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3199,12 +3159,12 @@ define void @test_vsoxseg7_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3220,13 +3180,13 @@ define void @test_vsoxseg8_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg8_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3239,13 +3199,13 @@ define void @test_vsoxseg8_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3261,13 +3221,13 @@ define void @test_vsoxseg8_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3280,13 +3240,13 @@ define void @test_vsoxseg8_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3302,13 +3262,13 @@ define void @test_vsoxseg8_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3321,13 +3281,13 @@ define void @test_vsoxseg8_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3342,7 +3302,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i16.nxv8i16(<vscale x 8 x i16>,<vscal
 define void @test_vsoxseg2_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -3356,7 +3315,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -3373,7 +3331,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i16.nxv8i8(<vscale x 8 x i16>,<vscale
 define void @test_vsoxseg2_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -3387,7 +3344,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -3404,7 +3360,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i16.nxv8i32(<vscale x 8 x i16>,<vscal
 define void @test_vsoxseg2_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v12
@@ -3417,7 +3372,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v12, v0.t
@@ -3434,8 +3388,8 @@ define void @test_vsoxseg3_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3448,8 +3402,8 @@ define void @test_vsoxseg3_mask_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3465,8 +3419,8 @@ define void @test_vsoxseg3_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg3_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3479,8 +3433,8 @@ define void @test_vsoxseg3_mask_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3495,11 +3449,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv8i16.nxv8i32(<vscale x 8 x i16>,<vscal
 define void @test_vsoxseg3_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v16, (a0), v12
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl)
@@ -3509,11 +3463,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl)
@@ -3527,10 +3481,10 @@ define void @test_vsoxseg4_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
-; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
 entry:
@@ -3542,9 +3496,9 @@ define void @test_vsoxseg4_mask_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3560,9 +3514,9 @@ define void @test_vsoxseg4_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg4_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3575,9 +3529,9 @@ define void @test_vsoxseg4_mask_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3593,9 +3547,9 @@ define void @test_vsoxseg4_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -3608,9 +3562,9 @@ define void @test_vsoxseg4_mask_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -3625,7 +3579,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i8.nxv8i16(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg2_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei16.v v8, (a0), v10
@@ -3638,7 +3591,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei16.v v8, (a0), v10, v0.t
@@ -3654,7 +3606,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i8.nxv8i8(<vscale x 8 x i8>,<vscale x
 define void @test_vsoxseg2_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -3668,7 +3619,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -3685,7 +3635,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i8.nxv8i32(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg2_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v12
@@ -3698,7 +3647,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v12, v0.t
@@ -3714,11 +3662,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv8i8.nxv8i16(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg3_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
-; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10
+; CHECK-NEXT:    vsoxseg3ei16.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl)
@@ -3728,11 +3676,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
-; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsoxseg3ei16.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl)
@@ -3746,8 +3694,8 @@ define void @test_vsoxseg3_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg3_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3760,8 +3708,8 @@ define void @test_vsoxseg3_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3776,7 +3724,6 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv8i8.nxv8i32(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg3_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -3790,7 +3737,6 @@ entry:
 define void @test_vsoxseg3_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -3808,9 +3754,9 @@ define void @test_vsoxseg4_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg4_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3823,9 +3769,9 @@ define void @test_vsoxseg4_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3841,9 +3787,9 @@ define void @test_vsoxseg4_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg4_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3856,9 +3802,9 @@ define void @test_vsoxseg4_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3873,7 +3819,6 @@ declare void @llvm.riscv.vsoxseg4.mask.nxv8i8.nxv8i32(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg4_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3888,7 +3833,6 @@ entry:
 define void @test_vsoxseg4_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3907,10 +3851,10 @@ define void @test_vsoxseg5_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg5_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3923,10 +3867,10 @@ define void @test_vsoxseg5_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3942,10 +3886,10 @@ define void @test_vsoxseg5_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg5_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3958,10 +3902,10 @@ define void @test_vsoxseg5_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3976,13 +3920,13 @@ declare void @llvm.riscv.vsoxseg5.mask.nxv8i8.nxv8i32(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg5_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg5_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
-; CHECK-NEXT:    vsoxseg5ei32.v v16, (a0), v12
+; CHECK-NEXT:    vsoxseg5ei32.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg5.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl)
@@ -3992,13 +3936,13 @@ entry:
 define void @test_vsoxseg5_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
-; CHECK-NEXT:    vsoxseg5ei32.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsoxseg5ei32.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg5.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl)
@@ -4012,11 +3956,11 @@ define void @test_vsoxseg6_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -4029,11 +3973,11 @@ define void @test_vsoxseg6_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -4049,11 +3993,11 @@ define void @test_vsoxseg6_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg6_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4066,11 +4010,11 @@ define void @test_vsoxseg6_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4086,11 +4030,11 @@ define void @test_vsoxseg6_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -4103,11 +4047,11 @@ define void @test_vsoxseg6_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -4123,12 +4067,12 @@ define void @test_vsoxseg7_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -4141,12 +4085,12 @@ define void @test_vsoxseg7_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -4162,12 +4106,12 @@ define void @test_vsoxseg7_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg7_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4180,12 +4124,12 @@ define void @test_vsoxseg7_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4201,12 +4145,12 @@ define void @test_vsoxseg7_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -4219,12 +4163,12 @@ define void @test_vsoxseg7_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -4240,13 +4184,13 @@ define void @test_vsoxseg8_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -4259,13 +4203,13 @@ define void @test_vsoxseg8_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -4281,13 +4225,13 @@ define void @test_vsoxseg8_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg8_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4300,13 +4244,13 @@ define void @test_vsoxseg8_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4322,13 +4266,13 @@ define void @test_vsoxseg8_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -4341,13 +4285,13 @@ define void @test_vsoxseg8_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -4362,7 +4306,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i32.nxv8i16(<vscale x 8 x i32>,<vscal
 define void @test_vsoxseg2_nxv8i32_nxv8i16(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i32_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -4376,7 +4319,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i32_nxv8i16(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i32_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -4393,7 +4335,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i32.nxv8i8(<vscale x 8 x i32>,<vscale
 define void @test_vsoxseg2_nxv8i32_nxv8i8(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i32_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -4407,7 +4348,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i32_nxv8i8(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i32_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -4424,7 +4364,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i32.nxv8i32(<vscale x 8 x i32>,<vscal
 define void @test_vsoxseg2_nxv8i32_nxv8i32(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i32_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -4438,7 +4377,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i32_nxv8i32(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i32_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -4455,7 +4393,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i8.nxv4i16(<vscale x 4 x i8>,<vscale
 define void @test_vsoxseg2_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -4469,7 +4406,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -4486,7 +4422,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i8.nxv4i8(<vscale x 4 x i8>,<vscale x
 define void @test_vsoxseg2_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -4500,7 +4435,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -4517,7 +4451,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i8.nxv4i32(<vscale x 4 x i8>,<vscale
 define void @test_vsoxseg2_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v10
@@ -4530,7 +4463,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v10, v0.t
@@ -4547,8 +4479,8 @@ define void @test_vsoxseg3_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg3_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4561,8 +4493,8 @@ define void @test_vsoxseg3_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4578,8 +4510,8 @@ define void @test_vsoxseg3_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg3_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4592,8 +4524,8 @@ define void @test_vsoxseg3_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4608,11 +4540,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv4i8.nxv4i32(<vscale x 4 x i8>,<vscale
 define void @test_vsoxseg3_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl)
@@ -4622,11 +4554,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl)
@@ -4640,9 +4572,9 @@ define void @test_vsoxseg4_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg4_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4655,9 +4587,9 @@ define void @test_vsoxseg4_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4673,9 +4605,9 @@ define void @test_vsoxseg4_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg4_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4688,9 +4620,9 @@ define void @test_vsoxseg4_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4706,9 +4638,9 @@ define void @test_vsoxseg4_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg4_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -4721,9 +4653,9 @@ define void @test_vsoxseg4_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -4739,10 +4671,10 @@ define void @test_vsoxseg5_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg5_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4755,10 +4687,10 @@ define void @test_vsoxseg5_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4774,10 +4706,10 @@ define void @test_vsoxseg5_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg5_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4790,10 +4722,10 @@ define void @test_vsoxseg5_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4809,10 +4741,10 @@ define void @test_vsoxseg5_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg5_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -4825,10 +4757,10 @@ define void @test_vsoxseg5_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -4844,11 +4776,11 @@ define void @test_vsoxseg6_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4861,11 +4793,11 @@ define void @test_vsoxseg6_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4881,11 +4813,11 @@ define void @test_vsoxseg6_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg6_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4898,11 +4830,11 @@ define void @test_vsoxseg6_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4918,11 +4850,11 @@ define void @test_vsoxseg6_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -4935,11 +4867,11 @@ define void @test_vsoxseg6_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -4955,12 +4887,12 @@ define void @test_vsoxseg7_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4973,12 +4905,12 @@ define void @test_vsoxseg7_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4994,12 +4926,12 @@ define void @test_vsoxseg7_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg7_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5012,12 +4944,12 @@ define void @test_vsoxseg7_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5033,12 +4965,12 @@ define void @test_vsoxseg7_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -5051,12 +4983,12 @@ define void @test_vsoxseg7_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -5072,13 +5004,13 @@ define void @test_vsoxseg8_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5091,13 +5023,13 @@ define void @test_vsoxseg8_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5113,13 +5045,13 @@ define void @test_vsoxseg8_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg8_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5132,13 +5064,13 @@ define void @test_vsoxseg8_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5154,13 +5086,13 @@ define void @test_vsoxseg8_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -5173,13 +5105,13 @@ define void @test_vsoxseg8_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -5194,7 +5126,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i16.nxv1i8(<vscale x 1 x i16>,<vscale
 define void @test_vsoxseg2_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -5208,7 +5139,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -5225,7 +5155,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i16.nxv1i32(<vscale x 1 x i16>,<vscal
 define void @test_vsoxseg2_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -5239,7 +5168,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -5256,7 +5184,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i16.nxv1i16(<vscale x 1 x i16>,<vscal
 define void @test_vsoxseg2_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -5270,7 +5197,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -5288,8 +5214,8 @@ define void @test_vsoxseg3_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg3_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5302,8 +5228,8 @@ define void @test_vsoxseg3_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5319,8 +5245,8 @@ define void @test_vsoxseg3_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5333,8 +5259,8 @@ define void @test_vsoxseg3_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5350,8 +5276,8 @@ define void @test_vsoxseg3_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5364,8 +5290,8 @@ define void @test_vsoxseg3_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5381,9 +5307,9 @@ define void @test_vsoxseg4_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg4_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5396,9 +5322,9 @@ define void @test_vsoxseg4_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5414,9 +5340,9 @@ define void @test_vsoxseg4_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5429,9 +5355,9 @@ define void @test_vsoxseg4_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5447,9 +5373,9 @@ define void @test_vsoxseg4_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5462,9 +5388,9 @@ define void @test_vsoxseg4_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5480,10 +5406,10 @@ define void @test_vsoxseg5_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg5_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5496,10 +5422,10 @@ define void @test_vsoxseg5_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5515,10 +5441,10 @@ define void @test_vsoxseg5_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5531,10 +5457,10 @@ define void @test_vsoxseg5_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5550,10 +5476,10 @@ define void @test_vsoxseg5_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5566,10 +5492,10 @@ define void @test_vsoxseg5_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5585,11 +5511,11 @@ define void @test_vsoxseg6_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg6_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5602,11 +5528,11 @@ define void @test_vsoxseg6_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5622,11 +5548,11 @@ define void @test_vsoxseg6_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5639,11 +5565,11 @@ define void @test_vsoxseg6_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5659,11 +5585,11 @@ define void @test_vsoxseg6_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5676,11 +5602,11 @@ define void @test_vsoxseg6_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5696,12 +5622,12 @@ define void @test_vsoxseg7_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg7_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5714,12 +5640,12 @@ define void @test_vsoxseg7_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5735,12 +5661,12 @@ define void @test_vsoxseg7_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5753,12 +5679,12 @@ define void @test_vsoxseg7_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5774,12 +5700,12 @@ define void @test_vsoxseg7_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5792,12 +5718,12 @@ define void @test_vsoxseg7_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5813,13 +5739,13 @@ define void @test_vsoxseg8_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg8_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5832,13 +5758,13 @@ define void @test_vsoxseg8_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5854,13 +5780,13 @@ define void @test_vsoxseg8_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5873,13 +5799,13 @@ define void @test_vsoxseg8_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5895,13 +5821,13 @@ define void @test_vsoxseg8_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5914,13 +5840,13 @@ define void @test_vsoxseg8_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5935,7 +5861,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv32i8.nxv32i16(<vscale x 32 x i8>,<vsca
 define void @test_vsoxseg2_nxv32i8_nxv32i16(<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv32i8_nxv32i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei16.v v8, (a0), v16
@@ -5948,7 +5873,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv32i8_nxv32i16(<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i16> %index, <vscale x 32 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv32i8_nxv32i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei16.v v8, (a0), v16, v0.t
@@ -5964,7 +5888,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv32i8.nxv32i8(<vscale x 32 x i8>,<vscal
 define void @test_vsoxseg2_nxv32i8_nxv32i8(<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv32i8_nxv32i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
@@ -5978,7 +5901,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv32i8_nxv32i8(<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i8> %index, <vscale x 32 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv32i8_nxv32i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
@@ -5995,7 +5917,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i8.nxv2i32(<vscale x 2 x i8>,<vscale
 define void @test_vsoxseg2_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -6009,7 +5930,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -6026,7 +5946,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i8.nxv2i8(<vscale x 2 x i8>,<vscale x
 define void @test_vsoxseg2_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -6040,7 +5959,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -6057,7 +5975,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i8.nxv2i16(<vscale x 2 x i8>,<vscale
 define void @test_vsoxseg2_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -6071,7 +5988,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -6089,8 +6005,8 @@ define void @test_vsoxseg3_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg3_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6103,8 +6019,8 @@ define void @test_vsoxseg3_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6120,8 +6036,8 @@ define void @test_vsoxseg3_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg3_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6134,8 +6050,8 @@ define void @test_vsoxseg3_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6151,8 +6067,8 @@ define void @test_vsoxseg3_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg3_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6165,8 +6081,8 @@ define void @test_vsoxseg3_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6182,9 +6098,9 @@ define void @test_vsoxseg4_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg4_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6197,9 +6113,9 @@ define void @test_vsoxseg4_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6215,9 +6131,9 @@ define void @test_vsoxseg4_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg4_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6230,9 +6146,9 @@ define void @test_vsoxseg4_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6248,9 +6164,9 @@ define void @test_vsoxseg4_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg4_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6263,9 +6179,9 @@ define void @test_vsoxseg4_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6281,10 +6197,10 @@ define void @test_vsoxseg5_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg5_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6297,10 +6213,10 @@ define void @test_vsoxseg5_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6316,10 +6232,10 @@ define void @test_vsoxseg5_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg5_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6332,10 +6248,10 @@ define void @test_vsoxseg5_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6351,10 +6267,10 @@ define void @test_vsoxseg5_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg5_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6367,10 +6283,10 @@ define void @test_vsoxseg5_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6386,11 +6302,11 @@ define void @test_vsoxseg6_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6403,11 +6319,11 @@ define void @test_vsoxseg6_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6423,11 +6339,11 @@ define void @test_vsoxseg6_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg6_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6440,11 +6356,11 @@ define void @test_vsoxseg6_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6460,11 +6376,11 @@ define void @test_vsoxseg6_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6477,11 +6393,11 @@ define void @test_vsoxseg6_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6497,12 +6413,12 @@ define void @test_vsoxseg7_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6515,12 +6431,12 @@ define void @test_vsoxseg7_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6536,12 +6452,12 @@ define void @test_vsoxseg7_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg7_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6554,12 +6470,12 @@ define void @test_vsoxseg7_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6575,12 +6491,12 @@ define void @test_vsoxseg7_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6593,12 +6509,12 @@ define void @test_vsoxseg7_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6614,13 +6530,13 @@ define void @test_vsoxseg8_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6633,13 +6549,13 @@ define void @test_vsoxseg8_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6655,13 +6571,13 @@ define void @test_vsoxseg8_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg8_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6674,13 +6590,13 @@ define void @test_vsoxseg8_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6696,13 +6612,13 @@ define void @test_vsoxseg8_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6715,13 +6631,13 @@ define void @test_vsoxseg8_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6736,7 +6652,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i16.nxv2i32(<vscale x 2 x i16>,<vscal
 define void @test_vsoxseg2_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -6750,7 +6665,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -6767,7 +6681,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i16.nxv2i8(<vscale x 2 x i16>,<vscale
 define void @test_vsoxseg2_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -6781,7 +6694,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -6798,7 +6710,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i16.nxv2i16(<vscale x 2 x i16>,<vscal
 define void @test_vsoxseg2_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -6812,7 +6723,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -6830,8 +6740,8 @@ define void @test_vsoxseg3_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6844,8 +6754,8 @@ define void @test_vsoxseg3_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6861,8 +6771,8 @@ define void @test_vsoxseg3_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg3_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6875,8 +6785,8 @@ define void @test_vsoxseg3_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6892,8 +6802,8 @@ define void @test_vsoxseg3_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6906,8 +6816,8 @@ define void @test_vsoxseg3_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6923,9 +6833,9 @@ define void @test_vsoxseg4_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6938,9 +6848,9 @@ define void @test_vsoxseg4_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6956,9 +6866,9 @@ define void @test_vsoxseg4_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg4_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6971,9 +6881,9 @@ define void @test_vsoxseg4_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6989,9 +6899,9 @@ define void @test_vsoxseg4_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7004,9 +6914,9 @@ define void @test_vsoxseg4_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7022,10 +6932,10 @@ define void @test_vsoxseg5_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7038,10 +6948,10 @@ define void @test_vsoxseg5_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7057,10 +6967,10 @@ define void @test_vsoxseg5_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg5_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7073,10 +6983,10 @@ define void @test_vsoxseg5_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7092,10 +7002,10 @@ define void @test_vsoxseg5_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7108,10 +7018,10 @@ define void @test_vsoxseg5_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7127,11 +7037,11 @@ define void @test_vsoxseg6_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7144,11 +7054,11 @@ define void @test_vsoxseg6_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7164,11 +7074,11 @@ define void @test_vsoxseg6_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg6_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7181,11 +7091,11 @@ define void @test_vsoxseg6_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7201,11 +7111,11 @@ define void @test_vsoxseg6_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7218,11 +7128,11 @@ define void @test_vsoxseg6_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7238,13 +7148,13 @@ define void @test_vsoxseg7_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
 entry:
@@ -7256,12 +7166,12 @@ define void @test_vsoxseg7_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7277,12 +7187,12 @@ define void @test_vsoxseg7_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg7_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7295,12 +7205,12 @@ define void @test_vsoxseg7_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7316,12 +7226,12 @@ define void @test_vsoxseg7_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7334,12 +7244,12 @@ define void @test_vsoxseg7_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7355,13 +7265,13 @@ define void @test_vsoxseg8_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7374,13 +7284,13 @@ define void @test_vsoxseg8_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7396,13 +7306,13 @@ define void @test_vsoxseg8_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg8_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7415,13 +7325,13 @@ define void @test_vsoxseg8_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7437,13 +7347,13 @@ define void @test_vsoxseg8_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7456,13 +7366,13 @@ define void @test_vsoxseg8_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7477,7 +7387,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i32.nxv4i16(<vscale x 4 x i32>,<vscal
 define void @test_vsoxseg2_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -7491,7 +7400,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -7508,7 +7416,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i32.nxv4i8(<vscale x 4 x i32>,<vscale
 define void @test_vsoxseg2_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -7522,7 +7429,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -7539,7 +7445,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i32.nxv4i32(<vscale x 4 x i32>,<vscal
 define void @test_vsoxseg2_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -7553,7 +7458,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -7571,8 +7475,8 @@ define void @test_vsoxseg3_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7585,8 +7489,8 @@ define void @test_vsoxseg3_mask_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7602,8 +7506,8 @@ define void @test_vsoxseg3_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg3_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7616,8 +7520,8 @@ define void @test_vsoxseg3_mask_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7633,8 +7537,8 @@ define void @test_vsoxseg3_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7647,8 +7551,8 @@ define void @test_vsoxseg3_mask_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7664,9 +7568,9 @@ define void @test_vsoxseg4_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7679,9 +7583,9 @@ define void @test_vsoxseg4_mask_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7697,9 +7601,9 @@ define void @test_vsoxseg4_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg4_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7712,9 +7616,9 @@ define void @test_vsoxseg4_mask_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7730,9 +7634,9 @@ define void @test_vsoxseg4_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7745,9 +7649,9 @@ define void @test_vsoxseg4_mask_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7762,7 +7666,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16f16.nxv16i16(<vscale x 16 x half>,<v
 define void @test_vsoxseg2_nxv16f16_nxv16i16(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16f16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -7776,7 +7679,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16f16_nxv16i16(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16f16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -7793,7 +7695,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16f16.nxv16i8(<vscale x 16 x half>,<vs
 define void @test_vsoxseg2_nxv16f16_nxv16i8(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16f16_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -7807,7 +7708,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16f16_nxv16i8(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16f16_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -7824,7 +7724,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16f16.nxv16i32(<vscale x 16 x half>,<v
 define void @test_vsoxseg2_nxv16f16_nxv16i32(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16f16_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v16
@@ -7837,7 +7736,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16f16_nxv16i32(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16f16_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v16, v0.t
@@ -7853,7 +7751,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f64.nxv4i16(<vscale x 4 x double>,<vs
 define void @test_vsoxseg2_nxv4f64_nxv4i16(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f64_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7867,7 +7764,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f64_nxv4i16(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f64_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7884,7 +7780,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f64.nxv4i8(<vscale x 4 x double>,<vsc
 define void @test_vsoxseg2_nxv4f64_nxv4i8(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f64_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7898,7 +7793,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f64_nxv4i8(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f64_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7915,7 +7809,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f64.nxv4i32(<vscale x 4 x double>,<vs
 define void @test_vsoxseg2_nxv4f64_nxv4i32(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f64_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7929,7 +7822,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f64_nxv4i32(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f64_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7946,7 +7838,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f64.nxv1i8(<vscale x 1 x double>,<vsc
 define void @test_vsoxseg2_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -7960,7 +7851,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -7977,7 +7867,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f64.nxv1i32(<vscale x 1 x double>,<vs
 define void @test_vsoxseg2_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -7991,7 +7880,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -8008,7 +7896,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f64.nxv1i16(<vscale x 1 x double>,<vs
 define void @test_vsoxseg2_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -8022,7 +7909,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -8040,8 +7926,8 @@ define void @test_vsoxseg3_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8054,8 +7940,8 @@ define void @test_vsoxseg3_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8071,8 +7957,8 @@ define void @test_vsoxseg3_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8085,8 +7971,8 @@ define void @test_vsoxseg3_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8102,8 +7988,8 @@ define void @test_vsoxseg3_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8116,8 +8002,8 @@ define void @test_vsoxseg3_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8133,9 +8019,9 @@ define void @test_vsoxseg4_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8148,9 +8034,9 @@ define void @test_vsoxseg4_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8166,9 +8052,9 @@ define void @test_vsoxseg4_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8181,9 +8067,9 @@ define void @test_vsoxseg4_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8199,9 +8085,9 @@ define void @test_vsoxseg4_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8214,9 +8100,9 @@ define void @test_vsoxseg4_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8232,10 +8118,10 @@ define void @test_vsoxseg5_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8248,10 +8134,10 @@ define void @test_vsoxseg5_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8267,10 +8153,10 @@ define void @test_vsoxseg5_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8283,10 +8169,10 @@ define void @test_vsoxseg5_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8302,10 +8188,10 @@ define void @test_vsoxseg5_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8318,10 +8204,10 @@ define void @test_vsoxseg5_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8337,11 +8223,11 @@ define void @test_vsoxseg6_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8354,11 +8240,11 @@ define void @test_vsoxseg6_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8374,11 +8260,11 @@ define void @test_vsoxseg6_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8391,11 +8277,11 @@ define void @test_vsoxseg6_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8411,11 +8297,11 @@ define void @test_vsoxseg6_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8428,11 +8314,11 @@ define void @test_vsoxseg6_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8448,12 +8334,12 @@ define void @test_vsoxseg7_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8466,12 +8352,12 @@ define void @test_vsoxseg7_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8487,12 +8373,12 @@ define void @test_vsoxseg7_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8505,12 +8391,12 @@ define void @test_vsoxseg7_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8526,12 +8412,12 @@ define void @test_vsoxseg7_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8544,12 +8430,12 @@ define void @test_vsoxseg7_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8565,13 +8451,13 @@ define void @test_vsoxseg8_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8584,13 +8470,13 @@ define void @test_vsoxseg8_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8606,13 +8492,13 @@ define void @test_vsoxseg8_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8625,13 +8511,13 @@ define void @test_vsoxseg8_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8647,13 +8533,13 @@ define void @test_vsoxseg8_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8666,13 +8552,13 @@ define void @test_vsoxseg8_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8687,7 +8573,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f32.nxv2i32(<vscale x 2 x float>,<vsc
 define void @test_vsoxseg2_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -8701,7 +8586,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -8718,7 +8602,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f32.nxv2i8(<vscale x 2 x float>,<vsca
 define void @test_vsoxseg2_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -8732,7 +8615,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -8749,7 +8631,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f32.nxv2i16(<vscale x 2 x float>,<vsc
 define void @test_vsoxseg2_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -8763,7 +8644,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -8781,8 +8661,8 @@ define void @test_vsoxseg3_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8795,8 +8675,8 @@ define void @test_vsoxseg3_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8812,8 +8692,8 @@ define void @test_vsoxseg3_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8826,8 +8706,8 @@ define void @test_vsoxseg3_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8843,8 +8723,8 @@ define void @test_vsoxseg3_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8857,8 +8737,8 @@ define void @test_vsoxseg3_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8874,9 +8754,9 @@ define void @test_vsoxseg4_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8889,9 +8769,9 @@ define void @test_vsoxseg4_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8907,9 +8787,9 @@ define void @test_vsoxseg4_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8922,9 +8802,9 @@ define void @test_vsoxseg4_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8940,9 +8820,9 @@ define void @test_vsoxseg4_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8955,9 +8835,9 @@ define void @test_vsoxseg4_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8973,10 +8853,10 @@ define void @test_vsoxseg5_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8989,10 +8869,10 @@ define void @test_vsoxseg5_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9008,10 +8888,10 @@ define void @test_vsoxseg5_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9024,10 +8904,10 @@ define void @test_vsoxseg5_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9043,10 +8923,10 @@ define void @test_vsoxseg5_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9059,10 +8939,10 @@ define void @test_vsoxseg5_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9078,11 +8958,11 @@ define void @test_vsoxseg6_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9095,11 +8975,11 @@ define void @test_vsoxseg6_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9115,11 +8995,11 @@ define void @test_vsoxseg6_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9132,11 +9012,11 @@ define void @test_vsoxseg6_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9152,11 +9032,11 @@ define void @test_vsoxseg6_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9169,11 +9049,11 @@ define void @test_vsoxseg6_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9189,12 +9069,12 @@ define void @test_vsoxseg7_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9207,12 +9087,12 @@ define void @test_vsoxseg7_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9228,12 +9108,12 @@ define void @test_vsoxseg7_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9246,12 +9126,12 @@ define void @test_vsoxseg7_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9267,12 +9147,12 @@ define void @test_vsoxseg7_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9285,12 +9165,12 @@ define void @test_vsoxseg7_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9306,13 +9186,13 @@ define void @test_vsoxseg8_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9325,13 +9205,13 @@ define void @test_vsoxseg8_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9347,13 +9227,13 @@ define void @test_vsoxseg8_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9366,13 +9246,13 @@ define void @test_vsoxseg8_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9388,13 +9268,13 @@ define void @test_vsoxseg8_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9407,13 +9287,13 @@ define void @test_vsoxseg8_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9428,7 +9308,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f16.nxv1i8(<vscale x 1 x half>,<vscal
 define void @test_vsoxseg2_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -9442,7 +9321,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -9459,7 +9337,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f16.nxv1i32(<vscale x 1 x half>,<vsca
 define void @test_vsoxseg2_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -9473,7 +9350,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -9490,7 +9366,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f16.nxv1i16(<vscale x 1 x half>,<vsca
 define void @test_vsoxseg2_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -9504,7 +9379,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -9522,8 +9396,8 @@ define void @test_vsoxseg3_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9536,8 +9410,8 @@ define void @test_vsoxseg3_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9553,8 +9427,8 @@ define void @test_vsoxseg3_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9567,8 +9441,8 @@ define void @test_vsoxseg3_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9584,8 +9458,8 @@ define void @test_vsoxseg3_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9598,8 +9472,8 @@ define void @test_vsoxseg3_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9615,9 +9489,9 @@ define void @test_vsoxseg4_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9630,9 +9504,9 @@ define void @test_vsoxseg4_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9648,9 +9522,9 @@ define void @test_vsoxseg4_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9663,9 +9537,9 @@ define void @test_vsoxseg4_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9681,9 +9555,9 @@ define void @test_vsoxseg4_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9696,9 +9570,9 @@ define void @test_vsoxseg4_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9714,10 +9588,10 @@ define void @test_vsoxseg5_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9730,10 +9604,10 @@ define void @test_vsoxseg5_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9749,10 +9623,10 @@ define void @test_vsoxseg5_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9765,10 +9639,10 @@ define void @test_vsoxseg5_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9784,10 +9658,10 @@ define void @test_vsoxseg5_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9800,10 +9674,10 @@ define void @test_vsoxseg5_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9819,11 +9693,11 @@ define void @test_vsoxseg6_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9836,11 +9710,11 @@ define void @test_vsoxseg6_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9856,11 +9730,11 @@ define void @test_vsoxseg6_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9873,11 +9747,11 @@ define void @test_vsoxseg6_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9893,11 +9767,11 @@ define void @test_vsoxseg6_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9910,11 +9784,11 @@ define void @test_vsoxseg6_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9930,12 +9804,12 @@ define void @test_vsoxseg7_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9948,12 +9822,12 @@ define void @test_vsoxseg7_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9969,12 +9843,12 @@ define void @test_vsoxseg7_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9987,12 +9861,12 @@ define void @test_vsoxseg7_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10008,12 +9882,12 @@ define void @test_vsoxseg7_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10026,12 +9900,12 @@ define void @test_vsoxseg7_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10047,13 +9921,13 @@ define void @test_vsoxseg8_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10066,13 +9940,13 @@ define void @test_vsoxseg8_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10088,13 +9962,13 @@ define void @test_vsoxseg8_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10107,13 +9981,13 @@ define void @test_vsoxseg8_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10129,13 +10003,13 @@ define void @test_vsoxseg8_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10148,13 +10022,13 @@ define void @test_vsoxseg8_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10169,7 +10043,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f32.nxv1i8(<vscale x 1 x float>,<vsca
 define void @test_vsoxseg2_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -10183,7 +10056,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -10200,7 +10072,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f32.nxv1i32(<vscale x 1 x float>,<vsc
 define void @test_vsoxseg2_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -10214,7 +10085,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -10231,7 +10101,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f32.nxv1i16(<vscale x 1 x float>,<vsc
 define void @test_vsoxseg2_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -10245,7 +10114,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -10263,8 +10131,8 @@ define void @test_vsoxseg3_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10277,8 +10145,8 @@ define void @test_vsoxseg3_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10294,8 +10162,8 @@ define void @test_vsoxseg3_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10308,8 +10176,8 @@ define void @test_vsoxseg3_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10325,8 +10193,8 @@ define void @test_vsoxseg3_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10339,8 +10207,8 @@ define void @test_vsoxseg3_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10356,9 +10224,9 @@ define void @test_vsoxseg4_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10371,9 +10239,9 @@ define void @test_vsoxseg4_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10389,9 +10257,9 @@ define void @test_vsoxseg4_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10404,9 +10272,9 @@ define void @test_vsoxseg4_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10422,9 +10290,9 @@ define void @test_vsoxseg4_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10437,9 +10305,9 @@ define void @test_vsoxseg4_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10455,10 +10323,10 @@ define void @test_vsoxseg5_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10471,10 +10339,10 @@ define void @test_vsoxseg5_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10490,10 +10358,10 @@ define void @test_vsoxseg5_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10506,10 +10374,10 @@ define void @test_vsoxseg5_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10525,10 +10393,10 @@ define void @test_vsoxseg5_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10541,10 +10409,10 @@ define void @test_vsoxseg5_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10560,11 +10428,11 @@ define void @test_vsoxseg6_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10577,11 +10445,11 @@ define void @test_vsoxseg6_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10597,11 +10465,11 @@ define void @test_vsoxseg6_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10614,11 +10482,11 @@ define void @test_vsoxseg6_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10634,11 +10502,11 @@ define void @test_vsoxseg6_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10651,11 +10519,11 @@ define void @test_vsoxseg6_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10671,12 +10539,12 @@ define void @test_vsoxseg7_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10689,12 +10557,12 @@ define void @test_vsoxseg7_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10710,12 +10578,12 @@ define void @test_vsoxseg7_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10728,12 +10596,12 @@ define void @test_vsoxseg7_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10749,12 +10617,12 @@ define void @test_vsoxseg7_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10767,12 +10635,12 @@ define void @test_vsoxseg7_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10788,13 +10656,13 @@ define void @test_vsoxseg8_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10807,13 +10675,13 @@ define void @test_vsoxseg8_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10829,13 +10697,13 @@ define void @test_vsoxseg8_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10848,13 +10716,13 @@ define void @test_vsoxseg8_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10870,13 +10738,13 @@ define void @test_vsoxseg8_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10889,13 +10757,13 @@ define void @test_vsoxseg8_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10910,7 +10778,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8f16.nxv8i16(<vscale x 8 x half>,<vsca
 define void @test_vsoxseg2_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -10924,7 +10791,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -10941,7 +10807,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8f16.nxv8i8(<vscale x 8 x half>,<vscal
 define void @test_vsoxseg2_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -10955,7 +10820,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -10972,7 +10836,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8f16.nxv8i32(<vscale x 8 x half>,<vsca
 define void @test_vsoxseg2_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v12
@@ -10985,7 +10848,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v12, v0.t
@@ -11002,8 +10864,8 @@ define void @test_vsoxseg3_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11016,8 +10878,8 @@ define void @test_vsoxseg3_mask_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11033,8 +10895,8 @@ define void @test_vsoxseg3_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11047,8 +10909,8 @@ define void @test_vsoxseg3_mask_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11063,11 +10925,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv8f16.nxv8i32(<vscale x 8 x half>,<vsca
 define void @test_vsoxseg3_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v16, (a0), v12
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl)
@@ -11077,11 +10939,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl)
@@ -11095,9 +10957,9 @@ define void @test_vsoxseg4_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11110,9 +10972,9 @@ define void @test_vsoxseg4_mask_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11128,9 +10990,9 @@ define void @test_vsoxseg4_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11143,9 +11005,9 @@ define void @test_vsoxseg4_mask_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11161,9 +11023,9 @@ define void @test_vsoxseg4_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -11176,9 +11038,9 @@ define void @test_vsoxseg4_mask_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -11193,7 +11055,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8f32.nxv8i16(<vscale x 8 x float>,<vsc
 define void @test_vsoxseg2_nxv8f32_nxv8i16(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8f32_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -11207,7 +11068,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8f32_nxv8i16(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8f32_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -11224,7 +11084,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8f32.nxv8i8(<vscale x 8 x float>,<vsca
 define void @test_vsoxseg2_nxv8f32_nxv8i8(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8f32_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -11238,7 +11097,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8f32_nxv8i8(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8f32_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -11255,7 +11113,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8f32.nxv8i32(<vscale x 8 x float>,<vsc
 define void @test_vsoxseg2_nxv8f32_nxv8i32(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8f32_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -11269,7 +11126,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8f32_nxv8i32(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8f32_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -11286,7 +11142,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f64.nxv2i32(<vscale x 2 x double>,<vs
 define void @test_vsoxseg2_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11300,7 +11155,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11317,7 +11171,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f64.nxv2i8(<vscale x 2 x double>,<vsc
 define void @test_vsoxseg2_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11331,7 +11184,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11348,7 +11200,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f64.nxv2i16(<vscale x 2 x double>,<vs
 define void @test_vsoxseg2_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11362,7 +11213,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11380,8 +11230,8 @@ define void @test_vsoxseg3_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11394,8 +11244,8 @@ define void @test_vsoxseg3_mask_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11411,8 +11261,8 @@ define void @test_vsoxseg3_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11425,8 +11275,8 @@ define void @test_vsoxseg3_mask_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11442,8 +11292,8 @@ define void @test_vsoxseg3_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11456,8 +11306,8 @@ define void @test_vsoxseg3_mask_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11473,9 +11323,9 @@ define void @test_vsoxseg4_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11488,9 +11338,9 @@ define void @test_vsoxseg4_mask_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11506,9 +11356,9 @@ define void @test_vsoxseg4_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11521,9 +11371,9 @@ define void @test_vsoxseg4_mask_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11539,9 +11389,9 @@ define void @test_vsoxseg4_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11554,9 +11404,9 @@ define void @test_vsoxseg4_mask_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11571,7 +11421,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f16.nxv4i16(<vscale x 4 x half>,<vsca
 define void @test_vsoxseg2_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -11585,7 +11434,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -11602,7 +11450,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f16.nxv4i8(<vscale x 4 x half>,<vscal
 define void @test_vsoxseg2_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -11616,7 +11463,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -11633,7 +11479,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f16.nxv4i32(<vscale x 4 x half>,<vsca
 define void @test_vsoxseg2_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v10
@@ -11646,7 +11491,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v10, v0.t
@@ -11663,8 +11507,8 @@ define void @test_vsoxseg3_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11677,8 +11521,8 @@ define void @test_vsoxseg3_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11694,8 +11538,8 @@ define void @test_vsoxseg3_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11708,8 +11552,8 @@ define void @test_vsoxseg3_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11724,11 +11568,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv4f16.nxv4i32(<vscale x 4 x half>,<vsca
 define void @test_vsoxseg3_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl)
@@ -11738,11 +11582,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl)
@@ -11756,9 +11600,9 @@ define void @test_vsoxseg4_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11771,9 +11615,9 @@ define void @test_vsoxseg4_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11789,9 +11633,9 @@ define void @test_vsoxseg4_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11804,9 +11648,9 @@ define void @test_vsoxseg4_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11822,9 +11666,9 @@ define void @test_vsoxseg4_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11837,9 +11681,9 @@ define void @test_vsoxseg4_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11855,10 +11699,10 @@ define void @test_vsoxseg5_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11871,10 +11715,10 @@ define void @test_vsoxseg5_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11890,10 +11734,10 @@ define void @test_vsoxseg5_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11906,10 +11750,10 @@ define void @test_vsoxseg5_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11925,10 +11769,10 @@ define void @test_vsoxseg5_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11941,10 +11785,10 @@ define void @test_vsoxseg5_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11960,11 +11804,11 @@ define void @test_vsoxseg6_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11977,11 +11821,11 @@ define void @test_vsoxseg6_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11997,11 +11841,11 @@ define void @test_vsoxseg6_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12014,11 +11858,11 @@ define void @test_vsoxseg6_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12034,11 +11878,11 @@ define void @test_vsoxseg6_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -12051,11 +11895,11 @@ define void @test_vsoxseg6_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -12071,12 +11915,12 @@ define void @test_vsoxseg7_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12089,12 +11933,12 @@ define void @test_vsoxseg7_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12110,12 +11954,12 @@ define void @test_vsoxseg7_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12128,12 +11972,12 @@ define void @test_vsoxseg7_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12149,12 +11993,12 @@ define void @test_vsoxseg7_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -12167,12 +12011,12 @@ define void @test_vsoxseg7_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -12188,13 +12032,13 @@ define void @test_vsoxseg8_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12207,13 +12051,13 @@ define void @test_vsoxseg8_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12229,13 +12073,13 @@ define void @test_vsoxseg8_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12248,13 +12092,13 @@ define void @test_vsoxseg8_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12270,13 +12114,13 @@ define void @test_vsoxseg8_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -12289,13 +12133,13 @@ define void @test_vsoxseg8_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -12310,7 +12154,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f16.nxv2i32(<vscale x 2 x half>,<vsca
 define void @test_vsoxseg2_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -12324,7 +12167,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -12341,7 +12183,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f16.nxv2i8(<vscale x 2 x half>,<vscal
 define void @test_vsoxseg2_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -12355,7 +12196,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -12372,7 +12212,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f16.nxv2i16(<vscale x 2 x half>,<vsca
 define void @test_vsoxseg2_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -12386,7 +12225,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -12404,8 +12242,8 @@ define void @test_vsoxseg3_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12418,8 +12256,8 @@ define void @test_vsoxseg3_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12435,8 +12273,8 @@ define void @test_vsoxseg3_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12449,8 +12287,8 @@ define void @test_vsoxseg3_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12466,8 +12304,8 @@ define void @test_vsoxseg3_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12480,8 +12318,8 @@ define void @test_vsoxseg3_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12497,9 +12335,9 @@ define void @test_vsoxseg4_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12512,9 +12350,9 @@ define void @test_vsoxseg4_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12530,9 +12368,9 @@ define void @test_vsoxseg4_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12545,9 +12383,9 @@ define void @test_vsoxseg4_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12563,9 +12401,9 @@ define void @test_vsoxseg4_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12578,9 +12416,9 @@ define void @test_vsoxseg4_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12596,10 +12434,10 @@ define void @test_vsoxseg5_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12612,10 +12450,10 @@ define void @test_vsoxseg5_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12631,10 +12469,10 @@ define void @test_vsoxseg5_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12647,10 +12485,10 @@ define void @test_vsoxseg5_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12666,10 +12504,10 @@ define void @test_vsoxseg5_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12682,10 +12520,10 @@ define void @test_vsoxseg5_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12701,11 +12539,11 @@ define void @test_vsoxseg6_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12718,11 +12556,11 @@ define void @test_vsoxseg6_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12738,11 +12576,11 @@ define void @test_vsoxseg6_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12755,11 +12593,11 @@ define void @test_vsoxseg6_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12775,11 +12613,11 @@ define void @test_vsoxseg6_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12792,11 +12630,11 @@ define void @test_vsoxseg6_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12812,12 +12650,12 @@ define void @test_vsoxseg7_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12830,12 +12668,12 @@ define void @test_vsoxseg7_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12851,12 +12689,12 @@ define void @test_vsoxseg7_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12869,12 +12707,12 @@ define void @test_vsoxseg7_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12890,12 +12728,12 @@ define void @test_vsoxseg7_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12908,12 +12746,12 @@ define void @test_vsoxseg7_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12929,13 +12767,13 @@ define void @test_vsoxseg8_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12948,13 +12786,13 @@ define void @test_vsoxseg8_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12970,13 +12808,13 @@ define void @test_vsoxseg8_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12989,13 +12827,13 @@ define void @test_vsoxseg8_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13011,13 +12849,13 @@ define void @test_vsoxseg8_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13030,13 +12868,13 @@ define void @test_vsoxseg8_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13051,7 +12889,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f32.nxv4i16(<vscale x 4 x float>,<vsc
 define void @test_vsoxseg2_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -13065,7 +12902,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -13082,7 +12918,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f32.nxv4i8(<vscale x 4 x float>,<vsca
 define void @test_vsoxseg2_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -13096,7 +12931,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -13113,7 +12947,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f32.nxv4i32(<vscale x 4 x float>,<vsc
 define void @test_vsoxseg2_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -13127,7 +12960,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -13145,8 +12977,8 @@ define void @test_vsoxseg3_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13159,8 +12991,8 @@ define void @test_vsoxseg3_mask_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13176,8 +13008,8 @@ define void @test_vsoxseg3_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13190,8 +13022,8 @@ define void @test_vsoxseg3_mask_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13207,8 +13039,8 @@ define void @test_vsoxseg3_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13221,8 +13053,8 @@ define void @test_vsoxseg3_mask_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13238,9 +13070,9 @@ define void @test_vsoxseg4_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13253,9 +13085,9 @@ define void @test_vsoxseg4_mask_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13271,9 +13103,9 @@ define void @test_vsoxseg4_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13286,9 +13118,9 @@ define void @test_vsoxseg4_mask_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13304,9 +13136,9 @@ define void @test_vsoxseg4_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13319,9 +13151,9 @@ define void @test_vsoxseg4_mask_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/rvv/vsoxseg-rv64.ll b/llvm/test/CodeGen/RISCV/rvv/vsoxseg-rv64.ll
index 7f84323b0e022..dacf1f99f242d 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vsoxseg-rv64.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vsoxseg-rv64.ll
@@ -8,7 +8,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16i16.nxv16i16(<vscale x 16 x i16>,<vs
 define void @test_vsoxseg2_nxv16i16_nxv16i16(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16i16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -22,7 +21,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16i16_nxv16i16(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16i16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -39,7 +37,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16i16.nxv16i8(<vscale x 16 x i16>,<vsc
 define void @test_vsoxseg2_nxv16i16_nxv16i8(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16i16_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -53,7 +50,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16i16_nxv16i8(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16i16_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -70,7 +66,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16i16.nxv16i32(<vscale x 16 x i16>,<vs
 define void @test_vsoxseg2_nxv16i16_nxv16i32(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16i16_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v16
@@ -83,7 +78,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16i16_nxv16i32(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16i16_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v16, v0.t
@@ -99,7 +93,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i32.nxv4i32(<vscale x 4 x i32>,<vscal
 define void @test_vsoxseg2_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -113,7 +106,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -130,7 +122,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i32.nxv4i8(<vscale x 4 x i32>,<vscale
 define void @test_vsoxseg2_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -144,7 +135,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -161,7 +151,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i32.nxv4i64(<vscale x 4 x i32>,<vscal
 define void @test_vsoxseg2_nxv4i32_nxv4i64(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v12
@@ -174,7 +163,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i32_nxv4i64(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v12, v0.t
@@ -190,7 +178,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i32.nxv4i16(<vscale x 4 x i32>,<vscal
 define void @test_vsoxseg2_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -204,7 +191,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -222,8 +208,8 @@ define void @test_vsoxseg3_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -236,8 +222,8 @@ define void @test_vsoxseg3_mask_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -253,8 +239,8 @@ define void @test_vsoxseg3_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg3_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -267,8 +253,8 @@ define void @test_vsoxseg3_mask_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -283,11 +269,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv4i32.nxv4i64(<vscale x 4 x i32>,<vscal
 define void @test_vsoxseg3_nxv4i32_nxv4i64(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv4i32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei64.v v16, (a0), v12
+; CHECK-NEXT:    vsoxseg3ei64.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv4i32.nxv4i64(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl)
@@ -297,11 +283,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv4i32_nxv4i64(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei64.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsoxseg3ei64.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv4i32.nxv4i64(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl)
@@ -315,8 +301,8 @@ define void @test_vsoxseg3_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -329,8 +315,8 @@ define void @test_vsoxseg3_mask_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -346,9 +332,9 @@ define void @test_vsoxseg4_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -361,9 +347,9 @@ define void @test_vsoxseg4_mask_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -379,9 +365,9 @@ define void @test_vsoxseg4_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg4_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -394,9 +380,9 @@ define void @test_vsoxseg4_mask_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -412,9 +398,9 @@ define void @test_vsoxseg4_nxv4i32_nxv4i64(<vscale x 4 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv4i32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -427,9 +413,9 @@ define void @test_vsoxseg4_mask_nxv4i32_nxv4i64(<vscale x 4 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -445,9 +431,9 @@ define void @test_vsoxseg4_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -460,9 +446,9 @@ define void @test_vsoxseg4_mask_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -477,7 +463,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16i8.nxv16i16(<vscale x 16 x i8>,<vsca
 define void @test_vsoxseg2_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei16.v v8, (a0), v12
@@ -490,7 +475,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei16.v v8, (a0), v12, v0.t
@@ -506,7 +490,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16i8.nxv16i8(<vscale x 16 x i8>,<vscal
 define void @test_vsoxseg2_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -520,7 +503,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -537,7 +519,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16i8.nxv16i32(<vscale x 16 x i8>,<vsca
 define void @test_vsoxseg2_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v16
@@ -550,7 +531,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v16, v0.t
@@ -566,11 +546,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv16i8.nxv16i16(<vscale x 16 x i8>,<vsca
 define void @test_vsoxseg3_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei16.v v16, (a0), v12
+; CHECK-NEXT:    vsoxseg3ei16.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, i64 %vl)
@@ -580,11 +560,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei16.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsoxseg3ei16.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl)
@@ -598,8 +578,8 @@ define void @test_vsoxseg3_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -612,8 +592,8 @@ define void @test_vsoxseg3_mask_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -628,7 +608,6 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv16i8.nxv16i32(<vscale x 16 x i8>,<vsca
 define void @test_vsoxseg3_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -642,7 +621,6 @@ entry:
 define void @test_vsoxseg3_mask_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -660,9 +638,9 @@ define void @test_vsoxseg4_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -675,9 +653,9 @@ define void @test_vsoxseg4_mask_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -693,9 +671,9 @@ define void @test_vsoxseg4_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -708,9 +686,9 @@ define void @test_vsoxseg4_mask_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -725,7 +703,6 @@ declare void @llvm.riscv.vsoxseg4.mask.nxv16i8.nxv16i32(<vscale x 16 x i8>,<vsca
 define void @test_vsoxseg4_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -740,7 +717,6 @@ entry:
 define void @test_vsoxseg4_mask_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -758,7 +734,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i64.nxv1i64(<vscale x 1 x i64>,<vscal
 define void @test_vsoxseg2_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -772,7 +747,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -789,7 +763,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i64.nxv1i32(<vscale x 1 x i64>,<vscal
 define void @test_vsoxseg2_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -803,7 +776,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -820,7 +792,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i64.nxv1i16(<vscale x 1 x i64>,<vscal
 define void @test_vsoxseg2_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -834,7 +805,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -851,7 +821,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i64.nxv1i8(<vscale x 1 x i64>,<vscale
 define void @test_vsoxseg2_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -865,7 +834,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -883,8 +851,8 @@ define void @test_vsoxseg3_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -897,8 +865,8 @@ define void @test_vsoxseg3_mask_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -914,8 +882,8 @@ define void @test_vsoxseg3_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -928,8 +896,8 @@ define void @test_vsoxseg3_mask_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -945,8 +913,8 @@ define void @test_vsoxseg3_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -959,8 +927,8 @@ define void @test_vsoxseg3_mask_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -976,8 +944,8 @@ define void @test_vsoxseg3_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg3_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -990,8 +958,8 @@ define void @test_vsoxseg3_mask_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1007,9 +975,9 @@ define void @test_vsoxseg4_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1022,9 +990,9 @@ define void @test_vsoxseg4_mask_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1040,9 +1008,9 @@ define void @test_vsoxseg4_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1055,9 +1023,9 @@ define void @test_vsoxseg4_mask_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1073,9 +1041,9 @@ define void @test_vsoxseg4_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1088,9 +1056,9 @@ define void @test_vsoxseg4_mask_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1106,9 +1074,9 @@ define void @test_vsoxseg4_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg4_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1121,9 +1089,9 @@ define void @test_vsoxseg4_mask_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1139,10 +1107,10 @@ define void @test_vsoxseg5_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1155,10 +1123,10 @@ define void @test_vsoxseg5_mask_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1174,10 +1142,10 @@ define void @test_vsoxseg5_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1190,10 +1158,10 @@ define void @test_vsoxseg5_mask_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1209,10 +1177,10 @@ define void @test_vsoxseg5_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1225,10 +1193,10 @@ define void @test_vsoxseg5_mask_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1244,10 +1212,10 @@ define void @test_vsoxseg5_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg5_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1260,10 +1228,10 @@ define void @test_vsoxseg5_mask_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1279,11 +1247,11 @@ define void @test_vsoxseg6_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1296,11 +1264,11 @@ define void @test_vsoxseg6_mask_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1316,11 +1284,11 @@ define void @test_vsoxseg6_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1333,11 +1301,11 @@ define void @test_vsoxseg6_mask_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1353,11 +1321,11 @@ define void @test_vsoxseg6_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1370,11 +1338,11 @@ define void @test_vsoxseg6_mask_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1390,11 +1358,11 @@ define void @test_vsoxseg6_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg6_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1407,11 +1375,11 @@ define void @test_vsoxseg6_mask_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1427,12 +1395,12 @@ define void @test_vsoxseg7_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1445,12 +1413,12 @@ define void @test_vsoxseg7_mask_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1466,12 +1434,12 @@ define void @test_vsoxseg7_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1484,12 +1452,12 @@ define void @test_vsoxseg7_mask_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1505,12 +1473,12 @@ define void @test_vsoxseg7_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1523,12 +1491,12 @@ define void @test_vsoxseg7_mask_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1544,15 +1512,15 @@ define void @test_vsoxseg7_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg7_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
-; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
-; CHECK-NEXT:    ret
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
+; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
+; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg7.nxv1i64.nxv1i8(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl)
   ret void
@@ -1562,12 +1530,12 @@ define void @test_vsoxseg7_mask_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1583,13 +1551,13 @@ define void @test_vsoxseg8_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1602,13 +1570,13 @@ define void @test_vsoxseg8_mask_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1624,13 +1592,13 @@ define void @test_vsoxseg8_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1643,13 +1611,13 @@ define void @test_vsoxseg8_mask_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1665,13 +1633,13 @@ define void @test_vsoxseg8_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1684,13 +1652,13 @@ define void @test_vsoxseg8_mask_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1706,13 +1674,13 @@ define void @test_vsoxseg8_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg8_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1725,13 +1693,13 @@ define void @test_vsoxseg8_mask_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1746,7 +1714,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i32.nxv1i64(<vscale x 1 x i32>,<vscal
 define void @test_vsoxseg2_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -1760,7 +1727,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -1777,7 +1743,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i32.nxv1i32(<vscale x 1 x i32>,<vscal
 define void @test_vsoxseg2_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -1791,7 +1756,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -1808,7 +1772,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i32.nxv1i16(<vscale x 1 x i32>,<vscal
 define void @test_vsoxseg2_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -1822,7 +1785,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -1839,7 +1801,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i32.nxv1i8(<vscale x 1 x i32>,<vscale
 define void @test_vsoxseg2_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -1853,7 +1814,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -1871,8 +1831,8 @@ define void @test_vsoxseg3_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1885,8 +1845,8 @@ define void @test_vsoxseg3_mask_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1902,8 +1862,8 @@ define void @test_vsoxseg3_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1916,8 +1876,8 @@ define void @test_vsoxseg3_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1933,8 +1893,8 @@ define void @test_vsoxseg3_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1947,8 +1907,8 @@ define void @test_vsoxseg3_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1964,8 +1924,8 @@ define void @test_vsoxseg3_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg3_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1978,8 +1938,8 @@ define void @test_vsoxseg3_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1995,9 +1955,9 @@ define void @test_vsoxseg4_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2010,9 +1970,9 @@ define void @test_vsoxseg4_mask_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2028,9 +1988,9 @@ define void @test_vsoxseg4_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2043,9 +2003,9 @@ define void @test_vsoxseg4_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2061,9 +2021,9 @@ define void @test_vsoxseg4_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2076,9 +2036,9 @@ define void @test_vsoxseg4_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2094,9 +2054,9 @@ define void @test_vsoxseg4_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg4_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2109,9 +2069,9 @@ define void @test_vsoxseg4_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2127,10 +2087,10 @@ define void @test_vsoxseg5_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2143,10 +2103,10 @@ define void @test_vsoxseg5_mask_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2162,10 +2122,10 @@ define void @test_vsoxseg5_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2178,10 +2138,10 @@ define void @test_vsoxseg5_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2197,10 +2157,10 @@ define void @test_vsoxseg5_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2213,10 +2173,10 @@ define void @test_vsoxseg5_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2232,10 +2192,10 @@ define void @test_vsoxseg5_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg5_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2248,10 +2208,10 @@ define void @test_vsoxseg5_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2267,11 +2227,11 @@ define void @test_vsoxseg6_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2284,11 +2244,11 @@ define void @test_vsoxseg6_mask_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2304,11 +2264,11 @@ define void @test_vsoxseg6_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2321,11 +2281,11 @@ define void @test_vsoxseg6_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2341,11 +2301,11 @@ define void @test_vsoxseg6_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2358,11 +2318,11 @@ define void @test_vsoxseg6_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2378,11 +2338,11 @@ define void @test_vsoxseg6_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg6_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2395,11 +2355,11 @@ define void @test_vsoxseg6_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2415,12 +2375,12 @@ define void @test_vsoxseg7_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2433,12 +2393,12 @@ define void @test_vsoxseg7_mask_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2454,12 +2414,12 @@ define void @test_vsoxseg7_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2472,12 +2432,12 @@ define void @test_vsoxseg7_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2493,12 +2453,12 @@ define void @test_vsoxseg7_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2511,16 +2471,16 @@ define void @test_vsoxseg7_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
-; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    ret
-entry:
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
+; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
+; CHECK-NEXT:    ret
+entry:
   tail call void @llvm.riscv.vsoxseg7.mask.nxv1i32.nxv1i16(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl)
   ret void
 }
@@ -2532,12 +2492,12 @@ define void @test_vsoxseg7_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg7_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2550,12 +2510,12 @@ define void @test_vsoxseg7_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2571,13 +2531,13 @@ define void @test_vsoxseg8_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2590,13 +2550,13 @@ define void @test_vsoxseg8_mask_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2612,13 +2572,13 @@ define void @test_vsoxseg8_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2631,13 +2591,13 @@ define void @test_vsoxseg8_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2653,13 +2613,13 @@ define void @test_vsoxseg8_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2672,13 +2632,13 @@ define void @test_vsoxseg8_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2694,13 +2654,13 @@ define void @test_vsoxseg8_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg8_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2713,13 +2673,13 @@ define void @test_vsoxseg8_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2734,7 +2694,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i16.nxv8i16(<vscale x 8 x i16>,<vscal
 define void @test_vsoxseg2_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -2748,7 +2707,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -2765,7 +2723,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i16.nxv8i8(<vscale x 8 x i16>,<vscale
 define void @test_vsoxseg2_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -2779,7 +2736,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -2796,7 +2752,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i16.nxv8i64(<vscale x 8 x i16>,<vscal
 define void @test_vsoxseg2_nxv8i16_nxv8i64(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v16
@@ -2809,7 +2764,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i16_nxv8i64(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v16, v0.t
@@ -2825,7 +2779,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i16.nxv8i32(<vscale x 8 x i16>,<vscal
 define void @test_vsoxseg2_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v12
@@ -2838,7 +2791,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v12, v0.t
@@ -2855,8 +2807,8 @@ define void @test_vsoxseg3_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -2869,8 +2821,8 @@ define void @test_vsoxseg3_mask_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -2886,8 +2838,8 @@ define void @test_vsoxseg3_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg3_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -2900,8 +2852,8 @@ define void @test_vsoxseg3_mask_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -2916,7 +2868,6 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv8i16.nxv8i64(<vscale x 8 x i16>,<vscal
 define void @test_vsoxseg3_nxv8i16_nxv8i64(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv8i16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -2930,7 +2881,6 @@ entry:
 define void @test_vsoxseg3_mask_nxv8i16_nxv8i64(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8i16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -2947,11 +2897,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv8i16.nxv8i32(<vscale x 8 x i16>,<vscal
 define void @test_vsoxseg3_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v16, (a0), v12
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl)
@@ -2961,11 +2911,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl)
@@ -2979,9 +2929,9 @@ define void @test_vsoxseg4_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -2994,9 +2944,9 @@ define void @test_vsoxseg4_mask_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3012,9 +2962,9 @@ define void @test_vsoxseg4_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg4_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3027,9 +2977,9 @@ define void @test_vsoxseg4_mask_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3044,7 +2994,6 @@ declare void @llvm.riscv.vsoxseg4.mask.nxv8i16.nxv8i64(<vscale x 8 x i16>,<vscal
 define void @test_vsoxseg4_nxv8i16_nxv8i64(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_nxv8i16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -3059,7 +3008,6 @@ entry:
 define void @test_vsoxseg4_mask_nxv8i16_nxv8i64(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8i16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -3078,9 +3026,9 @@ define void @test_vsoxseg4_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -3093,9 +3041,9 @@ define void @test_vsoxseg4_mask_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -3110,7 +3058,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i8.nxv4i32(<vscale x 4 x i8>,<vscale
 define void @test_vsoxseg2_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v10
@@ -3123,7 +3070,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v10, v0.t
@@ -3139,7 +3085,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i8.nxv4i8(<vscale x 4 x i8>,<vscale x
 define void @test_vsoxseg2_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -3153,7 +3098,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -3170,7 +3114,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i8.nxv4i64(<vscale x 4 x i8>,<vscale
 define void @test_vsoxseg2_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v12
@@ -3183,7 +3126,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v12, v0.t
@@ -3199,7 +3141,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i8.nxv4i16(<vscale x 4 x i8>,<vscale
 define void @test_vsoxseg2_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -3213,7 +3154,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -3230,11 +3170,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv4i8.nxv4i32(<vscale x 4 x i8>,<vscale
 define void @test_vsoxseg3_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl)
@@ -3244,11 +3184,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl)
@@ -3262,8 +3202,8 @@ define void @test_vsoxseg3_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg3_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3276,8 +3216,8 @@ define void @test_vsoxseg3_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3292,7 +3232,6 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv4i8.nxv4i64(<vscale x 4 x i8>,<vscale
 define void @test_vsoxseg3_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -3306,7 +3245,6 @@ entry:
 define void @test_vsoxseg3_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -3324,8 +3262,8 @@ define void @test_vsoxseg3_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg3_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3338,8 +3276,8 @@ define void @test_vsoxseg3_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3355,9 +3293,9 @@ define void @test_vsoxseg4_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg4_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3370,9 +3308,9 @@ define void @test_vsoxseg4_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3388,9 +3326,9 @@ define void @test_vsoxseg4_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg4_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3403,9 +3341,9 @@ define void @test_vsoxseg4_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3420,7 +3358,6 @@ declare void @llvm.riscv.vsoxseg4.mask.nxv4i8.nxv4i64(<vscale x 4 x i8>,<vscale
 define void @test_vsoxseg4_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3435,7 +3372,6 @@ entry:
 define void @test_vsoxseg4_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3454,9 +3390,9 @@ define void @test_vsoxseg4_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg4_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3469,9 +3405,9 @@ define void @test_vsoxseg4_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3487,10 +3423,10 @@ define void @test_vsoxseg5_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg5_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3503,10 +3439,10 @@ define void @test_vsoxseg5_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3522,10 +3458,10 @@ define void @test_vsoxseg5_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg5_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3538,10 +3474,10 @@ define void @test_vsoxseg5_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3556,13 +3492,13 @@ declare void @llvm.riscv.vsoxseg5.mask.nxv4i8.nxv4i64(<vscale x 4 x i8>,<vscale
 define void @test_vsoxseg5_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg5_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
-; CHECK-NEXT:    vsoxseg5ei64.v v16, (a0), v12
+; CHECK-NEXT:    vsoxseg5ei64.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg5.nxv4i8.nxv4i64(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl)
@@ -3572,13 +3508,13 @@ entry:
 define void @test_vsoxseg5_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
-; CHECK-NEXT:    vsoxseg5ei64.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsoxseg5ei64.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg5.mask.nxv4i8.nxv4i64(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl)
@@ -3592,10 +3528,10 @@ define void @test_vsoxseg5_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg5_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3608,10 +3544,10 @@ define void @test_vsoxseg5_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3627,11 +3563,11 @@ define void @test_vsoxseg6_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3644,11 +3580,11 @@ define void @test_vsoxseg6_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3664,11 +3600,11 @@ define void @test_vsoxseg6_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg6_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3681,11 +3617,11 @@ define void @test_vsoxseg6_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3701,11 +3637,11 @@ define void @test_vsoxseg6_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -3718,11 +3654,11 @@ define void @test_vsoxseg6_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -3738,11 +3674,11 @@ define void @test_vsoxseg6_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3755,11 +3691,11 @@ define void @test_vsoxseg6_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3775,12 +3711,12 @@ define void @test_vsoxseg7_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3793,12 +3729,12 @@ define void @test_vsoxseg7_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3814,12 +3750,12 @@ define void @test_vsoxseg7_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg7_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3832,12 +3768,12 @@ define void @test_vsoxseg7_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3853,12 +3789,12 @@ define void @test_vsoxseg7_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -3871,12 +3807,12 @@ define void @test_vsoxseg7_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -3892,12 +3828,12 @@ define void @test_vsoxseg7_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3910,12 +3846,12 @@ define void @test_vsoxseg7_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3931,13 +3867,13 @@ define void @test_vsoxseg8_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3950,13 +3886,13 @@ define void @test_vsoxseg8_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3972,13 +3908,13 @@ define void @test_vsoxseg8_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg8_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3991,13 +3927,13 @@ define void @test_vsoxseg8_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4013,13 +3949,13 @@ define void @test_vsoxseg8_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -4032,13 +3968,13 @@ define void @test_vsoxseg8_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -4054,13 +3990,13 @@ define void @test_vsoxseg8_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4073,13 +4009,13 @@ define void @test_vsoxseg8_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4094,7 +4030,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i16.nxv1i64(<vscale x 1 x i16>,<vscal
 define void @test_vsoxseg2_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -4108,7 +4043,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -4125,7 +4059,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i16.nxv1i32(<vscale x 1 x i16>,<vscal
 define void @test_vsoxseg2_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -4139,7 +4072,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -4156,7 +4088,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i16.nxv1i16(<vscale x 1 x i16>,<vscal
 define void @test_vsoxseg2_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -4170,7 +4101,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -4187,7 +4117,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i16.nxv1i8(<vscale x 1 x i16>,<vscale
 define void @test_vsoxseg2_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -4201,7 +4130,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -4219,8 +4147,8 @@ define void @test_vsoxseg3_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4233,8 +4161,8 @@ define void @test_vsoxseg3_mask_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4250,8 +4178,8 @@ define void @test_vsoxseg3_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4264,8 +4192,8 @@ define void @test_vsoxseg3_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4281,8 +4209,8 @@ define void @test_vsoxseg3_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4295,8 +4223,8 @@ define void @test_vsoxseg3_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4312,8 +4240,8 @@ define void @test_vsoxseg3_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg3_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4326,8 +4254,8 @@ define void @test_vsoxseg3_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4343,9 +4271,9 @@ define void @test_vsoxseg4_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4358,9 +4286,9 @@ define void @test_vsoxseg4_mask_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4376,9 +4304,9 @@ define void @test_vsoxseg4_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4391,9 +4319,9 @@ define void @test_vsoxseg4_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4409,9 +4337,9 @@ define void @test_vsoxseg4_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4424,9 +4352,9 @@ define void @test_vsoxseg4_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4442,9 +4370,9 @@ define void @test_vsoxseg4_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg4_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4457,9 +4385,9 @@ define void @test_vsoxseg4_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4475,10 +4403,10 @@ define void @test_vsoxseg5_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4491,10 +4419,10 @@ define void @test_vsoxseg5_mask_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4510,10 +4438,10 @@ define void @test_vsoxseg5_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4526,10 +4454,10 @@ define void @test_vsoxseg5_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4545,10 +4473,10 @@ define void @test_vsoxseg5_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4561,10 +4489,10 @@ define void @test_vsoxseg5_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4580,10 +4508,10 @@ define void @test_vsoxseg5_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg5_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4596,10 +4524,10 @@ define void @test_vsoxseg5_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4615,11 +4543,11 @@ define void @test_vsoxseg6_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4632,11 +4560,11 @@ define void @test_vsoxseg6_mask_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4652,11 +4580,11 @@ define void @test_vsoxseg6_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4669,11 +4597,11 @@ define void @test_vsoxseg6_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4689,11 +4617,11 @@ define void @test_vsoxseg6_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4706,11 +4634,11 @@ define void @test_vsoxseg6_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4726,11 +4654,11 @@ define void @test_vsoxseg6_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg6_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4743,11 +4671,11 @@ define void @test_vsoxseg6_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4763,12 +4691,12 @@ define void @test_vsoxseg7_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4781,12 +4709,12 @@ define void @test_vsoxseg7_mask_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4802,12 +4730,12 @@ define void @test_vsoxseg7_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4820,12 +4748,12 @@ define void @test_vsoxseg7_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4841,12 +4769,12 @@ define void @test_vsoxseg7_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4859,12 +4787,12 @@ define void @test_vsoxseg7_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4880,12 +4808,12 @@ define void @test_vsoxseg7_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg7_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4898,16 +4826,16 @@ define void @test_vsoxseg7_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
-; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    ret
-entry:
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
+; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
+; CHECK-NEXT:    ret
+entry:
   tail call void @llvm.riscv.vsoxseg7.mask.nxv1i16.nxv1i8(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl)
   ret void
 }
@@ -4919,13 +4847,13 @@ define void @test_vsoxseg8_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4938,13 +4866,13 @@ define void @test_vsoxseg8_mask_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4960,13 +4888,13 @@ define void @test_vsoxseg8_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4979,13 +4907,13 @@ define void @test_vsoxseg8_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5001,13 +4929,13 @@ define void @test_vsoxseg8_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5020,13 +4948,13 @@ define void @test_vsoxseg8_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5042,13 +4970,13 @@ define void @test_vsoxseg8_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg8_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5061,13 +4989,13 @@ define void @test_vsoxseg8_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5082,7 +5010,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i32.nxv2i32(<vscale x 2 x i32>,<vscal
 define void @test_vsoxseg2_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -5096,7 +5023,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -5113,7 +5039,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i32.nxv2i8(<vscale x 2 x i32>,<vscale
 define void @test_vsoxseg2_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -5127,7 +5052,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -5144,7 +5068,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i32.nxv2i16(<vscale x 2 x i32>,<vscal
 define void @test_vsoxseg2_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -5158,7 +5081,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -5175,7 +5097,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i32.nxv2i64(<vscale x 2 x i32>,<vscal
 define void @test_vsoxseg2_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v10
@@ -5188,7 +5109,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v10, v0.t
@@ -5205,8 +5125,8 @@ define void @test_vsoxseg3_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5219,8 +5139,8 @@ define void @test_vsoxseg3_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5236,8 +5156,8 @@ define void @test_vsoxseg3_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg3_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5250,8 +5170,8 @@ define void @test_vsoxseg3_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5267,8 +5187,8 @@ define void @test_vsoxseg3_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5281,8 +5201,8 @@ define void @test_vsoxseg3_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5297,11 +5217,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv2i32.nxv2i64(<vscale x 2 x i32>,<vscal
 define void @test_vsoxseg3_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
-; CHECK-NEXT:    vsoxseg3ei64.v v12, (a0), v10
+; CHECK-NEXT:    vsoxseg3ei64.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv2i32.nxv2i64(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl)
@@ -5311,11 +5231,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
-; CHECK-NEXT:    vsoxseg3ei64.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsoxseg3ei64.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv2i32.nxv2i64(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl)
@@ -5329,9 +5249,9 @@ define void @test_vsoxseg4_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5344,9 +5264,9 @@ define void @test_vsoxseg4_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5362,9 +5282,9 @@ define void @test_vsoxseg4_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg4_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5377,9 +5297,9 @@ define void @test_vsoxseg4_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5395,9 +5315,9 @@ define void @test_vsoxseg4_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5410,9 +5330,9 @@ define void @test_vsoxseg4_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5428,9 +5348,9 @@ define void @test_vsoxseg4_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -5443,9 +5363,9 @@ define void @test_vsoxseg4_mask_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -5461,10 +5381,10 @@ define void @test_vsoxseg5_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5477,10 +5397,10 @@ define void @test_vsoxseg5_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5496,10 +5416,10 @@ define void @test_vsoxseg5_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg5_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5512,10 +5432,10 @@ define void @test_vsoxseg5_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5531,10 +5451,10 @@ define void @test_vsoxseg5_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5547,10 +5467,10 @@ define void @test_vsoxseg5_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5566,10 +5486,10 @@ define void @test_vsoxseg5_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -5582,10 +5502,10 @@ define void @test_vsoxseg5_mask_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -5601,11 +5521,11 @@ define void @test_vsoxseg6_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5618,11 +5538,11 @@ define void @test_vsoxseg6_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5638,11 +5558,11 @@ define void @test_vsoxseg6_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg6_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5655,11 +5575,11 @@ define void @test_vsoxseg6_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5675,11 +5595,11 @@ define void @test_vsoxseg6_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5692,11 +5612,11 @@ define void @test_vsoxseg6_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5712,11 +5632,11 @@ define void @test_vsoxseg6_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -5729,11 +5649,11 @@ define void @test_vsoxseg6_mask_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -5749,12 +5669,12 @@ define void @test_vsoxseg7_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5767,12 +5687,12 @@ define void @test_vsoxseg7_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5788,12 +5708,12 @@ define void @test_vsoxseg7_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg7_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5806,12 +5726,12 @@ define void @test_vsoxseg7_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5827,12 +5747,12 @@ define void @test_vsoxseg7_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5845,12 +5765,12 @@ define void @test_vsoxseg7_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5866,12 +5786,12 @@ define void @test_vsoxseg7_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -5884,12 +5804,12 @@ define void @test_vsoxseg7_mask_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -5905,13 +5825,13 @@ define void @test_vsoxseg8_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5924,13 +5844,13 @@ define void @test_vsoxseg8_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5946,13 +5866,13 @@ define void @test_vsoxseg8_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg8_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5965,13 +5885,13 @@ define void @test_vsoxseg8_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5987,13 +5907,13 @@ define void @test_vsoxseg8_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6006,13 +5926,13 @@ define void @test_vsoxseg8_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6028,13 +5948,13 @@ define void @test_vsoxseg8_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -6047,13 +5967,13 @@ define void @test_vsoxseg8_mask_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -6068,7 +5988,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i8.nxv8i16(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg2_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei16.v v8, (a0), v10
@@ -6081,7 +6000,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei16.v v8, (a0), v10, v0.t
@@ -6097,7 +6015,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i8.nxv8i8(<vscale x 8 x i8>,<vscale x
 define void @test_vsoxseg2_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -6111,7 +6028,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -6128,7 +6044,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i8.nxv8i64(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg2_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v16
@@ -6141,7 +6056,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v16, v0.t
@@ -6157,7 +6071,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i8.nxv8i32(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg2_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v12
@@ -6170,7 +6083,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v12, v0.t
@@ -6186,11 +6098,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv8i8.nxv8i16(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg3_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
-; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10
+; CHECK-NEXT:    vsoxseg3ei16.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl)
@@ -6200,11 +6112,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
-; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsoxseg3ei16.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl)
@@ -6218,8 +6130,8 @@ define void @test_vsoxseg3_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg3_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6232,8 +6144,8 @@ define void @test_vsoxseg3_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6248,7 +6160,6 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv8i8.nxv8i64(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg3_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -6262,7 +6173,6 @@ entry:
 define void @test_vsoxseg3_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -6279,7 +6189,6 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv8i8.nxv8i32(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg3_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -6293,7 +6202,6 @@ entry:
 define void @test_vsoxseg3_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -6311,9 +6219,9 @@ define void @test_vsoxseg4_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg4_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -6326,9 +6234,9 @@ define void @test_vsoxseg4_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -6344,9 +6252,9 @@ define void @test_vsoxseg4_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg4_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6359,9 +6267,9 @@ define void @test_vsoxseg4_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6376,7 +6284,6 @@ declare void @llvm.riscv.vsoxseg4.mask.nxv8i8.nxv8i64(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg4_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6391,7 +6298,6 @@ entry:
 define void @test_vsoxseg4_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6409,7 +6315,6 @@ declare void @llvm.riscv.vsoxseg4.mask.nxv8i8.nxv8i32(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg4_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6424,7 +6329,6 @@ entry:
 define void @test_vsoxseg4_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6443,10 +6347,10 @@ define void @test_vsoxseg5_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg5_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -6459,10 +6363,10 @@ define void @test_vsoxseg5_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -6478,10 +6382,10 @@ define void @test_vsoxseg5_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg5_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6494,10 +6398,10 @@ define void @test_vsoxseg5_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6512,7 +6416,6 @@ declare void @llvm.riscv.vsoxseg5.mask.nxv8i8.nxv8i64(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg5_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg5_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6528,7 +6431,6 @@ entry:
 define void @test_vsoxseg5_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6547,13 +6449,13 @@ declare void @llvm.riscv.vsoxseg5.mask.nxv8i8.nxv8i32(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg5_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg5_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
-; CHECK-NEXT:    vsoxseg5ei32.v v16, (a0), v12
+; CHECK-NEXT:    vsoxseg5ei32.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg5.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl)
@@ -6563,13 +6465,13 @@ entry:
 define void @test_vsoxseg5_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
-; CHECK-NEXT:    vsoxseg5ei32.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsoxseg5ei32.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg5.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl)
@@ -6583,11 +6485,11 @@ define void @test_vsoxseg6_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -6600,11 +6502,11 @@ define void @test_vsoxseg6_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -6620,11 +6522,11 @@ define void @test_vsoxseg6_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg6_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6637,11 +6539,11 @@ define void @test_vsoxseg6_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6656,7 +6558,6 @@ declare void @llvm.riscv.vsoxseg6.mask.nxv8i8.nxv8i64(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg6_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg6_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6673,7 +6574,6 @@ entry:
 define void @test_vsoxseg6_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6694,11 +6594,11 @@ define void @test_vsoxseg6_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -6711,11 +6611,11 @@ define void @test_vsoxseg6_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -6731,12 +6631,12 @@ define void @test_vsoxseg7_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -6749,12 +6649,12 @@ define void @test_vsoxseg7_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -6770,12 +6670,12 @@ define void @test_vsoxseg7_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg7_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6788,12 +6688,12 @@ define void @test_vsoxseg7_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6808,7 +6708,6 @@ declare void @llvm.riscv.vsoxseg7.mask.nxv8i8.nxv8i64(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg7_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg7_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6826,7 +6725,6 @@ entry:
 define void @test_vsoxseg7_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6848,12 +6746,12 @@ define void @test_vsoxseg7_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -6866,12 +6764,12 @@ define void @test_vsoxseg7_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -6887,13 +6785,13 @@ define void @test_vsoxseg8_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -6906,13 +6804,13 @@ define void @test_vsoxseg8_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -6928,13 +6826,13 @@ define void @test_vsoxseg8_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg8_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6947,13 +6845,13 @@ define void @test_vsoxseg8_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6968,7 +6866,6 @@ declare void @llvm.riscv.vsoxseg8.mask.nxv8i8.nxv8i64(<vscale x 8 x i8>,<vscale
 define void @test_vsoxseg8_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg8_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6987,7 +6884,6 @@ entry:
 define void @test_vsoxseg8_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -7010,13 +6906,13 @@ define void @test_vsoxseg8_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -7029,13 +6925,13 @@ define void @test_vsoxseg8_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -7050,7 +6946,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i64.nxv4i32(<vscale x 4 x i64>,<vscal
 define void @test_vsoxseg2_nxv4i64_nxv4i32(<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i64_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7064,7 +6959,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i64_nxv4i32(<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i64_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7081,7 +6975,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i64.nxv4i8(<vscale x 4 x i64>,<vscale
 define void @test_vsoxseg2_nxv4i64_nxv4i8(<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i64_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7095,7 +6988,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i64_nxv4i8(<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i64_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7112,7 +7004,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i64.nxv4i64(<vscale x 4 x i64>,<vscal
 define void @test_vsoxseg2_nxv4i64_nxv4i64(<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i64_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7126,7 +7017,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i64_nxv4i64(<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i64_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7143,7 +7033,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i64.nxv4i16(<vscale x 4 x i64>,<vscal
 define void @test_vsoxseg2_nxv4i64_nxv4i16(<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i64_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7157,7 +7046,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i64_nxv4i16(<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i64_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7174,7 +7062,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i16.nxv4i32(<vscale x 4 x i16>,<vscal
 define void @test_vsoxseg2_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v10
@@ -7187,7 +7074,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v10, v0.t
@@ -7203,7 +7089,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i16.nxv4i8(<vscale x 4 x i16>,<vscale
 define void @test_vsoxseg2_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -7217,7 +7102,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -7234,7 +7118,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i16.nxv4i64(<vscale x 4 x i16>,<vscal
 define void @test_vsoxseg2_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v12
@@ -7247,7 +7130,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v12, v0.t
@@ -7263,7 +7145,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4i16.nxv4i16(<vscale x 4 x i16>,<vscal
 define void @test_vsoxseg2_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -7277,7 +7158,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -7294,11 +7174,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv4i16.nxv4i32(<vscale x 4 x i16>,<vscal
 define void @test_vsoxseg3_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl)
@@ -7308,11 +7188,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl)
@@ -7326,8 +7206,8 @@ define void @test_vsoxseg3_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg3_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7340,8 +7220,8 @@ define void @test_vsoxseg3_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7356,7 +7236,6 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv4i16.nxv4i64(<vscale x 4 x i16>,<vscal
 define void @test_vsoxseg3_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -7370,7 +7249,6 @@ entry:
 define void @test_vsoxseg3_mask_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -7388,8 +7266,8 @@ define void @test_vsoxseg3_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7402,8 +7280,8 @@ define void @test_vsoxseg3_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7419,9 +7297,9 @@ define void @test_vsoxseg4_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7434,9 +7312,9 @@ define void @test_vsoxseg4_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7452,9 +7330,9 @@ define void @test_vsoxseg4_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg4_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7467,9 +7345,9 @@ define void @test_vsoxseg4_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7484,7 +7362,6 @@ declare void @llvm.riscv.vsoxseg4.mask.nxv4i16.nxv4i64(<vscale x 4 x i16>,<vscal
 define void @test_vsoxseg4_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -7499,7 +7376,6 @@ entry:
 define void @test_vsoxseg4_mask_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -7518,9 +7394,9 @@ define void @test_vsoxseg4_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7533,9 +7409,9 @@ define void @test_vsoxseg4_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7551,10 +7427,10 @@ define void @test_vsoxseg5_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7567,10 +7443,10 @@ define void @test_vsoxseg5_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7586,10 +7462,10 @@ define void @test_vsoxseg5_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg5_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7602,10 +7478,10 @@ define void @test_vsoxseg5_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7620,13 +7496,13 @@ declare void @llvm.riscv.vsoxseg5.mask.nxv4i16.nxv4i64(<vscale x 4 x i16>,<vscal
 define void @test_vsoxseg5_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg5_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsoxseg5ei64.v v16, (a0), v12
+; CHECK-NEXT:    vsoxseg5ei64.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg5.nxv4i16.nxv4i64(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl)
@@ -7636,13 +7512,13 @@ entry:
 define void @test_vsoxseg5_mask_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsoxseg5ei64.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsoxseg5ei64.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg5.mask.nxv4i16.nxv4i64(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl)
@@ -7656,10 +7532,10 @@ define void @test_vsoxseg5_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7672,10 +7548,10 @@ define void @test_vsoxseg5_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7691,11 +7567,11 @@ define void @test_vsoxseg6_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7708,11 +7584,11 @@ define void @test_vsoxseg6_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7728,11 +7604,11 @@ define void @test_vsoxseg6_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg6_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7745,11 +7621,11 @@ define void @test_vsoxseg6_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7765,11 +7641,11 @@ define void @test_vsoxseg6_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -7782,11 +7658,11 @@ define void @test_vsoxseg6_mask_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -7802,11 +7678,11 @@ define void @test_vsoxseg6_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7819,11 +7695,11 @@ define void @test_vsoxseg6_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7839,12 +7715,12 @@ define void @test_vsoxseg7_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7857,12 +7733,12 @@ define void @test_vsoxseg7_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7878,12 +7754,12 @@ define void @test_vsoxseg7_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg7_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7896,12 +7772,12 @@ define void @test_vsoxseg7_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7917,12 +7793,12 @@ define void @test_vsoxseg7_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -7935,12 +7811,12 @@ define void @test_vsoxseg7_mask_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -7956,12 +7832,12 @@ define void @test_vsoxseg7_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7974,12 +7850,12 @@ define void @test_vsoxseg7_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7995,13 +7871,13 @@ define void @test_vsoxseg8_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -8014,13 +7890,13 @@ define void @test_vsoxseg8_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -8036,13 +7912,13 @@ define void @test_vsoxseg8_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg8_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8055,13 +7931,13 @@ define void @test_vsoxseg8_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8077,13 +7953,13 @@ define void @test_vsoxseg8_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -8096,13 +7972,13 @@ define void @test_vsoxseg8_mask_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -8118,13 +7994,13 @@ define void @test_vsoxseg8_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8137,13 +8013,13 @@ define void @test_vsoxseg8_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8158,7 +8034,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i8.nxv1i64(<vscale x 1 x i8>,<vscale
 define void @test_vsoxseg2_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -8172,7 +8047,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -8189,7 +8063,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i8.nxv1i32(<vscale x 1 x i8>,<vscale
 define void @test_vsoxseg2_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -8203,7 +8076,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -8220,7 +8092,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i8.nxv1i16(<vscale x 1 x i8>,<vscale
 define void @test_vsoxseg2_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -8234,7 +8105,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -8251,7 +8121,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1i8.nxv1i8(<vscale x 1 x i8>,<vscale x
 define void @test_vsoxseg2_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -8265,7 +8134,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -8283,8 +8151,8 @@ define void @test_vsoxseg3_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg3_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8297,8 +8165,8 @@ define void @test_vsoxseg3_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8314,8 +8182,8 @@ define void @test_vsoxseg3_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg3_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8328,8 +8196,8 @@ define void @test_vsoxseg3_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8345,8 +8213,8 @@ define void @test_vsoxseg3_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg3_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8359,8 +8227,8 @@ define void @test_vsoxseg3_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8376,8 +8244,8 @@ define void @test_vsoxseg3_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg3_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8390,8 +8258,8 @@ define void @test_vsoxseg3_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8407,9 +8275,9 @@ define void @test_vsoxseg4_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg4_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8422,9 +8290,9 @@ define void @test_vsoxseg4_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8440,9 +8308,9 @@ define void @test_vsoxseg4_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg4_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8455,9 +8323,9 @@ define void @test_vsoxseg4_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8473,9 +8341,9 @@ define void @test_vsoxseg4_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg4_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8488,9 +8356,9 @@ define void @test_vsoxseg4_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8506,9 +8374,9 @@ define void @test_vsoxseg4_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg4_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8521,9 +8389,9 @@ define void @test_vsoxseg4_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8539,10 +8407,10 @@ define void @test_vsoxseg5_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg5_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8555,10 +8423,10 @@ define void @test_vsoxseg5_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8574,10 +8442,10 @@ define void @test_vsoxseg5_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg5_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8590,10 +8458,10 @@ define void @test_vsoxseg5_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8609,10 +8477,10 @@ define void @test_vsoxseg5_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg5_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8625,10 +8493,10 @@ define void @test_vsoxseg5_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8644,10 +8512,10 @@ define void @test_vsoxseg5_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg5_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8660,10 +8528,10 @@ define void @test_vsoxseg5_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8679,11 +8547,11 @@ define void @test_vsoxseg6_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8696,11 +8564,11 @@ define void @test_vsoxseg6_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8716,11 +8584,11 @@ define void @test_vsoxseg6_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8733,11 +8601,11 @@ define void @test_vsoxseg6_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8753,11 +8621,11 @@ define void @test_vsoxseg6_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8770,11 +8638,11 @@ define void @test_vsoxseg6_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8790,11 +8658,11 @@ define void @test_vsoxseg6_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg6_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8807,11 +8675,11 @@ define void @test_vsoxseg6_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8827,12 +8695,12 @@ define void @test_vsoxseg7_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8845,12 +8713,12 @@ define void @test_vsoxseg7_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8866,12 +8734,12 @@ define void @test_vsoxseg7_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8884,12 +8752,12 @@ define void @test_vsoxseg7_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8905,12 +8773,12 @@ define void @test_vsoxseg7_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8923,12 +8791,12 @@ define void @test_vsoxseg7_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8944,12 +8812,12 @@ define void @test_vsoxseg7_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg7_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8962,12 +8830,12 @@ define void @test_vsoxseg7_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8983,13 +8851,13 @@ define void @test_vsoxseg8_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9002,13 +8870,13 @@ define void @test_vsoxseg8_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9024,13 +8892,13 @@ define void @test_vsoxseg8_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9043,13 +8911,13 @@ define void @test_vsoxseg8_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9065,13 +8933,13 @@ define void @test_vsoxseg8_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9084,13 +8952,13 @@ define void @test_vsoxseg8_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9106,13 +8974,13 @@ define void @test_vsoxseg8_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg8_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9125,13 +8993,13 @@ define void @test_vsoxseg8_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9146,7 +9014,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i8.nxv2i32(<vscale x 2 x i8>,<vscale
 define void @test_vsoxseg2_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -9160,7 +9027,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -9177,7 +9043,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i8.nxv2i8(<vscale x 2 x i8>,<vscale x
 define void @test_vsoxseg2_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -9191,7 +9056,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -9208,7 +9072,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i8.nxv2i16(<vscale x 2 x i8>,<vscale
 define void @test_vsoxseg2_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -9222,7 +9085,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -9239,7 +9101,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i8.nxv2i64(<vscale x 2 x i8>,<vscale
 define void @test_vsoxseg2_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v10
@@ -9252,7 +9113,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v10, v0.t
@@ -9269,8 +9129,8 @@ define void @test_vsoxseg3_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg3_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9283,8 +9143,8 @@ define void @test_vsoxseg3_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9300,8 +9160,8 @@ define void @test_vsoxseg3_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg3_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9314,8 +9174,8 @@ define void @test_vsoxseg3_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9331,8 +9191,8 @@ define void @test_vsoxseg3_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg3_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9345,8 +9205,8 @@ define void @test_vsoxseg3_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9361,11 +9221,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv2i8.nxv2i64(<vscale x 2 x i8>,<vscale
 define void @test_vsoxseg3_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
-; CHECK-NEXT:    vsoxseg3ei64.v v12, (a0), v10
+; CHECK-NEXT:    vsoxseg3ei64.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv2i8.nxv2i64(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl)
@@ -9375,11 +9235,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
-; CHECK-NEXT:    vsoxseg3ei64.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsoxseg3ei64.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv2i8.nxv2i64(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl)
@@ -9393,9 +9253,9 @@ define void @test_vsoxseg4_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg4_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9408,9 +9268,9 @@ define void @test_vsoxseg4_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9426,9 +9286,9 @@ define void @test_vsoxseg4_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg4_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9441,9 +9301,9 @@ define void @test_vsoxseg4_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9459,9 +9319,9 @@ define void @test_vsoxseg4_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg4_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9474,9 +9334,9 @@ define void @test_vsoxseg4_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9492,9 +9352,9 @@ define void @test_vsoxseg4_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg4_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -9507,9 +9367,9 @@ define void @test_vsoxseg4_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -9525,10 +9385,10 @@ define void @test_vsoxseg5_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg5_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9541,10 +9401,10 @@ define void @test_vsoxseg5_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9560,10 +9420,10 @@ define void @test_vsoxseg5_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg5_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9576,10 +9436,10 @@ define void @test_vsoxseg5_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9595,10 +9455,10 @@ define void @test_vsoxseg5_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg5_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9611,10 +9471,10 @@ define void @test_vsoxseg5_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9630,10 +9490,10 @@ define void @test_vsoxseg5_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg5_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -9646,10 +9506,10 @@ define void @test_vsoxseg5_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -9665,11 +9525,11 @@ define void @test_vsoxseg6_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9682,11 +9542,11 @@ define void @test_vsoxseg6_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9702,11 +9562,11 @@ define void @test_vsoxseg6_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg6_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9719,11 +9579,11 @@ define void @test_vsoxseg6_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9739,11 +9599,11 @@ define void @test_vsoxseg6_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9756,11 +9616,11 @@ define void @test_vsoxseg6_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9776,11 +9636,11 @@ define void @test_vsoxseg6_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg6_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -9793,11 +9653,11 @@ define void @test_vsoxseg6_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -9813,12 +9673,12 @@ define void @test_vsoxseg7_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9831,12 +9691,12 @@ define void @test_vsoxseg7_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9852,12 +9712,12 @@ define void @test_vsoxseg7_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg7_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9870,12 +9730,12 @@ define void @test_vsoxseg7_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9891,12 +9751,12 @@ define void @test_vsoxseg7_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9909,12 +9769,12 @@ define void @test_vsoxseg7_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9930,12 +9790,12 @@ define void @test_vsoxseg7_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg7_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -9948,12 +9808,12 @@ define void @test_vsoxseg7_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -9969,13 +9829,13 @@ define void @test_vsoxseg8_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9988,13 +9848,13 @@ define void @test_vsoxseg8_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10010,13 +9870,13 @@ define void @test_vsoxseg8_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsoxseg8_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10029,13 +9889,13 @@ define void @test_vsoxseg8_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10051,13 +9911,13 @@ define void @test_vsoxseg8_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10070,13 +9930,13 @@ define void @test_vsoxseg8_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10092,13 +9952,13 @@ define void @test_vsoxseg8_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsoxseg8_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -10111,13 +9971,13 @@ define void @test_vsoxseg8_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -10132,7 +9992,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i32.nxv8i16(<vscale x 8 x i32>,<vscal
 define void @test_vsoxseg2_nxv8i32_nxv8i16(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i32_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -10146,7 +10005,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i32_nxv8i16(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i32_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -10163,7 +10021,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i32.nxv8i8(<vscale x 8 x i32>,<vscale
 define void @test_vsoxseg2_nxv8i32_nxv8i8(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i32_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -10177,7 +10034,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i32_nxv8i8(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i32_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -10194,7 +10050,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i32.nxv8i64(<vscale x 8 x i32>,<vscal
 define void @test_vsoxseg2_nxv8i32_nxv8i64(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i32_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v16
@@ -10207,7 +10062,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i32_nxv8i64(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i32_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v16, v0.t
@@ -10223,7 +10077,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8i32.nxv8i32(<vscale x 8 x i32>,<vscal
 define void @test_vsoxseg2_nxv8i32_nxv8i32(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8i32_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -10237,7 +10090,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8i32_nxv8i32(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8i32_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -10254,7 +10106,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv32i8.nxv32i16(<vscale x 32 x i8>,<vsca
 define void @test_vsoxseg2_nxv32i8_nxv32i16(<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv32i8_nxv32i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei16.v v8, (a0), v16
@@ -10267,7 +10118,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv32i8_nxv32i16(<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i16> %index, <vscale x 32 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv32i8_nxv32i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei16.v v8, (a0), v16, v0.t
@@ -10283,7 +10133,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv32i8.nxv32i8(<vscale x 32 x i8>,<vscal
 define void @test_vsoxseg2_nxv32i8_nxv32i8(<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv32i8_nxv32i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
@@ -10297,7 +10146,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv32i8_nxv32i8(<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i8> %index, <vscale x 32 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv32i8_nxv32i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
@@ -10314,7 +10162,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i16.nxv2i32(<vscale x 2 x i16>,<vscal
 define void @test_vsoxseg2_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -10328,7 +10175,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -10345,7 +10191,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i16.nxv2i8(<vscale x 2 x i16>,<vscale
 define void @test_vsoxseg2_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -10359,7 +10204,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -10376,7 +10220,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i16.nxv2i16(<vscale x 2 x i16>,<vscal
 define void @test_vsoxseg2_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -10390,7 +10233,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -10407,7 +10249,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i16.nxv2i64(<vscale x 2 x i16>,<vscal
 define void @test_vsoxseg2_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v10
@@ -10420,7 +10261,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v10, v0.t
@@ -10437,8 +10277,8 @@ define void @test_vsoxseg3_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10451,8 +10291,8 @@ define void @test_vsoxseg3_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10468,8 +10308,8 @@ define void @test_vsoxseg3_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg3_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10482,8 +10322,8 @@ define void @test_vsoxseg3_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10499,8 +10339,8 @@ define void @test_vsoxseg3_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10513,8 +10353,8 @@ define void @test_vsoxseg3_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10529,11 +10369,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv2i16.nxv2i64(<vscale x 2 x i16>,<vscal
 define void @test_vsoxseg3_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei64.v v12, (a0), v10
+; CHECK-NEXT:    vsoxseg3ei64.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv2i16.nxv2i64(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl)
@@ -10543,11 +10383,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei64.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsoxseg3ei64.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv2i16.nxv2i64(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl)
@@ -10561,9 +10401,9 @@ define void @test_vsoxseg4_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10576,9 +10416,9 @@ define void @test_vsoxseg4_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10594,9 +10434,9 @@ define void @test_vsoxseg4_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg4_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10609,9 +10449,9 @@ define void @test_vsoxseg4_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10627,9 +10467,9 @@ define void @test_vsoxseg4_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10642,9 +10482,9 @@ define void @test_vsoxseg4_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10660,9 +10500,9 @@ define void @test_vsoxseg4_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -10675,9 +10515,9 @@ define void @test_vsoxseg4_mask_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -10693,10 +10533,10 @@ define void @test_vsoxseg5_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10709,10 +10549,10 @@ define void @test_vsoxseg5_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10728,10 +10568,10 @@ define void @test_vsoxseg5_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg5_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10744,10 +10584,10 @@ define void @test_vsoxseg5_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10763,10 +10603,10 @@ define void @test_vsoxseg5_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10779,10 +10619,10 @@ define void @test_vsoxseg5_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10798,10 +10638,10 @@ define void @test_vsoxseg5_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -10814,10 +10654,10 @@ define void @test_vsoxseg5_mask_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -10833,11 +10673,11 @@ define void @test_vsoxseg6_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10850,11 +10690,11 @@ define void @test_vsoxseg6_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10870,11 +10710,11 @@ define void @test_vsoxseg6_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg6_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10887,11 +10727,11 @@ define void @test_vsoxseg6_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10907,11 +10747,11 @@ define void @test_vsoxseg6_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10924,11 +10764,11 @@ define void @test_vsoxseg6_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10944,11 +10784,11 @@ define void @test_vsoxseg6_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -10961,11 +10801,11 @@ define void @test_vsoxseg6_mask_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -10981,12 +10821,12 @@ define void @test_vsoxseg7_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10999,12 +10839,12 @@ define void @test_vsoxseg7_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11020,12 +10860,12 @@ define void @test_vsoxseg7_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg7_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11038,12 +10878,12 @@ define void @test_vsoxseg7_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11059,12 +10899,12 @@ define void @test_vsoxseg7_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11077,12 +10917,12 @@ define void @test_vsoxseg7_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11098,12 +10938,12 @@ define void @test_vsoxseg7_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11116,12 +10956,12 @@ define void @test_vsoxseg7_mask_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11137,13 +10977,13 @@ define void @test_vsoxseg8_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11156,13 +10996,13 @@ define void @test_vsoxseg8_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11178,13 +11018,13 @@ define void @test_vsoxseg8_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg8_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11197,13 +11037,13 @@ define void @test_vsoxseg8_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11219,13 +11059,13 @@ define void @test_vsoxseg8_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11238,13 +11078,13 @@ define void @test_vsoxseg8_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11260,13 +11100,13 @@ define void @test_vsoxseg8_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11279,13 +11119,13 @@ define void @test_vsoxseg8_mask_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11300,7 +11140,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i64.nxv2i32(<vscale x 2 x i64>,<vscal
 define void @test_vsoxseg2_nxv2i64_nxv2i32(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11314,7 +11153,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i64_nxv2i32(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11331,7 +11169,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i64.nxv2i8(<vscale x 2 x i64>,<vscale
 define void @test_vsoxseg2_nxv2i64_nxv2i8(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11345,7 +11182,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i64_nxv2i8(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11362,7 +11198,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i64.nxv2i16(<vscale x 2 x i64>,<vscal
 define void @test_vsoxseg2_nxv2i64_nxv2i16(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11376,7 +11211,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i64_nxv2i16(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11393,7 +11227,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2i64.nxv2i64(<vscale x 2 x i64>,<vscal
 define void @test_vsoxseg2_nxv2i64_nxv2i64(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2i64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11407,7 +11240,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2i64_nxv2i64(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2i64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11425,8 +11257,8 @@ define void @test_vsoxseg3_nxv2i64_nxv2i32(<vscale x 2 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv2i64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11439,8 +11271,8 @@ define void @test_vsoxseg3_mask_nxv2i64_nxv2i32(<vscale x 2 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11456,8 +11288,8 @@ define void @test_vsoxseg3_nxv2i64_nxv2i8(<vscale x 2 x i64> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg3_nxv2i64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11470,8 +11302,8 @@ define void @test_vsoxseg3_mask_nxv2i64_nxv2i8(<vscale x 2 x i64> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11487,8 +11319,8 @@ define void @test_vsoxseg3_nxv2i64_nxv2i16(<vscale x 2 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv2i64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11501,8 +11333,8 @@ define void @test_vsoxseg3_mask_nxv2i64_nxv2i16(<vscale x 2 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11518,8 +11350,8 @@ define void @test_vsoxseg3_nxv2i64_nxv2i64(<vscale x 2 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv2i64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11532,8 +11364,8 @@ define void @test_vsoxseg3_mask_nxv2i64_nxv2i64(<vscale x 2 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2i64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11549,9 +11381,9 @@ define void @test_vsoxseg4_nxv2i64_nxv2i32(<vscale x 2 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv2i64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11564,9 +11396,9 @@ define void @test_vsoxseg4_mask_nxv2i64_nxv2i32(<vscale x 2 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11582,9 +11414,9 @@ define void @test_vsoxseg4_nxv2i64_nxv2i8(<vscale x 2 x i64> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsoxseg4_nxv2i64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11597,9 +11429,9 @@ define void @test_vsoxseg4_mask_nxv2i64_nxv2i8(<vscale x 2 x i64> %val, ptr %bas
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11615,9 +11447,9 @@ define void @test_vsoxseg4_nxv2i64_nxv2i16(<vscale x 2 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv2i64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11630,9 +11462,9 @@ define void @test_vsoxseg4_mask_nxv2i64_nxv2i16(<vscale x 2 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11648,9 +11480,9 @@ define void @test_vsoxseg4_nxv2i64_nxv2i64(<vscale x 2 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv2i64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11663,9 +11495,9 @@ define void @test_vsoxseg4_mask_nxv2i64_nxv2i64(<vscale x 2 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2i64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11680,7 +11512,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16f16.nxv16i16(<vscale x 16 x half>,<v
 define void @test_vsoxseg2_nxv16f16_nxv16i16(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16f16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -11694,7 +11525,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16f16_nxv16i16(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16f16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -11711,7 +11541,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16f16.nxv16i8(<vscale x 16 x half>,<vs
 define void @test_vsoxseg2_nxv16f16_nxv16i8(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16f16_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -11725,7 +11554,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16f16_nxv16i8(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16f16_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -11742,7 +11570,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16f16.nxv16i32(<vscale x 16 x half>,<v
 define void @test_vsoxseg2_nxv16f16_nxv16i32(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16f16_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v16
@@ -11755,7 +11582,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16f16_nxv16i32(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16f16_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v16, v0.t
@@ -11771,7 +11597,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f64.nxv4i32(<vscale x 4 x double>,<vs
 define void @test_vsoxseg2_nxv4f64_nxv4i32(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f64_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -11785,7 +11610,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f64_nxv4i32(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f64_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -11802,7 +11626,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f64.nxv4i8(<vscale x 4 x double>,<vsc
 define void @test_vsoxseg2_nxv4f64_nxv4i8(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f64_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -11816,7 +11639,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f64_nxv4i8(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f64_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -11833,7 +11655,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f64.nxv4i64(<vscale x 4 x double>,<vs
 define void @test_vsoxseg2_nxv4f64_nxv4i64(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f64_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -11847,7 +11668,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f64_nxv4i64(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f64_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -11864,7 +11684,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f64.nxv4i16(<vscale x 4 x double>,<vs
 define void @test_vsoxseg2_nxv4f64_nxv4i16(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f64_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -11878,7 +11697,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f64_nxv4i16(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f64_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -11895,7 +11713,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f64.nxv1i64(<vscale x 1 x double>,<vs
 define void @test_vsoxseg2_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -11909,7 +11726,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -11926,7 +11742,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f64.nxv1i32(<vscale x 1 x double>,<vs
 define void @test_vsoxseg2_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -11940,7 +11755,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -11957,7 +11771,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f64.nxv1i16(<vscale x 1 x double>,<vs
 define void @test_vsoxseg2_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -11971,7 +11784,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -11988,7 +11800,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f64.nxv1i8(<vscale x 1 x double>,<vsc
 define void @test_vsoxseg2_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -12002,7 +11813,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -12020,8 +11830,8 @@ define void @test_vsoxseg3_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12034,8 +11844,8 @@ define void @test_vsoxseg3_mask_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12051,8 +11861,8 @@ define void @test_vsoxseg3_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12065,8 +11875,8 @@ define void @test_vsoxseg3_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12082,8 +11892,8 @@ define void @test_vsoxseg3_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12096,8 +11906,8 @@ define void @test_vsoxseg3_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12113,8 +11923,8 @@ define void @test_vsoxseg3_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12127,8 +11937,8 @@ define void @test_vsoxseg3_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12144,9 +11954,9 @@ define void @test_vsoxseg4_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12159,9 +11969,9 @@ define void @test_vsoxseg4_mask_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12177,9 +11987,9 @@ define void @test_vsoxseg4_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12192,9 +12002,9 @@ define void @test_vsoxseg4_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12210,9 +12020,9 @@ define void @test_vsoxseg4_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12225,9 +12035,9 @@ define void @test_vsoxseg4_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12243,9 +12053,9 @@ define void @test_vsoxseg4_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12258,9 +12068,9 @@ define void @test_vsoxseg4_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12276,10 +12086,10 @@ define void @test_vsoxseg5_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12292,10 +12102,10 @@ define void @test_vsoxseg5_mask_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12311,10 +12121,10 @@ define void @test_vsoxseg5_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12327,10 +12137,10 @@ define void @test_vsoxseg5_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12346,10 +12156,10 @@ define void @test_vsoxseg5_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg5_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12362,10 +12172,10 @@ define void @test_vsoxseg5_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12381,10 +12191,10 @@ define void @test_vsoxseg5_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12397,10 +12207,10 @@ define void @test_vsoxseg5_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12416,11 +12226,11 @@ define void @test_vsoxseg6_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12433,11 +12243,11 @@ define void @test_vsoxseg6_mask_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12453,11 +12263,11 @@ define void @test_vsoxseg6_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12470,11 +12280,11 @@ define void @test_vsoxseg6_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12490,11 +12300,11 @@ define void @test_vsoxseg6_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg6_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12507,11 +12317,11 @@ define void @test_vsoxseg6_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12527,11 +12337,11 @@ define void @test_vsoxseg6_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12544,11 +12354,11 @@ define void @test_vsoxseg6_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12564,12 +12374,12 @@ define void @test_vsoxseg7_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12582,12 +12392,12 @@ define void @test_vsoxseg7_mask_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12603,12 +12413,12 @@ define void @test_vsoxseg7_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12621,12 +12431,12 @@ define void @test_vsoxseg7_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12642,12 +12452,12 @@ define void @test_vsoxseg7_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg7_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12660,12 +12470,12 @@ define void @test_vsoxseg7_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12681,12 +12491,12 @@ define void @test_vsoxseg7_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12699,12 +12509,12 @@ define void @test_vsoxseg7_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12720,13 +12530,13 @@ define void @test_vsoxseg8_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12739,13 +12549,13 @@ define void @test_vsoxseg8_mask_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12761,13 +12571,13 @@ define void @test_vsoxseg8_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12780,13 +12590,13 @@ define void @test_vsoxseg8_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12802,13 +12612,13 @@ define void @test_vsoxseg8_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg8_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12821,13 +12631,13 @@ define void @test_vsoxseg8_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12843,13 +12653,13 @@ define void @test_vsoxseg8_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12862,13 +12672,13 @@ define void @test_vsoxseg8_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12883,7 +12693,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f32.nxv2i32(<vscale x 2 x float>,<vsc
 define void @test_vsoxseg2_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -12897,7 +12706,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -12914,7 +12722,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f32.nxv2i8(<vscale x 2 x float>,<vsca
 define void @test_vsoxseg2_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -12928,7 +12735,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -12945,7 +12751,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f32.nxv2i16(<vscale x 2 x float>,<vsc
 define void @test_vsoxseg2_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -12959,7 +12764,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -12976,7 +12780,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f32.nxv2i64(<vscale x 2 x float>,<vsc
 define void @test_vsoxseg2_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v10
@@ -12989,7 +12792,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v10, v0.t
@@ -13006,8 +12808,8 @@ define void @test_vsoxseg3_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13020,8 +12822,8 @@ define void @test_vsoxseg3_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13037,8 +12839,8 @@ define void @test_vsoxseg3_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13051,8 +12853,8 @@ define void @test_vsoxseg3_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13068,8 +12870,8 @@ define void @test_vsoxseg3_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13082,8 +12884,8 @@ define void @test_vsoxseg3_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13098,11 +12900,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv2f32.nxv2i64(<vscale x 2 x float>,<vsc
 define void @test_vsoxseg3_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
-; CHECK-NEXT:    vsoxseg3ei64.v v12, (a0), v10
+; CHECK-NEXT:    vsoxseg3ei64.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv2f32.nxv2i64(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl)
@@ -13112,11 +12914,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
-; CHECK-NEXT:    vsoxseg3ei64.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsoxseg3ei64.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv2f32.nxv2i64(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl)
@@ -13130,9 +12932,9 @@ define void @test_vsoxseg4_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13145,9 +12947,9 @@ define void @test_vsoxseg4_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13163,9 +12965,9 @@ define void @test_vsoxseg4_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13178,9 +12980,9 @@ define void @test_vsoxseg4_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13196,9 +12998,9 @@ define void @test_vsoxseg4_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13211,9 +13013,9 @@ define void @test_vsoxseg4_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13229,9 +13031,9 @@ define void @test_vsoxseg4_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13244,9 +13046,9 @@ define void @test_vsoxseg4_mask_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13262,10 +13064,10 @@ define void @test_vsoxseg5_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13278,10 +13080,10 @@ define void @test_vsoxseg5_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13297,10 +13099,10 @@ define void @test_vsoxseg5_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13313,10 +13115,10 @@ define void @test_vsoxseg5_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13332,10 +13134,10 @@ define void @test_vsoxseg5_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13348,10 +13150,10 @@ define void @test_vsoxseg5_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13367,10 +13169,10 @@ define void @test_vsoxseg5_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13383,10 +13185,10 @@ define void @test_vsoxseg5_mask_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13402,11 +13204,11 @@ define void @test_vsoxseg6_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13419,11 +13221,11 @@ define void @test_vsoxseg6_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13439,11 +13241,11 @@ define void @test_vsoxseg6_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13456,11 +13258,11 @@ define void @test_vsoxseg6_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13476,11 +13278,11 @@ define void @test_vsoxseg6_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13493,11 +13295,11 @@ define void @test_vsoxseg6_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13513,11 +13315,11 @@ define void @test_vsoxseg6_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13530,11 +13332,11 @@ define void @test_vsoxseg6_mask_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13550,12 +13352,12 @@ define void @test_vsoxseg7_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13568,12 +13370,12 @@ define void @test_vsoxseg7_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13589,12 +13391,12 @@ define void @test_vsoxseg7_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13607,12 +13409,12 @@ define void @test_vsoxseg7_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13628,12 +13430,12 @@ define void @test_vsoxseg7_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13646,12 +13448,12 @@ define void @test_vsoxseg7_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13667,12 +13469,12 @@ define void @test_vsoxseg7_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13685,12 +13487,12 @@ define void @test_vsoxseg7_mask_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13706,13 +13508,13 @@ define void @test_vsoxseg8_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13725,13 +13527,13 @@ define void @test_vsoxseg8_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13747,13 +13549,13 @@ define void @test_vsoxseg8_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13766,13 +13568,13 @@ define void @test_vsoxseg8_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13788,13 +13590,13 @@ define void @test_vsoxseg8_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13807,13 +13609,13 @@ define void @test_vsoxseg8_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13829,13 +13631,13 @@ define void @test_vsoxseg8_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13848,13 +13650,13 @@ define void @test_vsoxseg8_mask_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13869,7 +13671,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f16.nxv1i64(<vscale x 1 x half>,<vsca
 define void @test_vsoxseg2_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -13883,7 +13684,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -13900,7 +13700,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f16.nxv1i32(<vscale x 1 x half>,<vsca
 define void @test_vsoxseg2_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -13914,7 +13713,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -13931,7 +13729,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f16.nxv1i16(<vscale x 1 x half>,<vsca
 define void @test_vsoxseg2_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -13945,7 +13742,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -13962,7 +13758,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f16.nxv1i8(<vscale x 1 x half>,<vscal
 define void @test_vsoxseg2_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -13976,7 +13771,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -13994,8 +13788,8 @@ define void @test_vsoxseg3_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14008,8 +13802,8 @@ define void @test_vsoxseg3_mask_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14025,8 +13819,8 @@ define void @test_vsoxseg3_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14039,8 +13833,8 @@ define void @test_vsoxseg3_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14056,8 +13850,8 @@ define void @test_vsoxseg3_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14070,8 +13864,8 @@ define void @test_vsoxseg3_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14087,8 +13881,8 @@ define void @test_vsoxseg3_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14101,8 +13895,8 @@ define void @test_vsoxseg3_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14118,9 +13912,9 @@ define void @test_vsoxseg4_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14133,9 +13927,9 @@ define void @test_vsoxseg4_mask_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14151,9 +13945,9 @@ define void @test_vsoxseg4_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14166,9 +13960,9 @@ define void @test_vsoxseg4_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14184,9 +13978,9 @@ define void @test_vsoxseg4_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14199,9 +13993,9 @@ define void @test_vsoxseg4_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14217,9 +14011,9 @@ define void @test_vsoxseg4_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14232,9 +14026,9 @@ define void @test_vsoxseg4_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14250,10 +14044,10 @@ define void @test_vsoxseg5_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14266,10 +14060,10 @@ define void @test_vsoxseg5_mask_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14285,10 +14079,10 @@ define void @test_vsoxseg5_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14301,10 +14095,10 @@ define void @test_vsoxseg5_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14320,10 +14114,10 @@ define void @test_vsoxseg5_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14336,10 +14130,10 @@ define void @test_vsoxseg5_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14355,10 +14149,10 @@ define void @test_vsoxseg5_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14371,10 +14165,10 @@ define void @test_vsoxseg5_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14390,11 +14184,11 @@ define void @test_vsoxseg6_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14407,11 +14201,11 @@ define void @test_vsoxseg6_mask_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14427,11 +14221,11 @@ define void @test_vsoxseg6_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14444,11 +14238,11 @@ define void @test_vsoxseg6_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14464,11 +14258,11 @@ define void @test_vsoxseg6_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14481,11 +14275,11 @@ define void @test_vsoxseg6_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14501,11 +14295,11 @@ define void @test_vsoxseg6_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14518,11 +14312,11 @@ define void @test_vsoxseg6_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14538,12 +14332,12 @@ define void @test_vsoxseg7_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14556,12 +14350,12 @@ define void @test_vsoxseg7_mask_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14577,12 +14371,12 @@ define void @test_vsoxseg7_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14595,12 +14389,12 @@ define void @test_vsoxseg7_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14616,12 +14410,12 @@ define void @test_vsoxseg7_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14634,12 +14428,12 @@ define void @test_vsoxseg7_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14655,12 +14449,12 @@ define void @test_vsoxseg7_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14673,12 +14467,12 @@ define void @test_vsoxseg7_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14694,13 +14488,13 @@ define void @test_vsoxseg8_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14713,13 +14507,13 @@ define void @test_vsoxseg8_mask_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14735,13 +14529,13 @@ define void @test_vsoxseg8_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14754,13 +14548,13 @@ define void @test_vsoxseg8_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14776,13 +14570,13 @@ define void @test_vsoxseg8_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14795,13 +14589,13 @@ define void @test_vsoxseg8_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14817,13 +14611,13 @@ define void @test_vsoxseg8_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14836,13 +14630,13 @@ define void @test_vsoxseg8_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14857,7 +14651,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f32.nxv1i64(<vscale x 1 x float>,<vsc
 define void @test_vsoxseg2_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -14871,7 +14664,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -14888,7 +14680,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f32.nxv1i32(<vscale x 1 x float>,<vsc
 define void @test_vsoxseg2_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -14902,7 +14693,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -14919,7 +14709,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f32.nxv1i16(<vscale x 1 x float>,<vsc
 define void @test_vsoxseg2_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -14933,7 +14722,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -14950,7 +14738,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv1f32.nxv1i8(<vscale x 1 x float>,<vsca
 define void @test_vsoxseg2_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -14964,7 +14751,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -14982,8 +14768,8 @@ define void @test_vsoxseg3_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14996,8 +14782,8 @@ define void @test_vsoxseg3_mask_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15013,8 +14799,8 @@ define void @test_vsoxseg3_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15027,8 +14813,8 @@ define void @test_vsoxseg3_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15044,8 +14830,8 @@ define void @test_vsoxseg3_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15058,8 +14844,8 @@ define void @test_vsoxseg3_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15075,8 +14861,8 @@ define void @test_vsoxseg3_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15089,8 +14875,8 @@ define void @test_vsoxseg3_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15106,9 +14892,9 @@ define void @test_vsoxseg4_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15121,9 +14907,9 @@ define void @test_vsoxseg4_mask_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15139,9 +14925,9 @@ define void @test_vsoxseg4_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15154,9 +14940,9 @@ define void @test_vsoxseg4_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15172,9 +14958,9 @@ define void @test_vsoxseg4_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15187,9 +14973,9 @@ define void @test_vsoxseg4_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15205,9 +14991,9 @@ define void @test_vsoxseg4_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15220,9 +15006,9 @@ define void @test_vsoxseg4_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15238,10 +15024,10 @@ define void @test_vsoxseg5_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15254,10 +15040,10 @@ define void @test_vsoxseg5_mask_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15273,10 +15059,10 @@ define void @test_vsoxseg5_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15289,10 +15075,10 @@ define void @test_vsoxseg5_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15308,10 +15094,10 @@ define void @test_vsoxseg5_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15324,10 +15110,10 @@ define void @test_vsoxseg5_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15343,10 +15129,10 @@ define void @test_vsoxseg5_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15359,10 +15145,10 @@ define void @test_vsoxseg5_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15378,11 +15164,11 @@ define void @test_vsoxseg6_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15395,11 +15181,11 @@ define void @test_vsoxseg6_mask_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15415,11 +15201,11 @@ define void @test_vsoxseg6_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15432,11 +15218,11 @@ define void @test_vsoxseg6_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15452,11 +15238,11 @@ define void @test_vsoxseg6_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15469,11 +15255,11 @@ define void @test_vsoxseg6_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15489,11 +15275,11 @@ define void @test_vsoxseg6_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15506,11 +15292,11 @@ define void @test_vsoxseg6_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15526,12 +15312,12 @@ define void @test_vsoxseg7_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15544,12 +15330,12 @@ define void @test_vsoxseg7_mask_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15565,12 +15351,12 @@ define void @test_vsoxseg7_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15583,12 +15369,12 @@ define void @test_vsoxseg7_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15604,12 +15390,12 @@ define void @test_vsoxseg7_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15622,12 +15408,12 @@ define void @test_vsoxseg7_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15643,12 +15429,12 @@ define void @test_vsoxseg7_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15661,12 +15447,12 @@ define void @test_vsoxseg7_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15682,13 +15468,13 @@ define void @test_vsoxseg8_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15701,13 +15487,13 @@ define void @test_vsoxseg8_mask_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15723,13 +15509,13 @@ define void @test_vsoxseg8_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15742,13 +15528,13 @@ define void @test_vsoxseg8_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15764,13 +15550,13 @@ define void @test_vsoxseg8_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15783,13 +15569,13 @@ define void @test_vsoxseg8_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15805,13 +15591,13 @@ define void @test_vsoxseg8_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15824,13 +15610,13 @@ define void @test_vsoxseg8_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15845,7 +15631,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8f16.nxv8i16(<vscale x 8 x half>,<vsca
 define void @test_vsoxseg2_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -15859,7 +15644,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -15876,7 +15660,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8f16.nxv8i8(<vscale x 8 x half>,<vscal
 define void @test_vsoxseg2_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -15890,7 +15673,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -15907,7 +15689,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8f16.nxv8i64(<vscale x 8 x half>,<vsca
 define void @test_vsoxseg2_nxv8f16_nxv8i64(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8f16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v16
@@ -15920,7 +15701,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8f16_nxv8i64(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8f16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v16, v0.t
@@ -15936,7 +15716,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8f16.nxv8i32(<vscale x 8 x half>,<vsca
 define void @test_vsoxseg2_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v12
@@ -15949,7 +15728,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v12, v0.t
@@ -15966,8 +15744,8 @@ define void @test_vsoxseg3_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -15980,8 +15758,8 @@ define void @test_vsoxseg3_mask_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -15997,8 +15775,8 @@ define void @test_vsoxseg3_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16011,8 +15789,8 @@ define void @test_vsoxseg3_mask_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16027,7 +15805,6 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv8f16.nxv8i64(<vscale x 8 x half>,<vsca
 define void @test_vsoxseg3_nxv8f16_nxv8i64(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv8f16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -16041,7 +15818,6 @@ entry:
 define void @test_vsoxseg3_mask_nxv8f16_nxv8i64(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8f16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -16058,11 +15834,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv8f16.nxv8i32(<vscale x 8 x half>,<vsca
 define void @test_vsoxseg3_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v16, (a0), v12
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl)
@@ -16072,11 +15848,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl)
@@ -16090,9 +15866,9 @@ define void @test_vsoxseg4_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16105,9 +15881,9 @@ define void @test_vsoxseg4_mask_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16123,9 +15899,9 @@ define void @test_vsoxseg4_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16138,9 +15914,9 @@ define void @test_vsoxseg4_mask_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16155,7 +15931,6 @@ declare void @llvm.riscv.vsoxseg4.mask.nxv8f16.nxv8i64(<vscale x 8 x half>,<vsca
 define void @test_vsoxseg4_nxv8f16_nxv8i64(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_nxv8f16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -16170,7 +15945,6 @@ entry:
 define void @test_vsoxseg4_mask_nxv8f16_nxv8i64(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8f16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -16189,9 +15963,9 @@ define void @test_vsoxseg4_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -16204,9 +15978,9 @@ define void @test_vsoxseg4_mask_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -16221,7 +15995,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8f32.nxv8i16(<vscale x 8 x float>,<vsc
 define void @test_vsoxseg2_nxv8f32_nxv8i16(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8f32_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -16235,7 +16008,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8f32_nxv8i16(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8f32_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -16252,7 +16024,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8f32.nxv8i8(<vscale x 8 x float>,<vsca
 define void @test_vsoxseg2_nxv8f32_nxv8i8(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8f32_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -16266,7 +16037,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8f32_nxv8i8(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8f32_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -16283,7 +16053,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8f32.nxv8i64(<vscale x 8 x float>,<vsc
 define void @test_vsoxseg2_nxv8f32_nxv8i64(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8f32_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v16
@@ -16296,7 +16065,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8f32_nxv8i64(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8f32_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v16, v0.t
@@ -16312,7 +16080,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv8f32.nxv8i32(<vscale x 8 x float>,<vsc
 define void @test_vsoxseg2_nxv8f32_nxv8i32(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv8f32_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -16326,7 +16093,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv8f32_nxv8i32(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv8f32_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -16343,7 +16109,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f64.nxv2i32(<vscale x 2 x double>,<vs
 define void @test_vsoxseg2_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -16357,7 +16122,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -16374,7 +16138,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f64.nxv2i8(<vscale x 2 x double>,<vsc
 define void @test_vsoxseg2_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -16388,7 +16151,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -16405,7 +16167,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f64.nxv2i16(<vscale x 2 x double>,<vs
 define void @test_vsoxseg2_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -16419,7 +16180,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -16436,7 +16196,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f64.nxv2i64(<vscale x 2 x double>,<vs
 define void @test_vsoxseg2_nxv2f64_nxv2i64(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -16450,7 +16209,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f64_nxv2i64(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -16468,8 +16226,8 @@ define void @test_vsoxseg3_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16482,8 +16240,8 @@ define void @test_vsoxseg3_mask_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16499,8 +16257,8 @@ define void @test_vsoxseg3_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16513,8 +16271,8 @@ define void @test_vsoxseg3_mask_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16530,8 +16288,8 @@ define void @test_vsoxseg3_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16544,8 +16302,8 @@ define void @test_vsoxseg3_mask_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16561,8 +16319,8 @@ define void @test_vsoxseg3_nxv2f64_nxv2i64(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg3_nxv2f64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16575,8 +16333,8 @@ define void @test_vsoxseg3_mask_nxv2f64_nxv2i64(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16592,9 +16350,9 @@ define void @test_vsoxseg4_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16607,9 +16365,9 @@ define void @test_vsoxseg4_mask_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16625,9 +16383,9 @@ define void @test_vsoxseg4_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16640,9 +16398,9 @@ define void @test_vsoxseg4_mask_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16658,9 +16416,9 @@ define void @test_vsoxseg4_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16673,9 +16431,9 @@ define void @test_vsoxseg4_mask_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16691,9 +16449,9 @@ define void @test_vsoxseg4_nxv2f64_nxv2i64(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsoxseg4_nxv2f64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16706,9 +16464,9 @@ define void @test_vsoxseg4_mask_nxv2f64_nxv2i64(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16723,7 +16481,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f16.nxv4i32(<vscale x 4 x half>,<vsca
 define void @test_vsoxseg2_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v10
@@ -16736,7 +16493,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei32.v v8, (a0), v10, v0.t
@@ -16752,7 +16508,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f16.nxv4i8(<vscale x 4 x half>,<vscal
 define void @test_vsoxseg2_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -16766,7 +16521,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -16783,7 +16537,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f16.nxv4i64(<vscale x 4 x half>,<vsca
 define void @test_vsoxseg2_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v12
@@ -16796,7 +16549,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v12, v0.t
@@ -16812,7 +16564,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f16.nxv4i16(<vscale x 4 x half>,<vsca
 define void @test_vsoxseg2_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -16826,7 +16577,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -16843,11 +16593,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv4f16.nxv4i32(<vscale x 4 x half>,<vsca
 define void @test_vsoxseg3_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl)
@@ -16857,11 +16607,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsoxseg3ei32.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl)
@@ -16875,8 +16625,8 @@ define void @test_vsoxseg3_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -16889,8 +16639,8 @@ define void @test_vsoxseg3_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -16905,7 +16655,6 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv4f16.nxv4i64(<vscale x 4 x half>,<vsca
 define void @test_vsoxseg3_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -16919,7 +16668,6 @@ entry:
 define void @test_vsoxseg3_mask_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -16937,8 +16685,8 @@ define void @test_vsoxseg3_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -16951,8 +16699,8 @@ define void @test_vsoxseg3_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -16968,9 +16716,9 @@ define void @test_vsoxseg4_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16983,9 +16731,9 @@ define void @test_vsoxseg4_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -17001,9 +16749,9 @@ define void @test_vsoxseg4_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17016,9 +16764,9 @@ define void @test_vsoxseg4_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17033,7 +16781,6 @@ declare void @llvm.riscv.vsoxseg4.mask.nxv4f16.nxv4i64(<vscale x 4 x half>,<vsca
 define void @test_vsoxseg4_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -17048,7 +16795,6 @@ entry:
 define void @test_vsoxseg4_mask_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -17067,9 +16813,9 @@ define void @test_vsoxseg4_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17082,9 +16828,9 @@ define void @test_vsoxseg4_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17100,10 +16846,10 @@ define void @test_vsoxseg5_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -17116,10 +16862,10 @@ define void @test_vsoxseg5_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -17135,10 +16881,10 @@ define void @test_vsoxseg5_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17151,10 +16897,10 @@ define void @test_vsoxseg5_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17169,13 +16915,13 @@ declare void @llvm.riscv.vsoxseg5.mask.nxv4f16.nxv4i64(<vscale x 4 x half>,<vsca
 define void @test_vsoxseg5_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg5_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsoxseg5ei64.v v16, (a0), v12
+; CHECK-NEXT:    vsoxseg5ei64.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg5.nxv4f16.nxv4i64(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl)
@@ -17185,13 +16931,13 @@ entry:
 define void @test_vsoxseg5_mask_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsoxseg5ei64.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsoxseg5ei64.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg5.mask.nxv4f16.nxv4i64(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl)
@@ -17205,10 +16951,10 @@ define void @test_vsoxseg5_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17221,10 +16967,10 @@ define void @test_vsoxseg5_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17240,11 +16986,11 @@ define void @test_vsoxseg6_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -17257,11 +17003,11 @@ define void @test_vsoxseg6_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -17277,11 +17023,11 @@ define void @test_vsoxseg6_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17294,11 +17040,11 @@ define void @test_vsoxseg6_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17314,11 +17060,11 @@ define void @test_vsoxseg6_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -17331,11 +17077,11 @@ define void @test_vsoxseg6_mask_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -17351,11 +17097,11 @@ define void @test_vsoxseg6_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17368,11 +17114,11 @@ define void @test_vsoxseg6_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17388,12 +17134,12 @@ define void @test_vsoxseg7_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -17406,12 +17152,12 @@ define void @test_vsoxseg7_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -17427,12 +17173,12 @@ define void @test_vsoxseg7_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17445,12 +17191,12 @@ define void @test_vsoxseg7_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17466,12 +17212,12 @@ define void @test_vsoxseg7_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -17484,12 +17230,12 @@ define void @test_vsoxseg7_mask_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -17505,12 +17251,12 @@ define void @test_vsoxseg7_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17523,12 +17269,12 @@ define void @test_vsoxseg7_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17544,13 +17290,13 @@ define void @test_vsoxseg8_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -17563,13 +17309,13 @@ define void @test_vsoxseg8_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -17585,13 +17331,13 @@ define void @test_vsoxseg8_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17604,13 +17350,13 @@ define void @test_vsoxseg8_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17626,13 +17372,13 @@ define void @test_vsoxseg8_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -17645,13 +17391,13 @@ define void @test_vsoxseg8_mask_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -17667,13 +17413,13 @@ define void @test_vsoxseg8_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17686,13 +17432,13 @@ define void @test_vsoxseg8_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17707,7 +17453,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f16.nxv2i32(<vscale x 2 x half>,<vsca
 define void @test_vsoxseg2_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -17721,7 +17466,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -17738,7 +17482,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f16.nxv2i8(<vscale x 2 x half>,<vscal
 define void @test_vsoxseg2_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -17752,7 +17495,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -17769,7 +17511,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f16.nxv2i16(<vscale x 2 x half>,<vsca
 define void @test_vsoxseg2_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -17783,7 +17524,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -17800,7 +17540,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv2f16.nxv2i64(<vscale x 2 x half>,<vsca
 define void @test_vsoxseg2_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v10
@@ -17813,7 +17552,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v10, v0.t
@@ -17830,8 +17568,8 @@ define void @test_vsoxseg3_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17844,8 +17582,8 @@ define void @test_vsoxseg3_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17861,8 +17599,8 @@ define void @test_vsoxseg3_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg3_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17875,8 +17613,8 @@ define void @test_vsoxseg3_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17892,8 +17630,8 @@ define void @test_vsoxseg3_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17906,8 +17644,8 @@ define void @test_vsoxseg3_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17922,11 +17660,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv2f16.nxv2i64(<vscale x 2 x half>,<vsca
 define void @test_vsoxseg3_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei64.v v12, (a0), v10
+; CHECK-NEXT:    vsoxseg3ei64.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv2f16.nxv2i64(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl)
@@ -17936,11 +17674,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei64.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsoxseg3ei64.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv2f16.nxv2i64(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl)
@@ -17954,9 +17692,9 @@ define void @test_vsoxseg4_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17969,9 +17707,9 @@ define void @test_vsoxseg4_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17987,9 +17725,9 @@ define void @test_vsoxseg4_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg4_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18002,9 +17740,9 @@ define void @test_vsoxseg4_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18020,9 +17758,9 @@ define void @test_vsoxseg4_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18035,9 +17773,9 @@ define void @test_vsoxseg4_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18053,9 +17791,9 @@ define void @test_vsoxseg4_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18068,9 +17806,9 @@ define void @test_vsoxseg4_mask_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -18086,10 +17824,10 @@ define void @test_vsoxseg5_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18102,10 +17840,10 @@ define void @test_vsoxseg5_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18121,10 +17859,10 @@ define void @test_vsoxseg5_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg5_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18137,10 +17875,10 @@ define void @test_vsoxseg5_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18156,10 +17894,10 @@ define void @test_vsoxseg5_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18172,10 +17910,10 @@ define void @test_vsoxseg5_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18191,10 +17929,10 @@ define void @test_vsoxseg5_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg5_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18207,10 +17945,10 @@ define void @test_vsoxseg5_mask_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg5_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -18226,11 +17964,11 @@ define void @test_vsoxseg6_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18243,11 +17981,11 @@ define void @test_vsoxseg6_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18263,11 +18001,11 @@ define void @test_vsoxseg6_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg6_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18280,11 +18018,11 @@ define void @test_vsoxseg6_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18300,11 +18038,11 @@ define void @test_vsoxseg6_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18317,11 +18055,11 @@ define void @test_vsoxseg6_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18337,11 +18075,11 @@ define void @test_vsoxseg6_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg6_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18354,11 +18092,11 @@ define void @test_vsoxseg6_mask_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg6_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -18374,12 +18112,12 @@ define void @test_vsoxseg7_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18392,12 +18130,12 @@ define void @test_vsoxseg7_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18413,12 +18151,12 @@ define void @test_vsoxseg7_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg7_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18431,12 +18169,12 @@ define void @test_vsoxseg7_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18452,12 +18190,12 @@ define void @test_vsoxseg7_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18470,12 +18208,12 @@ define void @test_vsoxseg7_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18491,12 +18229,12 @@ define void @test_vsoxseg7_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg7_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18509,12 +18247,12 @@ define void @test_vsoxseg7_mask_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg7_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -18530,13 +18268,13 @@ define void @test_vsoxseg8_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18549,13 +18287,13 @@ define void @test_vsoxseg8_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18571,13 +18309,13 @@ define void @test_vsoxseg8_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsoxseg8_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18590,13 +18328,13 @@ define void @test_vsoxseg8_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18612,13 +18350,13 @@ define void @test_vsoxseg8_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18631,13 +18369,13 @@ define void @test_vsoxseg8_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18653,13 +18391,13 @@ define void @test_vsoxseg8_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg8_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18672,13 +18410,13 @@ define void @test_vsoxseg8_mask_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg8_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsoxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -18693,7 +18431,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f32.nxv4i32(<vscale x 4 x float>,<vsc
 define void @test_vsoxseg2_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -18707,7 +18444,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -18724,7 +18460,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f32.nxv4i8(<vscale x 4 x float>,<vsca
 define void @test_vsoxseg2_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -18738,7 +18473,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -18755,7 +18489,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f32.nxv4i64(<vscale x 4 x float>,<vsc
 define void @test_vsoxseg2_nxv4f32_nxv4i64(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v12
@@ -18768,7 +18501,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f32_nxv4i64(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg2ei64.v v8, (a0), v12, v0.t
@@ -18784,7 +18516,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv4f32.nxv4i16(<vscale x 4 x float>,<vsc
 define void @test_vsoxseg2_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -18798,7 +18529,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -18816,8 +18546,8 @@ define void @test_vsoxseg3_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18830,8 +18560,8 @@ define void @test_vsoxseg3_mask_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -18847,8 +18577,8 @@ define void @test_vsoxseg3_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18861,8 +18591,8 @@ define void @test_vsoxseg3_mask_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -18877,11 +18607,11 @@ declare void @llvm.riscv.vsoxseg3.mask.nxv4f32.nxv4i64(<vscale x 4 x float>,<vsc
 define void @test_vsoxseg3_nxv4f32_nxv4i64(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_nxv4f32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei64.v v16, (a0), v12
+; CHECK-NEXT:    vsoxseg3ei64.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.nxv4f32.nxv4i64(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl)
@@ -18891,11 +18621,11 @@ entry:
 define void @test_vsoxseg3_mask_nxv4f32_nxv4i64(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4f32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
-; CHECK-NEXT:    vsoxseg3ei64.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsoxseg3ei64.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsoxseg3.mask.nxv4f32.nxv4i64(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl)
@@ -18909,8 +18639,8 @@ define void @test_vsoxseg3_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg3_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18923,8 +18653,8 @@ define void @test_vsoxseg3_mask_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg3_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -18940,9 +18670,9 @@ define void @test_vsoxseg4_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18955,9 +18685,9 @@ define void @test_vsoxseg4_mask_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -18973,9 +18703,9 @@ define void @test_vsoxseg4_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18988,9 +18718,9 @@ define void @test_vsoxseg4_mask_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -19006,9 +18736,9 @@ define void @test_vsoxseg4_nxv4f32_nxv4i64(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv4f32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -19021,9 +18751,9 @@ define void @test_vsoxseg4_mask_nxv4f32_nxv4i64(<vscale x 4 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4f32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -19039,9 +18769,9 @@ define void @test_vsoxseg4_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsoxseg4_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -19054,9 +18784,9 @@ define void @test_vsoxseg4_mask_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %
 ; CHECK-LABEL: test_vsoxseg4_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsoxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/rvv/vsplats-zfa.ll b/llvm/test/CodeGen/RISCV/rvv/vsplats-zfa.ll
new file mode 100644
index 0000000000000..e4ae59e012026
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/rvv/vsplats-zfa.ll
@@ -0,0 +1,41 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 -mattr=+zfh,+experimental-zfa,+experimental-zvfh,+v -target-abi ilp32d -verify-machineinstrs < %s \
+; RUN:   | FileCheck %s --check-prefixes=CHECK
+; RUN: llc -mtriple=riscv64 -mattr=+zfh,+experimental-zfa,+experimental-zvfh,+v -target-abi lp64d -verify-machineinstrs < %s \
+; RUN:   | FileCheck %s --check-prefixes=CHECK
+
+define <vscale x 8 x half> @vsplat_f16_0p625() {
+; CHECK-LABEL: vsplat_f16_0p625:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.h ft0, 0.625
+; CHECK-NEXT:    vsetvli a0, zero, e16, m2, ta, ma
+; CHECK-NEXT:    vfmv.v.f v8, ft0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x half> poison, half 0.625, i32 0
+  %splat = shufflevector <vscale x 8 x half> %head, <vscale x 8 x half> poison, <vscale x 8 x i32> zeroinitializer
+  ret <vscale x 8 x half> %splat
+}
+
+define <vscale x 8 x float> @vsplat_f32_0p75() {
+; CHECK-LABEL: vsplat_f32_0p75:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.s ft0, 0.75
+; CHECK-NEXT:    vsetvli a0, zero, e32, m4, ta, ma
+; CHECK-NEXT:    vfmv.v.f v8, ft0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x float> poison, float 0.75, i32 0
+  %splat = shufflevector <vscale x 8 x float> %head, <vscale x 8 x float> poison, <vscale x 8 x i32> zeroinitializer
+  ret <vscale x 8 x float> %splat
+}
+
+define <vscale x 8 x double> @vsplat_f64_neg1() {
+; CHECK-LABEL: vsplat_f64_neg1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fli.d ft0, -1.0
+; CHECK-NEXT:    vsetvli a0, zero, e64, m8, ta, ma
+; CHECK-NEXT:    vfmv.v.f v8, ft0
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x double> poison, double -1.0, i32 0
+  %splat = shufflevector <vscale x 8 x double> %head, <vscale x 8 x double> poison, <vscale x 8 x i32> zeroinitializer
+  ret <vscale x 8 x double> %splat
+}
diff --git a/llvm/test/CodeGen/RISCV/rvv/vsseg-rv32.ll b/llvm/test/CodeGen/RISCV/rvv/vsseg-rv32.ll
index f3ac92670e835..0cdef07b01109 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vsseg-rv32.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vsseg-rv32.ll
@@ -8,7 +8,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv16i16(<vscale x 16 x i16>,<vscale x 16 x
 define void @test_vsseg2_nxv16i16(<vscale x 16 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -21,7 +20,6 @@ entry:
 define void @test_vsseg2_mask_nxv16i16(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -37,7 +35,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>,
 define void @test_vsseg2_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0)
@@ -50,7 +47,6 @@ entry:
 define void @test_vsseg2_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0), v0.t
@@ -66,7 +62,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>,
 define void @test_vsseg3_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -80,7 +75,6 @@ entry:
 define void @test_vsseg3_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -97,7 +91,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>,
 define void @test_vsseg4_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -112,7 +105,6 @@ entry:
 define void @test_vsseg4_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -130,7 +122,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>,
 define void @test_vsseg5_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -146,7 +137,6 @@ entry:
 define void @test_vsseg5_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -165,7 +155,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>,
 define void @test_vsseg6_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -182,7 +171,6 @@ entry:
 define void @test_vsseg6_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -202,7 +190,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>,
 define void @test_vsseg7_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -220,7 +207,6 @@ entry:
 define void @test_vsseg7_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -241,7 +227,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>,
 define void @test_vsseg8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -260,7 +245,6 @@ entry:
 define void @test_vsseg8_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -282,7 +266,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv16i8(<vscale x 16 x i8>,<vscale x 16 x i
 define void @test_vsseg2_nxv16i8(<vscale x 16 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0)
@@ -295,7 +278,6 @@ entry:
 define void @test_vsseg2_mask_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0), v0.t
@@ -311,7 +293,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv16i8(<vscale x 16 x i8>,<vscale x 16 x i
 define void @test_vsseg3_nxv16i8(<vscale x 16 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -325,7 +306,6 @@ entry:
 define void @test_vsseg3_mask_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -342,7 +322,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv16i8(<vscale x 16 x i8>,<vscale x 16 x i
 define void @test_vsseg4_nxv16i8(<vscale x 16 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -357,7 +336,6 @@ entry:
 define void @test_vsseg4_mask_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -375,7 +353,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i3
 define void @test_vsseg2_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0)
@@ -388,7 +365,6 @@ entry:
 define void @test_vsseg2_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0), v0.t
@@ -404,7 +380,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i3
 define void @test_vsseg3_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -418,7 +393,6 @@ entry:
 define void @test_vsseg3_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -435,7 +409,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i3
 define void @test_vsseg4_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -450,7 +423,6 @@ entry:
 define void @test_vsseg4_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -468,7 +440,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i3
 define void @test_vsseg5_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -484,7 +455,6 @@ entry:
 define void @test_vsseg5_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -503,7 +473,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i3
 define void @test_vsseg6_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -520,7 +489,6 @@ entry:
 define void @test_vsseg6_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -540,7 +508,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i3
 define void @test_vsseg7_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -558,7 +525,6 @@ entry:
 define void @test_vsseg7_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -579,7 +545,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i3
 define void @test_vsseg8_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -598,7 +563,6 @@ entry:
 define void @test_vsseg8_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -620,7 +584,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i1
 define void @test_vsseg2_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -633,7 +596,6 @@ entry:
 define void @test_vsseg2_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -649,7 +611,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i1
 define void @test_vsseg3_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -663,7 +624,6 @@ entry:
 define void @test_vsseg3_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -680,7 +640,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i1
 define void @test_vsseg4_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -695,7 +654,6 @@ entry:
 define void @test_vsseg4_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -713,7 +671,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i1
 define void @test_vsseg5_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -729,7 +686,6 @@ entry:
 define void @test_vsseg5_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -748,7 +704,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i1
 define void @test_vsseg6_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -765,7 +720,6 @@ entry:
 define void @test_vsseg6_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -785,7 +739,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i1
 define void @test_vsseg7_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -803,7 +756,6 @@ entry:
 define void @test_vsseg7_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -824,7 +776,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i1
 define void @test_vsseg8_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -843,7 +794,6 @@ entry:
 define void @test_vsseg8_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -865,7 +815,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i3
 define void @test_vsseg2_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0)
@@ -878,7 +827,6 @@ entry:
 define void @test_vsseg2_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0), v0.t
@@ -894,7 +842,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i3
 define void @test_vsseg3_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -908,7 +855,6 @@ entry:
 define void @test_vsseg3_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -925,7 +871,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i3
 define void @test_vsseg4_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -940,7 +885,6 @@ entry:
 define void @test_vsseg4_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -958,7 +902,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i3
 define void @test_vsseg5_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -974,7 +917,6 @@ entry:
 define void @test_vsseg5_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -993,7 +935,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i3
 define void @test_vsseg6_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1010,7 +951,6 @@ entry:
 define void @test_vsseg6_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1030,7 +970,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i3
 define void @test_vsseg7_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1048,7 +987,6 @@ entry:
 define void @test_vsseg7_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1069,7 +1007,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i3
 define void @test_vsseg8_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1088,7 +1025,6 @@ entry:
 define void @test_vsseg8_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1110,7 +1046,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv8i16(<vscale x 8 x i16>,<vscale x 8 x i1
 define void @test_vsseg2_nxv8i16(<vscale x 8 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -1123,7 +1058,6 @@ entry:
 define void @test_vsseg2_mask_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -1139,7 +1073,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv8i16(<vscale x 8 x i16>,<vscale x 8 x i1
 define void @test_vsseg3_nxv8i16(<vscale x 8 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -1153,7 +1086,6 @@ entry:
 define void @test_vsseg3_mask_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -1170,7 +1102,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv8i16(<vscale x 8 x i16>,<vscale x 8 x i1
 define void @test_vsseg4_nxv8i16(<vscale x 8 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -1185,7 +1116,6 @@ entry:
 define void @test_vsseg4_mask_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -1203,7 +1133,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>,
 define void @test_vsseg2_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0)
@@ -1216,7 +1145,6 @@ entry:
 define void @test_vsseg2_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0), v0.t
@@ -1232,7 +1160,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>,
 define void @test_vsseg3_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -1246,7 +1173,6 @@ entry:
 define void @test_vsseg3_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -1263,7 +1189,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>,
 define void @test_vsseg4_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1278,7 +1203,6 @@ entry:
 define void @test_vsseg4_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1296,7 +1220,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>,
 define void @test_vsseg5_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1312,7 +1235,6 @@ entry:
 define void @test_vsseg5_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1331,7 +1253,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>,
 define void @test_vsseg6_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1348,7 +1269,6 @@ entry:
 define void @test_vsseg6_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1368,7 +1288,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>,
 define void @test_vsseg7_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1386,7 +1305,6 @@ entry:
 define void @test_vsseg7_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1407,7 +1325,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>,
 define void @test_vsseg8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1426,7 +1343,6 @@ entry:
 define void @test_vsseg8_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1448,7 +1364,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv8i32(<vscale x 8 x i32>,<vscale x 8 x i3
 define void @test_vsseg2_nxv8i32(<vscale x 8 x i32> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0)
@@ -1461,7 +1376,6 @@ entry:
 define void @test_vsseg2_mask_nxv8i32(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0), v0.t
@@ -1477,7 +1391,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>,
 define void @test_vsseg2_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0)
@@ -1490,7 +1403,6 @@ entry:
 define void @test_vsseg2_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0), v0.t
@@ -1506,7 +1418,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>,
 define void @test_vsseg3_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -1520,7 +1431,6 @@ entry:
 define void @test_vsseg3_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -1537,7 +1447,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>,
 define void @test_vsseg4_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1552,7 +1461,6 @@ entry:
 define void @test_vsseg4_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1570,7 +1478,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>,
 define void @test_vsseg5_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1586,7 +1493,6 @@ entry:
 define void @test_vsseg5_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1605,7 +1511,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>,
 define void @test_vsseg6_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1622,7 +1527,6 @@ entry:
 define void @test_vsseg6_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1642,7 +1546,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>,
 define void @test_vsseg7_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1660,7 +1563,6 @@ entry:
 define void @test_vsseg7_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1681,7 +1583,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>,
 define void @test_vsseg8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1700,7 +1601,6 @@ entry:
 define void @test_vsseg8_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1722,7 +1622,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i1
 define void @test_vsseg2_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -1735,7 +1634,6 @@ entry:
 define void @test_vsseg2_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -1751,7 +1649,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i1
 define void @test_vsseg3_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -1765,7 +1662,6 @@ entry:
 define void @test_vsseg3_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -1782,7 +1678,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i1
 define void @test_vsseg4_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1797,7 +1692,6 @@ entry:
 define void @test_vsseg4_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1815,7 +1709,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i1
 define void @test_vsseg5_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1831,7 +1724,6 @@ entry:
 define void @test_vsseg5_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1850,7 +1742,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i1
 define void @test_vsseg6_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1867,7 +1758,6 @@ entry:
 define void @test_vsseg6_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1887,7 +1777,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i1
 define void @test_vsseg7_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1905,7 +1794,6 @@ entry:
 define void @test_vsseg7_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1926,7 +1814,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i1
 define void @test_vsseg8_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1945,7 +1832,6 @@ entry:
 define void @test_vsseg8_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1967,7 +1853,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv32i8(<vscale x 32 x i8>,<vscale x 32 x i
 define void @test_vsseg2_nxv32i8(<vscale x 32 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv32i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0)
@@ -1980,7 +1865,6 @@ entry:
 define void @test_vsseg2_mask_nxv32i8(<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv32i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0), v0.t
@@ -1996,7 +1880,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>,
 define void @test_vsseg2_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0)
@@ -2009,7 +1892,6 @@ entry:
 define void @test_vsseg2_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0), v0.t
@@ -2025,7 +1907,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>,
 define void @test_vsseg3_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -2039,7 +1920,6 @@ entry:
 define void @test_vsseg3_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -2056,7 +1936,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>,
 define void @test_vsseg4_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2071,7 +1950,6 @@ entry:
 define void @test_vsseg4_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2089,7 +1967,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>,
 define void @test_vsseg5_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2105,7 +1982,6 @@ entry:
 define void @test_vsseg5_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2124,7 +2000,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>,
 define void @test_vsseg6_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2141,7 +2016,6 @@ entry:
 define void @test_vsseg6_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2161,7 +2035,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>,
 define void @test_vsseg7_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2179,7 +2052,6 @@ entry:
 define void @test_vsseg7_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2200,7 +2072,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>,
 define void @test_vsseg8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2219,7 +2090,6 @@ entry:
 define void @test_vsseg8_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2241,7 +2111,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i1
 define void @test_vsseg2_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -2254,7 +2123,6 @@ entry:
 define void @test_vsseg2_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -2270,7 +2138,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i1
 define void @test_vsseg3_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -2284,7 +2151,6 @@ entry:
 define void @test_vsseg3_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -2301,7 +2167,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i1
 define void @test_vsseg4_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2316,7 +2181,6 @@ entry:
 define void @test_vsseg4_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2334,7 +2198,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i1
 define void @test_vsseg5_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2350,7 +2213,6 @@ entry:
 define void @test_vsseg5_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2369,7 +2231,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i1
 define void @test_vsseg6_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2386,7 +2247,6 @@ entry:
 define void @test_vsseg6_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2406,7 +2266,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i1
 define void @test_vsseg7_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2424,7 +2283,6 @@ entry:
 define void @test_vsseg7_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2445,7 +2303,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i1
 define void @test_vsseg8_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2464,7 +2321,6 @@ entry:
 define void @test_vsseg8_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2486,7 +2342,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv4i32(<vscale x 4 x i32>,<vscale x 4 x i3
 define void @test_vsseg2_nxv4i32(<vscale x 4 x i32> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0)
@@ -2499,7 +2354,6 @@ entry:
 define void @test_vsseg2_mask_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0), v0.t
@@ -2515,7 +2369,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv4i32(<vscale x 4 x i32>,<vscale x 4 x i3
 define void @test_vsseg3_nxv4i32(<vscale x 4 x i32> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -2529,7 +2382,6 @@ entry:
 define void @test_vsseg3_mask_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -2546,7 +2398,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv4i32(<vscale x 4 x i32>,<vscale x 4 x i3
 define void @test_vsseg4_nxv4i32(<vscale x 4 x i32> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -2561,7 +2412,6 @@ entry:
 define void @test_vsseg4_mask_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -2579,7 +2429,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv16f16(<vscale x 16 x half>,<vscale x 16
 define void @test_vsseg2_nxv16f16(<vscale x 16 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv16f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -2592,7 +2441,6 @@ entry:
 define void @test_vsseg2_mask_nxv16f16(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv16f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -2608,7 +2456,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv4f64(<vscale x 4 x double>,<vscale x 4 x
 define void @test_vsseg2_nxv4f64(<vscale x 4 x double> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv4f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e64.v v8, (a0)
@@ -2621,7 +2468,6 @@ entry:
 define void @test_vsseg2_mask_nxv4f64(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv4f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e64.v v8, (a0), v0.t
@@ -2637,7 +2483,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1 x
 define void @test_vsseg2_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e64.v v8, (a0)
@@ -2650,7 +2495,6 @@ entry:
 define void @test_vsseg2_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e64.v v8, (a0), v0.t
@@ -2666,7 +2510,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1 x
 define void @test_vsseg3_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -2680,7 +2523,6 @@ entry:
 define void @test_vsseg3_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -2697,7 +2539,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1 x
 define void @test_vsseg4_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2712,7 +2553,6 @@ entry:
 define void @test_vsseg4_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2730,7 +2570,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1 x
 define void @test_vsseg5_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2746,7 +2585,6 @@ entry:
 define void @test_vsseg5_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2765,7 +2603,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1 x
 define void @test_vsseg6_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2782,7 +2619,6 @@ entry:
 define void @test_vsseg6_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2802,7 +2638,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1 x
 define void @test_vsseg7_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2820,7 +2655,6 @@ entry:
 define void @test_vsseg7_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2841,7 +2675,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1 x
 define void @test_vsseg8_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2860,7 +2693,6 @@ entry:
 define void @test_vsseg8_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2882,7 +2714,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vsseg2_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0)
@@ -2895,7 +2726,6 @@ entry:
 define void @test_vsseg2_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0), v0.t
@@ -2911,7 +2741,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vsseg3_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -2925,7 +2754,6 @@ entry:
 define void @test_vsseg3_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -2942,7 +2770,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vsseg4_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2957,7 +2784,6 @@ entry:
 define void @test_vsseg4_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2975,7 +2801,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vsseg5_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2991,7 +2816,6 @@ entry:
 define void @test_vsseg5_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3010,7 +2834,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vsseg6_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3027,7 +2850,6 @@ entry:
 define void @test_vsseg6_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3047,7 +2869,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vsseg7_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3065,7 +2886,6 @@ entry:
 define void @test_vsseg7_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3086,7 +2906,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vsseg8_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3105,7 +2924,6 @@ entry:
 define void @test_vsseg8_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3127,7 +2945,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x h
 define void @test_vsseg2_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -3140,7 +2957,6 @@ entry:
 define void @test_vsseg2_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -3156,7 +2972,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x h
 define void @test_vsseg3_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -3170,7 +2985,6 @@ entry:
 define void @test_vsseg3_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -3187,7 +3001,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x h
 define void @test_vsseg4_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3202,7 +3015,6 @@ entry:
 define void @test_vsseg4_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3220,7 +3032,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x h
 define void @test_vsseg5_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3236,7 +3047,6 @@ entry:
 define void @test_vsseg5_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3255,7 +3065,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x h
 define void @test_vsseg6_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3272,7 +3081,6 @@ entry:
 define void @test_vsseg6_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3292,7 +3100,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x h
 define void @test_vsseg7_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3310,7 +3117,6 @@ entry:
 define void @test_vsseg7_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3331,7 +3137,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x h
 define void @test_vsseg8_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3350,7 +3155,6 @@ entry:
 define void @test_vsseg8_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3372,7 +3176,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vsseg2_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0)
@@ -3385,7 +3188,6 @@ entry:
 define void @test_vsseg2_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0), v0.t
@@ -3401,7 +3203,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vsseg3_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -3415,7 +3216,6 @@ entry:
 define void @test_vsseg3_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -3432,7 +3232,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vsseg4_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3447,7 +3246,6 @@ entry:
 define void @test_vsseg4_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3465,7 +3263,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vsseg5_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3481,7 +3278,6 @@ entry:
 define void @test_vsseg5_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3500,7 +3296,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vsseg6_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3517,7 +3312,6 @@ entry:
 define void @test_vsseg6_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3537,7 +3331,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vsseg7_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3555,7 +3348,6 @@ entry:
 define void @test_vsseg7_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3576,7 +3368,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vsseg8_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3595,7 +3386,6 @@ entry:
 define void @test_vsseg8_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3617,7 +3407,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv8f16(<vscale x 8 x half>,<vscale x 8 x h
 define void @test_vsseg2_nxv8f16(<vscale x 8 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -3630,7 +3419,6 @@ entry:
 define void @test_vsseg2_mask_nxv8f16(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -3646,7 +3434,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv8f16(<vscale x 8 x half>,<vscale x 8 x h
 define void @test_vsseg3_nxv8f16(<vscale x 8 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -3660,7 +3447,6 @@ entry:
 define void @test_vsseg3_mask_nxv8f16(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -3677,7 +3463,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv8f16(<vscale x 8 x half>,<vscale x 8 x h
 define void @test_vsseg4_nxv8f16(<vscale x 8 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -3692,7 +3477,6 @@ entry:
 define void @test_vsseg4_mask_nxv8f16(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -3710,7 +3494,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv8f32(<vscale x 8 x float>,<vscale x 8 x
 define void @test_vsseg2_nxv8f32(<vscale x 8 x float> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv8f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0)
@@ -3723,7 +3506,6 @@ entry:
 define void @test_vsseg2_mask_nxv8f32(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv8f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0), v0.t
@@ -3739,7 +3521,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv2f64(<vscale x 2 x double>,<vscale x 2 x
 define void @test_vsseg2_nxv2f64(<vscale x 2 x double> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e64.v v8, (a0)
@@ -3752,7 +3533,6 @@ entry:
 define void @test_vsseg2_mask_nxv2f64(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e64.v v8, (a0), v0.t
@@ -3768,7 +3548,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv2f64(<vscale x 2 x double>,<vscale x 2 x
 define void @test_vsseg3_nxv2f64(<vscale x 2 x double> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -3782,7 +3561,6 @@ entry:
 define void @test_vsseg3_mask_nxv2f64(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -3799,7 +3577,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv2f64(<vscale x 2 x double>,<vscale x 2 x
 define void @test_vsseg4_nxv2f64(<vscale x 2 x double> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -3814,7 +3591,6 @@ entry:
 define void @test_vsseg4_mask_nxv2f64(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -3832,7 +3608,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x h
 define void @test_vsseg2_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -3845,7 +3620,6 @@ entry:
 define void @test_vsseg2_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -3861,7 +3635,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x h
 define void @test_vsseg3_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -3875,7 +3648,6 @@ entry:
 define void @test_vsseg3_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -3892,7 +3664,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x h
 define void @test_vsseg4_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3907,7 +3678,6 @@ entry:
 define void @test_vsseg4_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3925,7 +3695,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x h
 define void @test_vsseg5_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3941,7 +3710,6 @@ entry:
 define void @test_vsseg5_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3960,7 +3728,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x h
 define void @test_vsseg6_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3977,7 +3744,6 @@ entry:
 define void @test_vsseg6_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3997,7 +3763,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x h
 define void @test_vsseg7_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4015,7 +3780,6 @@ entry:
 define void @test_vsseg7_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4036,7 +3800,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x h
 define void @test_vsseg8_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4055,7 +3818,6 @@ entry:
 define void @test_vsseg8_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4077,7 +3839,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x h
 define void @test_vsseg2_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -4090,7 +3851,6 @@ entry:
 define void @test_vsseg2_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -4106,7 +3866,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x h
 define void @test_vsseg3_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -4120,7 +3879,6 @@ entry:
 define void @test_vsseg3_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -4137,7 +3895,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x h
 define void @test_vsseg4_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4152,7 +3909,6 @@ entry:
 define void @test_vsseg4_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4170,7 +3926,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x h
 define void @test_vsseg5_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4186,7 +3941,6 @@ entry:
 define void @test_vsseg5_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4205,7 +3959,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x h
 define void @test_vsseg6_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4222,7 +3975,6 @@ entry:
 define void @test_vsseg6_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4242,7 +3994,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x h
 define void @test_vsseg7_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4260,7 +4011,6 @@ entry:
 define void @test_vsseg7_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4281,7 +4031,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x h
 define void @test_vsseg8_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4300,7 +4049,6 @@ entry:
 define void @test_vsseg8_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4322,7 +4070,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv4f32(<vscale x 4 x float>,<vscale x 4 x
 define void @test_vsseg2_nxv4f32(<vscale x 4 x float> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0)
@@ -4335,7 +4082,6 @@ entry:
 define void @test_vsseg2_mask_nxv4f32(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0), v0.t
@@ -4351,7 +4097,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv4f32(<vscale x 4 x float>,<vscale x 4 x
 define void @test_vsseg3_nxv4f32(<vscale x 4 x float> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -4365,7 +4110,6 @@ entry:
 define void @test_vsseg3_mask_nxv4f32(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -4382,7 +4126,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv4f32(<vscale x 4 x float>,<vscale x 4 x
 define void @test_vsseg4_nxv4f32(<vscale x 4 x float> %val, ptr %base, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -4397,7 +4140,6 @@ entry:
 define void @test_vsseg4_mask_nxv4f32(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
diff --git a/llvm/test/CodeGen/RISCV/rvv/vsseg-rv64.ll b/llvm/test/CodeGen/RISCV/rvv/vsseg-rv64.ll
index 293bdbce00acf..3031cb4abc053 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vsseg-rv64.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vsseg-rv64.ll
@@ -8,7 +8,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv16i16(<vscale x 16 x i16>,<vscale x 16 x
 define void @test_vsseg2_nxv16i16(<vscale x 16 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -21,7 +20,6 @@ entry:
 define void @test_vsseg2_mask_nxv16i16(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -37,7 +35,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv4i32(<vscale x 4 x i32>,<vscale x 4 x i3
 define void @test_vsseg2_nxv4i32(<vscale x 4 x i32> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0)
@@ -50,7 +47,6 @@ entry:
 define void @test_vsseg2_mask_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0), v0.t
@@ -66,7 +62,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv4i32(<vscale x 4 x i32>,<vscale x 4 x i3
 define void @test_vsseg3_nxv4i32(<vscale x 4 x i32> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -80,7 +75,6 @@ entry:
 define void @test_vsseg3_mask_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -97,7 +91,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv4i32(<vscale x 4 x i32>,<vscale x 4 x i3
 define void @test_vsseg4_nxv4i32(<vscale x 4 x i32> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -112,7 +105,6 @@ entry:
 define void @test_vsseg4_mask_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -130,7 +122,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv16i8(<vscale x 16 x i8>,<vscale x 16 x i
 define void @test_vsseg2_nxv16i8(<vscale x 16 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0)
@@ -143,7 +134,6 @@ entry:
 define void @test_vsseg2_mask_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0), v0.t
@@ -159,7 +149,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv16i8(<vscale x 16 x i8>,<vscale x 16 x i
 define void @test_vsseg3_nxv16i8(<vscale x 16 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -173,7 +162,6 @@ entry:
 define void @test_vsseg3_mask_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -190,7 +178,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv16i8(<vscale x 16 x i8>,<vscale x 16 x i
 define void @test_vsseg4_nxv16i8(<vscale x 16 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -205,7 +192,6 @@ entry:
 define void @test_vsseg4_mask_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -223,7 +209,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv1i64(<vscale x 1 x i64>,<vscale x 1 x i6
 define void @test_vsseg2_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e64.v v8, (a0)
@@ -236,7 +221,6 @@ entry:
 define void @test_vsseg2_mask_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e64.v v8, (a0), v0.t
@@ -252,7 +236,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv1i64(<vscale x 1 x i64>,<vscale x 1 x i6
 define void @test_vsseg3_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -266,7 +249,6 @@ entry:
 define void @test_vsseg3_mask_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -283,7 +265,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv1i64(<vscale x 1 x i64>,<vscale x 1 x i6
 define void @test_vsseg4_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -298,7 +279,6 @@ entry:
 define void @test_vsseg4_mask_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -316,7 +296,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv1i64(<vscale x 1 x i64>,<vscale x 1 x i6
 define void @test_vsseg5_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -332,7 +311,6 @@ entry:
 define void @test_vsseg5_mask_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -351,7 +329,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv1i64(<vscale x 1 x i64>,<vscale x 1 x i6
 define void @test_vsseg6_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -368,7 +345,6 @@ entry:
 define void @test_vsseg6_mask_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -388,7 +364,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv1i64(<vscale x 1 x i64>,<vscale x 1 x i6
 define void @test_vsseg7_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -406,7 +381,6 @@ entry:
 define void @test_vsseg7_mask_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -427,7 +401,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv1i64(<vscale x 1 x i64>,<vscale x 1 x i6
 define void @test_vsseg8_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -446,7 +419,6 @@ entry:
 define void @test_vsseg8_mask_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -468,7 +440,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i3
 define void @test_vsseg2_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0)
@@ -481,7 +452,6 @@ entry:
 define void @test_vsseg2_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0), v0.t
@@ -497,7 +467,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i3
 define void @test_vsseg3_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -511,7 +480,6 @@ entry:
 define void @test_vsseg3_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -528,7 +496,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i3
 define void @test_vsseg4_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -543,7 +510,6 @@ entry:
 define void @test_vsseg4_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -561,7 +527,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i3
 define void @test_vsseg5_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -577,7 +542,6 @@ entry:
 define void @test_vsseg5_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -596,7 +560,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i3
 define void @test_vsseg6_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -613,7 +576,6 @@ entry:
 define void @test_vsseg6_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -633,7 +595,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i3
 define void @test_vsseg7_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -651,7 +612,6 @@ entry:
 define void @test_vsseg7_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -672,7 +632,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i3
 define void @test_vsseg8_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -691,7 +650,6 @@ entry:
 define void @test_vsseg8_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -713,7 +671,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv8i16(<vscale x 8 x i16>,<vscale x 8 x i1
 define void @test_vsseg2_nxv8i16(<vscale x 8 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -726,7 +683,6 @@ entry:
 define void @test_vsseg2_mask_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -742,7 +698,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv8i16(<vscale x 8 x i16>,<vscale x 8 x i1
 define void @test_vsseg3_nxv8i16(<vscale x 8 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -756,7 +711,6 @@ entry:
 define void @test_vsseg3_mask_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -773,7 +727,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv8i16(<vscale x 8 x i16>,<vscale x 8 x i1
 define void @test_vsseg4_nxv8i16(<vscale x 8 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -788,7 +741,6 @@ entry:
 define void @test_vsseg4_mask_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -806,7 +758,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>,
 define void @test_vsseg2_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0)
@@ -819,7 +770,6 @@ entry:
 define void @test_vsseg2_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0), v0.t
@@ -835,7 +785,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>,
 define void @test_vsseg3_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -849,7 +798,6 @@ entry:
 define void @test_vsseg3_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -866,7 +814,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>,
 define void @test_vsseg4_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -881,7 +828,6 @@ entry:
 define void @test_vsseg4_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -899,7 +845,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>,
 define void @test_vsseg5_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -915,7 +860,6 @@ entry:
 define void @test_vsseg5_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -934,7 +878,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>,
 define void @test_vsseg6_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -951,7 +894,6 @@ entry:
 define void @test_vsseg6_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -971,7 +913,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>,
 define void @test_vsseg7_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -989,7 +930,6 @@ entry:
 define void @test_vsseg7_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1010,7 +950,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>,
 define void @test_vsseg8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1029,7 +968,6 @@ entry:
 define void @test_vsseg8_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1051,7 +989,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i1
 define void @test_vsseg2_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -1064,7 +1001,6 @@ entry:
 define void @test_vsseg2_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -1080,7 +1016,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i1
 define void @test_vsseg3_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -1094,7 +1029,6 @@ entry:
 define void @test_vsseg3_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -1111,7 +1045,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i1
 define void @test_vsseg4_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1126,7 +1059,6 @@ entry:
 define void @test_vsseg4_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1144,7 +1076,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i1
 define void @test_vsseg5_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1160,7 +1091,6 @@ entry:
 define void @test_vsseg5_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1179,7 +1109,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i1
 define void @test_vsseg6_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1196,7 +1125,6 @@ entry:
 define void @test_vsseg6_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1216,7 +1144,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i1
 define void @test_vsseg7_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1234,7 +1161,6 @@ entry:
 define void @test_vsseg7_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1255,7 +1181,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i1
 define void @test_vsseg8_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1274,7 +1199,6 @@ entry:
 define void @test_vsseg8_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1296,7 +1220,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i3
 define void @test_vsseg2_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0)
@@ -1309,7 +1232,6 @@ entry:
 define void @test_vsseg2_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0), v0.t
@@ -1325,7 +1247,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i3
 define void @test_vsseg3_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -1339,7 +1260,6 @@ entry:
 define void @test_vsseg3_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -1356,7 +1276,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i3
 define void @test_vsseg4_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1371,7 +1290,6 @@ entry:
 define void @test_vsseg4_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1389,7 +1307,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i3
 define void @test_vsseg5_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1405,7 +1322,6 @@ entry:
 define void @test_vsseg5_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1424,7 +1340,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i3
 define void @test_vsseg6_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1441,7 +1356,6 @@ entry:
 define void @test_vsseg6_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1461,7 +1375,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i3
 define void @test_vsseg7_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1479,7 +1392,6 @@ entry:
 define void @test_vsseg7_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1500,7 +1412,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i3
 define void @test_vsseg8_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1519,7 +1430,6 @@ entry:
 define void @test_vsseg8_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1541,7 +1451,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>,
 define void @test_vsseg2_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0)
@@ -1554,7 +1463,6 @@ entry:
 define void @test_vsseg2_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0), v0.t
@@ -1570,7 +1478,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>,
 define void @test_vsseg3_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -1584,7 +1491,6 @@ entry:
 define void @test_vsseg3_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -1601,7 +1507,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>,
 define void @test_vsseg4_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1616,7 +1521,6 @@ entry:
 define void @test_vsseg4_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1634,7 +1538,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>,
 define void @test_vsseg5_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1650,7 +1553,6 @@ entry:
 define void @test_vsseg5_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1669,7 +1571,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>,
 define void @test_vsseg6_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1686,7 +1587,6 @@ entry:
 define void @test_vsseg6_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1706,7 +1606,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>,
 define void @test_vsseg7_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1724,7 +1623,6 @@ entry:
 define void @test_vsseg7_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1745,7 +1643,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>,
 define void @test_vsseg8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1764,7 +1661,6 @@ entry:
 define void @test_vsseg8_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1786,7 +1682,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv4i64(<vscale x 4 x i64>,<vscale x 4 x i6
 define void @test_vsseg2_nxv4i64(<vscale x 4 x i64> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e64.v v8, (a0)
@@ -1799,7 +1694,6 @@ entry:
 define void @test_vsseg2_mask_nxv4i64(<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e64.v v8, (a0), v0.t
@@ -1815,7 +1709,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i1
 define void @test_vsseg2_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -1828,7 +1721,6 @@ entry:
 define void @test_vsseg2_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -1844,7 +1736,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i1
 define void @test_vsseg3_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -1858,7 +1749,6 @@ entry:
 define void @test_vsseg3_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -1875,7 +1765,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i1
 define void @test_vsseg4_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1890,7 +1779,6 @@ entry:
 define void @test_vsseg4_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1908,7 +1796,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i1
 define void @test_vsseg5_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1924,7 +1811,6 @@ entry:
 define void @test_vsseg5_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1943,7 +1829,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i1
 define void @test_vsseg6_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1960,7 +1845,6 @@ entry:
 define void @test_vsseg6_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1980,7 +1864,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i1
 define void @test_vsseg7_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1998,7 +1881,6 @@ entry:
 define void @test_vsseg7_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2019,7 +1901,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i1
 define void @test_vsseg8_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2038,7 +1919,6 @@ entry:
 define void @test_vsseg8_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2060,7 +1940,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>,
 define void @test_vsseg2_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0)
@@ -2073,7 +1952,6 @@ entry:
 define void @test_vsseg2_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0), v0.t
@@ -2089,7 +1967,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>,
 define void @test_vsseg3_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -2103,7 +1980,6 @@ entry:
 define void @test_vsseg3_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -2120,7 +1996,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>,
 define void @test_vsseg4_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2135,7 +2010,6 @@ entry:
 define void @test_vsseg4_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2153,7 +2027,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>,
 define void @test_vsseg5_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2169,7 +2042,6 @@ entry:
 define void @test_vsseg5_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2188,7 +2060,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>,
 define void @test_vsseg6_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2205,7 +2076,6 @@ entry:
 define void @test_vsseg6_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2225,7 +2095,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>,
 define void @test_vsseg7_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2243,7 +2112,6 @@ entry:
 define void @test_vsseg7_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2264,7 +2132,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>,
 define void @test_vsseg8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2283,7 +2150,6 @@ entry:
 define void @test_vsseg8_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2305,7 +2171,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>,
 define void @test_vsseg2_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0)
@@ -2318,7 +2183,6 @@ entry:
 define void @test_vsseg2_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0), v0.t
@@ -2334,7 +2198,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>,
 define void @test_vsseg3_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -2348,7 +2211,6 @@ entry:
 define void @test_vsseg3_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -2365,7 +2227,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>,
 define void @test_vsseg4_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2380,7 +2241,6 @@ entry:
 define void @test_vsseg4_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2398,7 +2258,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>,
 define void @test_vsseg5_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2414,7 +2273,6 @@ entry:
 define void @test_vsseg5_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2433,7 +2291,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>,
 define void @test_vsseg6_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2450,7 +2307,6 @@ entry:
 define void @test_vsseg6_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2470,7 +2326,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>,
 define void @test_vsseg7_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2488,7 +2343,6 @@ entry:
 define void @test_vsseg7_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2509,7 +2363,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>,
 define void @test_vsseg8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2528,7 +2381,6 @@ entry:
 define void @test_vsseg8_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2550,7 +2402,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv8i32(<vscale x 8 x i32>,<vscale x 8 x i3
 define void @test_vsseg2_nxv8i32(<vscale x 8 x i32> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0)
@@ -2563,7 +2414,6 @@ entry:
 define void @test_vsseg2_mask_nxv8i32(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0), v0.t
@@ -2579,7 +2429,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv32i8(<vscale x 32 x i8>,<vscale x 32 x i
 define void @test_vsseg2_nxv32i8(<vscale x 32 x i8> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv32i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0)
@@ -2592,7 +2441,6 @@ entry:
 define void @test_vsseg2_mask_nxv32i8(<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv32i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e8.v v8, (a0), v0.t
@@ -2608,7 +2456,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i1
 define void @test_vsseg2_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -2621,7 +2468,6 @@ entry:
 define void @test_vsseg2_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -2637,7 +2483,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i1
 define void @test_vsseg3_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -2651,7 +2496,6 @@ entry:
 define void @test_vsseg3_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -2668,7 +2512,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i1
 define void @test_vsseg4_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2683,7 +2526,6 @@ entry:
 define void @test_vsseg4_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2701,7 +2543,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i1
 define void @test_vsseg5_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2717,7 +2558,6 @@ entry:
 define void @test_vsseg5_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2736,7 +2576,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i1
 define void @test_vsseg6_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2753,7 +2592,6 @@ entry:
 define void @test_vsseg6_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2773,7 +2611,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i1
 define void @test_vsseg7_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2791,7 +2628,6 @@ entry:
 define void @test_vsseg7_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2812,7 +2648,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i1
 define void @test_vsseg8_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2831,7 +2666,6 @@ entry:
 define void @test_vsseg8_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2853,7 +2687,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv2i64(<vscale x 2 x i64>,<vscale x 2 x i6
 define void @test_vsseg2_nxv2i64(<vscale x 2 x i64> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e64.v v8, (a0)
@@ -2866,7 +2699,6 @@ entry:
 define void @test_vsseg2_mask_nxv2i64(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e64.v v8, (a0), v0.t
@@ -2882,7 +2714,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv2i64(<vscale x 2 x i64>,<vscale x 2 x i6
 define void @test_vsseg3_nxv2i64(<vscale x 2 x i64> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -2896,7 +2727,6 @@ entry:
 define void @test_vsseg3_mask_nxv2i64(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -2913,7 +2743,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv2i64(<vscale x 2 x i64>,<vscale x 2 x i6
 define void @test_vsseg4_nxv2i64(<vscale x 2 x i64> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -2928,7 +2757,6 @@ entry:
 define void @test_vsseg4_mask_nxv2i64(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -2946,7 +2774,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv16f16(<vscale x 16 x half>,<vscale x 16
 define void @test_vsseg2_nxv16f16(<vscale x 16 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv16f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -2959,7 +2786,6 @@ entry:
 define void @test_vsseg2_mask_nxv16f16(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv16f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -2975,7 +2801,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv4f64(<vscale x 4 x double>,<vscale x 4 x
 define void @test_vsseg2_nxv4f64(<vscale x 4 x double> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv4f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e64.v v8, (a0)
@@ -2988,7 +2813,6 @@ entry:
 define void @test_vsseg2_mask_nxv4f64(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv4f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e64.v v8, (a0), v0.t
@@ -3004,7 +2828,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1 x
 define void @test_vsseg2_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e64.v v8, (a0)
@@ -3017,7 +2840,6 @@ entry:
 define void @test_vsseg2_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e64.v v8, (a0), v0.t
@@ -3033,7 +2855,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1 x
 define void @test_vsseg3_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -3047,7 +2868,6 @@ entry:
 define void @test_vsseg3_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -3064,7 +2884,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1 x
 define void @test_vsseg4_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3079,7 +2898,6 @@ entry:
 define void @test_vsseg4_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3097,7 +2915,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1 x
 define void @test_vsseg5_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3113,7 +2930,6 @@ entry:
 define void @test_vsseg5_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3132,7 +2948,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1 x
 define void @test_vsseg6_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3149,7 +2964,6 @@ entry:
 define void @test_vsseg6_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3169,7 +2983,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1 x
 define void @test_vsseg7_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3187,7 +3000,6 @@ entry:
 define void @test_vsseg7_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3208,7 +3020,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1 x
 define void @test_vsseg8_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3227,7 +3038,6 @@ entry:
 define void @test_vsseg8_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3249,7 +3059,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vsseg2_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0)
@@ -3262,7 +3071,6 @@ entry:
 define void @test_vsseg2_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0), v0.t
@@ -3278,7 +3086,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vsseg3_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -3292,7 +3099,6 @@ entry:
 define void @test_vsseg3_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -3309,7 +3115,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vsseg4_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3324,7 +3129,6 @@ entry:
 define void @test_vsseg4_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3342,7 +3146,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vsseg5_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3358,7 +3161,6 @@ entry:
 define void @test_vsseg5_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3377,7 +3179,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vsseg6_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3394,7 +3195,6 @@ entry:
 define void @test_vsseg6_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3414,7 +3214,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vsseg7_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3432,7 +3231,6 @@ entry:
 define void @test_vsseg7_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3453,7 +3251,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vsseg8_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3472,7 +3269,6 @@ entry:
 define void @test_vsseg8_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3494,7 +3290,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x h
 define void @test_vsseg2_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -3507,7 +3302,6 @@ entry:
 define void @test_vsseg2_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -3523,7 +3317,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x h
 define void @test_vsseg3_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -3537,7 +3330,6 @@ entry:
 define void @test_vsseg3_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -3554,7 +3346,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x h
 define void @test_vsseg4_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3569,7 +3360,6 @@ entry:
 define void @test_vsseg4_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3587,7 +3377,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x h
 define void @test_vsseg5_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3603,7 +3392,6 @@ entry:
 define void @test_vsseg5_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3622,7 +3410,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x h
 define void @test_vsseg6_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3639,7 +3426,6 @@ entry:
 define void @test_vsseg6_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3659,7 +3445,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x h
 define void @test_vsseg7_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3677,7 +3462,6 @@ entry:
 define void @test_vsseg7_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3698,7 +3482,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x h
 define void @test_vsseg8_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3717,7 +3500,6 @@ entry:
 define void @test_vsseg8_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3739,7 +3521,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vsseg2_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0)
@@ -3752,7 +3533,6 @@ entry:
 define void @test_vsseg2_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0), v0.t
@@ -3768,7 +3548,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vsseg3_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -3782,7 +3561,6 @@ entry:
 define void @test_vsseg3_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -3799,7 +3577,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vsseg4_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3814,7 +3591,6 @@ entry:
 define void @test_vsseg4_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3832,7 +3608,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vsseg5_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3848,7 +3623,6 @@ entry:
 define void @test_vsseg5_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3867,7 +3641,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vsseg6_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3884,7 +3657,6 @@ entry:
 define void @test_vsseg6_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3904,7 +3676,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vsseg7_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3922,7 +3693,6 @@ entry:
 define void @test_vsseg7_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3943,7 +3713,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vsseg8_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3962,7 +3731,6 @@ entry:
 define void @test_vsseg8_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3984,7 +3752,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv8f16(<vscale x 8 x half>,<vscale x 8 x h
 define void @test_vsseg2_nxv8f16(<vscale x 8 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -3997,7 +3764,6 @@ entry:
 define void @test_vsseg2_mask_nxv8f16(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -4013,7 +3779,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv8f16(<vscale x 8 x half>,<vscale x 8 x h
 define void @test_vsseg3_nxv8f16(<vscale x 8 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -4027,7 +3792,6 @@ entry:
 define void @test_vsseg3_mask_nxv8f16(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -4044,7 +3808,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv8f16(<vscale x 8 x half>,<vscale x 8 x h
 define void @test_vsseg4_nxv8f16(<vscale x 8 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -4059,7 +3822,6 @@ entry:
 define void @test_vsseg4_mask_nxv8f16(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -4077,7 +3839,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv8f32(<vscale x 8 x float>,<vscale x 8 x
 define void @test_vsseg2_nxv8f32(<vscale x 8 x float> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv8f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0)
@@ -4090,7 +3851,6 @@ entry:
 define void @test_vsseg2_mask_nxv8f32(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv8f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0), v0.t
@@ -4106,7 +3866,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv2f64(<vscale x 2 x double>,<vscale x 2 x
 define void @test_vsseg2_nxv2f64(<vscale x 2 x double> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e64.v v8, (a0)
@@ -4119,7 +3878,6 @@ entry:
 define void @test_vsseg2_mask_nxv2f64(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e64.v v8, (a0), v0.t
@@ -4135,7 +3893,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv2f64(<vscale x 2 x double>,<vscale x 2 x
 define void @test_vsseg3_nxv2f64(<vscale x 2 x double> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -4149,7 +3906,6 @@ entry:
 define void @test_vsseg3_mask_nxv2f64(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -4166,7 +3922,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv2f64(<vscale x 2 x double>,<vscale x 2 x
 define void @test_vsseg4_nxv2f64(<vscale x 2 x double> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -4181,7 +3936,6 @@ entry:
 define void @test_vsseg4_mask_nxv2f64(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -4199,7 +3953,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x h
 define void @test_vsseg2_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -4212,7 +3965,6 @@ entry:
 define void @test_vsseg2_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -4228,7 +3980,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x h
 define void @test_vsseg3_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -4242,7 +3993,6 @@ entry:
 define void @test_vsseg3_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -4259,7 +4009,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x h
 define void @test_vsseg4_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4274,7 +4023,6 @@ entry:
 define void @test_vsseg4_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4292,7 +4040,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x h
 define void @test_vsseg5_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4308,7 +4055,6 @@ entry:
 define void @test_vsseg5_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4327,7 +4073,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x h
 define void @test_vsseg6_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4344,7 +4089,6 @@ entry:
 define void @test_vsseg6_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4364,7 +4108,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x h
 define void @test_vsseg7_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4382,7 +4125,6 @@ entry:
 define void @test_vsseg7_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4403,7 +4145,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x h
 define void @test_vsseg8_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4422,7 +4163,6 @@ entry:
 define void @test_vsseg8_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4444,7 +4184,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x h
 define void @test_vsseg2_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -4457,7 +4196,6 @@ entry:
 define void @test_vsseg2_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -4473,7 +4211,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x h
 define void @test_vsseg3_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -4487,7 +4224,6 @@ entry:
 define void @test_vsseg3_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -4504,7 +4240,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x h
 define void @test_vsseg4_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4519,7 +4254,6 @@ entry:
 define void @test_vsseg4_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4537,7 +4271,6 @@ declare void @llvm.riscv.vsseg5.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x h
 define void @test_vsseg5_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4553,7 +4286,6 @@ entry:
 define void @test_vsseg5_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg5_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4572,7 +4304,6 @@ declare void @llvm.riscv.vsseg6.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x h
 define void @test_vsseg6_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4589,7 +4320,6 @@ entry:
 define void @test_vsseg6_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg6_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4609,7 +4339,6 @@ declare void @llvm.riscv.vsseg7.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x h
 define void @test_vsseg7_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4627,7 +4356,6 @@ entry:
 define void @test_vsseg7_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg7_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4648,7 +4376,6 @@ declare void @llvm.riscv.vsseg8.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x h
 define void @test_vsseg8_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4667,7 +4394,6 @@ entry:
 define void @test_vsseg8_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg8_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4689,7 +4415,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv4f32(<vscale x 4 x float>,<vscale x 4 x
 define void @test_vsseg2_nxv4f32(<vscale x 4 x float> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0)
@@ -4702,7 +4427,6 @@ entry:
 define void @test_vsseg2_mask_nxv4f32(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsseg2e32.v v8, (a0), v0.t
@@ -4718,7 +4442,6 @@ declare void @llvm.riscv.vsseg3.mask.nxv4f32(<vscale x 4 x float>,<vscale x 4 x
 define void @test_vsseg3_nxv4f32(<vscale x 4 x float> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -4732,7 +4455,6 @@ entry:
 define void @test_vsseg3_mask_nxv4f32(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg3_mask_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -4749,7 +4471,6 @@ declare void @llvm.riscv.vsseg4.mask.nxv4f32(<vscale x 4 x float>,<vscale x 4 x
 define void @test_vsseg4_nxv4f32(<vscale x 4 x float> %val, ptr %base, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -4764,7 +4485,6 @@ entry:
 define void @test_vsseg4_mask_nxv4f32(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsseg4_mask_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
diff --git a/llvm/test/CodeGen/RISCV/rvv/vssseg-rv32.ll b/llvm/test/CodeGen/RISCV/rvv/vssseg-rv32.ll
index 30b407124b3a1..01552a72cc930 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vssseg-rv32.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vssseg-rv32.ll
@@ -8,7 +8,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv16i16(<vscale x 16 x i16>,<vscale x 16
 define void @test_vssseg2_nxv16i16(<vscale x 16 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -21,7 +20,6 @@ entry:
 define void @test_vssseg2_mask_nxv16i16(<vscale x 16 x i16> %val, ptr %base, i32 %offset, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -37,7 +35,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>
 define void @test_vssseg2_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1
@@ -50,7 +47,6 @@ entry:
 define void @test_vssseg2_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1, v0.t
@@ -66,7 +62,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>
 define void @test_vssseg3_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
@@ -80,7 +75,6 @@ entry:
 define void @test_vssseg3_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
@@ -97,7 +91,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>
 define void @test_vssseg4_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -112,7 +105,6 @@ entry:
 define void @test_vssseg4_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -130,7 +122,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>
 define void @test_vssseg5_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -146,7 +137,6 @@ entry:
 define void @test_vssseg5_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -165,7 +155,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>
 define void @test_vssseg6_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -182,7 +171,6 @@ entry:
 define void @test_vssseg6_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -202,7 +190,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>
 define void @test_vssseg7_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -220,7 +207,6 @@ entry:
 define void @test_vssseg7_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -241,7 +227,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>
 define void @test_vssseg8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -260,7 +245,6 @@ entry:
 define void @test_vssseg8_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -282,7 +266,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv16i8(<vscale x 16 x i8>,<vscale x 16 x
 define void @test_vssseg2_nxv16i8(<vscale x 16 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1
@@ -295,7 +278,6 @@ entry:
 define void @test_vssseg2_mask_nxv16i8(<vscale x 16 x i8> %val, ptr %base, i32 %offset, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1, v0.t
@@ -311,7 +293,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv16i8(<vscale x 16 x i8>,<vscale x 16 x
 define void @test_vssseg3_nxv16i8(<vscale x 16 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
@@ -325,7 +306,6 @@ entry:
 define void @test_vssseg3_mask_nxv16i8(<vscale x 16 x i8> %val, ptr %base, i32 %offset, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
@@ -342,7 +322,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv16i8(<vscale x 16 x i8>,<vscale x 16 x
 define void @test_vssseg4_nxv16i8(<vscale x 16 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -357,7 +336,6 @@ entry:
 define void @test_vssseg4_mask_nxv16i8(<vscale x 16 x i8> %val, ptr %base, i32 %offset, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -375,7 +353,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i
 define void @test_vssseg2_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1
@@ -388,7 +365,6 @@ entry:
 define void @test_vssseg2_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1, v0.t
@@ -404,7 +380,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i
 define void @test_vssseg3_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
@@ -418,7 +393,6 @@ entry:
 define void @test_vssseg3_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
@@ -435,7 +409,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i
 define void @test_vssseg4_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -450,7 +423,6 @@ entry:
 define void @test_vssseg4_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -468,7 +440,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i
 define void @test_vssseg5_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -484,7 +455,6 @@ entry:
 define void @test_vssseg5_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -503,7 +473,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i
 define void @test_vssseg6_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -520,7 +489,6 @@ entry:
 define void @test_vssseg6_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -540,7 +508,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i
 define void @test_vssseg7_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -558,7 +525,6 @@ entry:
 define void @test_vssseg7_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -579,7 +545,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i
 define void @test_vssseg8_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -598,7 +563,6 @@ entry:
 define void @test_vssseg8_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -620,7 +584,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i
 define void @test_vssseg2_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -633,7 +596,6 @@ entry:
 define void @test_vssseg2_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -649,7 +611,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i
 define void @test_vssseg3_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
@@ -663,7 +624,6 @@ entry:
 define void @test_vssseg3_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
@@ -680,7 +640,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i
 define void @test_vssseg4_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -695,7 +654,6 @@ entry:
 define void @test_vssseg4_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -713,7 +671,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i
 define void @test_vssseg5_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -729,7 +686,6 @@ entry:
 define void @test_vssseg5_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -748,7 +704,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i
 define void @test_vssseg6_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -765,7 +720,6 @@ entry:
 define void @test_vssseg6_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -785,7 +739,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i
 define void @test_vssseg7_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -803,7 +756,6 @@ entry:
 define void @test_vssseg7_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -824,7 +776,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i
 define void @test_vssseg8_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -843,7 +794,6 @@ entry:
 define void @test_vssseg8_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -865,7 +815,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i
 define void @test_vssseg2_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1
@@ -878,7 +827,6 @@ entry:
 define void @test_vssseg2_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1, v0.t
@@ -894,7 +842,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i
 define void @test_vssseg3_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
@@ -908,7 +855,6 @@ entry:
 define void @test_vssseg3_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
@@ -925,7 +871,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i
 define void @test_vssseg4_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -940,7 +885,6 @@ entry:
 define void @test_vssseg4_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -958,7 +902,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i
 define void @test_vssseg5_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -974,7 +917,6 @@ entry:
 define void @test_vssseg5_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -993,7 +935,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i
 define void @test_vssseg6_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1010,7 +951,6 @@ entry:
 define void @test_vssseg6_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1030,7 +970,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i
 define void @test_vssseg7_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1048,7 +987,6 @@ entry:
 define void @test_vssseg7_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1069,7 +1007,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i
 define void @test_vssseg8_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1088,7 +1025,6 @@ entry:
 define void @test_vssseg8_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1110,7 +1046,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv8i16(<vscale x 8 x i16>,<vscale x 8 x i
 define void @test_vssseg2_nxv8i16(<vscale x 8 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -1123,7 +1058,6 @@ entry:
 define void @test_vssseg2_mask_nxv8i16(<vscale x 8 x i16> %val, ptr %base, i32 %offset, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -1139,7 +1073,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv8i16(<vscale x 8 x i16>,<vscale x 8 x i
 define void @test_vssseg3_nxv8i16(<vscale x 8 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
@@ -1153,7 +1086,6 @@ entry:
 define void @test_vssseg3_mask_nxv8i16(<vscale x 8 x i16> %val, ptr %base, i32 %offset, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
@@ -1170,7 +1102,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv8i16(<vscale x 8 x i16>,<vscale x 8 x i
 define void @test_vssseg4_nxv8i16(<vscale x 8 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -1185,7 +1116,6 @@ entry:
 define void @test_vssseg4_mask_nxv8i16(<vscale x 8 x i16> %val, ptr %base, i32 %offset, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -1203,7 +1133,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>
 define void @test_vssseg2_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1
@@ -1216,7 +1145,6 @@ entry:
 define void @test_vssseg2_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %offset, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1, v0.t
@@ -1232,7 +1160,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>
 define void @test_vssseg3_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
@@ -1246,7 +1173,6 @@ entry:
 define void @test_vssseg3_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %offset, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
@@ -1263,7 +1189,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>
 define void @test_vssseg4_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1278,7 +1203,6 @@ entry:
 define void @test_vssseg4_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %offset, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1296,7 +1220,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>
 define void @test_vssseg5_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1312,7 +1235,6 @@ entry:
 define void @test_vssseg5_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %offset, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1331,7 +1253,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>
 define void @test_vssseg6_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1348,7 +1269,6 @@ entry:
 define void @test_vssseg6_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %offset, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1368,7 +1288,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>
 define void @test_vssseg7_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1386,7 +1305,6 @@ entry:
 define void @test_vssseg7_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %offset, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1407,7 +1325,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>
 define void @test_vssseg8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1426,7 +1343,6 @@ entry:
 define void @test_vssseg8_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i32 %offset, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1448,7 +1364,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv8i32(<vscale x 8 x i32>,<vscale x 8 x i
 define void @test_vssseg2_nxv8i32(<vscale x 8 x i32> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1
@@ -1461,7 +1376,6 @@ entry:
 define void @test_vssseg2_mask_nxv8i32(<vscale x 8 x i32> %val, ptr %base, i32 %offset, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1, v0.t
@@ -1477,7 +1391,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>
 define void @test_vssseg2_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1
@@ -1490,7 +1403,6 @@ entry:
 define void @test_vssseg2_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1, v0.t
@@ -1506,7 +1418,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>
 define void @test_vssseg3_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
@@ -1520,7 +1431,6 @@ entry:
 define void @test_vssseg3_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
@@ -1537,7 +1447,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>
 define void @test_vssseg4_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1552,7 +1461,6 @@ entry:
 define void @test_vssseg4_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1570,7 +1478,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>
 define void @test_vssseg5_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1586,7 +1493,6 @@ entry:
 define void @test_vssseg5_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1605,7 +1511,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>
 define void @test_vssseg6_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1622,7 +1527,6 @@ entry:
 define void @test_vssseg6_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1642,7 +1546,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>
 define void @test_vssseg7_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1660,7 +1563,6 @@ entry:
 define void @test_vssseg7_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1681,7 +1583,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>
 define void @test_vssseg8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1700,7 +1601,6 @@ entry:
 define void @test_vssseg8_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1722,7 +1622,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i
 define void @test_vssseg2_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -1735,7 +1634,6 @@ entry:
 define void @test_vssseg2_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -1751,7 +1649,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i
 define void @test_vssseg3_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
@@ -1765,7 +1662,6 @@ entry:
 define void @test_vssseg3_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
@@ -1782,7 +1678,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i
 define void @test_vssseg4_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1797,7 +1692,6 @@ entry:
 define void @test_vssseg4_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1815,7 +1709,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i
 define void @test_vssseg5_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1831,7 +1724,6 @@ entry:
 define void @test_vssseg5_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1850,7 +1742,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i
 define void @test_vssseg6_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1867,7 +1758,6 @@ entry:
 define void @test_vssseg6_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1887,7 +1777,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i
 define void @test_vssseg7_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1905,7 +1794,6 @@ entry:
 define void @test_vssseg7_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1926,7 +1814,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i
 define void @test_vssseg8_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1945,7 +1832,6 @@ entry:
 define void @test_vssseg8_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1967,7 +1853,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv32i8(<vscale x 32 x i8>,<vscale x 32 x
 define void @test_vssseg2_nxv32i8(<vscale x 32 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv32i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1
@@ -1980,7 +1865,6 @@ entry:
 define void @test_vssseg2_mask_nxv32i8(<vscale x 32 x i8> %val, ptr %base, i32 %offset, <vscale x 32 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv32i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1, v0.t
@@ -1996,7 +1880,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>
 define void @test_vssseg2_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1
@@ -2009,7 +1892,6 @@ entry:
 define void @test_vssseg2_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1, v0.t
@@ -2025,7 +1907,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>
 define void @test_vssseg3_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
@@ -2039,7 +1920,6 @@ entry:
 define void @test_vssseg3_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
@@ -2056,7 +1936,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>
 define void @test_vssseg4_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2071,7 +1950,6 @@ entry:
 define void @test_vssseg4_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2089,7 +1967,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>
 define void @test_vssseg5_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2105,7 +1982,6 @@ entry:
 define void @test_vssseg5_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2124,7 +2000,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>
 define void @test_vssseg6_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2141,7 +2016,6 @@ entry:
 define void @test_vssseg6_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2161,7 +2035,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>
 define void @test_vssseg7_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2179,7 +2052,6 @@ entry:
 define void @test_vssseg7_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2200,7 +2072,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>
 define void @test_vssseg8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2219,7 +2090,6 @@ entry:
 define void @test_vssseg8_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2241,7 +2111,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i
 define void @test_vssseg2_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -2254,7 +2123,6 @@ entry:
 define void @test_vssseg2_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -2270,7 +2138,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i
 define void @test_vssseg3_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
@@ -2284,7 +2151,6 @@ entry:
 define void @test_vssseg3_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
@@ -2301,7 +2167,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i
 define void @test_vssseg4_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2316,7 +2181,6 @@ entry:
 define void @test_vssseg4_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2334,7 +2198,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i
 define void @test_vssseg5_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2350,7 +2213,6 @@ entry:
 define void @test_vssseg5_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2369,7 +2231,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i
 define void @test_vssseg6_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2386,7 +2247,6 @@ entry:
 define void @test_vssseg6_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2406,7 +2266,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i
 define void @test_vssseg7_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2424,7 +2283,6 @@ entry:
 define void @test_vssseg7_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2445,7 +2303,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i
 define void @test_vssseg8_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2464,7 +2321,6 @@ entry:
 define void @test_vssseg8_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2486,7 +2342,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv4i32(<vscale x 4 x i32>,<vscale x 4 x i
 define void @test_vssseg2_nxv4i32(<vscale x 4 x i32> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1
@@ -2499,7 +2354,6 @@ entry:
 define void @test_vssseg2_mask_nxv4i32(<vscale x 4 x i32> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1, v0.t
@@ -2515,7 +2369,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv4i32(<vscale x 4 x i32>,<vscale x 4 x i
 define void @test_vssseg3_nxv4i32(<vscale x 4 x i32> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
@@ -2529,7 +2382,6 @@ entry:
 define void @test_vssseg3_mask_nxv4i32(<vscale x 4 x i32> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
@@ -2546,7 +2398,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv4i32(<vscale x 4 x i32>,<vscale x 4 x i
 define void @test_vssseg4_nxv4i32(<vscale x 4 x i32> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -2561,7 +2412,6 @@ entry:
 define void @test_vssseg4_mask_nxv4i32(<vscale x 4 x i32> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -2579,7 +2429,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv16f16(<vscale x 16 x half>,<vscale x 16
 define void @test_vssseg2_nxv16f16(<vscale x 16 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv16f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -2592,7 +2441,6 @@ entry:
 define void @test_vssseg2_mask_nxv16f16(<vscale x 16 x half> %val, ptr %base, i32 %offset, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv16f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -2608,7 +2456,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv4f64(<vscale x 4 x double>,<vscale x 4
 define void @test_vssseg2_nxv4f64(<vscale x 4 x double> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv4f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e64.v v8, (a0), a1
@@ -2621,7 +2468,6 @@ entry:
 define void @test_vssseg2_mask_nxv4f64(<vscale x 4 x double> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv4f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e64.v v8, (a0), a1, v0.t
@@ -2637,7 +2483,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1
 define void @test_vssseg2_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e64.v v8, (a0), a1
@@ -2650,7 +2495,6 @@ entry:
 define void @test_vssseg2_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e64.v v8, (a0), a1, v0.t
@@ -2666,7 +2510,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1
 define void @test_vssseg3_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
@@ -2680,7 +2523,6 @@ entry:
 define void @test_vssseg3_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
@@ -2697,7 +2539,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1
 define void @test_vssseg4_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2712,7 +2553,6 @@ entry:
 define void @test_vssseg4_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2730,7 +2570,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1
 define void @test_vssseg5_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2746,7 +2585,6 @@ entry:
 define void @test_vssseg5_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2765,7 +2603,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1
 define void @test_vssseg6_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2782,7 +2619,6 @@ entry:
 define void @test_vssseg6_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2802,7 +2638,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1
 define void @test_vssseg7_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2820,7 +2655,6 @@ entry:
 define void @test_vssseg7_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2841,7 +2675,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1
 define void @test_vssseg8_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2860,7 +2693,6 @@ entry:
 define void @test_vssseg8_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2882,7 +2714,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vssseg2_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1
@@ -2895,7 +2726,6 @@ entry:
 define void @test_vssseg2_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1, v0.t
@@ -2911,7 +2741,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vssseg3_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
@@ -2925,7 +2754,6 @@ entry:
 define void @test_vssseg3_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
@@ -2942,7 +2770,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vssseg4_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2957,7 +2784,6 @@ entry:
 define void @test_vssseg4_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2975,7 +2801,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vssseg5_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2991,7 +2816,6 @@ entry:
 define void @test_vssseg5_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3010,7 +2834,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vssseg6_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3027,7 +2850,6 @@ entry:
 define void @test_vssseg6_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3047,7 +2869,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vssseg7_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3065,7 +2886,6 @@ entry:
 define void @test_vssseg7_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3086,7 +2906,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vssseg8_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3105,7 +2924,6 @@ entry:
 define void @test_vssseg8_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3127,7 +2945,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x
 define void @test_vssseg2_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -3140,7 +2957,6 @@ entry:
 define void @test_vssseg2_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -3156,7 +2972,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x
 define void @test_vssseg3_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
@@ -3170,7 +2985,6 @@ entry:
 define void @test_vssseg3_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
@@ -3187,7 +3001,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x
 define void @test_vssseg4_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3202,7 +3015,6 @@ entry:
 define void @test_vssseg4_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3220,7 +3032,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x
 define void @test_vssseg5_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3236,7 +3047,6 @@ entry:
 define void @test_vssseg5_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3255,7 +3065,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x
 define void @test_vssseg6_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3272,7 +3081,6 @@ entry:
 define void @test_vssseg6_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3292,7 +3100,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x
 define void @test_vssseg7_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3310,7 +3117,6 @@ entry:
 define void @test_vssseg7_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3331,7 +3137,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x
 define void @test_vssseg8_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3350,7 +3155,6 @@ entry:
 define void @test_vssseg8_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3372,7 +3176,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vssseg2_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1
@@ -3385,7 +3188,6 @@ entry:
 define void @test_vssseg2_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1, v0.t
@@ -3401,7 +3203,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vssseg3_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
@@ -3415,7 +3216,6 @@ entry:
 define void @test_vssseg3_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
@@ -3432,7 +3232,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vssseg4_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3447,7 +3246,6 @@ entry:
 define void @test_vssseg4_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3465,7 +3263,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vssseg5_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3481,7 +3278,6 @@ entry:
 define void @test_vssseg5_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3500,7 +3296,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vssseg6_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3517,7 +3312,6 @@ entry:
 define void @test_vssseg6_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3537,7 +3331,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vssseg7_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3555,7 +3348,6 @@ entry:
 define void @test_vssseg7_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3576,7 +3368,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vssseg8_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3595,7 +3386,6 @@ entry:
 define void @test_vssseg8_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, i32 %offset, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3617,7 +3407,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv8f16(<vscale x 8 x half>,<vscale x 8 x
 define void @test_vssseg2_nxv8f16(<vscale x 8 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -3630,7 +3419,6 @@ entry:
 define void @test_vssseg2_mask_nxv8f16(<vscale x 8 x half> %val, ptr %base, i32 %offset, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -3646,7 +3434,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv8f16(<vscale x 8 x half>,<vscale x 8 x
 define void @test_vssseg3_nxv8f16(<vscale x 8 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
@@ -3660,7 +3447,6 @@ entry:
 define void @test_vssseg3_mask_nxv8f16(<vscale x 8 x half> %val, ptr %base, i32 %offset, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
@@ -3677,7 +3463,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv8f16(<vscale x 8 x half>,<vscale x 8 x
 define void @test_vssseg4_nxv8f16(<vscale x 8 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -3692,7 +3477,6 @@ entry:
 define void @test_vssseg4_mask_nxv8f16(<vscale x 8 x half> %val, ptr %base, i32 %offset, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -3710,7 +3494,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv8f32(<vscale x 8 x float>,<vscale x 8 x
 define void @test_vssseg2_nxv8f32(<vscale x 8 x float> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv8f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1
@@ -3723,7 +3506,6 @@ entry:
 define void @test_vssseg2_mask_nxv8f32(<vscale x 8 x float> %val, ptr %base, i32 %offset, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv8f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1, v0.t
@@ -3739,7 +3521,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv2f64(<vscale x 2 x double>,<vscale x 2
 define void @test_vssseg2_nxv2f64(<vscale x 2 x double> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e64.v v8, (a0), a1
@@ -3752,7 +3533,6 @@ entry:
 define void @test_vssseg2_mask_nxv2f64(<vscale x 2 x double> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e64.v v8, (a0), a1, v0.t
@@ -3768,7 +3548,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv2f64(<vscale x 2 x double>,<vscale x 2
 define void @test_vssseg3_nxv2f64(<vscale x 2 x double> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
@@ -3782,7 +3561,6 @@ entry:
 define void @test_vssseg3_mask_nxv2f64(<vscale x 2 x double> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
@@ -3799,7 +3577,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv2f64(<vscale x 2 x double>,<vscale x 2
 define void @test_vssseg4_nxv2f64(<vscale x 2 x double> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -3814,7 +3591,6 @@ entry:
 define void @test_vssseg4_mask_nxv2f64(<vscale x 2 x double> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -3832,7 +3608,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x
 define void @test_vssseg2_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -3845,7 +3620,6 @@ entry:
 define void @test_vssseg2_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -3861,7 +3635,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x
 define void @test_vssseg3_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
@@ -3875,7 +3648,6 @@ entry:
 define void @test_vssseg3_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
@@ -3892,7 +3664,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x
 define void @test_vssseg4_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3907,7 +3678,6 @@ entry:
 define void @test_vssseg4_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3925,7 +3695,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x
 define void @test_vssseg5_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3941,7 +3710,6 @@ entry:
 define void @test_vssseg5_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3960,7 +3728,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x
 define void @test_vssseg6_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3977,7 +3744,6 @@ entry:
 define void @test_vssseg6_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3997,7 +3763,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x
 define void @test_vssseg7_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4015,7 +3780,6 @@ entry:
 define void @test_vssseg7_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4036,7 +3800,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x
 define void @test_vssseg8_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4055,7 +3818,6 @@ entry:
 define void @test_vssseg8_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4077,7 +3839,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x
 define void @test_vssseg2_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -4090,7 +3851,6 @@ entry:
 define void @test_vssseg2_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -4106,7 +3866,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x
 define void @test_vssseg3_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
@@ -4120,7 +3879,6 @@ entry:
 define void @test_vssseg3_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
@@ -4137,7 +3895,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x
 define void @test_vssseg4_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4152,7 +3909,6 @@ entry:
 define void @test_vssseg4_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4170,7 +3926,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x
 define void @test_vssseg5_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4186,7 +3941,6 @@ entry:
 define void @test_vssseg5_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4205,7 +3959,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x
 define void @test_vssseg6_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4222,7 +3975,6 @@ entry:
 define void @test_vssseg6_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4242,7 +3994,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x
 define void @test_vssseg7_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4260,7 +4011,6 @@ entry:
 define void @test_vssseg7_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4281,7 +4031,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x
 define void @test_vssseg8_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4300,7 +4049,6 @@ entry:
 define void @test_vssseg8_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, i32 %offset, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4322,7 +4070,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv4f32(<vscale x 4 x float>,<vscale x 4 x
 define void @test_vssseg2_nxv4f32(<vscale x 4 x float> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1
@@ -4335,7 +4082,6 @@ entry:
 define void @test_vssseg2_mask_nxv4f32(<vscale x 4 x float> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1, v0.t
@@ -4351,7 +4097,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv4f32(<vscale x 4 x float>,<vscale x 4 x
 define void @test_vssseg3_nxv4f32(<vscale x 4 x float> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
@@ -4365,7 +4110,6 @@ entry:
 define void @test_vssseg3_mask_nxv4f32(<vscale x 4 x float> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
@@ -4382,7 +4126,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv4f32(<vscale x 4 x float>,<vscale x 4 x
 define void @test_vssseg4_nxv4f32(<vscale x 4 x float> %val, ptr %base, i32 %offset, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -4397,7 +4140,6 @@ entry:
 define void @test_vssseg4_mask_nxv4f32(<vscale x 4 x float> %val, ptr %base, i32 %offset, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
diff --git a/llvm/test/CodeGen/RISCV/rvv/vssseg-rv64.ll b/llvm/test/CodeGen/RISCV/rvv/vssseg-rv64.ll
index 9778447cd848f..6e067e6358239 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vssseg-rv64.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vssseg-rv64.ll
@@ -8,7 +8,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv16i16(<vscale x 16 x i16>,<vscale x 16
 define void @test_vssseg2_nxv16i16(<vscale x 16 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -21,7 +20,6 @@ entry:
 define void @test_vssseg2_mask_nxv16i16(<vscale x 16 x i16> %val, ptr %base, i64 %offset, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -37,7 +35,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv4i32(<vscale x 4 x i32>,<vscale x 4 x i
 define void @test_vssseg2_nxv4i32(<vscale x 4 x i32> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1
@@ -50,7 +47,6 @@ entry:
 define void @test_vssseg2_mask_nxv4i32(<vscale x 4 x i32> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1, v0.t
@@ -66,7 +62,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv4i32(<vscale x 4 x i32>,<vscale x 4 x i
 define void @test_vssseg3_nxv4i32(<vscale x 4 x i32> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
@@ -80,7 +75,6 @@ entry:
 define void @test_vssseg3_mask_nxv4i32(<vscale x 4 x i32> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
@@ -97,7 +91,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv4i32(<vscale x 4 x i32>,<vscale x 4 x i
 define void @test_vssseg4_nxv4i32(<vscale x 4 x i32> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -112,7 +105,6 @@ entry:
 define void @test_vssseg4_mask_nxv4i32(<vscale x 4 x i32> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -130,7 +122,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv16i8(<vscale x 16 x i8>,<vscale x 16 x
 define void @test_vssseg2_nxv16i8(<vscale x 16 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1
@@ -143,7 +134,6 @@ entry:
 define void @test_vssseg2_mask_nxv16i8(<vscale x 16 x i8> %val, ptr %base, i64 %offset, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1, v0.t
@@ -159,7 +149,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv16i8(<vscale x 16 x i8>,<vscale x 16 x
 define void @test_vssseg3_nxv16i8(<vscale x 16 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
@@ -173,7 +162,6 @@ entry:
 define void @test_vssseg3_mask_nxv16i8(<vscale x 16 x i8> %val, ptr %base, i64 %offset, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
@@ -190,7 +178,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv16i8(<vscale x 16 x i8>,<vscale x 16 x
 define void @test_vssseg4_nxv16i8(<vscale x 16 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -205,7 +192,6 @@ entry:
 define void @test_vssseg4_mask_nxv16i8(<vscale x 16 x i8> %val, ptr %base, i64 %offset, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -223,7 +209,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv1i64(<vscale x 1 x i64>,<vscale x 1 x i
 define void @test_vssseg2_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e64.v v8, (a0), a1
@@ -236,7 +221,6 @@ entry:
 define void @test_vssseg2_mask_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e64.v v8, (a0), a1, v0.t
@@ -252,7 +236,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv1i64(<vscale x 1 x i64>,<vscale x 1 x i
 define void @test_vssseg3_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
@@ -266,7 +249,6 @@ entry:
 define void @test_vssseg3_mask_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
@@ -283,7 +265,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv1i64(<vscale x 1 x i64>,<vscale x 1 x i
 define void @test_vssseg4_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -298,7 +279,6 @@ entry:
 define void @test_vssseg4_mask_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -316,7 +296,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv1i64(<vscale x 1 x i64>,<vscale x 1 x i
 define void @test_vssseg5_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -332,7 +311,6 @@ entry:
 define void @test_vssseg5_mask_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -351,7 +329,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv1i64(<vscale x 1 x i64>,<vscale x 1 x i
 define void @test_vssseg6_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -368,7 +345,6 @@ entry:
 define void @test_vssseg6_mask_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -388,7 +364,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv1i64(<vscale x 1 x i64>,<vscale x 1 x i
 define void @test_vssseg7_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -406,7 +381,6 @@ entry:
 define void @test_vssseg7_mask_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -427,7 +401,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv1i64(<vscale x 1 x i64>,<vscale x 1 x i
 define void @test_vssseg8_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -446,7 +419,6 @@ entry:
 define void @test_vssseg8_mask_nxv1i64(<vscale x 1 x i64> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -468,7 +440,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i
 define void @test_vssseg2_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1
@@ -481,7 +452,6 @@ entry:
 define void @test_vssseg2_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1, v0.t
@@ -497,7 +467,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i
 define void @test_vssseg3_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
@@ -511,7 +480,6 @@ entry:
 define void @test_vssseg3_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
@@ -528,7 +496,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i
 define void @test_vssseg4_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -543,7 +510,6 @@ entry:
 define void @test_vssseg4_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -561,7 +527,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i
 define void @test_vssseg5_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -577,7 +542,6 @@ entry:
 define void @test_vssseg5_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -596,7 +560,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i
 define void @test_vssseg6_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -613,7 +576,6 @@ entry:
 define void @test_vssseg6_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -633,7 +595,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i
 define void @test_vssseg7_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -651,7 +612,6 @@ entry:
 define void @test_vssseg7_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -672,7 +632,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv1i32(<vscale x 1 x i32>,<vscale x 1 x i
 define void @test_vssseg8_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -691,7 +650,6 @@ entry:
 define void @test_vssseg8_mask_nxv1i32(<vscale x 1 x i32> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -713,7 +671,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv8i16(<vscale x 8 x i16>,<vscale x 8 x i
 define void @test_vssseg2_nxv8i16(<vscale x 8 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -726,7 +683,6 @@ entry:
 define void @test_vssseg2_mask_nxv8i16(<vscale x 8 x i16> %val, ptr %base, i64 %offset, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -742,7 +698,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv8i16(<vscale x 8 x i16>,<vscale x 8 x i
 define void @test_vssseg3_nxv8i16(<vscale x 8 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
@@ -756,7 +711,6 @@ entry:
 define void @test_vssseg3_mask_nxv8i16(<vscale x 8 x i16> %val, ptr %base, i64 %offset, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
@@ -773,7 +727,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv8i16(<vscale x 8 x i16>,<vscale x 8 x i
 define void @test_vssseg4_nxv8i16(<vscale x 8 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -788,7 +741,6 @@ entry:
 define void @test_vssseg4_mask_nxv8i16(<vscale x 8 x i16> %val, ptr %base, i64 %offset, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -806,7 +758,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>
 define void @test_vssseg2_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1
@@ -819,7 +770,6 @@ entry:
 define void @test_vssseg2_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1, v0.t
@@ -835,7 +785,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>
 define void @test_vssseg3_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
@@ -849,7 +798,6 @@ entry:
 define void @test_vssseg3_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf2, ta, ma
@@ -866,7 +814,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>
 define void @test_vssseg4_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -881,7 +828,6 @@ entry:
 define void @test_vssseg4_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -899,7 +845,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>
 define void @test_vssseg5_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -915,7 +860,6 @@ entry:
 define void @test_vssseg5_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -934,7 +878,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>
 define void @test_vssseg6_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -951,7 +894,6 @@ entry:
 define void @test_vssseg6_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -971,7 +913,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>
 define void @test_vssseg7_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -989,7 +930,6 @@ entry:
 define void @test_vssseg7_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1010,7 +950,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv4i8(<vscale x 4 x i8>,<vscale x 4 x i8>
 define void @test_vssseg8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1029,7 +968,6 @@ entry:
 define void @test_vssseg8_mask_nxv4i8(<vscale x 4 x i8> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1051,7 +989,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i
 define void @test_vssseg2_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -1064,7 +1001,6 @@ entry:
 define void @test_vssseg2_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -1080,7 +1016,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i
 define void @test_vssseg3_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
@@ -1094,7 +1029,6 @@ entry:
 define void @test_vssseg3_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
@@ -1111,7 +1045,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i
 define void @test_vssseg4_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1126,7 +1059,6 @@ entry:
 define void @test_vssseg4_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1144,7 +1076,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i
 define void @test_vssseg5_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1160,7 +1091,6 @@ entry:
 define void @test_vssseg5_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1179,7 +1109,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i
 define void @test_vssseg6_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1196,7 +1125,6 @@ entry:
 define void @test_vssseg6_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1216,7 +1144,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i
 define void @test_vssseg7_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1234,7 +1161,6 @@ entry:
 define void @test_vssseg7_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1255,7 +1181,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv1i16(<vscale x 1 x i16>,<vscale x 1 x i
 define void @test_vssseg8_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1274,7 +1199,6 @@ entry:
 define void @test_vssseg8_mask_nxv1i16(<vscale x 1 x i16> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1296,7 +1220,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i
 define void @test_vssseg2_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1
@@ -1309,7 +1232,6 @@ entry:
 define void @test_vssseg2_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1, v0.t
@@ -1325,7 +1247,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i
 define void @test_vssseg3_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
@@ -1339,7 +1260,6 @@ entry:
 define void @test_vssseg3_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
@@ -1356,7 +1276,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i
 define void @test_vssseg4_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1371,7 +1290,6 @@ entry:
 define void @test_vssseg4_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1389,7 +1307,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i
 define void @test_vssseg5_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1405,7 +1322,6 @@ entry:
 define void @test_vssseg5_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1424,7 +1340,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i
 define void @test_vssseg6_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1441,7 +1356,6 @@ entry:
 define void @test_vssseg6_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1461,7 +1375,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i
 define void @test_vssseg7_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1479,7 +1392,6 @@ entry:
 define void @test_vssseg7_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1500,7 +1412,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv2i32(<vscale x 2 x i32>,<vscale x 2 x i
 define void @test_vssseg8_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1519,7 +1430,6 @@ entry:
 define void @test_vssseg8_mask_nxv2i32(<vscale x 2 x i32> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1541,7 +1451,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>
 define void @test_vssseg2_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1
@@ -1554,7 +1463,6 @@ entry:
 define void @test_vssseg2_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %offset, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1, v0.t
@@ -1570,7 +1478,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>
 define void @test_vssseg3_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
@@ -1584,7 +1491,6 @@ entry:
 define void @test_vssseg3_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %offset, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m1, ta, ma
@@ -1601,7 +1507,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>
 define void @test_vssseg4_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1616,7 +1521,6 @@ entry:
 define void @test_vssseg4_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %offset, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1634,7 +1538,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>
 define void @test_vssseg5_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1650,7 +1553,6 @@ entry:
 define void @test_vssseg5_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %offset, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1669,7 +1571,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>
 define void @test_vssseg6_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1686,7 +1587,6 @@ entry:
 define void @test_vssseg6_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %offset, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1706,7 +1606,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>
 define void @test_vssseg7_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1724,7 +1623,6 @@ entry:
 define void @test_vssseg7_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %offset, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1745,7 +1643,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv8i8(<vscale x 8 x i8>,<vscale x 8 x i8>
 define void @test_vssseg8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1764,7 +1661,6 @@ entry:
 define void @test_vssseg8_mask_nxv8i8(<vscale x 8 x i8> %val, ptr %base, i64 %offset, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1786,7 +1682,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv4i64(<vscale x 4 x i64>,<vscale x 4 x i
 define void @test_vssseg2_nxv4i64(<vscale x 4 x i64> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e64.v v8, (a0), a1
@@ -1799,7 +1694,6 @@ entry:
 define void @test_vssseg2_mask_nxv4i64(<vscale x 4 x i64> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e64.v v8, (a0), a1, v0.t
@@ -1815,7 +1709,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i
 define void @test_vssseg2_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -1828,7 +1721,6 @@ entry:
 define void @test_vssseg2_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -1844,7 +1736,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i
 define void @test_vssseg3_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
@@ -1858,7 +1749,6 @@ entry:
 define void @test_vssseg3_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
@@ -1875,7 +1765,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i
 define void @test_vssseg4_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1890,7 +1779,6 @@ entry:
 define void @test_vssseg4_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1908,7 +1796,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i
 define void @test_vssseg5_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1924,7 +1811,6 @@ entry:
 define void @test_vssseg5_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1943,7 +1829,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i
 define void @test_vssseg6_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1960,7 +1845,6 @@ entry:
 define void @test_vssseg6_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1980,7 +1864,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i
 define void @test_vssseg7_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -1998,7 +1881,6 @@ entry:
 define void @test_vssseg7_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2019,7 +1901,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv4i16(<vscale x 4 x i16>,<vscale x 4 x i
 define void @test_vssseg8_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2038,7 +1919,6 @@ entry:
 define void @test_vssseg8_mask_nxv4i16(<vscale x 4 x i16> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2060,7 +1940,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>
 define void @test_vssseg2_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1
@@ -2073,7 +1952,6 @@ entry:
 define void @test_vssseg2_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1, v0.t
@@ -2089,7 +1967,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>
 define void @test_vssseg3_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
@@ -2103,7 +1980,6 @@ entry:
 define void @test_vssseg3_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf8, ta, ma
@@ -2120,7 +1996,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>
 define void @test_vssseg4_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2135,7 +2010,6 @@ entry:
 define void @test_vssseg4_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2153,7 +2027,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>
 define void @test_vssseg5_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2169,7 +2042,6 @@ entry:
 define void @test_vssseg5_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2188,7 +2060,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>
 define void @test_vssseg6_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2205,7 +2076,6 @@ entry:
 define void @test_vssseg6_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2225,7 +2095,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>
 define void @test_vssseg7_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2243,7 +2112,6 @@ entry:
 define void @test_vssseg7_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2264,7 +2132,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv1i8(<vscale x 1 x i8>,<vscale x 1 x i8>
 define void @test_vssseg8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2283,7 +2150,6 @@ entry:
 define void @test_vssseg8_mask_nxv1i8(<vscale x 1 x i8> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2305,7 +2171,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>
 define void @test_vssseg2_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1
@@ -2318,7 +2183,6 @@ entry:
 define void @test_vssseg2_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1, v0.t
@@ -2334,7 +2198,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>
 define void @test_vssseg3_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
@@ -2348,7 +2211,6 @@ entry:
 define void @test_vssseg3_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, mf4, ta, ma
@@ -2365,7 +2227,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>
 define void @test_vssseg4_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2380,7 +2241,6 @@ entry:
 define void @test_vssseg4_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2398,7 +2258,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>
 define void @test_vssseg5_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2414,7 +2273,6 @@ entry:
 define void @test_vssseg5_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2433,7 +2291,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>
 define void @test_vssseg6_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2450,7 +2307,6 @@ entry:
 define void @test_vssseg6_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2470,7 +2326,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>
 define void @test_vssseg7_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2488,7 +2343,6 @@ entry:
 define void @test_vssseg7_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2509,7 +2363,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv2i8(<vscale x 2 x i8>,<vscale x 2 x i8>
 define void @test_vssseg8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2528,7 +2381,6 @@ entry:
 define void @test_vssseg8_mask_nxv2i8(<vscale x 2 x i8> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2550,7 +2402,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv8i32(<vscale x 8 x i32>,<vscale x 8 x i
 define void @test_vssseg2_nxv8i32(<vscale x 8 x i32> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1
@@ -2563,7 +2414,6 @@ entry:
 define void @test_vssseg2_mask_nxv8i32(<vscale x 8 x i32> %val, ptr %base, i64 %offset, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1, v0.t
@@ -2579,7 +2429,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv32i8(<vscale x 32 x i8>,<vscale x 32 x
 define void @test_vssseg2_nxv32i8(<vscale x 32 x i8> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv32i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1
@@ -2592,7 +2441,6 @@ entry:
 define void @test_vssseg2_mask_nxv32i8(<vscale x 32 x i8> %val, ptr %base, i64 %offset, <vscale x 32 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv32i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e8.v v8, (a0), a1, v0.t
@@ -2608,7 +2456,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i
 define void @test_vssseg2_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -2621,7 +2468,6 @@ entry:
 define void @test_vssseg2_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -2637,7 +2483,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i
 define void @test_vssseg3_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
@@ -2651,7 +2496,6 @@ entry:
 define void @test_vssseg3_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
@@ -2668,7 +2512,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i
 define void @test_vssseg4_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2683,7 +2526,6 @@ entry:
 define void @test_vssseg4_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2701,7 +2543,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i
 define void @test_vssseg5_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2717,7 +2558,6 @@ entry:
 define void @test_vssseg5_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2736,7 +2576,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i
 define void @test_vssseg6_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2753,7 +2592,6 @@ entry:
 define void @test_vssseg6_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2773,7 +2611,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i
 define void @test_vssseg7_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2791,7 +2628,6 @@ entry:
 define void @test_vssseg7_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2812,7 +2648,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv2i16(<vscale x 2 x i16>,<vscale x 2 x i
 define void @test_vssseg8_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2831,7 +2666,6 @@ entry:
 define void @test_vssseg8_mask_nxv2i16(<vscale x 2 x i16> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -2853,7 +2687,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv2i64(<vscale x 2 x i64>,<vscale x 2 x i
 define void @test_vssseg2_nxv2i64(<vscale x 2 x i64> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e64.v v8, (a0), a1
@@ -2866,7 +2699,6 @@ entry:
 define void @test_vssseg2_mask_nxv2i64(<vscale x 2 x i64> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e64.v v8, (a0), a1, v0.t
@@ -2882,7 +2714,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv2i64(<vscale x 2 x i64>,<vscale x 2 x i
 define void @test_vssseg3_nxv2i64(<vscale x 2 x i64> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
@@ -2896,7 +2727,6 @@ entry:
 define void @test_vssseg3_mask_nxv2i64(<vscale x 2 x i64> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
@@ -2913,7 +2743,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv2i64(<vscale x 2 x i64>,<vscale x 2 x i
 define void @test_vssseg4_nxv2i64(<vscale x 2 x i64> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -2928,7 +2757,6 @@ entry:
 define void @test_vssseg4_mask_nxv2i64(<vscale x 2 x i64> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -2946,7 +2774,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv16f16(<vscale x 16 x half>,<vscale x 16
 define void @test_vssseg2_nxv16f16(<vscale x 16 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv16f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -2959,7 +2786,6 @@ entry:
 define void @test_vssseg2_mask_nxv16f16(<vscale x 16 x half> %val, ptr %base, i64 %offset, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv16f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -2975,7 +2801,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv4f64(<vscale x 4 x double>,<vscale x 4
 define void @test_vssseg2_nxv4f64(<vscale x 4 x double> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv4f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e64.v v8, (a0), a1
@@ -2988,7 +2813,6 @@ entry:
 define void @test_vssseg2_mask_nxv4f64(<vscale x 4 x double> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv4f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e64.v v8, (a0), a1, v0.t
@@ -3004,7 +2828,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1
 define void @test_vssseg2_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e64.v v8, (a0), a1
@@ -3017,7 +2840,6 @@ entry:
 define void @test_vssseg2_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e64.v v8, (a0), a1, v0.t
@@ -3033,7 +2855,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1
 define void @test_vssseg3_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
@@ -3047,7 +2868,6 @@ entry:
 define void @test_vssseg3_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m1, ta, ma
@@ -3064,7 +2884,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1
 define void @test_vssseg4_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3079,7 +2898,6 @@ entry:
 define void @test_vssseg4_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3097,7 +2915,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1
 define void @test_vssseg5_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3113,7 +2930,6 @@ entry:
 define void @test_vssseg5_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3132,7 +2948,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1
 define void @test_vssseg6_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3149,7 +2964,6 @@ entry:
 define void @test_vssseg6_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3169,7 +2983,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1
 define void @test_vssseg7_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3187,7 +3000,6 @@ entry:
 define void @test_vssseg7_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3208,7 +3020,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv1f64(<vscale x 1 x double>,<vscale x 1
 define void @test_vssseg8_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3227,7 +3038,6 @@ entry:
 define void @test_vssseg8_mask_nxv1f64(<vscale x 1 x double> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv1f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3249,7 +3059,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vssseg2_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1
@@ -3262,7 +3071,6 @@ entry:
 define void @test_vssseg2_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1, v0.t
@@ -3278,7 +3086,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vssseg3_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
@@ -3292,7 +3099,6 @@ entry:
 define void @test_vssseg3_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m1, ta, ma
@@ -3309,7 +3115,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vssseg4_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3324,7 +3129,6 @@ entry:
 define void @test_vssseg4_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3342,7 +3146,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vssseg5_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3358,7 +3161,6 @@ entry:
 define void @test_vssseg5_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3377,7 +3179,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vssseg6_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3394,7 +3195,6 @@ entry:
 define void @test_vssseg6_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3414,7 +3214,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vssseg7_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3432,7 +3231,6 @@ entry:
 define void @test_vssseg7_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3453,7 +3251,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv2f32(<vscale x 2 x float>,<vscale x 2 x
 define void @test_vssseg8_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3472,7 +3269,6 @@ entry:
 define void @test_vssseg8_mask_nxv2f32(<vscale x 2 x float> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv2f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3494,7 +3290,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x
 define void @test_vssseg2_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -3507,7 +3302,6 @@ entry:
 define void @test_vssseg2_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -3523,7 +3317,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x
 define void @test_vssseg3_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
@@ -3537,7 +3330,6 @@ entry:
 define void @test_vssseg3_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf4, ta, ma
@@ -3554,7 +3346,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x
 define void @test_vssseg4_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3569,7 +3360,6 @@ entry:
 define void @test_vssseg4_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3587,7 +3377,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x
 define void @test_vssseg5_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3603,7 +3392,6 @@ entry:
 define void @test_vssseg5_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3622,7 +3410,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x
 define void @test_vssseg6_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3639,7 +3426,6 @@ entry:
 define void @test_vssseg6_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3659,7 +3445,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x
 define void @test_vssseg7_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3677,7 +3462,6 @@ entry:
 define void @test_vssseg7_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3698,7 +3482,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv1f16(<vscale x 1 x half>,<vscale x 1 x
 define void @test_vssseg8_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3717,7 +3500,6 @@ entry:
 define void @test_vssseg8_mask_nxv1f16(<vscale x 1 x half> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv1f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3739,7 +3521,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vssseg2_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1
@@ -3752,7 +3533,6 @@ entry:
 define void @test_vssseg2_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1, v0.t
@@ -3768,7 +3548,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vssseg3_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
@@ -3782,7 +3561,6 @@ entry:
 define void @test_vssseg3_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, mf2, ta, ma
@@ -3799,7 +3577,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vssseg4_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3814,7 +3591,6 @@ entry:
 define void @test_vssseg4_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3832,7 +3608,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vssseg5_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3848,7 +3623,6 @@ entry:
 define void @test_vssseg5_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3867,7 +3641,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vssseg6_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3884,7 +3657,6 @@ entry:
 define void @test_vssseg6_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3904,7 +3676,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vssseg7_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3922,7 +3693,6 @@ entry:
 define void @test_vssseg7_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3943,7 +3713,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv1f32(<vscale x 1 x float>,<vscale x 1 x
 define void @test_vssseg8_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3962,7 +3731,6 @@ entry:
 define void @test_vssseg8_mask_nxv1f32(<vscale x 1 x float> %val, ptr %base, i64 %offset, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv1f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3984,7 +3752,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv8f16(<vscale x 8 x half>,<vscale x 8 x
 define void @test_vssseg2_nxv8f16(<vscale x 8 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -3997,7 +3764,6 @@ entry:
 define void @test_vssseg2_mask_nxv8f16(<vscale x 8 x half> %val, ptr %base, i64 %offset, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -4013,7 +3779,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv8f16(<vscale x 8 x half>,<vscale x 8 x
 define void @test_vssseg3_nxv8f16(<vscale x 8 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
@@ -4027,7 +3792,6 @@ entry:
 define void @test_vssseg3_mask_nxv8f16(<vscale x 8 x half> %val, ptr %base, i64 %offset, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m2, ta, ma
@@ -4044,7 +3808,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv8f16(<vscale x 8 x half>,<vscale x 8 x
 define void @test_vssseg4_nxv8f16(<vscale x 8 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -4059,7 +3822,6 @@ entry:
 define void @test_vssseg4_mask_nxv8f16(<vscale x 8 x half> %val, ptr %base, i64 %offset, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv8f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -4077,7 +3839,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv8f32(<vscale x 8 x float>,<vscale x 8 x
 define void @test_vssseg2_nxv8f32(<vscale x 8 x float> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv8f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1
@@ -4090,7 +3851,6 @@ entry:
 define void @test_vssseg2_mask_nxv8f32(<vscale x 8 x float> %val, ptr %base, i64 %offset, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv8f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1, v0.t
@@ -4106,7 +3866,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv2f64(<vscale x 2 x double>,<vscale x 2
 define void @test_vssseg2_nxv2f64(<vscale x 2 x double> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e64.v v8, (a0), a1
@@ -4119,7 +3878,6 @@ entry:
 define void @test_vssseg2_mask_nxv2f64(<vscale x 2 x double> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e64.v v8, (a0), a1, v0.t
@@ -4135,7 +3893,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv2f64(<vscale x 2 x double>,<vscale x 2
 define void @test_vssseg3_nxv2f64(<vscale x 2 x double> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
@@ -4149,7 +3906,6 @@ entry:
 define void @test_vssseg3_mask_nxv2f64(<vscale x 2 x double> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m2, ta, ma
@@ -4166,7 +3922,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv2f64(<vscale x 2 x double>,<vscale x 2
 define void @test_vssseg4_nxv2f64(<vscale x 2 x double> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -4181,7 +3936,6 @@ entry:
 define void @test_vssseg4_mask_nxv2f64(<vscale x 2 x double> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv2f64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -4199,7 +3953,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x
 define void @test_vssseg2_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -4212,7 +3965,6 @@ entry:
 define void @test_vssseg2_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -4228,7 +3980,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x
 define void @test_vssseg3_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
@@ -4242,7 +3993,6 @@ entry:
 define void @test_vssseg3_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m1, ta, ma
@@ -4259,7 +4009,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x
 define void @test_vssseg4_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4274,7 +4023,6 @@ entry:
 define void @test_vssseg4_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4292,7 +4040,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x
 define void @test_vssseg5_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4308,7 +4055,6 @@ entry:
 define void @test_vssseg5_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4327,7 +4073,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x
 define void @test_vssseg6_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4344,7 +4089,6 @@ entry:
 define void @test_vssseg6_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4364,7 +4108,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x
 define void @test_vssseg7_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4382,7 +4125,6 @@ entry:
 define void @test_vssseg7_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4403,7 +4145,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv4f16(<vscale x 4 x half>,<vscale x 4 x
 define void @test_vssseg8_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4422,7 +4163,6 @@ entry:
 define void @test_vssseg8_mask_nxv4f16(<vscale x 4 x half> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv4f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4444,7 +4184,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x
 define void @test_vssseg2_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -4457,7 +4196,6 @@ entry:
 define void @test_vssseg2_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -4473,7 +4211,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x
 define void @test_vssseg3_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
@@ -4487,7 +4224,6 @@ entry:
 define void @test_vssseg3_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e16, mf2, ta, ma
@@ -4504,7 +4240,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x
 define void @test_vssseg4_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4519,7 +4254,6 @@ entry:
 define void @test_vssseg4_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4537,7 +4271,6 @@ declare void @llvm.riscv.vssseg5.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x
 define void @test_vssseg5_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4553,7 +4286,6 @@ entry:
 define void @test_vssseg5_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg5_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4572,7 +4304,6 @@ declare void @llvm.riscv.vssseg6.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x
 define void @test_vssseg6_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4589,7 +4320,6 @@ entry:
 define void @test_vssseg6_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg6_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4609,7 +4339,6 @@ declare void @llvm.riscv.vssseg7.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x
 define void @test_vssseg7_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4627,7 +4356,6 @@ entry:
 define void @test_vssseg7_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg7_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4648,7 +4376,6 @@ declare void @llvm.riscv.vssseg8.mask.nxv2f16(<vscale x 2 x half>,<vscale x 2 x
 define void @test_vssseg8_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4667,7 +4394,6 @@ entry:
 define void @test_vssseg8_mask_nxv2f16(<vscale x 2 x half> %val, ptr %base, i64 %offset, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg8_mask_nxv2f16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -4689,7 +4415,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv4f32(<vscale x 4 x float>,<vscale x 4 x
 define void @test_vssseg2_nxv4f32(<vscale x 4 x float> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1
@@ -4702,7 +4427,6 @@ entry:
 define void @test_vssseg2_mask_nxv4f32(<vscale x 4 x float> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
 ; CHECK-NEXT:    vssseg2e32.v v8, (a0), a1, v0.t
@@ -4718,7 +4442,6 @@ declare void @llvm.riscv.vssseg3.mask.nxv4f32(<vscale x 4 x float>,<vscale x 4 x
 define void @test_vssseg3_nxv4f32(<vscale x 4 x float> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
@@ -4732,7 +4455,6 @@ entry:
 define void @test_vssseg3_mask_nxv4f32(<vscale x 4 x float> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg3_mask_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m2, ta, ma
@@ -4749,7 +4471,6 @@ declare void @llvm.riscv.vssseg4.mask.nxv4f32(<vscale x 4 x float>,<vscale x 4 x
 define void @test_vssseg4_nxv4f32(<vscale x 4 x float> %val, ptr %base, i64 %offset, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -4764,7 +4485,6 @@ entry:
 define void @test_vssseg4_mask_nxv4f32(<vscale x 4 x float> %val, ptr %base, i64 %offset, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vssseg4_mask_nxv4f32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
diff --git a/llvm/test/CodeGen/RISCV/rvv/vsuxseg-rv32.ll b/llvm/test/CodeGen/RISCV/rvv/vsuxseg-rv32.ll
index bbe7c8ac70cf6..c8a56f1ef7cce 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vsuxseg-rv32.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vsuxseg-rv32.ll
@@ -8,7 +8,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16i16.nxv16i16(<vscale x 16 x i16>,<vs
 define void @test_vsuxseg2_nxv16i16_nxv16i16(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16i16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -22,7 +21,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16i16_nxv16i16(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16i16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -39,7 +37,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16i16.nxv16i8(<vscale x 16 x i16>,<vsc
 define void @test_vsuxseg2_nxv16i16_nxv16i8(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16i16_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -53,7 +50,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16i16_nxv16i8(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16i16_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -70,7 +66,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16i16.nxv16i32(<vscale x 16 x i16>,<vs
 define void @test_vsuxseg2_nxv16i16_nxv16i32(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16i16_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v16
@@ -83,7 +78,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16i16_nxv16i32(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16i16_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v16, v0.t
@@ -99,7 +93,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i8.nxv1i8(<vscale x 1 x i8>,<vscale x
 define void @test_vsuxseg2_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -113,7 +106,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -130,7 +122,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i8.nxv1i32(<vscale x 1 x i8>,<vscale
 define void @test_vsuxseg2_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -144,7 +135,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -161,7 +151,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i8.nxv1i16(<vscale x 1 x i8>,<vscale
 define void @test_vsuxseg2_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -175,7 +164,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -193,8 +181,8 @@ define void @test_vsuxseg3_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg3_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -207,8 +195,8 @@ define void @test_vsuxseg3_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -224,8 +212,8 @@ define void @test_vsuxseg3_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg3_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -238,8 +226,8 @@ define void @test_vsuxseg3_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -255,8 +243,8 @@ define void @test_vsuxseg3_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg3_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -269,8 +257,8 @@ define void @test_vsuxseg3_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -286,9 +274,9 @@ define void @test_vsuxseg4_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg4_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -301,9 +289,9 @@ define void @test_vsuxseg4_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -319,9 +307,9 @@ define void @test_vsuxseg4_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg4_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -334,9 +322,9 @@ define void @test_vsuxseg4_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -352,9 +340,9 @@ define void @test_vsuxseg4_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg4_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -367,9 +355,9 @@ define void @test_vsuxseg4_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -385,10 +373,10 @@ define void @test_vsuxseg5_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg5_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -401,10 +389,10 @@ define void @test_vsuxseg5_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -420,10 +408,10 @@ define void @test_vsuxseg5_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg5_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -436,10 +424,10 @@ define void @test_vsuxseg5_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -455,10 +443,10 @@ define void @test_vsuxseg5_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg5_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -471,10 +459,10 @@ define void @test_vsuxseg5_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -490,11 +478,11 @@ define void @test_vsuxseg6_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg6_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -507,11 +495,11 @@ define void @test_vsuxseg6_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -527,11 +515,11 @@ define void @test_vsuxseg6_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -544,11 +532,11 @@ define void @test_vsuxseg6_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -564,11 +552,11 @@ define void @test_vsuxseg6_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -581,11 +569,11 @@ define void @test_vsuxseg6_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -601,12 +589,12 @@ define void @test_vsuxseg7_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg7_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -619,12 +607,12 @@ define void @test_vsuxseg7_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -640,12 +628,12 @@ define void @test_vsuxseg7_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -658,12 +646,12 @@ define void @test_vsuxseg7_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -679,12 +667,12 @@ define void @test_vsuxseg7_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -697,12 +685,12 @@ define void @test_vsuxseg7_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -718,13 +706,13 @@ define void @test_vsuxseg8_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg8_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -737,13 +725,13 @@ define void @test_vsuxseg8_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -759,13 +747,13 @@ define void @test_vsuxseg8_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -778,13 +766,13 @@ define void @test_vsuxseg8_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -800,13 +788,13 @@ define void @test_vsuxseg8_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -819,13 +807,13 @@ define void @test_vsuxseg8_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -840,7 +828,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16i8.nxv16i16(<vscale x 16 x i8>,<vsca
 define void @test_vsuxseg2_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei16.v v8, (a0), v12
@@ -853,7 +840,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei16.v v8, (a0), v12, v0.t
@@ -869,7 +855,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16i8.nxv16i8(<vscale x 16 x i8>,<vscal
 define void @test_vsuxseg2_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -883,7 +868,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -900,7 +884,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16i8.nxv16i32(<vscale x 16 x i8>,<vsca
 define void @test_vsuxseg2_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v16
@@ -913,7 +896,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v16, v0.t
@@ -929,11 +911,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv16i8.nxv16i16(<vscale x 16 x i8>,<vsca
 define void @test_vsuxseg3_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei16.v v16, (a0), v12
+; CHECK-NEXT:    vsuxseg3ei16.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, i32 %vl)
@@ -943,11 +925,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei16.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsuxseg3ei16.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl)
@@ -961,8 +943,8 @@ define void @test_vsuxseg3_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -975,8 +957,8 @@ define void @test_vsuxseg3_mask_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -991,7 +973,6 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv16i8.nxv16i32(<vscale x 16 x i8>,<vsca
 define void @test_vsuxseg3_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -1005,7 +986,6 @@ entry:
 define void @test_vsuxseg3_mask_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -1023,9 +1003,9 @@ define void @test_vsuxseg4_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -1038,9 +1018,9 @@ define void @test_vsuxseg4_mask_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -1056,9 +1036,9 @@ define void @test_vsuxseg4_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -1071,9 +1051,9 @@ define void @test_vsuxseg4_mask_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -1088,7 +1068,6 @@ declare void @llvm.riscv.vsuxseg4.mask.nxv16i8.nxv16i32(<vscale x 16 x i8>,<vsca
 define void @test_vsuxseg4_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -1103,7 +1082,6 @@ entry:
 define void @test_vsuxseg4_mask_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -1121,7 +1099,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i32.nxv2i32(<vscale x 2 x i32>,<vscal
 define void @test_vsuxseg2_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -1135,7 +1112,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -1152,7 +1128,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i32.nxv2i8(<vscale x 2 x i32>,<vscale
 define void @test_vsuxseg2_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -1166,7 +1141,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -1183,7 +1157,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i32.nxv2i16(<vscale x 2 x i32>,<vscal
 define void @test_vsuxseg2_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -1197,7 +1170,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -1215,8 +1187,8 @@ define void @test_vsuxseg3_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1229,8 +1201,8 @@ define void @test_vsuxseg3_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1246,8 +1218,8 @@ define void @test_vsuxseg3_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg3_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1260,8 +1232,8 @@ define void @test_vsuxseg3_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1277,8 +1249,8 @@ define void @test_vsuxseg3_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1291,8 +1263,8 @@ define void @test_vsuxseg3_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1308,9 +1280,9 @@ define void @test_vsuxseg4_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1323,9 +1295,9 @@ define void @test_vsuxseg4_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1341,9 +1313,9 @@ define void @test_vsuxseg4_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg4_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1356,9 +1328,9 @@ define void @test_vsuxseg4_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1374,9 +1346,9 @@ define void @test_vsuxseg4_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1389,9 +1361,9 @@ define void @test_vsuxseg4_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1407,10 +1379,10 @@ define void @test_vsuxseg5_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1423,10 +1395,10 @@ define void @test_vsuxseg5_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1442,10 +1414,10 @@ define void @test_vsuxseg5_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg5_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1458,10 +1430,10 @@ define void @test_vsuxseg5_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1477,10 +1449,10 @@ define void @test_vsuxseg5_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1493,10 +1465,10 @@ define void @test_vsuxseg5_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1512,11 +1484,11 @@ define void @test_vsuxseg6_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1529,11 +1501,11 @@ define void @test_vsuxseg6_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1549,11 +1521,11 @@ define void @test_vsuxseg6_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg6_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1566,11 +1538,11 @@ define void @test_vsuxseg6_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1586,11 +1558,11 @@ define void @test_vsuxseg6_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1603,11 +1575,11 @@ define void @test_vsuxseg6_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1623,12 +1595,12 @@ define void @test_vsuxseg7_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1641,12 +1613,12 @@ define void @test_vsuxseg7_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1662,12 +1634,12 @@ define void @test_vsuxseg7_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg7_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1680,12 +1652,12 @@ define void @test_vsuxseg7_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1701,12 +1673,12 @@ define void @test_vsuxseg7_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1719,12 +1691,12 @@ define void @test_vsuxseg7_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1740,13 +1712,13 @@ define void @test_vsuxseg8_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1759,13 +1731,13 @@ define void @test_vsuxseg8_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1781,13 +1753,13 @@ define void @test_vsuxseg8_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg8_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1800,13 +1772,13 @@ define void @test_vsuxseg8_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1822,13 +1794,13 @@ define void @test_vsuxseg8_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1841,13 +1813,13 @@ define void @test_vsuxseg8_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1862,7 +1834,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i16.nxv4i16(<vscale x 4 x i16>,<vscal
 define void @test_vsuxseg2_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -1876,7 +1847,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -1893,7 +1863,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i16.nxv4i8(<vscale x 4 x i16>,<vscale
 define void @test_vsuxseg2_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -1907,7 +1876,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -1924,7 +1892,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i16.nxv4i32(<vscale x 4 x i16>,<vscal
 define void @test_vsuxseg2_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v10
@@ -1937,7 +1904,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v10, v0.t
@@ -1954,8 +1920,8 @@ define void @test_vsuxseg3_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1968,8 +1934,8 @@ define void @test_vsuxseg3_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1985,8 +1951,8 @@ define void @test_vsuxseg3_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg3_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1999,8 +1965,8 @@ define void @test_vsuxseg3_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2015,11 +1981,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv4i16.nxv4i32(<vscale x 4 x i16>,<vscal
 define void @test_vsuxseg3_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl)
@@ -2029,11 +1995,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl)
@@ -2047,9 +2013,9 @@ define void @test_vsuxseg4_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2062,9 +2028,9 @@ define void @test_vsuxseg4_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2080,9 +2046,9 @@ define void @test_vsuxseg4_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg4_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2095,9 +2061,9 @@ define void @test_vsuxseg4_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2113,9 +2079,9 @@ define void @test_vsuxseg4_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -2128,9 +2094,9 @@ define void @test_vsuxseg4_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -2146,10 +2112,10 @@ define void @test_vsuxseg5_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2162,10 +2128,10 @@ define void @test_vsuxseg5_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2181,10 +2147,10 @@ define void @test_vsuxseg5_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg5_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2197,10 +2163,10 @@ define void @test_vsuxseg5_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2216,10 +2182,10 @@ define void @test_vsuxseg5_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -2232,10 +2198,10 @@ define void @test_vsuxseg5_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -2251,11 +2217,11 @@ define void @test_vsuxseg6_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2268,11 +2234,11 @@ define void @test_vsuxseg6_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2288,11 +2254,11 @@ define void @test_vsuxseg6_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg6_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2305,11 +2271,11 @@ define void @test_vsuxseg6_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2325,11 +2291,11 @@ define void @test_vsuxseg6_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -2342,11 +2308,11 @@ define void @test_vsuxseg6_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -2362,12 +2328,12 @@ define void @test_vsuxseg7_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2380,12 +2346,12 @@ define void @test_vsuxseg7_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2401,12 +2367,12 @@ define void @test_vsuxseg7_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg7_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2419,12 +2385,12 @@ define void @test_vsuxseg7_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2440,12 +2406,12 @@ define void @test_vsuxseg7_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -2458,12 +2424,12 @@ define void @test_vsuxseg7_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -2479,13 +2445,13 @@ define void @test_vsuxseg8_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2498,13 +2464,13 @@ define void @test_vsuxseg8_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2520,13 +2486,13 @@ define void @test_vsuxseg8_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg8_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2539,13 +2505,13 @@ define void @test_vsuxseg8_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2561,13 +2527,13 @@ define void @test_vsuxseg8_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -2580,13 +2546,13 @@ define void @test_vsuxseg8_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -2601,7 +2567,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i32.nxv1i8(<vscale x 1 x i32>,<vscale
 define void @test_vsuxseg2_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -2615,7 +2580,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -2632,7 +2596,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i32.nxv1i32(<vscale x 1 x i32>,<vscal
 define void @test_vsuxseg2_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -2646,7 +2609,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -2663,7 +2625,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i32.nxv1i16(<vscale x 1 x i32>,<vscal
 define void @test_vsuxseg2_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -2677,7 +2638,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -2695,8 +2655,8 @@ define void @test_vsuxseg3_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg3_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2709,8 +2669,8 @@ define void @test_vsuxseg3_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2726,8 +2686,8 @@ define void @test_vsuxseg3_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2740,8 +2700,8 @@ define void @test_vsuxseg3_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2757,8 +2717,8 @@ define void @test_vsuxseg3_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2771,8 +2731,8 @@ define void @test_vsuxseg3_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2788,9 +2748,9 @@ define void @test_vsuxseg4_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg4_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2803,9 +2763,9 @@ define void @test_vsuxseg4_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2821,9 +2781,9 @@ define void @test_vsuxseg4_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2836,9 +2796,9 @@ define void @test_vsuxseg4_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2854,9 +2814,9 @@ define void @test_vsuxseg4_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2869,9 +2829,9 @@ define void @test_vsuxseg4_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2887,10 +2847,10 @@ define void @test_vsuxseg5_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg5_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2903,10 +2863,10 @@ define void @test_vsuxseg5_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2922,10 +2882,10 @@ define void @test_vsuxseg5_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2938,10 +2898,10 @@ define void @test_vsuxseg5_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2957,10 +2917,10 @@ define void @test_vsuxseg5_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2973,10 +2933,10 @@ define void @test_vsuxseg5_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2992,11 +2952,11 @@ define void @test_vsuxseg6_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg6_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3009,11 +2969,11 @@ define void @test_vsuxseg6_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3029,11 +2989,11 @@ define void @test_vsuxseg6_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3046,11 +3006,11 @@ define void @test_vsuxseg6_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3066,11 +3026,11 @@ define void @test_vsuxseg6_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3083,11 +3043,11 @@ define void @test_vsuxseg6_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3103,12 +3063,12 @@ define void @test_vsuxseg7_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg7_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3121,12 +3081,12 @@ define void @test_vsuxseg7_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3142,12 +3102,12 @@ define void @test_vsuxseg7_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3160,12 +3120,12 @@ define void @test_vsuxseg7_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3181,12 +3141,12 @@ define void @test_vsuxseg7_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3199,12 +3159,12 @@ define void @test_vsuxseg7_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3220,13 +3180,13 @@ define void @test_vsuxseg8_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg8_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3239,13 +3199,13 @@ define void @test_vsuxseg8_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3261,13 +3221,13 @@ define void @test_vsuxseg8_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3280,13 +3240,13 @@ define void @test_vsuxseg8_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3302,13 +3262,13 @@ define void @test_vsuxseg8_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3321,13 +3281,13 @@ define void @test_vsuxseg8_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3342,7 +3302,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i16.nxv8i16(<vscale x 8 x i16>,<vscal
 define void @test_vsuxseg2_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -3356,7 +3315,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -3373,7 +3331,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i16.nxv8i8(<vscale x 8 x i16>,<vscale
 define void @test_vsuxseg2_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -3387,7 +3344,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -3404,7 +3360,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i16.nxv8i32(<vscale x 8 x i16>,<vscal
 define void @test_vsuxseg2_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v12
@@ -3417,7 +3372,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v12, v0.t
@@ -3434,8 +3388,8 @@ define void @test_vsuxseg3_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3448,8 +3402,8 @@ define void @test_vsuxseg3_mask_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3465,8 +3419,8 @@ define void @test_vsuxseg3_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg3_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3479,8 +3433,8 @@ define void @test_vsuxseg3_mask_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3495,11 +3449,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv8i16.nxv8i32(<vscale x 8 x i16>,<vscal
 define void @test_vsuxseg3_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v16, (a0), v12
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl)
@@ -3509,11 +3463,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl)
@@ -3527,10 +3481,10 @@ define void @test_vsuxseg4_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
-; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
 entry:
@@ -3542,9 +3496,9 @@ define void @test_vsuxseg4_mask_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3560,9 +3514,9 @@ define void @test_vsuxseg4_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg4_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3575,9 +3529,9 @@ define void @test_vsuxseg4_mask_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3593,9 +3547,9 @@ define void @test_vsuxseg4_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -3608,9 +3562,9 @@ define void @test_vsuxseg4_mask_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -3625,7 +3579,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i8.nxv8i16(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg2_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei16.v v8, (a0), v10
@@ -3638,7 +3591,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei16.v v8, (a0), v10, v0.t
@@ -3654,7 +3606,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i8.nxv8i8(<vscale x 8 x i8>,<vscale x
 define void @test_vsuxseg2_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -3668,7 +3619,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -3685,7 +3635,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i8.nxv8i32(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg2_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v12
@@ -3698,7 +3647,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v12, v0.t
@@ -3714,11 +3662,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv8i8.nxv8i16(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg3_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
-; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10
+; CHECK-NEXT:    vsuxseg3ei16.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl)
@@ -3728,11 +3676,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
-; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsuxseg3ei16.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl)
@@ -3746,8 +3694,8 @@ define void @test_vsuxseg3_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg3_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3760,8 +3708,8 @@ define void @test_vsuxseg3_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3776,7 +3724,6 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv8i8.nxv8i32(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg3_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -3790,7 +3737,6 @@ entry:
 define void @test_vsuxseg3_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -3808,9 +3754,9 @@ define void @test_vsuxseg4_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg4_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3823,9 +3769,9 @@ define void @test_vsuxseg4_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3841,9 +3787,9 @@ define void @test_vsuxseg4_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg4_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3856,9 +3802,9 @@ define void @test_vsuxseg4_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3873,7 +3819,6 @@ declare void @llvm.riscv.vsuxseg4.mask.nxv8i8.nxv8i32(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg4_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3888,7 +3833,6 @@ entry:
 define void @test_vsuxseg4_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3907,10 +3851,10 @@ define void @test_vsuxseg5_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg5_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3923,10 +3867,10 @@ define void @test_vsuxseg5_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3942,10 +3886,10 @@ define void @test_vsuxseg5_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg5_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3958,10 +3902,10 @@ define void @test_vsuxseg5_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3976,13 +3920,13 @@ declare void @llvm.riscv.vsuxseg5.mask.nxv8i8.nxv8i32(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg5_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg5_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
-; CHECK-NEXT:    vsuxseg5ei32.v v16, (a0), v12
+; CHECK-NEXT:    vsuxseg5ei32.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg5.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl)
@@ -3992,13 +3936,13 @@ entry:
 define void @test_vsuxseg5_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
-; CHECK-NEXT:    vsuxseg5ei32.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsuxseg5ei32.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg5.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl)
@@ -4012,11 +3956,11 @@ define void @test_vsuxseg6_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -4029,11 +3973,11 @@ define void @test_vsuxseg6_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -4049,11 +3993,11 @@ define void @test_vsuxseg6_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg6_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4066,11 +4010,11 @@ define void @test_vsuxseg6_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4086,11 +4030,11 @@ define void @test_vsuxseg6_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -4103,11 +4047,11 @@ define void @test_vsuxseg6_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -4123,12 +4067,12 @@ define void @test_vsuxseg7_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -4141,12 +4085,12 @@ define void @test_vsuxseg7_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -4162,12 +4106,12 @@ define void @test_vsuxseg7_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg7_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4180,12 +4124,12 @@ define void @test_vsuxseg7_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4201,12 +4145,12 @@ define void @test_vsuxseg7_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -4219,12 +4163,12 @@ define void @test_vsuxseg7_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -4240,13 +4184,13 @@ define void @test_vsuxseg8_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -4259,13 +4203,13 @@ define void @test_vsuxseg8_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -4281,13 +4225,13 @@ define void @test_vsuxseg8_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg8_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4300,13 +4244,13 @@ define void @test_vsuxseg8_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4322,13 +4266,13 @@ define void @test_vsuxseg8_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -4341,13 +4285,13 @@ define void @test_vsuxseg8_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -4362,7 +4306,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i32.nxv8i16(<vscale x 8 x i32>,<vscal
 define void @test_vsuxseg2_nxv8i32_nxv8i16(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i32_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -4376,7 +4319,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i32_nxv8i16(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i32_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -4393,7 +4335,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i32.nxv8i8(<vscale x 8 x i32>,<vscale
 define void @test_vsuxseg2_nxv8i32_nxv8i8(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i32_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -4407,7 +4348,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i32_nxv8i8(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i32_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -4424,7 +4364,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i32.nxv8i32(<vscale x 8 x i32>,<vscal
 define void @test_vsuxseg2_nxv8i32_nxv8i32(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i32_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -4438,7 +4377,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i32_nxv8i32(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i32_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -4455,7 +4393,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i8.nxv4i16(<vscale x 4 x i8>,<vscale
 define void @test_vsuxseg2_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -4469,7 +4406,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -4486,7 +4422,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i8.nxv4i8(<vscale x 4 x i8>,<vscale x
 define void @test_vsuxseg2_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -4500,7 +4435,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -4517,7 +4451,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i8.nxv4i32(<vscale x 4 x i8>,<vscale
 define void @test_vsuxseg2_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v10
@@ -4530,7 +4463,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v10, v0.t
@@ -4547,8 +4479,8 @@ define void @test_vsuxseg3_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg3_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4561,8 +4493,8 @@ define void @test_vsuxseg3_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4578,8 +4510,8 @@ define void @test_vsuxseg3_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg3_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4592,8 +4524,8 @@ define void @test_vsuxseg3_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4608,11 +4540,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv4i8.nxv4i32(<vscale x 4 x i8>,<vscale
 define void @test_vsuxseg3_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl)
@@ -4622,11 +4554,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl)
@@ -4640,9 +4572,9 @@ define void @test_vsuxseg4_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg4_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4655,9 +4587,9 @@ define void @test_vsuxseg4_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4673,9 +4605,9 @@ define void @test_vsuxseg4_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg4_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4688,9 +4620,9 @@ define void @test_vsuxseg4_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4706,9 +4638,9 @@ define void @test_vsuxseg4_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg4_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -4721,9 +4653,9 @@ define void @test_vsuxseg4_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -4739,10 +4671,10 @@ define void @test_vsuxseg5_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg5_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4755,10 +4687,10 @@ define void @test_vsuxseg5_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4774,10 +4706,10 @@ define void @test_vsuxseg5_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg5_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4790,10 +4722,10 @@ define void @test_vsuxseg5_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4809,10 +4741,10 @@ define void @test_vsuxseg5_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg5_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -4825,10 +4757,10 @@ define void @test_vsuxseg5_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -4844,11 +4776,11 @@ define void @test_vsuxseg6_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4861,11 +4793,11 @@ define void @test_vsuxseg6_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4881,11 +4813,11 @@ define void @test_vsuxseg6_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg6_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4898,11 +4830,11 @@ define void @test_vsuxseg6_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4918,11 +4850,11 @@ define void @test_vsuxseg6_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -4935,11 +4867,11 @@ define void @test_vsuxseg6_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -4955,12 +4887,12 @@ define void @test_vsuxseg7_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4973,12 +4905,12 @@ define void @test_vsuxseg7_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4994,12 +4926,12 @@ define void @test_vsuxseg7_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg7_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5012,12 +4944,12 @@ define void @test_vsuxseg7_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5033,12 +4965,12 @@ define void @test_vsuxseg7_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -5051,12 +4983,12 @@ define void @test_vsuxseg7_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -5072,13 +5004,13 @@ define void @test_vsuxseg8_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5091,13 +5023,13 @@ define void @test_vsuxseg8_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5113,13 +5045,13 @@ define void @test_vsuxseg8_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg8_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5132,13 +5064,13 @@ define void @test_vsuxseg8_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5154,13 +5086,13 @@ define void @test_vsuxseg8_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -5173,13 +5105,13 @@ define void @test_vsuxseg8_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -5194,7 +5126,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i16.nxv1i8(<vscale x 1 x i16>,<vscale
 define void @test_vsuxseg2_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -5208,7 +5139,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -5225,7 +5155,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i16.nxv1i32(<vscale x 1 x i16>,<vscal
 define void @test_vsuxseg2_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -5239,7 +5168,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -5256,7 +5184,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i16.nxv1i16(<vscale x 1 x i16>,<vscal
 define void @test_vsuxseg2_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -5270,7 +5197,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -5288,8 +5214,8 @@ define void @test_vsuxseg3_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg3_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5302,8 +5228,8 @@ define void @test_vsuxseg3_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5319,8 +5245,8 @@ define void @test_vsuxseg3_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5333,8 +5259,8 @@ define void @test_vsuxseg3_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5350,8 +5276,8 @@ define void @test_vsuxseg3_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5364,8 +5290,8 @@ define void @test_vsuxseg3_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5381,9 +5307,9 @@ define void @test_vsuxseg4_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg4_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5396,9 +5322,9 @@ define void @test_vsuxseg4_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5414,9 +5340,9 @@ define void @test_vsuxseg4_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5429,9 +5355,9 @@ define void @test_vsuxseg4_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5447,9 +5373,9 @@ define void @test_vsuxseg4_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5462,9 +5388,9 @@ define void @test_vsuxseg4_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5480,10 +5406,10 @@ define void @test_vsuxseg5_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg5_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5496,10 +5422,10 @@ define void @test_vsuxseg5_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5515,10 +5441,10 @@ define void @test_vsuxseg5_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5531,10 +5457,10 @@ define void @test_vsuxseg5_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5550,10 +5476,10 @@ define void @test_vsuxseg5_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5566,10 +5492,10 @@ define void @test_vsuxseg5_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5585,11 +5511,11 @@ define void @test_vsuxseg6_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg6_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5602,11 +5528,11 @@ define void @test_vsuxseg6_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5622,11 +5548,11 @@ define void @test_vsuxseg6_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5639,11 +5565,11 @@ define void @test_vsuxseg6_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5659,11 +5585,11 @@ define void @test_vsuxseg6_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5676,11 +5602,11 @@ define void @test_vsuxseg6_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5696,12 +5622,12 @@ define void @test_vsuxseg7_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg7_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5714,12 +5640,12 @@ define void @test_vsuxseg7_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5735,12 +5661,12 @@ define void @test_vsuxseg7_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5753,12 +5679,12 @@ define void @test_vsuxseg7_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5774,12 +5700,12 @@ define void @test_vsuxseg7_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5792,12 +5718,12 @@ define void @test_vsuxseg7_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5813,13 +5739,13 @@ define void @test_vsuxseg8_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg8_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5832,13 +5758,13 @@ define void @test_vsuxseg8_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5854,13 +5780,13 @@ define void @test_vsuxseg8_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5873,13 +5799,13 @@ define void @test_vsuxseg8_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5895,13 +5821,13 @@ define void @test_vsuxseg8_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5914,13 +5840,13 @@ define void @test_vsuxseg8_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5935,7 +5861,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv32i8.nxv32i16(<vscale x 32 x i8>,<vsca
 define void @test_vsuxseg2_nxv32i8_nxv32i16(<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv32i8_nxv32i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei16.v v8, (a0), v16
@@ -5948,7 +5873,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv32i8_nxv32i16(<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i16> %index, <vscale x 32 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv32i8_nxv32i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei16.v v8, (a0), v16, v0.t
@@ -5964,7 +5888,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv32i8.nxv32i8(<vscale x 32 x i8>,<vscal
 define void @test_vsuxseg2_nxv32i8_nxv32i8(<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv32i8_nxv32i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
@@ -5978,7 +5901,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv32i8_nxv32i8(<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i8> %index, <vscale x 32 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv32i8_nxv32i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
@@ -5995,7 +5917,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i8.nxv2i32(<vscale x 2 x i8>,<vscale
 define void @test_vsuxseg2_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -6009,7 +5930,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -6026,7 +5946,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i8.nxv2i8(<vscale x 2 x i8>,<vscale x
 define void @test_vsuxseg2_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -6040,7 +5959,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -6057,7 +5975,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i8.nxv2i16(<vscale x 2 x i8>,<vscale
 define void @test_vsuxseg2_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -6071,7 +5988,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -6089,8 +6005,8 @@ define void @test_vsuxseg3_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg3_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6103,8 +6019,8 @@ define void @test_vsuxseg3_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6120,8 +6036,8 @@ define void @test_vsuxseg3_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg3_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6134,8 +6050,8 @@ define void @test_vsuxseg3_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6151,8 +6067,8 @@ define void @test_vsuxseg3_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg3_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6165,8 +6081,8 @@ define void @test_vsuxseg3_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6182,9 +6098,9 @@ define void @test_vsuxseg4_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg4_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6197,9 +6113,9 @@ define void @test_vsuxseg4_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6215,9 +6131,9 @@ define void @test_vsuxseg4_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg4_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6230,9 +6146,9 @@ define void @test_vsuxseg4_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6248,9 +6164,9 @@ define void @test_vsuxseg4_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg4_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6263,9 +6179,9 @@ define void @test_vsuxseg4_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6281,10 +6197,10 @@ define void @test_vsuxseg5_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg5_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6297,10 +6213,10 @@ define void @test_vsuxseg5_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6316,10 +6232,10 @@ define void @test_vsuxseg5_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg5_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6332,10 +6248,10 @@ define void @test_vsuxseg5_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6351,10 +6267,10 @@ define void @test_vsuxseg5_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg5_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6367,10 +6283,10 @@ define void @test_vsuxseg5_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6386,11 +6302,11 @@ define void @test_vsuxseg6_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6403,11 +6319,11 @@ define void @test_vsuxseg6_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6423,11 +6339,11 @@ define void @test_vsuxseg6_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg6_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6440,11 +6356,11 @@ define void @test_vsuxseg6_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6460,11 +6376,11 @@ define void @test_vsuxseg6_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6477,11 +6393,11 @@ define void @test_vsuxseg6_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6497,12 +6413,12 @@ define void @test_vsuxseg7_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6515,12 +6431,12 @@ define void @test_vsuxseg7_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6536,12 +6452,12 @@ define void @test_vsuxseg7_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg7_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6554,12 +6470,12 @@ define void @test_vsuxseg7_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6575,12 +6491,12 @@ define void @test_vsuxseg7_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6593,12 +6509,12 @@ define void @test_vsuxseg7_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6614,13 +6530,13 @@ define void @test_vsuxseg8_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6633,13 +6549,13 @@ define void @test_vsuxseg8_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6655,13 +6571,13 @@ define void @test_vsuxseg8_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg8_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6674,13 +6590,13 @@ define void @test_vsuxseg8_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6696,13 +6612,13 @@ define void @test_vsuxseg8_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6715,13 +6631,13 @@ define void @test_vsuxseg8_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6736,7 +6652,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i16.nxv2i32(<vscale x 2 x i16>,<vscal
 define void @test_vsuxseg2_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -6750,7 +6665,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -6767,7 +6681,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i16.nxv2i8(<vscale x 2 x i16>,<vscale
 define void @test_vsuxseg2_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -6781,7 +6694,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -6798,7 +6710,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i16.nxv2i16(<vscale x 2 x i16>,<vscal
 define void @test_vsuxseg2_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -6812,7 +6723,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -6830,8 +6740,8 @@ define void @test_vsuxseg3_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6844,8 +6754,8 @@ define void @test_vsuxseg3_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6861,8 +6771,8 @@ define void @test_vsuxseg3_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg3_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6875,8 +6785,8 @@ define void @test_vsuxseg3_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6892,8 +6802,8 @@ define void @test_vsuxseg3_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6906,8 +6816,8 @@ define void @test_vsuxseg3_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6923,9 +6833,9 @@ define void @test_vsuxseg4_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6938,9 +6848,9 @@ define void @test_vsuxseg4_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6956,9 +6866,9 @@ define void @test_vsuxseg4_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg4_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6971,9 +6881,9 @@ define void @test_vsuxseg4_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6989,9 +6899,9 @@ define void @test_vsuxseg4_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7004,9 +6914,9 @@ define void @test_vsuxseg4_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7022,10 +6932,10 @@ define void @test_vsuxseg5_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7038,10 +6948,10 @@ define void @test_vsuxseg5_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7057,10 +6967,10 @@ define void @test_vsuxseg5_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg5_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7073,10 +6983,10 @@ define void @test_vsuxseg5_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7092,10 +7002,10 @@ define void @test_vsuxseg5_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7108,10 +7018,10 @@ define void @test_vsuxseg5_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7127,11 +7037,11 @@ define void @test_vsuxseg6_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7144,11 +7054,11 @@ define void @test_vsuxseg6_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7164,11 +7074,11 @@ define void @test_vsuxseg6_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg6_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7181,11 +7091,11 @@ define void @test_vsuxseg6_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7201,11 +7111,11 @@ define void @test_vsuxseg6_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7218,11 +7128,11 @@ define void @test_vsuxseg6_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7238,13 +7148,13 @@ define void @test_vsuxseg7_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
 entry:
@@ -7256,12 +7166,12 @@ define void @test_vsuxseg7_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7277,12 +7187,12 @@ define void @test_vsuxseg7_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg7_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7295,12 +7205,12 @@ define void @test_vsuxseg7_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7316,12 +7226,12 @@ define void @test_vsuxseg7_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7334,12 +7244,12 @@ define void @test_vsuxseg7_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7355,13 +7265,13 @@ define void @test_vsuxseg8_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7374,13 +7284,13 @@ define void @test_vsuxseg8_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7396,13 +7306,13 @@ define void @test_vsuxseg8_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg8_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7415,13 +7325,13 @@ define void @test_vsuxseg8_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7437,13 +7347,13 @@ define void @test_vsuxseg8_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7456,13 +7366,13 @@ define void @test_vsuxseg8_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7477,7 +7387,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i32.nxv4i16(<vscale x 4 x i32>,<vscal
 define void @test_vsuxseg2_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -7491,7 +7400,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -7508,7 +7416,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i32.nxv4i8(<vscale x 4 x i32>,<vscale
 define void @test_vsuxseg2_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -7522,7 +7429,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -7539,7 +7445,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i32.nxv4i32(<vscale x 4 x i32>,<vscal
 define void @test_vsuxseg2_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -7553,7 +7458,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -7571,8 +7475,8 @@ define void @test_vsuxseg3_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7585,8 +7489,8 @@ define void @test_vsuxseg3_mask_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7602,8 +7506,8 @@ define void @test_vsuxseg3_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg3_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7616,8 +7520,8 @@ define void @test_vsuxseg3_mask_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7633,8 +7537,8 @@ define void @test_vsuxseg3_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7647,8 +7551,8 @@ define void @test_vsuxseg3_mask_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7664,9 +7568,9 @@ define void @test_vsuxseg4_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7679,9 +7583,9 @@ define void @test_vsuxseg4_mask_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7697,9 +7601,9 @@ define void @test_vsuxseg4_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg4_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7712,9 +7616,9 @@ define void @test_vsuxseg4_mask_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7730,9 +7634,9 @@ define void @test_vsuxseg4_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7745,9 +7649,9 @@ define void @test_vsuxseg4_mask_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7762,7 +7666,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16f16.nxv16i16(<vscale x 16 x half>,<v
 define void @test_vsuxseg2_nxv16f16_nxv16i16(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16f16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -7776,7 +7679,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16f16_nxv16i16(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16f16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -7793,7 +7695,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16f16.nxv16i8(<vscale x 16 x half>,<vs
 define void @test_vsuxseg2_nxv16f16_nxv16i8(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16f16_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -7807,7 +7708,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16f16_nxv16i8(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16f16_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -7824,7 +7724,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16f16.nxv16i32(<vscale x 16 x half>,<v
 define void @test_vsuxseg2_nxv16f16_nxv16i32(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16f16_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v16
@@ -7837,7 +7736,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16f16_nxv16i32(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16f16_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v16, v0.t
@@ -7853,7 +7751,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f64.nxv4i16(<vscale x 4 x double>,<vs
 define void @test_vsuxseg2_nxv4f64_nxv4i16(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f64_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7867,7 +7764,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f64_nxv4i16(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f64_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7884,7 +7780,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f64.nxv4i8(<vscale x 4 x double>,<vsc
 define void @test_vsuxseg2_nxv4f64_nxv4i8(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f64_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7898,7 +7793,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f64_nxv4i8(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f64_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7915,7 +7809,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f64.nxv4i32(<vscale x 4 x double>,<vs
 define void @test_vsuxseg2_nxv4f64_nxv4i32(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f64_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7929,7 +7822,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f64_nxv4i32(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f64_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7946,7 +7838,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f64.nxv1i8(<vscale x 1 x double>,<vsc
 define void @test_vsuxseg2_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -7960,7 +7851,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -7977,7 +7867,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f64.nxv1i32(<vscale x 1 x double>,<vs
 define void @test_vsuxseg2_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -7991,7 +7880,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -8008,7 +7896,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f64.nxv1i16(<vscale x 1 x double>,<vs
 define void @test_vsuxseg2_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -8022,7 +7909,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -8040,8 +7926,8 @@ define void @test_vsuxseg3_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8054,8 +7940,8 @@ define void @test_vsuxseg3_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8071,8 +7957,8 @@ define void @test_vsuxseg3_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8085,8 +7971,8 @@ define void @test_vsuxseg3_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8102,8 +7988,8 @@ define void @test_vsuxseg3_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8116,8 +8002,8 @@ define void @test_vsuxseg3_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8133,9 +8019,9 @@ define void @test_vsuxseg4_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8148,9 +8034,9 @@ define void @test_vsuxseg4_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8166,9 +8052,9 @@ define void @test_vsuxseg4_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8181,9 +8067,9 @@ define void @test_vsuxseg4_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8199,9 +8085,9 @@ define void @test_vsuxseg4_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8214,9 +8100,9 @@ define void @test_vsuxseg4_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8232,10 +8118,10 @@ define void @test_vsuxseg5_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8248,10 +8134,10 @@ define void @test_vsuxseg5_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8267,10 +8153,10 @@ define void @test_vsuxseg5_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8283,10 +8169,10 @@ define void @test_vsuxseg5_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8302,10 +8188,10 @@ define void @test_vsuxseg5_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8318,10 +8204,10 @@ define void @test_vsuxseg5_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8337,11 +8223,11 @@ define void @test_vsuxseg6_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8354,11 +8240,11 @@ define void @test_vsuxseg6_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8374,11 +8260,11 @@ define void @test_vsuxseg6_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8391,11 +8277,11 @@ define void @test_vsuxseg6_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8411,11 +8297,11 @@ define void @test_vsuxseg6_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8428,11 +8314,11 @@ define void @test_vsuxseg6_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8448,12 +8334,12 @@ define void @test_vsuxseg7_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8466,12 +8352,12 @@ define void @test_vsuxseg7_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8487,12 +8373,12 @@ define void @test_vsuxseg7_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8505,12 +8391,12 @@ define void @test_vsuxseg7_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8526,12 +8412,12 @@ define void @test_vsuxseg7_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8544,12 +8430,12 @@ define void @test_vsuxseg7_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8565,13 +8451,13 @@ define void @test_vsuxseg8_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8584,13 +8470,13 @@ define void @test_vsuxseg8_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8606,13 +8492,13 @@ define void @test_vsuxseg8_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8625,13 +8511,13 @@ define void @test_vsuxseg8_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8647,13 +8533,13 @@ define void @test_vsuxseg8_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8666,13 +8552,13 @@ define void @test_vsuxseg8_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8687,7 +8573,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f32.nxv2i32(<vscale x 2 x float>,<vsc
 define void @test_vsuxseg2_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -8701,7 +8586,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -8718,7 +8602,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f32.nxv2i8(<vscale x 2 x float>,<vsca
 define void @test_vsuxseg2_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -8732,7 +8615,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -8749,7 +8631,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f32.nxv2i16(<vscale x 2 x float>,<vsc
 define void @test_vsuxseg2_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -8763,7 +8644,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -8781,8 +8661,8 @@ define void @test_vsuxseg3_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8795,8 +8675,8 @@ define void @test_vsuxseg3_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8812,8 +8692,8 @@ define void @test_vsuxseg3_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8826,8 +8706,8 @@ define void @test_vsuxseg3_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8843,8 +8723,8 @@ define void @test_vsuxseg3_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8857,8 +8737,8 @@ define void @test_vsuxseg3_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8874,9 +8754,9 @@ define void @test_vsuxseg4_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8889,9 +8769,9 @@ define void @test_vsuxseg4_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8907,9 +8787,9 @@ define void @test_vsuxseg4_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8922,9 +8802,9 @@ define void @test_vsuxseg4_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8940,9 +8820,9 @@ define void @test_vsuxseg4_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8955,9 +8835,9 @@ define void @test_vsuxseg4_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8973,10 +8853,10 @@ define void @test_vsuxseg5_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8989,10 +8869,10 @@ define void @test_vsuxseg5_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9008,10 +8888,10 @@ define void @test_vsuxseg5_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9024,10 +8904,10 @@ define void @test_vsuxseg5_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9043,10 +8923,10 @@ define void @test_vsuxseg5_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9059,10 +8939,10 @@ define void @test_vsuxseg5_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9078,11 +8958,11 @@ define void @test_vsuxseg6_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9095,11 +8975,11 @@ define void @test_vsuxseg6_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9115,11 +8995,11 @@ define void @test_vsuxseg6_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9132,11 +9012,11 @@ define void @test_vsuxseg6_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9152,11 +9032,11 @@ define void @test_vsuxseg6_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9169,11 +9049,11 @@ define void @test_vsuxseg6_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9189,12 +9069,12 @@ define void @test_vsuxseg7_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9207,12 +9087,12 @@ define void @test_vsuxseg7_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9228,12 +9108,12 @@ define void @test_vsuxseg7_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9246,12 +9126,12 @@ define void @test_vsuxseg7_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9267,12 +9147,12 @@ define void @test_vsuxseg7_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9285,12 +9165,12 @@ define void @test_vsuxseg7_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9306,13 +9186,13 @@ define void @test_vsuxseg8_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9325,13 +9205,13 @@ define void @test_vsuxseg8_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9347,13 +9227,13 @@ define void @test_vsuxseg8_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9366,13 +9246,13 @@ define void @test_vsuxseg8_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9388,13 +9268,13 @@ define void @test_vsuxseg8_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9407,13 +9287,13 @@ define void @test_vsuxseg8_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9428,7 +9308,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f16.nxv1i8(<vscale x 1 x half>,<vscal
 define void @test_vsuxseg2_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -9442,7 +9321,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -9459,7 +9337,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f16.nxv1i32(<vscale x 1 x half>,<vsca
 define void @test_vsuxseg2_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -9473,7 +9350,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -9490,7 +9366,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f16.nxv1i16(<vscale x 1 x half>,<vsca
 define void @test_vsuxseg2_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -9504,7 +9379,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -9522,8 +9396,8 @@ define void @test_vsuxseg3_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9536,8 +9410,8 @@ define void @test_vsuxseg3_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9553,8 +9427,8 @@ define void @test_vsuxseg3_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9567,8 +9441,8 @@ define void @test_vsuxseg3_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9584,8 +9458,8 @@ define void @test_vsuxseg3_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9598,8 +9472,8 @@ define void @test_vsuxseg3_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9615,9 +9489,9 @@ define void @test_vsuxseg4_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9630,9 +9504,9 @@ define void @test_vsuxseg4_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9648,9 +9522,9 @@ define void @test_vsuxseg4_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9663,9 +9537,9 @@ define void @test_vsuxseg4_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9681,9 +9555,9 @@ define void @test_vsuxseg4_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9696,9 +9570,9 @@ define void @test_vsuxseg4_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9714,10 +9588,10 @@ define void @test_vsuxseg5_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9730,10 +9604,10 @@ define void @test_vsuxseg5_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9749,10 +9623,10 @@ define void @test_vsuxseg5_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9765,10 +9639,10 @@ define void @test_vsuxseg5_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9784,10 +9658,10 @@ define void @test_vsuxseg5_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9800,10 +9674,10 @@ define void @test_vsuxseg5_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9819,11 +9693,11 @@ define void @test_vsuxseg6_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9836,11 +9710,11 @@ define void @test_vsuxseg6_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9856,11 +9730,11 @@ define void @test_vsuxseg6_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9873,11 +9747,11 @@ define void @test_vsuxseg6_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9893,11 +9767,11 @@ define void @test_vsuxseg6_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9910,11 +9784,11 @@ define void @test_vsuxseg6_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9930,12 +9804,12 @@ define void @test_vsuxseg7_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9948,12 +9822,12 @@ define void @test_vsuxseg7_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9969,12 +9843,12 @@ define void @test_vsuxseg7_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9987,12 +9861,12 @@ define void @test_vsuxseg7_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10008,12 +9882,12 @@ define void @test_vsuxseg7_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10026,12 +9900,12 @@ define void @test_vsuxseg7_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10047,13 +9921,13 @@ define void @test_vsuxseg8_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10066,13 +9940,13 @@ define void @test_vsuxseg8_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10088,13 +9962,13 @@ define void @test_vsuxseg8_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10107,13 +9981,13 @@ define void @test_vsuxseg8_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10129,13 +10003,13 @@ define void @test_vsuxseg8_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10148,13 +10022,13 @@ define void @test_vsuxseg8_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10169,7 +10043,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f32.nxv1i8(<vscale x 1 x float>,<vsca
 define void @test_vsuxseg2_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -10183,7 +10056,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -10200,7 +10072,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f32.nxv1i32(<vscale x 1 x float>,<vsc
 define void @test_vsuxseg2_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -10214,7 +10085,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -10231,7 +10101,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f32.nxv1i16(<vscale x 1 x float>,<vsc
 define void @test_vsuxseg2_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -10245,7 +10114,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -10263,8 +10131,8 @@ define void @test_vsuxseg3_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10277,8 +10145,8 @@ define void @test_vsuxseg3_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10294,8 +10162,8 @@ define void @test_vsuxseg3_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10308,8 +10176,8 @@ define void @test_vsuxseg3_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10325,8 +10193,8 @@ define void @test_vsuxseg3_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10339,8 +10207,8 @@ define void @test_vsuxseg3_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10356,9 +10224,9 @@ define void @test_vsuxseg4_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10371,9 +10239,9 @@ define void @test_vsuxseg4_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10389,9 +10257,9 @@ define void @test_vsuxseg4_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10404,9 +10272,9 @@ define void @test_vsuxseg4_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10422,9 +10290,9 @@ define void @test_vsuxseg4_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10437,9 +10305,9 @@ define void @test_vsuxseg4_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10455,10 +10323,10 @@ define void @test_vsuxseg5_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10471,10 +10339,10 @@ define void @test_vsuxseg5_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10490,10 +10358,10 @@ define void @test_vsuxseg5_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10506,10 +10374,10 @@ define void @test_vsuxseg5_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10525,10 +10393,10 @@ define void @test_vsuxseg5_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10541,10 +10409,10 @@ define void @test_vsuxseg5_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10560,11 +10428,11 @@ define void @test_vsuxseg6_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10577,11 +10445,11 @@ define void @test_vsuxseg6_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10597,11 +10465,11 @@ define void @test_vsuxseg6_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10614,11 +10482,11 @@ define void @test_vsuxseg6_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10634,11 +10502,11 @@ define void @test_vsuxseg6_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10651,11 +10519,11 @@ define void @test_vsuxseg6_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10671,12 +10539,12 @@ define void @test_vsuxseg7_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10689,12 +10557,12 @@ define void @test_vsuxseg7_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10710,12 +10578,12 @@ define void @test_vsuxseg7_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10728,12 +10596,12 @@ define void @test_vsuxseg7_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10749,12 +10617,12 @@ define void @test_vsuxseg7_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10767,12 +10635,12 @@ define void @test_vsuxseg7_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10788,13 +10656,13 @@ define void @test_vsuxseg8_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10807,13 +10675,13 @@ define void @test_vsuxseg8_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10829,13 +10697,13 @@ define void @test_vsuxseg8_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10848,13 +10716,13 @@ define void @test_vsuxseg8_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10870,13 +10738,13 @@ define void @test_vsuxseg8_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10889,13 +10757,13 @@ define void @test_vsuxseg8_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10910,7 +10778,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8f16.nxv8i16(<vscale x 8 x half>,<vsca
 define void @test_vsuxseg2_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -10924,7 +10791,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -10941,7 +10807,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8f16.nxv8i8(<vscale x 8 x half>,<vscal
 define void @test_vsuxseg2_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -10955,7 +10820,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -10972,7 +10836,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8f16.nxv8i32(<vscale x 8 x half>,<vsca
 define void @test_vsuxseg2_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v12
@@ -10985,7 +10848,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v12, v0.t
@@ -11002,8 +10864,8 @@ define void @test_vsuxseg3_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11016,8 +10878,8 @@ define void @test_vsuxseg3_mask_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11033,8 +10895,8 @@ define void @test_vsuxseg3_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11047,8 +10909,8 @@ define void @test_vsuxseg3_mask_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11063,11 +10925,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv8f16.nxv8i32(<vscale x 8 x half>,<vsca
 define void @test_vsuxseg3_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v16, (a0), v12
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl)
@@ -11077,11 +10939,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl)
@@ -11095,9 +10957,9 @@ define void @test_vsuxseg4_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11110,9 +10972,9 @@ define void @test_vsuxseg4_mask_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11128,9 +10990,9 @@ define void @test_vsuxseg4_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11143,9 +11005,9 @@ define void @test_vsuxseg4_mask_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11161,9 +11023,9 @@ define void @test_vsuxseg4_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -11176,9 +11038,9 @@ define void @test_vsuxseg4_mask_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -11193,7 +11055,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8f32.nxv8i16(<vscale x 8 x float>,<vsc
 define void @test_vsuxseg2_nxv8f32_nxv8i16(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8f32_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -11207,7 +11068,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8f32_nxv8i16(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8f32_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -11224,7 +11084,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8f32.nxv8i8(<vscale x 8 x float>,<vsca
 define void @test_vsuxseg2_nxv8f32_nxv8i8(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8f32_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -11238,7 +11097,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8f32_nxv8i8(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8f32_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -11255,7 +11113,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8f32.nxv8i32(<vscale x 8 x float>,<vsc
 define void @test_vsuxseg2_nxv8f32_nxv8i32(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8f32_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -11269,7 +11126,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8f32_nxv8i32(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8f32_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -11286,7 +11142,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f64.nxv2i32(<vscale x 2 x double>,<vs
 define void @test_vsuxseg2_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11300,7 +11155,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11317,7 +11171,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f64.nxv2i8(<vscale x 2 x double>,<vsc
 define void @test_vsuxseg2_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11331,7 +11184,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11348,7 +11200,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f64.nxv2i16(<vscale x 2 x double>,<vs
 define void @test_vsuxseg2_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11362,7 +11213,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11380,8 +11230,8 @@ define void @test_vsuxseg3_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11394,8 +11244,8 @@ define void @test_vsuxseg3_mask_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11411,8 +11261,8 @@ define void @test_vsuxseg3_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11425,8 +11275,8 @@ define void @test_vsuxseg3_mask_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11442,8 +11292,8 @@ define void @test_vsuxseg3_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11456,8 +11306,8 @@ define void @test_vsuxseg3_mask_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11473,9 +11323,9 @@ define void @test_vsuxseg4_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11488,9 +11338,9 @@ define void @test_vsuxseg4_mask_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11506,9 +11356,9 @@ define void @test_vsuxseg4_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11521,9 +11371,9 @@ define void @test_vsuxseg4_mask_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11539,9 +11389,9 @@ define void @test_vsuxseg4_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11554,9 +11404,9 @@ define void @test_vsuxseg4_mask_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11571,7 +11421,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f16.nxv4i16(<vscale x 4 x half>,<vsca
 define void @test_vsuxseg2_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -11585,7 +11434,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -11602,7 +11450,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f16.nxv4i8(<vscale x 4 x half>,<vscal
 define void @test_vsuxseg2_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -11616,7 +11463,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -11633,7 +11479,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f16.nxv4i32(<vscale x 4 x half>,<vsca
 define void @test_vsuxseg2_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v10
@@ -11646,7 +11491,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v10, v0.t
@@ -11663,8 +11507,8 @@ define void @test_vsuxseg3_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11677,8 +11521,8 @@ define void @test_vsuxseg3_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11694,8 +11538,8 @@ define void @test_vsuxseg3_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11708,8 +11552,8 @@ define void @test_vsuxseg3_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11724,11 +11568,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv4f16.nxv4i32(<vscale x 4 x half>,<vsca
 define void @test_vsuxseg3_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl)
@@ -11738,11 +11582,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl)
@@ -11756,9 +11600,9 @@ define void @test_vsuxseg4_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11771,9 +11615,9 @@ define void @test_vsuxseg4_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11789,9 +11633,9 @@ define void @test_vsuxseg4_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11804,9 +11648,9 @@ define void @test_vsuxseg4_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11822,9 +11666,9 @@ define void @test_vsuxseg4_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11837,9 +11681,9 @@ define void @test_vsuxseg4_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11855,10 +11699,10 @@ define void @test_vsuxseg5_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11871,10 +11715,10 @@ define void @test_vsuxseg5_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11890,10 +11734,10 @@ define void @test_vsuxseg5_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11906,10 +11750,10 @@ define void @test_vsuxseg5_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11925,10 +11769,10 @@ define void @test_vsuxseg5_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11941,10 +11785,10 @@ define void @test_vsuxseg5_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11960,11 +11804,11 @@ define void @test_vsuxseg6_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11977,11 +11821,11 @@ define void @test_vsuxseg6_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11997,11 +11841,11 @@ define void @test_vsuxseg6_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12014,11 +11858,11 @@ define void @test_vsuxseg6_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12034,11 +11878,11 @@ define void @test_vsuxseg6_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -12051,11 +11895,11 @@ define void @test_vsuxseg6_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -12071,12 +11915,12 @@ define void @test_vsuxseg7_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12089,12 +11933,12 @@ define void @test_vsuxseg7_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12110,12 +11954,12 @@ define void @test_vsuxseg7_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12128,12 +11972,12 @@ define void @test_vsuxseg7_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12149,12 +11993,12 @@ define void @test_vsuxseg7_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -12167,12 +12011,12 @@ define void @test_vsuxseg7_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -12188,13 +12032,13 @@ define void @test_vsuxseg8_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12207,13 +12051,13 @@ define void @test_vsuxseg8_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12229,13 +12073,13 @@ define void @test_vsuxseg8_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12248,13 +12092,13 @@ define void @test_vsuxseg8_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12270,13 +12114,13 @@ define void @test_vsuxseg8_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -12289,13 +12133,13 @@ define void @test_vsuxseg8_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -12310,7 +12154,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f16.nxv2i32(<vscale x 2 x half>,<vsca
 define void @test_vsuxseg2_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -12324,7 +12167,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -12341,7 +12183,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f16.nxv2i8(<vscale x 2 x half>,<vscal
 define void @test_vsuxseg2_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -12355,7 +12196,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -12372,7 +12212,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f16.nxv2i16(<vscale x 2 x half>,<vsca
 define void @test_vsuxseg2_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -12386,7 +12225,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -12404,8 +12242,8 @@ define void @test_vsuxseg3_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12418,8 +12256,8 @@ define void @test_vsuxseg3_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12435,8 +12273,8 @@ define void @test_vsuxseg3_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12449,8 +12287,8 @@ define void @test_vsuxseg3_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12466,8 +12304,8 @@ define void @test_vsuxseg3_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12480,8 +12318,8 @@ define void @test_vsuxseg3_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12497,9 +12335,9 @@ define void @test_vsuxseg4_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12512,9 +12350,9 @@ define void @test_vsuxseg4_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12530,9 +12368,9 @@ define void @test_vsuxseg4_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12545,9 +12383,9 @@ define void @test_vsuxseg4_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12563,9 +12401,9 @@ define void @test_vsuxseg4_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12578,9 +12416,9 @@ define void @test_vsuxseg4_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12596,10 +12434,10 @@ define void @test_vsuxseg5_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12612,10 +12450,10 @@ define void @test_vsuxseg5_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12631,10 +12469,10 @@ define void @test_vsuxseg5_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12647,10 +12485,10 @@ define void @test_vsuxseg5_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12666,10 +12504,10 @@ define void @test_vsuxseg5_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12682,10 +12520,10 @@ define void @test_vsuxseg5_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12701,11 +12539,11 @@ define void @test_vsuxseg6_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12718,11 +12556,11 @@ define void @test_vsuxseg6_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12738,11 +12576,11 @@ define void @test_vsuxseg6_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12755,11 +12593,11 @@ define void @test_vsuxseg6_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12775,11 +12613,11 @@ define void @test_vsuxseg6_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12792,11 +12630,11 @@ define void @test_vsuxseg6_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12812,12 +12650,12 @@ define void @test_vsuxseg7_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12830,12 +12668,12 @@ define void @test_vsuxseg7_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12851,12 +12689,12 @@ define void @test_vsuxseg7_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12869,12 +12707,12 @@ define void @test_vsuxseg7_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12890,12 +12728,12 @@ define void @test_vsuxseg7_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12908,12 +12746,12 @@ define void @test_vsuxseg7_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12929,13 +12767,13 @@ define void @test_vsuxseg8_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12948,13 +12786,13 @@ define void @test_vsuxseg8_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12970,13 +12808,13 @@ define void @test_vsuxseg8_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12989,13 +12827,13 @@ define void @test_vsuxseg8_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13011,13 +12849,13 @@ define void @test_vsuxseg8_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13030,13 +12868,13 @@ define void @test_vsuxseg8_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13051,7 +12889,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f32.nxv4i16(<vscale x 4 x float>,<vsc
 define void @test_vsuxseg2_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -13065,7 +12902,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -13082,7 +12918,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f32.nxv4i8(<vscale x 4 x float>,<vsca
 define void @test_vsuxseg2_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -13096,7 +12931,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -13113,7 +12947,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f32.nxv4i32(<vscale x 4 x float>,<vsc
 define void @test_vsuxseg2_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -13127,7 +12960,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i32 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -13145,8 +12977,8 @@ define void @test_vsuxseg3_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13159,8 +12991,8 @@ define void @test_vsuxseg3_mask_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13176,8 +13008,8 @@ define void @test_vsuxseg3_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13190,8 +13022,8 @@ define void @test_vsuxseg3_mask_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13207,8 +13039,8 @@ define void @test_vsuxseg3_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13221,8 +13053,8 @@ define void @test_vsuxseg3_mask_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13238,9 +13070,9 @@ define void @test_vsuxseg4_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13253,9 +13085,9 @@ define void @test_vsuxseg4_mask_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13271,9 +13103,9 @@ define void @test_vsuxseg4_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13286,9 +13118,9 @@ define void @test_vsuxseg4_mask_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13304,9 +13136,9 @@ define void @test_vsuxseg4_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13319,9 +13151,9 @@ define void @test_vsuxseg4_mask_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/rvv/vsuxseg-rv64.ll b/llvm/test/CodeGen/RISCV/rvv/vsuxseg-rv64.ll
index f4670bbcd3e9a..9cd5c28092333 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vsuxseg-rv64.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vsuxseg-rv64.ll
@@ -8,7 +8,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16i16.nxv16i16(<vscale x 16 x i16>,<vs
 define void @test_vsuxseg2_nxv16i16_nxv16i16(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16i16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -22,7 +21,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16i16_nxv16i16(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16i16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -39,7 +37,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16i16.nxv16i8(<vscale x 16 x i16>,<vsc
 define void @test_vsuxseg2_nxv16i16_nxv16i8(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16i16_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -53,7 +50,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16i16_nxv16i8(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16i16_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -70,7 +66,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16i16.nxv16i32(<vscale x 16 x i16>,<vs
 define void @test_vsuxseg2_nxv16i16_nxv16i32(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16i16_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v16
@@ -83,7 +78,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16i16_nxv16i32(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16i16_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v16, v0.t
@@ -99,7 +93,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i32.nxv4i32(<vscale x 4 x i32>,<vscal
 define void @test_vsuxseg2_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -113,7 +106,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -130,7 +122,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i32.nxv4i8(<vscale x 4 x i32>,<vscale
 define void @test_vsuxseg2_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -144,7 +135,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -161,7 +151,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i32.nxv4i64(<vscale x 4 x i32>,<vscal
 define void @test_vsuxseg2_nxv4i32_nxv4i64(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v12
@@ -174,7 +163,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i32_nxv4i64(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v12, v0.t
@@ -190,7 +178,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i32.nxv4i16(<vscale x 4 x i32>,<vscal
 define void @test_vsuxseg2_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -204,7 +191,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -222,8 +208,8 @@ define void @test_vsuxseg3_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -236,8 +222,8 @@ define void @test_vsuxseg3_mask_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -253,8 +239,8 @@ define void @test_vsuxseg3_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg3_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -267,8 +253,8 @@ define void @test_vsuxseg3_mask_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -283,11 +269,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv4i32.nxv4i64(<vscale x 4 x i32>,<vscal
 define void @test_vsuxseg3_nxv4i32_nxv4i64(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv4i32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei64.v v16, (a0), v12
+; CHECK-NEXT:    vsuxseg3ei64.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv4i32.nxv4i64(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl)
@@ -297,11 +283,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv4i32_nxv4i64(<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei64.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsuxseg3ei64.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv4i32.nxv4i64(<vscale x 4 x i32> %val,<vscale x 4 x i32> %val,<vscale x 4 x i32> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl)
@@ -315,8 +301,8 @@ define void @test_vsuxseg3_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -329,8 +315,8 @@ define void @test_vsuxseg3_mask_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -346,9 +332,9 @@ define void @test_vsuxseg4_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -361,9 +347,9 @@ define void @test_vsuxseg4_mask_nxv4i32_nxv4i32(<vscale x 4 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -379,9 +365,9 @@ define void @test_vsuxseg4_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg4_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -394,9 +380,9 @@ define void @test_vsuxseg4_mask_nxv4i32_nxv4i8(<vscale x 4 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -412,9 +398,9 @@ define void @test_vsuxseg4_nxv4i32_nxv4i64(<vscale x 4 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv4i32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -427,9 +413,9 @@ define void @test_vsuxseg4_mask_nxv4i32_nxv4i64(<vscale x 4 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -445,9 +431,9 @@ define void @test_vsuxseg4_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -460,9 +446,9 @@ define void @test_vsuxseg4_mask_nxv4i32_nxv4i16(<vscale x 4 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -477,7 +463,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16i8.nxv16i16(<vscale x 16 x i8>,<vsca
 define void @test_vsuxseg2_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei16.v v8, (a0), v12
@@ -490,7 +475,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei16.v v8, (a0), v12, v0.t
@@ -506,7 +490,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16i8.nxv16i8(<vscale x 16 x i8>,<vscal
 define void @test_vsuxseg2_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -520,7 +503,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -537,7 +519,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16i8.nxv16i32(<vscale x 16 x i8>,<vsca
 define void @test_vsuxseg2_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v16
@@ -550,7 +531,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v16, v0.t
@@ -566,11 +546,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv16i8.nxv16i16(<vscale x 16 x i8>,<vsca
 define void @test_vsuxseg3_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei16.v v16, (a0), v12
+; CHECK-NEXT:    vsuxseg3ei16.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, i64 %vl)
@@ -580,11 +560,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei16.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsuxseg3ei16.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv16i8.nxv16i16(<vscale x 16 x i8> %val,<vscale x 16 x i8> %val,<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl)
@@ -598,8 +578,8 @@ define void @test_vsuxseg3_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -612,8 +592,8 @@ define void @test_vsuxseg3_mask_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -628,7 +608,6 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv16i8.nxv16i32(<vscale x 16 x i8>,<vsca
 define void @test_vsuxseg3_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -642,7 +621,6 @@ entry:
 define void @test_vsuxseg3_mask_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
@@ -660,9 +638,9 @@ define void @test_vsuxseg4_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -675,9 +653,9 @@ define void @test_vsuxseg4_mask_nxv16i8_nxv16i16(<vscale x 16 x i8> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv16i8_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -693,9 +671,9 @@ define void @test_vsuxseg4_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -708,9 +686,9 @@ define void @test_vsuxseg4_mask_nxv16i8_nxv16i8(<vscale x 16 x i8> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv16i8_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -725,7 +703,6 @@ declare void @llvm.riscv.vsuxseg4.mask.nxv16i8.nxv16i32(<vscale x 16 x i8>,<vsca
 define void @test_vsuxseg4_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -740,7 +717,6 @@ entry:
 define void @test_vsuxseg4_mask_nxv16i8_nxv16i32(<vscale x 16 x i8> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv16i8_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -758,7 +734,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i64.nxv1i64(<vscale x 1 x i64>,<vscal
 define void @test_vsuxseg2_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -772,7 +747,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -789,7 +763,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i64.nxv1i32(<vscale x 1 x i64>,<vscal
 define void @test_vsuxseg2_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -803,7 +776,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -820,7 +792,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i64.nxv1i16(<vscale x 1 x i64>,<vscal
 define void @test_vsuxseg2_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -834,7 +805,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -851,7 +821,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i64.nxv1i8(<vscale x 1 x i64>,<vscale
 define void @test_vsuxseg2_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -865,7 +834,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -883,8 +851,8 @@ define void @test_vsuxseg3_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -897,8 +865,8 @@ define void @test_vsuxseg3_mask_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -914,8 +882,8 @@ define void @test_vsuxseg3_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -928,8 +896,8 @@ define void @test_vsuxseg3_mask_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -945,8 +913,8 @@ define void @test_vsuxseg3_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -959,8 +927,8 @@ define void @test_vsuxseg3_mask_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -976,8 +944,8 @@ define void @test_vsuxseg3_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg3_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -990,8 +958,8 @@ define void @test_vsuxseg3_mask_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1007,9 +975,9 @@ define void @test_vsuxseg4_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1022,9 +990,9 @@ define void @test_vsuxseg4_mask_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1040,9 +1008,9 @@ define void @test_vsuxseg4_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1055,9 +1023,9 @@ define void @test_vsuxseg4_mask_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1073,9 +1041,9 @@ define void @test_vsuxseg4_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1088,9 +1056,9 @@ define void @test_vsuxseg4_mask_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1106,9 +1074,9 @@ define void @test_vsuxseg4_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg4_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1121,9 +1089,9 @@ define void @test_vsuxseg4_mask_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1139,10 +1107,10 @@ define void @test_vsuxseg5_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1155,10 +1123,10 @@ define void @test_vsuxseg5_mask_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1174,10 +1142,10 @@ define void @test_vsuxseg5_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1190,10 +1158,10 @@ define void @test_vsuxseg5_mask_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1209,10 +1177,10 @@ define void @test_vsuxseg5_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1225,10 +1193,10 @@ define void @test_vsuxseg5_mask_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1244,10 +1212,10 @@ define void @test_vsuxseg5_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg5_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1260,10 +1228,10 @@ define void @test_vsuxseg5_mask_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1279,11 +1247,11 @@ define void @test_vsuxseg6_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1296,11 +1264,11 @@ define void @test_vsuxseg6_mask_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1316,11 +1284,11 @@ define void @test_vsuxseg6_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1333,11 +1301,11 @@ define void @test_vsuxseg6_mask_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1353,11 +1321,11 @@ define void @test_vsuxseg6_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1370,11 +1338,11 @@ define void @test_vsuxseg6_mask_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1390,11 +1358,11 @@ define void @test_vsuxseg6_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg6_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1407,11 +1375,11 @@ define void @test_vsuxseg6_mask_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1427,12 +1395,12 @@ define void @test_vsuxseg7_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1445,12 +1413,12 @@ define void @test_vsuxseg7_mask_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1466,12 +1434,12 @@ define void @test_vsuxseg7_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1484,12 +1452,12 @@ define void @test_vsuxseg7_mask_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1505,12 +1473,12 @@ define void @test_vsuxseg7_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1523,12 +1491,12 @@ define void @test_vsuxseg7_mask_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1544,15 +1512,15 @@ define void @test_vsuxseg7_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg7_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
-; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
-; CHECK-NEXT:    ret
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
+; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
+; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg7.nxv1i64.nxv1i8(<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val,<vscale x 1 x i64> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl)
   ret void
@@ -1562,12 +1530,12 @@ define void @test_vsuxseg7_mask_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1583,13 +1551,13 @@ define void @test_vsuxseg8_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1602,13 +1570,13 @@ define void @test_vsuxseg8_mask_nxv1i64_nxv1i64(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1624,13 +1592,13 @@ define void @test_vsuxseg8_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1643,13 +1611,13 @@ define void @test_vsuxseg8_mask_nxv1i64_nxv1i32(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1665,13 +1633,13 @@ define void @test_vsuxseg8_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1684,13 +1652,13 @@ define void @test_vsuxseg8_mask_nxv1i64_nxv1i16(<vscale x 1 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1706,13 +1674,13 @@ define void @test_vsuxseg8_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg8_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1725,13 +1693,13 @@ define void @test_vsuxseg8_mask_nxv1i64_nxv1i8(<vscale x 1 x i64> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1746,7 +1714,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i32.nxv1i64(<vscale x 1 x i32>,<vscal
 define void @test_vsuxseg2_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -1760,7 +1727,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -1777,7 +1743,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i32.nxv1i32(<vscale x 1 x i32>,<vscal
 define void @test_vsuxseg2_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -1791,7 +1756,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -1808,7 +1772,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i32.nxv1i16(<vscale x 1 x i32>,<vscal
 define void @test_vsuxseg2_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -1822,7 +1785,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -1839,7 +1801,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i32.nxv1i8(<vscale x 1 x i32>,<vscale
 define void @test_vsuxseg2_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -1853,7 +1814,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -1871,8 +1831,8 @@ define void @test_vsuxseg3_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1885,8 +1845,8 @@ define void @test_vsuxseg3_mask_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1902,8 +1862,8 @@ define void @test_vsuxseg3_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1916,8 +1876,8 @@ define void @test_vsuxseg3_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1933,8 +1893,8 @@ define void @test_vsuxseg3_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1947,8 +1907,8 @@ define void @test_vsuxseg3_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1964,8 +1924,8 @@ define void @test_vsuxseg3_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg3_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -1978,8 +1938,8 @@ define void @test_vsuxseg3_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -1995,9 +1955,9 @@ define void @test_vsuxseg4_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2010,9 +1970,9 @@ define void @test_vsuxseg4_mask_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2028,9 +1988,9 @@ define void @test_vsuxseg4_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2043,9 +2003,9 @@ define void @test_vsuxseg4_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2061,9 +2021,9 @@ define void @test_vsuxseg4_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2076,9 +2036,9 @@ define void @test_vsuxseg4_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2094,9 +2054,9 @@ define void @test_vsuxseg4_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg4_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2109,9 +2069,9 @@ define void @test_vsuxseg4_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2127,10 +2087,10 @@ define void @test_vsuxseg5_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2143,10 +2103,10 @@ define void @test_vsuxseg5_mask_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2162,10 +2122,10 @@ define void @test_vsuxseg5_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2178,10 +2138,10 @@ define void @test_vsuxseg5_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2197,10 +2157,10 @@ define void @test_vsuxseg5_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2213,10 +2173,10 @@ define void @test_vsuxseg5_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2232,10 +2192,10 @@ define void @test_vsuxseg5_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg5_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2248,10 +2208,10 @@ define void @test_vsuxseg5_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2267,11 +2227,11 @@ define void @test_vsuxseg6_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2284,11 +2244,11 @@ define void @test_vsuxseg6_mask_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2304,11 +2264,11 @@ define void @test_vsuxseg6_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2321,11 +2281,11 @@ define void @test_vsuxseg6_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2341,11 +2301,11 @@ define void @test_vsuxseg6_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2358,11 +2318,11 @@ define void @test_vsuxseg6_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2378,11 +2338,11 @@ define void @test_vsuxseg6_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg6_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2395,11 +2355,11 @@ define void @test_vsuxseg6_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2415,12 +2375,12 @@ define void @test_vsuxseg7_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2433,12 +2393,12 @@ define void @test_vsuxseg7_mask_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2454,12 +2414,12 @@ define void @test_vsuxseg7_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2472,12 +2432,12 @@ define void @test_vsuxseg7_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2493,12 +2453,12 @@ define void @test_vsuxseg7_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2511,16 +2471,16 @@ define void @test_vsuxseg7_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
-; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    ret
-entry:
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
+; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
+; CHECK-NEXT:    ret
+entry:
   tail call void @llvm.riscv.vsuxseg7.mask.nxv1i32.nxv1i16(<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val,<vscale x 1 x i32> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl)
   ret void
 }
@@ -2532,12 +2492,12 @@ define void @test_vsuxseg7_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg7_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2550,12 +2510,12 @@ define void @test_vsuxseg7_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2571,13 +2531,13 @@ define void @test_vsuxseg8_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2590,13 +2550,13 @@ define void @test_vsuxseg8_mask_nxv1i32_nxv1i64(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2612,13 +2572,13 @@ define void @test_vsuxseg8_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2631,13 +2591,13 @@ define void @test_vsuxseg8_mask_nxv1i32_nxv1i32(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2653,13 +2613,13 @@ define void @test_vsuxseg8_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2672,13 +2632,13 @@ define void @test_vsuxseg8_mask_nxv1i32_nxv1i16(<vscale x 1 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2694,13 +2654,13 @@ define void @test_vsuxseg8_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg8_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -2713,13 +2673,13 @@ define void @test_vsuxseg8_mask_nxv1i32_nxv1i8(<vscale x 1 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -2734,7 +2694,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i16.nxv8i16(<vscale x 8 x i16>,<vscal
 define void @test_vsuxseg2_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -2748,7 +2707,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -2765,7 +2723,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i16.nxv8i8(<vscale x 8 x i16>,<vscale
 define void @test_vsuxseg2_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -2779,7 +2736,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -2796,7 +2752,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i16.nxv8i64(<vscale x 8 x i16>,<vscal
 define void @test_vsuxseg2_nxv8i16_nxv8i64(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v16
@@ -2809,7 +2764,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i16_nxv8i64(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v16, v0.t
@@ -2825,7 +2779,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i16.nxv8i32(<vscale x 8 x i16>,<vscal
 define void @test_vsuxseg2_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v12
@@ -2838,7 +2791,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v12, v0.t
@@ -2855,8 +2807,8 @@ define void @test_vsuxseg3_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -2869,8 +2821,8 @@ define void @test_vsuxseg3_mask_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -2886,8 +2838,8 @@ define void @test_vsuxseg3_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg3_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -2900,8 +2852,8 @@ define void @test_vsuxseg3_mask_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -2916,7 +2868,6 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv8i16.nxv8i64(<vscale x 8 x i16>,<vscal
 define void @test_vsuxseg3_nxv8i16_nxv8i64(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv8i16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -2930,7 +2881,6 @@ entry:
 define void @test_vsuxseg3_mask_nxv8i16_nxv8i64(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8i16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -2947,11 +2897,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv8i16.nxv8i32(<vscale x 8 x i16>,<vscal
 define void @test_vsuxseg3_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v16, (a0), v12
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl)
@@ -2961,11 +2911,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv8i16.nxv8i32(<vscale x 8 x i16> %val,<vscale x 8 x i16> %val,<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl)
@@ -2979,9 +2929,9 @@ define void @test_vsuxseg4_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -2994,9 +2944,9 @@ define void @test_vsuxseg4_mask_nxv8i16_nxv8i16(<vscale x 8 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8i16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3012,9 +2962,9 @@ define void @test_vsuxseg4_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg4_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3027,9 +2977,9 @@ define void @test_vsuxseg4_mask_nxv8i16_nxv8i8(<vscale x 8 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8i16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3044,7 +2994,6 @@ declare void @llvm.riscv.vsuxseg4.mask.nxv8i16.nxv8i64(<vscale x 8 x i16>,<vscal
 define void @test_vsuxseg4_nxv8i16_nxv8i64(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_nxv8i16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -3059,7 +3008,6 @@ entry:
 define void @test_vsuxseg4_mask_nxv8i16_nxv8i64(<vscale x 8 x i16> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8i16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -3078,9 +3026,9 @@ define void @test_vsuxseg4_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -3093,9 +3041,9 @@ define void @test_vsuxseg4_mask_nxv8i16_nxv8i32(<vscale x 8 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8i16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -3110,7 +3058,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i8.nxv4i32(<vscale x 4 x i8>,<vscale
 define void @test_vsuxseg2_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v10
@@ -3123,7 +3070,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v10, v0.t
@@ -3139,7 +3085,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i8.nxv4i8(<vscale x 4 x i8>,<vscale x
 define void @test_vsuxseg2_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -3153,7 +3098,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -3170,7 +3114,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i8.nxv4i64(<vscale x 4 x i8>,<vscale
 define void @test_vsuxseg2_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v12
@@ -3183,7 +3126,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v12, v0.t
@@ -3199,7 +3141,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i8.nxv4i16(<vscale x 4 x i8>,<vscale
 define void @test_vsuxseg2_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -3213,7 +3154,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -3230,11 +3170,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv4i8.nxv4i32(<vscale x 4 x i8>,<vscale
 define void @test_vsuxseg3_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl)
@@ -3244,11 +3184,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv4i8.nxv4i32(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl)
@@ -3262,8 +3202,8 @@ define void @test_vsuxseg3_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg3_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3276,8 +3216,8 @@ define void @test_vsuxseg3_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3292,7 +3232,6 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv4i8.nxv4i64(<vscale x 4 x i8>,<vscale
 define void @test_vsuxseg3_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -3306,7 +3245,6 @@ entry:
 define void @test_vsuxseg3_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
@@ -3324,8 +3262,8 @@ define void @test_vsuxseg3_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg3_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3338,8 +3276,8 @@ define void @test_vsuxseg3_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3355,9 +3293,9 @@ define void @test_vsuxseg4_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg4_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3370,9 +3308,9 @@ define void @test_vsuxseg4_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3388,9 +3326,9 @@ define void @test_vsuxseg4_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg4_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3403,9 +3341,9 @@ define void @test_vsuxseg4_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3420,7 +3358,6 @@ declare void @llvm.riscv.vsuxseg4.mask.nxv4i8.nxv4i64(<vscale x 4 x i8>,<vscale
 define void @test_vsuxseg4_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3435,7 +3372,6 @@ entry:
 define void @test_vsuxseg4_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -3454,9 +3390,9 @@ define void @test_vsuxseg4_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg4_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3469,9 +3405,9 @@ define void @test_vsuxseg4_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3487,10 +3423,10 @@ define void @test_vsuxseg5_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg5_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3503,10 +3439,10 @@ define void @test_vsuxseg5_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3522,10 +3458,10 @@ define void @test_vsuxseg5_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg5_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3538,10 +3474,10 @@ define void @test_vsuxseg5_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3556,13 +3492,13 @@ declare void @llvm.riscv.vsuxseg5.mask.nxv4i8.nxv4i64(<vscale x 4 x i8>,<vscale
 define void @test_vsuxseg5_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg5_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
-; CHECK-NEXT:    vsuxseg5ei64.v v16, (a0), v12
+; CHECK-NEXT:    vsuxseg5ei64.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg5.nxv4i8.nxv4i64(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl)
@@ -3572,13 +3508,13 @@ entry:
 define void @test_vsuxseg5_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
-; CHECK-NEXT:    vsuxseg5ei64.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsuxseg5ei64.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg5.mask.nxv4i8.nxv4i64(<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val,<vscale x 4 x i8> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl)
@@ -3592,10 +3528,10 @@ define void @test_vsuxseg5_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg5_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3608,10 +3544,10 @@ define void @test_vsuxseg5_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3627,11 +3563,11 @@ define void @test_vsuxseg6_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3644,11 +3580,11 @@ define void @test_vsuxseg6_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3664,11 +3600,11 @@ define void @test_vsuxseg6_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg6_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3681,11 +3617,11 @@ define void @test_vsuxseg6_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3701,11 +3637,11 @@ define void @test_vsuxseg6_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -3718,11 +3654,11 @@ define void @test_vsuxseg6_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -3738,11 +3674,11 @@ define void @test_vsuxseg6_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3755,11 +3691,11 @@ define void @test_vsuxseg6_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3775,12 +3711,12 @@ define void @test_vsuxseg7_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3793,12 +3729,12 @@ define void @test_vsuxseg7_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3814,12 +3750,12 @@ define void @test_vsuxseg7_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg7_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3832,12 +3768,12 @@ define void @test_vsuxseg7_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3853,12 +3789,12 @@ define void @test_vsuxseg7_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -3871,12 +3807,12 @@ define void @test_vsuxseg7_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -3892,12 +3828,12 @@ define void @test_vsuxseg7_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3910,12 +3846,12 @@ define void @test_vsuxseg7_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -3931,13 +3867,13 @@ define void @test_vsuxseg8_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -3950,13 +3886,13 @@ define void @test_vsuxseg8_mask_nxv4i8_nxv4i32(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4i8_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -3972,13 +3908,13 @@ define void @test_vsuxseg8_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg8_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -3991,13 +3927,13 @@ define void @test_vsuxseg8_mask_nxv4i8_nxv4i8(<vscale x 4 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4i8_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4013,13 +3949,13 @@ define void @test_vsuxseg8_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -4032,13 +3968,13 @@ define void @test_vsuxseg8_mask_nxv4i8_nxv4i64(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4i8_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -4054,13 +3990,13 @@ define void @test_vsuxseg8_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4073,13 +4009,13 @@ define void @test_vsuxseg8_mask_nxv4i8_nxv4i16(<vscale x 4 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4i8_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4094,7 +4030,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i16.nxv1i64(<vscale x 1 x i16>,<vscal
 define void @test_vsuxseg2_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -4108,7 +4043,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -4125,7 +4059,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i16.nxv1i32(<vscale x 1 x i16>,<vscal
 define void @test_vsuxseg2_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -4139,7 +4072,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -4156,7 +4088,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i16.nxv1i16(<vscale x 1 x i16>,<vscal
 define void @test_vsuxseg2_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -4170,7 +4101,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -4187,7 +4117,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i16.nxv1i8(<vscale x 1 x i16>,<vscale
 define void @test_vsuxseg2_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -4201,7 +4130,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -4219,8 +4147,8 @@ define void @test_vsuxseg3_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4233,8 +4161,8 @@ define void @test_vsuxseg3_mask_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4250,8 +4178,8 @@ define void @test_vsuxseg3_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4264,8 +4192,8 @@ define void @test_vsuxseg3_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4281,8 +4209,8 @@ define void @test_vsuxseg3_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4295,8 +4223,8 @@ define void @test_vsuxseg3_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4312,8 +4240,8 @@ define void @test_vsuxseg3_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg3_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4326,8 +4254,8 @@ define void @test_vsuxseg3_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4343,9 +4271,9 @@ define void @test_vsuxseg4_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4358,9 +4286,9 @@ define void @test_vsuxseg4_mask_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4376,9 +4304,9 @@ define void @test_vsuxseg4_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4391,9 +4319,9 @@ define void @test_vsuxseg4_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4409,9 +4337,9 @@ define void @test_vsuxseg4_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4424,9 +4352,9 @@ define void @test_vsuxseg4_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4442,9 +4370,9 @@ define void @test_vsuxseg4_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg4_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4457,9 +4385,9 @@ define void @test_vsuxseg4_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4475,10 +4403,10 @@ define void @test_vsuxseg5_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4491,10 +4419,10 @@ define void @test_vsuxseg5_mask_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4510,10 +4438,10 @@ define void @test_vsuxseg5_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4526,10 +4454,10 @@ define void @test_vsuxseg5_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4545,10 +4473,10 @@ define void @test_vsuxseg5_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4561,10 +4489,10 @@ define void @test_vsuxseg5_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4580,10 +4508,10 @@ define void @test_vsuxseg5_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg5_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4596,10 +4524,10 @@ define void @test_vsuxseg5_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4615,11 +4543,11 @@ define void @test_vsuxseg6_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4632,11 +4560,11 @@ define void @test_vsuxseg6_mask_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4652,11 +4580,11 @@ define void @test_vsuxseg6_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4669,11 +4597,11 @@ define void @test_vsuxseg6_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4689,11 +4617,11 @@ define void @test_vsuxseg6_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4706,11 +4634,11 @@ define void @test_vsuxseg6_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4726,11 +4654,11 @@ define void @test_vsuxseg6_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg6_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4743,11 +4671,11 @@ define void @test_vsuxseg6_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4763,12 +4691,12 @@ define void @test_vsuxseg7_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4781,12 +4709,12 @@ define void @test_vsuxseg7_mask_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4802,12 +4730,12 @@ define void @test_vsuxseg7_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4820,12 +4748,12 @@ define void @test_vsuxseg7_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4841,12 +4769,12 @@ define void @test_vsuxseg7_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4859,12 +4787,12 @@ define void @test_vsuxseg7_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4880,12 +4808,12 @@ define void @test_vsuxseg7_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg7_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4898,16 +4826,16 @@ define void @test_vsuxseg7_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
-; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
-; CHECK-NEXT:    ret
-entry:
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
+; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
+; CHECK-NEXT:    ret
+entry:
   tail call void @llvm.riscv.vsuxseg7.mask.nxv1i16.nxv1i8(<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val,<vscale x 1 x i16> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl)
   ret void
 }
@@ -4919,13 +4847,13 @@ define void @test_vsuxseg8_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4938,13 +4866,13 @@ define void @test_vsuxseg8_mask_nxv1i16_nxv1i64(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -4960,13 +4888,13 @@ define void @test_vsuxseg8_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -4979,13 +4907,13 @@ define void @test_vsuxseg8_mask_nxv1i16_nxv1i32(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5001,13 +4929,13 @@ define void @test_vsuxseg8_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5020,13 +4948,13 @@ define void @test_vsuxseg8_mask_nxv1i16_nxv1i16(<vscale x 1 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5042,13 +4970,13 @@ define void @test_vsuxseg8_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg8_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5061,13 +4989,13 @@ define void @test_vsuxseg8_mask_nxv1i16_nxv1i8(<vscale x 1 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5082,7 +5010,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i32.nxv2i32(<vscale x 2 x i32>,<vscal
 define void @test_vsuxseg2_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -5096,7 +5023,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -5113,7 +5039,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i32.nxv2i8(<vscale x 2 x i32>,<vscale
 define void @test_vsuxseg2_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -5127,7 +5052,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -5144,7 +5068,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i32.nxv2i16(<vscale x 2 x i32>,<vscal
 define void @test_vsuxseg2_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -5158,7 +5081,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -5175,7 +5097,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i32.nxv2i64(<vscale x 2 x i32>,<vscal
 define void @test_vsuxseg2_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v10
@@ -5188,7 +5109,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v10, v0.t
@@ -5205,8 +5125,8 @@ define void @test_vsuxseg3_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5219,8 +5139,8 @@ define void @test_vsuxseg3_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5236,8 +5156,8 @@ define void @test_vsuxseg3_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg3_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5250,8 +5170,8 @@ define void @test_vsuxseg3_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5267,8 +5187,8 @@ define void @test_vsuxseg3_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5281,8 +5201,8 @@ define void @test_vsuxseg3_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5297,11 +5217,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv2i32.nxv2i64(<vscale x 2 x i32>,<vscal
 define void @test_vsuxseg3_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
-; CHECK-NEXT:    vsuxseg3ei64.v v12, (a0), v10
+; CHECK-NEXT:    vsuxseg3ei64.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv2i32.nxv2i64(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl)
@@ -5311,11 +5231,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
-; CHECK-NEXT:    vsuxseg3ei64.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsuxseg3ei64.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv2i32.nxv2i64(<vscale x 2 x i32> %val,<vscale x 2 x i32> %val,<vscale x 2 x i32> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl)
@@ -5329,9 +5249,9 @@ define void @test_vsuxseg4_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5344,9 +5264,9 @@ define void @test_vsuxseg4_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5362,9 +5282,9 @@ define void @test_vsuxseg4_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg4_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5377,9 +5297,9 @@ define void @test_vsuxseg4_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5395,9 +5315,9 @@ define void @test_vsuxseg4_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5410,9 +5330,9 @@ define void @test_vsuxseg4_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5428,9 +5348,9 @@ define void @test_vsuxseg4_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -5443,9 +5363,9 @@ define void @test_vsuxseg4_mask_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -5461,10 +5381,10 @@ define void @test_vsuxseg5_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5477,10 +5397,10 @@ define void @test_vsuxseg5_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5496,10 +5416,10 @@ define void @test_vsuxseg5_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg5_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5512,10 +5432,10 @@ define void @test_vsuxseg5_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5531,10 +5451,10 @@ define void @test_vsuxseg5_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5547,10 +5467,10 @@ define void @test_vsuxseg5_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5566,10 +5486,10 @@ define void @test_vsuxseg5_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -5582,10 +5502,10 @@ define void @test_vsuxseg5_mask_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -5601,11 +5521,11 @@ define void @test_vsuxseg6_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5618,11 +5538,11 @@ define void @test_vsuxseg6_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5638,11 +5558,11 @@ define void @test_vsuxseg6_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg6_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5655,11 +5575,11 @@ define void @test_vsuxseg6_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5675,11 +5595,11 @@ define void @test_vsuxseg6_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5692,11 +5612,11 @@ define void @test_vsuxseg6_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5712,11 +5632,11 @@ define void @test_vsuxseg6_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -5729,11 +5649,11 @@ define void @test_vsuxseg6_mask_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -5749,12 +5669,12 @@ define void @test_vsuxseg7_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5767,12 +5687,12 @@ define void @test_vsuxseg7_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5788,12 +5708,12 @@ define void @test_vsuxseg7_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg7_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5806,12 +5726,12 @@ define void @test_vsuxseg7_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5827,12 +5747,12 @@ define void @test_vsuxseg7_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5845,12 +5765,12 @@ define void @test_vsuxseg7_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5866,12 +5786,12 @@ define void @test_vsuxseg7_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -5884,12 +5804,12 @@ define void @test_vsuxseg7_mask_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -5905,13 +5825,13 @@ define void @test_vsuxseg8_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5924,13 +5844,13 @@ define void @test_vsuxseg8_mask_nxv2i32_nxv2i32(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5946,13 +5866,13 @@ define void @test_vsuxseg8_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg8_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -5965,13 +5885,13 @@ define void @test_vsuxseg8_mask_nxv2i32_nxv2i8(<vscale x 2 x i32> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -5987,13 +5907,13 @@ define void @test_vsuxseg8_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6006,13 +5926,13 @@ define void @test_vsuxseg8_mask_nxv2i32_nxv2i16(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6028,13 +5948,13 @@ define void @test_vsuxseg8_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -6047,13 +5967,13 @@ define void @test_vsuxseg8_mask_nxv2i32_nxv2i64(<vscale x 2 x i32> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -6068,7 +5988,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i8.nxv8i16(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg2_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei16.v v8, (a0), v10
@@ -6081,7 +6000,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei16.v v8, (a0), v10, v0.t
@@ -6097,7 +6015,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i8.nxv8i8(<vscale x 8 x i8>,<vscale x
 define void @test_vsuxseg2_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -6111,7 +6028,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -6128,7 +6044,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i8.nxv8i64(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg2_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v16
@@ -6141,7 +6056,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v16, v0.t
@@ -6157,7 +6071,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i8.nxv8i32(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg2_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v12
@@ -6170,7 +6083,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v12, v0.t
@@ -6186,11 +6098,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv8i8.nxv8i16(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg3_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
-; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10
+; CHECK-NEXT:    vsuxseg3ei16.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl)
@@ -6200,11 +6112,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
-; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsuxseg3ei16.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv8i8.nxv8i16(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl)
@@ -6218,8 +6130,8 @@ define void @test_vsuxseg3_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg3_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6232,8 +6144,8 @@ define void @test_vsuxseg3_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6248,7 +6160,6 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv8i8.nxv8i64(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg3_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -6262,7 +6173,6 @@ entry:
 define void @test_vsuxseg3_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -6279,7 +6189,6 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv8i8.nxv8i32(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg3_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -6293,7 +6202,6 @@ entry:
 define void @test_vsuxseg3_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
@@ -6311,9 +6219,9 @@ define void @test_vsuxseg4_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg4_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -6326,9 +6234,9 @@ define void @test_vsuxseg4_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -6344,9 +6252,9 @@ define void @test_vsuxseg4_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg4_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6359,9 +6267,9 @@ define void @test_vsuxseg4_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6376,7 +6284,6 @@ declare void @llvm.riscv.vsuxseg4.mask.nxv8i8.nxv8i64(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg4_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6391,7 +6298,6 @@ entry:
 define void @test_vsuxseg4_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6409,7 +6315,6 @@ declare void @llvm.riscv.vsuxseg4.mask.nxv8i8.nxv8i32(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg4_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6424,7 +6329,6 @@ entry:
 define void @test_vsuxseg4_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6443,10 +6347,10 @@ define void @test_vsuxseg5_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg5_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -6459,10 +6363,10 @@ define void @test_vsuxseg5_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -6478,10 +6382,10 @@ define void @test_vsuxseg5_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg5_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6494,10 +6398,10 @@ define void @test_vsuxseg5_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6512,7 +6416,6 @@ declare void @llvm.riscv.vsuxseg5.mask.nxv8i8.nxv8i64(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg5_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg5_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6528,7 +6431,6 @@ entry:
 define void @test_vsuxseg5_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6547,13 +6449,13 @@ declare void @llvm.riscv.vsuxseg5.mask.nxv8i8.nxv8i32(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg5_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg5_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
-; CHECK-NEXT:    vsuxseg5ei32.v v16, (a0), v12
+; CHECK-NEXT:    vsuxseg5ei32.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg5.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl)
@@ -6563,13 +6465,13 @@ entry:
 define void @test_vsuxseg5_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
-; CHECK-NEXT:    vsuxseg5ei32.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsuxseg5ei32.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg5.mask.nxv8i8.nxv8i32(<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val,<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl)
@@ -6583,11 +6485,11 @@ define void @test_vsuxseg6_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -6600,11 +6502,11 @@ define void @test_vsuxseg6_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -6620,11 +6522,11 @@ define void @test_vsuxseg6_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg6_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6637,11 +6539,11 @@ define void @test_vsuxseg6_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6656,7 +6558,6 @@ declare void @llvm.riscv.vsuxseg6.mask.nxv8i8.nxv8i64(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg6_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg6_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6673,7 +6574,6 @@ entry:
 define void @test_vsuxseg6_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6694,11 +6594,11 @@ define void @test_vsuxseg6_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -6711,11 +6611,11 @@ define void @test_vsuxseg6_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -6731,12 +6631,12 @@ define void @test_vsuxseg7_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -6749,12 +6649,12 @@ define void @test_vsuxseg7_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -6770,12 +6670,12 @@ define void @test_vsuxseg7_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg7_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6788,12 +6688,12 @@ define void @test_vsuxseg7_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6808,7 +6708,6 @@ declare void @llvm.riscv.vsuxseg7.mask.nxv8i8.nxv8i64(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg7_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg7_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6826,7 +6725,6 @@ entry:
 define void @test_vsuxseg7_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6848,12 +6746,12 @@ define void @test_vsuxseg7_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -6866,12 +6764,12 @@ define void @test_vsuxseg7_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -6887,13 +6785,13 @@ define void @test_vsuxseg8_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -6906,13 +6804,13 @@ define void @test_vsuxseg8_mask_nxv8i8_nxv8i16(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv8i8_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -6928,13 +6826,13 @@ define void @test_vsuxseg8_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg8_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -6947,13 +6845,13 @@ define void @test_vsuxseg8_mask_nxv8i8_nxv8i8(<vscale x 8 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv8i8_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -6968,7 +6866,6 @@ declare void @llvm.riscv.vsuxseg8.mask.nxv8i8.nxv8i64(<vscale x 8 x i8>,<vscale
 define void @test_vsuxseg8_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg8_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -6987,7 +6884,6 @@ entry:
 define void @test_vsuxseg8_mask_nxv8i8_nxv8i64(<vscale x 8 x i8> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv8i8_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11_v12_v13_v14_v15
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -7010,13 +6906,13 @@ define void @test_vsuxseg8_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -7029,13 +6925,13 @@ define void @test_vsuxseg8_mask_nxv8i8_nxv8i32(<vscale x 8 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv8i8_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -7050,7 +6946,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i64.nxv4i32(<vscale x 4 x i64>,<vscal
 define void @test_vsuxseg2_nxv4i64_nxv4i32(<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i64_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7064,7 +6959,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i64_nxv4i32(<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i64_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7081,7 +6975,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i64.nxv4i8(<vscale x 4 x i64>,<vscale
 define void @test_vsuxseg2_nxv4i64_nxv4i8(<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i64_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7095,7 +6988,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i64_nxv4i8(<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i64_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7112,7 +7004,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i64.nxv4i64(<vscale x 4 x i64>,<vscal
 define void @test_vsuxseg2_nxv4i64_nxv4i64(<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i64_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7126,7 +7017,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i64_nxv4i64(<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i64_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7143,7 +7033,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i64.nxv4i16(<vscale x 4 x i64>,<vscal
 define void @test_vsuxseg2_nxv4i64_nxv4i16(<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i64_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7157,7 +7046,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i64_nxv4i16(<vscale x 4 x i64> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i64_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -7174,7 +7062,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i16.nxv4i32(<vscale x 4 x i16>,<vscal
 define void @test_vsuxseg2_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v10
@@ -7187,7 +7074,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v10, v0.t
@@ -7203,7 +7089,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i16.nxv4i8(<vscale x 4 x i16>,<vscale
 define void @test_vsuxseg2_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -7217,7 +7102,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -7234,7 +7118,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i16.nxv4i64(<vscale x 4 x i16>,<vscal
 define void @test_vsuxseg2_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v12
@@ -7247,7 +7130,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v12, v0.t
@@ -7263,7 +7145,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4i16.nxv4i16(<vscale x 4 x i16>,<vscal
 define void @test_vsuxseg2_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -7277,7 +7158,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -7294,11 +7174,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv4i16.nxv4i32(<vscale x 4 x i16>,<vscal
 define void @test_vsuxseg3_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl)
@@ -7308,11 +7188,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv4i16.nxv4i32(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl)
@@ -7326,8 +7206,8 @@ define void @test_vsuxseg3_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg3_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7340,8 +7220,8 @@ define void @test_vsuxseg3_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7356,7 +7236,6 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv4i16.nxv4i64(<vscale x 4 x i16>,<vscal
 define void @test_vsuxseg3_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -7370,7 +7249,6 @@ entry:
 define void @test_vsuxseg3_mask_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -7388,8 +7266,8 @@ define void @test_vsuxseg3_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7402,8 +7280,8 @@ define void @test_vsuxseg3_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7419,9 +7297,9 @@ define void @test_vsuxseg4_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7434,9 +7312,9 @@ define void @test_vsuxseg4_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7452,9 +7330,9 @@ define void @test_vsuxseg4_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg4_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7467,9 +7345,9 @@ define void @test_vsuxseg4_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7484,7 +7362,6 @@ declare void @llvm.riscv.vsuxseg4.mask.nxv4i16.nxv4i64(<vscale x 4 x i16>,<vscal
 define void @test_vsuxseg4_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -7499,7 +7376,6 @@ entry:
 define void @test_vsuxseg4_mask_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -7518,9 +7394,9 @@ define void @test_vsuxseg4_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7533,9 +7409,9 @@ define void @test_vsuxseg4_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7551,10 +7427,10 @@ define void @test_vsuxseg5_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7567,10 +7443,10 @@ define void @test_vsuxseg5_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7586,10 +7462,10 @@ define void @test_vsuxseg5_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg5_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7602,10 +7478,10 @@ define void @test_vsuxseg5_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7620,13 +7496,13 @@ declare void @llvm.riscv.vsuxseg5.mask.nxv4i16.nxv4i64(<vscale x 4 x i16>,<vscal
 define void @test_vsuxseg5_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg5_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsuxseg5ei64.v v16, (a0), v12
+; CHECK-NEXT:    vsuxseg5ei64.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg5.nxv4i16.nxv4i64(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl)
@@ -7636,13 +7512,13 @@ entry:
 define void @test_vsuxseg5_mask_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsuxseg5ei64.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsuxseg5ei64.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg5.mask.nxv4i16.nxv4i64(<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val,<vscale x 4 x i16> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl)
@@ -7656,10 +7532,10 @@ define void @test_vsuxseg5_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7672,10 +7548,10 @@ define void @test_vsuxseg5_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7691,11 +7567,11 @@ define void @test_vsuxseg6_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7708,11 +7584,11 @@ define void @test_vsuxseg6_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7728,11 +7604,11 @@ define void @test_vsuxseg6_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg6_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7745,11 +7621,11 @@ define void @test_vsuxseg6_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7765,11 +7641,11 @@ define void @test_vsuxseg6_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -7782,11 +7658,11 @@ define void @test_vsuxseg6_mask_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -7802,11 +7678,11 @@ define void @test_vsuxseg6_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7819,11 +7695,11 @@ define void @test_vsuxseg6_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7839,12 +7715,12 @@ define void @test_vsuxseg7_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -7857,12 +7733,12 @@ define void @test_vsuxseg7_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -7878,12 +7754,12 @@ define void @test_vsuxseg7_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg7_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7896,12 +7772,12 @@ define void @test_vsuxseg7_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7917,12 +7793,12 @@ define void @test_vsuxseg7_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -7935,12 +7811,12 @@ define void @test_vsuxseg7_mask_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -7956,12 +7832,12 @@ define void @test_vsuxseg7_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -7974,12 +7850,12 @@ define void @test_vsuxseg7_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -7995,13 +7871,13 @@ define void @test_vsuxseg8_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -8014,13 +7890,13 @@ define void @test_vsuxseg8_mask_nxv4i16_nxv4i32(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4i16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -8036,13 +7912,13 @@ define void @test_vsuxseg8_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg8_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8055,13 +7931,13 @@ define void @test_vsuxseg8_mask_nxv4i16_nxv4i8(<vscale x 4 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4i16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8077,13 +7953,13 @@ define void @test_vsuxseg8_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -8096,13 +7972,13 @@ define void @test_vsuxseg8_mask_nxv4i16_nxv4i64(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4i16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -8118,13 +7994,13 @@ define void @test_vsuxseg8_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8137,13 +8013,13 @@ define void @test_vsuxseg8_mask_nxv4i16_nxv4i16(<vscale x 4 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4i16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8158,7 +8034,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i8.nxv1i64(<vscale x 1 x i8>,<vscale
 define void @test_vsuxseg2_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -8172,7 +8047,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -8189,7 +8063,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i8.nxv1i32(<vscale x 1 x i8>,<vscale
 define void @test_vsuxseg2_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -8203,7 +8076,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -8220,7 +8092,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i8.nxv1i16(<vscale x 1 x i8>,<vscale
 define void @test_vsuxseg2_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -8234,7 +8105,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -8251,7 +8121,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1i8.nxv1i8(<vscale x 1 x i8>,<vscale x
 define void @test_vsuxseg2_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -8265,7 +8134,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
@@ -8283,8 +8151,8 @@ define void @test_vsuxseg3_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg3_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8297,8 +8165,8 @@ define void @test_vsuxseg3_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8314,8 +8182,8 @@ define void @test_vsuxseg3_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg3_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8328,8 +8196,8 @@ define void @test_vsuxseg3_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8345,8 +8213,8 @@ define void @test_vsuxseg3_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg3_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8359,8 +8227,8 @@ define void @test_vsuxseg3_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8376,8 +8244,8 @@ define void @test_vsuxseg3_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg3_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8390,8 +8258,8 @@ define void @test_vsuxseg3_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8407,9 +8275,9 @@ define void @test_vsuxseg4_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg4_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8422,9 +8290,9 @@ define void @test_vsuxseg4_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8440,9 +8308,9 @@ define void @test_vsuxseg4_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg4_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8455,9 +8323,9 @@ define void @test_vsuxseg4_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8473,9 +8341,9 @@ define void @test_vsuxseg4_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg4_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8488,9 +8356,9 @@ define void @test_vsuxseg4_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8506,9 +8374,9 @@ define void @test_vsuxseg4_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg4_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8521,9 +8389,9 @@ define void @test_vsuxseg4_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8539,10 +8407,10 @@ define void @test_vsuxseg5_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg5_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8555,10 +8423,10 @@ define void @test_vsuxseg5_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8574,10 +8442,10 @@ define void @test_vsuxseg5_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg5_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8590,10 +8458,10 @@ define void @test_vsuxseg5_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8609,10 +8477,10 @@ define void @test_vsuxseg5_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg5_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8625,10 +8493,10 @@ define void @test_vsuxseg5_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8644,10 +8512,10 @@ define void @test_vsuxseg5_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg5_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8660,10 +8528,10 @@ define void @test_vsuxseg5_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8679,11 +8547,11 @@ define void @test_vsuxseg6_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8696,11 +8564,11 @@ define void @test_vsuxseg6_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8716,11 +8584,11 @@ define void @test_vsuxseg6_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8733,11 +8601,11 @@ define void @test_vsuxseg6_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8753,11 +8621,11 @@ define void @test_vsuxseg6_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8770,11 +8638,11 @@ define void @test_vsuxseg6_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8790,11 +8658,11 @@ define void @test_vsuxseg6_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg6_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8807,11 +8675,11 @@ define void @test_vsuxseg6_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8827,12 +8695,12 @@ define void @test_vsuxseg7_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8845,12 +8713,12 @@ define void @test_vsuxseg7_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8866,12 +8734,12 @@ define void @test_vsuxseg7_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8884,12 +8752,12 @@ define void @test_vsuxseg7_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8905,12 +8773,12 @@ define void @test_vsuxseg7_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8923,12 +8791,12 @@ define void @test_vsuxseg7_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8944,12 +8812,12 @@ define void @test_vsuxseg7_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg7_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -8962,12 +8830,12 @@ define void @test_vsuxseg7_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -8983,13 +8851,13 @@ define void @test_vsuxseg8_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9002,13 +8870,13 @@ define void @test_vsuxseg8_mask_nxv1i8_nxv1i64(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i8_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9024,13 +8892,13 @@ define void @test_vsuxseg8_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9043,13 +8911,13 @@ define void @test_vsuxseg8_mask_nxv1i8_nxv1i32(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i8_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9065,13 +8933,13 @@ define void @test_vsuxseg8_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9084,13 +8952,13 @@ define void @test_vsuxseg8_mask_nxv1i8_nxv1i16(<vscale x 1 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i8_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9106,13 +8974,13 @@ define void @test_vsuxseg8_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg8_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9125,13 +8993,13 @@ define void @test_vsuxseg8_mask_nxv1i8_nxv1i8(<vscale x 1 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1i8_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9146,7 +9014,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i8.nxv2i32(<vscale x 2 x i8>,<vscale
 define void @test_vsuxseg2_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -9160,7 +9027,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -9177,7 +9043,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i8.nxv2i8(<vscale x 2 x i8>,<vscale x
 define void @test_vsuxseg2_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -9191,7 +9056,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -9208,7 +9072,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i8.nxv2i16(<vscale x 2 x i8>,<vscale
 define void @test_vsuxseg2_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -9222,7 +9085,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
@@ -9239,7 +9101,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i8.nxv2i64(<vscale x 2 x i8>,<vscale
 define void @test_vsuxseg2_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v10
@@ -9252,7 +9113,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v10, v0.t
@@ -9269,8 +9129,8 @@ define void @test_vsuxseg3_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg3_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9283,8 +9143,8 @@ define void @test_vsuxseg3_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9300,8 +9160,8 @@ define void @test_vsuxseg3_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg3_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9314,8 +9174,8 @@ define void @test_vsuxseg3_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9331,8 +9191,8 @@ define void @test_vsuxseg3_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg3_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9345,8 +9205,8 @@ define void @test_vsuxseg3_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9361,11 +9221,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv2i8.nxv2i64(<vscale x 2 x i8>,<vscale
 define void @test_vsuxseg3_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
-; CHECK-NEXT:    vsuxseg3ei64.v v12, (a0), v10
+; CHECK-NEXT:    vsuxseg3ei64.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv2i8.nxv2i64(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl)
@@ -9375,11 +9235,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
-; CHECK-NEXT:    vsuxseg3ei64.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsuxseg3ei64.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv2i8.nxv2i64(<vscale x 2 x i8> %val,<vscale x 2 x i8> %val,<vscale x 2 x i8> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl)
@@ -9393,9 +9253,9 @@ define void @test_vsuxseg4_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg4_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9408,9 +9268,9 @@ define void @test_vsuxseg4_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9426,9 +9286,9 @@ define void @test_vsuxseg4_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg4_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9441,9 +9301,9 @@ define void @test_vsuxseg4_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9459,9 +9319,9 @@ define void @test_vsuxseg4_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg4_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9474,9 +9334,9 @@ define void @test_vsuxseg4_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9492,9 +9352,9 @@ define void @test_vsuxseg4_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg4_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -9507,9 +9367,9 @@ define void @test_vsuxseg4_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -9525,10 +9385,10 @@ define void @test_vsuxseg5_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg5_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9541,10 +9401,10 @@ define void @test_vsuxseg5_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9560,10 +9420,10 @@ define void @test_vsuxseg5_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg5_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9576,10 +9436,10 @@ define void @test_vsuxseg5_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9595,10 +9455,10 @@ define void @test_vsuxseg5_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg5_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9611,10 +9471,10 @@ define void @test_vsuxseg5_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9630,10 +9490,10 @@ define void @test_vsuxseg5_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg5_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -9646,10 +9506,10 @@ define void @test_vsuxseg5_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -9665,11 +9525,11 @@ define void @test_vsuxseg6_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9682,11 +9542,11 @@ define void @test_vsuxseg6_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9702,11 +9562,11 @@ define void @test_vsuxseg6_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg6_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9719,11 +9579,11 @@ define void @test_vsuxseg6_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9739,11 +9599,11 @@ define void @test_vsuxseg6_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9756,11 +9616,11 @@ define void @test_vsuxseg6_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9776,11 +9636,11 @@ define void @test_vsuxseg6_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg6_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -9793,11 +9653,11 @@ define void @test_vsuxseg6_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -9813,12 +9673,12 @@ define void @test_vsuxseg7_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9831,12 +9691,12 @@ define void @test_vsuxseg7_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9852,12 +9712,12 @@ define void @test_vsuxseg7_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg7_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9870,12 +9730,12 @@ define void @test_vsuxseg7_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9891,12 +9751,12 @@ define void @test_vsuxseg7_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9909,12 +9769,12 @@ define void @test_vsuxseg7_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -9930,12 +9790,12 @@ define void @test_vsuxseg7_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg7_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -9948,12 +9808,12 @@ define void @test_vsuxseg7_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -9969,13 +9829,13 @@ define void @test_vsuxseg8_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -9988,13 +9848,13 @@ define void @test_vsuxseg8_mask_nxv2i8_nxv2i32(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i8_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10010,13 +9870,13 @@ define void @test_vsuxseg8_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base, <vsc
 ; CHECK-LABEL: test_vsuxseg8_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10029,13 +9889,13 @@ define void @test_vsuxseg8_mask_nxv2i8_nxv2i8(<vscale x 2 x i8> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i8_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10051,13 +9911,13 @@ define void @test_vsuxseg8_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10070,13 +9930,13 @@ define void @test_vsuxseg8_mask_nxv2i8_nxv2i16(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i8_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10092,13 +9952,13 @@ define void @test_vsuxseg8_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base, <vs
 ; CHECK-LABEL: test_vsuxseg8_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -10111,13 +9971,13 @@ define void @test_vsuxseg8_mask_nxv2i8_nxv2i64(<vscale x 2 x i8> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i8_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -10132,7 +9992,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i32.nxv8i16(<vscale x 8 x i32>,<vscal
 define void @test_vsuxseg2_nxv8i32_nxv8i16(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i32_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -10146,7 +10005,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i32_nxv8i16(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i32_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -10163,7 +10021,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i32.nxv8i8(<vscale x 8 x i32>,<vscale
 define void @test_vsuxseg2_nxv8i32_nxv8i8(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i32_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -10177,7 +10034,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i32_nxv8i8(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i32_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -10194,7 +10050,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i32.nxv8i64(<vscale x 8 x i32>,<vscal
 define void @test_vsuxseg2_nxv8i32_nxv8i64(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i32_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v16
@@ -10207,7 +10062,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i32_nxv8i64(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i32_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v16, v0.t
@@ -10223,7 +10077,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8i32.nxv8i32(<vscale x 8 x i32>,<vscal
 define void @test_vsuxseg2_nxv8i32_nxv8i32(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8i32_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -10237,7 +10090,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8i32_nxv8i32(<vscale x 8 x i32> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8i32_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -10254,7 +10106,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv32i8.nxv32i16(<vscale x 32 x i8>,<vsca
 define void @test_vsuxseg2_nxv32i8_nxv32i16(<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv32i8_nxv32i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei16.v v8, (a0), v16
@@ -10267,7 +10118,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv32i8_nxv32i16(<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i16> %index, <vscale x 32 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv32i8_nxv32i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei16.v v8, (a0), v16, v0.t
@@ -10283,7 +10133,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv32i8.nxv32i8(<vscale x 32 x i8>,<vscal
 define void @test_vsuxseg2_nxv32i8_nxv32i8(<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv32i8_nxv32i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
@@ -10297,7 +10146,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv32i8_nxv32i8(<vscale x 32 x i8> %val, ptr %base, <vscale x 32 x i8> %index, <vscale x 32 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv32i8_nxv32i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
@@ -10314,7 +10162,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i16.nxv2i32(<vscale x 2 x i16>,<vscal
 define void @test_vsuxseg2_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -10328,7 +10175,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -10345,7 +10191,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i16.nxv2i8(<vscale x 2 x i16>,<vscale
 define void @test_vsuxseg2_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -10359,7 +10204,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -10376,7 +10220,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i16.nxv2i16(<vscale x 2 x i16>,<vscal
 define void @test_vsuxseg2_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -10390,7 +10233,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -10407,7 +10249,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i16.nxv2i64(<vscale x 2 x i16>,<vscal
 define void @test_vsuxseg2_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v10
@@ -10420,7 +10261,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v10, v0.t
@@ -10437,8 +10277,8 @@ define void @test_vsuxseg3_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10451,8 +10291,8 @@ define void @test_vsuxseg3_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10468,8 +10308,8 @@ define void @test_vsuxseg3_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg3_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10482,8 +10322,8 @@ define void @test_vsuxseg3_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10499,8 +10339,8 @@ define void @test_vsuxseg3_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10513,8 +10353,8 @@ define void @test_vsuxseg3_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10529,11 +10369,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv2i16.nxv2i64(<vscale x 2 x i16>,<vscal
 define void @test_vsuxseg3_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei64.v v12, (a0), v10
+; CHECK-NEXT:    vsuxseg3ei64.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv2i16.nxv2i64(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl)
@@ -10543,11 +10383,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei64.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsuxseg3ei64.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv2i16.nxv2i64(<vscale x 2 x i16> %val,<vscale x 2 x i16> %val,<vscale x 2 x i16> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl)
@@ -10561,9 +10401,9 @@ define void @test_vsuxseg4_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10576,9 +10416,9 @@ define void @test_vsuxseg4_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10594,9 +10434,9 @@ define void @test_vsuxseg4_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg4_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10609,9 +10449,9 @@ define void @test_vsuxseg4_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10627,9 +10467,9 @@ define void @test_vsuxseg4_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10642,9 +10482,9 @@ define void @test_vsuxseg4_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10660,9 +10500,9 @@ define void @test_vsuxseg4_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -10675,9 +10515,9 @@ define void @test_vsuxseg4_mask_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -10693,10 +10533,10 @@ define void @test_vsuxseg5_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10709,10 +10549,10 @@ define void @test_vsuxseg5_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10728,10 +10568,10 @@ define void @test_vsuxseg5_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg5_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10744,10 +10584,10 @@ define void @test_vsuxseg5_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10763,10 +10603,10 @@ define void @test_vsuxseg5_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10779,10 +10619,10 @@ define void @test_vsuxseg5_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10798,10 +10638,10 @@ define void @test_vsuxseg5_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -10814,10 +10654,10 @@ define void @test_vsuxseg5_mask_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -10833,11 +10673,11 @@ define void @test_vsuxseg6_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10850,11 +10690,11 @@ define void @test_vsuxseg6_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10870,11 +10710,11 @@ define void @test_vsuxseg6_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg6_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10887,11 +10727,11 @@ define void @test_vsuxseg6_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10907,11 +10747,11 @@ define void @test_vsuxseg6_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10924,11 +10764,11 @@ define void @test_vsuxseg6_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -10944,11 +10784,11 @@ define void @test_vsuxseg6_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -10961,11 +10801,11 @@ define void @test_vsuxseg6_mask_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -10981,12 +10821,12 @@ define void @test_vsuxseg7_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -10999,12 +10839,12 @@ define void @test_vsuxseg7_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11020,12 +10860,12 @@ define void @test_vsuxseg7_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg7_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11038,12 +10878,12 @@ define void @test_vsuxseg7_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11059,12 +10899,12 @@ define void @test_vsuxseg7_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11077,12 +10917,12 @@ define void @test_vsuxseg7_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11098,12 +10938,12 @@ define void @test_vsuxseg7_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11116,12 +10956,12 @@ define void @test_vsuxseg7_mask_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11137,13 +10977,13 @@ define void @test_vsuxseg8_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11156,13 +10996,13 @@ define void @test_vsuxseg8_mask_nxv2i16_nxv2i32(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11178,13 +11018,13 @@ define void @test_vsuxseg8_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg8_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11197,13 +11037,13 @@ define void @test_vsuxseg8_mask_nxv2i16_nxv2i8(<vscale x 2 x i16> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11219,13 +11059,13 @@ define void @test_vsuxseg8_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -11238,13 +11078,13 @@ define void @test_vsuxseg8_mask_nxv2i16_nxv2i16(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -11260,13 +11100,13 @@ define void @test_vsuxseg8_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11279,13 +11119,13 @@ define void @test_vsuxseg8_mask_nxv2i16_nxv2i64(<vscale x 2 x i16> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2i16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11300,7 +11140,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i64.nxv2i32(<vscale x 2 x i64>,<vscal
 define void @test_vsuxseg2_nxv2i64_nxv2i32(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11314,7 +11153,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i64_nxv2i32(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11331,7 +11169,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i64.nxv2i8(<vscale x 2 x i64>,<vscale
 define void @test_vsuxseg2_nxv2i64_nxv2i8(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11345,7 +11182,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i64_nxv2i8(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11362,7 +11198,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i64.nxv2i16(<vscale x 2 x i64>,<vscal
 define void @test_vsuxseg2_nxv2i64_nxv2i16(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11376,7 +11211,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i64_nxv2i16(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11393,7 +11227,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2i64.nxv2i64(<vscale x 2 x i64>,<vscal
 define void @test_vsuxseg2_nxv2i64_nxv2i64(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2i64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11407,7 +11240,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2i64_nxv2i64(<vscale x 2 x i64> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2i64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -11425,8 +11257,8 @@ define void @test_vsuxseg3_nxv2i64_nxv2i32(<vscale x 2 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv2i64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11439,8 +11271,8 @@ define void @test_vsuxseg3_mask_nxv2i64_nxv2i32(<vscale x 2 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11456,8 +11288,8 @@ define void @test_vsuxseg3_nxv2i64_nxv2i8(<vscale x 2 x i64> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg3_nxv2i64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11470,8 +11302,8 @@ define void @test_vsuxseg3_mask_nxv2i64_nxv2i8(<vscale x 2 x i64> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11487,8 +11319,8 @@ define void @test_vsuxseg3_nxv2i64_nxv2i16(<vscale x 2 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv2i64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11501,8 +11333,8 @@ define void @test_vsuxseg3_mask_nxv2i64_nxv2i16(<vscale x 2 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11518,8 +11350,8 @@ define void @test_vsuxseg3_nxv2i64_nxv2i64(<vscale x 2 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv2i64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11532,8 +11364,8 @@ define void @test_vsuxseg3_mask_nxv2i64_nxv2i64(<vscale x 2 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2i64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11549,9 +11381,9 @@ define void @test_vsuxseg4_nxv2i64_nxv2i32(<vscale x 2 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv2i64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11564,9 +11396,9 @@ define void @test_vsuxseg4_mask_nxv2i64_nxv2i32(<vscale x 2 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11582,9 +11414,9 @@ define void @test_vsuxseg4_nxv2i64_nxv2i8(<vscale x 2 x i64> %val, ptr %base, <v
 ; CHECK-LABEL: test_vsuxseg4_nxv2i64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11597,9 +11429,9 @@ define void @test_vsuxseg4_mask_nxv2i64_nxv2i8(<vscale x 2 x i64> %val, ptr %bas
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11615,9 +11447,9 @@ define void @test_vsuxseg4_nxv2i64_nxv2i16(<vscale x 2 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv2i64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11630,9 +11462,9 @@ define void @test_vsuxseg4_mask_nxv2i64_nxv2i16(<vscale x 2 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11648,9 +11480,9 @@ define void @test_vsuxseg4_nxv2i64_nxv2i64(<vscale x 2 x i64> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv2i64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -11663,9 +11495,9 @@ define void @test_vsuxseg4_mask_nxv2i64_nxv2i64(<vscale x 2 x i64> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2i64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -11680,7 +11512,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16f16.nxv16i16(<vscale x 16 x half>,<v
 define void @test_vsuxseg2_nxv16f16_nxv16i16(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16f16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -11694,7 +11525,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16f16_nxv16i16(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16f16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -11711,7 +11541,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16f16.nxv16i8(<vscale x 16 x half>,<vs
 define void @test_vsuxseg2_nxv16f16_nxv16i8(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16f16_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -11725,7 +11554,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16f16_nxv16i8(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i8> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16f16_nxv16i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
@@ -11742,7 +11570,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16f16.nxv16i32(<vscale x 16 x half>,<v
 define void @test_vsuxseg2_nxv16f16_nxv16i32(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16f16_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v16
@@ -11755,7 +11582,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16f16_nxv16i32(<vscale x 16 x half> %val, ptr %base, <vscale x 16 x i32> %index, <vscale x 16 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16f16_nxv16i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v16, v0.t
@@ -11771,7 +11597,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f64.nxv4i32(<vscale x 4 x double>,<vs
 define void @test_vsuxseg2_nxv4f64_nxv4i32(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f64_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -11785,7 +11610,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f64_nxv4i32(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f64_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -11802,7 +11626,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f64.nxv4i8(<vscale x 4 x double>,<vsc
 define void @test_vsuxseg2_nxv4f64_nxv4i8(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f64_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -11816,7 +11639,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f64_nxv4i8(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f64_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -11833,7 +11655,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f64.nxv4i64(<vscale x 4 x double>,<vs
 define void @test_vsuxseg2_nxv4f64_nxv4i64(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f64_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -11847,7 +11668,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f64_nxv4i64(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f64_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -11864,7 +11684,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f64.nxv4i16(<vscale x 4 x double>,<vs
 define void @test_vsuxseg2_nxv4f64_nxv4i16(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f64_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -11878,7 +11697,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f64_nxv4i16(<vscale x 4 x double> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f64_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m4, ta, ma
@@ -11895,7 +11713,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f64.nxv1i64(<vscale x 1 x double>,<vs
 define void @test_vsuxseg2_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -11909,7 +11726,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -11926,7 +11742,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f64.nxv1i32(<vscale x 1 x double>,<vs
 define void @test_vsuxseg2_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -11940,7 +11755,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -11957,7 +11771,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f64.nxv1i16(<vscale x 1 x double>,<vs
 define void @test_vsuxseg2_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -11971,7 +11784,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -11988,7 +11800,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f64.nxv1i8(<vscale x 1 x double>,<vsc
 define void @test_vsuxseg2_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -12002,7 +11813,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
@@ -12020,8 +11830,8 @@ define void @test_vsuxseg3_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12034,8 +11844,8 @@ define void @test_vsuxseg3_mask_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12051,8 +11861,8 @@ define void @test_vsuxseg3_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12065,8 +11875,8 @@ define void @test_vsuxseg3_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12082,8 +11892,8 @@ define void @test_vsuxseg3_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12096,8 +11906,8 @@ define void @test_vsuxseg3_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12113,8 +11923,8 @@ define void @test_vsuxseg3_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12127,8 +11937,8 @@ define void @test_vsuxseg3_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12144,9 +11954,9 @@ define void @test_vsuxseg4_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12159,9 +11969,9 @@ define void @test_vsuxseg4_mask_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12177,9 +11987,9 @@ define void @test_vsuxseg4_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12192,9 +12002,9 @@ define void @test_vsuxseg4_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12210,9 +12020,9 @@ define void @test_vsuxseg4_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12225,9 +12035,9 @@ define void @test_vsuxseg4_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12243,9 +12053,9 @@ define void @test_vsuxseg4_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12258,9 +12068,9 @@ define void @test_vsuxseg4_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12276,10 +12086,10 @@ define void @test_vsuxseg5_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12292,10 +12102,10 @@ define void @test_vsuxseg5_mask_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12311,10 +12121,10 @@ define void @test_vsuxseg5_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12327,10 +12137,10 @@ define void @test_vsuxseg5_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12346,10 +12156,10 @@ define void @test_vsuxseg5_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg5_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12362,10 +12172,10 @@ define void @test_vsuxseg5_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12381,10 +12191,10 @@ define void @test_vsuxseg5_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12397,10 +12207,10 @@ define void @test_vsuxseg5_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12416,11 +12226,11 @@ define void @test_vsuxseg6_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12433,11 +12243,11 @@ define void @test_vsuxseg6_mask_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12453,11 +12263,11 @@ define void @test_vsuxseg6_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12470,11 +12280,11 @@ define void @test_vsuxseg6_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12490,11 +12300,11 @@ define void @test_vsuxseg6_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg6_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12507,11 +12317,11 @@ define void @test_vsuxseg6_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12527,11 +12337,11 @@ define void @test_vsuxseg6_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12544,11 +12354,11 @@ define void @test_vsuxseg6_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12564,12 +12374,12 @@ define void @test_vsuxseg7_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12582,12 +12392,12 @@ define void @test_vsuxseg7_mask_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12603,12 +12413,12 @@ define void @test_vsuxseg7_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12621,12 +12431,12 @@ define void @test_vsuxseg7_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12642,12 +12452,12 @@ define void @test_vsuxseg7_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg7_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12660,12 +12470,12 @@ define void @test_vsuxseg7_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12681,12 +12491,12 @@ define void @test_vsuxseg7_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12699,12 +12509,12 @@ define void @test_vsuxseg7_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12720,13 +12530,13 @@ define void @test_vsuxseg8_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12739,13 +12549,13 @@ define void @test_vsuxseg8_mask_nxv1f64_nxv1i64(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f64_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12761,13 +12571,13 @@ define void @test_vsuxseg8_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12780,13 +12590,13 @@ define void @test_vsuxseg8_mask_nxv1f64_nxv1i32(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f64_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12802,13 +12612,13 @@ define void @test_vsuxseg8_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg8_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12821,13 +12631,13 @@ define void @test_vsuxseg8_mask_nxv1f64_nxv1i16(<vscale x 1 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f64_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12843,13 +12653,13 @@ define void @test_vsuxseg8_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -12862,13 +12672,13 @@ define void @test_vsuxseg8_mask_nxv1f64_nxv1i8(<vscale x 1 x double> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f64_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -12883,7 +12693,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f32.nxv2i32(<vscale x 2 x float>,<vsc
 define void @test_vsuxseg2_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -12897,7 +12706,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -12914,7 +12722,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f32.nxv2i8(<vscale x 2 x float>,<vsca
 define void @test_vsuxseg2_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -12928,7 +12735,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -12945,7 +12751,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f32.nxv2i16(<vscale x 2 x float>,<vsc
 define void @test_vsuxseg2_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -12959,7 +12764,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
@@ -12976,7 +12780,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f32.nxv2i64(<vscale x 2 x float>,<vsc
 define void @test_vsuxseg2_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v10
@@ -12989,7 +12792,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v10, v0.t
@@ -13006,8 +12808,8 @@ define void @test_vsuxseg3_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13020,8 +12822,8 @@ define void @test_vsuxseg3_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13037,8 +12839,8 @@ define void @test_vsuxseg3_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13051,8 +12853,8 @@ define void @test_vsuxseg3_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13068,8 +12870,8 @@ define void @test_vsuxseg3_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13082,8 +12884,8 @@ define void @test_vsuxseg3_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13098,11 +12900,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv2f32.nxv2i64(<vscale x 2 x float>,<vsc
 define void @test_vsuxseg3_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
-; CHECK-NEXT:    vsuxseg3ei64.v v12, (a0), v10
+; CHECK-NEXT:    vsuxseg3ei64.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv2f32.nxv2i64(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl)
@@ -13112,11 +12914,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
-; CHECK-NEXT:    vsuxseg3ei64.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsuxseg3ei64.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv2f32.nxv2i64(<vscale x 2 x float> %val,<vscale x 2 x float> %val,<vscale x 2 x float> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl)
@@ -13130,9 +12932,9 @@ define void @test_vsuxseg4_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13145,9 +12947,9 @@ define void @test_vsuxseg4_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13163,9 +12965,9 @@ define void @test_vsuxseg4_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13178,9 +12980,9 @@ define void @test_vsuxseg4_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13196,9 +12998,9 @@ define void @test_vsuxseg4_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13211,9 +13013,9 @@ define void @test_vsuxseg4_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13229,9 +13031,9 @@ define void @test_vsuxseg4_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13244,9 +13046,9 @@ define void @test_vsuxseg4_mask_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13262,10 +13064,10 @@ define void @test_vsuxseg5_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13278,10 +13080,10 @@ define void @test_vsuxseg5_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13297,10 +13099,10 @@ define void @test_vsuxseg5_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13313,10 +13115,10 @@ define void @test_vsuxseg5_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13332,10 +13134,10 @@ define void @test_vsuxseg5_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13348,10 +13150,10 @@ define void @test_vsuxseg5_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13367,10 +13169,10 @@ define void @test_vsuxseg5_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13383,10 +13185,10 @@ define void @test_vsuxseg5_mask_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13402,11 +13204,11 @@ define void @test_vsuxseg6_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13419,11 +13221,11 @@ define void @test_vsuxseg6_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13439,11 +13241,11 @@ define void @test_vsuxseg6_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13456,11 +13258,11 @@ define void @test_vsuxseg6_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13476,11 +13278,11 @@ define void @test_vsuxseg6_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13493,11 +13295,11 @@ define void @test_vsuxseg6_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13513,11 +13315,11 @@ define void @test_vsuxseg6_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13530,11 +13332,11 @@ define void @test_vsuxseg6_mask_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13550,12 +13352,12 @@ define void @test_vsuxseg7_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13568,12 +13370,12 @@ define void @test_vsuxseg7_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13589,12 +13391,12 @@ define void @test_vsuxseg7_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13607,12 +13409,12 @@ define void @test_vsuxseg7_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13628,12 +13430,12 @@ define void @test_vsuxseg7_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13646,12 +13448,12 @@ define void @test_vsuxseg7_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13667,12 +13469,12 @@ define void @test_vsuxseg7_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13685,12 +13487,12 @@ define void @test_vsuxseg7_mask_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13706,13 +13508,13 @@ define void @test_vsuxseg8_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13725,13 +13527,13 @@ define void @test_vsuxseg8_mask_nxv2f32_nxv2i32(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2f32_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13747,13 +13549,13 @@ define void @test_vsuxseg8_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13766,13 +13568,13 @@ define void @test_vsuxseg8_mask_nxv2f32_nxv2i8(<vscale x 2 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2f32_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13788,13 +13590,13 @@ define void @test_vsuxseg8_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -13807,13 +13609,13 @@ define void @test_vsuxseg8_mask_nxv2f32_nxv2i16(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2f32_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -13829,13 +13631,13 @@ define void @test_vsuxseg8_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -13848,13 +13650,13 @@ define void @test_vsuxseg8_mask_nxv2f32_nxv2i64(<vscale x 2 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2f32_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -13869,7 +13671,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f16.nxv1i64(<vscale x 1 x half>,<vsca
 define void @test_vsuxseg2_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -13883,7 +13684,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -13900,7 +13700,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f16.nxv1i32(<vscale x 1 x half>,<vsca
 define void @test_vsuxseg2_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -13914,7 +13713,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -13931,7 +13729,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f16.nxv1i16(<vscale x 1 x half>,<vsca
 define void @test_vsuxseg2_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -13945,7 +13742,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -13962,7 +13758,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f16.nxv1i8(<vscale x 1 x half>,<vscal
 define void @test_vsuxseg2_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -13976,7 +13771,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
@@ -13994,8 +13788,8 @@ define void @test_vsuxseg3_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14008,8 +13802,8 @@ define void @test_vsuxseg3_mask_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14025,8 +13819,8 @@ define void @test_vsuxseg3_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14039,8 +13833,8 @@ define void @test_vsuxseg3_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14056,8 +13850,8 @@ define void @test_vsuxseg3_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14070,8 +13864,8 @@ define void @test_vsuxseg3_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14087,8 +13881,8 @@ define void @test_vsuxseg3_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14101,8 +13895,8 @@ define void @test_vsuxseg3_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14118,9 +13912,9 @@ define void @test_vsuxseg4_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14133,9 +13927,9 @@ define void @test_vsuxseg4_mask_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14151,9 +13945,9 @@ define void @test_vsuxseg4_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14166,9 +13960,9 @@ define void @test_vsuxseg4_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14184,9 +13978,9 @@ define void @test_vsuxseg4_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14199,9 +13993,9 @@ define void @test_vsuxseg4_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14217,9 +14011,9 @@ define void @test_vsuxseg4_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14232,9 +14026,9 @@ define void @test_vsuxseg4_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14250,10 +14044,10 @@ define void @test_vsuxseg5_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14266,10 +14060,10 @@ define void @test_vsuxseg5_mask_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14285,10 +14079,10 @@ define void @test_vsuxseg5_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14301,10 +14095,10 @@ define void @test_vsuxseg5_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14320,10 +14114,10 @@ define void @test_vsuxseg5_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14336,10 +14130,10 @@ define void @test_vsuxseg5_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14355,10 +14149,10 @@ define void @test_vsuxseg5_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14371,10 +14165,10 @@ define void @test_vsuxseg5_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14390,11 +14184,11 @@ define void @test_vsuxseg6_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14407,11 +14201,11 @@ define void @test_vsuxseg6_mask_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14427,11 +14221,11 @@ define void @test_vsuxseg6_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14444,11 +14238,11 @@ define void @test_vsuxseg6_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14464,11 +14258,11 @@ define void @test_vsuxseg6_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14481,11 +14275,11 @@ define void @test_vsuxseg6_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14501,11 +14295,11 @@ define void @test_vsuxseg6_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14518,11 +14312,11 @@ define void @test_vsuxseg6_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14538,12 +14332,12 @@ define void @test_vsuxseg7_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14556,12 +14350,12 @@ define void @test_vsuxseg7_mask_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14577,12 +14371,12 @@ define void @test_vsuxseg7_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14595,12 +14389,12 @@ define void @test_vsuxseg7_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14616,12 +14410,12 @@ define void @test_vsuxseg7_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14634,12 +14428,12 @@ define void @test_vsuxseg7_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14655,12 +14449,12 @@ define void @test_vsuxseg7_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14673,12 +14467,12 @@ define void @test_vsuxseg7_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14694,13 +14488,13 @@ define void @test_vsuxseg8_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14713,13 +14507,13 @@ define void @test_vsuxseg8_mask_nxv1f16_nxv1i64(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f16_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14735,13 +14529,13 @@ define void @test_vsuxseg8_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14754,13 +14548,13 @@ define void @test_vsuxseg8_mask_nxv1f16_nxv1i32(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f16_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14776,13 +14570,13 @@ define void @test_vsuxseg8_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14795,13 +14589,13 @@ define void @test_vsuxseg8_mask_nxv1f16_nxv1i16(<vscale x 1 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f16_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14817,13 +14611,13 @@ define void @test_vsuxseg8_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14836,13 +14630,13 @@ define void @test_vsuxseg8_mask_nxv1f16_nxv1i8(<vscale x 1 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f16_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf4, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -14857,7 +14651,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f32.nxv1i64(<vscale x 1 x float>,<vsc
 define void @test_vsuxseg2_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -14871,7 +14664,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i64> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -14888,7 +14680,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f32.nxv1i32(<vscale x 1 x float>,<vsc
 define void @test_vsuxseg2_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -14902,7 +14693,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i32> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -14919,7 +14709,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f32.nxv1i16(<vscale x 1 x float>,<vsc
 define void @test_vsuxseg2_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -14933,7 +14722,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i16> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -14950,7 +14738,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv1f32.nxv1i8(<vscale x 1 x float>,<vsca
 define void @test_vsuxseg2_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -14964,7 +14751,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base, <vscale x 1 x i8> %index, <vscale x 1 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
@@ -14982,8 +14768,8 @@ define void @test_vsuxseg3_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -14996,8 +14782,8 @@ define void @test_vsuxseg3_mask_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15013,8 +14799,8 @@ define void @test_vsuxseg3_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15027,8 +14813,8 @@ define void @test_vsuxseg3_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15044,8 +14830,8 @@ define void @test_vsuxseg3_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15058,8 +14844,8 @@ define void @test_vsuxseg3_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15075,8 +14861,8 @@ define void @test_vsuxseg3_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15089,8 +14875,8 @@ define void @test_vsuxseg3_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15106,9 +14892,9 @@ define void @test_vsuxseg4_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15121,9 +14907,9 @@ define void @test_vsuxseg4_mask_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15139,9 +14925,9 @@ define void @test_vsuxseg4_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15154,9 +14940,9 @@ define void @test_vsuxseg4_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15172,9 +14958,9 @@ define void @test_vsuxseg4_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15187,9 +14973,9 @@ define void @test_vsuxseg4_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15205,9 +14991,9 @@ define void @test_vsuxseg4_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15220,9 +15006,9 @@ define void @test_vsuxseg4_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15238,10 +15024,10 @@ define void @test_vsuxseg5_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15254,10 +15040,10 @@ define void @test_vsuxseg5_mask_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15273,10 +15059,10 @@ define void @test_vsuxseg5_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15289,10 +15075,10 @@ define void @test_vsuxseg5_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15308,10 +15094,10 @@ define void @test_vsuxseg5_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15324,10 +15110,10 @@ define void @test_vsuxseg5_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15343,10 +15129,10 @@ define void @test_vsuxseg5_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15359,10 +15145,10 @@ define void @test_vsuxseg5_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15378,11 +15164,11 @@ define void @test_vsuxseg6_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15395,11 +15181,11 @@ define void @test_vsuxseg6_mask_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15415,11 +15201,11 @@ define void @test_vsuxseg6_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15432,11 +15218,11 @@ define void @test_vsuxseg6_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15452,11 +15238,11 @@ define void @test_vsuxseg6_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15469,11 +15255,11 @@ define void @test_vsuxseg6_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15489,11 +15275,11 @@ define void @test_vsuxseg6_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15506,11 +15292,11 @@ define void @test_vsuxseg6_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15526,12 +15312,12 @@ define void @test_vsuxseg7_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15544,12 +15330,12 @@ define void @test_vsuxseg7_mask_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15565,12 +15351,12 @@ define void @test_vsuxseg7_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15583,12 +15369,12 @@ define void @test_vsuxseg7_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15604,12 +15390,12 @@ define void @test_vsuxseg7_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15622,12 +15408,12 @@ define void @test_vsuxseg7_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15643,12 +15429,12 @@ define void @test_vsuxseg7_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15661,12 +15447,12 @@ define void @test_vsuxseg7_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15682,13 +15468,13 @@ define void @test_vsuxseg8_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15701,13 +15487,13 @@ define void @test_vsuxseg8_mask_nxv1f32_nxv1i64(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f32_nxv1i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15723,13 +15509,13 @@ define void @test_vsuxseg8_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15742,13 +15528,13 @@ define void @test_vsuxseg8_mask_nxv1f32_nxv1i32(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f32_nxv1i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15764,13 +15550,13 @@ define void @test_vsuxseg8_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15783,13 +15569,13 @@ define void @test_vsuxseg8_mask_nxv1f32_nxv1i16(<vscale x 1 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f32_nxv1i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15805,13 +15591,13 @@ define void @test_vsuxseg8_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -15824,13 +15610,13 @@ define void @test_vsuxseg8_mask_nxv1f32_nxv1i8(<vscale x 1 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv1f32_nxv1i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -15845,7 +15631,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8f16.nxv8i16(<vscale x 8 x half>,<vsca
 define void @test_vsuxseg2_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -15859,7 +15644,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -15876,7 +15660,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8f16.nxv8i8(<vscale x 8 x half>,<vscal
 define void @test_vsuxseg2_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -15890,7 +15673,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -15907,7 +15689,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8f16.nxv8i64(<vscale x 8 x half>,<vsca
 define void @test_vsuxseg2_nxv8f16_nxv8i64(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8f16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v16
@@ -15920,7 +15701,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8f16_nxv8i64(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8f16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v16, v0.t
@@ -15936,7 +15716,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8f16.nxv8i32(<vscale x 8 x half>,<vsca
 define void @test_vsuxseg2_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v12
@@ -15949,7 +15728,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v12, v0.t
@@ -15966,8 +15744,8 @@ define void @test_vsuxseg3_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -15980,8 +15758,8 @@ define void @test_vsuxseg3_mask_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -15997,8 +15775,8 @@ define void @test_vsuxseg3_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16011,8 +15789,8 @@ define void @test_vsuxseg3_mask_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16027,7 +15805,6 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv8f16.nxv8i64(<vscale x 8 x half>,<vsca
 define void @test_vsuxseg3_nxv8f16_nxv8i64(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv8f16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -16041,7 +15818,6 @@ entry:
 define void @test_vsuxseg3_mask_nxv8f16_nxv8i64(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8f16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
@@ -16058,11 +15834,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv8f16.nxv8i32(<vscale x 8 x half>,<vsca
 define void @test_vsuxseg3_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v16, (a0), v12
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl)
@@ -16072,11 +15848,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv8f16.nxv8i32(<vscale x 8 x half> %val,<vscale x 8 x half> %val,<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl)
@@ -16090,9 +15866,9 @@ define void @test_vsuxseg4_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16105,9 +15881,9 @@ define void @test_vsuxseg4_mask_nxv8f16_nxv8i16(<vscale x 8 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8f16_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16123,9 +15899,9 @@ define void @test_vsuxseg4_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16138,9 +15914,9 @@ define void @test_vsuxseg4_mask_nxv8f16_nxv8i8(<vscale x 8 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8f16_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16155,7 +15931,6 @@ declare void @llvm.riscv.vsuxseg4.mask.nxv8f16.nxv8i64(<vscale x 8 x half>,<vsca
 define void @test_vsuxseg4_nxv8f16_nxv8i64(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_nxv8f16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -16170,7 +15945,6 @@ entry:
 define void @test_vsuxseg4_mask_nxv8f16_nxv8i64(<vscale x 8 x half> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8f16_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2_v12m2_v14m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vmv2r.v v14, v8
@@ -16189,9 +15963,9 @@ define void @test_vsuxseg4_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -16204,9 +15978,9 @@ define void @test_vsuxseg4_mask_nxv8f16_nxv8i32(<vscale x 8 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv8f16_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -16221,7 +15995,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8f32.nxv8i16(<vscale x 8 x float>,<vsc
 define void @test_vsuxseg2_nxv8f32_nxv8i16(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8f32_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -16235,7 +16008,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8f32_nxv8i16(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i16> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8f32_nxv8i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv2r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -16252,7 +16024,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8f32.nxv8i8(<vscale x 8 x float>,<vsca
 define void @test_vsuxseg2_nxv8f32_nxv8i8(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8f32_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -16266,7 +16037,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8f32_nxv8i8(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i8> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8f32_nxv8i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv1r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -16283,7 +16053,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8f32.nxv8i64(<vscale x 8 x float>,<vsc
 define void @test_vsuxseg2_nxv8f32_nxv8i64(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8f32_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v16
@@ -16296,7 +16065,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8f32_nxv8i64(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i64> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8f32_nxv8i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v16, v0.t
@@ -16312,7 +16080,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv8f32.nxv8i32(<vscale x 8 x float>,<vsc
 define void @test_vsuxseg2_nxv8f32_nxv8i32(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv8f32_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -16326,7 +16093,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv8f32_nxv8i32(<vscale x 8 x float> %val, ptr %base, <vscale x 8 x i32> %index, <vscale x 8 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv8f32_nxv8i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m4, ta, ma
@@ -16343,7 +16109,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f64.nxv2i32(<vscale x 2 x double>,<vs
 define void @test_vsuxseg2_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -16357,7 +16122,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -16374,7 +16138,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f64.nxv2i8(<vscale x 2 x double>,<vsc
 define void @test_vsuxseg2_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -16388,7 +16151,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -16405,7 +16167,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f64.nxv2i16(<vscale x 2 x double>,<vs
 define void @test_vsuxseg2_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -16419,7 +16180,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -16436,7 +16196,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f64.nxv2i64(<vscale x 2 x double>,<vs
 define void @test_vsuxseg2_nxv2f64_nxv2i64(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -16450,7 +16209,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f64_nxv2i64(<vscale x 2 x double> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
@@ -16468,8 +16226,8 @@ define void @test_vsuxseg3_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16482,8 +16240,8 @@ define void @test_vsuxseg3_mask_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16499,8 +16257,8 @@ define void @test_vsuxseg3_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16513,8 +16271,8 @@ define void @test_vsuxseg3_mask_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16530,8 +16288,8 @@ define void @test_vsuxseg3_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16544,8 +16302,8 @@ define void @test_vsuxseg3_mask_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16561,8 +16319,8 @@ define void @test_vsuxseg3_nxv2f64_nxv2i64(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg3_nxv2f64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16575,8 +16333,8 @@ define void @test_vsuxseg3_mask_nxv2f64_nxv2i64(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16592,9 +16350,9 @@ define void @test_vsuxseg4_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16607,9 +16365,9 @@ define void @test_vsuxseg4_mask_nxv2f64_nxv2i32(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f64_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16625,9 +16383,9 @@ define void @test_vsuxseg4_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16640,9 +16398,9 @@ define void @test_vsuxseg4_mask_nxv2f64_nxv2i8(<vscale x 2 x double> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f64_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16658,9 +16416,9 @@ define void @test_vsuxseg4_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16673,9 +16431,9 @@ define void @test_vsuxseg4_mask_nxv2f64_nxv2i16(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f64_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16691,9 +16449,9 @@ define void @test_vsuxseg4_nxv2f64_nxv2i64(<vscale x 2 x double> %val, ptr %base
 ; CHECK-LABEL: test_vsuxseg4_nxv2f64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16706,9 +16464,9 @@ define void @test_vsuxseg4_mask_nxv2f64_nxv2i64(<vscale x 2 x double> %val, ptr
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f64_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -16723,7 +16481,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f16.nxv4i32(<vscale x 4 x half>,<vsca
 define void @test_vsuxseg2_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v10
@@ -16736,7 +16493,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei32.v v8, (a0), v10, v0.t
@@ -16752,7 +16508,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f16.nxv4i8(<vscale x 4 x half>,<vscal
 define void @test_vsuxseg2_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -16766,7 +16521,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -16783,7 +16537,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f16.nxv4i64(<vscale x 4 x half>,<vsca
 define void @test_vsuxseg2_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v12
@@ -16796,7 +16549,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v12, v0.t
@@ -16812,7 +16564,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f16.nxv4i16(<vscale x 4 x half>,<vsca
 define void @test_vsuxseg2_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -16826,7 +16577,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -16843,11 +16593,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv4f16.nxv4i32(<vscale x 4 x half>,<vsca
 define void @test_vsuxseg3_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl)
@@ -16857,11 +16607,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsuxseg3ei32.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv4f16.nxv4i32(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl)
@@ -16875,8 +16625,8 @@ define void @test_vsuxseg3_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -16889,8 +16639,8 @@ define void @test_vsuxseg3_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -16905,7 +16655,6 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv4f16.nxv4i64(<vscale x 4 x half>,<vsca
 define void @test_vsuxseg3_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -16919,7 +16668,6 @@ entry:
 define void @test_vsuxseg3_mask_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
@@ -16937,8 +16685,8 @@ define void @test_vsuxseg3_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -16951,8 +16699,8 @@ define void @test_vsuxseg3_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -16968,9 +16716,9 @@ define void @test_vsuxseg4_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -16983,9 +16731,9 @@ define void @test_vsuxseg4_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -17001,9 +16749,9 @@ define void @test_vsuxseg4_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17016,9 +16764,9 @@ define void @test_vsuxseg4_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17033,7 +16781,6 @@ declare void @llvm.riscv.vsuxseg4.mask.nxv4f16.nxv4i64(<vscale x 4 x half>,<vsca
 define void @test_vsuxseg4_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -17048,7 +16795,6 @@ entry:
 define void @test_vsuxseg4_mask_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9_v10_v11
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vmv1r.v v11, v8
@@ -17067,9 +16813,9 @@ define void @test_vsuxseg4_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17082,9 +16828,9 @@ define void @test_vsuxseg4_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17100,10 +16846,10 @@ define void @test_vsuxseg5_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -17116,10 +16862,10 @@ define void @test_vsuxseg5_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -17135,10 +16881,10 @@ define void @test_vsuxseg5_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17151,10 +16897,10 @@ define void @test_vsuxseg5_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17169,13 +16915,13 @@ declare void @llvm.riscv.vsuxseg5.mask.nxv4f16.nxv4i64(<vscale x 4 x half>,<vsca
 define void @test_vsuxseg5_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg5_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsuxseg5ei64.v v16, (a0), v12
+; CHECK-NEXT:    vsuxseg5ei64.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg5.nxv4f16.nxv4i64(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl)
@@ -17185,13 +16931,13 @@ entry:
 define void @test_vsuxseg5_mask_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv1r.v v10, v8
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
-; CHECK-NEXT:    vsuxseg5ei64.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsuxseg5ei64.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg5.mask.nxv4f16.nxv4i64(<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val,<vscale x 4 x half> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl)
@@ -17205,10 +16951,10 @@ define void @test_vsuxseg5_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17221,10 +16967,10 @@ define void @test_vsuxseg5_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17240,11 +16986,11 @@ define void @test_vsuxseg6_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -17257,11 +17003,11 @@ define void @test_vsuxseg6_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -17277,11 +17023,11 @@ define void @test_vsuxseg6_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17294,11 +17040,11 @@ define void @test_vsuxseg6_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17314,11 +17060,11 @@ define void @test_vsuxseg6_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -17331,11 +17077,11 @@ define void @test_vsuxseg6_mask_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -17351,11 +17097,11 @@ define void @test_vsuxseg6_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17368,11 +17114,11 @@ define void @test_vsuxseg6_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17388,12 +17134,12 @@ define void @test_vsuxseg7_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -17406,12 +17152,12 @@ define void @test_vsuxseg7_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -17427,12 +17173,12 @@ define void @test_vsuxseg7_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17445,12 +17191,12 @@ define void @test_vsuxseg7_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17466,12 +17212,12 @@ define void @test_vsuxseg7_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -17484,12 +17230,12 @@ define void @test_vsuxseg7_mask_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -17505,12 +17251,12 @@ define void @test_vsuxseg7_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17523,12 +17269,12 @@ define void @test_vsuxseg7_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17544,13 +17290,13 @@ define void @test_vsuxseg8_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -17563,13 +17309,13 @@ define void @test_vsuxseg8_mask_nxv4f16_nxv4i32(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4f16_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -17585,13 +17331,13 @@ define void @test_vsuxseg8_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17604,13 +17350,13 @@ define void @test_vsuxseg8_mask_nxv4f16_nxv4i8(<vscale x 4 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4f16_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17626,13 +17372,13 @@ define void @test_vsuxseg8_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -17645,13 +17391,13 @@ define void @test_vsuxseg8_mask_nxv4f16_nxv4i64(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4f16_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v16, v8
-; CHECK-NEXT:    vmv1r.v v17, v16
-; CHECK-NEXT:    vmv1r.v v18, v16
-; CHECK-NEXT:    vmv1r.v v19, v16
-; CHECK-NEXT:    vmv1r.v v20, v16
-; CHECK-NEXT:    vmv1r.v v21, v16
-; CHECK-NEXT:    vmv1r.v v22, v16
-; CHECK-NEXT:    vmv1r.v v23, v16
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
+; CHECK-NEXT:    vmv1r.v v20, v8
+; CHECK-NEXT:    vmv1r.v v21, v8
+; CHECK-NEXT:    vmv1r.v v22, v8
+; CHECK-NEXT:    vmv1r.v v23, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -17667,13 +17413,13 @@ define void @test_vsuxseg8_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17686,13 +17432,13 @@ define void @test_vsuxseg8_mask_nxv4f16_nxv4i16(<vscale x 4 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv4f16_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17707,7 +17453,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f16.nxv2i32(<vscale x 2 x half>,<vsca
 define void @test_vsuxseg2_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -17721,7 +17466,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i32> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -17738,7 +17482,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f16.nxv2i8(<vscale x 2 x half>,<vscal
 define void @test_vsuxseg2_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -17752,7 +17495,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i8> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -17769,7 +17511,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f16.nxv2i16(<vscale x 2 x half>,<vsca
 define void @test_vsuxseg2_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -17783,7 +17524,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i16> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 killed $v8_v9 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v10, v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
@@ -17800,7 +17540,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv2f16.nxv2i64(<vscale x 2 x half>,<vsca
 define void @test_vsuxseg2_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v10
@@ -17813,7 +17552,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8_v9
 ; CHECK-NEXT:    vmv1r.v v9, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v10, v0.t
@@ -17830,8 +17568,8 @@ define void @test_vsuxseg3_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17844,8 +17582,8 @@ define void @test_vsuxseg3_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17861,8 +17599,8 @@ define void @test_vsuxseg3_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg3_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17875,8 +17613,8 @@ define void @test_vsuxseg3_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17892,8 +17630,8 @@ define void @test_vsuxseg3_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17906,8 +17644,8 @@ define void @test_vsuxseg3_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17922,11 +17660,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv2f16.nxv2i64(<vscale x 2 x half>,<vsca
 define void @test_vsuxseg3_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei64.v v12, (a0), v10
+; CHECK-NEXT:    vsuxseg3ei64.v v8, (a0), v12
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv2f16.nxv2i64(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, i64 %vl)
@@ -17936,11 +17674,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
+; CHECK-NEXT:    vmv1r.v v9, v8
+; CHECK-NEXT:    vmv2r.v v12, v10
+; CHECK-NEXT:    vmv1r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei64.v v12, (a0), v10, v0.t
+; CHECK-NEXT:    vsuxseg3ei64.v v8, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv2f16.nxv2i64(<vscale x 2 x half> %val,<vscale x 2 x half> %val,<vscale x 2 x half> %val, ptr %base, <vscale x 2 x i64> %index, <vscale x 2 x i1> %mask, i64 %vl)
@@ -17954,9 +17692,9 @@ define void @test_vsuxseg4_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -17969,9 +17707,9 @@ define void @test_vsuxseg4_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -17987,9 +17725,9 @@ define void @test_vsuxseg4_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg4_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18002,9 +17740,9 @@ define void @test_vsuxseg4_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18020,9 +17758,9 @@ define void @test_vsuxseg4_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18035,9 +17773,9 @@ define void @test_vsuxseg4_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18053,9 +17791,9 @@ define void @test_vsuxseg4_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18068,9 +17806,9 @@ define void @test_vsuxseg4_mask_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -18086,10 +17824,10 @@ define void @test_vsuxseg5_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18102,10 +17840,10 @@ define void @test_vsuxseg5_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18121,10 +17859,10 @@ define void @test_vsuxseg5_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg5_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18137,10 +17875,10 @@ define void @test_vsuxseg5_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18156,10 +17894,10 @@ define void @test_vsuxseg5_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18172,10 +17910,10 @@ define void @test_vsuxseg5_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18191,10 +17929,10 @@ define void @test_vsuxseg5_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg5_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18207,10 +17945,10 @@ define void @test_vsuxseg5_mask_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg5_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg5ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -18226,11 +17964,11 @@ define void @test_vsuxseg6_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18243,11 +17981,11 @@ define void @test_vsuxseg6_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18263,11 +18001,11 @@ define void @test_vsuxseg6_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg6_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18280,11 +18018,11 @@ define void @test_vsuxseg6_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18300,11 +18038,11 @@ define void @test_vsuxseg6_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18317,11 +18055,11 @@ define void @test_vsuxseg6_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18337,11 +18075,11 @@ define void @test_vsuxseg6_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg6_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18354,11 +18092,11 @@ define void @test_vsuxseg6_mask_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg6_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg6ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -18374,12 +18112,12 @@ define void @test_vsuxseg7_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18392,12 +18130,12 @@ define void @test_vsuxseg7_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18413,12 +18151,12 @@ define void @test_vsuxseg7_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg7_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18431,12 +18169,12 @@ define void @test_vsuxseg7_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18452,12 +18190,12 @@ define void @test_vsuxseg7_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18470,12 +18208,12 @@ define void @test_vsuxseg7_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18491,12 +18229,12 @@ define void @test_vsuxseg7_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg7_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18509,12 +18247,12 @@ define void @test_vsuxseg7_mask_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg7_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg7ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -18530,13 +18268,13 @@ define void @test_vsuxseg8_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18549,13 +18287,13 @@ define void @test_vsuxseg8_mask_nxv2f16_nxv2i32(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2f16_nxv2i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei32.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18571,13 +18309,13 @@ define void @test_vsuxseg8_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %base, <
 ; CHECK-LABEL: test_vsuxseg8_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18590,13 +18328,13 @@ define void @test_vsuxseg8_mask_nxv2f16_nxv2i8(<vscale x 2 x half> %val, ptr %ba
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2f16_nxv2i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei8.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18612,13 +18350,13 @@ define void @test_vsuxseg8_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9
 ; CHECK-NEXT:    ret
@@ -18631,13 +18369,13 @@ define void @test_vsuxseg8_mask_nxv2f16_nxv2i16(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2f16_nxv2i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v10, v8
-; CHECK-NEXT:    vmv1r.v v11, v10
-; CHECK-NEXT:    vmv1r.v v12, v10
-; CHECK-NEXT:    vmv1r.v v13, v10
-; CHECK-NEXT:    vmv1r.v v14, v10
-; CHECK-NEXT:    vmv1r.v v15, v10
-; CHECK-NEXT:    vmv1r.v v16, v10
-; CHECK-NEXT:    vmv1r.v v17, v10
+; CHECK-NEXT:    vmv1r.v v11, v8
+; CHECK-NEXT:    vmv1r.v v12, v8
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei16.v v10, (a0), v9, v0.t
 ; CHECK-NEXT:    ret
@@ -18653,13 +18391,13 @@ define void @test_vsuxseg8_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg8_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18672,13 +18410,13 @@ define void @test_vsuxseg8_mask_nxv2f16_nxv2i64(<vscale x 2 x half> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg8_mask_nxv2f16_nxv2i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv1r.v v12, v8
-; CHECK-NEXT:    vmv1r.v v13, v12
-; CHECK-NEXT:    vmv1r.v v14, v12
-; CHECK-NEXT:    vmv1r.v v15, v12
-; CHECK-NEXT:    vmv1r.v v16, v12
-; CHECK-NEXT:    vmv1r.v v17, v12
-; CHECK-NEXT:    vmv1r.v v18, v12
-; CHECK-NEXT:    vmv1r.v v19, v12
+; CHECK-NEXT:    vmv1r.v v13, v8
+; CHECK-NEXT:    vmv1r.v v14, v8
+; CHECK-NEXT:    vmv1r.v v15, v8
+; CHECK-NEXT:    vmv1r.v v16, v8
+; CHECK-NEXT:    vmv1r.v v17, v8
+; CHECK-NEXT:    vmv1r.v v18, v8
+; CHECK-NEXT:    vmv1r.v v19, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e16, mf2, ta, ma
 ; CHECK-NEXT:    vsuxseg8ei64.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -18693,7 +18431,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f32.nxv4i32(<vscale x 4 x float>,<vsc
 define void @test_vsuxseg2_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -18707,7 +18444,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i32> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -18724,7 +18460,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f32.nxv4i8(<vscale x 4 x float>,<vsca
 define void @test_vsuxseg2_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -18738,7 +18473,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i8> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -18755,7 +18489,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f32.nxv4i64(<vscale x 4 x float>,<vsc
 define void @test_vsuxseg2_nxv4f32_nxv4i64(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v12
@@ -18768,7 +18501,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f32_nxv4i64(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg2ei64.v v8, (a0), v12, v0.t
@@ -18784,7 +18516,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv4f32.nxv4i16(<vscale x 4 x float>,<vsc
 define void @test_vsuxseg2_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -18798,7 +18529,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i16> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v8m2_v10m2 def $v8m2_v10m2
 ; CHECK-NEXT:    vmv1r.v v12, v10
 ; CHECK-NEXT:    vmv2r.v v10, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
@@ -18816,8 +18546,8 @@ define void @test_vsuxseg3_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18830,8 +18560,8 @@ define void @test_vsuxseg3_mask_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -18847,8 +18577,8 @@ define void @test_vsuxseg3_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18861,8 +18591,8 @@ define void @test_vsuxseg3_mask_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -18877,11 +18607,11 @@ declare void @llvm.riscv.vsuxseg3.mask.nxv4f32.nxv4i64(<vscale x 4 x float>,<vsc
 define void @test_vsuxseg3_nxv4f32_nxv4i64(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_nxv4f32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei64.v v16, (a0), v12
+; CHECK-NEXT:    vsuxseg3ei64.v v8, (a0), v16
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.nxv4f32.nxv4i64(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, i64 %vl)
@@ -18891,11 +18621,11 @@ entry:
 define void @test_vsuxseg3_mask_nxv4f32_nxv4i64(<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl) {
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4f32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
+; CHECK-NEXT:    vmv2r.v v10, v8
+; CHECK-NEXT:    vmv4r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v12, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
-; CHECK-NEXT:    vsuxseg3ei64.v v16, (a0), v12, v0.t
+; CHECK-NEXT:    vsuxseg3ei64.v v8, (a0), v16, v0.t
 ; CHECK-NEXT:    ret
 entry:
   tail call void @llvm.riscv.vsuxseg3.mask.nxv4f32.nxv4i64(<vscale x 4 x float> %val,<vscale x 4 x float> %val,<vscale x 4 x float> %val, ptr %base, <vscale x 4 x i64> %index, <vscale x 4 x i1> %mask, i64 %vl)
@@ -18909,8 +18639,8 @@ define void @test_vsuxseg3_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg3_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18923,8 +18653,8 @@ define void @test_vsuxseg3_mask_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg3_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg3ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -18940,9 +18670,9 @@ define void @test_vsuxseg4_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18955,9 +18685,9 @@ define void @test_vsuxseg4_mask_nxv4f32_nxv4i32(<vscale x 4 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4f32_nxv4i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei32.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -18973,9 +18703,9 @@ define void @test_vsuxseg4_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -18988,9 +18718,9 @@ define void @test_vsuxseg4_mask_nxv4f32_nxv4i8(<vscale x 4 x float> %val, ptr %b
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4f32_nxv4i8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei8.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
@@ -19006,9 +18736,9 @@ define void @test_vsuxseg4_nxv4f32_nxv4i64(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv4f32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v16, (a0), v12
 ; CHECK-NEXT:    ret
@@ -19021,9 +18751,9 @@ define void @test_vsuxseg4_mask_nxv4f32_nxv4i64(<vscale x 4 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4f32_nxv4i64:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v16, v8
-; CHECK-NEXT:    vmv2r.v v18, v16
-; CHECK-NEXT:    vmv2r.v v20, v16
-; CHECK-NEXT:    vmv2r.v v22, v16
+; CHECK-NEXT:    vmv2r.v v18, v8
+; CHECK-NEXT:    vmv2r.v v20, v8
+; CHECK-NEXT:    vmv2r.v v22, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei64.v v16, (a0), v12, v0.t
 ; CHECK-NEXT:    ret
@@ -19039,9 +18769,9 @@ define void @test_vsuxseg4_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %base,
 ; CHECK-LABEL: test_vsuxseg4_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10
 ; CHECK-NEXT:    ret
@@ -19054,9 +18784,9 @@ define void @test_vsuxseg4_mask_nxv4f32_nxv4i16(<vscale x 4 x float> %val, ptr %
 ; CHECK-LABEL: test_vsuxseg4_mask_nxv4f32_nxv4i16:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vmv2r.v v12, v8
-; CHECK-NEXT:    vmv2r.v v14, v12
-; CHECK-NEXT:    vmv2r.v v16, v12
-; CHECK-NEXT:    vmv2r.v v18, v12
+; CHECK-NEXT:    vmv2r.v v14, v8
+; CHECK-NEXT:    vmv2r.v v16, v8
+; CHECK-NEXT:    vmv2r.v v18, v8
 ; CHECK-NEXT:    vsetvli zero, a1, e32, m2, ta, ma
 ; CHECK-NEXT:    vsuxseg4ei16.v v12, (a0), v10, v0.t
 ; CHECK-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/rvv/vtrunc-vp.ll b/llvm/test/CodeGen/RISCV/rvv/vtrunc-vp.ll
index 24407b2ab4b0d..3ce9147622749 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vtrunc-vp.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vtrunc-vp.ll
@@ -285,29 +285,25 @@ define <vscale x 32 x i32> @vtrunc_nxv32i64_nxv32i32(<vscale x 32 x i64> %a, <vs
 ; CHECK-NEXT:    addi sp, sp, -16
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    csrr a1, vlenb
-; CHECK-NEXT:    li a3, 24
-; CHECK-NEXT:    mul a1, a1, a3
+; CHECK-NEXT:    slli a1, a1, 4
 ; CHECK-NEXT:    sub sp, sp, a1
-; CHECK-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x18, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 24 * vlenb
+; CHECK-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x10, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 16 * vlenb
 ; CHECK-NEXT:    vmv1r.v v1, v0
-; CHECK-NEXT:    csrr a1, vlenb
-; CHECK-NEXT:    slli a1, a1, 3
-; CHECK-NEXT:    add a1, sp, a1
-; CHECK-NEXT:    addi a1, a1, 16
+; CHECK-NEXT:    addi a1, sp, 16
 ; CHECK-NEXT:    vs8r.v v16, (a1) # Unknown-size Folded Spill
 ; CHECK-NEXT:    csrr a1, vlenb
-; CHECK-NEXT:    slli a1, a1, 4
+; CHECK-NEXT:    slli a1, a1, 3
 ; CHECK-NEXT:    add a1, sp, a1
 ; CHECK-NEXT:    addi a1, a1, 16
 ; CHECK-NEXT:    vs8r.v v8, (a1) # Unknown-size Folded Spill
 ; CHECK-NEXT:    csrr a1, vlenb
 ; CHECK-NEXT:    srli a3, a1, 2
 ; CHECK-NEXT:    vsetvli a4, zero, e8, mf2, ta, ma
-; CHECK-NEXT:    vslidedown.vx v25, v0, a3
+; CHECK-NEXT:    vslidedown.vx v16, v0, a3
 ; CHECK-NEXT:    vsetvli a3, zero, e8, mf4, ta, ma
 ; CHECK-NEXT:    slli a3, a1, 3
 ; CHECK-NEXT:    add a3, a0, a3
-; CHECK-NEXT:    vl8re64.v v8, (a3)
+; CHECK-NEXT:    vl8re64.v v24, (a3)
 ; CHECK-NEXT:    slli a3, a1, 1
 ; CHECK-NEXT:    sub a4, a2, a3
 ; CHECK-NEXT:    sltu a5, a2, a4
@@ -318,22 +314,18 @@ define <vscale x 32 x i32> @vtrunc_nxv32i64_nxv32i32(<vscale x 32 x i64> %a, <vs
 ; CHECK-NEXT:    addi a6, a6, -1
 ; CHECK-NEXT:    and a6, a6, a5
 ; CHECK-NEXT:    srli a5, a1, 3
-; CHECK-NEXT:    vl8re64.v v16, (a0)
-; CHECK-NEXT:    addi a0, sp, 16
-; CHECK-NEXT:    vs8r.v v16, (a0) # Unknown-size Folded Spill
-; CHECK-NEXT:    vslidedown.vx v0, v25, a5
+; CHECK-NEXT:    vl8re64.v v8, (a0)
+; CHECK-NEXT:    vslidedown.vx v0, v16, a5
 ; CHECK-NEXT:    vsetvli zero, a6, e32, m4, ta, ma
-; CHECK-NEXT:    vnsrl.wi v20, v8, 0, v0.t
+; CHECK-NEXT:    vnsrl.wi v20, v24, 0, v0.t
 ; CHECK-NEXT:    bltu a4, a1, .LBB17_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a4, a1
 ; CHECK-NEXT:  .LBB17_2:
 ; CHECK-NEXT:    vsetvli a0, zero, e8, mf4, ta, ma
-; CHECK-NEXT:    vslidedown.vx v26, v1, a5
+; CHECK-NEXT:    vslidedown.vx v2, v1, a5
 ; CHECK-NEXT:    vsetvli zero, a4, e32, m4, ta, ma
-; CHECK-NEXT:    vmv1r.v v0, v25
-; CHECK-NEXT:    addi a0, sp, 16
-; CHECK-NEXT:    vl8r.v v8, (a0) # Unknown-size Folded Reload
+; CHECK-NEXT:    vmv1r.v v0, v16
 ; CHECK-NEXT:    vnsrl.wi v16, v8, 0, v0.t
 ; CHECK-NEXT:    bltu a2, a3, .LBB17_4
 ; CHECK-NEXT:  # %bb.3:
@@ -344,13 +336,10 @@ define <vscale x 32 x i32> @vtrunc_nxv32i64_nxv32i32(<vscale x 32 x i64> %a, <vs
 ; CHECK-NEXT:    addi a3, a3, -1
 ; CHECK-NEXT:    and a0, a3, a0
 ; CHECK-NEXT:    vsetvli zero, a0, e32, m4, ta, ma
-; CHECK-NEXT:    vmv1r.v v0, v26
-; CHECK-NEXT:    csrr a0, vlenb
-; CHECK-NEXT:    slli a0, a0, 3
-; CHECK-NEXT:    add a0, sp, a0
-; CHECK-NEXT:    addi a0, a0, 16
-; CHECK-NEXT:    vl8r.v v24, (a0) # Unknown-size Folded Reload
-; CHECK-NEXT:    vnsrl.wi v12, v24, 0, v0.t
+; CHECK-NEXT:    vmv1r.v v0, v2
+; CHECK-NEXT:    addi a0, sp, 16
+; CHECK-NEXT:    vl8r.v v8, (a0) # Unknown-size Folded Reload
+; CHECK-NEXT:    vnsrl.wi v28, v8, 0, v0.t
 ; CHECK-NEXT:    bltu a2, a1, .LBB17_6
 ; CHECK-NEXT:  # %bb.5:
 ; CHECK-NEXT:    mv a2, a1
@@ -358,14 +347,14 @@ define <vscale x 32 x i32> @vtrunc_nxv32i64_nxv32i32(<vscale x 32 x i64> %a, <vs
 ; CHECK-NEXT:    vsetvli zero, a2, e32, m4, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1
 ; CHECK-NEXT:    csrr a0, vlenb
-; CHECK-NEXT:    slli a0, a0, 4
+; CHECK-NEXT:    slli a0, a0, 3
 ; CHECK-NEXT:    add a0, sp, a0
 ; CHECK-NEXT:    addi a0, a0, 16
-; CHECK-NEXT:    vl8r.v v24, (a0) # Unknown-size Folded Reload
-; CHECK-NEXT:    vnsrl.wi v8, v24, 0, v0.t
+; CHECK-NEXT:    vl8r.v v8, (a0) # Unknown-size Folded Reload
+; CHECK-NEXT:    vnsrl.wi v24, v8, 0, v0.t
+; CHECK-NEXT:    vmv8r.v v8, v24
 ; CHECK-NEXT:    csrr a0, vlenb
-; CHECK-NEXT:    li a1, 24
-; CHECK-NEXT:    mul a0, a0, a1
+; CHECK-NEXT:    slli a0, a0, 4
 ; CHECK-NEXT:    add sp, sp, a0
 ; CHECK-NEXT:    addi sp, sp, 16
 ; CHECK-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/rvv/vuitofp-vp.ll b/llvm/test/CodeGen/RISCV/rvv/vuitofp-vp.ll
index 7cdd0af87f714..5107e1942c9f3 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vuitofp-vp.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vuitofp-vp.ll
@@ -306,15 +306,7 @@ declare <vscale x 32 x half> @llvm.vp.uitofp.nxv32f16.nxv32i32(<vscale x 32 x i3
 define <vscale x 32 x half> @vuitofp_nxv32f16_nxv32i32(<vscale x 32 x i32> %va, <vscale x 32 x i1> %m, i32 zeroext %evl) {
 ; CHECK-LABEL: vuitofp_nxv32f16_nxv32i32:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    addi sp, sp, -16
-; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    csrr a1, vlenb
-; CHECK-NEXT:    slli a1, a1, 3
-; CHECK-NEXT:    sub sp, sp, a1
-; CHECK-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x08, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 8 * vlenb
 ; CHECK-NEXT:    vmv1r.v v24, v0
-; CHECK-NEXT:    addi a1, sp, 16
-; CHECK-NEXT:    vs8r.v v8, (a1) # Unknown-size Folded Spill
 ; CHECK-NEXT:    csrr a1, vlenb
 ; CHECK-NEXT:    srli a2, a1, 2
 ; CHECK-NEXT:    vsetvli a3, zero, e8, mf2, ta, ma
@@ -325,20 +317,15 @@ define <vscale x 32 x half> @vuitofp_nxv32f16_nxv32i32(<vscale x 32 x i32> %va,
 ; CHECK-NEXT:    addi a3, a3, -1
 ; CHECK-NEXT:    and a2, a3, a2
 ; CHECK-NEXT:    vsetvli zero, a2, e16, m4, ta, ma
-; CHECK-NEXT:    vfncvt.f.xu.w v12, v16, v0.t
+; CHECK-NEXT:    vfncvt.f.xu.w v28, v16, v0.t
 ; CHECK-NEXT:    bltu a0, a1, .LBB25_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a0, a1
 ; CHECK-NEXT:  .LBB25_2:
 ; CHECK-NEXT:    vsetvli zero, a0, e16, m4, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v24
-; CHECK-NEXT:    addi a0, sp, 16
-; CHECK-NEXT:    vl8r.v v16, (a0) # Unknown-size Folded Reload
-; CHECK-NEXT:    vfncvt.f.xu.w v8, v16, v0.t
-; CHECK-NEXT:    csrr a0, vlenb
-; CHECK-NEXT:    slli a0, a0, 3
-; CHECK-NEXT:    add sp, sp, a0
-; CHECK-NEXT:    addi sp, sp, 16
+; CHECK-NEXT:    vfncvt.f.xu.w v24, v8, v0.t
+; CHECK-NEXT:    vmv8r.v v8, v24
 ; CHECK-NEXT:    ret
   %v = call <vscale x 32 x half> @llvm.vp.uitofp.nxv32f16.nxv32i32(<vscale x 32 x i32> %va, <vscale x 32 x i1> %m, i32 %evl)
   ret <vscale x 32 x half> %v
diff --git a/llvm/test/CodeGen/RISCV/rvv/zvlsseg-zero-vl.ll b/llvm/test/CodeGen/RISCV/rvv/zvlsseg-zero-vl.ll
index ec14e4eac8d70..0f1e33e47f03e 100644
--- a/llvm/test/CodeGen/RISCV/rvv/zvlsseg-zero-vl.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/zvlsseg-zero-vl.ll
@@ -15,7 +15,6 @@ define <vscale x 16 x i16> @test_vlseg2_mask_nxv16i16(ptr %base, <vscale x 16 x
 ; CHECK-NEXT:    vlseg2e16.v v4, (a0)
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlseg2e16.v v4, (a0), v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vlseg2.nxv16i16(<vscale x 16 x i16> undef,<vscale x 16 x i16> undef, ptr %base, i64 0)
@@ -35,7 +34,6 @@ define <vscale x 16 x i16> @test_vlsseg2_mask_nxv16i16(ptr %base, i64 %offset, <
 ; CHECK-NEXT:    vlsseg2e16.v v4, (a0), a1
 ; CHECK-NEXT:    vmv4r.v v8, v4
 ; CHECK-NEXT:    vlsseg2e16.v v4, (a0), a1, v0.t
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>} @llvm.riscv.vlsseg2.nxv16i16(<vscale x 16 x i16> undef, <vscale x 16 x i16> undef, ptr %base, i64 %offset, i64 0)
@@ -94,7 +92,6 @@ define <vscale x 16 x i16> @test_vlseg2ff_nxv16i16(ptr %base, ptr %outvl) {
 ; CHECK-NEXT:    vlseg2e16ff.v v4, (a0)
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a1)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>, i64} @llvm.riscv.vlseg2ff.nxv16i16(<vscale x 16 x i16> undef, <vscale x 16 x i16> undef, ptr %base, i64 0)
@@ -112,7 +109,6 @@ define <vscale x 16 x i16> @test_vlseg2ff_mask_nxv16i16(<vscale x 16 x i16> %val
 ; CHECK-NEXT:    vlseg2e16ff.v v4, (a0), v0.t
 ; CHECK-NEXT:    csrr a0, vl
 ; CHECK-NEXT:    sd a0, 0(a1)
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call {<vscale x 16 x i16>,<vscale x 16 x i16>, i64} @llvm.riscv.vlseg2ff.mask.nxv16i16(<vscale x 16 x i16> %val,<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i1> %mask, i64 0, i64 1)
@@ -128,7 +124,6 @@ declare void @llvm.riscv.vsseg2.mask.nxv16i16(<vscale x 16 x i16>,<vscale x 16 x
 define void @test_vsseg2_nxv16i16(<vscale x 16 x i16> %val, ptr %base) {
 ; CHECK-LABEL: test_vsseg2_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetivli zero, 0, e16, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0)
@@ -141,7 +136,6 @@ entry:
 define void @test_vsseg2_mask_nxv16i16(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i1> %mask) {
 ; CHECK-LABEL: test_vsseg2_mask_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetivli zero, 0, e16, m4, ta, ma
 ; CHECK-NEXT:    vsseg2e16.v v8, (a0), v0.t
@@ -157,7 +151,6 @@ declare void @llvm.riscv.vssseg2.mask.nxv16i16(<vscale x 16 x i16>,<vscale x 16
 define void @test_vssseg2_nxv16i16(<vscale x 16 x i16> %val, ptr %base, i64 %offset) {
 ; CHECK-LABEL: test_vssseg2_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetivli zero, 0, e16, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1
@@ -170,7 +163,6 @@ entry:
 define void @test_vssseg2_mask_nxv16i16(<vscale x 16 x i16> %val, ptr %base, i64 %offset, <vscale x 16 x i1> %mask) {
 ; CHECK-LABEL: test_vssseg2_mask_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetivli zero, 0, e16, m4, ta, ma
 ; CHECK-NEXT:    vssseg2e16.v v8, (a0), a1, v0.t
@@ -186,7 +178,6 @@ declare void @llvm.riscv.vsoxseg2.mask.nxv16i16.nxv16i16(<vscale x 16 x i16>,<vs
 define void @test_vsoxseg2_nxv16i16_nxv16i16(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i16> %index) {
 ; CHECK-LABEL: test_vsoxseg2_nxv16i16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetivli zero, 0, e16, m4, ta, ma
@@ -200,7 +191,6 @@ entry:
 define void @test_vsoxseg2_mask_nxv16i16_nxv16i16(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask) {
 ; CHECK-LABEL: test_vsoxseg2_mask_nxv16i16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetivli zero, 0, e16, m4, ta, ma
@@ -217,7 +207,6 @@ declare void @llvm.riscv.vsuxseg2.mask.nxv16i16.nxv16i16(<vscale x 16 x i16>,<vs
 define void @test_vsuxseg2_nxv16i16_nxv16i16(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i16> %index) {
 ; CHECK-LABEL: test_vsuxseg2_nxv16i16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetivli zero, 0, e16, m4, ta, ma
@@ -231,7 +220,6 @@ entry:
 define void @test_vsuxseg2_mask_nxv16i16_nxv16i16(<vscale x 16 x i16> %val, ptr %base, <vscale x 16 x i16> %index, <vscale x 16 x i1> %mask) {
 ; CHECK-LABEL: test_vsuxseg2_mask_nxv16i16_nxv16i16:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v8m4_v12m4 def $v8m4_v12m4
 ; CHECK-NEXT:    vmv4r.v v16, v12
 ; CHECK-NEXT:    vmv4r.v v12, v8
 ; CHECK-NEXT:    vsetivli zero, 0, e16, m4, ta, ma
diff --git a/llvm/test/CodeGen/RISCV/sadd_sat.ll b/llvm/test/CodeGen/RISCV/sadd_sat.ll
index 3d7668b860ddf..8efeb3b1bb668 100644
--- a/llvm/test/CodeGen/RISCV/sadd_sat.ll
+++ b/llvm/test/CodeGen/RISCV/sadd_sat.ll
@@ -37,7 +37,7 @@ define signext i32 @func(i32 signext %x, i32 signext %y) nounwind {
 ; RV64I-NEXT:    ret
 ; RV64I-NEXT:  .LBB0_3:
 ; RV64I-NEXT:    mv a0, a2
-; RV64I-NEXT:    blt a1, a0, .LBB0_2
+; RV64I-NEXT:    blt a1, a2, .LBB0_2
 ; RV64I-NEXT:  .LBB0_4:
 ; RV64I-NEXT:    lui a0, 524288
 ; RV64I-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/sadd_sat_plus.ll b/llvm/test/CodeGen/RISCV/sadd_sat_plus.ll
index aa9496007adcb..48b7ead5e7320 100644
--- a/llvm/test/CodeGen/RISCV/sadd_sat_plus.ll
+++ b/llvm/test/CodeGen/RISCV/sadd_sat_plus.ll
@@ -40,7 +40,7 @@ define i32 @func32(i32 %x, i32 %y, i32 %z) nounwind {
 ; RV64I-NEXT:    ret
 ; RV64I-NEXT:  .LBB0_3:
 ; RV64I-NEXT:    mv a0, a2
-; RV64I-NEXT:    blt a1, a0, .LBB0_2
+; RV64I-NEXT:    blt a1, a2, .LBB0_2
 ; RV64I-NEXT:  .LBB0_4:
 ; RV64I-NEXT:    lui a0, 524288
 ; RV64I-NEXT:    ret
@@ -148,7 +148,7 @@ define i16 @func16(i16 %x, i16 %y, i16 %z) nounwind {
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    slli a0, a0, 48
 ; RV64I-NEXT:    srai a0, a0, 48
-; RV64I-NEXT:    mulw a1, a1, a2
+; RV64I-NEXT:    mul a1, a1, a2
 ; RV64I-NEXT:    slli a1, a1, 48
 ; RV64I-NEXT:    srai a1, a1, 48
 ; RV64I-NEXT:    add a0, a0, a1
@@ -184,7 +184,7 @@ define i16 @func16(i16 %x, i16 %y, i16 %z) nounwind {
 ; RV64IZbb-LABEL: func16:
 ; RV64IZbb:       # %bb.0:
 ; RV64IZbb-NEXT:    sext.h a0, a0
-; RV64IZbb-NEXT:    mulw a1, a1, a2
+; RV64IZbb-NEXT:    mul a1, a1, a2
 ; RV64IZbb-NEXT:    sext.h a1, a1
 ; RV64IZbb-NEXT:    add a0, a0, a1
 ; RV64IZbb-NEXT:    lui a1, 8
@@ -226,7 +226,7 @@ define i8 @func8(i8 %x, i8 %y, i8 %z) nounwind {
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    slli a0, a0, 56
 ; RV64I-NEXT:    srai a0, a0, 56
-; RV64I-NEXT:    mulw a1, a1, a2
+; RV64I-NEXT:    mul a1, a1, a2
 ; RV64I-NEXT:    slli a1, a1, 56
 ; RV64I-NEXT:    srai a1, a1, 56
 ; RV64I-NEXT:    add a0, a0, a1
@@ -260,7 +260,7 @@ define i8 @func8(i8 %x, i8 %y, i8 %z) nounwind {
 ; RV64IZbb-LABEL: func8:
 ; RV64IZbb:       # %bb.0:
 ; RV64IZbb-NEXT:    sext.b a0, a0
-; RV64IZbb-NEXT:    mulw a1, a1, a2
+; RV64IZbb-NEXT:    mul a1, a1, a2
 ; RV64IZbb-NEXT:    sext.b a1, a1
 ; RV64IZbb-NEXT:    add a0, a0, a1
 ; RV64IZbb-NEXT:    li a1, 127
@@ -301,7 +301,7 @@ define i4 @func4(i4 %x, i4 %y, i4 %z) nounwind {
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    slli a0, a0, 60
 ; RV64I-NEXT:    srai a0, a0, 60
-; RV64I-NEXT:    mulw a1, a1, a2
+; RV64I-NEXT:    mul a1, a1, a2
 ; RV64I-NEXT:    slli a1, a1, 60
 ; RV64I-NEXT:    srai a1, a1, 60
 ; RV64I-NEXT:    add a0, a0, a1
@@ -338,7 +338,7 @@ define i4 @func4(i4 %x, i4 %y, i4 %z) nounwind {
 ; RV64IZbb:       # %bb.0:
 ; RV64IZbb-NEXT:    slli a0, a0, 60
 ; RV64IZbb-NEXT:    srai a0, a0, 60
-; RV64IZbb-NEXT:    mulw a1, a1, a2
+; RV64IZbb-NEXT:    mul a1, a1, a2
 ; RV64IZbb-NEXT:    slli a1, a1, 60
 ; RV64IZbb-NEXT:    srai a1, a1, 60
 ; RV64IZbb-NEXT:    add a0, a0, a1
diff --git a/llvm/test/CodeGen/RISCV/select-optimize-multiple.ll b/llvm/test/CodeGen/RISCV/select-optimize-multiple.ll
index a1546a1a89fb2..0efe2996ec842 100644
--- a/llvm/test/CodeGen/RISCV/select-optimize-multiple.ll
+++ b/llvm/test/CodeGen/RISCV/select-optimize-multiple.ll
@@ -267,11 +267,9 @@ define i32 @cmovccdep(i32 signext %a, i32 %b, i32 %c, i32 %d) nounwind {
 ; RV32I-NEXT:    ret
 ; RV32I-NEXT:  .LBB6_3: # %entry
 ; RV32I-NEXT:    mv a1, a2
-; RV32I-NEXT:    mv a2, a1
 ; RV32I-NEXT:    beq a0, a4, .LBB6_2
 ; RV32I-NEXT:  .LBB6_4: # %entry
-; RV32I-NEXT:    mv a2, a3
-; RV32I-NEXT:    add a0, a1, a2
+; RV32I-NEXT:    add a0, a1, a3
 ; RV32I-NEXT:    ret
 ;
 ; RV64I-LABEL: cmovccdep:
@@ -286,11 +284,9 @@ define i32 @cmovccdep(i32 signext %a, i32 %b, i32 %c, i32 %d) nounwind {
 ; RV64I-NEXT:    ret
 ; RV64I-NEXT:  .LBB6_3: # %entry
 ; RV64I-NEXT:    mv a1, a2
-; RV64I-NEXT:    mv a2, a1
 ; RV64I-NEXT:    beq a0, a4, .LBB6_2
 ; RV64I-NEXT:  .LBB6_4: # %entry
-; RV64I-NEXT:    mv a2, a3
-; RV64I-NEXT:    addw a0, a1, a2
+; RV64I-NEXT:    addw a0, a1, a3
 ; RV64I-NEXT:    ret
 entry:
   %cmp = icmp eq i32 %a, 123
@@ -317,8 +313,7 @@ define i32 @cmovdiffcc(i1 %a, i1 %b, i32 %c, i32 %d, i32 %e, i32 %f) nounwind {
 ; RV32I-NEXT:    mv a2, a3
 ; RV32I-NEXT:    bnez a1, .LBB7_2
 ; RV32I-NEXT:  .LBB7_4: # %entry
-; RV32I-NEXT:    mv a4, a5
-; RV32I-NEXT:    add a0, a2, a4
+; RV32I-NEXT:    add a0, a2, a5
 ; RV32I-NEXT:    ret
 ;
 ; RV64I-LABEL: cmovdiffcc:
@@ -335,8 +330,7 @@ define i32 @cmovdiffcc(i1 %a, i1 %b, i32 %c, i32 %d, i32 %e, i32 %f) nounwind {
 ; RV64I-NEXT:    mv a2, a3
 ; RV64I-NEXT:    bnez a1, .LBB7_2
 ; RV64I-NEXT:  .LBB7_4: # %entry
-; RV64I-NEXT:    mv a4, a5
-; RV64I-NEXT:    addw a0, a2, a4
+; RV64I-NEXT:    addw a0, a2, a5
 ; RV64I-NEXT:    ret
 entry:
   %cond1 = select i1 %a, i32 %c, i32 %d
diff --git a/llvm/test/CodeGen/RISCV/sextw-removal.ll b/llvm/test/CodeGen/RISCV/sextw-removal.ll
index ffe64db623508..0fcf3194a3f44 100644
--- a/llvm/test/CodeGen/RISCV/sextw-removal.ll
+++ b/llvm/test/CodeGen/RISCV/sextw-removal.ll
@@ -198,7 +198,7 @@ define void @test5(i32 signext %arg, i32 signext %arg1) nounwind {
 ; RV64I-NEXT:    srli a2, a0, 4
 ; RV64I-NEXT:    add a0, a0, a2
 ; RV64I-NEXT:    and a0, a0, s2
-; RV64I-NEXT:    mulw a0, a0, s3
+; RV64I-NEXT:    mul a0, a0, s3
 ; RV64I-NEXT:    srliw a0, a0, 24
 ; RV64I-NEXT:    bnez a1, .LBB4_1
 ; RV64I-NEXT:  # %bb.2: # %bb7
diff --git a/llvm/test/CodeGen/RISCV/shadowcallstack.ll b/llvm/test/CodeGen/RISCV/shadowcallstack.ll
index 51df390df802d..4442372637f24 100644
--- a/llvm/test/CodeGen/RISCV/shadowcallstack.ll
+++ b/llvm/test/CodeGen/RISCV/shadowcallstack.ll
@@ -36,6 +36,7 @@ define i32 @f3() shadowcallstack {
 ; RV32:       # %bb.0:
 ; RV32-NEXT:    sw ra, 0(s2)
 ; RV32-NEXT:    addi s2, s2, 4
+; RV32-NEXT:    .cfi_escape 0x16, 0x12, 0x02, 0x82, 0x7c #
 ; RV32-NEXT:    addi sp, sp, -16
 ; RV32-NEXT:    .cfi_def_cfa_offset 16
 ; RV32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
@@ -45,12 +46,14 @@ define i32 @f3() shadowcallstack {
 ; RV32-NEXT:    addi sp, sp, 16
 ; RV32-NEXT:    lw ra, -4(s2)
 ; RV32-NEXT:    addi s2, s2, -4
+; RV32-NEXT:    .cfi_restore s2
 ; RV32-NEXT:    ret
 ;
 ; RV64-LABEL: f3:
 ; RV64:       # %bb.0:
 ; RV64-NEXT:    sd ra, 0(s2)
 ; RV64-NEXT:    addi s2, s2, 8
+; RV64-NEXT:    .cfi_escape 0x16, 0x12, 0x02, 0x82, 0x78 #
 ; RV64-NEXT:    addi sp, sp, -16
 ; RV64-NEXT:    .cfi_def_cfa_offset 16
 ; RV64-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
@@ -60,6 +63,7 @@ define i32 @f3() shadowcallstack {
 ; RV64-NEXT:    addi sp, sp, 16
 ; RV64-NEXT:    ld ra, -8(s2)
 ; RV64-NEXT:    addi s2, s2, -8
+; RV64-NEXT:    .cfi_restore s2
 ; RV64-NEXT:    ret
   %res = call i32 @bar()
   %res1 = add i32 %res, 1
@@ -71,6 +75,7 @@ define i32 @f4() shadowcallstack {
 ; RV32:       # %bb.0:
 ; RV32-NEXT:    sw ra, 0(s2)
 ; RV32-NEXT:    addi s2, s2, 4
+; RV32-NEXT:    .cfi_escape 0x16, 0x12, 0x02, 0x82, 0x7c #
 ; RV32-NEXT:    addi sp, sp, -16
 ; RV32-NEXT:    .cfi_def_cfa_offset 16
 ; RV32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
@@ -98,12 +103,14 @@ define i32 @f4() shadowcallstack {
 ; RV32-NEXT:    addi sp, sp, 16
 ; RV32-NEXT:    lw ra, -4(s2)
 ; RV32-NEXT:    addi s2, s2, -4
+; RV32-NEXT:    .cfi_restore s2
 ; RV32-NEXT:    ret
 ;
 ; RV64-LABEL: f4:
 ; RV64:       # %bb.0:
 ; RV64-NEXT:    sd ra, 0(s2)
 ; RV64-NEXT:    addi s2, s2, 8
+; RV64-NEXT:    .cfi_escape 0x16, 0x12, 0x02, 0x82, 0x78 #
 ; RV64-NEXT:    addi sp, sp, -32
 ; RV64-NEXT:    .cfi_def_cfa_offset 32
 ; RV64-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
@@ -131,6 +138,7 @@ define i32 @f4() shadowcallstack {
 ; RV64-NEXT:    addi sp, sp, 32
 ; RV64-NEXT:    ld ra, -8(s2)
 ; RV64-NEXT:    addi s2, s2, -8
+; RV64-NEXT:    .cfi_restore s2
 ; RV64-NEXT:    ret
   %res1 = call i32 @bar()
   %res2 = call i32 @bar()
diff --git a/llvm/test/CodeGen/RISCV/shuffle-reverse.ll b/llvm/test/CodeGen/RISCV/shuffle-reverse.ll
index 049e45bba8b6f..dcf1a05f034e2 100644
--- a/llvm/test/CodeGen/RISCV/shuffle-reverse.ll
+++ b/llvm/test/CodeGen/RISCV/shuffle-reverse.ll
@@ -117,7 +117,6 @@ define <32 x i8> @v16i8_2(<16 x i8> %a, <16 x i8> %b) {
 ; RV32-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; RV32-NEXT:    vle8.v v12, (a0)
 ; RV32-NEXT:    vmv1r.v v14, v9
-; RV32-NEXT:    # kill: def $v8 killed $v8 def $v8m2
 ; RV32-NEXT:    vrgather.vv v10, v8, v12
 ; RV32-NEXT:    vid.v v8
 ; RV32-NEXT:    vrsub.vi v8, v8, 15
@@ -138,7 +137,6 @@ define <32 x i8> @v16i8_2(<16 x i8> %a, <16 x i8> %b) {
 ; RV64-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
 ; RV64-NEXT:    vle8.v v12, (a0)
 ; RV64-NEXT:    vmv1r.v v14, v9
-; RV64-NEXT:    # kill: def $v8 killed $v8 def $v8m2
 ; RV64-NEXT:    vrgather.vv v10, v8, v12
 ; RV64-NEXT:    vid.v v8
 ; RV64-NEXT:    vrsub.vi v8, v8, 15
@@ -232,7 +230,6 @@ define <16 x i16> @v8i16_2(<8 x i16> %a, <8 x i16> %b) {
 ; CHECK-LABEL: v8i16_2:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vmv1r.v v12, v9
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8m2
 ; CHECK-NEXT:    vsetivli zero, 16, e16, m2, ta, mu
 ; CHECK-NEXT:    vid.v v14
 ; CHECK-NEXT:    vrsub.vi v16, v14, 15
@@ -267,19 +264,18 @@ define <32 x i16> @v16i16_2(<16 x i16> %a, <16 x i16> %b) {
 ; RV32-NEXT:    addi a0, a0, %lo(.LCPI15_0)
 ; RV32-NEXT:    li a1, 32
 ; RV32-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
-; RV32-NEXT:    vle16.v v16, (a0)
-; RV32-NEXT:    vmv2r.v v20, v10
-; RV32-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m4
-; RV32-NEXT:    vrgather.vv v12, v8, v16
-; RV32-NEXT:    vid.v v8
-; RV32-NEXT:    vrsub.vi v8, v8, 15
+; RV32-NEXT:    vle16.v v20, (a0)
+; RV32-NEXT:    vmv2r.v v16, v10
+; RV32-NEXT:    vmv2r.v v12, v8
+; RV32-NEXT:    vrgather.vv v8, v12, v20
+; RV32-NEXT:    vid.v v12
+; RV32-NEXT:    vrsub.vi v12, v12, 15
 ; RV32-NEXT:    lui a0, 16
 ; RV32-NEXT:    addi a0, a0, -1
 ; RV32-NEXT:    vsetivli zero, 1, e32, mf2, ta, ma
 ; RV32-NEXT:    vmv.s.x v0, a0
 ; RV32-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
-; RV32-NEXT:    vrgather.vv v12, v20, v8, v0.t
-; RV32-NEXT:    vmv.v.v v8, v12
+; RV32-NEXT:    vrgather.vv v8, v16, v12, v0.t
 ; RV32-NEXT:    ret
 ;
 ; RV64-LABEL: v16i16_2:
@@ -288,19 +284,18 @@ define <32 x i16> @v16i16_2(<16 x i16> %a, <16 x i16> %b) {
 ; RV64-NEXT:    addi a0, a0, %lo(.LCPI15_0)
 ; RV64-NEXT:    li a1, 32
 ; RV64-NEXT:    vsetvli zero, a1, e16, m4, ta, ma
-; RV64-NEXT:    vle16.v v16, (a0)
-; RV64-NEXT:    vmv2r.v v20, v10
-; RV64-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m4
-; RV64-NEXT:    vrgather.vv v12, v8, v16
-; RV64-NEXT:    vid.v v8
-; RV64-NEXT:    vrsub.vi v8, v8, 15
+; RV64-NEXT:    vle16.v v20, (a0)
+; RV64-NEXT:    vmv2r.v v16, v10
+; RV64-NEXT:    vmv2r.v v12, v8
+; RV64-NEXT:    vrgather.vv v8, v12, v20
+; RV64-NEXT:    vid.v v12
+; RV64-NEXT:    vrsub.vi v12, v12, 15
 ; RV64-NEXT:    lui a0, 16
 ; RV64-NEXT:    addiw a0, a0, -1
 ; RV64-NEXT:    vsetivli zero, 1, e32, mf2, ta, ma
 ; RV64-NEXT:    vmv.s.x v0, a0
 ; RV64-NEXT:    vsetvli zero, a1, e16, m4, ta, mu
-; RV64-NEXT:    vrgather.vv v12, v20, v8, v0.t
-; RV64-NEXT:    vmv.v.v v8, v12
+; RV64-NEXT:    vrgather.vv v8, v16, v12, v0.t
 ; RV64-NEXT:    ret
   %v32i16 = shufflevector <16 x i16> %a, <16 x i16> %b,  <32 x i32> <i32 31, i32 30, i32 29, i32 28, i32 27, i32 26, i32 25, i32 24, i32 23, i32 22, i32 21, i32 20, i32 19, i32 18, i32 17, i32 16, i32 15, i32 14, i32 13, i32 12, i32 11, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
   ret <32 x i16> %v32i16
@@ -354,7 +349,6 @@ define <8 x i32> @v4i32_2(<4 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: v4i32_2:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vmv1r.v v12, v9
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8m2
 ; CHECK-NEXT:    vsetivli zero, 8, e32, m2, ta, mu
 ; CHECK-NEXT:    vid.v v14
 ; CHECK-NEXT:    vrsub.vi v16, v14, 7
@@ -386,7 +380,6 @@ define <16 x i32> @v8i32_2(<8 x i32> %a, <8 x i32> %b) {
 ; CHECK-LABEL: v8i32_2:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vmv2r.v v16, v10
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m4
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, mu
 ; CHECK-NEXT:    vid.v v20
 ; CHECK-NEXT:    vrsub.vi v24, v20, 15
@@ -475,16 +468,15 @@ define <4 x i64> @v2i64_2(<2 x i64> %a, < 2 x i64> %b) {
 ; CHECK-LABEL: v2i64_2:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-NEXT:    vslidedown.vi v12, v8, 1
+; CHECK-NEXT:    vslidedown.vi v10, v8, 1
 ; CHECK-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
-; CHECK-NEXT:    vslideup.vi v12, v8, 1
+; CHECK-NEXT:    vslideup.vi v10, v8, 1
 ; CHECK-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-NEXT:    vslidedown.vi v10, v9, 1
+; CHECK-NEXT:    vslidedown.vi v8, v9, 1
 ; CHECK-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
-; CHECK-NEXT:    vslideup.vi v10, v9, 1
+; CHECK-NEXT:    vslideup.vi v8, v9, 1
 ; CHECK-NEXT:    vsetivli zero, 4, e64, m2, tu, ma
-; CHECK-NEXT:    vslideup.vi v10, v12, 2
-; CHECK-NEXT:    vmv2r.v v8, v10
+; CHECK-NEXT:    vslideup.vi v8, v10, 2
 ; CHECK-NEXT:    ret
   %v4i64 = shufflevector <2 x i64> %a, <2 x i64> %b, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
   ret <4 x i64> %v4i64
@@ -517,16 +509,15 @@ define <8 x i64> @v4i64_2(<4 x i64> %a, <4 x i64> %b) {
 ; RV32-LABEL: v4i64_2:
 ; RV32:       # %bb.0:
 ; RV32-NEXT:    vmv2r.v v16, v10
-; RV32-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m4
 ; RV32-NEXT:    vsetivli zero, 8, e16, m1, ta, ma
-; RV32-NEXT:    vid.v v20
-; RV32-NEXT:    vrsub.vi v21, v20, 7
+; RV32-NEXT:    vid.v v18
+; RV32-NEXT:    vrsub.vi v19, v18, 7
 ; RV32-NEXT:    vsetvli zero, zero, e64, m4, ta, ma
-; RV32-NEXT:    vrgatherei16.vv v12, v8, v21
+; RV32-NEXT:    vrgatherei16.vv v12, v8, v19
 ; RV32-NEXT:    li a0, 15
 ; RV32-NEXT:    vmv.s.x v0, a0
 ; RV32-NEXT:    vsetvli zero, zero, e16, m1, ta, ma
-; RV32-NEXT:    vrsub.vi v8, v20, 3
+; RV32-NEXT:    vrsub.vi v8, v18, 3
 ; RV32-NEXT:    vsetvli zero, zero, e64, m4, ta, mu
 ; RV32-NEXT:    vrgatherei16.vv v12, v16, v8, v0.t
 ; RV32-NEXT:    vmv.v.v v8, v12
@@ -535,7 +526,6 @@ define <8 x i64> @v4i64_2(<4 x i64> %a, <4 x i64> %b) {
 ; RV64-LABEL: v4i64_2:
 ; RV64:       # %bb.0:
 ; RV64-NEXT:    vmv2r.v v16, v10
-; RV64-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m4
 ; RV64-NEXT:    vsetivli zero, 8, e64, m4, ta, mu
 ; RV64-NEXT:    vid.v v20
 ; RV64-NEXT:    vrsub.vi v24, v20, 7
@@ -628,7 +618,6 @@ define <16 x half> @v8f16_2(<8 x half> %a, <8 x half> %b) {
 ; CHECK-LABEL: v8f16_2:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vmv1r.v v12, v9
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8m2
 ; CHECK-NEXT:    vsetivli zero, 16, e16, m2, ta, mu
 ; CHECK-NEXT:    vid.v v14
 ; CHECK-NEXT:    vrsub.vi v16, v14, 15
@@ -659,7 +648,6 @@ define <16 x half> @v16f16(<16 x half> %a) {
 define <32 x half> @v16f16_2(<16 x half> %a) {
 ; CHECK-LABEL: v16f16_2:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m4
 ; CHECK-NEXT:    lui a0, %hi(.LCPI35_0)
 ; CHECK-NEXT:    addi a0, a0, %lo(.LCPI35_0)
 ; CHECK-NEXT:    li a1, 32
@@ -723,7 +711,6 @@ define <8 x float> @v4f32_2(<4 x float> %a, <4 x float> %b) {
 ; CHECK-LABEL: v4f32_2:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vmv1r.v v12, v9
-; CHECK-NEXT:    # kill: def $v8 killed $v8 def $v8m2
 ; CHECK-NEXT:    vsetivli zero, 8, e32, m2, ta, mu
 ; CHECK-NEXT:    vid.v v14
 ; CHECK-NEXT:    vrsub.vi v16, v14, 7
@@ -755,7 +742,6 @@ define <16 x float> @v8f32_2(<8 x float> %a, <8 x float> %b) {
 ; CHECK-LABEL: v8f32_2:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vmv2r.v v16, v10
-; CHECK-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m4
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, mu
 ; CHECK-NEXT:    vid.v v20
 ; CHECK-NEXT:    vrsub.vi v24, v20, 15
@@ -787,16 +773,15 @@ define <4 x double> @v2f64_2(<2 x double> %a, < 2 x double> %b) {
 ; CHECK-LABEL: v2f64_2:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-NEXT:    vslidedown.vi v12, v8, 1
+; CHECK-NEXT:    vslidedown.vi v10, v8, 1
 ; CHECK-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
-; CHECK-NEXT:    vslideup.vi v12, v8, 1
+; CHECK-NEXT:    vslideup.vi v10, v8, 1
 ; CHECK-NEXT:    vsetivli zero, 1, e64, m1, ta, ma
-; CHECK-NEXT:    vslidedown.vi v10, v9, 1
+; CHECK-NEXT:    vslidedown.vi v8, v9, 1
 ; CHECK-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
-; CHECK-NEXT:    vslideup.vi v10, v9, 1
+; CHECK-NEXT:    vslideup.vi v8, v9, 1
 ; CHECK-NEXT:    vsetivli zero, 4, e64, m2, tu, ma
-; CHECK-NEXT:    vslideup.vi v10, v12, 2
-; CHECK-NEXT:    vmv2r.v v8, v10
+; CHECK-NEXT:    vslideup.vi v8, v10, 2
 ; CHECK-NEXT:    ret
   %v4f64 = shufflevector <2 x double> %a, <2 x double> %b, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
   ret <4 x double> %v4f64
@@ -829,16 +814,15 @@ define <8 x double> @v4f64_2(<4 x double> %a, <4 x double> %b) {
 ; RV32-LABEL: v4f64_2:
 ; RV32:       # %bb.0:
 ; RV32-NEXT:    vmv2r.v v16, v10
-; RV32-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m4
 ; RV32-NEXT:    vsetivli zero, 8, e16, m1, ta, ma
-; RV32-NEXT:    vid.v v20
-; RV32-NEXT:    vrsub.vi v21, v20, 7
+; RV32-NEXT:    vid.v v18
+; RV32-NEXT:    vrsub.vi v19, v18, 7
 ; RV32-NEXT:    vsetvli zero, zero, e64, m4, ta, ma
-; RV32-NEXT:    vrgatherei16.vv v12, v8, v21
+; RV32-NEXT:    vrgatherei16.vv v12, v8, v19
 ; RV32-NEXT:    li a0, 15
 ; RV32-NEXT:    vmv.s.x v0, a0
 ; RV32-NEXT:    vsetvli zero, zero, e16, m1, ta, ma
-; RV32-NEXT:    vrsub.vi v8, v20, 3
+; RV32-NEXT:    vrsub.vi v8, v18, 3
 ; RV32-NEXT:    vsetvli zero, zero, e64, m4, ta, mu
 ; RV32-NEXT:    vrgatherei16.vv v12, v16, v8, v0.t
 ; RV32-NEXT:    vmv.v.v v8, v12
@@ -847,7 +831,6 @@ define <8 x double> @v4f64_2(<4 x double> %a, <4 x double> %b) {
 ; RV64-LABEL: v4f64_2:
 ; RV64:       # %bb.0:
 ; RV64-NEXT:    vmv2r.v v16, v10
-; RV64-NEXT:    # kill: def $v8m2 killed $v8m2 def $v8m4
 ; RV64-NEXT:    vsetivli zero, 8, e64, m4, ta, mu
 ; RV64-NEXT:    vid.v v20
 ; RV64-NEXT:    vrsub.vi v24, v20, 7
diff --git a/llvm/test/CodeGen/RISCV/split-offsets.ll b/llvm/test/CodeGen/RISCV/split-offsets.ll
index 365b76b8c6636..e597bd1bfe80e 100644
--- a/llvm/test/CodeGen/RISCV/split-offsets.ll
+++ b/llvm/test/CodeGen/RISCV/split-offsets.ll
@@ -61,7 +61,7 @@ define void @test2(ptr %sp, ptr %t, i32 %n) {
 ; RV32I-NEXT:    addi a4, a4, -1920
 ; RV32I-NEXT:    add a1, a1, a4
 ; RV32I-NEXT:    add a0, a0, a4
-; RV32I-NEXT:    bge a3, a2, .LBB1_2
+; RV32I-NEXT:    blez a2, .LBB1_2
 ; RV32I-NEXT:  .LBB1_1: # %while_body
 ; RV32I-NEXT:    # =>This Inner Loop Header: Depth=1
 ; RV32I-NEXT:    addi a4, a3, 1
@@ -70,7 +70,7 @@ define void @test2(ptr %sp, ptr %t, i32 %n) {
 ; RV32I-NEXT:    sw a4, 0(a1)
 ; RV32I-NEXT:    sw a3, 4(a1)
 ; RV32I-NEXT:    mv a3, a4
-; RV32I-NEXT:    blt a3, a2, .LBB1_1
+; RV32I-NEXT:    blt a4, a2, .LBB1_1
 ; RV32I-NEXT:  .LBB1_2: # %while_end
 ; RV32I-NEXT:    ret
 ;
@@ -83,7 +83,7 @@ define void @test2(ptr %sp, ptr %t, i32 %n) {
 ; RV64I-NEXT:    add a1, a1, a4
 ; RV64I-NEXT:    add a0, a0, a4
 ; RV64I-NEXT:    sext.w a2, a2
-; RV64I-NEXT:    bge a3, a2, .LBB1_2
+; RV64I-NEXT:    blez a2, .LBB1_2
 ; RV64I-NEXT:  .LBB1_1: # %while_body
 ; RV64I-NEXT:    # =>This Inner Loop Header: Depth=1
 ; RV64I-NEXT:    addiw a4, a3, 1
@@ -92,7 +92,7 @@ define void @test2(ptr %sp, ptr %t, i32 %n) {
 ; RV64I-NEXT:    sw a4, 0(a1)
 ; RV64I-NEXT:    sw a3, 4(a1)
 ; RV64I-NEXT:    mv a3, a4
-; RV64I-NEXT:    blt a3, a2, .LBB1_1
+; RV64I-NEXT:    blt a4, a2, .LBB1_1
 ; RV64I-NEXT:  .LBB1_2: # %while_end
 ; RV64I-NEXT:    ret
 entry:
diff --git a/llvm/test/CodeGen/RISCV/srem-lkk.ll b/llvm/test/CodeGen/RISCV/srem-lkk.ll
index 1b8dee85296b0..24e740fd143d1 100644
--- a/llvm/test/CodeGen/RISCV/srem-lkk.ll
+++ b/llvm/test/CodeGen/RISCV/srem-lkk.ll
@@ -51,7 +51,7 @@ define i32 @fold_srem_positive_odd(i32 %x) nounwind {
 ; RV64IM-NEXT:    sraiw a1, a1, 6
 ; RV64IM-NEXT:    add a1, a1, a2
 ; RV64IM-NEXT:    li a2, 95
-; RV64IM-NEXT:    mulw a1, a1, a2
+; RV64IM-NEXT:    mul a1, a1, a2
 ; RV64IM-NEXT:    subw a0, a0, a1
 ; RV64IM-NEXT:    ret
   %1 = srem i32 %x, 95
@@ -99,7 +99,7 @@ define i32 @fold_srem_positive_even(i32 %x) nounwind {
 ; RV64IM-NEXT:    srai a1, a1, 40
 ; RV64IM-NEXT:    add a1, a1, a2
 ; RV64IM-NEXT:    li a2, 1060
-; RV64IM-NEXT:    mulw a1, a1, a2
+; RV64IM-NEXT:    mul a1, a1, a2
 ; RV64IM-NEXT:    subw a0, a0, a1
 ; RV64IM-NEXT:    ret
   %1 = srem i32 %x, 1060
@@ -147,7 +147,7 @@ define i32 @fold_srem_negative_odd(i32 %x) nounwind {
 ; RV64IM-NEXT:    srai a1, a1, 40
 ; RV64IM-NEXT:    add a1, a1, a2
 ; RV64IM-NEXT:    li a2, -723
-; RV64IM-NEXT:    mulw a1, a1, a2
+; RV64IM-NEXT:    mul a1, a1, a2
 ; RV64IM-NEXT:    subw a0, a0, a1
 ; RV64IM-NEXT:    ret
   %1 = srem i32 %x, -723
@@ -199,7 +199,7 @@ define i32 @fold_srem_negative_even(i32 %x) nounwind {
 ; RV64IM-NEXT:    add a1, a1, a2
 ; RV64IM-NEXT:    lui a2, 1048570
 ; RV64IM-NEXT:    addiw a2, a2, 1595
-; RV64IM-NEXT:    mulw a1, a1, a2
+; RV64IM-NEXT:    mul a1, a1, a2
 ; RV64IM-NEXT:    subw a0, a0, a1
 ; RV64IM-NEXT:    ret
   %1 = srem i32 %x, -22981
@@ -277,7 +277,7 @@ define i32 @combine_srem_sdiv(i32 %x) nounwind {
 ; RV64IM-NEXT:    sraiw a1, a1, 6
 ; RV64IM-NEXT:    add a1, a1, a2
 ; RV64IM-NEXT:    li a2, 95
-; RV64IM-NEXT:    mulw a2, a1, a2
+; RV64IM-NEXT:    mul a2, a1, a2
 ; RV64IM-NEXT:    add a0, a0, a1
 ; RV64IM-NEXT:    subw a0, a0, a2
 ; RV64IM-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/srem-seteq-illegal-types.ll b/llvm/test/CodeGen/RISCV/srem-seteq-illegal-types.ll
index 9e00b1282796a..91efe2c223a07 100644
--- a/llvm/test/CodeGen/RISCV/srem-seteq-illegal-types.ll
+++ b/llvm/test/CodeGen/RISCV/srem-seteq-illegal-types.ll
@@ -64,7 +64,7 @@ define i1 @test_srem_odd(i29 %X) nounwind {
 ; RV64M:       # %bb.0:
 ; RV64M-NEXT:    lui a1, 128424
 ; RV64M-NEXT:    addiw a1, a1, 331
-; RV64M-NEXT:    mulw a0, a0, a1
+; RV64M-NEXT:    mul a0, a0, a1
 ; RV64M-NEXT:    lui a1, 662
 ; RV64M-NEXT:    addiw a1, a1, -83
 ; RV64M-NEXT:    add a0, a0, a1
@@ -94,7 +94,7 @@ define i1 @test_srem_odd(i29 %X) nounwind {
 ; RV64MV:       # %bb.0:
 ; RV64MV-NEXT:    lui a1, 128424
 ; RV64MV-NEXT:    addiw a1, a1, 331
-; RV64MV-NEXT:    mulw a0, a0, a1
+; RV64MV-NEXT:    mul a0, a0, a1
 ; RV64MV-NEXT:    lui a1, 662
 ; RV64MV-NEXT:    addiw a1, a1, -83
 ; RV64MV-NEXT:    add a0, a0, a1
@@ -167,7 +167,7 @@ define i1 @test_srem_even(i4 %X) nounwind {
 ; RV64M-NEXT:    srli a1, a1, 63
 ; RV64M-NEXT:    add a1, a2, a1
 ; RV64M-NEXT:    li a2, 6
-; RV64M-NEXT:    mulw a1, a1, a2
+; RV64M-NEXT:    mul a1, a1, a2
 ; RV64M-NEXT:    subw a0, a0, a1
 ; RV64M-NEXT:    andi a0, a0, 15
 ; RV64M-NEXT:    addi a0, a0, -1
@@ -203,7 +203,7 @@ define i1 @test_srem_even(i4 %X) nounwind {
 ; RV64MV-NEXT:    srli a1, a1, 63
 ; RV64MV-NEXT:    add a1, a2, a1
 ; RV64MV-NEXT:    li a2, 6
-; RV64MV-NEXT:    mulw a1, a1, a2
+; RV64MV-NEXT:    mul a1, a1, a2
 ; RV64MV-NEXT:    subw a0, a0, a1
 ; RV64MV-NEXT:    andi a0, a0, 15
 ; RV64MV-NEXT:    addi a0, a0, -1
diff --git a/llvm/test/CodeGen/RISCV/srem-vector-lkk.ll b/llvm/test/CodeGen/RISCV/srem-vector-lkk.ll
index 50d4a55842f1a..231c066de5437 100644
--- a/llvm/test/CodeGen/RISCV/srem-vector-lkk.ll
+++ b/llvm/test/CodeGen/RISCV/srem-vector-lkk.ll
@@ -159,7 +159,7 @@ define <4 x i16> @fold_srem_vec_1(<4 x i16> %x) nounwind {
 ; RV64IM-NEXT:    lui a6, %hi(.LCPI0_1)
 ; RV64IM-NEXT:    ld a6, %lo(.LCPI0_1)(a6)
 ; RV64IM-NEXT:    li a7, 95
-; RV64IM-NEXT:    mulw a3, a3, a7
+; RV64IM-NEXT:    mul a3, a3, a7
 ; RV64IM-NEXT:    subw a2, a2, a3
 ; RV64IM-NEXT:    mulh a3, a1, a6
 ; RV64IM-NEXT:    sub a3, a3, a1
@@ -169,7 +169,7 @@ define <4 x i16> @fold_srem_vec_1(<4 x i16> %x) nounwind {
 ; RV64IM-NEXT:    lui a6, %hi(.LCPI0_2)
 ; RV64IM-NEXT:    ld a6, %lo(.LCPI0_2)(a6)
 ; RV64IM-NEXT:    li a7, -124
-; RV64IM-NEXT:    mulw a3, a3, a7
+; RV64IM-NEXT:    mul a3, a3, a7
 ; RV64IM-NEXT:    subw a1, a1, a3
 ; RV64IM-NEXT:    mulh a3, a5, a6
 ; RV64IM-NEXT:    srli a6, a3, 63
@@ -178,14 +178,14 @@ define <4 x i16> @fold_srem_vec_1(<4 x i16> %x) nounwind {
 ; RV64IM-NEXT:    lui a6, %hi(.LCPI0_3)
 ; RV64IM-NEXT:    ld a6, %lo(.LCPI0_3)(a6)
 ; RV64IM-NEXT:    li a7, 98
-; RV64IM-NEXT:    mulw a3, a3, a7
+; RV64IM-NEXT:    mul a3, a3, a7
 ; RV64IM-NEXT:    subw a5, a5, a3
 ; RV64IM-NEXT:    mulh a3, a4, a6
 ; RV64IM-NEXT:    srli a6, a3, 63
 ; RV64IM-NEXT:    srli a3, a3, 7
 ; RV64IM-NEXT:    add a3, a3, a6
 ; RV64IM-NEXT:    li a6, -1003
-; RV64IM-NEXT:    mulw a3, a3, a6
+; RV64IM-NEXT:    mul a3, a3, a6
 ; RV64IM-NEXT:    subw a4, a4, a3
 ; RV64IM-NEXT:    sh a4, 6(a0)
 ; RV64IM-NEXT:    sh a5, 4(a0)
@@ -338,28 +338,28 @@ define <4 x i16> @fold_srem_vec_2(<4 x i16> %x) nounwind {
 ; RV64IM-NEXT:    srli a6, a6, 6
 ; RV64IM-NEXT:    add a6, a6, a7
 ; RV64IM-NEXT:    li a7, 95
-; RV64IM-NEXT:    mulw a6, a6, a7
+; RV64IM-NEXT:    mul a6, a6, a7
 ; RV64IM-NEXT:    subw a2, a2, a6
 ; RV64IM-NEXT:    mulh a6, a1, a3
 ; RV64IM-NEXT:    add a6, a6, a1
 ; RV64IM-NEXT:    srli t0, a6, 63
 ; RV64IM-NEXT:    srli a6, a6, 6
 ; RV64IM-NEXT:    add a6, a6, t0
-; RV64IM-NEXT:    mulw a6, a6, a7
+; RV64IM-NEXT:    mul a6, a6, a7
 ; RV64IM-NEXT:    subw a1, a1, a6
 ; RV64IM-NEXT:    mulh a6, a5, a3
 ; RV64IM-NEXT:    add a6, a6, a5
 ; RV64IM-NEXT:    srli t0, a6, 63
 ; RV64IM-NEXT:    srli a6, a6, 6
 ; RV64IM-NEXT:    add a6, a6, t0
-; RV64IM-NEXT:    mulw a6, a6, a7
+; RV64IM-NEXT:    mul a6, a6, a7
 ; RV64IM-NEXT:    subw a5, a5, a6
 ; RV64IM-NEXT:    mulh a3, a4, a3
 ; RV64IM-NEXT:    add a3, a3, a4
 ; RV64IM-NEXT:    srli a6, a3, 63
 ; RV64IM-NEXT:    srli a3, a3, 6
 ; RV64IM-NEXT:    add a3, a3, a6
-; RV64IM-NEXT:    mulw a3, a3, a7
+; RV64IM-NEXT:    mul a3, a3, a7
 ; RV64IM-NEXT:    subw a4, a4, a3
 ; RV64IM-NEXT:    sh a4, 6(a0)
 ; RV64IM-NEXT:    sh a5, 4(a0)
@@ -574,25 +574,25 @@ define <4 x i16> @combine_srem_sdiv(<4 x i16> %x) nounwind {
 ; RV64IM-NEXT:    srai a6, a6, 6
 ; RV64IM-NEXT:    add a6, a6, a7
 ; RV64IM-NEXT:    li a7, 95
-; RV64IM-NEXT:    mulw t0, a6, a7
+; RV64IM-NEXT:    mul t0, a6, a7
 ; RV64IM-NEXT:    mulh t1, a1, a3
 ; RV64IM-NEXT:    add t1, t1, a1
 ; RV64IM-NEXT:    srli t2, t1, 63
 ; RV64IM-NEXT:    srai t1, t1, 6
 ; RV64IM-NEXT:    add t1, t1, t2
-; RV64IM-NEXT:    mulw t2, t1, a7
+; RV64IM-NEXT:    mul t2, t1, a7
 ; RV64IM-NEXT:    mulh t3, a5, a3
 ; RV64IM-NEXT:    add t3, t3, a5
 ; RV64IM-NEXT:    srli t4, t3, 63
 ; RV64IM-NEXT:    srai t3, t3, 6
 ; RV64IM-NEXT:    add t3, t3, t4
-; RV64IM-NEXT:    mulw t4, t3, a7
+; RV64IM-NEXT:    mul t4, t3, a7
 ; RV64IM-NEXT:    mulh a3, a4, a3
 ; RV64IM-NEXT:    add a3, a3, a4
 ; RV64IM-NEXT:    srli t5, a3, 63
 ; RV64IM-NEXT:    srai a3, a3, 6
 ; RV64IM-NEXT:    add a3, a3, t5
-; RV64IM-NEXT:    mulw a7, a3, a7
+; RV64IM-NEXT:    mul a7, a3, a7
 ; RV64IM-NEXT:    add a3, a4, a3
 ; RV64IM-NEXT:    subw a3, a3, a7
 ; RV64IM-NEXT:    add a5, a5, t3
@@ -740,7 +740,7 @@ define <4 x i16> @dont_fold_srem_power_of_two(<4 x i16> %x) nounwind {
 ; RV64IM-NEXT:    srli a3, a3, 6
 ; RV64IM-NEXT:    add a3, a3, a6
 ; RV64IM-NEXT:    li a6, 95
-; RV64IM-NEXT:    mulw a3, a3, a6
+; RV64IM-NEXT:    mul a3, a3, a6
 ; RV64IM-NEXT:    subw a2, a2, a3
 ; RV64IM-NEXT:    srli a3, a1, 58
 ; RV64IM-NEXT:    add a3, a1, a3
@@ -893,7 +893,7 @@ define <4 x i16> @dont_fold_srem_one(<4 x i16> %x) nounwind {
 ; RV64IM-NEXT:    lui a5, %hi(.LCPI4_1)
 ; RV64IM-NEXT:    ld a5, %lo(.LCPI4_1)(a5)
 ; RV64IM-NEXT:    li a6, 23
-; RV64IM-NEXT:    mulw a3, a3, a6
+; RV64IM-NEXT:    mul a3, a3, a6
 ; RV64IM-NEXT:    subw a2, a2, a3
 ; RV64IM-NEXT:    mulh a3, a1, a5
 ; RV64IM-NEXT:    srli a5, a3, 63
@@ -902,7 +902,7 @@ define <4 x i16> @dont_fold_srem_one(<4 x i16> %x) nounwind {
 ; RV64IM-NEXT:    lui a5, %hi(.LCPI4_2)
 ; RV64IM-NEXT:    ld a5, %lo(.LCPI4_2)(a5)
 ; RV64IM-NEXT:    li a6, 654
-; RV64IM-NEXT:    mulw a3, a3, a6
+; RV64IM-NEXT:    mul a3, a3, a6
 ; RV64IM-NEXT:    subw a1, a1, a3
 ; RV64IM-NEXT:    mulh a3, a4, a5
 ; RV64IM-NEXT:    srli a5, a3, 63
@@ -910,7 +910,7 @@ define <4 x i16> @dont_fold_srem_one(<4 x i16> %x) nounwind {
 ; RV64IM-NEXT:    add a3, a3, a5
 ; RV64IM-NEXT:    lui a5, 1
 ; RV64IM-NEXT:    addiw a5, a5, 1327
-; RV64IM-NEXT:    mulw a3, a3, a5
+; RV64IM-NEXT:    mul a3, a3, a5
 ; RV64IM-NEXT:    subw a4, a4, a3
 ; RV64IM-NEXT:    sh zero, 0(a0)
 ; RV64IM-NEXT:    sh a4, 6(a0)
@@ -1045,7 +1045,7 @@ define <4 x i16> @dont_fold_urem_i16_smax(<4 x i16> %x) nounwind {
 ; RV64IM-NEXT:    li a5, 23
 ; RV64IM-NEXT:    lui a6, %hi(.LCPI5_1)
 ; RV64IM-NEXT:    ld a6, %lo(.LCPI5_1)(a6)
-; RV64IM-NEXT:    mulw a3, a3, a5
+; RV64IM-NEXT:    mul a3, a3, a5
 ; RV64IM-NEXT:    lh a1, 8(a1)
 ; RV64IM-NEXT:    subw a2, a2, a3
 ; RV64IM-NEXT:    mulh a3, a4, a6
@@ -1054,7 +1054,7 @@ define <4 x i16> @dont_fold_urem_i16_smax(<4 x i16> %x) nounwind {
 ; RV64IM-NEXT:    add a3, a3, a5
 ; RV64IM-NEXT:    lui a5, 1
 ; RV64IM-NEXT:    addiw a5, a5, 1327
-; RV64IM-NEXT:    mulw a3, a3, a5
+; RV64IM-NEXT:    mul a3, a3, a5
 ; RV64IM-NEXT:    subw a4, a4, a3
 ; RV64IM-NEXT:    srli a3, a1, 49
 ; RV64IM-NEXT:    add a3, a1, a3
diff --git a/llvm/test/CodeGen/RISCV/ssub_sat.ll b/llvm/test/CodeGen/RISCV/ssub_sat.ll
index 9571ca317316a..7732661dd1306 100644
--- a/llvm/test/CodeGen/RISCV/ssub_sat.ll
+++ b/llvm/test/CodeGen/RISCV/ssub_sat.ll
@@ -37,7 +37,7 @@ define signext i32 @func(i32 signext %x, i32 signext %y) nounwind {
 ; RV64I-NEXT:    ret
 ; RV64I-NEXT:  .LBB0_3:
 ; RV64I-NEXT:    mv a0, a2
-; RV64I-NEXT:    blt a1, a0, .LBB0_2
+; RV64I-NEXT:    blt a1, a2, .LBB0_2
 ; RV64I-NEXT:  .LBB0_4:
 ; RV64I-NEXT:    lui a0, 524288
 ; RV64I-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/ssub_sat_plus.ll b/llvm/test/CodeGen/RISCV/ssub_sat_plus.ll
index 32e794fc778e6..6ff61ff415216 100644
--- a/llvm/test/CodeGen/RISCV/ssub_sat_plus.ll
+++ b/llvm/test/CodeGen/RISCV/ssub_sat_plus.ll
@@ -40,7 +40,7 @@ define i32 @func32(i32 %x, i32 %y, i32 %z) nounwind {
 ; RV64I-NEXT:    ret
 ; RV64I-NEXT:  .LBB0_3:
 ; RV64I-NEXT:    mv a0, a2
-; RV64I-NEXT:    blt a1, a0, .LBB0_2
+; RV64I-NEXT:    blt a1, a2, .LBB0_2
 ; RV64I-NEXT:  .LBB0_4:
 ; RV64I-NEXT:    lui a0, 524288
 ; RV64I-NEXT:    ret
@@ -128,7 +128,7 @@ define i16 @func16(i16 %x, i16 %y, i16 %z) nounwind {
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    slli a0, a0, 48
 ; RV64I-NEXT:    srai a0, a0, 48
-; RV64I-NEXT:    mulw a1, a1, a2
+; RV64I-NEXT:    mul a1, a1, a2
 ; RV64I-NEXT:    slli a1, a1, 48
 ; RV64I-NEXT:    srai a1, a1, 48
 ; RV64I-NEXT:    sub a0, a0, a1
@@ -164,7 +164,7 @@ define i16 @func16(i16 %x, i16 %y, i16 %z) nounwind {
 ; RV64IZbb-LABEL: func16:
 ; RV64IZbb:       # %bb.0:
 ; RV64IZbb-NEXT:    sext.h a0, a0
-; RV64IZbb-NEXT:    mulw a1, a1, a2
+; RV64IZbb-NEXT:    mul a1, a1, a2
 ; RV64IZbb-NEXT:    sext.h a1, a1
 ; RV64IZbb-NEXT:    sub a0, a0, a1
 ; RV64IZbb-NEXT:    lui a1, 8
@@ -206,7 +206,7 @@ define i8 @func8(i8 %x, i8 %y, i8 %z) nounwind {
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    slli a0, a0, 56
 ; RV64I-NEXT:    srai a0, a0, 56
-; RV64I-NEXT:    mulw a1, a1, a2
+; RV64I-NEXT:    mul a1, a1, a2
 ; RV64I-NEXT:    slli a1, a1, 56
 ; RV64I-NEXT:    srai a1, a1, 56
 ; RV64I-NEXT:    sub a0, a0, a1
@@ -240,7 +240,7 @@ define i8 @func8(i8 %x, i8 %y, i8 %z) nounwind {
 ; RV64IZbb-LABEL: func8:
 ; RV64IZbb:       # %bb.0:
 ; RV64IZbb-NEXT:    sext.b a0, a0
-; RV64IZbb-NEXT:    mulw a1, a1, a2
+; RV64IZbb-NEXT:    mul a1, a1, a2
 ; RV64IZbb-NEXT:    sext.b a1, a1
 ; RV64IZbb-NEXT:    sub a0, a0, a1
 ; RV64IZbb-NEXT:    li a1, 127
@@ -281,7 +281,7 @@ define i4 @func4(i4 %x, i4 %y, i4 %z) nounwind {
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    slli a0, a0, 60
 ; RV64I-NEXT:    srai a0, a0, 60
-; RV64I-NEXT:    mulw a1, a1, a2
+; RV64I-NEXT:    mul a1, a1, a2
 ; RV64I-NEXT:    slli a1, a1, 60
 ; RV64I-NEXT:    srai a1, a1, 60
 ; RV64I-NEXT:    sub a0, a0, a1
@@ -318,7 +318,7 @@ define i4 @func4(i4 %x, i4 %y, i4 %z) nounwind {
 ; RV64IZbb:       # %bb.0:
 ; RV64IZbb-NEXT:    slli a0, a0, 60
 ; RV64IZbb-NEXT:    srai a0, a0, 60
-; RV64IZbb-NEXT:    mulw a1, a1, a2
+; RV64IZbb-NEXT:    mul a1, a1, a2
 ; RV64IZbb-NEXT:    slli a1, a1, 60
 ; RV64IZbb-NEXT:    srai a1, a1, 60
 ; RV64IZbb-NEXT:    sub a0, a0, a1
diff --git a/llvm/test/CodeGen/RISCV/stack-slot-coloring.mir b/llvm/test/CodeGen/RISCV/stack-slot-coloring.mir
new file mode 100644
index 0000000000000..1fb2b7685bafb
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/stack-slot-coloring.mir
@@ -0,0 +1,213 @@
+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -march=riscv32 -run-pass=greedy,virtregrewriter,stack-slot-coloring %s -o - 2>&1 | FileCheck %s
+
+--- |
+  define dso_local i32 @main() local_unnamed_addr {
+  entry:
+    %a = alloca i32, align 4
+    ret i32 0
+  }
+
+...
+---
+name:            main
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+callsEHReturn:   false
+callsUnwindInit: false
+hasEHCatchret:   false
+hasEHScopes:     false
+hasEHFunclets:   false
+failsVerification: false
+tracksDebugUserValues: false
+registers:
+  - { id: 0, class: gpr, preferred-register: '' }
+  - { id: 1, class: gpr, preferred-register: '' }
+liveins:         []
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    4
+  adjustsStack:    false
+  hasCalls:        false
+  stackProtector:  ''
+  functionContext: ''
+  maxCallFrameSize: 4294967295
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  hasTailCall:     false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:
+  - { id: 0, name: a, type: default, offset: 0, size: 4, alignment: 4,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+callSites:       []
+debugValueSubstitutions: []
+constants:       []
+machineFunctionInfo:
+  varArgsFrameIndex: 0
+  varArgsSaveSize: 0
+body:             |
+  bb.0.entry:
+    ; CHECK-LABEL: name: main
+    ; CHECK: $x10 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x11 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x12 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x13 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x14 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x15 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x16 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x17 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x5 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x6 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x7 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x28 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x29 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x30 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x31 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x8 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x9 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x18 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x19 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x20 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x21 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x22 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x23 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x24 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x25 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x26 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: $x27 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: renamable $x1 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: SW killed renamable $x1, %stack.1, 0 :: (store (s32) into %stack.1)
+    ; CHECK-NEXT: renamable $x1 = LB %stack.0.a, 0 :: (volatile dereferenceable load (s8) from %ir.a)
+    ; CHECK-NEXT: SW killed renamable $x1, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: renamable $x1 = LW %stack.1, 0 :: (load (s32) from %stack.1)
+    ; CHECK-NEXT: SW killed renamable $x1, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: renamable $x1 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    ; CHECK-NEXT: SW killed renamable $x1, %stack.1, 0 :: (store (s32) into %stack.1)
+    ; CHECK-NEXT: renamable $x1 = LB %stack.0.a, 0 :: (volatile dereferenceable load (s8) from %ir.a)
+    ; CHECK-NEXT: SW killed renamable $x1, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: renamable $x1 = LW %stack.1, 0 :: (load (s32) from %stack.1)
+    ; CHECK-NEXT: SW killed renamable $x1, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x10, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x11, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x12, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x13, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x14, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x15, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x16, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x17, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x5, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x6, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x7, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x28, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x29, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x30, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x31, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x8, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x9, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x18, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x19, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x20, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x21, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x22, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x23, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x24, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x25, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x26, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: SW $x27, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    ; CHECK-NEXT: $x10 = COPY $x0
+    ; CHECK-NEXT: PseudoRET implicit $x10
+    $x10 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x11 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x12 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x13 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x14 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x15 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x16 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x17 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x5 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x6 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x7 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x28 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x29 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x30 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x31 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x8 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x9 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x18 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x19 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x20 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x21 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x22 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x23 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x24 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x25 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x26 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+    $x27 = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+
+    ; First vreg load
+    %1:gpr = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+
+    ; First faulty sequence; %1 spilt
+    %12:gpr = LB %stack.0.a, 0 :: (volatile dereferenceable load (s8) from %ir.a)
+    SW %12, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+
+    ; Store %1 to avoid it being optimised out, will result in a load-from-spill
+    SW %1, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+
+    ; That code sequence a second time, to generate a second spill slot that
+    ; will get coloured and merged.
+    %2:gpr = LW %stack.0.a, 0 :: (volatile dereferenceable load (s32) from %ir.a)
+
+    %22:gpr = LB %stack.0.a, 0 :: (volatile dereferenceable load (s8) from %ir.a)
+    SW %22, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+
+    SW %2, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+
+    SW $x10, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x11, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x12, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x13, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x14, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x15, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x16, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x17, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x5, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x6, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x7, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x28, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x29, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x30, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x31, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x8, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x9, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x18, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x19, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x20, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x21, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x22, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x23, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x24, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x25, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x26, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    SW $x27, %stack.0.a, 0 :: (volatile store (s32) into %ir.a)
+    $x10 = COPY $x0
+    PseudoRET implicit killed $x10
+
+...
diff --git a/llvm/test/CodeGen/RISCV/uadd_sat_plus.ll b/llvm/test/CodeGen/RISCV/uadd_sat_plus.ll
index 7a8c4c04fd0e1..219f0daf270bc 100644
--- a/llvm/test/CodeGen/RISCV/uadd_sat_plus.ll
+++ b/llvm/test/CodeGen/RISCV/uadd_sat_plus.ll
@@ -22,7 +22,7 @@ define i32 @func32(i32 %x, i32 %y, i32 %z) nounwind {
 ;
 ; RV64I-LABEL: func32:
 ; RV64I:       # %bb.0:
-; RV64I-NEXT:    mulw a1, a1, a2
+; RV64I-NEXT:    mul a1, a1, a2
 ; RV64I-NEXT:    addw a1, a0, a1
 ; RV64I-NEXT:    sext.w a0, a0
 ; RV64I-NEXT:    sltu a0, a1, a0
@@ -144,7 +144,7 @@ define i16 @func16(i16 %x, i16 %y, i16 %z) nounwind {
 ; RV64IZbb-LABEL: func16:
 ; RV64IZbb:       # %bb.0:
 ; RV64IZbb-NEXT:    zext.h a0, a0
-; RV64IZbb-NEXT:    mulw a1, a1, a2
+; RV64IZbb-NEXT:    mul a1, a1, a2
 ; RV64IZbb-NEXT:    zext.h a1, a1
 ; RV64IZbb-NEXT:    add a0, a0, a1
 ; RV64IZbb-NEXT:    lui a1, 16
@@ -173,7 +173,7 @@ define i8 @func8(i8 %x, i8 %y, i8 %z) nounwind {
 ; RV64I-LABEL: func8:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    andi a0, a0, 255
-; RV64I-NEXT:    mulw a1, a1, a2
+; RV64I-NEXT:    mul a1, a1, a2
 ; RV64I-NEXT:    andi a1, a1, 255
 ; RV64I-NEXT:    add a0, a0, a1
 ; RV64I-NEXT:    li a1, 255
@@ -196,7 +196,7 @@ define i8 @func8(i8 %x, i8 %y, i8 %z) nounwind {
 ; RV64IZbb-LABEL: func8:
 ; RV64IZbb:       # %bb.0:
 ; RV64IZbb-NEXT:    andi a0, a0, 255
-; RV64IZbb-NEXT:    mulw a1, a1, a2
+; RV64IZbb-NEXT:    mul a1, a1, a2
 ; RV64IZbb-NEXT:    andi a1, a1, 255
 ; RV64IZbb-NEXT:    add a0, a0, a1
 ; RV64IZbb-NEXT:    li a1, 255
@@ -224,7 +224,7 @@ define i4 @func4(i4 %x, i4 %y, i4 %z) nounwind {
 ; RV64I-LABEL: func4:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    andi a0, a0, 15
-; RV64I-NEXT:    mulw a1, a1, a2
+; RV64I-NEXT:    mul a1, a1, a2
 ; RV64I-NEXT:    andi a1, a1, 15
 ; RV64I-NEXT:    add a0, a0, a1
 ; RV64I-NEXT:    li a1, 15
@@ -247,7 +247,7 @@ define i4 @func4(i4 %x, i4 %y, i4 %z) nounwind {
 ; RV64IZbb-LABEL: func4:
 ; RV64IZbb:       # %bb.0:
 ; RV64IZbb-NEXT:    andi a0, a0, 15
-; RV64IZbb-NEXT:    mulw a1, a1, a2
+; RV64IZbb-NEXT:    mul a1, a1, a2
 ; RV64IZbb-NEXT:    andi a1, a1, 15
 ; RV64IZbb-NEXT:    add a0, a0, a1
 ; RV64IZbb-NEXT:    li a1, 15
diff --git a/llvm/test/CodeGen/RISCV/urem-lkk.ll b/llvm/test/CodeGen/RISCV/urem-lkk.ll
index 73ec7d23f554f..3d181c3a30d09 100644
--- a/llvm/test/CodeGen/RISCV/urem-lkk.ll
+++ b/llvm/test/CodeGen/RISCV/urem-lkk.ll
@@ -53,7 +53,7 @@ define i32 @fold_urem_positive_odd(i32 %x) nounwind {
 ; RV64IM-NEXT:    add a1, a2, a1
 ; RV64IM-NEXT:    srli a1, a1, 6
 ; RV64IM-NEXT:    li a2, 95
-; RV64IM-NEXT:    mulw a1, a1, a2
+; RV64IM-NEXT:    mul a1, a1, a2
 ; RV64IM-NEXT:    subw a0, a0, a1
 ; RV64IM-NEXT:    ret
   %1 = urem i32 %x, 95
@@ -99,7 +99,7 @@ define i32 @fold_urem_positive_even(i32 %x) nounwind {
 ; RV64IM-NEXT:    mulhu a1, a1, a2
 ; RV64IM-NEXT:    srli a1, a1, 42
 ; RV64IM-NEXT:    li a2, 1060
-; RV64IM-NEXT:    mulw a1, a1, a2
+; RV64IM-NEXT:    mul a1, a1, a2
 ; RV64IM-NEXT:    subw a0, a0, a1
 ; RV64IM-NEXT:    ret
   %1 = urem i32 %x, 1060
@@ -179,7 +179,7 @@ define i32 @combine_urem_udiv(i32 %x) nounwind {
 ; RV64IM-NEXT:    add a1, a2, a1
 ; RV64IM-NEXT:    srli a1, a1, 6
 ; RV64IM-NEXT:    li a2, 95
-; RV64IM-NEXT:    mulw a2, a1, a2
+; RV64IM-NEXT:    mul a2, a1, a2
 ; RV64IM-NEXT:    add a0, a0, a1
 ; RV64IM-NEXT:    subw a0, a0, a2
 ; RV64IM-NEXT:    ret
diff --git a/llvm/test/CodeGen/RISCV/urem-seteq-illegal-types.ll b/llvm/test/CodeGen/RISCV/urem-seteq-illegal-types.ll
index 1ac6179f62e45..36d064ea3d505 100644
--- a/llvm/test/CodeGen/RISCV/urem-seteq-illegal-types.ll
+++ b/llvm/test/CodeGen/RISCV/urem-seteq-illegal-types.ll
@@ -49,7 +49,7 @@ define i1 @test_urem_odd(i13 %X) nounwind {
 ; RV64M:       # %bb.0:
 ; RV64M-NEXT:    lui a1, 1
 ; RV64M-NEXT:    addiw a1, a1, -819
-; RV64M-NEXT:    mulw a0, a0, a1
+; RV64M-NEXT:    mul a0, a0, a1
 ; RV64M-NEXT:    slli a0, a0, 51
 ; RV64M-NEXT:    srli a0, a0, 51
 ; RV64M-NEXT:    sltiu a0, a0, 1639
@@ -69,7 +69,7 @@ define i1 @test_urem_odd(i13 %X) nounwind {
 ; RV64MV:       # %bb.0:
 ; RV64MV-NEXT:    lui a1, 1
 ; RV64MV-NEXT:    addiw a1, a1, -819
-; RV64MV-NEXT:    mulw a0, a0, a1
+; RV64MV-NEXT:    mul a0, a0, a1
 ; RV64MV-NEXT:    slli a0, a0, 51
 ; RV64MV-NEXT:    srli a0, a0, 51
 ; RV64MV-NEXT:    sltiu a0, a0, 1639
@@ -140,7 +140,7 @@ define i1 @test_urem_even(i27 %X) nounwind {
 ; RV64M:       # %bb.0:
 ; RV64M-NEXT:    lui a1, 28087
 ; RV64M-NEXT:    addiw a1, a1, -585
-; RV64M-NEXT:    mulw a0, a0, a1
+; RV64M-NEXT:    mul a0, a0, a1
 ; RV64M-NEXT:    slli a1, a0, 26
 ; RV64M-NEXT:    slli a0, a0, 37
 ; RV64M-NEXT:    srli a0, a0, 38
@@ -172,7 +172,7 @@ define i1 @test_urem_even(i27 %X) nounwind {
 ; RV64MV:       # %bb.0:
 ; RV64MV-NEXT:    lui a1, 28087
 ; RV64MV-NEXT:    addiw a1, a1, -585
-; RV64MV-NEXT:    mulw a0, a0, a1
+; RV64MV-NEXT:    mul a0, a0, a1
 ; RV64MV-NEXT:    slli a1, a0, 26
 ; RV64MV-NEXT:    slli a0, a0, 37
 ; RV64MV-NEXT:    srli a0, a0, 38
@@ -292,7 +292,7 @@ define i1 @test_urem_negative_odd(i9 %X) nounwind {
 ; RV64M-LABEL: test_urem_negative_odd:
 ; RV64M:       # %bb.0:
 ; RV64M-NEXT:    li a1, 307
-; RV64M-NEXT:    mulw a0, a0, a1
+; RV64M-NEXT:    mul a0, a0, a1
 ; RV64M-NEXT:    andi a0, a0, 511
 ; RV64M-NEXT:    sltiu a0, a0, 2
 ; RV64M-NEXT:    xori a0, a0, 1
@@ -310,7 +310,7 @@ define i1 @test_urem_negative_odd(i9 %X) nounwind {
 ; RV64MV-LABEL: test_urem_negative_odd:
 ; RV64MV:       # %bb.0:
 ; RV64MV-NEXT:    li a1, 307
-; RV64MV-NEXT:    mulw a0, a0, a1
+; RV64MV-NEXT:    mul a0, a0, a1
 ; RV64MV-NEXT:    andi a0, a0, 511
 ; RV64MV-NEXT:    sltiu a0, a0, 2
 ; RV64MV-NEXT:    xori a0, a0, 1
@@ -487,7 +487,7 @@ define void @test_urem_vec(ptr %X) nounwind {
 ; RV64M-NEXT:    srli a3, a1, 11
 ; RV64M-NEXT:    andi a1, a1, 2047
 ; RV64M-NEXT:    li a4, 683
-; RV64M-NEXT:    mulw a1, a1, a4
+; RV64M-NEXT:    mul a1, a1, a4
 ; RV64M-NEXT:    slli a4, a1, 10
 ; RV64M-NEXT:    slli a1, a1, 53
 ; RV64M-NEXT:    srli a1, a1, 54
@@ -495,12 +495,12 @@ define void @test_urem_vec(ptr %X) nounwind {
 ; RV64M-NEXT:    andi a1, a1, 2047
 ; RV64M-NEXT:    sltiu a1, a1, 342
 ; RV64M-NEXT:    li a4, 1463
-; RV64M-NEXT:    mulw a3, a3, a4
+; RV64M-NEXT:    mul a3, a3, a4
 ; RV64M-NEXT:    addiw a3, a3, -1463
 ; RV64M-NEXT:    andi a3, a3, 2047
 ; RV64M-NEXT:    sltiu a3, a3, 293
 ; RV64M-NEXT:    li a4, 819
-; RV64M-NEXT:    mulw a2, a2, a4
+; RV64M-NEXT:    mul a2, a2, a4
 ; RV64M-NEXT:    addiw a2, a2, -1638
 ; RV64M-NEXT:    andi a2, a2, 2047
 ; RV64M-NEXT:    sltiu a2, a2, 2
diff --git a/llvm/test/CodeGen/RISCV/urem-vector-lkk.ll b/llvm/test/CodeGen/RISCV/urem-vector-lkk.ll
index b2e50f9fc7c5b..d8f364ec8c00f 100644
--- a/llvm/test/CodeGen/RISCV/urem-vector-lkk.ll
+++ b/llvm/test/CodeGen/RISCV/urem-vector-lkk.ll
@@ -142,23 +142,23 @@ define <4 x i16> @fold_urem_vec_1(<4 x i16> %x) nounwind {
 ; RV64IM-NEXT:    lui a6, %hi(.LCPI0_1)
 ; RV64IM-NEXT:    ld a6, %lo(.LCPI0_1)(a6)
 ; RV64IM-NEXT:    li a7, 95
-; RV64IM-NEXT:    mulw a3, a3, a7
+; RV64IM-NEXT:    mul a3, a3, a7
 ; RV64IM-NEXT:    subw a2, a2, a3
 ; RV64IM-NEXT:    mulhu a3, a1, a6
 ; RV64IM-NEXT:    lui a6, %hi(.LCPI0_2)
 ; RV64IM-NEXT:    ld a6, %lo(.LCPI0_2)(a6)
 ; RV64IM-NEXT:    li a7, 124
-; RV64IM-NEXT:    mulw a3, a3, a7
+; RV64IM-NEXT:    mul a3, a3, a7
 ; RV64IM-NEXT:    subw a1, a1, a3
 ; RV64IM-NEXT:    mulhu a3, a5, a6
 ; RV64IM-NEXT:    lui a6, %hi(.LCPI0_3)
 ; RV64IM-NEXT:    ld a6, %lo(.LCPI0_3)(a6)
 ; RV64IM-NEXT:    li a7, 98
-; RV64IM-NEXT:    mulw a3, a3, a7
+; RV64IM-NEXT:    mul a3, a3, a7
 ; RV64IM-NEXT:    subw a5, a5, a3
 ; RV64IM-NEXT:    mulhu a3, a4, a6
 ; RV64IM-NEXT:    li a6, 1003
-; RV64IM-NEXT:    mulw a3, a3, a6
+; RV64IM-NEXT:    mul a3, a3, a6
 ; RV64IM-NEXT:    subw a4, a4, a3
 ; RV64IM-NEXT:    sh a4, 6(a0)
 ; RV64IM-NEXT:    sh a5, 4(a0)
@@ -291,16 +291,16 @@ define <4 x i16> @fold_urem_vec_2(<4 x i16> %x) nounwind {
 ; RV64IM-NEXT:    lhu a1, 8(a1)
 ; RV64IM-NEXT:    mulhu a6, a2, a3
 ; RV64IM-NEXT:    li a7, 95
-; RV64IM-NEXT:    mulw a6, a6, a7
+; RV64IM-NEXT:    mul a6, a6, a7
 ; RV64IM-NEXT:    subw a2, a2, a6
 ; RV64IM-NEXT:    mulhu a6, a1, a3
-; RV64IM-NEXT:    mulw a6, a6, a7
+; RV64IM-NEXT:    mul a6, a6, a7
 ; RV64IM-NEXT:    subw a1, a1, a6
 ; RV64IM-NEXT:    mulhu a6, a5, a3
-; RV64IM-NEXT:    mulw a6, a6, a7
+; RV64IM-NEXT:    mul a6, a6, a7
 ; RV64IM-NEXT:    subw a5, a5, a6
 ; RV64IM-NEXT:    mulhu a3, a4, a3
-; RV64IM-NEXT:    mulw a3, a3, a7
+; RV64IM-NEXT:    mul a3, a3, a7
 ; RV64IM-NEXT:    subw a4, a4, a3
 ; RV64IM-NEXT:    sh a4, 6(a0)
 ; RV64IM-NEXT:    sh a5, 4(a0)
@@ -495,13 +495,13 @@ define <4 x i16> @combine_urem_udiv(<4 x i16> %x) nounwind {
 ; RV64IM-NEXT:    lhu a1, 16(a1)
 ; RV64IM-NEXT:    mulhu a6, a2, a3
 ; RV64IM-NEXT:    li a7, 95
-; RV64IM-NEXT:    mulw t0, a6, a7
+; RV64IM-NEXT:    mul t0, a6, a7
 ; RV64IM-NEXT:    mulhu t1, a1, a3
-; RV64IM-NEXT:    mulw t2, t1, a7
+; RV64IM-NEXT:    mul t2, t1, a7
 ; RV64IM-NEXT:    mulhu t3, a5, a3
-; RV64IM-NEXT:    mulw t4, t3, a7
+; RV64IM-NEXT:    mul t4, t3, a7
 ; RV64IM-NEXT:    mulhu a3, a4, a3
-; RV64IM-NEXT:    mulw a7, a3, a7
+; RV64IM-NEXT:    mul a7, a3, a7
 ; RV64IM-NEXT:    add a3, a4, a3
 ; RV64IM-NEXT:    subw a3, a3, a7
 ; RV64IM-NEXT:    add a5, a5, t3
@@ -616,7 +616,7 @@ define <4 x i16> @dont_fold_urem_power_of_two(<4 x i16> %x) nounwind {
 ; RV64IM-NEXT:    lhu a1, 0(a1)
 ; RV64IM-NEXT:    mulhu a3, a2, a3
 ; RV64IM-NEXT:    li a6, 95
-; RV64IM-NEXT:    mulw a3, a3, a6
+; RV64IM-NEXT:    mul a3, a3, a6
 ; RV64IM-NEXT:    subw a2, a2, a3
 ; RV64IM-NEXT:    andi a1, a1, 63
 ; RV64IM-NEXT:    andi a5, a5, 31
@@ -745,18 +745,18 @@ define <4 x i16> @dont_fold_urem_one(<4 x i16> %x) nounwind {
 ; RV64IM-NEXT:    lui a5, %hi(.LCPI4_1)
 ; RV64IM-NEXT:    ld a5, %lo(.LCPI4_1)(a5)
 ; RV64IM-NEXT:    li a6, 654
-; RV64IM-NEXT:    mulw a3, a3, a6
+; RV64IM-NEXT:    mul a3, a3, a6
 ; RV64IM-NEXT:    subw a2, a2, a3
 ; RV64IM-NEXT:    mulhu a3, a1, a5
 ; RV64IM-NEXT:    lui a5, %hi(.LCPI4_2)
 ; RV64IM-NEXT:    ld a5, %lo(.LCPI4_2)(a5)
 ; RV64IM-NEXT:    li a6, 23
-; RV64IM-NEXT:    mulw a3, a3, a6
+; RV64IM-NEXT:    mul a3, a3, a6
 ; RV64IM-NEXT:    subw a1, a1, a3
 ; RV64IM-NEXT:    mulhu a3, a4, a5
 ; RV64IM-NEXT:    lui a5, 1
 ; RV64IM-NEXT:    addiw a5, a5, 1327
-; RV64IM-NEXT:    mulw a3, a3, a5
+; RV64IM-NEXT:    mul a3, a3, a5
 ; RV64IM-NEXT:    subw a4, a4, a3
 ; RV64IM-NEXT:    sh zero, 0(a0)
 ; RV64IM-NEXT:    sh a4, 6(a0)
diff --git a/llvm/test/CodeGen/RISCV/usub_sat_plus.ll b/llvm/test/CodeGen/RISCV/usub_sat_plus.ll
index 6f868b328b7cc..393466d89e8ca 100644
--- a/llvm/test/CodeGen/RISCV/usub_sat_plus.ll
+++ b/llvm/test/CodeGen/RISCV/usub_sat_plus.ll
@@ -22,7 +22,7 @@ define i32 @func32(i32 %x, i32 %y, i32 %z) nounwind {
 ;
 ; RV64I-LABEL: func32:
 ; RV64I:       # %bb.0:
-; RV64I-NEXT:    mulw a1, a1, a2
+; RV64I-NEXT:    mul a1, a1, a2
 ; RV64I-NEXT:    subw a1, a0, a1
 ; RV64I-NEXT:    sext.w a0, a0
 ; RV64I-NEXT:    sltu a0, a0, a1
@@ -143,7 +143,7 @@ define i16 @func16(i16 %x, i16 %y, i16 %z) nounwind {
 ; RV64IZbb-LABEL: func16:
 ; RV64IZbb:       # %bb.0:
 ; RV64IZbb-NEXT:    zext.h a0, a0
-; RV64IZbb-NEXT:    mulw a1, a1, a2
+; RV64IZbb-NEXT:    mul a1, a1, a2
 ; RV64IZbb-NEXT:    zext.h a1, a1
 ; RV64IZbb-NEXT:    maxu a0, a0, a1
 ; RV64IZbb-NEXT:    sub a0, a0, a1
@@ -168,7 +168,7 @@ define i8 @func8(i8 %x, i8 %y, i8 %z) nounwind {
 ; RV64I-LABEL: func8:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    andi a0, a0, 255
-; RV64I-NEXT:    mulw a1, a1, a2
+; RV64I-NEXT:    mul a1, a1, a2
 ; RV64I-NEXT:    andi a1, a1, 255
 ; RV64I-NEXT:    sub a1, a0, a1
 ; RV64I-NEXT:    sltu a0, a0, a1
@@ -188,7 +188,7 @@ define i8 @func8(i8 %x, i8 %y, i8 %z) nounwind {
 ; RV64IZbb-LABEL: func8:
 ; RV64IZbb:       # %bb.0:
 ; RV64IZbb-NEXT:    andi a0, a0, 255
-; RV64IZbb-NEXT:    mulw a1, a1, a2
+; RV64IZbb-NEXT:    mul a1, a1, a2
 ; RV64IZbb-NEXT:    andi a1, a1, 255
 ; RV64IZbb-NEXT:    maxu a0, a0, a1
 ; RV64IZbb-NEXT:    sub a0, a0, a1
@@ -213,7 +213,7 @@ define i4 @func4(i4 %x, i4 %y, i4 %z) nounwind {
 ; RV64I-LABEL: func4:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    andi a0, a0, 15
-; RV64I-NEXT:    mulw a1, a1, a2
+; RV64I-NEXT:    mul a1, a1, a2
 ; RV64I-NEXT:    andi a1, a1, 15
 ; RV64I-NEXT:    sub a1, a0, a1
 ; RV64I-NEXT:    sltu a0, a0, a1
@@ -233,7 +233,7 @@ define i4 @func4(i4 %x, i4 %y, i4 %z) nounwind {
 ; RV64IZbb-LABEL: func4:
 ; RV64IZbb:       # %bb.0:
 ; RV64IZbb-NEXT:    andi a0, a0, 15
-; RV64IZbb-NEXT:    mulw a1, a1, a2
+; RV64IZbb-NEXT:    mul a1, a1, a2
 ; RV64IZbb-NEXT:    andi a1, a1, 15
 ; RV64IZbb-NEXT:    maxu a0, a0, a1
 ; RV64IZbb-NEXT:    sub a0, a0, a1
diff --git a/llvm/test/CodeGen/SPARC/branches-relax.ll b/llvm/test/CodeGen/SPARC/branches-relax.ll
new file mode 100644
index 0000000000000..11324faf1146f
--- /dev/null
+++ b/llvm/test/CodeGen/SPARC/branches-relax.ll
@@ -0,0 +1,113 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=sparc64 -sparc-bpcc-offset-bits=4 | FileCheck --check-prefix=SPARC64 %s
+
+define i32 @branch_relax_int(i32 %in) {
+; SPARC64-LABEL: branch_relax_int:
+; SPARC64:         .cfi_startproc
+; SPARC64-NEXT:  ! %bb.0:
+; SPARC64-NEXT:    save %sp, -128, %sp
+; SPARC64-NEXT:    .cfi_def_cfa_register %fp
+; SPARC64-NEXT:    .cfi_window_save
+; SPARC64-NEXT:    .cfi_register %o7, %i7
+; SPARC64-NEXT:    cmp %i0, 0
+; SPARC64-NEXT:    bne %icc, .LBB0_1
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    ba .LBB0_2
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:  .LBB0_1: ! %false
+; SPARC64-NEXT:    !APP
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    !NO_APP
+; SPARC64-NEXT:    ret
+; SPARC64-NEXT:    restore %g0, %g0, %o0
+; SPARC64-NEXT:  .LBB0_2: ! %true
+; SPARC64-NEXT:    mov 4, %i0
+; SPARC64-NEXT:    !APP
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    !NO_APP
+; SPARC64-NEXT:    ret
+; SPARC64-NEXT:    restore
+  %tst = icmp eq i32 %in, 0
+  br i1 %tst, label %true, label %false
+
+true:
+  call void asm sideeffect "nop\0A\09nop\0A\09nop\0A\09nop\0A\09nop\0A\09nop\0A\09nop\0A\09nop", ""()
+  ret i32 4
+
+false:
+  call void asm sideeffect "nop\0A\09nop\0A\09nop\0A\09nop\0A\09nop\0A\09nop\0A\09nop\0A\09nop", ""()
+  ret i32 0
+}
+
+define float @branch_relax_float(float %in) {
+; SPARC64-LABEL: branch_relax_float:
+; SPARC64:         .cfi_startproc
+; SPARC64-NEXT:  ! %bb.0:
+; SPARC64-NEXT:    save %sp, -128, %sp
+; SPARC64-NEXT:    .cfi_def_cfa_register %fp
+; SPARC64-NEXT:    .cfi_window_save
+; SPARC64-NEXT:    .cfi_register %o7, %i7
+; SPARC64-NEXT:    sethi %h44(.LCPI1_0), %i0
+; SPARC64-NEXT:    add %i0, %m44(.LCPI1_0), %i0
+; SPARC64-NEXT:    sllx %i0, 12, %i0
+; SPARC64-NEXT:    ld [%i0+%l44(.LCPI1_0)], %f0
+; SPARC64-NEXT:    fcmps %fcc0, %f1, %f0
+; SPARC64-NEXT:    fbe %fcc0, .LBB1_1
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    ba .LBB1_2
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:  .LBB1_1: ! %true
+; SPARC64-NEXT:    sethi %h44(.LCPI1_1), %i0
+; SPARC64-NEXT:    add %i0, %m44(.LCPI1_1), %i0
+; SPARC64-NEXT:    sllx %i0, 12, %i0
+; SPARC64-NEXT:    ld [%i0+%l44(.LCPI1_1)], %f0
+; SPARC64-NEXT:    !APP
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    !NO_APP
+; SPARC64-NEXT:    ret
+; SPARC64-NEXT:    restore
+; SPARC64-NEXT:  .LBB1_2: ! %false
+; SPARC64-NEXT:    !APP
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    nop
+; SPARC64-NEXT:    !NO_APP
+; SPARC64-NEXT:    ret
+; SPARC64-NEXT:    restore
+  %tst = fcmp oeq float %in, 0.0
+  br i1 %tst, label %true, label %false
+
+true:
+  call void asm sideeffect "nop\0A\09nop\0A\09nop\0A\09nop\0A\09nop\0A\09nop\0A\09nop\0A\09nop", ""()
+  ret float 4.0
+
+false:
+  call void asm sideeffect "nop\0A\09nop\0A\09nop\0A\09nop\0A\09nop\0A\09nop\0A\09nop\0A\09nop", ""()
+  ret float 0.0
+}
diff --git a/llvm/test/CodeGen/SPARC/branches-v9.ll b/llvm/test/CodeGen/SPARC/branches-v9.ll
index b1315407194dd..c1c765d6025c8 100644
--- a/llvm/test/CodeGen/SPARC/branches-v9.ll
+++ b/llvm/test/CodeGen/SPARC/branches-v9.ll
@@ -20,7 +20,7 @@ define void @i(i32 signext %sel) {
 ; CHECK-NEXT:  ! %bb.1: ! %fbb
 ; CHECK-NEXT:    call f2
 ; CHECK-NEXT:    nop
-; CHECK-NEXT:    ba %icc, .LBB0_3
+; CHECK-NEXT:    ba .LBB0_3
 ; CHECK-NEXT:    nop
 ; CHECK-NEXT:  .LBB0_2: ! %tbb
 ; CHECK-NEXT:    call f1
@@ -61,7 +61,7 @@ define void @l(i64 %sel) {
 ; CHECK-NEXT:  ! %bb.1: ! %fbb
 ; CHECK-NEXT:    call f2
 ; CHECK-NEXT:    nop
-; CHECK-NEXT:    ba %icc, .LBB1_3
+; CHECK-NEXT:    ba .LBB1_3
 ; CHECK-NEXT:    nop
 ; CHECK-NEXT:  .LBB1_2: ! %tbb
 ; CHECK-NEXT:    call f1
diff --git a/llvm/test/CodeGen/SPARC/smulo-128-legalisation-lowering.ll b/llvm/test/CodeGen/SPARC/smulo-128-legalisation-lowering.ll
index 5af477c49f2bf..21ab49a341378 100644
--- a/llvm/test/CodeGen/SPARC/smulo-128-legalisation-lowering.ll
+++ b/llvm/test/CodeGen/SPARC/smulo-128-legalisation-lowering.ll
@@ -125,12 +125,12 @@ define { i128, i8 } @muloti_test(i128 %l, i128 %r) unnamed_addr #0 {
 ; SPARC-NEXT:    or %i2, %i4, %i2
 ; SPARC-NEXT:    or %i2, %i3, %i2
 ; SPARC-NEXT:    cmp %i2, 0
-; SPARC-NEXT:    bne .LBB0_1
+; SPARC-NEXT:    bne .LBB0_2
 ; SPARC-NEXT:    nop
-; SPARC-NEXT:  ! %bb.2: ! %start
+; SPARC-NEXT:  ! %bb.1: ! %start
 ; SPARC-NEXT:    ba .LBB0_3
 ; SPARC-NEXT:    mov %g0, %i4
-; SPARC-NEXT:  .LBB0_1:
+; SPARC-NEXT:  .LBB0_2:
 ; SPARC-NEXT:    mov 1, %i4
 ; SPARC-NEXT:  .LBB0_3: ! %start
 ; SPARC-NEXT:    ld [%fp+-4], %i2 ! 4-byte Folded Reload
diff --git a/llvm/test/CodeGen/SPIRV/constant/local-arbitrary-width-integers-constants-type-promotion.ll b/llvm/test/CodeGen/SPIRV/constant/local-arbitrary-width-integers-constants-type-promotion.ll
new file mode 100644
index 0000000000000..06ab469e70077
--- /dev/null
+++ b/llvm/test/CodeGen/SPIRV/constant/local-arbitrary-width-integers-constants-type-promotion.ll
@@ -0,0 +1,58 @@
+; RUN: llc -O0 -mtriple=spirv32-unknown-unknown %s -o - | FileCheck %s
+
+define i4 @getConstantI4() {
+  ret i4 2 ; i4 => OpTypeInt 8
+}
+
+define i11 @getConstantI11() {
+  ret i11 7 ; i11 => OpTypeInt 16
+}
+
+define i24 @getConstantI24() {
+  ret i24 42 ; i24 => OpTypeInt 32
+}
+
+define i63 @getConstantI63() {
+  ret i63 5705 ; i63 => OpTypeInt 64
+}
+
+;; Capabilities:
+; CHECK-DAG: OpCapability Int8
+; CHECK-DAG: OpCapability Int16
+; CHECK-DAG: OpCapability Int64
+
+; CHECK-NOT: DAG-FENCE
+
+;; Names:
+; CHECK-DAG: OpName %[[#GET_I4:]] "getConstantI4"
+; CHECK-DAG: OpName %[[#GET_I11:]] "getConstantI11"
+; CHECK-DAG: OpName %[[#GET_I24:]] "getConstantI24"
+; CHECK-DAG: OpName %[[#GET_I63:]] "getConstantI63"
+
+; CHECK-NOT: DAG-FENCE
+
+;; Types and Constants:
+; CHECK-DAG: %[[#I8:]] = OpTypeInt 8 0
+; CHECK-DAG: %[[#I16:]] = OpTypeInt 16 0
+; CHECK-DAG: %[[#I32:]] = OpTypeInt 32 0
+; CHECK-DAG: %[[#I64:]] = OpTypeInt 64 0
+; CHECK-DAG: %[[#CST_I8:]] = OpConstant %[[#I8]] 2
+; CHECK-DAG: %[[#CST_I16:]] = OpConstant %[[#I16]] 7
+; CHECK-DAG: %[[#CST_I32:]] = OpConstant %[[#I32]] 42
+; CHECK-DAG: %[[#CST_I64:]] = OpConstant %[[#I64]] 5705
+
+; CHECK: %[[#GET_I4]] = OpFunction %[[#I8]]
+; CHECK: OpReturnValue %[[#CST_I8]]
+; CHECK: OpFunctionEnd
+
+; CHECK: %[[#GET_I11]] = OpFunction %[[#I16]]
+; CHECK: OpReturnValue %[[#CST_I16]]
+; CHECK: OpFunctionEnd
+
+; CHECK: %[[#GET_I24]] = OpFunction %[[#I32]]
+; CHECK: OpReturnValue %[[#CST_I32]]
+; CHECK: OpFunctionEnd
+
+; CHECK: %[[#GET_I63]] = OpFunction %[[#I64]]
+; CHECK: OpReturnValue %[[#CST_I64]]
+; CHECK: OpFunctionEnd
diff --git a/llvm/test/CodeGen/SPIRV/constant/local-integers-constants.ll b/llvm/test/CodeGen/SPIRV/constant/local-integers-constants.ll
index 0530d1ffa4c62..9660ba446cfca 100644
--- a/llvm/test/CodeGen/SPIRV/constant/local-integers-constants.ll
+++ b/llvm/test/CodeGen/SPIRV/constant/local-integers-constants.ll
@@ -1,5 +1,9 @@
 ; RUN: llc -O0 -mtriple=spirv32-unknown-unknown %s -o - | FileCheck %s
 
+define i8 @getConstantI8() {
+  ret i8 2
+}
+
 define i16 @getConstantI16() {
   ret i16 -58
 }
@@ -17,12 +21,14 @@ define i64 @getLargeConstantI64() {
 }
 
 ;; Capabilities:
+; CHECK-DAG: OpCapability Int8
 ; CHECK-DAG: OpCapability Int16
 ; CHECK-DAG: OpCapability Int64
 
 ; CHECK-NOT: DAG-FENCE
 
 ;; Names:
+; CHECK-DAG: OpName %[[#GET_I8:]] "getConstantI8"
 ; CHECK-DAG: OpName %[[#GET_I16:]] "getConstantI16"
 ; CHECK-DAG: OpName %[[#GET_I32:]] "getConstantI32"
 ; CHECK-DAG: OpName %[[#GET_I64:]] "getConstantI64"
@@ -31,14 +37,20 @@ define i64 @getLargeConstantI64() {
 ; CHECK-NOT: DAG-FENCE
 
 ;; Types and Constants:
+; CHECK-DAG: %[[#I8:]] = OpTypeInt 8 0
 ; CHECK-DAG: %[[#I16:]] = OpTypeInt 16 0
 ; CHECK-DAG: %[[#I32:]] = OpTypeInt 32 0
 ; CHECK-DAG: %[[#I64:]] = OpTypeInt 64 0
+; CHECK-DAG: %[[#CST_I8:]] = OpConstant %[[#I8]] 2
 ; CHECK-DAG: %[[#CST_I16:]] = OpConstant %[[#I16]] 65478
 ; CHECK-DAG: %[[#CST_I32:]] = OpConstant %[[#I32]] 42
 ; CHECK-DAG: %[[#CST_I64:]] = OpConstant %[[#I64]] 123456789 0
 ; CHECK-DAG: %[[#CST_LARGE_I64:]] = OpConstant %[[#I64]] 0 8
 
+; CHECK: %[[#GET_I8]] = OpFunction %[[#I8]]
+; CHECK: OpReturnValue %[[#CST_I8]]
+; CHECK: OpFunctionEnd
+
 ; CHECK: %[[#GET_I16]] = OpFunction %[[#I16]]
 ; CHECK: OpReturnValue %[[#CST_I16]]
 ; CHECK: OpFunctionEnd
diff --git a/llvm/test/CodeGen/SystemZ/int-cmp-62.ll b/llvm/test/CodeGen/SystemZ/int-cmp-62.ll
new file mode 100644
index 0000000000000..c57cf5777d0f2
--- /dev/null
+++ b/llvm/test/CodeGen/SystemZ/int-cmp-62.ll
@@ -0,0 +1,39 @@
+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z15 | FileCheck %s
+;
+; Test that a CC result of a sub that can overflow is tested with the right predicate.
+
+define i32 @fun0(i32 %a, i32 %b, ptr %dest) {
+; CHECK-LABEL: fun0
+; CHECK: s %r2, 0(%r4)
+; CHECK: bner %r14
+entry:
+  %cur = load i32, ptr %dest
+  %res = sub nsw i32 %a, %cur
+  %cmp = icmp ne i32 %a, %cur
+  br i1 %cmp, label %exit, label %store
+
+store:
+  store i32 %b, ptr %dest
+  br label %exit
+
+exit:
+  ret i32 %res
+}
+
+define i32 @fun1(i32 %a, i32 %b, ptr %dest) {
+; CHECK-LABEL: fun1
+; CHECK: s %r2, 0(%r4)
+; CHECK: bner %r14
+entry:
+  %cur = load i32, ptr %dest
+  %res = sub nuw i32 %a, %cur
+  %cmp = icmp ne i32 %a, %cur
+  br i1 %cmp, label %exit, label %store
+
+store:
+  store i32 %b, ptr %dest
+  br label %exit
+
+exit:
+  ret i32 %res
+}
diff --git a/llvm/test/CodeGen/SystemZ/prefetch-04.ll b/llvm/test/CodeGen/SystemZ/prefetch-04.ll
index 6ab85321d9099..61a2a1460c583 100644
--- a/llvm/test/CodeGen/SystemZ/prefetch-04.ll
+++ b/llvm/test/CodeGen/SystemZ/prefetch-04.ll
@@ -5,7 +5,7 @@
 ; write prefetch.
 ;
 ; CHECK-LABEL: for.body
-; CHECK: call void @llvm.prefetch.p0(ptr %uglygep, i32 1, i32 3, i32 1
+; CHECK: call void @llvm.prefetch.p0(ptr %scevgep, i32 1, i32 3, i32 1
 ; CHECK-not: call void @llvm.prefetch
 
 define void @fun(ptr nocapture %Src, ptr nocapture readonly %Dst) {
diff --git a/llvm/test/CodeGen/Thumb2/mve-complex-deinterleaving-mixed-cases.ll b/llvm/test/CodeGen/Thumb2/mve-complex-deinterleaving-mixed-cases.ll
index b16b06bd45023..a529aa81467e0 100644
--- a/llvm/test/CodeGen/Thumb2/mve-complex-deinterleaving-mixed-cases.ll
+++ b/llvm/test/CodeGen/Thumb2/mve-complex-deinterleaving-mixed-cases.ll
@@ -385,3 +385,194 @@ entry:
   %interleaved.vec = shufflevector <2 x float> %11, <2 x float> %8, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
   ret <4 x float> %interleaved.vec
 }
+
+; Expected to transform
+define <4 x float> @mul_addequal(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
+; CHECK-LABEL: mul_addequal:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vmov d0, r0, r1
+; CHECK-NEXT:    mov r1, sp
+; CHECK-NEXT:    vldrw.u32 q2, [r1]
+; CHECK-NEXT:    vmov d1, r2, r3
+; CHECK-NEXT:    add r0, sp, #16
+; CHECK-NEXT:    vcmul.f32 q3, q0, q2, #0
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vcmla.f32 q3, q0, q2, #90
+; CHECK-NEXT:    vadd.f32 q0, q3, q1
+; CHECK-NEXT:    vmov r0, r1, d0
+; CHECK-NEXT:    vmov r2, r3, d1
+; CHECK-NEXT:    bx lr
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %c.real = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %c.imag = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %6 = fadd fast <2 x float> %5, %c.real
+  %7 = fadd fast <2 x float> %2, %c.imag
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}
+
+; Expected to transform
+define <4 x float> @mul_subequal(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
+; CHECK-LABEL: mul_subequal:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vmov d0, r0, r1
+; CHECK-NEXT:    mov r1, sp
+; CHECK-NEXT:    vldrw.u32 q2, [r1]
+; CHECK-NEXT:    vmov d1, r2, r3
+; CHECK-NEXT:    add r0, sp, #16
+; CHECK-NEXT:    vcmul.f32 q3, q0, q2, #0
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vcmla.f32 q3, q0, q2, #90
+; CHECK-NEXT:    vsub.f32 q0, q3, q1
+; CHECK-NEXT:    vmov r0, r1, d0
+; CHECK-NEXT:    vmov r2, r3, d1
+; CHECK-NEXT:    bx lr
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %c.real = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %c.imag = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %6 = fsub fast <2 x float> %5, %c.real
+  %7 = fsub fast <2 x float> %2, %c.imag
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}
+
+
+; Expected to transform
+define <4 x float> @mul_mulequal(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
+; CHECK-LABEL: mul_mulequal:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vmov d0, r0, r1
+; CHECK-NEXT:    mov r1, sp
+; CHECK-NEXT:    vldrw.u32 q2, [r1]
+; CHECK-NEXT:    vmov d1, r2, r3
+; CHECK-NEXT:    add r0, sp, #16
+; CHECK-NEXT:    vcmul.f32 q3, q0, q2, #0
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vcmla.f32 q3, q0, q2, #90
+; CHECK-NEXT:    vmul.f32 q0, q3, q1
+; CHECK-NEXT:    vmov r0, r1, d0
+; CHECK-NEXT:    vmov r2, r3, d1
+; CHECK-NEXT:    bx lr
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %c.real = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %c.imag = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %6 = fmul fast <2 x float> %5, %c.real
+  %7 = fmul fast <2 x float> %2, %c.imag
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}
+
+; Expected to not transform
+define <4 x float> @mul_divequal(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
+; CHECK-LABEL: mul_divequal:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    .vsave {d10, d11}
+; CHECK-NEXT:    vpush {d10, d11}
+; CHECK-NEXT:    .vsave {d8}
+; CHECK-NEXT:    vpush {d8}
+; CHECK-NEXT:    vmov d0, r0, r1
+; CHECK-NEXT:    add r0, sp, #24
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vmov d1, r2, r3
+; CHECK-NEXT:    vmov.f32 s16, s1
+; CHECK-NEXT:    add.w r12, sp, #40
+; CHECK-NEXT:    vmov.f32 s12, s5
+; CHECK-NEXT:    vmov.f32 s13, s7
+; CHECK-NEXT:    vmov.f32 s1, s2
+; CHECK-NEXT:    vmov.f32 s8, s4
+; CHECK-NEXT:    vmul.f32 q5, q3, q0
+; CHECK-NEXT:    vmov.f32 s9, s6
+; CHECK-NEXT:    vldrw.u32 q1, [r12]
+; CHECK-NEXT:    vmov.f32 s17, s3
+; CHECK-NEXT:    vfma.f32 q5, q2, q4
+; CHECK-NEXT:    vmul.f32 q3, q4, q3
+; CHECK-NEXT:    vdiv.f32 s3, s21, s7
+; CHECK-NEXT:    vneg.f32 q3, q3
+; CHECK-NEXT:    vfma.f32 q3, q2, q0
+; CHECK-NEXT:    vdiv.f32 s1, s20, s5
+; CHECK-NEXT:    vdiv.f32 s2, s13, s6
+; CHECK-NEXT:    vdiv.f32 s0, s12, s4
+; CHECK-NEXT:    vmov r0, r1, d0
+; CHECK-NEXT:    vmov r2, r3, d1
+; CHECK-NEXT:    vpop {d8}
+; CHECK-NEXT:    vpop {d10, d11}
+; CHECK-NEXT:    bx lr
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %c.real = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %c.imag = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %6 = fdiv fast <2 x float> %5, %c.real
+  %7 = fdiv fast <2 x float> %2, %c.imag
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}
+
+; Expected to transform
+define <4 x float> @mul_negequal(<4 x float> %a, <4 x float> %b) {
+; CHECK-LABEL: mul_negequal:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vmov d0, r0, r1
+; CHECK-NEXT:    mov r0, sp
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vmov d1, r2, r3
+; CHECK-NEXT:    vcmul.f32 q2, q0, q1, #0
+; CHECK-NEXT:    vcmla.f32 q2, q0, q1, #90
+; CHECK-NEXT:    vneg.f32 q0, q2
+; CHECK-NEXT:    vmov r0, r1, d0
+; CHECK-NEXT:    vmov r2, r3, d1
+; CHECK-NEXT:    bx lr
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %6 = fneg fast <2 x float> %5
+  %7 = fneg fast <2 x float> %2
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}
diff --git a/llvm/test/CodeGen/Thumb2/mve-laneinterleaving-reduct.ll b/llvm/test/CodeGen/Thumb2/mve-laneinterleaving-reduct.ll
index d13a3454906f6..a9cf69351a496 100644
--- a/llvm/test/CodeGen/Thumb2/mve-laneinterleaving-reduct.ll
+++ b/llvm/test/CodeGen/Thumb2/mve-laneinterleaving-reduct.ll
@@ -123,10 +123,10 @@ define void @correlate(ptr nocapture noundef readonly %ID, ptr nocapture noundef
 ; CHECK-NEXT:  @ %bb.1: @ %for.body.lr.ph
 ; CHECK-NEXT:    ldr r7, [sp, #48]
 ; CHECK-NEXT:    mov r0, r3
-; CHECK-NEXT:    ldr.w r10, [sp, #4] @ 4-byte Reload
+; CHECK-NEXT:    ldr.w r9, [sp, #4] @ 4-byte Reload
 ; CHECK-NEXT:    adds r3, r2, #3
 ; CHECK-NEXT:    mov.w r11, #0
-; CHECK-NEXT:    mov r9, r2
+; CHECK-NEXT:    mov r10, r2
 ; CHECK-NEXT:    uxth.w r12, r7
 ; CHECK-NEXT:    adr r7, .LCPI4_0
 ; CHECK-NEXT:    vldrw.u32 q0, [r7]
@@ -139,8 +139,8 @@ define void @correlate(ptr nocapture noundef readonly %ID, ptr nocapture noundef
 ; CHECK-NEXT:    ldr r7, [sp, #8] @ 4-byte Reload
 ; CHECK-NEXT:    lsrs r1, r6, #16
 ; CHECK-NEXT:    subs r3, #1
-; CHECK-NEXT:    add.w r10, r10, #2
-; CHECK-NEXT:    sub.w r9, r9, #1
+; CHECK-NEXT:    add.w r9, r9, #2
+; CHECK-NEXT:    sub.w r10, r10, #1
 ; CHECK-NEXT:    strh.w r1, [r7, r11, lsl #1]
 ; CHECK-NEXT:    add.w r11, r11, #1
 ; CHECK-NEXT:    cmp r11, r0
@@ -162,12 +162,13 @@ define void @correlate(ptr nocapture noundef readonly %ID, ptr nocapture noundef
 ; CHECK-NEXT:    b .LBB4_10
 ; CHECK-NEXT:  .LBB4_7: @ %vector.ph
 ; CHECK-NEXT:    @ in Loop: Header=BB4_4 Depth=1
-; CHECK-NEXT:    bic r7, r8, #7
+; CHECK-NEXT:    bic r7, r10, #7
 ; CHECK-NEXT:    movs r1, #1
-; CHECK-NEXT:    sub.w r6, r7, #8
-; CHECK-NEXT:    mov r5, r10
-; CHECK-NEXT:    add.w lr, r1, r6, lsr #3
+; CHECK-NEXT:    subs r7, #8
 ; CHECK-NEXT:    movs r6, #0
+; CHECK-NEXT:    mov r5, r9
+; CHECK-NEXT:    add.w lr, r1, r7, lsr #3
+; CHECK-NEXT:    bic r7, r8, #7
 ; CHECK-NEXT:    ldr r4, [sp, #4] @ 4-byte Reload
 ; CHECK-NEXT:  .LBB4_8: @ %vector.body
 ; CHECK-NEXT:    @ Parent Loop BB4_4 Depth=1
diff --git a/llvm/test/CodeGen/WebAssembly/call.ll b/llvm/test/CodeGen/WebAssembly/call.ll
index 0fcc2bed2463c..6e7a97118bc96 100644
--- a/llvm/test/CodeGen/WebAssembly/call.ll
+++ b/llvm/test/CodeGen/WebAssembly/call.ll
@@ -216,7 +216,10 @@ define void @coldcc_tail_call_void_nullary() {
 ; CHECK-NEXT: i32.const $push[[L0:[0-9]+]]=, 2{{$}}
 ; CHECK-NEXT: i32.const $push[[L1:[0-9]+]]=, 3{{$}}
 ; CHECK-NEXT: call .Lvararg_func_bitcast, $pop[[L0]], $pop[[L1]]{{$}}
-; CHECK-NEXT: call other_void_nullary{{$}}
+; CHECK-NEXT: i32.const	$push[[L3:[0-9]+]]=, void_nullary{{$}}
+; CHECK-NEXT: i32.const	$push[[L2:[0-9]+]]=, other_void_nullary{{$}}
+; CHECK-NEXT: i32.add 	$push[[L4:[0-9]+]]=, $pop[[L3]], $pop[[L2]]{{$}}
+; CHECK-NEXT: call_indirect	$pop[[L4]]{{$}}
 ; CHECK-NEXT: call void_nullary{{$}}
 ; CHECK-NEXT: return{{$}}
 declare void @vararg_func(...)
@@ -226,7 +229,7 @@ bb0:
   call void @vararg_func(i32 2, i32 3)
   br label %bb1
 bb1:
-  call void select (i1 0, ptr @void_nullary, ptr @other_void_nullary)()
+  call void getelementptr (i8, ptr @void_nullary, i32 ptrtoint (ptr @other_void_nullary to i32))()
   br label %bb2
 bb2:
   call void inttoptr (i32 ptrtoint (ptr @void_nullary to i32) to ptr)()
diff --git a/llvm/test/CodeGen/WebAssembly/lower-em-exceptions.ll b/llvm/test/CodeGen/WebAssembly/lower-em-exceptions.ll
index 1d75bafb102ab..d17a5b419e351 100644
--- a/llvm/test/CodeGen/WebAssembly/lower-em-exceptions.ll
+++ b/llvm/test/CodeGen/WebAssembly/lower-em-exceptions.ll
@@ -69,7 +69,7 @@ catch:                                            ; preds = %catch.dispatch
 ; Test invoke instruction with filters (functions with throw(...) declaration)
 ; Currently we don't support exception specifications correctly in JS glue code,
 ; so we ignore all filters here.
-; See https://bugs.llvm.org/show_bug.cgi?id=50396.
+; See https://github.com/llvm/llvm-project/issues/49740.
 define void @filter() personality ptr @__gxx_personality_v0 {
 ; CHECK-LABEL: @filter(
 entry:
diff --git a/llvm/test/CodeGen/WebAssembly/pr59625.ll b/llvm/test/CodeGen/WebAssembly/pr59625.ll
new file mode 100644
index 0000000000000..4c3d9226e9348
--- /dev/null
+++ b/llvm/test/CodeGen/WebAssembly/pr59625.ll
@@ -0,0 +1,18 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
+; RUN: llc < %s -mtriple=wasm32-- -mattr=+simd128 | FileCheck --check-prefix=CHECK %s
+; RUN: llc < %s -mtriple=wasm64-- -mattr=+simd128 | FileCheck --check-prefix=CHECK %s
+
+define <1 x i16> @f(<1 x i16> %0) {
+; CHECK-LABEL: f:
+; CHECK:         .functype f (v128) -> (v128)
+; CHECK-NEXT:  # %bb.0: # %BB
+; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    # fallthrough-return
+BB:
+  %B2 = srem <1 x i16> %0, %0
+  br label %BB1
+
+BB1:                                             ; preds = %BB
+  %B = urem <1 x i16> %B2, <i16 3>
+  ret <1 x i16> %B
+}
diff --git a/llvm/test/CodeGen/X86/2008-09-19-RegAllocBug.ll b/llvm/test/CodeGen/X86/2008-09-19-RegAllocBug.ll
index 61b54ee271c25..f027f3aa77d78 100644
--- a/llvm/test/CodeGen/X86/2008-09-19-RegAllocBug.ll
+++ b/llvm/test/CodeGen/X86/2008-09-19-RegAllocBug.ll
@@ -9,10 +9,11 @@ entry:
 	%1 = trunc i32 %0 to i8		; <i8> [#uses=1]
 	%2 = sub i8 1, %1		; <i8> [#uses=1]
 	%3 = sext i8 %2 to i32		; <i32> [#uses=1]
-	%.0 = ashr i32 %3, select (i1 icmp ne (i8 zext (i1 icmp ugt (i32 ptrtoint (ptr @func_4 to i32), i32 3) to i8), i8 0), i32 0, i32 ptrtoint (ptr @func_4 to i32))		; <i32> [#uses=1]
-	%4 = urem i32 %0, %.0		; <i32> [#uses=1]
-	%5 = icmp eq i32 %4, 0		; <i1> [#uses=1]
-	br i1 %5, label %return, label %bb4
+	%s = select i1 icmp ne (i8 zext (i1 icmp ugt (i32 ptrtoint (ptr @func_4 to i32), i32 3) to i8), i8 0), i32 0, i32 ptrtoint (ptr @func_4 to i32)
+	%ashr = ashr i32 %3, %s
+	%urem = urem i32 %0, %ashr
+	%cmp = icmp eq i32 %urem, 0
+	br i1 %cmp, label %return, label %bb4
 
 bb4:		; preds = %entry
 	ret i32 undef
diff --git a/llvm/test/CodeGen/X86/3addr-or.ll b/llvm/test/CodeGen/X86/3addr-or.ll
index ee560a9f9a24f..72e29b9faf4a6 100644
--- a/llvm/test/CodeGen/X86/3addr-or.ll
+++ b/llvm/test/CodeGen/X86/3addr-or.ll
@@ -20,7 +20,6 @@ define i32 @test1(i32 %x) nounwind ssp {
 define i64 @test2(i8 %A, i8 %B) nounwind {
 ; CHECK-LABEL: test2:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $esi killed $esi def $rsi
 ; CHECK-NEXT:    # kill: def $edi killed $edi def $rdi
 ; CHECK-NEXT:    shll $4, %edi
 ; CHECK-NEXT:    andl $48, %edi
diff --git a/llvm/test/CodeGen/X86/Inputs/fsloader_v1.afdo b/llvm/test/CodeGen/X86/Inputs/fsloader_v1.afdo
new file mode 100644
index 0000000000000..dbb5dd0e8eca9
--- /dev/null
+++ b/llvm/test/CodeGen/X86/Inputs/fsloader_v1.afdo
@@ -0,0 +1,35 @@
+work:42380966:1346190
+ 1: 1246499
+ 5: 1246499
+foo:28798256:4267
+ 0: 4267
+ 2.1: 255999
+ 4: 264627 bar:250018
+ 4.256: 269485 bar:278102
+ 4.512: 280297 bar:280933
+ 4.768: 278916 bar:267752
+ 5: 264627
+ 5.256: 269485
+ 5.512: 260670
+ 5.768: 278916
+ 6: 11541
+ 6.256: 278916 work:284547
+ 6.512: 260670 work:249428
+ 6.768: 11541
+ 7: 272442
+ 7.256: 283590
+ 7.512: 234082
+ 7.768: 279149
+ 8: 11541
+ 8.256: 283590 work:305061
+ 8.512: 279149 work:281368
+ 8.768: 234082 work:225786
+ 10: 4050
+bar:9504180:1076805
+ 2: 1056020
+ 3: 1056020
+main:20360:0
+ 0: 0
+ 2.1: 4045
+ 3: 4156 foo:4267
+ 5: 0
diff --git a/llvm/test/CodeGen/X86/abds-vector-128.ll b/llvm/test/CodeGen/X86/abds-vector-128.ll
index b0eb8e75d7909..a48781c6ebf8e 100644
--- a/llvm/test/CodeGen/X86/abds-vector-128.ll
+++ b/llvm/test/CodeGen/X86/abds-vector-128.ll
@@ -799,38 +799,17 @@ define <16 x i8> @abd_cmp_v16i8(<16 x i8> %a, <16 x i8> %b) nounwind {
 ; SSE42-LABEL: abd_cmp_v16i8:
 ; SSE42:       # %bb.0:
 ; SSE42-NEXT:    movdqa %xmm0, %xmm2
-; SSE42-NEXT:    pcmpgtb %xmm1, %xmm2
-; SSE42-NEXT:    movdqa %xmm0, %xmm3
-; SSE42-NEXT:    psubb %xmm1, %xmm3
-; SSE42-NEXT:    psubb %xmm0, %xmm1
-; SSE42-NEXT:    movdqa %xmm2, %xmm0
-; SSE42-NEXT:    pblendvb %xmm0, %xmm3, %xmm1
-; SSE42-NEXT:    movdqa %xmm1, %xmm0
+; SSE42-NEXT:    pminsb %xmm1, %xmm2
+; SSE42-NEXT:    pmaxsb %xmm1, %xmm0
+; SSE42-NEXT:    psubb %xmm2, %xmm0
 ; SSE42-NEXT:    retq
 ;
-; AVX1-LABEL: abd_cmp_v16i8:
-; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpcmpgtb %xmm1, %xmm0, %xmm2
-; AVX1-NEXT:    vpsubb %xmm1, %xmm0, %xmm3
-; AVX1-NEXT:    vpsubb %xmm0, %xmm1, %xmm0
-; AVX1-NEXT:    vpblendvb %xmm2, %xmm3, %xmm0, %xmm0
-; AVX1-NEXT:    retq
-;
-; AVX2-LABEL: abd_cmp_v16i8:
-; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpcmpgtb %xmm1, %xmm0, %xmm2
-; AVX2-NEXT:    vpsubb %xmm1, %xmm0, %xmm3
-; AVX2-NEXT:    vpsubb %xmm0, %xmm1, %xmm0
-; AVX2-NEXT:    vpblendvb %xmm2, %xmm3, %xmm0, %xmm0
-; AVX2-NEXT:    retq
-;
-; AVX512-LABEL: abd_cmp_v16i8:
-; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpsubb %xmm1, %xmm0, %xmm2
-; AVX512-NEXT:    vpsubb %xmm0, %xmm1, %xmm3
-; AVX512-NEXT:    vpcmpgtb %xmm1, %xmm0, %xmm0
-; AVX512-NEXT:    vpblendvb %xmm0, %xmm2, %xmm3, %xmm0
-; AVX512-NEXT:    retq
+; AVX-LABEL: abd_cmp_v16i8:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpminsb %xmm1, %xmm0, %xmm2
+; AVX-NEXT:    vpmaxsb %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vpsubb %xmm2, %xmm0, %xmm0
+; AVX-NEXT:    retq
   %cmp = icmp sgt <16 x i8> %a, %b
   %ab = sub <16 x i8> %a, %b
   %ba = sub <16 x i8> %b, %a
@@ -839,54 +818,20 @@ define <16 x i8> @abd_cmp_v16i8(<16 x i8> %a, <16 x i8> %b) nounwind {
 }
 
 define <8 x i16> @abd_cmp_v8i16(<8 x i16> %a, <8 x i16> %b) nounwind {
-; SSE2-LABEL: abd_cmp_v8i16:
-; SSE2:       # %bb.0:
-; SSE2-NEXT:    movdqa %xmm1, %xmm2
-; SSE2-NEXT:    pcmpgtw %xmm0, %xmm2
-; SSE2-NEXT:    movdqa %xmm0, %xmm3
-; SSE2-NEXT:    psubw %xmm1, %xmm3
-; SSE2-NEXT:    psubw %xmm0, %xmm1
-; SSE2-NEXT:    pand %xmm2, %xmm1
-; SSE2-NEXT:    pandn %xmm3, %xmm2
-; SSE2-NEXT:    por %xmm1, %xmm2
-; SSE2-NEXT:    movdqa %xmm2, %xmm0
-; SSE2-NEXT:    retq
-;
-; SSE42-LABEL: abd_cmp_v8i16:
-; SSE42:       # %bb.0:
-; SSE42-NEXT:    movdqa %xmm1, %xmm2
-; SSE42-NEXT:    pcmpgtw %xmm0, %xmm2
-; SSE42-NEXT:    movdqa %xmm0, %xmm3
-; SSE42-NEXT:    psubw %xmm1, %xmm3
-; SSE42-NEXT:    psubw %xmm0, %xmm1
-; SSE42-NEXT:    movdqa %xmm2, %xmm0
-; SSE42-NEXT:    pblendvb %xmm0, %xmm1, %xmm3
-; SSE42-NEXT:    movdqa %xmm3, %xmm0
-; SSE42-NEXT:    retq
-;
-; AVX1-LABEL: abd_cmp_v8i16:
-; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpcmpgtw %xmm0, %xmm1, %xmm2
-; AVX1-NEXT:    vpsubw %xmm1, %xmm0, %xmm3
-; AVX1-NEXT:    vpsubw %xmm0, %xmm1, %xmm0
-; AVX1-NEXT:    vpblendvb %xmm2, %xmm0, %xmm3, %xmm0
-; AVX1-NEXT:    retq
-;
-; AVX2-LABEL: abd_cmp_v8i16:
-; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpcmpgtw %xmm0, %xmm1, %xmm2
-; AVX2-NEXT:    vpsubw %xmm1, %xmm0, %xmm3
-; AVX2-NEXT:    vpsubw %xmm0, %xmm1, %xmm0
-; AVX2-NEXT:    vpblendvb %xmm2, %xmm0, %xmm3, %xmm0
-; AVX2-NEXT:    retq
+; SSE-LABEL: abd_cmp_v8i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movdqa %xmm0, %xmm2
+; SSE-NEXT:    pminsw %xmm1, %xmm2
+; SSE-NEXT:    pmaxsw %xmm1, %xmm0
+; SSE-NEXT:    psubw %xmm2, %xmm0
+; SSE-NEXT:    retq
 ;
-; AVX512-LABEL: abd_cmp_v8i16:
-; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpsubw %xmm1, %xmm0, %xmm2
-; AVX512-NEXT:    vpsubw %xmm0, %xmm1, %xmm3
-; AVX512-NEXT:    vpcmpgtw %xmm0, %xmm1, %xmm0
-; AVX512-NEXT:    vpblendvb %xmm0, %xmm3, %xmm2, %xmm0
-; AVX512-NEXT:    retq
+; AVX-LABEL: abd_cmp_v8i16:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpminsw %xmm1, %xmm0, %xmm2
+; AVX-NEXT:    vpmaxsw %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vpsubw %xmm2, %xmm0, %xmm0
+; AVX-NEXT:    retq
   %cmp = icmp sge <8 x i16> %a, %b
   %ab = sub <8 x i16> %a, %b
   %ba = sub <8 x i16> %b, %a
@@ -910,39 +855,18 @@ define <4 x i32> @abd_cmp_v4i32(<4 x i32> %a, <4 x i32> %b) nounwind {
 ;
 ; SSE42-LABEL: abd_cmp_v4i32:
 ; SSE42:       # %bb.0:
-; SSE42-NEXT:    movdqa %xmm1, %xmm2
-; SSE42-NEXT:    pcmpgtd %xmm0, %xmm2
-; SSE42-NEXT:    movdqa %xmm0, %xmm3
-; SSE42-NEXT:    psubd %xmm1, %xmm3
-; SSE42-NEXT:    psubd %xmm0, %xmm1
-; SSE42-NEXT:    movdqa %xmm2, %xmm0
-; SSE42-NEXT:    blendvps %xmm0, %xmm1, %xmm3
-; SSE42-NEXT:    movaps %xmm3, %xmm0
+; SSE42-NEXT:    movdqa %xmm0, %xmm2
+; SSE42-NEXT:    pminsd %xmm1, %xmm2
+; SSE42-NEXT:    pmaxsd %xmm1, %xmm0
+; SSE42-NEXT:    psubd %xmm2, %xmm0
 ; SSE42-NEXT:    retq
 ;
-; AVX1-LABEL: abd_cmp_v4i32:
-; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpcmpgtd %xmm0, %xmm1, %xmm2
-; AVX1-NEXT:    vpsubd %xmm1, %xmm0, %xmm3
-; AVX1-NEXT:    vpsubd %xmm0, %xmm1, %xmm0
-; AVX1-NEXT:    vblendvps %xmm2, %xmm0, %xmm3, %xmm0
-; AVX1-NEXT:    retq
-;
-; AVX2-LABEL: abd_cmp_v4i32:
-; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpcmpgtd %xmm0, %xmm1, %xmm2
-; AVX2-NEXT:    vpsubd %xmm1, %xmm0, %xmm3
-; AVX2-NEXT:    vpsubd %xmm0, %xmm1, %xmm0
-; AVX2-NEXT:    vblendvps %xmm2, %xmm0, %xmm3, %xmm0
-; AVX2-NEXT:    retq
-;
-; AVX512-LABEL: abd_cmp_v4i32:
-; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpcmpgtd %xmm0, %xmm1, %k1
-; AVX512-NEXT:    vpsubd %xmm1, %xmm0, %xmm2
-; AVX512-NEXT:    vpsubd %xmm0, %xmm1, %xmm2 {%k1}
-; AVX512-NEXT:    vmovdqa %xmm2, %xmm0
-; AVX512-NEXT:    retq
+; AVX-LABEL: abd_cmp_v4i32:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpminsd %xmm1, %xmm0, %xmm2
+; AVX-NEXT:    vpmaxsd %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vpsubd %xmm2, %xmm0, %xmm0
+; AVX-NEXT:    retq
   %cmp = icmp slt <4 x i32> %a, %b
   %ab = sub <4 x i32> %a, %b
   %ba = sub <4 x i32> %b, %a
@@ -968,16 +892,16 @@ define <2 x i64> @abd_cmp_v2i64(<2 x i64> %a, <2 x i64> %b) nounwind {
 ; SSE2-NEXT:    movdqa %xmm0, %xmm3
 ; SSE2-NEXT:    psubq %xmm1, %xmm3
 ; SSE2-NEXT:    psubq %xmm0, %xmm1
-; SSE2-NEXT:    pand %xmm2, %xmm3
-; SSE2-NEXT:    pandn %xmm1, %xmm2
-; SSE2-NEXT:    por %xmm3, %xmm2
+; SSE2-NEXT:    pand %xmm2, %xmm1
+; SSE2-NEXT:    pandn %xmm3, %xmm2
+; SSE2-NEXT:    por %xmm1, %xmm2
 ; SSE2-NEXT:    movdqa %xmm2, %xmm0
 ; SSE2-NEXT:    retq
 ;
 ; SSE42-LABEL: abd_cmp_v2i64:
 ; SSE42:       # %bb.0:
-; SSE42-NEXT:    movdqa %xmm1, %xmm2
-; SSE42-NEXT:    pcmpgtq %xmm0, %xmm2
+; SSE42-NEXT:    movdqa %xmm0, %xmm2
+; SSE42-NEXT:    pcmpgtq %xmm1, %xmm2
 ; SSE42-NEXT:    movdqa %xmm0, %xmm3
 ; SSE42-NEXT:    psubq %xmm1, %xmm3
 ; SSE42-NEXT:    psubq %xmm0, %xmm1
@@ -988,7 +912,7 @@ define <2 x i64> @abd_cmp_v2i64(<2 x i64> %a, <2 x i64> %b) nounwind {
 ;
 ; AVX1-LABEL: abd_cmp_v2i64:
 ; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpcmpgtq %xmm0, %xmm1, %xmm2
+; AVX1-NEXT:    vpcmpgtq %xmm1, %xmm0, %xmm2
 ; AVX1-NEXT:    vpsubq %xmm1, %xmm0, %xmm3
 ; AVX1-NEXT:    vpsubq %xmm0, %xmm1, %xmm0
 ; AVX1-NEXT:    vblendvpd %xmm2, %xmm3, %xmm0, %xmm0
@@ -996,7 +920,7 @@ define <2 x i64> @abd_cmp_v2i64(<2 x i64> %a, <2 x i64> %b) nounwind {
 ;
 ; AVX2-LABEL: abd_cmp_v2i64:
 ; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpcmpgtq %xmm0, %xmm1, %xmm2
+; AVX2-NEXT:    vpcmpgtq %xmm1, %xmm0, %xmm2
 ; AVX2-NEXT:    vpsubq %xmm1, %xmm0, %xmm3
 ; AVX2-NEXT:    vpsubq %xmm0, %xmm1, %xmm0
 ; AVX2-NEXT:    vblendvpd %xmm2, %xmm3, %xmm0, %xmm0
@@ -1004,15 +928,14 @@ define <2 x i64> @abd_cmp_v2i64(<2 x i64> %a, <2 x i64> %b) nounwind {
 ;
 ; AVX512-LABEL: abd_cmp_v2i64:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpcmpnltq %xmm1, %xmm0, %k1
-; AVX512-NEXT:    vpsubq %xmm1, %xmm0, %xmm2
-; AVX512-NEXT:    vpsubq %xmm0, %xmm1, %xmm2 {%k1}
-; AVX512-NEXT:    vmovdqa %xmm2, %xmm0
+; AVX512-NEXT:    vpminsq %xmm1, %xmm0, %xmm2
+; AVX512-NEXT:    vpmaxsq %xmm1, %xmm0, %xmm0
+; AVX512-NEXT:    vpsubq %xmm2, %xmm0, %xmm0
 ; AVX512-NEXT:    retq
   %cmp = icmp sge <2 x i64> %a, %b
   %ab = sub <2 x i64> %a, %b
   %ba = sub <2 x i64> %b, %a
-  %sel = select <2 x i1> %cmp, <2 x i64> %ba, <2 x i64> %ab
+  %sel = select <2 x i1> %cmp, <2 x i64> %ab, <2 x i64> %ba
   ret <2 x i64> %sel
 }
 
@@ -1174,44 +1097,47 @@ define <2 x i64> @abd_cmp_v2i64_multiuse_cmp(<2 x i64> %a, <2 x i64> %b) nounwin
 ; SSE42-LABEL: abd_cmp_v2i64_multiuse_cmp:
 ; SSE42:       # %bb.0:
 ; SSE42-NEXT:    movdqa %xmm0, %xmm2
-; SSE42-NEXT:    psubq %xmm1, %xmm2
-; SSE42-NEXT:    movdqa %xmm1, %xmm3
-; SSE42-NEXT:    psubq %xmm0, %xmm3
-; SSE42-NEXT:    pcmpgtq %xmm0, %xmm1
-; SSE42-NEXT:    movdqa %xmm1, %xmm0
-; SSE42-NEXT:    blendvpd %xmm0, %xmm3, %xmm2
+; SSE42-NEXT:    pcmpgtq %xmm1, %xmm0
+; SSE42-NEXT:    movdqa %xmm2, %xmm3
+; SSE42-NEXT:    psubq %xmm1, %xmm3
+; SSE42-NEXT:    movdqa %xmm1, %xmm4
+; SSE42-NEXT:    psubq %xmm2, %xmm4
+; SSE42-NEXT:    blendvpd %xmm0, %xmm3, %xmm4
+; SSE42-NEXT:    pcmpgtq %xmm2, %xmm1
 ; SSE42-NEXT:    pcmpeqd %xmm0, %xmm0
 ; SSE42-NEXT:    pxor %xmm1, %xmm0
-; SSE42-NEXT:    paddq %xmm2, %xmm0
+; SSE42-NEXT:    paddq %xmm4, %xmm0
 ; SSE42-NEXT:    retq
 ;
 ; AVX1-LABEL: abd_cmp_v2i64_multiuse_cmp:
 ; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpsubq %xmm1, %xmm0, %xmm2
-; AVX1-NEXT:    vpsubq %xmm0, %xmm1, %xmm3
+; AVX1-NEXT:    vpcmpgtq %xmm1, %xmm0, %xmm2
+; AVX1-NEXT:    vpsubq %xmm1, %xmm0, %xmm3
+; AVX1-NEXT:    vpsubq %xmm0, %xmm1, %xmm4
+; AVX1-NEXT:    vblendvpd %xmm2, %xmm3, %xmm4, %xmm2
 ; AVX1-NEXT:    vpcmpgtq %xmm0, %xmm1, %xmm0
-; AVX1-NEXT:    vblendvpd %xmm0, %xmm3, %xmm2, %xmm1
-; AVX1-NEXT:    vpcmpeqd %xmm2, %xmm2, %xmm2
-; AVX1-NEXT:    vpxor %xmm2, %xmm0, %xmm0
-; AVX1-NEXT:    vpaddq %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
+; AVX1-NEXT:    vpxor %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vpaddq %xmm2, %xmm0, %xmm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: abd_cmp_v2i64_multiuse_cmp:
 ; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpsubq %xmm1, %xmm0, %xmm2
-; AVX2-NEXT:    vpsubq %xmm0, %xmm1, %xmm3
+; AVX2-NEXT:    vpcmpgtq %xmm1, %xmm0, %xmm2
+; AVX2-NEXT:    vpsubq %xmm1, %xmm0, %xmm3
+; AVX2-NEXT:    vpsubq %xmm0, %xmm1, %xmm4
+; AVX2-NEXT:    vblendvpd %xmm2, %xmm3, %xmm4, %xmm2
 ; AVX2-NEXT:    vpcmpgtq %xmm0, %xmm1, %xmm0
-; AVX2-NEXT:    vblendvpd %xmm0, %xmm3, %xmm2, %xmm1
-; AVX2-NEXT:    vpcmpeqd %xmm2, %xmm2, %xmm2
-; AVX2-NEXT:    vpxor %xmm2, %xmm0, %xmm0
-; AVX2-NEXT:    vpaddq %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
+; AVX2-NEXT:    vpxor %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vpaddq %xmm2, %xmm0, %xmm0
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: abd_cmp_v2i64_multiuse_cmp:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpcmpnltq %xmm1, %xmm0, %k1
-; AVX512-NEXT:    vpsubq %xmm0, %xmm1, %xmm2
-; AVX512-NEXT:    vpsubq %xmm1, %xmm0, %xmm2 {%k1}
+; AVX512-NEXT:    vpminsq %xmm1, %xmm0, %xmm2
+; AVX512-NEXT:    vpmaxsq %xmm1, %xmm0, %xmm3
+; AVX512-NEXT:    vpsubq %xmm2, %xmm3, %xmm2
 ; AVX512-NEXT:    vpcmpgtq %xmm0, %xmm1, %xmm0
 ; AVX512-NEXT:    vpternlogq $15, %xmm0, %xmm0, %xmm0
 ; AVX512-NEXT:    vpaddq %xmm2, %xmm0, %xmm0
@@ -1226,60 +1152,25 @@ define <2 x i64> @abd_cmp_v2i64_multiuse_cmp(<2 x i64> %a, <2 x i64> %b) nounwin
 }
 
 define <8 x i16> @abd_cmp_v8i16_multiuse_sub(<8 x i16> %a, <8 x i16> %b) nounwind {
-; SSE2-LABEL: abd_cmp_v8i16_multiuse_sub:
-; SSE2:       # %bb.0:
-; SSE2-NEXT:    movdqa %xmm0, %xmm2
-; SSE2-NEXT:    pcmpgtw %xmm1, %xmm2
-; SSE2-NEXT:    movdqa %xmm0, %xmm3
-; SSE2-NEXT:    psubw %xmm1, %xmm3
-; SSE2-NEXT:    psubw %xmm0, %xmm1
-; SSE2-NEXT:    movdqa %xmm3, %xmm0
-; SSE2-NEXT:    pand %xmm2, %xmm0
-; SSE2-NEXT:    pandn %xmm1, %xmm2
-; SSE2-NEXT:    por %xmm0, %xmm2
-; SSE2-NEXT:    paddw %xmm3, %xmm2
-; SSE2-NEXT:    movdqa %xmm2, %xmm0
-; SSE2-NEXT:    retq
-;
-; SSE42-LABEL: abd_cmp_v8i16_multiuse_sub:
-; SSE42:       # %bb.0:
-; SSE42-NEXT:    movdqa %xmm0, %xmm2
-; SSE42-NEXT:    pcmpgtw %xmm1, %xmm2
-; SSE42-NEXT:    movdqa %xmm0, %xmm3
-; SSE42-NEXT:    psubw %xmm1, %xmm3
-; SSE42-NEXT:    psubw %xmm0, %xmm1
-; SSE42-NEXT:    movdqa %xmm2, %xmm0
-; SSE42-NEXT:    pblendvb %xmm0, %xmm3, %xmm1
-; SSE42-NEXT:    paddw %xmm1, %xmm3
-; SSE42-NEXT:    movdqa %xmm3, %xmm0
-; SSE42-NEXT:    retq
-;
-; AVX1-LABEL: abd_cmp_v8i16_multiuse_sub:
-; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpcmpgtw %xmm1, %xmm0, %xmm2
-; AVX1-NEXT:    vpsubw %xmm1, %xmm0, %xmm3
-; AVX1-NEXT:    vpsubw %xmm0, %xmm1, %xmm0
-; AVX1-NEXT:    vpblendvb %xmm2, %xmm3, %xmm0, %xmm0
-; AVX1-NEXT:    vpaddw %xmm0, %xmm3, %xmm0
-; AVX1-NEXT:    retq
-;
-; AVX2-LABEL: abd_cmp_v8i16_multiuse_sub:
-; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpcmpgtw %xmm1, %xmm0, %xmm2
-; AVX2-NEXT:    vpsubw %xmm1, %xmm0, %xmm3
-; AVX2-NEXT:    vpsubw %xmm0, %xmm1, %xmm0
-; AVX2-NEXT:    vpblendvb %xmm2, %xmm3, %xmm0, %xmm0
-; AVX2-NEXT:    vpaddw %xmm0, %xmm3, %xmm0
-; AVX2-NEXT:    retq
+; SSE-LABEL: abd_cmp_v8i16_multiuse_sub:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movdqa %xmm0, %xmm2
+; SSE-NEXT:    psubw %xmm1, %xmm2
+; SSE-NEXT:    movdqa %xmm0, %xmm3
+; SSE-NEXT:    pminsw %xmm1, %xmm3
+; SSE-NEXT:    pmaxsw %xmm1, %xmm0
+; SSE-NEXT:    psubw %xmm3, %xmm0
+; SSE-NEXT:    paddw %xmm2, %xmm0
+; SSE-NEXT:    retq
 ;
-; AVX512-LABEL: abd_cmp_v8i16_multiuse_sub:
-; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpsubw %xmm1, %xmm0, %xmm2
-; AVX512-NEXT:    vpsubw %xmm0, %xmm1, %xmm3
-; AVX512-NEXT:    vpcmpgtw %xmm1, %xmm0, %xmm0
-; AVX512-NEXT:    vpblendvb %xmm0, %xmm2, %xmm3, %xmm0
-; AVX512-NEXT:    vpaddw %xmm0, %xmm2, %xmm0
-; AVX512-NEXT:    retq
+; AVX-LABEL: abd_cmp_v8i16_multiuse_sub:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpsubw %xmm1, %xmm0, %xmm2
+; AVX-NEXT:    vpminsw %xmm1, %xmm0, %xmm3
+; AVX-NEXT:    vpmaxsw %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vpsubw %xmm3, %xmm0, %xmm0
+; AVX-NEXT:    vpaddw %xmm0, %xmm2, %xmm0
+; AVX-NEXT:    retq
   %cmp = icmp sgt <8 x i16> %a, %b
   %ab = sub <8 x i16> %a, %b
   %ba = sub <8 x i16> %b, %a
diff --git a/llvm/test/CodeGen/X86/abds-vector-256.ll b/llvm/test/CodeGen/X86/abds-vector-256.ll
index 880ce6434b67c..78190d2cb7d8b 100644
--- a/llvm/test/CodeGen/X86/abds-vector-256.ll
+++ b/llvm/test/CodeGen/X86/abds-vector-256.ll
@@ -441,34 +441,27 @@ define <32 x i8> @abd_cmp_v32i8(<32 x i8> %a, <32 x i8> %b) nounwind {
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm3
-; AVX1-NEXT:    vpcmpgtb %xmm2, %xmm3, %xmm4
-; AVX1-NEXT:    vpcmpgtb %xmm1, %xmm0, %xmm5
-; AVX1-NEXT:    vinsertf128 $1, %xmm4, %ymm5, %ymm4
-; AVX1-NEXT:    vpsubb %xmm2, %xmm3, %xmm5
-; AVX1-NEXT:    vpsubb %xmm1, %xmm0, %xmm6
-; AVX1-NEXT:    vinsertf128 $1, %xmm5, %ymm6, %ymm5
-; AVX1-NEXT:    vpsubb %xmm3, %xmm2, %xmm2
-; AVX1-NEXT:    vpsubb %xmm0, %xmm1, %xmm0
+; AVX1-NEXT:    vpminsb %xmm2, %xmm3, %xmm4
+; AVX1-NEXT:    vpmaxsb %xmm2, %xmm3, %xmm2
+; AVX1-NEXT:    vpsubb %xmm4, %xmm2, %xmm2
+; AVX1-NEXT:    vpminsb %xmm1, %xmm0, %xmm3
+; AVX1-NEXT:    vpmaxsb %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vpsubb %xmm3, %xmm0, %xmm0
 ; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm0, %ymm0
-; AVX1-NEXT:    vandnps %ymm0, %ymm4, %ymm0
-; AVX1-NEXT:    vandps %ymm4, %ymm5, %ymm1
-; AVX1-NEXT:    vorps %ymm0, %ymm1, %ymm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: abd_cmp_v32i8:
 ; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpcmpgtb %ymm1, %ymm0, %ymm2
-; AVX2-NEXT:    vpsubb %ymm1, %ymm0, %ymm3
-; AVX2-NEXT:    vpsubb %ymm0, %ymm1, %ymm0
-; AVX2-NEXT:    vpblendvb %ymm2, %ymm3, %ymm0, %ymm0
+; AVX2-NEXT:    vpminsb %ymm1, %ymm0, %ymm2
+; AVX2-NEXT:    vpmaxsb %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    vpsubb %ymm2, %ymm0, %ymm0
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: abd_cmp_v32i8:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpsubb %ymm1, %ymm0, %ymm2
-; AVX512-NEXT:    vpsubb %ymm0, %ymm1, %ymm3
-; AVX512-NEXT:    vpcmpgtb %ymm1, %ymm0, %ymm0
-; AVX512-NEXT:    vpblendvb %ymm0, %ymm2, %ymm3, %ymm0
+; AVX512-NEXT:    vpminsb %ymm1, %ymm0, %ymm2
+; AVX512-NEXT:    vpmaxsb %ymm1, %ymm0, %ymm0
+; AVX512-NEXT:    vpsubb %ymm2, %ymm0, %ymm0
 ; AVX512-NEXT:    retq
   %cmp = icmp sgt <32 x i8> %a, %b
   %ab = sub <32 x i8> %a, %b
@@ -480,36 +473,29 @@ define <32 x i8> @abd_cmp_v32i8(<32 x i8> %a, <32 x i8> %b) nounwind {
 define <16 x i16> @abd_cmp_v16i16(<16 x i16> %a, <16 x i16> %b) nounwind {
 ; AVX1-LABEL: abd_cmp_v16i16:
 ; AVX1:       # %bb.0:
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm2
-; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm3
-; AVX1-NEXT:    vpcmpgtw %xmm2, %xmm3, %xmm4
-; AVX1-NEXT:    vpcmpgtw %xmm0, %xmm1, %xmm5
-; AVX1-NEXT:    vinsertf128 $1, %xmm4, %ymm5, %ymm4
-; AVX1-NEXT:    vpsubw %xmm3, %xmm2, %xmm5
-; AVX1-NEXT:    vpsubw %xmm1, %xmm0, %xmm6
-; AVX1-NEXT:    vinsertf128 $1, %xmm5, %ymm6, %ymm5
-; AVX1-NEXT:    vpsubw %xmm2, %xmm3, %xmm2
-; AVX1-NEXT:    vpsubw %xmm0, %xmm1, %xmm0
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm3
+; AVX1-NEXT:    vpminsw %xmm2, %xmm3, %xmm4
+; AVX1-NEXT:    vpmaxsw %xmm2, %xmm3, %xmm2
+; AVX1-NEXT:    vpsubw %xmm4, %xmm2, %xmm2
+; AVX1-NEXT:    vpminsw %xmm1, %xmm0, %xmm3
+; AVX1-NEXT:    vpmaxsw %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vpsubw %xmm3, %xmm0, %xmm0
 ; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm0, %ymm0
-; AVX1-NEXT:    vandps %ymm0, %ymm4, %ymm0
-; AVX1-NEXT:    vandnps %ymm5, %ymm4, %ymm1
-; AVX1-NEXT:    vorps %ymm0, %ymm1, %ymm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: abd_cmp_v16i16:
 ; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpcmpgtw %ymm0, %ymm1, %ymm2
-; AVX2-NEXT:    vpsubw %ymm1, %ymm0, %ymm3
-; AVX2-NEXT:    vpsubw %ymm0, %ymm1, %ymm0
-; AVX2-NEXT:    vpblendvb %ymm2, %ymm0, %ymm3, %ymm0
+; AVX2-NEXT:    vpminsw %ymm1, %ymm0, %ymm2
+; AVX2-NEXT:    vpmaxsw %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    vpsubw %ymm2, %ymm0, %ymm0
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: abd_cmp_v16i16:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpsubw %ymm1, %ymm0, %ymm2
-; AVX512-NEXT:    vpsubw %ymm0, %ymm1, %ymm3
-; AVX512-NEXT:    vpcmpgtw %ymm0, %ymm1, %ymm0
-; AVX512-NEXT:    vpblendvb %ymm0, %ymm3, %ymm2, %ymm0
+; AVX512-NEXT:    vpminsw %ymm1, %ymm0, %ymm2
+; AVX512-NEXT:    vpmaxsw %ymm1, %ymm0, %ymm0
+; AVX512-NEXT:    vpsubw %ymm2, %ymm0, %ymm0
 ; AVX512-NEXT:    retq
   %cmp = icmp sge <16 x i16> %a, %b
   %ab = sub <16 x i16> %a, %b
@@ -521,33 +507,29 @@ define <16 x i16> @abd_cmp_v16i16(<16 x i16> %a, <16 x i16> %b) nounwind {
 define <8 x i32> @abd_cmp_v8i32(<8 x i32> %a, <8 x i32> %b) nounwind {
 ; AVX1-LABEL: abd_cmp_v8i32:
 ; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpcmpgtd %xmm0, %xmm1, %xmm2
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm3
-; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm4
-; AVX1-NEXT:    vpcmpgtd %xmm3, %xmm4, %xmm5
-; AVX1-NEXT:    vpsubd %xmm1, %xmm0, %xmm6
-; AVX1-NEXT:    vpsubd %xmm4, %xmm3, %xmm7
-; AVX1-NEXT:    vpsubd %xmm0, %xmm1, %xmm0
-; AVX1-NEXT:    vblendvps %xmm2, %xmm0, %xmm6, %xmm0
-; AVX1-NEXT:    vpsubd %xmm3, %xmm4, %xmm1
-; AVX1-NEXT:    vblendvps %xmm5, %xmm1, %xmm7, %xmm1
-; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
+; AVX1-NEXT:    vpminsd %xmm2, %xmm3, %xmm4
+; AVX1-NEXT:    vpmaxsd %xmm2, %xmm3, %xmm2
+; AVX1-NEXT:    vpsubd %xmm4, %xmm2, %xmm2
+; AVX1-NEXT:    vpminsd %xmm1, %xmm0, %xmm3
+; AVX1-NEXT:    vpmaxsd %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vpsubd %xmm3, %xmm0, %xmm0
+; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm0, %ymm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: abd_cmp_v8i32:
 ; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpcmpgtd %ymm0, %ymm1, %ymm2
-; AVX2-NEXT:    vpsubd %ymm1, %ymm0, %ymm3
-; AVX2-NEXT:    vpsubd %ymm0, %ymm1, %ymm0
-; AVX2-NEXT:    vblendvps %ymm2, %ymm0, %ymm3, %ymm0
+; AVX2-NEXT:    vpminsd %ymm1, %ymm0, %ymm2
+; AVX2-NEXT:    vpmaxsd %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    vpsubd %ymm2, %ymm0, %ymm0
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: abd_cmp_v8i32:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpcmpgtd %ymm0, %ymm1, %k1
-; AVX512-NEXT:    vpsubd %ymm1, %ymm0, %ymm2
-; AVX512-NEXT:    vpsubd %ymm0, %ymm1, %ymm2 {%k1}
-; AVX512-NEXT:    vmovdqa %ymm2, %ymm0
+; AVX512-NEXT:    vpminsd %ymm1, %ymm0, %ymm2
+; AVX512-NEXT:    vpmaxsd %ymm1, %ymm0, %ymm0
+; AVX512-NEXT:    vpsubd %ymm2, %ymm0, %ymm0
 ; AVX512-NEXT:    retq
   %cmp = icmp slt <8 x i32> %a, %b
   %ab = sub <8 x i32> %a, %b
@@ -559,22 +541,22 @@ define <8 x i32> @abd_cmp_v8i32(<8 x i32> %a, <8 x i32> %b) nounwind {
 define <4 x i64> @abd_cmp_v4i64(<4 x i64> %a, <4 x i64> %b) nounwind {
 ; AVX1-LABEL: abd_cmp_v4i64:
 ; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpcmpgtq %xmm0, %xmm1, %xmm2
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm3
-; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm4
-; AVX1-NEXT:    vpcmpgtq %xmm3, %xmm4, %xmm5
-; AVX1-NEXT:    vpsubq %xmm1, %xmm0, %xmm6
-; AVX1-NEXT:    vpsubq %xmm4, %xmm3, %xmm7
+; AVX1-NEXT:    vpcmpgtq %xmm2, %xmm3, %xmm4
+; AVX1-NEXT:    vpsubq %xmm2, %xmm3, %xmm5
+; AVX1-NEXT:    vpsubq %xmm3, %xmm2, %xmm2
+; AVX1-NEXT:    vblendvpd %xmm4, %xmm5, %xmm2, %xmm2
+; AVX1-NEXT:    vpcmpgtq %xmm1, %xmm0, %xmm3
+; AVX1-NEXT:    vpsubq %xmm1, %xmm0, %xmm4
 ; AVX1-NEXT:    vpsubq %xmm0, %xmm1, %xmm0
-; AVX1-NEXT:    vblendvpd %xmm2, %xmm6, %xmm0, %xmm0
-; AVX1-NEXT:    vpsubq %xmm3, %xmm4, %xmm1
-; AVX1-NEXT:    vblendvpd %xmm5, %xmm7, %xmm1, %xmm1
-; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
+; AVX1-NEXT:    vblendvpd %xmm3, %xmm4, %xmm0, %xmm0
+; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm0, %ymm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: abd_cmp_v4i64:
 ; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpcmpgtq %ymm0, %ymm1, %ymm2
+; AVX2-NEXT:    vpcmpgtq %ymm1, %ymm0, %ymm2
 ; AVX2-NEXT:    vpsubq %ymm1, %ymm0, %ymm3
 ; AVX2-NEXT:    vpsubq %ymm0, %ymm1, %ymm0
 ; AVX2-NEXT:    vblendvpd %ymm2, %ymm3, %ymm0, %ymm0
@@ -582,15 +564,14 @@ define <4 x i64> @abd_cmp_v4i64(<4 x i64> %a, <4 x i64> %b) nounwind {
 ;
 ; AVX512-LABEL: abd_cmp_v4i64:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpcmpnltq %ymm1, %ymm0, %k1
-; AVX512-NEXT:    vpsubq %ymm1, %ymm0, %ymm2
-; AVX512-NEXT:    vpsubq %ymm0, %ymm1, %ymm2 {%k1}
-; AVX512-NEXT:    vmovdqa %ymm2, %ymm0
+; AVX512-NEXT:    vpminsq %ymm1, %ymm0, %ymm2
+; AVX512-NEXT:    vpmaxsq %ymm1, %ymm0, %ymm0
+; AVX512-NEXT:    vpsubq %ymm2, %ymm0, %ymm0
 ; AVX512-NEXT:    retq
   %cmp = icmp sge <4 x i64> %a, %b
   %ab = sub <4 x i64> %a, %b
   %ba = sub <4 x i64> %b, %a
-  %sel = select <4 x i1> %cmp, <4 x i64> %ba, <4 x i64> %ab
+  %sel = select <4 x i1> %cmp, <4 x i64> %ab, <4 x i64> %ba
   ret <4 x i64> %sel
 }
 
diff --git a/llvm/test/CodeGen/X86/abds-vector-512.ll b/llvm/test/CodeGen/X86/abds-vector-512.ll
index b21cc31b9d238..359b962a152b2 100644
--- a/llvm/test/CodeGen/X86/abds-vector-512.ll
+++ b/llvm/test/CodeGen/X86/abds-vector-512.ll
@@ -267,26 +267,22 @@ define <8 x i64> @abd_minmax_v8i64(<8 x i64> %a, <8 x i64> %b) nounwind {
 define <64 x i8> @abd_cmp_v64i8(<64 x i8> %a, <64 x i8> %b) nounwind {
 ; AVX512BW-LABEL: abd_cmp_v64i8:
 ; AVX512BW:       # %bb.0:
-; AVX512BW-NEXT:    vpcmpgtb %zmm1, %zmm0, %k1
-; AVX512BW-NEXT:    vpsubb %zmm0, %zmm1, %zmm2
-; AVX512BW-NEXT:    vpsubb %zmm1, %zmm0, %zmm2 {%k1}
-; AVX512BW-NEXT:    vmovdqa64 %zmm2, %zmm0
+; AVX512BW-NEXT:    vpminsb %zmm1, %zmm0, %zmm2
+; AVX512BW-NEXT:    vpmaxsb %zmm1, %zmm0, %zmm0
+; AVX512BW-NEXT:    vpsubb %zmm2, %zmm0, %zmm0
 ; AVX512BW-NEXT:    retq
 ;
 ; AVX512DQ-LABEL: abd_cmp_v64i8:
 ; AVX512DQ:       # %bb.0:
 ; AVX512DQ-NEXT:    vextracti64x4 $1, %zmm1, %ymm2
 ; AVX512DQ-NEXT:    vextracti64x4 $1, %zmm0, %ymm3
-; AVX512DQ-NEXT:    vpcmpgtb %ymm2, %ymm3, %ymm4
-; AVX512DQ-NEXT:    vpcmpgtb %ymm1, %ymm0, %ymm5
-; AVX512DQ-NEXT:    vinserti64x4 $1, %ymm4, %zmm5, %zmm4
-; AVX512DQ-NEXT:    vpsubb %ymm2, %ymm3, %ymm5
-; AVX512DQ-NEXT:    vpsubb %ymm1, %ymm0, %ymm6
-; AVX512DQ-NEXT:    vinserti64x4 $1, %ymm5, %zmm6, %zmm5
-; AVX512DQ-NEXT:    vpsubb %ymm3, %ymm2, %ymm2
-; AVX512DQ-NEXT:    vpsubb %ymm0, %ymm1, %ymm0
+; AVX512DQ-NEXT:    vpminsb %ymm2, %ymm3, %ymm4
+; AVX512DQ-NEXT:    vpmaxsb %ymm2, %ymm3, %ymm2
+; AVX512DQ-NEXT:    vpsubb %ymm4, %ymm2, %ymm2
+; AVX512DQ-NEXT:    vpminsb %ymm1, %ymm0, %ymm3
+; AVX512DQ-NEXT:    vpmaxsb %ymm1, %ymm0, %ymm0
+; AVX512DQ-NEXT:    vpsubb %ymm3, %ymm0, %ymm0
 ; AVX512DQ-NEXT:    vinserti64x4 $1, %ymm2, %zmm0, %zmm0
-; AVX512DQ-NEXT:    vpternlogq $184, %zmm5, %zmm4, %zmm0
 ; AVX512DQ-NEXT:    retq
   %cmp = icmp sgt <64 x i8> %a, %b
   %ab = sub <64 x i8> %a, %b
@@ -298,26 +294,22 @@ define <64 x i8> @abd_cmp_v64i8(<64 x i8> %a, <64 x i8> %b) nounwind {
 define <32 x i16> @abd_cmp_v32i16(<32 x i16> %a, <32 x i16> %b) nounwind {
 ; AVX512BW-LABEL: abd_cmp_v32i16:
 ; AVX512BW:       # %bb.0:
-; AVX512BW-NEXT:    vpcmpnltw %zmm1, %zmm0, %k1
-; AVX512BW-NEXT:    vpsubw %zmm0, %zmm1, %zmm2
-; AVX512BW-NEXT:    vpsubw %zmm1, %zmm0, %zmm2 {%k1}
-; AVX512BW-NEXT:    vmovdqa64 %zmm2, %zmm0
+; AVX512BW-NEXT:    vpminsw %zmm1, %zmm0, %zmm2
+; AVX512BW-NEXT:    vpmaxsw %zmm1, %zmm0, %zmm0
+; AVX512BW-NEXT:    vpsubw %zmm2, %zmm0, %zmm0
 ; AVX512BW-NEXT:    retq
 ;
 ; AVX512DQ-LABEL: abd_cmp_v32i16:
 ; AVX512DQ:       # %bb.0:
-; AVX512DQ-NEXT:    vextracti64x4 $1, %zmm0, %ymm2
-; AVX512DQ-NEXT:    vextracti64x4 $1, %zmm1, %ymm3
-; AVX512DQ-NEXT:    vpcmpgtw %ymm2, %ymm3, %ymm4
-; AVX512DQ-NEXT:    vpcmpgtw %ymm0, %ymm1, %ymm5
-; AVX512DQ-NEXT:    vinserti64x4 $1, %ymm4, %zmm5, %zmm4
-; AVX512DQ-NEXT:    vpsubw %ymm3, %ymm2, %ymm5
-; AVX512DQ-NEXT:    vpsubw %ymm1, %ymm0, %ymm6
-; AVX512DQ-NEXT:    vinserti64x4 $1, %ymm5, %zmm6, %zmm5
-; AVX512DQ-NEXT:    vpsubw %ymm2, %ymm3, %ymm2
-; AVX512DQ-NEXT:    vpsubw %ymm0, %ymm1, %ymm0
+; AVX512DQ-NEXT:    vextracti64x4 $1, %zmm1, %ymm2
+; AVX512DQ-NEXT:    vextracti64x4 $1, %zmm0, %ymm3
+; AVX512DQ-NEXT:    vpminsw %ymm2, %ymm3, %ymm4
+; AVX512DQ-NEXT:    vpmaxsw %ymm2, %ymm3, %ymm2
+; AVX512DQ-NEXT:    vpsubw %ymm4, %ymm2, %ymm2
+; AVX512DQ-NEXT:    vpminsw %ymm1, %ymm0, %ymm3
+; AVX512DQ-NEXT:    vpmaxsw %ymm1, %ymm0, %ymm0
+; AVX512DQ-NEXT:    vpsubw %ymm3, %ymm0, %ymm0
 ; AVX512DQ-NEXT:    vinserti64x4 $1, %ymm2, %zmm0, %zmm0
-; AVX512DQ-NEXT:    vpternlogq $226, %zmm5, %zmm4, %zmm0
 ; AVX512DQ-NEXT:    retq
   %cmp = icmp sge <32 x i16> %a, %b
   %ab = sub <32 x i16> %a, %b
@@ -329,10 +321,9 @@ define <32 x i16> @abd_cmp_v32i16(<32 x i16> %a, <32 x i16> %b) nounwind {
 define <16 x i32> @abd_cmp_v16i32(<16 x i32> %a, <16 x i32> %b) nounwind {
 ; AVX512-LABEL: abd_cmp_v16i32:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpcmpgtd %zmm0, %zmm1, %k1
-; AVX512-NEXT:    vpsubd %zmm1, %zmm0, %zmm2
-; AVX512-NEXT:    vpsubd %zmm0, %zmm1, %zmm2 {%k1}
-; AVX512-NEXT:    vmovdqa64 %zmm2, %zmm0
+; AVX512-NEXT:    vpminsd %zmm1, %zmm0, %zmm2
+; AVX512-NEXT:    vpmaxsd %zmm1, %zmm0, %zmm0
+; AVX512-NEXT:    vpsubd %zmm2, %zmm0, %zmm0
 ; AVX512-NEXT:    retq
   %cmp = icmp slt <16 x i32> %a, %b
   %ab = sub <16 x i32> %a, %b
@@ -344,15 +335,14 @@ define <16 x i32> @abd_cmp_v16i32(<16 x i32> %a, <16 x i32> %b) nounwind {
 define <8 x i64> @abd_cmp_v8i64(<8 x i64> %a, <8 x i64> %b) nounwind {
 ; AVX512-LABEL: abd_cmp_v8i64:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpcmpnltq %zmm1, %zmm0, %k1
-; AVX512-NEXT:    vpsubq %zmm1, %zmm0, %zmm2
-; AVX512-NEXT:    vpsubq %zmm0, %zmm1, %zmm2 {%k1}
-; AVX512-NEXT:    vmovdqa64 %zmm2, %zmm0
+; AVX512-NEXT:    vpminsq %zmm1, %zmm0, %zmm2
+; AVX512-NEXT:    vpmaxsq %zmm1, %zmm0, %zmm0
+; AVX512-NEXT:    vpsubq %zmm2, %zmm0, %zmm0
 ; AVX512-NEXT:    retq
   %cmp = icmp sge <8 x i64> %a, %b
   %ab = sub <8 x i64> %a, %b
   %ba = sub <8 x i64> %b, %a
-  %sel = select <8 x i1> %cmp, <8 x i64> %ba, <8 x i64> %ab
+  %sel = select <8 x i1> %cmp, <8 x i64> %ab, <8 x i64> %ba
   ret <8 x i64> %sel
 }
 
diff --git a/llvm/test/CodeGen/X86/abdu-vector-128.ll b/llvm/test/CodeGen/X86/abdu-vector-128.ll
index a2a6831249127..88496032aa307 100644
--- a/llvm/test/CodeGen/X86/abdu-vector-128.ll
+++ b/llvm/test/CodeGen/X86/abdu-vector-128.ll
@@ -641,59 +641,20 @@ define <2 x i64> @abd_minmax_v2i64(<2 x i64> %a, <2 x i64> %b) nounwind {
 ;
 
 define <16 x i8> @abd_cmp_v16i8(<16 x i8> %a, <16 x i8> %b) nounwind {
-; SSE2-LABEL: abd_cmp_v16i8:
-; SSE2:       # %bb.0:
-; SSE2-NEXT:    movdqa %xmm0, %xmm2
-; SSE2-NEXT:    pminub %xmm1, %xmm2
-; SSE2-NEXT:    pcmpeqb %xmm0, %xmm2
-; SSE2-NEXT:    movdqa %xmm0, %xmm3
-; SSE2-NEXT:    psubb %xmm1, %xmm3
-; SSE2-NEXT:    psubb %xmm0, %xmm1
-; SSE2-NEXT:    pand %xmm2, %xmm1
-; SSE2-NEXT:    pandn %xmm3, %xmm2
-; SSE2-NEXT:    por %xmm1, %xmm2
-; SSE2-NEXT:    movdqa %xmm2, %xmm0
-; SSE2-NEXT:    retq
-;
-; SSE42-LABEL: abd_cmp_v16i8:
-; SSE42:       # %bb.0:
-; SSE42-NEXT:    movdqa %xmm0, %xmm2
-; SSE42-NEXT:    pminub %xmm1, %xmm2
-; SSE42-NEXT:    pcmpeqb %xmm0, %xmm2
-; SSE42-NEXT:    movdqa %xmm0, %xmm3
-; SSE42-NEXT:    psubb %xmm1, %xmm3
-; SSE42-NEXT:    psubb %xmm0, %xmm1
-; SSE42-NEXT:    movdqa %xmm2, %xmm0
-; SSE42-NEXT:    pblendvb %xmm0, %xmm1, %xmm3
-; SSE42-NEXT:    movdqa %xmm3, %xmm0
-; SSE42-NEXT:    retq
-;
-; AVX1-LABEL: abd_cmp_v16i8:
-; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpminub %xmm1, %xmm0, %xmm2
-; AVX1-NEXT:    vpcmpeqb %xmm2, %xmm0, %xmm2
-; AVX1-NEXT:    vpsubb %xmm1, %xmm0, %xmm3
-; AVX1-NEXT:    vpsubb %xmm0, %xmm1, %xmm0
-; AVX1-NEXT:    vpblendvb %xmm2, %xmm0, %xmm3, %xmm0
-; AVX1-NEXT:    retq
-;
-; AVX2-LABEL: abd_cmp_v16i8:
-; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpminub %xmm1, %xmm0, %xmm2
-; AVX2-NEXT:    vpcmpeqb %xmm2, %xmm0, %xmm2
-; AVX2-NEXT:    vpsubb %xmm1, %xmm0, %xmm3
-; AVX2-NEXT:    vpsubb %xmm0, %xmm1, %xmm0
-; AVX2-NEXT:    vpblendvb %xmm2, %xmm0, %xmm3, %xmm0
-; AVX2-NEXT:    retq
+; SSE-LABEL: abd_cmp_v16i8:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movdqa %xmm0, %xmm2
+; SSE-NEXT:    pminub %xmm1, %xmm2
+; SSE-NEXT:    pmaxub %xmm1, %xmm0
+; SSE-NEXT:    psubb %xmm2, %xmm0
+; SSE-NEXT:    retq
 ;
-; AVX512-LABEL: abd_cmp_v16i8:
-; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpsubb %xmm1, %xmm0, %xmm2
-; AVX512-NEXT:    vpsubb %xmm0, %xmm1, %xmm3
-; AVX512-NEXT:    vpminub %xmm1, %xmm0, %xmm1
-; AVX512-NEXT:    vpcmpeqb %xmm1, %xmm0, %xmm0
-; AVX512-NEXT:    vpblendvb %xmm0, %xmm3, %xmm2, %xmm0
-; AVX512-NEXT:    retq
+; AVX-LABEL: abd_cmp_v16i8:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpminub %xmm1, %xmm0, %xmm2
+; AVX-NEXT:    vpmaxub %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vpsubb %xmm2, %xmm0, %xmm0
+; AVX-NEXT:    retq
   %cmp = icmp ugt <16 x i8> %a, %b
   %ab = sub <16 x i8> %a, %b
   %ba = sub <16 x i8> %b, %a
@@ -720,42 +681,17 @@ define <8 x i16> @abd_cmp_v8i16(<8 x i16> %a, <8 x i16> %b) nounwind {
 ; SSE42-LABEL: abd_cmp_v8i16:
 ; SSE42:       # %bb.0:
 ; SSE42-NEXT:    movdqa %xmm0, %xmm2
-; SSE42-NEXT:    pmaxuw %xmm1, %xmm2
-; SSE42-NEXT:    pcmpeqw %xmm0, %xmm2
-; SSE42-NEXT:    movdqa %xmm0, %xmm3
-; SSE42-NEXT:    psubw %xmm1, %xmm3
-; SSE42-NEXT:    psubw %xmm0, %xmm1
-; SSE42-NEXT:    movdqa %xmm2, %xmm0
-; SSE42-NEXT:    pblendvb %xmm0, %xmm3, %xmm1
-; SSE42-NEXT:    movdqa %xmm1, %xmm0
+; SSE42-NEXT:    pminuw %xmm1, %xmm2
+; SSE42-NEXT:    pmaxuw %xmm1, %xmm0
+; SSE42-NEXT:    psubw %xmm2, %xmm0
 ; SSE42-NEXT:    retq
 ;
-; AVX1-LABEL: abd_cmp_v8i16:
-; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpmaxuw %xmm1, %xmm0, %xmm2
-; AVX1-NEXT:    vpcmpeqw %xmm2, %xmm0, %xmm2
-; AVX1-NEXT:    vpsubw %xmm1, %xmm0, %xmm3
-; AVX1-NEXT:    vpsubw %xmm0, %xmm1, %xmm0
-; AVX1-NEXT:    vpblendvb %xmm2, %xmm3, %xmm0, %xmm0
-; AVX1-NEXT:    retq
-;
-; AVX2-LABEL: abd_cmp_v8i16:
-; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpmaxuw %xmm1, %xmm0, %xmm2
-; AVX2-NEXT:    vpcmpeqw %xmm2, %xmm0, %xmm2
-; AVX2-NEXT:    vpsubw %xmm1, %xmm0, %xmm3
-; AVX2-NEXT:    vpsubw %xmm0, %xmm1, %xmm0
-; AVX2-NEXT:    vpblendvb %xmm2, %xmm3, %xmm0, %xmm0
-; AVX2-NEXT:    retq
-;
-; AVX512-LABEL: abd_cmp_v8i16:
-; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpsubw %xmm1, %xmm0, %xmm2
-; AVX512-NEXT:    vpsubw %xmm0, %xmm1, %xmm3
-; AVX512-NEXT:    vpmaxuw %xmm1, %xmm0, %xmm1
-; AVX512-NEXT:    vpcmpeqw %xmm1, %xmm0, %xmm0
-; AVX512-NEXT:    vpblendvb %xmm0, %xmm2, %xmm3, %xmm0
-; AVX512-NEXT:    retq
+; AVX-LABEL: abd_cmp_v8i16:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpminuw %xmm1, %xmm0, %xmm2
+; AVX-NEXT:    vpmaxuw %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vpsubw %xmm2, %xmm0, %xmm0
+; AVX-NEXT:    retq
   %cmp = icmp uge <8 x i16> %a, %b
   %ab = sub <8 x i16> %a, %b
   %ba = sub <8 x i16> %b, %a
@@ -783,41 +719,17 @@ define <4 x i32> @abd_cmp_v4i32(<4 x i32> %a, <4 x i32> %b) nounwind {
 ; SSE42-LABEL: abd_cmp_v4i32:
 ; SSE42:       # %bb.0:
 ; SSE42-NEXT:    movdqa %xmm0, %xmm2
-; SSE42-NEXT:    pmaxud %xmm1, %xmm2
-; SSE42-NEXT:    pcmpeqd %xmm0, %xmm2
-; SSE42-NEXT:    movdqa %xmm0, %xmm3
-; SSE42-NEXT:    psubd %xmm1, %xmm3
-; SSE42-NEXT:    psubd %xmm0, %xmm1
-; SSE42-NEXT:    movdqa %xmm2, %xmm0
-; SSE42-NEXT:    blendvps %xmm0, %xmm3, %xmm1
-; SSE42-NEXT:    movaps %xmm1, %xmm0
+; SSE42-NEXT:    pminud %xmm1, %xmm2
+; SSE42-NEXT:    pmaxud %xmm1, %xmm0
+; SSE42-NEXT:    psubd %xmm2, %xmm0
 ; SSE42-NEXT:    retq
 ;
-; AVX1-LABEL: abd_cmp_v4i32:
-; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpmaxud %xmm1, %xmm0, %xmm2
-; AVX1-NEXT:    vpcmpeqd %xmm2, %xmm0, %xmm2
-; AVX1-NEXT:    vpsubd %xmm1, %xmm0, %xmm3
-; AVX1-NEXT:    vpsubd %xmm0, %xmm1, %xmm0
-; AVX1-NEXT:    vblendvps %xmm2, %xmm3, %xmm0, %xmm0
-; AVX1-NEXT:    retq
-;
-; AVX2-LABEL: abd_cmp_v4i32:
-; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpmaxud %xmm1, %xmm0, %xmm2
-; AVX2-NEXT:    vpcmpeqd %xmm2, %xmm0, %xmm2
-; AVX2-NEXT:    vpsubd %xmm1, %xmm0, %xmm3
-; AVX2-NEXT:    vpsubd %xmm0, %xmm1, %xmm0
-; AVX2-NEXT:    vblendvps %xmm2, %xmm3, %xmm0, %xmm0
-; AVX2-NEXT:    retq
-;
-; AVX512-LABEL: abd_cmp_v4i32:
-; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpcmpltud %xmm1, %xmm0, %k1
-; AVX512-NEXT:    vpsubd %xmm1, %xmm0, %xmm2
-; AVX512-NEXT:    vpsubd %xmm0, %xmm1, %xmm2 {%k1}
-; AVX512-NEXT:    vmovdqa %xmm2, %xmm0
-; AVX512-NEXT:    retq
+; AVX-LABEL: abd_cmp_v4i32:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpminud %xmm1, %xmm0, %xmm2
+; AVX-NEXT:    vpmaxud %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vpsubd %xmm2, %xmm0, %xmm0
+; AVX-NEXT:    retq
   %cmp = icmp ult <4 x i32> %a, %b
   %ab = sub <4 x i32> %a, %b
   %ba = sub <4 x i32> %b, %a
@@ -843,18 +755,18 @@ define <2 x i64> @abd_cmp_v2i64(<2 x i64> %a, <2 x i64> %b) nounwind {
 ; SSE2-NEXT:    movdqa %xmm0, %xmm3
 ; SSE2-NEXT:    psubq %xmm1, %xmm3
 ; SSE2-NEXT:    psubq %xmm0, %xmm1
-; SSE2-NEXT:    pand %xmm2, %xmm3
-; SSE2-NEXT:    pandn %xmm1, %xmm2
-; SSE2-NEXT:    por %xmm3, %xmm2
+; SSE2-NEXT:    pand %xmm2, %xmm1
+; SSE2-NEXT:    pandn %xmm3, %xmm2
+; SSE2-NEXT:    por %xmm1, %xmm2
 ; SSE2-NEXT:    movdqa %xmm2, %xmm0
 ; SSE2-NEXT:    retq
 ;
 ; SSE42-LABEL: abd_cmp_v2i64:
 ; SSE42:       # %bb.0:
 ; SSE42-NEXT:    movdqa {{.*#+}} xmm2 = [9223372036854775808,9223372036854775808]
-; SSE42-NEXT:    movdqa %xmm0, %xmm3
+; SSE42-NEXT:    movdqa %xmm1, %xmm3
 ; SSE42-NEXT:    pxor %xmm2, %xmm3
-; SSE42-NEXT:    pxor %xmm1, %xmm2
+; SSE42-NEXT:    pxor %xmm0, %xmm2
 ; SSE42-NEXT:    pcmpgtq %xmm3, %xmm2
 ; SSE42-NEXT:    movdqa %xmm0, %xmm3
 ; SSE42-NEXT:    psubq %xmm1, %xmm3
@@ -867,8 +779,8 @@ define <2 x i64> @abd_cmp_v2i64(<2 x i64> %a, <2 x i64> %b) nounwind {
 ; AVX1-LABEL: abd_cmp_v2i64:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    vmovdqa {{.*#+}} xmm2 = [9223372036854775808,9223372036854775808]
-; AVX1-NEXT:    vpxor %xmm2, %xmm0, %xmm3
-; AVX1-NEXT:    vpxor %xmm2, %xmm1, %xmm2
+; AVX1-NEXT:    vpxor %xmm2, %xmm1, %xmm3
+; AVX1-NEXT:    vpxor %xmm2, %xmm0, %xmm2
 ; AVX1-NEXT:    vpcmpgtq %xmm3, %xmm2, %xmm2
 ; AVX1-NEXT:    vpsubq %xmm1, %xmm0, %xmm3
 ; AVX1-NEXT:    vpsubq %xmm0, %xmm1, %xmm0
@@ -878,8 +790,8 @@ define <2 x i64> @abd_cmp_v2i64(<2 x i64> %a, <2 x i64> %b) nounwind {
 ; AVX2-LABEL: abd_cmp_v2i64:
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    vmovdqa {{.*#+}} xmm2 = [9223372036854775808,9223372036854775808]
-; AVX2-NEXT:    vpxor %xmm2, %xmm0, %xmm3
-; AVX2-NEXT:    vpxor %xmm2, %xmm1, %xmm2
+; AVX2-NEXT:    vpxor %xmm2, %xmm1, %xmm3
+; AVX2-NEXT:    vpxor %xmm2, %xmm0, %xmm2
 ; AVX2-NEXT:    vpcmpgtq %xmm3, %xmm2, %xmm2
 ; AVX2-NEXT:    vpsubq %xmm1, %xmm0, %xmm3
 ; AVX2-NEXT:    vpsubq %xmm0, %xmm1, %xmm0
@@ -888,15 +800,14 @@ define <2 x i64> @abd_cmp_v2i64(<2 x i64> %a, <2 x i64> %b) nounwind {
 ;
 ; AVX512-LABEL: abd_cmp_v2i64:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpcmpnltuq %xmm1, %xmm0, %k1
-; AVX512-NEXT:    vpsubq %xmm1, %xmm0, %xmm2
-; AVX512-NEXT:    vpsubq %xmm0, %xmm1, %xmm2 {%k1}
-; AVX512-NEXT:    vmovdqa %xmm2, %xmm0
+; AVX512-NEXT:    vpminuq %xmm1, %xmm0, %xmm2
+; AVX512-NEXT:    vpmaxuq %xmm1, %xmm0, %xmm0
+; AVX512-NEXT:    vpsubq %xmm2, %xmm0, %xmm0
 ; AVX512-NEXT:    retq
   %cmp = icmp uge <2 x i64> %a, %b
   %ab = sub <2 x i64> %a, %b
   %ba = sub <2 x i64> %b, %a
-  %sel = select <2 x i1> %cmp, <2 x i64> %ba, <2 x i64> %ab
+  %sel = select <2 x i1> %cmp, <2 x i64> %ab, <2 x i64> %ba
   ret <2 x i64> %sel
 }
 
@@ -932,50 +843,52 @@ define <2 x i64> @abd_cmp_v2i64_multiuse_cmp(<2 x i64> %a, <2 x i64> %b) nounwin
 ;
 ; SSE42-LABEL: abd_cmp_v2i64_multiuse_cmp:
 ; SSE42:       # %bb.0:
-; SSE42-NEXT:    movdqa %xmm0, %xmm2
-; SSE42-NEXT:    psubq %xmm1, %xmm2
+; SSE42-NEXT:    movdqa {{.*#+}} xmm2 = [9223372036854775808,9223372036854775808]
 ; SSE42-NEXT:    movdqa %xmm1, %xmm3
-; SSE42-NEXT:    psubq %xmm0, %xmm3
-; SSE42-NEXT:    movdqa {{.*#+}} xmm4 = [9223372036854775808,9223372036854775808]
-; SSE42-NEXT:    pxor %xmm4, %xmm1
-; SSE42-NEXT:    pxor %xmm4, %xmm0
-; SSE42-NEXT:    pcmpgtq %xmm1, %xmm0
-; SSE42-NEXT:    blendvpd %xmm0, %xmm2, %xmm3
-; SSE42-NEXT:    paddq %xmm3, %xmm0
+; SSE42-NEXT:    pxor %xmm2, %xmm3
+; SSE42-NEXT:    pxor %xmm0, %xmm2
+; SSE42-NEXT:    pcmpgtq %xmm3, %xmm2
+; SSE42-NEXT:    movdqa %xmm0, %xmm3
+; SSE42-NEXT:    psubq %xmm1, %xmm3
+; SSE42-NEXT:    psubq %xmm0, %xmm1
+; SSE42-NEXT:    movdqa %xmm2, %xmm0
+; SSE42-NEXT:    blendvpd %xmm0, %xmm3, %xmm1
+; SSE42-NEXT:    paddq %xmm1, %xmm2
+; SSE42-NEXT:    movdqa %xmm2, %xmm0
 ; SSE42-NEXT:    retq
 ;
 ; AVX1-LABEL: abd_cmp_v2i64_multiuse_cmp:
 ; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpsubq %xmm1, %xmm0, %xmm2
-; AVX1-NEXT:    vpsubq %xmm0, %xmm1, %xmm3
-; AVX1-NEXT:    vmovdqa {{.*#+}} xmm4 = [9223372036854775808,9223372036854775808]
-; AVX1-NEXT:    vpxor %xmm4, %xmm1, %xmm1
-; AVX1-NEXT:    vpxor %xmm4, %xmm0, %xmm0
-; AVX1-NEXT:    vpcmpgtq %xmm1, %xmm0, %xmm0
-; AVX1-NEXT:    vblendvpd %xmm0, %xmm2, %xmm3, %xmm1
-; AVX1-NEXT:    vpaddq %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vmovdqa {{.*#+}} xmm2 = [9223372036854775808,9223372036854775808]
+; AVX1-NEXT:    vpxor %xmm2, %xmm1, %xmm3
+; AVX1-NEXT:    vpxor %xmm2, %xmm0, %xmm2
+; AVX1-NEXT:    vpcmpgtq %xmm3, %xmm2, %xmm2
+; AVX1-NEXT:    vpsubq %xmm1, %xmm0, %xmm3
+; AVX1-NEXT:    vpsubq %xmm0, %xmm1, %xmm0
+; AVX1-NEXT:    vblendvpd %xmm2, %xmm3, %xmm0, %xmm0
+; AVX1-NEXT:    vpaddq %xmm0, %xmm2, %xmm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: abd_cmp_v2i64_multiuse_cmp:
 ; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpsubq %xmm1, %xmm0, %xmm2
-; AVX2-NEXT:    vpsubq %xmm0, %xmm1, %xmm3
-; AVX2-NEXT:    vmovdqa {{.*#+}} xmm4 = [9223372036854775808,9223372036854775808]
-; AVX2-NEXT:    vpxor %xmm4, %xmm1, %xmm1
-; AVX2-NEXT:    vpxor %xmm4, %xmm0, %xmm0
-; AVX2-NEXT:    vpcmpgtq %xmm1, %xmm0, %xmm0
-; AVX2-NEXT:    vblendvpd %xmm0, %xmm2, %xmm3, %xmm1
-; AVX2-NEXT:    vpaddq %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vmovdqa {{.*#+}} xmm2 = [9223372036854775808,9223372036854775808]
+; AVX2-NEXT:    vpxor %xmm2, %xmm1, %xmm3
+; AVX2-NEXT:    vpxor %xmm2, %xmm0, %xmm2
+; AVX2-NEXT:    vpcmpgtq %xmm3, %xmm2, %xmm2
+; AVX2-NEXT:    vpsubq %xmm1, %xmm0, %xmm3
+; AVX2-NEXT:    vpsubq %xmm0, %xmm1, %xmm0
+; AVX2-NEXT:    vblendvpd %xmm2, %xmm3, %xmm0, %xmm0
+; AVX2-NEXT:    vpaddq %xmm0, %xmm2, %xmm0
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: abd_cmp_v2i64_multiuse_cmp:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vpcmpnleuq %xmm1, %xmm0, %k1
-; AVX512-NEXT:    vpsubq %xmm0, %xmm1, %xmm2
-; AVX512-NEXT:    vpsubq %xmm1, %xmm0, %xmm2 {%k1}
-; AVX512-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; AVX512-NEXT:    vpaddq %xmm0, %xmm2, %xmm2 {%k1}
-; AVX512-NEXT:    vmovdqa %xmm2, %xmm0
+; AVX512-NEXT:    vpminuq %xmm1, %xmm0, %xmm2
+; AVX512-NEXT:    vpmaxuq %xmm1, %xmm0, %xmm0
+; AVX512-NEXT:    vpsubq %xmm2, %xmm0, %xmm0
+; AVX512-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
+; AVX512-NEXT:    vpaddq %xmm1, %xmm0, %xmm0 {%k1}
 ; AVX512-NEXT:    retq
   %cmp = icmp ugt <2 x i64> %a, %b
   %ab = sub <2 x i64> %a, %b
@@ -1007,46 +920,22 @@ define <8 x i16> @abd_cmp_v8i16_multiuse_sub(<8 x i16> %a, <8 x i16> %b) nounwin
 ; SSE42-LABEL: abd_cmp_v8i16_multiuse_sub:
 ; SSE42:       # %bb.0:
 ; SSE42-NEXT:    movdqa %xmm0, %xmm2
-; SSE42-NEXT:    pmaxuw %xmm1, %xmm2
-; SSE42-NEXT:    pcmpeqw %xmm0, %xmm2
+; SSE42-NEXT:    psubw %xmm1, %xmm2
 ; SSE42-NEXT:    movdqa %xmm0, %xmm3
-; SSE42-NEXT:    psubw %xmm1, %xmm3
-; SSE42-NEXT:    psubw %xmm0, %xmm1
-; SSE42-NEXT:    movdqa %xmm2, %xmm0
-; SSE42-NEXT:    pblendvb %xmm0, %xmm3, %xmm1
-; SSE42-NEXT:    paddw %xmm1, %xmm3
-; SSE42-NEXT:    movdqa %xmm3, %xmm0
+; SSE42-NEXT:    pminuw %xmm1, %xmm3
+; SSE42-NEXT:    pmaxuw %xmm1, %xmm0
+; SSE42-NEXT:    psubw %xmm3, %xmm0
+; SSE42-NEXT:    paddw %xmm2, %xmm0
 ; SSE42-NEXT:    retq
 ;
-; AVX1-LABEL: abd_cmp_v8i16_multiuse_sub:
-; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpmaxuw %xmm1, %xmm0, %xmm2
-; AVX1-NEXT:    vpcmpeqw %xmm2, %xmm0, %xmm2
-; AVX1-NEXT:    vpsubw %xmm1, %xmm0, %xmm3
-; AVX1-NEXT:    vpsubw %xmm0, %xmm1, %xmm0
-; AVX1-NEXT:    vpblendvb %xmm2, %xmm3, %xmm0, %xmm0
-; AVX1-NEXT:    vpaddw %xmm0, %xmm3, %xmm0
-; AVX1-NEXT:    retq
-;
-; AVX2-LABEL: abd_cmp_v8i16_multiuse_sub:
-; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpmaxuw %xmm1, %xmm0, %xmm2
-; AVX2-NEXT:    vpcmpeqw %xmm2, %xmm0, %xmm2
-; AVX2-NEXT:    vpsubw %xmm1, %xmm0, %xmm3
-; AVX2-NEXT:    vpsubw %xmm0, %xmm1, %xmm0
-; AVX2-NEXT:    vpblendvb %xmm2, %xmm3, %xmm0, %xmm0
-; AVX2-NEXT:    vpaddw %xmm0, %xmm3, %xmm0
-; AVX2-NEXT:    retq
-;
-; AVX512-LABEL: abd_cmp_v8i16_multiuse_sub:
-; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpsubw %xmm1, %xmm0, %xmm2
-; AVX512-NEXT:    vpsubw %xmm0, %xmm1, %xmm3
-; AVX512-NEXT:    vpmaxuw %xmm1, %xmm0, %xmm1
-; AVX512-NEXT:    vpcmpeqw %xmm1, %xmm0, %xmm0
-; AVX512-NEXT:    vpblendvb %xmm0, %xmm2, %xmm3, %xmm0
-; AVX512-NEXT:    vpaddw %xmm0, %xmm2, %xmm0
-; AVX512-NEXT:    retq
+; AVX-LABEL: abd_cmp_v8i16_multiuse_sub:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpsubw %xmm1, %xmm0, %xmm2
+; AVX-NEXT:    vpminuw %xmm1, %xmm0, %xmm3
+; AVX-NEXT:    vpmaxuw %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vpsubw %xmm3, %xmm0, %xmm0
+; AVX-NEXT:    vpaddw %xmm0, %xmm2, %xmm0
+; AVX-NEXT:    retq
   %cmp = icmp uge <8 x i16> %a, %b
   %ab = sub <8 x i16> %a, %b
   %ba = sub <8 x i16> %b, %a
diff --git a/llvm/test/CodeGen/X86/abdu-vector-256.ll b/llvm/test/CodeGen/X86/abdu-vector-256.ll
index d4e9d8acc97f6..3957133574ce5 100644
--- a/llvm/test/CodeGen/X86/abdu-vector-256.ll
+++ b/llvm/test/CodeGen/X86/abdu-vector-256.ll
@@ -466,37 +466,26 @@ define <32 x i8> @abd_cmp_v32i8(<32 x i8> %a, <32 x i8> %b) nounwind {
 ; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm3
 ; AVX1-NEXT:    vpminub %xmm2, %xmm3, %xmm4
-; AVX1-NEXT:    vpcmpeqb %xmm4, %xmm3, %xmm4
-; AVX1-NEXT:    vpminub %xmm1, %xmm0, %xmm5
-; AVX1-NEXT:    vpcmpeqb %xmm5, %xmm0, %xmm5
-; AVX1-NEXT:    vinsertf128 $1, %xmm4, %ymm5, %ymm4
-; AVX1-NEXT:    vpsubb %xmm2, %xmm3, %xmm5
-; AVX1-NEXT:    vpsubb %xmm1, %xmm0, %xmm6
-; AVX1-NEXT:    vinsertf128 $1, %xmm5, %ymm6, %ymm5
-; AVX1-NEXT:    vpsubb %xmm3, %xmm2, %xmm2
-; AVX1-NEXT:    vpsubb %xmm0, %xmm1, %xmm0
+; AVX1-NEXT:    vpmaxub %xmm2, %xmm3, %xmm2
+; AVX1-NEXT:    vpsubb %xmm4, %xmm2, %xmm2
+; AVX1-NEXT:    vpminub %xmm1, %xmm0, %xmm3
+; AVX1-NEXT:    vpmaxub %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vpsubb %xmm3, %xmm0, %xmm0
 ; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm0, %ymm0
-; AVX1-NEXT:    vandps %ymm0, %ymm4, %ymm0
-; AVX1-NEXT:    vandnps %ymm5, %ymm4, %ymm1
-; AVX1-NEXT:    vorps %ymm0, %ymm1, %ymm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: abd_cmp_v32i8:
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    vpminub %ymm1, %ymm0, %ymm2
-; AVX2-NEXT:    vpcmpeqb %ymm2, %ymm0, %ymm2
-; AVX2-NEXT:    vpsubb %ymm1, %ymm0, %ymm3
-; AVX2-NEXT:    vpsubb %ymm0, %ymm1, %ymm0
-; AVX2-NEXT:    vpblendvb %ymm2, %ymm0, %ymm3, %ymm0
+; AVX2-NEXT:    vpmaxub %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    vpsubb %ymm2, %ymm0, %ymm0
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: abd_cmp_v32i8:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpsubb %ymm1, %ymm0, %ymm2
-; AVX512-NEXT:    vpsubb %ymm0, %ymm1, %ymm3
-; AVX512-NEXT:    vpminub %ymm1, %ymm0, %ymm1
-; AVX512-NEXT:    vpcmpeqb %ymm1, %ymm0, %ymm0
-; AVX512-NEXT:    vpblendvb %ymm0, %ymm3, %ymm2, %ymm0
+; AVX512-NEXT:    vpminub %ymm1, %ymm0, %ymm2
+; AVX512-NEXT:    vpmaxub %ymm1, %ymm0, %ymm0
+; AVX512-NEXT:    vpsubb %ymm2, %ymm0, %ymm0
 ; AVX512-NEXT:    retq
   %cmp = icmp ugt <32 x i8> %a, %b
   %ab = sub <32 x i8> %a, %b
@@ -510,38 +499,27 @@ define <16 x i16> @abd_cmp_v16i16(<16 x i16> %a, <16 x i16> %b) nounwind {
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm3
-; AVX1-NEXT:    vpmaxuw %xmm2, %xmm3, %xmm4
-; AVX1-NEXT:    vpcmpeqw %xmm4, %xmm3, %xmm4
-; AVX1-NEXT:    vpmaxuw %xmm1, %xmm0, %xmm5
-; AVX1-NEXT:    vpcmpeqw %xmm5, %xmm0, %xmm5
-; AVX1-NEXT:    vinsertf128 $1, %xmm4, %ymm5, %ymm4
-; AVX1-NEXT:    vpsubw %xmm2, %xmm3, %xmm5
-; AVX1-NEXT:    vpsubw %xmm1, %xmm0, %xmm6
-; AVX1-NEXT:    vinsertf128 $1, %xmm5, %ymm6, %ymm5
-; AVX1-NEXT:    vpsubw %xmm3, %xmm2, %xmm2
-; AVX1-NEXT:    vpsubw %xmm0, %xmm1, %xmm0
+; AVX1-NEXT:    vpminuw %xmm2, %xmm3, %xmm4
+; AVX1-NEXT:    vpmaxuw %xmm2, %xmm3, %xmm2
+; AVX1-NEXT:    vpsubw %xmm4, %xmm2, %xmm2
+; AVX1-NEXT:    vpminuw %xmm1, %xmm0, %xmm3
+; AVX1-NEXT:    vpmaxuw %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vpsubw %xmm3, %xmm0, %xmm0
 ; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm0, %ymm0
-; AVX1-NEXT:    vandnps %ymm0, %ymm4, %ymm0
-; AVX1-NEXT:    vandps %ymm4, %ymm5, %ymm1
-; AVX1-NEXT:    vorps %ymm0, %ymm1, %ymm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: abd_cmp_v16i16:
 ; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpmaxuw %ymm1, %ymm0, %ymm2
-; AVX2-NEXT:    vpcmpeqw %ymm2, %ymm0, %ymm2
-; AVX2-NEXT:    vpsubw %ymm1, %ymm0, %ymm3
-; AVX2-NEXT:    vpsubw %ymm0, %ymm1, %ymm0
-; AVX2-NEXT:    vpblendvb %ymm2, %ymm3, %ymm0, %ymm0
+; AVX2-NEXT:    vpminuw %ymm1, %ymm0, %ymm2
+; AVX2-NEXT:    vpmaxuw %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    vpsubw %ymm2, %ymm0, %ymm0
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: abd_cmp_v16i16:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpsubw %ymm1, %ymm0, %ymm2
-; AVX512-NEXT:    vpsubw %ymm0, %ymm1, %ymm3
-; AVX512-NEXT:    vpmaxuw %ymm1, %ymm0, %ymm1
-; AVX512-NEXT:    vpcmpeqw %ymm1, %ymm0, %ymm0
-; AVX512-NEXT:    vpblendvb %ymm0, %ymm2, %ymm3, %ymm0
+; AVX512-NEXT:    vpminuw %ymm1, %ymm0, %ymm2
+; AVX512-NEXT:    vpmaxuw %ymm1, %ymm0, %ymm0
+; AVX512-NEXT:    vpsubw %ymm2, %ymm0, %ymm0
 ; AVX512-NEXT:    retq
   %cmp = icmp uge <16 x i16> %a, %b
   %ab = sub <16 x i16> %a, %b
@@ -553,36 +531,29 @@ define <16 x i16> @abd_cmp_v16i16(<16 x i16> %a, <16 x i16> %b) nounwind {
 define <8 x i32> @abd_cmp_v8i32(<8 x i32> %a, <8 x i32> %b) nounwind {
 ; AVX1-LABEL: abd_cmp_v8i32:
 ; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpmaxud %xmm1, %xmm0, %xmm2
-; AVX1-NEXT:    vpcmpeqd %xmm2, %xmm0, %xmm2
-; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm3
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm4
-; AVX1-NEXT:    vpmaxud %xmm3, %xmm4, %xmm5
-; AVX1-NEXT:    vpcmpeqd %xmm5, %xmm4, %xmm5
-; AVX1-NEXT:    vpsubd %xmm1, %xmm0, %xmm6
-; AVX1-NEXT:    vpsubd %xmm3, %xmm4, %xmm7
-; AVX1-NEXT:    vpsubd %xmm0, %xmm1, %xmm0
-; AVX1-NEXT:    vblendvps %xmm2, %xmm6, %xmm0, %xmm0
-; AVX1-NEXT:    vpsubd %xmm4, %xmm3, %xmm1
-; AVX1-NEXT:    vblendvps %xmm5, %xmm7, %xmm1, %xmm1
-; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm3
+; AVX1-NEXT:    vpminud %xmm2, %xmm3, %xmm4
+; AVX1-NEXT:    vpmaxud %xmm2, %xmm3, %xmm2
+; AVX1-NEXT:    vpsubd %xmm4, %xmm2, %xmm2
+; AVX1-NEXT:    vpminud %xmm1, %xmm0, %xmm3
+; AVX1-NEXT:    vpmaxud %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vpsubd %xmm3, %xmm0, %xmm0
+; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm0, %ymm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: abd_cmp_v8i32:
 ; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpmaxud %ymm1, %ymm0, %ymm2
-; AVX2-NEXT:    vpcmpeqd %ymm2, %ymm0, %ymm2
-; AVX2-NEXT:    vpsubd %ymm1, %ymm0, %ymm3
-; AVX2-NEXT:    vpsubd %ymm0, %ymm1, %ymm0
-; AVX2-NEXT:    vblendvps %ymm2, %ymm3, %ymm0, %ymm0
+; AVX2-NEXT:    vpminud %ymm1, %ymm0, %ymm2
+; AVX2-NEXT:    vpmaxud %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    vpsubd %ymm2, %ymm0, %ymm0
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: abd_cmp_v8i32:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpcmpltud %ymm1, %ymm0, %k1
-; AVX512-NEXT:    vpsubd %ymm1, %ymm0, %ymm2
-; AVX512-NEXT:    vpsubd %ymm0, %ymm1, %ymm2 {%k1}
-; AVX512-NEXT:    vmovdqa %ymm2, %ymm0
+; AVX512-NEXT:    vpminud %ymm1, %ymm0, %ymm2
+; AVX512-NEXT:    vpmaxud %ymm1, %ymm0, %ymm0
+; AVX512-NEXT:    vpsubd %ymm2, %ymm0, %ymm0
 ; AVX512-NEXT:    retq
   %cmp = icmp ult <8 x i32> %a, %b
   %ab = sub <8 x i32> %a, %b
@@ -594,29 +565,29 @@ define <8 x i32> @abd_cmp_v8i32(<8 x i32> %a, <8 x i32> %b) nounwind {
 define <4 x i64> @abd_cmp_v4i64(<4 x i64> %a, <4 x i64> %b) nounwind {
 ; AVX1-LABEL: abd_cmp_v4i64:
 ; AVX1:       # %bb.0:
-; AVX1-NEXT:    vmovdqa {{.*#+}} xmm2 = [9223372036854775808,9223372036854775808]
-; AVX1-NEXT:    vpxor %xmm2, %xmm0, %xmm3
-; AVX1-NEXT:    vpxor %xmm2, %xmm1, %xmm4
-; AVX1-NEXT:    vpcmpgtq %xmm3, %xmm4, %xmm3
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm4
-; AVX1-NEXT:    vpxor %xmm2, %xmm4, %xmm5
-; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm6
-; AVX1-NEXT:    vpxor %xmm2, %xmm6, %xmm2
-; AVX1-NEXT:    vpcmpgtq %xmm5, %xmm2, %xmm2
-; AVX1-NEXT:    vpsubq %xmm1, %xmm0, %xmm5
-; AVX1-NEXT:    vpsubq %xmm6, %xmm4, %xmm7
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
+; AVX1-NEXT:    vmovdqa {{.*#+}} xmm3 = [9223372036854775808,9223372036854775808]
+; AVX1-NEXT:    vpxor %xmm3, %xmm2, %xmm4
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm5
+; AVX1-NEXT:    vpxor %xmm3, %xmm5, %xmm6
+; AVX1-NEXT:    vpcmpgtq %xmm4, %xmm6, %xmm4
+; AVX1-NEXT:    vpsubq %xmm2, %xmm5, %xmm6
+; AVX1-NEXT:    vpsubq %xmm5, %xmm2, %xmm2
+; AVX1-NEXT:    vblendvpd %xmm4, %xmm6, %xmm2, %xmm2
+; AVX1-NEXT:    vpxor %xmm3, %xmm1, %xmm4
+; AVX1-NEXT:    vpxor %xmm3, %xmm0, %xmm3
+; AVX1-NEXT:    vpcmpgtq %xmm4, %xmm3, %xmm3
+; AVX1-NEXT:    vpsubq %xmm1, %xmm0, %xmm4
 ; AVX1-NEXT:    vpsubq %xmm0, %xmm1, %xmm0
-; AVX1-NEXT:    vblendvpd %xmm3, %xmm5, %xmm0, %xmm0
-; AVX1-NEXT:    vpsubq %xmm4, %xmm6, %xmm1
-; AVX1-NEXT:    vblendvpd %xmm2, %xmm7, %xmm1, %xmm1
-; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
+; AVX1-NEXT:    vblendvpd %xmm3, %xmm4, %xmm0, %xmm0
+; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm0, %ymm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: abd_cmp_v4i64:
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    vpbroadcastq {{.*#+}} ymm2 = [9223372036854775808,9223372036854775808,9223372036854775808,9223372036854775808]
-; AVX2-NEXT:    vpxor %ymm2, %ymm0, %ymm3
-; AVX2-NEXT:    vpxor %ymm2, %ymm1, %ymm2
+; AVX2-NEXT:    vpxor %ymm2, %ymm1, %ymm3
+; AVX2-NEXT:    vpxor %ymm2, %ymm0, %ymm2
 ; AVX2-NEXT:    vpcmpgtq %ymm3, %ymm2, %ymm2
 ; AVX2-NEXT:    vpsubq %ymm1, %ymm0, %ymm3
 ; AVX2-NEXT:    vpsubq %ymm0, %ymm1, %ymm0
@@ -625,15 +596,14 @@ define <4 x i64> @abd_cmp_v4i64(<4 x i64> %a, <4 x i64> %b) nounwind {
 ;
 ; AVX512-LABEL: abd_cmp_v4i64:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpcmpnltuq %ymm1, %ymm0, %k1
-; AVX512-NEXT:    vpsubq %ymm1, %ymm0, %ymm2
-; AVX512-NEXT:    vpsubq %ymm0, %ymm1, %ymm2 {%k1}
-; AVX512-NEXT:    vmovdqa %ymm2, %ymm0
+; AVX512-NEXT:    vpminuq %ymm1, %ymm0, %ymm2
+; AVX512-NEXT:    vpmaxuq %ymm1, %ymm0, %ymm0
+; AVX512-NEXT:    vpsubq %ymm2, %ymm0, %ymm0
 ; AVX512-NEXT:    retq
   %cmp = icmp uge <4 x i64> %a, %b
   %ab = sub <4 x i64> %a, %b
   %ba = sub <4 x i64> %b, %a
-  %sel = select <4 x i1> %cmp, <4 x i64> %ba, <4 x i64> %ab
+  %sel = select <4 x i1> %cmp, <4 x i64> %ab, <4 x i64> %ba
   ret <4 x i64> %sel
 }
 
diff --git a/llvm/test/CodeGen/X86/abdu-vector-512.ll b/llvm/test/CodeGen/X86/abdu-vector-512.ll
index d10fc73c9946b..a855bea61e530 100644
--- a/llvm/test/CodeGen/X86/abdu-vector-512.ll
+++ b/llvm/test/CodeGen/X86/abdu-vector-512.ll
@@ -267,10 +267,9 @@ define <8 x i64> @abd_minmax_v8i64(<8 x i64> %a, <8 x i64> %b) nounwind {
 define <64 x i8> @abd_cmp_v64i8(<64 x i8> %a, <64 x i8> %b) nounwind {
 ; AVX512BW-LABEL: abd_cmp_v64i8:
 ; AVX512BW:       # %bb.0:
-; AVX512BW-NEXT:    vpcmpnleub %zmm1, %zmm0, %k1
-; AVX512BW-NEXT:    vpsubb %zmm0, %zmm1, %zmm2
-; AVX512BW-NEXT:    vpsubb %zmm1, %zmm0, %zmm2 {%k1}
-; AVX512BW-NEXT:    vmovdqa64 %zmm2, %zmm0
+; AVX512BW-NEXT:    vpminub %zmm1, %zmm0, %zmm2
+; AVX512BW-NEXT:    vpmaxub %zmm1, %zmm0, %zmm0
+; AVX512BW-NEXT:    vpsubb %zmm2, %zmm0, %zmm0
 ; AVX512BW-NEXT:    retq
 ;
 ; AVX512DQ-LABEL: abd_cmp_v64i8:
@@ -278,17 +277,12 @@ define <64 x i8> @abd_cmp_v64i8(<64 x i8> %a, <64 x i8> %b) nounwind {
 ; AVX512DQ-NEXT:    vextracti64x4 $1, %zmm1, %ymm2
 ; AVX512DQ-NEXT:    vextracti64x4 $1, %zmm0, %ymm3
 ; AVX512DQ-NEXT:    vpminub %ymm2, %ymm3, %ymm4
-; AVX512DQ-NEXT:    vpcmpeqb %ymm4, %ymm3, %ymm4
-; AVX512DQ-NEXT:    vpminub %ymm1, %ymm0, %ymm5
-; AVX512DQ-NEXT:    vpcmpeqb %ymm5, %ymm0, %ymm5
-; AVX512DQ-NEXT:    vinserti64x4 $1, %ymm4, %zmm5, %zmm4
-; AVX512DQ-NEXT:    vpsubb %ymm2, %ymm3, %ymm5
-; AVX512DQ-NEXT:    vpsubb %ymm1, %ymm0, %ymm6
-; AVX512DQ-NEXT:    vinserti64x4 $1, %ymm5, %zmm6, %zmm5
-; AVX512DQ-NEXT:    vpsubb %ymm3, %ymm2, %ymm2
-; AVX512DQ-NEXT:    vpsubb %ymm0, %ymm1, %ymm0
+; AVX512DQ-NEXT:    vpmaxub %ymm2, %ymm3, %ymm2
+; AVX512DQ-NEXT:    vpsubb %ymm4, %ymm2, %ymm2
+; AVX512DQ-NEXT:    vpminub %ymm1, %ymm0, %ymm3
+; AVX512DQ-NEXT:    vpmaxub %ymm1, %ymm0, %ymm0
+; AVX512DQ-NEXT:    vpsubb %ymm3, %ymm0, %ymm0
 ; AVX512DQ-NEXT:    vinserti64x4 $1, %ymm2, %zmm0, %zmm0
-; AVX512DQ-NEXT:    vpternlogq $226, %zmm5, %zmm4, %zmm0
 ; AVX512DQ-NEXT:    retq
   %cmp = icmp ugt <64 x i8> %a, %b
   %ab = sub <64 x i8> %a, %b
@@ -300,28 +294,22 @@ define <64 x i8> @abd_cmp_v64i8(<64 x i8> %a, <64 x i8> %b) nounwind {
 define <32 x i16> @abd_cmp_v32i16(<32 x i16> %a, <32 x i16> %b) nounwind {
 ; AVX512BW-LABEL: abd_cmp_v32i16:
 ; AVX512BW:       # %bb.0:
-; AVX512BW-NEXT:    vpcmpnltuw %zmm1, %zmm0, %k1
-; AVX512BW-NEXT:    vpsubw %zmm0, %zmm1, %zmm2
-; AVX512BW-NEXT:    vpsubw %zmm1, %zmm0, %zmm2 {%k1}
-; AVX512BW-NEXT:    vmovdqa64 %zmm2, %zmm0
+; AVX512BW-NEXT:    vpminuw %zmm1, %zmm0, %zmm2
+; AVX512BW-NEXT:    vpmaxuw %zmm1, %zmm0, %zmm0
+; AVX512BW-NEXT:    vpsubw %zmm2, %zmm0, %zmm0
 ; AVX512BW-NEXT:    retq
 ;
 ; AVX512DQ-LABEL: abd_cmp_v32i16:
 ; AVX512DQ:       # %bb.0:
 ; AVX512DQ-NEXT:    vextracti64x4 $1, %zmm1, %ymm2
 ; AVX512DQ-NEXT:    vextracti64x4 $1, %zmm0, %ymm3
-; AVX512DQ-NEXT:    vpmaxuw %ymm2, %ymm3, %ymm4
-; AVX512DQ-NEXT:    vpcmpeqw %ymm4, %ymm3, %ymm4
-; AVX512DQ-NEXT:    vpmaxuw %ymm1, %ymm0, %ymm5
-; AVX512DQ-NEXT:    vpcmpeqw %ymm5, %ymm0, %ymm5
-; AVX512DQ-NEXT:    vinserti64x4 $1, %ymm4, %zmm5, %zmm4
-; AVX512DQ-NEXT:    vpsubw %ymm2, %ymm3, %ymm5
-; AVX512DQ-NEXT:    vpsubw %ymm1, %ymm0, %ymm6
-; AVX512DQ-NEXT:    vinserti64x4 $1, %ymm5, %zmm6, %zmm5
-; AVX512DQ-NEXT:    vpsubw %ymm3, %ymm2, %ymm2
-; AVX512DQ-NEXT:    vpsubw %ymm0, %ymm1, %ymm0
+; AVX512DQ-NEXT:    vpminuw %ymm2, %ymm3, %ymm4
+; AVX512DQ-NEXT:    vpmaxuw %ymm2, %ymm3, %ymm2
+; AVX512DQ-NEXT:    vpsubw %ymm4, %ymm2, %ymm2
+; AVX512DQ-NEXT:    vpminuw %ymm1, %ymm0, %ymm3
+; AVX512DQ-NEXT:    vpmaxuw %ymm1, %ymm0, %ymm0
+; AVX512DQ-NEXT:    vpsubw %ymm3, %ymm0, %ymm0
 ; AVX512DQ-NEXT:    vinserti64x4 $1, %ymm2, %zmm0, %zmm0
-; AVX512DQ-NEXT:    vpternlogq $184, %zmm5, %zmm4, %zmm0
 ; AVX512DQ-NEXT:    retq
   %cmp = icmp uge <32 x i16> %a, %b
   %ab = sub <32 x i16> %a, %b
@@ -333,10 +321,9 @@ define <32 x i16> @abd_cmp_v32i16(<32 x i16> %a, <32 x i16> %b) nounwind {
 define <16 x i32> @abd_cmp_v16i32(<16 x i32> %a, <16 x i32> %b) nounwind {
 ; AVX512-LABEL: abd_cmp_v16i32:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpcmpltud %zmm1, %zmm0, %k1
-; AVX512-NEXT:    vpsubd %zmm1, %zmm0, %zmm2
-; AVX512-NEXT:    vpsubd %zmm0, %zmm1, %zmm2 {%k1}
-; AVX512-NEXT:    vmovdqa64 %zmm2, %zmm0
+; AVX512-NEXT:    vpminud %zmm1, %zmm0, %zmm2
+; AVX512-NEXT:    vpmaxud %zmm1, %zmm0, %zmm0
+; AVX512-NEXT:    vpsubd %zmm2, %zmm0, %zmm0
 ; AVX512-NEXT:    retq
   %cmp = icmp ult <16 x i32> %a, %b
   %ab = sub <16 x i32> %a, %b
@@ -348,15 +335,14 @@ define <16 x i32> @abd_cmp_v16i32(<16 x i32> %a, <16 x i32> %b) nounwind {
 define <8 x i64> @abd_cmp_v8i64(<8 x i64> %a, <8 x i64> %b) nounwind {
 ; AVX512-LABEL: abd_cmp_v8i64:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    vpcmpnltuq %zmm1, %zmm0, %k1
-; AVX512-NEXT:    vpsubq %zmm1, %zmm0, %zmm2
-; AVX512-NEXT:    vpsubq %zmm0, %zmm1, %zmm2 {%k1}
-; AVX512-NEXT:    vmovdqa64 %zmm2, %zmm0
+; AVX512-NEXT:    vpminuq %zmm1, %zmm0, %zmm2
+; AVX512-NEXT:    vpmaxuq %zmm1, %zmm0, %zmm0
+; AVX512-NEXT:    vpsubq %zmm2, %zmm0, %zmm0
 ; AVX512-NEXT:    retq
   %cmp = icmp uge <8 x i64> %a, %b
   %ab = sub <8 x i64> %a, %b
   %ba = sub <8 x i64> %b, %a
-  %sel = select <8 x i1> %cmp, <8 x i64> %ba, <8 x i64> %ab
+  %sel = select <8 x i1> %cmp, <8 x i64> %ab, <8 x i64> %ba
   ret <8 x i64> %sel
 }
 
diff --git a/llvm/test/CodeGen/X86/abdu.ll b/llvm/test/CodeGen/X86/abdu.ll
index 20ec1643e0b20..7095f7712082f 100644
--- a/llvm/test/CodeGen/X86/abdu.ll
+++ b/llvm/test/CodeGen/X86/abdu.ll
@@ -20,8 +20,6 @@ define i8 @abd_ext_i8(i8 %a, i8 %b) nounwind {
 ;
 ; X64-LABEL: abd_ext_i8:
 ; X64:       # %bb.0:
-; X64-NEXT:    # kill: def $esi killed $esi def $rsi
-; X64-NEXT:    # kill: def $edi killed $edi def $rdi
 ; X64-NEXT:    movzbl %dil, %ecx
 ; X64-NEXT:    movzbl %sil, %eax
 ; X64-NEXT:    subq %rax, %rcx
@@ -52,8 +50,6 @@ define i8 @abd_ext_i8_undef(i8 %a, i8 %b) nounwind {
 ;
 ; X64-LABEL: abd_ext_i8_undef:
 ; X64:       # %bb.0:
-; X64-NEXT:    # kill: def $esi killed $esi def $rsi
-; X64-NEXT:    # kill: def $edi killed $edi def $rdi
 ; X64-NEXT:    movzbl %dil, %ecx
 ; X64-NEXT:    movzbl %sil, %eax
 ; X64-NEXT:    subq %rax, %rcx
@@ -84,8 +80,6 @@ define i16 @abd_ext_i16(i16 %a, i16 %b) nounwind {
 ;
 ; X64-LABEL: abd_ext_i16:
 ; X64:       # %bb.0:
-; X64-NEXT:    # kill: def $esi killed $esi def $rsi
-; X64-NEXT:    # kill: def $edi killed $edi def $rdi
 ; X64-NEXT:    movzwl %di, %ecx
 ; X64-NEXT:    movzwl %si, %eax
 ; X64-NEXT:    subq %rax, %rcx
@@ -116,8 +110,6 @@ define i16 @abd_ext_i16_undef(i16 %a, i16 %b) nounwind {
 ;
 ; X64-LABEL: abd_ext_i16_undef:
 ; X64:       # %bb.0:
-; X64-NEXT:    # kill: def $esi killed $esi def $rsi
-; X64-NEXT:    # kill: def $edi killed $edi def $rdi
 ; X64-NEXT:    movzwl %di, %ecx
 ; X64-NEXT:    movzwl %si, %eax
 ; X64-NEXT:    subq %rax, %rcx
diff --git a/llvm/test/CodeGen/X86/add-and-not.ll b/llvm/test/CodeGen/X86/add-and-not.ll
index 99bfb94e689e2..d4dc732ce782c 100644
--- a/llvm/test/CodeGen/X86/add-and-not.ll
+++ b/llvm/test/CodeGen/X86/add-and-not.ll
@@ -1,15 +1,22 @@
 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc < %s -mtriple=x86_64-linux | FileCheck %s
+; RUN: llc < %s -mtriple=i686-linux | FileCheck %s --check-prefixes=X86
+; RUN: llc < %s -mtriple=x86_64-linux | FileCheck %s --check-prefixes=X64
 
 declare void @use(i8)
 
 define i8 @add_and_xor(i8 %x, i8 %y) {
-; CHECK-LABEL: add_and_xor:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movl %edi, %eax
-; CHECK-NEXT:    orl %esi, %eax
-; CHECK-NEXT:    # kill: def $al killed $al killed $eax
-; CHECK-NEXT:    retq
+; X86-LABEL: add_and_xor:
+; X86:       # %bb.0:
+; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    orb {{[0-9]+}}(%esp), %al
+; X86-NEXT:    retl
+;
+; X64-LABEL: add_and_xor:
+; X64:       # %bb.0:
+; X64-NEXT:    movl %edi, %eax
+; X64-NEXT:    orl %esi, %eax
+; X64-NEXT:    # kill: def $al killed $al killed $eax
+; X64-NEXT:    retq
   %xor = xor i8 %x, -1
   %and = and i8 %xor, %y
   %add = add i8 %and, %x
@@ -17,13 +24,22 @@ define i8 @add_and_xor(i8 %x, i8 %y) {
 }
 
 define i8 @add_and_xor_wrong_const(i8 %x, i8 %y) {
-; CHECK-LABEL: add_and_xor_wrong_const:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movl %edi, %eax
-; CHECK-NEXT:    xorb $-2, %al
-; CHECK-NEXT:    andb %sil, %al
-; CHECK-NEXT:    addb %dil, %al
-; CHECK-NEXT:    retq
+; X86-LABEL: add_and_xor_wrong_const:
+; X86:       # %bb.0:
+; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    movl %ecx, %eax
+; X86-NEXT:    xorb $-2, %al
+; X86-NEXT:    andb {{[0-9]+}}(%esp), %al
+; X86-NEXT:    addb %cl, %al
+; X86-NEXT:    retl
+;
+; X64-LABEL: add_and_xor_wrong_const:
+; X64:       # %bb.0:
+; X64-NEXT:    movl %edi, %eax
+; X64-NEXT:    xorb $-2, %al
+; X64-NEXT:    andb %sil, %al
+; X64-NEXT:    addb %dil, %al
+; X64-NEXT:    retq
   %xor = xor i8 %x, -2
   %and = and i8 %xor, %y
   %add = add i8 %and, %x
@@ -31,15 +47,23 @@ define i8 @add_and_xor_wrong_const(i8 %x, i8 %y) {
 }
 
 define i8 @add_and_xor_wrong_op(i8 %x, i8 %y, i8 %z) {
-; CHECK-LABEL: add_and_xor_wrong_op:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $edx killed $edx def $rdx
-; CHECK-NEXT:    # kill: def $edi killed $edi def $rdi
-; CHECK-NEXT:    notb %dl
-; CHECK-NEXT:    andb %sil, %dl
-; CHECK-NEXT:    leal (%rdx,%rdi), %eax
-; CHECK-NEXT:    # kill: def $al killed $al killed $eax
-; CHECK-NEXT:    retq
+; X86-LABEL: add_and_xor_wrong_op:
+; X86:       # %bb.0:
+; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    notb %al
+; X86-NEXT:    andb {{[0-9]+}}(%esp), %al
+; X86-NEXT:    addb {{[0-9]+}}(%esp), %al
+; X86-NEXT:    retl
+;
+; X64-LABEL: add_and_xor_wrong_op:
+; X64:       # %bb.0:
+; X64-NEXT:    # kill: def $edx killed $edx def $rdx
+; X64-NEXT:    # kill: def $edi killed $edi def $rdi
+; X64-NEXT:    notb %dl
+; X64-NEXT:    andb %sil, %dl
+; X64-NEXT:    leal (%rdx,%rdi), %eax
+; X64-NEXT:    # kill: def $al killed $al killed $eax
+; X64-NEXT:    retq
   %xor = xor i8 %z, -1
   %and = and i8 %xor, %y
   %add = add i8 %and, %x
@@ -47,12 +71,18 @@ define i8 @add_and_xor_wrong_op(i8 %x, i8 %y, i8 %z) {
 }
 
 define i8 @add_and_xor_commuted1(i8 %x, i8 %y) {
-; CHECK-LABEL: add_and_xor_commuted1:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movl %edi, %eax
-; CHECK-NEXT:    orl %esi, %eax
-; CHECK-NEXT:    # kill: def $al killed $al killed $eax
-; CHECK-NEXT:    retq
+; X86-LABEL: add_and_xor_commuted1:
+; X86:       # %bb.0:
+; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    orb {{[0-9]+}}(%esp), %al
+; X86-NEXT:    retl
+;
+; X64-LABEL: add_and_xor_commuted1:
+; X64:       # %bb.0:
+; X64-NEXT:    movl %edi, %eax
+; X64-NEXT:    orl %esi, %eax
+; X64-NEXT:    # kill: def $al killed $al killed $eax
+; X64-NEXT:    retq
   %xor = xor i8 %x, -1
   %and = and i8 %y, %xor
   %add = add i8 %and, %x
@@ -60,12 +90,18 @@ define i8 @add_and_xor_commuted1(i8 %x, i8 %y) {
 }
 
 define i8 @add_and_xor_commuted2(i8 %x, i8 %y) {
-; CHECK-LABEL: add_and_xor_commuted2:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movl %edi, %eax
-; CHECK-NEXT:    orl %esi, %eax
-; CHECK-NEXT:    # kill: def $al killed $al killed $eax
-; CHECK-NEXT:    retq
+; X86-LABEL: add_and_xor_commuted2:
+; X86:       # %bb.0:
+; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    orb {{[0-9]+}}(%esp), %al
+; X86-NEXT:    retl
+;
+; X64-LABEL: add_and_xor_commuted2:
+; X64:       # %bb.0:
+; X64-NEXT:    movl %edi, %eax
+; X64-NEXT:    orl %esi, %eax
+; X64-NEXT:    # kill: def $al killed $al killed $eax
+; X64-NEXT:    retq
   %xor = xor i8 %x, -1
   %and = and i8 %xor, %y
   %add = add i8 %x, %and
@@ -73,12 +109,18 @@ define i8 @add_and_xor_commuted2(i8 %x, i8 %y) {
 }
 
 define i8 @add_and_xor_commuted3(i8 %x, i8 %y) {
-; CHECK-LABEL: add_and_xor_commuted3:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movl %edi, %eax
-; CHECK-NEXT:    orl %esi, %eax
-; CHECK-NEXT:    # kill: def $al killed $al killed $eax
-; CHECK-NEXT:    retq
+; X86-LABEL: add_and_xor_commuted3:
+; X86:       # %bb.0:
+; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    orb {{[0-9]+}}(%esp), %al
+; X86-NEXT:    retl
+;
+; X64-LABEL: add_and_xor_commuted3:
+; X64:       # %bb.0:
+; X64-NEXT:    movl %edi, %eax
+; X64-NEXT:    orl %esi, %eax
+; X64-NEXT:    # kill: def $al killed $al killed $eax
+; X64-NEXT:    retq
   %xor = xor i8 %x, -1
   %and = and i8 %y, %xor
   %add = add i8 %x, %and
@@ -86,27 +128,47 @@ define i8 @add_and_xor_commuted3(i8 %x, i8 %y) {
 }
 
 define i8 @add_and_xor_extra_use(i8 %x, i8 %y) nounwind {
-; CHECK-LABEL: add_and_xor_extra_use:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    pushq %rbp
-; CHECK-NEXT:    pushq %r14
-; CHECK-NEXT:    pushq %rbx
-; CHECK-NEXT:    movl %esi, %ebx
-; CHECK-NEXT:    movl %edi, %ebp
-; CHECK-NEXT:    movl %ebp, %eax
-; CHECK-NEXT:    notb %al
-; CHECK-NEXT:    movzbl %al, %r14d
-; CHECK-NEXT:    movl %r14d, %edi
-; CHECK-NEXT:    callq use@PLT
-; CHECK-NEXT:    andb %bl, %r14b
-; CHECK-NEXT:    movzbl %r14b, %edi
-; CHECK-NEXT:    callq use@PLT
-; CHECK-NEXT:    orb %bpl, %bl
-; CHECK-NEXT:    movl %ebx, %eax
-; CHECK-NEXT:    popq %rbx
-; CHECK-NEXT:    popq %r14
-; CHECK-NEXT:    popq %rbp
-; CHECK-NEXT:    retq
+; X86-LABEL: add_and_xor_extra_use:
+; X86:       # %bb.0:
+; X86-NEXT:    pushl %ebx
+; X86-NEXT:    subl $8, %esp
+; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %ebx
+; X86-NEXT:    movb {{[0-9]+}}(%esp), %bh
+; X86-NEXT:    notb %bh
+; X86-NEXT:    movzbl %bh, %eax
+; X86-NEXT:    movl %eax, (%esp)
+; X86-NEXT:    calll use@PLT
+; X86-NEXT:    andb %bl, %bh
+; X86-NEXT:    movzbl %bh, %eax
+; X86-NEXT:    movl %eax, (%esp)
+; X86-NEXT:    calll use@PLT
+; X86-NEXT:    orb {{[0-9]+}}(%esp), %bl
+; X86-NEXT:    movl %ebx, %eax
+; X86-NEXT:    addl $8, %esp
+; X86-NEXT:    popl %ebx
+; X86-NEXT:    retl
+;
+; X64-LABEL: add_and_xor_extra_use:
+; X64:       # %bb.0:
+; X64-NEXT:    pushq %rbp
+; X64-NEXT:    pushq %r14
+; X64-NEXT:    pushq %rbx
+; X64-NEXT:    movl %esi, %ebx
+; X64-NEXT:    movl %edi, %ebp
+; X64-NEXT:    movl %ebp, %eax
+; X64-NEXT:    notb %al
+; X64-NEXT:    movzbl %al, %r14d
+; X64-NEXT:    movl %r14d, %edi
+; X64-NEXT:    callq use@PLT
+; X64-NEXT:    andb %bl, %r14b
+; X64-NEXT:    movzbl %r14b, %edi
+; X64-NEXT:    callq use@PLT
+; X64-NEXT:    orb %bpl, %bl
+; X64-NEXT:    movl %ebx, %eax
+; X64-NEXT:    popq %rbx
+; X64-NEXT:    popq %r14
+; X64-NEXT:    popq %rbp
+; X64-NEXT:    retq
   %xor = xor i8 %x, -1
   call void @use(i8 %xor)
   %and = and i8 %xor, %y
@@ -116,11 +178,18 @@ define i8 @add_and_xor_extra_use(i8 %x, i8 %y) nounwind {
 }
 
 define i64 @add_and_xor_const(i64 %x) {
-; CHECK-LABEL: add_and_xor_const:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movq %rdi, %rax
-; CHECK-NEXT:    orq $1, %rax
-; CHECK-NEXT:    retq
+; X86-LABEL: add_and_xor_const:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-NEXT:    orl $1, %eax
+; X86-NEXT:    retl
+;
+; X64-LABEL: add_and_xor_const:
+; X64:       # %bb.0:
+; X64-NEXT:    movq %rdi, %rax
+; X64-NEXT:    orq $1, %rax
+; X64-NEXT:    retq
   %xor = xor i64 %x, -1
   %and = and i64 %xor, 1
   %add = add i64 %and, %x
@@ -128,12 +197,22 @@ define i64 @add_and_xor_const(i64 %x) {
 }
 
 define i64 @add_and_xor_const_wrong_op(i64 %x, i64 %y) {
-; CHECK-LABEL: add_and_xor_const_wrong_op:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    notl %esi
-; CHECK-NEXT:    andl $1, %esi
-; CHECK-NEXT:    leaq (%rsi,%rdi), %rax
-; CHECK-NEXT:    retq
+; X86-LABEL: add_and_xor_const_wrong_op:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    notl %eax
+; X86-NEXT:    andl $1, %eax
+; X86-NEXT:    addl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    adcl $0, %edx
+; X86-NEXT:    retl
+;
+; X64-LABEL: add_and_xor_const_wrong_op:
+; X64:       # %bb.0:
+; X64-NEXT:    notl %esi
+; X64-NEXT:    andl $1, %esi
+; X64-NEXT:    leaq (%rsi,%rdi), %rax
+; X64-NEXT:    retq
   %xor = xor i64 %y, -1
   %and = and i64 %xor, 1
   %add = add i64 %and, %x
@@ -141,11 +220,18 @@ define i64 @add_and_xor_const_wrong_op(i64 %x, i64 %y) {
 }
 
 define i64 @add_and_xor_const_explicit_trunc(i64 %x) {
-; CHECK-LABEL: add_and_xor_const_explicit_trunc:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movq %rdi, %rax
-; CHECK-NEXT:    orq $1, %rax
-; CHECK-NEXT:    retq
+; X86-LABEL: add_and_xor_const_explicit_trunc:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-NEXT:    orl $1, %eax
+; X86-NEXT:    retl
+;
+; X64-LABEL: add_and_xor_const_explicit_trunc:
+; X64:       # %bb.0:
+; X64-NEXT:    movq %rdi, %rax
+; X64-NEXT:    orq $1, %rax
+; X64-NEXT:    retq
   %trunc = trunc i64 %x to i32
   %xor = xor i32 %trunc, -1
   %ext = sext i32 %xor to i64
@@ -155,15 +241,27 @@ define i64 @add_and_xor_const_explicit_trunc(i64 %x) {
 }
 
 define i64 @add_and_xor_const_explicit_trunc_wrong_mask(i64 %x) {
-; CHECK-LABEL: add_and_xor_const_explicit_trunc_wrong_mask:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movl %edi, %eax
-; CHECK-NEXT:    notl %eax
-; CHECK-NEXT:    movslq %eax, %rcx
-; CHECK-NEXT:    movabsq $4294967297, %rax # imm = 0x100000001
-; CHECK-NEXT:    andq %rcx, %rax
-; CHECK-NEXT:    addq %rdi, %rax
-; CHECK-NEXT:    retq
+; X86-LABEL: add_and_xor_const_explicit_trunc_wrong_mask:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    movl %ecx, %eax
+; X86-NEXT:    notl %eax
+; X86-NEXT:    movl %eax, %edx
+; X86-NEXT:    shrl $31, %edx
+; X86-NEXT:    andl $1, %eax
+; X86-NEXT:    addl %ecx, %eax
+; X86-NEXT:    adcl {{[0-9]+}}(%esp), %edx
+; X86-NEXT:    retl
+;
+; X64-LABEL: add_and_xor_const_explicit_trunc_wrong_mask:
+; X64:       # %bb.0:
+; X64-NEXT:    movl %edi, %eax
+; X64-NEXT:    notl %eax
+; X64-NEXT:    movslq %eax, %rcx
+; X64-NEXT:    movabsq $4294967297, %rax # imm = 0x100000001
+; X64-NEXT:    andq %rcx, %rax
+; X64-NEXT:    addq %rdi, %rax
+; X64-NEXT:    retq
   %trunc = trunc i64 %x to i32
   %xor = xor i32 %trunc, -1
   %ext = sext i32 %xor to i64
@@ -173,11 +271,17 @@ define i64 @add_and_xor_const_explicit_trunc_wrong_mask(i64 %x) {
 }
 
 define ptr @gep_and_xor(ptr %a, i64 %m) {
-; CHECK-LABEL: gep_and_xor:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movq %rdi, %rax
-; CHECK-NEXT:    orq %rsi, %rax
-; CHECK-NEXT:    retq
+; X86-LABEL: gep_and_xor:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    orl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    retl
+;
+; X64-LABEL: gep_and_xor:
+; X64:       # %bb.0:
+; X64-NEXT:    movq %rdi, %rax
+; X64-NEXT:    orq %rsi, %rax
+; X64-NEXT:    retq
   %old = ptrtoint ptr %a to i64
   %old.not = and i64 %old, %m
   %offset = xor i64 %old.not, %m
@@ -186,11 +290,17 @@ define ptr @gep_and_xor(ptr %a, i64 %m) {
 }
 
 define ptr @gep_and_xor_const(ptr %a) {
-; CHECK-LABEL: gep_and_xor_const:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movq %rdi, %rax
-; CHECK-NEXT:    orq $1, %rax
-; CHECK-NEXT:    retq
+; X86-LABEL: gep_and_xor_const:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    orl $1, %eax
+; X86-NEXT:    retl
+;
+; X64-LABEL: gep_and_xor_const:
+; X64:       # %bb.0:
+; X64-NEXT:    movq %rdi, %rax
+; X64-NEXT:    orq $1, %rax
+; X64-NEXT:    retq
   %old = ptrtoint ptr %a to i64
   %old.not = and i64 %old, 1
   %offset = xor i64 %old.not, 1
diff --git a/llvm/test/CodeGen/X86/addcarry.ll b/llvm/test/CodeGen/X86/addcarry.ll
index 95ee94551a7e5..0410b7efb08ff 100644
--- a/llvm/test/CodeGen/X86/addcarry.ll
+++ b/llvm/test/CodeGen/X86/addcarry.ll
@@ -683,7 +683,6 @@ define { i64, i1 } @addcarry_carry_not_zext(i64 %a, i64 %b, i64 %carryin) nounwi
 define { i64, i1 } @addcarry_carry_not_i1(i64 %a, i64 %b, i8 %carryin) nounwind {
 ; CHECK-LABEL: addcarry_carry_not_i1:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $edx killed $edx def $rdx
 ; CHECK-NEXT:    addq %rsi, %rdi
 ; CHECK-NEXT:    setb %cl
 ; CHECK-NEXT:    movzbl %dl, %eax
diff --git a/llvm/test/CodeGen/X86/and-shift.ll b/llvm/test/CodeGen/X86/and-shift.ll
new file mode 100644
index 0000000000000..e406a28c7f4db
--- /dev/null
+++ b/llvm/test/CodeGen/X86/and-shift.ll
@@ -0,0 +1,63 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=i386-unknown-unknown   | FileCheck %s --check-prefixes=X86
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown | FileCheck %s --check-prefixes=X64
+; RUN: llc < %s -mtriple=x86_64-unknown-gnux32 | FileCheck %s --check-prefixes=X64
+
+define i32 @shift30_and2_i32(i32 %x) {
+; X86-LABEL: shift30_and2_i32:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    shrl $30, %eax
+; X86-NEXT:    andl $-2, %eax
+; X86-NEXT:    retl
+;
+; X64-LABEL: shift30_and2_i32:
+; X64:       # %bb.0:
+; X64-NEXT:    movl %edi, %eax
+; X64-NEXT:    shrl $30, %eax
+; X64-NEXT:    andl $-2, %eax
+; X64-NEXT:    retq
+  %shr = lshr i32 %x, 30
+  %and = and i32 %shr, 2
+  ret i32 %and
+}
+
+define i64 @shift62_and2_i64(i64 %x) {
+; X86-LABEL: shift62_and2_i64:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    shrl $30, %eax
+; X86-NEXT:    andl $-2, %eax
+; X86-NEXT:    xorl %edx, %edx
+; X86-NEXT:    retl
+;
+; X64-LABEL: shift62_and2_i64:
+; X64:       # %bb.0:
+; X64-NEXT:    movq %rdi, %rax
+; X64-NEXT:    shrq $62, %rax
+; X64-NEXT:    andl $-2, %eax
+; X64-NEXT:    retq
+  %shr = lshr i64 %x, 62
+  %and = and i64 %shr, 2
+  ret i64 %and
+}
+
+define i64 @shift30_and2_i64(i64 %x) {
+; X86-LABEL: shift30_and2_i64:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    shrl $30, %eax
+; X86-NEXT:    andl $-2, %eax
+; X86-NEXT:    xorl %edx, %edx
+; X86-NEXT:    retl
+;
+; X64-LABEL: shift30_and2_i64:
+; X64:       # %bb.0:
+; X64-NEXT:    movq %rdi, %rax
+; X64-NEXT:    shrq $30, %rax
+; X64-NEXT:    andl $2, %eax
+; X64-NEXT:    retq
+  %shr = lshr i64 %x, 30
+  %and = and i64 %shr, 2
+  ret i64 %and
+}
diff --git a/llvm/test/CodeGen/X86/avx512-intrinsics-fast-isel.ll b/llvm/test/CodeGen/X86/avx512-intrinsics-fast-isel.ll
index 38b1524ab7498..231335591c0ac 100644
--- a/llvm/test/CodeGen/X86/avx512-intrinsics-fast-isel.ll
+++ b/llvm/test/CodeGen/X86/avx512-intrinsics-fast-isel.ll
@@ -998,13 +998,13 @@ define <16 x float> @test_mm512_mask_permute_ps(<16 x float> %a0, i16 %a1, <16 x
 ; X86:       # %bb.0:
 ; X86-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    kmovw %eax, %k1
-; X86-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} = zmm1[2,0,0,0,6,4,4,4,10,8,8,8,14,12,12,12]
+; X86-NEXT:    vshufps {{.*#+}} zmm0 {%k1} = zmm1[2,0,0,0,6,4,4,4,10,8,8,8,14,12,12,12]
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_mask_permute_ps:
 ; X64:       # %bb.0:
 ; X64-NEXT:    kmovw %edi, %k1
-; X64-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} = zmm1[2,0,0,0,6,4,4,4,10,8,8,8,14,12,12,12]
+; X64-NEXT:    vshufps {{.*#+}} zmm0 {%k1} = zmm1[2,0,0,0,6,4,4,4,10,8,8,8,14,12,12,12]
 ; X64-NEXT:    retq
   %arg1 = bitcast i16 %a1 to <16 x i1>
   %res0 = shufflevector <16 x float> %a2, <16 x float> undef, <16 x i32> <i32 2, i32 0, i32 0, i32 0, i32 6, i32 4, i32 4, i32 4, i32 10, i32 8, i32 8, i32 8, i32 14, i32 12, i32 12, i32 12>
@@ -1017,13 +1017,13 @@ define <16 x float> @test_mm512_maskz_permute_ps(i16 %a0, <16 x float> %a1) {
 ; X86:       # %bb.0:
 ; X86-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    kmovw %eax, %k1
-; X86-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = zmm0[2,0,0,0,6,4,4,4,10,8,8,8,14,12,12,12]
+; X86-NEXT:    vshufps {{.*#+}} zmm0 {%k1} {z} = zmm0[2,0,0,0,6,4,4,4,10,8,8,8,14,12,12,12]
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_maskz_permute_ps:
 ; X64:       # %bb.0:
 ; X64-NEXT:    kmovw %edi, %k1
-; X64-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = zmm0[2,0,0,0,6,4,4,4,10,8,8,8,14,12,12,12]
+; X64-NEXT:    vshufps {{.*#+}} zmm0 {%k1} {z} = zmm0[2,0,0,0,6,4,4,4,10,8,8,8,14,12,12,12]
 ; X64-NEXT:    retq
   %arg0 = bitcast i16 %a0 to <16 x i1>
   %res0 = shufflevector <16 x float> %a1, <16 x float> undef, <16 x i32> <i32 2, i32 0, i32 0, i32 0, i32 6, i32 4, i32 4, i32 4, i32 10, i32 8, i32 8, i32 8, i32 14, i32 12, i32 12, i32 12>
diff --git a/llvm/test/CodeGen/X86/avx512-intrinsics-upgrade.ll b/llvm/test/CodeGen/X86/avx512-intrinsics-upgrade.ll
index 86922d2ccf0fc..8989e2ebd8895 100644
--- a/llvm/test/CodeGen/X86/avx512-intrinsics-upgrade.ll
+++ b/llvm/test/CodeGen/X86/avx512-intrinsics-upgrade.ll
@@ -1033,7 +1033,7 @@ define <16 x float>@test_int_x86_avx512_mask_vpermil_ps_512(<16 x float> %x0, <1
 ; X86-LABEL: test_int_x86_avx512_mask_vpermil_ps_512:
 ; X86:       ## %bb.0:
 ; X86-NEXT:    kmovw {{[0-9]+}}(%esp), %k1 ## encoding: [0xc5,0xf8,0x90,0x4c,0x24,0x04]
-; X86-NEXT:    vpermilps $22, %zmm0, %zmm1 {%k1} ## encoding: [0x62,0xf3,0x7d,0x49,0x04,0xc8,0x16]
+; X86-NEXT:    vshufps $22, %zmm0, %zmm0, %zmm1 {%k1} ## encoding: [0x62,0xf1,0x7c,0x49,0xc6,0xc8,0x16]
 ; X86-NEXT:    ## zmm1 {%k1} = zmm0[2,1,1,0,6,5,5,4,10,9,9,8,14,13,13,12]
 ; X86-NEXT:    vmovaps %zmm1, %zmm0 ## encoding: [0x62,0xf1,0x7c,0x48,0x28,0xc1]
 ; X86-NEXT:    retl ## encoding: [0xc3]
@@ -1041,7 +1041,7 @@ define <16 x float>@test_int_x86_avx512_mask_vpermil_ps_512(<16 x float> %x0, <1
 ; X64-LABEL: test_int_x86_avx512_mask_vpermil_ps_512:
 ; X64:       ## %bb.0:
 ; X64-NEXT:    kmovw %edi, %k1 ## encoding: [0xc5,0xf8,0x92,0xcf]
-; X64-NEXT:    vpermilps $22, %zmm0, %zmm1 {%k1} ## encoding: [0x62,0xf3,0x7d,0x49,0x04,0xc8,0x16]
+; X64-NEXT:    vshufps $22, %zmm0, %zmm0, %zmm1 {%k1} ## encoding: [0x62,0xf1,0x7c,0x49,0xc6,0xc8,0x16]
 ; X64-NEXT:    ## zmm1 {%k1} = zmm0[2,1,1,0,6,5,5,4,10,9,9,8,14,13,13,12]
 ; X64-NEXT:    vmovaps %zmm1, %zmm0 ## encoding: [0x62,0xf1,0x7c,0x48,0x28,0xc1]
 ; X64-NEXT:    retq ## encoding: [0xc3]
@@ -1053,14 +1053,14 @@ define <16 x float>@test_int_x86_avx512_maskz_vpermil_ps_512(<16 x float> %x0, i
 ; X86-LABEL: test_int_x86_avx512_maskz_vpermil_ps_512:
 ; X86:       ## %bb.0:
 ; X86-NEXT:    kmovw {{[0-9]+}}(%esp), %k1 ## encoding: [0xc5,0xf8,0x90,0x4c,0x24,0x04]
-; X86-NEXT:    vpermilps $22, %zmm0, %zmm0 {%k1} {z} ## encoding: [0x62,0xf3,0x7d,0xc9,0x04,0xc0,0x16]
+; X86-NEXT:    vshufps $22, %zmm0, %zmm0, %zmm0 {%k1} {z} ## encoding: [0x62,0xf1,0x7c,0xc9,0xc6,0xc0,0x16]
 ; X86-NEXT:    ## zmm0 {%k1} {z} = zmm0[2,1,1,0,6,5,5,4,10,9,9,8,14,13,13,12]
 ; X86-NEXT:    retl ## encoding: [0xc3]
 ;
 ; X64-LABEL: test_int_x86_avx512_maskz_vpermil_ps_512:
 ; X64:       ## %bb.0:
 ; X64-NEXT:    kmovw %edi, %k1 ## encoding: [0xc5,0xf8,0x92,0xcf]
-; X64-NEXT:    vpermilps $22, %zmm0, %zmm0 {%k1} {z} ## encoding: [0x62,0xf3,0x7d,0xc9,0x04,0xc0,0x16]
+; X64-NEXT:    vshufps $22, %zmm0, %zmm0, %zmm0 {%k1} {z} ## encoding: [0x62,0xf1,0x7c,0xc9,0xc6,0xc0,0x16]
 ; X64-NEXT:    ## zmm0 {%k1} {z} = zmm0[2,1,1,0,6,5,5,4,10,9,9,8,14,13,13,12]
 ; X64-NEXT:    retq ## encoding: [0xc3]
   %res = call <16 x float> @llvm.x86.avx512.mask.vpermil.ps.512(<16 x float> %x0, i32 22, <16 x float> zeroinitializer, i16 %x3)
diff --git a/llvm/test/CodeGen/X86/avx512-shuffles/in_lane_permute.ll b/llvm/test/CodeGen/X86/avx512-shuffles/in_lane_permute.ll
index c57e2fb88e21b..56af39c3b90f4 100644
--- a/llvm/test/CodeGen/X86/avx512-shuffles/in_lane_permute.ll
+++ b/llvm/test/CodeGen/X86/avx512-shuffles/in_lane_permute.ll
@@ -16,7 +16,7 @@ define <4 x float> @test_masked_4xfloat_perm_mask0(<4 x float> %vec, <4 x float>
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm3, %xmm3, %xmm3
 ; CHECK-NEXT:    vcmpeqps %xmm3, %xmm2, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} xmm1 {%k1} = xmm0[2,1,3,1]
+; CHECK-NEXT:    vshufps {{.*#+}} xmm1 {%k1} = xmm0[2,1,3,1]
 ; CHECK-NEXT:    vmovaps %xmm1, %xmm0
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <4 x float> %vec, <4 x float> undef, <4 x i32> <i32 2, i32 1, i32 3, i32 1>
@@ -30,7 +30,7 @@ define <4 x float> @test_masked_z_4xfloat_perm_mask0(<4 x float> %vec, <4 x floa
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    vcmpeqps %xmm2, %xmm1, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} xmm0 {%k1} {z} = xmm0[2,1,3,1]
+; CHECK-NEXT:    vshufps {{.*#+}} xmm0 {%k1} {z} = xmm0[2,1,3,1]
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <4 x float> %vec, <4 x float> undef, <4 x i32> <i32 2, i32 1, i32 3, i32 1>
   %cmp = fcmp oeq <4 x float> %mask, zeroinitializer
@@ -42,7 +42,7 @@ define <4 x float> @test_masked_4xfloat_perm_mask1(<4 x float> %vec, <4 x float>
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm3, %xmm3, %xmm3
 ; CHECK-NEXT:    vcmpeqps %xmm3, %xmm2, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} xmm1 {%k1} = xmm0[1,2,3,2]
+; CHECK-NEXT:    vshufps {{.*#+}} xmm1 {%k1} = xmm0[1,2,3,2]
 ; CHECK-NEXT:    vmovaps %xmm1, %xmm0
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <4 x float> %vec, <4 x float> undef, <4 x i32> <i32 1, i32 2, i32 3, i32 2>
@@ -56,7 +56,7 @@ define <4 x float> @test_masked_z_4xfloat_perm_mask1(<4 x float> %vec, <4 x floa
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    vcmpeqps %xmm2, %xmm1, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} xmm0 {%k1} {z} = xmm0[1,2,3,2]
+; CHECK-NEXT:    vshufps {{.*#+}} xmm0 {%k1} {z} = xmm0[1,2,3,2]
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <4 x float> %vec, <4 x float> undef, <4 x i32> <i32 1, i32 2, i32 3, i32 2>
   %cmp = fcmp oeq <4 x float> %mask, zeroinitializer
@@ -68,7 +68,7 @@ define <4 x float> @test_masked_4xfloat_perm_mask2(<4 x float> %vec, <4 x float>
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm3, %xmm3, %xmm3
 ; CHECK-NEXT:    vcmpeqps %xmm3, %xmm2, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} xmm1 {%k1} = xmm0[1,3,2,1]
+; CHECK-NEXT:    vshufps {{.*#+}} xmm1 {%k1} = xmm0[1,3,2,1]
 ; CHECK-NEXT:    vmovaps %xmm1, %xmm0
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <4 x float> %vec, <4 x float> undef, <4 x i32> <i32 1, i32 3, i32 2, i32 1>
@@ -82,7 +82,7 @@ define <4 x float> @test_masked_z_4xfloat_perm_mask2(<4 x float> %vec, <4 x floa
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    vcmpeqps %xmm2, %xmm1, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} xmm0 {%k1} {z} = xmm0[1,3,2,1]
+; CHECK-NEXT:    vshufps {{.*#+}} xmm0 {%k1} {z} = xmm0[1,3,2,1]
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <4 x float> %vec, <4 x float> undef, <4 x i32> <i32 1, i32 3, i32 2, i32 1>
   %cmp = fcmp oeq <4 x float> %mask, zeroinitializer
@@ -102,7 +102,7 @@ define <4 x float> @test_masked_4xfloat_perm_mask3(<4 x float> %vec, <4 x float>
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm3, %xmm3, %xmm3
 ; CHECK-NEXT:    vcmpeqps %xmm3, %xmm2, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} xmm1 {%k1} = xmm0[1,2,3,2]
+; CHECK-NEXT:    vshufps {{.*#+}} xmm1 {%k1} = xmm0[1,2,3,2]
 ; CHECK-NEXT:    vmovaps %xmm1, %xmm0
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <4 x float> %vec, <4 x float> undef, <4 x i32> <i32 1, i32 2, i32 3, i32 2>
@@ -116,7 +116,7 @@ define <4 x float> @test_masked_z_4xfloat_perm_mask3(<4 x float> %vec, <4 x floa
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    vcmpeqps %xmm2, %xmm1, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} xmm0 {%k1} {z} = xmm0[1,2,3,2]
+; CHECK-NEXT:    vshufps {{.*#+}} xmm0 {%k1} {z} = xmm0[1,2,3,2]
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <4 x float> %vec, <4 x float> undef, <4 x i32> <i32 1, i32 2, i32 3, i32 2>
   %cmp = fcmp oeq <4 x float> %mask, zeroinitializer
@@ -292,7 +292,7 @@ define <8 x float> @test_masked_8xfloat_perm_imm_mask1(<8 x float> %vec, <8 x fl
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm3, %xmm3, %xmm3
 ; CHECK-NEXT:    vcmpeqps %ymm3, %ymm2, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} ymm1 {%k1} = ymm0[3,2,3,2,7,6,7,6]
+; CHECK-NEXT:    vshufps {{.*#+}} ymm1 {%k1} = ymm0[3,2,3,2,7,6,7,6]
 ; CHECK-NEXT:    vmovaps %ymm1, %ymm0
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <8 x float> %vec, <8 x float> undef, <8 x i32> <i32 3, i32 2, i32 3, i32 2, i32 7, i32 6, i32 7, i32 6>
@@ -306,7 +306,7 @@ define <8 x float> @test_masked_z_8xfloat_perm_imm_mask1(<8 x float> %vec, <8 x
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    vcmpeqps %ymm2, %ymm1, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} ymm0 {%k1} {z} = ymm0[3,2,3,2,7,6,7,6]
+; CHECK-NEXT:    vshufps {{.*#+}} ymm0 {%k1} {z} = ymm0[3,2,3,2,7,6,7,6]
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <8 x float> %vec, <8 x float> undef, <8 x i32> <i32 3, i32 2, i32 3, i32 2, i32 7, i32 6, i32 7, i32 6>
   %cmp = fcmp oeq <8 x float> %mask, zeroinitializer
@@ -352,7 +352,7 @@ define <8 x float> @test_masked_8xfloat_perm_imm_mask3(<8 x float> %vec, <8 x fl
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm3, %xmm3, %xmm3
 ; CHECK-NEXT:    vcmpeqps %ymm3, %ymm2, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} ymm1 {%k1} = ymm0[2,2,1,0,6,6,5,4]
+; CHECK-NEXT:    vshufps {{.*#+}} ymm1 {%k1} = ymm0[2,2,1,0,6,6,5,4]
 ; CHECK-NEXT:    vmovaps %ymm1, %ymm0
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <8 x float> %vec, <8 x float> undef, <8 x i32> <i32 2, i32 2, i32 1, i32 0, i32 6, i32 6, i32 5, i32 4>
@@ -366,7 +366,7 @@ define <8 x float> @test_masked_z_8xfloat_perm_imm_mask3(<8 x float> %vec, <8 x
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    vcmpeqps %ymm2, %ymm1, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} ymm0 {%k1} {z} = ymm0[2,2,1,0,6,6,5,4]
+; CHECK-NEXT:    vshufps {{.*#+}} ymm0 {%k1} {z} = ymm0[2,2,1,0,6,6,5,4]
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <8 x float> %vec, <8 x float> undef, <8 x i32> <i32 2, i32 2, i32 1, i32 0, i32 6, i32 6, i32 5, i32 4>
   %cmp = fcmp oeq <8 x float> %mask, zeroinitializer
@@ -404,7 +404,7 @@ define <8 x float> @test_masked_8xfloat_perm_imm_mask5(<8 x float> %vec, <8 x fl
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm3, %xmm3, %xmm3
 ; CHECK-NEXT:    vcmpeqps %ymm3, %ymm2, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} ymm1 {%k1} = ymm0[2,1,3,3,6,5,7,7]
+; CHECK-NEXT:    vshufps {{.*#+}} ymm1 {%k1} = ymm0[2,1,3,3,6,5,7,7]
 ; CHECK-NEXT:    vmovaps %ymm1, %ymm0
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <8 x float> %vec, <8 x float> undef, <8 x i32> <i32 2, i32 1, i32 3, i32 3, i32 6, i32 5, i32 7, i32 7>
@@ -418,7 +418,7 @@ define <8 x float> @test_masked_z_8xfloat_perm_imm_mask5(<8 x float> %vec, <8 x
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    vcmpeqps %ymm2, %ymm1, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} ymm0 {%k1} {z} = ymm0[2,1,3,3,6,5,7,7]
+; CHECK-NEXT:    vshufps {{.*#+}} ymm0 {%k1} {z} = ymm0[2,1,3,3,6,5,7,7]
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <8 x float> %vec, <8 x float> undef, <8 x i32> <i32 2, i32 1, i32 3, i32 3, i32 6, i32 5, i32 7, i32 7>
   %cmp = fcmp oeq <8 x float> %mask, zeroinitializer
@@ -464,7 +464,7 @@ define <8 x float> @test_masked_8xfloat_perm_imm_mask7(<8 x float> %vec, <8 x fl
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm3, %xmm3, %xmm3
 ; CHECK-NEXT:    vcmpeqps %ymm3, %ymm2, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} ymm1 {%k1} = ymm0[3,0,2,1,7,4,6,5]
+; CHECK-NEXT:    vshufps {{.*#+}} ymm1 {%k1} = ymm0[3,0,2,1,7,4,6,5]
 ; CHECK-NEXT:    vmovaps %ymm1, %ymm0
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <8 x float> %vec, <8 x float> undef, <8 x i32> <i32 3, i32 0, i32 2, i32 1, i32 7, i32 4, i32 6, i32 5>
@@ -478,7 +478,7 @@ define <8 x float> @test_masked_z_8xfloat_perm_imm_mask7(<8 x float> %vec, <8 x
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    vcmpeqps %ymm2, %ymm1, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} ymm0 {%k1} {z} = ymm0[3,0,2,1,7,4,6,5]
+; CHECK-NEXT:    vshufps {{.*#+}} ymm0 {%k1} {z} = ymm0[3,0,2,1,7,4,6,5]
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <8 x float> %vec, <8 x float> undef, <8 x i32> <i32 3, i32 0, i32 2, i32 1, i32 7, i32 4, i32 6, i32 5>
   %cmp = fcmp oeq <8 x float> %mask, zeroinitializer
@@ -785,7 +785,7 @@ define <16 x float> @test_masked_16xfloat_perm_imm_mask1(<16 x float> %vec, <16
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm3, %xmm3, %xmm3
 ; CHECK-NEXT:    vcmpeqps %zmm3, %zmm2, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} zmm1 {%k1} = zmm0[2,2,2,1,6,6,6,5,10,10,10,9,14,14,14,13]
+; CHECK-NEXT:    vshufps {{.*#+}} zmm1 {%k1} = zmm0[2,2,2,1,6,6,6,5,10,10,10,9,14,14,14,13]
 ; CHECK-NEXT:    vmovaps %zmm1, %zmm0
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <16 x float> %vec, <16 x float> undef, <16 x i32> <i32 2, i32 2, i32 2, i32 1, i32 6, i32 6, i32 6, i32 5, i32 10, i32 10, i32 10, i32 9, i32 14, i32 14, i32 14, i32 13>
@@ -799,7 +799,7 @@ define <16 x float> @test_masked_z_16xfloat_perm_imm_mask1(<16 x float> %vec, <1
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    vcmpeqps %zmm2, %zmm1, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = zmm0[2,2,2,1,6,6,6,5,10,10,10,9,14,14,14,13]
+; CHECK-NEXT:    vshufps {{.*#+}} zmm0 {%k1} {z} = zmm0[2,2,2,1,6,6,6,5,10,10,10,9,14,14,14,13]
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <16 x float> %vec, <16 x float> undef, <16 x i32> <i32 2, i32 2, i32 2, i32 1, i32 6, i32 6, i32 6, i32 5, i32 10, i32 10, i32 10, i32 9, i32 14, i32 14, i32 14, i32 13>
   %cmp = fcmp oeq <16 x float> %mask, zeroinitializer
@@ -845,7 +845,7 @@ define <16 x float> @test_masked_16xfloat_perm_imm_mask3(<16 x float> %vec, <16
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm3, %xmm3, %xmm3
 ; CHECK-NEXT:    vcmpeqps %zmm3, %zmm2, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} zmm1 {%k1} = zmm0[1,1,0,2,5,5,4,6,9,9,8,10,13,13,12,14]
+; CHECK-NEXT:    vshufps {{.*#+}} zmm1 {%k1} = zmm0[1,1,0,2,5,5,4,6,9,9,8,10,13,13,12,14]
 ; CHECK-NEXT:    vmovaps %zmm1, %zmm0
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <16 x float> %vec, <16 x float> undef, <16 x i32> <i32 1, i32 1, i32 0, i32 2, i32 5, i32 5, i32 4, i32 6, i32 9, i32 9, i32 8, i32 10, i32 13, i32 13, i32 12, i32 14>
@@ -859,7 +859,7 @@ define <16 x float> @test_masked_z_16xfloat_perm_imm_mask3(<16 x float> %vec, <1
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    vcmpeqps %zmm2, %zmm1, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = zmm0[1,1,0,2,5,5,4,6,9,9,8,10,13,13,12,14]
+; CHECK-NEXT:    vshufps {{.*#+}} zmm0 {%k1} {z} = zmm0[1,1,0,2,5,5,4,6,9,9,8,10,13,13,12,14]
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <16 x float> %vec, <16 x float> undef, <16 x i32> <i32 1, i32 1, i32 0, i32 2, i32 5, i32 5, i32 4, i32 6, i32 9, i32 9, i32 8, i32 10, i32 13, i32 13, i32 12, i32 14>
   %cmp = fcmp oeq <16 x float> %mask, zeroinitializer
@@ -897,7 +897,7 @@ define <16 x float> @test_masked_16xfloat_perm_imm_mask5(<16 x float> %vec, <16
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm3, %xmm3, %xmm3
 ; CHECK-NEXT:    vcmpeqps %zmm3, %zmm2, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} zmm1 {%k1} = zmm0[1,2,1,0,5,6,5,4,9,10,9,8,13,14,13,12]
+; CHECK-NEXT:    vshufps {{.*#+}} zmm1 {%k1} = zmm0[1,2,1,0,5,6,5,4,9,10,9,8,13,14,13,12]
 ; CHECK-NEXT:    vmovaps %zmm1, %zmm0
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <16 x float> %vec, <16 x float> undef, <16 x i32> <i32 1, i32 2, i32 1, i32 0, i32 5, i32 6, i32 5, i32 4, i32 9, i32 10, i32 9, i32 8, i32 13, i32 14, i32 13, i32 12>
@@ -911,7 +911,7 @@ define <16 x float> @test_masked_z_16xfloat_perm_imm_mask5(<16 x float> %vec, <1
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    vcmpeqps %zmm2, %zmm1, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = zmm0[1,2,1,0,5,6,5,4,9,10,9,8,13,14,13,12]
+; CHECK-NEXT:    vshufps {{.*#+}} zmm0 {%k1} {z} = zmm0[1,2,1,0,5,6,5,4,9,10,9,8,13,14,13,12]
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <16 x float> %vec, <16 x float> undef, <16 x i32> <i32 1, i32 2, i32 1, i32 0, i32 5, i32 6, i32 5, i32 4, i32 9, i32 10, i32 9, i32 8, i32 13, i32 14, i32 13, i32 12>
   %cmp = fcmp oeq <16 x float> %mask, zeroinitializer
@@ -957,7 +957,7 @@ define <16 x float> @test_masked_16xfloat_perm_imm_mask7(<16 x float> %vec, <16
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm3, %xmm3, %xmm3
 ; CHECK-NEXT:    vcmpeqps %zmm3, %zmm2, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} zmm1 {%k1} = zmm0[3,3,0,2,7,7,4,6,11,11,8,10,15,15,12,14]
+; CHECK-NEXT:    vshufps {{.*#+}} zmm1 {%k1} = zmm0[3,3,0,2,7,7,4,6,11,11,8,10,15,15,12,14]
 ; CHECK-NEXT:    vmovaps %zmm1, %zmm0
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <16 x float> %vec, <16 x float> undef, <16 x i32> <i32 3, i32 3, i32 0, i32 2, i32 7, i32 7, i32 4, i32 6, i32 11, i32 11, i32 8, i32 10, i32 15, i32 15, i32 12, i32 14>
@@ -971,7 +971,7 @@ define <16 x float> @test_masked_z_16xfloat_perm_imm_mask7(<16 x float> %vec, <1
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    vcmpeqps %zmm2, %zmm1, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = zmm0[3,3,0,2,7,7,4,6,11,11,8,10,15,15,12,14]
+; CHECK-NEXT:    vshufps {{.*#+}} zmm0 {%k1} {z} = zmm0[3,3,0,2,7,7,4,6,11,11,8,10,15,15,12,14]
 ; CHECK-NEXT:    retq
   %shuf = shufflevector <16 x float> %vec, <16 x float> undef, <16 x i32> <i32 3, i32 3, i32 0, i32 2, i32 7, i32 7, i32 4, i32 6, i32 11, i32 11, i32 8, i32 10, i32 15, i32 15, i32 12, i32 14>
   %cmp = fcmp oeq <16 x float> %mask, zeroinitializer
diff --git a/llvm/test/CodeGen/X86/avx512cd-intrinsics-upgrade.ll b/llvm/test/CodeGen/X86/avx512cd-intrinsics-upgrade.ll
index 5e076c1f38c48..ce5d2e7e8f348 100644
--- a/llvm/test/CodeGen/X86/avx512cd-intrinsics-upgrade.ll
+++ b/llvm/test/CodeGen/X86/avx512cd-intrinsics-upgrade.ll
@@ -181,7 +181,6 @@ define <8 x i64> @test_x86_broadcastmb_512(i8 %a0) {
 ;
 ; X64-LABEL: test_x86_broadcastmb_512:
 ; X64:       # %bb.0:
-; X64-NEXT:    # kill: def $edi killed $edi def $rdi
 ; X64-NEXT:    movzbl %dil, %eax
 ; X64-NEXT:    vpbroadcastq %rax, %zmm0
 ; X64-NEXT:    retq
diff --git a/llvm/test/CodeGen/X86/avx512cdvl-intrinsics-upgrade.ll b/llvm/test/CodeGen/X86/avx512cdvl-intrinsics-upgrade.ll
index ec07484bce4ad..6d1631bff3d85 100644
--- a/llvm/test/CodeGen/X86/avx512cdvl-intrinsics-upgrade.ll
+++ b/llvm/test/CodeGen/X86/avx512cdvl-intrinsics-upgrade.ll
@@ -183,7 +183,6 @@ define <4 x i64> @test_x86_broadcastmb_256(i8 %a0) {
 ;
 ; X64-LABEL: test_x86_broadcastmb_256:
 ; X64:       # %bb.0:
-; X64-NEXT:    # kill: def $edi killed $edi def $rdi
 ; X64-NEXT:    movzbl %dil, %eax
 ; X64-NEXT:    vpbroadcastq %rax, %ymm0
 ; X64-NEXT:    retq
@@ -202,7 +201,6 @@ define <2 x i64> @test_x86_broadcastmb_128(i8 %a0) {
 ;
 ; X64-LABEL: test_x86_broadcastmb_128:
 ; X64:       # %bb.0:
-; X64-NEXT:    # kill: def $edi killed $edi def $rdi
 ; X64-NEXT:    movzbl %dil, %eax
 ; X64-NEXT:    vpbroadcastq %rax, %xmm0
 ; X64-NEXT:    retq
diff --git a/llvm/test/CodeGen/X86/avx512fp16-mov.ll b/llvm/test/CodeGen/X86/avx512fp16-mov.ll
index bcaa4a733713e..1fd6f649de7b4 100644
--- a/llvm/test/CodeGen/X86/avx512fp16-mov.ll
+++ b/llvm/test/CodeGen/X86/avx512fp16-mov.ll
@@ -2130,9 +2130,8 @@ define <16 x i32> @pr52561(<16 x i32> %a, <16 x i32> %b) "min-legal-vector-width
 define <8 x i16> @pr59628_xmm(i16 %arg) {
 ; X64-LABEL: pr59628_xmm:
 ; X64:       # %bb.0:
-; X64-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; X64-NEXT:    vmovw %edi, %xmm0
 ; X64-NEXT:    vpbroadcastw %edi, %xmm1
-; X64-NEXT:    vmovsh %xmm1, %xmm0, %xmm0
 ; X64-NEXT:    vpcmpneqw {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm1, %k1
 ; X64-NEXT:    vmovdqu16 %xmm0, %xmm0 {%k1} {z}
 ; X64-NEXT:    retq
diff --git a/llvm/test/CodeGen/X86/avx512vl-intrinsics-upgrade.ll b/llvm/test/CodeGen/X86/avx512vl-intrinsics-upgrade.ll
index 7390a5a5259bd..ff217a592c27e 100644
--- a/llvm/test/CodeGen/X86/avx512vl-intrinsics-upgrade.ll
+++ b/llvm/test/CodeGen/X86/avx512vl-intrinsics-upgrade.ll
@@ -903,7 +903,7 @@ define <8 x float>@test_int_x86_avx512_mask_vpermil_ps_256(<8 x float> %x0, <8 x
 ; X86:       # %bb.0:
 ; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax # encoding: [0x0f,0xb6,0x44,0x24,0x04]
 ; X86-NEXT:    kmovw %eax, %k1 # encoding: [0xc5,0xf8,0x92,0xc8]
-; X86-NEXT:    vpermilps $22, %ymm0, %ymm1 {%k1} # encoding: [0x62,0xf3,0x7d,0x29,0x04,0xc8,0x16]
+; X86-NEXT:    vshufps $22, %ymm0, %ymm0, %ymm1 {%k1} # encoding: [0x62,0xf1,0x7c,0x29,0xc6,0xc8,0x16]
 ; X86-NEXT:    # ymm1 {%k1} = ymm0[2,1,1,0,6,5,5,4]
 ; X86-NEXT:    vmovaps %ymm1, %ymm0 # EVEX TO VEX Compression encoding: [0xc5,0xfc,0x28,0xc1]
 ; X86-NEXT:    retl # encoding: [0xc3]
@@ -911,7 +911,7 @@ define <8 x float>@test_int_x86_avx512_mask_vpermil_ps_256(<8 x float> %x0, <8 x
 ; X64-LABEL: test_int_x86_avx512_mask_vpermil_ps_256:
 ; X64:       # %bb.0:
 ; X64-NEXT:    kmovw %edi, %k1 # encoding: [0xc5,0xf8,0x92,0xcf]
-; X64-NEXT:    vpermilps $22, %ymm0, %ymm1 {%k1} # encoding: [0x62,0xf3,0x7d,0x29,0x04,0xc8,0x16]
+; X64-NEXT:    vshufps $22, %ymm0, %ymm0, %ymm1 {%k1} # encoding: [0x62,0xf1,0x7c,0x29,0xc6,0xc8,0x16]
 ; X64-NEXT:    # ymm1 {%k1} = ymm0[2,1,1,0,6,5,5,4]
 ; X64-NEXT:    vmovaps %ymm1, %ymm0 # EVEX TO VEX Compression encoding: [0xc5,0xfc,0x28,0xc1]
 ; X64-NEXT:    retq # encoding: [0xc3]
@@ -924,14 +924,14 @@ define <8 x float>@test_int_x86_avx512_maskz_vpermil_ps_256(<8 x float> %x0, i8
 ; X86:       # %bb.0:
 ; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax # encoding: [0x0f,0xb6,0x44,0x24,0x04]
 ; X86-NEXT:    kmovw %eax, %k1 # encoding: [0xc5,0xf8,0x92,0xc8]
-; X86-NEXT:    vpermilps $22, %ymm0, %ymm0 {%k1} {z} # encoding: [0x62,0xf3,0x7d,0xa9,0x04,0xc0,0x16]
+; X86-NEXT:    vshufps $22, %ymm0, %ymm0, %ymm0 {%k1} {z} # encoding: [0x62,0xf1,0x7c,0xa9,0xc6,0xc0,0x16]
 ; X86-NEXT:    # ymm0 {%k1} {z} = ymm0[2,1,1,0,6,5,5,4]
 ; X86-NEXT:    retl # encoding: [0xc3]
 ;
 ; X64-LABEL: test_int_x86_avx512_maskz_vpermil_ps_256:
 ; X64:       # %bb.0:
 ; X64-NEXT:    kmovw %edi, %k1 # encoding: [0xc5,0xf8,0x92,0xcf]
-; X64-NEXT:    vpermilps $22, %ymm0, %ymm0 {%k1} {z} # encoding: [0x62,0xf3,0x7d,0xa9,0x04,0xc0,0x16]
+; X64-NEXT:    vshufps $22, %ymm0, %ymm0, %ymm0 {%k1} {z} # encoding: [0x62,0xf1,0x7c,0xa9,0xc6,0xc0,0x16]
 ; X64-NEXT:    # ymm0 {%k1} {z} = ymm0[2,1,1,0,6,5,5,4]
 ; X64-NEXT:    retq # encoding: [0xc3]
   %res = call <8 x float> @llvm.x86.avx512.mask.vpermil.ps.256(<8 x float> %x0, i32 22, <8 x float> zeroinitializer, i8 %x3)
@@ -955,7 +955,7 @@ define <4 x float>@test_int_x86_avx512_mask_vpermil_ps_128(<4 x float> %x0, <4 x
 ; X86:       # %bb.0:
 ; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax # encoding: [0x0f,0xb6,0x44,0x24,0x04]
 ; X86-NEXT:    kmovw %eax, %k1 # encoding: [0xc5,0xf8,0x92,0xc8]
-; X86-NEXT:    vpermilps $22, %xmm0, %xmm1 {%k1} # encoding: [0x62,0xf3,0x7d,0x09,0x04,0xc8,0x16]
+; X86-NEXT:    vshufps $22, %xmm0, %xmm0, %xmm1 {%k1} # encoding: [0x62,0xf1,0x7c,0x09,0xc6,0xc8,0x16]
 ; X86-NEXT:    # xmm1 {%k1} = xmm0[2,1,1,0]
 ; X86-NEXT:    vmovaps %xmm1, %xmm0 # EVEX TO VEX Compression encoding: [0xc5,0xf8,0x28,0xc1]
 ; X86-NEXT:    retl # encoding: [0xc3]
@@ -963,7 +963,7 @@ define <4 x float>@test_int_x86_avx512_mask_vpermil_ps_128(<4 x float> %x0, <4 x
 ; X64-LABEL: test_int_x86_avx512_mask_vpermil_ps_128:
 ; X64:       # %bb.0:
 ; X64-NEXT:    kmovw %edi, %k1 # encoding: [0xc5,0xf8,0x92,0xcf]
-; X64-NEXT:    vpermilps $22, %xmm0, %xmm1 {%k1} # encoding: [0x62,0xf3,0x7d,0x09,0x04,0xc8,0x16]
+; X64-NEXT:    vshufps $22, %xmm0, %xmm0, %xmm1 {%k1} # encoding: [0x62,0xf1,0x7c,0x09,0xc6,0xc8,0x16]
 ; X64-NEXT:    # xmm1 {%k1} = xmm0[2,1,1,0]
 ; X64-NEXT:    vmovaps %xmm1, %xmm0 # EVEX TO VEX Compression encoding: [0xc5,0xf8,0x28,0xc1]
 ; X64-NEXT:    retq # encoding: [0xc3]
@@ -976,14 +976,14 @@ define <4 x float>@test_int_x86_avx512_maskz_vpermil_ps_128(<4 x float> %x0, i8
 ; X86:       # %bb.0:
 ; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax # encoding: [0x0f,0xb6,0x44,0x24,0x04]
 ; X86-NEXT:    kmovw %eax, %k1 # encoding: [0xc5,0xf8,0x92,0xc8]
-; X86-NEXT:    vpermilps $22, %xmm0, %xmm0 {%k1} {z} # encoding: [0x62,0xf3,0x7d,0x89,0x04,0xc0,0x16]
+; X86-NEXT:    vshufps $22, %xmm0, %xmm0, %xmm0 {%k1} {z} # encoding: [0x62,0xf1,0x7c,0x89,0xc6,0xc0,0x16]
 ; X86-NEXT:    # xmm0 {%k1} {z} = xmm0[2,1,1,0]
 ; X86-NEXT:    retl # encoding: [0xc3]
 ;
 ; X64-LABEL: test_int_x86_avx512_maskz_vpermil_ps_128:
 ; X64:       # %bb.0:
 ; X64-NEXT:    kmovw %edi, %k1 # encoding: [0xc5,0xf8,0x92,0xcf]
-; X64-NEXT:    vpermilps $22, %xmm0, %xmm0 {%k1} {z} # encoding: [0x62,0xf3,0x7d,0x89,0x04,0xc0,0x16]
+; X64-NEXT:    vshufps $22, %xmm0, %xmm0, %xmm0 {%k1} {z} # encoding: [0x62,0xf1,0x7c,0x89,0xc6,0xc0,0x16]
 ; X64-NEXT:    # xmm0 {%k1} {z} = xmm0[2,1,1,0]
 ; X64-NEXT:    retq # encoding: [0xc3]
   %res = call <4 x float> @llvm.x86.avx512.mask.vpermil.ps.128(<4 x float> %x0, i32 22, <4 x float> zeroinitializer, i8 %x3)
diff --git a/llvm/test/CodeGen/X86/bitcast-and-setcc-256.ll b/llvm/test/CodeGen/X86/bitcast-and-setcc-256.ll
index 225ebaceeae63..8699f17ecd43c 100644
--- a/llvm/test/CodeGen/X86/bitcast-and-setcc-256.ll
+++ b/llvm/test/CodeGen/X86/bitcast-and-setcc-256.ll
@@ -445,6 +445,188 @@ define i8 @v8i32_or_and(<8 x i32> %a, <8 x i32> %b, <8 x i32> %c, <8 x i32> %d,
   ret i8 %res
 }
 
+; PR61104 - peek through vselect allones operand
+; combineBitcastvxi1 may be called before the fold vselect(c,-1,x) --> or(c,x) vXi1 fold
+
+define i8 @v8i32_or_vselect(<8 x i32> %a0, <8 x i32> %a1, <8 x i32> %a2) {
+; SSE2-SSSE3-LABEL: v8i32_or_vselect:
+; SSE2-SSSE3:       # %bb.0:
+; SSE2-SSSE3-NEXT:    pcmpeqd %xmm3, %xmm1
+; SSE2-SSSE3-NEXT:    pcmpeqd %xmm2, %xmm0
+; SSE2-SSSE3-NEXT:    packssdw %xmm1, %xmm0
+; SSE2-SSSE3-NEXT:    packssdw %xmm5, %xmm4
+; SSE2-SSSE3-NEXT:    por %xmm0, %xmm4
+; SSE2-SSSE3-NEXT:    packsswb %xmm4, %xmm4
+; SSE2-SSSE3-NEXT:    pmovmskb %xmm4, %eax
+; SSE2-SSSE3-NEXT:    # kill: def $al killed $al killed $eax
+; SSE2-SSSE3-NEXT:    retq
+;
+; AVX1-LABEL: v8i32_or_vselect:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm3
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm4
+; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm4, %xmm3
+; AVX1-NEXT:    vpcmpeqd %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vinsertf128 $1, %xmm3, %ymm0, %ymm0
+; AVX1-NEXT:    vorps %ymm2, %ymm0, %ymm0
+; AVX1-NEXT:    vmovmskps %ymm0, %eax
+; AVX1-NEXT:    # kill: def $al killed $al killed $eax
+; AVX1-NEXT:    vzeroupper
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v8i32_or_vselect:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vpcmpeqd %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    vpor %ymm2, %ymm0, %ymm0
+; AVX2-NEXT:    vmovmskps %ymm0, %eax
+; AVX2-NEXT:    # kill: def $al killed $al killed $eax
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512F-LABEL: v8i32_or_vselect:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    vpcmpeqd %ymm1, %ymm0, %k0
+; AVX512F-NEXT:    vpxor %xmm0, %xmm0, %xmm0
+; AVX512F-NEXT:    vpcmpgtd %ymm2, %ymm0, %k1
+; AVX512F-NEXT:    korw %k1, %k0, %k0
+; AVX512F-NEXT:    kmovw %k0, %eax
+; AVX512F-NEXT:    # kill: def $al killed $al killed $eax
+; AVX512F-NEXT:    vzeroupper
+; AVX512F-NEXT:    retq
+;
+; AVX512BW-LABEL: v8i32_or_vselect:
+; AVX512BW:       # %bb.0:
+; AVX512BW-NEXT:    vpcmpeqd %ymm1, %ymm0, %k0
+; AVX512BW-NEXT:    vpxor %xmm0, %xmm0, %xmm0
+; AVX512BW-NEXT:    vpcmpgtd %ymm2, %ymm0, %k1
+; AVX512BW-NEXT:    korw %k1, %k0, %k0
+; AVX512BW-NEXT:    kmovd %k0, %eax
+; AVX512BW-NEXT:    # kill: def $al killed $al killed $eax
+; AVX512BW-NEXT:    vzeroupper
+; AVX512BW-NEXT:    retq
+  %cmp = icmp eq <8 x i32> %a0, %a1
+  %slt = icmp slt <8 x i32> %a2, zeroinitializer
+  %sel = select <8 x i1> %cmp, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i1> %slt
+  %res = bitcast <8 x i1> %sel to i8
+  ret i8 %res
+}
+
+define i8 @v8i32_or_select(<8 x i32> %a0, <8 x i32> %a1, <8 x i32> %a2, <8 x i32> %a3, i1 %a4) {
+; SSE2-SSSE3-LABEL: v8i32_or_select:
+; SSE2-SSSE3:       # %bb.0:
+; SSE2-SSSE3-NEXT:    pcmpeqd %xmm1, %xmm3
+; SSE2-SSSE3-NEXT:    pcmpeqd %xmm0, %xmm2
+; SSE2-SSSE3-NEXT:    packssdw %xmm3, %xmm2
+; SSE2-SSSE3-NEXT:    packssdw %xmm7, %xmm6
+; SSE2-SSSE3-NEXT:    testb $1, %dil
+; SSE2-SSSE3-NEXT:    jne .LBB7_1
+; SSE2-SSSE3-NEXT:  # %bb.2:
+; SSE2-SSSE3-NEXT:    pxor %xmm0, %xmm0
+; SSE2-SSSE3-NEXT:    jmp .LBB7_3
+; SSE2-SSSE3-NEXT:  .LBB7_1:
+; SSE2-SSSE3-NEXT:    pcmpeqd %xmm5, %xmm1
+; SSE2-SSSE3-NEXT:    pcmpeqd %xmm4, %xmm0
+; SSE2-SSSE3-NEXT:    packssdw %xmm1, %xmm0
+; SSE2-SSSE3-NEXT:  .LBB7_3:
+; SSE2-SSSE3-NEXT:    por %xmm2, %xmm6
+; SSE2-SSSE3-NEXT:    por %xmm0, %xmm6
+; SSE2-SSSE3-NEXT:    packsswb %xmm6, %xmm6
+; SSE2-SSSE3-NEXT:    pmovmskb %xmm6, %eax
+; SSE2-SSSE3-NEXT:    # kill: def $al killed $al killed $eax
+; SSE2-SSSE3-NEXT:    retq
+;
+; AVX1-LABEL: v8i32_or_select:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm4
+; AVX1-NEXT:    testb $1, %dil
+; AVX1-NEXT:    jne .LBB7_1
+; AVX1-NEXT:  # %bb.2:
+; AVX1-NEXT:    vxorps %xmm2, %xmm2, %xmm2
+; AVX1-NEXT:    jmp .LBB7_3
+; AVX1-NEXT:  .LBB7_1:
+; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm5
+; AVX1-NEXT:    vpcmpeqd %xmm5, %xmm4, %xmm5
+; AVX1-NEXT:    vpcmpeqd %xmm2, %xmm0, %xmm2
+; AVX1-NEXT:    vinsertf128 $1, %xmm5, %ymm2, %ymm2
+; AVX1-NEXT:  .LBB7_3:
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm5
+; AVX1-NEXT:    vpcmpeqd %xmm5, %xmm4, %xmm4
+; AVX1-NEXT:    vpcmpeqd %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vinsertf128 $1, %xmm4, %ymm0, %ymm0
+; AVX1-NEXT:    vorps %ymm2, %ymm3, %ymm1
+; AVX1-NEXT:    vorps %ymm0, %ymm1, %ymm0
+; AVX1-NEXT:    vmovmskps %ymm0, %eax
+; AVX1-NEXT:    # kill: def $al killed $al killed $eax
+; AVX1-NEXT:    vzeroupper
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v8i32_or_select:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    testb $1, %dil
+; AVX2-NEXT:    jne .LBB7_1
+; AVX2-NEXT:  # %bb.2:
+; AVX2-NEXT:    vpxor %xmm2, %xmm2, %xmm2
+; AVX2-NEXT:    jmp .LBB7_3
+; AVX2-NEXT:  .LBB7_1:
+; AVX2-NEXT:    vpcmpeqd %ymm2, %ymm0, %ymm2
+; AVX2-NEXT:  .LBB7_3:
+; AVX2-NEXT:    vpcmpeqd %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    vpor %ymm2, %ymm3, %ymm1
+; AVX2-NEXT:    vpor %ymm0, %ymm1, %ymm0
+; AVX2-NEXT:    vmovmskps %ymm0, %eax
+; AVX2-NEXT:    # kill: def $al killed $al killed $eax
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512F-LABEL: v8i32_or_select:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    vpcmpeqd %ymm1, %ymm0, %k0
+; AVX512F-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; AVX512F-NEXT:    vpcmpgtd %ymm3, %ymm1, %k1
+; AVX512F-NEXT:    testb $1, %dil
+; AVX512F-NEXT:    jne .LBB7_1
+; AVX512F-NEXT:  # %bb.2:
+; AVX512F-NEXT:    kxorw %k0, %k0, %k2
+; AVX512F-NEXT:    jmp .LBB7_3
+; AVX512F-NEXT:  .LBB7_1:
+; AVX512F-NEXT:    vpcmpeqd %ymm2, %ymm0, %k2
+; AVX512F-NEXT:  .LBB7_3:
+; AVX512F-NEXT:    korw %k0, %k1, %k0
+; AVX512F-NEXT:    korw %k2, %k0, %k0
+; AVX512F-NEXT:    kmovw %k0, %eax
+; AVX512F-NEXT:    # kill: def $al killed $al killed $eax
+; AVX512F-NEXT:    vzeroupper
+; AVX512F-NEXT:    retq
+;
+; AVX512BW-LABEL: v8i32_or_select:
+; AVX512BW:       # %bb.0:
+; AVX512BW-NEXT:    vpcmpeqd %ymm1, %ymm0, %k0
+; AVX512BW-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; AVX512BW-NEXT:    vpcmpgtd %ymm3, %ymm1, %k1
+; AVX512BW-NEXT:    testb $1, %dil
+; AVX512BW-NEXT:    jne .LBB7_1
+; AVX512BW-NEXT:  # %bb.2:
+; AVX512BW-NEXT:    kxorw %k0, %k0, %k2
+; AVX512BW-NEXT:    jmp .LBB7_3
+; AVX512BW-NEXT:  .LBB7_1:
+; AVX512BW-NEXT:    vpcmpeqd %ymm2, %ymm0, %k2
+; AVX512BW-NEXT:  .LBB7_3:
+; AVX512BW-NEXT:    korw %k0, %k1, %k0
+; AVX512BW-NEXT:    korw %k2, %k0, %k0
+; AVX512BW-NEXT:    kmovd %k0, %eax
+; AVX512BW-NEXT:    # kill: def $al killed $al killed $eax
+; AVX512BW-NEXT:    vzeroupper
+; AVX512BW-NEXT:    retq
+  %cmp0 = icmp eq <8 x i32> %a0, %a1
+  %cmp1 = icmp eq <8 x i32> %a0, %a2
+  %cmp2 = icmp slt <8 x i32> %a3, zeroinitializer
+  %sel = select i1 %a4, <8 x i1> %cmp1, <8 x i1> zeroinitializer
+  %or0 = or <8 x i1> %cmp2, %cmp0
+  %or1 = or <8 x i1> %or0, %sel
+  %res = bitcast <8 x i1> %or1 to i8
+  ret i8 %res
+}
+
 define i8 @v8f32_and(<8 x float> %a, <8 x float> %b, <8 x float> %c, <8 x float> %d) {
 ; SSE2-SSSE3-LABEL: v8f32_and:
 ; SSE2-SSSE3:       # %bb.0:
diff --git a/llvm/test/CodeGen/X86/clear_upper_vector_element_bits.ll b/llvm/test/CodeGen/X86/clear_upper_vector_element_bits.ll
index 45d74ea5b196c..ef21006ab64d3 100644
--- a/llvm/test/CodeGen/X86/clear_upper_vector_element_bits.ll
+++ b/llvm/test/CodeGen/X86/clear_upper_vector_element_bits.ll
@@ -542,7 +542,7 @@ define <2 x i64> @_clearupper2xi64b(<2 x i64>) nounwind {
 define <4 x i64> @_clearupper4xi64b(<4 x i64>) nounwind {
 ; SSE2-LABEL: _clearupper4xi64b:
 ; SSE2:       # %bb.0:
-; SSE2-NEXT:    movaps {{.*#+}} xmm2 = [NaN,0.0E+0,NaN,0.0E+0]
+; SSE2-NEXT:    movaps {{.*#+}} xmm2 = [4294967295,0,4294967295,0]
 ; SSE2-NEXT:    andps %xmm2, %xmm0
 ; SSE2-NEXT:    andps %xmm2, %xmm1
 ; SSE2-NEXT:    retq
@@ -689,165 +689,14 @@ define <16 x i16> @_clearupper16xi16b(<16 x i16>) nounwind {
 }
 
 define <16 x i8> @_clearupper16xi8b(<16 x i8>) nounwind {
-; SSE2-LABEL: _clearupper16xi8b:
-; SSE2:       # %bb.0:
-; SSE2-NEXT:    pushq %rbx
-; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm0[2,3,2,3]
-; SSE2-NEXT:    movq %xmm1, %rdx
-; SSE2-NEXT:    movq %rdx, %rax
-; SSE2-NEXT:    shrq $56, %rax
-; SSE2-NEXT:    andl $15, %eax
-; SSE2-NEXT:    movq %rdx, %rcx
-; SSE2-NEXT:    shrq $48, %rcx
-; SSE2-NEXT:    andl $15, %ecx
-; SSE2-NEXT:    movq %rdx, %rsi
-; SSE2-NEXT:    shrq $40, %rsi
-; SSE2-NEXT:    andl $15, %esi
-; SSE2-NEXT:    movq %rdx, %r8
-; SSE2-NEXT:    shrq $32, %r8
-; SSE2-NEXT:    andl $15, %r8d
-; SSE2-NEXT:    movq %xmm0, %r10
-; SSE2-NEXT:    movq %r10, %rdi
-; SSE2-NEXT:    shrq $56, %rdi
-; SSE2-NEXT:    andl $15, %edi
-; SSE2-NEXT:    movq %r10, %r9
-; SSE2-NEXT:    shrq $48, %r9
-; SSE2-NEXT:    andl $15, %r9d
-; SSE2-NEXT:    movq %r10, %r11
-; SSE2-NEXT:    shrq $40, %r11
-; SSE2-NEXT:    andl $15, %r11d
-; SSE2-NEXT:    movq %r10, %rbx
-; SSE2-NEXT:    shrq $32, %rbx
-; SSE2-NEXT:    andl $15, %ebx
-; SSE2-NEXT:    shlq $32, %rbx
-; SSE2-NEXT:    andl $252645135, %r10d # imm = 0xF0F0F0F
-; SSE2-NEXT:    orq %rbx, %r10
-; SSE2-NEXT:    shlq $40, %r11
-; SSE2-NEXT:    orq %r10, %r11
-; SSE2-NEXT:    shlq $48, %r9
-; SSE2-NEXT:    orq %r11, %r9
-; SSE2-NEXT:    shlq $56, %rdi
-; SSE2-NEXT:    orq %r9, %rdi
-; SSE2-NEXT:    shlq $32, %r8
-; SSE2-NEXT:    andl $252645135, %edx # imm = 0xF0F0F0F
-; SSE2-NEXT:    orq %r8, %rdx
-; SSE2-NEXT:    shlq $40, %rsi
-; SSE2-NEXT:    orq %rdx, %rsi
-; SSE2-NEXT:    shlq $48, %rcx
-; SSE2-NEXT:    orq %rsi, %rcx
-; SSE2-NEXT:    shlq $56, %rax
-; SSE2-NEXT:    orq %rcx, %rax
-; SSE2-NEXT:    movq %rdi, %xmm0
-; SSE2-NEXT:    movq %rax, %xmm1
-; SSE2-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm1[0]
-; SSE2-NEXT:    popq %rbx
-; SSE2-NEXT:    retq
-;
-; SSE42-LABEL: _clearupper16xi8b:
-; SSE42:       # %bb.0:
-; SSE42-NEXT:    pushq %rbx
-; SSE42-NEXT:    pextrq $1, %xmm0, %rdx
-; SSE42-NEXT:    movq %rdx, %rax
-; SSE42-NEXT:    shrq $56, %rax
-; SSE42-NEXT:    andl $15, %eax
-; SSE42-NEXT:    movq %rdx, %rcx
-; SSE42-NEXT:    shrq $48, %rcx
-; SSE42-NEXT:    andl $15, %ecx
-; SSE42-NEXT:    movq %rdx, %rsi
-; SSE42-NEXT:    shrq $40, %rsi
-; SSE42-NEXT:    andl $15, %esi
-; SSE42-NEXT:    movq %rdx, %r8
-; SSE42-NEXT:    shrq $32, %r8
-; SSE42-NEXT:    andl $15, %r8d
-; SSE42-NEXT:    movq %xmm0, %r10
-; SSE42-NEXT:    movq %r10, %rdi
-; SSE42-NEXT:    shrq $56, %rdi
-; SSE42-NEXT:    andl $15, %edi
-; SSE42-NEXT:    movq %r10, %r9
-; SSE42-NEXT:    shrq $48, %r9
-; SSE42-NEXT:    andl $15, %r9d
-; SSE42-NEXT:    movq %r10, %r11
-; SSE42-NEXT:    shrq $40, %r11
-; SSE42-NEXT:    andl $15, %r11d
-; SSE42-NEXT:    movq %r10, %rbx
-; SSE42-NEXT:    shrq $32, %rbx
-; SSE42-NEXT:    andl $15, %ebx
-; SSE42-NEXT:    shlq $32, %rbx
-; SSE42-NEXT:    andl $252645135, %r10d # imm = 0xF0F0F0F
-; SSE42-NEXT:    orq %rbx, %r10
-; SSE42-NEXT:    shlq $40, %r11
-; SSE42-NEXT:    orq %r10, %r11
-; SSE42-NEXT:    shlq $48, %r9
-; SSE42-NEXT:    orq %r11, %r9
-; SSE42-NEXT:    shlq $56, %rdi
-; SSE42-NEXT:    orq %r9, %rdi
-; SSE42-NEXT:    shlq $32, %r8
-; SSE42-NEXT:    andl $252645135, %edx # imm = 0xF0F0F0F
-; SSE42-NEXT:    orq %r8, %rdx
-; SSE42-NEXT:    shlq $40, %rsi
-; SSE42-NEXT:    orq %rdx, %rsi
-; SSE42-NEXT:    shlq $48, %rcx
-; SSE42-NEXT:    orq %rsi, %rcx
-; SSE42-NEXT:    shlq $56, %rax
-; SSE42-NEXT:    orq %rcx, %rax
-; SSE42-NEXT:    movq %rax, %xmm1
-; SSE42-NEXT:    movq %rdi, %xmm0
-; SSE42-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm1[0]
-; SSE42-NEXT:    popq %rbx
-; SSE42-NEXT:    retq
+; SSE-LABEL: _clearupper16xi8b:
+; SSE:       # %bb.0:
+; SSE-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
+; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: _clearupper16xi8b:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    pushq %rbx
-; AVX-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; AVX-NEXT:    movq -{{[0-9]+}}(%rsp), %rdi
-; AVX-NEXT:    movq %rcx, %rax
-; AVX-NEXT:    shrq $48, %rax
-; AVX-NEXT:    andl $15, %eax
-; AVX-NEXT:    movq %rcx, %rdx
-; AVX-NEXT:    shrq $40, %rdx
-; AVX-NEXT:    andl $15, %edx
-; AVX-NEXT:    movq %rcx, %rsi
-; AVX-NEXT:    shrq $32, %rsi
-; AVX-NEXT:    andl $15, %esi
-; AVX-NEXT:    movq %rdi, %r9
-; AVX-NEXT:    shrq $48, %r9
-; AVX-NEXT:    andl $15, %r9d
-; AVX-NEXT:    movq %rdi, %r10
-; AVX-NEXT:    shrq $40, %r10
-; AVX-NEXT:    andl $15, %r10d
-; AVX-NEXT:    movq %rdi, %r11
-; AVX-NEXT:    shrq $32, %r11
-; AVX-NEXT:    andl $15, %r11d
-; AVX-NEXT:    movq %rcx, %r8
-; AVX-NEXT:    shrq $56, %r8
-; AVX-NEXT:    andl $15, %r8d
-; AVX-NEXT:    movq %rdi, %rbx
-; AVX-NEXT:    shrq $56, %rbx
-; AVX-NEXT:    andl $15, %ebx
-; AVX-NEXT:    shlq $32, %r11
-; AVX-NEXT:    andl $252645135, %edi # imm = 0xF0F0F0F
-; AVX-NEXT:    orq %r11, %rdi
-; AVX-NEXT:    shlq $40, %r10
-; AVX-NEXT:    orq %rdi, %r10
-; AVX-NEXT:    shlq $48, %r9
-; AVX-NEXT:    orq %r10, %r9
-; AVX-NEXT:    shlq $56, %rbx
-; AVX-NEXT:    orq %r9, %rbx
-; AVX-NEXT:    movq %rbx, -{{[0-9]+}}(%rsp)
-; AVX-NEXT:    shlq $32, %rsi
-; AVX-NEXT:    andl $252645135, %ecx # imm = 0xF0F0F0F
-; AVX-NEXT:    orq %rsi, %rcx
-; AVX-NEXT:    shlq $40, %rdx
-; AVX-NEXT:    orq %rcx, %rdx
-; AVX-NEXT:    shlq $48, %rax
-; AVX-NEXT:    orq %rdx, %rax
-; AVX-NEXT:    shlq $56, %r8
-; AVX-NEXT:    orq %rax, %r8
-; AVX-NEXT:    movq %r8, -{{[0-9]+}}(%rsp)
-; AVX-NEXT:    vmovaps -{{[0-9]+}}(%rsp), %xmm0
-; AVX-NEXT:    popq %rbx
+; AVX-NEXT:    vandps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
 ; AVX-NEXT:    retq
   %x4  = bitcast <16 x i8> %0 to <32 x i4>
   %r0  = insertelement <32 x i4> %x4,  i4 zeroinitializer, i32 1
@@ -871,165 +720,14 @@ define <16 x i8> @_clearupper16xi8b(<16 x i8>) nounwind {
 }
 
 define <32 x i8> @_clearupper32xi8b(<32 x i8>) nounwind {
-; SSE2-LABEL: _clearupper32xi8b:
-; SSE2:       # %bb.0:
-; SSE2-NEXT:    pushq %rbx
-; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm0[2,3,2,3]
-; SSE2-NEXT:    movq %xmm2, %rdx
-; SSE2-NEXT:    movq %rdx, %rax
-; SSE2-NEXT:    shrq $56, %rax
-; SSE2-NEXT:    andl $15, %eax
-; SSE2-NEXT:    movq %rdx, %rcx
-; SSE2-NEXT:    shrq $48, %rcx
-; SSE2-NEXT:    andl $15, %ecx
-; SSE2-NEXT:    movq %rdx, %rsi
-; SSE2-NEXT:    shrq $40, %rsi
-; SSE2-NEXT:    andl $15, %esi
-; SSE2-NEXT:    movq %rdx, %r8
-; SSE2-NEXT:    shrq $32, %r8
-; SSE2-NEXT:    andl $15, %r8d
-; SSE2-NEXT:    movq %xmm0, %r10
-; SSE2-NEXT:    movq %r10, %rdi
-; SSE2-NEXT:    shrq $56, %rdi
-; SSE2-NEXT:    andl $15, %edi
-; SSE2-NEXT:    movq %r10, %r9
-; SSE2-NEXT:    shrq $48, %r9
-; SSE2-NEXT:    andl $15, %r9d
-; SSE2-NEXT:    movq %r10, %r11
-; SSE2-NEXT:    shrq $40, %r11
-; SSE2-NEXT:    andl $15, %r11d
-; SSE2-NEXT:    movq %r10, %rbx
-; SSE2-NEXT:    shrq $32, %rbx
-; SSE2-NEXT:    andl $15, %ebx
-; SSE2-NEXT:    shlq $32, %rbx
-; SSE2-NEXT:    andl $252645135, %r10d # imm = 0xF0F0F0F
-; SSE2-NEXT:    orq %rbx, %r10
-; SSE2-NEXT:    shlq $40, %r11
-; SSE2-NEXT:    orq %r10, %r11
-; SSE2-NEXT:    shlq $48, %r9
-; SSE2-NEXT:    orq %r11, %r9
-; SSE2-NEXT:    shlq $56, %rdi
-; SSE2-NEXT:    orq %r9, %rdi
-; SSE2-NEXT:    shlq $32, %r8
-; SSE2-NEXT:    andl $252645135, %edx # imm = 0xF0F0F0F
-; SSE2-NEXT:    orq %r8, %rdx
-; SSE2-NEXT:    shlq $40, %rsi
-; SSE2-NEXT:    orq %rdx, %rsi
-; SSE2-NEXT:    shlq $48, %rcx
-; SSE2-NEXT:    orq %rsi, %rcx
-; SSE2-NEXT:    shlq $56, %rax
-; SSE2-NEXT:    orq %rcx, %rax
-; SSE2-NEXT:    movq %rdi, %xmm0
-; SSE2-NEXT:    movq %rax, %xmm2
-; SSE2-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
-; SSE2-NEXT:    popq %rbx
-; SSE2-NEXT:    retq
-;
-; SSE42-LABEL: _clearupper32xi8b:
-; SSE42:       # %bb.0:
-; SSE42-NEXT:    pushq %rbx
-; SSE42-NEXT:    pextrq $1, %xmm0, %rdx
-; SSE42-NEXT:    movq %rdx, %rax
-; SSE42-NEXT:    shrq $56, %rax
-; SSE42-NEXT:    andl $15, %eax
-; SSE42-NEXT:    movq %rdx, %rcx
-; SSE42-NEXT:    shrq $48, %rcx
-; SSE42-NEXT:    andl $15, %ecx
-; SSE42-NEXT:    movq %rdx, %rsi
-; SSE42-NEXT:    shrq $40, %rsi
-; SSE42-NEXT:    andl $15, %esi
-; SSE42-NEXT:    movq %rdx, %r8
-; SSE42-NEXT:    shrq $32, %r8
-; SSE42-NEXT:    andl $15, %r8d
-; SSE42-NEXT:    movq %xmm0, %r10
-; SSE42-NEXT:    movq %r10, %rdi
-; SSE42-NEXT:    shrq $56, %rdi
-; SSE42-NEXT:    andl $15, %edi
-; SSE42-NEXT:    movq %r10, %r9
-; SSE42-NEXT:    shrq $48, %r9
-; SSE42-NEXT:    andl $15, %r9d
-; SSE42-NEXT:    movq %r10, %r11
-; SSE42-NEXT:    shrq $40, %r11
-; SSE42-NEXT:    andl $15, %r11d
-; SSE42-NEXT:    movq %r10, %rbx
-; SSE42-NEXT:    shrq $32, %rbx
-; SSE42-NEXT:    andl $15, %ebx
-; SSE42-NEXT:    shlq $32, %rbx
-; SSE42-NEXT:    andl $252645135, %r10d # imm = 0xF0F0F0F
-; SSE42-NEXT:    orq %rbx, %r10
-; SSE42-NEXT:    shlq $40, %r11
-; SSE42-NEXT:    orq %r10, %r11
-; SSE42-NEXT:    shlq $48, %r9
-; SSE42-NEXT:    orq %r11, %r9
-; SSE42-NEXT:    shlq $56, %rdi
-; SSE42-NEXT:    orq %r9, %rdi
-; SSE42-NEXT:    shlq $32, %r8
-; SSE42-NEXT:    andl $252645135, %edx # imm = 0xF0F0F0F
-; SSE42-NEXT:    orq %r8, %rdx
-; SSE42-NEXT:    shlq $40, %rsi
-; SSE42-NEXT:    orq %rdx, %rsi
-; SSE42-NEXT:    shlq $48, %rcx
-; SSE42-NEXT:    orq %rsi, %rcx
-; SSE42-NEXT:    shlq $56, %rax
-; SSE42-NEXT:    orq %rcx, %rax
-; SSE42-NEXT:    movq %rax, %xmm2
-; SSE42-NEXT:    movq %rdi, %xmm0
-; SSE42-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
-; SSE42-NEXT:    popq %rbx
-; SSE42-NEXT:    retq
+; SSE-LABEL: _clearupper32xi8b:
+; SSE:       # %bb.0:
+; SSE-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
+; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: _clearupper32xi8b:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    pushq %rbx
-; AVX-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; AVX-NEXT:    movq -{{[0-9]+}}(%rsp), %rdi
-; AVX-NEXT:    movq %rcx, %rax
-; AVX-NEXT:    shrq $48, %rax
-; AVX-NEXT:    andl $15, %eax
-; AVX-NEXT:    movq %rcx, %rdx
-; AVX-NEXT:    shrq $40, %rdx
-; AVX-NEXT:    andl $15, %edx
-; AVX-NEXT:    movq %rcx, %rsi
-; AVX-NEXT:    shrq $32, %rsi
-; AVX-NEXT:    andl $15, %esi
-; AVX-NEXT:    movq %rdi, %r9
-; AVX-NEXT:    shrq $48, %r9
-; AVX-NEXT:    andl $15, %r9d
-; AVX-NEXT:    movq %rdi, %r10
-; AVX-NEXT:    shrq $40, %r10
-; AVX-NEXT:    andl $15, %r10d
-; AVX-NEXT:    movq %rdi, %r11
-; AVX-NEXT:    shrq $32, %r11
-; AVX-NEXT:    andl $15, %r11d
-; AVX-NEXT:    movq %rcx, %r8
-; AVX-NEXT:    shrq $56, %r8
-; AVX-NEXT:    andl $15, %r8d
-; AVX-NEXT:    movq %rdi, %rbx
-; AVX-NEXT:    shrq $56, %rbx
-; AVX-NEXT:    andl $15, %ebx
-; AVX-NEXT:    shlq $32, %r11
-; AVX-NEXT:    andl $252645135, %edi # imm = 0xF0F0F0F
-; AVX-NEXT:    orq %r11, %rdi
-; AVX-NEXT:    shlq $40, %r10
-; AVX-NEXT:    orq %rdi, %r10
-; AVX-NEXT:    shlq $48, %r9
-; AVX-NEXT:    orq %r10, %r9
-; AVX-NEXT:    shlq $56, %rbx
-; AVX-NEXT:    orq %r9, %rbx
-; AVX-NEXT:    movq %rbx, -{{[0-9]+}}(%rsp)
-; AVX-NEXT:    shlq $32, %rsi
-; AVX-NEXT:    andl $252645135, %ecx # imm = 0xF0F0F0F
-; AVX-NEXT:    orq %rsi, %rcx
-; AVX-NEXT:    shlq $40, %rdx
-; AVX-NEXT:    orq %rcx, %rdx
-; AVX-NEXT:    shlq $48, %rax
-; AVX-NEXT:    orq %rdx, %rax
-; AVX-NEXT:    shlq $56, %r8
-; AVX-NEXT:    orq %rax, %r8
-; AVX-NEXT:    movq %r8, -{{[0-9]+}}(%rsp)
-; AVX-NEXT:    vinsertf128 $0, -{{[0-9]+}}(%rsp), %ymm0, %ymm0
-; AVX-NEXT:    popq %rbx
+; AVX-NEXT:    vandps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
 ; AVX-NEXT:    retq
   %x4  = bitcast <32 x i8> %0 to <64 x i4>
   %r0  = insertelement <64 x i4> %x4,  i4 zeroinitializer, i32 1
diff --git a/llvm/test/CodeGen/X86/cmp-concat.ll b/llvm/test/CodeGen/X86/cmp-concat.ll
index ada0d86dcd873..33834bc4470ae 100644
--- a/llvm/test/CodeGen/X86/cmp-concat.ll
+++ b/llvm/test/CodeGen/X86/cmp-concat.ll
@@ -33,8 +33,6 @@ define i1 @cmp_anybits_concat_i32(i32 %x, i32 %y) {
 define i1 @cmp_anybits_concat_shl_shl_i16(i16 %x, i16 %y) {
 ; CHECK-LABEL: cmp_anybits_concat_shl_shl_i16:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $esi killed $esi def $rsi
-; CHECK-NEXT:    # kill: def $edi killed $edi def $rdi
 ; CHECK-NEXT:    movzwl %di, %eax
 ; CHECK-NEXT:    movzwl %si, %ecx
 ; CHECK-NEXT:    shlq $8, %rcx
@@ -53,8 +51,6 @@ define i1 @cmp_anybits_concat_shl_shl_i16(i16 %x, i16 %y) {
 define i1 @cmp_anybits_concat_shl_shl_i16_commute(i16 %x, i16 %y) {
 ; CHECK-LABEL: cmp_anybits_concat_shl_shl_i16_commute:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $esi killed $esi def $rsi
-; CHECK-NEXT:    # kill: def $edi killed $edi def $rdi
 ; CHECK-NEXT:    movzwl %di, %eax
 ; CHECK-NEXT:    movzwl %si, %ecx
 ; CHECK-NEXT:    shlq $8, %rcx
diff --git a/llvm/test/CodeGen/X86/combine-add.ll b/llvm/test/CodeGen/X86/combine-add.ll
index f96968d5d50f3..72b987d452f5c 100644
--- a/llvm/test/CodeGen/X86/combine-add.ll
+++ b/llvm/test/CodeGen/X86/combine-add.ll
@@ -510,3 +510,53 @@ define i1 @PR51238(i1 %b, i8 %x, i8 %y, i8 %z) {
    %r = add i1 %cmpyz, true
    ret i1 %r
 }
+
+define <2 x i64> @add_vec_x_notx(<2 x i64> %v0) nounwind {
+; SSE-LABEL: add_vec_x_notx:
+; SSE:       # %bb.0:
+; SSE-NEXT:    pcmpeqd %xmm0, %xmm0
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: add_vec_x_notx:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
+; AVX-NEXT:    retq
+  %x = xor <2 x i64> %v0, <i64 -1, i64 -1>
+  %y = add <2 x i64> %v0, %x
+  ret <2 x i64> %y
+}
+
+define <2 x i64> @add_vec_notx_x(<2 x i64> %v0) nounwind {
+; SSE-LABEL: add_vec_notx_x:
+; SSE:       # %bb.0:
+; SSE-NEXT:    pcmpeqd %xmm0, %xmm0
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: add_vec_notx_x:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
+; AVX-NEXT:    retq
+  %x = xor <2 x i64> %v0, <i64 -1, i64 -1>
+  %y = add <2 x i64> %x, %v0
+  ret <2 x i64> %y
+}
+
+define i64 @add_x_notx(i64 %v0) nounwind {
+; CHECK-LABEL: add_x_notx:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movq $-1, %rax
+; CHECK-NEXT:    retq
+  %x = xor i64 %v0, -1
+  %y = add i64 %v0, %x
+  ret i64 %y
+}
+
+define i64 @add_notx_x(i64 %v0) nounwind {
+; CHECK-LABEL: add_notx_x:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movq $-1, %rax
+; CHECK-NEXT:    retq
+  %x = xor i64 %v0, -1
+  %y = add i64 %x, %v0
+  ret i64 %y
+}
diff --git a/llvm/test/CodeGen/X86/combine-and.ll b/llvm/test/CodeGen/X86/combine-and.ll
index 9eca86f201228..da52c627ef786 100644
--- a/llvm/test/CodeGen/X86/combine-and.ll
+++ b/llvm/test/CodeGen/X86/combine-and.ll
@@ -1176,3 +1176,53 @@ define <4 x i32> @neg_scalar_broadcast_two_uses(i32 %a0, <4 x i32> %a1, ptr %a2)
   %4 = and <4 x i32> %3, %a1
   ret <4 x i32> %4
 }
+
+define <2 x i64> @andnp_xx(<2 x i64> %v0) nounwind {
+; SSE-LABEL: andnp_xx:
+; SSE:       # %bb.0:
+; SSE-NEXT:    xorps %xmm0, %xmm0
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: andnp_xx:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AVX-NEXT:    retq
+  %x = xor <2 x i64> %v0, <i64 -1, i64 -1>
+  %y = and <2 x i64> %v0, %x
+  ret <2 x i64> %y
+}
+
+define <2 x i64> @andnp_xx_2(<2 x i64> %v0) nounwind {
+; SSE-LABEL: andnp_xx_2:
+; SSE:       # %bb.0:
+; SSE-NEXT:    xorps %xmm0, %xmm0
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: andnp_xx_2:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AVX-NEXT:    retq
+  %x = xor <2 x i64> %v0, <i64 -1, i64 -1>
+  %y = and <2 x i64> %x, %v0
+  ret <2 x i64> %y
+}
+
+define i64 @andn_xx(i64 %v0) nounwind {
+; CHECK-LABEL: andn_xx:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    retq
+  %x = xor i64 %v0, -1
+  %y = and i64 %v0, %x
+  ret i64 %y
+}
+
+define i64 @andn_xx_2(i64 %v0) nounwind {
+; CHECK-LABEL: andn_xx_2:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    retq
+  %x = xor i64 %v0, -1
+  %y = and i64 %x, %v0
+  ret i64 %y
+}
diff --git a/llvm/test/CodeGen/X86/dag-update-nodetomatch.ll b/llvm/test/CodeGen/X86/dag-update-nodetomatch.ll
index 9620882854b2c..ec2926b31b938 100644
--- a/llvm/test/CodeGen/X86/dag-update-nodetomatch.ll
+++ b/llvm/test/CodeGen/X86/dag-update-nodetomatch.ll
@@ -116,9 +116,8 @@ define void @_Z2x6v() local_unnamed_addr {
 ; CHECK-NEXT:    .cfi_offset %rbp, -16
 ; CHECK-NEXT:    movq x1@GOTPCREL(%rip), %rax
 ; CHECK-NEXT:    movl (%rax), %ebx
-; CHECK-NEXT:    movl %ebx, %r9d
-; CHECK-NEXT:    andl $511, %r9d # imm = 0x1FF
-; CHECK-NEXT:    leaq 1(%r9), %rax
+; CHECK-NEXT:    andl $511, %ebx # imm = 0x1FF
+; CHECK-NEXT:    leaq 1(%rbx), %rax
 ; CHECK-NEXT:    movq x4@GOTPCREL(%rip), %rcx
 ; CHECK-NEXT:    movl %eax, (%rcx)
 ; CHECK-NEXT:    movq x3@GOTPCREL(%rip), %rcx
@@ -135,12 +134,11 @@ define void @_Z2x6v() local_unnamed_addr {
 ; CHECK-NEXT:    addq %rsi, %rdi
 ; CHECK-NEXT:    movq x2@GOTPCREL(%rip), %r8
 ; CHECK-NEXT:    movl (%r8), %edx
-; CHECK-NEXT:    leal 8(,%r9,8), %eax
+; CHECK-NEXT:    leal 8(,%rbx,8), %eax
 ; CHECK-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; CHECK-NEXT:    leaq 8(%rsi), %rax
 ; CHECK-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; CHECK-NEXT:    leaq 32(%rsi), %r11
-; CHECK-NEXT:    andl $511, %ebx # imm = 0x1FF
 ; CHECK-NEXT:    leaq 8(,%rbx,8), %rbx
 ; CHECK-NEXT:    xorl %r14d, %r14d
 ; CHECK-NEXT:    movq x0@GOTPCREL(%rip), %r15
diff --git a/llvm/test/CodeGen/X86/fastcc-byval.ll b/llvm/test/CodeGen/X86/fastcc-byval.ll
index aee07caf4efb5..920291a73ecd6 100644
--- a/llvm/test/CodeGen/X86/fastcc-byval.ll
+++ b/llvm/test/CodeGen/X86/fastcc-byval.ll
@@ -16,8 +16,7 @@ define fastcc i32 @bar() nounwind {
 ; CHECK:       ## %bb.0:
 ; CHECK-NEXT:    subl $12, %esp
 ; CHECK-NEXT:    movl $1, 8(%esp)
-; CHECK-NEXT:    movl 8(%esp), %eax
-; CHECK-NEXT:    movl %eax, (%esp)
+; CHECK-NEXT:    movl $1, (%esp)
 ; CHECK-NEXT:    calll _foo
 ; CHECK-NEXT:    movl 8(%esp), %eax
 ; CHECK-NEXT:    addl $12, %esp
diff --git a/llvm/test/CodeGen/X86/fold-masked-merge.ll b/llvm/test/CodeGen/X86/fold-masked-merge.ll
index d2c00df281610..135494ac25f8c 100644
--- a/llvm/test/CodeGen/X86/fold-masked-merge.ll
+++ b/llvm/test/CodeGen/X86/fold-masked-merge.ll
@@ -187,21 +187,11 @@ define i32 @not_a_masked_merge3(i32 %a0, i32 %a1, i32 %a2) {
 
 ; not a masked merge: `not` operand must not be on same `and`.
 define i32 @not_a_masked_merge4(i32 %a0, i32 %a1, i32 %a2) {
-; NOBMI-LABEL: not_a_masked_merge4:
-; NOBMI:       # %bb.0:
-; NOBMI-NEXT:    andl %esi, %edi
-; NOBMI-NEXT:    movl %edx, %eax
-; NOBMI-NEXT:    notl %eax
-; NOBMI-NEXT:    andl %edx, %eax
-; NOBMI-NEXT:    orl %edi, %eax
-; NOBMI-NEXT:    retq
-;
-; BMI-LABEL: not_a_masked_merge4:
-; BMI:       # %bb.0:
-; BMI-NEXT:    andl %esi, %edi
-; BMI-NEXT:    andnl %edx, %edx, %eax
-; BMI-NEXT:    orl %edi, %eax
-; BMI-NEXT:    retq
+; CHECK-LABEL: not_a_masked_merge4:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    andl %esi, %eax
+; CHECK-NEXT:    retq
   %and0 = and i32 %a0, %a1
   %not = xor i32 %a2, -1
   %and1 = and i32 %not, %a2
diff --git a/llvm/test/CodeGen/X86/fsafdo_test1.ll b/llvm/test/CodeGen/X86/fsafdo_test1.ll
index dae098dbd6305..b5ae3915294cd 100644
--- a/llvm/test/CodeGen/X86/fsafdo_test1.ll
+++ b/llvm/test/CodeGen/X86/fsafdo_test1.ll
@@ -1,17 +1,20 @@
-; RUN: llc -enable-fs-discriminator < %s | FileCheck %s
+; RUN: llc -enable-fs-discriminator -improved-fs-discriminator=false < %s | FileCheck %s --check-prefixes=V0,V01
+; RUN: llc -enable-fs-discriminator -improved-fs-discriminator=true < %s | FileCheck %s --check-prefixes=V1,V01
 ;
 ; Check that fs-afdo discriminators are generated.
-; CHECK: .loc    1 7 3 is_stmt 0 discriminator 2 # foo.c:7:3
-; ChECK: .loc    1 9 5 is_stmt 1 discriminator 2 # foo.c:9:5
-; CHECK: .loc    1 9 5 is_stmt 0 discriminator 11266 # foo.c:9:5
-; CHECK: .loc    1 7 3 is_stmt 1 discriminator 11266 # foo.c:7:3
+; V01: .loc    1 7 3 is_stmt 0 discriminator 2 # foo.c:7:3
+; V01: .loc    1 9 5 is_stmt 1 discriminator 2 # foo.c:9:5
+; V0: .loc    1 9 5 is_stmt 0 discriminator 11266 # foo.c:9:5
+; V0: .loc    1 7 3 is_stmt 1 discriminator 11266 # foo.c:7:3
+; V1: .loc    1 9 5 is_stmt 0 discriminator 258 # foo.c:9:5
+; V1: .loc    1 7 3 is_stmt 1 discriminator 258 # foo.c:7:3
 ; Check that variable __llvm_fs_discriminator__ is generated.
-; CHECK: .type   __llvm_fs_discriminator__,@object # @__llvm_fs_discriminator__
-; CHECK: .section        .rodata,"a",@progbits
-; CHECK: .weak   __llvm_fs_discriminator__
-; CHECK: __llvm_fs_discriminator__:
-; CHECK: .byte   1
-; CHECK: .size   __llvm_fs_discriminator__, 1
+; V01: .type   __llvm_fs_discriminator__,@object # @__llvm_fs_discriminator__
+; V01: .section        .rodata,"a",@progbits
+; V01: .weak   __llvm_fs_discriminator__
+; V01: __llvm_fs_discriminator__:
+; V01: .byte   1
+; V01: .size   __llvm_fs_discriminator__, 1
 
 target triple = "x86_64-unknown-linux-gnu"
 
diff --git a/llvm/test/CodeGen/X86/fsafdo_test2.ll b/llvm/test/CodeGen/X86/fsafdo_test2.ll
index c0ae5d2f36768..c473e0ccacbfc 100644
--- a/llvm/test/CodeGen/X86/fsafdo_test2.ll
+++ b/llvm/test/CodeGen/X86/fsafdo_test2.ll
@@ -1,7 +1,10 @@
 ; REQUIRES: asserts
-; RUN: llc -enable-fs-discriminator < %s | FileCheck %s
-; RUN: llvm-profdata merge --sample -profile-isfs -o %t.afdo %S/Inputs/fsloader.afdo
-; RUN: llc -enable-fs-discriminator -fs-profile-file=%t.afdo -show-fs-branchprob -disable-ra-fsprofile-loader=false -disable-layout-fsprofile-loader=false < %s 2>&1 | FileCheck %s --check-prefix=LOADER
+; RUN: llc -enable-fs-discriminator -improved-fs-discriminator=false < %s | FileCheck %s --check-prefixes=V0,V01
+; RUN: llvm-profdata merge --sample -profile-isfs -o %t0.afdo %S/Inputs/fsloader.afdo
+; RUN: llc -enable-fs-discriminator -improved-fs-discriminator=false -fs-profile-file=%t0.afdo -show-fs-branchprob -disable-ra-fsprofile-loader=false -disable-layout-fsprofile-loader=false < %s 2>&1 | FileCheck %s --check-prefixes=LOADERV0,LOADER
+; RUN: llc -enable-fs-discriminator -improved-fs-discriminator=true < %s | FileCheck %s --check-prefixes=V1,V01
+; RUN: llvm-profdata merge --sample -profile-isfs -o %t1.afdo %S/Inputs/fsloader_v1.afdo
+; RUN: llc -enable-fs-discriminator -improved-fs-discriminator=true -fs-profile-file=%t1.afdo -show-fs-branchprob -disable-ra-fsprofile-loader=false -disable-layout-fsprofile-loader=false < %s 2>&1 | FileCheck %s --check-prefixes=LOADERV1,LOADER
 ;
 ;;
 ;; C source code for the test (compiler at -O3):
@@ -41,18 +44,21 @@
 ;; }
 ;;
 ;; Check that fs-afdo discriminators are generated.
-; CHECK: .loc    1 23 9 is_stmt 0 discriminator 1 # unroll.c:23:9
-; CHECK: .loc    1 23 9 is_stmt 0 discriminator 3585 # unroll.c:23:9
-; CHECK: .loc    1 23 9 is_stmt 0 discriminator 8705 # unroll.c:23:9
-; CHECK: .loc    1 23 9 is_stmt 0 discriminator 4097 # unroll.c:23:9
+; V01: .loc    1 23 9 is_stmt 0 discriminator 1 # unroll.c:23:9
+; V0: .loc    1 23 9 is_stmt 0 discriminator 3585 # unroll.c:23:9
+; V0: .loc    1 23 9 is_stmt 0 discriminator 8705 # unroll.c:23:9
+; V0: .loc    1 23 9 is_stmt 0 discriminator 4097 # unroll.c:23:9
+; V1: .loc    1 23 9 is_stmt 0 discriminator 257 # unroll.c:23:9
+; V1: .loc    1 23 9 is_stmt 0 discriminator 513 # unroll.c:23:9
+; V1: .loc    1 23 9 is_stmt 0 discriminator 769 # unroll.c:23:9
 ;;
 ;; Check that variable __llvm_fs_discriminator__ is generated.
-; CHECK: .type   __llvm_fs_discriminator__,@object # @__llvm_fs_discriminator__
-; CHECK: .section        .rodata,"a",@progbits
-; CHECK: .weak   __llvm_fs_discriminator__
-; CHECK: __llvm_fs_discriminator__:
-; CHECK: .byte   1
-; CHECK: .size   __llvm_fs_discriminator__, 1
+; V01: .type   __llvm_fs_discriminator__,@object # @__llvm_fs_discriminator__
+; V01: .section        .rodata,"a",@progbits
+; V01: .weak   __llvm_fs_discriminator__
+; V01: __llvm_fs_discriminator__:
+; V01: .byte   1
+; V01: .size   __llvm_fs_discriminator__, 1
 
 ;; Check that new branch probs are generated.
 ; LOADER: Set branch fs prob: MBB (1 -> 3): unroll.c:22:11-->unroll.c:24:11 W=283590  0x40000000 / 0x80000000 = 50.00% --> 0x7aca7894 / 0x80000000 = 95.93%
@@ -63,16 +69,19 @@
 ; LOADER: Set branch fs prob: MBB (5 -> 7): unroll.c:22:11 W=283590  0x40000000 / 0x80000000 = 50.00% --> 0x7de3eed2 / 0x80000000 = 98.35%
 ; LOADER: Set branch fs prob: MBB (8 -> 10): unroll.c:24:11-->unroll.c:22:11 W=283590  0x30000000 / 0x80000000 = 37.50% --> 0x00000000 / 0x80000000 = 0.00%
 ; LOADER: Set branch fs prob: MBB (8 -> 9): unroll.c:24:11 W=283590  0x50000000 / 0x80000000 = 62.50% --> 0x80000000 / 0x80000000 = 100.00%
-; LOADER: Set branch fs prob: MBB (10 -> 12): unroll.c:22:11-->unroll.c:24:11 W=283590  0x40000000 / 0x80000000 = 50.00% --> 0x7aca7894 / 0x80000000 = 95.93%
-; LOADER: Set branch fs prob: MBB (10 -> 11): unroll.c:22:11 W=283590  0x40000000 / 0x80000000 = 50.00% --> 0x0535876c / 0x80000000 = 4.07%
+; LOADERV0: Set branch fs prob: MBB (10 -> 12): unroll.c:22:11-->unroll.c:24:11 W=283590  0x40000000 / 0x80000000 = 50.00% --> 0x7aca7894 / 0x80000000 = 95.93%
+; LOADERV1: Set branch fs prob: MBB (10 -> 12): unroll.c:22:11-->unroll.c:24:11 W=283590  0x40000000 / 0x80000000 = 50.00% --> 0x0a5856e1 / 0x80000000 = 8.08%
+; LOADERV0: Set branch fs prob: MBB (10 -> 11): unroll.c:22:11 W=283590  0x40000000 / 0x80000000 = 50.00% --> 0x0535876c / 0x80000000 = 4.07%
+; LOADERV1: Set branch fs prob: MBB (10 -> 11): unroll.c:22:11 W=283590  0x40000000 / 0x80000000 = 50.00% --> 0x75a7a91f / 0x80000000 = 91.92%
 ; LOADER: Set branch fs prob: MBB (12 -> 14): unroll.c:24:11-->unroll.c:22:11 W=283590  0x30000000 / 0x80000000 = 37.50% --> 0x02012507 / 0x80000000 = 1.57%
 ; LOADER: Set branch fs prob: MBB (12 -> 13): unroll.c:24:11 W=283590  0x50000000 / 0x80000000 = 62.50% --> 0x7dfedaf9 / 0x80000000 = 98.43%
-; LOADER: Set branch fs prob: MBB (14 -> 16): unroll.c:22:11-->unroll.c:24:11 W=283590  0x40000000 / 0x80000000 = 50.00% --> 0x0a5856e1 / 0x80000000 = 8.08%
-; LOADER: Set branch fs prob: MBB (14 -> 15): unroll.c:22:11 W=283590  0x40000000 / 0x80000000 = 50.00% --> 0x75a7a91f / 0x80000000 = 91.92%
+; LOADERV0: Set branch fs prob: MBB (14 -> 16): unroll.c:22:11-->unroll.c:24:11 W=283590  0x40000000 / 0x80000000 = 50.00% --> 0x0a5856e1 / 0x80000000 = 8.08%
+; LOADERV1: Set branch fs prob: MBB (14 -> 16): unroll.c:22:11-->unroll.c:24:11 W=283590  0x40000000 / 0x80000000 = 50.00% --> 0x7aca7894 / 0x80000000 = 95.93%
+; LOADERV0: Set branch fs prob: MBB (14 -> 15): unroll.c:22:11 W=283590  0x40000000 / 0x80000000 = 50.00% --> 0x75a7a91f / 0x80000000 = 91.92%
+; LOADERV1: Set branch fs prob: MBB (14 -> 15): unroll.c:22:11 W=283590  0x40000000 / 0x80000000 = 50.00% --> 0x0535876c / 0x80000000 = 4.07%
 ; LOADER: Set branch fs prob: MBB (16 -> 18): unroll.c:24:11-->unroll.c:19:3 W=283590  0x30000000 / 0x80000000 = 37.50% --> 0x16588166 / 0x80000000 = 17.46%
 ; LOADER: Set branch fs prob: MBB (16 -> 17): unroll.c:24:11 W=283590  0x50000000 / 0x80000000 = 62.50% --> 0x69a77e9a / 0x80000000 = 82.54%
 
-
 target triple = "x86_64-unknown-linux-gnu"
 
 @sum = dso_local local_unnamed_addr global i32 0, align 4
diff --git a/llvm/test/CodeGen/X86/fsafdo_test3.ll b/llvm/test/CodeGen/X86/fsafdo_test3.ll
index 4a5ad040e9321..bbcc3ff59ec35 100644
--- a/llvm/test/CodeGen/X86/fsafdo_test3.ll
+++ b/llvm/test/CodeGen/X86/fsafdo_test3.ll
@@ -1,5 +1,7 @@
-; RUN: llvm-profdata merge --sample -profile-isfs -o %t.afdo %S/Inputs/fsloader.afdo
-; RUN: llc -enable-fs-discriminator -fs-profile-file=%t.afdo -disable-ra-fsprofile-loader=false -disable-layout-fsprofile-loader=false -print-machine-bfi -print-bfi-func-name=foo -print-before=fs-profile-loader -stop-after=fs-profile-loader < %s 2>&1 | FileCheck %s --check-prefix=BFI
+; RUN: llvm-profdata merge --sample -profile-isfs -o %t0.afdo %S/Inputs/fsloader.afdo
+; RUN: llc -enable-fs-discriminator -improved-fs-discriminator=false -fs-profile-file=%t0.afdo -disable-ra-fsprofile-loader=false -disable-layout-fsprofile-loader=false -print-machine-bfi -print-bfi-func-name=foo -print-before=fs-profile-loader -stop-after=fs-profile-loader < %s 2>&1 | FileCheck %s --check-prefixes=BFI,BFIV0
+; RUN: llvm-profdata merge --sample -profile-isfs -o %t1.afdo %S/Inputs/fsloader_v1.afdo
+; RUN: llc -enable-fs-discriminator -improved-fs-discriminator=true -fs-profile-file=%t1.afdo -disable-ra-fsprofile-loader=false -disable-layout-fsprofile-loader=false -print-machine-bfi -print-bfi-func-name=foo -print-before=fs-profile-loader -stop-after=fs-profile-loader < %s 2>&1 | FileCheck %s --check-prefixes=BFI,BFIV1
 ;
 ;;
 ;; C source code for the test (compiler at -O3):
@@ -63,7 +65,7 @@
 ;
 ; BFI: # *** IR Dump Before SampleFDO loader in MIR (fs-profile-loader) ***:
 ; BFI: # End machine code for function foo.
-;
+; BFI-EMPTY:
 ; BFI: block-frequency-info: foo
 ; BFI:  - BB0[entry]: float = 1.0, int = 8, count = 4268
 ; BFI:  - BB1[for.cond1.preheader]: float = 66.446, int = 531, count = 283289
@@ -75,11 +77,13 @@
 ; BFI:  - BB7[if.end.1]: float = 66.446, int = 531, count = 283289
 ; BFI:  - BB8[if.then7.1]: float = 66.446, int = 531, count = 283289
 ; BFI:  - BB9[if.end9.1]: float = 66.446, int = 531, count = 283289
-; BFI:  - BB10[if.then.2]: float = 2.7041, int = 21, count = 11204
+; BFIV0:  - BB10[if.then.2]: float = 2.7041, int = 21, count = 11204
+; BFIV1:  - BB10[if.then.2]: float = 61.075, int = 488, count = 260348
 ; BFI:  - BB11[if.end.2]: float = 66.446, int = 531, count = 283289
 ; BFI:  - BB12[if.then7.2]: float = 65.405, int = 523, count = 279021
 ; BFI:  - BB13[if.end9.2]: float = 66.446, int = 531, count = 283289
-; BFI:  - BB14[if.then.3]: float = 61.075, int = 488, count = 260348
+; BFIV0:  - BB14[if.then.3]: float = 61.075, int = 488, count = 260348
+; BFIV1:  - BB14[if.then.3]: float = 2.7041, int = 21, count = 11204
 ; BFI:  - BB15[if.end.3]: float = 66.446, int = 531, count = 283289
 ; BFI:  - BB16[if.then7.3]: float = 54.846, int = 438, count = 233673
 ; BFI:  - BB17[if.end9.3]: float = 66.446, int = 531, count = 283289
diff --git a/llvm/test/CodeGen/X86/fsafdo_test4.ll b/llvm/test/CodeGen/X86/fsafdo_test4.ll
index b5a3a591e9fd4..6a22ea9822412 100644
--- a/llvm/test/CodeGen/X86/fsafdo_test4.ll
+++ b/llvm/test/CodeGen/X86/fsafdo_test4.ll
@@ -1,10 +1,11 @@
-; RUN: llc -enable-fs-discriminator < %s | FileCheck %s
+; RUN: llc -enable-fs-discriminator -improved-fs-discriminator=false < %s | FileCheck %s
+; RUN: llc -enable-fs-discriminator -improved-fs-discriminator=true < %s | FileCheck %s
 ;
 ; Check that fs-afdo discriminators are NOT generated, as debugInfoForProfiling is false (not set).
 ; CHECK: .loc    1 7 3 is_stmt 0 discriminator 2 # foo.c:7:3
 ; CHECK: .loc    1 9 5 is_stmt 1 discriminator 2 # foo.c:9:5
-; CHECK-NOT: .loc    1 9 5 is_stmt 0 discriminator 11266 # foo.c:9:5
-; CHECK-NOT: .loc    1 7 3 is_stmt 1 discriminator 11266 # foo.c:7:3
+; CHECK-NOT: .loc    1 9 5 is_stmt 0 discriminator
+; CHECK-NOT: .loc    1 7 3 is_stmt 1 discriminator
 ; Check that variable __llvm_fs_discriminator__ is NOT generated.
 ; CHECK-NOT: __llvm_fs_discriminator__:
 
diff --git a/llvm/test/CodeGen/X86/half.ll b/llvm/test/CodeGen/X86/half.ll
index 14a1016fa9f4e..2761d3b099946 100644
--- a/llvm/test/CodeGen/X86/half.ll
+++ b/llvm/test/CodeGen/X86/half.ll
@@ -1355,4 +1355,800 @@ define <8 x half> @shuffle(ptr %p) {
   ret <8 x half> %2
 }
 
+declare half @llvm.minnum.f16(half, half)
+
+define half @pr61271(half %0, half %1) #0 {
+; CHECK-LIBCALL-LABEL: pr61271:
+; CHECK-LIBCALL:       # %bb.0:
+; CHECK-LIBCALL-NEXT:    subq $40, %rsp
+; CHECK-LIBCALL-NEXT:    movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps %xmm1, %xmm0
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    movss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Reload
+; CHECK-LIBCALL-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movaps %xmm0, %xmm1
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm2 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    cmpltss %xmm2, %xmm1
+; CHECK-LIBCALL-NEXT:    andps %xmm1, %xmm0
+; CHECK-LIBCALL-NEXT:    andnps %xmm2, %xmm1
+; CHECK-LIBCALL-NEXT:    orps %xmm1, %xmm0
+; CHECK-LIBCALL-NEXT:    callq __truncsfhf2@PLT
+; CHECK-LIBCALL-NEXT:    addq $40, %rsp
+; CHECK-LIBCALL-NEXT:    retq
+;
+; BWON-F16C-LABEL: pr61271:
+; BWON-F16C:       # %bb.0:
+; BWON-F16C-NEXT:    vpextrw $0, %xmm0, %eax
+; BWON-F16C-NEXT:    vpextrw $0, %xmm1, %ecx
+; BWON-F16C-NEXT:    movzwl %cx, %ecx
+; BWON-F16C-NEXT:    vmovd %ecx, %xmm0
+; BWON-F16C-NEXT:    vcvtph2ps %xmm0, %xmm0
+; BWON-F16C-NEXT:    movzwl %ax, %eax
+; BWON-F16C-NEXT:    vmovd %eax, %xmm1
+; BWON-F16C-NEXT:    vcvtph2ps %xmm1, %xmm1
+; BWON-F16C-NEXT:    vcmpltss %xmm0, %xmm1, %xmm2
+; BWON-F16C-NEXT:    vblendvps %xmm2, %xmm1, %xmm0, %xmm0
+; BWON-F16C-NEXT:    vcvtps2ph $4, %xmm0, %xmm0
+; BWON-F16C-NEXT:    vmovd %xmm0, %eax
+; BWON-F16C-NEXT:    vpinsrw $0, %eax, %xmm0, %xmm0
+; BWON-F16C-NEXT:    retq
+;
+; CHECK-I686-LABEL: pr61271:
+; CHECK-I686:       # %bb.0:
+; CHECK-I686-NEXT:    subl $44, %esp
+; CHECK-I686-NEXT:    pinsrw $0, {{[0-9]+}}(%esp), %xmm0
+; CHECK-I686-NEXT:    movdqa %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; CHECK-I686-NEXT:    pinsrw $0, {{[0-9]+}}(%esp), %xmm0
+; CHECK-I686-NEXT:    pextrw $0, %xmm0, %eax
+; CHECK-I686-NEXT:    movw %ax, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    movdqa {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-I686-NEXT:    pextrw $0, %xmm0, %eax
+; CHECK-I686-NEXT:    movw %ax, (%esp)
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    movaps %xmm1, %xmm2
+; CHECK-I686-NEXT:    cmpltss %xmm0, %xmm2
+; CHECK-I686-NEXT:    andps %xmm2, %xmm1
+; CHECK-I686-NEXT:    andnps %xmm0, %xmm2
+; CHECK-I686-NEXT:    orps %xmm1, %xmm2
+; CHECK-I686-NEXT:    movss %xmm2, (%esp)
+; CHECK-I686-NEXT:    calll __truncsfhf2
+; CHECK-I686-NEXT:    addl $44, %esp
+; CHECK-I686-NEXT:    retl
+  %3 = call fast half @llvm.minnum.f16(half %0, half %1)
+  ret half %3
+}
+
+declare <8 x half> @llvm.maxnum.v8f16(<8 x half>, <8 x half>)
+
+define <8 x half> @maxnum_v8f16(<8 x half> %0, <8 x half> %1) #0 {
+; CHECK-LIBCALL-LABEL: maxnum_v8f16:
+; CHECK-LIBCALL:       # %bb.0:
+; CHECK-LIBCALL-NEXT:    subq $184, %rsp
+; CHECK-LIBCALL-NEXT:    movaps %xmm1, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    movdqa %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    psrldq {{.*#+}} xmm0 = xmm0[14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; CHECK-LIBCALL-NEXT:    movdqa %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movd %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Folded Spill
+; CHECK-LIBCALL-NEXT:    movdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    psrldq {{.*#+}} xmm0 = xmm0[14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; CHECK-LIBCALL-NEXT:    movdqa %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movd %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Folded Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    ucomiss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Folded Reload
+; CHECK-LIBCALL-NEXT:    movss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Reload
+; CHECK-LIBCALL-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-LIBCALL-NEXT:    ja .LBB26_2
+; CHECK-LIBCALL-NEXT:  # %bb.1:
+; CHECK-LIBCALL-NEXT:    movss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Reload
+; CHECK-LIBCALL-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-LIBCALL-NEXT:  .LBB26_2:
+; CHECK-LIBCALL-NEXT:    callq __truncsfhf2@PLT
+; CHECK-LIBCALL-NEXT:    movaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    shufps {{.*#+}} xmm0 = xmm0[3,3,3,3]
+; CHECK-LIBCALL-NEXT:    movaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    shufps {{.*#+}} xmm0 = xmm0[3,3,3,3]
+; CHECK-LIBCALL-NEXT:    movaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movss %xmm0, (%rsp) # 4-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    ucomiss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Folded Reload
+; CHECK-LIBCALL-NEXT:    movss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Reload
+; CHECK-LIBCALL-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-LIBCALL-NEXT:    ja .LBB26_4
+; CHECK-LIBCALL-NEXT:  # %bb.3:
+; CHECK-LIBCALL-NEXT:    movss (%rsp), %xmm0 # 4-byte Reload
+; CHECK-LIBCALL-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-LIBCALL-NEXT:  .LBB26_4:
+; CHECK-LIBCALL-NEXT:    callq __truncsfhf2@PLT
+; CHECK-LIBCALL-NEXT:    movaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    movdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    psrldq {{.*#+}} xmm0 = xmm0[10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; CHECK-LIBCALL-NEXT:    movdqa %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movd %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Folded Spill
+; CHECK-LIBCALL-NEXT:    movdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    psrldq {{.*#+}} xmm0 = xmm0[10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; CHECK-LIBCALL-NEXT:    movdqa %xmm0, (%rsp) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movd %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Folded Spill
+; CHECK-LIBCALL-NEXT:    movaps (%rsp), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movss %xmm0, (%rsp) # 4-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    ucomiss (%rsp), %xmm0 # 4-byte Folded Reload
+; CHECK-LIBCALL-NEXT:    movss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Reload
+; CHECK-LIBCALL-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-LIBCALL-NEXT:    ja .LBB26_6
+; CHECK-LIBCALL-NEXT:  # %bb.5:
+; CHECK-LIBCALL-NEXT:    movss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Reload
+; CHECK-LIBCALL-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-LIBCALL-NEXT:  .LBB26_6:
+; CHECK-LIBCALL-NEXT:    callq __truncsfhf2@PLT
+; CHECK-LIBCALL-NEXT:    movaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    movhlps {{.*#+}} xmm0 = xmm0[1,1]
+; CHECK-LIBCALL-NEXT:    movaps %xmm0, (%rsp) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    movhlps {{.*#+}} xmm0 = xmm0[1,1]
+; CHECK-LIBCALL-NEXT:    movaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps (%rsp), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    ucomiss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Folded Reload
+; CHECK-LIBCALL-NEXT:    movss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Reload
+; CHECK-LIBCALL-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-LIBCALL-NEXT:    ja .LBB26_8
+; CHECK-LIBCALL-NEXT:  # %bb.7:
+; CHECK-LIBCALL-NEXT:    movss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Reload
+; CHECK-LIBCALL-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-LIBCALL-NEXT:  .LBB26_8:
+; CHECK-LIBCALL-NEXT:    callq __truncsfhf2@PLT
+; CHECK-LIBCALL-NEXT:    movaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    movdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    psrlq $48, %xmm0
+; CHECK-LIBCALL-NEXT:    movdqa %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movd %xmm0, (%rsp) # 4-byte Folded Spill
+; CHECK-LIBCALL-NEXT:    movdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    psrlq $48, %xmm0
+; CHECK-LIBCALL-NEXT:    movdqa %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movd %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Folded Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    ucomiss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Folded Reload
+; CHECK-LIBCALL-NEXT:    movss (%rsp), %xmm0 # 4-byte Reload
+; CHECK-LIBCALL-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-LIBCALL-NEXT:    ja .LBB26_10
+; CHECK-LIBCALL-NEXT:  # %bb.9:
+; CHECK-LIBCALL-NEXT:    movss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Reload
+; CHECK-LIBCALL-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-LIBCALL-NEXT:  .LBB26_10:
+; CHECK-LIBCALL-NEXT:    callq __truncsfhf2@PLT
+; CHECK-LIBCALL-NEXT:    movaps %xmm0, (%rsp) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    shufps {{.*#+}} xmm0 = xmm0[1,1,1,1]
+; CHECK-LIBCALL-NEXT:    movaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    shufps {{.*#+}} xmm0 = xmm0[1,1,1,1]
+; CHECK-LIBCALL-NEXT:    movaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movss {{[-0-9]+}}(%r{{[sb]}}p), %xmm1 # 4-byte Reload
+; CHECK-LIBCALL-NEXT:    # xmm1 = mem[0],zero,zero,zero
+; CHECK-LIBCALL-NEXT:    ucomiss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Folded Reload
+; CHECK-LIBCALL-NEXT:    ja .LBB26_12
+; CHECK-LIBCALL-NEXT:  # %bb.11:
+; CHECK-LIBCALL-NEXT:    movss {{[-0-9]+}}(%r{{[sb]}}p), %xmm1 # 4-byte Reload
+; CHECK-LIBCALL-NEXT:    # xmm1 = mem[0],zero,zero,zero
+; CHECK-LIBCALL-NEXT:  .LBB26_12:
+; CHECK-LIBCALL-NEXT:    movdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    punpcklwd {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Folded Reload
+; CHECK-LIBCALL-NEXT:    # xmm0 = xmm0[0],mem[0],xmm0[1],mem[1],xmm0[2],mem[2],xmm0[3],mem[3]
+; CHECK-LIBCALL-NEXT:    movdqa %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    movdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    punpcklwd {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Folded Reload
+; CHECK-LIBCALL-NEXT:    # xmm0 = xmm0[0],mem[0],xmm0[1],mem[1],xmm0[2],mem[2],xmm0[3],mem[3]
+; CHECK-LIBCALL-NEXT:    movdqa %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps %xmm1, %xmm0
+; CHECK-LIBCALL-NEXT:    callq __truncsfhf2@PLT
+; CHECK-LIBCALL-NEXT:    movaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movss {{[-0-9]+}}(%r{{[sb]}}p), %xmm1 # 4-byte Reload
+; CHECK-LIBCALL-NEXT:    # xmm1 = mem[0],zero,zero,zero
+; CHECK-LIBCALL-NEXT:    ucomiss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Folded Reload
+; CHECK-LIBCALL-NEXT:    ja .LBB26_14
+; CHECK-LIBCALL-NEXT:  # %bb.13:
+; CHECK-LIBCALL-NEXT:    movss {{[-0-9]+}}(%r{{[sb]}}p), %xmm1 # 4-byte Reload
+; CHECK-LIBCALL-NEXT:    # xmm1 = mem[0],zero,zero,zero
+; CHECK-LIBCALL-NEXT:  .LBB26_14:
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    unpcklps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Folded Reload
+; CHECK-LIBCALL-NEXT:    # xmm0 = xmm0[0],mem[0],xmm0[1],mem[1]
+; CHECK-LIBCALL-NEXT:    movaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    movdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    punpcklwd (%rsp), %xmm0 # 16-byte Folded Reload
+; CHECK-LIBCALL-NEXT:    # xmm0 = xmm0[0],mem[0],xmm0[1],mem[1],xmm0[2],mem[2],xmm0[3],mem[3]
+; CHECK-LIBCALL-NEXT:    movdqa %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps %xmm1, %xmm0
+; CHECK-LIBCALL-NEXT:    callq __truncsfhf2@PLT
+; CHECK-LIBCALL-NEXT:    movaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    movdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    psrld $16, %xmm0
+; CHECK-LIBCALL-NEXT:    movdqa %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movd %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Folded Spill
+; CHECK-LIBCALL-NEXT:    movdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    psrld $16, %xmm0
+; CHECK-LIBCALL-NEXT:    movdqa %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movd %xmm0, (%rsp) # 4-byte Folded Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; CHECK-LIBCALL-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    callq __extendhfsf2@PLT
+; CHECK-LIBCALL-NEXT:    ucomiss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Folded Reload
+; CHECK-LIBCALL-NEXT:    movd {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Folded Reload
+; CHECK-LIBCALL-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-LIBCALL-NEXT:    ja .LBB26_16
+; CHECK-LIBCALL-NEXT:  # %bb.15:
+; CHECK-LIBCALL-NEXT:    movd (%rsp), %xmm0 # 4-byte Folded Reload
+; CHECK-LIBCALL-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-LIBCALL-NEXT:  .LBB26_16:
+; CHECK-LIBCALL-NEXT:    callq __truncsfhf2@PLT
+; CHECK-LIBCALL-NEXT:    movdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm1 # 16-byte Reload
+; CHECK-LIBCALL-NEXT:    punpcklwd {{.*#+}} xmm1 = xmm1[0],xmm0[0],xmm1[1],xmm0[1],xmm1[2],xmm0[2],xmm1[3],xmm0[3]
+; CHECK-LIBCALL-NEXT:    punpckldq {{[-0-9]+}}(%r{{[sb]}}p), %xmm1 # 16-byte Folded Reload
+; CHECK-LIBCALL-NEXT:    # xmm1 = xmm1[0],mem[0],xmm1[1],mem[1]
+; CHECK-LIBCALL-NEXT:    punpcklqdq {{[-0-9]+}}(%r{{[sb]}}p), %xmm1 # 16-byte Folded Reload
+; CHECK-LIBCALL-NEXT:    # xmm1 = xmm1[0],mem[0]
+; CHECK-LIBCALL-NEXT:    movdqa %xmm1, %xmm0
+; CHECK-LIBCALL-NEXT:    addq $184, %rsp
+; CHECK-LIBCALL-NEXT:    retq
+;
+; BWON-F16C-LABEL: maxnum_v8f16:
+; BWON-F16C:       # %bb.0:
+; BWON-F16C-NEXT:    vpsrldq {{.*#+}} xmm2 = xmm1[14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; BWON-F16C-NEXT:    vpextrw $0, %xmm2, %eax
+; BWON-F16C-NEXT:    movzwl %ax, %eax
+; BWON-F16C-NEXT:    vmovd %eax, %xmm2
+; BWON-F16C-NEXT:    vcvtph2ps %xmm2, %xmm2
+; BWON-F16C-NEXT:    vpsrldq {{.*#+}} xmm3 = xmm0[14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; BWON-F16C-NEXT:    vpextrw $0, %xmm3, %eax
+; BWON-F16C-NEXT:    movzwl %ax, %eax
+; BWON-F16C-NEXT:    vmovd %eax, %xmm3
+; BWON-F16C-NEXT:    vcvtph2ps %xmm3, %xmm3
+; BWON-F16C-NEXT:    vucomiss %xmm2, %xmm3
+; BWON-F16C-NEXT:    ja .LBB26_2
+; BWON-F16C-NEXT:  # %bb.1:
+; BWON-F16C-NEXT:    vmovaps %xmm2, %xmm3
+; BWON-F16C-NEXT:  .LBB26_2:
+; BWON-F16C-NEXT:    vcvtps2ph $4, %xmm3, %xmm2
+; BWON-F16C-NEXT:    vpshufd {{.*#+}} xmm3 = xmm1[3,3,3,3]
+; BWON-F16C-NEXT:    vpextrw $0, %xmm3, %eax
+; BWON-F16C-NEXT:    movzwl %ax, %eax
+; BWON-F16C-NEXT:    vmovd %eax, %xmm3
+; BWON-F16C-NEXT:    vcvtph2ps %xmm3, %xmm3
+; BWON-F16C-NEXT:    vpshufd {{.*#+}} xmm4 = xmm0[3,3,3,3]
+; BWON-F16C-NEXT:    vpextrw $0, %xmm4, %eax
+; BWON-F16C-NEXT:    movzwl %ax, %eax
+; BWON-F16C-NEXT:    vmovd %eax, %xmm4
+; BWON-F16C-NEXT:    vcvtph2ps %xmm4, %xmm4
+; BWON-F16C-NEXT:    vucomiss %xmm3, %xmm4
+; BWON-F16C-NEXT:    ja .LBB26_4
+; BWON-F16C-NEXT:  # %bb.3:
+; BWON-F16C-NEXT:    vmovaps %xmm3, %xmm4
+; BWON-F16C-NEXT:  .LBB26_4:
+; BWON-F16C-NEXT:    vmovd %xmm2, %eax
+; BWON-F16C-NEXT:    vcvtps2ph $4, %xmm4, %xmm2
+; BWON-F16C-NEXT:    vmovd %xmm2, %ecx
+; BWON-F16C-NEXT:    vpsrldq {{.*#+}} xmm2 = xmm1[10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; BWON-F16C-NEXT:    vpextrw $0, %xmm2, %edx
+; BWON-F16C-NEXT:    movzwl %dx, %edx
+; BWON-F16C-NEXT:    vmovd %edx, %xmm2
+; BWON-F16C-NEXT:    vcvtph2ps %xmm2, %xmm2
+; BWON-F16C-NEXT:    vpsrldq {{.*#+}} xmm3 = xmm0[10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; BWON-F16C-NEXT:    vpextrw $0, %xmm3, %edx
+; BWON-F16C-NEXT:    movzwl %dx, %edx
+; BWON-F16C-NEXT:    vmovd %edx, %xmm3
+; BWON-F16C-NEXT:    vcvtph2ps %xmm3, %xmm3
+; BWON-F16C-NEXT:    vucomiss %xmm2, %xmm3
+; BWON-F16C-NEXT:    ja .LBB26_6
+; BWON-F16C-NEXT:  # %bb.5:
+; BWON-F16C-NEXT:    vmovaps %xmm2, %xmm3
+; BWON-F16C-NEXT:  .LBB26_6:
+; BWON-F16C-NEXT:    vcvtps2ph $4, %xmm3, %xmm2
+; BWON-F16C-NEXT:    vmovd %xmm2, %edx
+; BWON-F16C-NEXT:    vpermilpd {{.*#+}} xmm2 = xmm1[1,0]
+; BWON-F16C-NEXT:    vpextrw $0, %xmm2, %esi
+; BWON-F16C-NEXT:    movzwl %si, %esi
+; BWON-F16C-NEXT:    vmovd %esi, %xmm2
+; BWON-F16C-NEXT:    vcvtph2ps %xmm2, %xmm2
+; BWON-F16C-NEXT:    vpermilpd {{.*#+}} xmm3 = xmm0[1,0]
+; BWON-F16C-NEXT:    vpextrw $0, %xmm3, %esi
+; BWON-F16C-NEXT:    movzwl %si, %esi
+; BWON-F16C-NEXT:    vmovd %esi, %xmm3
+; BWON-F16C-NEXT:    vcvtph2ps %xmm3, %xmm3
+; BWON-F16C-NEXT:    vucomiss %xmm2, %xmm3
+; BWON-F16C-NEXT:    ja .LBB26_8
+; BWON-F16C-NEXT:  # %bb.7:
+; BWON-F16C-NEXT:    vmovaps %xmm2, %xmm3
+; BWON-F16C-NEXT:  .LBB26_8:
+; BWON-F16C-NEXT:    vcvtps2ph $4, %xmm3, %xmm2
+; BWON-F16C-NEXT:    vmovd %xmm2, %esi
+; BWON-F16C-NEXT:    vpsrlq $48, %xmm1, %xmm2
+; BWON-F16C-NEXT:    vpextrw $0, %xmm2, %edi
+; BWON-F16C-NEXT:    movzwl %di, %edi
+; BWON-F16C-NEXT:    vmovd %edi, %xmm2
+; BWON-F16C-NEXT:    vcvtph2ps %xmm2, %xmm2
+; BWON-F16C-NEXT:    vpsrlq $48, %xmm0, %xmm3
+; BWON-F16C-NEXT:    vpextrw $0, %xmm3, %edi
+; BWON-F16C-NEXT:    movzwl %di, %edi
+; BWON-F16C-NEXT:    vmovd %edi, %xmm3
+; BWON-F16C-NEXT:    vcvtph2ps %xmm3, %xmm6
+; BWON-F16C-NEXT:    vucomiss %xmm2, %xmm6
+; BWON-F16C-NEXT:    ja .LBB26_10
+; BWON-F16C-NEXT:  # %bb.9:
+; BWON-F16C-NEXT:    vmovaps %xmm2, %xmm6
+; BWON-F16C-NEXT:  .LBB26_10:
+; BWON-F16C-NEXT:    vpinsrw $0, %eax, %xmm0, %xmm2
+; BWON-F16C-NEXT:    vpinsrw $0, %ecx, %xmm0, %xmm3
+; BWON-F16C-NEXT:    vpinsrw $0, %edx, %xmm0, %xmm4
+; BWON-F16C-NEXT:    vpinsrw $0, %esi, %xmm0, %xmm5
+; BWON-F16C-NEXT:    vcvtps2ph $4, %xmm6, %xmm6
+; BWON-F16C-NEXT:    vmovd %xmm6, %eax
+; BWON-F16C-NEXT:    vmovshdup {{.*#+}} xmm6 = xmm1[1,1,3,3]
+; BWON-F16C-NEXT:    vpextrw $0, %xmm6, %ecx
+; BWON-F16C-NEXT:    movzwl %cx, %ecx
+; BWON-F16C-NEXT:    vmovd %ecx, %xmm6
+; BWON-F16C-NEXT:    vcvtph2ps %xmm6, %xmm6
+; BWON-F16C-NEXT:    vmovshdup {{.*#+}} xmm7 = xmm0[1,1,3,3]
+; BWON-F16C-NEXT:    vpextrw $0, %xmm7, %ecx
+; BWON-F16C-NEXT:    movzwl %cx, %ecx
+; BWON-F16C-NEXT:    vmovd %ecx, %xmm7
+; BWON-F16C-NEXT:    vcvtph2ps %xmm7, %xmm7
+; BWON-F16C-NEXT:    vucomiss %xmm6, %xmm7
+; BWON-F16C-NEXT:    ja .LBB26_12
+; BWON-F16C-NEXT:  # %bb.11:
+; BWON-F16C-NEXT:    vmovaps %xmm6, %xmm7
+; BWON-F16C-NEXT:  .LBB26_12:
+; BWON-F16C-NEXT:    vpunpcklwd {{.*#+}} xmm2 = xmm3[0],xmm2[0],xmm3[1],xmm2[1],xmm3[2],xmm2[2],xmm3[3],xmm2[3]
+; BWON-F16C-NEXT:    vpunpcklwd {{.*#+}} xmm3 = xmm5[0],xmm4[0],xmm5[1],xmm4[1],xmm5[2],xmm4[2],xmm5[3],xmm4[3]
+; BWON-F16C-NEXT:    vpinsrw $0, %eax, %xmm0, %xmm4
+; BWON-F16C-NEXT:    vcvtps2ph $4, %xmm7, %xmm5
+; BWON-F16C-NEXT:    vmovd %xmm5, %eax
+; BWON-F16C-NEXT:    vpinsrw $0, %eax, %xmm0, %xmm5
+; BWON-F16C-NEXT:    vpextrw $0, %xmm1, %eax
+; BWON-F16C-NEXT:    movzwl %ax, %eax
+; BWON-F16C-NEXT:    vmovd %eax, %xmm6
+; BWON-F16C-NEXT:    vcvtph2ps %xmm6, %xmm6
+; BWON-F16C-NEXT:    vpextrw $0, %xmm0, %eax
+; BWON-F16C-NEXT:    movzwl %ax, %eax
+; BWON-F16C-NEXT:    vmovd %eax, %xmm7
+; BWON-F16C-NEXT:    vcvtph2ps %xmm7, %xmm7
+; BWON-F16C-NEXT:    vucomiss %xmm6, %xmm7
+; BWON-F16C-NEXT:    ja .LBB26_14
+; BWON-F16C-NEXT:  # %bb.13:
+; BWON-F16C-NEXT:    vmovaps %xmm6, %xmm7
+; BWON-F16C-NEXT:  .LBB26_14:
+; BWON-F16C-NEXT:    vpunpckldq {{.*#+}} xmm2 = xmm3[0],xmm2[0],xmm3[1],xmm2[1]
+; BWON-F16C-NEXT:    vpunpcklwd {{.*#+}} xmm3 = xmm5[0],xmm4[0],xmm5[1],xmm4[1],xmm5[2],xmm4[2],xmm5[3],xmm4[3]
+; BWON-F16C-NEXT:    vcvtps2ph $4, %xmm7, %xmm4
+; BWON-F16C-NEXT:    vmovd %xmm4, %eax
+; BWON-F16C-NEXT:    vpinsrw $0, %eax, %xmm0, %xmm4
+; BWON-F16C-NEXT:    vpsrld $16, %xmm1, %xmm1
+; BWON-F16C-NEXT:    vpextrw $0, %xmm1, %eax
+; BWON-F16C-NEXT:    movzwl %ax, %eax
+; BWON-F16C-NEXT:    vmovd %eax, %xmm1
+; BWON-F16C-NEXT:    vcvtph2ps %xmm1, %xmm1
+; BWON-F16C-NEXT:    vpsrld $16, %xmm0, %xmm0
+; BWON-F16C-NEXT:    vpextrw $0, %xmm0, %eax
+; BWON-F16C-NEXT:    movzwl %ax, %eax
+; BWON-F16C-NEXT:    vmovd %eax, %xmm0
+; BWON-F16C-NEXT:    vcvtph2ps %xmm0, %xmm0
+; BWON-F16C-NEXT:    vucomiss %xmm1, %xmm0
+; BWON-F16C-NEXT:    ja .LBB26_16
+; BWON-F16C-NEXT:  # %bb.15:
+; BWON-F16C-NEXT:    vmovaps %xmm1, %xmm0
+; BWON-F16C-NEXT:  .LBB26_16:
+; BWON-F16C-NEXT:    vcvtps2ph $4, %xmm0, %xmm0
+; BWON-F16C-NEXT:    vmovd %xmm0, %eax
+; BWON-F16C-NEXT:    vpinsrw $0, %eax, %xmm0, %xmm0
+; BWON-F16C-NEXT:    vpunpcklwd {{.*#+}} xmm0 = xmm4[0],xmm0[0],xmm4[1],xmm0[1],xmm4[2],xmm0[2],xmm4[3],xmm0[3]
+; BWON-F16C-NEXT:    vpunpckldq {{.*#+}} xmm0 = xmm0[0],xmm3[0],xmm0[1],xmm3[1]
+; BWON-F16C-NEXT:    vpunpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
+; BWON-F16C-NEXT:    retq
+;
+; CHECK-I686-LABEL: maxnum_v8f16:
+; CHECK-I686:       # %bb.0:
+; CHECK-I686-NEXT:    pushl %ebx
+; CHECK-I686-NEXT:    pushl %edi
+; CHECK-I686-NEXT:    pushl %esi
+; CHECK-I686-NEXT:    subl $336, %esp # imm = 0x150
+; CHECK-I686-NEXT:    movaps %xmm1, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; CHECK-I686-NEXT:    movaps %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; CHECK-I686-NEXT:    movaps %xmm1, %xmm0
+; CHECK-I686-NEXT:    shufps {{.*#+}} xmm0 = xmm0[1,1],xmm1[1,1]
+; CHECK-I686-NEXT:    pextrw $0, %xmm0, %edi
+; CHECK-I686-NEXT:    movw %di, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-I686-NEXT:    shufps {{.*#+}} xmm0 = xmm0[1,1,1,1]
+; CHECK-I686-NEXT:    pextrw $0, %xmm0, %esi
+; CHECK-I686-NEXT:    movw %si, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movw %di, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movdqa {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-I686-NEXT:    pextrw $0, %xmm0, %edi
+; CHECK-I686-NEXT:    movw %di, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movdqa {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-I686-NEXT:    pextrw $0, %xmm0, %ebx
+; CHECK-I686-NEXT:    movw %bx, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movw %di, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movw %bx, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    movw %si, (%esp)
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    ucomiss {{[0-9]+}}(%esp), %xmm0
+; CHECK-I686-NEXT:    ja .LBB26_1
+; CHECK-I686-NEXT:  # %bb.2:
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    jmp .LBB26_3
+; CHECK-I686-NEXT:  .LBB26_1:
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:  .LBB26_3:
+; CHECK-I686-NEXT:    movss %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    movss {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 4-byte Reload
+; CHECK-I686-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    movss %xmm0, (%esp)
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    ucomiss {{[0-9]+}}(%esp), %xmm0
+; CHECK-I686-NEXT:    ja .LBB26_4
+; CHECK-I686-NEXT:  # %bb.5:
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    jmp .LBB26_6
+; CHECK-I686-NEXT:  .LBB26_4:
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:  .LBB26_6:
+; CHECK-I686-NEXT:    movss %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; CHECK-I686-NEXT:    calll __truncsfhf2
+; CHECK-I686-NEXT:    movaps %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; CHECK-I686-NEXT:    movss {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 4-byte Reload
+; CHECK-I686-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    movss %xmm0, (%esp)
+; CHECK-I686-NEXT:    calll __truncsfhf2
+; CHECK-I686-NEXT:    movaps %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; CHECK-I686-NEXT:    movdqa {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-I686-NEXT:    punpckhqdq {{.*#+}} xmm0 = xmm0[1,1]
+; CHECK-I686-NEXT:    pextrw $0, %xmm0, %edi
+; CHECK-I686-NEXT:    movw %di, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movdqa {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-I686-NEXT:    punpckhqdq {{.*#+}} xmm0 = xmm0[1,1]
+; CHECK-I686-NEXT:    pextrw $0, %xmm0, %esi
+; CHECK-I686-NEXT:    movw %si, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movw %di, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movdqa {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-I686-NEXT:    psrlq $48, %xmm0
+; CHECK-I686-NEXT:    pextrw $0, %xmm0, %edi
+; CHECK-I686-NEXT:    movw %di, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movdqa {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-I686-NEXT:    psrlq $48, %xmm0
+; CHECK-I686-NEXT:    pextrw $0, %xmm0, %ebx
+; CHECK-I686-NEXT:    movw %bx, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movw %di, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movw %bx, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    movw %si, (%esp)
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    ucomiss {{[0-9]+}}(%esp), %xmm0
+; CHECK-I686-NEXT:    ja .LBB26_7
+; CHECK-I686-NEXT:  # %bb.8:
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    jmp .LBB26_9
+; CHECK-I686-NEXT:  .LBB26_7:
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:  .LBB26_9:
+; CHECK-I686-NEXT:    movss %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    movss {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 4-byte Reload
+; CHECK-I686-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    movss %xmm0, (%esp)
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    ucomiss {{[0-9]+}}(%esp), %xmm0
+; CHECK-I686-NEXT:    ja .LBB26_10
+; CHECK-I686-NEXT:  # %bb.11:
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    jmp .LBB26_12
+; CHECK-I686-NEXT:  .LBB26_10:
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:  .LBB26_12:
+; CHECK-I686-NEXT:    movss %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; CHECK-I686-NEXT:    calll __truncsfhf2
+; CHECK-I686-NEXT:    movaps %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; CHECK-I686-NEXT:    movss {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 4-byte Reload
+; CHECK-I686-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    movss %xmm0, (%esp)
+; CHECK-I686-NEXT:    calll __truncsfhf2
+; CHECK-I686-NEXT:    movaps %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; CHECK-I686-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-I686-NEXT:    shufps {{.*#+}} xmm0 = xmm0[3,3,3,3]
+; CHECK-I686-NEXT:    pextrw $0, %xmm0, %edi
+; CHECK-I686-NEXT:    movw %di, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-I686-NEXT:    shufps {{.*#+}} xmm0 = xmm0[3,3,3,3]
+; CHECK-I686-NEXT:    pextrw $0, %xmm0, %esi
+; CHECK-I686-NEXT:    movw %si, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movw %di, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movdqa {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-I686-NEXT:    psrldq {{.*#+}} xmm0 = xmm0[10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; CHECK-I686-NEXT:    pextrw $0, %xmm0, %edi
+; CHECK-I686-NEXT:    movw %di, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movdqa {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-I686-NEXT:    psrldq {{.*#+}} xmm0 = xmm0[10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; CHECK-I686-NEXT:    pextrw $0, %xmm0, %ebx
+; CHECK-I686-NEXT:    movw %bx, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movw %di, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movw %bx, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    movw %si, (%esp)
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    ucomiss {{[0-9]+}}(%esp), %xmm0
+; CHECK-I686-NEXT:    ja .LBB26_13
+; CHECK-I686-NEXT:  # %bb.14:
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    jmp .LBB26_15
+; CHECK-I686-NEXT:  .LBB26_13:
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:  .LBB26_15:
+; CHECK-I686-NEXT:    movss %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    movss {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 4-byte Reload
+; CHECK-I686-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    movss %xmm0, (%esp)
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    ucomiss {{[0-9]+}}(%esp), %xmm0
+; CHECK-I686-NEXT:    ja .LBB26_16
+; CHECK-I686-NEXT:  # %bb.17:
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    jmp .LBB26_18
+; CHECK-I686-NEXT:  .LBB26_16:
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:  .LBB26_18:
+; CHECK-I686-NEXT:    movss %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; CHECK-I686-NEXT:    calll __truncsfhf2
+; CHECK-I686-NEXT:    movaps %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; CHECK-I686-NEXT:    movss {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 4-byte Reload
+; CHECK-I686-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    movss %xmm0, (%esp)
+; CHECK-I686-NEXT:    calll __truncsfhf2
+; CHECK-I686-NEXT:    movaps %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; CHECK-I686-NEXT:    movdqa {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-I686-NEXT:    psrld $16, %xmm0
+; CHECK-I686-NEXT:    pextrw $0, %xmm0, %edi
+; CHECK-I686-NEXT:    movw %di, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movdqa {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-I686-NEXT:    psrld $16, %xmm0
+; CHECK-I686-NEXT:    pextrw $0, %xmm0, %esi
+; CHECK-I686-NEXT:    movw %si, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movw %di, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movdqa {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-I686-NEXT:    psrldq {{.*#+}} xmm0 = xmm0[14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; CHECK-I686-NEXT:    pextrw $0, %xmm0, %edi
+; CHECK-I686-NEXT:    movw %di, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movdqa {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-I686-NEXT:    psrldq {{.*#+}} xmm0 = xmm0[14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; CHECK-I686-NEXT:    pextrw $0, %xmm0, %ebx
+; CHECK-I686-NEXT:    movw %bx, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movw %di, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    fstpt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Spill
+; CHECK-I686-NEXT:    movw %bx, (%esp)
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    movw %si, (%esp)
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    ucomiss {{[0-9]+}}(%esp), %xmm0
+; CHECK-I686-NEXT:    ja .LBB26_19
+; CHECK-I686-NEXT:  # %bb.20:
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    jmp .LBB26_21
+; CHECK-I686-NEXT:  .LBB26_19:
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:  .LBB26_21:
+; CHECK-I686-NEXT:    movss %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; CHECK-I686-NEXT:    calll __extendhfsf2
+; CHECK-I686-NEXT:    movss {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 4-byte Reload
+; CHECK-I686-NEXT:    # xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    movss %xmm0, (%esp)
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    fldt {{[-0-9]+}}(%e{{[sb]}}p) # 10-byte Folded Reload
+; CHECK-I686-NEXT:    fstps {{[0-9]+}}(%esp)
+; CHECK-I686-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    ucomiss {{[0-9]+}}(%esp), %xmm0
+; CHECK-I686-NEXT:    ja .LBB26_22
+; CHECK-I686-NEXT:  # %bb.23:
+; CHECK-I686-NEXT:    movd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    jmp .LBB26_24
+; CHECK-I686-NEXT:  .LBB26_22:
+; CHECK-I686-NEXT:    movd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:  .LBB26_24:
+; CHECK-I686-NEXT:    movd %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
+; CHECK-I686-NEXT:    calll __truncsfhf2
+; CHECK-I686-NEXT:    movss {{[-0-9]+}}(%e{{[sb]}}p), %xmm1 # 4-byte Reload
+; CHECK-I686-NEXT:    # xmm1 = mem[0],zero,zero,zero
+; CHECK-I686-NEXT:    movss %xmm1, (%esp)
+; CHECK-I686-NEXT:    movdqa {{[-0-9]+}}(%e{{[sb]}}p), %xmm1 # 16-byte Reload
+; CHECK-I686-NEXT:    punpcklwd {{.*#+}} xmm1 = xmm1[0],xmm0[0],xmm1[1],xmm0[1],xmm1[2],xmm0[2],xmm1[3],xmm0[3]
+; CHECK-I686-NEXT:    movdqa {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-I686-NEXT:    punpcklwd {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Folded Reload
+; CHECK-I686-NEXT:    # xmm0 = xmm0[0],mem[0],xmm0[1],mem[1],xmm0[2],mem[2],xmm0[3],mem[3]
+; CHECK-I686-NEXT:    punpckldq {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
+; CHECK-I686-NEXT:    movdqa %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; CHECK-I686-NEXT:    movdqa {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-I686-NEXT:    punpcklwd {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Folded Reload
+; CHECK-I686-NEXT:    # xmm0 = xmm0[0],mem[0],xmm0[1],mem[1],xmm0[2],mem[2],xmm0[3],mem[3]
+; CHECK-I686-NEXT:    movdqa %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; CHECK-I686-NEXT:    calll __truncsfhf2
+; CHECK-I686-NEXT:    movdqa {{[-0-9]+}}(%e{{[sb]}}p), %xmm1 # 16-byte Reload
+; CHECK-I686-NEXT:    punpcklwd {{.*#+}} xmm1 = xmm1[0],xmm0[0],xmm1[1],xmm0[1],xmm1[2],xmm0[2],xmm1[3],xmm0[3]
+; CHECK-I686-NEXT:    punpckldq {{[-0-9]+}}(%e{{[sb]}}p), %xmm1 # 16-byte Folded Reload
+; CHECK-I686-NEXT:    # xmm1 = xmm1[0],mem[0],xmm1[1],mem[1]
+; CHECK-I686-NEXT:    punpcklqdq {{[-0-9]+}}(%e{{[sb]}}p), %xmm1 # 16-byte Folded Reload
+; CHECK-I686-NEXT:    # xmm1 = xmm1[0],mem[0]
+; CHECK-I686-NEXT:    movdqa %xmm1, %xmm0
+; CHECK-I686-NEXT:    addl $336, %esp # imm = 0x150
+; CHECK-I686-NEXT:    popl %esi
+; CHECK-I686-NEXT:    popl %edi
+; CHECK-I686-NEXT:    popl %ebx
+; CHECK-I686-NEXT:    retl
+  %3 = call fast <8 x half> @llvm.maxnum.v8f16(<8 x half> %0, <8 x half> %1)
+  ret <8 x half> %3
+}
+
 attributes #0 = { nounwind }
diff --git a/llvm/test/CodeGen/X86/i386-baseptr.ll b/llvm/test/CodeGen/X86/i386-baseptr.ll
index 448658d93ee72..8baa16c076bac 100644
--- a/llvm/test/CodeGen/X86/i386-baseptr.ll
+++ b/llvm/test/CodeGen/X86/i386-baseptr.ll
@@ -6,11 +6,15 @@ define void @base() #0 {
 ; CHECK-LABEL: base:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    pushl %ebp
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-NEXT:    .cfi_offset %ebp, -8
 ; CHECK-NEXT:    movl %esp, %ebp
+; CHECK-NEXT:    .cfi_def_cfa_register %ebp
 ; CHECK-NEXT:    pushl %esi
 ; CHECK-NEXT:    andl $-32, %esp
 ; CHECK-NEXT:    subl $32, %esp
 ; CHECK-NEXT:    movl %esp, %esi
+; CHECK-NEXT:    .cfi_offset %esi, -12
 ; CHECK-NEXT:    calll helper@PLT
 ; CHECK-NEXT:    movl %esp, %ecx
 ; CHECK-NEXT:    leal 31(,%eax,4), %eax
@@ -23,6 +27,7 @@ define void @base() #0 {
 ; CHECK-NEXT:    leal -4(%ebp), %esp
 ; CHECK-NEXT:    popl %esi
 ; CHECK-NEXT:    popl %ebp
+; CHECK-NEXT:    .cfi_def_cfa %esp, 4
 ; CHECK-NEXT:    retl
 entry:
   %k = call i32 @helper()
@@ -35,11 +40,15 @@ define void @clobber_base() #0 {
 ; CHECK-LABEL: clobber_base:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    pushl %ebp
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-NEXT:    .cfi_offset %ebp, -8
 ; CHECK-NEXT:    movl %esp, %ebp
+; CHECK-NEXT:    .cfi_def_cfa_register %ebp
 ; CHECK-NEXT:    pushl %esi
 ; CHECK-NEXT:    andl $-128, %esp
 ; CHECK-NEXT:    subl $128, %esp
 ; CHECK-NEXT:    movl %esp, %esi
+; CHECK-NEXT:    .cfi_offset %esi, -12
 ; CHECK-NEXT:    calll helper@PLT
 ; CHECK-NEXT:    movl %esp, %ecx
 ; CHECK-NEXT:    leal 31(,%eax,4), %eax
@@ -60,6 +69,7 @@ define void @clobber_base() #0 {
 ; CHECK-NEXT:    leal -4(%ebp), %esp
 ; CHECK-NEXT:    popl %esi
 ; CHECK-NEXT:    popl %ebp
+; CHECK-NEXT:    .cfi_def_cfa %esp, 4
 ; CHECK-NEXT:    retl
 entry:
   %k = call i32 @helper()
@@ -72,6 +82,59 @@ entry:
   ret void
 }
 
-attributes #0 = { nounwind "frame-pointer"="all"}
+define x86_regcallcc void @clobber_baseptr_argptr(i32 %param1, i32 %param2, i32 %param3, i32 %param4, i32 %param5, i32 %param6) #0 {
+; CHECK-LABEL: clobber_baseptr_argptr:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    pushl %ebp
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-NEXT:    .cfi_offset %ebp, -8
+; CHECK-NEXT:    movl %esp, %ebp
+; CHECK-NEXT:    .cfi_def_cfa_register %ebp
+; CHECK-NEXT:    pushl %ebx
+; CHECK-NEXT:    andl $-128, %esp
+; CHECK-NEXT:    subl $128, %esp
+; CHECK-NEXT:    movl %esp, %esi
+; CHECK-NEXT:    .cfi_offset %ebx, -12
+; CHECK-NEXT:    movl 8(%ebp), %edi
+; CHECK-NEXT:    calll helper@PLT
+; CHECK-NEXT:    movl %esp, %ecx
+; CHECK-NEXT:    leal 31(,%eax,4), %eax
+; CHECK-NEXT:    andl $-32, %eax
+; CHECK-NEXT:    movl %ecx, %edx
+; CHECK-NEXT:    subl %eax, %edx
+; CHECK-NEXT:    movl %edx, %esp
+; CHECK-NEXT:    negl %eax
+; CHECK-NEXT:    movl $405, %esi # imm = 0x195
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    movl $405, %ebx # imm = 0x195
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    movl $8, %edx
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    movl %edx, (%esi)
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    movl %edi, (%ecx,%eax)
+; CHECK-NEXT:    leal -4(%ebp), %esp
+; CHECK-NEXT:    popl %ebx
+; CHECK-NEXT:    popl %ebp
+; CHECK-NEXT:    .cfi_def_cfa %esp, 4
+; CHECK-NEXT:    retl
+entry:
+  %k = call i32 @helper()
+  %a = alloca i32, align 128
+  %b = alloca i32, i32 %k, align 4
+  ; clobber base pointer register
+  tail call void asm sideeffect "nop", "{si}"(i32 405)
+  ; clobber argument pointer register
+  tail call void asm sideeffect "nop", "{bx}"(i32 405)
+  call void asm sideeffect "movl $0, $1", "r,*m"(i32 8, ptr elementtype(i32) %a)
+  store i32 %param6, ptr %b, align 4
+  ret void
+}
+
+attributes #0 = { "frame-pointer"="all"}
 !llvm.module.flags = !{!0}
 !0 = !{i32 2, !"override-stack-alignment", i32 32}
diff --git a/llvm/test/CodeGen/X86/illegal-bitfield-loadstore.ll b/llvm/test/CodeGen/X86/illegal-bitfield-loadstore.ll
index 05ad92cc0b330..7fb07c6b3163e 100644
--- a/llvm/test/CodeGen/X86/illegal-bitfield-loadstore.ll
+++ b/llvm/test/CodeGen/X86/illegal-bitfield-loadstore.ll
@@ -5,21 +5,19 @@
 define void @i24_or(ptr %a) {
 ; X86-LABEL: i24_or:
 ; X86:       # %bb.0:
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X86-NEXT:    movzwl (%ecx), %edx
-; X86-NEXT:    movzbl 2(%ecx), %eax
-; X86-NEXT:    movb %al, 2(%ecx)
-; X86-NEXT:    shll $16, %eax
-; X86-NEXT:    orl %edx, %eax
-; X86-NEXT:    orl $384, %eax # imm = 0x180
-; X86-NEXT:    movw %ax, (%ecx)
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movzwl (%eax), %ecx
+; X86-NEXT:    movzbl 2(%eax), %edx
+; X86-NEXT:    shll $16, %edx
+; X86-NEXT:    orl %ecx, %edx
+; X86-NEXT:    orl $384, %edx # imm = 0x180
+; X86-NEXT:    movw %dx, (%eax)
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: i24_or:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movzwl (%rdi), %eax
 ; X64-NEXT:    movzbl 2(%rdi), %ecx
-; X64-NEXT:    movb %cl, 2(%rdi)
 ; X64-NEXT:    shll $16, %ecx
 ; X64-NEXT:    orl %eax, %ecx
 ; X64-NEXT:    orl $384, %ecx # imm = 0x180
@@ -35,21 +33,19 @@ define void @i24_and_or(ptr %a) {
 ; X86-LABEL: i24_and_or:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X86-NEXT:    movzwl (%eax), %edx
-; X86-NEXT:    movzbl 2(%eax), %ecx
-; X86-NEXT:    movb %cl, 2(%eax)
-; X86-NEXT:    shll $16, %ecx
-; X86-NEXT:    orl %edx, %ecx
-; X86-NEXT:    orl $384, %ecx # imm = 0x180
-; X86-NEXT:    andl $-128, %ecx
-; X86-NEXT:    movw %cx, (%eax)
+; X86-NEXT:    movzwl (%eax), %ecx
+; X86-NEXT:    movzbl 2(%eax), %edx
+; X86-NEXT:    shll $16, %edx
+; X86-NEXT:    orl %ecx, %edx
+; X86-NEXT:    orl $384, %edx # imm = 0x180
+; X86-NEXT:    andl $-128, %edx
+; X86-NEXT:    movw %dx, (%eax)
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: i24_and_or:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movzwl (%rdi), %eax
 ; X64-NEXT:    movzbl 2(%rdi), %ecx
-; X64-NEXT:    movb %cl, 2(%rdi)
 ; X64-NEXT:    shll $16, %ecx
 ; X64-NEXT:    orl %eax, %ecx
 ; X64-NEXT:    orl $384, %ecx # imm = 0x180
@@ -66,21 +62,20 @@ define void @i24_and_or(ptr %a) {
 define void @i24_insert_bit(ptr %a, i1 zeroext %bit) {
 ; X86-LABEL: i24_insert_bit:
 ; X86:       # %bb.0:
-; X86-NEXT:    pushl %ebx
+; X86-NEXT:    pushl %esi
 ; X86-NEXT:    .cfi_def_cfa_offset 8
-; X86-NEXT:    .cfi_offset %ebx, -8
+; X86-NEXT:    .cfi_offset %esi, -8
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movzwl (%eax), %edx
-; X86-NEXT:    movzbl 2(%eax), %ebx
-; X86-NEXT:    movb %bl, 2(%eax)
-; X86-NEXT:    shll $16, %ebx
-; X86-NEXT:    orl %edx, %ebx
+; X86-NEXT:    movzbl 2(%eax), %esi
+; X86-NEXT:    shll $16, %esi
+; X86-NEXT:    orl %edx, %esi
 ; X86-NEXT:    shll $13, %ecx
-; X86-NEXT:    andl $16769023, %ebx # imm = 0xFFDFFF
-; X86-NEXT:    orl %ecx, %ebx
-; X86-NEXT:    movw %bx, (%eax)
-; X86-NEXT:    popl %ebx
+; X86-NEXT:    andl $16769023, %esi # imm = 0xFFDFFF
+; X86-NEXT:    orl %ecx, %esi
+; X86-NEXT:    movw %si, (%eax)
+; X86-NEXT:    popl %esi
 ; X86-NEXT:    .cfi_def_cfa_offset 4
 ; X86-NEXT:    retl
 ;
@@ -88,7 +83,6 @@ define void @i24_insert_bit(ptr %a, i1 zeroext %bit) {
 ; X64:       # %bb.0:
 ; X64-NEXT:    movzwl (%rdi), %eax
 ; X64-NEXT:    movzbl 2(%rdi), %ecx
-; X64-NEXT:    movb %cl, 2(%rdi)
 ; X64-NEXT:    shll $16, %ecx
 ; X64-NEXT:    orl %eax, %ecx
 ; X64-NEXT:    shll $13, %esi
@@ -114,8 +108,6 @@ define void @i56_or(ptr %a) {
 ;
 ; X64-LABEL: i56_or:
 ; X64:       # %bb.0:
-; X64-NEXT:    movzwl 4(%rdi), %eax
-; X64-NEXT:    movw %ax, 4(%rdi)
 ; X64-NEXT:    orl $384, (%rdi) # imm = 0x180
 ; X64-NEXT:    retq
   %aa = load i56, ptr %a, align 1
@@ -138,8 +130,6 @@ define void @i56_and_or(ptr %a) {
 ; X64:       # %bb.0:
 ; X64-NEXT:    movzwl 4(%rdi), %eax
 ; X64-NEXT:    movzbl 6(%rdi), %ecx
-; X64-NEXT:    movb %cl, 6(%rdi)
-; X64-NEXT:    # kill: def $ecx killed $ecx def $rcx
 ; X64-NEXT:    shll $16, %ecx
 ; X64-NEXT:    orl %eax, %ecx
 ; X64-NEXT:    shlq $32, %rcx
@@ -175,8 +165,6 @@ define void @i56_insert_bit(ptr %a, i1 zeroext %bit) {
 ; X64:       # %bb.0:
 ; X64-NEXT:    movzwl 4(%rdi), %eax
 ; X64-NEXT:    movzbl 6(%rdi), %ecx
-; X64-NEXT:    movb %cl, 6(%rdi)
-; X64-NEXT:    # kill: def $ecx killed $ecx def $rcx
 ; X64-NEXT:    shll $16, %ecx
 ; X64-NEXT:    orl %eax, %ecx
 ; X64-NEXT:    shlq $32, %rcx
diff --git a/llvm/test/CodeGen/X86/insertelement-zero.ll b/llvm/test/CodeGen/X86/insertelement-zero.ll
index 6a2174626aced..952940d141808 100644
--- a/llvm/test/CodeGen/X86/insertelement-zero.ll
+++ b/llvm/test/CodeGen/X86/insertelement-zero.ll
@@ -4,9 +4,9 @@
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+ssse3 | FileCheck %s --check-prefixes=SSE,SSSE3
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse4.1 | FileCheck %s --check-prefixes=SSE,SSE41
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx | FileCheck %s --check-prefixes=AVX,AVX1
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx2 | FileCheck %s --check-prefixes=AVX,AVX2,AVX2-SLOW
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx2,+fast-variable-crosslane-shuffle,+fast-variable-perlane-shuffle | FileCheck %s --check-prefixes=AVX,AVX2,AVX2-FAST
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx2,+fast-variable-perlane-shuffle | FileCheck %s --check-prefixes=AVX,AVX2,AVX2-FAST
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx2 | FileCheck %s --check-prefixes=AVX,AVX2
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx2,+fast-variable-crosslane-shuffle,+fast-variable-perlane-shuffle | FileCheck %s --check-prefixes=AVX,AVX2
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx2,+fast-variable-perlane-shuffle | FileCheck %s --check-prefixes=AVX,AVX2
 
 define <2 x double> @insert_v2f64_z1(<2 x double> %a) {
 ; SSE2-LABEL: insert_v2f64_z1:
@@ -268,29 +268,20 @@ define <4 x i32> @insert_v4i32_01z3(<4 x i32> %a) {
 define <8 x i32> @insert_v8i32_z12345z7(<8 x i32> %a) {
 ; SSE2-LABEL: insert_v8i32_z12345z7:
 ; SSE2:       # %bb.0:
-; SSE2-NEXT:    xorps %xmm2, %xmm2
-; SSE2-NEXT:    movss {{.*#+}} xmm0 = xmm2[0],xmm0[1,2,3]
-; SSE2-NEXT:    xorps %xmm2, %xmm2
-; SSE2-NEXT:    shufps {{.*#+}} xmm2 = xmm2[0,0],xmm1[3,0]
-; SSE2-NEXT:    shufps {{.*#+}} xmm1 = xmm1[0,1],xmm2[0,2]
+; SSE2-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
+; SSE2-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm1
 ; SSE2-NEXT:    retq
 ;
 ; SSE3-LABEL: insert_v8i32_z12345z7:
 ; SSE3:       # %bb.0:
-; SSE3-NEXT:    xorps %xmm2, %xmm2
-; SSE3-NEXT:    movss {{.*#+}} xmm0 = xmm2[0],xmm0[1,2,3]
-; SSE3-NEXT:    xorps %xmm2, %xmm2
-; SSE3-NEXT:    shufps {{.*#+}} xmm2 = xmm2[0,0],xmm1[3,0]
-; SSE3-NEXT:    shufps {{.*#+}} xmm1 = xmm1[0,1],xmm2[0,2]
+; SSE3-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
+; SSE3-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm1
 ; SSE3-NEXT:    retq
 ;
 ; SSSE3-LABEL: insert_v8i32_z12345z7:
 ; SSSE3:       # %bb.0:
-; SSSE3-NEXT:    xorps %xmm2, %xmm2
-; SSSE3-NEXT:    movss {{.*#+}} xmm0 = xmm2[0],xmm0[1,2,3]
-; SSSE3-NEXT:    xorps %xmm2, %xmm2
-; SSSE3-NEXT:    shufps {{.*#+}} xmm2 = xmm2[0,0],xmm1[3,0]
-; SSSE3-NEXT:    shufps {{.*#+}} xmm1 = xmm1[0,1],xmm2[0,2]
+; SSSE3-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
+; SSSE3-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm1
 ; SSSE3-NEXT:    retq
 ;
 ; SSE41-LABEL: insert_v8i32_z12345z7:
@@ -313,23 +304,17 @@ define <8 x i32> @insert_v8i32_z12345z7(<8 x i32> %a) {
 define <8 x i16> @insert_v8i16_z12345z7(<8 x i16> %a) {
 ; SSE2-LABEL: insert_v8i16_z12345z7:
 ; SSE2:       # %bb.0:
-; SSE2-NEXT:    xorl %eax, %eax
-; SSE2-NEXT:    pinsrw $0, %eax, %xmm0
-; SSE2-NEXT:    pinsrw $6, %eax, %xmm0
+; SSE2-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
 ; SSE2-NEXT:    retq
 ;
 ; SSE3-LABEL: insert_v8i16_z12345z7:
 ; SSE3:       # %bb.0:
-; SSE3-NEXT:    xorl %eax, %eax
-; SSE3-NEXT:    pinsrw $0, %eax, %xmm0
-; SSE3-NEXT:    pinsrw $6, %eax, %xmm0
+; SSE3-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
 ; SSE3-NEXT:    retq
 ;
 ; SSSE3-LABEL: insert_v8i16_z12345z7:
 ; SSSE3:       # %bb.0:
-; SSSE3-NEXT:    xorl %eax, %eax
-; SSSE3-NEXT:    pinsrw $0, %eax, %xmm0
-; SSSE3-NEXT:    pinsrw $6, %eax, %xmm0
+; SSSE3-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
 ; SSSE3-NEXT:    retq
 ;
 ; SSE41-LABEL: insert_v8i16_z12345z7:
@@ -351,26 +336,20 @@ define <8 x i16> @insert_v8i16_z12345z7(<8 x i16> %a) {
 define <16 x i16> @insert_v16i16_z12345z789ABCDEz(<16 x i16> %a) {
 ; SSE2-LABEL: insert_v16i16_z12345z789ABCDEz:
 ; SSE2:       # %bb.0:
-; SSE2-NEXT:    xorl %eax, %eax
-; SSE2-NEXT:    pinsrw $0, %eax, %xmm0
-; SSE2-NEXT:    pinsrw $6, %eax, %xmm0
-; SSE2-NEXT:    pinsrw $7, %eax, %xmm1
+; SSE2-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
+; SSE2-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm1
 ; SSE2-NEXT:    retq
 ;
 ; SSE3-LABEL: insert_v16i16_z12345z789ABCDEz:
 ; SSE3:       # %bb.0:
-; SSE3-NEXT:    xorl %eax, %eax
-; SSE3-NEXT:    pinsrw $0, %eax, %xmm0
-; SSE3-NEXT:    pinsrw $6, %eax, %xmm0
-; SSE3-NEXT:    pinsrw $7, %eax, %xmm1
+; SSE3-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
+; SSE3-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm1
 ; SSE3-NEXT:    retq
 ;
 ; SSSE3-LABEL: insert_v16i16_z12345z789ABCDEz:
 ; SSSE3:       # %bb.0:
-; SSSE3-NEXT:    xorl %eax, %eax
-; SSSE3-NEXT:    pinsrw $0, %eax, %xmm0
-; SSSE3-NEXT:    pinsrw $6, %eax, %xmm0
-; SSSE3-NEXT:    pinsrw $7, %eax, %xmm1
+; SSSE3-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
+; SSSE3-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm1
 ; SSSE3-NEXT:    retq
 ;
 ; SSE41-LABEL: insert_v16i16_z12345z789ABCDEz:
@@ -391,46 +370,15 @@ define <16 x i16> @insert_v16i16_z12345z789ABCDEz(<16 x i16> %a) {
 }
 
 define <16 x i8> @insert_v16i8_z123456789ABCDEz(<16 x i8> %a) {
-; SSE2-LABEL: insert_v16i8_z123456789ABCDEz:
-; SSE2:       # %bb.0:
-; SSE2-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
-; SSE2-NEXT:    retq
-;
-; SSE3-LABEL: insert_v16i8_z123456789ABCDEz:
-; SSE3:       # %bb.0:
-; SSE3-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
-; SSE3-NEXT:    retq
-;
-; SSSE3-LABEL: insert_v16i8_z123456789ABCDEz:
-; SSSE3:       # %bb.0:
-; SSSE3-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
-; SSSE3-NEXT:    retq
-;
-; SSE41-LABEL: insert_v16i8_z123456789ABCDEz:
-; SSE41:       # %bb.0:
-; SSE41-NEXT:    xorl %eax, %eax
-; SSE41-NEXT:    pinsrb $0, %eax, %xmm0
-; SSE41-NEXT:    pinsrb $15, %eax, %xmm0
-; SSE41-NEXT:    retq
-;
-; AVX1-LABEL: insert_v16i8_z123456789ABCDEz:
-; AVX1:       # %bb.0:
-; AVX1-NEXT:    xorl %eax, %eax
-; AVX1-NEXT:    vpinsrb $0, %eax, %xmm0, %xmm0
-; AVX1-NEXT:    vpinsrb $15, %eax, %xmm0, %xmm0
-; AVX1-NEXT:    retq
+; SSE-LABEL: insert_v16i8_z123456789ABCDEz:
+; SSE:       # %bb.0:
+; SSE-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
+; SSE-NEXT:    retq
 ;
-; AVX2-SLOW-LABEL: insert_v16i8_z123456789ABCDEz:
-; AVX2-SLOW:       # %bb.0:
-; AVX2-SLOW-NEXT:    xorl %eax, %eax
-; AVX2-SLOW-NEXT:    vpinsrb $0, %eax, %xmm0, %xmm0
-; AVX2-SLOW-NEXT:    vpinsrb $15, %eax, %xmm0, %xmm0
-; AVX2-SLOW-NEXT:    retq
-;
-; AVX2-FAST-LABEL: insert_v16i8_z123456789ABCDEz:
-; AVX2-FAST:       # %bb.0:
-; AVX2-FAST-NEXT:    vandps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
-; AVX2-FAST-NEXT:    retq
+; AVX-LABEL: insert_v16i8_z123456789ABCDEz:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vandps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
+; AVX-NEXT:    retq
   %1 = insertelement <16 x i8> %a, i8 0, i32 0
   %2 = insertelement <16 x i8> %1, i8 0, i32 15
   ret <16 x i8> %2
@@ -457,11 +405,9 @@ define <32 x i8> @insert_v32i8_z123456789ABCDEzGHIJKLMNOPQRSTzz(<32 x i8> %a) {
 ;
 ; SSE41-LABEL: insert_v32i8_z123456789ABCDEzGHIJKLMNOPQRSTzz:
 ; SSE41:       # %bb.0:
-; SSE41-NEXT:    xorl %eax, %eax
-; SSE41-NEXT:    pinsrb $0, %eax, %xmm0
-; SSE41-NEXT:    pinsrb $15, %eax, %xmm0
 ; SSE41-NEXT:    pxor %xmm2, %xmm2
 ; SSE41-NEXT:    pblendw {{.*#+}} xmm1 = xmm1[0,1,2,3,4,5,6],xmm2[7]
+; SSE41-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
 ; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: insert_v32i8_z123456789ABCDEzGHIJKLMNOPQRSTzz:
diff --git a/llvm/test/CodeGen/X86/is_fpclass-fp80.ll b/llvm/test/CodeGen/X86/is_fpclass-fp80.ll
index 17569234abd0f..4ebcce5d4b462 100644
--- a/llvm/test/CodeGen/X86/is_fpclass-fp80.ll
+++ b/llvm/test/CodeGen/X86/is_fpclass-fp80.ll
@@ -267,10 +267,10 @@ define i1 @is_inf_f80(x86_fp80 %x) {
 ; CHECK-64:       # %bb.0: # %entry
 ; CHECK-64-NEXT:    movl {{[0-9]+}}(%rsp), %eax
 ; CHECK-64-NEXT:    notl %eax
-; CHECK-64-NEXT:    andl $32767, %eax # imm = 0x7FFF
 ; CHECK-64-NEXT:    movabsq $-9223372036854775808, %rcx # imm = 0x8000000000000000
 ; CHECK-64-NEXT:    xorq {{[0-9]+}}(%rsp), %rcx
-; CHECK-64-NEXT:    orq %rax, %rcx
+; CHECK-64-NEXT:    andl $32767, %eax # imm = 0x7FFF
+; CHECK-64-NEXT:    orq %rcx, %rax
 ; CHECK-64-NEXT:    sete %al
 ; CHECK-64-NEXT:    retq
 entry:
@@ -318,12 +318,11 @@ define i1 @is_neginf_f80(x86_fp80 %x) {
 ;
 ; CHECK-64-LABEL: is_neginf_f80:
 ; CHECK-64:       # %bb.0: # %entry
-; CHECK-64-NEXT:    movl {{[0-9]+}}(%rsp), %eax
-; CHECK-64-NEXT:    notl %eax
-; CHECK-64-NEXT:    movzwl %ax, %eax
+; CHECK-64-NEXT:    movzwl {{[0-9]+}}(%rsp), %eax
+; CHECK-64-NEXT:    xorl $65535, %eax # imm = 0xFFFF
 ; CHECK-64-NEXT:    movabsq $-9223372036854775808, %rcx # imm = 0x8000000000000000
 ; CHECK-64-NEXT:    xorq {{[0-9]+}}(%rsp), %rcx
-; CHECK-64-NEXT:    orq %rax, %rcx
+; CHECK-64-NEXT:    orq %rcx, %rax
 ; CHECK-64-NEXT:    sete %al
 ; CHECK-64-NEXT:    retq
 entry:
diff --git a/llvm/test/CodeGen/X86/is_fpclass.ll b/llvm/test/CodeGen/X86/is_fpclass.ll
index edeadd01893cb..a12efe7aa8a25 100644
--- a/llvm/test/CodeGen/X86/is_fpclass.ll
+++ b/llvm/test/CodeGen/X86/is_fpclass.ll
@@ -71,6 +71,33 @@ entry:
   ret i1 %0
 }
 
+define i1 @not_issignaling_f(float %x) {
+; CHECK-32-LABEL: not_issignaling_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2143289344, %eax # imm = 0x7FC00000
+; CHECK-32-NEXT:    setge %cl
+; CHECK-32-NEXT:    cmpl $2139095041, %eax # imm = 0x7F800001
+; CHECK-32-NEXT:    setl %al
+; CHECK-32-NEXT:    orb %cl, %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_issignaling_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2143289344, %eax # imm = 0x7FC00000
+; CHECK-64-NEXT:    setge %cl
+; CHECK-64-NEXT:    cmpl $2139095041, %eax # imm = 0x7F800001
+; CHECK-64-NEXT:    setl %al
+; CHECK-64-NEXT:    orb %cl, %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 1022)  ; ~"snan"
+  ret i1 %0
+}
+
 define i1 @isquiet_f(float %x) {
 ; CHECK-32-LABEL: isquiet_f:
 ; CHECK-32:       # %bb.0: # %entry
@@ -92,6 +119,27 @@ entry:
   ret i1 %0
 }
 
+define i1 @not_isquiet_f(float %x) {
+; CHECK-32-LABEL: not_isquiet_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2143289344, %eax # imm = 0x7FC00000
+; CHECK-32-NEXT:    setl %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_isquiet_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2143289344, %eax # imm = 0x7FC00000
+; CHECK-64-NEXT:    setl %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 1021)  ; ~"qnan"
+  ret i1 %0
+}
+
 define i1 @isinf_f(float %x) {
 ; CHECK-32-LABEL: isinf_f:
 ; CHECK-32:       # %bb.0: # %entry
@@ -113,6 +161,27 @@ entry:
   ret i1 %0
 }
 
+define i1 @not_isinf_f(float %x) {
+; CHECK-32-LABEL: not_isinf_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-32-NEXT:    setne %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_isinf_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-64-NEXT:    setne %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 507)  ; ~0x204 = "~inf"
+  ret i1 %0
+}
+
 define i1 @is_plus_inf_f(float %x) {
 ; CHECK-32-LABEL: is_plus_inf_f:
 ; CHECK-32:       # %bb.0: # %entry
@@ -149,6 +218,24 @@ entry:
   ret i1 %0
 }
 
+define i1 @not_is_minus_inf_f(float %x) {
+; CHECK-32-LABEL: not_is_minus_inf_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    cmpl $-8388608, {{[0-9]+}}(%esp) # imm = 0xFF800000
+; CHECK-32-NEXT:    setne %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_is_minus_inf_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    cmpl $-8388608, %eax # imm = 0xFF800000
+; CHECK-64-NEXT:    setne %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 1019)  ; ~"-inf"
+  ret i1 %0
+}
+
 define i1 @isfinite_f(float %x) {
 ; CHECK-32-LABEL: isfinite_f:
 ; CHECK-32:       # %bb.0: # %entry
@@ -170,6 +257,27 @@ entry:
   ret i1 %0
 }
 
+define i1 @not_isfinite_f(float %x) {
+; CHECK-32-LABEL: not_isfinite_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-32-NEXT:    setge %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_isfinite_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-64-NEXT:    setge %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 519)  ; ~0x1f8 = "~finite"
+  ret i1 %0
+}
+
 define i1 @is_plus_finite_f(float %x) {
 ; CHECK-32-LABEL: is_plus_finite_f:
 ; CHECK-32:       # %bb.0: # %entry
@@ -188,6 +296,24 @@ entry:
   ret i1 %0
 }
 
+define i1 @not_is_plus_finite_f(float %x) {
+; CHECK-32-LABEL: not_is_plus_finite_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    cmpl $2139095040, {{[0-9]+}}(%esp) # imm = 0x7F800000
+; CHECK-32-NEXT:    setae %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_is_plus_finite_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-64-NEXT:    setae %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 575)  ; ~0x1c0 = ~"+finite"
+  ret i1 %0
+}
+
 define i1 @is_minus_finite_f(float %x) {
 ; CHECK-32-LABEL: is_minus_finite_f:
 ; CHECK-32:       # %bb.0: # %entry
@@ -215,7 +341,34 @@ entry:
   ret i1 %0
 }
 
-define i1 @isnormal_f(float %x) {
+define i1 @not_is_minus_finite_f(float %x) {
+; CHECK-32-LABEL: not_is_minus_finite_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    testl %eax, %eax
+; CHECK-32-NEXT:    setns %cl
+; CHECK-32-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-32-NEXT:    setge %al
+; CHECK-32-NEXT:    orb %cl, %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_is_minus_finite_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    testl %eax, %eax
+; CHECK-64-NEXT:    setns %cl
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-64-NEXT:    setge %al
+; CHECK-64-NEXT:    orb %cl, %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 967)  ; ~0x38 = ~"-finite"
+  ret i1 %0
+}
+
+define i1 @isnormal_f(float %x) #1 {
 ; CHECK-32-LABEL: isnormal_f:
 ; CHECK-32:       # %bb.0: # %entry
 ; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
@@ -238,6 +391,29 @@ entry:
   ret i1 %0
 }
 
+define i1 @not_isnormal_f(float %x) #1 {
+; CHECK-32-LABEL: not_isnormal_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    addl $-8388608, %eax # imm = 0xFF800000
+; CHECK-32-NEXT:    cmpl $2130706432, %eax # imm = 0x7F000000
+; CHECK-32-NEXT:    setae %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_isnormal_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    addl $-8388608, %eax # imm = 0xFF800000
+; CHECK-64-NEXT:    cmpl $2130706432, %eax # imm = 0x7F000000
+; CHECK-64-NEXT:    setae %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 759)  ; ~0x108 = "~normal"
+  ret i1 %0
+}
+
 define i1 @is_plus_normal_f(float %x) {
 ; CHECK-32-LABEL: is_plus_normal_f:
 ; CHECK-32:       # %bb.0: # %entry
@@ -336,6 +512,75 @@ entry:
   ret i1 %0
 }
 
+define i1 @not_issubnormal_f(float %x) {
+; CHECK-32-LABEL: not_issubnormal_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    decl %eax
+; CHECK-32-NEXT:    cmpl $8388607, %eax # imm = 0x7FFFFF
+; CHECK-32-NEXT:    setae %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_issubnormal_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    decl %eax
+; CHECK-64-NEXT:    cmpl $8388607, %eax # imm = 0x7FFFFF
+; CHECK-64-NEXT:    setae %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 879)  ; ~0x90 = "~subnormal"
+  ret i1 %0
+}
+
+define i1 @not_issubnormal_f_daz(float %x) #0 {
+; CHECK-32-LABEL: not_issubnormal_f_daz:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    decl %eax
+; CHECK-32-NEXT:    cmpl $8388607, %eax # imm = 0x7FFFFF
+; CHECK-32-NEXT:    setae %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_issubnormal_f_daz:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    decl %eax
+; CHECK-64-NEXT:    cmpl $8388607, %eax # imm = 0x7FFFFF
+; CHECK-64-NEXT:    setae %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 879)  ; ~0x90 = "~subnormal"
+  ret i1 %0
+}
+
+define i1 @not_issubnormal_f_maybe_daz(float %x) #1 {
+; CHECK-32-LABEL: not_issubnormal_f_maybe_daz:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    decl %eax
+; CHECK-32-NEXT:    cmpl $8388607, %eax # imm = 0x7FFFFF
+; CHECK-32-NEXT:    setae %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_issubnormal_f_maybe_daz:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    decl %eax
+; CHECK-64-NEXT:    cmpl $8388607, %eax # imm = 0x7FFFFF
+; CHECK-64-NEXT:    setae %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 879)  ; ~0x90 = "~subnormal"
+  ret i1 %0
+}
+
 define i1 @is_plus_subnormal_f(float %x) {
 ; CHECK-32-LABEL: is_plus_subnormal_f:
 ; CHECK-32:       # %bb.0: # %entry
@@ -357,6 +602,27 @@ entry:
   ret i1 %0
 }
 
+define i1 @not_is_plus_subnormal_f(float %x) {
+; CHECK-32-LABEL: not_is_plus_subnormal_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    decl %eax
+; CHECK-32-NEXT:    cmpl $8388607, %eax # imm = 0x7FFFFF
+; CHECK-32-NEXT:    setae %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_is_plus_subnormal_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    decl %eax
+; CHECK-64-NEXT:    cmpl $8388607, %eax # imm = 0x7FFFFF
+; CHECK-64-NEXT:    setae %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 895)  ; ~0x80 = ~"+subnormal"
+  ret i1 %0
+}
+
 define i1 @is_minus_subnormal_f(float %x) {
 ; CHECK-32-LABEL: is_minus_subnormal_f:
 ; CHECK-32:       # %bb.0: # %entry
@@ -386,6 +652,35 @@ entry:
   ret i1 %0
 }
 
+define i1 @not_is_minus_subnormal_f(float %x) {
+; CHECK-32-LABEL: not_is_minus_subnormal_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    testl %eax, %eax
+; CHECK-32-NEXT:    setns %cl
+; CHECK-32-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    decl %eax
+; CHECK-32-NEXT:    cmpl $8388607, %eax # imm = 0x7FFFFF
+; CHECK-32-NEXT:    setae %al
+; CHECK-32-NEXT:    orb %cl, %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_is_minus_subnormal_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    testl %eax, %eax
+; CHECK-64-NEXT:    setns %cl
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    decl %eax
+; CHECK-64-NEXT:    cmpl $8388607, %eax # imm = 0x7FFFFF
+; CHECK-64-NEXT:    setae %al
+; CHECK-64-NEXT:    orb %cl, %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 1007)  ; ~0x10 = ~"-subnormal"
+  ret i1 %0
+}
+
 define i1 @iszero_f(float %x) {
 ; CHECK-32-LABEL: iszero_f:
 ; CHECK-32:       # %bb.0: # %entry
@@ -449,38 +744,101 @@ entry:
   ret i1 %0
 }
 
-define i1 @issubnormal_or_zero_f(float %x) {
-; CHECK-32-LABEL: issubnormal_or_zero_f:
+define i1 @not_iszero_f(float %x) {
+; CHECK-32-LABEL: not_iszero_f:
 ; CHECK-32:       # %bb.0: # %entry
-; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
-; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
-; CHECK-32-NEXT:    sete %cl
-; CHECK-32-NEXT:    decl %eax
-; CHECK-32-NEXT:    cmpl $8388607, %eax # imm = 0x7FFFFF
-; CHECK-32-NEXT:    setb %al
+; CHECK-32-NEXT:    flds {{[0-9]+}}(%esp)
+; CHECK-32-NEXT:    fldz
+; CHECK-32-NEXT:    fucompp
+; CHECK-32-NEXT:    fnstsw %ax
+; CHECK-32-NEXT:    # kill: def $ah killed $ah killed $ax
+; CHECK-32-NEXT:    sahf
+; CHECK-32-NEXT:    setp %cl
+; CHECK-32-NEXT:    setne %al
 ; CHECK-32-NEXT:    orb %cl, %al
 ; CHECK-32-NEXT:    retl
 ;
-; CHECK-64-LABEL: issubnormal_or_zero_f:
+; CHECK-64-LABEL: not_iszero_f:
 ; CHECK-64:       # %bb.0: # %entry
-; CHECK-64-NEXT:    movd %xmm0, %eax
-; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
-; CHECK-64-NEXT:    sete %cl
-; CHECK-64-NEXT:    decl %eax
-; CHECK-64-NEXT:    cmpl $8388607, %eax # imm = 0x7FFFFF
-; CHECK-64-NEXT:    setb %al
-; CHECK-64-NEXT:    orb %cl, %al
+; CHECK-64-NEXT:    xorps %xmm1, %xmm1
+; CHECK-64-NEXT:    cmpneqss %xmm0, %xmm1
+; CHECK-64-NEXT:    movd %xmm1, %eax
+; CHECK-64-NEXT:    andl $1, %eax
+; CHECK-64-NEXT:    # kill: def $al killed $al killed $eax
 ; CHECK-64-NEXT:    retq
 entry:
-  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 240)  ; 0xf0 = "subnormal|zero"
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 927)  ; ~0x60 = "~zero"
   ret i1 %0
 }
 
-define i1 @issubnormal_or_zero_f_daz(float %x) #0 {
-; CHECK-32-LABEL: issubnormal_or_zero_f_daz:
+define i1 @not_iszero_f_daz(float %x) #0 {
+; CHECK-32-LABEL: not_iszero_f_daz:
 ; CHECK-32:       # %bb.0: # %entry
-; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
-; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    testl $2147483647, {{[0-9]+}}(%esp) # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    setne %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_iszero_f_daz:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    testl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    setne %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 927)  ; ~0x60 = "~zero"
+  ret i1 %0
+}
+
+define i1 @not_iszero_f_maybe_daz(float %x) #1 {
+; CHECK-32-LABEL: not_iszero_f_maybe_daz:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    testl $2147483647, {{[0-9]+}}(%esp) # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    setne %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_iszero_f_maybe_daz:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    testl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    setne %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 927)  ; ~0x60 = "~zero"
+  ret i1 %0
+}
+
+define i1 @issubnormal_or_zero_f(float %x) {
+; CHECK-32-LABEL: issubnormal_or_zero_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    sete %cl
+; CHECK-32-NEXT:    decl %eax
+; CHECK-32-NEXT:    cmpl $8388607, %eax # imm = 0x7FFFFF
+; CHECK-32-NEXT:    setb %al
+; CHECK-32-NEXT:    orb %cl, %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: issubnormal_or_zero_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    sete %cl
+; CHECK-64-NEXT:    decl %eax
+; CHECK-64-NEXT:    cmpl $8388607, %eax # imm = 0x7FFFFF
+; CHECK-64-NEXT:    setb %al
+; CHECK-64-NEXT:    orb %cl, %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 240)  ; 0xf0 = "subnormal|zero"
+  ret i1 %0
+}
+
+define i1 @issubnormal_or_zero_f_daz(float %x) #0 {
+; CHECK-32-LABEL: issubnormal_or_zero_f_daz:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
 ; CHECK-32-NEXT:    sete %cl
 ; CHECK-32-NEXT:    decl %eax
 ; CHECK-32-NEXT:    cmpl $8388607, %eax # imm = 0x7FFFFF
@@ -530,6 +888,93 @@ entry:
   ret i1 %0
 }
 
+define i1 @not_issubnormal_or_zero_f(float %x) {
+; CHECK-32-LABEL: not_issubnormal_or_zero_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-32-NEXT:    setge %cl
+; CHECK-32-NEXT:    addl $-8388608, %eax # imm = 0xFF800000
+; CHECK-32-NEXT:    cmpl $2130706432, %eax # imm = 0x7F000000
+; CHECK-32-NEXT:    setb %al
+; CHECK-32-NEXT:    orb %cl, %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_issubnormal_or_zero_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-64-NEXT:    setge %cl
+; CHECK-64-NEXT:    addl $-8388608, %eax # imm = 0xFF800000
+; CHECK-64-NEXT:    cmpl $2130706432, %eax # imm = 0x7F000000
+; CHECK-64-NEXT:    setb %al
+; CHECK-64-NEXT:    orb %cl, %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 783)  ; ~0xf0 = "~(subnormal|zero)"
+  ret i1 %0
+}
+
+define i1 @not_issubnormal_or_zero_f_daz(float %x) #0 {
+; CHECK-32-LABEL: not_issubnormal_or_zero_f_daz:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-32-NEXT:    setge %cl
+; CHECK-32-NEXT:    addl $-8388608, %eax # imm = 0xFF800000
+; CHECK-32-NEXT:    cmpl $2130706432, %eax # imm = 0x7F000000
+; CHECK-32-NEXT:    setb %al
+; CHECK-32-NEXT:    orb %cl, %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_issubnormal_or_zero_f_daz:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-64-NEXT:    setge %cl
+; CHECK-64-NEXT:    addl $-8388608, %eax # imm = 0xFF800000
+; CHECK-64-NEXT:    cmpl $2130706432, %eax # imm = 0x7F000000
+; CHECK-64-NEXT:    setb %al
+; CHECK-64-NEXT:    orb %cl, %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 783)  ; ~0xf0 = "~(subnormal|zero)"
+  ret i1 %0
+}
+
+define i1 @not_issubnormal_or_zero_f_maybe_daz(float %x) #1 {
+; CHECK-32-LABEL: not_issubnormal_or_zero_f_maybe_daz:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-32-NEXT:    setge %cl
+; CHECK-32-NEXT:    addl $-8388608, %eax # imm = 0xFF800000
+; CHECK-32-NEXT:    cmpl $2130706432, %eax # imm = 0x7F000000
+; CHECK-32-NEXT:    setb %al
+; CHECK-32-NEXT:    orb %cl, %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_issubnormal_or_zero_f_maybe_daz:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-64-NEXT:    setge %cl
+; CHECK-64-NEXT:    addl $-8388608, %eax # imm = 0xFF800000
+; CHECK-64-NEXT:    cmpl $2130706432, %eax # imm = 0x7F000000
+; CHECK-64-NEXT:    setb %al
+; CHECK-64-NEXT:    orb %cl, %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 783)  ; ~0xf0 = "~(subnormal|zero)"
+  ret i1 %0
+}
+
 define i1 @is_plus_zero_f(float %x) {
 ; CHECK-32-LABEL: is_plus_zero_f:
 ; CHECK-32:       # %bb.0: # %entry
@@ -548,6 +993,24 @@ entry:
   ret i1 %0
 }
 
+define i1 @not_is_plus_zero_f(float %x) {
+; CHECK-32-LABEL: not_is_plus_zero_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    cmpl $0, {{[0-9]+}}(%esp)
+; CHECK-32-NEXT:    setne %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_is_plus_zero_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    testl %eax, %eax
+; CHECK-64-NEXT:    setne %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 959)  ; ~0x40 = ~"+zero"
+  ret i1 %0
+}
+
 define i1 @is_minus_zero_f(float %x) {
 ; CHECK-32-LABEL: is_minus_zero_f:
 ; CHECK-32:       # %bb.0: # %entry
@@ -566,7 +1029,23 @@ entry:
   ret i1 %0
 }
 
-
+define i1 @not_is_minus_zero_f(float %x) {
+; CHECK-32-LABEL: not_is_minus_zero_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    cmpl $-2147483648, {{[0-9]+}}(%esp) # imm = 0x80000000
+; CHECK-32-NEXT:    setne %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_is_minus_zero_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    cmpl $-2147483648, %eax # imm = 0x80000000
+; CHECK-64-NEXT:    setne %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 991)  ; ~0x20 = ~"-zero"
+  ret i1 %0
+}
 
 define i1 @isnan_f_strictfp(float %x) strictfp {
 ; CHECK-32-LABEL: isnan_f_strictfp:
@@ -589,6 +1068,27 @@ entry:
   ret i1 %0
 }
 
+define i1 @not_isnan_f_strictfp(float %x) strictfp {
+; CHECK-32-LABEL: not_isnan_f_strictfp:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2139095041, %eax # imm = 0x7F800001
+; CHECK-32-NEXT:    setl %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_isnan_f_strictfp:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095041, %eax # imm = 0x7F800001
+; CHECK-64-NEXT:    setl %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 1020)  ; ~"nan"
+  ret i1 %0
+}
+
 define i1 @isfinite_f_strictfp(float %x) strictfp {
 ; CHECK-32-LABEL: isfinite_f_strictfp:
 ; CHECK-32:       # %bb.0: # %entry
@@ -610,6 +1110,27 @@ entry:
   ret i1 %0
 }
 
+define i1 @not_isfinite_f_strictfp(float %x) strictfp {
+; CHECK-32-LABEL: not_isfinite_f_strictfp:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-32-NEXT:    setge %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_isfinite_f_strictfp:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-64-NEXT:    setge %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 519)  ; ~0x1f8 = ~"finite"
+  ret i1 %0
+}
+
 define i1 @iszero_f_strictfp(float %x) strictfp {
 ; CHECK-32-LABEL: iszero_f_strictfp:
 ; CHECK-32:       # %bb.0: # %entry
@@ -628,6 +1149,23 @@ entry:
   ret i1 %0
 }
 
+define i1 @not_iszero_f_strictfp(float %x) strictfp {
+; CHECK-32-LABEL: not_iszero_f_strictfp:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    testl $2147483647, {{[0-9]+}}(%esp) # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    setne %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_iszero_f_strictfp:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    testl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    setne %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 927)  ; ~0x60 = ~"zero"
+  ret i1 %0
+}
 
 define i1 @isnan_d(double %x) {
 ; CHECK-32-LABEL: isnan_d:
@@ -1129,6 +1667,438 @@ entry:
 }
 
 
+define i1 @iszero_or_nan_f(float %x) {
+; CHECK-32-LABEL: iszero_or_nan_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2139095041, %eax # imm = 0x7F800001
+; CHECK-32-NEXT:    setge %cl
+; CHECK-32-NEXT:    testl %eax, %eax
+; CHECK-32-NEXT:    sete %al
+; CHECK-32-NEXT:    orb %cl, %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: iszero_or_nan_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095041, %eax # imm = 0x7F800001
+; CHECK-64-NEXT:    setge %cl
+; CHECK-64-NEXT:    testl %eax, %eax
+; CHECK-64-NEXT:    sete %al
+; CHECK-64-NEXT:    orb %cl, %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 99)  ; 0x60|0x3 = "zero|nan"
+  ret i1 %0
+}
+
+define i1 @iszero_or_nan_f_daz(float %x) #0 {
+; CHECK-32-LABEL: iszero_or_nan_f_daz:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2139095041, %eax # imm = 0x7F800001
+; CHECK-32-NEXT:    setge %cl
+; CHECK-32-NEXT:    testl %eax, %eax
+; CHECK-32-NEXT:    sete %al
+; CHECK-32-NEXT:    orb %cl, %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: iszero_or_nan_f_daz:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095041, %eax # imm = 0x7F800001
+; CHECK-64-NEXT:    setge %cl
+; CHECK-64-NEXT:    testl %eax, %eax
+; CHECK-64-NEXT:    sete %al
+; CHECK-64-NEXT:    orb %cl, %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 99)  ; 0x60|0x3 = "zero|nan"
+  ret i1 %0
+}
+
+define i1 @iszero_or_nan_f_maybe_daz(float %x) #1 {
+; CHECK-32-LABEL: iszero_or_nan_f_maybe_daz:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2139095041, %eax # imm = 0x7F800001
+; CHECK-32-NEXT:    setge %cl
+; CHECK-32-NEXT:    testl %eax, %eax
+; CHECK-32-NEXT:    sete %al
+; CHECK-32-NEXT:    orb %cl, %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: iszero_or_nan_f_maybe_daz:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095041, %eax # imm = 0x7F800001
+; CHECK-64-NEXT:    setge %cl
+; CHECK-64-NEXT:    testl %eax, %eax
+; CHECK-64-NEXT:    sete %al
+; CHECK-64-NEXT:    orb %cl, %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 99)  ; 0x60|0x3 = "zero|nan"
+  ret i1 %0
+}
+
+define i1 @not_iszero_or_nan_f(float %x) {
+; CHECK-32-LABEL: not_iszero_or_nan_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-32-NEXT:    sete %cl
+; CHECK-32-NEXT:    leal -1(%eax), %edx
+; CHECK-32-NEXT:    cmpl $8388607, %edx # imm = 0x7FFFFF
+; CHECK-32-NEXT:    setb %dl
+; CHECK-32-NEXT:    orb %cl, %dl
+; CHECK-32-NEXT:    addl $-8388608, %eax # imm = 0xFF800000
+; CHECK-32-NEXT:    cmpl $2130706432, %eax # imm = 0x7F000000
+; CHECK-32-NEXT:    setb %al
+; CHECK-32-NEXT:    orb %dl, %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_iszero_or_nan_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-64-NEXT:    sete %cl
+; CHECK-64-NEXT:    leal -1(%rax), %edx
+; CHECK-64-NEXT:    cmpl $8388607, %edx # imm = 0x7FFFFF
+; CHECK-64-NEXT:    setb %dl
+; CHECK-64-NEXT:    orb %cl, %dl
+; CHECK-64-NEXT:    addl $-8388608, %eax # imm = 0xFF800000
+; CHECK-64-NEXT:    cmpl $2130706432, %eax # imm = 0x7F000000
+; CHECK-64-NEXT:    setb %al
+; CHECK-64-NEXT:    orb %dl, %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 924)  ; ~0x60 = "~(zero|nan)"
+  ret i1 %0
+}
+
+define i1 @not_iszero_or_nan_f_daz(float %x) #0 {
+; CHECK-32-LABEL: not_iszero_or_nan_f_daz:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-32-NEXT:    sete %cl
+; CHECK-32-NEXT:    leal -1(%eax), %edx
+; CHECK-32-NEXT:    cmpl $8388607, %edx # imm = 0x7FFFFF
+; CHECK-32-NEXT:    setb %dl
+; CHECK-32-NEXT:    orb %cl, %dl
+; CHECK-32-NEXT:    addl $-8388608, %eax # imm = 0xFF800000
+; CHECK-32-NEXT:    cmpl $2130706432, %eax # imm = 0x7F000000
+; CHECK-32-NEXT:    setb %al
+; CHECK-32-NEXT:    orb %dl, %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_iszero_or_nan_f_daz:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-64-NEXT:    sete %cl
+; CHECK-64-NEXT:    leal -1(%rax), %edx
+; CHECK-64-NEXT:    cmpl $8388607, %edx # imm = 0x7FFFFF
+; CHECK-64-NEXT:    setb %dl
+; CHECK-64-NEXT:    orb %cl, %dl
+; CHECK-64-NEXT:    addl $-8388608, %eax # imm = 0xFF800000
+; CHECK-64-NEXT:    cmpl $2130706432, %eax # imm = 0x7F000000
+; CHECK-64-NEXT:    setb %al
+; CHECK-64-NEXT:    orb %dl, %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 924)  ; ~(0x60|0x3) = "~(zero|nan)"
+  ret i1 %0
+}
+
+define i1 @not_iszero_or_nan_f_maybe_daz(float %x) #1 {
+; CHECK-32-LABEL: not_iszero_or_nan_f_maybe_daz:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-32-NEXT:    sete %cl
+; CHECK-32-NEXT:    leal -1(%eax), %edx
+; CHECK-32-NEXT:    cmpl $8388607, %edx # imm = 0x7FFFFF
+; CHECK-32-NEXT:    setb %dl
+; CHECK-32-NEXT:    orb %cl, %dl
+; CHECK-32-NEXT:    addl $-8388608, %eax # imm = 0xFF800000
+; CHECK-32-NEXT:    cmpl $2130706432, %eax # imm = 0x7F000000
+; CHECK-32-NEXT:    setb %al
+; CHECK-32-NEXT:    orb %dl, %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_iszero_or_nan_f_maybe_daz:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-64-NEXT:    sete %cl
+; CHECK-64-NEXT:    leal -1(%rax), %edx
+; CHECK-64-NEXT:    cmpl $8388607, %edx # imm = 0x7FFFFF
+; CHECK-64-NEXT:    setb %dl
+; CHECK-64-NEXT:    orb %cl, %dl
+; CHECK-64-NEXT:    addl $-8388608, %eax # imm = 0xFF800000
+; CHECK-64-NEXT:    cmpl $2130706432, %eax # imm = 0x7F000000
+; CHECK-64-NEXT:    setb %al
+; CHECK-64-NEXT:    orb %dl, %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 924)  ; ~(0x60|0x3) = "~(zero|nan)"
+  ret i1 %0
+}
+
+define i1 @iszero_or_qnan_f(float %x) {
+; CHECK-32-LABEL: iszero_or_qnan_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2143289344, %eax # imm = 0x7FC00000
+; CHECK-32-NEXT:    setge %cl
+; CHECK-32-NEXT:    testl %eax, %eax
+; CHECK-32-NEXT:    sete %al
+; CHECK-32-NEXT:    orb %cl, %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: iszero_or_qnan_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2143289344, %eax # imm = 0x7FC00000
+; CHECK-64-NEXT:    setge %cl
+; CHECK-64-NEXT:    testl %eax, %eax
+; CHECK-64-NEXT:    sete %al
+; CHECK-64-NEXT:    orb %cl, %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 98)  ; 0x60|0x2 = "zero|qnan"
+  ret i1 %0
+}
+
+define i1 @iszero_or_snan_f(float %x) {
+; CHECK-32-LABEL: iszero_or_snan_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2143289344, %eax # imm = 0x7FC00000
+; CHECK-32-NEXT:    setl %cl
+; CHECK-32-NEXT:    cmpl $2139095041, %eax # imm = 0x7F800001
+; CHECK-32-NEXT:    setge %dl
+; CHECK-32-NEXT:    andb %cl, %dl
+; CHECK-32-NEXT:    testl %eax, %eax
+; CHECK-32-NEXT:    sete %al
+; CHECK-32-NEXT:    orb %dl, %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: iszero_or_snan_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2143289344, %eax # imm = 0x7FC00000
+; CHECK-64-NEXT:    setl %cl
+; CHECK-64-NEXT:    cmpl $2139095041, %eax # imm = 0x7F800001
+; CHECK-64-NEXT:    setge %dl
+; CHECK-64-NEXT:    andb %cl, %dl
+; CHECK-64-NEXT:    testl %eax, %eax
+; CHECK-64-NEXT:    sete %al
+; CHECK-64-NEXT:    orb %dl, %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 97)  ; 0x60|0x1 = "zero|snan"
+  ret i1 %0
+}
+
+define i1 @not_iszero_or_qnan_f(float %x) {
+; CHECK-32-LABEL: not_iszero_or_qnan_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    pushl %esi
+; CHECK-32-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-32-NEXT:    .cfi_offset %esi, -8
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2143289344, %eax # imm = 0x7FC00000
+; CHECK-32-NEXT:    setl %cl
+; CHECK-32-NEXT:    cmpl $2139095041, %eax # imm = 0x7F800001
+; CHECK-32-NEXT:    setge %dl
+; CHECK-32-NEXT:    andb %cl, %dl
+; CHECK-32-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-32-NEXT:    sete %cl
+; CHECK-32-NEXT:    leal -1(%eax), %esi
+; CHECK-32-NEXT:    cmpl $8388607, %esi # imm = 0x7FFFFF
+; CHECK-32-NEXT:    setb %ch
+; CHECK-32-NEXT:    orb %cl, %ch
+; CHECK-32-NEXT:    addl $-8388608, %eax # imm = 0xFF800000
+; CHECK-32-NEXT:    cmpl $2130706432, %eax # imm = 0x7F000000
+; CHECK-32-NEXT:    setb %al
+; CHECK-32-NEXT:    orb %dl, %al
+; CHECK-32-NEXT:    orb %ch, %al
+; CHECK-32-NEXT:    popl %esi
+; CHECK-32-NEXT:    .cfi_def_cfa_offset 4
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_iszero_or_qnan_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2143289344, %eax # imm = 0x7FC00000
+; CHECK-64-NEXT:    setl %cl
+; CHECK-64-NEXT:    cmpl $2139095041, %eax # imm = 0x7F800001
+; CHECK-64-NEXT:    setge %dl
+; CHECK-64-NEXT:    andb %cl, %dl
+; CHECK-64-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-64-NEXT:    sete %cl
+; CHECK-64-NEXT:    leal -1(%rax), %esi
+; CHECK-64-NEXT:    cmpl $8388607, %esi # imm = 0x7FFFFF
+; CHECK-64-NEXT:    setb %sil
+; CHECK-64-NEXT:    orb %cl, %sil
+; CHECK-64-NEXT:    addl $-8388608, %eax # imm = 0xFF800000
+; CHECK-64-NEXT:    cmpl $2130706432, %eax # imm = 0x7F000000
+; CHECK-64-NEXT:    setb %al
+; CHECK-64-NEXT:    orb %dl, %al
+; CHECK-64-NEXT:    orb %sil, %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 925)  ; ~(0x60|0x2) = "~(zero|qnan)"
+  ret i1 %0
+}
+
+define i1 @not_iszero_or_snan_f(float %x) {
+; CHECK-32-LABEL: not_iszero_or_snan_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-32-NEXT:    sete %cl
+; CHECK-32-NEXT:    leal -1(%eax), %edx
+; CHECK-32-NEXT:    cmpl $8388607, %edx # imm = 0x7FFFFF
+; CHECK-32-NEXT:    setb %dl
+; CHECK-32-NEXT:    cmpl $2143289344, %eax # imm = 0x7FC00000
+; CHECK-32-NEXT:    setge %ch
+; CHECK-32-NEXT:    orb %cl, %ch
+; CHECK-32-NEXT:    orb %dl, %ch
+; CHECK-32-NEXT:    addl $-8388608, %eax # imm = 0xFF800000
+; CHECK-32-NEXT:    cmpl $2130706432, %eax # imm = 0x7F000000
+; CHECK-32-NEXT:    setb %al
+; CHECK-32-NEXT:    orb %ch, %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_iszero_or_snan_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-64-NEXT:    sete %cl
+; CHECK-64-NEXT:    leal -1(%rax), %edx
+; CHECK-64-NEXT:    cmpl $8388607, %edx # imm = 0x7FFFFF
+; CHECK-64-NEXT:    setb %dl
+; CHECK-64-NEXT:    cmpl $2143289344, %eax # imm = 0x7FC00000
+; CHECK-64-NEXT:    setge %sil
+; CHECK-64-NEXT:    orb %cl, %sil
+; CHECK-64-NEXT:    orb %dl, %sil
+; CHECK-64-NEXT:    addl $-8388608, %eax # imm = 0xFF800000
+; CHECK-64-NEXT:    cmpl $2130706432, %eax # imm = 0x7F000000
+; CHECK-64-NEXT:    setb %al
+; CHECK-64-NEXT:    orb %sil, %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 926)  ; ~(0x60|0x1) = "~(zero|snan)"
+  ret i1 %0
+}
+
+define i1 @isinf_or_nan_f(float %x) {
+; CHECK-32-LABEL: isinf_or_nan_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-32-NEXT:    setge %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: isinf_or_nan_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-64-NEXT:    setge %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 519)  ; 0x204|0x3 = "inf|nan"
+  ret i1 %0
+}
+
+define i1 @not_isinf_or_nan_f(float %x) {
+; CHECK-32-LABEL: not_isinf_or_nan_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-32-NEXT:    setl %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_isinf_or_nan_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-64-NEXT:    setl %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 504)  ; ~(0x204|0x3) = "~(inf|nan)"
+  ret i1 %0
+}
+
+define i1 @isfinite_or_nan_f(float %x) {
+; CHECK-32-LABEL: isfinite_or_nan_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-32-NEXT:    setne %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: isfinite_or_nan_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-64-NEXT:    setne %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 507)  ; 0x1f8|0x3 = "finite|nan"
+  ret i1 %0
+}
+
+define i1 @not_isfinite_or_nan_f(float %x) {
+; CHECK-32-LABEL: not_isfinite_or_nan_f:
+; CHECK-32:       # %bb.0: # %entry
+; CHECK-32-NEXT:    movl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-32-NEXT:    andl {{[0-9]+}}(%esp), %eax
+; CHECK-32-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-32-NEXT:    sete %al
+; CHECK-32-NEXT:    retl
+;
+; CHECK-64-LABEL: not_isfinite_or_nan_f:
+; CHECK-64:       # %bb.0: # %entry
+; CHECK-64-NEXT:    movd %xmm0, %eax
+; CHECK-64-NEXT:    andl $2147483647, %eax # imm = 0x7FFFFFFF
+; CHECK-64-NEXT:    cmpl $2139095040, %eax # imm = 0x7F800000
+; CHECK-64-NEXT:    sete %al
+; CHECK-64-NEXT:    retq
+entry:
+  %0 = tail call i1 @llvm.is.fpclass.f32(float %x, i32 516)  ; ~(0x1f8|0x3) = "~(finite|nan)"
+  ret i1 %0
+}
 
 declare i1 @llvm.is.fpclass.f32(float, i32)
 declare i1 @llvm.is.fpclass.f64(double, i32)
diff --git a/llvm/test/CodeGen/X86/lea-dagdag.ll b/llvm/test/CodeGen/X86/lea-dagdag.ll
index 4c53f569b0d2c..2705bd00f5d2c 100644
--- a/llvm/test/CodeGen/X86/lea-dagdag.ll
+++ b/llvm/test/CodeGen/X86/lea-dagdag.ll
@@ -150,6 +150,48 @@ define i64 @and_i32_shl_zext_add_i64(i64 %t0, i32 %t1) {
   ret i64 %t6
 }
 
+define i64 @shl_and_i8_zext_add_i64(i64 %t0, i8 %t1) {
+; CHECK-LABEL: shl_and_i8_zext_add_i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    andb $15, %sil
+; CHECK-NEXT:    movzbl %sil, %eax
+; CHECK-NEXT:    leaq (%rdi,%rax,4), %rax
+; CHECK-NEXT:    retq
+  %s = shl i8 %t1, 2
+  %m = and i8 %s, 60
+  %z = zext i8 %m to i64
+  %a = add i64 %t0, %z
+  ret i64 %a
+}
+
+define i64 @shl_and_i16_zext_add_i64(i64 %t0, i16 %t1) {
+; CHECK-LABEL: shl_and_i16_zext_add_i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    # kill: def $esi killed $esi def $rsi
+; CHECK-NEXT:    andl $8, %esi
+; CHECK-NEXT:    leaq (%rdi,%rsi,2), %rax
+; CHECK-NEXT:    retq
+  %s = shl i16 %t1, 1
+  %m = and i16 %s, 17
+  %z = zext i16 %m to i64
+  %a = add i64 %t0, %z
+  ret i64 %a
+}
+
+define i64 @shl_and_i32_zext_add_i64(i64 %t0, i32 %t1) {
+; CHECK-LABEL: shl_and_i32_zext_add_i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    # kill: def $esi killed $esi def $rsi
+; CHECK-NEXT:    andl $749, %esi # imm = 0x2ED
+; CHECK-NEXT:    leaq (%rdi,%rsi,8), %rax
+; CHECK-NEXT:    retq
+  %s = shl i32 %t1, 3
+  %m = and i32 %s, 5999
+  %z = zext i32 %m to i64
+  %a = add i64 %t0, %z
+  ret i64 %a
+}
+
 ; Negative test - shift can't be converted to scale factor.
 
 define i64 @and_i32_zext_shl_add_i64_overshift(i64 %t0, i32 %t1) {
diff --git a/llvm/test/CodeGen/X86/lzcnt-cmp.ll b/llvm/test/CodeGen/X86/lzcnt-cmp.ll
index 01d6c0ed483b7..b2c1ffd799e31 100644
--- a/llvm/test/CodeGen/X86/lzcnt-cmp.ll
+++ b/llvm/test/CodeGen/X86/lzcnt-cmp.ll
@@ -1,6 +1,6 @@
 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i686-- | FileCheck %s --check-prefixes=X86,X86-BSR
-; RUN: llc < %s -mtriple=i686-- -mattr=+lzcnt | FileCheck %s --check-prefixes=X86,X86-LZCNT
+; RUN: llc < %s -mtriple=i686-- -mattr=+lzcnt,+cmov | FileCheck %s --check-prefixes=X86,X86-LZCNT
 ; RUN: llc < %s -mtriple=x86_64-- | FileCheck %s --check-prefixes=X64,X64-BSR
 ; RUN: llc < %s -mtriple=x86_64-- -mattr=+lzcnt | FileCheck %s --check-prefixes=X64,X64-LZCNT
 
@@ -56,14 +56,12 @@ define i1 @lshr_ctlz_undef_cmpeq_one_i64(i64 %in) nounwind {
 ;
 ; X86-LZCNT-LABEL: lshr_ctlz_undef_cmpeq_one_i64:
 ; X86-LZCNT:       # %bb.0:
-; X86-LZCNT-NEXT:    xorl %eax, %eax
-; X86-LZCNT-NEXT:    cmpl $0, {{[0-9]+}}(%esp)
-; X86-LZCNT-NEXT:    jne .LBB1_2
-; X86-LZCNT-NEXT:  # %bb.1:
 ; X86-LZCNT-NEXT:    lzcntl {{[0-9]+}}(%esp), %eax
 ; X86-LZCNT-NEXT:    addl $32, %eax
-; X86-LZCNT-NEXT:  .LBB1_2:
-; X86-LZCNT-NEXT:    testb $64, %al
+; X86-LZCNT-NEXT:    xorl %ecx, %ecx
+; X86-LZCNT-NEXT:    cmpl $0, {{[0-9]+}}(%esp)
+; X86-LZCNT-NEXT:    cmovel %eax, %ecx
+; X86-LZCNT-NEXT:    testb $64, %cl
 ; X86-LZCNT-NEXT:    setne %al
 ; X86-LZCNT-NEXT:    retl
 ;
@@ -139,14 +137,12 @@ define i1 @lshr_ctlz_undef_cmpne_zero_i64(i64 %in) nounwind {
 ;
 ; X86-LZCNT-LABEL: lshr_ctlz_undef_cmpne_zero_i64:
 ; X86-LZCNT:       # %bb.0:
-; X86-LZCNT-NEXT:    xorl %eax, %eax
-; X86-LZCNT-NEXT:    cmpl $0, {{[0-9]+}}(%esp)
-; X86-LZCNT-NEXT:    jne .LBB3_2
-; X86-LZCNT-NEXT:  # %bb.1:
 ; X86-LZCNT-NEXT:    lzcntl {{[0-9]+}}(%esp), %eax
 ; X86-LZCNT-NEXT:    addl $32, %eax
-; X86-LZCNT-NEXT:  .LBB3_2:
-; X86-LZCNT-NEXT:    testb $64, %al
+; X86-LZCNT-NEXT:    xorl %ecx, %ecx
+; X86-LZCNT-NEXT:    cmpl $0, {{[0-9]+}}(%esp)
+; X86-LZCNT-NEXT:    cmovel %eax, %ecx
+; X86-LZCNT-NEXT:    testb $64, %cl
 ; X86-LZCNT-NEXT:    setne %al
 ; X86-LZCNT-NEXT:    retl
 ;
diff --git a/llvm/test/CodeGen/X86/machine-combiner-dbg.mir b/llvm/test/CodeGen/X86/machine-combiner-dbg.mir
new file mode 100644
index 0000000000000..6817d7e0ef00a
--- /dev/null
+++ b/llvm/test/CodeGen/X86/machine-combiner-dbg.mir
@@ -0,0 +1,58 @@
+# RUN: llc -mtriple=x86_64-gnu-linux -run-pass=machine-combiner %s -o - | FileCheck %s
+
+--- |
+  define float @reassoc_me(float %f1, float %f2, float %f3, float %f4) !dbg !4 {
+    %add = fadd reassoc nsz float %f1, %f2, !dbg !10
+    %add1 = fadd reassoc nsz float %add, %f3, !dbg !10
+    %add2 = fadd reassoc nsz float %add1, %f4, !dbg !10
+    call void @llvm.dbg.value(metadata float %add2, metadata !9, metadata !DIExpression()), !dbg !10
+    ret float %add2, !dbg !10
+  }
+
+  declare void @llvm.dbg.value(metadata, metadata, metadata)
+
+  !llvm.dbg.cu = !{!0}
+  !llvm.module.flags = !{!2, !3}
+
+  !0 = distinct !DICompileUnit(language: DW_LANG_C11, file: !1, producer: "hello", isOptimized: true, emissionKind: FullDebug)
+  !1 = !DIFile(filename: "reassoc.c", directory: "blah")
+  !2 = !{i32 7, !"Dwarf Version", i32 4}
+  !3 = !{i32 2, !"Debug Info Version", i32 3}
+  !4 = distinct !DISubprogram(name: "reassoc_me", scope: !1, file: !1, line: 1, type: !5, scopeLine: 1, unit: !0, retainedNodes: !8)
+  !5 = !DISubroutineType(types: !6)
+  !6 = !{!7, !7, !7, !7, !7}
+  !7 = !DIBasicType(name: "float", size: 32, encoding: DW_ATE_float)
+  !8 = !{!9}
+  !9 = !DILocalVariable(name: "temp3", scope: !4, file: !1, line: 4, type: !7)
+  !10 = !DILocation(line: 0, scope: !4)
+
+...
+---
+name:            reassoc_me
+alignment:       16
+debugInstrRef:   true
+body:             |
+  bb.0 (%ir-block.0):
+    liveins: $xmm0, $xmm1, $xmm2, $xmm3
+
+    %3:fr32 = COPY $xmm3
+    %2:fr32 = COPY $xmm2
+    %1:fr32 = COPY $xmm1
+    %0:fr32 = COPY $xmm0
+    %4:fr32 = nsz reassoc nofpexcept ADDSSrr %0, %1, implicit $mxcsr, debug-location !10
+    %5:fr32 = nsz reassoc nofpexcept ADDSSrr %4, %2, implicit $mxcsr, debug-location !10
+    %6:fr32 = nsz reassoc nofpexcept ADDSSrr %5, %3, implicit $mxcsr, debug-instr-number 1, debug-location !10
+    DBG_INSTR_REF !9, !DIExpression(DW_OP_LLVM_arg, 0), dbg-instr-ref(1, 0), debug-location !10
+    $xmm0 = COPY %6, debug-location !10
+    RET 0, $xmm0, debug-location !10
+
+  ; CHECK: ![[VAR:.*]] = !DILocalVariable(name: "temp3"
+  ; CHECK:      %[[arg3:.*]]:fr32 = COPY $xmm3
+  ; CHECK-NEXT: %[[arg2:.*]]:fr32 = COPY $xmm2
+  ; CHECK-NEXT: %[[arg1:.*]]:fr32 = COPY $xmm1
+  ; CHECK-NEXT: %[[arg0:.*]]:fr32 = COPY $xmm0
+  ; CHECK-NEXT: %[[add1:.*]]:fr32 = {{.*}} ADDSSrr %[[arg0]], %[[arg1]]
+  ; CHECK-NEXT: %[[add2:.*]]:fr32 = {{.*}} ADDSSrr %[[arg2]], %[[arg3]]
+  ; CHECK-NEXT:                            ADDSSrr %[[add1]], killed %[[add2]], {{.*}} debug-instr-number 1
+  ; CHECK-NEXT: DBG_INSTR_REF ![[VAR]], !DIExpression(DW_OP_LLVM_arg, 0), dbg-instr-ref(1, 0)
+...
diff --git a/llvm/test/CodeGen/X86/memset-inline.ll b/llvm/test/CodeGen/X86/memset-inline.ll
index 0679fb342ac3b..905a0ffda061f 100644
--- a/llvm/test/CodeGen/X86/memset-inline.ll
+++ b/llvm/test/CodeGen/X86/memset-inline.ll
@@ -44,7 +44,6 @@ define void @memset_4(ptr %a, i8 %value) nounwind {
 define void @memset_8(ptr %a, i8 %value) nounwind {
 ; GPR-LABEL: memset_8:
 ; GPR:       # %bb.0:
-; GPR-NEXT:    # kill: def $esi killed $esi def $rsi
 ; GPR-NEXT:    movzbl %sil, %eax
 ; GPR-NEXT:    movabsq $72340172838076673, %rcx # imm = 0x101010101010101
 ; GPR-NEXT:    imulq %rax, %rcx
@@ -57,7 +56,6 @@ define void @memset_8(ptr %a, i8 %value) nounwind {
 define void @memset_16(ptr %a, i8 %value) nounwind {
 ; SSE2-LABEL: memset_16:
 ; SSE2:       # %bb.0:
-; SSE2-NEXT:    # kill: def $esi killed $esi def $rsi
 ; SSE2-NEXT:    movzbl %sil, %eax
 ; SSE2-NEXT:    movabsq $72340172838076673, %rcx # imm = 0x101010101010101
 ; SSE2-NEXT:    imulq %rax, %rcx
@@ -94,7 +92,6 @@ define void @memset_16(ptr %a, i8 %value) nounwind {
 define void @memset_32(ptr %a, i8 %value) nounwind {
 ; SSE2-LABEL: memset_32:
 ; SSE2:       # %bb.0:
-; SSE2-NEXT:    # kill: def $esi killed $esi def $rsi
 ; SSE2-NEXT:    movzbl %sil, %eax
 ; SSE2-NEXT:    movabsq $72340172838076673, %rcx # imm = 0x101010101010101
 ; SSE2-NEXT:    imulq %rax, %rcx
@@ -136,7 +133,6 @@ define void @memset_32(ptr %a, i8 %value) nounwind {
 define void @memset_64(ptr %a, i8 %value) nounwind {
 ; SSE2-LABEL: memset_64:
 ; SSE2:       # %bb.0:
-; SSE2-NEXT:    # kill: def $esi killed $esi def $rsi
 ; SSE2-NEXT:    movzbl %sil, %eax
 ; SSE2-NEXT:    movabsq $72340172838076673, %rcx # imm = 0x101010101010101
 ; SSE2-NEXT:    imulq %rax, %rcx
diff --git a/llvm/test/CodeGen/X86/memset-nonzero.ll b/llvm/test/CodeGen/X86/memset-nonzero.ll
index 60d912382000c..5a0c703ae2ea4 100644
--- a/llvm/test/CodeGen/X86/memset-nonzero.ll
+++ b/llvm/test/CodeGen/X86/memset-nonzero.ll
@@ -279,7 +279,6 @@ declare ptr @__memset_chk(ptr, i32, i64, i64)
 define void @memset_16_nonconst_bytes(ptr %x, i8 %c) {
 ; SSE-LABEL: memset_16_nonconst_bytes:
 ; SSE:       # %bb.0:
-; SSE-NEXT:    # kill: def $esi killed $esi def $rsi
 ; SSE-NEXT:    movzbl %sil, %eax
 ; SSE-NEXT:    movabsq $72340172838076673, %rcx # imm = 0x101010101010101
 ; SSE-NEXT:    imulq %rax, %rcx
@@ -324,7 +323,6 @@ define void @memset_16_nonconst_bytes(ptr %x, i8 %c) {
 define void @memset_32_nonconst_bytes(ptr %x, i8 %c) {
 ; SSE-LABEL: memset_32_nonconst_bytes:
 ; SSE:       # %bb.0:
-; SSE-NEXT:    # kill: def $esi killed $esi def $rsi
 ; SSE-NEXT:    movzbl %sil, %eax
 ; SSE-NEXT:    movabsq $72340172838076673, %rcx # imm = 0x101010101010101
 ; SSE-NEXT:    imulq %rax, %rcx
@@ -375,7 +373,6 @@ define void @memset_32_nonconst_bytes(ptr %x, i8 %c) {
 define void @memset_64_nonconst_bytes(ptr %x, i8 %c) {
 ; SSE-LABEL: memset_64_nonconst_bytes:
 ; SSE:       # %bb.0:
-; SSE-NEXT:    # kill: def $esi killed $esi def $rsi
 ; SSE-NEXT:    movzbl %sil, %eax
 ; SSE-NEXT:    movabsq $72340172838076673, %rcx # imm = 0x101010101010101
 ; SSE-NEXT:    imulq %rax, %rcx
@@ -443,7 +440,6 @@ define void @memset_64_nonconst_bytes(ptr %x, i8 %c) {
 define void @memset_128_nonconst_bytes(ptr %x, i8 %c) {
 ; SSE-LABEL: memset_128_nonconst_bytes:
 ; SSE:       # %bb.0:
-; SSE-NEXT:    # kill: def $esi killed $esi def $rsi
 ; SSE-NEXT:    movzbl %sil, %eax
 ; SSE-NEXT:    movabsq $72340172838076673, %rcx # imm = 0x101010101010101
 ; SSE-NEXT:    imulq %rax, %rcx
diff --git a/llvm/test/CodeGen/X86/memset-vs-memset-inline.ll b/llvm/test/CodeGen/X86/memset-vs-memset-inline.ll
index d1d5c9f4a4d03..b8fdd936b4389 100644
--- a/llvm/test/CodeGen/X86/memset-vs-memset-inline.ll
+++ b/llvm/test/CodeGen/X86/memset-vs-memset-inline.ll
@@ -7,7 +7,6 @@ declare void @llvm.memset.inline.p0.i64(ptr nocapture, i8, i64, i1) nounwind
 define void @test1(ptr %a, i8 %value) nounwind {
 ; CHECK-LABEL: test1:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $esi killed $esi def $rsi
 ; CHECK-NEXT:    movzbl %sil, %eax
 ; CHECK-NEXT:    movabsq $72340172838076673, %rcx # imm = 0x101010101010101
 ; CHECK-NEXT:    imulq %rax, %rcx
@@ -29,12 +28,138 @@ define void @regular_memset_calls_external_function(ptr %a, i8 %value) nounwind
 define void @inlined_set_doesnt_call_external_function(ptr %a, i8 %value) nounwind {
 ; CHECK-LABEL: inlined_set_doesnt_call_external_function:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $esi killed $esi def $rsi
 ; CHECK-NEXT:    movzbl %sil, %ecx
 ; CHECK-NEXT:    movabsq $72340172838076673, %rax # imm = 0x101010101010101
 ; CHECK-NEXT:    imulq %rcx, %rax
 ; CHECK-NEXT:    movq %rax, 1016(%rdi)
 ; CHECK-NEXT:    movq %rax, 1008(%rdi)
+; CHECK-NEXT:    movq %rax, 1000(%rdi)
+; CHECK-NEXT:    movq %rax, 992(%rdi)
+; CHECK-NEXT:    movq %rax, 984(%rdi)
+; CHECK-NEXT:    movq %rax, 976(%rdi)
+; CHECK-NEXT:    movq %rax, 968(%rdi)
+; CHECK-NEXT:    movq %rax, 960(%rdi)
+; CHECK-NEXT:    movq %rax, 952(%rdi)
+; CHECK-NEXT:    movq %rax, 944(%rdi)
+; CHECK-NEXT:    movq %rax, 936(%rdi)
+; CHECK-NEXT:    movq %rax, 928(%rdi)
+; CHECK-NEXT:    movq %rax, 920(%rdi)
+; CHECK-NEXT:    movq %rax, 912(%rdi)
+; CHECK-NEXT:    movq %rax, 904(%rdi)
+; CHECK-NEXT:    movq %rax, 896(%rdi)
+; CHECK-NEXT:    movq %rax, 888(%rdi)
+; CHECK-NEXT:    movq %rax, 880(%rdi)
+; CHECK-NEXT:    movq %rax, 872(%rdi)
+; CHECK-NEXT:    movq %rax, 864(%rdi)
+; CHECK-NEXT:    movq %rax, 856(%rdi)
+; CHECK-NEXT:    movq %rax, 848(%rdi)
+; CHECK-NEXT:    movq %rax, 840(%rdi)
+; CHECK-NEXT:    movq %rax, 832(%rdi)
+; CHECK-NEXT:    movq %rax, 824(%rdi)
+; CHECK-NEXT:    movq %rax, 816(%rdi)
+; CHECK-NEXT:    movq %rax, 808(%rdi)
+; CHECK-NEXT:    movq %rax, 800(%rdi)
+; CHECK-NEXT:    movq %rax, 792(%rdi)
+; CHECK-NEXT:    movq %rax, 784(%rdi)
+; CHECK-NEXT:    movq %rax, 776(%rdi)
+; CHECK-NEXT:    movq %rax, 768(%rdi)
+; CHECK-NEXT:    movq %rax, 760(%rdi)
+; CHECK-NEXT:    movq %rax, 752(%rdi)
+; CHECK-NEXT:    movq %rax, 744(%rdi)
+; CHECK-NEXT:    movq %rax, 736(%rdi)
+; CHECK-NEXT:    movq %rax, 728(%rdi)
+; CHECK-NEXT:    movq %rax, 720(%rdi)
+; CHECK-NEXT:    movq %rax, 712(%rdi)
+; CHECK-NEXT:    movq %rax, 704(%rdi)
+; CHECK-NEXT:    movq %rax, 696(%rdi)
+; CHECK-NEXT:    movq %rax, 688(%rdi)
+; CHECK-NEXT:    movq %rax, 680(%rdi)
+; CHECK-NEXT:    movq %rax, 672(%rdi)
+; CHECK-NEXT:    movq %rax, 664(%rdi)
+; CHECK-NEXT:    movq %rax, 656(%rdi)
+; CHECK-NEXT:    movq %rax, 648(%rdi)
+; CHECK-NEXT:    movq %rax, 640(%rdi)
+; CHECK-NEXT:    movq %rax, 632(%rdi)
+; CHECK-NEXT:    movq %rax, 624(%rdi)
+; CHECK-NEXT:    movq %rax, 616(%rdi)
+; CHECK-NEXT:    movq %rax, 608(%rdi)
+; CHECK-NEXT:    movq %rax, 600(%rdi)
+; CHECK-NEXT:    movq %rax, 592(%rdi)
+; CHECK-NEXT:    movq %rax, 584(%rdi)
+; CHECK-NEXT:    movq %rax, 576(%rdi)
+; CHECK-NEXT:    movq %rax, 568(%rdi)
+; CHECK-NEXT:    movq %rax, 560(%rdi)
+; CHECK-NEXT:    movq %rax, 552(%rdi)
+; CHECK-NEXT:    movq %rax, 544(%rdi)
+; CHECK-NEXT:    movq %rax, 536(%rdi)
+; CHECK-NEXT:    movq %rax, 528(%rdi)
+; CHECK-NEXT:    movq %rax, 520(%rdi)
+; CHECK-NEXT:    movq %rax, 512(%rdi)
+; CHECK-NEXT:    movq %rax, 504(%rdi)
+; CHECK-NEXT:    movq %rax, 496(%rdi)
+; CHECK-NEXT:    movq %rax, 488(%rdi)
+; CHECK-NEXT:    movq %rax, 480(%rdi)
+; CHECK-NEXT:    movq %rax, 472(%rdi)
+; CHECK-NEXT:    movq %rax, 464(%rdi)
+; CHECK-NEXT:    movq %rax, 456(%rdi)
+; CHECK-NEXT:    movq %rax, 448(%rdi)
+; CHECK-NEXT:    movq %rax, 440(%rdi)
+; CHECK-NEXT:    movq %rax, 432(%rdi)
+; CHECK-NEXT:    movq %rax, 424(%rdi)
+; CHECK-NEXT:    movq %rax, 416(%rdi)
+; CHECK-NEXT:    movq %rax, 408(%rdi)
+; CHECK-NEXT:    movq %rax, 400(%rdi)
+; CHECK-NEXT:    movq %rax, 392(%rdi)
+; CHECK-NEXT:    movq %rax, 384(%rdi)
+; CHECK-NEXT:    movq %rax, 376(%rdi)
+; CHECK-NEXT:    movq %rax, 368(%rdi)
+; CHECK-NEXT:    movq %rax, 360(%rdi)
+; CHECK-NEXT:    movq %rax, 352(%rdi)
+; CHECK-NEXT:    movq %rax, 344(%rdi)
+; CHECK-NEXT:    movq %rax, 336(%rdi)
+; CHECK-NEXT:    movq %rax, 328(%rdi)
+; CHECK-NEXT:    movq %rax, 320(%rdi)
+; CHECK-NEXT:    movq %rax, 312(%rdi)
+; CHECK-NEXT:    movq %rax, 304(%rdi)
+; CHECK-NEXT:    movq %rax, 296(%rdi)
+; CHECK-NEXT:    movq %rax, 288(%rdi)
+; CHECK-NEXT:    movq %rax, 280(%rdi)
+; CHECK-NEXT:    movq %rax, 272(%rdi)
+; CHECK-NEXT:    movq %rax, 264(%rdi)
+; CHECK-NEXT:    movq %rax, 256(%rdi)
+; CHECK-NEXT:    movq %rax, 248(%rdi)
+; CHECK-NEXT:    movq %rax, 240(%rdi)
+; CHECK-NEXT:    movq %rax, 232(%rdi)
+; CHECK-NEXT:    movq %rax, 224(%rdi)
+; CHECK-NEXT:    movq %rax, 216(%rdi)
+; CHECK-NEXT:    movq %rax, 208(%rdi)
+; CHECK-NEXT:    movq %rax, 200(%rdi)
+; CHECK-NEXT:    movq %rax, 192(%rdi)
+; CHECK-NEXT:    movq %rax, 184(%rdi)
+; CHECK-NEXT:    movq %rax, 176(%rdi)
+; CHECK-NEXT:    movq %rax, 168(%rdi)
+; CHECK-NEXT:    movq %rax, 160(%rdi)
+; CHECK-NEXT:    movq %rax, 152(%rdi)
+; CHECK-NEXT:    movq %rax, 144(%rdi)
+; CHECK-NEXT:    movq %rax, 136(%rdi)
+; CHECK-NEXT:    movq %rax, 128(%rdi)
+; CHECK-NEXT:    movq %rax, 120(%rdi)
+; CHECK-NEXT:    movq %rax, 112(%rdi)
+; CHECK-NEXT:    movq %rax, 104(%rdi)
+; CHECK-NEXT:    movq %rax, 96(%rdi)
+; CHECK-NEXT:    movq %rax, 88(%rdi)
+; CHECK-NEXT:    movq %rax, 80(%rdi)
+; CHECK-NEXT:    movq %rax, 72(%rdi)
+; CHECK-NEXT:    movq %rax, 64(%rdi)
+; CHECK-NEXT:    movq %rax, 56(%rdi)
+; CHECK-NEXT:    movq %rax, 48(%rdi)
+; CHECK-NEXT:    movq %rax, 40(%rdi)
+; CHECK-NEXT:    movq %rax, 32(%rdi)
+; CHECK-NEXT:    movq %rax, 24(%rdi)
+; CHECK-NEXT:    movq %rax, 16(%rdi)
+; CHECK-NEXT:    movq %rax, 8(%rdi)
+; CHECK-NEXT:    movq %rax, (%rdi)
+; CHECK-NEXT:    retq
   tail call void @llvm.memset.inline.p0.i64(ptr %a, i8 %value, i64 1024, i1 0)
   ret void
 }
diff --git a/llvm/test/CodeGen/X86/merge-store-dependency.ll b/llvm/test/CodeGen/X86/merge-store-dependency.ll
new file mode 100644
index 0000000000000..1e1802b1d50b3
--- /dev/null
+++ b/llvm/test/CodeGen/X86/merge-store-dependency.ll
@@ -0,0 +1,48 @@
+; RUN: llc < %s -mtriple=i686-unknown-linux-android24 -verify-machineinstrs \
+; RUN:   -debug-only=machine-scheduler -o - 2>&1 | FileCheck %s
+; REQUIRES: asserts
+
+;; MOVUPSmr is a merged store from stack objects %ir.arg1, %ir.arg2, %ir.arg3,
+;; %ir.arg4.
+;; Check that the merged store has dependency with %ir.arg4.
+
+; CHECK:       ********** MI Scheduling **********
+; CHECK-LABEL: f:%bb.0 bb
+; CHECK:       SU([[ARG4:[0-9]+]]):{{.*}}MOV32rm{{.*}}load (s32) from %ir.arg4
+; CHECK:       SU([[#WIDEN:]]):{{.*}}MOVUPSmr{{.*}}store (s128) into
+; CHECK:         Predecessors:
+; CHECK:           SU([[ARG4]]):{{.*}}Memory
+; CHECK:       SU([[#WIDEN+1]])
+;
+
+define void @f(ptr %arg, ptr byval(ptr) %arg1, ptr byval(ptr) %arg2, ptr byval(ptr) %arg3, ptr byval(ptr) %arg4) #0 {
+bb:
+  %inst = alloca ptr, align 4
+  %inst5 = alloca ptr, align 4
+  %inst6 = alloca ptr, align 4
+  %inst7 = alloca ptr, align 4
+  %inst9 = load ptr, ptr %arg1, align 4
+  store ptr null, ptr %arg1, align 4
+  store ptr %inst9, ptr %inst, align 4
+  %inst10 = load ptr, ptr %arg2, align 4
+  store ptr null, ptr %arg2, align 4
+  store ptr %inst10, ptr %inst5, align 4
+  %inst11 = load ptr, ptr %arg3, align 4
+  store ptr null, ptr %arg3, align 4
+  store ptr %inst11, ptr %inst6, align 4
+  %inst12 = load ptr, ptr %arg4, align 4
+  store ptr null, ptr %arg4, align 4
+  store ptr %inst12, ptr %inst7, align 4
+  call void @g(ptr %arg, ptr byval(ptr) %inst, ptr byval(ptr) %inst5, ptr byval(ptr) %inst6, ptr byval(ptr) %inst7)
+  call void @h(ptr %arg4)
+  call void @h(ptr %arg3)
+  call void @h(ptr %arg2)
+  call void @h(ptr %arg1)
+  ret void
+}
+
+declare void @g(ptr, ptr, ptr, ptr, ptr)
+
+declare void @h(ptr)
+
+attributes #0 = { optsize "frame-pointer"="non-leaf" "target-cpu"="i686" "target-features"="+sse,+sse2" "tune-cpu"="generic" }
diff --git a/llvm/test/CodeGen/X86/mulvi32.ll b/llvm/test/CodeGen/X86/mulvi32.ll
index 226ec4f63a1b8..88bc356d56952 100644
--- a/llvm/test/CodeGen/X86/mulvi32.ll
+++ b/llvm/test/CodeGen/X86/mulvi32.ll
@@ -288,11 +288,11 @@ define <4 x i64> @_mul4xi32toi64c(<4 x i32>, <4 x i32>) {
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    vpmovzxdq {{.*#+}} xmm2 = xmm0[0],zero,xmm0[1],zero
 ; AVX2-NEXT:    vpmovzxdq {{.*#+}} xmm3 = xmm1[0],zero,xmm1[1],zero
-; AVX2-NEXT:    vpmuludq %xmm3, %xmm2, %xmm2
 ; AVX2-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[2,2,3,3]
 ; AVX2-NEXT:    vpshufd {{.*#+}} xmm1 = xmm1[2,2,3,3]
-; AVX2-NEXT:    vpmuludq %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vinserti128 $1, %xmm1, %ymm3, %ymm1
 ; AVX2-NEXT:    vinserti128 $1, %xmm0, %ymm2, %ymm0
+; AVX2-NEXT:    vpmuludq %ymm1, %ymm0, %ymm0
 ; AVX2-NEXT:    retq
   %lower0 = shufflevector <4 x i32> %0, <4 x i32> undef, <4 x i32> <i32 0, i32 undef, i32 1, i32 undef>
   %lower1 = shufflevector <4 x i32> %1, <4 x i32> undef, <4 x i32> <i32 0, i32 undef, i32 1, i32 undef>
diff --git a/llvm/test/CodeGen/X86/narrow-shl-cst.ll b/llvm/test/CodeGen/X86/narrow-shl-cst.ll
index ed87be0fe532f..296ef52c3bff9 100644
--- a/llvm/test/CodeGen/X86/narrow-shl-cst.ll
+++ b/llvm/test/CodeGen/X86/narrow-shl-cst.ll
@@ -212,7 +212,7 @@ define i64 @test18(i64 %x) nounwind {
 ; CHECK-LABEL: test18:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    movzbl %dil, %eax
-; CHECK-NEXT:    shlq $10, %rax
+; CHECK-NEXT:    shll $10, %eax
 ; CHECK-NEXT:    retq
   %and = shl i64 %x, 10
   %shl = and i64 %and, 261120
@@ -234,7 +234,7 @@ define i64 @test20(i64 %x) nounwind {
 ; CHECK-LABEL: test20:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    movzwl %di, %eax
-; CHECK-NEXT:    shlq $10, %rax
+; CHECK-NEXT:    shll $10, %eax
 ; CHECK-NEXT:    retq
   %and = shl i64 %x, 10
   %shl = and i64 %and, 67107840
diff --git a/llvm/test/CodeGen/X86/no-plt.ll b/llvm/test/CodeGen/X86/no-plt.ll
index d6285a828a3b3..807731cea3145 100644
--- a/llvm/test/CodeGen/X86/no-plt.ll
+++ b/llvm/test/CodeGen/X86/no-plt.ll
@@ -5,7 +5,8 @@
 
 define i32 @fp_weakfunc() {
 ; X64: weakfunc@GOTPCREL(%rip)
-  ret i32 select (i1 icmp ne (ptr @weakfunc, ptr null), i32 1, i32 0)
+  %s = select i1 icmp ne (ptr @weakfunc, ptr null), i32 1, i32 0
+  ret i32 %s
 }
 declare extern_weak i32 @weakfunc() nonlazybind
 
diff --git a/llvm/test/CodeGen/X86/pcsections-memops.ll b/llvm/test/CodeGen/X86/pcsections-memops.ll
new file mode 100644
index 0000000000000..3e6acda18422d
--- /dev/null
+++ b/llvm/test/CodeGen/X86/pcsections-memops.ll
@@ -0,0 +1,244 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -O0 < %s | FileCheck %s --check-prefixes=O0
+; RUN: llc -O1 < %s | FileCheck %s --check-prefixes=OPT
+; RUN: llc -O2 < %s | FileCheck %s --check-prefixes=OPT
+; RUN: llc -O3 < %s | FileCheck %s --check-prefixes=OPT
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+@__llvm_gcov_ctr.89 = internal global [2 x i64] zeroinitializer
+@__llvm_gcov_ctr.109 = internal global [2 x i64] zeroinitializer
+@__llvm_gcov_ctr.112 = internal global [2 x i64] zeroinitializer
+@__llvm_gcov_ctr.113 = internal global [2 x i64] zeroinitializer
+@__llvm_gcov_ctr.154 = internal global [1 x i64] zeroinitializer
+@__llvm_gcov_ctr.155 = internal global [2 x i64] zeroinitializer
+@__llvm_gcov_ctr.156 = internal global [2 x i64] zeroinitializer
+@__llvm_gcov_ctr.157 = internal global [2 x i64] zeroinitializer
+@__llvm_gcov_ctr.160 = internal global [2 x i64] zeroinitializer
+@__llvm_gcov_ctr.161 = internal global [2 x i64] zeroinitializer
+@__llvm_gcov_ctr.189 = internal global [2 x i64] zeroinitializer
+@__llvm_gcov_ctr.1109 = internal global [2 x i64] zeroinitializer
+@__llvm_gcov_ctr.1112 = internal global [2 x i64] zeroinitializer
+@__llvm_gcov_ctr.1113 = internal global [2 x i64] zeroinitializer
+@__llvm_gcov_ctr.1154 = internal global [1 x i64] zeroinitializer
+@__llvm_gcov_ctr.1155 = internal global [2 x i64] zeroinitializer
+@__llvm_gcov_ctr.1156 = internal global [2 x i64] zeroinitializer
+@__llvm_gcov_ctr.1157 = internal global [2 x i64] zeroinitializer
+@__llvm_gcov_ctr.1160 = internal global [2 x i64] zeroinitializer
+@__llvm_gcov_ctr.1161 = internal global [2 x i64] zeroinitializer
+
+define dso_local void @independent_load_stores() {
+; O0-LABEL: independent_load_stores:
+; O0:       # %bb.0:
+; O0-NEXT:  .Lpcsection0:
+; O0-NEXT:    movq __llvm_gcov_ctr.109, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection1:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.109
+; O0-NEXT:  .Lpcsection2:
+; O0-NEXT:    movq __llvm_gcov_ctr.112, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection3:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.112
+; O0-NEXT:  .Lpcsection4:
+; O0-NEXT:    movq __llvm_gcov_ctr.113, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection5:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.113
+; O0-NEXT:  .Lpcsection6:
+; O0-NEXT:    movq __llvm_gcov_ctr.154, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection7:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.154
+; O0-NEXT:  .Lpcsection8:
+; O0-NEXT:    movq __llvm_gcov_ctr.155, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection9:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.155
+; O0-NEXT:  .Lpcsection10:
+; O0-NEXT:    movq __llvm_gcov_ctr.89, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection11:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.89
+; O0-NEXT:  .Lpcsection12:
+; O0-NEXT:    movq __llvm_gcov_ctr.160, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection13:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.160
+; O0-NEXT:  .Lpcsection14:
+; O0-NEXT:    movq __llvm_gcov_ctr.156, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection15:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.156
+; O0-NEXT:  .Lpcsection16:
+; O0-NEXT:    movq __llvm_gcov_ctr.157, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection17:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.157
+; O0-NEXT:  .Lpcsection18:
+; O0-NEXT:    movq __llvm_gcov_ctr.161, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection19:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.161
+; O0-NEXT:  .Lpcsection20:
+; O0-NEXT:    movq __llvm_gcov_ctr.1109, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection21:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.1109
+; O0-NEXT:  .Lpcsection22:
+; O0-NEXT:    movq __llvm_gcov_ctr.1112, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection23:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.1112
+; O0-NEXT:  .Lpcsection24:
+; O0-NEXT:    movq __llvm_gcov_ctr.1113, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection25:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.1113
+; O0-NEXT:  .Lpcsection26:
+; O0-NEXT:    movq __llvm_gcov_ctr.1154, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection27:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.1154
+; O0-NEXT:  .Lpcsection28:
+; O0-NEXT:    movq __llvm_gcov_ctr.1155, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection29:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.1155
+; O0-NEXT:  .Lpcsection30:
+; O0-NEXT:    movq __llvm_gcov_ctr.189, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection31:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.189
+; O0-NEXT:  .Lpcsection32:
+; O0-NEXT:    movq __llvm_gcov_ctr.1160, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection33:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.1160
+; O0-NEXT:  .Lpcsection34:
+; O0-NEXT:    movq __llvm_gcov_ctr.1156, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection35:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.1156
+; O0-NEXT:  .Lpcsection36:
+; O0-NEXT:    movq __llvm_gcov_ctr.1157, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection37:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.1157
+; O0-NEXT:  .Lpcsection38:
+; O0-NEXT:    movq __llvm_gcov_ctr.1161, %rax
+; O0-NEXT:    addq $1, %rax
+; O0-NEXT:  .Lpcsection39:
+; O0-NEXT:    movq %rax, __llvm_gcov_ctr.1161
+; O0-NEXT:    retq
+;
+; OPT-LABEL: independent_load_stores:
+; OPT:       # %bb.0:
+; OPT-NEXT:  .Lpcsection0:
+; OPT-NEXT:    incq __llvm_gcov_ctr.109(%rip)
+; OPT-NEXT:  .Lpcsection1:
+; OPT-NEXT:    incq __llvm_gcov_ctr.112(%rip)
+; OPT-NEXT:  .Lpcsection2:
+; OPT-NEXT:    incq __llvm_gcov_ctr.113(%rip)
+; OPT-NEXT:  .Lpcsection3:
+; OPT-NEXT:    incq __llvm_gcov_ctr.154(%rip)
+; OPT-NEXT:  .Lpcsection4:
+; OPT-NEXT:    incq __llvm_gcov_ctr.155(%rip)
+; OPT-NEXT:  .Lpcsection5:
+; OPT-NEXT:    incq __llvm_gcov_ctr.89(%rip)
+; OPT-NEXT:  .Lpcsection6:
+; OPT-NEXT:    incq __llvm_gcov_ctr.160(%rip)
+; OPT-NEXT:  .Lpcsection7:
+; OPT-NEXT:    incq __llvm_gcov_ctr.156(%rip)
+; OPT-NEXT:  .Lpcsection8:
+; OPT-NEXT:    incq __llvm_gcov_ctr.157(%rip)
+; OPT-NEXT:  .Lpcsection9:
+; OPT-NEXT:    incq __llvm_gcov_ctr.161(%rip)
+; OPT-NEXT:  .Lpcsection10:
+; OPT-NEXT:    incq __llvm_gcov_ctr.1109(%rip)
+; OPT-NEXT:  .Lpcsection11:
+; OPT-NEXT:    incq __llvm_gcov_ctr.1112(%rip)
+; OPT-NEXT:  .Lpcsection12:
+; OPT-NEXT:    incq __llvm_gcov_ctr.1113(%rip)
+; OPT-NEXT:  .Lpcsection13:
+; OPT-NEXT:    incq __llvm_gcov_ctr.1154(%rip)
+; OPT-NEXT:  .Lpcsection14:
+; OPT-NEXT:    incq __llvm_gcov_ctr.1155(%rip)
+; OPT-NEXT:  .Lpcsection15:
+; OPT-NEXT:    incq __llvm_gcov_ctr.189(%rip)
+; OPT-NEXT:  .Lpcsection16:
+; OPT-NEXT:    incq __llvm_gcov_ctr.1160(%rip)
+; OPT-NEXT:  .Lpcsection17:
+; OPT-NEXT:    incq __llvm_gcov_ctr.1156(%rip)
+; OPT-NEXT:  .Lpcsection18:
+; OPT-NEXT:    incq __llvm_gcov_ctr.1157(%rip)
+; OPT-NEXT:  .Lpcsection19:
+; OPT-NEXT:    incq __llvm_gcov_ctr.1161(%rip)
+; OPT-NEXT:    retq
+  %1 = load i64, ptr @__llvm_gcov_ctr.109, align 8, !pcsections !0
+  %2 = add i64 %1, 1
+  store i64 %2, ptr @__llvm_gcov_ctr.109, align 8, !pcsections !0
+  %3 = load i64, ptr @__llvm_gcov_ctr.112, align 8, !pcsections !0
+  %4 = add i64 %3, 1
+  store i64 %4, ptr @__llvm_gcov_ctr.112, align 8, !pcsections !0
+  %5 = load i64, ptr @__llvm_gcov_ctr.113, align 8, !pcsections !0
+  %6 = add i64 %5, 1
+  store i64 %6, ptr @__llvm_gcov_ctr.113, align 8, !pcsections !0
+  %7 = load i64, ptr @__llvm_gcov_ctr.154, align 8, !pcsections !0
+  %8 = add i64 %7, 1
+  store i64 %8, ptr @__llvm_gcov_ctr.154, align 8, !pcsections !0
+  %9 = load i64, ptr @__llvm_gcov_ctr.155, align 8, !pcsections !0
+  %10 = add i64 %9, 1
+  store i64 %10, ptr @__llvm_gcov_ctr.155, align 8, !pcsections !0
+  %11 = load i64, ptr @__llvm_gcov_ctr.89, align 8, !pcsections !0
+  %12 = add i64 %11, 1
+  store i64 %12, ptr @__llvm_gcov_ctr.89, align 8, !pcsections !0
+  %13 = load i64, ptr @__llvm_gcov_ctr.160, align 8, !pcsections !0
+  %14 = add i64 %13, 1
+  store i64 %14, ptr @__llvm_gcov_ctr.160, align 8, !pcsections !0
+  %15 = load i64, ptr @__llvm_gcov_ctr.156, align 8, !pcsections !0
+  %16 = add i64 %15, 1
+  store i64 %16, ptr @__llvm_gcov_ctr.156, align 8, !pcsections !0
+  %17 = load i64, ptr @__llvm_gcov_ctr.157, align 8, !pcsections !0
+  %18 = add i64 %17, 1
+  store i64 %18, ptr @__llvm_gcov_ctr.157, align 8, !pcsections !0
+  %19 = load i64, ptr @__llvm_gcov_ctr.161, align 8, !pcsections !0
+  %20 = add i64 %19, 1
+  store i64 %20, ptr @__llvm_gcov_ctr.161, align 8, !pcsections !0
+
+  %21 = load i64, ptr @__llvm_gcov_ctr.1109, align 8, !pcsections !0
+  %22 = add i64 %21, 1
+  store i64 %22, ptr @__llvm_gcov_ctr.1109, align 8, !pcsections !0
+  %23 = load i64, ptr @__llvm_gcov_ctr.1112, align 8, !pcsections !0
+  %24 = add i64 %23, 1
+  store i64 %24, ptr @__llvm_gcov_ctr.1112, align 8, !pcsections !0
+  %25 = load i64, ptr @__llvm_gcov_ctr.1113, align 8, !pcsections !0
+  %26 = add i64 %25, 1
+  store i64 %26, ptr @__llvm_gcov_ctr.1113, align 8, !pcsections !0
+  %27 = load i64, ptr @__llvm_gcov_ctr.1154, align 8, !pcsections !0
+  %28 = add i64 %27, 1
+  store i64 %28, ptr @__llvm_gcov_ctr.1154, align 8, !pcsections !0
+  %29 = load i64, ptr @__llvm_gcov_ctr.1155, align 8, !pcsections !0
+  %30 = add i64 %29, 1
+  store i64 %30, ptr @__llvm_gcov_ctr.1155, align 8, !pcsections !0
+  %31 = load i64, ptr @__llvm_gcov_ctr.189, align 8, !pcsections !0
+  %32 = add i64 %31, 1
+  store i64 %32, ptr @__llvm_gcov_ctr.189, align 8, !pcsections !0
+  %33 = load i64, ptr @__llvm_gcov_ctr.1160, align 8, !pcsections !0
+  %34 = add i64 %33, 1
+  store i64 %34, ptr @__llvm_gcov_ctr.1160, align 8, !pcsections !0
+  %35 = load i64, ptr @__llvm_gcov_ctr.1156, align 8, !pcsections !0
+  %36 = add i64 %35, 1
+  store i64 %36, ptr @__llvm_gcov_ctr.1156, align 8, !pcsections !0
+  %37 = load i64, ptr @__llvm_gcov_ctr.1157, align 8, !pcsections !0
+  %38 = add i64 %37, 1
+  store i64 %38, ptr @__llvm_gcov_ctr.1157, align 8, !pcsections !0
+  %39 = load i64, ptr @__llvm_gcov_ctr.1161, align 8, !pcsections !0
+  %40 = add i64 %39, 1
+  store i64 %40, ptr @__llvm_gcov_ctr.1161, align 8, !pcsections !0
+
+  ret void
+}
+
+!0 = !{!"foo"}
diff --git a/llvm/test/CodeGen/X86/pr30562.ll b/llvm/test/CodeGen/X86/pr30562.ll
index e04790d9dfce5..2745514cff8d4 100644
--- a/llvm/test/CodeGen/X86/pr30562.ll
+++ b/llvm/test/CodeGen/X86/pr30562.ll
@@ -20,7 +20,6 @@ define i32 @foo(ptr nocapture %perm, i32 %n) {
 ; CHECK-NEXT:    movaps %xmm1, %xmm0
 ; CHECK-NEXT:    jne .LBB0_1
 ; CHECK-NEXT:  # %bb.2: # %exit
-; CHECK-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-NEXT:    retq
 entry:
   br label %body
diff --git a/llvm/test/CodeGen/X86/pr35763.ll b/llvm/test/CodeGen/X86/pr35763.ll
index 8cc4b1b906785..9d2ee84cf675b 100644
--- a/llvm/test/CodeGen/X86/pr35763.ll
+++ b/llvm/test/CodeGen/X86/pr35763.ll
@@ -10,10 +10,10 @@
 define dso_local void @PR35763() {
 ; CHECK-LABEL: PR35763:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    movl z(%rip), %eax
-; CHECK-NEXT:    orl z+2(%rip), %eax
-; CHECK-NEXT:    movzwl %ax, %eax
-; CHECK-NEXT:    movq %rax, tf_3_var_136(%rip)
+; CHECK-NEXT:    movzwl z(%rip), %eax
+; CHECK-NEXT:    movzwl z+2(%rip), %ecx
+; CHECK-NEXT:    orl %eax, %ecx
+; CHECK-NEXT:    movq %rcx, tf_3_var_136(%rip)
 ; CHECK-NEXT:    movl z+6(%rip), %eax
 ; CHECK-NEXT:    movzbl z+10(%rip), %ecx
 ; CHECK-NEXT:    shlq $32, %rcx
diff --git a/llvm/test/CodeGen/X86/pr44749.ll b/llvm/test/CodeGen/X86/pr44749.ll
index b2c0e17fc6684..cc9963dc2d8cc 100644
--- a/llvm/test/CodeGen/X86/pr44749.ll
+++ b/llvm/test/CodeGen/X86/pr44749.ll
@@ -14,9 +14,9 @@ define i32 @a() {
 ; CHECK-NEXT:    subq $-1, %rax
 ; CHECK-NEXT:    setne %al
 ; CHECK-NEXT:    movzbl %al, %eax
-; CHECK-NEXT:    movl %eax, %ecx
-; CHECK-NEXT:    leaq {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %rax
-; CHECK-NEXT:    movsd {{.*#+}} xmm0 = mem[0],zero
+; CHECK-NEXT:    cvtsi2sd %eax, %xmm0
+; CHECK-NEXT:    movsd {{.*#+}} xmm2 = mem[0],zero
+; CHECK-NEXT:    subsd %xmm2, %xmm0
 ; CHECK-NEXT:    movsd {{.*#+}} xmm3 = mem[0],zero
 ; CHECK-NEXT:    movsd {{.*#+}} xmm2 = mem[0],zero
 ; CHECK-NEXT:    cmplesd %xmm1, %xmm0
@@ -30,7 +30,9 @@ define i32 @a() {
 entry:
   %call = call i32 (...) @b()
   %conv = sitofp i32 %call to double
-  %fsub = fsub double sitofp (i32 select (i1 icmp ne (ptr getelementptr (i8, ptr @calloc, i64 1), ptr null), i32 1, i32 0) to double), 1.000000e+02
+  %sel = select i1 icmp ne (ptr getelementptr (i8, ptr @calloc, i64 1), ptr null), i32 1, i32 0
+  %sitofp = sitofp i32 %sel to double
+  %fsub = fsub double %sitofp, 1.000000e+02
   %cmp = fcmp ole double %fsub, %conv
   %cond = select i1 %cmp, double 1.000000e+00, double 3.140000e+00
   %conv2 = fptosi double %cond to i32
diff --git a/llvm/test/CodeGen/X86/pr58914.mir b/llvm/test/CodeGen/X86/pr58914.mir
new file mode 100644
index 0000000000000..f1cafc99d5505
--- /dev/null
+++ b/llvm/test/CodeGen/X86/pr58914.mir
@@ -0,0 +1,36 @@
+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 2
+# RUN: llc -x mir < %s -mtriple=x86_64-unknown-unknown -mcpu=skylake-avx512 -O1 -run-pass=livevars,twoaddressinstruction | FileCheck %s
+
+---
+name: PR58914
+tracksRegLiveness: true
+stack:
+  - { id: 0, offset: 0, size: 768 }
+body:             |
+  ; CHECK-LABEL: name: PR58914
+  ; CHECK: bb.0:
+  ; CHECK-NEXT:   successors: %bb.1(0x80000000)
+  ; CHECK-NEXT:   liveins: $rcx, $ymm0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   [[COPY:%[0-9]+]]:gr64_nosp = COPY killed $rcx
+  ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:vr256x = COPY killed $ymm0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.1:
+  ; CHECK-NEXT:   successors: %bb.1(0x80000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   [[COPY2:%[0-9]+]]:vk16wm = COPY $rsp
+  ; CHECK-NEXT:   [[VBLENDMPDZ256rmk:%[0-9]+]]:vr256x = VBLENDMPDZ256rmk killed [[COPY2]], [[COPY1]], %stack.0, 8, [[COPY]], 0, $noreg
+  ; CHECK-NEXT:   RET killed [[VBLENDMPDZ256rmk]]
+  bb.0:
+    successors: %bb.1(0x7c000000)
+    liveins: $rcx, $ymm0
+    %0:gr64_nosp = COPY $rcx
+    %1:vr256x = COPY $ymm0
+
+  bb.1:
+    successors: %bb.1(0x7c000000)
+    %2:vk16wm = COPY $rsp
+    %3:vr256x = VMOVUPDZ256rmk %1, killed %2, %stack.0, 8, %0, 0, $noreg
+    RET %3
+
+...
diff --git a/llvm/test/CodeGen/X86/pr61348.ll b/llvm/test/CodeGen/X86/pr61348.ll
new file mode 100644
index 0000000000000..f17cbf3b62583
--- /dev/null
+++ b/llvm/test/CodeGen/X86/pr61348.ll
@@ -0,0 +1,23 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
+; REQUIRES: asserts
+; RUN: llc < %s -mtriple=x86_64-- | FileCheck %s
+
+define i32 @PR61348() optsize {
+; CHECK-LABEL: PR61348:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movl $3, %ecx
+; CHECK-NEXT:    xorl %edi, %edi
+; CHECK-NEXT:    xorl %esi, %esi
+; CHECK-NEXT:    rep;movsq (%rsi), %es:(%rdi)
+; CHECK-NEXT:    movb 30, %al
+; CHECK-NEXT:    movb %al, 30
+; CHECK-NEXT:    movzwl 28, %eax
+; CHECK-NEXT:    movw %ax, 28
+; CHECK-NEXT:    movl 24, %eax
+; CHECK-NEXT:    movl %eax, 24
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    retq
+  tail call void @llvm.memcpy.p0.p0.i64(ptr align 1 null, ptr align 1 null, i64 31, i1 true)
+  ret i32 0
+}
+declare void @llvm.memcpy.p0.p0.i64(ptr noalias nocapture writeonly, ptr noalias nocapture readonly, i64, i1 immarg)
diff --git a/llvm/test/CodeGen/X86/pr61384.ll b/llvm/test/CodeGen/X86/pr61384.ll
new file mode 100644
index 0000000000000..e5178b4904b7e
--- /dev/null
+++ b/llvm/test/CodeGen/X86/pr61384.ll
@@ -0,0 +1,19 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -show-mc-encoding < %s | FileCheck %s
+
+@a = external dso_local global { { i64 } }
+
+define i32 @atomic_global() nounwind {
+; CHECK-LABEL: atomic_global:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    xorl %eax, %eax # encoding: [0x31,0xc0]
+; CHECK-NEXT:    lock btsq %rax, a(%rip) # encoding: [0xf0,0x48,0x0f,0xab,0x05,A,A,A,A]
+; CHECK-NEXT:    # fixup A - offset: 5, value: a-4, kind: reloc_riprel_4byte
+; CHECK-NEXT:    xorl %eax, %eax # encoding: [0x31,0xc0]
+; CHECK-NEXT:    retq # encoding: [0xc3]
+entry:
+  %shl.i = shl i64 1, 0
+  %0 = atomicrmw or ptr @a, i64 %shl.i monotonic, align 8
+  %and.i = and i64 %shl.i, %0
+  ret i32 0
+}
diff --git a/llvm/test/CodeGen/X86/prefer-avx256-lzcnt.ll b/llvm/test/CodeGen/X86/prefer-avx256-lzcnt.ll
index 0655cc9c9f470..056aeaba9271c 100644
--- a/llvm/test/CodeGen/X86/prefer-avx256-lzcnt.ll
+++ b/llvm/test/CodeGen/X86/prefer-avx256-lzcnt.ll
@@ -64,17 +64,15 @@ define <16 x i8> @testv16i8(<16 x i8> %in) {
 define <16 x i16> @testv16i16(<16 x i16> %in) {
 ; AVX256-LABEL: testv16i16:
 ; AVX256:       # %bb.0:
-; AVX256-NEXT:    vextracti128 $1, %ymm0, %xmm1
-; AVX256-NEXT:    vpmovzxwd {{.*#+}} ymm1 = xmm1[0],zero,xmm1[1],zero,xmm1[2],zero,xmm1[3],zero,xmm1[4],zero,xmm1[5],zero,xmm1[6],zero,xmm1[7],zero
+; AVX256-NEXT:    vpmovzxwd {{.*#+}} ymm1 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero
 ; AVX256-NEXT:    vplzcntd %ymm1, %ymm1
 ; AVX256-NEXT:    vpmovdw %ymm1, %xmm1
-; AVX256-NEXT:    vmovdqa {{.*#+}} xmm2 = [16,16,16,16,16,16,16,16]
-; AVX256-NEXT:    vpsubw %xmm2, %xmm1, %xmm1
+; AVX256-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX256-NEXT:    vpmovzxwd {{.*#+}} ymm0 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero
 ; AVX256-NEXT:    vplzcntd %ymm0, %ymm0
 ; AVX256-NEXT:    vpmovdw %ymm0, %xmm0
-; AVX256-NEXT:    vpsubw %xmm2, %xmm0, %xmm0
-; AVX256-NEXT:    vinserti128 $1, %xmm1, %ymm0, %ymm0
+; AVX256-NEXT:    vinserti128 $1, %xmm0, %ymm1, %ymm0
+; AVX256-NEXT:    vpsubw {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
 ; AVX256-NEXT:    retq
 ;
 ; AVX512-LABEL: testv16i16:
@@ -104,17 +102,15 @@ define <32 x i8> @testv32i8(<32 x i8> %in) {
 ;
 ; AVX512-LABEL: testv32i8:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    vextracti128 $1, %ymm0, %xmm1
-; AVX512-NEXT:    vpmovzxbd {{.*#+}} zmm1 = xmm1[0],zero,zero,zero,xmm1[1],zero,zero,zero,xmm1[2],zero,zero,zero,xmm1[3],zero,zero,zero,xmm1[4],zero,zero,zero,xmm1[5],zero,zero,zero,xmm1[6],zero,zero,zero,xmm1[7],zero,zero,zero,xmm1[8],zero,zero,zero,xmm1[9],zero,zero,zero,xmm1[10],zero,zero,zero,xmm1[11],zero,zero,zero,xmm1[12],zero,zero,zero,xmm1[13],zero,zero,zero,xmm1[14],zero,zero,zero,xmm1[15],zero,zero,zero
+; AVX512-NEXT:    vpmovzxbd {{.*#+}} zmm1 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero,xmm0[2],zero,zero,zero,xmm0[3],zero,zero,zero,xmm0[4],zero,zero,zero,xmm0[5],zero,zero,zero,xmm0[6],zero,zero,zero,xmm0[7],zero,zero,zero,xmm0[8],zero,zero,zero,xmm0[9],zero,zero,zero,xmm0[10],zero,zero,zero,xmm0[11],zero,zero,zero,xmm0[12],zero,zero,zero,xmm0[13],zero,zero,zero,xmm0[14],zero,zero,zero,xmm0[15],zero,zero,zero
 ; AVX512-NEXT:    vplzcntd %zmm1, %zmm1
 ; AVX512-NEXT:    vpmovdb %zmm1, %xmm1
-; AVX512-NEXT:    vmovdqa {{.*#+}} xmm2 = [24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24]
-; AVX512-NEXT:    vpsubb %xmm2, %xmm1, %xmm1
+; AVX512-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX512-NEXT:    vpmovzxbd {{.*#+}} zmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero,xmm0[2],zero,zero,zero,xmm0[3],zero,zero,zero,xmm0[4],zero,zero,zero,xmm0[5],zero,zero,zero,xmm0[6],zero,zero,zero,xmm0[7],zero,zero,zero,xmm0[8],zero,zero,zero,xmm0[9],zero,zero,zero,xmm0[10],zero,zero,zero,xmm0[11],zero,zero,zero,xmm0[12],zero,zero,zero,xmm0[13],zero,zero,zero,xmm0[14],zero,zero,zero,xmm0[15],zero,zero,zero
 ; AVX512-NEXT:    vplzcntd %zmm0, %zmm0
 ; AVX512-NEXT:    vpmovdb %zmm0, %xmm0
-; AVX512-NEXT:    vpsubb %xmm2, %xmm0, %xmm0
-; AVX512-NEXT:    vinserti128 $1, %xmm1, %ymm0, %ymm0
+; AVX512-NEXT:    vinserti128 $1, %xmm0, %ymm1, %ymm0
+; AVX512-NEXT:    vpsubb {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
 ; AVX512-NEXT:    retq
   %out = call <32 x i8> @llvm.ctlz.v32i8(<32 x i8> %in, i1 false)
   ret <32 x i8> %out
diff --git a/llvm/test/CodeGen/X86/setcc-combine.ll b/llvm/test/CodeGen/X86/setcc-combine.ll
index d225d420987b3..c2ee78989ba16 100644
--- a/llvm/test/CodeGen/X86/setcc-combine.ll
+++ b/llvm/test/CodeGen/X86/setcc-combine.ll
@@ -244,14 +244,9 @@ define void @test_i1_uge(ptr%A2) {
 ; CHECK-LABEL: test_i1_uge:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    movzbl (%rdi), %eax
-; CHECK-NEXT:    movl %eax, %ecx
-; CHECK-NEXT:    xorb $1, %cl
-; CHECK-NEXT:    andb %cl, %al
-; CHECK-NEXT:    movzbl %al, %eax
-; CHECK-NEXT:    andl $1, %eax
-; CHECK-NEXT:    negq %rax
-; CHECK-NEXT:    andb $1, %cl
-; CHECK-NEXT:    movb %cl, (%rdi,%rax)
+; CHECK-NEXT:    notb %al
+; CHECK-NEXT:    andb $1, %al
+; CHECK-NEXT:    movb %al, (%rdi)
 ; CHECK-NEXT:    retq
   %L5 = load i1, ptr %A2
   %C3 = icmp ne i1 %L5, true
diff --git a/llvm/test/CodeGen/X86/subcarry.ll b/llvm/test/CodeGen/X86/subcarry.ll
index 28a48af14d29f..9538ea1061cd1 100644
--- a/llvm/test/CodeGen/X86/subcarry.ll
+++ b/llvm/test/CodeGen/X86/subcarry.ll
@@ -392,7 +392,6 @@ define { i64, i1 } @subcarry_carry_not_zext(i64 %a, i64 %b, i64 %carryin) {
 define { i64, i1 } @subcarry_carry_not_i1(i64 %a, i64 %b, i8 %carryin) {
 ; CHECK-LABEL: subcarry_carry_not_i1:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $edx killed $edx def $rdx
 ; CHECK-NEXT:    movq %rdi, %rax
 ; CHECK-NEXT:    subq %rsi, %rax
 ; CHECK-NEXT:    setb %cl
diff --git a/llvm/test/CodeGen/X86/switch-phi-const.ll b/llvm/test/CodeGen/X86/switch-phi-const.ll
index 448e7b543d21c..08ff1a03d8e4b 100644
--- a/llvm/test/CodeGen/X86/switch-phi-const.ll
+++ b/llvm/test/CodeGen/X86/switch-phi-const.ll
@@ -91,13 +91,11 @@ default:
 define void @switch_trunc_phi_const(i32 %x) {
 ; CHECK-LABEL: switch_trunc_phi_const:
 ; CHECK:       # %bb.0: # %bb0
-; CHECK-NEXT:    # kill: def $edi killed $edi def $rdi
-; CHECK-NEXT:    movzbl %dil, %ecx
-; CHECK-NEXT:    decl %ecx
+; CHECK-NEXT:    movzbl %dil, %eax
+; CHECK-NEXT:    leal -1(%rax), %ecx
 ; CHECK-NEXT:    cmpl $54, %ecx
 ; CHECK-NEXT:    ja .LBB1_8
 ; CHECK-NEXT:  # %bb.1: # %bb0
-; CHECK-NEXT:    movzbl %dil, %eax
 ; CHECK-NEXT:    movl $3895, %edx # imm = 0xF37
 ; CHECK-NEXT:    jmpq *.LJTI1_0(,%rcx,8)
 ; CHECK-NEXT:  .LBB1_8: # %default
diff --git a/llvm/test/CodeGen/X86/switch.ll b/llvm/test/CodeGen/X86/switch.ll
index cc6aba92bf708..f5040f2b2bab5 100644
--- a/llvm/test/CodeGen/X86/switch.ll
+++ b/llvm/test/CodeGen/X86/switch.ll
@@ -1416,7 +1416,6 @@ sw:
 define void @int_max_table_cluster(i8 %x) {
 ; CHECK-LABEL: int_max_table_cluster:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    # kill: def $edi killed $edi def $rdi
 ; CHECK-NEXT:    cmpb $-9, %dil
 ; CHECK-NEXT:    ja .LBB15_4
 ; CHECK-NEXT:  # %bb.1: # %entry
diff --git a/llvm/test/CodeGen/X86/test-vs-bittest.ll b/llvm/test/CodeGen/X86/test-vs-bittest.ll
index b22336e62a14e..0998a0f1d6ba9 100644
--- a/llvm/test/CodeGen/X86/test-vs-bittest.ll
+++ b/llvm/test/CodeGen/X86/test-vs-bittest.ll
@@ -571,6 +571,20 @@ define i64 @is_upper_bit_clear_i64(i64 %x) {
   ret i64 %r
 }
 
+define i32 @is_upper_bit_clear_i64_trunc(i64 %x) {
+; CHECK-LABEL: is_upper_bit_clear_i64_trunc:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    btq $42, %rdi
+; CHECK-NEXT:    setae %al
+; CHECK-NEXT:    retq
+  %sh = lshr i64 %x, 42
+  %t = trunc i64 %sh to i32
+  %m = and i32 %t, 1
+  %r = xor i32 %m, 1
+  ret i32 %r
+}
+
 define i64 @is_upper_bit_clear_i64_not(i64 %x) {
 ; CHECK-LABEL: is_upper_bit_clear_i64_not:
 ; CHECK:       # %bb.0:
@@ -688,16 +702,13 @@ define i8 @is_bit_clear_i8_not(i8 %x) {
   ret i8 %r
 }
 
-; TODO: We could use bt/test on the 64-bit value.
+; Use bt/test on the 64-bit value and truncate result.
 
 define i8 @overshift(i64 %x) {
 ; CHECK-LABEL: overshift:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    movq %rdi, %rax
-; CHECK-NEXT:    shrq $42, %rax
-; CHECK-NEXT:    notb %al
-; CHECK-NEXT:    andb $1, %al
-; CHECK-NEXT:    # kill: def $al killed $al killed $rax
+; CHECK-NEXT:    btq $42, %rdi
+; CHECK-NEXT:    setae %al
 ; CHECK-NEXT:    retq
   %a = lshr i64 %x, 42
   %t = trunc i64 %a to i8
diff --git a/llvm/test/CodeGen/X86/tuning-shuffle-permilps-avx512.ll b/llvm/test/CodeGen/X86/tuning-shuffle-permilps-avx512.ll
index d7f2116084c57..2e1462d60fd37 100644
--- a/llvm/test/CodeGen/X86/tuning-shuffle-permilps-avx512.ll
+++ b/llvm/test/CodeGen/X86/tuning-shuffle-permilps-avx512.ll
@@ -35,7 +35,7 @@ define <16 x float> @transform_VPERMILPSZrrkz(<16 x float> %a, i16 %mask_int) no
 ; CHECK-LABEL: transform_VPERMILPSZrrkz:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = zmm0[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
+; CHECK-NEXT:    vshufps {{.*#+}} zmm0 {%k1} {z} = zmm0[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
 ; CHECK-NEXT:    retq
   %mask = bitcast i16 %mask_int to <16 x i1>
   %shufp = shufflevector <16 x float> %a, <16 x float> poison, <16 x i32> <i32 3, i32 2, i32 1, i32 0, i32 7, i32 6, i32 5, i32 4, i32 11, i32 10, i32 9, i32 8, i32 15, i32 14, i32 13, i32 12>
@@ -47,7 +47,7 @@ define <8 x float> @transform_VPERMILPSYrrkz(<8 x float> %a, i8 %mask_int) nounw
 ; CHECK-LABEL: transform_VPERMILPSYrrkz:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} ymm0 {%k1} {z} = ymm0[3,2,1,0,7,6,5,4]
+; CHECK-NEXT:    vshufps {{.*#+}} ymm0 {%k1} {z} = ymm0[3,2,1,0,7,6,5,4]
 ; CHECK-NEXT:    retq
   %mask = bitcast i8 %mask_int to <8 x i1>
   %shufp = shufflevector <8 x float> %a, <8 x float> poison, <8 x i32> <i32 3, i32 2, i32 1, i32 0, i32 7, i32 6, i32 5, i32 4>
@@ -59,7 +59,7 @@ define <4 x float> @transform_VPERMILPSrrkz(<4 x float> %a, i4 %mask_int) nounwi
 ; CHECK-LABEL: transform_VPERMILPSrrkz:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} xmm0 {%k1} {z} = xmm0[3,2,1,0]
+; CHECK-NEXT:    vshufps {{.*#+}} xmm0 {%k1} {z} = xmm0[3,2,1,0]
 ; CHECK-NEXT:    retq
   %mask = bitcast i4 %mask_int to <4 x i1>
   %shufp = shufflevector <4 x float> %a, <4 x float> poison, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
@@ -71,7 +71,7 @@ define <16 x float> @transform_VPERMILPSZrrk(<16 x float> %a, <16 x float> %b, i
 ; CHECK-LABEL: transform_VPERMILPSZrrk:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} zmm1 {%k1} = zmm0[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
+; CHECK-NEXT:    vshufps {{.*#+}} zmm1 {%k1} = zmm0[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
 ; CHECK-NEXT:    vmovaps %zmm1, %zmm0
 ; CHECK-NEXT:    retq
   %mask = bitcast i16 %mask_int to <16 x i1>
@@ -84,7 +84,7 @@ define <8 x float> @transform_VPERMILPSYrrk(<8 x float> %a, <8 x float> %b, i8 %
 ; CHECK-LABEL: transform_VPERMILPSYrrk:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} ymm1 {%k1} = ymm0[3,2,1,0,7,6,5,4]
+; CHECK-NEXT:    vshufps {{.*#+}} ymm1 {%k1} = ymm0[3,2,1,0,7,6,5,4]
 ; CHECK-NEXT:    vmovaps %ymm1, %ymm0
 ; CHECK-NEXT:    retq
   %mask = bitcast i8 %mask_int to <8 x i1>
@@ -97,7 +97,7 @@ define <4 x float> @transform_VPERMILPSrrk(<4 x float> %a, <4 x float> %b, i4 %m
 ; CHECK-LABEL: transform_VPERMILPSrrk:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} xmm1 {%k1} = xmm0[3,2,1,0]
+; CHECK-NEXT:    vshufps {{.*#+}} xmm1 {%k1} = xmm0[3,2,1,0]
 ; CHECK-NEXT:    vmovaps %xmm1, %xmm0
 ; CHECK-NEXT:    retq
   %mask = bitcast i4 %mask_int to <4 x i1>
@@ -182,11 +182,29 @@ define <4 x float> @transform_VPERMILPSrm(ptr %ap) nounwind {
 }
 
 define <16 x float> @transform_VPERMILPSZrmkz(ptr %ap, i16 %mask_int) nounwind {
-; CHECK-LABEL: transform_VPERMILPSZrmkz:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    kmovd %esi, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = mem[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
-; CHECK-NEXT:    retq
+; CHECK-ICX-LABEL: transform_VPERMILPSZrmkz:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    kmovd %esi, %k1
+; CHECK-ICX-NEXT:    vpshufd {{.*#+}} zmm0 {%k1} {z} = mem[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
+; CHECK-ICX-NEXT:    retq
+;
+; CHECK-V4-LABEL: transform_VPERMILPSZrmkz:
+; CHECK-V4:       # %bb.0:
+; CHECK-V4-NEXT:    kmovd %esi, %k1
+; CHECK-V4-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = mem[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
+; CHECK-V4-NEXT:    retq
+;
+; CHECK-AVX512-LABEL: transform_VPERMILPSZrmkz:
+; CHECK-AVX512:       # %bb.0:
+; CHECK-AVX512-NEXT:    kmovd %esi, %k1
+; CHECK-AVX512-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = mem[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
+; CHECK-AVX512-NEXT:    retq
+;
+; CHECK-ZNVER4-LABEL: transform_VPERMILPSZrmkz:
+; CHECK-ZNVER4:       # %bb.0:
+; CHECK-ZNVER4-NEXT:    kmovd %esi, %k1
+; CHECK-ZNVER4-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = mem[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
+; CHECK-ZNVER4-NEXT:    retq
   %mask = bitcast i16 %mask_int to <16 x i1>
   %a = load <16 x float>, ptr %ap
   %shufp = shufflevector <16 x float> %a, <16 x float> poison, <16 x i32> <i32 3, i32 2, i32 1, i32 0, i32 7, i32 6, i32 5, i32 4, i32 11, i32 10, i32 9, i32 8, i32 15, i32 14, i32 13, i32 12>
@@ -195,11 +213,29 @@ define <16 x float> @transform_VPERMILPSZrmkz(ptr %ap, i16 %mask_int) nounwind {
 }
 
 define <8 x float> @transform_VPERMILPSYrmkz(ptr %ap, i8 %mask_int) nounwind {
-; CHECK-LABEL: transform_VPERMILPSYrmkz:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    kmovd %esi, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} ymm0 {%k1} {z} = mem[3,2,1,0,7,6,5,4]
-; CHECK-NEXT:    retq
+; CHECK-ICX-LABEL: transform_VPERMILPSYrmkz:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    kmovd %esi, %k1
+; CHECK-ICX-NEXT:    vpshufd {{.*#+}} ymm0 {%k1} {z} = mem[3,2,1,0,7,6,5,4]
+; CHECK-ICX-NEXT:    retq
+;
+; CHECK-V4-LABEL: transform_VPERMILPSYrmkz:
+; CHECK-V4:       # %bb.0:
+; CHECK-V4-NEXT:    kmovd %esi, %k1
+; CHECK-V4-NEXT:    vpermilps {{.*#+}} ymm0 {%k1} {z} = mem[3,2,1,0,7,6,5,4]
+; CHECK-V4-NEXT:    retq
+;
+; CHECK-AVX512-LABEL: transform_VPERMILPSYrmkz:
+; CHECK-AVX512:       # %bb.0:
+; CHECK-AVX512-NEXT:    kmovd %esi, %k1
+; CHECK-AVX512-NEXT:    vpermilps {{.*#+}} ymm0 {%k1} {z} = mem[3,2,1,0,7,6,5,4]
+; CHECK-AVX512-NEXT:    retq
+;
+; CHECK-ZNVER4-LABEL: transform_VPERMILPSYrmkz:
+; CHECK-ZNVER4:       # %bb.0:
+; CHECK-ZNVER4-NEXT:    kmovd %esi, %k1
+; CHECK-ZNVER4-NEXT:    vpermilps {{.*#+}} ymm0 {%k1} {z} = mem[3,2,1,0,7,6,5,4]
+; CHECK-ZNVER4-NEXT:    retq
   %mask = bitcast i8 %mask_int to <8 x i1>
   %a = load <8 x float>, ptr %ap
   %shufp = shufflevector <8 x float> %a, <8 x float> poison, <8 x i32> <i32 3, i32 2, i32 1, i32 0, i32 7, i32 6, i32 5, i32 4>
@@ -208,11 +244,29 @@ define <8 x float> @transform_VPERMILPSYrmkz(ptr %ap, i8 %mask_int) nounwind {
 }
 
 define <4 x float> @transform_VPERMILPSrmkz(ptr %ap, i4 %mask_int) nounwind {
-; CHECK-LABEL: transform_VPERMILPSrmkz:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    kmovd %esi, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} xmm0 {%k1} {z} = mem[3,2,1,0]
-; CHECK-NEXT:    retq
+; CHECK-ICX-LABEL: transform_VPERMILPSrmkz:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    kmovd %esi, %k1
+; CHECK-ICX-NEXT:    vpshufd {{.*#+}} xmm0 {%k1} {z} = mem[3,2,1,0]
+; CHECK-ICX-NEXT:    retq
+;
+; CHECK-V4-LABEL: transform_VPERMILPSrmkz:
+; CHECK-V4:       # %bb.0:
+; CHECK-V4-NEXT:    kmovd %esi, %k1
+; CHECK-V4-NEXT:    vpermilps {{.*#+}} xmm0 {%k1} {z} = mem[3,2,1,0]
+; CHECK-V4-NEXT:    retq
+;
+; CHECK-AVX512-LABEL: transform_VPERMILPSrmkz:
+; CHECK-AVX512:       # %bb.0:
+; CHECK-AVX512-NEXT:    kmovd %esi, %k1
+; CHECK-AVX512-NEXT:    vpermilps {{.*#+}} xmm0 {%k1} {z} = mem[3,2,1,0]
+; CHECK-AVX512-NEXT:    retq
+;
+; CHECK-ZNVER4-LABEL: transform_VPERMILPSrmkz:
+; CHECK-ZNVER4:       # %bb.0:
+; CHECK-ZNVER4-NEXT:    kmovd %esi, %k1
+; CHECK-ZNVER4-NEXT:    vpermilps {{.*#+}} xmm0 {%k1} {z} = mem[3,2,1,0]
+; CHECK-ZNVER4-NEXT:    retq
   %mask = bitcast i4 %mask_int to <4 x i1>
   %a = load <4 x float>, ptr %ap
   %shufp = shufflevector <4 x float> %a, <4 x float> poison, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
@@ -221,11 +275,29 @@ define <4 x float> @transform_VPERMILPSrmkz(ptr %ap, i4 %mask_int) nounwind {
 }
 
 define <16 x float> @transform_VPERMILPSZrmk(ptr %ap, <16 x float> %b, i16 %mask_int) nounwind {
-; CHECK-LABEL: transform_VPERMILPSZrmk:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    kmovd %esi, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} = mem[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
-; CHECK-NEXT:    retq
+; CHECK-ICX-LABEL: transform_VPERMILPSZrmk:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    kmovd %esi, %k1
+; CHECK-ICX-NEXT:    vpshufd {{.*#+}} zmm0 {%k1} = mem[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
+; CHECK-ICX-NEXT:    retq
+;
+; CHECK-V4-LABEL: transform_VPERMILPSZrmk:
+; CHECK-V4:       # %bb.0:
+; CHECK-V4-NEXT:    kmovd %esi, %k1
+; CHECK-V4-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} = mem[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
+; CHECK-V4-NEXT:    retq
+;
+; CHECK-AVX512-LABEL: transform_VPERMILPSZrmk:
+; CHECK-AVX512:       # %bb.0:
+; CHECK-AVX512-NEXT:    kmovd %esi, %k1
+; CHECK-AVX512-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} = mem[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
+; CHECK-AVX512-NEXT:    retq
+;
+; CHECK-ZNVER4-LABEL: transform_VPERMILPSZrmk:
+; CHECK-ZNVER4:       # %bb.0:
+; CHECK-ZNVER4-NEXT:    kmovd %esi, %k1
+; CHECK-ZNVER4-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} = mem[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
+; CHECK-ZNVER4-NEXT:    retq
   %mask = bitcast i16 %mask_int to <16 x i1>
   %a = load <16 x float>, ptr %ap
   %shufp = shufflevector <16 x float> %a, <16 x float> poison, <16 x i32> <i32 3, i32 2, i32 1, i32 0, i32 7, i32 6, i32 5, i32 4, i32 11, i32 10, i32 9, i32 8, i32 15, i32 14, i32 13, i32 12>
@@ -234,11 +306,29 @@ define <16 x float> @transform_VPERMILPSZrmk(ptr %ap, <16 x float> %b, i16 %mask
 }
 
 define <8 x float> @transform_VPERMILPSYrmk(ptr %ap, <8 x float> %b, i8 %mask_int) nounwind {
-; CHECK-LABEL: transform_VPERMILPSYrmk:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    kmovd %esi, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} ymm0 {%k1} = mem[3,2,1,0,7,6,5,4]
-; CHECK-NEXT:    retq
+; CHECK-ICX-LABEL: transform_VPERMILPSYrmk:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    kmovd %esi, %k1
+; CHECK-ICX-NEXT:    vpshufd {{.*#+}} ymm0 {%k1} = mem[3,2,1,0,7,6,5,4]
+; CHECK-ICX-NEXT:    retq
+;
+; CHECK-V4-LABEL: transform_VPERMILPSYrmk:
+; CHECK-V4:       # %bb.0:
+; CHECK-V4-NEXT:    kmovd %esi, %k1
+; CHECK-V4-NEXT:    vpermilps {{.*#+}} ymm0 {%k1} = mem[3,2,1,0,7,6,5,4]
+; CHECK-V4-NEXT:    retq
+;
+; CHECK-AVX512-LABEL: transform_VPERMILPSYrmk:
+; CHECK-AVX512:       # %bb.0:
+; CHECK-AVX512-NEXT:    kmovd %esi, %k1
+; CHECK-AVX512-NEXT:    vpermilps {{.*#+}} ymm0 {%k1} = mem[3,2,1,0,7,6,5,4]
+; CHECK-AVX512-NEXT:    retq
+;
+; CHECK-ZNVER4-LABEL: transform_VPERMILPSYrmk:
+; CHECK-ZNVER4:       # %bb.0:
+; CHECK-ZNVER4-NEXT:    kmovd %esi, %k1
+; CHECK-ZNVER4-NEXT:    vpermilps {{.*#+}} ymm0 {%k1} = mem[3,2,1,0,7,6,5,4]
+; CHECK-ZNVER4-NEXT:    retq
   %mask = bitcast i8 %mask_int to <8 x i1>
   %a = load <8 x float>, ptr %ap
   %shufp = shufflevector <8 x float> %a, <8 x float> poison, <8 x i32> <i32 3, i32 2, i32 1, i32 0, i32 7, i32 6, i32 5, i32 4>
@@ -247,11 +337,29 @@ define <8 x float> @transform_VPERMILPSYrmk(ptr %ap, <8 x float> %b, i8 %mask_in
 }
 
 define <4 x float> @transform_VPERMILPSrmk(ptr %ap, <4 x float> %b, i4 %mask_int) nounwind {
-; CHECK-LABEL: transform_VPERMILPSrmk:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    kmovd %esi, %k1
-; CHECK-NEXT:    vpermilps {{.*#+}} xmm0 {%k1} = mem[3,2,1,0]
-; CHECK-NEXT:    retq
+; CHECK-ICX-LABEL: transform_VPERMILPSrmk:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    kmovd %esi, %k1
+; CHECK-ICX-NEXT:    vpshufd {{.*#+}} xmm0 {%k1} = mem[3,2,1,0]
+; CHECK-ICX-NEXT:    retq
+;
+; CHECK-V4-LABEL: transform_VPERMILPSrmk:
+; CHECK-V4:       # %bb.0:
+; CHECK-V4-NEXT:    kmovd %esi, %k1
+; CHECK-V4-NEXT:    vpermilps {{.*#+}} xmm0 {%k1} = mem[3,2,1,0]
+; CHECK-V4-NEXT:    retq
+;
+; CHECK-AVX512-LABEL: transform_VPERMILPSrmk:
+; CHECK-AVX512:       # %bb.0:
+; CHECK-AVX512-NEXT:    kmovd %esi, %k1
+; CHECK-AVX512-NEXT:    vpermilps {{.*#+}} xmm0 {%k1} = mem[3,2,1,0]
+; CHECK-AVX512-NEXT:    retq
+;
+; CHECK-ZNVER4-LABEL: transform_VPERMILPSrmk:
+; CHECK-ZNVER4:       # %bb.0:
+; CHECK-ZNVER4-NEXT:    kmovd %esi, %k1
+; CHECK-ZNVER4-NEXT:    vpermilps {{.*#+}} xmm0 {%k1} = mem[3,2,1,0]
+; CHECK-ZNVER4-NEXT:    retq
   %mask = bitcast i4 %mask_int to <4 x i1>
   %a = load <4 x float>, ptr %ap
   %shufp = shufflevector <4 x float> %a, <4 x float> poison, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
diff --git a/llvm/test/CodeGen/X86/tuning-shuffle-unpckpd-avx512.ll b/llvm/test/CodeGen/X86/tuning-shuffle-unpckpd-avx512.ll
index 5b96196a4ec59..d40ad3dd9e52d 100644
--- a/llvm/test/CodeGen/X86/tuning-shuffle-unpckpd-avx512.ll
+++ b/llvm/test/CodeGen/X86/tuning-shuffle-unpckpd-avx512.ll
@@ -25,37 +25,117 @@ define <16 x float> @transform_VUNPCKHPDZrr(<16 x float> %a, <16 x float> %b) no
 }
 
 define <8 x float> @transform_VUNPCKLPDYrr(<8 x float> %a, <8 x float> %b) nounwind {
-; CHECK-LABEL: transform_VUNPCKLPDYrr:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vunpcklpd {{.*#+}} ymm0 = ymm0[0],ymm1[0],ymm0[2],ymm1[2]
-; CHECK-NEXT:    retq
+; CHECK-SKX-LABEL: transform_VUNPCKLPDYrr:
+; CHECK-SKX:       # %bb.0:
+; CHECK-SKX-NEXT:    vunpcklpd {{.*#+}} ymm0 = ymm0[0],ymm1[0],ymm0[2],ymm1[2]
+; CHECK-SKX-NEXT:    retq
+;
+; CHECK-ICX-LABEL: transform_VUNPCKLPDYrr:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    vshufps {{.*#+}} ymm0 = ymm0[0,1],ymm1[0,1],ymm0[4,5],ymm1[4,5]
+; CHECK-ICX-NEXT:    retq
+;
+; CHECK-V4-LABEL: transform_VUNPCKLPDYrr:
+; CHECK-V4:       # %bb.0:
+; CHECK-V4-NEXT:    vunpcklpd {{.*#+}} ymm0 = ymm0[0],ymm1[0],ymm0[2],ymm1[2]
+; CHECK-V4-NEXT:    retq
+;
+; CHECK-AVX512-LABEL: transform_VUNPCKLPDYrr:
+; CHECK-AVX512:       # %bb.0:
+; CHECK-AVX512-NEXT:    vunpcklpd {{.*#+}} ymm0 = ymm0[0],ymm1[0],ymm0[2],ymm1[2]
+; CHECK-AVX512-NEXT:    retq
+;
+; CHECK-ZNVER4-LABEL: transform_VUNPCKLPDYrr:
+; CHECK-ZNVER4:       # %bb.0:
+; CHECK-ZNVER4-NEXT:    vunpcklpd {{.*#+}} ymm0 = ymm0[0],ymm1[0],ymm0[2],ymm1[2]
+; CHECK-ZNVER4-NEXT:    retq
   %shufp = shufflevector <8 x float> %a, <8 x float> %b, <8 x i32> <i32 0, i32 1, i32 8, i32 9, i32 4, i32 5, i32 12, i32 13>
   ret <8 x float> %shufp
 }
 
 define <8 x float> @transform_VUNPCKHPDYrr(<8 x float> %a, <8 x float> %b) nounwind {
-; CHECK-LABEL: transform_VUNPCKHPDYrr:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vunpckhpd {{.*#+}} ymm0 = ymm0[1],ymm1[1],ymm0[3],ymm1[3]
-; CHECK-NEXT:    retq
+; CHECK-SKX-LABEL: transform_VUNPCKHPDYrr:
+; CHECK-SKX:       # %bb.0:
+; CHECK-SKX-NEXT:    vunpckhpd {{.*#+}} ymm0 = ymm0[1],ymm1[1],ymm0[3],ymm1[3]
+; CHECK-SKX-NEXT:    retq
+;
+; CHECK-ICX-LABEL: transform_VUNPCKHPDYrr:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    vshufps {{.*#+}} ymm0 = ymm0[2,3],ymm1[2,3],ymm0[6,7],ymm1[6,7]
+; CHECK-ICX-NEXT:    retq
+;
+; CHECK-V4-LABEL: transform_VUNPCKHPDYrr:
+; CHECK-V4:       # %bb.0:
+; CHECK-V4-NEXT:    vunpckhpd {{.*#+}} ymm0 = ymm0[1],ymm1[1],ymm0[3],ymm1[3]
+; CHECK-V4-NEXT:    retq
+;
+; CHECK-AVX512-LABEL: transform_VUNPCKHPDYrr:
+; CHECK-AVX512:       # %bb.0:
+; CHECK-AVX512-NEXT:    vunpckhpd {{.*#+}} ymm0 = ymm0[1],ymm1[1],ymm0[3],ymm1[3]
+; CHECK-AVX512-NEXT:    retq
+;
+; CHECK-ZNVER4-LABEL: transform_VUNPCKHPDYrr:
+; CHECK-ZNVER4:       # %bb.0:
+; CHECK-ZNVER4-NEXT:    vunpckhpd {{.*#+}} ymm0 = ymm0[1],ymm1[1],ymm0[3],ymm1[3]
+; CHECK-ZNVER4-NEXT:    retq
   %shufp = shufflevector <8 x float> %a, <8 x float> %b, <8 x i32> <i32 2, i32 3, i32 10, i32 11, i32 6, i32 7, i32 14, i32 15>
   ret <8 x float> %shufp
 }
 
 define <4 x float> @transform_VUNPCKLPDrr(<4 x float> %a, <4 x float> %b) nounwind {
-; CHECK-LABEL: transform_VUNPCKLPDrr:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vmovlhps {{.*#+}} xmm0 = xmm0[0],xmm1[0]
-; CHECK-NEXT:    retq
+; CHECK-SKX-LABEL: transform_VUNPCKLPDrr:
+; CHECK-SKX:       # %bb.0:
+; CHECK-SKX-NEXT:    vmovlhps {{.*#+}} xmm0 = xmm0[0],xmm1[0]
+; CHECK-SKX-NEXT:    retq
+;
+; CHECK-ICX-LABEL: transform_VUNPCKLPDrr:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    vshufps {{.*#+}} xmm0 = xmm0[0,1],xmm1[0,1]
+; CHECK-ICX-NEXT:    retq
+;
+; CHECK-V4-LABEL: transform_VUNPCKLPDrr:
+; CHECK-V4:       # %bb.0:
+; CHECK-V4-NEXT:    vmovlhps {{.*#+}} xmm0 = xmm0[0],xmm1[0]
+; CHECK-V4-NEXT:    retq
+;
+; CHECK-AVX512-LABEL: transform_VUNPCKLPDrr:
+; CHECK-AVX512:       # %bb.0:
+; CHECK-AVX512-NEXT:    vmovlhps {{.*#+}} xmm0 = xmm0[0],xmm1[0]
+; CHECK-AVX512-NEXT:    retq
+;
+; CHECK-ZNVER4-LABEL: transform_VUNPCKLPDrr:
+; CHECK-ZNVER4:       # %bb.0:
+; CHECK-ZNVER4-NEXT:    vmovlhps {{.*#+}} xmm0 = xmm0[0],xmm1[0]
+; CHECK-ZNVER4-NEXT:    retq
   %shufp = shufflevector <4 x float> %a, <4 x float> %b, <4 x i32> <i32 0, i32 1, i32 4, i32 5>
   ret <4 x float> %shufp
 }
 
 define <4 x float> @transform_VUNPCKHPDrr(<4 x float> %a, <4 x float> %b) nounwind {
-; CHECK-LABEL: transform_VUNPCKHPDrr:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vunpckhpd {{.*#+}} xmm0 = xmm0[1],xmm1[1]
-; CHECK-NEXT:    retq
+; CHECK-SKX-LABEL: transform_VUNPCKHPDrr:
+; CHECK-SKX:       # %bb.0:
+; CHECK-SKX-NEXT:    vunpckhpd {{.*#+}} xmm0 = xmm0[1],xmm1[1]
+; CHECK-SKX-NEXT:    retq
+;
+; CHECK-ICX-LABEL: transform_VUNPCKHPDrr:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    vshufps {{.*#+}} xmm0 = xmm0[2,3],xmm1[2,3]
+; CHECK-ICX-NEXT:    retq
+;
+; CHECK-V4-LABEL: transform_VUNPCKHPDrr:
+; CHECK-V4:       # %bb.0:
+; CHECK-V4-NEXT:    vunpckhpd {{.*#+}} xmm0 = xmm0[1],xmm1[1]
+; CHECK-V4-NEXT:    retq
+;
+; CHECK-AVX512-LABEL: transform_VUNPCKHPDrr:
+; CHECK-AVX512:       # %bb.0:
+; CHECK-AVX512-NEXT:    vunpckhpd {{.*#+}} xmm0 = xmm0[1],xmm1[1]
+; CHECK-AVX512-NEXT:    retq
+;
+; CHECK-ZNVER4-LABEL: transform_VUNPCKHPDrr:
+; CHECK-ZNVER4:       # %bb.0:
+; CHECK-ZNVER4-NEXT:    vunpckhpd {{.*#+}} xmm0 = xmm0[1],xmm1[1]
+; CHECK-ZNVER4-NEXT:    retq
   %shufp = shufflevector <4 x float> %a, <4 x float> %b, <4 x i32> <i32 2, i32 3, i32 6, i32 7>
   ret <4 x float> %shufp
 }
@@ -85,11 +165,29 @@ define <8 x double> @transform_VUNPCKHPDZrrkz(<8 x double> %a, <8 x double> %b,
 }
 
 define <4 x double> @transform_VUNPCKLPDYrrkz(<4 x double> %a, <4 x double> %b, i4 %mask_int) nounwind {
-; CHECK-LABEL: transform_VUNPCKLPDYrrkz:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vunpcklpd {{.*#+}} ymm0 {%k1} {z} = ymm0[0],ymm1[0],ymm0[2],ymm1[2]
-; CHECK-NEXT:    retq
+; CHECK-SKX-LABEL: transform_VUNPCKLPDYrrkz:
+; CHECK-SKX:       # %bb.0:
+; CHECK-SKX-NEXT:    kmovd %edi, %k1
+; CHECK-SKX-NEXT:    vunpcklpd {{.*#+}} ymm0 {%k1} {z} = ymm0[0],ymm1[0],ymm0[2],ymm1[2]
+; CHECK-SKX-NEXT:    retq
+;
+; CHECK-ICX-LABEL: transform_VUNPCKLPDYrrkz:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    kmovd %edi, %k1
+; CHECK-ICX-NEXT:    vshufps {{.*#+}} ymm0 {%k1} {z} = ymm0[0,1],ymm1[0,1],ymm0[4,5],ymm1[4,5]
+; CHECK-ICX-NEXT:    retq
+;
+; CHECK-V4-LABEL: transform_VUNPCKLPDYrrkz:
+; CHECK-V4:       # %bb.0:
+; CHECK-V4-NEXT:    kmovd %edi, %k1
+; CHECK-V4-NEXT:    vunpcklpd {{.*#+}} ymm0 {%k1} {z} = ymm0[0],ymm1[0],ymm0[2],ymm1[2]
+; CHECK-V4-NEXT:    retq
+;
+; CHECK-ZNVER4-LABEL: transform_VUNPCKLPDYrrkz:
+; CHECK-ZNVER4:       # %bb.0:
+; CHECK-ZNVER4-NEXT:    kmovd %edi, %k1
+; CHECK-ZNVER4-NEXT:    vunpcklpd {{.*#+}} ymm0 {%k1} {z} = ymm0[0],ymm1[0],ymm0[2],ymm1[2]
+; CHECK-ZNVER4-NEXT:    retq
   %mask = bitcast i4 %mask_int to <4 x i1>
   %shufp = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 0, i32 4, i32 2, i32 6>
   %res = select <4 x i1> %mask, <4 x double> %shufp, <4 x double> zeroinitializer
@@ -97,11 +195,29 @@ define <4 x double> @transform_VUNPCKLPDYrrkz(<4 x double> %a, <4 x double> %b,
 }
 
 define <4 x double> @transform_VUNPCKHPDYrrkz(<4 x double> %a, <4 x double> %b, i4 %mask_int) nounwind {
-; CHECK-LABEL: transform_VUNPCKHPDYrrkz:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vunpckhpd {{.*#+}} ymm0 {%k1} {z} = ymm0[1],ymm1[1],ymm0[3],ymm1[3]
-; CHECK-NEXT:    retq
+; CHECK-SKX-LABEL: transform_VUNPCKHPDYrrkz:
+; CHECK-SKX:       # %bb.0:
+; CHECK-SKX-NEXT:    kmovd %edi, %k1
+; CHECK-SKX-NEXT:    vunpckhpd {{.*#+}} ymm0 {%k1} {z} = ymm0[1],ymm1[1],ymm0[3],ymm1[3]
+; CHECK-SKX-NEXT:    retq
+;
+; CHECK-ICX-LABEL: transform_VUNPCKHPDYrrkz:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    kmovd %edi, %k1
+; CHECK-ICX-NEXT:    vshufps {{.*#+}} ymm0 {%k1} {z} = ymm0[2,3],ymm1[2,3],ymm0[6,7],ymm1[6,7]
+; CHECK-ICX-NEXT:    retq
+;
+; CHECK-V4-LABEL: transform_VUNPCKHPDYrrkz:
+; CHECK-V4:       # %bb.0:
+; CHECK-V4-NEXT:    kmovd %edi, %k1
+; CHECK-V4-NEXT:    vunpckhpd {{.*#+}} ymm0 {%k1} {z} = ymm0[1],ymm1[1],ymm0[3],ymm1[3]
+; CHECK-V4-NEXT:    retq
+;
+; CHECK-ZNVER4-LABEL: transform_VUNPCKHPDYrrkz:
+; CHECK-ZNVER4:       # %bb.0:
+; CHECK-ZNVER4-NEXT:    kmovd %edi, %k1
+; CHECK-ZNVER4-NEXT:    vunpckhpd {{.*#+}} ymm0 {%k1} {z} = ymm0[1],ymm1[1],ymm0[3],ymm1[3]
+; CHECK-ZNVER4-NEXT:    retq
   %mask = bitcast i4 %mask_int to <4 x i1>
   %shufp = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 1, i32 5, i32 3, i32 7>
   %res = select <4 x i1> %mask, <4 x double> %shufp, <4 x double> zeroinitializer
@@ -109,11 +225,29 @@ define <4 x double> @transform_VUNPCKHPDYrrkz(<4 x double> %a, <4 x double> %b,
 }
 
 define <2 x double> @transform_VUNPCKLPDrrkz(<2 x double> %a, <2 x double> %b, i2 %mask_int) nounwind {
-; CHECK-LABEL: transform_VUNPCKLPDrrkz:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vunpcklpd {{.*#+}} xmm0 {%k1} {z} = xmm0[0],xmm1[0]
-; CHECK-NEXT:    retq
+; CHECK-SKX-LABEL: transform_VUNPCKLPDrrkz:
+; CHECK-SKX:       # %bb.0:
+; CHECK-SKX-NEXT:    kmovd %edi, %k1
+; CHECK-SKX-NEXT:    vunpcklpd {{.*#+}} xmm0 {%k1} {z} = xmm0[0],xmm1[0]
+; CHECK-SKX-NEXT:    retq
+;
+; CHECK-ICX-LABEL: transform_VUNPCKLPDrrkz:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    kmovd %edi, %k1
+; CHECK-ICX-NEXT:    vshufps {{.*#+}} xmm0 {%k1} {z} = xmm0[0,1],xmm1[0,1]
+; CHECK-ICX-NEXT:    retq
+;
+; CHECK-V4-LABEL: transform_VUNPCKLPDrrkz:
+; CHECK-V4:       # %bb.0:
+; CHECK-V4-NEXT:    kmovd %edi, %k1
+; CHECK-V4-NEXT:    vunpcklpd {{.*#+}} xmm0 {%k1} {z} = xmm0[0],xmm1[0]
+; CHECK-V4-NEXT:    retq
+;
+; CHECK-ZNVER4-LABEL: transform_VUNPCKLPDrrkz:
+; CHECK-ZNVER4:       # %bb.0:
+; CHECK-ZNVER4-NEXT:    kmovd %edi, %k1
+; CHECK-ZNVER4-NEXT:    vunpcklpd {{.*#+}} xmm0 {%k1} {z} = xmm0[0],xmm1[0]
+; CHECK-ZNVER4-NEXT:    retq
   %mask = bitcast i2 %mask_int to <2 x i1>
   %shufp = shufflevector <2 x double> %a, <2 x double> %b, <2 x i32> <i32 0, i32 2>
   %res = select <2 x i1> %mask, <2 x double> %shufp, <2 x double> zeroinitializer
@@ -121,11 +255,29 @@ define <2 x double> @transform_VUNPCKLPDrrkz(<2 x double> %a, <2 x double> %b, i
 }
 
 define <2 x double> @transform_VUNPCKHPDrrkz(<2 x double> %a, <2 x double> %b, i2 %mask_int) nounwind {
-; CHECK-LABEL: transform_VUNPCKHPDrrkz:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vunpckhpd {{.*#+}} xmm0 {%k1} {z} = xmm0[1],xmm1[1]
-; CHECK-NEXT:    retq
+; CHECK-SKX-LABEL: transform_VUNPCKHPDrrkz:
+; CHECK-SKX:       # %bb.0:
+; CHECK-SKX-NEXT:    kmovd %edi, %k1
+; CHECK-SKX-NEXT:    vunpckhpd {{.*#+}} xmm0 {%k1} {z} = xmm0[1],xmm1[1]
+; CHECK-SKX-NEXT:    retq
+;
+; CHECK-ICX-LABEL: transform_VUNPCKHPDrrkz:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    kmovd %edi, %k1
+; CHECK-ICX-NEXT:    vshufps {{.*#+}} xmm0 {%k1} {z} = xmm0[2,3],xmm1[2,3]
+; CHECK-ICX-NEXT:    retq
+;
+; CHECK-V4-LABEL: transform_VUNPCKHPDrrkz:
+; CHECK-V4:       # %bb.0:
+; CHECK-V4-NEXT:    kmovd %edi, %k1
+; CHECK-V4-NEXT:    vunpckhpd {{.*#+}} xmm0 {%k1} {z} = xmm0[1],xmm1[1]
+; CHECK-V4-NEXT:    retq
+;
+; CHECK-ZNVER4-LABEL: transform_VUNPCKHPDrrkz:
+; CHECK-ZNVER4:       # %bb.0:
+; CHECK-ZNVER4-NEXT:    kmovd %edi, %k1
+; CHECK-ZNVER4-NEXT:    vunpckhpd {{.*#+}} xmm0 {%k1} {z} = xmm0[1],xmm1[1]
+; CHECK-ZNVER4-NEXT:    retq
   %mask = bitcast i2 %mask_int to <2 x i1>
   %shufp = shufflevector <2 x double> %a, <2 x double> %b, <2 x i32> <i32 1, i32 3>
   %res = select <2 x i1> %mask, <2 x double> %shufp, <2 x double> zeroinitializer
@@ -159,12 +311,33 @@ define <8 x double> @transform_VUNPCKHPDZrrk(<8 x double> %a, <8 x double> %b, <
 }
 
 define <4 x double> @transform_VUNPCKLPDYrrk(<4 x double> %a, <4 x double> %b, <4 x double> %c, i4 %mask_int) nounwind {
-; CHECK-LABEL: transform_VUNPCKLPDYrrk:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vunpcklpd {{.*#+}} ymm2 {%k1} = ymm0[0],ymm1[0],ymm0[2],ymm1[2]
-; CHECK-NEXT:    vmovapd %ymm2, %ymm0
-; CHECK-NEXT:    retq
+; CHECK-SKX-LABEL: transform_VUNPCKLPDYrrk:
+; CHECK-SKX:       # %bb.0:
+; CHECK-SKX-NEXT:    kmovd %edi, %k1
+; CHECK-SKX-NEXT:    vunpcklpd {{.*#+}} ymm2 {%k1} = ymm0[0],ymm1[0],ymm0[2],ymm1[2]
+; CHECK-SKX-NEXT:    vmovapd %ymm2, %ymm0
+; CHECK-SKX-NEXT:    retq
+;
+; CHECK-ICX-LABEL: transform_VUNPCKLPDYrrk:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    kmovd %edi, %k1
+; CHECK-ICX-NEXT:    vshufps {{.*#+}} ymm2 {%k1} = ymm0[0,1],ymm1[0,1],ymm0[4,5],ymm1[4,5]
+; CHECK-ICX-NEXT:    vmovapd %ymm2, %ymm0
+; CHECK-ICX-NEXT:    retq
+;
+; CHECK-V4-LABEL: transform_VUNPCKLPDYrrk:
+; CHECK-V4:       # %bb.0:
+; CHECK-V4-NEXT:    kmovd %edi, %k1
+; CHECK-V4-NEXT:    vunpcklpd {{.*#+}} ymm2 {%k1} = ymm0[0],ymm1[0],ymm0[2],ymm1[2]
+; CHECK-V4-NEXT:    vmovapd %ymm2, %ymm0
+; CHECK-V4-NEXT:    retq
+;
+; CHECK-ZNVER4-LABEL: transform_VUNPCKLPDYrrk:
+; CHECK-ZNVER4:       # %bb.0:
+; CHECK-ZNVER4-NEXT:    kmovd %edi, %k1
+; CHECK-ZNVER4-NEXT:    vunpcklpd {{.*#+}} ymm2 {%k1} = ymm0[0],ymm1[0],ymm0[2],ymm1[2]
+; CHECK-ZNVER4-NEXT:    vmovapd %ymm2, %ymm0
+; CHECK-ZNVER4-NEXT:    retq
   %mask = bitcast i4 %mask_int to <4 x i1>
   %shufp = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 0, i32 4, i32 2, i32 6>
   %res = select <4 x i1> %mask, <4 x double> %shufp, <4 x double> %c
@@ -172,12 +345,33 @@ define <4 x double> @transform_VUNPCKLPDYrrk(<4 x double> %a, <4 x double> %b, <
 }
 
 define <4 x double> @transform_VUNPCKHPDYrrk(<4 x double> %a, <4 x double> %b, <4 x double> %c, i4 %mask_int) nounwind {
-; CHECK-LABEL: transform_VUNPCKHPDYrrk:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vunpckhpd {{.*#+}} ymm2 {%k1} = ymm0[1],ymm1[1],ymm0[3],ymm1[3]
-; CHECK-NEXT:    vmovapd %ymm2, %ymm0
-; CHECK-NEXT:    retq
+; CHECK-SKX-LABEL: transform_VUNPCKHPDYrrk:
+; CHECK-SKX:       # %bb.0:
+; CHECK-SKX-NEXT:    kmovd %edi, %k1
+; CHECK-SKX-NEXT:    vunpckhpd {{.*#+}} ymm2 {%k1} = ymm0[1],ymm1[1],ymm0[3],ymm1[3]
+; CHECK-SKX-NEXT:    vmovapd %ymm2, %ymm0
+; CHECK-SKX-NEXT:    retq
+;
+; CHECK-ICX-LABEL: transform_VUNPCKHPDYrrk:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    kmovd %edi, %k1
+; CHECK-ICX-NEXT:    vshufps {{.*#+}} ymm2 {%k1} = ymm0[2,3],ymm1[2,3],ymm0[6,7],ymm1[6,7]
+; CHECK-ICX-NEXT:    vmovapd %ymm2, %ymm0
+; CHECK-ICX-NEXT:    retq
+;
+; CHECK-V4-LABEL: transform_VUNPCKHPDYrrk:
+; CHECK-V4:       # %bb.0:
+; CHECK-V4-NEXT:    kmovd %edi, %k1
+; CHECK-V4-NEXT:    vunpckhpd {{.*#+}} ymm2 {%k1} = ymm0[1],ymm1[1],ymm0[3],ymm1[3]
+; CHECK-V4-NEXT:    vmovapd %ymm2, %ymm0
+; CHECK-V4-NEXT:    retq
+;
+; CHECK-ZNVER4-LABEL: transform_VUNPCKHPDYrrk:
+; CHECK-ZNVER4:       # %bb.0:
+; CHECK-ZNVER4-NEXT:    kmovd %edi, %k1
+; CHECK-ZNVER4-NEXT:    vunpckhpd {{.*#+}} ymm2 {%k1} = ymm0[1],ymm1[1],ymm0[3],ymm1[3]
+; CHECK-ZNVER4-NEXT:    vmovapd %ymm2, %ymm0
+; CHECK-ZNVER4-NEXT:    retq
   %mask = bitcast i4 %mask_int to <4 x i1>
   %shufp = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 1, i32 5, i32 3, i32 7>
   %res = select <4 x i1> %mask, <4 x double> %shufp, <4 x double> %c
@@ -185,12 +379,33 @@ define <4 x double> @transform_VUNPCKHPDYrrk(<4 x double> %a, <4 x double> %b, <
 }
 
 define <2 x double> @transform_VUNPCKLPDrrk(<2 x double> %a, <2 x double> %b, <2 x double> %c, i2 %mask_int) nounwind {
-; CHECK-LABEL: transform_VUNPCKLPDrrk:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vunpcklpd {{.*#+}} xmm2 {%k1} = xmm0[0],xmm1[0]
-; CHECK-NEXT:    vmovapd %xmm2, %xmm0
-; CHECK-NEXT:    retq
+; CHECK-SKX-LABEL: transform_VUNPCKLPDrrk:
+; CHECK-SKX:       # %bb.0:
+; CHECK-SKX-NEXT:    kmovd %edi, %k1
+; CHECK-SKX-NEXT:    vunpcklpd {{.*#+}} xmm2 {%k1} = xmm0[0],xmm1[0]
+; CHECK-SKX-NEXT:    vmovapd %xmm2, %xmm0
+; CHECK-SKX-NEXT:    retq
+;
+; CHECK-ICX-LABEL: transform_VUNPCKLPDrrk:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    kmovd %edi, %k1
+; CHECK-ICX-NEXT:    vshufps {{.*#+}} xmm2 {%k1} = xmm0[0,1],xmm1[0,1]
+; CHECK-ICX-NEXT:    vmovapd %xmm2, %xmm0
+; CHECK-ICX-NEXT:    retq
+;
+; CHECK-V4-LABEL: transform_VUNPCKLPDrrk:
+; CHECK-V4:       # %bb.0:
+; CHECK-V4-NEXT:    kmovd %edi, %k1
+; CHECK-V4-NEXT:    vunpcklpd {{.*#+}} xmm2 {%k1} = xmm0[0],xmm1[0]
+; CHECK-V4-NEXT:    vmovapd %xmm2, %xmm0
+; CHECK-V4-NEXT:    retq
+;
+; CHECK-ZNVER4-LABEL: transform_VUNPCKLPDrrk:
+; CHECK-ZNVER4:       # %bb.0:
+; CHECK-ZNVER4-NEXT:    kmovd %edi, %k1
+; CHECK-ZNVER4-NEXT:    vunpcklpd {{.*#+}} xmm2 {%k1} = xmm0[0],xmm1[0]
+; CHECK-ZNVER4-NEXT:    vmovapd %xmm2, %xmm0
+; CHECK-ZNVER4-NEXT:    retq
   %mask = bitcast i2 %mask_int to <2 x i1>
   %shufp = shufflevector <2 x double> %a, <2 x double> %b, <2 x i32> <i32 0, i32 2>
   %res = select <2 x i1> %mask, <2 x double> %shufp, <2 x double> %c
@@ -198,12 +413,33 @@ define <2 x double> @transform_VUNPCKLPDrrk(<2 x double> %a, <2 x double> %b, <2
 }
 
 define <2 x double> @transform_VUNPCKHPDrrk(<2 x double> %a, <2 x double> %b, <2 x double> %c, i2 %mask_int) nounwind {
-; CHECK-LABEL: transform_VUNPCKHPDrrk:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vunpckhpd {{.*#+}} xmm2 {%k1} = xmm0[1],xmm1[1]
-; CHECK-NEXT:    vmovapd %xmm2, %xmm0
-; CHECK-NEXT:    retq
+; CHECK-SKX-LABEL: transform_VUNPCKHPDrrk:
+; CHECK-SKX:       # %bb.0:
+; CHECK-SKX-NEXT:    kmovd %edi, %k1
+; CHECK-SKX-NEXT:    vunpckhpd {{.*#+}} xmm2 {%k1} = xmm0[1],xmm1[1]
+; CHECK-SKX-NEXT:    vmovapd %xmm2, %xmm0
+; CHECK-SKX-NEXT:    retq
+;
+; CHECK-ICX-LABEL: transform_VUNPCKHPDrrk:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    kmovd %edi, %k1
+; CHECK-ICX-NEXT:    vshufps {{.*#+}} xmm2 {%k1} = xmm0[2,3],xmm1[2,3]
+; CHECK-ICX-NEXT:    vmovapd %xmm2, %xmm0
+; CHECK-ICX-NEXT:    retq
+;
+; CHECK-V4-LABEL: transform_VUNPCKHPDrrk:
+; CHECK-V4:       # %bb.0:
+; CHECK-V4-NEXT:    kmovd %edi, %k1
+; CHECK-V4-NEXT:    vunpckhpd {{.*#+}} xmm2 {%k1} = xmm0[1],xmm1[1]
+; CHECK-V4-NEXT:    vmovapd %xmm2, %xmm0
+; CHECK-V4-NEXT:    retq
+;
+; CHECK-ZNVER4-LABEL: transform_VUNPCKHPDrrk:
+; CHECK-ZNVER4:       # %bb.0:
+; CHECK-ZNVER4-NEXT:    kmovd %edi, %k1
+; CHECK-ZNVER4-NEXT:    vunpckhpd {{.*#+}} xmm2 {%k1} = xmm0[1],xmm1[1]
+; CHECK-ZNVER4-NEXT:    vmovapd %xmm2, %xmm0
+; CHECK-ZNVER4-NEXT:    retq
   %mask = bitcast i2 %mask_int to <2 x i1>
   %shufp = shufflevector <2 x double> %a, <2 x double> %b, <2 x i32> <i32 1, i32 3>
   %res = select <2 x i1> %mask, <2 x double> %shufp, <2 x double> %c
@@ -431,9 +667,3 @@ define <2 x double> @transform_VUNPCKHPDrmk(<2 x double> %a, ptr %pb, <2 x doubl
   %res = select <2 x i1> %mask, <2 x double> %shufp, <2 x double> %c
   ret <2 x double> %res
 }
-;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
-; CHECK-AVX512: {{.*}}
-; CHECK-ICX: {{.*}}
-; CHECK-SKX: {{.*}}
-; CHECK-V4: {{.*}}
-; CHECK-ZNVER4: {{.*}}
diff --git a/llvm/test/CodeGen/X86/tuning-shuffle-unpckpd.ll b/llvm/test/CodeGen/X86/tuning-shuffle-unpckpd.ll
index c5dcd55dc7be1..9de90fd8da4a9 100644
--- a/llvm/test/CodeGen/X86/tuning-shuffle-unpckpd.ll
+++ b/llvm/test/CodeGen/X86/tuning-shuffle-unpckpd.ll
@@ -1,39 +1,60 @@
 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mcpu=skylake  | FileCheck %s --check-prefixes=CHECK,CHECK-SKL
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mcpu=x86-64-v3  | FileCheck %s --check-prefixes=CHECK,CHECK-V3
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mcpu=skylake  | FileCheck %s --check-prefixes=CHECK,CHECK-AVX2,CHECK-SKL
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mcpu=x86-64-v3  | FileCheck %s --check-prefixes=CHECK,CHECK-AVX2,CHECK-V3
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mcpu=icelake-server  | FileCheck %s --check-prefixes=CHECK,CHECK-ICX
 
 define <8 x float> @transform_VUNPCKLPDYrr(<8 x float> %a, <8 x float> %b) nounwind {
-; CHECK-LABEL: transform_VUNPCKLPDYrr:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vunpcklpd {{.*#+}} ymm0 = ymm0[0],ymm1[0],ymm0[2],ymm1[2]
-; CHECK-NEXT:    retq
+; CHECK-AVX2-LABEL: transform_VUNPCKLPDYrr:
+; CHECK-AVX2:       # %bb.0:
+; CHECK-AVX2-NEXT:    vunpcklpd {{.*#+}} ymm0 = ymm0[0],ymm1[0],ymm0[2],ymm1[2]
+; CHECK-AVX2-NEXT:    retq
+;
+; CHECK-ICX-LABEL: transform_VUNPCKLPDYrr:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    vshufps {{.*#+}} ymm0 = ymm0[0,1],ymm1[0,1],ymm0[4,5],ymm1[4,5]
+; CHECK-ICX-NEXT:    retq
   %shufp = shufflevector <8 x float> %a, <8 x float> %b, <8 x i32> <i32 0, i32 1, i32 8, i32 9, i32 4, i32 5, i32 12, i32 13>
   ret <8 x float> %shufp
 }
 
 define <8 x float> @transform_VUNPCKHPDYrr(<8 x float> %a, <8 x float> %b) nounwind {
-; CHECK-LABEL: transform_VUNPCKHPDYrr:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vunpckhpd {{.*#+}} ymm0 = ymm0[1],ymm1[1],ymm0[3],ymm1[3]
-; CHECK-NEXT:    retq
+; CHECK-AVX2-LABEL: transform_VUNPCKHPDYrr:
+; CHECK-AVX2:       # %bb.0:
+; CHECK-AVX2-NEXT:    vunpckhpd {{.*#+}} ymm0 = ymm0[1],ymm1[1],ymm0[3],ymm1[3]
+; CHECK-AVX2-NEXT:    retq
+;
+; CHECK-ICX-LABEL: transform_VUNPCKHPDYrr:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    vshufps {{.*#+}} ymm0 = ymm0[2,3],ymm1[2,3],ymm0[6,7],ymm1[6,7]
+; CHECK-ICX-NEXT:    retq
   %shufp = shufflevector <8 x float> %a, <8 x float> %b, <8 x i32> <i32 2, i32 3, i32 10, i32 11, i32 6, i32 7, i32 14, i32 15>
   ret <8 x float> %shufp
 }
 
 define <4 x float> @transform_VUNPCKLPDrr(<4 x float> %a, <4 x float> %b) nounwind {
-; CHECK-LABEL: transform_VUNPCKLPDrr:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vmovlhps {{.*#+}} xmm0 = xmm0[0],xmm1[0]
-; CHECK-NEXT:    retq
+; CHECK-AVX2-LABEL: transform_VUNPCKLPDrr:
+; CHECK-AVX2:       # %bb.0:
+; CHECK-AVX2-NEXT:    vmovlhps {{.*#+}} xmm0 = xmm0[0],xmm1[0]
+; CHECK-AVX2-NEXT:    retq
+;
+; CHECK-ICX-LABEL: transform_VUNPCKLPDrr:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    vshufps {{.*#+}} xmm0 = xmm0[0,1],xmm1[0,1]
+; CHECK-ICX-NEXT:    retq
   %shufp = shufflevector <4 x float> %a, <4 x float> %b, <4 x i32> <i32 0, i32 1, i32 4, i32 5>
   ret <4 x float> %shufp
 }
 
 define <4 x float> @transform_VUNPCKHPDrr(<4 x float> %a, <4 x float> %b) nounwind {
-; CHECK-LABEL: transform_VUNPCKHPDrr:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vunpckhpd {{.*#+}} xmm0 = xmm0[1],xmm1[1]
-; CHECK-NEXT:    retq
+; CHECK-AVX2-LABEL: transform_VUNPCKHPDrr:
+; CHECK-AVX2:       # %bb.0:
+; CHECK-AVX2-NEXT:    vunpckhpd {{.*#+}} xmm0 = xmm0[1],xmm1[1]
+; CHECK-AVX2-NEXT:    retq
+;
+; CHECK-ICX-LABEL: transform_VUNPCKHPDrr:
+; CHECK-ICX:       # %bb.0:
+; CHECK-ICX-NEXT:    vshufps {{.*#+}} xmm0 = xmm0[2,3],xmm1[2,3]
+; CHECK-ICX-NEXT:    retq
   %shufp = shufflevector <4 x float> %a, <4 x float> %b, <4 x i32> <i32 2, i32 3, i32 6, i32 7>
   ret <4 x float> %shufp
 }
diff --git a/llvm/test/CodeGen/X86/umax.ll b/llvm/test/CodeGen/X86/umax.ll
index e4b465449e259..e37760d1e0b09 100644
--- a/llvm/test/CodeGen/X86/umax.ll
+++ b/llvm/test/CodeGen/X86/umax.ll
@@ -17,6 +17,7 @@ declare <3 x i32> @llvm.umax.v3i32(<3 x i32>, <3 x i32>)
 declare <4 x i32> @llvm.umax.v4i32(<4 x i32>, <4 x i32>)
 declare <8 x i32> @llvm.umax.v8i32(<8 x i32>, <8 x i32>)
 
+declare <2 x i64> @llvm.umax.v2i64(<2 x i64>, <2 x i64>)
 declare <8 x i16> @llvm.umax.v8i16(<8 x i16>, <8 x i16>)
 declare <16 x i8> @llvm.umax.v16i8(<16 x i8>, <16 x i8>)
 
@@ -44,18 +45,17 @@ define i8 @test_i8(i8 %a, i8 %b) nounwind {
 define i8 @test_i8_1(i8 %a) nounwind {
 ; X64-LABEL: test_i8_1:
 ; X64:       # %bb.0:
-; X64-NEXT:    cmpb $2, %dil
-; X64-NEXT:    movl $1, %eax
-; X64-NEXT:    cmovael %edi, %eax
+; X64-NEXT:    movl %edi, %eax
+; X64-NEXT:    cmpb $1, %al
+; X64-NEXT:    adcl $0, %eax
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: test_i8_1:
 ; X86:       # %bb.0:
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X86-NEXT:    cmpb $2, %cl
-; X86-NEXT:    movl $1, %eax
-; X86-NEXT:    cmovael %ecx, %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    cmpb $1, %al
+; X86-NEXT:    adcl $0, %eax
 ; X86-NEXT:    # kill: def $al killed $al killed $eax
 ; X86-NEXT:    retl
   %r = call i8 @llvm.umax.i8(i8 %a, i8 1)
@@ -86,18 +86,17 @@ define i16 @test_i16(i16 %a, i16 %b) nounwind {
 define i16 @test_i16_1(i16 %a) nounwind {
 ; X64-LABEL: test_i16_1:
 ; X64:       # %bb.0:
-; X64-NEXT:    cmpw $2, %di
-; X64-NEXT:    movl $1, %eax
-; X64-NEXT:    cmovael %edi, %eax
+; X64-NEXT:    movl %edi, %eax
+; X64-NEXT:    cmpw $1, %ax
+; X64-NEXT:    adcl $0, %eax
 ; X64-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: test_i16_1:
 ; X86:       # %bb.0:
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X86-NEXT:    cmpw $2, %cx
-; X86-NEXT:    movl $1, %eax
-; X86-NEXT:    cmovael %ecx, %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    cmpw $1, %ax
+; X86-NEXT:    adcl $0, %eax
 ; X86-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X86-NEXT:    retl
   %r = call i16 @llvm.umax.i16(i16 %a, i16 1)
@@ -149,17 +148,16 @@ define i32 @test_i32(i32 %a, i32 %b) nounwind {
 define i32 @test_i32_1(i32 %a) nounwind {
 ; X64-LABEL: test_i32_1:
 ; X64:       # %bb.0:
-; X64-NEXT:    cmpl $2, %edi
-; X64-NEXT:    movl $1, %eax
-; X64-NEXT:    cmovael %edi, %eax
+; X64-NEXT:    movl %edi, %eax
+; X64-NEXT:    cmpl $1, %edi
+; X64-NEXT:    adcl $0, %eax
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: test_i32_1:
 ; X86:       # %bb.0:
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X86-NEXT:    cmpl $2, %ecx
-; X86-NEXT:    movl $1, %eax
-; X86-NEXT:    cmovael %ecx, %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    cmpl $1, %eax
+; X86-NEXT:    adcl $0, %eax
 ; X86-NEXT:    retl
   %r = call i32 @llvm.umax.i32(i32 %a, i32 1)
   ret i32 %r
@@ -198,9 +196,9 @@ define i64 @test_i64(i64 %a, i64 %b) nounwind {
 define i64 @test_i64_1(i64 %a) nounwind {
 ; X64-LABEL: test_i64_1:
 ; X64:       # %bb.0:
-; X64-NEXT:    cmpq $2, %rdi
-; X64-NEXT:    movl $1, %eax
-; X64-NEXT:    cmovaeq %rdi, %rax
+; X64-NEXT:    movq %rdi, %rax
+; X64-NEXT:    cmpq $1, %rdi
+; X64-NEXT:    adcq $0, %rax
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: test_i64_1:
@@ -208,11 +206,11 @@ define i64 @test_i64_1(i64 %a) nounwind {
 ; X86-NEXT:    pushl %esi
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; X86-NEXT:    cmpl $2, %ecx
-; X86-NEXT:    movl $1, %eax
-; X86-NEXT:    movl $1, %esi
-; X86-NEXT:    cmovael %ecx, %esi
+; X86-NEXT:    cmpl $1, %ecx
+; X86-NEXT:    movl %ecx, %esi
+; X86-NEXT:    adcl $0, %esi
 ; X86-NEXT:    testl %edx, %edx
+; X86-NEXT:    movl $1, %eax
 ; X86-NEXT:    cmovnel %ecx, %eax
 ; X86-NEXT:    cmovel %esi, %eax
 ; X86-NEXT:    popl %esi
@@ -295,6 +293,187 @@ define i128 @test_i128(i128 %a, i128 %b) nounwind {
   ret i128 %r
 }
 
+define i128 @test_i128_1(i128 %a) nounwind {
+; X64-LABEL: test_i128_1:
+; X64:       # %bb.0:
+; X64-NEXT:    movq %rsi, %rdx
+; X64-NEXT:    cmpq $1, %rdi
+; X64-NEXT:    movq %rdi, %rcx
+; X64-NEXT:    adcq $0, %rcx
+; X64-NEXT:    testq %rsi, %rsi
+; X64-NEXT:    movl $1, %eax
+; X64-NEXT:    cmovneq %rdi, %rax
+; X64-NEXT:    cmoveq %rcx, %rax
+; X64-NEXT:    retq
+;
+; X86-LABEL: test_i128_1:
+; X86:       # %bb.0:
+; X86-NEXT:    pushl %ebp
+; X86-NEXT:    pushl %ebx
+; X86-NEXT:    pushl %edi
+; X86-NEXT:    pushl %esi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
+; X86-NEXT:    cmpl $1, %eax
+; X86-NEXT:    movl %eax, %ebx
+; X86-NEXT:    adcl $0, %ebx
+; X86-NEXT:    testl %esi, %esi
+; X86-NEXT:    movl $1, %edi
+; X86-NEXT:    cmovnel %eax, %edi
+; X86-NEXT:    cmovel %ebx, %edi
+; X86-NEXT:    xorl %ebx, %ebx
+; X86-NEXT:    movl %ecx, %ebp
+; X86-NEXT:    negl %ebp
+; X86-NEXT:    movl $0, %ebp
+; X86-NEXT:    sbbl %edx, %ebp
+; X86-NEXT:    movl $1, %ebp
+; X86-NEXT:    cmovbl %eax, %ebp
+; X86-NEXT:    movl %ecx, %eax
+; X86-NEXT:    cmovbl %esi, %ebx
+; X86-NEXT:    orl %edx, %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movl %edx, 12(%eax)
+; X86-NEXT:    movl %ecx, 8(%eax)
+; X86-NEXT:    cmovel %edi, %ebp
+; X86-NEXT:    cmovel %esi, %ebx
+; X86-NEXT:    movl %ebx, 4(%eax)
+; X86-NEXT:    movl %ebp, (%eax)
+; X86-NEXT:    popl %esi
+; X86-NEXT:    popl %edi
+; X86-NEXT:    popl %ebx
+; X86-NEXT:    popl %ebp
+; X86-NEXT:    retl $4
+  %r = call i128 @llvm.umax.i128(i128 %a, i128 1)
+  ret i128 %r
+}
+
+define <2 x i64> @test_v2i64(<2 x i64> %a, <2 x i64> %b) nounwind {
+; SSE-LABEL: test_v2i64:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movdqa {{.*#+}} xmm2 = [9223372039002259456,9223372039002259456]
+; SSE-NEXT:    movdqa %xmm1, %xmm3
+; SSE-NEXT:    pxor %xmm2, %xmm3
+; SSE-NEXT:    pxor %xmm0, %xmm2
+; SSE-NEXT:    movdqa %xmm2, %xmm4
+; SSE-NEXT:    pcmpgtd %xmm3, %xmm4
+; SSE-NEXT:    pshufd {{.*#+}} xmm5 = xmm4[0,0,2,2]
+; SSE-NEXT:    pcmpeqd %xmm3, %xmm2
+; SSE-NEXT:    pshufd {{.*#+}} xmm2 = xmm2[1,1,3,3]
+; SSE-NEXT:    pand %xmm5, %xmm2
+; SSE-NEXT:    pshufd {{.*#+}} xmm3 = xmm4[1,1,3,3]
+; SSE-NEXT:    por %xmm2, %xmm3
+; SSE-NEXT:    pand %xmm3, %xmm0
+; SSE-NEXT:    pandn %xmm1, %xmm3
+; SSE-NEXT:    por %xmm3, %xmm0
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: test_v2i64:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vmovdqa {{.*#+}} xmm2 = [9223372036854775808,9223372036854775808]
+; AVX-NEXT:    vpxor %xmm2, %xmm1, %xmm3
+; AVX-NEXT:    vpxor %xmm2, %xmm0, %xmm2
+; AVX-NEXT:    vpcmpgtq %xmm3, %xmm2, %xmm2
+; AVX-NEXT:    vblendvpd %xmm2, %xmm0, %xmm1, %xmm0
+; AVX-NEXT:    retq
+;
+; X86-LABEL: test_v2i64:
+; X86:       # %bb.0:
+; X86-NEXT:    pushl %ebp
+; X86-NEXT:    pushl %ebx
+; X86-NEXT:    pushl %edi
+; X86-NEXT:    pushl %esi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebx
+; X86-NEXT:    cmpl %edx, %eax
+; X86-NEXT:    movl %edx, %ebp
+; X86-NEXT:    cmoval %eax, %ebp
+; X86-NEXT:    cmpl %edi, %ebx
+; X86-NEXT:    cmoval %eax, %edx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    cmovel %ebp, %edx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebp
+; X86-NEXT:    cmoval %ebx, %edi
+; X86-NEXT:    cmpl %ecx, %eax
+; X86-NEXT:    movl %ecx, %ebx
+; X86-NEXT:    cmoval %eax, %ebx
+; X86-NEXT:    cmpl %esi, %ebp
+; X86-NEXT:    cmoval %eax, %ecx
+; X86-NEXT:    cmovel %ebx, %ecx
+; X86-NEXT:    cmoval %ebp, %esi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movl %esi, 12(%eax)
+; X86-NEXT:    movl %ecx, 8(%eax)
+; X86-NEXT:    movl %edi, 4(%eax)
+; X86-NEXT:    movl %edx, (%eax)
+; X86-NEXT:    popl %esi
+; X86-NEXT:    popl %edi
+; X86-NEXT:    popl %ebx
+; X86-NEXT:    popl %ebp
+; X86-NEXT:    retl $4
+  %r = call <2 x i64> @llvm.umax.v2i64(<2 x i64> %a, <2 x i64> %b)
+  ret <2 x i64> %r
+}
+
+define <2 x i64> @test_v2i64_1(<2 x i64> %a) nounwind {
+; SSE-LABEL: test_v2i64_1:
+; SSE:       # %bb.0:
+; SSE-NEXT:    pxor %xmm1, %xmm1
+; SSE-NEXT:    pcmpeqd %xmm0, %xmm1
+; SSE-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[1,0,3,2]
+; SSE-NEXT:    pand %xmm1, %xmm2
+; SSE-NEXT:    psubq %xmm2, %xmm0
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: test_v2i64_1:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; AVX-NEXT:    vpcmpeqq %xmm1, %xmm0, %xmm1
+; AVX-NEXT:    vpsubq %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    retq
+;
+; X86-LABEL: test_v2i64_1:
+; X86:       # %bb.0:
+; X86-NEXT:    pushl %ebp
+; X86-NEXT:    pushl %ebx
+; X86-NEXT:    pushl %edi
+; X86-NEXT:    pushl %esi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
+; X86-NEXT:    cmpl $1, %ecx
+; X86-NEXT:    movl %ecx, %ebx
+; X86-NEXT:    adcl $0, %ebx
+; X86-NEXT:    testl %esi, %esi
+; X86-NEXT:    movl $1, %ebp
+; X86-NEXT:    cmovel %ebp, %ecx
+; X86-NEXT:    cmovel %ebx, %ecx
+; X86-NEXT:    cmpl $1, %edi
+; X86-NEXT:    movl %edi, %ebx
+; X86-NEXT:    adcl $0, %ebx
+; X86-NEXT:    testl %edx, %edx
+; X86-NEXT:    cmovnel %edi, %ebp
+; X86-NEXT:    cmovel %ebx, %ebp
+; X86-NEXT:    movl %edx, 12(%eax)
+; X86-NEXT:    movl %ebp, 8(%eax)
+; X86-NEXT:    movl %esi, 4(%eax)
+; X86-NEXT:    movl %ecx, (%eax)
+; X86-NEXT:    popl %esi
+; X86-NEXT:    popl %edi
+; X86-NEXT:    popl %ebx
+; X86-NEXT:    popl %ebp
+; X86-NEXT:    retl $4
+  %r = call <2 x i64> @llvm.umax.v2i64(<2 x i64> %a, <2 x i64> <i64 1, i64 1>)
+  ret <2 x i64> %r
+}
+
 define <1 x i32> @test_v1i32(<1 x i32> %a, <1 x i32> %b) nounwind {
 ; X64-LABEL: test_v1i32:
 ; X64:       # %bb.0:
@@ -442,6 +621,53 @@ define <4 x i32> @test_v4i32(<4 x i32> %a, <4 x i32> %b) nounwind {
   ret <4 x i32> %r
 }
 
+define <4 x i32> @test_v4i32_1(<4 x i32> %a) nounwind {
+; SSE-LABEL: test_v4i32_1:
+; SSE:       # %bb.0:
+; SSE-NEXT:    pxor %xmm1, %xmm1
+; SSE-NEXT:    pcmpeqd %xmm0, %xmm1
+; SSE-NEXT:    psubd %xmm1, %xmm0
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: test_v4i32_1:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vpmaxud {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: test_v4i32_1:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vpbroadcastd {{.*#+}} xmm1 = [1,1,1,1]
+; AVX2-NEXT:    vpmaxud %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    retq
+;
+; X86-LABEL: test_v4i32_1:
+; X86:       # %bb.0:
+; X86-NEXT:    pushl %edi
+; X86-NEXT:    pushl %esi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    cmpl $1, %ecx
+; X86-NEXT:    adcl $0, %ecx
+; X86-NEXT:    cmpl $1, %edx
+; X86-NEXT:    adcl $0, %edx
+; X86-NEXT:    cmpl $1, %esi
+; X86-NEXT:    adcl $0, %esi
+; X86-NEXT:    cmpl $1, %edi
+; X86-NEXT:    adcl $0, %edi
+; X86-NEXT:    movl %edi, 12(%eax)
+; X86-NEXT:    movl %esi, 8(%eax)
+; X86-NEXT:    movl %edx, 4(%eax)
+; X86-NEXT:    movl %ecx, (%eax)
+; X86-NEXT:    popl %esi
+; X86-NEXT:    popl %edi
+; X86-NEXT:    retl $4
+  %r = call <4 x i32> @llvm.umax.v4i32(<4 x i32> %a, <4 x i32> <i32 1, i32 1, i32 1, i32 1>)
+  ret <4 x i32> %r
+}
+
 define <8 x i32> @test_v8i32(<8 x i32> %a, <8 x i32> %b) nounwind {
 ; SSE-LABEL: test_v8i32:
 ; SSE:       # %bb.0:
@@ -540,6 +766,87 @@ define <8 x i32> @test_v8i32(<8 x i32> %a, <8 x i32> %b) nounwind {
   ret <8 x i32> %r
 }
 
+define <8 x i32> @test_v8i32_1(<8 x i32> %a) nounwind {
+; SSE-LABEL: test_v8i32_1:
+; SSE:       # %bb.0:
+; SSE-NEXT:    pxor %xmm2, %xmm2
+; SSE-NEXT:    movdqa %xmm0, %xmm3
+; SSE-NEXT:    pcmpeqd %xmm2, %xmm3
+; SSE-NEXT:    psubd %xmm3, %xmm0
+; SSE-NEXT:    pcmpeqd %xmm1, %xmm2
+; SSE-NEXT:    psubd %xmm2, %xmm1
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: test_v8i32_1:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm1
+; AVX1-NEXT:    vmovdqa {{.*#+}} xmm2 = [1,1,1,1]
+; AVX1-NEXT:    vpmaxud %xmm2, %xmm1, %xmm1
+; AVX1-NEXT:    vpmaxud %xmm2, %xmm0, %xmm0
+; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: test_v8i32_1:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vpbroadcastd {{.*#+}} ymm1 = [1,1,1,1,1,1,1,1]
+; AVX2-NEXT:    vpmaxud %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    retq
+;
+; X86-LABEL: test_v8i32_1:
+; X86:       # %bb.0:
+; X86-NEXT:    pushl %ebp
+; X86-NEXT:    pushl %ebx
+; X86-NEXT:    pushl %edi
+; X86-NEXT:    pushl %esi
+; X86-NEXT:    subl $8, %esp
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebp
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    cmpl $1, %eax
+; X86-NEXT:    adcl $0, %eax
+; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT:    cmpl $1, %edx
+; X86-NEXT:    adcl $0, %edx
+; X86-NEXT:    cmpl $1, %esi
+; X86-NEXT:    adcl $0, %esi
+; X86-NEXT:    cmpl $1, %edi
+; X86-NEXT:    adcl $0, %edi
+; X86-NEXT:    cmpl $1, %ebx
+; X86-NEXT:    adcl $0, %ebx
+; X86-NEXT:    cmpl $1, %ebp
+; X86-NEXT:    adcl $0, %ebp
+; X86-NEXT:    cmpl $1, %ecx
+; X86-NEXT:    adcl $0, %ecx
+; X86-NEXT:    movl %ecx, (%esp) # 4-byte Spill
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    cmpl $1, %eax
+; X86-NEXT:    adcl $0, %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    movl %eax, 28(%ecx)
+; X86-NEXT:    movl (%esp), %eax # 4-byte Reload
+; X86-NEXT:    movl %eax, 24(%ecx)
+; X86-NEXT:    movl %ebp, 20(%ecx)
+; X86-NEXT:    movl %ebx, 16(%ecx)
+; X86-NEXT:    movl %edi, 12(%ecx)
+; X86-NEXT:    movl %esi, 8(%ecx)
+; X86-NEXT:    movl %edx, 4(%ecx)
+; X86-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
+; X86-NEXT:    movl %eax, (%ecx)
+; X86-NEXT:    movl %ecx, %eax
+; X86-NEXT:    addl $8, %esp
+; X86-NEXT:    popl %esi
+; X86-NEXT:    popl %edi
+; X86-NEXT:    popl %ebx
+; X86-NEXT:    popl %ebp
+; X86-NEXT:    retl $4
+  %r = call <8 x i32> @llvm.umax.v8i32(<8 x i32> %a, <8 x i32> <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>)
+  ret <8 x i32> %r
+}
+
 define <8 x i16> @test_v8i16(<8 x i16> %a, <8 x i16> %b) nounwind {
 ; SSE-LABEL: test_v8i16:
 ; SSE:       # %bb.0:
@@ -615,6 +922,74 @@ define <8 x i16> @test_v8i16(<8 x i16> %a, <8 x i16> %b) nounwind {
   ret <8 x i16> %r
 }
 
+define <8 x i16> @test_v8i16_1(<8 x i16> %a) nounwind {
+; SSE-LABEL: test_v8i16_1:
+; SSE:       # %bb.0:
+; SSE-NEXT:    pxor %xmm1, %xmm1
+; SSE-NEXT:    pcmpeqw %xmm0, %xmm1
+; SSE-NEXT:    psubw %xmm1, %xmm0
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: test_v8i16_1:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpmaxuw {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
+; AVX-NEXT:    retq
+;
+; X86-LABEL: test_v8i16_1:
+; X86:       # %bb.0:
+; X86-NEXT:    pushl %ebp
+; X86-NEXT:    pushl %ebx
+; X86-NEXT:    pushl %edi
+; X86-NEXT:    pushl %esi
+; X86-NEXT:    subl $8, %esp
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebp
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-NEXT:    cmpw $1, %dx
+; X86-NEXT:    adcl $0, %edx
+; X86-NEXT:    cmpw $1, %bp
+; X86-NEXT:    adcl $0, %ebp
+; X86-NEXT:    cmpw $1, %bx
+; X86-NEXT:    adcl $0, %ebx
+; X86-NEXT:    cmpw $1, %di
+; X86-NEXT:    adcl $0, %edi
+; X86-NEXT:    cmpw $1, %si
+; X86-NEXT:    adcl $0, %esi
+; X86-NEXT:    cmpw $1, %cx
+; X86-NEXT:    adcl $0, %ecx
+; X86-NEXT:    movl %ecx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT:    cmpw $1, %ax
+; X86-NEXT:    adcl $0, %eax
+; X86-NEXT:    movl %eax, (%esp) # 4-byte Spill
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    cmpw $1, %ax
+; X86-NEXT:    adcl $0, %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    movw %ax, 14(%ecx)
+; X86-NEXT:    movl (%esp), %eax # 4-byte Reload
+; X86-NEXT:    movw %ax, 12(%ecx)
+; X86-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
+; X86-NEXT:    movw %ax, 10(%ecx)
+; X86-NEXT:    movw %si, 8(%ecx)
+; X86-NEXT:    movw %di, 6(%ecx)
+; X86-NEXT:    movw %bx, 4(%ecx)
+; X86-NEXT:    movw %bp, 2(%ecx)
+; X86-NEXT:    movw %dx, (%ecx)
+; X86-NEXT:    movl %ecx, %eax
+; X86-NEXT:    addl $8, %esp
+; X86-NEXT:    popl %esi
+; X86-NEXT:    popl %edi
+; X86-NEXT:    popl %ebx
+; X86-NEXT:    popl %ebp
+; X86-NEXT:    retl $4
+  %r = call <8 x i16> @llvm.umax.v8i16(<8 x i16> %a, <8 x i16> <i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1>)
+  ret <8 x i16> %r
+}
+
 define <16 x i8> @test_v16i8(<16 x i8> %a, <16 x i8> %b) nounwind {
 ; SSE-LABEL: test_v16i8:
 ; SSE:       # %bb.0:
@@ -750,6 +1125,125 @@ define <16 x i8> @test_v16i8(<16 x i8> %a, <16 x i8> %b) nounwind {
   ret <16 x i8> %r
 }
 
+define <16 x i8> @test_v16i8_1(<16 x i8> %a) nounwind {
+; SSE-LABEL: test_v16i8_1:
+; SSE:       # %bb.0:
+; SSE-NEXT:    pmaxub {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: test_v16i8_1:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpmaxub {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
+; AVX-NEXT:    retq
+;
+; X86-LABEL: test_v16i8_1:
+; X86:       # %bb.0:
+; X86-NEXT:    pushl %ebp
+; X86-NEXT:    pushl %ebx
+; X86-NEXT:    pushl %edi
+; X86-NEXT:    pushl %esi
+; X86-NEXT:    subl $40, %esp
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebx
+; X86-NEXT:    cmpb $1, %bl
+; X86-NEXT:    adcl $0, %ebx
+; X86-NEXT:    movl %ebx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT:    cmpb $1, %dl
+; X86-NEXT:    adcl $0, %edx
+; X86-NEXT:    movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT:    cmpb $1, %cl
+; X86-NEXT:    adcl $0, %ecx
+; X86-NEXT:    movl %ecx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT:    cmpb $1, %al
+; X86-NEXT:    adcl $0, %eax
+; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    cmpb $1, %al
+; X86-NEXT:    adcl $0, %eax
+; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    cmpb $1, %al
+; X86-NEXT:    adcl $0, %eax
+; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    cmpb $1, %al
+; X86-NEXT:    adcl $0, %eax
+; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    cmpb $1, %al
+; X86-NEXT:    adcl $0, %eax
+; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    cmpb $1, %al
+; X86-NEXT:    adcl $0, %eax
+; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    cmpb $1, %al
+; X86-NEXT:    adcl $0, %eax
+; X86-NEXT:    movl %eax, (%esp) # 4-byte Spill
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    cmpb $1, %al
+; X86-NEXT:    adcl $0, %eax
+; X86-NEXT:    movl %eax, %ebp
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    cmpb $1, %al
+; X86-NEXT:    adcl $0, %eax
+; X86-NEXT:    movl %eax, %edi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    cmpb $1, %al
+; X86-NEXT:    adcl $0, %eax
+; X86-NEXT:    movl %eax, %esi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebx
+; X86-NEXT:    cmpb $1, %bl
+; X86-NEXT:    adcl $0, %ebx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-NEXT:    cmpb $1, %dl
+; X86-NEXT:    adcl $0, %edx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    cmpb $1, %cl
+; X86-NEXT:    adcl $0, %ecx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movb %cl, 15(%eax)
+; X86-NEXT:    movb %dl, 14(%eax)
+; X86-NEXT:    movb %bl, 13(%eax)
+; X86-NEXT:    movl %esi, %ecx
+; X86-NEXT:    movb %cl, 12(%eax)
+; X86-NEXT:    movl %edi, %ecx
+; X86-NEXT:    movb %cl, 11(%eax)
+; X86-NEXT:    movl %ebp, %ecx
+; X86-NEXT:    movb %cl, 10(%eax)
+; X86-NEXT:    movl (%esp), %ecx # 4-byte Reload
+; X86-NEXT:    movb %cl, 9(%eax)
+; X86-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
+; X86-NEXT:    movb %cl, 8(%eax)
+; X86-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
+; X86-NEXT:    movb %cl, 7(%eax)
+; X86-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
+; X86-NEXT:    movb %cl, 6(%eax)
+; X86-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
+; X86-NEXT:    movb %cl, 5(%eax)
+; X86-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
+; X86-NEXT:    movb %cl, 4(%eax)
+; X86-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
+; X86-NEXT:    movb %cl, 3(%eax)
+; X86-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
+; X86-NEXT:    movb %cl, 2(%eax)
+; X86-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
+; X86-NEXT:    movb %cl, 1(%eax)
+; X86-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
+; X86-NEXT:    movb %cl, (%eax)
+; X86-NEXT:    addl $40, %esp
+; X86-NEXT:    popl %esi
+; X86-NEXT:    popl %edi
+; X86-NEXT:    popl %ebx
+; X86-NEXT:    popl %ebp
+; X86-NEXT:    retl $4
+  %r = call <16 x i8> @llvm.umax.v16i8(<16 x i8> %a, <16 x i8> <i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1>)
+  ret <16 x i8> %r
+}
+
 define i16 @test_signbits_i16(i16 %a, i16 %b) nounwind {
 ; X64-LABEL: test_signbits_i16:
 ; X64:       # %bb.0:
diff --git a/llvm/test/CodeGen/X86/vector-lzcnt-256.ll b/llvm/test/CodeGen/X86/vector-lzcnt-256.ll
index 7c481b3d2b378..f9cac4ba97149 100644
--- a/llvm/test/CodeGen/X86/vector-lzcnt-256.ll
+++ b/llvm/test/CodeGen/X86/vector-lzcnt-256.ll
@@ -976,17 +976,15 @@ define <32 x i8> @testv32i8(<32 x i8> %in) nounwind {
 ;
 ; AVX512-LABEL: testv32i8:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    vextracti128 $1, %ymm0, %xmm1
-; AVX512-NEXT:    vpmovzxbd {{.*#+}} zmm1 = xmm1[0],zero,zero,zero,xmm1[1],zero,zero,zero,xmm1[2],zero,zero,zero,xmm1[3],zero,zero,zero,xmm1[4],zero,zero,zero,xmm1[5],zero,zero,zero,xmm1[6],zero,zero,zero,xmm1[7],zero,zero,zero,xmm1[8],zero,zero,zero,xmm1[9],zero,zero,zero,xmm1[10],zero,zero,zero,xmm1[11],zero,zero,zero,xmm1[12],zero,zero,zero,xmm1[13],zero,zero,zero,xmm1[14],zero,zero,zero,xmm1[15],zero,zero,zero
+; AVX512-NEXT:    vpmovzxbd {{.*#+}} zmm1 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero,xmm0[2],zero,zero,zero,xmm0[3],zero,zero,zero,xmm0[4],zero,zero,zero,xmm0[5],zero,zero,zero,xmm0[6],zero,zero,zero,xmm0[7],zero,zero,zero,xmm0[8],zero,zero,zero,xmm0[9],zero,zero,zero,xmm0[10],zero,zero,zero,xmm0[11],zero,zero,zero,xmm0[12],zero,zero,zero,xmm0[13],zero,zero,zero,xmm0[14],zero,zero,zero,xmm0[15],zero,zero,zero
 ; AVX512-NEXT:    vplzcntd %zmm1, %zmm1
 ; AVX512-NEXT:    vpmovdb %zmm1, %xmm1
-; AVX512-NEXT:    vmovdqa {{.*#+}} xmm2 = [24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24]
-; AVX512-NEXT:    vpsubb %xmm2, %xmm1, %xmm1
+; AVX512-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX512-NEXT:    vpmovzxbd {{.*#+}} zmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero,xmm0[2],zero,zero,zero,xmm0[3],zero,zero,zero,xmm0[4],zero,zero,zero,xmm0[5],zero,zero,zero,xmm0[6],zero,zero,zero,xmm0[7],zero,zero,zero,xmm0[8],zero,zero,zero,xmm0[9],zero,zero,zero,xmm0[10],zero,zero,zero,xmm0[11],zero,zero,zero,xmm0[12],zero,zero,zero,xmm0[13],zero,zero,zero,xmm0[14],zero,zero,zero,xmm0[15],zero,zero,zero
 ; AVX512-NEXT:    vplzcntd %zmm0, %zmm0
 ; AVX512-NEXT:    vpmovdb %zmm0, %xmm0
-; AVX512-NEXT:    vpsubb %xmm2, %xmm0, %xmm0
-; AVX512-NEXT:    vinserti128 $1, %xmm1, %ymm0, %ymm0
+; AVX512-NEXT:    vinserti128 $1, %xmm0, %ymm1, %ymm0
+; AVX512-NEXT:    vpsubb {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
 ; AVX512-NEXT:    retq
 ;
 ; X32-AVX-LABEL: testv32i8:
@@ -1070,17 +1068,15 @@ define <32 x i8> @testv32i8u(<32 x i8> %in) nounwind {
 ;
 ; AVX512-LABEL: testv32i8u:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    vextracti128 $1, %ymm0, %xmm1
-; AVX512-NEXT:    vpmovzxbd {{.*#+}} zmm1 = xmm1[0],zero,zero,zero,xmm1[1],zero,zero,zero,xmm1[2],zero,zero,zero,xmm1[3],zero,zero,zero,xmm1[4],zero,zero,zero,xmm1[5],zero,zero,zero,xmm1[6],zero,zero,zero,xmm1[7],zero,zero,zero,xmm1[8],zero,zero,zero,xmm1[9],zero,zero,zero,xmm1[10],zero,zero,zero,xmm1[11],zero,zero,zero,xmm1[12],zero,zero,zero,xmm1[13],zero,zero,zero,xmm1[14],zero,zero,zero,xmm1[15],zero,zero,zero
+; AVX512-NEXT:    vpmovzxbd {{.*#+}} zmm1 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero,xmm0[2],zero,zero,zero,xmm0[3],zero,zero,zero,xmm0[4],zero,zero,zero,xmm0[5],zero,zero,zero,xmm0[6],zero,zero,zero,xmm0[7],zero,zero,zero,xmm0[8],zero,zero,zero,xmm0[9],zero,zero,zero,xmm0[10],zero,zero,zero,xmm0[11],zero,zero,zero,xmm0[12],zero,zero,zero,xmm0[13],zero,zero,zero,xmm0[14],zero,zero,zero,xmm0[15],zero,zero,zero
 ; AVX512-NEXT:    vplzcntd %zmm1, %zmm1
 ; AVX512-NEXT:    vpmovdb %zmm1, %xmm1
-; AVX512-NEXT:    vmovdqa {{.*#+}} xmm2 = [24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24]
-; AVX512-NEXT:    vpsubb %xmm2, %xmm1, %xmm1
+; AVX512-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX512-NEXT:    vpmovzxbd {{.*#+}} zmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero,xmm0[2],zero,zero,zero,xmm0[3],zero,zero,zero,xmm0[4],zero,zero,zero,xmm0[5],zero,zero,zero,xmm0[6],zero,zero,zero,xmm0[7],zero,zero,zero,xmm0[8],zero,zero,zero,xmm0[9],zero,zero,zero,xmm0[10],zero,zero,zero,xmm0[11],zero,zero,zero,xmm0[12],zero,zero,zero,xmm0[13],zero,zero,zero,xmm0[14],zero,zero,zero,xmm0[15],zero,zero,zero
 ; AVX512-NEXT:    vplzcntd %zmm0, %zmm0
 ; AVX512-NEXT:    vpmovdb %zmm0, %xmm0
-; AVX512-NEXT:    vpsubb %xmm2, %xmm0, %xmm0
-; AVX512-NEXT:    vinserti128 $1, %xmm1, %ymm0, %ymm0
+; AVX512-NEXT:    vinserti128 $1, %xmm0, %ymm1, %ymm0
+; AVX512-NEXT:    vpsubb {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
 ; AVX512-NEXT:    retq
 ;
 ; X32-AVX-LABEL: testv32i8u:
diff --git a/llvm/test/CodeGen/X86/vector-lzcnt-512.ll b/llvm/test/CodeGen/X86/vector-lzcnt-512.ll
index 5b7cc54c8eb69..78be6dfbff06e 100644
--- a/llvm/test/CodeGen/X86/vector-lzcnt-512.ll
+++ b/llvm/test/CodeGen/X86/vector-lzcnt-512.ll
@@ -346,17 +346,15 @@ define <32 x i16> @testv32i16(<32 x i16> %in) nounwind {
 ;
 ; AVX512CDBW-LABEL: testv32i16:
 ; AVX512CDBW:       # %bb.0:
-; AVX512CDBW-NEXT:    vextracti64x4 $1, %zmm0, %ymm1
-; AVX512CDBW-NEXT:    vpmovzxwd {{.*#+}} zmm1 = ymm1[0],zero,ymm1[1],zero,ymm1[2],zero,ymm1[3],zero,ymm1[4],zero,ymm1[5],zero,ymm1[6],zero,ymm1[7],zero,ymm1[8],zero,ymm1[9],zero,ymm1[10],zero,ymm1[11],zero,ymm1[12],zero,ymm1[13],zero,ymm1[14],zero,ymm1[15],zero
+; AVX512CDBW-NEXT:    vpmovzxwd {{.*#+}} zmm1 = ymm0[0],zero,ymm0[1],zero,ymm0[2],zero,ymm0[3],zero,ymm0[4],zero,ymm0[5],zero,ymm0[6],zero,ymm0[7],zero,ymm0[8],zero,ymm0[9],zero,ymm0[10],zero,ymm0[11],zero,ymm0[12],zero,ymm0[13],zero,ymm0[14],zero,ymm0[15],zero
 ; AVX512CDBW-NEXT:    vplzcntd %zmm1, %zmm1
 ; AVX512CDBW-NEXT:    vpmovdw %zmm1, %ymm1
-; AVX512CDBW-NEXT:    vmovdqa {{.*#+}} ymm2 = [16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16]
-; AVX512CDBW-NEXT:    vpsubw %ymm2, %ymm1, %ymm1
+; AVX512CDBW-NEXT:    vextracti64x4 $1, %zmm0, %ymm0
 ; AVX512CDBW-NEXT:    vpmovzxwd {{.*#+}} zmm0 = ymm0[0],zero,ymm0[1],zero,ymm0[2],zero,ymm0[3],zero,ymm0[4],zero,ymm0[5],zero,ymm0[6],zero,ymm0[7],zero,ymm0[8],zero,ymm0[9],zero,ymm0[10],zero,ymm0[11],zero,ymm0[12],zero,ymm0[13],zero,ymm0[14],zero,ymm0[15],zero
 ; AVX512CDBW-NEXT:    vplzcntd %zmm0, %zmm0
 ; AVX512CDBW-NEXT:    vpmovdw %zmm0, %ymm0
-; AVX512CDBW-NEXT:    vpsubw %ymm2, %ymm0, %ymm0
-; AVX512CDBW-NEXT:    vinserti64x4 $1, %ymm1, %zmm0, %zmm0
+; AVX512CDBW-NEXT:    vinserti64x4 $1, %ymm0, %zmm1, %zmm0
+; AVX512CDBW-NEXT:    vpsubw {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %zmm0, %zmm0
 ; AVX512CDBW-NEXT:    retq
 ;
 ; AVX512BW-LABEL: testv32i16:
@@ -432,17 +430,15 @@ define <32 x i16> @testv32i16u(<32 x i16> %in) nounwind {
 ;
 ; AVX512CDBW-LABEL: testv32i16u:
 ; AVX512CDBW:       # %bb.0:
-; AVX512CDBW-NEXT:    vextracti64x4 $1, %zmm0, %ymm1
-; AVX512CDBW-NEXT:    vpmovzxwd {{.*#+}} zmm1 = ymm1[0],zero,ymm1[1],zero,ymm1[2],zero,ymm1[3],zero,ymm1[4],zero,ymm1[5],zero,ymm1[6],zero,ymm1[7],zero,ymm1[8],zero,ymm1[9],zero,ymm1[10],zero,ymm1[11],zero,ymm1[12],zero,ymm1[13],zero,ymm1[14],zero,ymm1[15],zero
+; AVX512CDBW-NEXT:    vpmovzxwd {{.*#+}} zmm1 = ymm0[0],zero,ymm0[1],zero,ymm0[2],zero,ymm0[3],zero,ymm0[4],zero,ymm0[5],zero,ymm0[6],zero,ymm0[7],zero,ymm0[8],zero,ymm0[9],zero,ymm0[10],zero,ymm0[11],zero,ymm0[12],zero,ymm0[13],zero,ymm0[14],zero,ymm0[15],zero
 ; AVX512CDBW-NEXT:    vplzcntd %zmm1, %zmm1
 ; AVX512CDBW-NEXT:    vpmovdw %zmm1, %ymm1
-; AVX512CDBW-NEXT:    vmovdqa {{.*#+}} ymm2 = [16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16]
-; AVX512CDBW-NEXT:    vpsubw %ymm2, %ymm1, %ymm1
+; AVX512CDBW-NEXT:    vextracti64x4 $1, %zmm0, %ymm0
 ; AVX512CDBW-NEXT:    vpmovzxwd {{.*#+}} zmm0 = ymm0[0],zero,ymm0[1],zero,ymm0[2],zero,ymm0[3],zero,ymm0[4],zero,ymm0[5],zero,ymm0[6],zero,ymm0[7],zero,ymm0[8],zero,ymm0[9],zero,ymm0[10],zero,ymm0[11],zero,ymm0[12],zero,ymm0[13],zero,ymm0[14],zero,ymm0[15],zero
 ; AVX512CDBW-NEXT:    vplzcntd %zmm0, %zmm0
 ; AVX512CDBW-NEXT:    vpmovdw %zmm0, %ymm0
-; AVX512CDBW-NEXT:    vpsubw %ymm2, %ymm0, %ymm0
-; AVX512CDBW-NEXT:    vinserti64x4 $1, %ymm1, %zmm0, %zmm0
+; AVX512CDBW-NEXT:    vinserti64x4 $1, %ymm0, %zmm1, %zmm0
+; AVX512CDBW-NEXT:    vpsubw {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %zmm0, %zmm0
 ; AVX512CDBW-NEXT:    retq
 ;
 ; AVX512BW-LABEL: testv32i16u:
@@ -504,55 +500,49 @@ define <64 x i8> @testv64i8(<64 x i8> %in) nounwind {
 ; AVX512CD-LABEL: testv64i8:
 ; AVX512CD:       # %bb.0:
 ; AVX512CD-NEXT:    vextracti64x4 $1, %zmm0, %ymm1
-; AVX512CD-NEXT:    vextracti128 $1, %ymm1, %xmm2
-; AVX512CD-NEXT:    vpmovzxbd {{.*#+}} zmm2 = xmm2[0],zero,zero,zero,xmm2[1],zero,zero,zero,xmm2[2],zero,zero,zero,xmm2[3],zero,zero,zero,xmm2[4],zero,zero,zero,xmm2[5],zero,zero,zero,xmm2[6],zero,zero,zero,xmm2[7],zero,zero,zero,xmm2[8],zero,zero,zero,xmm2[9],zero,zero,zero,xmm2[10],zero,zero,zero,xmm2[11],zero,zero,zero,xmm2[12],zero,zero,zero,xmm2[13],zero,zero,zero,xmm2[14],zero,zero,zero,xmm2[15],zero,zero,zero
+; AVX512CD-NEXT:    vpmovzxbd {{.*#+}} zmm2 = xmm1[0],zero,zero,zero,xmm1[1],zero,zero,zero,xmm1[2],zero,zero,zero,xmm1[3],zero,zero,zero,xmm1[4],zero,zero,zero,xmm1[5],zero,zero,zero,xmm1[6],zero,zero,zero,xmm1[7],zero,zero,zero,xmm1[8],zero,zero,zero,xmm1[9],zero,zero,zero,xmm1[10],zero,zero,zero,xmm1[11],zero,zero,zero,xmm1[12],zero,zero,zero,xmm1[13],zero,zero,zero,xmm1[14],zero,zero,zero,xmm1[15],zero,zero,zero
 ; AVX512CD-NEXT:    vplzcntd %zmm2, %zmm2
 ; AVX512CD-NEXT:    vpmovdb %zmm2, %xmm2
-; AVX512CD-NEXT:    vmovdqa {{.*#+}} xmm3 = [24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24]
-; AVX512CD-NEXT:    vpsubb %xmm3, %xmm2, %xmm2
+; AVX512CD-NEXT:    vextracti128 $1, %ymm1, %xmm1
 ; AVX512CD-NEXT:    vpmovzxbd {{.*#+}} zmm1 = xmm1[0],zero,zero,zero,xmm1[1],zero,zero,zero,xmm1[2],zero,zero,zero,xmm1[3],zero,zero,zero,xmm1[4],zero,zero,zero,xmm1[5],zero,zero,zero,xmm1[6],zero,zero,zero,xmm1[7],zero,zero,zero,xmm1[8],zero,zero,zero,xmm1[9],zero,zero,zero,xmm1[10],zero,zero,zero,xmm1[11],zero,zero,zero,xmm1[12],zero,zero,zero,xmm1[13],zero,zero,zero,xmm1[14],zero,zero,zero,xmm1[15],zero,zero,zero
 ; AVX512CD-NEXT:    vplzcntd %zmm1, %zmm1
 ; AVX512CD-NEXT:    vpmovdb %zmm1, %xmm1
-; AVX512CD-NEXT:    vpsubb %xmm3, %xmm1, %xmm1
-; AVX512CD-NEXT:    vinserti128 $1, %xmm2, %ymm1, %ymm1
-; AVX512CD-NEXT:    vextracti128 $1, %ymm0, %xmm2
-; AVX512CD-NEXT:    vpmovzxbd {{.*#+}} zmm2 = xmm2[0],zero,zero,zero,xmm2[1],zero,zero,zero,xmm2[2],zero,zero,zero,xmm2[3],zero,zero,zero,xmm2[4],zero,zero,zero,xmm2[5],zero,zero,zero,xmm2[6],zero,zero,zero,xmm2[7],zero,zero,zero,xmm2[8],zero,zero,zero,xmm2[9],zero,zero,zero,xmm2[10],zero,zero,zero,xmm2[11],zero,zero,zero,xmm2[12],zero,zero,zero,xmm2[13],zero,zero,zero,xmm2[14],zero,zero,zero,xmm2[15],zero,zero,zero
-; AVX512CD-NEXT:    vplzcntd %zmm2, %zmm2
-; AVX512CD-NEXT:    vpmovdb %zmm2, %xmm2
-; AVX512CD-NEXT:    vpsubb %xmm3, %xmm2, %xmm2
+; AVX512CD-NEXT:    vinserti128 $1, %xmm1, %ymm2, %ymm1
+; AVX512CD-NEXT:    vmovdqa {{.*#+}} ymm2 = [24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24]
+; AVX512CD-NEXT:    vpsubb %ymm2, %ymm1, %ymm1
+; AVX512CD-NEXT:    vpmovzxbd {{.*#+}} zmm3 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero,xmm0[2],zero,zero,zero,xmm0[3],zero,zero,zero,xmm0[4],zero,zero,zero,xmm0[5],zero,zero,zero,xmm0[6],zero,zero,zero,xmm0[7],zero,zero,zero,xmm0[8],zero,zero,zero,xmm0[9],zero,zero,zero,xmm0[10],zero,zero,zero,xmm0[11],zero,zero,zero,xmm0[12],zero,zero,zero,xmm0[13],zero,zero,zero,xmm0[14],zero,zero,zero,xmm0[15],zero,zero,zero
+; AVX512CD-NEXT:    vplzcntd %zmm3, %zmm3
+; AVX512CD-NEXT:    vpmovdb %zmm3, %xmm3
+; AVX512CD-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX512CD-NEXT:    vpmovzxbd {{.*#+}} zmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero,xmm0[2],zero,zero,zero,xmm0[3],zero,zero,zero,xmm0[4],zero,zero,zero,xmm0[5],zero,zero,zero,xmm0[6],zero,zero,zero,xmm0[7],zero,zero,zero,xmm0[8],zero,zero,zero,xmm0[9],zero,zero,zero,xmm0[10],zero,zero,zero,xmm0[11],zero,zero,zero,xmm0[12],zero,zero,zero,xmm0[13],zero,zero,zero,xmm0[14],zero,zero,zero,xmm0[15],zero,zero,zero
 ; AVX512CD-NEXT:    vplzcntd %zmm0, %zmm0
 ; AVX512CD-NEXT:    vpmovdb %zmm0, %xmm0
-; AVX512CD-NEXT:    vpsubb %xmm3, %xmm0, %xmm0
-; AVX512CD-NEXT:    vinserti128 $1, %xmm2, %ymm0, %ymm0
+; AVX512CD-NEXT:    vinserti128 $1, %xmm0, %ymm3, %ymm0
+; AVX512CD-NEXT:    vpsubb %ymm2, %ymm0, %ymm0
 ; AVX512CD-NEXT:    vinserti64x4 $1, %ymm1, %zmm0, %zmm0
 ; AVX512CD-NEXT:    retq
 ;
 ; AVX512CDBW-LABEL: testv64i8:
 ; AVX512CDBW:       # %bb.0:
 ; AVX512CDBW-NEXT:    vextracti64x4 $1, %zmm0, %ymm1
-; AVX512CDBW-NEXT:    vextracti128 $1, %ymm1, %xmm2
-; AVX512CDBW-NEXT:    vpmovzxbd {{.*#+}} zmm2 = xmm2[0],zero,zero,zero,xmm2[1],zero,zero,zero,xmm2[2],zero,zero,zero,xmm2[3],zero,zero,zero,xmm2[4],zero,zero,zero,xmm2[5],zero,zero,zero,xmm2[6],zero,zero,zero,xmm2[7],zero,zero,zero,xmm2[8],zero,zero,zero,xmm2[9],zero,zero,zero,xmm2[10],zero,zero,zero,xmm2[11],zero,zero,zero,xmm2[12],zero,zero,zero,xmm2[13],zero,zero,zero,xmm2[14],zero,zero,zero,xmm2[15],zero,zero,zero
+; AVX512CDBW-NEXT:    vpmovzxbd {{.*#+}} zmm2 = xmm1[0],zero,zero,zero,xmm1[1],zero,zero,zero,xmm1[2],zero,zero,zero,xmm1[3],zero,zero,zero,xmm1[4],zero,zero,zero,xmm1[5],zero,zero,zero,xmm1[6],zero,zero,zero,xmm1[7],zero,zero,zero,xmm1[8],zero,zero,zero,xmm1[9],zero,zero,zero,xmm1[10],zero,zero,zero,xmm1[11],zero,zero,zero,xmm1[12],zero,zero,zero,xmm1[13],zero,zero,zero,xmm1[14],zero,zero,zero,xmm1[15],zero,zero,zero
 ; AVX512CDBW-NEXT:    vplzcntd %zmm2, %zmm2
 ; AVX512CDBW-NEXT:    vpmovdb %zmm2, %xmm2
-; AVX512CDBW-NEXT:    vmovdqa {{.*#+}} xmm3 = [24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24]
-; AVX512CDBW-NEXT:    vpsubb %xmm3, %xmm2, %xmm2
+; AVX512CDBW-NEXT:    vextracti128 $1, %ymm1, %xmm1
 ; AVX512CDBW-NEXT:    vpmovzxbd {{.*#+}} zmm1 = xmm1[0],zero,zero,zero,xmm1[1],zero,zero,zero,xmm1[2],zero,zero,zero,xmm1[3],zero,zero,zero,xmm1[4],zero,zero,zero,xmm1[5],zero,zero,zero,xmm1[6],zero,zero,zero,xmm1[7],zero,zero,zero,xmm1[8],zero,zero,zero,xmm1[9],zero,zero,zero,xmm1[10],zero,zero,zero,xmm1[11],zero,zero,zero,xmm1[12],zero,zero,zero,xmm1[13],zero,zero,zero,xmm1[14],zero,zero,zero,xmm1[15],zero,zero,zero
 ; AVX512CDBW-NEXT:    vplzcntd %zmm1, %zmm1
 ; AVX512CDBW-NEXT:    vpmovdb %zmm1, %xmm1
-; AVX512CDBW-NEXT:    vpsubb %xmm3, %xmm1, %xmm1
-; AVX512CDBW-NEXT:    vinserti128 $1, %xmm2, %ymm1, %ymm1
-; AVX512CDBW-NEXT:    vextracti128 $1, %ymm0, %xmm2
-; AVX512CDBW-NEXT:    vpmovzxbd {{.*#+}} zmm2 = xmm2[0],zero,zero,zero,xmm2[1],zero,zero,zero,xmm2[2],zero,zero,zero,xmm2[3],zero,zero,zero,xmm2[4],zero,zero,zero,xmm2[5],zero,zero,zero,xmm2[6],zero,zero,zero,xmm2[7],zero,zero,zero,xmm2[8],zero,zero,zero,xmm2[9],zero,zero,zero,xmm2[10],zero,zero,zero,xmm2[11],zero,zero,zero,xmm2[12],zero,zero,zero,xmm2[13],zero,zero,zero,xmm2[14],zero,zero,zero,xmm2[15],zero,zero,zero
+; AVX512CDBW-NEXT:    vinserti128 $1, %xmm1, %ymm2, %ymm1
+; AVX512CDBW-NEXT:    vpmovzxbd {{.*#+}} zmm2 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero,xmm0[2],zero,zero,zero,xmm0[3],zero,zero,zero,xmm0[4],zero,zero,zero,xmm0[5],zero,zero,zero,xmm0[6],zero,zero,zero,xmm0[7],zero,zero,zero,xmm0[8],zero,zero,zero,xmm0[9],zero,zero,zero,xmm0[10],zero,zero,zero,xmm0[11],zero,zero,zero,xmm0[12],zero,zero,zero,xmm0[13],zero,zero,zero,xmm0[14],zero,zero,zero,xmm0[15],zero,zero,zero
 ; AVX512CDBW-NEXT:    vplzcntd %zmm2, %zmm2
 ; AVX512CDBW-NEXT:    vpmovdb %zmm2, %xmm2
-; AVX512CDBW-NEXT:    vpsubb %xmm3, %xmm2, %xmm2
+; AVX512CDBW-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX512CDBW-NEXT:    vpmovzxbd {{.*#+}} zmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero,xmm0[2],zero,zero,zero,xmm0[3],zero,zero,zero,xmm0[4],zero,zero,zero,xmm0[5],zero,zero,zero,xmm0[6],zero,zero,zero,xmm0[7],zero,zero,zero,xmm0[8],zero,zero,zero,xmm0[9],zero,zero,zero,xmm0[10],zero,zero,zero,xmm0[11],zero,zero,zero,xmm0[12],zero,zero,zero,xmm0[13],zero,zero,zero,xmm0[14],zero,zero,zero,xmm0[15],zero,zero,zero
 ; AVX512CDBW-NEXT:    vplzcntd %zmm0, %zmm0
 ; AVX512CDBW-NEXT:    vpmovdb %zmm0, %xmm0
-; AVX512CDBW-NEXT:    vpsubb %xmm3, %xmm0, %xmm0
-; AVX512CDBW-NEXT:    vinserti128 $1, %xmm2, %ymm0, %ymm0
+; AVX512CDBW-NEXT:    vinserti128 $1, %xmm0, %ymm2, %ymm0
 ; AVX512CDBW-NEXT:    vinserti64x4 $1, %ymm1, %zmm0, %zmm0
+; AVX512CDBW-NEXT:    vpsubb {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %zmm0, %zmm0
 ; AVX512CDBW-NEXT:    retq
 ;
 ; AVX512BW-LABEL: testv64i8:
@@ -598,55 +588,49 @@ define <64 x i8> @testv64i8u(<64 x i8> %in) nounwind {
 ; AVX512CD-LABEL: testv64i8u:
 ; AVX512CD:       # %bb.0:
 ; AVX512CD-NEXT:    vextracti64x4 $1, %zmm0, %ymm1
-; AVX512CD-NEXT:    vextracti128 $1, %ymm1, %xmm2
-; AVX512CD-NEXT:    vpmovzxbd {{.*#+}} zmm2 = xmm2[0],zero,zero,zero,xmm2[1],zero,zero,zero,xmm2[2],zero,zero,zero,xmm2[3],zero,zero,zero,xmm2[4],zero,zero,zero,xmm2[5],zero,zero,zero,xmm2[6],zero,zero,zero,xmm2[7],zero,zero,zero,xmm2[8],zero,zero,zero,xmm2[9],zero,zero,zero,xmm2[10],zero,zero,zero,xmm2[11],zero,zero,zero,xmm2[12],zero,zero,zero,xmm2[13],zero,zero,zero,xmm2[14],zero,zero,zero,xmm2[15],zero,zero,zero
+; AVX512CD-NEXT:    vpmovzxbd {{.*#+}} zmm2 = xmm1[0],zero,zero,zero,xmm1[1],zero,zero,zero,xmm1[2],zero,zero,zero,xmm1[3],zero,zero,zero,xmm1[4],zero,zero,zero,xmm1[5],zero,zero,zero,xmm1[6],zero,zero,zero,xmm1[7],zero,zero,zero,xmm1[8],zero,zero,zero,xmm1[9],zero,zero,zero,xmm1[10],zero,zero,zero,xmm1[11],zero,zero,zero,xmm1[12],zero,zero,zero,xmm1[13],zero,zero,zero,xmm1[14],zero,zero,zero,xmm1[15],zero,zero,zero
 ; AVX512CD-NEXT:    vplzcntd %zmm2, %zmm2
 ; AVX512CD-NEXT:    vpmovdb %zmm2, %xmm2
-; AVX512CD-NEXT:    vmovdqa {{.*#+}} xmm3 = [24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24]
-; AVX512CD-NEXT:    vpsubb %xmm3, %xmm2, %xmm2
+; AVX512CD-NEXT:    vextracti128 $1, %ymm1, %xmm1
 ; AVX512CD-NEXT:    vpmovzxbd {{.*#+}} zmm1 = xmm1[0],zero,zero,zero,xmm1[1],zero,zero,zero,xmm1[2],zero,zero,zero,xmm1[3],zero,zero,zero,xmm1[4],zero,zero,zero,xmm1[5],zero,zero,zero,xmm1[6],zero,zero,zero,xmm1[7],zero,zero,zero,xmm1[8],zero,zero,zero,xmm1[9],zero,zero,zero,xmm1[10],zero,zero,zero,xmm1[11],zero,zero,zero,xmm1[12],zero,zero,zero,xmm1[13],zero,zero,zero,xmm1[14],zero,zero,zero,xmm1[15],zero,zero,zero
 ; AVX512CD-NEXT:    vplzcntd %zmm1, %zmm1
 ; AVX512CD-NEXT:    vpmovdb %zmm1, %xmm1
-; AVX512CD-NEXT:    vpsubb %xmm3, %xmm1, %xmm1
-; AVX512CD-NEXT:    vinserti128 $1, %xmm2, %ymm1, %ymm1
-; AVX512CD-NEXT:    vextracti128 $1, %ymm0, %xmm2
-; AVX512CD-NEXT:    vpmovzxbd {{.*#+}} zmm2 = xmm2[0],zero,zero,zero,xmm2[1],zero,zero,zero,xmm2[2],zero,zero,zero,xmm2[3],zero,zero,zero,xmm2[4],zero,zero,zero,xmm2[5],zero,zero,zero,xmm2[6],zero,zero,zero,xmm2[7],zero,zero,zero,xmm2[8],zero,zero,zero,xmm2[9],zero,zero,zero,xmm2[10],zero,zero,zero,xmm2[11],zero,zero,zero,xmm2[12],zero,zero,zero,xmm2[13],zero,zero,zero,xmm2[14],zero,zero,zero,xmm2[15],zero,zero,zero
-; AVX512CD-NEXT:    vplzcntd %zmm2, %zmm2
-; AVX512CD-NEXT:    vpmovdb %zmm2, %xmm2
-; AVX512CD-NEXT:    vpsubb %xmm3, %xmm2, %xmm2
+; AVX512CD-NEXT:    vinserti128 $1, %xmm1, %ymm2, %ymm1
+; AVX512CD-NEXT:    vmovdqa {{.*#+}} ymm2 = [24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24]
+; AVX512CD-NEXT:    vpsubb %ymm2, %ymm1, %ymm1
+; AVX512CD-NEXT:    vpmovzxbd {{.*#+}} zmm3 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero,xmm0[2],zero,zero,zero,xmm0[3],zero,zero,zero,xmm0[4],zero,zero,zero,xmm0[5],zero,zero,zero,xmm0[6],zero,zero,zero,xmm0[7],zero,zero,zero,xmm0[8],zero,zero,zero,xmm0[9],zero,zero,zero,xmm0[10],zero,zero,zero,xmm0[11],zero,zero,zero,xmm0[12],zero,zero,zero,xmm0[13],zero,zero,zero,xmm0[14],zero,zero,zero,xmm0[15],zero,zero,zero
+; AVX512CD-NEXT:    vplzcntd %zmm3, %zmm3
+; AVX512CD-NEXT:    vpmovdb %zmm3, %xmm3
+; AVX512CD-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX512CD-NEXT:    vpmovzxbd {{.*#+}} zmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero,xmm0[2],zero,zero,zero,xmm0[3],zero,zero,zero,xmm0[4],zero,zero,zero,xmm0[5],zero,zero,zero,xmm0[6],zero,zero,zero,xmm0[7],zero,zero,zero,xmm0[8],zero,zero,zero,xmm0[9],zero,zero,zero,xmm0[10],zero,zero,zero,xmm0[11],zero,zero,zero,xmm0[12],zero,zero,zero,xmm0[13],zero,zero,zero,xmm0[14],zero,zero,zero,xmm0[15],zero,zero,zero
 ; AVX512CD-NEXT:    vplzcntd %zmm0, %zmm0
 ; AVX512CD-NEXT:    vpmovdb %zmm0, %xmm0
-; AVX512CD-NEXT:    vpsubb %xmm3, %xmm0, %xmm0
-; AVX512CD-NEXT:    vinserti128 $1, %xmm2, %ymm0, %ymm0
+; AVX512CD-NEXT:    vinserti128 $1, %xmm0, %ymm3, %ymm0
+; AVX512CD-NEXT:    vpsubb %ymm2, %ymm0, %ymm0
 ; AVX512CD-NEXT:    vinserti64x4 $1, %ymm1, %zmm0, %zmm0
 ; AVX512CD-NEXT:    retq
 ;
 ; AVX512CDBW-LABEL: testv64i8u:
 ; AVX512CDBW:       # %bb.0:
 ; AVX512CDBW-NEXT:    vextracti64x4 $1, %zmm0, %ymm1
-; AVX512CDBW-NEXT:    vextracti128 $1, %ymm1, %xmm2
-; AVX512CDBW-NEXT:    vpmovzxbd {{.*#+}} zmm2 = xmm2[0],zero,zero,zero,xmm2[1],zero,zero,zero,xmm2[2],zero,zero,zero,xmm2[3],zero,zero,zero,xmm2[4],zero,zero,zero,xmm2[5],zero,zero,zero,xmm2[6],zero,zero,zero,xmm2[7],zero,zero,zero,xmm2[8],zero,zero,zero,xmm2[9],zero,zero,zero,xmm2[10],zero,zero,zero,xmm2[11],zero,zero,zero,xmm2[12],zero,zero,zero,xmm2[13],zero,zero,zero,xmm2[14],zero,zero,zero,xmm2[15],zero,zero,zero
+; AVX512CDBW-NEXT:    vpmovzxbd {{.*#+}} zmm2 = xmm1[0],zero,zero,zero,xmm1[1],zero,zero,zero,xmm1[2],zero,zero,zero,xmm1[3],zero,zero,zero,xmm1[4],zero,zero,zero,xmm1[5],zero,zero,zero,xmm1[6],zero,zero,zero,xmm1[7],zero,zero,zero,xmm1[8],zero,zero,zero,xmm1[9],zero,zero,zero,xmm1[10],zero,zero,zero,xmm1[11],zero,zero,zero,xmm1[12],zero,zero,zero,xmm1[13],zero,zero,zero,xmm1[14],zero,zero,zero,xmm1[15],zero,zero,zero
 ; AVX512CDBW-NEXT:    vplzcntd %zmm2, %zmm2
 ; AVX512CDBW-NEXT:    vpmovdb %zmm2, %xmm2
-; AVX512CDBW-NEXT:    vmovdqa {{.*#+}} xmm3 = [24,24,24,24,24,24,24,24,24,24,24,24,24,24,24,24]
-; AVX512CDBW-NEXT:    vpsubb %xmm3, %xmm2, %xmm2
+; AVX512CDBW-NEXT:    vextracti128 $1, %ymm1, %xmm1
 ; AVX512CDBW-NEXT:    vpmovzxbd {{.*#+}} zmm1 = xmm1[0],zero,zero,zero,xmm1[1],zero,zero,zero,xmm1[2],zero,zero,zero,xmm1[3],zero,zero,zero,xmm1[4],zero,zero,zero,xmm1[5],zero,zero,zero,xmm1[6],zero,zero,zero,xmm1[7],zero,zero,zero,xmm1[8],zero,zero,zero,xmm1[9],zero,zero,zero,xmm1[10],zero,zero,zero,xmm1[11],zero,zero,zero,xmm1[12],zero,zero,zero,xmm1[13],zero,zero,zero,xmm1[14],zero,zero,zero,xmm1[15],zero,zero,zero
 ; AVX512CDBW-NEXT:    vplzcntd %zmm1, %zmm1
 ; AVX512CDBW-NEXT:    vpmovdb %zmm1, %xmm1
-; AVX512CDBW-NEXT:    vpsubb %xmm3, %xmm1, %xmm1
-; AVX512CDBW-NEXT:    vinserti128 $1, %xmm2, %ymm1, %ymm1
-; AVX512CDBW-NEXT:    vextracti128 $1, %ymm0, %xmm2
-; AVX512CDBW-NEXT:    vpmovzxbd {{.*#+}} zmm2 = xmm2[0],zero,zero,zero,xmm2[1],zero,zero,zero,xmm2[2],zero,zero,zero,xmm2[3],zero,zero,zero,xmm2[4],zero,zero,zero,xmm2[5],zero,zero,zero,xmm2[6],zero,zero,zero,xmm2[7],zero,zero,zero,xmm2[8],zero,zero,zero,xmm2[9],zero,zero,zero,xmm2[10],zero,zero,zero,xmm2[11],zero,zero,zero,xmm2[12],zero,zero,zero,xmm2[13],zero,zero,zero,xmm2[14],zero,zero,zero,xmm2[15],zero,zero,zero
+; AVX512CDBW-NEXT:    vinserti128 $1, %xmm1, %ymm2, %ymm1
+; AVX512CDBW-NEXT:    vpmovzxbd {{.*#+}} zmm2 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero,xmm0[2],zero,zero,zero,xmm0[3],zero,zero,zero,xmm0[4],zero,zero,zero,xmm0[5],zero,zero,zero,xmm0[6],zero,zero,zero,xmm0[7],zero,zero,zero,xmm0[8],zero,zero,zero,xmm0[9],zero,zero,zero,xmm0[10],zero,zero,zero,xmm0[11],zero,zero,zero,xmm0[12],zero,zero,zero,xmm0[13],zero,zero,zero,xmm0[14],zero,zero,zero,xmm0[15],zero,zero,zero
 ; AVX512CDBW-NEXT:    vplzcntd %zmm2, %zmm2
 ; AVX512CDBW-NEXT:    vpmovdb %zmm2, %xmm2
-; AVX512CDBW-NEXT:    vpsubb %xmm3, %xmm2, %xmm2
+; AVX512CDBW-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX512CDBW-NEXT:    vpmovzxbd {{.*#+}} zmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero,xmm0[2],zero,zero,zero,xmm0[3],zero,zero,zero,xmm0[4],zero,zero,zero,xmm0[5],zero,zero,zero,xmm0[6],zero,zero,zero,xmm0[7],zero,zero,zero,xmm0[8],zero,zero,zero,xmm0[9],zero,zero,zero,xmm0[10],zero,zero,zero,xmm0[11],zero,zero,zero,xmm0[12],zero,zero,zero,xmm0[13],zero,zero,zero,xmm0[14],zero,zero,zero,xmm0[15],zero,zero,zero
 ; AVX512CDBW-NEXT:    vplzcntd %zmm0, %zmm0
 ; AVX512CDBW-NEXT:    vpmovdb %zmm0, %xmm0
-; AVX512CDBW-NEXT:    vpsubb %xmm3, %xmm0, %xmm0
-; AVX512CDBW-NEXT:    vinserti128 $1, %xmm2, %ymm0, %ymm0
+; AVX512CDBW-NEXT:    vinserti128 $1, %xmm0, %ymm2, %ymm0
 ; AVX512CDBW-NEXT:    vinserti64x4 $1, %ymm1, %zmm0, %zmm0
+; AVX512CDBW-NEXT:    vpsubb {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %zmm0, %zmm0
 ; AVX512CDBW-NEXT:    retq
 ;
 ; AVX512BW-LABEL: testv64i8u:
diff --git a/llvm/test/CodeGen/X86/vector-shuffle-avx512.ll b/llvm/test/CodeGen/X86/vector-shuffle-avx512.ll
index edfedae3943c1..0dca69aaaf0b4 100644
--- a/llvm/test/CodeGen/X86/vector-shuffle-avx512.ll
+++ b/llvm/test/CodeGen/X86/vector-shuffle-avx512.ll
@@ -1,46 +1,48 @@
 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc < %s -mtriple=x86_64-pc-linux-gnu -mcpu=skx | FileCheck %s --check-prefixes=CHECK,SKX,X64,SKX64
-; RUN: llc < %s -mtriple=x86_64-pc-linux-gnu -mcpu=knl | FileCheck %s --check-prefixes=CHECK,KNL,X64,KNL64
-; RUN: llc < %s -mtriple=i386-pc-linux-gnu -mcpu=skx | FileCheck %s --check-prefixes=CHECK,SKX,X86,SKX32
-; RUN: llc < %s -mtriple=i386-pc-linux-gnu -mcpu=knl | FileCheck %s --check-prefixes=CHECK,KNL,X86,KNL32
+; RUN: llc < %s -mtriple=i386-pc-linux-gnu -mattr=+avx512f,+avx512vl,+avx512bw,+avx512dq | FileCheck %s --check-prefixes=CHECK,AVX512,X86-AVX512,AVX512-SLOW,X86-AVX512-SLOW
+; RUN: llc < %s -mtriple=x86_64-pc-linux-gnu -mattr=+avx512f,+avx512vl,+avx512bw,+avx512dq | FileCheck %s --check-prefixes=CHECK,AVX512,X64-AVX512,AVX512-SLOW,X64-AVX512-SLOW
+; RUN: llc < %s -mtriple=i386-pc-linux-gnu -mattr=+avx512f,+avx512vl,+avx512bw,+avx512dq,+fast-variable-crosslane-shuffle,+fast-variable-perlane-shuffle | FileCheck %s --check-prefixes=CHECK,AVX512,X86-AVX512,AVX512-FAST,X86-AVX512-FAST
+; RUN: llc < %s -mtriple=x86_64-pc-linux-gnu -mattr=+avx512f,+avx512vl,+avx512bw,+avx512dq,+fast-variable-crosslane-shuffle,+fast-variable-perlane-shuffle | FileCheck %s --check-prefixes=CHECK,AVX512,X64-AVX512,AVX512-FAST,X64-AVX512-FAST
+; RUN: llc < %s -mtriple=i386-pc-linux-gnu -mattr=+avx512f | FileCheck %s --check-prefixes=CHECK,AVX512F,X86-AVX512F
+; RUN: llc < %s -mtriple=x86_64-pc-linux-gnu -mattr=+avx512f | FileCheck %s --check-prefixes=CHECK,AVX512F,X64-AVX512F
 
 ;expand 128 -> 256 include <4 x float> <2 x double>
 define <8 x float> @expand(<4 x float> %a) {
-; SKX-LABEL: expand:
-; SKX:       # %bb.0:
-; SKX-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
-; SKX-NEXT:    movb $5, %al
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vexpandps %ymm0, %ymm0 {%k1} {z}
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL-LABEL: expand:
-; KNL:       # %bb.0:
-; KNL-NEXT:    vshufps {{.*#+}} xmm0 = xmm0[0,1,1,3]
-; KNL-NEXT:    vxorps %xmm1, %xmm1, %xmm1
-; KNL-NEXT:    vblendps {{.*#+}} ymm0 = ymm0[0],ymm1[1],ymm0[2],ymm1[3,4,5,6,7]
-; KNL-NEXT:    ret{{[l|q]}}
+; AVX512-LABEL: expand:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
+; AVX512-NEXT:    movb $5, %al
+; AVX512-NEXT:    kmovd %eax, %k1
+; AVX512-NEXT:    vexpandps %ymm0, %ymm0 {%k1} {z}
+; AVX512-NEXT:    ret{{[l|q]}}
+;
+; AVX512F-LABEL: expand:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    vshufps {{.*#+}} xmm0 = xmm0[0,1,1,3]
+; AVX512F-NEXT:    vxorps %xmm1, %xmm1, %xmm1
+; AVX512F-NEXT:    vblendps {{.*#+}} ymm0 = ymm0[0],ymm1[1],ymm0[2],ymm1[3,4,5,6,7]
+; AVX512F-NEXT:    ret{{[l|q]}}
    %res = shufflevector <4 x float> %a, <4 x float> zeroinitializer, <8 x i32> <i32 0, i32 5, i32 1, i32 5, i32 5, i32 5, i32 5, i32 5>
    ret <8 x float> %res
 }
 
 define <8 x float> @expand1(<4 x float> %a ) {
-; SKX-LABEL: expand1:
-; SKX:       # %bb.0:
-; SKX-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
-; SKX-NEXT:    movb $-86, %al
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vexpandps %ymm0, %ymm0 {%k1} {z}
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL-LABEL: expand1:
-; KNL:       # %bb.0:
-; KNL-NEXT:    # kill: def $xmm0 killed $xmm0 def $zmm0
-; KNL-NEXT:    vmovaps {{.*#+}} ymm1 = [16,0,18,1,20,2,22,3]
-; KNL-NEXT:    vxorps %xmm2, %xmm2, %xmm2
-; KNL-NEXT:    vpermt2ps %zmm2, %zmm1, %zmm0
-; KNL-NEXT:    # kill: def $ymm0 killed $ymm0 killed $zmm0
-; KNL-NEXT:    ret{{[l|q]}}
+; AVX512-LABEL: expand1:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
+; AVX512-NEXT:    movb $-86, %al
+; AVX512-NEXT:    kmovd %eax, %k1
+; AVX512-NEXT:    vexpandps %ymm0, %ymm0 {%k1} {z}
+; AVX512-NEXT:    ret{{[l|q]}}
+;
+; AVX512F-LABEL: expand1:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    # kill: def $xmm0 killed $xmm0 def $zmm0
+; AVX512F-NEXT:    vmovaps {{.*#+}} ymm1 = [16,0,18,1,20,2,22,3]
+; AVX512F-NEXT:    vxorps %xmm2, %xmm2, %xmm2
+; AVX512F-NEXT:    vpermt2ps %zmm2, %zmm1, %zmm0
+; AVX512F-NEXT:    # kill: def $ymm0 killed $ymm0 killed $zmm0
+; AVX512F-NEXT:    ret{{[l|q]}}
    %res = shufflevector <4 x float> zeroinitializer, <4 x float> %a, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
    ret <8 x float> %res
 }
@@ -60,61 +62,68 @@ define <4 x double> @expand2(<2 x double> %a) {
 
 ;expand 128 -> 256 include case <4 x i32> <8 x i32>
 define <8 x i32> @expand3(<4 x i32> %a ) {
-; SKX-LABEL: expand3:
-; SKX:       # %bb.0:
-; SKX-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
-; SKX-NEXT:    movb $-127, %al
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vpexpandd %ymm0, %ymm0 {%k1} {z}
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL-LABEL: expand3:
-; KNL:       # %bb.0:
-; KNL-NEXT:    vbroadcastsd %xmm0, %ymm0
-; KNL-NEXT:    vxorps %xmm1, %xmm1, %xmm1
-; KNL-NEXT:    vblendps {{.*#+}} ymm0 = ymm0[0],ymm1[1,2,3,4,5,6],ymm0[7]
-; KNL-NEXT:    ret{{[l|q]}}
+; AVX512-LABEL: expand3:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
+; AVX512-NEXT:    movb $-127, %al
+; AVX512-NEXT:    kmovd %eax, %k1
+; AVX512-NEXT:    vpexpandd %ymm0, %ymm0 {%k1} {z}
+; AVX512-NEXT:    ret{{[l|q]}}
+;
+; AVX512F-LABEL: expand3:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    vbroadcastsd %xmm0, %ymm0
+; AVX512F-NEXT:    vxorps %xmm1, %xmm1, %xmm1
+; AVX512F-NEXT:    vblendps {{.*#+}} ymm0 = ymm0[0],ymm1[1,2,3,4,5,6],ymm0[7]
+; AVX512F-NEXT:    ret{{[l|q]}}
    %res = shufflevector <4 x i32> zeroinitializer, <4 x i32> %a, <8 x i32> <i32 4, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0,i32 5>
    ret <8 x i32> %res
 }
 
 ;expand 128 -> 256 include case <2 x i64> <4 x i64>
 define <4 x i64> @expand4(<2 x i64> %a ) {
-; SKX-LABEL: expand4:
-; SKX:       # %bb.0:
-; SKX-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
-; SKX-NEXT:    movb $9, %al
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vpexpandq %ymm0, %ymm0 {%k1} {z}
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL-LABEL: expand4:
-; KNL:       # %bb.0:
-; KNL-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
-; KNL-NEXT:    vperm2f128 {{.*#+}} ymm1 = zero,zero,ymm0[0,1]
-; KNL-NEXT:    vmovaps %xmm0, %xmm0
-; KNL-NEXT:    vblendps {{.*#+}} ymm0 = ymm0[0,1],ymm1[2,3],ymm0[4,5],ymm1[6,7]
-; KNL-NEXT:    ret{{[l|q]}}
+; AVX512-LABEL: expand4:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
+; AVX512-NEXT:    movb $9, %al
+; AVX512-NEXT:    kmovd %eax, %k1
+; AVX512-NEXT:    vpexpandq %ymm0, %ymm0 {%k1} {z}
+; AVX512-NEXT:    ret{{[l|q]}}
+;
+; AVX512F-LABEL: expand4:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
+; AVX512F-NEXT:    vperm2f128 {{.*#+}} ymm1 = zero,zero,ymm0[0,1]
+; AVX512F-NEXT:    vmovaps %xmm0, %xmm0
+; AVX512F-NEXT:    vblendps {{.*#+}} ymm0 = ymm0[0,1],ymm1[2,3],ymm0[4,5],ymm1[6,7]
+; AVX512F-NEXT:    ret{{[l|q]}}
    %res = shufflevector <2 x i64> zeroinitializer, <2 x i64> %a, <4 x i32> <i32 2, i32 0, i32 0, i32 3>
    ret <4 x i64> %res
 }
 
 ;Negative test for 128-> 256
 define <8 x float> @expand5(<4 x float> %a ) {
-; SKX-LABEL: expand5:
-; SKX:       # %bb.0:
-; SKX-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
-; SKX-NEXT:    vxorps %xmm1, %xmm1, %xmm1
-; SKX-NEXT:    vmovaps {{.*#+}} ymm2 = [8,0,10,0,12,0,14,0]
-; SKX-NEXT:    vpermt2ps %ymm1, %ymm2, %ymm0
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL-LABEL: expand5:
-; KNL:       # %bb.0:
-; KNL-NEXT:    vbroadcastss %xmm0, %ymm0
-; KNL-NEXT:    vxorps %xmm1, %xmm1, %xmm1
-; KNL-NEXT:    vblendps {{.*#+}} ymm0 = ymm1[0],ymm0[1],ymm1[2],ymm0[3],ymm1[4],ymm0[5],ymm1[6],ymm0[7]
-; KNL-NEXT:    ret{{[l|q]}}
+; AVX512-SLOW-LABEL: expand5:
+; AVX512-SLOW:       # %bb.0:
+; AVX512-SLOW-NEXT:    vbroadcastss %xmm0, %ymm0
+; AVX512-SLOW-NEXT:    vxorps %xmm1, %xmm1, %xmm1
+; AVX512-SLOW-NEXT:    vblendps {{.*#+}} ymm0 = ymm1[0],ymm0[1],ymm1[2],ymm0[3],ymm1[4],ymm0[5],ymm1[6],ymm0[7]
+; AVX512-SLOW-NEXT:    ret{{[l|q]}}
+;
+; AVX512-FAST-LABEL: expand5:
+; AVX512-FAST:       # %bb.0:
+; AVX512-FAST-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
+; AVX512-FAST-NEXT:    vxorps %xmm1, %xmm1, %xmm1
+; AVX512-FAST-NEXT:    vmovaps {{.*#+}} ymm2 = [8,0,10,0,12,0,14,0]
+; AVX512-FAST-NEXT:    vpermt2ps %ymm1, %ymm2, %ymm0
+; AVX512-FAST-NEXT:    ret{{[l|q]}}
+;
+; AVX512F-LABEL: expand5:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    vbroadcastss %xmm0, %ymm0
+; AVX512F-NEXT:    vxorps %xmm1, %xmm1, %xmm1
+; AVX512F-NEXT:    vblendps {{.*#+}} ymm0 = ymm1[0],ymm0[1],ymm1[2],ymm0[3],ymm1[4],ymm0[5],ymm1[6],ymm0[7]
+; AVX512F-NEXT:    ret{{[l|q]}}
    %res = shufflevector <4 x float> zeroinitializer, <4 x float> %a, <8 x i32> <i32 0, i32 4, i32 1, i32 4, i32 2, i32 4, i32 3, i32 4>
    ret <8 x float> %res
 }
@@ -131,102 +140,102 @@ define <8 x float> @expand6(<4 x float> %a ) {
 }
 
 define <16 x float> @expand7(<8 x float> %a) {
-; SKX-LABEL: expand7:
-; SKX:       # %bb.0:
-; SKX-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
-; SKX-NEXT:    movw $1285, %ax # imm = 0x505
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vexpandps %zmm0, %zmm0 {%k1} {z}
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL-LABEL: expand7:
-; KNL:       # %bb.0:
-; KNL-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
-; KNL-NEXT:    movw $1285, %ax # imm = 0x505
-; KNL-NEXT:    kmovw %eax, %k1
-; KNL-NEXT:    vexpandps %zmm0, %zmm0 {%k1} {z}
-; KNL-NEXT:    ret{{[l|q]}}
+; AVX512-LABEL: expand7:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
+; AVX512-NEXT:    movw $1285, %ax # imm = 0x505
+; AVX512-NEXT:    kmovd %eax, %k1
+; AVX512-NEXT:    vexpandps %zmm0, %zmm0 {%k1} {z}
+; AVX512-NEXT:    ret{{[l|q]}}
+;
+; AVX512F-LABEL: expand7:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
+; AVX512F-NEXT:    movw $1285, %ax # imm = 0x505
+; AVX512F-NEXT:    kmovw %eax, %k1
+; AVX512F-NEXT:    vexpandps %zmm0, %zmm0 {%k1} {z}
+; AVX512F-NEXT:    ret{{[l|q]}}
    %res = shufflevector <8 x float> %a, <8 x float> zeroinitializer, <16 x i32> <i32 0, i32 8, i32 1, i32 8, i32 8, i32 8, i32 8, i32 8, i32 2, i32 8, i32 3, i32 8, i32 8, i32 8, i32 8, i32 8>
    ret <16 x float> %res
 }
 
 define <16 x float> @expand8(<8 x float> %a ) {
-; SKX-LABEL: expand8:
-; SKX:       # %bb.0:
-; SKX-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
-; SKX-NEXT:    movw $-21846, %ax # imm = 0xAAAA
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vexpandps %zmm0, %zmm0 {%k1} {z}
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL-LABEL: expand8:
-; KNL:       # %bb.0:
-; KNL-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
-; KNL-NEXT:    movw $-21846, %ax # imm = 0xAAAA
-; KNL-NEXT:    kmovw %eax, %k1
-; KNL-NEXT:    vexpandps %zmm0, %zmm0 {%k1} {z}
-; KNL-NEXT:    ret{{[l|q]}}
+; AVX512-LABEL: expand8:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
+; AVX512-NEXT:    movw $-21846, %ax # imm = 0xAAAA
+; AVX512-NEXT:    kmovd %eax, %k1
+; AVX512-NEXT:    vexpandps %zmm0, %zmm0 {%k1} {z}
+; AVX512-NEXT:    ret{{[l|q]}}
+;
+; AVX512F-LABEL: expand8:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
+; AVX512F-NEXT:    movw $-21846, %ax # imm = 0xAAAA
+; AVX512F-NEXT:    kmovw %eax, %k1
+; AVX512F-NEXT:    vexpandps %zmm0, %zmm0 {%k1} {z}
+; AVX512F-NEXT:    ret{{[l|q]}}
    %res = shufflevector <8 x float> zeroinitializer, <8 x float> %a, <16 x i32> <i32 0, i32 8, i32 1, i32 9, i32 2, i32 10, i32 3, i32 11, i32 4, i32 12, i32 5, i32 13, i32 6, i32 14, i32 7, i32 15>
    ret <16 x float> %res
 }
 
 ;expand 256 -> 512 include <4 x double> <8 x double>
 define <8 x double> @expand9(<4 x double> %a) {
-; SKX-LABEL: expand9:
-; SKX:       # %bb.0:
-; SKX-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
-; SKX-NEXT:    movb $-127, %al
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vexpandpd %zmm0, %zmm0 {%k1} {z}
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL-LABEL: expand9:
-; KNL:       # %bb.0:
-; KNL-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
-; KNL-NEXT:    movb $-127, %al
-; KNL-NEXT:    kmovw %eax, %k1
-; KNL-NEXT:    vexpandpd %zmm0, %zmm0 {%k1} {z}
-; KNL-NEXT:    ret{{[l|q]}}
+; AVX512-LABEL: expand9:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
+; AVX512-NEXT:    movb $-127, %al
+; AVX512-NEXT:    kmovd %eax, %k1
+; AVX512-NEXT:    vexpandpd %zmm0, %zmm0 {%k1} {z}
+; AVX512-NEXT:    ret{{[l|q]}}
+;
+; AVX512F-LABEL: expand9:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
+; AVX512F-NEXT:    movb $-127, %al
+; AVX512F-NEXT:    kmovw %eax, %k1
+; AVX512F-NEXT:    vexpandpd %zmm0, %zmm0 {%k1} {z}
+; AVX512F-NEXT:    ret{{[l|q]}}
    %res = shufflevector <4 x double> %a, <4 x double> zeroinitializer, <8 x i32> <i32 0, i32 4, i32 4, i32 4, i32 4, i32 4, i32 4, i32 1>
    ret <8 x double> %res
 }
 
 define <16 x i32> @expand10(<8 x i32> %a ) {
-; SKX-LABEL: expand10:
-; SKX:       # %bb.0:
-; SKX-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
-; SKX-NEXT:    movw $-21846, %ax # imm = 0xAAAA
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vpexpandd %zmm0, %zmm0 {%k1} {z}
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL-LABEL: expand10:
-; KNL:       # %bb.0:
-; KNL-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
-; KNL-NEXT:    movw $-21846, %ax # imm = 0xAAAA
-; KNL-NEXT:    kmovw %eax, %k1
-; KNL-NEXT:    vpexpandd %zmm0, %zmm0 {%k1} {z}
-; KNL-NEXT:    ret{{[l|q]}}
+; AVX512-LABEL: expand10:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
+; AVX512-NEXT:    movw $-21846, %ax # imm = 0xAAAA
+; AVX512-NEXT:    kmovd %eax, %k1
+; AVX512-NEXT:    vpexpandd %zmm0, %zmm0 {%k1} {z}
+; AVX512-NEXT:    ret{{[l|q]}}
+;
+; AVX512F-LABEL: expand10:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
+; AVX512F-NEXT:    movw $-21846, %ax # imm = 0xAAAA
+; AVX512F-NEXT:    kmovw %eax, %k1
+; AVX512F-NEXT:    vpexpandd %zmm0, %zmm0 {%k1} {z}
+; AVX512F-NEXT:    ret{{[l|q]}}
    %res = shufflevector <8 x i32> zeroinitializer, <8 x i32> %a, <16 x i32> <i32 0, i32 8, i32 1, i32 9, i32 2, i32 10, i32 3, i32 11, i32 4, i32 12, i32 5, i32 13, i32 6, i32 14, i32 7, i32 15>
    ret <16 x i32> %res
 }
 
 define <8 x i64> @expand11(<4 x i64> %a) {
-; SKX-LABEL: expand11:
-; SKX:       # %bb.0:
-; SKX-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
-; SKX-NEXT:    movb $-127, %al
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vpexpandq %zmm0, %zmm0 {%k1} {z}
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL-LABEL: expand11:
-; KNL:       # %bb.0:
-; KNL-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
-; KNL-NEXT:    movb $-127, %al
-; KNL-NEXT:    kmovw %eax, %k1
-; KNL-NEXT:    vpexpandq %zmm0, %zmm0 {%k1} {z}
-; KNL-NEXT:    ret{{[l|q]}}
+; AVX512-LABEL: expand11:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
+; AVX512-NEXT:    movb $-127, %al
+; AVX512-NEXT:    kmovd %eax, %k1
+; AVX512-NEXT:    vpexpandq %zmm0, %zmm0 {%k1} {z}
+; AVX512-NEXT:    ret{{[l|q]}}
+;
+; AVX512F-LABEL: expand11:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
+; AVX512F-NEXT:    movb $-127, %al
+; AVX512F-NEXT:    kmovw %eax, %k1
+; AVX512F-NEXT:    vpexpandq %zmm0, %zmm0 {%k1} {z}
+; AVX512F-NEXT:    ret{{[l|q]}}
    %res = shufflevector <4 x i64> %a, <4 x i64> zeroinitializer, <8 x i32> <i32 0, i32 4, i32 4, i32 4, i32 4, i32 4, i32 4, i32 1>
    ret <8 x i64> %res
 }
@@ -258,22 +267,22 @@ define <16 x float> @expand13(<8 x float> %a ) {
 ; The function checks for a case where the vector is mixed values vector ,and the mask points on zero elements from this vector.
 
 define <8 x float> @expand14(<4 x float> %a) {
-; SKX-LABEL: expand14:
-; SKX:       # %bb.0:
-; SKX-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
-; SKX-NEXT:    movb $20, %al
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vexpandps %ymm0, %ymm0 {%k1} {z}
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL-LABEL: expand14:
-; KNL:       # %bb.0:
-; KNL-NEXT:    # kill: def $xmm0 killed $xmm0 def $zmm0
-; KNL-NEXT:    vmovaps {{.*#+}} ymm1 = [16,17,0,19,1,21,22,23]
-; KNL-NEXT:    vxorps %xmm2, %xmm2, %xmm2
-; KNL-NEXT:    vpermt2ps %zmm2, %zmm1, %zmm0
-; KNL-NEXT:    # kill: def $ymm0 killed $ymm0 killed $zmm0
-; KNL-NEXT:    ret{{[l|q]}}
+; AVX512-LABEL: expand14:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
+; AVX512-NEXT:    movb $20, %al
+; AVX512-NEXT:    kmovd %eax, %k1
+; AVX512-NEXT:    vexpandps %ymm0, %ymm0 {%k1} {z}
+; AVX512-NEXT:    ret{{[l|q]}}
+;
+; AVX512F-LABEL: expand14:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    # kill: def $xmm0 killed $xmm0 def $zmm0
+; AVX512F-NEXT:    vmovaps {{.*#+}} ymm1 = [16,17,0,19,1,21,22,23]
+; AVX512F-NEXT:    vxorps %xmm2, %xmm2, %xmm2
+; AVX512F-NEXT:    vpermt2ps %zmm2, %zmm1, %zmm0
+; AVX512F-NEXT:    # kill: def $ymm0 killed $ymm0 killed $zmm0
+; AVX512F-NEXT:    ret{{[l|q]}}
    %addV = fadd <4 x float> <float 0.0,float 1.0,float 2.0,float 0.0> , <float 0.0,float 1.0,float 2.0,float 0.0>
    %res = shufflevector <4 x float> %addV, <4 x float> %a, <8 x i32> <i32 3, i32 3, i32 4, i32 0, i32 5, i32 0, i32 0, i32 0>
    ret <8 x float> %res
@@ -281,148 +290,154 @@ define <8 x float> @expand14(<4 x float> %a) {
 
 ;Negative test.
 define <8 x float> @expand15(<4 x float> %a) {
-; SKX-LABEL: expand15:
-; SKX:       # %bb.0:
-; SKX-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
-; SKX-NEXT:    vmovaps {{.*#+}} ymm1 = <u,u,0,u,1,u,u,u>
-; SKX-NEXT:    vpermps %ymm0, %ymm1, %ymm0
-; SKX-NEXT:    vblendps {{.*#+}} ymm0 = mem[0,1],ymm0[2],mem[3],ymm0[4],mem[5,6,7]
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL-LABEL: expand15:
-; KNL:       # %bb.0:
-; KNL-NEXT:    vshufps {{.*#+}} xmm0 = xmm0[0,1,1,3]
-; KNL-NEXT:    vpermpd {{.*#+}} ymm0 = ymm0[0,0,1,3]
-; KNL-NEXT:    vblendps {{.*#+}} ymm0 = mem[0,1],ymm0[2],mem[3],ymm0[4],mem[5,6,7]
-; KNL-NEXT:    ret{{[l|q]}}
+; AVX512-SLOW-LABEL: expand15:
+; AVX512-SLOW:       # %bb.0:
+; AVX512-SLOW-NEXT:    vshufps {{.*#+}} xmm0 = xmm0[0,1,1,3]
+; AVX512-SLOW-NEXT:    vpermpd {{.*#+}} ymm0 = ymm0[0,0,1,3]
+; AVX512-SLOW-NEXT:    vblendps {{.*#+}} ymm0 = mem[0,1],ymm0[2],mem[3],ymm0[4],mem[5,6,7]
+; AVX512-SLOW-NEXT:    ret{{[l|q]}}
+;
+; AVX512-FAST-LABEL: expand15:
+; AVX512-FAST:       # %bb.0:
+; AVX512-FAST-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
+; AVX512-FAST-NEXT:    vmovaps {{.*#+}} ymm1 = <u,u,0,u,1,u,u,u>
+; AVX512-FAST-NEXT:    vpermps %ymm0, %ymm1, %ymm0
+; AVX512-FAST-NEXT:    vblendps {{.*#+}} ymm0 = mem[0,1],ymm0[2],mem[3],ymm0[4],mem[5,6,7]
+; AVX512-FAST-NEXT:    ret{{[l|q]}}
+;
+; AVX512F-LABEL: expand15:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    vshufps {{.*#+}} xmm0 = xmm0[0,1,1,3]
+; AVX512F-NEXT:    vpermpd {{.*#+}} ymm0 = ymm0[0,0,1,3]
+; AVX512F-NEXT:    vblendps {{.*#+}} ymm0 = mem[0,1],ymm0[2],mem[3],ymm0[4],mem[5,6,7]
+; AVX512F-NEXT:    ret{{[l|q]}}
    %addV = fadd <4 x float> <float 0.0,float 1.0,float 2.0,float 0.0> , <float 0.0,float 1.0,float 2.0,float 0.0>
    %res = shufflevector <4 x float> %addV, <4 x float> %a, <8 x i32> <i32 0, i32 1, i32 4, i32 0, i32 5, i32 0, i32 0, i32 0>
    ret <8 x float> %res
 }
 
-
 ; Shuffle to blend test
 
 define <64 x i8> @test_mm512_mask_blend_epi8(<64 x i8> %A, <64 x i8> %W){
-; SKX64-LABEL: test_mm512_mask_blend_epi8:
-; SKX64:       # %bb.0: # %entry
-; SKX64-NEXT:    movabsq $-6148914691236517206, %rax # imm = 0xAAAAAAAAAAAAAAAA
-; SKX64-NEXT:    kmovq %rax, %k1
-; SKX64-NEXT:    vpblendmb %zmm0, %zmm1, %zmm0 {%k1}
-; SKX64-NEXT:    retq
-;
-; KNL-LABEL: test_mm512_mask_blend_epi8:
-; KNL:       # %bb.0: # %entry
-; KNL-NEXT:    vpbroadcastw {{.*#+}} ymm2 = [255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255]
-; KNL-NEXT:    vinserti64x4 $1, %ymm2, %zmm2, %zmm2
-; KNL-NEXT:    vpternlogq $216, %zmm2, %zmm1, %zmm0
-; KNL-NEXT:    ret{{[l|q]}}
-;
-; SKX32-LABEL: test_mm512_mask_blend_epi8:
-; SKX32:       # %bb.0: # %entry
-; SKX32-NEXT:    movl $-1431655766, %eax # imm = 0xAAAAAAAA
-; SKX32-NEXT:    kmovd %eax, %k0
-; SKX32-NEXT:    kunpckdq %k0, %k0, %k1
-; SKX32-NEXT:    vpblendmb %zmm0, %zmm1, %zmm0 {%k1}
-; SKX32-NEXT:    retl
+; X86-AVX512-LABEL: test_mm512_mask_blend_epi8:
+; X86-AVX512:       # %bb.0: # %entry
+; X86-AVX512-NEXT:    movl $-1431655766, %eax # imm = 0xAAAAAAAA
+; X86-AVX512-NEXT:    kmovd %eax, %k0
+; X86-AVX512-NEXT:    kunpckdq %k0, %k0, %k1
+; X86-AVX512-NEXT:    vpblendmb %zmm0, %zmm1, %zmm0 {%k1}
+; X86-AVX512-NEXT:    retl
+;
+; X64-AVX512-LABEL: test_mm512_mask_blend_epi8:
+; X64-AVX512:       # %bb.0: # %entry
+; X64-AVX512-NEXT:    movabsq $-6148914691236517206, %rax # imm = 0xAAAAAAAAAAAAAAAA
+; X64-AVX512-NEXT:    kmovq %rax, %k1
+; X64-AVX512-NEXT:    vpblendmb %zmm0, %zmm1, %zmm0 {%k1}
+; X64-AVX512-NEXT:    retq
+;
+; AVX512F-LABEL: test_mm512_mask_blend_epi8:
+; AVX512F:       # %bb.0: # %entry
+; AVX512F-NEXT:    vpbroadcastw {{.*#+}} ymm2 = [255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255]
+; AVX512F-NEXT:    vinserti64x4 $1, %ymm2, %zmm2, %zmm2
+; AVX512F-NEXT:    vpternlogq $216, %zmm2, %zmm1, %zmm0
+; AVX512F-NEXT:    ret{{[l|q]}}
 entry:
   %0 = shufflevector <64 x i8> %A, <64 x i8> %W, <64 x i32>  <i32 64, i32 1, i32 66, i32 3, i32 68, i32 5, i32 70, i32 7, i32 72, i32 9, i32 74, i32 11, i32 76, i32 13, i32 78, i32 15, i32 80, i32 17, i32 82, i32 19, i32 84, i32 21, i32 86, i32 23, i32 88, i32 25, i32 90, i32 27, i32 92, i32 29, i32 94, i32 31, i32 96, i32 33, i32 98, i32 35, i32 100, i32 37, i32 102, i32 39, i32 104, i32 41, i32 106, i32 43, i32 108, i32 45, i32 110, i32 47, i32 112, i32 49, i32 114, i32 51, i32 116, i32 53, i32 118, i32 55, i32 120, i32 57, i32 122, i32 59, i32 124, i32 61, i32 126, i32 63>
   ret <64 x i8> %0
 }
 
 define <32 x i16> @test_mm512_mask_blend_epi16(<32 x i16> %A, <32 x i16> %W){
-; SKX-LABEL: test_mm512_mask_blend_epi16:
-; SKX:       # %bb.0: # %entry
-; SKX-NEXT:    movl $-1431655766, %eax # imm = 0xAAAAAAAA
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vpblendmw %zmm0, %zmm1, %zmm0 {%k1}
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL64-LABEL: test_mm512_mask_blend_epi16:
-; KNL64:       # %bb.0: # %entry
-; KNL64-NEXT:    vpternlogd $216, {{\.?LCPI[0-9]+_[0-9]+}}(%rip){1to16}, %zmm1, %zmm0
-; KNL64-NEXT:    retq
-;
-; KNL32-LABEL: test_mm512_mask_blend_epi16:
-; KNL32:       # %bb.0: # %entry
-; KNL32-NEXT:    vpternlogd $216, {{\.?LCPI[0-9]+_[0-9]+}}{1to16}, %zmm1, %zmm0
-; KNL32-NEXT:    retl
+; AVX512-LABEL: test_mm512_mask_blend_epi16:
+; AVX512:       # %bb.0: # %entry
+; AVX512-NEXT:    movl $-1431655766, %eax # imm = 0xAAAAAAAA
+; AVX512-NEXT:    kmovd %eax, %k1
+; AVX512-NEXT:    vpblendmw %zmm0, %zmm1, %zmm0 {%k1}
+; AVX512-NEXT:    ret{{[l|q]}}
+;
+; X86-AVX512F-LABEL: test_mm512_mask_blend_epi16:
+; X86-AVX512F:       # %bb.0: # %entry
+; X86-AVX512F-NEXT:    vpternlogd $216, {{\.?LCPI[0-9]+_[0-9]+}}{1to16}, %zmm1, %zmm0
+; X86-AVX512F-NEXT:    retl
+;
+; X64-AVX512F-LABEL: test_mm512_mask_blend_epi16:
+; X64-AVX512F:       # %bb.0: # %entry
+; X64-AVX512F-NEXT:    vpternlogd $216, {{\.?LCPI[0-9]+_[0-9]+}}(%rip){1to16}, %zmm1, %zmm0
+; X64-AVX512F-NEXT:    retq
 entry:
   %0 = shufflevector <32 x i16> %A, <32 x i16> %W, <32 x i32>  <i32 32, i32 1, i32 34, i32 3, i32 36, i32 5, i32 38, i32 7, i32 40, i32 9, i32 42, i32 11, i32 44, i32 13, i32 46, i32 15, i32 48, i32 17, i32 50, i32 19, i32 52, i32 21, i32 54, i32 23, i32 56, i32 25, i32 58, i32 27, i32 60, i32 29, i32 62, i32 31>
   ret <32 x i16> %0
 }
 
 define <16 x i32> @test_mm512_mask_blend_epi32(<16 x i32> %A, <16 x i32> %W){
-; SKX-LABEL: test_mm512_mask_blend_epi32:
-; SKX:       # %bb.0: # %entry
-; SKX-NEXT:    movw $-21846, %ax # imm = 0xAAAA
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vpblendmd %zmm0, %zmm1, %zmm0 {%k1}
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL-LABEL: test_mm512_mask_blend_epi32:
-; KNL:       # %bb.0: # %entry
-; KNL-NEXT:    movw $-21846, %ax # imm = 0xAAAA
-; KNL-NEXT:    kmovw %eax, %k1
-; KNL-NEXT:    vpblendmd %zmm0, %zmm1, %zmm0 {%k1}
-; KNL-NEXT:    ret{{[l|q]}}
+; AVX512-LABEL: test_mm512_mask_blend_epi32:
+; AVX512:       # %bb.0: # %entry
+; AVX512-NEXT:    movw $-21846, %ax # imm = 0xAAAA
+; AVX512-NEXT:    kmovd %eax, %k1
+; AVX512-NEXT:    vpblendmd %zmm0, %zmm1, %zmm0 {%k1}
+; AVX512-NEXT:    ret{{[l|q]}}
+;
+; AVX512F-LABEL: test_mm512_mask_blend_epi32:
+; AVX512F:       # %bb.0: # %entry
+; AVX512F-NEXT:    movw $-21846, %ax # imm = 0xAAAA
+; AVX512F-NEXT:    kmovw %eax, %k1
+; AVX512F-NEXT:    vpblendmd %zmm0, %zmm1, %zmm0 {%k1}
+; AVX512F-NEXT:    ret{{[l|q]}}
 entry:
   %0 = shufflevector <16 x i32> %A, <16 x i32> %W, <16 x i32>  <i32 16, i32 1, i32 18, i32 3, i32 20, i32 5, i32 22, i32 7, i32 24, i32 9, i32 26, i32 11, i32 28, i32 13, i32 30, i32 15>
   ret <16 x i32> %0
 }
 
 define <8 x i64> @test_mm512_mask_blend_epi64(<8 x i64> %A, <8 x i64> %W){
-; SKX-LABEL: test_mm512_mask_blend_epi64:
-; SKX:       # %bb.0: # %entry
-; SKX-NEXT:    movb $-86, %al
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vpblendmq %zmm0, %zmm1, %zmm0 {%k1}
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL-LABEL: test_mm512_mask_blend_epi64:
-; KNL:       # %bb.0: # %entry
-; KNL-NEXT:    movb $-86, %al
-; KNL-NEXT:    kmovw %eax, %k1
-; KNL-NEXT:    vpblendmq %zmm0, %zmm1, %zmm0 {%k1}
-; KNL-NEXT:    ret{{[l|q]}}
+; AVX512-LABEL: test_mm512_mask_blend_epi64:
+; AVX512:       # %bb.0: # %entry
+; AVX512-NEXT:    movb $-86, %al
+; AVX512-NEXT:    kmovd %eax, %k1
+; AVX512-NEXT:    vpblendmq %zmm0, %zmm1, %zmm0 {%k1}
+; AVX512-NEXT:    ret{{[l|q]}}
+;
+; AVX512F-LABEL: test_mm512_mask_blend_epi64:
+; AVX512F:       # %bb.0: # %entry
+; AVX512F-NEXT:    movb $-86, %al
+; AVX512F-NEXT:    kmovw %eax, %k1
+; AVX512F-NEXT:    vpblendmq %zmm0, %zmm1, %zmm0 {%k1}
+; AVX512F-NEXT:    ret{{[l|q]}}
 entry:
   %0 = shufflevector <8 x i64> %A, <8 x i64> %W, <8 x i32>  <i32 8, i32 1, i32 10, i32 3, i32 12, i32 5, i32 14, i32 7>
   ret <8 x i64> %0
 }
 
 define <16 x float> @test_mm512_mask_blend_ps(<16 x float> %A, <16 x float> %W){
-; SKX-LABEL: test_mm512_mask_blend_ps:
-; SKX:       # %bb.0: # %entry
-; SKX-NEXT:    movw $-21846, %ax # imm = 0xAAAA
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vblendmps %zmm0, %zmm1, %zmm0 {%k1}
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL-LABEL: test_mm512_mask_blend_ps:
-; KNL:       # %bb.0: # %entry
-; KNL-NEXT:    movw $-21846, %ax # imm = 0xAAAA
-; KNL-NEXT:    kmovw %eax, %k1
-; KNL-NEXT:    vblendmps %zmm0, %zmm1, %zmm0 {%k1}
-; KNL-NEXT:    ret{{[l|q]}}
+; AVX512-LABEL: test_mm512_mask_blend_ps:
+; AVX512:       # %bb.0: # %entry
+; AVX512-NEXT:    movw $-21846, %ax # imm = 0xAAAA
+; AVX512-NEXT:    kmovd %eax, %k1
+; AVX512-NEXT:    vblendmps %zmm0, %zmm1, %zmm0 {%k1}
+; AVX512-NEXT:    ret{{[l|q]}}
+;
+; AVX512F-LABEL: test_mm512_mask_blend_ps:
+; AVX512F:       # %bb.0: # %entry
+; AVX512F-NEXT:    movw $-21846, %ax # imm = 0xAAAA
+; AVX512F-NEXT:    kmovw %eax, %k1
+; AVX512F-NEXT:    vblendmps %zmm0, %zmm1, %zmm0 {%k1}
+; AVX512F-NEXT:    ret{{[l|q]}}
 entry:
   %0 = shufflevector <16 x float> %A, <16 x float> %W, <16 x i32>  <i32 16, i32 1, i32 18, i32 3, i32 20, i32 5, i32 22, i32 7, i32 24, i32 9, i32 26, i32 11, i32 28, i32 13, i32 30, i32 15>
   ret <16 x float> %0
 }
 
 define <8 x double> @test_mm512_mask_blend_pd(<8 x double> %A, <8 x double> %W){
-; SKX-LABEL: test_mm512_mask_blend_pd:
-; SKX:       # %bb.0: # %entry
-; SKX-NEXT:    movb $-88, %al
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vblendmpd %zmm0, %zmm1, %zmm0 {%k1}
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL-LABEL: test_mm512_mask_blend_pd:
-; KNL:       # %bb.0: # %entry
-; KNL-NEXT:    movb $-88, %al
-; KNL-NEXT:    kmovw %eax, %k1
-; KNL-NEXT:    vblendmpd %zmm0, %zmm1, %zmm0 {%k1}
-; KNL-NEXT:    ret{{[l|q]}}
+; AVX512-LABEL: test_mm512_mask_blend_pd:
+; AVX512:       # %bb.0: # %entry
+; AVX512-NEXT:    movb $-88, %al
+; AVX512-NEXT:    kmovd %eax, %k1
+; AVX512-NEXT:    vblendmpd %zmm0, %zmm1, %zmm0 {%k1}
+; AVX512-NEXT:    ret{{[l|q]}}
+;
+; AVX512F-LABEL: test_mm512_mask_blend_pd:
+; AVX512F:       # %bb.0: # %entry
+; AVX512F-NEXT:    movb $-88, %al
+; AVX512F-NEXT:    kmovw %eax, %k1
+; AVX512F-NEXT:    vblendmpd %zmm0, %zmm1, %zmm0 {%k1}
+; AVX512F-NEXT:    ret{{[l|q]}}
 entry:
   %0 = shufflevector <8 x double> %A, <8 x double> %W, <8 x i32>  <i32 8, i32 9, i32 10, i32 3, i32 12, i32 5, i32 14, i32 7>
   ret <8 x double> %0
@@ -430,36 +445,36 @@ entry:
 
 
 define <32 x i8> @test_mm256_mask_blend_epi8(<32 x i8> %A, <32 x i8> %W){
-; SKX-LABEL: test_mm256_mask_blend_epi8:
-; SKX:       # %bb.0: # %entry
-; SKX-NEXT:    movl $-1431655766, %eax # imm = 0xAAAAAAAA
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vpblendmb %ymm0, %ymm1, %ymm0 {%k1}
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL-LABEL: test_mm256_mask_blend_epi8:
-; KNL:       # %bb.0: # %entry
-; KNL-NEXT:    vmovdqa {{.*#+}} ymm2 = [255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0]
-; KNL-NEXT:    vpblendvb %ymm2, %ymm1, %ymm0, %ymm0
-; KNL-NEXT:    ret{{[l|q]}}
+; AVX512-LABEL: test_mm256_mask_blend_epi8:
+; AVX512:       # %bb.0: # %entry
+; AVX512-NEXT:    movl $-1431655766, %eax # imm = 0xAAAAAAAA
+; AVX512-NEXT:    kmovd %eax, %k1
+; AVX512-NEXT:    vpblendmb %ymm0, %ymm1, %ymm0 {%k1}
+; AVX512-NEXT:    ret{{[l|q]}}
+;
+; AVX512F-LABEL: test_mm256_mask_blend_epi8:
+; AVX512F:       # %bb.0: # %entry
+; AVX512F-NEXT:    vmovdqa {{.*#+}} ymm2 = [255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0]
+; AVX512F-NEXT:    vpblendvb %ymm2, %ymm1, %ymm0, %ymm0
+; AVX512F-NEXT:    ret{{[l|q]}}
 entry:
   %0 = shufflevector <32 x i8> %A, <32 x i8> %W, <32 x i32>  <i32 32, i32 1, i32 34, i32 3, i32 36, i32 5, i32 38, i32 7, i32 40, i32 9, i32 42, i32 11, i32 44, i32 13, i32 46, i32 15, i32 48, i32 17, i32 50, i32 19, i32 52, i32 21, i32 54, i32 23, i32 56, i32 25, i32 58, i32 27, i32 60, i32 29, i32 62, i32 31>
   ret <32 x i8> %0
 }
 
 define <16 x i8> @test_mm_mask_blend_epi8(<16 x i8> %A, <16 x i8> %W){
-; SKX-LABEL: test_mm_mask_blend_epi8:
-; SKX:       # %bb.0: # %entry
-; SKX-NEXT:    movw $-21846, %ax # imm = 0xAAAA
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vpblendmb %xmm0, %xmm1, %xmm0 {%k1}
-; SKX-NEXT:    ret{{[l|q]}}
-;
-; KNL-LABEL: test_mm_mask_blend_epi8:
-; KNL:       # %bb.0: # %entry
-; KNL-NEXT:    vmovdqa {{.*#+}} xmm2 = [255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0]
-; KNL-NEXT:    vpblendvb %xmm2, %xmm1, %xmm0, %xmm0
-; KNL-NEXT:    ret{{[l|q]}}
+; AVX512-LABEL: test_mm_mask_blend_epi8:
+; AVX512:       # %bb.0: # %entry
+; AVX512-NEXT:    movw $-21846, %ax # imm = 0xAAAA
+; AVX512-NEXT:    kmovd %eax, %k1
+; AVX512-NEXT:    vpblendmb %xmm0, %xmm1, %xmm0 {%k1}
+; AVX512-NEXT:    ret{{[l|q]}}
+;
+; AVX512F-LABEL: test_mm_mask_blend_epi8:
+; AVX512F:       # %bb.0: # %entry
+; AVX512F-NEXT:    vmovdqa {{.*#+}} xmm2 = [255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0]
+; AVX512F-NEXT:    vpblendvb %xmm2, %xmm1, %xmm0, %xmm0
+; AVX512F-NEXT:    ret{{[l|q]}}
 entry:
   %0 = shufflevector <16 x i8> %A, <16 x i8> %W, <16 x i32>  <i32 16, i32 1, i32 18, i32 3, i32 20, i32 5, i32 22, i32 7, i32 24, i32 9, i32 26, i32 11, i32 28, i32 13, i32 30, i32 15>
   ret <16 x i8> %0
@@ -467,41 +482,41 @@ entry:
 
 ; PR34370
 define <8 x float> @test_masked_permps_v8f32(ptr %vp, <8 x float> %vec2) {
-; SKX64-LABEL: test_masked_permps_v8f32:
-; SKX64:       # %bb.0:
-; SKX64-NEXT:    vmovaps (%rdi), %ymm2
-; SKX64-NEXT:    vmovaps {{.*#+}} ymm1 = [7,6,3,11,7,6,14,15]
-; SKX64-NEXT:    vpermi2ps %ymm0, %ymm2, %ymm1
-; SKX64-NEXT:    vmovaps %ymm1, %ymm0
-; SKX64-NEXT:    retq
-;
-; KNL64-LABEL: test_masked_permps_v8f32:
-; KNL64:       # %bb.0:
-; KNL64-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
-; KNL64-NEXT:    vmovaps (%rdi), %ymm1
-; KNL64-NEXT:    vmovaps {{.*#+}} ymm2 = [7,6,3,19,7,6,22,23]
-; KNL64-NEXT:    vpermt2ps %zmm0, %zmm2, %zmm1
-; KNL64-NEXT:    vmovaps %ymm1, %ymm0
-; KNL64-NEXT:    retq
-;
-; SKX32-LABEL: test_masked_permps_v8f32:
-; SKX32:       # %bb.0:
-; SKX32-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; SKX32-NEXT:    vmovaps (%eax), %ymm2
-; SKX32-NEXT:    vmovaps {{.*#+}} ymm1 = [7,6,3,11,7,6,14,15]
-; SKX32-NEXT:    vpermi2ps %ymm0, %ymm2, %ymm1
-; SKX32-NEXT:    vmovaps %ymm1, %ymm0
-; SKX32-NEXT:    retl
-;
-; KNL32-LABEL: test_masked_permps_v8f32:
-; KNL32:       # %bb.0:
-; KNL32-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
-; KNL32-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; KNL32-NEXT:    vmovaps (%eax), %ymm1
-; KNL32-NEXT:    vmovaps {{.*#+}} ymm2 = [7,6,3,19,7,6,22,23]
-; KNL32-NEXT:    vpermt2ps %zmm0, %zmm2, %zmm1
-; KNL32-NEXT:    vmovaps %ymm1, %ymm0
-; KNL32-NEXT:    retl
+; X86-AVX512-LABEL: test_masked_permps_v8f32:
+; X86-AVX512:       # %bb.0:
+; X86-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-AVX512-NEXT:    vmovaps (%eax), %ymm2
+; X86-AVX512-NEXT:    vmovaps {{.*#+}} ymm1 = [7,6,3,11,7,6,14,15]
+; X86-AVX512-NEXT:    vpermi2ps %ymm0, %ymm2, %ymm1
+; X86-AVX512-NEXT:    vmovaps %ymm1, %ymm0
+; X86-AVX512-NEXT:    retl
+;
+; X64-AVX512-LABEL: test_masked_permps_v8f32:
+; X64-AVX512:       # %bb.0:
+; X64-AVX512-NEXT:    vmovaps (%rdi), %ymm2
+; X64-AVX512-NEXT:    vmovaps {{.*#+}} ymm1 = [7,6,3,11,7,6,14,15]
+; X64-AVX512-NEXT:    vpermi2ps %ymm0, %ymm2, %ymm1
+; X64-AVX512-NEXT:    vmovaps %ymm1, %ymm0
+; X64-AVX512-NEXT:    retq
+;
+; X86-AVX512F-LABEL: test_masked_permps_v8f32:
+; X86-AVX512F:       # %bb.0:
+; X86-AVX512F-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
+; X86-AVX512F-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-AVX512F-NEXT:    vmovaps (%eax), %ymm1
+; X86-AVX512F-NEXT:    vmovaps {{.*#+}} ymm2 = [7,6,3,19,7,6,22,23]
+; X86-AVX512F-NEXT:    vpermt2ps %zmm0, %zmm2, %zmm1
+; X86-AVX512F-NEXT:    vmovaps %ymm1, %ymm0
+; X86-AVX512F-NEXT:    retl
+;
+; X64-AVX512F-LABEL: test_masked_permps_v8f32:
+; X64-AVX512F:       # %bb.0:
+; X64-AVX512F-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
+; X64-AVX512F-NEXT:    vmovaps (%rdi), %ymm1
+; X64-AVX512F-NEXT:    vmovaps {{.*#+}} ymm2 = [7,6,3,19,7,6,22,23]
+; X64-AVX512F-NEXT:    vpermt2ps %zmm0, %zmm2, %zmm1
+; X64-AVX512F-NEXT:    vmovaps %ymm1, %ymm0
+; X64-AVX512F-NEXT:    retq
   %vec = load <8 x float>, ptr %vp
   %shuf = shufflevector <8 x float> %vec, <8 x float> undef, <8 x i32> <i32 7, i32 6, i32 3, i32 0, i32 7, i32 6, i32 3, i32 0>
   %res = select <8 x i1> <i1 1, i1 1, i1 1, i1 0, i1 1, i1 1, i1 0, i1 0>, <8 x float> %shuf, <8 x float> %vec2
@@ -509,22 +524,39 @@ define <8 x float> @test_masked_permps_v8f32(ptr %vp, <8 x float> %vec2) {
 }
 
 define <16 x float> @test_masked_permps_v16f32(ptr %vp, <16 x float> %vec2) {
-; X64-LABEL: test_masked_permps_v16f32:
-; X64:       # %bb.0:
-; X64-NEXT:    vmovaps (%rdi), %zmm2
-; X64-NEXT:    vmovaps {{.*#+}} zmm1 = [15,13,11,19,14,12,22,23,7,6,3,27,7,29,3,31]
-; X64-NEXT:    vpermi2ps %zmm0, %zmm2, %zmm1
-; X64-NEXT:    vmovaps %zmm1, %zmm0
-; X64-NEXT:    retq
-;
-; X86-LABEL: test_masked_permps_v16f32:
-; X86:       # %bb.0:
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X86-NEXT:    vmovaps (%eax), %zmm2
-; X86-NEXT:    vmovaps {{.*#+}} zmm1 = [15,13,11,19,14,12,22,23,7,6,3,27,7,29,3,31]
-; X86-NEXT:    vpermi2ps %zmm0, %zmm2, %zmm1
-; X86-NEXT:    vmovaps %zmm1, %zmm0
-; X86-NEXT:    retl
+; X86-AVX512-LABEL: test_masked_permps_v16f32:
+; X86-AVX512:       # %bb.0:
+; X86-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-AVX512-NEXT:    vmovaps (%eax), %zmm2
+; X86-AVX512-NEXT:    vmovaps {{.*#+}} zmm1 = [15,13,11,19,14,12,22,23,7,6,3,27,7,29,3,31]
+; X86-AVX512-NEXT:    vpermi2ps %zmm0, %zmm2, %zmm1
+; X86-AVX512-NEXT:    vmovaps %zmm1, %zmm0
+; X86-AVX512-NEXT:    retl
+;
+; X64-AVX512-LABEL: test_masked_permps_v16f32:
+; X64-AVX512:       # %bb.0:
+; X64-AVX512-NEXT:    vmovaps (%rdi), %zmm2
+; X64-AVX512-NEXT:    vmovaps {{.*#+}} zmm1 = [15,13,11,19,14,12,22,23,7,6,3,27,7,29,3,31]
+; X64-AVX512-NEXT:    vpermi2ps %zmm0, %zmm2, %zmm1
+; X64-AVX512-NEXT:    vmovaps %zmm1, %zmm0
+; X64-AVX512-NEXT:    retq
+;
+; X86-AVX512F-LABEL: test_masked_permps_v16f32:
+; X86-AVX512F:       # %bb.0:
+; X86-AVX512F-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-AVX512F-NEXT:    vmovaps (%eax), %zmm2
+; X86-AVX512F-NEXT:    vmovaps {{.*#+}} zmm1 = [15,13,11,19,14,12,22,23,7,6,3,27,7,29,3,31]
+; X86-AVX512F-NEXT:    vpermi2ps %zmm0, %zmm2, %zmm1
+; X86-AVX512F-NEXT:    vmovaps %zmm1, %zmm0
+; X86-AVX512F-NEXT:    retl
+;
+; X64-AVX512F-LABEL: test_masked_permps_v16f32:
+; X64-AVX512F:       # %bb.0:
+; X64-AVX512F-NEXT:    vmovaps (%rdi), %zmm2
+; X64-AVX512F-NEXT:    vmovaps {{.*#+}} zmm1 = [15,13,11,19,14,12,22,23,7,6,3,27,7,29,3,31]
+; X64-AVX512F-NEXT:    vpermi2ps %zmm0, %zmm2, %zmm1
+; X64-AVX512F-NEXT:    vmovaps %zmm1, %zmm0
+; X64-AVX512F-NEXT:    retq
   %vec = load <16 x float>, ptr %vp
   %shuf = shufflevector <16 x float> %vec, <16 x float> undef, <16 x i32> <i32 15, i32 13, i32 11, i32 9, i32 14, i32 12, i32 10, i32 8, i32 7, i32 6, i32 3, i32 0, i32 7, i32 6, i32 3, i32 0>
   %res = select <16 x i1> <i1 1, i1 1, i1 1, i1 0, i1 1, i1 1, i1 0, i1 0, i1 1, i1 1, i1 1, i1 0, i1 1, i1 0, i1 1, i1 0>, <16 x float> %shuf, <16 x float> %vec2
@@ -532,51 +564,77 @@ define <16 x float> @test_masked_permps_v16f32(ptr %vp, <16 x float> %vec2) {
 }
 
 define void @test_demandedelts_pshufb_v32i8_v16i8(ptr %src, ptr %dst) {
-; SKX64-LABEL: test_demandedelts_pshufb_v32i8_v16i8:
-; SKX64:       # %bb.0:
-; SKX64-NEXT:    vpbroadcastd 44(%rdi), %xmm0
-; SKX64-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
-; SKX64-NEXT:    vmovdqa %ymm0, 672(%rsi)
-; SKX64-NEXT:    vmovdqa 208(%rdi), %xmm0
-; SKX64-NEXT:    vpshufb {{.*#+}} xmm0 = xmm0[4,5,6,7,0,1,2,3],zero,zero,zero,zero,zero,zero,zero,zero
-; SKX64-NEXT:    vmovdqa %ymm0, 832(%rsi)
-; SKX64-NEXT:    vzeroupper
-; SKX64-NEXT:    retq
-;
-; KNL64-LABEL: test_demandedelts_pshufb_v32i8_v16i8:
-; KNL64:       # %bb.0:
-; KNL64-NEXT:    vpbroadcastd 44(%rdi), %xmm0
-; KNL64-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
-; KNL64-NEXT:    vmovdqa %ymm0, 672(%rsi)
-; KNL64-NEXT:    vpshufd {{.*#+}} xmm0 = mem[1,0,2,3]
-; KNL64-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
-; KNL64-NEXT:    vmovdqa %ymm0, 832(%rsi)
-; KNL64-NEXT:    retq
-;
-; SKX32-LABEL: test_demandedelts_pshufb_v32i8_v16i8:
-; SKX32:       # %bb.0:
-; SKX32-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; SKX32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; SKX32-NEXT:    vpbroadcastd 44(%ecx), %xmm0
-; SKX32-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
-; SKX32-NEXT:    vmovdqa %ymm0, 672(%eax)
-; SKX32-NEXT:    vmovdqa 208(%ecx), %xmm0
-; SKX32-NEXT:    vpshufb {{.*#+}} xmm0 = xmm0[4,5,6,7,0,1,2,3],zero,zero,zero,zero,zero,zero,zero,zero
-; SKX32-NEXT:    vmovdqa %ymm0, 832(%eax)
-; SKX32-NEXT:    vzeroupper
-; SKX32-NEXT:    retl
-;
-; KNL32-LABEL: test_demandedelts_pshufb_v32i8_v16i8:
-; KNL32:       # %bb.0:
-; KNL32-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; KNL32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; KNL32-NEXT:    vpbroadcastd 44(%ecx), %xmm0
-; KNL32-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
-; KNL32-NEXT:    vmovdqa %ymm0, 672(%eax)
-; KNL32-NEXT:    vpshufd {{.*#+}} xmm0 = mem[1,0,2,3]
-; KNL32-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
-; KNL32-NEXT:    vmovdqa %ymm0, 832(%eax)
-; KNL32-NEXT:    retl
+; X86-AVX512-SLOW-LABEL: test_demandedelts_pshufb_v32i8_v16i8:
+; X86-AVX512-SLOW:       # %bb.0:
+; X86-AVX512-SLOW-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-AVX512-SLOW-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-AVX512-SLOW-NEXT:    vpbroadcastd 44(%ecx), %xmm0
+; X86-AVX512-SLOW-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
+; X86-AVX512-SLOW-NEXT:    vmovdqa %ymm0, 672(%eax)
+; X86-AVX512-SLOW-NEXT:    vpshufd {{.*#+}} xmm0 = mem[1,0,2,3]
+; X86-AVX512-SLOW-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
+; X86-AVX512-SLOW-NEXT:    vmovdqa %ymm0, 832(%eax)
+; X86-AVX512-SLOW-NEXT:    vzeroupper
+; X86-AVX512-SLOW-NEXT:    retl
+;
+; X64-AVX512-SLOW-LABEL: test_demandedelts_pshufb_v32i8_v16i8:
+; X64-AVX512-SLOW:       # %bb.0:
+; X64-AVX512-SLOW-NEXT:    vpbroadcastd 44(%rdi), %xmm0
+; X64-AVX512-SLOW-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
+; X64-AVX512-SLOW-NEXT:    vmovdqa %ymm0, 672(%rsi)
+; X64-AVX512-SLOW-NEXT:    vpshufd {{.*#+}} xmm0 = mem[1,0,2,3]
+; X64-AVX512-SLOW-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
+; X64-AVX512-SLOW-NEXT:    vmovdqa %ymm0, 832(%rsi)
+; X64-AVX512-SLOW-NEXT:    vzeroupper
+; X64-AVX512-SLOW-NEXT:    retq
+;
+; X86-AVX512-FAST-LABEL: test_demandedelts_pshufb_v32i8_v16i8:
+; X86-AVX512-FAST:       # %bb.0:
+; X86-AVX512-FAST-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-AVX512-FAST-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-AVX512-FAST-NEXT:    vpbroadcastd 44(%ecx), %xmm0
+; X86-AVX512-FAST-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
+; X86-AVX512-FAST-NEXT:    vmovdqa %ymm0, 672(%eax)
+; X86-AVX512-FAST-NEXT:    vmovdqa 208(%ecx), %xmm0
+; X86-AVX512-FAST-NEXT:    vpshufb {{.*#+}} xmm0 = xmm0[4,5,6,7,0,1,2,3],zero,zero,zero,zero,zero,zero,zero,zero
+; X86-AVX512-FAST-NEXT:    vmovdqa %ymm0, 832(%eax)
+; X86-AVX512-FAST-NEXT:    vzeroupper
+; X86-AVX512-FAST-NEXT:    retl
+;
+; X64-AVX512-FAST-LABEL: test_demandedelts_pshufb_v32i8_v16i8:
+; X64-AVX512-FAST:       # %bb.0:
+; X64-AVX512-FAST-NEXT:    vpbroadcastd 44(%rdi), %xmm0
+; X64-AVX512-FAST-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
+; X64-AVX512-FAST-NEXT:    vmovdqa %ymm0, 672(%rsi)
+; X64-AVX512-FAST-NEXT:    vmovdqa 208(%rdi), %xmm0
+; X64-AVX512-FAST-NEXT:    vpshufb {{.*#+}} xmm0 = xmm0[4,5,6,7,0,1,2,3],zero,zero,zero,zero,zero,zero,zero,zero
+; X64-AVX512-FAST-NEXT:    vmovdqa %ymm0, 832(%rsi)
+; X64-AVX512-FAST-NEXT:    vzeroupper
+; X64-AVX512-FAST-NEXT:    retq
+;
+; X86-AVX512F-LABEL: test_demandedelts_pshufb_v32i8_v16i8:
+; X86-AVX512F:       # %bb.0:
+; X86-AVX512F-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-AVX512F-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-AVX512F-NEXT:    vpbroadcastd 44(%ecx), %xmm0
+; X86-AVX512F-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
+; X86-AVX512F-NEXT:    vmovdqa %ymm0, 672(%eax)
+; X86-AVX512F-NEXT:    vpshufd {{.*#+}} xmm0 = mem[1,0,2,3]
+; X86-AVX512F-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
+; X86-AVX512F-NEXT:    vmovdqa %ymm0, 832(%eax)
+; X86-AVX512F-NEXT:    vzeroupper
+; X86-AVX512F-NEXT:    retl
+;
+; X64-AVX512F-LABEL: test_demandedelts_pshufb_v32i8_v16i8:
+; X64-AVX512F:       # %bb.0:
+; X64-AVX512F-NEXT:    vpbroadcastd 44(%rdi), %xmm0
+; X64-AVX512F-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
+; X64-AVX512F-NEXT:    vmovdqa %ymm0, 672(%rsi)
+; X64-AVX512F-NEXT:    vpshufd {{.*#+}} xmm0 = mem[1,0,2,3]
+; X64-AVX512F-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
+; X64-AVX512F-NEXT:    vmovdqa %ymm0, 832(%rsi)
+; X64-AVX512F-NEXT:    vzeroupper
+; X64-AVX512F-NEXT:    retq
   %t87 = load <16 x i32>, ptr %src, align 64
   %t88 = extractelement <16 x i32> %t87, i64 11
   %t89 = insertelement <8 x i32> <i32 undef, i32 undef, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0>, i32 %t88, i64 0
@@ -613,33 +671,35 @@ define <32 x float> @PR47534(<8 x float> %tmp) {
 @src1 = external dso_local local_unnamed_addr global %union1, align 64
 
 define void @PR43170(ptr %a0) {
-; SKX64-LABEL: PR43170:
-; SKX64:       # %bb.0: # %entry
-; SKX64-NEXT:    vmovaps src1(%rip), %ymm0
-; SKX64-NEXT:    vmovaps %zmm0, (%rdi)
-; SKX64-NEXT:    vzeroupper
-; SKX64-NEXT:    retq
-;
-; KNL64-LABEL: PR43170:
-; KNL64:       # %bb.0: # %entry
-; KNL64-NEXT:    vmovaps src1(%rip), %ymm0
-; KNL64-NEXT:    vmovaps %zmm0, (%rdi)
-; KNL64-NEXT:    retq
-;
-; SKX32-LABEL: PR43170:
-; SKX32:       # %bb.0: # %entry
-; SKX32-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; SKX32-NEXT:    vmovaps src1, %ymm0
-; SKX32-NEXT:    vmovaps %zmm0, (%eax)
-; SKX32-NEXT:    vzeroupper
-; SKX32-NEXT:    retl
-;
-; KNL32-LABEL: PR43170:
-; KNL32:       # %bb.0: # %entry
-; KNL32-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; KNL32-NEXT:    vmovaps src1, %ymm0
-; KNL32-NEXT:    vmovaps %zmm0, (%eax)
-; KNL32-NEXT:    retl
+; X86-AVX512-LABEL: PR43170:
+; X86-AVX512:       # %bb.0: # %entry
+; X86-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-AVX512-NEXT:    vmovaps src1, %ymm0
+; X86-AVX512-NEXT:    vmovaps %zmm0, (%eax)
+; X86-AVX512-NEXT:    vzeroupper
+; X86-AVX512-NEXT:    retl
+;
+; X64-AVX512-LABEL: PR43170:
+; X64-AVX512:       # %bb.0: # %entry
+; X64-AVX512-NEXT:    vmovaps src1(%rip), %ymm0
+; X64-AVX512-NEXT:    vmovaps %zmm0, (%rdi)
+; X64-AVX512-NEXT:    vzeroupper
+; X64-AVX512-NEXT:    retq
+;
+; X86-AVX512F-LABEL: PR43170:
+; X86-AVX512F:       # %bb.0: # %entry
+; X86-AVX512F-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-AVX512F-NEXT:    vmovaps src1, %ymm0
+; X86-AVX512F-NEXT:    vmovaps %zmm0, (%eax)
+; X86-AVX512F-NEXT:    vzeroupper
+; X86-AVX512F-NEXT:    retl
+;
+; X64-AVX512F-LABEL: PR43170:
+; X64-AVX512F:       # %bb.0: # %entry
+; X64-AVX512F-NEXT:    vmovaps src1(%rip), %ymm0
+; X64-AVX512F-NEXT:    vmovaps %zmm0, (%rdi)
+; X64-AVX512F-NEXT:    vzeroupper
+; X64-AVX512F-NEXT:    retq
 entry:
   %0 = load <8 x float>, ptr @src1, align 64
   %1 = shufflevector <8 x float> %0, <8 x float> zeroinitializer, <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
diff --git a/llvm/test/CodeGen/X86/vector-shuffle-combining-avx512f.ll b/llvm/test/CodeGen/X86/vector-shuffle-combining-avx512f.ll
index 04ae2f6fb83a7..27375e96b3160 100644
--- a/llvm/test/CodeGen/X86/vector-shuffle-combining-avx512f.ll
+++ b/llvm/test/CodeGen/X86/vector-shuffle-combining-avx512f.ll
@@ -362,19 +362,19 @@ define <16 x float> @combine_vpermt2var_16f32_vmovddup_mask(<16 x float> %x0, <1
 ; X86-LABEL: combine_vpermt2var_16f32_vmovddup_mask:
 ; X86:       # %bb.0:
 ; X86-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
-; X86-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = zmm0[0,1,0,1,4,5,4,5,8,9,8,9,12,13,12,13]
+; X86-NEXT:    vshufps {{.*#+}} zmm0 {%k1} {z} = zmm0[0,1,0,1,4,5,4,5,8,9,8,9,12,13,12,13]
 ; X86-NEXT:    retl
 ;
 ; X64-AVX512F-LABEL: combine_vpermt2var_16f32_vmovddup_mask:
 ; X64-AVX512F:       # %bb.0:
 ; X64-AVX512F-NEXT:    kmovw %edi, %k1
-; X64-AVX512F-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = zmm0[0,1,0,1,4,5,4,5,8,9,8,9,12,13,12,13]
+; X64-AVX512F-NEXT:    vshufps {{.*#+}} zmm0 {%k1} {z} = zmm0[0,1,0,1,4,5,4,5,8,9,8,9,12,13,12,13]
 ; X64-AVX512F-NEXT:    retq
 ;
 ; X64-AVX512BW-LABEL: combine_vpermt2var_16f32_vmovddup_mask:
 ; X64-AVX512BW:       # %bb.0:
 ; X64-AVX512BW-NEXT:    kmovd %edi, %k1
-; X64-AVX512BW-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = zmm0[0,1,0,1,4,5,4,5,8,9,8,9,12,13,12,13]
+; X64-AVX512BW-NEXT:    vshufps {{.*#+}} zmm0 {%k1} {z} = zmm0[0,1,0,1,4,5,4,5,8,9,8,9,12,13,12,13]
 ; X64-AVX512BW-NEXT:    retq
   %res0 = call <16 x float> @llvm.x86.avx512.maskz.vpermt2var.ps.512(<16 x i32> <i32 0, i32 1, i32 0, i32 1, i32 4, i32 5, i32 4, i32 5, i32 8, i32 9, i32 8, i32 9, i32 12, i32 13, i32 12, i32 13>, <16 x float> %x0, <16 x float> %x1, i16 %m)
   ret <16 x float> %res0
@@ -543,19 +543,19 @@ define <16 x float> @combine_vpermt2var_16f32_vpermilps_mask(<16 x float> %x0, <
 ; X86-LABEL: combine_vpermt2var_16f32_vpermilps_mask:
 ; X86:       # %bb.0:
 ; X86-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
-; X86-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = zmm0[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
+; X86-NEXT:    vshufps {{.*#+}} zmm0 {%k1} {z} = zmm0[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
 ; X86-NEXT:    retl
 ;
 ; X64-AVX512F-LABEL: combine_vpermt2var_16f32_vpermilps_mask:
 ; X64-AVX512F:       # %bb.0:
 ; X64-AVX512F-NEXT:    kmovw %edi, %k1
-; X64-AVX512F-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = zmm0[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
+; X64-AVX512F-NEXT:    vshufps {{.*#+}} zmm0 {%k1} {z} = zmm0[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
 ; X64-AVX512F-NEXT:    retq
 ;
 ; X64-AVX512BW-LABEL: combine_vpermt2var_16f32_vpermilps_mask:
 ; X64-AVX512BW:       # %bb.0:
 ; X64-AVX512BW-NEXT:    kmovd %edi, %k1
-; X64-AVX512BW-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = zmm0[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
+; X64-AVX512BW-NEXT:    vshufps {{.*#+}} zmm0 {%k1} {z} = zmm0[3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12]
 ; X64-AVX512BW-NEXT:    retq
   %res0 = call <16 x float> @llvm.x86.avx512.maskz.vpermt2var.ps.512(<16 x i32> <i32 3, i32 2, i32 1, i32 0, i32 7, i32 6, i32 5, i32 4, i32 11, i32 10, i32 9, i32 8, i32 15, i32 14, i32 13, i32 12>, <16 x float> %x0, <16 x float> %x1, i16 %m)
   ret <16 x float> %res0
diff --git a/llvm/test/CodeGen/X86/vector-trunc.ll b/llvm/test/CodeGen/X86/vector-trunc.ll
index c7bf6daf803ab..4b73e5a590a39 100644
--- a/llvm/test/CodeGen/X86/vector-trunc.ll
+++ b/llvm/test/CodeGen/X86/vector-trunc.ll
@@ -1392,37 +1392,12 @@ define <8 x i32> @trunc2x4i64_8i32(<4 x i64> %a, <4 x i64> %b) {
 ; AVX2-FAST-PERLANE-NEXT:    vshufps {{.*#+}} ymm0 = ymm0[0,2],ymm2[0,2],ymm0[4,6],ymm2[4,6]
 ; AVX2-FAST-PERLANE-NEXT:    retq
 ;
-; AVX512F-LABEL: trunc2x4i64_8i32:
-; AVX512F:       # %bb.0: # %entry
-; AVX512F-NEXT:    # kill: def $ymm1 killed $ymm1 def $zmm1
-; AVX512F-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
-; AVX512F-NEXT:    vpmovqd %zmm0, %ymm0
-; AVX512F-NEXT:    vpmovqd %zmm1, %ymm1
-; AVX512F-NEXT:    vinserti128 $1, %xmm1, %ymm0, %ymm0
-; AVX512F-NEXT:    retq
-;
-; AVX512VL-LABEL: trunc2x4i64_8i32:
-; AVX512VL:       # %bb.0: # %entry
-; AVX512VL-NEXT:    vpmovqd %ymm0, %xmm0
-; AVX512VL-NEXT:    vpmovqd %ymm1, %xmm1
-; AVX512VL-NEXT:    vinserti128 $1, %xmm1, %ymm0, %ymm0
-; AVX512VL-NEXT:    retq
-;
-; AVX512BW-LABEL: trunc2x4i64_8i32:
-; AVX512BW:       # %bb.0: # %entry
-; AVX512BW-NEXT:    # kill: def $ymm1 killed $ymm1 def $zmm1
-; AVX512BW-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
-; AVX512BW-NEXT:    vpmovqd %zmm0, %ymm0
-; AVX512BW-NEXT:    vpmovqd %zmm1, %ymm1
-; AVX512BW-NEXT:    vinserti128 $1, %xmm1, %ymm0, %ymm0
-; AVX512BW-NEXT:    retq
-;
-; AVX512BWVL-LABEL: trunc2x4i64_8i32:
-; AVX512BWVL:       # %bb.0: # %entry
-; AVX512BWVL-NEXT:    vpmovqd %ymm0, %xmm0
-; AVX512BWVL-NEXT:    vpmovqd %ymm1, %xmm1
-; AVX512BWVL-NEXT:    vinserti128 $1, %xmm1, %ymm0, %ymm0
-; AVX512BWVL-NEXT:    retq
+; AVX512-LABEL: trunc2x4i64_8i32:
+; AVX512:       # %bb.0: # %entry
+; AVX512-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
+; AVX512-NEXT:    vinserti64x4 $1, %ymm1, %zmm0, %zmm0
+; AVX512-NEXT:    vpmovqd %zmm0, %ymm0
+; AVX512-NEXT:    retq
 entry:
   %0 = trunc <4 x i64> %a to <4 x i32>
   %1 = trunc <4 x i64> %b to <4 x i32>
@@ -1743,6 +1718,78 @@ entry:
   ret i64 %1
 }
 
+define <32 x i8> @trunc2x16i16_32i8(<16 x i16> %a, <16 x i16> %b) {
+; SSE-LABEL: trunc2x16i16_32i8:
+; SSE:       # %bb.0: # %entry
+; SSE-NEXT:    movdqa {{.*#+}} xmm4 = [255,0,255,0,255,0,255,0,255,0,255,0,255,0,255,0]
+; SSE-NEXT:    pand %xmm4, %xmm1
+; SSE-NEXT:    pand %xmm4, %xmm0
+; SSE-NEXT:    packuswb %xmm1, %xmm0
+; SSE-NEXT:    pand %xmm4, %xmm3
+; SSE-NEXT:    pand %xmm2, %xmm4
+; SSE-NEXT:    packuswb %xmm3, %xmm4
+; SSE-NEXT:    movdqa %xmm4, %xmm1
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: trunc2x16i16_32i8:
+; AVX1:       # %bb.0: # %entry
+; AVX1-NEXT:    vmovaps {{.*#+}} ymm2 = [255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255]
+; AVX1-NEXT:    vandps %ymm2, %ymm0, %ymm0
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm3
+; AVX1-NEXT:    vpackuswb %xmm3, %xmm0, %xmm0
+; AVX1-NEXT:    vandps %ymm2, %ymm1, %ymm1
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
+; AVX1-NEXT:    vpackuswb %xmm2, %xmm1, %xmm1
+; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: trunc2x16i16_32i8:
+; AVX2:       # %bb.0: # %entry
+; AVX2-NEXT:    vpbroadcastw {{.*#+}} ymm2 = [255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255]
+; AVX2-NEXT:    vpand %ymm2, %ymm0, %ymm0
+; AVX2-NEXT:    vpand %ymm2, %ymm1, %ymm1
+; AVX2-NEXT:    vpackuswb %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    vpermq {{.*#+}} ymm0 = ymm0[0,2,1,3]
+; AVX2-NEXT:    retq
+;
+; AVX512F-LABEL: trunc2x16i16_32i8:
+; AVX512F:       # %bb.0: # %entry
+; AVX512F-NEXT:    vpmovzxwd {{.*#+}} zmm0 = ymm0[0],zero,ymm0[1],zero,ymm0[2],zero,ymm0[3],zero,ymm0[4],zero,ymm0[5],zero,ymm0[6],zero,ymm0[7],zero,ymm0[8],zero,ymm0[9],zero,ymm0[10],zero,ymm0[11],zero,ymm0[12],zero,ymm0[13],zero,ymm0[14],zero,ymm0[15],zero
+; AVX512F-NEXT:    vpmovdb %zmm0, %xmm0
+; AVX512F-NEXT:    vpmovzxwd {{.*#+}} zmm1 = ymm1[0],zero,ymm1[1],zero,ymm1[2],zero,ymm1[3],zero,ymm1[4],zero,ymm1[5],zero,ymm1[6],zero,ymm1[7],zero,ymm1[8],zero,ymm1[9],zero,ymm1[10],zero,ymm1[11],zero,ymm1[12],zero,ymm1[13],zero,ymm1[14],zero,ymm1[15],zero
+; AVX512F-NEXT:    vpmovdb %zmm1, %xmm1
+; AVX512F-NEXT:    vinserti128 $1, %xmm1, %ymm0, %ymm0
+; AVX512F-NEXT:    retq
+;
+; AVX512VL-LABEL: trunc2x16i16_32i8:
+; AVX512VL:       # %bb.0: # %entry
+; AVX512VL-NEXT:    vpmovzxwd {{.*#+}} zmm0 = ymm0[0],zero,ymm0[1],zero,ymm0[2],zero,ymm0[3],zero,ymm0[4],zero,ymm0[5],zero,ymm0[6],zero,ymm0[7],zero,ymm0[8],zero,ymm0[9],zero,ymm0[10],zero,ymm0[11],zero,ymm0[12],zero,ymm0[13],zero,ymm0[14],zero,ymm0[15],zero
+; AVX512VL-NEXT:    vpmovdb %zmm0, %xmm0
+; AVX512VL-NEXT:    vpmovzxwd {{.*#+}} zmm1 = ymm1[0],zero,ymm1[1],zero,ymm1[2],zero,ymm1[3],zero,ymm1[4],zero,ymm1[5],zero,ymm1[6],zero,ymm1[7],zero,ymm1[8],zero,ymm1[9],zero,ymm1[10],zero,ymm1[11],zero,ymm1[12],zero,ymm1[13],zero,ymm1[14],zero,ymm1[15],zero
+; AVX512VL-NEXT:    vpmovdb %zmm1, %xmm1
+; AVX512VL-NEXT:    vinserti128 $1, %xmm1, %ymm0, %ymm0
+; AVX512VL-NEXT:    retq
+;
+; AVX512BW-LABEL: trunc2x16i16_32i8:
+; AVX512BW:       # %bb.0: # %entry
+; AVX512BW-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
+; AVX512BW-NEXT:    vinserti64x4 $1, %ymm1, %zmm0, %zmm0
+; AVX512BW-NEXT:    vpmovwb %zmm0, %ymm0
+; AVX512BW-NEXT:    retq
+;
+; AVX512BWVL-LABEL: trunc2x16i16_32i8:
+; AVX512BWVL:       # %bb.0: # %entry
+; AVX512BWVL-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
+; AVX512BWVL-NEXT:    vinserti64x4 $1, %ymm1, %zmm0, %zmm0
+; AVX512BWVL-NEXT:    vpmovwb %zmm0, %ymm0
+; AVX512BWVL-NEXT:    retq
+entry:
+  %0 = trunc <16 x i16> %a to <16 x i8>
+  %1 = trunc <16 x i16> %b to <16 x i8>
+  %2 = shufflevector <16 x i8> %0, <16 x i8> %1, <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31>
+  ret <32 x i8> %2
+}
+
 define <16 x i8> @trunc2x8i16_16i8(<8 x i16> %a, <8 x i16> %b) {
 ; SSE2-LABEL: trunc2x8i16_16i8:
 ; SSE2:       # %bb.0: # %entry
diff --git a/llvm/test/CodeGen/X86/x86-64-baseptr.ll b/llvm/test/CodeGen/X86/x86-64-baseptr.ll
index decf1f0e437ec..914a5a4796247 100644
--- a/llvm/test/CodeGen/X86/x86-64-baseptr.ll
+++ b/llvm/test/CodeGen/X86/x86-64-baseptr.ll
@@ -15,11 +15,15 @@ define void @base() #0 {
 ; CHECK-LABEL: base:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    pushq %rbp
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset %rbp, -16
 ; CHECK-NEXT:    movq %rsp, %rbp
+; CHECK-NEXT:    .cfi_def_cfa_register %rbp
 ; CHECK-NEXT:    pushq %rbx
 ; CHECK-NEXT:    andq $-32, %rsp
 ; CHECK-NEXT:    subq $32, %rsp
 ; CHECK-NEXT:    movq %rsp, %rbx
+; CHECK-NEXT:    .cfi_offset %rbx, -24
 ; CHECK-NEXT:    callq helper@PLT
 ; CHECK-NEXT:    movq %rsp, %rcx
 ; CHECK-NEXT:    movl %eax, %eax
@@ -33,16 +37,21 @@ define void @base() #0 {
 ; CHECK-NEXT:    leaq -8(%rbp), %rsp
 ; CHECK-NEXT:    popq %rbx
 ; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:    .cfi_def_cfa %rsp, 8
 ; CHECK-NEXT:    retq
 ;
 ; X32ABI-LABEL: base:
 ; X32ABI:       # %bb.0: # %entry
 ; X32ABI-NEXT:    pushq %rbp
+; X32ABI-NEXT:    .cfi_def_cfa_offset 16
+; X32ABI-NEXT:    .cfi_offset %rbp, -16
 ; X32ABI-NEXT:    movl %esp, %ebp
+; X32ABI-NEXT:    .cfi_def_cfa_register %rbp
 ; X32ABI-NEXT:    pushq %rbx
 ; X32ABI-NEXT:    andl $-32, %esp
 ; X32ABI-NEXT:    subl $32, %esp
 ; X32ABI-NEXT:    movl %esp, %ebx
+; X32ABI-NEXT:    .cfi_offset %rbx, -24
 ; X32ABI-NEXT:    callq helper@PLT
 ; X32ABI-NEXT:    # kill: def $eax killed $eax def $rax
 ; X32ABI-NEXT:    leal 31(,%rax,4), %eax
@@ -56,6 +65,7 @@ define void @base() #0 {
 ; X32ABI-NEXT:    leal -8(%ebp), %esp
 ; X32ABI-NEXT:    popq %rbx
 ; X32ABI-NEXT:    popq %rbp
+; X32ABI-NEXT:    .cfi_def_cfa %rsp, 8
 ; X32ABI-NEXT:    retq
 entry:
   %k = call i32 @helper()
@@ -68,11 +78,15 @@ define void @clobber_base() #0 {
 ; CHECK-LABEL: clobber_base:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    pushq %rbp
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset %rbp, -16
 ; CHECK-NEXT:    movq %rsp, %rbp
+; CHECK-NEXT:    .cfi_def_cfa_register %rbp
 ; CHECK-NEXT:    pushq %rbx
 ; CHECK-NEXT:    andq $-128, %rsp
 ; CHECK-NEXT:    subq $128, %rsp
 ; CHECK-NEXT:    movq %rsp, %rbx
+; CHECK-NEXT:    .cfi_offset %rbx, -24
 ; CHECK-NEXT:    callq helper@PLT
 ; CHECK-NEXT:    movq %rsp, %rcx
 ; CHECK-NEXT:    movl %eax, %eax
@@ -94,16 +108,21 @@ define void @clobber_base() #0 {
 ; CHECK-NEXT:    leaq -8(%rbp), %rsp
 ; CHECK-NEXT:    popq %rbx
 ; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:    .cfi_def_cfa %rsp, 8
 ; CHECK-NEXT:    retq
 ;
 ; X32ABI-LABEL: clobber_base:
 ; X32ABI:       # %bb.0: # %entry
 ; X32ABI-NEXT:    pushq %rbp
+; X32ABI-NEXT:    .cfi_def_cfa_offset 16
+; X32ABI-NEXT:    .cfi_offset %rbp, -16
 ; X32ABI-NEXT:    movl %esp, %ebp
+; X32ABI-NEXT:    .cfi_def_cfa_register %rbp
 ; X32ABI-NEXT:    pushq %rbx
 ; X32ABI-NEXT:    andl $-128, %esp
 ; X32ABI-NEXT:    subl $128, %esp
 ; X32ABI-NEXT:    movl %esp, %ebx
+; X32ABI-NEXT:    .cfi_offset %rbx, -24
 ; X32ABI-NEXT:    callq helper@PLT
 ; X32ABI-NEXT:    # kill: def $eax killed $eax def $rax
 ; X32ABI-NEXT:    leal 31(,%rax,4), %eax
@@ -125,6 +144,7 @@ define void @clobber_base() #0 {
 ; X32ABI-NEXT:    leal -8(%ebp), %esp
 ; X32ABI-NEXT:    popq %rbx
 ; X32ABI-NEXT:    popq %rbp
+; X32ABI-NEXT:    .cfi_def_cfa %rsp, 8
 ; X32ABI-NEXT:    retq
 entry:
   %k = call i32 @helper()
@@ -137,6 +157,147 @@ entry:
   ret void
 }
 
-attributes #0 = { nounwind "frame-pointer"="all"}
+define x86_regcallcc void @clobber_baseptr_argptr(i32 %param1, i32 %param2, i32 %param3, i32 %param4, i32 %param5, i32 %param6, i32 %param7, i32 %param8, i32 %param9, i32 %param10, i32 %param11, i32 %param12) #0 {
+; CHECK-LABEL: clobber_baseptr_argptr:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    pushq %rbp
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset %rbp, -16
+; CHECK-NEXT:    movq %rsp, %rbp
+; CHECK-NEXT:    .cfi_def_cfa_register %rbp
+; CHECK-NEXT:    pushq %rbx
+; CHECK-NEXT:    andq $-128, %rsp
+; CHECK-NEXT:    subq $256, %rsp # imm = 0x100
+; CHECK-NEXT:    movaps %xmm15, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-NEXT:    movaps %xmm14, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-NEXT:    movaps %xmm13, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-NEXT:    movaps %xmm12, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-NEXT:    movaps %xmm11, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-NEXT:    movaps %xmm10, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-NEXT:    movaps %xmm9, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-NEXT:    movaps %xmm8, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-NEXT:    movq %rsp, %rbx
+; CHECK-NEXT:    .cfi_offset %rbx, -24
+; CHECK-NEXT:    .cfi_offset %xmm8, -160
+; CHECK-NEXT:    .cfi_offset %xmm9, -144
+; CHECK-NEXT:    .cfi_offset %xmm10, -128
+; CHECK-NEXT:    .cfi_offset %xmm11, -112
+; CHECK-NEXT:    .cfi_offset %xmm12, -96
+; CHECK-NEXT:    .cfi_offset %xmm13, -80
+; CHECK-NEXT:    .cfi_offset %xmm14, -64
+; CHECK-NEXT:    .cfi_offset %xmm15, -48
+; CHECK-NEXT:    movl 16(%rbp), %r14d
+; CHECK-NEXT:    callq helper@PLT
+; CHECK-NEXT:    movq %rsp, %rcx
+; CHECK-NEXT:    movl %eax, %eax
+; CHECK-NEXT:    leaq 31(,%rax,4), %rax
+; CHECK-NEXT:    andq $-32, %rax
+; CHECK-NEXT:    movq %rcx, %rdx
+; CHECK-NEXT:    subq %rax, %rdx
+; CHECK-NEXT:    movq %rdx, %rsp
+; CHECK-NEXT:    negq %rax
+; CHECK-NEXT:    movl $405, %ebx # imm = 0x195
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    movl $8, %edx
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    movl %edx, (%rbx)
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    movl %r14d, (%rcx,%rax)
+; CHECK-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm8 # 16-byte Reload
+; CHECK-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm9 # 16-byte Reload
+; CHECK-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm10 # 16-byte Reload
+; CHECK-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm11 # 16-byte Reload
+; CHECK-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm12 # 16-byte Reload
+; CHECK-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm13 # 16-byte Reload
+; CHECK-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm14 # 16-byte Reload
+; CHECK-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm15 # 16-byte Reload
+; CHECK-NEXT:    leaq -8(%rbp), %rsp
+; CHECK-NEXT:    popq %rbx
+; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:    .cfi_def_cfa %rsp, 8
+; CHECK-NEXT:    retq
+;
+; X32ABI-LABEL: clobber_baseptr_argptr:
+; X32ABI:       # %bb.0: # %entry
+; X32ABI-NEXT:    pushq %rbp
+; X32ABI-NEXT:    .cfi_def_cfa_offset 16
+; X32ABI-NEXT:    .cfi_offset %rbp, -16
+; X32ABI-NEXT:    movl %esp, %ebp
+; X32ABI-NEXT:    .cfi_def_cfa_register %rbp
+; X32ABI-NEXT:    pushq %rbx
+; X32ABI-NEXT:    andl $-128, %esp
+; X32ABI-NEXT:    subl $256, %esp # imm = 0x100
+; X32ABI-NEXT:    movaps %xmm15, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; X32ABI-NEXT:    movaps %xmm14, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; X32ABI-NEXT:    movaps %xmm13, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; X32ABI-NEXT:    movaps %xmm12, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; X32ABI-NEXT:    movaps %xmm11, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; X32ABI-NEXT:    movaps %xmm10, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; X32ABI-NEXT:    movaps %xmm9, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; X32ABI-NEXT:    movaps %xmm8, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; X32ABI-NEXT:    movl %esp, %ebx
+; X32ABI-NEXT:    .cfi_offset %rbx, -24
+; X32ABI-NEXT:    .cfi_offset %xmm8, -160
+; X32ABI-NEXT:    .cfi_offset %xmm9, -144
+; X32ABI-NEXT:    .cfi_offset %xmm10, -128
+; X32ABI-NEXT:    .cfi_offset %xmm11, -112
+; X32ABI-NEXT:    .cfi_offset %xmm12, -96
+; X32ABI-NEXT:    .cfi_offset %xmm13, -80
+; X32ABI-NEXT:    .cfi_offset %xmm14, -64
+; X32ABI-NEXT:    .cfi_offset %xmm15, -48
+; X32ABI-NEXT:    movl 16(%ebp), %r14d
+; X32ABI-NEXT:    callq helper@PLT
+; X32ABI-NEXT:    # kill: def $eax killed $eax def $rax
+; X32ABI-NEXT:    leal 31(,%rax,4), %eax
+; X32ABI-NEXT:    andl $-32, %eax
+; X32ABI-NEXT:    movl %esp, %ecx
+; X32ABI-NEXT:    movl %ecx, %edx
+; X32ABI-NEXT:    subl %eax, %edx
+; X32ABI-NEXT:    negl %eax
+; X32ABI-NEXT:    movl %edx, %esp
+; X32ABI-NEXT:    movl $405, %ebx # imm = 0x195
+; X32ABI-NEXT:    #APP
+; X32ABI-NEXT:    nop
+; X32ABI-NEXT:    #NO_APP
+; X32ABI-NEXT:    #APP
+; X32ABI-NEXT:    nop
+; X32ABI-NEXT:    #NO_APP
+; X32ABI-NEXT:    movl $8, %edx
+; X32ABI-NEXT:    #APP
+; X32ABI-NEXT:    movl %edx, (%ebx)
+; X32ABI-NEXT:    #NO_APP
+; X32ABI-NEXT:    movl %r14d, (%ecx,%eax)
+; X32ABI-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm8 # 16-byte Reload
+; X32ABI-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm9 # 16-byte Reload
+; X32ABI-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm10 # 16-byte Reload
+; X32ABI-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm11 # 16-byte Reload
+; X32ABI-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm12 # 16-byte Reload
+; X32ABI-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm13 # 16-byte Reload
+; X32ABI-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm14 # 16-byte Reload
+; X32ABI-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm15 # 16-byte Reload
+; X32ABI-NEXT:    leal -8(%ebp), %esp
+; X32ABI-NEXT:    popq %rbx
+; X32ABI-NEXT:    popq %rbp
+; X32ABI-NEXT:    .cfi_def_cfa %rsp, 8
+; X32ABI-NEXT:    retq
+entry:
+  %k = call i32 @helper()
+  %a = alloca i32, align 128
+  %b = alloca i32, i32 %k, align 4
+  ; clobber base pointer register
+  tail call void asm sideeffect "nop", "{bx}"(i32 405)
+  ; clobber argument pointer register
+  tail call void asm sideeffect "nop", "~{bx},~{r10},~{r11}"()
+  call void asm sideeffect "movl $0, $1", "r,*m"(i32 8, ptr elementtype(i32) %a)
+  store i32 %param12, ptr %b, align 4
+  ret void
+}
+
+attributes #0 = { "frame-pointer"="all"}
 !llvm.module.flags = !{!0}
 !0 = !{i32 2, !"override-stack-alignment", i32 32}
diff --git a/llvm/test/CodeGen/X86/zext-lshr.ll b/llvm/test/CodeGen/X86/zext-lshr.ll
new file mode 100644
index 0000000000000..fbfd7b7e1bfe0
--- /dev/null
+++ b/llvm/test/CodeGen/X86/zext-lshr.ll
@@ -0,0 +1,137 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=i686-unknown-unknown | FileCheck %s --check-prefixes=CHECK,X86
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown | FileCheck %s --check-prefixes=CHECK,X64
+
+define i32 @i32_zext_shift_i16_zext_i1(i1 %a0) nounwind {
+; CHECK-LABEL: i32_zext_shift_i16_zext_i1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    ret{{[l|q]}}
+  %t0 = zext i1 %a0 to i16
+  %t1 = lshr i16 %t0, 5
+  %t2 = zext i16 %t1 to i32
+  ret i32 %t2
+}
+
+define i32 @i32_zext_shift_i16_zext_i8(i8 %a0) nounwind {
+; X86-LABEL: i32_zext_shift_i16_zext_i8:
+; X86:       # %bb.0:
+; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    shrl $5, %eax
+; X86-NEXT:    retl
+;
+; X64-LABEL: i32_zext_shift_i16_zext_i8:
+; X64:       # %bb.0:
+; X64-NEXT:    movzbl %dil, %eax
+; X64-NEXT:    shrl $5, %eax
+; X64-NEXT:    retq
+  %t0 = zext i8 %a0 to i16
+  %t1 = lshr i16 %t0, 5
+  %t2 = zext i16 %t1 to i32
+  ret i32 %t2
+}
+
+define i64 @i64_zext_shift_i16_zext_i8(i8 %a0) nounwind {
+; X86-LABEL: i64_zext_shift_i16_zext_i8:
+; X86:       # %bb.0:
+; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    shrl $5, %eax
+; X86-NEXT:    xorl %edx, %edx
+; X86-NEXT:    retl
+;
+; X64-LABEL: i64_zext_shift_i16_zext_i8:
+; X64:       # %bb.0:
+; X64-NEXT:    movzbl %dil, %eax
+; X64-NEXT:    shrq $5, %rax
+; X64-NEXT:    retq
+  %t0 = zext i8 %a0 to i16
+  %t1 = lshr i16 %t0, 5
+  %t2 = zext i16 %t1 to i64
+  ret i64 %t2
+}
+
+define i64 @i64_zext_shift_i32_zext_i8(i8 %a0) nounwind {
+; X86-LABEL: i64_zext_shift_i32_zext_i8:
+; X86:       # %bb.0:
+; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    shrl $3, %eax
+; X86-NEXT:    xorl %edx, %edx
+; X86-NEXT:    retl
+;
+; X64-LABEL: i64_zext_shift_i32_zext_i8:
+; X64:       # %bb.0:
+; X64-NEXT:    movzbl %dil, %eax
+; X64-NEXT:    shrl $3, %eax
+; X64-NEXT:    retq
+  %t0 = zext i8 %a0 to i32
+  %t1 = lshr i32 %t0, 3
+  %t2 = zext i32 %t1 to i64
+  ret i64 %t2
+}
+
+define i64 @i64_zext_shift_i32_zext_i16(i16 %a0) nounwind {
+; X86-LABEL: i64_zext_shift_i32_zext_i16:
+; X86:       # %bb.0:
+; X86-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    shrl $5, %eax
+; X86-NEXT:    xorl %edx, %edx
+; X86-NEXT:    retl
+;
+; X64-LABEL: i64_zext_shift_i32_zext_i16:
+; X64:       # %bb.0:
+; X64-NEXT:    movzwl %di, %eax
+; X64-NEXT:    shrl $5, %eax
+; X64-NEXT:    retq
+  %t0 = zext i16 %a0 to i32
+  %t1 = lshr i32 %t0, 5
+  %t2 = zext i32 %t1 to i64
+  ret i64 %t2
+}
+
+define i128 @i128_zext_shift_i64_zext_i8(i8 %a0) nounwind {
+; X86-LABEL: i128_zext_shift_i64_zext_i8:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    shrl $4, %ecx
+; X86-NEXT:    movl %ecx, (%eax)
+; X86-NEXT:    movl $0, 12(%eax)
+; X86-NEXT:    movl $0, 8(%eax)
+; X86-NEXT:    movl $0, 4(%eax)
+; X86-NEXT:    retl $4
+;
+; X64-LABEL: i128_zext_shift_i64_zext_i8:
+; X64:       # %bb.0:
+; X64-NEXT:    movzbl %dil, %eax
+; X64-NEXT:    shrq $4, %rax
+; X64-NEXT:    xorl %edx, %edx
+; X64-NEXT:    retq
+  %t0 = zext i8 %a0 to i64
+  %t1 = lshr i64 %t0, 4
+  %t2 = zext i64 %t1 to i128
+  ret i128 %t2
+}
+
+define i128 @i128_zext_shift_i64_zext_i16(i16 %a0) nounwind {
+; X86-LABEL: i128_zext_shift_i64_zext_i16:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movzwl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    shrl $7, %ecx
+; X86-NEXT:    movl %ecx, (%eax)
+; X86-NEXT:    movl $0, 12(%eax)
+; X86-NEXT:    movl $0, 8(%eax)
+; X86-NEXT:    movl $0, 4(%eax)
+; X86-NEXT:    retl $4
+;
+; X64-LABEL: i128_zext_shift_i64_zext_i16:
+; X64:       # %bb.0:
+; X64-NEXT:    movzwl %di, %eax
+; X64-NEXT:    shrq $7, %rax
+; X64-NEXT:    xorl %edx, %edx
+; X64-NEXT:    retq
+  %t0 = zext i16 %a0 to i64
+  %t1 = lshr i64 %t0,7
+  %t2 = zext i64 %t1 to i128
+  ret i128 %t2
+}
diff --git a/llvm/test/CodeGen/X86/zext-shl.ll b/llvm/test/CodeGen/X86/zext-shl.ll
index 1b9c813bc1e60..8c27e0da6acf7 100644
--- a/llvm/test/CodeGen/X86/zext-shl.ll
+++ b/llvm/test/CodeGen/X86/zext-shl.ll
@@ -1,26 +1,146 @@
 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc < %s -mtriple=i686-unknown-unknown | FileCheck %s
+; RUN: llc < %s -mtriple=i686-unknown-unknown | FileCheck %s --check-prefixes=X86
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown | FileCheck %s --check-prefixes=X64
 
-define i32 @t1(i8 zeroext %x) nounwind {
-; CHECK-LABEL: t1:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
-; CHECK-NEXT:    shll $5, %eax
-; CHECK-NEXT:    retl
-  %t0 = zext i8 %x to i16
+define i32 @i32_zext_shift_i16_zext_i1(i1 %a0) nounwind {
+; X86-LABEL: i32_zext_shift_i16_zext_i1:
+; X86:       # %bb.0:
+; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    andl $1, %eax
+; X86-NEXT:    shll $5, %eax
+; X86-NEXT:    retl
+;
+; X64-LABEL: i32_zext_shift_i16_zext_i1:
+; X64:       # %bb.0:
+; X64-NEXT:    movl %edi, %eax
+; X64-NEXT:    andl $1, %eax
+; X64-NEXT:    shll $5, %eax
+; X64-NEXT:    retq
+  %t0 = zext i1 %a0 to i16
   %t1 = shl i16 %t0, 5
   %t2 = zext i16 %t1 to i32
   ret i32 %t2
 }
 
-define i32 @t2(i8 zeroext %x) nounwind {
-; CHECK-LABEL: t2:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
-; CHECK-NEXT:    shrl $3, %eax
-; CHECK-NEXT:    retl
-  %t0 = zext i8 %x to i16
-  %t1 = lshr i16 %t0, 3
+define i32 @i32_zext_shift_i16_zext_i8(i8 %a0) nounwind {
+; X86-LABEL: i32_zext_shift_i16_zext_i8:
+; X86:       # %bb.0:
+; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    shll $5, %eax
+; X86-NEXT:    retl
+;
+; X64-LABEL: i32_zext_shift_i16_zext_i8:
+; X64:       # %bb.0:
+; X64-NEXT:    movzbl %dil, %eax
+; X64-NEXT:    shll $5, %eax
+; X64-NEXT:    retq
+  %t0 = zext i8 %a0 to i16
+  %t1 = shl i16 %t0, 5
   %t2 = zext i16 %t1 to i32
   ret i32 %t2
 }
+
+define i64 @i64_zext_shift_i16_zext_i8(i8 %a0) nounwind {
+; X86-LABEL: i64_zext_shift_i16_zext_i8:
+; X86:       # %bb.0:
+; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    shll $5, %eax
+; X86-NEXT:    xorl %edx, %edx
+; X86-NEXT:    retl
+;
+; X64-LABEL: i64_zext_shift_i16_zext_i8:
+; X64:       # %bb.0:
+; X64-NEXT:    movzbl %dil, %eax
+; X64-NEXT:    shlq $5, %rax
+; X64-NEXT:    retq
+  %t0 = zext i8 %a0 to i16
+  %t1 = shl i16 %t0, 5
+  %t2 = zext i16 %t1 to i64
+  ret i64 %t2
+}
+
+define i64 @i64_zext_shift_i32_zext_i8(i8 %a0) nounwind {
+; X86-LABEL: i64_zext_shift_i32_zext_i8:
+; X86:       # %bb.0:
+; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    shll $3, %eax
+; X86-NEXT:    xorl %edx, %edx
+; X86-NEXT:    retl
+;
+; X64-LABEL: i64_zext_shift_i32_zext_i8:
+; X64:       # %bb.0:
+; X64-NEXT:    movzbl %dil, %eax
+; X64-NEXT:    shll $3, %eax
+; X64-NEXT:    retq
+  %t0 = zext i8 %a0 to i32
+  %t1 = shl i32 %t0, 3
+  %t2 = zext i32 %t1 to i64
+  ret i64 %t2
+}
+
+define i64 @i64_zext_shift_i32_zext_i16(i16 %a0) nounwind {
+; X86-LABEL: i64_zext_shift_i32_zext_i16:
+; X86:       # %bb.0:
+; X86-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    shll $5, %eax
+; X86-NEXT:    xorl %edx, %edx
+; X86-NEXT:    retl
+;
+; X64-LABEL: i64_zext_shift_i32_zext_i16:
+; X64:       # %bb.0:
+; X64-NEXT:    movzwl %di, %eax
+; X64-NEXT:    shll $5, %eax
+; X64-NEXT:    retq
+  %t0 = zext i16 %a0 to i32
+  %t1 = shl i32 %t0, 5
+  %t2 = zext i32 %t1 to i64
+  ret i64 %t2
+}
+
+define i128 @i128_zext_shift_i64_zext_i8(i8 %a0) nounwind {
+; X86-LABEL: i128_zext_shift_i64_zext_i8:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    shll $4, %ecx
+; X86-NEXT:    movl %ecx, (%eax)
+; X86-NEXT:    movl $0, 12(%eax)
+; X86-NEXT:    movl $0, 8(%eax)
+; X86-NEXT:    movl $0, 4(%eax)
+; X86-NEXT:    retl $4
+;
+; X64-LABEL: i128_zext_shift_i64_zext_i8:
+; X64:       # %bb.0:
+; X64-NEXT:    movzbl %dil, %eax
+; X64-NEXT:    shlq $4, %rax
+; X64-NEXT:    xorl %edx, %edx
+; X64-NEXT:    retq
+  %t0 = zext i8 %a0 to i64
+  %t1 = shl i64 %t0, 4
+  %t2 = zext i64 %t1 to i128
+  ret i128 %t2
+}
+
+define i128 @i128_zext_shift_i64_zext_i16(i16 %a0) nounwind {
+; X86-LABEL: i128_zext_shift_i64_zext_i16:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movzwl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    shll $7, %ecx
+; X86-NEXT:    movl %ecx, (%eax)
+; X86-NEXT:    movl $0, 12(%eax)
+; X86-NEXT:    movl $0, 8(%eax)
+; X86-NEXT:    movl $0, 4(%eax)
+; X86-NEXT:    retl $4
+;
+; X64-LABEL: i128_zext_shift_i64_zext_i16:
+; X64:       # %bb.0:
+; X64-NEXT:    movzwl %di, %eax
+; X64-NEXT:    shlq $7, %rax
+; X64-NEXT:    xorl %edx, %edx
+; X64-NEXT:    retq
+  %t0 = zext i16 %a0 to i64
+  %t1 = shl i64 %t0,7
+  %t2 = zext i64 %t1 to i128
+  ret i128 %t2
+}
diff --git a/llvm/test/DebugInfo/COFF/language.ll b/llvm/test/DebugInfo/COFF/language.ll
index 3db67e3ebbbe1..8870acca6abdf 100644
--- a/llvm/test/DebugInfo/COFF/language.ll
+++ b/llvm/test/DebugInfo/COFF/language.ll
@@ -39,6 +39,12 @@
 ;
 ; RUN: sed -e 's/<LANG1>/DW_LANG_Rust/;s/<LANG2>/Rust/' %s > %t
 ; RUN: llc -filetype=obj -o - %t | llvm-readobj --codeview - | FileCheck %t
+;
+; RUN: sed -e 's/<LANG1>/DW_LANG_ObjC/;s/<LANG2>/ObjC/' %s > %t
+; RUN: llc -filetype=obj -o - %t | llvm-readobj --codeview - | FileCheck %t
+;
+; RUN: sed -e 's/<LANG1>/DW_LANG_ObjC_plus_plus/;s/<LANG2>/ObjCpp/' %s > %t
+; RUN: llc -filetype=obj -o - %t | llvm-readobj --codeview - | FileCheck %t
 
 ; CHECK:      CodeViewDebugInfo [
 ; CHECK:        Subsection [
diff --git a/llvm/test/DebugInfo/COFF/objc.ll b/llvm/test/DebugInfo/COFF/objc.ll
new file mode 100644
index 0000000000000..7e4ddf88ed787
--- /dev/null
+++ b/llvm/test/DebugInfo/COFF/objc.ll
@@ -0,0 +1,35 @@
+; RUN: llc < %s | FileCheck %s --check-prefix=ASM
+; RUN: llc -filetype=obj < %s | llvm-readobj --codeview - | FileCheck %s --check-prefix=OBJ
+
+; ASM:      .short  4412                    # Record kind: S_COMPILE3
+; ASM-NEXT: .long   17                      # Flags and language
+
+; OBJ:       Kind: S_COMPILE3 (0x113C)
+; OBJ-NEXT:  Language: ObjC (0x11)
+
+; ModuleID = 'objc.m'
+source_filename = "objc.m"
+target datalayout = "e-m:w-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-pc-windows-msvc"
+
+; Function Attrs: uwtable
+define void @f() unnamed_addr #0 !dbg !5 {
+entry:
+  ret void, !dbg !9
+}
+
+attributes #0 = { uwtable "target-cpu"="x86-64" }
+
+!llvm.module.flags = !{!0, !1, !2}
+!llvm.dbg.cu = !{!3}
+
+!0 = !{i32 8, !"PIC Level", i32 2}
+!1 = !{i32 2, !"CodeView", i32 1}
+!2 = !{i32 2, !"Debug Info Version", i32 3}
+!3 = distinct !DICompileUnit(language: DW_LANG_ObjC, file: !4, producer: "clang version 17.0.0 (https://github.com/llvm/llvm-project a8e9beca6bee1f248ef4be7892802c4d091b7fcb)", isOptimized: false, runtimeVersion: 1, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!4 = !DIFile(filename: "objc.m", directory: "src", checksumkind: CSK_MD5, checksum: "e6ab1d5b7f82464c963a8522037dfa72")
+!5 = distinct !DISubprogram(name: "f", scope: !4, file: !4, line: 1, type: !6, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !3, retainedNodes: !8)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!8 = !{}
+!9 = !DILocation(line: 1, scope: !5)
diff --git a/llvm/test/DebugInfo/COFF/objcpp.ll b/llvm/test/DebugInfo/COFF/objcpp.ll
new file mode 100644
index 0000000000000..49786afaf05f6
--- /dev/null
+++ b/llvm/test/DebugInfo/COFF/objcpp.ll
@@ -0,0 +1,35 @@
+; RUN: llc < %s | FileCheck %s --check-prefix=ASM
+; RUN: llc -filetype=obj < %s | llvm-readobj --codeview - | FileCheck %s --check-prefix=OBJ
+
+; ASM:      .short  4412                    # Record kind: S_COMPILE3
+; ASM-NEXT: .long   18                      # Flags and language
+
+; OBJ:       Kind: S_COMPILE3 (0x113C)
+; OBJ-NEXT:  Language: ObjCpp (0x12)
+
+; ModuleID = 'objcpp.mm'
+source_filename = "objcpp.mm"
+target datalayout = "e-m:w-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-pc-windows-msvc"
+
+; Function Attrs: uwtable
+define void @"?f@@YAXXZ"() #0 !dbg !5 {
+entry:
+  ret void, !dbg !9
+}
+
+attributes #0 = { uwtable "target-cpu"="x86-64" }
+
+!llvm.module.flags = !{!0, !1, !2}
+!llvm.dbg.cu = !{!3}
+
+!0 = !{i32 8, !"PIC Level", i32 2}
+!1 = !{i32 2, !"CodeView", i32 1}
+!2 = !{i32 2, !"Debug Info Version", i32 3}
+!3 = distinct !DICompileUnit(language: DW_LANG_ObjC_plus_plus, file: !4, producer: "clang version 17.0.0 (https://github.com/llvm/llvm-project a8e9beca6bee1f248ef4be7892802c4d091b7fcb)", isOptimized: false, runtimeVersion: 1, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!4 = !DIFile(filename: "objcpp.mm", directory: "src", checksumkind: CSK_MD5, checksum: "e6ab1d5b7f82464c963a8522037dfa72")
+!5 = distinct !DISubprogram(name: "f", linkageName: "?f@@YAXXZ", scope: !4, file: !4, line: 1, type: !6, scopeLine: 1, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !3, retainedNodes: !8)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!8 = !{}
+!9 = !DILocation(line: 1, scope: !5)
diff --git a/llvm/test/DebugInfo/Generic/assignment-tracking/salvage-value.ll b/llvm/test/DebugInfo/Generic/assignment-tracking/salvage-value.ll
index 29d5770211bd0..ba591b1aa5425 100644
--- a/llvm/test/DebugInfo/Generic/assignment-tracking/salvage-value.ll
+++ b/llvm/test/DebugInfo/Generic/assignment-tracking/salvage-value.ll
@@ -14,9 +14,8 @@ entry:
 
  %add1 = add nsw i32 %x, %y, !dbg !29
  call void @llvm.dbg.assign(metadata i32 %add1, metadata !32, metadata !DIExpression(), metadata !31, metadata ptr %p, metadata !DIExpression()), !dbg !16
-;; %add1 is not salvaged as it requires two values and DIArgList is
-;; not (yet) supported for dbg.assigns.
-; CHECK-NEXT: call void @llvm.dbg.assign(metadata i32 poison,{{.+}}, metadata !DIExpression(),{{.+}}, metadata ptr %p, metadata !DIExpression())
+;; %add1 is salvaged using a variadic expression.
+; CHECK-NEXT: call void @llvm.dbg.assign(metadata !DIArgList(i32 %x, i32 %y), metadata ![[#]], metadata !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus, DW_OP_stack_value), metadata ![[#]], metadata ptr %p, metadata !DIExpression())
 
   %arrayidx0 = getelementptr inbounds i32, ptr %p, i32 0
   call void @llvm.dbg.assign(metadata i32 %x, metadata !14, metadata !DIExpression(), metadata !17, metadata ptr %arrayidx0, metadata !DIExpression()), !dbg !16
diff --git a/llvm/test/DebugInfo/MIR/X86/merge-inline-loc1.mir b/llvm/test/DebugInfo/MIR/X86/merge-inline-loc1.mir
new file mode 100644
index 0000000000000..1504789cd5854
--- /dev/null
+++ b/llvm/test/DebugInfo/MIR/X86/merge-inline-loc1.mir
@@ -0,0 +1,222 @@
+# RUN: llc -mtriple=x86_64-pc-linux %s -run-pass=branch-folder -o - | FileCheck %s
+
+--- |
+  ; ModuleID = 'case1.c'
+  source_filename = "case1.c"
+  target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+  target triple = "x86_64-unknown-linux-gnu"
+
+  @q1 = dso_local local_unnamed_addr global i32 1, align 4
+  @q2 = dso_local local_unnamed_addr global i32 4, align 4
+  @q3 = dso_local local_unnamed_addr global i32 2, align 4
+  @g1 = dso_local local_unnamed_addr global i32 0, align 4
+  @g2 = dso_local local_unnamed_addr global i32 0, align 4
+  @g3 = dso_local local_unnamed_addr global i32 0, align 4
+
+  ; Function Attrs: nounwind uwtable
+  define dso_local i32 @multiple_inl_one_loc() local_unnamed_addr #0 !dbg !9 {
+  entry:
+    %0 = load i32, ptr @q1, align 4, !dbg !12, !tbaa !13
+    %cmp.i = icmp sgt i32 %0, 3, !dbg !17
+    br i1 %cmp.i, label %if.then.i, label %inl1.exit, !dbg !20
+
+  if.then.i:                                        ; preds = %entry
+    tail call void @abort() #2, !dbg !21
+    unreachable, !dbg !21
+
+  inl1.exit:                                        ; preds = %entry
+    %mul.i = mul nsw i32 %0, 152, !dbg !22
+    %add.i = add nsw i32 %mul.i, 100, !dbg !23
+    store i32 %add.i, ptr @g1, align 4, !dbg !24, !tbaa !13
+    %1 = load i32, ptr @q2, align 4, !dbg !25, !tbaa !13
+    %cmp.i3 = icmp sgt i32 %1, 3, !dbg !26
+    br i1 %cmp.i3, label %if.then.i4, label %inl1.exit7, !dbg !28
+
+  if.then.i4:                                       ; preds = %inl1.exit
+    tail call void @abort() #2, !dbg !29
+    unreachable, !dbg !29
+
+  inl1.exit7:                                       ; preds = %inl1.exit
+    %mul.i5 = mul nsw i32 %1, 152, !dbg !30
+    %add.i6 = add nsw i32 %mul.i5, 200, !dbg !31
+    store i32 %add.i6, ptr @g2, align 4, !dbg !32, !tbaa !13
+    %2 = load i32, ptr @q3, align 4, !dbg !33, !tbaa !13
+    %cmp.i8 = icmp sgt i32 %2, 3, !dbg !34
+    br i1 %cmp.i8, label %if.then.i9, label %inl1.exit12, !dbg !36
+
+  if.then.i9:                                       ; preds = %inl1.exit7
+    tail call void @abort() #2, !dbg !37
+    unreachable, !dbg !37
+
+  inl1.exit12:                                      ; preds = %inl1.exit7
+    %mul.i10 = mul nsw i32 %2, 152, !dbg !38
+    %add.i11 = add nsw i32 %mul.i10, 300, !dbg !39
+    store i32 %add.i11, ptr @q3, align 4, !dbg !40, !tbaa !13
+    ret i32 0, !dbg !41
+  }
+
+  ; Function Attrs: noreturn nounwind
+  declare !dbg !42 void @abort() local_unnamed_addr #1
+
+  attributes #0 = { nounwind uwtable "frame-pointer"="none" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
+  attributes #1 = { noreturn nounwind "frame-pointer"="none" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
+  attributes #2 = { noreturn nounwind }
+
+  !llvm.dbg.cu = !{!0}
+  !llvm.module.flags = !{!2, !3, !4, !5, !6, !7}
+  !llvm.ident = !{!8}
+
+  !0 = distinct !DICompileUnit(language: DW_LANG_C11, file: !1, producer: "clang version 16.0.0.prerel", isOptimized: true, runtimeVersion: 0, emissionKind: LineTablesOnly, splitDebugInlining: false, nameTableKind: None)
+  !1 = !DIFile(filename: "case1.c", directory: "/", checksumkind: CSK_MD5, checksum: "36d7da9616644fc8b011f5c49108ab31")
+  !2 = !{i32 7, !"Dwarf Version", i32 5}
+  !3 = !{i32 2, !"Debug Info Version", i32 3}
+  !4 = !{i32 1, !"wchar_size", i32 4}
+  !5 = !{i32 8, !"PIC Level", i32 2}
+  !6 = !{i32 7, !"PIE Level", i32 2}
+  !7 = !{i32 7, !"uwtable", i32 2}
+  !8 = !{!"clang version 16.0.0.prerel"}
+  !9 = distinct !DISubprogram(name: "multiple_inl_one_loc", scope: !1, file: !1, line: 13, type: !10, scopeLine: 14, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !11)
+  !10 = !DISubroutineType(types: !11)
+  !11 = !{}
+  !12 = !DILocation(line: 15, column: 13, scope: !9)
+  !13 = !{!14, !14, i64 0}
+  !14 = !{!"int", !15, i64 0}
+  !15 = !{!"omnipotent char", !16, i64 0}
+  !16 = !{!"Simple C/C++ TBAA"}
+  !17 = !DILocation(line: 5, column: 9, scope: !18, inlinedAt: !19)
+  !18 = distinct !DISubprogram(name: "inl1", scope: !1, file: !1, line: 3, type: !10, scopeLine: 4, flags: DIFlagPrototyped | DIFlagAllCallsDescribed, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !11)
+  !19 = distinct !DILocation(line: 15, column: 8, scope: !9)
+  !20 = !DILocation(line: 5, column: 7, scope: !18, inlinedAt: !19)
+  !21 = !DILocation(line: 6, column: 5, scope: !18, inlinedAt: !19)
+  !22 = !DILocation(line: 7, column: 12, scope: !18, inlinedAt: !19)
+  !23 = !DILocation(line: 7, column: 18, scope: !18, inlinedAt: !19)
+  !24 = !DILocation(line: 15, column: 6, scope: !9)
+  !25 = !DILocation(line: 16, column: 13, scope: !9)
+  !26 = !DILocation(line: 5, column: 9, scope: !18, inlinedAt: !27)
+  !27 = distinct !DILocation(line: 16, column: 8, scope: !9)
+  !28 = !DILocation(line: 5, column: 7, scope: !18, inlinedAt: !27)
+  !29 = !DILocation(line: 6, column: 5, scope: !18, inlinedAt: !27)
+  !30 = !DILocation(line: 7, column: 12, scope: !18, inlinedAt: !27)
+  !31 = !DILocation(line: 7, column: 18, scope: !18, inlinedAt: !27)
+  !32 = !DILocation(line: 16, column: 6, scope: !9)
+  !33 = !DILocation(line: 17, column: 13, scope: !9)
+  !34 = !DILocation(line: 5, column: 9, scope: !18, inlinedAt: !35)
+  !35 = distinct !DILocation(line: 17, column: 8, scope: !9)
+  !36 = !DILocation(line: 5, column: 7, scope: !18, inlinedAt: !35)
+  !37 = !DILocation(line: 6, column: 5, scope: !18, inlinedAt: !35)
+  !38 = !DILocation(line: 7, column: 12, scope: !18, inlinedAt: !35)
+  !39 = !DILocation(line: 7, column: 18, scope: !18, inlinedAt: !35)
+  !40 = !DILocation(line: 17, column: 6, scope: !9)
+  !41 = !DILocation(line: 18, column: 3, scope: !9)
+  !42 = !DISubprogram(name: "abort", scope: !43, file: !43, line: 514, type: !10, flags: DIFlagPrototyped, spFlags: DISPFlagOptimized, retainedNodes: !11)
+  !43 = !DIFile(filename: "/usr/include/stdlib.h", directory: "", checksumkind: CSK_MD5, checksum: "f7a1412d75d9e3df251dfc21b02d59ef")
+
+...
+---
+name:            multiple_inl_one_loc
+alignment:       16
+tracksRegLiveness: true
+tracksDebugUserValues: true
+frameInfo:
+  stackSize:       8
+  offsetAdjustment: -8
+  maxAlignment:    1
+  adjustsStack:    true
+  hasCalls:        true
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0.entry:
+    successors: %bb.1(0x00000800), %bb.2(0x7ffff800)
+
+    frame-setup PUSH64r undef $rax, implicit-def $rsp, implicit $rsp
+    frame-setup CFI_INSTRUCTION def_cfa_offset 16
+    renamable $eax = MOV32rm $rip, 1, $noreg, @q1, $noreg, debug-location !12 :: (dereferenceable load (s32) from @q1, !tbaa !13)
+    CMP32ri8 renamable $eax, 4, implicit-def $eflags, debug-location !17
+    JCC_1 %bb.2, 12, implicit killed $eflags, debug-location !20
+    JMP_1 %bb.1, debug-location !20
+
+  bb.1.if.then.i:
+    successors:
+
+    CALL64pcrel32 target-flags(x86-plt) @abort, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, debug-location !21
+
+  bb.2.inl1.exit:
+    successors: %bb.3(0x00000800), %bb.4(0x7ffff800)
+    liveins: $eax
+
+    renamable $eax = nsw IMUL32rri killed renamable $eax, 152, implicit-def dead $eflags, debug-location !22
+    renamable $eax = nsw ADD32ri8 killed renamable $eax, 100, implicit-def dead $eflags, debug-location !23
+    MOV32mr $rip, 1, $noreg, @g1, $noreg, killed renamable $eax, debug-location !24 :: (store (s32) into @g1, !tbaa !13)
+    renamable $eax = MOV32rm $rip, 1, $noreg, @q2, $noreg, debug-location !25 :: (dereferenceable load (s32) from @q2, !tbaa !13)
+    CMP32ri8 renamable $eax, 4, implicit-def $eflags, debug-location !26
+    JCC_1 %bb.4, 12, implicit killed $eflags, debug-location !28
+    JMP_1 %bb.3, debug-location !28
+
+  bb.3.if.then.i4:
+    successors:
+
+    CALL64pcrel32 target-flags(x86-plt) @abort, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, debug-location !29
+
+  bb.4.inl1.exit7:
+    successors: %bb.5(0x00000800), %bb.6(0x7ffff800)
+    liveins: $eax
+
+    renamable $eax = nsw IMUL32rri killed renamable $eax, 152, implicit-def dead $eflags, debug-location !30
+    renamable $eax = nsw ADD32ri killed renamable $eax, 200, implicit-def dead $eflags, debug-location !31
+    MOV32mr $rip, 1, $noreg, @g2, $noreg, killed renamable $eax, debug-location !32 :: (store (s32) into @g2, !tbaa !13)
+    renamable $eax = MOV32rm $rip, 1, $noreg, @q3, $noreg, debug-location !33 :: (dereferenceable load (s32) from @q3, !tbaa !13)
+    CMP32ri8 renamable $eax, 4, implicit-def $eflags, debug-location !34
+    JCC_1 %bb.6, 12, implicit killed $eflags, debug-location !36
+    JMP_1 %bb.5, debug-location !36
+
+  bb.5.if.then.i9:
+    successors:
+
+    CALL64pcrel32 target-flags(x86-plt) @abort, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, debug-location !37
+
+  bb.6.inl1.exit12:
+    liveins: $eax
+
+    renamable $eax = nsw IMUL32rri killed renamable $eax, 152, implicit-def dead $eflags, debug-location !38
+    renamable $eax = nsw ADD32ri killed renamable $eax, 300, implicit-def dead $eflags, debug-location !39
+    MOV32mr $rip, 1, $noreg, @q3, $noreg, killed renamable $eax, debug-location !40 :: (store (s32) into @q3, !tbaa !13)
+    $eax = MOV32r0 implicit-def dead $eflags, debug-location !41
+    $rcx = frame-destroy POP64r implicit-def $rsp, implicit $rsp, debug-location !41
+    frame-destroy CFI_INSTRUCTION def_cfa_offset 8, debug-location !41
+    RET 0, $eax, debug-location !41
+
+...
+
+# In this case we have a abort call block folded from one single location in
+# three inlined instances of inl1():
+#
+#  1 | #include <stdlib.h>
+#  2 |
+#  3 | static inline int inl1(int q, int n)
+#  4 | {
+#  5 |   if (q > 3)
+#  6 |     abort();
+#  7 |   return q * 152 + n;
+#  8 | }
+#  9 |
+# 10 | int q1 = 1, q2 = 4, q3 = 2;
+# 11 | int g1, g2, g3;
+# 12 |
+# 13 | int multiple_inl_one_loc()
+# 14 | {
+# 15 |   g1 = inl1(q1, 100);
+# 16 |   g2 = inl1(q2, 200);
+# 17 |   q3 = inl1(q3, 300);
+# 18 |   return 0;
+# 19 | }
+#
+# We should produce a merged location describing that the abort call is located
+# at line 6 in inl1() inlined at line 0.
+
+# CHECK-DAG: [[INLINER:![0-9]+]] = distinct !DISubprogram(name: "multiple_inl_one_loc"
+# CHECK-DAG: [[INLINEE:![0-9]+]] = distinct !DISubprogram(name: "inl1"
+
+# CHECK-NOT: CALL64pcrel32
+# CHECK: CALL64pcrel32 target-flags(x86-plt) @abort, {{.*}} debug-location !DILocation(line: 6, column: 5, scope: [[INLINEE]], inlinedAt: !DILocation(line: 0, scope: [[INLINER]]))
+# CHECK-NOT: CALL64pcrel32
diff --git a/llvm/test/DebugInfo/MIR/X86/merge-inline-loc2.mir b/llvm/test/DebugInfo/MIR/X86/merge-inline-loc2.mir
new file mode 100644
index 0000000000000..0564b49fcafdd
--- /dev/null
+++ b/llvm/test/DebugInfo/MIR/X86/merge-inline-loc2.mir
@@ -0,0 +1,339 @@
+# RUN: llc -mtriple=x86_64-pc-linux %s -run-pass=branch-folder -o - | FileCheck %s
+
+--- |
+  ; ModuleID = 'case2.c'
+  source_filename = "case2.c"
+  target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+  target triple = "x86_64-unknown-linux-gnu"
+
+  @q1 = dso_local local_unnamed_addr global i32 1, align 4, !dbg !0
+  @q2 = dso_local local_unnamed_addr global i32 4, align 4, !dbg !5
+  @q3 = dso_local local_unnamed_addr global i32 2, align 4, !dbg !8
+  @g1 = dso_local local_unnamed_addr global i32 0, align 4, !dbg !10
+  @g2 = dso_local local_unnamed_addr global i32 0, align 4, !dbg !12
+  @g3 = dso_local local_unnamed_addr global i32 0, align 4, !dbg !14
+
+  ; Function Attrs: nounwind uwtable
+  define dso_local i32 @multiple_inl_multiple_loc() local_unnamed_addr #0 !dbg !23 {
+  entry:
+    %0 = load i32, ptr @q1, align 4, !dbg !27, !tbaa !28
+    call void @llvm.dbg.value(metadata i32 %0, metadata !32, metadata !DIExpression()), !dbg !38
+    call void @llvm.dbg.value(metadata i32 100, metadata !37, metadata !DIExpression()), !dbg !38
+    %cmp.i = icmp sgt i32 %0, 3, !dbg !40
+    br i1 %cmp.i, label %if.then.i, label %if.end.i, !dbg !42
+
+  if.then.i:                                        ; preds = %entry
+    tail call void @abort() #3, !dbg !43
+    unreachable, !dbg !43
+
+  if.end.i:                                         ; preds = %entry
+    %cmp1.i = icmp slt i32 %0, 1, !dbg !44
+    br i1 %cmp1.i, label %if.then2.i, label %inl2.exit, !dbg !46
+
+  if.then2.i:                                       ; preds = %if.end.i
+    tail call void @abort() #3, !dbg !47
+    unreachable, !dbg !47
+
+  inl2.exit:                                        ; preds = %if.end.i
+    %mul.i = mul nuw nsw i32 %0, 152, !dbg !48
+    %add.i = add nuw nsw i32 %mul.i, 100, !dbg !49
+    store i32 %add.i, ptr @g1, align 4, !dbg !50, !tbaa !28
+    %1 = load i32, ptr @q2, align 4, !dbg !51, !tbaa !28
+    call void @llvm.dbg.value(metadata i32 %1, metadata !32, metadata !DIExpression()), !dbg !52
+    call void @llvm.dbg.value(metadata i32 200, metadata !37, metadata !DIExpression()), !dbg !52
+    %cmp.i3 = icmp sgt i32 %1, 3, !dbg !54
+    br i1 %cmp.i3, label %if.then.i4, label %if.end.i6, !dbg !55
+
+  if.then.i4:                                       ; preds = %inl2.exit
+    tail call void @abort() #3, !dbg !56
+    unreachable, !dbg !56
+
+  if.end.i6:                                        ; preds = %inl2.exit
+    %cmp1.i5 = icmp slt i32 %1, 1, !dbg !57
+    br i1 %cmp1.i5, label %if.then2.i7, label %inl2.exit10, !dbg !58
+
+  if.then2.i7:                                      ; preds = %if.end.i6
+    tail call void @abort() #3, !dbg !59
+    unreachable, !dbg !59
+
+  inl2.exit10:                                      ; preds = %if.end.i6
+    %mul.i8 = mul nuw nsw i32 %1, 152, !dbg !60
+    %add.i9 = add nuw nsw i32 %mul.i8, 200, !dbg !61
+    store i32 %add.i9, ptr @g2, align 4, !dbg !62, !tbaa !28
+    %2 = load i32, ptr @q3, align 4, !dbg !63, !tbaa !28
+    call void @llvm.dbg.value(metadata i32 %2, metadata !32, metadata !DIExpression()), !dbg !64
+    call void @llvm.dbg.value(metadata i32 300, metadata !37, metadata !DIExpression()), !dbg !64
+    %cmp.i11 = icmp sgt i32 %2, 3, !dbg !66
+    br i1 %cmp.i11, label %if.then.i12, label %if.end.i14, !dbg !67
+
+  if.then.i12:                                      ; preds = %inl2.exit10
+    tail call void @abort() #3, !dbg !68
+    unreachable, !dbg !68
+
+  if.end.i14:                                       ; preds = %inl2.exit10
+    %cmp1.i13 = icmp slt i32 %2, 1, !dbg !69
+    br i1 %cmp1.i13, label %if.then2.i15, label %inl2.exit18, !dbg !70
+
+  if.then2.i15:                                     ; preds = %if.end.i14
+    tail call void @abort() #3, !dbg !71
+    unreachable, !dbg !71
+
+  inl2.exit18:                                      ; preds = %if.end.i14
+    %mul.i16 = mul nuw nsw i32 %2, 152, !dbg !72
+    %add.i17 = add nuw nsw i32 %mul.i16, 300, !dbg !73
+    store i32 %add.i17, ptr @q3, align 4, !dbg !74, !tbaa !28
+    ret i32 0, !dbg !75
+  }
+
+  ; Function Attrs: noreturn nounwind
+  declare !dbg !76 void @abort() local_unnamed_addr #1
+
+  ; Function Attrs: nocallback nofree nosync nounwind speculatable willreturn memory(none)
+  declare void @llvm.dbg.value(metadata, metadata, metadata) #2
+
+  attributes #0 = { nounwind uwtable "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
+  attributes #1 = { noreturn nounwind "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
+  attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memory(none) }
+  attributes #3 = { noreturn nounwind }
+
+  !llvm.dbg.cu = !{!2}
+  !llvm.module.flags = !{!16, !17, !18, !19, !20, !21}
+  !llvm.ident = !{!22}
+
+  !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+  !1 = distinct !DIGlobalVariable(name: "q1", scope: !2, file: !3, line: 12, type: !7, isLocal: false, isDefinition: true)
+  !2 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, producer: "clang version 16.0.0.prerel", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, globals: !4, splitDebugInlining: false, nameTableKind: None)
+  !3 = !DIFile(filename: "case2.c", directory: "/", checksumkind: CSK_MD5, checksum: "5093c50294e57eaa7f9ed00bfd62075c")
+  !4 = !{!0, !5, !8, !10, !12, !14}
+  !5 = !DIGlobalVariableExpression(var: !6, expr: !DIExpression())
+  !6 = distinct !DIGlobalVariable(name: "q2", scope: !2, file: !3, line: 12, type: !7, isLocal: false, isDefinition: true)
+  !7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+  !8 = !DIGlobalVariableExpression(var: !9, expr: !DIExpression())
+  !9 = distinct !DIGlobalVariable(name: "q3", scope: !2, file: !3, line: 12, type: !7, isLocal: false, isDefinition: true)
+  !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+  !11 = distinct !DIGlobalVariable(name: "g1", scope: !2, file: !3, line: 13, type: !7, isLocal: false, isDefinition: true)
+  !12 = !DIGlobalVariableExpression(var: !13, expr: !DIExpression())
+  !13 = distinct !DIGlobalVariable(name: "g2", scope: !2, file: !3, line: 13, type: !7, isLocal: false, isDefinition: true)
+  !14 = !DIGlobalVariableExpression(var: !15, expr: !DIExpression())
+  !15 = distinct !DIGlobalVariable(name: "g3", scope: !2, file: !3, line: 13, type: !7, isLocal: false, isDefinition: true)
+  !16 = !{i32 7, !"Dwarf Version", i32 5}
+  !17 = !{i32 2, !"Debug Info Version", i32 3}
+  !18 = !{i32 1, !"wchar_size", i32 4}
+  !19 = !{i32 8, !"PIC Level", i32 2}
+  !20 = !{i32 7, !"PIE Level", i32 2}
+  !21 = !{i32 7, !"uwtable", i32 2}
+  !22 = !{!"clang version 16.0.0.prerel"}
+  !23 = distinct !DISubprogram(name: "multiple_inl_multiple_loc", scope: !3, file: !3, line: 15, type: !24, scopeLine: 16, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2, retainedNodes: !26)
+  !24 = !DISubroutineType(types: !25)
+  !25 = !{!7}
+  !26 = !{}
+  !27 = !DILocation(line: 17, column: 13, scope: !23)
+  !28 = !{!29, !29, i64 0}
+  !29 = !{!"int", !30, i64 0}
+  !30 = !{!"omnipotent char", !31, i64 0}
+  !31 = !{!"Simple C/C++ TBAA"}
+  !32 = !DILocalVariable(name: "q", arg: 1, scope: !33, file: !3, line: 3, type: !7)
+  !33 = distinct !DISubprogram(name: "inl2", scope: !3, file: !3, line: 3, type: !34, scopeLine: 4, flags: DIFlagPrototyped | DIFlagAllCallsDescribed, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition | DISPFlagOptimized, unit: !2, retainedNodes: !36)
+  !34 = !DISubroutineType(types: !35)
+  !35 = !{!7, !7, !7}
+  !36 = !{!32, !37}
+  !37 = !DILocalVariable(name: "n", arg: 2, scope: !33, file: !3, line: 3, type: !7)
+  !38 = !DILocation(line: 0, scope: !33, inlinedAt: !39)
+  !39 = distinct !DILocation(line: 17, column: 8, scope: !23)
+  !40 = !DILocation(line: 5, column: 9, scope: !41, inlinedAt: !39)
+  !41 = distinct !DILexicalBlock(scope: !33, file: !3, line: 5, column: 7)
+  !42 = !DILocation(line: 5, column: 7, scope: !33, inlinedAt: !39)
+  !43 = !DILocation(line: 6, column: 5, scope: !41, inlinedAt: !39)
+  !44 = !DILocation(line: 7, column: 9, scope: !45, inlinedAt: !39)
+  !45 = distinct !DILexicalBlock(scope: !33, file: !3, line: 7, column: 7)
+  !46 = !DILocation(line: 7, column: 7, scope: !33, inlinedAt: !39)
+  !47 = !DILocation(line: 8, column: 5, scope: !45, inlinedAt: !39)
+  !48 = !DILocation(line: 9, column: 12, scope: !33, inlinedAt: !39)
+  !49 = !DILocation(line: 9, column: 18, scope: !33, inlinedAt: !39)
+  !50 = !DILocation(line: 17, column: 6, scope: !23)
+  !51 = !DILocation(line: 18, column: 13, scope: !23)
+  !52 = !DILocation(line: 0, scope: !33, inlinedAt: !53)
+  !53 = distinct !DILocation(line: 18, column: 8, scope: !23)
+  !54 = !DILocation(line: 5, column: 9, scope: !41, inlinedAt: !53)
+  !55 = !DILocation(line: 5, column: 7, scope: !33, inlinedAt: !53)
+  !56 = !DILocation(line: 6, column: 5, scope: !41, inlinedAt: !53)
+  !57 = !DILocation(line: 7, column: 9, scope: !45, inlinedAt: !53)
+  !58 = !DILocation(line: 7, column: 7, scope: !33, inlinedAt: !53)
+  !59 = !DILocation(line: 8, column: 5, scope: !45, inlinedAt: !53)
+  !60 = !DILocation(line: 9, column: 12, scope: !33, inlinedAt: !53)
+  !61 = !DILocation(line: 9, column: 18, scope: !33, inlinedAt: !53)
+  !62 = !DILocation(line: 18, column: 6, scope: !23)
+  !63 = !DILocation(line: 19, column: 13, scope: !23)
+  !64 = !DILocation(line: 0, scope: !33, inlinedAt: !65)
+  !65 = distinct !DILocation(line: 19, column: 8, scope: !23)
+  !66 = !DILocation(line: 5, column: 9, scope: !41, inlinedAt: !65)
+  !67 = !DILocation(line: 5, column: 7, scope: !33, inlinedAt: !65)
+  !68 = !DILocation(line: 6, column: 5, scope: !41, inlinedAt: !65)
+  !69 = !DILocation(line: 7, column: 9, scope: !45, inlinedAt: !65)
+  !70 = !DILocation(line: 7, column: 7, scope: !33, inlinedAt: !65)
+  !71 = !DILocation(line: 8, column: 5, scope: !45, inlinedAt: !65)
+  !72 = !DILocation(line: 9, column: 12, scope: !33, inlinedAt: !65)
+  !73 = !DILocation(line: 9, column: 18, scope: !33, inlinedAt: !65)
+  !74 = !DILocation(line: 19, column: 6, scope: !23)
+  !75 = !DILocation(line: 20, column: 3, scope: !23)
+  !76 = !DISubprogram(name: "abort", scope: !77, file: !77, line: 514, type: !78, flags: DIFlagPrototyped | DIFlagNoReturn, spFlags: DISPFlagOptimized, retainedNodes: !26)
+  !77 = !DIFile(filename: "/usr/include/stdlib.h", directory: "", checksumkind: CSK_MD5, checksum: "f7a1412d75d9e3df251dfc21b02d59ef")
+  !78 = !DISubroutineType(types: !79)
+  !79 = !{null}
+
+...
+---
+name:            multiple_inl_multiple_loc
+alignment:       16
+tracksRegLiveness: true
+tracksDebugUserValues: true
+frameInfo:
+  stackSize:       8
+  offsetAdjustment: -8
+  maxAlignment:    1
+  adjustsStack:    true
+  hasCalls:        true
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0.entry:
+    successors: %bb.1(0x00000800), %bb.2(0x7ffff800)
+
+    frame-setup PUSH64r undef $rax, implicit-def $rsp, implicit $rsp
+    frame-setup CFI_INSTRUCTION def_cfa_offset 16
+    renamable $eax = MOV32rm $rip, 1, $noreg, @q1, $noreg, debug-instr-number 1, debug-location !27 :: (dereferenceable load (s32) from @q1, !tbaa !28)
+    DBG_INSTR_REF !32, !DIExpression(DW_OP_LLVM_arg, 0), dbg-instr-ref(1, 0), debug-location !38
+    DBG_VALUE 100, $noreg, !37, !DIExpression(), debug-location !38
+    CMP32ri8 renamable $eax, 4, implicit-def $eflags, debug-location !40
+    JCC_1 %bb.2, 12, implicit killed $eflags, debug-location !42
+    JMP_1 %bb.1, debug-location !42
+
+  bb.1.if.then.i:
+    successors:
+
+    CALL64pcrel32 target-flags(x86-plt) @abort, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, debug-location !43
+
+  bb.2.if.end.i:
+    successors: %bb.3(0x00000800), %bb.4(0x7ffff800)
+    liveins: $eax
+
+    TEST32rr renamable $eax, renamable $eax, implicit-def $eflags, debug-location !44
+    JCC_1 %bb.4, 15, implicit killed $eflags, debug-location !46
+    JMP_1 %bb.3, debug-location !46
+
+  bb.3.if.then2.i:
+    successors:
+
+    CALL64pcrel32 target-flags(x86-plt) @abort, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, debug-location !47
+
+  bb.4.inl2.exit:
+    successors: %bb.5(0x00000800), %bb.6(0x7ffff800)
+    liveins: $eax
+
+    renamable $eax = nuw nsw IMUL32rri killed renamable $eax, 152, implicit-def dead $eflags, debug-location !48
+    renamable $eax = nuw nsw ADD32ri8 killed renamable $eax, 100, implicit-def dead $eflags, debug-location !49
+    MOV32mr $rip, 1, $noreg, @g1, $noreg, killed renamable $eax, debug-location !50 :: (store (s32) into @g1, !tbaa !28)
+    renamable $eax = MOV32rm $rip, 1, $noreg, @q2, $noreg, debug-instr-number 2, debug-location !51 :: (dereferenceable load (s32) from @q2, !tbaa !28)
+    DBG_INSTR_REF !32, !DIExpression(DW_OP_LLVM_arg, 0), dbg-instr-ref(2, 0), debug-location !52
+    DBG_VALUE 200, $noreg, !37, !DIExpression(), debug-location !52
+    CMP32ri8 renamable $eax, 4, implicit-def $eflags, debug-location !54
+    JCC_1 %bb.6, 12, implicit killed $eflags, debug-location !55
+    JMP_1 %bb.5, debug-location !55
+
+  bb.5.if.then.i4:
+    successors:
+
+    CALL64pcrel32 target-flags(x86-plt) @abort, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, debug-location !56
+
+  bb.6.if.end.i6:
+    successors: %bb.7(0x00000800), %bb.8(0x7ffff800)
+    liveins: $eax
+
+    TEST32rr renamable $eax, renamable $eax, implicit-def $eflags, debug-location !57
+    JCC_1 %bb.8, 15, implicit killed $eflags, debug-location !58
+    JMP_1 %bb.7, debug-location !58
+
+  bb.7.if.then2.i7:
+    successors:
+
+    CALL64pcrel32 target-flags(x86-plt) @abort, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, debug-location !59
+
+  bb.8.inl2.exit10:
+    successors: %bb.9(0x00000800), %bb.10(0x7ffff800)
+    liveins: $eax
+
+    renamable $eax = nuw nsw IMUL32rri killed renamable $eax, 152, implicit-def dead $eflags, debug-location !60
+    renamable $eax = nuw nsw ADD32ri killed renamable $eax, 200, implicit-def dead $eflags, debug-location !61
+    MOV32mr $rip, 1, $noreg, @g2, $noreg, killed renamable $eax, debug-location !62 :: (store (s32) into @g2, !tbaa !28)
+    renamable $eax = MOV32rm $rip, 1, $noreg, @q3, $noreg, debug-instr-number 3, debug-location !63 :: (dereferenceable load (s32) from @q3, !tbaa !28)
+    DBG_INSTR_REF !32, !DIExpression(DW_OP_LLVM_arg, 0), dbg-instr-ref(3, 0), debug-location !64
+    DBG_VALUE 300, $noreg, !37, !DIExpression(), debug-location !64
+    CMP32ri8 renamable $eax, 4, implicit-def $eflags, debug-location !66
+    JCC_1 %bb.10, 12, implicit killed $eflags, debug-location !67
+    JMP_1 %bb.9, debug-location !67
+
+  bb.9.if.then.i12:
+    successors:
+
+    CALL64pcrel32 target-flags(x86-plt) @abort, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, debug-location !68
+
+  bb.10.if.end.i14:
+    successors: %bb.11(0x00000800), %bb.12(0x7ffff800)
+    liveins: $eax
+
+    TEST32rr renamable $eax, renamable $eax, implicit-def $eflags, debug-location !69
+    JCC_1 %bb.12, 15, implicit killed $eflags, debug-location !70
+    JMP_1 %bb.11, debug-location !70
+
+  bb.11.if.then2.i15:
+    successors:
+
+    CALL64pcrel32 target-flags(x86-plt) @abort, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, debug-location !71
+
+  bb.12.inl2.exit18:
+    liveins: $eax
+
+    renamable $eax = nuw nsw IMUL32rri killed renamable $eax, 152, implicit-def dead $eflags, debug-location !72
+    renamable $eax = nuw nsw ADD32ri killed renamable $eax, 300, implicit-def dead $eflags, debug-location !73
+    MOV32mr $rip, 1, $noreg, @q3, $noreg, killed renamable $eax, debug-location !74 :: (store (s32) into @q3, !tbaa !28)
+    $eax = MOV32r0 implicit-def dead $eflags, debug-location !75
+    $rcx = frame-destroy POP64r implicit-def $rsp, implicit $rsp, debug-location !75
+    frame-destroy CFI_INSTRUCTION def_cfa_offset 8, debug-location !75
+    RET 0, $eax, debug-location !75
+
+...
+
+# In this case we have a abort call block folded from two locations in
+# three inlined instances of inl1():
+#
+#  1 | #include <stdlib.h>
+#  2 |
+#  3 | static inline int inl2(int q, int n)
+#  4 | {
+#  5 |   if (q > 3)
+#  6 |     abort();
+#  7 |   if (q < 1)
+#  8 |     abort();
+#  9 |   return q * 152 + n;
+# 10 | }
+# 11 |
+# 12 | int q1 = 1, q2 = 4, q3 = 2;
+# 13 | int g1, g2, g3;
+# 14 |
+# 15 | int multiple_inl_multiple_loc()
+# 16 | {
+# 17 |   g1 = inl2(q1, 100);
+# 18 |   g2 = inl2(q2, 200);
+# 19 |   q3 = inl2(q3, 300);
+# 20 |   return 0;
+# 21 | }
+#
+# We should produce a merged location describing that the abort call is located
+# at line 0 in inl2() inlined at line 0 in multiple_inl_multiple_loc().
+
+# CHECK-DAG: [[INLINER:![0-9]+]] = distinct !DISubprogram(name: "multiple_inl_multiple_loc"
+# CHECK-DAG: [[INLINEE:![0-9]+]] = distinct !DISubprogram(name: "inl2"
+
+# CHECK-NOT: CALL64pcrel32
+# CHECK: CALL64pcrel32 target-flags(x86-plt) @abort, {{.*}} debug-location !DILocation(line: 0, scope: [[INLINEE]], inlinedAt: !DILocation(line: 0, scope: [[INLINER]]))
+# CHECK-NOT: CALL64pcrel32
diff --git a/llvm/test/DebugInfo/MIR/X86/merge-inline-loc3.mir b/llvm/test/DebugInfo/MIR/X86/merge-inline-loc3.mir
new file mode 100644
index 0000000000000..2f21c7912c723
--- /dev/null
+++ b/llvm/test/DebugInfo/MIR/X86/merge-inline-loc3.mir
@@ -0,0 +1,187 @@
+# RUN: llc -mtriple=x86_64-pc-linux %s -run-pass=branch-folder -o - | FileCheck %s
+
+--- |
+  ; ModuleID = 'case3.c'
+  source_filename = "case3.c"
+  target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+  target triple = "x86_64-unknown-linux-gnu"
+
+  @q1 = dso_local local_unnamed_addr global i32 1, align 4
+  @q2 = dso_local local_unnamed_addr global i32 6, align 4
+  @g1 = dso_local local_unnamed_addr global i32 0, align 4
+  @g2 = dso_local local_unnamed_addr global i32 0, align 4
+
+  ; Function Attrs: nounwind uwtable
+  define dso_local i32 @multiple_inl_funcs() local_unnamed_addr #0 !dbg !9 {
+  entry:
+    %0 = load i32, ptr @q1, align 4, !dbg !12, !tbaa !13
+    %cmp.i = icmp sgt i32 %0, 3, !dbg !17
+    br i1 %cmp.i, label %if.then.i, label %inl3.exit, !dbg !20
+
+  if.then.i:                                        ; preds = %entry
+    tail call void @abort() #2, !dbg !21
+    unreachable, !dbg !21
+
+  inl3.exit:                                        ; preds = %entry
+    %mul.i = mul nsw i32 %0, 152, !dbg !22
+    %add.i = add nsw i32 %mul.i, 100, !dbg !23
+    store i32 %add.i, ptr @g1, align 4, !dbg !24, !tbaa !13
+    %1 = load i32, ptr @q2, align 4, !dbg !25, !tbaa !13
+    %cmp.i2 = icmp sgt i32 %1, 5, !dbg !26
+    br i1 %cmp.i2, label %if.then.i3, label %inl4.exit, !dbg !29
+
+  if.then.i3:                                       ; preds = %inl3.exit
+    tail call void @abort() #2, !dbg !30
+    unreachable, !dbg !30
+
+  inl4.exit:                                        ; preds = %inl3.exit
+    %mul.i4 = mul nsw i32 %1, %1, !dbg !31
+    %add.i5 = add nuw nsw i32 %mul.i4, 200, !dbg !32
+    store i32 %add.i5, ptr @g2, align 4, !dbg !33, !tbaa !13
+    ret i32 0, !dbg !34
+  }
+
+  ; Function Attrs: noreturn nounwind
+  declare !dbg !35 void @abort() local_unnamed_addr #1
+
+  attributes #0 = { nounwind uwtable "frame-pointer"="none" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
+  attributes #1 = { noreturn nounwind "frame-pointer"="none" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
+  attributes #2 = { noreturn nounwind }
+
+  !llvm.dbg.cu = !{!0}
+  !llvm.module.flags = !{!2, !3, !4, !5, !6, !7}
+  !llvm.ident = !{!8}
+
+  !0 = distinct !DICompileUnit(language: DW_LANG_C11, file: !1, producer: "clang version 16.0.0.prerel", isOptimized: true, runtimeVersion: 0, emissionKind: LineTablesOnly, splitDebugInlining: false, nameTableKind: None)
+  !1 = !DIFile(filename: "case3.c", directory: "/", checksumkind: CSK_MD5, checksum: "53e9893099480164de1f5ee265c0cf01")
+  !2 = !{i32 7, !"Dwarf Version", i32 5}
+  !3 = !{i32 2, !"Debug Info Version", i32 3}
+  !4 = !{i32 1, !"wchar_size", i32 4}
+  !5 = !{i32 8, !"PIC Level", i32 2}
+  !6 = !{i32 7, !"PIE Level", i32 2}
+  !7 = !{i32 7, !"uwtable", i32 2}
+  !8 = !{!"clang version 16.0.0.prerel"}
+  !9 = distinct !DISubprogram(name: "multiple_inl_funcs", scope: !1, file: !1, line: 18, type: !10, scopeLine: 19, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !11)
+  !10 = !DISubroutineType(types: !11)
+  !11 = !{}
+  !12 = !DILocation(line: 20, column: 13, scope: !9)
+  !13 = !{!14, !14, i64 0}
+  !14 = !{!"int", !15, i64 0}
+  !15 = !{!"omnipotent char", !16, i64 0}
+  !16 = !{!"Simple C/C++ TBAA"}
+  !17 = !DILocation(line: 4, column: 9, scope: !18, inlinedAt: !19)
+  !18 = distinct !DISubprogram(name: "inl3", scope: !1, file: !1, line: 3, type: !10, scopeLine: 3, flags: DIFlagPrototyped | DIFlagAllCallsDescribed, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !11)
+  !19 = distinct !DILocation(line: 20, column: 8, scope: !9)
+  !20 = !DILocation(line: 4, column: 7, scope: !18, inlinedAt: !19)
+  !21 = !DILocation(line: 5, column: 5, scope: !18, inlinedAt: !19)
+  !22 = !DILocation(line: 6, column: 12, scope: !18, inlinedAt: !19)
+  !23 = !DILocation(line: 6, column: 18, scope: !18, inlinedAt: !19)
+  !24 = !DILocation(line: 20, column: 6, scope: !9)
+  !25 = !DILocation(line: 21, column: 13, scope: !9)
+  !26 = !DILocation(line: 10, column: 9, scope: !27, inlinedAt: !28)
+  !27 = distinct !DISubprogram(name: "inl4", scope: !1, file: !1, line: 9, type: !10, scopeLine: 9, flags: DIFlagPrototyped | DIFlagAllCallsDescribed, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !11)
+  !28 = distinct !DILocation(line: 21, column: 8, scope: !9)
+  !29 = !DILocation(line: 10, column: 7, scope: !27, inlinedAt: !28)
+  !30 = !DILocation(line: 11, column: 5, scope: !27, inlinedAt: !28)
+  !31 = !DILocation(line: 12, column: 12, scope: !27, inlinedAt: !28)
+  !32 = !DILocation(line: 12, column: 16, scope: !27, inlinedAt: !28)
+  !33 = !DILocation(line: 21, column: 6, scope: !9)
+  !34 = !DILocation(line: 22, column: 3, scope: !9)
+  !35 = !DISubprogram(name: "abort", scope: !36, file: !36, line: 514, type: !10, flags: DIFlagPrototyped, spFlags: DISPFlagOptimized, retainedNodes: !11)
+  !36 = !DIFile(filename: "/usr/include/stdlib.h", directory: "", checksumkind: CSK_MD5, checksum: "f7a1412d75d9e3df251dfc21b02d59ef")
+
+...
+---
+name:            multiple_inl_funcs
+alignment:       16
+tracksRegLiveness: true
+tracksDebugUserValues: true
+frameInfo:
+  stackSize:       8
+  offsetAdjustment: -8
+  maxAlignment:    1
+  adjustsStack:    true
+  hasCalls:        true
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0.entry:
+    successors: %bb.1(0x00000800), %bb.2(0x7ffff800)
+
+    frame-setup PUSH64r undef $rax, implicit-def $rsp, implicit $rsp
+    frame-setup CFI_INSTRUCTION def_cfa_offset 16
+    renamable $eax = MOV32rm $rip, 1, $noreg, @q1, $noreg, debug-location !12 :: (dereferenceable load (s32) from @q1, !tbaa !13)
+    CMP32ri8 renamable $eax, 4, implicit-def $eflags, debug-location !17
+    JCC_1 %bb.2, 12, implicit killed $eflags, debug-location !20
+    JMP_1 %bb.1, debug-location !20
+
+  bb.1.if.then.i:
+    successors:
+
+    CALL64pcrel32 target-flags(x86-plt) @abort, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, debug-location !21
+
+  bb.2.inl3.exit:
+    successors: %bb.3(0x00000800), %bb.4(0x7ffff800)
+    liveins: $eax
+
+    renamable $eax = nsw IMUL32rri killed renamable $eax, 152, implicit-def dead $eflags, debug-location !22
+    renamable $eax = nsw ADD32ri8 killed renamable $eax, 100, implicit-def dead $eflags, debug-location !23
+    MOV32mr $rip, 1, $noreg, @g1, $noreg, killed renamable $eax, debug-location !24 :: (store (s32) into @g1, !tbaa !13)
+    renamable $eax = MOV32rm $rip, 1, $noreg, @q2, $noreg, debug-location !25 :: (dereferenceable load (s32) from @q2, !tbaa !13)
+    CMP32ri8 renamable $eax, 6, implicit-def $eflags, debug-location !26
+    JCC_1 %bb.4, 12, implicit killed $eflags, debug-location !29
+    JMP_1 %bb.3, debug-location !29
+
+  bb.3.if.then.i3:
+    successors:
+
+    CALL64pcrel32 target-flags(x86-plt) @abort, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, debug-location !30
+
+  bb.4.inl4.exit:
+    liveins: $eax
+
+    renamable $eax = nsw IMUL32rr killed renamable $eax, renamable $eax, implicit-def dead $eflags, debug-location !31
+    renamable $eax = nuw nsw ADD32ri killed renamable $eax, 200, implicit-def dead $eflags, debug-location !32
+    MOV32mr $rip, 1, $noreg, @g2, $noreg, killed renamable $eax, debug-location !33 :: (store (s32) into @g2, !tbaa !13)
+    $eax = MOV32r0 implicit-def dead $eflags, debug-location !34
+    $rcx = frame-destroy POP64r implicit-def $rsp, implicit $rsp, debug-location !34
+    frame-destroy CFI_INSTRUCTION def_cfa_offset 8, debug-location !34
+    RET 0, $eax, debug-location !34
+
+...
+
+# In this case we get a single abort call originated from two separate
+# inlined functions:
+#
+#  1 | #include <stdlib.h>
+#  2 |
+#  3 | static inline int inl3(int q, int n) {
+#  4 |   if (q > 3)
+#  5 |     abort();
+#  6 |   return q * 152 + n;
+#  7 | }
+#  8 |
+#  9 | static inline int inl4(int q, int n) {
+# 10 |   if (q > 5)
+# 11 |     abort();
+# 12 |   return q * q + n;
+# 13 | }
+# 14 |
+# 15 | int q1 = 1, q2 = 6;
+# 16 | int g1, g2;
+# 17 |
+# 18 | int multiple_inl_funcs()
+# 19 | {
+# 20 |   g1 = inl3(q1, 100);
+# 21 |   g2 = inl4(q2, 200);
+# 22 |   return 0;
+# 23 | }
+#
+# We should produce a location at line 0 in the most common scope,
+# multiple_inl_funcs(), without any inline information.
+
+# CHECK: [[INLINER:![0-9]+]] = distinct !DISubprogram(name: "multiple_inl_funcs"
+
+# CHECK-NOT: CALL64pcrel32
+# CHECK: CALL64pcrel32 target-flags(x86-plt) @abort, {{.*}} debug-location !DILocation(line: 0, scope: [[INLINER]])
+# CHECK-NOT: CALL64pcrel32
diff --git a/llvm/test/DebugInfo/MIR/X86/merge-inline-loc4.mir b/llvm/test/DebugInfo/MIR/X86/merge-inline-loc4.mir
new file mode 100644
index 0000000000000..7ebeda4c7e826
--- /dev/null
+++ b/llvm/test/DebugInfo/MIR/X86/merge-inline-loc4.mir
@@ -0,0 +1,166 @@
+# RUN: llc -mtriple=x86_64-pc-linux %s -run-pass=branch-folder -o - | FileCheck %s
+
+--- |
+  ; ModuleID = 'case4.c'
+  source_filename = "case4.c"
+  target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+  target triple = "x86_64-unknown-linux-gnu"
+
+  @q1 = dso_local local_unnamed_addr global i32 1, align 4
+  @q2 = dso_local local_unnamed_addr global i32 6, align 4
+  @g1 = dso_local local_unnamed_addr global i32 0, align 4
+
+  ; Function Attrs: nounwind uwtable
+  define dso_local i32 @merge_inl_and_non_inl() local_unnamed_addr #0 !dbg !9 {
+  entry:
+    %0 = load i32, ptr @q1, align 4, !dbg !12, !tbaa !13
+    %cmp.i = icmp sgt i32 %0, 3, !dbg !17
+    br i1 %cmp.i, label %if.then.i, label %inl5.exit, !dbg !20
+
+  if.then.i:                                        ; preds = %entry
+    tail call void @abort() #2, !dbg !21
+    unreachable, !dbg !21
+
+  inl5.exit:                                        ; preds = %entry
+    %mul.i = mul nsw i32 %0, 152, !dbg !22
+    %add.i = add nsw i32 %mul.i, 100, !dbg !23
+    store i32 %add.i, ptr @g1, align 4, !dbg !24, !tbaa !13
+    %1 = load i32, ptr @q2, align 4, !dbg !25, !tbaa !13
+    %cmp = icmp sgt i32 %1, 5, !dbg !26
+    br i1 %cmp, label %if.then, label %if.end, !dbg !25
+
+  if.then:                                          ; preds = %inl5.exit
+    tail call void @abort() #2, !dbg !27
+    unreachable, !dbg !27
+
+  if.end:                                           ; preds = %inl5.exit
+    ret i32 0, !dbg !28
+  }
+
+  ; Function Attrs: noreturn nounwind
+  declare !dbg !29 void @abort() local_unnamed_addr #1
+
+  attributes #0 = { nounwind uwtable "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
+  attributes #1 = { noreturn nounwind "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
+  attributes #2 = { noreturn nounwind }
+
+  !llvm.dbg.cu = !{!0}
+  !llvm.module.flags = !{!2, !3, !4, !5, !6, !7}
+  !llvm.ident = !{!8}
+
+  !0 = distinct !DICompileUnit(language: DW_LANG_C11, file: !1, producer: "clang version 16.0.0.prerel", isOptimized: true, runtimeVersion: 0, emissionKind: LineTablesOnly, splitDebugInlining: false, nameTableKind: None)
+  !1 = !DIFile(filename: "case4.c", directory: "/", checksumkind: CSK_MD5, checksum: "6ffe3d1878ac05f79e8c3a30566e1b2a")
+  !2 = !{i32 7, !"Dwarf Version", i32 5}
+  !3 = !{i32 2, !"Debug Info Version", i32 3}
+  !4 = !{i32 1, !"wchar_size", i32 4}
+  !5 = !{i32 8, !"PIC Level", i32 2}
+  !6 = !{i32 7, !"PIE Level", i32 2}
+  !7 = !{i32 7, !"uwtable", i32 2}
+  !8 = !{!"clang version 16.0.0.prerel"}
+  !9 = distinct !DISubprogram(name: "merge_inl_and_non_inl", scope: !1, file: !1, line: 12, type: !10, scopeLine: 13, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !11)
+  !10 = !DISubroutineType(types: !11)
+  !11 = !{}
+  !12 = !DILocation(line: 14, column: 13, scope: !9)
+  !13 = !{!14, !14, i64 0}
+  !14 = !{!"int", !15, i64 0}
+  !15 = !{!"omnipotent char", !16, i64 0}
+  !16 = !{!"Simple C/C++ TBAA"}
+  !17 = !DILocation(line: 4, column: 9, scope: !18, inlinedAt: !19)
+  !18 = distinct !DISubprogram(name: "inl5", scope: !1, file: !1, line: 3, type: !10, scopeLine: 3, flags: DIFlagPrototyped | DIFlagAllCallsDescribed, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !11)
+  !19 = distinct !DILocation(line: 14, column: 8, scope: !9)
+  !20 = !DILocation(line: 4, column: 7, scope: !18, inlinedAt: !19)
+  !21 = !DILocation(line: 5, column: 5, scope: !18, inlinedAt: !19)
+  !22 = !DILocation(line: 6, column: 12, scope: !18, inlinedAt: !19)
+  !23 = !DILocation(line: 6, column: 18, scope: !18, inlinedAt: !19)
+  !24 = !DILocation(line: 14, column: 6, scope: !9)
+  !25 = !DILocation(line: 15, column: 7, scope: !9)
+  !26 = !DILocation(line: 15, column: 10, scope: !9)
+  !27 = !DILocation(line: 16, column: 5, scope: !9)
+  !28 = !DILocation(line: 17, column: 3, scope: !9)
+  !29 = !DISubprogram(name: "abort", scope: !30, file: !30, line: 514, type: !10, flags: DIFlagPrototyped, spFlags: DISPFlagOptimized, retainedNodes: !11)
+  !30 = !DIFile(filename: "/usr/include/stdlib.h", directory: "", checksumkind: CSK_MD5, checksum: "f7a1412d75d9e3df251dfc21b02d59ef")
+
+...
+---
+name:            merge_inl_and_non_inl
+alignment:       16
+tracksRegLiveness: true
+tracksDebugUserValues: true
+frameInfo:
+  stackSize:       8
+  offsetAdjustment: -8
+  maxAlignment:    1
+  adjustsStack:    true
+  hasCalls:        true
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0.entry:
+    successors: %bb.1(0x00000800), %bb.2(0x7ffff800)
+
+    frame-setup PUSH64r undef $rax, implicit-def $rsp, implicit $rsp
+    frame-setup CFI_INSTRUCTION def_cfa_offset 16
+    renamable $eax = MOV32rm $rip, 1, $noreg, @q1, $noreg, debug-location !12 :: (dereferenceable load (s32) from @q1, !tbaa !13)
+    CMP32ri8 renamable $eax, 4, implicit-def $eflags, debug-location !17
+    JCC_1 %bb.2, 12, implicit killed $eflags, debug-location !20
+    JMP_1 %bb.1, debug-location !20
+
+  bb.1.if.then.i:
+    successors:
+
+    CALL64pcrel32 target-flags(x86-plt) @abort, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, debug-location !21
+
+  bb.2.inl5.exit:
+    successors: %bb.3(0x00000800), %bb.4(0x7ffff800)
+    liveins: $eax
+
+    renamable $eax = nsw IMUL32rri killed renamable $eax, 152, implicit-def dead $eflags, debug-location !22
+    renamable $eax = nsw ADD32ri8 killed renamable $eax, 100, implicit-def dead $eflags, debug-location !23
+    MOV32mr $rip, 1, $noreg, @g1, $noreg, killed renamable $eax, debug-location !24 :: (store (s32) into @g1, !tbaa !13)
+    CMP32mi8 $rip, 1, $noreg, @q2, $noreg, 6, implicit-def $eflags, debug-location !26 :: (dereferenceable load (s32) from @q2, !tbaa !13)
+    JCC_1 %bb.4, 12, implicit killed $eflags, debug-location !25
+    JMP_1 %bb.3, debug-location !25
+
+  bb.3.if.then:
+    successors:
+
+    CALL64pcrel32 target-flags(x86-plt) @abort, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, debug-location !27
+
+  bb.4.if.end:
+    $eax = MOV32r0 implicit-def dead $eflags, debug-location !28
+    $rcx = frame-destroy POP64r implicit-def $rsp, implicit $rsp, debug-location !28
+    frame-destroy CFI_INSTRUCTION def_cfa_offset 8, debug-location !28
+    RET 0, $eax, debug-location !28
+
+...
+
+# In this case we get a single abort call originating from an inlined instance
+# of inl5() and the function that is inlined in:
+#
+#  1 | #include <stdlib.h>
+#  2 |
+#  3 | static inline int inl5(int q, int n) {
+#  4 |   if (q > 3)
+#  5 |     abort();
+#  6 |   return q * 152 + n;
+#  7 | }
+#  8 |
+#  9 | int q1 = 1, q2 = 6;
+# 10 | int g1;
+# 11 |
+# 12 | int merge_inl_and_non_inl()
+# 13 | {
+# 14 |   g1 = inl5(q1, 100);
+# 15 |   if (q2 > 5)
+# 16 |     abort();
+# 17 |   return 0;
+# 18 | }
+#
+# We should produce a location at line 0 in the most common scope,
+# merge_inl_and_non_inl(), without any inline information.
+
+# CHECK: [[INLINER:![0-9]+]] = distinct !DISubprogram(name: "merge_inl_and_non_inl"
+
+# CHECK-NOT: CALL64pcrel32
+# CHECK: CALL64pcrel32 target-flags(x86-plt) @abort, {{.*}} debug-location !DILocation(line: 0, scope: [[INLINER]])
+# CHECK-NOT: CALL64pcrel32
diff --git a/llvm/test/DebugInfo/WebAssembly/dbg-value-list.ll b/llvm/test/DebugInfo/WebAssembly/dbg-value-list.ll
index c33113673a2a5..dfd4fbbcf3e34 100644
--- a/llvm/test/DebugInfo/WebAssembly/dbg-value-list.ll
+++ b/llvm/test/DebugInfo/WebAssembly/dbg-value-list.ll
@@ -5,12 +5,11 @@ target datalayout = "e-m:e-p:32:32-i64:64-n32:64-S128"
 target triple = "wasm32-unknown-unknown"
 
 ; WebAssembly backend does not currently handle DBG_VALUE_LIST instructions
-; correctly. In the meantime, they are converted to 'DBG_VALUE $noreg's in
+; correctly. In the meantime, they are converted to undefs in
 ; WebAssemblyNullifyDebugValueLists pass.
 
-; BEFORE: DBG_VALUE_LIST
-; AFTER-NOT: DBG_VALUE_LIST
-; AFTER: DBG_VALUE $noreg, $noreg
+; BEFORE: DBG_VALUE_LIST !{{[0-9]+}}, !DIExpression(), %{{[0-9]+}}, %{{[0-9]+}}
+; AFTER: DBG_VALUE_LIST !{{[0-9]+}}, !DIExpression(), $noreg, $noreg
 define i32 @dbg_value_list_test() !dbg !6 {
 entry:
   %0 = call i32 @foo(), !dbg !9
diff --git a/llvm/test/DebugInfo/WebAssembly/tls_pic_globals.ll b/llvm/test/DebugInfo/WebAssembly/tls_pic_globals.ll
new file mode 100644
index 0000000000000..cd837107ecc0b
--- /dev/null
+++ b/llvm/test/DebugInfo/WebAssembly/tls_pic_globals.ll
@@ -0,0 +1,182 @@
+; Non-PIC tests
+; RUN: sed -e 's/\[\[TLS_MODE\]\]/(localexec)/' %s | llc -filetype=obj -mattr=+bulk-memory,atomics - -o %t.localexec.o
+; RUN: sed -e 's/\[\[TLS_MODE\]\]//' %s | llc -filetype=obj -mattr=+bulk-memory,atomics - -o %t.generaldynamic.o
+; RUN: llvm-dwarfdump %t.localexec.o | FileCheck %s --check-prefixes=CHECK,NOPIC
+; RUN: llvm-dwarfdump %t.generaldynamic.o | FileCheck %s --check-prefixes=CHECK,NOPIC
+; RUN: llvm-readobj -r %t.localexec.o | FileCheck %s --check-prefixes=RELOCS-NOSPLIT
+; RUN: llvm-readobj -r %t.generaldynamic.o | FileCheck %s --check-prefixes=RELOCS-NOSPLIT
+
+; PIC tests
+; RUN: sed -e 's/\[\[TLS_MODE\]\]/(localexec)/' %s | llc -filetype=obj -mattr=+bulk-memory,atomics -relocation-model=pic - -o %t.localexec.pic.o
+; RUN: sed -e 's/\[\[TLS_MODE\]\]//' %s | llc -filetype=obj -mattr=+bulk-memory,atomics -relocation-model=pic - -o %t.generaldynamic.pic.o
+; RUN: llvm-dwarfdump %t.localexec.pic.o | FileCheck %s --check-prefixes=CHECK,PIC
+; RUN: llvm-dwarfdump %t.generaldynamic.pic.o | FileCheck %s --check-prefixes=CHECK,PIC
+; RUN: llvm-readobj -r %t.localexec.pic.o | FileCheck %s --check-prefixes=RELOCS-NOSPLIT,RELOCS-PIC-NOSPLIT
+; RUN: llvm-readobj -r %t.generaldynamic.pic.o | FileCheck %s --check-prefixes=RELOCS-NOSPLIT,RELOCS-PIC-NOSPLIT
+
+; Non-PIC + split DWARF tests
+; RUN: sed -e 's/\[\[TLS_MODE\]\]/(localexec)/' %s | llc -filetype=obj -mattr=+bulk-memory,atomics -split-dwarf-file=%t.localexec.split.dwo -split-dwarf-output=%t.localexec.split.dwo - -o %t.localexec.split.o
+; RUN: sed -e 's/\[\[TLS_MODE\]\]//' %s | llc -filetype=obj -mattr=+bulk-memory,atomics -split-dwarf-file=%t.generaldynamic.split.dwo -split-dwarf-output=%t.generaldynamic.split.dwo - -o %t.generaldynamic.split.o
+; RUN: llvm-dwarfdump %t.localexec.split.dwo | FileCheck %s --check-prefixes=CHECK,NOPIC
+; RUN: llvm-dwarfdump %t.generaldynamic.split.dwo | FileCheck %s --check-prefixes=CHECK,NOPIC
+; RUN: llvm-readobj -r %t.localexec.split.dwo | FileCheck %s --check-prefixes=RELOCS-SPLIT
+; RUN: llvm-readobj -r %t.generaldynamic.split.dwo | FileCheck %s --check-prefixes=RELOCS-SPLIT
+
+; PIC + split DWARF tests
+; RUN: sed -e 's/\[\[TLS_MODE\]\]/(localexec)/' %s | llc -filetype=obj -mattr=+bulk-memory,atomics -relocation-model=pic -split-dwarf-file=%t.localexec.pic.split.dwo -split-dwarf-output=%t.localexec.pic.split.dwo - -o %t.localexec.pic.split.o
+; RUN: sed -e 's/\[\[TLS_MODE\]\]//' %s | llc -filetype=obj -mattr=+bulk-memory,atomics -relocation-model=pic -split-dwarf-file=%t.generaldynamic.pic.split.dwo -split-dwarf-output=%t.generaldynamic.pic.split.dwo - -o %t.generaldynamic.pic.split.o
+; RUN: llvm-dwarfdump %t.localexec.pic.split.dwo | FileCheck %s --check-prefixes=CHECK,PIC
+; RUN: llvm-dwarfdump %t.generaldynamic.pic.split.dwo | FileCheck %s --check-prefixes=CHECK,PIC
+; RUN: llvm-readobj -r %t.localexec.pic.split.dwo | FileCheck %s --check-prefixes=RELOCS-SPLIT
+; RUN: llvm-readobj -r %t.generaldynamic.pic.split.dwo | FileCheck %s --check-prefixes=RELOCS-SPLIT
+
+; This test is generated from the following C code, after which some unnecessary
+; debug info is removed.
+
+; int external_var0 = 111;
+; int external_var1 = 222;
+; static int internal_var0 = 333;
+; static int internal_var1 = 444;
+; _Thread_local int external_tls_var0 = 555;
+; _Thread_local int external_tls_var1 = 666;
+; _Thread_local int internal_tls_var0 = 777;
+; _Thread_local int internal_tls_var1 = 888;
+;
+; void foo(int, int, int, int, int, int, int, int);
+;
+; void test_tls_pic_globals() {
+;   foo(external_var0, external_var1, internal_var0, internal_var1,
+;       external_tls_var0, external_tls_var1, internal_tls_var0,
+;       internal_tls_var1);
+; }
+
+target datalayout = "e-m:e-p:32:32-p10:8:8-p20:8:8-i64:64-f128:64-n32:64-S128-ni:1:10:20"
+target triple = "wasm32-unknown-emscripten"
+
+@external_var0 = global i32 111, align 4, !dbg !0
+@external_var1 = global i32 222, align 4, !dbg !5
+@internal_var0 = internal global i32 333, align 4, !dbg !8
+@internal_var1 = internal global i32 444, align 4, !dbg !10
+@external_tls_var0 = thread_local[[TLS_MODE]] global i32 555, align 4, !dbg !12
+@external_tls_var1 = thread_local[[TLS_MODE]] global i32 666, align 4, !dbg !14
+@internal_tls_var0 = internal thread_local[[TLS_MODE]] global i32 777, align 4, !dbg !16
+@internal_tls_var1 = internal thread_local[[TLS_MODE]] global i32 888, align 4, !dbg !18
+
+define void @foo(i32, i32, i32, i32, i32, i32, i32, i32) {
+  ret void
+}
+
+define void @test_tls_pic_globals() !dbg !24 {
+entry:
+  %0 = load i32, ptr @external_var0, align 4
+  %1 = load i32, ptr @external_var1, align 4
+  %2 = load i32, ptr @internal_var0, align 4
+  %3 = load i32, ptr @internal_var1, align 4
+  %4 = call align 4 ptr @llvm.threadlocal.address.p0(ptr align 4 @external_tls_var0)
+  %5 = load i32, ptr %4, align 4
+  %6 = call align 4 ptr @llvm.threadlocal.address.p0(ptr align 4 @external_tls_var1)
+  %7 = load i32, ptr %6, align 4
+  %8 = call align 4 ptr @llvm.threadlocal.address.p0(ptr align 4 @internal_tls_var0)
+  %9 = load i32, ptr %8, align 4
+  %10 = call align 4 ptr @llvm.threadlocal.address.p0(ptr align 4 @internal_tls_var1)
+  %11 = load i32, ptr %10, align 4
+  call void @foo(i32 %0, i32 %1, i32 %2, i32 %3, i32 %5, i32 %7, i32 %9, i32 %11)
+  ret void
+}
+
+; Function Attrs: nocallback nofree nosync nounwind speculatable willreturn memory(none)
+declare nonnull ptr @llvm.threadlocal.address.p0(ptr nonnull) #0
+
+attributes #0 = { nocallback nofree nosync nounwind speculatable willreturn memory(none) }
+
+!llvm.dbg.cu = !{!2}
+!llvm.module.flags = !{!20, !21, !22, !23}
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "external_var0", scope: !2, file: !3, line: 4, type: !7, isLocal: false, isDefinition: true)
+!2 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !4)
+!3 = !DIFile(filename: "test.c", directory: "")
+!4 = !{!0, !5, !8, !10, !12, !14, !16, !18}
+!5 = !DIGlobalVariableExpression(var: !6, expr: !DIExpression())
+!6 = distinct !DIGlobalVariable(name: "external_var1", scope: !2, file: !3, line: 4, type: !7, isLocal: false, isDefinition: true)
+!7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!8 = !DIGlobalVariableExpression(var: !9, expr: !DIExpression())
+!9 = distinct !DIGlobalVariable(name: "internal_var0", scope: !2, file: !3, line: 6, type: !7, isLocal: true, isDefinition: true)
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "internal_var1", scope: !2, file: !3, line: 7, type: !7, isLocal: true, isDefinition: true)
+!12 = !DIGlobalVariableExpression(var: !13, expr: !DIExpression())
+!13 = distinct !DIGlobalVariable(name: "external_tls_var0", scope: !2, file: !3, line: 8, type: !7, isLocal: false, isDefinition: true)
+!14 = !DIGlobalVariableExpression(var: !15, expr: !DIExpression())
+!15 = distinct !DIGlobalVariable(name: "external_tls_var1", scope: !2, file: !3, line: 9, type: !7, isLocal: false, isDefinition: true)
+!16 = !DIGlobalVariableExpression(var: !17, expr: !DIExpression())
+!17 = distinct !DIGlobalVariable(name: "internal_tls_var0", scope: !2, file: !3, line: 9, type: !7, isLocal: true, isDefinition: true)
+!18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
+!19 = distinct !DIGlobalVariable(name: "internal_tls_var1", scope: !2, file: !3, line: 9, type: !7, isLocal: true, isDefinition: true)
+!20 = !{i32 7, !"Dwarf Version", i32 5}
+!21 = !{i32 2, !"Debug Info Version", i32 3}
+!22 = !{i32 1, !"wchar_size", i32 4}
+!23 = !{i32 8, !"PIC Level", i32 2}
+!24 = distinct !DISubprogram(name: "main", scope: !3, file: !3, line: 16, type: !25, spFlags: DISPFlagDefinition, unit: !2)
+!25 = !DISubroutineType(types: !26)
+!26 = !{null}
+
+; Tests if TLS variables and global variables in PIC mode have a correct debug
+; info location. TLS variables should have their location relative to __tls_base
+; global, and global variables in PIC objects should have their location
+; relative to __memory_base global.
+
+; CHECK:      DW_TAG_variable
+; CHECK-NEXT:   DW_AT_name  ("external_var0")
+; CHECK:        DW_AT_external  (true)
+; NOPIC:        DW_AT_location  (DW_OP_addrx 0x0)
+; PIC:          DW_AT_location  (DW_OP_WASM_location 0x3 0x{{[0-9]+}}, DW_OP_addrx 0x0, DW_OP_plus)
+
+; CHECK:      DW_TAG_variable
+; CHECK-NEXT:   DW_AT_name  ("external_var1")
+; CHECK:        DW_AT_external  (true)
+; NOPIC:        DW_AT_location  (DW_OP_addrx 0x1)
+; PIC:          DW_AT_location  (DW_OP_WASM_location 0x3 0x{{[0-9]+}}, DW_OP_addrx 0x1, DW_OP_plus)
+
+; CHECK:      DW_TAG_variable
+; CHECK-NEXT:   DW_AT_name  ("internal_var0")
+; NOPIC:        DW_AT_location  (DW_OP_addrx 0x2)
+; PIC:          DW_AT_location  (DW_OP_WASM_location 0x3 0x{{[0-9]+}}, DW_OP_addrx 0x2, DW_OP_plus)
+
+; CHECK:      DW_TAG_variable
+; CHECK-NEXT:   DW_AT_name  ("internal_var1")
+; NOPIC:        DW_AT_location  (DW_OP_addrx 0x3)
+; PIC:          DW_AT_location  (DW_OP_WASM_location 0x3 0x{{[0-9]+}}, DW_OP_addrx 0x3, DW_OP_plus)
+
+; CHECK:      DW_TAG_variable
+; CHECK-NEXT:   DW_AT_name  ("external_tls_var0")
+; CHECK:        DW_AT_external  (true)
+; CHECK:        DW_AT_location  (DW_OP_WASM_location 0x3 0x{{[0-9]+}}, DW_OP_addrx 0x4, DW_OP_plus)
+
+; CHECK:      DW_TAG_variable
+; CHECK-NEXT:   DW_AT_name  ("external_tls_var1")
+; CHECK:        DW_AT_external  (true)
+; CHECK:        DW_AT_location  (DW_OP_WASM_location 0x3 0x{{[0-9]+}}, DW_OP_addrx 0x5, DW_OP_plus)
+
+; CHECK:      DW_TAG_variable
+; CHECK-NEXT:   DW_AT_name  ("internal_tls_var0")
+; CHECK:        DW_AT_location  (DW_OP_WASM_location 0x3 0x{{[0-9]+}}, DW_OP_addrx 0x6, DW_OP_plus)
+
+; CHECK:      DW_TAG_variable
+; CHECK-NEXT:   DW_AT_name  ("internal_tls_var1")
+; CHECK:        DW_AT_location  (DW_OP_WASM_location 0x3 0x{{[0-9]+}}, DW_OP_addrx 0x7, DW_OP_plus)
+
+; In non-split DWARF, .debug_info section contains relocations referring to
+; __stack_pointer, __tls_base, and __memory_base (if used)
+
+; RELOCS-NOSPLIT:         Relocations [
+; RELOCS-NOSPLIT:           Section (8) .debug_info {
+; RELOCS-NOSPLIT-DAG:         0x{{.*}} R_WASM_GLOBAL_INDEX_I32 __tls_base
+; RELOCS-NOSPLIT-DAG:         0x{{.*}} R_WASM_GLOBAL_INDEX_I32 __stack_pointer
+; RELOCS-PIC-NOSPLIT-DAG:     0x{{.*}} R_WASM_GLOBAL_INDEX_I32 __memory_base
+; RELOCS-NOSPLIT:           }
+; RELOCS-NOSPLIT:           Section (9) .debug_str_offsets {
+
+; In split DWARF, there should be no relocations in .dwo files.
+
+; RELOCS-SPLIT:           Relocations [
+; RELOCS-SPLIT-NEXT:      ]
diff --git a/llvm/test/DebugInfo/X86/debug_value_list_selectiondag.ll b/llvm/test/DebugInfo/X86/debug_value_list_selectiondag.ll
index 4ba8f5fabfc9f..e53b03a5f5f39 100644
--- a/llvm/test/DebugInfo/X86/debug_value_list_selectiondag.ll
+++ b/llvm/test/DebugInfo/X86/debug_value_list_selectiondag.ll
@@ -1,58 +1,66 @@
-; RUN: llc %s -start-after=codegenprepare -stop-before=finalize-isel -experimental-debug-variable-locations=false -o - | FileCheck --check-prefixes=CHECK,DAG %s
-; RUN: llc %s -fast-isel=true -start-after=codegenprepare -stop-before=finalize-isel -experimental-debug-variable-locations=false -o - | FileCheck --check-prefixes=CHECK,FAST %s
-; RUN: llc %s -global-isel=true -start-after=codegenprepare -stop-before=finalize-isel -experimental-debug-variable-locations=false -o - | FileCheck --check-prefixes=CHECK,GLOBAL %s
-
-;; Test that a dbg.value that uses a DIArgList is correctly converted to a
-;; DBG_VALUE_LIST that uses the registers corresponding to its operands.
-
-; CHECK-DAG: [[A_VAR:![0-9]+]] = !DILocalVariable(name: "a"
-; CHECK-DAG: [[B_VAR:![0-9]+]] = !DILocalVariable(name: "b"
-; CHECK-DAG: [[C_VAR:![0-9]+]] = !DILocalVariable(name: "c"
-; CHECK-LABEL: bb.{{(0|1)}}.entry
-; DAG-DAG: DBG_VALUE_LIST [[A_VAR]], !DIExpression(DW_OP_LLVM_arg, 0), %0, debug-location
-; DAG-DAG: DBG_VALUE_LIST [[B_VAR]], !DIExpression(DW_OP_LLVM_arg, 0), %1, debug-location
-; DAG: DBG_VALUE_LIST [[C_VAR]], !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus), %0, %1, debug-location
-; FAST-DAG: DBG_VALUE $noreg, $noreg, [[A_VAR]], !DIExpression(DW_OP_LLVM_arg, 0), debug-location
-; FAST-DAG: DBG_VALUE $noreg, $noreg, [[B_VAR]], !DIExpression(DW_OP_LLVM_arg, 0), debug-location
-; FAST: DBG_VALUE $noreg, $noreg, [[C_VAR]], !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus), debug-location
-; GLOBAL-DAG: DBG_VALUE $noreg, 0, [[A_VAR]], !DIExpression(DW_OP_LLVM_arg, 0), debug-location
-; GLOBAL-DAG: DBG_VALUE $noreg, 0, [[B_VAR]], !DIExpression(DW_OP_LLVM_arg, 0), debug-location
-; GLOBAL: DBG_VALUE $noreg, 0, [[C_VAR]], !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus), debug-location
-
-target datalayout = "e-m:w-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
-target triple = "x86_64-pc-windows-msvc19.16.27034"
-
-define dso_local i32 @"?foo@@YAHHH@Z"(i32 %a, i32 %b) local_unnamed_addr !dbg !8 {
-entry:
-  call void @llvm.dbg.value(metadata !DIArgList(i32 %b), metadata !14, metadata !DIExpression(DW_OP_LLVM_arg, 0)), !dbg !17
-  call void @llvm.dbg.value(metadata !DIArgList(i32 %a), metadata !15, metadata !DIExpression(DW_OP_LLVM_arg, 0)), !dbg !17
-  call void @llvm.dbg.value(metadata !DIArgList(i32 %a, i32 %b), metadata !16, metadata !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus)), !dbg !17
-  %mul = mul nsw i32 %b, %a, !dbg !18
-  ret i32 %mul, !dbg !18
-}
-
-declare void @llvm.dbg.value(metadata, metadata, metadata)
-
-!llvm.dbg.cu = !{!0}
-!llvm.module.flags = !{!3, !4, !5, !6}
-!llvm.ident = !{!7}
-
-!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !1, producer: "clang version 11.0.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, nameTableKind: None)
-!1 = !DIFile(filename: "debug_value_list_selectiondag.cpp", directory: "/")
-!2 = !{}
-!3 = !{i32 2, !"CodeView", i32 1}
-!4 = !{i32 2, !"Debug Info Version", i32 3}
-!5 = !{i32 1, !"wchar_size", i32 2}
-!6 = !{i32 7, !"PIC Level", i32 2}
-!7 = !{!"clang version 11.0.0"}
-!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAHHH@Z", scope: !9, file: !9, line: 1, type: !10, scopeLine: 1, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !13)
-!9 = !DIFile(filename: ".\\debug_value_list.cpp", directory: "/tmp")
-!10 = !DISubroutineType(types: !11)
-!11 = !{!12, !12, !12}
-!12 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!13 = !{!14, !15, !16}
-!14 = !DILocalVariable(name: "b", arg: 2, scope: !8, file: !9, line: 1, type: !12)
-!15 = !DILocalVariable(name: "a", arg: 1, scope: !8, file: !9, line: 1, type: !12)
-!16 = !DILocalVariable(name: "c", scope: !8, file: !9, line: 2, type: !12)
-!17 = !DILocation(line: 0, scope: !8)
-!18 = !DILocation(line: 3, scope: !8)
+; RUN: llc %s -start-after=codegenprepare -stop-before=finalize-isel -experimental-debug-variable-locations=false -o - | FileCheck --check-prefixes=CHECK,DAG %s
+; RUN: llc %s -fast-isel=true -start-after=codegenprepare -stop-before=finalize-isel -experimental-debug-variable-locations=false -o - | FileCheck --check-prefixes=CHECK,FAST %s
+; RUN: llc %s -global-isel=true -start-after=codegenprepare -stop-before=finalize-isel -experimental-debug-variable-locations=false -o - | FileCheck --check-prefixes=CHECK,GLOBAL %s
+
+;; Run with assignment tracking enabled (use sed to add the module flag).
+; RUN: sed 's/;Uncomment-with-sed//g' < %s \
+; RUN: | llc -global-isel=false -start-after=codegenprepare -stop-before=finalize-isel -experimental-debug-variable-locations=false -o - \
+; RUN: | FileCheck --check-prefixes=CHECK,DAG %s
+
+;; Test that a dbg.value that uses a DIArgList is correctly converted to a
+;; DBG_VALUE_LIST that uses the registers corresponding to its operands.
+
+; CHECK-DAG: [[A_VAR:![0-9]+]] = !DILocalVariable(name: "a"
+; CHECK-DAG: [[B_VAR:![0-9]+]] = !DILocalVariable(name: "b"
+; CHECK-DAG: [[C_VAR:![0-9]+]] = !DILocalVariable(name: "c"
+; CHECK-LABEL: bb.{{(0|1)}}.entry
+; DAG-DAG: DBG_VALUE_LIST [[A_VAR]], !DIExpression(DW_OP_LLVM_arg, 0), %0, debug-location
+; DAG-DAG: DBG_VALUE_LIST [[B_VAR]], !DIExpression(DW_OP_LLVM_arg, 0), %1, debug-location
+; DAG: DBG_VALUE_LIST [[C_VAR]], !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus), %0, %1, debug-location
+; FAST-DAG: DBG_VALUE $noreg, $noreg, [[A_VAR]], !DIExpression(DW_OP_LLVM_arg, 0), debug-location
+; FAST-DAG: DBG_VALUE $noreg, $noreg, [[B_VAR]], !DIExpression(DW_OP_LLVM_arg, 0), debug-location
+; FAST: DBG_VALUE $noreg, $noreg, [[C_VAR]], !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus), debug-location
+; GLOBAL-DAG: DBG_VALUE $noreg, 0, [[A_VAR]], !DIExpression(DW_OP_LLVM_arg, 0), debug-location
+; GLOBAL-DAG: DBG_VALUE $noreg, 0, [[B_VAR]], !DIExpression(DW_OP_LLVM_arg, 0), debug-location
+; GLOBAL: DBG_VALUE $noreg, 0, [[C_VAR]], !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus), debug-location
+
+target datalayout = "e-m:w-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-pc-windows-msvc19.16.27034"
+
+define dso_local i32 @"?foo@@YAHHH@Z"(i32 %a, i32 %b) local_unnamed_addr !dbg !8 {
+entry:
+  call void @llvm.dbg.value(metadata !DIArgList(i32 %b), metadata !14, metadata !DIExpression(DW_OP_LLVM_arg, 0)), !dbg !17
+  call void @llvm.dbg.value(metadata !DIArgList(i32 %a), metadata !15, metadata !DIExpression(DW_OP_LLVM_arg, 0)), !dbg !17
+  call void @llvm.dbg.value(metadata !DIArgList(i32 %a, i32 %b), metadata !16, metadata !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus)), !dbg !17
+  %mul = mul nsw i32 %b, %a, !dbg !18
+  ret i32 %mul, !dbg !18
+}
+
+declare void @llvm.dbg.value(metadata, metadata, metadata)
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4, !5,
+;Uncomment-with-sed !19,
+!6}
+!llvm.ident = !{!7}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !1, producer: "clang version 11.0.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, nameTableKind: None)
+!1 = !DIFile(filename: "debug_value_list_selectiondag.cpp", directory: "/")
+!2 = !{}
+!3 = !{i32 2, !"CodeView", i32 1}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = !{i32 1, !"wchar_size", i32 2}
+!6 = !{i32 7, !"PIC Level", i32 2}
+!7 = !{!"clang version 11.0.0"}
+!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAHHH@Z", scope: !9, file: !9, line: 1, type: !10, scopeLine: 1, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !13)
+!9 = !DIFile(filename: ".\\debug_value_list.cpp", directory: "/tmp")
+!10 = !DISubroutineType(types: !11)
+!11 = !{!12, !12, !12}
+!12 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!13 = !{!14, !15, !16}
+!14 = !DILocalVariable(name: "b", arg: 2, scope: !8, file: !9, line: 1, type: !12)
+!15 = !DILocalVariable(name: "a", arg: 1, scope: !8, file: !9, line: 1, type: !12)
+!16 = !DILocalVariable(name: "c", scope: !8, file: !9, line: 2, type: !12)
+!17 = !DILocation(line: 0, scope: !8)
+!18 = !DILocation(line: 3, scope: !8)
+!19 = !{i32 7, !"debug-info-assignment-tracking", i1 true}
diff --git a/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll b/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
index 3e0b5c69dcbda..94b7deec95f41 100644
--- a/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
+++ b/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
@@ -20,10 +20,10 @@
 %struct.BSS1 = type <{ [12 x i8] }>
 
 ;CHECK: @.BSS1.0 = internal unnamed_addr global i32 0, align 32, !dbg ![[GVE1:.*]]
-;CHECK: @.BSS1.1 = internal unnamed_addr global i32 0, align 32, !dbg ![[GVE2:.*]]
+;CHECK: @.BSS1.1 = internal unnamed_addr global i32 0, align 32, !dbg ![[GVE2:.*]], !dbg ![[GVE4:.*]]
 ;CHECK: @.BSS1.2 = internal unnamed_addr global i32 0, align 8, !dbg ![[GVE3:.*]]
 
-@.BSS1 = internal global %struct.BSS1 zeroinitializer, align 32, !dbg !0, !dbg !7, !dbg !10, !dbg !27
+@.BSS1 = internal global %struct.BSS1 zeroinitializer, align 32, !dbg !0, !dbg !7, !dbg !10, !dbg !27, !dbg !29
 @.C330_sub_ = internal constant i32 0
 @.C332_sub_ = internal constant i32 1
 
@@ -82,6 +82,8 @@ declare void @llvm.dbg.declare(metadata, metadata, metadata)
 ; CHECK-DAG: ![[GV2]] = distinct !DIGlobalVariable(name: "nodeb"
 ; CHECK-DAG: ![[GVE3]] = !DIGlobalVariableExpression(var: ![[GV3:.*]], expr: !DIExpression())
 ; CHECK-DAG: ![[GV3]] = distinct !DIGlobalVariable(name: "jmax"
+; CHECK-DAG: ![[GVE4]] = !DIGlobalVariableExpression(var: ![[GV4:.*]], expr: !DIExpression(DW_OP_plus_uconst, 2))
+; CHECK-DAG: ![[GV4]] = distinct !DIGlobalVariable(name: "ivar2"
 
 ;; This does not make sense should be removed.
 ; CHECK-NOT: !DIGlobalVariable(name: "doesnotexist"
@@ -92,7 +94,7 @@ declare void @llvm.dbg.declare(metadata, metadata, metadata)
 !3 = !DIFile(filename: "global-sra-struct-fit-segment.f90", directory: "/tmp")
 !4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g -O0 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
 !5 = !{}
-!6 = !{!0, !7, !10}
+!6 = !{!0, !7, !10, !29}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression(DW_OP_plus_uconst, 4))
 !8 = distinct !DIGlobalVariable(name: "nodeb", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
 !9 = !DIBasicType(name: "integer", size: 32, align: 32, encoding: DW_ATE_signed)
@@ -115,3 +117,6 @@ declare void @llvm.dbg.declare(metadata, metadata, metadata)
 !26 = !DILocation(line: 13, column: 1, scope: !2)
 !27 = !DIGlobalVariableExpression(var: !28, expr: !DIExpression(DW_OP_constu, 4, DW_OP_minus))
 !28 = distinct !DIGlobalVariable(name: "doesnotexist", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
+!29 = !DIGlobalVariableExpression(var: !30, expr: !DIExpression(DW_OP_plus_uconst, 6))
+!30 = distinct !DIGlobalVariable(name: "ivar2", scope: !2, file: !3, line: 3, type: !31, isLocal: true, isDefinition: true)
+!31 = !DIBasicType(name: "integer*2", size: 16, align: 16, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll b/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
new file mode 100644
index 0000000000000..11e5356ea2074
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
@@ -0,0 +1,199 @@
+; RUN: opt -S -p globalopt < %s | FileCheck %s
+
+;; Generated at -g from:
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;;MODULE mymod
+;;   IMPLICIT NONE
+;;   INTEGER, PARAMETER :: dp = SELECTED_REAL_KIND(14, 200)
+;;CONTAINS
+;;   FUNCTION foo(arg)
+;;      REAL(KIND=dp), DIMENSION(3, 3), INTENT(IN)         :: arg
+;;      INTEGER, DIMENSION(3)                              :: foo
+;;      foo = bar(arg)
+;;   END FUNCTION foo
+;;   FUNCTION bar(arg)
+;;      REAL(KIND=dp), DIMENSION(3, 3), INTENT(IN)         :: arg
+;;      INTEGER, DIMENSION(3)                              :: bar
+;;      REAL(KIND=dp)                                      :: rvar(3)
+;;      rvar = arg(1,:)
+;;      IF(rvar(1) == 0._dp) print *,"IF"
+;;      bar = FLOOR(rvar)
+;;   END FUNCTION bar
+;;END MODULE mymod
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+
+%struct.BSS3 = type <{ [24 x i8] }>
+
+@.BSS3 = internal unnamed_addr global %struct.BSS3 zeroinitializer, align 32, !dbg !0, !dbg !7, !dbg !29
+;CHECK: @.BSS3.0 = internal unnamed_addr global double 0.000000e+00, align 32, !dbg ![[GVE1:.*]], !dbg ![[GVE2:.*]]
+;CHECK: @.BSS3.1 = internal unnamed_addr global double 0.000000e+00, align 32, !dbg ![[GVE3:.*]], !dbg ![[GVE4:.*]], !dbg ![[GVE6:.*]]
+;CHECK: @.BSS3.2 = internal unnamed_addr global double 0.000000e+00, align 16, !dbg ![[GVE5:.*]]
+
+@.C363_mymod_bar_ = internal constant [2 x i8] c"IF"
+@.C330_mymod_bar_ = internal constant i32 0
+@.C360_mymod_bar_ = internal constant i32 6
+@.C357_mymod_bar_ = internal constant [9 x i8] c"frag1.f90"
+@.C359_mymod_bar_ = internal constant i32 15
+
+define float @mymod_() local_unnamed_addr {
+.L.entry:
+  ret float undef
+}
+
+define void @mymod_foo_(i64* noalias nocapture writeonly %foo, i64* noalias nocapture readonly %arg) local_unnamed_addr !dbg !22 {
+L.entry:
+  tail call void @llvm.experimental.noalias.scope.decl(metadata !23)
+  tail call void @llvm.experimental.noalias.scope.decl(metadata !26)
+  %0 = bitcast i64* %arg to double*
+  %1 = load double, double* %0, align 8, !noalias !23
+  store double %1, double* bitcast (%struct.BSS3* @.BSS3 to double*), align 32, !noalias !28
+  %2 = getelementptr i64, i64* %arg, i64 3
+  %3 = bitcast i64* %2 to double*
+  %4 = load double, double* %3, align 8, !alias.scope !26, !noalias !23
+  store double %4, double* bitcast (i8* getelementptr inbounds (%struct.BSS3, %struct.BSS3* @.BSS3, i64 0, i32 0, i64 8) to double*), align 8, !noalias !28
+  %5 = getelementptr i64, i64* %arg, i64 6
+  %6 = bitcast i64* %5 to double*
+  %7 = load double, double* %6, align 8, !alias.scope !26, !noalias !23
+  store double %7, double* bitcast (i8* getelementptr inbounds (%struct.BSS3, %struct.BSS3* @.BSS3, i64 0, i32 0, i64 16) to double*), align 16, !noalias !28
+  %8 = fcmp une double %1, 0.000000e+00
+  br i1 %8, label %L.LB3_377.i, label %L.LB3_417.i
+
+L.LB3_417.i:                                      ; preds = %L.entry
+  tail call void (i8*, i8*, i64, ...) bitcast (void (...)* @f90io_src_info03a to void (i8*, i8*, i64, ...)*)(i8* bitcast (i32* @.C359_mymod_bar_ to i8*), i8* getelementptr inbounds ([9 x i8], [9 x i8]* @.C357_mymod_bar_, i64 0, i64 0), i64 9), !noalias !28
+  %9 = tail call i32 (i8*, i8*, i8*, i8*, ...) bitcast (i32 (...)* @f90io_print_init to i32 (i8*, i8*, i8*, i8*, ...)*)(i8* bitcast (i32* @.C360_mymod_bar_ to i8*), i8* null, i8* bitcast (i32* @.C330_mymod_bar_ to i8*), i8* bitcast (i32* @.C330_mymod_bar_ to i8*)), !noalias !28
+  %10 = tail call i32 (i8*, i32, i64, ...) bitcast (i32 (...)* @f90io_sc_ch_ldw to i32 (i8*, i32, i64, ...)*)(i8* getelementptr inbounds ([2 x i8], [2 x i8]* @.C363_mymod_bar_, i64 0, i64 0), i32 14, i64 2), !noalias !28
+  %11 = tail call i32 (...) @f90io_ldw_end(), !noalias !28
+  %.pre = load double, double* bitcast (%struct.BSS3* @.BSS3 to double*), align 32, !noalias !28
+  %.pre1 = load double, double* bitcast (i8* getelementptr inbounds (%struct.BSS3, %struct.BSS3* @.BSS3, i64 0, i32 0, i64 8) to double*), align 8, !noalias !28
+  %.pre2 = load double, double* bitcast (i8* getelementptr inbounds (%struct.BSS3, %struct.BSS3* @.BSS3, i64 0, i32 0, i64 16) to double*), align 16, !noalias !28
+  br label %L.LB3_377.i
+
+L.LB3_377.i:                                      ; preds = %L.LB3_417.i, %L.entry
+  %12 = phi double [ %.pre2, %L.LB3_417.i ], [ %7, %L.entry ]
+  %13 = phi double [ %.pre1, %L.LB3_417.i ], [ %4, %L.entry ]
+  %14 = phi double [ %.pre, %L.LB3_417.i ], [ %1, %L.entry ]
+  %15 = bitcast i64* %foo to i8*
+  %16 = tail call double @llvm.floor.f64(double %14)
+  %17 = fptosi double %16 to i32
+  %18 = bitcast i64* %foo to i32*
+  store i32 %17, i32* %18, align 4, !alias.scope !23, !noalias !26
+  %19 = tail call double @llvm.floor.f64(double %13)
+  %20 = fptosi double %19 to i32
+  %21 = getelementptr i8, i8* %15, i64 4
+  %22 = bitcast i8* %21 to i32*
+  store i32 %20, i32* %22, align 4, !alias.scope !23, !noalias !26
+  %23 = tail call double @llvm.floor.f64(double %12)
+  %24 = fptosi double %23 to i32
+  %25 = getelementptr i64, i64* %foo, i64 1
+  %26 = bitcast i64* %25 to i32*
+  store i32 %24, i32* %26, align 4, !alias.scope !23, !noalias !26
+  ret void
+}
+
+define void @mymod_bar_(i64* noalias nocapture writeonly %bar, i64* noalias nocapture readonly %arg) local_unnamed_addr !dbg !9 {
+L.entry:
+  %0 = bitcast i64* %arg to double*
+  %1 = load double, double* %0, align 8
+  store double %1, double* bitcast (%struct.BSS3* @.BSS3 to double*), align 32
+  %2 = getelementptr i64, i64* %arg, i64 3
+  %3 = bitcast i64* %2 to double*
+  %4 = load double, double* %3, align 8
+  store double %4, double* bitcast (i8* getelementptr inbounds (%struct.BSS3, %struct.BSS3* @.BSS3, i64 0, i32 0, i64 8) to double*), align 8
+  %5 = getelementptr i64, i64* %arg, i64 6
+  %6 = bitcast i64* %5 to double*
+  %7 = load double, double* %6, align 8
+  store double %7, double* bitcast (i8* getelementptr inbounds (%struct.BSS3, %struct.BSS3* @.BSS3, i64 0, i32 0, i64 16) to double*), align 16
+  %8 = fcmp une double %1, 0.000000e+00
+  br i1 %8, label %L.LB3_377, label %L.LB3_417
+
+L.LB3_417:                                        ; preds = %L.entry
+  tail call void (i8*, i8*, i64, ...) bitcast (void (...)* @f90io_src_info03a to void (i8*, i8*, i64, ...)*)(i8* bitcast (i32* @.C359_mymod_bar_ to i8*), i8* getelementptr inbounds ([9 x i8], [9 x i8]* @.C357_mymod_bar_, i64 0, i64 0), i64 9)
+  %9 = tail call i32 (i8*, i8*, i8*, i8*, ...) bitcast (i32 (...)* @f90io_print_init to i32 (i8*, i8*, i8*, i8*, ...)*)(i8* bitcast (i32* @.C360_mymod_bar_ to i8*), i8* null, i8* bitcast (i32* @.C330_mymod_bar_ to i8*), i8* bitcast (i32* @.C330_mymod_bar_ to i8*))
+  %10 = tail call i32 (i8*, i32, i64, ...) bitcast (i32 (...)* @f90io_sc_ch_ldw to i32 (i8*, i32, i64, ...)*)(i8* getelementptr inbounds ([2 x i8], [2 x i8]* @.C363_mymod_bar_, i64 0, i64 0), i32 14, i64 2)
+  %11 = tail call i32 (...) @f90io_ldw_end()
+  %.pre = load double, double* bitcast (%struct.BSS3* @.BSS3 to double*), align 32
+  %.pre1 = load double, double* bitcast (i8* getelementptr inbounds (%struct.BSS3, %struct.BSS3* @.BSS3, i64 0, i32 0, i64 8) to double*), align 8
+  %.pre2 = load double, double* bitcast (i8* getelementptr inbounds (%struct.BSS3, %struct.BSS3* @.BSS3, i64 0, i32 0, i64 16) to double*), align 16
+  br label %L.LB3_377
+
+L.LB3_377:                                        ; preds = %L.LB3_417, %L.entry
+  %12 = phi double [ %.pre2, %L.LB3_417 ], [ %7, %L.entry ]
+  %13 = phi double [ %.pre1, %L.LB3_417 ], [ %4, %L.entry ]
+  %14 = phi double [ %.pre, %L.LB3_417 ], [ %1, %L.entry ]
+  %15 = bitcast i64* %bar to i8*
+  %16 = tail call double @llvm.floor.f64(double %14)
+  %17 = fptosi double %16 to i32
+  %18 = bitcast i64* %bar to i32*
+  store i32 %17, i32* %18, align 4
+  %19 = tail call double @llvm.floor.f64(double %13)
+  %20 = fptosi double %19 to i32
+  %21 = getelementptr i8, i8* %15, i64 4
+  %22 = bitcast i8* %21 to i32*
+  store i32 %20, i32* %22, align 4
+  %23 = tail call double @llvm.floor.f64(double %12)
+  %24 = fptosi double %23 to i32
+  %25 = getelementptr i64, i64* %bar, i64 1
+  %26 = bitcast i64* %25 to i32*
+  store i32 %24, i32* %26, align 4
+  ret void
+}
+
+declare signext i32 @f90io_ldw_end(...) local_unnamed_addr
+
+declare signext i32 @f90io_sc_ch_ldw(...) local_unnamed_addr
+
+declare signext i32 @f90io_print_init(...) local_unnamed_addr
+
+declare void @f90io_src_info03a(...) local_unnamed_addr
+
+; Function Attrs: nocallback nofree nosync nounwind readnone speculatable willreturn
+declare double @llvm.floor.f64(double)
+
+; Function Attrs: inaccessiblememonly nocallback nofree nosync nounwind willreturn
+declare void @llvm.experimental.noalias.scope.decl(metadata)
+
+!llvm.module.flags = !{!20, !21}
+!llvm.dbg.cu = !{!3}
+
+; CHECK-DAG: ![[GVE1]] = !DIGlobalVariableExpression(var: ![[GV1:.*]], expr: !DIExpression(DW_OP_LLVM_fragment, 0, 64))
+; CHECK-DAG: ![[GV1]] = distinct !DIGlobalVariable(name: "bar1"
+; CHECK-DAG: ![[GVE2]] = !DIGlobalVariableExpression(var: ![[GV2:.*]], expr: !DIExpression(DW_OP_LLVM_fragment, 0, 64))
+; CHECK-DAG: ![[GV2]] = distinct !DIGlobalVariable(name: "rvar"
+; CHECK-DAG: ![[GVE3]] = !DIGlobalVariableExpression(var: ![[GV1]], expr: !DIExpression(DW_OP_LLVM_fragment, 64, 32))
+; CHECK-DAG: ![[GVE4]] = !DIGlobalVariableExpression(var: ![[GV2]], expr: !DIExpression(DW_OP_LLVM_fragment, 64, 64))
+; CHECK-DAG: ![[GVE5]] = !DIGlobalVariableExpression(var: ![[GV2]], expr: !DIExpression(DW_OP_LLVM_fragment, 128, 64))
+; CHECK-DAG: ![[GVE6]] = !DIGlobalVariableExpression(var: ![[GV3:.*]], expr: !DIExpression(DW_OP_LLVM_fragment, 32, 32))
+; CHECK-DAG: ![[GV3]] = distinct !DIGlobalVariable(name: "ivar"
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "bar1", scope: !2, file: !4, type: !12, isLocal: true, isDefinition: true)
+!2 = !DIModule(scope: !3, name: "mymod", isDecl: true)
+!3 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !4, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: true, flags: "'+flang -g -O3 -S -emit-llvm -o -O1'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
+!4 = !DIFile(filename: "global-sra-struct-part-overlap-segment_0.f90", directory: "/tmp")
+!5 = !{}
+!6 = !{!0, !7, !29}
+!7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression())
+!8 = distinct !DIGlobalVariable(name: "rvar", scope: !9, file: !4, line: 13, type: !19, isLocal: true, isDefinition: true)
+!9 = distinct !DISubprogram(name: "bar", scope: !2, file: !4, line: 10, type: !10, scopeLine: 10, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !3)
+!10 = !DISubroutineType(types: !11)
+!11 = !{null, !12, !16}
+!12 = !DICompositeType(tag: DW_TAG_array_type, baseType: !13, size: 96, align: 32, elements: !14)
+!13 = !DIBasicType(name: "integer", size: 32, align: 32, encoding: DW_ATE_signed)
+!14 = !{!15}
+!15 = !DISubrange(lowerBound: 1, upperBound: 3)
+!16 = !DICompositeType(tag: DW_TAG_array_type, baseType: !17, size: 576, align: 64, elements: !18)
+!17 = !DIBasicType(name: "double precision", size: 64, align: 64, encoding: DW_ATE_float)
+!18 = !{!15, !15}
+!19 = !DICompositeType(tag: DW_TAG_array_type, baseType: !17, size: 192, align: 64, elements: !14)
+!20 = !{i32 2, !"Dwarf Version", i32 4}
+!21 = !{i32 2, !"Debug Info Version", i32 3}
+!22 = distinct !DISubprogram(name: "foo", scope: !2, file: !4, line: 5, type: !10, scopeLine: 5, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !3)
+!23 = !{!24}
+!24 = distinct !{!24, !25, !"mymod_bar_: %bar"}
+!25 = distinct !{!25, !"mymod_bar_"}
+!26 = !{!27}
+!27 = distinct !{!27, !25, !"mymod_bar_: %arg"}
+!28 = !{!24, !27}
+!29 = !DIGlobalVariableExpression(var: !30, expr: !DIExpression(DW_OP_plus_uconst, 4))
+!30 = distinct !DIGlobalVariable(name: "ivar", scope: !9, file: !4, line: 13, type: !31, isLocal: true, isDefinition: true)
+!31 = !DIBasicType(name: "integer*8", size: 64, align: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/merge_inlined_loc.ll b/llvm/test/DebugInfo/X86/merge_inlined_loc.ll
index 5961f843173a2..07a72d6d30d70 100644
--- a/llvm/test/DebugInfo/X86/merge_inlined_loc.ll
+++ b/llvm/test/DebugInfo/X86/merge_inlined_loc.ll
@@ -6,8 +6,8 @@
 ; locations with the same source line but different scopes should be merged to
 ; a line zero location at the nearest common scope and inlining. The location
 ; of the single call to "common" (the two calls are collapsed together by
-; BranchFolding) should be attributed to line zero inside the wrapper2 inlined
-; scope within f1.
+; BranchFolding) should be attributed to line 2 inside the wrapper inlined
+; scope within wrapper2 at line 0 inlined within f1 at line 13.
 
 ; void common();
 ; inline void wrapper() { common(); }
@@ -22,10 +22,12 @@
 ; }
 ; void f1() { wrapper2(); }
 
-; Ensure there is only one inlined_subroutine (for wrapper2, none for wrapper)
-; & that its address range includes the call to 'common'.
+; Ensure there are two inlined_subroutine (for wrapper and wrapper2).
 
-; CHECK: jmp _Z6commonv
+; CHECK: .loc 1 2 25 epilogue_begin
+; CHECK-NEXT: popq %rax
+; CHECK-NEXT: .cfi_def_cfa_offset 8
+; CHECK-NEXT: jmp _Z6commonv
 ; CHECK-NEXT: [[LABEL:.*]]:
 
 ; CHECK: .section .debug_info
@@ -35,6 +37,9 @@
 ; CHECK:   DW_TAG_inlined_subroutine
 ; CHECK-NOT: {{DW_TAG\|End Of Children}}
 ; CHECK:     [[LABEL]]-{{.*}} DW_AT_high_pc
+; CHECK:     .byte 13 # DW_AT_call_line
+; CHECK:   DW_TAG_inlined_subroutine
+; CHECK:     .byte 0 # DW_AT_call_line
 ; CHECK-NOT: DW_TAG
 
 
diff --git a/llvm/test/DebugInfo/XCOFF/empty.ll b/llvm/test/DebugInfo/XCOFF/empty.ll
index 737436846f1cd..22aff206e6b96 100644
--- a/llvm/test/DebugInfo/XCOFF/empty.ll
+++ b/llvm/test/DebugInfo/XCOFF/empty.ll
@@ -67,7 +67,7 @@ entry:
 ; ASM32-NEXT:                                          # -IsFloatingPointOperationLogOrAbortEnabled
 ; ASM32-NEXT:          .byte   0x40                            # -IsInterruptHandler, +IsFunctionNamePresent, -IsAllocaUsed
 ; ASM32-NEXT:                                          # OnConditionDirective = 0, -IsCRSaved, -IsLRSaved
-; ASM32-NEXT:          .byte   0x80                            # +IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
+; ASM32-NEXT:          .byte   0x00                            # -IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
 ; ASM32-NEXT:          .byte   0x00                            # -HasExtensionTable, -HasVectorInfo, NumOfGPRsSaved = 0
 ; ASM32-NEXT:          .byte   0x00                            # NumberOfFixedParms = 0
 ; ASM32-NEXT:          .byte   0x01                            # NumberOfFPParms = 0, +HasParmsOnStack
@@ -268,7 +268,7 @@ entry:
 ; ASM64-NEXT:                                          # -IsFloatingPointOperationLogOrAbortEnabled
 ; ASM64-NEXT:          .byte   0x40                            # -IsInterruptHandler, +IsFunctionNamePresent, -IsAllocaUsed
 ; ASM64-NEXT:                                          # OnConditionDirective = 0, -IsCRSaved, -IsLRSaved
-; ASM64-NEXT:          .byte   0x80                            # +IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
+; ASM64-NEXT:          .byte   0x00                            # -IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
 ; ASM64-NEXT:          .byte   0x00                            # -HasExtensionTable, -HasVectorInfo, NumOfGPRsSaved = 0
 ; ASM64-NEXT:          .byte   0x00                            # NumberOfFixedParms = 0
 ; ASM64-NEXT:          .byte   0x01                            # NumberOfFPParms = 0, +HasParmsOnStack
diff --git a/llvm/test/DebugInfo/XCOFF/explicit-section.ll b/llvm/test/DebugInfo/XCOFF/explicit-section.ll
index 9a78c07bf62a6..2360ad8f55d82 100644
--- a/llvm/test/DebugInfo/XCOFF/explicit-section.ll
+++ b/llvm/test/DebugInfo/XCOFF/explicit-section.ll
@@ -71,7 +71,7 @@ entry:
 ; CHECK-NEXT:                                          # -IsFloatingPointOperationLogOrAbortEnabled
 ; CHECK-NEXT:          .byte   0x40                            # -IsInterruptHandler, +IsFunctionNamePresent, -IsAllocaUsed
 ; CHECK-NEXT:                                          # OnConditionDirective = 0, -IsCRSaved, -IsLRSaved
-; CHECK-NEXT:          .byte   0x80                            # +IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
+; CHECK-NEXT:          .byte   0x00                            # -IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
 ; CHECK-NEXT:          .byte   0x00                            # -HasExtensionTable, -HasVectorInfo, NumOfGPRsSaved = 0
 ; CHECK-NEXT:          .byte   0x00                            # NumberOfFixedParms = 0
 ; CHECK-NEXT:          .byte   0x01                            # NumberOfFPParms = 0, +HasParmsOnStack
diff --git a/llvm/test/DebugInfo/XCOFF/function-sections.ll b/llvm/test/DebugInfo/XCOFF/function-sections.ll
index f7347dc7bb44b..2b4d236763a77 100644
--- a/llvm/test/DebugInfo/XCOFF/function-sections.ll
+++ b/llvm/test/DebugInfo/XCOFF/function-sections.ll
@@ -66,7 +66,7 @@ entry:
 ; CHECK-NEXT:                                          # -IsFloatingPointOperationLogOrAbortEnabled
 ; CHECK-NEXT:          .byte   0x40                            # -IsInterruptHandler, +IsFunctionNamePresent, -IsAllocaUsed
 ; CHECK-NEXT:                                          # OnConditionDirective = 0, -IsCRSaved, -IsLRSaved
-; CHECK-NEXT:          .byte   0x80                            # +IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
+; CHECK-NEXT:          .byte   0x00                            # -IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
 ; CHECK-NEXT:          .byte   0x00                            # -HasExtensionTable, -HasVectorInfo, NumOfGPRsSaved = 0
 ; CHECK-NEXT:          .byte   0x00                            # NumberOfFixedParms = 0
 ; CHECK-NEXT:          .byte   0x01                            # NumberOfFPParms = 0, +HasParmsOnStack
@@ -102,7 +102,7 @@ entry:
 ; CHECK-NEXT:                                          # -IsFloatingPointOperationLogOrAbortEnabled
 ; CHECK-NEXT:          .byte   0x40                            # -IsInterruptHandler, +IsFunctionNamePresent, -IsAllocaUsed
 ; CHECK-NEXT:                                          # OnConditionDirective = 0, -IsCRSaved, -IsLRSaved
-; CHECK-NEXT:          .byte   0x80                            # +IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
+; CHECK-NEXT:          .byte   0x00                            # -IsBackChainStored, -IsFixup, NumOfFPRsSaved = 0
 ; CHECK-NEXT:          .byte   0x00                            # -HasExtensionTable, -HasVectorInfo, NumOfGPRsSaved = 0
 ; CHECK-NEXT:          .byte   0x00                            # NumberOfFixedParms = 0
 ; CHECK-NEXT:          .byte   0x01                            # NumberOfFPParms = 0, +HasParmsOnStack
diff --git a/llvm/test/ExecutionEngine/JITLink/X86/COFF_comdat_associative_dead_strip.test b/llvm/test/ExecutionEngine/JITLink/X86/COFF_comdat_associative_dead_strip.test
index dd2e4bc7c9055..ad6b5a06abfca 100644
--- a/llvm/test/ExecutionEngine/JITLink/X86/COFF_comdat_associative_dead_strip.test
+++ b/llvm/test/ExecutionEngine/JITLink/X86/COFF_comdat_associative_dead_strip.test
@@ -6,9 +6,10 @@
 # Check a comdat child block connected by associative selection type is dead strip when
 # parent block is dead.
 #
-# CHECK: section parent:
+# CHECK: Link graph
+# CHECK-DAG: section parent:
 # CHECK-EMPTY:
-# CHECK-NEXT: section child:
+# CHECK-DAG: section child:
 # CHECK-EMPTY:
 
 --- !COFF
diff --git a/llvm/test/ExecutionEngine/JITLink/X86/ELF_R_X86_64_32.yaml b/llvm/test/ExecutionEngine/JITLink/X86/ELF_R_X86_64_32.yaml
new file mode 100644
index 0000000000000..a352855970597
--- /dev/null
+++ b/llvm/test/ExecutionEngine/JITLink/X86/ELF_R_X86_64_32.yaml
@@ -0,0 +1,83 @@
+# Check success and failure cases of R_X86_64_32 handling.
+# This file contains a single R_X86_64_32 relocation. We expect the relocation
+# to apply successfully when we assign x an address in the low 32-bits of the
+# address space, and fail if we assign x an address outside that range.
+
+# RUN: yaml2obj -o %t.o %s
+# RUN: llvm-jitlink -abs x=0x12345678 -noexec -check=%s %t.o
+# RUN: not llvm-jitlink -abs x=0x123456789 -noexec %t.o 2>&1 | \
+# RUN:   FileCheck -check-prefix=CHECK-ERROR %s
+
+# jitlink-check: *{8}xptr = x
+
+# CHECK-ERROR: relocation target "x" {{.*}} is out of range of Pointer32 fixup
+
+--- !ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_REL
+  Machine:         EM_X86_64
+  SectionHeaderStringTable: .strtab
+Sections:
+  - Name:            .text
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_ALLOC, SHF_EXECINSTR ]
+    AddressAlign:    0x10
+    Content:         31C0C3
+  - Name:            .data
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_WRITE, SHF_ALLOC ]
+    AddressAlign:    0x8
+    Content:         '0000000000000000'
+  - Name:            .comment
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_MERGE, SHF_STRINGS ]
+    AddressAlign:    0x1
+    EntSize:         0x1
+    Content:         00636C616E672076657273696F6E2031372E302E302028676974406769746875622E636F6D3A6C6C766D2F6C6C766D2D70726F6A6563742E67697420393361303461303539316331393538643533343466333534313135376662643862386666373337302900
+  - Name:            .note.GNU-stack
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x1
+  - Name:            .rela.data
+    Type:            SHT_RELA
+    Flags:           [ SHF_INFO_LINK ]
+    Link:            .symtab
+    AddressAlign:    0x8
+    Info:            .data
+    Relocations:
+      - Symbol:          x
+        Type:            R_X86_64_32
+  - Name:            .llvm_addrsig
+    Type:            SHT_LLVM_ADDRSIG
+    Flags:           [ SHF_EXCLUDE ]
+    Link:            .symtab
+    AddressAlign:    0x1
+    Symbols:         [ x ]
+  - Type:            SectionHeaderTable
+    Sections:
+      - Name:            .strtab
+      - Name:            .text
+      - Name:            .data
+      - Name:            .rela.data
+      - Name:            .comment
+      - Name:            .note.GNU-stack
+      - Name:            .llvm_addrsig
+      - Name:            .symtab
+Symbols:
+  - Name:            testcase.c
+    Type:            STT_FILE
+    Index:           SHN_ABS
+  - Name:            main
+    Type:            STT_FUNC
+    Section:         .text
+    Binding:         STB_GLOBAL
+    Size:            0x3
+  - Name:            xptr
+    Type:            STT_OBJECT
+    Section:         .data
+    Binding:         STB_GLOBAL
+    Size:            0x8
+  - Name:            x
+    Binding:         STB_GLOBAL
+...
diff --git a/llvm/test/ExecutionEngine/JITLink/X86/ELF_debug_section_lifetime_is_NoAlloc.yaml b/llvm/test/ExecutionEngine/JITLink/X86/ELF_debug_section_lifetime_is_NoAlloc.yaml
new file mode 100644
index 0000000000000..0afcda467c326
--- /dev/null
+++ b/llvm/test/ExecutionEngine/JITLink/X86/ELF_debug_section_lifetime_is_NoAlloc.yaml
@@ -0,0 +1,195 @@
+# REQUIRES: asserts
+# RUN: yaml2obj -o %t.o %s
+# RUN: llvm-jitlink -debug-only=jitlink -noexec %t.o 2>&1 | FileCheck %s
+#
+# Check that debug sections get NoAlloc lifetimes.
+#
+# CHECK: ".debug_str" is not a SHF_ALLOC section. Using NoAlloc lifetime.
+# CHECK: ".debug_abbrev" is not a SHF_ALLOC section. Using NoAlloc lifetime.
+# CHECK: ".debug_info" is not a SHF_ALLOC section. Using NoAlloc lifetime.
+# CHECK: ".debug_line" is not a SHF_ALLOC section. Using NoAlloc lifetime.
+
+
+--- !ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_REL
+  Machine:         EM_X86_64
+  SectionHeaderStringTable: .strtab
+Sections:
+  - Name:            .text
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_ALLOC, SHF_EXECINSTR ]
+    AddressAlign:    0x10
+    Content:         B82A000000C3662E0F1F840000000000B82A000000C3
+  - Name:            .debug_abbrev
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x1
+    Content:         011101250E1305030E10171B0E110112060000022E00110112064018974219030E3A0B3B0B271949133F190000032E01110112064018974219030E3A0B3B0B271949133F1900000405000218030E3A0B3B0B49130000052400030E3E0B0B0B0000060F004913000000
+  - Name:            .debug_info
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x1
+    Content:         8C0000000400000000000801000000000C000000000000000000000000000000000000000000160000000200000000000000000600000001570000000001017700000003000000000000000006000000015700000000010577000000040155000000000105770000000401540000000001057E0000000005000000000504068300000006880000000500000000060100
+  - Name:            .comment
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_MERGE, SHF_STRINGS ]
+    AddressAlign:    0x1
+    EntSize:         0x1
+    Content:         00636C616E672076657273696F6E2031302E302E302D347562756E7475312000
+  - Name:            .note.GNU-stack
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x1
+  - Name:            .debug_line
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x1
+    Content:         58000000040036000000010101FB0E0D0001010101000000010000012F746D700000454C465F736B69705F64656275675F73656374696F6E732E63000100000000090200000000000000000105030A130500F505030A130206000101
+  - Name:            .eh_frame
+    Type:            SHT_X86_64_UNWIND
+    Flags:           [ SHF_ALLOC ]
+    AddressAlign:    0x8
+    Content:         1400000000000000017A5200017810011B0C070890010000100000001C0000000000000006000000000000001000000030000000000000000600000000000000
+  - Name:            .rela.debug_info
+    Type:            SHT_RELA
+    Flags:           [ SHF_INFO_LINK ]
+    Link:            .symtab
+    AddressAlign:    0x8
+    Info:            .debug_info
+    Relocations:
+      - Offset:          0x6
+        Symbol:          .debug_abbrev
+        Type:            R_X86_64_32
+      - Offset:          0xC
+        Symbol:          .debug_str
+        Type:            R_X86_64_32
+      - Offset:          0x12
+        Symbol:          .debug_str
+        Type:            R_X86_64_32
+        Addend:          31
+      - Offset:          0x16
+        Symbol:          .debug_line
+        Type:            R_X86_64_32
+      - Offset:          0x1A
+        Symbol:          .debug_str
+        Type:            R_X86_64_32
+        Addend:          57
+      - Offset:          0x1E
+        Symbol:          .text
+        Type:            R_X86_64_64
+      - Offset:          0x2B
+        Symbol:          .text
+        Type:            R_X86_64_64
+      - Offset:          0x39
+        Symbol:          .debug_str
+        Type:            R_X86_64_32
+        Addend:          62
+      - Offset:          0x44
+        Symbol:          .text
+        Type:            R_X86_64_64
+        Addend:          16
+      - Offset:          0x52
+        Symbol:          .debug_str
+        Type:            R_X86_64_32
+        Addend:          70
+      - Offset:          0x5F
+        Symbol:          .debug_str
+        Type:            R_X86_64_32
+        Addend:          75
+      - Offset:          0x6C
+        Symbol:          .debug_str
+        Type:            R_X86_64_32
+        Addend:          80
+      - Offset:          0x78
+        Symbol:          .debug_str
+        Type:            R_X86_64_32
+        Addend:          66
+      - Offset:          0x89
+        Symbol:          .debug_str
+        Type:            R_X86_64_32
+        Addend:          85
+  - Name:            .rela.debug_line
+    Type:            SHT_RELA
+    Flags:           [ SHF_INFO_LINK ]
+    Link:            .symtab
+    AddressAlign:    0x8
+    Info:            .debug_line
+    Relocations:
+      - Offset:          0x43
+        Symbol:          .text
+        Type:            R_X86_64_64
+  - Name:            .rela.eh_frame
+    Type:            SHT_RELA
+    Flags:           [ SHF_INFO_LINK ]
+    Link:            .symtab
+    AddressAlign:    0x8
+    Info:            .eh_frame
+    Relocations:
+      - Offset:          0x20
+        Symbol:          .text
+        Type:            R_X86_64_PC32
+      - Offset:          0x34
+        Symbol:          .text
+        Type:            R_X86_64_PC32
+        Addend:          16
+  - Name:            .llvm_addrsig
+    Type:            SHT_LLVM_ADDRSIG
+    Flags:           [ SHF_EXCLUDE ]
+    Link:            .symtab
+    AddressAlign:    0x1
+    Offset:          0x4C0
+    Symbols:         [  ]
+  - Type:            SectionHeaderTable
+    Sections:
+      - Name:            .strtab
+      - Name:            .text
+      - Name:            .debug_str
+      - Name:            .debug_abbrev
+      - Name:            .debug_info
+      - Name:            .rela.debug_info
+      - Name:            .comment
+      - Name:            .note.GNU-stack
+      - Name:            .debug_line
+      - Name:            .rela.debug_line
+      - Name:            .eh_frame
+      - Name:            .rela.eh_frame
+      - Name:            .llvm_addrsig
+      - Name:            .symtab
+Symbols:
+  - Name:            ELF_skip_debug_sections.c
+    Type:            STT_FILE
+    Index:           SHN_ABS
+  - Name:            .text
+    Type:            STT_SECTION
+    Section:         .text
+  - Name:            .debug_str
+    Type:            STT_SECTION
+    Section:         .debug_str
+  - Name:            .debug_abbrev
+    Type:            STT_SECTION
+    Section:         .debug_abbrev
+  - Name:            .debug_line
+    Type:            STT_SECTION
+    Section:         .debug_line
+  - Name:            foo
+    Type:            STT_FUNC
+    Section:         .text
+    Binding:         STB_GLOBAL
+    Size:            0x6
+  - Name:            main
+    Type:            STT_FUNC
+    Section:         .text
+    Binding:         STB_GLOBAL
+    Value:           0x10
+    Size:            0x6
+DWARF:
+  debug_str:
+    - 'clang version 10.0.0-4ubuntu1 '
+    - ELF_skip_debug_sections.c
+    - '/tmp'
+    - foo
+    - int
+    - main
+    - argc
+    - argv
+    - char
+...
diff --git a/llvm/test/ExecutionEngine/JITLink/X86/ELF_skip_debug_sections.s b/llvm/test/ExecutionEngine/JITLink/X86/ELF_skip_debug_sections.s
deleted file mode 100644
index acd3ae7ad8b1f..0000000000000
--- a/llvm/test/ExecutionEngine/JITLink/X86/ELF_skip_debug_sections.s
+++ /dev/null
@@ -1,241 +0,0 @@
-# REQUIRES: asserts
-# RUN: llvm-mc -triple=x86_64-pc-linux-gnu -filetype=obj -o %t %s
-# RUN: llvm-jitlink -debug-only=jitlink -noexec %t 2>&1 | FileCheck %s
-#
-# Check that debug sections are not emitted.
-#
-# CHECK: ".debug_info" is a debug section: No graph section will be created.
-
-	.text
-	.file	"ELF_skip_debug_sections.c"
-	.globl	foo
-	.p2align	4, 0x90
-	.type	foo,@function
-foo:
-.Lfunc_begin0:
-	.file	1 "/tmp" "ELF_skip_debug_sections.c"
-	.loc	1 1 0
-	.cfi_startproc
-
-	.loc	1 2 3 prologue_end
-	movl	$42, %eax
-	retq
-.Ltmp0:
-.Lfunc_end0:
-	.size	foo, .Lfunc_end0-foo
-	.cfi_endproc
-
-	.globl	main
-	.p2align	4, 0x90
-	.type	main,@function
-main:
-.Lfunc_begin1:
-	.loc	1 5 0
-	.cfi_startproc
-
-
-
-	.loc	1 6 3 prologue_end
-	movl	$42, %eax
-	retq
-.Ltmp1:
-.Lfunc_end1:
-	.size	main, .Lfunc_end1-main
-	.cfi_endproc
-
-	.section	.debug_str,"MS",@progbits,1
-.Linfo_string0:
-	.asciz	"clang version 10.0.0-4ubuntu1 "
-.Linfo_string1:
-	.asciz	"ELF_skip_debug_sections.c"
-.Linfo_string2:
-	.asciz	"/tmp"
-.Linfo_string3:
-	.asciz	"foo"
-.Linfo_string4:
-	.asciz	"int"
-.Linfo_string5:
-	.asciz	"main"
-.Linfo_string6:
-	.asciz	"argc"
-.Linfo_string7:
-	.asciz	"argv"
-.Linfo_string8:
-	.asciz	"char"
-	.section	.debug_abbrev,"",@progbits
-	.byte	1
-	.byte	17
-	.byte	1
-	.byte	37
-	.byte	14
-	.byte	19
-	.byte	5
-	.byte	3
-	.byte	14
-	.byte	16
-	.byte	23
-	.byte	27
-	.byte	14
-	.byte	17
-	.byte	1
-	.byte	18
-	.byte	6
-	.byte	0
-	.byte	0
-	.byte	2
-	.byte	46
-	.byte	0
-	.byte	17
-	.byte	1
-	.byte	18
-	.byte	6
-	.byte	64
-	.byte	24
-	.ascii	"\227B"
-	.byte	25
-	.byte	3
-	.byte	14
-	.byte	58
-	.byte	11
-	.byte	59
-	.byte	11
-	.byte	39
-	.byte	25
-	.byte	73
-	.byte	19
-	.byte	63
-	.byte	25
-	.byte	0
-	.byte	0
-	.byte	3
-	.byte	46
-	.byte	1
-	.byte	17
-	.byte	1
-	.byte	18
-	.byte	6
-	.byte	64
-	.byte	24
-	.ascii	"\227B"
-	.byte	25
-	.byte	3
-	.byte	14
-	.byte	58
-	.byte	11
-	.byte	59
-	.byte	11
-	.byte	39
-	.byte	25
-	.byte	73
-	.byte	19
-	.byte	63
-	.byte	25
-	.byte	0
-	.byte	0
-	.byte	4
-	.byte	5
-	.byte	0
-	.byte	2
-	.byte	24
-	.byte	3
-	.byte	14
-	.byte	58
-	.byte	11
-	.byte	59
-	.byte	11
-	.byte	73
-	.byte	19
-	.byte	0
-	.byte	0
-	.byte	5
-	.byte	36
-	.byte	0
-	.byte	3
-	.byte	14
-	.byte	62
-	.byte	11
-	.byte	11
-	.byte	11
-	.byte	0
-	.byte	0
-	.byte	6
-	.byte	15
-	.byte	0
-	.byte	73
-	.byte	19
-	.byte	0
-	.byte	0
-	.byte	0
-	.section	.debug_info,"",@progbits
-.Lcu_begin0:
-	.long	.Ldebug_info_end0-.Ldebug_info_start0
-.Ldebug_info_start0:
-	.short	4
-	.long	.debug_abbrev
-	.byte	8
-	.byte	1
-	.long	.Linfo_string0
-	.short	12
-	.long	.Linfo_string1
-	.long	.Lline_table_start0
-	.long	.Linfo_string2
-	.quad	.Lfunc_begin0
-	.long	.Lfunc_end1-.Lfunc_begin0
-	.byte	2
-	.quad	.Lfunc_begin0
-	.long	.Lfunc_end0-.Lfunc_begin0
-	.byte	1
-	.byte	87
-
-	.long	.Linfo_string3
-	.byte	1
-	.byte	1
-
-	.long	119
-
-	.byte	3
-	.quad	.Lfunc_begin1
-	.long	.Lfunc_end1-.Lfunc_begin1
-	.byte	1
-	.byte	87
-
-	.long	.Linfo_string5
-	.byte	1
-	.byte	5
-
-	.long	119
-
-	.byte	4
-	.byte	1
-	.byte	85
-	.long	.Linfo_string6
-	.byte	1
-	.byte	5
-	.long	119
-	.byte	4
-	.byte	1
-	.byte	84
-	.long	.Linfo_string7
-	.byte	1
-	.byte	5
-	.long	126
-	.byte	0
-	.byte	5
-	.long	.Linfo_string4
-	.byte	5
-	.byte	4
-	.byte	6
-	.long	131
-	.byte	6
-	.long	136
-	.byte	5
-	.long	.Linfo_string8
-	.byte	6
-	.byte	1
-	.byte	0
-.Ldebug_info_end0:
-	.ident	"clang version 10.0.0-4ubuntu1 "
-	.section	".note.GNU-stack","",@progbits
-	.addrsig
-	.section	.debug_line,"",@progbits
-.Lline_table_start0:
diff --git a/llvm/test/Feature/load_extension.ll b/llvm/test/Feature/load_extension.ll
index 7c97262fdaa8d..dd47f2abc5417 100644
--- a/llvm/test/Feature/load_extension.ll
+++ b/llvm/test/Feature/load_extension.ll
@@ -7,10 +7,6 @@
 ; REQUIRES: plugins, examples
 ; UNSUPPORTED: target={{.*windows.*}}
 ; CHECK: Bye
-;
-; Specifying a new PM pass plugin with the old PM is an error.
-; RUN: ! opt %s %loadnewpmbye -goodbye -wave-goodbye -disable-output -enable-new-pm=0 2>&1 | FileCheck %s --check-prefix=ERROR
-; ERROR: load-pass-plugin specified with legacy PM.
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
diff --git a/llvm/test/Instrumentation/AddressSanitizer/asan-masked-load-store.ll b/llvm/test/Instrumentation/AddressSanitizer/asan-masked-load-store.ll
index a6cde287a8120..45a7977865652 100644
--- a/llvm/test/Instrumentation/AddressSanitizer/asan-masked-load-store.ll
+++ b/llvm/test/Instrumentation/AddressSanitizer/asan-masked-load-store.ll
@@ -308,3 +308,68 @@ define <4 x float> @load.v4f32.1001.after.full.load(ptr %p, <4 x float> %arg) sa
   %res2 = tail call <4 x float> @llvm.masked.load.v4f32.p0(ptr %p, i32 4, <4 x i1> <i1 false, i1 false, i1 false, i1 true>, <4 x float> %arg)
   ret <4 x float> %res2
 }
+
+;; Scalable vector tests
+;; ---------------------------
+declare <vscale x 4 x float> @llvm.masked.load.nxv4f32.p0(ptr, i32, <vscale x 4 x i1>, <vscale x 4 x float>)
+declare void @llvm.masked.store.nxv4f32.p0(<vscale x 4 x float>, ptr, i32, <vscale x 4 x i1>)
+
+define <vscale x 4 x float> @scalable.load.nxv4f32(ptr %p, <vscale x 4 x i1> %mask) sanitize_address {
+; CHECK-LABEL: @scalable.load.nxv4f32(
+; CHECK-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 4
+; CHECK-NEXT:    br label [[DOTSPLIT:%.*]]
+; CHECK:       .split:
+; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ 0, [[TMP0:%.*]] ], [ [[IV_NEXT:%.*]], [[TMP7:%.*]] ]
+; CHECK-NEXT:    [[TMP3:%.*]] = extractelement <vscale x 4 x i1> [[MASK:%.*]], i64 [[IV]]
+; CHECK-NEXT:    br i1 [[TMP3]], label [[TMP4:%.*]], label [[TMP7]]
+; CHECK:       4:
+; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr <vscale x 4 x float>, ptr [[P:%.*]], i64 0, i64 [[IV]]
+; CHECK-NEXT:    [[TMP6:%.*]] = ptrtoint ptr [[TMP5]] to i64
+; CHECK-NEXT:    call void @__asan_load4(i64 [[TMP6]])
+; CHECK-NEXT:    br label [[TMP7]]
+; CHECK:       7:
+; CHECK-NEXT:    [[IV_NEXT]] = add nuw nsw i64 [[IV]], 1
+; CHECK-NEXT:    [[IV_CHECK:%.*]] = icmp eq i64 [[IV_NEXT]], [[TMP2]]
+; CHECK-NEXT:    br i1 [[IV_CHECK]], label [[DOTSPLIT_SPLIT:%.*]], label [[DOTSPLIT]]
+; CHECK:       .split.split:
+; CHECK-NEXT:    [[RES:%.*]] = tail call <vscale x 4 x float> @llvm.masked.load.nxv4f32.p0(ptr [[P]], i32 4, <vscale x 4 x i1> [[MASK]], <vscale x 4 x float> undef)
+; CHECK-NEXT:    ret <vscale x 4 x float> [[RES]]
+;
+; DISABLED-LABEL: @scalable.load.nxv4f32(
+; DISABLED-NEXT:    [[RES:%.*]] = tail call <vscale x 4 x float> @llvm.masked.load.nxv4f32.p0(ptr [[P:%.*]], i32 4, <vscale x 4 x i1> [[MASK:%.*]], <vscale x 4 x float> undef)
+; DISABLED-NEXT:    ret <vscale x 4 x float> [[RES]]
+;
+  %res = tail call <vscale x 4 x float> @llvm.masked.load.nxv4f32.p0(ptr %p, i32 4, <vscale x 4 x i1> %mask, <vscale x 4 x float> undef)
+  ret <vscale x 4 x float> %res
+}
+
+define void @scalable.store.nxv4f32(ptr %p, <vscale x 4 x float> %arg, <vscale x 4 x i1> %mask) sanitize_address {
+; CHECK-LABEL: @scalable.store.nxv4f32(
+; CHECK-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 4
+; CHECK-NEXT:    br label [[DOTSPLIT:%.*]]
+; CHECK:       .split:
+; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ 0, [[TMP0:%.*]] ], [ [[IV_NEXT:%.*]], [[TMP7:%.*]] ]
+; CHECK-NEXT:    [[TMP3:%.*]] = extractelement <vscale x 4 x i1> [[MASK:%.*]], i64 [[IV]]
+; CHECK-NEXT:    br i1 [[TMP3]], label [[TMP4:%.*]], label [[TMP7]]
+; CHECK:       4:
+; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr <vscale x 4 x float>, ptr [[P:%.*]], i64 0, i64 [[IV]]
+; CHECK-NEXT:    [[TMP6:%.*]] = ptrtoint ptr [[TMP5]] to i64
+; CHECK-NEXT:    call void @__asan_store4(i64 [[TMP6]])
+; CHECK-NEXT:    br label [[TMP7]]
+; CHECK:       7:
+; CHECK-NEXT:    [[IV_NEXT]] = add nuw nsw i64 [[IV]], 1
+; CHECK-NEXT:    [[IV_CHECK:%.*]] = icmp eq i64 [[IV_NEXT]], [[TMP2]]
+; CHECK-NEXT:    br i1 [[IV_CHECK]], label [[DOTSPLIT_SPLIT:%.*]], label [[DOTSPLIT]]
+; CHECK:       .split.split:
+; CHECK-NEXT:    tail call void @llvm.masked.store.nxv4f32.p0(<vscale x 4 x float> [[ARG:%.*]], ptr [[P]], i32 4, <vscale x 4 x i1> [[MASK]])
+; CHECK-NEXT:    ret void
+;
+; DISABLED-LABEL: @scalable.store.nxv4f32(
+; DISABLED-NEXT:    tail call void @llvm.masked.store.nxv4f32.p0(<vscale x 4 x float> [[ARG:%.*]], ptr [[P:%.*]], i32 4, <vscale x 4 x i1> [[MASK:%.*]])
+; DISABLED-NEXT:    ret void
+;
+  tail call void @llvm.masked.store.nxv4f32.p0(<vscale x 4 x float> %arg, ptr %p, i32 4, <vscale x 4 x i1> %mask)
+  ret void
+}
diff --git a/llvm/test/Instrumentation/AddressSanitizer/vector-load-store.ll b/llvm/test/Instrumentation/AddressSanitizer/vector-load-store.ll
index 72eca56f4cd57..750c65d8d91f1 100644
--- a/llvm/test/Instrumentation/AddressSanitizer/vector-load-store.ll
+++ b/llvm/test/Instrumentation/AddressSanitizer/vector-load-store.ll
@@ -22,7 +22,7 @@ define void @load.v1i32(ptr %p) sanitize_address {
 ; CHECK-NEXT:    [[TMP11:%.*]] = icmp sge i8 [[TMP10]], [[TMP5]]
 ; CHECK-NEXT:    br i1 [[TMP11]], label [[TMP12:%.*]], label [[TMP13]]
 ; CHECK:       12:
-; CHECK-NEXT:    call void @__asan_report_load4(i64 [[TMP1]]) #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    call void @__asan_report_load4(i64 [[TMP1]]) #[[ATTR4:[0-9]+]]
 ; CHECK-NEXT:    unreachable
 ; CHECK:       13:
 ; CHECK-NEXT:    [[TMP14:%.*]] = load <1 x i32>, ptr [[P]], align 4
@@ -48,7 +48,7 @@ define void @load.v2i32(ptr %p) sanitize_address {
 ; CHECK-NEXT:    [[TMP6:%.*]] = icmp ne i8 [[TMP5]], 0
 ; CHECK-NEXT:    br i1 [[TMP6]], label [[TMP7:%.*]], label [[TMP8:%.*]]
 ; CHECK:       7:
-; CHECK-NEXT:    call void @__asan_report_load8(i64 [[TMP1]]) #[[ATTR3]]
+; CHECK-NEXT:    call void @__asan_report_load8(i64 [[TMP1]]) #[[ATTR4]]
 ; CHECK-NEXT:    unreachable
 ; CHECK:       8:
 ; CHECK-NEXT:    [[TMP9:%.*]] = load <2 x i32>, ptr [[P]], align 8
@@ -74,7 +74,7 @@ define void @load.v4i32(ptr %p) sanitize_address {
 ; CHECK-NEXT:    [[TMP6:%.*]] = icmp ne i16 [[TMP5]], 0
 ; CHECK-NEXT:    br i1 [[TMP6]], label [[TMP7:%.*]], label [[TMP8:%.*]]
 ; CHECK:       7:
-; CHECK-NEXT:    call void @__asan_report_load16(i64 [[TMP1]]) #[[ATTR3]]
+; CHECK-NEXT:    call void @__asan_report_load16(i64 [[TMP1]]) #[[ATTR4]]
 ; CHECK-NEXT:    unreachable
 ; CHECK:       8:
 ; CHECK-NEXT:    [[TMP9:%.*]] = load <4 x i32>, ptr [[P]], align 16
@@ -108,7 +108,7 @@ define void @load.v8i32(ptr %p) sanitize_address {
 ; CHECK-NEXT:    [[TMP13:%.*]] = icmp sge i8 [[TMP12]], [[TMP8]]
 ; CHECK-NEXT:    br i1 [[TMP13]], label [[TMP14:%.*]], label [[TMP15]]
 ; CHECK:       14:
-; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP4]], i64 32) #[[ATTR3]]
+; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP4]], i64 32) #[[ATTR4]]
 ; CHECK-NEXT:    unreachable
 ; CHECK:       15:
 ; CHECK-NEXT:    [[TMP16:%.*]] = ptrtoint ptr [[TMP3]] to i64
@@ -124,7 +124,7 @@ define void @load.v8i32(ptr %p) sanitize_address {
 ; CHECK-NEXT:    [[TMP25:%.*]] = icmp sge i8 [[TMP24]], [[TMP20]]
 ; CHECK-NEXT:    br i1 [[TMP25]], label [[TMP26:%.*]], label [[TMP27]]
 ; CHECK:       26:
-; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP16]], i64 32) #[[ATTR3]]
+; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP16]], i64 32) #[[ATTR4]]
 ; CHECK-NEXT:    unreachable
 ; CHECK:       27:
 ; CHECK-NEXT:    [[TMP28:%.*]] = load <8 x i32>, ptr [[P]], align 32
@@ -158,7 +158,7 @@ define void @load.v16i32(ptr %p) sanitize_address {
 ; CHECK-NEXT:    [[TMP13:%.*]] = icmp sge i8 [[TMP12]], [[TMP8]]
 ; CHECK-NEXT:    br i1 [[TMP13]], label [[TMP14:%.*]], label [[TMP15]]
 ; CHECK:       14:
-; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP4]], i64 64) #[[ATTR3]]
+; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP4]], i64 64) #[[ATTR4]]
 ; CHECK-NEXT:    unreachable
 ; CHECK:       15:
 ; CHECK-NEXT:    [[TMP16:%.*]] = ptrtoint ptr [[TMP3]] to i64
@@ -174,7 +174,7 @@ define void @load.v16i32(ptr %p) sanitize_address {
 ; CHECK-NEXT:    [[TMP25:%.*]] = icmp sge i8 [[TMP24]], [[TMP20]]
 ; CHECK-NEXT:    br i1 [[TMP25]], label [[TMP26:%.*]], label [[TMP27]]
 ; CHECK:       26:
-; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP16]], i64 64) #[[ATTR3]]
+; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP16]], i64 64) #[[ATTR4]]
 ; CHECK-NEXT:    unreachable
 ; CHECK:       27:
 ; CHECK-NEXT:    [[TMP28:%.*]] = load <16 x i32>, ptr [[P]], align 64
@@ -207,7 +207,7 @@ define void @store.v1i32(ptr %p) sanitize_address {
 ; CHECK-NEXT:    [[TMP11:%.*]] = icmp sge i8 [[TMP10]], [[TMP5]]
 ; CHECK-NEXT:    br i1 [[TMP11]], label [[TMP12:%.*]], label [[TMP13]]
 ; CHECK:       12:
-; CHECK-NEXT:    call void @__asan_report_store4(i64 [[TMP1]]) #[[ATTR3]]
+; CHECK-NEXT:    call void @__asan_report_store4(i64 [[TMP1]]) #[[ATTR4]]
 ; CHECK-NEXT:    unreachable
 ; CHECK:       13:
 ; CHECK-NEXT:    store <1 x i32> zeroinitializer, ptr [[P]], align 4
@@ -233,7 +233,7 @@ define void @store.v2i32(ptr %p) sanitize_address {
 ; CHECK-NEXT:    [[TMP6:%.*]] = icmp ne i8 [[TMP5]], 0
 ; CHECK-NEXT:    br i1 [[TMP6]], label [[TMP7:%.*]], label [[TMP8:%.*]]
 ; CHECK:       7:
-; CHECK-NEXT:    call void @__asan_report_store8(i64 [[TMP1]]) #[[ATTR3]]
+; CHECK-NEXT:    call void @__asan_report_store8(i64 [[TMP1]]) #[[ATTR4]]
 ; CHECK-NEXT:    unreachable
 ; CHECK:       8:
 ; CHECK-NEXT:    store <2 x i32> zeroinitializer, ptr [[P]], align 8
@@ -259,7 +259,7 @@ define void @store.v4i32(ptr %p) sanitize_address {
 ; CHECK-NEXT:    [[TMP6:%.*]] = icmp ne i16 [[TMP5]], 0
 ; CHECK-NEXT:    br i1 [[TMP6]], label [[TMP7:%.*]], label [[TMP8:%.*]]
 ; CHECK:       7:
-; CHECK-NEXT:    call void @__asan_report_store16(i64 [[TMP1]]) #[[ATTR3]]
+; CHECK-NEXT:    call void @__asan_report_store16(i64 [[TMP1]]) #[[ATTR4]]
 ; CHECK-NEXT:    unreachable
 ; CHECK:       8:
 ; CHECK-NEXT:    store <4 x i32> zeroinitializer, ptr [[P]], align 16
@@ -293,7 +293,7 @@ define void @store.v8i32(ptr %p) sanitize_address {
 ; CHECK-NEXT:    [[TMP13:%.*]] = icmp sge i8 [[TMP12]], [[TMP8]]
 ; CHECK-NEXT:    br i1 [[TMP13]], label [[TMP14:%.*]], label [[TMP15]]
 ; CHECK:       14:
-; CHECK-NEXT:    call void @__asan_report_store_n(i64 [[TMP4]], i64 32) #[[ATTR3]]
+; CHECK-NEXT:    call void @__asan_report_store_n(i64 [[TMP4]], i64 32) #[[ATTR4]]
 ; CHECK-NEXT:    unreachable
 ; CHECK:       15:
 ; CHECK-NEXT:    [[TMP16:%.*]] = ptrtoint ptr [[TMP3]] to i64
@@ -309,7 +309,7 @@ define void @store.v8i32(ptr %p) sanitize_address {
 ; CHECK-NEXT:    [[TMP25:%.*]] = icmp sge i8 [[TMP24]], [[TMP20]]
 ; CHECK-NEXT:    br i1 [[TMP25]], label [[TMP26:%.*]], label [[TMP27]]
 ; CHECK:       26:
-; CHECK-NEXT:    call void @__asan_report_store_n(i64 [[TMP16]], i64 32) #[[ATTR3]]
+; CHECK-NEXT:    call void @__asan_report_store_n(i64 [[TMP16]], i64 32) #[[ATTR4]]
 ; CHECK-NEXT:    unreachable
 ; CHECK:       27:
 ; CHECK-NEXT:    store <8 x i32> zeroinitializer, ptr [[P]], align 32
@@ -343,7 +343,7 @@ define void @store.v16i32(ptr %p) sanitize_address {
 ; CHECK-NEXT:    [[TMP13:%.*]] = icmp sge i8 [[TMP12]], [[TMP8]]
 ; CHECK-NEXT:    br i1 [[TMP13]], label [[TMP14:%.*]], label [[TMP15]]
 ; CHECK:       14:
-; CHECK-NEXT:    call void @__asan_report_store_n(i64 [[TMP4]], i64 64) #[[ATTR3]]
+; CHECK-NEXT:    call void @__asan_report_store_n(i64 [[TMP4]], i64 64) #[[ATTR4]]
 ; CHECK-NEXT:    unreachable
 ; CHECK:       15:
 ; CHECK-NEXT:    [[TMP16:%.*]] = ptrtoint ptr [[TMP3]] to i64
@@ -359,7 +359,7 @@ define void @store.v16i32(ptr %p) sanitize_address {
 ; CHECK-NEXT:    [[TMP25:%.*]] = icmp sge i8 [[TMP24]], [[TMP20]]
 ; CHECK-NEXT:    br i1 [[TMP25]], label [[TMP26:%.*]], label [[TMP27]]
 ; CHECK:       26:
-; CHECK-NEXT:    call void @__asan_report_store_n(i64 [[TMP16]], i64 64) #[[ATTR3]]
+; CHECK-NEXT:    call void @__asan_report_store_n(i64 [[TMP16]], i64 64) #[[ATTR4]]
 ; CHECK-NEXT:    unreachable
 ; CHECK:       27:
 ; CHECK-NEXT:    store <16 x i32> zeroinitializer, ptr [[P]], align 64
@@ -375,3 +375,639 @@ define void @store.v16i32(ptr %p) sanitize_address {
   ret void
 }
 
+
+define void @load.nxv1i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @load.nxv1i32(
+; CHECK-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 32
+; CHECK-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CHECK-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CHECK-NEXT:    [[TMP5:%.*]] = sub i64 [[TMP3]], 1
+; CHECK-NEXT:    [[TMP6:%.*]] = add i64 [[TMP4]], [[TMP5]]
+; CHECK-NEXT:    [[TMP7:%.*]] = inttoptr i64 [[TMP6]] to ptr
+; CHECK-NEXT:    [[TMP8:%.*]] = ptrtoint ptr [[P]] to i64
+; CHECK-NEXT:    [[TMP9:%.*]] = lshr i64 [[TMP8]], 3
+; CHECK-NEXT:    [[TMP10:%.*]] = or i64 [[TMP9]], 17592186044416
+; CHECK-NEXT:    [[TMP11:%.*]] = inttoptr i64 [[TMP10]] to ptr
+; CHECK-NEXT:    [[TMP12:%.*]] = load i8, ptr [[TMP11]], align 1
+; CHECK-NEXT:    [[TMP13:%.*]] = icmp ne i8 [[TMP12]], 0
+; CHECK-NEXT:    br i1 [[TMP13]], label [[TMP14:%.*]], label [[TMP19:%.*]], !prof [[PROF0]]
+; CHECK:       14:
+; CHECK-NEXT:    [[TMP15:%.*]] = and i64 [[TMP8]], 7
+; CHECK-NEXT:    [[TMP16:%.*]] = trunc i64 [[TMP15]] to i8
+; CHECK-NEXT:    [[TMP17:%.*]] = icmp sge i8 [[TMP16]], [[TMP12]]
+; CHECK-NEXT:    br i1 [[TMP17]], label [[TMP18:%.*]], label [[TMP19]]
+; CHECK:       18:
+; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP8]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       19:
+; CHECK-NEXT:    [[TMP20:%.*]] = ptrtoint ptr [[TMP7]] to i64
+; CHECK-NEXT:    [[TMP21:%.*]] = lshr i64 [[TMP20]], 3
+; CHECK-NEXT:    [[TMP22:%.*]] = or i64 [[TMP21]], 17592186044416
+; CHECK-NEXT:    [[TMP23:%.*]] = inttoptr i64 [[TMP22]] to ptr
+; CHECK-NEXT:    [[TMP24:%.*]] = load i8, ptr [[TMP23]], align 1
+; CHECK-NEXT:    [[TMP25:%.*]] = icmp ne i8 [[TMP24]], 0
+; CHECK-NEXT:    br i1 [[TMP25]], label [[TMP26:%.*]], label [[TMP31:%.*]], !prof [[PROF0]]
+; CHECK:       26:
+; CHECK-NEXT:    [[TMP27:%.*]] = and i64 [[TMP20]], 7
+; CHECK-NEXT:    [[TMP28:%.*]] = trunc i64 [[TMP27]] to i8
+; CHECK-NEXT:    [[TMP29:%.*]] = icmp sge i8 [[TMP28]], [[TMP24]]
+; CHECK-NEXT:    br i1 [[TMP29]], label [[TMP30:%.*]], label [[TMP31]]
+; CHECK:       30:
+; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP20]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       31:
+; CHECK-NEXT:    [[TMP32:%.*]] = load <vscale x 1 x i32>, ptr [[P]], align 4
+; CHECK-NEXT:    ret void
+;
+; CALLS-LABEL: @load.nxv1i32(
+; CALLS-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CALLS-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 32
+; CALLS-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CALLS-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CALLS-NEXT:    call void @__asan_loadN(i64 [[TMP4]], i64 [[TMP3]])
+; CALLS-NEXT:    [[TMP5:%.*]] = load <vscale x 1 x i32>, ptr [[P]], align 4
+; CALLS-NEXT:    ret void
+;
+  load <vscale x 1 x i32>, ptr %p
+  ret void
+}
+
+define void @load.nxv2i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @load.nxv2i32(
+; CHECK-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 64
+; CHECK-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CHECK-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CHECK-NEXT:    [[TMP5:%.*]] = sub i64 [[TMP3]], 1
+; CHECK-NEXT:    [[TMP6:%.*]] = add i64 [[TMP4]], [[TMP5]]
+; CHECK-NEXT:    [[TMP7:%.*]] = inttoptr i64 [[TMP6]] to ptr
+; CHECK-NEXT:    [[TMP8:%.*]] = ptrtoint ptr [[P]] to i64
+; CHECK-NEXT:    [[TMP9:%.*]] = lshr i64 [[TMP8]], 3
+; CHECK-NEXT:    [[TMP10:%.*]] = or i64 [[TMP9]], 17592186044416
+; CHECK-NEXT:    [[TMP11:%.*]] = inttoptr i64 [[TMP10]] to ptr
+; CHECK-NEXT:    [[TMP12:%.*]] = load i8, ptr [[TMP11]], align 1
+; CHECK-NEXT:    [[TMP13:%.*]] = icmp ne i8 [[TMP12]], 0
+; CHECK-NEXT:    br i1 [[TMP13]], label [[TMP14:%.*]], label [[TMP19:%.*]], !prof [[PROF0]]
+; CHECK:       14:
+; CHECK-NEXT:    [[TMP15:%.*]] = and i64 [[TMP8]], 7
+; CHECK-NEXT:    [[TMP16:%.*]] = trunc i64 [[TMP15]] to i8
+; CHECK-NEXT:    [[TMP17:%.*]] = icmp sge i8 [[TMP16]], [[TMP12]]
+; CHECK-NEXT:    br i1 [[TMP17]], label [[TMP18:%.*]], label [[TMP19]]
+; CHECK:       18:
+; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP8]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       19:
+; CHECK-NEXT:    [[TMP20:%.*]] = ptrtoint ptr [[TMP7]] to i64
+; CHECK-NEXT:    [[TMP21:%.*]] = lshr i64 [[TMP20]], 3
+; CHECK-NEXT:    [[TMP22:%.*]] = or i64 [[TMP21]], 17592186044416
+; CHECK-NEXT:    [[TMP23:%.*]] = inttoptr i64 [[TMP22]] to ptr
+; CHECK-NEXT:    [[TMP24:%.*]] = load i8, ptr [[TMP23]], align 1
+; CHECK-NEXT:    [[TMP25:%.*]] = icmp ne i8 [[TMP24]], 0
+; CHECK-NEXT:    br i1 [[TMP25]], label [[TMP26:%.*]], label [[TMP31:%.*]], !prof [[PROF0]]
+; CHECK:       26:
+; CHECK-NEXT:    [[TMP27:%.*]] = and i64 [[TMP20]], 7
+; CHECK-NEXT:    [[TMP28:%.*]] = trunc i64 [[TMP27]] to i8
+; CHECK-NEXT:    [[TMP29:%.*]] = icmp sge i8 [[TMP28]], [[TMP24]]
+; CHECK-NEXT:    br i1 [[TMP29]], label [[TMP30:%.*]], label [[TMP31]]
+; CHECK:       30:
+; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP20]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       31:
+; CHECK-NEXT:    [[TMP32:%.*]] = load <vscale x 2 x i32>, ptr [[P]], align 8
+; CHECK-NEXT:    ret void
+;
+; CALLS-LABEL: @load.nxv2i32(
+; CALLS-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CALLS-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 64
+; CALLS-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CALLS-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CALLS-NEXT:    call void @__asan_loadN(i64 [[TMP4]], i64 [[TMP3]])
+; CALLS-NEXT:    [[TMP5:%.*]] = load <vscale x 2 x i32>, ptr [[P]], align 8
+; CALLS-NEXT:    ret void
+;
+  load <vscale x 2 x i32>, ptr %p
+  ret void
+}
+
+define void @load.nxv4i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @load.nxv4i32(
+; CHECK-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 128
+; CHECK-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CHECK-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CHECK-NEXT:    [[TMP5:%.*]] = sub i64 [[TMP3]], 1
+; CHECK-NEXT:    [[TMP6:%.*]] = add i64 [[TMP4]], [[TMP5]]
+; CHECK-NEXT:    [[TMP7:%.*]] = inttoptr i64 [[TMP6]] to ptr
+; CHECK-NEXT:    [[TMP8:%.*]] = ptrtoint ptr [[P]] to i64
+; CHECK-NEXT:    [[TMP9:%.*]] = lshr i64 [[TMP8]], 3
+; CHECK-NEXT:    [[TMP10:%.*]] = or i64 [[TMP9]], 17592186044416
+; CHECK-NEXT:    [[TMP11:%.*]] = inttoptr i64 [[TMP10]] to ptr
+; CHECK-NEXT:    [[TMP12:%.*]] = load i8, ptr [[TMP11]], align 1
+; CHECK-NEXT:    [[TMP13:%.*]] = icmp ne i8 [[TMP12]], 0
+; CHECK-NEXT:    br i1 [[TMP13]], label [[TMP14:%.*]], label [[TMP19:%.*]], !prof [[PROF0]]
+; CHECK:       14:
+; CHECK-NEXT:    [[TMP15:%.*]] = and i64 [[TMP8]], 7
+; CHECK-NEXT:    [[TMP16:%.*]] = trunc i64 [[TMP15]] to i8
+; CHECK-NEXT:    [[TMP17:%.*]] = icmp sge i8 [[TMP16]], [[TMP12]]
+; CHECK-NEXT:    br i1 [[TMP17]], label [[TMP18:%.*]], label [[TMP19]]
+; CHECK:       18:
+; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP8]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       19:
+; CHECK-NEXT:    [[TMP20:%.*]] = ptrtoint ptr [[TMP7]] to i64
+; CHECK-NEXT:    [[TMP21:%.*]] = lshr i64 [[TMP20]], 3
+; CHECK-NEXT:    [[TMP22:%.*]] = or i64 [[TMP21]], 17592186044416
+; CHECK-NEXT:    [[TMP23:%.*]] = inttoptr i64 [[TMP22]] to ptr
+; CHECK-NEXT:    [[TMP24:%.*]] = load i8, ptr [[TMP23]], align 1
+; CHECK-NEXT:    [[TMP25:%.*]] = icmp ne i8 [[TMP24]], 0
+; CHECK-NEXT:    br i1 [[TMP25]], label [[TMP26:%.*]], label [[TMP31:%.*]], !prof [[PROF0]]
+; CHECK:       26:
+; CHECK-NEXT:    [[TMP27:%.*]] = and i64 [[TMP20]], 7
+; CHECK-NEXT:    [[TMP28:%.*]] = trunc i64 [[TMP27]] to i8
+; CHECK-NEXT:    [[TMP29:%.*]] = icmp sge i8 [[TMP28]], [[TMP24]]
+; CHECK-NEXT:    br i1 [[TMP29]], label [[TMP30:%.*]], label [[TMP31]]
+; CHECK:       30:
+; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP20]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       31:
+; CHECK-NEXT:    [[TMP32:%.*]] = load <vscale x 4 x i32>, ptr [[P]], align 16
+; CHECK-NEXT:    ret void
+;
+; CALLS-LABEL: @load.nxv4i32(
+; CALLS-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CALLS-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 128
+; CALLS-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CALLS-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CALLS-NEXT:    call void @__asan_loadN(i64 [[TMP4]], i64 [[TMP3]])
+; CALLS-NEXT:    [[TMP5:%.*]] = load <vscale x 4 x i32>, ptr [[P]], align 16
+; CALLS-NEXT:    ret void
+;
+  load <vscale x 4 x i32>, ptr %p
+  ret void
+}
+
+define void @load.nxv8i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @load.nxv8i32(
+; CHECK-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 256
+; CHECK-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CHECK-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CHECK-NEXT:    [[TMP5:%.*]] = sub i64 [[TMP3]], 1
+; CHECK-NEXT:    [[TMP6:%.*]] = add i64 [[TMP4]], [[TMP5]]
+; CHECK-NEXT:    [[TMP7:%.*]] = inttoptr i64 [[TMP6]] to ptr
+; CHECK-NEXT:    [[TMP8:%.*]] = ptrtoint ptr [[P]] to i64
+; CHECK-NEXT:    [[TMP9:%.*]] = lshr i64 [[TMP8]], 3
+; CHECK-NEXT:    [[TMP10:%.*]] = or i64 [[TMP9]], 17592186044416
+; CHECK-NEXT:    [[TMP11:%.*]] = inttoptr i64 [[TMP10]] to ptr
+; CHECK-NEXT:    [[TMP12:%.*]] = load i8, ptr [[TMP11]], align 1
+; CHECK-NEXT:    [[TMP13:%.*]] = icmp ne i8 [[TMP12]], 0
+; CHECK-NEXT:    br i1 [[TMP13]], label [[TMP14:%.*]], label [[TMP19:%.*]], !prof [[PROF0]]
+; CHECK:       14:
+; CHECK-NEXT:    [[TMP15:%.*]] = and i64 [[TMP8]], 7
+; CHECK-NEXT:    [[TMP16:%.*]] = trunc i64 [[TMP15]] to i8
+; CHECK-NEXT:    [[TMP17:%.*]] = icmp sge i8 [[TMP16]], [[TMP12]]
+; CHECK-NEXT:    br i1 [[TMP17]], label [[TMP18:%.*]], label [[TMP19]]
+; CHECK:       18:
+; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP8]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       19:
+; CHECK-NEXT:    [[TMP20:%.*]] = ptrtoint ptr [[TMP7]] to i64
+; CHECK-NEXT:    [[TMP21:%.*]] = lshr i64 [[TMP20]], 3
+; CHECK-NEXT:    [[TMP22:%.*]] = or i64 [[TMP21]], 17592186044416
+; CHECK-NEXT:    [[TMP23:%.*]] = inttoptr i64 [[TMP22]] to ptr
+; CHECK-NEXT:    [[TMP24:%.*]] = load i8, ptr [[TMP23]], align 1
+; CHECK-NEXT:    [[TMP25:%.*]] = icmp ne i8 [[TMP24]], 0
+; CHECK-NEXT:    br i1 [[TMP25]], label [[TMP26:%.*]], label [[TMP31:%.*]], !prof [[PROF0]]
+; CHECK:       26:
+; CHECK-NEXT:    [[TMP27:%.*]] = and i64 [[TMP20]], 7
+; CHECK-NEXT:    [[TMP28:%.*]] = trunc i64 [[TMP27]] to i8
+; CHECK-NEXT:    [[TMP29:%.*]] = icmp sge i8 [[TMP28]], [[TMP24]]
+; CHECK-NEXT:    br i1 [[TMP29]], label [[TMP30:%.*]], label [[TMP31]]
+; CHECK:       30:
+; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP20]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       31:
+; CHECK-NEXT:    [[TMP32:%.*]] = load <vscale x 8 x i32>, ptr [[P]], align 32
+; CHECK-NEXT:    ret void
+;
+; CALLS-LABEL: @load.nxv8i32(
+; CALLS-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CALLS-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 256
+; CALLS-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CALLS-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CALLS-NEXT:    call void @__asan_loadN(i64 [[TMP4]], i64 [[TMP3]])
+; CALLS-NEXT:    [[TMP5:%.*]] = load <vscale x 8 x i32>, ptr [[P]], align 32
+; CALLS-NEXT:    ret void
+;
+  load <vscale x 8 x i32>, ptr %p
+  ret void
+}
+
+define void @load.nxv16i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @load.nxv16i32(
+; CHECK-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 512
+; CHECK-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CHECK-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CHECK-NEXT:    [[TMP5:%.*]] = sub i64 [[TMP3]], 1
+; CHECK-NEXT:    [[TMP6:%.*]] = add i64 [[TMP4]], [[TMP5]]
+; CHECK-NEXT:    [[TMP7:%.*]] = inttoptr i64 [[TMP6]] to ptr
+; CHECK-NEXT:    [[TMP8:%.*]] = ptrtoint ptr [[P]] to i64
+; CHECK-NEXT:    [[TMP9:%.*]] = lshr i64 [[TMP8]], 3
+; CHECK-NEXT:    [[TMP10:%.*]] = or i64 [[TMP9]], 17592186044416
+; CHECK-NEXT:    [[TMP11:%.*]] = inttoptr i64 [[TMP10]] to ptr
+; CHECK-NEXT:    [[TMP12:%.*]] = load i8, ptr [[TMP11]], align 1
+; CHECK-NEXT:    [[TMP13:%.*]] = icmp ne i8 [[TMP12]], 0
+; CHECK-NEXT:    br i1 [[TMP13]], label [[TMP14:%.*]], label [[TMP19:%.*]], !prof [[PROF0]]
+; CHECK:       14:
+; CHECK-NEXT:    [[TMP15:%.*]] = and i64 [[TMP8]], 7
+; CHECK-NEXT:    [[TMP16:%.*]] = trunc i64 [[TMP15]] to i8
+; CHECK-NEXT:    [[TMP17:%.*]] = icmp sge i8 [[TMP16]], [[TMP12]]
+; CHECK-NEXT:    br i1 [[TMP17]], label [[TMP18:%.*]], label [[TMP19]]
+; CHECK:       18:
+; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP8]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       19:
+; CHECK-NEXT:    [[TMP20:%.*]] = ptrtoint ptr [[TMP7]] to i64
+; CHECK-NEXT:    [[TMP21:%.*]] = lshr i64 [[TMP20]], 3
+; CHECK-NEXT:    [[TMP22:%.*]] = or i64 [[TMP21]], 17592186044416
+; CHECK-NEXT:    [[TMP23:%.*]] = inttoptr i64 [[TMP22]] to ptr
+; CHECK-NEXT:    [[TMP24:%.*]] = load i8, ptr [[TMP23]], align 1
+; CHECK-NEXT:    [[TMP25:%.*]] = icmp ne i8 [[TMP24]], 0
+; CHECK-NEXT:    br i1 [[TMP25]], label [[TMP26:%.*]], label [[TMP31:%.*]], !prof [[PROF0]]
+; CHECK:       26:
+; CHECK-NEXT:    [[TMP27:%.*]] = and i64 [[TMP20]], 7
+; CHECK-NEXT:    [[TMP28:%.*]] = trunc i64 [[TMP27]] to i8
+; CHECK-NEXT:    [[TMP29:%.*]] = icmp sge i8 [[TMP28]], [[TMP24]]
+; CHECK-NEXT:    br i1 [[TMP29]], label [[TMP30:%.*]], label [[TMP31]]
+; CHECK:       30:
+; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP20]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       31:
+; CHECK-NEXT:    [[TMP32:%.*]] = load <vscale x 16 x i32>, ptr [[P]], align 64
+; CHECK-NEXT:    ret void
+;
+; CALLS-LABEL: @load.nxv16i32(
+; CALLS-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CALLS-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 512
+; CALLS-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CALLS-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CALLS-NEXT:    call void @__asan_loadN(i64 [[TMP4]], i64 [[TMP3]])
+; CALLS-NEXT:    [[TMP5:%.*]] = load <vscale x 16 x i32>, ptr [[P]], align 64
+; CALLS-NEXT:    ret void
+;
+  load <vscale x 16 x i32>, ptr %p
+  ret void
+}
+
+
+define void @store.nxv1i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @store.nxv1i32(
+; CHECK-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 32
+; CHECK-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CHECK-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CHECK-NEXT:    [[TMP5:%.*]] = sub i64 [[TMP3]], 1
+; CHECK-NEXT:    [[TMP6:%.*]] = add i64 [[TMP4]], [[TMP5]]
+; CHECK-NEXT:    [[TMP7:%.*]] = inttoptr i64 [[TMP6]] to ptr
+; CHECK-NEXT:    [[TMP8:%.*]] = ptrtoint ptr [[P]] to i64
+; CHECK-NEXT:    [[TMP9:%.*]] = lshr i64 [[TMP8]], 3
+; CHECK-NEXT:    [[TMP10:%.*]] = or i64 [[TMP9]], 17592186044416
+; CHECK-NEXT:    [[TMP11:%.*]] = inttoptr i64 [[TMP10]] to ptr
+; CHECK-NEXT:    [[TMP12:%.*]] = load i8, ptr [[TMP11]], align 1
+; CHECK-NEXT:    [[TMP13:%.*]] = icmp ne i8 [[TMP12]], 0
+; CHECK-NEXT:    br i1 [[TMP13]], label [[TMP14:%.*]], label [[TMP19:%.*]], !prof [[PROF0]]
+; CHECK:       14:
+; CHECK-NEXT:    [[TMP15:%.*]] = and i64 [[TMP8]], 7
+; CHECK-NEXT:    [[TMP16:%.*]] = trunc i64 [[TMP15]] to i8
+; CHECK-NEXT:    [[TMP17:%.*]] = icmp sge i8 [[TMP16]], [[TMP12]]
+; CHECK-NEXT:    br i1 [[TMP17]], label [[TMP18:%.*]], label [[TMP19]]
+; CHECK:       18:
+; CHECK-NEXT:    call void @__asan_report_store_n(i64 [[TMP8]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       19:
+; CHECK-NEXT:    [[TMP20:%.*]] = ptrtoint ptr [[TMP7]] to i64
+; CHECK-NEXT:    [[TMP21:%.*]] = lshr i64 [[TMP20]], 3
+; CHECK-NEXT:    [[TMP22:%.*]] = or i64 [[TMP21]], 17592186044416
+; CHECK-NEXT:    [[TMP23:%.*]] = inttoptr i64 [[TMP22]] to ptr
+; CHECK-NEXT:    [[TMP24:%.*]] = load i8, ptr [[TMP23]], align 1
+; CHECK-NEXT:    [[TMP25:%.*]] = icmp ne i8 [[TMP24]], 0
+; CHECK-NEXT:    br i1 [[TMP25]], label [[TMP26:%.*]], label [[TMP31:%.*]], !prof [[PROF0]]
+; CHECK:       26:
+; CHECK-NEXT:    [[TMP27:%.*]] = and i64 [[TMP20]], 7
+; CHECK-NEXT:    [[TMP28:%.*]] = trunc i64 [[TMP27]] to i8
+; CHECK-NEXT:    [[TMP29:%.*]] = icmp sge i8 [[TMP28]], [[TMP24]]
+; CHECK-NEXT:    br i1 [[TMP29]], label [[TMP30:%.*]], label [[TMP31]]
+; CHECK:       30:
+; CHECK-NEXT:    call void @__asan_report_store_n(i64 [[TMP20]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       31:
+; CHECK-NEXT:    store <vscale x 1 x i32> zeroinitializer, ptr [[P]], align 4
+; CHECK-NEXT:    ret void
+;
+; CALLS-LABEL: @store.nxv1i32(
+; CALLS-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CALLS-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 32
+; CALLS-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CALLS-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CALLS-NEXT:    call void @__asan_storeN(i64 [[TMP4]], i64 [[TMP3]])
+; CALLS-NEXT:    store <vscale x 1 x i32> zeroinitializer, ptr [[P]], align 4
+; CALLS-NEXT:    ret void
+;
+  store <vscale x 1 x i32> zeroinitializer, ptr %p
+  ret void
+}
+
+define void @store.nxv2i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @store.nxv2i32(
+; CHECK-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 64
+; CHECK-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CHECK-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CHECK-NEXT:    [[TMP5:%.*]] = sub i64 [[TMP3]], 1
+; CHECK-NEXT:    [[TMP6:%.*]] = add i64 [[TMP4]], [[TMP5]]
+; CHECK-NEXT:    [[TMP7:%.*]] = inttoptr i64 [[TMP6]] to ptr
+; CHECK-NEXT:    [[TMP8:%.*]] = ptrtoint ptr [[P]] to i64
+; CHECK-NEXT:    [[TMP9:%.*]] = lshr i64 [[TMP8]], 3
+; CHECK-NEXT:    [[TMP10:%.*]] = or i64 [[TMP9]], 17592186044416
+; CHECK-NEXT:    [[TMP11:%.*]] = inttoptr i64 [[TMP10]] to ptr
+; CHECK-NEXT:    [[TMP12:%.*]] = load i8, ptr [[TMP11]], align 1
+; CHECK-NEXT:    [[TMP13:%.*]] = icmp ne i8 [[TMP12]], 0
+; CHECK-NEXT:    br i1 [[TMP13]], label [[TMP14:%.*]], label [[TMP19:%.*]], !prof [[PROF0]]
+; CHECK:       14:
+; CHECK-NEXT:    [[TMP15:%.*]] = and i64 [[TMP8]], 7
+; CHECK-NEXT:    [[TMP16:%.*]] = trunc i64 [[TMP15]] to i8
+; CHECK-NEXT:    [[TMP17:%.*]] = icmp sge i8 [[TMP16]], [[TMP12]]
+; CHECK-NEXT:    br i1 [[TMP17]], label [[TMP18:%.*]], label [[TMP19]]
+; CHECK:       18:
+; CHECK-NEXT:    call void @__asan_report_store_n(i64 [[TMP8]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       19:
+; CHECK-NEXT:    [[TMP20:%.*]] = ptrtoint ptr [[TMP7]] to i64
+; CHECK-NEXT:    [[TMP21:%.*]] = lshr i64 [[TMP20]], 3
+; CHECK-NEXT:    [[TMP22:%.*]] = or i64 [[TMP21]], 17592186044416
+; CHECK-NEXT:    [[TMP23:%.*]] = inttoptr i64 [[TMP22]] to ptr
+; CHECK-NEXT:    [[TMP24:%.*]] = load i8, ptr [[TMP23]], align 1
+; CHECK-NEXT:    [[TMP25:%.*]] = icmp ne i8 [[TMP24]], 0
+; CHECK-NEXT:    br i1 [[TMP25]], label [[TMP26:%.*]], label [[TMP31:%.*]], !prof [[PROF0]]
+; CHECK:       26:
+; CHECK-NEXT:    [[TMP27:%.*]] = and i64 [[TMP20]], 7
+; CHECK-NEXT:    [[TMP28:%.*]] = trunc i64 [[TMP27]] to i8
+; CHECK-NEXT:    [[TMP29:%.*]] = icmp sge i8 [[TMP28]], [[TMP24]]
+; CHECK-NEXT:    br i1 [[TMP29]], label [[TMP30:%.*]], label [[TMP31]]
+; CHECK:       30:
+; CHECK-NEXT:    call void @__asan_report_store_n(i64 [[TMP20]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       31:
+; CHECK-NEXT:    store <vscale x 2 x i32> zeroinitializer, ptr [[P]], align 8
+; CHECK-NEXT:    ret void
+;
+; CALLS-LABEL: @store.nxv2i32(
+; CALLS-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CALLS-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 64
+; CALLS-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CALLS-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CALLS-NEXT:    call void @__asan_storeN(i64 [[TMP4]], i64 [[TMP3]])
+; CALLS-NEXT:    store <vscale x 2 x i32> zeroinitializer, ptr [[P]], align 8
+; CALLS-NEXT:    ret void
+;
+  store <vscale x 2 x i32> zeroinitializer, ptr %p
+  ret void
+}
+
+define void @store.nxv4i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @store.nxv4i32(
+; CHECK-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 128
+; CHECK-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CHECK-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CHECK-NEXT:    [[TMP5:%.*]] = sub i64 [[TMP3]], 1
+; CHECK-NEXT:    [[TMP6:%.*]] = add i64 [[TMP4]], [[TMP5]]
+; CHECK-NEXT:    [[TMP7:%.*]] = inttoptr i64 [[TMP6]] to ptr
+; CHECK-NEXT:    [[TMP8:%.*]] = ptrtoint ptr [[P]] to i64
+; CHECK-NEXT:    [[TMP9:%.*]] = lshr i64 [[TMP8]], 3
+; CHECK-NEXT:    [[TMP10:%.*]] = or i64 [[TMP9]], 17592186044416
+; CHECK-NEXT:    [[TMP11:%.*]] = inttoptr i64 [[TMP10]] to ptr
+; CHECK-NEXT:    [[TMP12:%.*]] = load i8, ptr [[TMP11]], align 1
+; CHECK-NEXT:    [[TMP13:%.*]] = icmp ne i8 [[TMP12]], 0
+; CHECK-NEXT:    br i1 [[TMP13]], label [[TMP14:%.*]], label [[TMP19:%.*]], !prof [[PROF0]]
+; CHECK:       14:
+; CHECK-NEXT:    [[TMP15:%.*]] = and i64 [[TMP8]], 7
+; CHECK-NEXT:    [[TMP16:%.*]] = trunc i64 [[TMP15]] to i8
+; CHECK-NEXT:    [[TMP17:%.*]] = icmp sge i8 [[TMP16]], [[TMP12]]
+; CHECK-NEXT:    br i1 [[TMP17]], label [[TMP18:%.*]], label [[TMP19]]
+; CHECK:       18:
+; CHECK-NEXT:    call void @__asan_report_store_n(i64 [[TMP8]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       19:
+; CHECK-NEXT:    [[TMP20:%.*]] = ptrtoint ptr [[TMP7]] to i64
+; CHECK-NEXT:    [[TMP21:%.*]] = lshr i64 [[TMP20]], 3
+; CHECK-NEXT:    [[TMP22:%.*]] = or i64 [[TMP21]], 17592186044416
+; CHECK-NEXT:    [[TMP23:%.*]] = inttoptr i64 [[TMP22]] to ptr
+; CHECK-NEXT:    [[TMP24:%.*]] = load i8, ptr [[TMP23]], align 1
+; CHECK-NEXT:    [[TMP25:%.*]] = icmp ne i8 [[TMP24]], 0
+; CHECK-NEXT:    br i1 [[TMP25]], label [[TMP26:%.*]], label [[TMP31:%.*]], !prof [[PROF0]]
+; CHECK:       26:
+; CHECK-NEXT:    [[TMP27:%.*]] = and i64 [[TMP20]], 7
+; CHECK-NEXT:    [[TMP28:%.*]] = trunc i64 [[TMP27]] to i8
+; CHECK-NEXT:    [[TMP29:%.*]] = icmp sge i8 [[TMP28]], [[TMP24]]
+; CHECK-NEXT:    br i1 [[TMP29]], label [[TMP30:%.*]], label [[TMP31]]
+; CHECK:       30:
+; CHECK-NEXT:    call void @__asan_report_store_n(i64 [[TMP20]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       31:
+; CHECK-NEXT:    store <vscale x 4 x i32> zeroinitializer, ptr [[P]], align 16
+; CHECK-NEXT:    ret void
+;
+; CALLS-LABEL: @store.nxv4i32(
+; CALLS-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CALLS-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 128
+; CALLS-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CALLS-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CALLS-NEXT:    call void @__asan_storeN(i64 [[TMP4]], i64 [[TMP3]])
+; CALLS-NEXT:    store <vscale x 4 x i32> zeroinitializer, ptr [[P]], align 16
+; CALLS-NEXT:    ret void
+;
+  store <vscale x 4 x i32> zeroinitializer, ptr %p
+  ret void
+}
+
+define void @store.nxv8i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @store.nxv8i32(
+; CHECK-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 256
+; CHECK-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CHECK-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CHECK-NEXT:    [[TMP5:%.*]] = sub i64 [[TMP3]], 1
+; CHECK-NEXT:    [[TMP6:%.*]] = add i64 [[TMP4]], [[TMP5]]
+; CHECK-NEXT:    [[TMP7:%.*]] = inttoptr i64 [[TMP6]] to ptr
+; CHECK-NEXT:    [[TMP8:%.*]] = ptrtoint ptr [[P]] to i64
+; CHECK-NEXT:    [[TMP9:%.*]] = lshr i64 [[TMP8]], 3
+; CHECK-NEXT:    [[TMP10:%.*]] = or i64 [[TMP9]], 17592186044416
+; CHECK-NEXT:    [[TMP11:%.*]] = inttoptr i64 [[TMP10]] to ptr
+; CHECK-NEXT:    [[TMP12:%.*]] = load i8, ptr [[TMP11]], align 1
+; CHECK-NEXT:    [[TMP13:%.*]] = icmp ne i8 [[TMP12]], 0
+; CHECK-NEXT:    br i1 [[TMP13]], label [[TMP14:%.*]], label [[TMP19:%.*]], !prof [[PROF0]]
+; CHECK:       14:
+; CHECK-NEXT:    [[TMP15:%.*]] = and i64 [[TMP8]], 7
+; CHECK-NEXT:    [[TMP16:%.*]] = trunc i64 [[TMP15]] to i8
+; CHECK-NEXT:    [[TMP17:%.*]] = icmp sge i8 [[TMP16]], [[TMP12]]
+; CHECK-NEXT:    br i1 [[TMP17]], label [[TMP18:%.*]], label [[TMP19]]
+; CHECK:       18:
+; CHECK-NEXT:    call void @__asan_report_store_n(i64 [[TMP8]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       19:
+; CHECK-NEXT:    [[TMP20:%.*]] = ptrtoint ptr [[TMP7]] to i64
+; CHECK-NEXT:    [[TMP21:%.*]] = lshr i64 [[TMP20]], 3
+; CHECK-NEXT:    [[TMP22:%.*]] = or i64 [[TMP21]], 17592186044416
+; CHECK-NEXT:    [[TMP23:%.*]] = inttoptr i64 [[TMP22]] to ptr
+; CHECK-NEXT:    [[TMP24:%.*]] = load i8, ptr [[TMP23]], align 1
+; CHECK-NEXT:    [[TMP25:%.*]] = icmp ne i8 [[TMP24]], 0
+; CHECK-NEXT:    br i1 [[TMP25]], label [[TMP26:%.*]], label [[TMP31:%.*]], !prof [[PROF0]]
+; CHECK:       26:
+; CHECK-NEXT:    [[TMP27:%.*]] = and i64 [[TMP20]], 7
+; CHECK-NEXT:    [[TMP28:%.*]] = trunc i64 [[TMP27]] to i8
+; CHECK-NEXT:    [[TMP29:%.*]] = icmp sge i8 [[TMP28]], [[TMP24]]
+; CHECK-NEXT:    br i1 [[TMP29]], label [[TMP30:%.*]], label [[TMP31]]
+; CHECK:       30:
+; CHECK-NEXT:    call void @__asan_report_store_n(i64 [[TMP20]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       31:
+; CHECK-NEXT:    store <vscale x 8 x i32> zeroinitializer, ptr [[P]], align 32
+; CHECK-NEXT:    ret void
+;
+; CALLS-LABEL: @store.nxv8i32(
+; CALLS-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CALLS-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 256
+; CALLS-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CALLS-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CALLS-NEXT:    call void @__asan_storeN(i64 [[TMP4]], i64 [[TMP3]])
+; CALLS-NEXT:    store <vscale x 8 x i32> zeroinitializer, ptr [[P]], align 32
+; CALLS-NEXT:    ret void
+;
+  store <vscale x 8 x i32> zeroinitializer, ptr %p
+  ret void
+}
+
+define void @store.nxv16i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @store.nxv16i32(
+; CHECK-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 512
+; CHECK-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CHECK-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CHECK-NEXT:    [[TMP5:%.*]] = sub i64 [[TMP3]], 1
+; CHECK-NEXT:    [[TMP6:%.*]] = add i64 [[TMP4]], [[TMP5]]
+; CHECK-NEXT:    [[TMP7:%.*]] = inttoptr i64 [[TMP6]] to ptr
+; CHECK-NEXT:    [[TMP8:%.*]] = ptrtoint ptr [[P]] to i64
+; CHECK-NEXT:    [[TMP9:%.*]] = lshr i64 [[TMP8]], 3
+; CHECK-NEXT:    [[TMP10:%.*]] = or i64 [[TMP9]], 17592186044416
+; CHECK-NEXT:    [[TMP11:%.*]] = inttoptr i64 [[TMP10]] to ptr
+; CHECK-NEXT:    [[TMP12:%.*]] = load i8, ptr [[TMP11]], align 1
+; CHECK-NEXT:    [[TMP13:%.*]] = icmp ne i8 [[TMP12]], 0
+; CHECK-NEXT:    br i1 [[TMP13]], label [[TMP14:%.*]], label [[TMP19:%.*]], !prof [[PROF0]]
+; CHECK:       14:
+; CHECK-NEXT:    [[TMP15:%.*]] = and i64 [[TMP8]], 7
+; CHECK-NEXT:    [[TMP16:%.*]] = trunc i64 [[TMP15]] to i8
+; CHECK-NEXT:    [[TMP17:%.*]] = icmp sge i8 [[TMP16]], [[TMP12]]
+; CHECK-NEXT:    br i1 [[TMP17]], label [[TMP18:%.*]], label [[TMP19]]
+; CHECK:       18:
+; CHECK-NEXT:    call void @__asan_report_store_n(i64 [[TMP8]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       19:
+; CHECK-NEXT:    [[TMP20:%.*]] = ptrtoint ptr [[TMP7]] to i64
+; CHECK-NEXT:    [[TMP21:%.*]] = lshr i64 [[TMP20]], 3
+; CHECK-NEXT:    [[TMP22:%.*]] = or i64 [[TMP21]], 17592186044416
+; CHECK-NEXT:    [[TMP23:%.*]] = inttoptr i64 [[TMP22]] to ptr
+; CHECK-NEXT:    [[TMP24:%.*]] = load i8, ptr [[TMP23]], align 1
+; CHECK-NEXT:    [[TMP25:%.*]] = icmp ne i8 [[TMP24]], 0
+; CHECK-NEXT:    br i1 [[TMP25]], label [[TMP26:%.*]], label [[TMP31:%.*]], !prof [[PROF0]]
+; CHECK:       26:
+; CHECK-NEXT:    [[TMP27:%.*]] = and i64 [[TMP20]], 7
+; CHECK-NEXT:    [[TMP28:%.*]] = trunc i64 [[TMP27]] to i8
+; CHECK-NEXT:    [[TMP29:%.*]] = icmp sge i8 [[TMP28]], [[TMP24]]
+; CHECK-NEXT:    br i1 [[TMP29]], label [[TMP30:%.*]], label [[TMP31]]
+; CHECK:       30:
+; CHECK-NEXT:    call void @__asan_report_store_n(i64 [[TMP20]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       31:
+; CHECK-NEXT:    store <vscale x 16 x i32> zeroinitializer, ptr [[P]], align 64
+; CHECK-NEXT:    ret void
+;
+; CALLS-LABEL: @store.nxv16i32(
+; CALLS-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CALLS-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 512
+; CALLS-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CALLS-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CALLS-NEXT:    call void @__asan_storeN(i64 [[TMP4]], i64 [[TMP3]])
+; CALLS-NEXT:    store <vscale x 16 x i32> zeroinitializer, ptr [[P]], align 64
+; CALLS-NEXT:    ret void
+;
+  store <vscale x 16 x i32> zeroinitializer, ptr %p
+  ret void
+}
+
+declare void @clobber(ptr)
+
+define <vscale x 2 x i32> @local_alloca() sanitize_address {
+; CHECK-LABEL: @local_alloca(
+; CHECK-NEXT:    [[A:%.*]] = alloca <vscale x 2 x i32>, align 8
+; CHECK-NEXT:    call void @clobber(ptr [[A]])
+; CHECK-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 64
+; CHECK-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CHECK-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[A]] to i64
+; CHECK-NEXT:    [[TMP5:%.*]] = sub i64 [[TMP3]], 1
+; CHECK-NEXT:    [[TMP6:%.*]] = add i64 [[TMP4]], [[TMP5]]
+; CHECK-NEXT:    [[TMP7:%.*]] = inttoptr i64 [[TMP6]] to ptr
+; CHECK-NEXT:    [[TMP8:%.*]] = ptrtoint ptr [[A]] to i64
+; CHECK-NEXT:    [[TMP9:%.*]] = lshr i64 [[TMP8]], 3
+; CHECK-NEXT:    [[TMP10:%.*]] = or i64 [[TMP9]], 17592186044416
+; CHECK-NEXT:    [[TMP11:%.*]] = inttoptr i64 [[TMP10]] to ptr
+; CHECK-NEXT:    [[TMP12:%.*]] = load i8, ptr [[TMP11]], align 1
+; CHECK-NEXT:    [[TMP13:%.*]] = icmp ne i8 [[TMP12]], 0
+; CHECK-NEXT:    br i1 [[TMP13]], label [[TMP14:%.*]], label [[TMP19:%.*]], !prof [[PROF0]]
+; CHECK:       14:
+; CHECK-NEXT:    [[TMP15:%.*]] = and i64 [[TMP8]], 7
+; CHECK-NEXT:    [[TMP16:%.*]] = trunc i64 [[TMP15]] to i8
+; CHECK-NEXT:    [[TMP17:%.*]] = icmp sge i8 [[TMP16]], [[TMP12]]
+; CHECK-NEXT:    br i1 [[TMP17]], label [[TMP18:%.*]], label [[TMP19]]
+; CHECK:       18:
+; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP8]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       19:
+; CHECK-NEXT:    [[TMP20:%.*]] = ptrtoint ptr [[TMP7]] to i64
+; CHECK-NEXT:    [[TMP21:%.*]] = lshr i64 [[TMP20]], 3
+; CHECK-NEXT:    [[TMP22:%.*]] = or i64 [[TMP21]], 17592186044416
+; CHECK-NEXT:    [[TMP23:%.*]] = inttoptr i64 [[TMP22]] to ptr
+; CHECK-NEXT:    [[TMP24:%.*]] = load i8, ptr [[TMP23]], align 1
+; CHECK-NEXT:    [[TMP25:%.*]] = icmp ne i8 [[TMP24]], 0
+; CHECK-NEXT:    br i1 [[TMP25]], label [[TMP26:%.*]], label [[TMP31:%.*]], !prof [[PROF0]]
+; CHECK:       26:
+; CHECK-NEXT:    [[TMP27:%.*]] = and i64 [[TMP20]], 7
+; CHECK-NEXT:    [[TMP28:%.*]] = trunc i64 [[TMP27]] to i8
+; CHECK-NEXT:    [[TMP29:%.*]] = icmp sge i8 [[TMP28]], [[TMP24]]
+; CHECK-NEXT:    br i1 [[TMP29]], label [[TMP30:%.*]], label [[TMP31]]
+; CHECK:       30:
+; CHECK-NEXT:    call void @__asan_report_load_n(i64 [[TMP20]], i64 [[TMP3]]) #[[ATTR4]]
+; CHECK-NEXT:    unreachable
+; CHECK:       31:
+; CHECK-NEXT:    [[RES:%.*]] = load <vscale x 2 x i32>, ptr [[A]], align 8
+; CHECK-NEXT:    ret <vscale x 2 x i32> [[RES]]
+;
+; CALLS-LABEL: @local_alloca(
+; CALLS-NEXT:    [[A:%.*]] = alloca <vscale x 2 x i32>, align 8
+; CALLS-NEXT:    call void @clobber(ptr [[A]])
+; CALLS-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CALLS-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 64
+; CALLS-NEXT:    [[TMP3:%.*]] = lshr i64 [[TMP2]], 3
+; CALLS-NEXT:    [[TMP4:%.*]] = ptrtoint ptr [[A]] to i64
+; CALLS-NEXT:    call void @__asan_loadN(i64 [[TMP4]], i64 [[TMP3]])
+; CALLS-NEXT:    [[RES:%.*]] = load <vscale x 2 x i32>, ptr [[A]], align 8
+; CALLS-NEXT:    ret <vscale x 2 x i32> [[RES]]
+;
+  %a = alloca <vscale x 2 x i32>
+  call void @clobber(ptr %a)
+  %res = load <vscale x 2 x i32>, ptr %a
+  ret <vscale x 2 x i32> %res
+}
diff --git a/llvm/test/Instrumentation/MemorySanitizer/vector-load-store.ll b/llvm/test/Instrumentation/MemorySanitizer/vector-load-store.ll
new file mode 100644
index 0000000000000..52c60e9b83b29
--- /dev/null
+++ b/llvm/test/Instrumentation/MemorySanitizer/vector-load-store.ll
@@ -0,0 +1,280 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -msan-check-access-address=0 -S -passes=msan 2>&1 | FileCheck %s --implicit-check-not="call void @__msan_warning"
+; RUN: opt < %s -msan-check-access-address=1 -S -passes=msan 2>&1 | FileCheck %s --check-prefixes=ADDR --implicit-check-not="call void @__msan_warning"
+; RUN: opt < %s -msan-check-access-address=0 -msan-track-origins=1 -S -passes=msan 2>&1 | FileCheck %s --check-prefixes=ORIGINS --implicit-check-not="call void @__msan_warning"
+
+target triple = "x86_64-unknown-linux-gnu"
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+
+define void @load.v1i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @load.v1i32(
+; CHECK-NEXT:    call void @llvm.donothing()
+; CHECK-NEXT:    [[TMP1:%.*]] = load <1 x i32>, ptr [[P:%.*]], align 4
+; CHECK-NEXT:    ret void
+;
+; ADDR-LABEL: @load.v1i32(
+; ADDR-NEXT:    call void @llvm.donothing()
+; ADDR-NEXT:    [[TMP1:%.*]] = load <1 x i32>, ptr [[P:%.*]], align 4
+; ADDR-NEXT:    ret void
+;
+; ORIGINS-LABEL: @load.v1i32(
+; ORIGINS-NEXT:    call void @llvm.donothing()
+; ORIGINS-NEXT:    [[TMP1:%.*]] = load <1 x i32>, ptr [[P:%.*]], align 4
+; ORIGINS-NEXT:    ret void
+;
+  load <1 x i32>, ptr %p
+  ret void
+}
+
+define void @load.v2i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @load.v2i32(
+; CHECK-NEXT:    call void @llvm.donothing()
+; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x i32>, ptr [[P:%.*]], align 8
+; CHECK-NEXT:    ret void
+;
+; ADDR-LABEL: @load.v2i32(
+; ADDR-NEXT:    call void @llvm.donothing()
+; ADDR-NEXT:    [[TMP1:%.*]] = load <2 x i32>, ptr [[P:%.*]], align 8
+; ADDR-NEXT:    ret void
+;
+; ORIGINS-LABEL: @load.v2i32(
+; ORIGINS-NEXT:    call void @llvm.donothing()
+; ORIGINS-NEXT:    [[TMP1:%.*]] = load <2 x i32>, ptr [[P:%.*]], align 8
+; ORIGINS-NEXT:    ret void
+;
+  load <2 x i32>, ptr %p
+  ret void
+}
+
+define void @load.v4i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @load.v4i32(
+; CHECK-NEXT:    call void @llvm.donothing()
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, ptr [[P:%.*]], align 16
+; CHECK-NEXT:    ret void
+;
+; ADDR-LABEL: @load.v4i32(
+; ADDR-NEXT:    call void @llvm.donothing()
+; ADDR-NEXT:    [[TMP1:%.*]] = load <4 x i32>, ptr [[P:%.*]], align 16
+; ADDR-NEXT:    ret void
+;
+; ORIGINS-LABEL: @load.v4i32(
+; ORIGINS-NEXT:    call void @llvm.donothing()
+; ORIGINS-NEXT:    [[TMP1:%.*]] = load <4 x i32>, ptr [[P:%.*]], align 16
+; ORIGINS-NEXT:    ret void
+;
+  load <4 x i32>, ptr %p
+  ret void
+}
+
+define void @load.v8i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @load.v8i32(
+; CHECK-NEXT:    call void @llvm.donothing()
+; CHECK-NEXT:    [[TMP1:%.*]] = load <8 x i32>, ptr [[P:%.*]], align 32
+; CHECK-NEXT:    ret void
+;
+; ADDR-LABEL: @load.v8i32(
+; ADDR-NEXT:    call void @llvm.donothing()
+; ADDR-NEXT:    [[TMP1:%.*]] = load <8 x i32>, ptr [[P:%.*]], align 32
+; ADDR-NEXT:    ret void
+;
+; ORIGINS-LABEL: @load.v8i32(
+; ORIGINS-NEXT:    call void @llvm.donothing()
+; ORIGINS-NEXT:    [[TMP1:%.*]] = load <8 x i32>, ptr [[P:%.*]], align 32
+; ORIGINS-NEXT:    ret void
+;
+  load <8 x i32>, ptr %p
+  ret void
+}
+
+define void @load.v16i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @load.v16i32(
+; CHECK-NEXT:    call void @llvm.donothing()
+; CHECK-NEXT:    [[TMP1:%.*]] = load <16 x i32>, ptr [[P:%.*]], align 64
+; CHECK-NEXT:    ret void
+;
+; ADDR-LABEL: @load.v16i32(
+; ADDR-NEXT:    call void @llvm.donothing()
+; ADDR-NEXT:    [[TMP1:%.*]] = load <16 x i32>, ptr [[P:%.*]], align 64
+; ADDR-NEXT:    ret void
+;
+; ORIGINS-LABEL: @load.v16i32(
+; ORIGINS-NEXT:    call void @llvm.donothing()
+; ORIGINS-NEXT:    [[TMP1:%.*]] = load <16 x i32>, ptr [[P:%.*]], align 64
+; ORIGINS-NEXT:    ret void
+;
+  load <16 x i32>, ptr %p
+  ret void
+}
+
+
+define void @store.v1i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @store.v1i32(
+; CHECK-NEXT:    call void @llvm.donothing()
+; CHECK-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CHECK-NEXT:    [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
+; CHECK-NEXT:    [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
+; CHECK-NEXT:    store <1 x i32> zeroinitializer, ptr [[TMP3]], align 4
+; CHECK-NEXT:    store <1 x i32> zeroinitializer, ptr [[P]], align 4
+; CHECK-NEXT:    ret void
+;
+; ADDR-LABEL: @store.v1i32(
+; ADDR-NEXT:    call void @llvm.donothing()
+; ADDR-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; ADDR-NEXT:    [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
+; ADDR-NEXT:    [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
+; ADDR-NEXT:    store <1 x i32> zeroinitializer, ptr [[TMP3]], align 4
+; ADDR-NEXT:    store <1 x i32> zeroinitializer, ptr [[P]], align 4
+; ADDR-NEXT:    ret void
+;
+; ORIGINS-LABEL: @store.v1i32(
+; ORIGINS-NEXT:    call void @llvm.donothing()
+; ORIGINS-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; ORIGINS-NEXT:    [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
+; ORIGINS-NEXT:    [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
+; ORIGINS-NEXT:    [[TMP4:%.*]] = add i64 [[TMP2]], 17592186044416
+; ORIGINS-NEXT:    [[TMP5:%.*]] = inttoptr i64 [[TMP4]] to ptr
+; ORIGINS-NEXT:    store <1 x i32> zeroinitializer, ptr [[TMP3]], align 4
+; ORIGINS-NEXT:    store <1 x i32> zeroinitializer, ptr [[P]], align 4
+; ORIGINS-NEXT:    ret void
+;
+  store <1 x i32> zeroinitializer, ptr %p
+  ret void
+}
+
+define void @store.v2i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @store.v2i32(
+; CHECK-NEXT:    call void @llvm.donothing()
+; CHECK-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CHECK-NEXT:    [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
+; CHECK-NEXT:    [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
+; CHECK-NEXT:    store <2 x i32> zeroinitializer, ptr [[TMP3]], align 8
+; CHECK-NEXT:    store <2 x i32> zeroinitializer, ptr [[P]], align 8
+; CHECK-NEXT:    ret void
+;
+; ADDR-LABEL: @store.v2i32(
+; ADDR-NEXT:    call void @llvm.donothing()
+; ADDR-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; ADDR-NEXT:    [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
+; ADDR-NEXT:    [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
+; ADDR-NEXT:    store <2 x i32> zeroinitializer, ptr [[TMP3]], align 8
+; ADDR-NEXT:    store <2 x i32> zeroinitializer, ptr [[P]], align 8
+; ADDR-NEXT:    ret void
+;
+; ORIGINS-LABEL: @store.v2i32(
+; ORIGINS-NEXT:    call void @llvm.donothing()
+; ORIGINS-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; ORIGINS-NEXT:    [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
+; ORIGINS-NEXT:    [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
+; ORIGINS-NEXT:    [[TMP4:%.*]] = add i64 [[TMP2]], 17592186044416
+; ORIGINS-NEXT:    [[TMP5:%.*]] = inttoptr i64 [[TMP4]] to ptr
+; ORIGINS-NEXT:    store <2 x i32> zeroinitializer, ptr [[TMP3]], align 8
+; ORIGINS-NEXT:    store <2 x i32> zeroinitializer, ptr [[P]], align 8
+; ORIGINS-NEXT:    ret void
+;
+  store <2 x i32> zeroinitializer, ptr %p
+  ret void
+}
+
+define void @store.v4i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @store.v4i32(
+; CHECK-NEXT:    call void @llvm.donothing()
+; CHECK-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CHECK-NEXT:    [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
+; CHECK-NEXT:    [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
+; CHECK-NEXT:    store <4 x i32> zeroinitializer, ptr [[TMP3]], align 16
+; CHECK-NEXT:    store <4 x i32> zeroinitializer, ptr [[P]], align 16
+; CHECK-NEXT:    ret void
+;
+; ADDR-LABEL: @store.v4i32(
+; ADDR-NEXT:    call void @llvm.donothing()
+; ADDR-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; ADDR-NEXT:    [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
+; ADDR-NEXT:    [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
+; ADDR-NEXT:    store <4 x i32> zeroinitializer, ptr [[TMP3]], align 16
+; ADDR-NEXT:    store <4 x i32> zeroinitializer, ptr [[P]], align 16
+; ADDR-NEXT:    ret void
+;
+; ORIGINS-LABEL: @store.v4i32(
+; ORIGINS-NEXT:    call void @llvm.donothing()
+; ORIGINS-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; ORIGINS-NEXT:    [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
+; ORIGINS-NEXT:    [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
+; ORIGINS-NEXT:    [[TMP4:%.*]] = add i64 [[TMP2]], 17592186044416
+; ORIGINS-NEXT:    [[TMP5:%.*]] = inttoptr i64 [[TMP4]] to ptr
+; ORIGINS-NEXT:    store <4 x i32> zeroinitializer, ptr [[TMP3]], align 16
+; ORIGINS-NEXT:    store <4 x i32> zeroinitializer, ptr [[P]], align 16
+; ORIGINS-NEXT:    ret void
+;
+  store <4 x i32> zeroinitializer, ptr %p
+  ret void
+}
+
+define void @store.v8i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @store.v8i32(
+; CHECK-NEXT:    call void @llvm.donothing()
+; CHECK-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CHECK-NEXT:    [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
+; CHECK-NEXT:    [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
+; CHECK-NEXT:    store <8 x i32> zeroinitializer, ptr [[TMP3]], align 32
+; CHECK-NEXT:    store <8 x i32> zeroinitializer, ptr [[P]], align 32
+; CHECK-NEXT:    ret void
+;
+; ADDR-LABEL: @store.v8i32(
+; ADDR-NEXT:    call void @llvm.donothing()
+; ADDR-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; ADDR-NEXT:    [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
+; ADDR-NEXT:    [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
+; ADDR-NEXT:    store <8 x i32> zeroinitializer, ptr [[TMP3]], align 32
+; ADDR-NEXT:    store <8 x i32> zeroinitializer, ptr [[P]], align 32
+; ADDR-NEXT:    ret void
+;
+; ORIGINS-LABEL: @store.v8i32(
+; ORIGINS-NEXT:    call void @llvm.donothing()
+; ORIGINS-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; ORIGINS-NEXT:    [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
+; ORIGINS-NEXT:    [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
+; ORIGINS-NEXT:    [[TMP4:%.*]] = add i64 [[TMP2]], 17592186044416
+; ORIGINS-NEXT:    [[TMP5:%.*]] = inttoptr i64 [[TMP4]] to ptr
+; ORIGINS-NEXT:    store <8 x i32> zeroinitializer, ptr [[TMP3]], align 32
+; ORIGINS-NEXT:    store <8 x i32> zeroinitializer, ptr [[P]], align 32
+; ORIGINS-NEXT:    ret void
+;
+  store <8 x i32> zeroinitializer, ptr %p
+  ret void
+}
+
+define void @store.v16i32(ptr %p) sanitize_address {
+; CHECK-LABEL: @store.v16i32(
+; CHECK-NEXT:    call void @llvm.donothing()
+; CHECK-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CHECK-NEXT:    [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
+; CHECK-NEXT:    [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
+; CHECK-NEXT:    store <16 x i32> zeroinitializer, ptr [[TMP3]], align 64
+; CHECK-NEXT:    store <16 x i32> zeroinitializer, ptr [[P]], align 64
+; CHECK-NEXT:    ret void
+;
+; ADDR-LABEL: @store.v16i32(
+; ADDR-NEXT:    call void @llvm.donothing()
+; ADDR-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; ADDR-NEXT:    [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
+; ADDR-NEXT:    [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
+; ADDR-NEXT:    store <16 x i32> zeroinitializer, ptr [[TMP3]], align 64
+; ADDR-NEXT:    store <16 x i32> zeroinitializer, ptr [[P]], align 64
+; ADDR-NEXT:    ret void
+;
+; ORIGINS-LABEL: @store.v16i32(
+; ORIGINS-NEXT:    call void @llvm.donothing()
+; ORIGINS-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; ORIGINS-NEXT:    [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
+; ORIGINS-NEXT:    [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
+; ORIGINS-NEXT:    [[TMP4:%.*]] = add i64 [[TMP2]], 17592186044416
+; ORIGINS-NEXT:    [[TMP5:%.*]] = inttoptr i64 [[TMP4]] to ptr
+; ORIGINS-NEXT:    store <16 x i32> zeroinitializer, ptr [[TMP3]], align 64
+; ORIGINS-NEXT:    store <16 x i32> zeroinitializer, ptr [[P]], align 64
+; ORIGINS-NEXT:    ret void
+;
+  store <16 x i32> zeroinitializer, ptr %p
+  ret void
+}
+
+
diff --git a/llvm/test/LTO/X86/strip-debug-info.ll b/llvm/test/LTO/X86/strip-debug-info.ll
index 321bd556e07ab..c478f641eba85 100644
--- a/llvm/test/LTO/X86/strip-debug-info.ll
+++ b/llvm/test/LTO/X86/strip-debug-info.ll
@@ -24,10 +24,18 @@
 ; RUN:     -exported-symbol foo -exported-symbol _foo \
 ; RUN:     %t-stripped.bc -disable-verify 2>&1 | \
 ; RUN:     FileCheck %s -allow-empty -check-prefix=CHECK-WARN
+; ---- Thin LTO (optimize, don't strip imported file)
+; RUN: llvm-lto -thinlto-action=thinlink -o %t.index.bc %t-stripped.bc %t2.bc
+; RUN: llvm-lto -thinlto -thinlto-action=import -disable-auto-upgrade-debug-info \
+; RUN:     -thinlto-index=%t.index.bc \
+; RUN:     -exported-symbol foo -exported-symbol _foo \
+; RUN:     %t-stripped.bc -disable-verify 2>&1 | \
+; RUN:     FileCheck %s -allow-empty -check-prefix=CHECK-NO-WARN
 
 ; CHECK-WARN: warning{{.*}} ignoring invalid debug info
 ; CHECK-WARN-NOT: Broken module found
 ; CHECK: foo
+; CHECK-NO-WARN-NOT: ignoring invalid debug info
 target datalayout = "e-m:o-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-apple-macosx10.12"
 
diff --git a/llvm/test/Linker/Inputs/metadata-function.ll b/llvm/test/Linker/Inputs/metadata-function.ll
index 8572ff1198372..4109c57c14e12 100644
--- a/llvm/test/Linker/Inputs/metadata-function.ll
+++ b/llvm/test/Linker/Inputs/metadata-function.ll
@@ -10,4 +10,13 @@ define void @b() !b !0 {
   unreachable
 }
 
+%AltHandle = type { i8* }
+declare !types !1 %AltHandle @init.AltHandle()
+
+define void @uses.AltHandle() {
+  %.res = call %AltHandle @init.AltHandle()
+  unreachable
+}
+
 !0 = !{!"b"}
+!1 = !{%AltHandle undef}
diff --git a/llvm/test/Linker/Inputs/nofpclass.ll b/llvm/test/Linker/Inputs/nofpclass.ll
new file mode 100644
index 0000000000000..e921da6a30fcb
--- /dev/null
+++ b/llvm/test/Linker/Inputs/nofpclass.ll
@@ -0,0 +1,4 @@
+define float @declared_as_nonan(float %arg) {
+  %add = fadd float %arg, 1.0
+  ret float %add
+}
diff --git a/llvm/test/Linker/metadata-function.ll b/llvm/test/Linker/metadata-function.ll
index 0b17e0c58588f..3fe3b524b145f 100644
--- a/llvm/test/Linker/metadata-function.ll
+++ b/llvm/test/Linker/metadata-function.ll
@@ -21,6 +21,15 @@ define void @a() !a !0 {
   unreachable
 }
 
+; CHECK-DAG: define %[[HandleType:[A-Za-z]+]] @init.Handle() {
+; CHECK-DAG: declare !types ![[C:[0-9]+]] %[[HandleType]] @init.AltHandle()
+; CHECK-DAG: define void @uses.AltHandle() {
+%Handle = type { i8* }
+define %Handle @init.Handle() {
+  unreachable
+}
+
 ; CHECK-DAG: ![[A]] = !{!"a"}
 ; CHECK-DAG: ![[B]] = !{!"b"}
+; CHECK-DAG: ![[C]] = !{%[[HandleType]] undef}
 !0 = !{!"a"}
diff --git a/llvm/test/Linker/nofpclass.ll b/llvm/test/Linker/nofpclass.ll
new file mode 100644
index 0000000000000..a43a5fdcd19f5
--- /dev/null
+++ b/llvm/test/Linker/nofpclass.ll
@@ -0,0 +1,30 @@
+; RUN: llvm-link %s %S/Inputs/nofpclass.ll -S -o - | FileCheck -check-prefix=ORDER1 %s
+; RUN: llvm-link %S/Inputs/nofpclass.ll %s -S -o - | FileCheck -check-prefix=ORDER2 %s
+
+; Make sure nofpclass is dropped if the function was declared as
+; nofpclass, but not defined with nofpclass.
+
+; ORDER1: define float @caller(float %arg) {
+; ORDER1-NEXT: %result = call float @declared_as_nonan(float %arg)
+; ORDER1-NEXT: ret float %result
+
+; ORDER1: define float @declared_as_nonan(float %arg) {
+; ORDER1-NEXT: %add = fadd float %arg, 1.000000e+00
+; ORDER1-NEXT: ret float %add
+
+
+; ORDER2: define float @declared_as_nonan(float %arg) {
+; ORDER2-NEXT: %add = fadd float %arg, 1.000000e+00
+; ORDER2-NEXT: ret float %add
+
+; ORDER2: define float @caller(float %arg) {
+; ORDER2-NEXT: %result = call float @declared_as_nonan(float %arg)
+; ORDER2-NEXT: ret float %result
+
+
+declare nofpclass(nan) float @declared_as_nonan(float nofpclass(nan))
+
+define float @caller(float %arg) {
+  %result = call float @declared_as_nonan(float %arg)
+  ret float %result
+}
diff --git a/llvm/test/MC/AArch64/armv9.4a-chk.s b/llvm/test/MC/AArch64/armv9.4a-chk.s
new file mode 100644
index 0000000000000..95acee370dcb1
--- /dev/null
+++ b/llvm/test/MC/AArch64/armv9.4a-chk.s
@@ -0,0 +1,21 @@
+// RUN: llvm-mc -triple aarch64 -mattr=+chk -show-encoding %s | FileCheck %s
+// RUN: llvm-mc -triple aarch64 -mattr=+v8.9a -show-encoding %s | FileCheck %s
+// RUN: llvm-mc -triple aarch64 -mattr=+v9.4a -show-encoding %s | FileCheck %s
+// RUN: llvm-mc -triple aarch64 -mattr=+v8a -show-encoding %s | FileCheck %s --check-prefix=NO-CHK
+
+// FEAT_CHK is mandatory from v8.0-a, but a clang user may not be using the LLVM
+// integrated assembler, so we cannot just print `chkfeat x16` in all
+// circumstances. Thankfully, we can always print `hint #40` when we cannot
+// print `chkfeat x16`.
+//
+// So, in this case, we only print `chkfeat x16` from v8.9-a onwards, as an
+// assembler that understands v8.9-a will understand `chkfeat x16`, and those
+// that understand previous versions may not.
+
+chkfeat x16
+// CHECK: chkfeat x16                       // encoding: [0x1f,0x25,0x03,0xd5]
+// NO-CHK: hint #40                              // encoding: [0x1f,0x25,0x03,0xd5]
+
+hint #40
+// CHECK: chkfeat x16                      // encoding: [0x1f,0x25,0x03,0xd5]
+// NO-CHK: hint #40                             // encoding: [0x1f,0x25,0x03,0xd5]
diff --git a/llvm/test/MC/AArch64/armv9.4a-gcs.s b/llvm/test/MC/AArch64/armv9.4a-gcs.s
new file mode 100644
index 0000000000000..8910229b8dace
--- /dev/null
+++ b/llvm/test/MC/AArch64/armv9.4a-gcs.s
@@ -0,0 +1,115 @@
+// RUN: llvm-mc -triple aarch64 -mattr +gcs -show-encoding %s | FileCheck %s
+// RUN: not llvm-mc -triple aarch64 -show-encoding %s 2>%t | FileCheck %s --check-prefix=NO-GCS
+// RUN: FileCheck --check-prefix=ERROR-NO-GCS %s < %t
+
+msr GCSCR_EL1, x0
+mrs x1, GCSCR_EL1
+// CHECK: msr     GCSCR_EL1, x0                   // encoding: [0x00,0x25,0x18,0xd5]
+// CHECK: mrs     x1, GCSCR_EL1                   // encoding: [0x01,0x25,0x38,0xd5]
+
+msr GCSPR_EL1, x2
+mrs x3, GCSPR_EL1
+// CHECK: msr     GCSPR_EL1, x2                   // encoding: [0x22,0x25,0x18,0xd5]
+// CHECK: mrs     x3, GCSPR_EL1                   // encoding: [0x23,0x25,0x38,0xd5]
+
+msr GCSCRE0_EL1, x4
+mrs x5, GCSCRE0_EL1
+// CHECK: msr     GCSCRE0_EL1, x4                 // encoding: [0x44,0x25,0x18,0xd5]
+// CHECK: mrs     x5, GCSCRE0_EL1                 // encoding: [0x45,0x25,0x38,0xd5]
+
+msr GCSPR_EL0, x6
+mrs x7, GCSPR_EL0
+// CHECK: msr     GCSPR_EL0, x6                   // encoding: [0x26,0x25,0x1b,0xd5]
+// CHECK: mrs     x7, GCSPR_EL0                   // encoding: [0x27,0x25,0x3b,0xd5]
+
+msr GCSCR_EL2, x10
+mrs x11, GCSCR_EL2
+// CHECK: msr     GCSCR_EL2, x10                  // encoding: [0x0a,0x25,0x1c,0xd5]
+// CHECK: mrs     x11, GCSCR_EL2                  // encoding: [0x0b,0x25,0x3c,0xd5]
+
+msr GCSPR_EL2, x12
+mrs x13, GCSPR_EL2
+// CHECK: msr     GCSPR_EL2, x12                  // encoding: [0x2c,0x25,0x1c,0xd5]
+// CHECK: mrs     x13, GCSPR_EL2                  // encoding: [0x2d,0x25,0x3c,0xd5]
+
+msr GCSCR_EL12, x14
+mrs x15, GCSCR_EL12
+// CHECK: msr     GCSCR_EL12, x14                 // encoding: [0x0e,0x25,0x1d,0xd5]
+// CHECK: mrs     x15, GCSCR_EL12                 // encoding: [0x0f,0x25,0x3d,0xd5]
+
+msr GCSPR_EL12, x16
+mrs x17, GCSPR_EL12
+// CHECK: msr     GCSPR_EL12, x16                 // encoding: [0x30,0x25,0x1d,0xd5]
+// CHECK: mrs     x17, GCSPR_EL12                 // encoding: [0x31,0x25,0x3d,0xd5]
+
+msr GCSCR_EL3, x18
+mrs x19, GCSCR_EL3
+// CHECK: msr     GCSCR_EL3, x18                  // encoding: [0x12,0x25,0x1e,0xd5]
+// CHECK: mrs     x19, GCSCR_EL3                  // encoding: [0x13,0x25,0x3e,0xd5]
+
+msr GCSPR_EL3, x20
+mrs x21, GCSPR_EL3
+// CHECK: msr     GCSPR_EL3, x20                  // encoding: [0x34,0x25,0x1e,0xd5]
+// CHECK: mrs     x21, GCSPR_EL3                  // encoding: [0x35,0x25,0x3e,0xd5]
+
+gcsss1 x21
+// CHECK: gcsss1  x21                        // encoding: [0x55,0x77,0x0b,0xd5]
+// ERROR-NO-GCS: [[@LINE-2]]:1: error: instruction requires: gcs
+
+gcsss2 x22
+// CHECK: gcsss2  x22                        // encoding: [0x76,0x77,0x2b,0xd5]
+// ERROR-NO-GCS: [[@LINE-2]]:1: error: instruction requires: gcs
+
+gcspushm x25
+// CHECK: gcspushm x25                       // encoding: [0x19,0x77,0x0b,0xd5]
+// ERROR-NO-GCS: [[@LINE-2]]:1: error: instruction requires: gcs
+
+gcspopm
+// CHECK: gcspopm                             // encoding: [0x3f,0x77,0x2b,0xd5]
+// ERROR-NO-GCS: [[@LINE-2]]:1: error: instruction requires: gcs
+
+gcspopm xzr
+// CHECK: gcspopm                            // encoding: [0x3f,0x77,0x2b,0xd5]
+// ERROR-NO-GCS: [[@LINE-2]]:1: error: instruction requires: gcs
+
+gcspopm x25
+// CHECK: gcspopm  x25                        // encoding: [0x39,0x77,0x2b,0xd5]
+// ERROR-NO-GCS: [[@LINE-2]]:1: error: instruction requires: gcs
+
+gcsb dsync
+// CHECK: gcsb    dsync                           // encoding: [0x7f,0x22,0x03,0xd5]
+// ERROR-NO-GCS-NOT: [[@LINE-2]]:1: error: instruction requires: gcs
+// NO-GCS: hint #19                              // encoding: [0x7f,0x22,0x03,0xd5]
+
+hint #19
+// CHECK: gcsb    dsync                           // encoding: [0x7f,0x22,0x03,0xd5]
+// ERROR-NO-GCS-NOT: [[@LINE-2]]:1: error: instruction requires: gcs
+// NO-GCS: hint #19                              // encoding: [0x7f,0x22,0x03,0xd5]
+
+gcsstr x26, x27
+// CHECK: gcsstr x26, x27                          // encoding: [0x7a,0x0f,0x1f,0xd9]
+// ERROR-NO-GCS: [[@LINE-2]]:1: error: instruction requires: gcs
+
+gcsstr x26, sp
+// CHECK: gcsstr x26, sp                           // encoding: [0xfa,0x0f,0x1f,0xd9]
+// ERROR-NO-GCS: [[@LINE-2]]:1: error: instruction requires: gcs
+
+gcssttr x26, x27
+// CHECK: gcssttr x26, x27                         // encoding: [0x7a,0x1f,0x1f,0xd9]
+// ERROR-NO-GCS: [[@LINE-2]]:1: error: instruction requires: gcs
+
+gcssttr x26, sp
+// CHECK: gcssttr x26, sp                          // encoding: [0xfa,0x1f,0x1f,0xd9]
+// ERROR-NO-GCS: [[@LINE-2]]:1: error: instruction requires: gcs
+
+gcspushx
+// CHECK: gcspushx                          // encoding: [0x9f,0x77,0x08,0xd5]
+// ERROR-NO-GCS: [[@LINE-2]]:1: error: instruction requires: gcs
+
+gcspopcx
+// CHECK: gcspopcx                          // encoding: [0xbf,0x77,0x08,0xd5]
+// ERROR-NO-GCS: [[@LINE-2]]:1: error: instruction requires: gcs
+
+gcspopx
+// CHECK: gcspopx                           // encoding: [0xdf,0x77,0x08,0xd5]
+// ERROR-NO-GCS: [[@LINE-2]]:1: error: instruction requires: gcs
diff --git a/llvm/test/MC/AArch64/directive-arch_extension-negative.s b/llvm/test/MC/AArch64/directive-arch_extension-negative.s
index b6686998d2ea0..1c1cfc9d33e3e 100644
--- a/llvm/test/MC/AArch64/directive-arch_extension-negative.s
+++ b/llvm/test/MC/AArch64/directive-arch_extension-negative.s
@@ -1,6 +1,6 @@
 // RUN: not llvm-mc -triple aarch64 \
 // RUN: -mattr=+crc,+sm4,+sha3,+sha2,+aes,+fp,+neon,+ras,+lse,+predres,+ccdp,+mte,+tlb-rmi,+pan-rwv,+ccpp,+rcpc,+ls64,+flagm,+hbc,+mops \
-// RUN: -mattr=+rcpc3,+lse128,+d128,+the,+rasv2,+ite,+cssc,+specres2 \
+// RUN: -mattr=+rcpc3,+lse128,+d128,+the,+rasv2,+ite,+cssc,+specres2,+gcs \
 // RUN: -filetype asm -o - %s 2>&1 | FileCheck %s
 
 .arch_extension axp64
@@ -210,3 +210,10 @@ mrs x0, ERXGSR_EL1
 mrs x0, ERXGSR_EL1
 // CHECK: [[@LINE-1]]:9: error: expected readable system register
 // CHECK-NEXT: mrs x0, ERXGSR_EL1
+
+gcspushm x0
+// CHECK-NOT: [[@LINE-1]]:1: error: instruction requires: gcs
+.arch_extension nogcs
+gcspushm x0
+// CHECK: [[@LINE-1]]:1: error: instruction requires: gcs
+// CHECK-NEXT: gcspushm x0
diff --git a/llvm/test/MC/AArch64/directive-arch_extension.s b/llvm/test/MC/AArch64/directive-arch_extension.s
index 03ca1fe87988b..b5bbee7910561 100644
--- a/llvm/test/MC/AArch64/directive-arch_extension.s
+++ b/llvm/test/MC/AArch64/directive-arch_extension.s
@@ -128,3 +128,7 @@ trcit x0
 .arch_extension cssc
 umax x0, x1, x2
 // CHECK: umax x0, x1, x2
+
+.arch_extension gcs
+gcspushm x0
+// CHECK: gcspushm x0
diff --git a/llvm/test/MC/AArch64/elf-reloc-plt32.s b/llvm/test/MC/AArch64/elf-reloc-plt32.s
index 5bde72d5f44cb..fdf603ffe3bae 100644
--- a/llvm/test/MC/AArch64/elf-reloc-plt32.s
+++ b/llvm/test/MC/AArch64/elf-reloc-plt32.s
@@ -1,10 +1,14 @@
 // RUN: llvm-mc -triple=aarch64 -filetype=obj %s -o - | \
 // RUN:   llvm-readobj -r - | FileCheck %s
+.globl func
+func:
 
         .section .data
 this:
         .word extern_func@PLT - this + 4
+        .word func@plt - . + 8
 
 // CHECK:      Section ({{.*}}) .rela.data
 // CHECK-NEXT:   0x0 R_AARCH64_PLT32 extern_func 0x4
+// CHECK-NEXT:   0x4 R_AARCH64_PLT32 func 0x8
 // CHECK-NEXT: }
diff --git a/llvm/test/MC/AMDGPU/hsa-diag-v3.s b/llvm/test/MC/AMDGPU/hsa-diag-v3.s
index aef2ecadb1673..369ac905ad2b2 100644
--- a/llvm/test/MC/AMDGPU/hsa-diag-v3.s
+++ b/llvm/test/MC/AMDGPU/hsa-diag-v3.s
@@ -1,13 +1,15 @@
-// RUN: not llvm-mc --amdhsa-code-object-version=3 -triple amdgcn-amd-amdhsa -mcpu=gfx810 -mattr=+xnack -show-encoding %s 2>&1 >/dev/null | FileCheck %s --check-prefixes=GCN,GFX8,NONGFX10,AMDHSA
-// RUN: not llvm-mc --amdhsa-code-object-version=3 -triple amdgcn-amd-amdhsa -mcpu=gfx1010 -mattr=+xnack -show-encoding %s 2>&1 >/dev/null | FileCheck %s --check-prefixes=GCN,GFX10,AMDHSA
+// RUN: not llvm-mc --amdhsa-code-object-version=3 -triple amdgcn-amd-amdhsa -mcpu=gfx810 -mattr=+xnack -show-encoding %s 2>&1 >/dev/null | FileCheck %s --check-prefixes=GCN,GFX8,PREGFX10,AMDHSA
+// RUN: not llvm-mc --amdhsa-code-object-version=3 -triple amdgcn-amd-amdhsa -mcpu=gfx1010 -mattr=+xnack -show-encoding %s 2>&1 >/dev/null | FileCheck %s --check-prefixes=GCN,GFX10PLUS,GFX10,AMDHSA
+// RUN: not llvm-mc --amdhsa-code-object-version=3 -triple amdgcn-amd-amdhsa -mcpu=gfx1100 -show-encoding %s 2>&1 >/dev/null | FileCheck %s --check-prefixes=GCN,GFX10PLUS,GFX11,AMDHSA
 // RUN: not llvm-mc --amdhsa-code-object-version=3 -triple amdgcn-amd- -mcpu=gfx810 -mattr=+xnack -show-encoding %s 2>&1 >/dev/null | FileCheck %s --check-prefixes=GCN,NONAMDHSA
-// RUN: not llvm-mc --amdhsa-code-object-version=3 -triple amdgcn-amd-amdhsa -mcpu=gfx90a -mattr=+xnack -show-encoding %s 2>&1 >/dev/null | FileCheck %s --check-prefixes=GFX90A,NONGFX10,AMDHSA,ALL
+// RUN: not llvm-mc --amdhsa-code-object-version=3 -triple amdgcn-amd-amdhsa -mcpu=gfx90a -mattr=+xnack -show-encoding %s 2>&1 >/dev/null | FileCheck %s --check-prefixes=GFX90A,PREGFX10,AMDHSA,ALL
 
 .text
 
 // GCN-LABEL: warning: test_target
 // GFX8-NOT: error:
 // GFX10: error: .amdgcn_target directive's target id amdgcn-amd-amdhsa--gfx810+xnack does not match the specified target id amdgcn-amd-amdhsa--gfx1010+xnack
+// GFX11: error: .amdgcn_target directive's target id amdgcn-amd-amdhsa--gfx810+xnack does not match the specified target id amdgcn-amd-amdhsa--gfx1100
 // NONAMDHSA: error: .amdgcn_target directive's target id amdgcn-amd-amdhsa--gfx810+xnack does not match the specified target id amdgcn-amd-unknown--gfx810
 .warning "test_target"
 .amdgcn_target "amdgcn-amd-amdhsa--gfx810+xnack"
@@ -154,8 +156,8 @@
 .end_amdhsa_kernel
 
 // ALL-LABEL: warning: test_amdhsa_wavefront_size32
-// NONGFX10: error: directive requires gfx10+
-// GFX10: error: .amdhsa_next_free_vgpr directive is required
+// PREGFX10: error: directive requires gfx10+
+// GFX10PLUS: error: .amdhsa_next_free_vgpr directive is required
 // NONAMDHSA: error: unknown directive
 .warning "test_amdhsa_wavefront_size32"
 .amdhsa_kernel test_amdhsa_wavefront_size32
@@ -163,8 +165,8 @@
 .end_amdhsa_kernel
 
 // GCN-LABEL: warning: test_amdhsa_wavefront_size32_invalid
-// NONGFX10: error: directive requires gfx10+
-// GFX10: error: value out of range
+// PREGFX10: error: directive requires gfx10+
+// GFX10PLUS: error: value out of range
 // NONAMDHSA: error: unknown directive
 .warning "test_amdhsa_wavefront_size32_invalid"
 .amdhsa_kernel test_amdhsa_wavefront_size32_invalid
@@ -172,8 +174,8 @@
 .end_amdhsa_kernel
 
 // GCN-LABEL: warning: test_amdhsa_workgroup_processor_mode
-// NONGFX10: error: directive requires gfx10+
-// GFX10: error: .amdhsa_next_free_vgpr directive is required
+// PREGFX10: error: directive requires gfx10+
+// GFX10PLUS: error: .amdhsa_next_free_vgpr directive is required
 // NONAMDHSA: error: unknown directive
 .warning "test_amdhsa_workgroup_processor_mode"
 .amdhsa_kernel test_amdhsa_workgroup_processor_mode
@@ -181,8 +183,8 @@
 .end_amdhsa_kernel
 
 // GCN-LABEL: warning: test_amdhsa_workgroup_processor_mode_invalid
-// NONGFX10: error: directive requires gfx10+
-// GFX10: error: value out of range
+// PREGFX10: error: directive requires gfx10+
+// GFX10PLUS: error: value out of range
 // NONAMDHSA: error: unknown directive
 .warning "test_amdhsa_workgroup_processor_mode_invalid"
 .amdhsa_kernel test_amdhsa_workgroup_processor_mode_invalid
@@ -190,8 +192,8 @@
 .end_amdhsa_kernel
 
 // GCN-LABEL: warning: test_amdhsa_memory_ordered
-// NONGFX10: error: directive requires gfx10+
-// GFX10: error: .amdhsa_next_free_vgpr directive is required
+// PREGFX10: error: directive requires gfx10+
+// GFX10PLUS: error: .amdhsa_next_free_vgpr directive is required
 // NONAMDHSA: error: unknown directive
 .warning "test_amdhsa_memory_ordered"
 .amdhsa_kernel test_amdhsa_memory_ordered
@@ -199,8 +201,8 @@
 .end_amdhsa_kernel
 
 // GCN-LABEL: warning: test_amdhsa_memory_ordered_invalid
-// NONGFX10: error: directive requires gfx10+
-// GFX10: error: value out of range
+// PREGFX10: error: directive requires gfx10+
+// GFX10PLUS: error: value out of range
 // NONAMDHSA: error: unknown directive
 .warning "test_amdhsa_memory_ordered_invalid"
 .amdhsa_kernel test_amdhsa_memory_ordered_invalid
@@ -208,8 +210,8 @@
 .end_amdhsa_kernel
 
 // GCN-LABEL: warning: test_amdhsa_forward_progress
-// NONGFX10: error: directive requires gfx10+
-// GFX10: error: .amdhsa_next_free_vgpr directive is required
+// PREGFX10: error: directive requires gfx10+
+// GFX10PLUS: error: .amdhsa_next_free_vgpr directive is required
 // NONAMDHSA: error: unknown directive
 .warning "test_amdhsa_forward_progress"
 .amdhsa_kernel test_amdhsa_forward_progress
@@ -217,8 +219,8 @@
 .end_amdhsa_kernel
 
 // GCN-LABEL: warning: test_amdhsa_forward_progress_invalid
-// NONGFX10: error: directive requires gfx10+
-// GFX10: error: value out of range
+// PREGFX10: error: directive requires gfx10+
+// GFX10PLUS: error: value out of range
 // NONAMDHSA: error: unknown directive
 .warning "test_amdhsa_forward_progress_invalid"
 .amdhsa_kernel test_amdhsa_forward_progress_invalid
@@ -226,8 +228,8 @@
 .end_amdhsa_kernel
 
 // GCN-LABEL: warning: test_amdhsa_shared_vgpr_count_invalid1
-// NONGFX10: error: directive requires gfx10+
-// GFX10: error: .amdhsa_next_free_vgpr directive is required
+// PREGFX10: error: directive requires gfx10+
+// GFX10PLUS: error: .amdhsa_next_free_vgpr directive is required
 // NONAMDHSA: error: unknown directive
 .warning "test_amdhsa_shared_vgpr_count_invalid1"
 .amdhsa_kernel test_amdhsa_shared_vgpr_count_invalid1
@@ -235,8 +237,8 @@
 .end_amdhsa_kernel
 
 // GCN-LABEL: warning: test_amdhsa_shared_vgpr_count_invalid2
-// NONGFX10: error: directive requires gfx10+
-// GFX10: error: shared_vgpr_count directive not valid on wavefront size 32
+// PREGFX10: error: directive requires gfx10+
+// GFX10PLUS: error: shared_vgpr_count directive not valid on wavefront size 32
 // NONAMDHSA: error: unknown directive
 .warning "test_amdhsa_shared_vgpr_count_invalid2"
 .amdhsa_kernel test_amdhsa_shared_vgpr_count_invalid2
@@ -247,8 +249,8 @@
 .end_amdhsa_kernel
 
 // GCN-LABEL: warning: test_amdhsa_shared_vgpr_count_invalid3
-// NONGFX10: error: directive requires gfx10+
-// GFX10: error: value out of range
+// PREGFX10: error: directive requires gfx10+
+// GFX10PLUS: error: value out of range
 // NONAMDHSA: error: unknown directive
 .warning "test_amdhsa_shared_vgpr_count_invalid3"
 .amdhsa_kernel test_amdhsa_shared_vgpr_count_invalid3
@@ -258,8 +260,8 @@
 .end_amdhsa_kernel
 
 // GCN-LABEL: warning: test_amdhsa_shared_vgpr_count_invalid4
-// NONGFX10: error: directive requires gfx10+
-// GFX10: error: shared_vgpr_count*2 + compute_pgm_rsrc1.GRANULATED_WORKITEM_VGPR_COUNT cannot exceed 63
+// PREGFX10: error: directive requires gfx10+
+// GFX10PLUS: error: shared_vgpr_count*2 + compute_pgm_rsrc1.GRANULATED_WORKITEM_VGPR_COUNT cannot exceed 63
 // NONAMDHSA: error: unknown directive
 .warning "test_amdhsa_shared_vgpr_count_invalid4"
 .amdhsa_kernel test_amdhsa_shared_vgpr_count_invalid4
diff --git a/llvm/test/MC/AsmParser/unmatched-if-macro.s b/llvm/test/MC/AsmParser/unmatched-if-macro.s
index 8ccbd5c043b45..d158d2cd10c28 100644
--- a/llvm/test/MC/AsmParser/unmatched-if-macro.s
+++ b/llvm/test/MC/AsmParser/unmatched-if-macro.s
@@ -1,3 +1,4 @@
+# RUN: export LSAN_OPTIONS=detect_leaks=0
 # RUN: not llvm-mc -triple x86_64 %s -o /dev/null 2>&1 | FileCheck %s
 
 ## This also tests that we don't assert due to an active macro instantiation.
diff --git a/llvm/test/MC/Disassembler/AArch64/armv9.4a-chk.txt b/llvm/test/MC/Disassembler/AArch64/armv9.4a-chk.txt
new file mode 100644
index 0000000000000..730f4440402e1
--- /dev/null
+++ b/llvm/test/MC/Disassembler/AArch64/armv9.4a-chk.txt
@@ -0,0 +1,8 @@
+# RUN: llvm-mc -triple=aarch64 -mattr=+v8.9a -disassemble %s 2> %t | FileCheck %s
+# RUN: llvm-mc -triple=aarch64 -mattr=+v9.4a -disassemble %s 2> %t | FileCheck %s
+# RUN: llvm-mc -triple=aarch64 -mattr=+chk   -disassemble %s 2> %t | FileCheck %s
+# RUN: llvm-mc -triple=aarch64 -mattr=+v8a   -disassemble %s 2> %t | FileCheck %s --check-prefix=NO-CHK
+
+[0x1f,0x25,0x03,0xd5]
+// CHECK: chkfeat x16
+// NO-CHK: hint #40
diff --git a/llvm/test/MC/Disassembler/AArch64/armv9.4a-gcs.txt b/llvm/test/MC/Disassembler/AArch64/armv9.4a-gcs.txt
new file mode 100644
index 0000000000000..7e2802b263858
--- /dev/null
+++ b/llvm/test/MC/Disassembler/AArch64/armv9.4a-gcs.txt
@@ -0,0 +1,90 @@
+# RUN: llvm-mc -triple=aarch64 -mattr +gcs -disassemble %s 2> %t | FileCheck %s
+
+[0x00,0x25,0x18,0xd5]
+[0x01,0x25,0x38,0xd5]
+// CHECK: msr     GCSCR_EL1, x0
+// CHECK: mrs     x1, GCSCR_EL1
+
+[0x22,0x25,0x18,0xd5]
+[0x23,0x25,0x38,0xd5]
+// CHECK: msr     GCSPR_EL1, x2
+// CHECK: mrs     x3, GCSPR_EL1
+
+[0x44,0x25,0x18,0xd5]
+[0x45,0x25,0x38,0xd5]
+// CHECK: msr     GCSCRE0_EL1, x4
+// CHECK: mrs     x5, GCSCRE0_EL1
+
+[0x26,0x25,0x1b,0xd5]
+[0x27,0x25,0x3b,0xd5]
+// CHECK: msr     GCSPR_EL0, x6
+// CHECK: mrs     x7, GCSPR_EL0
+
+[0x0a,0x25,0x1c,0xd5]
+[0x0b,0x25,0x3c,0xd5]
+// CHECK: msr     GCSCR_EL2, x10
+// CHECK: mrs     x11, GCSCR_EL2
+
+[0x2c,0x25,0x1c,0xd5]
+[0x2d,0x25,0x3c,0xd5]
+// CHECK: msr     GCSPR_EL2, x12
+// CHECK: mrs     x13, GCSPR_EL2
+
+[0x0e,0x25,0x1d,0xd5]
+[0x0f,0x25,0x3d,0xd5]
+// CHECK: msr     GCSCR_EL12, x14
+// CHECK: mrs     x15, GCSCR_EL12
+
+[0x30,0x25,0x1d,0xd5]
+[0x31,0x25,0x3d,0xd5]
+// CHECK: msr     GCSPR_EL12, x16
+// CHECK: mrs     x17, GCSPR_EL12
+
+[0x12,0x25,0x1e,0xd5]
+[0x13,0x25,0x3e,0xd5]
+// CHECK: msr     GCSCR_EL3, x18
+// CHECK: mrs     x19, GCSCR_EL3
+
+[0x34,0x25,0x1e,0xd5]
+[0x35,0x25,0x3e,0xd5]
+// CHECK: msr     GCSPR_EL3, x20
+// CHECK: mrs     x21, GCSPR_EL3
+
+[0x55,0x77,0x0b,0xd5]
+// CHECK: gcsss1 x21
+
+[0x76,0x77,0x2b,0xd5]
+// CHECK: gcsss2    x22
+
+[0x19,0x77,0x0b,0xd5]
+// CHECK: gcspushm x25
+
+[0x3f,0x77,0x2b,0xd5]
+// CHECK: gcspopm
+
+[0x39,0x77,0x2b,0xd5]
+// CHECK: gcspopm    x25
+
+[0x7f,0x22,0x03,0xd5]
+// CHECK: gcsb    dsync
+
+[0x7a,0x0f,0x1f,0xd9]
+// CHECK: gcsstr   x26, x27
+
+[0xfa,0x0f,0x1f,0xd9]
+// CHECK: gcsstr   x26, sp
+
+[0x7a,0x1f,0x1f,0xd9]
+// CHECK: gcssttr  x26, x27
+
+[0xfa,0x1f,0x1f,0xd9]
+// CHECK: gcssttr  x26, sp
+
+[0x9f,0x77,0x08,0xd5]
+// CHECK: gcspushx
+
+[0xbf,0x77,0x08,0xd5]
+// CHECK: gcspopcx
+
+[0xdf,0x77,0x08,0xd5]
+// CHECK: gcspopx
diff --git a/llvm/test/MC/RISCV/elf-reloc-plt32.s b/llvm/test/MC/RISCV/elf-reloc-plt32.s
index 8f3a40d36a390..f93262a138a91 100644
--- a/llvm/test/MC/RISCV/elf-reloc-plt32.s
+++ b/llvm/test/MC/RISCV/elf-reloc-plt32.s
@@ -3,9 +3,14 @@
 # RUN: llvm-mc -triple=riscv32 -filetype=obj %s -o - \
 # RUN:   | llvm-readobj -r - | FileCheck %s
 
+.globl func
+func:
+
 .data
 .word extern_func@PLT - . + 4
+.word func@PLT - . + 8
 
 # CHECK:      Section ({{.*}}) .rela.data {
 # CHECK-NEXT:   0x0 R_RISCV_PLT32 extern_func 0x4
+# CHECK-NEXT:   0x4 R_RISCV_PLT32 func 0x8
 # CHECK-NEXT: }
diff --git a/llvm/test/MC/RISCV/riscv64-64b-pcrel.s b/llvm/test/MC/RISCV/riscv64-64b-pcrel.s
index 5f5870ee73319..9dd7879fc8056 100644
--- a/llvm/test/MC/RISCV/riscv64-64b-pcrel.s
+++ b/llvm/test/MC/RISCV/riscv64-64b-pcrel.s
@@ -1,46 +1,90 @@
 # RUN: llvm-mc -triple riscv64-unknown-linux-gnu -filetype obj -o - %s \
 # RUN:   | llvm-readobj -r - | FileCheck %s
 # RUN: not llvm-mc -triple riscv64-unknown-linux-gnu -filetype obj --defsym ERR=1 -o /dev/null %s 2>&1 \
-# RUN:   | FileCheck %s --check-prefix CHECK-ERROR
+# RUN:   | FileCheck %s --check-prefix=ERROR
 
-# CHECK: Relocations [
-# CHECK:  .relasx {
-# CHECK-NEXT:    0x0 R_RISCV_ADD64 y 0x0
-# CHECK-NEXT:    0x0 R_RISCV_SUB64 x 0x0
-# CHECK:  }
-# CHECK:  .relasy {
-# CHECK-NEXT:    0x0 R_RISCV_ADD64 x 0x0
-# CHECK-NEXT:    0x0 R_RISCV_SUB64 y 0x0
-# CHECK:  }
-# CHECK:  .relasa {
-# CHECK-NEXT:    0x0 R_RISCV_ADD64 a 0x0
-# CHECK-NEXT:    0x0 R_RISCV_SUB64 z 0x0
-# CHECK:  }
+# CHECK:      Relocations [
+# CHECK-NEXT:   Section ({{.*}}) .rela.note {
+# CHECK-NEXT:     0x0 R_RISCV_ADD64 extern 0x0
+# CHECK-NEXT:     0x0 R_RISCV_SUB64 note 0x0
+# CHECK-NEXT:   }
+# CHECK-NEXT:   Section ({{.*}}) .rela.rodata {
+# CHECK-NEXT:     0x0 R_RISCV_ADD64 extern 0x0
+# CHECK-NEXT:     0x0 R_RISCV_SUB64 rodata 0x0
+# CHECK-NEXT:   }
+# CHECK-NEXT:   Section ({{.*}}) .rela.alloc_w {
+# CHECK-NEXT:     0x0 R_RISCV_ADD64 extern 0x0
+# CHECK-NEXT:     0x0 R_RISCV_SUB64 w 0x0
+# CHECK-NEXT:     0x8 R_RISCV_ADD64 w 0x0
+# CHECK-NEXT:     0x8 R_RISCV_SUB64 extern 0x0
+# CHECK-NEXT:     0x10 R_RISCV_ADD32 x 0x0
+# CHECK-NEXT:     0x10 R_RISCV_SUB32 w 0x0
+# CHECK-NEXT:     0x14 R_RISCV_ADD32 w1 0x0
+# CHECK-NEXT:     0x14 R_RISCV_SUB32 w 0x0
+# CHECK-NEXT:     0x18 R_RISCV_ADD32 .L.str 0x0
+# CHECK-NEXT:     0x18 R_RISCV_SUB32 w 0x0
+# CHECK-NEXT:   }
+# CHECK-NEXT:   Section ({{.*}}) .rela.alloc_x {
+# CHECK-NEXT:     0x0 R_RISCV_ADD64 y 0x0
+# CHECK-NEXT:     0x0 R_RISCV_SUB64 x 0x0
+# CHECK-NEXT:   }
+# CHECK-NEXT:   Section ({{.*}}) .rela.alloc_y {
+# CHECK-NEXT:     0x0 R_RISCV_ADD64 x 0x0
+# CHECK-NEXT:     0x0 R_RISCV_SUB64 y 0x0
+# CHECK-NEXT:   }
+# CHECK-NEXT:   Section ({{.*}}) .rela.nonalloc_w {
+# CHECK-NEXT:     0x0 R_RISCV_ADD64 extern 0x0
+# CHECK-NEXT:     0x0 R_RISCV_SUB64 nw 0x0
+# CHECK-NEXT:   }
+# CHECK-NEXT:   Section ({{.*}}) .rela.nonalloc_x {
+# CHECK-NEXT:     0x0 R_RISCV_ADD64 ny 0x0
+# CHECK-NEXT:     0x0 R_RISCV_SUB64 nx 0x0
+# CHECK-NEXT:   }
+# CHECK-NEXT:   Section ({{.*}}) .rela.nonalloc_y {
+# CHECK-NEXT:     0x0 R_RISCV_ADD64 nx 0x0
+# CHECK-NEXT:     0x0 R_RISCV_SUB64 ny 0x0
+# CHECK-NEXT:   }
 # CHECK-NEXT: ]
 
-	.section	sx,"aw",@progbits
-x:
-	.quad y-x
+.section .note,"a",@note; note:
+.quad extern-note
+.section .rodata,"a",@progbits; rodata:
+.quad extern-rodata
 
-	.section	sy,"aw",@progbits
-y:
-	.quad x-y
+.section .alloc_w,"aw",@progbits; w:
+.quad extern-w   # extern is undefined
+.quad w-extern
+.long x-w        # A is later defined in another section
+.long w1-w       # A is later defined in the same section
+.long .L.str-w   # A is temporary
+w1:
 
-	.section	sz
-z:
-.ifdef ERR
-	.quad z-a
+.section .alloc_x,"aw",@progbits; x:
+.quad y-x
+.section .alloc_y,"aw",@progbits; y:
+.quad x-y
 
-# CHECK-ERROR: Cannot represent a difference across sections
-# CHECK-ERROR: 	.quad z-a
-# CHECK-ERROR:        ^
-.else
-	.quad 0
+.section .nonalloc_w; nw:
+.quad extern-nw
+.ifdef ERR
+# ERROR: :[[#@LINE+1]]:7: error: symbol 'extern' can not be undefined in a subtraction expression
+.quad nw-extern
 .endif
+.section .nonalloc_x; nx:
+.quad ny-nx
+.section .nonalloc_y; ny:
+.quad nx-ny
 
-	.section	sa
-a:
-	.quad a-z
+## -gdwarf-aranges generated assembly expects no relocation.
+## Otherwise, a .Lsec_end0 symbol (defined at the end of .rodata.str1.1)
+## will be rejected by linkers.
+.section .rodata.str1.1,"aMS",@progbits,1
+.L.str:
+  .asciz  "hello"
+.section .rodata.str1.1,"aMS",@progbits,1
+.Lsec_end0:
+.section .debug_aranges,"",@progbits
+.quad .Lsec_end0-.L.str
 
 ## .apple_names/.apple_types are fixed-size and do not need fixups.
 ## llvm-dwarfdump --apple-names does not process R_RISCV_{ADD,SUB}32 in them.
diff --git a/llvm/test/MC/RISCV/zfa-invalid.s b/llvm/test/MC/RISCV/zfa-invalid.s
index 067b0317db5d0..e48618506626b 100644
--- a/llvm/test/MC/RISCV/zfa-invalid.s
+++ b/llvm/test/MC/RISCV/zfa-invalid.s
@@ -21,3 +21,58 @@ fcvtmod.w.d a1, ft1, rdn
 # CHECK-NO-RV64: error: operand must be 'rtz' floating-point rounding mode
 # CHECK-NO-RV32: error: operand must be 'rtz' floating-point rounding mode
 fcvtmod.w.d a1, ft1, rup
+
+# Invalid floating-point immediate
+# CHECK-NO-RV64: error: operand must be a valid floating-point constant
+# CHECK-NO-RV32: error: operand must be a valid floating-point constant
+fli.s ft1, 5.250000e-01
+
+# CHECK-NO-RV64: error: operand must be a valid floating-point constant
+# CHECK-NO-RV32: error: operand must be a valid floating-point constant
+fli.d ft1, 3.560000e+02
+
+# CHECK-NO-RV64: error: operand must be a valid floating-point constant
+# CHECK-NO-RV32: error: operand must be a valid floating-point constant
+fli.h ft1, 1.600000e+00
+
+# CHECK-NO-RV64: error: invalid floating point immediate
+# CHECK-NO-RV32: error: invalid floating point immediate
+fli.s ft1, -min
+
+# CHECK-NO-RV64: error: invalid floating point immediate
+# CHECK-NO-RV32: error: invalid floating point immediate
+fli.s ft1, -inf
+
+# CHECK-NO-RV64: error: invalid floating point immediate
+# CHECK-NO-RV32: error: invalid floating point immediate
+fli.s ft1, -nan
+
+# Don't accept decimal minimum.
+# CHECK-NO-RV64: error: operand must be a valid floating-point constant
+# CHECK-NO-RV32: error: operand must be a valid floating-point constant
+fli.s ft1, 1.1754943508222875079687365372222456778186655567720875215087517062784172594547271728515625e-38
+
+# Don't accept decimal minimum.
+# CHECK-NO-RV64: error: operand must be a valid floating-point constant
+# CHECK-NO-RV32: error: operand must be a valid floating-point constant
+fli.d ft1, 2.225073858507201383090232717332404064219215980462331830553327416887204434813918195854283159012511020564067339731035811005152434161553460108856012385377718821130777993532002330479610147442583636071921565046942503734208375250806650616658158948720491179968591639648500635908770118304874799780887753749949451580451605050915399856582470818645113537935804992115981085766051992433352114352390148795699609591288891602992641511063466313393663477586513029371762047325631781485664350872122828637642044846811407613911477062801689853244110024161447421618567166150540154285084716752901903161322778896729707373123334086988983175067838846926092773977972858659654941091369095406136467568702398678315290680984617210924625396728515625e-308
+
+# Don't accept decimal minimum.
+# CHECK-NO-RV64: error: operand must be a valid floating-point constant
+# CHECK-NO-RV32: error: operand must be a valid floating-point constant
+fli.h ft1, 6.103516e-05
+
+# Don't accept single precision minimum for double.
+# CHECK-NO-RV64: error: operand must be a valid floating-point constant
+# CHECK-NO-RV32: error: operand must be a valid floating-point constant
+fli.d ft1, 1.1754943508222875079687365372222456778186655567720875215087517062784172594547271728515625e-38
+
+# Don't accept single precision minimum for half.
+# CHECK-NO-RV64: error: operand must be a valid floating-point constant
+# CHECK-NO-RV32: error: operand must be a valid floating-point constant
+fli.h ft1, 1.1754943508222875079687365372222456778186655567720875215087517062784172594547271728515625e-38
+
+# Don't accept integers.
+# CHECK-NO-RV32: error: invalid floating point immediate
+# CHECK-NO-RV64: error: invalid floating point immediate
+fli.s ft1, 1
diff --git a/llvm/test/MC/RISCV/zfa-valid.s b/llvm/test/MC/RISCV/zfa-valid.s
index 0d307fc527b57..6892a7a5b0b0e 100644
--- a/llvm/test/MC/RISCV/zfa-valid.s
+++ b/llvm/test/MC/RISCV/zfa-valid.s
@@ -16,6 +16,486 @@
 # RUN:     -riscv-no-aliases -show-encoding < %s 2>&1 \
 # RUN:     | FileCheck -check-prefixes=CHECK-NO-EXT %s
 
+# CHECK-ASM-AND-OBJ: fli.s ft1, -1.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x10,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, -1.000000e+00
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, min
+# CHECK-ASM: encoding: [0xd3,0x80,0x10,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, min
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 1.52587890625e-05
+# CHECK-ASM: encoding: [0xd3,0x00,0x11,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 1.52587890625e-05
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 3.0517578125e-05
+# CHECK-ASM: encoding: [0xd3,0x80,0x11,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 3.0517578125e-05
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 0.00390625
+# CHECK-ASM: encoding: [0xd3,0x00,0x12,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 3.906250e-03
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 0.0078125
+# CHECK-ASM: encoding: [0xd3,0x80,0x12,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 7.812500e-03
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 0.0625
+# CHECK-ASM: encoding: [0xd3,0x00,0x13,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 6.250000e-02
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 0.125
+# CHECK-ASM: encoding: [0xd3,0x80,0x13,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 1.250000e-01
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 0.25
+# CHECK-ASM: encoding: [0xd3,0x00,0x14,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 2.500000e-01
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 0.3125
+# CHECK-ASM: encoding: [0xd3,0x80,0x14,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 3.125000e-01
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 0.375
+# CHECK-ASM: encoding: [0xd3,0x00,0x15,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 3.750000e-01
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 0.4375
+# CHECK-ASM: encoding: [0xd3,0x80,0x15,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 4.375000e-01
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 0.5
+# CHECK-ASM: encoding: [0xd3,0x00,0x16,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 5.000000e-01
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 0.625
+# CHECK-ASM: encoding: [0xd3,0x80,0x16,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 6.250000e-01
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 0.75
+# CHECK-ASM: encoding: [0xd3,0x00,0x17,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 7.500000e-01
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 0.875
+# CHECK-ASM: encoding: [0xd3,0x80,0x17,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 8.750000e-01
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 1.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x18,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 1.000000e+00
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 1.25
+# CHECK-ASM: encoding: [0xd3,0x80,0x18,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 1.250000e+00
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 1.5
+# CHECK-ASM: encoding: [0xd3,0x00,0x19,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 1.500000e+00
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 1.75
+# CHECK-ASM: encoding: [0xd3,0x80,0x19,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 1.750000e+00
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 2.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x1a,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 2.000000e+00
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 2.5
+# CHECK-ASM: encoding: [0xd3,0x80,0x1a,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 2.500000e+00
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 3.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x1b,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 3.000000e+00
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 4.0
+# CHECK-ASM: encoding: [0xd3,0x80,0x1b,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 4.000000e+00
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 8.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x1c,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 8.000000e+00
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 16.0
+# CHECK-ASM: encoding: [0xd3,0x80,0x1c,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 1.600000e+01
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 128.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x1d,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 1.280000e+02
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 256.0
+# CHECK-ASM: encoding: [0xd3,0x80,0x1d,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 2.560000e+02
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 32768.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x1e,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 3.276800e+04
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, 65536.0
+# CHECK-ASM: encoding: [0xd3,0x80,0x1e,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, 6.553600e+04
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, inf
+# CHECK-ASM: encoding: [0xd3,0x00,0x1f,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, INF
+
+# CHECK-ASM-AND-OBJ: fli.s ft1, nan
+# CHECK-ASM: encoding: [0xd3,0x80,0x1f,0xf0]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.s ft1, nan
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, -1.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x10,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, -1.000000e+00
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, min
+# CHECK-ASM: encoding: [0xd3,0x80,0x10,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, min
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 1.52587890625e-05
+# CHECK-ASM: encoding: [0xd3,0x00,0x11,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 1.52587890625e-05
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 3.0517578125e-05
+# CHECK-ASM: encoding: [0xd3,0x80,0x11,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 3.0517578125e-05
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 0.00390625
+# CHECK-ASM: encoding: [0xd3,0x00,0x12,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 3.906250e-03
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 0.0078125
+# CHECK-ASM: encoding: [0xd3,0x80,0x12,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 7.812500e-03
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 0.0625
+# CHECK-ASM: encoding: [0xd3,0x00,0x13,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 6.250000e-02
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 0.125
+# CHECK-ASM: encoding: [0xd3,0x80,0x13,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 1.250000e-01
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 0.25
+# CHECK-ASM: encoding: [0xd3,0x00,0x14,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 2.500000e-01
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 0.3125
+# CHECK-ASM: encoding: [0xd3,0x80,0x14,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 3.125000e-01
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 0.375
+# CHECK-ASM: encoding: [0xd3,0x00,0x15,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 3.750000e-01
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 0.4375
+# CHECK-ASM: encoding: [0xd3,0x80,0x15,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 4.375000e-01
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 0.5
+# CHECK-ASM: encoding: [0xd3,0x00,0x16,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 5.000000e-01
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 0.625
+# CHECK-ASM: encoding: [0xd3,0x80,0x16,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 6.250000e-01
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 0.75
+# CHECK-ASM: encoding: [0xd3,0x00,0x17,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 7.500000e-01
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 0.875
+# CHECK-ASM: encoding: [0xd3,0x80,0x17,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 8.750000e-01
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 1.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x18,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 1.000000e+00
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 1.25
+# CHECK-ASM: encoding: [0xd3,0x80,0x18,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 1.250000e+00
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 1.5
+# CHECK-ASM: encoding: [0xd3,0x00,0x19,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 1.500000e+00
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 1.75
+# CHECK-ASM: encoding: [0xd3,0x80,0x19,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 1.750000e+00
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 2.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x1a,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 2.000000e+00
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 2.5
+# CHECK-ASM: encoding: [0xd3,0x80,0x1a,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 2.500000e+00
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 3.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x1b,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 3.000000e+00
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 4.0
+# CHECK-ASM: encoding: [0xd3,0x80,0x1b,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 4.000000e+00
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 8.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x1c,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 8.000000e+00
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 16.0
+# CHECK-ASM: encoding: [0xd3,0x80,0x1c,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 1.600000e+01
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 128.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x1d,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 1.280000e+02
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 256.0
+# CHECK-ASM: encoding: [0xd3,0x80,0x1d,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 2.560000e+02
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 32768.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x1e,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 3.276800e+04
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, 65536.0
+# CHECK-ASM: encoding: [0xd3,0x80,0x1e,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, 6.553600e+04
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, inf
+# CHECK-ASM: encoding: [0xd3,0x00,0x1f,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, INF
+
+# CHECK-ASM-AND-OBJ: fli.d ft1, nan
+# CHECK-ASM: encoding: [0xd3,0x80,0x1f,0xf2]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.d ft1, nan
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, -1.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x10,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, -1.000000e+00
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, min
+# CHECK-ASM: encoding: [0xd3,0x80,0x10,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, min
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 1.52587890625e-05
+# CHECK-ASM: encoding: [0xd3,0x00,0x11,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 1.52587890625e-05
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 3.0517578125e-05
+# CHECK-ASM: encoding: [0xd3,0x80,0x11,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 3.0517578125e-05
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 0.00390625
+# CHECK-ASM: encoding: [0xd3,0x00,0x12,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 3.906250e-03
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 0.0078125
+# CHECK-ASM: encoding: [0xd3,0x80,0x12,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 7.812500e-03
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 0.0625
+# CHECK-ASM: encoding: [0xd3,0x00,0x13,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 6.250000e-02
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 0.125
+# CHECK-ASM: encoding: [0xd3,0x80,0x13,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 1.250000e-01
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 0.25
+# CHECK-ASM: encoding: [0xd3,0x00,0x14,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 2.500000e-01
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 0.3125
+# CHECK-ASM: encoding: [0xd3,0x80,0x14,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 3.125000e-01
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 0.375
+# CHECK-ASM: encoding: [0xd3,0x00,0x15,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 3.750000e-01
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 0.4375
+# CHECK-ASM: encoding: [0xd3,0x80,0x15,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 4.375000e-01
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 0.5
+# CHECK-ASM: encoding: [0xd3,0x00,0x16,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 5.000000e-01
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 0.625
+# CHECK-ASM: encoding: [0xd3,0x80,0x16,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 6.250000e-01
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 0.75
+# CHECK-ASM: encoding: [0xd3,0x00,0x17,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 7.500000e-01
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 0.875
+# CHECK-ASM: encoding: [0xd3,0x80,0x17,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 8.750000e-01
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 1.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x18,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 1.000000e+00
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 1.25
+# CHECK-ASM: encoding: [0xd3,0x80,0x18,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 1.250000e+00
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 1.5
+# CHECK-ASM: encoding: [0xd3,0x00,0x19,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 1.500000e+00
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 1.75
+# CHECK-ASM: encoding: [0xd3,0x80,0x19,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 1.750000e+00
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 2.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x1a,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 2.000000e+00
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 2.5
+# CHECK-ASM: encoding: [0xd3,0x80,0x1a,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 2.500000e+00
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 3.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x1b,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 3.000000e+00
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 4.0
+# CHECK-ASM: encoding: [0xd3,0x80,0x1b,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 4.000000e+00
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 8.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x1c,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 8.000000e+00
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 16.0
+# CHECK-ASM: encoding: [0xd3,0x80,0x1c,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 1.600000e+01
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 128.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x1d,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 1.280000e+02
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 256.0
+# CHECK-ASM: encoding: [0xd3,0x80,0x1d,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 2.560000e+02
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 32768.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x1e,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 3.276800e+04
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, 65536.0
+# CHECK-ASM: encoding: [0xd3,0x80,0x1e,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, 6.553600e+04
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, inf
+# CHECK-ASM: encoding: [0xd3,0x00,0x1f,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, INF
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, nan
+# CHECK-ASM: encoding: [0xd3,0x80,0x1f,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
+fli.h ft1, nan
+
 # CHECK-ASM-AND-OBJ: fminm.s fa0, fa1, fa2
 # CHECK-ASM: encoding: [0x53,0xa5,0xc5,0x28]
 # CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point){{$}}
diff --git a/llvm/test/MC/RISCV/zfa-zfhmin-zvfh-valid.s b/llvm/test/MC/RISCV/zfa-zfhmin-zvfh-valid.s
new file mode 100644
index 0000000000000..aa458129ecd26
--- /dev/null
+++ b/llvm/test/MC/RISCV/zfa-zfhmin-zvfh-valid.s
@@ -0,0 +1,22 @@
+# RUN: llvm-mc %s -triple=riscv32 -mattr=+experimental-zfa,+zfhmin,+experimental-zvfh -riscv-no-aliases -show-encoding \
+# RUN:     | FileCheck -check-prefixes=CHECK-ASM,CHECK-ASM-AND-OBJ %s
+# RUN: llvm-mc %s -triple=riscv64 -mattr=+experimental-zfa,+zfhmin,+experimental-zvfh -riscv-no-aliases -show-encoding \
+# RUN:     | FileCheck -check-prefixes=CHECK-ASM,CHECK-ASM-AND-OBJ %s
+# RUN: llvm-mc -filetype=obj -triple=riscv32 -mattr=+experimental-zfa,+zfhmin,+experimental-zvfh < %s \
+# RUN:     | llvm-objdump --mattr=+experimental-zfa,+zfhmin,+experimental-zvfh -M no-aliases -d -r - \
+# RUN:     | FileCheck --check-prefix=CHECK-ASM-AND-OBJ %s
+# RUN: llvm-mc -filetype=obj -triple=riscv64 -mattr=+experimental-zfa,+zfhmin,+experimental-zvfh < %s \
+# RUN:     | llvm-objdump --mattr=+experimental-zfa,+zfhmin,+experimental-zvfh -M no-aliases -d -r - \
+# RUN:     | FileCheck --check-prefix=CHECK-ASM-AND-OBJ %s
+#
+# RUN: not llvm-mc -triple riscv32 -riscv-no-aliases -show-encoding < %s 2>&1 \
+# RUN:     | FileCheck -check-prefixes=CHECK-NO-EXT %s
+# RUN: not llvm-mc -triple riscv64 -riscv-no-aliases -show-encoding < %s 2>&1 \
+# RUN:     | FileCheck -check-prefixes=CHECK-NO-EXT %s
+
+# This test makes sure fli.h is supported with Zvfh.
+
+# CHECK-ASM-AND-OBJ: fli.h ft1, -1.0
+# CHECK-ASM: encoding: [0xd3,0x00,0x10,0xf4]
+# CHECK-NO-EXT: error: instruction requires the following: 'Zfa' (Additional Floating-Point), 'Zfh' (Half-Precision Floating-Point) or 'Zvfh' (Vector Half-Precision Floating-Point){{$}}
+fli.h ft1, -1.0
diff --git a/llvm/test/MC/WebAssembly/section-symbol.s b/llvm/test/MC/WebAssembly/section-symbol.s
index b55221c0947f3..adf6111a02b0d 100644
--- a/llvm/test/MC/WebAssembly/section-symbol.s
+++ b/llvm/test/MC/WebAssembly/section-symbol.s
@@ -3,7 +3,7 @@
 # check that we can refer to section symbols of other sections.
 # getWasmSection currently forces the section symbol to have a suffix.
 
-# TODO: fix the 0-suffix: https://bugs.llvm.org/show_bug.cgi?id=49252
+# TODO: fix the 0-suffix: https://github.com/llvm/llvm-project/issues/48596
 
     .section    .debug_abbrev,"",@
     .int8       1
diff --git a/llvm/test/MC/X86/index-operations.s b/llvm/test/MC/X86/index-operations.s
index 0c39fe8a90da0..a355b7ae0760d 100644
--- a/llvm/test/MC/X86/index-operations.s
+++ b/llvm/test/MC/X86/index-operations.s
@@ -160,3 +160,29 @@ insw %dx, (%rbx)
 // ERR32: 64-bit
 // ERR16: 64-bit
 
+movdir64b	291(%si), %ecx
+// ERR32: invalid operand
+// ERR16: invalid operand
+
+movdir64b	291(%esi), %cx
+// ERR32: invalid operand
+// ERR16: invalid operand
+
+movdir64b (%rdx), %r15d
+// ERR64: invalid operand
+
+movdir64b (%edx), %r15
+// ERR64: invalid operand
+
+movdir64b (%eip), %ebx
+// 64: movdir64b (%eip), %ebx # encoding: [0x67,0x66,0x0f,0x38,0xf8,0x1d,0x00,0x00,0x00,0x00]
+
+movdir64b (%rip), %rbx
+// 64: movdir64b (%rip), %rbx # encoding: [0x66,0x0f,0x38,0xf8,0x1d,0x00,0x00,0x00,0x00]
+
+movdir64b 291(%esi, %eiz, 4), %ebx
+// 64: movdir64b 291(%esi,%eiz,4), %ebx # encoding: [0x67,0x66,0x0f,0x38,0xf8,0x9c,0xa6,0x23,0x01,0x00,0x00]
+// 32: movdir64b 291(%esi,%eiz,4), %ebx # encoding: [0x66,0x0f,0x38,0xf8,0x9c,0xa6,0x23,0x01,0x00,0x00]
+
+movdir64b 291(%rsi, %riz, 4), %rbx
+// 64: movdir64b 291(%rsi,%riz,4), %rbx # encoding: [0x66,0x0f,0x38,0xf8,0x9c,0xa6,0x23,0x01,0x00,0x00]
\ No newline at end of file
diff --git a/llvm/test/Other/constant-fold-gep.ll b/llvm/test/Other/constant-fold-gep.ll
index 4bc9d29553df3..0c1ca129bdb38 100644
--- a/llvm/test/Other/constant-fold-gep.ll
+++ b/llvm/test/Other/constant-fold-gep.ll
@@ -457,7 +457,7 @@ define ptr @different_addrspace() nounwind noinline {
   %p = getelementptr inbounds i8, ptr addrspacecast (ptr addrspace(12) @p12 to ptr),
                                   i32 2
   ret ptr %p
-; OPT: ret ptr getelementptr (i8, ptr addrspacecast (ptr addrspace(12) @p12 to ptr), i64 2)
+; OPT: ret ptr getelementptr inbounds (i8, ptr addrspacecast (ptr addrspace(12) @p12 to ptr), i64 2)
 }
 
 define ptr @same_addrspace() nounwind noinline {
diff --git a/llvm/test/Other/loop-pm-invalidation.ll b/llvm/test/Other/loop-pm-invalidation.ll
index e215fb5d847c2..25552f7f139fd 100644
--- a/llvm/test/Other/loop-pm-invalidation.ll
+++ b/llvm/test/Other/loop-pm-invalidation.ll
@@ -2,18 +2,18 @@
 ;
 ; Check that we always nuke the LPM stuff when the loops themselves are
 ; invalidated.
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager -aa-pipeline= %s 2>&1 \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager -aa-pipeline= %s 2>&1 \
 ; RUN:     -passes='loop(no-op-loop),invalidate<loops>,loop(no-op-loop)' \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-LOOP-INV
 ;
 ; If we ended up building the standard analyses, their invalidation should nuke
 ; stuff as well.
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager %s -aa-pipeline= 2>&1 \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager %s -aa-pipeline= 2>&1 \
 ; RUN:     -passes='loop(no-op-loop),invalidate<scalar-evolution>,loop(no-op-loop)' \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-SCEV-INV
 ;
 ; Also provide a test that can delete loops after populating analyses for them.
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0  -debug-pass-manager %s -aa-pipeline= 2>&1 \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0  -debug-pass-manager %s -aa-pipeline= 2>&1 \
 ; RUN:     -passes='loop(no-op-loop,loop-deletion),invalidate<scalar-evolution>,loop(no-op-loop)' \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-SCEV-INV-AFTER-DELETE
 
diff --git a/llvm/test/Other/new-pass-manager.ll b/llvm/test/Other/new-pass-manager.ll
index 720392668c685..627b8b9a397eb 100644
--- a/llvm/test/Other/new-pass-manager.ll
+++ b/llvm/test/Other/new-pass-manager.ll
@@ -5,18 +5,18 @@
 ; files, but for now this is just going to step the new process through its
 ; paces.
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager \
 ; RUN:     -passes=no-op-module %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-MODULE-PASS
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager \
 ; RUN:     -p no-op-module %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-MODULE-PASS
 ; CHECK-MODULE-PASS: Running pass: NoOpModulePass
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager \
 ; RUN:     -passes=no-op-cgscc %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-CGSCC-PASS
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager \
 ; RUN:     -passes='cgscc(no-op-cgscc)' %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-CGSCC-PASS
 ; CHECK-CGSCC-PASS: Running analysis: InnerAnalysisManagerProxy<{{.*(CGSCCAnalysisManager|AnalysisManager<.*LazyCallGraph::SCC.*>).*}},{{.*}}Module>
@@ -27,10 +27,10 @@
 ; CHECK-CGSCC-PASS-NEXT: Running analysis: OuterAnalysisManagerProxy<{{.*}}LazyCallGraph::SCC{{.*}}>
 ; CHECK-CGSCC-PASS-NEXT: Running pass: NoOpCGSCCPass
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager \
 ; RUN:     -passes=no-op-function %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-FUNCTION-PASS
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager \
 ; RUN:     -passes='function(no-op-function)' %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-FUNCTION-PASS
 ; CHECK-FUNCTION-PASS: Running analysis: InnerAnalysisManagerProxy<{{.*}}>
@@ -43,7 +43,7 @@
 ; CHECK-MODULE-PRINT: define void @foo(i1 %x, ptr %p1, ptr %p2)
 ; CHECK-MODULE-PRINT: Running pass: VerifierPass
 
-; RUN: opt -disable-output -debug-pass-manager -disable-verify -verify-cfg-preserved=0 -passes='print,verify' %s 2>&1 \
+; RUN: opt -disable-output -debug-pass-manager -disable-verify -verify-analysis-invalidation=0 -passes='print,verify' %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-MODULE-VERIFY
 ; CHECK-MODULE-VERIFY: Running pass: PrintModulePass
 ; CHECK-MODULE-VERIFY: ModuleID
@@ -58,7 +58,7 @@
 ; CHECK-FUNCTION-PRINT: define void @foo(i1 %x, ptr %p1, ptr %p2)
 ; CHECK-FUNCTION-PRINT: Running pass: VerifierPass
 
-; RUN: opt -disable-output -debug-pass-manager -disable-verify -verify-cfg-preserved=0 -passes='function(print,verify)' %s 2>&1 \
+; RUN: opt -disable-output -debug-pass-manager -disable-verify -verify-analysis-invalidation=0 -passes='function(print,verify)' %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-FUNCTION-VERIFY
 ; CHECK-FUNCTION-VERIFY: Running pass: PrintFunctionPass
 ; CHECK-FUNCTION-VERIFY-NOT: ModuleID
@@ -83,7 +83,7 @@
 ; RUN:     | llvm-dis \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-NOOP
 
-; RUN: opt -disable-output -debug-pass-manager -disable-verify -verify-cfg-preserved=0 -passes='no-op-module,function(no-op-function)' %s 2>&1 \
+; RUN: opt -disable-output -debug-pass-manager -disable-verify -verify-analysis-invalidation=0 -passes='no-op-module,function(no-op-function)' %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-NO-VERIFY
 ; CHECK-NO-VERIFY-NOT: VerifierPass
 ; CHECK-NO-VERIFY: Running pass: NoOpModulePass
@@ -153,7 +153,7 @@
 ; CHECK-DO-INVALIDATE-FUNCTION-ANALYSIS-RESULTS: Invalidating analysis: NoOpFunctionAnalysis
 ; CHECK-DO-INVALIDATE-FUNCTION-ANALYSIS-RESULTS: Running analysis: NoOpFunctionAnalysis
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager \
 ; RUN:     -passes='require<no-op-module>,module(require<no-op-module>,function(require<no-op-function>,invalidate<all>,require<no-op-function>),require<no-op-module>),require<no-op-module>' %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-INVALIDATE-ALL
 ; CHECK-INVALIDATE-ALL: Running pass: RequireAnalysisPass
@@ -173,7 +173,7 @@
 ; CHECK-INVALIDATE-ALL: Running pass: RequireAnalysisPass
 ; CHECK-INVALIDATE-ALL-NOT: Running analysis: NoOpModuleAnalysis
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager \
 ; RUN:     -passes='require<no-op-module>,module(require<no-op-module>,cgscc(require<no-op-cgscc>,function(require<no-op-function>,invalidate<all>,require<no-op-function>),require<no-op-cgscc>),require<no-op-module>),require<no-op-module>' %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-INVALIDATE-ALL-CG
 ; CHECK-INVALIDATE-ALL-CG: Running pass: RequireAnalysisPass
@@ -200,7 +200,7 @@
 ; CHECK-INVALIDATE-ALL-CG: Running pass: RequireAnalysisPass
 ; CHECK-INVALIDATE-ALL-CG-NOT: Running analysis: NoOpModuleAnalysis
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager %s 2>&1 \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager %s 2>&1 \
 ; RUN:     -passes='require<targetlibinfo>,invalidate<all>,require<targetlibinfo>' \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-TLI
 ; CHECK-TLI: Running pass: RequireAnalysisPass
@@ -210,7 +210,7 @@
 ; CHECK-TLI: Running pass: RequireAnalysisPass
 ; CHECK-TLI-NOT: Running analysis: TargetLibraryAnalysis
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager %s 2>&1 \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager %s 2>&1 \
 ; RUN:     -passes='require<targetir>,invalidate<all>,require<targetir>' \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-TIRA
 ; CHECK-TIRA: Running pass: RequireAnalysisPass
@@ -220,26 +220,26 @@
 ; CHECK-TIRA: Running pass: RequireAnalysisPass
 ; CHECK-TIRA-NOT: Running analysis: TargetIRAnalysis
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager %s 2>&1 \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager %s 2>&1 \
 ; RUN:     -passes='require<domtree>' \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-DT
 ; CHECK-DT: Running pass: RequireAnalysisPass
 ; CHECK-DT: Running analysis: DominatorTreeAnalysis
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager %s 2>&1 \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager %s 2>&1 \
 ; RUN:     -passes='require<basic-aa>' \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-BASIC-AA
 ; CHECK-BASIC-AA: Running pass: RequireAnalysisPass
 ; CHECK-BASIC-AA: Running analysis: BasicAA
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager %s 2>&1 \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager %s 2>&1 \
 ; RUN:     -passes='require<aa>' -aa-pipeline='basic-aa' \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-AA
 ; CHECK-AA: Running pass: RequireAnalysisPass
 ; CHECK-AA: Running analysis: AAManager
 ; CHECK-AA: Running analysis: BasicAA
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager %s 2>&1 \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager %s 2>&1 \
 ; RUN:     -passes='require<aa>' -aa-pipeline='default' \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-AA-DEFAULT
 ; CHECK-AA-DEFAULT: Running pass: RequireAnalysisPass
@@ -248,7 +248,7 @@
 ; CHECK-AA-DEFAULT: Running analysis: ScopedNoAliasAA
 ; CHECK-AA-DEFAULT: Running analysis: TypeBasedAA
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager %s 2>&1 \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager %s 2>&1 \
 ; RUN:     -passes='require<aa>,invalidate<domtree>,aa-eval' -aa-pipeline='basic-aa' \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-AA-FUNCTION-INVALIDATE
 ; CHECK-AA-FUNCTION-INVALIDATE: Running pass: RequireAnalysisPass
@@ -262,7 +262,7 @@
 ; CHECK-AA-FUNCTION-INVALIDATE: Running analysis: AAManager
 ; CHECK-AA-FUNCTION-INVALIDATE: Running analysis: BasicAA
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager %s 2>&1 \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager %s 2>&1 \
 ; RUN:     -passes='require<globals-aa>,function(require<aa>),invalidate<globals-aa>,require<globals-aa>,function(aa-eval)' -aa-pipeline='globals-aa' \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-AA-MODULE-INVALIDATE
 ; CHECK-AA-MODULE-INVALIDATE: Running pass: RequireAnalysisPass
@@ -275,19 +275,19 @@
 ; CHECK-AA-MODULE-INVALIDATE: Running analysis: GlobalsAA
 ; CHECK-AA-MODULE-INVALIDATE: Running pass: AAEvaluator
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager %s 2>&1 \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager %s 2>&1 \
 ; RUN:     -passes='require<memdep>' \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-MEMDEP
 ; CHECK-MEMDEP: Running pass: RequireAnalysisPass
 ; CHECK-MEMDEP: Running analysis: MemoryDependenceAnalysis
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager %s 2>&1 \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager %s 2>&1 \
 ; RUN:     -passes='require<callgraph>' \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-CALLGRAPH
 ; CHECK-CALLGRAPH: Running pass: RequireAnalysisPass
 ; CHECK-CALLGRAPH: Running analysis: CallGraphAnalysis
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager \
 ; RUN:     -passes='default<O0>' %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-O0 --check-prefix=%llvmcheckext
 ; CHECK-O0: Running pass: AlwaysInlinerPass
@@ -298,7 +298,7 @@
 ; avoiding FileCheck complaining about the unused prefix.
 ; CHECK-NOEXT: {{.*}}
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager \
 ; RUN:     -passes='repeat<3>(no-op-module)' %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-REPEAT-MODULE-PASS
 ; CHECK-REPEAT-MODULE-PASS: Running pass: RepeatedPass
@@ -306,7 +306,7 @@
 ; CHECK-REPEAT-MODULE-PASS-NEXT: Running pass: NoOpModulePass
 ; CHECK-REPEAT-MODULE-PASS-NEXT: Running pass: NoOpModulePass
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager \
 ; RUN:     -passes='cgscc(repeat<3>(no-op-cgscc))' %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-REPEAT-CGSCC-PASS
 ; CHECK-REPEAT-CGSCC-PASS: Running analysis: InnerAnalysisManagerProxy<{{.*(CGSCCAnalysisManager|AnalysisManager<.*LazyCallGraph::SCC.*>).*}},{{.*}}Module>
@@ -320,7 +320,7 @@
 ; CHECK-REPEAT-CGSCC-PASS-NEXT: Running pass: NoOpCGSCCPass
 ; CHECK-REPEAT-CGSCC-PASS-NEXT: Running pass: NoOpCGSCCPass
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager \
 ; RUN:     -passes='function(repeat<3>(no-op-function))' %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-REPEAT-FUNCTION-PASS
 ; CHECK-REPEAT-FUNCTION-PASS: Running analysis: InnerAnalysisManagerProxy<{{.*}}>
@@ -329,7 +329,7 @@
 ; CHECK-REPEAT-FUNCTION-PASS-NEXT: Running pass: NoOpFunctionPass
 ; CHECK-REPEAT-FUNCTION-PASS-NEXT: Running pass: NoOpFunctionPass
 
-; RUN: opt -disable-output -disable-verify -verify-cfg-preserved=0 -debug-pass-manager \
+; RUN: opt -disable-output -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager \
 ; RUN:     -passes='loop(repeat<3>(no-op-loop))' %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-REPEAT-LOOP-PASS
 ; CHECK-REPEAT-LOOP-PASS: Running analysis: InnerAnalysisManagerProxy<{{.*}}>
diff --git a/llvm/test/Other/new-pm-O0-defaults.ll b/llvm/test/Other/new-pm-O0-defaults.ll
index 8e97793bd2111..b191794444981 100644
--- a/llvm/test/Other/new-pm-O0-defaults.ll
+++ b/llvm/test/Other/new-pm-O0-defaults.ll
@@ -7,25 +7,25 @@
 ; Any invalidation that shows up here is a bug, unless we started modifying
 ; the IR, in which case we need to make it immutable harder.
 
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager \
 ; RUN:     -passes='default<O0>' -S %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK,CHECK-DEFAULT,CHECK-CORO
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -debug-pass-manager -enable-matrix \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager -enable-matrix \
 ; RUN:     -passes='default<O0>' -S %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK,CHECK-DEFAULT,CHECK-MATRIX,CHECK-CORO
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -debug-pass-manager -debug-info-for-profiling \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager -debug-info-for-profiling \
 ; RUN:     -passes='default<O0>' -S %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK,CHECK-DIS,CHECK-CORO
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager \
 ; RUN:     -passes='thinlto-pre-link<O0>' -S %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK,CHECK-DEFAULT,CHECK-PRE-LINK,CHECK-CORO
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager \
 ; RUN:     -passes='lto-pre-link<O0>' -S %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK,CHECK-DEFAULT,CHECK-PRE-LINK,CHECK-CORO
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager \
 ; RUN:     -passes='thinlto<O0>' -S %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK,CHECK-THINLTO
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -debug-pass-manager \
 ; RUN:     -passes='lto<O0>' -S %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK,CHECK-LTO
 
diff --git a/llvm/test/Other/new-pm-defaults.ll b/llvm/test/Other/new-pm-defaults.ll
index 8b2bc4c84bcad..c2b44b7150cdf 100644
--- a/llvm/test/Other/new-pm-defaults.ll
+++ b/llvm/test/Other/new-pm-defaults.ll
@@ -7,85 +7,85 @@
 ; Any invalidation that shows up here is a bug, unless we started modifying
 ; the IR, in which case we need to make it immutable harder.
 
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='default<O1>' -S %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-O123,CHECK-O1,%llvmcheckext
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-FUNC-SPEC,CHECK-O1,%llvmcheckext
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='default<O2>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-O123,CHECK-O2,CHECK-O23SZ,%llvmcheckext
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-FUNC-SPEC,CHECK-O2,CHECK-O23SZ,%llvmcheckext
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='default<O3>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-O123,CHECK-O3,CHECK-O23SZ,%llvmcheckext
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-FUNC-SPEC,CHECK-O3,CHECK-O23SZ,%llvmcheckext
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='default<Os>' -S %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-OSZ,CHECK-Os,CHECK-O23SZ,%llvmcheckext
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-NO-FUNC-SPEC,CHECK-Os,CHECK-O23SZ,%llvmcheckext
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='default<Oz>' -S %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-OSZ,CHECK-Oz,CHECK-O23SZ,%llvmcheckext
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-NO-FUNC-SPEC,CHECK-Oz,CHECK-O23SZ,%llvmcheckext
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='lto-pre-link<O2>' -S %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-LTO,CHECK-O123,CHECK-O2,CHECK-O23SZ,%llvmcheckext
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-LTO,CHECK-NO-FUNC-SPEC,CHECK-O2,CHECK-O23SZ,%llvmcheckext
 
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes-ep-peephole='no-op-function' \
 ; RUN:     -passes='default<O3>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-O123,CHECK-O3,%llvmcheckext,CHECK-EP-PEEPHOLE,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-FUNC-SPEC,CHECK-O3,%llvmcheckext,CHECK-EP-PEEPHOLE,CHECK-O23SZ
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes-ep-late-loop-optimizations='no-op-loop' \
 ; RUN:     -passes='default<O3>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-O123,CHECK-O3,%llvmcheckext,CHECK-EP-LOOP-LATE,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-FUNC-SPEC,CHECK-O3,%llvmcheckext,CHECK-EP-LOOP-LATE,CHECK-O23SZ
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes-ep-loop-optimizer-end='no-op-loop' \
 ; RUN:     -passes='default<O3>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-O123,CHECK-O3,%llvmcheckext,CHECK-EP-LOOP-END,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-FUNC-SPEC,CHECK-O3,%llvmcheckext,CHECK-EP-LOOP-END,CHECK-O23SZ
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes-ep-scalar-optimizer-late='no-op-function' \
 ; RUN:     -passes='default<O3>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-O123,CHECK-O3,%llvmcheckext,CHECK-EP-SCALAR-LATE,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-FUNC-SPEC,CHECK-O3,%llvmcheckext,CHECK-EP-SCALAR-LATE,CHECK-O23SZ
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes-ep-cgscc-optimizer-late='no-op-cgscc' \
 ; RUN:     -passes='default<O3>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-O123,CHECK-O3,%llvmcheckext,CHECK-EP-CGSCC-LATE,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-FUNC-SPEC,CHECK-O3,%llvmcheckext,CHECK-EP-CGSCC-LATE,CHECK-O23SZ
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes-ep-vectorizer-start='no-op-function' \
 ; RUN:     -passes='default<O3>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-O123,CHECK-O3,%llvmcheckext,CHECK-EP-VECTORIZER-START,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-FUNC-SPEC,CHECK-O3,%llvmcheckext,CHECK-EP-VECTORIZER-START,CHECK-O23SZ
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes-ep-pipeline-start='no-op-module' \
 ; RUN:     -passes='default<O3>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-O123,CHECK-O3,%llvmcheckext,CHECK-EP-PIPELINE-START,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-FUNC-SPEC,CHECK-O3,%llvmcheckext,CHECK-EP-PIPELINE-START,CHECK-O23SZ
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes-ep-pipeline-early-simplification='no-op-module' \
 ; RUN:     -passes='default<O3>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-O123,CHECK-O3,%llvmcheckext,CHECK-EP-PIPELINE-EARLY-SIMPLIFICATION,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-FUNC-SPEC,CHECK-O3,%llvmcheckext,CHECK-EP-PIPELINE-EARLY-SIMPLIFICATION,CHECK-O23SZ
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes-ep-pipeline-start='no-op-module' \
 ; RUN:     -passes='lto-pre-link<O3>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-LTO,CHECK-O123,CHECK-O3,%llvmcheckext,CHECK-EP-PIPELINE-START,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-LTO,CHECK-NO-FUNC-SPEC,CHECK-O3,%llvmcheckext,CHECK-EP-PIPELINE-START,CHECK-O23SZ
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes-ep-optimizer-early='no-op-module' \
 ; RUN:     -passes='default<O3>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-O123,CHECK-O3,%llvmcheckext,CHECK-EP-OPTIMIZER-EARLY,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-FUNC-SPEC,CHECK-O3,%llvmcheckext,CHECK-EP-OPTIMIZER-EARLY,CHECK-O23SZ
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes-ep-optimizer-last='no-op-module' \
 ; RUN:     -passes='default<O3>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-O123,CHECK-O3,%llvmcheckext,CHECK-EP-OPTIMIZER-LAST,CHECK-O23SZ
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-FUNC-SPEC,CHECK-O3,%llvmcheckext,CHECK-EP-OPTIMIZER-LAST,CHECK-O23SZ
 
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='default<O3>' -enable-matrix -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-O123,CHECK-O3,CHECK-O23SZ,%llvmcheckext,CHECK-MATRIX
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-FUNC-SPEC,CHECK-O3,CHECK-O23SZ,%llvmcheckext,CHECK-MATRIX
 
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='default<O3>' -enable-merge-functions -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-O123,CHECK-O3,CHECK-O23SZ,%llvmcheckext,CHECK-MERGE-FUNCS
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-FUNC-SPEC,CHECK-O3,CHECK-O23SZ,%llvmcheckext,CHECK-MERGE-FUNCS
 
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='default<O3>' -ir-outliner -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-O123,CHECK-O3,CHECK-O23SZ,%llvmcheckext,CHECK-IR-OUTLINER
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-FUNC-SPEC,CHECK-O3,CHECK-O23SZ,%llvmcheckext,CHECK-IR-OUTLINER
 
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='default<O3>' -hot-cold-split -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-O123,CHECK-O3,CHECK-O23SZ,%llvmcheckext,CHECK-HOT-COLD-SPLIT
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-DEFAULT,CHECK-FUNC-SPEC,CHECK-O3,CHECK-O23SZ,%llvmcheckext,CHECK-HOT-COLD-SPLIT
 
 ; Suppress FileCheck --allow-unused-prefixes=false diagnostics.
 ; CHECK-Oz: {{^}}
@@ -109,7 +109,7 @@
 ; CHECK-O-NEXT: Running pass: OpenMPOptPass
 ; CHECK-EP-PIPELINE-EARLY-SIMPLIFICATION-NEXT: Running pass: NoOpModulePass
 ; CHECK-O-NEXT: Running pass: IPSCCPPass
-; CHECK-O123-NEXT: Running analysis: LoopAnalysis
+; CHECK-FUNC-SPEC-NEXT: Running analysis: LoopAnalysis
 ; CHECK-O-NEXT: Running pass: CalledValuePropagationPass
 ; CHECK-O-NEXT: Running pass: GlobalOptPass
 ; CHECK-O-NEXT: Running pass: PromotePass
@@ -139,7 +139,6 @@
 ; CHECK-O-NEXT: Running pass: InlinerPass
 ; CHECK-O-NEXT: Running pass: InlinerPass
 ; CHECK-O-NEXT: Running pass: PostOrderFunctionAttrsPass
-; CHECK-O-NEXT: Running analysis: AAManager
 ; CHECK-O3-NEXT: Running pass: ArgumentPromotionPass
 ; CHECK-O2-NEXT: Running pass: OpenMPOptCGSCCPass on (foo)
 ; CHECK-O3-NEXT: Running pass: OpenMPOptCGSCCPass on (foo)
@@ -147,6 +146,7 @@
 ; CHECK-O-NEXT: Running pass: SROAPass
 ; CHECK-O-NEXT: Running pass: EarlyCSEPass
 ; CHECK-O-NEXT: Running analysis: MemorySSAAnalysis
+; CHECK-O-NEXT: Running analysis: AAManager
 ; CHECK-O23SZ-NEXT: Running pass: SpeculativeExecutionPass
 ; CHECK-O23SZ-NEXT: Running pass: JumpThreadingPass
 ; CHECK-O23SZ-NEXT: Running analysis: LazyValueAnalysis
@@ -165,7 +165,7 @@
 ; CHECK-O-NEXT: Running pass: ReassociatePass
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}OptimizationRemarkEmitterAnalysis
 ; CHECK-O-NEXT: Running pass: LoopSimplifyPass
-; CHECK-OSZ-NEXT: Running analysis: LoopAnalysis
+; CHECK-NO-FUNC-SPEC-NEXT: Running analysis: LoopAnalysis
 ; CHECK-O-NEXT: Running pass: LCSSAPass
 ; CHECK-O-NEXT: Running analysis: ScalarEvolutionAnalysis
 ; CHECK-O-NEXT: Running analysis: InnerAnalysisManagerProxy
@@ -214,6 +214,8 @@
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
 ; CHECK-EP-PEEPHOLE-NEXT: Running pass: NoOpFunctionPass
+; CHECK-O-NEXT: Running pass: PostOrderFunctionAttrsPass
+; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}ShouldNotRunFunctionPassesAnalysis
 ; CHECK-O-NEXT: Running analysis: ShouldNotRunFunctionPassesAnalysis
 ; CHECK-O-NEXT: Running pass: CoroSplitPass
 ; CHECK-O-NEXT: Running pass: InvalidateAnalysisPass<{{.*}}ShouldNotRunFunctionPassesAnalysis
diff --git a/llvm/test/Other/new-pm-lto-defaults.ll b/llvm/test/Other/new-pm-lto-defaults.ll
index f93a39b9578c1..2e0cd5669c78a 100644
--- a/llvm/test/Other/new-pm-lto-defaults.ll
+++ b/llvm/test/Other/new-pm-lto-defaults.ll
@@ -1,29 +1,29 @@
 ; Basic test for the new LTO pipeline.
 
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='lto<O1>' -S %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O1
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='lto<O1>' -S %s -passes-ep-full-link-time-optimization-early=no-op-module \
 ; RUN:     -passes-ep-full-link-time-optimization-last=no-op-module 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O1,CHECK-EP
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='lto<O2>' -S  %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O2,CHECK-O23,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='lto<O2>' -S %s -passes-ep-full-link-time-optimization-early=no-op-module \
 ; RUN:     -passes-ep-full-link-time-optimization-last=no-op-module 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O2,CHECK-O23,CHECK-O23SZ,CHECK-EP
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='lto<O3>' -S  %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O3,CHECK-O23,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='lto<Os>' -S %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-OS,CHECK-OSZ,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='lto<Oz>' -S %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-OSZ,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='lto<O3>' -S  %s -passes-ep-peephole='no-op-function' 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O3,CHECK-O23,CHECK-O23SZ,CHECK-EP-Peephole
 
@@ -68,6 +68,7 @@
 ; CHECK-O23SZ-NEXT: Running pass: GlobalOptPass
 ; CHECK-O23SZ-NEXT: Running pass: PromotePass
 ; CHECK-O23SZ-NEXT: Running pass: ConstantMergePass
+; CHECK-O23SZ-NEXT: Running pass: DeadArgumentEliminationPass
 ; CHECK-O23SZ-NEXT: Running pass: InstCombinePass
 ; CHECK-O3-NEXT: Running pass: AggressiveInstCombinePass
 ; CHECK-EP-Peephole-NEXT: Running pass: NoOpFunctionPass
@@ -80,7 +81,6 @@
 ; CHECK-O23SZ-NEXT: Running pass: OpenMPOptPass
 ; CHECK-O23SZ-NEXT: Running pass: GlobalDCEPass
 ; CHECK-O23SZ-NEXT: Running pass: ArgumentPromotionPass
-; CHECK-O23SZ-NEXT: Running pass: DeadArgumentEliminationPass
 ; CHECK-O23SZ-NEXT: Running pass: InstCombinePass
 ; CHECK-EP-Peephole-NEXT: Running pass: NoOpFunctionPass
 ; CHECK-O23SZ-NEXT: Running pass: ConstraintEliminationPass
diff --git a/llvm/test/Other/new-pm-pgo.ll b/llvm/test/Other/new-pm-pgo.ll
index b5de48a847414..c4926f4ce648d 100644
--- a/llvm/test/Other/new-pm-pgo.ll
+++ b/llvm/test/Other/new-pm-pgo.ll
@@ -24,7 +24,6 @@
 ; SAMPLE_USE: Running pass: SimplifyCFGPass
 ; SAMPLE_USE: Running pass: SROAPass
 ; SAMPLE_USE: Running pass: EarlyCSEPass
-; SAMPLE_USE_POST_LINK: Running pass: InstCombinePass
 ; SAMPLE_USE: Running pass: SampleProfileLoaderPass
 ; SAMPLE_USE_O: Running pass: PGOIndirectCallPromotion
 ; SAMPLE_USE_POST_LINK-NOT: Running pass: GlobalOptPass
diff --git a/llvm/test/Other/new-pm-print-pipeline.ll b/llvm/test/Other/new-pm-print-pipeline.ll
index 3ddc3698ea175..2d52569d6d8ea 100644
--- a/llvm/test/Other/new-pm-print-pipeline.ll
+++ b/llvm/test/Other/new-pm-print-pipeline.ll
@@ -97,3 +97,17 @@
 ;; Test InstCombine options - the first pass checks default settings, and the second checks customized options.
 ; RUN: opt -disable-output -disable-verify -print-pipeline-passes -passes='function(instcombine,instcombine<use-loop-info;max-iterations=42>)' < %s | FileCheck %s --match-full-lines --check-prefixes=CHECK-28
 ; CHECK-28: function(instcombine<max-iterations=1000;no-use-loop-info>,instcombine<max-iterations=42;use-loop-info>)
+
+;; Test function-attrs
+; RUN: opt -disable-output -disable-verify -print-pipeline-passes -passes='cgscc(function-attrs<skip-non-recursive>)' < %s | FileCheck %s --match-full-lines --check-prefixes=CHECK-29
+; CHECK-29: cgscc(function-attrs<skip-non-recursive>)
+
+;; Test cgscc -> function adaptor
+; RUN: opt -disable-output -disable-verify -print-pipeline-passes -passes='cgscc(function<eager-inv;no-rerun>(no-op-function))' < %s | FileCheck %s --match-full-lines --check-prefixes=CHECK-30
+; CHECK-30: cgscc(function<eager-inv;no-rerun>(no-op-function))
+
+; RUN: opt -disable-output -disable-verify -print-pipeline-passes -passes='cgscc(function<no-rerun;eager-inv>(no-op-function))' < %s | FileCheck %s --match-full-lines --check-prefixes=CHECK-31
+; CHECK-31: cgscc(function<eager-inv;no-rerun>(no-op-function))
+
+; RUN: opt -disable-output -disable-verify -print-pipeline-passes -passes='cgscc(function<no-rerun>(no-op-function))' < %s | FileCheck %s --match-full-lines --check-prefixes=CHECK-32
+; CHECK-32: cgscc(function<no-rerun>(no-op-function))
diff --git a/llvm/test/Other/new-pm-thinlto-defaults.ll b/llvm/test/Other/new-pm-thinlto-defaults.ll
index 6d73544b191c0..e89a62112db77 100644
--- a/llvm/test/Other/new-pm-thinlto-defaults.ll
+++ b/llvm/test/Other/new-pm-thinlto-defaults.ll
@@ -8,56 +8,56 @@
 ; the IR, in which case we need to make it immutable harder.
 ;
 ; Prelink pipelines:
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='thinlto-pre-link<O1>' -S %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O1,CHECK-PRELINK-O,CHECK-PRELINK-O-NODIS
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-NO-FUNC-SPEC,CHECK-O1,CHECK-PRELINK-O,CHECK-PRELINK-O-NODIS
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='thinlto-pre-link<O2>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O2,CHECK-O23SZ,CHECK-PRELINK-O,CHECK-PRELINK-O-NODIS
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-NO-FUNC-SPEC,CHECK-O2,CHECK-O23SZ,CHECK-PRELINK-O,CHECK-PRELINK-O-NODIS
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='thinlto-pre-link<O3>' -S -passes-ep-pipeline-start='no-op-module' %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O3,CHECK-O23SZ,CHECK-PRELINK-O,CHECK-PRELINK-O-NODIS,CHECK-EP-PIPELINE-START
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-NO-FUNC-SPEC,CHECK-O3,CHECK-O23SZ,CHECK-PRELINK-O,CHECK-PRELINK-O-NODIS,CHECK-EP-PIPELINE-START
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='thinlto-pre-link<O3>' -S -passes-ep-optimizer-early='no-op-module' %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O3,CHECK-O23SZ,CHECK-PRELINK-O,CHECK-PRELINK-O-NODIS,CHECK-EP-OPT-EARLY
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-NO-FUNC-SPEC,CHECK-O3,CHECK-O23SZ,CHECK-PRELINK-O,CHECK-PRELINK-O-NODIS,CHECK-EP-OPT-EARLY
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='thinlto-pre-link<O3>' -S -passes-ep-optimizer-last='no-op-module' %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O3,CHECK-O23SZ,CHECK-PRELINK-O,CHECK-PRELINK-O-NODIS,CHECK-EP-OPT-LAST
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-NO-FUNC-SPEC,CHECK-O3,CHECK-O23SZ,CHECK-PRELINK-O,CHECK-PRELINK-O-NODIS,CHECK-EP-OPT-LAST
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='thinlto-pre-link<Os>' -S %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-OSZ,CHECK-O23SZ,CHECK-PRELINK-O,CHECK-PRELINK-O-NODIS
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-NO-FUNC-SPEC,CHECK-O23SZ,CHECK-PRELINK-O,CHECK-PRELINK-O-NODIS
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='thinlto-pre-link<Oz>' -S %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-OSZ,CHECK-O23SZ,CHECK-PRELINK-O,CHECK-PRELINK-O-NODIS
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -debug-info-for-profiling \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-NO-FUNC-SPEC,CHECK-O23SZ,CHECK-PRELINK-O,CHECK-PRELINK-O-NODIS
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -debug-info-for-profiling \
 ; RUN:     -passes='thinlto-pre-link<O2>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-DIS,CHECK-O,CHECK-O123,CHECK-O2,CHECK-O23SZ,CHECK-PRELINK-O
+; RUN:     | FileCheck %s --check-prefixes=CHECK-DIS,CHECK-O,CHECK-NO-FUNC-SPEC,CHECK-O2,CHECK-O23SZ,CHECK-PRELINK-O
 ;
 ; Postlink pipelines:
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='thinlto<O1>' -S %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O1,CHECK-POSTLINK-O,%llvmcheckext
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-FUNC-SPEC,CHECK-O1,CHECK-POSTLINK-O,%llvmcheckext
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='thinlto<O2>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O2,CHECK-O23SZ,CHECK-POSTLINK-O,%llvmcheckext,CHECK-POSTLINK-O2
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -passes-ep-pipeline-start='no-op-module' \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-FUNC-SPEC,CHECK-O2,CHECK-O23SZ,CHECK-POSTLINK-O,%llvmcheckext,CHECK-POSTLINK-O2
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -passes-ep-pipeline-start='no-op-module' \
 ; RUN:     -passes='thinlto<O3>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O3,CHECK-O23SZ,CHECK-POSTLINK-O,%llvmcheckext,CHECK-POSTLINK-O3
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -passes-ep-optimizer-early='no-op-module' \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-FUNC-SPEC,CHECK-O3,CHECK-O23SZ,CHECK-POSTLINK-O,%llvmcheckext,CHECK-POSTLINK-O3
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -passes-ep-optimizer-early='no-op-module' \
 ; RUN:     -passes='thinlto<O3>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O3,CHECK-O23SZ,CHECK-POSTLINK-O,%llvmcheckext,CHECK-POSTLINK-O3,CHECK-POST-EP-OPT-EARLY
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -passes-ep-optimizer-last='no-op-module' \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-FUNC-SPEC,CHECK-O3,CHECK-O23SZ,CHECK-POSTLINK-O,%llvmcheckext,CHECK-POSTLINK-O3,CHECK-POST-EP-OPT-EARLY
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -passes-ep-optimizer-last='no-op-module' \
 ; RUN:     -passes='thinlto<O3>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O3,CHECK-O23SZ,CHECK-POSTLINK-O,%llvmcheckext,CHECK-POSTLINK-O3,CHECK-POST-EP-OPT-LAST
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-FUNC-SPEC,CHECK-O3,CHECK-O23SZ,CHECK-POSTLINK-O,%llvmcheckext,CHECK-POSTLINK-O3,CHECK-POST-EP-OPT-LAST
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='thinlto<Os>' -S %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-OSZ,CHECK-O23SZ,CHECK-POSTLINK-O,%llvmcheckext,CHECK-POSTLINK-Os
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-NO-FUNC-SPEC,CHECK-O23SZ,CHECK-POSTLINK-O,%llvmcheckext,CHECK-POSTLINK-Os
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='thinlto<Oz>' -S %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-OSZ,CHECK-O23SZ,CHECK-POSTLINK-O,%llvmcheckext
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -debug-info-for-profiling \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-NO-FUNC-SPEC,CHECK-O23SZ,CHECK-POSTLINK-O,%llvmcheckext
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -debug-info-for-profiling \
 ; RUN:     -passes='thinlto<O2>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O2,CHECK-O23SZ,CHECK-POSTLINK-O,%llvmcheckext,CHECK-POSTLINK-O2
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-FUNC-SPEC,CHECK-O2,CHECK-O23SZ,CHECK-POSTLINK-O,%llvmcheckext,CHECK-POSTLINK-O2
 
 ; Suppress FileCheck --allow-unused-prefixes=false diagnostics.
 ; CHECK-NOEXT: {{^}}
@@ -87,7 +87,7 @@
 ; CHECK-O-NEXT: Running pass: OpenMPOptPass
 ; CHECK-POSTLINK-O-NEXT: Running pass: LowerTypeTestsPass
 ; CHECK-O-NEXT: Running pass: IPSCCPPass
-; CHECK-O123-NEXT: Running analysis: LoopAnalysis
+; CHECK-FUNC-SPEC-NEXT: Running analysis: LoopAnalysis
 ; CHECK-O-NEXT: Running pass: CalledValuePropagationPass
 ; CHECK-O-NEXT: Running pass: GlobalOptPass
 ; CHECK-O-NEXT: Running pass: PromotePass
@@ -116,13 +116,13 @@
 ; CHECK-O-NEXT: Running pass: InlinerPass
 ; CHECK-O-NEXT: Running pass: InlinerPass
 ; CHECK-O-NEXT: Running pass: PostOrderFunctionAttrsPass
-; CHECK-O-NEXT: Running analysis: AAManager
 ; CHECK-O3-NEXT: Running pass: ArgumentPromotionPass
 ; CHECK-O2-NEXT: Running pass: OpenMPOptCGSCCPass on (foo)
 ; CHECK-O3-NEXT: Running pass: OpenMPOptCGSCCPass on (foo)
 ; CHECK-O-NEXT: Running pass: SROAPass
 ; CHECK-O-NEXT: Running pass: EarlyCSEPass
 ; CHECK-O-NEXT: Running analysis: MemorySSAAnalysis
+; CHECK-O-NEXT: Running analysis: AAManager
 ; CHECK-O23SZ-NEXT: Running pass: SpeculativeExecutionPass
 ; CHECK-O23SZ-NEXT: Running pass: JumpThreadingPass
 ; CHECK-O23SZ-NEXT: Running analysis: LazyValueAnalysis
@@ -140,7 +140,7 @@
 ; CHECK-O-NEXT: Running pass: ReassociatePass
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}OptimizationRemarkEmitterAnalysis
 ; CHECK-O-NEXT: Running pass: LoopSimplifyPass
-; CHECK-OSZ-NEXT: Running analysis: LoopAnalysis
+; CHECK-NO-FUNC-SPEC-NEXT: Running analysis: LoopAnalysis
 ; CHECK-O-NEXT: Running pass: LCSSAPass
 ; CHECK-O-NEXT: Running analysis: ScalarEvolutionAnalysis
 ; CHECK-O-NEXT: Running analysis: InnerAnalysisManagerProxy
@@ -184,6 +184,8 @@
 ; CHECK-O23SZ-NEXT: Running pass: CoroElidePass
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
+; CHECK-O-NEXT: Running pass: PostOrderFunctionAttrsPass
+; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}ShouldNotRunFunctionPassesAnalysis
 ; CHECK-O-NEXT: Running analysis: ShouldNotRunFunctionPassesAnalysis
 ; CHECK-O-NEXT: Running pass: CoroSplitPass
 ; CHECK-O-NEXT: Running pass: InvalidateAnalysisPass<{{.*}}ShouldNotRunFunctionPassesAnalysis
diff --git a/llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll b/llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll
index fddb16e387f60..c1e2fd2230056 100644
--- a/llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll
+++ b/llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll
@@ -1,22 +1,22 @@
 ; Validate ThinLTO post link pipeline when we have instrumentation PGO
 ;
 ; Postlink pipelines:
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='thinlto<O1>' -S %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O1,%llvmcheckext
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='thinlto<O2>' -S  %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O2,CHECK-O23SZ,%llvmcheckext
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -passes-ep-pipeline-start='no-op-module' \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -passes-ep-pipeline-start='no-op-module' \
 ; RUN:     -passes='thinlto<O3>' -S  %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O3,CHECK-O23SZ,%llvmcheckext
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='thinlto<Os>' -S %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-OSZ,CHECK-Os,CHECK-O23SZ,%llvmcheckext
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -passes='thinlto<Oz>' -S %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-OSZ,CHECK-O23SZ,%llvmcheckext
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -debug-info-for-profiling \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -debug-info-for-profiling \
 ; RUN:     -passes='thinlto<O2>' -S  %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O2,CHECK-O23SZ,%llvmcheckext
 
@@ -79,13 +79,13 @@
 ; CHECK-O-NEXT: Running pass: InlinerPass
 ; CHECK-O-NEXT: Running pass: InlinerPass
 ; CHECK-O-NEXT: Running pass: PostOrderFunctionAttrsPass
-; CHECK-O-NEXT: Running analysis: AAManager
 ; CHECK-O3-NEXT: Running pass: ArgumentPromotionPass
 ; CHECK-O2-NEXT: Running pass: OpenMPOptCGSCCPass
 ; CHECK-O3-NEXT: Running pass: OpenMPOptCGSCCPass
 ; CHECK-O-NEXT: Running pass: SROAPass
 ; CHECK-O-NEXT: Running pass: EarlyCSEPass
 ; CHECK-O-NEXT: Running analysis: MemorySSAAnalysis
+; CHECK-O-NEXT: Running analysis: AAManager
 ; CHECK-O23SZ-NEXT: Running pass: SpeculativeExecutionPass
 ; CHECK-O23SZ-NEXT: Running pass: JumpThreadingPass
 ; CHECK-O23SZ-NEXT: Running analysis: LazyValueAnalysis
@@ -145,6 +145,8 @@
 ; CHECK-O23SZ-NEXT: Running pass: CoroElidePass
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
+; CHECK-O-NEXT: Running pass: PostOrderFunctionAttrsPass
+; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}ShouldNotRunFunctionPassesAnalysis
 ; CHECK-O-NEXT: Running analysis: ShouldNotRunFunctionPassesAnalysis
 ; CHECK-O-NEXT: Running pass: CoroSplitPass
 ; CHECK-O-NEXT: Running pass: InvalidateAnalysisPass<{{.*}}ShouldNotRunFunctionPassesAnalysis
diff --git a/llvm/test/Other/new-pm-thinlto-postlink-samplepgo-defaults.ll b/llvm/test/Other/new-pm-thinlto-postlink-samplepgo-defaults.ll
index eb79d471eea97..a275cf18cbf95 100644
--- a/llvm/test/Other/new-pm-thinlto-postlink-samplepgo-defaults.ll
+++ b/llvm/test/Other/new-pm-thinlto-postlink-samplepgo-defaults.ll
@@ -1,26 +1,26 @@
 ; Validate ThinLTO postlink pipeline when we have Sample PGO
 ;
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -pgo-kind=pgo-sample-use-pipeline -profile-file='%S/Inputs/new-pm-thinlto-samplepgo-defaults.prof' \
 ; RUN:     -passes='thinlto<O1>' -S %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O1,%llvmcheckext
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -pgo-kind=pgo-sample-use-pipeline -profile-file='%S/Inputs/new-pm-thinlto-samplepgo-defaults.prof' \
 ; RUN:     -passes='thinlto<O2>' -S  %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O2,CHECK-O23SZ,%llvmcheckext
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -passes-ep-pipeline-start='no-op-module' \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -passes-ep-pipeline-start='no-op-module' \
 ; RUN:     -pgo-kind=pgo-sample-use-pipeline -profile-file='%S/Inputs/new-pm-thinlto-samplepgo-defaults.prof' \
 ; RUN:     -passes='thinlto<O3>' -S  %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O3,CHECK-O23SZ,%llvmcheckext
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -pgo-kind=pgo-sample-use-pipeline -profile-file='%S/Inputs/new-pm-thinlto-samplepgo-defaults.prof' \
 ; RUN:     -passes='thinlto<Os>' -S %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-OSZ,CHECK-Os,CHECK-O23SZ,%llvmcheckext
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -pgo-kind=pgo-sample-use-pipeline -profile-file='%S/Inputs/new-pm-thinlto-samplepgo-defaults.prof' \
 ; RUN:     -passes='thinlto<Oz>' -S %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-OSZ,CHECK-O23SZ,%llvmcheckext
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -debug-info-for-profiling \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -debug-info-for-profiling \
 ; RUN:     -pgo-kind=pgo-sample-use-pipeline -profile-file='%S/Inputs/new-pm-thinlto-samplepgo-defaults.prof' \
 ; RUN:     -passes='thinlto<O2>' -S  %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O2,CHECK-O23SZ,%llvmcheckext
@@ -44,18 +44,12 @@
 ; CHECK-O-NEXT: Running pass: EarlyCSEPass
 ; CHECK-O-NEXT: Running analysis: TargetLibraryAnalysis
 ; CHECK-O3-NEXT: Running pass: CallSiteSplittingPass
-; CHECK-O-NEXT: Running pass: InstCombinePass on foo
-; CHECK-O-NEXT: Running analysis: OptimizationRemarkEmitterAnalysis on foo
-; CHECK-O-NEXT: Running analysis: AAManager on foo
-; CHECK-O-NEXT: Running analysis: BasicAA
-; CHECK-O-NEXT: Running analysis: ScopedNoAliasAA
-; CHECK-O-NEXT: Running analysis: TypeBasedAA
-; CHECK-O-NEXT: Running analysis: OuterAnalysisManagerProxy
 ; CHECK-O-NEXT: Running pass: SampleProfileLoaderPass
 ; CHECK-O-NEXT: Running analysis: ProfileSummaryAnalysis
 ; CHECK-O-NEXT: Running analysis: CallGraphAnalysis
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}ProfileSummaryAnalysis
 ; CHECK-O-NEXT: Running pass: PGOIndirectCallPromotion
+; CHECK-O-NEXT: Running analysis: OptimizationRemarkEmitterAnalysis
 ; CHECK-O-NEXT: Running pass: OpenMPOptPass
 ; CHECK-O-NEXT: Running pass: LowerTypeTestsPass
 ; CHECK-O-NEXT: Running pass: IPSCCPPass
@@ -64,6 +58,11 @@
 ; CHECK-O-NEXT: Running pass: GlobalOptPass
 ; CHECK-O-NEXT: Running pass: PromotePass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
+; CHECK-O-NEXT: Running analysis: AAManager on foo
+; CHECK-O-NEXT: Running analysis: BasicAA
+; CHECK-O-NEXT: Running analysis: ScopedNoAliasAA
+; CHECK-O-NEXT: Running analysis: TypeBasedAA
+; CHECK-O-NEXT: Running analysis: OuterAnalysisManagerProxy
 ; CHECK-O-NEXT: Running analysis: BlockFrequencyAnalysis on foo
 ; These next two can appear in any order since they are accessed as parameters
 ; on the same call to BlockFrequencyInfo::calculate.
@@ -88,13 +87,13 @@
 ; CHECK-O-NEXT: Running pass: InlinerPass
 ; CHECK-O-NEXT: Running pass: InlinerPass
 ; CHECK-O-NEXT: Running pass: PostOrderFunctionAttrsPass
-; CHECK-O-NEXT: Running analysis: AAManager
 ; CHECK-O3-NEXT: Running pass: ArgumentPromotionPass
 ; CHECK-O2-NEXT: Running pass: OpenMPOptCGSCCPass
 ; CHECK-O3-NEXT: Running pass: OpenMPOptCGSCCPass
 ; CHECK-O-NEXT: Running pass: SROAPass
 ; CHECK-O-NEXT: Running pass: EarlyCSEPass
 ; CHECK-O-NEXT: Running analysis: MemorySSAAnalysis
+; CHECK-O-NEXT: Running analysis: AAManager
 ; CHECK-O23SZ-NEXT: Running pass: SpeculativeExecutionPass
 ; CHECK-O23SZ-NEXT: Running pass: JumpThreadingPass
 ; CHECK-O23SZ-NEXT: Running analysis: LazyValueAnalysis
@@ -154,6 +153,8 @@
 ; CHECK-O23SZ-NEXT: Running pass: CoroElidePass
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
+; CHECK-O-NEXT: Running pass: PostOrderFunctionAttrsPass
+; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}ShouldNotRunFunctionPassesAnalysis
 ; CHECK-O-NEXT: Running analysis: ShouldNotRunFunctionPassesAnalysis
 ; CHECK-O-NEXT: Running pass: CoroSplitPass
 ; CHECK-O-NEXT: Running pass: InvalidateAnalysisPass<{{.*}}ShouldNotRunFunctionPassesAnalysis
diff --git a/llvm/test/Other/new-pm-thinlto-prelink-pgo-defaults.ll b/llvm/test/Other/new-pm-thinlto-prelink-pgo-defaults.ll
index 4153502b5180b..f6fbeff0acb44 100644
--- a/llvm/test/Other/new-pm-thinlto-prelink-pgo-defaults.ll
+++ b/llvm/test/Other/new-pm-thinlto-prelink-pgo-defaults.ll
@@ -2,30 +2,30 @@
 ;
 ; RUN: llvm-profdata merge %S/Inputs/new-pm-thinlto-prelink-pgo-defaults.proftext -o %t.profdata
 ;
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -pgo-kind=pgo-instr-use-pipeline -profile-file='%t.profdata' \
 ; RUN:     -passes='thinlto-pre-link<O1>' -S %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O1,CHECK-O123SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O1,CHECK-O123SZ
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -pgo-kind=pgo-instr-use-pipeline -profile-file='%t.profdata' \
 ; RUN:     -passes='thinlto-pre-link<O2>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O2,CHECK-O23SZ,CHECK-O123SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O2,CHECK-O23SZ,CHECK-O123SZ
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -pgo-kind=pgo-instr-use-pipeline -profile-file='%t.profdata' \
 ; RUN:     -passes='thinlto-pre-link<O3>' -S -passes-ep-pipeline-start='no-op-module' %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O3,CHECK-O23SZ,CHECK-O123SZ,CHECK-EP-PIPELINE-START
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O3,CHECK-O23SZ,CHECK-O123SZ,CHECK-EP-PIPELINE-START
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -pgo-kind=pgo-instr-use-pipeline -profile-file='%t.profdata' \
 ; RUN:     -passes='thinlto-pre-link<Os>' -S %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-OSZ,CHECK-O123SZ,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123SZ,CHECK-O23SZ
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -pgo-kind=pgo-instr-use-pipeline -profile-file='%t.profdata' \
 ; RUN:     -passes='thinlto-pre-link<Oz>' -S %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-OSZ,CHECK-O123SZ,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -debug-info-for-profiling \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123SZ,CHECK-O23SZ
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -debug-info-for-profiling \
 ; RUN:     -pgo-kind=pgo-instr-use-pipeline -profile-file='%t.profdata' \
 ; RUN:     -passes='thinlto-pre-link<O2>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O2,CHECK-O23SZ,CHECK-O123SZ
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O2,CHECK-O23SZ,CHECK-O123SZ
 ;
 ; CHECK-O: Running pass: Annotation2Metadata
 ; CHECK-O-NEXT: Running pass: ForceFunctionAttrsPass
@@ -45,7 +45,6 @@
 ; CHECK-O3-NEXT: Running pass: CallSiteSplittingPass
 ; CHECK-O-NEXT: Running pass: OpenMPOptPass
 ; CHECK-O-NEXT: Running pass: IPSCCPPass
-; CHECK-O123-NEXT: Running analysis: LoopAnalysis
 ; CHECK-O-NEXT: Running pass: CalledValuePropagationPass
 ; CHECK-O-NEXT: Running pass: GlobalOptPass
 ; CHECK-O-NEXT: Running pass: PromotePass
@@ -74,11 +73,9 @@
 ; CHECK-O-NEXT: Running pass: PGOInstrumentationUse
 ; These next two can appear in any order since they are accessed as parameters
 ; on the same call to BlockFrequencyInfo::calculate.
-; CHECK-OSZ-DAG: Running analysis: BranchProbabilityAnalysis on foo
-; CHECK-OSZ-DAG: Running analysis: PostDominatorTreeAnalysis on foo
-; CHECK-OSZ-DAG: Running analysis: LoopAnalysis on foo
-; CHECK-O123-DAG: Running analysis: BranchProbabilityAnalysis on foo
-; CHECK-O123-DAG: Running analysis: PostDominatorTreeAnalysis on foo
+; CHECK-O-DAG: Running analysis: BranchProbabilityAnalysis on foo
+; CHECK-O-DAG: Running analysis: PostDominatorTreeAnalysis on foo
+; CHECK-O-DAG: Running analysis: LoopAnalysis on foo
 ; CHECK-O-NEXT: Running analysis: BlockFrequencyAnalysis on foo
 ; CHECK-O-NEXT: Invalidating analysis: InnerAnalysisManagerProxy
 ; CHECK-O123SZ-NEXT: Invalidating analysis: LazyCallGraphAnalysis on
@@ -103,20 +100,20 @@
 ; CHECK-O-NEXT: Running pass: InlinerPass
 ; CHECK-O-NEXT: Running pass: InlinerPass
 ; CHECK-O-NEXT: Running pass: PostOrderFunctionAttrsPass
-; CHECK-O-NEXT: Running analysis: AAManager
-; CHECK-O-NEXT: Running analysis: BasicAA
-; CHECK-O-NEXT: Running analysis: AssumptionAnalysis
-; CHECK-O-NEXT: Running analysis: TargetIRAnalysis
-; CHECK-O-NEXT: Running analysis: DominatorTreeAnalysis
-; CHECK-O-NEXT: Running analysis: ScopedNoAliasAA
-; CHECK-O-NEXT: Running analysis: TypeBasedAA
-; CHECK-O-NEXT: Running analysis: OuterAnalysisManagerProxy
 ; CHECK-O3-NEXT: Running pass: ArgumentPromotionPass
 ; CHECK-O2-NEXT: Running pass: OpenMPOptCGSCCPass
 ; CHECK-O3-NEXT: Running pass: OpenMPOptCGSCCPass
 ; CHECK-O-NEXT: Running pass: SROAPass
+; CHECK-O-NEXT: Running analysis: DominatorTreeAnalysis
+; CHECK-O-NEXT: Running analysis: AssumptionAnalysis
+; CHECK-O-NEXT: Running analysis: TargetIRAnalysis
 ; CHECK-O-NEXT: Running pass: EarlyCSEPass
 ; CHECK-O-NEXT: Running analysis: MemorySSAAnalysis
+; CHECK-O-NEXT: Running analysis: AAManager
+; CHECK-O-NEXT: Running analysis: BasicAA
+; CHECK-O-NEXT: Running analysis: ScopedNoAliasAA
+; CHECK-O-NEXT: Running analysis: TypeBasedAA
+; CHECK-O-NEXT: Running analysis: OuterAnalysisManagerProxy
 ; CHECK-O23SZ-NEXT: Running pass: SpeculativeExecutionPass
 ; CHECK-O23SZ-NEXT: Running pass: JumpThreadingPass
 ; CHECK-O23SZ-NEXT: Running analysis: LazyValueAnalysis
@@ -184,6 +181,8 @@
 ; CHECK-O23SZ-NEXT: Running pass: CoroElidePass
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
+; CHECK-O-NEXT: Running pass: PostOrderFunctionAttrsPass
+; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}ShouldNotRunFunctionPassesAnalysis
 ; CHECK-O-NEXT: Running analysis: ShouldNotRunFunctionPassesAnalysis
 ; CHECK-O-NEXT: Running pass: CoroSplitPass
 ; CHECK-O-NEXT: Running pass: InvalidateAnalysisPass<{{.*}}ShouldNotRunFunctionPassesAnalysis
diff --git a/llvm/test/Other/new-pm-thinlto-prelink-samplepgo-defaults.ll b/llvm/test/Other/new-pm-thinlto-prelink-samplepgo-defaults.ll
index 663af16ae836f..ec4e1236abb6c 100644
--- a/llvm/test/Other/new-pm-thinlto-prelink-samplepgo-defaults.ll
+++ b/llvm/test/Other/new-pm-thinlto-prelink-samplepgo-defaults.ll
@@ -1,29 +1,29 @@
 ; Validate ThinLTO prelink pipeline when we have Sample PGO
 ;
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -pgo-kind=pgo-sample-use-pipeline -profile-file='%S/Inputs/new-pm-thinlto-samplepgo-defaults.prof' \
 ; RUN:     -passes='thinlto-pre-link<O1>' -S %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O1
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O1
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -pgo-kind=pgo-sample-use-pipeline -profile-file='%S/Inputs/new-pm-thinlto-samplepgo-defaults.prof' \
 ; RUN:     -passes='thinlto-pre-link<O2>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O2,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O2,CHECK-O23SZ
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -pgo-kind=pgo-sample-use-pipeline -profile-file='%S/Inputs/new-pm-thinlto-samplepgo-defaults.prof' \
 ; RUN:     -passes='thinlto-pre-link<O3>' -S -passes-ep-pipeline-start='no-op-module' %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O3,CHECK-O23SZ,CHECK-EP-PIPELINE-START
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O3,CHECK-O23SZ,CHECK-EP-PIPELINE-START
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -pgo-kind=pgo-sample-use-pipeline -profile-file='%S/Inputs/new-pm-thinlto-samplepgo-defaults.prof' \
 ; RUN:     -passes='thinlto-pre-link<Os>' -S %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-OSZ,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O23SZ
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager \
 ; RUN:     -pgo-kind=pgo-sample-use-pipeline -profile-file='%S/Inputs/new-pm-thinlto-samplepgo-defaults.prof' \
 ; RUN:     -passes='thinlto-pre-link<Oz>' -S %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-OSZ,CHECK-O23SZ
-; RUN: opt -disable-verify -verify-cfg-preserved=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -debug-info-for-profiling \
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O23SZ
+; RUN: opt -disable-verify -verify-analysis-invalidation=0 -eagerly-invalidate-analyses=0 -debug-pass-manager -debug-info-for-profiling \
 ; RUN:     -pgo-kind=pgo-sample-use-pipeline -profile-file='%S/Inputs/new-pm-thinlto-samplepgo-defaults.prof' \
 ; RUN:     -passes='thinlto-pre-link<O2>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O123,CHECK-O2,CHECK-O23SZ
+; RUN:     | FileCheck %s --check-prefixes=CHECK-O,CHECK-O2,CHECK-O23SZ
 ;
 ; CHECK-O: Running pass: Annotation2Metadata
 ; CHECK-O-NEXT: Running pass: ForceFunctionAttrsPass
@@ -42,30 +42,27 @@
 ; CHECK-O-NEXT: Running pass: EarlyCSEPass
 ; CHECK-O-NEXT: Running analysis: TargetLibraryAnalysis
 ; CHECK-O3-NEXT: Running pass: CallSiteSplittingPass
-; CHECK-O-NEXT: Running pass: InstCombinePass on foo
-; CHECK-O-NEXT: Running analysis: OptimizationRemarkEmitterAnalysis on foo
-; CHECK-O-NEXT: Running analysis: AAManager on foo
-; CHECK-O-NEXT: Running analysis: BasicAA
-; CHECK-O-NEXT: Running analysis: ScopedNoAliasAA
-; CHECK-O-NEXT: Running analysis: TypeBasedAA
-; CHECK-O-NEXT: Running analysis: OuterAnalysisManagerProxy
 ; CHECK-O-NEXT: Running pass: SampleProfileLoaderPass
 ; CHECK-O-NEXT: Running analysis: ProfileSummaryAnalysis
 ; CHECK-O-NEXT: Running analysis: CallGraphAnalysis
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}ProfileSummaryAnalysis
 ; CHECK-O-NEXT: Running pass: OpenMPOptPass
 ; CHECK-O-NEXT: Running pass: IPSCCPPass
-; CHECK-O123-NEXT: Running analysis: LoopAnalysis on foo
 ; CHECK-O-NEXT: Running pass: CalledValuePropagationPass
 ; CHECK-O-NEXT: Running pass: GlobalOptPass
 ; CHECK-O-NEXT: Running pass: PromotePass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
+; CHECK-O-NEXT: Running analysis: OptimizationRemarkEmitterAnalysis on foo
+; CHECK-O-NEXT: Running analysis: AAManager on foo
+; CHECK-O-NEXT: Running analysis: BasicAA
+; CHECK-O-NEXT: Running analysis: ScopedNoAliasAA
+; CHECK-O-NEXT: Running analysis: TypeBasedAA
+; CHECK-O-NEXT: Running analysis: OuterAnalysisManagerProxy
 ; CHECK-O-NEXT: Running analysis: BlockFrequencyAnalysis on foo
 ; These next two can appear in any order since they are accessed as parameters
 ; on the same call to BlockFrequencyInfo::calculate.
-; CHECK-OSZ-DAG: Running analysis: LoopAnalysis on foo
-; CHECK-OSZ-DAG: Running analysis: BranchProbabilityAnalysis on foo
-; CHECK-O123-NEXT: Running analysis: BranchProbabilityAnalysis on foo
+; CHECK-O-DAG: Running analysis: LoopAnalysis on foo
+; CHECK-O-DAG: Running analysis: BranchProbabilityAnalysis on foo
 ; CHECK-O-NEXT: Running analysis: PostDominatorTreeAnalysis on foo
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass on foo
 ; CHECK-O-NEXT: Running pass: ModuleInlinerWrapperPass
@@ -83,13 +80,13 @@
 ; CHECK-O-NEXT: Running pass: InlinerPass
 ; CHECK-O-NEXT: Running pass: InlinerPass
 ; CHECK-O-NEXT: Running pass: PostOrderFunctionAttrsPass
-; CHECK-O-NEXT: Running analysis: AAManager
 ; CHECK-O3-NEXT: Running pass: ArgumentPromotionPass
 ; CHECK-O2-NEXT: Running pass: OpenMPOptCGSCCPass
 ; CHECK-O3-NEXT: Running pass: OpenMPOptCGSCCPass
 ; CHECK-O-NEXT: Running pass: SROAPass
 ; CHECK-O-NEXT: Running pass: EarlyCSEPass
 ; CHECK-O-NEXT: Running analysis: MemorySSAAnalysis
+; CHECK-O-NEXT: Running analysis: AAManager
 ; CHECK-O23SZ-NEXT: Running pass: SpeculativeExecutionPass
 ; CHECK-O23SZ-NEXT: Running pass: JumpThreadingPass
 ; CHECK-O23SZ-NEXT: Running analysis: LazyValueAnalysis
@@ -148,6 +145,8 @@
 ; CHECK-O23SZ-NEXT: Running pass: CoroElidePass
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
+; CHECK-O-NEXT: Running pass: PostOrderFunctionAttrsPass
+; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}ShouldNotRunFunctionPassesAnalysis
 ; CHECK-O-NEXT: Running analysis: ShouldNotRunFunctionPassesAnalysis
 ; CHECK-O-NEXT: Running pass: CoroSplitPass
 ; CHECK-O-NEXT: Running pass: InvalidateAnalysisPass<{{.*}}ShouldNotRunFunctionPassesAnalysis
diff --git a/llvm/test/Other/opt-On.ll b/llvm/test/Other/opt-On.ll
index 49f2d9d10deef..522f35578f961 100644
--- a/llvm/test/Other/opt-On.ll
+++ b/llvm/test/Other/opt-On.ll
@@ -19,16 +19,3 @@ define void @f() {
   unreachable
 }
 
-
-; Legacy PM deprecation tests (tests should be removed in the future).
-;
-; RUN: not opt -enable-new-pm=0 -O0 < %s -S 2>&1 | FileCheck %s --check-prefix=LEGACYPM-ERROR
-; RUN: not opt -enable-new-pm=0 -O1 < %s -S 2>&1 | FileCheck %s --check-prefix=LEGACYPM-ERROR
-; RUN: not opt -enable-new-pm=0 -O2 < %s -S 2>&1 | FileCheck %s --check-prefix=LEGACYPM-ERROR
-; RUN: not opt -enable-new-pm=0 -O3 < %s -S 2>&1 | FileCheck %s --check-prefix=LEGACYPM-ERROR
-; RUN: not opt -enable-new-pm=0 -Os < %s -S 2>&1 | FileCheck %s --check-prefix=LEGACYPM-ERROR
-; RUN: not opt -enable-new-pm=0 -Oz < %s -S 2>&1 | FileCheck %s --check-prefix=LEGACYPM-ERROR
-; RUN: not opt -O1 -codegenprepare < %s -S 2>&1 | FileCheck %s --check-prefix=LEGACYPM-ERROR
-; RUN: not opt -codegenprepare -O2 < %s -S 2>&1 | FileCheck %s --check-prefix=LEGACYPM-ERROR
-
-; LEGACYPM-ERROR: Cannot use -O# with legacy PM
diff --git a/llvm/test/Other/opt-legacy-syntax-deprecation.ll b/llvm/test/Other/opt-legacy-syntax-deprecation.ll
index a5a33564fd7ea..50162b333524e 100644
--- a/llvm/test/Other/opt-legacy-syntax-deprecation.ll
+++ b/llvm/test/Other/opt-legacy-syntax-deprecation.ll
@@ -4,7 +4,6 @@
 ; RUN: opt /dev/null -disable-output -passes=instcombine 2>&1 | FileCheck %s --check-prefix=OK --allow-empty
 ; RUN: not opt /dev/null -disable-output -instcombine 2>&1 | FileCheck %s --check-prefix=WARN
 ; RUN: not opt /dev/null -disable-output -instcombine -always-inline 2>&1 | FileCheck %s --check-prefix=WARN
-; RUN: opt /dev/null -disable-output -instcombine -enable-new-pm=0 2>&1 | FileCheck %s --check-prefix=OK --allow-empty
 ; RUN: opt /dev/null -disable-output -codegenprepare -mtriple=x86_64-unknown-linux-gnu 2>&1 | FileCheck %s --check-prefix=OK --allow-empty
 
 ; OK-NOT: deprecated
diff --git a/llvm/test/Other/pass-pipeline-parsing.ll b/llvm/test/Other/pass-pipeline-parsing.ll
index e84afe9ee7975..fd65224b1ef0d 100644
--- a/llvm/test/Other/pass-pipeline-parsing.ll
+++ b/llvm/test/Other/pass-pipeline-parsing.ll
@@ -57,6 +57,10 @@
 ; CHECK-DEFAULT-AA-DAG: Running analysis: BasicAA
 ; CHECK-DEFAULT-AA-DAG: Running analysis: TypeBasedAA
 
+; RUN: not opt -passes='function<no-rerun>(no-op-function)' %s 2>&1 \
+; RUN:     | FileCheck %s --check-prefix=CHECK-RERUN-BAD
+; CHECK-RERUN-BAD: cannot have a no-rerun module to function adaptor
+
 ; RUN: not opt -disable-output -debug-pass-manager \
 ; RUN:     -passes='no-op-module)' %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-UNBALANCED1
diff --git a/llvm/test/Other/scc-deleted-printer.ll b/llvm/test/Other/scc-deleted-printer.ll
index 86171d914c19e..c7455ff8ebd43 100644
--- a/llvm/test/Other/scc-deleted-printer.ll
+++ b/llvm/test/Other/scc-deleted-printer.ll
@@ -4,7 +4,7 @@
 ; RUN: 	   -passes=inline -print-before-all -print-after-all -print-module-scope | FileCheck %s
 
 ; CHECK: IR Dump Before InlinerPass on (tester, foo)
-; CHECK: IR Dump After InlinerPass on (tester, foo) (invalidated)
+; CHECK: IR Dump After InlinerPass on (tester, foo)
 ; CHECK: IR Dump Before InlinerPass on (tester)
 ; CHECK: IR Dump After InlinerPass on (tester)
 
diff --git a/llvm/test/TableGen/HwModeEncodeDecode.td b/llvm/test/TableGen/HwModeEncodeDecode.td
index bac432271888b..f8fcfafdd968f 100644
--- a/llvm/test/TableGen/HwModeEncodeDecode.td
+++ b/llvm/test/TableGen/HwModeEncodeDecode.td
@@ -13,8 +13,11 @@ def  Myi32  : Operand<i32> {
   let DecoderMethod = "DecodeMyi32";
 }
 
-def ModeA : HwMode<"+a">;
-def ModeB : HwMode<"+b">;
+def HasA : Predicate<"Subtarget->hasA()">;
+def HasB : Predicate<"Subtarget->hasB()">;
+
+def ModeA : HwMode<"+a", [HasA]>;
+def ModeB : HwMode<"+b", [HasB]>;
 
 
 def fooTypeEncA : InstructionEncoding {
diff --git a/llvm/test/TableGen/HwModeSelect.td b/llvm/test/TableGen/HwModeSelect.td
index 0c63e140f3902..e849febe0c4cb 100644
--- a/llvm/test/TableGen/HwModeSelect.td
+++ b/llvm/test/TableGen/HwModeSelect.td
@@ -18,8 +18,11 @@ def TestTarget : Target {
 def TestReg : Register<"testreg">;
 def TestClass : RegisterClass<"TestTarget", [i32], 32, (add TestReg)>;
 
-def TestMode1 : HwMode<"+feat1">;
-def TestMode2 : HwMode<"+feat2">;
+def HasFeat1 : Predicate<"Subtarget->hasFeat1()">;
+def HasFeat2 : Predicate<"Subtarget->hasFeat2()">;
+
+def TestMode1 : HwMode<"+feat1", [HasFeat1]>;
+def TestMode2 : HwMode<"+feat2", [HasFeat2]>;
 
 def BadDef : ValueTypeByHwMode<[TestMode1, TestMode2, DefaultMode],
                                [i8, i16, i32, i64]>;
diff --git a/llvm/test/TableGen/InsufficientPositionalOperands.td b/llvm/test/TableGen/InsufficientPositionalOperands.td
deleted file mode 100644
index 131c0861ad3c5..0000000000000
--- a/llvm/test/TableGen/InsufficientPositionalOperands.td
+++ /dev/null
@@ -1,35 +0,0 @@
-// RUN: not llvm-tblgen -gen-emitter -I %p/../../include %s 2>&1 | FileCheck %s --implicit-check-not=error:
-
-// Check that TableGen doesn't crash on insufficient positional
-// instruction operands.
-
-// TODO: remove this test when removing useDeprecatedPositionallyEncodedOperands
-
-include "llvm/Target/Target.td"
-
-def ArchInstrInfo : InstrInfo {
-  let useDeprecatedPositionallyEncodedOperands = 1;
-}
-
-def Arch : Target {
-  let InstructionSet = ArchInstrInfo;
-}
-
-def Reg : Register<"reg">;
-
-def Regs : RegisterClass<"foo", [i32], 0, (add Reg)>;
-
-// CHECK: error: Too few operands in record foo (no match for variable rs)
-// CHECK: note: Dumping record for previous error:
-def foo : Instruction {
-  bits<3> rd;
-  bits<3> rs;
-
-  bits<8> Inst;
-  let Inst{1-0} = 0;
-  let Inst{4-2} = rd;
-  let Inst{7-5} = rs;
-
-  let OutOperandList = (outs Regs:$xd);
-  let InOperandList = (ins);
-}
diff --git a/llvm/test/TableGen/MissingOperandField.td b/llvm/test/TableGen/MissingOperandField.td
index 3b550813540a6..37d6695ad4d59 100644
--- a/llvm/test/TableGen/MissingOperandField.td
+++ b/llvm/test/TableGen/MissingOperandField.td
@@ -15,8 +15,8 @@ def Reg : Register<"reg">;
 
 def Regs : RegisterClass<"foo", [i32], 0, (add Reg)>;
 
-// CHECK: error: No operand named rd in record foo (would've used positional operand #0 ('xd') sub-op #0 with useDeprecatedPositionallyEncodedOperands=true)
-// CHECK: error: No operand named rs in record foo (would've given 'too few operands' error with useDeprecatedPositionallyEncodedOperands=true)
+// CHECK: error: No operand named rd in record foo
+// CHECK: error: No operand named rs in record foo
 // CHECK: note: Dumping record for previous error:
 def foo : Instruction {
   bits<3> rd;
diff --git a/llvm/test/TableGen/if-unresolved-cast.td b/llvm/test/TableGen/if-unresolved-cast.td
new file mode 100644
index 0000000000000..2174d7e54c72b
--- /dev/null
+++ b/llvm/test/TableGen/if-unresolved-cast.td
@@ -0,0 +1,8 @@
+// RUN: not llvm-tblgen %s 2>&1 | FileCheck %s
+// XFAIL: vg_leak
+
+class A { int x; }
+
+// CHECK: Undefined reference to record: ''
+if !cast<A>("").x then
+  def x;
diff --git a/llvm/test/TableGen/intrinsic-pointer-to-any.td b/llvm/test/TableGen/intrinsic-pointer-to-any.td
index ca977d7b5078c..1b090b98cef45 100644
--- a/llvm/test/TableGen/intrinsic-pointer-to-any.td
+++ b/llvm/test/TableGen/intrinsic-pointer-to-any.td
@@ -1,4 +1,4 @@
-// RUN: llvm-tblgen -gen-intrinsic-impl %s | FileCheck %s
+// RUN: llvm-tblgen -gen-intrinsic-impl -I %p/../../include %s -DTEST_INTRINSICS_SUPPRESS_DEFS | FileCheck %s
 
 // This test is validating that it an Intrinsic with an LLVMPointerType to
 // llvm_any_ty still properly work after r363233. That patch rewrote the
@@ -6,52 +6,7 @@
 // case, so TableGen would hit an assertion in EncodeFixedType that was checking
 // to ensure that the substitution being processed was correctly replaced.
 
-class IntrinsicProperty<bit is_default = 0> {
-  bit IsDefault = is_default;
-}
-
-class SDNodeProperty;
-
-class ValueType<int size, int value> {
-  string Namespace = "MVT";
-  int Size = size;
-  int Value = value;
-}
-
-def iPTR   : ValueType<0  , 254>;
-def Any    : ValueType<0  , 255>;
-
-class LLVMType<ValueType vt> {
-  ValueType VT = vt;
-  int isAny = 0;
-}
-
-
-class Intrinsic<list<LLVMType> ret_types> {
-  string LLVMName = "";
-  string TargetPrefix = "";   // Set to a prefix for target-specific intrinsics.
-  list<LLVMType> RetTypes = ret_types;
-  list<LLVMType> ParamTypes = [];
-  list<IntrinsicProperty> IntrProperties = [];
-  list<SDNodeProperty> Properties = [];
-  bit isTarget = 0;
-  bit DisableDefaultAttributes = 1;
-}
-
-class LLVMQualPointerType<LLVMType elty>
-  : LLVMType<iPTR>{
-  LLVMType ElTy = elty;
-  int AddrSpace = 0;
-}
-
-class LLVMPointerType<LLVMType elty>
-  : LLVMQualPointerType<elty>;
-
-let isAny = 1 in {
-  def llvm_any_ty        : LLVMType<Any>;
-}
-def i8 : ValueType<8, 3>;
-def llvm_i8_ty : LLVMType<i8>;
+include "llvm/IR/Intrinsics.td"
 
 def int_has_ptr_to_any : Intrinsic<[LLVMPointerType<llvm_any_ty>, llvm_i8_ty]>;
-// CHECK: /* 0 */ 21, 14, 15, 0, 57, 0
+// CHECK: /* 0 */ 21, 14, 15, 0, 2, 0
diff --git a/llvm/test/TableGen/intrinsic-varargs.td b/llvm/test/TableGen/intrinsic-varargs.td
index da860ed0129c8..3634e16e20565 100644
--- a/llvm/test/TableGen/intrinsic-varargs.td
+++ b/llvm/test/TableGen/intrinsic-varargs.td
@@ -1,29 +1,7 @@
-// RUN: llvm-tblgen -gen-intrinsic-impl -I %p/../../include %s | FileCheck %s
+// RUN: llvm-tblgen -gen-intrinsic-impl -I %p/../../include %s -DTEST_INTRINSICS_SUPPRESS_DEFS | FileCheck %s
 // XFAIL: vg_leak
 
-include "llvm/CodeGen/ValueTypes.td"
-
-class IntrinsicProperty<bit is_default = 0> {
-  bit IsDefault = is_default;
-}
-class SDNodeProperty;
-
-class LLVMType<ValueType vt> {
-  ValueType VT = vt;
-}
-
-class Intrinsic<string name, list<LLVMType> param_types = []> {
-  string LLVMName = name;
-  bit isTarget = 0;
-  string TargetPrefix = "";
-  list<LLVMType> RetTypes = [];
-  list<LLVMType> ParamTypes = param_types;
-  list<IntrinsicProperty> IntrProperties = [];
-  list<SDNodeProperty> Properties = [];
-  bit DisableDefaultAttributes = 1;
-}
-
-def llvm_vararg_ty : LLVMType<isVoid>;   // this means vararg here
+include "llvm/IR/Intrinsics.td"
 
 // CHECK: /* 0 */ 0, 29, 0,
-def int_foo : Intrinsic<"llvm.foo", [llvm_vararg_ty]>;
+def int_foo : Intrinsic<[], [llvm_vararg_ty]>;
diff --git a/llvm/test/TableGen/string_ops.td b/llvm/test/TableGen/string_ops.td
new file mode 100644
index 0000000000000..68581dd9cd655
--- /dev/null
+++ b/llvm/test/TableGen/string_ops.td
@@ -0,0 +1,39 @@
+// RUN: llvm-tblgen %s | FileCheck %s
+
+class ToLower<string str> {
+  string s = !tolower(str);
+}
+
+class ToUpper<string str> {
+  string s = !toupper(str);
+}
+
+// CHECK-LABEL: def LOWER1 {
+// CHECK: string s = "str";
+// CHECK: }
+def LOWER1: ToLower<"STR">;
+
+// CHECK-LABEL: def LOWER2 {
+// CHECK: string s = "str";
+// CHECK: }
+def LOWER2 : ToLower<"Str">;
+
+// CHECK-LABEL: def LOWER3 {
+// CHECK: string s = "str";
+// CHECK: }
+def LOWER3 : ToLower<"STr">;
+
+// CHECK-LABEL: def UPPER1 {
+// CHECK: string s = "STR";
+// CHECK: }
+def UPPER1 : ToUpper<"str">;
+
+// CHECK-LABEL: def UPPER2 {
+// CHECK: string s = "STR";
+// CHECK: }
+def UPPER2 : ToUpper<"sTr">;
+
+// CHECK-LABEL: def UPPER3 {
+// CHECK: string s = "STR";
+// CHECK: }
+def UPPER3 : ToUpper<"sTR">;
\ No newline at end of file
diff --git a/llvm/test/TableGen/x86-auto-memfold.td b/llvm/test/TableGen/x86-auto-memfold.td
new file mode 100644
index 0000000000000..55bb8875ef205
--- /dev/null
+++ b/llvm/test/TableGen/x86-auto-memfold.td
@@ -0,0 +1,2 @@
+// RUN: llvm-tblgen -gen-x86-fold-tables -asmwriternum=1 %p/../../lib/Target/X86/X86.td -I %p/../../include -I %p/../../lib/Target/X86/ -I %p/../../include/ -I %p/../../lib/Target/ --write-if-changed  -o %t1
+// RUN: cmp  --ignore-initial=0:568 %p/../../lib/Target/X86/X86MemFoldTables.inc %t1
diff --git a/llvm/test/ThinLTO/X86/funcattrs-prop-unknown.ll b/llvm/test/ThinLTO/X86/funcattrs-prop-unknown.ll
index f6f2dc8b1e82b..48fa8f6bbffed 100644
--- a/llvm/test/ThinLTO/X86/funcattrs-prop-unknown.ll
+++ b/llvm/test/ThinLTO/X86/funcattrs-prop-unknown.ll
@@ -26,7 +26,7 @@ entry:
 ; CHECK: define void @selectcallee() {
 define void @selectcallee() {
     ; Test calls that aren't handled either as direct or indirect.
-    call void select (i1 icmp eq (ptr @global, ptr null), ptr @f, ptr @g)()
+    call void getelementptr (i8, ptr @f, i64 ptrtoint (ptr @g to i64))()
     ret void
 }
 
diff --git a/llvm/test/Transforms/ADCE/preserve-memoryssa-if-only-remove-debug.ll b/llvm/test/Transforms/ADCE/preserve-memoryssa-if-only-remove-debug.ll
new file mode 100644
index 0000000000000..339bc758344f9
--- /dev/null
+++ b/llvm/test/Transforms/ADCE/preserve-memoryssa-if-only-remove-debug.ll
@@ -0,0 +1,70 @@
+; RUN: opt -passes='require<memoryssa>,adce' -o - -S -debug-pass-manager < %s 2>&1 | FileCheck %s
+
+; ADCE should remove the dbg.declare in test1, but it should preserve
+; MemorySSA since it only removed a debug instruction.
+
+; Invalidating MemorySSA when only removing debug instructions may lead
+; to different code with/without debug info present.
+; In
+;  https://github.com/llvm/llvm-project/issues/58285
+; we saw how ADCE removed a dbg.declare, invalidated all analyses, and later
+; DSE behaved in some way. Without the dbg.declare in the input ADCE kept
+; analysis info, and DSE behaved differently.
+
+; CHECK: Running analysis: MemorySSAAnalysis on test1
+; CHECK: Running pass: ADCEPass on test1 (1 instruction)
+; CHECK-NOT: Invalidating analysis: MemorySSAAnalysis on test1
+
+; In test2 ADCE also removes an instruction, but since we remove a non-debug
+; instruction as well we invalidate several analyses, including MemorySSA.
+
+; CHECK: Running analysis: MemorySSAAnalysis on test2
+; CHECK: Running pass: ADCEPass on test2 (2 instructions)
+; CHECK: Invalidating analysis: MemorySSAAnalysis on test2
+; CHECK: Running pass: PrintModulePass
+
+; CHECK-LABEL: @test1(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    ret i16 0
+
+; CHECK-LABEL: @test2(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    ret i16 0
+
+define i16 @test1() {
+entry:
+  call void @llvm.dbg.declare(metadata ptr undef, metadata !4, metadata !DIExpression()), !dbg !16
+  ret i16 0
+}
+
+define i16 @test2() {
+entry:
+  %dead = add i16 1, 2
+  ret i16 0
+}
+
+; Function Attrs: nocallback nofree nosync nounwind speculatable willreturn memory(none)
+declare void @llvm.dbg.declare(metadata, metadata, metadata) #0
+
+attributes #0 = { nocallback nofree nosync nounwind speculatable willreturn memory(none) }
+
+!llvm.dbg.cu = !{}
+!llvm.module.flags = !{!0, !1, !2, !3}
+
+!0 = !{i32 7, !"Dwarf Version", i32 4}
+!1 = !{i32 2, !"Debug Info Version", i32 3}
+!2 = !{i32 1, !"wchar_size", i32 1}
+!3 = !{i32 7, !"frame-pointer", i32 2}
+!4 = !DILocalVariable(name: "w", scope: !5, file: !6, line: 18, type: !11)
+!5 = distinct !DILexicalBlock(scope: !7, file: !6, line: 18, column: 8)
+!6 = !DIFile(filename: "foo2.c", directory: "/llvm")
+!7 = distinct !DISubprogram(name: "test1", scope: !6, file: !6, line: 14, type: !8, scopeLine: 14, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !10)
+!8 = !DISubroutineType(types: !9)
+!9 = !{}
+!10 = distinct !DICompileUnit(language: DW_LANG_C99, file: !6, producer: "clang version 16.0.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, globals: !9, splitDebugInlining: false, nameTableKind: None)
+!11 = !DIDerivedType(tag: DW_TAG_typedef, name: "uint64_t", file: !12, line: 60, baseType: !13)
+!12 = !DIFile(filename: "/include/sys/_stdint.h", directory: "")
+!13 = !DIDerivedType(tag: DW_TAG_typedef, name: "__uint64_t", file: !14, line: 108, baseType: !15)
+!14 = !DIFile(filename: "/include/machine/_default_types.h", directory: "")
+!15 = !DIBasicType(name: "unsigned long long", size: 64, encoding: DW_ATE_unsigned)
+!16 = !DILocation(line: 18, column: 8, scope: !5)
\ No newline at end of file
diff --git a/llvm/test/Transforms/AggressiveInstCombine/logic-combine.ll b/llvm/test/Transforms/AggressiveInstCombine/logic-combine.ll
index e08834ddb7af8..291963b5e2e2b 100644
--- a/llvm/test/Transforms/AggressiveInstCombine/logic-combine.ll
+++ b/llvm/test/Transforms/AggressiveInstCombine/logic-combine.ll
@@ -1,9 +1,10 @@
 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=aggressive-instcombine -logic-combine-max-depth=6 -S | FileCheck %s
+; RUN: opt < %s -passes=aggressive-instcombine -S | FileCheck %s
 
 define i8 @leaf1_and_aa(i8 %a)  {
 ; CHECK-LABEL: @leaf1_and_aa(
-; CHECK-NEXT:    ret i8 [[A:%.*]]
+; CHECK-NEXT:    [[AND_AA:%.*]] = and i8 [[A:%.*]], [[A]]
+; CHECK-NEXT:    ret i8 [[AND_AA]]
 ;
   %and.aa = and i8 %a, %a
   ret i8 %and.aa
@@ -11,7 +12,8 @@ define i8 @leaf1_and_aa(i8 %a)  {
 
 define i8 @leaf1_and_a_false(i8 %a)  {
 ; CHECK-LABEL: @leaf1_and_a_false(
-; CHECK-NEXT:    ret i8 0
+; CHECK-NEXT:    [[AND_AA:%.*]] = and i8 [[A:%.*]], 0
+; CHECK-NEXT:    ret i8 [[AND_AA]]
 ;
   %and.aa = and i8 %a, 0
   ret i8 %and.aa
@@ -19,7 +21,8 @@ define i8 @leaf1_and_a_false(i8 %a)  {
 
 define i8 @leaf1_xor_aa(i8 %a)  {
 ; CHECK-LABEL: @leaf1_xor_aa(
-; CHECK-NEXT:    ret i8 0
+; CHECK-NEXT:    [[XOR_AA:%.*]] = xor i8 [[A:%.*]], [[A]]
+; CHECK-NEXT:    ret i8 [[XOR_AA]]
 ;
   %xor.aa = xor i8 %a, %a
   ret i8 %xor.aa
@@ -27,7 +30,9 @@ define i8 @leaf1_xor_aa(i8 %a)  {
 
 define i8 @leaf1_and_not(i8 %a)  {
 ; CHECK-LABEL: @leaf1_and_not(
-; CHECK-NEXT:    ret i8 0
+; CHECK-NEXT:    [[NOT_A:%.*]] = xor i8 [[A:%.*]], -1
+; CHECK-NEXT:    [[AND:%.*]] = and i8 [[A]], [[NOT_A]]
+; CHECK-NEXT:    ret i8 [[AND]]
 ;
   %not.a = xor i8 %a, -1
   %and = and i8 %a, %not.a
@@ -36,7 +41,9 @@ define i8 @leaf1_and_not(i8 %a)  {
 
 define i8 @leaf1_or_not(i8 %a)  {
 ; CHECK-LABEL: @leaf1_or_not(
-; CHECK-NEXT:    ret i8 -1
+; CHECK-NEXT:    [[NOT_A:%.*]] = xor i8 [[A:%.*]], -1
+; CHECK-NEXT:    [[OR:%.*]] = or i8 [[A]], [[NOT_A]]
+; CHECK-NEXT:    ret i8 [[OR]]
 ;
   %not.a = xor i8 %a, -1
   %or = or i8 %a, %not.a
@@ -45,7 +52,9 @@ define i8 @leaf1_or_not(i8 %a)  {
 
 define i8 @leaf2_xor(i8 %a, i8 %b)  {
 ; CHECK-LABEL: @leaf2_xor(
-; CHECK-NEXT:    ret i8 [[B:%.*]]
+; CHECK-NEXT:    [[AB:%.*]] = xor i8 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[XOR_AB_A:%.*]] = xor i8 [[AB]], [[A]]
+; CHECK-NEXT:    ret i8 [[XOR_AB_A]]
 ;
   %ab = xor i8 %a, %b
   %xor.ab.a = xor i8 %ab, %a
@@ -54,7 +63,10 @@ define i8 @leaf2_xor(i8 %a, i8 %b)  {
 
 define i8 @leaf2_xor_ret_const_false(i8 %a, i8 %b)  {
 ; CHECK-LABEL: @leaf2_xor_ret_const_false(
-; CHECK-NEXT:    ret i8 0
+; CHECK-NEXT:    [[XOR_AB:%.*]] = xor i8 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[XOR_AB_A:%.*]] = xor i8 [[XOR_AB]], [[A]]
+; CHECK-NEXT:    [[XOR_AB_A_B:%.*]] = xor i8 [[XOR_AB_A]], [[B]]
+; CHECK-NEXT:    ret i8 [[XOR_AB_A_B]]
 ;
   %xor.ab = xor i8 %a, %b
   %xor.ab.a = xor i8 %xor.ab, %a
@@ -64,7 +76,11 @@ define i8 @leaf2_xor_ret_const_false(i8 %a, i8 %b)  {
 
 define i8 @leaf2_or_ret_leaf(i8 %a, i8 %b)  {
 ; CHECK-LABEL: @leaf2_or_ret_leaf(
-; CHECK-NEXT:    ret i8 [[B:%.*]]
+; CHECK-NEXT:    [[OR_AB:%.*]] = or i8 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[AND_AB:%.*]] = and i8 [[A]], [[B]]
+; CHECK-NEXT:    [[XOR1:%.*]] = xor i8 [[OR_AB]], [[AND_AB]]
+; CHECK-NEXT:    [[XOR2:%.*]] = xor i8 [[XOR1]], [[A]]
+; CHECK-NEXT:    ret i8 [[XOR2]]
 ;
   %or.ab = or i8 %a, %b
   %and.ab = and i8 %a, %b
@@ -75,19 +91,28 @@ define i8 @leaf2_or_ret_leaf(i8 %a, i8 %b)  {
 
 define i8 @leaf2_or_ret_const_false(i8 %a, i8 %b)  {
 ; CHECK-LABEL: @leaf2_or_ret_const_false(
-; CHECK-NEXT:    ret i8 0
+; CHECK-NEXT:    [[OR_AB:%.*]] = or i8 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[AND_AB:%.*]] = and i8 [[A]], [[B]]
+; CHECK-NEXT:    [[XOR1:%.*]] = xor i8 [[OR_AB]], [[AND_AB]]
+; CHECK-NEXT:    [[XOR2:%.*]] = xor i8 [[XOR1]], [[A]]
+; CHECK-NEXT:    [[XOR3:%.*]] = xor i8 [[XOR1]], [[B]]
+; CHECK-NEXT:    ret i8 [[XOR3]]
 ;
   %or.ab = or i8 %a, %b
   %and.ab = and i8 %a, %b
   %xor1 = xor i8 %or.ab, %and.ab
   %xor2 = xor i8 %xor1, %a
-  %xor3 = xor i8 %xor2, %b
+  %xor3 = xor i8 %xor1, %b
   ret i8 %xor3
 }
 
 define i1 @leaf2_type_is_i1(i1 %a, i1 %b) {
 ; CHECK-LABEL: @leaf2_type_is_i1(
-; CHECK-NEXT:    ret i1 true
+; CHECK-NEXT:    [[XOR_AB:%.*]] = xor i1 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[NOT_A:%.*]] = xor i1 [[A]], true
+; CHECK-NEXT:    [[XOR2:%.*]] = xor i1 [[NOT_A]], [[B]]
+; CHECK-NEXT:    [[OR:%.*]] = or i1 [[XOR2]], [[XOR_AB]]
+; CHECK-NEXT:    ret i1 [[OR]]
 ;
   %xor.ab = xor i1 %a, %b
   %not.a = xor i1 %a, true
@@ -98,7 +123,11 @@ define i1 @leaf2_type_is_i1(i1 %a, i1 %b) {
 
 define i8 @leaf3_complex_ret_const_false(i8 %a, i8 %b, i8 %c)  {
 ; CHECK-LABEL: @leaf3_complex_ret_const_false(
-; CHECK-NEXT:    ret i8 0
+; CHECK-NEXT:    [[AB:%.*]] = or i8 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[ABC:%.*]] = or i8 [[AB]], [[C:%.*]]
+; CHECK-NEXT:    [[NOT_ABC:%.*]] = xor i8 [[ABC]], -1
+; CHECK-NEXT:    [[R:%.*]] = and i8 [[NOT_ABC]], [[A]]
+; CHECK-NEXT:    ret i8 [[R]]
 ;
   %ab = or i8 %a, %b
   %abc = or i8 %ab, %c
@@ -109,7 +138,14 @@ define i8 @leaf3_complex_ret_const_false(i8 %a, i8 %b, i8 %c)  {
 
 define i8 @leaf3_complex_ret_leaf(i8 %a, i8 %b, i8 %c) {
 ; CHECK-LABEL: @leaf3_complex_ret_leaf(
-; CHECK-NEXT:    ret i8 [[C:%.*]]
+; CHECK-NEXT:    [[AB:%.*]] = and i8 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[BC:%.*]] = and i8 [[B]], [[C:%.*]]
+; CHECK-NEXT:    [[XOR_AC:%.*]] = xor i8 [[A]], [[C]]
+; CHECK-NEXT:    [[OR:%.*]] = or i8 [[AB]], [[XOR_AC]]
+; CHECK-NEXT:    [[NOT_BC:%.*]] = xor i8 [[BC]], -1
+; CHECK-NEXT:    [[AND:%.*]] = and i8 [[NOT_BC]], [[A]]
+; CHECK-NEXT:    [[COND:%.*]] = xor i8 [[AND]], [[OR]]
+; CHECK-NEXT:    ret i8 [[COND]]
 ;
   %ab = and i8 %a, %b
   %bc = and i8 %b, %c
@@ -123,7 +159,13 @@ define i8 @leaf3_complex_ret_leaf(i8 %a, i8 %b, i8 %c) {
 
 define i8 @leaf4_ret_const_true(i8 %a, i8 %b, i8 %c, i8 %d)  {
 ; CHECK-LABEL: @leaf4_ret_const_true(
-; CHECK-NEXT:    ret i8 -1
+; CHECK-NEXT:    [[BD:%.*]] = and i8 [[B:%.*]], [[D:%.*]]
+; CHECK-NEXT:    [[NOT_BD:%.*]] = xor i8 [[BD]], -1
+; CHECK-NEXT:    [[XOR_AB:%.*]] = xor i8 [[A:%.*]], [[B]]
+; CHECK-NEXT:    [[OR1:%.*]] = or i8 [[XOR_AB]], [[C:%.*]]
+; CHECK-NEXT:    [[OR2:%.*]] = or i8 [[OR1]], [[NOT_BD]]
+; CHECK-NEXT:    [[OR3:%.*]] = or i8 [[OR2]], [[A]]
+; CHECK-NEXT:    ret i8 [[OR3]]
 ;
   %bd = and i8 %b, %d
   %not.bd = xor i8 %bd, -1
@@ -136,7 +178,15 @@ define i8 @leaf4_ret_const_true(i8 %a, i8 %b, i8 %c, i8 %d)  {
 
 define i8 @leaf4_ret_leaf(i8 %a, i8 %b, i8 %c, i8 %d)  {
 ; CHECK-LABEL: @leaf4_ret_leaf(
-; CHECK-NEXT:    ret i8 [[B:%.*]]
+; CHECK-NEXT:    [[BD:%.*]] = and i8 [[B:%.*]], [[D:%.*]]
+; CHECK-NEXT:    [[XOR:%.*]] = xor i8 [[BD]], [[C:%.*]]
+; CHECK-NEXT:    [[NOT_BD:%.*]] = xor i8 [[XOR]], -1
+; CHECK-NEXT:    [[XOR_AB:%.*]] = xor i8 [[A:%.*]], [[B]]
+; CHECK-NEXT:    [[OR1:%.*]] = or i8 [[XOR_AB]], [[C]]
+; CHECK-NEXT:    [[OR2:%.*]] = or i8 [[OR1]], [[NOT_BD]]
+; CHECK-NEXT:    [[OR3:%.*]] = or i8 [[OR2]], [[A]]
+; CHECK-NEXT:    [[AND:%.*]] = and i8 [[OR3]], [[B]]
+; CHECK-NEXT:    ret i8 [[AND]]
 ;
   %bd = and i8 %b, %d
   %xor = xor i8 %bd, %c
@@ -151,7 +201,15 @@ define i8 @leaf4_ret_leaf(i8 %a, i8 %b, i8 %c, i8 %d)  {
 
 define i8 @leaf4_ret_leaf2(i8 %a, i8 %b, i8 %c, i8 %d)  {
 ; CHECK-LABEL: @leaf4_ret_leaf2(
-; CHECK-NEXT:    ret i8 [[B:%.*]]
+; CHECK-NEXT:    [[BD:%.*]] = and i8 [[B:%.*]], [[D:%.*]]
+; CHECK-NEXT:    [[XOR:%.*]] = xor i8 [[BD]], [[C:%.*]]
+; CHECK-NEXT:    [[NOT_BD:%.*]] = xor i8 [[XOR]], -1
+; CHECK-NEXT:    [[XOR_AB:%.*]] = xor i8 [[A:%.*]], [[B]]
+; CHECK-NEXT:    [[OR1:%.*]] = or i8 [[XOR_AB]], [[C]]
+; CHECK-NEXT:    [[OR2:%.*]] = or i8 [[OR1]], [[NOT_BD]]
+; CHECK-NEXT:    [[OR3:%.*]] = or i8 [[OR2]], [[A]]
+; CHECK-NEXT:    [[AND:%.*]] = and i8 [[OR3]], [[B]]
+; CHECK-NEXT:    ret i8 [[AND]]
 ;
   %bd = and i8 %b, %d
   %xor = xor i8 %bd, %c
@@ -163,88 +221,3 @@ define i8 @leaf4_ret_leaf2(i8 %a, i8 %b, i8 %c, i8 %d)  {
   %and = and i8 %or3, %b
   ret i8 %and
 }
-
-; Negative test case 1 for max leaf number:
-; This case's max leaf number is 9, if we adjust max depth limitation
-; to larger than 8, it will return %a1
-
-define i8 @leaf8_negative_leafnum(i8 %a1, i8 %a2, i8 %a3, i8 %a4, i8 %a5, i8 %a6, i8 %a7, i8 %a8, i8 %a9)  {
-; CHECK-LABEL: @leaf8_negative_leafnum(
-; CHECK-NEXT:    [[A12:%.*]] = xor i8 [[A1:%.*]], [[A2:%.*]]
-; CHECK-NEXT:    [[A34:%.*]] = xor i8 [[A3:%.*]], [[A4:%.*]]
-; CHECK-NEXT:    [[A56:%.*]] = xor i8 [[A5:%.*]], [[A6:%.*]]
-; CHECK-NEXT:    [[A78:%.*]] = xor i8 [[A7:%.*]], [[A8:%.*]]
-; CHECK-NEXT:    [[A14:%.*]] = xor i8 [[A12]], [[A34]]
-; CHECK-NEXT:    [[A58:%.*]] = xor i8 [[A56]], [[A78]]
-; CHECK-NEXT:    [[A18:%.*]] = xor i8 [[A14]], [[A58]]
-; CHECK-NEXT:    [[A19:%.*]] = xor i8 [[A18]], [[A9:%.*]]
-; CHECK-NEXT:    [[A23:%.*]] = xor i8 [[A2]], [[A3]]
-; CHECK-NEXT:    [[A45:%.*]] = xor i8 [[A4]], [[A5]]
-; CHECK-NEXT:    [[A67:%.*]] = xor i8 [[A6]], [[A7]]
-; CHECK-NEXT:    [[A89:%.*]] = xor i8 [[A8]], [[A9]]
-; CHECK-NEXT:    [[A25:%.*]] = xor i8 [[A23]], [[A45]]
-; CHECK-NEXT:    [[A69:%.*]] = xor i8 [[A67]], [[A89]]
-; CHECK-NEXT:    [[A29:%.*]] = xor i8 [[A25]], [[A69]]
-; CHECK-NEXT:    [[R:%.*]] = xor i8 [[A19]], [[A29]]
-; CHECK-NEXT:    ret i8 [[R]]
-;
-  %a12 = xor i8 %a1, %a2
-  %a34 = xor i8 %a3, %a4
-  %a56 = xor i8 %a5, %a6
-  %a78 = xor i8 %a7, %a8
-  %a14 = xor i8 %a12, %a34
-  %a58 = xor i8 %a56, %a78
-  %a18 = xor i8 %a14, %a58
-  %a19 = xor i8 %a18, %a9
-  %a23 = xor i8 %a2, %a3
-  %a45 = xor i8 %a4, %a5
-  %a67 = xor i8 %a6, %a7
-  %a89 = xor i8 %a8, %a9
-  %a25 = xor i8 %a23, %a45
-  %a69 = xor i8 %a67, %a89
-  %a29 = xor i8 %a25, %a69
-  %r = xor i8 %a19, %a29
-  ret i8 %r
-}
-
-; Negative test case 2 for max leaf number:
-; Constant value is also a leaf node.
-
-define i8 @leaf8_negative_leafnum_const(i8 %a1, i8 %a2)  {
-; CHECK-LABEL: @leaf8_negative_leafnum_const(
-; CHECK-NEXT:    [[AND1:%.*]] = and i8 [[A1:%.*]], 1
-; CHECK-NEXT:    call void @use8(i8 [[AND1]])
-; CHECK-NEXT:    [[AND2:%.*]] = and i8 [[A1]], 2
-; CHECK-NEXT:    call void @use8(i8 [[AND2]])
-; CHECK-NEXT:    [[AND3:%.*]] = and i8 [[A1]], 3
-; CHECK-NEXT:    call void @use8(i8 [[AND3]])
-; CHECK-NEXT:    [[AND4:%.*]] = and i8 [[A1]], 4
-; CHECK-NEXT:    call void @use8(i8 [[AND4]])
-; CHECK-NEXT:    [[AND5:%.*]] = and i8 [[A1]], 5
-; CHECK-NEXT:    call void @use8(i8 [[AND5]])
-; CHECK-NEXT:    [[AND6:%.*]] = and i8 [[A1]], 6
-; CHECK-NEXT:    call void @use8(i8 [[AND6]])
-; CHECK-NEXT:    [[AND7:%.*]] = and i8 [[A1]], 7
-; CHECK-NEXT:    call void @use8(i8 [[AND7]])
-; CHECK-NEXT:    [[R:%.*]] = xor i8 [[A2:%.*]], [[A2]]
-; CHECK-NEXT:    ret i8 [[R]]
-;
-  %and1 = and i8 %a1, 1
-  call void @use8(i8 %and1)
-  %and2 = and i8 %a1, 2
-  call void @use8(i8 %and2)
-  %and3 = and i8 %a1, 3
-  call void @use8(i8 %and3)
-  %and4 = and i8 %a1, 4
-  call void @use8(i8 %and4)
-  %and5 = and i8 %a1, 5
-  call void @use8(i8 %and5)
-  %and6 = and i8 %a1, 6
-  call void @use8(i8 %and6)
-  %and7 = and i8 %a1, 7
-  call void @use8(i8 %and7)
-  %r = xor i8 %a2, %a2
-  ret i8 %r
-}
-
-declare void @use8(i8)
diff --git a/llvm/test/Transforms/Attributor/IPConstantProp/openmp_parallel_for.ll b/llvm/test/Transforms/Attributor/IPConstantProp/openmp_parallel_for.ll
index 5c2671d2fab3e..8a437422ae20e 100644
--- a/llvm/test/Transforms/Attributor/IPConstantProp/openmp_parallel_for.ll
+++ b/llvm/test/Transforms/Attributor/IPConstantProp/openmp_parallel_for.ll
@@ -101,7 +101,7 @@ define internal void @.omp_outlined.(ptr noalias %.global_tid., ptr noalias %.bo
 ; TUNIT:       omp.inner.for.body:
 ; TUNIT-NEXT:    [[ADD10:%.*]] = add nsw i32 [[DOTOMP_IV_0]], 2
 ; TUNIT-NEXT:    [[TMP11:%.*]] = load double, ptr [[Q_ADDR]], align 8
-; TUNIT-NEXT:    call void @bar(i32 [[ADD10]], float 3.000000e+00, double [[TMP11]])
+; TUNIT-NEXT:    call void @bar(i32 [[ADD10]], float nofpclass(nan inf zero sub nnorm) 3.000000e+00, double [[TMP11]])
 ; TUNIT-NEXT:    br label [[OMP_BODY_CONTINUE:%.*]]
 ; TUNIT:       omp.body.continue:
 ; TUNIT-NEXT:    br label [[OMP_INNER_FOR_INC]]
diff --git a/llvm/test/Transforms/Attributor/nofpclass.ll b/llvm/test/Transforms/Attributor/nofpclass.ll
new file mode 100644
index 0000000000000..78ba8d678de89
--- /dev/null
+++ b/llvm/test/Transforms/Attributor/nofpclass.ll
@@ -0,0 +1,662 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --check-attributes --version 2
+; RUN: opt -aa-pipeline=basic-aa -passes=attributor -attributor-manifest-internal  -attributor-max-iterations-verify -attributor-annotate-decl-cs -attributor-max-iterations=3 -S < %s | FileCheck %s --check-prefixes=CHECK,TUNIT
+; RUN: opt -aa-pipeline=basic-aa -passes=attributor-cgscc -attributor-manifest-internal  -attributor-annotate-decl-cs -S < %s | FileCheck %s --check-prefixes=CHECK,CGSCC
+
+declare nofpclass(nan) float @ret_nofpclass_nan()
+declare [2 x [3 x float]] @ret_array()
+declare float @extern()
+declare void @extern.use(float)
+declare void @extern.use.array([2 x [3 x float]])
+declare void @llvm.assume(i1 noundef)
+declare void @unknown()
+declare half @llvm.fabs.f16(half)
+declare void @extern.use.f16(half)
+declare i1 @llvm.is.fpclass.f32(float, i32 immarg)
+
+define float @returned_0() {
+; CHECK-LABEL: define noundef nofpclass(nan inf nzero sub norm) float @returned_0() {
+; CHECK-NEXT:    call void @unknown()
+; CHECK-NEXT:    ret float 0.000000e+00
+;
+  call void @unknown()
+  ret float 0.0
+}
+
+define float @returned_neg0() {
+; CHECK-LABEL: define noundef nofpclass(nan inf pzero sub norm) float @returned_neg0() {
+; CHECK-NEXT:    call void @unknown()
+; CHECK-NEXT:    ret float -0.000000e+00
+;
+  call void @unknown()
+  ret float -0.0
+}
+
+define float @returned_undef() {
+; CHECK-LABEL: define nofpclass(all) float @returned_undef() {
+; CHECK-NEXT:    call void @unknown()
+; CHECK-NEXT:    ret float undef
+;
+  call void @unknown()
+  ret float undef
+}
+
+define float @returned_poison() {
+; CHECK-LABEL: define nofpclass(all) float @returned_poison() {
+; CHECK-NEXT:    call void @unknown()
+; CHECK-NEXT:    ret float poison
+;
+  call void @unknown()
+  ret float poison
+}
+
+define double @returned_snan() {
+; CHECK-LABEL: define noundef nofpclass(qnan inf zero sub norm) double @returned_snan() {
+; CHECK-NEXT:    call void @unknown()
+; CHECK-NEXT:    ret double 0x7FF0000000000001
+;
+  call void @unknown()
+  ret double 0x7FF0000000000001
+}
+
+define double @returned_qnan() {
+; CHECK-LABEL: define noundef nofpclass(snan inf zero sub norm) double @returned_qnan() {
+; CHECK-NEXT:    call void @unknown()
+; CHECK-NEXT:    ret double 0x7FF8000000000000
+;
+  call void @unknown()
+  ret double 0x7FF8000000000000
+}
+
+define <2 x double> @returned_zero_vector() {
+; CHECK-LABEL: define noundef nofpclass(nan inf nzero sub norm) <2 x double> @returned_zero_vector() {
+; CHECK-NEXT:    call void @unknown()
+; CHECK-NEXT:    ret <2 x double> zeroinitializer
+;
+  call void @unknown()
+  ret <2 x double> zeroinitializer
+}
+
+define <2 x double> @returned_negzero_vector() {
+; CHECK-LABEL: define noundef nofpclass(nan inf pzero sub norm) <2 x double> @returned_negzero_vector() {
+; CHECK-NEXT:    call void @unknown()
+; CHECK-NEXT:    ret <2 x double> <double -0.000000e+00, double -0.000000e+00>
+;
+  call void @unknown()
+  ret <2 x double> <double -0.0, double -0.0>
+}
+
+define <2 x double> @returned_qnan_zero_vector() {
+; CHECK-LABEL: define noundef <2 x double> @returned_qnan_zero_vector() {
+; CHECK-NEXT:    call void @unknown()
+; CHECK-NEXT:    ret <2 x double> <double 0x7FF8000000000000, double 0.000000e+00>
+;
+  call void @unknown()
+  ret <2 x double> <double 0x7FF8000000000000, double 0.0>
+}
+
+; Return a float trivially nofpclass(nan) (call return attribute)
+define float @return_nofpclass_nan_decl_return() {
+; CHECK-LABEL: define nofpclass(nan) float @return_nofpclass_nan_decl_return() {
+; CHECK-NEXT:    [[RET:%.*]] = call nofpclass(nan) float @ret_nofpclass_nan()
+; CHECK-NEXT:    ret float [[RET]]
+;
+  %ret = call float @ret_nofpclass_nan()
+  ret float %ret
+}
+
+; Return a float trivially nofpclass(nan) (argument attribute)
+define float @return_nofpclass_nan_arg(float returned nofpclass(nan) %p) {
+; CHECK: Function Attrs: nofree norecurse nosync nounwind willreturn memory(none)
+; CHECK-LABEL: define nofpclass(nan) float @return_nofpclass_nan_arg
+; CHECK-SAME: (float returned nofpclass(nan) [[P:%.*]]) #[[ATTR2:[0-9]+]] {
+; CHECK-NEXT:    ret float [[P]]
+;
+  ret float %p
+}
+
+define [2 x [3 x float]] @return_nofpclass_inf_ret_array() {
+; CHECK-LABEL: define nofpclass(inf) [2 x [3 x float]] @return_nofpclass_inf_ret_array() {
+; CHECK-NEXT:    [[RET:%.*]] = call nofpclass(inf) [2 x [3 x float]] @ret_array()
+; CHECK-NEXT:    ret [2 x [3 x float]] [[RET]]
+;
+  %ret = call nofpclass(inf) [2 x [3 x float]]  @ret_array()
+  ret [2 x [3 x float]] %ret
+}
+
+define float @returned_nnan_fadd(float %arg0, float %arg1) {
+; CHECK: Function Attrs: nofree norecurse nosync nounwind willreturn memory(none)
+; CHECK-LABEL: define nofpclass(nan) float @returned_nnan_fadd
+; CHECK-SAME: (float [[ARG0:%.*]], float [[ARG1:%.*]]) #[[ATTR2]] {
+; CHECK-NEXT:    [[FADD:%.*]] = fadd nnan float [[ARG0]], [[ARG1]]
+; CHECK-NEXT:    ret float [[FADD]]
+;
+  %fadd = fadd nnan float %arg0, %arg1
+  ret float %fadd
+}
+
+define float @return_nofpclass_nan_callsite() {
+; CHECK-LABEL: define nofpclass(nan) float @return_nofpclass_nan_callsite() {
+; CHECK-NEXT:    [[CALL:%.*]] = call nofpclass(nan) float @extern()
+; CHECK-NEXT:    ret float [[CALL]]
+;
+  %call = call nofpclass(nan) float @extern()
+  ret float %call
+}
+
+; Can union the return classes
+define nofpclass(inf) float @return_ninf_nofpclass_nan_callsite() {
+; CHECK-LABEL: define nofpclass(nan inf) float @return_ninf_nofpclass_nan_callsite() {
+; CHECK-NEXT:    [[CALL:%.*]] = call nofpclass(nan) float @extern()
+; CHECK-NEXT:    ret float [[CALL]]
+;
+  %call = call nofpclass(nan) float @extern()
+  ret float %call
+}
+
+define void @arg_used_by_nofpclass_nan_callsite(float %arg) {
+; CHECK-LABEL: define void @arg_used_by_nofpclass_nan_callsite
+; CHECK-SAME: (float [[ARG:%.*]]) {
+; CHECK-NEXT:    call void @extern.use(float nofpclass(nan) [[ARG]])
+; CHECK-NEXT:    ret void
+;
+  call void @extern.use(float nofpclass(nan) %arg)
+  ret void
+}
+
+; Callsite can union the incoming and outgoing
+define void @ninf_arg_used_by_nofpclass_nan_callsite(float nofpclass(inf) %arg) {
+; CHECK-LABEL: define void @ninf_arg_used_by_nofpclass_nan_callsite
+; CHECK-SAME: (float nofpclass(inf) [[ARG:%.*]]) {
+; CHECK-NEXT:    call void @extern.use(float nofpclass(nan inf) [[ARG]])
+; CHECK-NEXT:    ret void
+;
+  call void @extern.use(float nofpclass(nan) %arg)
+  ret void
+}
+
+define void @ninf_arg_used_by_callsite_array([2 x [3 x float]] nofpclass(inf) %arg) {
+; CHECK-LABEL: define void @ninf_arg_used_by_callsite_array
+; CHECK-SAME: ([2 x [3 x float]] nofpclass(inf) [[ARG:%.*]]) {
+; CHECK-NEXT:    call void @extern.use.array([2 x [3 x float]] nofpclass(inf) [[ARG]])
+; CHECK-NEXT:    ret void
+;
+  call void @extern.use.array([2 x [3 x float]]  %arg)
+  ret void
+}
+
+define void @nofpclass_call_use_after_unannotated_use(float %arg) {
+; CHECK-LABEL: define void @nofpclass_call_use_after_unannotated_use
+; CHECK-SAME: (float [[ARG:%.*]]) {
+; CHECK-NEXT:    call void @extern(float [[ARG]]) #[[ATTR5:[0-9]+]]
+; CHECK-NEXT:    call void @extern(float nofpclass(nan inf) [[ARG]])
+; CHECK-NEXT:    ret void
+;
+  call void @extern(float %arg) willreturn nounwind ; < annotate this use
+  call void @extern(float nofpclass(nan inf) %arg)
+  ret void
+}
+
+define float @mutually_recursive0(float %arg) {
+; TUNIT: Function Attrs: nofree nosync nounwind willreturn memory(none)
+; TUNIT-LABEL: define nofpclass(all) float @mutually_recursive0
+; TUNIT-SAME: (float [[ARG:%.*]]) #[[ATTR3:[0-9]+]] {
+; TUNIT-NEXT:    ret float undef
+;
+; CGSCC: Function Attrs: nofree norecurse nosync nounwind willreturn memory(none)
+; CGSCC-LABEL: define nofpclass(all) float @mutually_recursive0
+; CGSCC-SAME: (float [[ARG:%.*]]) #[[ATTR2]] {
+; CGSCC-NEXT:    ret float undef
+;
+  %call = call float @mutually_recursive1(float %arg)
+  ret float %call
+}
+
+define float @mutually_recursive1(float %arg) {
+; TUNIT: Function Attrs: nofree nosync nounwind willreturn memory(none)
+; TUNIT-LABEL: define nofpclass(all) float @mutually_recursive1
+; TUNIT-SAME: (float [[ARG:%.*]]) #[[ATTR3]] {
+; TUNIT-NEXT:    ret float undef
+;
+; CGSCC: Function Attrs: nofree norecurse nosync nounwind willreturn memory(none)
+; CGSCC-LABEL: define nofpclass(all) float @mutually_recursive1
+; CGSCC-SAME: (float [[ARG:%.*]]) #[[ATTR2]] {
+; CGSCC-NEXT:    ret float undef
+;
+  %call = call float @mutually_recursive0(float %arg)
+  ret float %call
+}
+
+define float @recursive_phi(ptr %ptr) {
+; CHECK-LABEL: define nofpclass(nan) float @recursive_phi
+; CHECK-SAME: (ptr nofree [[PTR:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[RET:%.*]] = call nofpclass(nan) float @ret_nofpclass_nan()
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[PHI:%.*]] = phi float [ [[RET]], [[ENTRY:%.*]] ], [ [[RET]], [[LOOP]] ]
+; CHECK-NEXT:    [[COND:%.*]] = load volatile i1, ptr [[PTR]], align 1
+; CHECK-NEXT:    br i1 [[COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    ret float [[RET]]
+;
+entry:
+  %ret = call float @ret_nofpclass_nan()
+  br label %loop
+
+loop:
+  %phi = phi float [%ret, %entry], [%phi, %loop]
+  %cond = load volatile i1, ptr %ptr
+  br i1 %cond, label %loop, label %exit
+
+exit:
+  ret float %phi
+}
+
+; Should be able to infer nofpclass(nan) return
+define float @fcmp_uno_check(float %arg) local_unnamed_addr {
+; CHECK-LABEL: define float @fcmp_uno_check
+; CHECK-SAME: (float [[ARG:%.*]]) local_unnamed_addr {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[ISNAN:%.*]] = fcmp uno float [[ARG]], 0.000000e+00
+; CHECK-NEXT:    br i1 [[ISNAN]], label [[BB0:%.*]], label [[BB1:%.*]]
+; CHECK:       bb0:
+; CHECK-NEXT:    [[CALL:%.*]] = call nofpclass(nan) float @ret_nofpclass_nan()
+; CHECK-NEXT:    br label [[BB1]]
+; CHECK:       bb1:
+; CHECK-NEXT:    [[PHI:%.*]] = phi float [ [[CALL]], [[BB0]] ], [ [[ARG]], [[ENTRY:%.*]] ]
+; CHECK-NEXT:    ret float [[PHI]]
+;
+entry:
+  %isnan = fcmp uno float %arg, 0.0
+  br i1 %isnan, label %bb0, label %bb1
+
+bb0:
+  %call = call float @ret_nofpclass_nan()
+  br label %bb1
+
+bb1:
+  %phi = phi float [ %call, %bb0 ], [ %arg, %entry ]
+  ret float %phi
+}
+
+; Should be able to infer nofpclass(nan) on %arg use
+define void @fcmp_ord_guard_callsite_arg(float %arg) {
+; CHECK-LABEL: define void @fcmp_ord_guard_callsite_arg
+; CHECK-SAME: (float [[ARG:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[IS_NOT_NAN:%.*]] = fcmp ord float [[ARG]], 0.000000e+00
+; CHECK-NEXT:    br i1 [[IS_NOT_NAN]], label [[BB0:%.*]], label [[BB1:%.*]]
+; CHECK:       bb0:
+; CHECK-NEXT:    call void @extern.use(float [[ARG]])
+; CHECK-NEXT:    br label [[BB1]]
+; CHECK:       bb1:
+; CHECK-NEXT:    ret void
+;
+entry:
+  %is.not.nan = fcmp ord float %arg, 0.0
+  br i1 %is.not.nan, label %bb0, label %bb1
+
+bb0:
+  call void @extern.use(float %arg)
+  br label %bb1
+
+bb1:
+  ret void
+}
+
+; Should be able to infer nofpclass on both %arg uses
+define float @fcmp_ord_assume_callsite_arg_return(float %arg) {
+; CHECK-LABEL: define nofpclass(inf zero sub norm) float @fcmp_ord_assume_callsite_arg_return
+; CHECK-SAME: (float returned nofpclass(inf zero sub norm) [[ARG:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[IS_NOT_NAN:%.*]] = fcmp ord float [[ARG]], 0.000000e+00
+; CHECK-NEXT:    call void @llvm.assume(i1 noundef [[IS_NOT_NAN]]) #[[ATTR6:[0-9]+]]
+; CHECK-NEXT:    call void @extern.use(float nofpclass(inf zero sub norm) [[ARG]])
+; CHECK-NEXT:    ret float [[ARG]]
+;
+entry:
+  %is.not.nan = fcmp ord float %arg, 0.0
+  call void @llvm.assume(i1 %is.not.nan)
+  call void @extern.use(float %arg)
+  ret float %arg
+}
+
+define internal float @returned_dead() {
+; CHECK-LABEL: define internal nofpclass(nan inf nzero sub norm) float @returned_dead() {
+; CHECK-NEXT:    call void @unknown()
+; CHECK-NEXT:    ret float undef
+;
+  call void @unknown()
+  ret float 0.0
+}
+
+define void @returned_dead_caller() {
+; CHECK-LABEL: define void @returned_dead_caller() {
+; CHECK-NEXT:    [[TMP1:%.*]] = call float @returned_dead()
+; CHECK-NEXT:    ret void
+;
+  call float @returned_dead()
+  ret void
+}
+
+define internal float @only_nofpclass_inf_callers(float %arg) {
+; CHECK: Function Attrs: nofree norecurse nosync nounwind willreturn memory(none)
+; CHECK-LABEL: define internal float @only_nofpclass_inf_callers
+; CHECK-SAME: (float nofpclass(inf) [[ARG:%.*]]) #[[ATTR2]] {
+; CHECK-NEXT:    [[ADD:%.*]] = fadd float [[ARG]], [[ARG]]
+; CHECK-NEXT:    ret float [[ADD]]
+;
+  %add = fadd float %arg, %arg
+  ret float %add
+}
+
+define float @call_noinf_0(float nofpclass(inf) %arg) {
+; TUNIT: Function Attrs: nofree norecurse nosync nounwind willreturn memory(none)
+; TUNIT-LABEL: define float @call_noinf_0
+; TUNIT-SAME: (float nofpclass(inf) [[ARG:%.*]]) #[[ATTR2]] {
+; TUNIT-NEXT:    [[RESULT:%.*]] = call float @only_nofpclass_inf_callers(float nofpclass(inf) [[ARG]]) #[[ATTR7:[0-9]+]]
+; TUNIT-NEXT:    ret float [[RESULT]]
+;
+; CGSCC: Function Attrs: nofree nosync nounwind willreturn memory(none)
+; CGSCC-LABEL: define float @call_noinf_0
+; CGSCC-SAME: (float nofpclass(inf) [[ARG:%.*]]) #[[ATTR3:[0-9]+]] {
+; CGSCC-NEXT:    [[RESULT:%.*]] = call float @only_nofpclass_inf_callers(float nofpclass(inf) [[ARG]]) #[[ATTR6]]
+; CGSCC-NEXT:    ret float [[RESULT]]
+;
+  %result = call float @only_nofpclass_inf_callers(float %arg)
+  ret float %result
+}
+
+define float @call_noinf_1(float %arg) {
+; TUNIT: Function Attrs: nofree norecurse nosync nounwind willreturn memory(none)
+; TUNIT-LABEL: define float @call_noinf_1
+; TUNIT-SAME: (float [[ARG:%.*]]) #[[ATTR2]] {
+; TUNIT-NEXT:    [[RESULT:%.*]] = call float @only_nofpclass_inf_callers(float nofpclass(inf) [[ARG]]) #[[ATTR7]]
+; TUNIT-NEXT:    ret float [[RESULT]]
+;
+; CGSCC: Function Attrs: nofree nosync nounwind willreturn memory(none)
+; CGSCC-LABEL: define float @call_noinf_1
+; CGSCC-SAME: (float [[ARG:%.*]]) #[[ATTR3]] {
+; CGSCC-NEXT:    [[RESULT:%.*]] = call float @only_nofpclass_inf_callers(float nofpclass(inf) [[ARG]]) #[[ATTR6]]
+; CGSCC-NEXT:    ret float [[RESULT]]
+;
+  %result = call float @only_nofpclass_inf_callers(float nofpclass(inf) %arg)
+  ret float %result
+}
+
+; TODO: Should be able to infer nofpclass(inf) on return
+define internal float @only_nofpclass_inf_return_users(float %arg) {
+; CHECK: Function Attrs: nofree norecurse nosync nounwind willreturn memory(none)
+; CHECK-LABEL: define internal float @only_nofpclass_inf_return_users
+; CHECK-SAME: (float [[ARG:%.*]]) #[[ATTR2]] {
+; CHECK-NEXT:    [[ADD:%.*]] = fadd float [[ARG]], [[ARG]]
+; CHECK-NEXT:    ret float [[ADD]]
+;
+  %add = fadd float %arg, %arg
+  ret float %add
+}
+
+define float @call_noinf_return_0(float %arg) {
+; TUNIT: Function Attrs: nofree norecurse nosync nounwind willreturn memory(none)
+; TUNIT-LABEL: define nofpclass(inf) float @call_noinf_return_0
+; TUNIT-SAME: (float [[ARG:%.*]]) #[[ATTR2]] {
+; TUNIT-NEXT:    [[RESULT:%.*]] = call nofpclass(inf) float @only_nofpclass_inf_return_users(float [[ARG]]) #[[ATTR7]]
+; TUNIT-NEXT:    ret float [[RESULT]]
+;
+; CGSCC: Function Attrs: nofree nosync nounwind willreturn memory(none)
+; CGSCC-LABEL: define nofpclass(inf) float @call_noinf_return_0
+; CGSCC-SAME: (float [[ARG:%.*]]) #[[ATTR3]] {
+; CGSCC-NEXT:    [[RESULT:%.*]] = call nofpclass(inf) float @only_nofpclass_inf_return_users(float [[ARG]]) #[[ATTR6]]
+; CGSCC-NEXT:    ret float [[RESULT]]
+;
+  %result = call nofpclass(inf) float @only_nofpclass_inf_return_users(float %arg)
+  ret float %result
+}
+
+define float @call_noinf_return_1(float %arg) {
+; TUNIT: Function Attrs: nofree norecurse nosync nounwind willreturn memory(none)
+; TUNIT-LABEL: define nofpclass(inf) float @call_noinf_return_1
+; TUNIT-SAME: (float [[ARG:%.*]]) #[[ATTR2]] {
+; TUNIT-NEXT:    [[RESULT:%.*]] = call nofpclass(inf) float @only_nofpclass_inf_return_users(float [[ARG]]) #[[ATTR7]]
+; TUNIT-NEXT:    ret float [[RESULT]]
+;
+; CGSCC: Function Attrs: nofree nosync nounwind willreturn memory(none)
+; CGSCC-LABEL: define nofpclass(inf) float @call_noinf_return_1
+; CGSCC-SAME: (float [[ARG:%.*]]) #[[ATTR3]] {
+; CGSCC-NEXT:    [[RESULT:%.*]] = call nofpclass(inf) float @only_nofpclass_inf_return_users(float [[ARG]]) #[[ATTR6]]
+; CGSCC-NEXT:    ret float [[RESULT]]
+;
+  %result = call nofpclass(inf) float @only_nofpclass_inf_return_users(float %arg)
+  ret float %result
+}
+
+define float @fcmp_olt_assume_one_0_callsite_arg_return(float %arg) {
+; CHECK-LABEL: define nofpclass(inf sub norm) float @fcmp_olt_assume_one_0_callsite_arg_return
+; CHECK-SAME: (float returned nofpclass(inf sub norm) [[ARG:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[IS_NOT_ZERO_OR_NAN:%.*]] = fcmp one float [[ARG]], 0.000000e+00
+; CHECK-NEXT:    call void @llvm.assume(i1 noundef [[IS_NOT_ZERO_OR_NAN]]) #[[ATTR6]]
+; CHECK-NEXT:    call void @extern.use(float nofpclass(inf sub norm) [[ARG]])
+; CHECK-NEXT:    ret float [[ARG]]
+;
+entry:
+  %is.not.zero.or.nan = fcmp one float %arg, 0.0
+  call void @llvm.assume(i1 %is.not.zero.or.nan)
+  call void @extern.use(float %arg)
+  ret float %arg
+}
+
+define float @fcmp_olt_assume_une_0_callsite_arg_return(float %arg) {
+; CHECK-LABEL: define nofpclass(nan inf sub norm) float @fcmp_olt_assume_une_0_callsite_arg_return
+; CHECK-SAME: (float returned nofpclass(nan inf sub norm) [[ARG:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[IS_NOT_ZERO_OR_NAN:%.*]] = fcmp une float [[ARG]], 0.000000e+00
+; CHECK-NEXT:    call void @llvm.assume(i1 noundef [[IS_NOT_ZERO_OR_NAN]]) #[[ATTR6]]
+; CHECK-NEXT:    call void @extern.use(float nofpclass(nan inf sub norm) [[ARG]])
+; CHECK-NEXT:    ret float [[ARG]]
+;
+entry:
+  %is.not.zero.or.nan = fcmp une float %arg, 0.0
+  call void @llvm.assume(i1 %is.not.zero.or.nan)
+  call void @extern.use(float %arg)
+  ret float %arg
+}
+
+define half @fcmp_assume_issubnormal_callsite_arg_return(half %arg) {
+; CHECK-LABEL: define nofpclass(zero sub) half @fcmp_assume_issubnormal_callsite_arg_return
+; CHECK-SAME: (half returned nofpclass(zero sub) [[ARG:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half nofpclass(zero sub) [[ARG]]) #[[ATTR6]]
+; CHECK-NEXT:    [[IS_SUBNORMAL:%.*]] = fcmp olt half [[FABS]], 0xH0400
+; CHECK-NEXT:    call void @llvm.assume(i1 noundef [[IS_SUBNORMAL]]) #[[ATTR6]]
+; CHECK-NEXT:    call void @extern.use.f16(half nofpclass(zero sub) [[ARG]])
+; CHECK-NEXT:    ret half [[ARG]]
+;
+entry:
+  %fabs = call half @llvm.fabs.f16(half %arg)
+  %is.subnormal = fcmp olt half %fabs, 0xH0400
+  call void @llvm.assume(i1 %is.subnormal)
+  call void @extern.use.f16(half %arg)
+  ret half %arg
+}
+
+; Assume is after the call, shouldn't mark callsite.
+define half @fcmp_assume_not_inf_after_call(half %arg) {
+; CHECK-LABEL: define half @fcmp_assume_not_inf_after_call
+; CHECK-SAME: (half returned [[ARG:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    call void @extern.use.f16(half [[ARG]])
+; CHECK-NEXT:    [[NOT_INF:%.*]] = fcmp oeq half [[ARG]], 0xH7C00
+; CHECK-NEXT:    call void @llvm.assume(i1 noundef [[NOT_INF]])
+; CHECK-NEXT:    ret half [[ARG]]
+;
+entry:
+  call void @extern.use.f16(half %arg)
+  %not.inf = fcmp oeq half %arg, 0xH7C00
+  call void @llvm.assume(i1 %not.inf)
+  ret half %arg
+}
+
+; Assume not subnormal or zero, and not infinity
+define half @fcmp_assume2_callsite_arg_return(half %arg) {
+; CHECK-LABEL: define nofpclass(inf norm) half @fcmp_assume2_callsite_arg_return
+; CHECK-SAME: (half returned nofpclass(inf norm) [[ARG:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[FABS:%.*]] = call nofpclass(inf) half @llvm.fabs.f16(half nofpclass(inf norm) [[ARG]]) #[[ATTR6]]
+; CHECK-NEXT:    [[NOT_SUBNORMAL_OR_ZERO:%.*]] = fcmp oge half [[FABS]], 0xH0400
+; CHECK-NEXT:    call void @llvm.assume(i1 noundef [[NOT_SUBNORMAL_OR_ZERO]]) #[[ATTR6]]
+; CHECK-NEXT:    [[NOT_INF:%.*]] = fcmp oeq half [[ARG]], 0xH7C00
+; CHECK-NEXT:    call void @llvm.assume(i1 noundef [[NOT_INF]]) #[[ATTR6]]
+; CHECK-NEXT:    call void @extern.use.f16(half nofpclass(inf norm) [[ARG]])
+; CHECK-NEXT:    ret half [[ARG]]
+;
+entry:
+  %fabs = call half @llvm.fabs.f16(half %arg)
+  %not.subnormal.or.zero = fcmp oge half %fabs, 0xH0400
+  call void @llvm.assume(i1 %not.subnormal.or.zero)
+
+  %not.inf = fcmp oeq half %arg, 0xH7C00
+  call void @llvm.assume(i1 %not.inf)
+
+  call void @extern.use.f16(half %arg)
+  ret half %arg
+}
+
+define float @is_fpclass_assume_arg_return(float %arg) {
+; CHECK-LABEL: define nofpclass(ninf nzero pnorm) float @is_fpclass_assume_arg_return
+; CHECK-SAME: (float returned nofpclass(ninf nzero pnorm) [[ARG:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[CLASS_TEST:%.*]] = call i1 @llvm.is.fpclass.f32(float nofpclass(ninf nzero pnorm) [[ARG]], i32 noundef 292) #[[ATTR6]]
+; CHECK-NEXT:    call void @llvm.assume(i1 noundef [[CLASS_TEST]]) #[[ATTR6]]
+; CHECK-NEXT:    call void @extern.use(float nofpclass(ninf nzero pnorm) [[ARG]])
+; CHECK-NEXT:    ret float [[ARG]]
+;
+entry:
+  %class.test = call i1 @llvm.is.fpclass.f32(float %arg, i32 292)
+  call void @llvm.assume(i1 %class.test)
+  call void @extern.use(float %arg)
+  ret float %arg
+}
+
+; Make sure we don't get confused by looking at an unrelated assume
+; based on the fabs of the value.
+define half @assume_fcmp_fabs_with_other_fabs_assume(half %arg) {
+; CHECK-LABEL: define nofpclass(zero sub) half @assume_fcmp_fabs_with_other_fabs_assume
+; CHECK-SAME: (half returned nofpclass(zero sub) [[ARG:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[FABS:%.*]] = call nofpclass(inf zero sub norm) half @llvm.fabs.f16(half nofpclass(zero sub) [[ARG]]) #[[ATTR6]]
+; CHECK-NEXT:    [[UNRELATED_FABS:%.*]] = fcmp one half [[FABS]], 0xH0000
+; CHECK-NEXT:    call void @llvm.assume(i1 noundef [[UNRELATED_FABS]]) #[[ATTR6]]
+; CHECK-NEXT:    [[IS_SUBNORMAL:%.*]] = fcmp olt half [[FABS]], 0xH0400
+; CHECK-NEXT:    call void @llvm.assume(i1 noundef [[IS_SUBNORMAL]]) #[[ATTR6]]
+; CHECK-NEXT:    call void @extern.use.f16(half nofpclass(zero sub) [[ARG]])
+; CHECK-NEXT:    call void @extern.use.f16(half nofpclass(inf zero sub norm) [[FABS]])
+; CHECK-NEXT:    ret half [[ARG]]
+;
+entry:
+
+  %fabs = call half @llvm.fabs.f16(half %arg)
+  %unrelated.fabs = fcmp one half %fabs, 0.0
+  call void @llvm.assume(i1 %unrelated.fabs)
+  %is.subnormal = fcmp olt half %fabs, 0xH0400
+  call void @llvm.assume(i1 %is.subnormal)
+  call void @extern.use.f16(half %arg)
+  call void @extern.use.f16(half %fabs)
+  ret half %arg
+}
+
+; Make sure if looking through the fabs finds a different source
+; value, we still identify a test mask by ignoring the fabs
+define half @assume_fcmp_fabs_with_other_fabs_assume_fallback(half %arg) {
+; CHECK-LABEL: define nofpclass(pinf zero sub) half @assume_fcmp_fabs_with_other_fabs_assume_fallback
+; CHECK-SAME: (half returned nofpclass(pinf zero sub) [[ARG:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[FABS:%.*]] = call nofpclass(inf zero sub nnorm) half @llvm.fabs.f16(half nofpclass(pinf zero sub) [[ARG]]) #[[ATTR6]]
+; CHECK-NEXT:    [[ONE_INF:%.*]] = fcmp oeq half [[ARG]], 0xH7C00
+; CHECK-NEXT:    call void @llvm.assume(i1 noundef [[ONE_INF]]) #[[ATTR6]]
+; CHECK-NEXT:    [[UNRELATED_FABS:%.*]] = fcmp oeq half [[FABS]], 0xH0000
+; CHECK-NEXT:    call void @llvm.assume(i1 noundef [[UNRELATED_FABS]]) #[[ATTR6]]
+; CHECK-NEXT:    [[IS_SUBNORMAL:%.*]] = fcmp olt half [[FABS]], 0xH0400
+; CHECK-NEXT:    call void @llvm.assume(i1 noundef [[IS_SUBNORMAL]]) #[[ATTR6]]
+; CHECK-NEXT:    call void @extern.use.f16(half nofpclass(pinf zero sub) [[ARG]])
+; CHECK-NEXT:    call void @extern.use.f16(half nofpclass(inf zero sub nnorm) [[FABS]])
+; CHECK-NEXT:    ret half [[ARG]]
+;
+entry:
+
+  %fabs = call half @llvm.fabs.f16(half %arg)
+
+  %one.inf = fcmp oeq half %arg, 0xH7C00
+  call void @llvm.assume(i1 %one.inf)
+
+  %unrelated.fabs = fcmp oeq half %fabs, 0.0
+  call void @llvm.assume(i1 %unrelated.fabs)
+
+  %is.subnormal = fcmp olt half %fabs, 0xH0400
+  call void @llvm.assume(i1 %is.subnormal)
+  call void @extern.use.f16(half %arg)
+  call void @extern.use.f16(half %fabs)
+  ret half %arg
+}
+
+define float @assume_bundles(i1 %c, float %ret) {
+; CHECK-LABEL: define float @assume_bundles
+; CHECK-SAME: (i1 noundef [[C:%.*]], float returned [[RET:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br i1 [[C]], label [[A:%.*]], label [[B:%.*]]
+; CHECK:       A:
+; CHECK-NEXT:    call void @llvm.assume(i1 noundef true) #[[ATTR6]] [ "nofpclass"(float [[RET]], i32 3) ]
+; CHECK-NEXT:    call void @extern.use(float nofpclass(nan) [[RET]])
+; CHECK-NEXT:    ret float [[RET]]
+; CHECK:       B:
+; CHECK-NEXT:    call void @llvm.assume(i1 noundef true) [ "nofpclass"(float [[RET]], i32 12) ]
+; CHECK-NEXT:    call void @extern.use(float nofpclass(ninf nnorm) [[RET]])
+; CHECK-NEXT:    ret float [[RET]]
+;
+entry:
+  br i1 %c, label %A, label %B
+
+A:
+  call void @llvm.assume(i1 true) [ "nofpclass"(float %ret, i32 3) ]
+  call void @extern.use(float %ret)
+  ret float %ret
+
+B:
+  call void @llvm.assume(i1 true) [ "nofpclass"(float %ret, i32 12) ]
+  call void @extern.use(float %ret)
+  ret float %ret
+}
+
+define float @returned_load(ptr %ptr) {
+; CHECK: Function Attrs: nofree norecurse nosync nounwind willreturn memory(argmem: read)
+; CHECK-LABEL: define float @returned_load
+; CHECK-SAME: (ptr nocapture nofree noundef nonnull readonly align 4 dereferenceable(4) [[PTR:%.*]]) #[[ATTR4:[0-9]+]] {
+; CHECK-NEXT:    [[LOAD:%.*]] = load float, ptr [[PTR]], align 4
+; CHECK-NEXT:    ret float [[LOAD]]
+;
+  %load = load float, ptr %ptr
+  ret float %load
+}
+
+define float @pass_nofpclass_inf_through_memory(float nofpclass(inf) %arg) {
+; TUNIT: Function Attrs: nofree norecurse nosync nounwind willreturn memory(none)
+; TUNIT-LABEL: define float @pass_nofpclass_inf_through_memory
+; TUNIT-SAME: (float nofpclass(inf) [[ARG:%.*]]) #[[ATTR2]] {
+; TUNIT-NEXT:    [[ALLOCA:%.*]] = alloca float, align 4
+; TUNIT-NEXT:    store float [[ARG]], ptr [[ALLOCA]], align 4
+; TUNIT-NEXT:    [[RET:%.*]] = call float @returned_load(ptr noalias nocapture nofree noundef nonnull readonly align 4 dereferenceable(4) [[ALLOCA]]) #[[ATTR7]]
+; TUNIT-NEXT:    ret float [[RET]]
+;
+; CGSCC: Function Attrs: nofree nosync nounwind willreturn memory(none)
+; CGSCC-LABEL: define float @pass_nofpclass_inf_through_memory
+; CGSCC-SAME: (float nofpclass(inf) [[ARG:%.*]]) #[[ATTR3]] {
+; CGSCC-NEXT:    [[ALLOCA:%.*]] = alloca float, align 4
+; CGSCC-NEXT:    store float [[ARG]], ptr [[ALLOCA]], align 4
+; CGSCC-NEXT:    [[RET:%.*]] = call float @returned_load(ptr noalias nocapture nofree noundef nonnull readonly align 4 dereferenceable(4) [[ALLOCA]]) #[[ATTR6]]
+; CGSCC-NEXT:    ret float [[RET]]
+;
+  %alloca = alloca float
+  store float %arg, ptr %alloca
+  %ret = call float @returned_load(ptr %alloca)
+  ret float %ret
+}
diff --git a/llvm/test/Transforms/ConstraintElimination/analysis-invalidation.ll b/llvm/test/Transforms/ConstraintElimination/analysis-invalidation.ll
index 7eed403f5b08a..2e82cbed101eb 100644
--- a/llvm/test/Transforms/ConstraintElimination/analysis-invalidation.ll
+++ b/llvm/test/Transforms/ConstraintElimination/analysis-invalidation.ll
@@ -1,4 +1,4 @@
-; RUN: opt -passes='require<demanded-bits>,constraint-elimination,require<demanded-bits>' -disable-verify -verify-cfg-preserved=false -debug-pass-manager -disable-output %s 2>&1 | FileCheck %s
+; RUN: opt -passes='require<demanded-bits>,constraint-elimination,require<demanded-bits>' -disable-verify -verify-analysis-invalidation=false -debug-pass-manager -disable-output %s 2>&1 | FileCheck %s
 
 ; Check that constraint-elimination properly invalidates anlyses.
 
diff --git a/llvm/test/Transforms/ConstraintElimination/loops.ll b/llvm/test/Transforms/ConstraintElimination/loops.ll
index 90c4574e21a0c..df008a9c33ebe 100644
--- a/llvm/test/Transforms/ConstraintElimination/loops.ll
+++ b/llvm/test/Transforms/ConstraintElimination/loops.ll
@@ -155,12 +155,12 @@ exit:
   ret i32 20
 }
 
-define void @loop_header_dom_or(i32 %y, i1 %c) {
+define void @loop_header_dom_or(i32 %y, i1 %c, i32 %start) {
 ; CHECK-LABEL: @loop_header_dom_or(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br i1 [[C:%.*]], label [[LOOP_HEADER:%.*]], label [[EXIT:%.*]]
 ; CHECK:       loop.header:
-; CHECK-NEXT:    [[X:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[X_NEXT:%.*]], [[LOOP_LATCH:%.*]] ]
+; CHECK-NEXT:    [[X:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[X_NEXT:%.*]], [[LOOP_LATCH:%.*]] ]
 ; CHECK-NEXT:    [[X_1:%.*]] = icmp ule i32 [[X]], 10
 ; CHECK-NEXT:    [[Y_1:%.*]] = icmp ugt i32 [[Y:%.*]], 99
 ; CHECK-NEXT:    [[OR:%.*]] = or i1 [[X_1]], [[Y_1]]
@@ -193,7 +193,7 @@ entry:
   br i1 %c, label %loop.header, label %exit
 
 loop.header:
-  %x = phi i32 [ 0, %entry ], [ %x.next, %loop.latch ]
+  %x = phi i32 [ %start, %entry ], [ %x.next, %loop.latch ]
   %x.1 = icmp ule i32 %x, 10
   %y.1 = icmp ugt i32 %y, 99
   %or = or i1 %x.1, %y.1
diff --git a/llvm/test/Transforms/ConstraintElimination/or.ll b/llvm/test/Transforms/ConstraintElimination/or.ll
index 4fa8edd83c291..511aef06ce2f2 100644
--- a/llvm/test/Transforms/ConstraintElimination/or.ll
+++ b/llvm/test/Transforms/ConstraintElimination/or.ll
@@ -454,13 +454,13 @@ define void @test_or_as_add_ult(i8 %init_val, i8 %high) {
 ; CHECK-NEXT:    call void @use(i1 true)
 ; CHECK-NEXT:    [[START_1:%.*]] = or i8 [[START]], 1
 ; CHECK-NEXT:    [[T_1:%.*]] = icmp ult i8 [[START_1]], [[HIGH]]
-; CHECK-NEXT:    call void @use(i1 [[T_1]])
+; CHECK-NEXT:    call void @use(i1 true)
 ; CHECK-NEXT:    [[START_2:%.*]] = or i8 [[START]], 2
 ; CHECK-NEXT:    [[T_2:%.*]] = icmp ult i8 [[START_2]], [[HIGH]]
-; CHECK-NEXT:    call void @use(i1 [[T_2]])
+; CHECK-NEXT:    call void @use(i1 true)
 ; CHECK-NEXT:    [[START_3:%.*]] = or i8 [[START]], 3
 ; CHECK-NEXT:    [[T_3:%.*]] = icmp ult i8 [[START_3]], [[HIGH]]
-; CHECK-NEXT:    call void @use(i1 [[T_3]])
+; CHECK-NEXT:    call void @use(i1 true)
 ; CHECK-NEXT:    [[START_4:%.*]] = or i8 [[START]], 4
 ; CHECK-NEXT:    [[C_4:%.*]] = icmp ult i8 [[START_4]], [[HIGH]]
 ; CHECK-NEXT:    call void @use(i1 [[C_4]])
@@ -523,13 +523,13 @@ define void @test_or_as_add_ule(i8 %init_val, i8 %high) {
 ; CHECK-NEXT:    call void @use(i1 true)
 ; CHECK-NEXT:    [[START_1:%.*]] = or i8 [[START]], 1
 ; CHECK-NEXT:    [[T_1:%.*]] = icmp ule i8 [[START_1]], [[HIGH]]
-; CHECK-NEXT:    call void @use(i1 [[T_1]])
+; CHECK-NEXT:    call void @use(i1 true)
 ; CHECK-NEXT:    [[START_2:%.*]] = or i8 [[START]], 2
 ; CHECK-NEXT:    [[T_2:%.*]] = icmp ule i8 [[START_2]], [[HIGH]]
-; CHECK-NEXT:    call void @use(i1 [[T_2]])
+; CHECK-NEXT:    call void @use(i1 true)
 ; CHECK-NEXT:    [[START_3:%.*]] = or i8 [[START]], 3
 ; CHECK-NEXT:    [[T_3:%.*]] = icmp ule i8 [[START_3]], [[HIGH]]
-; CHECK-NEXT:    call void @use(i1 [[T_3]])
+; CHECK-NEXT:    call void @use(i1 true)
 ; CHECK-NEXT:    [[START_4:%.*]] = or i8 [[START]], 4
 ; CHECK-NEXT:    [[T_4:%.*]] = icmp ule i8 [[START_4]], [[HIGH]]
 ; CHECK-NEXT:    call void @use(i1 [[T_4]])
@@ -599,13 +599,13 @@ define void @test_or_as_add_ugt(i8 %init_val, i8 %high) {
 ; CHECK-NEXT:    call void @use(i1 false)
 ; CHECK-NEXT:    [[START_1:%.*]] = or i8 [[START]], 1
 ; CHECK-NEXT:    [[F_1:%.*]] = icmp ugt i8 [[START_1]], [[HIGH]]
-; CHECK-NEXT:    call void @use(i1 [[F_1]])
+; CHECK-NEXT:    call void @use(i1 false)
 ; CHECK-NEXT:    [[START_2:%.*]] = or i8 [[START]], 2
 ; CHECK-NEXT:    [[F_2:%.*]] = icmp ugt i8 [[START_2]], [[HIGH]]
-; CHECK-NEXT:    call void @use(i1 [[F_2]])
+; CHECK-NEXT:    call void @use(i1 false)
 ; CHECK-NEXT:    [[START_3:%.*]] = or i8 [[START]], 3
 ; CHECK-NEXT:    [[F_3:%.*]] = icmp ugt i8 [[START_3]], [[HIGH]]
-; CHECK-NEXT:    call void @use(i1 [[F_3]])
+; CHECK-NEXT:    call void @use(i1 false)
 ; CHECK-NEXT:    [[START_4:%.*]] = or i8 [[START]], 4
 ; CHECK-NEXT:    [[F_4:%.*]] = icmp ugt i8 [[START_4]], [[HIGH]]
 ; CHECK-NEXT:    call void @use(i1 [[F_4]])
@@ -674,13 +674,13 @@ define void @test_or_as_add_uge(i8 %init_val, i8 %high) {
 ; CHECK-NEXT:    call void @use(i1 false)
 ; CHECK-NEXT:    [[START_1:%.*]] = or i8 [[START]], 1
 ; CHECK-NEXT:    [[F_1:%.*]] = icmp uge i8 [[START_1]], [[HIGH]]
-; CHECK-NEXT:    call void @use(i1 [[F_1]])
+; CHECK-NEXT:    call void @use(i1 false)
 ; CHECK-NEXT:    [[START_2:%.*]] = or i8 [[START]], 2
 ; CHECK-NEXT:    [[F_2:%.*]] = icmp uge i8 [[START_2]], [[HIGH]]
-; CHECK-NEXT:    call void @use(i1 [[F_2]])
+; CHECK-NEXT:    call void @use(i1 false)
 ; CHECK-NEXT:    [[START_3:%.*]] = or i8 [[START]], 3
 ; CHECK-NEXT:    [[F_3:%.*]] = icmp uge i8 [[START_3]], [[HIGH]]
-; CHECK-NEXT:    call void @use(i1 [[F_3]])
+; CHECK-NEXT:    call void @use(i1 false)
 ; CHECK-NEXT:    [[START_4:%.*]] = or i8 [[START]], 4
 ; CHECK-NEXT:    [[C_4:%.*]] = icmp uge i8 [[START_4]], [[HIGH]]
 ; CHECK-NEXT:    call void @use(i1 [[C_4]])
diff --git a/llvm/test/Transforms/ConstraintElimination/shl.ll b/llvm/test/Transforms/ConstraintElimination/shl.ll
index e11fdacbd5497..6d196f77da1c9 100644
--- a/llvm/test/Transforms/ConstraintElimination/shl.ll
+++ b/llvm/test/Transforms/ConstraintElimination/shl.ll
@@ -1223,3 +1223,68 @@ entry:
   %f.1 = icmp ule i8 %start.add.2.13, %start.shl.4
   ret i1 %f.1
 }
+
+define i1 @shl_overflow(i64 %start) {
+; CHECK-LABEL: @shl_overflow(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[PRE_COND:%.*]] = icmp ugt i64 [[START:%.*]], 0
+; CHECK-NEXT:    br i1 [[PRE_COND]], label [[MAIN:%.*]], label [[EXIT:%.*]]
+; CHECK:       main:
+; CHECK-NEXT:    [[TMP0:%.*]] = shl nuw nsw i64 [[START]], -1
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp uge i64 [[TMP0]], [[START]]
+; CHECK-NEXT:    ret i1 [[TMP1]]
+; CHECK:       exit:
+; CHECK-NEXT:    ret i1 false
+;
+entry:
+  %pre.cond = icmp ugt i64 %start, 0
+  br i1 %pre.cond, label %main, label %exit
+
+main:
+  %0 = shl nuw nsw i64 %start, -1
+  %1 = icmp uge i64 %0, %start
+  ret i1 %1
+
+exit:
+  ret i1 0
+}
+
+
+define i1 @shl_overflow_2() {
+; CHECK-LABEL: @shl_overflow_2(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[SHL_UB:%.*]] = shl nuw nsw i256 0, 64
+; CHECK-NEXT:    [[SHL_CMP:%.*]] = icmp uge i256 [[SHL_UB]], 0
+; CHECK-NEXT:    ret i1 [[SHL_CMP]]
+;
+entry:
+  %shl.ub = shl nuw nsw i256 0, 64
+  %shl.cmp = icmp uge i256 %shl.ub, 0
+  ret i1 %shl.cmp
+}
+
+define i1 @shl_overflow_3() {
+; CHECK-LABEL: @shl_overflow_3(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[SHL_UB:%.*]] = shl nuw nsw i256 0, 65
+; CHECK-NEXT:    [[SHL_CMP:%.*]] = icmp uge i256 [[SHL_UB]], 0
+; CHECK-NEXT:    ret i1 [[SHL_CMP]]
+;
+entry:
+  %shl.ub = shl nuw nsw i256 0, 65
+  %shl.cmp = icmp uge i256 %shl.ub, 0
+  ret i1 %shl.cmp
+}
+
+define i1 @shl_55() {
+; CHECK-LABEL: @shl_55(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[SHL_UB:%.*]] = shl nuw nsw i256 1, 55
+; CHECK-NEXT:    [[SHL_CMP:%.*]] = icmp uge i256 [[SHL_UB]], 1
+; CHECK-NEXT:    ret i1 true
+;
+entry:
+  %shl.ub = shl nuw nsw i256 1, 55
+  %shl.cmp = icmp uge i256 %shl.ub, 1
+  ret i1 %shl.cmp
+}
diff --git a/llvm/test/Transforms/Coroutines/remarks.ll b/llvm/test/Transforms/Coroutines/remarks.ll
new file mode 100644
index 0000000000000..be1a213f67a92
--- /dev/null
+++ b/llvm/test/Transforms/Coroutines/remarks.ll
@@ -0,0 +1,71 @@
+; Test coroutine remarks.
+; RUN: opt < %s --disable-output -S -passes='default<O1>' \
+; RUN: --pass-remarks="coro-split|coro-elide" \
+; RUN: --pass-remarks-missed="coro-split|coro-elide" \
+; RUN: --pass-remarks-with-hotness 2>&1 | FileCheck %s
+
+; CHECK: Split 'foo' (frame_size=24, align=8) (hotness: 400)
+; CHECK: 'foo' not elided in 'bar' (frame_size=24, align=8) (hotness: 100)
+; CHECK: 'foo' elided in 'baz' (frame_size=24, align=8) (hotness: 200)
+
+define ptr @foo() presplitcoroutine !prof !0 {
+entry:
+  %id = call token @llvm.coro.id(i32 0, ptr null, ptr null, ptr null)
+  %need.dyn.alloc = call i1 @llvm.coro.alloc(token %id)
+  br i1 %need.dyn.alloc, label %dyn.alloc, label %coro.begin
+dyn.alloc:
+  %size = call i32 @llvm.coro.size.i32()
+  %alloc = call ptr @malloc(i32 %size)
+  br label %coro.begin
+coro.begin:
+  %phi = phi ptr [ null, %entry ], [ %alloc, %dyn.alloc ]
+  %hdl = call ptr @llvm.coro.begin(token %id, ptr %phi)
+  call void @print(i32 0)
+  %0 = call i8 @llvm.coro.suspend(token none, i1 false)
+  switch i8 %0, label %suspend [i8 0, label %resume
+                                i8 1, label %cleanup]
+resume:
+  call void @print(i32 1)
+  br label %cleanup
+
+cleanup:
+  %mem = call ptr @llvm.coro.free(token %id, ptr %hdl)
+  call void @free(ptr %mem)
+  br label %suspend
+suspend:
+  call i1 @llvm.coro.end(ptr %hdl, i1 0)
+  ret ptr %hdl
+}
+
+define i32 @bar() !prof !1 {
+entry:
+  %hdl = call ptr @foo()
+  call void @llvm.coro.resume(ptr %hdl)
+  ret i32 0
+}
+
+define i32 @baz() !prof !2 {
+entry:
+  %hdl = call ptr @foo()
+  call void @llvm.coro.destroy(ptr %hdl)
+  ret i32 0
+}
+
+declare ptr @llvm.coro.free(token, ptr)
+declare i32 @llvm.coro.size.i32()
+declare i8  @llvm.coro.suspend(token, i1)
+declare void @llvm.coro.resume(ptr)
+declare void @llvm.coro.destroy(ptr)
+
+declare token @llvm.coro.id(i32, ptr, ptr, ptr)
+declare i1 @llvm.coro.alloc(token)
+declare ptr @llvm.coro.begin(token, ptr)
+declare i1 @llvm.coro.end(ptr, i1)
+
+declare noalias ptr @malloc(i32)
+declare void @print(i32)
+declare void @free(ptr)
+
+!0 = !{!"function_entry_count", i64 400}
+!1 = !{!"function_entry_count", i64 100}
+!2 = !{!"function_entry_count", i64 200}
diff --git a/llvm/test/Transforms/DeadArgElim/musttail-indirect.ll b/llvm/test/Transforms/DeadArgElim/musttail-indirect.ll
new file mode 100644
index 0000000000000..cdcccb44c0c56
--- /dev/null
+++ b/llvm/test/Transforms/DeadArgElim/musttail-indirect.ll
@@ -0,0 +1,65 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: -p --function-signature
+; RUN: opt -passes=deadargelim -S < %s | FileCheck %s
+
+define internal i32 @test_caller(ptr %fptr, i32 %a, i32 %b) {
+; CHECK-LABEL: define {{[^@]+}}@test_caller(ptr %fptr, i32 %a, i32 %b) {
+; CHECK-NEXT:    %r = musttail call i32 @test(ptr %fptr, i32 %a, i32 poison)
+; CHECK-NEXT:    ret i32 %r
+;
+  %r = musttail call i32 @test(ptr %fptr, i32 %a, i32 %b)
+  ret i32 %r
+}
+
+define internal i32 @test(ptr %fptr, i32 %a, i32 %b) {
+; CHECK-LABEL: define {{[^@]+}}@test(ptr %fptr, i32 %a, i32 %b) {
+; CHECK-NEXT:    %r = musttail call i32 %fptr(ptr %fptr, i32 %a, i32 0)
+; CHECK-NEXT:    ret i32 %r
+;
+  %r = musttail call i32 %fptr(ptr %fptr, i32 %a, i32 0)
+  ret i32 %r
+}
+
+define internal i32 @direct_test() {
+; CHECK-LABEL: define {{[^@]+}}@direct_test() {
+; CHECK-NEXT:    %r = musttail call i32 @foo()
+; CHECK-NEXT:    ret i32 %r
+;
+  %r = musttail call i32 @foo()
+  ret i32 %r
+}
+
+declare i32 @foo()
+
+define internal i32 @ping(i32 %x) {
+; CHECK-LABEL: define {{[^@]+}}@ping(i32 %x) {
+; CHECK-NEXT:    %r = musttail call i32 @pong(i32 %x)
+; CHECK-NEXT:    ret i32 %r
+;
+  %r = musttail call i32 @pong(i32 %x)
+  ret i32 %r
+}
+
+define internal i32 @pong(i32 %x) {
+; CHECK-LABEL: define {{[^@]+}}@pong(i32 %x) {
+; CHECK-NEXT:    %cond = icmp eq i32 %x, 2
+; CHECK-NEXT:    br i1 %cond, label %yes, label %no
+; CHECK:       yes:
+; CHECK-NEXT:    %r1 = musttail call i32 @ping(i32 %x)
+; CHECK-NEXT:    ret i32 %r1
+; CHECK:       no:
+; CHECK-NEXT:    %r2 = musttail call i32 @bar(i32 %x)
+; CHECK-NEXT:    ret i32 %r2
+;
+  %cond = icmp eq i32 %x, 2
+  br i1 %cond, label %yes, label %no
+
+yes:
+  %r1 = musttail call i32 @ping(i32 %x)
+  ret i32 %r1
+no:
+  %r2 = musttail call i32 @bar(i32 %x)
+  ret i32 %r2
+}
+
+declare i32 @bar(i32 %x)
+
diff --git a/llvm/test/Transforms/FunctionSpecialization/bug55000-read-uninitialized-value.ll b/llvm/test/Transforms/FunctionSpecialization/bug55000-read-uninitialized-value.ll
index bae203a202c86..0cd1538445434 100644
--- a/llvm/test/Transforms/FunctionSpecialization/bug55000-read-uninitialized-value.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/bug55000-read-uninitialized-value.ll
@@ -1,4 +1,4 @@
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -func-specialization-max-iters=2 -func-specialization-max-clones=1 -function-specialization-for-literal-constant=true -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -funcspec-max-iters=2 -funcspec-max-clones=1 -funcspec-for-literal-constant=true -S < %s | FileCheck %s
 
 declare hidden i1 @compare(ptr) align 2
 declare hidden { i8, ptr } @getType(ptr) align 2
diff --git a/llvm/test/Transforms/FunctionSpecialization/compiler-crash-60191.ll b/llvm/test/Transforms/FunctionSpecialization/compiler-crash-60191.ll
index 9d41cc82fb8ac..35364c4b0ad2b 100644
--- a/llvm/test/Transforms/FunctionSpecialization/compiler-crash-60191.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/compiler-crash-60191.ll
@@ -1,4 +1,4 @@
-; RUN: opt -S --passes="ipsccp<func-spec>" -force-function-specialization < %s | FileCheck %s
+; RUN: opt -S --passes="ipsccp<func-spec>" -force-specialization < %s | FileCheck %s
 
 @A = private constant [6 x i32] [i32 1, i32 2, i32 0, i32 0, i32 0, i32 0], align 16
 @B = external global ptr, align 8
diff --git a/llvm/test/Transforms/FunctionSpecialization/compiler-crash-promote-alloca.ll b/llvm/test/Transforms/FunctionSpecialization/compiler-crash-promote-alloca.ll
index 3946b2dd97f73..4d00d7a284ab7 100644
--- a/llvm/test/Transforms/FunctionSpecialization/compiler-crash-promote-alloca.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/compiler-crash-promote-alloca.ll
@@ -1,4 +1,4 @@
-; RUN: opt -S --passes="ipsccp<func-spec>" -force-function-specialization < %s | FileCheck %s
+; RUN: opt -S --passes="ipsccp<func-spec>" -force-specialization < %s | FileCheck %s
 
 ; Tests that `bar` has been specialized and that the compiler did not crash
 ; while attempting to promote the alloca in `entry`.
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-always-inline.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-always-inline.ll
index bac810cf42155..b6e726ea46af0 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-always-inline.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-always-inline.ll
@@ -1,4 +1,4 @@
-; RUN: opt -passes="ipsccp<func-spec>" -func-specialization-avg-iters-cost=3 -func-specialization-size-threshold=10 -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -funcspec-avg-loop-iters=3 -funcspec-min-function-size=10 -S < %s | FileCheck %s
 
 ; CHECK-NOT: foo.{{[0-9]+}}
 
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression.ll
index 0950cac7be80f..a9cb4184bb8a7 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression.ll
@@ -4,7 +4,7 @@
 ; Note that this test case shows that function specialization pass would
 ; transform the function even if no specialization happened.
 
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -S < %s | FileCheck %s
 
 %struct = type { i8, i16, i32, i64, i64}
 @Global = internal constant %struct {i8 0, i16 1, i32 2, i64 3, i64 4}
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression2.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression2.ll
index edd68e27b515d..2dd9e2c461b60 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression2.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression2.ll
@@ -1,5 +1,5 @@
 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -S < %s | FileCheck %s
 
 ; Check that we don't crash and specialise on a constant expression.
 
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression3.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression3.ll
index 5e6feec776ebf..b1b7f1fd820d6 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression3.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression3.ll
@@ -1,5 +1,5 @@
 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -S < %s | FileCheck %s
 
 define i32 @main() {
 ; CHECK-LABEL: @main(
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression4.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression4.ll
index 6b22858c7e19c..9cac9ca95f8bc 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression4.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression4.ll
@@ -1,4 +1,4 @@
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -S < %s | FileCheck %s
 
 ; Check that we don't crash and specialise on a function call with byval attribute.
 
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression5.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression5.ll
index cc70df6b9fe2f..c53673cf84b63 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression5.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-expression5.ll
@@ -1,5 +1,5 @@
 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -func-specialization-on-address -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -funcspec-on-address -S < %s | FileCheck %s
 
 ; Check that we don't crash and specialise on a scalar global variable with byval attribute.
 
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-integers.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-integers.ll
index 80a10ac35bc56..1b00e1032bd43 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-integers.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-constant-integers.ll
@@ -1,4 +1,4 @@
-; RUN: opt -passes="ipsccp<func-spec>" -function-specialization-for-literal-constant=true -func-specialization-size-threshold=10 -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -funcspec-for-literal-constant=true -funcspec-min-function-size=10 -S < %s | FileCheck %s
 
 ; Check that the literal constant parameter could be specialized.
 ; CHECK: @foo.1(
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-loop.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-loop.ll
index cb4674502ccf0..0e6fe4ce19872 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-loop.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-loop.ll
@@ -1,4 +1,4 @@
-; RUN: opt -passes="ipsccp<func-spec>" -func-specialization-avg-iters-cost=5 -func-specialization-size-threshold=10 -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -funcspec-avg-loop-iters=5 -funcspec-min-function-size=10 -S < %s | FileCheck %s
 
 ; Check that the loop depth results in a larger specialization bonus.
 ; CHECK: @foo.1(
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-minsize2.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-minsize2.ll
index 6129a6ef5e6ae..2d0e04d01dc37 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-minsize2.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-minsize2.ll
@@ -1,4 +1,4 @@
-; RUN: opt -passes="ipsccp<func-spec>" -func-specialization-size-threshold=3 -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -funcspec-min-function-size=3 -S < %s | FileCheck %s
 
 ; Checks for callsites that have been annotated with MinSize. No specialisation
 ; expected here:
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-minsize3.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-minsize3.ll
index ae8b1b6e53fe8..525721f03cfb2 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-minsize3.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-minsize3.ll
@@ -1,4 +1,4 @@
-; RUN: opt -passes="ipsccp<func-spec>" -func-specialization-size-threshold=3 -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -funcspec-min-function-size=3 -S < %s | FileCheck %s
 
 ; Checks for callsites that have been annotated with MinSize. We only expect
 ; specialisation for the call that does not have the attribute:
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-nodup.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-nodup.ll
index 8b0878477367c..d9dcb44dcdb52 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-nodup.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-nodup.ll
@@ -1,4 +1,4 @@
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -S < %s | FileCheck %s
 
 ; Function @foo has function attribute 'noduplicate', so check that we don't
 ; specialize it:
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-nodup2.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-nodup2.ll
index 04e49e6e869d9..c950dfa31e4b2 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-nodup2.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-nodup2.ll
@@ -1,4 +1,4 @@
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -S < %s | FileCheck %s
 
 ; Check that function foo does not gets specialised as it contains an intrinsic
 ; that is marked as NoDuplicate.
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-noexec.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-noexec.ll
index 90e76217ab928..d1e2a77dfc19c 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-noexec.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-noexec.ll
@@ -1,4 +1,4 @@
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -S < %s | FileCheck %s
 
 ; The if.then block is not executed, so check that we don't specialise here.
 
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-nonconst-glob.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-nonconst-glob.ll
index 345bf952a12da..54eed8d1346fe 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-nonconst-glob.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-nonconst-glob.ll
@@ -1,8 +1,8 @@
 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -S < %s | FileCheck %s
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -func-specialization-on-address=0 -S < %s | FileCheck %s
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -func-specialization-on-address=1 -S < %s | FileCheck %s --check-prefix=ON-ADDRESS
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -funcspec-on-address=0 -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -funcspec-on-address=1 -S < %s | FileCheck %s --check-prefix=ON-ADDRESS
 
 ; Global B is not constant. We do not specialise on addresses unless we
 ; enable that:
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-poison.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-poison.ll
index 1617f86aaa1fb..76cf05ab2aa14 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-poison.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-poison.ll
@@ -1,5 +1,5 @@
 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -S < %s | FileCheck %s
 
 ; Check that we don't crash and specialise on a poison value.
 
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-recursive.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-recursive.ll
index c12e02e1ecc92..2cee26a911a47 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-recursive.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-recursive.ll
@@ -1,6 +1,6 @@
-; RUN: opt -passes="ipsccp<func-spec>,inline,instcombine" -force-function-specialization -func-specialization-max-iters=2 -S < %s | FileCheck %s --check-prefix=ITERS2
-; RUN: opt -passes="ipsccp<func-spec>,inline,instcombine" -force-function-specialization -func-specialization-max-iters=3 -S < %s | FileCheck %s --check-prefix=ITERS3
-; RUN: opt -passes="ipsccp<func-spec>,inline,instcombine" -force-function-specialization -func-specialization-max-iters=4 -S < %s | FileCheck %s --check-prefix=ITERS4
+; RUN: opt -passes="ipsccp<func-spec>,inline,instcombine" -force-specialization -funcspec-max-iters=2 -S < %s | FileCheck %s --check-prefix=ITERS2
+; RUN: opt -passes="ipsccp<func-spec>,inline,instcombine" -force-specialization -funcspec-max-iters=3 -S < %s | FileCheck %s --check-prefix=ITERS3
+; RUN: opt -passes="ipsccp<func-spec>,inline,instcombine" -force-specialization -funcspec-max-iters=4 -S < %s | FileCheck %s --check-prefix=ITERS4
 
 @low = internal constant i32 0, align 4
 @high = internal constant i32 6, align 4
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-recursive2.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-recursive2.ll
index 9ab2328452d6c..9c7d3b22c917d 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-recursive2.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-recursive2.ll
@@ -1,4 +1,4 @@
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -func-specialization-max-iters=2 -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -funcspec-max-iters=2 -S < %s | FileCheck %s
 
 ; Volatile store preventing recursive specialisation:
 ;
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-recursive3.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-recursive3.ll
index 9956fe1fb9635..633138721e554 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-recursive3.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-recursive3.ll
@@ -1,4 +1,4 @@
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -func-specialization-max-iters=2 -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -funcspec-max-iters=2 -S < %s | FileCheck %s
 
 ; Duplicate store preventing recursive specialisation:
 ;
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-recursive4.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-recursive4.ll
index 6e3f6762d11a8..6dca04c17bf4d 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-recursive4.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-recursive4.ll
@@ -1,4 +1,4 @@
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -func-specialization-max-iters=2 -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -funcspec-max-iters=2 -S < %s | FileCheck %s
 
 ; Alloca is not an integer type:
 ;
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization-stats.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization-stats.ll
index 42a7676cb54f3..c1ca3eb3ae299 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization-stats.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization-stats.ll
@@ -1,7 +1,7 @@
 ; REQUIRES: asserts
-; RUN: opt -stats -passes="ipsccp<func-spec>" -S -force-function-specialization < %s 2>&1 | FileCheck %s
+; RUN: opt -stats -passes="ipsccp<func-spec>" -S -force-specialization < %s 2>&1 | FileCheck %s
 
-; CHECK: 2 function-specialization - Number of functions specialized
+; CHECK: 2 function-specialization - Number of specializations created
 
 define i64 @main(i64 %x, i1 %flag) {
 entry:
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization.ll
index 90b1aedc37a48..b5d16f6dab1c0 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization.ll
@@ -1,5 +1,5 @@
-; RUN: opt -passes="ipsccp<func-spec>" -func-specialization-size-threshold=3 -S < %s | FileCheck %s
-; RUN: opt -passes="ipsccp<no-func-spec>" -func-specialization-size-threshold=3 -S < %s | FileCheck %s --check-prefix=NOFSPEC
+; RUN: opt -passes="ipsccp<func-spec>" -funcspec-min-function-size=3 -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<no-func-spec>" -funcspec-min-function-size=3 -S < %s | FileCheck %s --check-prefix=NOFSPEC
 
 define i64 @main(i64 %x, i1 %flag) {
 ;
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization2.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization2.ll
index d3344fdbfe27a..c7ef3e8ef520a 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization2.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization2.ll
@@ -1,8 +1,8 @@
 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -passes="ipsccp<func-spec>,deadargelim" -force-function-specialization -S < %s | FileCheck %s
-; RUN: opt -passes="ipsccp<func-spec>,deadargelim" -func-specialization-max-iters=1 -force-function-specialization -S < %s | FileCheck %s
-; RUN: opt -passes="ipsccp<func-spec>,deadargelim" -func-specialization-max-iters=0 -force-function-specialization -S < %s | FileCheck %s --check-prefix=DISABLED
-; RUN: opt -passes="ipsccp<func-spec>,deadargelim" -func-specialization-avg-iters-cost=1 -force-function-specialization -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>,deadargelim" -force-specialization -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>,deadargelim" -funcspec-max-iters=1 -force-specialization -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>,deadargelim" -funcspec-max-iters=0 -force-specialization -S < %s | FileCheck %s --check-prefix=DISABLED
+; RUN: opt -passes="ipsccp<func-spec>,deadargelim" -funcspec-avg-loop-iters=1 -force-specialization -S < %s | FileCheck %s
 
 ; DISABLED-NOT: @func.1(
 ; DISABLED-NOT: @func.2(
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization3.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization3.ll
index 8625a82a07e95..83e6b6660dc09 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization3.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization3.ll
@@ -1,8 +1,8 @@
-; RUN: opt -passes="ipsccp<func-spec>" -func-specialization-avg-iters-cost=3 -S < %s | \
+; RUN: opt -passes="ipsccp<func-spec>" -funcspec-avg-loop-iters=3 -S < %s | \
 ; RUN:   FileCheck %s --check-prefixes=COMMON,DISABLED
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -S < %s | \
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -S < %s | \
 ; RUN:   FileCheck %s --check-prefixes=COMMON,FORCE
-; RUN: opt -passes="ipsccp<func-spec>" -func-specialization-avg-iters-cost=3 -force-function-specialization -S < %s | \
+; RUN: opt -passes="ipsccp<func-spec>" -funcspec-avg-loop-iters=3 -force-specialization -S < %s | \
 ; RUN:   FileCheck %s --check-prefixes=COMMON,FORCE
 
 ; Test for specializing a constant global.
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization4.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization4.ll
index 8c00a466611b6..0e859a0a24feb 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization4.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization4.ll
@@ -1,8 +1,8 @@
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization \
-; RUN:   -func-specialization-max-clones=2 -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization \
+; RUN:   -funcspec-max-clones=2 -S < %s | FileCheck %s
 
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization \
-; RUN:   -func-specialization-max-clones=1 -S < %s | FileCheck %s --check-prefix=CONST1
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization \
+; RUN:   -funcspec-max-clones=1 -S < %s | FileCheck %s --check-prefix=CONST1
 
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
 
diff --git a/llvm/test/Transforms/FunctionSpecialization/function-specialization5.ll b/llvm/test/Transforms/FunctionSpecialization/function-specialization5.ll
index fd0557a9e27d2..b272510b3939f 100644
--- a/llvm/test/Transforms/FunctionSpecialization/function-specialization5.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/function-specialization5.ll
@@ -1,4 +1,4 @@
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -S < %s | FileCheck %s
 
 ; There's nothing to specialize here as both calls are the same, so check that:
 ;
diff --git a/llvm/test/Transforms/FunctionSpecialization/global-rank.ll b/llvm/test/Transforms/FunctionSpecialization/global-rank.ll
index 055a82003dba2..541faa2e19515 100644
--- a/llvm/test/Transforms/FunctionSpecialization/global-rank.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/global-rank.ll
@@ -1,4 +1,4 @@
-; RUN: opt -S --passes="ipsccp<func-spec>" -func-specialization-max-clones=1 < %s | FileCheck %s
+; RUN: opt -S --passes="ipsccp<func-spec>" -funcspec-max-clones=1 < %s | FileCheck %s
 define internal i32 @f(i32 noundef %x, ptr nocapture noundef readonly %p, ptr nocapture noundef readonly %q) noinline {
 entry:
   %call = tail call i32 %p(i32 noundef %x)
diff --git a/llvm/test/Transforms/FunctionSpecialization/identical-specializations.ll b/llvm/test/Transforms/FunctionSpecialization/identical-specializations.ll
index d2c6de3a8c036..c2ba0920c2be3 100644
--- a/llvm/test/Transforms/FunctionSpecialization/identical-specializations.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/identical-specializations.ll
@@ -1,5 +1,5 @@
 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -passes="ipsccp<func-spec>" -force-function-specialization -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -force-specialization -S < %s | FileCheck %s
 
 define i64 @main(i64 %x, i64 %y, i1 %flag) {
 ; CHECK-LABEL: @main(
diff --git a/llvm/test/Transforms/FunctionSpecialization/literal-const.ll b/llvm/test/Transforms/FunctionSpecialization/literal-const.ll
index 824c719285a3f..fc400202ab91e 100644
--- a/llvm/test/Transforms/FunctionSpecialization/literal-const.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/literal-const.ll
@@ -1,8 +1,8 @@
 ; RUN: opt -S --passes="ipsccp<func-spec>" \
-; RUN:        -force-function-specialization < %s | FileCheck %s -check-prefix CHECK-NOLIT
+; RUN:        -force-specialization < %s | FileCheck %s -check-prefix CHECK-NOLIT
 ; RUN: opt -S --passes="ipsccp<func-spec>" \
-; RUN:        -function-specialization-for-literal-constant \
-; RUN:        -force-function-specialization < %s | FileCheck %s -check-prefix CHECK-LIT
+; RUN:        -funcspec-for-literal-constant \
+; RUN:        -force-specialization < %s | FileCheck %s -check-prefix CHECK-LIT
 
 define i32 @f0(i32 noundef %x) {
 entry:
diff --git a/llvm/test/Transforms/FunctionSpecialization/no-spec-unused-arg.ll b/llvm/test/Transforms/FunctionSpecialization/no-spec-unused-arg.ll
index 17dfe9aa4066f..38345652bfc8c 100644
--- a/llvm/test/Transforms/FunctionSpecialization/no-spec-unused-arg.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/no-spec-unused-arg.ll
@@ -1,4 +1,4 @@
-; RUN: opt -S --passes="ipsccp<func-spec>" -force-function-specialization -function-specialization-for-literal-constant < %s | FileCheck %s
+; RUN: opt -S --passes="ipsccp<func-spec>" -force-specialization -funcspec-for-literal-constant < %s | FileCheck %s
 define internal i32 @f(i32 %x, i32 %y) noinline {
     ret i32 %x
 }
diff --git a/llvm/test/Transforms/FunctionSpecialization/remove-dead-recursive-function.ll b/llvm/test/Transforms/FunctionSpecialization/remove-dead-recursive-function.ll
index 6563f3b79e03d..4233998ad9f6d 100644
--- a/llvm/test/Transforms/FunctionSpecialization/remove-dead-recursive-function.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/remove-dead-recursive-function.ll
@@ -1,4 +1,4 @@
-; RUN: opt -passes="ipsccp<func-spec>" -func-specialization-size-threshold=3 -S < %s | FileCheck %s
+; RUN: opt -passes="ipsccp<func-spec>" -funcspec-min-function-size=3 -S < %s | FileCheck %s
 
 define i64 @main(i64 %x, i1 %flag) {
 entry:
diff --git a/llvm/test/Transforms/FunctionSpecialization/specialization-order.ll b/llvm/test/Transforms/FunctionSpecialization/specialization-order.ll
index cc1675db5137c..e0afb3bef1039 100644
--- a/llvm/test/Transforms/FunctionSpecialization/specialization-order.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/specialization-order.ll
@@ -1,4 +1,4 @@
-; RUN: opt -S --passes="ipsccp<func-spec>,deadargelim" -force-function-specialization < %s | FileCheck %s
+; RUN: opt -S --passes="ipsccp<func-spec>,deadargelim" -force-specialization < %s | FileCheck %s
 define dso_local i32 @add(i32 %x, i32 %y) {
 entry:
   %add = add nsw i32 %y, %x
diff --git a/llvm/test/Transforms/FunctionSpecialization/specialize-multiple-arguments.ll b/llvm/test/Transforms/FunctionSpecialization/specialize-multiple-arguments.ll
index 35b65828052ef..d1c23e07d5972 100644
--- a/llvm/test/Transforms/FunctionSpecialization/specialize-multiple-arguments.ll
+++ b/llvm/test/Transforms/FunctionSpecialization/specialize-multiple-arguments.ll
@@ -1,8 +1,8 @@
 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -passes="ipsccp<func-spec>" -func-specialization-max-clones=0 -func-specialization-size-threshold=14 -S < %s | FileCheck %s --check-prefix=NONE
-; RUN: opt -passes="ipsccp<func-spec>" -func-specialization-max-clones=1 -func-specialization-size-threshold=14 -S < %s | FileCheck %s --check-prefix=ONE
-; RUN: opt -passes="ipsccp<func-spec>" -func-specialization-max-clones=2 -func-specialization-size-threshold=14 -S < %s | FileCheck %s --check-prefix=TWO
-; RUN: opt -passes="ipsccp<func-spec>" -func-specialization-max-clones=3 -func-specialization-size-threshold=14 -S < %s | FileCheck %s --check-prefix=THREE
+; RUN: opt -passes="ipsccp<func-spec>" -funcspec-max-clones=0 -funcspec-min-function-size=14 -S < %s | FileCheck %s --check-prefix=NONE
+; RUN: opt -passes="ipsccp<func-spec>" -funcspec-max-clones=1 -funcspec-min-function-size=14 -S < %s | FileCheck %s --check-prefix=ONE
+; RUN: opt -passes="ipsccp<func-spec>" -funcspec-max-clones=2 -funcspec-min-function-size=14 -S < %s | FileCheck %s --check-prefix=TWO
+; RUN: opt -passes="ipsccp<func-spec>" -funcspec-max-clones=3 -funcspec-min-function-size=14 -S < %s | FileCheck %s --check-prefix=THREE
 
 ; Make sure that we iterate correctly after sorting the specializations:
 ; FnSpecialization: Specializations for function compute
diff --git a/llvm/test/Transforms/GVN/constexpr-vector-constainsundef-crash-inseltpoison.ll b/llvm/test/Transforms/GVN/constexpr-vector-constainsundef-crash-inseltpoison.ll
index fb2133182503a..a40811aa1dde1 100644
--- a/llvm/test/Transforms/GVN/constexpr-vector-constainsundef-crash-inseltpoison.ll
+++ b/llvm/test/Transforms/GVN/constexpr-vector-constainsundef-crash-inseltpoison.ll
@@ -9,7 +9,7 @@ define <4 x ptr> @test(ptr %ptr) {
 ; CHECK-LABEL: @test(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[L3:%.*]] = load i64, ptr [[PTR:%.*]], align 4
-; CHECK-NEXT:    [[I6:%.*]] = insertelement <4 x ptr> getelementptr (i64, ptr null, <4 x i64> <i64 poison, i64 poison, i64 poison, i64 -128>), ptr undef, i64 [[L3]]
+; CHECK-NEXT:    [[I6:%.*]] = insertelement <4 x ptr> getelementptr inbounds (i64, ptr null, <4 x i64> <i64 poison, i64 poison, i64 poison, i64 -128>), ptr undef, i64 [[L3]]
 ; CHECK-NEXT:    ret <4 x ptr> [[I6]]
 ;
 entry:
diff --git a/llvm/test/Transforms/GVN/constexpr-vector-constainsundef-crash.ll b/llvm/test/Transforms/GVN/constexpr-vector-constainsundef-crash.ll
index 276008f6348a3..ddb223fffab5e 100644
--- a/llvm/test/Transforms/GVN/constexpr-vector-constainsundef-crash.ll
+++ b/llvm/test/Transforms/GVN/constexpr-vector-constainsundef-crash.ll
@@ -8,7 +8,9 @@
 define <4 x ptr> @test(ptr %ptr) {
 ; CHECK-LABEL: @test(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    ret <4 x ptr> getelementptr (i64, ptr null, <4 x i64> <i64 0, i64 0, i64 0, i64 -128>)
+; CHECK-NEXT:    [[L3:%.*]] = load i64, ptr [[PTR:%.*]], align 4
+; CHECK-NEXT:    [[I6:%.*]] = insertelement <4 x ptr> getelementptr inbounds (i64, ptr null, <4 x i64> <i64 0, i64 0, i64 0, i64 -128>), ptr undef, i64 [[L3]]
+; CHECK-NEXT:    ret <4 x ptr> [[I6]]
 ;
 entry:
   %B9 = sdiv i16 -32768, 256
diff --git a/llvm/test/Transforms/GVNHoist/hoist-call.ll b/llvm/test/Transforms/GVNHoist/hoist-call.ll
index 206b782567854..7c1fe26a827cc 100644
--- a/llvm/test/Transforms/GVNHoist/hoist-call.ll
+++ b/llvm/test/Transforms/GVNHoist/hoist-call.ll
@@ -1,13 +1,20 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -S -passes=gvn-hoist < %s | FileCheck %s
 
 ; Check that the call and fcmp are hoisted.
-; CHECK-LABEL: define void @fun(
-; CHECK: call float
-; CHECK: fcmp oeq
-; CHECK-NOT: call float
-; CHECK-NOT: fcmp oeq
-
 define void @fun(float %__b) minsize {
+; CHECK-LABEL: @fun(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[IF_THEN:%.*]]
+; CHECK:       if.then:
+; CHECK-NEXT:    [[TMP0:%.*]] = call float @llvm.fabs.f32(float [[__B:%.*]])
+; CHECK-NEXT:    [[CMPINF7:%.*]] = fcmp oeq float [[TMP0]], 0x7FF0000000000000
+; CHECK-NEXT:    br i1 undef, label [[IF_THEN8:%.*]], label [[LOR_LHS_FALSE:%.*]]
+; CHECK:       lor.lhs.false:
+; CHECK-NEXT:    unreachable
+; CHECK:       if.then8:
+; CHECK-NEXT:    ret void
+;
 entry:
   br label %if.then
 
@@ -26,3 +33,171 @@ if.then8:                                         ; preds = %if.then
 }
 
 declare float @llvm.fabs.f32(float)
+
+; Check that extractvalues are not hoisted into entry, but that non-dependent
+; adds are.
+define i32 @foo(i32 %x) {
+; CHECK-LABEL: @foo(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[X:%.*]], 1
+; CHECK-NEXT:    [[TMP0:%.*]] = callbr { i32, i32 } asm sideeffect "somestuff", "=r,=r,!i"()
+; CHECK-NEXT:    to label [[ASM_FALLTHROUGH:%.*]] [label %err.split]
+; CHECK:       asm.fallthrough:
+; CHECK-NEXT:    [[ASMRESULT:%.*]] = extractvalue { i32, i32 } [[TMP0]], 0
+; CHECK-NEXT:    ret i32 [[ADD]]
+; CHECK:       err.split:
+; CHECK-NEXT:    [[ASMRESULT2:%.*]] = extractvalue { i32, i32 } [[TMP0]], 0
+; CHECK-NEXT:    ret i32 [[ADD]]
+;
+entry:
+  %0 = callbr { i32, i32 } asm sideeffect "somestuff", "=r,=r,!i"()
+  to label %asm.fallthrough [label %err.split]
+
+asm.fallthrough:                                  ; preds = %entry
+  %asmresult = extractvalue { i32, i32 } %0, 0
+  %add = add nsw i32 %x, 1
+  ret i32 %add
+
+err.split:                                        ; preds = %entry
+  %asmresult2 = extractvalue { i32, i32 } %0, 0
+  %add2 = add nsw i32 %x, 1
+  ret i32 %add2
+}
+
+; Check that extractvalues and dependent adds are not hoisted into entry.
+define i32 @foo2() {
+; CHECK-LABEL: @foo2(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = callbr { i32, i32 } asm sideeffect "somestuff", "=r,=r,!i"()
+; CHECK-NEXT:    to label [[ASM_FALLTHROUGH:%.*]] [label %err.split]
+; CHECK:       asm.fallthrough:
+; CHECK-NEXT:    [[ASMRESULT:%.*]] = extractvalue { i32, i32 } [[TMP0]], 0
+; CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[ASMRESULT]], 1
+; CHECK-NEXT:    ret i32 [[ADD]]
+; CHECK:       err.split:
+; CHECK-NEXT:    [[ASMRESULT2:%.*]] = extractvalue { i32, i32 } [[TMP0]], 0
+; CHECK-NEXT:    [[ADD2:%.*]] = add nsw i32 [[ASMRESULT2]], 1
+; CHECK-NEXT:    ret i32 [[ADD2]]
+;
+entry:
+  %0 = callbr { i32, i32 } asm sideeffect "somestuff", "=r,=r,!i"()
+  to label %asm.fallthrough [label %err.split]
+
+asm.fallthrough:                                  ; preds = %entry
+  %asmresult = extractvalue { i32, i32 } %0, 0
+  %add = add nsw i32 %asmresult, 1
+  ret i32 %add
+
+err.split:                                        ; preds = %entry
+  %asmresult2 = extractvalue { i32, i32 } %0, 0
+  %add2 = add nsw i32 %asmresult2, 1
+  ret i32 %add2
+}
+
+; Ensure we don't hoist loads that are modified by callbr.
+@x = global i32 0
+define i32 @foo3() {
+; CHECK-LABEL: @foo3(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    callbr void asm "", "=*m,!i"(ptr elementtype(i32) @x)
+; CHECK-NEXT:    to label [[ASM_FALLTHROUGH:%.*]] [label %err.split]
+; CHECK:       asm.fallthrough:
+; CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr @x, align 4
+; CHECK-NEXT:    ret i32 [[TMP0]]
+; CHECK:       err.split:
+; CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr @x, align 4
+; CHECK-NEXT:    ret i32 [[TMP1]]
+;
+entry:
+  callbr void asm "", "=*m,!i"(ptr elementtype(i32) @x)
+  to label %asm.fallthrough [label %err.split]
+
+asm.fallthrough:                                  ; preds = %entry
+  %0 = load i32, ptr @x
+  ret i32 %0
+
+err.split:                                        ; preds = %entry
+  %1 = load i32, ptr @x
+  ret i32 %1
+}
+
+; Ensure we do hoist loads that aren't modified by callbr, if the callbr has
+; the attribute memory(argmem:readwrite).
+@y = global i32 0
+define i32 @foo4() {
+; CHECK-LABEL: @foo4(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr @y, align 4
+; CHECK-NEXT:    callbr void asm "", "=*m,!i"(ptr elementtype(i32) @x) #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    to label [[A:%.*]] [label %b]
+; CHECK:       a:
+; CHECK-NEXT:    ret i32 [[TMP0]]
+; CHECK:       b:
+; CHECK-NEXT:    ret i32 [[TMP0]]
+;
+entry:
+  callbr void asm "", "=*m,!i"(ptr elementtype(i32) @x) memory(argmem: readwrite)
+  to label %a [label %b]
+
+a:                                  ; preds = %entry
+  %0 = load i32, ptr @y
+  ret i32 %0
+
+b:                                        ; preds = %entry
+  %1 = load i32, ptr @y
+  ret i32 %1
+}
+
+; Ensure we don't hoist loads that are modified by callbr, if the callbr has
+; the attribute memory(argmem:readwrite).
+define i32 @foo5() {
+; CHECK-LABEL: @foo5(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    callbr void asm "", "=*m,!i"(ptr elementtype(i32) @x) #[[ATTR2]]
+; CHECK-NEXT:    to label [[A:%.*]] [label %b]
+; CHECK:       a:
+; CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr @x, align 4
+; CHECK-NEXT:    ret i32 [[TMP0]]
+; CHECK:       b:
+; CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr @x, align 4
+; CHECK-NEXT:    ret i32 [[TMP1]]
+;
+entry:
+  callbr void asm "", "=*m,!i"(ptr elementtype(i32) @x) memory(argmem: readwrite)
+  to label %a [label %b]
+
+a:                                  ; preds = %entry
+  %0 = load i32, ptr @x
+  ret i32 %0
+
+b:                                        ; preds = %entry
+  %1 = load i32, ptr @x
+  ret i32 %1
+}
+
+; Ensure we hoist loads that are modified by callbr, if the callbr has the
+; attribute memory(argmem:none).
+define i32 @foo6() {
+; CHECK-LABEL: @foo6(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr @x, align 4
+; CHECK-NEXT:    callbr void asm "", "=*m,!i"(ptr elementtype(i32) @x) #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    to label [[A:%.*]] [label %b]
+; CHECK:       a:
+; CHECK-NEXT:    ret i32 [[TMP0]]
+; CHECK:       b:
+; CHECK-NEXT:    ret i32 [[TMP0]]
+;
+entry:
+  callbr void asm "", "=*m,!i"(ptr elementtype(i32) @x) memory(argmem: none)
+  to label %a [label %b]
+
+a:                                  ; preds = %entry
+  %0 = load i32, ptr @x
+  ret i32 %0
+
+b:                                        ; preds = %entry
+  %1 = load i32, ptr @x
+  ret i32 %1
+}
+
diff --git a/llvm/test/Transforms/GlobalOpt/load-failed-to-evaluate-initializer.ll b/llvm/test/Transforms/GlobalOpt/load-failed-to-evaluate-initializer.ll
new file mode 100644
index 0000000000000..9ceac10ad7659
--- /dev/null
+++ b/llvm/test/Transforms/GlobalOpt/load-failed-to-evaluate-initializer.ll
@@ -0,0 +1,19 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
+; RUN: opt -passes=globalopt -S -debug %s 2>&1 | FileCheck %s
+
+; REQUIRES: asserts
+
+@g = internal global { double } { double 0x40E7606FBE1650A4 }
+
+; CHECK: Global SRA: Failed to evaluate initializer of @g =
+
+define [1 x double] @test () {
+; CHECK-LABEL: define [1 x double] @test() local_unnamed_addr {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[L:%.*]] = load [1 x double], ptr @g, align 8
+; CHECK-NEXT:    ret [1 x double] [[L]]
+;
+entry:
+  %l = load [1 x double], ptr @g, align 8
+  ret [1 x double] %l
+}
diff --git a/llvm/test/Transforms/GlobalOpt/sra-many-stores-initializers.ll b/llvm/test/Transforms/GlobalOpt/sra-many-stores-initializers.ll
new file mode 100644
index 0000000000000..9c85f3b342470
--- /dev/null
+++ b/llvm/test/Transforms/GlobalOpt/sra-many-stores-initializers.ll
@@ -0,0 +1,470 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --check-globals
+; RUN: opt -passes=globalopt -S %s | FileCheck %s
+
+%struct.20ptr = type { ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr }
+
+@global.20ptr = internal global %struct.20ptr zeroinitializer
+
+@c = global ptr null
+
+;.
+; CHECK: @[[C:[a-zA-Z0-9_$"\\.-]+]] = global ptr null
+; CHECK: @[[GLOBAL_20PTR_3:[a-zA-Z0-9_$"\\.-]+]] = internal unnamed_addr global [[STRUCT_20PTR:%.*]] { ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr null, ptr null, ptr null, ptr @c }
+; CHECK: @[[GLOBAL_20PTR_4_16:[a-zA-Z0-9_$"\\.-]+]] = internal unnamed_addr global ptr null
+;.
+define void @store_initializer() {
+; CHECK-LABEL: @store_initializer(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    ret void
+;
+entry:
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 0), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 1), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 2), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 3), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 4), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 5), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 6), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 7), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 8), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 9), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 10), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 11), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 12), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 13), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 14), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 15), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 16), align 8
+
+  %l0 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 0), align 8
+  store volatile ptr %l0, ptr @c
+  %l1 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 1), align 8
+  store volatile ptr %l1, ptr @c
+  %l2 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 2), align 8
+  store volatile ptr %l2, ptr @c
+  %l3 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 3), align 8
+  store volatile ptr %l3, ptr @c
+  %l4 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 4), align 8
+  store volatile ptr %l4, ptr @c
+  %l5 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 5), align 8
+  store volatile ptr %l5, ptr @c
+  %l6 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 6), align 8
+  store volatile ptr %l6, ptr @c
+  %l7 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 7), align 8
+  store volatile ptr %l7, ptr @c
+  %l8 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 8), align 8
+  store volatile ptr %l8, ptr @c
+  %l9 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 9), align 8
+  store volatile ptr %l9, ptr @c
+  %l10 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 10), align 8
+  store volatile ptr %l10, ptr @c
+  %l11 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 11), align 8
+  store volatile ptr %l11, ptr @c
+  %l12 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 12), align 8
+  store volatile ptr %l12, ptr @c
+  %l13 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 13), align 8
+  store volatile ptr %l13, ptr @c
+  %l14 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 14), align 8
+  store volatile ptr %l14, ptr @c
+  %l15 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 15), align 8
+  store volatile ptr %l15, ptr @c
+  %l16 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 16), align 8
+  store volatile ptr %l16, ptr @c
+
+  ret void
+}
+
+; Global with initializer where some elements in the initializer are not null.
+
+@global.20ptr.2 = internal global %struct.20ptr { ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null, ptr @c }
+
+define void @store_null_initializer_2() {
+; CHECK-LABEL: @store_null_initializer_2(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    ret void
+;
+entry:
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 0), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 1), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 2), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 3), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 4), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 5), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 6), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 7), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 8), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 9), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 10), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 11), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 12), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 13), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 14), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 15), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 16), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 19), align 8
+
+  %l0 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 0), align 8
+  store volatile ptr %l0, ptr @c
+  %l1 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 1), align 8
+  store volatile ptr %l1, ptr @c
+  %l2 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 2), align 8
+  store volatile ptr %l2, ptr @c
+  %l3 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 3), align 8
+  store volatile ptr %l3, ptr @c
+  %l4 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 4), align 8
+  store volatile ptr %l4, ptr @c
+  %l5 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 5), align 8
+  store volatile ptr %l5, ptr @c
+  %l6 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 6), align 8
+  store volatile ptr %l6, ptr @c
+  %l7 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 7), align 8
+  store volatile ptr %l7, ptr @c
+  %l8 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 8), align 8
+  store volatile ptr %l8, ptr @c
+  %l9 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 9), align 8
+  store volatile ptr %l9, ptr @c
+  %l10 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 10), align 8
+  store volatile ptr %l10, ptr @c
+  %l11 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 11), align 8
+  store volatile ptr %l11, ptr @c
+  %l12 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 12), align 8
+  store volatile ptr %l12, ptr @c
+  %l13 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 13), align 8
+  store volatile ptr %l13, ptr @c
+  %l14 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 14), align 8
+  store volatile ptr %l14, ptr @c
+  %l15 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 15), align 8
+  store volatile ptr %l15, ptr @c
+  %l16 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 16), align 8
+  store volatile ptr %l16, ptr @c
+  %l19 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.2, i64 0, i32 19), align 8
+  store volatile ptr %l19, ptr @c
+
+  ret void
+}
+
+
+@global.20ptr.3 = internal global %struct.20ptr { ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr null, ptr null, ptr null, ptr @c }
+
+; Stores writing a different value than the initializer.
+define void @store_mixed_initializer_negative() {
+; CHECK-LABEL: @store_mixed_initializer_negative(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    store ptr null, ptr @global.20ptr.3, align 8
+; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR:%.*]], ptr @global.20ptr.3, i64 0, i32 1), align 8
+; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 2), align 8
+; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 3), align 8
+; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 4), align 8
+; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 5), align 8
+; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 6), align 8
+; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 7), align 8
+; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 8), align 8
+; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 9), align 8
+; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 10), align 8
+; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 11), align 8
+; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 12), align 8
+; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 13), align 8
+; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 14), align 8
+; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 15), align 8
+; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 16), align 8
+; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 19), align 8
+; CHECK-NEXT:    [[L0:%.*]] = load ptr, ptr @global.20ptr.3, align 8
+; CHECK-NEXT:    store volatile ptr [[L0]], ptr @c, align 8
+; CHECK-NEXT:    [[L1:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 1), align 8
+; CHECK-NEXT:    store volatile ptr [[L1]], ptr @c, align 8
+; CHECK-NEXT:    [[L2:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 2), align 8
+; CHECK-NEXT:    store volatile ptr [[L2]], ptr @c, align 8
+; CHECK-NEXT:    [[L3:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 3), align 8
+; CHECK-NEXT:    store volatile ptr [[L3]], ptr @c, align 8
+; CHECK-NEXT:    [[L4:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 4), align 8
+; CHECK-NEXT:    store volatile ptr [[L4]], ptr @c, align 8
+; CHECK-NEXT:    [[L5:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 5), align 8
+; CHECK-NEXT:    store volatile ptr [[L5]], ptr @c, align 8
+; CHECK-NEXT:    [[L6:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 6), align 8
+; CHECK-NEXT:    store volatile ptr [[L6]], ptr @c, align 8
+; CHECK-NEXT:    [[L7:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 7), align 8
+; CHECK-NEXT:    store volatile ptr [[L7]], ptr @c, align 8
+; CHECK-NEXT:    [[L8:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 8), align 8
+; CHECK-NEXT:    store volatile ptr [[L8]], ptr @c, align 8
+; CHECK-NEXT:    [[L9:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 9), align 8
+; CHECK-NEXT:    store volatile ptr [[L9]], ptr @c, align 8
+; CHECK-NEXT:    [[L10:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 10), align 8
+; CHECK-NEXT:    store volatile ptr [[L10]], ptr @c, align 8
+; CHECK-NEXT:    [[L11:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 11), align 8
+; CHECK-NEXT:    store volatile ptr [[L11]], ptr @c, align 8
+; CHECK-NEXT:    [[L12:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 12), align 8
+; CHECK-NEXT:    store volatile ptr [[L12]], ptr @c, align 8
+; CHECK-NEXT:    [[L13:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 13), align 8
+; CHECK-NEXT:    store volatile ptr [[L13]], ptr @c, align 8
+; CHECK-NEXT:    [[L14:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 14), align 8
+; CHECK-NEXT:    store volatile ptr [[L14]], ptr @c, align 8
+; CHECK-NEXT:    [[L15:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 15), align 8
+; CHECK-NEXT:    store volatile ptr [[L15]], ptr @c, align 8
+; CHECK-NEXT:    [[L16:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 16), align 8
+; CHECK-NEXT:    store volatile ptr [[L16]], ptr @c, align 8
+; CHECK-NEXT:    [[L19:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr.3, i64 0, i32 19), align 8
+; CHECK-NEXT:    store volatile ptr [[L19]], ptr @c, align 8
+; CHECK-NEXT:    ret void
+;
+entry:
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 0), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 1), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 2), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 3), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 4), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 5), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 6), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 7), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 8), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 9), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 10), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 11), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 12), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 13), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 14), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 15), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 16), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 19), align 8
+
+  %l0 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 0), align 8
+  store volatile ptr %l0, ptr @c
+  %l1 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 1), align 8
+  store volatile ptr %l1, ptr @c
+  %l2 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 2), align 8
+  store volatile ptr %l2, ptr @c
+  %l3 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 3), align 8
+  store volatile ptr %l3, ptr @c
+  %l4 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 4), align 8
+  store volatile ptr %l4, ptr @c
+  %l5 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 5), align 8
+  store volatile ptr %l5, ptr @c
+  %l6 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 6), align 8
+  store volatile ptr %l6, ptr @c
+  %l7 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 7), align 8
+  store volatile ptr %l7, ptr @c
+  %l8 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 8), align 8
+  store volatile ptr %l8, ptr @c
+  %l9 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 9), align 8
+  store volatile ptr %l9, ptr @c
+  %l10 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 10), align 8
+  store volatile ptr %l10, ptr @c
+  %l11 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 11), align 8
+  store volatile ptr %l11, ptr @c
+  %l12 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 12), align 8
+  store volatile ptr %l12, ptr @c
+  %l13 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 13), align 8
+  store volatile ptr %l13, ptr @c
+  %l14 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 14), align 8
+  store volatile ptr %l14, ptr @c
+  %l15 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 15), align 8
+  store volatile ptr %l15, ptr @c
+  %l16 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 16), align 8
+  store volatile ptr %l16, ptr @c
+  %l19 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.3, i64 0, i32 19), align 8
+  store volatile ptr %l19, ptr @c
+
+  ret void
+}
+
+
+@global.20ptr.4 = internal global %struct.20ptr { ptr null, ptr null, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr null, ptr null, ptr null, ptr @c }
+
+; Stores writing a different value than the initializer.
+define void @store_mixed_initializer() {
+; CHECK-LABEL: @store_mixed_initializer(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    store ptr @c, ptr @global.20ptr.4.16, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    [[L16:%.*]] = load ptr, ptr @global.20ptr.4.16, align 8
+; CHECK-NEXT:    store volatile ptr [[L16]], ptr @c, align 8
+; CHECK-NEXT:    ret void
+;
+entry:
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 0), align 8
+  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 1), align 8
+  store ptr @c, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 2), align 8
+  store ptr @c, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 3), align 8
+  store ptr @c, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 4), align 8
+  store ptr @c, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 5), align 8
+  store ptr @c, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 6), align 8
+  store ptr @c, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 7), align 8
+  store ptr @c, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 8), align 8
+  store ptr @c, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 9), align 8
+  store ptr @c, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 10), align 8
+  store ptr @c, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 11), align 8
+  store ptr @c, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 12), align 8
+  store ptr @c, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 13), align 8
+  store ptr @c, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 14), align 8
+  store ptr @c, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 15), align 8
+  store ptr @c, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 16), align 8
+
+  %l0 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 0), align 8
+  store volatile ptr %l0, ptr @c
+  %l1 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 1), align 8
+  store volatile ptr %l1, ptr @c
+  %l2 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 2), align 8
+  store volatile ptr %l2, ptr @c
+  %l3 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 3), align 8
+  store volatile ptr %l3, ptr @c
+  %l4 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 4), align 8
+  store volatile ptr %l4, ptr @c
+  %l5 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 5), align 8
+  store volatile ptr %l5, ptr @c
+  %l6 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 6), align 8
+  store volatile ptr %l6, ptr @c
+  %l7 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 7), align 8
+  store volatile ptr %l7, ptr @c
+  %l8 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 8), align 8
+  store volatile ptr %l8, ptr @c
+  %l9 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 9), align 8
+  store volatile ptr %l9, ptr @c
+  %l10 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 10), align 8
+  store volatile ptr %l10, ptr @c
+  %l11 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 11), align 8
+  store volatile ptr %l11, ptr @c
+  %l12 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 12), align 8
+  store volatile ptr %l12, ptr @c
+  %l13 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 13), align 8
+  store volatile ptr %l13, ptr @c
+  %l14 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 14), align 8
+  store volatile ptr %l14, ptr @c
+  %l15 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 15), align 8
+  store volatile ptr %l15, ptr @c
+  %l16 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 16), align 8
+  store volatile ptr %l16, ptr @c
+
+  ret void
+}
+
+; Same as @store_mixed_initializer, but the GEP constant expressions are not
+; inbounds.
+define void @store_mixed_initializer_geps_without_inbounds() {
+; CHECK-LABEL: @store_mixed_initializer_geps_without_inbounds(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    store ptr @c, ptr @global.20ptr.4.16, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr null, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    store volatile ptr @c, ptr @c, align 8
+; CHECK-NEXT:    [[L16:%.*]] = load ptr, ptr @global.20ptr.4.16, align 8
+; CHECK-NEXT:    store volatile ptr [[L16]], ptr @c, align 8
+; CHECK-NEXT:    ret void
+;
+entry:
+  store ptr null, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 0), align 8
+  store ptr null, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 1), align 8
+  store ptr @c, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 2), align 8
+  store ptr @c, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 3), align 8
+  store ptr @c, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 4), align 8
+  store ptr @c, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 5), align 8
+  store ptr @c, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 6), align 8
+  store ptr @c, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 7), align 8
+  store ptr @c, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 8), align 8
+  store ptr @c, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 9), align 8
+  store ptr @c, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 10), align 8
+  store ptr @c, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 11), align 8
+  store ptr @c, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 12), align 8
+  store ptr @c, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 13), align 8
+  store ptr @c, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 14), align 8
+  store ptr @c, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 15), align 8
+  store ptr @c, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 16), align 8
+
+  %l0 = load ptr, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 0), align 8
+  store volatile ptr %l0, ptr @c
+  %l1 = load ptr, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 1), align 8
+  store volatile ptr %l1, ptr @c
+  %l2 = load ptr, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 2), align 8
+  store volatile ptr %l2, ptr @c
+  %l3 = load ptr, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 3), align 8
+  store volatile ptr %l3, ptr @c
+  %l4 = load ptr, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 4), align 8
+  store volatile ptr %l4, ptr @c
+  %l5 = load ptr, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 5), align 8
+  store volatile ptr %l5, ptr @c
+  %l6 = load ptr, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 6), align 8
+  store volatile ptr %l6, ptr @c
+  %l7 = load ptr, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 7), align 8
+  store volatile ptr %l7, ptr @c
+  %l8 = load ptr, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 8), align 8
+  store volatile ptr %l8, ptr @c
+  %l9 = load ptr, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 9), align 8
+  store volatile ptr %l9, ptr @c
+  %l10 = load ptr, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 10), align 8
+  store volatile ptr %l10, ptr @c
+  %l11 = load ptr, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 11), align 8
+  store volatile ptr %l11, ptr @c
+  %l12 = load ptr, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 12), align 8
+  store volatile ptr %l12, ptr @c
+  %l13 = load ptr, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 13), align 8
+  store volatile ptr %l13, ptr @c
+  %l14 = load ptr, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 14), align 8
+  store volatile ptr %l14, ptr @c
+  %l15 = load ptr, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 15), align 8
+  store volatile ptr %l15, ptr @c
+  %l16 = load ptr, ptr getelementptr (%struct.20ptr, ptr @global.20ptr.4, i64 0, i32 16), align 8
+  store volatile ptr %l16, ptr @c
+
+  ret void
+}
diff --git a/llvm/test/Transforms/GlobalOpt/sra-many-stores.ll b/llvm/test/Transforms/GlobalOpt/sra-many-stores.ll
index 689ef6d776205..900bcc2582f66 100644
--- a/llvm/test/Transforms/GlobalOpt/sra-many-stores.ll
+++ b/llvm/test/Transforms/GlobalOpt/sra-many-stores.ll
@@ -5,11 +5,8 @@
 
 @global = internal global %struct.widget zeroinitializer
 
-
 ;.
 ; CHECK: @[[A:[a-zA-Z0-9_$"\\.-]+]] = global i8 0, align 4
-; CHECK: @[[GLOBAL_20PTR:[a-zA-Z0-9_$"\\.-]+]] = internal unnamed_addr global [[STRUCT_20PTR:%.*]] zeroinitializer
-; CHECK: @[[C:[a-zA-Z0-9_$"\\.-]+]] = global ptr null
 ;.
 define internal void @read_struct() {
 ; CHECK-LABEL: @read_struct(
@@ -180,123 +177,3 @@ define void @test_single_write_to_global_b() {
   store volatile i8 %i19, ptr @a, align 4
   ret void
 }
-
-
-%struct.20ptr = type { ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr }
-
-@global.20ptr = internal global %struct.20ptr zeroinitializer
-
-@c = global ptr null
-
-define void @store_initializer() {
-; CHECK-LABEL: @store_initializer(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    store ptr null, ptr @global.20ptr, align 8
-; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR:%.*]], ptr @global.20ptr, i64 0, i32 1), align 8
-; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 2), align 8
-; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 3), align 8
-; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 4), align 8
-; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 5), align 8
-; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 6), align 8
-; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 7), align 8
-; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 8), align 8
-; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 9), align 8
-; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 10), align 8
-; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 11), align 8
-; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 12), align 8
-; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 13), align 8
-; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 14), align 8
-; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 15), align 8
-; CHECK-NEXT:    store ptr null, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 16), align 8
-; CHECK-NEXT:    [[L0:%.*]] = load ptr, ptr @global.20ptr, align 8
-; CHECK-NEXT:    store volatile ptr [[L0]], ptr @c, align 8
-; CHECK-NEXT:    [[L1:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 1), align 8
-; CHECK-NEXT:    store volatile ptr [[L1]], ptr @c, align 8
-; CHECK-NEXT:    [[L2:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 2), align 8
-; CHECK-NEXT:    store volatile ptr [[L2]], ptr @c, align 8
-; CHECK-NEXT:    [[L3:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 3), align 8
-; CHECK-NEXT:    store volatile ptr [[L3]], ptr @c, align 8
-; CHECK-NEXT:    [[L4:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 4), align 8
-; CHECK-NEXT:    store volatile ptr [[L4]], ptr @c, align 8
-; CHECK-NEXT:    [[L5:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 5), align 8
-; CHECK-NEXT:    store volatile ptr [[L5]], ptr @c, align 8
-; CHECK-NEXT:    [[L6:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 6), align 8
-; CHECK-NEXT:    store volatile ptr [[L6]], ptr @c, align 8
-; CHECK-NEXT:    [[L7:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 7), align 8
-; CHECK-NEXT:    store volatile ptr [[L7]], ptr @c, align 8
-; CHECK-NEXT:    [[L8:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 8), align 8
-; CHECK-NEXT:    store volatile ptr [[L8]], ptr @c, align 8
-; CHECK-NEXT:    [[L9:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 9), align 8
-; CHECK-NEXT:    store volatile ptr [[L9]], ptr @c, align 8
-; CHECK-NEXT:    [[L10:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 10), align 8
-; CHECK-NEXT:    store volatile ptr [[L10]], ptr @c, align 8
-; CHECK-NEXT:    [[L11:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 11), align 8
-; CHECK-NEXT:    store volatile ptr [[L11]], ptr @c, align 8
-; CHECK-NEXT:    [[L12:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 12), align 8
-; CHECK-NEXT:    store volatile ptr [[L12]], ptr @c, align 8
-; CHECK-NEXT:    [[L13:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 13), align 8
-; CHECK-NEXT:    store volatile ptr [[L13]], ptr @c, align 8
-; CHECK-NEXT:    [[L14:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 14), align 8
-; CHECK-NEXT:    store volatile ptr [[L14]], ptr @c, align 8
-; CHECK-NEXT:    [[L15:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 15), align 8
-; CHECK-NEXT:    store volatile ptr [[L15]], ptr @c, align 8
-; CHECK-NEXT:    [[L16:%.*]] = load ptr, ptr getelementptr inbounds ([[STRUCT_20PTR]], ptr @global.20ptr, i64 0, i32 16), align 8
-; CHECK-NEXT:    store volatile ptr [[L16]], ptr @c, align 8
-; CHECK-NEXT:    ret void
-;
-entry:
-  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 0), align 8
-  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 1), align 8
-  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 2), align 8
-  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 3), align 8
-  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 4), align 8
-  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 5), align 8
-  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 6), align 8
-  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 7), align 8
-  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 8), align 8
-  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 9), align 8
-  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 10), align 8
-  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 11), align 8
-  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 12), align 8
-  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 13), align 8
-  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 14), align 8
-  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 15), align 8
-  store ptr null, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 16), align 8
-
-  %l0 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 0), align 8
-  store volatile ptr %l0, ptr @c
-  %l1 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 1), align 8
-  store volatile ptr %l1, ptr @c
-  %l2 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 2), align 8
-  store volatile ptr %l2, ptr @c
-  %l3 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 3), align 8
-  store volatile ptr %l3, ptr @c
-  %l4 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 4), align 8
-  store volatile ptr %l4, ptr @c
-  %l5 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 5), align 8
-  store volatile ptr %l5, ptr @c
-  %l6 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 6), align 8
-  store volatile ptr %l6, ptr @c
-  %l7 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 7), align 8
-  store volatile ptr %l7, ptr @c
-  %l8 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 8), align 8
-  store volatile ptr %l8, ptr @c
-  %l9 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 9), align 8
-  store volatile ptr %l9, ptr @c
-  %l10 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 10), align 8
-  store volatile ptr %l10, ptr @c
-  %l11 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 11), align 8
-  store volatile ptr %l11, ptr @c
-  %l12 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 12), align 8
-  store volatile ptr %l12, ptr @c
-  %l13 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 13), align 8
-  store volatile ptr %l13, ptr @c
-  %l14 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 14), align 8
-  store volatile ptr %l14, ptr @c
-  %l15 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 15), align 8
-  store volatile ptr %l15, ptr @c
-  %l16 = load ptr, ptr getelementptr inbounds (%struct.20ptr, ptr @global.20ptr, i64 0, i32 16), align 8
-  store volatile ptr %l16, ptr @c
-
-  ret void
-}
diff --git a/llvm/test/Transforms/GuardWidening/two_forms_behavior_consistency.ll b/llvm/test/Transforms/GuardWidening/two_forms_behavior_consistency.ll
new file mode 100644
index 0000000000000..9b60f4e3e62b7
--- /dev/null
+++ b/llvm/test/Transforms/GuardWidening/two_forms_behavior_consistency.ll
@@ -0,0 +1,125 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
+; RUN: opt -S -passes="guard-widening,make-guards-explicit,simplifycfg" < %s        | FileCheck %s --check-prefixes=INTRINSIC_FORM
+; RUN: opt -S -passes="make-guards-explicit,guard-widening,simplifycfg" < %s        | FileCheck %s --check-prefixes=BRANCH_FORM
+; RUN: opt -S -passes="make-guards-explicit,loop-mssa(licm),guard-widening,simplifycfg" < %s        | FileCheck %s --check-prefixes=BRANCH_FORM_LICM
+
+declare i1 @cond() readonly
+
+; FIXME We want to make sure that guard widening works in the same way, no matter what form of
+;       guards it is dealing with.
+; We also want to make sure that LICM doesn't mess with widenable conditions, what might
+; make things more complex.
+
+define void @test_01(i32 %a, i32 %b, i32 %c, i32 %d) {
+; INTRINSIC_FORM-LABEL: define void @test_01
+; INTRINSIC_FORM-SAME: (i32 [[A:%.*]], i32 [[B:%.*]], i32 [[C:%.*]], i32 [[D:%.*]]) {
+; INTRINSIC_FORM-NEXT:  entry:
+; INTRINSIC_FORM-NEXT:    br label [[LOOP:%.*]]
+; INTRINSIC_FORM:       loop:
+; INTRINSIC_FORM-NEXT:    [[IV:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[GUARDED:%.*]] ]
+; INTRINSIC_FORM-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; INTRINSIC_FORM-NEXT:    [[C1:%.*]] = icmp ult i32 [[IV]], [[A]]
+; INTRINSIC_FORM-NEXT:    [[C2:%.*]] = icmp ult i32 [[IV]], [[B]]
+; INTRINSIC_FORM-NEXT:    [[WIDE_CHK:%.*]] = and i1 [[C1]], [[C2]]
+; INTRINSIC_FORM-NEXT:    [[C3:%.*]] = icmp ult i32 [[IV]], [[C]]
+; INTRINSIC_FORM-NEXT:    [[WIDE_CHK1:%.*]] = and i1 [[WIDE_CHK]], [[C3]]
+; INTRINSIC_FORM-NEXT:    [[C4:%.*]] = icmp ult i32 [[IV]], [[D]]
+; INTRINSIC_FORM-NEXT:    [[WIDE_CHK2:%.*]] = and i1 [[WIDE_CHK1]], [[C4]]
+; INTRINSIC_FORM-NEXT:    [[WIDENABLE_COND:%.*]] = call i1 @llvm.experimental.widenable.condition()
+; INTRINSIC_FORM-NEXT:    [[EXIPLICIT_GUARD_COND:%.*]] = and i1 [[WIDE_CHK2]], [[WIDENABLE_COND]]
+; INTRINSIC_FORM-NEXT:    br i1 [[EXIPLICIT_GUARD_COND]], label [[GUARDED]], label [[DEOPT:%.*]], !prof [[PROF0:![0-9]+]]
+; INTRINSIC_FORM:       deopt:
+; INTRINSIC_FORM-NEXT:    call void (...) @llvm.experimental.deoptimize.isVoid() [ "deopt"() ]
+; INTRINSIC_FORM-NEXT:    ret void
+; INTRINSIC_FORM:       guarded:
+; INTRINSIC_FORM-NEXT:    [[LOOP_COND:%.*]] = call i1 @cond()
+; INTRINSIC_FORM-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; INTRINSIC_FORM:       exit:
+; INTRINSIC_FORM-NEXT:    ret void
+;
+; BRANCH_FORM-LABEL: define void @test_01
+; BRANCH_FORM-SAME: (i32 [[A:%.*]], i32 [[B:%.*]], i32 [[C:%.*]], i32 [[D:%.*]]) {
+; BRANCH_FORM-NEXT:  entry:
+; BRANCH_FORM-NEXT:    br label [[LOOP:%.*]]
+; BRANCH_FORM:       loop:
+; BRANCH_FORM-NEXT:    [[IV:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[GUARDED:%.*]] ]
+; BRANCH_FORM-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; BRANCH_FORM-NEXT:    [[C1:%.*]] = icmp ult i32 [[IV]], [[A]]
+; BRANCH_FORM-NEXT:    [[C2:%.*]] = icmp ult i32 [[IV]], [[B]]
+; BRANCH_FORM-NEXT:    [[WIDE_CHK:%.*]] = and i1 [[C1]], [[C2]]
+; BRANCH_FORM-NEXT:    [[C3:%.*]] = icmp ult i32 [[IV]], [[C]]
+; BRANCH_FORM-NEXT:    [[WIDE_CHK13:%.*]] = and i1 [[WIDE_CHK]], [[C3]]
+; BRANCH_FORM-NEXT:    [[C4:%.*]] = icmp ult i32 [[IV]], [[D]]
+; BRANCH_FORM-NEXT:    [[WIDE_CHK14:%.*]] = and i1 [[WIDE_CHK13]], [[C4]]
+; BRANCH_FORM-NEXT:    [[WIDENABLE_COND:%.*]] = call i1 @llvm.experimental.widenable.condition()
+; BRANCH_FORM-NEXT:    [[EXIPLICIT_GUARD_COND:%.*]] = and i1 [[WIDE_CHK14]], [[WIDENABLE_COND]]
+; BRANCH_FORM-NEXT:    br i1 [[EXIPLICIT_GUARD_COND]], label [[GUARDED]], label [[DEOPT:%.*]], !prof [[PROF0:![0-9]+]]
+; BRANCH_FORM:       deopt:
+; BRANCH_FORM-NEXT:    call void (...) @llvm.experimental.deoptimize.isVoid() [ "deopt"() ]
+; BRANCH_FORM-NEXT:    ret void
+; BRANCH_FORM:       guarded:
+; BRANCH_FORM-NEXT:    [[WIDENABLE_COND3:%.*]] = call i1 @llvm.experimental.widenable.condition()
+; BRANCH_FORM-NEXT:    [[EXIPLICIT_GUARD_COND4:%.*]] = and i1 [[C2]], [[WIDENABLE_COND3]]
+; BRANCH_FORM-NEXT:    [[WIDENABLE_COND7:%.*]] = call i1 @llvm.experimental.widenable.condition()
+; BRANCH_FORM-NEXT:    [[EXIPLICIT_GUARD_COND8:%.*]] = and i1 [[C3]], [[WIDENABLE_COND7]]
+; BRANCH_FORM-NEXT:    [[WIDENABLE_COND11:%.*]] = call i1 @llvm.experimental.widenable.condition()
+; BRANCH_FORM-NEXT:    [[EXIPLICIT_GUARD_COND12:%.*]] = and i1 [[C4]], [[WIDENABLE_COND11]]
+; BRANCH_FORM-NEXT:    [[LOOP_COND:%.*]] = call i1 @cond()
+; BRANCH_FORM-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; BRANCH_FORM:       exit:
+; BRANCH_FORM-NEXT:    ret void
+;
+; BRANCH_FORM_LICM-LABEL: define void @test_01
+; BRANCH_FORM_LICM-SAME: (i32 [[A:%.*]], i32 [[B:%.*]], i32 [[C:%.*]], i32 [[D:%.*]]) {
+; BRANCH_FORM_LICM-NEXT:  entry:
+; BRANCH_FORM_LICM-NEXT:    br label [[LOOP:%.*]]
+; BRANCH_FORM_LICM:       loop:
+; BRANCH_FORM_LICM-NEXT:    [[IV:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[GUARDED:%.*]] ]
+; BRANCH_FORM_LICM-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; BRANCH_FORM_LICM-NEXT:    [[C1:%.*]] = icmp ult i32 [[IV]], [[A]]
+; BRANCH_FORM_LICM-NEXT:    [[C2:%.*]] = icmp ult i32 [[IV]], [[B]]
+; BRANCH_FORM_LICM-NEXT:    [[WIDE_CHK:%.*]] = and i1 [[C1]], [[C2]]
+; BRANCH_FORM_LICM-NEXT:    [[C3:%.*]] = icmp ult i32 [[IV]], [[C]]
+; BRANCH_FORM_LICM-NEXT:    [[WIDE_CHK13:%.*]] = and i1 [[WIDE_CHK]], [[C3]]
+; BRANCH_FORM_LICM-NEXT:    [[C4:%.*]] = icmp ult i32 [[IV]], [[D]]
+; BRANCH_FORM_LICM-NEXT:    [[WIDE_CHK14:%.*]] = and i1 [[WIDE_CHK13]], [[C4]]
+; BRANCH_FORM_LICM-NEXT:    [[WIDENABLE_COND:%.*]] = call i1 @llvm.experimental.widenable.condition()
+; BRANCH_FORM_LICM-NEXT:    [[EXIPLICIT_GUARD_COND:%.*]] = and i1 [[WIDE_CHK14]], [[WIDENABLE_COND]]
+; BRANCH_FORM_LICM-NEXT:    br i1 [[EXIPLICIT_GUARD_COND]], label [[GUARDED]], label [[DEOPT:%.*]], !prof [[PROF0:![0-9]+]]
+; BRANCH_FORM_LICM:       deopt:
+; BRANCH_FORM_LICM-NEXT:    call void (...) @llvm.experimental.deoptimize.isVoid() [ "deopt"() ]
+; BRANCH_FORM_LICM-NEXT:    ret void
+; BRANCH_FORM_LICM:       guarded:
+; BRANCH_FORM_LICM-NEXT:    [[WIDENABLE_COND3:%.*]] = call i1 @llvm.experimental.widenable.condition()
+; BRANCH_FORM_LICM-NEXT:    [[EXIPLICIT_GUARD_COND4:%.*]] = and i1 [[C2]], [[WIDENABLE_COND3]]
+; BRANCH_FORM_LICM-NEXT:    [[WIDENABLE_COND7:%.*]] = call i1 @llvm.experimental.widenable.condition()
+; BRANCH_FORM_LICM-NEXT:    [[EXIPLICIT_GUARD_COND8:%.*]] = and i1 [[C3]], [[WIDENABLE_COND7]]
+; BRANCH_FORM_LICM-NEXT:    [[WIDENABLE_COND11:%.*]] = call i1 @llvm.experimental.widenable.condition()
+; BRANCH_FORM_LICM-NEXT:    [[EXIPLICIT_GUARD_COND12:%.*]] = and i1 [[C4]], [[WIDENABLE_COND11]]
+; BRANCH_FORM_LICM-NEXT:    [[LOOP_COND:%.*]] = call i1 @cond()
+; BRANCH_FORM_LICM-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; BRANCH_FORM_LICM:       exit:
+; BRANCH_FORM_LICM-NEXT:    ret void
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [0, %entry], [%iv.next, %loop]
+  %iv.next = add i32 %iv, 1
+  %c1 = icmp ult i32 %iv, %a
+  call void(i1, ...) @llvm.experimental.guard(i1 %c1) [ "deopt"() ]
+  %c2 = icmp ult i32 %iv, %b
+  call void(i1, ...) @llvm.experimental.guard(i1 %c2) [ "deopt"() ]
+  %c3 = icmp ult i32 %iv, %c
+  call void(i1, ...) @llvm.experimental.guard(i1 %c3) [ "deopt"() ]
+  %c4 = icmp ult i32 %iv, %d
+  call void(i1, ...) @llvm.experimental.guard(i1 %c4) [ "deopt"() ]
+  %loop_cond = call i1 @cond()
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret void
+}
+
+declare void @llvm.experimental.guard(i1, ...)
diff --git a/llvm/test/Transforms/IndVarSimplify/lftr-reuse.ll b/llvm/test/Transforms/IndVarSimplify/lftr-reuse.ll
index 4a9f6a10a0f49..8aa698a4cb51d 100644
--- a/llvm/test/Transforms/IndVarSimplify/lftr-reuse.ll
+++ b/llvm/test/Transforms/IndVarSimplify/lftr-reuse.ll
@@ -126,6 +126,8 @@ exit:
 
 ; Force SCEVExpander to look for an existing well-formed phi.
 ; Perform LFTR without generating extra preheader code.
+; TODO: Recover regression after not strengthening AddRec flags during
+; get[Sign,Zero]ExtendExpr for INDVARS_IV_NEXT.
 define void @guardedloop(ptr %matrix, ptr %vector,
 ;
 ; CHECK-LABEL: @guardedloop(
@@ -146,7 +148,7 @@ define void @guardedloop(ptr %matrix, ptr %vector,
 ; CHECK-NEXT:    [[VECTORP:%.*]] = getelementptr inbounds [0 x double], ptr [[VECTOR:%.*]], i32 0, i64 [[INDVARS_IV2]]
 ; CHECK-NEXT:    [[V2:%.*]] = load double, ptr [[VECTORP]], align 8
 ; CHECK-NEXT:    call void @use(double [[V2]])
-; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nsw i64 [[INDVARS_IV]], [[TMP0]]
+; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add i64 [[INDVARS_IV]], [[TMP0]]
 ; CHECK-NEXT:    [[INDVARS_IV_NEXT3]] = add nuw nsw i64 [[INDVARS_IV2]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp ne i64 [[INDVARS_IV_NEXT3]], [[WIDE_TRIP_COUNT]]
 ; CHECK-NEXT:    br i1 [[EXITCOND]], label [[LOOP]], label [[RETURN_LOOPEXIT:%.*]]
diff --git a/llvm/test/Transforms/IndVarSimplify/pr60944.ll b/llvm/test/Transforms/IndVarSimplify/pr60944.ll
new file mode 100644
index 0000000000000..bb5b9044ac314
--- /dev/null
+++ b/llvm/test/Transforms/IndVarSimplify/pr60944.ll
@@ -0,0 +1,55 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
+; RUN: opt -S -passes=indvars < %s | FileCheck %s
+
+@x = global i32 0
+
+; %loop2 is never entered and we cannot derive any fact about %iv from it.
+define i32 @main(i32 %iv.start, i32 %arg2) mustprogress {
+; CHECK-LABEL: define i32 @main
+; CHECK-SAME: (i32 [[IV_START:%.*]], i32 [[ARG2:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[IV_NEXT:%.*]], [[LOOP_LATCH:%.*]] ], [ [[IV_START]], [[ENTRY:%.*]] ]
+; CHECK-NEXT:    br i1 false, label [[LOOP2_PREHEADER:%.*]], label [[LOOP_LATCH]]
+; CHECK:       loop2.preheader:
+; CHECK-NEXT:    br label [[LOOP2:%.*]]
+; CHECK:       loop2:
+; CHECK-NEXT:    br i1 true, label [[LOOP2_EXIT:%.*]], label [[LOOP2]]
+; CHECK:       loop2.exit:
+; CHECK-NEXT:    br label [[LOOP_LATCH]]
+; CHECK:       loop.latch:
+; CHECK-NEXT:    [[IV_NEXT]] = sdiv i32 [[ARG2]], [[IV]]
+; CHECK-NEXT:    [[CMP2:%.*]] = icmp eq i32 [[IV]], -1
+; CHECK-NEXT:    br i1 [[CMP2]], label [[LOOP_EXIT:%.*]], label [[LOOP]]
+; CHECK:       loop.exit:
+; CHECK-NEXT:    [[IV_NEXT_LCSSA:%.*]] = phi i32 [ [[IV_NEXT]], [[LOOP_LATCH]] ]
+; CHECK-NEXT:    ret i32 [[IV_NEXT_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [ %iv.next, %loop.latch ], [ %iv.start, %entry ]
+  br i1 false, label %loop2.preheader, label %loop.latch
+
+loop2.preheader:
+  br label %loop2
+
+loop2:
+  %x = load i32, ptr @x, align 4
+  %cmp1 = icmp ult i32 %iv, %x
+  br i1 %cmp1, label %loop2.exit, label %loop2
+
+loop2.exit:
+  br label %loop.latch
+
+loop.latch:
+  %iv.next = sdiv i32 %arg2, %iv
+  %cmp2 = icmp eq i32 %iv, -1
+  br i1 %cmp2, label %loop.exit, label %loop
+
+loop.exit:
+  %iv.next.lcssa = phi i32 [ %iv.next, %loop.latch ]
+  ret i32 %iv.next.lcssa
+}
diff --git a/llvm/test/Transforms/IndVarSimplify/range-iter-threshold.ll b/llvm/test/Transforms/IndVarSimplify/range-iter-threshold.ll
index 294e9a24d27c4..9c442710e2191 100644
--- a/llvm/test/Transforms/IndVarSimplify/range-iter-threshold.ll
+++ b/llvm/test/Transforms/IndVarSimplify/range-iter-threshold.ll
@@ -1,37 +1,35 @@
-; RUN: opt -passes=indvars -S %s | FileCheck --check-prefix=COMMON --check-prefix=DEFAULT %s
-; RUN: opt -passes=indvars -scev-range-iter-threshold=1 -S %s | FileCheck --check-prefix=COMMON --check-prefix=LIMIT %s
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
+; RUN: opt -passes=indvars -S %s | FileCheck --check-prefix=COMMON %s
+; RUN: opt -passes=indvars -scev-range-iter-threshold=1 -S %s | FileCheck --check-prefix=COMMON %s
 
 target datalayout = "e-m:o-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
 
 define i32 @test(i1 %c.0, i32 %m) {
-; COMMON-LABEL: @test(
+; COMMON-LABEL: define i32 @test
+; COMMON-SAME: (i1 [[C_0:%.*]], i32 [[M:%.*]]) {
 ; COMMON-NEXT:  entry:
 ; COMMON-NEXT:    br label [[OUTER_HEADER:%.*]]
 ; COMMON:       outer.header:
-; COMMON-NEXT:    [[INDVARS_IV1:%.*]] = phi i64 [ [[INDVARS_IV_NEXT2:%.*]], [[OUTER_LATCH:%.*]] ], [ 0, [[ENTRY:%.*]] ]
-; DEFAULT-NEXT:    [[INDVARS_IV:%.*]] = phi i32 [ [[INDVARS_IV_NEXT:%.*]], [[OUTER_LATCH]] ], [ 2, [[ENTRY]] ]
-; COMMON-NEXT:    [[IV_1:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[IV_1_NEXT:%.*]], [[OUTER_LATCH]] ]
+; COMMON-NEXT:    [[IV_1:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[IV_1_NEXT:%.*]], [[OUTER_LATCH:%.*]] ]
 ; COMMON-NEXT:    [[MAX_0:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[MAX_1:%.*]], [[OUTER_LATCH]] ]
+; COMMON-NEXT:    [[TMP0:%.*]] = sext i32 [[IV_1]] to i64
 ; COMMON-NEXT:    br label [[INNER_1:%.*]]
 ; COMMON:       inner.1:
-; COMMON-NEXT:    [[C_1:%.*]] = icmp ult i64 0, [[INDVARS_IV1]]
+; COMMON-NEXT:    [[C_1:%.*]] = icmp slt i64 0, [[TMP0]]
 ; COMMON-NEXT:    br i1 [[C_1]], label [[INNER_1]], label [[INNER_2_HEADER_PREHEADER:%.*]]
 ; COMMON:       inner.2.header.preheader:
 ; COMMON-NEXT:    br label [[INNER_2_HEADER:%.*]]
 ; COMMON:       inner.2.header:
 ; COMMON-NEXT:    [[IV_3:%.*]] = phi i32 [ [[IV_3_NEXT:%.*]], [[INNER_2_LATCH:%.*]] ], [ 0, [[INNER_2_HEADER_PREHEADER]] ]
-; COMMON-NEXT:    br i1 [[C_0:%.*]], label [[OUTER_LATCH]], label [[INNER_2_LATCH]]
+; COMMON-NEXT:    br i1 [[C_0]], label [[OUTER_LATCH]], label [[INNER_2_LATCH]]
 ; COMMON:       inner.2.latch:
 ; COMMON-NEXT:    [[IV_3_NEXT]] = add i32 [[IV_3]], 1
-; DEFAULT-NEXT:   [[EXITCOND:%.*]] = icmp eq i32 [[IV_3_NEXT]], [[INDVARS_IV]]
-; LIMIT-NEXT:     [[EXITCOND:%.*]] = icmp ugt i32 [[IV_3]], [[IV_1]]
-; COMMON-NEXT:    br i1 [[EXITCOND]], label [[OUTER_LATCH]], label [[INNER_2_HEADER]]
+; COMMON-NEXT:    [[C_2:%.*]] = icmp ugt i32 [[IV_3]], [[IV_1]]
+; COMMON-NEXT:    br i1 [[C_2]], label [[OUTER_LATCH]], label [[INNER_2_HEADER]]
 ; COMMON:       outer.latch:
-; COMMON-NEXT:    [[MAX_1]] = phi i32 [ [[M:%.*]], [[INNER_2_LATCH]] ], [ 0, [[INNER_2_HEADER]] ]
-; COMMON-NEXT:    [[INDVARS_IV_NEXT2]] = add nuw nsw i64 [[INDVARS_IV1]], 1
-; COMMON-NEXT:    [[IV_1_NEXT]] = add nuw i32 [[IV_1]], 1
+; COMMON-NEXT:    [[MAX_1]] = phi i32 [ [[M]], [[INNER_2_LATCH]] ], [ 0, [[INNER_2_HEADER]] ]
+; COMMON-NEXT:    [[IV_1_NEXT]] = add i32 [[IV_1]], 1
 ; COMMON-NEXT:    [[C_3:%.*]] = icmp ugt i32 [[IV_1]], [[MAX_0]]
-; DEFAULT-NEXT:   [[INDVARS_IV_NEXT]] = add i32 [[INDVARS_IV]], 1
 ; COMMON-NEXT:    br i1 [[C_3]], label [[EXIT:%.*]], label [[OUTER_HEADER]], !llvm.loop [[LOOP0:![0-9]+]]
 ; COMMON:       exit:
 ; COMMON-NEXT:    ret i32 0
diff --git a/llvm/test/Transforms/IndVarSimplify/rewrite-loop-exit-value.ll b/llvm/test/Transforms/IndVarSimplify/rewrite-loop-exit-value.ll
index 23412ed978d29..0d3d2425806ff 100644
--- a/llvm/test/Transforms/IndVarSimplify/rewrite-loop-exit-value.ll
+++ b/llvm/test/Transforms/IndVarSimplify/rewrite-loop-exit-value.ll
@@ -169,9 +169,9 @@ define i16 @pr57336(i16 %end, i16 %m) mustprogress {
 ; CHECK-NEXT:    [[CMP_NOT:%.*]] = icmp sgt i16 [[MUL]], [[END:%.*]]
 ; CHECK-NEXT:    br i1 [[CMP_NOT]], label [[CRIT_EDGE:%.*]], label [[FOR_BODY]]
 ; CHECK:       crit_edge:
-; CHECK-NEXT:    [[TMP0:%.*]] = call i16 @llvm.smax.i16(i16 [[END]], i16 -1)
-; CHECK-NEXT:    [[SMAX:%.*]] = add nsw i16 [[TMP0]], 1
-; CHECK-NEXT:    [[TMP1:%.*]] = icmp ne i16 [[SMAX]], 0
+; CHECK-NEXT:    [[TMP0:%.*]] = add i16 [[END]], 1
+; CHECK-NEXT:    [[SMAX:%.*]] = call i16 @llvm.smax.i16(i16 [[TMP0]], i16 0)
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp ult i16 [[END]], 32767
 ; CHECK-NEXT:    [[UMIN:%.*]] = zext i1 [[TMP1]] to i16
 ; CHECK-NEXT:    [[TMP2:%.*]] = sub nsw i16 [[SMAX]], [[UMIN]]
 ; CHECK-NEXT:    [[UMAX:%.*]] = call i16 @llvm.umax.i16(i16 [[M]], i16 1)
diff --git a/llvm/test/Transforms/InferAddressSpaces/AMDGPU/infer-address-space.ll b/llvm/test/Transforms/InferAddressSpaces/AMDGPU/infer-address-space.ll
index 57935cf7df141..72109d0cff437 100644
--- a/llvm/test/Transforms/InferAddressSpaces/AMDGPU/infer-address-space.ll
+++ b/llvm/test/Transforms/InferAddressSpaces/AMDGPU/infer-address-space.ll
@@ -166,10 +166,12 @@ exit:                                             ; preds = %loop
 }
 
 ; CHECK-LABEL: @select_bug(
-; CHECK: %add.ptr157 = getelementptr inbounds i64, ptr undef, i64 select (i1 icmp ne (ptr inttoptr (i64 4873 to ptr), ptr null), i64 73, i64 93)
+; CHECK: %sel = select i1 icmp ne (ptr inttoptr (i64 4873 to ptr), ptr null), i64 73, i64 93
+; CHECK: %add.ptr157 = getelementptr inbounds i64, ptr undef, i64 %sel
 ; CHECK: %cmp169 = icmp uge ptr undef, %add.ptr157
 define void @select_bug() #0 {
-  %add.ptr157 = getelementptr inbounds i64, ptr undef, i64 select (i1 icmp ne (ptr inttoptr (i64 4873 to ptr), ptr null), i64 73, i64 93)
+  %sel = select i1 icmp ne (ptr inttoptr (i64 4873 to ptr), ptr null), i64 73, i64 93
+  %add.ptr157 = getelementptr inbounds i64, ptr undef, i64 %sel
   %cmp169 = icmp uge ptr undef, %add.ptr157
   unreachable
 }
diff --git a/llvm/test/Transforms/InferAddressSpaces/AMDGPU/select.ll b/llvm/test/Transforms/InferAddressSpaces/AMDGPU/select.ll
index 5562843a7dbf2..9495c5566b36c 100644
--- a/llvm/test/Transforms/InferAddressSpaces/AMDGPU/select.ll
+++ b/llvm/test/Transforms/InferAddressSpaces/AMDGPU/select.ll
@@ -54,22 +54,6 @@ define amdgpu_kernel void @store_select_mismatch_group_private_flat(i1 %c, ptr a
 @lds0 = internal addrspace(3) global i32 123, align 4
 @lds1 = internal addrspace(3) global i32 456, align 4
 
-; CHECK-LABEL: @constexpr_select_group_flat(
-; CHECK: %tmp = load i32, ptr addrspace(3) select (i1 icmp eq (i32 ptrtoint (ptr addrspace(3) @lds1 to i32), i32 4), ptr addrspace(3) @lds0, ptr addrspace(3) @lds1)
-define i32 @constexpr_select_group_flat() #0 {
-bb:
-  %tmp = load i32, ptr select (i1 icmp eq (i32 ptrtoint (ptr addrspace(3) @lds1 to i32), i32 4), ptr addrspacecast (ptr addrspace(3) @lds0 to ptr), ptr addrspacecast (ptr addrspace(3) @lds1 to ptr))
-  ret i32 %tmp
-}
-
-; CHECK-LABEL: @constexpr_select_group_global_flat_mismatch(
-; CHECK: %tmp = load i32, ptr select (i1 icmp eq (i32 ptrtoint (ptr addrspace(3) @lds1 to i32), i32 4), ptr addrspacecast (ptr addrspace(3) @lds0 to ptr), ptr addrspacecast (ptr addrspace(1) @global0 to ptr))
-define i32 @constexpr_select_group_global_flat_mismatch() #0 {
-bb:
-  %tmp = load i32, ptr select (i1 icmp eq (i32 ptrtoint (ptr addrspace(3) @lds1 to i32), i32 4), ptr addrspacecast (ptr addrspace(3) @lds0 to ptr), ptr addrspacecast (ptr addrspace(1) @global0 to ptr))
-  ret i32 %tmp
-}
-
 ; CHECK-LABEL: @store_select_group_flat_null(
 ; CHECK: %select = select i1 %c, ptr addrspace(3) %group.ptr.0, ptr addrspace(3) addrspacecast (ptr null to ptr addrspace(3))
 ; CHECK: store i32 -1, ptr addrspace(3) %select
@@ -185,48 +169,6 @@ define amdgpu_kernel void @store_select_group_global_mismatch_null_null(i1 %c) #
   ret void
 }
 
-; CHECK-LABEL: @store_select_group_global_mismatch_null_null_constexpr(
-; CHECK: store i32 7, ptr select (i1 icmp eq (i32 ptrtoint (ptr addrspace(3) @lds1 to i32), i32 4), ptr addrspacecast (ptr addrspace(3) null to ptr), ptr addrspacecast (ptr addrspace(1) null to ptr)), align 4
-define amdgpu_kernel void @store_select_group_global_mismatch_null_null_constexpr() #0 {
-  store i32 7, ptr select (i1 icmp eq (i32 ptrtoint (ptr addrspace(3) @lds1 to i32), i32 4), ptr addrspacecast (ptr addrspace(3) null to ptr), ptr addrspacecast (ptr addrspace(1) null to ptr)), align 4
-  ret void
-}
-
-; CHECK-LABEL: @store_select_group_global_mismatch_gv_null_constexpr(
-; CHECK: store i32 7, ptr select (i1 icmp eq (i32 ptrtoint (ptr addrspace(3) @lds1 to i32), i32 4), ptr addrspacecast (ptr addrspace(3) @lds0 to ptr), ptr addrspacecast (ptr addrspace(1) null to ptr)), align 4
-define amdgpu_kernel void @store_select_group_global_mismatch_gv_null_constexpr() #0 {
-  store i32 7, ptr select (i1 icmp eq (i32 ptrtoint (ptr addrspace(3) @lds1 to i32), i32 4), ptr addrspacecast (ptr addrspace(3) @lds0 to ptr), ptr addrspacecast (ptr addrspace(1) null to ptr)), align 4
-  ret void
-}
-
-; CHECK-LABEL: @store_select_group_global_mismatch_null_gv_constexpr(
-; CHECK: store i32 7, ptr select (i1 icmp eq (i32 ptrtoint (ptr addrspace(3) @lds1 to i32), i32 4), ptr addrspacecast (ptr addrspace(3) null to ptr), ptr addrspacecast (ptr addrspace(1) @global0 to ptr)), align 4
-define amdgpu_kernel void @store_select_group_global_mismatch_null_gv_constexpr() #0 {
-  store i32 7, ptr select (i1 icmp eq (i32 ptrtoint (ptr addrspace(3) @lds1 to i32), i32 4), ptr addrspacecast (ptr addrspace(3) null to ptr), ptr addrspacecast (ptr addrspace(1) @global0 to ptr)), align 4
-  ret void
-}
-
-; CHECK-LABEL: @store_select_group_global_mismatch_inttoptr_null_constexpr(
-; CHECK: store i32 7, ptr select (i1 icmp eq (i32 ptrtoint (ptr addrspace(3) @lds1 to i32), i32 4), ptr addrspacecast (ptr addrspace(3) inttoptr (i64 123 to ptr addrspace(3)) to ptr), ptr addrspacecast (ptr addrspace(1) null to ptr)), align 4
-define amdgpu_kernel void @store_select_group_global_mismatch_inttoptr_null_constexpr() #0 {
-  store i32 7, ptr select (i1 icmp eq (i32 ptrtoint (ptr addrspace(3) @lds1 to i32), i32 4), ptr addrspacecast (ptr addrspace(3) inttoptr (i64 123 to ptr addrspace(3)) to ptr), ptr addrspacecast (ptr addrspace(1) null to ptr)), align 4
-  ret void
-}
-
-; CHECK-LABEL: @store_select_group_global_mismatch_inttoptr_flat_null_constexpr(
-; CHECK: store i32 7, ptr addrspace(1) select (i1 icmp eq (i32 ptrtoint (ptr addrspace(3) @lds1 to i32), i32 4), ptr addrspace(1) addrspacecast (ptr inttoptr (i64 123 to ptr) to ptr addrspace(1)), ptr addrspace(1) null), align 4
-define amdgpu_kernel void @store_select_group_global_mismatch_inttoptr_flat_null_constexpr() #0 {
-  store i32 7, ptr select (i1 icmp eq (i32 ptrtoint (ptr addrspace(3) @lds1 to i32), i32 4), ptr inttoptr (i64 123 to ptr), ptr addrspacecast (ptr addrspace(1) null to ptr)), align 4
-  ret void
-}
-
-; CHECK-LABEL: @store_select_group_global_mismatch_undef_undef_constexpr(
-; CHECK: store i32 7, ptr select (i1 icmp eq (i32 ptrtoint (ptr addrspace(3) @lds1 to i32), i32 4), ptr addrspacecast (ptr addrspace(3) null to ptr), ptr undef), align 4
-define amdgpu_kernel void @store_select_group_global_mismatch_undef_undef_constexpr() #0 {
-  store i32 7, ptr select (i1 icmp eq (i32 ptrtoint (ptr addrspace(3) @lds1 to i32), i32 4), ptr addrspacecast (ptr addrspace(3) null to ptr), ptr addrspacecast (ptr addrspace(1) undef to ptr)), align 4
-  ret void
-}
-
 @lds2 = external addrspace(3) global [1024 x i32], align 4
 
 ; CHECK-LABEL: @store_select_group_constexpr_ptrtoint(
diff --git a/llvm/test/Transforms/Inline/byval-dbg.ll b/llvm/test/Transforms/Inline/byval-dbg.ll
new file mode 100644
index 0000000000000..51dc686664edc
--- /dev/null
+++ b/llvm/test/Transforms/Inline/byval-dbg.ll
@@ -0,0 +1,49 @@
+; RUN: opt < %s -passes=inline -S | FileCheck %s
+
+; Check that the memcpy generated by byval lowering is decorated by debuginfo.
+
+target datalayout = "p:32:32-p1:64:64-p2:16:16-n16:32:64"
+
+%struct.ss = type { i32, i64 }
+
+define internal void @f(ptr byval(%struct.ss)  %b) nounwind !dbg !4 {
+entry:
+	%tmp = getelementptr %struct.ss, ptr %b, i32 0, i32 0		; <ptr> [#uses=2]
+	%tmp1 = load i32, ptr %tmp, align 4		; <i32> [#uses=1]
+	%tmp2 = add i32 %tmp1, 1		; <i32> [#uses=1]
+	store i32 %tmp2, ptr %tmp, align 4
+	ret void
+}
+
+define i32 @test1() nounwind !dbg !10 {
+entry:
+	%S = alloca %struct.ss		; <ptr> [#uses=4]
+	%tmp1 = getelementptr %struct.ss, ptr %S, i32 0, i32 0		; <ptr> [#uses=1]
+	store i32 1, ptr %tmp1, align 8
+	%tmp4 = getelementptr %struct.ss, ptr %S, i32 0, i32 1		; <ptr> [#uses=1]
+	store i64 2, ptr %tmp4, align 4
+	call void @f(ptr byval(%struct.ss) %S) nounwind, !dbg !14
+	ret i32 0
+; CHECK: @test1()
+; CHECK: call void @llvm.memcpy{{.*}} !dbg
+; CHECK: ret i32 0
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!2, !3}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!1 = !DIFile(filename: "/app/example.c", directory: "/app")
+!2 = !{i32 7, !"Dwarf Version", i32 4}
+!3 = !{i32 2, !"Debug Info Version", i32 3}
+!4 = distinct !DISubprogram(name: "f", scope: !5, file: !5, line: 1, type: !6, scopeLine: 1, flags: DIFlagPrototyped | DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !9)
+!5 = !DIFile(filename: "example.c", directory: "/app")
+!6 = !DISubroutineType(types: !7)
+!7 = !{null, !8}
+!8 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: null, size: 64)
+!9 = !{}
+!10 = distinct !DISubprogram(name: "test1", scope: !5, file: !5, line: 4, type: !11, scopeLine: 4, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !9)
+!11 = !DISubroutineType(types: !12)
+!12 = !{!13}
+!13 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!14 = !DILocation(line: 6, column: 1, scope: !10)
diff --git a/llvm/test/Transforms/InstCombine/2010-03-03-ExtElim.ll b/llvm/test/Transforms/InstCombine/2010-03-03-ExtElim.ll
index c9b2d6b41aa9c..73fa613d0568c 100644
--- a/llvm/test/Transforms/InstCombine/2010-03-03-ExtElim.ll
+++ b/llvm/test/Transforms/InstCombine/2010-03-03-ExtElim.ll
@@ -21,12 +21,19 @@ define i1 @PR6486() nounwind {
 
 define i1 @PR16462_1() nounwind {
 ; CHECK-LABEL: @PR16462_1(
-  ret i1 icmp sgt (i32 sext (i16 trunc (i32 select (i1 icmp eq (ptr @a, ptr @d), i32 0, i32 1) to i16) to i32), i32 65535)
+  %constexpr = select i1 icmp eq (ptr @a, ptr @d), i32 0, i32 1
+  %constexpr1 = trunc i32 %constexpr to i16
+  %constexpr2 = sext i16 %constexpr1 to i32
+  %constexpr3 = icmp sgt i32 %constexpr2, 65535
+  ret i1 %constexpr3
 ; CHECK: ret i1 false
 }
 
 define i1 @PR16462_2() nounwind {
 ; CHECK-LABEL: @PR16462_2(
-  ret i1 icmp sgt (i32 sext (i16 trunc (i32 select (i1 icmp eq (ptr @a, ptr @d), i32 0, i32 1) to i16) to i32), i32 42)
+  %constexpr = select i1 icmp eq (ptr @a, ptr @d), i32 0, i32 1
+  %constexpr1 = trunc i32 %constexpr to i16
+  %constexpr2 = icmp sgt i16 %constexpr1, 42
+  ret i1 %constexpr2
 ; CHECK: ret i1 false
 }
diff --git a/llvm/test/Transforms/InstCombine/AArch64/sve-intrinsic-muladdsub.ll b/llvm/test/Transforms/InstCombine/AArch64/sve-intrinsic-muladdsub.ll
index 00213f8bb17b0..18d3f0a6891c5 100644
--- a/llvm/test/Transforms/InstCombine/AArch64/sve-intrinsic-muladdsub.ll
+++ b/llvm/test/Transforms/InstCombine/AArch64/sve-intrinsic-muladdsub.ll
@@ -15,6 +15,18 @@ define <vscale x 8 x half> @combine_fmla(<vscale x 16 x i1> %p, <vscale x 8 x ha
   ret <vscale x 8 x half> %3
 }
 
+define <vscale x 8 x half> @combine_fmla_u(<vscale x 16 x i1> %p, <vscale x 8 x half> %a, <vscale x 8 x half> %b, <vscale x 8 x half> %c) #0 {
+; CHECK-LABEL: @combine_fmla_u(
+; CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[P:%.*]])
+; CHECK-NEXT:    [[TMP2:%.*]] = call fast <vscale x 8 x half> @llvm.aarch64.sve.fmla.u.nxv8f16(<vscale x 8 x i1> [[TMP1]], <vscale x 8 x half> [[C:%.*]], <vscale x 8 x half> [[A:%.*]], <vscale x 8 x half> [[B:%.*]])
+; CHECK-NEXT:    ret <vscale x 8 x half> [[TMP2]]
+;
+  %1 = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> %p)
+  %2 = tail call fast <vscale x 8 x half> @llvm.aarch64.sve.fmul.u.nxv8f16(<vscale x 8 x i1> %1, <vscale x 8 x half> %a, <vscale x 8 x half> %b)
+  %3 = tail call fast <vscale x 8 x half> @llvm.aarch64.sve.fadd.u.nxv8f16(<vscale x 8 x i1> %1, <vscale x 8 x half> %c, <vscale x 8 x half> %2)
+  ret <vscale x 8 x half> %3
+}
+
 define <vscale x 16 x i8> @combine_mla_i8(<vscale x 16 x i1> %p, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b, <vscale x 16 x i8> %c) #0 {
 ; CHECK-LABEL: @combine_mla_i8(
 ; CHECK-NEXT:    [[TMP1:%.*]] = call <vscale x 16 x i8> @llvm.aarch64.sve.mla.nxv16i8(<vscale x 16 x i1> [[P:%.*]], <vscale x 16 x i8> [[C:%.*]], <vscale x 16 x i8> [[A:%.*]], <vscale x 16 x i8> [[B:%.*]])
@@ -59,6 +71,18 @@ define <vscale x 8 x half> @combine_fmls(<vscale x 16 x i1> %p, <vscale x 8 x ha
   ret <vscale x 8 x half> %3
 }
 
+define <vscale x 8 x half> @combine_fmls_u(<vscale x 16 x i1> %p, <vscale x 8 x half> %a, <vscale x 8 x half> %b, <vscale x 8 x half> %c) #0 {
+; CHECK-LABEL: @combine_fmls_u(
+; CHECK-NEXT:    [[TMP1:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[P:%.*]])
+; CHECK-NEXT:    [[TMP2:%.*]] = call fast <vscale x 8 x half> @llvm.aarch64.sve.fmls.u.nxv8f16(<vscale x 8 x i1> [[TMP1]], <vscale x 8 x half> [[C:%.*]], <vscale x 8 x half> [[A:%.*]], <vscale x 8 x half> [[B:%.*]])
+; CHECK-NEXT:    ret <vscale x 8 x half> [[TMP2]]
+;
+  %1 = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> %p)
+  %2 = tail call fast <vscale x 8 x half> @llvm.aarch64.sve.fmul.u.nxv8f16(<vscale x 8 x i1> %1, <vscale x 8 x half> %a, <vscale x 8 x half> %b)
+  %3 = tail call fast <vscale x 8 x half> @llvm.aarch64.sve.fsub.u.nxv8f16(<vscale x 8 x i1> %1, <vscale x 8 x half> %c, <vscale x 8 x half> %2)
+  ret <vscale x 8 x half> %3
+}
+
 define <vscale x 16 x i8> @combine_mls_i8(<vscale x 16 x i1> %p, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b, <vscale x 16 x i8> %c) #0 {
 ; CHECK-LABEL: @combine_mls_i8(
 ; CHECK-NEXT:    [[TMP1:%.*]] = call <vscale x 16 x i8> @llvm.aarch64.sve.mls.nxv16i8(<vscale x 16 x i1> [[P:%.*]], <vscale x 16 x i8> [[C:%.*]], <vscale x 16 x i8> [[A:%.*]], <vscale x 16 x i8> [[B:%.*]])
@@ -173,6 +197,9 @@ declare <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x
 declare <vscale x 8 x half> @llvm.aarch64.sve.fmul.nxv8f16(<vscale x 8 x i1>, <vscale x 8 x half>, <vscale x 8 x half>)
 declare <vscale x 8 x half> @llvm.aarch64.sve.fadd.nxv8f16(<vscale x 8 x i1>, <vscale x 8 x half>, <vscale x 8 x half>)
 declare <vscale x 8 x half> @llvm.aarch64.sve.fsub.nxv8f16(<vscale x 8 x i1>, <vscale x 8 x half>, <vscale x 8 x half>)
+declare <vscale x 8 x half> @llvm.aarch64.sve.fmul.u.nxv8f16(<vscale x 8 x i1>, <vscale x 8 x half>, <vscale x 8 x half>)
+declare <vscale x 8 x half> @llvm.aarch64.sve.fadd.u.nxv8f16(<vscale x 8 x i1>, <vscale x 8 x half>, <vscale x 8 x half>)
+declare <vscale x 8 x half> @llvm.aarch64.sve.fsub.u.nxv8f16(<vscale x 8 x i1>, <vscale x 8 x half>, <vscale x 8 x half>)
 declare <vscale x 16 x i8> @llvm.aarch64.sve.mul.nxv16i8(<vscale x 16 x i1>, <vscale x 16 x i8>, <vscale x 16 x i8>)
 declare <vscale x 16 x i8> @llvm.aarch64.sve.add.nxv16i8(<vscale x 16 x i1>, <vscale x 16 x i8>, <vscale x 16 x i8>)
 declare <vscale x 16 x i8> @llvm.aarch64.sve.sub.nxv16i8(<vscale x 16 x i1>, <vscale x 16 x i8>, <vscale x 16 x i8>)
diff --git a/llvm/test/Transforms/InstCombine/abs-1.ll b/llvm/test/Transforms/InstCombine/abs-1.ll
index b0a1044902bc4..7355c560c820b 100644
--- a/llvm/test/Transforms/InstCombine/abs-1.ll
+++ b/llvm/test/Transforms/InstCombine/abs-1.ll
@@ -679,6 +679,8 @@ define i8 @nabs_extra_use_icmp_sub(i8 %x) {
   ret i8 %s
 }
 
+; TODO: negate-of-abs-diff
+
 define i32 @nabs_diff_signed_slt(i32 %a, i32 %b) {
 ; CHECK-LABEL: @nabs_diff_signed_slt(
 ; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 [[A:%.*]], [[B:%.*]]
@@ -694,6 +696,8 @@ define i32 @nabs_diff_signed_slt(i32 %a, i32 %b) {
   ret i32 %cond
 }
 
+; TODO: negate-of-abs-diff
+
 define <2 x i8> @nabs_diff_signed_sle(<2 x i8> %a, <2 x i8> %b) {
 ; CHECK-LABEL: @nabs_diff_signed_sle(
 ; CHECK-NEXT:    [[CMP_NOT:%.*]] = icmp sgt <2 x i8> [[A:%.*]], [[B:%.*]]
@@ -711,11 +715,9 @@ define <2 x i8> @nabs_diff_signed_sle(<2 x i8> %a, <2 x i8> %b) {
 
 define i8 @abs_diff_signed_sgt(i8 %a, i8 %b) {
 ; CHECK-LABEL: @abs_diff_signed_sgt(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 [[A:%.*]], [[B:%.*]]
-; CHECK-NEXT:    [[SUB_BA:%.*]] = sub nsw i8 [[B]], [[A]]
-; CHECK-NEXT:    [[SUB_AB:%.*]] = sub nsw i8 [[A]], [[B]]
+; CHECK-NEXT:    [[SUB_AB:%.*]] = sub nsw i8 [[A:%.*]], [[B:%.*]]
 ; CHECK-NEXT:    call void @extra_use(i8 [[SUB_AB]])
-; CHECK-NEXT:    [[COND:%.*]] = select i1 [[CMP]], i8 [[SUB_AB]], i8 [[SUB_BA]]
+; CHECK-NEXT:    [[COND:%.*]] = call i8 @llvm.abs.i8(i8 [[SUB_AB]], i1 true)
 ; CHECK-NEXT:    ret i8 [[COND]]
 ;
   %cmp = icmp sgt i8 %a, %b
@@ -728,12 +730,11 @@ define i8 @abs_diff_signed_sgt(i8 %a, i8 %b) {
 
 define i8 @abs_diff_signed_sge(i8 %a, i8 %b) {
 ; CHECK-LABEL: @abs_diff_signed_sge(
-; CHECK-NEXT:    [[CMP_NOT:%.*]] = icmp slt i8 [[A:%.*]], [[B:%.*]]
-; CHECK-NEXT:    [[SUB_BA:%.*]] = sub nsw i8 [[B]], [[A]]
+; CHECK-NEXT:    [[SUB_BA:%.*]] = sub nsw i8 [[B:%.*]], [[A:%.*]]
 ; CHECK-NEXT:    call void @extra_use(i8 [[SUB_BA]])
 ; CHECK-NEXT:    [[SUB_AB:%.*]] = sub nsw i8 [[A]], [[B]]
 ; CHECK-NEXT:    call void @extra_use(i8 [[SUB_AB]])
-; CHECK-NEXT:    [[COND:%.*]] = select i1 [[CMP_NOT]], i8 [[SUB_BA]], i8 [[SUB_AB]]
+; CHECK-NEXT:    [[COND:%.*]] = call i8 @llvm.abs.i8(i8 [[SUB_AB]], i1 true)
 ; CHECK-NEXT:    ret i8 [[COND]]
 ;
   %cmp = icmp sge i8 %a, %b
@@ -745,6 +746,8 @@ define i8 @abs_diff_signed_sge(i8 %a, i8 %b) {
   ret i8 %cond
 }
 
+; negative test - need nsw
+
 define i32 @abs_diff_signed_slt_no_nsw(i32 %a, i32 %b) {
 ; CHECK-LABEL: @abs_diff_signed_slt_no_nsw(
 ; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 [[A:%.*]], [[B:%.*]]
@@ -760,12 +763,12 @@ define i32 @abs_diff_signed_slt_no_nsw(i32 %a, i32 %b) {
   ret i32 %cond
 }
 
+; bonus nuw - it's fine to match the pattern, but nuw can't propagate
+
 define i8 @abs_diff_signed_sgt_nsw_nuw(i8 %a, i8 %b) {
 ; CHECK-LABEL: @abs_diff_signed_sgt_nsw_nuw(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 [[A:%.*]], [[B:%.*]]
-; CHECK-NEXT:    [[SUB_BA:%.*]] = sub nuw nsw i8 [[B]], [[A]]
-; CHECK-NEXT:    [[SUB_AB:%.*]] = sub nuw nsw i8 [[A]], [[B]]
-; CHECK-NEXT:    [[COND:%.*]] = select i1 [[CMP]], i8 [[SUB_AB]], i8 [[SUB_BA]]
+; CHECK-NEXT:    [[SUB_AB:%.*]] = sub nsw i8 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[COND:%.*]] = call i8 @llvm.abs.i8(i8 [[SUB_AB]], i1 true)
 ; CHECK-NEXT:    ret i8 [[COND]]
 ;
   %cmp = icmp sgt i8 %a, %b
@@ -775,12 +778,12 @@ define i8 @abs_diff_signed_sgt_nsw_nuw(i8 %a, i8 %b) {
   ret i8 %cond
 }
 
+; this is absolute diff, but nuw can't propagate and nsw can be set.
+
 define i8 @abs_diff_signed_sgt_nuw(i8 %a, i8 %b) {
 ; CHECK-LABEL: @abs_diff_signed_sgt_nuw(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 [[A:%.*]], [[B:%.*]]
-; CHECK-NEXT:    [[SUB_BA:%.*]] = sub nuw i8 [[B]], [[A]]
-; CHECK-NEXT:    [[SUB_AB:%.*]] = sub nuw i8 [[A]], [[B]]
-; CHECK-NEXT:    [[COND:%.*]] = select i1 [[CMP]], i8 [[SUB_AB]], i8 [[SUB_BA]]
+; CHECK-NEXT:    [[SUB_AB:%.*]] = sub nsw i8 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[COND:%.*]] = call i8 @llvm.abs.i8(i8 [[SUB_AB]], i1 true)
 ; CHECK-NEXT:    ret i8 [[COND]]
 ;
   %cmp = icmp sgt i8 %a, %b
@@ -790,13 +793,14 @@ define i8 @abs_diff_signed_sgt_nuw(i8 %a, i8 %b) {
   ret i8 %cond
 }
 
+; same as above
+
 define i8 @abs_diff_signed_sgt_nuw_extra_use1(i8 %a, i8 %b) {
 ; CHECK-LABEL: @abs_diff_signed_sgt_nuw_extra_use1(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 [[A:%.*]], [[B:%.*]]
-; CHECK-NEXT:    [[SUB_BA:%.*]] = sub nuw i8 [[B]], [[A]]
+; CHECK-NEXT:    [[SUB_BA:%.*]] = sub nuw i8 [[B:%.*]], [[A:%.*]]
 ; CHECK-NEXT:    call void @extra_use(i8 [[SUB_BA]])
-; CHECK-NEXT:    [[SUB_AB:%.*]] = sub nuw i8 [[A]], [[B]]
-; CHECK-NEXT:    [[COND:%.*]] = select i1 [[CMP]], i8 [[SUB_AB]], i8 [[SUB_BA]]
+; CHECK-NEXT:    [[SUB_AB:%.*]] = sub nsw i8 [[A]], [[B]]
+; CHECK-NEXT:    [[COND:%.*]] = call i8 @llvm.abs.i8(i8 [[SUB_AB]], i1 true)
 ; CHECK-NEXT:    ret i8 [[COND]]
 ;
   %cmp = icmp sgt i8 %a, %b
@@ -807,13 +811,13 @@ define i8 @abs_diff_signed_sgt_nuw_extra_use1(i8 %a, i8 %b) {
   ret i8 %cond
 }
 
+; nuw can't propagate, and the extra use prevents applying nsw
+
 define i8 @abs_diff_signed_sgt_nuw_extra_use2(i8 %a, i8 %b) {
 ; CHECK-LABEL: @abs_diff_signed_sgt_nuw_extra_use2(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 [[A:%.*]], [[B:%.*]]
-; CHECK-NEXT:    [[SUB_BA:%.*]] = sub nuw i8 [[B]], [[A]]
-; CHECK-NEXT:    [[SUB_AB:%.*]] = sub nuw i8 [[A]], [[B]]
+; CHECK-NEXT:    [[SUB_AB:%.*]] = sub i8 [[A:%.*]], [[B:%.*]]
 ; CHECK-NEXT:    call void @extra_use(i8 [[SUB_AB]])
-; CHECK-NEXT:    [[COND:%.*]] = select i1 [[CMP]], i8 [[SUB_AB]], i8 [[SUB_BA]]
+; CHECK-NEXT:    [[COND:%.*]] = call i8 @llvm.abs.i8(i8 [[SUB_AB]], i1 true)
 ; CHECK-NEXT:    ret i8 [[COND]]
 ;
   %cmp = icmp sgt i8 %a, %b
@@ -824,14 +828,15 @@ define i8 @abs_diff_signed_sgt_nuw_extra_use2(i8 %a, i8 %b) {
   ret i8 %cond
 }
 
+; same as above
+
 define i8 @abs_diff_signed_sgt_nuw_extra_use3(i8 %a, i8 %b) {
 ; CHECK-LABEL: @abs_diff_signed_sgt_nuw_extra_use3(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 [[A:%.*]], [[B:%.*]]
-; CHECK-NEXT:    [[SUB_BA:%.*]] = sub nuw i8 [[B]], [[A]]
+; CHECK-NEXT:    [[SUB_BA:%.*]] = sub nuw i8 [[B:%.*]], [[A:%.*]]
 ; CHECK-NEXT:    call void @extra_use(i8 [[SUB_BA]])
-; CHECK-NEXT:    [[SUB_AB:%.*]] = sub nuw i8 [[A]], [[B]]
+; CHECK-NEXT:    [[SUB_AB:%.*]] = sub i8 [[A]], [[B]]
 ; CHECK-NEXT:    call void @extra_use(i8 [[SUB_AB]])
-; CHECK-NEXT:    [[COND:%.*]] = select i1 [[CMP]], i8 [[SUB_AB]], i8 [[SUB_BA]]
+; CHECK-NEXT:    [[COND:%.*]] = call i8 @llvm.abs.i8(i8 [[SUB_AB]], i1 true)
 ; CHECK-NEXT:    ret i8 [[COND]]
 ;
   %cmp = icmp sgt i8 %a, %b
@@ -843,6 +848,8 @@ define i8 @abs_diff_signed_sgt_nuw_extra_use3(i8 %a, i8 %b) {
   ret i8 %cond
 }
 
+; negative test - wrong predicate
+
 define i32 @abs_diff_signed_slt_swap_wrong_pred1(i32 %a, i32 %b) {
 ; CHECK-LABEL: @abs_diff_signed_slt_swap_wrong_pred1(
 ; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i32 [[A:%.*]], [[B:%.*]]
@@ -858,6 +865,8 @@ define i32 @abs_diff_signed_slt_swap_wrong_pred1(i32 %a, i32 %b) {
   ret i32 %cond
 }
 
+; negative test - wrong predicate
+
 define i32 @abs_diff_signed_slt_swap_wrong_pred2(i32 %a, i32 %b) {
 ; CHECK-LABEL: @abs_diff_signed_slt_swap_wrong_pred2(
 ; CHECK-NEXT:    [[CMP:%.*]] = icmp ult i32 [[A:%.*]], [[B:%.*]]
@@ -873,6 +882,8 @@ define i32 @abs_diff_signed_slt_swap_wrong_pred2(i32 %a, i32 %b) {
   ret i32 %cond
 }
 
+; negative test - need common operands
+
 define i32 @abs_diff_signed_slt_swap_wrong_op(i32 %a, i32 %b, i32 %z) {
 ; CHECK-LABEL: @abs_diff_signed_slt_swap_wrong_op(
 ; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i32 [[A:%.*]], [[B:%.*]]
@@ -890,10 +901,8 @@ define i32 @abs_diff_signed_slt_swap_wrong_op(i32 %a, i32 %b, i32 %z) {
 
 define i32 @abs_diff_signed_slt_swap(i32 %a, i32 %b) {
 ; CHECK-LABEL: @abs_diff_signed_slt_swap(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 [[A:%.*]], [[B:%.*]]
-; CHECK-NEXT:    [[SUB_BA:%.*]] = sub nsw i32 [[B]], [[A]]
-; CHECK-NEXT:    [[SUB_AB:%.*]] = sub nsw i32 [[A]], [[B]]
-; CHECK-NEXT:    [[COND:%.*]] = select i1 [[CMP]], i32 [[SUB_BA]], i32 [[SUB_AB]]
+; CHECK-NEXT:    [[SUB_AB:%.*]] = sub nsw i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[COND:%.*]] = call i32 @llvm.abs.i32(i32 [[SUB_AB]], i1 true)
 ; CHECK-NEXT:    ret i32 [[COND]]
 ;
   %cmp = icmp slt i32 %a, %b
@@ -905,10 +914,8 @@ define i32 @abs_diff_signed_slt_swap(i32 %a, i32 %b) {
 
 define <2 x i8> @abs_diff_signed_sle_swap(<2 x i8> %a, <2 x i8> %b) {
 ; CHECK-LABEL: @abs_diff_signed_sle_swap(
-; CHECK-NEXT:    [[CMP_NOT:%.*]] = icmp sgt <2 x i8> [[A:%.*]], [[B:%.*]]
-; CHECK-NEXT:    [[SUB_BA:%.*]] = sub nsw <2 x i8> [[B]], [[A]]
-; CHECK-NEXT:    [[SUB_AB:%.*]] = sub nsw <2 x i8> [[A]], [[B]]
-; CHECK-NEXT:    [[COND:%.*]] = select <2 x i1> [[CMP_NOT]], <2 x i8> [[SUB_AB]], <2 x i8> [[SUB_BA]]
+; CHECK-NEXT:    [[SUB_AB:%.*]] = sub nsw <2 x i8> [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[COND:%.*]] = call <2 x i8> @llvm.abs.v2i8(<2 x i8> [[SUB_AB]], i1 true)
 ; CHECK-NEXT:    ret <2 x i8> [[COND]]
 ;
   %cmp = icmp sle <2 x i8> %a, %b
@@ -918,6 +925,8 @@ define <2 x i8> @abs_diff_signed_sle_swap(<2 x i8> %a, <2 x i8> %b) {
   ret <2 x i8> %cond
 }
 
+; TODO: negate-of-abs-diff
+
 define i8 @nabs_diff_signed_sgt_swap(i8 %a, i8 %b) {
 ; CHECK-LABEL: @nabs_diff_signed_sgt_swap(
 ; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 [[A:%.*]], [[B:%.*]]
@@ -935,6 +944,8 @@ define i8 @nabs_diff_signed_sgt_swap(i8 %a, i8 %b) {
   ret i8 %cond
 }
 
+; TODO: negate-of-abs-diff, but too many uses?
+
 define i8 @nabs_diff_signed_sge_swap(i8 %a, i8 %b) {
 ; CHECK-LABEL: @nabs_diff_signed_sge_swap(
 ; CHECK-NEXT:    [[CMP_NOT:%.*]] = icmp slt i8 [[A:%.*]], [[B:%.*]]
@@ -954,6 +965,8 @@ define i8 @nabs_diff_signed_sge_swap(i8 %a, i8 %b) {
   ret i8 %cond
 }
 
+; negative test - need nsw
+
 define i32 @abs_diff_signed_slt_no_nsw_swap(i32 %a, i32 %b) {
 ; CHECK-LABEL: @abs_diff_signed_slt_no_nsw_swap(
 ; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 [[A:%.*]], [[B:%.*]]
diff --git a/llvm/test/Transforms/InstCombine/bit_ceil.ll b/llvm/test/Transforms/InstCombine/bit_ceil.ll
new file mode 100644
index 0000000000000..aa98896aac549
--- /dev/null
+++ b/llvm/test/Transforms/InstCombine/bit_ceil.ll
@@ -0,0 +1,156 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -passes=instcombine -S | FileCheck %s
+
+; std::bit_ceil<uint32_t>(x)
+define i32 @bit_ceil_32(i32 %x) {
+; CHECK-LABEL: @bit_ceil_32(
+; CHECK-NEXT:    [[DEC:%.*]] = add i32 [[X:%.*]], -1
+; CHECK-NEXT:    [[CTLZ:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[DEC]], i1 false), !range [[RNG0:![0-9]+]]
+; CHECK-NEXT:    [[SUB:%.*]] = sub nuw nsw i32 32, [[CTLZ]]
+; CHECK-NEXT:    [[SHL:%.*]] = shl nuw i32 1, [[SUB]]
+; CHECK-NEXT:    [[UGT:%.*]] = icmp ugt i32 [[X]], 1
+; CHECK-NEXT:    [[SEL:%.*]] = select i1 [[UGT]], i32 [[SHL]], i32 1
+; CHECK-NEXT:    ret i32 [[SEL]]
+;
+  %dec = add i32 %x, -1
+  %ctlz = tail call i32 @llvm.ctlz.i32(i32 %dec, i1 false)
+  %sub = sub i32 32, %ctlz
+  %shl = shl i32 1, %sub
+  %ugt = icmp ugt i32 %x, 1
+  %sel = select i1 %ugt, i32 %shl, i32 1
+  ret i32 %sel
+}
+
+; std::bit_ceil<uint64_t>(x)
+define i64 @bit_ceil_64(i64 %x) {
+; CHECK-LABEL: @bit_ceil_64(
+; CHECK-NEXT:    [[DEC:%.*]] = add i64 [[X:%.*]], -1
+; CHECK-NEXT:    [[CTLZ:%.*]] = tail call i64 @llvm.ctlz.i64(i64 [[DEC]], i1 false), !range [[RNG1:![0-9]+]]
+; CHECK-NEXT:    [[SUB:%.*]] = sub nuw nsw i64 64, [[CTLZ]]
+; CHECK-NEXT:    [[SHL:%.*]] = shl nuw i64 1, [[SUB]]
+; CHECK-NEXT:    [[UGT:%.*]] = icmp ugt i64 [[X]], 1
+; CHECK-NEXT:    [[SEL:%.*]] = select i1 [[UGT]], i64 [[SHL]], i64 1
+; CHECK-NEXT:    ret i64 [[SEL]]
+;
+  %dec = add i64 %x, -1
+  %ctlz = tail call i64 @llvm.ctlz.i64(i64 %dec, i1 false)
+  %sub = sub i64 64, %ctlz
+  %shl = shl i64 1, %sub
+  %ugt = icmp ugt i64 %x, 1
+  %sel = select i1 %ugt, i64 %shl, i64 1
+  ret i64 %sel
+}
+
+; std::bit_ceil<uint32_t>(x - 1)
+define i32 @bit_ceil_32_minus_1(i32 %x) {
+; CHECK-LABEL: @bit_ceil_32_minus_1(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[SUB:%.*]] = add i32 [[X:%.*]], -2
+; CHECK-NEXT:    [[CTLZ:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[SUB]], i1 false), !range [[RNG0]]
+; CHECK-NEXT:    [[SUB2:%.*]] = sub nuw nsw i32 32, [[CTLZ]]
+; CHECK-NEXT:    [[SHL:%.*]] = shl nuw i32 1, [[SUB2]]
+; CHECK-NEXT:    [[ADD:%.*]] = add i32 [[X]], -3
+; CHECK-NEXT:    [[ULT:%.*]] = icmp ult i32 [[ADD]], -2
+; CHECK-NEXT:    [[SEL:%.*]] = select i1 [[ULT]], i32 [[SHL]], i32 1
+; CHECK-NEXT:    ret i32 [[SEL]]
+;
+entry:
+  %sub = add i32 %x, -2
+  %ctlz = tail call i32 @llvm.ctlz.i32(i32 %sub, i1 false)
+  %sub2 = sub nuw nsw i32 32, %ctlz
+  %shl = shl nuw i32 1, %sub2
+  %add = add i32 %x, -3
+  %ult = icmp ult i32 %add, -2
+  %sel = select i1 %ult, i32 %shl, i32 1
+  ret i32 %sel
+}
+
+; std::bit_ceil<uint32_t>(x + 1)
+define i32 @bit_ceil_32_plus_1(i32 %x) {
+; CHECK-LABEL: @bit_ceil_32_plus_1(
+; CHECK-NEXT:    [[CTLZ:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[X:%.*]], i1 false), !range [[RNG0]]
+; CHECK-NEXT:    [[SUB:%.*]] = sub nuw nsw i32 32, [[CTLZ]]
+; CHECK-NEXT:    [[SHL:%.*]] = shl nuw i32 1, [[SUB]]
+; CHECK-NEXT:    [[DEC:%.*]] = add i32 [[X]], -1
+; CHECK-NEXT:    [[ULT:%.*]] = icmp ult i32 [[DEC]], -2
+; CHECK-NEXT:    [[SEL:%.*]] = select i1 [[ULT]], i32 [[SHL]], i32 1
+; CHECK-NEXT:    ret i32 [[SEL]]
+;
+  %ctlz = tail call i32 @llvm.ctlz.i32(i32 %x, i1 false)
+  %sub = sub i32 32, %ctlz
+  %shl = shl i32 1, %sub
+  %dec = add i32 %x, -1
+  %ult = icmp ult i32 %dec, -2
+  %sel = select i1 %ult, i32 %shl, i32 1
+  ret i32 %sel
+}
+
+define i32 @bit_ceil_plus_2(i32 %x) {
+; CHECK-LABEL: @bit_ceil_plus_2(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[SUB:%.*]] = add i32 [[X:%.*]], 1
+; CHECK-NEXT:    [[CTLZ:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[SUB]], i1 false), !range [[RNG0]]
+; CHECK-NEXT:    [[SUB2:%.*]] = sub nuw nsw i32 32, [[CTLZ]]
+; CHECK-NEXT:    [[SHL:%.*]] = shl nuw i32 1, [[SUB2]]
+; CHECK-NEXT:    [[ULT:%.*]] = icmp ult i32 [[X]], -2
+; CHECK-NEXT:    [[SEL:%.*]] = select i1 [[ULT]], i32 [[SHL]], i32 1
+; CHECK-NEXT:    ret i32 [[SEL]]
+;
+entry:
+  %sub = add i32 %x, 1
+  %ctlz = tail call i32 @llvm.ctlz.i32(i32 %sub, i1 false)
+  %sub2 = sub nuw nsw i32 32, %ctlz
+  %shl = shl nuw i32 1, %sub2
+  %ult = icmp ult i32 %x, -2
+  %sel = select i1 %ult, i32 %shl, i32 1
+  ret i32 %sel
+}
+
+; std::bit_ceil<uint32_t>(-x)
+define i32 @bit_ceil_32_neg(i32 %x) {
+; CHECK-LABEL: @bit_ceil_32_neg(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[SUB:%.*]] = xor i32 [[X:%.*]], -1
+; CHECK-NEXT:    [[CTLZ:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[SUB]], i1 false), !range [[RNG0]]
+; CHECK-NEXT:    [[SUB2:%.*]] = sub nuw nsw i32 32, [[CTLZ]]
+; CHECK-NEXT:    [[SHL:%.*]] = shl nuw i32 1, [[SUB2]]
+; CHECK-NEXT:    [[NOTSUB:%.*]] = add i32 [[X]], -1
+; CHECK-NEXT:    [[ULT:%.*]] = icmp ult i32 [[NOTSUB]], -2
+; CHECK-NEXT:    [[SEL:%.*]] = select i1 [[ULT]], i32 [[SHL]], i32 1
+; CHECK-NEXT:    ret i32 [[SEL]]
+;
+entry:
+  %sub = xor i32 %x, -1
+  %ctlz = tail call i32 @llvm.ctlz.i32(i32 %sub, i1 false)
+  %sub2 = sub nuw nsw i32 32, %ctlz
+  %shl = shl nuw i32 1, %sub2
+  %notsub = add i32 %x, -1
+  %ult = icmp ult i32 %notsub, -2
+  %sel = select i1 %ult, i32 %shl, i32 1
+  ret i32 %sel
+}
+
+; std::bit_ceil<uint32_t>(~x)
+define i32 @bit_ceil_not(i32 %x) {
+; CHECK-LABEL: @bit_ceil_not(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[SUB:%.*]] = sub i32 -2, [[X:%.*]]
+; CHECK-NEXT:    [[CTLZ:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[SUB]], i1 false), !range [[RNG0]]
+; CHECK-NEXT:    [[SUB2:%.*]] = sub nuw nsw i32 32, [[CTLZ]]
+; CHECK-NEXT:    [[SHL:%.*]] = shl nuw i32 1, [[SUB2]]
+; CHECK-NEXT:    [[ULT:%.*]] = icmp ult i32 [[X]], -2
+; CHECK-NEXT:    [[SEL:%.*]] = select i1 [[ULT]], i32 [[SHL]], i32 1
+; CHECK-NEXT:    ret i32 [[SEL]]
+;
+entry:
+  %sub = sub i32 -2, %x
+  %ctlz = tail call i32 @llvm.ctlz.i32(i32 %sub, i1 false)
+  %sub2 = sub nuw nsw i32 32, %ctlz
+  %shl = shl nuw i32 1, %sub2
+  %ult = icmp ult i32 %x, -2
+  %sel = select i1 %ult, i32 %shl, i32 1
+  ret i32 %sel
+}
+
+declare i32 @llvm.ctlz.i32(i32, i1 immarg)
+declare i64 @llvm.ctlz.i64(i64, i1 immarg)
diff --git a/llvm/test/Transforms/InstCombine/bit_floor.ll b/llvm/test/Transforms/InstCombine/bit_floor.ll
new file mode 100644
index 0000000000000..0ef7fe3d22e0f
--- /dev/null
+++ b/llvm/test/Transforms/InstCombine/bit_floor.ll
@@ -0,0 +1,43 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -passes=instcombine -S | FileCheck %s
+
+define i32 @bit_floor_32(i32 %x) {
+; CHECK-LABEL: @bit_floor_32(
+; CHECK-NEXT:    [[EQ0:%.*]] = icmp eq i32 [[X:%.*]], 0
+; CHECK-NEXT:    [[LSHR:%.*]] = lshr i32 [[X]], 1
+; CHECK-NEXT:    [[CTLZ:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[LSHR]], i1 false), !range [[RNG0:![0-9]+]]
+; CHECK-NEXT:    [[SUB:%.*]] = sub nuw nsw i32 32, [[CTLZ]]
+; CHECK-NEXT:    [[SHL:%.*]] = shl nuw i32 1, [[SUB]]
+; CHECK-NEXT:    [[SEL:%.*]] = select i1 [[EQ0]], i32 0, i32 [[SHL]]
+; CHECK-NEXT:    ret i32 [[SEL]]
+;
+  %eq0 = icmp eq i32 %x, 0
+  %lshr = lshr i32 %x, 1
+  %ctlz = tail call i32 @llvm.ctlz.i32(i32 %lshr, i1 false)
+  %sub = sub i32 32, %ctlz
+  %shl = shl i32 1, %sub
+  %sel = select i1 %eq0, i32 0, i32 %shl
+  ret i32 %sel
+}
+
+define i64 @bit_floor_64(i64 %x) {
+; CHECK-LABEL: @bit_floor_64(
+; CHECK-NEXT:    [[EQ0:%.*]] = icmp eq i64 [[X:%.*]], 0
+; CHECK-NEXT:    [[LSHR:%.*]] = lshr i64 [[X]], 1
+; CHECK-NEXT:    [[CTLZ:%.*]] = tail call i64 @llvm.ctlz.i64(i64 [[LSHR]], i1 false), !range [[RNG1:![0-9]+]]
+; CHECK-NEXT:    [[SUB:%.*]] = sub nuw nsw i64 64, [[CTLZ]]
+; CHECK-NEXT:    [[SHL:%.*]] = shl nuw i64 1, [[SUB]]
+; CHECK-NEXT:    [[SEL:%.*]] = select i1 [[EQ0]], i64 0, i64 [[SHL]]
+; CHECK-NEXT:    ret i64 [[SEL]]
+;
+  %eq0 = icmp eq i64 %x, 0
+  %lshr = lshr i64 %x, 1
+  %ctlz = tail call i64 @llvm.ctlz.i64(i64 %lshr, i1 false)
+  %sub = sub i64 64, %ctlz
+  %shl = shl i64 1, %sub
+  %sel = select i1 %eq0, i64 0, i64 %shl
+  ret i64 %sel
+}
+
+declare i32 @llvm.ctlz.i32(i32, i1 immarg)
+declare i64 @llvm.ctlz.i64(i64, i1 immarg)
diff --git a/llvm/test/Transforms/InstCombine/cast.ll b/llvm/test/Transforms/InstCombine/cast.ll
index cd3b032d739de..5b480b1157936 100644
--- a/llvm/test/Transforms/InstCombine/cast.ll
+++ b/llvm/test/Transforms/InstCombine/cast.ll
@@ -1441,7 +1441,8 @@ define i64 @PR28745() {
 ; LE-LABEL: @PR28745(
 ; LE-NEXT:    ret i64 0
 ;
-  %e = extractvalue { i32 } select (i1 icmp eq (i16 extractelement (<2 x i16> bitcast (<1 x i32> <i32 1> to <2 x i16>), i32 0), i16 0), { i32 } { i32 1 }, { i32 } zeroinitializer), 0
+  %s = select i1 icmp eq (i16 extractelement (<2 x i16> bitcast (<1 x i32> <i32 1> to <2 x i16>), i32 0), i16 0), { i32 } { i32 1 }, { i32 } zeroinitializer
+  %e = extractvalue { i32 } %s, 0
   %b = zext i32 %e to i64
   ret i64 %b
 }
diff --git a/llvm/test/Transforms/InstCombine/cmp-intrinsic.ll b/llvm/test/Transforms/InstCombine/cmp-intrinsic.ll
index accbfdc5d3ef4..f2cecd34228d0 100644
--- a/llvm/test/Transforms/InstCombine/cmp-intrinsic.ll
+++ b/llvm/test/Transforms/InstCombine/cmp-intrinsic.ll
@@ -13,6 +13,8 @@ declare i11 @llvm.ctpop.i11(i11)
 declare <2 x i32> @llvm.cttz.v2i32(<2 x i32>, i1)
 declare <2 x i32> @llvm.ctlz.v2i32(<2 x i32>, i1)
 declare <2 x i32> @llvm.ctpop.v2i32(<2 x i32>)
+declare i8 @llvm.bitreverse.i8(i8)
+declare <2 x i8> @llvm.bitreverse.v2i8(<2 x i8>)
 declare void @use6(i6)
 
 define i1 @bswap_eq_i16(i16 %x) {
@@ -762,3 +764,46 @@ define i1 @trunc_negative_destbits_not_enough(i33 %x) {
   %cmp = icmp ult i4 %trunc, 7
   ret i1 %cmp
 }
+
+define i1 @bitreverse_ne_22(i8 %x) {
+; CHECK-LABEL: @bitreverse_ne_22(
+; CHECK-NEXT:    [[Z:%.*]] = icmp ne i8 [[X:%.*]], 104
+; CHECK-NEXT:    ret i1 [[Z]]
+;
+  %y = call i8 @llvm.bitreverse.i8(i8 %x)
+  %z = icmp ne i8 %y, 22
+  ret i1 %z
+}
+
+define i1 @bitreverse_ult_22_fail_not_equality_pred(i8 %x) {
+; CHECK-LABEL: @bitreverse_ult_22_fail_not_equality_pred(
+; CHECK-NEXT:    [[Y:%.*]] = call i8 @llvm.bitreverse.i8(i8 [[X:%.*]])
+; CHECK-NEXT:    [[Z:%.*]] = icmp ult i8 [[Y]], 22
+; CHECK-NEXT:    ret i1 [[Z]]
+;
+  %y = call i8 @llvm.bitreverse.i8(i8 %x)
+  %z = icmp ult i8 %y, 22
+  ret i1 %z
+}
+
+
+define <2 x i1> @bitreverse_vec_eq_2_2(<2 x i8> %x) {
+; CHECK-LABEL: @bitreverse_vec_eq_2_2(
+; CHECK-NEXT:    [[Z:%.*]] = icmp eq <2 x i8> [[X:%.*]], <i8 64, i8 64>
+; CHECK-NEXT:    ret <2 x i1> [[Z]]
+;
+  %y = call <2 x i8> @llvm.bitreverse.v2i8(<2 x i8> %x)
+  %z = icmp eq <2 x i8> %y, <i8 2, i8 2>
+  ret <2 x i1> %z
+}
+
+define <2 x i1> @bitreverse_vec_eq_1_2_todo_no_splat(<2 x i8> %x) {
+; CHECK-LABEL: @bitreverse_vec_eq_1_2_todo_no_splat(
+; CHECK-NEXT:    [[Y:%.*]] = call <2 x i8> @llvm.bitreverse.v2i8(<2 x i8> [[X:%.*]])
+; CHECK-NEXT:    [[Z:%.*]] = icmp eq <2 x i8> [[Y]], <i8 1, i8 2>
+; CHECK-NEXT:    ret <2 x i1> [[Z]]
+;
+  %y = call <2 x i8> @llvm.bitreverse.v2i8(<2 x i8> %x)
+  %z = icmp eq <2 x i8> %y, <i8 1, i8 2>
+  ret <2 x i1> %z
+}
diff --git a/llvm/test/Transforms/InstCombine/combine-is.fpclass-and-fcmp.ll b/llvm/test/Transforms/InstCombine/combine-is.fpclass-and-fcmp.ll
index a4ec83661c66e..66ee31f57f961 100644
--- a/llvm/test/Transforms/InstCombine/combine-is.fpclass-and-fcmp.ll
+++ b/llvm/test/Transforms/InstCombine/combine-is.fpclass-and-fcmp.ll
@@ -211,10 +211,8 @@ define <2 x i1> @class_finite_or_fcmp_issubnormal_vector(<2 x half> %x) {
 
 define i1 @fcmp_oeq_zero_or_class_normal(half %x) {
 ; CHECK-LABEL: @fcmp_oeq_zero_or_class_normal(
-; CHECK-NEXT:    [[OEQ_INF:%.*]] = fcmp oeq half [[X:%.*]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X]], i32 264)
-; CHECK-NEXT:    [[OR:%.*]] = or i1 [[OEQ_INF]], [[CLASS]]
-; CHECK-NEXT:    ret i1 [[OR]]
+; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 360)
+; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %oeq.inf = fcmp oeq half %x, 0.0
   %class = call i1 @llvm.is.fpclass.f16(half %x, i32 264)
@@ -276,10 +274,8 @@ define <2 x i1> @fcmp_oeq_zero_or_class_normal_dynamic_v2f16(<2 x half> %x) #2 {
 
 define i1 @class_normal_or_fcmp_oeq_zero(half %x) {
 ; CHECK-LABEL: @class_normal_or_fcmp_oeq_zero(
-; CHECK-NEXT:    [[OEQ_INF:%.*]] = fcmp oeq half [[X:%.*]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X]], i32 264)
-; CHECK-NEXT:    [[OR:%.*]] = or i1 [[OEQ_INF]], [[CLASS]]
-; CHECK-NEXT:    ret i1 [[OR]]
+; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 360)
+; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %oeq.inf = fcmp oeq half %x, 0.0
   %class = call i1 @llvm.is.fpclass.f16(half %x, i32 264)
@@ -289,10 +285,8 @@ define i1 @class_normal_or_fcmp_oeq_zero(half %x) {
 
 define i1 @fcmp_ueq_zero_or_class_normal(half %x) {
 ; CHECK-LABEL: @fcmp_ueq_zero_or_class_normal(
-; CHECK-NEXT:    [[UEQ_INF:%.*]] = fcmp ueq half [[X:%.*]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X]], i32 264)
-; CHECK-NEXT:    [[OR:%.*]] = or i1 [[UEQ_INF]], [[CLASS]]
-; CHECK-NEXT:    ret i1 [[OR]]
+; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 363)
+; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %ueq.inf = fcmp ueq half %x, 0.0
   %class = call i1 @llvm.is.fpclass.f16(half %x, i32 264)
@@ -302,10 +296,8 @@ define i1 @fcmp_ueq_zero_or_class_normal(half %x) {
 
 define i1 @class_normal_or_fcmp_ueq_zero(half %x) {
 ; CHECK-LABEL: @class_normal_or_fcmp_ueq_zero(
-; CHECK-NEXT:    [[UEQ_INF:%.*]] = fcmp ueq half [[X:%.*]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X]], i32 264)
-; CHECK-NEXT:    [[OR:%.*]] = or i1 [[UEQ_INF]], [[CLASS]]
-; CHECK-NEXT:    ret i1 [[OR]]
+; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 363)
+; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %ueq.inf = fcmp ueq half %x, 0.0
   %class = call i1 @llvm.is.fpclass.f16(half %x, i32 264)
@@ -315,10 +307,8 @@ define i1 @class_normal_or_fcmp_ueq_zero(half %x) {
 
 define i1 @fcmp_one_zero_or_class_normal(half %x) {
 ; CHECK-LABEL: @fcmp_one_zero_or_class_normal(
-; CHECK-NEXT:    [[ONE_INF:%.*]] = fcmp one half [[X:%.*]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X]], i32 264)
-; CHECK-NEXT:    [[OR:%.*]] = or i1 [[ONE_INF]], [[CLASS]]
-; CHECK-NEXT:    ret i1 [[OR]]
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp one half [[X:%.*]], 0xH0000
+; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %one.inf = fcmp one half %x, 0.0
   %class = call i1 @llvm.is.fpclass.f16(half %x, i32 264)
@@ -354,10 +344,8 @@ define i1 @fcmp_one_zero_or_class_normal_dynamic(half %x) #2 {
 
 define i1 @class_normal_or_fcmp_one_zero(half %x) {
 ; CHECK-LABEL: @class_normal_or_fcmp_one_zero(
-; CHECK-NEXT:    [[ONE_INF:%.*]] = fcmp one half [[X:%.*]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X]], i32 264)
-; CHECK-NEXT:    [[OR:%.*]] = or i1 [[ONE_INF]], [[CLASS]]
-; CHECK-NEXT:    ret i1 [[OR]]
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp one half [[X:%.*]], 0xH0000
+; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %one.inf = fcmp one half %x, 0.0
   %class = call i1 @llvm.is.fpclass.f16(half %x, i32 264)
@@ -367,10 +355,8 @@ define i1 @class_normal_or_fcmp_one_zero(half %x) {
 
 define i1 @fcmp_une_zero_or_class_normal(half %x) {
 ; CHECK-LABEL: @fcmp_une_zero_or_class_normal(
-; CHECK-NEXT:    [[UNE_INF:%.*]] = fcmp une half [[X:%.*]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X]], i32 264)
-; CHECK-NEXT:    [[OR:%.*]] = or i1 [[UNE_INF]], [[CLASS]]
-; CHECK-NEXT:    ret i1 [[OR]]
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp une half [[X:%.*]], 0xH0000
+; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %une.inf = fcmp une half %x, 0.0
   %class = call i1 @llvm.is.fpclass.f16(half %x, i32 264)
@@ -380,10 +366,8 @@ define i1 @fcmp_une_zero_or_class_normal(half %x) {
 
 define i1 @class_normal_or_fcmp_une_zero(half %x) {
 ; CHECK-LABEL: @class_normal_or_fcmp_une_zero(
-; CHECK-NEXT:    [[UNE_INF:%.*]] = fcmp une half [[X:%.*]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X]], i32 264)
-; CHECK-NEXT:    [[OR:%.*]] = or i1 [[UNE_INF]], [[CLASS]]
-; CHECK-NEXT:    ret i1 [[OR]]
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp une half [[X:%.*]], 0xH0000
+; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %une.inf = fcmp une half %x, 0.0
   %class = call i1 @llvm.is.fpclass.f16(half %x, i32 264)
diff --git a/llvm/test/Transforms/InstCombine/compare-3way.ll b/llvm/test/Transforms/InstCombine/compare-3way.ll
index 15d48f3064464..e2067368fb4c3 100644
--- a/llvm/test/Transforms/InstCombine/compare-3way.ll
+++ b/llvm/test/Transforms/InstCombine/compare-3way.ll
@@ -1,15 +1,25 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
 ; RUN: opt -S -passes=instcombine < %s | FileCheck %s
 
 declare void @use(i32)
 
 ; These 18 exercise all combinations of signed comparison
-; for each of the three values produced by your typical 
+; for each of the three values produced by your typical
 ; 3way compare function (-1, 0, 1)
 
 define void @test_low_sgt(i64 %a, i64 %b) {
-; CHECK-LABEL: @test_low_sgt
-; CHECK: [[TMP1:%.*]] = icmp slt i64 %a, %b
-; CHECK: br i1 [[TMP1]], label %normal, label %unreached
+; CHECK-LABEL: define void @test_low_sgt
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    [[CMP_NOT:%.*]] = icmp slt i64 [[A]], [[B]]
+; CHECK-NEXT:    br i1 [[CMP_NOT]], label [[NORMAL:%.*]], label [[UNREACHED:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    [[EQ:%.*]] = icmp ne i64 [[A]], [[B]]
+; CHECK-NEXT:    [[RESULT:%.*]] = zext i1 [[EQ]] to i32
+; CHECK-NEXT:    call void @use(i32 [[RESULT]])
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -1, i32 1
@@ -24,8 +34,14 @@ unreached:
 }
 
 define void @test_low_slt(i64 %a, i64 %b) {
-; CHECK-LABEL: @test_low_slt
-; CHECK: br i1 false, label %unreached, label %normal
+; CHECK-LABEL: define void @test_low_slt
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    br i1 false, label [[UNREACHED:%.*]], label [[NORMAL:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -1, i32 1
@@ -40,8 +56,19 @@ unreached:
 }
 
 define void @test_low_sge(i64 %a, i64 %b) {
-; CHECK-LABEL: @test_low_sge
-; CHECK: br i1 true, label %unreached, label %normal
+; CHECK-LABEL: define void @test_low_sge
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    br i1 true, label [[UNREACHED:%.*]], label [[NORMAL:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    [[EQ:%.*]] = icmp eq i64 [[A]], [[B]]
+; CHECK-NEXT:    [[SLT:%.*]] = icmp slt i64 [[A]], [[B]]
+; CHECK-NEXT:    [[DOT:%.*]] = select i1 [[SLT]], i32 -1, i32 1
+; CHECK-NEXT:    [[RESULT:%.*]] = select i1 [[EQ]], i32 0, i32 [[DOT]]
+; CHECK-NEXT:    call void @use(i32 [[RESULT]])
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -1, i32 1
@@ -56,9 +83,16 @@ unreached:
 }
 
 define void @test_low_sle(i64 %a, i64 %b) {
-; CHECK-LABEL: @test_low_sle
-; CHECK: [[TMP1:%.*]] = icmp slt i64 %a, %b
-; CHECK: br i1 [[TMP1]], label %unreached, label %normal
+; CHECK-LABEL: define void @test_low_sle
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp slt i64 [[A]], [[B]]
+; CHECK-NEXT:    br i1 [[TMP1]], label [[UNREACHED:%.*]], label [[NORMAL:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    call void @use(i32 -1)
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -1, i32 1
@@ -73,9 +107,18 @@ unreached:
 }
 
 define void @test_low_ne(i64 %a, i64 %b) {
-; CHECK-LABEL: @test_low_ne
-; CHECK: [[TMP1:%.*]] = icmp slt i64 %a, %b
-; CHECK: br i1 [[TMP1]], label %normal, label %unreached
+; CHECK-LABEL: define void @test_low_ne
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp slt i64 [[A]], [[B]]
+; CHECK-NEXT:    br i1 [[TMP1]], label [[NORMAL:%.*]], label [[UNREACHED:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    [[EQ:%.*]] = icmp ne i64 [[A]], [[B]]
+; CHECK-NEXT:    [[RESULT:%.*]] = zext i1 [[EQ]] to i32
+; CHECK-NEXT:    call void @use(i32 [[RESULT]])
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -1, i32 1
@@ -90,9 +133,16 @@ unreached:
 }
 
 define void @test_low_eq(i64 %a, i64 %b) {
-; CHECK-LABEL: @test_low_eq
-; CHECK: [[TMP1:%.*]] = icmp slt i64 %a, %b
-; CHECK: br i1 [[TMP1]], label %unreached, label %normal
+; CHECK-LABEL: define void @test_low_eq
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp slt i64 [[A]], [[B]]
+; CHECK-NEXT:    br i1 [[TMP1]], label [[UNREACHED:%.*]], label [[NORMAL:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    call void @use(i32 -1)
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -1, i32 1
@@ -107,9 +157,16 @@ unreached:
 }
 
 define void @test_mid_sgt(i64 %a, i64 %b) {
-; CHECK-LABEL: @test_mid_sgt
-; CHECK: [[TMP1:%.*]] = icmp sgt i64 %a, %b
-; CHECK: br i1 [[TMP1]], label %unreached, label %normal
+; CHECK-LABEL: define void @test_mid_sgt
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp sgt i64 [[A]], [[B]]
+; CHECK-NEXT:    br i1 [[TMP1]], label [[UNREACHED:%.*]], label [[NORMAL:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    call void @use(i32 1)
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -1, i32 1
@@ -124,9 +181,16 @@ unreached:
 }
 
 define void @test_mid_slt(i64 %a, i64 %b) {
-; CHECK-LABEL: @test_mid_slt
-; CHECK: [[TMP1:%.*]] = icmp slt i64 %a, %b
-; CHECK: br i1 [[TMP1]], label %unreached, label %normal
+; CHECK-LABEL: define void @test_mid_slt
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp slt i64 [[A]], [[B]]
+; CHECK-NEXT:    br i1 [[TMP1]], label [[UNREACHED:%.*]], label [[NORMAL:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    call void @use(i32 -1)
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -1, i32 1
@@ -141,9 +205,18 @@ unreached:
 }
 
 define void @test_mid_sge(i64 %a, i64 %b) {
-; CHECK-LABEL: @test_mid_sge
-; CHECK: [[TMP1:%.*]] = icmp slt i64 %a, %b
-; CHECK: br i1 [[TMP1]], label %normal, label %unreached
+; CHECK-LABEL: define void @test_mid_sge
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    [[CMP_NOT:%.*]] = icmp slt i64 [[A]], [[B]]
+; CHECK-NEXT:    br i1 [[CMP_NOT]], label [[NORMAL:%.*]], label [[UNREACHED:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    [[EQ:%.*]] = icmp ne i64 [[A]], [[B]]
+; CHECK-NEXT:    [[RESULT:%.*]] = zext i1 [[EQ]] to i32
+; CHECK-NEXT:    call void @use(i32 [[RESULT]])
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -1, i32 1
@@ -158,9 +231,20 @@ unreached:
 }
 
 define void @test_mid_sle(i64 %a, i64 %b) {
-; CHECK-LABEL: @test_mid_sle
-; CHECK: [[TMP1:%.*]] = icmp sgt i64 %a, %b
-; CHECK: br i1 [[TMP1]], label %normal, label %unreached
+; CHECK-LABEL: define void @test_mid_sle
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    [[CMP_NOT:%.*]] = icmp sgt i64 [[A]], [[B]]
+; CHECK-NEXT:    br i1 [[CMP_NOT]], label [[NORMAL:%.*]], label [[UNREACHED:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    [[EQ:%.*]] = icmp eq i64 [[A]], [[B]]
+; CHECK-NEXT:    [[SLT:%.*]] = icmp slt i64 [[A]], [[B]]
+; CHECK-NEXT:    [[DOT:%.*]] = select i1 [[SLT]], i32 -1, i32 1
+; CHECK-NEXT:    [[RESULT:%.*]] = select i1 [[EQ]], i32 0, i32 [[DOT]]
+; CHECK-NEXT:    call void @use(i32 [[RESULT]])
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -1, i32 1
@@ -175,9 +259,18 @@ unreached:
 }
 
 define void @test_mid_ne(i64 %a, i64 %b) {
-; CHECK-LABEL: @test_mid_ne
-; CHECK: [[TMP1:%.*]] = icmp eq i64 %a, %b
-; CHECK: br i1 [[TMP1]], label %normal, label %unreached
+; CHECK-LABEL: define void @test_mid_ne
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    [[EQ:%.*]] = icmp eq i64 [[A]], [[B]]
+; CHECK-NEXT:    br i1 [[EQ]], label [[NORMAL:%.*]], label [[UNREACHED:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    [[SLT:%.*]] = icmp slt i64 [[A]], [[B]]
+; CHECK-NEXT:    [[DOT:%.*]] = select i1 [[SLT]], i32 -1, i32 1
+; CHECK-NEXT:    call void @use(i32 [[DOT]])
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -1, i32 1
@@ -192,9 +285,16 @@ unreached:
 }
 
 define void @test_mid_eq(i64 %a, i64 %b) {
-; CHECK-LABEL: @test_mid_eq
-; CHECK: icmp eq i64 %a, %b
-; CHECK: br i1 %eq, label %unreached, label %normal
+; CHECK-LABEL: define void @test_mid_eq
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    [[EQ:%.*]] = icmp eq i64 [[A]], [[B]]
+; CHECK-NEXT:    br i1 [[EQ]], label [[UNREACHED:%.*]], label [[NORMAL:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    call void @use(i32 0)
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -1, i32 1
@@ -209,8 +309,14 @@ unreached:
 }
 
 define void @test_high_sgt(i64 %a, i64 %b) {
-; CHECK-LABEL: @test_high_sgt
-; CHECK: br i1 false, label %unreached, label %normal
+; CHECK-LABEL: define void @test_high_sgt
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    br i1 false, label [[UNREACHED:%.*]], label [[NORMAL:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -1, i32 1
@@ -225,9 +331,20 @@ unreached:
 }
 
 define void @test_high_slt(i64 %a, i64 %b) {
-; CHECK-LABEL: @test_high_slt
-; CHECK: [[TMP1:%.*]] = icmp sgt i64 %a, %b
-; CHECK: br i1 [[TMP1]], label %normal, label %unreached
+; CHECK-LABEL: define void @test_high_slt
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    [[CMP_NOT:%.*]] = icmp sgt i64 [[A]], [[B]]
+; CHECK-NEXT:    br i1 [[CMP_NOT]], label [[NORMAL:%.*]], label [[UNREACHED:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    [[EQ:%.*]] = icmp eq i64 [[A]], [[B]]
+; CHECK-NEXT:    [[SLT:%.*]] = icmp slt i64 [[A]], [[B]]
+; CHECK-NEXT:    [[DOT:%.*]] = select i1 [[SLT]], i32 -1, i32 1
+; CHECK-NEXT:    [[RESULT:%.*]] = select i1 [[EQ]], i32 0, i32 [[DOT]]
+; CHECK-NEXT:    call void @use(i32 [[RESULT]])
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -1, i32 1
@@ -242,9 +359,16 @@ unreached:
 }
 
 define void @test_high_sge(i64 %a, i64 %b) {
-; CHECK-LABEL: @test_high_sge
-; CHECK: [[TMP1:%.*]] = icmp sgt i64 %a, %b
-; CHECK: br i1 [[TMP1]], label %unreached, label %normal
+; CHECK-LABEL: define void @test_high_sge
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp sgt i64 [[A]], [[B]]
+; CHECK-NEXT:    br i1 [[TMP1]], label [[UNREACHED:%.*]], label [[NORMAL:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    call void @use(i32 1)
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -1, i32 1
@@ -259,8 +383,19 @@ unreached:
 }
 
 define void @test_high_sle(i64 %a, i64 %b) {
-; CHECK-LABEL: @test_high_sle
-; CHECK: br i1 true, label %unreached, label %normal
+; CHECK-LABEL: define void @test_high_sle
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    br i1 true, label [[UNREACHED:%.*]], label [[NORMAL:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    [[EQ:%.*]] = icmp eq i64 [[A]], [[B]]
+; CHECK-NEXT:    [[SLT:%.*]] = icmp slt i64 [[A]], [[B]]
+; CHECK-NEXT:    [[DOT:%.*]] = select i1 [[SLT]], i32 -1, i32 1
+; CHECK-NEXT:    [[RESULT:%.*]] = select i1 [[EQ]], i32 0, i32 [[DOT]]
+; CHECK-NEXT:    call void @use(i32 [[RESULT]])
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -1, i32 1
@@ -275,9 +410,20 @@ unreached:
 }
 
 define void @test_high_ne(i64 %a, i64 %b) {
-; CHECK-LABEL: @test_high_ne
-; CHECK: [[TMP1:%.*]] = icmp sgt i64 %a, %b
-; CHECK: br i1 [[TMP1]], label %normal, label %unreached
+; CHECK-LABEL: define void @test_high_ne
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp sgt i64 [[A]], [[B]]
+; CHECK-NEXT:    br i1 [[TMP1]], label [[NORMAL:%.*]], label [[UNREACHED:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    [[EQ:%.*]] = icmp eq i64 [[A]], [[B]]
+; CHECK-NEXT:    [[SLT:%.*]] = icmp slt i64 [[A]], [[B]]
+; CHECK-NEXT:    [[DOT:%.*]] = select i1 [[SLT]], i32 -1, i32 1
+; CHECK-NEXT:    [[RESULT:%.*]] = select i1 [[EQ]], i32 0, i32 [[DOT]]
+; CHECK-NEXT:    call void @use(i32 [[RESULT]])
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -1, i32 1
@@ -292,9 +438,16 @@ unreached:
 }
 
 define void @test_high_eq(i64 %a, i64 %b) {
-; CHECK-LABEL: @test_high_eq
-; CHECK: [[TMP1:%.*]] = icmp sgt i64 %a, %b
-; CHECK: br i1 [[TMP1]], label %unreached, label %normal
+; CHECK-LABEL: define void @test_high_eq
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp sgt i64 [[A]], [[B]]
+; CHECK-NEXT:    br i1 [[TMP1]], label [[UNREACHED:%.*]], label [[NORMAL:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    call void @use(i32 1)
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -1, i32 1
@@ -312,9 +465,16 @@ unreached:
 ; produced values
 
 define void @non_standard_low(i64 %a, i64 %b) {
-; CHECK-LABEL: @non_standard_low
-; CHECK: [[TMP1:%.*]] = icmp slt i64 %a, %b
-; CHECK: br i1 [[TMP1]], label %unreached, label %normal
+; CHECK-LABEL: define void @non_standard_low
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp slt i64 [[A]], [[B]]
+; CHECK-NEXT:    br i1 [[TMP1]], label [[UNREACHED:%.*]], label [[NORMAL:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    call void @use(i32 -3)
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -3, i32 -1
@@ -329,9 +489,16 @@ unreached:
 }
 
 define void @non_standard_mid(i64 %a, i64 %b) {
-; CHECK-LABEL: @non_standard_mid
-; CHECK: icmp eq i64 %a, %b
-; CHECK: br i1 %eq, label %unreached, label %normal
+; CHECK-LABEL: define void @non_standard_mid
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    [[EQ:%.*]] = icmp eq i64 [[A]], [[B]]
+; CHECK-NEXT:    br i1 [[EQ]], label [[UNREACHED:%.*]], label [[NORMAL:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    call void @use(i32 -2)
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -3, i32 -1
@@ -346,9 +513,16 @@ unreached:
 }
 
 define void @non_standard_high(i64 %a, i64 %b) {
-; CHECK-LABEL: @non_standard_high
-; CHECK: [[TMP1:%.*]] = icmp sgt i64 %a, %b
-; CHECK: br i1 [[TMP1]], label %unreached, label %normal
+; CHECK-LABEL: define void @non_standard_high
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp sgt i64 [[A]], [[B]]
+; CHECK-NEXT:    br i1 [[TMP1]], label [[UNREACHED:%.*]], label [[NORMAL:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    call void @use(i32 -1)
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -3, i32 -1
@@ -363,8 +537,14 @@ unreached:
 }
 
 define void @non_standard_bound1(i64 %a, i64 %b) {
-; CHECK-LABEL: @non_standard_bound1
-; CHECK: br i1 false, label %unreached, label %normal
+; CHECK-LABEL: define void @non_standard_bound1
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    br i1 false, label [[UNREACHED:%.*]], label [[NORMAL:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -3, i32 -1
@@ -379,8 +559,14 @@ unreached:
 }
 
 define void @non_standard_bound2(i64 %a, i64 %b) {
-; CHECK-LABEL: @non_standard_bound2
-; CHECK: br i1 false, label %unreached, label %normal
+; CHECK-LABEL: define void @non_standard_bound2
+; CHECK-SAME: (i64 [[A:%.*]], i64 [[B:%.*]]) {
+; CHECK-NEXT:    br i1 false, label [[UNREACHED:%.*]], label [[NORMAL:%.*]]
+; CHECK:       normal:
+; CHECK-NEXT:    ret void
+; CHECK:       unreached:
+; CHECK-NEXT:    ret void
+;
   %eq = icmp eq i64 %a, %b
   %slt = icmp slt i64 %a, %b
   %. = select i1 %slt, i32 -3, i32 -1
diff --git a/llvm/test/Transforms/InstCombine/create-class-from-logic-fcmp.ll b/llvm/test/Transforms/InstCombine/create-class-from-logic-fcmp.ll
index 8230f9615de59..cd69bc83c6521 100644
--- a/llvm/test/Transforms/InstCombine/create-class-from-logic-fcmp.ll
+++ b/llvm/test/Transforms/InstCombine/create-class-from-logic-fcmp.ll
@@ -10,10 +10,7 @@
 ; Base pattern !isfinite(x) || x == 0.0
 define i1 @not_isfinite_or_zero_f16(half %x) {
 ; CHECK-LABEL: @not_isfinite_or_zero_f16(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp ueq half [[FABS]], 0xH7C00
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp oeq half [[X]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = or i1 [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 615)
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -26,10 +23,7 @@ define i1 @not_isfinite_or_zero_f16(half %x) {
 ; Base pattern x == 0.0 || !isfinite(x)
 define i1 @not_isfinite_or_zero_f16_commute_or(half %x) {
 ; CHECK-LABEL: @not_isfinite_or_zero_f16_commute_or(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp ueq half [[FABS]], 0xH7C00
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp oeq half [[X]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = or i1 [[CMPINF]], [[CMPZERO]]
+; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 615)
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -42,10 +36,7 @@ define i1 @not_isfinite_or_zero_f16_commute_or(half %x) {
 ; Base pattern !isfinite(x) || x == -0.0
 define i1 @not_isfinite_or_zero_f16_negzero(half %x) {
 ; CHECK-LABEL: @not_isfinite_or_zero_f16_negzero(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp ueq half [[FABS]], 0xH7C00
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp oeq half [[X]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = or i1 [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 615)
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -57,10 +48,7 @@ define i1 @not_isfinite_or_zero_f16_negzero(half %x) {
 
 define i1 @not_isfinite_or_fabs_oeq_zero_f16(half %x) {
 ; CHECK-LABEL: @not_isfinite_or_fabs_oeq_zero_f16(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp ueq half [[FABS]], 0xH7C00
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp oeq half [[X]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = or i1 [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 615)
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -73,10 +61,7 @@ define i1 @not_isfinite_or_fabs_oeq_zero_f16(half %x) {
 ; Base pattern !isfinite(x) || x == 0.0
 define <2 x i1> @not_isfinite_or_zero_v2f16(<2 x half> %x) {
 ; CHECK-LABEL: @not_isfinite_or_zero_v2f16(
-; CHECK-NEXT:    [[FABS:%.*]] = call <2 x half> @llvm.fabs.v2f16(<2 x half> [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp ueq <2 x half> [[FABS]], <half 0xH7C00, half 0xH7C00>
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp oeq <2 x half> [[X]], zeroinitializer
-; CHECK-NEXT:    [[CLASS:%.*]] = or <2 x i1> [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[CLASS:%.*]] = call <2 x i1> @llvm.is.fpclass.v2f16(<2 x half> [[X:%.*]], i32 615)
 ; CHECK-NEXT:    ret <2 x i1> [[CLASS]]
 ;
   %fabs = call <2 x half> @llvm.fabs.v2f16(<2 x half> %x)
@@ -89,10 +74,7 @@ define <2 x i1> @not_isfinite_or_zero_v2f16(<2 x half> %x) {
 ; Base pattern !isfinite(x) || x == <0.0, -0.0>
 define <2 x i1> @not_isfinite_or_zero_v2f16_pos0_neg0_vec(<2 x half> %x) {
 ; CHECK-LABEL: @not_isfinite_or_zero_v2f16_pos0_neg0_vec(
-; CHECK-NEXT:    [[FABS:%.*]] = call <2 x half> @llvm.fabs.v2f16(<2 x half> [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp ueq <2 x half> [[FABS]], <half 0xH7C00, half 0xH7C00>
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp oeq <2 x half> [[X]], zeroinitializer
-; CHECK-NEXT:    [[CLASS:%.*]] = or <2 x i1> [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[CLASS:%.*]] = call <2 x i1> @llvm.is.fpclass.v2f16(<2 x half> [[X:%.*]], i32 615)
 ; CHECK-NEXT:    ret <2 x i1> [[CLASS]]
 ;
   %fabs = call <2 x half> @llvm.fabs.v2f16(<2 x half> %x)
@@ -105,10 +87,7 @@ define <2 x i1> @not_isfinite_or_zero_v2f16_pos0_neg0_vec(<2 x half> %x) {
 ; Base pattern x == 0.0 || !isfinite(x)
 define <2 x i1> @not_isfinite_or_zero_v2f16_commute_or(<2 x half> %x) {
 ; CHECK-LABEL: @not_isfinite_or_zero_v2f16_commute_or(
-; CHECK-NEXT:    [[FABS:%.*]] = call <2 x half> @llvm.fabs.v2f16(<2 x half> [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp ueq <2 x half> [[FABS]], <half 0xH7C00, half 0xH7C00>
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp oeq <2 x half> [[X]], zeroinitializer
-; CHECK-NEXT:    [[CLASS:%.*]] = or <2 x i1> [[CMPINF]], [[CMPZERO]]
+; CHECK-NEXT:    [[CLASS:%.*]] = call <2 x i1> @llvm.is.fpclass.v2f16(<2 x half> [[X:%.*]], i32 615)
 ; CHECK-NEXT:    ret <2 x i1> [[CLASS]]
 ;
   %fabs = call <2 x half> @llvm.fabs.v2f16(<2 x half> %x)
@@ -121,10 +100,7 @@ define <2 x i1> @not_isfinite_or_zero_v2f16_commute_or(<2 x half> %x) {
 ; Positive test
 define i1 @oeq_isinf_or_oeq_zero(half %x) {
 ; CHECK-LABEL: @oeq_isinf_or_oeq_zero(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp oeq half [[FABS]], 0xH7C00
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp oeq half [[X]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = or i1 [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 612)
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -137,9 +113,7 @@ define i1 @oeq_isinf_or_oeq_zero(half %x) {
 ; Missing fabs for infinity check
 define i1 @ueq_inf_or_oeq_zero(half %x) {
 ; CHECK-LABEL: @ueq_inf_or_oeq_zero(
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp ueq half [[X:%.*]], 0xH7C00
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp oeq half [[X]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = or i1 [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 611)
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %cmpinf = fcmp ueq half %x, 0xH7C00
@@ -151,10 +125,7 @@ define i1 @ueq_inf_or_oeq_zero(half %x) {
 ; Extra fabs.
 define i1 @oeq_isinf_or_fabs_oeq_zero(half %x) {
 ; CHECK-LABEL: @oeq_isinf_or_fabs_oeq_zero(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp oeq half [[FABS]], 0xH7C00
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp oeq half [[X]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = or i1 [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 612)
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -167,9 +138,7 @@ define i1 @oeq_isinf_or_fabs_oeq_zero(half %x) {
 ; Positive test
 define i1 @ueq_0_or_oeq_inf(half %x) {
 ; CHECK-LABEL: @ueq_0_or_oeq_inf(
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp ueq half [[X:%.*]], 0xH0000
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp oeq half [[X]], 0xH7C00
-; CHECK-NEXT:    [[CLASS:%.*]] = or i1 [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 611)
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -194,10 +163,7 @@ define i1 @not_isfinite_or_zero_f16_not_inf(half %x) {
 ; Positive test
 define i1 @ueq_inf_or_ueq_zero(half %x) {
 ; CHECK-LABEL: @ueq_inf_or_ueq_zero(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp ueq half [[FABS]], 0xH7C00
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp ueq half [[X]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = or i1 [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 615)
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -210,11 +176,7 @@ define i1 @ueq_inf_or_ueq_zero(half %x) {
 ; Positive test
 define i1 @not_isfinite_and_zero_f16(half %x) {
 ; CHECK-LABEL: @not_isfinite_and_zero_f16(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp ueq half [[FABS]], 0xH7C00
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp oeq half [[X]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = and i1 [[CMPZERO]], [[CMPINF]]
-; CHECK-NEXT:    ret i1 [[CLASS]]
+; CHECK-NEXT:    ret i1 false
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
   %cmpinf = fcmp ueq half %fabs, 0xH7C00
@@ -262,10 +224,7 @@ define i1 @not_isfinite_or_zero_f16_multi_use_cmp1(half %x, ptr %ptr) {
 ; Negative test
 define i1 @not_isfinite_or_zero_f16_neg_inf(half %x) {
 ; CHECK-LABEL: @not_isfinite_or_zero_f16_neg_inf(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp ueq half [[FABS]], 0xHFC00
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp oeq half [[X]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = or i1 [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp ueq half [[X:%.*]], 0xH0000
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -294,10 +253,7 @@ define i1 @olt_0_or_fabs_ueq_inf(half %x) {
 ; Negative test
 define i1 @oeq_0_or_fabs_ult_inf(half %x) {
 ; CHECK-LABEL: @oeq_0_or_fabs_ult_inf(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp ult half [[FABS]], 0xH7C00
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp oeq half [[X]], 0xH0000
-; CHECK-NEXT:    [[CLASS:%.*]] = or i1 [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 507)
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -362,10 +318,7 @@ define i1 @not_isfinite_or_zero_f16_not_fabs(half %x) {
 ; Negation of base pattern, isfinite(x) && !(x == 0.0)
 define i1 @negated_isfinite_or_zero_f16(half %x) {
 ; CHECK-LABEL: @negated_isfinite_or_zero_f16(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp one half [[FABS]], 0xH7C00
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp une half [[X]], 0xH0000
-; CHECK-NEXT:    [[NOT_CLASS:%.*]] = and i1 [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[NOT_CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 408)
 ; CHECK-NEXT:    ret i1 [[NOT_CLASS]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -378,10 +331,7 @@ define i1 @negated_isfinite_or_zero_f16(half %x) {
 ; Commuted !(x == 0.0) && isfinite(x)
 define i1 @negated_isfinite_or_zero_f16_commute_and(half %x) {
 ; CHECK-LABEL: @negated_isfinite_or_zero_f16_commute_and(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp one half [[FABS]], 0xH7C00
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp une half [[X]], 0xH0000
-; CHECK-NEXT:    [[NOT_CLASS:%.*]] = and i1 [[CMPINF]], [[CMPZERO]]
+; CHECK-NEXT:    [[NOT_CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 408)
 ; CHECK-NEXT:    ret i1 [[NOT_CLASS]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -394,10 +344,7 @@ define i1 @negated_isfinite_or_zero_f16_commute_and(half %x) {
 ; isfinite(x) && !(x == -0.0)
 define i1 @negated_isfinite_or_zero_f16_negzero(half %x) {
 ; CHECK-LABEL: @negated_isfinite_or_zero_f16_negzero(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp one half [[FABS]], 0xH7C00
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp une half [[X]], 0xH0000
-; CHECK-NEXT:    [[NOT_CLASS:%.*]] = and i1 [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[NOT_CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 408)
 ; CHECK-NEXT:    ret i1 [[NOT_CLASS]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -410,10 +357,7 @@ define i1 @negated_isfinite_or_zero_f16_negzero(half %x) {
 ; Negated pattern
 define <2 x i1> @negated_isfinite_or_zero_v2f16(<2 x half> %x) {
 ; CHECK-LABEL: @negated_isfinite_or_zero_v2f16(
-; CHECK-NEXT:    [[FABS:%.*]] = call <2 x half> @llvm.fabs.v2f16(<2 x half> [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp one <2 x half> [[FABS]], <half 0xH7C00, half 0xH7C00>
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp une <2 x half> [[X]], zeroinitializer
-; CHECK-NEXT:    [[NOT_CLASS:%.*]] = and <2 x i1> [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[NOT_CLASS:%.*]] = call <2 x i1> @llvm.is.fpclass.v2f16(<2 x half> [[X:%.*]], i32 408)
 ; CHECK-NEXT:    ret <2 x i1> [[NOT_CLASS]]
 ;
   %fabs = call <2 x half> @llvm.fabs.v2f16(<2 x half> %x)
@@ -426,10 +370,7 @@ define <2 x i1> @negated_isfinite_or_zero_v2f16(<2 x half> %x) {
 ; Negated pattern, commuted vector and
 define <2 x i1> @negated_isfinite_or_zero_v2f16_comumte(<2 x half> %x) {
 ; CHECK-LABEL: @negated_isfinite_or_zero_v2f16_comumte(
-; CHECK-NEXT:    [[FABS:%.*]] = call <2 x half> @llvm.fabs.v2f16(<2 x half> [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp one <2 x half> [[FABS]], <half 0xH7C00, half 0xH7C00>
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp une <2 x half> [[X]], zeroinitializer
-; CHECK-NEXT:    [[NOT_CLASS:%.*]] = and <2 x i1> [[CMPINF]], [[CMPZERO]]
+; CHECK-NEXT:    [[NOT_CLASS:%.*]] = call <2 x i1> @llvm.is.fpclass.v2f16(<2 x half> [[X:%.*]], i32 408)
 ; CHECK-NEXT:    ret <2 x i1> [[NOT_CLASS]]
 ;
   %fabs = call <2 x half> @llvm.fabs.v2f16(<2 x half> %x)
@@ -442,10 +383,7 @@ define <2 x i1> @negated_isfinite_or_zero_v2f16_comumte(<2 x half> %x) {
 ; Positive test
 define i1 @negated_isfinite_or_zero_f16_not_une_zero(half %x) {
 ; CHECK-LABEL: @negated_isfinite_or_zero_f16_not_une_zero(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp one half [[FABS]], 0xH7C00
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp one half [[X]], 0xH0000
-; CHECK-NEXT:    [[NOT_CLASS:%.*]] = and i1 [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[NOT_CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 408)
 ; CHECK-NEXT:    ret i1 [[NOT_CLASS]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -458,11 +396,7 @@ define i1 @negated_isfinite_or_zero_f16_not_une_zero(half %x) {
 ; Positive test
 define i1 @negated_isfinite_and_zero_f16(half %x) {
 ; CHECK-LABEL: @negated_isfinite_and_zero_f16(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp one half [[FABS]], 0xH7C00
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp une half [[X]], 0xH0000
-; CHECK-NEXT:    [[NOT_CLASS:%.*]] = or i1 [[CMPZERO]], [[CMPINF]]
-; CHECK-NEXT:    ret i1 [[NOT_CLASS]]
+; CHECK-NEXT:    ret i1 true
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
   %cmpinf = fcmp one half %fabs, 0xH7C00
@@ -474,9 +408,7 @@ define i1 @negated_isfinite_and_zero_f16(half %x) {
 ; Negative test
 define i1 @negated_isfinite_or_zero_f16_swapped_constants(half %x) {
 ; CHECK-LABEL: @negated_isfinite_or_zero_f16_swapped_constants(
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp one half [[X:%.*]], 0xH0000
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp une half [[X]], 0xH7C00
-; CHECK-NEXT:    [[NOT_CLASS:%.*]] = and i1 [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[NOT_CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 412)
 ; CHECK-NEXT:    ret i1 [[NOT_CLASS]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -525,10 +457,7 @@ define i1 @negated_isfinite_or_zero_f16_multi_use_cmp1(half %x, ptr %ptr) {
 ; Negative test
 define i1 @negated_isfinite_or_zero_f16_multi_use_cmp0_not_one_inf(half %x) {
 ; CHECK-LABEL: @negated_isfinite_or_zero_f16_multi_use_cmp0_not_one_inf(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp une half [[FABS]], 0xH7C00
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp une half [[X]], 0xH0000
-; CHECK-NEXT:    [[NOT_CLASS:%.*]] = and i1 [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[NOT_CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 411)
 ; CHECK-NEXT:    ret i1 [[NOT_CLASS]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -560,10 +489,7 @@ define i1 @negated_isfinite_or_zero_f16_fabs_wrong_value(half %x, half %y) {
 
 define i1 @fcmp_une_0_or_fcmp_une_inf(half %x) {
 ; CHECK-LABEL: @fcmp_une_0_or_fcmp_une_inf(
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp une half [[X:%.*]], 0xH0000
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp une half [[X]], 0xH7C00
-; CHECK-NEXT:    [[OR:%.*]] = or i1 [[CMPZERO]], [[CMPINF]]
-; CHECK-NEXT:    ret i1 [[OR]]
+; CHECK-NEXT:    ret i1 true
 ;
   %cmpzero = fcmp une half %x, 0.0
   %cmpinf = fcmp une half %x, 0xH7C00
@@ -573,10 +499,7 @@ define i1 @fcmp_une_0_or_fcmp_une_inf(half %x) {
 
 define i1 @fcmp_one_0_and_fcmp_une_fabs_inf(half %x) {
 ; CHECK-LABEL: @fcmp_one_0_and_fcmp_une_fabs_inf(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp one half [[X]], 0xH0000
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp une half [[FABS]], 0xH7C00
-; CHECK-NEXT:    [[AND:%.*]] = and i1 [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[AND:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 408)
 ; CHECK-NEXT:    ret i1 [[AND]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -588,10 +511,7 @@ define i1 @fcmp_one_0_and_fcmp_une_fabs_inf(half %x) {
 
 define i1 @fcmp_une_0_and_fcmp_une_fabs_inf(half %x) {
 ; CHECK-LABEL: @fcmp_une_0_and_fcmp_une_fabs_inf(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp une half [[X]], 0xH0000
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp une half [[FABS]], 0xH7C00
-; CHECK-NEXT:    [[AND:%.*]] = and i1 [[CMPZERO]], [[CMPINF]]
+; CHECK-NEXT:    [[AND:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 411)
 ; CHECK-NEXT:    ret i1 [[AND]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -603,10 +523,7 @@ define i1 @fcmp_une_0_and_fcmp_une_fabs_inf(half %x) {
 
 define i1 @fcmp_une_0_and_fcmp_une_neginf(half %x) {
 ; CHECK-LABEL: @fcmp_une_0_and_fcmp_une_neginf(
-; CHECK-NEXT:    [[CMPZERO:%.*]] = fcmp une half [[X:%.*]], 0xH0000
-; CHECK-NEXT:    [[CMPINF:%.*]] = fcmp une half [[X]], 0xHFC00
-; CHECK-NEXT:    [[OR:%.*]] = or i1 [[CMPZERO]], [[CMPINF]]
-; CHECK-NEXT:    ret i1 [[OR]]
+; CHECK-NEXT:    ret i1 true
 ;
   %cmpzero = fcmp une half %x, 0.0
   %cmpinf = fcmp une half %x, 0xHFC00
@@ -746,10 +663,7 @@ define i1 @issubnormal_and_finite_olt(half %x) {
 
 define i1 @not_zero_and_subnormal(half %x) {
 ; CHECK-LABEL: @not_zero_and_subnormal(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[CMP_ZERO:%.*]] = fcmp one half [[X]], 0xH0000
-; CHECK-NEXT:    [[CMP_SMALLEST_NORMAL:%.*]] = fcmp olt half [[FABS]], 0xH0400
-; CHECK-NEXT:    [[OR:%.*]] = or i1 [[CMP_SMALLEST_NORMAL]], [[CMP_ZERO]]
+; CHECK-NEXT:    [[OR:%.*]] = fcmp ord half [[X:%.*]], 0xH0000
 ; CHECK-NEXT:    ret i1 [[OR]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -877,10 +791,8 @@ define i1 @oeq_isinf_and_ord(half %x) {
 define i1 @isnormal_or_zero(half %x) #0 {
 ; CHECK-LABEL: @isnormal_or_zero(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[AND1:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 264)
-; CHECK-NEXT:    [[CMP:%.*]] = fcmp oeq half [[X]], 0xH0000
-; CHECK-NEXT:    [[SPEC_SELECT:%.*]] = or i1 [[CMP]], [[AND1]]
-; CHECK-NEXT:    ret i1 [[SPEC_SELECT]]
+; CHECK-NEXT:    [[AND1:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 360)
+; CHECK-NEXT:    ret i1 [[AND1]]
 ;
 entry:
   %iseq = fcmp ord half %x, 0xH0000
@@ -897,10 +809,7 @@ entry:
 define i1 @isnormal_uge_or_zero_oeq(half %x) #0 {
 ; CHECK-LABEL: @isnormal_uge_or_zero_oeq(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[FABS:%.*]] = tail call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[IS_NORMAL:%.*]] = fcmp uge half [[FABS]], 0xH0400
-; CHECK-NEXT:    [[IS_ZERO:%.*]] = fcmp oeq half [[X]], 0xH0000
-; CHECK-NEXT:    [[OR:%.*]] = or i1 [[IS_NORMAL]], [[IS_ZERO]]
+; CHECK-NEXT:    [[OR:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 879)
 ; CHECK-NEXT:    ret i1 [[OR]]
 ;
 entry:
@@ -945,9 +854,7 @@ define i1 @ord_or_isnormalinf(half %x) #0 {
 ; -> iszero
 define i1 @une_or_oge_smallest_normal(half %x) #0 {
 ; CHECK-LABEL: @une_or_oge_smallest_normal(
-; CHECK-NEXT:    [[IS_NORMAL_INF:%.*]] = fcmp oge half [[X:%.*]], 0xH0400
-; CHECK-NEXT:    [[IS_UNE:%.*]] = fcmp une half [[X]], 0xH0000
-; CHECK-NEXT:    [[OR:%.*]] = or i1 [[IS_UNE]], [[IS_NORMAL_INF]]
+; CHECK-NEXT:    [[OR:%.*]] = fcmp une half [[X:%.*]], 0xH0000
 ; CHECK-NEXT:    ret i1 [[OR]]
 ;
   %is.normal.inf = fcmp oge half %x, 0xH0400
@@ -1011,7 +918,7 @@ define i1 @isnormalinf_and_inf(half %x) #0 {
 ; -> pinf
 define i1 @posisnormalinf_and_posinf(half %x) #0 {
 ; CHECK-LABEL: @posisnormalinf_and_posinf(
-; CHECK-NEXT:    [[AND:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 512)
+; CHECK-NEXT:    [[AND:%.*]] = fcmp oeq half [[X:%.*]], 0xH7C00
 ; CHECK-NEXT:    ret i1 [[AND]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -1024,7 +931,7 @@ define i1 @posisnormalinf_and_posinf(half %x) #0 {
 ; -> pinf
 define i1 @isnormalinf_and_posinf(half %x) #0 {
 ; CHECK-LABEL: @isnormalinf_and_posinf(
-; CHECK-NEXT:    [[AND:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 512)
+; CHECK-NEXT:    [[AND:%.*]] = fcmp oeq half [[X:%.*]], 0xH7C00
 ; CHECK-NEXT:    ret i1 [[AND]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -1166,7 +1073,7 @@ define i1 @olt_smallest_normal_or_uno(half %x) #0 {
 
 define i1 @olt_smallest_normal_or_finite(half %x) #0 {
 ; CHECK-LABEL: @olt_smallest_normal_or_finite(
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 508)
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp one half [[X:%.*]], 0xH7C00
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
@@ -1213,7 +1120,7 @@ define i1 @uge_smallest_normal_and_uno(half %x) #0 {
 ; -> true
 define i1 @olt_infinity_or_finite(half %x) #0 {
 ; CHECK-LABEL: @olt_infinity_or_finite(
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 508)
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp one half [[X:%.*]], 0xH7C00
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %lt.infinity = fcmp olt half %x, 0xH7C00
@@ -1249,7 +1156,7 @@ define i1 @olt_infinity_or_ord(half %x) #0 {
 ; -> ~posinf
 define i1 @olt_infinity_or_uno(half %x) #0 {
 ; CHECK-LABEL: @olt_infinity_or_uno(
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 511)
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp une half [[X:%.*]], 0xH7C00
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %lt.infinity = fcmp olt half %x, 0xH7C00
@@ -1260,7 +1167,7 @@ define i1 @olt_infinity_or_uno(half %x) #0 {
 
 define i1 @olt_infinity_or_subnormal(half %x) #0 {
 ; CHECK-LABEL: @olt_infinity_or_subnormal(
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 508)
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp one half [[X:%.*]], 0xH7C00
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %lt.infinity = fcmp olt half %x, 0xH7C00
@@ -1298,7 +1205,7 @@ define i1 @olt_infinity_and_not_subnormal(half %x) #0 {
 ; -> ninf
 define i1 @olt_infinity_and_ueq_inf(half %x) #0 {
 ; CHECK-LABEL: @olt_infinity_and_ueq_inf(
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 4)
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp oeq half [[X:%.*]], 0xHFC00
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %lt.infinity = fcmp olt half %x, 0xH7C00
@@ -1334,7 +1241,7 @@ define i1 @olt_smallest_normal_or_ueq_inf(half %x) #0 {
 ; -> ~pinf
 define i1 @olt_smallest_normal_or_une_inf(half %x) #0 {
 ; CHECK-LABEL: @olt_smallest_normal_or_une_inf(
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 511)
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp une half [[X:%.*]], 0xH7C00
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %lt.normal = fcmp olt half %x, 0xH0400
@@ -1395,7 +1302,7 @@ define i1 @oge_fabs_eq_inf_and_ord(half %x) #0 {
 
 define i1 @oge_eq_inf_and_ord(half %x) #0 {
 ; CHECK-LABEL: @oge_eq_inf_and_ord(
-; CHECK-NEXT:    [[AND:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 512)
+; CHECK-NEXT:    [[AND:%.*]] = fcmp oeq half [[X:%.*]], 0xH7C00
 ; CHECK-NEXT:    ret i1 [[AND]]
 ;
   %oge.fabs.inf = fcmp oge half %x, 0xH7C00
@@ -1418,7 +1325,7 @@ define i1 @oge_fabs_eq_inf_or_uno(half %x) #0 {
 
 define i1 @oge_eq_inf_or_uno(half %x) #0 {
 ; CHECK-LABEL: @oge_eq_inf_or_uno(
-; CHECK-NEXT:    [[OR:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 515)
+; CHECK-NEXT:    [[OR:%.*]] = fcmp ueq half [[X:%.*]], 0xH7C00
 ; CHECK-NEXT:    ret i1 [[OR]]
 ;
   %oge.fabs.inf = fcmp oge half %x, 0xH7C00
@@ -1465,7 +1372,7 @@ define i1 @ult_fabs_eq_inf_or_uno(half %x) #0 {
 
 define i1 @ult_eq_inf_or_uno(half %x) #0 {
 ; CHECK-LABEL: @ult_eq_inf_or_uno(
-; CHECK-NEXT:    [[OR:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 511)
+; CHECK-NEXT:    [[OR:%.*]] = fcmp une half [[X:%.*]], 0xH7C00
 ; CHECK-NEXT:    ret i1 [[OR]]
 ;
   %ult.fabs.inf = fcmp ult half %x, 0xH7C00
@@ -1504,7 +1411,7 @@ define i1 @oeq_neginfinity_or_olt_smallest_normal(half %x) #0 {
 ; -> ninf
 define i1 @oeq_neginfinity_and_olt_smallest_normal(half %x) #0 {
 ; CHECK-LABEL: @oeq_neginfinity_and_olt_smallest_normal(
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 4)
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp oeq half [[X:%.*]], 0xHFC00
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %oeq.neg.infinity = fcmp oeq half %x, 0xHFC00
@@ -1551,7 +1458,7 @@ define i1 @oeq_neginfinity_or_ord(half %x) #0 {
 ; -> ninf
 define i1 @oeq_neginfinity_and_ord(half %x) #0 {
 ; CHECK-LABEL: @oeq_neginfinity_and_ord(
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 4)
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp oeq half [[X:%.*]], 0xHFC00
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %oeq.neg.infinity = fcmp oeq half %x, 0xHFC00
@@ -1624,7 +1531,7 @@ define i1 @one_neginfinity_and_olt_smallest_normal(half %x) #0 {
 ; -> ~ninf
 define i1 @one_neginfinity_or_uno(half %x) #0 {
 ; CHECK-LABEL: @one_neginfinity_or_uno(
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 1019)
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp une half [[X:%.*]], 0xHFC00
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %one.neg.infinity = fcmp one half %x, 0xHFC00
@@ -1671,7 +1578,7 @@ define i1 @ueq_neginfinity_or_olt_smallest_normal(half %x) #0 {
 ; -> ninf
 define i1 @ueq_neginfinity_and_olt_smallest_normal(half %x) #0 {
 ; CHECK-LABEL: @ueq_neginfinity_and_olt_smallest_normal(
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 4)
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp oeq half [[X:%.*]], 0xHFC00
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %ueq.neg.infinity = fcmp ueq half %x, 0xHFC00
@@ -1683,7 +1590,7 @@ define i1 @ueq_neginfinity_and_olt_smallest_normal(half %x) #0 {
 ; -> nan|ninf
 define i1 @ueq_neginfinity_or_uno(half %x) #0 {
 ; CHECK-LABEL: @ueq_neginfinity_or_uno(
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 7)
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp ueq half [[X:%.*]], 0xHFC00
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %ueq.neg.infinity = fcmp ueq half %x, 0xHFC00
@@ -1938,10 +1845,7 @@ define i1 @not_zero_and_subnormal_dynamic(half %x) #2 {
 ; TODO: This could fold to just fcmp olt half %fabs, 0xH0400
 define i1 @subnormal_or_zero_ieee(half %x) #0 {
 ; CHECK-LABEL: @subnormal_or_zero_ieee(
-; CHECK-NEXT:    [[FABS:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    [[IS_SUBNORMAL:%.*]] = fcmp olt half [[FABS]], 0xH0400
-; CHECK-NEXT:    [[IS_ZERO:%.*]] = fcmp oeq half [[X]], 0xH0000
-; CHECK-NEXT:    [[AND:%.*]] = or i1 [[IS_SUBNORMAL]], [[IS_ZERO]]
+; CHECK-NEXT:    [[AND:%.*]] = call i1 @llvm.is.fpclass.f16(half [[X:%.*]], i32 240)
 ; CHECK-NEXT:    ret i1 [[AND]]
 ;
   %fabs = call half @llvm.fabs.f16(half %x)
diff --git a/llvm/test/Transforms/InstCombine/ctpop.ll b/llvm/test/Transforms/InstCombine/ctpop.ll
index 863b9a88c41bd..cf1d72383e694 100644
--- a/llvm/test/Transforms/InstCombine/ctpop.ll
+++ b/llvm/test/Transforms/InstCombine/ctpop.ll
@@ -72,14 +72,9 @@ define i1 @test5(i32 %arg) {
 
 ; Test when the number of possible known bits isn't one less than a power of 2
 ; and the compare value is greater but less than the next power of 2.
-; TODO: The icmp is unnecessary given the known bits of the input, but range
-; metadata doesn't support vectors
 define <2 x i1> @test5vec(<2 x i32> %arg) {
 ; CHECK-LABEL: @test5vec(
-; CHECK-NEXT:    [[AND:%.*]] = and <2 x i32> [[ARG:%.*]], <i32 3, i32 3>
-; CHECK-NEXT:    [[CNT:%.*]] = call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[AND]])
-; CHECK-NEXT:    [[RES:%.*]] = icmp eq <2 x i32> [[CNT]], <i32 3, i32 3>
-; CHECK-NEXT:    ret <2 x i1> [[RES]]
+; CHECK-NEXT:    ret <2 x i1> zeroinitializer
 ;
   %and = and <2 x i32> %arg, <i32 3, i32 3>
   %cnt = call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> %and)
@@ -150,7 +145,7 @@ define i7 @_parity_of_not_odd_type(i7 %x) {
 
 define <2 x i32> @_parity_of_not_vec(<2 x i32> %x) {
 ; CHECK-LABEL: @_parity_of_not_vec(
-; CHECK-NEXT:    [[TMP1:%.*]] = call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[X:%.*]])
+; CHECK-NEXT:    [[TMP1:%.*]] = call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[X:%.*]]), !range [[RNG1]]
 ; CHECK-NEXT:    [[R:%.*]] = and <2 x i32> [[TMP1]], <i32 1, i32 1>
 ; CHECK-NEXT:    ret <2 x i32> [[R]]
 ;
@@ -162,7 +157,7 @@ define <2 x i32> @_parity_of_not_vec(<2 x i32> %x) {
 
 define <2 x i32> @_parity_of_not_undef(<2 x i32> %x) {
 ; CHECK-LABEL: @_parity_of_not_undef(
-; CHECK-NEXT:    [[TMP1:%.*]] = call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[X:%.*]])
+; CHECK-NEXT:    [[TMP1:%.*]] = call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[X:%.*]]), !range [[RNG1]]
 ; CHECK-NEXT:    [[R:%.*]] = and <2 x i32> [[TMP1]], <i32 1, i32 1>
 ; CHECK-NEXT:    ret <2 x i32> [[R]]
 ;
@@ -175,7 +170,7 @@ define <2 x i32> @_parity_of_not_undef(<2 x i32> %x) {
 define <2 x i32> @_parity_of_not_undef2(<2 x i32> %x) {
 ; CHECK-LABEL: @_parity_of_not_undef2(
 ; CHECK-NEXT:    [[NEG:%.*]] = xor <2 x i32> [[X:%.*]], <i32 -1, i32 -1>
-; CHECK-NEXT:    [[CNT:%.*]] = tail call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[NEG]])
+; CHECK-NEXT:    [[CNT:%.*]] = tail call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[NEG]]), !range [[RNG1]]
 ; CHECK-NEXT:    [[R:%.*]] = and <2 x i32> [[CNT]], <i32 1, i32 undef>
 ; CHECK-NEXT:    ret <2 x i32> [[R]]
 ;
@@ -206,8 +201,8 @@ define i32 @ctpop_add(i32 %a, i32 %b) {
 define i32 @ctpop_add_no_common_bits(i32 %a, i32 %b) {
 ; CHECK-LABEL: @ctpop_add_no_common_bits(
 ; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.fshl.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 16)
-; CHECK-NEXT:    [[TMP2:%.*]] = call i32 @llvm.ctpop.i32(i32 [[TMP1]]), !range [[RNG1]]
-; CHECK-NEXT:    ret i32 [[TMP2]]
+; CHECK-NEXT:    [[RES:%.*]] = call i32 @llvm.ctpop.i32(i32 [[TMP1]]), !range [[RNG1]]
+; CHECK-NEXT:    ret i32 [[RES]]
 ;
   %shl16 = shl i32 %a, 16
   %ctpop1 = tail call i32 @llvm.ctpop.i32(i32 %shl16)
@@ -220,8 +215,8 @@ define i32 @ctpop_add_no_common_bits(i32 %a, i32 %b) {
 define <2 x i32> @ctpop_add_no_common_bits_vec(<2 x i32> %a, <2 x i32> %b) {
 ; CHECK-LABEL: @ctpop_add_no_common_bits_vec(
 ; CHECK-NEXT:    [[TMP1:%.*]] = call <2 x i32> @llvm.fshl.v2i32(<2 x i32> [[A:%.*]], <2 x i32> [[B:%.*]], <2 x i32> <i32 16, i32 16>)
-; CHECK-NEXT:    [[TMP2:%.*]] = call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[TMP1]])
-; CHECK-NEXT:    ret <2 x i32> [[TMP2]]
+; CHECK-NEXT:    [[RES:%.*]] = call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[TMP1]]), !range [[RNG1]]
+; CHECK-NEXT:    ret <2 x i32> [[RES]]
 ;
   %shl16 = shl <2 x i32> %a, <i32 16, i32 16>
   %ctpop1 = tail call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> %shl16)
@@ -234,9 +229,9 @@ define <2 x i32> @ctpop_add_no_common_bits_vec(<2 x i32> %a, <2 x i32> %b) {
 define <2 x i32> @ctpop_add_no_common_bits_vec_use(<2 x i32> %a, <2 x i32> %b, ptr %p) {
 ; CHECK-LABEL: @ctpop_add_no_common_bits_vec_use(
 ; CHECK-NEXT:    [[SHL16:%.*]] = shl <2 x i32> [[A:%.*]], <i32 16, i32 16>
-; CHECK-NEXT:    [[CTPOP1:%.*]] = tail call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[SHL16]])
+; CHECK-NEXT:    [[CTPOP1:%.*]] = tail call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[SHL16]]), !range [[RNG3:![0-9]+]]
 ; CHECK-NEXT:    [[LSHL16:%.*]] = lshr <2 x i32> [[B:%.*]], <i32 16, i32 16>
-; CHECK-NEXT:    [[CTPOP2:%.*]] = tail call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[LSHL16]])
+; CHECK-NEXT:    [[CTPOP2:%.*]] = tail call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[LSHL16]]), !range [[RNG3]]
 ; CHECK-NEXT:    store <2 x i32> [[CTPOP2]], ptr [[P:%.*]], align 8
 ; CHECK-NEXT:    [[RES:%.*]] = add nuw nsw <2 x i32> [[CTPOP1]], [[CTPOP2]]
 ; CHECK-NEXT:    ret <2 x i32> [[RES]]
@@ -253,10 +248,10 @@ define <2 x i32> @ctpop_add_no_common_bits_vec_use(<2 x i32> %a, <2 x i32> %b, p
 define <2 x i32> @ctpop_add_no_common_bits_vec_use2(<2 x i32> %a, <2 x i32> %b, ptr %p) {
 ; CHECK-LABEL: @ctpop_add_no_common_bits_vec_use2(
 ; CHECK-NEXT:    [[SHL16:%.*]] = shl <2 x i32> [[A:%.*]], <i32 16, i32 16>
-; CHECK-NEXT:    [[CTPOP1:%.*]] = tail call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[SHL16]])
+; CHECK-NEXT:    [[CTPOP1:%.*]] = tail call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[SHL16]]), !range [[RNG3]]
 ; CHECK-NEXT:    store <2 x i32> [[CTPOP1]], ptr [[P:%.*]], align 8
 ; CHECK-NEXT:    [[LSHL16:%.*]] = lshr <2 x i32> [[B:%.*]], <i32 16, i32 16>
-; CHECK-NEXT:    [[CTPOP2:%.*]] = tail call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[LSHL16]])
+; CHECK-NEXT:    [[CTPOP2:%.*]] = tail call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[LSHL16]]), !range [[RNG3]]
 ; CHECK-NEXT:    [[RES:%.*]] = add nuw nsw <2 x i32> [[CTPOP1]], [[CTPOP2]]
 ; CHECK-NEXT:    ret <2 x i32> [[RES]]
 ;
@@ -295,8 +290,8 @@ declare i8 @llvm.fshr.i8(i8, i8, i8)
 define i8 @sub_ctpop(i8 %a)  {
 ; CHECK-LABEL: @sub_ctpop(
 ; CHECK-NEXT:    [[TMP1:%.*]] = xor i8 [[A:%.*]], -1
-; CHECK-NEXT:    [[TMP2:%.*]] = call i8 @llvm.ctpop.i8(i8 [[TMP1]]), !range [[RNG0]]
-; CHECK-NEXT:    ret i8 [[TMP2]]
+; CHECK-NEXT:    [[RES:%.*]] = call i8 @llvm.ctpop.i8(i8 [[TMP1]]), !range [[RNG0]]
+; CHECK-NEXT:    ret i8 [[RES]]
 ;
   %cnt = tail call i8 @llvm.ctpop.i8(i8 %a)
   %res = sub i8 8, %cnt
@@ -328,8 +323,8 @@ define i8 @sub_ctpop_unknown(i8 %a, i8 %b)  {
 define <2 x i32> @sub_ctpop_vec(<2 x i32> %a) {
 ; CHECK-LABEL: @sub_ctpop_vec(
 ; CHECK-NEXT:    [[TMP1:%.*]] = xor <2 x i32> [[A:%.*]], <i32 -1, i32 -1>
-; CHECK-NEXT:    [[TMP2:%.*]] = call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[TMP1]])
-; CHECK-NEXT:    ret <2 x i32> [[TMP2]]
+; CHECK-NEXT:    [[RES:%.*]] = call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[TMP1]]), !range [[RNG1]]
+; CHECK-NEXT:    ret <2 x i32> [[RES]]
 ;
   %cnt = tail call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> %a)
   %res = sub <2 x i32> <i32 32, i32 32>, %cnt
@@ -338,7 +333,7 @@ define <2 x i32> @sub_ctpop_vec(<2 x i32> %a) {
 
 define <2 x i32> @sub_ctpop_vec_extra_use(<2 x i32> %a, ptr %p) {
 ; CHECK-LABEL: @sub_ctpop_vec_extra_use(
-; CHECK-NEXT:    [[CNT:%.*]] = tail call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[A:%.*]])
+; CHECK-NEXT:    [[CNT:%.*]] = tail call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[A:%.*]]), !range [[RNG1]]
 ; CHECK-NEXT:    store <2 x i32> [[CNT]], ptr [[P:%.*]], align 8
 ; CHECK-NEXT:    [[RES:%.*]] = sub nuw nsw <2 x i32> <i32 32, i32 32>, [[CNT]]
 ; CHECK-NEXT:    ret <2 x i32> [[RES]]
@@ -351,7 +346,7 @@ define <2 x i32> @sub_ctpop_vec_extra_use(<2 x i32> %a, ptr %p) {
 
 define i32 @zext_ctpop(i16 %x) {
 ; CHECK-LABEL: @zext_ctpop(
-; CHECK-NEXT:    [[TMP1:%.*]] = call i16 @llvm.ctpop.i16(i16 [[X:%.*]]), !range [[RNG3:![0-9]+]]
+; CHECK-NEXT:    [[TMP1:%.*]] = call i16 @llvm.ctpop.i16(i16 [[X:%.*]]), !range [[RNG4:![0-9]+]]
 ; CHECK-NEXT:    [[P:%.*]] = zext i16 [[TMP1]] to i32
 ; CHECK-NEXT:    ret i32 [[P]]
 ;
@@ -362,7 +357,7 @@ define i32 @zext_ctpop(i16 %x) {
 
 define <2 x i32> @zext_ctpop_vec(<2 x i7> %x) {
 ; CHECK-LABEL: @zext_ctpop_vec(
-; CHECK-NEXT:    [[TMP1:%.*]] = call <2 x i7> @llvm.ctpop.v2i7(<2 x i7> [[X:%.*]])
+; CHECK-NEXT:    [[TMP1:%.*]] = call <2 x i7> @llvm.ctpop.v2i7(<2 x i7> [[X:%.*]]), !range [[RNG2]]
 ; CHECK-NEXT:    [[P:%.*]] = zext <2 x i7> [[TMP1]] to <2 x i32>
 ; CHECK-NEXT:    ret <2 x i32> [[P]]
 ;
@@ -375,7 +370,7 @@ define i32 @zext_ctpop_extra_use(i16 %x, ptr %q) {
 ; CHECK-LABEL: @zext_ctpop_extra_use(
 ; CHECK-NEXT:    [[Z:%.*]] = zext i16 [[X:%.*]] to i32
 ; CHECK-NEXT:    store i32 [[Z]], ptr [[Q:%.*]], align 4
-; CHECK-NEXT:    [[P:%.*]] = call i32 @llvm.ctpop.i32(i32 [[Z]]), !range [[RNG4:![0-9]+]]
+; CHECK-NEXT:    [[P:%.*]] = call i32 @llvm.ctpop.i32(i32 [[Z]]), !range [[RNG3]]
 ; CHECK-NEXT:    ret i32 [[P]]
 ;
   %z = zext i16 %x to i32
@@ -417,7 +412,7 @@ define i32 @parity_xor_trunc(i64 %arg, i64 %arg1) {
 define <2 x i32> @parity_xor_vec(<2 x i32> %arg, <2 x i32> %arg1) {
 ; CHECK-LABEL: @parity_xor_vec(
 ; CHECK-NEXT:    [[TMP1:%.*]] = xor <2 x i32> [[ARG1:%.*]], [[ARG:%.*]]
-; CHECK-NEXT:    [[TMP2:%.*]] = call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[TMP1]])
+; CHECK-NEXT:    [[TMP2:%.*]] = call <2 x i32> @llvm.ctpop.v2i32(<2 x i32> [[TMP1]]), !range [[RNG1]]
 ; CHECK-NEXT:    [[I4:%.*]] = and <2 x i32> [[TMP2]], <i32 1, i32 1>
 ; CHECK-NEXT:    ret <2 x i32> [[I4]]
 ;
diff --git a/llvm/test/Transforms/InstCombine/cttz.ll b/llvm/test/Transforms/InstCombine/cttz.ll
index 114ef4292dfc3..197eae184f8c3 100644
--- a/llvm/test/Transforms/InstCombine/cttz.ll
+++ b/llvm/test/Transforms/InstCombine/cttz.ll
@@ -8,8 +8,8 @@ declare void @use(i32)
 define i32 @cttz_zext_zero_undef(i16 %x) {
 ; CHECK-LABEL: @cttz_zext_zero_undef(
 ; CHECK-NEXT:    [[TMP1:%.*]] = call i16 @llvm.cttz.i16(i16 [[X:%.*]], i1 true), !range [[RNG0:![0-9]+]]
-; CHECK-NEXT:    [[TMP2:%.*]] = zext i16 [[TMP1]] to i32
-; CHECK-NEXT:    ret i32 [[TMP2]]
+; CHECK-NEXT:    [[TZ:%.*]] = zext i16 [[TMP1]] to i32
+; CHECK-NEXT:    ret i32 [[TZ]]
 ;
   %z = zext i16 %x to i32
   %tz = call i32 @llvm.cttz.i32(i32 %z, i1 true)
@@ -42,9 +42,9 @@ define i32 @cttz_zext_zero_undef_extra_use(i16 %x) {
 
 define <2 x i64> @cttz_zext_zero_undef_vec(<2 x i32> %x) {
 ; CHECK-LABEL: @cttz_zext_zero_undef_vec(
-; CHECK-NEXT:    [[TMP1:%.*]] = call <2 x i32> @llvm.cttz.v2i32(<2 x i32> [[X:%.*]], i1 true)
-; CHECK-NEXT:    [[TMP2:%.*]] = zext <2 x i32> [[TMP1]] to <2 x i64>
-; CHECK-NEXT:    ret <2 x i64> [[TMP2]]
+; CHECK-NEXT:    [[TMP1:%.*]] = call <2 x i32> @llvm.cttz.v2i32(<2 x i32> [[X:%.*]], i1 true), !range [[RNG1]]
+; CHECK-NEXT:    [[TZ:%.*]] = zext <2 x i32> [[TMP1]] to <2 x i64>
+; CHECK-NEXT:    ret <2 x i64> [[TZ]]
 ;
   %z = zext <2 x i32> %x to <2 x i64>
   %tz = tail call <2 x i64> @llvm.cttz.v2i64(<2 x i64> %z, i1 true)
@@ -54,7 +54,7 @@ define <2 x i64> @cttz_zext_zero_undef_vec(<2 x i32> %x) {
 define <2 x i64> @cttz_zext_zero_def_vec(<2 x i32> %x) {
 ; CHECK-LABEL: @cttz_zext_zero_def_vec(
 ; CHECK-NEXT:    [[Z:%.*]] = zext <2 x i32> [[X:%.*]] to <2 x i64>
-; CHECK-NEXT:    [[TZ:%.*]] = tail call <2 x i64> @llvm.cttz.v2i64(<2 x i64> [[Z]], i1 false)
+; CHECK-NEXT:    [[TZ:%.*]] = tail call <2 x i64> @llvm.cttz.v2i64(<2 x i64> [[Z]], i1 false), !range [[RNG2:![0-9]+]]
 ; CHECK-NEXT:    ret <2 x i64> [[TZ]]
 ;
   %z = zext <2 x i32> %x to <2 x i64>
@@ -65,8 +65,8 @@ define <2 x i64> @cttz_zext_zero_def_vec(<2 x i32> %x) {
 define i32 @cttz_sext_zero_undef(i16 %x) {
 ; CHECK-LABEL: @cttz_sext_zero_undef(
 ; CHECK-NEXT:    [[TMP1:%.*]] = call i16 @llvm.cttz.i16(i16 [[X:%.*]], i1 true), !range [[RNG0]]
-; CHECK-NEXT:    [[TMP2:%.*]] = zext i16 [[TMP1]] to i32
-; CHECK-NEXT:    ret i32 [[TMP2]]
+; CHECK-NEXT:    [[TZ:%.*]] = zext i16 [[TMP1]] to i32
+; CHECK-NEXT:    ret i32 [[TZ]]
 ;
   %s = sext i16 %x to i32
   %tz = call i32 @llvm.cttz.i32(i32 %s, i1 true)
@@ -76,8 +76,8 @@ define i32 @cttz_sext_zero_undef(i16 %x) {
 define i32 @cttz_sext_zero_def(i16 %x) {
 ; CHECK-LABEL: @cttz_sext_zero_def(
 ; CHECK-NEXT:    [[TMP1:%.*]] = zext i16 [[X:%.*]] to i32
-; CHECK-NEXT:    [[TMP2:%.*]] = call i32 @llvm.cttz.i32(i32 [[TMP1]], i1 false), !range [[RNG1]]
-; CHECK-NEXT:    ret i32 [[TMP2]]
+; CHECK-NEXT:    [[TZ:%.*]] = call i32 @llvm.cttz.i32(i32 [[TMP1]], i1 false), !range [[RNG1]]
+; CHECK-NEXT:    ret i32 [[TZ]]
 ;
   %s = sext i16 %x to i32
   %tz = call i32 @llvm.cttz.i32(i32 %s, i1 false)
@@ -99,9 +99,9 @@ define i32 @cttz_sext_zero_undef_extra_use(i16 %x) {
 
 define <2 x i64> @cttz_sext_zero_undef_vec(<2 x i32> %x) {
 ; CHECK-LABEL: @cttz_sext_zero_undef_vec(
-; CHECK-NEXT:    [[TMP1:%.*]] = call <2 x i32> @llvm.cttz.v2i32(<2 x i32> [[X:%.*]], i1 true)
-; CHECK-NEXT:    [[TMP2:%.*]] = zext <2 x i32> [[TMP1]] to <2 x i64>
-; CHECK-NEXT:    ret <2 x i64> [[TMP2]]
+; CHECK-NEXT:    [[TMP1:%.*]] = call <2 x i32> @llvm.cttz.v2i32(<2 x i32> [[X:%.*]], i1 true), !range [[RNG1]]
+; CHECK-NEXT:    [[TZ:%.*]] = zext <2 x i32> [[TMP1]] to <2 x i64>
+; CHECK-NEXT:    ret <2 x i64> [[TZ]]
 ;
   %s = sext <2 x i32> %x to <2 x i64>
   %tz = tail call <2 x i64> @llvm.cttz.v2i64(<2 x i64> %s, i1 true)
@@ -111,8 +111,8 @@ define <2 x i64> @cttz_sext_zero_undef_vec(<2 x i32> %x) {
 define <2 x i64> @cttz_sext_zero_def_vec(<2 x i32> %x) {
 ; CHECK-LABEL: @cttz_sext_zero_def_vec(
 ; CHECK-NEXT:    [[TMP1:%.*]] = zext <2 x i32> [[X:%.*]] to <2 x i64>
-; CHECK-NEXT:    [[TMP2:%.*]] = call <2 x i64> @llvm.cttz.v2i64(<2 x i64> [[TMP1]], i1 false)
-; CHECK-NEXT:    ret <2 x i64> [[TMP2]]
+; CHECK-NEXT:    [[TZ:%.*]] = call <2 x i64> @llvm.cttz.v2i64(<2 x i64> [[TMP1]], i1 false), !range [[RNG2]]
+; CHECK-NEXT:    ret <2 x i64> [[TZ]]
 ;
   %s = sext <2 x i32> %x to <2 x i64>
   %tz = tail call <2 x i64> @llvm.cttz.v2i64(<2 x i64> %s, i1 false)
diff --git a/llvm/test/Transforms/InstCombine/dbg-scalable-store-fixed-frag.ll b/llvm/test/Transforms/InstCombine/dbg-scalable-store-fixed-frag.ll
new file mode 100644
index 0000000000000..5307ebe399379
--- /dev/null
+++ b/llvm/test/Transforms/InstCombine/dbg-scalable-store-fixed-frag.ll
@@ -0,0 +1,63 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -passes='instcombine' -S | FileCheck %s
+
+define i32 @foo(<vscale x 2 x i32> %x) {
+; CHECK-LABEL: @foo(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[ARR:%.*]] = alloca i32, align 8
+; CHECK-NEXT:    call void @llvm.dbg.value(metadata <vscale x 2 x i32> undef, metadata [[META8:![0-9]+]], metadata !DIExpression()), !dbg [[DBG14:![0-9]+]]
+; CHECK-NEXT:    store <vscale x 2 x i32> [[X:%.*]], ptr [[ARR]], align 8
+; CHECK-NEXT:    [[RES:%.*]] = load i32, ptr [[ARR]], align 8
+; CHECK-NEXT:    ret i32 [[RES]]
+;
+entry:
+  %arr = alloca i32, align 4
+  call void @llvm.dbg.declare(metadata ptr %arr, metadata !8, metadata !DIExpression()), !dbg !14
+  store <vscale x 2 x i32> %x, ptr %arr, align 4
+  %res = load i32, ptr %arr
+  ret i32 %res
+}
+
+define i32 @foo2(<vscale x 2 x i32> %x) {
+; CHECK-LABEL: @foo2(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[ARR:%.*]] = alloca [4 x i32], align 8
+; CHECK-NEXT:    call void @llvm.dbg.declare(metadata ptr [[ARR]], metadata [[META15:![0-9]+]], metadata !DIExpression()), !dbg [[DBG17:![0-9]+]]
+; CHECK-NEXT:    store <vscale x 2 x i32> [[X:%.*]], ptr [[ARR]], align 8
+; CHECK-NEXT:    [[RES:%.*]] = load i32, ptr [[ARR]], align 8
+; CHECK-NEXT:    ret i32 [[RES]]
+;
+entry:
+  %arr = alloca [4 x i32], align 4
+  call void @llvm.dbg.declare(metadata ptr %arr, metadata !15, metadata !DIExpression()), !dbg !17
+  store <vscale x 2 x i32> %x, ptr %arr, align 4
+  %res = load i32, ptr %arr
+  ret i32 %res
+}
+
+; Function Attrs: nocallback nofree nosync nounwind speculatable willreturn memory(none)
+declare void @llvm.dbg.declare(metadata, metadata, metadata) #0
+
+attributes #0 = { nocallback nofree nosync nounwind speculatable willreturn memory(none) }
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4, !5, !6, !7}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !1, producer: "clang version 17.0.0 (git@github.com:llvm/llvm-project.git a489e11439e36c7e0ec83b28a6fb1596a5c21faa)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, retainedTypes: !2, splitDebugInlining: false, nameTableKind: None)
+!1 = !DIFile(filename: "u.cpp", directory: "/path/to/test", checksumkind: CSK_MD5, checksum: "42f62c17fb0f0110f515890bc6d69cb5")
+!2 = !{}
+!3 = !{i32 7, !"Dwarf Version", i32 5}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = !{i32 1, !"wchar_size", i32 4}
+!6 = !{i32 1, !"target-abi", !"lp64d"}
+!7 = !{i32 8, !"SmallDataLimit", i32 8}
+!8 = !DILocalVariable(name: "arr", scope: !9, file: !1, line: 3, type: !12)
+!9 = distinct !DISubprogram(name: "foo", linkageName: "foo", scope: !1, file: !1, line: 2, type: !10, scopeLine: 2, flags: DIFlagPrototyped | DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !2)
+!10 = !DISubroutineType(types: !11)
+!11 = !{null}
+!12 = !DICompositeType(tag: DW_TAG_array_type, baseType: !13, size: 128, elements: !2)
+!13 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!14 = !DILocation(line: 3, column: 7, scope: !9)
+!15 = !DILocalVariable(name: "arr", scope: !16, file: !1, line: 6, type: !12)
+!16 = distinct !DISubprogram(name: "foo2", linkageName: "foo2", scope: !1, file: !1, line: 5, type: !10, scopeLine: 5, flags: DIFlagPrototyped | DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !2)
+!17 = !DILocation(line: 6, column: 7, scope: !16)
diff --git a/llvm/test/Transforms/InstCombine/div.ll b/llvm/test/Transforms/InstCombine/div.ll
index 59871f51b9298..1f0081befb07d 100644
--- a/llvm/test/Transforms/InstCombine/div.ll
+++ b/llvm/test/Transforms/InstCombine/div.ll
@@ -400,7 +400,7 @@ define i32 @test28(i32 %a) {
 
 define i32 @test29(i32 %a) {
 ; CHECK-LABEL: @test29(
-; CHECK-NEXT:    [[TMP1:%.*]] = icmp eq i32 [[A:%.*]], -1
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp ne i32 [[A:%.*]], 0
 ; CHECK-NEXT:    [[DIV:%.*]] = zext i1 [[TMP1]] to i32
 ; CHECK-NEXT:    ret i32 [[DIV]]
 ;
diff --git a/llvm/test/Transforms/InstCombine/icmp-range.ll b/llvm/test/Transforms/InstCombine/icmp-range.ll
index fcdf76be8f677..4281e09cb0309 100644
--- a/llvm/test/Transforms/InstCombine/icmp-range.ll
+++ b/llvm/test/Transforms/InstCombine/icmp-range.ll
@@ -873,6 +873,70 @@ define <2 x i1> @vector_zext_sext_add_icmp_slt_1_poison(<2 x i1> %a, <2 x i1> %b
   ret <2 x i1> %r
 }
 
+define i1 @zext_sext_add_icmp_slt_minus_1_no_oneuse(i1 %a, i1 %b) {
+; CHECK-LABEL: @zext_sext_add_icmp_slt_minus_1_no_oneuse(
+; CHECK-NEXT:    [[ZEXT_A:%.*]] = zext i1 [[A:%.*]] to i8
+; CHECK-NEXT:    [[SEXT_B:%.*]] = sext i1 [[B:%.*]] to i8
+; CHECK-NEXT:    [[ADD:%.*]] = add nsw i8 [[ZEXT_A]], [[SEXT_B]]
+; CHECK-NEXT:    call void @use(i8 [[ADD]])
+; CHECK-NEXT:    ret i1 false
+;
+  %zext.a = zext i1 %a to i8
+  %sext.b = sext i1 %b to i8
+  %add = add i8 %zext.a, %sext.b
+  call void @use(i8 %add)
+  %r = icmp slt i8 %add, -1
+  ret i1 %r
+}
+
+define i1 @zext_sext_add_icmp_sgt_1_no_oneuse(i1 %a, i1 %b) {
+; CHECK-LABEL: @zext_sext_add_icmp_sgt_1_no_oneuse(
+; CHECK-NEXT:    [[ZEXT_A:%.*]] = zext i1 [[A:%.*]] to i8
+; CHECK-NEXT:    [[SEXT_B:%.*]] = sext i1 [[B:%.*]] to i8
+; CHECK-NEXT:    [[ADD:%.*]] = add nsw i8 [[ZEXT_A]], [[SEXT_B]]
+; CHECK-NEXT:    call void @use(i8 [[ADD]])
+; CHECK-NEXT:    ret i1 false
+;
+  %zext.a = zext i1 %a to i8
+  %sext.b = sext i1 %b to i8
+  %add = add i8 %zext.a, %sext.b
+  call void @use(i8 %add)
+  %r = icmp sgt i8 %add, 1
+  ret i1 %r
+}
+
+define i1 @zext_sext_add_icmp_slt_2_no_oneuse(i1 %a, i1 %b) {
+; CHECK-LABEL: @zext_sext_add_icmp_slt_2_no_oneuse(
+; CHECK-NEXT:    [[ZEXT_A:%.*]] = zext i1 [[A:%.*]] to i8
+; CHECK-NEXT:    [[SEXT_B:%.*]] = sext i1 [[B:%.*]] to i8
+; CHECK-NEXT:    [[ADD:%.*]] = add nsw i8 [[ZEXT_A]], [[SEXT_B]]
+; CHECK-NEXT:    call void @use(i8 [[ADD]])
+; CHECK-NEXT:    ret i1 true
+;
+  %zext.a = zext i1 %a to i8
+  %sext.b = sext i1 %b to i8
+  %add = add i8 %zext.a, %sext.b
+  call void @use(i8 %add)
+  %r = icmp slt i8 %add, 2
+  ret i1 %r
+}
+
+define i1 @zext_sext_add_icmp_sgt_mins_2_no_oneuse(i1 %a, i1 %b) {
+; CHECK-LABEL: @zext_sext_add_icmp_sgt_mins_2_no_oneuse(
+; CHECK-NEXT:    [[ZEXT_A:%.*]] = zext i1 [[A:%.*]] to i8
+; CHECK-NEXT:    [[SEXT_B:%.*]] = sext i1 [[B:%.*]] to i8
+; CHECK-NEXT:    [[ADD:%.*]] = add nsw i8 [[ZEXT_A]], [[SEXT_B]]
+; CHECK-NEXT:    call void @use(i8 [[ADD]])
+; CHECK-NEXT:    ret i1 true
+;
+  %zext.a = zext i1 %a to i8
+  %sext.b = sext i1 %b to i8
+  %add = add i8 %zext.a, %sext.b
+  call void @use(i8 %add)
+  %r = icmp sgt i8 %add, -2
+  ret i1 %r
+}
+
 ; Negative test, more than one use for icmp LHS
 
 define i1 @zext_sext_add_icmp_slt_1_no_oneuse(i1 %a, i1 %b) {
@@ -926,6 +990,50 @@ define i1 @zext_sext_add_icmp_slt_1_type_not_i1(i2 %a, i1 %b) {
   ret i1 %r
 }
 
+define i1 @icmp_eq_bool_0(ptr %ptr) {
+; CHECK-LABEL: @icmp_eq_bool_0(
+; CHECK-NEXT:    [[VAL:%.*]] = load i64, ptr [[PTR:%.*]], align 8, !range [[RNG6:![0-9]+]]
+; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i64 [[VAL]], 0
+; CHECK-NEXT:    ret i1 [[CMP]]
+;
+  %val = load i64, ptr %ptr, align 8, !range !{i64 0, i64 2}
+  %cmp = icmp eq i64 %val, 0
+  ret i1 %cmp
+}
+
+define i1 @icmp_eq_bool_1(ptr %ptr) {
+; CHECK-LABEL: @icmp_eq_bool_1(
+; CHECK-NEXT:    [[VAL:%.*]] = load i64, ptr [[PTR:%.*]], align 8, !range [[RNG6]]
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ne i64 [[VAL]], 0
+; CHECK-NEXT:    ret i1 [[CMP]]
+;
+  %val = load i64, ptr %ptr, align 8, !range !{i64 0, i64 2}
+  %cmp = icmp eq i64 %val, 1
+  ret i1 %cmp
+}
+
+define i1 @icmp_ne_bool_0(ptr %ptr) {
+; CHECK-LABEL: @icmp_ne_bool_0(
+; CHECK-NEXT:    [[VAL:%.*]] = load i64, ptr [[PTR:%.*]], align 8, !range [[RNG6]]
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ne i64 [[VAL]], 0
+; CHECK-NEXT:    ret i1 [[CMP]]
+;
+  %val = load i64, ptr %ptr, align 8, !range !{i64 0, i64 2}
+  %cmp = icmp ne i64 %val, 0
+  ret i1 %cmp
+}
+
+define i1 @icmp_ne_bool_1(ptr %ptr) {
+; CHECK-LABEL: @icmp_ne_bool_1(
+; CHECK-NEXT:    [[VAL:%.*]] = load i64, ptr [[PTR:%.*]], align 8, !range [[RNG6]]
+; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i64 [[VAL]], 0
+; CHECK-NEXT:    ret i1 [[CMP]]
+;
+  %val = load i64, ptr %ptr, align 8, !range !{i64 0, i64 2}
+  %cmp = icmp ne i64 %val, 1
+  ret i1 %cmp
+}
+
 !0 = !{i32 1, i32 6}
 !1 = !{i32 0, i32 6}
 !2 = !{i8 0, i8 1}
diff --git a/llvm/test/Transforms/InstCombine/icmp-shl-nsw.ll b/llvm/test/Transforms/InstCombine/icmp-shl-nsw.ll
index 012ed586194d4..6b9ea1f8ef97e 100644
--- a/llvm/test/Transforms/InstCombine/icmp-shl-nsw.ll
+++ b/llvm/test/Transforms/InstCombine/icmp-shl-nsw.ll
@@ -6,7 +6,7 @@
 
 define i1 @icmp_shl_nsw_sgt(i32 %x) {
 ; CHECK-LABEL: @icmp_shl_nsw_sgt(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i32 %x, 0
+; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[X:%.*]], 0
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i32 %x, 21
@@ -16,7 +16,7 @@ define i1 @icmp_shl_nsw_sgt(i32 %x) {
 
 define i1 @icmp_shl_nsw_sge0(i32 %x) {
 ; CHECK-LABEL: @icmp_shl_nsw_sge0(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i32 %x, -1
+; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[X:%.*]], -1
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i32 %x, 21
@@ -26,7 +26,7 @@ define i1 @icmp_shl_nsw_sge0(i32 %x) {
 
 define i1 @icmp_shl_nsw_sge1(i32 %x) {
 ; CHECK-LABEL: @icmp_shl_nsw_sge1(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i32 %x, 0
+; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[X:%.*]], 0
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i32 %x, 21
@@ -36,7 +36,7 @@ define i1 @icmp_shl_nsw_sge1(i32 %x) {
 
 define <2 x i1> @icmp_shl_nsw_sge1_vec(<2 x i32> %x) {
 ; CHECK-LABEL: @icmp_shl_nsw_sge1_vec(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt <2 x i32> %x, zeroinitializer
+; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt <2 x i32> [[X:%.*]], zeroinitializer
 ; CHECK-NEXT:    ret <2 x i1> [[CMP]]
 ;
   %shl = shl nsw <2 x i32> %x, <i32 21, i32 21>
@@ -48,7 +48,7 @@ define <2 x i1> @icmp_shl_nsw_sge1_vec(<2 x i32> %x) {
 
 define i1 @icmp_shl_nsw_eq(i32 %x) {
 ; CHECK-LABEL: @icmp_shl_nsw_eq(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i32 %x, 0
+; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i32 [[X:%.*]], 0
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %mul = shl nsw i32 %x, 5
@@ -58,7 +58,7 @@ define i1 @icmp_shl_nsw_eq(i32 %x) {
 
 define <2 x i1> @icmp_shl_nsw_eq_vec(<2 x i32> %x) {
 ; CHECK-LABEL: @icmp_shl_nsw_eq_vec(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp eq <2 x i32> %x, zeroinitializer
+; CHECK-NEXT:    [[CMP:%.*]] = icmp eq <2 x i32> [[X:%.*]], zeroinitializer
 ; CHECK-NEXT:    ret <2 x i1> [[CMP]]
 ;
   %mul = shl nsw <2 x i32> %x, <i32 5, i32 5>
@@ -73,7 +73,7 @@ define <2 x i1> @icmp_shl_nsw_eq_vec(<2 x i32> %x) {
 
 define i1 @icmp_sgt1(i8 %x) {
 ; CHECK-LABEL: @icmp_sgt1(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ne i8 %x, -64
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ne i8 [[X:%.*]], -64
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 1
@@ -83,7 +83,7 @@ define i1 @icmp_sgt1(i8 %x) {
 
 define i1 @icmp_sgt2(i8 %x) {
 ; CHECK-LABEL: @icmp_sgt2(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 %x, -64
+; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 [[X:%.*]], -64
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 1
@@ -93,7 +93,7 @@ define i1 @icmp_sgt2(i8 %x) {
 
 define i1 @icmp_sgt3(i8 %x) {
 ; CHECK-LABEL: @icmp_sgt3(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 %x, -8
+; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 [[X:%.*]], -8
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 1
@@ -103,7 +103,7 @@ define i1 @icmp_sgt3(i8 %x) {
 
 define i1 @icmp_sgt4(i8 %x) {
 ; CHECK-LABEL: @icmp_sgt4(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 %x, -1
+; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 [[X:%.*]], -1
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 1
@@ -116,7 +116,7 @@ define i1 @icmp_sgt4(i8 %x) {
 
 define i1 @icmp_sgt5(i8 %x) {
 ; CHECK-LABEL: @icmp_sgt5(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 %x, 0
+; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 [[X:%.*]], 0
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 1
@@ -126,7 +126,7 @@ define i1 @icmp_sgt5(i8 %x) {
 
 define i1 @icmp_sgt6(i8 %x) {
 ; CHECK-LABEL: @icmp_sgt6(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 %x, 8
+; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 [[X:%.*]], 8
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 1
@@ -136,7 +136,7 @@ define i1 @icmp_sgt6(i8 %x) {
 
 define i1 @icmp_sgt7(i8 %x) {
 ; CHECK-LABEL: @icmp_sgt7(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 %x, 62
+; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 [[X:%.*]], 62
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 1
@@ -148,7 +148,7 @@ define i1 @icmp_sgt7(i8 %x) {
 
 define i1 @icmp_sgt8(i8 %x) {
 ; CHECK-LABEL: @icmp_sgt8(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i8 %x, 63
+; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i8 [[X:%.*]], 63
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 1
@@ -162,7 +162,7 @@ define i1 @icmp_sgt8(i8 %x) {
 
 define i1 @icmp_sgt9(i8 %x) {
 ; CHECK-LABEL: @icmp_sgt9(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ne i8 %x, -1
+; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i8 [[X:%.*]], 0
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 7
@@ -172,7 +172,7 @@ define i1 @icmp_sgt9(i8 %x) {
 
 define i1 @icmp_sgt10(i8 %x) {
 ; CHECK-LABEL: @icmp_sgt10(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 %x, -1
+; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 [[X:%.*]], -1
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 7
@@ -182,7 +182,7 @@ define i1 @icmp_sgt10(i8 %x) {
 
 define i1 @icmp_sgt11(i8 %x) {
 ; CHECK-LABEL: @icmp_sgt11(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 %x, -1
+; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i8 [[X:%.*]], -1
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 7
@@ -194,7 +194,7 @@ define i1 @icmp_sgt11(i8 %x) {
 
 define <2 x i1> @icmp_sgt11_vec(<2 x i8> %x) {
 ; CHECK-LABEL: @icmp_sgt11_vec(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt <2 x i8> %x, <i8 -1, i8 -1>
+; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt <2 x i8> [[X:%.*]], <i8 -1, i8 -1>
 ; CHECK-NEXT:    ret <2 x i1> [[CMP]]
 ;
   %shl = shl nsw <2 x i8> %x, <i8 7, i8 7>
@@ -214,7 +214,7 @@ define <2 x i1> @icmp_sgt11_vec(<2 x i8> %x) {
 
 define i1 @icmp_sle1(i8 %x) {
 ; CHECK-LABEL: @icmp_sle1(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i8 %x, -64
+; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i8 [[X:%.*]], -64
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 1
@@ -224,7 +224,7 @@ define i1 @icmp_sle1(i8 %x) {
 
 define i1 @icmp_sle2(i8 %x) {
 ; CHECK-LABEL: @icmp_sle2(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i8 %x, -63
+; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i8 [[X:%.*]], -63
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 1
@@ -234,7 +234,7 @@ define i1 @icmp_sle2(i8 %x) {
 
 define i1 @icmp_sle3(i8 %x) {
 ; CHECK-LABEL: @icmp_sle3(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i8 %x, -7
+; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i8 [[X:%.*]], -7
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 1
@@ -244,7 +244,7 @@ define i1 @icmp_sle3(i8 %x) {
 
 define i1 @icmp_sle4(i8 %x) {
 ; CHECK-LABEL: @icmp_sle4(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i8 %x, 0
+; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i8 [[X:%.*]], 0
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 1
@@ -257,7 +257,7 @@ define i1 @icmp_sle4(i8 %x) {
 
 define i1 @icmp_sle5(i8 %x) {
 ; CHECK-LABEL: @icmp_sle5(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i8 %x, 1
+; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i8 [[X:%.*]], 1
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 1
@@ -267,7 +267,7 @@ define i1 @icmp_sle5(i8 %x) {
 
 define i1 @icmp_sle6(i8 %x) {
 ; CHECK-LABEL: @icmp_sle6(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i8 %x, 9
+; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i8 [[X:%.*]], 9
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 1
@@ -277,7 +277,7 @@ define i1 @icmp_sle6(i8 %x) {
 
 define i1 @icmp_sle7(i8 %x) {
 ; CHECK-LABEL: @icmp_sle7(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i8 %x, 63
+; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i8 [[X:%.*]], 63
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 1
@@ -289,7 +289,7 @@ define i1 @icmp_sle7(i8 %x) {
 
 define i1 @icmp_sle8(i8 %x) {
 ; CHECK-LABEL: @icmp_sle8(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ne i8 %x, 63
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ne i8 [[X:%.*]], 63
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 1
@@ -303,7 +303,7 @@ define i1 @icmp_sle8(i8 %x) {
 
 define i1 @icmp_sle9(i8 %x) {
 ; CHECK-LABEL: @icmp_sle9(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i8 %x, -1
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ne i8 [[X:%.*]], 0
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 7
@@ -313,7 +313,7 @@ define i1 @icmp_sle9(i8 %x) {
 
 define i1 @icmp_sle10(i8 %x) {
 ; CHECK-LABEL: @icmp_sle10(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i8 %x, 0
+; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i8 [[X:%.*]], 0
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 7
@@ -323,7 +323,7 @@ define i1 @icmp_sle10(i8 %x) {
 
 define i1 @icmp_sle11(i8 %x) {
 ; CHECK-LABEL: @icmp_sle11(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i8 %x, 0
+; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i8 [[X:%.*]], 0
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 7
@@ -336,7 +336,7 @@ define i1 @icmp_sle11(i8 %x) {
 
 define i1 @icmp_eq1(i8 %x) {
 ; CHECK-LABEL: @icmp_eq1(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i8 %x, 6
+; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i8 [[X:%.*]], 6
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 1
@@ -346,7 +346,7 @@ define i1 @icmp_eq1(i8 %x) {
 
 define i1 @icmp_ne1(i8 %x) {
 ; CHECK-LABEL: @icmp_ne1(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ne i8 %x, -2
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ne i8 [[X:%.*]], -2
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %shl = shl nsw i8 %x, 6
diff --git a/llvm/test/Transforms/InstCombine/icmp-shr-lt-gt.ll b/llvm/test/Transforms/InstCombine/icmp-shr-lt-gt.ll
index 6ef1c6ca211d5..33c9b59551667 100644
--- a/llvm/test/Transforms/InstCombine/icmp-shr-lt-gt.ll
+++ b/llvm/test/Transforms/InstCombine/icmp-shr-lt-gt.ll
@@ -2767,7 +2767,7 @@ define i1 @lshrult_03_00_exact(i4 %x) {
 
 define i1 @lshrult_03_01_exact(i4 %x) {
 ; CHECK-LABEL: @lshrult_03_01_exact(
-; CHECK-NEXT:    [[C:%.*]] = icmp ne i4 [[X:%.*]], -8
+; CHECK-NEXT:    [[C:%.*]] = icmp eq i4 [[X:%.*]], 0
 ; CHECK-NEXT:    ret i1 [[C]]
 ;
   %s = lshr exact i4 %x, 3
diff --git a/llvm/test/Transforms/InstCombine/icmp-vscale.ll b/llvm/test/Transforms/InstCombine/icmp-vscale.ll
index c85a9e8a406d6..baac5ce495985 100644
--- a/llvm/test/Transforms/InstCombine/icmp-vscale.ll
+++ b/llvm/test/Transforms/InstCombine/icmp-vscale.ll
@@ -82,6 +82,111 @@ entry:
   ret i1 %res
 }
 
+define i1 @vscale_ule_max() vscale_range(5,10) {
+; CHECK-LABEL: @vscale_ule_max(
+; CHECK-NEXT:    ret i1 true
+;
+  %vscale = call i16 @llvm.vscale.i16()
+  %res = icmp ule i16 %vscale, 10
+  ret i1 %res
+}
+
+define i1 @vscale_ult_max() vscale_range(5,10) {
+; CHECK-LABEL: @vscale_ult_max(
+; CHECK-NEXT:    [[VSCALE:%.*]] = call i16 @llvm.vscale.i16()
+; CHECK-NEXT:    [[RES:%.*]] = icmp ult i16 [[VSCALE]], 10
+; CHECK-NEXT:    ret i1 [[RES]]
+;
+  %vscale = call i16 @llvm.vscale.i16()
+  %res = icmp ult i16 %vscale, 10
+  ret i1 %res
+}
+
+define i1 @vscale_uge_min() vscale_range(5,10) {
+; CHECK-LABEL: @vscale_uge_min(
+; CHECK-NEXT:    ret i1 true
+;
+  %vscale = call i16 @llvm.vscale.i16()
+  %res = icmp uge i16 %vscale, 5
+  ret i1 %res
+}
+
+define i1 @vscale_ugt_min() vscale_range(5,10) {
+; CHECK-LABEL: @vscale_ugt_min(
+; CHECK-NEXT:    [[VSCALE:%.*]] = call i16 @llvm.vscale.i16()
+; CHECK-NEXT:    [[RES:%.*]] = icmp ugt i16 [[VSCALE]], 5
+; CHECK-NEXT:    ret i1 [[RES]]
+;
+  %vscale = call i16 @llvm.vscale.i16()
+  %res = icmp ugt i16 %vscale, 5
+  ret i1 %res
+}
+
+define i1 @vscale_uge_no_max() vscale_range(5) {
+; CHECK-LABEL: @vscale_uge_no_max(
+; CHECK-NEXT:    ret i1 true
+;
+  %vscale = call i8 @llvm.vscale.i8()
+  %res = icmp uge i8 %vscale, 5
+  ret i1 %res
+}
+
+define i1 @vscale_ugt_no_max() vscale_range(5) {
+; CHECK-LABEL: @vscale_ugt_no_max(
+; CHECK-NEXT:    ret i1 false
+;
+  %vscale = call i8 @llvm.vscale.i8()
+  %res = icmp ugt i8 %vscale, 5
+  ret i1 %res
+}
+
+define i1 @vscale_uge_max_overflow() vscale_range(5,256) {
+; CHECK-LABEL: @vscale_uge_max_overflow(
+; CHECK-NEXT:    ret i1 true
+;
+  %vscale = call i8 @llvm.vscale.i8()
+  %res = icmp uge i8 %vscale, 5
+  ret i1 %res
+}
+
+define i1 @vscale_ugt_max_overflow() vscale_range(5,256) {
+; CHECK-LABEL: @vscale_ugt_max_overflow(
+; CHECK-NEXT:    [[VSCALE:%.*]] = call i8 @llvm.vscale.i8()
+; CHECK-NEXT:    [[RES:%.*]] = icmp ugt i8 [[VSCALE]], 5
+; CHECK-NEXT:    ret i1 [[RES]]
+;
+  %vscale = call i8 @llvm.vscale.i8()
+  %res = icmp ugt i8 %vscale, 5
+  ret i1 %res
+}
+
+define i1 @vscale_eq_min_overflow() vscale_range(256,300) {
+; CHECK-LABEL: @vscale_eq_min_overflow(
+; CHECK-NEXT:    ret i1 true
+;
+  %vscale = call i8 @llvm.vscale.i8()
+  %res = icmp eq i8 %vscale, 42
+  ret i1 %res
+}
+
+define i1 @vscale_ult_min_overflow() vscale_range(256,300) {
+; CHECK-LABEL: @vscale_ult_min_overflow(
+; CHECK-NEXT:    ret i1 true
+;
+  %vscale = call i8 @llvm.vscale.i8()
+  %res = icmp ult i8 %vscale, 42
+  ret i1 %res
+}
+
+define i1 @vscale_ugt_min_overflow() vscale_range(256,300) {
+; CHECK-LABEL: @vscale_ugt_min_overflow(
+; CHECK-NEXT:    ret i1 true
+;
+  %vscale = call i8 @llvm.vscale.i8()
+  %res = icmp ugt i8 %vscale, 42
+  ret i1 %res
+}
+
 declare i8 @llvm.vscale.i8()
 declare i16 @llvm.vscale.i16()
 declare i32 @llvm.vscale.i32()
diff --git a/llvm/test/Transforms/InstCombine/is_fpclass.ll b/llvm/test/Transforms/InstCombine/is_fpclass.ll
index f746215dbbdf3..eba5b25dc5886 100644
--- a/llvm/test/Transforms/InstCombine/is_fpclass.ll
+++ b/llvm/test/Transforms/InstCombine/is_fpclass.ll
@@ -116,7 +116,7 @@ define i1 @test_class_isnan_f32_strict(float %x) {
 
 define i1 @test_class_is_p0_n0_f32(float %x) {
 ; CHECK-LABEL: @test_class_is_p0_n0_f32(
-; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 96)
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp oeq float [[X:%.*]], 0.000000e+00
 ; CHECK-NEXT:    ret i1 [[VAL]]
 ;
   %val = call i1 @llvm.is.fpclass.f32(float %x, i32 96) ; fcZero
@@ -125,7 +125,7 @@ define i1 @test_class_is_p0_n0_f32(float %x) {
 
 define <2 x i1> @test_class_is_p0_n0_v2f32(<2 x float> %x) {
 ; CHECK-LABEL: @test_class_is_p0_n0_v2f32(
-; CHECK-NEXT:    [[VAL:%.*]] = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> [[X:%.*]], i32 96)
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp oeq <2 x float> [[X:%.*]], zeroinitializer
 ; CHECK-NEXT:    ret <2 x i1> [[VAL]]
 ;
   %val = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> %x, i32 96) ; fcZero
@@ -152,7 +152,7 @@ define <2 x i1> @test_class_is_p0_n0_v2f32_dynamic(<2 x float> %x) "denormal-fp-
 
 define i1 @test_class_is_p0_n0_or_nan_f32(float %x) {
 ; CHECK-LABEL: @test_class_is_p0_n0_or_nan_f32(
-; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 99)
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp ueq float [[X:%.*]], 0.000000e+00
 ; CHECK-NEXT:    ret i1 [[VAL]]
 ;
   %val = call i1 @llvm.is.fpclass.f32(float %x, i32 99) ; fcZero|fcNan
@@ -161,6 +161,24 @@ define i1 @test_class_is_p0_n0_or_nan_f32(float %x) {
 
 define <2 x i1> @test_class_is_p0_n0_or_nan_v2f32(<2 x float> %x) {
 ; CHECK-LABEL: @test_class_is_p0_n0_or_nan_v2f32(
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp ueq <2 x float> [[X:%.*]], zeroinitializer
+; CHECK-NEXT:    ret <2 x i1> [[VAL]]
+;
+  %val = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> %x, i32 99) ; fcZero|fcNan
+  ret <2 x i1> %val
+}
+
+define i1 @test_class_is_p0_n0_or_nan_f32_daz(float %x) "denormal-fp-math-f32"="ieee,preserve-sign" {
+; CHECK-LABEL: @test_class_is_p0_n0_or_nan_f32_daz(
+; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 99)
+; CHECK-NEXT:    ret i1 [[VAL]]
+;
+  %val = call i1 @llvm.is.fpclass.f32(float %x, i32 99) ; fcZero|fcNan
+  ret i1 %val
+}
+
+define <2 x i1> @test_class_is_p0_n0_or_nan_v2f32_daz(<2 x float> %x) "denormal-fp-math-f32"="ieee,preserve-sign" {
+; CHECK-LABEL: @test_class_is_p0_n0_or_nan_v2f32_daz(
 ; CHECK-NEXT:    [[VAL:%.*]] = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> [[X:%.*]], i32 99)
 ; CHECK-NEXT:    ret <2 x i1> [[VAL]]
 ;
@@ -168,9 +186,136 @@ define <2 x i1> @test_class_is_p0_n0_or_nan_v2f32(<2 x float> %x) {
   ret <2 x i1> %val
 }
 
+define i1 @test_class_is_p0_n0_or_sub_or_nan_f32(float %x) {
+; CHECK-LABEL: @test_class_is_p0_n0_or_sub_or_nan_f32(
+; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 243)
+; CHECK-NEXT:    ret i1 [[VAL]]
+;
+;`
+  %val = call i1 @llvm.is.fpclass.f32(float %x, i32 243) ; fcZero|fcNan|fcSubnormal
+  ret i1 %val
+}
+
+define <2 x i1> @test_class_is_p0_n0_or_sub_or_nan_v2f32(<2 x float> %x) {
+; CHECK-LABEL: @test_class_is_p0_n0_or_sub_or_nan_v2f32(
+; CHECK-NEXT:    [[VAL:%.*]] = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> [[X:%.*]], i32 243)
+; CHECK-NEXT:    ret <2 x i1> [[VAL]]
+;
+  %val = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> %x, i32 243) ; fcZero|fcNan|fcSubnormal
+  ret <2 x i1> %val
+}
+
+define i1 @test_class_is_p0_n0_or_sub_or_nan_f32_daz(float %x) "denormal-fp-math-f32"="ieee,preserve-sign" {
+; CHECK-LABEL: @test_class_is_p0_n0_or_sub_or_nan_f32_daz(
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp ueq float [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    ret i1 [[VAL]]
+;
+  %val = call i1 @llvm.is.fpclass.f32(float %x, i32 243) ; fcZero|fcNan|fcSubnormal
+  ret i1 %val
+}
+
+define <2 x i1> @test_class_is_p0_n0_or_sub_or_nan_v2f32_daz(<2 x float> %x) "denormal-fp-math-f32"="ieee,preserve-sign" {
+; CHECK-LABEL: @test_class_is_p0_n0_or_sub_or_nan_v2f32_daz(
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp ueq <2 x float> [[X:%.*]], zeroinitializer
+; CHECK-NEXT:    ret <2 x i1> [[VAL]]
+;
+  %val = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> %x, i32 243) ; fcZero|fcNan|fcSubnormal
+  ret <2 x i1> %val
+}
+
+define i1 @test_class_is_p0_n0_or_sub_or_snan_f32_daz(float %x) "denormal-fp-math-f32"="ieee,preserve-sign" {
+; CHECK-LABEL: @test_class_is_p0_n0_or_sub_or_snan_f32_daz(
+; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 241)
+; CHECK-NEXT:    ret i1 [[VAL]]
+;
+  %val = call i1 @llvm.is.fpclass.f32(float %x, i32 241) ; fcZero|fcSNan|fcSubnormal
+  ret i1 %val
+}
+
+define i1 @test_class_is_p0_n0_or_sub_or_qnan_f32_daz(float %x) "denormal-fp-math-f32"="ieee,preserve-sign" {
+; CHECK-LABEL: @test_class_is_p0_n0_or_sub_or_qnan_f32_daz(
+; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 242)
+; CHECK-NEXT:    ret i1 [[VAL]]
+;
+  %val = call i1 @llvm.is.fpclass.f32(float %x, i32 242) ; fcZero|fcQNan|fcSubnormal
+  ret i1 %val
+}
+
+define i1 @test_class_is_not_p0_n0_or_nan_f32(float %x) {
+; CHECK-LABEL: @test_class_is_not_p0_n0_or_nan_f32(
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp one float [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    ret i1 [[VAL]]
+;
+  %val = call i1 @llvm.is.fpclass.f32(float %x, i32 924) ; ~fcZero & ~fcNan & fcAllFlags
+  ret i1 %val
+}
+
+define i1 @test_class_is_not_p0_n0_or_qnan_f32(float %x) {
+; CHECK-LABEL: @test_class_is_not_p0_n0_or_qnan_f32(
+; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 926)
+; CHECK-NEXT:    ret i1 [[VAL]]
+;
+  %val = call i1 @llvm.is.fpclass.f32(float %x, i32 926) ; ~fcZero & ~fcSNan & fcAllFlags
+  ret i1 %val
+}
+
+define i1 @test_class_is_not_p0_n0_or_snan_f32(float %x) {
+; CHECK-LABEL: @test_class_is_not_p0_n0_or_snan_f32(
+; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 925)
+; CHECK-NEXT:    ret i1 [[VAL]]
+;
+  %val = call i1 @llvm.is.fpclass.f32(float %x, i32 925) ; ~fcZero & ~fcQNan & fcAllFlags
+  ret i1 %val
+}
+
+define i1 @test_class_is_not_p0_n0_or_nan_f32_daz(float %x) "denormal-fp-math-f32"="ieee,preserve-sign" {
+; CHECK-LABEL: @test_class_is_not_p0_n0_or_nan_f32_daz(
+; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 924)
+; CHECK-NEXT:    ret i1 [[VAL]]
+;
+  %val = call i1 @llvm.is.fpclass.f32(float %x, i32 924) ; ~fcZero & ~fcNan & fcAllFlags
+  ret i1 %val
+}
+
+define i1 @test_class_is_not_p0_n0_or_sub_or_nan_f32(float %x) {
+; CHECK-LABEL: @test_class_is_not_p0_n0_or_sub_or_nan_f32(
+; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 783)
+; CHECK-NEXT:    ret i1 [[VAL]]
+;
+  %val = call i1 @llvm.is.fpclass.f32(float %x, i32 783) ; ~(fcZero|fcSubnormal) | fcNan
+  ret i1 %val
+}
+
+define i1 @test_class_is_not_p0_n0_or_sub_or_nan_f32_daz(float %x) "denormal-fp-math-f32"="ieee,preserve-sign" {
+; CHECK-LABEL: @test_class_is_not_p0_n0_or_sub_or_nan_f32_daz(
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp une float [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    ret i1 [[VAL]]
+;
+  %val = call i1 @llvm.is.fpclass.f32(float %x, i32 783) ; ~(fcZero|fcSubnormal) | fcNan
+  ret i1 %val
+}
+
+define i1 @test_class_is_not_p0_n0_or_sub_and_not_nan_f32(float %x) {
+; CHECK-LABEL: @test_class_is_not_p0_n0_or_sub_and_not_nan_f32(
+; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 780)
+; CHECK-NEXT:    ret i1 [[VAL]]
+;
+  %val = call i1 @llvm.is.fpclass.f32(float %x, i32 780) ; ~(fcZero|fcSubnormal) & ~fcNan & fcAllFlags
+  ret i1 %val
+}
+
+define i1 @test_class_is_not_p0_n0_or_sub_and_not_nan_f32_daz(float %x) "denormal-fp-math-f32"="ieee,preserve-sign" {
+; CHECK-LABEL: @test_class_is_not_p0_n0_or_sub_and_not_nan_f32_daz(
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp one float [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    ret i1 [[VAL]]
+;
+  %val = call i1 @llvm.is.fpclass.f32(float %x, i32 780) ; ~(fcZero|fcSubonrmal) & ~fcNan & fcAllFlags
+  ret i1 %val
+}
+
 define i1 @test_class_is_not_p0_n0_f32(float %x) {
 ; CHECK-LABEL: @test_class_is_not_p0_n0_f32(
-; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 927)
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp une float [[X:%.*]], 0.000000e+00
 ; CHECK-NEXT:    ret i1 [[VAL]]
 ;
   %val = call i1 @llvm.is.fpclass.f32(float %x, i32 927) ; ~fcZero & fcAllFlags
@@ -179,7 +324,7 @@ define i1 @test_class_is_not_p0_n0_f32(float %x) {
 
 define <2 x i1> @test_class_is_not_p0_n0_v2f32(<2 x float> %x) {
 ; CHECK-LABEL: @test_class_is_not_p0_n0_v2f32(
-; CHECK-NEXT:    [[VAL:%.*]] = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> [[X:%.*]], i32 927)
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp une <2 x float> [[X:%.*]], zeroinitializer
 ; CHECK-NEXT:    ret <2 x i1> [[VAL]]
 ;
   %val = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> %x, i32 927) ; ~fcZero & fcAllFlags
@@ -215,7 +360,7 @@ define i1 @test_class_is_not_p0_n0_f32_dynamic(float %x) "denormal-fp-math"="iee
 
 define i1 @test_class_is_not_p0_n0_psub_nsub_f32_daz(float %x) "denormal-fp-math"="ieee,preserve-sign" {
 ; CHECK-LABEL: @test_class_is_not_p0_n0_psub_nsub_f32_daz(
-; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 783)
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp une float [[X:%.*]], 0.000000e+00
 ; CHECK-NEXT:    ret i1 [[VAL]]
 ;
   %val = call i1 @llvm.is.fpclass.f32(float %x, i32 783) ; ~(fcZero|fcSubnormal) & fcAllFlags
@@ -224,7 +369,7 @@ define i1 @test_class_is_not_p0_n0_psub_nsub_f32_daz(float %x) "denormal-fp-math
 
 define i1 @test_class_is_not_p0_n0_psub_nsub_f32_dapz(float %x) "denormal-fp-math"="ieee,positive-zero" {
 ; CHECK-LABEL: @test_class_is_not_p0_n0_psub_nsub_f32_dapz(
-; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 783)
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp une float [[X:%.*]], 0.000000e+00
 ; CHECK-NEXT:    ret i1 [[VAL]]
 ;
   %val = call i1 @llvm.is.fpclass.f32(float %x, i32 783) ; ~(fcZero|fcSubnormal) & fcAllFlags
@@ -278,7 +423,7 @@ define i1 @test_class_is_p0_n0_psub_nsub_f32(float %x) {
 
 define i1 @test_class_is_p0_n0_psub_nsub_f32_daz(float %x) "denormal-fp-math"="ieee,preserve-sign" {
 ; CHECK-LABEL: @test_class_is_p0_n0_psub_nsub_f32_daz(
-; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 240)
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp oeq float [[X:%.*]], 0.000000e+00
 ; CHECK-NEXT:    ret i1 [[VAL]]
 ;
   %val = call i1 @llvm.is.fpclass.f32(float %x, i32 240) ; fcZero | fcSubnormal
@@ -287,7 +432,7 @@ define i1 @test_class_is_p0_n0_psub_nsub_f32_daz(float %x) "denormal-fp-math"="i
 
 define i1 @test_class_is_p0_n0_psub_nsub_f32_dapz(float %x) "denormal-fp-math"="ieee,positive-zero" {
 ; CHECK-LABEL: @test_class_is_p0_n0_psub_nsub_f32_dapz(
-; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 240)
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp oeq float [[X:%.*]], 0.000000e+00
 ; CHECK-NEXT:    ret i1 [[VAL]]
 ;
   %val = call i1 @llvm.is.fpclass.f32(float %x, i32 240) ; fcZero | fcSubnormal
@@ -314,7 +459,7 @@ define <2 x i1> @test_class_is_p0_n0_psub_nsub_v2f32(<2 x float> %x) {
 
 define <2 x i1> @test_class_is_p0_n0_psub_nsub_v2f32_daz(<2 x float> %x) "denormal-fp-math"="ieee,preserve-sign" {
 ; CHECK-LABEL: @test_class_is_p0_n0_psub_nsub_v2f32_daz(
-; CHECK-NEXT:    [[VAL:%.*]] = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> [[X:%.*]], i32 240)
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp oeq <2 x float> [[X:%.*]], zeroinitializer
 ; CHECK-NEXT:    ret <2 x i1> [[VAL]]
 ;
   %val = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> %x, i32 240) ; fcZero | fcSubnormal
@@ -323,7 +468,7 @@ define <2 x i1> @test_class_is_p0_n0_psub_nsub_v2f32_daz(<2 x float> %x) "denorm
 
 define <2 x i1> @test_class_is_p0_n0_psub_nsub_v2f32_dapz(<2 x float> %x) "denormal-fp-math"="ieee,positive-zero" {
 ; CHECK-LABEL: @test_class_is_p0_n0_psub_nsub_v2f32_dapz(
-; CHECK-NEXT:    [[VAL:%.*]] = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> [[X:%.*]], i32 240)
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp oeq <2 x float> [[X:%.*]], zeroinitializer
 ; CHECK-NEXT:    ret <2 x i1> [[VAL]]
 ;
   %val = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> %x, i32 240) ; fcZero | fcSubnormal
@@ -341,7 +486,7 @@ define <2 x i1> @test_class_is_p0_n0_psub_nsub_v2f32_dynamic(<2 x float> %x) "de
 
 define i1 @test_class_is_pinf_f32(float %x) {
 ; CHECK-LABEL: @test_class_is_pinf_f32(
-; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 512)
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp oeq float [[X:%.*]], 0x7FF0000000000000
 ; CHECK-NEXT:    ret i1 [[VAL]]
 ;
   %val = call i1 @llvm.is.fpclass.f32(float %x, i32 512) ; fcPosInf
@@ -350,7 +495,7 @@ define i1 @test_class_is_pinf_f32(float %x) {
 
 define i1 @test_class_is_pinf_or_nan_f32(float %x) {
 ; CHECK-LABEL: @test_class_is_pinf_or_nan_f32(
-; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 515)
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp ueq float [[X:%.*]], 0x7FF0000000000000
 ; CHECK-NEXT:    ret i1 [[VAL]]
 ;
   %val = call i1 @llvm.is.fpclass.f32(float %x, i32 515) ; fcPosInf|fcNan
@@ -359,7 +504,7 @@ define i1 @test_class_is_pinf_or_nan_f32(float %x) {
 
 define <2 x i1> @test_class_is_pinf_v2f32(<2 x float> %x) {
 ; CHECK-LABEL: @test_class_is_pinf_v2f32(
-; CHECK-NEXT:    [[VAL:%.*]] = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> [[X:%.*]], i32 512)
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp oeq <2 x float> [[X:%.*]], <float 0x7FF0000000000000, float 0x7FF0000000000000>
 ; CHECK-NEXT:    ret <2 x i1> [[VAL]]
 ;
   %val = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> %x, i32 512) ; fcPosInf
@@ -368,7 +513,7 @@ define <2 x i1> @test_class_is_pinf_v2f32(<2 x float> %x) {
 
 define i1 @test_class_is_ninf_f32(float %x) {
 ; CHECK-LABEL: @test_class_is_ninf_f32(
-; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 4)
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp oeq float [[X:%.*]], 0xFFF0000000000000
 ; CHECK-NEXT:    ret i1 [[VAL]]
 ;
   %val = call i1 @llvm.is.fpclass.f32(float %x, i32 4) ; fcNegInf
@@ -377,7 +522,7 @@ define i1 @test_class_is_ninf_f32(float %x) {
 
 define i1 @test_class_is_ninf_or_nan_f32(float %x) {
 ; CHECK-LABEL: @test_class_is_ninf_or_nan_f32(
-; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.is.fpclass.f32(float [[X:%.*]], i32 7)
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp ueq float [[X:%.*]], 0xFFF0000000000000
 ; CHECK-NEXT:    ret i1 [[VAL]]
 ;
   %val = call i1 @llvm.is.fpclass.f32(float %x, i32 7) ; fcNegInf|fcNan
@@ -386,7 +531,7 @@ define i1 @test_class_is_ninf_or_nan_f32(float %x) {
 
 define <2 x i1> @test_class_is_ninf_v2f32(<2 x float> %x) {
 ; CHECK-LABEL: @test_class_is_ninf_v2f32(
-; CHECK-NEXT:    [[VAL:%.*]] = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> [[X:%.*]], i32 4)
+; CHECK-NEXT:    [[VAL:%.*]] = fcmp oeq <2 x float> [[X:%.*]], <float 0xFFF0000000000000, float 0xFFF0000000000000>
 ; CHECK-NEXT:    ret <2 x i1> [[VAL]]
 ;
   %val = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> %x, i32 4) ; fcNegInf
@@ -870,8 +1015,8 @@ define i1 @test_fold_or_class_f32_0(float %a) {
 
 define i1 @test_fold_or3_class_f32_0(float %a) {
 ; CHECK-LABEL: @test_fold_or3_class_f32_0(
-; CHECK-NEXT:    [[CLASS2:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A:%.*]], i32 7)
-; CHECK-NEXT:    ret i1 [[CLASS2]]
+; CHECK-NEXT:    [[OR_1:%.*]] = fcmp ueq float [[A:%.*]], 0xFFF0000000000000
+; CHECK-NEXT:    ret i1 [[OR_1]]
 ;
   %class0 = call i1 @llvm.is.fpclass.f32(float %a, i32 1)
   %class1 = call i1 @llvm.is.fpclass.f32(float %a, i32 2)
@@ -909,8 +1054,8 @@ define i1 @test_fold_or_all_tests_class_f32_0(float %a) {
 
 define i1 @test_fold_or_class_f32_1(float %a) {
 ; CHECK-LABEL: @test_fold_or_class_f32_1(
-; CHECK-NEXT:    [[CLASS0:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A:%.*]], i32 12)
-; CHECK-NEXT:    ret i1 [[CLASS0]]
+; CHECK-NEXT:    [[CLASS1:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A:%.*]], i32 12)
+; CHECK-NEXT:    ret i1 [[CLASS1]]
 ;
   %class0 = call i1 @llvm.is.fpclass.f32(float %a, i32 4)
   %class1 = call i1 @llvm.is.fpclass.f32(float %a, i32 8)
@@ -920,7 +1065,7 @@ define i1 @test_fold_or_class_f32_1(float %a) {
 
 define i1 @test_no_fold_or_class_f32_multi_use0(float %a, ptr %ptr) {
 ; CHECK-LABEL: @test_no_fold_or_class_f32_multi_use0(
-; CHECK-NEXT:    [[CLASS0:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A:%.*]], i32 4)
+; CHECK-NEXT:    [[CLASS0:%.*]] = fcmp oeq float [[A:%.*]], 0xFFF0000000000000
 ; CHECK-NEXT:    store i1 [[CLASS0]], ptr [[PTR:%.*]], align 1
 ; CHECK-NEXT:    [[CLASS1:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A]], i32 8)
 ; CHECK-NEXT:    [[OR:%.*]] = or i1 [[CLASS0]], [[CLASS1]]
@@ -935,7 +1080,7 @@ define i1 @test_no_fold_or_class_f32_multi_use0(float %a, ptr %ptr) {
 
 define i1 @test_no_fold_or_class_f32_multi_use1(float %a, ptr %ptr) {
 ; CHECK-LABEL: @test_no_fold_or_class_f32_multi_use1(
-; CHECK-NEXT:    [[CLASS0:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A:%.*]], i32 4)
+; CHECK-NEXT:    [[CLASS0:%.*]] = fcmp oeq float [[A:%.*]], 0xFFF0000000000000
 ; CHECK-NEXT:    [[CLASS1:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A]], i32 8)
 ; CHECK-NEXT:    store i1 [[CLASS1]], ptr [[PTR:%.*]], align 1
 ; CHECK-NEXT:    [[OR:%.*]] = or i1 [[CLASS0]], [[CLASS1]]
@@ -950,8 +1095,8 @@ define i1 @test_no_fold_or_class_f32_multi_use1(float %a, ptr %ptr) {
 
 define i1 @test_fold_or_class_f32_2(float %a) {
 ; CHECK-LABEL: @test_fold_or_class_f32_2(
-; CHECK-NEXT:    [[CLASS0:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A:%.*]], i32 7)
-; CHECK-NEXT:    ret i1 [[CLASS0]]
+; CHECK-NEXT:    [[OR:%.*]] = fcmp ueq float [[A:%.*]], 0xFFF0000000000000
+; CHECK-NEXT:    ret i1 [[OR]]
 ;
   %class0 = call i1 @llvm.is.fpclass.f32(float %a, i32 7)
   %class1 = call i1 @llvm.is.fpclass.f32(float %a, i32 7)
@@ -961,7 +1106,7 @@ define i1 @test_fold_or_class_f32_2(float %a) {
 
 define i1 @test_no_fold_or_class_f32_0(float %a, float %b) {
 ; CHECK-LABEL: @test_no_fold_or_class_f32_0(
-; CHECK-NEXT:    [[CLASS0:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A:%.*]], i32 4)
+; CHECK-NEXT:    [[CLASS0:%.*]] = fcmp oeq float [[A:%.*]], 0xFFF0000000000000
 ; CHECK-NEXT:    [[CLASS1:%.*]] = call i1 @llvm.is.fpclass.f32(float [[B:%.*]], i32 8)
 ; CHECK-NEXT:    [[OR:%.*]] = or i1 [[CLASS0]], [[CLASS1]]
 ; CHECK-NEXT:    ret i1 [[OR]]
@@ -974,8 +1119,8 @@ define i1 @test_no_fold_or_class_f32_0(float %a, float %b) {
 
 define <2 x i1> @test_fold_or_class_v2f32(<2 x float> %a) {
 ; CHECK-LABEL: @test_fold_or_class_v2f32(
-; CHECK-NEXT:    [[CLASS0:%.*]] = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> [[A:%.*]], i32 12)
-; CHECK-NEXT:    ret <2 x i1> [[CLASS0]]
+; CHECK-NEXT:    [[CLASS1:%.*]] = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> [[A:%.*]], i32 12)
+; CHECK-NEXT:    ret <2 x i1> [[CLASS1]]
 ;
   %class0 = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> %a, i32 4)
   %class1 = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> %a, i32 8)
@@ -1105,8 +1250,8 @@ define i1 @test_no_fold_and_class_f32_multi_use1(float %a, ptr %ptr) {
 
 define i1 @test_fold_and_class_f32_2(float %a) {
 ; CHECK-LABEL: @test_fold_and_class_f32_2(
-; CHECK-NEXT:    [[CLASS0:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A:%.*]], i32 7)
-; CHECK-NEXT:    ret i1 [[CLASS0]]
+; CHECK-NEXT:    [[AND:%.*]] = fcmp ueq float [[A:%.*]], 0xFFF0000000000000
+; CHECK-NEXT:    ret i1 [[AND]]
 ;
   %class0 = call i1 @llvm.is.fpclass.f32(float %a, i32 7)
   %class1 = call i1 @llvm.is.fpclass.f32(float %a, i32 7)
@@ -1138,7 +1283,7 @@ define i1 @test_fold_and_class_f32_4(float %a) {
 
 define i1 @test_no_fold_and_class_f32_0(float %a, float %b) {
 ; CHECK-LABEL: @test_no_fold_and_class_f32_0(
-; CHECK-NEXT:    [[CLASS0:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A:%.*]], i32 7)
+; CHECK-NEXT:    [[CLASS0:%.*]] = fcmp ueq float [[A:%.*]], 0xFFF0000000000000
 ; CHECK-NEXT:    [[CLASS1:%.*]] = call i1 @llvm.is.fpclass.f32(float [[B:%.*]], i32 15)
 ; CHECK-NEXT:    [[AND:%.*]] = and i1 [[CLASS0]], [[CLASS1]]
 ; CHECK-NEXT:    ret i1 [[AND]]
@@ -1151,8 +1296,8 @@ define i1 @test_no_fold_and_class_f32_0(float %a, float %b) {
 
 define <2 x i1> @test_fold_and_class_v2f32(<2 x float> %a) {
 ; CHECK-LABEL: @test_fold_and_class_v2f32(
-; CHECK-NEXT:    [[CLASS0:%.*]] = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> [[A:%.*]], i32 7)
-; CHECK-NEXT:    ret <2 x i1> [[CLASS0]]
+; CHECK-NEXT:    [[CLASS1:%.*]] = fcmp ueq <2 x float> [[A:%.*]], <float 0xFFF0000000000000, float 0xFFF0000000000000>
+; CHECK-NEXT:    ret <2 x i1> [[CLASS1]]
 ;
   %class0 = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> %a, i32 7)
   %class1 = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> %a, i32 15)
@@ -1177,8 +1322,10 @@ define i1 @test_fold_xor_class_f32_0(float %a) {
 
 define i1 @test_fold_xor3_class_f32_0(float %a) {
 ; CHECK-LABEL: @test_fold_xor3_class_f32_0(
-; CHECK-NEXT:    [[CLASS2:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A:%.*]], i32 7)
-; CHECK-NEXT:    ret i1 [[CLASS2]]
+; CHECK-NEXT:    [[CLASS0:%.*]] = fcmp uno float [[A:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[CLASS2:%.*]] = fcmp oeq float [[A]], 0xFFF0000000000000
+; CHECK-NEXT:    [[XOR_1:%.*]] = xor i1 [[CLASS0]], [[CLASS2]]
+; CHECK-NEXT:    ret i1 [[XOR_1]]
 ;
   %class0 = call i1 @llvm.is.fpclass.f32(float %a, i32 1)
   %class1 = call i1 @llvm.is.fpclass.f32(float %a, i32 2)
@@ -1190,7 +1337,24 @@ define i1 @test_fold_xor3_class_f32_0(float %a) {
 
 define i1 @test_fold_xor_all_tests_class_f32_0(float %a) {
 ; CHECK-LABEL: @test_fold_xor_all_tests_class_f32_0(
-; CHECK-NEXT:    ret i1 true
+; CHECK-NEXT:    [[CLASS0:%.*]] = fcmp uno float [[A:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[CLASS2:%.*]] = fcmp oeq float [[A]], 0xFFF0000000000000
+; CHECK-NEXT:    [[CLASS3:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A]], i32 8)
+; CHECK-NEXT:    [[CLASS4:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A]], i32 16)
+; CHECK-NEXT:    [[CLASS5:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A]], i32 32)
+; CHECK-NEXT:    [[CLASS6:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A]], i32 64)
+; CHECK-NEXT:    [[CLASS7:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A]], i32 128)
+; CHECK-NEXT:    [[CLASS8:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A]], i32 256)
+; CHECK-NEXT:    [[CLASS9:%.*]] = fcmp oeq float [[A]], 0x7FF0000000000000
+; CHECK-NEXT:    [[XOR_1:%.*]] = xor i1 [[CLASS0]], [[CLASS2]]
+; CHECK-NEXT:    [[XOR_2:%.*]] = xor i1 [[XOR_1]], [[CLASS3]]
+; CHECK-NEXT:    [[XOR_3:%.*]] = xor i1 [[XOR_2]], [[CLASS4]]
+; CHECK-NEXT:    [[XOR_4:%.*]] = xor i1 [[XOR_3]], [[CLASS5]]
+; CHECK-NEXT:    [[XOR_5:%.*]] = xor i1 [[XOR_4]], [[CLASS6]]
+; CHECK-NEXT:    [[XOR_6:%.*]] = xor i1 [[XOR_5]], [[CLASS7]]
+; CHECK-NEXT:    [[XOR_7:%.*]] = xor i1 [[XOR_6]], [[CLASS8]]
+; CHECK-NEXT:    [[XOR_8:%.*]] = xor i1 [[XOR_7]], [[CLASS9]]
+; CHECK-NEXT:    ret i1 [[XOR_8]]
 ;
   %class0 = call i1 @llvm.is.fpclass.f32(float %a, i32 1)
   %class1 = call i1 @llvm.is.fpclass.f32(float %a, i32 2)
@@ -1216,8 +1380,8 @@ define i1 @test_fold_xor_all_tests_class_f32_0(float %a) {
 
 define i1 @test_fold_xor_class_f32_1(float %a) {
 ; CHECK-LABEL: @test_fold_xor_class_f32_1(
-; CHECK-NEXT:    [[CLASS0:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A:%.*]], i32 12)
-; CHECK-NEXT:    ret i1 [[CLASS0]]
+; CHECK-NEXT:    [[CLASS1:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A:%.*]], i32 12)
+; CHECK-NEXT:    ret i1 [[CLASS1]]
 ;
   %class0 = call i1 @llvm.is.fpclass.f32(float %a, i32 4)
   %class1 = call i1 @llvm.is.fpclass.f32(float %a, i32 8)
@@ -1227,7 +1391,7 @@ define i1 @test_fold_xor_class_f32_1(float %a) {
 
 define i1 @test_no_fold_xor_class_f32_multi_use0(float %a, ptr %ptr) {
 ; CHECK-LABEL: @test_no_fold_xor_class_f32_multi_use0(
-; CHECK-NEXT:    [[CLASS0:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A:%.*]], i32 4)
+; CHECK-NEXT:    [[CLASS0:%.*]] = fcmp oeq float [[A:%.*]], 0xFFF0000000000000
 ; CHECK-NEXT:    store i1 [[CLASS0]], ptr [[PTR:%.*]], align 1
 ; CHECK-NEXT:    [[CLASS1:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A]], i32 8)
 ; CHECK-NEXT:    [[XOR:%.*]] = xor i1 [[CLASS0]], [[CLASS1]]
@@ -1242,7 +1406,7 @@ define i1 @test_no_fold_xor_class_f32_multi_use0(float %a, ptr %ptr) {
 
 define i1 @test_no_fold_xor_class_f32_multi_use1(float %a, ptr %ptr) {
 ; CHECK-LABEL: @test_no_fold_xor_class_f32_multi_use1(
-; CHECK-NEXT:    [[CLASS0:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A:%.*]], i32 4)
+; CHECK-NEXT:    [[CLASS0:%.*]] = fcmp oeq float [[A:%.*]], 0xFFF0000000000000
 ; CHECK-NEXT:    [[CLASS1:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A]], i32 8)
 ; CHECK-NEXT:    store i1 [[CLASS1]], ptr [[PTR:%.*]], align 1
 ; CHECK-NEXT:    [[XOR:%.*]] = xor i1 [[CLASS0]], [[CLASS1]]
@@ -1257,7 +1421,10 @@ define i1 @test_no_fold_xor_class_f32_multi_use1(float %a, ptr %ptr) {
 
 define i1 @test_fold_xor_class_f32_2(float %a) {
 ; CHECK-LABEL: @test_fold_xor_class_f32_2(
-; CHECK-NEXT:    ret i1 false
+; CHECK-NEXT:    [[CLASS0:%.*]] = fcmp ueq float [[A:%.*]], 0xFFF0000000000000
+; CHECK-NEXT:    [[CLASS1:%.*]] = fcmp ueq float [[A]], 0xFFF0000000000000
+; CHECK-NEXT:    [[XOR:%.*]] = xor i1 [[CLASS0]], [[CLASS1]]
+; CHECK-NEXT:    ret i1 [[XOR]]
 ;
   %class0 = call i1 @llvm.is.fpclass.f32(float %a, i32 7)
   %class1 = call i1 @llvm.is.fpclass.f32(float %a, i32 7)
@@ -1267,7 +1434,7 @@ define i1 @test_fold_xor_class_f32_2(float %a) {
 
 define i1 @test_no_fold_xor_class_f32_0(float %a, float %b) {
 ; CHECK-LABEL: @test_no_fold_xor_class_f32_0(
-; CHECK-NEXT:    [[CLASS0:%.*]] = call i1 @llvm.is.fpclass.f32(float [[A:%.*]], i32 4)
+; CHECK-NEXT:    [[CLASS0:%.*]] = fcmp oeq float [[A:%.*]], 0xFFF0000000000000
 ; CHECK-NEXT:    [[CLASS1:%.*]] = call i1 @llvm.is.fpclass.f32(float [[B:%.*]], i32 8)
 ; CHECK-NEXT:    [[XOR:%.*]] = xor i1 [[CLASS0]], [[CLASS1]]
 ; CHECK-NEXT:    ret i1 [[XOR]]
@@ -1280,8 +1447,8 @@ define i1 @test_no_fold_xor_class_f32_0(float %a, float %b) {
 
 define <2 x i1> @test_fold_xor_class_v2f32(<2 x float> %a) {
 ; CHECK-LABEL: @test_fold_xor_class_v2f32(
-; CHECK-NEXT:    [[CLASS0:%.*]] = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> [[A:%.*]], i32 9)
-; CHECK-NEXT:    ret <2 x i1> [[CLASS0]]
+; CHECK-NEXT:    [[CLASS1:%.*]] = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> [[A:%.*]], i32 9)
+; CHECK-NEXT:    ret <2 x i1> [[CLASS1]]
 ;
   %class0 = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> %a, i32 4)
   %class1 = call <2 x i1> @llvm.is.fpclass.v2f32(<2 x float> %a, i32 13)
@@ -1338,7 +1505,7 @@ define i1 @test_class_fneg_qnan(float %arg) {
 ; -> posinf
 define i1 @test_class_fneg_neginf(float %arg) {
 ; CHECK-LABEL: @test_class_fneg_neginf(
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f32(float [[ARG:%.*]], i32 512)
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp oeq float [[ARG:%.*]], 0x7FF0000000000000
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %fneg = fneg float %arg
@@ -1415,7 +1582,7 @@ define i1 @test_class_fneg_posnormal(float %arg) {
 ; -> neginf
 define i1 @test_class_fneg_posinf(float %arg) {
 ; CHECK-LABEL: @test_class_fneg_posinf(
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f32(float [[ARG:%.*]], i32 4)
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp oeq float [[ARG:%.*]], 0xFFF0000000000000
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %fneg = fneg float %arg
@@ -1460,7 +1627,7 @@ define i1 @test_class_fneg_normal(float %arg) {
 define i1 @test_class_fneg_zero(float %arg) {
 ;
 ; CHECK-LABEL: @test_class_fneg_zero(
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f32(float [[ARG:%.*]], i32 96)
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp oeq float [[ARG:%.*]], 0.000000e+00
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %fneg = fneg float %arg
@@ -1715,7 +1882,7 @@ define i1 @test_class_fabs_negzero(float %arg) {
 ; -> poszero
 define i1 @test_class_fabs_poszero(float %arg) {
 ; CHECK-LABEL: @test_class_fabs_poszero(
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f32(float [[ARG:%.*]], i32 96)
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp oeq float [[ARG:%.*]], 0.000000e+00
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %fabs = call float @llvm.fabs.f32(float %arg)
@@ -1792,7 +1959,7 @@ define i1 @test_class_fabs_normal(float %arg) {
 ; -> poszero
 define i1 @test_class_fabs_zero(float %arg) {
 ; CHECK-LABEL: @test_class_fabs_zero(
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f32(float [[ARG:%.*]], i32 96)
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp oeq float [[ARG:%.*]], 0.000000e+00
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %fabs = call float @llvm.fabs.f32(float %arg)
@@ -2038,7 +2205,7 @@ define i1 @test_class_fneg_fabs_negsubnormal(float %arg) {
 
 define i1 @test_class_fneg_fabs_negzero(float %arg) {
 ; CHECK-LABEL: @test_class_fneg_fabs_negzero(
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f32(float [[ARG:%.*]], i32 96)
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp oeq float [[ARG:%.*]], 0.000000e+00
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %fabs = call float @llvm.fabs.f32(float %arg)
@@ -2122,7 +2289,7 @@ define i1 @test_class_fneg_fabs_normal(float %arg) {
 
 define i1 @test_class_fneg_fabs_zero(float %arg) {
 ; CHECK-LABEL: @test_class_fneg_fabs_zero(
-; CHECK-NEXT:    [[CLASS:%.*]] = call i1 @llvm.is.fpclass.f32(float [[ARG:%.*]], i32 96)
+; CHECK-NEXT:    [[CLASS:%.*]] = fcmp oeq float [[ARG:%.*]], 0.000000e+00
 ; CHECK-NEXT:    ret i1 [[CLASS]]
 ;
   %fabs = call float @llvm.fabs.f32(float %arg)
diff --git a/llvm/test/Transforms/InstCombine/logical-select.ll b/llvm/test/Transforms/InstCombine/logical-select.ll
index 14782732d8666..2bdb3021523fb 100644
--- a/llvm/test/Transforms/InstCombine/logical-select.ll
+++ b/llvm/test/Transforms/InstCombine/logical-select.ll
@@ -1126,8 +1126,8 @@ define i1 @not_d_bools_negative_use2(i1 %c, i1 %x, i1 %y) {
 }
 
 ; A & (~C | B)
-define i1 @test1(i1 %a, i1 %b, i1 %c) {
-; CHECK-LABEL: @test1(
+define i1 @logical_and_or_with_not_op(i1 %a, i1 %b, i1 %c) {
+; CHECK-LABEL: @logical_and_or_with_not_op(
 ; CHECK-NEXT:    [[NOT:%.*]] = xor i1 [[C:%.*]], true
 ; CHECK-NEXT:    [[OR:%.*]] = or i1 [[NOT]], [[B:%.*]]
 ; CHECK-NEXT:    [[AND:%.*]] = select i1 [[A:%.*]], i1 [[OR]], i1 false
@@ -1139,13 +1139,11 @@ define i1 @test1(i1 %a, i1 %b, i1 %c) {
   ret i1 %and
 }
 
-; As test1 but with C=A
+; As logical_and_or_with_not_op but with C=A
 ; A & (~A | B) --> A & B
-define i1 @test1_variant1(i1 %a, i1 %b) {
-; CHECK-LABEL: @test1_variant1(
-; CHECK-NEXT:    [[NOT:%.*]] = xor i1 [[A:%.*]], true
-; CHECK-NEXT:    [[OR:%.*]] = or i1 [[NOT]], [[B:%.*]]
-; CHECK-NEXT:    [[AND:%.*]] = select i1 [[A]], i1 [[OR]], i1 false
+define i1 @logical_and_or_with_common_not_op_variant1(i1 %a, i1 %b) {
+; CHECK-LABEL: @logical_and_or_with_common_not_op_variant1(
+; CHECK-NEXT:    [[AND:%.*]] = select i1 [[A:%.*]], i1 [[B:%.*]], i1 false
 ; CHECK-NEXT:    ret i1 [[AND]]
 ;
   %not = xor i1 %a, true
@@ -1154,13 +1152,11 @@ define i1 @test1_variant1(i1 %a, i1 %b) {
   ret i1 %and
 }
 
-; As test1_variant1 but operating on vectors
+; As logical_and_or_with_common_not_op_variant1 but operating on vectors
 ; A & (~A | B) --> A & B
-define <2 x i1> @test1_variant2(<2 x i1> %a, <2 x i1> %b) {
-; CHECK-LABEL: @test1_variant2(
-; CHECK-NEXT:    [[NOT:%.*]] = xor <2 x i1> [[A:%.*]], <i1 true, i1 true>
-; CHECK-NEXT:    [[OR:%.*]] = or <2 x i1> [[NOT]], [[B:%.*]]
-; CHECK-NEXT:    [[AND:%.*]] = select <2 x i1> [[A]], <2 x i1> [[OR]], <2 x i1> zeroinitializer
+define <2 x i1> @logical_and_or_with_common_not_op_variant2(<2 x i1> %a, <2 x i1> %b) {
+; CHECK-LABEL: @logical_and_or_with_common_not_op_variant2(
+; CHECK-NEXT:    [[AND:%.*]] = select <2 x i1> [[A:%.*]], <2 x i1> [[B:%.*]], <2 x i1> zeroinitializer
 ; CHECK-NEXT:    ret <2 x i1> [[AND]]
 ;
   %not = xor <2 x i1> %a, <i1 true, i1 true>
@@ -1169,13 +1165,12 @@ define <2 x i1> @test1_variant2(<2 x i1> %a, <2 x i1> %b) {
   ret <2 x i1> %and
 }
 
-; As test1_variant1 but with "or" implemented as "select X, true, Y"
+; As logical_and_or_with_common_not_op_variant1 but with "or" implemented as
+; "select X, true, Y"
 ; A & (~A | B) --> A & B
-define i1 @test1_variant3(i1 %a, i1 %b) {
-; CHECK-LABEL: @test1_variant3(
-; CHECK-NEXT:    [[NOT:%.*]] = xor i1 [[A:%.*]], true
-; CHECK-NEXT:    [[OR:%.*]] = select i1 [[NOT]], i1 true, i1 [[B:%.*]]
-; CHECK-NEXT:    [[AND:%.*]] = select i1 [[A]], i1 [[OR]], i1 false
+define i1 @logical_and_or_with_common_not_op_variant3(i1 %a, i1 %b) {
+; CHECK-LABEL: @logical_and_or_with_common_not_op_variant3(
+; CHECK-NEXT:    [[AND:%.*]] = select i1 [[A:%.*]], i1 [[B:%.*]], i1 false
 ; CHECK-NEXT:    ret i1 [[AND]]
 ;
   %not = xor i1 %a, true
@@ -1184,13 +1179,14 @@ define i1 @test1_variant3(i1 %a, i1 %b) {
   ret i1 %and
 }
 
-; As test1_variant3 but operating on vectors where each operand has other uses
+; As logical_and_or_with_common_not_op_variant3 but operating on vectors where
+; each operand has other uses
 ; A & (~A | B) --> A & B
-define <2 x i1> @test1_variant4(<2 x i1> %a, <2 x i1> %b) {
-; CHECK-LABEL: @test1_variant4(
+define <2 x i1> @logical_and_or_with_common_not_op_variant4(<2 x i1> %a, <2 x i1> %b) {
+; CHECK-LABEL: @logical_and_or_with_common_not_op_variant4(
 ; CHECK-NEXT:    [[NOT:%.*]] = xor <2 x i1> [[A:%.*]], <i1 true, i1 true>
 ; CHECK-NEXT:    [[OR:%.*]] = select <2 x i1> [[NOT]], <2 x i1> <i1 true, i1 true>, <2 x i1> [[B:%.*]]
-; CHECK-NEXT:    [[AND:%.*]] = select <2 x i1> [[A]], <2 x i1> [[OR]], <2 x i1> zeroinitializer
+; CHECK-NEXT:    [[AND:%.*]] = select <2 x i1> [[A]], <2 x i1> [[B]], <2 x i1> zeroinitializer
 ; CHECK-NEXT:    call void @use2(<2 x i1> [[A]])
 ; CHECK-NEXT:    call void @use2(<2 x i1> [[B]])
 ; CHECK-NEXT:    call void @use2(<2 x i1> [[OR]])
@@ -1205,14 +1201,12 @@ define <2 x i1> @test1_variant4(<2 x i1> %a, <2 x i1> %b) {
   ret <2 x i1> %and
 }
 
-; As test1_variant1 but with |'s operands swapped
+; As logical_and_or_with_common_not_op_variant1 but with |'s operands swapped
 ; A & (B | ~A) --> A & B
-define i1 @test1_variant5(i1 %a) {
-; CHECK-LABEL: @test1_variant5(
+define i1 @logical_and_or_with_common_not_op_variant5(i1 %a) {
+; CHECK-LABEL: @logical_and_or_with_common_not_op_variant5(
 ; CHECK-NEXT:    [[B:%.*]] = call i1 @gen()
-; CHECK-NEXT:    [[NOT:%.*]] = xor i1 [[A:%.*]], true
-; CHECK-NEXT:    [[OR:%.*]] = or i1 [[B]], [[NOT]]
-; CHECK-NEXT:    [[AND:%.*]] = select i1 [[A]], i1 [[OR]], i1 false
+; CHECK-NEXT:    [[AND:%.*]] = select i1 [[A:%.*]], i1 [[B]], i1 false
 ; CHECK-NEXT:    ret i1 [[AND]]
 ;
   %b = call i1 @gen()
@@ -1223,8 +1217,8 @@ define i1 @test1_variant5(i1 %a) {
 }
 
 ; A | (~C & B)
-define i1 @test2(i1 %a, i1 %b, i1 %c) {
-; CHECK-LABEL: @test2(
+define i1 @logical_or_and_with_not_op(i1 %a, i1 %b, i1 %c) {
+; CHECK-LABEL: @logical_or_and_with_not_op(
 ; CHECK-NEXT:    [[NOT:%.*]] = xor i1 [[C:%.*]], true
 ; CHECK-NEXT:    [[AND:%.*]] = and i1 [[NOT]], [[B:%.*]]
 ; CHECK-NEXT:    [[OR:%.*]] = select i1 [[A:%.*]], i1 true, i1 [[AND]]
@@ -1236,13 +1230,11 @@ define i1 @test2(i1 %a, i1 %b, i1 %c) {
   ret i1 %or
 }
 
-; As test2 but with C=A
+; As logical_or_and_with_not_op but with C=A
 ; A | (~A & B) --> A | B
-define i1 @test2_variant1(i1 %a, i1 %b) {
-; CHECK-LABEL: @test2_variant1(
-; CHECK-NEXT:    [[NOT:%.*]] = xor i1 [[A:%.*]], true
-; CHECK-NEXT:    [[AND:%.*]] = and i1 [[NOT]], [[B:%.*]]
-; CHECK-NEXT:    [[OR:%.*]] = select i1 [[A]], i1 true, i1 [[AND]]
+define i1 @logical_or_and_with_common_not_op_variant1(i1 %a, i1 %b) {
+; CHECK-LABEL: @logical_or_and_with_common_not_op_variant1(
+; CHECK-NEXT:    [[OR:%.*]] = select i1 [[A:%.*]], i1 true, i1 [[B:%.*]]
 ; CHECK-NEXT:    ret i1 [[OR]]
 ;
   %not = xor i1 %a, true
@@ -1251,13 +1243,11 @@ define i1 @test2_variant1(i1 %a, i1 %b) {
   ret i1 %or
 }
 
-; As test2_variant1 but operating on vectors
+; As logical_or_and_with_common_not_op_variant1 but operating on vectors
 ; A | (~A & B) --> A | B
-define <2 x i1> @test2_variant2(<2 x i1> %a, <2 x i1> %b) {
-; CHECK-LABEL: @test2_variant2(
-; CHECK-NEXT:    [[NOT:%.*]] = xor <2 x i1> [[A:%.*]], <i1 true, i1 true>
-; CHECK-NEXT:    [[AND:%.*]] = and <2 x i1> [[NOT]], [[B:%.*]]
-; CHECK-NEXT:    [[OR:%.*]] = select <2 x i1> [[A]], <2 x i1> <i1 true, i1 true>, <2 x i1> [[AND]]
+define <2 x i1> @logical_or_and_with_common_not_op_variant2(<2 x i1> %a, <2 x i1> %b) {
+; CHECK-LABEL: @logical_or_and_with_common_not_op_variant2(
+; CHECK-NEXT:    [[OR:%.*]] = select <2 x i1> [[A:%.*]], <2 x i1> <i1 true, i1 true>, <2 x i1> [[B:%.*]]
 ; CHECK-NEXT:    ret <2 x i1> [[OR]]
 ;
   %not = xor <2 x i1> %a, <i1 true, i1 true>
@@ -1266,13 +1256,12 @@ define <2 x i1> @test2_variant2(<2 x i1> %a, <2 x i1> %b) {
   ret <2 x i1> %or
 }
 
-; As test2_variant1 but with "and" implemented as "select X, Y, false"
+; As logical_or_and_with_common_not_op_variant1 but with "and" implemented as
+; "select X, Y, false"
 ; A | (~A & B) --> A | B
-define i1 @test2_variant3(i1 %a, i1 %b) {
-; CHECK-LABEL: @test2_variant3(
-; CHECK-NEXT:    [[NOT:%.*]] = xor i1 [[A:%.*]], true
-; CHECK-NEXT:    [[AND:%.*]] = select i1 [[NOT]], i1 [[B:%.*]], i1 false
-; CHECK-NEXT:    [[OR:%.*]] = select i1 [[A]], i1 true, i1 [[AND]]
+define i1 @logical_or_and_with_common_not_op_variant3(i1 %a, i1 %b) {
+; CHECK-LABEL: @logical_or_and_with_common_not_op_variant3(
+; CHECK-NEXT:    [[OR:%.*]] = select i1 [[A:%.*]], i1 true, i1 [[B:%.*]]
 ; CHECK-NEXT:    ret i1 [[OR]]
 ;
   %not = xor i1 %a, true
@@ -1281,13 +1270,14 @@ define i1 @test2_variant3(i1 %a, i1 %b) {
   ret i1 %or
 }
 
-; As test2_variant3 but operating on vectors where each operand has other uses
+; As logical_or_and_with_common_not_op_variant3 but operating on vectors where
+; each operand has other uses
 ; A | (~A & B) --> A | B
-define <2 x i1> @test2_variant4(<2 x i1> %a, <2 x i1> %b) {
-; CHECK-LABEL: @test2_variant4(
+define <2 x i1> @logical_or_and_with_common_not_op_variant4(<2 x i1> %a, <2 x i1> %b) {
+; CHECK-LABEL: @logical_or_and_with_common_not_op_variant4(
 ; CHECK-NEXT:    [[NOT:%.*]] = xor <2 x i1> [[A:%.*]], <i1 true, i1 true>
 ; CHECK-NEXT:    [[AND:%.*]] = select <2 x i1> [[NOT]], <2 x i1> [[B:%.*]], <2 x i1> zeroinitializer
-; CHECK-NEXT:    [[OR:%.*]] = select <2 x i1> [[A]], <2 x i1> <i1 true, i1 true>, <2 x i1> [[AND]]
+; CHECK-NEXT:    [[OR:%.*]] = select <2 x i1> [[A]], <2 x i1> <i1 true, i1 true>, <2 x i1> [[B]]
 ; CHECK-NEXT:    call void @use2(<2 x i1> [[A]])
 ; CHECK-NEXT:    call void @use2(<2 x i1> [[B]])
 ; CHECK-NEXT:    call void @use2(<2 x i1> [[AND]])
@@ -1302,14 +1292,12 @@ define <2 x i1> @test2_variant4(<2 x i1> %a, <2 x i1> %b) {
   ret <2 x i1> %or
 }
 
-; As test2_variant1 but with &'s operands swapped
+; As logical_or_and_with_common_not_op_variant1 but with &'s operands swapped
 ; A | (B & ~A) --> A | B
-define i1 @test2_variant5(i1 %a) {
-; CHECK-LABEL: @test2_variant5(
+define i1 @logical_or_and_with_common_not_op_variant5(i1 %a) {
+; CHECK-LABEL: @logical_or_and_with_common_not_op_variant5(
 ; CHECK-NEXT:    [[B:%.*]] = call i1 @gen()
-; CHECK-NEXT:    [[NOT:%.*]] = xor i1 [[A:%.*]], true
-; CHECK-NEXT:    [[AND:%.*]] = and i1 [[B]], [[NOT]]
-; CHECK-NEXT:    [[OR:%.*]] = select i1 [[A]], i1 true, i1 [[AND]]
+; CHECK-NEXT:    [[OR:%.*]] = select i1 [[A:%.*]], i1 true, i1 [[B]]
 ; CHECK-NEXT:    ret i1 [[OR]]
 ;
   %b = call i1 @gen()
diff --git a/llvm/test/Transforms/InstCombine/pr28725.ll b/llvm/test/Transforms/InstCombine/pr28725.ll
index 26772eb7fe3af..688409c41f3c1 100644
--- a/llvm/test/Transforms/InstCombine/pr28725.ll
+++ b/llvm/test/Transforms/InstCombine/pr28725.ll
@@ -3,7 +3,8 @@
 
 define <2 x i16> @test1() {
 entry:
-  %e = extractvalue %S select (i1 icmp eq (i16 extractelement (<2 x i16> bitcast (<1 x i32> <i32 1> to <2 x i16>), i32 0), i16 0), %S zeroinitializer, %S { i16 0, i32 1 }), 0
+  %s = select i1 icmp eq (i16 extractelement (<2 x i16> bitcast (<1 x i32> <i32 1> to <2 x i16>), i32 0), i16 0), %S zeroinitializer, %S { i16 0, i32 1 }
+  %e = extractvalue %S %s, 0
   %b = insertelement <2 x i16> <i16 undef, i16 0>, i16 %e, i32 0
   ret <2 x i16> %b
 }
diff --git a/llvm/test/Transforms/InstCombine/rem-mul-shl.ll b/llvm/test/Transforms/InstCombine/rem-mul-shl.ll
new file mode 100644
index 0000000000000..c7cb9c954bb3a
--- /dev/null
+++ b/llvm/test/Transforms/InstCombine/rem-mul-shl.ll
@@ -0,0 +1,807 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -passes=instcombine -S | FileCheck %s
+declare void @use8(i8)
+
+define i8 @srem_non_matching(i8 %X, i8 %Y) {
+; CHECK-LABEL: @srem_non_matching(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw nsw i8 [[X:%.*]], 15
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw i8 [[Y:%.*]], 5
+; CHECK-NEXT:    [[R:%.*]] = srem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nsw nuw i8 %X, 15
+  %BO1 = mul nsw nuw i8 %Y, 5
+  %r = srem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_1_shl(i8 %X, i8 %Y) {
+; CHECK-LABEL: @urem_1_shl(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 1, [[X:%.*]]
+; CHECK-NEXT:    [[NOTMASK:%.*]] = shl nsw i8 -1, [[Y:%.*]]
+; CHECK-NEXT:    [[TMP1:%.*]] = xor i8 [[NOTMASK]], -1
+; CHECK-NEXT:    [[R:%.*]] = and i8 [[BO0]], [[TMP1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nsw nuw i8 1, %X
+  %BO1 = shl nsw nuw i8 1, %Y
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define <vscale x 16 x i8> @urem_XY_XZ_with_CY_rem_CZ_eq_0_scalable(<vscale x 16 x i8> %X) {
+; CHECK-LABEL: @urem_XY_XZ_with_CY_rem_CZ_eq_0_scalable(
+; CHECK-NEXT:    ret <vscale x 16 x i8> zeroinitializer
+;
+  %BO0 = mul nuw <vscale x 16 x i8> %X, shufflevector(<vscale x 16 x i8> insertelement(<vscale x 16 x i8> poison, i8 15, i64 0) , <vscale x 16 x i8> poison, <vscale x 16 x i32> zeroinitializer)
+  %BO1 = mul <vscale x 16 x i8> %X, shufflevector(<vscale x 16 x i8> insertelement(<vscale x 16 x i8> poison, i8 5, i64 0) , <vscale x 16 x i8> poison, <vscale x 16 x i32> zeroinitializer)
+  %r = urem <vscale x 16 x i8> %BO0, %BO1
+  ret <vscale x 16 x i8> %r
+}
+
+define i8 @urem_XY_XZ_with_CY_rem_CZ_eq_0(i8 %X) {
+; CHECK-LABEL: @urem_XY_XZ_with_CY_rem_CZ_eq_0(
+; CHECK-NEXT:    ret i8 0
+;
+  %BO0 = mul nuw i8 %X, 15
+  %BO1 = mul i8 %X, 5
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_XY_XZ_with_CY_rem_CZ_eq_0_fail_missing_flag(i8 %X) {
+; CHECK-LABEL: @urem_XY_XZ_with_CY_rem_CZ_eq_0_fail_missing_flag(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nsw i8 [[X:%.*]], 15
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw i8 [[X]], 5
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nsw i8 %X, 15
+  %BO1 = mul nsw nuw i8 %X, 5
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_XY_XZ_with_CY_lt_CZ(i8 %X) {
+; CHECK-LABEL: @urem_XY_XZ_with_CY_lt_CZ(
+; CHECK-NEXT:    [[R:%.*]] = mul nuw i8 [[X:%.*]], 3
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul i8 %X, 3
+  %BO1 = mul nuw i8 %X, 12
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define <2 x i8> @urem_XY_XZ_with_CY_lt_CZ_with_nsw_out(<2 x i8> %X) {
+; CHECK-LABEL: @urem_XY_XZ_with_CY_lt_CZ_with_nsw_out(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nsw <2 x i8> [[X:%.*]], <i8 2, i8 2>
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw <2 x i8> [[X]], <i8 12, i8 12>
+; CHECK-NEXT:    [[R:%.*]] = urem <2 x i8> [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret <2 x i8> [[R]]
+;
+  %BO0 = shl nsw <2 x i8> %X, <i8 2, i8 2>
+  %BO1 = mul nuw <2 x i8> %X, <i8 12, i8 12>
+  %r = urem <2 x i8> %BO0, %BO1
+  ret <2 x i8> %r
+}
+
+define i8 @urem_XY_XZ_with_CY_lt_CZ_no_nsw_out(i8 %X) {
+; CHECK-LABEL: @urem_XY_XZ_with_CY_lt_CZ_no_nsw_out(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw i8 [[X:%.*]], 3
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[X]], 3
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw i8 %X, 3
+  %BO1 = shl nsw nuw i8 %X, 3
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_XY_XZ_with_CY_lt_CZ_fail_missing_flag(i8 %X) {
+; CHECK-LABEL: @urem_XY_XZ_with_CY_lt_CZ_fail_missing_flag(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw nsw i8 [[X:%.*]], 3
+; CHECK-NEXT:    [[BO1:%.*]] = mul nsw i8 [[X]], 12
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw nsw i8 %X, 3
+  %BO1 = mul nsw i8 %X, 12
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_XY_XZ_with_CY_gt_CZ(i8 %X) {
+; CHECK-LABEL: @urem_XY_XZ_with_CY_gt_CZ(
+; CHECK-NEXT:    [[R:%.*]] = mul nuw nsw i8 [[X:%.*]], 3
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw i8 %X, 21
+  %BO1 = mul i8 %X, 6
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_XY_XZ_with_CY_gt_CZ_fail_missing_flag(i8 %X) {
+; CHECK-LABEL: @urem_XY_XZ_with_CY_gt_CZ_fail_missing_flag(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nsw i8 [[X:%.*]], 21
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw i8 [[X]], 6
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nsw i8 %X, 21
+  %BO1 = mul nsw nuw i8 %X, 6
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_XY_XZ_with_Y_Z_is_mul_X_RemYZ(i8 %X, i8 %Y, i8 %Z) {
+; CHECK-LABEL: @urem_XY_XZ_with_Y_Z_is_mul_X_RemYZ(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw i8 [[X:%.*]], [[Y:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw i8 [[Z:%.*]], [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw i8 %X, %Y
+  %BO1 = mul nuw i8 %Z, %X
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_XY_XZ_with_CX_Y_Z_is_mul_X_RemYZ(i8 %Y, i8 %Z) {
+; CHECK-LABEL: @urem_XY_XZ_with_CX_Y_Z_is_mul_X_RemYZ(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw i8 [[Y:%.*]], 10
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw i8 10, [[Z:%.*]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw i8 10, %Y
+  %BO1 = shl nuw i8 10, %Z
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_XY_XZ_with_Y_Z_is_mul_X_RemYZ_with_nsw_out1(i8 %X, i8 %Y, i8 %Z) {
+; CHECK-LABEL: @urem_XY_XZ_with_Y_Z_is_mul_X_RemYZ_with_nsw_out1(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw nsw i8 [[X:%.*]], [[Y:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw i8 [[X]], [[Z:%.*]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw nsw i8 %X, %Y
+  %BO1 = shl nuw i8 %X, %Z
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define <2 x i8> @urem_XY_XZ_with_Y_Z_is_mul_X_RemYZ_with_nsw_out2(<2 x i8> %X, <2 x i8> %Y, <2 x i8> %Z) {
+; CHECK-LABEL: @urem_XY_XZ_with_Y_Z_is_mul_X_RemYZ_with_nsw_out2(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw <2 x i8> [[Y:%.*]], [[X:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw <2 x i8> [[Z:%.*]], [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem <2 x i8> [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret <2 x i8> [[R]]
+;
+  %BO0 = shl nuw <2 x i8> %Y, %X
+  %BO1 = shl nuw nsw <2 x i8> %Z, %X
+  %r = urem <2 x i8> %BO0, %BO1
+  ret <2 x i8> %r
+}
+
+define i8 @urem_XY_XZ_with_Y_Z_is_mul_X_RemYZ_fail_reused1(i8 %X, i8 %Y, i8 %Z) {
+; CHECK-LABEL: @urem_XY_XZ_with_Y_Z_is_mul_X_RemYZ_fail_reused1(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw nsw i8 [[X:%.*]], [[Y:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw i8 [[X]], [[Z:%.*]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    call void @use8(i8 [[BO0]])
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nsw nuw i8 %X, %Y
+  %BO1 = mul nsw nuw i8 %X, %Z
+  %r = urem i8 %BO0, %BO1
+  call void @use8(i8 %BO0)
+  ret i8 %r
+}
+
+define <2 x i8> @urem_XY_XZ_with_Y_Z_is_mul_X_RemYZ_fail_missing_flags1(<2 x i8> %X, <2 x i8> %Y, <2 x i8> %Z) {
+; CHECK-LABEL: @urem_XY_XZ_with_Y_Z_is_mul_X_RemYZ_fail_missing_flags1(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nsw <2 x i8> [[X:%.*]], [[Y:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw <2 x i8> [[X]], [[Z:%.*]]
+; CHECK-NEXT:    [[R:%.*]] = urem <2 x i8> [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret <2 x i8> [[R]]
+;
+  %BO0 = mul nsw <2 x i8> %X, %Y
+  %BO1 = mul nsw nuw <2 x i8> %X, %Z
+  %r = urem <2 x i8> %BO0, %BO1
+  ret <2 x i8> %r
+}
+
+define i8 @urem_XY_XZ_with_Y_Z_is_mul_X_RemYZ_fail_missing_flags2(i8 %X, i8 %Y, i8 %Z) {
+; CHECK-LABEL: @urem_XY_XZ_with_Y_Z_is_mul_X_RemYZ_fail_missing_flags2(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw nsw i8 [[X:%.*]], [[Y:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nsw i8 [[X]], [[Z:%.*]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nsw nuw i8 %X, %Y
+  %BO1 = shl nsw i8 %X, %Z
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+;; Signed Verions
+define <vscale x 16 x i8> @srem_XY_XZ_with_CY_rem_CZ_eq_0_scalable(<vscale x 16 x i8> %X) {
+; CHECK-LABEL: @srem_XY_XZ_with_CY_rem_CZ_eq_0_scalable(
+; CHECK-NEXT:    ret <vscale x 16 x i8> zeroinitializer
+;
+  %BO0 = mul nsw <vscale x 16 x i8> %X, shufflevector(<vscale x 16 x i8> insertelement(<vscale x 16 x i8> poison, i8 15, i64 0) , <vscale x 16 x i8> poison, <vscale x 16 x i32> zeroinitializer)
+  %BO1 = mul <vscale x 16 x i8> %X, shufflevector(<vscale x 16 x i8> insertelement(<vscale x 16 x i8> poison, i8 5, i64 0) , <vscale x 16 x i8> poison, <vscale x 16 x i32> zeroinitializer)
+  %r = srem <vscale x 16 x i8> %BO0, %BO1
+  ret <vscale x 16 x i8> %r
+}
+
+define i8 @srem_XY_XZ_with_CY_rem_CZ_eq_0(i8 %X) {
+; CHECK-LABEL: @srem_XY_XZ_with_CY_rem_CZ_eq_0(
+; CHECK-NEXT:    ret i8 0
+;
+  %BO0 = mul nsw i8 %X, 9
+  %BO1 = mul i8 %X, 3
+  %r = srem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @srem_XY_XZ_with_CY_rem_CZ_eq_0_fail_missing_flag(i8 %X) {
+; CHECK-LABEL: @srem_XY_XZ_with_CY_rem_CZ_eq_0_fail_missing_flag(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw i8 [[X:%.*]], 9
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw i8 [[X]], 3
+; CHECK-NEXT:    [[R:%.*]] = srem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw i8 %X, 9
+  %BO1 = mul nsw nuw i8 %X, 3
+  %r = srem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define <2 x i8> @srem_XY_XZ_with_CY_lt_CZ(<2 x i8> %X) {
+; CHECK-LABEL: @srem_XY_XZ_with_CY_lt_CZ(
+; CHECK-NEXT:    [[BO0:%.*]] = shl <2 x i8> [[X:%.*]], <i8 3, i8 3>
+; CHECK-NEXT:    [[BO1:%.*]] = mul nsw <2 x i8> [[X]], <i8 15, i8 15>
+; CHECK-NEXT:    [[R:%.*]] = srem <2 x i8> [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret <2 x i8> [[R]]
+;
+  %BO0 = shl <2 x i8> %X, <i8 3, i8 3>
+  %BO1 = mul nsw <2 x i8> %X, <i8 15, i8 15>
+  %r = srem <2 x i8> %BO0, %BO1
+  ret <2 x i8> %r
+}
+
+define i8 @srem_XY_XZ_with_CY_lt_CZ_with_nuw_out(i8 %X) {
+; CHECK-LABEL: @srem_XY_XZ_with_CY_lt_CZ_with_nuw_out(
+; CHECK-NEXT:    [[R:%.*]] = mul nuw nsw i8 [[X:%.*]], 5
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw i8 %X, 5
+  %BO1 = mul nsw i8 %X, 15
+  %r = srem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @srem_XY_XZ_with_CY_lt_CZ_no_nsw_out(i8 %X) {
+; CHECK-LABEL: @srem_XY_XZ_with_CY_lt_CZ_no_nsw_out(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nsw i8 [[X:%.*]], 5
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[X]], 4
+; CHECK-NEXT:    [[R:%.*]] = srem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nsw i8 %X, 5
+  %BO1 = shl nsw nuw i8 %X, 4
+  %r = srem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @srem_XY_XZ_with_CY_lt_CZ_fail_missing_flag(i8 %X) {
+; CHECK-LABEL: @srem_XY_XZ_with_CY_lt_CZ_fail_missing_flag(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw nsw i8 [[X:%.*]], 5
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw i8 [[X]], 4
+; CHECK-NEXT:    [[R:%.*]] = srem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw nsw i8 %X, 5
+  %BO1 = shl nuw i8 %X, 4
+  %r = srem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @srem_XY_XZ_with_CY_gt_CZ(i8 %X) {
+; CHECK-LABEL: @srem_XY_XZ_with_CY_gt_CZ(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nsw i8 [[X:%.*]], 3
+; CHECK-NEXT:    [[BO1:%.*]] = mul nsw i8 [[X]], 6
+; CHECK-NEXT:    [[R:%.*]] = srem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nsw i8 %X, 3
+  %BO1 = mul nsw i8 %X, 6
+  %r = srem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @srem_XY_XZ_with_CY_gt_CZ_with_nuw_out(i8 %X) {
+; CHECK-LABEL: @srem_XY_XZ_with_CY_gt_CZ_with_nuw_out(
+; CHECK-NEXT:    [[R:%.*]] = shl nuw nsw i8 [[X:%.*]], 2
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nsw nuw i8 %X, 10
+  %BO1 = mul nsw i8 %X, 6
+  %r = srem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define <2 x i8> @srem_XY_XZ_with_CY_gt_CZ_no_nuw_out(<2 x i8> %X) {
+; CHECK-LABEL: @srem_XY_XZ_with_CY_gt_CZ_no_nuw_out(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nsw <2 x i8> [[X:%.*]], <i8 10, i8 10>
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw <2 x i8> [[X]], <i8 3, i8 3>
+; CHECK-NEXT:    [[R:%.*]] = srem <2 x i8> [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret <2 x i8> [[R]]
+;
+  %BO0 = mul nsw <2 x i8> %X, <i8 10, i8 10>
+  %BO1 = shl nsw nuw <2 x i8> %X, <i8 3, i8 3>
+  %r = srem <2 x i8> %BO0, %BO1
+  ret <2 x i8> %r
+}
+
+define i8 @srem_XY_XZ_with_CY_gt_CZ_fail_missing_flag1(i8 %X) {
+; CHECK-LABEL: @srem_XY_XZ_with_CY_gt_CZ_fail_missing_flag1(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw nsw i8 [[X:%.*]], 10
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw i8 [[X]], 6
+; CHECK-NEXT:    [[R:%.*]] = srem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nsw nuw i8 %X, 10
+  %BO1 = mul nuw i8 %X, 6
+  %r = srem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @srem_XY_XZ_with_CY_gt_CZ_fail_missing_flag2(i8 %X) {
+; CHECK-LABEL: @srem_XY_XZ_with_CY_gt_CZ_fail_missing_flag2(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw i8 [[X:%.*]], 4
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw i8 [[X]], 5
+; CHECK-NEXT:    [[R:%.*]] = srem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw i8 %X, 4
+  %BO1 = mul nsw nuw i8 %X, 5
+  %r = srem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @srem_XY_XZ_with_Y_Z_is_mul_X_RemYZ(i8 %X, i8 %Y, i8 %Z) {
+; CHECK-LABEL: @srem_XY_XZ_with_Y_Z_is_mul_X_RemYZ(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nsw i8 [[Y:%.*]], [[X:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw i8 [[X]], [[Z:%.*]]
+; CHECK-NEXT:    [[R:%.*]] = srem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nsw i8 %Y, %X
+  %BO1 = mul nsw nuw i8 %X, %Z
+  %r = srem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @srem_XY_XZ_with_Y_Z_is_mul_X_RemYZ_with_nuw_out(i8 %X, i8 %Y, i8 %Z) {
+; CHECK-LABEL: @srem_XY_XZ_with_Y_Z_is_mul_X_RemYZ_with_nuw_out(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw nsw i8 [[Y:%.*]], [[X:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw i8 [[Z:%.*]], [[X]]
+; CHECK-NEXT:    [[R:%.*]] = srem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nsw nuw i8 %Y, %X
+  %BO1 = mul nsw nuw i8 %Z, %X
+  %r = srem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @srem_XY_XZ_with_Y_Z_is_mul_X_RemYZ_fail_shl(i8 %X, i8 %Y, i8 %Z) {
+; CHECK-LABEL: @srem_XY_XZ_with_Y_Z_is_mul_X_RemYZ_fail_shl(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 [[X:%.*]], [[Y:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[X]], [[Z:%.*]]
+; CHECK-NEXT:    [[R:%.*]] = srem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nsw nuw i8 %X, %Y
+  %BO1 = shl nsw nuw i8 %X, %Z
+  %r = srem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+
+define i8 @srem_XY_XZ_with_Y_Z_is_mul_X_RemYZ_fail_missing_flags1(i8 %X, i8 %Y, i8 %Z) {
+; CHECK-LABEL: @srem_XY_XZ_with_Y_Z_is_mul_X_RemYZ_fail_missing_flags1(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw i8 [[X:%.*]], [[Y:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw i8 [[X]], [[Z:%.*]]
+; CHECK-NEXT:    [[R:%.*]] = srem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw i8 %X, %Y
+  %BO1 = mul nsw nuw i8 %X, %Z
+  %r = srem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @srem_XY_XZ_with_Y_Z_is_mul_X_RemYZ_fail_missing_flags2(i8 %X, i8 %Y, i8 %Z) {
+; CHECK-LABEL: @srem_XY_XZ_with_Y_Z_is_mul_X_RemYZ_fail_missing_flags2(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw nsw i8 [[X:%.*]], [[Y:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw i8 [[X]], [[Z:%.*]]
+; CHECK-NEXT:    [[R:%.*]] = srem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nsw nuw i8 %X, %Y
+  %BO1 = mul nuw i8 %X, %Z
+  %r = srem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_XY_shl_ZX_fail(i8 %X, i8 %Z, i8 %Y) {
+; CHECK-LABEL: @urem_shl_XY_shl_ZX_fail(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 [[X:%.*]], [[Y:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[Z:%.*]], [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 %X, %Y
+  %BO1 = shl nuw nsw i8 %Z, %X
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_mul_XY_shl_ZX_fail(i8 %X, i8 %Z, i8 %Y) {
+; CHECK-LABEL: @urem_mul_XY_shl_ZX_fail(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw nsw i8 [[X:%.*]], [[Y:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[Z:%.*]], [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw nsw i8 %X, %Y
+  %BO1 = shl nuw nsw i8 %Z, %X
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_YX_shl_XZ_fail(i8 %X, i8 %Z, i8 %Y) {
+; CHECK-LABEL: @urem_shl_YX_shl_XZ_fail(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 [[Y:%.*]], [[X:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[X]], [[Z:%.*]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 %Y, %X
+  %BO1 = shl nuw nsw i8 %X, %Z
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_YX_mul_XZ_fail(i8 %X, i8 %Z, i8 %Y) {
+; CHECK-LABEL: @urem_shl_YX_mul_XZ_fail(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 [[Y:%.*]], [[X:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw i8 [[X]], [[Z:%.*]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 %Y, %X
+  %BO1 = mul nuw nsw i8 %X, %Z
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_YX_mul_ZX_fail(i8 %X, i8 %Z, i8 %Y) {
+; CHECK-LABEL: @urem_shl_YX_mul_ZX_fail(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 [[Y:%.*]], [[X:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw i8 [[Z:%.*]], [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 %Y, %X
+  %BO1 = mul nuw nsw i8 %Z, %X
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_mul_YX_shl_ZX_fail(i8 %X, i8 %Z, i8 %Y) {
+; CHECK-LABEL: @urem_mul_YX_shl_ZX_fail(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw nsw i8 [[Y:%.*]], [[X:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[Z:%.*]], [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw nsw i8 %Y, %X
+  %BO1 = shl nuw nsw i8 %Z, %X
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_CXY_shl_ZCX_fail(i8 %Z, i8 %Y) {
+; CHECK-LABEL: @urem_shl_CXY_shl_ZCX_fail(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 3, [[Y:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[Z:%.*]], 3
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 3, %Y
+  %BO1 = shl nuw nsw i8 %Z, 3
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_YCX_shl_CXZ_fail(i8 %Z, i8 %Y) {
+; CHECK-LABEL: @urem_shl_YCX_shl_CXZ_fail(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 [[Y:%.*]], 3
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 3, [[Z:%.*]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 %Y, 3
+  %BO1 = shl nuw nsw i8 3, %Z
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_YCX_mul_ZCX_fail(i8 %Z, i8 %Y) {
+; CHECK-LABEL: @urem_shl_YCX_mul_ZCX_fail(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 [[Y:%.*]], 3
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw i8 [[Z:%.*]], 10
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 %Y, 3
+  %BO1 = mul nuw nsw i8 %Z, 10
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_mul_YCX_shl_ZCX_fail(i8 %Z, i8 %Y) {
+; CHECK-LABEL: @urem_mul_YCX_shl_ZCX_fail(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw nsw i8 [[Y:%.*]], 3
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[Z:%.*]], 3
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw nsw i8 %Y, 3
+  %BO1 = shl nuw nsw i8 %Z, 3
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_XCY_shl_CZX_fail(i8 %X) {
+; CHECK-LABEL: @urem_shl_XCY_shl_CZX_fail(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 [[X:%.*]], 3
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 6, [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 %X, 3
+  %BO1 = shl nuw nsw i8 6, %X
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_mul_XCY_shl_CZX_fail(i8 %X) {
+; CHECK-LABEL: @urem_mul_XCY_shl_CZX_fail(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw nsw i8 [[X:%.*]], 6
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 3, [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw nsw i8 %X, 6
+  %BO1 = shl nuw nsw i8 3, %X
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_CYX_shl_XCZ_fail(i8 %X) {
+; CHECK-LABEL: @urem_shl_CYX_shl_XCZ_fail(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 3, [[X:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[X]], 6
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 3, %X
+  %BO1 = shl nuw nsw i8 %X, 6
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_CYX_mul_XCZ_fail(i8 %X) {
+; CHECK-LABEL: @urem_shl_CYX_mul_XCZ_fail(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 3, [[X:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw i8 [[X]], 10
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 3, %X
+  %BO1 = mul nuw nsw i8 %X, 10
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_XX_shl_ZX(i8 %X, i8 %Z) {
+; CHECK-LABEL: @urem_shl_XX_shl_ZX(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 [[X:%.*]], [[X]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[Z:%.*]], [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 %X, %X
+  %BO1 = shl nuw nsw i8 %Z, %X
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_YX_shl_XX(i8 %X, i8 %Y) {
+; CHECK-LABEL: @urem_shl_YX_shl_XX(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 [[Y:%.*]], [[X:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[X]], [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 %Y, %X
+  %BO1 = shl nuw nsw i8 %X, %X
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_XX_shl_XZ(i8 %X, i8 %Z) {
+; CHECK-LABEL: @urem_shl_XX_shl_XZ(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 [[X:%.*]], [[X]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[X]], [[Z:%.*]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 %X, %X
+  %BO1 = shl nuw nsw i8 %X, %Z
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_XY_shl_XX(i8 %X, i8 %Y) {
+; CHECK-LABEL: @urem_shl_XY_shl_XX(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 [[X:%.*]], [[Y:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[X]], [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 %X, %Y
+  %BO1 = shl nuw nsw i8 %X, %X
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_mul_XX_shl_ZX(i8 %X, i8 %Z) {
+; CHECK-LABEL: @urem_mul_XX_shl_ZX(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw nsw i8 [[X:%.*]], [[X]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[Z:%.*]], [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw nsw i8 %X, %X
+  %BO1 = shl nuw nsw i8 %Z, %X
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_mul_YX_shl_XX(i8 %X, i8 %Y) {
+; CHECK-LABEL: @urem_mul_YX_shl_XX(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw nsw i8 [[Y:%.*]], [[X:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[X]], [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw nsw i8 %Y, %X
+  %BO1 = shl nuw nsw i8 %X, %X
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_mul_XX_shl_XZ(i8 %X, i8 %Z) {
+; CHECK-LABEL: @urem_mul_XX_shl_XZ(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw nsw i8 [[X:%.*]], [[X]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[X]], [[Z:%.*]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw nsw i8 %X, %X
+  %BO1 = shl nuw nsw i8 %X, %Z
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_mul_XY_shl_XX(i8 %X, i8 %Y) {
+; CHECK-LABEL: @urem_mul_XY_shl_XX(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw nsw i8 [[X:%.*]], [[Y:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[X]], [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw nsw i8 %X, %Y
+  %BO1 = shl nuw nsw i8 %X, %X
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_XX_mul_ZX(i8 %X, i8 %Z) {
+; CHECK-LABEL: @urem_shl_XX_mul_ZX(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 [[X:%.*]], [[X]]
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw i8 [[Z:%.*]], [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 %X, %X
+  %BO1 = mul nuw nsw i8 %Z, %X
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_YX_mul_XX(i8 %X, i8 %Y) {
+; CHECK-LABEL: @urem_shl_YX_mul_XX(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 [[Y:%.*]], [[X:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw i8 [[X]], [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 %Y, %X
+  %BO1 = mul nuw nsw i8 %X, %X
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_XX_mul_XZ(i8 %X, i8 %Z) {
+; CHECK-LABEL: @urem_shl_XX_mul_XZ(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 [[X:%.*]], [[X]]
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw i8 [[X]], [[Z:%.*]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 %X, %X
+  %BO1 = mul nuw nsw i8 %X, %Z
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_XY_mul_XX(i8 %X, i8 %Y) {
+; CHECK-LABEL: @urem_shl_XY_mul_XX(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 [[X:%.*]], [[Y:%.*]]
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw i8 [[X]], [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 %X, %Y
+  %BO1 = mul nuw nsw i8 %X, %X
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_mul_XX_shl_XX(i8 %X) {
+; CHECK-LABEL: @urem_mul_XX_shl_XX(
+; CHECK-NEXT:    [[BO0:%.*]] = mul nuw nsw i8 [[X:%.*]], [[X]]
+; CHECK-NEXT:    [[BO1:%.*]] = shl nuw nsw i8 [[X]], [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = mul nuw nsw i8 %X, %X
+  %BO1 = shl nuw nsw i8 %X, %X
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
+
+define i8 @urem_shl_XX_mul_XX(i8 %X) {
+; CHECK-LABEL: @urem_shl_XX_mul_XX(
+; CHECK-NEXT:    [[BO0:%.*]] = shl nuw nsw i8 [[X:%.*]], [[X]]
+; CHECK-NEXT:    [[BO1:%.*]] = mul nuw nsw i8 [[X]], [[X]]
+; CHECK-NEXT:    [[R:%.*]] = urem i8 [[BO0]], [[BO1]]
+; CHECK-NEXT:    ret i8 [[R]]
+;
+  %BO0 = shl nuw nsw i8 %X, %X
+  %BO1 = mul nuw nsw i8 %X, %X
+  %r = urem i8 %BO0, %BO1
+  ret i8 %r
+}
diff --git a/llvm/test/Transforms/InstCombine/select-select.ll b/llvm/test/Transforms/InstCombine/select-select.ll
index 8cb7cc8a98e34..7d7b67ad89011 100644
--- a/llvm/test/Transforms/InstCombine/select-select.ll
+++ b/llvm/test/Transforms/InstCombine/select-select.ll
@@ -175,3 +175,426 @@ define <2 x i8> @sel_shuf_narrowing_commute2(<4 x i8> %x, <4 x i8> %y, <2 x i8>
   %r = select <2 x i1> %cmp, <2 x i8> %x2, <2 x i8> %blend
   ret <2 x i8> %r
 }
+
+define i8 @strong_order_cmp_slt_eq(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_slt_eq(
+; CHECK-NEXT:    [[CMP_LT:%.*]] = icmp slt i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEL_LT:%.*]] = select i1 [[CMP_LT]], i8 -1, i8 1
+; CHECK-NEXT:    [[CMP_EQ:%.*]] = icmp eq i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = select i1 [[CMP_EQ]], i8 0, i8 [[SEL_LT]]
+; CHECK-NEXT:    ret i8 [[SEL_EQ]]
+;
+  %cmp.lt = icmp slt i32 %a, %b
+  %sel.lt = select i1 %cmp.lt, i8 -1, i8 1
+  %cmp.eq = icmp eq i32 %a, %b
+  %sel.eq = select i1 %cmp.eq, i8 0, i8 %sel.lt
+  ret i8 %sel.eq
+}
+
+define i8 @strong_order_cmp_ult_eq(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_ult_eq(
+; CHECK-NEXT:    [[CMP_LT:%.*]] = icmp ult i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEL_LT:%.*]] = select i1 [[CMP_LT]], i8 -1, i8 1
+; CHECK-NEXT:    [[CMP_EQ:%.*]] = icmp eq i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = select i1 [[CMP_EQ]], i8 0, i8 [[SEL_LT]]
+; CHECK-NEXT:    ret i8 [[SEL_EQ]]
+;
+  %cmp.lt = icmp ult i32 %a, %b
+  %sel.lt = select i1 %cmp.lt, i8 -1, i8 1
+  %cmp.eq = icmp eq i32 %a, %b
+  %sel.eq = select i1 %cmp.eq, i8 0, i8 %sel.lt
+  ret i8 %sel.eq
+}
+
+define i8 @strong_order_cmp_slt_eq_wrong_const(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_slt_eq_wrong_const(
+; CHECK-NEXT:    [[CMP_LT:%.*]] = icmp slt i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEL_LT:%.*]] = select i1 [[CMP_LT]], i8 -2, i8 1
+; CHECK-NEXT:    [[CMP_EQ:%.*]] = icmp eq i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = select i1 [[CMP_EQ]], i8 0, i8 [[SEL_LT]]
+; CHECK-NEXT:    ret i8 [[SEL_EQ]]
+;
+  %cmp.lt = icmp slt i32 %a, %b
+  %sel.lt = select i1 %cmp.lt, i8 -2, i8 1
+  %cmp.eq = icmp eq i32 %a, %b
+  %sel.eq = select i1 %cmp.eq, i8 0, i8 %sel.lt
+  ret i8 %sel.eq
+}
+
+define i8 @strong_order_cmp_ult_eq_wrong_const(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_ult_eq_wrong_const(
+; CHECK-NEXT:    [[CMP_LT:%.*]] = icmp ult i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEL_LT:%.*]] = select i1 [[CMP_LT]], i8 -1, i8 3
+; CHECK-NEXT:    [[CMP_EQ:%.*]] = icmp eq i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = select i1 [[CMP_EQ]], i8 0, i8 [[SEL_LT]]
+; CHECK-NEXT:    ret i8 [[SEL_EQ]]
+;
+  %cmp.lt = icmp ult i32 %a, %b
+  %sel.lt = select i1 %cmp.lt, i8 -1, i8 3
+  %cmp.eq = icmp eq i32 %a, %b
+  %sel.eq = select i1 %cmp.eq, i8 0, i8 %sel.lt
+  ret i8 %sel.eq
+}
+
+define i8 @strong_order_cmp_slt_ult_wrong_pred(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_slt_ult_wrong_pred(
+; CHECK-NEXT:    [[CMP_LT:%.*]] = icmp slt i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEL_LT:%.*]] = select i1 [[CMP_LT]], i8 -1, i8 1
+; CHECK-NEXT:    [[CMP_EQ:%.*]] = icmp ult i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = select i1 [[CMP_EQ]], i8 0, i8 [[SEL_LT]]
+; CHECK-NEXT:    ret i8 [[SEL_EQ]]
+;
+  %cmp.lt = icmp slt i32 %a, %b
+  %sel.lt = select i1 %cmp.lt, i8 -1, i8 1
+  %cmp.eq = icmp ult i32 %a, %b
+  %sel.eq = select i1 %cmp.eq, i8 0, i8 %sel.lt
+  ret i8 %sel.eq
+}
+
+define i8 @strong_order_cmp_sgt_eq(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_sgt_eq(
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp sgt i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEL_GT:%.*]] = select i1 [[CMP_GT]], i8 1, i8 -1
+; CHECK-NEXT:    [[CMP_EQ:%.*]] = icmp eq i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = select i1 [[CMP_EQ]], i8 0, i8 [[SEL_GT]]
+; CHECK-NEXT:    ret i8 [[SEL_EQ]]
+;
+  %cmp.gt = icmp sgt i32 %a, %b
+  %sel.gt = select i1 %cmp.gt, i8 1, i8 -1
+  %cmp.eq = icmp eq i32 %a, %b
+  %sel.eq = select i1 %cmp.eq, i8 0, i8 %sel.gt
+  ret i8 %sel.eq
+}
+
+define i8 @strong_order_cmp_ugt_eq(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_ugt_eq(
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp ugt i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEL_GT:%.*]] = select i1 [[CMP_GT]], i8 1, i8 -1
+; CHECK-NEXT:    [[CMP_EQ:%.*]] = icmp eq i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = select i1 [[CMP_EQ]], i8 0, i8 [[SEL_GT]]
+; CHECK-NEXT:    ret i8 [[SEL_EQ]]
+;
+  %cmp.gt = icmp ugt i32 %a, %b
+  %sel.gt = select i1 %cmp.gt, i8 1, i8 -1
+  %cmp.eq = icmp eq i32 %a, %b
+  %sel.eq = select i1 %cmp.eq, i8 0, i8 %sel.gt
+  ret i8 %sel.eq
+}
+
+define i8 @strong_order_cmp_eq_slt(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_eq_slt(
+; CHECK-NEXT:    [[CMP_EQ:%.*]] = icmp ne i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = zext i1 [[CMP_EQ]] to i8
+; CHECK-NEXT:    [[CMP_LT:%.*]] = icmp slt i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_LT:%.*]] = select i1 [[CMP_LT]], i8 -1, i8 [[SEL_EQ]]
+; CHECK-NEXT:    ret i8 [[SEL_LT]]
+;
+  %cmp.eq = icmp eq i32 %a, %b
+  %sel.eq = select i1 %cmp.eq, i8 0, i8 1
+  %cmp.lt = icmp slt i32 %a, %b
+  %sel.lt = select i1 %cmp.lt, i8 -1, i8 %sel.eq
+  ret i8 %sel.lt
+}
+
+define i8 @strong_order_cmp_eq_sgt(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_eq_sgt(
+; CHECK-NEXT:    [[CMP_EQ:%.*]] = icmp ne i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = sext i1 [[CMP_EQ]] to i8
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp sgt i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_GT:%.*]] = select i1 [[CMP_GT]], i8 1, i8 [[SEL_EQ]]
+; CHECK-NEXT:    ret i8 [[SEL_GT]]
+;
+  %cmp.eq = icmp eq i32 %a, %b
+  %sel.eq = select i1 %cmp.eq, i8 0, i8 -1
+  %cmp.gt = icmp sgt i32 %a, %b
+  %sel.gt = select i1 %cmp.gt, i8 1, i8 %sel.eq
+  ret i8 %sel.gt
+}
+
+define i8 @strong_order_cmp_eq_ult(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_eq_ult(
+; CHECK-NEXT:    [[CMP_EQ:%.*]] = icmp ne i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = zext i1 [[CMP_EQ]] to i8
+; CHECK-NEXT:    [[CMP_LT:%.*]] = icmp ult i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_LT:%.*]] = select i1 [[CMP_LT]], i8 -1, i8 [[SEL_EQ]]
+; CHECK-NEXT:    ret i8 [[SEL_LT]]
+;
+  %cmp.eq = icmp eq i32 %a, %b
+  %sel.eq = select i1 %cmp.eq, i8 0, i8 1
+  %cmp.lt = icmp ult i32 %a, %b
+  %sel.lt = select i1 %cmp.lt, i8 -1, i8 %sel.eq
+  ret i8 %sel.lt
+}
+
+define i8 @strong_order_cmp_eq_ugt(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_eq_ugt(
+; CHECK-NEXT:    [[CMP_EQ:%.*]] = icmp ne i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = sext i1 [[CMP_EQ]] to i8
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp ugt i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_GT:%.*]] = select i1 [[CMP_GT]], i8 1, i8 [[SEL_EQ]]
+; CHECK-NEXT:    ret i8 [[SEL_GT]]
+;
+  %cmp.eq = icmp eq i32 %a, %b
+  %sel.eq = select i1 %cmp.eq, i8 0, i8 -1
+  %cmp.gt = icmp ugt i32 %a, %b
+  %sel.gt = select i1 %cmp.gt, i8 1, i8 %sel.eq
+  ret i8 %sel.gt
+}
+
+define i8 @strong_order_cmp_slt_sgt(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_slt_sgt(
+; CHECK-NEXT:    [[CMP_LT:%.*]] = icmp slt i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEXT:%.*]] = sext i1 [[CMP_LT]] to i8
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp sgt i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_GT:%.*]] = select i1 [[CMP_GT]], i8 1, i8 [[SEXT]]
+; CHECK-NEXT:    ret i8 [[SEL_GT]]
+;
+  %cmp.lt = icmp slt i32 %a, %b
+  %sext = sext i1 %cmp.lt to i8
+  %cmp.gt = icmp sgt i32 %a, %b
+  %sel.gt = select i1 %cmp.gt, i8 1, i8 %sext
+  ret i8 %sel.gt
+}
+
+define i8 @strong_order_cmp_ult_ugt(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_ult_ugt(
+; CHECK-NEXT:    [[CMP_LT:%.*]] = icmp ult i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEXT:%.*]] = sext i1 [[CMP_LT]] to i8
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp ugt i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_GT:%.*]] = select i1 [[CMP_GT]], i8 1, i8 [[SEXT]]
+; CHECK-NEXT:    ret i8 [[SEL_GT]]
+;
+  %cmp.lt = icmp ult i32 %a, %b
+  %sext = sext i1 %cmp.lt to i8
+  %cmp.gt = icmp ugt i32 %a, %b
+  %sel.gt = select i1 %cmp.gt, i8 1, i8 %sext
+  ret i8 %sel.gt
+}
+
+define i8 @strong_order_cmp_sgt_slt(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_sgt_slt(
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp sgt i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[ZEXT:%.*]] = zext i1 [[CMP_GT]] to i8
+; CHECK-NEXT:    [[CMP_LT:%.*]] = icmp slt i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_LT:%.*]] = select i1 [[CMP_LT]], i8 -1, i8 [[ZEXT]]
+; CHECK-NEXT:    ret i8 [[SEL_LT]]
+;
+  %cmp.gt = icmp sgt i32 %a, %b
+  %zext = zext i1 %cmp.gt to i8
+  %cmp.lt = icmp slt i32 %a, %b
+  %sel.lt = select i1 %cmp.lt, i8 -1, i8 %zext
+  ret i8 %sel.lt
+}
+
+define i8 @strong_order_cmp_ugt_ult(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_ugt_ult(
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp ugt i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[ZEXT:%.*]] = zext i1 [[CMP_GT]] to i8
+; CHECK-NEXT:    [[CMP_LT:%.*]] = icmp ult i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_LT:%.*]] = select i1 [[CMP_LT]], i8 -1, i8 [[ZEXT]]
+; CHECK-NEXT:    ret i8 [[SEL_LT]]
+;
+  %cmp.gt = icmp ugt i32 %a, %b
+  %zext = zext i1 %cmp.gt to i8
+  %cmp.lt = icmp ult i32 %a, %b
+  %sel.lt = select i1 %cmp.lt, i8 -1, i8 %zext
+  ret i8 %sel.lt
+}
+
+define i8 @strong_order_cmp_ne_ugt_ne_not_one_use(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_ne_ugt_ne_not_one_use(
+; CHECK-NEXT:    [[CMP_NE:%.*]] = icmp ne i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    call void @use1(i1 [[CMP_NE]])
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = sext i1 [[CMP_NE]] to i8
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp ugt i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_GT:%.*]] = select i1 [[CMP_GT]], i8 1, i8 [[SEL_EQ]]
+; CHECK-NEXT:    ret i8 [[SEL_GT]]
+;
+  %cmp.ne = icmp ne i32 %a, %b
+  call void @use1(i1 %cmp.ne)
+  %sel.eq = sext i1 %cmp.ne to i8
+  %cmp.gt = icmp ugt i32 %a, %b
+  %sel.gt = select i1 %cmp.gt, i8 1, i8 %sel.eq
+  ret i8 %sel.gt
+}
+
+define i8 @strong_order_cmp_slt_eq_slt_not_oneuse(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_slt_eq_slt_not_oneuse(
+; CHECK-NEXT:    [[CMP_LT:%.*]] = icmp slt i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    call void @use1(i1 [[CMP_LT]])
+; CHECK-NEXT:    [[SEL_LT:%.*]] = select i1 [[CMP_LT]], i8 -1, i8 1
+; CHECK-NEXT:    [[CMP_EQ:%.*]] = icmp eq i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = select i1 [[CMP_EQ]], i8 0, i8 [[SEL_LT]]
+; CHECK-NEXT:    ret i8 [[SEL_EQ]]
+;
+  %cmp.lt = icmp slt i32 %a, %b
+  call void @use1(i1 %cmp.lt)
+  %sel.lt = select i1 %cmp.lt, i8 -1, i8 1
+  %cmp.eq = icmp eq i32 %a, %b
+  %sel.eq = select i1 %cmp.eq, i8 0, i8 %sel.lt
+  ret i8 %sel.eq
+}
+
+define i8 @strong_order_cmp_sgt_eq_eq_not_oneuse(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_sgt_eq_eq_not_oneuse(
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp sgt i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEL_GT:%.*]] = select i1 [[CMP_GT]], i8 1, i8 -1
+; CHECK-NEXT:    [[CMP_EQ:%.*]] = icmp eq i32 [[A]], [[B]]
+; CHECK-NEXT:    call void @use1(i1 [[CMP_EQ]])
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = select i1 [[CMP_EQ]], i8 0, i8 [[SEL_GT]]
+; CHECK-NEXT:    ret i8 [[SEL_EQ]]
+;
+  %cmp.gt = icmp sgt i32 %a, %b
+  %sel.gt = select i1 %cmp.gt, i8 1, i8 -1
+  %cmp.eq = icmp eq i32 %a, %b
+  call void @use1(i1 %cmp.eq)
+  %sel.eq = select i1 %cmp.eq, i8 0, i8 %sel.gt
+  ret i8 %sel.eq
+}
+
+define i8 @strong_order_cmp_eq_ugt_eq_not_oneuse(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_eq_ugt_eq_not_oneuse(
+; CHECK-NEXT:    [[CMP_EQ:%.*]] = icmp eq i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    call void @use1(i1 [[CMP_EQ]])
+; CHECK-NEXT:    [[NOT_CMP_EQ:%.*]] = xor i1 [[CMP_EQ]], true
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = sext i1 [[NOT_CMP_EQ]] to i8
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp ugt i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_GT:%.*]] = select i1 [[CMP_GT]], i8 1, i8 [[SEL_EQ]]
+; CHECK-NEXT:    ret i8 [[SEL_GT]]
+;
+  %cmp.eq = icmp eq i32 %a, %b
+  call void @use1(i1 %cmp.eq)
+  %sel.eq = select i1 %cmp.eq, i8 0, i8 -1
+  %cmp.gt = icmp ugt i32 %a, %b
+  %sel.gt = select i1 %cmp.gt, i8 1, i8 %sel.eq
+  ret i8 %sel.gt
+}
+
+define i8 @strong_order_cmp_ugt_ult_zext_not_oneuse(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_ugt_ult_zext_not_oneuse(
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp ugt i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[ZEXT:%.*]] = zext i1 [[CMP_GT]] to i8
+; CHECK-NEXT:    call void @use8(i8 [[ZEXT]])
+; CHECK-NEXT:    [[CMP_LT:%.*]] = icmp ult i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_LT:%.*]] = select i1 [[CMP_LT]], i8 -1, i8 [[ZEXT]]
+; CHECK-NEXT:    ret i8 [[SEL_LT]]
+;
+  %cmp.gt = icmp ugt i32 %a, %b
+  %zext = zext i1 %cmp.gt to i8
+  call void @use8(i8 %zext)
+  %cmp.lt = icmp ult i32 %a, %b
+  %sel.lt = select i1 %cmp.lt, i8 -1, i8 %zext
+  ret i8 %sel.lt
+}
+
+define i8 @strong_order_cmp_slt_sgt_sext_not_oneuse(i32 %a, i32 %b) {
+; CHECK-LABEL: @strong_order_cmp_slt_sgt_sext_not_oneuse(
+; CHECK-NEXT:    [[CMP_LT:%.*]] = icmp slt i32 [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEXT:%.*]] = sext i1 [[CMP_LT]] to i8
+; CHECK-NEXT:    call void @use8(i8 [[SEXT]])
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp sgt i32 [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_GT:%.*]] = select i1 [[CMP_GT]], i8 1, i8 [[SEXT]]
+; CHECK-NEXT:    ret i8 [[SEL_GT]]
+;
+  %cmp.lt = icmp slt i32 %a, %b
+  %sext = sext i1 %cmp.lt to i8
+  call void @use8(i8 %sext)
+  %cmp.gt = icmp sgt i32 %a, %b
+  %sel.gt = select i1 %cmp.gt, i8 1, i8 %sext
+  ret i8 %sel.gt
+}
+
+define <2 x i8> @strong_order_cmp_ugt_ult_vector(<2 x i32> %a, <2 x i32> %b) {
+; CHECK-LABEL: @strong_order_cmp_ugt_ult_vector(
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp ugt <2 x i32> [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[ZEXT:%.*]] = zext <2 x i1> [[CMP_GT]] to <2 x i8>
+; CHECK-NEXT:    [[CMP_LT:%.*]] = icmp ult <2 x i32> [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_LT:%.*]] = select <2 x i1> [[CMP_LT]], <2 x i8> <i8 -1, i8 -1>, <2 x i8> [[ZEXT]]
+; CHECK-NEXT:    ret <2 x i8> [[SEL_LT]]
+;
+  %cmp.gt = icmp ugt <2 x i32> %a, %b
+  %zext = zext <2 x i1> %cmp.gt to <2 x i8>
+  %cmp.lt = icmp ult <2 x i32> %a, %b
+  %sel.lt = select <2 x i1> %cmp.lt, <2 x i8> <i8 -1, i8 -1>, <2 x i8> %zext
+  ret <2 x i8> %sel.lt
+}
+
+define <2 x i8> @strong_order_cmp_ugt_ult_vector_poison(<2 x i32> %a, <2 x i32> %b) {
+; CHECK-LABEL: @strong_order_cmp_ugt_ult_vector_poison(
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp ugt <2 x i32> [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[ZEXT:%.*]] = zext <2 x i1> [[CMP_GT]] to <2 x i8>
+; CHECK-NEXT:    [[CMP_LT:%.*]] = icmp ult <2 x i32> [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_LT:%.*]] = select <2 x i1> [[CMP_LT]], <2 x i8> <i8 poison, i8 -1>, <2 x i8> [[ZEXT]]
+; CHECK-NEXT:    ret <2 x i8> [[SEL_LT]]
+;
+  %cmp.gt = icmp ugt <2 x i32> %a, %b
+  %zext = zext <2 x i1> %cmp.gt to <2 x i8>
+  %cmp.lt = icmp ult <2 x i32> %a, %b
+  %sel.lt = select <2 x i1> %cmp.lt, <2 x i8> <i8 poison, i8 -1>, <2 x i8> %zext
+  ret <2 x i8> %sel.lt
+}
+
+define <2 x i8> @strong_order_cmp_eq_ugt_vector(<2 x i32> %a, <2 x i32> %b) {
+; CHECK-LABEL: @strong_order_cmp_eq_ugt_vector(
+; CHECK-NEXT:    [[CMP_EQ:%.*]] = icmp ne <2 x i32> [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = sext <2 x i1> [[CMP_EQ]] to <2 x i8>
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp ugt <2 x i32> [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_GT:%.*]] = select <2 x i1> [[CMP_GT]], <2 x i8> <i8 1, i8 1>, <2 x i8> [[SEL_EQ]]
+; CHECK-NEXT:    ret <2 x i8> [[SEL_GT]]
+;
+  %cmp.eq = icmp eq <2 x i32> %a, %b
+  %sel.eq = select <2 x i1> %cmp.eq, <2 x i8> <i8 0, i8 0>, <2 x i8> <i8 -1, i8 -1>
+  %cmp.gt = icmp ugt <2 x i32> %a, %b
+  %sel.gt = select <2 x i1> %cmp.gt, <2 x i8> <i8 1, i8 1>, <2 x i8> %sel.eq
+  ret <2 x i8> %sel.gt
+}
+
+define <2 x i8> @strong_order_cmp_eq_ugt_vector_poison1(<2 x i32> %a, <2 x i32> %b) {
+; CHECK-LABEL: @strong_order_cmp_eq_ugt_vector_poison1(
+; CHECK-NEXT:    [[CMP_EQ:%.*]] = icmp ne <2 x i32> [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = sext <2 x i1> [[CMP_EQ]] to <2 x i8>
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp ugt <2 x i32> [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_GT:%.*]] = select <2 x i1> [[CMP_GT]], <2 x i8> <i8 1, i8 1>, <2 x i8> [[SEL_EQ]]
+; CHECK-NEXT:    ret <2 x i8> [[SEL_GT]]
+;
+  %cmp.eq = icmp eq <2 x i32> %a, %b
+  %sel.eq = select <2 x i1> %cmp.eq, <2 x i8> <i8 0, i8 poison>, <2 x i8> <i8 -1, i8 -1>
+  %cmp.gt = icmp ugt <2 x i32> %a, %b
+  %sel.gt = select <2 x i1> %cmp.gt, <2 x i8> <i8 1, i8 1>, <2 x i8> %sel.eq
+  ret <2 x i8> %sel.gt
+}
+
+define <2 x i8> @strong_order_cmp_eq_ugt_vector_poison2(<2 x i32> %a, <2 x i32> %b) {
+; CHECK-LABEL: @strong_order_cmp_eq_ugt_vector_poison2(
+; CHECK-NEXT:    [[CMP_EQ:%.*]] = icmp ne <2 x i32> [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = sext <2 x i1> [[CMP_EQ]] to <2 x i8>
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp ugt <2 x i32> [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_GT:%.*]] = select <2 x i1> [[CMP_GT]], <2 x i8> <i8 1, i8 1>, <2 x i8> [[SEL_EQ]]
+; CHECK-NEXT:    ret <2 x i8> [[SEL_GT]]
+;
+  %cmp.eq = icmp eq <2 x i32> %a, %b
+  %sel.eq = select <2 x i1> %cmp.eq, <2 x i8> <i8 0, i8 0>, <2 x i8> <i8 poison, i8 -1>
+  %cmp.gt = icmp ugt <2 x i32> %a, %b
+  %sel.gt = select <2 x i1> %cmp.gt, <2 x i8> <i8 1, i8 1>, <2 x i8> %sel.eq
+  ret <2 x i8> %sel.gt
+}
+
+define <2 x i8> @strong_order_cmp_eq_ugt_vector_poison3(<2 x i32> %a, <2 x i32> %b) {
+; CHECK-LABEL: @strong_order_cmp_eq_ugt_vector_poison3(
+; CHECK-NEXT:    [[CMP_EQ:%.*]] = icmp ne <2 x i32> [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[SEL_EQ:%.*]] = sext <2 x i1> [[CMP_EQ]] to <2 x i8>
+; CHECK-NEXT:    [[CMP_GT:%.*]] = icmp ugt <2 x i32> [[A]], [[B]]
+; CHECK-NEXT:    [[SEL_GT:%.*]] = select <2 x i1> [[CMP_GT]], <2 x i8> <i8 1, i8 poison>, <2 x i8> [[SEL_EQ]]
+; CHECK-NEXT:    ret <2 x i8> [[SEL_GT]]
+;
+  %cmp.eq = icmp eq <2 x i32> %a, %b
+  %sel.eq = select <2 x i1> %cmp.eq, <2 x i8> <i8 0, i8 0>, <2 x i8> <i8 -1, i8 -1>
+  %cmp.gt = icmp ugt <2 x i32> %a, %b
+  %sel.gt = select <2 x i1> %cmp.gt, <2 x i8> <i8 1, i8 poison>, <2 x i8> %sel.eq
+  ret <2 x i8> %sel.gt
+}
+
+
+
+declare void @use1(i1)
+declare void @use8(i8)
diff --git a/llvm/test/Transforms/InstCombine/select.ll b/llvm/test/Transforms/InstCombine/select.ll
index 20e9a3aa0312a..ccb62b027c655 100644
--- a/llvm/test/Transforms/InstCombine/select.ll
+++ b/llvm/test/Transforms/InstCombine/select.ll
@@ -3566,3 +3566,18 @@ define i8 @not_clamp_smax2(i8 %x) {
   %sel = select i1 %cmp, i8 125, i8 %x
   ret i8 %sel
 }
+
+; Used to infinite loop.
+define i32 @pr61361(i32 %arg) {
+; CHECK-LABEL: @pr61361(
+; CHECK-NEXT:    [[CMP2:%.*]] = icmp eq i32 [[ARG:%.*]], 0
+; CHECK-NEXT:    [[SEL2:%.*]] = select i1 [[CMP2]], i32 16777215, i32 0
+; CHECK-NEXT:    ret i32 [[SEL2]]
+;
+  %cmp1 = icmp eq i32 %arg, 1
+  %sel1 = select i1 %cmp1, i32 0, i32 33554431
+  %cmp2 = icmp eq i32 %arg, 0
+  %sel2 = select i1 %cmp2, i32 %sel1, i32 0
+  %ashr = ashr i32 %sel2, 1
+  ret i32 %ashr
+}
diff --git a/llvm/test/Transforms/InstCombine/shift.ll b/llvm/test/Transforms/InstCombine/shift.ll
index 652b2a095e8d5..91036b60c270c 100644
--- a/llvm/test/Transforms/InstCombine/shift.ll
+++ b/llvm/test/Transforms/InstCombine/shift.ll
@@ -2067,3 +2067,48 @@ define i32 @shl2_cttz(i32 %x) {
   %shl = shl i32 2, %tz
   ret i32 %shl
 }
+
+; shift (X, amt | bitwidth - 1) -> shift (X, bitwidth - 1)
+define i6 @shl_or7_eq_shl7(i6 %x, i6 %c) {
+; CHECK-LABEL: @shl_or7_eq_shl7(
+; CHECK-NEXT:    [[Y:%.*]] = shl nsw i6 [[X:%.*]], 5
+; CHECK-NEXT:    ret i6 [[Y]]
+;
+  %amt = or i6 %c, 5
+  ;; nsw not needed for transform, just check that we propagate.
+  %y = shl nsw i6 %x, %amt
+  ret i6 %y
+}
+
+define <2 x i8> @lshr_vec_or7_eq_shl7(<2 x i8> %x, <2 x i8> %c) {
+; CHECK-LABEL: @lshr_vec_or7_eq_shl7(
+; CHECK-NEXT:    [[Y:%.*]] = lshr exact <2 x i8> [[X:%.*]], <i8 7, i8 7>
+; CHECK-NEXT:    ret <2 x i8> [[Y]]
+;
+  %amt = or <2 x i8> %c, <i8 7, i8 7>
+  ;; exact not needed for transform, just check that we propagate.
+  %y = lshr exact <2 x i8> %x, %amt
+  ret <2 x i8> %y
+}
+
+define <2 x i8> @ashr_vec_or7_eq_ashr7(<2 x i8> %x, <2 x i8> %c) {
+; CHECK-LABEL: @ashr_vec_or7_eq_ashr7(
+; CHECK-NEXT:    [[Y:%.*]] = ashr <2 x i8> [[X:%.*]], <i8 7, i8 7>
+; CHECK-NEXT:    ret <2 x i8> [[Y]]
+;
+  %amt = or <2 x i8> %c, <i8 7, i8 7>
+  %y = ashr <2 x i8> %x, %amt
+  ret <2 x i8> %y
+}
+
+; Negative test not bitwidth - 1
+define <2 x i8> @ashr_vec_or6_fail(<2 x i8> %x, <2 x i8> %c) {
+; CHECK-LABEL: @ashr_vec_or6_fail(
+; CHECK-NEXT:    [[AMT:%.*]] = or <2 x i8> [[C:%.*]], <i8 6, i8 6>
+; CHECK-NEXT:    [[Y:%.*]] = ashr <2 x i8> [[X:%.*]], [[AMT]]
+; CHECK-NEXT:    ret <2 x i8> [[Y]]
+;
+  %amt = or <2 x i8> %c, <i8 6, i8 6>
+  %y = ashr <2 x i8> %x, %amt
+  ret <2 x i8> %y
+}
diff --git a/llvm/test/Transforms/InstCombine/sincospi.ll b/llvm/test/Transforms/InstCombine/sincospi.ll
index fb2117945eb7b..5386669bba511 100644
--- a/llvm/test/Transforms/InstCombine/sincospi.ll
+++ b/llvm/test/Transforms/InstCombine/sincospi.ll
@@ -24,8 +24,7 @@ define float @test_instbased_f32() {
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[SINCOSPI:%.*]] = call <2 x float> @__sincospif_stret(float [[VAL]])
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[SINPI:%.*]] = extractelement <2 x float> [[SINCOSPI]], i64 0
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[COSPI:%.*]] = extractelement <2 x float> [[SINCOSPI]], i64 1
-; CHECK-FLOAT-IN-VEC-NEXT:    [[SIN:%.*]] = call float @__sinpif(float [[VAL]]) #[[ATTR0:[0-9]+]]
-; CHECK-FLOAT-IN-VEC-NEXT:    [[COS:%.*]] = call float @__cospif(float [[VAL]]) #[[ATTR0]]
+; CHECK-FLOAT-IN-VEC-NEXT:    [[COS:%.*]] = call float @__cospif(float [[VAL]]) #[[ATTR0:[0-9]+]]
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[RES:%.*]] = fadd float [[SINPI]], [[COSPI]]
 ; CHECK-FLOAT-IN-VEC-NEXT:    ret float [[RES]]
 ;
@@ -34,8 +33,7 @@ define float @test_instbased_f32() {
 ; CHECK-NEXT:    [[SINCOSPI:%.*]] = call { float, float } @__sincospif_stret(float [[VAL]])
 ; CHECK-NEXT:    [[SINPI:%.*]] = extractvalue { float, float } [[SINCOSPI]], 0
 ; CHECK-NEXT:    [[COSPI:%.*]] = extractvalue { float, float } [[SINCOSPI]], 1
-; CHECK-NEXT:    [[SIN:%.*]] = call float @__sinpif(float [[VAL]]) #[[ATTR0:[0-9]+]]
-; CHECK-NEXT:    [[COS:%.*]] = call float @__cospif(float [[VAL]]) #[[ATTR0]]
+; CHECK-NEXT:    [[COS:%.*]] = call float @__cospif(float [[VAL]]) #[[ATTR0:[0-9]+]]
 ; CHECK-NEXT:    [[RES:%.*]] = fadd float [[SINPI]], [[COSPI]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
@@ -60,7 +58,6 @@ define float @test_instbased_f32_other_user(ptr %ptr) {
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[SINPI:%.*]] = extractelement <2 x float> [[SINCOSPI]], i64 0
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[COSPI:%.*]] = extractelement <2 x float> [[SINCOSPI]], i64 1
 ; CHECK-FLOAT-IN-VEC-NEXT:    store float [[VAL]], ptr [[PTR:%.*]], align 4
-; CHECK-FLOAT-IN-VEC-NEXT:    [[SIN:%.*]] = call float @__sinpif(float [[VAL]]) #[[ATTR0]]
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[COS:%.*]] = call float @__cospif(float [[VAL]]) #[[ATTR0]]
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[RES:%.*]] = fadd float [[SINPI]], [[COSPI]]
 ; CHECK-FLOAT-IN-VEC-NEXT:    ret float [[RES]]
@@ -71,7 +68,6 @@ define float @test_instbased_f32_other_user(ptr %ptr) {
 ; CHECK-NEXT:    [[SINPI:%.*]] = extractvalue { float, float } [[SINCOSPI]], 0
 ; CHECK-NEXT:    [[COSPI:%.*]] = extractvalue { float, float } [[SINCOSPI]], 1
 ; CHECK-NEXT:    store float [[VAL]], ptr [[PTR:%.*]], align 4
-; CHECK-NEXT:    [[SIN:%.*]] = call float @__sinpif(float [[VAL]]) #[[ATTR0]]
 ; CHECK-NEXT:    [[COS:%.*]] = call float @__cospif(float [[VAL]]) #[[ATTR0]]
 ; CHECK-NEXT:    [[RES:%.*]] = fadd float [[SINPI]], [[COSPI]]
 ; CHECK-NEXT:    ret float [[RES]]
@@ -97,7 +93,6 @@ define float @test_constant_f32() {
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[SINCOSPI:%.*]] = call <2 x float> @__sincospif_stret(float 1.000000e+00)
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[SINPI:%.*]] = extractelement <2 x float> [[SINCOSPI]], i64 0
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[COSPI:%.*]] = extractelement <2 x float> [[SINCOSPI]], i64 1
-; CHECK-FLOAT-IN-VEC-NEXT:    [[SIN:%.*]] = call float @__sinpif(float 1.000000e+00) #[[ATTR0]]
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[COS:%.*]] = call float @__cospif(float 1.000000e+00) #[[ATTR0]]
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[RES:%.*]] = fadd float [[SINPI]], [[COSPI]]
 ; CHECK-FLOAT-IN-VEC-NEXT:    ret float [[RES]]
@@ -106,7 +101,6 @@ define float @test_constant_f32() {
 ; CHECK-NEXT:    [[SINCOSPI:%.*]] = call { float, float } @__sincospif_stret(float 1.000000e+00)
 ; CHECK-NEXT:    [[SINPI:%.*]] = extractvalue { float, float } [[SINCOSPI]], 0
 ; CHECK-NEXT:    [[COSPI:%.*]] = extractvalue { float, float } [[SINCOSPI]], 1
-; CHECK-NEXT:    [[SIN:%.*]] = call float @__sinpif(float 1.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    [[COS:%.*]] = call float @__cospif(float 1.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    [[RES:%.*]] = fadd float [[SINPI]], [[COSPI]]
 ; CHECK-NEXT:    ret float [[RES]]
@@ -131,7 +125,6 @@ define double @test_instbased_f64() {
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[SINCOSPI:%.*]] = call { double, double } @__sincospi_stret(double [[VAL]])
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[SINPI:%.*]] = extractvalue { double, double } [[SINCOSPI]], 0
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[COSPI:%.*]] = extractvalue { double, double } [[SINCOSPI]], 1
-; CHECK-FLOAT-IN-VEC-NEXT:    [[SIN:%.*]] = call double @__sinpi(double [[VAL]]) #[[ATTR0]]
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[COS:%.*]] = call double @__cospi(double [[VAL]]) #[[ATTR0]]
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[RES:%.*]] = fadd double [[SINPI]], [[COSPI]]
 ; CHECK-FLOAT-IN-VEC-NEXT:    ret double [[RES]]
@@ -141,7 +134,6 @@ define double @test_instbased_f64() {
 ; CHECK-NEXT:    [[SINCOSPI:%.*]] = call { double, double } @__sincospi_stret(double [[VAL]])
 ; CHECK-NEXT:    [[SINPI:%.*]] = extractvalue { double, double } [[SINCOSPI]], 0
 ; CHECK-NEXT:    [[COSPI:%.*]] = extractvalue { double, double } [[SINCOSPI]], 1
-; CHECK-NEXT:    [[SIN:%.*]] = call double @__sinpi(double [[VAL]]) #[[ATTR0]]
 ; CHECK-NEXT:    [[COS:%.*]] = call double @__cospi(double [[VAL]]) #[[ATTR0]]
 ; CHECK-NEXT:    [[RES:%.*]] = fadd double [[SINPI]], [[COSPI]]
 ; CHECK-NEXT:    ret double [[RES]]
@@ -167,7 +159,6 @@ define double @test_constant_f64() {
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[SINCOSPI:%.*]] = call { double, double } @__sincospi_stret(double 1.000000e+00)
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[SINPI:%.*]] = extractvalue { double, double } [[SINCOSPI]], 0
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[COSPI:%.*]] = extractvalue { double, double } [[SINCOSPI]], 1
-; CHECK-FLOAT-IN-VEC-NEXT:    [[SIN:%.*]] = call double @__sinpi(double 1.000000e+00) #[[ATTR0]]
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[COS:%.*]] = call double @__cospi(double 1.000000e+00) #[[ATTR0]]
 ; CHECK-FLOAT-IN-VEC-NEXT:    [[RES:%.*]] = fadd double [[SINPI]], [[COSPI]]
 ; CHECK-FLOAT-IN-VEC-NEXT:    ret double [[RES]]
@@ -176,7 +167,6 @@ define double @test_constant_f64() {
 ; CHECK-NEXT:    [[SINCOSPI:%.*]] = call { double, double } @__sincospi_stret(double 1.000000e+00)
 ; CHECK-NEXT:    [[SINPI:%.*]] = extractvalue { double, double } [[SINCOSPI]], 0
 ; CHECK-NEXT:    [[COSPI:%.*]] = extractvalue { double, double } [[SINCOSPI]], 1
-; CHECK-NEXT:    [[SIN:%.*]] = call double @__sinpi(double 1.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    [[COS:%.*]] = call double @__cospi(double 1.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    [[RES:%.*]] = fadd double [[SINPI]], [[COSPI]]
 ; CHECK-NEXT:    ret double [[RES]]
diff --git a/llvm/test/Transforms/InstCombine/sub-minmax.ll b/llvm/test/Transforms/InstCombine/sub-minmax.ll
index 79396ca099831..c9ce165c38988 100644
--- a/llvm/test/Transforms/InstCombine/sub-minmax.ll
+++ b/llvm/test/Transforms/InstCombine/sub-minmax.ll
@@ -31,8 +31,8 @@ define i32 @na_minus_max_na_b(i32 %A, i32 %B) {
 ; CHECK-LABEL: define {{[^@]+}}@na_minus_max_na_b
 ; CHECK-SAME: (i32 [[A:%.*]], i32 [[B:%.*]]) {
 ; CHECK-NEXT:    [[NOT:%.*]] = xor i32 [[A]], -1
-; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.usub.sat.i32(i32 [[NOT]], i32 [[B]])
-; CHECK-NEXT:    ret i32 [[TMP1]]
+; CHECK-NEXT:    [[X:%.*]] = call i32 @llvm.usub.sat.i32(i32 [[NOT]], i32 [[B]])
+; CHECK-NEXT:    ret i32 [[X]]
 ;
   %not = xor i32 %A, -1
   %l0 = icmp ult i32 %not, %B
@@ -44,8 +44,8 @@ define i32 @na_minus_max_na_b(i32 %A, i32 %B) {
 define i5 @sub_umin(i5 %a, i5 %b) {
 ; CHECK-LABEL: define {{[^@]+}}@sub_umin
 ; CHECK-SAME: (i5 [[A:%.*]], i5 [[B:%.*]]) {
-; CHECK-NEXT:    [[TMP1:%.*]] = call i5 @llvm.usub.sat.i5(i5 [[A]], i5 [[B]])
-; CHECK-NEXT:    ret i5 [[TMP1]]
+; CHECK-NEXT:    [[R:%.*]] = call i5 @llvm.usub.sat.i5(i5 [[A]], i5 [[B]])
+; CHECK-NEXT:    ret i5 [[R]]
 ;
   %umin = call i5 @llvm.umin.i5(i5 %a, i5 %b)
   %r = sub i5 %a, %umin
@@ -55,8 +55,8 @@ define i5 @sub_umin(i5 %a, i5 %b) {
 define <2 x i8> @sub_umin_commute_vec(<2 x i8> %a, <2 x i8> %b) {
 ; CHECK-LABEL: define {{[^@]+}}@sub_umin_commute_vec
 ; CHECK-SAME: (<2 x i8> [[A:%.*]], <2 x i8> [[B:%.*]]) {
-; CHECK-NEXT:    [[TMP1:%.*]] = call <2 x i8> @llvm.usub.sat.v2i8(<2 x i8> [[B]], <2 x i8> [[A]])
-; CHECK-NEXT:    ret <2 x i8> [[TMP1]]
+; CHECK-NEXT:    [[R:%.*]] = call <2 x i8> @llvm.usub.sat.v2i8(<2 x i8> [[B]], <2 x i8> [[A]])
+; CHECK-NEXT:    ret <2 x i8> [[R]]
 ;
   %umin = call <2 x i8> @llvm.umin.v2i8(<2 x i8> %b, <2 x i8> %a)
   %r = sub <2 x i8> %b, %umin
@@ -108,8 +108,8 @@ define i32 @na_minus_max_b_na(i32 %A, i32 %B) {
 ; CHECK-LABEL: define {{[^@]+}}@na_minus_max_b_na
 ; CHECK-SAME: (i32 [[A:%.*]], i32 [[B:%.*]]) {
 ; CHECK-NEXT:    [[NOT:%.*]] = xor i32 [[A]], -1
-; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.usub.sat.i32(i32 [[NOT]], i32 [[B]])
-; CHECK-NEXT:    ret i32 [[TMP1]]
+; CHECK-NEXT:    [[X:%.*]] = call i32 @llvm.usub.sat.i32(i32 [[NOT]], i32 [[B]])
+; CHECK-NEXT:    ret i32 [[X]]
 ;
   %not = xor i32 %A, -1
   %l0 = icmp ugt i32 %not, %B
@@ -137,8 +137,8 @@ define i32 @max_na_bi_minux_na(i32 %A, i32 %Bi) {
 define i32 @na_minus_max_na_bi(i32 %A, i32 %Bi) {
 ; CHECK-LABEL: define {{[^@]+}}@na_minus_max_na_bi
 ; CHECK-SAME: (i32 [[A:%.*]], i32 [[BI:%.*]]) {
-; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.usub.sat.i32(i32 [[BI]], i32 [[A]])
-; CHECK-NEXT:    ret i32 [[TMP1]]
+; CHECK-NEXT:    [[X:%.*]] = call i32 @llvm.usub.sat.i32(i32 [[BI]], i32 [[A]])
+; CHECK-NEXT:    ret i32 [[X]]
 ;
   %B =  xor i32 %Bi, -1
   %not = xor i32 %A, -1
@@ -166,8 +166,8 @@ define i32 @max_bi_na_minus_na(i32 %A, i32 %Bi) {
 define i32 @na_minus_max_bi_na(i32 %A, i32 %Bi) {
 ; CHECK-LABEL: define {{[^@]+}}@na_minus_max_bi_na
 ; CHECK-SAME: (i32 [[A:%.*]], i32 [[BI:%.*]]) {
-; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.usub.sat.i32(i32 [[BI]], i32 [[A]])
-; CHECK-NEXT:    ret i32 [[TMP1]]
+; CHECK-NEXT:    [[X:%.*]] = call i32 @llvm.usub.sat.i32(i32 [[BI]], i32 [[A]])
+; CHECK-NEXT:    ret i32 [[X]]
 ;
   %B =  xor i32 %Bi, -1
   %not = xor i32 %A, -1
@@ -182,9 +182,9 @@ define i32 @max_na_bi_minux_na_use(i32 %A, i32 %Bi) {
 ; CHECK-LABEL: define {{[^@]+}}@max_na_bi_minux_na_use
 ; CHECK-SAME: (i32 [[A:%.*]], i32 [[BI:%.*]]) {
 ; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.umax.i32(i32 [[A]], i32 -32)
-; CHECK-NEXT:    [[TMP2:%.*]] = xor i32 [[TMP1]], -1
+; CHECK-NEXT:    [[L1:%.*]] = xor i32 [[TMP1]], -1
 ; CHECK-NEXT:    [[X:%.*]] = sub i32 [[A]], [[TMP1]]
-; CHECK-NEXT:    call void @use32(i32 [[TMP2]])
+; CHECK-NEXT:    call void @use32(i32 [[L1]])
 ; CHECK-NEXT:    ret i32 [[X]]
 ;
   %not = xor i32 %A, -1
@@ -199,9 +199,9 @@ define i32 @na_minus_max_na_bi_use(i32 %A, i32 %Bi) {
 ; CHECK-LABEL: define {{[^@]+}}@na_minus_max_na_bi_use
 ; CHECK-SAME: (i32 [[A:%.*]], i32 [[BI:%.*]]) {
 ; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.umax.i32(i32 [[A]], i32 -32)
-; CHECK-NEXT:    [[TMP2:%.*]] = xor i32 [[TMP1]], -1
+; CHECK-NEXT:    [[L1:%.*]] = xor i32 [[TMP1]], -1
 ; CHECK-NEXT:    [[X:%.*]] = sub i32 [[TMP1]], [[A]]
-; CHECK-NEXT:    call void @use32(i32 [[TMP2]])
+; CHECK-NEXT:    call void @use32(i32 [[L1]])
 ; CHECK-NEXT:    ret i32 [[X]]
 ;
   %not = xor i32 %A, -1
@@ -216,9 +216,9 @@ define i32 @max_bi_na_minus_na_use(i32 %A, i32 %Bi) {
 ; CHECK-LABEL: define {{[^@]+}}@max_bi_na_minus_na_use
 ; CHECK-SAME: (i32 [[A:%.*]], i32 [[BI:%.*]]) {
 ; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.umax.i32(i32 [[BI]], i32 [[A]])
-; CHECK-NEXT:    [[TMP2:%.*]] = xor i32 [[TMP1]], -1
+; CHECK-NEXT:    [[L1:%.*]] = xor i32 [[TMP1]], -1
 ; CHECK-NEXT:    [[X:%.*]] = sub i32 [[A]], [[TMP1]]
-; CHECK-NEXT:    call void @use32(i32 [[TMP2]])
+; CHECK-NEXT:    call void @use32(i32 [[L1]])
 ; CHECK-NEXT:    ret i32 [[X]]
 ;
   %not = xor i32 %A, -1
@@ -234,9 +234,9 @@ define i32 @na_minus_max_bi_na_use(i32 %A, i32 %Bi) {
 ; CHECK-LABEL: define {{[^@]+}}@na_minus_max_bi_na_use
 ; CHECK-SAME: (i32 [[A:%.*]], i32 [[BI:%.*]]) {
 ; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.umax.i32(i32 [[BI]], i32 [[A]])
-; CHECK-NEXT:    [[TMP2:%.*]] = xor i32 [[TMP1]], -1
+; CHECK-NEXT:    [[L1:%.*]] = xor i32 [[TMP1]], -1
 ; CHECK-NEXT:    [[X:%.*]] = sub i32 [[TMP1]], [[A]]
-; CHECK-NEXT:    call void @use32(i32 [[TMP2]])
+; CHECK-NEXT:    call void @use32(i32 [[L1]])
 ; CHECK-NEXT:    ret i32 [[X]]
 ;
   %not = xor i32 %A, -1
@@ -253,9 +253,9 @@ define i32 @max_na_bi_minux_na_use2(i32 %A, i32 %Bi) {
 ; CHECK-LABEL: define {{[^@]+}}@max_na_bi_minux_na_use2
 ; CHECK-SAME: (i32 [[A:%.*]], i32 [[BI:%.*]]) {
 ; CHECK-NEXT:    [[NOT:%.*]] = xor i32 [[A]], -1
-; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.umin.i32(i32 [[NOT]], i32 31)
-; CHECK-NEXT:    [[X:%.*]] = sub i32 [[TMP1]], [[NOT]]
-; CHECK-NEXT:    call void @use32(i32 [[TMP1]])
+; CHECK-NEXT:    [[L1:%.*]] = call i32 @llvm.umin.i32(i32 [[NOT]], i32 31)
+; CHECK-NEXT:    [[X:%.*]] = sub i32 [[L1]], [[NOT]]
+; CHECK-NEXT:    call void @use32(i32 [[L1]])
 ; CHECK-NEXT:    call void @use32(i32 [[NOT]])
 ; CHECK-NEXT:    ret i32 [[X]]
 ;
@@ -272,9 +272,9 @@ define i32 @na_minus_max_na_bi_use2(i32 %A, i32 %Bi) {
 ; CHECK-LABEL: define {{[^@]+}}@na_minus_max_na_bi_use2
 ; CHECK-SAME: (i32 [[A:%.*]], i32 [[BI:%.*]]) {
 ; CHECK-NEXT:    [[NOT:%.*]] = xor i32 [[A]], -1
-; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.umin.i32(i32 [[NOT]], i32 31)
-; CHECK-NEXT:    [[X:%.*]] = sub i32 [[NOT]], [[TMP1]]
-; CHECK-NEXT:    call void @use32(i32 [[TMP1]])
+; CHECK-NEXT:    [[L1:%.*]] = call i32 @llvm.umin.i32(i32 [[NOT]], i32 31)
+; CHECK-NEXT:    [[X:%.*]] = sub i32 [[NOT]], [[L1]]
+; CHECK-NEXT:    call void @use32(i32 [[L1]])
 ; CHECK-NEXT:    call void @use32(i32 [[NOT]])
 ; CHECK-NEXT:    ret i32 [[X]]
 ;
@@ -292,9 +292,9 @@ define i32 @max_bi_na_minus_na_use2(i32 %A, i32 %Bi) {
 ; CHECK-SAME: (i32 [[A:%.*]], i32 [[BI:%.*]]) {
 ; CHECK-NEXT:    [[NOT:%.*]] = xor i32 [[A]], -1
 ; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.umax.i32(i32 [[BI]], i32 [[A]])
-; CHECK-NEXT:    [[TMP2:%.*]] = xor i32 [[TMP1]], -1
+; CHECK-NEXT:    [[L1:%.*]] = xor i32 [[TMP1]], -1
 ; CHECK-NEXT:    [[X:%.*]] = sub i32 [[A]], [[TMP1]]
-; CHECK-NEXT:    call void @use32(i32 [[TMP2]])
+; CHECK-NEXT:    call void @use32(i32 [[L1]])
 ; CHECK-NEXT:    call void @use32(i32 [[NOT]])
 ; CHECK-NEXT:    ret i32 [[X]]
 ;
@@ -313,9 +313,9 @@ define i32 @na_minus_max_bi_na_use2(i32 %A, i32 %Bi) {
 ; CHECK-SAME: (i32 [[A:%.*]], i32 [[BI:%.*]]) {
 ; CHECK-NEXT:    [[NOT:%.*]] = xor i32 [[A]], -1
 ; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.umax.i32(i32 [[BI]], i32 [[A]])
-; CHECK-NEXT:    [[TMP2:%.*]] = xor i32 [[TMP1]], -1
+; CHECK-NEXT:    [[L1:%.*]] = xor i32 [[TMP1]], -1
 ; CHECK-NEXT:    [[X:%.*]] = sub i32 [[TMP1]], [[A]]
-; CHECK-NEXT:    call void @use32(i32 [[TMP2]])
+; CHECK-NEXT:    call void @use32(i32 [[L1]])
 ; CHECK-NEXT:    call void @use32(i32 [[NOT]])
 ; CHECK-NEXT:    ret i32 [[X]]
 ;
@@ -333,12 +333,12 @@ define i8 @umin_not_sub(i8 %x, i8 %y) {
 ; CHECK-LABEL: define {{[^@]+}}@umin_not_sub
 ; CHECK-SAME: (i8 [[X:%.*]], i8 [[Y:%.*]]) {
 ; CHECK-NEXT:    [[TMP1:%.*]] = call i8 @llvm.umax.i8(i8 [[X]], i8 [[Y]])
-; CHECK-NEXT:    [[TMP2:%.*]] = xor i8 [[TMP1]], -1
+; CHECK-NEXT:    [[MINXY:%.*]] = xor i8 [[TMP1]], -1
 ; CHECK-NEXT:    [[SUBX:%.*]] = sub i8 [[TMP1]], [[X]]
 ; CHECK-NEXT:    [[SUBY:%.*]] = sub i8 [[TMP1]], [[Y]]
 ; CHECK-NEXT:    call void @use8(i8 [[SUBX]])
 ; CHECK-NEXT:    call void @use8(i8 [[SUBY]])
-; CHECK-NEXT:    ret i8 [[TMP2]]
+; CHECK-NEXT:    ret i8 [[MINXY]]
 ;
   %nx = xor i8 %x, -1
   %ny = xor i8 %y, -1
@@ -355,12 +355,12 @@ define i8 @umin_not_sub_rev(i8 %x, i8 %y) {
 ; CHECK-LABEL: define {{[^@]+}}@umin_not_sub_rev
 ; CHECK-SAME: (i8 [[X:%.*]], i8 [[Y:%.*]]) {
 ; CHECK-NEXT:    [[TMP1:%.*]] = call i8 @llvm.umax.i8(i8 [[X]], i8 [[Y]])
-; CHECK-NEXT:    [[TMP2:%.*]] = xor i8 [[TMP1]], -1
+; CHECK-NEXT:    [[MINXY:%.*]] = xor i8 [[TMP1]], -1
 ; CHECK-NEXT:    [[SUBX:%.*]] = sub i8 [[X]], [[TMP1]]
 ; CHECK-NEXT:    [[SUBY:%.*]] = sub i8 [[Y]], [[TMP1]]
 ; CHECK-NEXT:    call void @use8(i8 [[SUBX]])
 ; CHECK-NEXT:    call void @use8(i8 [[SUBY]])
-; CHECK-NEXT:    ret i8 [[TMP2]]
+; CHECK-NEXT:    ret i8 [[MINXY]]
 ;
   %nx = xor i8 %x, -1
   %ny = xor i8 %y, -1
@@ -378,11 +378,11 @@ define void @umin3_not_all_ops_extra_uses_invert_subs(i8 %x, i8 %y, i8 %z) {
 ; CHECK-SAME: (i8 [[X:%.*]], i8 [[Y:%.*]], i8 [[Z:%.*]]) {
 ; CHECK-NEXT:    [[TMP1:%.*]] = call i8 @llvm.umax.i8(i8 [[X]], i8 [[Z]])
 ; CHECK-NEXT:    [[TMP2:%.*]] = call i8 @llvm.umax.i8(i8 [[Y]], i8 [[TMP1]])
-; CHECK-NEXT:    [[TMP3:%.*]] = xor i8 [[TMP2]], -1
+; CHECK-NEXT:    [[MINXYZ:%.*]] = xor i8 [[TMP2]], -1
 ; CHECK-NEXT:    [[XMIN:%.*]] = sub i8 [[TMP2]], [[X]]
 ; CHECK-NEXT:    [[YMIN:%.*]] = sub i8 [[TMP2]], [[Y]]
 ; CHECK-NEXT:    [[ZMIN:%.*]] = sub i8 [[TMP2]], [[Z]]
-; CHECK-NEXT:    call void @use8(i8 [[TMP3]])
+; CHECK-NEXT:    call void @use8(i8 [[MINXYZ]])
 ; CHECK-NEXT:    call void @use8(i8 [[XMIN]])
 ; CHECK-NEXT:    call void @use8(i8 [[YMIN]])
 ; CHECK-NEXT:    call void @use8(i8 [[ZMIN]])
@@ -518,8 +518,8 @@ define i8 @umin_not_sub_intrinsic_uses(i8 %x, i8 %y) {
 define i8 @umax_sub_op0(i8 %x, i8 %y) {
 ; CHECK-LABEL: define {{[^@]+}}@umax_sub_op0
 ; CHECK-SAME: (i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[TMP1:%.*]] = call i8 @llvm.usub.sat.i8(i8 [[X]], i8 [[Y]])
-; CHECK-NEXT:    ret i8 [[TMP1]]
+; CHECK-NEXT:    [[R:%.*]] = call i8 @llvm.usub.sat.i8(i8 [[X]], i8 [[Y]])
+; CHECK-NEXT:    ret i8 [[R]]
 ;
   %u = call i8 @llvm.umax.i8(i8 %y, i8 %x)
   %r = sub i8 %u, %y
@@ -529,8 +529,8 @@ define i8 @umax_sub_op0(i8 %x, i8 %y) {
 define <2 x i8> @umax_sub_op0_vec_commute(<2 x i8> %x, <2 x i8> %y) {
 ; CHECK-LABEL: define {{[^@]+}}@umax_sub_op0_vec_commute
 ; CHECK-SAME: (<2 x i8> [[X:%.*]], <2 x i8> [[Y:%.*]]) {
-; CHECK-NEXT:    [[TMP1:%.*]] = call <2 x i8> @llvm.usub.sat.v2i8(<2 x i8> [[X]], <2 x i8> [[Y]])
-; CHECK-NEXT:    ret <2 x i8> [[TMP1]]
+; CHECK-NEXT:    [[R:%.*]] = call <2 x i8> @llvm.usub.sat.v2i8(<2 x i8> [[X]], <2 x i8> [[Y]])
+; CHECK-NEXT:    ret <2 x i8> [[R]]
 ;
   %u = call <2 x i8> @llvm.umax.v2i8(<2 x i8> %x, <2 x i8> %y)
   %r = sub <2 x i8> %u, %y
@@ -592,8 +592,8 @@ define i8 @umax_sub_op1_use(i8 %x, i8 %y) {
 define i8 @umin_sub_op1(i8 %x, i8 %y) {
 ; CHECK-LABEL: define {{[^@]+}}@umin_sub_op1
 ; CHECK-SAME: (i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[TMP1:%.*]] = call i8 @llvm.usub.sat.i8(i8 [[Y]], i8 [[X]])
-; CHECK-NEXT:    ret i8 [[TMP1]]
+; CHECK-NEXT:    [[R:%.*]] = call i8 @llvm.usub.sat.i8(i8 [[Y]], i8 [[X]])
+; CHECK-NEXT:    ret i8 [[R]]
 ;
   %u = call i8 @llvm.umin.i8(i8 %y, i8 %x)
   %r = sub i8 %y, %u
@@ -603,8 +603,8 @@ define i8 @umin_sub_op1(i8 %x, i8 %y) {
 define i8 @umin_sub_op1_commute(i8 %x, i8 %y) {
 ; CHECK-LABEL: define {{[^@]+}}@umin_sub_op1_commute
 ; CHECK-SAME: (i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[TMP1:%.*]] = call i8 @llvm.usub.sat.i8(i8 [[Y]], i8 [[X]])
-; CHECK-NEXT:    ret i8 [[TMP1]]
+; CHECK-NEXT:    [[R:%.*]] = call i8 @llvm.usub.sat.i8(i8 [[Y]], i8 [[X]])
+; CHECK-NEXT:    ret i8 [[R]]
 ;
   %u = call i8 @llvm.umin.i8(i8 %x, i8 %y)
   %r = sub i8 %y, %u
@@ -998,5 +998,149 @@ define i8 @sub_smin0_sub_nsw_commute(i8 %x, i8 %y) {
   ret i8 %r
 }
 
+define i8 @sub_max_min_nsw(i8 %a, i8 %b) {
+; CHECK-LABEL: define {{[^@]+}}@sub_max_min_nsw
+; CHECK-SAME: (i8 [[A:%.*]], i8 [[B:%.*]]) {
+; CHECK-NEXT:    [[SUB:%.*]] = sub nsw i8 [[A]], [[B]]
+; CHECK-NEXT:    [[AB:%.*]] = call i8 @llvm.abs.i8(i8 [[SUB]], i1 true)
+; CHECK-NEXT:    ret i8 [[AB]]
+;
+  %min = call i8 @llvm.smin.i8(i8 %a, i8 %b)
+  %max = call i8 @llvm.smax.i8(i8 %a, i8 %b)
+  %ab = sub nsw i8 %max, %min
+  ret i8 %ab
+}
+
+define i8 @sub_max_min_nuw(i8 %a, i8 %b) {
+; CHECK-LABEL: define {{[^@]+}}@sub_max_min_nuw
+; CHECK-SAME: (i8 [[A:%.*]], i8 [[B:%.*]]) {
+; CHECK-NEXT:    [[SUB:%.*]] = sub nsw i8 [[A]], [[B]]
+; CHECK-NEXT:    [[AB:%.*]] = call i8 @llvm.abs.i8(i8 [[SUB]], i1 true)
+; CHECK-NEXT:    ret i8 [[AB]]
+;
+  %min = call i8 @llvm.smin.i8(i8 %a, i8 %b)
+  %max = call i8 @llvm.smax.i8(i8 %a, i8 %b)
+  %ab = sub nuw i8 %max, %min
+  ret i8 %ab
+}
+
+define i8 @sub_max_min_nsw_commute(i8 %a, i8 %b) {
+; CHECK-LABEL: define {{[^@]+}}@sub_max_min_nsw_commute
+; CHECK-SAME: (i8 [[A:%.*]], i8 [[B:%.*]]) {
+; CHECK-NEXT:    [[SUB:%.*]] = sub nsw i8 [[A]], [[B]]
+; CHECK-NEXT:    [[AB:%.*]] = call i8 @llvm.abs.i8(i8 [[SUB]], i1 true)
+; CHECK-NEXT:    ret i8 [[AB]]
+;
+  %min = call i8 @llvm.smin.i8(i8 %b, i8 %a)
+  %max = call i8 @llvm.smax.i8(i8 %a, i8 %b)
+  %ab = sub nsw i8 %max, %min
+  ret i8 %ab
+}
+
+define i8 @sub_max_min_nuw_commute(i8 %a, i8 %b) {
+; CHECK-LABEL: define {{[^@]+}}@sub_max_min_nuw_commute
+; CHECK-SAME: (i8 [[A:%.*]], i8 [[B:%.*]]) {
+; CHECK-NEXT:    [[SUB:%.*]] = sub nsw i8 [[A]], [[B]]
+; CHECK-NEXT:    [[AB:%.*]] = call i8 @llvm.abs.i8(i8 [[SUB]], i1 true)
+; CHECK-NEXT:    ret i8 [[AB]]
+;
+  %min = call i8 @llvm.smin.i8(i8 %b, i8 %a)
+  %max = call i8 @llvm.smax.i8(i8 %a, i8 %b)
+  %ab = sub nuw i8 %max, %min
+  ret i8 %ab
+}
+
+define <2 x i8> @sub_max_min_vec_nsw(<2 x i8> %a, <2 x i8> %b) {
+; CHECK-LABEL: define {{[^@]+}}@sub_max_min_vec_nsw
+; CHECK-SAME: (<2 x i8> [[A:%.*]], <2 x i8> [[B:%.*]]) {
+; CHECK-NEXT:    [[SUB:%.*]] = sub nsw <2 x i8> [[A]], [[B]]
+; CHECK-NEXT:    [[AB:%.*]] = call <2 x i8> @llvm.abs.v2i8(<2 x i8> [[SUB]], i1 true)
+; CHECK-NEXT:    ret <2 x i8> [[AB]]
+;
+  %min = call <2 x i8> @llvm.smin.v2i8(<2 x i8> %a, <2 x i8> %b)
+  %max = call <2 x i8> @llvm.smax.v2i8(<2 x i8> %a, <2 x i8> %b)
+  %ab = sub nsw <2 x i8> %max, %min
+  ret <2 x i8> %ab
+}
+
+define <2 x i8> @sub_max_min_vec_nuw(<2 x i8> %a, <2 x i8> %b) {
+; CHECK-LABEL: define {{[^@]+}}@sub_max_min_vec_nuw
+; CHECK-SAME: (<2 x i8> [[A:%.*]], <2 x i8> [[B:%.*]]) {
+; CHECK-NEXT:    [[SUB:%.*]] = sub nsw <2 x i8> [[A]], [[B]]
+; CHECK-NEXT:    [[AB:%.*]] = call <2 x i8> @llvm.abs.v2i8(<2 x i8> [[SUB]], i1 true)
+; CHECK-NEXT:    ret <2 x i8> [[AB]]
+;
+  %min = call <2 x i8> @llvm.smin.v2i8(<2 x i8> %a, <2 x i8> %b)
+  %max = call <2 x i8> @llvm.smax.v2i8(<2 x i8> %a, <2 x i8> %b)
+  %ab = sub nuw <2 x i8> %max, %min
+  ret <2 x i8> %ab
+}
+
+define <2 x i8> @sub_max_min_vec_nsw_commute(<2 x i8> %a, <2 x i8> %b) {
+; CHECK-LABEL: define {{[^@]+}}@sub_max_min_vec_nsw_commute
+; CHECK-SAME: (<2 x i8> [[A:%.*]], <2 x i8> [[B:%.*]]) {
+; CHECK-NEXT:    [[SUB:%.*]] = sub nsw <2 x i8> [[A]], [[B]]
+; CHECK-NEXT:    [[AB:%.*]] = call <2 x i8> @llvm.abs.v2i8(<2 x i8> [[SUB]], i1 true)
+; CHECK-NEXT:    ret <2 x i8> [[AB]]
+;
+  %min = call <2 x i8> @llvm.smin.v2i8(<2 x i8> %b, <2 x i8> %a)
+  %max = call <2 x i8> @llvm.smax.v2i8(<2 x i8> %a, <2 x i8> %b)
+  %ab = sub nsw <2 x i8> %max, %min
+  ret <2 x i8> %ab
+}
+
+define <2 x i8> @sub_max_min_vec_nuw_commute(<2 x i8> %a, <2 x i8> %b) {
+; CHECK-LABEL: define {{[^@]+}}@sub_max_min_vec_nuw_commute
+; CHECK-SAME: (<2 x i8> [[A:%.*]], <2 x i8> [[B:%.*]]) {
+; CHECK-NEXT:    [[SUB:%.*]] = sub nsw <2 x i8> [[A]], [[B]]
+; CHECK-NEXT:    [[AB:%.*]] = call <2 x i8> @llvm.abs.v2i8(<2 x i8> [[SUB]], i1 true)
+; CHECK-NEXT:    ret <2 x i8> [[AB]]
+;
+  %min = call <2 x i8> @llvm.smin.v2i8(<2 x i8> %b, <2 x i8> %a)
+  %max = call <2 x i8> @llvm.smax.v2i8(<2 x i8> %a, <2 x i8> %b)
+  %ab = sub nuw <2 x i8> %max, %min
+  ret <2 x i8> %ab
+}
+
+; negative test - multiple use
+
+define i8 @sub_max_min_multi_use(i8 %a, i8 %b) {
+; CHECK-LABEL: define {{[^@]+}}@sub_max_min_multi_use
+; CHECK-SAME: (i8 [[A:%.*]], i8 [[B:%.*]]) {
+; CHECK-NEXT:    [[MIN:%.*]] = call i8 @llvm.smin.i8(i8 [[A]], i8 [[B]])
+; CHECK-NEXT:    call void @use8(i8 [[MIN]])
+; CHECK-NEXT:    [[MAX:%.*]] = call i8 @llvm.smax.i8(i8 [[A]], i8 [[B]])
+; CHECK-NEXT:    call void @use8(i8 [[MAX]])
+; CHECK-NEXT:    [[AB:%.*]] = sub nsw i8 [[MAX]], [[MIN]]
+; CHECK-NEXT:    ret i8 [[AB]]
+;
+  %min = call i8 @llvm.smin.i8(i8 %a, i8 %b)
+  call void @use8(i8 %min)
+  %max = call i8 @llvm.smax.i8(i8 %a, i8 %b)
+  call void @use8(i8 %max)
+  %ab = sub nsw i8 %max, %min
+  ret i8 %ab
+}
+
+define <2 x i8> @sub_max_min_vec_multi_use(<2 x i8> %a, <2 x i8> %b) {
+; CHECK-LABEL: define {{[^@]+}}@sub_max_min_vec_multi_use
+; CHECK-SAME: (<2 x i8> [[A:%.*]], <2 x i8> [[B:%.*]]) {
+; CHECK-NEXT:    [[MIN:%.*]] = call <2 x i8> @llvm.smin.v2i8(<2 x i8> [[A]], <2 x i8> [[B]])
+; CHECK-NEXT:    call void @use8v2(<2 x i8> [[MIN]])
+; CHECK-NEXT:    [[MAX:%.*]] = call <2 x i8> @llvm.smax.v2i8(<2 x i8> [[A]], <2 x i8> [[B]])
+; CHECK-NEXT:    call void @use8v2(<2 x i8> [[MAX]])
+; CHECK-NEXT:    [[AB:%.*]] = sub nsw <2 x i8> [[MAX]], [[MIN]]
+; CHECK-NEXT:    ret <2 x i8> [[AB]]
+;
+  %min = call <2 x i8> @llvm.smin.v2i8(<2 x i8> %a, <2 x i8> %b)
+  call void @use8v2(<2 x i8> %min)
+  %max = call <2 x i8> @llvm.smax.v2i8(<2 x i8> %a, <2 x i8> %b)
+  call void @use8v2(<2 x i8> %max)
+  %ab = sub nsw <2 x i8> %max, %min
+  ret <2 x i8> %ab
+}
+
 declare void @use8(i8)
 declare void @use32(i32 %u)
+
+declare void @use8v2(i8)
diff --git a/llvm/test/Transforms/InstCombine/unused-nonnull.ll b/llvm/test/Transforms/InstCombine/unused-nonnull.ll
index 32dd65977ea86..169cab455f606 100644
--- a/llvm/test/Transforms/InstCombine/unused-nonnull.ll
+++ b/llvm/test/Transforms/InstCombine/unused-nonnull.ll
@@ -9,7 +9,7 @@ target triple = "x86_64-unknown-linux-gnu"
 
 define i32 @main(i32 %argc, ptr %argv) #0 {
 ; CHECK-LABEL: define {{[^@]+}}@main
-; CHECK-SAME: (i32 [[ARGC:%.*]], ptr nocapture readonly [[ARGV:%.*]]) local_unnamed_addr #[[ATTR0:[0-9]+]] {
+; CHECK-SAME: (i32 [[ARGC:%.*]], ptr nocapture readnone [[ARGV:%.*]]) local_unnamed_addr #[[ATTR0:[0-9]+]] {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = icmp slt i32 [[ARGC]], 2
 ; CHECK-NEXT:    [[SPEC_SELECT:%.*]] = select i1 [[TMP0]], i32 0, i32 [[ARGC]]
diff --git a/llvm/test/Transforms/InstCombine/zext.ll b/llvm/test/Transforms/InstCombine/zext.ll
index d223c94e553e1..8aa2a10e6abb2 100644
--- a/llvm/test/Transforms/InstCombine/zext.ll
+++ b/llvm/test/Transforms/InstCombine/zext.ll
@@ -702,3 +702,49 @@ define i8 @zext_icmp_eq_pow2(i8 %y, i8 %x) {
   %r = zext i1 %c to i8
   ret i8 %r
 }
+
+define i64 @zext_icmp_eq_bool_0(ptr %ptr) {
+; CHECK-LABEL: @zext_icmp_eq_bool_0(
+; CHECK-NEXT:    [[VAL:%.*]] = load i64, ptr [[PTR:%.*]], align 8, !range [[RNG0:![0-9]+]]
+; CHECK-NEXT:    [[LEN:%.*]] = xor i64 [[VAL]], 1
+; CHECK-NEXT:    ret i64 [[LEN]]
+;
+  %val = load i64, ptr %ptr, align 8, !range !{i64 0, i64 2}
+  %cmp = icmp eq i64 %val, 0
+  %len = zext i1 %cmp to i64
+  ret i64 %len
+}
+
+define i64 @zext_icmp_eq_bool_1(ptr %ptr) {
+; CHECK-LABEL: @zext_icmp_eq_bool_1(
+; CHECK-NEXT:    [[VAL:%.*]] = load i64, ptr [[PTR:%.*]], align 8, !range [[RNG0]]
+; CHECK-NEXT:    ret i64 [[VAL]]
+;
+  %val = load i64, ptr %ptr, align 8, !range !{i64 0, i64 2}
+  %cmp = icmp eq i64 %val, 1
+  %len = zext i1 %cmp to i64
+  ret i64 %len
+}
+
+define i64 @zext_icmp_ne_bool_0(ptr %ptr) {
+; CHECK-LABEL: @zext_icmp_ne_bool_0(
+; CHECK-NEXT:    [[VAL:%.*]] = load i64, ptr [[PTR:%.*]], align 8, !range [[RNG0]]
+; CHECK-NEXT:    ret i64 [[VAL]]
+;
+  %val = load i64, ptr %ptr, align 8, !range !{i64 0, i64 2}
+  %cmp = icmp ne i64 %val, 0
+  %len = zext i1 %cmp to i64
+  ret i64 %len
+}
+
+define i64 @zext_icmp_ne_bool_1(ptr %ptr) {
+; CHECK-LABEL: @zext_icmp_ne_bool_1(
+; CHECK-NEXT:    [[VAL:%.*]] = load i64, ptr [[PTR:%.*]], align 8, !range [[RNG0]]
+; CHECK-NEXT:    [[LEN:%.*]] = xor i64 [[VAL]], 1
+; CHECK-NEXT:    ret i64 [[LEN]]
+;
+  %val = load i64, ptr %ptr, align 8, !range !{i64 0, i64 2}
+  %cmp = icmp ne i64 %val, 1
+  %len = zext i1 %cmp to i64
+  ret i64 %len
+}
diff --git a/llvm/test/Transforms/InstSimplify/ConstProp/cast-vector.ll b/llvm/test/Transforms/InstSimplify/ConstProp/cast-vector.ll
index 685dfcc3cf2a5..3e4504a166366 100644
--- a/llvm/test/Transforms/InstSimplify/ConstProp/cast-vector.ll
+++ b/llvm/test/Transforms/InstSimplify/ConstProp/cast-vector.ll
@@ -1,32 +1,32 @@
-; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=instsimplify -S | FileCheck %s
-
-; Test constant fold of constant expression GEP used by ptrtoint (the
-; "offsetof-like expression" case).
-; This used to hit an assert due to not supporting vectors in
-; llvm::ConstantFoldCastInstruction when handling ptrtoint.
-define <2 x i16> @test1() {
-; CHECK-LABEL: @test1(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    ret <2 x i16> ptrtoint (<2 x ptr> getelementptr ([10 x i32], ptr null, <2 x i64> zeroinitializer, <2 x i64> <i64 5, i64 7>) to <2 x i16>)
-;
-entry:
-  %gep = getelementptr inbounds [10 x i32], ptr null, i16 0, <2 x i16> <i16 5, i16 7>
-  %vec = ptrtoint <2 x ptr> %gep to <2 x i16>
-  ret <2 x i16> %vec
-}
-
-; Test constant fold of constant expression GEP used by ptrtoint (the
-; "sizeof-like expression" case).
-; This used to hit an assert due to not supporting vectors in
-; llvm::ConstantFoldCastInstruction when handling ptrtoint.
-define <2 x i16> @test2() {
-; CHECK-LABEL: @test2(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    ret <2 x i16> ptrtoint (<2 x ptr> getelementptr (i32, ptr null, <2 x i64> <i64 5, i64 7>) to <2 x i16>)
-;
-entry:
-  %gep = getelementptr i32, ptr null, <2 x i16> <i16 5, i16 7>
-  %vec = ptrtoint <2 x ptr> %gep to <2 x i16>
-  ret <2 x i16> %vec
-}
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -passes=instsimplify -S | FileCheck %s
+
+; Test constant fold of constant expression GEP used by ptrtoint (the
+; "offsetof-like expression" case).
+; This used to hit an assert due to not supporting vectors in
+; llvm::ConstantFoldCastInstruction when handling ptrtoint.
+define <2 x i16> @test1() {
+; CHECK-LABEL: @test1(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    ret <2 x i16> ptrtoint (<2 x ptr> getelementptr inbounds ([10 x i32], ptr null, <2 x i64> zeroinitializer, <2 x i64> <i64 5, i64 7>) to <2 x i16>)
+;
+entry:
+  %gep = getelementptr inbounds [10 x i32], ptr null, i16 0, <2 x i16> <i16 5, i16 7>
+  %vec = ptrtoint <2 x ptr> %gep to <2 x i16>
+  ret <2 x i16> %vec
+}
+
+; Test constant fold of constant expression GEP used by ptrtoint (the
+; "sizeof-like expression" case).
+; This used to hit an assert due to not supporting vectors in
+; llvm::ConstantFoldCastInstruction when handling ptrtoint.
+define <2 x i16> @test2() {
+; CHECK-LABEL: @test2(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    ret <2 x i16> ptrtoint (<2 x ptr> getelementptr (i32, ptr null, <2 x i64> <i64 5, i64 7>) to <2 x i16>)
+;
+entry:
+  %gep = getelementptr i32, ptr null, <2 x i16> <i16 5, i16 7>
+  %vec = ptrtoint <2 x ptr> %gep to <2 x i16>
+  ret <2 x i16> %vec
+}
diff --git a/llvm/test/Transforms/InstSimplify/ConstProp/constant-expr.ll b/llvm/test/Transforms/InstSimplify/ConstProp/constant-expr.ll
index 8978d9200afb6..bdecda847d372 100644
--- a/llvm/test/Transforms/InstSimplify/ConstProp/constant-expr.ll
+++ b/llvm/test/Transforms/InstSimplify/ConstProp/constant-expr.ll
@@ -84,12 +84,3 @@
 ; CHECK: pr9011_14 = constant i128 0
 @pr9011_15 = constant i128 bitcast (<4 x i32> zeroinitializer to i128)
 ; CHECK: pr9011_15 = constant i128 0
-
-@select = internal constant
-          i32 select (i1 icmp ult (i32 ptrtoint (ptr @X to i32),
-                                   i32 ptrtoint (ptr @Y to i32)),
-            i32 select (i1 icmp ult (i32 ptrtoint (ptr @X to i32),
-                                     i32 ptrtoint (ptr @Y to i32)),
-               i32 10, i32 20),
-            i32 30)
-; CHECK: select = internal constant i32 select {{.*}} i32 10, i32 30
diff --git a/llvm/test/Transforms/InstSimplify/ConstProp/vscale-inseltpoison.ll b/llvm/test/Transforms/InstSimplify/ConstProp/vscale-inseltpoison.ll
index b5bdbfcf24d82..63b9e12abfde5 100644
--- a/llvm/test/Transforms/InstSimplify/ConstProp/vscale-inseltpoison.ll
+++ b/llvm/test/Transforms/InstSimplify/ConstProp/vscale-inseltpoison.ll
@@ -202,19 +202,6 @@ define <vscale x 4 x i32> @shufflevector() {
   ret <vscale x 4 x i32> %i2
 }
 
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-;; Memory Access and Addressing Operations
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-
-define <vscale x 2 x double> @load() {
-; CHECK-LABEL: @load(
-; CHECK-NEXT:    [[R:%.*]] = load <vscale x 2 x double>, ptr getelementptr (<vscale x 2 x double>, ptr null, i64 1), align 16
-; CHECK-NEXT:    ret <vscale x 2 x double> [[R]]
-;
-  %r = load <vscale x 2 x double>, ptr getelementptr (<vscale x 2 x double>, ptr null, i64 1)
-  ret <vscale x 2 x double> %r
-}
-
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; Conversion Operations
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
diff --git a/llvm/test/Transforms/InstSimplify/ConstProp/vscale.ll b/llvm/test/Transforms/InstSimplify/ConstProp/vscale.ll
index 0d597939cd13d..585e736650b78 100644
--- a/llvm/test/Transforms/InstSimplify/ConstProp/vscale.ll
+++ b/llvm/test/Transforms/InstSimplify/ConstProp/vscale.ll
@@ -202,19 +202,6 @@ define <vscale x 4 x i32> @shufflevector() {
   ret <vscale x 4 x i32> %i2
 }
 
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-;; Memory Access and Addressing Operations
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-
-define <vscale x 2 x double> @load() {
-; CHECK-LABEL: @load(
-; CHECK-NEXT:    [[R:%.*]] = load <vscale x 2 x double>, ptr getelementptr (<vscale x 2 x double>, ptr null, i64 1), align 16
-; CHECK-NEXT:    ret <vscale x 2 x double> [[R]]
-;
-  %r = load <vscale x 2 x double>, ptr getelementptr (<vscale x 2 x double>, ptr null, i64 1)
-  ret <vscale x 2 x double> %r
-}
-
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; Conversion Operations
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
diff --git a/llvm/test/Transforms/InstSimplify/phi.ll b/llvm/test/Transforms/InstSimplify/phi.ll
index f1cdd8b960cbb..c97326ea05137 100644
--- a/llvm/test/Transforms/InstSimplify/phi.ll
+++ b/llvm/test/Transforms/InstSimplify/phi.ll
@@ -152,3 +152,37 @@ EXIT:
   %r = phi i8 [poison, %A], [poison, %B]
   ret i8 %r
 }
+
+; Should not fold srem to -1 due to incorrect context instruction when
+; threading over phi.
+define i32 @pr61312() {
+; CHECK-LABEL: @pr61312(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[FOR_COND:%.*]]
+; CHECK:       for.cond:
+; CHECK-NEXT:    [[A_0:%.*]] = phi i32 [ 2, [[ENTRY:%.*]] ], [ [[DEC:%.*]], [[FOR_INC:%.*]] ]
+; CHECK-NEXT:    [[CMP:%.*]] = icmp sge i32 [[A_0]], 0
+; CHECK-NEXT:    br i1 [[CMP]], label [[FOR_INC]], label [[FOR_END:%.*]]
+; CHECK:       for.inc:
+; CHECK-NEXT:    [[DEC]] = add nsw i32 [[A_0]], -1
+; CHECK-NEXT:    br label [[FOR_COND]]
+; CHECK:       for.end:
+; CHECK-NEXT:    [[REM:%.*]] = srem i32 -1, [[A_0]]
+; CHECK-NEXT:    ret i32 [[REM]]
+;
+entry:
+  br label %for.cond
+
+for.cond:
+  %a.0 = phi i32 [ 2, %entry ], [ %dec, %for.inc ]
+  %cmp = icmp sge i32 %a.0, 0
+  br i1 %cmp, label %for.inc, label %for.end
+
+for.inc:
+  %dec = add nsw i32 %a.0, -1
+  br label %for.cond
+
+for.end:
+  %rem = srem i32 -1, %a.0
+  ret i32 %rem
+}
diff --git a/llvm/test/Transforms/InstSimplify/pr28725.ll b/llvm/test/Transforms/InstSimplify/pr28725.ll
index 67e6170c4f1bc..8adfecc5128ec 100644
--- a/llvm/test/Transforms/InstSimplify/pr28725.ll
+++ b/llvm/test/Transforms/InstSimplify/pr28725.ll
@@ -6,12 +6,14 @@
 define <2 x i16> @test1() {
 ; CHECK-LABEL: @test1(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[E:%.*]] = extractvalue [[S:%.*]] select (i1 icmp eq (i16 extractelement (<2 x i16> bitcast (<1 x i32> <i32 1> to <2 x i16>), i32 0), i16 0), [[S]] zeroinitializer, [[S]] { i16 0, i32 1 }), 0
+; CHECK-NEXT:    [[SEL:%.*]] = select i1 icmp eq (i16 extractelement (<2 x i16> bitcast (<1 x i32> <i32 1> to <2 x i16>), i32 0), i16 0), [[S:%.*]] zeroinitializer, [[S]] { i16 0, i32 1 }
+; CHECK-NEXT:    [[E:%.*]] = extractvalue [[S]] [[SEL]], 0
 ; CHECK-NEXT:    [[B:%.*]] = insertelement <2 x i16> <i16 undef, i16 0>, i16 [[E]], i32 0
 ; CHECK-NEXT:    ret <2 x i16> [[B]]
 ;
 entry:
-  %e = extractvalue %S select (i1 icmp eq (i16 extractelement (<2 x i16> bitcast (<1 x i32> <i32 1> to <2 x i16>), i32 0), i16 0), %S zeroinitializer, %S { i16 0, i32 1 }), 0
+  %sel = select i1 icmp eq (i16 extractelement (<2 x i16> bitcast (<1 x i32> <i32 1> to <2 x i16>), i32 0), i16 0), %S zeroinitializer, %S { i16 0, i32 1 }
+  %e = extractvalue %S %sel, 0
   %b = insertelement <2 x i16> <i16 undef, i16 0>, i16 %e, i32 0
   ret <2 x i16> %b
 }
diff --git a/llvm/test/Transforms/InstSimplify/shift.ll b/llvm/test/Transforms/InstSimplify/shift.ll
index 9b145813b587e..83751765df839 100644
--- a/llvm/test/Transforms/InstSimplify/shift.ll
+++ b/llvm/test/Transforms/InstSimplify/shift.ll
@@ -350,3 +350,51 @@ define <vscale x 4 x i16> @lshr_scalable_overshift(<vscale x 4 x i16> %va) {
   %vc = lshr <vscale x 4 x i16> %va, shufflevector (<vscale x 4 x i16> insertelement (<vscale x 4 x i16> poison, i16 16, i32 0), <vscale x 4 x i16> poison, <vscale x 4 x i32> zeroinitializer)
   ret <vscale x 4 x i16> %vc
 }
+
+; shl nsw+nuw is 0
+define i8 @shl_nsw_nuw_7_eq_0(i8 %x) {
+; CHECK-LABEL: @shl_nsw_nuw_7_eq_0(
+; CHECK-NEXT:    ret i8 0
+;
+  %y = shl nsw nuw i8 %x, 7
+  ret i8 %y
+}
+
+; Make sure we match the element width
+define <2 x i8> @shl_vec_nsw_nuw_7_eq_0(<2 x i8> %x) {
+; CHECK-LABEL: @shl_vec_nsw_nuw_7_eq_0(
+; CHECK-NEXT:    ret <2 x i8> zeroinitializer
+;
+  %y = shl nsw nuw <2 x i8> %x, <i8 7, i8 7>
+  ret <2 x i8> %y
+}
+
+; negative test (missing nuw)
+define i8 @shl_nsw_7_fail_missing_nuw(i8 %x) {
+; CHECK-LABEL: @shl_nsw_7_fail_missing_nuw(
+; CHECK-NEXT:    [[Y:%.*]] = shl nsw i8 [[X:%.*]], 7
+; CHECK-NEXT:    ret i8 [[Y]]
+;
+  %y = shl nsw i8 %x, 7
+  ret i8 %y
+}
+
+; negative test (missing nsw)
+define i8 @shl_nuw_7_fail_missing_nsw(i8 %x) {
+; CHECK-LABEL: @shl_nuw_7_fail_missing_nsw(
+; CHECK-NEXT:    [[Y:%.*]] = shl nuw i8 [[X:%.*]], 7
+; CHECK-NEXT:    ret i8 [[Y]]
+;
+  %y = shl nuw i8 %x, 7
+  ret i8 %y
+}
+
+; negative test (shift value != bitwdth - 1)
+define i8 @shl_nsw_nuw_6_do_nothing(i8 %x) {
+; CHECK-LABEL: @shl_nsw_nuw_6_do_nothing(
+; CHECK-NEXT:    [[Y:%.*]] = shl nuw nsw i8 [[X:%.*]], 6
+; CHECK-NEXT:    ret i8 [[Y]]
+;
+  %y = shl nsw nuw i8 %x, 6
+  ret i8 %y
+}
diff --git a/llvm/test/Transforms/InstSimplify/vscale-inseltpoison.ll b/llvm/test/Transforms/InstSimplify/vscale-inseltpoison.ll
index 7d4a7c2e6c3a7..5b34f417e4093 100644
--- a/llvm/test/Transforms/InstSimplify/vscale-inseltpoison.ll
+++ b/llvm/test/Transforms/InstSimplify/vscale-inseltpoison.ll
@@ -152,51 +152,6 @@ define <vscale x 4 x float> @bitcast() {
 ;; Memory Access and Addressing Operations
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 
-; getelementptr
-
-define <vscale x 4 x ptr> @getelementptr_constant_foldable_1() {
-; CHECK-LABEL: @getelementptr_constant_foldable_1(
-; CHECK-NEXT:    ret <vscale x 4 x ptr> zeroinitializer
-;
-  %ptr = getelementptr i32, <vscale x 4 x ptr> zeroinitializer, <vscale x 4 x i64> undef
-  ret <vscale x 4 x ptr> %ptr
-}
-
-define <vscale x 4 x ptr> @getelementptr_constant_foldable_2() {
-; CHECK-LABEL: @getelementptr_constant_foldable_2(
-; CHECK-NEXT:    ret <vscale x 4 x ptr> zeroinitializer
-;
-  %ptr = getelementptr <vscale x 4 x i32>, ptr null, <vscale x 4 x i64> undef
-  ret <vscale x 4 x ptr> %ptr
-}
-
-; fold getelementptr P, 0 -> P.
-define ptr @getelementptr_constant_foldable_3() {
-; CHECK-LABEL: @getelementptr_constant_foldable_3(
-; CHECK-NEXT:    ret ptr null
-;
-  ret ptr null
-}
-
-define ptr @getelementptr_not_constant_foldable(i64 %x) {
-; CHECK-LABEL: @getelementptr_not_constant_foldable(
-; CHECK-NEXT:    [[PTR:%.*]] = getelementptr <vscale x 4 x i32>, ptr null, i64 [[X:%.*]]
-; CHECK-NEXT:    ret ptr [[PTR]]
-;
-  %ptr = getelementptr <vscale x 4 x i32>, ptr null, i64 %x
-  ret ptr %ptr
-}
-
-; Check GEP's result is known to be non-null.
-define i1 @getelementptr_check_non_null(ptr %ptr) {
-; CHECK-LABEL: @getelementptr_check_non_null(
-; CHECK-NEXT:    ret i1 false
-;
-  %x = getelementptr inbounds <vscale x 16 x i8>, ptr %ptr, i32 1
-  %cmp = icmp eq ptr %x, null
-  ret i1 %cmp
-}
-
 define i32 @extractelement_splat_constant_index(i32 %v) {
 ; CHECK-LABEL: @extractelement_splat_constant_index(
 ; CHECK-NEXT:    ret i32 [[V:%.*]]
diff --git a/llvm/test/Transforms/InstSimplify/vscale.ll b/llvm/test/Transforms/InstSimplify/vscale.ll
index 47d281ea2d46d..66f629aac3a52 100644
--- a/llvm/test/Transforms/InstSimplify/vscale.ll
+++ b/llvm/test/Transforms/InstSimplify/vscale.ll
@@ -192,10 +192,10 @@ define ptr @getelementptr_constant_foldable_3() {
 
 define ptr @getelementptr_not_constant_foldable(i64 %x) {
 ; CHECK-LABEL: @getelementptr_not_constant_foldable(
-; CHECK-NEXT:    [[PTR:%.*]] = getelementptr <vscale x 4 x i32>, ptr null, i64 [[X:%.*]]
+; CHECK-NEXT:    [[PTR:%.*]] = getelementptr <vscale x 4 x i32>, ptr null, i64 1
 ; CHECK-NEXT:    ret ptr [[PTR]]
 ;
-  %ptr = getelementptr <vscale x 4 x i32>, ptr null, i64 %x
+  %ptr = getelementptr <vscale x 4 x i32>, ptr null, i64 1
   ret ptr %ptr
 }
 
diff --git a/llvm/test/Transforms/LICM/2011-04-06-PromoteResultOfPromotion.ll b/llvm/test/Transforms/LICM/2011-04-06-PromoteResultOfPromotion.ll
index 69a82d8444b55..33027189dc5c0 100644
--- a/llvm/test/Transforms/LICM/2011-04-06-PromoteResultOfPromotion.ll
+++ b/llvm/test/Transforms/LICM/2011-04-06-PromoteResultOfPromotion.ll
@@ -1,15 +1,29 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
 ; RUN: opt < %s -passes=licm -S | FileCheck %s
 ; PR9634
 
 @g_58 = common global i32 0, align 4
 @g_116 = common global ptr null, align 8
 
-define void @f() nounwind {
-
-; CHECK: entry:
-; CHECK: alloca [9 x i16]
-; CHECK: load i32, ptr @g_58
-; CHECK: br label %for.body
+define void @f() {
+; CHECK-LABEL: define void @f() {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[L_87_I:%.*]] = alloca [9 x i16], align 16
+; CHECK-NEXT:    [[G_58_PROMOTED:%.*]] = load i32, ptr @g_58, align 4, !tbaa [[TBAA0:![0-9]+]]
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[TMP31:%.*]] = phi i32 [ [[G_58_PROMOTED]], [[ENTRY:%.*]] ], [ [[OR:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[INC12:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[INC:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[OR]] = or i32 [[TMP31]], 10
+; CHECK-NEXT:    [[INC]] = add nsw i32 [[INC12]], 1
+; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 [[INC]], 4
+; CHECK-NEXT:    br i1 [[CMP]], label [[FOR_BODY]], label [[FOR_END:%.*]]
+; CHECK:       for.end:
+; CHECK-NEXT:    [[OR_LCSSA:%.*]] = phi i32 [ [[OR]], [[FOR_BODY]] ]
+; CHECK-NEXT:    store ptr @g_58, ptr @g_116, align 8, !tbaa [[TBAA4:![0-9]+]]
+; CHECK-NEXT:    store i32 [[OR_LCSSA]], ptr @g_58, align 4, !tbaa [[TBAA0]]
+; CHECK-NEXT:    ret void
+;
 
 entry:
   %l_87.i = alloca [9 x i16], align 16
diff --git a/llvm/test/Transforms/LICM/explicit_guards.ll b/llvm/test/Transforms/LICM/explicit_guards.ll
index be963b0b01994..469acf1bfffe2 100644
--- a/llvm/test/Transforms/LICM/explicit_guards.ll
+++ b/llvm/test/Transforms/LICM/explicit_guards.ll
@@ -5,16 +5,27 @@ declare void @llvm.experimental.guard(i1,...)
 declare void @maythrow()
 
 ; Make sure that we do not hoist widenable_cond out of loop.
-define void @hoist_widenable_cond(i1 %cond, i32 %N, i32 %M) {
-; CHECK-LABEL: @hoist_widenable_cond(
+; Widenable conditions don't actually alias anything or throw, however
+; hoisting them is not profitable because it can prevent guard widening
+; in loop.
+; If we want to widen guards in branch form, we need to generate them
+; at the point of widenable_condition call
+; (see https://github.com/llvm/llvm-project/issues/60234). Because of
+; this, hoisting a widenable condition out of loop can make widening of
+; in-loop conditions impossible (e.g. if they depend on loop variant). On
+; the other hand, hoisting of the WC doesn't really see profitable,
+; because it will be optimized away later anyways. So the test checks that
+; LICM will not move it out of loop.
+define void @do_not_hoist_widenable_cond(i1 %cond, i32 %N, i32 %M) {
+; CHECK-LABEL: @do_not_hoist_widenable_cond(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[WIDENABLE_COND:%.*]] = call i1 @llvm.experimental.widenable.condition()
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[GUARDED:%.*]] ]
 ; CHECK-NEXT:    [[GUARD_COND:%.*]] = icmp slt i32 [[IV]], [[N:%.*]]
+; CHECK-NEXT:    [[WIDENABLE_COND:%.*]] = call i1 @llvm.experimental.widenable.condition()
 ; CHECK-NEXT:    [[EXIPLICIT_GUARD_COND:%.*]] = and i1 [[GUARD_COND]], [[WIDENABLE_COND]]
-; CHECK-NEXT:    br i1 [[EXIPLICIT_GUARD_COND]], label [[GUARDED]], label [[DEOPT:%.*]], !prof !0
+; CHECK-NEXT:    br i1 [[EXIPLICIT_GUARD_COND]], label [[GUARDED]], label [[DEOPT:%.*]], !prof [[PROF0:![0-9]+]]
 ; CHECK:       deopt:
 ; CHECK-NEXT:    call void (...) @llvm.experimental.deoptimize.isVoid() [ "deopt"() ]
 ; CHECK-NEXT:    ret void
@@ -40,17 +51,17 @@ exit:
   ret void
 }
 
-define void @hoist_widenable_cond_speculate(i1 %cond, i32 %N, i32 %M) {
-; CHECK-LABEL: @hoist_widenable_cond_speculate(
+define void @do_not_hoist_widenable_cond_speculate(i1 %cond, i32 %N, i32 %M) {
+; CHECK-LABEL: @do_not_hoist_widenable_cond_speculate(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[WIDENABLE_COND:%.*]] = call i1 @llvm.experimental.widenable.condition()
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[GUARDED:%.*]] ]
 ; CHECK-NEXT:    [[GUARD_COND:%.*]] = icmp slt i32 [[IV]], [[N:%.*]]
 ; CHECK-NEXT:    call void @maythrow()
+; CHECK-NEXT:    [[WIDENABLE_COND:%.*]] = call i1 @llvm.experimental.widenable.condition()
 ; CHECK-NEXT:    [[EXIPLICIT_GUARD_COND:%.*]] = and i1 [[GUARD_COND]], [[WIDENABLE_COND]]
-; CHECK-NEXT:    br i1 [[EXIPLICIT_GUARD_COND]], label [[GUARDED]], label [[DEOPT:%.*]], !prof !0
+; CHECK-NEXT:    br i1 [[EXIPLICIT_GUARD_COND]], label [[GUARDED]], label [[DEOPT:%.*]], !prof [[PROF0]]
 ; CHECK:       deopt:
 ; CHECK-NEXT:    call void (...) @llvm.experimental.deoptimize.isVoid() [ "deopt"() ]
 ; CHECK-NEXT:    ret void
@@ -81,14 +92,14 @@ exit:
 define void @hoist_invariant_load(i1 %cond, ptr %np, i32 %M) {
 ; CHECK-LABEL: @hoist_invariant_load(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[N:%.*]] = load i32, ptr [[NP:%.*]]
-; CHECK-NEXT:    [[WIDENABLE_COND:%.*]] = call i1 @llvm.experimental.widenable.condition()
+; CHECK-NEXT:    [[N:%.*]] = load i32, ptr [[NP:%.*]], align 4
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[GUARDED:%.*]] ]
 ; CHECK-NEXT:    [[GUARD_COND:%.*]] = icmp slt i32 [[IV]], [[N]]
+; CHECK-NEXT:    [[WIDENABLE_COND:%.*]] = call i1 @llvm.experimental.widenable.condition()
 ; CHECK-NEXT:    [[EXIPLICIT_GUARD_COND:%.*]] = and i1 [[GUARD_COND]], [[WIDENABLE_COND]]
-; CHECK-NEXT:    br i1 [[EXIPLICIT_GUARD_COND]], label [[GUARDED]], label [[DEOPT:%.*]], !prof !0
+; CHECK-NEXT:    br i1 [[EXIPLICIT_GUARD_COND]], label [[GUARDED]], label [[DEOPT:%.*]], !prof [[PROF0]]
 ; CHECK:       deopt:
 ; CHECK-NEXT:    call void (...) @llvm.experimental.deoptimize.isVoid() [ "deopt"() ]
 ; CHECK-NEXT:    ret void
diff --git a/llvm/test/Transforms/LICM/min_max.ll b/llvm/test/Transforms/LICM/min_max.ll
index 0eeef1880c897..c2bf0a7f20cc1 100644
--- a/llvm/test/Transforms/LICM/min_max.ll
+++ b/llvm/test/Transforms/LICM/min_max.ll
@@ -1,16 +1,15 @@
 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -S %s -passes='loop-mssa(licm)'  -verify-memoryssa  | FileCheck %s
 
-; TODO: turn to %iv <u umin(inv_1, inv_2) and hoist it out of loop.
+; turn to %iv <u umin(inv_1, inv_2) and hoist it out of loop.
 define i32 @test_ult(i32 %start, i32 %inv_1, i32 %inv_2) {
 ; CHECK-LABEL: @test_ult(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_UMIN:%.*]] = call i32 @llvm.umin.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ult i32 [[IV]], [[INV_1:%.*]]
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ult i32 [[IV]], [[INV_2:%.*]]
-; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp ult i32 [[IV]], [[INVARIANT_UMIN]]
 ; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
 ; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
@@ -32,16 +31,46 @@ exit:
   ret i32 %iv
 }
 
-; TODO: turn to %iv <=u umin(inv_1, inv_2) and hoist it out of loop.
+; Do not optimize: pointer types.
+define ptr @test_ult_ptr_neg(ptr %start, ptr %inv_1, ptr %inv_2) {
+; CHECK-LABEL: @test_ult_ptr_neg(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi ptr [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ult ptr [[IV]], [[INV_1:%.*]]
+; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ult ptr [[IV]], [[INV_2:%.*]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[IV_NEXT]] = getelementptr i32, ptr [[IV]], i64 4
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi ptr [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret ptr [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi ptr [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp ult ptr %iv, %inv_1
+  %cmp_2 = icmp ult ptr %iv, %inv_2
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv.next = getelementptr i32, ptr %iv, i64 4
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret ptr %iv
+}
+
+; turn to %iv <=u umin(inv_1, inv_2) and hoist it out of loop.
 define i32 @test_ule(i32 %start, i32 %inv_1, i32 %inv_2) {
 ; CHECK-LABEL: @test_ule(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_UMIN:%.*]] = call i32 @llvm.umin.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ule i32 [[IV]], [[INV_1:%.*]]
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ule i32 [[IV]], [[INV_2:%.*]]
-; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp ule i32 [[IV]], [[INVARIANT_UMIN]]
 ; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
 ; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
@@ -63,16 +92,15 @@ exit:
   ret i32 %iv
 }
 
-; TODO: turn to %iv <s smin(inv_1, inv_2) and hoist it out of loop.
+; turn to %iv <s smin(inv_1, inv_2) and hoist it out of loop.
 define i32 @test_slt(i32 %start, i32 %inv_1, i32 %inv_2) {
 ; CHECK-LABEL: @test_slt(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_SMIN:%.*]] = call i32 @llvm.smin.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp slt i32 [[IV]], [[INV_1:%.*]]
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp slt i32 [[IV]], [[INV_2:%.*]]
-; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp slt i32 [[IV]], [[INVARIANT_SMIN]]
 ; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
 ; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
@@ -94,16 +122,15 @@ exit:
   ret i32 %iv
 }
 
-; TODO: turn to %iv <=s smin(inv_1, inv_2) and hoist it out of loop.
+; turn to %iv <=s smin(inv_1, inv_2) and hoist it out of loop.
 define i32 @test_sle(i32 %start, i32 %inv_1, i32 %inv_2) {
 ; CHECK-LABEL: @test_sle(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_SMIN:%.*]] = call i32 @llvm.smin.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp sle i32 [[IV]], [[INV_1:%.*]]
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp sle i32 [[IV]], [[INV_2:%.*]]
-; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp sle i32 [[IV]], [[INVARIANT_SMIN]]
 ; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
 ; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
@@ -125,16 +152,15 @@ exit:
   ret i32 %iv
 }
 
-; TODO: turn to %iv >u umax(inv_1, inv_2) and hoist it out of loop.
+; turn to %iv >u umax(inv_1, inv_2) and hoist it out of loop.
 define i32 @test_ugt(i32 %start, i32 %inv_1, i32 %inv_2) {
 ; CHECK-LABEL: @test_ugt(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_UMAX:%.*]] = call i32 @llvm.umax.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ugt i32 [[IV]], [[INV_1:%.*]]
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ugt i32 [[IV]], [[INV_2:%.*]]
-; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp ugt i32 [[IV]], [[INVARIANT_UMAX]]
 ; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
 ; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
@@ -156,16 +182,15 @@ exit:
   ret i32 %iv
 }
 
-; TODO: turn to %iv >=u umax(inv_1, inv_2) and hoist it out of loop.
+; turn to %iv >=u umax(inv_1, inv_2) and hoist it out of loop.
 define i32 @test_uge(i32 %start, i32 %inv_1, i32 %inv_2) {
 ; CHECK-LABEL: @test_uge(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_UMAX:%.*]] = call i32 @llvm.umax.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp uge i32 [[IV]], [[INV_1:%.*]]
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp uge i32 [[IV]], [[INV_2:%.*]]
-; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp uge i32 [[IV]], [[INVARIANT_UMAX]]
 ; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
 ; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
@@ -187,16 +212,15 @@ exit:
   ret i32 %iv
 }
 
-; TODO: turn to %iv >s smax(inv_1, inv_2) and hoist it out of loop.
+; turn to %iv >s smax(inv_1, inv_2) and hoist it out of loop.
 define i32 @test_sgt(i32 %start, i32 %inv_1, i32 %inv_2) {
 ; CHECK-LABEL: @test_sgt(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_SMAX:%.*]] = call i32 @llvm.smax.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp sgt i32 [[IV]], [[INV_1:%.*]]
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp sgt i32 [[IV]], [[INV_2:%.*]]
-; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp sgt i32 [[IV]], [[INVARIANT_SMAX]]
 ; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
 ; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
@@ -218,16 +242,15 @@ exit:
   ret i32 %iv
 }
 
-; TODO: turn to %iv >=s smax(inv_1, inv_2) and hoist it out of loop.
+; turn to %iv >=s smax(inv_1, inv_2) and hoist it out of loop.
 define i32 @test_sge(i32 %start, i32 %inv_1, i32 %inv_2) {
 ; CHECK-LABEL: @test_sge(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_SMAX:%.*]] = call i32 @llvm.smax.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp sge i32 [[IV]], [[INV_1:%.*]]
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp sge i32 [[IV]], [[INV_2:%.*]]
-; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp sge i32 [[IV]], [[INVARIANT_SMAX]]
 ; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
 ; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
@@ -249,16 +272,15 @@ exit:
   ret i32 %iv
 }
 
-; TODO: Turn OR to AND and handle accordingly.
+; Turn OR to AND and handle accordingly.
 define i32 @test_ult_inv(i32 %start, i32 %inv_1, i32 %inv_2) {
 ; CHECK-LABEL: @test_ult_inv(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_UMAX:%.*]] = call i32 @llvm.umax.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ult i32 [[IV]], [[INV_1:%.*]]
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ult i32 [[IV]], [[INV_2:%.*]]
-; CHECK-NEXT:    [[LOOP_COND:%.*]] = or i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp ult i32 [[IV]], [[INVARIANT_UMAX]]
 ; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
 ; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
@@ -280,16 +302,15 @@ exit:
   ret i32 %iv
 }
 
-; TODO: Turn OR to AND and handle accordingly.
+; Turn OR to AND and handle accordingly.
 define i32 @test_ule_inv(i32 %start, i32 %inv_1, i32 %inv_2) {
 ; CHECK-LABEL: @test_ule_inv(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_UMAX:%.*]] = call i32 @llvm.umax.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ule i32 [[IV]], [[INV_1:%.*]]
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ule i32 [[IV]], [[INV_2:%.*]]
-; CHECK-NEXT:    [[LOOP_COND:%.*]] = or i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp ule i32 [[IV]], [[INVARIANT_UMAX]]
 ; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
 ; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
@@ -311,16 +332,15 @@ exit:
   ret i32 %iv
 }
 
-; TODO: Turn OR to AND and handle accordingly.
+; Turn OR to AND and handle accordingly.
 define i32 @test_slt_inv(i32 %start, i32 %inv_1, i32 %inv_2) {
 ; CHECK-LABEL: @test_slt_inv(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_SMAX:%.*]] = call i32 @llvm.smax.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp slt i32 [[IV]], [[INV_1:%.*]]
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp slt i32 [[IV]], [[INV_2:%.*]]
-; CHECK-NEXT:    [[LOOP_COND:%.*]] = or i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp slt i32 [[IV]], [[INVARIANT_SMAX]]
 ; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
 ; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
@@ -342,16 +362,15 @@ exit:
   ret i32 %iv
 }
 
-; TODO: Turn OR to AND and handle accordingly.
+; Turn OR to AND and handle accordingly.
 define i32 @test_sle_inv(i32 %start, i32 %inv_1, i32 %inv_2) {
 ; CHECK-LABEL: @test_sle_inv(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_SMAX:%.*]] = call i32 @llvm.smax.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp sle i32 [[IV]], [[INV_1:%.*]]
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp sle i32 [[IV]], [[INV_2:%.*]]
-; CHECK-NEXT:    [[LOOP_COND:%.*]] = or i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp sle i32 [[IV]], [[INVARIANT_SMAX]]
 ; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
 ; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
@@ -373,16 +392,15 @@ exit:
   ret i32 %iv
 }
 
-; TODO: Turn OR to AND and handle accordingly.
+; Turn OR to AND and handle accordingly.
 define i32 @test_ugt_inv(i32 %start, i32 %inv_1, i32 %inv_2) {
 ; CHECK-LABEL: @test_ugt_inv(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_UMIN:%.*]] = call i32 @llvm.umin.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ugt i32 [[IV]], [[INV_1:%.*]]
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ugt i32 [[IV]], [[INV_2:%.*]]
-; CHECK-NEXT:    [[LOOP_COND:%.*]] = or i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp ugt i32 [[IV]], [[INVARIANT_UMIN]]
 ; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
 ; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
@@ -404,16 +422,15 @@ exit:
   ret i32 %iv
 }
 
-; TODO: Turn OR to AND and handle accordingly.
+; Turn OR to AND and handle accordingly.
 define i32 @test_uge_inv(i32 %start, i32 %inv_1, i32 %inv_2) {
 ; CHECK-LABEL: @test_uge_inv(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_UMIN:%.*]] = call i32 @llvm.umin.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp uge i32 [[IV]], [[INV_1:%.*]]
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp uge i32 [[IV]], [[INV_2:%.*]]
-; CHECK-NEXT:    [[LOOP_COND:%.*]] = or i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp uge i32 [[IV]], [[INVARIANT_UMIN]]
 ; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
 ; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
@@ -435,16 +452,15 @@ exit:
   ret i32 %iv
 }
 
-; TODO: Turn OR to AND and handle accordingly.
+; Turn OR to AND and handle accordingly.
 define i32 @test_sgt_inv(i32 %start, i32 %inv_1, i32 %inv_2) {
 ; CHECK-LABEL: @test_sgt_inv(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_SMIN:%.*]] = call i32 @llvm.smin.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp sgt i32 [[IV]], [[INV_1:%.*]]
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp sgt i32 [[IV]], [[INV_2:%.*]]
-; CHECK-NEXT:    [[LOOP_COND:%.*]] = or i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp sgt i32 [[IV]], [[INVARIANT_SMIN]]
 ; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
 ; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
@@ -466,16 +482,15 @@ exit:
   ret i32 %iv
 }
 
-; TODO: Turn OR to AND and handle accordingly.
+; Turn OR to AND and handle accordingly.
 define i32 @test_sge_inv(i32 %start, i32 %inv_1, i32 %inv_2) {
 ; CHECK-LABEL: @test_sge_inv(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_SMIN:%.*]] = call i32 @llvm.smin.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp sge i32 [[IV]], [[INV_1:%.*]]
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp sge i32 [[IV]], [[INV_2:%.*]]
-; CHECK-NEXT:    [[LOOP_COND:%.*]] = or i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp sge i32 [[IV]], [[INVARIANT_SMIN]]
 ; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
 ; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
@@ -496,3 +511,731 @@ loop:
 exit:
   ret i32 %iv
 }
+
+; turn to %iv <u umin(inv_1, inv_2) and hoist it out of loop.
+define i32 @test_ult_swapped(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_ult_swapped(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_UMIN:%.*]] = call i32 @llvm.umin.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp ult i32 [[IV]], [[INVARIANT_UMIN]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp ugt i32 %inv_1, %iv
+  %cmp_2 = icmp ugt i32 %inv_2, %iv
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv
+}
+
+; turn to %iv <=u umin(inv_1, inv_2) and hoist it out of loop.
+define i32 @test_ule_swapped(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_ule_swapped(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_UMIN:%.*]] = call i32 @llvm.umin.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp ule i32 [[IV]], [[INVARIANT_UMIN]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp uge i32 %inv_1, %iv
+  %cmp_2 = icmp uge i32 %inv_2, %iv
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv
+}
+
+; turn to %iv <s smin(inv_1, inv_2) and hoist it out of loop.
+define i32 @test_slt_swapped(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_slt_swapped(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_SMIN:%.*]] = call i32 @llvm.smin.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp slt i32 [[IV]], [[INVARIANT_SMIN]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp sgt i32 %inv_1, %iv
+  %cmp_2 = icmp sgt i32 %inv_2, %iv
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv
+}
+
+; turn to %iv <=s smin(inv_1, inv_2) and hoist it out of loop.
+define i32 @test_sle_swapped(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_sle_swapped(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_SMIN:%.*]] = call i32 @llvm.smin.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp sle i32 [[IV]], [[INVARIANT_SMIN]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp sge i32 %inv_1, %iv
+  %cmp_2 = icmp sge i32 %inv_2, %iv
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv
+}
+
+; turn to %iv >u umax(inv_1, inv_2) and hoist it out of loop.
+define i32 @test_ugt_swapped(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_ugt_swapped(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_UMAX:%.*]] = call i32 @llvm.umax.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp ugt i32 [[IV]], [[INVARIANT_UMAX]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp ult i32 %inv_1, %iv
+  %cmp_2 = icmp ult i32 %inv_2, %iv
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv
+}
+
+; turn to %iv >=u umax(inv_1, inv_2) and hoist it out of loop.
+define i32 @test_uge_swapped(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_uge_swapped(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_UMAX:%.*]] = call i32 @llvm.umax.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp uge i32 [[IV]], [[INVARIANT_UMAX]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp ule i32 %inv_1, %iv
+  %cmp_2 = icmp ule i32 %inv_2, %iv
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv
+}
+
+; turn to %iv >s smax(inv_1, inv_2) and hoist it out of loop.
+define i32 @test_sgt_swapped(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_sgt_swapped(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_SMAX:%.*]] = call i32 @llvm.smax.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp sgt i32 [[IV]], [[INVARIANT_SMAX]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp slt i32 %inv_1, %iv
+  %cmp_2 = icmp slt i32 %inv_2, %iv
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv
+}
+
+; turn to %iv >=s smax(inv_1, inv_2) and hoist it out of loop.
+define i32 @test_sge_swapped(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_sge_swapped(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_SMAX:%.*]] = call i32 @llvm.smax.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp sge i32 [[IV]], [[INVARIANT_SMAX]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp sle i32 %inv_1, %iv
+  %cmp_2 = icmp sle i32 %inv_2, %iv
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv
+}
+
+; inv_2 needs freeze because hoisted umax would create a new use that didn't exist before.
+; Counter-example: %cmp_1 = false, %inv_2 = poison.
+define i32 @test_logical_and_ult_needs_freeze(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_logical_and_ult_needs_freeze(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INV_2_FR:%.*]] = freeze i32 [[INV_2:%.*]]
+; CHECK-NEXT:    [[INVARIANT_UMIN:%.*]] = call i32 @llvm.umin.i32(i32 [[INV_1:%.*]], i32 [[INV_2_FR]])
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp ult i32 [[IV]], [[INVARIANT_UMIN]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp ult i32 %iv, %inv_1
+  %cmp_2 = icmp ult i32 %iv, %inv_2
+  %loop_cond = select i1 %cmp_1, i1 %cmp_2, i1 false
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv
+}
+
+; turn to %iv <u umin(inv_1, inv_2) and hoist it out of loop.
+; https://alive2.llvm.org/ce/z/mhKGtT
+define i32 @test_logical_and_ult_pos(i32 %start, i32 %inv_1, i32 noundef %inv_2) {
+; CHECK-LABEL: @test_logical_and_ult_pos(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INV_2_FR:%.*]] = freeze i32 [[INV_2:%.*]]
+; CHECK-NEXT:    [[INVARIANT_UMIN:%.*]] = call i32 @llvm.umin.i32(i32 [[INV_1:%.*]], i32 [[INV_2_FR]])
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp ult i32 [[IV]], [[INVARIANT_UMIN]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp ult i32 %iv, %inv_1
+  %cmp_2 = icmp ult i32 %iv, %inv_2
+  %loop_cond = select i1 %cmp_1, i1 %cmp_2, i1 false
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv
+}
+
+; inv_2 needs freeze because hoisted umax would create a new use that didn't exist before.
+define i32 @test_logical_or_ult_inv_needs_freeze(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_logical_or_ult_inv_needs_freeze(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INV_2_FR:%.*]] = freeze i32 [[INV_2:%.*]]
+; CHECK-NEXT:    [[INVARIANT_UMAX:%.*]] = call i32 @llvm.umax.i32(i32 [[INV_1:%.*]], i32 [[INV_2_FR]])
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp ult i32 [[IV]], [[INVARIANT_UMAX]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp ult i32 %iv, %inv_1
+  %cmp_2 = icmp ult i32 %iv, %inv_2
+  %loop_cond = select i1 %cmp_1, i1 true, i1 %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv
+}
+
+; turn to %iv <u umax(inv_1, inv_2) and hoist it out of loop.
+; https://alive2.llvm.org/ce/z/zgyG5N
+define i32 @test_logical_or_ult_inv_pos(i32 %start, i32 %inv_1, i32 noundef %inv_2) {
+; CHECK-LABEL: @test_logical_or_ult_inv_pos(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INV_2_FR:%.*]] = freeze i32 [[INV_2:%.*]]
+; CHECK-NEXT:    [[INVARIANT_UMAX:%.*]] = call i32 @llvm.umax.i32(i32 [[INV_1:%.*]], i32 [[INV_2_FR]])
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp ult i32 [[IV]], [[INVARIANT_UMAX]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp ult i32 %iv, %inv_1
+  %cmp_2 = icmp ult i32 %iv, %inv_2
+  %loop_cond = select i1 %cmp_1, i1 true, i1 %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv
+}
+
+; Mismatched predicates, do not optimize.
+define i32 @test_mismatched_predicates_neg(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_mismatched_predicates_neg(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ult i32 [[IV]], [[INV_1:%.*]]
+; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ule i32 [[IV]], [[INV_2:%.*]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp ult i32 %iv, %inv_1
+  %cmp_2 = icmp ule i32 %iv, %inv_2
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv
+}
+
+; TODO: Turn ule against constant into ult against constant, then do the same as in test_ult.
+; https://alive2.llvm.org/ce/z/rucunQ
+define i32 @test_mismatched_predicates_constant_pos_1(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_mismatched_predicates_constant_pos_1(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ult i32 [[IV]], [[INV_1:%.*]]
+; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ule i32 [[IV]], 100
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp ult i32 %iv, %inv_1
+  %cmp_2 = icmp ule i32 %iv, 100
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv
+}
+
+; TODO: Turn ult against a constant into ule, then do the same as in test_ule.
+; https://alive2.llvm.org/ce/z/-ht-pU
+define i32 @test_mismatched_predicates_constant_pos_2(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_mismatched_predicates_constant_pos_2(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ult i32 [[IV]], 100
+; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ule i32 [[IV]], [[INV_2:%.*]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp ult i32 %iv, 100
+  %cmp_2 = icmp ule i32 %iv, %inv_2
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv
+}
+
+; Do not optimize: `ule -1` cannot be turned into `ult 0` because of overflow.
+; However, this condition is trivially true anyways, so do not bother about it.
+define i32 @test_mismatched_predicates_constant_neg(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_mismatched_predicates_constant_neg(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ult i32 [[IV]], [[INV_1:%.*]]
+; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ule i32 [[IV]], -1
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp ult i32 %iv, %inv_1
+  %cmp_2 = icmp ule i32 %iv, -1
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv
+}
+
+; Do not optimize: predicate eq.
+define i32 @test_eq_neg(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_eq_neg(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[CMP_1:%.*]] = icmp eq i32 [[IV]], [[INV_1:%.*]]
+; CHECK-NEXT:    [[CMP_2:%.*]] = icmp eq i32 [[IV]], [[INV_2:%.*]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp eq i32 %iv, %inv_1
+  %cmp_2 = icmp eq i32 %iv, %inv_2
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv
+}
+
+; Do not optimize: predicate ne.
+define i32 @test_ne_neg(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_ne_neg(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ne i32 [[IV]], [[INV_1:%.*]]
+; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ne i32 [[IV]], [[INV_2:%.*]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp ne i32 %iv, %inv_1
+  %cmp_2 = icmp ne i32 %iv, %inv_2
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv
+}
+
+; Do not optimize: extra use.
+define i32 @test_ult_extra_use_1_neg(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_ult_extra_use_1_neg(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ult i32 [[IV]], [[INV_1:%.*]]
+; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ult i32 [[IV]], [[INV_2:%.*]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    [[CMP_1_LCSSA:%.*]] = phi i1 [ [[CMP_1]], [[LOOP]] ]
+; CHECK-NEXT:    call void @use(i1 [[CMP_1_LCSSA]])
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp ult i32 %iv, %inv_1
+  %cmp_2 = icmp ult i32 %iv, %inv_2
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  call void @use(i1 %cmp_1)
+  ret i32 %iv
+}
+
+define i32 @test_ult_extra_use_2_neg(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_ult_extra_use_2_neg(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ult i32 [[IV]], [[INV_1:%.*]]
+; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ult i32 [[IV]], [[INV_2:%.*]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    [[CMP_2_LCSSA:%.*]] = phi i1 [ [[CMP_2]], [[LOOP]] ]
+; CHECK-NEXT:    call void @use(i1 [[CMP_2_LCSSA]])
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp ult i32 %iv, %inv_1
+  %cmp_2 = icmp ult i32 %iv, %inv_2
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  call void @use(i1 %cmp_2)
+  ret i32 %iv
+}
+
+; This can be optimized, despite the fact that loop_cond has other uses.
+define i32 @test_ult_extra_use_result_pos(i32 %start, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_ult_extra_use_result_pos(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[INVARIANT_UMIN:%.*]] = call i32 @llvm.umin.i32(i32 [[INV_1:%.*]], i32 [[INV_2:%.*]])
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ [[START:%.*]], [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = icmp ult i32 [[IV]], [[INVARIANT_UMIN]]
+; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_LCSSA:%.*]] = phi i32 [ [[IV]], [[LOOP]] ]
+; CHECK-NEXT:    [[LOOP_COND_LCSSA:%.*]] = phi i1 [ [[LOOP_COND]], [[LOOP]] ]
+; CHECK-NEXT:    call void @use(i1 [[LOOP_COND_LCSSA]])
+; CHECK-NEXT:    ret i32 [[IV_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i32 [%start, %entry], [%iv.next, %loop]
+  %cmp_1 = icmp ult i32 %iv, %inv_1
+  %cmp_2 = icmp ult i32 %iv, %inv_2
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv.next = add i32 %iv, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  call void @use(i1 %loop_cond)
+  ret i32 %iv
+}
+
+; Do not optimize: comparison against different variant values.
+define i32 @test_ult_different_variants_neg(i32 %start_1, i32 %start_2, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_ult_different_variants_neg(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV_1:%.*]] = phi i32 [ [[START_1:%.*]], [[ENTRY:%.*]] ], [ [[IV_1_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[IV_2:%.*]] = phi i32 [ [[START_2:%.*]], [[ENTRY]] ], [ [[IV_2_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ult i32 [[IV_1]], [[INV_1:%.*]]
+; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ult i32 [[IV_2]], [[INV_2:%.*]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[IV_1_NEXT]] = add i32 [[IV_1]], 1
+; CHECK-NEXT:    [[IV_2_NEXT]] = add i32 [[IV_2]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_1_LCSSA:%.*]] = phi i32 [ [[IV_1]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_1_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv_1 = phi i32 [%start_1, %entry], [%iv_1.next, %loop]
+  %iv_2 = phi i32 [%start_2, %entry], [%iv_2.next, %loop]
+  %cmp_1 = icmp ult i32 %iv_1, %inv_1
+  %cmp_2 = icmp ult i32 %iv_2, %inv_2
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv_1.next = add i32 %iv_1, 1
+  %iv_2.next = add i32 %iv_2, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv_1
+}
+
+; Do not optimize: one of comparisons is against loop-variant.
+define i32 @test_ult_compare_against_variant_neg(i32 %start_1, i32 %start_2, i32 %inv_1, i32 %inv_2) {
+; CHECK-LABEL: @test_ult_compare_against_variant_neg(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV_1:%.*]] = phi i32 [ [[START_1:%.*]], [[ENTRY:%.*]] ], [ [[IV_1_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[IV_2:%.*]] = phi i32 [ [[START_2:%.*]], [[ENTRY]] ], [ [[IV_2_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ult i32 [[IV_1]], [[INV_1:%.*]]
+; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ult i32 [[IV_1]], [[IV_2]]
+; CHECK-NEXT:    [[LOOP_COND:%.*]] = and i1 [[CMP_1]], [[CMP_2]]
+; CHECK-NEXT:    [[IV_1_NEXT]] = add i32 [[IV_1]], 1
+; CHECK-NEXT:    [[IV_2_NEXT]] = add i32 [[IV_2]], 1
+; CHECK-NEXT:    br i1 [[LOOP_COND]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK:       exit:
+; CHECK-NEXT:    [[IV_1_LCSSA:%.*]] = phi i32 [ [[IV_1]], [[LOOP]] ]
+; CHECK-NEXT:    ret i32 [[IV_1_LCSSA]]
+;
+entry:
+  br label %loop
+
+loop:
+  %iv_1 = phi i32 [%start_1, %entry], [%iv_1.next, %loop]
+  %iv_2 = phi i32 [%start_2, %entry], [%iv_2.next, %loop]
+  %cmp_1 = icmp ult i32 %iv_1, %inv_1
+  %cmp_2 = icmp ult i32 %iv_1, %iv_2
+  %loop_cond = and i1 %cmp_1, %cmp_2
+  %iv_1.next = add i32 %iv_1, 1
+  %iv_2.next = add i32 %iv_2, 1
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret i32 %iv_1
+}
+
+declare void @use(i1)
diff --git a/llvm/test/Transforms/LICM/promote-single-thread.ll b/llvm/test/Transforms/LICM/promote-single-thread.ll
index 7e717615d543b..ff0798a877603 100644
--- a/llvm/test/Transforms/LICM/promote-single-thread.ll
+++ b/llvm/test/Transforms/LICM/promote-single-thread.ll
@@ -102,6 +102,75 @@ exit:
   ret void
 }
 
+; if %c is false and %ptr == @g, then this should store 42 to the pointer.
+; However, if we perform load+store promotion, then we would instead store the
+; original value of the global.
+; FIXME: This is a miscompile.
+define void @promote_global_noalias(i1 %c, i1 %c2, ptr noalias %ptr) {
+; MT-LABEL: @promote_global_noalias(
+; MT-NEXT:  entry:
+; MT-NEXT:    [[G_PROMOTED:%.*]] = load i32, ptr @g, align 4
+; MT-NEXT:    br label [[LOOP:%.*]]
+; MT:       loop:
+; MT-NEXT:    [[V_INC2:%.*]] = phi i32 [ [[V_INC1:%.*]], [[LATCH:%.*]] ], [ [[G_PROMOTED]], [[ENTRY:%.*]] ]
+; MT-NEXT:    br i1 [[C:%.*]], label [[IF:%.*]], label [[ELSE:%.*]]
+; MT:       if:
+; MT-NEXT:    [[V_INC:%.*]] = add i32 [[V_INC2]], 1
+; MT-NEXT:    store i32 [[V_INC]], ptr @g, align 4
+; MT-NEXT:    br label [[LATCH]]
+; MT:       else:
+; MT-NEXT:    store i32 42, ptr [[PTR:%.*]], align 4
+; MT-NEXT:    br label [[LATCH]]
+; MT:       latch:
+; MT-NEXT:    [[V_INC1]] = phi i32 [ [[V_INC2]], [[ELSE]] ], [ [[V_INC]], [[IF]] ]
+; MT-NEXT:    br i1 [[C2:%.*]], label [[EXIT:%.*]], label [[LOOP]]
+; MT:       exit:
+; MT-NEXT:    ret void
+;
+; ST-LABEL: @promote_global_noalias(
+; ST-NEXT:  entry:
+; ST-NEXT:    [[G_PROMOTED:%.*]] = load i32, ptr @g, align 4
+; ST-NEXT:    br label [[LOOP:%.*]]
+; ST:       loop:
+; ST-NEXT:    [[V_INC2:%.*]] = phi i32 [ [[V_INC1:%.*]], [[LATCH:%.*]] ], [ [[G_PROMOTED]], [[ENTRY:%.*]] ]
+; ST-NEXT:    br i1 [[C:%.*]], label [[IF:%.*]], label [[ELSE:%.*]]
+; ST:       if:
+; ST-NEXT:    [[V_INC:%.*]] = add i32 [[V_INC2]], 1
+; ST-NEXT:    br label [[LATCH]]
+; ST:       else:
+; ST-NEXT:    store i32 42, ptr [[PTR:%.*]], align 4
+; ST-NEXT:    br label [[LATCH]]
+; ST:       latch:
+; ST-NEXT:    [[V_INC1]] = phi i32 [ [[V_INC2]], [[ELSE]] ], [ [[V_INC]], [[IF]] ]
+; ST-NEXT:    br i1 [[C2:%.*]], label [[EXIT:%.*]], label [[LOOP]]
+; ST:       exit:
+; ST-NEXT:    [[V_INC1_LCSSA:%.*]] = phi i32 [ [[V_INC1]], [[LATCH]] ]
+; ST-NEXT:    store i32 [[V_INC1_LCSSA]], ptr @g, align 4
+; ST-NEXT:    ret void
+;
+entry:
+  br label %loop
+
+loop:
+  br i1 %c, label %if, label %else
+
+if:
+  %v = load i32, ptr @g
+  %v.inc = add i32 %v, 1
+  store i32 %v.inc, ptr @g
+  br label %latch
+
+else:
+  store i32 42, ptr %ptr
+  br label %latch
+
+latch:
+  br i1 %c2, label %exit, label %loop
+
+exit:
+  ret void
+}
+
 ; In single-thread mode both loads and stores can be promoted. In multi-thread
 ; mode only loads can be promoted, as a different thread might write to the
 ; captured alloca.
diff --git a/llvm/test/Transforms/LoopRotate/pr35210.ll b/llvm/test/Transforms/LoopRotate/pr35210.ll
index 836db38f2c47d..12df8f5a33e7e 100644
--- a/llvm/test/Transforms/LoopRotate/pr35210.ll
+++ b/llvm/test/Transforms/LoopRotate/pr35210.ll
@@ -1,5 +1,5 @@
-;RUN: opt %s -aa-pipeline= -passes='adce,loop(loop-rotate),adce' -S -verify-cfg-preserved=0 -debug-pass-manager -debug-only=loop-rotate 2>&1 | FileCheck %s
-;RUN: opt %s -aa-pipeline= -passes='adce,loop-mssa(loop-rotate),adce' -S -verify-cfg-preserved=0 -debug-pass-manager -debug-only=loop-rotate -verify-memoryssa 2>&1 | FileCheck %s --check-prefix=MSSA
+;RUN: opt %s -aa-pipeline= -passes='adce,loop(loop-rotate),adce' -S -verify-analysis-invalidation=0 -debug-pass-manager -debug-only=loop-rotate 2>&1 | FileCheck %s
+;RUN: opt %s -aa-pipeline= -passes='adce,loop-mssa(loop-rotate),adce' -S -verify-analysis-invalidation=0 -debug-pass-manager -debug-only=loop-rotate -verify-memoryssa 2>&1 | FileCheck %s --check-prefix=MSSA
 ;REQUIRES: asserts
 
 ; This test is to make sure we invalidate the post dominator pass after loop rotate simplifies the loop latch.
diff --git a/llvm/test/Transforms/LoopStrengthReduce/AArch64/vscale-factor-out-constant.ll b/llvm/test/Transforms/LoopStrengthReduce/AArch64/vscale-factor-out-constant.ll
index 97c3147c54c29..c70537f6e561e 100644
--- a/llvm/test/Transforms/LoopStrengthReduce/AArch64/vscale-factor-out-constant.ll
+++ b/llvm/test/Transforms/LoopStrengthReduce/AArch64/vscale-factor-out-constant.ll
@@ -8,7 +8,7 @@ target triple = "aarch64-unknown-linux-gnu"
 ; CHECK-LABEL: test
 ; FIXME: Handle VectorType in SCEVExpander::expandAddToGEP.
 ; The generated IR is not ideal with base 'scalar_vector' cast to i8*, and do ugly getelementptr over casted base.
-; CHECK: uglygep
+; CHECK: scevgep
 define void @test(ptr %a, i32 %v, i64 %n) {
 entry:
   %scalar_vector = alloca <vscale x 4 x i32>, align 16
diff --git a/llvm/test/Transforms/LoopStrengthReduce/AMDGPU/atomics.ll b/llvm/test/Transforms/LoopStrengthReduce/AMDGPU/atomics.ll
index b67235c0cc803..b23ef8f4c2185 100644
--- a/llvm/test/Transforms/LoopStrengthReduce/AMDGPU/atomics.ll
+++ b/llvm/test/Transforms/LoopStrengthReduce/AMDGPU/atomics.ll
@@ -7,16 +7,16 @@ target datalayout = "e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:3
 ; OPT-LABEL: @test_local_atomicrmw_addressing_loop_uniform_index_max_offset_i32(
 
 ; OPT: .lr.ph.preheader:
-; OPT: %uglygep2 = getelementptr i8, ptr addrspace(3) %arg1, i32 65532
+; OPT: %scevgep2 = getelementptr i8, ptr addrspace(3) %arg1, i32 65532
 ; OPT: br label %.lr.ph
 ; OPT: .lr.ph:
-; OPT: %lsr.iv3 = phi ptr addrspace(3) [ %uglygep4, %.lr.ph ], [ %uglygep2, %.lr.ph.preheader ]
-; OPT: %lsr.iv1 = phi ptr addrspace(3) [ %uglygep, %.lr.ph ], [ %arg0, %.lr.ph.preheader ]
+; OPT: %lsr.iv3 = phi ptr addrspace(3) [ %scevgep4, %.lr.ph ], [ %scevgep2, %.lr.ph.preheader ]
+; OPT: %lsr.iv1 = phi ptr addrspace(3) [ %scevgep, %.lr.ph ], [ %arg0, %.lr.ph.preheader ]
 ; OPT: %lsr.iv = phi i32 [ %lsr.iv.next, %.lr.ph ], [ %n, %.lr.ph.preheader ]
 ; OPT: %tmp4 = atomicrmw add ptr addrspace(3) %lsr.iv3, i32 undef seq_cst, align 4
 ; OPT: %tmp7 = atomicrmw add ptr addrspace(3) %lsr.iv1, i32 undef seq_cst, align 4
 ; OPT: %0 = atomicrmw add ptr addrspace(3) %lsr.iv1, i32 %tmp8 seq_cst, align 4
-; OPT: %uglygep4 = getelementptr i8, ptr addrspace(3) %lsr.iv3, i32 4
+; OPT: %scevgep4 = getelementptr i8, ptr addrspace(3) %lsr.iv3, i32 4
 ; OPT: br i1 %exitcond
 define amdgpu_kernel void @test_local_atomicrmw_addressing_loop_uniform_index_max_offset_i32(ptr addrspace(3) noalias nocapture %arg0, ptr addrspace(3) noalias nocapture readonly %arg1, i32 %n) #0 {
 bb:
@@ -49,14 +49,14 @@ bb:
 ; OPT-LABEL: test_local_cmpxchg_addressing_loop_uniform_index_max_offset_i32(
 
 ; OPT: .lr.ph.preheader:
-; OPT: %uglygep2 = getelementptr i8, ptr addrspace(3)  %arg1, i32 65532
+; OPT: %scevgep2 = getelementptr i8, ptr addrspace(3)  %arg1, i32 65532
 ; OPT: br label %.lr.ph
 ; OPT: .lr.ph:
-; OPT: %lsr.iv3 = phi ptr addrspace(3) [ %uglygep4, %.lr.ph ], [ %uglygep2, %.lr.ph.preheader ]
-; OPT: %lsr.iv1 = phi ptr addrspace(3) [ %uglygep, %.lr.ph ], [ %arg0, %.lr.ph.preheader ]
+; OPT: %lsr.iv3 = phi ptr addrspace(3) [ %scevgep4, %.lr.ph ], [ %scevgep2, %.lr.ph.preheader ]
+; OPT: %lsr.iv1 = phi ptr addrspace(3) [ %scevgep, %.lr.ph ], [ %arg0, %.lr.ph.preheader ]
 ; OPT: %lsr.iv = phi i32 [ %lsr.iv.next, %.lr.ph ], [ %n, %.lr.ph.preheader ]
 ; OPT: %tmp4 =  cmpxchg ptr addrspace(3) %lsr.iv3, i32 undef, i32 undef seq_cst monotonic, align 4
-; OPT: %uglygep4 = getelementptr i8, ptr addrspace(3) %lsr.iv3, i32 4
+; OPT: %scevgep4 = getelementptr i8, ptr addrspace(3) %lsr.iv3, i32 4
 define amdgpu_kernel void @test_local_cmpxchg_addressing_loop_uniform_index_max_offset_i32(ptr addrspace(3) noalias nocapture %arg0, ptr addrspace(3) noalias nocapture readonly %arg1, i32 %n) #0 {
 bb:
   %tmp = icmp sgt i32 %n, 0
@@ -90,15 +90,15 @@ bb:
 ; OPT-LABEL: @test_local_atomicinc_addressing_loop_uniform_index_max_offset_i32(
 
 ; OPT: .lr.ph.preheader:
-; OPT: %uglygep2 = getelementptr i8, ptr addrspace(3)  %arg1, i32 65532
+; OPT: %scevgep2 = getelementptr i8, ptr addrspace(3)  %arg1, i32 65532
 ; OPT: br label %.lr.ph
 ; OPT: .lr.ph:
-; OPT: %lsr.iv3 = phi ptr addrspace(3) [ %uglygep4, %.lr.ph ], [ %uglygep2, %.lr.ph.preheader ]
-; OPT: %lsr.iv1 = phi ptr addrspace(3) [ %uglygep, %.lr.ph ], [ %arg0, %.lr.ph.preheader ]
+; OPT: %lsr.iv3 = phi ptr addrspace(3) [ %scevgep4, %.lr.ph ], [ %scevgep2, %.lr.ph.preheader ]
+; OPT: %lsr.iv1 = phi ptr addrspace(3) [ %scevgep, %.lr.ph ], [ %arg0, %.lr.ph.preheader ]
 ; OPT: %lsr.iv = phi i32 [ %lsr.iv.next, %.lr.ph ], [ %n, %.lr.ph.preheader ]
 ; OPT: %tmp4 = call i32 @llvm.amdgcn.atomic.inc.i32.p3(ptr addrspace(3) %lsr.iv3, i32 undef, i32 0, i32 0, i1 false)
 ; OPT: %tmp7 = call i32 @llvm.amdgcn.atomic.inc.i32.p3(ptr addrspace(3) %lsr.iv1, i32 undef, i32 0, i32 0, i1 false)
-; OPT: %uglygep4 = getelementptr i8, ptr addrspace(3) %lsr.iv3, i32 4
+; OPT: %scevgep4 = getelementptr i8, ptr addrspace(3) %lsr.iv3, i32 4
 define amdgpu_kernel void @test_local_atomicinc_addressing_loop_uniform_index_max_offset_i32(ptr addrspace(3) noalias nocapture %arg0, ptr addrspace(3) noalias nocapture readonly %arg1, i32 %n) #0 {
 bb:
   %tmp = icmp sgt i32 %n, 0
@@ -129,15 +129,15 @@ bb:
 
 ; OPT-LABEL: @test_local_atomicdec_addressing_loop_uniform_index_max_offset_i32(
 ; OPT: .lr.ph.preheader:
-; OPT: %uglygep2 = getelementptr i8, ptr addrspace(3) %arg1, i32 65532
+; OPT: %scevgep2 = getelementptr i8, ptr addrspace(3) %arg1, i32 65532
 ; OPT: br label %.lr.ph
 ; OPT: .lr.ph:
-; OPT: %lsr.iv3 = phi ptr addrspace(3) [ %uglygep4, %.lr.ph ], [ %uglygep2, %.lr.ph.preheader ]
-; OPT: %lsr.iv1 = phi ptr addrspace(3) [ %uglygep, %.lr.ph ], [ %arg0, %.lr.ph.preheader ]
+; OPT: %lsr.iv3 = phi ptr addrspace(3) [ %scevgep4, %.lr.ph ], [ %scevgep2, %.lr.ph.preheader ]
+; OPT: %lsr.iv1 = phi ptr addrspace(3) [ %scevgep, %.lr.ph ], [ %arg0, %.lr.ph.preheader ]
 ; OPT: %lsr.iv = phi i32 [ %lsr.iv.next, %.lr.ph ], [ %n, %.lr.ph.preheader ]
 ; OPT: %tmp4 = call i32 @llvm.amdgcn.atomic.dec.i32.p3(ptr addrspace(3) %lsr.iv3, i32 undef, i32 0, i32 0, i1 false)
 ; OPT: %tmp7 = call i32 @llvm.amdgcn.atomic.dec.i32.p3(ptr addrspace(3) %lsr.iv1, i32 undef, i32 0, i32 0, i1 false)
-; OPT: %uglygep4 = getelementptr i8, ptr addrspace(3) %lsr.iv3, i32 4
+; OPT: %scevgep4 = getelementptr i8, ptr addrspace(3) %lsr.iv3, i32 4
 define amdgpu_kernel void @test_local_atomicdec_addressing_loop_uniform_index_max_offset_i32(ptr addrspace(3) noalias nocapture %arg0, ptr addrspace(3) noalias nocapture readonly %arg1, i32 %n) #0 {
 bb:
   %tmp = icmp sgt i32 %n, 0
diff --git a/llvm/test/Transforms/LoopStrengthReduce/AMDGPU/different-addrspace-addressing-mode-loops.ll b/llvm/test/Transforms/LoopStrengthReduce/AMDGPU/different-addrspace-addressing-mode-loops.ll
index 0a757419b59b5..40306e7998927 100644
--- a/llvm/test/Transforms/LoopStrengthReduce/AMDGPU/different-addrspace-addressing-mode-loops.ll
+++ b/llvm/test/Transforms/LoopStrengthReduce/AMDGPU/different-addrspace-addressing-mode-loops.ll
@@ -7,12 +7,12 @@ target datalayout = "e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:3
 
 ; OPT-LABEL: @test_global_addressing_loop_uniform_index_max_offset_i32(
 ; OPT: .lr.ph.preheader:
-; OPT: %uglygep2 = getelementptr i8, ptr addrspace(1) %arg1, i64 4095
+; OPT: %scevgep2 = getelementptr i8, ptr addrspace(1) %arg1, i64 4095
 ; OPT: br label %.lr.ph
 ; OPT: {{^}}.lr.ph:
-; OPT: %lsr.iv3 = phi ptr addrspace(1) [ %uglygep4, %.lr.ph ], [ %uglygep2, %.lr.ph.preheader ]
+; OPT: %lsr.iv3 = phi ptr addrspace(1) [ %scevgep4, %.lr.ph ], [ %scevgep2, %.lr.ph.preheader ]
 ; OPT: load i8, ptr addrspace(1) %lsr.iv3, align 1
-; OPT: %uglygep4 = getelementptr i8, ptr addrspace(1) %lsr.iv3, i64 1
+; OPT: %scevgep4 = getelementptr i8, ptr addrspace(1) %lsr.iv3, i64 1
 define amdgpu_kernel void @test_global_addressing_loop_uniform_index_max_offset_i32(ptr addrspace(1) noalias nocapture %arg0, ptr addrspace(1) noalias nocapture readonly %arg1, i32 %n) #0 {
 bb:
   %tmp = icmp sgt i32 %n, 0
@@ -45,12 +45,12 @@ bb:
 
 ; OPT-LABEL: @test_global_addressing_loop_uniform_index_max_offset_p1_i32(
 ; OPT: {{^}}.lr.ph.preheader:
-; OPT: %uglygep2 = getelementptr i8, ptr addrspace(1) %arg1, i64 4096
+; OPT: %scevgep2 = getelementptr i8, ptr addrspace(1) %arg1, i64 4096
 ; OPT: br label %.lr.ph
 
 ; OPT: {{^}}.lr.ph:
-; OPT: %lsr.iv3 = phi ptr addrspace(1) [ %uglygep4, %.lr.ph ], [ %uglygep2, %.lr.ph.preheader ]
-; OPT: %uglygep4 = getelementptr i8, ptr addrspace(1) %lsr.iv3, i64 1
+; OPT: %lsr.iv3 = phi ptr addrspace(1) [ %scevgep4, %.lr.ph ], [ %scevgep2, %.lr.ph.preheader ]
+; OPT: %scevgep4 = getelementptr i8, ptr addrspace(1) %lsr.iv3, i64 1
 define amdgpu_kernel void @test_global_addressing_loop_uniform_index_max_offset_p1_i32(ptr addrspace(1) noalias nocapture %arg0, ptr addrspace(1) noalias nocapture readonly %arg1, i32 %n) #0 {
 bb:
   %tmp = icmp sgt i32 %n, 0
@@ -83,12 +83,12 @@ bb:
 
 ; OPT-LABEL: @test_local_addressing_loop_uniform_index_max_offset_i32(
 ; OPT: .lr.ph.preheader:
-; OPT: %uglygep2 = getelementptr i8, ptr addrspace(3) %arg1, i32 65535
+; OPT: %scevgep2 = getelementptr i8, ptr addrspace(3) %arg1, i32 65535
 ; OPT: br label %.lr.ph
 ; OPT: {{^}}.lr.ph
-; OPT: %lsr.iv3 = phi ptr addrspace(3) [ %uglygep4, %.lr.ph ], [ %uglygep2, %.lr.ph.preheader ]
+; OPT: %lsr.iv3 = phi ptr addrspace(3) [ %scevgep4, %.lr.ph ], [ %scevgep2, %.lr.ph.preheader ]
 ; OPT: %tmp4 = load i8, ptr addrspace(3) %lsr.iv3, align 1
-; OPT: %uglygep4 = getelementptr i8, ptr addrspace(3) %lsr.iv3, i32 1
+; OPT: %scevgep4 = getelementptr i8, ptr addrspace(3) %lsr.iv3, i32 1
 define amdgpu_kernel void @test_local_addressing_loop_uniform_index_max_offset_i32(ptr addrspace(1) noalias nocapture %arg0, ptr addrspace(3) noalias nocapture readonly %arg1, i32 %n) #0 {
 bb:
   %tmp = icmp sgt i32 %n, 0
@@ -122,12 +122,12 @@ bb:
 
 ; OPT-LABEL: @test_local_addressing_loop_uniform_index_max_offset_p1_i32(
 ; OPT: {{^}}.lr.ph.preheader:
-; OPT: %uglygep2 = getelementptr i8, ptr addrspace(3) %arg1, i32 65536
+; OPT: %scevgep2 = getelementptr i8, ptr addrspace(3) %arg1, i32 65536
 ; OPT: br label %.lr.ph
 
 ; OPT: {{^}}.lr.ph:
-; OPT: %lsr.iv3 = phi ptr addrspace(3) [ %uglygep4, %.lr.ph ], [ %uglygep2, %.lr.ph.preheader ]
-; OPT: %uglygep4 = getelementptr i8, ptr addrspace(3) %lsr.iv3, i32 1
+; OPT: %lsr.iv3 = phi ptr addrspace(3) [ %scevgep4, %.lr.ph ], [ %scevgep2, %.lr.ph.preheader ]
+; OPT: %scevgep4 = getelementptr i8, ptr addrspace(3) %lsr.iv3, i32 1
 define amdgpu_kernel void @test_local_addressing_loop_uniform_index_max_offset_p1_i32(ptr addrspace(1) noalias nocapture %arg0, ptr addrspace(3) noalias nocapture readonly %arg1, i32 %n) #0 {
 bb:
   %tmp = icmp sgt i32 %n, 0
diff --git a/llvm/test/Transforms/LoopStrengthReduce/AMDGPU/preserve-addrspace-assert.ll b/llvm/test/Transforms/LoopStrengthReduce/AMDGPU/preserve-addrspace-assert.ll
index 2006ac0e6de6d..99432cc86691e 100644
--- a/llvm/test/Transforms/LoopStrengthReduce/AMDGPU/preserve-addrspace-assert.ll
+++ b/llvm/test/Transforms/LoopStrengthReduce/AMDGPU/preserve-addrspace-assert.ll
@@ -8,11 +8,11 @@ target datalayout = "e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:3
 %0 = type { i32, double, i32, float }
 
 ; CHECK-LABEL: @lsr_crash_preserve_addrspace_unknown_type(
-; CHECK: %uglygep1 = getelementptr i8, ptr addrspace(3) %tmp, i32 8
-; CHECK: load double, ptr addrspace(3) %uglygep1
+; CHECK: %scevgep1 = getelementptr i8, ptr addrspace(3) %tmp, i32 8
+; CHECK: load double, ptr addrspace(3) %scevgep1
 
-; CHECK: %uglygep = getelementptr i8, ptr addrspace(3) %tmp, i32 16
-; CHECK: %tmp14 = load i32, ptr addrspace(3) %uglygep
+; CHECK: %scevgep = getelementptr i8, ptr addrspace(3) %tmp, i32 16
+; CHECK: %tmp14 = load i32, ptr addrspace(3) %scevgep
 define amdgpu_kernel void @lsr_crash_preserve_addrspace_unknown_type() #0 {
 bb:
   br label %bb1
diff --git a/llvm/test/Transforms/LoopStrengthReduce/RISCV/lsr-drop-solution.ll b/llvm/test/Transforms/LoopStrengthReduce/RISCV/lsr-drop-solution.ll
index f0d22c9a33b01..1ab19081b7de5 100644
--- a/llvm/test/Transforms/LoopStrengthReduce/RISCV/lsr-drop-solution.ll
+++ b/llvm/test/Transforms/LoopStrengthReduce/RISCV/lsr-drop-solution.ll
@@ -10,10 +10,9 @@ define ptr @foo(ptr %a0, ptr %a1, i64 %a2) {
 ; CHECK-NEXT:    vsetvli a4, a2, e8, m8, ta, ma
 ; CHECK-NEXT:    bne a4, a2, .LBB0_2
 ; CHECK-NEXT:  # %bb.1:
-; CHECK-NEXT:    mv a3, a0
 ; CHECK-NEXT:    vsetvli zero, a2, e8, m8, ta, ma
 ; CHECK-NEXT:    vle8.v v8, (a1)
-; CHECK-NEXT:    vse8.v v8, (a3)
+; CHECK-NEXT:    vse8.v v8, (a0)
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB0_2: # %if.then
 ; CHECK-NEXT:    add a2, a0, a2
diff --git a/llvm/test/Transforms/LoopStrengthReduce/lsr-rewrite-to-add-one.ll b/llvm/test/Transforms/LoopStrengthReduce/lsr-rewrite-to-add-one.ll
index ecbd34732b2b2..74d861cf5168c 100644
--- a/llvm/test/Transforms/LoopStrengthReduce/lsr-rewrite-to-add-one.ll
+++ b/llvm/test/Transforms/LoopStrengthReduce/lsr-rewrite-to-add-one.ll
@@ -8,7 +8,7 @@ define i32 @test(i1 %c.1, ptr %src) {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br label [[LOOP_HEADER:%.*]]
 ; CHECK:       loop.header:
-; CHECK-NEXT:    [[IV:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP_LATCH:%.*]] ]
+; CHECK-NEXT:    [[LSR_IV:%.*]] = phi i32 [ [[LSR_IV_NEXT:%.*]], [[LOOP_LATCH:%.*]] ], [ 0, [[ENTRY:%.*]] ]
 ; CHECK-NEXT:    br i1 [[C_1:%.*]], label [[LOOP_LATCH]], label [[LOOP_THEN:%.*]]
 ; CHECK:       loop.then:
 ; CHECK-NEXT:    [[L:%.*]] = load i32, ptr [[SRC:%.*]], align 4
@@ -16,11 +16,12 @@ define i32 @test(i1 %c.1, ptr %src) {
 ; CHECK-NEXT:    br label [[LOOP_LATCH]]
 ; CHECK:       loop.latch:
 ; CHECK-NEXT:    [[P:%.*]] = phi i1 [ [[C_2]], [[LOOP_THEN]] ], [ false, [[LOOP_HEADER]] ]
-; CHECK-NEXT:    [[T:%.*]] = icmp sgt i32 [[IV]], -1050
+; CHECK-NEXT:    [[T:%.*]] = icmp sgt i32 [[LSR_IV]], -1050
 ; CHECK-NEXT:    [[OR:%.*]] = or i1 [[P]], [[T]]
 ; CHECK-NEXT:    [[ZEXT_OR:%.*]] = zext i1 [[OR]] to i32
-; CHECK-NEXT:    [[IV_NEXT]] = add i32 [[IV]], [[ZEXT_OR]]
-; CHECK-NEXT:    [[LOOP_HEADER_TERMCOND:%.*]] = icmp sgt i32 [[IV]], -1050
+; CHECK-NEXT:    [[LSR_IV_NEXT]] = add nuw i32 [[LSR_IV]], 1
+; CHECK-NEXT:    [[TMP0:%.*]] = add i32 [[LSR_IV_NEXT]], -1
+; CHECK-NEXT:    [[LOOP_HEADER_TERMCOND:%.*]] = icmp sgt i32 [[TMP0]], -1050
 ; CHECK-NEXT:    br i1 [[LOOP_HEADER_TERMCOND]], label [[LOOP_HEADER]], label [[EXIT:%.*]]
 ; CHECK:       exit:
 ; CHECK-NEXT:    [[ZEXT_OR_LCSSA:%.*]] = phi i32 [ [[ZEXT_OR]], [[LOOP_LATCH]] ]
diff --git a/llvm/test/Transforms/LoopStrengthReduce/lsr-term-fold-negative-testcase.ll b/llvm/test/Transforms/LoopStrengthReduce/lsr-term-fold-negative-testcase.ll
index 2856da967ffa2..895ce1c6a6965 100644
--- a/llvm/test/Transforms/LoopStrengthReduce/lsr-term-fold-negative-testcase.ll
+++ b/llvm/test/Transforms/LoopStrengthReduce/lsr-term-fold-negative-testcase.ll
@@ -1,10 +1,23 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
 ; REQUIRES: asserts
 ; RUN: opt < %s -passes="loop-reduce" -S -debug -lsr-term-fold 2>&1 | FileCheck %s
 
-target datalayout = "e-p:32:32:32-n32"
+target datalayout = "e-p:64:64:64-n64"
 
 define i32 @loop_variant(ptr %ar, i32 %n, i32 %m) {
-; CHECK: Cannot fold on backedge that is loop variant
+; CHECK-LABEL: define i32 @loop_variant
+; CHECK-SAME: (ptr [[AR:%.*]], i32 [[N:%.*]], i32 [[M:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[FOR_COND:%.*]]
+; CHECK:       for.cond:
+; CHECK-NEXT:    [[N_ADDR_0:%.*]] = phi i32 [ [[N]], [[ENTRY:%.*]] ], [ [[MUL:%.*]], [[FOR_COND]] ]
+; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 [[N_ADDR_0]], [[M]]
+; CHECK-NEXT:    [[MUL]] = shl nsw i32 [[N_ADDR_0]], 1
+; CHECK-NEXT:    br i1 [[CMP]], label [[FOR_COND]], label [[FOR_END:%.*]]
+; CHECK:       for.end:
+; CHECK-NEXT:    [[N_ADDR_0_LCSSA:%.*]] = phi i32 [ [[N_ADDR_0]], [[FOR_COND]] ]
+; CHECK-NEXT:    ret i32 [[N_ADDR_0_LCSSA]]
+;
 entry:
   br label %for.cond
 
@@ -19,8 +32,43 @@ for.end:                                          ; preds = %for.cond
 }
 
 define i32 @nested_loop(ptr %ar, i32 %n, i32 %m, i32 %o) {
-; CHECK: Cannot fold on backedge that is loop variant
-; CHECK: Cannot fold on non-innermost loop
+; CHECK-LABEL: define i32 @nested_loop
+; CHECK-SAME: (ptr [[AR:%.*]], i32 [[N:%.*]], i32 [[M:%.*]], i32 [[O:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[CMP15:%.*]] = icmp sgt i32 [[O]], 0
+; CHECK-NEXT:    br i1 [[CMP15]], label [[FOR_BODY_PREHEADER:%.*]], label [[FOR_COND_CLEANUP:%.*]]
+; CHECK:       for.body.preheader:
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.cond.cleanup.loopexit:
+; CHECK-NEXT:    [[CNT_1_LCSSA_LCSSA:%.*]] = phi i32 [ [[CNT_1_LCSSA:%.*]], [[FOR_COND_CLEANUP3:%.*]] ]
+; CHECK-NEXT:    br label [[FOR_COND_CLEANUP]]
+; CHECK:       for.cond.cleanup:
+; CHECK-NEXT:    [[CNT_0_LCSSA:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[CNT_1_LCSSA_LCSSA]], [[FOR_COND_CLEANUP_LOOPEXIT:%.*]] ]
+; CHECK-NEXT:    ret i32 [[CNT_0_LCSSA]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[I_017:%.*]] = phi i32 [ [[INC6:%.*]], [[FOR_COND_CLEANUP3]] ], [ 0, [[FOR_BODY_PREHEADER]] ]
+; CHECK-NEXT:    [[CNT_016:%.*]] = phi i32 [ [[CNT_1_LCSSA]], [[FOR_COND_CLEANUP3]] ], [ 0, [[FOR_BODY_PREHEADER]] ]
+; CHECK-NEXT:    [[SUB:%.*]] = sub nsw i32 [[N]], [[I_017]]
+; CHECK-NEXT:    [[CMP212:%.*]] = icmp slt i32 [[SUB]], [[M]]
+; CHECK-NEXT:    br i1 [[CMP212]], label [[FOR_BODY4_PREHEADER:%.*]], label [[FOR_COND_CLEANUP3]]
+; CHECK:       for.body4.preheader:
+; CHECK-NEXT:    br label [[FOR_BODY4:%.*]]
+; CHECK:       for.cond.cleanup3.loopexit:
+; CHECK-NEXT:    [[INC_LCSSA:%.*]] = phi i32 [ [[INC:%.*]], [[FOR_BODY4]] ]
+; CHECK-NEXT:    br label [[FOR_COND_CLEANUP3]]
+; CHECK:       for.cond.cleanup3:
+; CHECK-NEXT:    [[CNT_1_LCSSA]] = phi i32 [ [[CNT_016]], [[FOR_BODY]] ], [ [[INC_LCSSA]], [[FOR_COND_CLEANUP3_LOOPEXIT:%.*]] ]
+; CHECK-NEXT:    [[INC6]] = add nuw nsw i32 [[I_017]], 1
+; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 [[INC6]], [[O]]
+; CHECK-NEXT:    br i1 [[CMP]], label [[FOR_BODY]], label [[FOR_COND_CLEANUP_LOOPEXIT]]
+; CHECK:       for.body4:
+; CHECK-NEXT:    [[J_014:%.*]] = phi i32 [ [[MUL:%.*]], [[FOR_BODY4]] ], [ [[SUB]], [[FOR_BODY4_PREHEADER]] ]
+; CHECK-NEXT:    [[CNT_113:%.*]] = phi i32 [ [[INC]], [[FOR_BODY4]] ], [ [[CNT_016]], [[FOR_BODY4_PREHEADER]] ]
+; CHECK-NEXT:    [[INC]] = add nsw i32 [[CNT_113]], 1
+; CHECK-NEXT:    [[MUL]] = shl nsw i32 [[J_014]], 1
+; CHECK-NEXT:    [[CMP2:%.*]] = icmp slt i32 [[MUL]], [[M]]
+; CHECK-NEXT:    br i1 [[CMP2]], label [[FOR_BODY4]], label [[FOR_COND_CLEANUP3_LOOPEXIT]]
+;
 entry:
   %cmp15 = icmp sgt i32 %o, 0
   br i1 %cmp15, label %for.body, label %for.cond.cleanup
@@ -54,8 +102,22 @@ for.body4:                                        ; preds = %for.body, %for.body
 declare void @foo(ptr)
 
 define void @NonAddRecIV(ptr %a) {
-; CHECK: SCEV of phi '  %lsr.iv = phi i32 [ %lsr.iv.next, %for.body ], [ 1, %entry ]'
-; CHECK: is not an affine add recursion, not qualified for the terminating condition folding.
+; CHECK-LABEL: define void @NonAddRecIV
+; CHECK-SAME: (ptr [[A:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[UGLYGEP:%.*]] = getelementptr i8, ptr [[A]], i32 84
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[LSR_IV1:%.*]] = phi ptr [ [[UGLYGEP2:%.*]], [[FOR_BODY]] ], [ [[UGLYGEP]], [[ENTRY:%.*]] ]
+; CHECK-NEXT:    [[LSR_IV:%.*]] = phi i32 [ [[LSR_IV_NEXT:%.*]], [[FOR_BODY]] ], [ 1, [[ENTRY]] ]
+; CHECK-NEXT:    store i32 1, ptr [[LSR_IV1]], align 4
+; CHECK-NEXT:    [[LSR_IV_NEXT]] = mul nsw i32 [[LSR_IV]], 2
+; CHECK-NEXT:    [[UGLYGEP2]] = getelementptr i8, ptr [[LSR_IV1]], i64 4
+; CHECK-NEXT:    [[EXITCOND_NOT:%.*]] = icmp eq i32 [[LSR_IV_NEXT]], 65536
+; CHECK-NEXT:    br i1 [[EXITCOND_NOT]], label [[FOR_END:%.*]], label [[FOR_BODY]]
+; CHECK:       for.end:
+; CHECK-NEXT:    ret void
+;
 entry:
   %uglygep = getelementptr i8, ptr %a, i32 84
   br label %for.body
@@ -76,8 +138,25 @@ for.end:                                          ; preds = %for.body
 @fp_inc = common global float 0.000000e+00, align 4
 
 define void @NonSCEVableIV(float %init, float* %A, i32 %N) {
-; CHECK: IV of phi '  %x.05 = phi float [ %init, %entry ], [ %add, %for.body ]'
-; CHECK: is not SCEV-able, not qualified for the terminating condition folding.
+; CHECK-LABEL: define void @NonSCEVableIV
+; CHECK-SAME: (float [[INIT:%.*]], ptr [[A:%.*]], i32 [[N:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load float, ptr @fp_inc, align 4
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[LSR_IV1:%.*]] = phi ptr [ [[SCEVGEP:%.*]], [[FOR_BODY]] ], [ [[A]], [[ENTRY:%.*]] ]
+; CHECK-NEXT:    [[LSR_IV:%.*]] = phi i64 [ [[LSR_IV_NEXT:%.*]], [[FOR_BODY]] ], [ 1, [[ENTRY]] ]
+; CHECK-NEXT:    [[X_05:%.*]] = phi float [ [[INIT]], [[ENTRY]] ], [ [[ADD:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    store float [[X_05]], ptr [[LSR_IV1]], align 4
+; CHECK-NEXT:    [[ADD]] = fsub float [[X_05]], [[TMP0]]
+; CHECK-NEXT:    [[LFTR_WIDEIV:%.*]] = trunc i64 [[LSR_IV]] to i32
+; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i32 [[LFTR_WIDEIV]], [[N]]
+; CHECK-NEXT:    [[LSR_IV_NEXT]] = add nuw nsw i64 [[LSR_IV]], 1
+; CHECK-NEXT:    [[SCEVGEP]] = getelementptr i8, ptr [[LSR_IV1]], i64 4
+; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_END:%.*]], label [[FOR_BODY]]
+; CHECK:       for.end:
+; CHECK-NEXT:    ret void
+;
 entry:
   %0 = load float, float* @fp_inc, align 4
   br label %for.body
@@ -98,7 +177,22 @@ for.end:                                          ; preds = %for.end
 }
 
 define void @NonIcmpEqNe(ptr %a) {
-; CHECK: Cannot fold on branching condition that is not an ICmpInst::eq / ICmpInst::ne
+; CHECK-LABEL: define void @NonIcmpEqNe
+; CHECK-SAME: (ptr [[A:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[UGLYGEP:%.*]] = getelementptr i8, ptr [[A]], i64 84
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[LSR_IV1:%.*]] = phi ptr [ [[UGLYGEP2:%.*]], [[FOR_BODY]] ], [ [[UGLYGEP]], [[ENTRY:%.*]] ]
+; CHECK-NEXT:    [[LSR_IV:%.*]] = phi i64 [ [[LSR_IV_NEXT:%.*]], [[FOR_BODY]] ], [ 379, [[ENTRY]] ]
+; CHECK-NEXT:    store i32 1, ptr [[LSR_IV1]], align 4
+; CHECK-NEXT:    [[LSR_IV_NEXT]] = add nsw i64 [[LSR_IV]], -1
+; CHECK-NEXT:    [[UGLYGEP2]] = getelementptr i8, ptr [[LSR_IV1]], i64 4
+; CHECK-NEXT:    [[EXITCOND_NOT:%.*]] = icmp sle i64 [[LSR_IV_NEXT]], 0
+; CHECK-NEXT:    br i1 [[EXITCOND_NOT]], label [[FOR_END:%.*]], label [[FOR_BODY]]
+; CHECK:       for.end:
+; CHECK-NEXT:    ret void
+;
 entry:
   %uglygep = getelementptr i8, ptr %a, i64 84
   br label %for.body
@@ -117,7 +211,25 @@ for.end:                                          ; preds = %for.body
 }
 
 define void @TermCondMoreThanOneUse(ptr %a) {
-;CHECK: Cannot replace terminating condition with more than one use
+; CHECK-LABEL: define void @TermCondMoreThanOneUse
+; CHECK-SAME: (ptr [[A:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[UGLYGEP:%.*]] = getelementptr i8, ptr [[A]], i64 84
+; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i8, ptr [[A]], i64 1600
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[LSR_IV2:%.*]] = phi i64 [ [[LSR_IV_NEXT3:%.*]], [[FOR_BODY]] ], [ -378, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    [[LSR_IV1:%.*]] = phi ptr [ [[UGLYGEP2:%.*]], [[FOR_BODY]] ], [ [[UGLYGEP]], [[ENTRY]] ]
+; CHECK-NEXT:    store i32 1, ptr [[LSR_IV1]], align 4
+; CHECK-NEXT:    [[UGLYGEP2]] = getelementptr i8, ptr [[LSR_IV1]], i64 4
+; CHECK-NEXT:    [[EXITCOND_NOT:%.*]] = icmp eq i64 [[LSR_IV2]], 0
+; CHECK-NEXT:    [[DUMMY:%.*]] = select i1 [[EXITCOND_NOT]], i8 0, i8 1
+; CHECK-NEXT:    [[LSR_IV_NEXT3]] = add nsw i64 [[LSR_IV2]], 1
+; CHECK-NEXT:    [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND:%.*]] = icmp eq ptr [[UGLYGEP2]], [[SCEVGEP]]
+; CHECK-NEXT:    br i1 [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND]], label [[FOR_END:%.*]], label [[FOR_BODY]]
+; CHECK:       for.end:
+; CHECK-NEXT:    ret void
+;
 entry:
   %uglygep = getelementptr i8, ptr %a, i64 84
   br label %for.body
@@ -143,7 +255,59 @@ for.end:                                          ; preds = %for.body
 @histogram_energy_boundaries = global [1001 x double] zeroinitializer, align 8
 
 define void @ebur128_calc_gating_block(ptr %st, ptr %optional_output) {
-; CHECK: Is not safe to expand terminating value for phi node  %i.026 = phi i64 [ 0, %for.body7.lr.ph ], [ %inc, %for.body7 ]
+; CHECK-LABEL: define void @ebur128_calc_gating_block
+; CHECK-SAME: (ptr [[ST:%.*]], ptr [[OPTIONAL_OUTPUT:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[ST]], align 8
+; CHECK-NEXT:    [[CONV:%.*]] = zext i32 [[TMP0]] to i64
+; CHECK-NEXT:    [[CMP28_NOT:%.*]] = icmp eq i32 [[TMP0]], 0
+; CHECK-NEXT:    br i1 [[CMP28_NOT]], label [[FOR_END13:%.*]], label [[FOR_COND2_PREHEADER_LR_PH:%.*]]
+; CHECK:       for.cond2.preheader.lr.ph:
+; CHECK-NEXT:    [[AUDIO_DATA_INDEX:%.*]] = getelementptr inbounds [[STRUCT_FFEBUR128STATE:%.*]], ptr [[ST]], i64 0, i32 3
+; CHECK-NEXT:    [[TMP1:%.*]] = load i64, ptr [[AUDIO_DATA_INDEX]], align 8
+; CHECK-NEXT:    [[DIV:%.*]] = udiv i64 [[TMP1]], [[CONV]]
+; CHECK-NEXT:    [[CMP525_NOT:%.*]] = icmp ult i64 [[TMP1]], [[CONV]]
+; CHECK-NEXT:    [[AUDIO_DATA:%.*]] = getelementptr inbounds [[STRUCT_FFEBUR128STATE]], ptr [[ST]], i64 0, i32 1
+; CHECK-NEXT:    [[UMAX:%.*]] = tail call i64 @llvm.umax.i64(i64 [[DIV]], i64 1)
+; CHECK-NEXT:    [[TMP2:%.*]] = shl nuw nsw i64 [[CONV]], 3
+; CHECK-NEXT:    br label [[FOR_COND2_PREHEADER:%.*]]
+; CHECK:       for.cond2.preheader:
+; CHECK-NEXT:    [[LSR_IV1:%.*]] = phi i64 [ [[LSR_IV_NEXT2:%.*]], [[FOR_INC11:%.*]] ], [ 0, [[FOR_COND2_PREHEADER_LR_PH]] ]
+; CHECK-NEXT:    [[CHANNEL_SUM_030:%.*]] = phi double [ 0.000000e+00, [[FOR_COND2_PREHEADER_LR_PH]] ], [ [[CHANNEL_SUM_1_LCSSA:%.*]], [[FOR_INC11]] ]
+; CHECK-NEXT:    [[C_029:%.*]] = phi i64 [ 0, [[FOR_COND2_PREHEADER_LR_PH]] ], [ [[INC12:%.*]], [[FOR_INC11]] ]
+; CHECK-NEXT:    br i1 [[CMP525_NOT]], label [[FOR_INC11]], label [[FOR_BODY7_LR_PH:%.*]]
+; CHECK:       for.body7.lr.ph:
+; CHECK-NEXT:    [[TMP3:%.*]] = load ptr, ptr [[AUDIO_DATA]], align 8
+; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i8, ptr [[TMP3]], i64 [[LSR_IV1]]
+; CHECK-NEXT:    br label [[FOR_BODY7:%.*]]
+; CHECK:       for.body7:
+; CHECK-NEXT:    [[LSR_IV3:%.*]] = phi ptr [ [[SCEVGEP4:%.*]], [[FOR_BODY7]] ], [ [[SCEVGEP]], [[FOR_BODY7_LR_PH]] ]
+; CHECK-NEXT:    [[LSR_IV:%.*]] = phi i64 [ [[LSR_IV_NEXT:%.*]], [[FOR_BODY7]] ], [ [[UMAX]], [[FOR_BODY7_LR_PH]] ]
+; CHECK-NEXT:    [[CHANNEL_SUM_127:%.*]] = phi double [ [[CHANNEL_SUM_030]], [[FOR_BODY7_LR_PH]] ], [ [[ADD10:%.*]], [[FOR_BODY7]] ]
+; CHECK-NEXT:    [[TMP4:%.*]] = load double, ptr [[LSR_IV3]], align 8
+; CHECK-NEXT:    [[ADD10]] = fadd double [[CHANNEL_SUM_127]], [[TMP4]]
+; CHECK-NEXT:    [[LSR_IV_NEXT]] = add i64 [[LSR_IV]], -1
+; CHECK-NEXT:    [[SCEVGEP4]] = getelementptr i8, ptr [[LSR_IV3]], i64 [[TMP2]]
+; CHECK-NEXT:    [[EXITCOND_NOT:%.*]] = icmp eq i64 [[LSR_IV_NEXT]], 0
+; CHECK-NEXT:    br i1 [[EXITCOND_NOT]], label [[FOR_INC11_LOOPEXIT:%.*]], label [[FOR_BODY7]]
+; CHECK:       for.inc11.loopexit:
+; CHECK-NEXT:    [[ADD10_LCSSA:%.*]] = phi double [ [[ADD10]], [[FOR_BODY7]] ]
+; CHECK-NEXT:    br label [[FOR_INC11]]
+; CHECK:       for.inc11:
+; CHECK-NEXT:    [[CHANNEL_SUM_1_LCSSA]] = phi double [ [[CHANNEL_SUM_030]], [[FOR_COND2_PREHEADER]] ], [ [[ADD10_LCSSA]], [[FOR_INC11_LOOPEXIT]] ]
+; CHECK-NEXT:    [[INC12]] = add nuw nsw i64 [[C_029]], 1
+; CHECK-NEXT:    [[LSR_IV_NEXT2]] = add nuw nsw i64 [[LSR_IV1]], 8
+; CHECK-NEXT:    [[EXITCOND32_NOT:%.*]] = icmp eq i64 [[INC12]], [[CONV]]
+; CHECK-NEXT:    br i1 [[EXITCOND32_NOT]], label [[FOR_END13_LOOPEXIT:%.*]], label [[FOR_COND2_PREHEADER]]
+; CHECK:       for.end13.loopexit:
+; CHECK-NEXT:    [[CHANNEL_SUM_1_LCSSA_LCSSA:%.*]] = phi double [ [[CHANNEL_SUM_1_LCSSA]], [[FOR_INC11]] ]
+; CHECK-NEXT:    br label [[FOR_END13]]
+; CHECK:       for.end13:
+; CHECK-NEXT:    [[CHANNEL_SUM_0_LCSSA:%.*]] = phi double [ 0.000000e+00, [[ENTRY:%.*]] ], [ [[CHANNEL_SUM_1_LCSSA_LCSSA]], [[FOR_END13_LOOPEXIT]] ]
+; CHECK-NEXT:    [[ADD14:%.*]] = fadd double [[CHANNEL_SUM_0_LCSSA]], 0.000000e+00
+; CHECK-NEXT:    store double [[ADD14]], ptr [[OPTIONAL_OUTPUT]], align 8
+; CHECK-NEXT:    ret void
+;
 entry:
   %0 = load i32, ptr %st, align 8
   %conv = zext i32 %0 to i64
@@ -198,7 +362,24 @@ declare i64 @llvm.umax.i64(i64, i64)
 %struct.PAKT_INFO = type { i32, i32, i32, [0 x i32] }
 
 define i64 @alac_seek(ptr %0) {
-; CHECK: Is not safe to expand terminating value for phi node  %indvars.iv.i = phi i64 [ 0, %entry ], [ %indvars.iv.next.i, %for.body.i ]
+; CHECK-LABEL: define i64 @alac_seek
+; CHECK-SAME: (ptr [[TMP0:%.*]]) {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[DIV:%.*]] = udiv i64 1, 0
+; CHECK-NEXT:    [[TMP1:%.*]] = add nuw nsw i64 [[DIV]], 1
+; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i8, ptr [[TMP0]], i64 12
+; CHECK-NEXT:    br label [[FOR_BODY_I:%.*]]
+; CHECK:       for.body.i:
+; CHECK-NEXT:    [[LSR_IV1:%.*]] = phi ptr [ [[SCEVGEP2:%.*]], [[FOR_BODY_I]] ], [ [[SCEVGEP]], [[ENTRY:%.*]] ]
+; CHECK-NEXT:    [[LSR_IV:%.*]] = phi i64 [ [[LSR_IV_NEXT:%.*]], [[FOR_BODY_I]] ], [ [[TMP1]], [[ENTRY]] ]
+; CHECK-NEXT:    [[TMP2:%.*]] = load i32, ptr [[LSR_IV1]], align 4
+; CHECK-NEXT:    [[LSR_IV_NEXT]] = add i64 [[LSR_IV]], -1
+; CHECK-NEXT:    [[SCEVGEP2]] = getelementptr i8, ptr [[LSR_IV1]], i64 4
+; CHECK-NEXT:    [[EXITCOND_NOT_I:%.*]] = icmp eq i64 [[LSR_IV_NEXT]], 0
+; CHECK-NEXT:    br i1 [[EXITCOND_NOT_I]], label [[ALAC_PAKT_BLOCK_OFFSET_EXIT:%.*]], label [[FOR_BODY_I]]
+; CHECK:       alac_pakt_block_offset.exit:
+; CHECK-NEXT:    ret i64 0
+;
 entry:
   %div = udiv i64 1, 0
   br label %for.body.i
diff --git a/llvm/test/Transforms/LoopStrengthReduce/lsr-term-fold.ll b/llvm/test/Transforms/LoopStrengthReduce/lsr-term-fold.ll
index 23b06ba8de249..0755b5ee33589 100644
--- a/llvm/test/Transforms/LoopStrengthReduce/lsr-term-fold.ll
+++ b/llvm/test/Transforms/LoopStrengthReduce/lsr-term-fold.ll
@@ -1,32 +1,19 @@
 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -passes="loop-reduce" -S -lsr-term-fold | FileCheck %s
 
-; There are 3 test cases here regarding the replacing the primary iv with
-; other affine AddRec IV.
-; 1. Loop with constant tripcount
-; 2. Loop with runtime tripcount
-; 3. The replacing AddRec IV is a complicated AddRec. This tests whether
-;    the fold terminating condition transformation is writing new terminating
-;    condition in the correct type.
-
-target datalayout = "e-p:32:32:32-n32"
-
-; This is equivalent to the following code, loop guard is removed.
-;   void const_tripcount(int *a) {
-;     for (int i = 21; i < 400; i++)
-;       a[i] = 1;
-;   }
+target datalayout = "e-p:64:64:64-n64"
+
 define void @const_tripcount(ptr %a) {
 ; CHECK-LABEL: @const_tripcount(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[UGLYGEP:%.*]] = getelementptr i8, ptr [[A:%.*]], i64 84
-; CHECK-NEXT:    [[UGLYGEP1:%.*]] = getelementptr i8, ptr [[A]], i32 1600
+; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i8, ptr [[A]], i64 1600
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[LSR_IV1:%.*]] = phi ptr [ [[UGLYGEP2:%.*]], [[FOR_BODY]] ], [ [[UGLYGEP]], [[ENTRY:%.*]] ]
 ; CHECK-NEXT:    store i32 1, ptr [[LSR_IV1]], align 4
 ; CHECK-NEXT:    [[UGLYGEP2]] = getelementptr i8, ptr [[LSR_IV1]], i64 4
-; CHECK-NEXT:    [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND:%.*]] = icmp eq ptr [[UGLYGEP2]], [[UGLYGEP1]]
+; CHECK-NEXT:    [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND:%.*]] = icmp eq ptr [[UGLYGEP2]], [[SCEVGEP]]
 ; CHECK-NEXT:    br i1 [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND]], label [[FOR_END:%.*]], label [[FOR_BODY]]
 ; CHECK:       for.end:
 ; CHECK-NEXT:    ret void
@@ -48,25 +35,20 @@ for.end:                                          ; preds = %for.body
   ret void
 }
 
-; This is equivalent to the following code, loop guard is removed.
-;   void runtime_tripcount(int *a, int N) {
-;     for (int i = 21; i < N; i++)
-;       a[i] = 1;
-;   }
-
 define void @runtime_tripcount(ptr %a, i32 %N) {
 ; CHECK-LABEL: @runtime_tripcount(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[UGLYGEP:%.*]] = getelementptr i8, ptr [[A:%.*]], i32 84
-; CHECK-NEXT:    [[TMP0:%.*]] = shl i32 [[N:%.*]], 2
-; CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[TMP0]], 84
-; CHECK-NEXT:    [[UGLYGEP1:%.*]] = getelementptr i8, ptr [[A]], i32 [[TMP1]]
+; CHECK-NEXT:    [[TMP0:%.*]] = zext i32 [[N:%.*]] to i64
+; CHECK-NEXT:    [[TMP1:%.*]] = shl nuw nsw i64 [[TMP0]], 2
+; CHECK-NEXT:    [[TMP2:%.*]] = add nuw nsw i64 [[TMP1]], 84
+; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i8, ptr [[A]], i64 [[TMP2]]
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[LSR_IV1:%.*]] = phi ptr [ [[UGLYGEP2:%.*]], [[FOR_BODY]] ], [ [[UGLYGEP]], [[ENTRY:%.*]] ]
 ; CHECK-NEXT:    store i32 1, ptr [[LSR_IV1]], align 4
 ; CHECK-NEXT:    [[UGLYGEP2]] = getelementptr i8, ptr [[LSR_IV1]], i64 4
-; CHECK-NEXT:    [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND:%.*]] = icmp eq ptr [[UGLYGEP2]], [[UGLYGEP1]]
+; CHECK-NEXT:    [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND:%.*]] = icmp eq ptr [[UGLYGEP2]], [[SCEVGEP]]
 ; CHECK-NEXT:    br i1 [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND]], label [[FOR_END:%.*]], label [[FOR_BODY]]
 ; CHECK:       for.end:
 ; CHECK-NEXT:    ret void
@@ -88,28 +70,23 @@ for.end:                                          ; preds = %for.body
   ret void
 }
 
-; This is modified from IR of the following code, loop guard is removed.
-;   void foo(int*);
-;   void ptr_of_ptr_addrec(int **ptrptr, int length) {
-;     int **it = ptrptr;
-;       for (int i = length; i; i--) {
-;           foo(*it);
-;           it++;
-;       }
-;   }
+; The replacing AddRec IV is a complicated AddRec. This tests whether
+; the fold terminating condition transformation is writing new terminating
+; condition in the correct type.
 define void @ptr_of_ptr_addrec(ptr %ptrptr, i32 %length) {
 ; CHECK-LABEL: @ptr_of_ptr_addrec(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[START_PTRPTR:%.*]] = getelementptr ptr, ptr [[PTRPTR:%.*]]
-; CHECK-NEXT:    [[TMP0:%.*]] = shl i32 [[LENGTH:%.*]], 2
-; CHECK-NEXT:    [[UGLYGEP:%.*]] = getelementptr i8, ptr [[START_PTRPTR]], i32 [[TMP0]]
+; CHECK-NEXT:    [[TMP0:%.*]] = zext i32 [[LENGTH:%.*]] to i64
+; CHECK-NEXT:    [[TMP1:%.*]] = shl nuw nsw i64 [[TMP0]], 3
+; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i8, ptr [[START_PTRPTR]], i64 [[TMP1]]
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[IT_04:%.*]] = phi ptr [ [[INCDEC_PTR:%.*]], [[FOR_BODY]] ], [ [[START_PTRPTR]], [[ENTRY:%.*]] ]
-; CHECK-NEXT:    [[TMP1:%.*]] = load ptr, ptr [[IT_04]], align 8
-; CHECK-NEXT:    tail call void @foo(ptr [[TMP1]])
+; CHECK-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[IT_04]], align 8
+; CHECK-NEXT:    tail call void @foo(ptr [[TMP2]])
 ; CHECK-NEXT:    [[INCDEC_PTR]] = getelementptr inbounds ptr, ptr [[IT_04]], i64 1
-; CHECK-NEXT:    [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND:%.*]] = icmp eq ptr [[INCDEC_PTR]], [[UGLYGEP]]
+; CHECK-NEXT:    [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND:%.*]] = icmp eq ptr [[INCDEC_PTR]], [[SCEVGEP]]
 ; CHECK-NEXT:    br i1 [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND]], label [[FOR_END:%.*]], label [[FOR_BODY]]
 ; CHECK:       for.end:
 ; CHECK-NEXT:    ret void
@@ -140,8 +117,9 @@ define void @iv_start_non_preheader(ptr %mark, i32 signext %length) {
 ; CHECK-NEXT:    [[TOBOOL_NOT3:%.*]] = icmp eq i32 [[LENGTH:%.*]], 0
 ; CHECK-NEXT:    br i1 [[TOBOOL_NOT3]], label [[FOR_COND_CLEANUP:%.*]], label [[FOR_BODY_PREHEADER:%.*]]
 ; CHECK:       for.body.preheader:
-; CHECK-NEXT:    [[TMP0:%.*]] = shl i32 [[LENGTH]], 2
-; CHECK-NEXT:    [[UGLYGEP:%.*]] = getelementptr i8, ptr [[MARK:%.*]], i32 [[TMP0]]
+; CHECK-NEXT:    [[TMP0:%.*]] = zext i32 [[LENGTH]] to i64
+; CHECK-NEXT:    [[TMP1:%.*]] = shl nuw nsw i64 [[TMP0]], 3
+; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i8, ptr [[MARK:%.*]], i64 [[TMP1]]
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.cond.cleanup.loopexit:
 ; CHECK-NEXT:    br label [[FOR_COND_CLEANUP]]
@@ -149,10 +127,10 @@ define void @iv_start_non_preheader(ptr %mark, i32 signext %length) {
 ; CHECK-NEXT:    ret void
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[DST_04:%.*]] = phi ptr [ [[INCDEC_PTR:%.*]], [[FOR_BODY]] ], [ [[MARK]], [[FOR_BODY_PREHEADER]] ]
-; CHECK-NEXT:    [[TMP1:%.*]] = load ptr, ptr [[DST_04]], align 8
-; CHECK-NEXT:    [[TMP2:%.*]] = call ptr @foo(ptr [[TMP1]])
+; CHECK-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[DST_04]], align 8
+; CHECK-NEXT:    [[TMP3:%.*]] = call ptr @foo(ptr [[TMP2]])
 ; CHECK-NEXT:    [[INCDEC_PTR]] = getelementptr inbounds ptr, ptr [[DST_04]], i64 1
-; CHECK-NEXT:    [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND:%.*]] = icmp eq ptr [[INCDEC_PTR]], [[UGLYGEP]]
+; CHECK-NEXT:    [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND:%.*]] = icmp eq ptr [[INCDEC_PTR]], [[SCEVGEP]]
 ; CHECK-NEXT:    br i1 [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND]], label [[FOR_COND_CLEANUP_LOOPEXIT:%.*]], label [[FOR_BODY]]
 ;
 entry:
@@ -172,3 +150,77 @@ for.body:                                         ; preds = %entry, %for.body
   %tobool.not = icmp eq i32 %dec, 0
   br i1 %tobool.not, label %for.cond.cleanup, label %for.body
 }
+
+; Consider the case where %a points to a buffer exactly 17 bytes long.  The
+; loop below will access bytes: 0, 4, 8, and 16.  The key bit is that we
+; advance the pointer IV by *4* each time, and thus on the iteration we write
+; byte 16, %uglygep2 (the pointer increment) is past the end of the underlying
+; storage and thus violates the inbounds requirements.  As a result, %uglygep2
+; is poison on the final iteration.  If we insert a branch on that value, we
+; have inserted undefined behavior where it did not previously exist.
+; FIXME: miscompile
+define void @inbounds_poison_use(ptr %a) {
+; CHECK-LABEL: @inbounds_poison_use(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i8, ptr [[A:%.*]], i64 16
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[LSR_IV1:%.*]] = phi ptr [ [[UGLYGEP2:%.*]], [[FOR_BODY]] ], [ [[A]], [[ENTRY:%.*]] ]
+; CHECK-NEXT:    store i8 1, ptr [[LSR_IV1]], align 4
+; CHECK-NEXT:    [[UGLYGEP2]] = getelementptr inbounds i8, ptr [[LSR_IV1]], i64 4
+; CHECK-NEXT:    [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND:%.*]] = icmp eq ptr [[UGLYGEP2]], [[SCEVGEP]]
+; CHECK-NEXT:    br i1 [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND]], label [[FOR_END:%.*]], label [[FOR_BODY]]
+; CHECK:       for.end:
+; CHECK-NEXT:    ret void
+;
+entry:
+  br label %for.body
+
+for.body:                                         ; preds = %for.body, %entry
+  %lsr.iv1 = phi ptr [ %uglygep2, %for.body ], [ %a, %entry ]
+  %lsr.iv = phi i32 [ %lsr.iv.next, %for.body ], [ 4, %entry ]
+  store i8 1, ptr %lsr.iv1, align 4
+  %lsr.iv.next = add nsw i32 %lsr.iv, -1
+  %uglygep2 = getelementptr inbounds i8, ptr %lsr.iv1, i64 4
+  %exitcond.not = icmp eq i32 %lsr.iv.next, 0
+  br i1 %exitcond.not, label %for.end, label %for.body
+
+for.end:                                          ; preds = %for.body
+  ret void
+}
+
+; In this case, the integer IV has a larger bitwidth than the pointer IV.
+; This means that the smaller IV may wrap around multiple times before
+; the original loop exit is taken.
+; FIXME: miscompile
+define void @iv_size(ptr %a, i128 %N) {
+; CHECK-LABEL: @iv_size(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = trunc i128 [[N:%.*]] to i64
+; CHECK-NEXT:    [[TMP1:%.*]] = shl i64 [[TMP0]], 2
+; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i8, ptr [[A:%.*]], i64 [[TMP1]]
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[LSR_IV1:%.*]] = phi ptr [ [[UGLYGEP2:%.*]], [[FOR_BODY]] ], [ [[A]], [[ENTRY:%.*]] ]
+; CHECK-NEXT:    store i32 1, ptr [[LSR_IV1]], align 4
+; CHECK-NEXT:    [[UGLYGEP2]] = getelementptr i8, ptr [[LSR_IV1]], i64 4
+; CHECK-NEXT:    [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND:%.*]] = icmp eq ptr [[UGLYGEP2]], [[SCEVGEP]]
+; CHECK-NEXT:    br i1 [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND]], label [[FOR_END:%.*]], label [[FOR_BODY]]
+; CHECK:       for.end:
+; CHECK-NEXT:    ret void
+;
+entry:
+  br label %for.body
+
+for.body:                                         ; preds = %for.body, %entry
+  %lsr.iv1 = phi ptr [ %uglygep2, %for.body ], [ %a, %entry ]
+  %lsr.iv = phi i128 [ %lsr.iv.next, %for.body ], [ %N, %entry ]
+  store i32 1, ptr %lsr.iv1, align 4
+  %lsr.iv.next = add nsw i128 %lsr.iv, -1
+  %uglygep2 = getelementptr i8, ptr %lsr.iv1, i64 4
+  %exitcond.not = icmp eq i128 %lsr.iv.next, 0
+  br i1 %exitcond.not, label %for.end, label %for.body
+
+for.end:                                          ; preds = %for.body
+  ret void
+}
diff --git a/llvm/test/Transforms/LoopStrengthReduce/scev-after-loopinstsimplify.ll b/llvm/test/Transforms/LoopStrengthReduce/scev-after-loopinstsimplify.ll
index 6c91f643e7a9d..6e059b5d8fa59 100644
--- a/llvm/test/Transforms/LoopStrengthReduce/scev-after-loopinstsimplify.ll
+++ b/llvm/test/Transforms/LoopStrengthReduce/scev-after-loopinstsimplify.ll
@@ -1,4 +1,4 @@
-; RUN: opt %s -indvars -loop-instsimplify -loop-reduce
+; RUN: opt %s -passes=indvars,loop-instsimplify,loop-reduce
 ; We are only checking that there is no crash!
 
 ; https://bugs.llvm.org/show_bug.cgi?id=37936
diff --git a/llvm/test/Transforms/LoopUnroll/guard-cost-for-unrolling.ll b/llvm/test/Transforms/LoopUnroll/guard-cost-for-unrolling.ll
new file mode 100644
index 0000000000000..596c12dae0779
--- /dev/null
+++ b/llvm/test/Transforms/LoopUnroll/guard-cost-for-unrolling.ll
@@ -0,0 +1,69 @@
+; RUN: opt -S -disable-output -unroll-partial-threshold=16 -debug-only=loop-unroll -passes='loop-unroll<runtime>' < %s 2>&1 | FileCheck %s
+
+; REQUIRES: asserts
+
+; This test is needed to make sure that the guard cost remains the same,
+; independently on guard representation form (either intrinsic call or branch with
+; widenable condition).
+
+define void @test_guard_as_intrinsic(ptr %arr, i64 %n, i64 %bound) {
+; CHECK-LABEL: Loop Unroll: F[test_guard_as_intrinsic] Loop %loop
+; CHECK-NEXT:    Loop Size = 8
+; CHECK-NEXT:    runtime unrolling with count: 2
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i64 [0, %entry], [%iv.next, %loop]
+  %gep = getelementptr i64, ptr %arr, i64 %iv
+  %bound_check = icmp ult i64 %iv, %bound
+  call void(i1, ...) @llvm.experimental.guard(i1 %bound_check) [ "deopt"() ]
+  store i64 %iv, ptr %gep, align 8
+  store i64 %iv, ptr %gep, align 8
+  %iv.next = add i64 %iv, 1
+  %loop_cond = icmp ult i64 %iv, %n
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:
+  ret void
+}
+
+define void @test_guard_as_branch(ptr %arr, i64 %n, i64 %bound) {
+; CHECK-LABEL: Loop Unroll: F[test_guard_as_branch] Loop %loop
+; CHECK-NEXT:    Loop Size = 8
+; CHECK-NEXT:    runtime unrolling with count: 2
+entry:
+  br label %loop
+
+loop:                                             ; preds = %guarded, %entry
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %guarded ]
+  %gep = getelementptr i64, ptr %arr, i64 %iv
+  %bound_check = icmp ult i64 %iv, %bound
+  %widenable_cond = call i1 @llvm.experimental.widenable.condition()
+  %exiplicit_guard_cond = and i1 %bound_check, %widenable_cond
+  br i1 %exiplicit_guard_cond, label %guarded, label %deopt, !prof !0
+
+deopt:                                            ; preds = %loop
+  call void (...) @llvm.experimental.deoptimize.isVoid() [ "deopt"() ]
+  ret void
+
+guarded:                                          ; preds = %loop
+  store i64 %iv, ptr %gep, align 8
+  store i64 %iv, ptr %gep, align 8
+  %iv.next = add i64 %iv, 1
+  %loop_cond = icmp ult i64 %iv, %n
+  br i1 %loop_cond, label %loop, label %exit
+
+exit:                                             ; preds = %guarded
+  ret void
+}
+
+; Function Attrs: nocallback nofree nosync willreturn
+declare void @llvm.experimental.guard(i1, ...) #0
+
+declare void @llvm.experimental.deoptimize.isVoid(...)
+
+; Function Attrs: nocallback nofree nosync nounwind speculatable willreturn memory(inaccessiblemem: readwrite)
+declare i1 @llvm.experimental.widenable.condition() #1
+
+!0 = !{!"branch_weights", i32 1048576, i32 1}
diff --git a/llvm/test/Transforms/LoopUnroll/peel-loop-conditions.ll b/llvm/test/Transforms/LoopUnroll/peel-loop-conditions.ll
index f3b82a9442635..84cedb6e8f377 100644
--- a/llvm/test/Transforms/LoopUnroll/peel-loop-conditions.ll
+++ b/llvm/test/Transforms/LoopUnroll/peel-loop-conditions.ll
@@ -644,23 +644,64 @@ for.end:
   ret void
 }
 
-; Comparison with non-monotonic predicate due to possible wrapping, loop
-; body cannot be simplified.
 define void @test8(i32 %k) {
 ; CHECK-LABEL: @test8(
 ; CHECK-NEXT:  for.body.lr.ph:
+; CHECK-NEXT:    br label [[FOR_BODY_PEEL_BEGIN:%.*]]
+; CHECK:       for.body.peel.begin:
+; CHECK-NEXT:    br label [[FOR_BODY_PEEL:%.*]]
+; CHECK:       for.body.peel:
+; CHECK-NEXT:    [[CMP1_PEEL:%.*]] = icmp slt i32 0, 3
+; CHECK-NEXT:    br i1 [[CMP1_PEEL]], label [[IF_THEN_PEEL:%.*]], label [[FOR_INC_PEEL:%.*]]
+; CHECK:       if.then.peel:
+; CHECK-NEXT:    call void @f1()
+; CHECK-NEXT:    br label [[FOR_INC_PEEL]]
+; CHECK:       for.inc.peel:
+; CHECK-NEXT:    [[INC_PEEL:%.*]] = add i32 0, 1
+; CHECK-NEXT:    [[CMP_PEEL:%.*]] = icmp slt i32 [[INC_PEEL]], [[K:%.*]]
+; CHECK-NEXT:    br i1 [[CMP_PEEL]], label [[FOR_BODY_PEEL_NEXT:%.*]], label [[FOR_END:%.*]]
+; CHECK:       for.body.peel.next:
+; CHECK-NEXT:    br label [[FOR_BODY_PEEL2:%.*]]
+; CHECK:       for.body.peel2:
+; CHECK-NEXT:    [[CMP1_PEEL3:%.*]] = icmp slt i32 [[INC_PEEL]], 3
+; CHECK-NEXT:    br i1 [[CMP1_PEEL3]], label [[IF_THEN_PEEL4:%.*]], label [[FOR_INC_PEEL5:%.*]]
+; CHECK:       if.then.peel4:
+; CHECK-NEXT:    call void @f1()
+; CHECK-NEXT:    br label [[FOR_INC_PEEL5]]
+; CHECK:       for.inc.peel5:
+; CHECK-NEXT:    [[INC_PEEL6:%.*]] = add i32 [[INC_PEEL]], 1
+; CHECK-NEXT:    [[CMP_PEEL7:%.*]] = icmp slt i32 [[INC_PEEL6]], [[K]]
+; CHECK-NEXT:    br i1 [[CMP_PEEL7]], label [[FOR_BODY_PEEL_NEXT1:%.*]], label [[FOR_END]]
+; CHECK:       for.body.peel.next1:
+; CHECK-NEXT:    br label [[FOR_BODY_PEEL9:%.*]]
+; CHECK:       for.body.peel9:
+; CHECK-NEXT:    [[CMP1_PEEL10:%.*]] = icmp slt i32 [[INC_PEEL6]], 3
+; CHECK-NEXT:    br i1 [[CMP1_PEEL10]], label [[IF_THEN_PEEL11:%.*]], label [[FOR_INC_PEEL12:%.*]]
+; CHECK:       if.then.peel11:
+; CHECK-NEXT:    call void @f1()
+; CHECK-NEXT:    br label [[FOR_INC_PEEL12]]
+; CHECK:       for.inc.peel12:
+; CHECK-NEXT:    [[INC_PEEL13:%.*]] = add i32 [[INC_PEEL6]], 1
+; CHECK-NEXT:    [[CMP_PEEL14:%.*]] = icmp slt i32 [[INC_PEEL13]], [[K]]
+; CHECK-NEXT:    br i1 [[CMP_PEEL14]], label [[FOR_BODY_PEEL_NEXT8:%.*]], label [[FOR_END]]
+; CHECK:       for.body.peel.next8:
+; CHECK-NEXT:    br label [[FOR_BODY_PEEL_NEXT15:%.*]]
+; CHECK:       for.body.peel.next15:
+; CHECK-NEXT:    br label [[FOR_BODY_LR_PH_PEEL_NEWPH:%.*]]
+; CHECK:       for.body.lr.ph.peel.newph:
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.body:
-; CHECK-NEXT:    [[I_05:%.*]] = phi i32 [ 0, [[FOR_BODY_LR_PH:%.*]] ], [ [[INC:%.*]], [[FOR_INC:%.*]] ]
-; CHECK-NEXT:    [[CMP1:%.*]] = icmp slt i32 [[I_05]], 3
-; CHECK-NEXT:    br i1 [[CMP1]], label [[IF_THEN:%.*]], label [[FOR_INC]]
+; CHECK-NEXT:    [[I_05:%.*]] = phi i32 [ [[INC_PEEL13]], [[FOR_BODY_LR_PH_PEEL_NEWPH]] ], [ [[INC:%.*]], [[FOR_INC:%.*]] ]
+; CHECK-NEXT:    br i1 false, label [[IF_THEN:%.*]], label [[FOR_INC]]
 ; CHECK:       if.then:
 ; CHECK-NEXT:    call void @f1()
 ; CHECK-NEXT:    br label [[FOR_INC]]
 ; CHECK:       for.inc:
-; CHECK-NEXT:    [[INC]] = add i32 [[I_05]], 1
-; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 [[INC]], [[K:%.*]]
-; CHECK-NEXT:    br i1 [[CMP]], label [[FOR_BODY]], label [[FOR_END:%.*]]
+; CHECK-NEXT:    [[INC]] = add nuw nsw i32 [[I_05]], 1
+; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 [[INC]], [[K]]
+; CHECK-NEXT:    br i1 [[CMP]], label [[FOR_BODY]], label [[FOR_END_LOOPEXIT:%.*]], !llvm.loop [[LOOP8:![0-9]+]]
+; CHECK:       for.end.loopexit:
+; CHECK-NEXT:    br label [[FOR_END]]
 ; CHECK:       for.end:
 ; CHECK-NEXT:    ret void
 ;
@@ -728,7 +769,7 @@ define void @test_9__peel_first_iter_via_slt_pred(i32 %len) {
 ; CHECK-NEXT:    call void @sink()
 ; CHECK-NEXT:    [[INC]] = add nuw nsw i32 [[I_06]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i32 [[INC]], [[LEN]]
-; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_COND_CLEANUP_LOOPEXIT_LOOPEXIT:%.*]], label [[FOR_BODY]], !llvm.loop !8
+; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_COND_CLEANUP_LOOPEXIT_LOOPEXIT:%.*]], label [[FOR_BODY]], !llvm.loop [[LOOP9:![0-9]+]]
 ;
 entry:
   %cmp5 = icmp sgt i32 %len, 0
@@ -795,7 +836,7 @@ define void @test_10__peel_first_iter_via_sgt_pred(i32 %len) {
 ; CHECK-NEXT:    call void @sink()
 ; CHECK-NEXT:    [[INC]] = add nuw nsw i32 [[I_06]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i32 [[INC]], [[LEN]]
-; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_COND_CLEANUP_LOOPEXIT_LOOPEXIT:%.*]], label [[FOR_BODY]], !llvm.loop !10
+; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_COND_CLEANUP_LOOPEXIT_LOOPEXIT:%.*]], label [[FOR_BODY]], !llvm.loop [[LOOP11:![0-9]+]]
 ;
 entry:
   %cmp5 = icmp sgt i32 %len, 0
@@ -864,7 +905,7 @@ define void @test11__peel_first_iter_via_eq_pred(i32 %len) {
 ; CHECK-NEXT:    call void @sink()
 ; CHECK-NEXT:    [[INC]] = add nuw nsw i32 [[I_06]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i32 [[INC]], [[LEN]]
-; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_COND_CLEANUP_LOOPEXIT_LOOPEXIT:%.*]], label [[FOR_BODY]], !llvm.loop !11
+; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_COND_CLEANUP_LOOPEXIT_LOOPEXIT:%.*]], label [[FOR_BODY]], !llvm.loop [[LOOP12:![0-9]+]]
 ;
 entry:
   %cmp5 = icmp sgt i32 %len, 0
@@ -933,7 +974,7 @@ define void @test12__peel_first_iter_via_ne_pred(i32 %len) {
 ; CHECK-NEXT:    call void @sink()
 ; CHECK-NEXT:    [[INC]] = add nuw nsw i32 [[I_06]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i32 [[INC]], [[LEN]]
-; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_COND_CLEANUP_LOOPEXIT_LOOPEXIT:%.*]], label [[FOR_BODY]], !llvm.loop !12
+; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_COND_CLEANUP_LOOPEXIT_LOOPEXIT:%.*]], label [[FOR_BODY]], !llvm.loop [[LOOP13:![0-9]+]]
 ;
 entry:
   %cmp5 = icmp sgt i32 %len, 0
@@ -1157,7 +1198,7 @@ define void @test17() personality ptr undef{
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    invoke void @f1()
-; CHECK-NEXT:    to label [[LOOP]] unwind label [[EH_UNW_LOOPEXIT_LOOPEXIT:%.*]], !llvm.loop [[LOOP13:![0-9]+]]
+; CHECK-NEXT:    to label [[LOOP]] unwind label [[EH_UNW_LOOPEXIT_LOOPEXIT:%.*]], !llvm.loop [[LOOP14:![0-9]+]]
 ; CHECK:       eh.Unw.loopexit.loopexit:
 ; CHECK-NEXT:    [[LPAD_LOOPEXIT2:%.*]] = landingpad { ptr, i32 }
 ; CHECK-NEXT:    catch ptr null
@@ -1209,7 +1250,7 @@ define void @test18(ptr %p) {
 ; CHECK-NEXT:    [[CONTROL:%.*]] = load volatile i32, ptr [[P]], align 4
 ; CHECK-NEXT:    switch i32 [[CONTROL]], label [[EXIT_LOOPEXIT:%.*]] [
 ; CHECK-NEXT:    i32 2, label [[LOOP]]
-; CHECK-NEXT:    ], !llvm.loop [[LOOP14:![0-9]+]]
+; CHECK-NEXT:    ], !llvm.loop [[LOOP15:![0-9]+]]
 ; CHECK:       exit.loopexit:
 ; CHECK-NEXT:    br label [[EXIT]]
 ; CHECK:       exit:
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/interleave-allocsize-not-equal-typesize.ll b/llvm/test/Transforms/LoopVectorize/AArch64/interleave-allocsize-not-equal-typesize.ll
index 32ca12ee7e0ef..014620487c202 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/interleave-allocsize-not-equal-typesize.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/interleave-allocsize-not-equal-typesize.ll
@@ -29,42 +29,31 @@ define void @pr58722_load_interleave_group(ptr %src, ptr %dst) {
 ; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr inbounds i64, ptr [[SRC]], i64 [[TMP1]]
 ; CHECK-NEXT:    [[TMP6:%.*]] = getelementptr inbounds i64, ptr [[SRC]], i64 [[TMP2]]
 ; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr inbounds i64, ptr [[SRC]], i64 [[TMP3]]
-; CHECK-NEXT:    [[TMP8:%.*]] = load i32, ptr [[TMP4]], align 4, !alias.scope !0
-; CHECK-NEXT:    [[TMP9:%.*]] = load i32, ptr [[TMP5]], align 4, !alias.scope !0
-; CHECK-NEXT:    [[TMP10:%.*]] = insertelement <2 x i32> poison, i32 [[TMP8]], i32 0
-; CHECK-NEXT:    [[TMP11:%.*]] = insertelement <2 x i32> [[TMP10]], i32 [[TMP9]], i32 1
-; CHECK-NEXT:    [[TMP12:%.*]] = load i32, ptr [[TMP6]], align 4, !alias.scope !0
-; CHECK-NEXT:    [[TMP13:%.*]] = load i32, ptr [[TMP7]], align 4, !alias.scope !0
-; CHECK-NEXT:    [[TMP14:%.*]] = insertelement <2 x i32> poison, i32 [[TMP12]], i32 0
-; CHECK-NEXT:    [[TMP15:%.*]] = insertelement <2 x i32> [[TMP14]], i32 [[TMP13]], i32 1
-; CHECK-NEXT:    [[TMP16:%.*]] = getelementptr inbounds i32, ptr [[TMP4]], i64 1
-; CHECK-NEXT:    [[TMP17:%.*]] = getelementptr inbounds i32, ptr [[TMP5]], i64 1
-; CHECK-NEXT:    [[TMP18:%.*]] = getelementptr inbounds i32, ptr [[TMP6]], i64 1
-; CHECK-NEXT:    [[TMP19:%.*]] = getelementptr inbounds i32, ptr [[TMP7]], i64 1
-; CHECK-NEXT:    [[TMP20:%.*]] = load i24, ptr [[TMP16]], align 4, !alias.scope !0
-; CHECK-NEXT:    [[TMP21:%.*]] = load i24, ptr [[TMP17]], align 4, !alias.scope !0
-; CHECK-NEXT:    [[TMP22:%.*]] = insertelement <2 x i24> poison, i24 [[TMP20]], i32 0
-; CHECK-NEXT:    [[TMP23:%.*]] = insertelement <2 x i24> [[TMP22]], i24 [[TMP21]], i32 1
-; CHECK-NEXT:    [[TMP24:%.*]] = load i24, ptr [[TMP18]], align 4, !alias.scope !0
-; CHECK-NEXT:    [[TMP25:%.*]] = load i24, ptr [[TMP19]], align 4, !alias.scope !0
-; CHECK-NEXT:    [[TMP26:%.*]] = insertelement <2 x i24> poison, i24 [[TMP24]], i32 0
-; CHECK-NEXT:    [[TMP27:%.*]] = insertelement <2 x i24> [[TMP26]], i24 [[TMP25]], i32 1
-; CHECK-NEXT:    [[TMP28:%.*]] = zext <2 x i24> [[TMP23]] to <2 x i32>
-; CHECK-NEXT:    [[TMP29:%.*]] = zext <2 x i24> [[TMP27]] to <2 x i32>
-; CHECK-NEXT:    [[TMP30:%.*]] = add <2 x i32> [[TMP11]], [[TMP28]]
-; CHECK-NEXT:    [[TMP31:%.*]] = add <2 x i32> [[TMP15]], [[TMP29]]
-; CHECK-NEXT:    [[TMP32:%.*]] = getelementptr inbounds i32, ptr [[DST]], i64 [[TMP0]]
-; CHECK-NEXT:    [[TMP33:%.*]] = getelementptr inbounds i32, ptr [[DST]], i64 [[TMP2]]
-; CHECK-NEXT:    [[TMP34:%.*]] = getelementptr inbounds i32, ptr [[TMP32]], i32 0
-; CHECK-NEXT:    store <2 x i32> [[TMP30]], ptr [[TMP34]], align 4, !alias.scope !3, !noalias !0
-; CHECK-NEXT:    [[TMP35:%.*]] = getelementptr inbounds i32, ptr [[TMP32]], i32 2
-; CHECK-NEXT:    store <2 x i32> [[TMP31]], ptr [[TMP35]], align 4, !alias.scope !3, !noalias !0
+; CHECK-NEXT:    [[TMP8:%.*]] = getelementptr inbounds i32, ptr [[TMP4]], i32 0
+; CHECK-NEXT:    [[WIDE_VEC:%.*]] = load <8 x i32>, ptr [[TMP8]], align 4
+; CHECK-NEXT:    [[STRIDED_VEC:%.*]] = shufflevector <8 x i32> [[WIDE_VEC]], <8 x i32> poison, <4 x i32> <i32 0, i32 2, i32 4, i32 6>
+; CHECK-NEXT:    [[TMP9:%.*]] = getelementptr inbounds i32, ptr [[TMP4]], i64 1
+; CHECK-NEXT:    [[TMP10:%.*]] = getelementptr inbounds i32, ptr [[TMP5]], i64 1
+; CHECK-NEXT:    [[TMP11:%.*]] = getelementptr inbounds i32, ptr [[TMP6]], i64 1
+; CHECK-NEXT:    [[TMP12:%.*]] = getelementptr inbounds i32, ptr [[TMP7]], i64 1
+; CHECK-NEXT:    [[TMP13:%.*]] = load i24, ptr [[TMP9]], align 4, !alias.scope !0
+; CHECK-NEXT:    [[TMP14:%.*]] = load i24, ptr [[TMP10]], align 4, !alias.scope !0
+; CHECK-NEXT:    [[TMP15:%.*]] = load i24, ptr [[TMP11]], align 4, !alias.scope !0
+; CHECK-NEXT:    [[TMP16:%.*]] = load i24, ptr [[TMP12]], align 4, !alias.scope !0
+; CHECK-NEXT:    [[TMP17:%.*]] = insertelement <4 x i24> poison, i24 [[TMP13]], i32 0
+; CHECK-NEXT:    [[TMP18:%.*]] = insertelement <4 x i24> [[TMP17]], i24 [[TMP14]], i32 1
+; CHECK-NEXT:    [[TMP19:%.*]] = insertelement <4 x i24> [[TMP18]], i24 [[TMP15]], i32 2
+; CHECK-NEXT:    [[TMP20:%.*]] = insertelement <4 x i24> [[TMP19]], i24 [[TMP16]], i32 3
+; CHECK-NEXT:    [[TMP21:%.*]] = zext <4 x i24> [[TMP20]] to <4 x i32>
+; CHECK-NEXT:    [[TMP22:%.*]] = add <4 x i32> [[STRIDED_VEC]], [[TMP21]]
+; CHECK-NEXT:    [[TMP23:%.*]] = getelementptr inbounds i32, ptr [[DST]], i64 [[TMP0]]
+; CHECK-NEXT:    [[TMP24:%.*]] = getelementptr inbounds i32, ptr [[TMP23]], i32 0
+; CHECK-NEXT:    store <4 x i32> [[TMP22]], ptr [[TMP24]], align 4, !alias.scope !3, !noalias !0
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
-; CHECK-NEXT:    [[TMP36:%.*]] = icmp eq i64 [[INDEX_NEXT]], 10000
-; CHECK-NEXT:    br i1 [[TMP36]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP5:![0-9]+]]
+; CHECK-NEXT:    [[TMP25:%.*]] = icmp eq i64 [[INDEX_NEXT]], 10000
+; CHECK-NEXT:    br i1 [[TMP25]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP5:![0-9]+]]
 ; CHECK:       middle.block:
-; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 10001, 10000
-; CHECK-NEXT:    br i1 [[CMP_N]], label [[EXIT:%.*]], label [[SCALAR_PH]]
+; CHECK-NEXT:    br label [[SCALAR_PH]]
 ; CHECK:       scalar.ph:
 ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ 10000, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ], [ 0, [[VECTOR_MEMCHECK]] ]
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
@@ -80,7 +69,7 @@ define void @pr58722_load_interleave_group(ptr %src, ptr %dst) {
 ; CHECK-NEXT:    store i32 [[SUM]], ptr [[GEP_DST]], align 4
 ; CHECK-NEXT:    [[IV_NEXT]] = add i64 [[IV]], 1
 ; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i64 [[IV]], 10000
-; CHECK-NEXT:    br i1 [[CMP]], label [[EXIT]], label [[LOOP]], !llvm.loop [[LOOP7:![0-9]+]]
+; CHECK-NEXT:    br i1 [[CMP]], label [[EXIT:%.*]], label [[LOOP]], !llvm.loop [[LOOP8:![0-9]+]]
 ; CHECK:       exit:
 ; CHECK-NEXT:    ret void
 ;
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sleef-calls-aarch64.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sleef-calls-aarch64.ll
index 623ac986fcace..be4309996fbb5 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sleef-calls-aarch64.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sleef-calls-aarch64.ll
@@ -1,5 +1,5 @@
 ; Do NOT use -O3. It will lower exp2 to ldexp, and the test will fail.
-; RUN: opt -vector-library=sleefgnuabi -replace-with-veclib -loop-unroll -loop-vectorize -S < %s | FileCheck %s
+; RUN: opt -vector-library=sleefgnuabi -replace-with-veclib < %s | opt -vector-library=sleefgnuabi -passes=inject-tli-mappings,loop-unroll,loop-vectorize -S | FileCheck %s
 
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
 target triple = "aarch64-unknown-linux-gnu"
@@ -922,6 +922,53 @@ define void @sqrt_f32(float* nocapture %varray) {
   ret void
 }
 
+	
+define void @llvm_sqrt_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @llvm_sqrt_f64(
+  ; CHECK:    [[TMP5:%.*]] = call fast <2 x double> @llvm.sqrt.v2f64(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call fast double @llvm.sqrt.f64(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @llvm_sqrt_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @llvm_sqrt_f32(
+  ; CHECK:    [[TMP5:%.*]] = call fast <4 x float> @llvm.sqrt.v4f32(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call fast float @llvm.sqrt.f32(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
 declare double @tan(double) #0
 declare float @tanf(float) #0
 declare double @llvm.tan.f64(double) #0
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-tail-folding-forced.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-tail-folding-forced.ll
index 6324805120077..5ad0994d02e0b 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-tail-folding-forced.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-tail-folding-forced.ll
@@ -10,10 +10,12 @@ target triple = "aarch64-unknown-linux-gnu"
 
 ; VPLANS-LABEL: Checking a loop in 'simple_memset'
 ; VPLANS:      VPlan 'Initial VPlan for VF={vscale x 1,vscale x 2,vscale x 4},UF>=1' {
+; VPLANS-NEXT: Live-in vp<[[TC:%[0-9]+]]> = original trip-count
+; VPLANS-EMPTY:
 ; VPLANS-NEXT: vector.ph:
 ; VPLANS-NEXT:   EMIT vp<[[VF:%[0-9]+]]> = VF * Part +  ir<0>
-; VPLANS-NEXT:   EMIT vp<[[NEWTC:%[0-9]+]]> = TC > VF ? TC - VF : 0 <badref>
-; VPLANS-NEXT:   EMIT vp<[[LANEMASK_ENTRY:%[0-9]+]]> = active lane mask vp<[[VF]]> <badref>
+; VPLANS-NEXT:   EMIT vp<[[NEWTC:%[0-9]+]]> = TC > VF ? TC - VF : 0 vp<[[TC]]>
+; VPLANS-NEXT:   EMIT vp<[[LANEMASK_ENTRY:%[0-9]+]]> = active lane mask vp<[[VF]]> vp<[[TC]]>
 ; VPLANS-NEXT: Successor(s): vector loop
 ; VPLANS-EMPTY:
 ; VPLANS-NEXT: <x1> vector loop: {
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/interleaved-accesses-zve32x.ll b/llvm/test/Transforms/LoopVectorize/RISCV/interleaved-accesses-zve32x.ll
new file mode 100644
index 0000000000000..241fa9d8cd49b
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/interleaved-accesses-zve32x.ll
@@ -0,0 +1,49 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -passes=loop-vectorize -mtriple=riscv64 -mattr=+zve32x,+zvl1024b -S | FileCheck %s
+
+; This element type isn't a supported SEW so this shouldn't be interleaved
+define void @load_store_zve32x(ptr %p) {
+; CHECK-LABEL: @load_store_zve32x(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[I:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[NEXTI:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[OFFSET0:%.*]] = shl i64 [[I]], 1
+; CHECK-NEXT:    [[Q0:%.*]] = getelementptr i64, ptr [[P:%.*]], i64 [[OFFSET0]]
+; CHECK-NEXT:    [[X0:%.*]] = load i64, ptr [[Q0]], align 4
+; CHECK-NEXT:    [[Y0:%.*]] = add i64 [[X0]], 1
+; CHECK-NEXT:    store i64 [[Y0]], ptr [[Q0]], align 4
+; CHECK-NEXT:    [[OFFSET1:%.*]] = add i64 [[OFFSET0]], 1
+; CHECK-NEXT:    [[Q1:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET1]]
+; CHECK-NEXT:    [[X1:%.*]] = load i64, ptr [[Q1]], align 4
+; CHECK-NEXT:    [[Y1:%.*]] = add i64 [[X1]], 2
+; CHECK-NEXT:    store i64 [[Y1]], ptr [[Q1]], align 4
+; CHECK-NEXT:    [[NEXTI]] = add i64 [[I]], 1
+; CHECK-NEXT:    [[DONE:%.*]] = icmp eq i64 [[NEXTI]], 1024
+; CHECK-NEXT:    br i1 [[DONE]], label [[EXIT:%.*]], label [[LOOP]]
+; CHECK:       exit:
+; CHECK-NEXT:    ret void
+;
+entry:
+  br label %loop
+loop:
+  %i = phi i64 [0, %entry], [%nexti, %loop]
+
+  %offset0 = shl i64 %i, 1
+  %q0 = getelementptr i64, ptr %p, i64 %offset0
+  %x0 = load i64, ptr %q0
+  %y0 = add i64 %x0, 1
+  store i64 %y0, ptr %q0
+
+  %offset1 = add i64 %offset0, 1
+  %q1 = getelementptr i64, ptr %p, i64 %offset1
+  %x1 = load i64, ptr %q1
+  %y1 = add i64 %x1, 2
+  store i64 %y1, ptr %q1
+
+  %nexti = add i64 %i, 1
+  %done = icmp eq i64 %nexti, 1024
+  br i1 %done, label %exit, label %loop
+exit:
+  ret void
+}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/interleaved-accesses.ll b/llvm/test/Transforms/LoopVectorize/RISCV/interleaved-accesses.ll
new file mode 100644
index 0000000000000..b81d14c520770
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/interleaved-accesses.ll
@@ -0,0 +1,518 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -passes=loop-vectorize -mtriple=riscv64 -mattr=+v -S | FileCheck %s
+
+define void @load_store_factor2_i32(ptr %p) {
+; CHECK-LABEL: @load_store_factor2_i32(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
+; CHECK:       vector.ph:
+; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
+; CHECK:       vector.body:
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[TMP0:%.*]] = add i64 [[INDEX]], 0
+; CHECK-NEXT:    [[TMP1:%.*]] = shl i64 [[TMP0]], 1
+; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr i32, ptr [[P:%.*]], i64 [[TMP1]]
+; CHECK-NEXT:    [[TMP3:%.*]] = getelementptr i32, ptr [[TMP2]], i32 0
+; CHECK-NEXT:    [[WIDE_VEC:%.*]] = load <8 x i32>, ptr [[TMP3]], align 4
+; CHECK-NEXT:    [[STRIDED_VEC:%.*]] = shufflevector <8 x i32> [[WIDE_VEC]], <8 x i32> poison, <4 x i32> <i32 0, i32 2, i32 4, i32 6>
+; CHECK-NEXT:    [[STRIDED_VEC1:%.*]] = shufflevector <8 x i32> [[WIDE_VEC]], <8 x i32> poison, <4 x i32> <i32 1, i32 3, i32 5, i32 7>
+; CHECK-NEXT:    [[TMP4:%.*]] = add <4 x i32> [[STRIDED_VEC]], <i32 1, i32 1, i32 1, i32 1>
+; CHECK-NEXT:    [[TMP5:%.*]] = add i64 [[TMP1]], 1
+; CHECK-NEXT:    [[TMP6:%.*]] = getelementptr i32, ptr [[P]], i64 [[TMP5]]
+; CHECK-NEXT:    [[TMP7:%.*]] = add <4 x i32> [[STRIDED_VEC1]], <i32 2, i32 2, i32 2, i32 2>
+; CHECK-NEXT:    [[TMP8:%.*]] = getelementptr i32, ptr [[TMP6]], i32 -1
+; CHECK-NEXT:    [[TMP9:%.*]] = shufflevector <4 x i32> [[TMP4]], <4 x i32> [[TMP7]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+; CHECK-NEXT:    [[INTERLEAVED_VEC:%.*]] = shufflevector <8 x i32> [[TMP9]], <8 x i32> poison, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
+; CHECK-NEXT:    store <8 x i32> [[INTERLEAVED_VEC]], ptr [[TMP8]], align 4
+; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; CHECK-NEXT:    [[TMP10:%.*]] = icmp eq i64 [[INDEX_NEXT]], 1024
+; CHECK-NEXT:    br i1 [[TMP10]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; CHECK:       middle.block:
+; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 1024, 1024
+; CHECK-NEXT:    br i1 [[CMP_N]], label [[EXIT:%.*]], label [[SCALAR_PH]]
+; CHECK:       scalar.ph:
+; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ 1024, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[NEXTI:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[OFFSET0:%.*]] = shl i64 [[I]], 1
+; CHECK-NEXT:    [[Q0:%.*]] = getelementptr i32, ptr [[P]], i64 [[OFFSET0]]
+; CHECK-NEXT:    [[X0:%.*]] = load i32, ptr [[Q0]], align 4
+; CHECK-NEXT:    [[Y0:%.*]] = add i32 [[X0]], 1
+; CHECK-NEXT:    store i32 [[Y0]], ptr [[Q0]], align 4
+; CHECK-NEXT:    [[OFFSET1:%.*]] = add i64 [[OFFSET0]], 1
+; CHECK-NEXT:    [[Q1:%.*]] = getelementptr i32, ptr [[P]], i64 [[OFFSET1]]
+; CHECK-NEXT:    [[X1:%.*]] = load i32, ptr [[Q1]], align 4
+; CHECK-NEXT:    [[Y1:%.*]] = add i32 [[X1]], 2
+; CHECK-NEXT:    store i32 [[Y1]], ptr [[Q1]], align 4
+; CHECK-NEXT:    [[NEXTI]] = add i64 [[I]], 1
+; CHECK-NEXT:    [[DONE:%.*]] = icmp eq i64 [[NEXTI]], 1024
+; CHECK-NEXT:    br i1 [[DONE]], label [[EXIT]], label [[LOOP]], !llvm.loop [[LOOP3:![0-9]+]]
+; CHECK:       exit:
+; CHECK-NEXT:    ret void
+;
+entry:
+  br label %loop
+loop:
+  %i = phi i64 [0, %entry], [%nexti, %loop]
+
+  %offset0 = shl i64 %i, 1
+  %q0 = getelementptr i32, ptr %p, i64 %offset0
+  %x0 = load i32, ptr %q0
+  %y0 = add i32 %x0, 1
+  store i32 %y0, ptr %q0
+
+  %offset1 = add i64 %offset0, 1
+  %q1 = getelementptr i32, ptr %p, i64 %offset1
+  %x1 = load i32, ptr %q1
+  %y1 = add i32 %x1, 2
+  store i32 %y1, ptr %q1
+
+  %nexti = add i64 %i, 1
+  %done = icmp eq i64 %nexti, 1024
+  br i1 %done, label %exit, label %loop
+exit:
+  ret void
+}
+
+define void @load_store_factor2_i64(ptr %p) {
+; CHECK-LABEL: @load_store_factor2_i64(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[I:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[NEXTI:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[OFFSET0:%.*]] = shl i64 [[I]], 1
+; CHECK-NEXT:    [[Q0:%.*]] = getelementptr i64, ptr [[P:%.*]], i64 [[OFFSET0]]
+; CHECK-NEXT:    [[X0:%.*]] = load i64, ptr [[Q0]], align 4
+; CHECK-NEXT:    [[Y0:%.*]] = add i64 [[X0]], 1
+; CHECK-NEXT:    store i64 [[Y0]], ptr [[Q0]], align 4
+; CHECK-NEXT:    [[OFFSET1:%.*]] = add i64 [[OFFSET0]], 1
+; CHECK-NEXT:    [[Q1:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET1]]
+; CHECK-NEXT:    [[X1:%.*]] = load i64, ptr [[Q1]], align 4
+; CHECK-NEXT:    [[Y1:%.*]] = add i64 [[X1]], 2
+; CHECK-NEXT:    store i64 [[Y1]], ptr [[Q1]], align 4
+; CHECK-NEXT:    [[NEXTI]] = add i64 [[I]], 1
+; CHECK-NEXT:    [[DONE:%.*]] = icmp eq i64 [[NEXTI]], 1024
+; CHECK-NEXT:    br i1 [[DONE]], label [[EXIT:%.*]], label [[LOOP]]
+; CHECK:       exit:
+; CHECK-NEXT:    ret void
+;
+entry:
+  br label %loop
+loop:
+  %i = phi i64 [0, %entry], [%nexti, %loop]
+
+  %offset0 = shl i64 %i, 1
+  %q0 = getelementptr i64, ptr %p, i64 %offset0
+  %x0 = load i64, ptr %q0
+  %y0 = add i64 %x0, 1
+  store i64 %y0, ptr %q0
+
+  %offset1 = add i64 %offset0, 1
+  %q1 = getelementptr i64, ptr %p, i64 %offset1
+  %x1 = load i64, ptr %q1
+  %y1 = add i64 %x1, 2
+  store i64 %y1, ptr %q1
+
+  %nexti = add i64 %i, 1
+  %done = icmp eq i64 %nexti, 1024
+  br i1 %done, label %exit, label %loop
+exit:
+  ret void
+}
+
+define void @load_store_factor3_i32(ptr %p) {
+; CHECK-LABEL: @load_store_factor3_i32(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP1:%.*]] = mul i64 [[TMP0]], 2
+; CHECK-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 1024, [[TMP1]]
+; CHECK-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
+; CHECK:       vector.ph:
+; CHECK-NEXT:    [[TMP2:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP3:%.*]] = mul i64 [[TMP2]], 2
+; CHECK-NEXT:    [[N_MOD_VF:%.*]] = urem i64 1024, [[TMP3]]
+; CHECK-NEXT:    [[N_VEC:%.*]] = sub i64 1024, [[N_MOD_VF]]
+; CHECK-NEXT:    [[TMP4:%.*]] = call <vscale x 2 x i64> @llvm.experimental.stepvector.nxv2i64()
+; CHECK-NEXT:    [[TMP5:%.*]] = add <vscale x 2 x i64> [[TMP4]], zeroinitializer
+; CHECK-NEXT:    [[TMP6:%.*]] = mul <vscale x 2 x i64> [[TMP5]], shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 1, i64 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+; CHECK-NEXT:    [[INDUCTION:%.*]] = add <vscale x 2 x i64> zeroinitializer, [[TMP6]]
+; CHECK-NEXT:    [[TMP7:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP8:%.*]] = mul i64 [[TMP7]], 2
+; CHECK-NEXT:    [[TMP9:%.*]] = mul i64 1, [[TMP8]]
+; CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 2 x i64> poison, i64 [[TMP9]], i64 0
+; CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 2 x i64> [[DOTSPLATINSERT]], <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer
+; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
+; CHECK:       vector.body:
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[VEC_IND:%.*]] = phi <vscale x 2 x i64> [ [[INDUCTION]], [[VECTOR_PH]] ], [ [[VEC_IND_NEXT:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[TMP10:%.*]] = mul <vscale x 2 x i64> [[VEC_IND]], shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 3, i64 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+; CHECK-NEXT:    [[TMP11:%.*]] = getelementptr i32, ptr [[P:%.*]], <vscale x 2 x i64> [[TMP10]]
+; CHECK-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <vscale x 2 x i32> @llvm.masked.gather.nxv2i32.nxv2p0(<vscale x 2 x ptr> [[TMP11]], i32 4, <vscale x 2 x i1> shufflevector (<vscale x 2 x i1> insertelement (<vscale x 2 x i1> poison, i1 true, i64 0), <vscale x 2 x i1> poison, <vscale x 2 x i32> zeroinitializer), <vscale x 2 x i32> poison)
+; CHECK-NEXT:    [[TMP12:%.*]] = add <vscale x 2 x i32> [[WIDE_MASKED_GATHER]], shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> poison, i32 1, i64 0), <vscale x 2 x i32> poison, <vscale x 2 x i32> zeroinitializer)
+; CHECK-NEXT:    call void @llvm.masked.scatter.nxv2i32.nxv2p0(<vscale x 2 x i32> [[TMP12]], <vscale x 2 x ptr> [[TMP11]], i32 4, <vscale x 2 x i1> shufflevector (<vscale x 2 x i1> insertelement (<vscale x 2 x i1> poison, i1 true, i64 0), <vscale x 2 x i1> poison, <vscale x 2 x i32> zeroinitializer))
+; CHECK-NEXT:    [[TMP13:%.*]] = add <vscale x 2 x i64> [[TMP10]], shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 1, i64 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+; CHECK-NEXT:    [[TMP14:%.*]] = getelementptr i32, ptr [[P]], <vscale x 2 x i64> [[TMP13]]
+; CHECK-NEXT:    [[WIDE_MASKED_GATHER1:%.*]] = call <vscale x 2 x i32> @llvm.masked.gather.nxv2i32.nxv2p0(<vscale x 2 x ptr> [[TMP14]], i32 4, <vscale x 2 x i1> shufflevector (<vscale x 2 x i1> insertelement (<vscale x 2 x i1> poison, i1 true, i64 0), <vscale x 2 x i1> poison, <vscale x 2 x i32> zeroinitializer), <vscale x 2 x i32> poison)
+; CHECK-NEXT:    [[TMP15:%.*]] = add <vscale x 2 x i32> [[WIDE_MASKED_GATHER1]], shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> poison, i32 2, i64 0), <vscale x 2 x i32> poison, <vscale x 2 x i32> zeroinitializer)
+; CHECK-NEXT:    call void @llvm.masked.scatter.nxv2i32.nxv2p0(<vscale x 2 x i32> [[TMP15]], <vscale x 2 x ptr> [[TMP14]], i32 4, <vscale x 2 x i1> shufflevector (<vscale x 2 x i1> insertelement (<vscale x 2 x i1> poison, i1 true, i64 0), <vscale x 2 x i1> poison, <vscale x 2 x i32> zeroinitializer))
+; CHECK-NEXT:    [[TMP16:%.*]] = add <vscale x 2 x i64> [[TMP13]], shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 1, i64 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+; CHECK-NEXT:    [[TMP17:%.*]] = getelementptr i32, ptr [[P]], <vscale x 2 x i64> [[TMP16]]
+; CHECK-NEXT:    [[WIDE_MASKED_GATHER2:%.*]] = call <vscale x 2 x i32> @llvm.masked.gather.nxv2i32.nxv2p0(<vscale x 2 x ptr> [[TMP17]], i32 4, <vscale x 2 x i1> shufflevector (<vscale x 2 x i1> insertelement (<vscale x 2 x i1> poison, i1 true, i64 0), <vscale x 2 x i1> poison, <vscale x 2 x i32> zeroinitializer), <vscale x 2 x i32> poison)
+; CHECK-NEXT:    [[TMP18:%.*]] = add <vscale x 2 x i32> [[WIDE_MASKED_GATHER2]], shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> poison, i32 3, i64 0), <vscale x 2 x i32> poison, <vscale x 2 x i32> zeroinitializer)
+; CHECK-NEXT:    call void @llvm.masked.scatter.nxv2i32.nxv2p0(<vscale x 2 x i32> [[TMP18]], <vscale x 2 x ptr> [[TMP17]], i32 4, <vscale x 2 x i1> shufflevector (<vscale x 2 x i1> insertelement (<vscale x 2 x i1> poison, i1 true, i64 0), <vscale x 2 x i1> poison, <vscale x 2 x i32> zeroinitializer))
+; CHECK-NEXT:    [[TMP19:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP20:%.*]] = mul i64 [[TMP19]], 2
+; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], [[TMP20]]
+; CHECK-NEXT:    [[VEC_IND_NEXT]] = add <vscale x 2 x i64> [[VEC_IND]], [[DOTSPLAT]]
+; CHECK-NEXT:    [[TMP21:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; CHECK-NEXT:    br i1 [[TMP21]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP4:![0-9]+]]
+; CHECK:       middle.block:
+; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 1024, [[N_VEC]]
+; CHECK-NEXT:    br i1 [[CMP_N]], label [[EXIT:%.*]], label [[SCALAR_PH]]
+; CHECK:       scalar.ph:
+; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[NEXTI:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[OFFSET0:%.*]] = mul i64 [[I]], 3
+; CHECK-NEXT:    [[Q0:%.*]] = getelementptr i32, ptr [[P]], i64 [[OFFSET0]]
+; CHECK-NEXT:    [[X0:%.*]] = load i32, ptr [[Q0]], align 4
+; CHECK-NEXT:    [[Y0:%.*]] = add i32 [[X0]], 1
+; CHECK-NEXT:    store i32 [[Y0]], ptr [[Q0]], align 4
+; CHECK-NEXT:    [[OFFSET1:%.*]] = add i64 [[OFFSET0]], 1
+; CHECK-NEXT:    [[Q1:%.*]] = getelementptr i32, ptr [[P]], i64 [[OFFSET1]]
+; CHECK-NEXT:    [[X1:%.*]] = load i32, ptr [[Q1]], align 4
+; CHECK-NEXT:    [[Y1:%.*]] = add i32 [[X1]], 2
+; CHECK-NEXT:    store i32 [[Y1]], ptr [[Q1]], align 4
+; CHECK-NEXT:    [[OFFSET2:%.*]] = add i64 [[OFFSET1]], 1
+; CHECK-NEXT:    [[Q2:%.*]] = getelementptr i32, ptr [[P]], i64 [[OFFSET2]]
+; CHECK-NEXT:    [[X2:%.*]] = load i32, ptr [[Q2]], align 4
+; CHECK-NEXT:    [[Y2:%.*]] = add i32 [[X2]], 3
+; CHECK-NEXT:    store i32 [[Y2]], ptr [[Q2]], align 4
+; CHECK-NEXT:    [[NEXTI]] = add i64 [[I]], 1
+; CHECK-NEXT:    [[DONE:%.*]] = icmp eq i64 [[NEXTI]], 1024
+; CHECK-NEXT:    br i1 [[DONE]], label [[EXIT]], label [[LOOP]], !llvm.loop [[LOOP5:![0-9]+]]
+; CHECK:       exit:
+; CHECK-NEXT:    ret void
+;
+entry:
+  br label %loop
+loop:
+  %i = phi i64 [0, %entry], [%nexti, %loop]
+
+  %offset0 = mul i64 %i, 3
+  %q0 = getelementptr i32, ptr %p, i64 %offset0
+  %x0 = load i32, ptr %q0
+  %y0 = add i32 %x0, 1
+  store i32 %y0, ptr %q0
+
+  %offset1 = add i64 %offset0, 1
+  %q1 = getelementptr i32, ptr %p, i64 %offset1
+  %x1 = load i32, ptr %q1
+  %y1 = add i32 %x1, 2
+  store i32 %y1, ptr %q1
+
+  %offset2 = add i64 %offset1, 1
+  %q2 = getelementptr i32, ptr %p, i64 %offset2
+  %x2 = load i32, ptr %q2
+  %y2 = add i32 %x2, 3
+  store i32 %y2, ptr %q2
+
+  %nexti = add i64 %i, 1
+  %done = icmp eq i64 %nexti, 1024
+  br i1 %done, label %exit, label %loop
+exit:
+  ret void
+}
+
+define void @load_store_factor3_i64(ptr %p) {
+; CHECK-LABEL: @load_store_factor3_i64(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[I:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[NEXTI:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[OFFSET0:%.*]] = mul i64 [[I]], 3
+; CHECK-NEXT:    [[Q0:%.*]] = getelementptr i64, ptr [[P:%.*]], i64 [[OFFSET0]]
+; CHECK-NEXT:    [[X0:%.*]] = load i64, ptr [[Q0]], align 4
+; CHECK-NEXT:    [[Y0:%.*]] = add i64 [[X0]], 1
+; CHECK-NEXT:    store i64 [[Y0]], ptr [[Q0]], align 4
+; CHECK-NEXT:    [[OFFSET1:%.*]] = add i64 [[OFFSET0]], 1
+; CHECK-NEXT:    [[Q1:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET1]]
+; CHECK-NEXT:    [[X1:%.*]] = load i64, ptr [[Q1]], align 4
+; CHECK-NEXT:    [[Y1:%.*]] = add i64 [[X1]], 2
+; CHECK-NEXT:    store i64 [[Y1]], ptr [[Q1]], align 4
+; CHECK-NEXT:    [[OFFSET2:%.*]] = add i64 [[OFFSET1]], 1
+; CHECK-NEXT:    [[Q2:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET2]]
+; CHECK-NEXT:    [[X2:%.*]] = load i64, ptr [[Q2]], align 4
+; CHECK-NEXT:    [[Y2:%.*]] = add i64 [[X2]], 3
+; CHECK-NEXT:    store i64 [[Y2]], ptr [[Q2]], align 4
+; CHECK-NEXT:    [[NEXTI]] = add i64 [[I]], 1
+; CHECK-NEXT:    [[DONE:%.*]] = icmp eq i64 [[NEXTI]], 1024
+; CHECK-NEXT:    br i1 [[DONE]], label [[EXIT:%.*]], label [[LOOP]]
+; CHECK:       exit:
+; CHECK-NEXT:    ret void
+;
+entry:
+  br label %loop
+loop:
+  %i = phi i64 [0, %entry], [%nexti, %loop]
+
+  %offset0 = mul i64 %i, 3
+  %q0 = getelementptr i64, ptr %p, i64 %offset0
+  %x0 = load i64, ptr %q0
+  %y0 = add i64 %x0, 1
+  store i64 %y0, ptr %q0
+
+  %offset1 = add i64 %offset0, 1
+  %q1 = getelementptr i64, ptr %p, i64 %offset1
+  %x1 = load i64, ptr %q1
+  %y1 = add i64 %x1, 2
+  store i64 %y1, ptr %q1
+
+  %offset2 = add i64 %offset1, 1
+  %q2 = getelementptr i64, ptr %p, i64 %offset2
+  %x2 = load i64, ptr %q2
+  %y2 = add i64 %x2, 3
+  store i64 %y2, ptr %q2
+
+  %nexti = add i64 %i, 1
+  %done = icmp eq i64 %nexti, 1024
+  br i1 %done, label %exit, label %loop
+exit:
+  ret void
+}
+
+define void @load_store_factor8(ptr %p) {
+; CHECK-LABEL: @load_store_factor8(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[I:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[NEXTI:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[OFFSET0:%.*]] = shl i64 [[I]], 3
+; CHECK-NEXT:    [[Q0:%.*]] = getelementptr i64, ptr [[P:%.*]], i64 [[OFFSET0]]
+; CHECK-NEXT:    [[X0:%.*]] = load i64, ptr [[Q0]], align 4
+; CHECK-NEXT:    [[Y0:%.*]] = add i64 [[X0]], 1
+; CHECK-NEXT:    store i64 [[Y0]], ptr [[Q0]], align 4
+; CHECK-NEXT:    [[OFFSET1:%.*]] = add i64 [[OFFSET0]], 1
+; CHECK-NEXT:    [[Q1:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET1]]
+; CHECK-NEXT:    [[X1:%.*]] = load i64, ptr [[Q1]], align 4
+; CHECK-NEXT:    [[Y1:%.*]] = add i64 [[X1]], 2
+; CHECK-NEXT:    store i64 [[Y1]], ptr [[Q1]], align 4
+; CHECK-NEXT:    [[OFFSET2:%.*]] = add i64 [[OFFSET1]], 1
+; CHECK-NEXT:    [[Q2:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET2]]
+; CHECK-NEXT:    [[X2:%.*]] = load i64, ptr [[Q2]], align 4
+; CHECK-NEXT:    [[Y2:%.*]] = add i64 [[X2]], 3
+; CHECK-NEXT:    store i64 [[Y2]], ptr [[Q2]], align 4
+; CHECK-NEXT:    [[OFFSET3:%.*]] = add i64 [[OFFSET2]], 1
+; CHECK-NEXT:    [[Q3:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET3]]
+; CHECK-NEXT:    [[X3:%.*]] = load i64, ptr [[Q3]], align 4
+; CHECK-NEXT:    [[Y3:%.*]] = add i64 [[X3]], 4
+; CHECK-NEXT:    store i64 [[Y3]], ptr [[Q3]], align 4
+; CHECK-NEXT:    [[OFFSET4:%.*]] = add i64 [[OFFSET3]], 1
+; CHECK-NEXT:    [[Q4:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET4]]
+; CHECK-NEXT:    [[X4:%.*]] = load i64, ptr [[Q4]], align 4
+; CHECK-NEXT:    [[Y4:%.*]] = add i64 [[X4]], 5
+; CHECK-NEXT:    store i64 [[Y4]], ptr [[Q4]], align 4
+; CHECK-NEXT:    [[OFFSET5:%.*]] = add i64 [[OFFSET4]], 1
+; CHECK-NEXT:    [[Q5:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET5]]
+; CHECK-NEXT:    [[X5:%.*]] = load i64, ptr [[Q5]], align 4
+; CHECK-NEXT:    [[Y5:%.*]] = add i64 [[X5]], 6
+; CHECK-NEXT:    store i64 [[Y5]], ptr [[Q5]], align 4
+; CHECK-NEXT:    [[OFFSET6:%.*]] = add i64 [[OFFSET5]], 1
+; CHECK-NEXT:    [[Q6:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET6]]
+; CHECK-NEXT:    [[X6:%.*]] = load i64, ptr [[Q6]], align 4
+; CHECK-NEXT:    [[Y6:%.*]] = add i64 [[X6]], 7
+; CHECK-NEXT:    store i64 [[Y6]], ptr [[Q6]], align 4
+; CHECK-NEXT:    [[OFFSET7:%.*]] = add i64 [[OFFSET6]], 1
+; CHECK-NEXT:    [[Q7:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET7]]
+; CHECK-NEXT:    [[X7:%.*]] = load i64, ptr [[Q7]], align 4
+; CHECK-NEXT:    [[Y7:%.*]] = add i64 [[X7]], 8
+; CHECK-NEXT:    store i64 [[Y7]], ptr [[Q7]], align 4
+; CHECK-NEXT:    [[NEXTI]] = add i64 [[I]], 1
+; CHECK-NEXT:    [[DONE:%.*]] = icmp eq i64 [[NEXTI]], 1024
+; CHECK-NEXT:    br i1 [[DONE]], label [[EXIT:%.*]], label [[LOOP]]
+; CHECK:       exit:
+; CHECK-NEXT:    ret void
+;
+entry:
+  br label %loop
+loop:
+  %i = phi i64 [0, %entry], [%nexti, %loop]
+
+  %offset0 = shl i64 %i, 3
+  %q0 = getelementptr i64, ptr %p, i64 %offset0
+  %x0 = load i64, ptr %q0
+  %y0 = add i64 %x0, 1
+  store i64 %y0, ptr %q0
+
+  %offset1 = add i64 %offset0, 1
+  %q1 = getelementptr i64, ptr %p, i64 %offset1
+  %x1 = load i64, ptr %q1
+  %y1 = add i64 %x1, 2
+  store i64 %y1, ptr %q1
+
+  %offset2 = add i64 %offset1, 1
+  %q2 = getelementptr i64, ptr %p, i64 %offset2
+  %x2 = load i64, ptr %q2
+  %y2 = add i64 %x2, 3
+  store i64 %y2, ptr %q2
+
+  %offset3 = add i64 %offset2, 1
+  %q3 = getelementptr i64, ptr %p, i64 %offset3
+  %x3 = load i64, ptr %q3
+  %y3 = add i64 %x3, 4
+  store i64 %y3, ptr %q3
+
+  %offset4 = add i64 %offset3, 1
+  %q4 = getelementptr i64, ptr %p, i64 %offset4
+  %x4 = load i64, ptr %q4
+  %y4 = add i64 %x4, 5
+  store i64 %y4, ptr %q4
+
+  %offset5 = add i64 %offset4, 1
+  %q5 = getelementptr i64, ptr %p, i64 %offset5
+  %x5 = load i64, ptr %q5
+  %y5 = add i64 %x5, 6
+  store i64 %y5, ptr %q5
+
+  %offset6 = add i64 %offset5, 1
+  %q6 = getelementptr i64, ptr %p, i64 %offset6
+  %x6 = load i64, ptr %q6
+  %y6 = add i64 %x6, 7
+  store i64 %y6, ptr %q6
+
+  %offset7 = add i64 %offset6, 1
+  %q7 = getelementptr i64, ptr %p, i64 %offset7
+  %x7 = load i64, ptr %q7
+  %y7 = add i64 %x7, 8
+  store i64 %y7, ptr %q7
+
+  %nexti = add i64 %i, 1
+  %done = icmp eq i64 %nexti, 1024
+  br i1 %done, label %exit, label %loop
+exit:
+  ret void
+}
+
+define void @combine_load_factor2_i32(ptr %p) {
+; CHECK-LABEL: @combine_load_factor2_i32(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
+; CHECK:       vector.ph:
+; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
+; CHECK:       vector.body:
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[VEC_IND:%.*]] = phi <4 x i64> [ <i64 0, i64 1, i64 2, i64 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[STEP_ADD:%.*]] = add <4 x i64> [[VEC_IND]], <i64 4, i64 4, i64 4, i64 4>
+; CHECK-NEXT:    [[TMP0:%.*]] = shl <4 x i64> [[VEC_IND]], <i64 1, i64 1, i64 1, i64 1>
+; CHECK-NEXT:    [[TMP1:%.*]] = shl <4 x i64> [[STEP_ADD]], <i64 1, i64 1, i64 1, i64 1>
+; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr i32, ptr [[P:%.*]], <4 x i64> [[TMP0]]
+; CHECK-NEXT:    [[TMP3:%.*]] = getelementptr i32, ptr [[P]], <4 x i64> [[TMP1]]
+; CHECK-NEXT:    [[TMP4:%.*]] = extractelement <4 x ptr> [[TMP2]], i32 0
+; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr i32, ptr [[TMP4]], i32 0
+; CHECK-NEXT:    [[TMP6:%.*]] = extractelement <4 x ptr> [[TMP3]], i32 0
+; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr i32, ptr [[TMP6]], i32 0
+; CHECK-NEXT:    [[WIDE_VEC:%.*]] = load <8 x i32>, ptr [[TMP5]], align 4
+; CHECK-NEXT:    [[WIDE_VEC2:%.*]] = load <8 x i32>, ptr [[TMP7]], align 4
+; CHECK-NEXT:    [[STRIDED_VEC:%.*]] = shufflevector <8 x i32> [[WIDE_VEC]], <8 x i32> poison, <4 x i32> <i32 0, i32 2, i32 4, i32 6>
+; CHECK-NEXT:    [[STRIDED_VEC3:%.*]] = shufflevector <8 x i32> [[WIDE_VEC2]], <8 x i32> poison, <4 x i32> <i32 0, i32 2, i32 4, i32 6>
+; CHECK-NEXT:    [[STRIDED_VEC4:%.*]] = shufflevector <8 x i32> [[WIDE_VEC]], <8 x i32> poison, <4 x i32> <i32 1, i32 3, i32 5, i32 7>
+; CHECK-NEXT:    [[STRIDED_VEC5:%.*]] = shufflevector <8 x i32> [[WIDE_VEC2]], <8 x i32> poison, <4 x i32> <i32 1, i32 3, i32 5, i32 7>
+; CHECK-NEXT:    [[TMP8:%.*]] = add <4 x i32> [[STRIDED_VEC]], [[STRIDED_VEC4]]
+; CHECK-NEXT:    [[TMP9:%.*]] = add <4 x i32> [[STRIDED_VEC3]], [[STRIDED_VEC5]]
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4i32.v4p0(<4 x i32> [[TMP8]], <4 x ptr> [[TMP2]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>)
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4i32.v4p0(<4 x i32> [[TMP9]], <4 x ptr> [[TMP3]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>)
+; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 8
+; CHECK-NEXT:    [[VEC_IND_NEXT]] = add <4 x i64> [[STEP_ADD]], <i64 4, i64 4, i64 4, i64 4>
+; CHECK-NEXT:    [[TMP10:%.*]] = icmp eq i64 [[INDEX_NEXT]], 1024
+; CHECK-NEXT:    br i1 [[TMP10]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP6:![0-9]+]]
+; CHECK:       middle.block:
+; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 1024, 1024
+; CHECK-NEXT:    br i1 [[CMP_N]], label [[EXIT:%.*]], label [[SCALAR_PH]]
+; CHECK:       scalar.ph:
+; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ 1024, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[NEXTI:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[OFFSET0:%.*]] = shl i64 [[I]], 1
+; CHECK-NEXT:    [[Q0:%.*]] = getelementptr i32, ptr [[P]], i64 [[OFFSET0]]
+; CHECK-NEXT:    [[X0:%.*]] = load i32, ptr [[Q0]], align 4
+; CHECK-NEXT:    [[OFFSET1:%.*]] = add i64 [[OFFSET0]], 1
+; CHECK-NEXT:    [[Q1:%.*]] = getelementptr i32, ptr [[P]], i64 [[OFFSET1]]
+; CHECK-NEXT:    [[X1:%.*]] = load i32, ptr [[Q1]], align 4
+; CHECK-NEXT:    [[RES:%.*]] = add i32 [[X0]], [[X1]]
+; CHECK-NEXT:    store i32 [[RES]], ptr [[Q0]], align 4
+; CHECK-NEXT:    [[NEXTI]] = add i64 [[I]], 1
+; CHECK-NEXT:    [[DONE:%.*]] = icmp eq i64 [[NEXTI]], 1024
+; CHECK-NEXT:    br i1 [[DONE]], label [[EXIT]], label [[LOOP]], !llvm.loop [[LOOP7:![0-9]+]]
+; CHECK:       exit:
+; CHECK-NEXT:    ret void
+;
+entry:
+  br label %loop
+loop:
+  %i = phi i64 [0, %entry], [%nexti, %loop]
+
+  %offset0 = shl i64 %i, 1
+  %q0 = getelementptr i32, ptr %p, i64 %offset0
+  %x0 = load i32, ptr %q0
+
+  %offset1 = add i64 %offset0, 1
+  %q1 = getelementptr i32, ptr %p, i64 %offset1
+  %x1 = load i32, ptr %q1
+
+  %res = add i32 %x0, %x1
+
+  store i32 %res, ptr %q0
+
+  %nexti = add i64 %i, 1
+  %done = icmp eq i64 %nexti, 1024
+  br i1 %done, label %exit, label %loop
+exit:
+  ret void
+}
+
+define void @combine_load_factor2_i64(ptr %p) {
+; CHECK-LABEL: @combine_load_factor2_i64(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[I:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[NEXTI:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[OFFSET0:%.*]] = shl i64 [[I]], 1
+; CHECK-NEXT:    [[Q0:%.*]] = getelementptr i64, ptr [[P:%.*]], i64 [[OFFSET0]]
+; CHECK-NEXT:    [[X0:%.*]] = load i64, ptr [[Q0]], align 4
+; CHECK-NEXT:    [[OFFSET1:%.*]] = add i64 [[OFFSET0]], 1
+; CHECK-NEXT:    [[Q1:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET1]]
+; CHECK-NEXT:    [[X1:%.*]] = load i64, ptr [[Q1]], align 4
+; CHECK-NEXT:    [[RES:%.*]] = add i64 [[X0]], [[X1]]
+; CHECK-NEXT:    store i64 [[RES]], ptr [[Q0]], align 4
+; CHECK-NEXT:    [[NEXTI]] = add i64 [[I]], 1
+; CHECK-NEXT:    [[DONE:%.*]] = icmp eq i64 [[NEXTI]], 1024
+; CHECK-NEXT:    br i1 [[DONE]], label [[EXIT:%.*]], label [[LOOP]]
+; CHECK:       exit:
+; CHECK-NEXT:    ret void
+;
+entry:
+  br label %loop
+loop:
+  %i = phi i64 [0, %entry], [%nexti, %loop]
+
+  %offset0 = shl i64 %i, 1
+  %q0 = getelementptr i64, ptr %p, i64 %offset0
+  %x0 = load i64, ptr %q0
+
+  %offset1 = add i64 %offset0, 1
+  %q1 = getelementptr i64, ptr %p, i64 %offset1
+  %x1 = load i64, ptr %q1
+
+  %res = add i64 %x0, %x1
+
+  store i64 %res, ptr %q0
+
+  %nexti = add i64 %i, 1
+  %done = icmp eq i64 %nexti, 1024
+  br i1 %done, label %exit, label %loop
+exit:
+  ret void
+}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/interleaved-cost.ll b/llvm/test/Transforms/LoopVectorize/RISCV/interleaved-cost.ll
new file mode 100644
index 0000000000000..327d622ce4301
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/interleaved-cost.ll
@@ -0,0 +1,101 @@
+; REQUIRES: asserts
+; RUN: opt -passes=loop-vectorize -mtriple=riscv64 -mattr=+v -force-vector-width=2 -debug-only=loop-vectorize -disable-output < %s 2>&1 | FileCheck %s --check-prefix=VF_2
+; RUN: opt -passes=loop-vectorize -mtriple=riscv64 -mattr=+v -force-vector-width=4 -debug-only=loop-vectorize -disable-output < %s 2>&1 | FileCheck %s --check-prefix=VF_4
+; RUN: opt -passes=loop-vectorize -mtriple=riscv64 -mattr=+v -force-vector-width=8 -debug-only=loop-vectorize -disable-output < %s 2>&1 | FileCheck %s --check-prefix=VF_8
+; RUN: opt -passes=loop-vectorize -mtriple=riscv64 -mattr=+v -force-vector-width=16 -debug-only=loop-vectorize -disable-output < %s 2>&1 | FileCheck %s --check-prefix=VF_16
+
+%i8.2 = type {i8, i8}
+define void @i8_factor_2(ptr %data, i64 %n) {
+entry:
+  br label %for.body
+; VF_2-LABEL: Checking a loop in 'i8_factor_2'
+; VF_2:       Found an estimated cost of 3 for VF 2 For instruction:   %l0 = load i8, ptr %p0, align 1
+; VF_2-NEXT:  Found an estimated cost of 0 for VF 2 For instruction:   %l1 = load i8, ptr %p1, align 1
+; VF_2:       Found an estimated cost of 0 for VF 2 For instruction:   store i8 %a0, ptr %p0, align 1
+; VF_2-NEXT:  Found an estimated cost of 3 for VF 2 For instruction:   store i8 %a1, ptr %p1, align 1
+; VF_4-LABEL: Checking a loop in 'i8_factor_2'
+; VF_4:       Found an estimated cost of 3 for VF 4 For instruction:   %l0 = load i8, ptr %p0, align 1
+; VF_4-NEXT:  Found an estimated cost of 0 for VF 4 For instruction:   %l1 = load i8, ptr %p1, align 1
+; VF_4:       Found an estimated cost of 0 for VF 4 For instruction:   store i8 %a0, ptr %p0, align 1
+; VF_4-NEXT:  Found an estimated cost of 3 for VF 4 For instruction:   store i8 %a1, ptr %p1, align 1
+; VF_8-LABEL: Checking a loop in 'i8_factor_2'
+; VF_8:       Found an estimated cost of 3 for VF 8 For instruction:   %l0 = load i8, ptr %p0, align 1
+; VF_8-NEXT:  Found an estimated cost of 0 for VF 8 For instruction:   %l1 = load i8, ptr %p1, align 1
+; VF_8:       Found an estimated cost of 0 for VF 8 For instruction:   store i8 %a0, ptr %p0, align 1
+; VF_8-NEXT:  Found an estimated cost of 3 for VF 8 For instruction:   store i8 %a1, ptr %p1, align 1
+; VF_16-LABEL: Checking a loop in 'i8_factor_2'
+; VF_16:       Found an estimated cost of 3 for VF 16 For instruction:   %l0 = load i8, ptr %p0, align 1
+; VF_16-NEXT:  Found an estimated cost of 0 for VF 16 For instruction:   %l1 = load i8, ptr %p1, align 1
+; VF_16:       Found an estimated cost of 0 for VF 16 For instruction:   store i8 %a0, ptr %p0, align 1
+; VF_16-NEXT:  Found an estimated cost of 5 for VF 16 For instruction:   store i8 %a1, ptr %p1, align 1
+for.body:
+  %i = phi i64 [ 0, %entry ], [ %i.next, %for.body ]
+  %p0 = getelementptr inbounds %i8.2, ptr %data, i64 %i, i32 0
+  %p1 = getelementptr inbounds %i8.2, ptr %data, i64 %i, i32 1
+  %l0 = load i8, ptr %p0, align 1
+  %l1 = load i8, ptr %p1, align 1
+  %a0 = add i8 %l0, 1
+  %a1 = add i8 %l1, 2
+  store i8 %a0, ptr %p0, align 1
+  store i8 %a1, ptr %p1, align 1
+  %i.next = add nuw nsw i64 %i, 1
+  %cond = icmp slt i64 %i.next, %n
+  br i1 %cond, label %for.body, label %for.end
+
+for.end:
+  ret void
+}
+
+%i8.3 = type {i8, i8, i8}
+define void @i8_factor_3(ptr %data, i64 %n) {
+entry:
+  br label %for.body
+; VF_2-LABEL: Checking a loop in 'i8_factor_3'
+; VF_2:       Found an estimated cost of 6 for VF 2 For instruction:   %l0 = load i8, ptr %p0, align 1
+; VF_2-NEXT:  Found an estimated cost of 0 for VF 2 For instruction:   %l1 = load i8, ptr %p1, align 1
+; VF_2-NEXT:  Found an estimated cost of 0 for VF 2 For instruction:   %l2 = load i8, ptr %p2, align 1
+; VF_2:       Found an estimated cost of 0 for VF 2 For instruction:   store i8 %a0, ptr %p0, align 1
+; VF_2:       Found an estimated cost of 0 for VF 2 For instruction:   store i8 %a1, ptr %p1, align 1
+; VF_2-NEXT:  Found an estimated cost of 6 for VF 2 For instruction:   store i8 %a2, ptr %p2, align 1
+; VF_4-LABEL: Checking a loop in 'i8_factor_3'
+; VF_4:       Found an estimated cost of 12 for VF 4 For instruction:   %l0 = load i8, ptr %p0, align 1
+; VF_4-NEXT:  Found an estimated cost of 0 for VF 4 For instruction:   %l1 = load i8, ptr %p1, align 1
+; VF_4-NEXT:  Found an estimated cost of 0 for VF 4 For instruction:   %l2 = load i8, ptr %p2, align 1
+; VF_4:       Found an estimated cost of 0 for VF 4 For instruction:   store i8 %a0, ptr %p0, align 1
+; VF_4:       Found an estimated cost of 0 for VF 4 For instruction:   store i8 %a1, ptr %p1, align 1
+; VF_4-NEXT:  Found an estimated cost of 12 for VF 4 For instruction:   store i8 %a2, ptr %p2, align 1
+; VF_8-LABEL: Checking a loop in 'i8_factor_3'
+; VF_8:       Found an estimated cost of 24 for VF 8 For instruction:   %l0 = load i8, ptr %p0, align 1
+; VF_8-NEXT:  Found an estimated cost of 0 for VF 8 For instruction:   %l1 = load i8, ptr %p1, align 1
+; VF_8-NEXT:  Found an estimated cost of 0 for VF 8 For instruction:   %l2 = load i8, ptr %p2, align 1
+; VF_8:       Found an estimated cost of 0 for VF 8 For instruction:   store i8 %a0, ptr %p0, align 1
+; VF_8:       Found an estimated cost of 0 for VF 8 For instruction:   store i8 %a1, ptr %p1, align 1
+; VF_8-NEXT:  Found an estimated cost of 24 for VF 8 For instruction:   store i8 %a2, ptr %p2, align 1
+; VF_16-LABEL: Checking a loop in 'i8_factor_3'
+; VF_16:       Found an estimated cost of 48 for VF 16 For instruction:   %l0 = load i8, ptr %p0, align 1
+; VF_16-NEXT:  Found an estimated cost of 0 for VF 16 For instruction:   %l1 = load i8, ptr %p1, align 1
+; VF_16-NEXT:  Found an estimated cost of 0 for VF 16 For instruction:   %l2 = load i8, ptr %p2, align 1
+; VF_16:       Found an estimated cost of 0 for VF 16 For instruction:   store i8 %a0, ptr %p0, align 1
+; VF_16:       Found an estimated cost of 0 for VF 16 For instruction:   store i8 %a1, ptr %p1, align 1
+; VF_16-NEXT:  Found an estimated cost of 48 for VF 16 For instruction:   store i8 %a2, ptr %p2, align 1
+for.body:
+  %i = phi i64 [ 0, %entry ], [ %i.next, %for.body ]
+  %p0 = getelementptr inbounds %i8.3, ptr %data, i64 %i, i32 0
+  %p1 = getelementptr inbounds %i8.3, ptr %data, i64 %i, i32 1
+  %p2 = getelementptr inbounds %i8.3, ptr %data, i64 %i, i32 2
+  %l0 = load i8, ptr %p0, align 1
+  %l1 = load i8, ptr %p1, align 1
+  %l2 = load i8, ptr %p2, align 1
+  %a0 = add i8 %l0, 1
+  %a1 = add i8 %l1, 2
+  %a2 = add i8 %l2, 3
+  store i8 %a0, ptr %p0, align 1
+  store i8 %a1, ptr %p1, align 1
+  store i8 %a2, ptr %p2, align 1
+  %i.next = add nuw nsw i64 %i, 1
+  %cond = icmp slt i64 %i.next, %n
+  br i1 %cond, label %for.body, label %for.end
+
+for.end:
+  ret void
+}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/ordered-reduction.ll b/llvm/test/Transforms/LoopVectorize/RISCV/ordered-reduction.ll
index 2e6a01e2fb81f..dfe52eaf13bb4 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/ordered-reduction.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/ordered-reduction.ll
@@ -1,5 +1,5 @@
 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -passes=loop-vectorize -hints-allow-reordering=false -S < %s | FileCheck %s --check-prefix=CHECK-NOT-VECTORIZED
+; RUN: opt -passes=loop-vectorize -hints-allow-reordering=false -S < %s | FileCheck %s --check-prefix=CHECK-ORDERED
 ; RUN: opt -passes=loop-vectorize -S < %s | FileCheck %s --check-prefix=CHECK-UNORDERED
 ; RUN: opt -passes=loop-vectorize -force-ordered-reductions=false -hints-allow-reordering=false -S < %s | FileCheck %s --check-prefix=CHECK-NOT-VECTORIZED
 ; RUN: opt -passes=loop-vectorize -force-ordered-reductions=true -hints-allow-reordering=false -S < %s | FileCheck %s --check-prefix=CHECK-ORDERED
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/zvl32b.ll b/llvm/test/Transforms/LoopVectorize/RISCV/zvl32b.ll
index 4c89588af13ec..0e2f916ac08b6 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/zvl32b.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/zvl32b.ll
@@ -14,8 +14,8 @@ define void @vector_add_i16(ptr noalias nocapture %a, i16 %v, i64 %n) {
 ; CHECK:       vector.ph:
 ; CHECK-NEXT:    [[BROADCAST_SPLATINSERT:%.*]] = insertelement <2 x i16> poison, i16 [[V:%.*]], i64 0
 ; CHECK-NEXT:    [[BROADCAST_SPLAT:%.*]] = shufflevector <2 x i16> [[BROADCAST_SPLATINSERT]], <2 x i16> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[BROADCAST_SPLATINSERT3:%.*]] = insertelement <2 x i16> poison, i16 [[V]], i64 0
-; CHECK-NEXT:    [[BROADCAST_SPLAT4:%.*]] = shufflevector <2 x i16> [[BROADCAST_SPLATINSERT3]], <2 x i16> poison, <2 x i32> zeroinitializer
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT4:%.*]] = insertelement <2 x i16> poison, i16 [[V]], i64 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT5:%.*]] = shufflevector <2 x i16> [[BROADCAST_SPLATINSERT4]], <2 x i16> poison, <2 x i32> zeroinitializer
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
 ; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
@@ -23,21 +23,26 @@ define void @vector_add_i16(ptr noalias nocapture %a, i16 %v, i64 %n) {
 ; CHECK-NEXT:    [[STEP_ADD:%.*]] = add <2 x i64> [[VEC_IND]], <i64 2, i64 2>
 ; CHECK-NEXT:    [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[A:%.*]], <2 x i64> [[VEC_IND]]
 ; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr inbounds i32, ptr [[A]], <2 x i64> [[STEP_ADD]]
-; CHECK-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <2 x i16> @llvm.masked.gather.v2i16.v2p0(<2 x ptr> [[TMP0]], i32 2, <2 x i1> <i1 true, i1 true>, <2 x i16> poison)
-; CHECK-NEXT:    [[WIDE_MASKED_GATHER2:%.*]] = call <2 x i16> @llvm.masked.gather.v2i16.v2p0(<2 x ptr> [[TMP1]], i32 2, <2 x i1> <i1 true, i1 true>, <2 x i16> poison)
-; CHECK-NEXT:    [[TMP2:%.*]] = add <2 x i16> [[WIDE_MASKED_GATHER]], [[BROADCAST_SPLAT]]
-; CHECK-NEXT:    [[TMP3:%.*]] = add <2 x i16> [[WIDE_MASKED_GATHER2]], [[BROADCAST_SPLAT4]]
-; CHECK-NEXT:    call void @llvm.masked.scatter.v2i16.v2p0(<2 x i16> [[TMP2]], <2 x ptr> [[TMP0]], i32 2, <2 x i1> <i1 true, i1 true>)
-; CHECK-NEXT:    call void @llvm.masked.scatter.v2i16.v2p0(<2 x i16> [[TMP3]], <2 x ptr> [[TMP1]], i32 2, <2 x i1> <i1 true, i1 true>)
+; CHECK-NEXT:    [[TMP2:%.*]] = extractelement <2 x ptr> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP3:%.*]] = getelementptr i16, ptr [[TMP2]], i32 0
+; CHECK-NEXT:    [[TMP4:%.*]] = extractelement <2 x ptr> [[TMP1]], i32 0
+; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr i16, ptr [[TMP4]], i32 0
+; CHECK-NEXT:    [[WIDE_VEC:%.*]] = load <4 x i16>, ptr [[TMP3]], align 2
+; CHECK-NEXT:    [[WIDE_VEC2:%.*]] = load <4 x i16>, ptr [[TMP5]], align 2
+; CHECK-NEXT:    [[STRIDED_VEC:%.*]] = shufflevector <4 x i16> [[WIDE_VEC]], <4 x i16> poison, <2 x i32> <i32 0, i32 2>
+; CHECK-NEXT:    [[STRIDED_VEC3:%.*]] = shufflevector <4 x i16> [[WIDE_VEC2]], <4 x i16> poison, <2 x i32> <i32 0, i32 2>
+; CHECK-NEXT:    [[TMP6:%.*]] = add <2 x i16> [[STRIDED_VEC]], [[BROADCAST_SPLAT]]
+; CHECK-NEXT:    [[TMP7:%.*]] = add <2 x i16> [[STRIDED_VEC3]], [[BROADCAST_SPLAT5]]
+; CHECK-NEXT:    call void @llvm.masked.scatter.v2i16.v2p0(<2 x i16> [[TMP6]], <2 x ptr> [[TMP0]], i32 2, <2 x i1> <i1 true, i1 true>)
+; CHECK-NEXT:    call void @llvm.masked.scatter.v2i16.v2p0(<2 x i16> [[TMP7]], <2 x ptr> [[TMP1]], i32 2, <2 x i1> <i1 true, i1 true>)
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
 ; CHECK-NEXT:    [[VEC_IND_NEXT]] = add <2 x i64> [[STEP_ADD]], <i64 2, i64 2>
-; CHECK-NEXT:    [[TMP4:%.*]] = icmp eq i64 [[INDEX_NEXT]], 1024
-; CHECK-NEXT:    br i1 [[TMP4]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; CHECK-NEXT:    [[TMP8:%.*]] = icmp eq i64 [[INDEX_NEXT]], 1020
+; CHECK-NEXT:    br i1 [[TMP8]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
 ; CHECK:       middle.block:
-; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 1024, 1024
-; CHECK-NEXT:    br i1 [[CMP_N]], label [[FOR_END:%.*]], label [[SCALAR_PH]]
+; CHECK-NEXT:    br label [[SCALAR_PH]]
 ; CHECK:       scalar.ph:
-; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ 1024, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ 1020, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[IV_NEXT:%.*]], [[FOR_BODY]] ]
@@ -47,7 +52,7 @@ define void @vector_add_i16(ptr noalias nocapture %a, i16 %v, i64 %n) {
 ; CHECK-NEXT:    store i16 [[ADD]], ptr [[ARRAYIDX]], align 2
 ; CHECK-NEXT:    [[IV_NEXT]] = add nuw nsw i64 [[IV]], 1
 ; CHECK-NEXT:    [[EXITCOND_NOT:%.*]] = icmp eq i64 [[IV_NEXT]], 1024
-; CHECK-NEXT:    br i1 [[EXITCOND_NOT]], label [[FOR_END]], label [[FOR_BODY]], !llvm.loop [[LOOP2:![0-9]+]]
+; CHECK-NEXT:    br i1 [[EXITCOND_NOT]], label [[FOR_END:%.*]], label [[FOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
 ; CHECK:       for.end:
 ; CHECK-NEXT:    ret void
 ;
diff --git a/llvm/test/Transforms/LoopVectorize/X86/consecutive-ptr-uniforms.ll b/llvm/test/Transforms/LoopVectorize/X86/consecutive-ptr-uniforms.ll
index 7ca6b38827226..51f4b01196220 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/consecutive-ptr-uniforms.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/consecutive-ptr-uniforms.ll
@@ -87,23 +87,23 @@ attributes #0 = { "target-cpu"="knl" }
 ; FORCE:       vector.ph:
 ; FORCE-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; FORCE:       vector.body:
-; FORCE-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_LOAD_CONTINUE4:%.*]] ]
-; FORCE-NEXT:    [[VEC_IND:%.*]] = phi <2 x i32> [ <i32 0, i32 1>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT:%.*]], [[PRED_LOAD_CONTINUE4]] ]
+; FORCE-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE4:%.*]] ]
+; FORCE-NEXT:    [[VEC_IND:%.*]] = phi <2 x i32> [ <i32 0, i32 1>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT:%.*]], [[PRED_STORE_CONTINUE4]] ]
 ; FORCE-NEXT:    [[TMP2:%.*]] = icmp ule <2 x i32> [[VEC_IND]], <i32 2, i32 2>
 ; FORCE-NEXT:    [[TMP3:%.*]] = extractelement <2 x i1> [[TMP2]], i32 0
-; FORCE-NEXT:    br i1 [[TMP3]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
-; FORCE:       pred.load.if:
+; FORCE-NEXT:    br i1 [[TMP3]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
+; FORCE:       pred.store.if:
 ; FORCE-NEXT:    [[TMP0:%.*]] = add i32 [[INDEX]], 0
 ; FORCE-NEXT:    store i32 [[TMP0]], i32* @b, align 1
-; FORCE-NEXT:    br label [[PRED_LOAD_CONTINUE]]
-; FORCE:       pred.load.continue:
+; FORCE-NEXT:    br label [[PRED_STORE_CONTINUE]]
+; FORCE:       pred.store.continue:
 ; FORCE-NEXT:    [[TMP10:%.*]] = extractelement <2 x i1> [[TMP2]], i32 1
-; FORCE-NEXT:    br i1 [[TMP10]], label [[PRED_LOAD_IF3:%.*]], label [[PRED_LOAD_CONTINUE4]]
-; FORCE:       pred.load.if1:
+; FORCE-NEXT:    br i1 [[TMP10]], label [[PRED_STORE_IF3:%.*]], label [[PRED_STORE_CONTINUE4]]
+; FORCE:       pred.store.if1:
 ; FORCE-NEXT:    [[TMP1:%.*]] = add i32 [[INDEX]], 1
 ; FORCE-NEXT:    store i32 [[TMP1]], i32* @b, align 1
-; FORCE-NEXT:    br label [[PRED_LOAD_CONTINUE4]]
-; FORCE:       pred.load.continue2:
+; FORCE-NEXT:    br label [[PRED_STORE_CONTINUE4]]
+; FORCE:       pred.store.continue2:
 ; FORCE-NEXT:    [[INDEX_NEXT]] = add i32 [[INDEX]], 2
 ; FORCE-NEXT:    [[VEC_IND_NEXT]] = add <2 x i32> [[VEC_IND]], <i32 2, i32 2>
 ; FORCE-NEXT:    [[TMP15:%.*]] = icmp eq i32 [[INDEX_NEXT]], 4
diff --git a/llvm/test/Transforms/LoopVectorize/X86/load-deref-pred.ll b/llvm/test/Transforms/LoopVectorize/X86/load-deref-pred.ll
index cf1207d468f44..dfb210772f44a 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/load-deref-pred.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/load-deref-pred.ll
@@ -101,7 +101,7 @@ define i32 @test_explicit_pred(i64 %len) {
 ; CHECK-NEXT:    [[VAL_PHI:%.*]] = phi i32 [ 0, [[LOOP]] ], [ [[VAL]], [[PRED]] ]
 ; CHECK-NEXT:    [[ACCUM_NEXT]] = add i32 [[ACCUM]], [[VAL_PHI]]
 ; CHECK-NEXT:    [[EXIT:%.*]] = icmp ugt i64 [[IV]], 4094
-; CHECK-NEXT:    br i1 [[EXIT]], label [[LOOP_EXIT]], label [[LOOP]], !llvm.loop [[LOOP2:![0-9]+]]
+; CHECK-NEXT:    br i1 [[EXIT]], label [[LOOP_EXIT]], label [[LOOP]], !llvm.loop [[LOOP3:![0-9]+]]
 ; CHECK:       loop_exit:
 ; CHECK-NEXT:    [[ACCUM_NEXT_LCSSA:%.*]] = phi i32 [ [[ACCUM_NEXT]], [[LATCH]] ], [ [[TMP25]], [[MIDDLE_BLOCK]] ]
 ; CHECK-NEXT:    ret i32 [[ACCUM_NEXT_LCSSA]]
@@ -2609,3 +2609,809 @@ latch:
 loop_exit:
   ret i32 %accum.next
 }
+
+
+; All non-unit stride test cases below.
+; These will be masked gathers with AVX-512 support.
+define i32 @test_stride_three(i64 %len, ptr %test_base) {
+; CHECK-LABEL: @test_stride_three(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[ALLOCA:%.*]] = alloca [105 x i32], align 4
+; CHECK-NEXT:    call void @init(ptr [[ALLOCA]])
+; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
+; CHECK:       vector.ph:
+; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
+; CHECK:       vector.body:
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_LOAD_CONTINUE6:%.*]] ]
+; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP37:%.*]], [[PRED_LOAD_CONTINUE6]] ]
+; CHECK-NEXT:    [[OFFSET_IDX:%.*]] = mul i64 [[INDEX]], 3
+; CHECK-NEXT:    [[TMP0:%.*]] = add i64 [[OFFSET_IDX]], 0
+; CHECK-NEXT:    [[TMP1:%.*]] = add i64 [[OFFSET_IDX]], 3
+; CHECK-NEXT:    [[TMP2:%.*]] = add i64 [[OFFSET_IDX]], 6
+; CHECK-NEXT:    [[TMP3:%.*]] = add i64 [[OFFSET_IDX]], 9
+; CHECK-NEXT:    [[TMP4:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE:%.*]], i64 [[TMP0]]
+; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP1]]
+; CHECK-NEXT:    [[TMP6:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP2]]
+; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP3]]
+; CHECK-NEXT:    [[TMP8:%.*]] = load i1, ptr [[TMP4]], align 1
+; CHECK-NEXT:    [[TMP9:%.*]] = load i1, ptr [[TMP5]], align 1
+; CHECK-NEXT:    [[TMP10:%.*]] = load i1, ptr [[TMP6]], align 1
+; CHECK-NEXT:    [[TMP11:%.*]] = load i1, ptr [[TMP7]], align 1
+; CHECK-NEXT:    [[TMP12:%.*]] = insertelement <4 x i1> poison, i1 [[TMP8]], i32 0
+; CHECK-NEXT:    [[TMP13:%.*]] = insertelement <4 x i1> [[TMP12]], i1 [[TMP9]], i32 1
+; CHECK-NEXT:    [[TMP14:%.*]] = insertelement <4 x i1> [[TMP13]], i1 [[TMP10]], i32 2
+; CHECK-NEXT:    [[TMP15:%.*]] = insertelement <4 x i1> [[TMP14]], i1 [[TMP11]], i32 3
+; CHECK-NEXT:    [[TMP16:%.*]] = extractelement <4 x i1> [[TMP15]], i32 0
+; CHECK-NEXT:    br i1 [[TMP16]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
+; CHECK:       pred.load.if:
+; CHECK-NEXT:    [[TMP17:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP0]]
+; CHECK-NEXT:    [[TMP18:%.*]] = load i32, ptr [[TMP17]], align 4
+; CHECK-NEXT:    [[TMP19:%.*]] = insertelement <4 x i32> poison, i32 [[TMP18]], i32 0
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE]]
+; CHECK:       pred.load.continue:
+; CHECK-NEXT:    [[TMP20:%.*]] = phi <4 x i32> [ poison, [[VECTOR_BODY]] ], [ [[TMP19]], [[PRED_LOAD_IF]] ]
+; CHECK-NEXT:    [[TMP21:%.*]] = extractelement <4 x i1> [[TMP15]], i32 1
+; CHECK-NEXT:    br i1 [[TMP21]], label [[PRED_LOAD_IF1:%.*]], label [[PRED_LOAD_CONTINUE2:%.*]]
+; CHECK:       pred.load.if1:
+; CHECK-NEXT:    [[TMP22:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP1]]
+; CHECK-NEXT:    [[TMP23:%.*]] = load i32, ptr [[TMP22]], align 4
+; CHECK-NEXT:    [[TMP24:%.*]] = insertelement <4 x i32> [[TMP20]], i32 [[TMP23]], i32 1
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE2]]
+; CHECK:       pred.load.continue2:
+; CHECK-NEXT:    [[TMP25:%.*]] = phi <4 x i32> [ [[TMP20]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP24]], [[PRED_LOAD_IF1]] ]
+; CHECK-NEXT:    [[TMP26:%.*]] = extractelement <4 x i1> [[TMP15]], i32 2
+; CHECK-NEXT:    br i1 [[TMP26]], label [[PRED_LOAD_IF3:%.*]], label [[PRED_LOAD_CONTINUE4:%.*]]
+; CHECK:       pred.load.if3:
+; CHECK-NEXT:    [[TMP27:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP2]]
+; CHECK-NEXT:    [[TMP28:%.*]] = load i32, ptr [[TMP27]], align 4
+; CHECK-NEXT:    [[TMP29:%.*]] = insertelement <4 x i32> [[TMP25]], i32 [[TMP28]], i32 2
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE4]]
+; CHECK:       pred.load.continue4:
+; CHECK-NEXT:    [[TMP30:%.*]] = phi <4 x i32> [ [[TMP25]], [[PRED_LOAD_CONTINUE2]] ], [ [[TMP29]], [[PRED_LOAD_IF3]] ]
+; CHECK-NEXT:    [[TMP31:%.*]] = extractelement <4 x i1> [[TMP15]], i32 3
+; CHECK-NEXT:    br i1 [[TMP31]], label [[PRED_LOAD_IF5:%.*]], label [[PRED_LOAD_CONTINUE6]]
+; CHECK:       pred.load.if5:
+; CHECK-NEXT:    [[TMP32:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP3]]
+; CHECK-NEXT:    [[TMP33:%.*]] = load i32, ptr [[TMP32]], align 4
+; CHECK-NEXT:    [[TMP34:%.*]] = insertelement <4 x i32> [[TMP30]], i32 [[TMP33]], i32 3
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE6]]
+; CHECK:       pred.load.continue6:
+; CHECK-NEXT:    [[TMP35:%.*]] = phi <4 x i32> [ [[TMP30]], [[PRED_LOAD_CONTINUE4]] ], [ [[TMP34]], [[PRED_LOAD_IF5]] ]
+; CHECK-NEXT:    [[TMP36:%.*]] = xor <4 x i1> [[TMP15]], <i1 true, i1 true, i1 true, i1 true>
+; CHECK-NEXT:    [[PREDPHI:%.*]] = select <4 x i1> [[TMP15]], <4 x i32> [[TMP35]], <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP37]] = add <4 x i32> [[VEC_PHI]], [[PREDPHI]]
+; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; CHECK-NEXT:    [[TMP38:%.*]] = icmp eq i64 [[INDEX_NEXT]], 32
+; CHECK-NEXT:    br i1 [[TMP38]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP30:![0-9]+]]
+; CHECK:       middle.block:
+; CHECK-NEXT:    [[TMP39:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[TMP37]])
+; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 35, 32
+; CHECK-NEXT:    br i1 [[CMP_N]], label [[LOOP_EXIT:%.*]], label [[SCALAR_PH]]
+; CHECK:       scalar.ph:
+; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ 96, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    [[BC_MERGE_RDX:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[TMP39]], [[MIDDLE_BLOCK]] ]
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[IV_NEXT:%.*]], [[LATCH:%.*]] ]
+; CHECK-NEXT:    [[ACCUM:%.*]] = phi i32 [ [[BC_MERGE_RDX]], [[SCALAR_PH]] ], [ [[ACCUM_NEXT:%.*]], [[LATCH]] ]
+; CHECK-NEXT:    [[IV_NEXT]] = add i64 [[IV]], 3
+; CHECK-NEXT:    [[TEST_ADDR:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[IV]]
+; CHECK-NEXT:    [[EARLYCND:%.*]] = load i1, ptr [[TEST_ADDR]], align 1
+; CHECK-NEXT:    br i1 [[EARLYCND]], label [[PRED:%.*]], label [[LATCH]]
+; CHECK:       pred:
+; CHECK-NEXT:    [[ADDR:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[IV]]
+; CHECK-NEXT:    [[VAL:%.*]] = load i32, ptr [[ADDR]], align 4
+; CHECK-NEXT:    br label [[LATCH]]
+; CHECK:       latch:
+; CHECK-NEXT:    [[VAL_PHI:%.*]] = phi i32 [ 0, [[LOOP]] ], [ [[VAL]], [[PRED]] ]
+; CHECK-NEXT:    [[ACCUM_NEXT]] = add i32 [[ACCUM]], [[VAL_PHI]]
+; CHECK-NEXT:    [[EXIT:%.*]] = icmp ugt i64 [[IV]], 100
+; CHECK-NEXT:    br i1 [[EXIT]], label [[LOOP_EXIT]], label [[LOOP]], !llvm.loop [[LOOP31:![0-9]+]]
+; CHECK:       loop_exit:
+; CHECK-NEXT:    [[ACCUM_NEXT_LCSSA:%.*]] = phi i32 [ [[ACCUM_NEXT]], [[LATCH]] ], [ [[TMP39]], [[MIDDLE_BLOCK]] ]
+; CHECK-NEXT:    ret i32 [[ACCUM_NEXT_LCSSA]]
+;
+entry:
+  %alloca = alloca [105 x i32]
+  call void @init(ptr %alloca)
+  br label %loop
+loop:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %latch ]
+  %accum = phi i32 [ 0, %entry ], [ %accum.next, %latch ]
+  %iv.next = add i64 %iv, 3
+  %test_addr = getelementptr inbounds i1, ptr %test_base, i64 %iv
+  %earlycnd = load i1, ptr %test_addr
+  br i1 %earlycnd, label %pred, label %latch
+pred:
+  %addr = getelementptr inbounds i32, ptr %alloca, i64 %iv
+  %val = load i32, ptr %addr
+  br label %latch
+latch:
+  %val.phi = phi i32 [0, %loop], [%val, %pred]
+  %accum.next = add i32 %accum, %val.phi
+  %exit = icmp ugt i64 %iv, 100
+  br i1 %exit, label %loop_exit, label %loop
+
+loop_exit:
+  ret i32 %accum.next
+}
+
+define i32 @test_non_unit_stride_four(i64 %len, ptr %test_base) {
+; CHECK-LABEL: @test_non_unit_stride_four(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[ALLOCA:%.*]] = alloca [108 x i32], align 4
+; CHECK-NEXT:    call void @init(ptr [[ALLOCA]])
+; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
+; CHECK:       vector.ph:
+; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
+; CHECK:       vector.body:
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_LOAD_CONTINUE6:%.*]] ]
+; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP37:%.*]], [[PRED_LOAD_CONTINUE6]] ]
+; CHECK-NEXT:    [[OFFSET_IDX:%.*]] = mul i64 [[INDEX]], 4
+; CHECK-NEXT:    [[TMP0:%.*]] = add i64 [[OFFSET_IDX]], 0
+; CHECK-NEXT:    [[TMP1:%.*]] = add i64 [[OFFSET_IDX]], 4
+; CHECK-NEXT:    [[TMP2:%.*]] = add i64 [[OFFSET_IDX]], 8
+; CHECK-NEXT:    [[TMP3:%.*]] = add i64 [[OFFSET_IDX]], 12
+; CHECK-NEXT:    [[TMP4:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE:%.*]], i64 [[TMP0]]
+; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP1]]
+; CHECK-NEXT:    [[TMP6:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP2]]
+; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP3]]
+; CHECK-NEXT:    [[TMP8:%.*]] = load i1, ptr [[TMP4]], align 1
+; CHECK-NEXT:    [[TMP9:%.*]] = load i1, ptr [[TMP5]], align 1
+; CHECK-NEXT:    [[TMP10:%.*]] = load i1, ptr [[TMP6]], align 1
+; CHECK-NEXT:    [[TMP11:%.*]] = load i1, ptr [[TMP7]], align 1
+; CHECK-NEXT:    [[TMP12:%.*]] = insertelement <4 x i1> poison, i1 [[TMP8]], i32 0
+; CHECK-NEXT:    [[TMP13:%.*]] = insertelement <4 x i1> [[TMP12]], i1 [[TMP9]], i32 1
+; CHECK-NEXT:    [[TMP14:%.*]] = insertelement <4 x i1> [[TMP13]], i1 [[TMP10]], i32 2
+; CHECK-NEXT:    [[TMP15:%.*]] = insertelement <4 x i1> [[TMP14]], i1 [[TMP11]], i32 3
+; CHECK-NEXT:    [[TMP16:%.*]] = extractelement <4 x i1> [[TMP15]], i32 0
+; CHECK-NEXT:    br i1 [[TMP16]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
+; CHECK:       pred.load.if:
+; CHECK-NEXT:    [[TMP17:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP0]]
+; CHECK-NEXT:    [[TMP18:%.*]] = load i32, ptr [[TMP17]], align 4
+; CHECK-NEXT:    [[TMP19:%.*]] = insertelement <4 x i32> poison, i32 [[TMP18]], i32 0
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE]]
+; CHECK:       pred.load.continue:
+; CHECK-NEXT:    [[TMP20:%.*]] = phi <4 x i32> [ poison, [[VECTOR_BODY]] ], [ [[TMP19]], [[PRED_LOAD_IF]] ]
+; CHECK-NEXT:    [[TMP21:%.*]] = extractelement <4 x i1> [[TMP15]], i32 1
+; CHECK-NEXT:    br i1 [[TMP21]], label [[PRED_LOAD_IF1:%.*]], label [[PRED_LOAD_CONTINUE2:%.*]]
+; CHECK:       pred.load.if1:
+; CHECK-NEXT:    [[TMP22:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP1]]
+; CHECK-NEXT:    [[TMP23:%.*]] = load i32, ptr [[TMP22]], align 4
+; CHECK-NEXT:    [[TMP24:%.*]] = insertelement <4 x i32> [[TMP20]], i32 [[TMP23]], i32 1
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE2]]
+; CHECK:       pred.load.continue2:
+; CHECK-NEXT:    [[TMP25:%.*]] = phi <4 x i32> [ [[TMP20]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP24]], [[PRED_LOAD_IF1]] ]
+; CHECK-NEXT:    [[TMP26:%.*]] = extractelement <4 x i1> [[TMP15]], i32 2
+; CHECK-NEXT:    br i1 [[TMP26]], label [[PRED_LOAD_IF3:%.*]], label [[PRED_LOAD_CONTINUE4:%.*]]
+; CHECK:       pred.load.if3:
+; CHECK-NEXT:    [[TMP27:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP2]]
+; CHECK-NEXT:    [[TMP28:%.*]] = load i32, ptr [[TMP27]], align 4
+; CHECK-NEXT:    [[TMP29:%.*]] = insertelement <4 x i32> [[TMP25]], i32 [[TMP28]], i32 2
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE4]]
+; CHECK:       pred.load.continue4:
+; CHECK-NEXT:    [[TMP30:%.*]] = phi <4 x i32> [ [[TMP25]], [[PRED_LOAD_CONTINUE2]] ], [ [[TMP29]], [[PRED_LOAD_IF3]] ]
+; CHECK-NEXT:    [[TMP31:%.*]] = extractelement <4 x i1> [[TMP15]], i32 3
+; CHECK-NEXT:    br i1 [[TMP31]], label [[PRED_LOAD_IF5:%.*]], label [[PRED_LOAD_CONTINUE6]]
+; CHECK:       pred.load.if5:
+; CHECK-NEXT:    [[TMP32:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP3]]
+; CHECK-NEXT:    [[TMP33:%.*]] = load i32, ptr [[TMP32]], align 4
+; CHECK-NEXT:    [[TMP34:%.*]] = insertelement <4 x i32> [[TMP30]], i32 [[TMP33]], i32 3
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE6]]
+; CHECK:       pred.load.continue6:
+; CHECK-NEXT:    [[TMP35:%.*]] = phi <4 x i32> [ [[TMP30]], [[PRED_LOAD_CONTINUE4]] ], [ [[TMP34]], [[PRED_LOAD_IF5]] ]
+; CHECK-NEXT:    [[TMP36:%.*]] = xor <4 x i1> [[TMP15]], <i1 true, i1 true, i1 true, i1 true>
+; CHECK-NEXT:    [[PREDPHI:%.*]] = select <4 x i1> [[TMP15]], <4 x i32> [[TMP35]], <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP37]] = add <4 x i32> [[VEC_PHI]], [[PREDPHI]]
+; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; CHECK-NEXT:    [[TMP38:%.*]] = icmp eq i64 [[INDEX_NEXT]], 24
+; CHECK-NEXT:    br i1 [[TMP38]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP32:![0-9]+]]
+; CHECK:       middle.block:
+; CHECK-NEXT:    [[TMP39:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[TMP37]])
+; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 27, 24
+; CHECK-NEXT:    br i1 [[CMP_N]], label [[LOOP_EXIT:%.*]], label [[SCALAR_PH]]
+; CHECK:       scalar.ph:
+; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ 96, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    [[BC_MERGE_RDX:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[TMP39]], [[MIDDLE_BLOCK]] ]
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[IV_NEXT:%.*]], [[LATCH:%.*]] ]
+; CHECK-NEXT:    [[ACCUM:%.*]] = phi i32 [ [[BC_MERGE_RDX]], [[SCALAR_PH]] ], [ [[ACCUM_NEXT:%.*]], [[LATCH]] ]
+; CHECK-NEXT:    [[IV_NEXT]] = add i64 [[IV]], 4
+; CHECK-NEXT:    [[TEST_ADDR:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[IV]]
+; CHECK-NEXT:    [[EARLYCND:%.*]] = load i1, ptr [[TEST_ADDR]], align 1
+; CHECK-NEXT:    br i1 [[EARLYCND]], label [[PRED:%.*]], label [[LATCH]]
+; CHECK:       pred:
+; CHECK-NEXT:    [[ADDR:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[IV]]
+; CHECK-NEXT:    [[VAL:%.*]] = load i32, ptr [[ADDR]], align 4
+; CHECK-NEXT:    br label [[LATCH]]
+; CHECK:       latch:
+; CHECK-NEXT:    [[VAL_PHI:%.*]] = phi i32 [ 0, [[LOOP]] ], [ [[VAL]], [[PRED]] ]
+; CHECK-NEXT:    [[ACCUM_NEXT]] = add i32 [[ACCUM]], [[VAL_PHI]]
+; CHECK-NEXT:    [[EXIT:%.*]] = icmp ugt i64 [[IV]], 100
+; CHECK-NEXT:    br i1 [[EXIT]], label [[LOOP_EXIT]], label [[LOOP]], !llvm.loop [[LOOP33:![0-9]+]]
+; CHECK:       loop_exit:
+; CHECK-NEXT:    [[ACCUM_NEXT_LCSSA:%.*]] = phi i32 [ [[ACCUM_NEXT]], [[LATCH]] ], [ [[TMP39]], [[MIDDLE_BLOCK]] ]
+; CHECK-NEXT:    ret i32 [[ACCUM_NEXT_LCSSA]]
+;
+entry:
+  %alloca = alloca [108 x i32]
+  call void @init(ptr %alloca)
+  br label %loop
+loop:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %latch ]
+  %accum = phi i32 [ 0, %entry ], [ %accum.next, %latch ]
+  %iv.next = add i64 %iv, 4
+  %test_addr = getelementptr inbounds i1, ptr %test_base, i64 %iv
+  %earlycnd = load i1, ptr %test_addr
+  br i1 %earlycnd, label %pred, label %latch
+pred:
+  %addr = getelementptr inbounds i32, ptr %alloca, i64 %iv
+  %val = load i32, ptr %addr
+  br label %latch
+latch:
+  %val.phi = phi i32 [0, %loop], [%val, %pred]
+  %accum.next = add i32 %accum, %val.phi
+  %exit = icmp ugt i64 %iv, 100
+  br i1 %exit, label %loop_exit, label %loop
+
+loop_exit:
+  ret i32 %accum.next
+}
+
+define i32 @test_non_unit_stride_five(i64 %len, ptr %test_base) {
+; CHECK-LABEL: @test_non_unit_stride_five(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[ALLOCA:%.*]] = alloca [110 x i32], align 4
+; CHECK-NEXT:    call void @init(ptr [[ALLOCA]])
+; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
+; CHECK:       vector.ph:
+; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
+; CHECK:       vector.body:
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_LOAD_CONTINUE6:%.*]] ]
+; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP37:%.*]], [[PRED_LOAD_CONTINUE6]] ]
+; CHECK-NEXT:    [[OFFSET_IDX:%.*]] = mul i64 [[INDEX]], 5
+; CHECK-NEXT:    [[TMP0:%.*]] = add i64 [[OFFSET_IDX]], 0
+; CHECK-NEXT:    [[TMP1:%.*]] = add i64 [[OFFSET_IDX]], 5
+; CHECK-NEXT:    [[TMP2:%.*]] = add i64 [[OFFSET_IDX]], 10
+; CHECK-NEXT:    [[TMP3:%.*]] = add i64 [[OFFSET_IDX]], 15
+; CHECK-NEXT:    [[TMP4:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE:%.*]], i64 [[TMP0]]
+; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP1]]
+; CHECK-NEXT:    [[TMP6:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP2]]
+; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP3]]
+; CHECK-NEXT:    [[TMP8:%.*]] = load i1, ptr [[TMP4]], align 1
+; CHECK-NEXT:    [[TMP9:%.*]] = load i1, ptr [[TMP5]], align 1
+; CHECK-NEXT:    [[TMP10:%.*]] = load i1, ptr [[TMP6]], align 1
+; CHECK-NEXT:    [[TMP11:%.*]] = load i1, ptr [[TMP7]], align 1
+; CHECK-NEXT:    [[TMP12:%.*]] = insertelement <4 x i1> poison, i1 [[TMP8]], i32 0
+; CHECK-NEXT:    [[TMP13:%.*]] = insertelement <4 x i1> [[TMP12]], i1 [[TMP9]], i32 1
+; CHECK-NEXT:    [[TMP14:%.*]] = insertelement <4 x i1> [[TMP13]], i1 [[TMP10]], i32 2
+; CHECK-NEXT:    [[TMP15:%.*]] = insertelement <4 x i1> [[TMP14]], i1 [[TMP11]], i32 3
+; CHECK-NEXT:    [[TMP16:%.*]] = extractelement <4 x i1> [[TMP15]], i32 0
+; CHECK-NEXT:    br i1 [[TMP16]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
+; CHECK:       pred.load.if:
+; CHECK-NEXT:    [[TMP17:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP0]]
+; CHECK-NEXT:    [[TMP18:%.*]] = load i32, ptr [[TMP17]], align 4
+; CHECK-NEXT:    [[TMP19:%.*]] = insertelement <4 x i32> poison, i32 [[TMP18]], i32 0
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE]]
+; CHECK:       pred.load.continue:
+; CHECK-NEXT:    [[TMP20:%.*]] = phi <4 x i32> [ poison, [[VECTOR_BODY]] ], [ [[TMP19]], [[PRED_LOAD_IF]] ]
+; CHECK-NEXT:    [[TMP21:%.*]] = extractelement <4 x i1> [[TMP15]], i32 1
+; CHECK-NEXT:    br i1 [[TMP21]], label [[PRED_LOAD_IF1:%.*]], label [[PRED_LOAD_CONTINUE2:%.*]]
+; CHECK:       pred.load.if1:
+; CHECK-NEXT:    [[TMP22:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP1]]
+; CHECK-NEXT:    [[TMP23:%.*]] = load i32, ptr [[TMP22]], align 4
+; CHECK-NEXT:    [[TMP24:%.*]] = insertelement <4 x i32> [[TMP20]], i32 [[TMP23]], i32 1
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE2]]
+; CHECK:       pred.load.continue2:
+; CHECK-NEXT:    [[TMP25:%.*]] = phi <4 x i32> [ [[TMP20]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP24]], [[PRED_LOAD_IF1]] ]
+; CHECK-NEXT:    [[TMP26:%.*]] = extractelement <4 x i1> [[TMP15]], i32 2
+; CHECK-NEXT:    br i1 [[TMP26]], label [[PRED_LOAD_IF3:%.*]], label [[PRED_LOAD_CONTINUE4:%.*]]
+; CHECK:       pred.load.if3:
+; CHECK-NEXT:    [[TMP27:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP2]]
+; CHECK-NEXT:    [[TMP28:%.*]] = load i32, ptr [[TMP27]], align 4
+; CHECK-NEXT:    [[TMP29:%.*]] = insertelement <4 x i32> [[TMP25]], i32 [[TMP28]], i32 2
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE4]]
+; CHECK:       pred.load.continue4:
+; CHECK-NEXT:    [[TMP30:%.*]] = phi <4 x i32> [ [[TMP25]], [[PRED_LOAD_CONTINUE2]] ], [ [[TMP29]], [[PRED_LOAD_IF3]] ]
+; CHECK-NEXT:    [[TMP31:%.*]] = extractelement <4 x i1> [[TMP15]], i32 3
+; CHECK-NEXT:    br i1 [[TMP31]], label [[PRED_LOAD_IF5:%.*]], label [[PRED_LOAD_CONTINUE6]]
+; CHECK:       pred.load.if5:
+; CHECK-NEXT:    [[TMP32:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP3]]
+; CHECK-NEXT:    [[TMP33:%.*]] = load i32, ptr [[TMP32]], align 4
+; CHECK-NEXT:    [[TMP34:%.*]] = insertelement <4 x i32> [[TMP30]], i32 [[TMP33]], i32 3
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE6]]
+; CHECK:       pred.load.continue6:
+; CHECK-NEXT:    [[TMP35:%.*]] = phi <4 x i32> [ [[TMP30]], [[PRED_LOAD_CONTINUE4]] ], [ [[TMP34]], [[PRED_LOAD_IF5]] ]
+; CHECK-NEXT:    [[TMP36:%.*]] = xor <4 x i1> [[TMP15]], <i1 true, i1 true, i1 true, i1 true>
+; CHECK-NEXT:    [[PREDPHI:%.*]] = select <4 x i1> [[TMP15]], <4 x i32> [[TMP35]], <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP37]] = add <4 x i32> [[VEC_PHI]], [[PREDPHI]]
+; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; CHECK-NEXT:    [[TMP38:%.*]] = icmp eq i64 [[INDEX_NEXT]], 20
+; CHECK-NEXT:    br i1 [[TMP38]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP34:![0-9]+]]
+; CHECK:       middle.block:
+; CHECK-NEXT:    [[TMP39:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[TMP37]])
+; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 22, 20
+; CHECK-NEXT:    br i1 [[CMP_N]], label [[LOOP_EXIT:%.*]], label [[SCALAR_PH]]
+; CHECK:       scalar.ph:
+; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ 100, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    [[BC_MERGE_RDX:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[TMP39]], [[MIDDLE_BLOCK]] ]
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[IV_NEXT:%.*]], [[LATCH:%.*]] ]
+; CHECK-NEXT:    [[ACCUM:%.*]] = phi i32 [ [[BC_MERGE_RDX]], [[SCALAR_PH]] ], [ [[ACCUM_NEXT:%.*]], [[LATCH]] ]
+; CHECK-NEXT:    [[IV_NEXT]] = add i64 [[IV]], 5
+; CHECK-NEXT:    [[TEST_ADDR:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[IV]]
+; CHECK-NEXT:    [[EARLYCND:%.*]] = load i1, ptr [[TEST_ADDR]], align 1
+; CHECK-NEXT:    br i1 [[EARLYCND]], label [[PRED:%.*]], label [[LATCH]]
+; CHECK:       pred:
+; CHECK-NEXT:    [[ADDR:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[IV]]
+; CHECK-NEXT:    [[VAL:%.*]] = load i32, ptr [[ADDR]], align 4
+; CHECK-NEXT:    br label [[LATCH]]
+; CHECK:       latch:
+; CHECK-NEXT:    [[VAL_PHI:%.*]] = phi i32 [ 0, [[LOOP]] ], [ [[VAL]], [[PRED]] ]
+; CHECK-NEXT:    [[ACCUM_NEXT]] = add i32 [[ACCUM]], [[VAL_PHI]]
+; CHECK-NEXT:    [[EXIT:%.*]] = icmp ugt i64 [[IV]], 100
+; CHECK-NEXT:    br i1 [[EXIT]], label [[LOOP_EXIT]], label [[LOOP]], !llvm.loop [[LOOP35:![0-9]+]]
+; CHECK:       loop_exit:
+; CHECK-NEXT:    [[ACCUM_NEXT_LCSSA:%.*]] = phi i32 [ [[ACCUM_NEXT]], [[LATCH]] ], [ [[TMP39]], [[MIDDLE_BLOCK]] ]
+; CHECK-NEXT:    ret i32 [[ACCUM_NEXT_LCSSA]]
+;
+entry:
+  %alloca = alloca [110 x i32]
+  call void @init(ptr %alloca)
+  br label %loop
+loop:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %latch ]
+  %accum = phi i32 [ 0, %entry ], [ %accum.next, %latch ]
+  %iv.next = add i64 %iv, 5
+  %test_addr = getelementptr inbounds i1, ptr %test_base, i64 %iv
+  %earlycnd = load i1, ptr %test_addr
+  br i1 %earlycnd, label %pred, label %latch
+pred:
+  %addr = getelementptr inbounds i32, ptr %alloca, i64 %iv
+  %val = load i32, ptr %addr
+  br label %latch
+latch:
+  %val.phi = phi i32 [0, %loop], [%val, %pred]
+  %accum.next = add i32 %accum, %val.phi
+  %exit = icmp ugt i64 %iv, 100
+  br i1 %exit, label %loop_exit, label %loop
+
+loop_exit:
+  ret i32 %accum.next
+}
+
+define i32 @neg_test_non_unit_stride_off_by_four_bytes(i64 %len, ptr %test_base) {
+; CHECK-LABEL: @neg_test_non_unit_stride_off_by_four_bytes(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[ALLOCA:%.*]] = alloca [103 x i32], align 4
+; CHECK-NEXT:    call void @init(ptr [[ALLOCA]])
+; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
+; CHECK:       vector.ph:
+; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
+; CHECK:       vector.body:
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_LOAD_CONTINUE6:%.*]] ]
+; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP37:%.*]], [[PRED_LOAD_CONTINUE6]] ]
+; CHECK-NEXT:    [[OFFSET_IDX:%.*]] = mul i64 [[INDEX]], 2
+; CHECK-NEXT:    [[TMP0:%.*]] = add i64 [[OFFSET_IDX]], 0
+; CHECK-NEXT:    [[TMP1:%.*]] = add i64 [[OFFSET_IDX]], 2
+; CHECK-NEXT:    [[TMP2:%.*]] = add i64 [[OFFSET_IDX]], 4
+; CHECK-NEXT:    [[TMP3:%.*]] = add i64 [[OFFSET_IDX]], 6
+; CHECK-NEXT:    [[TMP4:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE:%.*]], i64 [[TMP0]]
+; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP1]]
+; CHECK-NEXT:    [[TMP6:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP2]]
+; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP3]]
+; CHECK-NEXT:    [[TMP8:%.*]] = load i1, ptr [[TMP4]], align 1
+; CHECK-NEXT:    [[TMP9:%.*]] = load i1, ptr [[TMP5]], align 1
+; CHECK-NEXT:    [[TMP10:%.*]] = load i1, ptr [[TMP6]], align 1
+; CHECK-NEXT:    [[TMP11:%.*]] = load i1, ptr [[TMP7]], align 1
+; CHECK-NEXT:    [[TMP12:%.*]] = insertelement <4 x i1> poison, i1 [[TMP8]], i32 0
+; CHECK-NEXT:    [[TMP13:%.*]] = insertelement <4 x i1> [[TMP12]], i1 [[TMP9]], i32 1
+; CHECK-NEXT:    [[TMP14:%.*]] = insertelement <4 x i1> [[TMP13]], i1 [[TMP10]], i32 2
+; CHECK-NEXT:    [[TMP15:%.*]] = insertelement <4 x i1> [[TMP14]], i1 [[TMP11]], i32 3
+; CHECK-NEXT:    [[TMP16:%.*]] = extractelement <4 x i1> [[TMP15]], i32 0
+; CHECK-NEXT:    br i1 [[TMP16]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
+; CHECK:       pred.load.if:
+; CHECK-NEXT:    [[TMP17:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP0]]
+; CHECK-NEXT:    [[TMP18:%.*]] = load i32, ptr [[TMP17]], align 4
+; CHECK-NEXT:    [[TMP19:%.*]] = insertelement <4 x i32> poison, i32 [[TMP18]], i32 0
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE]]
+; CHECK:       pred.load.continue:
+; CHECK-NEXT:    [[TMP20:%.*]] = phi <4 x i32> [ poison, [[VECTOR_BODY]] ], [ [[TMP19]], [[PRED_LOAD_IF]] ]
+; CHECK-NEXT:    [[TMP21:%.*]] = extractelement <4 x i1> [[TMP15]], i32 1
+; CHECK-NEXT:    br i1 [[TMP21]], label [[PRED_LOAD_IF1:%.*]], label [[PRED_LOAD_CONTINUE2:%.*]]
+; CHECK:       pred.load.if1:
+; CHECK-NEXT:    [[TMP22:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP1]]
+; CHECK-NEXT:    [[TMP23:%.*]] = load i32, ptr [[TMP22]], align 4
+; CHECK-NEXT:    [[TMP24:%.*]] = insertelement <4 x i32> [[TMP20]], i32 [[TMP23]], i32 1
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE2]]
+; CHECK:       pred.load.continue2:
+; CHECK-NEXT:    [[TMP25:%.*]] = phi <4 x i32> [ [[TMP20]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP24]], [[PRED_LOAD_IF1]] ]
+; CHECK-NEXT:    [[TMP26:%.*]] = extractelement <4 x i1> [[TMP15]], i32 2
+; CHECK-NEXT:    br i1 [[TMP26]], label [[PRED_LOAD_IF3:%.*]], label [[PRED_LOAD_CONTINUE4:%.*]]
+; CHECK:       pred.load.if3:
+; CHECK-NEXT:    [[TMP27:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP2]]
+; CHECK-NEXT:    [[TMP28:%.*]] = load i32, ptr [[TMP27]], align 4
+; CHECK-NEXT:    [[TMP29:%.*]] = insertelement <4 x i32> [[TMP25]], i32 [[TMP28]], i32 2
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE4]]
+; CHECK:       pred.load.continue4:
+; CHECK-NEXT:    [[TMP30:%.*]] = phi <4 x i32> [ [[TMP25]], [[PRED_LOAD_CONTINUE2]] ], [ [[TMP29]], [[PRED_LOAD_IF3]] ]
+; CHECK-NEXT:    [[TMP31:%.*]] = extractelement <4 x i1> [[TMP15]], i32 3
+; CHECK-NEXT:    br i1 [[TMP31]], label [[PRED_LOAD_IF5:%.*]], label [[PRED_LOAD_CONTINUE6]]
+; CHECK:       pred.load.if5:
+; CHECK-NEXT:    [[TMP32:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP3]]
+; CHECK-NEXT:    [[TMP33:%.*]] = load i32, ptr [[TMP32]], align 4
+; CHECK-NEXT:    [[TMP34:%.*]] = insertelement <4 x i32> [[TMP30]], i32 [[TMP33]], i32 3
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE6]]
+; CHECK:       pred.load.continue6:
+; CHECK-NEXT:    [[TMP35:%.*]] = phi <4 x i32> [ [[TMP30]], [[PRED_LOAD_CONTINUE4]] ], [ [[TMP34]], [[PRED_LOAD_IF5]] ]
+; CHECK-NEXT:    [[TMP36:%.*]] = xor <4 x i1> [[TMP15]], <i1 true, i1 true, i1 true, i1 true>
+; CHECK-NEXT:    [[PREDPHI:%.*]] = select <4 x i1> [[TMP15]], <4 x i32> [[TMP35]], <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP37]] = add <4 x i32> [[VEC_PHI]], [[PREDPHI]]
+; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; CHECK-NEXT:    [[TMP38:%.*]] = icmp eq i64 [[INDEX_NEXT]], 52
+; CHECK-NEXT:    br i1 [[TMP38]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP36:![0-9]+]]
+; CHECK:       middle.block:
+; CHECK-NEXT:    [[TMP39:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[TMP37]])
+; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 52, 52
+; CHECK-NEXT:    br i1 [[CMP_N]], label [[LOOP_EXIT:%.*]], label [[SCALAR_PH]]
+; CHECK:       scalar.ph:
+; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ 104, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    [[BC_MERGE_RDX:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[TMP39]], [[MIDDLE_BLOCK]] ]
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[IV_NEXT:%.*]], [[LATCH:%.*]] ]
+; CHECK-NEXT:    [[ACCUM:%.*]] = phi i32 [ [[BC_MERGE_RDX]], [[SCALAR_PH]] ], [ [[ACCUM_NEXT:%.*]], [[LATCH]] ]
+; CHECK-NEXT:    [[IV_NEXT]] = add i64 [[IV]], 2
+; CHECK-NEXT:    [[TEST_ADDR:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[IV]]
+; CHECK-NEXT:    [[EARLYCND:%.*]] = load i1, ptr [[TEST_ADDR]], align 1
+; CHECK-NEXT:    br i1 [[EARLYCND]], label [[PRED:%.*]], label [[LATCH]]
+; CHECK:       pred:
+; CHECK-NEXT:    [[ADDR:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[IV]]
+; CHECK-NEXT:    [[VAL:%.*]] = load i32, ptr [[ADDR]], align 4
+; CHECK-NEXT:    br label [[LATCH]]
+; CHECK:       latch:
+; CHECK-NEXT:    [[VAL_PHI:%.*]] = phi i32 [ 0, [[LOOP]] ], [ [[VAL]], [[PRED]] ]
+; CHECK-NEXT:    [[ACCUM_NEXT]] = add i32 [[ACCUM]], [[VAL_PHI]]
+; CHECK-NEXT:    [[EXIT:%.*]] = icmp ugt i64 [[IV]], 100
+; CHECK-NEXT:    br i1 [[EXIT]], label [[LOOP_EXIT]], label [[LOOP]], !llvm.loop [[LOOP37:![0-9]+]]
+; CHECK:       loop_exit:
+; CHECK-NEXT:    [[ACCUM_NEXT_LCSSA:%.*]] = phi i32 [ [[ACCUM_NEXT]], [[LATCH]] ], [ [[TMP39]], [[MIDDLE_BLOCK]] ]
+; CHECK-NEXT:    ret i32 [[ACCUM_NEXT_LCSSA]]
+;
+entry:
+  %alloca = alloca [103 x i32]
+  call void @init(ptr %alloca)
+  br label %loop
+loop:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %latch ]
+  %accum = phi i32 [ 0, %entry ], [ %accum.next, %latch ]
+  %iv.next = add i64 %iv, 2
+  %test_addr = getelementptr inbounds i1, ptr %test_base, i64 %iv
+  %earlycnd = load i1, ptr %test_addr
+  br i1 %earlycnd, label %pred, label %latch
+pred:
+  %addr = getelementptr inbounds i32, ptr %alloca, i64 %iv
+  %val = load i32, ptr %addr
+  br label %latch
+latch:
+  %val.phi = phi i32 [0, %loop], [%val, %pred]
+  %accum.next = add i32 %accum, %val.phi
+  %exit = icmp ugt i64 %iv, 100
+  br i1 %exit, label %loop_exit, label %loop
+
+loop_exit:
+  ret i32 %accum.next
+}
+
+; Start value of AddRec is not a value (%alloca + 4)
+define i32 @test_non_unit_stride_with_first_iteration_step_access(i64 %len, ptr %test_base) {
+; CHECK-LABEL: @test_non_unit_stride_with_first_iteration_step_access(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[ALLOCA:%.*]] = alloca [4096 x i32], align 4
+; CHECK-NEXT:    call void @init(ptr [[ALLOCA]])
+; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
+; CHECK:       vector.ph:
+; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
+; CHECK:       vector.body:
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_LOAD_CONTINUE33:%.*]] ]
+; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP164:%.*]], [[PRED_LOAD_CONTINUE33]] ]
+; CHECK-NEXT:    [[VEC_PHI1:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP165:%.*]], [[PRED_LOAD_CONTINUE33]] ]
+; CHECK-NEXT:    [[VEC_PHI2:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP166:%.*]], [[PRED_LOAD_CONTINUE33]] ]
+; CHECK-NEXT:    [[VEC_PHI3:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP167:%.*]], [[PRED_LOAD_CONTINUE33]] ]
+; CHECK-NEXT:    [[OFFSET_IDX:%.*]] = mul i64 [[INDEX]], 2
+; CHECK-NEXT:    [[TMP0:%.*]] = add i64 [[OFFSET_IDX]], 0
+; CHECK-NEXT:    [[TMP1:%.*]] = add i64 [[OFFSET_IDX]], 2
+; CHECK-NEXT:    [[TMP2:%.*]] = add i64 [[OFFSET_IDX]], 4
+; CHECK-NEXT:    [[TMP3:%.*]] = add i64 [[OFFSET_IDX]], 6
+; CHECK-NEXT:    [[TMP4:%.*]] = add i64 [[OFFSET_IDX]], 8
+; CHECK-NEXT:    [[TMP5:%.*]] = add i64 [[OFFSET_IDX]], 10
+; CHECK-NEXT:    [[TMP6:%.*]] = add i64 [[OFFSET_IDX]], 12
+; CHECK-NEXT:    [[TMP7:%.*]] = add i64 [[OFFSET_IDX]], 14
+; CHECK-NEXT:    [[TMP8:%.*]] = add i64 [[OFFSET_IDX]], 16
+; CHECK-NEXT:    [[TMP9:%.*]] = add i64 [[OFFSET_IDX]], 18
+; CHECK-NEXT:    [[TMP10:%.*]] = add i64 [[OFFSET_IDX]], 20
+; CHECK-NEXT:    [[TMP11:%.*]] = add i64 [[OFFSET_IDX]], 22
+; CHECK-NEXT:    [[TMP12:%.*]] = add i64 [[OFFSET_IDX]], 24
+; CHECK-NEXT:    [[TMP13:%.*]] = add i64 [[OFFSET_IDX]], 26
+; CHECK-NEXT:    [[TMP14:%.*]] = add i64 [[OFFSET_IDX]], 28
+; CHECK-NEXT:    [[TMP15:%.*]] = add i64 [[OFFSET_IDX]], 30
+; CHECK-NEXT:    [[TMP16:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE:%.*]], i64 [[TMP0]]
+; CHECK-NEXT:    [[TMP17:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP1]]
+; CHECK-NEXT:    [[TMP18:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP2]]
+; CHECK-NEXT:    [[TMP19:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP3]]
+; CHECK-NEXT:    [[TMP20:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP4]]
+; CHECK-NEXT:    [[TMP21:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP5]]
+; CHECK-NEXT:    [[TMP22:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP6]]
+; CHECK-NEXT:    [[TMP23:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP7]]
+; CHECK-NEXT:    [[TMP24:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP8]]
+; CHECK-NEXT:    [[TMP25:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP9]]
+; CHECK-NEXT:    [[TMP26:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP10]]
+; CHECK-NEXT:    [[TMP27:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP11]]
+; CHECK-NEXT:    [[TMP28:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP12]]
+; CHECK-NEXT:    [[TMP29:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP13]]
+; CHECK-NEXT:    [[TMP30:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP14]]
+; CHECK-NEXT:    [[TMP31:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[TMP15]]
+; CHECK-NEXT:    [[TMP32:%.*]] = load i1, ptr [[TMP16]], align 1
+; CHECK-NEXT:    [[TMP33:%.*]] = load i1, ptr [[TMP17]], align 1
+; CHECK-NEXT:    [[TMP34:%.*]] = load i1, ptr [[TMP18]], align 1
+; CHECK-NEXT:    [[TMP35:%.*]] = load i1, ptr [[TMP19]], align 1
+; CHECK-NEXT:    [[TMP36:%.*]] = insertelement <4 x i1> poison, i1 [[TMP32]], i32 0
+; CHECK-NEXT:    [[TMP37:%.*]] = insertelement <4 x i1> [[TMP36]], i1 [[TMP33]], i32 1
+; CHECK-NEXT:    [[TMP38:%.*]] = insertelement <4 x i1> [[TMP37]], i1 [[TMP34]], i32 2
+; CHECK-NEXT:    [[TMP39:%.*]] = insertelement <4 x i1> [[TMP38]], i1 [[TMP35]], i32 3
+; CHECK-NEXT:    [[TMP40:%.*]] = load i1, ptr [[TMP20]], align 1
+; CHECK-NEXT:    [[TMP41:%.*]] = load i1, ptr [[TMP21]], align 1
+; CHECK-NEXT:    [[TMP42:%.*]] = load i1, ptr [[TMP22]], align 1
+; CHECK-NEXT:    [[TMP43:%.*]] = load i1, ptr [[TMP23]], align 1
+; CHECK-NEXT:    [[TMP44:%.*]] = insertelement <4 x i1> poison, i1 [[TMP40]], i32 0
+; CHECK-NEXT:    [[TMP45:%.*]] = insertelement <4 x i1> [[TMP44]], i1 [[TMP41]], i32 1
+; CHECK-NEXT:    [[TMP46:%.*]] = insertelement <4 x i1> [[TMP45]], i1 [[TMP42]], i32 2
+; CHECK-NEXT:    [[TMP47:%.*]] = insertelement <4 x i1> [[TMP46]], i1 [[TMP43]], i32 3
+; CHECK-NEXT:    [[TMP48:%.*]] = load i1, ptr [[TMP24]], align 1
+; CHECK-NEXT:    [[TMP49:%.*]] = load i1, ptr [[TMP25]], align 1
+; CHECK-NEXT:    [[TMP50:%.*]] = load i1, ptr [[TMP26]], align 1
+; CHECK-NEXT:    [[TMP51:%.*]] = load i1, ptr [[TMP27]], align 1
+; CHECK-NEXT:    [[TMP52:%.*]] = insertelement <4 x i1> poison, i1 [[TMP48]], i32 0
+; CHECK-NEXT:    [[TMP53:%.*]] = insertelement <4 x i1> [[TMP52]], i1 [[TMP49]], i32 1
+; CHECK-NEXT:    [[TMP54:%.*]] = insertelement <4 x i1> [[TMP53]], i1 [[TMP50]], i32 2
+; CHECK-NEXT:    [[TMP55:%.*]] = insertelement <4 x i1> [[TMP54]], i1 [[TMP51]], i32 3
+; CHECK-NEXT:    [[TMP56:%.*]] = load i1, ptr [[TMP28]], align 1
+; CHECK-NEXT:    [[TMP57:%.*]] = load i1, ptr [[TMP29]], align 1
+; CHECK-NEXT:    [[TMP58:%.*]] = load i1, ptr [[TMP30]], align 1
+; CHECK-NEXT:    [[TMP59:%.*]] = load i1, ptr [[TMP31]], align 1
+; CHECK-NEXT:    [[TMP60:%.*]] = insertelement <4 x i1> poison, i1 [[TMP56]], i32 0
+; CHECK-NEXT:    [[TMP61:%.*]] = insertelement <4 x i1> [[TMP60]], i1 [[TMP57]], i32 1
+; CHECK-NEXT:    [[TMP62:%.*]] = insertelement <4 x i1> [[TMP61]], i1 [[TMP58]], i32 2
+; CHECK-NEXT:    [[TMP63:%.*]] = insertelement <4 x i1> [[TMP62]], i1 [[TMP59]], i32 3
+; CHECK-NEXT:    [[TMP64:%.*]] = extractelement <4 x i1> [[TMP39]], i32 0
+; CHECK-NEXT:    br i1 [[TMP64]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
+; CHECK:       pred.load.if:
+; CHECK-NEXT:    [[TMP65:%.*]] = add i64 [[TMP0]], 2
+; CHECK-NEXT:    [[TMP66:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP65]]
+; CHECK-NEXT:    [[TMP67:%.*]] = load i32, ptr [[TMP66]], align 4
+; CHECK-NEXT:    [[TMP68:%.*]] = insertelement <4 x i32> poison, i32 [[TMP67]], i32 0
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE]]
+; CHECK:       pred.load.continue:
+; CHECK-NEXT:    [[TMP69:%.*]] = phi <4 x i32> [ poison, [[VECTOR_BODY]] ], [ [[TMP68]], [[PRED_LOAD_IF]] ]
+; CHECK-NEXT:    [[TMP70:%.*]] = extractelement <4 x i1> [[TMP39]], i32 1
+; CHECK-NEXT:    br i1 [[TMP70]], label [[PRED_LOAD_IF4:%.*]], label [[PRED_LOAD_CONTINUE5:%.*]]
+; CHECK:       pred.load.if4:
+; CHECK-NEXT:    [[TMP71:%.*]] = add i64 [[TMP1]], 2
+; CHECK-NEXT:    [[TMP72:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP71]]
+; CHECK-NEXT:    [[TMP73:%.*]] = load i32, ptr [[TMP72]], align 4
+; CHECK-NEXT:    [[TMP74:%.*]] = insertelement <4 x i32> [[TMP69]], i32 [[TMP73]], i32 1
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE5]]
+; CHECK:       pred.load.continue5:
+; CHECK-NEXT:    [[TMP75:%.*]] = phi <4 x i32> [ [[TMP69]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP74]], [[PRED_LOAD_IF4]] ]
+; CHECK-NEXT:    [[TMP76:%.*]] = extractelement <4 x i1> [[TMP39]], i32 2
+; CHECK-NEXT:    br i1 [[TMP76]], label [[PRED_LOAD_IF6:%.*]], label [[PRED_LOAD_CONTINUE7:%.*]]
+; CHECK:       pred.load.if6:
+; CHECK-NEXT:    [[TMP77:%.*]] = add i64 [[TMP2]], 2
+; CHECK-NEXT:    [[TMP78:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP77]]
+; CHECK-NEXT:    [[TMP79:%.*]] = load i32, ptr [[TMP78]], align 4
+; CHECK-NEXT:    [[TMP80:%.*]] = insertelement <4 x i32> [[TMP75]], i32 [[TMP79]], i32 2
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE7]]
+; CHECK:       pred.load.continue7:
+; CHECK-NEXT:    [[TMP81:%.*]] = phi <4 x i32> [ [[TMP75]], [[PRED_LOAD_CONTINUE5]] ], [ [[TMP80]], [[PRED_LOAD_IF6]] ]
+; CHECK-NEXT:    [[TMP82:%.*]] = extractelement <4 x i1> [[TMP39]], i32 3
+; CHECK-NEXT:    br i1 [[TMP82]], label [[PRED_LOAD_IF8:%.*]], label [[PRED_LOAD_CONTINUE9:%.*]]
+; CHECK:       pred.load.if8:
+; CHECK-NEXT:    [[TMP83:%.*]] = add i64 [[TMP3]], 2
+; CHECK-NEXT:    [[TMP84:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP83]]
+; CHECK-NEXT:    [[TMP85:%.*]] = load i32, ptr [[TMP84]], align 4
+; CHECK-NEXT:    [[TMP86:%.*]] = insertelement <4 x i32> [[TMP81]], i32 [[TMP85]], i32 3
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE9]]
+; CHECK:       pred.load.continue9:
+; CHECK-NEXT:    [[TMP87:%.*]] = phi <4 x i32> [ [[TMP81]], [[PRED_LOAD_CONTINUE7]] ], [ [[TMP86]], [[PRED_LOAD_IF8]] ]
+; CHECK-NEXT:    [[TMP88:%.*]] = extractelement <4 x i1> [[TMP47]], i32 0
+; CHECK-NEXT:    br i1 [[TMP88]], label [[PRED_LOAD_IF10:%.*]], label [[PRED_LOAD_CONTINUE11:%.*]]
+; CHECK:       pred.load.if10:
+; CHECK-NEXT:    [[TMP89:%.*]] = add i64 [[TMP4]], 2
+; CHECK-NEXT:    [[TMP90:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP89]]
+; CHECK-NEXT:    [[TMP91:%.*]] = load i32, ptr [[TMP90]], align 4
+; CHECK-NEXT:    [[TMP92:%.*]] = insertelement <4 x i32> poison, i32 [[TMP91]], i32 0
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE11]]
+; CHECK:       pred.load.continue11:
+; CHECK-NEXT:    [[TMP93:%.*]] = phi <4 x i32> [ poison, [[PRED_LOAD_CONTINUE9]] ], [ [[TMP92]], [[PRED_LOAD_IF10]] ]
+; CHECK-NEXT:    [[TMP94:%.*]] = extractelement <4 x i1> [[TMP47]], i32 1
+; CHECK-NEXT:    br i1 [[TMP94]], label [[PRED_LOAD_IF12:%.*]], label [[PRED_LOAD_CONTINUE13:%.*]]
+; CHECK:       pred.load.if12:
+; CHECK-NEXT:    [[TMP95:%.*]] = add i64 [[TMP5]], 2
+; CHECK-NEXT:    [[TMP96:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP95]]
+; CHECK-NEXT:    [[TMP97:%.*]] = load i32, ptr [[TMP96]], align 4
+; CHECK-NEXT:    [[TMP98:%.*]] = insertelement <4 x i32> [[TMP93]], i32 [[TMP97]], i32 1
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE13]]
+; CHECK:       pred.load.continue13:
+; CHECK-NEXT:    [[TMP99:%.*]] = phi <4 x i32> [ [[TMP93]], [[PRED_LOAD_CONTINUE11]] ], [ [[TMP98]], [[PRED_LOAD_IF12]] ]
+; CHECK-NEXT:    [[TMP100:%.*]] = extractelement <4 x i1> [[TMP47]], i32 2
+; CHECK-NEXT:    br i1 [[TMP100]], label [[PRED_LOAD_IF14:%.*]], label [[PRED_LOAD_CONTINUE15:%.*]]
+; CHECK:       pred.load.if14:
+; CHECK-NEXT:    [[TMP101:%.*]] = add i64 [[TMP6]], 2
+; CHECK-NEXT:    [[TMP102:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP101]]
+; CHECK-NEXT:    [[TMP103:%.*]] = load i32, ptr [[TMP102]], align 4
+; CHECK-NEXT:    [[TMP104:%.*]] = insertelement <4 x i32> [[TMP99]], i32 [[TMP103]], i32 2
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE15]]
+; CHECK:       pred.load.continue15:
+; CHECK-NEXT:    [[TMP105:%.*]] = phi <4 x i32> [ [[TMP99]], [[PRED_LOAD_CONTINUE13]] ], [ [[TMP104]], [[PRED_LOAD_IF14]] ]
+; CHECK-NEXT:    [[TMP106:%.*]] = extractelement <4 x i1> [[TMP47]], i32 3
+; CHECK-NEXT:    br i1 [[TMP106]], label [[PRED_LOAD_IF16:%.*]], label [[PRED_LOAD_CONTINUE17:%.*]]
+; CHECK:       pred.load.if16:
+; CHECK-NEXT:    [[TMP107:%.*]] = add i64 [[TMP7]], 2
+; CHECK-NEXT:    [[TMP108:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP107]]
+; CHECK-NEXT:    [[TMP109:%.*]] = load i32, ptr [[TMP108]], align 4
+; CHECK-NEXT:    [[TMP110:%.*]] = insertelement <4 x i32> [[TMP105]], i32 [[TMP109]], i32 3
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE17]]
+; CHECK:       pred.load.continue17:
+; CHECK-NEXT:    [[TMP111:%.*]] = phi <4 x i32> [ [[TMP105]], [[PRED_LOAD_CONTINUE15]] ], [ [[TMP110]], [[PRED_LOAD_IF16]] ]
+; CHECK-NEXT:    [[TMP112:%.*]] = extractelement <4 x i1> [[TMP55]], i32 0
+; CHECK-NEXT:    br i1 [[TMP112]], label [[PRED_LOAD_IF18:%.*]], label [[PRED_LOAD_CONTINUE19:%.*]]
+; CHECK:       pred.load.if18:
+; CHECK-NEXT:    [[TMP113:%.*]] = add i64 [[TMP8]], 2
+; CHECK-NEXT:    [[TMP114:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP113]]
+; CHECK-NEXT:    [[TMP115:%.*]] = load i32, ptr [[TMP114]], align 4
+; CHECK-NEXT:    [[TMP116:%.*]] = insertelement <4 x i32> poison, i32 [[TMP115]], i32 0
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE19]]
+; CHECK:       pred.load.continue19:
+; CHECK-NEXT:    [[TMP117:%.*]] = phi <4 x i32> [ poison, [[PRED_LOAD_CONTINUE17]] ], [ [[TMP116]], [[PRED_LOAD_IF18]] ]
+; CHECK-NEXT:    [[TMP118:%.*]] = extractelement <4 x i1> [[TMP55]], i32 1
+; CHECK-NEXT:    br i1 [[TMP118]], label [[PRED_LOAD_IF20:%.*]], label [[PRED_LOAD_CONTINUE21:%.*]]
+; CHECK:       pred.load.if20:
+; CHECK-NEXT:    [[TMP119:%.*]] = add i64 [[TMP9]], 2
+; CHECK-NEXT:    [[TMP120:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP119]]
+; CHECK-NEXT:    [[TMP121:%.*]] = load i32, ptr [[TMP120]], align 4
+; CHECK-NEXT:    [[TMP122:%.*]] = insertelement <4 x i32> [[TMP117]], i32 [[TMP121]], i32 1
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE21]]
+; CHECK:       pred.load.continue21:
+; CHECK-NEXT:    [[TMP123:%.*]] = phi <4 x i32> [ [[TMP117]], [[PRED_LOAD_CONTINUE19]] ], [ [[TMP122]], [[PRED_LOAD_IF20]] ]
+; CHECK-NEXT:    [[TMP124:%.*]] = extractelement <4 x i1> [[TMP55]], i32 2
+; CHECK-NEXT:    br i1 [[TMP124]], label [[PRED_LOAD_IF22:%.*]], label [[PRED_LOAD_CONTINUE23:%.*]]
+; CHECK:       pred.load.if22:
+; CHECK-NEXT:    [[TMP125:%.*]] = add i64 [[TMP10]], 2
+; CHECK-NEXT:    [[TMP126:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP125]]
+; CHECK-NEXT:    [[TMP127:%.*]] = load i32, ptr [[TMP126]], align 4
+; CHECK-NEXT:    [[TMP128:%.*]] = insertelement <4 x i32> [[TMP123]], i32 [[TMP127]], i32 2
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE23]]
+; CHECK:       pred.load.continue23:
+; CHECK-NEXT:    [[TMP129:%.*]] = phi <4 x i32> [ [[TMP123]], [[PRED_LOAD_CONTINUE21]] ], [ [[TMP128]], [[PRED_LOAD_IF22]] ]
+; CHECK-NEXT:    [[TMP130:%.*]] = extractelement <4 x i1> [[TMP55]], i32 3
+; CHECK-NEXT:    br i1 [[TMP130]], label [[PRED_LOAD_IF24:%.*]], label [[PRED_LOAD_CONTINUE25:%.*]]
+; CHECK:       pred.load.if24:
+; CHECK-NEXT:    [[TMP131:%.*]] = add i64 [[TMP11]], 2
+; CHECK-NEXT:    [[TMP132:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP131]]
+; CHECK-NEXT:    [[TMP133:%.*]] = load i32, ptr [[TMP132]], align 4
+; CHECK-NEXT:    [[TMP134:%.*]] = insertelement <4 x i32> [[TMP129]], i32 [[TMP133]], i32 3
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE25]]
+; CHECK:       pred.load.continue25:
+; CHECK-NEXT:    [[TMP135:%.*]] = phi <4 x i32> [ [[TMP129]], [[PRED_LOAD_CONTINUE23]] ], [ [[TMP134]], [[PRED_LOAD_IF24]] ]
+; CHECK-NEXT:    [[TMP136:%.*]] = extractelement <4 x i1> [[TMP63]], i32 0
+; CHECK-NEXT:    br i1 [[TMP136]], label [[PRED_LOAD_IF26:%.*]], label [[PRED_LOAD_CONTINUE27:%.*]]
+; CHECK:       pred.load.if26:
+; CHECK-NEXT:    [[TMP137:%.*]] = add i64 [[TMP12]], 2
+; CHECK-NEXT:    [[TMP138:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP137]]
+; CHECK-NEXT:    [[TMP139:%.*]] = load i32, ptr [[TMP138]], align 4
+; CHECK-NEXT:    [[TMP140:%.*]] = insertelement <4 x i32> poison, i32 [[TMP139]], i32 0
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE27]]
+; CHECK:       pred.load.continue27:
+; CHECK-NEXT:    [[TMP141:%.*]] = phi <4 x i32> [ poison, [[PRED_LOAD_CONTINUE25]] ], [ [[TMP140]], [[PRED_LOAD_IF26]] ]
+; CHECK-NEXT:    [[TMP142:%.*]] = extractelement <4 x i1> [[TMP63]], i32 1
+; CHECK-NEXT:    br i1 [[TMP142]], label [[PRED_LOAD_IF28:%.*]], label [[PRED_LOAD_CONTINUE29:%.*]]
+; CHECK:       pred.load.if28:
+; CHECK-NEXT:    [[TMP143:%.*]] = add i64 [[TMP13]], 2
+; CHECK-NEXT:    [[TMP144:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP143]]
+; CHECK-NEXT:    [[TMP145:%.*]] = load i32, ptr [[TMP144]], align 4
+; CHECK-NEXT:    [[TMP146:%.*]] = insertelement <4 x i32> [[TMP141]], i32 [[TMP145]], i32 1
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE29]]
+; CHECK:       pred.load.continue29:
+; CHECK-NEXT:    [[TMP147:%.*]] = phi <4 x i32> [ [[TMP141]], [[PRED_LOAD_CONTINUE27]] ], [ [[TMP146]], [[PRED_LOAD_IF28]] ]
+; CHECK-NEXT:    [[TMP148:%.*]] = extractelement <4 x i1> [[TMP63]], i32 2
+; CHECK-NEXT:    br i1 [[TMP148]], label [[PRED_LOAD_IF30:%.*]], label [[PRED_LOAD_CONTINUE31:%.*]]
+; CHECK:       pred.load.if30:
+; CHECK-NEXT:    [[TMP149:%.*]] = add i64 [[TMP14]], 2
+; CHECK-NEXT:    [[TMP150:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP149]]
+; CHECK-NEXT:    [[TMP151:%.*]] = load i32, ptr [[TMP150]], align 4
+; CHECK-NEXT:    [[TMP152:%.*]] = insertelement <4 x i32> [[TMP147]], i32 [[TMP151]], i32 2
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE31]]
+; CHECK:       pred.load.continue31:
+; CHECK-NEXT:    [[TMP153:%.*]] = phi <4 x i32> [ [[TMP147]], [[PRED_LOAD_CONTINUE29]] ], [ [[TMP152]], [[PRED_LOAD_IF30]] ]
+; CHECK-NEXT:    [[TMP154:%.*]] = extractelement <4 x i1> [[TMP63]], i32 3
+; CHECK-NEXT:    br i1 [[TMP154]], label [[PRED_LOAD_IF32:%.*]], label [[PRED_LOAD_CONTINUE33]]
+; CHECK:       pred.load.if32:
+; CHECK-NEXT:    [[TMP155:%.*]] = add i64 [[TMP15]], 2
+; CHECK-NEXT:    [[TMP156:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[TMP155]]
+; CHECK-NEXT:    [[TMP157:%.*]] = load i32, ptr [[TMP156]], align 4
+; CHECK-NEXT:    [[TMP158:%.*]] = insertelement <4 x i32> [[TMP153]], i32 [[TMP157]], i32 3
+; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE33]]
+; CHECK:       pred.load.continue33:
+; CHECK-NEXT:    [[TMP159:%.*]] = phi <4 x i32> [ [[TMP153]], [[PRED_LOAD_CONTINUE31]] ], [ [[TMP158]], [[PRED_LOAD_IF32]] ]
+; CHECK-NEXT:    [[TMP160:%.*]] = xor <4 x i1> [[TMP39]], <i1 true, i1 true, i1 true, i1 true>
+; CHECK-NEXT:    [[TMP161:%.*]] = xor <4 x i1> [[TMP47]], <i1 true, i1 true, i1 true, i1 true>
+; CHECK-NEXT:    [[TMP162:%.*]] = xor <4 x i1> [[TMP55]], <i1 true, i1 true, i1 true, i1 true>
+; CHECK-NEXT:    [[TMP163:%.*]] = xor <4 x i1> [[TMP63]], <i1 true, i1 true, i1 true, i1 true>
+; CHECK-NEXT:    [[PREDPHI:%.*]] = select <4 x i1> [[TMP39]], <4 x i32> [[TMP87]], <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[PREDPHI34:%.*]] = select <4 x i1> [[TMP47]], <4 x i32> [[TMP111]], <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[PREDPHI35:%.*]] = select <4 x i1> [[TMP55]], <4 x i32> [[TMP135]], <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[PREDPHI36:%.*]] = select <4 x i1> [[TMP63]], <4 x i32> [[TMP159]], <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP164]] = add <4 x i32> [[VEC_PHI]], [[PREDPHI]]
+; CHECK-NEXT:    [[TMP165]] = add <4 x i32> [[VEC_PHI1]], [[PREDPHI34]]
+; CHECK-NEXT:    [[TMP166]] = add <4 x i32> [[VEC_PHI2]], [[PREDPHI35]]
+; CHECK-NEXT:    [[TMP167]] = add <4 x i32> [[VEC_PHI3]], [[PREDPHI36]]
+; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 16
+; CHECK-NEXT:    [[TMP168:%.*]] = icmp eq i64 [[INDEX_NEXT]], 144
+; CHECK-NEXT:    br i1 [[TMP168]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP38:![0-9]+]]
+; CHECK:       middle.block:
+; CHECK-NEXT:    [[BIN_RDX:%.*]] = add <4 x i32> [[TMP165]], [[TMP164]]
+; CHECK-NEXT:    [[BIN_RDX37:%.*]] = add <4 x i32> [[TMP166]], [[BIN_RDX]]
+; CHECK-NEXT:    [[BIN_RDX38:%.*]] = add <4 x i32> [[TMP167]], [[BIN_RDX37]]
+; CHECK-NEXT:    [[TMP169:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[BIN_RDX38]])
+; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 152, 144
+; CHECK-NEXT:    br i1 [[CMP_N]], label [[LOOP_EXIT:%.*]], label [[SCALAR_PH]]
+; CHECK:       scalar.ph:
+; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ 288, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    [[BC_MERGE_RDX:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[TMP169]], [[MIDDLE_BLOCK]] ]
+; CHECK-NEXT:    br label [[LOOP:%.*]]
+; CHECK:       loop:
+; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[IV_NEXT:%.*]], [[LATCH:%.*]] ]
+; CHECK-NEXT:    [[ACCUM:%.*]] = phi i32 [ [[BC_MERGE_RDX]], [[SCALAR_PH]] ], [ [[ACCUM_NEXT:%.*]], [[LATCH]] ]
+; CHECK-NEXT:    [[IV_NEXT]] = add i64 [[IV]], 2
+; CHECK-NEXT:    [[TEST_ADDR:%.*]] = getelementptr inbounds i1, ptr [[TEST_BASE]], i64 [[IV]]
+; CHECK-NEXT:    [[EARLYCND:%.*]] = load i1, ptr [[TEST_ADDR]], align 1
+; CHECK-NEXT:    br i1 [[EARLYCND]], label [[PRED:%.*]], label [[LATCH]]
+; CHECK:       pred:
+; CHECK-NEXT:    [[ADDR:%.*]] = getelementptr inbounds i32, ptr [[ALLOCA]], i64 [[IV_NEXT]]
+; CHECK-NEXT:    [[VAL:%.*]] = load i32, ptr [[ADDR]], align 4
+; CHECK-NEXT:    br label [[LATCH]]
+; CHECK:       latch:
+; CHECK-NEXT:    [[VAL_PHI:%.*]] = phi i32 [ 0, [[LOOP]] ], [ [[VAL]], [[PRED]] ]
+; CHECK-NEXT:    [[ACCUM_NEXT]] = add i32 [[ACCUM]], [[VAL_PHI]]
+; CHECK-NEXT:    [[EXIT:%.*]] = icmp ugt i64 [[IV]], 300
+; CHECK-NEXT:    br i1 [[EXIT]], label [[LOOP_EXIT]], label [[LOOP]], !llvm.loop [[LOOP39:![0-9]+]]
+; CHECK:       loop_exit:
+; CHECK-NEXT:    [[ACCUM_NEXT_LCSSA:%.*]] = phi i32 [ [[ACCUM_NEXT]], [[LATCH]] ], [ [[TMP169]], [[MIDDLE_BLOCK]] ]
+; CHECK-NEXT:    ret i32 [[ACCUM_NEXT_LCSSA]]
+;
+entry:
+  %alloca = alloca [4096 x i32]
+  call void @init(ptr %alloca)
+  br label %loop
+loop:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %latch ]
+  %accum = phi i32 [ 0, %entry ], [ %accum.next, %latch ]
+  %iv.next = add i64 %iv, 2
+  %test_addr = getelementptr inbounds i1, ptr %test_base, i64 %iv
+  %earlycnd = load i1, ptr %test_addr
+  br i1 %earlycnd, label %pred, label %latch
+pred:
+  %addr = getelementptr inbounds i32, ptr %alloca, i64 %iv.next
+  %val = load i32, ptr %addr
+  br label %latch
+latch:
+  %val.phi = phi i32 [0, %loop], [%val, %pred]
+  %accum.next = add i32 %accum, %val.phi
+  %exit = icmp ugt i64 %iv,300
+  br i1 %exit, label %loop_exit, label %loop
+
+loop_exit:
+  ret i32 %accum.next
+}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/x86-interleaved-accesses-masked-group.ll b/llvm/test/Transforms/LoopVectorize/X86/x86-interleaved-accesses-masked-group.ll
index 6ed4cda9fb282..bb76b4d6c1dc7 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/x86-interleaved-accesses-masked-group.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/x86-interleaved-accesses-masked-group.ll
@@ -125,8 +125,8 @@ define dso_local void @masked_strided1(ptr noalias nocapture readonly %p, ptr no
 ; DISABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v8i8.p0(<8 x i8> [[TMP49]], ptr [[TMP50]], i32 1, <8 x i1> [[TMP0]])
 ; DISABLED_MASKED_STRIDED-NEXT:    [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 8
 ; DISABLED_MASKED_STRIDED-NEXT:    [[VEC_IND_NEXT]] = add <8 x i32> [[VEC_IND]], <i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8>
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP52:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP52]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP51:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP51]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
 ; DISABLED_MASKED_STRIDED:       for.end:
 ; DISABLED_MASKED_STRIDED-NEXT:    ret void
 ;
@@ -145,12 +145,12 @@ define dso_local void @masked_strided1(ptr noalias nocapture readonly %p, ptr no
 ; ENABLED_MASKED_STRIDED-NEXT:    [[INTERLEAVED_MASK:%.*]] = shufflevector <8 x i1> [[TMP0]], <8 x i1> poison, <16 x i32> <i32 0, i32 0, i32 1, i32 1, i32 2, i32 2, i32 3, i32 3, i32 4, i32 4, i32 5, i32 5, i32 6, i32 6, i32 7, i32 7>
 ; ENABLED_MASKED_STRIDED-NEXT:    [[WIDE_MASKED_VEC:%.*]] = call <16 x i8> @llvm.masked.load.v16i8.p0(ptr [[TMP2]], i32 1, <16 x i1> [[INTERLEAVED_MASK]], <16 x i8> poison)
 ; ENABLED_MASKED_STRIDED-NEXT:    [[STRIDED_VEC:%.*]] = shufflevector <16 x i8> [[WIDE_MASKED_VEC]], <16 x i8> poison, <8 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP4:%.*]] = getelementptr i8, ptr [[Q:%.*]], i32 [[INDEX]]
-; ENABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v8i8.p0(<8 x i8> [[STRIDED_VEC]], ptr [[TMP4]], i32 1, <8 x i1> [[TMP0]])
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP3:%.*]] = getelementptr i8, ptr [[Q:%.*]], i32 [[INDEX]]
+; ENABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v8i8.p0(<8 x i8> [[STRIDED_VEC]], ptr [[TMP3]], i32 1, <8 x i1> [[TMP0]])
 ; ENABLED_MASKED_STRIDED-NEXT:    [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 8
 ; ENABLED_MASKED_STRIDED-NEXT:    [[VEC_IND_NEXT]] = add <8 x i32> [[VEC_IND]], <i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP6:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1016
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP6]], label [[FOR_BODY:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP4:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1016
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP4]], label [[FOR_BODY:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
 ; ENABLED_MASKED_STRIDED:       for.body:
 ; ENABLED_MASKED_STRIDED-NEXT:    [[IX_09:%.*]] = phi i32 [ [[INC:%.*]], [[FOR_INC:%.*]] ], [ 1016, [[VECTOR_BODY]] ]
 ; ENABLED_MASKED_STRIDED-NEXT:    [[CMP1:%.*]] = icmp ugt i32 [[IX_09]], [[CONV]]
@@ -158,14 +158,14 @@ define dso_local void @masked_strided1(ptr noalias nocapture readonly %p, ptr no
 ; ENABLED_MASKED_STRIDED:       if.then:
 ; ENABLED_MASKED_STRIDED-NEXT:    [[MUL:%.*]] = shl nuw nsw i32 [[IX_09]], 1
 ; ENABLED_MASKED_STRIDED-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[MUL]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP7:%.*]] = load i8, ptr [[ARRAYIDX]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP5:%.*]] = load i8, ptr [[ARRAYIDX]], align 1
 ; ENABLED_MASKED_STRIDED-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[IX_09]]
-; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP7]], ptr [[ARRAYIDX3]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP5]], ptr [[ARRAYIDX3]], align 1
 ; ENABLED_MASKED_STRIDED-NEXT:    br label [[FOR_INC]]
 ; ENABLED_MASKED_STRIDED:       for.inc:
 ; ENABLED_MASKED_STRIDED-NEXT:    [[INC]] = add nuw nsw i32 [[IX_09]], 1
 ; ENABLED_MASKED_STRIDED-NEXT:    [[EXITCOND:%.*]] = icmp eq i32 [[INC]], 1024
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[EXITCOND]], label [[FOR_END:%.*]], label [[FOR_BODY]], !llvm.loop [[LOOP2:![0-9]+]]
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[EXITCOND]], label [[FOR_END:%.*]], label [[FOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
 ; ENABLED_MASKED_STRIDED:       for.end:
 ; ENABLED_MASKED_STRIDED-NEXT:    ret void
 ;
@@ -298,8 +298,8 @@ define dso_local void @masked_strided1_optsize(ptr noalias nocapture readonly %p
 ; DISABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v8i8.p0(<8 x i8> [[TMP49]], ptr [[TMP50]], i32 1, <8 x i1> [[TMP0]])
 ; DISABLED_MASKED_STRIDED-NEXT:    [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 8
 ; DISABLED_MASKED_STRIDED-NEXT:    [[VEC_IND_NEXT]] = add <8 x i32> [[VEC_IND]], <i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8>
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP52:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP52]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP2:![0-9]+]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP51:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP51]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
 ; DISABLED_MASKED_STRIDED:       for.end:
 ; DISABLED_MASKED_STRIDED-NEXT:    ret void
 ;
@@ -316,15 +316,15 @@ define dso_local void @masked_strided1_optsize(ptr noalias nocapture readonly %p
 ; ENABLED_MASKED_STRIDED-NEXT:    [[TMP1:%.*]] = shl i32 [[INDEX]], 1
 ; ENABLED_MASKED_STRIDED-NEXT:    [[TMP2:%.*]] = getelementptr i8, ptr [[P:%.*]], i32 [[TMP1]]
 ; ENABLED_MASKED_STRIDED-NEXT:    [[INTERLEAVED_MASK:%.*]] = shufflevector <8 x i1> [[TMP0]], <8 x i1> poison, <16 x i32> <i32 0, i32 0, i32 1, i32 1, i32 2, i32 2, i32 3, i32 3, i32 4, i32 4, i32 5, i32 5, i32 6, i32 6, i32 7, i32 7>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP4:%.*]] = and <16 x i1> [[INTERLEAVED_MASK]], <i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false>
-; ENABLED_MASKED_STRIDED-NEXT:    [[WIDE_MASKED_VEC:%.*]] = call <16 x i8> @llvm.masked.load.v16i8.p0(ptr [[TMP2]], i32 1, <16 x i1> [[TMP4]], <16 x i8> poison)
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP3:%.*]] = and <16 x i1> [[INTERLEAVED_MASK]], <i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false>
+; ENABLED_MASKED_STRIDED-NEXT:    [[WIDE_MASKED_VEC:%.*]] = call <16 x i8> @llvm.masked.load.v16i8.p0(ptr [[TMP2]], i32 1, <16 x i1> [[TMP3]], <16 x i8> poison)
 ; ENABLED_MASKED_STRIDED-NEXT:    [[STRIDED_VEC:%.*]] = shufflevector <16 x i8> [[WIDE_MASKED_VEC]], <16 x i8> poison, <8 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP5:%.*]] = getelementptr i8, ptr [[Q:%.*]], i32 [[INDEX]]
-; ENABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v8i8.p0(<8 x i8> [[STRIDED_VEC]], ptr [[TMP5]], i32 1, <8 x i1> [[TMP0]])
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP4:%.*]] = getelementptr i8, ptr [[Q:%.*]], i32 [[INDEX]]
+; ENABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v8i8.p0(<8 x i8> [[STRIDED_VEC]], ptr [[TMP4]], i32 1, <8 x i1> [[TMP0]])
 ; ENABLED_MASKED_STRIDED-NEXT:    [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 8
 ; ENABLED_MASKED_STRIDED-NEXT:    [[VEC_IND_NEXT]] = add <8 x i32> [[VEC_IND]], <i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP7:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP7]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP4:![0-9]+]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP5:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP5]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP4:![0-9]+]]
 ; ENABLED_MASKED_STRIDED:       for.end:
 ; ENABLED_MASKED_STRIDED-NEXT:    ret void
 ;
@@ -487,8 +487,8 @@ define dso_local void @masked_strided1_optsize_unknown_tc(ptr noalias nocapture
 ; DISABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v8i8.p0(<8 x i8> [[TMP51]], ptr [[TMP52]], i32 1, <8 x i1> [[TMP3]])
 ; DISABLED_MASKED_STRIDED-NEXT:    [[INDEX_NEXT]] = add i32 [[INDEX]], 8
 ; DISABLED_MASKED_STRIDED-NEXT:    [[VEC_IND_NEXT]] = add <8 x i32> [[VEC_IND]], <i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8>
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP54:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP54]], label [[FOR_END]], label [[VECTOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP53:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP53]], label [[FOR_END]], label [[VECTOR_BODY]], !llvm.loop [[LOOP4:![0-9]+]]
 ; DISABLED_MASKED_STRIDED:       for.end:
 ; DISABLED_MASKED_STRIDED-NEXT:    ret void
 ;
@@ -515,15 +515,15 @@ define dso_local void @masked_strided1_optsize_unknown_tc(ptr noalias nocapture
 ; ENABLED_MASKED_STRIDED-NEXT:    [[TMP3:%.*]] = getelementptr i8, ptr [[P:%.*]], i32 [[TMP2]]
 ; ENABLED_MASKED_STRIDED-NEXT:    [[TMP4:%.*]] = select <8 x i1> [[TMP0]], <8 x i1> [[TMP1]], <8 x i1> zeroinitializer
 ; ENABLED_MASKED_STRIDED-NEXT:    [[INTERLEAVED_MASK:%.*]] = shufflevector <8 x i1> [[TMP4]], <8 x i1> poison, <16 x i32> <i32 0, i32 0, i32 1, i32 1, i32 2, i32 2, i32 3, i32 3, i32 4, i32 4, i32 5, i32 5, i32 6, i32 6, i32 7, i32 7>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP6:%.*]] = and <16 x i1> [[INTERLEAVED_MASK]], <i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false>
-; ENABLED_MASKED_STRIDED-NEXT:    [[WIDE_MASKED_VEC:%.*]] = call <16 x i8> @llvm.masked.load.v16i8.p0(ptr [[TMP3]], i32 1, <16 x i1> [[TMP6]], <16 x i8> poison)
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP5:%.*]] = and <16 x i1> [[INTERLEAVED_MASK]], <i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false>
+; ENABLED_MASKED_STRIDED-NEXT:    [[WIDE_MASKED_VEC:%.*]] = call <16 x i8> @llvm.masked.load.v16i8.p0(ptr [[TMP3]], i32 1, <16 x i1> [[TMP5]], <16 x i8> poison)
 ; ENABLED_MASKED_STRIDED-NEXT:    [[STRIDED_VEC:%.*]] = shufflevector <16 x i8> [[WIDE_MASKED_VEC]], <16 x i8> poison, <8 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP7:%.*]] = getelementptr i8, ptr [[Q:%.*]], i32 [[INDEX]]
-; ENABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v8i8.p0(<8 x i8> [[STRIDED_VEC]], ptr [[TMP7]], i32 1, <8 x i1> [[TMP4]])
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP6:%.*]] = getelementptr i8, ptr [[Q:%.*]], i32 [[INDEX]]
+; ENABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v8i8.p0(<8 x i8> [[STRIDED_VEC]], ptr [[TMP6]], i32 1, <8 x i1> [[TMP4]])
 ; ENABLED_MASKED_STRIDED-NEXT:    [[INDEX_NEXT]] = add i32 [[INDEX]], 8
 ; ENABLED_MASKED_STRIDED-NEXT:    [[VEC_IND_NEXT]] = add <8 x i32> [[VEC_IND]], <i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP9:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP9]], label [[FOR_END]], label [[VECTOR_BODY]], !llvm.loop [[LOOP5:![0-9]+]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP7:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP7]], label [[FOR_END]], label [[VECTOR_BODY]], !llvm.loop [[LOOP5:![0-9]+]]
 ; ENABLED_MASKED_STRIDED:       for.end:
 ; ENABLED_MASKED_STRIDED-NEXT:    ret void
 ;
@@ -691,8 +691,8 @@ define dso_local void @masked_strided3_optsize_unknown_tc(ptr noalias nocapture
 ; DISABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v8i8.p0(<8 x i8> [[TMP51]], ptr [[TMP52]], i32 1, <8 x i1> [[TMP3]])
 ; DISABLED_MASKED_STRIDED-NEXT:    [[INDEX_NEXT]] = add i32 [[INDEX]], 8
 ; DISABLED_MASKED_STRIDED-NEXT:    [[VEC_IND_NEXT]] = add <8 x i32> [[VEC_IND]], <i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8>
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP54:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP54]], label [[FOR_END]], label [[VECTOR_BODY]], !llvm.loop [[LOOP4:![0-9]+]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP53:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP53]], label [[FOR_END]], label [[VECTOR_BODY]], !llvm.loop [[LOOP5:![0-9]+]]
 ; DISABLED_MASKED_STRIDED:       for.end:
 ; DISABLED_MASKED_STRIDED-NEXT:    ret void
 ;
@@ -719,15 +719,15 @@ define dso_local void @masked_strided3_optsize_unknown_tc(ptr noalias nocapture
 ; ENABLED_MASKED_STRIDED-NEXT:    [[TMP3:%.*]] = getelementptr i8, ptr [[P:%.*]], i32 [[TMP2]]
 ; ENABLED_MASKED_STRIDED-NEXT:    [[TMP4:%.*]] = select <8 x i1> [[TMP0]], <8 x i1> [[TMP1]], <8 x i1> zeroinitializer
 ; ENABLED_MASKED_STRIDED-NEXT:    [[INTERLEAVED_MASK:%.*]] = shufflevector <8 x i1> [[TMP4]], <8 x i1> poison, <24 x i32> <i32 0, i32 0, i32 0, i32 1, i32 1, i32 1, i32 2, i32 2, i32 2, i32 3, i32 3, i32 3, i32 4, i32 4, i32 4, i32 5, i32 5, i32 5, i32 6, i32 6, i32 6, i32 7, i32 7, i32 7>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP6:%.*]] = and <24 x i1> [[INTERLEAVED_MASK]], <i1 true, i1 false, i1 false, i1 true, i1 false, i1 false, i1 true, i1 false, i1 false, i1 true, i1 false, i1 false, i1 true, i1 false, i1 false, i1 true, i1 false, i1 false, i1 true, i1 false, i1 false, i1 true, i1 false, i1 false>
-; ENABLED_MASKED_STRIDED-NEXT:    [[WIDE_MASKED_VEC:%.*]] = call <24 x i8> @llvm.masked.load.v24i8.p0(ptr [[TMP3]], i32 1, <24 x i1> [[TMP6]], <24 x i8> poison)
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP5:%.*]] = and <24 x i1> [[INTERLEAVED_MASK]], <i1 true, i1 false, i1 false, i1 true, i1 false, i1 false, i1 true, i1 false, i1 false, i1 true, i1 false, i1 false, i1 true, i1 false, i1 false, i1 true, i1 false, i1 false, i1 true, i1 false, i1 false, i1 true, i1 false, i1 false>
+; ENABLED_MASKED_STRIDED-NEXT:    [[WIDE_MASKED_VEC:%.*]] = call <24 x i8> @llvm.masked.load.v24i8.p0(ptr [[TMP3]], i32 1, <24 x i1> [[TMP5]], <24 x i8> poison)
 ; ENABLED_MASKED_STRIDED-NEXT:    [[STRIDED_VEC:%.*]] = shufflevector <24 x i8> [[WIDE_MASKED_VEC]], <24 x i8> poison, <8 x i32> <i32 0, i32 3, i32 6, i32 9, i32 12, i32 15, i32 18, i32 21>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP7:%.*]] = getelementptr i8, ptr [[Q:%.*]], i32 [[INDEX]]
-; ENABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v8i8.p0(<8 x i8> [[STRIDED_VEC]], ptr [[TMP7]], i32 1, <8 x i1> [[TMP4]])
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP6:%.*]] = getelementptr i8, ptr [[Q:%.*]], i32 [[INDEX]]
+; ENABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v8i8.p0(<8 x i8> [[STRIDED_VEC]], ptr [[TMP6]], i32 1, <8 x i1> [[TMP4]])
 ; ENABLED_MASKED_STRIDED-NEXT:    [[INDEX_NEXT]] = add i32 [[INDEX]], 8
 ; ENABLED_MASKED_STRIDED-NEXT:    [[VEC_IND_NEXT]] = add <8 x i32> [[VEC_IND]], <i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP9:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP9]], label [[FOR_END]], label [[VECTOR_BODY]], !llvm.loop [[LOOP6:![0-9]+]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP7:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP7]], label [[FOR_END]], label [[VECTOR_BODY]], !llvm.loop [[LOOP6:![0-9]+]]
 ; ENABLED_MASKED_STRIDED:       for.end:
 ; ENABLED_MASKED_STRIDED-NEXT:    ret void
 ;
@@ -829,8 +829,8 @@ define dso_local void @unconditional_strided1_optsize(ptr noalias nocapture read
 ; DISABLED_MASKED_STRIDED-NEXT:    store <8 x i8> [[TMP32]], ptr [[TMP33]], align 1
 ; DISABLED_MASKED_STRIDED-NEXT:    [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 8
 ; DISABLED_MASKED_STRIDED-NEXT:    [[VEC_IND_NEXT]] = add <8 x i32> [[VEC_IND]], <i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8>
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP35:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP35]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP5:![0-9]+]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP34:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP34]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP6:![0-9]+]]
 ; DISABLED_MASKED_STRIDED:       for.end:
 ; DISABLED_MASKED_STRIDED-NEXT:    ret void
 ;
@@ -843,11 +843,11 @@ define dso_local void @unconditional_strided1_optsize(ptr noalias nocapture read
 ; ENABLED_MASKED_STRIDED-NEXT:    [[TMP1:%.*]] = getelementptr inbounds i8, ptr [[P:%.*]], i32 [[TMP0]]
 ; ENABLED_MASKED_STRIDED-NEXT:    [[WIDE_MASKED_VEC:%.*]] = call <16 x i8> @llvm.masked.load.v16i8.p0(ptr [[TMP1]], i32 1, <16 x i1> <i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false>, <16 x i8> poison)
 ; ENABLED_MASKED_STRIDED-NEXT:    [[STRIDED_VEC:%.*]] = shufflevector <16 x i8> [[WIDE_MASKED_VEC]], <16 x i8> poison, <8 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP3:%.*]] = getelementptr inbounds i8, ptr [[Q:%.*]], i32 [[INDEX]]
-; ENABLED_MASKED_STRIDED-NEXT:    store <8 x i8> [[STRIDED_VEC]], ptr [[TMP3]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP2:%.*]] = getelementptr inbounds i8, ptr [[Q:%.*]], i32 [[INDEX]]
+; ENABLED_MASKED_STRIDED-NEXT:    store <8 x i8> [[STRIDED_VEC]], ptr [[TMP2]], align 1
 ; ENABLED_MASKED_STRIDED-NEXT:    [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 8
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP5:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP5]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP7:![0-9]+]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP3:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP3]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP7:![0-9]+]]
 ; ENABLED_MASKED_STRIDED:       for.end:
 ; ENABLED_MASKED_STRIDED-NEXT:    ret void
 ;
@@ -991,8 +991,8 @@ define dso_local void @unconditional_strided1_optsize_unknown_tc(ptr noalias noc
 ; DISABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v8i8.p0(<8 x i8> [[TMP49]], ptr [[TMP50]], i32 1, <8 x i1> [[TMP0]])
 ; DISABLED_MASKED_STRIDED-NEXT:    [[INDEX_NEXT]] = add i32 [[INDEX]], 8
 ; DISABLED_MASKED_STRIDED-NEXT:    [[VEC_IND_NEXT]] = add <8 x i32> [[VEC_IND]], <i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8>
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP52:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP52]], label [[FOR_END]], label [[VECTOR_BODY]], !llvm.loop [[LOOP6:![0-9]+]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP51:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP51]], label [[FOR_END]], label [[VECTOR_BODY]], !llvm.loop [[LOOP7:![0-9]+]]
 ; DISABLED_MASKED_STRIDED:       for.end:
 ; DISABLED_MASKED_STRIDED-NEXT:    ret void
 ;
@@ -1016,14 +1016,14 @@ define dso_local void @unconditional_strided1_optsize_unknown_tc(ptr noalias noc
 ; ENABLED_MASKED_STRIDED-NEXT:    [[TMP1:%.*]] = shl nuw nsw i32 [[INDEX]], 1
 ; ENABLED_MASKED_STRIDED-NEXT:    [[TMP2:%.*]] = getelementptr inbounds i8, ptr [[P:%.*]], i32 [[TMP1]]
 ; ENABLED_MASKED_STRIDED-NEXT:    [[INTERLEAVED_MASK:%.*]] = shufflevector <8 x i1> [[TMP0]], <8 x i1> poison, <16 x i32> <i32 0, i32 0, i32 1, i32 1, i32 2, i32 2, i32 3, i32 3, i32 4, i32 4, i32 5, i32 5, i32 6, i32 6, i32 7, i32 7>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP4:%.*]] = and <16 x i1> [[INTERLEAVED_MASK]], <i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false>
-; ENABLED_MASKED_STRIDED-NEXT:    [[WIDE_MASKED_VEC:%.*]] = call <16 x i8> @llvm.masked.load.v16i8.p0(ptr [[TMP2]], i32 1, <16 x i1> [[TMP4]], <16 x i8> poison)
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP3:%.*]] = and <16 x i1> [[INTERLEAVED_MASK]], <i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false>
+; ENABLED_MASKED_STRIDED-NEXT:    [[WIDE_MASKED_VEC:%.*]] = call <16 x i8> @llvm.masked.load.v16i8.p0(ptr [[TMP2]], i32 1, <16 x i1> [[TMP3]], <16 x i8> poison)
 ; ENABLED_MASKED_STRIDED-NEXT:    [[STRIDED_VEC:%.*]] = shufflevector <16 x i8> [[WIDE_MASKED_VEC]], <16 x i8> poison, <8 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP5:%.*]] = getelementptr inbounds i8, ptr [[Q:%.*]], i32 [[INDEX]]
-; ENABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v8i8.p0(<8 x i8> [[STRIDED_VEC]], ptr [[TMP5]], i32 1, <8 x i1> [[TMP0]])
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP4:%.*]] = getelementptr inbounds i8, ptr [[Q:%.*]], i32 [[INDEX]]
+; ENABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v8i8.p0(<8 x i8> [[STRIDED_VEC]], ptr [[TMP4]], i32 1, <8 x i1> [[TMP0]])
 ; ENABLED_MASKED_STRIDED-NEXT:    [[INDEX_NEXT]] = add i32 [[INDEX]], 8
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP7:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP7]], label [[FOR_END]], label [[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP5:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP5]], label [[FOR_END]], label [[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
 ; ENABLED_MASKED_STRIDED:       for.end:
 ; ENABLED_MASKED_STRIDED-NEXT:    ret void
 ;
@@ -1399,7 +1399,7 @@ define dso_local void @masked_strided2(ptr noalias nocapture readonly %p, ptr no
 ; DISABLED_MASKED_STRIDED-NEXT:    [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 8
 ; DISABLED_MASKED_STRIDED-NEXT:    [[VEC_IND_NEXT]] = add <8 x i32> [[VEC_IND]], <i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8>
 ; DISABLED_MASKED_STRIDED-NEXT:    [[TMP165:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP165]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP7:![0-9]+]]
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP165]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
 ; DISABLED_MASKED_STRIDED:       for.end:
 ; DISABLED_MASKED_STRIDED-NEXT:    ret void
 ;
@@ -1419,17 +1419,17 @@ define dso_local void @masked_strided2(ptr noalias nocapture readonly %p, ptr no
 ; ENABLED_MASKED_STRIDED-NEXT:    [[WIDE_MASKED_VEC:%.*]] = call <16 x i8> @llvm.masked.load.v16i8.p0(ptr [[TMP2]], i32 1, <16 x i1> [[INTERLEAVED_MASK]], <16 x i8> poison)
 ; ENABLED_MASKED_STRIDED-NEXT:    [[STRIDED_VEC:%.*]] = shufflevector <16 x i8> [[WIDE_MASKED_VEC]], <16 x i8> poison, <8 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14>
 ; ENABLED_MASKED_STRIDED-NEXT:    [[STRIDED_VEC1:%.*]] = shufflevector <16 x i8> [[WIDE_MASKED_VEC]], <16 x i8> poison, <8 x i32> <i32 1, i32 3, i32 5, i32 7, i32 9, i32 11, i32 13, i32 15>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP4:%.*]] = or i32 [[TMP1]], 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP5:%.*]] = call <8 x i8> @llvm.smax.v8i8(<8 x i8> [[STRIDED_VEC]], <8 x i8> [[STRIDED_VEC1]])
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP6:%.*]] = sub <8 x i8> zeroinitializer, [[TMP5]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP7:%.*]] = getelementptr i8, ptr [[Q:%.*]], i32 -1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP8:%.*]] = getelementptr i8, ptr [[TMP7]], i32 [[TMP4]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[INTERLEAVED_VEC:%.*]] = shufflevector <8 x i8> [[TMP5]], <8 x i8> [[TMP6]], <16 x i32> <i32 0, i32 8, i32 1, i32 9, i32 2, i32 10, i32 3, i32 11, i32 4, i32 12, i32 5, i32 13, i32 6, i32 14, i32 7, i32 15>
-; ENABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v16i8.p0(<16 x i8> [[INTERLEAVED_VEC]], ptr [[TMP8]], i32 1, <16 x i1> [[INTERLEAVED_MASK]])
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP3:%.*]] = or i32 [[TMP1]], 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP4:%.*]] = call <8 x i8> @llvm.smax.v8i8(<8 x i8> [[STRIDED_VEC]], <8 x i8> [[STRIDED_VEC1]])
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP5:%.*]] = sub <8 x i8> zeroinitializer, [[TMP4]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP6:%.*]] = getelementptr i8, ptr [[Q:%.*]], i32 -1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP7:%.*]] = getelementptr i8, ptr [[TMP6]], i32 [[TMP3]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[INTERLEAVED_VEC:%.*]] = shufflevector <8 x i8> [[TMP4]], <8 x i8> [[TMP5]], <16 x i32> <i32 0, i32 8, i32 1, i32 9, i32 2, i32 10, i32 3, i32 11, i32 4, i32 12, i32 5, i32 13, i32 6, i32 14, i32 7, i32 15>
+; ENABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v16i8.p0(<16 x i8> [[INTERLEAVED_VEC]], ptr [[TMP7]], i32 1, <16 x i1> [[INTERLEAVED_MASK]])
 ; ENABLED_MASKED_STRIDED-NEXT:    [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 8
 ; ENABLED_MASKED_STRIDED-NEXT:    [[VEC_IND_NEXT]] = add <8 x i32> [[VEC_IND]], <i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP10:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP10]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP9:![0-9]+]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP8:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP8]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP9:![0-9]+]]
 ; ENABLED_MASKED_STRIDED:       for.end:
 ; ENABLED_MASKED_STRIDED-NEXT:    ret void
 ;
@@ -1488,712 +1488,650 @@ define dso_local void @masked_strided2_reverse(ptr noalias nocapture readonly %p
 ; DISABLED_MASKED_STRIDED-LABEL: @masked_strided2_reverse(
 ; DISABLED_MASKED_STRIDED-NEXT:  entry:
 ; DISABLED_MASKED_STRIDED-NEXT:    [[CONV:%.*]] = zext i8 [[GUARD:%.*]] to i32
-; DISABLED_MASKED_STRIDED-NEXT:    [[SCEVGEP:%.*]] = getelementptr i8, ptr [[Q:%.*]], i32 2048
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP0:%.*]] = getelementptr i8, ptr [[Q]], i32 2
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP1:%.*]] = icmp ugt ptr [[TMP0]], [[SCEVGEP]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[SCEVGEP2:%.*]] = getelementptr i8, ptr [[Q]], i32 2049
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP2:%.*]] = getelementptr i8, ptr [[Q]], i32 3
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP3:%.*]] = icmp ugt ptr [[TMP2]], [[SCEVGEP2]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP4:%.*]] = or i1 [[TMP1]], [[TMP3]]
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP4]], label [[FOR_BODY:%.*]], label [[VECTOR_PH:%.*]]
-; DISABLED_MASKED_STRIDED:       vector.ph:
 ; DISABLED_MASKED_STRIDED-NEXT:    [[BROADCAST_SPLATINSERT:%.*]] = insertelement <8 x i32> poison, i32 [[CONV]], i64 0
 ; DISABLED_MASKED_STRIDED-NEXT:    [[BROADCAST_SPLAT:%.*]] = shufflevector <8 x i32> [[BROADCAST_SPLATINSERT]], <8 x i32> poison, <8 x i32> zeroinitializer
 ; DISABLED_MASKED_STRIDED-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; DISABLED_MASKED_STRIDED:       vector.body:
-; DISABLED_MASKED_STRIDED-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE65:%.*]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[VEC_IND:%.*]] = phi <8 x i32> [ <i32 1024, i32 1023, i32 1022, i32 1021, i32 1020, i32 1019, i32 1018, i32 1017>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT:%.*]], [[PRED_STORE_CONTINUE65]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP5:%.*]] = icmp ugt <8 x i32> [[VEC_IND]], [[BROADCAST_SPLAT]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP6:%.*]] = shl nuw nsw <8 x i32> [[VEC_IND]], <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP7:%.*]] = extractelement <8 x i1> [[TMP5]], i64 0
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP7]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE60:%.*]] ]
+; DISABLED_MASKED_STRIDED-NEXT:    [[VEC_IND:%.*]] = phi <8 x i32> [ <i32 1024, i32 1023, i32 1022, i32 1021, i32 1020, i32 1019, i32 1018, i32 1017>, [[ENTRY]] ], [ [[VEC_IND_NEXT:%.*]], [[PRED_STORE_CONTINUE60]] ]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP0:%.*]] = icmp ugt <8 x i32> [[VEC_IND]], [[BROADCAST_SPLAT]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP1:%.*]] = shl nuw nsw <8 x i32> [[VEC_IND]], <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP2:%.*]] = extractelement <8 x i1> [[TMP0]], i64 0
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP2]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
 ; DISABLED_MASKED_STRIDED:       pred.load.if:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP8:%.*]] = extractelement <8 x i32> [[TMP6]], i64 0
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP9:%.*]] = getelementptr inbounds i8, ptr [[P:%.*]], i32 [[TMP8]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP10:%.*]] = load i8, ptr [[TMP9]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP11:%.*]] = insertelement <8 x i8> poison, i8 [[TMP10]], i64 0
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP3:%.*]] = extractelement <8 x i32> [[TMP1]], i64 0
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP4:%.*]] = getelementptr inbounds i8, ptr [[P:%.*]], i32 [[TMP3]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP5:%.*]] = load i8, ptr [[TMP4]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP6:%.*]] = insertelement <8 x i8> poison, i8 [[TMP5]], i64 0
 ; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE]]
 ; DISABLED_MASKED_STRIDED:       pred.load.continue:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP12:%.*]] = phi <8 x i8> [ poison, [[VECTOR_BODY]] ], [ [[TMP11]], [[PRED_LOAD_IF]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP13:%.*]] = extractelement <8 x i1> [[TMP5]], i64 1
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP13]], label [[PRED_LOAD_IF6:%.*]], label [[PRED_LOAD_CONTINUE7:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.load.if6:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP14:%.*]] = extractelement <8 x i32> [[TMP6]], i64 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP15:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP14]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP16:%.*]] = load i8, ptr [[TMP15]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP17:%.*]] = insertelement <8 x i8> [[TMP12]], i8 [[TMP16]], i64 1
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE7]]
-; DISABLED_MASKED_STRIDED:       pred.load.continue7:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP18:%.*]] = phi <8 x i8> [ [[TMP12]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP17]], [[PRED_LOAD_IF6]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP19:%.*]] = extractelement <8 x i1> [[TMP5]], i64 2
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP19]], label [[PRED_LOAD_IF8:%.*]], label [[PRED_LOAD_CONTINUE9:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.load.if8:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP20:%.*]] = extractelement <8 x i32> [[TMP6]], i64 2
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP21:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP20]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP22:%.*]] = load i8, ptr [[TMP21]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP23:%.*]] = insertelement <8 x i8> [[TMP18]], i8 [[TMP22]], i64 2
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE9]]
-; DISABLED_MASKED_STRIDED:       pred.load.continue9:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP24:%.*]] = phi <8 x i8> [ [[TMP18]], [[PRED_LOAD_CONTINUE7]] ], [ [[TMP23]], [[PRED_LOAD_IF8]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP25:%.*]] = extractelement <8 x i1> [[TMP5]], i64 3
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP25]], label [[PRED_LOAD_IF10:%.*]], label [[PRED_LOAD_CONTINUE11:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.load.if10:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP26:%.*]] = extractelement <8 x i32> [[TMP6]], i64 3
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP27:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP26]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP28:%.*]] = load i8, ptr [[TMP27]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP29:%.*]] = insertelement <8 x i8> [[TMP24]], i8 [[TMP28]], i64 3
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE11]]
-; DISABLED_MASKED_STRIDED:       pred.load.continue11:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP30:%.*]] = phi <8 x i8> [ [[TMP24]], [[PRED_LOAD_CONTINUE9]] ], [ [[TMP29]], [[PRED_LOAD_IF10]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP31:%.*]] = extractelement <8 x i1> [[TMP5]], i64 4
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP31]], label [[PRED_LOAD_IF12:%.*]], label [[PRED_LOAD_CONTINUE13:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.load.if12:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP32:%.*]] = extractelement <8 x i32> [[TMP6]], i64 4
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP33:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP32]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP34:%.*]] = load i8, ptr [[TMP33]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP35:%.*]] = insertelement <8 x i8> [[TMP30]], i8 [[TMP34]], i64 4
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE13]]
-; DISABLED_MASKED_STRIDED:       pred.load.continue13:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP36:%.*]] = phi <8 x i8> [ [[TMP30]], [[PRED_LOAD_CONTINUE11]] ], [ [[TMP35]], [[PRED_LOAD_IF12]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP37:%.*]] = extractelement <8 x i1> [[TMP5]], i64 5
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP37]], label [[PRED_LOAD_IF14:%.*]], label [[PRED_LOAD_CONTINUE15:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.load.if14:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP38:%.*]] = extractelement <8 x i32> [[TMP6]], i64 5
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP39:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP38]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP40:%.*]] = load i8, ptr [[TMP39]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP41:%.*]] = insertelement <8 x i8> [[TMP36]], i8 [[TMP40]], i64 5
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE15]]
-; DISABLED_MASKED_STRIDED:       pred.load.continue15:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP42:%.*]] = phi <8 x i8> [ [[TMP36]], [[PRED_LOAD_CONTINUE13]] ], [ [[TMP41]], [[PRED_LOAD_IF14]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP43:%.*]] = extractelement <8 x i1> [[TMP5]], i64 6
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP43]], label [[PRED_LOAD_IF16:%.*]], label [[PRED_LOAD_CONTINUE17:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.load.if16:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP44:%.*]] = extractelement <8 x i32> [[TMP6]], i64 6
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP45:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP44]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP46:%.*]] = load i8, ptr [[TMP45]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP47:%.*]] = insertelement <8 x i8> [[TMP42]], i8 [[TMP46]], i64 6
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE17]]
-; DISABLED_MASKED_STRIDED:       pred.load.continue17:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP48:%.*]] = phi <8 x i8> [ [[TMP42]], [[PRED_LOAD_CONTINUE15]] ], [ [[TMP47]], [[PRED_LOAD_IF16]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP49:%.*]] = extractelement <8 x i1> [[TMP5]], i64 7
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP49]], label [[PRED_LOAD_IF18:%.*]], label [[PRED_LOAD_CONTINUE19:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.load.if18:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP50:%.*]] = extractelement <8 x i32> [[TMP6]], i64 7
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP51:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP50]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP52:%.*]] = load i8, ptr [[TMP51]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP53:%.*]] = insertelement <8 x i8> [[TMP48]], i8 [[TMP52]], i64 7
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE19]]
-; DISABLED_MASKED_STRIDED:       pred.load.continue19:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP54:%.*]] = phi <8 x i8> [ [[TMP48]], [[PRED_LOAD_CONTINUE17]] ], [ [[TMP53]], [[PRED_LOAD_IF18]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP55:%.*]] = or <8 x i32> [[TMP6]], <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP56:%.*]] = extractelement <8 x i1> [[TMP5]], i64 0
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP56]], label [[PRED_LOAD_IF20:%.*]], label [[PRED_LOAD_CONTINUE21:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.load.if20:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP57:%.*]] = extractelement <8 x i32> [[TMP55]], i64 0
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP58:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP57]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP59:%.*]] = load i8, ptr [[TMP58]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP60:%.*]] = insertelement <8 x i8> poison, i8 [[TMP59]], i64 0
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE21]]
-; DISABLED_MASKED_STRIDED:       pred.load.continue21:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP61:%.*]] = phi <8 x i8> [ poison, [[PRED_LOAD_CONTINUE19]] ], [ [[TMP60]], [[PRED_LOAD_IF20]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP62:%.*]] = extractelement <8 x i1> [[TMP5]], i64 1
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP62]], label [[PRED_LOAD_IF22:%.*]], label [[PRED_LOAD_CONTINUE23:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.load.if22:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP63:%.*]] = extractelement <8 x i32> [[TMP55]], i64 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP64:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP63]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP65:%.*]] = load i8, ptr [[TMP64]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP66:%.*]] = insertelement <8 x i8> [[TMP61]], i8 [[TMP65]], i64 1
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE23]]
-; DISABLED_MASKED_STRIDED:       pred.load.continue23:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP67:%.*]] = phi <8 x i8> [ [[TMP61]], [[PRED_LOAD_CONTINUE21]] ], [ [[TMP66]], [[PRED_LOAD_IF22]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP68:%.*]] = extractelement <8 x i1> [[TMP5]], i64 2
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP68]], label [[PRED_LOAD_IF24:%.*]], label [[PRED_LOAD_CONTINUE25:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.load.if24:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP69:%.*]] = extractelement <8 x i32> [[TMP55]], i64 2
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP70:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP69]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP71:%.*]] = load i8, ptr [[TMP70]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP72:%.*]] = insertelement <8 x i8> [[TMP67]], i8 [[TMP71]], i64 2
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE25]]
-; DISABLED_MASKED_STRIDED:       pred.load.continue25:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP73:%.*]] = phi <8 x i8> [ [[TMP67]], [[PRED_LOAD_CONTINUE23]] ], [ [[TMP72]], [[PRED_LOAD_IF24]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP74:%.*]] = extractelement <8 x i1> [[TMP5]], i64 3
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP74]], label [[PRED_LOAD_IF26:%.*]], label [[PRED_LOAD_CONTINUE27:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.load.if26:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP75:%.*]] = extractelement <8 x i32> [[TMP55]], i64 3
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP76:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP75]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP77:%.*]] = load i8, ptr [[TMP76]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP78:%.*]] = insertelement <8 x i8> [[TMP73]], i8 [[TMP77]], i64 3
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE27]]
-; DISABLED_MASKED_STRIDED:       pred.load.continue27:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP79:%.*]] = phi <8 x i8> [ [[TMP73]], [[PRED_LOAD_CONTINUE25]] ], [ [[TMP78]], [[PRED_LOAD_IF26]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP80:%.*]] = extractelement <8 x i1> [[TMP5]], i64 4
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP80]], label [[PRED_LOAD_IF28:%.*]], label [[PRED_LOAD_CONTINUE29:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.load.if28:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP81:%.*]] = extractelement <8 x i32> [[TMP55]], i64 4
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP82:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP81]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP83:%.*]] = load i8, ptr [[TMP82]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP84:%.*]] = insertelement <8 x i8> [[TMP79]], i8 [[TMP83]], i64 4
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE29]]
-; DISABLED_MASKED_STRIDED:       pred.load.continue29:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP85:%.*]] = phi <8 x i8> [ [[TMP79]], [[PRED_LOAD_CONTINUE27]] ], [ [[TMP84]], [[PRED_LOAD_IF28]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP86:%.*]] = extractelement <8 x i1> [[TMP5]], i64 5
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP86]], label [[PRED_LOAD_IF30:%.*]], label [[PRED_LOAD_CONTINUE31:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.load.if30:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP87:%.*]] = extractelement <8 x i32> [[TMP55]], i64 5
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP88:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP87]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP89:%.*]] = load i8, ptr [[TMP88]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP90:%.*]] = insertelement <8 x i8> [[TMP85]], i8 [[TMP89]], i64 5
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE31]]
-; DISABLED_MASKED_STRIDED:       pred.load.continue31:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP91:%.*]] = phi <8 x i8> [ [[TMP85]], [[PRED_LOAD_CONTINUE29]] ], [ [[TMP90]], [[PRED_LOAD_IF30]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP92:%.*]] = extractelement <8 x i1> [[TMP5]], i64 6
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP92]], label [[PRED_LOAD_IF32:%.*]], label [[PRED_LOAD_CONTINUE33:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.load.if32:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP93:%.*]] = extractelement <8 x i32> [[TMP55]], i64 6
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP94:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP93]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP95:%.*]] = load i8, ptr [[TMP94]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP96:%.*]] = insertelement <8 x i8> [[TMP91]], i8 [[TMP95]], i64 6
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE33]]
-; DISABLED_MASKED_STRIDED:       pred.load.continue33:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP97:%.*]] = phi <8 x i8> [ [[TMP91]], [[PRED_LOAD_CONTINUE31]] ], [ [[TMP96]], [[PRED_LOAD_IF32]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP98:%.*]] = extractelement <8 x i1> [[TMP5]], i64 7
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP98]], label [[PRED_LOAD_IF34:%.*]], label [[PRED_LOAD_CONTINUE35:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.load.if34:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP99:%.*]] = extractelement <8 x i32> [[TMP55]], i64 7
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP100:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP99]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP101:%.*]] = load i8, ptr [[TMP100]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP102:%.*]] = insertelement <8 x i8> [[TMP97]], i8 [[TMP101]], i64 7
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE35]]
-; DISABLED_MASKED_STRIDED:       pred.load.continue35:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP103:%.*]] = phi <8 x i8> [ [[TMP97]], [[PRED_LOAD_CONTINUE33]] ], [ [[TMP102]], [[PRED_LOAD_IF34]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP104:%.*]] = call <8 x i8> @llvm.smax.v8i8(<8 x i8> [[TMP54]], <8 x i8> [[TMP103]])
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP105:%.*]] = extractelement <8 x i1> [[TMP5]], i64 0
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP105]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP7:%.*]] = phi <8 x i8> [ poison, [[VECTOR_BODY]] ], [ [[TMP6]], [[PRED_LOAD_IF]] ]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP8:%.*]] = extractelement <8 x i1> [[TMP0]], i64 1
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP8]], label [[PRED_LOAD_IF1:%.*]], label [[PRED_LOAD_CONTINUE2:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.load.if1:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP9:%.*]] = extractelement <8 x i32> [[TMP1]], i64 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP10:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP9]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP11:%.*]] = load i8, ptr [[TMP10]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP12:%.*]] = insertelement <8 x i8> [[TMP7]], i8 [[TMP11]], i64 1
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE2]]
+; DISABLED_MASKED_STRIDED:       pred.load.continue2:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP13:%.*]] = phi <8 x i8> [ [[TMP7]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP12]], [[PRED_LOAD_IF1]] ]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP14:%.*]] = extractelement <8 x i1> [[TMP0]], i64 2
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP14]], label [[PRED_LOAD_IF3:%.*]], label [[PRED_LOAD_CONTINUE4:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.load.if3:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP15:%.*]] = extractelement <8 x i32> [[TMP1]], i64 2
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP16:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP15]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP17:%.*]] = load i8, ptr [[TMP16]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP18:%.*]] = insertelement <8 x i8> [[TMP13]], i8 [[TMP17]], i64 2
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE4]]
+; DISABLED_MASKED_STRIDED:       pred.load.continue4:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP19:%.*]] = phi <8 x i8> [ [[TMP13]], [[PRED_LOAD_CONTINUE2]] ], [ [[TMP18]], [[PRED_LOAD_IF3]] ]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP20:%.*]] = extractelement <8 x i1> [[TMP0]], i64 3
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP20]], label [[PRED_LOAD_IF5:%.*]], label [[PRED_LOAD_CONTINUE6:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.load.if5:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP21:%.*]] = extractelement <8 x i32> [[TMP1]], i64 3
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP22:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP21]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP23:%.*]] = load i8, ptr [[TMP22]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP24:%.*]] = insertelement <8 x i8> [[TMP19]], i8 [[TMP23]], i64 3
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE6]]
+; DISABLED_MASKED_STRIDED:       pred.load.continue6:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP25:%.*]] = phi <8 x i8> [ [[TMP19]], [[PRED_LOAD_CONTINUE4]] ], [ [[TMP24]], [[PRED_LOAD_IF5]] ]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP26:%.*]] = extractelement <8 x i1> [[TMP0]], i64 4
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP26]], label [[PRED_LOAD_IF7:%.*]], label [[PRED_LOAD_CONTINUE8:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.load.if7:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP27:%.*]] = extractelement <8 x i32> [[TMP1]], i64 4
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP28:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP27]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP29:%.*]] = load i8, ptr [[TMP28]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP30:%.*]] = insertelement <8 x i8> [[TMP25]], i8 [[TMP29]], i64 4
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE8]]
+; DISABLED_MASKED_STRIDED:       pred.load.continue8:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP31:%.*]] = phi <8 x i8> [ [[TMP25]], [[PRED_LOAD_CONTINUE6]] ], [ [[TMP30]], [[PRED_LOAD_IF7]] ]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP32:%.*]] = extractelement <8 x i1> [[TMP0]], i64 5
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP32]], label [[PRED_LOAD_IF9:%.*]], label [[PRED_LOAD_CONTINUE10:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.load.if9:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP33:%.*]] = extractelement <8 x i32> [[TMP1]], i64 5
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP34:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP33]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP35:%.*]] = load i8, ptr [[TMP34]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP36:%.*]] = insertelement <8 x i8> [[TMP31]], i8 [[TMP35]], i64 5
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE10]]
+; DISABLED_MASKED_STRIDED:       pred.load.continue10:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP37:%.*]] = phi <8 x i8> [ [[TMP31]], [[PRED_LOAD_CONTINUE8]] ], [ [[TMP36]], [[PRED_LOAD_IF9]] ]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP38:%.*]] = extractelement <8 x i1> [[TMP0]], i64 6
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP38]], label [[PRED_LOAD_IF11:%.*]], label [[PRED_LOAD_CONTINUE12:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.load.if11:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP39:%.*]] = extractelement <8 x i32> [[TMP1]], i64 6
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP40:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP39]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP41:%.*]] = load i8, ptr [[TMP40]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP42:%.*]] = insertelement <8 x i8> [[TMP37]], i8 [[TMP41]], i64 6
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE12]]
+; DISABLED_MASKED_STRIDED:       pred.load.continue12:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP43:%.*]] = phi <8 x i8> [ [[TMP37]], [[PRED_LOAD_CONTINUE10]] ], [ [[TMP42]], [[PRED_LOAD_IF11]] ]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP44:%.*]] = extractelement <8 x i1> [[TMP0]], i64 7
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP44]], label [[PRED_LOAD_IF13:%.*]], label [[PRED_LOAD_CONTINUE14:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.load.if13:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP45:%.*]] = extractelement <8 x i32> [[TMP1]], i64 7
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP46:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP45]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP47:%.*]] = load i8, ptr [[TMP46]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP48:%.*]] = insertelement <8 x i8> [[TMP43]], i8 [[TMP47]], i64 7
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE14]]
+; DISABLED_MASKED_STRIDED:       pred.load.continue14:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP49:%.*]] = phi <8 x i8> [ [[TMP43]], [[PRED_LOAD_CONTINUE12]] ], [ [[TMP48]], [[PRED_LOAD_IF13]] ]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP50:%.*]] = or <8 x i32> [[TMP1]], <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP51:%.*]] = extractelement <8 x i1> [[TMP0]], i64 0
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP51]], label [[PRED_LOAD_IF15:%.*]], label [[PRED_LOAD_CONTINUE16:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.load.if15:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP52:%.*]] = extractelement <8 x i32> [[TMP50]], i64 0
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP53:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP52]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP54:%.*]] = load i8, ptr [[TMP53]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP55:%.*]] = insertelement <8 x i8> poison, i8 [[TMP54]], i64 0
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE16]]
+; DISABLED_MASKED_STRIDED:       pred.load.continue16:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP56:%.*]] = phi <8 x i8> [ poison, [[PRED_LOAD_CONTINUE14]] ], [ [[TMP55]], [[PRED_LOAD_IF15]] ]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP57:%.*]] = extractelement <8 x i1> [[TMP0]], i64 1
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP57]], label [[PRED_LOAD_IF17:%.*]], label [[PRED_LOAD_CONTINUE18:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.load.if17:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP58:%.*]] = extractelement <8 x i32> [[TMP50]], i64 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP59:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP58]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP60:%.*]] = load i8, ptr [[TMP59]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP61:%.*]] = insertelement <8 x i8> [[TMP56]], i8 [[TMP60]], i64 1
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE18]]
+; DISABLED_MASKED_STRIDED:       pred.load.continue18:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP62:%.*]] = phi <8 x i8> [ [[TMP56]], [[PRED_LOAD_CONTINUE16]] ], [ [[TMP61]], [[PRED_LOAD_IF17]] ]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP63:%.*]] = extractelement <8 x i1> [[TMP0]], i64 2
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP63]], label [[PRED_LOAD_IF19:%.*]], label [[PRED_LOAD_CONTINUE20:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.load.if19:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP64:%.*]] = extractelement <8 x i32> [[TMP50]], i64 2
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP65:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP64]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP66:%.*]] = load i8, ptr [[TMP65]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP67:%.*]] = insertelement <8 x i8> [[TMP62]], i8 [[TMP66]], i64 2
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE20]]
+; DISABLED_MASKED_STRIDED:       pred.load.continue20:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP68:%.*]] = phi <8 x i8> [ [[TMP62]], [[PRED_LOAD_CONTINUE18]] ], [ [[TMP67]], [[PRED_LOAD_IF19]] ]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP69:%.*]] = extractelement <8 x i1> [[TMP0]], i64 3
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP69]], label [[PRED_LOAD_IF21:%.*]], label [[PRED_LOAD_CONTINUE22:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.load.if21:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP70:%.*]] = extractelement <8 x i32> [[TMP50]], i64 3
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP71:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP70]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP72:%.*]] = load i8, ptr [[TMP71]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP73:%.*]] = insertelement <8 x i8> [[TMP68]], i8 [[TMP72]], i64 3
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE22]]
+; DISABLED_MASKED_STRIDED:       pred.load.continue22:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP74:%.*]] = phi <8 x i8> [ [[TMP68]], [[PRED_LOAD_CONTINUE20]] ], [ [[TMP73]], [[PRED_LOAD_IF21]] ]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP75:%.*]] = extractelement <8 x i1> [[TMP0]], i64 4
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP75]], label [[PRED_LOAD_IF23:%.*]], label [[PRED_LOAD_CONTINUE24:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.load.if23:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP76:%.*]] = extractelement <8 x i32> [[TMP50]], i64 4
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP77:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP76]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP78:%.*]] = load i8, ptr [[TMP77]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP79:%.*]] = insertelement <8 x i8> [[TMP74]], i8 [[TMP78]], i64 4
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE24]]
+; DISABLED_MASKED_STRIDED:       pred.load.continue24:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP80:%.*]] = phi <8 x i8> [ [[TMP74]], [[PRED_LOAD_CONTINUE22]] ], [ [[TMP79]], [[PRED_LOAD_IF23]] ]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP81:%.*]] = extractelement <8 x i1> [[TMP0]], i64 5
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP81]], label [[PRED_LOAD_IF25:%.*]], label [[PRED_LOAD_CONTINUE26:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.load.if25:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP82:%.*]] = extractelement <8 x i32> [[TMP50]], i64 5
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP83:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP82]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP84:%.*]] = load i8, ptr [[TMP83]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP85:%.*]] = insertelement <8 x i8> [[TMP80]], i8 [[TMP84]], i64 5
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE26]]
+; DISABLED_MASKED_STRIDED:       pred.load.continue26:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP86:%.*]] = phi <8 x i8> [ [[TMP80]], [[PRED_LOAD_CONTINUE24]] ], [ [[TMP85]], [[PRED_LOAD_IF25]] ]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP87:%.*]] = extractelement <8 x i1> [[TMP0]], i64 6
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP87]], label [[PRED_LOAD_IF27:%.*]], label [[PRED_LOAD_CONTINUE28:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.load.if27:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP88:%.*]] = extractelement <8 x i32> [[TMP50]], i64 6
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP89:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP88]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP90:%.*]] = load i8, ptr [[TMP89]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP91:%.*]] = insertelement <8 x i8> [[TMP86]], i8 [[TMP90]], i64 6
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE28]]
+; DISABLED_MASKED_STRIDED:       pred.load.continue28:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP92:%.*]] = phi <8 x i8> [ [[TMP86]], [[PRED_LOAD_CONTINUE26]] ], [ [[TMP91]], [[PRED_LOAD_IF27]] ]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP93:%.*]] = extractelement <8 x i1> [[TMP0]], i64 7
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP93]], label [[PRED_LOAD_IF29:%.*]], label [[PRED_LOAD_CONTINUE30:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.load.if29:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP94:%.*]] = extractelement <8 x i32> [[TMP50]], i64 7
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP95:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP94]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP96:%.*]] = load i8, ptr [[TMP95]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP97:%.*]] = insertelement <8 x i8> [[TMP92]], i8 [[TMP96]], i64 7
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE30]]
+; DISABLED_MASKED_STRIDED:       pred.load.continue30:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP98:%.*]] = phi <8 x i8> [ [[TMP92]], [[PRED_LOAD_CONTINUE28]] ], [ [[TMP97]], [[PRED_LOAD_IF29]] ]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP99:%.*]] = call <8 x i8> @llvm.smax.v8i8(<8 x i8> [[TMP49]], <8 x i8> [[TMP98]])
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP100:%.*]] = extractelement <8 x i1> [[TMP0]], i64 0
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP100]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
 ; DISABLED_MASKED_STRIDED:       pred.store.if:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP106:%.*]] = extractelement <8 x i32> [[TMP6]], i64 0
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP107:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP106]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP108:%.*]] = extractelement <8 x i8> [[TMP104]], i64 0
-; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP108]], ptr [[TMP107]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP101:%.*]] = extractelement <8 x i32> [[TMP1]], i64 0
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP102:%.*]] = getelementptr inbounds i8, ptr [[Q:%.*]], i32 [[TMP101]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP103:%.*]] = extractelement <8 x i8> [[TMP99]], i64 0
+; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP103]], ptr [[TMP102]], align 1
 ; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE]]
 ; DISABLED_MASKED_STRIDED:       pred.store.continue:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP109:%.*]] = extractelement <8 x i1> [[TMP5]], i64 1
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP109]], label [[PRED_STORE_IF36:%.*]], label [[PRED_STORE_CONTINUE37:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.store.if36:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP110:%.*]] = extractelement <8 x i32> [[TMP6]], i64 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP111:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP110]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP112:%.*]] = extractelement <8 x i8> [[TMP104]], i64 1
-; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP112]], ptr [[TMP111]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE37]]
-; DISABLED_MASKED_STRIDED:       pred.store.continue37:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP113:%.*]] = extractelement <8 x i1> [[TMP5]], i64 2
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP113]], label [[PRED_STORE_IF38:%.*]], label [[PRED_STORE_CONTINUE39:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.store.if38:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP114:%.*]] = extractelement <8 x i32> [[TMP6]], i64 2
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP115:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP114]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP116:%.*]] = extractelement <8 x i8> [[TMP104]], i64 2
-; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP116]], ptr [[TMP115]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE39]]
-; DISABLED_MASKED_STRIDED:       pred.store.continue39:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP117:%.*]] = extractelement <8 x i1> [[TMP5]], i64 3
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP117]], label [[PRED_STORE_IF40:%.*]], label [[PRED_STORE_CONTINUE41:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.store.if40:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP118:%.*]] = extractelement <8 x i32> [[TMP6]], i64 3
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP119:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP118]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP120:%.*]] = extractelement <8 x i8> [[TMP104]], i64 3
-; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP120]], ptr [[TMP119]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE41]]
-; DISABLED_MASKED_STRIDED:       pred.store.continue41:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP121:%.*]] = extractelement <8 x i1> [[TMP5]], i64 4
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP121]], label [[PRED_STORE_IF42:%.*]], label [[PRED_STORE_CONTINUE43:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.store.if42:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP122:%.*]] = extractelement <8 x i32> [[TMP6]], i64 4
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP123:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP122]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP124:%.*]] = extractelement <8 x i8> [[TMP104]], i64 4
-; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP124]], ptr [[TMP123]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE43]]
-; DISABLED_MASKED_STRIDED:       pred.store.continue43:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP125:%.*]] = extractelement <8 x i1> [[TMP5]], i64 5
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP125]], label [[PRED_STORE_IF44:%.*]], label [[PRED_STORE_CONTINUE45:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.store.if44:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP126:%.*]] = extractelement <8 x i32> [[TMP6]], i64 5
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP127:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP126]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP128:%.*]] = extractelement <8 x i8> [[TMP104]], i64 5
-; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP128]], ptr [[TMP127]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE45]]
-; DISABLED_MASKED_STRIDED:       pred.store.continue45:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP129:%.*]] = extractelement <8 x i1> [[TMP5]], i64 6
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP129]], label [[PRED_STORE_IF46:%.*]], label [[PRED_STORE_CONTINUE47:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.store.if46:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP130:%.*]] = extractelement <8 x i32> [[TMP6]], i64 6
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP131:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP130]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP132:%.*]] = extractelement <8 x i8> [[TMP104]], i64 6
-; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP132]], ptr [[TMP131]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE47]]
-; DISABLED_MASKED_STRIDED:       pred.store.continue47:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP133:%.*]] = extractelement <8 x i1> [[TMP5]], i64 7
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP133]], label [[PRED_STORE_IF48:%.*]], label [[PRED_STORE_CONTINUE49:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.store.if48:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP134:%.*]] = extractelement <8 x i32> [[TMP6]], i64 7
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP104:%.*]] = extractelement <8 x i1> [[TMP0]], i64 1
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP104]], label [[PRED_STORE_IF31:%.*]], label [[PRED_STORE_CONTINUE32:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.store.if31:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP105:%.*]] = extractelement <8 x i32> [[TMP1]], i64 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP106:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP105]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP107:%.*]] = extractelement <8 x i8> [[TMP99]], i64 1
+; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP107]], ptr [[TMP106]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE32]]
+; DISABLED_MASKED_STRIDED:       pred.store.continue32:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP108:%.*]] = extractelement <8 x i1> [[TMP0]], i64 2
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP108]], label [[PRED_STORE_IF33:%.*]], label [[PRED_STORE_CONTINUE34:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.store.if33:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP109:%.*]] = extractelement <8 x i32> [[TMP1]], i64 2
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP110:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP109]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP111:%.*]] = extractelement <8 x i8> [[TMP99]], i64 2
+; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP111]], ptr [[TMP110]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE34]]
+; DISABLED_MASKED_STRIDED:       pred.store.continue34:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP112:%.*]] = extractelement <8 x i1> [[TMP0]], i64 3
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP112]], label [[PRED_STORE_IF35:%.*]], label [[PRED_STORE_CONTINUE36:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.store.if35:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP113:%.*]] = extractelement <8 x i32> [[TMP1]], i64 3
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP114:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP113]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP115:%.*]] = extractelement <8 x i8> [[TMP99]], i64 3
+; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP115]], ptr [[TMP114]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE36]]
+; DISABLED_MASKED_STRIDED:       pred.store.continue36:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP116:%.*]] = extractelement <8 x i1> [[TMP0]], i64 4
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP116]], label [[PRED_STORE_IF37:%.*]], label [[PRED_STORE_CONTINUE38:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.store.if37:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP117:%.*]] = extractelement <8 x i32> [[TMP1]], i64 4
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP118:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP117]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP119:%.*]] = extractelement <8 x i8> [[TMP99]], i64 4
+; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP119]], ptr [[TMP118]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE38]]
+; DISABLED_MASKED_STRIDED:       pred.store.continue38:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP120:%.*]] = extractelement <8 x i1> [[TMP0]], i64 5
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP120]], label [[PRED_STORE_IF39:%.*]], label [[PRED_STORE_CONTINUE40:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.store.if39:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP121:%.*]] = extractelement <8 x i32> [[TMP1]], i64 5
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP122:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP121]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP123:%.*]] = extractelement <8 x i8> [[TMP99]], i64 5
+; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP123]], ptr [[TMP122]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE40]]
+; DISABLED_MASKED_STRIDED:       pred.store.continue40:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP124:%.*]] = extractelement <8 x i1> [[TMP0]], i64 6
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP124]], label [[PRED_STORE_IF41:%.*]], label [[PRED_STORE_CONTINUE42:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.store.if41:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP125:%.*]] = extractelement <8 x i32> [[TMP1]], i64 6
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP126:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP125]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP127:%.*]] = extractelement <8 x i8> [[TMP99]], i64 6
+; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP127]], ptr [[TMP126]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE42]]
+; DISABLED_MASKED_STRIDED:       pred.store.continue42:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP128:%.*]] = extractelement <8 x i1> [[TMP0]], i64 7
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP128]], label [[PRED_STORE_IF43:%.*]], label [[PRED_STORE_CONTINUE44:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.store.if43:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP129:%.*]] = extractelement <8 x i32> [[TMP1]], i64 7
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP130:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP129]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP131:%.*]] = extractelement <8 x i8> [[TMP99]], i64 7
+; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP131]], ptr [[TMP130]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE44]]
+; DISABLED_MASKED_STRIDED:       pred.store.continue44:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP132:%.*]] = sub <8 x i8> zeroinitializer, [[TMP99]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP133:%.*]] = extractelement <8 x i1> [[TMP0]], i64 0
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP133]], label [[PRED_STORE_IF45:%.*]], label [[PRED_STORE_CONTINUE46:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.store.if45:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP134:%.*]] = extractelement <8 x i32> [[TMP50]], i64 0
 ; DISABLED_MASKED_STRIDED-NEXT:    [[TMP135:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP134]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP136:%.*]] = extractelement <8 x i8> [[TMP104]], i64 7
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP136:%.*]] = extractelement <8 x i8> [[TMP132]], i64 0
 ; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP136]], ptr [[TMP135]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE49]]
-; DISABLED_MASKED_STRIDED:       pred.store.continue49:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP137:%.*]] = sub <8 x i8> zeroinitializer, [[TMP104]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP138:%.*]] = extractelement <8 x i1> [[TMP5]], i64 0
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP138]], label [[PRED_STORE_IF50:%.*]], label [[PRED_STORE_CONTINUE51:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.store.if50:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP139:%.*]] = extractelement <8 x i32> [[TMP55]], i64 0
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP140:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP139]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP141:%.*]] = extractelement <8 x i8> [[TMP137]], i64 0
-; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP141]], ptr [[TMP140]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE51]]
-; DISABLED_MASKED_STRIDED:       pred.store.continue51:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP142:%.*]] = extractelement <8 x i1> [[TMP5]], i64 1
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP142]], label [[PRED_STORE_IF52:%.*]], label [[PRED_STORE_CONTINUE53:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.store.if52:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP143:%.*]] = extractelement <8 x i32> [[TMP55]], i64 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP144:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP143]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP145:%.*]] = extractelement <8 x i8> [[TMP137]], i64 1
-; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP145]], ptr [[TMP144]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE53]]
-; DISABLED_MASKED_STRIDED:       pred.store.continue53:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP146:%.*]] = extractelement <8 x i1> [[TMP5]], i64 2
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP146]], label [[PRED_STORE_IF54:%.*]], label [[PRED_STORE_CONTINUE55:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.store.if54:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP147:%.*]] = extractelement <8 x i32> [[TMP55]], i64 2
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP148:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP147]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP149:%.*]] = extractelement <8 x i8> [[TMP137]], i64 2
-; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP149]], ptr [[TMP148]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE55]]
-; DISABLED_MASKED_STRIDED:       pred.store.continue55:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP150:%.*]] = extractelement <8 x i1> [[TMP5]], i64 3
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP150]], label [[PRED_STORE_IF56:%.*]], label [[PRED_STORE_CONTINUE57:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.store.if56:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP151:%.*]] = extractelement <8 x i32> [[TMP55]], i64 3
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP152:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP151]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP153:%.*]] = extractelement <8 x i8> [[TMP137]], i64 3
-; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP153]], ptr [[TMP152]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE57]]
-; DISABLED_MASKED_STRIDED:       pred.store.continue57:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP154:%.*]] = extractelement <8 x i1> [[TMP5]], i64 4
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP154]], label [[PRED_STORE_IF58:%.*]], label [[PRED_STORE_CONTINUE59:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.store.if58:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP155:%.*]] = extractelement <8 x i32> [[TMP55]], i64 4
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP156:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP155]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP157:%.*]] = extractelement <8 x i8> [[TMP137]], i64 4
-; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP157]], ptr [[TMP156]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE59]]
-; DISABLED_MASKED_STRIDED:       pred.store.continue59:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP158:%.*]] = extractelement <8 x i1> [[TMP5]], i64 5
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP158]], label [[PRED_STORE_IF60:%.*]], label [[PRED_STORE_CONTINUE61:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.store.if60:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP159:%.*]] = extractelement <8 x i32> [[TMP55]], i64 5
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP160:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP159]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP161:%.*]] = extractelement <8 x i8> [[TMP137]], i64 5
-; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP161]], ptr [[TMP160]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE61]]
-; DISABLED_MASKED_STRIDED:       pred.store.continue61:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP162:%.*]] = extractelement <8 x i1> [[TMP5]], i64 6
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP162]], label [[PRED_STORE_IF62:%.*]], label [[PRED_STORE_CONTINUE63:%.*]]
-; DISABLED_MASKED_STRIDED:       pred.store.if62:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP163:%.*]] = extractelement <8 x i32> [[TMP55]], i64 6
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP164:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP163]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP165:%.*]] = extractelement <8 x i8> [[TMP137]], i64 6
-; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP165]], ptr [[TMP164]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE63]]
-; DISABLED_MASKED_STRIDED:       pred.store.continue63:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP166:%.*]] = extractelement <8 x i1> [[TMP5]], i64 7
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP166]], label [[PRED_STORE_IF64:%.*]], label [[PRED_STORE_CONTINUE65]]
-; DISABLED_MASKED_STRIDED:       pred.store.if64:
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP167:%.*]] = extractelement <8 x i32> [[TMP55]], i64 7
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP168:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP167]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP169:%.*]] = extractelement <8 x i8> [[TMP137]], i64 7
-; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP169]], ptr [[TMP168]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE65]]
-; DISABLED_MASKED_STRIDED:       pred.store.continue65:
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE46]]
+; DISABLED_MASKED_STRIDED:       pred.store.continue46:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP137:%.*]] = extractelement <8 x i1> [[TMP0]], i64 1
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP137]], label [[PRED_STORE_IF47:%.*]], label [[PRED_STORE_CONTINUE48:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.store.if47:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP138:%.*]] = extractelement <8 x i32> [[TMP50]], i64 1
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP139:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP138]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP140:%.*]] = extractelement <8 x i8> [[TMP132]], i64 1
+; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP140]], ptr [[TMP139]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE48]]
+; DISABLED_MASKED_STRIDED:       pred.store.continue48:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP141:%.*]] = extractelement <8 x i1> [[TMP0]], i64 2
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP141]], label [[PRED_STORE_IF49:%.*]], label [[PRED_STORE_CONTINUE50:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.store.if49:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP142:%.*]] = extractelement <8 x i32> [[TMP50]], i64 2
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP143:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP142]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP144:%.*]] = extractelement <8 x i8> [[TMP132]], i64 2
+; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP144]], ptr [[TMP143]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE50]]
+; DISABLED_MASKED_STRIDED:       pred.store.continue50:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP145:%.*]] = extractelement <8 x i1> [[TMP0]], i64 3
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP145]], label [[PRED_STORE_IF51:%.*]], label [[PRED_STORE_CONTINUE52:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.store.if51:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP146:%.*]] = extractelement <8 x i32> [[TMP50]], i64 3
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP147:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP146]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP148:%.*]] = extractelement <8 x i8> [[TMP132]], i64 3
+; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP148]], ptr [[TMP147]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE52]]
+; DISABLED_MASKED_STRIDED:       pred.store.continue52:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP149:%.*]] = extractelement <8 x i1> [[TMP0]], i64 4
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP149]], label [[PRED_STORE_IF53:%.*]], label [[PRED_STORE_CONTINUE54:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.store.if53:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP150:%.*]] = extractelement <8 x i32> [[TMP50]], i64 4
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP151:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP150]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP152:%.*]] = extractelement <8 x i8> [[TMP132]], i64 4
+; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP152]], ptr [[TMP151]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE54]]
+; DISABLED_MASKED_STRIDED:       pred.store.continue54:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP153:%.*]] = extractelement <8 x i1> [[TMP0]], i64 5
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP153]], label [[PRED_STORE_IF55:%.*]], label [[PRED_STORE_CONTINUE56:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.store.if55:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP154:%.*]] = extractelement <8 x i32> [[TMP50]], i64 5
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP155:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP154]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP156:%.*]] = extractelement <8 x i8> [[TMP132]], i64 5
+; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP156]], ptr [[TMP155]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE56]]
+; DISABLED_MASKED_STRIDED:       pred.store.continue56:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP157:%.*]] = extractelement <8 x i1> [[TMP0]], i64 6
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP157]], label [[PRED_STORE_IF57:%.*]], label [[PRED_STORE_CONTINUE58:%.*]]
+; DISABLED_MASKED_STRIDED:       pred.store.if57:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP158:%.*]] = extractelement <8 x i32> [[TMP50]], i64 6
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP159:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP158]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP160:%.*]] = extractelement <8 x i8> [[TMP132]], i64 6
+; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP160]], ptr [[TMP159]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE58]]
+; DISABLED_MASKED_STRIDED:       pred.store.continue58:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP161:%.*]] = extractelement <8 x i1> [[TMP0]], i64 7
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP161]], label [[PRED_STORE_IF59:%.*]], label [[PRED_STORE_CONTINUE60]]
+; DISABLED_MASKED_STRIDED:       pred.store.if59:
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP162:%.*]] = extractelement <8 x i32> [[TMP50]], i64 7
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP163:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP162]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP164:%.*]] = extractelement <8 x i8> [[TMP132]], i64 7
+; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP164]], ptr [[TMP163]], align 1
+; DISABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE60]]
+; DISABLED_MASKED_STRIDED:       pred.store.continue60:
 ; DISABLED_MASKED_STRIDED-NEXT:    [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 8
 ; DISABLED_MASKED_STRIDED-NEXT:    [[VEC_IND_NEXT]] = add <8 x i32> [[VEC_IND]], <i32 -8, i32 -8, i32 -8, i32 -8, i32 -8, i32 -8, i32 -8, i32 -8>
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP170:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP170]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
-; DISABLED_MASKED_STRIDED:       for.body:
-; DISABLED_MASKED_STRIDED-NEXT:    [[IX_024:%.*]] = phi i32 [ [[INC:%.*]], [[FOR_INC:%.*]] ], [ 1024, [[ENTRY:%.*]] ]
-; DISABLED_MASKED_STRIDED-NEXT:    [[CMP1:%.*]] = icmp ugt i32 [[IX_024]], [[CONV]]
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[CMP1]], label [[IF_THEN:%.*]], label [[FOR_INC]]
-; DISABLED_MASKED_STRIDED:       if.then:
-; DISABLED_MASKED_STRIDED-NEXT:    [[MUL:%.*]] = shl nuw nsw i32 [[IX_024]], 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[MUL]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP171:%.*]] = load i8, ptr [[ARRAYIDX]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[ADD:%.*]] = or i32 [[MUL]], 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[ARRAYIDX4:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[ADD]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP172:%.*]] = load i8, ptr [[ARRAYIDX4]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[TMP173:%.*]] = call i8 @llvm.smax.i8(i8 [[TMP171]], i8 [[TMP172]])
-; DISABLED_MASKED_STRIDED-NEXT:    [[ARRAYIDX6:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[MUL]]
-; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP173]], ptr [[ARRAYIDX6]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    [[SUB:%.*]] = sub i8 0, [[TMP173]]
-; DISABLED_MASKED_STRIDED-NEXT:    [[ARRAYIDX11:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[ADD]]
-; DISABLED_MASKED_STRIDED-NEXT:    store i8 [[SUB]], ptr [[ARRAYIDX11]], align 1
-; DISABLED_MASKED_STRIDED-NEXT:    br label [[FOR_INC]]
-; DISABLED_MASKED_STRIDED:       for.inc:
-; DISABLED_MASKED_STRIDED-NEXT:    [[INC]] = add nsw i32 [[IX_024]], -1
-; DISABLED_MASKED_STRIDED-NEXT:    [[EXITCOND:%.*]] = icmp eq i32 [[INC]], 0
-; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[EXITCOND]], label [[FOR_END]], label [[FOR_BODY]], !llvm.loop [[LOOP9:![0-9]+]]
+; DISABLED_MASKED_STRIDED-NEXT:    [[TMP165:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024
+; DISABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP165]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP9:![0-9]+]]
 ; DISABLED_MASKED_STRIDED:       for.end:
 ; DISABLED_MASKED_STRIDED-NEXT:    ret void
 ;
 ; ENABLED_MASKED_STRIDED-LABEL: @masked_strided2_reverse(
 ; ENABLED_MASKED_STRIDED-NEXT:  entry:
 ; ENABLED_MASKED_STRIDED-NEXT:    [[CONV:%.*]] = zext i8 [[GUARD:%.*]] to i32
-; ENABLED_MASKED_STRIDED-NEXT:    [[SCEVGEP:%.*]] = getelementptr i8, ptr [[Q:%.*]], i32 2048
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP0:%.*]] = getelementptr i8, ptr [[Q]], i32 2
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP1:%.*]] = icmp ugt ptr [[TMP0]], [[SCEVGEP]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[SCEVGEP2:%.*]] = getelementptr i8, ptr [[Q]], i32 2049
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP2:%.*]] = getelementptr i8, ptr [[Q]], i32 3
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP3:%.*]] = icmp ugt ptr [[TMP2]], [[SCEVGEP2]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP4:%.*]] = or i1 [[TMP1]], [[TMP3]]
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP4]], label [[FOR_BODY:%.*]], label [[VECTOR_PH:%.*]]
-; ENABLED_MASKED_STRIDED:       vector.ph:
 ; ENABLED_MASKED_STRIDED-NEXT:    [[BROADCAST_SPLATINSERT:%.*]] = insertelement <8 x i32> poison, i32 [[CONV]], i64 0
 ; ENABLED_MASKED_STRIDED-NEXT:    [[BROADCAST_SPLAT:%.*]] = shufflevector <8 x i32> [[BROADCAST_SPLATINSERT]], <8 x i32> poison, <8 x i32> zeroinitializer
 ; ENABLED_MASKED_STRIDED-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; ENABLED_MASKED_STRIDED:       vector.body:
-; ENABLED_MASKED_STRIDED-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE65:%.*]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[VEC_IND:%.*]] = phi <8 x i32> [ <i32 1024, i32 1023, i32 1022, i32 1021, i32 1020, i32 1019, i32 1018, i32 1017>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT:%.*]], [[PRED_STORE_CONTINUE65]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP5:%.*]] = icmp ugt <8 x i32> [[VEC_IND]], [[BROADCAST_SPLAT]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP6:%.*]] = shl nuw nsw <8 x i32> [[VEC_IND]], <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP7:%.*]] = extractelement <8 x i1> [[TMP5]], i64 0
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP7]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE60:%.*]] ]
+; ENABLED_MASKED_STRIDED-NEXT:    [[VEC_IND:%.*]] = phi <8 x i32> [ <i32 1024, i32 1023, i32 1022, i32 1021, i32 1020, i32 1019, i32 1018, i32 1017>, [[ENTRY]] ], [ [[VEC_IND_NEXT:%.*]], [[PRED_STORE_CONTINUE60]] ]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP0:%.*]] = icmp ugt <8 x i32> [[VEC_IND]], [[BROADCAST_SPLAT]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP1:%.*]] = shl nuw nsw <8 x i32> [[VEC_IND]], <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP2:%.*]] = extractelement <8 x i1> [[TMP0]], i64 0
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP2]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
 ; ENABLED_MASKED_STRIDED:       pred.load.if:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP8:%.*]] = extractelement <8 x i32> [[TMP6]], i64 0
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP9:%.*]] = getelementptr inbounds i8, ptr [[P:%.*]], i32 [[TMP8]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP10:%.*]] = load i8, ptr [[TMP9]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP11:%.*]] = insertelement <8 x i8> poison, i8 [[TMP10]], i64 0
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP3:%.*]] = extractelement <8 x i32> [[TMP1]], i64 0
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP4:%.*]] = getelementptr inbounds i8, ptr [[P:%.*]], i32 [[TMP3]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP5:%.*]] = load i8, ptr [[TMP4]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP6:%.*]] = insertelement <8 x i8> poison, i8 [[TMP5]], i64 0
 ; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE]]
 ; ENABLED_MASKED_STRIDED:       pred.load.continue:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP12:%.*]] = phi <8 x i8> [ poison, [[VECTOR_BODY]] ], [ [[TMP11]], [[PRED_LOAD_IF]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP13:%.*]] = extractelement <8 x i1> [[TMP5]], i64 1
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP13]], label [[PRED_LOAD_IF6:%.*]], label [[PRED_LOAD_CONTINUE7:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.load.if6:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP14:%.*]] = extractelement <8 x i32> [[TMP6]], i64 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP15:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP14]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP16:%.*]] = load i8, ptr [[TMP15]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP17:%.*]] = insertelement <8 x i8> [[TMP12]], i8 [[TMP16]], i64 1
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE7]]
-; ENABLED_MASKED_STRIDED:       pred.load.continue7:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP18:%.*]] = phi <8 x i8> [ [[TMP12]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP17]], [[PRED_LOAD_IF6]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP19:%.*]] = extractelement <8 x i1> [[TMP5]], i64 2
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP19]], label [[PRED_LOAD_IF8:%.*]], label [[PRED_LOAD_CONTINUE9:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.load.if8:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP20:%.*]] = extractelement <8 x i32> [[TMP6]], i64 2
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP21:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP20]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP22:%.*]] = load i8, ptr [[TMP21]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP23:%.*]] = insertelement <8 x i8> [[TMP18]], i8 [[TMP22]], i64 2
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE9]]
-; ENABLED_MASKED_STRIDED:       pred.load.continue9:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP24:%.*]] = phi <8 x i8> [ [[TMP18]], [[PRED_LOAD_CONTINUE7]] ], [ [[TMP23]], [[PRED_LOAD_IF8]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP25:%.*]] = extractelement <8 x i1> [[TMP5]], i64 3
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP25]], label [[PRED_LOAD_IF10:%.*]], label [[PRED_LOAD_CONTINUE11:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.load.if10:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP26:%.*]] = extractelement <8 x i32> [[TMP6]], i64 3
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP27:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP26]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP28:%.*]] = load i8, ptr [[TMP27]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP29:%.*]] = insertelement <8 x i8> [[TMP24]], i8 [[TMP28]], i64 3
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE11]]
-; ENABLED_MASKED_STRIDED:       pred.load.continue11:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP30:%.*]] = phi <8 x i8> [ [[TMP24]], [[PRED_LOAD_CONTINUE9]] ], [ [[TMP29]], [[PRED_LOAD_IF10]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP31:%.*]] = extractelement <8 x i1> [[TMP5]], i64 4
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP31]], label [[PRED_LOAD_IF12:%.*]], label [[PRED_LOAD_CONTINUE13:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.load.if12:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP32:%.*]] = extractelement <8 x i32> [[TMP6]], i64 4
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP33:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP32]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP34:%.*]] = load i8, ptr [[TMP33]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP35:%.*]] = insertelement <8 x i8> [[TMP30]], i8 [[TMP34]], i64 4
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE13]]
-; ENABLED_MASKED_STRIDED:       pred.load.continue13:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP36:%.*]] = phi <8 x i8> [ [[TMP30]], [[PRED_LOAD_CONTINUE11]] ], [ [[TMP35]], [[PRED_LOAD_IF12]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP37:%.*]] = extractelement <8 x i1> [[TMP5]], i64 5
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP37]], label [[PRED_LOAD_IF14:%.*]], label [[PRED_LOAD_CONTINUE15:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.load.if14:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP38:%.*]] = extractelement <8 x i32> [[TMP6]], i64 5
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP39:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP38]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP40:%.*]] = load i8, ptr [[TMP39]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP41:%.*]] = insertelement <8 x i8> [[TMP36]], i8 [[TMP40]], i64 5
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE15]]
-; ENABLED_MASKED_STRIDED:       pred.load.continue15:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP42:%.*]] = phi <8 x i8> [ [[TMP36]], [[PRED_LOAD_CONTINUE13]] ], [ [[TMP41]], [[PRED_LOAD_IF14]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP43:%.*]] = extractelement <8 x i1> [[TMP5]], i64 6
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP43]], label [[PRED_LOAD_IF16:%.*]], label [[PRED_LOAD_CONTINUE17:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.load.if16:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP44:%.*]] = extractelement <8 x i32> [[TMP6]], i64 6
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP45:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP44]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP46:%.*]] = load i8, ptr [[TMP45]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP47:%.*]] = insertelement <8 x i8> [[TMP42]], i8 [[TMP46]], i64 6
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE17]]
-; ENABLED_MASKED_STRIDED:       pred.load.continue17:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP48:%.*]] = phi <8 x i8> [ [[TMP42]], [[PRED_LOAD_CONTINUE15]] ], [ [[TMP47]], [[PRED_LOAD_IF16]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP49:%.*]] = extractelement <8 x i1> [[TMP5]], i64 7
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP49]], label [[PRED_LOAD_IF18:%.*]], label [[PRED_LOAD_CONTINUE19:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.load.if18:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP50:%.*]] = extractelement <8 x i32> [[TMP6]], i64 7
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP51:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP50]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP52:%.*]] = load i8, ptr [[TMP51]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP53:%.*]] = insertelement <8 x i8> [[TMP48]], i8 [[TMP52]], i64 7
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE19]]
-; ENABLED_MASKED_STRIDED:       pred.load.continue19:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP54:%.*]] = phi <8 x i8> [ [[TMP48]], [[PRED_LOAD_CONTINUE17]] ], [ [[TMP53]], [[PRED_LOAD_IF18]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP55:%.*]] = or <8 x i32> [[TMP6]], <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP56:%.*]] = extractelement <8 x i1> [[TMP5]], i64 0
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP56]], label [[PRED_LOAD_IF20:%.*]], label [[PRED_LOAD_CONTINUE21:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.load.if20:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP57:%.*]] = extractelement <8 x i32> [[TMP55]], i64 0
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP58:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP57]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP59:%.*]] = load i8, ptr [[TMP58]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP60:%.*]] = insertelement <8 x i8> poison, i8 [[TMP59]], i64 0
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE21]]
-; ENABLED_MASKED_STRIDED:       pred.load.continue21:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP61:%.*]] = phi <8 x i8> [ poison, [[PRED_LOAD_CONTINUE19]] ], [ [[TMP60]], [[PRED_LOAD_IF20]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP62:%.*]] = extractelement <8 x i1> [[TMP5]], i64 1
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP62]], label [[PRED_LOAD_IF22:%.*]], label [[PRED_LOAD_CONTINUE23:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.load.if22:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP63:%.*]] = extractelement <8 x i32> [[TMP55]], i64 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP64:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP63]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP65:%.*]] = load i8, ptr [[TMP64]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP66:%.*]] = insertelement <8 x i8> [[TMP61]], i8 [[TMP65]], i64 1
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE23]]
-; ENABLED_MASKED_STRIDED:       pred.load.continue23:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP67:%.*]] = phi <8 x i8> [ [[TMP61]], [[PRED_LOAD_CONTINUE21]] ], [ [[TMP66]], [[PRED_LOAD_IF22]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP68:%.*]] = extractelement <8 x i1> [[TMP5]], i64 2
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP68]], label [[PRED_LOAD_IF24:%.*]], label [[PRED_LOAD_CONTINUE25:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.load.if24:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP69:%.*]] = extractelement <8 x i32> [[TMP55]], i64 2
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP70:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP69]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP71:%.*]] = load i8, ptr [[TMP70]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP72:%.*]] = insertelement <8 x i8> [[TMP67]], i8 [[TMP71]], i64 2
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE25]]
-; ENABLED_MASKED_STRIDED:       pred.load.continue25:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP73:%.*]] = phi <8 x i8> [ [[TMP67]], [[PRED_LOAD_CONTINUE23]] ], [ [[TMP72]], [[PRED_LOAD_IF24]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP74:%.*]] = extractelement <8 x i1> [[TMP5]], i64 3
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP74]], label [[PRED_LOAD_IF26:%.*]], label [[PRED_LOAD_CONTINUE27:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.load.if26:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP75:%.*]] = extractelement <8 x i32> [[TMP55]], i64 3
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP76:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP75]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP77:%.*]] = load i8, ptr [[TMP76]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP78:%.*]] = insertelement <8 x i8> [[TMP73]], i8 [[TMP77]], i64 3
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE27]]
-; ENABLED_MASKED_STRIDED:       pred.load.continue27:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP79:%.*]] = phi <8 x i8> [ [[TMP73]], [[PRED_LOAD_CONTINUE25]] ], [ [[TMP78]], [[PRED_LOAD_IF26]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP80:%.*]] = extractelement <8 x i1> [[TMP5]], i64 4
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP80]], label [[PRED_LOAD_IF28:%.*]], label [[PRED_LOAD_CONTINUE29:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.load.if28:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP81:%.*]] = extractelement <8 x i32> [[TMP55]], i64 4
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP82:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP81]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP83:%.*]] = load i8, ptr [[TMP82]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP84:%.*]] = insertelement <8 x i8> [[TMP79]], i8 [[TMP83]], i64 4
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE29]]
-; ENABLED_MASKED_STRIDED:       pred.load.continue29:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP85:%.*]] = phi <8 x i8> [ [[TMP79]], [[PRED_LOAD_CONTINUE27]] ], [ [[TMP84]], [[PRED_LOAD_IF28]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP86:%.*]] = extractelement <8 x i1> [[TMP5]], i64 5
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP86]], label [[PRED_LOAD_IF30:%.*]], label [[PRED_LOAD_CONTINUE31:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.load.if30:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP87:%.*]] = extractelement <8 x i32> [[TMP55]], i64 5
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP88:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP87]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP89:%.*]] = load i8, ptr [[TMP88]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP90:%.*]] = insertelement <8 x i8> [[TMP85]], i8 [[TMP89]], i64 5
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE31]]
-; ENABLED_MASKED_STRIDED:       pred.load.continue31:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP91:%.*]] = phi <8 x i8> [ [[TMP85]], [[PRED_LOAD_CONTINUE29]] ], [ [[TMP90]], [[PRED_LOAD_IF30]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP92:%.*]] = extractelement <8 x i1> [[TMP5]], i64 6
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP92]], label [[PRED_LOAD_IF32:%.*]], label [[PRED_LOAD_CONTINUE33:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.load.if32:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP93:%.*]] = extractelement <8 x i32> [[TMP55]], i64 6
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP94:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP93]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP95:%.*]] = load i8, ptr [[TMP94]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP96:%.*]] = insertelement <8 x i8> [[TMP91]], i8 [[TMP95]], i64 6
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE33]]
-; ENABLED_MASKED_STRIDED:       pred.load.continue33:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP97:%.*]] = phi <8 x i8> [ [[TMP91]], [[PRED_LOAD_CONTINUE31]] ], [ [[TMP96]], [[PRED_LOAD_IF32]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP98:%.*]] = extractelement <8 x i1> [[TMP5]], i64 7
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP98]], label [[PRED_LOAD_IF34:%.*]], label [[PRED_LOAD_CONTINUE35:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.load.if34:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP99:%.*]] = extractelement <8 x i32> [[TMP55]], i64 7
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP100:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP99]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP101:%.*]] = load i8, ptr [[TMP100]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP102:%.*]] = insertelement <8 x i8> [[TMP97]], i8 [[TMP101]], i64 7
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE35]]
-; ENABLED_MASKED_STRIDED:       pred.load.continue35:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP103:%.*]] = phi <8 x i8> [ [[TMP97]], [[PRED_LOAD_CONTINUE33]] ], [ [[TMP102]], [[PRED_LOAD_IF34]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP104:%.*]] = call <8 x i8> @llvm.smax.v8i8(<8 x i8> [[TMP54]], <8 x i8> [[TMP103]])
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP105:%.*]] = extractelement <8 x i1> [[TMP5]], i64 0
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP105]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP7:%.*]] = phi <8 x i8> [ poison, [[VECTOR_BODY]] ], [ [[TMP6]], [[PRED_LOAD_IF]] ]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP8:%.*]] = extractelement <8 x i1> [[TMP0]], i64 1
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP8]], label [[PRED_LOAD_IF1:%.*]], label [[PRED_LOAD_CONTINUE2:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.load.if1:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP9:%.*]] = extractelement <8 x i32> [[TMP1]], i64 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP10:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP9]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP11:%.*]] = load i8, ptr [[TMP10]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP12:%.*]] = insertelement <8 x i8> [[TMP7]], i8 [[TMP11]], i64 1
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE2]]
+; ENABLED_MASKED_STRIDED:       pred.load.continue2:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP13:%.*]] = phi <8 x i8> [ [[TMP7]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP12]], [[PRED_LOAD_IF1]] ]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP14:%.*]] = extractelement <8 x i1> [[TMP0]], i64 2
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP14]], label [[PRED_LOAD_IF3:%.*]], label [[PRED_LOAD_CONTINUE4:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.load.if3:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP15:%.*]] = extractelement <8 x i32> [[TMP1]], i64 2
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP16:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP15]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP17:%.*]] = load i8, ptr [[TMP16]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP18:%.*]] = insertelement <8 x i8> [[TMP13]], i8 [[TMP17]], i64 2
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE4]]
+; ENABLED_MASKED_STRIDED:       pred.load.continue4:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP19:%.*]] = phi <8 x i8> [ [[TMP13]], [[PRED_LOAD_CONTINUE2]] ], [ [[TMP18]], [[PRED_LOAD_IF3]] ]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP20:%.*]] = extractelement <8 x i1> [[TMP0]], i64 3
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP20]], label [[PRED_LOAD_IF5:%.*]], label [[PRED_LOAD_CONTINUE6:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.load.if5:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP21:%.*]] = extractelement <8 x i32> [[TMP1]], i64 3
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP22:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP21]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP23:%.*]] = load i8, ptr [[TMP22]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP24:%.*]] = insertelement <8 x i8> [[TMP19]], i8 [[TMP23]], i64 3
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE6]]
+; ENABLED_MASKED_STRIDED:       pred.load.continue6:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP25:%.*]] = phi <8 x i8> [ [[TMP19]], [[PRED_LOAD_CONTINUE4]] ], [ [[TMP24]], [[PRED_LOAD_IF5]] ]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP26:%.*]] = extractelement <8 x i1> [[TMP0]], i64 4
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP26]], label [[PRED_LOAD_IF7:%.*]], label [[PRED_LOAD_CONTINUE8:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.load.if7:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP27:%.*]] = extractelement <8 x i32> [[TMP1]], i64 4
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP28:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP27]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP29:%.*]] = load i8, ptr [[TMP28]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP30:%.*]] = insertelement <8 x i8> [[TMP25]], i8 [[TMP29]], i64 4
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE8]]
+; ENABLED_MASKED_STRIDED:       pred.load.continue8:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP31:%.*]] = phi <8 x i8> [ [[TMP25]], [[PRED_LOAD_CONTINUE6]] ], [ [[TMP30]], [[PRED_LOAD_IF7]] ]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP32:%.*]] = extractelement <8 x i1> [[TMP0]], i64 5
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP32]], label [[PRED_LOAD_IF9:%.*]], label [[PRED_LOAD_CONTINUE10:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.load.if9:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP33:%.*]] = extractelement <8 x i32> [[TMP1]], i64 5
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP34:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP33]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP35:%.*]] = load i8, ptr [[TMP34]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP36:%.*]] = insertelement <8 x i8> [[TMP31]], i8 [[TMP35]], i64 5
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE10]]
+; ENABLED_MASKED_STRIDED:       pred.load.continue10:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP37:%.*]] = phi <8 x i8> [ [[TMP31]], [[PRED_LOAD_CONTINUE8]] ], [ [[TMP36]], [[PRED_LOAD_IF9]] ]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP38:%.*]] = extractelement <8 x i1> [[TMP0]], i64 6
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP38]], label [[PRED_LOAD_IF11:%.*]], label [[PRED_LOAD_CONTINUE12:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.load.if11:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP39:%.*]] = extractelement <8 x i32> [[TMP1]], i64 6
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP40:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP39]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP41:%.*]] = load i8, ptr [[TMP40]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP42:%.*]] = insertelement <8 x i8> [[TMP37]], i8 [[TMP41]], i64 6
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE12]]
+; ENABLED_MASKED_STRIDED:       pred.load.continue12:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP43:%.*]] = phi <8 x i8> [ [[TMP37]], [[PRED_LOAD_CONTINUE10]] ], [ [[TMP42]], [[PRED_LOAD_IF11]] ]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP44:%.*]] = extractelement <8 x i1> [[TMP0]], i64 7
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP44]], label [[PRED_LOAD_IF13:%.*]], label [[PRED_LOAD_CONTINUE14:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.load.if13:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP45:%.*]] = extractelement <8 x i32> [[TMP1]], i64 7
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP46:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP45]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP47:%.*]] = load i8, ptr [[TMP46]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP48:%.*]] = insertelement <8 x i8> [[TMP43]], i8 [[TMP47]], i64 7
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE14]]
+; ENABLED_MASKED_STRIDED:       pred.load.continue14:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP49:%.*]] = phi <8 x i8> [ [[TMP43]], [[PRED_LOAD_CONTINUE12]] ], [ [[TMP48]], [[PRED_LOAD_IF13]] ]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP50:%.*]] = or <8 x i32> [[TMP1]], <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP51:%.*]] = extractelement <8 x i1> [[TMP0]], i64 0
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP51]], label [[PRED_LOAD_IF15:%.*]], label [[PRED_LOAD_CONTINUE16:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.load.if15:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP52:%.*]] = extractelement <8 x i32> [[TMP50]], i64 0
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP53:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP52]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP54:%.*]] = load i8, ptr [[TMP53]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP55:%.*]] = insertelement <8 x i8> poison, i8 [[TMP54]], i64 0
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE16]]
+; ENABLED_MASKED_STRIDED:       pred.load.continue16:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP56:%.*]] = phi <8 x i8> [ poison, [[PRED_LOAD_CONTINUE14]] ], [ [[TMP55]], [[PRED_LOAD_IF15]] ]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP57:%.*]] = extractelement <8 x i1> [[TMP0]], i64 1
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP57]], label [[PRED_LOAD_IF17:%.*]], label [[PRED_LOAD_CONTINUE18:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.load.if17:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP58:%.*]] = extractelement <8 x i32> [[TMP50]], i64 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP59:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP58]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP60:%.*]] = load i8, ptr [[TMP59]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP61:%.*]] = insertelement <8 x i8> [[TMP56]], i8 [[TMP60]], i64 1
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE18]]
+; ENABLED_MASKED_STRIDED:       pred.load.continue18:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP62:%.*]] = phi <8 x i8> [ [[TMP56]], [[PRED_LOAD_CONTINUE16]] ], [ [[TMP61]], [[PRED_LOAD_IF17]] ]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP63:%.*]] = extractelement <8 x i1> [[TMP0]], i64 2
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP63]], label [[PRED_LOAD_IF19:%.*]], label [[PRED_LOAD_CONTINUE20:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.load.if19:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP64:%.*]] = extractelement <8 x i32> [[TMP50]], i64 2
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP65:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP64]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP66:%.*]] = load i8, ptr [[TMP65]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP67:%.*]] = insertelement <8 x i8> [[TMP62]], i8 [[TMP66]], i64 2
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE20]]
+; ENABLED_MASKED_STRIDED:       pred.load.continue20:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP68:%.*]] = phi <8 x i8> [ [[TMP62]], [[PRED_LOAD_CONTINUE18]] ], [ [[TMP67]], [[PRED_LOAD_IF19]] ]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP69:%.*]] = extractelement <8 x i1> [[TMP0]], i64 3
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP69]], label [[PRED_LOAD_IF21:%.*]], label [[PRED_LOAD_CONTINUE22:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.load.if21:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP70:%.*]] = extractelement <8 x i32> [[TMP50]], i64 3
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP71:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP70]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP72:%.*]] = load i8, ptr [[TMP71]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP73:%.*]] = insertelement <8 x i8> [[TMP68]], i8 [[TMP72]], i64 3
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE22]]
+; ENABLED_MASKED_STRIDED:       pred.load.continue22:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP74:%.*]] = phi <8 x i8> [ [[TMP68]], [[PRED_LOAD_CONTINUE20]] ], [ [[TMP73]], [[PRED_LOAD_IF21]] ]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP75:%.*]] = extractelement <8 x i1> [[TMP0]], i64 4
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP75]], label [[PRED_LOAD_IF23:%.*]], label [[PRED_LOAD_CONTINUE24:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.load.if23:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP76:%.*]] = extractelement <8 x i32> [[TMP50]], i64 4
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP77:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP76]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP78:%.*]] = load i8, ptr [[TMP77]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP79:%.*]] = insertelement <8 x i8> [[TMP74]], i8 [[TMP78]], i64 4
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE24]]
+; ENABLED_MASKED_STRIDED:       pred.load.continue24:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP80:%.*]] = phi <8 x i8> [ [[TMP74]], [[PRED_LOAD_CONTINUE22]] ], [ [[TMP79]], [[PRED_LOAD_IF23]] ]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP81:%.*]] = extractelement <8 x i1> [[TMP0]], i64 5
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP81]], label [[PRED_LOAD_IF25:%.*]], label [[PRED_LOAD_CONTINUE26:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.load.if25:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP82:%.*]] = extractelement <8 x i32> [[TMP50]], i64 5
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP83:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP82]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP84:%.*]] = load i8, ptr [[TMP83]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP85:%.*]] = insertelement <8 x i8> [[TMP80]], i8 [[TMP84]], i64 5
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE26]]
+; ENABLED_MASKED_STRIDED:       pred.load.continue26:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP86:%.*]] = phi <8 x i8> [ [[TMP80]], [[PRED_LOAD_CONTINUE24]] ], [ [[TMP85]], [[PRED_LOAD_IF25]] ]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP87:%.*]] = extractelement <8 x i1> [[TMP0]], i64 6
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP87]], label [[PRED_LOAD_IF27:%.*]], label [[PRED_LOAD_CONTINUE28:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.load.if27:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP88:%.*]] = extractelement <8 x i32> [[TMP50]], i64 6
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP89:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP88]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP90:%.*]] = load i8, ptr [[TMP89]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP91:%.*]] = insertelement <8 x i8> [[TMP86]], i8 [[TMP90]], i64 6
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE28]]
+; ENABLED_MASKED_STRIDED:       pred.load.continue28:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP92:%.*]] = phi <8 x i8> [ [[TMP86]], [[PRED_LOAD_CONTINUE26]] ], [ [[TMP91]], [[PRED_LOAD_IF27]] ]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP93:%.*]] = extractelement <8 x i1> [[TMP0]], i64 7
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP93]], label [[PRED_LOAD_IF29:%.*]], label [[PRED_LOAD_CONTINUE30:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.load.if29:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP94:%.*]] = extractelement <8 x i32> [[TMP50]], i64 7
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP95:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[TMP94]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP96:%.*]] = load i8, ptr [[TMP95]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP97:%.*]] = insertelement <8 x i8> [[TMP92]], i8 [[TMP96]], i64 7
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_LOAD_CONTINUE30]]
+; ENABLED_MASKED_STRIDED:       pred.load.continue30:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP98:%.*]] = phi <8 x i8> [ [[TMP92]], [[PRED_LOAD_CONTINUE28]] ], [ [[TMP97]], [[PRED_LOAD_IF29]] ]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP99:%.*]] = call <8 x i8> @llvm.smax.v8i8(<8 x i8> [[TMP49]], <8 x i8> [[TMP98]])
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP100:%.*]] = extractelement <8 x i1> [[TMP0]], i64 0
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP100]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
 ; ENABLED_MASKED_STRIDED:       pred.store.if:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP106:%.*]] = extractelement <8 x i32> [[TMP6]], i64 0
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP107:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP106]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP108:%.*]] = extractelement <8 x i8> [[TMP104]], i64 0
-; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP108]], ptr [[TMP107]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP101:%.*]] = extractelement <8 x i32> [[TMP1]], i64 0
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP102:%.*]] = getelementptr inbounds i8, ptr [[Q:%.*]], i32 [[TMP101]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP103:%.*]] = extractelement <8 x i8> [[TMP99]], i64 0
+; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP103]], ptr [[TMP102]], align 1
 ; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE]]
 ; ENABLED_MASKED_STRIDED:       pred.store.continue:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP109:%.*]] = extractelement <8 x i1> [[TMP5]], i64 1
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP109]], label [[PRED_STORE_IF36:%.*]], label [[PRED_STORE_CONTINUE37:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.store.if36:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP110:%.*]] = extractelement <8 x i32> [[TMP6]], i64 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP111:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP110]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP112:%.*]] = extractelement <8 x i8> [[TMP104]], i64 1
-; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP112]], ptr [[TMP111]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE37]]
-; ENABLED_MASKED_STRIDED:       pred.store.continue37:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP113:%.*]] = extractelement <8 x i1> [[TMP5]], i64 2
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP113]], label [[PRED_STORE_IF38:%.*]], label [[PRED_STORE_CONTINUE39:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.store.if38:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP114:%.*]] = extractelement <8 x i32> [[TMP6]], i64 2
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP115:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP114]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP116:%.*]] = extractelement <8 x i8> [[TMP104]], i64 2
-; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP116]], ptr [[TMP115]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE39]]
-; ENABLED_MASKED_STRIDED:       pred.store.continue39:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP117:%.*]] = extractelement <8 x i1> [[TMP5]], i64 3
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP117]], label [[PRED_STORE_IF40:%.*]], label [[PRED_STORE_CONTINUE41:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.store.if40:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP118:%.*]] = extractelement <8 x i32> [[TMP6]], i64 3
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP119:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP118]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP120:%.*]] = extractelement <8 x i8> [[TMP104]], i64 3
-; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP120]], ptr [[TMP119]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE41]]
-; ENABLED_MASKED_STRIDED:       pred.store.continue41:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP121:%.*]] = extractelement <8 x i1> [[TMP5]], i64 4
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP121]], label [[PRED_STORE_IF42:%.*]], label [[PRED_STORE_CONTINUE43:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.store.if42:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP122:%.*]] = extractelement <8 x i32> [[TMP6]], i64 4
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP123:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP122]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP124:%.*]] = extractelement <8 x i8> [[TMP104]], i64 4
-; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP124]], ptr [[TMP123]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE43]]
-; ENABLED_MASKED_STRIDED:       pred.store.continue43:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP125:%.*]] = extractelement <8 x i1> [[TMP5]], i64 5
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP125]], label [[PRED_STORE_IF44:%.*]], label [[PRED_STORE_CONTINUE45:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.store.if44:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP126:%.*]] = extractelement <8 x i32> [[TMP6]], i64 5
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP127:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP126]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP128:%.*]] = extractelement <8 x i8> [[TMP104]], i64 5
-; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP128]], ptr [[TMP127]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE45]]
-; ENABLED_MASKED_STRIDED:       pred.store.continue45:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP129:%.*]] = extractelement <8 x i1> [[TMP5]], i64 6
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP129]], label [[PRED_STORE_IF46:%.*]], label [[PRED_STORE_CONTINUE47:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.store.if46:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP130:%.*]] = extractelement <8 x i32> [[TMP6]], i64 6
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP131:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP130]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP132:%.*]] = extractelement <8 x i8> [[TMP104]], i64 6
-; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP132]], ptr [[TMP131]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE47]]
-; ENABLED_MASKED_STRIDED:       pred.store.continue47:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP133:%.*]] = extractelement <8 x i1> [[TMP5]], i64 7
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP133]], label [[PRED_STORE_IF48:%.*]], label [[PRED_STORE_CONTINUE49:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.store.if48:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP134:%.*]] = extractelement <8 x i32> [[TMP6]], i64 7
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP104:%.*]] = extractelement <8 x i1> [[TMP0]], i64 1
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP104]], label [[PRED_STORE_IF31:%.*]], label [[PRED_STORE_CONTINUE32:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.store.if31:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP105:%.*]] = extractelement <8 x i32> [[TMP1]], i64 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP106:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP105]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP107:%.*]] = extractelement <8 x i8> [[TMP99]], i64 1
+; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP107]], ptr [[TMP106]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE32]]
+; ENABLED_MASKED_STRIDED:       pred.store.continue32:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP108:%.*]] = extractelement <8 x i1> [[TMP0]], i64 2
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP108]], label [[PRED_STORE_IF33:%.*]], label [[PRED_STORE_CONTINUE34:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.store.if33:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP109:%.*]] = extractelement <8 x i32> [[TMP1]], i64 2
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP110:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP109]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP111:%.*]] = extractelement <8 x i8> [[TMP99]], i64 2
+; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP111]], ptr [[TMP110]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE34]]
+; ENABLED_MASKED_STRIDED:       pred.store.continue34:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP112:%.*]] = extractelement <8 x i1> [[TMP0]], i64 3
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP112]], label [[PRED_STORE_IF35:%.*]], label [[PRED_STORE_CONTINUE36:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.store.if35:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP113:%.*]] = extractelement <8 x i32> [[TMP1]], i64 3
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP114:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP113]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP115:%.*]] = extractelement <8 x i8> [[TMP99]], i64 3
+; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP115]], ptr [[TMP114]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE36]]
+; ENABLED_MASKED_STRIDED:       pred.store.continue36:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP116:%.*]] = extractelement <8 x i1> [[TMP0]], i64 4
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP116]], label [[PRED_STORE_IF37:%.*]], label [[PRED_STORE_CONTINUE38:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.store.if37:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP117:%.*]] = extractelement <8 x i32> [[TMP1]], i64 4
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP118:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP117]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP119:%.*]] = extractelement <8 x i8> [[TMP99]], i64 4
+; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP119]], ptr [[TMP118]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE38]]
+; ENABLED_MASKED_STRIDED:       pred.store.continue38:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP120:%.*]] = extractelement <8 x i1> [[TMP0]], i64 5
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP120]], label [[PRED_STORE_IF39:%.*]], label [[PRED_STORE_CONTINUE40:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.store.if39:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP121:%.*]] = extractelement <8 x i32> [[TMP1]], i64 5
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP122:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP121]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP123:%.*]] = extractelement <8 x i8> [[TMP99]], i64 5
+; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP123]], ptr [[TMP122]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE40]]
+; ENABLED_MASKED_STRIDED:       pred.store.continue40:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP124:%.*]] = extractelement <8 x i1> [[TMP0]], i64 6
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP124]], label [[PRED_STORE_IF41:%.*]], label [[PRED_STORE_CONTINUE42:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.store.if41:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP125:%.*]] = extractelement <8 x i32> [[TMP1]], i64 6
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP126:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP125]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP127:%.*]] = extractelement <8 x i8> [[TMP99]], i64 6
+; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP127]], ptr [[TMP126]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE42]]
+; ENABLED_MASKED_STRIDED:       pred.store.continue42:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP128:%.*]] = extractelement <8 x i1> [[TMP0]], i64 7
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP128]], label [[PRED_STORE_IF43:%.*]], label [[PRED_STORE_CONTINUE44:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.store.if43:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP129:%.*]] = extractelement <8 x i32> [[TMP1]], i64 7
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP130:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP129]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP131:%.*]] = extractelement <8 x i8> [[TMP99]], i64 7
+; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP131]], ptr [[TMP130]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE44]]
+; ENABLED_MASKED_STRIDED:       pred.store.continue44:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP132:%.*]] = sub <8 x i8> zeroinitializer, [[TMP99]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP133:%.*]] = extractelement <8 x i1> [[TMP0]], i64 0
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP133]], label [[PRED_STORE_IF45:%.*]], label [[PRED_STORE_CONTINUE46:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.store.if45:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP134:%.*]] = extractelement <8 x i32> [[TMP50]], i64 0
 ; ENABLED_MASKED_STRIDED-NEXT:    [[TMP135:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP134]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP136:%.*]] = extractelement <8 x i8> [[TMP104]], i64 7
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP136:%.*]] = extractelement <8 x i8> [[TMP132]], i64 0
 ; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP136]], ptr [[TMP135]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE49]]
-; ENABLED_MASKED_STRIDED:       pred.store.continue49:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP137:%.*]] = sub <8 x i8> zeroinitializer, [[TMP104]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP138:%.*]] = extractelement <8 x i1> [[TMP5]], i64 0
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP138]], label [[PRED_STORE_IF50:%.*]], label [[PRED_STORE_CONTINUE51:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.store.if50:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP139:%.*]] = extractelement <8 x i32> [[TMP55]], i64 0
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP140:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP139]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP141:%.*]] = extractelement <8 x i8> [[TMP137]], i64 0
-; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP141]], ptr [[TMP140]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE51]]
-; ENABLED_MASKED_STRIDED:       pred.store.continue51:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP142:%.*]] = extractelement <8 x i1> [[TMP5]], i64 1
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP142]], label [[PRED_STORE_IF52:%.*]], label [[PRED_STORE_CONTINUE53:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.store.if52:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP143:%.*]] = extractelement <8 x i32> [[TMP55]], i64 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP144:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP143]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP145:%.*]] = extractelement <8 x i8> [[TMP137]], i64 1
-; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP145]], ptr [[TMP144]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE53]]
-; ENABLED_MASKED_STRIDED:       pred.store.continue53:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP146:%.*]] = extractelement <8 x i1> [[TMP5]], i64 2
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP146]], label [[PRED_STORE_IF54:%.*]], label [[PRED_STORE_CONTINUE55:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.store.if54:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP147:%.*]] = extractelement <8 x i32> [[TMP55]], i64 2
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP148:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP147]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP149:%.*]] = extractelement <8 x i8> [[TMP137]], i64 2
-; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP149]], ptr [[TMP148]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE55]]
-; ENABLED_MASKED_STRIDED:       pred.store.continue55:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP150:%.*]] = extractelement <8 x i1> [[TMP5]], i64 3
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP150]], label [[PRED_STORE_IF56:%.*]], label [[PRED_STORE_CONTINUE57:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.store.if56:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP151:%.*]] = extractelement <8 x i32> [[TMP55]], i64 3
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP152:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP151]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP153:%.*]] = extractelement <8 x i8> [[TMP137]], i64 3
-; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP153]], ptr [[TMP152]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE57]]
-; ENABLED_MASKED_STRIDED:       pred.store.continue57:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP154:%.*]] = extractelement <8 x i1> [[TMP5]], i64 4
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP154]], label [[PRED_STORE_IF58:%.*]], label [[PRED_STORE_CONTINUE59:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.store.if58:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP155:%.*]] = extractelement <8 x i32> [[TMP55]], i64 4
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP156:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP155]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP157:%.*]] = extractelement <8 x i8> [[TMP137]], i64 4
-; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP157]], ptr [[TMP156]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE59]]
-; ENABLED_MASKED_STRIDED:       pred.store.continue59:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP158:%.*]] = extractelement <8 x i1> [[TMP5]], i64 5
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP158]], label [[PRED_STORE_IF60:%.*]], label [[PRED_STORE_CONTINUE61:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.store.if60:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP159:%.*]] = extractelement <8 x i32> [[TMP55]], i64 5
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP160:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP159]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP161:%.*]] = extractelement <8 x i8> [[TMP137]], i64 5
-; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP161]], ptr [[TMP160]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE61]]
-; ENABLED_MASKED_STRIDED:       pred.store.continue61:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP162:%.*]] = extractelement <8 x i1> [[TMP5]], i64 6
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP162]], label [[PRED_STORE_IF62:%.*]], label [[PRED_STORE_CONTINUE63:%.*]]
-; ENABLED_MASKED_STRIDED:       pred.store.if62:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP163:%.*]] = extractelement <8 x i32> [[TMP55]], i64 6
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP164:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP163]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP165:%.*]] = extractelement <8 x i8> [[TMP137]], i64 6
-; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP165]], ptr [[TMP164]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE63]]
-; ENABLED_MASKED_STRIDED:       pred.store.continue63:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP166:%.*]] = extractelement <8 x i1> [[TMP5]], i64 7
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP166]], label [[PRED_STORE_IF64:%.*]], label [[PRED_STORE_CONTINUE65]]
-; ENABLED_MASKED_STRIDED:       pred.store.if64:
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP167:%.*]] = extractelement <8 x i32> [[TMP55]], i64 7
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP168:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP167]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP169:%.*]] = extractelement <8 x i8> [[TMP137]], i64 7
-; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP169]], ptr [[TMP168]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE65]]
-; ENABLED_MASKED_STRIDED:       pred.store.continue65:
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE46]]
+; ENABLED_MASKED_STRIDED:       pred.store.continue46:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP137:%.*]] = extractelement <8 x i1> [[TMP0]], i64 1
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP137]], label [[PRED_STORE_IF47:%.*]], label [[PRED_STORE_CONTINUE48:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.store.if47:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP138:%.*]] = extractelement <8 x i32> [[TMP50]], i64 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP139:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP138]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP140:%.*]] = extractelement <8 x i8> [[TMP132]], i64 1
+; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP140]], ptr [[TMP139]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE48]]
+; ENABLED_MASKED_STRIDED:       pred.store.continue48:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP141:%.*]] = extractelement <8 x i1> [[TMP0]], i64 2
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP141]], label [[PRED_STORE_IF49:%.*]], label [[PRED_STORE_CONTINUE50:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.store.if49:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP142:%.*]] = extractelement <8 x i32> [[TMP50]], i64 2
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP143:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP142]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP144:%.*]] = extractelement <8 x i8> [[TMP132]], i64 2
+; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP144]], ptr [[TMP143]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE50]]
+; ENABLED_MASKED_STRIDED:       pred.store.continue50:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP145:%.*]] = extractelement <8 x i1> [[TMP0]], i64 3
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP145]], label [[PRED_STORE_IF51:%.*]], label [[PRED_STORE_CONTINUE52:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.store.if51:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP146:%.*]] = extractelement <8 x i32> [[TMP50]], i64 3
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP147:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP146]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP148:%.*]] = extractelement <8 x i8> [[TMP132]], i64 3
+; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP148]], ptr [[TMP147]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE52]]
+; ENABLED_MASKED_STRIDED:       pred.store.continue52:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP149:%.*]] = extractelement <8 x i1> [[TMP0]], i64 4
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP149]], label [[PRED_STORE_IF53:%.*]], label [[PRED_STORE_CONTINUE54:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.store.if53:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP150:%.*]] = extractelement <8 x i32> [[TMP50]], i64 4
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP151:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP150]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP152:%.*]] = extractelement <8 x i8> [[TMP132]], i64 4
+; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP152]], ptr [[TMP151]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE54]]
+; ENABLED_MASKED_STRIDED:       pred.store.continue54:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP153:%.*]] = extractelement <8 x i1> [[TMP0]], i64 5
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP153]], label [[PRED_STORE_IF55:%.*]], label [[PRED_STORE_CONTINUE56:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.store.if55:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP154:%.*]] = extractelement <8 x i32> [[TMP50]], i64 5
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP155:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP154]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP156:%.*]] = extractelement <8 x i8> [[TMP132]], i64 5
+; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP156]], ptr [[TMP155]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE56]]
+; ENABLED_MASKED_STRIDED:       pred.store.continue56:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP157:%.*]] = extractelement <8 x i1> [[TMP0]], i64 6
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP157]], label [[PRED_STORE_IF57:%.*]], label [[PRED_STORE_CONTINUE58:%.*]]
+; ENABLED_MASKED_STRIDED:       pred.store.if57:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP158:%.*]] = extractelement <8 x i32> [[TMP50]], i64 6
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP159:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP158]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP160:%.*]] = extractelement <8 x i8> [[TMP132]], i64 6
+; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP160]], ptr [[TMP159]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE58]]
+; ENABLED_MASKED_STRIDED:       pred.store.continue58:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP161:%.*]] = extractelement <8 x i1> [[TMP0]], i64 7
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP161]], label [[PRED_STORE_IF59:%.*]], label [[PRED_STORE_CONTINUE60]]
+; ENABLED_MASKED_STRIDED:       pred.store.if59:
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP162:%.*]] = extractelement <8 x i32> [[TMP50]], i64 7
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP163:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[TMP162]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP164:%.*]] = extractelement <8 x i8> [[TMP132]], i64 7
+; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP164]], ptr [[TMP163]], align 1
+; ENABLED_MASKED_STRIDED-NEXT:    br label [[PRED_STORE_CONTINUE60]]
+; ENABLED_MASKED_STRIDED:       pred.store.continue60:
 ; ENABLED_MASKED_STRIDED-NEXT:    [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 8
 ; ENABLED_MASKED_STRIDED-NEXT:    [[VEC_IND_NEXT]] = add <8 x i32> [[VEC_IND]], <i32 -8, i32 -8, i32 -8, i32 -8, i32 -8, i32 -8, i32 -8, i32 -8>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP170:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP170]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP10:![0-9]+]]
-; ENABLED_MASKED_STRIDED:       for.body:
-; ENABLED_MASKED_STRIDED-NEXT:    [[IX_024:%.*]] = phi i32 [ [[INC:%.*]], [[FOR_INC:%.*]] ], [ 1024, [[ENTRY:%.*]] ]
-; ENABLED_MASKED_STRIDED-NEXT:    [[CMP1:%.*]] = icmp ugt i32 [[IX_024]], [[CONV]]
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[CMP1]], label [[IF_THEN:%.*]], label [[FOR_INC]]
-; ENABLED_MASKED_STRIDED:       if.then:
-; ENABLED_MASKED_STRIDED-NEXT:    [[MUL:%.*]] = shl nuw nsw i32 [[IX_024]], 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[MUL]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP171:%.*]] = load i8, ptr [[ARRAYIDX]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[ADD:%.*]] = or i32 [[MUL]], 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[ARRAYIDX4:%.*]] = getelementptr inbounds i8, ptr [[P]], i32 [[ADD]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP172:%.*]] = load i8, ptr [[ARRAYIDX4]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP173:%.*]] = call i8 @llvm.smax.i8(i8 [[TMP171]], i8 [[TMP172]])
-; ENABLED_MASKED_STRIDED-NEXT:    [[ARRAYIDX6:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[MUL]]
-; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[TMP173]], ptr [[ARRAYIDX6]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[SUB:%.*]] = sub i8 0, [[TMP173]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[ARRAYIDX11:%.*]] = getelementptr inbounds i8, ptr [[Q]], i32 [[ADD]]
-; ENABLED_MASKED_STRIDED-NEXT:    store i8 [[SUB]], ptr [[ARRAYIDX11]], align 1
-; ENABLED_MASKED_STRIDED-NEXT:    br label [[FOR_INC]]
-; ENABLED_MASKED_STRIDED:       for.inc:
-; ENABLED_MASKED_STRIDED-NEXT:    [[INC]] = add nsw i32 [[IX_024]], -1
-; ENABLED_MASKED_STRIDED-NEXT:    [[EXITCOND:%.*]] = icmp eq i32 [[INC]], 0
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[EXITCOND]], label [[FOR_END]], label [[FOR_BODY]], !llvm.loop [[LOOP11:![0-9]+]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP165:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP165]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP10:![0-9]+]]
 ; ENABLED_MASKED_STRIDED:       for.end:
 ; ENABLED_MASKED_STRIDED-NEXT:    ret void
 ;
@@ -2617,17 +2555,17 @@ define dso_local void @masked_strided2_unknown_tc(ptr noalias nocapture readonly
 ; ENABLED_MASKED_STRIDED-NEXT:    [[WIDE_MASKED_VEC:%.*]] = call <16 x i8> @llvm.masked.load.v16i8.p0(ptr [[TMP3]], i32 1, <16 x i1> [[INTERLEAVED_MASK]], <16 x i8> poison)
 ; ENABLED_MASKED_STRIDED-NEXT:    [[STRIDED_VEC:%.*]] = shufflevector <16 x i8> [[WIDE_MASKED_VEC]], <16 x i8> poison, <8 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14>
 ; ENABLED_MASKED_STRIDED-NEXT:    [[STRIDED_VEC3:%.*]] = shufflevector <16 x i8> [[WIDE_MASKED_VEC]], <16 x i8> poison, <8 x i32> <i32 1, i32 3, i32 5, i32 7, i32 9, i32 11, i32 13, i32 15>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP6:%.*]] = or i32 [[TMP2]], 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP7:%.*]] = call <8 x i8> @llvm.smax.v8i8(<8 x i8> [[STRIDED_VEC]], <8 x i8> [[STRIDED_VEC3]])
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP8:%.*]] = sub <8 x i8> zeroinitializer, [[TMP7]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP9:%.*]] = getelementptr i8, ptr [[Q:%.*]], i32 -1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP10:%.*]] = getelementptr i8, ptr [[TMP9]], i32 [[TMP6]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[INTERLEAVED_VEC:%.*]] = shufflevector <8 x i8> [[TMP7]], <8 x i8> [[TMP8]], <16 x i32> <i32 0, i32 8, i32 1, i32 9, i32 2, i32 10, i32 3, i32 11, i32 4, i32 12, i32 5, i32 13, i32 6, i32 14, i32 7, i32 15>
-; ENABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v16i8.p0(<16 x i8> [[INTERLEAVED_VEC]], ptr [[TMP10]], i32 1, <16 x i1> [[INTERLEAVED_MASK]])
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP5:%.*]] = or i32 [[TMP2]], 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP6:%.*]] = call <8 x i8> @llvm.smax.v8i8(<8 x i8> [[STRIDED_VEC]], <8 x i8> [[STRIDED_VEC3]])
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP7:%.*]] = sub <8 x i8> zeroinitializer, [[TMP6]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP8:%.*]] = getelementptr i8, ptr [[Q:%.*]], i32 -1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP9:%.*]] = getelementptr i8, ptr [[TMP8]], i32 [[TMP5]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[INTERLEAVED_VEC:%.*]] = shufflevector <8 x i8> [[TMP6]], <8 x i8> [[TMP7]], <16 x i32> <i32 0, i32 8, i32 1, i32 9, i32 2, i32 10, i32 3, i32 11, i32 4, i32 12, i32 5, i32 13, i32 6, i32 14, i32 7, i32 15>
+; ENABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v16i8.p0(<16 x i8> [[INTERLEAVED_VEC]], ptr [[TMP9]], i32 1, <16 x i1> [[INTERLEAVED_MASK]])
 ; ENABLED_MASKED_STRIDED-NEXT:    [[INDEX_NEXT]] = add i32 [[INDEX]], 8
 ; ENABLED_MASKED_STRIDED-NEXT:    [[VEC_IND_NEXT]] = add <8 x i32> [[VEC_IND]], <i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP12:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP12]], label [[FOR_END]], label [[VECTOR_BODY]], !llvm.loop [[LOOP12:![0-9]+]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP10:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP10]], label [[FOR_END]], label [[VECTOR_BODY]], !llvm.loop [[LOOP11:![0-9]+]]
 ; ENABLED_MASKED_STRIDED:       for.end:
 ; ENABLED_MASKED_STRIDED-NEXT:    ret void
 ;
@@ -3051,16 +2989,16 @@ define dso_local void @unconditional_masked_strided2_unknown_tc(ptr noalias noca
 ; ENABLED_MASKED_STRIDED-NEXT:    [[WIDE_MASKED_VEC:%.*]] = call <16 x i8> @llvm.masked.load.v16i8.p0(ptr [[TMP2]], i32 1, <16 x i1> [[INTERLEAVED_MASK]], <16 x i8> poison)
 ; ENABLED_MASKED_STRIDED-NEXT:    [[STRIDED_VEC:%.*]] = shufflevector <16 x i8> [[WIDE_MASKED_VEC]], <16 x i8> poison, <8 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14>
 ; ENABLED_MASKED_STRIDED-NEXT:    [[STRIDED_VEC3:%.*]] = shufflevector <16 x i8> [[WIDE_MASKED_VEC]], <16 x i8> poison, <8 x i32> <i32 1, i32 3, i32 5, i32 7, i32 9, i32 11, i32 13, i32 15>
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP4:%.*]] = or i32 [[TMP1]], 1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP5:%.*]] = call <8 x i8> @llvm.smax.v8i8(<8 x i8> [[STRIDED_VEC]], <8 x i8> [[STRIDED_VEC3]])
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP6:%.*]] = sub <8 x i8> zeroinitializer, [[TMP5]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP7:%.*]] = getelementptr i8, ptr [[Q:%.*]], i32 -1
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP8:%.*]] = getelementptr i8, ptr [[TMP7]], i32 [[TMP4]]
-; ENABLED_MASKED_STRIDED-NEXT:    [[INTERLEAVED_VEC:%.*]] = shufflevector <8 x i8> [[TMP5]], <8 x i8> [[TMP6]], <16 x i32> <i32 0, i32 8, i32 1, i32 9, i32 2, i32 10, i32 3, i32 11, i32 4, i32 12, i32 5, i32 13, i32 6, i32 14, i32 7, i32 15>
-; ENABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v16i8.p0(<16 x i8> [[INTERLEAVED_VEC]], ptr [[TMP8]], i32 1, <16 x i1> [[INTERLEAVED_MASK]])
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP3:%.*]] = or i32 [[TMP1]], 1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP4:%.*]] = call <8 x i8> @llvm.smax.v8i8(<8 x i8> [[STRIDED_VEC]], <8 x i8> [[STRIDED_VEC3]])
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP5:%.*]] = sub <8 x i8> zeroinitializer, [[TMP4]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP6:%.*]] = getelementptr i8, ptr [[Q:%.*]], i32 -1
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP7:%.*]] = getelementptr i8, ptr [[TMP6]], i32 [[TMP3]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[INTERLEAVED_VEC:%.*]] = shufflevector <8 x i8> [[TMP4]], <8 x i8> [[TMP5]], <16 x i32> <i32 0, i32 8, i32 1, i32 9, i32 2, i32 10, i32 3, i32 11, i32 4, i32 12, i32 5, i32 13, i32 6, i32 14, i32 7, i32 15>
+; ENABLED_MASKED_STRIDED-NEXT:    call void @llvm.masked.store.v16i8.p0(<16 x i8> [[INTERLEAVED_VEC]], ptr [[TMP7]], i32 1, <16 x i1> [[INTERLEAVED_MASK]])
 ; ENABLED_MASKED_STRIDED-NEXT:    [[INDEX_NEXT]] = add i32 [[INDEX]], 8
-; ENABLED_MASKED_STRIDED-NEXT:    [[TMP10:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
-; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP10]], label [[FOR_END]], label [[VECTOR_BODY]], !llvm.loop [[LOOP13:![0-9]+]]
+; ENABLED_MASKED_STRIDED-NEXT:    [[TMP8:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
+; ENABLED_MASKED_STRIDED-NEXT:    br i1 [[TMP8]], label [[FOR_END]], label [[VECTOR_BODY]], !llvm.loop [[LOOP12:![0-9]+]]
 ; ENABLED_MASKED_STRIDED:       for.end:
 ; ENABLED_MASKED_STRIDED-NEXT:    ret void
 ;
diff --git a/llvm/test/Transforms/LoopVectorize/consecutive-ptr-uniforms.ll b/llvm/test/Transforms/LoopVectorize/consecutive-ptr-uniforms.ll
index 40337e7da7a46..29b2c4533e394 100644
--- a/llvm/test/Transforms/LoopVectorize/consecutive-ptr-uniforms.ll
+++ b/llvm/test/Transforms/LoopVectorize/consecutive-ptr-uniforms.ll
@@ -1,6 +1,6 @@
 ; REQUIRES: asserts
-; RUN: opt -opaque-pointers=0 < %s -passes=loop-vectorize,instcombine -force-vector-width=4 -force-vector-interleave=1 -debug-only=loop-vectorize -disable-output -print-after=instcombine 2>&1 | FileCheck %s
-; RUN: opt -opaque-pointers=0 < %s -passes=loop-vectorize,instcombine -force-vector-width=4 -force-vector-interleave=1 -enable-interleaved-mem-accesses -debug-only=loop-vectorize -disable-output -print-after=instcombine 2>&1 | FileCheck %s --check-prefix=INTER
+; RUN: opt %s -passes=loop-vectorize,instcombine -force-vector-width=4 -force-vector-interleave=1 -debug-only=loop-vectorize -disable-output -print-after=instcombine 2>&1 | FileCheck %s
+; RUN: opt %s -passes=loop-vectorize,instcombine -force-vector-width=4 -force-vector-interleave=1 -enable-interleaved-mem-accesses -debug-only=loop-vectorize -disable-output -print-after=instcombine 2>&1 | FileCheck %s --check-prefix=INTER
 
 target datalayout = "e-m:e-i64:64-i128:128-n32:64-S128"
 
@@ -11,23 +11,23 @@ target datalayout = "e-m:e-i64:64-i128:128-n32:64-S128"
 ; Check that a forward consecutive pointer is recognized as uniform and remains
 ; uniform after vectorization.
 ;
-; CHECK:     LV: Found uniform instruction: %tmp1 = getelementptr inbounds i32, i32* %a, i64 %i
+; CHECK:     LV: Found uniform instruction: %tmp1 = getelementptr inbounds i32, ptr %a, i64 %i
 ; CHECK:     vector.body
 ; CHECK:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
 ; CHECK-NOT:   getelementptr
-; CHECK:       getelementptr inbounds i32, i32* %a, i64 %index
+; CHECK:       getelementptr inbounds i32, ptr %a, i64 %index
 ; CHECK-NOT:   getelementptr
 ; CHECK:       br i1 {{.*}}, label %middle.block, label %vector.body
 ;
-define i32 @consecutive_ptr_forward(i32* %a, i64 %n) {
+define i32 @consecutive_ptr_forward(ptr %a, i64 %n) {
 entry:
   br label %for.body
 
 for.body:
   %i = phi i64 [ %i.next, %for.body ], [ 0, %entry ]
   %tmp0 = phi i32 [ %tmp3, %for.body ], [ 0, %entry ]
-  %tmp1 = getelementptr inbounds i32, i32* %a, i64 %i
-  %tmp2 = load i32, i32* %tmp1, align 8
+  %tmp1 = getelementptr inbounds i32, ptr %a, i64 %i
+  %tmp2 = load i32, ptr %tmp1, align 8
   %tmp3 = add i32 %tmp0, %tmp2
   %i.next = add nuw nsw i64 %i, 1
   %cond = icmp slt i64 %i.next, %n
@@ -43,25 +43,25 @@ for.end:
 ; Check that a reverse consecutive pointer is recognized as uniform and remains
 ; uniform after vectorization.
 ;
-; CHECK:     LV: Found uniform instruction: %tmp1 = getelementptr inbounds i32, i32* %a, i64 %i
+; CHECK:     LV: Found uniform instruction: %tmp1 = getelementptr inbounds i32, ptr %a, i64 %i
 ; CHECK:     vector.body
 ; CHECK:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
 ; CHECK:       %offset.idx = sub i64 %n, %index
 ; CHECK-NOT:   getelementptr
-; CHECK:       %[[G0:.+]] = getelementptr i32, i32* %a, i64 -3
-; CHECK:       getelementptr i32, i32* %[[G0]], i64 %offset.idx
+; CHECK:       %[[G0:.+]] = getelementptr i32, ptr %a, i64 -3
+; CHECK:       getelementptr i32, ptr %[[G0]], i64 %offset.idx
 ; CHECK-NOT:   getelementptr
 ; CHECK:       br i1 {{.*}}, label %middle.block, label %vector.body
 ;
-define i32 @consecutive_ptr_reverse(i32* %a, i64 %n) {
+define i32 @consecutive_ptr_reverse(ptr %a, i64 %n) {
 entry:
   br label %for.body
 
 for.body:
   %i = phi i64 [ %i.next, %for.body ], [ %n, %entry ]
   %tmp0 = phi i32 [ %tmp3, %for.body ], [ 0, %entry ]
-  %tmp1 = getelementptr inbounds i32, i32* %a, i64 %i
-  %tmp2 = load i32, i32* %tmp1, align 8
+  %tmp1 = getelementptr inbounds i32, ptr %a, i64 %i
+  %tmp2 = load i32, ptr %tmp1, align 8
   %tmp3 = add i32 %tmp0, %tmp2
   %i.next = add nsw i64 %i, -1
   %cond = icmp sgt i64 %i.next, 0
@@ -80,43 +80,43 @@ for.end:
 ; interleaved memory accesses aren't enabled, the pointer should not be
 ; recognized as uniform, and it should not be uniform after vectorization.
 ;
-; CHECK-NOT: LV: Found uniform instruction: %tmp1 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 0
-; CHECK-NOT: LV: Found uniform instruction: %tmp2 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 1
+; CHECK-NOT: LV: Found uniform instruction: %tmp1 = getelementptr inbounds %pair, ptr %p, i64 %i, i32 0
+; CHECK-NOT: LV: Found uniform instruction: %tmp2 = getelementptr inbounds %pair, ptr %p, i64 %i, i32 1
 ; CHECK:     vector.body
 ; CHECK:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
 ; CHECK:       %[[I1:.+]] = or i64 %index, 1
 ; CHECK:       %[[I2:.+]] = or i64 %index, 2
 ; CHECK:       %[[I3:.+]] = or i64 %index, 3
-; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %index, i32 0
-; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I1]], i32 0
-; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I2]], i32 0
-; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I3]], i32 0
-; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %index, i32 1
-; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I1]], i32 1
-; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I2]], i32 1
-; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I3]], i32 1
+; CHECK:       getelementptr inbounds %pair, ptr %p, i64 %index, i32 0
+; CHECK:       getelementptr inbounds %pair, ptr %p, i64 %[[I1]], i32 0
+; CHECK:       getelementptr inbounds %pair, ptr %p, i64 %[[I2]], i32 0
+; CHECK:       getelementptr inbounds %pair, ptr %p, i64 %[[I3]], i32 0
+; CHECK:       getelementptr inbounds %pair, ptr %p, i64 %index, i32 1
+; CHECK:       getelementptr inbounds %pair, ptr %p, i64 %[[I1]], i32 1
+; CHECK:       getelementptr inbounds %pair, ptr %p, i64 %[[I2]], i32 1
+; CHECK:       getelementptr inbounds %pair, ptr %p, i64 %[[I3]], i32 1
 ; CHECK:       br i1 {{.*}}, label %middle.block, label %vector.body
 ;
-; INTER:     LV: Found uniform instruction: %tmp1 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 0
-; INTER:     LV: Found uniform instruction: %tmp2 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 1
+; INTER:     LV: Found uniform instruction: %tmp1 = getelementptr inbounds %pair, ptr %p, i64 %i, i32 0
+; INTER:     LV: Found uniform instruction: %tmp2 = getelementptr inbounds %pair, ptr %p, i64 %i, i32 1
 ; INTER:     vector.body
 ; INTER:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
 ; INTER-NOT:   getelementptr
-; INTER:       getelementptr inbounds %pair, %pair* %p, i64 %index, i32 0
+; INTER:       getelementptr inbounds %pair, ptr %p, i64 %index, i32 0
 ; INTER-NOT:   getelementptr
 ; INTER:       br i1 {{.*}}, label %middle.block, label %vector.body
 ;
-define i32 @interleaved_access_forward(%pair* %p, i64 %n) {
+define i32 @interleaved_access_forward(ptr %p, i64 %n) {
 entry:
   br label %for.body
 
 for.body:
   %i = phi i64 [ %i.next, %for.body ], [ 0, %entry ]
   %tmp0 = phi i32 [ %tmp6, %for.body ], [ 0, %entry ]
-  %tmp1 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 0
-  %tmp2 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 1
-  %tmp3 = load i32, i32* %tmp1, align 8
-  %tmp4 = load i32, i32* %tmp2, align 8
+  %tmp1 = getelementptr inbounds %pair, ptr %p, i64 %i, i32 0
+  %tmp2 = getelementptr inbounds %pair, ptr %p, i64 %i, i32 1
+  %tmp3 = load i32, ptr %tmp1, align 8
+  %tmp4 = load i32, ptr %tmp2, align 8
   %tmp5 = add i32 %tmp3, %tmp4
   %tmp6 = add i32 %tmp0, %tmp5
   %i.next = add nuw nsw i64 %i, 1
@@ -137,46 +137,46 @@ for.end:
 ; recognized as uniform, and it should not be uniform after vectorization.
 ;
 ; recognized as uniform, and it should not be uniform after vectorization.
-; CHECK-NOT: LV: Found uniform instruction: %tmp1 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 0
-; CHECK-NOT: LV: Found uniform instruction: %tmp2 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 1
+; CHECK-NOT: LV: Found uniform instruction: %tmp1 = getelementptr inbounds %pair, ptr %p, i64 %i, i32 0
+; CHECK-NOT: LV: Found uniform instruction: %tmp2 = getelementptr inbounds %pair, ptr %p, i64 %i, i32 1
 ; CHECK:     vector.body
 ; CHECK:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
 ; CHECK:       %offset.idx = sub i64 %n, %index
 ; CHECK:       %[[I1:.+]] = add i64 %offset.idx, -1
 ; CHECK:       %[[I2:.+]] = add i64 %offset.idx, -2
 ; CHECK:       %[[I3:.+]] = add i64 %offset.idx, -3
-; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %offset.idx, i32 0
-; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I1]], i32 0
-; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I2]], i32 0
-; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I3]], i32 0
-; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %offset.idx, i32 1
-; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I1]], i32 1
-; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I2]], i32 1
-; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I3]], i32 1
+; CHECK:       getelementptr inbounds %pair, ptr %p, i64 %offset.idx, i32 0
+; CHECK:       getelementptr inbounds %pair, ptr %p, i64 %[[I1]], i32 0
+; CHECK:       getelementptr inbounds %pair, ptr %p, i64 %[[I2]], i32 0
+; CHECK:       getelementptr inbounds %pair, ptr %p, i64 %[[I3]], i32 0
+; CHECK:       getelementptr inbounds %pair, ptr %p, i64 %offset.idx, i32 1
+; CHECK:       getelementptr inbounds %pair, ptr %p, i64 %[[I1]], i32 1
+; CHECK:       getelementptr inbounds %pair, ptr %p, i64 %[[I2]], i32 1
+; CHECK:       getelementptr inbounds %pair, ptr %p, i64 %[[I3]], i32 1
 ; CHECK:       br i1 {{.*}}, label %middle.block, label %vector.body
 ;
-; INTER:     LV: Found uniform instruction: %tmp1 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 0
-; INTER:     LV: Found uniform instruction: %tmp2 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 1
+; INTER:     LV: Found uniform instruction: %tmp1 = getelementptr inbounds %pair, ptr %p, i64 %i, i32 0
+; INTER:     LV: Found uniform instruction: %tmp2 = getelementptr inbounds %pair, ptr %p, i64 %i, i32 1
 ; INTER:     vector.body
 ; INTER:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
 ; INTER:       %offset.idx = sub i64 %n, %index
 ; INTER-NOT:   getelementptr
-; INTER:       %[[G0:.+]] = getelementptr inbounds %pair, %pair* %p, i64 %offset.idx, i32 0
-; INTER:       getelementptr inbounds i32, i32* %[[G0]], i64 -6
+; INTER:       %[[G0:.+]] = getelementptr inbounds %pair, ptr %p, i64 %offset.idx, i32 0
+; INTER:       getelementptr inbounds i32, ptr %[[G0]], i64 -6
 ; INTER-NOT:   getelementptr
 ; INTER:       br i1 {{.*}}, label %middle.block, label %vector.body
 ;
-define i32 @interleaved_access_reverse(%pair* %p, i64 %n) {
+define i32 @interleaved_access_reverse(ptr %p, i64 %n) {
 entry:
   br label %for.body
 
 for.body:
   %i = phi i64 [ %i.next, %for.body ], [ %n, %entry ]
   %tmp0 = phi i32 [ %tmp6, %for.body ], [ 0, %entry ]
-  %tmp1 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 0
-  %tmp2 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 1
-  %tmp3 = load i32, i32* %tmp1, align 8
-  %tmp4 = load i32, i32* %tmp2, align 8
+  %tmp1 = getelementptr inbounds %pair, ptr %p, i64 %i, i32 0
+  %tmp2 = getelementptr inbounds %pair, ptr %p, i64 %i, i32 1
+  %tmp3 = load i32, ptr %tmp1, align 8
+  %tmp4 = load i32, ptr %tmp2, align 8
   %tmp5 = add i32 %tmp3, %tmp4
   %tmp6 = add i32 %tmp0, %tmp5
   %i.next = add nsw i64 %i, -1
@@ -197,18 +197,17 @@ for.end:
 ; as if it were uniform, the store is scalarized, making the pointer
 ; non-uniform.
 ;
-; INTER-NOT: LV: Found uniform instruction: %tmp0 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 0
+; INTER-NOT: LV: Found uniform instruction: %tmp0 = getelementptr inbounds %pair, ptr %p, i64 %i, i32 0
 ; INTER:     vector.body
 ; INTER:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, {{.*}} ]
-; INTER:       %[[G0:.+]] = getelementptr inbounds %pair, %pair* %p, i64 %index, i32 0
-; INTER:       %[[B0:.+]] = bitcast i32* %[[G0]] to <8 x i32>*
-; INTER:       %wide.vec = load <8 x i32>, <8 x i32>* %[[B0]], align 8
+; INTER:       %[[G0:.+]] = getelementptr inbounds %pair, ptr %p, i64 %index, i32 0
+; INTER:       %wide.vec = load <8 x i32>, ptr %[[G0]], align 8
 ; INTER:       %[[I1:.+]] = or i64 %index, 1
-; INTER:       getelementptr inbounds %pair, %pair* %p, i64 %[[I1]], i32 0
+; INTER:       getelementptr inbounds %pair, ptr %p, i64 %[[I1]], i32 0
 ; INTER:       %[[I2:.+]] = or i64 %index, 2
-; INTER:       getelementptr inbounds %pair, %pair* %p, i64 %[[I2]], i32 0
+; INTER:       getelementptr inbounds %pair, ptr %p, i64 %[[I2]], i32 0
 ; INTER:       %[[I3:.+]] = or i64 %index, 3
-; INTER:       getelementptr inbounds %pair, %pair* %p, i64 %[[I3]], i32 0
+; INTER:       getelementptr inbounds %pair, ptr %p, i64 %[[I3]], i32 0
 ; INTER:       br i1 {{.*}}, label %middle.block, label %vector.body
 ;
 define void @predicated_store(%pair *%p, i32 %x, i64 %n) {
@@ -217,13 +216,13 @@ entry:
 
 for.body:
   %i  = phi i64 [ %i.next, %if.merge ], [ 0, %entry ]
-  %tmp0 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 0
-  %tmp1 = load i32, i32* %tmp0, align 8
+  %tmp0 = getelementptr inbounds %pair, ptr %p, i64 %i, i32 0
+  %tmp1 = load i32, ptr %tmp0, align 8
   %tmp2 = icmp eq i32 %tmp1, %x
   br i1 %tmp2, label %if.then, label %if.merge
 
 if.then:
-  store i32 %tmp1, i32* %tmp0, align 8
+  store i32 %tmp1, ptr %tmp0, align 8
   br label %if.merge
 
 if.merge:
@@ -241,27 +240,27 @@ for.end:
 ; as uniform and is not uniform after vectorization. The store is scalarized
 ; because the stored type may required padding.
 ;
-; CHECK-NOT: LV: Found uniform instruction: %tmp1 = getelementptr inbounds x86_fp80, x86_fp80* %a, i64 %i
+; CHECK-NOT: LV: Found uniform instruction: %tmp1 = getelementptr inbounds x86_fp80, ptr %a, i64 %i
 ; CHECK:     vector.body
 ; CHECK:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
 ; CHECK:       %[[I1:.+]] = or i64 %index, 1
 ; CHECK:       %[[I2:.+]] = or i64 %index, 2
 ; CHECK:       %[[I3:.+]] = or i64 %index, 3
-; CHECK:       getelementptr inbounds x86_fp80, x86_fp80* %a, i64 %index
-; CHECK:       getelementptr inbounds x86_fp80, x86_fp80* %a, i64 %[[I1]]
-; CHECK:       getelementptr inbounds x86_fp80, x86_fp80* %a, i64 %[[I2]]
-; CHECK:       getelementptr inbounds x86_fp80, x86_fp80* %a, i64 %[[I3]]
+; CHECK:       getelementptr inbounds x86_fp80, ptr %a, i64 %index
+; CHECK:       getelementptr inbounds x86_fp80, ptr %a, i64 %[[I1]]
+; CHECK:       getelementptr inbounds x86_fp80, ptr %a, i64 %[[I2]]
+; CHECK:       getelementptr inbounds x86_fp80, ptr %a, i64 %[[I3]]
 ; CHECK:       br i1 {{.*}}, label %middle.block, label %vector.body
 ;
-define void @irregular_type(x86_fp80* %a, i64 %n) {
+define void @irregular_type(ptr %a, i64 %n) {
 entry:
   br label %for.body
 
 for.body:
   %i = phi i64 [ 0, %entry ], [ %i.next, %for.body ]
   %tmp0 = sitofp i32 1 to x86_fp80
-  %tmp1 = getelementptr inbounds x86_fp80, x86_fp80* %a, i64 %i
-  store x86_fp80 %tmp0, x86_fp80* %tmp1, align 16
+  %tmp1 = getelementptr inbounds x86_fp80, ptr %a, i64 %i
+  store x86_fp80 %tmp0, ptr %tmp1, align 16
   %i.next = add i64 %i, 1
   %cond = icmp slt i64 %i.next, %n
   br i1 %cond, label %for.body, label %for.end
@@ -275,23 +274,24 @@ for.end:
 ; Check that a pointer induction variable is recognized as uniform and remains
 ; uniform after vectorization.
 ;
-; CHECK:     LV: Found uniform instruction: %p = phi i32* [ %tmp03, %for.body ], [ %a, %entry ]
+; CHECK:     LV: Found uniform instruction: %p = phi ptr [ %tmp03, %for.body ], [ %a, %entry ]
 ; CHECK:     vector.body
 ; CHECK:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
 ; CHECK-NOT:   getelementptr
-; CHECK:       %next.gep = getelementptr i32, i32* %a, i64 %index
+; CHECK:       [[SHL:%.+]] = shl i64 %index, 2
+; CHECK:       %next.gep = getelementptr i8, ptr %a, i64 [[SHL]]
 ; CHECK-NOT:   getelementptr
 ; CHECK:       br i1 {{.*}}, label %middle.block, label %vector.body
 ;
-define void @pointer_iv_uniform(i32* %a, i32 %x, i64 %n) {
+define void @pointer_iv_uniform(ptr %a, i32 %x, i64 %n) {
 entry:
   br label %for.body
 
 for.body:
   %i = phi i64 [ %i.next, %for.body ], [ 0, %entry ]
-  %p = phi i32* [ %tmp03, %for.body ], [ %a, %entry ]
-  store i32 %x, i32* %p, align 8
-  %tmp03 = getelementptr inbounds i32, i32* %p, i32 1
+  %p = phi ptr [ %tmp03, %for.body ], [ %a, %entry ]
+  store i32 %x, ptr %p, align 8
+  %tmp03 = getelementptr inbounds i32, ptr %p, i32 1
   %i.next = add nuw nsw i64 %i, 1
   %cond = icmp slt i64 %i.next, %n
   br i1 %cond, label %for.body, label %for.end
@@ -307,42 +307,42 @@ for.end:
 ; induction variable is used by getelementptr instructions that are non-uniform
 ; due to scalarization of the stores.
 ;
-; INTER-NOT: LV: Found uniform instruction: %p = phi i32* [ %tmp03, %for.body ], [ %a, %entry ]
+; INTER-NOT: LV: Found uniform instruction: %p = phi ptr [ %tmp03, %for.body ], [ %a, %entry ]
 ; INTER:     vector.body
 ; INTER:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
-; INTER:       %[[I0:.+]] = shl i64 %index, 2
-; INTER:       %next.gep = getelementptr i32, i32* %a, i64 %[[I0]]
-; INTER:       %[[S1:.+]] = shl i64 %index, 2
-; INTER:       %[[I1:.+]] = or i64 %[[S1]], 4
-; INTER:       %next.gep2 = getelementptr i32, i32* %a, i64 %[[I1]]
-; INTER:       %[[S2:.+]] = shl i64 %index, 2
-; INTER:       %[[I2:.+]] = or i64 %[[S2]], 8
-; INTER:       %next.gep3 = getelementptr i32, i32* %a, i64 %[[I2]]
-; INTER:       %[[S3:.+]] = shl i64 %index, 2
-; INTER:       %[[I3:.+]] = or i64 %[[S3]], 12
-; INTER:       %next.gep4 = getelementptr i32, i32* %a, i64 %[[I3]]
+; INTER:       %[[I0:.+]] = shl i64 %index, 4
+; INTER:       %next.gep = getelementptr i8, ptr %a, i64 %[[I0]]
+; INTER:       %[[S1:.+]] = shl i64 %index, 4
+; INTER:       %[[I1:.+]] = or i64 %[[S1]], 16
+; INTER:       %next.gep2 = getelementptr i8, ptr %a, i64 %[[I1]]
+; INTER:       %[[S2:.+]] = shl i64 %index, 4
+; INTER:       %[[I2:.+]] = or i64 %[[S2]], 32
+; INTER:       %next.gep3 = getelementptr i8, ptr %a, i64 %[[I2]]
+; INTER:       %[[S3:.+]] = shl i64 %index, 4
+; INTER:       %[[I3:.+]] = or i64 %[[S3]], 48
+; INTER:       %next.gep4 = getelementptr i8, ptr %a, i64 %[[I3]]
 ; INTER:       br i1 {{.*}}, label %middle.block, label %vector.body
 ;
-define void @pointer_iv_non_uniform_0(i32* %a, i64 %n) {
+define void @pointer_iv_non_uniform_0(ptr %a, i64 %n) {
 entry:
   br label %for.body
 
 for.body:
   %i = phi i64 [ %i.next, %for.body ], [ 0, %entry ]
-  %p = phi i32* [ %tmp03, %for.body ], [ %a, %entry ]
-  %tmp00 = load i32, i32* %p, align 8
-  %tmp01 = getelementptr inbounds i32, i32* %p, i32 1
-  %tmp02 = load i32, i32* %tmp01, align 8
-  %tmp03 = getelementptr inbounds i32, i32* %p, i32 4
-  %tmp04 = load i32, i32* %tmp03, align 8
-  %tmp05 = getelementptr inbounds i32, i32* %p, i32 5
-  %tmp06 = load i32, i32* %tmp05, align 8
+  %p = phi ptr [ %tmp03, %for.body ], [ %a, %entry ]
+  %tmp00 = load i32, ptr %p, align 8
+  %tmp01 = getelementptr inbounds i32, ptr %p, i32 1
+  %tmp02 = load i32, ptr %tmp01, align 8
+  %tmp03 = getelementptr inbounds i32, ptr %p, i32 4
+  %tmp04 = load i32, ptr %tmp03, align 8
+  %tmp05 = getelementptr inbounds i32, ptr %p, i32 5
+  %tmp06 = load i32, ptr %tmp05, align 8
   %tmp07 = sub i32 %tmp04, %tmp00
   %tmp08 = sub i32 %tmp02, %tmp02
-  %tmp09 = getelementptr inbounds i32, i32* %p, i32 2
-  store i32 %tmp07, i32* %tmp09, align 8
-  %tmp10 = getelementptr inbounds i32, i32* %p, i32 3
-  store i32 %tmp08, i32* %tmp10, align 8
+  %tmp09 = getelementptr inbounds i32, ptr %p, i32 2
+  store i32 %tmp07, ptr %tmp09, align 8
+  %tmp10 = getelementptr inbounds i32, ptr %p, i32 3
+  store i32 %tmp08, ptr %tmp10, align 8
   %i.next = add nuw nsw i64 %i, 1
   %cond = icmp slt i64 %i.next, %n
   br i1 %cond, label %for.body, label %for.end
@@ -357,28 +357,32 @@ for.end:
 ; recognized as uniform and is not uniform after vectorization. The pointer
 ; induction variable is used by a store that will be scalarized.
 ;
-; CHECK-NOT: LV: Found uniform instruction: %p = phi x86_fp80* [%tmp1, %for.body], [%a, %entry]
+; CHECK-NOT: LV: Found uniform instruction: %p = phi ptr [%tmp1, %for.body], [%a, %entry]
 ; CHECK:     vector.body
 ; CHECK:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
-; CHECK:       %next.gep = getelementptr x86_fp80, x86_fp80* %a, i64 %index
-; CHECK:       %[[I1:.+]] = or i64 %index, 1
-; CHECK:       %next.gep2 = getelementptr x86_fp80, x86_fp80* %a, i64 %[[I1]]
-; CHECK:       %[[I2:.+]] = or i64 %index, 2
-; CHECK:       %next.gep3 = getelementptr x86_fp80, x86_fp80* %a, i64 %[[I2]]
-; CHECK:       %[[I3:.+]] = or i64 %index, 3
-; CHECK:       %next.gep4 = getelementptr x86_fp80, x86_fp80* %a, i64 %[[I3]]
+; CHECK:       [[SHL1:%.+]] = shl i64 %index, 4
+; CHECK:       %next.gep = getelementptr i8, ptr %a, i64 [[SHL1]]
+; CHECK:       [[SHL2:%.+]] = shl i64 %index, 4
+; CHECK:       %[[I1:.+]] = or i64 [[SHL2]], 16
+; CHECK:       %next.gep2 = getelementptr i8, ptr %a, i64 %[[I1]]
+; CHECK:       [[SHL3:%.+]] = shl i64 %index, 4
+; CHECK:       %[[I2:.+]] = or i64 [[SHL3]], 32
+; CHECK:       %next.gep3 = getelementptr i8, ptr %a, i64 %[[I2]]
+; CHECK:       [[SHL4:%.+]] = shl i64 %index, 4
+; CHECK:       %[[I3:.+]] = or i64 [[SHL4]], 48
+; CHECK:       %next.gep4 = getelementptr i8, ptr %a, i64 %[[I3]]
 ; CHECK:       br i1 {{.*}}, label %middle.block, label %vector.body
 ;
-define void @pointer_iv_non_uniform_1(x86_fp80* %a, i64 %n) {
+define void @pointer_iv_non_uniform_1(ptr %a, i64 %n) {
 entry:
   br label %for.body
 
 for.body:
   %i = phi i64 [ %i.next, %for.body ], [ 0, %entry ]
-  %p = phi x86_fp80* [%tmp1, %for.body], [%a, %entry]
+  %p = phi ptr [%tmp1, %for.body], [%a, %entry]
   %tmp0 = sitofp i32 1 to x86_fp80
-  store x86_fp80 %tmp0, x86_fp80* %p, align 16
-  %tmp1 = getelementptr inbounds x86_fp80, x86_fp80* %p, i32 1
+  store x86_fp80 %tmp0, ptr %p, align 16
+  %tmp1 = getelementptr inbounds x86_fp80, ptr %p, i32 1
   %i.next = add i64 %i, 1
   %cond = icmp slt i64 %i.next, %n
   br i1 %cond, label %for.body, label %for.end
@@ -394,32 +398,32 @@ for.end:
 ; variable is not recognized as uniform and is not uniform after vectorization
 ; because it is stored to memory.
 ;
-; CHECK-NOT: LV: Found uniform instruction: %p = phi i32* [ %tmp3, %for.body ], [ %a, %entry ]
-; CHECK:     LV: Found uniform instruction: %q = phi i32** [ %tmp4, %for.body ], [ %b, %entry ]
+; CHECK-NOT: LV: Found uniform instruction: %p = phi ptr [ %tmp3, %for.body ], [ %a, %entry ]
+; CHECK:     LV: Found uniform instruction: %q = phi ptr [ %tmp4, %for.body ], [ %b, %entry ]
 ; CHECK:     vector.body
-; CHECK:       %pointer.phi = phi i32* [ %a, %vector.ph ], [ %ptr.ind, %vector.body ]
+; CHECK:       %pointer.phi = phi ptr [ %a, %vector.ph ], [ %ptr.ind, %vector.body ]
 ; CHECK:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
-; CHECK:       %[[PTRVEC:.+]] = getelementptr i32, i32* %pointer.phi, <4 x i64> <i64 0, i64 1, i64 2, i64 3>
-; CHECK:       %next.gep = getelementptr i32*, i32** %b, i64 %index
-; CHECK:       %[[NEXTGEPBC:.+]] = bitcast i32** %next.gep to <4 x i32*>*
-; CHECK:       store <4 x i32*> %[[PTRVEC]], <4 x i32*>* %[[NEXTGEPBC]], align 8
-; CHECK:       %ptr.ind = getelementptr i32, i32* %pointer.phi, i64 4
+; CHECK:       %[[PTRVEC:.+]] = getelementptr i8, ptr %pointer.phi, <4 x i64> <i64 0, i64 4, i64 8, i64 12>
+; CHECK:       [[SHL:%.+]] = shl i64 %index, 3
+; CHECK:       %next.gep = getelementptr i8, ptr %b, i64 [[SHL]]
+; CHECK:       store <4 x ptr> %[[PTRVEC]], ptr %next.gep, align 8
+; CHECK:       %ptr.ind = getelementptr i8, ptr %pointer.phi, i64 16
 ; CHECK:       br i1 {{.*}}, label %middle.block, label %vector.body
 ;
-define i32 @pointer_iv_mixed(i32* %a, i32** %b, i64 %n) {
+define i32 @pointer_iv_mixed(ptr %a, ptr %b, i64 %n) {
 entry:
   br label %for.body
 
 for.body:
   %i = phi i64 [ %i.next, %for.body ], [ 0, %entry ]
-  %p = phi i32* [ %tmp3, %for.body ], [ %a, %entry ]
-  %q = phi i32** [ %tmp4, %for.body ], [ %b, %entry ]
+  %p = phi ptr [ %tmp3, %for.body ], [ %a, %entry ]
+  %q = phi ptr [ %tmp4, %for.body ], [ %b, %entry ]
   %tmp0 = phi i32 [ %tmp2, %for.body ], [ 0, %entry ]
-  %tmp1 = load i32, i32* %p, align 8
+  %tmp1 = load i32, ptr %p, align 8
   %tmp2 = add i32 %tmp1, %tmp0
-  store i32* %p, i32** %q, align 8
-  %tmp3 = getelementptr inbounds i32, i32* %p, i32 1
-  %tmp4 = getelementptr inbounds i32*, i32** %q, i32 1
+  store ptr %p, ptr %q, align 8
+  %tmp3 = getelementptr inbounds i32, ptr %p, i32 1
+  %tmp4 = getelementptr inbounds ptr, ptr %q, i32 1
   %i.next = add nuw nsw i64 %i, 1
   %cond = icmp slt i64 %i.next, %n
   br i1 %cond, label %for.body, label %for.end
@@ -429,49 +433,47 @@ for.end:
   ret i32 %tmp5
 }
 
-; INTER-LABEL: bitcast_pointer_operand
+; INTER-LABEL: pointer_operand_geps_with_different_indexed_types
 ;
 ; Check that a pointer operand having a user other than a memory access is
-; recognized as uniform after vectorization. In this test case, %tmp1 is a
-; bitcast that is used by a load and a getelementptr instruction (%tmp2). Once
-; %tmp2 is marked uniform, %tmp1 should be marked uniform as well.
+; recognized as uniform after vectorization. In this test case, %tmp0 is a
+; GEP that is used by a load and a getelementptr instruction (%tmp2). Once
+; %tmp2 is marked uniform, %tmp0 should be marked uniform as well.
 ;
 ; INTER:       LV: Found uniform instruction: %cond = icmp slt i64 %i.next, %n
-; INTER-NEXT:  LV: Found uniform instruction: %tmp2 = getelementptr inbounds i8, i8* %tmp1, i64 3
-; INTER-NEXT:  LV: Found uniform instruction: %tmp6 = getelementptr inbounds i8, i8* %B, i64 %i
-; INTER-NEXT:  LV: Found uniform instruction: %tmp1 = bitcast i64* %tmp0 to i8*
-; INTER-NEXT:  LV: Found uniform instruction: %tmp0 = getelementptr inbounds i64, i64* %A, i64 %i
+; INTER-NEXT:  LV: Found uniform instruction: %tmp2 = getelementptr inbounds i8, ptr %tmp0, i64 3
+; INTER-NEXT:  LV: Found uniform instruction: %tmp6 = getelementptr inbounds i8, ptr %B, i64 %i
+; INTER-NEXT:  LV: Found uniform instruction: %tmp0 = getelementptr inbounds i64, ptr %A, i64 %i
 ; INTER-NEXT:  LV: Found uniform instruction: %i = phi i64 [ 0, %entry ], [ %i.next, %for.body ]
 ; INTER-NEXT:  LV: Found uniform instruction: %i.next = add nuw nsw i64 %i, 1
-; INTER:       define void @bitcast_pointer_operand(
+; INTER:       define void @pointer_operand_geps_with_different_indexed_types(
 ; INTER:       vector.body:
 ; INTER-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, %vector.ph ], [ [[INDEX_NEXT:%.*]], %vector.body ]
-; INTER-NEXT:    [[TMP4:%.*]] = getelementptr inbounds i64, i64* %A, i64 [[INDEX]]
-; INTER-NEXT:    [[TMP5:%.*]] = bitcast i64* [[TMP4]] to <32 x i8>*
-; INTER-NEXT:    [[WIDE_VEC:%.*]] = load <32 x i8>, <32 x i8>* [[TMP5]], align 1
+; INTER-NEXT:    [[TMP4:%.*]] = getelementptr inbounds i64, ptr %A, i64 [[INDEX]]
+; INTER-NEXT:    [[TMP5:%.*]] = getelementptr inbounds i8, ptr [[TMP4]], i64 3
+; INTER-NEXT:    [[TMP6:%.*]] = getelementptr inbounds i8, ptr [[TMP5]], i64 -3
+; INTER-NEXT:    [[WIDE_VEC:%.*]] = load <32 x i8>, ptr [[TMP6]], align 1
 ; INTER-NEXT:    [[STRIDED_VEC:%.*]] = shufflevector <32 x i8> [[WIDE_VEC]], <32 x i8> poison, <4 x i32> <i32 0, i32 8, i32 16, i32 24>
 ; INTER-NEXT:    [[STRIDED_VEC5:%.*]] = shufflevector <32 x i8> [[WIDE_VEC]], <32 x i8> poison, <4 x i32> <i32 3, i32 11, i32 19, i32 27>
 ; INTER-NEXT:    [[TMP6:%.*]] = xor <4 x i8> [[STRIDED_VEC5]], [[STRIDED_VEC]]
-; INTER-NEXT:    [[TMP7:%.*]] = getelementptr inbounds i8, i8* %B, i64 [[INDEX]]
-; INTER-NEXT:    [[TMP8:%.*]] = bitcast i8* [[TMP7]] to <4 x i8>*
-; INTER-NEXT:    store <4 x i8> [[TMP6]], <4 x i8>* [[TMP8]], align 1
+; INTER-NEXT:    [[TMP7:%.*]] = getelementptr inbounds i8, ptr %B, i64 [[INDEX]]
+; INTER-NEXT:    store <4 x i8> [[TMP6]], ptr [[TMP7]], align 1
 ; INTER-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
 ; INTER:         br i1 {{.*}}, label %middle.block, label %vector.body
 ;
-define void @bitcast_pointer_operand(i64* %A, i8* %B, i64 %n) {
+define void @pointer_operand_geps_with_different_indexed_types(i64* %A, i8* %B, i64 %n) {
 entry:
   br label %for.body
 
 for.body:
   %i = phi i64 [ 0, %entry ], [ %i.next, %for.body ]
-  %tmp0 = getelementptr inbounds i64, i64* %A, i64 %i
-  %tmp1 = bitcast i64* %tmp0 to i8*
-  %tmp2 = getelementptr inbounds i8, i8* %tmp1, i64 3
-  %tmp3 = load i8, i8* %tmp2, align 1
-  %tmp4 = load i8, i8* %tmp1, align 1
+  %tmp0 = getelementptr inbounds i64, ptr %A, i64 %i
+  %tmp2 = getelementptr inbounds i8, ptr %tmp0, i64 3
+  %tmp3 = load i8, ptr %tmp2, align 1
+  %tmp4 = load i8, ptr %tmp0, align 1
   %tmp5 = xor i8 %tmp3, %tmp4
-  %tmp6 = getelementptr inbounds i8, i8* %B, i64 %i
-  store i8 %tmp5, i8* %tmp6
+  %tmp6 = getelementptr inbounds i8, ptr %B, i64 %i
+  store i8 %tmp5, ptr %tmp6
   %i.next = add nuw nsw i64 %i, 1
   %cond = icmp slt i64 %i.next, %n
   br i1 %cond, label %for.body, label %for.end
@@ -479,3 +481,31 @@ for.body:
 for.end:
   ret void
 }
+
+; CHECK-LABEL: pr61396_pointer_used_as_both_stored_value_and_pointer_operand_by_store
+; CHECK-NOT: LV: Found uniform instruction: %cur.ptr = getelementptr inbounds ptr, ptr %ary, i64 %iv
+
+; CHECK:       define void @pr61396_pointer_used_as_both_stored_value_and_pointer_operand_by_store(
+; CHECK:       vector.body:
+; CHECK-NEXT:    %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
+; CHECK-NEXT:    [[VEC_IND:%.+]] = phi <4 x i64> [ <i64 0, i64 1, i64 2, i64 3>, %vector.ph ], [ %vec.ind.next, %vector.body ]
+; CHECK-NEXT:    [[GEP:%.+]] = getelementptr inbounds ptr, ptr %ary, <4 x i64> [[VEC_IND]]
+; CHECK-NEXT:    [[EXT:%.+]] = extractelement <4 x ptr> [[GEP]], i64 0
+; CHECK-NEXT:    store <4 x ptr> [[GEP]], ptr [[EXT]], align 8
+;
+
+define void @pr61396_pointer_used_as_both_stored_value_and_pointer_operand_by_store(ptr %ary) {
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
+  %cur.ptr = getelementptr inbounds ptr, ptr %ary, i64 %iv
+  store ptr %cur.ptr, ptr %cur.ptr, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %done = icmp eq i64 %iv, 10240
+  br i1 %done, label %exit, label %loop
+
+exit:
+  ret void
+}
diff --git a/llvm/test/Transforms/LoopVectorize/first-order-recurrence-sink-replicate-region.ll b/llvm/test/Transforms/LoopVectorize/first-order-recurrence-sink-replicate-region.ll
index 7de4ef1fc0e3b..37480019e9ddb 100644
--- a/llvm/test/Transforms/LoopVectorize/first-order-recurrence-sink-replicate-region.ll
+++ b/llvm/test/Transforms/LoopVectorize/first-order-recurrence-sink-replicate-region.ll
@@ -10,7 +10,6 @@ define void @sink_replicate_region_1(i32 %x, ptr %ptr, ptr noalias %dst) optsize
 ; CHECK-LABEL: sink_replicate_region_1
 ; CHECK:      VPlan 'Initial VPlan for VF={2},UF>=1' {
 ; CHECK-NEXT: Live-in vp<[[VEC_TC:%.+]]> = vector-trip-count
-; CHECK-EMPTY:
 ; CHECK-NEXT: Live-in vp<[[BTC:%.+]]> = backedge-taken count
 ; CHECK-EMPTY:
 ; CHECK-NEXT: vector.ph:
@@ -39,9 +38,9 @@ define void @sink_replicate_region_1(i32 %x, ptr %ptr, ptr noalias %dst) optsize
 ; CHECK-NEXT:     PHI-PREDICATED-INSTRUCTION vp<[[PRED1:%.+]]> = ir<%lv>
 ; CHECK-NEXT:   No successors
 ; CHECK-NEXT: }
-; CHECK-NEXT: Successor(s): loop.1
+; CHECK-NEXT: Successor(s): loop.0
 ; CHECK-EMPTY:
-; CHECK-NEXT: loop.1:
+; CHECK-NEXT: loop.0:
 ; CHECK-NEXT:   WIDEN ir<%conv> = sext vp<[[PRED1]]>
 ; CHECK-NEXT:   EMIT vp<[[SPLICE:%.+]]> = first-order splice ir<%0> ir<%conv>
 ; CHECK-NEXT: Successor(s): pred.store
@@ -100,7 +99,6 @@ define void @sink_replicate_region_2(i32 %x, i8 %y, ptr %ptr) optsize {
 ; CHECK-LABEL: sink_replicate_region_2
 ; CHECK:      VPlan 'Initial VPlan for VF={2},UF>=1' {
 ; CHECK-NEXT: Live-in vp<[[VEC_TC:%.+]]> = vector-trip-count
-; CHECK-EMPTY:
 ; CHECK-NEXT: Live-in vp<[[BTC:%.+]]> = backedge-taken count
 ; CHECK-EMPTY:
 ; CHECK-NEXT: vector.ph:
@@ -169,7 +167,6 @@ define i32 @sink_replicate_region_3_reduction(i32 %x, i8 %y, ptr %ptr) optsize {
 ; CHECK-LABEL: sink_replicate_region_3_reduction
 ; CHECK:      VPlan 'Initial VPlan for VF={2},UF>=1' {
 ; CHECK-NEXT: Live-in vp<[[VEC_TC:%.+]]> = vector-trip-count
-; CHECK-EMPTY:
 ; CHECK-NEXT: Live-in vp<[[BTC:%.+]]> = backedge-taken count
 ; CHECK-EMPTY:
 ; CHECK-NEXT: vector.ph:
@@ -199,9 +196,9 @@ define i32 @sink_replicate_region_3_reduction(i32 %x, i8 %y, ptr %ptr) optsize {
 ; CHECK-NEXT:     PHI-PREDICATED-INSTRUCTION vp<[[PRED:%.+]]> = ir<%rem>
 ; CHECK-NEXT:   No successors
 ; CHECK-NEXT: }
-; CHECK-NEXT: Successor(s): loop.0.split
+; CHECK-NEXT: Successor(s): loop.0
 ; CHECK-EMPTY:
-; CHECK-NEXT: loop.0.split:
+; CHECK-NEXT: loop.0:
 ; CHECK-NEXT:   WIDEN ir<%add> = add vp<[[PRED]]>, ir<%recur.next>
 ; CHECK-NEXT:   WIDEN ir<%and.red.next> = and ir<%and.red>, ir<%add>
 ; CHECK-NEXT:   EMIT vp<[[SEL:%.+]]> = select vp<[[MASK]]> ir<%and.red.next> ir<%and.red>
@@ -243,7 +240,6 @@ define void @sink_replicate_region_4_requires_split_at_end_of_block(i32 %x, ptr
 ; CHECK-LABEL: sink_replicate_region_4_requires_split_at_end_of_block
 ; CHECK:      VPlan 'Initial VPlan for VF={2},UF>=1' {
 ; CHECK-NEXT: Live-in vp<[[VEC_TC:%.+]]> = vector-trip-count
-; CHECK-EMPTY:
 ; CHECK-NEXT: Live-in vp<[[BTC:%.+]]> = backedge-taken count
 ; CHECK-EMPTY:
 ; CHECK-NEXT: vector.ph:
@@ -272,9 +268,9 @@ define void @sink_replicate_region_4_requires_split_at_end_of_block(i32 %x, ptr
 ; CHECK-NEXT:     PHI-PREDICATED-INSTRUCTION vp<[[PRED:%.+]]> = ir<%lv>
 ; CHECK-NEXT:   No successors
 ; CHECK-NEXT: }
-; CHECK-NEXT: Successor(s): loop.1
+; CHECK-NEXT: Successor(s): loop.0
 ; CHECK-EMPTY:
-; CHECK-NEXT: loop.1:
+; CHECK-NEXT: loop.0:
 ; CHECK-NEXT:   WIDEN ir<%conv> = sext vp<[[PRED]]>
 ; CHECK-NEXT:   EMIT vp<[[SPLICE:%.+]]> = first-order splice ir<%0> ir<%conv>
 ; CHECK-NEXT: Successor(s): pred.store
@@ -341,7 +337,6 @@ define void @sink_replicate_region_after_replicate_region(ptr %ptr, ptr noalias
 ; CHECK-LABEL: sink_replicate_region_after_replicate_region
 ; CHECK:      VPlan 'Initial VPlan for VF={2},UF>=1' {
 ; CHECK-NEXT: Live-in vp<[[VEC_TC:%.+]]> = vector-trip-count
-; CHECK-EMPTY:
 ; CHECK-NEXT: Live-in vp<[[BTC:%.+]]> = backedge-taken count
 ; CHECK-EMPTY:
 ; CHECK-NEXT: vector.ph:
@@ -363,8 +358,8 @@ define void @sink_replicate_region_after_replicate_region(ptr %ptr, ptr noalias
 ; CHECK-NEXT:   Successor(s): pred.store.if, pred.store.continue
 ; CHECK-EMPTY:
 ; CHECK-NEXT:   pred.store.if:
-; CHECK-NEXT:     REPLICATE ir<%rem> = srem vp<[[SPLICE]]>, ir<%x>
 ; CHECK-NEXT:     vp<[[STEPS:%.+]]> = SCALAR-STEPS vp<[[CAN_IV]]>, ir<1>
+; CHECK-NEXT:     REPLICATE ir<%rem> = srem vp<[[SPLICE]]>, ir<%x>
 ; CHECK-NEXT:     REPLICATE ir<%rem.div> = sdiv ir<20>, ir<%rem>
 ; CHECK-NEXT:     REPLICATE ir<%gep> = getelementptr ir<%ptr>, vp<[[STEPS]]>
 ; CHECK-NEXT:     REPLICATE store ir<%rem.div>, ir<%gep>
@@ -415,7 +410,6 @@ define void @need_new_block_after_sinking_pr56146(i32 %x, ptr %src, ptr noalias
 ; CHECK-LABEL: need_new_block_after_sinking_pr56146
 ; CHECK:      VPlan 'Initial VPlan for VF={2},UF>=1' {
 ; CHECK-NEXT: Live-in vp<[[VEC_TC:%.+]]> = vector-trip-count
-; CHECK-EMPTY:
 ; CHECK-NEXT: Live-in vp<[[BTC:%.+]]> = backedge-taken count
 ; CHECK-EMPTY:
 ; CHECK-NEXT: vector.ph:
diff --git a/llvm/test/Transforms/LoopVectorize/first-order-recurrence.ll b/llvm/test/Transforms/LoopVectorize/first-order-recurrence.ll
index 4af3beae88e89..8aa82478813d0 100644
--- a/llvm/test/Transforms/LoopVectorize/first-order-recurrence.ll
+++ b/llvm/test/Transforms/LoopVectorize/first-order-recurrence.ll
@@ -912,13 +912,15 @@ define i32 @PR27246() {
 ; UNROLL-NO-IC-NEXT:  entry:
 ; UNROLL-NO-IC-NEXT:    br label [[FOR_COND1_PREHEADER:%.*]]
 ; UNROLL-NO-IC:       for.cond1.preheader:
-; UNROLL-NO-IC-NEXT:    [[I_016:%.*]] = phi i32 [ 1, [[ENTRY:%.*]] ], [ [[INC:%.*]], [[FOR_COND_CLEANUP3:%.*]] ]
+; UNROLL-NO-IC-NEXT:    [[INDVAR:%.*]] = phi i32 [ [[INDVAR_NEXT:%.*]], [[FOR_COND_CLEANUP3:%.*]] ], [ 0, [[ENTRY:%.*]] ]
+; UNROLL-NO-IC-NEXT:    [[I_016:%.*]] = phi i32 [ 1, [[ENTRY]] ], [ [[INC:%.*]], [[FOR_COND_CLEANUP3]] ]
 ; UNROLL-NO-IC-NEXT:    [[E_015:%.*]] = phi i32 [ poison, [[ENTRY]] ], [ [[E_1_LCSSA:%.*]], [[FOR_COND_CLEANUP3]] ]
-; UNROLL-NO-IC-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i32 [[I_016]], 8
+; UNROLL-NO-IC-NEXT:    [[TMP0:%.*]] = add i32 [[INDVAR]], 1
+; UNROLL-NO-IC-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i32 [[TMP0]], 8
 ; UNROLL-NO-IC-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
 ; UNROLL-NO-IC:       vector.ph:
-; UNROLL-NO-IC-NEXT:    [[N_MOD_VF:%.*]] = urem i32 [[I_016]], 8
-; UNROLL-NO-IC-NEXT:    [[N_VEC:%.*]] = sub i32 [[I_016]], [[N_MOD_VF]]
+; UNROLL-NO-IC-NEXT:    [[N_MOD_VF:%.*]] = urem i32 [[TMP0]], 8
+; UNROLL-NO-IC-NEXT:    [[N_VEC:%.*]] = sub i32 [[TMP0]], [[N_MOD_VF]]
 ; UNROLL-NO-IC-NEXT:    [[IND_END:%.*]] = sub i32 [[I_016]], [[N_VEC]]
 ; UNROLL-NO-IC-NEXT:    [[VECTOR_RECUR_INIT:%.*]] = insertelement <4 x i32> poison, i32 [[E_015]], i32 3
 ; UNROLL-NO-IC-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <4 x i32> poison, i32 [[I_016]], i64 0
@@ -930,14 +932,14 @@ define i32 @PR27246() {
 ; UNROLL-NO-IC-NEXT:    [[VECTOR_RECUR:%.*]] = phi <4 x i32> [ [[VECTOR_RECUR_INIT]], [[VECTOR_PH]] ], [ [[STEP_ADD:%.*]], [[VECTOR_BODY]] ]
 ; UNROLL-NO-IC-NEXT:    [[VEC_IND:%.*]] = phi <4 x i32> [ [[INDUCTION]], [[VECTOR_PH]] ], [ [[VEC_IND_NEXT:%.*]], [[VECTOR_BODY]] ]
 ; UNROLL-NO-IC-NEXT:    [[STEP_ADD]] = add <4 x i32> [[VEC_IND]], <i32 -4, i32 -4, i32 -4, i32 -4>
-; UNROLL-NO-IC-NEXT:    [[TMP0:%.*]] = shufflevector <4 x i32> [[VECTOR_RECUR]], <4 x i32> [[VEC_IND]], <4 x i32> <i32 3, i32 4, i32 5, i32 6>
-; UNROLL-NO-IC-NEXT:    [[TMP1:%.*]] = shufflevector <4 x i32> [[VEC_IND]], <4 x i32> [[STEP_ADD]], <4 x i32> <i32 3, i32 4, i32 5, i32 6>
+; UNROLL-NO-IC-NEXT:    [[TMP1:%.*]] = shufflevector <4 x i32> [[VECTOR_RECUR]], <4 x i32> [[VEC_IND]], <4 x i32> <i32 3, i32 4, i32 5, i32 6>
+; UNROLL-NO-IC-NEXT:    [[TMP2:%.*]] = shufflevector <4 x i32> [[VEC_IND]], <4 x i32> [[STEP_ADD]], <4 x i32> <i32 3, i32 4, i32 5, i32 6>
 ; UNROLL-NO-IC-NEXT:    [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 8
 ; UNROLL-NO-IC-NEXT:    [[VEC_IND_NEXT]] = add <4 x i32> [[STEP_ADD]], <i32 -4, i32 -4, i32 -4, i32 -4>
-; UNROLL-NO-IC-NEXT:    [[TMP2:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
-; UNROLL-NO-IC-NEXT:    br i1 [[TMP2]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
+; UNROLL-NO-IC-NEXT:    [[TMP3:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
+; UNROLL-NO-IC-NEXT:    br i1 [[TMP3]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
 ; UNROLL-NO-IC:       middle.block:
-; UNROLL-NO-IC-NEXT:    [[CMP_N:%.*]] = icmp eq i32 [[I_016]], [[N_VEC]]
+; UNROLL-NO-IC-NEXT:    [[CMP_N:%.*]] = icmp eq i32 [[TMP0]], [[N_VEC]]
 ; UNROLL-NO-IC-NEXT:    [[VECTOR_RECUR_EXTRACT:%.*]] = extractelement <4 x i32> [[STEP_ADD]], i32 3
 ; UNROLL-NO-IC-NEXT:    [[VECTOR_RECUR_EXTRACT_FOR_PHI:%.*]] = extractelement <4 x i32> [[STEP_ADD]], i32 2
 ; UNROLL-NO-IC-NEXT:    br i1 [[CMP_N]], label [[FOR_COND_CLEANUP3]], label [[SCALAR_PH]]
@@ -958,35 +960,38 @@ define i32 @PR27246() {
 ; UNROLL-NO-IC-NEXT:    [[E_1_LCSSA]] = phi i32 [ [[SCALAR_RECUR]], [[FOR_COND1]] ], [ [[VECTOR_RECUR_EXTRACT_FOR_PHI]], [[MIDDLE_BLOCK]] ]
 ; UNROLL-NO-IC-NEXT:    [[INC]] = add nuw nsw i32 [[I_016]], 1
 ; UNROLL-NO-IC-NEXT:    [[EXITCOND:%.*]] = icmp eq i32 [[INC]], 49
+; UNROLL-NO-IC-NEXT:    [[INDVAR_NEXT]] = add i32 [[INDVAR]], 1
 ; UNROLL-NO-IC-NEXT:    br i1 [[EXITCOND]], label [[FOR_COND_CLEANUP:%.*]], label [[FOR_COND1_PREHEADER]]
 ;
 ; UNROLL-NO-VF-LABEL: @PR27246(
 ; UNROLL-NO-VF-NEXT:  entry:
 ; UNROLL-NO-VF-NEXT:    br label [[FOR_COND1_PREHEADER:%.*]]
 ; UNROLL-NO-VF:       for.cond1.preheader:
-; UNROLL-NO-VF-NEXT:    [[I_016:%.*]] = phi i32 [ 1, [[ENTRY:%.*]] ], [ [[INC:%.*]], [[FOR_COND_CLEANUP3:%.*]] ]
+; UNROLL-NO-VF-NEXT:    [[INDVAR:%.*]] = phi i32 [ [[INDVAR_NEXT:%.*]], [[FOR_COND_CLEANUP3:%.*]] ], [ 0, [[ENTRY:%.*]] ]
+; UNROLL-NO-VF-NEXT:    [[I_016:%.*]] = phi i32 [ 1, [[ENTRY]] ], [ [[INC:%.*]], [[FOR_COND_CLEANUP3]] ]
 ; UNROLL-NO-VF-NEXT:    [[E_015:%.*]] = phi i32 [ poison, [[ENTRY]] ], [ [[E_1_LCSSA:%.*]], [[FOR_COND_CLEANUP3]] ]
-; UNROLL-NO-VF-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i32 [[I_016]], 2
+; UNROLL-NO-VF-NEXT:    [[TMP0:%.*]] = add i32 [[INDVAR]], 1
+; UNROLL-NO-VF-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i32 [[TMP0]], 2
 ; UNROLL-NO-VF-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
 ; UNROLL-NO-VF:       vector.ph:
-; UNROLL-NO-VF-NEXT:    [[N_MOD_VF:%.*]] = urem i32 [[I_016]], 2
-; UNROLL-NO-VF-NEXT:    [[N_VEC:%.*]] = sub i32 [[I_016]], [[N_MOD_VF]]
+; UNROLL-NO-VF-NEXT:    [[N_MOD_VF:%.*]] = urem i32 [[TMP0]], 2
+; UNROLL-NO-VF-NEXT:    [[N_VEC:%.*]] = sub i32 [[TMP0]], [[N_MOD_VF]]
 ; UNROLL-NO-VF-NEXT:    [[IND_END:%.*]] = sub i32 [[I_016]], [[N_VEC]]
 ; UNROLL-NO-VF-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; UNROLL-NO-VF:       vector.body:
 ; UNROLL-NO-VF-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
 ; UNROLL-NO-VF-NEXT:    [[VECTOR_RECUR:%.*]] = phi i32 [ [[E_015]], [[VECTOR_PH]] ], [ [[TMP1:%.*]], [[VECTOR_BODY]] ]
 ; UNROLL-NO-VF-NEXT:    [[OFFSET_IDX:%.*]] = sub i32 [[I_016]], [[INDEX]]
-; UNROLL-NO-VF-NEXT:    [[TMP0:%.*]] = add i32 [[OFFSET_IDX]], 0
-; UNROLL-NO-VF-NEXT:    [[TMP1]] = add i32 [[OFFSET_IDX]], -1
+; UNROLL-NO-VF-NEXT:    [[TMP2:%.*]] = add i32 [[OFFSET_IDX]], 0
+; UNROLL-NO-VF-NEXT:    [[TMP3:%.*]] = add i32 [[OFFSET_IDX]], -1
 ; UNROLL-NO-VF-NEXT:    [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 2
-; UNROLL-NO-VF-NEXT:    [[TMP2:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
-; UNROLL-NO-VF-NEXT:    br i1 [[TMP2]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP7:![0-9]+]]
+; UNROLL-NO-VF-NEXT:    [[TMP1:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
+; UNROLL-NO-VF-NEXT:    br i1 [[TMP1]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP7:![0-9]+]]
 ; UNROLL-NO-VF:       middle.block:
-; UNROLL-NO-VF-NEXT:    [[CMP_N:%.*]] = icmp eq i32 [[I_016]], [[N_VEC]]
+; UNROLL-NO-VF-NEXT:    [[CMP_N:%.*]] = icmp eq i32 [[TMP0]], [[N_VEC]]
 ; UNROLL-NO-VF-NEXT:    br i1 [[CMP_N]], label [[FOR_COND_CLEANUP3]], label [[SCALAR_PH]]
 ; UNROLL-NO-VF:       scalar.ph:
-; UNROLL-NO-VF-NEXT:    [[SCALAR_RECUR_INIT:%.*]] = phi i32 [ [[E_015]], [[FOR_COND1_PREHEADER]] ], [ [[TMP1]], [[MIDDLE_BLOCK]] ]
+; UNROLL-NO-VF-NEXT:    [[SCALAR_RECUR_INIT:%.*]] = phi i32 [ [[E_015]], [[FOR_COND1_PREHEADER]] ], [ [[TMP3]], [[MIDDLE_BLOCK]] ]
 ; UNROLL-NO-VF-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i32 [ [[IND_END]], [[MIDDLE_BLOCK]] ], [ [[I_016]], [[FOR_COND1_PREHEADER]] ]
 ; UNROLL-NO-VF-NEXT:    br label [[FOR_COND1:%.*]]
 ; UNROLL-NO-VF:       for.cond.cleanup:
@@ -999,22 +1004,25 @@ define i32 @PR27246() {
 ; UNROLL-NO-VF-NEXT:    [[DEC]] = add nsw i32 [[K_0]], -1
 ; UNROLL-NO-VF-NEXT:    br i1 [[CMP2]], label [[FOR_COND1]], label [[FOR_COND_CLEANUP3]], !llvm.loop [[LOOP8:![0-9]+]]
 ; UNROLL-NO-VF:       for.cond.cleanup3:
-; UNROLL-NO-VF-NEXT:    [[E_1_LCSSA]] = phi i32 [ [[SCALAR_RECUR]], [[FOR_COND1]] ], [ [[TMP0]], [[MIDDLE_BLOCK]] ]
+; UNROLL-NO-VF-NEXT:    [[E_1_LCSSA]] = phi i32 [ [[SCALAR_RECUR]], [[FOR_COND1]] ], [ [[TMP2]], [[MIDDLE_BLOCK]] ]
 ; UNROLL-NO-VF-NEXT:    [[INC]] = add nuw nsw i32 [[I_016]], 1
 ; UNROLL-NO-VF-NEXT:    [[EXITCOND:%.*]] = icmp eq i32 [[INC]], 49
+; UNROLL-NO-VF-NEXT:    [[INDVAR_NEXT]] = add i32 [[INDVAR]], 1
 ; UNROLL-NO-VF-NEXT:    br i1 [[EXITCOND]], label [[FOR_COND_CLEANUP:%.*]], label [[FOR_COND1_PREHEADER]]
 ;
 ; SINK-AFTER-LABEL: @PR27246(
 ; SINK-AFTER-NEXT:  entry:
 ; SINK-AFTER-NEXT:    br label [[FOR_COND1_PREHEADER:%.*]]
 ; SINK-AFTER:       for.cond1.preheader:
-; SINK-AFTER-NEXT:    [[I_016:%.*]] = phi i32 [ 1, [[ENTRY:%.*]] ], [ [[INC:%.*]], [[FOR_COND_CLEANUP3:%.*]] ]
+; SINK-AFTER-NEXT:    [[INDVAR:%.*]] = phi i32 [ [[INDVAR_NEXT:%.*]], [[FOR_COND_CLEANUP3:%.*]] ], [ 0, [[ENTRY:%.*]] ]
+; SINK-AFTER-NEXT:    [[I_016:%.*]] = phi i32 [ 1, [[ENTRY]] ], [ [[INC:%.*]], [[FOR_COND_CLEANUP3]] ]
 ; SINK-AFTER-NEXT:    [[E_015:%.*]] = phi i32 [ poison, [[ENTRY]] ], [ [[E_1_LCSSA:%.*]], [[FOR_COND_CLEANUP3]] ]
-; SINK-AFTER-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i32 [[I_016]], 4
+; SINK-AFTER-NEXT:    [[TMP0:%.*]] = add i32 [[INDVAR]], 1
+; SINK-AFTER-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i32 [[TMP0]], 4
 ; SINK-AFTER-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
 ; SINK-AFTER:       vector.ph:
-; SINK-AFTER-NEXT:    [[N_MOD_VF:%.*]] = urem i32 [[I_016]], 4
-; SINK-AFTER-NEXT:    [[N_VEC:%.*]] = sub i32 [[I_016]], [[N_MOD_VF]]
+; SINK-AFTER-NEXT:    [[N_MOD_VF:%.*]] = urem i32 [[TMP0]], 4
+; SINK-AFTER-NEXT:    [[N_VEC:%.*]] = sub i32 [[TMP0]], [[N_MOD_VF]]
 ; SINK-AFTER-NEXT:    [[IND_END:%.*]] = sub i32 [[I_016]], [[N_VEC]]
 ; SINK-AFTER-NEXT:    [[VECTOR_RECUR_INIT:%.*]] = insertelement <4 x i32> poison, i32 [[E_015]], i32 3
 ; SINK-AFTER-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <4 x i32> poison, i32 [[I_016]], i64 0
@@ -1025,13 +1033,13 @@ define i32 @PR27246() {
 ; SINK-AFTER-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
 ; SINK-AFTER-NEXT:    [[VECTOR_RECUR:%.*]] = phi <4 x i32> [ [[VECTOR_RECUR_INIT]], [[VECTOR_PH]] ], [ [[VEC_IND:%.*]], [[VECTOR_BODY]] ]
 ; SINK-AFTER-NEXT:    [[VEC_IND]] = phi <4 x i32> [ [[INDUCTION]], [[VECTOR_PH]] ], [ [[VEC_IND_NEXT:%.*]], [[VECTOR_BODY]] ]
-; SINK-AFTER-NEXT:    [[TMP0:%.*]] = shufflevector <4 x i32> [[VECTOR_RECUR]], <4 x i32> [[VEC_IND]], <4 x i32> <i32 3, i32 4, i32 5, i32 6>
+; SINK-AFTER-NEXT:    [[TMP1:%.*]] = shufflevector <4 x i32> [[VECTOR_RECUR]], <4 x i32> [[VEC_IND]], <4 x i32> <i32 3, i32 4, i32 5, i32 6>
 ; SINK-AFTER-NEXT:    [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 4
 ; SINK-AFTER-NEXT:    [[VEC_IND_NEXT]] = add <4 x i32> [[VEC_IND]], <i32 -4, i32 -4, i32 -4, i32 -4>
-; SINK-AFTER-NEXT:    [[TMP1:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
-; SINK-AFTER-NEXT:    br i1 [[TMP1]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
+; SINK-AFTER-NEXT:    [[TMP2:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
+; SINK-AFTER-NEXT:    br i1 [[TMP2]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
 ; SINK-AFTER:       middle.block:
-; SINK-AFTER-NEXT:    [[CMP_N:%.*]] = icmp eq i32 [[I_016]], [[N_VEC]]
+; SINK-AFTER-NEXT:    [[CMP_N:%.*]] = icmp eq i32 [[TMP0]], [[N_VEC]]
 ; SINK-AFTER-NEXT:    [[VECTOR_RECUR_EXTRACT:%.*]] = extractelement <4 x i32> [[VEC_IND]], i32 3
 ; SINK-AFTER-NEXT:    [[VECTOR_RECUR_EXTRACT_FOR_PHI:%.*]] = extractelement <4 x i32> [[VEC_IND]], i32 2
 ; SINK-AFTER-NEXT:    br i1 [[CMP_N]], label [[FOR_COND_CLEANUP3]], label [[SCALAR_PH]]
@@ -1052,6 +1060,7 @@ define i32 @PR27246() {
 ; SINK-AFTER-NEXT:    [[E_1_LCSSA]] = phi i32 [ [[SCALAR_RECUR]], [[FOR_COND1]] ], [ [[VECTOR_RECUR_EXTRACT_FOR_PHI]], [[MIDDLE_BLOCK]] ]
 ; SINK-AFTER-NEXT:    [[INC]] = add nuw nsw i32 [[I_016]], 1
 ; SINK-AFTER-NEXT:    [[EXITCOND:%.*]] = icmp eq i32 [[INC]], 49
+; SINK-AFTER-NEXT:    [[INDVAR_NEXT]] = add i32 [[INDVAR]], 1
 ; SINK-AFTER-NEXT:    br i1 [[EXITCOND]], label [[FOR_COND_CLEANUP:%.*]], label [[FOR_COND1_PREHEADER]]
 ;
 entry:
diff --git a/llvm/test/Transforms/LoopVectorize/icmp-uniforms.ll b/llvm/test/Transforms/LoopVectorize/icmp-uniforms.ll
index 7b3f03e339f2e..217810c61eb17 100644
--- a/llvm/test/Transforms/LoopVectorize/icmp-uniforms.ll
+++ b/llvm/test/Transforms/LoopVectorize/icmp-uniforms.ll
@@ -38,7 +38,6 @@ for.end:
 ; CHECK-LABEL: 'test'
 ; CHECK:      VPlan 'Initial VPlan for VF={4},UF>=1' {
 ; CHECK-NEXT: Live-in vp<[[VEC_TC:%.+]]> = vector-trip-count
-; CHECK-EMPTY:
 ; CHECK-NEXT: Live-in vp<[[BTC:%.+]]> = backedge-taken count
 ; CHECK-EMPTY:
 ; CHECK-NEXT: vector.ph:
diff --git a/llvm/test/Transforms/LoopVectorize/opaque-ptr.ll b/llvm/test/Transforms/LoopVectorize/opaque-ptr.ll
index 5ec6535313792..bb58173f63e15 100644
--- a/llvm/test/Transforms/LoopVectorize/opaque-ptr.ll
+++ b/llvm/test/Transforms/LoopVectorize/opaque-ptr.ll
@@ -57,20 +57,16 @@ define void @store_pointer_induction(ptr %start, ptr %end) {
 ; CHECK-NEXT:    [[IND_END:%.*]] = getelementptr i8, ptr [[START]], i64 [[TMP4]]
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
+; CHECK-NEXT:    [[POINTER_PHI:%.*]] = phi ptr [ [[START]], [[VECTOR_PH]] ], [ [[PTR_IND:%.*]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[TMP5:%.*]] = add i64 [[INDEX]], 0
-; CHECK-NEXT:    [[TMP6:%.*]] = mul i64 [[TMP5]], 8
-; CHECK-NEXT:    [[NEXT_GEP:%.*]] = getelementptr i8, ptr [[START]], i64 [[TMP6]]
-; CHECK-NEXT:    [[TMP7:%.*]] = add i64 [[INDEX]], 1
-; CHECK-NEXT:    [[TMP8:%.*]] = mul i64 [[TMP7]], 8
-; CHECK-NEXT:    [[NEXT_GEP3:%.*]] = getelementptr i8, ptr [[START]], i64 [[TMP8]]
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <2 x ptr> poison, ptr [[NEXT_GEP]], i32 0
-; CHECK-NEXT:    [[TMP10:%.*]] = insertelement <2 x ptr> [[TMP9]], ptr [[NEXT_GEP3]], i32 1
-; CHECK-NEXT:    [[TMP11:%.*]] = getelementptr ptr, ptr [[NEXT_GEP]], i32 0
-; CHECK-NEXT:    store <2 x ptr> [[TMP10]], ptr [[TMP11]], align 4
+; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr i8, ptr [[POINTER_PHI]], <2 x i64> <i64 0, i64 8>
+; CHECK-NEXT:    [[TMP6:%.*]] = extractelement <2 x ptr> [[TMP5]], i32 0
+; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr ptr, ptr [[TMP6]], i32 0
+; CHECK-NEXT:    store <2 x ptr> [[TMP5]], ptr [[TMP7]], align 4
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 2
-; CHECK-NEXT:    [[TMP12:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
-; CHECK-NEXT:    br i1 [[TMP12]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; CHECK-NEXT:    [[PTR_IND]] = getelementptr i8, ptr [[POINTER_PHI]], i64 16
+; CHECK-NEXT:    [[TMP8:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; CHECK-NEXT:    br i1 [[TMP8]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
 ; CHECK:       middle.block:
 ; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 [[TMP3]], [[N_VEC]]
 ; CHECK-NEXT:    br i1 [[CMP_N]], label [[EXIT:%.*]], label [[SCALAR_PH]]
@@ -82,7 +78,7 @@ define void @store_pointer_induction(ptr %start, ptr %end) {
 ; CHECK-NEXT:    store ptr [[IV]], ptr [[IV]], align 4
 ; CHECK-NEXT:    [[IV_NEXT]] = getelementptr inbounds ptr, ptr [[IV]], i32 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq ptr [[IV_NEXT]], [[END]]
-; CHECK-NEXT:    br i1 [[EXITCOND]], label [[EXIT]], label [[LOOP]], !llvm.loop [[LOOP2:![0-9]+]]
+; CHECK-NEXT:    br i1 [[EXITCOND]], label [[EXIT]], label [[LOOP]], !llvm.loop [[LOOP3:![0-9]+]]
 ; CHECK:       exit:
 ; CHECK-NEXT:    ret void
 ;
diff --git a/llvm/test/Transforms/LoopVectorize/pr55100-expand-scev-predicate-used.ll b/llvm/test/Transforms/LoopVectorize/pr55100-expand-scev-predicate-used.ll
index f40176ff2e7d5..d3b8dafaf0002 100644
--- a/llvm/test/Transforms/LoopVectorize/pr55100-expand-scev-predicate-used.ll
+++ b/llvm/test/Transforms/LoopVectorize/pr55100-expand-scev-predicate-used.ll
@@ -10,11 +10,8 @@ define void @test_pr55100(i32 %N) {
 ; CHECK-NEXT:    [[IV_1:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[IV_1_NEXT:%.*]], [[LOOP_1_LATCH:%.*]] ]
 ; CHECK-NEXT:    [[TMP1:%.*]] = mul nuw nsw i32 [[IV_1]], -1
 ; CHECK-NEXT:    [[TMP2:%.*]] = add i32 [[TMP0]], [[TMP1]]
-; CHECK-NEXT:    [[UMIN2:%.*]] = call i32 @llvm.umin.i32(i32 [[TMP2]], i32 18)
-; CHECK-NEXT:    [[TMP3:%.*]] = add nuw nsw i32 [[UMIN2]], 1
-; CHECK-NEXT:    [[TMP4:%.*]] = mul nuw nsw i32 [[IV_1]], -1
-; CHECK-NEXT:    [[TMP5:%.*]] = add i32 [[TMP0]], [[TMP4]]
-; CHECK-NEXT:    [[UMIN:%.*]] = call i32 @llvm.umin.i32(i32 [[TMP5]], i32 18)
+; CHECK-NEXT:    [[UMIN:%.*]] = call i32 @llvm.umin.i32(i32 [[TMP2]], i32 18)
+; CHECK-NEXT:    [[TMP3:%.*]] = add nuw nsw i32 [[UMIN]], 1
 ; CHECK-NEXT:    [[C_2:%.*]] = icmp ugt i32 [[IV_1]], 10
 ; CHECK-NEXT:    br i1 [[C_2]], label [[LOOP_2_HEADER_PREHEADER:%.*]], label [[EXIT_LOOPEXIT1:%.*]]
 ; CHECK:       loop.2.header.preheader:
@@ -22,24 +19,21 @@ define void @test_pr55100(i32 %N) {
 ; CHECK-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
 ; CHECK:       vector.ph:
 ; CHECK-NEXT:    [[N_MOD_VF:%.*]] = urem i32 [[TMP3]], 2
-; CHECK-NEXT:    [[TMP6:%.*]] = icmp eq i32 [[N_MOD_VF]], 0
-; CHECK-NEXT:    [[TMP7:%.*]] = select i1 [[TMP6]], i32 2, i32 [[N_MOD_VF]]
-; CHECK-NEXT:    [[N_VEC:%.*]] = sub i32 [[TMP3]], [[TMP7]]
+; CHECK-NEXT:    [[TMP4:%.*]] = icmp eq i32 [[N_MOD_VF]], 0
+; CHECK-NEXT:    [[TMP5:%.*]] = select i1 [[TMP4]], i32 2, i32 [[N_MOD_VF]]
+; CHECK-NEXT:    [[N_VEC:%.*]] = sub i32 [[TMP3]], [[TMP5]]
 ; CHECK-NEXT:    [[IND_END:%.*]] = trunc i32 [[N_VEC]] to i16
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
 ; CHECK-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 2
-; CHECK-NEXT:    [[TMP8:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
-; CHECK-NEXT:    br i1 [[TMP8]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; CHECK-NEXT:    [[TMP6:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
+; CHECK-NEXT:    br i1 [[TMP6]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
 ; CHECK:       middle.block:
 ; CHECK-NEXT:    br label [[SCALAR_PH]]
 ; CHECK:       scalar.ph:
 ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i16 [ [[IND_END]], [[MIDDLE_BLOCK]] ], [ 0, [[LOOP_2_HEADER_PREHEADER]] ]
 ; CHECK-NEXT:    br label [[LOOP_2_HEADER:%.*]]
-; CHECK:       vector.scevcheck:
-; CHECK-NEXT:    [[TMP9:%.*]] = trunc i32 [[UMIN]] to i16
-; CHECK-NEXT:    unreachable
 ; CHECK:       loop.2.header:
 ; CHECK-NEXT:    [[IV_2:%.*]] = phi i16 [ [[IV_2_NEXT:%.*]], [[LOOP_2_LATCH:%.*]] ], [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ]
 ; CHECK-NEXT:    [[C_3:%.*]] = icmp slt i16 [[IV_2]], 18
diff --git a/llvm/test/Transforms/LoopVectorize/reduction-inloop-cond.ll b/llvm/test/Transforms/LoopVectorize/reduction-inloop-cond.ll
index 34767500dfd85..0836d8b79513e 100644
--- a/llvm/test/Transforms/LoopVectorize/reduction-inloop-cond.ll
+++ b/llvm/test/Transforms/LoopVectorize/reduction-inloop-cond.ll
@@ -353,124 +353,104 @@ define i32 @simple_chained_rdx(ptr noalias %a, ptr noalias %b, ptr noalias %cond
 ; CHECK-NEXT:    [[N_VEC:%.*]] = and i64 [[N]], -4
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
-; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_LOAD_CONTINUE14:%.*]] ]
-; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi i32 [ 5, [[VECTOR_PH]] ], [ [[TMP51:%.*]], [[PRED_LOAD_CONTINUE14]] ]
-; CHECK-NEXT:    [[TMP0:%.*]] = or i64 [[INDEX]], 1
-; CHECK-NEXT:    [[TMP1:%.*]] = or i64 [[INDEX]], 2
-; CHECK-NEXT:    [[TMP2:%.*]] = or i64 [[INDEX]], 3
-; CHECK-NEXT:    [[TMP3:%.*]] = getelementptr inbounds i32, ptr [[COND:%.*]], i64 [[INDEX]]
-; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP3]], align 4
-; CHECK-NEXT:    [[TMP5:%.*]] = icmp ne <4 x i32> [[WIDE_LOAD]], zeroinitializer
-; CHECK-NEXT:    [[TMP6:%.*]] = extractelement <4 x i1> [[TMP5]], i64 0
-; CHECK-NEXT:    br i1 [[TMP6]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_LOAD_CONTINUE6:%.*]] ]
+; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi i32 [ 5, [[VECTOR_PH]] ], [ [[TMP46:%.*]], [[PRED_LOAD_CONTINUE6]] ]
+; CHECK-NEXT:    [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[COND:%.*]], i64 [[INDEX]]
+; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp ne <4 x i32> [[WIDE_LOAD]], zeroinitializer
+; CHECK-NEXT:    [[TMP2:%.*]] = extractelement <4 x i1> [[TMP1]], i64 0
+; CHECK-NEXT:    br i1 [[TMP2]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
 ; CHECK:       pred.load.if:
-; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr inbounds i32, ptr [[A:%.*]], i64 [[INDEX]]
-; CHECK-NEXT:    [[TMP8:%.*]] = load i32, ptr [[TMP7]], align 4
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <4 x i32> poison, i32 [[TMP8]], i64 0
+; CHECK-NEXT:    [[TMP3:%.*]] = getelementptr inbounds i32, ptr [[A:%.*]], i64 [[INDEX]]
+; CHECK-NEXT:    [[TMP4:%.*]] = load i32, ptr [[TMP3]], align 4
+; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <4 x i32> poison, i32 [[TMP4]], i64 0
+; CHECK-NEXT:    [[TMP6:%.*]] = getelementptr inbounds i32, ptr [[B:%.*]], i64 [[INDEX]]
+; CHECK-NEXT:    [[TMP7:%.*]] = load i32, ptr [[TMP6]], align 4
+; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <4 x i32> poison, i32 [[TMP7]], i64 0
 ; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE]]
 ; CHECK:       pred.load.continue:
-; CHECK-NEXT:    [[TMP10:%.*]] = phi <4 x i32> [ poison, [[VECTOR_BODY]] ], [ [[TMP9]], [[PRED_LOAD_IF]] ]
-; CHECK-NEXT:    [[TMP11:%.*]] = extractelement <4 x i1> [[TMP5]], i64 1
+; CHECK-NEXT:    [[TMP9:%.*]] = phi <4 x i32> [ poison, [[VECTOR_BODY]] ], [ [[TMP5]], [[PRED_LOAD_IF]] ]
+; CHECK-NEXT:    [[TMP10:%.*]] = phi <4 x i32> [ poison, [[VECTOR_BODY]] ], [ [[TMP8]], [[PRED_LOAD_IF]] ]
+; CHECK-NEXT:    [[TMP11:%.*]] = extractelement <4 x i1> [[TMP1]], i64 1
 ; CHECK-NEXT:    br i1 [[TMP11]], label [[PRED_LOAD_IF1:%.*]], label [[PRED_LOAD_CONTINUE2:%.*]]
 ; CHECK:       pred.load.if1:
-; CHECK-NEXT:    [[TMP12:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[TMP0]]
-; CHECK-NEXT:    [[TMP13:%.*]] = load i32, ptr [[TMP12]], align 4
-; CHECK-NEXT:    [[TMP14:%.*]] = insertelement <4 x i32> [[TMP10]], i32 [[TMP13]], i64 1
+; CHECK-NEXT:    [[TMP12:%.*]] = or i64 [[INDEX]], 1
+; CHECK-NEXT:    [[TMP13:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[TMP12]]
+; CHECK-NEXT:    [[TMP14:%.*]] = load i32, ptr [[TMP13]], align 4
+; CHECK-NEXT:    [[TMP15:%.*]] = insertelement <4 x i32> [[TMP9]], i32 [[TMP14]], i64 1
+; CHECK-NEXT:    [[TMP16:%.*]] = getelementptr inbounds i32, ptr [[B]], i64 [[TMP12]]
+; CHECK-NEXT:    [[TMP17:%.*]] = load i32, ptr [[TMP16]], align 4
+; CHECK-NEXT:    [[TMP18:%.*]] = insertelement <4 x i32> [[TMP10]], i32 [[TMP17]], i64 1
 ; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE2]]
 ; CHECK:       pred.load.continue2:
-; CHECK-NEXT:    [[TMP15:%.*]] = phi <4 x i32> [ [[TMP10]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP14]], [[PRED_LOAD_IF1]] ]
-; CHECK-NEXT:    [[TMP16:%.*]] = extractelement <4 x i1> [[TMP5]], i64 2
-; CHECK-NEXT:    br i1 [[TMP16]], label [[PRED_LOAD_IF3:%.*]], label [[PRED_LOAD_CONTINUE4:%.*]]
+; CHECK-NEXT:    [[TMP19:%.*]] = phi <4 x i32> [ [[TMP9]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP15]], [[PRED_LOAD_IF1]] ]
+; CHECK-NEXT:    [[TMP20:%.*]] = phi <4 x i32> [ [[TMP10]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP18]], [[PRED_LOAD_IF1]] ]
+; CHECK-NEXT:    [[TMP21:%.*]] = extractelement <4 x i1> [[TMP1]], i64 2
+; CHECK-NEXT:    br i1 [[TMP21]], label [[PRED_LOAD_IF3:%.*]], label [[PRED_LOAD_CONTINUE4:%.*]]
 ; CHECK:       pred.load.if3:
-; CHECK-NEXT:    [[TMP17:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[TMP1]]
-; CHECK-NEXT:    [[TMP18:%.*]] = load i32, ptr [[TMP17]], align 4
-; CHECK-NEXT:    [[TMP19:%.*]] = insertelement <4 x i32> [[TMP15]], i32 [[TMP18]], i64 2
+; CHECK-NEXT:    [[TMP22:%.*]] = or i64 [[INDEX]], 2
+; CHECK-NEXT:    [[TMP23:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[TMP22]]
+; CHECK-NEXT:    [[TMP24:%.*]] = load i32, ptr [[TMP23]], align 4
+; CHECK-NEXT:    [[TMP25:%.*]] = insertelement <4 x i32> [[TMP19]], i32 [[TMP24]], i64 2
+; CHECK-NEXT:    [[TMP26:%.*]] = getelementptr inbounds i32, ptr [[B]], i64 [[TMP22]]
+; CHECK-NEXT:    [[TMP27:%.*]] = load i32, ptr [[TMP26]], align 4
+; CHECK-NEXT:    [[TMP28:%.*]] = insertelement <4 x i32> [[TMP20]], i32 [[TMP27]], i64 2
 ; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE4]]
 ; CHECK:       pred.load.continue4:
-; CHECK-NEXT:    [[TMP20:%.*]] = phi <4 x i32> [ [[TMP15]], [[PRED_LOAD_CONTINUE2]] ], [ [[TMP19]], [[PRED_LOAD_IF3]] ]
-; CHECK-NEXT:    [[TMP21:%.*]] = extractelement <4 x i1> [[TMP5]], i64 3
-; CHECK-NEXT:    br i1 [[TMP21]], label [[PRED_LOAD_IF5:%.*]], label [[PRED_LOAD_CONTINUE6:%.*]]
+; CHECK-NEXT:    [[TMP29:%.*]] = phi <4 x i32> [ [[TMP19]], [[PRED_LOAD_CONTINUE2]] ], [ [[TMP25]], [[PRED_LOAD_IF3]] ]
+; CHECK-NEXT:    [[TMP30:%.*]] = phi <4 x i32> [ [[TMP20]], [[PRED_LOAD_CONTINUE2]] ], [ [[TMP28]], [[PRED_LOAD_IF3]] ]
+; CHECK-NEXT:    [[TMP31:%.*]] = extractelement <4 x i1> [[TMP1]], i64 3
+; CHECK-NEXT:    br i1 [[TMP31]], label [[PRED_LOAD_IF5:%.*]], label [[PRED_LOAD_CONTINUE6]]
 ; CHECK:       pred.load.if5:
-; CHECK-NEXT:    [[TMP22:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[TMP2]]
-; CHECK-NEXT:    [[TMP23:%.*]] = load i32, ptr [[TMP22]], align 4
-; CHECK-NEXT:    [[TMP24:%.*]] = insertelement <4 x i32> [[TMP20]], i32 [[TMP23]], i64 3
+; CHECK-NEXT:    [[TMP32:%.*]] = or i64 [[INDEX]], 3
+; CHECK-NEXT:    [[TMP33:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[TMP32]]
+; CHECK-NEXT:    [[TMP34:%.*]] = load i32, ptr [[TMP33]], align 4
+; CHECK-NEXT:    [[TMP35:%.*]] = insertelement <4 x i32> [[TMP29]], i32 [[TMP34]], i64 3
+; CHECK-NEXT:    [[TMP36:%.*]] = getelementptr inbounds i32, ptr [[B]], i64 [[TMP32]]
+; CHECK-NEXT:    [[TMP37:%.*]] = load i32, ptr [[TMP36]], align 4
+; CHECK-NEXT:    [[TMP38:%.*]] = insertelement <4 x i32> [[TMP30]], i32 [[TMP37]], i64 3
 ; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE6]]
 ; CHECK:       pred.load.continue6:
-; CHECK-NEXT:    [[TMP25:%.*]] = phi <4 x i32> [ [[TMP20]], [[PRED_LOAD_CONTINUE4]] ], [ [[TMP24]], [[PRED_LOAD_IF5]] ]
-; CHECK-NEXT:    [[TMP26:%.*]] = select <4 x i1> [[TMP5]], <4 x i32> [[TMP25]], <4 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP27:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[TMP26]])
-; CHECK-NEXT:    [[TMP28:%.*]] = add i32 [[TMP27]], [[VEC_PHI]]
-; CHECK-NEXT:    [[TMP29:%.*]] = extractelement <4 x i1> [[TMP5]], i64 0
-; CHECK-NEXT:    br i1 [[TMP29]], label [[PRED_LOAD_IF7:%.*]], label [[PRED_LOAD_CONTINUE8:%.*]]
-; CHECK:       pred.load.if7:
-; CHECK-NEXT:    [[TMP30:%.*]] = getelementptr inbounds i32, ptr [[B:%.*]], i64 [[INDEX]]
-; CHECK-NEXT:    [[TMP31:%.*]] = load i32, ptr [[TMP30]], align 4
-; CHECK-NEXT:    [[TMP32:%.*]] = insertelement <4 x i32> poison, i32 [[TMP31]], i64 0
-; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE8]]
-; CHECK:       pred.load.continue8:
-; CHECK-NEXT:    [[TMP33:%.*]] = phi <4 x i32> [ poison, [[PRED_LOAD_CONTINUE6]] ], [ [[TMP32]], [[PRED_LOAD_IF7]] ]
-; CHECK-NEXT:    [[TMP34:%.*]] = extractelement <4 x i1> [[TMP5]], i64 1
-; CHECK-NEXT:    br i1 [[TMP34]], label [[PRED_LOAD_IF9:%.*]], label [[PRED_LOAD_CONTINUE10:%.*]]
-; CHECK:       pred.load.if9:
-; CHECK-NEXT:    [[TMP35:%.*]] = getelementptr inbounds i32, ptr [[B]], i64 [[TMP0]]
-; CHECK-NEXT:    [[TMP36:%.*]] = load i32, ptr [[TMP35]], align 4
-; CHECK-NEXT:    [[TMP37:%.*]] = insertelement <4 x i32> [[TMP33]], i32 [[TMP36]], i64 1
-; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE10]]
-; CHECK:       pred.load.continue10:
-; CHECK-NEXT:    [[TMP38:%.*]] = phi <4 x i32> [ [[TMP33]], [[PRED_LOAD_CONTINUE8]] ], [ [[TMP37]], [[PRED_LOAD_IF9]] ]
-; CHECK-NEXT:    [[TMP39:%.*]] = extractelement <4 x i1> [[TMP5]], i64 2
-; CHECK-NEXT:    br i1 [[TMP39]], label [[PRED_LOAD_IF11:%.*]], label [[PRED_LOAD_CONTINUE12:%.*]]
-; CHECK:       pred.load.if11:
-; CHECK-NEXT:    [[TMP40:%.*]] = getelementptr inbounds i32, ptr [[B]], i64 [[TMP1]]
-; CHECK-NEXT:    [[TMP41:%.*]] = load i32, ptr [[TMP40]], align 4
-; CHECK-NEXT:    [[TMP42:%.*]] = insertelement <4 x i32> [[TMP38]], i32 [[TMP41]], i64 2
-; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE12]]
-; CHECK:       pred.load.continue12:
-; CHECK-NEXT:    [[TMP43:%.*]] = phi <4 x i32> [ [[TMP38]], [[PRED_LOAD_CONTINUE10]] ], [ [[TMP42]], [[PRED_LOAD_IF11]] ]
-; CHECK-NEXT:    [[TMP44:%.*]] = extractelement <4 x i1> [[TMP5]], i64 3
-; CHECK-NEXT:    br i1 [[TMP44]], label [[PRED_LOAD_IF13:%.*]], label [[PRED_LOAD_CONTINUE14]]
-; CHECK:       pred.load.if13:
-; CHECK-NEXT:    [[TMP45:%.*]] = getelementptr inbounds i32, ptr [[B]], i64 [[TMP2]]
-; CHECK-NEXT:    [[TMP46:%.*]] = load i32, ptr [[TMP45]], align 4
-; CHECK-NEXT:    [[TMP47:%.*]] = insertelement <4 x i32> [[TMP43]], i32 [[TMP46]], i64 3
-; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE14]]
-; CHECK:       pred.load.continue14:
-; CHECK-NEXT:    [[TMP48:%.*]] = phi <4 x i32> [ [[TMP43]], [[PRED_LOAD_CONTINUE12]] ], [ [[TMP47]], [[PRED_LOAD_IF13]] ]
-; CHECK-NEXT:    [[TMP49:%.*]] = select <4 x i1> [[TMP5]], <4 x i32> [[TMP48]], <4 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP50:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[TMP49]])
-; CHECK-NEXT:    [[TMP51]] = add i32 [[TMP50]], [[TMP28]]
+; CHECK-NEXT:    [[TMP39:%.*]] = phi <4 x i32> [ [[TMP29]], [[PRED_LOAD_CONTINUE4]] ], [ [[TMP35]], [[PRED_LOAD_IF5]] ]
+; CHECK-NEXT:    [[TMP40:%.*]] = phi <4 x i32> [ [[TMP30]], [[PRED_LOAD_CONTINUE4]] ], [ [[TMP38]], [[PRED_LOAD_IF5]] ]
+; CHECK-NEXT:    [[TMP41:%.*]] = select <4 x i1> [[TMP1]], <4 x i32> [[TMP39]], <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP42:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[TMP41]])
+; CHECK-NEXT:    [[TMP43:%.*]] = add i32 [[TMP42]], [[VEC_PHI]]
+; CHECK-NEXT:    [[TMP44:%.*]] = select <4 x i1> [[TMP1]], <4 x i32> [[TMP40]], <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP45:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[TMP44]])
+; CHECK-NEXT:    [[TMP46]] = add i32 [[TMP45]], [[TMP43]]
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
-; CHECK-NEXT:    [[TMP52:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
-; CHECK-NEXT:    br i1 [[TMP52]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; CHECK-NEXT:    [[TMP47:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; CHECK-NEXT:    br i1 [[TMP47]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
 ; CHECK:       middle.block:
 ; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 [[N_VEC]], [[N]]
 ; CHECK-NEXT:    br i1 [[CMP_N]], label [[FOR_END:%.*]], label [[SCALAR_PH]]
 ; CHECK:       scalar.ph:
 ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
-; CHECK-NEXT:    [[BC_MERGE_RDX:%.*]] = phi i32 [ [[TMP51]], [[MIDDLE_BLOCK]] ], [ 5, [[ENTRY]] ]
+; CHECK-NEXT:    [[BC_MERGE_RDX:%.*]] = phi i32 [ [[TMP46]], [[MIDDLE_BLOCK]] ], [ 5, [[ENTRY]] ]
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ [[IV_NEXT:%.*]], [[FOR_INC:%.*]] ], [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ]
 ; CHECK-NEXT:    [[RDX:%.*]] = phi i32 [ [[RES:%.*]], [[FOR_INC]] ], [ [[BC_MERGE_RDX]], [[SCALAR_PH]] ]
 ; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[IV]]
-; CHECK-NEXT:    [[TMP53:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
-; CHECK-NEXT:    [[TOBOOL_NOT:%.*]] = icmp eq i32 [[TMP53]], 0
+; CHECK-NEXT:    [[TMP48:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
+; CHECK-NEXT:    [[TOBOOL_NOT:%.*]] = icmp eq i32 [[TMP48]], 0
 ; CHECK-NEXT:    br i1 [[TOBOOL_NOT]], label [[FOR_INC]], label [[IF_THEN:%.*]]
 ; CHECK:       if.then:
 ; CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[IV]]
-; CHECK-NEXT:    [[TMP54:%.*]] = load i32, ptr [[ARRAYIDX1]], align 4
-; CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP54]], [[RDX]]
+; CHECK-NEXT:    [[TMP49:%.*]] = load i32, ptr [[ARRAYIDX1]], align 4
+; CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP49]], [[RDX]]
 ; CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds i32, ptr [[B]], i64 [[IV]]
-; CHECK-NEXT:    [[TMP55:%.*]] = load i32, ptr [[ARRAYIDX2]], align 4
-; CHECK-NEXT:    [[ADD3:%.*]] = add nsw i32 [[ADD]], [[TMP55]]
+; CHECK-NEXT:    [[TMP50:%.*]] = load i32, ptr [[ARRAYIDX2]], align 4
+; CHECK-NEXT:    [[ADD3:%.*]] = add nsw i32 [[ADD]], [[TMP50]]
 ; CHECK-NEXT:    br label [[FOR_INC]]
 ; CHECK:       for.inc:
 ; CHECK-NEXT:    [[RES]] = phi i32 [ [[ADD3]], [[IF_THEN]] ], [ [[RDX]], [[FOR_BODY]] ]
 ; CHECK-NEXT:    [[IV_NEXT]] = add nuw nsw i64 [[IV]], 1
 ; CHECK-NEXT:    [[EXITCOND_NOT:%.*]] = icmp eq i64 [[IV_NEXT]], [[N]]
-; CHECK-NEXT:    br i1 [[EXITCOND_NOT]], label [[FOR_END]], label [[FOR_BODY]], !llvm.loop [[LOOP2:![0-9]+]]
+; CHECK-NEXT:    br i1 [[EXITCOND_NOT]], label [[FOR_END]], label [[FOR_BODY]], !llvm.loop [[LOOP9:![0-9]+]]
 ; CHECK:       for.end:
-; CHECK-NEXT:    [[RES_LCSSA:%.*]] = phi i32 [ [[RES]], [[FOR_INC]] ], [ [[TMP51]], [[MIDDLE_BLOCK]] ]
+; CHECK-NEXT:    [[RES_LCSSA:%.*]] = phi i32 [ [[RES]], [[FOR_INC]] ], [ [[TMP46]], [[MIDDLE_BLOCK]] ]
 ; CHECK-NEXT:    ret i32 [[RES_LCSSA]]
 ;
 entry:
diff --git a/llvm/test/Transforms/LoopVectorize/vplan-sink-scalars-and-merge.ll b/llvm/test/Transforms/LoopVectorize/vplan-sink-scalars-and-merge.ll
index 98749daec9c90..405fa0f628593 100644
--- a/llvm/test/Transforms/LoopVectorize/vplan-sink-scalars-and-merge.ll
+++ b/llvm/test/Transforms/LoopVectorize/vplan-sink-scalars-and-merge.ll
@@ -12,7 +12,6 @@ target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f3
 ; CHECK-LABEL: LV: Checking a loop in 'sink1'
 ; CHECK:      VPlan 'Initial VPlan for VF={2},UF>=1' {
 ; CHECK-NEXT: Live-in vp<[[VEC_TC:%.+]]> = vector-trip-count
-; CHECK-EMPTY:
 ; CHECK-NEXT: Live-in vp<[[BTC:%.+]]> = backedge-taken count
 ; CHECK-EMPTY:
 ; CHECK-NEXT: vector.ph:
@@ -76,7 +75,6 @@ exit:
 ; CHECK-LABEL: LV: Checking a loop in 'sink2'
 ; CHECK:      VPlan 'Initial VPlan for VF={2},UF>=1' {
 ; CHECK-NEXT: Live-in vp<[[VEC_TC:%.+]]> = vector-trip-count
-; CHECK-EMPTY:
 ; CHECK-NEXT: Live-in vp<[[BTC:%.+]]> = backedge-taken count
 ; CHECK-EMPTY:
 ; CHECK-NEXT: vector.ph:
@@ -155,7 +153,6 @@ exit:
 ; CHECK-LABEL: LV: Checking a loop in 'sink3'
 ; CHECK:      VPlan 'Initial VPlan for VF={2},UF>=1' {
 ; CHECK-NEXT: Live-in vp<[[VEC_TC:%.+]]> = vector-trip-count
-; CHECK-EMPTY:
 ; CHECK-NEXT: Live-in vp<[[BTC:%.+]]> = backedge-taken count
 ; CHECK-EMPTY:
 ; CHECK-NEXT: vector.ph:
@@ -236,7 +233,6 @@ define void @uniform_gep(i64 %k, ptr noalias %A, ptr noalias %B) {
 ; CHECK-LABEL: LV: Checking a loop in 'uniform_gep'
 ; CHECK:      VPlan 'Initial VPlan for VF={2},UF>=1' {
 ; CHECK-NEXT: Live-in vp<[[VEC_TC:%.+]]> = vector-trip-count
-; CHECK-EMPTY:
 ; CHECK-NEXT: Live-in vp<[[BTC:%.+]]> = backedge-taken count
 ; CHECK-EMPTY:
 ; CHECK-NEXT: vector.ph:
@@ -306,7 +302,6 @@ define void @pred_cfg1(i32 %k, i32 %j) {
 ; CHECK-LABEL: LV: Checking a loop in 'pred_cfg1'
 ; CHECK:      VPlan 'Initial VPlan for VF={2},UF>=1' {
 ; CHECK-NEXT: Live-in vp<[[VEC_TC:%.+]]> = vector-trip-count
-; CHECK-EMPTY:
 ; CHECK-NEXT: Live-in vp<[[BTC:%.+]]> = backedge-taken count
 ; CHECK-EMPTY:
 ; CHECK-NEXT: vector.ph:
@@ -359,9 +354,9 @@ define void @pred_cfg1(i32 %k, i32 %j) {
 ; CHECK-NEXT:   pred.store.continue:
 ; CHECK-NEXT:   No successors
 ; CHECK-NEXT: }
-; CHECK-NEXT: Successor(s): next.0.0
+; CHECK-NEXT: Successor(s): next.0.1
 ; CHECK-EMPTY:
-; CHECK-NEXT: next.0.0:
+; CHECK-NEXT: next.0.1:
 ; CHECK-NEXT:   EMIT vp<[[CAN_IV_NEXT:%.+]]> = VF * UF + vp<[[CAN_IV]]>
 ; CHECK-NEXT:   EMIT branch-on-count vp<[[CAN_IV_NEXT]]> vp<[[VEC_TC]]>
 ; CHECK-NEXT: No successors
@@ -401,7 +396,6 @@ define void @pred_cfg2(i32 %k, i32 %j) {
 ; CHECK-LABEL: LV: Checking a loop in 'pred_cfg2'
 ; CHECK:      VPlan 'Initial VPlan for VF={2},UF>=1' {
 ; CHECK-NEXT: Live-in vp<[[VEC_TC:%.+]]> = vector-trip-count
-; CHECK-EMPTY:
 ; CHECK-NEXT: Live-in vp<[[BTC:%.+]]> = backedge-taken count
 ; CHECK-EMPTY:
 ; CHECK-NEXT: vector.ph:
@@ -456,9 +450,9 @@ define void @pred_cfg2(i32 %k, i32 %j) {
 ; CHECK-NEXT:   pred.store.continue:
 ; CHECK-NEXT:   No successors
 ; CHECK-NEXT: }
-; CHECK-NEXT: Successor(s): then.1.0
+; CHECK-NEXT: Successor(s): then.1.1
 ; CHECK-EMPTY:
-; CHECK-NEXT: then.1.0:
+; CHECK-NEXT: then.1.1:
 ; CHECK-NEXT:   EMIT vp<[[CAN_IV_NEXT:%.+]]> = VF * UF + vp<[[CAN_IV]]>
 ; CHECK-NEXT:   EMIT branch-on-count vp<[[CAN_IV_NEXT]]> vp<[[VEC_TC]]>
 ; CHECK-NEXT: No successors
@@ -505,7 +499,6 @@ define void @pred_cfg3(i32 %k, i32 %j) {
 ; CHECK-LABEL: LV: Checking a loop in 'pred_cfg3'
 ; CHECK:      VPlan 'Initial VPlan for VF={2},UF>=1' {
 ; CHECK-NEXT: Live-in vp<[[VEC_TC:%.+]]> = vector-trip-count
-; CHECK-EMPTY:
 ; CHECK-NEXT: Live-in vp<[[BTC:%.+]]> = backedge-taken count
 ; CHECK-EMPTY:
 ; CHECK-NEXT: vector.ph:
@@ -561,9 +554,9 @@ define void @pred_cfg3(i32 %k, i32 %j) {
 ; CHECK-NEXT:   pred.store.continue:
 ; CHECK-NEXT:   No successors
 ; CHECK-NEXT: }
-; CHECK-NEXT: Successor(s): then.1.1
+; CHECK-NEXT: Successor(s): then.1.2
 ; CHECK-EMPTY:
-; CHECK-NEXT: then.1.1:
+; CHECK-NEXT: then.1.2:
 ; CHECK-NEXT:   EMIT vp<[[CAN_IV_NEXT:%.+]]> = VF * UF + vp<[[CAN_IV]]>
 ; CHECK-NEXT:   EMIT branch-on-count vp<[[CAN_IV_NEXT]]> vp<[[VEC_TC]]>
 ; CHECK-NEXT: No successors
@@ -609,7 +602,6 @@ define void @merge_3_replicate_region(i32 %k, i32 %j) {
 ; CHECK-LABEL: LV: Checking a loop in 'merge_3_replicate_region'
 ; CHECK:      VPlan 'Initial VPlan for VF={2},UF>=1' {
 ; CHECK-NEXT: Live-in vp<[[VEC_TC:%.+]]> = vector-trip-count
-; CHECK-EMPTY:
 ; CHECK-NEXT: Live-in vp<[[BTC:%.+]]> = backedge-taken count
 ; CHECK-EMPTY:
 ; CHECK-NEXT: vector.ph:
@@ -664,9 +656,9 @@ define void @merge_3_replicate_region(i32 %k, i32 %j) {
 ; CHECK-NEXT:   pred.store.continue:
 ; CHECK-NEXT:   No successors
 ; CHECK-NEXT: }
-; CHECK-NEXT: Successor(s): then.0.0
+; CHECK-NEXT: Successor(s): then.0.4
 ; CHECK-EMPTY:
-; CHECK-NEXT: then.0.0:
+; CHECK-NEXT: then.0.4:
 ; CHECK-NEXT:   EMIT vp<[[CAN_IV_NEXT:%.+]]> = VF * UF + vp<[[CAN_IV]]>
 ; CHECK-NEXT:   EMIT branch-on-count vp<[[CAN_IV_NEXT]]> vp<[[VEC_TC]]>
 ; CHECK-NEXT: No successors
@@ -709,7 +701,6 @@ define void @update_2_uses_in_same_recipe_in_merged_block(i32 %k) {
 ; CHECK-LABEL: LV: Checking a loop in 'update_2_uses_in_same_recipe_in_merged_block'
 ; CHECK:      VPlan 'Initial VPlan for VF={2},UF>=1' {
 ; CHECK-NEXT: Live-in vp<[[VEC_TC:%.+]]> = vector-trip-count
-; CHECK-EMPTY:
 ; CHECK-NEXT: Live-in vp<[[BTC:%.+]]> = backedge-taken count
 ; CHECK-EMPTY:
 ; CHECK-NEXT: vector.ph:
@@ -771,7 +762,6 @@ define void @recipe_in_merge_candidate_used_by_first_order_recurrence(i32 %k) {
 ; CHECK-LABEL: LV: Checking a loop in 'recipe_in_merge_candidate_used_by_first_order_recurrence'
 ; CHECK:      VPlan 'Initial VPlan for VF={2},UF>=1' {
 ; CHECK-NEXT: Live-in vp<[[VEC_TC:%.+]]> = vector-trip-count
-; CHECK-EMPTY:
 ; CHECK-NEXT: Live-in vp<[[BTC:%.+]]> = backedge-taken count
 ; CHECK-EMPTY:
 ; CHECK-NEXT: vector.ph:
@@ -982,7 +972,6 @@ define void @merge_with_dead_gep_between_regions(i32 %n, ptr noalias %src, ptr n
 ; CHECK-LABEL: LV: Checking a loop in 'merge_with_dead_gep_between_regions'
 ; CHECK:      VPlan 'Initial VPlan for VF={2},UF>=1' {
 ; CHECK-NEXT: Live-in vp<[[VEC_TC:%.+]]> = vector-trip-count
-; CHECK-EMPTY:
 ; CHECK-NEXT: Live-in vp<[[BTC:%.+]]> = backedge-taken count
 ; CHECK-EMPTY:
 ; CHECK-NEXT: vector.ph:
diff --git a/llvm/test/Transforms/LowerTypeTests/function-weak.ll b/llvm/test/Transforms/LowerTypeTests/function-weak.ll
index f372fa0c8990a..ff69abacc8e93 100644
--- a/llvm/test/Transforms/LowerTypeTests/function-weak.ll
+++ b/llvm/test/Transforms/LowerTypeTests/function-weak.ll
@@ -47,6 +47,65 @@ entry:
   ret void
 }
 
+define void @phi(i1 %c) {
+; CHECK-LABEL: define void @phi(i1 %c) {
+; CHECK: entry:
+; CHECK:   %0 = select i1 icmp ne (ptr @f, ptr null), ptr @.cfi.jumptable, ptr null
+; CHECK:   br i1 %c, label %if, label %join
+; CHECK: if:
+; CHECK:   %1 = select i1 icmp ne (ptr @f, ptr null), ptr @.cfi.jumptable, ptr null
+; CHECK:   br label %join
+; CHECK: join:
+; CHECK:   %phi = phi ptr [ %1, %if ], [ null, %entry ]
+; CHECK:   %phi2 = phi ptr [ null, %if ], [ %0, %entry ]
+
+entry:
+  br i1 %c, label %if, label %join
+
+if:
+  br label %join
+
+join:
+  %phi = phi ptr [ @f, %if ], [ null, %entry ]
+  %phi2 = phi ptr [ null, %if ], [ @f, %entry ]
+  ret void
+}
+
+define void @phi2(i1 %c, i32 %x) {
+; CHECK-LABEL: define void @phi2(i1 %c, i32 %x) {
+; CHECK: entry:
+; CHECK:   br i1 %c, label %if, label %else
+; CHECK: if:                                               ; preds = %entry
+; CHECK:   %0 = select i1 icmp ne (ptr @f, ptr null), ptr @.cfi.jumptable, ptr null
+; CHECK:   switch i32 %x, label %join [
+; CHECK:     i32 0, label %join
+; CHECK:   ]
+; CHECK: else:                                             ; preds = %entry
+; CHECK:   %1 = select i1 icmp ne (ptr @f, ptr null), ptr @.cfi.jumptable, ptr null
+; CHECK:   switch i32 %x, label %join [
+; CHECK:     i32 0, label %join
+; CHECK:   ]
+; CHECK: join:                                             ; preds = %else, %else, %if, %if
+; CHECK:   %phi2 = phi ptr [ %0, %if ], [ %0, %if ], [ %1, %else ], [ %1, %else ]
+
+entry:
+  br i1 %c, label %if, label %else
+
+if:
+  switch i32 %x, label %join [
+    i32 0, label %join
+  ]
+
+else:
+  switch i32 %x, label %join [
+    i32 0, label %join
+  ]
+
+join:
+  %phi2 = phi ptr [ @f, %if ], [ @f, %if ], [ @f, %else ], [ @f, %else ]
+  ret void
+}
+
 declare i1 @llvm.type.test(ptr %ptr, metadata %bitset) nounwind readnone
 
 define i1 @foo(ptr %p) {
diff --git a/llvm/test/Transforms/MetaRenamer/opcodes.ll b/llvm/test/Transforms/MetaRenamer/opcodes.ll
new file mode 100644
index 0000000000000..8a78677edbff1
--- /dev/null
+++ b/llvm/test/Transforms/MetaRenamer/opcodes.ll
@@ -0,0 +1,45 @@
+; RUN: opt -passes=metarenamer -S < %s | FileCheck %s
+
+define void @opcodes(ptr %p, ptr %arr) {
+; CHECK-LABEL: bb:
+; CHECK:         %load = load i32, ptr %arg, align 4
+; CHECK:         br label %bb2
+; CHECK-LABEL: bb2:                                              ; preds = %bb5, %bb
+; CHECK:         %phi = phi i32 [ %load, %bb ], [ %sub, %bb5 ]
+; CHECK:         %icmp = icmp eq i32 %phi, 0
+; CHECK:         br i1 %icmp, label %bb8, label %bb3
+; CHECK-LABEL: bb3:                                              ; preds = %bb2
+; CHECK:         %sub = sub i32 %phi, 1
+; CHECK:         %icmp4 = icmp ult i32 %sub, %load
+; CHECK:         br i1 %icmp4, label %bb5, label %bb9
+; CHECK-LABEL: bb5:                                              ; preds = %bb3
+; CHECK:         %getelementptr = getelementptr i32, ptr %arg, i32 %phi
+; CHECK:         %load6 = load i32, ptr %getelementptr, align 4
+; CHECK:         %icmp7 = icmp eq i32 %load6, 0
+; CHECK:         br i1 %icmp7, label %bb2, label %bb8
+preheader:
+  %len = load i32, ptr %p
+  br label %loop
+
+loop:
+  %iv = phi i32 [%len, %preheader], [%iv.next, %backedge]
+  %zero_cond = icmp eq i32 %iv, 0
+  br i1 %zero_cond, label %exit, label %range_check_block
+
+range_check_block:
+  %iv.next = sub i32 %iv, 1
+  %range_check = icmp ult i32 %iv.next, %len
+  br i1 %range_check, label %backedge, label %fail
+
+backedge:
+  %el.ptr = getelementptr i32, ptr %p, i32 %iv
+  %el = load i32, ptr %el.ptr
+  %loop.cond = icmp eq i32 %el, 0
+  br i1 %loop.cond, label %loop, label %exit
+
+exit:
+  ret void
+
+fail:
+  unreachable
+}
diff --git a/llvm/test/Transforms/PGOProfile/Inputs/memprof.exe b/llvm/test/Transforms/PGOProfile/Inputs/memprof.exe
index 6dd5783eb5acc..b10c2f9c72147 100755
Binary files a/llvm/test/Transforms/PGOProfile/Inputs/memprof.exe and b/llvm/test/Transforms/PGOProfile/Inputs/memprof.exe differ
diff --git a/llvm/test/Transforms/PGOProfile/Inputs/memprof.memprofraw b/llvm/test/Transforms/PGOProfile/Inputs/memprof.memprofraw
index a5e3a517f1b32..790249a44b0d1 100644
Binary files a/llvm/test/Transforms/PGOProfile/Inputs/memprof.memprofraw and b/llvm/test/Transforms/PGOProfile/Inputs/memprof.memprofraw differ
diff --git a/llvm/test/Transforms/PGOProfile/Inputs/memprof_pgo.profraw b/llvm/test/Transforms/PGOProfile/Inputs/memprof_pgo.profraw
index 87f1cded0988f..d5160e75d5ec2 100644
Binary files a/llvm/test/Transforms/PGOProfile/Inputs/memprof_pgo.profraw and b/llvm/test/Transforms/PGOProfile/Inputs/memprof_pgo.profraw differ
diff --git a/llvm/test/Transforms/PGOProfile/Inputs/update_memprof_inputs.sh b/llvm/test/Transforms/PGOProfile/Inputs/update_memprof_inputs.sh
new file mode 100755
index 0000000000000..0570e2cf2eefa
--- /dev/null
+++ b/llvm/test/Transforms/PGOProfile/Inputs/update_memprof_inputs.sh
@@ -0,0 +1,89 @@
+#!/bin/bash
+
+if [ -z $1 ]; then
+  echo "Path to clang required!"
+  echo "Usage: update_memprof_inputs.sh /path/to/updated/clang"
+  exit 1
+else
+  CLANG=$1
+fi
+
+# Allows the script to be invoked from other directories.
+OUTDIR=$(dirname $(realpath -s $0))
+
+# Note that changes in the code below which affect relative line number
+# offsets of calls from their parent function can affect callsite matching in
+# the LLVM IR.
+cat > ${OUTDIR}/memprof.cc << EOF
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+char *foo() {
+  return new char[10];
+}
+char *foo2() {
+  return foo();
+}
+char *bar() {
+  return foo2();
+}
+char *baz() {
+  return foo2();
+}
+char *recurse(unsigned n) {
+  if (!n)
+    return foo();
+  return recurse(n-1);
+}
+int main(int argc, char **argv) {
+  // Test allocations with different combinations of stack contexts and
+  // coldness (based on lifetime, since they are all accessed a single time
+  // per byte via the memset).
+  char *a = new char[10];
+  char *b = new char[10];
+  char *c = foo();
+  char *d = foo();
+  char *e = bar();
+  char *f = baz();
+  memset(a, 0, 10);
+  memset(b, 0, 10);
+  memset(c, 0, 10);
+  memset(d, 0, 10);
+  memset(e, 0, 10);
+  memset(f, 0, 10);
+  // a and c have short lifetimes
+  delete[] a;
+  delete[] c;
+  // b, d, e, and f have long lifetimes and will be detected as cold by default.
+  sleep(200);
+  delete[] b;
+  delete[] d;
+  delete[] e;
+  delete[] f;
+
+  // Loop ensures the two calls to recurse have stack contexts that only differ
+  // in one level of recursion. We should get two stack contexts reflecting the
+  // different levels of recursion and different allocation behavior (since the
+  // first has a very long lifetime and the second has a short lifetime).
+  for (unsigned i = 0; i < 2; i++) {
+    char *g = recurse(i + 3);
+    memset(g, 0, 10);
+    if (!i)
+      sleep(200);
+    delete[] g;
+  }
+  return 0;
+}
+EOF
+
+COMMON_FLAGS="-fuse-ld=lld -Wl,--no-rosegment -gmlt -fdebug-info-for-profiling -mno-omit-leaf-frame-pointer -fno-omit-frame-pointer -fno-optimize-sibling-calls -m64 -Wl,-build-id -no-pie"
+
+${CLANG} ${COMMON_FLAGS} -fmemory-profile ${OUTDIR}/memprof.cc -o ${OUTDIR}/memprof.exe
+env MEMPROF_OPTIONS=log_path=stdout ${OUTDIR}/memprof.exe > ${OUTDIR}/memprof.memprofraw
+
+${CLANG} ${COMMON_FLAGS} -fprofile-generate=. \
+  ${OUTDIR}/memprof.cc -o ${OUTDIR}/pgo.exe
+env LLVM_PROFILE_FILE=${OUTDIR}/memprof_pgo.profraw ${OUTDIR}/pgo.exe
+
+rm ${OUTDIR}/memprof.cc
+rm ${OUTDIR}/pgo.exe
diff --git a/llvm/test/Transforms/PGOProfile/chr-poison.ll b/llvm/test/Transforms/PGOProfile/chr-poison.ll
new file mode 100644
index 0000000000000..52d555e0b019e
--- /dev/null
+++ b/llvm/test/Transforms/PGOProfile/chr-poison.ll
@@ -0,0 +1,89 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -passes='require<profile-summary>,function(chr)' -S | FileCheck %s
+
+; Make sure we freeze poison.
+define void @chr_poison(i1 %arg) !prof !15 {
+; CHECK-LABEL: @chr_poison(
+; CHECK-NEXT:    [[TMP1:%.*]] = freeze i1 poison
+; CHECK-NEXT:    [[TMP2:%.*]] = select i1 true, i1 [[TMP1]], i1 false
+; CHECK-NEXT:    [[TMP3:%.*]] = freeze i1 [[ARG:%.*]]
+; CHECK-NEXT:    [[TMP4:%.*]] = select i1 [[TMP2]], i1 [[TMP3]], i1 false
+; CHECK-NEXT:    br i1 [[TMP4]], label [[DOTSPLIT:%.*]], label [[DOTSPLIT_NONCHR:%.*]], !prof [[PROF30:![0-9]+]]
+; CHECK:       .split:
+; CHECK-NEXT:    br i1 false, label [[BB3:%.*]], label [[BB4:%.*]]
+; CHECK:       bb3:
+; CHECK-NEXT:    [[SEL:%.*]] = select i1 false, ptr null, ptr null
+; CHECK-NEXT:    br label [[BB6:%.*]]
+; CHECK:       bb4:
+; CHECK-NEXT:    br i1 true, label [[BB6]], label [[BB5:%.*]], !prof [[PROF31:![0-9]+]]
+; CHECK:       bb5:
+; CHECK-NEXT:    br label [[BB6]]
+; CHECK:       bb6:
+; CHECK-NEXT:    br i1 false, label [[BB9:%.*]], label [[BB7:%.*]]
+; CHECK:       bb7:
+; CHECK-NEXT:    br i1 true, label [[BB9]], label [[BB8:%.*]], !prof [[PROF32:![0-9]+]]
+; CHECK:       bb8:
+; CHECK-NEXT:    br label [[BB9]]
+; CHECK:       .split.nonchr:
+; CHECK-NEXT:    br i1 false, label [[BB3_NONCHR:%.*]], label [[BB4_NONCHR:%.*]]
+; CHECK:       bb3.nonchr:
+; CHECK-NEXT:    [[SEL_NONCHR:%.*]] = select i1 false, ptr null, ptr null
+; CHECK-NEXT:    br label [[BB6_NONCHR:%.*]]
+; CHECK:       bb4.nonchr:
+; CHECK-NEXT:    br i1 poison, label [[BB6_NONCHR]], label [[BB5_NONCHR:%.*]], !prof [[PROF31]]
+; CHECK:       bb5.nonchr:
+; CHECK-NEXT:    br label [[BB6_NONCHR]]
+; CHECK:       bb6.nonchr:
+; CHECK-NEXT:    br i1 false, label [[BB9]], label [[BB7_NONCHR:%.*]]
+; CHECK:       bb7.nonchr:
+; CHECK-NEXT:    br i1 [[ARG]], label [[BB9]], label [[BB8_NONCHR:%.*]], !prof [[PROF32]]
+; CHECK:       bb8.nonchr:
+; CHECK-NEXT:    br label [[BB9]]
+; CHECK:       bb9:
+; CHECK-NEXT:    ret void
+;
+  br i1 false, label %bb3, label %bb4
+
+bb3:
+  %sel = select i1 false, ptr null, ptr null
+  br label %bb6
+
+bb4:
+  br i1 poison, label %bb6, label %bb5, !prof !16
+
+bb5:
+  br label %bb6
+
+bb6:
+  br i1 false, label %bb9, label %bb7
+
+bb7:
+  br i1 %arg, label %bb9, label %bb8, !prof !17
+
+bb8:
+  br label %bb9
+
+bb9:
+  ret void
+}
+
+!llvm.module.flags = !{!0}
+
+!0 = !{i32 1, !"ProfileSummary", !1}
+!1 = !{!2, !3, !4, !5, !6, !7, !8, !9, !10, !11}
+!2 = !{!"ProfileFormat", !"SampleProfile"}
+!3 = !{!"TotalCount", i64 2625732223}
+!4 = !{!"MaxCount", i64 6099111}
+!5 = !{!"MaxInternalCount", i64 0}
+!6 = !{!"MaxFunctionCount", i64 1000000}
+!7 = !{!"NumCounts", i64 1000000}
+!8 = !{!"NumFunctions", i64 14940}
+!9 = !{!"IsPartialProfile", i64 0}
+!10 = !{!"PartialProfileRatio", double 0.000000e+00}
+!11 = !{!"DetailedSummary", !12}
+!12 = !{!13, !14}
+!13 = !{i32 100000, i64 6083920, i32 44}
+!14 = !{i32 999999, i64 1, i32 1000000}
+!15 = !{!"function_entry_count", i64 1000}
+!16 = !{!"branch_weights", i32 1, i32 0}
+!17 = !{!"branch_weights", i32 1, i32 0}
diff --git a/llvm/test/Transforms/PGOProfile/chr.ll b/llvm/test/Transforms/PGOProfile/chr.ll
index 124bc5010fe5c..43cb6a2224de9 100644
--- a/llvm/test/Transforms/PGOProfile/chr.ll
+++ b/llvm/test/Transforms/PGOProfile/chr.ll
@@ -26,7 +26,8 @@ define void @test_chr_1(ptr %i) !prof !14 {
 ; CHECK-LABEL: @test_chr_1(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[I:%.*]], align 4
-; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[TMP0]], 3
+; CHECK-NEXT:    [[DOTFR1:%.*]] = freeze i32 [[TMP0]]
+; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[DOTFR1]], 3
 ; CHECK-NEXT:    [[TMP2:%.*]] = icmp eq i32 [[TMP1]], 3
 ; CHECK-NEXT:    br i1 [[TMP2]], label [[BB0:%.*]], label [[ENTRY_SPLIT_NONCHR:%.*]], !prof [[PROF15:![0-9]+]]
 ; CHECK:       bb0:
@@ -34,14 +35,14 @@ define void @test_chr_1(ptr %i) !prof !14 {
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB3:%.*]]
 ; CHECK:       entry.split.nonchr:
-; CHECK-NEXT:    [[TMP3:%.*]] = and i32 [[TMP0]], 1
+; CHECK-NEXT:    [[TMP3:%.*]] = and i32 [[DOTFR1]], 1
 ; CHECK-NEXT:    [[DOTNOT:%.*]] = icmp eq i32 [[TMP3]], 0
 ; CHECK-NEXT:    br i1 [[DOTNOT]], label [[BB1_NONCHR:%.*]], label [[BB0_NONCHR:%.*]], !prof [[PROF16:![0-9]+]]
 ; CHECK:       bb0.nonchr:
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB1_NONCHR]]
 ; CHECK:       bb1.nonchr:
-; CHECK-NEXT:    [[TMP4:%.*]] = and i32 [[TMP0]], 2
+; CHECK-NEXT:    [[TMP4:%.*]] = and i32 [[DOTFR1]], 2
 ; CHECK-NEXT:    [[TMP5:%.*]] = icmp eq i32 [[TMP4]], 0
 ; CHECK-NEXT:    br i1 [[TMP5]], label [[BB3]], label [[BB2_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb2.nonchr:
@@ -99,7 +100,8 @@ define void @test_chr_1_1(ptr %i) !prof !14 {
 ; CHECK-LABEL: @test_chr_1_1(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[I:%.*]], align 4
-; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[TMP0]], 7
+; CHECK-NEXT:    [[DOTFR1:%.*]] = freeze i32 [[TMP0]]
+; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[DOTFR1]], 7
 ; CHECK-NEXT:    [[TMP2:%.*]] = icmp eq i32 [[TMP1]], 7
 ; CHECK-NEXT:    br i1 [[TMP2]], label [[BB0:%.*]], label [[ENTRY_SPLIT_NONCHR:%.*]], !prof [[PROF15]]
 ; CHECK:       bb0:
@@ -107,21 +109,21 @@ define void @test_chr_1_1(ptr %i) !prof !14 {
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB5:%.*]]
 ; CHECK:       entry.split.nonchr:
-; CHECK-NEXT:    [[TMP3:%.*]] = and i32 [[TMP0]], 1
+; CHECK-NEXT:    [[TMP3:%.*]] = and i32 [[DOTFR1]], 1
 ; CHECK-NEXT:    [[DOTNOT:%.*]] = icmp eq i32 [[TMP3]], 0
 ; CHECK-NEXT:    br i1 [[DOTNOT]], label [[BB1_NONCHR:%.*]], label [[BB0_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb0.nonchr:
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB1_NONCHR]]
 ; CHECK:       bb1.nonchr:
-; CHECK-NEXT:    [[TMP4:%.*]] = and i32 [[TMP0]], 2
+; CHECK-NEXT:    [[TMP4:%.*]] = and i32 [[DOTFR1]], 2
 ; CHECK-NEXT:    [[TMP5:%.*]] = icmp eq i32 [[TMP4]], 0
 ; CHECK-NEXT:    br i1 [[TMP5]], label [[BB2_NONCHR:%.*]], label [[BB3_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb2.nonchr:
 ; CHECK-NEXT:    call void @bar()
 ; CHECK-NEXT:    br label [[BB3_NONCHR]]
 ; CHECK:       bb3.nonchr:
-; CHECK-NEXT:    [[TMP6:%.*]] = and i32 [[TMP0]], 4
+; CHECK-NEXT:    [[TMP6:%.*]] = and i32 [[DOTFR1]], 4
 ; CHECK-NEXT:    [[TMP7:%.*]] = icmp eq i32 [[TMP6]], 0
 ; CHECK-NEXT:    br i1 [[TMP7]], label [[BB5]], label [[BB4_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb4.nonchr:
@@ -185,7 +187,8 @@ define void @test_chr_2(ptr %i) !prof !14 {
 ; CHECK-LABEL: @test_chr_2(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[I:%.*]], align 4
-; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[TMP0]], 3
+; CHECK-NEXT:    [[DOTFR1:%.*]] = freeze i32 [[TMP0]]
+; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[DOTFR1]], 3
 ; CHECK-NEXT:    [[TMP2:%.*]] = icmp eq i32 [[TMP1]], 3
 ; CHECK-NEXT:    br i1 [[TMP2]], label [[BB1:%.*]], label [[ENTRY_SPLIT_NONCHR:%.*]], !prof [[PROF15]]
 ; CHECK:       bb1:
@@ -193,15 +196,15 @@ define void @test_chr_2(ptr %i) !prof !14 {
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB4:%.*]]
 ; CHECK:       entry.split.nonchr:
-; CHECK-NEXT:    [[TMP3:%.*]] = and i32 [[TMP0]], 255
+; CHECK-NEXT:    [[TMP3:%.*]] = and i32 [[DOTFR1]], 255
 ; CHECK-NEXT:    [[DOTNOT:%.*]] = icmp eq i32 [[TMP3]], 0
 ; CHECK-NEXT:    br i1 [[DOTNOT]], label [[BB4]], label [[BB0_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb0.nonchr:
-; CHECK-NEXT:    [[TMP4:%.*]] = and i32 [[TMP0]], 1
+; CHECK-NEXT:    [[TMP4:%.*]] = and i32 [[DOTFR1]], 1
 ; CHECK-NEXT:    [[TMP5:%.*]] = icmp eq i32 [[TMP4]], 0
 ; CHECK-NEXT:    br i1 [[TMP5]], label [[BB2_NONCHR:%.*]], label [[BB1_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb2.nonchr:
-; CHECK-NEXT:    [[TMP6:%.*]] = and i32 [[TMP0]], 2
+; CHECK-NEXT:    [[TMP6:%.*]] = and i32 [[DOTFR1]], 2
 ; CHECK-NEXT:    [[TMP7:%.*]] = icmp eq i32 [[TMP6]], 0
 ; CHECK-NEXT:    br i1 [[TMP7]], label [[BB4]], label [[BB3_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb3.nonchr:
@@ -278,7 +281,8 @@ define void @test_chr_3(ptr %i) !prof !14 {
 ; CHECK-LABEL: @test_chr_3(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[I:%.*]], align 4
-; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[TMP0]], 3
+; CHECK-NEXT:    [[DOTFR1:%.*]] = freeze i32 [[TMP0]]
+; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[DOTFR1]], 3
 ; CHECK-NEXT:    [[TMP2:%.*]] = icmp eq i32 [[TMP1]], 3
 ; CHECK-NEXT:    br i1 [[TMP2]], label [[BB0:%.*]], label [[ENTRY_SPLIT_NONCHR:%.*]], !prof [[PROF15]]
 ; CHECK:       bb0:
@@ -286,14 +290,14 @@ define void @test_chr_3(ptr %i) !prof !14 {
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB3:%.*]]
 ; CHECK:       entry.split.nonchr:
-; CHECK-NEXT:    [[TMP3:%.*]] = and i32 [[TMP0]], 1
+; CHECK-NEXT:    [[TMP3:%.*]] = and i32 [[DOTFR1]], 1
 ; CHECK-NEXT:    [[DOTNOT:%.*]] = icmp eq i32 [[TMP3]], 0
 ; CHECK-NEXT:    br i1 [[DOTNOT]], label [[BB1_NONCHR:%.*]], label [[BB0_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb0.nonchr:
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB1_NONCHR]]
 ; CHECK:       bb1.nonchr:
-; CHECK-NEXT:    [[TMP4:%.*]] = and i32 [[TMP0]], 2
+; CHECK-NEXT:    [[TMP4:%.*]] = and i32 [[DOTFR1]], 2
 ; CHECK-NEXT:    [[TMP5:%.*]] = icmp eq i32 [[TMP4]], 0
 ; CHECK-NEXT:    br i1 [[TMP5]], label [[BB3]], label [[BB2_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb2.nonchr:
@@ -301,7 +305,8 @@ define void @test_chr_3(ptr %i) !prof !14 {
 ; CHECK-NEXT:    br label [[BB3]]
 ; CHECK:       bb3:
 ; CHECK-NEXT:    [[TMP6:%.*]] = load i32, ptr [[I]], align 4
-; CHECK-NEXT:    [[TMP7:%.*]] = and i32 [[TMP6]], 12
+; CHECK-NEXT:    [[DOTFR2:%.*]] = freeze i32 [[TMP6]]
+; CHECK-NEXT:    [[TMP7:%.*]] = and i32 [[DOTFR2]], 12
 ; CHECK-NEXT:    [[TMP8:%.*]] = icmp eq i32 [[TMP7]], 12
 ; CHECK-NEXT:    br i1 [[TMP8]], label [[BB4:%.*]], label [[BB3_SPLIT_NONCHR:%.*]], !prof [[PROF15]]
 ; CHECK:       bb4:
@@ -309,14 +314,14 @@ define void @test_chr_3(ptr %i) !prof !14 {
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB7:%.*]]
 ; CHECK:       bb3.split.nonchr:
-; CHECK-NEXT:    [[TMP9:%.*]] = and i32 [[TMP6]], 4
-; CHECK-NEXT:    [[DOTNOT1:%.*]] = icmp eq i32 [[TMP9]], 0
-; CHECK-NEXT:    br i1 [[DOTNOT1]], label [[BB5_NONCHR:%.*]], label [[BB4_NONCHR:%.*]], !prof [[PROF16]]
+; CHECK-NEXT:    [[TMP9:%.*]] = and i32 [[DOTFR2]], 4
+; CHECK-NEXT:    [[DOTNOT3:%.*]] = icmp eq i32 [[TMP9]], 0
+; CHECK-NEXT:    br i1 [[DOTNOT3]], label [[BB5_NONCHR:%.*]], label [[BB4_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb4.nonchr:
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB5_NONCHR]]
 ; CHECK:       bb5.nonchr:
-; CHECK-NEXT:    [[TMP10:%.*]] = and i32 [[TMP6]], 8
+; CHECK-NEXT:    [[TMP10:%.*]] = and i32 [[DOTFR2]], 8
 ; CHECK-NEXT:    [[TMP11:%.*]] = icmp eq i32 [[TMP10]], 0
 ; CHECK-NEXT:    br i1 [[TMP11]], label [[BB7]], label [[BB6_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb6.nonchr:
@@ -546,17 +551,18 @@ bb3:
 define i32 @test_chr_5_1(ptr %i, i32 %sum0) !prof !14 {
 ; CHECK-LABEL: @test_chr_5_1(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[SUM0_FR:%.*]] = freeze i32 [[SUM0:%.*]]
 ; CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[I:%.*]], align 4
 ; CHECK-NEXT:    [[DOTFR1:%.*]] = freeze i32 [[TMP0]]
-; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[SUM0:%.*]], 4
+; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[SUM0_FR]], 4
 ; CHECK-NEXT:    [[TMP2:%.*]] = icmp ne i32 [[TMP1]], 0
 ; CHECK-NEXT:    [[TMP3:%.*]] = and i32 [[DOTFR1]], 11
 ; CHECK-NEXT:    [[TMP4:%.*]] = icmp eq i32 [[TMP3]], 11
-; CHECK-NEXT:    [[TMP5:%.*]] = select i1 [[TMP4]], i1 [[TMP2]], i1 false
+; CHECK-NEXT:    [[TMP5:%.*]] = and i1 [[TMP4]], [[TMP2]]
 ; CHECK-NEXT:    br i1 [[TMP5]], label [[BB0:%.*]], label [[ENTRY_SPLIT_NONCHR:%.*]], !prof [[PROF15]]
 ; CHECK:       bb0:
-; CHECK-NEXT:    [[TMP6:%.*]] = add i32 [[SUM0]], 85
-; CHECK-NEXT:    [[TMP7:%.*]] = add i32 [[SUM0]], 173
+; CHECK-NEXT:    [[TMP6:%.*]] = add i32 [[SUM0_FR]], 85
+; CHECK-NEXT:    [[TMP7:%.*]] = add i32 [[SUM0_FR]], 173
 ; CHECK-NEXT:    br label [[BB3:%.*]]
 ; CHECK:       entry.split.nonchr:
 ; CHECK-NEXT:    [[TMP8:%.*]] = and i32 [[DOTFR1]], 255
@@ -565,13 +571,13 @@ define i32 @test_chr_5_1(ptr %i, i32 %sum0) !prof !14 {
 ; CHECK:       bb0.nonchr:
 ; CHECK-NEXT:    [[TMP9:%.*]] = and i32 [[DOTFR1]], 1
 ; CHECK-NEXT:    [[TMP10:%.*]] = icmp eq i32 [[TMP9]], 0
-; CHECK-NEXT:    [[TMP11:%.*]] = add i32 [[SUM0]], 42
-; CHECK-NEXT:    [[SUM1_NONCHR:%.*]] = select i1 [[TMP10]], i32 [[SUM0]], i32 [[TMP11]], !prof [[PROF16]]
+; CHECK-NEXT:    [[TMP11:%.*]] = add i32 [[SUM0_FR]], 42
+; CHECK-NEXT:    [[SUM1_NONCHR:%.*]] = select i1 [[TMP10]], i32 [[SUM0_FR]], i32 [[TMP11]], !prof [[PROF16]]
 ; CHECK-NEXT:    [[TMP12:%.*]] = and i32 [[DOTFR1]], 2
 ; CHECK-NEXT:    [[TMP13:%.*]] = icmp eq i32 [[TMP12]], 0
 ; CHECK-NEXT:    [[TMP14:%.*]] = add i32 [[SUM1_NONCHR]], 43
 ; CHECK-NEXT:    [[SUM2_NONCHR:%.*]] = select i1 [[TMP13]], i32 [[SUM1_NONCHR]], i32 [[TMP14]], !prof [[PROF16]]
-; CHECK-NEXT:    [[TMP15:%.*]] = and i32 [[SUM0]], 4
+; CHECK-NEXT:    [[TMP15:%.*]] = and i32 [[SUM0_FR]], 4
 ; CHECK-NEXT:    [[TMP16:%.*]] = icmp eq i32 [[TMP15]], 0
 ; CHECK-NEXT:    [[TMP17:%.*]] = and i32 [[DOTFR1]], 8
 ; CHECK-NEXT:    [[TMP18:%.*]] = icmp eq i32 [[TMP17]], 0
@@ -580,7 +586,7 @@ define i32 @test_chr_5_1(ptr %i, i32 %sum0) !prof !14 {
 ; CHECK-NEXT:    [[SUM5_NONCHR:%.*]] = select i1 [[TMP16]], i32 [[SUM2_NONCHR]], i32 [[SUM4_NONCHR]], !prof [[PROF16]]
 ; CHECK-NEXT:    br label [[BB3]]
 ; CHECK:       bb3:
-; CHECK-NEXT:    [[SUM6:%.*]] = phi i32 [ [[TMP7]], [[BB0]] ], [ [[SUM0]], [[ENTRY_SPLIT_NONCHR]] ], [ [[SUM5_NONCHR]], [[BB0_NONCHR]] ]
+; CHECK-NEXT:    [[SUM6:%.*]] = phi i32 [ [[TMP7]], [[BB0]] ], [ [[SUM0_FR]], [[ENTRY_SPLIT_NONCHR]] ], [ [[SUM5_NONCHR]], [[BB0_NONCHR]] ]
 ; CHECK-NEXT:    ret i32 [[SUM6]]
 ;
 entry:
@@ -650,11 +656,12 @@ define i32 @test_chr_6(ptr %i, ptr %j, i32 %sum0) !prof !14 {
 ; CHECK-NEXT:    [[I0:%.*]] = load i32, ptr [[I:%.*]], align 4
 ; CHECK-NEXT:    [[I0_FR:%.*]] = freeze i32 [[I0]]
 ; CHECK-NEXT:    [[J0:%.*]] = load i32, ptr [[J:%.*]], align 4
-; CHECK-NEXT:    [[V9:%.*]] = and i32 [[J0]], 4
+; CHECK-NEXT:    [[J0_FR:%.*]] = freeze i32 [[J0]]
+; CHECK-NEXT:    [[V9:%.*]] = and i32 [[J0_FR]], 4
 ; CHECK-NEXT:    [[V10:%.*]] = icmp ne i32 [[V9]], 0
 ; CHECK-NEXT:    [[TMP0:%.*]] = and i32 [[I0_FR]], 10
 ; CHECK-NEXT:    [[TMP1:%.*]] = icmp eq i32 [[TMP0]], 10
-; CHECK-NEXT:    [[TMP2:%.*]] = select i1 [[TMP1]], i1 [[V10]], i1 false
+; CHECK-NEXT:    [[TMP2:%.*]] = and i1 [[TMP1]], [[V10]]
 ; CHECK-NEXT:    br i1 [[TMP2]], label [[BB0:%.*]], label [[ENTRY_SPLIT_NONCHR:%.*]], !prof [[PROF15]]
 ; CHECK:       bb0:
 ; CHECK-NEXT:    [[V8:%.*]] = add i32 [[SUM0:%.*]], 43
@@ -669,7 +676,7 @@ define i32 @test_chr_6(ptr %i, ptr %j, i32 %sum0) !prof !14 {
 ; CHECK-NEXT:    [[V4_NONCHR:%.*]] = icmp eq i32 [[V3_NONCHR]], 0
 ; CHECK-NEXT:    [[V8_NONCHR:%.*]] = add i32 [[SUM0]], 43
 ; CHECK-NEXT:    [[SUM2_NONCHR:%.*]] = select i1 [[V4_NONCHR]], i32 [[SUM0]], i32 [[V8_NONCHR]], !prof [[PROF16]]
-; CHECK-NEXT:    [[V9_NONCHR:%.*]] = and i32 [[J0]], 4
+; CHECK-NEXT:    [[V9_NONCHR:%.*]] = and i32 [[J0_FR]], 4
 ; CHECK-NEXT:    [[V10_NONCHR:%.*]] = icmp eq i32 [[V9_NONCHR]], 0
 ; CHECK-NEXT:    [[V11_NONCHR:%.*]] = and i32 [[I0_FR]], 8
 ; CHECK-NEXT:    [[V12_NONCHR:%.*]] = icmp eq i32 [[V11_NONCHR]], 0
@@ -804,7 +811,8 @@ define i32 @test_chr_7_1(ptr %i, ptr %j, i32 %sum0) !prof !14 {
 ; CHECK-NEXT:    [[I0:%.*]] = load i32, ptr [[I:%.*]], align 4
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    [[J0:%.*]] = load i32, ptr [[J:%.*]], align 4
-; CHECK-NEXT:    [[TMP0:%.*]] = and i32 [[J0]], 12
+; CHECK-NEXT:    [[J0_FR:%.*]] = freeze i32 [[J0]]
+; CHECK-NEXT:    [[TMP0:%.*]] = and i32 [[J0_FR]], 12
 ; CHECK-NEXT:    [[TMP1:%.*]] = icmp eq i32 [[TMP0]], 12
 ; CHECK-NEXT:    br i1 [[TMP1]], label [[BB0:%.*]], label [[ENTRY_SPLIT_NONCHR:%.*]], !prof [[PROF15]]
 ; CHECK:       bb0:
@@ -812,14 +820,14 @@ define i32 @test_chr_7_1(ptr %i, ptr %j, i32 %sum0) !prof !14 {
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB3:%.*]]
 ; CHECK:       entry.split.nonchr:
-; CHECK-NEXT:    [[V9:%.*]] = and i32 [[J0]], 4
+; CHECK-NEXT:    [[V9:%.*]] = and i32 [[J0_FR]], 4
 ; CHECK-NEXT:    [[V10_NOT:%.*]] = icmp eq i32 [[V9]], 0
 ; CHECK-NEXT:    br i1 [[V10_NOT]], label [[BB1_NONCHR:%.*]], label [[BB0_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb0.nonchr:
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB1_NONCHR]]
 ; CHECK:       bb1.nonchr:
-; CHECK-NEXT:    [[V11_NONCHR:%.*]] = and i32 [[J0]], 8
+; CHECK-NEXT:    [[V11_NONCHR:%.*]] = and i32 [[J0_FR]], 8
 ; CHECK-NEXT:    [[V12_NONCHR:%.*]] = icmp eq i32 [[V11_NONCHR]], 0
 ; CHECK-NEXT:    br i1 [[V12_NONCHR]], label [[BB3]], label [[BB2_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb2.nonchr:
@@ -944,7 +952,8 @@ define i32 @test_chr_9(ptr %i, ptr %j) !prof !14 {
 ; CHECK-LABEL: @test_chr_9(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[I:%.*]], align 4
-; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[TMP0]], 3
+; CHECK-NEXT:    [[DOTFR1:%.*]] = freeze i32 [[TMP0]]
+; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[DOTFR1]], 3
 ; CHECK-NEXT:    [[TMP2:%.*]] = icmp eq i32 [[TMP1]], 3
 ; CHECK-NEXT:    br i1 [[TMP2]], label [[BB0:%.*]], label [[ENTRY_SPLIT_NONCHR:%.*]], !prof [[PROF15]]
 ; CHECK:       bb0:
@@ -953,14 +962,14 @@ define i32 @test_chr_9(ptr %i, ptr %j) !prof !14 {
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB3:%.*]]
 ; CHECK:       entry.split.nonchr:
-; CHECK-NEXT:    [[TMP4:%.*]] = and i32 [[TMP0]], 1
+; CHECK-NEXT:    [[TMP4:%.*]] = and i32 [[DOTFR1]], 1
 ; CHECK-NEXT:    [[DOTNOT:%.*]] = icmp eq i32 [[TMP4]], 0
 ; CHECK-NEXT:    br i1 [[DOTNOT]], label [[BB1_NONCHR:%.*]], label [[BB0_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb0.nonchr:
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB1_NONCHR]]
 ; CHECK:       bb1.nonchr:
-; CHECK-NEXT:    [[TMP5:%.*]] = and i32 [[TMP0]], 2
+; CHECK-NEXT:    [[TMP5:%.*]] = and i32 [[DOTFR1]], 2
 ; CHECK-NEXT:    [[TMP6:%.*]] = icmp eq i32 [[TMP5]], 0
 ; CHECK-NEXT:    br i1 [[TMP6]], label [[BB3]], label [[BB2_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb2.nonchr:
@@ -968,7 +977,7 @@ define i32 @test_chr_9(ptr %i, ptr %j) !prof !14 {
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB3]]
 ; CHECK:       bb3:
-; CHECK-NEXT:    [[TMP8:%.*]] = phi i32 [ [[TMP3]], [[BB0]] ], [ [[TMP0]], [[BB1_NONCHR]] ], [ [[TMP7]], [[BB2_NONCHR]] ]
+; CHECK-NEXT:    [[TMP8:%.*]] = phi i32 [ [[TMP3]], [[BB0]] ], [ [[DOTFR1]], [[BB1_NONCHR]] ], [ [[TMP7]], [[BB2_NONCHR]] ]
 ; CHECK-NEXT:    ret i32 [[TMP8]]
 ;
 entry:
@@ -1025,7 +1034,8 @@ define i32 @test_chr_10(ptr %i, ptr %j) !prof !14 {
 ; CHECK-LABEL: @test_chr_10(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[I:%.*]], align 4
-; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[TMP0]], 3
+; CHECK-NEXT:    [[DOTFR1:%.*]] = freeze i32 [[TMP0]]
+; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[DOTFR1]], 3
 ; CHECK-NEXT:    [[TMP2:%.*]] = icmp eq i32 [[TMP1]], 3
 ; CHECK-NEXT:    br i1 [[TMP2]], label [[BB0:%.*]], label [[ENTRY_SPLIT_NONCHR:%.*]], !prof [[PROF15]]
 ; CHECK:       bb0:
@@ -1034,7 +1044,7 @@ define i32 @test_chr_10(ptr %i, ptr %j) !prof !14 {
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB3:%.*]]
 ; CHECK:       entry.split.nonchr:
-; CHECK-NEXT:    [[TMP4:%.*]] = and i32 [[TMP0]], 1
+; CHECK-NEXT:    [[TMP4:%.*]] = and i32 [[DOTFR1]], 1
 ; CHECK-NEXT:    [[DOTNOT:%.*]] = icmp eq i32 [[TMP4]], 0
 ; CHECK-NEXT:    br i1 [[DOTNOT]], label [[BB1_NONCHR:%.*]], label [[BB0_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb0.nonchr:
@@ -1042,7 +1052,7 @@ define i32 @test_chr_10(ptr %i, ptr %j) !prof !14 {
 ; CHECK-NEXT:    br label [[BB1_NONCHR]]
 ; CHECK:       bb1.nonchr:
 ; CHECK-NEXT:    [[TMP5:%.*]] = load i32, ptr [[J]], align 4
-; CHECK-NEXT:    [[TMP6:%.*]] = and i32 [[TMP0]], 2
+; CHECK-NEXT:    [[TMP6:%.*]] = and i32 [[DOTFR1]], 2
 ; CHECK-NEXT:    [[TMP7:%.*]] = icmp eq i32 [[TMP6]], 0
 ; CHECK-NEXT:    br i1 [[TMP7]], label [[BB3]], label [[BB2_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb2.nonchr:
@@ -1115,14 +1125,16 @@ define void @test_chr_11(ptr %i, i32 %x) !prof !14 {
 ; CHECK-LABEL: @test_chr_11(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[I:%.*]], align 4
-; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[TMP0]], 1
+; CHECK-NEXT:    [[DOTFR1:%.*]] = freeze i32 [[TMP0]]
+; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[DOTFR1]], 1
 ; CHECK-NEXT:    [[TMP2:%.*]] = icmp ne i32 [[TMP1]], 0
-; CHECK-NEXT:    [[CONV:%.*]] = sitofp i32 [[TMP0]] to double
+; CHECK-NEXT:    [[CONV:%.*]] = sitofp i32 [[DOTFR1]] to double
 ; CHECK-NEXT:    [[DIV:%.*]] = fdiv double 1.000000e+00, [[CONV]]
 ; CHECK-NEXT:    [[MUL16:%.*]] = fmul double [[DIV]], [[CONV]]
 ; CHECK-NEXT:    [[CONV717:%.*]] = fptosi double [[MUL16]] to i32
-; CHECK-NEXT:    [[CMP18:%.*]] = icmp sgt i32 [[CONV717]], 0
-; CHECK-NEXT:    [[TMP3:%.*]] = select i1 [[TMP2]], i1 [[CMP18]], i1 false
+; CHECK-NEXT:    [[CONV717_FR:%.*]] = freeze i32 [[CONV717]]
+; CHECK-NEXT:    [[CMP18:%.*]] = icmp sgt i32 [[CONV717_FR]], 0
+; CHECK-NEXT:    [[TMP3:%.*]] = and i1 [[TMP2]], [[CMP18]]
 ; CHECK-NEXT:    br i1 [[TMP3]], label [[BB0:%.*]], label [[ENTRY_SPLIT_NONCHR:%.*]], !prof [[PROF15]]
 ; CHECK:       bb0:
 ; CHECK-NEXT:    call void @foo()
@@ -1134,7 +1146,7 @@ define void @test_chr_11(ptr %i, i32 %x) !prof !14 {
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB1_NONCHR]]
 ; CHECK:       bb1.nonchr:
-; CHECK-NEXT:    [[CONV_NONCHR:%.*]] = sitofp i32 [[TMP0]] to double
+; CHECK-NEXT:    [[CONV_NONCHR:%.*]] = sitofp i32 [[DOTFR1]] to double
 ; CHECK-NEXT:    [[DIV_NONCHR:%.*]] = fdiv double 1.000000e+00, [[CONV_NONCHR]]
 ; CHECK-NEXT:    [[MUL16_NONCHR:%.*]] = fmul double [[DIV_NONCHR]], [[CONV_NONCHR]]
 ; CHECK-NEXT:    [[CONV717_NONCHR:%.*]] = fptosi double [[MUL16_NONCHR]] to i32
@@ -1177,22 +1189,23 @@ define i32 @test_chr_12(ptr %i, i32 %sum0) !prof !14 {
 ; CHECK-LABEL: @test_chr_12(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[I:%.*]], align 4
-; CHECK-NEXT:    [[DOTFR1:%.*]] = freeze i32 [[TMP0]]
-; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[DOTFR1]], 255
+; CHECK-NEXT:    [[DOTFR2:%.*]] = freeze i32 [[TMP0]]
+; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[DOTFR2]], 255
 ; CHECK-NEXT:    [[TMP2:%.*]] = icmp eq i32 [[TMP1]], 0
 ; CHECK-NEXT:    br i1 [[TMP2]], label [[BB3:%.*]], label [[BB0:%.*]], !prof [[PROF16]]
 ; CHECK:       bb0:
-; CHECK-NEXT:    [[TMP3:%.*]] = and i32 [[DOTFR1]], 1
+; CHECK-NEXT:    [[TMP3:%.*]] = and i32 [[DOTFR2]], 1
 ; CHECK-NEXT:    [[TMP4:%.*]] = icmp eq i32 [[TMP3]], 0
 ; CHECK-NEXT:    [[TMP5:%.*]] = add i32 [[SUM0:%.*]], 42
 ; CHECK-NEXT:    [[SUM1:%.*]] = select i1 [[TMP4]], i32 [[SUM0]], i32 [[TMP5]], !prof [[PROF16]]
-; CHECK-NEXT:    [[TMP6:%.*]] = and i32 [[DOTFR1]], 2
+; CHECK-NEXT:    [[TMP6:%.*]] = and i32 [[DOTFR2]], 2
 ; CHECK-NEXT:    [[TMP7:%.*]] = icmp eq i32 [[TMP6]], 0
 ; CHECK-NEXT:    [[TMP8:%.*]] = add i32 [[SUM1]], 43
 ; CHECK-NEXT:    [[SUM2:%.*]] = select i1 [[TMP7]], i32 [[SUM1]], i32 [[TMP8]], !prof [[PROF16]]
 ; CHECK-NEXT:    [[TMP9:%.*]] = load i32, ptr [[I]], align 4
-; CHECK-NEXT:    [[TMP10:%.*]] = icmp ne i32 [[TMP9]], 0
-; CHECK-NEXT:    [[TMP11:%.*]] = and i32 [[DOTFR1]], 8
+; CHECK-NEXT:    [[DOTFR:%.*]] = freeze i32 [[TMP9]]
+; CHECK-NEXT:    [[TMP10:%.*]] = icmp ne i32 [[DOTFR]], 0
+; CHECK-NEXT:    [[TMP11:%.*]] = and i32 [[DOTFR2]], 8
 ; CHECK-NEXT:    [[TMP12:%.*]] = icmp ne i32 [[TMP11]], 0
 ; CHECK-NEXT:    [[TMP13:%.*]] = and i1 [[TMP10]], [[TMP12]]
 ; CHECK-NEXT:    br i1 [[TMP13]], label [[BB1:%.*]], label [[BB0_SPLIT_NONCHR:%.*]], !prof [[PROF15]]
@@ -1202,7 +1215,7 @@ define i32 @test_chr_12(ptr %i, i32 %sum0) !prof !14 {
 ; CHECK:       bb0.split.nonchr:
 ; CHECK-NEXT:    br i1 [[TMP10]], label [[BB1_NONCHR:%.*]], label [[BB3]], !prof [[PROF18]]
 ; CHECK:       bb1.nonchr:
-; CHECK-NEXT:    [[TMP15:%.*]] = and i32 [[DOTFR1]], 8
+; CHECK-NEXT:    [[TMP15:%.*]] = and i32 [[DOTFR2]], 8
 ; CHECK-NEXT:    [[TMP16:%.*]] = icmp eq i32 [[TMP15]], 0
 ; CHECK-NEXT:    [[SUM4_NONCHR_V:%.*]] = select i1 [[TMP16]], i32 44, i32 88, !prof [[PROF16]]
 ; CHECK-NEXT:    [[SUM4_NONCHR:%.*]] = add i32 [[SUM2]], [[SUM4_NONCHR_V]]
@@ -1285,11 +1298,12 @@ define i32 @test_chr_14(ptr %i, ptr %j, i32 %sum0, i1 %pred, i32 %z) !prof !14 {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[Z_FR:%.*]] = freeze i32 [[Z:%.*]]
 ; CHECK-NEXT:    [[I0:%.*]] = load i32, ptr [[I:%.*]], align 4
-; CHECK-NEXT:    [[V1:%.*]] = icmp eq i32 [[Z_FR]], 1
-; CHECK-NEXT:    br i1 [[V1]], label [[BB1:%.*]], label [[ENTRY_SPLIT_NONCHR:%.*]], !prof [[PROF15]]
+; CHECK-NEXT:    [[V1_NOT:%.*]] = icmp eq i32 [[Z_FR]], 1
+; CHECK-NEXT:    br i1 [[V1_NOT]], label [[BB1:%.*]], label [[ENTRY_SPLIT_NONCHR:%.*]], !prof [[PROF15]]
 ; CHECK:       entry.split.nonchr:
+; CHECK-NEXT:    [[PRED_FR:%.*]] = freeze i1 [[PRED:%.*]]
 ; CHECK-NEXT:    [[V0:%.*]] = icmp eq i32 [[Z_FR]], 0
-; CHECK-NEXT:    [[V3_NONCHR:%.*]] = and i1 [[V0]], [[PRED:%.*]]
+; CHECK-NEXT:    [[V3_NONCHR:%.*]] = and i1 [[V0]], [[PRED_FR]]
 ; CHECK-NEXT:    br i1 [[V3_NONCHR]], label [[BB0_NONCHR:%.*]], label [[BB1]], !prof [[PROF16]]
 ; CHECK:       bb0.nonchr:
 ; CHECK-NEXT:    call void @foo()
@@ -1476,29 +1490,30 @@ define i32 @test_chr_16(ptr %i) !prof !14 {
 ; CHECK-LABEL: @test_chr_16(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[I:%.*]], align 4
-; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[TMP0]], 3
+; CHECK-NEXT:    [[DOTFR1:%.*]] = freeze i32 [[TMP0]]
+; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[DOTFR1]], 3
 ; CHECK-NEXT:    [[TMP2:%.*]] = icmp eq i32 [[TMP1]], 3
 ; CHECK-NEXT:    br i1 [[TMP2]], label [[BB0:%.*]], label [[ENTRY_SPLIT_NONCHR:%.*]], !prof [[PROF15]]
 ; CHECK:       bb0:
 ; CHECK-NEXT:    call void @foo()
-; CHECK-NEXT:    [[V40:%.*]] = add i32 [[TMP0]], 44
-; CHECK-NEXT:    [[V41:%.*]] = add i32 [[TMP0]], 99
+; CHECK-NEXT:    [[V40:%.*]] = add i32 [[DOTFR1]], 44
+; CHECK-NEXT:    [[V41:%.*]] = add i32 [[DOTFR1]], 99
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB3:%.*]]
 ; CHECK:       entry.split.nonchr:
-; CHECK-NEXT:    [[TMP3:%.*]] = and i32 [[TMP0]], 1
+; CHECK-NEXT:    [[TMP3:%.*]] = and i32 [[DOTFR1]], 1
 ; CHECK-NEXT:    [[DOTNOT:%.*]] = icmp eq i32 [[TMP3]], 0
 ; CHECK-NEXT:    br i1 [[DOTNOT]], label [[BB1_NONCHR:%.*]], label [[BB0_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb0.nonchr:
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB1_NONCHR]]
 ; CHECK:       bb1.nonchr:
-; CHECK-NEXT:    [[V40_NONCHR:%.*]] = add i32 [[TMP0]], 44
-; CHECK-NEXT:    [[TMP4:%.*]] = and i32 [[TMP0]], 2
+; CHECK-NEXT:    [[V40_NONCHR:%.*]] = add i32 [[DOTFR1]], 44
+; CHECK-NEXT:    [[TMP4:%.*]] = and i32 [[DOTFR1]], 2
 ; CHECK-NEXT:    [[TMP5:%.*]] = icmp eq i32 [[TMP4]], 0
 ; CHECK-NEXT:    br i1 [[TMP5]], label [[BB3]], label [[BB2_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb2.nonchr:
-; CHECK-NEXT:    [[V41_NONCHR:%.*]] = add i32 [[TMP0]], 99
+; CHECK-NEXT:    [[V41_NONCHR:%.*]] = add i32 [[DOTFR1]], 99
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB3]]
 ; CHECK:       bb3:
@@ -1915,15 +1930,18 @@ bb4:
 define i32 @test_chr_21(i64 %i, i64 %k, i64 %j) !prof !14 {
 ; CHECK-LABEL: @test_chr_21(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[CMP0:%.*]] = icmp ne i64 [[J:%.*]], [[K:%.*]]
-; CHECK-NEXT:    [[CMP3:%.*]] = icmp ne i64 [[J]], [[I:%.*]]
-; CHECK-NEXT:    [[CMP_I:%.*]] = icmp ne i64 [[I]], 86
-; CHECK-NEXT:    [[TMP0:%.*]] = select i1 [[CMP0]], i1 [[CMP3]], i1 false
-; CHECK-NEXT:    [[TMP1:%.*]] = select i1 [[TMP0]], i1 [[CMP_I]], i1 false
+; CHECK-NEXT:    [[J_FR:%.*]] = freeze i64 [[J:%.*]]
+; CHECK-NEXT:    [[I_FR:%.*]] = freeze i64 [[I:%.*]]
+; CHECK-NEXT:    [[K_FR:%.*]] = freeze i64 [[K:%.*]]
+; CHECK-NEXT:    [[CMP0:%.*]] = icmp ne i64 [[J_FR]], [[K_FR]]
+; CHECK-NEXT:    [[CMP3:%.*]] = icmp ne i64 [[I_FR]], [[J_FR]]
+; CHECK-NEXT:    [[CMP_I:%.*]] = icmp ne i64 [[I_FR]], 86
+; CHECK-NEXT:    [[TMP0:%.*]] = and i1 [[CMP0]], [[CMP3]]
+; CHECK-NEXT:    [[TMP1:%.*]] = and i1 [[TMP0]], [[CMP_I]]
 ; CHECK-NEXT:    br i1 [[TMP1]], label [[BB1:%.*]], label [[ENTRY_SPLIT_NONCHR:%.*]], !prof [[PROF15]]
 ; CHECK:       bb1:
-; CHECK-NEXT:    [[CMP2:%.*]] = icmp ne i64 [[I]], 2
-; CHECK-NEXT:    switch i64 [[I]], label [[BB2:%.*]] [
+; CHECK-NEXT:    [[CMP2:%.*]] = icmp ne i64 [[I_FR]], 2
+; CHECK-NEXT:    switch i64 [[I_FR]], label [[BB2:%.*]] [
 ; CHECK-NEXT:    i64 2, label [[BB3_NONCHR2:%.*]]
 ; CHECK-NEXT:    i64 86, label [[BB2_NONCHR1:%.*]]
 ; CHECK-NEXT:    ], !prof [[PROF19:![0-9]+]]
@@ -1946,13 +1964,13 @@ define i32 @test_chr_21(i64 %i, i64 %k, i64 %j) !prof !14 {
 ; CHECK:       entry.split.nonchr:
 ; CHECK-NEXT:    br i1 [[CMP0]], label [[BB1_NONCHR:%.*]], label [[BB10]], !prof [[PROF18]]
 ; CHECK:       bb1.nonchr:
-; CHECK-NEXT:    [[CMP2_NONCHR:%.*]] = icmp eq i64 [[I]], 2
+; CHECK-NEXT:    [[CMP2_NONCHR:%.*]] = icmp eq i64 [[I_FR]], 2
 ; CHECK-NEXT:    br i1 [[CMP2_NONCHR]], label [[BB3_NONCHR:%.*]], label [[BB2_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb3.nonchr:
-; CHECK-NEXT:    [[CMP_I_NONCHR:%.*]] = icmp eq i64 [[I]], 86
+; CHECK-NEXT:    [[CMP_I_NONCHR:%.*]] = icmp eq i64 [[I_FR]], 86
 ; CHECK-NEXT:    br i1 [[CMP_I_NONCHR]], label [[BB6_NONCHR:%.*]], label [[BB4_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb6.nonchr:
-; CHECK-NEXT:    [[CMP3_NONCHR:%.*]] = icmp eq i64 [[J]], [[I]]
+; CHECK-NEXT:    [[CMP3_NONCHR:%.*]] = icmp eq i64 [[J_FR]], [[I_FR]]
 ; CHECK-NEXT:    br i1 [[CMP3_NONCHR]], label [[BB8_NONCHR:%.*]], label [[BB7_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb8.nonchr:
 ; CHECK-NEXT:    br i1 [[CMP_I_NONCHR]], label [[BB10]], label [[BB9_NONCHR:%.*]], !prof [[PROF16]]
@@ -2350,7 +2368,8 @@ bb0:
 define i64 @test_chr_23(i64 %v0) !prof !14 {
 ; CHECK-LABEL: @test_chr_23(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[TMP0:%.*]] = mul i64 [[V0:%.*]], 50
+; CHECK-NEXT:    [[V0_FR:%.*]] = freeze i64 [[V0:%.*]]
+; CHECK-NEXT:    [[TMP0:%.*]] = mul i64 [[V0_FR]], 50
 ; CHECK-NEXT:    [[V10_NOT:%.*]] = icmp eq i64 [[TMP0]], -50
 ; CHECK-NEXT:    ret i64 99
 ;
@@ -2599,7 +2618,8 @@ define void @test_chr_with_bbs_address_taken2(ptr %i) !prof !14 {
 ; CHECK-LABEL: @test_chr_with_bbs_address_taken2(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[I:%.*]], align 4
-; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[TMP0]], 3
+; CHECK-NEXT:    [[DOTFR1:%.*]] = freeze i32 [[TMP0]]
+; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[DOTFR1]], 3
 ; CHECK-NEXT:    [[TMP2:%.*]] = icmp eq i32 [[TMP1]], 3
 ; CHECK-NEXT:    br i1 [[TMP2]], label [[BB0:%.*]], label [[ENTRY_SPLIT_NONCHR:%.*]], !prof [[PROF15]]
 ; CHECK:       bb0:
@@ -2607,14 +2627,14 @@ define void @test_chr_with_bbs_address_taken2(ptr %i) !prof !14 {
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB6:%.*]]
 ; CHECK:       entry.split.nonchr:
-; CHECK-NEXT:    [[TMP3:%.*]] = and i32 [[TMP0]], 1
+; CHECK-NEXT:    [[TMP3:%.*]] = and i32 [[DOTFR1]], 1
 ; CHECK-NEXT:    [[DOTNOT:%.*]] = icmp eq i32 [[TMP3]], 0
 ; CHECK-NEXT:    br i1 [[DOTNOT]], label [[BB1_NONCHR:%.*]], label [[BB0_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb0.nonchr:
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    br label [[BB1_NONCHR]]
 ; CHECK:       bb1.nonchr:
-; CHECK-NEXT:    [[TMP4:%.*]] = and i32 [[TMP0]], 2
+; CHECK-NEXT:    [[TMP4:%.*]] = and i32 [[DOTFR1]], 2
 ; CHECK-NEXT:    [[TMP5:%.*]] = icmp eq i32 [[TMP4]], 0
 ; CHECK-NEXT:    br i1 [[TMP5]], label [[BB6]], label [[BB2_NONCHR:%.*]], !prof [[PROF16]]
 ; CHECK:       bb2.nonchr:
diff --git a/llvm/test/Transforms/PGOProfile/chr_coro.ll b/llvm/test/Transforms/PGOProfile/chr_coro.ll
index 233948fa4a8fb..12654eccd5689 100644
--- a/llvm/test/Transforms/PGOProfile/chr_coro.ll
+++ b/llvm/test/Transforms/PGOProfile/chr_coro.ll
@@ -36,26 +36,38 @@ define ptr @test_chr_with_coro_id(ptr %i) !prof !14 {
 ; CHECK-LABEL: @test_chr_with_coro_id(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[I:%.*]], align 4
-; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[TMP0]], 3
+; CHECK-NEXT:    [[DOTFR1:%.*]] = freeze i32 [[TMP0]]
+; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[DOTFR1]], 3
 ; CHECK-NEXT:    [[TMP2:%.*]] = icmp eq i32 [[TMP1]], 3
-; CHECK-NEXT:    br i1 [[TMP2]], label %[[BB0:.*]], label %[[ENTRY_SPLIT_NONCHR:.*]], !prof !15
-; CHECK:       [[BB0]]:
+; CHECK-NEXT:    br i1 [[TMP2]], label [[BB0:%.*]], label [[ENTRY_SPLIT_NONCHR:%.*]], !prof [[PROF15:![0-9]+]]
+; CHECK:       bb0:
 ; CHECK-NEXT:    call void @foo()
-; CHECK-NEXT:    br label %[[BB_CORO_ID:.*]]
+; CHECK-NEXT:    br label [[BB_CORO_ID:%.*]]
+; CHECK:       entry.split.nonchr:
+; CHECK-NEXT:    [[TMP3:%.*]] = and i32 [[DOTFR1]], 1
+; CHECK-NEXT:    [[DOTNOT:%.*]] = icmp eq i32 [[TMP3]], 0
+; CHECK-NEXT:    br i1 [[DOTNOT]], label [[BB1_NONCHR:%.*]], label [[BB0_NONCHR:%.*]], !prof [[PROF16:![0-9]+]]
+; CHECK:       bb0.nonchr:
+; CHECK-NEXT:    call void @foo()
+; CHECK-NEXT:    br label [[BB1_NONCHR]]
 ; CHECK:       bb1.nonchr:
-; CHECK-NEXT:    [[TMP4:%.*]] = and i32 [[TMP0]], 2
+; CHECK-NEXT:    [[TMP4:%.*]] = and i32 [[DOTFR1]], 2
 ; CHECK-NEXT:    [[TMP5:%.*]] = icmp eq i32 [[TMP4]], 0
-; CHECK-NEXT:    br i1 [[TMP5]], label %[[BB2_NONCHR:.*]], label %[[BB_CORO_ID]], !prof !16
-; CHECK:       [[BB2_NONCHR]]:
+; CHECK-NEXT:    br i1 [[TMP5]], label [[BB2_NONCHR:%.*]], label [[BB_CORO_ID]], !prof [[PROF16]]
+; CHECK:       bb2.nonchr:
 ; CHECK-NEXT:    call void @foo()
-; CHECK-NEXT:    br label %[[BB_CORO_ID]]
-; CHECK:       [[BB_CORO_ID]]:
-; CHECK-NEXT:    [[ID:%.*]] = call token @llvm.coro.id
+; CHECK-NEXT:    br label [[BB_CORO_ID]]
+; CHECK:       bb.coro.id:
+; CHECK-NEXT:    [[ID:%.*]] = call token @llvm.coro.id(i32 0, ptr null, ptr null, ptr nonnull @f.resumers)
 ; CHECK-NEXT:    [[NEED_DYN_ALLOC:%.*]] = call i1 @llvm.coro.alloc(token [[ID]])
-; CHECK-NEXT:    br i1 [[NEED_DYN_ALLOC]], label %[[BB_CORO_DYN_ALLOC:.*]], label %[[BB_CORO_BEGIN:.*]]
-; CHECK:       [[BB_CORO_BEGIN]]:
-; CHECK-NEXT:    [[PHI:%.*]] = phi ptr [ null, %[[BB_CORO_ID]] ], [ %alloc, %[[BB_CORO_DYN_ALLOC]] ]
+; CHECK-NEXT:    br i1 [[NEED_DYN_ALLOC]], label [[BB_CORO_DYN_ALLOC:%.*]], label [[BB_CORO_BEGIN:%.*]]
+; CHECK:       bb.coro.dyn.alloc:
+; CHECK-NEXT:    [[ALLOC:%.*]] = call ptr @malloc(i32 24)
+; CHECK-NEXT:    br label [[BB_CORO_BEGIN]]
+; CHECK:       bb.coro.begin:
+; CHECK-NEXT:    [[PHI:%.*]] = phi ptr [ null, [[BB_CORO_ID]] ], [ [[ALLOC]], [[BB_CORO_DYN_ALLOC]] ]
 ; CHECK-NEXT:    [[HDL:%.*]] = call noalias nonnull ptr @llvm.coro.begin(token [[ID]], ptr [[PHI]])
+; CHECK-NEXT:    ret ptr [[HDL]]
 ;
 entry:
   %0 = load i32, ptr %i
diff --git a/llvm/test/Transforms/PGOProfile/memprof.ll b/llvm/test/Transforms/PGOProfile/memprof.ll
index a000453628d78..7296de87e5a63 100644
--- a/llvm/test/Transforms/PGOProfile/memprof.ll
+++ b/llvm/test/Transforms/PGOProfile/memprof.ll
@@ -7,87 +7,8 @@
 ; REQUIRES: x86_64-linux
 
 ;; TODO: Use text profile inputs once that is available for memprof.
-
-;; The input IR and raw profiles have been generated from the following source:
-;;
-;; #include <stdlib.h>
-;; #include <string.h>
-;; #include <unistd.h>
-;; char *foo() {
-;;   return new char[10];
-;; }
-;; char *foo2() {
-;;   return foo();
-;; }
-;; char *bar() {
-;;   return foo2();
-;; }
-;; char *baz() {
-;;   return foo2();
-;; }
-;; char *recurse(unsigned n) {
-;;   if (!n)
-;;     return foo();
-;;   return recurse(n-1);
-;; }
-;; int main(int argc, char **argv) {
-;;   // Test allocations with different combinations of stack contexts and
-;;   // coldness (based on lifetime, since they are all accessed a single time
-;;   // per byte via the memset).
-;;   char *a = new char[10];
-;;   char *b = new char[10];
-;;   char *c = foo();
-;;   char *d = foo();
-;;   char *e = bar();
-;;   char *f = baz();
-;;   memset(a, 0, 10);
-;;   memset(b, 0, 10);
-;;   memset(c, 0, 10);
-;;   memset(d, 0, 10);
-;;   memset(e, 0, 10);
-;;   memset(f, 0, 10);
-;;   // a and c have short lifetimes
-;;   delete[] a;
-;;   delete[] c;
-;;   // b, d, e, and f have long lifetimes and will be detected as cold by default.
-;;   sleep(200);
-;;   delete[] b;
-;;   delete[] d;
-;;   delete[] e;
-;;   delete[] f;
-;;   // Loop ensures the two calls to recurse have stack contexts that only differ
-;;   // in one level of recursion. We should get two stack contexts reflecting the
-;;   // different levels of recursion and different allocation behavior (since the
-;;   // first has a very long lifetime and the second has a short lifetime).
-;;   for (unsigned i = 0; i < 2; i++) {
-;;     char *g = recurse(i + 3);
-;;     memset(g, 0, 10);
-;;     if (!i)
-;;       sleep(200);
-;;     delete[] g;
-;;   }
-;;   return 0;
-;; }
-;;
-;; The following commands were used to compile the source to instrumented
-;; executables and collect raw binary format profiles:
-;;
-;; # Collect memory profile:
-;; $ clang++ -fuse-ld=lld -no-pie -Wl,--no-rosegment -gmlt \
-;; 	-fdebug-info-for-profiling -mno-omit-leaf-frame-pointer \
-;;	-fno-omit-frame-pointer -fno-optimize-sibling-calls -m64 -Wl,-build-id \
-;; 	memprof.cc -o memprof.exe -fmemory-profile
-;; $ env MEMPROF_OPTIONS=log_path=stdout ./memprof.exe > memprof.memprofraw
-;;
-;; # Collect IR PGO profile:
-;; $ clang++ -fuse-ld=lld -no-pie -Wl,--no-rosegment -gmlt \
-;; 	-fdebug-info-for-profiling -mno-omit-leaf-frame-pointer \
-;;	-fno-omit-frame-pointer -fno-optimize-sibling-calls -m64 -Wl,-build-id \
-;; 	memprof.cc -o pgo.exe -fprofile-generate=.
-;; $ ./pgo.exe
-;; $ mv default_*.profraw memprof_pgo.profraw
-;;
-;; # Generate below LLVM IR for use in matching:
+;; # To update the Inputs below, run Inputs/update_memprof_inputs.sh.
+;; # To generate below LLVM IR for use in matching:
 ;; $ clang++ -gmlt -fdebug-info-for-profiling -fno-omit-frame-pointer \
 ;;	-fno-optimize-sibling-calls memprof.cc -S -emit-llvm
 
diff --git a/llvm/test/Transforms/PhaseOrdering/dae-dce.ll b/llvm/test/Transforms/PhaseOrdering/dae-dce.ll
new file mode 100644
index 0000000000000..7ff3c5dc5536f
--- /dev/null
+++ b/llvm/test/Transforms/PhaseOrdering/dae-dce.ll
@@ -0,0 +1,52 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -S -passes='default<O3>' < %s | FileCheck %s --check-prefixes=CHECK,DEFAULT
+; RUN: opt -S -passes='lto<O3>' < %s | FileCheck %s --check-prefixes=CHECK,LTO
+
+declare void @llvm.trap()
+
+define void @do_trap(ptr %ptr) {
+; CHECK-LABEL: @do_trap(
+; CHECK-NEXT:    tail call void @llvm.trap()
+; CHECK-NEXT:    unreachable
+;
+  call void @llvm.trap()
+  unreachable
+}
+
+define internal void @capture_and_trap(ptr %ptr) noinline {
+; CHECK-LABEL: @capture_and_trap(
+; CHECK-NEXT:    tail call void @llvm.trap()
+; CHECK-NEXT:    unreachable
+;
+  %alloca = alloca ptr, align 4
+  store ptr %ptr, ptr %alloca, align 4
+  call void @do_trap(ptr noundef nonnull %alloca)
+  unreachable
+}
+
+define internal void @dead_fn1() {
+  ret void
+}
+
+define internal void @dead_fn2() {
+  ret void
+}
+
+define void @test(i1 %c) {
+; CHECK-LABEL: @test(
+; CHECK-NEXT:    tail call fastcc void @capture_and_trap()
+; CHECK-NEXT:    unreachable
+;
+  br i1 %c, label %if, label %else
+
+if:
+  call void @capture_and_trap(ptr @dead_fn1)
+  unreachable
+
+else:
+  call void @capture_and_trap(ptr @dead_fn2)
+  unreachable
+}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; DEFAULT: {{.*}}
+; LTO: {{.*}}
diff --git a/llvm/test/Transforms/PhaseOrdering/func-attrs.ll b/llvm/test/Transforms/PhaseOrdering/func-attrs.ll
index 64b33e794ed8e..187e32afe00a3 100644
--- a/llvm/test/Transforms/PhaseOrdering/func-attrs.ll
+++ b/llvm/test/Transforms/PhaseOrdering/func-attrs.ll
@@ -16,7 +16,7 @@ define internal i32 @h2(i32 %a, i32 %b) {
 }
 
 define void @f(i32 %a, i32 %b) noinline {
-; CHECK: Function Attrs: noinline
+; CHECK: Function Attrs: mustprogress nofree noinline norecurse nosync nounwind willreturn memory(none)
 ; CHECK-LABEL: @f(
 ; CHECK-NEXT:  end:
 ; CHECK-NEXT:    ret void
diff --git a/llvm/test/Transforms/SCCP/ip-add-range-to-call.ll b/llvm/test/Transforms/SCCP/ip-add-range-to-call.ll
index cc5c93fa4d4c9..64c1b9020a054 100644
--- a/llvm/test/Transforms/SCCP/ip-add-range-to-call.ll
+++ b/llvm/test/Transforms/SCCP/ip-add-range-to-call.ll
@@ -4,7 +4,7 @@
 ; Test 1.
 ; Both arguments and return value of @callee can be tracked. The inferred range
 ; can be added to call sites.
-define internal noundef i32 @callee(i32 %x) {
+define internal i32 @callee(i32 %x) {
 ; CHECK-LABEL: @callee(
 ; CHECK-NEXT:    ret i32 [[X:%.*]]
 ;
@@ -42,7 +42,7 @@ define i32 @caller2(i32 %x) {
 
 declare void @use_cb1(ptr)
 
-define internal noundef i32 @callee2(i32 %x) {
+define internal i32 @callee2(i32 %x) {
 ; CHECK-LABEL: @callee2(
 ; CHECK-NEXT:    ret i32 [[X:%.*]]
 ;
@@ -69,7 +69,7 @@ define void @caller_cb1() {
 
 declare void @use_cb2(ptr)
 
-define internal noundef i32 @callee3(i32 %x) {
+define internal i32 @callee3(i32 %x) {
 ; CHECK-LABEL: @callee3(
 ; CHECK-NEXT:    [[C:%.*]] = icmp eq i32 [[X:%.*]], 10
 ; CHECK-NEXT:    [[S:%.*]] = select i1 [[C]], i32 500, i32 600
@@ -99,7 +99,7 @@ define void @caller_cb2() {
 
 declare void @use_cb3(ptr)
 
-define internal noundef i32 @callee4(i32 %x, i32 %y) {
+define internal i32 @callee4(i32 %x, i32 %y) {
 ; CHECK-LABEL: @callee4(
 ; CHECK-NEXT:    [[C:%.*]] = icmp eq i32 [[X:%.*]], 10
 ; CHECK-NEXT:    [[S:%.*]] = select i1 [[C]], i32 500, i32 [[Y:%.*]]
@@ -128,7 +128,7 @@ define void @caller_cb3() {
 ; Test 5.
 ; Range for the return value of callee5 includes undef. No range metadata
 ; should be added at call sites.
-define internal noundef i32 @callee5(i32 %x, i32 %y) {
+define internal i32 @callee5(i32 %x, i32 %y) {
 ; CHECK-LABEL: @callee5(
 ; CHECK-NEXT:    [[C:%.*]] = icmp slt i32 [[X:%.*]], 15
 ; CHECK-NEXT:    br i1 [[C]], label [[BB1:%.*]], label [[BB2:%.*]]
@@ -169,5 +169,26 @@ define i32 @caller5() {
   ret i32 %a
 }
 
+define internal <2 x i64> @ctlz(<2 x i64> %arg) {
+; CHECK-LABEL: @ctlz(
+; CHECK-NEXT:    [[RES:%.*]] = call <2 x i64> @llvm.ctlz.v2i64(<2 x i64> [[ARG:%.*]], i1 false)
+; CHECK-NEXT:    ret <2 x i64> [[RES]]
+;
+  %res = call <2 x i64> @llvm.ctlz.v2i64(<2 x i64> %arg, i1 false)
+  ret <2 x i64> %res
+}
+
+define <2 x i64> @ctlz_caller(<2 x i64> %arg) {
+; CHECK-LABEL: @ctlz_caller(
+; CHECK-NEXT:    [[RES:%.*]] = call <2 x i64> @ctlz(<2 x i64> [[ARG:%.*]]), !range [[RNG2:![0-9]+]]
+; CHECK-NEXT:    ret <2 x i64> [[RES]]
+;
+  %res = call <2 x i64> @ctlz(<2 x i64> %arg)
+  ret <2 x i64> %res
+}
+
+declare <2 x i64> @llvm.ctlz.v2i64(<2 x i64>, i1)
+
 ; CHECK: [[RNG0]] = !{i32 0, i32 21}
 ; CHECK: [[RNG1]] = !{i32 500, i32 601}
+; CHECK: [[RNG2]] = !{i64 0, i64 65}
diff --git a/llvm/test/Transforms/SLPVectorizer/AArch64/accelerate-vector-functions-inseltpoison.ll b/llvm/test/Transforms/SLPVectorizer/AArch64/accelerate-vector-functions-inseltpoison.ll
index 5e8616673bbcd..ee24944e5cec6 100644
--- a/llvm/test/Transforms/SLPVectorizer/AArch64/accelerate-vector-functions-inseltpoison.ll
+++ b/llvm/test/Transforms/SLPVectorizer/AArch64/accelerate-vector-functions-inseltpoison.ll
@@ -24,13 +24,10 @@ define <4 x float> @int_sin_4x(ptr %a) {
 ; NOACCELERATE-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
 ; NOACCELERATE-NEXT:    [[TMP2:%.*]] = tail call fast float @llvm.sin.f32(float [[VECEXT_1]])
 ; NOACCELERATE-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
-; NOACCELERATE-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
-; NOACCELERATE-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
-; NOACCELERATE-NEXT:    [[TMP3:%.*]] = insertelement <2 x float> poison, float [[VECEXT_2]], i32 0
-; NOACCELERATE-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> [[TMP3]], float [[VECEXT_3]], i32 1
-; NOACCELERATE-NEXT:    [[TMP5:%.*]] = call fast <2 x float> @llvm.sin.v2f32(<2 x float> [[TMP4]])
-; NOACCELERATE-NEXT:    [[TMP6:%.*]] = shufflevector <2 x float> [[TMP5]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP6]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
+; NOACCELERATE-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[TMP0]], <4 x float> poison, <2 x i32> <i32 2, i32 3>
+; NOACCELERATE-NEXT:    [[TMP4:%.*]] = call fast <2 x float> @llvm.sin.v2f32(<2 x float> [[TMP3]])
+; NOACCELERATE-NEXT:    [[TMP5:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP5]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
 ; NOACCELERATE-NEXT:    ret <4 x float> [[VECINS_31]]
 ;
 entry:
@@ -220,13 +217,10 @@ define <4 x float> @exp_4x(ptr %a) {
 ; NOACCELERATE-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
 ; NOACCELERATE-NEXT:    [[TMP2:%.*]] = tail call fast float @expf(float [[VECEXT_1]])
 ; NOACCELERATE-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
-; NOACCELERATE-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
-; NOACCELERATE-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
-; NOACCELERATE-NEXT:    [[TMP3:%.*]] = insertelement <2 x float> poison, float [[VECEXT_2]], i32 0
-; NOACCELERATE-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> [[TMP3]], float [[VECEXT_3]], i32 1
-; NOACCELERATE-NEXT:    [[TMP5:%.*]] = call fast <2 x float> @llvm.exp.v2f32(<2 x float> [[TMP4]])
-; NOACCELERATE-NEXT:    [[TMP6:%.*]] = shufflevector <2 x float> [[TMP5]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP6]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
+; NOACCELERATE-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[TMP0]], <4 x float> poison, <2 x i32> <i32 2, i32 3>
+; NOACCELERATE-NEXT:    [[TMP4:%.*]] = call fast <2 x float> @llvm.exp.v2f32(<2 x float> [[TMP3]])
+; NOACCELERATE-NEXT:    [[TMP5:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP5]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
 ; NOACCELERATE-NEXT:    ret <4 x float> [[VECINS_31]]
 ;
 entry:
@@ -303,13 +297,10 @@ define <4 x float> @log_4x(ptr %a) {
 ; NOACCELERATE-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
 ; NOACCELERATE-NEXT:    [[TMP2:%.*]] = tail call fast float @logf(float [[VECEXT_1]])
 ; NOACCELERATE-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
-; NOACCELERATE-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
-; NOACCELERATE-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
-; NOACCELERATE-NEXT:    [[TMP3:%.*]] = insertelement <2 x float> poison, float [[VECEXT_2]], i32 0
-; NOACCELERATE-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> [[TMP3]], float [[VECEXT_3]], i32 1
-; NOACCELERATE-NEXT:    [[TMP5:%.*]] = call fast <2 x float> @llvm.log.v2f32(<2 x float> [[TMP4]])
-; NOACCELERATE-NEXT:    [[TMP6:%.*]] = shufflevector <2 x float> [[TMP5]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP6]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
+; NOACCELERATE-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[TMP0]], <4 x float> poison, <2 x i32> <i32 2, i32 3>
+; NOACCELERATE-NEXT:    [[TMP4:%.*]] = call fast <2 x float> @llvm.log.v2f32(<2 x float> [[TMP3]])
+; NOACCELERATE-NEXT:    [[TMP5:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP5]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
 ; NOACCELERATE-NEXT:    ret <4 x float> [[VECINS_31]]
 ;
 entry:
@@ -479,13 +470,10 @@ define <4 x float> @sin_4x(ptr %a) {
 ; NOACCELERATE-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
 ; NOACCELERATE-NEXT:    [[TMP2:%.*]] = tail call fast float @sinf(float [[VECEXT_1]])
 ; NOACCELERATE-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
-; NOACCELERATE-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
-; NOACCELERATE-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
-; NOACCELERATE-NEXT:    [[TMP3:%.*]] = insertelement <2 x float> poison, float [[VECEXT_2]], i32 0
-; NOACCELERATE-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> [[TMP3]], float [[VECEXT_3]], i32 1
-; NOACCELERATE-NEXT:    [[TMP5:%.*]] = call fast <2 x float> @llvm.sin.v2f32(<2 x float> [[TMP4]])
-; NOACCELERATE-NEXT:    [[TMP6:%.*]] = shufflevector <2 x float> [[TMP5]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP6]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
+; NOACCELERATE-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[TMP0]], <4 x float> poison, <2 x i32> <i32 2, i32 3>
+; NOACCELERATE-NEXT:    [[TMP4:%.*]] = call fast <2 x float> @llvm.sin.v2f32(<2 x float> [[TMP3]])
+; NOACCELERATE-NEXT:    [[TMP5:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP5]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
 ; NOACCELERATE-NEXT:    ret <4 x float> [[VECINS_31]]
 ;
 entry:
@@ -521,13 +509,10 @@ define <4 x float> @cos_4x(ptr %a) {
 ; NOACCELERATE-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
 ; NOACCELERATE-NEXT:    [[TMP2:%.*]] = tail call fast float @cosf(float [[VECEXT_1]])
 ; NOACCELERATE-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
-; NOACCELERATE-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
-; NOACCELERATE-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
-; NOACCELERATE-NEXT:    [[TMP3:%.*]] = insertelement <2 x float> poison, float [[VECEXT_2]], i32 0
-; NOACCELERATE-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> [[TMP3]], float [[VECEXT_3]], i32 1
-; NOACCELERATE-NEXT:    [[TMP5:%.*]] = call fast <2 x float> @llvm.cos.v2f32(<2 x float> [[TMP4]])
-; NOACCELERATE-NEXT:    [[TMP6:%.*]] = shufflevector <2 x float> [[TMP5]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP6]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
+; NOACCELERATE-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[TMP0]], <4 x float> poison, <2 x i32> <i32 2, i32 3>
+; NOACCELERATE-NEXT:    [[TMP4:%.*]] = call fast <2 x float> @llvm.cos.v2f32(<2 x float> [[TMP3]])
+; NOACCELERATE-NEXT:    [[TMP5:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP5]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
 ; NOACCELERATE-NEXT:    ret <4 x float> [[VECINS_31]]
 ;
 entry:
@@ -1012,13 +997,10 @@ define <4 x float> @int_cos_4x(ptr %a) {
 ; NOACCELERATE-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
 ; NOACCELERATE-NEXT:    [[TMP2:%.*]] = tail call fast float @llvm.cos.f32(float [[VECEXT_1]])
 ; NOACCELERATE-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
-; NOACCELERATE-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
-; NOACCELERATE-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
-; NOACCELERATE-NEXT:    [[TMP3:%.*]] = insertelement <2 x float> poison, float [[VECEXT_2]], i32 0
-; NOACCELERATE-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> [[TMP3]], float [[VECEXT_3]], i32 1
-; NOACCELERATE-NEXT:    [[TMP5:%.*]] = call fast <2 x float> @llvm.cos.v2f32(<2 x float> [[TMP4]])
-; NOACCELERATE-NEXT:    [[TMP6:%.*]] = shufflevector <2 x float> [[TMP5]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP6]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
+; NOACCELERATE-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[TMP0]], <4 x float> poison, <2 x i32> <i32 2, i32 3>
+; NOACCELERATE-NEXT:    [[TMP4:%.*]] = call fast <2 x float> @llvm.cos.v2f32(<2 x float> [[TMP3]])
+; NOACCELERATE-NEXT:    [[TMP5:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP5]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
 ; NOACCELERATE-NEXT:    ret <4 x float> [[VECINS_31]]
 ;
 entry:
diff --git a/llvm/test/Transforms/SLPVectorizer/AArch64/accelerate-vector-functions.ll b/llvm/test/Transforms/SLPVectorizer/AArch64/accelerate-vector-functions.ll
index 4df52503e1e2c..2bac49a58cd47 100644
--- a/llvm/test/Transforms/SLPVectorizer/AArch64/accelerate-vector-functions.ll
+++ b/llvm/test/Transforms/SLPVectorizer/AArch64/accelerate-vector-functions.ll
@@ -24,13 +24,10 @@ define <4 x float> @int_sin_4x(ptr %a) {
 ; NOACCELERATE-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
 ; NOACCELERATE-NEXT:    [[TMP2:%.*]] = tail call fast float @llvm.sin.f32(float [[VECEXT_1]])
 ; NOACCELERATE-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
-; NOACCELERATE-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
-; NOACCELERATE-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
-; NOACCELERATE-NEXT:    [[TMP3:%.*]] = insertelement <2 x float> poison, float [[VECEXT_2]], i32 0
-; NOACCELERATE-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> [[TMP3]], float [[VECEXT_3]], i32 1
-; NOACCELERATE-NEXT:    [[TMP5:%.*]] = call fast <2 x float> @llvm.sin.v2f32(<2 x float> [[TMP4]])
-; NOACCELERATE-NEXT:    [[TMP6:%.*]] = shufflevector <2 x float> [[TMP5]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP6]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
+; NOACCELERATE-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[TMP0]], <4 x float> poison, <2 x i32> <i32 2, i32 3>
+; NOACCELERATE-NEXT:    [[TMP4:%.*]] = call fast <2 x float> @llvm.sin.v2f32(<2 x float> [[TMP3]])
+; NOACCELERATE-NEXT:    [[TMP5:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP5]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
 ; NOACCELERATE-NEXT:    ret <4 x float> [[VECINS_31]]
 ;
 entry:
@@ -220,13 +217,10 @@ define <4 x float> @exp_4x(ptr %a) {
 ; NOACCELERATE-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
 ; NOACCELERATE-NEXT:    [[TMP2:%.*]] = tail call fast float @expf(float [[VECEXT_1]])
 ; NOACCELERATE-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
-; NOACCELERATE-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
-; NOACCELERATE-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
-; NOACCELERATE-NEXT:    [[TMP3:%.*]] = insertelement <2 x float> poison, float [[VECEXT_2]], i32 0
-; NOACCELERATE-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> [[TMP3]], float [[VECEXT_3]], i32 1
-; NOACCELERATE-NEXT:    [[TMP5:%.*]] = call fast <2 x float> @llvm.exp.v2f32(<2 x float> [[TMP4]])
-; NOACCELERATE-NEXT:    [[TMP6:%.*]] = shufflevector <2 x float> [[TMP5]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP6]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
+; NOACCELERATE-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[TMP0]], <4 x float> poison, <2 x i32> <i32 2, i32 3>
+; NOACCELERATE-NEXT:    [[TMP4:%.*]] = call fast <2 x float> @llvm.exp.v2f32(<2 x float> [[TMP3]])
+; NOACCELERATE-NEXT:    [[TMP5:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP5]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
 ; NOACCELERATE-NEXT:    ret <4 x float> [[VECINS_31]]
 ;
 entry:
@@ -303,13 +297,10 @@ define <4 x float> @log_4x(ptr %a) {
 ; NOACCELERATE-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
 ; NOACCELERATE-NEXT:    [[TMP2:%.*]] = tail call fast float @logf(float [[VECEXT_1]])
 ; NOACCELERATE-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
-; NOACCELERATE-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
-; NOACCELERATE-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
-; NOACCELERATE-NEXT:    [[TMP3:%.*]] = insertelement <2 x float> poison, float [[VECEXT_2]], i32 0
-; NOACCELERATE-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> [[TMP3]], float [[VECEXT_3]], i32 1
-; NOACCELERATE-NEXT:    [[TMP5:%.*]] = call fast <2 x float> @llvm.log.v2f32(<2 x float> [[TMP4]])
-; NOACCELERATE-NEXT:    [[TMP6:%.*]] = shufflevector <2 x float> [[TMP5]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP6]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
+; NOACCELERATE-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[TMP0]], <4 x float> poison, <2 x i32> <i32 2, i32 3>
+; NOACCELERATE-NEXT:    [[TMP4:%.*]] = call fast <2 x float> @llvm.log.v2f32(<2 x float> [[TMP3]])
+; NOACCELERATE-NEXT:    [[TMP5:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP5]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
 ; NOACCELERATE-NEXT:    ret <4 x float> [[VECINS_31]]
 ;
 entry:
@@ -479,13 +470,10 @@ define <4 x float> @sin_4x(ptr %a) {
 ; NOACCELERATE-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
 ; NOACCELERATE-NEXT:    [[TMP2:%.*]] = tail call fast float @sinf(float [[VECEXT_1]])
 ; NOACCELERATE-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
-; NOACCELERATE-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
-; NOACCELERATE-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
-; NOACCELERATE-NEXT:    [[TMP3:%.*]] = insertelement <2 x float> poison, float [[VECEXT_2]], i32 0
-; NOACCELERATE-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> [[TMP3]], float [[VECEXT_3]], i32 1
-; NOACCELERATE-NEXT:    [[TMP5:%.*]] = call fast <2 x float> @llvm.sin.v2f32(<2 x float> [[TMP4]])
-; NOACCELERATE-NEXT:    [[TMP6:%.*]] = shufflevector <2 x float> [[TMP5]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP6]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
+; NOACCELERATE-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[TMP0]], <4 x float> poison, <2 x i32> <i32 2, i32 3>
+; NOACCELERATE-NEXT:    [[TMP4:%.*]] = call fast <2 x float> @llvm.sin.v2f32(<2 x float> [[TMP3]])
+; NOACCELERATE-NEXT:    [[TMP5:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP5]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
 ; NOACCELERATE-NEXT:    ret <4 x float> [[VECINS_31]]
 ;
 entry:
@@ -521,13 +509,10 @@ define <4 x float> @cos_4x(ptr %a) {
 ; NOACCELERATE-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
 ; NOACCELERATE-NEXT:    [[TMP2:%.*]] = tail call fast float @cosf(float [[VECEXT_1]])
 ; NOACCELERATE-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
-; NOACCELERATE-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
-; NOACCELERATE-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
-; NOACCELERATE-NEXT:    [[TMP3:%.*]] = insertelement <2 x float> poison, float [[VECEXT_2]], i32 0
-; NOACCELERATE-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> [[TMP3]], float [[VECEXT_3]], i32 1
-; NOACCELERATE-NEXT:    [[TMP5:%.*]] = call fast <2 x float> @llvm.cos.v2f32(<2 x float> [[TMP4]])
-; NOACCELERATE-NEXT:    [[TMP6:%.*]] = shufflevector <2 x float> [[TMP5]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP6]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
+; NOACCELERATE-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[TMP0]], <4 x float> poison, <2 x i32> <i32 2, i32 3>
+; NOACCELERATE-NEXT:    [[TMP4:%.*]] = call fast <2 x float> @llvm.cos.v2f32(<2 x float> [[TMP3]])
+; NOACCELERATE-NEXT:    [[TMP5:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP5]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
 ; NOACCELERATE-NEXT:    ret <4 x float> [[VECINS_31]]
 ;
 entry:
@@ -1012,13 +997,10 @@ define <4 x float> @int_cos_4x(ptr %a) {
 ; NOACCELERATE-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
 ; NOACCELERATE-NEXT:    [[TMP2:%.*]] = tail call fast float @llvm.cos.f32(float [[VECEXT_1]])
 ; NOACCELERATE-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
-; NOACCELERATE-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
-; NOACCELERATE-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
-; NOACCELERATE-NEXT:    [[TMP3:%.*]] = insertelement <2 x float> poison, float [[VECEXT_2]], i32 0
-; NOACCELERATE-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> [[TMP3]], float [[VECEXT_3]], i32 1
-; NOACCELERATE-NEXT:    [[TMP5:%.*]] = call fast <2 x float> @llvm.cos.v2f32(<2 x float> [[TMP4]])
-; NOACCELERATE-NEXT:    [[TMP6:%.*]] = shufflevector <2 x float> [[TMP5]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP6]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
+; NOACCELERATE-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[TMP0]], <4 x float> poison, <2 x i32> <i32 2, i32 3>
+; NOACCELERATE-NEXT:    [[TMP4:%.*]] = call fast <2 x float> @llvm.cos.v2f32(<2 x float> [[TMP3]])
+; NOACCELERATE-NEXT:    [[TMP5:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; NOACCELERATE-NEXT:    [[VECINS_31:%.*]] = shufflevector <4 x float> [[VECINS_1]], <4 x float> [[TMP5]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
 ; NOACCELERATE-NEXT:    ret <4 x float> [[VECINS_31]]
 ;
 entry:
diff --git a/llvm/test/Transforms/SLPVectorizer/AArch64/slp-fma-loss.ll b/llvm/test/Transforms/SLPVectorizer/AArch64/slp-fma-loss.ll
index 3327cb0e51a8e..4214ecf0bb787 100644
--- a/llvm/test/Transforms/SLPVectorizer/AArch64/slp-fma-loss.ll
+++ b/llvm/test/Transforms/SLPVectorizer/AArch64/slp-fma-loss.ll
@@ -7,21 +7,20 @@ define void @slp_not_profitable_with_fast_fmf(ptr %A, ptr %B) {
 ; CHECK-LABEL: @slp_not_profitable_with_fast_fmf(
 ; CHECK-NEXT:    [[GEP_B_1:%.*]] = getelementptr inbounds float, ptr [[B:%.*]], i64 1
 ; CHECK-NEXT:    [[A_0:%.*]] = load float, ptr [[A:%.*]], align 4
+; CHECK-NEXT:    [[B_1:%.*]] = load float, ptr [[GEP_B_1]], align 4
+; CHECK-NEXT:    [[MUL_0:%.*]] = fmul fast float [[B_1]], [[A_0]]
 ; CHECK-NEXT:    [[B_0:%.*]] = load float, ptr [[B]], align 4
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x float>, ptr [[GEP_B_1]], align 4
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x float> poison, float [[B_0]], i32 0
-; CHECK-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <2 x float> [[TMP2]], <2 x float> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP3:%.*]] = fmul fast <2 x float> [[SHUFFLE1]], [[TMP1]]
-; CHECK-NEXT:    [[SHUFFLE2:%.*]] = shufflevector <2 x float> [[TMP3]], <2 x float> poison, <2 x i32> <i32 1, i32 0>
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> poison, float [[A_0]], i32 0
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP5:%.*]] = fmul fast <2 x float> [[TMP1]], [[SHUFFLE]]
-; CHECK-NEXT:    [[TMP6:%.*]] = fsub fast <2 x float> [[TMP5]], [[SHUFFLE2]]
-; CHECK-NEXT:    [[TMP7:%.*]] = fadd fast <2 x float> [[TMP5]], [[SHUFFLE2]]
-; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <2 x float> [[TMP6]], <2 x float> [[TMP7]], <2 x i32> <i32 0, i32 3>
-; CHECK-NEXT:    store <2 x float> [[TMP8]], ptr [[A]], align 4
-; CHECK-NEXT:    [[TMP9:%.*]] = extractelement <2 x float> [[TMP1]], i32 1
-; CHECK-NEXT:    store float [[TMP9]], ptr [[B]], align 4
+; CHECK-NEXT:    [[GEP_B_2:%.*]] = getelementptr inbounds float, ptr [[B]], i64 2
+; CHECK-NEXT:    [[B_2:%.*]] = load float, ptr [[GEP_B_2]], align 4
+; CHECK-NEXT:    [[MUL_1:%.*]] = fmul fast float [[B_2]], [[B_0]]
+; CHECK-NEXT:    [[SUB:%.*]] = fsub fast float [[MUL_0]], [[MUL_1]]
+; CHECK-NEXT:    [[MUL_2:%.*]] = fmul fast float [[B_0]], [[B_1]]
+; CHECK-NEXT:    [[MUL_3:%.*]] = fmul fast float [[B_2]], [[A_0]]
+; CHECK-NEXT:    [[ADD:%.*]] = fadd fast float [[MUL_3]], [[MUL_2]]
+; CHECK-NEXT:    store float [[SUB]], ptr [[A]], align 4
+; CHECK-NEXT:    [[GEP_A_1:%.*]] = getelementptr inbounds float, ptr [[A]], i64 1
+; CHECK-NEXT:    store float [[ADD]], ptr [[GEP_A_1]], align 4
+; CHECK-NEXT:    store float [[B_2]], ptr [[B]], align 4
 ; CHECK-NEXT:    ret void
 ;
   %gep.B.1 = getelementptr inbounds float, ptr %B, i64 1
@@ -47,21 +46,20 @@ define void @slp_not_profitable_with_reassoc_fmf(ptr %A, ptr %B) {
 ; CHECK-LABEL: @slp_not_profitable_with_reassoc_fmf(
 ; CHECK-NEXT:    [[GEP_B_1:%.*]] = getelementptr inbounds float, ptr [[B:%.*]], i64 1
 ; CHECK-NEXT:    [[A_0:%.*]] = load float, ptr [[A:%.*]], align 4
+; CHECK-NEXT:    [[B_1:%.*]] = load float, ptr [[GEP_B_1]], align 4
+; CHECK-NEXT:    [[MUL_0:%.*]] = fmul reassoc float [[B_1]], [[A_0]]
 ; CHECK-NEXT:    [[B_0:%.*]] = load float, ptr [[B]], align 4
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x float>, ptr [[GEP_B_1]], align 4
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x float> poison, float [[B_0]], i32 0
-; CHECK-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <2 x float> [[TMP2]], <2 x float> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP3:%.*]] = fmul <2 x float> [[SHUFFLE1]], [[TMP1]]
-; CHECK-NEXT:    [[SHUFFLE2:%.*]] = shufflevector <2 x float> [[TMP3]], <2 x float> poison, <2 x i32> <i32 1, i32 0>
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> poison, float [[A_0]], i32 0
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP5:%.*]] = fmul reassoc <2 x float> [[TMP1]], [[SHUFFLE]]
-; CHECK-NEXT:    [[TMP6:%.*]] = fsub reassoc <2 x float> [[TMP5]], [[SHUFFLE2]]
-; CHECK-NEXT:    [[TMP7:%.*]] = fadd reassoc <2 x float> [[TMP5]], [[SHUFFLE2]]
-; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <2 x float> [[TMP6]], <2 x float> [[TMP7]], <2 x i32> <i32 0, i32 3>
-; CHECK-NEXT:    store <2 x float> [[TMP8]], ptr [[A]], align 4
-; CHECK-NEXT:    [[TMP9:%.*]] = extractelement <2 x float> [[TMP1]], i32 1
-; CHECK-NEXT:    store float [[TMP9]], ptr [[B]], align 4
+; CHECK-NEXT:    [[GEP_B_2:%.*]] = getelementptr inbounds float, ptr [[B]], i64 2
+; CHECK-NEXT:    [[B_2:%.*]] = load float, ptr [[GEP_B_2]], align 4
+; CHECK-NEXT:    [[MUL_1:%.*]] = fmul float [[B_2]], [[B_0]]
+; CHECK-NEXT:    [[SUB:%.*]] = fsub reassoc float [[MUL_0]], [[MUL_1]]
+; CHECK-NEXT:    [[MUL_2:%.*]] = fmul float [[B_0]], [[B_1]]
+; CHECK-NEXT:    [[MUL_3:%.*]] = fmul reassoc float [[B_2]], [[A_0]]
+; CHECK-NEXT:    [[ADD:%.*]] = fadd reassoc float [[MUL_3]], [[MUL_2]]
+; CHECK-NEXT:    store float [[SUB]], ptr [[A]], align 4
+; CHECK-NEXT:    [[GEP_A_1:%.*]] = getelementptr inbounds float, ptr [[A]], i64 1
+; CHECK-NEXT:    store float [[ADD]], ptr [[GEP_A_1]], align 4
+; CHECK-NEXT:    store float [[B_2]], ptr [[B]], align 4
 ; CHECK-NEXT:    ret void
 ;
   %gep.B.1 = getelementptr inbounds float, ptr %B, i64 1
@@ -88,21 +86,20 @@ define void @slp_profitable_missing_fmf_on_fadd_fsub(ptr %A, ptr %B) {
 ; CHECK-LABEL: @slp_profitable_missing_fmf_on_fadd_fsub(
 ; CHECK-NEXT:    [[GEP_B_1:%.*]] = getelementptr inbounds float, ptr [[B:%.*]], i64 1
 ; CHECK-NEXT:    [[A_0:%.*]] = load float, ptr [[A:%.*]], align 4
+; CHECK-NEXT:    [[B_1:%.*]] = load float, ptr [[GEP_B_1]], align 4
+; CHECK-NEXT:    [[MUL_0:%.*]] = fmul fast float [[B_1]], [[A_0]]
 ; CHECK-NEXT:    [[B_0:%.*]] = load float, ptr [[B]], align 4
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x float>, ptr [[GEP_B_1]], align 4
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x float> poison, float [[B_0]], i32 0
-; CHECK-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <2 x float> [[TMP2]], <2 x float> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP3:%.*]] = fmul fast <2 x float> [[SHUFFLE1]], [[TMP1]]
-; CHECK-NEXT:    [[SHUFFLE2:%.*]] = shufflevector <2 x float> [[TMP3]], <2 x float> poison, <2 x i32> <i32 1, i32 0>
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> poison, float [[A_0]], i32 0
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP5:%.*]] = fmul fast <2 x float> [[TMP1]], [[SHUFFLE]]
-; CHECK-NEXT:    [[TMP6:%.*]] = fsub <2 x float> [[TMP5]], [[SHUFFLE2]]
-; CHECK-NEXT:    [[TMP7:%.*]] = fadd <2 x float> [[TMP5]], [[SHUFFLE2]]
-; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <2 x float> [[TMP6]], <2 x float> [[TMP7]], <2 x i32> <i32 0, i32 3>
-; CHECK-NEXT:    store <2 x float> [[TMP8]], ptr [[A]], align 4
-; CHECK-NEXT:    [[TMP9:%.*]] = extractelement <2 x float> [[TMP1]], i32 1
-; CHECK-NEXT:    store float [[TMP9]], ptr [[B]], align 4
+; CHECK-NEXT:    [[GEP_B_2:%.*]] = getelementptr inbounds float, ptr [[B]], i64 2
+; CHECK-NEXT:    [[B_2:%.*]] = load float, ptr [[GEP_B_2]], align 4
+; CHECK-NEXT:    [[MUL_1:%.*]] = fmul fast float [[B_2]], [[B_0]]
+; CHECK-NEXT:    [[SUB:%.*]] = fsub float [[MUL_0]], [[MUL_1]]
+; CHECK-NEXT:    [[MUL_2:%.*]] = fmul fast float [[B_0]], [[B_1]]
+; CHECK-NEXT:    [[MUL_3:%.*]] = fmul fast float [[B_2]], [[A_0]]
+; CHECK-NEXT:    [[ADD:%.*]] = fadd float [[MUL_3]], [[MUL_2]]
+; CHECK-NEXT:    store float [[SUB]], ptr [[A]], align 4
+; CHECK-NEXT:    [[GEP_A_1:%.*]] = getelementptr inbounds float, ptr [[A]], i64 1
+; CHECK-NEXT:    store float [[ADD]], ptr [[GEP_A_1]], align 4
+; CHECK-NEXT:    store float [[B_2]], ptr [[B]], align 4
 ; CHECK-NEXT:    ret void
 ;
   %gep.B.1 = getelementptr inbounds float, ptr %B, i64 1
@@ -129,21 +126,20 @@ define void @slp_profitable_missing_fmf_on_fmul_fadd_fsub(ptr %A, ptr %B) {
 ; CHECK-LABEL: @slp_profitable_missing_fmf_on_fmul_fadd_fsub(
 ; CHECK-NEXT:    [[GEP_B_1:%.*]] = getelementptr inbounds float, ptr [[B:%.*]], i64 1
 ; CHECK-NEXT:    [[A_0:%.*]] = load float, ptr [[A:%.*]], align 4
+; CHECK-NEXT:    [[B_1:%.*]] = load float, ptr [[GEP_B_1]], align 4
+; CHECK-NEXT:    [[MUL_0:%.*]] = fmul float [[B_1]], [[A_0]]
 ; CHECK-NEXT:    [[B_0:%.*]] = load float, ptr [[B]], align 4
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x float>, ptr [[GEP_B_1]], align 4
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x float> poison, float [[B_0]], i32 0
-; CHECK-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <2 x float> [[TMP2]], <2 x float> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP3:%.*]] = fmul <2 x float> [[SHUFFLE1]], [[TMP1]]
-; CHECK-NEXT:    [[SHUFFLE2:%.*]] = shufflevector <2 x float> [[TMP3]], <2 x float> poison, <2 x i32> <i32 1, i32 0>
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> poison, float [[A_0]], i32 0
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP5:%.*]] = fmul <2 x float> [[TMP1]], [[SHUFFLE]]
-; CHECK-NEXT:    [[TMP6:%.*]] = fsub <2 x float> [[TMP5]], [[SHUFFLE2]]
-; CHECK-NEXT:    [[TMP7:%.*]] = fadd <2 x float> [[TMP5]], [[SHUFFLE2]]
-; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <2 x float> [[TMP6]], <2 x float> [[TMP7]], <2 x i32> <i32 0, i32 3>
-; CHECK-NEXT:    store <2 x float> [[TMP8]], ptr [[A]], align 4
-; CHECK-NEXT:    [[TMP9:%.*]] = extractelement <2 x float> [[TMP1]], i32 1
-; CHECK-NEXT:    store float [[TMP9]], ptr [[B]], align 4
+; CHECK-NEXT:    [[GEP_B_2:%.*]] = getelementptr inbounds float, ptr [[B]], i64 2
+; CHECK-NEXT:    [[B_2:%.*]] = load float, ptr [[GEP_B_2]], align 4
+; CHECK-NEXT:    [[MUL_1:%.*]] = fmul float [[B_2]], [[B_0]]
+; CHECK-NEXT:    [[SUB:%.*]] = fsub float [[MUL_0]], [[MUL_1]]
+; CHECK-NEXT:    [[MUL_2:%.*]] = fmul float [[B_0]], [[B_1]]
+; CHECK-NEXT:    [[MUL_3:%.*]] = fmul float [[B_2]], [[A_0]]
+; CHECK-NEXT:    [[ADD:%.*]] = fadd float [[MUL_3]], [[MUL_2]]
+; CHECK-NEXT:    store float [[SUB]], ptr [[A]], align 4
+; CHECK-NEXT:    [[GEP_A_1:%.*]] = getelementptr inbounds float, ptr [[A]], i64 1
+; CHECK-NEXT:    store float [[ADD]], ptr [[GEP_A_1]], align 4
+; CHECK-NEXT:    store float [[B_2]], ptr [[B]], align 4
 ; CHECK-NEXT:    ret void
 ;
   %gep.B.1 = getelementptr inbounds float, ptr %B, i64 1
@@ -170,21 +166,20 @@ define void @slp_profitable_missing_fmf_nnans_only(ptr %A, ptr %B) {
 ; CHECK-LABEL: @slp_profitable_missing_fmf_nnans_only(
 ; CHECK-NEXT:    [[GEP_B_1:%.*]] = getelementptr inbounds float, ptr [[B:%.*]], i64 1
 ; CHECK-NEXT:    [[A_0:%.*]] = load float, ptr [[A:%.*]], align 4
+; CHECK-NEXT:    [[B_1:%.*]] = load float, ptr [[GEP_B_1]], align 4
+; CHECK-NEXT:    [[MUL_0:%.*]] = fmul nnan float [[B_1]], [[A_0]]
 ; CHECK-NEXT:    [[B_0:%.*]] = load float, ptr [[B]], align 4
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x float>, ptr [[GEP_B_1]], align 4
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x float> poison, float [[B_0]], i32 0
-; CHECK-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <2 x float> [[TMP2]], <2 x float> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP3:%.*]] = fmul nnan <2 x float> [[SHUFFLE1]], [[TMP1]]
-; CHECK-NEXT:    [[SHUFFLE2:%.*]] = shufflevector <2 x float> [[TMP3]], <2 x float> poison, <2 x i32> <i32 1, i32 0>
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> poison, float [[A_0]], i32 0
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP5:%.*]] = fmul nnan <2 x float> [[TMP1]], [[SHUFFLE]]
-; CHECK-NEXT:    [[TMP6:%.*]] = fsub nnan <2 x float> [[TMP5]], [[SHUFFLE2]]
-; CHECK-NEXT:    [[TMP7:%.*]] = fadd nnan <2 x float> [[TMP5]], [[SHUFFLE2]]
-; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <2 x float> [[TMP6]], <2 x float> [[TMP7]], <2 x i32> <i32 0, i32 3>
-; CHECK-NEXT:    store <2 x float> [[TMP8]], ptr [[A]], align 4
-; CHECK-NEXT:    [[TMP9:%.*]] = extractelement <2 x float> [[TMP1]], i32 1
-; CHECK-NEXT:    store float [[TMP9]], ptr [[B]], align 4
+; CHECK-NEXT:    [[GEP_B_2:%.*]] = getelementptr inbounds float, ptr [[B]], i64 2
+; CHECK-NEXT:    [[B_2:%.*]] = load float, ptr [[GEP_B_2]], align 4
+; CHECK-NEXT:    [[MUL_1:%.*]] = fmul nnan float [[B_2]], [[B_0]]
+; CHECK-NEXT:    [[SUB:%.*]] = fsub nnan float [[MUL_0]], [[MUL_1]]
+; CHECK-NEXT:    [[MUL_2:%.*]] = fmul nnan float [[B_0]], [[B_1]]
+; CHECK-NEXT:    [[MUL_3:%.*]] = fmul nnan float [[B_2]], [[A_0]]
+; CHECK-NEXT:    [[ADD:%.*]] = fadd nnan float [[MUL_3]], [[MUL_2]]
+; CHECK-NEXT:    store float [[SUB]], ptr [[A]], align 4
+; CHECK-NEXT:    [[GEP_A_1:%.*]] = getelementptr inbounds float, ptr [[A]], i64 1
+; CHECK-NEXT:    store float [[ADD]], ptr [[GEP_A_1]], align 4
+; CHECK-NEXT:    store float [[B_2]], ptr [[B]], align 4
 ; CHECK-NEXT:    ret void
 ;
   %gep.B.1 = getelementptr inbounds float, ptr %B, i64 1
@@ -267,16 +262,16 @@ define void @slp_profitable(ptr %A, ptr %B, float %0) {
 ; CHECK-NEXT:    [[SUB_I1096:%.*]] = fsub fast float 1.000000e+00, [[TMP0:%.*]]
 ; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x float>, ptr [[A:%.*]], align 4
 ; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x float> poison, float [[TMP0]], i32 0
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x float> [[TMP2]], <2 x float> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP3:%.*]] = fmul fast <2 x float> [[TMP1]], [[SHUFFLE]]
-; CHECK-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <2 x float> [[TMP3]], <2 x float> poison, <2 x i32> <i32 1, i32 0>
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> poison, float [[SUB_I1096]], i32 0
-; CHECK-NEXT:    [[SHUFFLE2:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP5:%.*]] = fmul fast <2 x float> [[TMP1]], [[SHUFFLE2]]
-; CHECK-NEXT:    [[TMP6:%.*]] = fadd fast <2 x float> [[SHUFFLE1]], [[TMP5]]
-; CHECK-NEXT:    [[TMP7:%.*]] = fsub fast <2 x float> [[SHUFFLE1]], [[TMP5]]
-; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <2 x float> [[TMP6]], <2 x float> [[TMP7]], <2 x i32> <i32 0, i32 3>
-; CHECK-NEXT:    store <2 x float> [[TMP8]], ptr [[B:%.*]], align 4
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <2 x float> [[TMP2]], <2 x float> poison, <2 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP4:%.*]] = fmul fast <2 x float> [[TMP1]], [[TMP3]]
+; CHECK-NEXT:    [[TMP5:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <2 x i32> <i32 1, i32 0>
+; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x float> poison, float [[SUB_I1096]], i32 0
+; CHECK-NEXT:    [[TMP7:%.*]] = shufflevector <2 x float> [[TMP6]], <2 x float> poison, <2 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP8:%.*]] = fmul fast <2 x float> [[TMP1]], [[TMP7]]
+; CHECK-NEXT:    [[TMP9:%.*]] = fadd fast <2 x float> [[TMP5]], [[TMP8]]
+; CHECK-NEXT:    [[TMP10:%.*]] = fsub fast <2 x float> [[TMP5]], [[TMP8]]
+; CHECK-NEXT:    [[TMP11:%.*]] = shufflevector <2 x float> [[TMP9]], <2 x float> [[TMP10]], <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    store <2 x float> [[TMP11]], ptr [[B:%.*]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
diff --git a/llvm/test/Transforms/SLPVectorizer/AArch64/vectorize-free-extracts-inserts.ll b/llvm/test/Transforms/SLPVectorizer/AArch64/vectorize-free-extracts-inserts.ll
index 1cdac15dba943..48ebff8ebabb6 100644
--- a/llvm/test/Transforms/SLPVectorizer/AArch64/vectorize-free-extracts-inserts.ll
+++ b/llvm/test/Transforms/SLPVectorizer/AArch64/vectorize-free-extracts-inserts.ll
@@ -13,16 +13,13 @@ define void @noop_extracts_first_2_lanes(ptr %ptr.1, ptr %ptr.2) {
 ; CHECK-NEXT:  bb:
 ; CHECK-NEXT:    [[V_1:%.*]] = load <2 x double>, ptr [[PTR_1:%.*]], align 8
 ; CHECK-NEXT:    [[V_2:%.*]] = load <4 x double>, ptr [[PTR_2:%.*]], align 16
-; CHECK-NEXT:    [[V2_LANE_2:%.*]] = extractelement <4 x double> [[V_2]], i32 2
-; CHECK-NEXT:    [[V2_LANE_3:%.*]] = extractelement <4 x double> [[V_2]], i32 3
-; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <2 x double> poison, double [[V2_LANE_2]], i32 0
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x double> [[TMP0]], double [[V2_LANE_3]], i32 1
-; CHECK-NEXT:    [[TMP2:%.*]] = fmul <2 x double> [[V_1]], [[TMP1]]
-; CHECK-NEXT:    [[TMP3:%.*]] = extractelement <2 x double> [[V_1]], i32 0
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <4 x double> [[V_2]], <4 x double> poison, <2 x i32> <i32 2, i32 3>
+; CHECK-NEXT:    [[TMP1:%.*]] = fmul <2 x double> [[V_1]], [[TMP0]]
+; CHECK-NEXT:    [[TMP2:%.*]] = extractelement <2 x double> [[V_1]], i32 0
+; CHECK-NEXT:    call void @use(double [[TMP2]])
+; CHECK-NEXT:    [[TMP3:%.*]] = extractelement <2 x double> [[V_1]], i32 1
 ; CHECK-NEXT:    call void @use(double [[TMP3]])
-; CHECK-NEXT:    [[TMP4:%.*]] = extractelement <2 x double> [[V_1]], i32 1
-; CHECK-NEXT:    call void @use(double [[TMP4]])
-; CHECK-NEXT:    store <2 x double> [[TMP2]], ptr [[PTR_1]], align 8
+; CHECK-NEXT:    store <2 x double> [[TMP1]], ptr [[PTR_1]], align 8
 ; CHECK-NEXT:    ret void
 ;
 bb:
@@ -56,15 +53,12 @@ define void @extracts_first_2_lanes_different_vectors(ptr %ptr.1, ptr %ptr.2, pt
 ; CHECK-NEXT:    [[V_3:%.*]] = load <2 x double>, ptr [[PTR_3:%.*]], align 8
 ; CHECK-NEXT:    [[V3_LANE_1:%.*]] = extractelement <2 x double> [[V_3]], i32 1
 ; CHECK-NEXT:    [[V_2:%.*]] = load <4 x double>, ptr [[PTR_2:%.*]], align 16
-; CHECK-NEXT:    [[V2_LANE_2:%.*]] = extractelement <4 x double> [[V_2]], i32 2
-; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <2 x double> poison, double [[V1_LANE_0]], i32 0
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x double> [[TMP0]], double [[V3_LANE_1]], i32 1
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[V2_LANE_2]], i32 0
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x double> [[TMP2]], <2 x double> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP3:%.*]] = fmul <2 x double> [[TMP1]], [[SHUFFLE]]
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <2 x double> [[V_1]], <2 x double> [[V_3]], <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x double> [[V_2]], <4 x double> poison, <2 x i32> <i32 2, i32 2>
+; CHECK-NEXT:    [[TMP2:%.*]] = fmul <2 x double> [[TMP0]], [[TMP1]]
 ; CHECK-NEXT:    call void @use(double [[V1_LANE_0]])
 ; CHECK-NEXT:    call void @use(double [[V3_LANE_1]])
-; CHECK-NEXT:    store <2 x double> [[TMP3]], ptr [[PTR_1]], align 8
+; CHECK-NEXT:    store <2 x double> [[TMP2]], ptr [[PTR_1]], align 8
 ; CHECK-NEXT:    ret void
 ;
 bb:
@@ -98,16 +92,13 @@ define void @noop_extract_second_2_lanes(ptr %ptr.1, ptr %ptr.2) {
 ; CHECK-NEXT:    [[V1_LANE_2:%.*]] = extractelement <4 x double> [[V_1]], i32 2
 ; CHECK-NEXT:    [[V1_LANE_3:%.*]] = extractelement <4 x double> [[V_1]], i32 3
 ; CHECK-NEXT:    [[V_2:%.*]] = load <4 x double>, ptr [[PTR_2:%.*]], align 16
-; CHECK-NEXT:    [[V2_LANE_2:%.*]] = extractelement <4 x double> [[V_2]], i32 2
-; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <2 x double> poison, double [[V1_LANE_2]], i32 0
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x double> [[TMP0]], double [[V1_LANE_3]], i32 1
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[V2_LANE_2]], i32 0
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x double> [[TMP2]], <2 x double> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP3:%.*]] = fmul <2 x double> [[TMP1]], [[SHUFFLE]]
-; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <2 x double> [[TMP3]], <2 x double> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <4 x double> [[V_1]], <4 x double> poison, <2 x i32> <i32 2, i32 3>
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x double> [[V_2]], <4 x double> poison, <2 x i32> <i32 2, i32 2>
+; CHECK-NEXT:    [[TMP2:%.*]] = fmul <2 x double> [[TMP0]], [[TMP1]]
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <2 x double> [[TMP2]], <2 x double> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
 ; CHECK-NEXT:    call void @use(double [[V1_LANE_2]])
 ; CHECK-NEXT:    call void @use(double [[V1_LANE_3]])
-; CHECK-NEXT:    store <4 x double> [[TMP4]], ptr [[PTR_1]], align 8
+; CHECK-NEXT:    store <4 x double> [[TMP3]], ptr [[PTR_1]], align 8
 ; CHECK-NEXT:    ret void
 ;
 bb:
@@ -137,10 +128,8 @@ define void @extract_reverse_order(ptr %ptr.1, ptr %ptr.2) {
 ; CHECK-NEXT:  bb:
 ; CHECK-NEXT:    [[V_1:%.*]] = load <2 x double>, ptr [[PTR_1:%.*]], align 8
 ; CHECK-NEXT:    [[V_2:%.*]] = load <4 x double>, ptr [[PTR_2:%.*]], align 16
-; CHECK-NEXT:    [[V2_LANE_2:%.*]] = extractelement <4 x double> [[V_2]], i32 2
-; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <2 x double> poison, double [[V2_LANE_2]], i32 0
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x double> [[TMP0]], <2 x double> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP1:%.*]] = fmul <2 x double> [[V_1]], [[SHUFFLE]]
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <4 x double> [[V_2]], <4 x double> poison, <2 x i32> <i32 2, i32 2>
+; CHECK-NEXT:    [[TMP1:%.*]] = fmul <2 x double> [[V_1]], [[TMP0]]
 ; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <2 x double> [[TMP1]], <2 x double> poison, <2 x i32> <i32 1, i32 0>
 ; CHECK-NEXT:    [[TMP3:%.*]] = extractelement <2 x double> [[V_1]], i32 0
 ; CHECK-NEXT:    call void @use(double [[TMP3]])
@@ -178,16 +167,13 @@ define void @extract_lanes_1_and_2(ptr %ptr.1, ptr %ptr.2) {
 ; CHECK-NEXT:    [[V1_LANE_1:%.*]] = extractelement <4 x double> [[V_1]], i32 1
 ; CHECK-NEXT:    [[V1_LANE_2:%.*]] = extractelement <4 x double> [[V_1]], i32 2
 ; CHECK-NEXT:    [[V_2:%.*]] = load <4 x double>, ptr [[PTR_2:%.*]], align 16
-; CHECK-NEXT:    [[V2_LANE_2:%.*]] = extractelement <4 x double> [[V_2]], i32 2
-; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <2 x double> poison, double [[V1_LANE_1]], i32 0
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x double> [[TMP0]], double [[V1_LANE_2]], i32 1
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[V2_LANE_2]], i32 0
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x double> [[TMP2]], <2 x double> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP3:%.*]] = fmul <2 x double> [[TMP1]], [[SHUFFLE]]
-; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <2 x double> [[TMP3]], <2 x double> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <4 x double> [[V_1]], <4 x double> poison, <2 x i32> <i32 1, i32 2>
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x double> [[V_2]], <4 x double> poison, <2 x i32> <i32 2, i32 2>
+; CHECK-NEXT:    [[TMP2:%.*]] = fmul <2 x double> [[TMP0]], [[TMP1]]
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <2 x double> [[TMP2]], <2 x double> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
 ; CHECK-NEXT:    call void @use(double [[V1_LANE_1]])
 ; CHECK-NEXT:    call void @use(double [[V1_LANE_2]])
-; CHECK-NEXT:    store <4 x double> [[TMP4]], ptr [[PTR_1]], align 8
+; CHECK-NEXT:    store <4 x double> [[TMP3]], ptr [[PTR_1]], align 8
 ; CHECK-NEXT:    ret void
 ;
 bb:
@@ -223,23 +209,16 @@ define void @noop_extracts_existing_vector_4_lanes(ptr %ptr.1, ptr %ptr.2) {
 ; CHECK-NEXT:    [[V1_LANE_2:%.*]] = extractelement <9 x double> [[V_1]], i32 2
 ; CHECK-NEXT:    [[V1_LANE_3:%.*]] = extractelement <9 x double> [[V_1]], i32 3
 ; CHECK-NEXT:    [[V_2:%.*]] = load <4 x double>, ptr [[PTR_2:%.*]], align 16
-; CHECK-NEXT:    [[V2_LANE_0:%.*]] = extractelement <4 x double> [[V_2]], i32 0
 ; CHECK-NEXT:    [[V2_LANE_1:%.*]] = extractelement <4 x double> [[V_2]], i32 1
-; CHECK-NEXT:    [[V2_LANE_2:%.*]] = extractelement <4 x double> [[V_2]], i32 2
-; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <4 x double> poison, double [[V1_LANE_2]], i32 0
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <4 x double> [[TMP0]], double [[V1_LANE_3]], i32 1
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <4 x double> [[TMP1]], double [[V1_LANE_0]], i32 2
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <4 x double> [[TMP2]], double [[V1_LANE_1]], i32 3
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <4 x double> poison, double [[V2_LANE_2]], i32 0
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <4 x double> [[TMP4]], double [[V2_LANE_0]], i32 1
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <4 x double> [[TMP5]], <4 x double> poison, <4 x i32> <i32 0, i32 0, i32 0, i32 1>
-; CHECK-NEXT:    [[TMP6:%.*]] = fmul <4 x double> [[TMP3]], [[SHUFFLE]]
-; CHECK-NEXT:    [[TMP7:%.*]] = shufflevector <4 x double> [[TMP6]], <4 x double> poison, <9 x i32> <i32 0, i32 1, i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <9 x double> [[V_1]], <9 x double> poison, <4 x i32> <i32 2, i32 3, i32 0, i32 1>
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x double> [[V_2]], <4 x double> poison, <4 x i32> <i32 2, i32 2, i32 2, i32 0>
+; CHECK-NEXT:    [[TMP2:%.*]] = fmul <4 x double> [[TMP0]], [[TMP1]]
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <4 x double> [[TMP2]], <4 x double> poison, <9 x i32> <i32 0, i32 1, i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
 ; CHECK-NEXT:    call void @use(double [[V1_LANE_0]])
 ; CHECK-NEXT:    call void @use(double [[V1_LANE_1]])
 ; CHECK-NEXT:    call void @use(double [[V1_LANE_2]])
 ; CHECK-NEXT:    call void @use(double [[V1_LANE_3]])
-; CHECK-NEXT:    store <9 x double> [[TMP7]], ptr [[PTR_1]], align 8
+; CHECK-NEXT:    store <9 x double> [[TMP3]], ptr [[PTR_1]], align 8
 ; CHECK-NEXT:    ret void
 ;
 bb:
@@ -279,24 +258,15 @@ define void @extracts_jumbled_4_lanes(ptr %ptr.1, ptr %ptr.2) {
 ; CHECK-NEXT:    [[V1_LANE_2:%.*]] = extractelement <9 x double> [[V_1]], i32 2
 ; CHECK-NEXT:    [[V1_LANE_3:%.*]] = extractelement <9 x double> [[V_1]], i32 3
 ; CHECK-NEXT:    [[V_2:%.*]] = load <4 x double>, ptr [[PTR_2:%.*]], align 16
-; CHECK-NEXT:    [[V2_LANE_0:%.*]] = extractelement <4 x double> [[V_2]], i32 0
-; CHECK-NEXT:    [[V2_LANE_1:%.*]] = extractelement <4 x double> [[V_2]], i32 1
-; CHECK-NEXT:    [[V2_LANE_2:%.*]] = extractelement <4 x double> [[V_2]], i32 2
-; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <4 x double> poison, double [[V1_LANE_0]], i32 0
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <4 x double> [[TMP0]], double [[V1_LANE_2]], i32 1
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <4 x double> [[TMP1]], double [[V1_LANE_1]], i32 2
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <4 x double> [[TMP2]], double [[V1_LANE_3]], i32 3
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <4 x double> poison, double [[V2_LANE_2]], i32 0
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <4 x double> [[TMP4]], double [[V2_LANE_1]], i32 1
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <4 x double> [[TMP5]], double [[V2_LANE_0]], i32 3
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <4 x double> [[TMP6]], <4 x double> poison, <4 x i32> <i32 0, i32 1, i32 0, i32 3>
-; CHECK-NEXT:    [[TMP7:%.*]] = fmul <4 x double> [[TMP3]], [[SHUFFLE]]
-; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <4 x double> [[TMP7]], <4 x double> poison, <9 x i32> <i32 0, i32 1, i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <9 x double> [[V_1]], <9 x double> poison, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x double> [[V_2]], <4 x double> poison, <4 x i32> <i32 2, i32 1, i32 2, i32 0>
+; CHECK-NEXT:    [[TMP2:%.*]] = fmul <4 x double> [[TMP0]], [[TMP1]]
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <4 x double> [[TMP2]], <4 x double> poison, <9 x i32> <i32 0, i32 1, i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
 ; CHECK-NEXT:    call void @use(double [[V1_LANE_0]])
 ; CHECK-NEXT:    call void @use(double [[V1_LANE_1]])
 ; CHECK-NEXT:    call void @use(double [[V1_LANE_2]])
 ; CHECK-NEXT:    call void @use(double [[V1_LANE_3]])
-; CHECK-NEXT:    store <9 x double> [[TMP8]], ptr [[PTR_1]], align 8
+; CHECK-NEXT:    store <9 x double> [[TMP3]], ptr [[PTR_1]], align 8
 ; CHECK-NEXT:    ret void
 ;
 bb:
@@ -333,51 +303,22 @@ define void @noop_extracts_9_lanes(ptr %ptr.1, ptr %ptr.2) {
 ; CHECK-LABEL: @noop_extracts_9_lanes(
 ; CHECK-NEXT:  bb:
 ; CHECK-NEXT:    [[V_1:%.*]] = load <9 x double>, ptr [[PTR_1:%.*]], align 8
-; CHECK-NEXT:    [[V1_LANE_0:%.*]] = extractelement <9 x double> [[V_1]], i32 0
-; CHECK-NEXT:    [[V1_LANE_1:%.*]] = extractelement <9 x double> [[V_1]], i32 1
 ; CHECK-NEXT:    [[V1_LANE_2:%.*]] = extractelement <9 x double> [[V_1]], i32 2
-; CHECK-NEXT:    [[V1_LANE_3:%.*]] = extractelement <9 x double> [[V_1]], i32 3
-; CHECK-NEXT:    [[V1_LANE_4:%.*]] = extractelement <9 x double> [[V_1]], i32 4
 ; CHECK-NEXT:    [[V1_LANE_5:%.*]] = extractelement <9 x double> [[V_1]], i32 5
-; CHECK-NEXT:    [[V1_LANE_6:%.*]] = extractelement <9 x double> [[V_1]], i32 6
-; CHECK-NEXT:    [[V1_LANE_7:%.*]] = extractelement <9 x double> [[V_1]], i32 7
-; CHECK-NEXT:    [[V1_LANE_8:%.*]] = extractelement <9 x double> [[V_1]], i32 8
 ; CHECK-NEXT:    [[V_2:%.*]] = load <4 x double>, ptr [[PTR_2:%.*]], align 16
 ; CHECK-NEXT:    [[V2_LANE_0:%.*]] = extractelement <4 x double> [[V_2]], i32 0
-; CHECK-NEXT:    [[V2_LANE_1:%.*]] = extractelement <4 x double> [[V_2]], i32 1
-; CHECK-NEXT:    [[V2_LANE_2:%.*]] = extractelement <4 x double> [[V_2]], i32 2
-; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <8 x double> poison, double [[V1_LANE_3]], i32 0
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <8 x double> [[TMP0]], double [[V1_LANE_4]], i32 1
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <8 x double> [[TMP1]], double [[V1_LANE_5]], i32 2
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <8 x double> [[TMP2]], double [[V1_LANE_6]], i32 3
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <8 x double> [[TMP3]], double [[V1_LANE_7]], i32 4
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <8 x double> [[TMP4]], double [[V1_LANE_8]], i32 5
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <8 x double> [[TMP5]], double [[V1_LANE_0]], i32 6
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <8 x double> [[TMP6]], double [[V1_LANE_1]], i32 7
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <8 x double> poison, double [[V2_LANE_0]], i32 0
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <8 x double> [[TMP8]], double [[V2_LANE_2]], i32 1
-; CHECK-NEXT:    [[TMP10:%.*]] = insertelement <8 x double> [[TMP9]], double [[V2_LANE_1]], i32 2
-; CHECK-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <8 x double> [[TMP10]], <8 x double> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 0, i32 1, i32 0, i32 1, i32 2>
-; CHECK-NEXT:    [[TMP11:%.*]] = fmul <8 x double> [[TMP7]], [[SHUFFLE1]]
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <9 x double> [[V_1]], <9 x double> poison, <8 x i32> <i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 0, i32 1>
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x double> [[V_2]], <4 x double> poison, <8 x i32> <i32 0, i32 2, i32 1, i32 0, i32 2, i32 0, i32 2, i32 1>
+; CHECK-NEXT:    [[TMP2:%.*]] = fmul <8 x double> [[TMP0]], [[TMP1]]
 ; CHECK-NEXT:    [[A_LANE_8:%.*]] = fmul double [[V1_LANE_2]], [[V2_LANE_0]]
-; CHECK-NEXT:    [[TMP12:%.*]] = shufflevector <8 x double> [[TMP11]], <8 x double> poison, <9 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 undef>
-; CHECK-NEXT:    [[A_INS_8:%.*]] = insertelement <9 x double> [[TMP12]], double [[A_LANE_8]], i32 8
-; CHECK-NEXT:    [[TMP13:%.*]] = insertelement <8 x double> poison, double [[V1_LANE_6]], i32 0
-; CHECK-NEXT:    [[TMP14:%.*]] = insertelement <8 x double> [[TMP13]], double [[V1_LANE_7]], i32 1
-; CHECK-NEXT:    [[TMP15:%.*]] = insertelement <8 x double> [[TMP14]], double [[V1_LANE_8]], i32 2
-; CHECK-NEXT:    [[TMP16:%.*]] = insertelement <8 x double> [[TMP15]], double [[V1_LANE_0]], i32 3
-; CHECK-NEXT:    [[TMP17:%.*]] = insertelement <8 x double> [[TMP16]], double [[V1_LANE_1]], i32 4
-; CHECK-NEXT:    [[TMP18:%.*]] = insertelement <8 x double> [[TMP17]], double [[V1_LANE_2]], i32 5
-; CHECK-NEXT:    [[TMP19:%.*]] = insertelement <8 x double> [[TMP18]], double [[V1_LANE_3]], i32 6
-; CHECK-NEXT:    [[TMP20:%.*]] = insertelement <8 x double> [[TMP19]], double [[V1_LANE_4]], i32 7
-; CHECK-NEXT:    [[TMP21:%.*]] = insertelement <8 x double> poison, double [[V2_LANE_2]], i32 0
-; CHECK-NEXT:    [[TMP22:%.*]] = insertelement <8 x double> [[TMP21]], double [[V2_LANE_1]], i32 1
-; CHECK-NEXT:    [[TMP23:%.*]] = insertelement <8 x double> [[TMP22]], double [[V2_LANE_0]], i32 2
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <8 x double> [[TMP23]], <8 x double> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 0, i32 1, i32 2, i32 0, i32 1>
-; CHECK-NEXT:    [[TMP24:%.*]] = fmul <8 x double> [[TMP20]], [[SHUFFLE]]
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <8 x double> [[TMP2]], <8 x double> poison, <9 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 undef>
+; CHECK-NEXT:    [[A_INS_8:%.*]] = insertelement <9 x double> [[TMP3]], double [[A_LANE_8]], i32 8
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <9 x double> [[V_1]], <9 x double> poison, <8 x i32> <i32 6, i32 7, i32 8, i32 0, i32 1, i32 2, i32 3, i32 4>
+; CHECK-NEXT:    [[TMP5:%.*]] = shufflevector <4 x double> [[V_2]], <4 x double> poison, <8 x i32> <i32 2, i32 1, i32 0, i32 2, i32 1, i32 0, i32 2, i32 1>
+; CHECK-NEXT:    [[TMP6:%.*]] = fmul <8 x double> [[TMP4]], [[TMP5]]
 ; CHECK-NEXT:    [[B_LANE_8:%.*]] = fmul double [[V1_LANE_5]], [[V2_LANE_0]]
-; CHECK-NEXT:    [[TMP25:%.*]] = shufflevector <8 x double> [[TMP24]], <8 x double> poison, <9 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 undef>
-; CHECK-NEXT:    [[B_INS_8:%.*]] = insertelement <9 x double> [[TMP25]], double [[B_LANE_8]], i32 8
+; CHECK-NEXT:    [[TMP7:%.*]] = shufflevector <8 x double> [[TMP6]], <8 x double> poison, <9 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 undef>
+; CHECK-NEXT:    [[B_INS_8:%.*]] = insertelement <9 x double> [[TMP7]], double [[B_LANE_8]], i32 8
 ; CHECK-NEXT:    [[RES:%.*]] = fsub <9 x double> [[A_INS_8]], [[B_INS_8]]
 ; CHECK-NEXT:    store <9 x double> [[RES]], ptr [[PTR_1]], align 8
 ; CHECK-NEXT:    ret void
@@ -450,47 +391,22 @@ define void @first_mul_chain_jumbled(ptr %ptr.1, ptr %ptr.2) {
 ; CHECK-LABEL: @first_mul_chain_jumbled(
 ; CHECK-NEXT:  bb:
 ; CHECK-NEXT:    [[V_1:%.*]] = load <9 x double>, ptr [[PTR_1:%.*]], align 8
-; CHECK-NEXT:    [[V1_LANE_0:%.*]] = extractelement <9 x double> [[V_1]], i32 0
-; CHECK-NEXT:    [[V1_LANE_1:%.*]] = extractelement <9 x double> [[V_1]], i32 1
 ; CHECK-NEXT:    [[V1_LANE_2:%.*]] = extractelement <9 x double> [[V_1]], i32 2
-; CHECK-NEXT:    [[V1_LANE_3:%.*]] = extractelement <9 x double> [[V_1]], i32 3
-; CHECK-NEXT:    [[V1_LANE_4:%.*]] = extractelement <9 x double> [[V_1]], i32 4
 ; CHECK-NEXT:    [[V1_LANE_5:%.*]] = extractelement <9 x double> [[V_1]], i32 5
-; CHECK-NEXT:    [[V1_LANE_6:%.*]] = extractelement <9 x double> [[V_1]], i32 6
-; CHECK-NEXT:    [[V1_LANE_7:%.*]] = extractelement <9 x double> [[V_1]], i32 7
-; CHECK-NEXT:    [[V1_LANE_8:%.*]] = extractelement <9 x double> [[V_1]], i32 8
 ; CHECK-NEXT:    [[V_2:%.*]] = load <4 x double>, ptr [[PTR_2:%.*]], align 16
 ; CHECK-NEXT:    [[V2_LANE_0:%.*]] = extractelement <4 x double> [[V_2]], i32 0
 ; CHECK-NEXT:    [[V2_LANE_1:%.*]] = extractelement <4 x double> [[V_2]], i32 1
-; CHECK-NEXT:    [[V2_LANE_2:%.*]] = extractelement <4 x double> [[V_2]], i32 2
-; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <8 x double> poison, double [[V1_LANE_4]], i32 0
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <8 x double> [[TMP0]], double [[V1_LANE_3]], i32 1
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <8 x double> [[TMP1]], double [[V1_LANE_6]], i32 2
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <8 x double> [[TMP2]], double [[V1_LANE_5]], i32 3
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <8 x double> [[TMP3]], double [[V1_LANE_8]], i32 4
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <8 x double> [[TMP4]], double [[V1_LANE_7]], i32 5
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <8 x double> [[TMP5]], double [[V1_LANE_1]], i32 6
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <8 x double> [[TMP6]], double [[V1_LANE_0]], i32 7
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <8 x double> poison, double [[V2_LANE_1]], i32 0
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <8 x double> [[TMP8]], double [[V2_LANE_0]], i32 1
-; CHECK-NEXT:    [[TMP10:%.*]] = insertelement <8 x double> [[TMP9]], double [[V2_LANE_2]], i32 2
-; CHECK-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <8 x double> [[TMP10]], <8 x double> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 1, i32 2, i32 0, i32 1, i32 2>
-; CHECK-NEXT:    [[TMP11:%.*]] = fmul <8 x double> [[TMP7]], [[SHUFFLE1]]
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <9 x double> [[V_1]], <9 x double> poison, <8 x i32> <i32 4, i32 3, i32 6, i32 5, i32 8, i32 7, i32 1, i32 0>
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x double> [[V_2]], <4 x double> poison, <8 x i32> <i32 1, i32 0, i32 2, i32 0, i32 2, i32 1, i32 0, i32 2>
+; CHECK-NEXT:    [[TMP2:%.*]] = fmul <8 x double> [[TMP0]], [[TMP1]]
 ; CHECK-NEXT:    [[A_LANE_8:%.*]] = fmul double [[V1_LANE_2]], [[V2_LANE_1]]
-; CHECK-NEXT:    [[TMP12:%.*]] = shufflevector <8 x double> [[TMP11]], <8 x double> poison, <9 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 undef>
-; CHECK-NEXT:    [[A_INS_8:%.*]] = insertelement <9 x double> [[TMP12]], double [[A_LANE_8]], i32 8
-; CHECK-NEXT:    [[TMP13:%.*]] = insertelement <8 x double> poison, double [[V1_LANE_6]], i32 0
-; CHECK-NEXT:    [[TMP14:%.*]] = insertelement <8 x double> [[TMP13]], double [[V1_LANE_7]], i32 1
-; CHECK-NEXT:    [[TMP15:%.*]] = insertelement <8 x double> [[TMP14]], double [[V1_LANE_8]], i32 2
-; CHECK-NEXT:    [[TMP16:%.*]] = insertelement <8 x double> [[TMP15]], double [[V1_LANE_0]], i32 3
-; CHECK-NEXT:    [[TMP17:%.*]] = insertelement <8 x double> [[TMP16]], double [[V1_LANE_1]], i32 4
-; CHECK-NEXT:    [[TMP18:%.*]] = insertelement <8 x double> [[TMP17]], double [[V1_LANE_2]], i32 5
-; CHECK-NEXT:    [[TMP19:%.*]] = insertelement <8 x double> [[TMP18]], double [[V1_LANE_3]], i32 6
-; CHECK-NEXT:    [[TMP20:%.*]] = insertelement <8 x double> [[TMP19]], double [[V1_LANE_4]], i32 7
-; CHECK-NEXT:    [[TMP21:%.*]] = fmul <8 x double> [[TMP20]], [[SHUFFLE1]]
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <8 x double> [[TMP2]], <8 x double> poison, <9 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 undef>
+; CHECK-NEXT:    [[A_INS_8:%.*]] = insertelement <9 x double> [[TMP3]], double [[A_LANE_8]], i32 8
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <9 x double> [[V_1]], <9 x double> poison, <8 x i32> <i32 6, i32 7, i32 8, i32 0, i32 1, i32 2, i32 3, i32 4>
+; CHECK-NEXT:    [[TMP5:%.*]] = fmul <8 x double> [[TMP4]], [[TMP1]]
 ; CHECK-NEXT:    [[B_LANE_8:%.*]] = fmul double [[V1_LANE_5]], [[V2_LANE_0]]
-; CHECK-NEXT:    [[TMP22:%.*]] = shufflevector <8 x double> [[TMP21]], <8 x double> poison, <9 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 undef>
-; CHECK-NEXT:    [[B_INS_8:%.*]] = insertelement <9 x double> [[TMP22]], double [[B_LANE_8]], i32 8
+; CHECK-NEXT:    [[TMP6:%.*]] = shufflevector <8 x double> [[TMP5]], <8 x double> poison, <9 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 undef>
+; CHECK-NEXT:    [[B_INS_8:%.*]] = insertelement <9 x double> [[TMP6]], double [[B_LANE_8]], i32 8
 ; CHECK-NEXT:    [[RES:%.*]] = fsub <9 x double> [[A_INS_8]], [[B_INS_8]]
 ; CHECK-NEXT:    store <9 x double> [[RES]], ptr [[PTR_1]], align 8
 ; CHECK-NEXT:    ret void
@@ -563,51 +479,23 @@ define void @first_and_second_mul_chain_jumbled(ptr %ptr.1, ptr %ptr.2) {
 ; CHECK-LABEL: @first_and_second_mul_chain_jumbled(
 ; CHECK-NEXT:  bb:
 ; CHECK-NEXT:    [[V_1:%.*]] = load <9 x double>, ptr [[PTR_1:%.*]], align 8
-; CHECK-NEXT:    [[V1_LANE_0:%.*]] = extractelement <9 x double> [[V_1]], i32 0
-; CHECK-NEXT:    [[V1_LANE_1:%.*]] = extractelement <9 x double> [[V_1]], i32 1
 ; CHECK-NEXT:    [[V1_LANE_2:%.*]] = extractelement <9 x double> [[V_1]], i32 2
-; CHECK-NEXT:    [[V1_LANE_3:%.*]] = extractelement <9 x double> [[V_1]], i32 3
 ; CHECK-NEXT:    [[V1_LANE_4:%.*]] = extractelement <9 x double> [[V_1]], i32 4
-; CHECK-NEXT:    [[V1_LANE_5:%.*]] = extractelement <9 x double> [[V_1]], i32 5
-; CHECK-NEXT:    [[V1_LANE_6:%.*]] = extractelement <9 x double> [[V_1]], i32 6
-; CHECK-NEXT:    [[V1_LANE_7:%.*]] = extractelement <9 x double> [[V_1]], i32 7
-; CHECK-NEXT:    [[V1_LANE_8:%.*]] = extractelement <9 x double> [[V_1]], i32 8
 ; CHECK-NEXT:    [[V_2:%.*]] = load <4 x double>, ptr [[PTR_2:%.*]], align 16
 ; CHECK-NEXT:    [[V2_LANE_0:%.*]] = extractelement <4 x double> [[V_2]], i32 0
-; CHECK-NEXT:    [[V2_LANE_1:%.*]] = extractelement <4 x double> [[V_2]], i32 1
 ; CHECK-NEXT:    [[V2_LANE_2:%.*]] = extractelement <4 x double> [[V_2]], i32 2
-; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <8 x double> poison, double [[V1_LANE_4]], i32 0
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <8 x double> [[TMP0]], double [[V1_LANE_3]], i32 1
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <8 x double> [[TMP1]], double [[V1_LANE_5]], i32 2
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <8 x double> [[TMP2]], double [[V1_LANE_6]], i32 3
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <8 x double> [[TMP3]], double [[V1_LANE_8]], i32 4
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <8 x double> [[TMP4]], double [[V1_LANE_7]], i32 5
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <8 x double> [[TMP5]], double [[V1_LANE_1]], i32 6
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <8 x double> [[TMP6]], double [[V1_LANE_0]], i32 7
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <8 x double> poison, double [[V2_LANE_0]], i32 0
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <8 x double> [[TMP8]], double [[V2_LANE_2]], i32 1
-; CHECK-NEXT:    [[TMP10:%.*]] = insertelement <8 x double> [[TMP9]], double [[V2_LANE_1]], i32 2
-; CHECK-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <8 x double> [[TMP10]], <8 x double> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 1, i32 2, i32 0, i32 1, i32 2>
-; CHECK-NEXT:    [[TMP11:%.*]] = fmul <8 x double> [[TMP7]], [[SHUFFLE1]]
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <9 x double> [[V_1]], <9 x double> poison, <8 x i32> <i32 4, i32 3, i32 5, i32 6, i32 8, i32 7, i32 1, i32 0>
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x double> [[V_2]], <4 x double> poison, <8 x i32> <i32 0, i32 2, i32 1, i32 2, i32 1, i32 0, i32 2, i32 1>
+; CHECK-NEXT:    [[TMP2:%.*]] = fmul <8 x double> [[TMP0]], [[TMP1]]
 ; CHECK-NEXT:    [[A_LANE_8:%.*]] = fmul double [[V1_LANE_2]], [[V2_LANE_0]]
-; CHECK-NEXT:    [[TMP12:%.*]] = shufflevector <8 x double> [[TMP11]], <8 x double> poison, <9 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 undef>
-; CHECK-NEXT:    [[A_INS_8:%.*]] = insertelement <9 x double> [[TMP12]], double [[A_LANE_8]], i32 8
-; CHECK-NEXT:    [[TMP13:%.*]] = insertelement <8 x double> poison, double [[V1_LANE_7]], i32 0
-; CHECK-NEXT:    [[TMP14:%.*]] = insertelement <8 x double> [[TMP13]], double [[V1_LANE_6]], i32 1
-; CHECK-NEXT:    [[TMP15:%.*]] = insertelement <8 x double> [[TMP14]], double [[V1_LANE_8]], i32 2
-; CHECK-NEXT:    [[TMP16:%.*]] = insertelement <8 x double> [[TMP15]], double [[V1_LANE_1]], i32 3
-; CHECK-NEXT:    [[TMP17:%.*]] = insertelement <8 x double> [[TMP16]], double [[V1_LANE_0]], i32 4
-; CHECK-NEXT:    [[TMP18:%.*]] = insertelement <8 x double> [[TMP17]], double [[V1_LANE_3]], i32 5
-; CHECK-NEXT:    [[TMP19:%.*]] = insertelement <8 x double> [[TMP18]], double [[V1_LANE_2]], i32 6
-; CHECK-NEXT:    [[TMP20:%.*]] = insertelement <8 x double> [[TMP19]], double [[V1_LANE_5]], i32 7
-; CHECK-NEXT:    [[TMP21:%.*]] = insertelement <8 x double> poison, double [[V2_LANE_2]], i32 0
-; CHECK-NEXT:    [[TMP22:%.*]] = insertelement <8 x double> [[TMP21]], double [[V2_LANE_1]], i32 1
-; CHECK-NEXT:    [[TMP23:%.*]] = insertelement <8 x double> [[TMP22]], double [[V2_LANE_0]], i32 2
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <8 x double> [[TMP23]], <8 x double> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 0, i32 2, i32 0, i32 1, i32 2>
-; CHECK-NEXT:    [[TMP24:%.*]] = fmul <8 x double> [[TMP20]], [[SHUFFLE]]
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <8 x double> [[TMP2]], <8 x double> poison, <9 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 undef>
+; CHECK-NEXT:    [[A_INS_8:%.*]] = insertelement <9 x double> [[TMP3]], double [[A_LANE_8]], i32 8
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <9 x double> [[V_1]], <9 x double> poison, <8 x i32> <i32 7, i32 6, i32 8, i32 1, i32 0, i32 3, i32 2, i32 5>
+; CHECK-NEXT:    [[TMP5:%.*]] = shufflevector <4 x double> [[V_2]], <4 x double> poison, <8 x i32> <i32 2, i32 1, i32 0, i32 2, i32 0, i32 2, i32 1, i32 0>
+; CHECK-NEXT:    [[TMP6:%.*]] = fmul <8 x double> [[TMP4]], [[TMP5]]
 ; CHECK-NEXT:    [[B_LANE_8:%.*]] = fmul double [[V1_LANE_4]], [[V2_LANE_2]]
-; CHECK-NEXT:    [[TMP25:%.*]] = shufflevector <8 x double> [[TMP24]], <8 x double> poison, <9 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 undef>
-; CHECK-NEXT:    [[B_INS_8:%.*]] = insertelement <9 x double> [[TMP25]], double [[B_LANE_8]], i32 8
+; CHECK-NEXT:    [[TMP7:%.*]] = shufflevector <8 x double> [[TMP6]], <8 x double> poison, <9 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 undef>
+; CHECK-NEXT:    [[B_INS_8:%.*]] = insertelement <9 x double> [[TMP7]], double [[B_LANE_8]], i32 8
 ; CHECK-NEXT:    [[RES:%.*]] = fsub <9 x double> [[A_INS_8]], [[B_INS_8]]
 ; CHECK-NEXT:    store <9 x double> [[RES]], ptr [[PTR_1]], align 8
 ; CHECK-NEXT:    ret void
diff --git a/llvm/test/Transforms/SLPVectorizer/AMDGPU/add_sub_sat-inseltpoison.ll b/llvm/test/Transforms/SLPVectorizer/AMDGPU/add_sub_sat-inseltpoison.ll
index 6fd4b14eb799e..e6d80675635a2 100644
--- a/llvm/test/Transforms/SLPVectorizer/AMDGPU/add_sub_sat-inseltpoison.ll
+++ b/llvm/test/Transforms/SLPVectorizer/AMDGPU/add_sub_sat-inseltpoison.ll
@@ -244,8 +244,8 @@ define <3 x i16> @uadd_sat_v3i16(<3 x i16> %arg0, <3 x i16> %arg1) {
 ; GFX8-NEXT:  bb:
 ; GFX8-NEXT:    [[ARG0_2:%.*]] = extractelement <3 x i16> [[ARG0:%.*]], i64 2
 ; GFX8-NEXT:    [[ARG1_2:%.*]] = extractelement <3 x i16> [[ARG1:%.*]], i64 2
-; GFX8-NEXT:    [[TMP0:%.*]] = shufflevector <3 x i16> [[ARG0]], <3 x i16> undef, <2 x i32> <i32 0, i32 1>
-; GFX8-NEXT:    [[TMP1:%.*]] = shufflevector <3 x i16> [[ARG1]], <3 x i16> undef, <2 x i32> <i32 0, i32 1>
+; GFX8-NEXT:    [[TMP0:%.*]] = shufflevector <3 x i16> [[ARG0]], <3 x i16> poison, <2 x i32> <i32 0, i32 1>
+; GFX8-NEXT:    [[TMP1:%.*]] = shufflevector <3 x i16> [[ARG1]], <3 x i16> poison, <2 x i32> <i32 0, i32 1>
 ; GFX8-NEXT:    [[TMP2:%.*]] = call <2 x i16> @llvm.uadd.sat.v2i16(<2 x i16> [[TMP0]], <2 x i16> [[TMP1]])
 ; GFX8-NEXT:    [[ADD_2:%.*]] = call i16 @llvm.uadd.sat.i16(i16 [[ARG0_2]], i16 [[ARG1_2]])
 ; GFX8-NEXT:    [[TMP3:%.*]] = shufflevector <2 x i16> [[TMP2]], <2 x i16> poison, <3 x i32> <i32 0, i32 1, i32 undef>
@@ -291,11 +291,11 @@ define <4 x i16> @uadd_sat_v4i16(<4 x i16> %arg0, <4 x i16> %arg1) {
 ;
 ; GFX8-LABEL: @uadd_sat_v4i16(
 ; GFX8-NEXT:  bb:
-; GFX8-NEXT:    [[TMP0:%.*]] = shufflevector <4 x i16> [[ARG0:%.*]], <4 x i16> undef, <2 x i32> <i32 0, i32 1>
-; GFX8-NEXT:    [[TMP1:%.*]] = shufflevector <4 x i16> [[ARG1:%.*]], <4 x i16> undef, <2 x i32> <i32 0, i32 1>
+; GFX8-NEXT:    [[TMP0:%.*]] = shufflevector <4 x i16> [[ARG0:%.*]], <4 x i16> poison, <2 x i32> <i32 0, i32 1>
+; GFX8-NEXT:    [[TMP1:%.*]] = shufflevector <4 x i16> [[ARG1:%.*]], <4 x i16> poison, <2 x i32> <i32 0, i32 1>
 ; GFX8-NEXT:    [[TMP2:%.*]] = call <2 x i16> @llvm.uadd.sat.v2i16(<2 x i16> [[TMP0]], <2 x i16> [[TMP1]])
-; GFX8-NEXT:    [[TMP3:%.*]] = shufflevector <4 x i16> [[ARG0]], <4 x i16> undef, <2 x i32> <i32 2, i32 3>
-; GFX8-NEXT:    [[TMP4:%.*]] = shufflevector <4 x i16> [[ARG1]], <4 x i16> undef, <2 x i32> <i32 2, i32 3>
+; GFX8-NEXT:    [[TMP3:%.*]] = shufflevector <4 x i16> [[ARG0]], <4 x i16> poison, <2 x i32> <i32 2, i32 3>
+; GFX8-NEXT:    [[TMP4:%.*]] = shufflevector <4 x i16> [[ARG1]], <4 x i16> poison, <2 x i32> <i32 2, i32 3>
 ; GFX8-NEXT:    [[TMP5:%.*]] = call <2 x i16> @llvm.uadd.sat.v2i16(<2 x i16> [[TMP3]], <2 x i16> [[TMP4]])
 ; GFX8-NEXT:    [[INS_31:%.*]] = shufflevector <2 x i16> [[TMP2]], <2 x i16> [[TMP5]], <4 x i32> <i32 0, i32 1, i32 2, i32 3>
 ; GFX8-NEXT:    ret <4 x i16> [[INS_31]]
diff --git a/llvm/test/Transforms/SLPVectorizer/AMDGPU/add_sub_sat.ll b/llvm/test/Transforms/SLPVectorizer/AMDGPU/add_sub_sat.ll
index 24fee6387beb5..e99be89ef9bba 100644
--- a/llvm/test/Transforms/SLPVectorizer/AMDGPU/add_sub_sat.ll
+++ b/llvm/test/Transforms/SLPVectorizer/AMDGPU/add_sub_sat.ll
@@ -244,8 +244,8 @@ define <3 x i16> @uadd_sat_v3i16(<3 x i16> %arg0, <3 x i16> %arg1) {
 ; GFX8-NEXT:  bb:
 ; GFX8-NEXT:    [[ARG0_2:%.*]] = extractelement <3 x i16> [[ARG0:%.*]], i64 2
 ; GFX8-NEXT:    [[ARG1_2:%.*]] = extractelement <3 x i16> [[ARG1:%.*]], i64 2
-; GFX8-NEXT:    [[TMP0:%.*]] = shufflevector <3 x i16> [[ARG0]], <3 x i16> undef, <2 x i32> <i32 0, i32 1>
-; GFX8-NEXT:    [[TMP1:%.*]] = shufflevector <3 x i16> [[ARG1]], <3 x i16> undef, <2 x i32> <i32 0, i32 1>
+; GFX8-NEXT:    [[TMP0:%.*]] = shufflevector <3 x i16> [[ARG0]], <3 x i16> poison, <2 x i32> <i32 0, i32 1>
+; GFX8-NEXT:    [[TMP1:%.*]] = shufflevector <3 x i16> [[ARG1]], <3 x i16> poison, <2 x i32> <i32 0, i32 1>
 ; GFX8-NEXT:    [[TMP2:%.*]] = call <2 x i16> @llvm.uadd.sat.v2i16(<2 x i16> [[TMP0]], <2 x i16> [[TMP1]])
 ; GFX8-NEXT:    [[ADD_2:%.*]] = call i16 @llvm.uadd.sat.i16(i16 [[ARG0_2]], i16 [[ARG1_2]])
 ; GFX8-NEXT:    [[TMP3:%.*]] = shufflevector <2 x i16> [[TMP2]], <2 x i16> poison, <3 x i32> <i32 0, i32 1, i32 undef>
@@ -291,11 +291,11 @@ define <4 x i16> @uadd_sat_v4i16(<4 x i16> %arg0, <4 x i16> %arg1) {
 ;
 ; GFX8-LABEL: @uadd_sat_v4i16(
 ; GFX8-NEXT:  bb:
-; GFX8-NEXT:    [[TMP0:%.*]] = shufflevector <4 x i16> [[ARG0:%.*]], <4 x i16> undef, <2 x i32> <i32 0, i32 1>
-; GFX8-NEXT:    [[TMP1:%.*]] = shufflevector <4 x i16> [[ARG1:%.*]], <4 x i16> undef, <2 x i32> <i32 0, i32 1>
+; GFX8-NEXT:    [[TMP0:%.*]] = shufflevector <4 x i16> [[ARG0:%.*]], <4 x i16> poison, <2 x i32> <i32 0, i32 1>
+; GFX8-NEXT:    [[TMP1:%.*]] = shufflevector <4 x i16> [[ARG1:%.*]], <4 x i16> poison, <2 x i32> <i32 0, i32 1>
 ; GFX8-NEXT:    [[TMP2:%.*]] = call <2 x i16> @llvm.uadd.sat.v2i16(<2 x i16> [[TMP0]], <2 x i16> [[TMP1]])
-; GFX8-NEXT:    [[TMP3:%.*]] = shufflevector <4 x i16> [[ARG0]], <4 x i16> undef, <2 x i32> <i32 2, i32 3>
-; GFX8-NEXT:    [[TMP4:%.*]] = shufflevector <4 x i16> [[ARG1]], <4 x i16> undef, <2 x i32> <i32 2, i32 3>
+; GFX8-NEXT:    [[TMP3:%.*]] = shufflevector <4 x i16> [[ARG0]], <4 x i16> poison, <2 x i32> <i32 2, i32 3>
+; GFX8-NEXT:    [[TMP4:%.*]] = shufflevector <4 x i16> [[ARG1]], <4 x i16> poison, <2 x i32> <i32 2, i32 3>
 ; GFX8-NEXT:    [[TMP5:%.*]] = call <2 x i16> @llvm.uadd.sat.v2i16(<2 x i16> [[TMP3]], <2 x i16> [[TMP4]])
 ; GFX8-NEXT:    [[INS_31:%.*]] = shufflevector <2 x i16> [[TMP2]], <2 x i16> [[TMP5]], <4 x i32> <i32 0, i32 1, i32 2, i32 3>
 ; GFX8-NEXT:    ret <4 x i16> [[INS_31]]
diff --git a/llvm/test/Transforms/SLPVectorizer/AMDGPU/crash_extract_subvector_cost.ll b/llvm/test/Transforms/SLPVectorizer/AMDGPU/crash_extract_subvector_cost.ll
index 7471d0f7d329c..68295bb0d78e3 100644
--- a/llvm/test/Transforms/SLPVectorizer/AMDGPU/crash_extract_subvector_cost.ll
+++ b/llvm/test/Transforms/SLPVectorizer/AMDGPU/crash_extract_subvector_cost.ll
@@ -4,16 +4,10 @@
 define <2 x i16> @uadd_sat_v9i16_combine_vi16(<9 x i16> %arg0, <9 x i16> %arg1) {
 ; CHECK-LABEL: @uadd_sat_v9i16_combine_vi16(
 ; CHECK-NEXT:  bb:
-; CHECK-NEXT:    [[ARG0_1:%.*]] = extractelement <9 x i16> undef, i64 7
-; CHECK-NEXT:    [[ARG0_2:%.*]] = extractelement <9 x i16> [[ARG0:%.*]], i64 8
-; CHECK-NEXT:    [[ARG1_1:%.*]] = extractelement <9 x i16> [[ARG1:%.*]], i64 7
-; CHECK-NEXT:    [[ARG1_2:%.*]] = extractelement <9 x i16> [[ARG1]], i64 8
-; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <2 x i16> poison, i16 [[ARG0_1]], i32 0
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x i16> [[TMP0]], i16 [[ARG0_2]], i32 1
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x i16> poison, i16 [[ARG1_1]], i32 0
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x i16> [[TMP2]], i16 [[ARG1_2]], i32 1
-; CHECK-NEXT:    [[TMP4:%.*]] = call <2 x i16> @llvm.uadd.sat.v2i16(<2 x i16> [[TMP1]], <2 x i16> [[TMP3]])
-; CHECK-NEXT:    ret <2 x i16> [[TMP4]]
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <9 x i16> [[ARG0:%.*]], <9 x i16> poison, <2 x i32> <i32 undef, i32 8>
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <9 x i16> [[ARG1:%.*]], <9 x i16> poison, <2 x i32> <i32 7, i32 8>
+; CHECK-NEXT:    [[TMP2:%.*]] = call <2 x i16> @llvm.uadd.sat.v2i16(<2 x i16> [[TMP0]], <2 x i16> [[TMP1]])
+; CHECK-NEXT:    ret <2 x i16> [[TMP2]]
 ;
 bb:
   %arg0.1 = extractelement <9 x i16> undef, i64 7
diff --git a/llvm/test/Transforms/SLPVectorizer/AMDGPU/phi-result-use-order.ll b/llvm/test/Transforms/SLPVectorizer/AMDGPU/phi-result-use-order.ll
index f870fb3a9bc1f..eb4117bbf7209 100644
--- a/llvm/test/Transforms/SLPVectorizer/AMDGPU/phi-result-use-order.ll
+++ b/llvm/test/Transforms/SLPVectorizer/AMDGPU/phi-result-use-order.ll
@@ -1,34 +1,24 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -passes=slp-vectorizer -S -mtriple=amdgcn-amd-amdhsa -mcpu=gfx906 < %s | FileCheck %s
 
 define <4 x half> @phis(i1 %cmp1, <4 x half> %in1, <4 x half> %in2)  {
 ; CHECK-LABEL: @phis(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[A0:%.*]] = extractelement <4 x half> [[IN1:%.*]], i64 0
-; CHECK-NEXT:    [[A1:%.*]] = extractelement <4 x half> [[IN1]], i64 1
-; CHECK-NEXT:    [[A2:%.*]] = extractelement <4 x half> [[IN1]], i64 2
-; CHECK-NEXT:    [[A3:%.*]] = extractelement <4 x half> [[IN1]], i64 3
-; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <2 x half> poison, half [[A0]], i32 0
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x half> [[TMP0]], half [[A1]], i32 1
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x half> poison, half [[A2]], i32 0
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x half> [[TMP2]], half [[A3]], i32 1
-; CHECK-NEXT:    br i1 [[CMP:%.*]], label [[BB1:%.*]], label [[BB0:%.*]]
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <4 x half> [[IN1:%.*]], <4 x half> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x half> [[IN1]], <4 x half> poison, <2 x i32> <i32 2, i32 3>
+; CHECK-NEXT:    br i1 [[CMP1:%.*]], label [[BB1:%.*]], label [[BB0:%.*]]
 ; CHECK:       bb0:
-; CHECK-NEXT:    [[B0:%.*]] = extractelement <4 x half> [[IN2:%.*]], i64 0
-; CHECK-NEXT:    [[B1:%.*]] = extractelement <4 x half> [[IN2]], i64 1
-; CHECK-NEXT:    [[B2:%.*]] = extractelement <4 x half> [[IN2]], i64 2
-; CHECK-NEXT:    [[B3:%.*]] = extractelement <4 x half> [[IN2]], i64 3
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x half> poison, half [[B0]], i32 0
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x half> [[TMP4]], half [[B1]], i32 1
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x half> poison, half [[B2]], i32 0
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <2 x half> [[TMP6]], half [[B3]], i32 1
-; CHECK-NEXT:    br label [[BB1:%.*]]
+; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <4 x half> [[IN2:%.*]], <4 x half> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <4 x half> [[IN2]], <4 x half> poison, <2 x i32> <i32 2, i32 3>
+; CHECK-NEXT:    br label [[BB1]]
 ; CHECK:       bb1:
-; CHECK-NEXT:    [[TMP8:%.*]] = phi <2 x half> [ [[TMP1]], %entry ], [ [[TMP5]], %bb0 ]
-; CHECK-NEXT:    [[TMP9:%.*]] = phi <2 x half> [ [[TMP3]], %entry ], [ [[TMP7]], %bb0 ]
-; CHECK-NEXT:    [[TMP10:%.*]] = shufflevector <2 x half> [[TMP8]], <2 x half> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; CHECK-NEXT:    [[TMP11:%.*]] = shufflevector <2 x half> [[TMP9]], <2 x half> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; CHECK-NEXT:    [[TMP12:%.*]] = shufflevector <4 x half> [[TMP10]], <4 x half> [[TMP11]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
-; CHECK-NEXT:    ret <4 x half> [[TMP12]]
+; CHECK-NEXT:    [[TMP4:%.*]] = phi <2 x half> [ [[TMP0]], [[ENTRY:%.*]] ], [ [[TMP2]], [[BB0]] ]
+; CHECK-NEXT:    [[TMP5:%.*]] = phi <2 x half> [ [[TMP1]], [[ENTRY]] ], [ [[TMP3]], [[BB0]] ]
+; CHECK-NEXT:    [[TMP6:%.*]] = shufflevector <2 x half> [[TMP4]], <2 x half> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; CHECK-NEXT:    [[TMP7:%.*]] = shufflevector <2 x half> [[TMP5]], <2 x half> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; CHECK-NEXT:    [[O31:%.*]] = shufflevector <4 x half> [[TMP6]], <4 x half> [[TMP7]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
+; CHECK-NEXT:    ret <4 x half> [[O31]]
+;
 entry:
   %a0 = extractelement <4 x half> %in1, i64 0
   %a1 = extractelement <4 x half> %in1, i64 1
@@ -59,32 +49,21 @@ bb1:
 define <4 x half> @phis_reverse(i1 %cmp1, <4 x half> %in1, <4 x half> %in2)  {
 ; CHECK-LABEL: @phis_reverse(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[A0:%.*]] = extractelement <4 x half> [[IN1:%.*]], i64 0
-; CHECK-NEXT:    [[A1:%.*]] = extractelement <4 x half> [[IN1]], i64 1
-; CHECK-NEXT:    [[A2:%.*]] = extractelement <4 x half> [[IN1]], i64 2
-; CHECK-NEXT:    [[A3:%.*]] = extractelement <4 x half> [[IN1]], i64 3
-; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <2 x half> poison, half [[A0]], i32 0
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x half> [[TMP0]], half [[A1]], i32 1
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x half> poison, half [[A2]], i32 0
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x half> [[TMP2]], half [[A3]], i32 1
-; CHECK-NEXT:    br i1 [[CMP:%.*]], label [[BB1:%.*]], label [[BB0:%.*]]
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <4 x half> [[IN1:%.*]], <4 x half> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x half> [[IN1]], <4 x half> poison, <2 x i32> <i32 2, i32 3>
+; CHECK-NEXT:    br i1 [[CMP1:%.*]], label [[BB1:%.*]], label [[BB0:%.*]]
 ; CHECK:       bb0:
-; CHECK-NEXT:    [[B0:%.*]] = extractelement <4 x half> [[IN2:%.*]], i64 0
-; CHECK-NEXT:    [[B1:%.*]] = extractelement <4 x half> [[IN2]], i64 1
-; CHECK-NEXT:    [[B2:%.*]] = extractelement <4 x half> [[IN2]], i64 2
-; CHECK-NEXT:    [[B3:%.*]] = extractelement <4 x half> [[IN2]], i64 3
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x half> poison, half [[B0]], i32 0
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x half> [[TMP4]], half [[B1]], i32 1
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x half> poison, half [[B2]], i32 0
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <2 x half> [[TMP6]], half [[B3]], i32 1
-; CHECK-NEXT:    br label [[BB1:%.*]]
+; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <4 x half> [[IN2:%.*]], <4 x half> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <4 x half> [[IN2]], <4 x half> poison, <2 x i32> <i32 2, i32 3>
+; CHECK-NEXT:    br label [[BB1]]
 ; CHECK:       bb1:
-; CHECK-NEXT:    [[TMP8:%.*]] = phi <2 x half> [ [[TMP1]], %entry ], [ [[TMP5]], %bb0 ]
-; CHECK-NEXT:    [[TMP9:%.*]] = phi <2 x half> [ [[TMP3]], %entry ], [ [[TMP7]], %bb0 ]
-; CHECK-NEXT:    [[TMP10:%.*]] = shufflevector <2 x half> [[TMP8]], <2 x half> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; CHECK-NEXT:    [[TMP11:%.*]] = shufflevector <2 x half> [[TMP9]], <2 x half> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; CHECK-NEXT:    [[TMP12:%.*]] = shufflevector <4 x half> [[TMP10]], <4 x half> [[TMP11]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
-; CHECK-NEXT:    ret <4 x half> [[TMP12]]
+; CHECK-NEXT:    [[TMP4:%.*]] = phi <2 x half> [ [[TMP0]], [[ENTRY:%.*]] ], [ [[TMP2]], [[BB0]] ]
+; CHECK-NEXT:    [[TMP5:%.*]] = phi <2 x half> [ [[TMP1]], [[ENTRY]] ], [ [[TMP3]], [[BB0]] ]
+; CHECK-NEXT:    [[TMP6:%.*]] = shufflevector <2 x half> [[TMP4]], <2 x half> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; CHECK-NEXT:    [[TMP7:%.*]] = shufflevector <2 x half> [[TMP5]], <2 x half> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; CHECK-NEXT:    [[O31:%.*]] = shufflevector <4 x half> [[TMP6]], <4 x half> [[TMP7]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
+; CHECK-NEXT:    ret <4 x half> [[O31]]
+;
 entry:
   %a0 = extractelement <4 x half> %in1, i64 0
   %a1 = extractelement <4 x half> %in1, i64 1
diff --git a/llvm/test/Transforms/SLPVectorizer/RISCV/floating-point.ll b/llvm/test/Transforms/SLPVectorizer/RISCV/floating-point.ll
new file mode 100644
index 0000000000000..984529e86a5ab
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/RISCV/floating-point.ll
@@ -0,0 +1,459 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
+; RUN: opt < %s -passes=slp-vectorizer -S -mtriple=riscv64 -mattr=+v,+f \
+; RUN:     -riscv-v-vector-bits-min=-1 -riscv-v-slp-max-vf=0 \
+; RUN:     | FileCheck %s
+; RUN: opt < %s -passes=slp-vectorizer -S -mtriple=riscv64 -mattr=+v,+f \
+; RUN:     | FileCheck %s --check-prefix=DEFAULT
+
+define void @fp_add(ptr %dst, ptr %p, ptr %q) {
+; CHECK-LABEL: define void @fp_add
+; CHECK-SAME: (ptr [[DST:%.*]], ptr [[P:%.*]], ptr [[Q:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[P]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x float>, ptr [[Q]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = fadd <4 x float> [[TMP0]], [[TMP1]]
+; CHECK-NEXT:    store <4 x float> [[TMP2]], ptr [[DST]], align 4
+; CHECK-NEXT:    ret void
+;
+; DEFAULT-LABEL: define void @fp_add
+; DEFAULT-SAME: (ptr [[DST:%.*]], ptr [[P:%.*]], ptr [[Q:%.*]]) #[[ATTR0:[0-9]+]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[E0:%.*]] = load float, ptr [[P]], align 4
+; DEFAULT-NEXT:    [[PE1:%.*]] = getelementptr inbounds float, ptr [[P]], i64 1
+; DEFAULT-NEXT:    [[E1:%.*]] = load float, ptr [[PE1]], align 4
+; DEFAULT-NEXT:    [[PE2:%.*]] = getelementptr inbounds float, ptr [[P]], i64 2
+; DEFAULT-NEXT:    [[E2:%.*]] = load float, ptr [[PE2]], align 4
+; DEFAULT-NEXT:    [[PE3:%.*]] = getelementptr inbounds float, ptr [[P]], i64 3
+; DEFAULT-NEXT:    [[E3:%.*]] = load float, ptr [[PE3]], align 4
+; DEFAULT-NEXT:    [[F0:%.*]] = load float, ptr [[Q]], align 4
+; DEFAULT-NEXT:    [[PF1:%.*]] = getelementptr inbounds float, ptr [[Q]], i64 1
+; DEFAULT-NEXT:    [[F1:%.*]] = load float, ptr [[PF1]], align 4
+; DEFAULT-NEXT:    [[PF2:%.*]] = getelementptr inbounds float, ptr [[Q]], i64 2
+; DEFAULT-NEXT:    [[F2:%.*]] = load float, ptr [[PF2]], align 4
+; DEFAULT-NEXT:    [[PF3:%.*]] = getelementptr inbounds float, ptr [[Q]], i64 3
+; DEFAULT-NEXT:    [[F3:%.*]] = load float, ptr [[PF3]], align 4
+; DEFAULT-NEXT:    [[A0:%.*]] = fadd float [[E0]], [[F0]]
+; DEFAULT-NEXT:    [[A1:%.*]] = fadd float [[E1]], [[F1]]
+; DEFAULT-NEXT:    [[A2:%.*]] = fadd float [[E2]], [[F2]]
+; DEFAULT-NEXT:    [[A3:%.*]] = fadd float [[E3]], [[F3]]
+; DEFAULT-NEXT:    store float [[A0]], ptr [[DST]], align 4
+; DEFAULT-NEXT:    [[PA1:%.*]] = getelementptr inbounds float, ptr [[DST]], i64 1
+; DEFAULT-NEXT:    store float [[A1]], ptr [[PA1]], align 4
+; DEFAULT-NEXT:    [[PA2:%.*]] = getelementptr inbounds float, ptr [[DST]], i64 2
+; DEFAULT-NEXT:    store float [[A2]], ptr [[PA2]], align 4
+; DEFAULT-NEXT:    [[PA3:%.*]] = getelementptr inbounds float, ptr [[DST]], i64 3
+; DEFAULT-NEXT:    store float [[A3]], ptr [[PA3]], align 4
+; DEFAULT-NEXT:    ret void
+;
+entry:
+  %e0 = load float, ptr %p, align 4
+  %pe1 = getelementptr inbounds float, ptr %p, i64 1
+  %e1 = load float, ptr %pe1, align 4
+  %pe2 = getelementptr inbounds float, ptr %p, i64 2
+  %e2 = load float, ptr %pe2, align 4
+  %pe3 = getelementptr inbounds float, ptr %p, i64 3
+  %e3 = load float, ptr %pe3, align 4
+
+  %f0 = load float, ptr %q, align 4
+  %pf1 = getelementptr inbounds float, ptr %q, i64 1
+  %f1 = load float, ptr %pf1, align 4
+  %pf2 = getelementptr inbounds float, ptr %q, i64 2
+  %f2 = load float, ptr %pf2, align 4
+  %pf3 = getelementptr inbounds float, ptr %q, i64 3
+  %f3 = load float, ptr %pf3, align 4
+
+  %a0 = fadd float %e0, %f0
+  %a1 = fadd float %e1, %f1
+  %a2 = fadd float %e2, %f2
+  %a3 = fadd float %e3, %f3
+
+  store float %a0, ptr %dst, align 4
+  %pa1 = getelementptr inbounds float, ptr %dst, i64 1
+  store float %a1, ptr %pa1, align 4
+  %pa2 = getelementptr inbounds float, ptr %dst, i64 2
+  store float %a2, ptr %pa2, align 4
+  %pa3 = getelementptr inbounds float, ptr %dst, i64 3
+  store float %a3, ptr %pa3, align 4
+
+  ret void
+}
+
+define void @fp_sub(ptr %dst, ptr %p) {
+; CHECK-LABEL: define void @fp_sub
+; CHECK-SAME: (ptr [[DST:%.*]], ptr [[P:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[P]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = fsub <4 x float> [[TMP0]], <float 3.000000e+00, float 3.000000e+00, float 3.000000e+00, float 3.000000e+00>
+; CHECK-NEXT:    store <4 x float> [[TMP1]], ptr [[DST]], align 4
+; CHECK-NEXT:    ret void
+;
+; DEFAULT-LABEL: define void @fp_sub
+; DEFAULT-SAME: (ptr [[DST:%.*]], ptr [[P:%.*]]) #[[ATTR0]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[E0:%.*]] = load float, ptr [[P]], align 4
+; DEFAULT-NEXT:    [[PE1:%.*]] = getelementptr inbounds float, ptr [[P]], i64 1
+; DEFAULT-NEXT:    [[E1:%.*]] = load float, ptr [[PE1]], align 4
+; DEFAULT-NEXT:    [[PE2:%.*]] = getelementptr inbounds float, ptr [[P]], i64 2
+; DEFAULT-NEXT:    [[E2:%.*]] = load float, ptr [[PE2]], align 4
+; DEFAULT-NEXT:    [[PE3:%.*]] = getelementptr inbounds float, ptr [[P]], i64 3
+; DEFAULT-NEXT:    [[E3:%.*]] = load float, ptr [[PE3]], align 4
+; DEFAULT-NEXT:    [[A0:%.*]] = fsub float [[E0]], 3.000000e+00
+; DEFAULT-NEXT:    [[A1:%.*]] = fsub float [[E1]], 3.000000e+00
+; DEFAULT-NEXT:    [[A2:%.*]] = fsub float [[E2]], 3.000000e+00
+; DEFAULT-NEXT:    [[A3:%.*]] = fsub float [[E3]], 3.000000e+00
+; DEFAULT-NEXT:    store float [[A0]], ptr [[DST]], align 4
+; DEFAULT-NEXT:    [[PA1:%.*]] = getelementptr inbounds float, ptr [[DST]], i64 1
+; DEFAULT-NEXT:    store float [[A1]], ptr [[PA1]], align 4
+; DEFAULT-NEXT:    [[PA2:%.*]] = getelementptr inbounds float, ptr [[DST]], i64 2
+; DEFAULT-NEXT:    store float [[A2]], ptr [[PA2]], align 4
+; DEFAULT-NEXT:    [[PA3:%.*]] = getelementptr inbounds float, ptr [[DST]], i64 3
+; DEFAULT-NEXT:    store float [[A3]], ptr [[PA3]], align 4
+; DEFAULT-NEXT:    ret void
+;
+entry:
+  %e0 = load float, ptr %p, align 4
+  %pe1 = getelementptr inbounds float, ptr %p, i64 1
+  %e1 = load float, ptr %pe1, align 4
+  %pe2 = getelementptr inbounds float, ptr %p, i64 2
+  %e2 = load float, ptr %pe2, align 4
+  %pe3 = getelementptr inbounds float, ptr %p, i64 3
+  %e3 = load float, ptr %pe3, align 4
+
+  %a0 = fsub float %e0, 3.0
+  %a1 = fsub float %e1, 3.0
+  %a2 = fsub float %e2, 3.0
+  %a3 = fsub float %e3, 3.0
+
+  store float %a0, ptr %dst, align 4
+  %pa1 = getelementptr inbounds float, ptr %dst, i64 1
+  store float %a1, ptr %pa1, align 4
+  %pa2 = getelementptr inbounds float, ptr %dst, i64 2
+  store float %a2, ptr %pa2, align 4
+  %pa3 = getelementptr inbounds float, ptr %dst, i64 3
+  store float %a3, ptr %pa3, align 4
+
+  ret void
+}
+
+define void @fp_mul(ptr %dst, ptr %p, ptr %q) {
+; CHECK-LABEL: define void @fp_mul
+; CHECK-SAME: (ptr [[DST:%.*]], ptr [[P:%.*]], ptr [[Q:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[P]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x float>, ptr [[Q]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = fmul <4 x float> [[TMP0]], [[TMP1]]
+; CHECK-NEXT:    store <4 x float> [[TMP2]], ptr [[DST]], align 4
+; CHECK-NEXT:    ret void
+;
+; DEFAULT-LABEL: define void @fp_mul
+; DEFAULT-SAME: (ptr [[DST:%.*]], ptr [[P:%.*]], ptr [[Q:%.*]]) #[[ATTR0]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[E0:%.*]] = load float, ptr [[P]], align 4
+; DEFAULT-NEXT:    [[PE1:%.*]] = getelementptr inbounds float, ptr [[P]], i64 1
+; DEFAULT-NEXT:    [[E1:%.*]] = load float, ptr [[PE1]], align 4
+; DEFAULT-NEXT:    [[PE2:%.*]] = getelementptr inbounds float, ptr [[P]], i64 2
+; DEFAULT-NEXT:    [[E2:%.*]] = load float, ptr [[PE2]], align 4
+; DEFAULT-NEXT:    [[PE3:%.*]] = getelementptr inbounds float, ptr [[P]], i64 3
+; DEFAULT-NEXT:    [[E3:%.*]] = load float, ptr [[PE3]], align 4
+; DEFAULT-NEXT:    [[F0:%.*]] = load float, ptr [[Q]], align 4
+; DEFAULT-NEXT:    [[PF1:%.*]] = getelementptr inbounds float, ptr [[Q]], i64 1
+; DEFAULT-NEXT:    [[F1:%.*]] = load float, ptr [[PF1]], align 4
+; DEFAULT-NEXT:    [[PF2:%.*]] = getelementptr inbounds float, ptr [[Q]], i64 2
+; DEFAULT-NEXT:    [[F2:%.*]] = load float, ptr [[PF2]], align 4
+; DEFAULT-NEXT:    [[PF3:%.*]] = getelementptr inbounds float, ptr [[Q]], i64 3
+; DEFAULT-NEXT:    [[F3:%.*]] = load float, ptr [[PF3]], align 4
+; DEFAULT-NEXT:    [[A0:%.*]] = fmul float [[E0]], [[F0]]
+; DEFAULT-NEXT:    [[A1:%.*]] = fmul float [[E1]], [[F1]]
+; DEFAULT-NEXT:    [[A2:%.*]] = fmul float [[E2]], [[F2]]
+; DEFAULT-NEXT:    [[A3:%.*]] = fmul float [[E3]], [[F3]]
+; DEFAULT-NEXT:    store float [[A0]], ptr [[DST]], align 4
+; DEFAULT-NEXT:    [[PA1:%.*]] = getelementptr inbounds float, ptr [[DST]], i64 1
+; DEFAULT-NEXT:    store float [[A1]], ptr [[PA1]], align 4
+; DEFAULT-NEXT:    [[PA2:%.*]] = getelementptr inbounds float, ptr [[DST]], i64 2
+; DEFAULT-NEXT:    store float [[A2]], ptr [[PA2]], align 4
+; DEFAULT-NEXT:    [[PA3:%.*]] = getelementptr inbounds float, ptr [[DST]], i64 3
+; DEFAULT-NEXT:    store float [[A3]], ptr [[PA3]], align 4
+; DEFAULT-NEXT:    ret void
+;
+entry:
+  %e0 = load float, ptr %p, align 4
+  %pe1 = getelementptr inbounds float, ptr %p, i64 1
+  %e1 = load float, ptr %pe1, align 4
+  %pe2 = getelementptr inbounds float, ptr %p, i64 2
+  %e2 = load float, ptr %pe2, align 4
+  %pe3 = getelementptr inbounds float, ptr %p, i64 3
+  %e3 = load float, ptr %pe3, align 4
+
+  %f0 = load float, ptr %q, align 4
+  %pf1 = getelementptr inbounds float, ptr %q, i64 1
+  %f1 = load float, ptr %pf1, align 4
+  %pf2 = getelementptr inbounds float, ptr %q, i64 2
+  %f2 = load float, ptr %pf2, align 4
+  %pf3 = getelementptr inbounds float, ptr %q, i64 3
+  %f3 = load float, ptr %pf3, align 4
+
+  %a0 = fmul float %e0, %f0
+  %a1 = fmul float %e1, %f1
+  %a2 = fmul float %e2, %f2
+  %a3 = fmul float %e3, %f3
+
+  store float %a0, ptr %dst, align 4
+  %pa1 = getelementptr inbounds float, ptr %dst, i64 1
+  store float %a1, ptr %pa1, align 4
+  %pa2 = getelementptr inbounds float, ptr %dst, i64 2
+  store float %a2, ptr %pa2, align 4
+  %pa3 = getelementptr inbounds float, ptr %dst, i64 3
+  store float %a3, ptr %pa3, align 4
+
+  ret void
+}
+
+define void @fp_div(ptr %dst, ptr %p) {
+; CHECK-LABEL: define void @fp_div
+; CHECK-SAME: (ptr [[DST:%.*]], ptr [[P:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[P]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = fdiv <4 x float> [[TMP0]], <float 1.050000e+01, float 1.050000e+01, float 1.050000e+01, float 1.050000e+01>
+; CHECK-NEXT:    store <4 x float> [[TMP1]], ptr [[DST]], align 4
+; CHECK-NEXT:    ret void
+;
+; DEFAULT-LABEL: define void @fp_div
+; DEFAULT-SAME: (ptr [[DST:%.*]], ptr [[P:%.*]]) #[[ATTR0]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[E0:%.*]] = load float, ptr [[P]], align 4
+; DEFAULT-NEXT:    [[PE1:%.*]] = getelementptr inbounds float, ptr [[P]], i64 1
+; DEFAULT-NEXT:    [[E1:%.*]] = load float, ptr [[PE1]], align 4
+; DEFAULT-NEXT:    [[PE2:%.*]] = getelementptr inbounds float, ptr [[P]], i64 2
+; DEFAULT-NEXT:    [[E2:%.*]] = load float, ptr [[PE2]], align 4
+; DEFAULT-NEXT:    [[PE3:%.*]] = getelementptr inbounds float, ptr [[P]], i64 3
+; DEFAULT-NEXT:    [[E3:%.*]] = load float, ptr [[PE3]], align 4
+; DEFAULT-NEXT:    [[A0:%.*]] = fdiv float [[E0]], 1.050000e+01
+; DEFAULT-NEXT:    [[A1:%.*]] = fdiv float [[E1]], 1.050000e+01
+; DEFAULT-NEXT:    [[A2:%.*]] = fdiv float [[E2]], 1.050000e+01
+; DEFAULT-NEXT:    [[A3:%.*]] = fdiv float [[E3]], 1.050000e+01
+; DEFAULT-NEXT:    store float [[A0]], ptr [[DST]], align 4
+; DEFAULT-NEXT:    [[PA1:%.*]] = getelementptr inbounds float, ptr [[DST]], i64 1
+; DEFAULT-NEXT:    store float [[A1]], ptr [[PA1]], align 4
+; DEFAULT-NEXT:    [[PA2:%.*]] = getelementptr inbounds float, ptr [[DST]], i64 2
+; DEFAULT-NEXT:    store float [[A2]], ptr [[PA2]], align 4
+; DEFAULT-NEXT:    [[PA3:%.*]] = getelementptr inbounds float, ptr [[DST]], i64 3
+; DEFAULT-NEXT:    store float [[A3]], ptr [[PA3]], align 4
+; DEFAULT-NEXT:    ret void
+;
+entry:
+  %e0 = load float, ptr %p, align 4
+  %pe1 = getelementptr inbounds float, ptr %p, i64 1
+  %e1 = load float, ptr %pe1, align 4
+  %pe2 = getelementptr inbounds float, ptr %p, i64 2
+  %e2 = load float, ptr %pe2, align 4
+  %pe3 = getelementptr inbounds float, ptr %p, i64 3
+  %e3 = load float, ptr %pe3, align 4
+
+  %a0 = fdiv float %e0, 10.5
+  %a1 = fdiv float %e1, 10.5
+  %a2 = fdiv float %e2, 10.5
+  %a3 = fdiv float %e3, 10.5
+
+  store float %a0, ptr %dst, align 4
+  %pa1 = getelementptr inbounds float, ptr %dst, i64 1
+  store float %a1, ptr %pa1, align 4
+  %pa2 = getelementptr inbounds float, ptr %dst, i64 2
+  store float %a2, ptr %pa2, align 4
+  %pa3 = getelementptr inbounds float, ptr %dst, i64 3
+  store float %a3, ptr %pa3, align 4
+
+  ret void
+}
+
+declare float @llvm.maxnum.f32(float, float)
+
+define void @fp_max(ptr %dst, ptr %p, ptr %q) {
+; CHECK-LABEL: define void @fp_max
+; CHECK-SAME: (ptr [[DST:%.*]], ptr [[P:%.*]], ptr [[Q:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[P]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x float>, ptr [[Q]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = call <4 x float> @llvm.maxnum.v4f32(<4 x float> [[TMP0]], <4 x float> [[TMP1]])
+; CHECK-NEXT:    store <4 x float> [[TMP2]], ptr [[DST]], align 4
+; CHECK-NEXT:    ret void
+;
+; DEFAULT-LABEL: define void @fp_max
+; DEFAULT-SAME: (ptr [[DST:%.*]], ptr [[P:%.*]], ptr [[Q:%.*]]) #[[ATTR0]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[E0:%.*]] = load float, ptr [[P]], align 4
+; DEFAULT-NEXT:    [[PE1:%.*]] = getelementptr inbounds float, ptr [[P]], i64 1
+; DEFAULT-NEXT:    [[E1:%.*]] = load float, ptr [[PE1]], align 4
+; DEFAULT-NEXT:    [[PE2:%.*]] = getelementptr inbounds float, ptr [[P]], i64 2
+; DEFAULT-NEXT:    [[E2:%.*]] = load float, ptr [[PE2]], align 4
+; DEFAULT-NEXT:    [[PE3:%.*]] = getelementptr inbounds float, ptr [[P]], i64 3
+; DEFAULT-NEXT:    [[E3:%.*]] = load float, ptr [[PE3]], align 4
+; DEFAULT-NEXT:    [[F0:%.*]] = load float, ptr [[Q]], align 4
+; DEFAULT-NEXT:    [[PF1:%.*]] = getelementptr inbounds float, ptr [[Q]], i64 1
+; DEFAULT-NEXT:    [[F1:%.*]] = load float, ptr [[PF1]], align 4
+; DEFAULT-NEXT:    [[PF2:%.*]] = getelementptr inbounds float, ptr [[Q]], i64 2
+; DEFAULT-NEXT:    [[F2:%.*]] = load float, ptr [[PF2]], align 4
+; DEFAULT-NEXT:    [[PF3:%.*]] = getelementptr inbounds float, ptr [[Q]], i64 3
+; DEFAULT-NEXT:    [[F3:%.*]] = load float, ptr [[PF3]], align 4
+; DEFAULT-NEXT:    [[A0:%.*]] = tail call float @llvm.maxnum.f32(float [[E0]], float [[F0]])
+; DEFAULT-NEXT:    [[A1:%.*]] = tail call float @llvm.maxnum.f32(float [[E1]], float [[F1]])
+; DEFAULT-NEXT:    [[A2:%.*]] = tail call float @llvm.maxnum.f32(float [[E2]], float [[F2]])
+; DEFAULT-NEXT:    [[A3:%.*]] = tail call float @llvm.maxnum.f32(float [[E3]], float [[F3]])
+; DEFAULT-NEXT:    store float [[A0]], ptr [[DST]], align 4
+; DEFAULT-NEXT:    [[PA1:%.*]] = getelementptr inbounds float, ptr [[DST]], i64 1
+; DEFAULT-NEXT:    store float [[A1]], ptr [[PA1]], align 4
+; DEFAULT-NEXT:    [[PA2:%.*]] = getelementptr inbounds float, ptr [[DST]], i64 2
+; DEFAULT-NEXT:    store float [[A2]], ptr [[PA2]], align 4
+; DEFAULT-NEXT:    [[PA3:%.*]] = getelementptr inbounds float, ptr [[DST]], i64 3
+; DEFAULT-NEXT:    store float [[A3]], ptr [[PA3]], align 4
+; DEFAULT-NEXT:    ret void
+;
+entry:
+  %e0 = load float, ptr %p, align 4
+  %pe1 = getelementptr inbounds float, ptr %p, i64 1
+  %e1 = load float, ptr %pe1, align 4
+  %pe2 = getelementptr inbounds float, ptr %p, i64 2
+  %e2 = load float, ptr %pe2, align 4
+  %pe3 = getelementptr inbounds float, ptr %p, i64 3
+  %e3 = load float, ptr %pe3, align 4
+
+  %f0 = load float, ptr %q, align 4
+  %pf1 = getelementptr inbounds float, ptr %q, i64 1
+  %f1 = load float, ptr %pf1, align 4
+  %pf2 = getelementptr inbounds float, ptr %q, i64 2
+  %f2 = load float, ptr %pf2, align 4
+  %pf3 = getelementptr inbounds float, ptr %q, i64 3
+  %f3 = load float, ptr %pf3, align 4
+
+  %a0 = tail call float @llvm.maxnum.f32(float %e0, float %f0)
+  %a1 = tail call float @llvm.maxnum.f32(float %e1, float %f1)
+  %a2 = tail call float @llvm.maxnum.f32(float %e2, float %f2)
+  %a3 = tail call float @llvm.maxnum.f32(float %e3, float %f3)
+
+  store float %a0, ptr %dst, align 4
+  %pa1 = getelementptr inbounds float, ptr %dst, i64 1
+  store float %a1, ptr %pa1, align 4
+  %pa2 = getelementptr inbounds float, ptr %dst, i64 2
+  store float %a2, ptr %pa2, align 4
+  %pa3 = getelementptr inbounds float, ptr %dst, i64 3
+  store float %a3, ptr %pa3, align 4
+
+  ret void
+}
+
+declare float @llvm.minnum.f32(float, float)
+
+define void @fp_min(ptr %dst, ptr %p) {
+; CHECK-LABEL: define void @fp_min
+; CHECK-SAME: (ptr [[DST:%.*]], ptr [[P:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[P]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = call <4 x float> @llvm.minnum.v4f32(<4 x float> [[TMP0]], <4 x float> <float 1.250000e+00, float 1.250000e+00, float 1.250000e+00, float 1.250000e+00>)
+; CHECK-NEXT:    store <4 x float> [[TMP1]], ptr [[DST]], align 4
+; CHECK-NEXT:    ret void
+;
+; DEFAULT-LABEL: define void @fp_min
+; DEFAULT-SAME: (ptr [[DST:%.*]], ptr [[P:%.*]]) #[[ATTR0]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[E0:%.*]] = load float, ptr [[P]], align 4
+; DEFAULT-NEXT:    [[PE1:%.*]] = getelementptr inbounds float, ptr [[P]], i64 1
+; DEFAULT-NEXT:    [[E1:%.*]] = load float, ptr [[PE1]], align 4
+; DEFAULT-NEXT:    [[PE2:%.*]] = getelementptr inbounds float, ptr [[P]], i64 2
+; DEFAULT-NEXT:    [[E2:%.*]] = load float, ptr [[PE2]], align 4
+; DEFAULT-NEXT:    [[PE3:%.*]] = getelementptr inbounds float, ptr [[P]], i64 3
+; DEFAULT-NEXT:    [[E3:%.*]] = load float, ptr [[PE3]], align 4
+; DEFAULT-NEXT:    [[A0:%.*]] = tail call float @llvm.minnum.f32(float [[E0]], float 1.250000e+00)
+; DEFAULT-NEXT:    [[A1:%.*]] = tail call float @llvm.minnum.f32(float [[E1]], float 1.250000e+00)
+; DEFAULT-NEXT:    [[A2:%.*]] = tail call float @llvm.minnum.f32(float [[E2]], float 1.250000e+00)
+; DEFAULT-NEXT:    [[A3:%.*]] = tail call float @llvm.minnum.f32(float [[E3]], float 1.250000e+00)
+; DEFAULT-NEXT:    store float [[A0]], ptr [[DST]], align 4
+; DEFAULT-NEXT:    [[PA1:%.*]] = getelementptr inbounds float, ptr [[DST]], i64 1
+; DEFAULT-NEXT:    store float [[A1]], ptr [[PA1]], align 4
+; DEFAULT-NEXT:    [[PA2:%.*]] = getelementptr inbounds float, ptr [[DST]], i64 2
+; DEFAULT-NEXT:    store float [[A2]], ptr [[PA2]], align 4
+; DEFAULT-NEXT:    [[PA3:%.*]] = getelementptr inbounds float, ptr [[DST]], i64 3
+; DEFAULT-NEXT:    store float [[A3]], ptr [[PA3]], align 4
+; DEFAULT-NEXT:    ret void
+;
+entry:
+  %e0 = load float, ptr %p, align 4
+  %pe1 = getelementptr inbounds float, ptr %p, i64 1
+  %e1 = load float, ptr %pe1, align 4
+  %pe2 = getelementptr inbounds float, ptr %p, i64 2
+  %e2 = load float, ptr %pe2, align 4
+  %pe3 = getelementptr inbounds float, ptr %p, i64 3
+  %e3 = load float, ptr %pe3, align 4
+
+  %a0 = tail call float @llvm.minnum.f32(float %e0, float 1.25)
+  %a1 = tail call float @llvm.minnum.f32(float %e1, float 1.25)
+  %a2 = tail call float @llvm.minnum.f32(float %e2, float 1.25)
+  %a3 = tail call float @llvm.minnum.f32(float %e3, float 1.25)
+
+  store float %a0, ptr %dst, align 4
+  %pa1 = getelementptr inbounds float, ptr %dst, i64 1
+  store float %a1, ptr %pa1, align 4
+  %pa2 = getelementptr inbounds float, ptr %dst, i64 2
+  store float %a2, ptr %pa2, align 4
+  %pa3 = getelementptr inbounds float, ptr %dst, i64 3
+  store float %a3, ptr %pa3, align 4
+
+  ret void
+}
+
+declare i32 @llvm.fptosi.sat.i32.f32(float)
+
+define void @fp_convert(ptr %dst, ptr %p) {
+; CHECK-LABEL: define void @fp_convert
+; CHECK-SAME: (ptr [[DST:%.*]], ptr [[P:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[P]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = call <4 x i32> @llvm.fptosi.sat.v4i32.v4f32(<4 x float> [[TMP0]])
+; CHECK-NEXT:    store <4 x i32> [[TMP1]], ptr [[DST]], align 4
+; CHECK-NEXT:    ret void
+;
+; DEFAULT-LABEL: define void @fp_convert
+; DEFAULT-SAME: (ptr [[DST:%.*]], ptr [[P:%.*]]) #[[ATTR0]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[E0:%.*]] = load float, ptr [[P]], align 4
+; DEFAULT-NEXT:    [[PE1:%.*]] = getelementptr inbounds float, ptr [[P]], i64 1
+; DEFAULT-NEXT:    [[E1:%.*]] = load float, ptr [[PE1]], align 4
+; DEFAULT-NEXT:    [[PE2:%.*]] = getelementptr inbounds float, ptr [[P]], i64 2
+; DEFAULT-NEXT:    [[E2:%.*]] = load float, ptr [[PE2]], align 4
+; DEFAULT-NEXT:    [[PE3:%.*]] = getelementptr inbounds float, ptr [[P]], i64 3
+; DEFAULT-NEXT:    [[E3:%.*]] = load float, ptr [[PE3]], align 4
+; DEFAULT-NEXT:    [[A0:%.*]] = tail call i32 @llvm.fptosi.sat.i32.f32(float [[E0]])
+; DEFAULT-NEXT:    [[A1:%.*]] = tail call i32 @llvm.fptosi.sat.i32.f32(float [[E1]])
+; DEFAULT-NEXT:    [[A2:%.*]] = tail call i32 @llvm.fptosi.sat.i32.f32(float [[E2]])
+; DEFAULT-NEXT:    [[A3:%.*]] = tail call i32 @llvm.fptosi.sat.i32.f32(float [[E3]])
+; DEFAULT-NEXT:    store i32 [[A0]], ptr [[DST]], align 4
+; DEFAULT-NEXT:    [[PA1:%.*]] = getelementptr inbounds i32, ptr [[DST]], i64 1
+; DEFAULT-NEXT:    store i32 [[A1]], ptr [[PA1]], align 4
+; DEFAULT-NEXT:    [[PA2:%.*]] = getelementptr inbounds i32, ptr [[DST]], i64 2
+; DEFAULT-NEXT:    store i32 [[A2]], ptr [[PA2]], align 4
+; DEFAULT-NEXT:    [[PA3:%.*]] = getelementptr inbounds i32, ptr [[DST]], i64 3
+; DEFAULT-NEXT:    store i32 [[A3]], ptr [[PA3]], align 4
+; DEFAULT-NEXT:    ret void
+;
+entry:
+  %e0 = load float, ptr %p, align 4
+  %pe1 = getelementptr inbounds float, ptr %p, i64 1
+  %e1 = load float, ptr %pe1, align 4
+  %pe2 = getelementptr inbounds float, ptr %p, i64 2
+  %e2 = load float, ptr %pe2, align 4
+  %pe3 = getelementptr inbounds float, ptr %p, i64 3
+  %e3 = load float, ptr %pe3, align 4
+
+  %a0 = tail call i32 @llvm.fptosi.sat.i32.f32(float %e0)
+  %a1 = tail call i32 @llvm.fptosi.sat.i32.f32(float %e1)
+  %a2 = tail call i32 @llvm.fptosi.sat.i32.f32(float %e2)
+  %a3 = tail call i32 @llvm.fptosi.sat.i32.f32(float %e3)
+
+  store i32 %a0, ptr %dst, align 4
+  %pa1 = getelementptr inbounds i32, ptr %dst, i64 1
+  store i32 %a1, ptr %pa1, align 4
+  %pa2 = getelementptr inbounds i32, ptr %dst, i64 2
+  store i32 %a2, ptr %pa2, align 4
+  %pa3 = getelementptr inbounds i32, ptr %dst, i64 3
+  store i32 %a3, ptr %pa3, align 4
+
+  ret void
+}
diff --git a/llvm/test/Transforms/SLPVectorizer/RISCV/math-function.ll b/llvm/test/Transforms/SLPVectorizer/RISCV/math-function.ll
new file mode 100644
index 0000000000000..acf826ae69980
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/RISCV/math-function.ll
@@ -0,0 +1,870 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
+; RUN: opt < %s -passes=slp-vectorizer -S -mtriple=riscv64 -mattr=+v,+f \
+; RUN:     -riscv-v-vector-bits-min=-1 -riscv-v-slp-max-vf=0 \
+; RUN:     | FileCheck %s
+; RUN: opt < %s -passes=slp-vectorizer -S -mtriple=riscv64 -mattr=+v,+f \
+; RUN:     | FileCheck %s --check-prefix=DEFAULT
+
+declare float @fabsf(float) readonly nounwind willreturn
+
+define <4 x float> @fabs_4x(ptr %a) {
+; CHECK-LABEL: define <4 x float> @fabs_4x
+; CHECK-SAME: (ptr [[A:%.*]]) #[[ATTR1:[0-9]+]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; CHECK-NEXT:    [[TMP1:%.*]] = call fast <4 x float> @llvm.fabs.v4f32(<4 x float> [[TMP0]])
+; CHECK-NEXT:    ret <4 x float> [[TMP1]]
+;
+; DEFAULT-LABEL: define <4 x float> @fabs_4x
+; DEFAULT-SAME: (ptr [[A:%.*]]) #[[ATTR1:[0-9]+]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; DEFAULT-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; DEFAULT-NEXT:    [[TMP1:%.*]] = tail call fast float @fabsf(float [[VECEXT]])
+; DEFAULT-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; DEFAULT-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; DEFAULT-NEXT:    [[TMP2:%.*]] = tail call fast float @fabsf(float [[VECEXT_1]])
+; DEFAULT-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; DEFAULT-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; DEFAULT-NEXT:    [[TMP3:%.*]] = tail call fast float @fabsf(float [[VECEXT_2]])
+; DEFAULT-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; DEFAULT-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; DEFAULT-NEXT:    [[TMP4:%.*]] = tail call fast float @fabsf(float [[VECEXT_3]])
+; DEFAULT-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; DEFAULT-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+entry:
+  %0 = load <4 x float>, ptr %a, align 16
+  %vecext = extractelement <4 x float> %0, i32 0
+  %1 = tail call fast float @fabsf(float %vecext)
+  %vecins = insertelement <4 x float> undef, float %1, i32 0
+  %vecext.1 = extractelement <4 x float> %0, i32 1
+  %2 = tail call fast float @fabsf(float %vecext.1)
+  %vecins.1 = insertelement <4 x float> %vecins, float %2, i32 1
+  %vecext.2 = extractelement <4 x float> %0, i32 2
+  %3 = tail call fast float @fabsf(float %vecext.2)
+  %vecins.2 = insertelement <4 x float> %vecins.1, float %3, i32 2
+  %vecext.3 = extractelement <4 x float> %0, i32 3
+  %4 = tail call fast float @fabsf(float %vecext.3)
+  %vecins.3 = insertelement <4 x float> %vecins.2, float %4, i32 3
+  ret <4 x float> %vecins.3
+}
+
+declare float @llvm.fabs.f32(float)
+
+define <4 x float> @int_fabs_4x(ptr %a) {
+; CHECK-LABEL: define <4 x float> @int_fabs_4x
+; CHECK-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; CHECK-NEXT:    [[TMP1:%.*]] = call fast <4 x float> @llvm.fabs.v4f32(<4 x float> [[TMP0]])
+; CHECK-NEXT:    ret <4 x float> [[TMP1]]
+;
+; DEFAULT-LABEL: define <4 x float> @int_fabs_4x
+; DEFAULT-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; DEFAULT-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; DEFAULT-NEXT:    [[TMP1:%.*]] = tail call fast float @llvm.fabs.f32(float [[VECEXT]])
+; DEFAULT-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; DEFAULT-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; DEFAULT-NEXT:    [[TMP2:%.*]] = tail call fast float @llvm.fabs.f32(float [[VECEXT_1]])
+; DEFAULT-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; DEFAULT-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; DEFAULT-NEXT:    [[TMP3:%.*]] = tail call fast float @llvm.fabs.f32(float [[VECEXT_2]])
+; DEFAULT-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; DEFAULT-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; DEFAULT-NEXT:    [[TMP4:%.*]] = tail call fast float @llvm.fabs.f32(float [[VECEXT_3]])
+; DEFAULT-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; DEFAULT-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+entry:
+  %0 = load <4 x float>, ptr %a, align 16
+  %vecext = extractelement <4 x float> %0, i32 0
+  %1 = tail call fast float @llvm.fabs.f32(float %vecext)
+  %vecins = insertelement <4 x float> undef, float %1, i32 0
+  %vecext.1 = extractelement <4 x float> %0, i32 1
+  %2 = tail call fast float @llvm.fabs.f32(float %vecext.1)
+  %vecins.1 = insertelement <4 x float> %vecins, float %2, i32 1
+  %vecext.2 = extractelement <4 x float> %0, i32 2
+  %3 = tail call fast float @llvm.fabs.f32(float %vecext.2)
+  %vecins.2 = insertelement <4 x float> %vecins.1, float %3, i32 2
+  %vecext.3 = extractelement <4 x float> %0, i32 3
+  %4 = tail call fast float @llvm.fabs.f32(float %vecext.3)
+  %vecins.3 = insertelement <4 x float> %vecins.2, float %4, i32 3
+  ret <4 x float> %vecins.3
+}
+
+declare float @sqrtf(float) readonly nounwind willreturn
+
+define <4 x float> @sqrt_4x(ptr %a) {
+; CHECK-LABEL: define <4 x float> @sqrt_4x
+; CHECK-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; CHECK-NEXT:    [[TMP1:%.*]] = call fast <4 x float> @llvm.sqrt.v4f32(<4 x float> [[TMP0]])
+; CHECK-NEXT:    ret <4 x float> [[TMP1]]
+;
+; DEFAULT-LABEL: define <4 x float> @sqrt_4x
+; DEFAULT-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; DEFAULT-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; DEFAULT-NEXT:    [[TMP1:%.*]] = tail call fast float @sqrtf(float [[VECEXT]])
+; DEFAULT-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; DEFAULT-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; DEFAULT-NEXT:    [[TMP2:%.*]] = tail call fast float @sqrtf(float [[VECEXT_1]])
+; DEFAULT-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; DEFAULT-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; DEFAULT-NEXT:    [[TMP3:%.*]] = tail call fast float @sqrtf(float [[VECEXT_2]])
+; DEFAULT-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; DEFAULT-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; DEFAULT-NEXT:    [[TMP4:%.*]] = tail call fast float @sqrtf(float [[VECEXT_3]])
+; DEFAULT-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; DEFAULT-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+entry:
+  %0 = load <4 x float>, ptr %a, align 16
+  %vecext = extractelement <4 x float> %0, i32 0
+  %1 = tail call fast float @sqrtf(float %vecext)
+  %vecins = insertelement <4 x float> undef, float %1, i32 0
+  %vecext.1 = extractelement <4 x float> %0, i32 1
+  %2 = tail call fast float @sqrtf(float %vecext.1)
+  %vecins.1 = insertelement <4 x float> %vecins, float %2, i32 1
+  %vecext.2 = extractelement <4 x float> %0, i32 2
+  %3 = tail call fast float @sqrtf(float %vecext.2)
+  %vecins.2 = insertelement <4 x float> %vecins.1, float %3, i32 2
+  %vecext.3 = extractelement <4 x float> %0, i32 3
+  %4 = tail call fast float @sqrtf(float %vecext.3)
+  %vecins.3 = insertelement <4 x float> %vecins.2, float %4, i32 3
+  ret <4 x float> %vecins.3
+}
+
+declare float @llvm.sqrt.f32(float)
+
+define <4 x float> @int_sqrt_4x(ptr %a) {
+; CHECK-LABEL: define <4 x float> @int_sqrt_4x
+; CHECK-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; CHECK-NEXT:    [[TMP1:%.*]] = call fast <4 x float> @llvm.sqrt.v4f32(<4 x float> [[TMP0]])
+; CHECK-NEXT:    ret <4 x float> [[TMP1]]
+;
+; DEFAULT-LABEL: define <4 x float> @int_sqrt_4x
+; DEFAULT-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; DEFAULT-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; DEFAULT-NEXT:    [[TMP1:%.*]] = tail call fast float @llvm.sqrt.f32(float [[VECEXT]])
+; DEFAULT-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; DEFAULT-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; DEFAULT-NEXT:    [[TMP2:%.*]] = tail call fast float @llvm.sqrt.f32(float [[VECEXT_1]])
+; DEFAULT-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; DEFAULT-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; DEFAULT-NEXT:    [[TMP3:%.*]] = tail call fast float @llvm.sqrt.f32(float [[VECEXT_2]])
+; DEFAULT-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; DEFAULT-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; DEFAULT-NEXT:    [[TMP4:%.*]] = tail call fast float @llvm.sqrt.f32(float [[VECEXT_3]])
+; DEFAULT-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; DEFAULT-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+entry:
+  %0 = load <4 x float>, ptr %a, align 16
+  %vecext = extractelement <4 x float> %0, i32 0
+  %1 = tail call fast float @llvm.sqrt.f32(float %vecext)
+  %vecins = insertelement <4 x float> undef, float %1, i32 0
+  %vecext.1 = extractelement <4 x float> %0, i32 1
+  %2 = tail call fast float @llvm.sqrt.f32(float %vecext.1)
+  %vecins.1 = insertelement <4 x float> %vecins, float %2, i32 1
+  %vecext.2 = extractelement <4 x float> %0, i32 2
+  %3 = tail call fast float @llvm.sqrt.f32(float %vecext.2)
+  %vecins.2 = insertelement <4 x float> %vecins.1, float %3, i32 2
+  %vecext.3 = extractelement <4 x float> %0, i32 3
+  %4 = tail call fast float @llvm.sqrt.f32(float %vecext.3)
+  %vecins.3 = insertelement <4 x float> %vecins.2, float %4, i32 3
+  ret <4 x float> %vecins.3
+}
+
+declare float @expf(float) readonly nounwind willreturn
+
+; We can not vectorized exp since RISCV has no such instruction.
+define <4 x float> @exp_4x(ptr %a) {
+; CHECK-LABEL: define <4 x float> @exp_4x
+; CHECK-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; CHECK-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = tail call fast float @expf(float [[VECEXT]])
+; CHECK-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; CHECK-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = tail call fast float @expf(float [[VECEXT_1]])
+; CHECK-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; CHECK-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = tail call fast float @expf(float [[VECEXT_2]])
+; CHECK-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; CHECK-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; CHECK-NEXT:    [[TMP4:%.*]] = tail call fast float @expf(float [[VECEXT_3]])
+; CHECK-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; CHECK-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+; DEFAULT-LABEL: define <4 x float> @exp_4x
+; DEFAULT-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; DEFAULT-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; DEFAULT-NEXT:    [[TMP1:%.*]] = tail call fast float @expf(float [[VECEXT]])
+; DEFAULT-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; DEFAULT-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; DEFAULT-NEXT:    [[TMP2:%.*]] = tail call fast float @expf(float [[VECEXT_1]])
+; DEFAULT-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; DEFAULT-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; DEFAULT-NEXT:    [[TMP3:%.*]] = tail call fast float @expf(float [[VECEXT_2]])
+; DEFAULT-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; DEFAULT-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; DEFAULT-NEXT:    [[TMP4:%.*]] = tail call fast float @expf(float [[VECEXT_3]])
+; DEFAULT-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; DEFAULT-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+entry:
+  %0 = load <4 x float>, ptr %a, align 16
+  %vecext = extractelement <4 x float> %0, i32 0
+  %1 = tail call fast float @expf(float %vecext)
+  %vecins = insertelement <4 x float> undef, float %1, i32 0
+  %vecext.1 = extractelement <4 x float> %0, i32 1
+  %2 = tail call fast float @expf(float %vecext.1)
+  %vecins.1 = insertelement <4 x float> %vecins, float %2, i32 1
+  %vecext.2 = extractelement <4 x float> %0, i32 2
+  %3 = tail call fast float @expf(float %vecext.2)
+  %vecins.2 = insertelement <4 x float> %vecins.1, float %3, i32 2
+  %vecext.3 = extractelement <4 x float> %0, i32 3
+  %4 = tail call fast float @expf(float %vecext.3)
+  %vecins.3 = insertelement <4 x float> %vecins.2, float %4, i32 3
+  ret <4 x float> %vecins.3
+}
+
+declare float @llvm.exp.f32(float)
+
+; We can not vectorized exp since RISCV has no such instruction.
+define <4 x float> @int_exp_4x(ptr %a) {
+; CHECK-LABEL: define <4 x float> @int_exp_4x
+; CHECK-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; CHECK-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = tail call fast float @llvm.exp.f32(float [[VECEXT]])
+; CHECK-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; CHECK-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = tail call fast float @llvm.exp.f32(float [[VECEXT_1]])
+; CHECK-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; CHECK-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = tail call fast float @llvm.exp.f32(float [[VECEXT_2]])
+; CHECK-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; CHECK-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; CHECK-NEXT:    [[TMP4:%.*]] = tail call fast float @llvm.exp.f32(float [[VECEXT_3]])
+; CHECK-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; CHECK-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+; DEFAULT-LABEL: define <4 x float> @int_exp_4x
+; DEFAULT-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; DEFAULT-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; DEFAULT-NEXT:    [[TMP1:%.*]] = tail call fast float @llvm.exp.f32(float [[VECEXT]])
+; DEFAULT-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; DEFAULT-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; DEFAULT-NEXT:    [[TMP2:%.*]] = tail call fast float @llvm.exp.f32(float [[VECEXT_1]])
+; DEFAULT-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; DEFAULT-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; DEFAULT-NEXT:    [[TMP3:%.*]] = tail call fast float @llvm.exp.f32(float [[VECEXT_2]])
+; DEFAULT-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; DEFAULT-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; DEFAULT-NEXT:    [[TMP4:%.*]] = tail call fast float @llvm.exp.f32(float [[VECEXT_3]])
+; DEFAULT-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; DEFAULT-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+entry:
+  %0 = load <4 x float>, ptr %a, align 16
+  %vecext = extractelement <4 x float> %0, i32 0
+  %1 = tail call fast float @llvm.exp.f32(float %vecext)
+  %vecins = insertelement <4 x float> undef, float %1, i32 0
+  %vecext.1 = extractelement <4 x float> %0, i32 1
+  %2 = tail call fast float @llvm.exp.f32(float %vecext.1)
+  %vecins.1 = insertelement <4 x float> %vecins, float %2, i32 1
+  %vecext.2 = extractelement <4 x float> %0, i32 2
+  %3 = tail call fast float @llvm.exp.f32(float %vecext.2)
+  %vecins.2 = insertelement <4 x float> %vecins.1, float %3, i32 2
+  %vecext.3 = extractelement <4 x float> %0, i32 3
+  %4 = tail call fast float @llvm.exp.f32(float %vecext.3)
+  %vecins.3 = insertelement <4 x float> %vecins.2, float %4, i32 3
+  ret <4 x float> %vecins.3
+}
+
+declare float @logf(float) readonly nounwind willreturn
+
+; We can not vectorized log since RISCV has no such instruction.
+define <4 x float> @log_4x(ptr %a) {
+; CHECK-LABEL: define <4 x float> @log_4x
+; CHECK-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; CHECK-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = tail call fast float @logf(float [[VECEXT]])
+; CHECK-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; CHECK-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = tail call fast float @logf(float [[VECEXT_1]])
+; CHECK-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; CHECK-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = tail call fast float @logf(float [[VECEXT_2]])
+; CHECK-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; CHECK-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; CHECK-NEXT:    [[TMP4:%.*]] = tail call fast float @logf(float [[VECEXT_3]])
+; CHECK-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; CHECK-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+; DEFAULT-LABEL: define <4 x float> @log_4x
+; DEFAULT-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; DEFAULT-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; DEFAULT-NEXT:    [[TMP1:%.*]] = tail call fast float @logf(float [[VECEXT]])
+; DEFAULT-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; DEFAULT-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; DEFAULT-NEXT:    [[TMP2:%.*]] = tail call fast float @logf(float [[VECEXT_1]])
+; DEFAULT-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; DEFAULT-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; DEFAULT-NEXT:    [[TMP3:%.*]] = tail call fast float @logf(float [[VECEXT_2]])
+; DEFAULT-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; DEFAULT-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; DEFAULT-NEXT:    [[TMP4:%.*]] = tail call fast float @logf(float [[VECEXT_3]])
+; DEFAULT-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; DEFAULT-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+entry:
+  %0 = load <4 x float>, ptr %a, align 16
+  %vecext = extractelement <4 x float> %0, i32 0
+  %1 = tail call fast float @logf(float %vecext)
+  %vecins = insertelement <4 x float> undef, float %1, i32 0
+  %vecext.1 = extractelement <4 x float> %0, i32 1
+  %2 = tail call fast float @logf(float %vecext.1)
+  %vecins.1 = insertelement <4 x float> %vecins, float %2, i32 1
+  %vecext.2 = extractelement <4 x float> %0, i32 2
+  %3 = tail call fast float @logf(float %vecext.2)
+  %vecins.2 = insertelement <4 x float> %vecins.1, float %3, i32 2
+  %vecext.3 = extractelement <4 x float> %0, i32 3
+  %4 = tail call fast float @logf(float %vecext.3)
+  %vecins.3 = insertelement <4 x float> %vecins.2, float %4, i32 3
+  ret <4 x float> %vecins.3
+}
+
+declare float @llvm.log.f32(float)
+
+; We can not vectorized log since RISCV has no such instruction.
+define <4 x float> @int_log_4x(ptr %a) {
+; CHECK-LABEL: define <4 x float> @int_log_4x
+; CHECK-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; CHECK-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = tail call fast float @llvm.log.f32(float [[VECEXT]])
+; CHECK-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; CHECK-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = tail call fast float @llvm.log.f32(float [[VECEXT_1]])
+; CHECK-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; CHECK-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = tail call fast float @llvm.log.f32(float [[VECEXT_2]])
+; CHECK-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; CHECK-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; CHECK-NEXT:    [[TMP4:%.*]] = tail call fast float @llvm.log.f32(float [[VECEXT_3]])
+; CHECK-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; CHECK-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+; DEFAULT-LABEL: define <4 x float> @int_log_4x
+; DEFAULT-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; DEFAULT-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; DEFAULT-NEXT:    [[TMP1:%.*]] = tail call fast float @llvm.log.f32(float [[VECEXT]])
+; DEFAULT-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; DEFAULT-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; DEFAULT-NEXT:    [[TMP2:%.*]] = tail call fast float @llvm.log.f32(float [[VECEXT_1]])
+; DEFAULT-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; DEFAULT-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; DEFAULT-NEXT:    [[TMP3:%.*]] = tail call fast float @llvm.log.f32(float [[VECEXT_2]])
+; DEFAULT-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; DEFAULT-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; DEFAULT-NEXT:    [[TMP4:%.*]] = tail call fast float @llvm.log.f32(float [[VECEXT_3]])
+; DEFAULT-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; DEFAULT-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+entry:
+  %0 = load <4 x float>, ptr %a, align 16
+  %vecext = extractelement <4 x float> %0, i32 0
+  %1 = tail call fast float @llvm.log.f32(float %vecext)
+  %vecins = insertelement <4 x float> undef, float %1, i32 0
+  %vecext.1 = extractelement <4 x float> %0, i32 1
+  %2 = tail call fast float @llvm.log.f32(float %vecext.1)
+  %vecins.1 = insertelement <4 x float> %vecins, float %2, i32 1
+  %vecext.2 = extractelement <4 x float> %0, i32 2
+  %3 = tail call fast float @llvm.log.f32(float %vecext.2)
+  %vecins.2 = insertelement <4 x float> %vecins.1, float %3, i32 2
+  %vecext.3 = extractelement <4 x float> %0, i32 3
+  %4 = tail call fast float @llvm.log.f32(float %vecext.3)
+  %vecins.3 = insertelement <4 x float> %vecins.2, float %4, i32 3
+  ret <4 x float> %vecins.3
+}
+
+declare float @sinf(float) readonly nounwind willreturn
+
+; We can not vectorized sin since RISCV has no such instruction.
+define <4 x float> @sin_4x(ptr %a) {
+; CHECK-LABEL: define <4 x float> @sin_4x
+; CHECK-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; CHECK-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = tail call fast float @sinf(float [[VECEXT]])
+; CHECK-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; CHECK-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = tail call fast float @sinf(float [[VECEXT_1]])
+; CHECK-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; CHECK-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = tail call fast float @sinf(float [[VECEXT_2]])
+; CHECK-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; CHECK-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; CHECK-NEXT:    [[TMP4:%.*]] = tail call fast float @sinf(float [[VECEXT_3]])
+; CHECK-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; CHECK-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+; DEFAULT-LABEL: define <4 x float> @sin_4x
+; DEFAULT-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; DEFAULT-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; DEFAULT-NEXT:    [[TMP1:%.*]] = tail call fast float @sinf(float [[VECEXT]])
+; DEFAULT-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; DEFAULT-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; DEFAULT-NEXT:    [[TMP2:%.*]] = tail call fast float @sinf(float [[VECEXT_1]])
+; DEFAULT-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; DEFAULT-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; DEFAULT-NEXT:    [[TMP3:%.*]] = tail call fast float @sinf(float [[VECEXT_2]])
+; DEFAULT-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; DEFAULT-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; DEFAULT-NEXT:    [[TMP4:%.*]] = tail call fast float @sinf(float [[VECEXT_3]])
+; DEFAULT-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; DEFAULT-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+entry:
+  %0 = load <4 x float>, ptr %a, align 16
+  %vecext = extractelement <4 x float> %0, i32 0
+  %1 = tail call fast float @sinf(float %vecext)
+  %vecins = insertelement <4 x float> undef, float %1, i32 0
+  %vecext.1 = extractelement <4 x float> %0, i32 1
+  %2 = tail call fast float @sinf(float %vecext.1)
+  %vecins.1 = insertelement <4 x float> %vecins, float %2, i32 1
+  %vecext.2 = extractelement <4 x float> %0, i32 2
+  %3 = tail call fast float @sinf(float %vecext.2)
+  %vecins.2 = insertelement <4 x float> %vecins.1, float %3, i32 2
+  %vecext.3 = extractelement <4 x float> %0, i32 3
+  %4 = tail call fast float @sinf(float %vecext.3)
+  %vecins.3 = insertelement <4 x float> %vecins.2, float %4, i32 3
+  ret <4 x float> %vecins.3
+}
+
+declare float @llvm.sin.f32(float)
+
+; We can not vectorized sin since RISCV has no such instruction.
+define <4 x float> @int_sin_4x(ptr %a) {
+; CHECK-LABEL: define <4 x float> @int_sin_4x
+; CHECK-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; CHECK-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = tail call fast float @llvm.sin.f32(float [[VECEXT]])
+; CHECK-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; CHECK-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = tail call fast float @llvm.sin.f32(float [[VECEXT_1]])
+; CHECK-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; CHECK-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = tail call fast float @llvm.sin.f32(float [[VECEXT_2]])
+; CHECK-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; CHECK-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; CHECK-NEXT:    [[TMP4:%.*]] = tail call fast float @llvm.sin.f32(float [[VECEXT_3]])
+; CHECK-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; CHECK-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+; DEFAULT-LABEL: define <4 x float> @int_sin_4x
+; DEFAULT-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; DEFAULT-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; DEFAULT-NEXT:    [[TMP1:%.*]] = tail call fast float @llvm.sin.f32(float [[VECEXT]])
+; DEFAULT-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; DEFAULT-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; DEFAULT-NEXT:    [[TMP2:%.*]] = tail call fast float @llvm.sin.f32(float [[VECEXT_1]])
+; DEFAULT-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; DEFAULT-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; DEFAULT-NEXT:    [[TMP3:%.*]] = tail call fast float @llvm.sin.f32(float [[VECEXT_2]])
+; DEFAULT-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; DEFAULT-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; DEFAULT-NEXT:    [[TMP4:%.*]] = tail call fast float @llvm.sin.f32(float [[VECEXT_3]])
+; DEFAULT-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; DEFAULT-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+entry:
+  %0 = load <4 x float>, ptr %a, align 16
+  %vecext = extractelement <4 x float> %0, i32 0
+  %1 = tail call fast float @llvm.sin.f32(float %vecext)
+  %vecins = insertelement <4 x float> undef, float %1, i32 0
+  %vecext.1 = extractelement <4 x float> %0, i32 1
+  %2 = tail call fast float @llvm.sin.f32(float %vecext.1)
+  %vecins.1 = insertelement <4 x float> %vecins, float %2, i32 1
+  %vecext.2 = extractelement <4 x float> %0, i32 2
+  %3 = tail call fast float @llvm.sin.f32(float %vecext.2)
+  %vecins.2 = insertelement <4 x float> %vecins.1, float %3, i32 2
+  %vecext.3 = extractelement <4 x float> %0, i32 3
+  %4 = tail call fast float @llvm.sin.f32(float %vecext.3)
+  %vecins.3 = insertelement <4 x float> %vecins.2, float %4, i32 3
+  ret <4 x float> %vecins.3
+}
+
+declare float @asinf(float) readonly nounwind willreturn
+
+; We can not vectorized asin since RISCV has no such instruction.
+define <4 x float> @asin_4x(ptr %a) {
+; CHECK-LABEL: define <4 x float> @asin_4x
+; CHECK-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; CHECK-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = tail call fast float @asinf(float [[VECEXT]])
+; CHECK-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; CHECK-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = tail call fast float @asinf(float [[VECEXT_1]])
+; CHECK-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; CHECK-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = tail call fast float @asinf(float [[VECEXT_2]])
+; CHECK-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; CHECK-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; CHECK-NEXT:    [[TMP4:%.*]] = tail call fast float @asinf(float [[VECEXT_3]])
+; CHECK-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; CHECK-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+; DEFAULT-LABEL: define <4 x float> @asin_4x
+; DEFAULT-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; DEFAULT-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; DEFAULT-NEXT:    [[TMP1:%.*]] = tail call fast float @asinf(float [[VECEXT]])
+; DEFAULT-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; DEFAULT-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; DEFAULT-NEXT:    [[TMP2:%.*]] = tail call fast float @asinf(float [[VECEXT_1]])
+; DEFAULT-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; DEFAULT-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; DEFAULT-NEXT:    [[TMP3:%.*]] = tail call fast float @asinf(float [[VECEXT_2]])
+; DEFAULT-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; DEFAULT-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; DEFAULT-NEXT:    [[TMP4:%.*]] = tail call fast float @asinf(float [[VECEXT_3]])
+; DEFAULT-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; DEFAULT-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+entry:
+  %0 = load <4 x float>, ptr %a, align 16
+  %vecext = extractelement <4 x float> %0, i32 0
+  %1 = tail call fast float @asinf(float %vecext)
+  %vecins = insertelement <4 x float> undef, float %1, i32 0
+  %vecext.1 = extractelement <4 x float> %0, i32 1
+  %2 = tail call fast float @asinf(float %vecext.1)
+  %vecins.1 = insertelement <4 x float> %vecins, float %2, i32 1
+  %vecext.2 = extractelement <4 x float> %0, i32 2
+  %3 = tail call fast float @asinf(float %vecext.2)
+  %vecins.2 = insertelement <4 x float> %vecins.1, float %3, i32 2
+  %vecext.3 = extractelement <4 x float> %0, i32 3
+  %4 = tail call fast float @asinf(float %vecext.3)
+  %vecins.3 = insertelement <4 x float> %vecins.2, float %4, i32 3
+  ret <4 x float> %vecins.3
+}
+
+declare float @llvm.asin.f32(float)
+
+; We can not vectorized asin since RISCV has no such instruction.
+define <4 x float> @int_asin_4x(ptr %a) {
+; CHECK-LABEL: define <4 x float> @int_asin_4x
+; CHECK-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; CHECK-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = tail call fast float @llvm.asin.f32(float [[VECEXT]])
+; CHECK-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; CHECK-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = tail call fast float @llvm.asin.f32(float [[VECEXT_1]])
+; CHECK-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; CHECK-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = tail call fast float @llvm.asin.f32(float [[VECEXT_2]])
+; CHECK-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; CHECK-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; CHECK-NEXT:    [[TMP4:%.*]] = tail call fast float @llvm.asin.f32(float [[VECEXT_3]])
+; CHECK-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; CHECK-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+; DEFAULT-LABEL: define <4 x float> @int_asin_4x
+; DEFAULT-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; DEFAULT-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; DEFAULT-NEXT:    [[TMP1:%.*]] = tail call fast float @llvm.asin.f32(float [[VECEXT]])
+; DEFAULT-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; DEFAULT-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; DEFAULT-NEXT:    [[TMP2:%.*]] = tail call fast float @llvm.asin.f32(float [[VECEXT_1]])
+; DEFAULT-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; DEFAULT-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; DEFAULT-NEXT:    [[TMP3:%.*]] = tail call fast float @llvm.asin.f32(float [[VECEXT_2]])
+; DEFAULT-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; DEFAULT-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; DEFAULT-NEXT:    [[TMP4:%.*]] = tail call fast float @llvm.asin.f32(float [[VECEXT_3]])
+; DEFAULT-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; DEFAULT-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+entry:
+  %0 = load <4 x float>, ptr %a, align 16
+  %vecext = extractelement <4 x float> %0, i32 0
+  %1 = tail call fast float @llvm.asin.f32(float %vecext)
+  %vecins = insertelement <4 x float> undef, float %1, i32 0
+  %vecext.1 = extractelement <4 x float> %0, i32 1
+  %2 = tail call fast float @llvm.asin.f32(float %vecext.1)
+  %vecins.1 = insertelement <4 x float> %vecins, float %2, i32 1
+  %vecext.2 = extractelement <4 x float> %0, i32 2
+  %3 = tail call fast float @llvm.asin.f32(float %vecext.2)
+  %vecins.2 = insertelement <4 x float> %vecins.1, float %3, i32 2
+  %vecext.3 = extractelement <4 x float> %0, i32 3
+  %4 = tail call fast float @llvm.asin.f32(float %vecext.3)
+  %vecins.3 = insertelement <4 x float> %vecins.2, float %4, i32 3
+  ret <4 x float> %vecins.3
+}
+
+declare float @coshf(float) readonly nounwind willreturn
+
+; We can not vectorized cosh since RISCV has no such instruction.
+define <4 x float> @cosh_4x(ptr %a) {
+; CHECK-LABEL: define <4 x float> @cosh_4x
+; CHECK-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; CHECK-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = tail call fast float @coshf(float [[VECEXT]])
+; CHECK-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; CHECK-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = tail call fast float @coshf(float [[VECEXT_1]])
+; CHECK-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; CHECK-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = tail call fast float @coshf(float [[VECEXT_2]])
+; CHECK-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; CHECK-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; CHECK-NEXT:    [[TMP4:%.*]] = tail call fast float @coshf(float [[VECEXT_3]])
+; CHECK-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; CHECK-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+; DEFAULT-LABEL: define <4 x float> @cosh_4x
+; DEFAULT-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; DEFAULT-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; DEFAULT-NEXT:    [[TMP1:%.*]] = tail call fast float @coshf(float [[VECEXT]])
+; DEFAULT-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; DEFAULT-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; DEFAULT-NEXT:    [[TMP2:%.*]] = tail call fast float @coshf(float [[VECEXT_1]])
+; DEFAULT-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; DEFAULT-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; DEFAULT-NEXT:    [[TMP3:%.*]] = tail call fast float @coshf(float [[VECEXT_2]])
+; DEFAULT-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; DEFAULT-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; DEFAULT-NEXT:    [[TMP4:%.*]] = tail call fast float @coshf(float [[VECEXT_3]])
+; DEFAULT-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; DEFAULT-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+entry:
+  %0 = load <4 x float>, ptr %a, align 16
+  %vecext = extractelement <4 x float> %0, i32 0
+  %1 = tail call fast float @coshf(float %vecext)
+  %vecins = insertelement <4 x float> undef, float %1, i32 0
+  %vecext.1 = extractelement <4 x float> %0, i32 1
+  %2 = tail call fast float @coshf(float %vecext.1)
+  %vecins.1 = insertelement <4 x float> %vecins, float %2, i32 1
+  %vecext.2 = extractelement <4 x float> %0, i32 2
+  %3 = tail call fast float @coshf(float %vecext.2)
+  %vecins.2 = insertelement <4 x float> %vecins.1, float %3, i32 2
+  %vecext.3 = extractelement <4 x float> %0, i32 3
+  %4 = tail call fast float @coshf(float %vecext.3)
+  %vecins.3 = insertelement <4 x float> %vecins.2, float %4, i32 3
+  ret <4 x float> %vecins.3
+}
+
+declare float @llvm.cosh.f32(float)
+
+; We can not vectorized cosh since RISCV has no such instruction.
+define <4 x float> @int_cosh_4x(ptr %a) {
+; CHECK-LABEL: define <4 x float> @int_cosh_4x
+; CHECK-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; CHECK-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = tail call fast float @llvm.cosh.f32(float [[VECEXT]])
+; CHECK-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; CHECK-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = tail call fast float @llvm.cosh.f32(float [[VECEXT_1]])
+; CHECK-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; CHECK-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = tail call fast float @llvm.cosh.f32(float [[VECEXT_2]])
+; CHECK-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; CHECK-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; CHECK-NEXT:    [[TMP4:%.*]] = tail call fast float @llvm.cosh.f32(float [[VECEXT_3]])
+; CHECK-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; CHECK-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+; DEFAULT-LABEL: define <4 x float> @int_cosh_4x
+; DEFAULT-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; DEFAULT-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; DEFAULT-NEXT:    [[TMP1:%.*]] = tail call fast float @llvm.cosh.f32(float [[VECEXT]])
+; DEFAULT-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; DEFAULT-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; DEFAULT-NEXT:    [[TMP2:%.*]] = tail call fast float @llvm.cosh.f32(float [[VECEXT_1]])
+; DEFAULT-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; DEFAULT-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; DEFAULT-NEXT:    [[TMP3:%.*]] = tail call fast float @llvm.cosh.f32(float [[VECEXT_2]])
+; DEFAULT-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; DEFAULT-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; DEFAULT-NEXT:    [[TMP4:%.*]] = tail call fast float @llvm.cosh.f32(float [[VECEXT_3]])
+; DEFAULT-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; DEFAULT-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+entry:
+  %0 = load <4 x float>, ptr %a, align 16
+  %vecext = extractelement <4 x float> %0, i32 0
+  %1 = tail call fast float @llvm.cosh.f32(float %vecext)
+  %vecins = insertelement <4 x float> undef, float %1, i32 0
+  %vecext.1 = extractelement <4 x float> %0, i32 1
+  %2 = tail call fast float @llvm.cosh.f32(float %vecext.1)
+  %vecins.1 = insertelement <4 x float> %vecins, float %2, i32 1
+  %vecext.2 = extractelement <4 x float> %0, i32 2
+  %3 = tail call fast float @llvm.cosh.f32(float %vecext.2)
+  %vecins.2 = insertelement <4 x float> %vecins.1, float %3, i32 2
+  %vecext.3 = extractelement <4 x float> %0, i32 3
+  %4 = tail call fast float @llvm.cosh.f32(float %vecext.3)
+  %vecins.3 = insertelement <4 x float> %vecins.2, float %4, i32 3
+  ret <4 x float> %vecins.3
+}
+
+declare float @atanhf(float) readonly nounwind willreturn
+
+; We can not vectorized atanh since RISCV has no such instruction.
+define <4 x float> @atanh_4x(ptr %a) {
+; CHECK-LABEL: define <4 x float> @atanh_4x
+; CHECK-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; CHECK-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = tail call fast float @atanhf(float [[VECEXT]])
+; CHECK-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; CHECK-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = tail call fast float @atanhf(float [[VECEXT_1]])
+; CHECK-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; CHECK-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = tail call fast float @atanhf(float [[VECEXT_2]])
+; CHECK-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; CHECK-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; CHECK-NEXT:    [[TMP4:%.*]] = tail call fast float @atanhf(float [[VECEXT_3]])
+; CHECK-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; CHECK-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+; DEFAULT-LABEL: define <4 x float> @atanh_4x
+; DEFAULT-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; DEFAULT-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; DEFAULT-NEXT:    [[TMP1:%.*]] = tail call fast float @atanhf(float [[VECEXT]])
+; DEFAULT-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; DEFAULT-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; DEFAULT-NEXT:    [[TMP2:%.*]] = tail call fast float @atanhf(float [[VECEXT_1]])
+; DEFAULT-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; DEFAULT-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; DEFAULT-NEXT:    [[TMP3:%.*]] = tail call fast float @atanhf(float [[VECEXT_2]])
+; DEFAULT-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; DEFAULT-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; DEFAULT-NEXT:    [[TMP4:%.*]] = tail call fast float @atanhf(float [[VECEXT_3]])
+; DEFAULT-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; DEFAULT-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+entry:
+  %0 = load <4 x float>, ptr %a, align 16
+  %vecext = extractelement <4 x float> %0, i32 0
+  %1 = tail call fast float @atanhf(float %vecext)
+  %vecins = insertelement <4 x float> undef, float %1, i32 0
+  %vecext.1 = extractelement <4 x float> %0, i32 1
+  %2 = tail call fast float @atanhf(float %vecext.1)
+  %vecins.1 = insertelement <4 x float> %vecins, float %2, i32 1
+  %vecext.2 = extractelement <4 x float> %0, i32 2
+  %3 = tail call fast float @atanhf(float %vecext.2)
+  %vecins.2 = insertelement <4 x float> %vecins.1, float %3, i32 2
+  %vecext.3 = extractelement <4 x float> %0, i32 3
+  %4 = tail call fast float @atanhf(float %vecext.3)
+  %vecins.3 = insertelement <4 x float> %vecins.2, float %4, i32 3
+  ret <4 x float> %vecins.3
+}
+
+declare float @llvm.atanh.f32(float)
+
+; We can not vectorized atanh since RISCV has no such instruction.
+define <4 x float> @int_atanh_4x(ptr %a) {
+; CHECK-LABEL: define <4 x float> @int_atanh_4x
+; CHECK-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; CHECK-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = tail call fast float @llvm.atanh.f32(float [[VECEXT]])
+; CHECK-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; CHECK-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = tail call fast float @llvm.atanh.f32(float [[VECEXT_1]])
+; CHECK-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; CHECK-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = tail call fast float @llvm.atanh.f32(float [[VECEXT_2]])
+; CHECK-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; CHECK-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; CHECK-NEXT:    [[TMP4:%.*]] = tail call fast float @llvm.atanh.f32(float [[VECEXT_3]])
+; CHECK-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; CHECK-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+; DEFAULT-LABEL: define <4 x float> @int_atanh_4x
+; DEFAULT-SAME: (ptr [[A:%.*]]) #[[ATTR1]] {
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load <4 x float>, ptr [[A]], align 16
+; DEFAULT-NEXT:    [[VECEXT:%.*]] = extractelement <4 x float> [[TMP0]], i32 0
+; DEFAULT-NEXT:    [[TMP1:%.*]] = tail call fast float @llvm.atanh.f32(float [[VECEXT]])
+; DEFAULT-NEXT:    [[VECINS:%.*]] = insertelement <4 x float> undef, float [[TMP1]], i32 0
+; DEFAULT-NEXT:    [[VECEXT_1:%.*]] = extractelement <4 x float> [[TMP0]], i32 1
+; DEFAULT-NEXT:    [[TMP2:%.*]] = tail call fast float @llvm.atanh.f32(float [[VECEXT_1]])
+; DEFAULT-NEXT:    [[VECINS_1:%.*]] = insertelement <4 x float> [[VECINS]], float [[TMP2]], i32 1
+; DEFAULT-NEXT:    [[VECEXT_2:%.*]] = extractelement <4 x float> [[TMP0]], i32 2
+; DEFAULT-NEXT:    [[TMP3:%.*]] = tail call fast float @llvm.atanh.f32(float [[VECEXT_2]])
+; DEFAULT-NEXT:    [[VECINS_2:%.*]] = insertelement <4 x float> [[VECINS_1]], float [[TMP3]], i32 2
+; DEFAULT-NEXT:    [[VECEXT_3:%.*]] = extractelement <4 x float> [[TMP0]], i32 3
+; DEFAULT-NEXT:    [[TMP4:%.*]] = tail call fast float @llvm.atanh.f32(float [[VECEXT_3]])
+; DEFAULT-NEXT:    [[VECINS_3:%.*]] = insertelement <4 x float> [[VECINS_2]], float [[TMP4]], i32 3
+; DEFAULT-NEXT:    ret <4 x float> [[VECINS_3]]
+;
+entry:
+  %0 = load <4 x float>, ptr %a, align 16
+  %vecext = extractelement <4 x float> %0, i32 0
+  %1 = tail call fast float @llvm.atanh.f32(float %vecext)
+  %vecins = insertelement <4 x float> undef, float %1, i32 0
+  %vecext.1 = extractelement <4 x float> %0, i32 1
+  %2 = tail call fast float @llvm.atanh.f32(float %vecext.1)
+  %vecins.1 = insertelement <4 x float> %vecins, float %2, i32 1
+  %vecext.2 = extractelement <4 x float> %0, i32 2
+  %3 = tail call fast float @llvm.atanh.f32(float %vecext.2)
+  %vecins.2 = insertelement <4 x float> %vecins.1, float %3, i32 2
+  %vecext.3 = extractelement <4 x float> %0, i32 3
+  %4 = tail call fast float @llvm.atanh.f32(float %vecext.3)
+  %vecins.3 = insertelement <4 x float> %vecins.2, float %4, i32 3
+  ret <4 x float> %vecins.3
+}
diff --git a/llvm/test/Transforms/SLPVectorizer/RISCV/reductions.ll b/llvm/test/Transforms/SLPVectorizer/RISCV/reductions.ll
index 5ea7d14e9e5b4..268e4f3189d5e 100644
--- a/llvm/test/Transforms/SLPVectorizer/RISCV/reductions.ll
+++ b/llvm/test/Transforms/SLPVectorizer/RISCV/reductions.ll
@@ -400,6 +400,261 @@ entry:
   ret i8 %add13.7
 }
 
+declare i8 @llvm.smin.i8(i8, i8)
+
+define i8 @reduce_smin(ptr %a, ptr %b) {
+; CHECK-LABEL: @reduce_smin(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_BUF:%.*]], ptr [[A:%.*]], i64 0, i32 0, i64 0
+; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds [[STRUCT_BUF]], ptr [[B:%.*]], i64 0, i32 0, i64 0
+; CHECK-NEXT:    [[TMP0:%.*]] = load <8 x i8>, ptr [[ARRAYIDX]], align 1
+; CHECK-NEXT:    [[TMP1:%.*]] = load <8 x i8>, ptr [[ARRAYIDX3]], align 1
+; CHECK-NEXT:    [[TMP2:%.*]] = and <8 x i8> [[TMP1]], [[TMP0]]
+; CHECK-NEXT:    [[TMP3:%.*]] = call i8 @llvm.vector.reduce.smin.v8i8(<8 x i8> [[TMP2]])
+; CHECK-NEXT:    ret i8 [[TMP3]]
+;
+entry:
+  %arrayidx = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 0
+  %0 = load i8, i8* %arrayidx, align 1
+  %arrayidx3 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 0
+  %1 = load i8, i8* %arrayidx3, align 1
+  %and12 = and i8 %1, %0
+  %arrayidx.1 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 1
+  %2 = load i8, i8* %arrayidx.1, align 1
+  %arrayidx3.1 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 1
+  %3 = load i8, i8* %arrayidx3.1, align 1
+  %and12.1 = and i8 %3, %2
+  %4 = tail call i8 @llvm.smin.i8(i8 %and12, i8 %and12.1)
+  %arrayidx.2 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 2
+  %5 = load i8, i8* %arrayidx.2, align 1
+  %arrayidx3.2 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 2
+  %6 = load i8, i8* %arrayidx3.2, align 1
+  %and12.2 = and i8 %6, %5
+  %7 = tail call i8 @llvm.smin.i8(i8 %4, i8 %and12.2)
+  %arrayidx.3 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 3
+  %8 = load i8, i8* %arrayidx.3, align 1
+  %arrayidx3.3 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 3
+  %9 = load i8, i8* %arrayidx3.3, align 1
+  %and12.3 = and i8 %9, %8
+  %10 = tail call i8 @llvm.smin.i8(i8 %7, i8 %and12.3)
+  %arrayidx.4 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 4
+  %11 = load i8, i8* %arrayidx.4, align 1
+  %arrayidx3.4 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 4
+  %12 = load i8, i8* %arrayidx3.4, align 1
+  %and12.4 = and i8 %12, %11
+  %13 = tail call i8 @llvm.smin.i8(i8 %10, i8 %and12.4)
+  %arrayidx.5 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 5
+  %14 = load i8, i8* %arrayidx.5, align 1
+  %arrayidx3.5 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 5
+  %15 = load i8, i8* %arrayidx3.5, align 1
+  %and12.5 = and i8 %15, %14
+  %16 = tail call i8 @llvm.smin.i8(i8 %13, i8 %and12.5)
+  %arrayidx.6 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 6
+  %17 = load i8, i8* %arrayidx.6, align 1
+  %arrayidx3.6 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 6
+  %18 = load i8, i8* %arrayidx3.6, align 1
+  %and12.6 = and i8 %18, %17
+  %19 = tail call i8 @llvm.smin.i8(i8 %16, i8 %and12.6)
+  %arrayidx.7 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 7
+  %20 = load i8, i8* %arrayidx.7, align 1
+  %arrayidx3.7 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 7
+  %21 = load i8, i8* %arrayidx3.7, align 1
+  %and12.7 = and i8 %21, %20
+  %22 = tail call i8 @llvm.smin.i8(i8 %19, i8 %and12.7)
+  ret i8 %22
+}
+
+declare i8 @llvm.smax.i8(i8, i8)
+
+define i8 @reduce_smax(ptr %a, ptr %b) {
+; CHECK-LABEL: @reduce_smax(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_BUF:%.*]], ptr [[A:%.*]], i64 0, i32 0, i64 0
+; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds [[STRUCT_BUF]], ptr [[B:%.*]], i64 0, i32 0, i64 0
+; CHECK-NEXT:    [[TMP0:%.*]] = load <8 x i8>, ptr [[ARRAYIDX]], align 1
+; CHECK-NEXT:    [[TMP1:%.*]] = load <8 x i8>, ptr [[ARRAYIDX3]], align 1
+; CHECK-NEXT:    [[TMP2:%.*]] = and <8 x i8> [[TMP1]], [[TMP0]]
+; CHECK-NEXT:    [[TMP3:%.*]] = call i8 @llvm.vector.reduce.smax.v8i8(<8 x i8> [[TMP2]])
+; CHECK-NEXT:    ret i8 [[TMP3]]
+;
+entry:
+  %arrayidx = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 0
+  %0 = load i8, i8* %arrayidx, align 1
+  %arrayidx3 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 0
+  %1 = load i8, i8* %arrayidx3, align 1
+  %and12 = and i8 %1, %0
+  %arrayidx.1 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 1
+  %2 = load i8, i8* %arrayidx.1, align 1
+  %arrayidx3.1 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 1
+  %3 = load i8, i8* %arrayidx3.1, align 1
+  %and12.1 = and i8 %3, %2
+  %4 = tail call i8 @llvm.smax.i8(i8 %and12, i8 %and12.1)
+  %arrayidx.2 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 2
+  %5 = load i8, i8* %arrayidx.2, align 1
+  %arrayidx3.2 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 2
+  %6 = load i8, i8* %arrayidx3.2, align 1
+  %and12.2 = and i8 %6, %5
+  %7 = tail call i8 @llvm.smax.i8(i8 %4, i8 %and12.2)
+  %arrayidx.3 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 3
+  %8 = load i8, i8* %arrayidx.3, align 1
+  %arrayidx3.3 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 3
+  %9 = load i8, i8* %arrayidx3.3, align 1
+  %and12.3 = and i8 %9, %8
+  %10 = tail call i8 @llvm.smax.i8(i8 %7, i8 %and12.3)
+  %arrayidx.4 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 4
+  %11 = load i8, i8* %arrayidx.4, align 1
+  %arrayidx3.4 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 4
+  %12 = load i8, i8* %arrayidx3.4, align 1
+  %and12.4 = and i8 %12, %11
+  %13 = tail call i8 @llvm.smax.i8(i8 %10, i8 %and12.4)
+  %arrayidx.5 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 5
+  %14 = load i8, i8* %arrayidx.5, align 1
+  %arrayidx3.5 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 5
+  %15 = load i8, i8* %arrayidx3.5, align 1
+  %and12.5 = and i8 %15, %14
+  %16 = tail call i8 @llvm.smax.i8(i8 %13, i8 %and12.5)
+  %arrayidx.6 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 6
+  %17 = load i8, i8* %arrayidx.6, align 1
+  %arrayidx3.6 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 6
+  %18 = load i8, i8* %arrayidx3.6, align 1
+  %and12.6 = and i8 %18, %17
+  %19 = tail call i8 @llvm.smax.i8(i8 %16, i8 %and12.6)
+  %arrayidx.7 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 7
+  %20 = load i8, i8* %arrayidx.7, align 1
+  %arrayidx3.7 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 7
+  %21 = load i8, i8* %arrayidx3.7, align 1
+  %and12.7 = and i8 %21, %20
+  %22 = tail call i8 @llvm.smax.i8(i8 %19, i8 %and12.7)
+  ret i8 %22
+}
+
+declare i8 @llvm.umax.i8(i8, i8)
+
+define i8 @reduce_umax(ptr %a, ptr %b) {
+; CHECK-LABEL: @reduce_umax(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_BUF:%.*]], ptr [[A:%.*]], i64 0, i32 0, i64 0
+; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds [[STRUCT_BUF]], ptr [[B:%.*]], i64 0, i32 0, i64 0
+; CHECK-NEXT:    [[TMP0:%.*]] = load <8 x i8>, ptr [[ARRAYIDX]], align 1
+; CHECK-NEXT:    [[TMP1:%.*]] = load <8 x i8>, ptr [[ARRAYIDX3]], align 1
+; CHECK-NEXT:    [[TMP2:%.*]] = and <8 x i8> [[TMP1]], [[TMP0]]
+; CHECK-NEXT:    [[TMP3:%.*]] = call i8 @llvm.vector.reduce.umax.v8i8(<8 x i8> [[TMP2]])
+; CHECK-NEXT:    ret i8 [[TMP3]]
+;
+entry:
+  %arrayidx = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 0
+  %0 = load i8, i8* %arrayidx, align 1
+  %arrayidx3 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 0
+  %1 = load i8, i8* %arrayidx3, align 1
+  %and12 = and i8 %1, %0
+  %arrayidx.1 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 1
+  %2 = load i8, i8* %arrayidx.1, align 1
+  %arrayidx3.1 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 1
+  %3 = load i8, i8* %arrayidx3.1, align 1
+  %and12.1 = and i8 %3, %2
+  %4 = tail call i8 @llvm.umax.i8(i8 %and12, i8 %and12.1)
+  %arrayidx.2 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 2
+  %5 = load i8, i8* %arrayidx.2, align 1
+  %arrayidx3.2 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 2
+  %6 = load i8, i8* %arrayidx3.2, align 1
+  %and12.2 = and i8 %6, %5
+  %7 = tail call i8 @llvm.umax.i8(i8 %4, i8 %and12.2)
+  %arrayidx.3 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 3
+  %8 = load i8, i8* %arrayidx.3, align 1
+  %arrayidx3.3 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 3
+  %9 = load i8, i8* %arrayidx3.3, align 1
+  %and12.3 = and i8 %9, %8
+  %10 = tail call i8 @llvm.umax.i8(i8 %7, i8 %and12.3)
+  %arrayidx.4 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 4
+  %11 = load i8, i8* %arrayidx.4, align 1
+  %arrayidx3.4 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 4
+  %12 = load i8, i8* %arrayidx3.4, align 1
+  %and12.4 = and i8 %12, %11
+  %13 = tail call i8 @llvm.umax.i8(i8 %10, i8 %and12.4)
+  %arrayidx.5 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 5
+  %14 = load i8, i8* %arrayidx.5, align 1
+  %arrayidx3.5 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 5
+  %15 = load i8, i8* %arrayidx3.5, align 1
+  %and12.5 = and i8 %15, %14
+  %16 = tail call i8 @llvm.umax.i8(i8 %13, i8 %and12.5)
+  %arrayidx.6 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 6
+  %17 = load i8, i8* %arrayidx.6, align 1
+  %arrayidx3.6 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 6
+  %18 = load i8, i8* %arrayidx3.6, align 1
+  %and12.6 = and i8 %18, %17
+  %19 = tail call i8 @llvm.umax.i8(i8 %16, i8 %and12.6)
+  %arrayidx.7 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 7
+  %20 = load i8, i8* %arrayidx.7, align 1
+  %arrayidx3.7 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 7
+  %21 = load i8, i8* %arrayidx3.7, align 1
+  %and12.7 = and i8 %21, %20
+  %22 = tail call i8 @llvm.umax.i8(i8 %19, i8 %and12.7)
+  ret i8 %22
+}
+
+declare i8 @llvm.umin.i8(i8, i8)
+
+define i8 @reduce_umin(ptr %a, ptr %b) {
+; CHECK-LABEL: @reduce_umin(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_BUF:%.*]], ptr [[A:%.*]], i64 0, i32 0, i64 0
+; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds [[STRUCT_BUF]], ptr [[B:%.*]], i64 0, i32 0, i64 0
+; CHECK-NEXT:    [[TMP0:%.*]] = load <8 x i8>, ptr [[ARRAYIDX]], align 1
+; CHECK-NEXT:    [[TMP1:%.*]] = load <8 x i8>, ptr [[ARRAYIDX3]], align 1
+; CHECK-NEXT:    [[TMP2:%.*]] = and <8 x i8> [[TMP1]], [[TMP0]]
+; CHECK-NEXT:    [[TMP3:%.*]] = call i8 @llvm.vector.reduce.umin.v8i8(<8 x i8> [[TMP2]])
+; CHECK-NEXT:    ret i8 [[TMP3]]
+;
+entry:
+  %arrayidx = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 0
+  %0 = load i8, i8* %arrayidx, align 1
+  %arrayidx3 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 0
+  %1 = load i8, i8* %arrayidx3, align 1
+  %and12 = and i8 %1, %0
+  %arrayidx.1 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 1
+  %2 = load i8, i8* %arrayidx.1, align 1
+  %arrayidx3.1 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 1
+  %3 = load i8, i8* %arrayidx3.1, align 1
+  %and12.1 = and i8 %3, %2
+  %4 = tail call i8 @llvm.umin.i8(i8 %and12, i8 %and12.1)
+  %arrayidx.2 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 2
+  %5 = load i8, i8* %arrayidx.2, align 1
+  %arrayidx3.2 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 2
+  %6 = load i8, i8* %arrayidx3.2, align 1
+  %and12.2 = and i8 %6, %5
+  %7 = tail call i8 @llvm.umin.i8(i8 %4, i8 %and12.2)
+  %arrayidx.3 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 3
+  %8 = load i8, i8* %arrayidx.3, align 1
+  %arrayidx3.3 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 3
+  %9 = load i8, i8* %arrayidx3.3, align 1
+  %and12.3 = and i8 %9, %8
+  %10 = tail call i8 @llvm.umin.i8(i8 %7, i8 %and12.3)
+  %arrayidx.4 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 4
+  %11 = load i8, i8* %arrayidx.4, align 1
+  %arrayidx3.4 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 4
+  %12 = load i8, i8* %arrayidx3.4, align 1
+  %and12.4 = and i8 %12, %11
+  %13 = tail call i8 @llvm.umin.i8(i8 %10, i8 %and12.4)
+  %arrayidx.5 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 5
+  %14 = load i8, i8* %arrayidx.5, align 1
+  %arrayidx3.5 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 5
+  %15 = load i8, i8* %arrayidx3.5, align 1
+  %and12.5 = and i8 %15, %14
+  %16 = tail call i8 @llvm.umin.i8(i8 %13, i8 %and12.5)
+  %arrayidx.6 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 6
+  %17 = load i8, i8* %arrayidx.6, align 1
+  %arrayidx3.6 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 6
+  %18 = load i8, i8* %arrayidx3.6, align 1
+  %and12.6 = and i8 %18, %17
+  %19 = tail call i8 @llvm.umin.i8(i8 %16, i8 %and12.6)
+  %arrayidx.7 = getelementptr inbounds %struct.buf, ptr %a, i64 0, i32 0, i64 7
+  %20 = load i8, i8* %arrayidx.7, align 1
+  %arrayidx3.7 = getelementptr inbounds %struct.buf, ptr %b, i64 0, i32 0, i64 7
+  %21 = load i8, i8* %arrayidx3.7, align 1
+  %and12.7 = and i8 %21, %20
+  %22 = tail call i8 @llvm.umin.i8(i8 %19, i8 %and12.7)
+  ret i8 %22
+}
 
 ; Next batch exercise reductions involing zext of narrower loads
 
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/PR35865-inseltpoison.ll b/llvm/test/Transforms/SLPVectorizer/X86/PR35865-inseltpoison.ll
index 7080b693e0ff6..cc1fa07d4c4f3 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/PR35865-inseltpoison.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/PR35865-inseltpoison.ll
@@ -4,14 +4,11 @@
 define void @test(<16 x half> %v) {
 ; CHECK-LABEL: @test(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[TMP0:%.*]] = extractelement <16 x half> [[V:%.*]], i32 4
-; CHECK-NEXT:    [[TMP1:%.*]] = extractelement <16 x half> [[V]], i32 5
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x half> poison, half [[TMP0]], i32 0
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x half> [[TMP2]], half [[TMP1]], i32 1
-; CHECK-NEXT:    [[TMP4:%.*]] = fpext <2 x half> [[TMP3]] to <2 x float>
-; CHECK-NEXT:    [[TMP5:%.*]] = bitcast <2 x float> [[TMP4]] to <2 x i32>
-; CHECK-NEXT:    [[TMP6:%.*]] = shufflevector <2 x i32> [[TMP5]], <2 x i32> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-; CHECK-NEXT:    [[VECINS_I_5_I1:%.*]] = shufflevector <8 x i32> [[TMP6]], <8 x i32> poison, <8 x i32> <i32 undef, i32 undef, i32 undef, i32 undef, i32 0, i32 1, i32 undef, i32 undef>
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <16 x half> [[V:%.*]], <16 x half> poison, <2 x i32> <i32 4, i32 5>
+; CHECK-NEXT:    [[TMP1:%.*]] = fpext <2 x half> [[TMP0]] to <2 x float>
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast <2 x float> [[TMP1]] to <2 x i32>
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <2 x i32> [[TMP2]], <2 x i32> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+; CHECK-NEXT:    [[VECINS_I_5_I1:%.*]] = shufflevector <8 x i32> [[TMP3]], <8 x i32> poison, <8 x i32> <i32 undef, i32 undef, i32 undef, i32 undef, i32 0, i32 1, i32 undef, i32 undef>
 ; CHECK-NEXT:    ret void
 ;
 entry:
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/PR35865.ll b/llvm/test/Transforms/SLPVectorizer/X86/PR35865.ll
index 244faa4b3e78e..a9e3aa33ae92d 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/PR35865.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/PR35865.ll
@@ -4,14 +4,11 @@
 define void @test(<16 x half> %v) {
 ; CHECK-LABEL: @test(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[TMP0:%.*]] = extractelement <16 x half> [[V:%.*]], i32 4
-; CHECK-NEXT:    [[TMP1:%.*]] = extractelement <16 x half> [[V]], i32 5
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x half> poison, half [[TMP0]], i32 0
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x half> [[TMP2]], half [[TMP1]], i32 1
-; CHECK-NEXT:    [[TMP4:%.*]] = fpext <2 x half> [[TMP3]] to <2 x float>
-; CHECK-NEXT:    [[TMP5:%.*]] = bitcast <2 x float> [[TMP4]] to <2 x i32>
-; CHECK-NEXT:    [[TMP6:%.*]] = shufflevector <2 x i32> [[TMP5]], <2 x i32> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-; CHECK-NEXT:    [[VECINS_I_5_I1:%.*]] = shufflevector <8 x i32> [[TMP6]], <8 x i32> undef, <8 x i32> <i32 8, i32 9, i32 10, i32 11, i32 0, i32 1, i32 14, i32 15>
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <16 x half> [[V:%.*]], <16 x half> poison, <2 x i32> <i32 4, i32 5>
+; CHECK-NEXT:    [[TMP1:%.*]] = fpext <2 x half> [[TMP0]] to <2 x float>
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast <2 x float> [[TMP1]] to <2 x i32>
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <2 x i32> [[TMP2]], <2 x i32> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+; CHECK-NEXT:    [[VECINS_I_5_I1:%.*]] = shufflevector <8 x i32> [[TMP3]], <8 x i32> undef, <8 x i32> <i32 8, i32 9, i32 10, i32 11, i32 0, i32 1, i32 14, i32 15>
 ; CHECK-NEXT:    ret void
 ;
 entry:
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/PR39774.ll b/llvm/test/Transforms/SLPVectorizer/X86/PR39774.ll
index a1babad830b88..abf1d7abdc122 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/PR39774.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/PR39774.ll
@@ -7,18 +7,16 @@ define void @Test(i32) {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
-; CHECK-NEXT:    [[TMP1:%.*]] = phi <2 x i32> [ [[TMP11:%.*]], [[LOOP]] ], [ zeroinitializer, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    [[TMP1:%.*]] = phi <2 x i32> [ [[TMP9:%.*]], [[LOOP]] ], [ zeroinitializer, [[ENTRY:%.*]] ]
 ; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <2 x i32> [[TMP1]], <2 x i32> poison, <8 x i32> <i32 0, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
-; CHECK-NEXT:    [[TMP3:%.*]] = extractelement <8 x i32> [[TMP2]], i32 1
-; CHECK-NEXT:    [[TMP4:%.*]] = add <8 x i32> [[TMP2]], <i32 0, i32 55, i32 285, i32 1240, i32 1496, i32 8555, i32 12529, i32 13685>
-; CHECK-NEXT:    [[TMP5:%.*]] = call i32 @llvm.vector.reduce.and.v8i32(<8 x i32> [[TMP4]])
-; CHECK-NEXT:    [[OP_RDX:%.*]] = and i32 [[TMP0:%.*]], [[TMP5]]
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x i32> <i32 poison, i32 14910>, i32 [[OP_RDX]], i32 0
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <2 x i32> poison, i32 [[TMP3]], i32 0
-; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <2 x i32> [[TMP7]], <2 x i32> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP9:%.*]] = and <2 x i32> [[TMP6]], [[TMP8]]
-; CHECK-NEXT:    [[TMP10:%.*]] = add <2 x i32> [[TMP6]], [[TMP8]]
-; CHECK-NEXT:    [[TMP11]] = shufflevector <2 x i32> [[TMP9]], <2 x i32> [[TMP10]], <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    [[TMP3:%.*]] = add <8 x i32> [[TMP2]], <i32 0, i32 55, i32 285, i32 1240, i32 1496, i32 8555, i32 12529, i32 13685>
+; CHECK-NEXT:    [[TMP4:%.*]] = call i32 @llvm.vector.reduce.and.v8i32(<8 x i32> [[TMP3]])
+; CHECK-NEXT:    [[OP_RDX:%.*]] = and i32 [[TMP0:%.*]], [[TMP4]]
+; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x i32> <i32 poison, i32 14910>, i32 [[OP_RDX]], i32 0
+; CHECK-NEXT:    [[TMP6:%.*]] = shufflevector <2 x i32> [[TMP1]], <2 x i32> poison, <2 x i32> <i32 1, i32 1>
+; CHECK-NEXT:    [[TMP7:%.*]] = and <2 x i32> [[TMP5]], [[TMP6]]
+; CHECK-NEXT:    [[TMP8:%.*]] = add <2 x i32> [[TMP5]], [[TMP6]]
+; CHECK-NEXT:    [[TMP9]] = shufflevector <2 x i32> [[TMP7]], <2 x i32> [[TMP8]], <2 x i32> <i32 0, i32 3>
 ; CHECK-NEXT:    br label [[LOOP]]
 ;
 ; FORCE_REDUCTION-LABEL: @Test(
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/alternate-calls-inseltpoison.ll b/llvm/test/Transforms/SLPVectorizer/X86/alternate-calls-inseltpoison.ll
index c801cd296917f..ee217c07895ee 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/alternate-calls-inseltpoison.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/alternate-calls-inseltpoison.ll
@@ -11,12 +11,12 @@ define <8 x float> @ceil_floor(<8 x float> %a) {
 ; SSE-NEXT:    [[A0:%.*]] = extractelement <8 x float> [[A:%.*]], i64 0
 ; SSE-NEXT:    [[A3:%.*]] = extractelement <8 x float> [[A]], i64 3
 ; SSE-NEXT:    [[AB0:%.*]] = call float @llvm.ceil.f32(float [[A0]])
-; SSE-NEXT:    [[TMP1:%.*]] = shufflevector <8 x float> [[A]], <8 x float> undef, <2 x i32> <i32 1, i32 2>
+; SSE-NEXT:    [[TMP1:%.*]] = shufflevector <8 x float> [[A]], <8 x float> poison, <2 x i32> <i32 1, i32 2>
 ; SSE-NEXT:    [[TMP2:%.*]] = call <2 x float> @llvm.floor.v2f32(<2 x float> [[TMP1]])
 ; SSE-NEXT:    [[AB3:%.*]] = call float @llvm.ceil.f32(float [[A3]])
-; SSE-NEXT:    [[TMP3:%.*]] = shufflevector <8 x float> [[A]], <8 x float> undef, <2 x i32> <i32 4, i32 5>
+; SSE-NEXT:    [[TMP3:%.*]] = shufflevector <8 x float> [[A]], <8 x float> poison, <2 x i32> <i32 4, i32 5>
 ; SSE-NEXT:    [[TMP4:%.*]] = call <2 x float> @llvm.ceil.v2f32(<2 x float> [[TMP3]])
-; SSE-NEXT:    [[TMP5:%.*]] = shufflevector <8 x float> [[A]], <8 x float> undef, <2 x i32> <i32 6, i32 7>
+; SSE-NEXT:    [[TMP5:%.*]] = shufflevector <8 x float> [[A]], <8 x float> poison, <2 x i32> <i32 6, i32 7>
 ; SSE-NEXT:    [[TMP6:%.*]] = call <2 x float> @llvm.floor.v2f32(<2 x float> [[TMP5]])
 ; SSE-NEXT:    [[R0:%.*]] = insertelement <8 x float> poison, float [[AB0]], i64 0
 ; SSE-NEXT:    [[TMP7:%.*]] = shufflevector <2 x float> [[TMP2]], <2 x float> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
@@ -32,12 +32,12 @@ define <8 x float> @ceil_floor(<8 x float> %a) {
 ; SLM-NEXT:    [[A0:%.*]] = extractelement <8 x float> [[A:%.*]], i64 0
 ; SLM-NEXT:    [[A3:%.*]] = extractelement <8 x float> [[A]], i64 3
 ; SLM-NEXT:    [[AB0:%.*]] = call float @llvm.ceil.f32(float [[A0]])
-; SLM-NEXT:    [[TMP1:%.*]] = shufflevector <8 x float> [[A]], <8 x float> undef, <2 x i32> <i32 1, i32 2>
+; SLM-NEXT:    [[TMP1:%.*]] = shufflevector <8 x float> [[A]], <8 x float> poison, <2 x i32> <i32 1, i32 2>
 ; SLM-NEXT:    [[TMP2:%.*]] = call <2 x float> @llvm.floor.v2f32(<2 x float> [[TMP1]])
 ; SLM-NEXT:    [[AB3:%.*]] = call float @llvm.ceil.f32(float [[A3]])
-; SLM-NEXT:    [[TMP3:%.*]] = shufflevector <8 x float> [[A]], <8 x float> undef, <2 x i32> <i32 4, i32 5>
+; SLM-NEXT:    [[TMP3:%.*]] = shufflevector <8 x float> [[A]], <8 x float> poison, <2 x i32> <i32 4, i32 5>
 ; SLM-NEXT:    [[TMP4:%.*]] = call <2 x float> @llvm.ceil.v2f32(<2 x float> [[TMP3]])
-; SLM-NEXT:    [[TMP5:%.*]] = shufflevector <8 x float> [[A]], <8 x float> undef, <2 x i32> <i32 6, i32 7>
+; SLM-NEXT:    [[TMP5:%.*]] = shufflevector <8 x float> [[A]], <8 x float> poison, <2 x i32> <i32 6, i32 7>
 ; SLM-NEXT:    [[TMP6:%.*]] = call <2 x float> @llvm.floor.v2f32(<2 x float> [[TMP5]])
 ; SLM-NEXT:    [[R0:%.*]] = insertelement <8 x float> poison, float [[AB0]], i64 0
 ; SLM-NEXT:    [[TMP7:%.*]] = shufflevector <2 x float> [[TMP2]], <2 x float> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
@@ -53,12 +53,12 @@ define <8 x float> @ceil_floor(<8 x float> %a) {
 ; AVX-NEXT:    [[A0:%.*]] = extractelement <8 x float> [[A:%.*]], i64 0
 ; AVX-NEXT:    [[A3:%.*]] = extractelement <8 x float> [[A]], i64 3
 ; AVX-NEXT:    [[AB0:%.*]] = call float @llvm.ceil.f32(float [[A0]])
-; AVX-NEXT:    [[TMP1:%.*]] = shufflevector <8 x float> [[A]], <8 x float> undef, <2 x i32> <i32 1, i32 2>
+; AVX-NEXT:    [[TMP1:%.*]] = shufflevector <8 x float> [[A]], <8 x float> poison, <2 x i32> <i32 1, i32 2>
 ; AVX-NEXT:    [[TMP2:%.*]] = call <2 x float> @llvm.floor.v2f32(<2 x float> [[TMP1]])
 ; AVX-NEXT:    [[AB3:%.*]] = call float @llvm.ceil.f32(float [[A3]])
-; AVX-NEXT:    [[TMP3:%.*]] = shufflevector <8 x float> [[A]], <8 x float> undef, <2 x i32> <i32 4, i32 5>
+; AVX-NEXT:    [[TMP3:%.*]] = shufflevector <8 x float> [[A]], <8 x float> poison, <2 x i32> <i32 4, i32 5>
 ; AVX-NEXT:    [[TMP4:%.*]] = call <2 x float> @llvm.ceil.v2f32(<2 x float> [[TMP3]])
-; AVX-NEXT:    [[TMP5:%.*]] = shufflevector <8 x float> [[A]], <8 x float> undef, <2 x i32> <i32 6, i32 7>
+; AVX-NEXT:    [[TMP5:%.*]] = shufflevector <8 x float> [[A]], <8 x float> poison, <2 x i32> <i32 6, i32 7>
 ; AVX-NEXT:    [[TMP6:%.*]] = call <2 x float> @llvm.floor.v2f32(<2 x float> [[TMP5]])
 ; AVX-NEXT:    [[R0:%.*]] = insertelement <8 x float> poison, float [[AB0]], i64 0
 ; AVX-NEXT:    [[TMP7:%.*]] = shufflevector <2 x float> [[TMP2]], <2 x float> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/alternate-calls.ll b/llvm/test/Transforms/SLPVectorizer/X86/alternate-calls.ll
index b4d525fc4db5a..0f7d3b5647ff9 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/alternate-calls.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/alternate-calls.ll
@@ -11,12 +11,12 @@ define <8 x float> @ceil_floor(<8 x float> %a) {
 ; SSE-NEXT:    [[A0:%.*]] = extractelement <8 x float> [[A:%.*]], i64 0
 ; SSE-NEXT:    [[A3:%.*]] = extractelement <8 x float> [[A]], i64 3
 ; SSE-NEXT:    [[AB0:%.*]] = call float @llvm.ceil.f32(float [[A0]])
-; SSE-NEXT:    [[TMP1:%.*]] = shufflevector <8 x float> [[A]], <8 x float> undef, <2 x i32> <i32 1, i32 2>
+; SSE-NEXT:    [[TMP1:%.*]] = shufflevector <8 x float> [[A]], <8 x float> poison, <2 x i32> <i32 1, i32 2>
 ; SSE-NEXT:    [[TMP2:%.*]] = call <2 x float> @llvm.floor.v2f32(<2 x float> [[TMP1]])
 ; SSE-NEXT:    [[AB3:%.*]] = call float @llvm.ceil.f32(float [[A3]])
-; SSE-NEXT:    [[TMP3:%.*]] = shufflevector <8 x float> [[A]], <8 x float> undef, <2 x i32> <i32 4, i32 5>
+; SSE-NEXT:    [[TMP3:%.*]] = shufflevector <8 x float> [[A]], <8 x float> poison, <2 x i32> <i32 4, i32 5>
 ; SSE-NEXT:    [[TMP4:%.*]] = call <2 x float> @llvm.ceil.v2f32(<2 x float> [[TMP3]])
-; SSE-NEXT:    [[TMP5:%.*]] = shufflevector <8 x float> [[A]], <8 x float> undef, <2 x i32> <i32 6, i32 7>
+; SSE-NEXT:    [[TMP5:%.*]] = shufflevector <8 x float> [[A]], <8 x float> poison, <2 x i32> <i32 6, i32 7>
 ; SSE-NEXT:    [[TMP6:%.*]] = call <2 x float> @llvm.floor.v2f32(<2 x float> [[TMP5]])
 ; SSE-NEXT:    [[R0:%.*]] = insertelement <8 x float> undef, float [[AB0]], i64 0
 ; SSE-NEXT:    [[TMP7:%.*]] = shufflevector <2 x float> [[TMP2]], <2 x float> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
@@ -32,12 +32,12 @@ define <8 x float> @ceil_floor(<8 x float> %a) {
 ; SLM-NEXT:    [[A0:%.*]] = extractelement <8 x float> [[A:%.*]], i64 0
 ; SLM-NEXT:    [[A3:%.*]] = extractelement <8 x float> [[A]], i64 3
 ; SLM-NEXT:    [[AB0:%.*]] = call float @llvm.ceil.f32(float [[A0]])
-; SLM-NEXT:    [[TMP1:%.*]] = shufflevector <8 x float> [[A]], <8 x float> undef, <2 x i32> <i32 1, i32 2>
+; SLM-NEXT:    [[TMP1:%.*]] = shufflevector <8 x float> [[A]], <8 x float> poison, <2 x i32> <i32 1, i32 2>
 ; SLM-NEXT:    [[TMP2:%.*]] = call <2 x float> @llvm.floor.v2f32(<2 x float> [[TMP1]])
 ; SLM-NEXT:    [[AB3:%.*]] = call float @llvm.ceil.f32(float [[A3]])
-; SLM-NEXT:    [[TMP3:%.*]] = shufflevector <8 x float> [[A]], <8 x float> undef, <2 x i32> <i32 4, i32 5>
+; SLM-NEXT:    [[TMP3:%.*]] = shufflevector <8 x float> [[A]], <8 x float> poison, <2 x i32> <i32 4, i32 5>
 ; SLM-NEXT:    [[TMP4:%.*]] = call <2 x float> @llvm.ceil.v2f32(<2 x float> [[TMP3]])
-; SLM-NEXT:    [[TMP5:%.*]] = shufflevector <8 x float> [[A]], <8 x float> undef, <2 x i32> <i32 6, i32 7>
+; SLM-NEXT:    [[TMP5:%.*]] = shufflevector <8 x float> [[A]], <8 x float> poison, <2 x i32> <i32 6, i32 7>
 ; SLM-NEXT:    [[TMP6:%.*]] = call <2 x float> @llvm.floor.v2f32(<2 x float> [[TMP5]])
 ; SLM-NEXT:    [[R0:%.*]] = insertelement <8 x float> undef, float [[AB0]], i64 0
 ; SLM-NEXT:    [[TMP7:%.*]] = shufflevector <2 x float> [[TMP2]], <2 x float> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
@@ -53,12 +53,12 @@ define <8 x float> @ceil_floor(<8 x float> %a) {
 ; AVX-NEXT:    [[A0:%.*]] = extractelement <8 x float> [[A:%.*]], i64 0
 ; AVX-NEXT:    [[A3:%.*]] = extractelement <8 x float> [[A]], i64 3
 ; AVX-NEXT:    [[AB0:%.*]] = call float @llvm.ceil.f32(float [[A0]])
-; AVX-NEXT:    [[TMP1:%.*]] = shufflevector <8 x float> [[A]], <8 x float> undef, <2 x i32> <i32 1, i32 2>
+; AVX-NEXT:    [[TMP1:%.*]] = shufflevector <8 x float> [[A]], <8 x float> poison, <2 x i32> <i32 1, i32 2>
 ; AVX-NEXT:    [[TMP2:%.*]] = call <2 x float> @llvm.floor.v2f32(<2 x float> [[TMP1]])
 ; AVX-NEXT:    [[AB3:%.*]] = call float @llvm.ceil.f32(float [[A3]])
-; AVX-NEXT:    [[TMP3:%.*]] = shufflevector <8 x float> [[A]], <8 x float> undef, <2 x i32> <i32 4, i32 5>
+; AVX-NEXT:    [[TMP3:%.*]] = shufflevector <8 x float> [[A]], <8 x float> poison, <2 x i32> <i32 4, i32 5>
 ; AVX-NEXT:    [[TMP4:%.*]] = call <2 x float> @llvm.ceil.v2f32(<2 x float> [[TMP3]])
-; AVX-NEXT:    [[TMP5:%.*]] = shufflevector <8 x float> [[A]], <8 x float> undef, <2 x i32> <i32 6, i32 7>
+; AVX-NEXT:    [[TMP5:%.*]] = shufflevector <8 x float> [[A]], <8 x float> poison, <2 x i32> <i32 6, i32 7>
 ; AVX-NEXT:    [[TMP6:%.*]] = call <2 x float> @llvm.floor.v2f32(<2 x float> [[TMP5]])
 ; AVX-NEXT:    [[R0:%.*]] = insertelement <8 x float> undef, float [[AB0]], i64 0
 ; AVX-NEXT:    [[TMP7:%.*]] = shufflevector <2 x float> [[TMP2]], <2 x float> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/alternate-cast-inseltpoison.ll b/llvm/test/Transforms/SLPVectorizer/X86/alternate-cast-inseltpoison.ll
index 40678b9c87829..11eddced91a12 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/alternate-cast-inseltpoison.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/alternate-cast-inseltpoison.ll
@@ -163,7 +163,7 @@ define <8 x i32> @sext_zext(<8 x i16> %a) {
 define <8 x float> @sitofp_4i32_8i16(<4 x i32> %a, <8 x i16> %b) {
 ; CHECK-LABEL: @sitofp_4i32_8i16(
 ; CHECK-NEXT:    [[TMP1:%.*]] = sitofp <4 x i32> [[A:%.*]] to <4 x float>
-; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i16> [[B:%.*]], <8 x i16> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i16> [[B:%.*]], <8 x i16> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
 ; CHECK-NEXT:    [[TMP3:%.*]] = sitofp <4 x i16> [[TMP2]] to <4 x float>
 ; CHECK-NEXT:    [[R71:%.*]] = shufflevector <4 x float> [[TMP1]], <4 x float> [[TMP3]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
 ; CHECK-NEXT:    ret <8 x float> [[R71]]
@@ -201,11 +201,11 @@ define <8 x float> @sitofp_uitofp_4i32_8i16_16i8(<4 x i32> %a, <8 x i16> %b, <16
 ; CHECK-NEXT:    [[TMP1:%.*]] = sitofp <4 x i32> [[A:%.*]] to <4 x float>
 ; CHECK-NEXT:    [[TMP2:%.*]] = uitofp <4 x i32> [[A]] to <4 x float>
 ; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[TMP1]], <4 x float> [[TMP2]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
-; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <8 x i16> [[B:%.*]], <8 x i16> undef, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <8 x i16> [[B:%.*]], <8 x i16> poison, <2 x i32> <i32 0, i32 1>
 ; CHECK-NEXT:    [[TMP5:%.*]] = sitofp <2 x i16> [[TMP4]] to <2 x float>
 ; CHECK-NEXT:    [[TMP6:%.*]] = uitofp <2 x i16> [[TMP4]] to <2 x float>
 ; CHECK-NEXT:    [[TMP7:%.*]] = shufflevector <2 x float> [[TMP5]], <2 x float> [[TMP6]], <2 x i32> <i32 0, i32 3>
-; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <16 x i8> [[C:%.*]], <16 x i8> undef, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <16 x i8> [[C:%.*]], <16 x i8> poison, <2 x i32> <i32 0, i32 1>
 ; CHECK-NEXT:    [[TMP9:%.*]] = sitofp <2 x i8> [[TMP8]] to <2 x float>
 ; CHECK-NEXT:    [[TMP10:%.*]] = uitofp <2 x i8> [[TMP8]] to <2 x float>
 ; CHECK-NEXT:    [[TMP11:%.*]] = shufflevector <2 x float> [[TMP9]], <2 x float> [[TMP10]], <2 x i32> <i32 0, i32 3>
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/alternate-cast.ll b/llvm/test/Transforms/SLPVectorizer/X86/alternate-cast.ll
index b09c142cca50c..59a77598b6640 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/alternate-cast.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/alternate-cast.ll
@@ -163,7 +163,7 @@ define <8 x i32> @sext_zext(<8 x i16> %a) {
 define <8 x float> @sitofp_4i32_8i16(<4 x i32> %a, <8 x i16> %b) {
 ; CHECK-LABEL: @sitofp_4i32_8i16(
 ; CHECK-NEXT:    [[TMP1:%.*]] = sitofp <4 x i32> [[A:%.*]] to <4 x float>
-; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i16> [[B:%.*]], <8 x i16> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i16> [[B:%.*]], <8 x i16> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
 ; CHECK-NEXT:    [[TMP3:%.*]] = sitofp <4 x i16> [[TMP2]] to <4 x float>
 ; CHECK-NEXT:    [[R71:%.*]] = shufflevector <4 x float> [[TMP1]], <4 x float> [[TMP3]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
 ; CHECK-NEXT:    ret <8 x float> [[R71]]
@@ -201,11 +201,11 @@ define <8 x float> @sitofp_uitofp_4i32_8i16_16i8(<4 x i32> %a, <8 x i16> %b, <16
 ; CHECK-NEXT:    [[TMP1:%.*]] = sitofp <4 x i32> [[A:%.*]] to <4 x float>
 ; CHECK-NEXT:    [[TMP2:%.*]] = uitofp <4 x i32> [[A]] to <4 x float>
 ; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[TMP1]], <4 x float> [[TMP2]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
-; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <8 x i16> [[B:%.*]], <8 x i16> undef, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <8 x i16> [[B:%.*]], <8 x i16> poison, <2 x i32> <i32 0, i32 1>
 ; CHECK-NEXT:    [[TMP5:%.*]] = sitofp <2 x i16> [[TMP4]] to <2 x float>
 ; CHECK-NEXT:    [[TMP6:%.*]] = uitofp <2 x i16> [[TMP4]] to <2 x float>
 ; CHECK-NEXT:    [[TMP7:%.*]] = shufflevector <2 x float> [[TMP5]], <2 x float> [[TMP6]], <2 x i32> <i32 0, i32 3>
-; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <16 x i8> [[C:%.*]], <16 x i8> undef, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <16 x i8> [[C:%.*]], <16 x i8> poison, <2 x i32> <i32 0, i32 1>
 ; CHECK-NEXT:    [[TMP9:%.*]] = sitofp <2 x i8> [[TMP8]] to <2 x float>
 ; CHECK-NEXT:    [[TMP10:%.*]] = uitofp <2 x i8> [[TMP8]] to <2 x float>
 ; CHECK-NEXT:    [[TMP11:%.*]] = shufflevector <2 x float> [[TMP9]], <2 x float> [[TMP10]], <2 x i32> <i32 0, i32 3>
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/alternate-fp-inseltpoison.ll b/llvm/test/Transforms/SLPVectorizer/X86/alternate-fp-inseltpoison.ll
index 2dcaec8d11023..0f59f595f911e 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/alternate-fp-inseltpoison.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/alternate-fp-inseltpoison.ll
@@ -98,7 +98,7 @@ define <4 x float> @fmul_fdiv_v4f32_const(<4 x float> %a) {
 ; SLM-LABEL: @fmul_fdiv_v4f32_const(
 ; SLM-NEXT:    [[A2:%.*]] = extractelement <4 x float> [[A:%.*]], i64 2
 ; SLM-NEXT:    [[A3:%.*]] = extractelement <4 x float> [[A]], i64 3
-; SLM-NEXT:    [[TMP1:%.*]] = shufflevector <4 x float> [[A]], <4 x float> undef, <2 x i32> <i32 0, i32 1>
+; SLM-NEXT:    [[TMP1:%.*]] = shufflevector <4 x float> [[A]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
 ; SLM-NEXT:    [[TMP2:%.*]] = fmul <2 x float> [[TMP1]], <float 2.000000e+00, float 1.000000e+00>
 ; SLM-NEXT:    [[AB3:%.*]] = fmul float [[A3]], 2.000000e+00
 ; SLM-NEXT:    [[TMP3:%.*]] = shufflevector <2 x float> [[TMP2]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/alternate-fp.ll b/llvm/test/Transforms/SLPVectorizer/X86/alternate-fp.ll
index 99e673ee52f54..ef13f7956e16d 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/alternate-fp.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/alternate-fp.ll
@@ -98,7 +98,7 @@ define <4 x float> @fmul_fdiv_v4f32_const(<4 x float> %a) {
 ; SLM-LABEL: @fmul_fdiv_v4f32_const(
 ; SLM-NEXT:    [[A2:%.*]] = extractelement <4 x float> [[A:%.*]], i64 2
 ; SLM-NEXT:    [[A3:%.*]] = extractelement <4 x float> [[A]], i64 3
-; SLM-NEXT:    [[TMP1:%.*]] = shufflevector <4 x float> [[A]], <4 x float> undef, <2 x i32> <i32 0, i32 1>
+; SLM-NEXT:    [[TMP1:%.*]] = shufflevector <4 x float> [[A]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
 ; SLM-NEXT:    [[TMP2:%.*]] = fmul <2 x float> [[TMP1]], <float 2.000000e+00, float 1.000000e+00>
 ; SLM-NEXT:    [[AB3:%.*]] = fmul float [[A3]], 2.000000e+00
 ; SLM-NEXT:    [[TMP3:%.*]] = shufflevector <2 x float> [[TMP2]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/alternate-int-inseltpoison.ll b/llvm/test/Transforms/SLPVectorizer/X86/alternate-int-inseltpoison.ll
index 19e7ed8d52117..9cf0e1184c4fe 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/alternate-int-inseltpoison.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/alternate-int-inseltpoison.ll
@@ -170,17 +170,17 @@ define <8 x i32> @ashr_shl_v8i32(<8 x i32> %a, <8 x i32> %b) {
 
 define <8 x i32> @ashr_shl_v8i32_const(<8 x i32> %a) {
 ; SSE-LABEL: @ashr_shl_v8i32_const(
-; SSE-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; SSE-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
 ; SSE-NEXT:    [[TMP2:%.*]] = ashr <4 x i32> [[TMP1]], <i32 2, i32 2, i32 2, i32 2>
-; SSE-NEXT:    [[TMP3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; SSE-NEXT:    [[TMP3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
 ; SSE-NEXT:    [[TMP4:%.*]] = shl <4 x i32> [[TMP3]], <i32 3, i32 3, i32 3, i32 3>
 ; SSE-NEXT:    [[R71:%.*]] = shufflevector <4 x i32> [[TMP2]], <4 x i32> [[TMP4]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
 ; SSE-NEXT:    ret <8 x i32> [[R71]]
 ;
 ; SLM-LABEL: @ashr_shl_v8i32_const(
-; SLM-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; SLM-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
 ; SLM-NEXT:    [[TMP2:%.*]] = ashr <4 x i32> [[TMP1]], <i32 2, i32 2, i32 2, i32 2>
-; SLM-NEXT:    [[TMP3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; SLM-NEXT:    [[TMP3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
 ; SLM-NEXT:    [[TMP4:%.*]] = shl <4 x i32> [[TMP3]], <i32 3, i32 3, i32 3, i32 3>
 ; SLM-NEXT:    [[R71:%.*]] = shufflevector <4 x i32> [[TMP2]], <4 x i32> [[TMP4]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
 ; SLM-NEXT:    ret <8 x i32> [[R71]]
@@ -236,8 +236,8 @@ define <8 x i32> @ashr_lshr_shl_v8i32(<8 x i32> %a, <8 x i32> %b) {
 ; SSE-NEXT:    [[A7:%.*]] = extractelement <8 x i32> [[A]], i64 7
 ; SSE-NEXT:    [[B6:%.*]] = extractelement <8 x i32> [[B:%.*]], i64 6
 ; SSE-NEXT:    [[B7:%.*]] = extractelement <8 x i32> [[B]], i64 7
-; SSE-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
-; SSE-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; SSE-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; SSE-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
 ; SSE-NEXT:    [[TMP3:%.*]] = ashr <4 x i32> [[TMP1]], [[TMP2]]
 ; SSE-NEXT:    [[TMP4:%.*]] = lshr <4 x i32> [[TMP1]], [[TMP2]]
 ; SSE-NEXT:    [[TMP5:%.*]] = shufflevector <4 x i32> [[TMP3]], <4 x i32> [[TMP4]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
@@ -253,13 +253,13 @@ define <8 x i32> @ashr_lshr_shl_v8i32(<8 x i32> %a, <8 x i32> %b) {
 ; SSE-NEXT:    ret <8 x i32> [[R7]]
 ;
 ; SLM-LABEL: @ashr_lshr_shl_v8i32(
-; SLM-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
-; SLM-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; SLM-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; SLM-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
 ; SLM-NEXT:    [[TMP3:%.*]] = ashr <4 x i32> [[TMP1]], [[TMP2]]
 ; SLM-NEXT:    [[TMP4:%.*]] = lshr <4 x i32> [[TMP1]], [[TMP2]]
 ; SLM-NEXT:    [[TMP5:%.*]] = shufflevector <4 x i32> [[TMP3]], <4 x i32> [[TMP4]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
-; SLM-NEXT:    [[TMP6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
-; SLM-NEXT:    [[TMP7:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; SLM-NEXT:    [[TMP6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; SLM-NEXT:    [[TMP7:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
 ; SLM-NEXT:    [[TMP8:%.*]] = lshr <4 x i32> [[TMP6]], [[TMP7]]
 ; SLM-NEXT:    [[TMP9:%.*]] = shl <4 x i32> [[TMP6]], [[TMP7]]
 ; SLM-NEXT:    [[TMP10:%.*]] = shufflevector <4 x i32> [[TMP8]], <4 x i32> [[TMP9]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
@@ -267,13 +267,13 @@ define <8 x i32> @ashr_lshr_shl_v8i32(<8 x i32> %a, <8 x i32> %b) {
 ; SLM-NEXT:    ret <8 x i32> [[R71]]
 ;
 ; AVX1-LABEL: @ashr_lshr_shl_v8i32(
-; AVX1-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
-; AVX1-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; AVX1-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; AVX1-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
 ; AVX1-NEXT:    [[TMP3:%.*]] = ashr <4 x i32> [[TMP1]], [[TMP2]]
 ; AVX1-NEXT:    [[TMP4:%.*]] = lshr <4 x i32> [[TMP1]], [[TMP2]]
 ; AVX1-NEXT:    [[TMP5:%.*]] = shufflevector <4 x i32> [[TMP3]], <4 x i32> [[TMP4]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
-; AVX1-NEXT:    [[TMP6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
-; AVX1-NEXT:    [[TMP7:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; AVX1-NEXT:    [[TMP6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; AVX1-NEXT:    [[TMP7:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
 ; AVX1-NEXT:    [[TMP8:%.*]] = lshr <4 x i32> [[TMP6]], [[TMP7]]
 ; AVX1-NEXT:    [[TMP9:%.*]] = shl <4 x i32> [[TMP6]], [[TMP7]]
 ; AVX1-NEXT:    [[TMP10:%.*]] = shufflevector <4 x i32> [[TMP8]], <4 x i32> [[TMP9]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
@@ -281,13 +281,13 @@ define <8 x i32> @ashr_lshr_shl_v8i32(<8 x i32> %a, <8 x i32> %b) {
 ; AVX1-NEXT:    ret <8 x i32> [[R71]]
 ;
 ; AVX2-LABEL: @ashr_lshr_shl_v8i32(
-; AVX2-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
-; AVX2-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; AVX2-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; AVX2-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
 ; AVX2-NEXT:    [[TMP3:%.*]] = ashr <4 x i32> [[TMP1]], [[TMP2]]
 ; AVX2-NEXT:    [[TMP4:%.*]] = lshr <4 x i32> [[TMP1]], [[TMP2]]
 ; AVX2-NEXT:    [[TMP5:%.*]] = shufflevector <4 x i32> [[TMP3]], <4 x i32> [[TMP4]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
-; AVX2-NEXT:    [[TMP6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
-; AVX2-NEXT:    [[TMP7:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; AVX2-NEXT:    [[TMP6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; AVX2-NEXT:    [[TMP7:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
 ; AVX2-NEXT:    [[TMP8:%.*]] = lshr <4 x i32> [[TMP6]], [[TMP7]]
 ; AVX2-NEXT:    [[TMP9:%.*]] = shl <4 x i32> [[TMP6]], [[TMP7]]
 ; AVX2-NEXT:    [[TMP10:%.*]] = shufflevector <4 x i32> [[TMP8]], <4 x i32> [[TMP9]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
@@ -295,13 +295,13 @@ define <8 x i32> @ashr_lshr_shl_v8i32(<8 x i32> %a, <8 x i32> %b) {
 ; AVX2-NEXT:    ret <8 x i32> [[R71]]
 ;
 ; AVX512-LABEL: @ashr_lshr_shl_v8i32(
-; AVX512-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
-; AVX512-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; AVX512-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; AVX512-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
 ; AVX512-NEXT:    [[TMP3:%.*]] = ashr <4 x i32> [[TMP1]], [[TMP2]]
 ; AVX512-NEXT:    [[TMP4:%.*]] = lshr <4 x i32> [[TMP1]], [[TMP2]]
 ; AVX512-NEXT:    [[TMP5:%.*]] = shufflevector <4 x i32> [[TMP3]], <4 x i32> [[TMP4]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
-; AVX512-NEXT:    [[TMP6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
-; AVX512-NEXT:    [[TMP7:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; AVX512-NEXT:    [[TMP6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; AVX512-NEXT:    [[TMP7:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
 ; AVX512-NEXT:    [[TMP8:%.*]] = lshr <4 x i32> [[TMP6]], [[TMP7]]
 ; AVX512-NEXT:    [[TMP9:%.*]] = shl <4 x i32> [[TMP6]], [[TMP7]]
 ; AVX512-NEXT:    [[TMP10:%.*]] = shufflevector <4 x i32> [[TMP8]], <4 x i32> [[TMP9]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
@@ -443,10 +443,10 @@ define <8 x i32> @sdiv_v8i32_undefs(<8 x i32> %a) {
 ; AVX2-NEXT:    [[A1:%.*]] = extractelement <8 x i32> [[A:%.*]], i64 1
 ; AVX2-NEXT:    [[A5:%.*]] = extractelement <8 x i32> [[A]], i64 5
 ; AVX2-NEXT:    [[AB1:%.*]] = sdiv i32 [[A1]], 4
-; AVX2-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <2 x i32> <i32 2, i32 3>
+; AVX2-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <2 x i32> <i32 2, i32 3>
 ; AVX2-NEXT:    [[TMP2:%.*]] = sdiv <2 x i32> [[TMP1]], <i32 8, i32 16>
 ; AVX2-NEXT:    [[AB5:%.*]] = sdiv i32 [[A5]], 4
-; AVX2-NEXT:    [[TMP3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <2 x i32> <i32 6, i32 7>
+; AVX2-NEXT:    [[TMP3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <2 x i32> <i32 6, i32 7>
 ; AVX2-NEXT:    [[TMP4:%.*]] = sdiv <2 x i32> [[TMP3]], <i32 8, i32 16>
 ; AVX2-NEXT:    [[R1:%.*]] = insertelement <8 x i32> poison, i32 [[AB1]], i64 1
 ; AVX2-NEXT:    [[TMP5:%.*]] = shufflevector <2 x i32> [[TMP2]], <2 x i32> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
@@ -460,10 +460,10 @@ define <8 x i32> @sdiv_v8i32_undefs(<8 x i32> %a) {
 ; AVX512-NEXT:    [[A1:%.*]] = extractelement <8 x i32> [[A:%.*]], i64 1
 ; AVX512-NEXT:    [[A5:%.*]] = extractelement <8 x i32> [[A]], i64 5
 ; AVX512-NEXT:    [[AB1:%.*]] = sdiv i32 [[A1]], 4
-; AVX512-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <2 x i32> <i32 2, i32 3>
+; AVX512-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <2 x i32> <i32 2, i32 3>
 ; AVX512-NEXT:    [[TMP2:%.*]] = sdiv <2 x i32> [[TMP1]], <i32 8, i32 16>
 ; AVX512-NEXT:    [[AB5:%.*]] = sdiv i32 [[A5]], 4
-; AVX512-NEXT:    [[TMP3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <2 x i32> <i32 6, i32 7>
+; AVX512-NEXT:    [[TMP3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <2 x i32> <i32 6, i32 7>
 ; AVX512-NEXT:    [[TMP4:%.*]] = sdiv <2 x i32> [[TMP3]], <i32 8, i32 16>
 ; AVX512-NEXT:    [[R1:%.*]] = insertelement <8 x i32> poison, i32 [[AB1]], i64 1
 ; AVX512-NEXT:    [[TMP5:%.*]] = shufflevector <2 x i32> [[TMP2]], <2 x i32> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
@@ -503,11 +503,11 @@ define <8 x i32> @sdiv_v8i32_undefs(<8 x i32> %a) {
 define <8 x i32> @add_sub_v8i32_splat(<8 x i32> %a, i32 %b) {
 ; CHECK-LABEL: @add_sub_v8i32_splat(
 ; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <8 x i32> poison, i32 [[B:%.*]], i64 0
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <8 x i32> [[TMP1]], <8 x i32> poison, <8 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP2:%.*]] = add <8 x i32> [[SHUFFLE]], [[A:%.*]]
-; CHECK-NEXT:    [[TMP3:%.*]] = sub <8 x i32> [[SHUFFLE]], [[A]]
-; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <8 x i32> [[TMP2]], <8 x i32> [[TMP3]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 12, i32 13, i32 14, i32 15>
-; CHECK-NEXT:    ret <8 x i32> [[TMP4]]
+; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[TMP1]], <8 x i32> poison, <8 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP3:%.*]] = add <8 x i32> [[TMP2]], [[A:%.*]]
+; CHECK-NEXT:    [[TMP4:%.*]] = sub <8 x i32> [[TMP2]], [[A]]
+; CHECK-NEXT:    [[TMP5:%.*]] = shufflevector <8 x i32> [[TMP3]], <8 x i32> [[TMP4]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 12, i32 13, i32 14, i32 15>
+; CHECK-NEXT:    ret <8 x i32> [[TMP5]]
 ;
   %a0 = extractelement <8 x i32> %a, i32 0
   %a1 = extractelement <8 x i32> %a, i32 1
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/alternate-int.ll b/llvm/test/Transforms/SLPVectorizer/X86/alternate-int.ll
index c07c4b794f150..976be7d7f1c7e 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/alternate-int.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/alternate-int.ll
@@ -170,17 +170,17 @@ define <8 x i32> @ashr_shl_v8i32(<8 x i32> %a, <8 x i32> %b) {
 
 define <8 x i32> @ashr_shl_v8i32_const(<8 x i32> %a) {
 ; SSE-LABEL: @ashr_shl_v8i32_const(
-; SSE-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; SSE-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
 ; SSE-NEXT:    [[TMP2:%.*]] = ashr <4 x i32> [[TMP1]], <i32 2, i32 2, i32 2, i32 2>
-; SSE-NEXT:    [[TMP3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; SSE-NEXT:    [[TMP3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
 ; SSE-NEXT:    [[TMP4:%.*]] = shl <4 x i32> [[TMP3]], <i32 3, i32 3, i32 3, i32 3>
 ; SSE-NEXT:    [[R71:%.*]] = shufflevector <4 x i32> [[TMP2]], <4 x i32> [[TMP4]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
 ; SSE-NEXT:    ret <8 x i32> [[R71]]
 ;
 ; SLM-LABEL: @ashr_shl_v8i32_const(
-; SLM-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; SLM-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
 ; SLM-NEXT:    [[TMP2:%.*]] = ashr <4 x i32> [[TMP1]], <i32 2, i32 2, i32 2, i32 2>
-; SLM-NEXT:    [[TMP3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; SLM-NEXT:    [[TMP3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
 ; SLM-NEXT:    [[TMP4:%.*]] = shl <4 x i32> [[TMP3]], <i32 3, i32 3, i32 3, i32 3>
 ; SLM-NEXT:    [[R71:%.*]] = shufflevector <4 x i32> [[TMP2]], <4 x i32> [[TMP4]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
 ; SLM-NEXT:    ret <8 x i32> [[R71]]
@@ -236,8 +236,8 @@ define <8 x i32> @ashr_lshr_shl_v8i32(<8 x i32> %a, <8 x i32> %b) {
 ; SSE-NEXT:    [[A7:%.*]] = extractelement <8 x i32> [[A]], i64 7
 ; SSE-NEXT:    [[B6:%.*]] = extractelement <8 x i32> [[B:%.*]], i64 6
 ; SSE-NEXT:    [[B7:%.*]] = extractelement <8 x i32> [[B]], i64 7
-; SSE-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
-; SSE-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; SSE-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; SSE-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
 ; SSE-NEXT:    [[TMP3:%.*]] = ashr <4 x i32> [[TMP1]], [[TMP2]]
 ; SSE-NEXT:    [[TMP4:%.*]] = lshr <4 x i32> [[TMP1]], [[TMP2]]
 ; SSE-NEXT:    [[TMP5:%.*]] = shufflevector <4 x i32> [[TMP3]], <4 x i32> [[TMP4]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
@@ -253,13 +253,13 @@ define <8 x i32> @ashr_lshr_shl_v8i32(<8 x i32> %a, <8 x i32> %b) {
 ; SSE-NEXT:    ret <8 x i32> [[R7]]
 ;
 ; SLM-LABEL: @ashr_lshr_shl_v8i32(
-; SLM-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
-; SLM-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; SLM-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; SLM-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
 ; SLM-NEXT:    [[TMP3:%.*]] = ashr <4 x i32> [[TMP1]], [[TMP2]]
 ; SLM-NEXT:    [[TMP4:%.*]] = lshr <4 x i32> [[TMP1]], [[TMP2]]
 ; SLM-NEXT:    [[TMP5:%.*]] = shufflevector <4 x i32> [[TMP3]], <4 x i32> [[TMP4]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
-; SLM-NEXT:    [[TMP6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
-; SLM-NEXT:    [[TMP7:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; SLM-NEXT:    [[TMP6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; SLM-NEXT:    [[TMP7:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
 ; SLM-NEXT:    [[TMP8:%.*]] = lshr <4 x i32> [[TMP6]], [[TMP7]]
 ; SLM-NEXT:    [[TMP9:%.*]] = shl <4 x i32> [[TMP6]], [[TMP7]]
 ; SLM-NEXT:    [[TMP10:%.*]] = shufflevector <4 x i32> [[TMP8]], <4 x i32> [[TMP9]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
@@ -267,13 +267,13 @@ define <8 x i32> @ashr_lshr_shl_v8i32(<8 x i32> %a, <8 x i32> %b) {
 ; SLM-NEXT:    ret <8 x i32> [[R71]]
 ;
 ; AVX1-LABEL: @ashr_lshr_shl_v8i32(
-; AVX1-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
-; AVX1-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; AVX1-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; AVX1-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
 ; AVX1-NEXT:    [[TMP3:%.*]] = ashr <4 x i32> [[TMP1]], [[TMP2]]
 ; AVX1-NEXT:    [[TMP4:%.*]] = lshr <4 x i32> [[TMP1]], [[TMP2]]
 ; AVX1-NEXT:    [[TMP5:%.*]] = shufflevector <4 x i32> [[TMP3]], <4 x i32> [[TMP4]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
-; AVX1-NEXT:    [[TMP6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
-; AVX1-NEXT:    [[TMP7:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; AVX1-NEXT:    [[TMP6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; AVX1-NEXT:    [[TMP7:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
 ; AVX1-NEXT:    [[TMP8:%.*]] = lshr <4 x i32> [[TMP6]], [[TMP7]]
 ; AVX1-NEXT:    [[TMP9:%.*]] = shl <4 x i32> [[TMP6]], [[TMP7]]
 ; AVX1-NEXT:    [[TMP10:%.*]] = shufflevector <4 x i32> [[TMP8]], <4 x i32> [[TMP9]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
@@ -281,13 +281,13 @@ define <8 x i32> @ashr_lshr_shl_v8i32(<8 x i32> %a, <8 x i32> %b) {
 ; AVX1-NEXT:    ret <8 x i32> [[R71]]
 ;
 ; AVX2-LABEL: @ashr_lshr_shl_v8i32(
-; AVX2-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
-; AVX2-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; AVX2-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; AVX2-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
 ; AVX2-NEXT:    [[TMP3:%.*]] = ashr <4 x i32> [[TMP1]], [[TMP2]]
 ; AVX2-NEXT:    [[TMP4:%.*]] = lshr <4 x i32> [[TMP1]], [[TMP2]]
 ; AVX2-NEXT:    [[TMP5:%.*]] = shufflevector <4 x i32> [[TMP3]], <4 x i32> [[TMP4]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
-; AVX2-NEXT:    [[TMP6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
-; AVX2-NEXT:    [[TMP7:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; AVX2-NEXT:    [[TMP6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; AVX2-NEXT:    [[TMP7:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
 ; AVX2-NEXT:    [[TMP8:%.*]] = lshr <4 x i32> [[TMP6]], [[TMP7]]
 ; AVX2-NEXT:    [[TMP9:%.*]] = shl <4 x i32> [[TMP6]], [[TMP7]]
 ; AVX2-NEXT:    [[TMP10:%.*]] = shufflevector <4 x i32> [[TMP8]], <4 x i32> [[TMP9]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
@@ -295,13 +295,13 @@ define <8 x i32> @ashr_lshr_shl_v8i32(<8 x i32> %a, <8 x i32> %b) {
 ; AVX2-NEXT:    ret <8 x i32> [[R71]]
 ;
 ; AVX512-LABEL: @ashr_lshr_shl_v8i32(
-; AVX512-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
-; AVX512-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B:%.*]], <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; AVX512-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; AVX512-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[B:%.*]], <8 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
 ; AVX512-NEXT:    [[TMP3:%.*]] = ashr <4 x i32> [[TMP1]], [[TMP2]]
 ; AVX512-NEXT:    [[TMP4:%.*]] = lshr <4 x i32> [[TMP1]], [[TMP2]]
 ; AVX512-NEXT:    [[TMP5:%.*]] = shufflevector <4 x i32> [[TMP3]], <4 x i32> [[TMP4]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
-; AVX512-NEXT:    [[TMP6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
-; AVX512-NEXT:    [[TMP7:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; AVX512-NEXT:    [[TMP6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; AVX512-NEXT:    [[TMP7:%.*]] = shufflevector <8 x i32> [[B]], <8 x i32> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
 ; AVX512-NEXT:    [[TMP8:%.*]] = lshr <4 x i32> [[TMP6]], [[TMP7]]
 ; AVX512-NEXT:    [[TMP9:%.*]] = shl <4 x i32> [[TMP6]], [[TMP7]]
 ; AVX512-NEXT:    [[TMP10:%.*]] = shufflevector <4 x i32> [[TMP8]], <4 x i32> [[TMP9]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
@@ -443,10 +443,10 @@ define <8 x i32> @sdiv_v8i32_undefs(<8 x i32> %a) {
 ; AVX2-NEXT:    [[A1:%.*]] = extractelement <8 x i32> [[A:%.*]], i64 1
 ; AVX2-NEXT:    [[A5:%.*]] = extractelement <8 x i32> [[A]], i64 5
 ; AVX2-NEXT:    [[AB1:%.*]] = sdiv i32 [[A1]], 4
-; AVX2-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <2 x i32> <i32 2, i32 3>
+; AVX2-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <2 x i32> <i32 2, i32 3>
 ; AVX2-NEXT:    [[TMP2:%.*]] = sdiv <2 x i32> [[TMP1]], <i32 8, i32 16>
 ; AVX2-NEXT:    [[AB5:%.*]] = sdiv i32 [[A5]], 4
-; AVX2-NEXT:    [[TMP3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <2 x i32> <i32 6, i32 7>
+; AVX2-NEXT:    [[TMP3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <2 x i32> <i32 6, i32 7>
 ; AVX2-NEXT:    [[TMP4:%.*]] = sdiv <2 x i32> [[TMP3]], <i32 8, i32 16>
 ; AVX2-NEXT:    [[R1:%.*]] = insertelement <8 x i32> <i32 poison, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>, i32 [[AB1]], i64 1
 ; AVX2-NEXT:    [[TMP5:%.*]] = shufflevector <2 x i32> [[TMP2]], <2 x i32> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
@@ -460,10 +460,10 @@ define <8 x i32> @sdiv_v8i32_undefs(<8 x i32> %a) {
 ; AVX512-NEXT:    [[A1:%.*]] = extractelement <8 x i32> [[A:%.*]], i64 1
 ; AVX512-NEXT:    [[A5:%.*]] = extractelement <8 x i32> [[A]], i64 5
 ; AVX512-NEXT:    [[AB1:%.*]] = sdiv i32 [[A1]], 4
-; AVX512-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <2 x i32> <i32 2, i32 3>
+; AVX512-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <2 x i32> <i32 2, i32 3>
 ; AVX512-NEXT:    [[TMP2:%.*]] = sdiv <2 x i32> [[TMP1]], <i32 8, i32 16>
 ; AVX512-NEXT:    [[AB5:%.*]] = sdiv i32 [[A5]], 4
-; AVX512-NEXT:    [[TMP3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> undef, <2 x i32> <i32 6, i32 7>
+; AVX512-NEXT:    [[TMP3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <2 x i32> <i32 6, i32 7>
 ; AVX512-NEXT:    [[TMP4:%.*]] = sdiv <2 x i32> [[TMP3]], <i32 8, i32 16>
 ; AVX512-NEXT:    [[R1:%.*]] = insertelement <8 x i32> <i32 poison, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>, i32 [[AB1]], i64 1
 ; AVX512-NEXT:    [[TMP5:%.*]] = shufflevector <2 x i32> [[TMP2]], <2 x i32> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
@@ -503,11 +503,11 @@ define <8 x i32> @sdiv_v8i32_undefs(<8 x i32> %a) {
 define <8 x i32> @add_sub_v8i32_splat(<8 x i32> %a, i32 %b) {
 ; CHECK-LABEL: @add_sub_v8i32_splat(
 ; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <8 x i32> poison, i32 [[B:%.*]], i64 0
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <8 x i32> [[TMP1]], <8 x i32> poison, <8 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP2:%.*]] = add <8 x i32> [[SHUFFLE]], [[A:%.*]]
-; CHECK-NEXT:    [[TMP3:%.*]] = sub <8 x i32> [[SHUFFLE]], [[A]]
-; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <8 x i32> [[TMP2]], <8 x i32> [[TMP3]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 12, i32 13, i32 14, i32 15>
-; CHECK-NEXT:    ret <8 x i32> [[TMP4]]
+; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[TMP1]], <8 x i32> poison, <8 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP3:%.*]] = add <8 x i32> [[TMP2]], [[A:%.*]]
+; CHECK-NEXT:    [[TMP4:%.*]] = sub <8 x i32> [[TMP2]], [[A]]
+; CHECK-NEXT:    [[TMP5:%.*]] = shufflevector <8 x i32> [[TMP3]], <8 x i32> [[TMP4]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 12, i32 13, i32 14, i32 15>
+; CHECK-NEXT:    ret <8 x i32> [[TMP5]]
 ;
   %a0 = extractelement <8 x i32> %a, i32 0
   %a1 = extractelement <8 x i32> %a, i32 1
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/arith-fp-inseltpoison.ll b/llvm/test/Transforms/SLPVectorizer/X86/arith-fp-inseltpoison.ll
index a1e47e5b31aa0..c556ea7491d48 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/arith-fp-inseltpoison.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/arith-fp-inseltpoison.ll
@@ -607,49 +607,25 @@ define <8 x double> @buildvector_div_8f64(<8 x double> %a, <8 x double> %b) {
 ; SSE-NEXT:    ret <8 x double> [[TMP1]]
 ;
 ; SLM-LABEL: @buildvector_div_8f64(
-; SLM-NEXT:    [[A0:%.*]] = extractelement <8 x double> [[A:%.*]], i32 0
-; SLM-NEXT:    [[A1:%.*]] = extractelement <8 x double> [[A]], i32 1
-; SLM-NEXT:    [[A2:%.*]] = extractelement <8 x double> [[A]], i32 2
-; SLM-NEXT:    [[A3:%.*]] = extractelement <8 x double> [[A]], i32 3
-; SLM-NEXT:    [[A4:%.*]] = extractelement <8 x double> [[A]], i32 4
-; SLM-NEXT:    [[A5:%.*]] = extractelement <8 x double> [[A]], i32 5
-; SLM-NEXT:    [[A6:%.*]] = extractelement <8 x double> [[A]], i32 6
-; SLM-NEXT:    [[A7:%.*]] = extractelement <8 x double> [[A]], i32 7
-; SLM-NEXT:    [[B0:%.*]] = extractelement <8 x double> [[B:%.*]], i32 0
-; SLM-NEXT:    [[B1:%.*]] = extractelement <8 x double> [[B]], i32 1
-; SLM-NEXT:    [[B2:%.*]] = extractelement <8 x double> [[B]], i32 2
-; SLM-NEXT:    [[B3:%.*]] = extractelement <8 x double> [[B]], i32 3
-; SLM-NEXT:    [[B4:%.*]] = extractelement <8 x double> [[B]], i32 4
-; SLM-NEXT:    [[B5:%.*]] = extractelement <8 x double> [[B]], i32 5
-; SLM-NEXT:    [[B6:%.*]] = extractelement <8 x double> [[B]], i32 6
-; SLM-NEXT:    [[B7:%.*]] = extractelement <8 x double> [[B]], i32 7
-; SLM-NEXT:    [[TMP1:%.*]] = insertelement <2 x double> poison, double [[A0]], i32 0
-; SLM-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP1]], double [[A1]], i32 1
-; SLM-NEXT:    [[TMP3:%.*]] = insertelement <2 x double> poison, double [[B0]], i32 0
-; SLM-NEXT:    [[TMP4:%.*]] = insertelement <2 x double> [[TMP3]], double [[B1]], i32 1
-; SLM-NEXT:    [[TMP5:%.*]] = fdiv <2 x double> [[TMP2]], [[TMP4]]
-; SLM-NEXT:    [[TMP6:%.*]] = insertelement <2 x double> poison, double [[A2]], i32 0
-; SLM-NEXT:    [[TMP7:%.*]] = insertelement <2 x double> [[TMP6]], double [[A3]], i32 1
-; SLM-NEXT:    [[TMP8:%.*]] = insertelement <2 x double> poison, double [[B2]], i32 0
-; SLM-NEXT:    [[TMP9:%.*]] = insertelement <2 x double> [[TMP8]], double [[B3]], i32 1
-; SLM-NEXT:    [[TMP10:%.*]] = fdiv <2 x double> [[TMP7]], [[TMP9]]
-; SLM-NEXT:    [[TMP11:%.*]] = insertelement <2 x double> poison, double [[A4]], i32 0
-; SLM-NEXT:    [[TMP12:%.*]] = insertelement <2 x double> [[TMP11]], double [[A5]], i32 1
-; SLM-NEXT:    [[TMP13:%.*]] = insertelement <2 x double> poison, double [[B4]], i32 0
-; SLM-NEXT:    [[TMP14:%.*]] = insertelement <2 x double> [[TMP13]], double [[B5]], i32 1
-; SLM-NEXT:    [[TMP15:%.*]] = fdiv <2 x double> [[TMP12]], [[TMP14]]
-; SLM-NEXT:    [[TMP16:%.*]] = insertelement <2 x double> poison, double [[A6]], i32 0
-; SLM-NEXT:    [[TMP17:%.*]] = insertelement <2 x double> [[TMP16]], double [[A7]], i32 1
-; SLM-NEXT:    [[TMP18:%.*]] = insertelement <2 x double> poison, double [[B6]], i32 0
-; SLM-NEXT:    [[TMP19:%.*]] = insertelement <2 x double> [[TMP18]], double [[B7]], i32 1
-; SLM-NEXT:    [[TMP20:%.*]] = fdiv <2 x double> [[TMP17]], [[TMP19]]
-; SLM-NEXT:    [[TMP21:%.*]] = shufflevector <2 x double> [[TMP5]], <2 x double> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-; SLM-NEXT:    [[TMP22:%.*]] = shufflevector <2 x double> [[TMP10]], <2 x double> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-; SLM-NEXT:    [[R31:%.*]] = shufflevector <8 x double> [[TMP21]], <8 x double> [[TMP22]], <8 x i32> <i32 0, i32 1, i32 8, i32 9, i32 4, i32 5, i32 6, i32 7>
-; SLM-NEXT:    [[TMP23:%.*]] = shufflevector <2 x double> [[TMP15]], <2 x double> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-; SLM-NEXT:    [[R52:%.*]] = shufflevector <8 x double> [[R31]], <8 x double> [[TMP23]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 8, i32 9, i32 6, i32 7>
-; SLM-NEXT:    [[TMP24:%.*]] = shufflevector <2 x double> [[TMP20]], <2 x double> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-; SLM-NEXT:    [[R73:%.*]] = shufflevector <8 x double> [[R52]], <8 x double> [[TMP24]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 8, i32 9>
+; SLM-NEXT:    [[TMP1:%.*]] = shufflevector <8 x double> [[A:%.*]], <8 x double> poison, <2 x i32> <i32 0, i32 1>
+; SLM-NEXT:    [[TMP2:%.*]] = shufflevector <8 x double> [[B:%.*]], <8 x double> poison, <2 x i32> <i32 0, i32 1>
+; SLM-NEXT:    [[TMP3:%.*]] = fdiv <2 x double> [[TMP1]], [[TMP2]]
+; SLM-NEXT:    [[TMP4:%.*]] = shufflevector <8 x double> [[A]], <8 x double> poison, <2 x i32> <i32 2, i32 3>
+; SLM-NEXT:    [[TMP5:%.*]] = shufflevector <8 x double> [[B]], <8 x double> poison, <2 x i32> <i32 2, i32 3>
+; SLM-NEXT:    [[TMP6:%.*]] = fdiv <2 x double> [[TMP4]], [[TMP5]]
+; SLM-NEXT:    [[TMP7:%.*]] = shufflevector <8 x double> [[A]], <8 x double> poison, <2 x i32> <i32 4, i32 5>
+; SLM-NEXT:    [[TMP8:%.*]] = shufflevector <8 x double> [[B]], <8 x double> poison, <2 x i32> <i32 4, i32 5>
+; SLM-NEXT:    [[TMP9:%.*]] = fdiv <2 x double> [[TMP7]], [[TMP8]]
+; SLM-NEXT:    [[TMP10:%.*]] = shufflevector <8 x double> [[A]], <8 x double> poison, <2 x i32> <i32 6, i32 7>
+; SLM-NEXT:    [[TMP11:%.*]] = shufflevector <8 x double> [[B]], <8 x double> poison, <2 x i32> <i32 6, i32 7>
+; SLM-NEXT:    [[TMP12:%.*]] = fdiv <2 x double> [[TMP10]], [[TMP11]]
+; SLM-NEXT:    [[TMP13:%.*]] = shufflevector <2 x double> [[TMP3]], <2 x double> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+; SLM-NEXT:    [[TMP14:%.*]] = shufflevector <2 x double> [[TMP6]], <2 x double> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+; SLM-NEXT:    [[R31:%.*]] = shufflevector <8 x double> [[TMP13]], <8 x double> [[TMP14]], <8 x i32> <i32 0, i32 1, i32 8, i32 9, i32 4, i32 5, i32 6, i32 7>
+; SLM-NEXT:    [[TMP15:%.*]] = shufflevector <2 x double> [[TMP9]], <2 x double> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+; SLM-NEXT:    [[R52:%.*]] = shufflevector <8 x double> [[R31]], <8 x double> [[TMP15]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 8, i32 9, i32 6, i32 7>
+; SLM-NEXT:    [[TMP16:%.*]] = shufflevector <2 x double> [[TMP12]], <2 x double> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+; SLM-NEXT:    [[R73:%.*]] = shufflevector <8 x double> [[R52]], <8 x double> [[TMP16]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 8, i32 9>
 ; SLM-NEXT:    ret <8 x double> [[R73]]
 ;
 ; AVX-LABEL: @buildvector_div_8f64(
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/arith-fp.ll b/llvm/test/Transforms/SLPVectorizer/X86/arith-fp.ll
index 8022b09f05c0b..f331735558b0a 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/arith-fp.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/arith-fp.ll
@@ -607,49 +607,25 @@ define <8 x double> @buildvector_div_8f64(<8 x double> %a, <8 x double> %b) {
 ; SSE-NEXT:    ret <8 x double> [[TMP1]]
 ;
 ; SLM-LABEL: @buildvector_div_8f64(
-; SLM-NEXT:    [[A0:%.*]] = extractelement <8 x double> [[A:%.*]], i32 0
-; SLM-NEXT:    [[A1:%.*]] = extractelement <8 x double> [[A]], i32 1
-; SLM-NEXT:    [[A2:%.*]] = extractelement <8 x double> [[A]], i32 2
-; SLM-NEXT:    [[A3:%.*]] = extractelement <8 x double> [[A]], i32 3
-; SLM-NEXT:    [[A4:%.*]] = extractelement <8 x double> [[A]], i32 4
-; SLM-NEXT:    [[A5:%.*]] = extractelement <8 x double> [[A]], i32 5
-; SLM-NEXT:    [[A6:%.*]] = extractelement <8 x double> [[A]], i32 6
-; SLM-NEXT:    [[A7:%.*]] = extractelement <8 x double> [[A]], i32 7
-; SLM-NEXT:    [[B0:%.*]] = extractelement <8 x double> [[B:%.*]], i32 0
-; SLM-NEXT:    [[B1:%.*]] = extractelement <8 x double> [[B]], i32 1
-; SLM-NEXT:    [[B2:%.*]] = extractelement <8 x double> [[B]], i32 2
-; SLM-NEXT:    [[B3:%.*]] = extractelement <8 x double> [[B]], i32 3
-; SLM-NEXT:    [[B4:%.*]] = extractelement <8 x double> [[B]], i32 4
-; SLM-NEXT:    [[B5:%.*]] = extractelement <8 x double> [[B]], i32 5
-; SLM-NEXT:    [[B6:%.*]] = extractelement <8 x double> [[B]], i32 6
-; SLM-NEXT:    [[B7:%.*]] = extractelement <8 x double> [[B]], i32 7
-; SLM-NEXT:    [[TMP1:%.*]] = insertelement <2 x double> poison, double [[A0]], i32 0
-; SLM-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP1]], double [[A1]], i32 1
-; SLM-NEXT:    [[TMP3:%.*]] = insertelement <2 x double> poison, double [[B0]], i32 0
-; SLM-NEXT:    [[TMP4:%.*]] = insertelement <2 x double> [[TMP3]], double [[B1]], i32 1
-; SLM-NEXT:    [[TMP5:%.*]] = fdiv <2 x double> [[TMP2]], [[TMP4]]
-; SLM-NEXT:    [[TMP6:%.*]] = insertelement <2 x double> poison, double [[A2]], i32 0
-; SLM-NEXT:    [[TMP7:%.*]] = insertelement <2 x double> [[TMP6]], double [[A3]], i32 1
-; SLM-NEXT:    [[TMP8:%.*]] = insertelement <2 x double> poison, double [[B2]], i32 0
-; SLM-NEXT:    [[TMP9:%.*]] = insertelement <2 x double> [[TMP8]], double [[B3]], i32 1
-; SLM-NEXT:    [[TMP10:%.*]] = fdiv <2 x double> [[TMP7]], [[TMP9]]
-; SLM-NEXT:    [[TMP11:%.*]] = insertelement <2 x double> poison, double [[A4]], i32 0
-; SLM-NEXT:    [[TMP12:%.*]] = insertelement <2 x double> [[TMP11]], double [[A5]], i32 1
-; SLM-NEXT:    [[TMP13:%.*]] = insertelement <2 x double> poison, double [[B4]], i32 0
-; SLM-NEXT:    [[TMP14:%.*]] = insertelement <2 x double> [[TMP13]], double [[B5]], i32 1
-; SLM-NEXT:    [[TMP15:%.*]] = fdiv <2 x double> [[TMP12]], [[TMP14]]
-; SLM-NEXT:    [[TMP16:%.*]] = insertelement <2 x double> poison, double [[A6]], i32 0
-; SLM-NEXT:    [[TMP17:%.*]] = insertelement <2 x double> [[TMP16]], double [[A7]], i32 1
-; SLM-NEXT:    [[TMP18:%.*]] = insertelement <2 x double> poison, double [[B6]], i32 0
-; SLM-NEXT:    [[TMP19:%.*]] = insertelement <2 x double> [[TMP18]], double [[B7]], i32 1
-; SLM-NEXT:    [[TMP20:%.*]] = fdiv <2 x double> [[TMP17]], [[TMP19]]
-; SLM-NEXT:    [[TMP21:%.*]] = shufflevector <2 x double> [[TMP5]], <2 x double> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-; SLM-NEXT:    [[TMP22:%.*]] = shufflevector <2 x double> [[TMP10]], <2 x double> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-; SLM-NEXT:    [[R31:%.*]] = shufflevector <8 x double> [[TMP21]], <8 x double> [[TMP22]], <8 x i32> <i32 0, i32 1, i32 8, i32 9, i32 4, i32 5, i32 6, i32 7>
-; SLM-NEXT:    [[TMP23:%.*]] = shufflevector <2 x double> [[TMP15]], <2 x double> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-; SLM-NEXT:    [[R52:%.*]] = shufflevector <8 x double> [[R31]], <8 x double> [[TMP23]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 8, i32 9, i32 6, i32 7>
-; SLM-NEXT:    [[TMP24:%.*]] = shufflevector <2 x double> [[TMP20]], <2 x double> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-; SLM-NEXT:    [[R73:%.*]] = shufflevector <8 x double> [[R52]], <8 x double> [[TMP24]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 8, i32 9>
+; SLM-NEXT:    [[TMP1:%.*]] = shufflevector <8 x double> [[A:%.*]], <8 x double> poison, <2 x i32> <i32 0, i32 1>
+; SLM-NEXT:    [[TMP2:%.*]] = shufflevector <8 x double> [[B:%.*]], <8 x double> poison, <2 x i32> <i32 0, i32 1>
+; SLM-NEXT:    [[TMP3:%.*]] = fdiv <2 x double> [[TMP1]], [[TMP2]]
+; SLM-NEXT:    [[TMP4:%.*]] = shufflevector <8 x double> [[A]], <8 x double> poison, <2 x i32> <i32 2, i32 3>
+; SLM-NEXT:    [[TMP5:%.*]] = shufflevector <8 x double> [[B]], <8 x double> poison, <2 x i32> <i32 2, i32 3>
+; SLM-NEXT:    [[TMP6:%.*]] = fdiv <2 x double> [[TMP4]], [[TMP5]]
+; SLM-NEXT:    [[TMP7:%.*]] = shufflevector <8 x double> [[A]], <8 x double> poison, <2 x i32> <i32 4, i32 5>
+; SLM-NEXT:    [[TMP8:%.*]] = shufflevector <8 x double> [[B]], <8 x double> poison, <2 x i32> <i32 4, i32 5>
+; SLM-NEXT:    [[TMP9:%.*]] = fdiv <2 x double> [[TMP7]], [[TMP8]]
+; SLM-NEXT:    [[TMP10:%.*]] = shufflevector <8 x double> [[A]], <8 x double> poison, <2 x i32> <i32 6, i32 7>
+; SLM-NEXT:    [[TMP11:%.*]] = shufflevector <8 x double> [[B]], <8 x double> poison, <2 x i32> <i32 6, i32 7>
+; SLM-NEXT:    [[TMP12:%.*]] = fdiv <2 x double> [[TMP10]], [[TMP11]]
+; SLM-NEXT:    [[TMP13:%.*]] = shufflevector <2 x double> [[TMP3]], <2 x double> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+; SLM-NEXT:    [[TMP14:%.*]] = shufflevector <2 x double> [[TMP6]], <2 x double> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+; SLM-NEXT:    [[R31:%.*]] = shufflevector <8 x double> [[TMP13]], <8 x double> [[TMP14]], <8 x i32> <i32 0, i32 1, i32 8, i32 9, i32 4, i32 5, i32 6, i32 7>
+; SLM-NEXT:    [[TMP15:%.*]] = shufflevector <2 x double> [[TMP9]], <2 x double> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+; SLM-NEXT:    [[R52:%.*]] = shufflevector <8 x double> [[R31]], <8 x double> [[TMP15]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 8, i32 9, i32 6, i32 7>
+; SLM-NEXT:    [[TMP16:%.*]] = shufflevector <2 x double> [[TMP12]], <2 x double> poison, <8 x i32> <i32 0, i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+; SLM-NEXT:    [[R73:%.*]] = shufflevector <8 x double> [[R52]], <8 x double> [[TMP16]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 8, i32 9>
 ; SLM-NEXT:    ret <8 x double> [[R73]]
 ;
 ; AVX-LABEL: @buildvector_div_8f64(
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/buildvector-nodes-dependency.ll b/llvm/test/Transforms/SLPVectorizer/X86/buildvector-nodes-dependency.ll
new file mode 100644
index 0000000000000..001da64c60a93
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/X86/buildvector-nodes-dependency.ll
@@ -0,0 +1,76 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
+; RUN: opt -passes=slp-vectorizer -S -mtriple=x86_64 < %s | FileCheck %s
+
+define double @test() {
+; CHECK-LABEL: define double @test() {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load double, ptr null, align 8
+; CHECK-NEXT:    br label [[COND_TRUE:%.*]]
+; CHECK:       cond.true:
+; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x double> <double 0.000000e+00, double poison>, double [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = fmul <2 x double> zeroinitializer, [[TMP1]]
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <2 x double> [[TMP1]], <2 x double> poison, <2 x i32> <i32 1, i32 1>
+; CHECK-NEXT:    [[TMP4:%.*]] = fmul <2 x double> [[TMP3]], zeroinitializer
+; CHECK-NEXT:    [[TMP5:%.*]] = fmul <2 x double> [[TMP3]], zeroinitializer
+; CHECK-NEXT:    [[TMP6:%.*]] = shufflevector <2 x double> [[TMP2]], <2 x double> [[TMP1]], <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    [[TMP7:%.*]] = fmul <2 x double> [[TMP6]], zeroinitializer
+; CHECK-NEXT:    [[TMP8:%.*]] = fsub <2 x double> [[TMP7]], zeroinitializer
+; CHECK-NEXT:    [[TMP9:%.*]] = fmul <2 x double> [[TMP7]], zeroinitializer
+; CHECK-NEXT:    [[TMP10:%.*]] = shufflevector <2 x double> [[TMP8]], <2 x double> [[TMP9]], <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    [[TMP11:%.*]] = fadd <2 x double> zeroinitializer, [[TMP10]]
+; CHECK-NEXT:    [[TMP12:%.*]] = fmul <2 x double> zeroinitializer, [[TMP10]]
+; CHECK-NEXT:    [[TMP13:%.*]] = shufflevector <2 x double> [[TMP11]], <2 x double> [[TMP12]], <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    [[TMP14:%.*]] = fsub <2 x double> [[TMP13]], [[TMP2]]
+; CHECK-NEXT:    [[TMP15:%.*]] = fadd <2 x double> [[TMP13]], [[TMP2]]
+; CHECK-NEXT:    [[TMP16:%.*]] = shufflevector <2 x double> [[TMP14]], <2 x double> [[TMP15]], <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    [[TMP17:%.*]] = fsub <2 x double> [[TMP16]], zeroinitializer
+; CHECK-NEXT:    [[TMP18:%.*]] = fmul <2 x double> [[TMP4]], zeroinitializer
+; CHECK-NEXT:    [[TMP19:%.*]] = fmul <2 x double> zeroinitializer, [[TMP18]]
+; CHECK-NEXT:    [[TMP20:%.*]] = fadd <2 x double> [[TMP19]], [[TMP17]]
+; CHECK-NEXT:    [[TMP21:%.*]] = fsub <2 x double> [[TMP20]], zeroinitializer
+; CHECK-NEXT:    [[TMP22:%.*]] = fmul <2 x double> [[TMP5]], zeroinitializer
+; CHECK-NEXT:    [[TMP23:%.*]] = fmul <2 x double> zeroinitializer, [[TMP22]]
+; CHECK-NEXT:    [[TMP24:%.*]] = fadd <2 x double> [[TMP23]], [[TMP21]]
+; CHECK-NEXT:    [[TMP25:%.*]] = extractelement <2 x double> [[TMP24]], i32 0
+; CHECK-NEXT:    [[TMP26:%.*]] = extractelement <2 x double> [[TMP24]], i32 1
+; CHECK-NEXT:    [[ADD29:%.*]] = fadd double [[TMP25]], [[TMP26]]
+; CHECK-NEXT:    ret double [[ADD29]]
+;
+entry:
+  %0 = load double, ptr null, align 8
+  br label %cond.true
+
+cond.true:
+  %mul13 = fmul double %0, 0.000000e+00
+  %mul14 = fmul double %0, 0.000000e+00
+  %mul15 = fmul double %mul14, 0.000000e+00
+  %mul16 = fmul double 0.000000e+00, %mul15
+  %add17 = fadd double %mul13, %mul16
+  %sub18 = fsub double %add17, 0.000000e+00
+  %mul19 = fmul double %0, 0.000000e+00
+  %mul20 = fmul double %mul19, 0.000000e+00
+  %mul21 = fmul double %mul20, 0.000000e+00
+  %add22 = fadd double %sub18, %mul21
+  %sub23 = fsub double %add22, 0.000000e+00
+  %mul24 = fmul double %0, 0.000000e+00
+  %mul25 = fmul double %mul24, 0.000000e+00
+  %mul26 = fmul double 0.000000e+00, %mul25
+  %add27 = fadd double %mul26, %sub23
+  %mul = fmul double 0.000000e+00, 0.000000e+00
+  %mul1 = fmul double %mul, 0.000000e+00
+  %sub = fsub double %mul1, 0.000000e+00
+  %add = fadd double 0.000000e+00, %sub
+  %sub2 = fsub double %add, %mul
+  %sub3 = fsub double %sub2, 0.000000e+00
+  %mul4 = fmul double %0, 0.000000e+00
+  %mul5 = fmul double %mul4, 0.000000e+00
+  %mul6 = fmul double 0.000000e+00, %mul5
+  %add7 = fadd double %mul6, %sub3
+  %sub8 = fsub double %add7, 0.000000e+00
+  %mul9 = fmul double %0, 0.000000e+00
+  %mul10 = fmul double %mul9, 0.000000e+00
+  %mul11 = fmul double 0.000000e+00, %mul10
+  %add12 = fadd double %mul11, %sub8
+  %add29 = fadd double %add12, %add27
+  ret double %add29
+}
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/c-ray.ll b/llvm/test/Transforms/SLPVectorizer/X86/c-ray.ll
index 6b104f33e0c6a..b7f161b056fe2 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/c-ray.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/c-ray.ll
@@ -67,23 +67,23 @@ define i32 @ray_sphere(ptr nocapture noundef readonly %sph, ptr nocapture nounde
 ; CHECK-NEXT:    [[MUL88:%.*]] = fmul double [[TMP4]], 2.000000e+00
 ; CHECK-NEXT:    [[TMP26:%.*]] = insertelement <2 x double> poison, double [[FNEG87]], i32 0
 ; CHECK-NEXT:    [[TMP27:%.*]] = insertelement <2 x double> [[TMP26]], double [[CALL]], i32 1
-; CHECK-NEXT:    [[TMP28:%.*]] = insertelement <2 x double> poison, double [[CALL]], i32 0
+; CHECK-NEXT:    [[TMP28:%.*]] = shufflevector <2 x double> [[TMP27]], <2 x double> poison, <2 x i32> <i32 1, i32 undef>
 ; CHECK-NEXT:    [[TMP29:%.*]] = insertelement <2 x double> [[TMP28]], double [[TMP12]], i32 1
 ; CHECK-NEXT:    [[TMP30:%.*]] = fsub <2 x double> [[TMP27]], [[TMP29]]
 ; CHECK-NEXT:    [[TMP31:%.*]] = insertelement <2 x double> poison, double [[MUL88]], i32 0
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x double> [[TMP31]], <2 x double> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP32:%.*]] = fdiv <2 x double> [[TMP30]], [[SHUFFLE]]
-; CHECK-NEXT:    [[TMP33:%.*]] = extractelement <2 x double> [[TMP32]], i32 1
-; CHECK-NEXT:    [[CMP93:%.*]] = fcmp olt double [[TMP33]], 0x3EB0C6F7A0B5ED8D
-; CHECK-NEXT:    [[TMP34:%.*]] = extractelement <2 x double> [[TMP32]], i32 0
-; CHECK-NEXT:    [[CMP94:%.*]] = fcmp olt double [[TMP34]], 0x3EB0C6F7A0B5ED8D
+; CHECK-NEXT:    [[TMP32:%.*]] = shufflevector <2 x double> [[TMP31]], <2 x double> poison, <2 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP33:%.*]] = fdiv <2 x double> [[TMP30]], [[TMP32]]
+; CHECK-NEXT:    [[TMP34:%.*]] = extractelement <2 x double> [[TMP33]], i32 1
+; CHECK-NEXT:    [[CMP93:%.*]] = fcmp olt double [[TMP34]], 0x3EB0C6F7A0B5ED8D
+; CHECK-NEXT:    [[TMP35:%.*]] = extractelement <2 x double> [[TMP33]], i32 0
+; CHECK-NEXT:    [[CMP94:%.*]] = fcmp olt double [[TMP35]], 0x3EB0C6F7A0B5ED8D
 ; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[CMP93]], i1 [[CMP94]], i1 false
 ; CHECK-NEXT:    br i1 [[OR_COND]], label [[CLEANUP]], label [[LOR_LHS_FALSE:%.*]]
 ; CHECK:       lor.lhs.false:
-; CHECK-NEXT:    [[TMP35:%.*]] = fcmp ule <2 x double> [[TMP32]], <double 1.000000e+00, double 1.000000e+00>
-; CHECK-NEXT:    [[TMP36:%.*]] = extractelement <2 x i1> [[TMP35]], i32 0
-; CHECK-NEXT:    [[TMP37:%.*]] = extractelement <2 x i1> [[TMP35]], i32 1
-; CHECK-NEXT:    [[OR_COND106:%.*]] = select i1 [[TMP37]], i1 true, i1 [[TMP36]]
+; CHECK-NEXT:    [[TMP36:%.*]] = fcmp ule <2 x double> [[TMP33]], <double 1.000000e+00, double 1.000000e+00>
+; CHECK-NEXT:    [[TMP37:%.*]] = extractelement <2 x i1> [[TMP36]], i32 0
+; CHECK-NEXT:    [[TMP38:%.*]] = extractelement <2 x i1> [[TMP36]], i32 1
+; CHECK-NEXT:    [[OR_COND106:%.*]] = select i1 [[TMP38]], i1 true, i1 [[TMP37]]
 ; CHECK-NEXT:    [[SPEC_SELECT:%.*]] = zext i1 [[OR_COND106]] to i32
 ; CHECK-NEXT:    br label [[CLEANUP]]
 ; CHECK:       cleanup:
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/commutativity.ll b/llvm/test/Transforms/SLPVectorizer/X86/commutativity.ll
index 8f592783d8d76..4ebb78017db1c 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/commutativity.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/commutativity.ll
@@ -95,11 +95,10 @@ define void @same_opcode_on_one_side(i32 %a, i32 %b, i32 %c) {
 ; AVX-NEXT:    [[TMP3:%.*]] = insertelement <4 x i32> poison, i32 [[A:%.*]], i32 0
 ; AVX-NEXT:    [[TMP4:%.*]] = shufflevector <4 x i32> [[TMP3]], <4 x i32> poison, <4 x i32> zeroinitializer
 ; AVX-NEXT:    [[TMP5:%.*]] = add <4 x i32> [[TMP2]], [[TMP4]]
-; AVX-NEXT:    [[TMP6:%.*]] = insertelement <4 x i32> [[TMP3]], i32 [[B:%.*]], i32 1
-; AVX-NEXT:    [[TMP7:%.*]] = insertelement <4 x i32> [[TMP6]], i32 [[C]], i32 2
-; AVX-NEXT:    [[TMP8:%.*]] = shufflevector <4 x i32> [[TMP7]], <4 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 0>
-; AVX-NEXT:    [[TMP9:%.*]] = xor <4 x i32> [[TMP5]], [[TMP8]]
-; AVX-NEXT:    store <4 x i32> [[TMP9]], ptr @cle32, align 16
+; AVX-NEXT:    [[TMP6:%.*]] = shufflevector <4 x i32> [[TMP4]], <4 x i32> [[TMP2]], <4 x i32> <i32 0, i32 undef, i32 4, i32 0>
+; AVX-NEXT:    [[TMP7:%.*]] = insertelement <4 x i32> [[TMP6]], i32 [[B:%.*]], i32 1
+; AVX-NEXT:    [[TMP8:%.*]] = xor <4 x i32> [[TMP5]], [[TMP7]]
+; AVX-NEXT:    store <4 x i32> [[TMP8]], ptr @cle32, align 16
 ; AVX-NEXT:    ret void
 ;
   %add1 = add i32 %c, %a
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/crash_clear_undefs.ll b/llvm/test/Transforms/SLPVectorizer/X86/crash_clear_undefs.ll
index f3c9847f55b14..4eb63cbd44d5a 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/crash_clear_undefs.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/crash_clear_undefs.ll
@@ -5,23 +5,19 @@ target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16
 define i1 @foo() {
 ; CHECK-LABEL: @foo(
 ; CHECK-NEXT:    [[TMP1:%.*]] = load float, ptr null, align 4
-; CHECK-NEXT:    br i1 false, label [[TMP15:%.*]], label [[TMP2:%.*]]
+; CHECK-NEXT:    br i1 false, label [[TMP11:%.*]], label [[TMP2:%.*]]
 ; CHECK:       2:
-; CHECK-NEXT:    [[TMP3:%.*]] = extractelement <4 x float> zeroinitializer, i64 0
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> poison, float [[TMP1]], i32 0
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x float> [[TMP4]], float [[TMP3]], i32 1
-; CHECK-NEXT:    [[TMP6:%.*]] = fadd <2 x float> [[TMP5]], zeroinitializer
-; CHECK-NEXT:    [[TMP7:%.*]] = shufflevector <2 x float> [[TMP6]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 0, i32 1>
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <4 x float> poison, float [[TMP1]], i32 0
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <4 x float> [[TMP8]], float [[TMP3]], i32 1
-; CHECK-NEXT:    [[TMP10:%.*]] = shufflevector <4 x float> [[TMP9]], <4 x float> poison, <4 x i32> <i32 0, i32 1, i32 0, i32 1>
-; CHECK-NEXT:    [[TMP11:%.*]] = select <4 x i1> zeroinitializer, <4 x float> [[TMP7]], <4 x float> [[TMP10]]
-; CHECK-NEXT:    [[TMP12:%.*]] = fsub <4 x float> [[TMP11]], zeroinitializer
-; CHECK-NEXT:    [[TMP13:%.*]] = fadd <4 x float> [[TMP11]], zeroinitializer
-; CHECK-NEXT:    [[TMP14:%.*]] = shufflevector <4 x float> [[TMP12]], <4 x float> [[TMP13]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
-; CHECK-NEXT:    br label [[TMP15]]
-; CHECK:       15:
-; CHECK-NEXT:    [[TMP16:%.*]] = phi <4 x float> [ [[TMP14]], [[TMP2]] ], [ zeroinitializer, [[TMP0:%.*]] ]
+; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x float> <float undef, float 0.000000e+00>, float [[TMP1]], i32 0
+; CHECK-NEXT:    [[TMP4:%.*]] = fadd <2 x float> [[TMP3]], zeroinitializer
+; CHECK-NEXT:    [[TMP5:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 0, i32 1>
+; CHECK-NEXT:    [[TMP6:%.*]] = shufflevector <2 x float> [[TMP3]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 0, i32 1>
+; CHECK-NEXT:    [[TMP7:%.*]] = select <4 x i1> zeroinitializer, <4 x float> [[TMP5]], <4 x float> [[TMP6]]
+; CHECK-NEXT:    [[TMP8:%.*]] = fsub <4 x float> [[TMP7]], zeroinitializer
+; CHECK-NEXT:    [[TMP9:%.*]] = fadd <4 x float> [[TMP7]], zeroinitializer
+; CHECK-NEXT:    [[TMP10:%.*]] = shufflevector <4 x float> [[TMP8]], <4 x float> [[TMP9]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
+; CHECK-NEXT:    br label [[TMP11]]
+; CHECK:       11:
+; CHECK-NEXT:    [[TMP12:%.*]] = phi <4 x float> [ [[TMP10]], [[TMP2]] ], [ zeroinitializer, [[TMP0:%.*]] ]
 ; CHECK-NEXT:    ret i1 false
 ;
   %1 = load float, ptr null, align 4
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/crash_exceed_scheduling.ll b/llvm/test/Transforms/SLPVectorizer/X86/crash_exceed_scheduling.ll
index 65817d74ee54f..18d9f8c903c5b 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/crash_exceed_scheduling.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/crash_exceed_scheduling.ll
@@ -5,19 +5,19 @@ define void @exceed(double %0, double %1) {
 ; CHECK-LABEL: @exceed(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[TMP0:%.*]], i32 0
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x double> [[TMP2]], <2 x double> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x double> poison, double [[TMP1:%.*]], i32 0
-; CHECK-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <2 x double> [[TMP3]], <2 x double> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP4:%.*]] = fdiv fast <2 x double> [[SHUFFLE]], [[SHUFFLE1]]
-; CHECK-NEXT:    [[TMP5:%.*]] = extractelement <2 x double> [[TMP4]], i32 1
-; CHECK-NEXT:    [[IX:%.*]] = fmul double [[TMP5]], undef
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <2 x double> [[TMP2]], <2 x double> poison, <2 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x double> poison, double [[TMP1:%.*]], i32 0
+; CHECK-NEXT:    [[TMP5:%.*]] = shufflevector <2 x double> [[TMP4]], <2 x double> poison, <2 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP6:%.*]] = fdiv fast <2 x double> [[TMP3]], [[TMP5]]
+; CHECK-NEXT:    [[TMP7:%.*]] = extractelement <2 x double> [[TMP6]], i32 1
+; CHECK-NEXT:    [[IX:%.*]] = fmul double [[TMP7]], undef
 ; CHECK-NEXT:    [[IXX0:%.*]] = fsub double undef, undef
 ; CHECK-NEXT:    [[IXX1:%.*]] = fsub double undef, undef
 ; CHECK-NEXT:    [[IXX2:%.*]] = fsub double undef, undef
 ; CHECK-NEXT:    [[IXX3:%.*]] = fsub double undef, undef
 ; CHECK-NEXT:    [[IXX4:%.*]] = fsub double undef, undef
 ; CHECK-NEXT:    [[IXX5:%.*]] = fsub double undef, undef
-; CHECK-NEXT:    [[IX1:%.*]] = fmul double [[TMP5]], undef
+; CHECK-NEXT:    [[IX1:%.*]] = fmul double [[TMP7]], undef
 ; CHECK-NEXT:    [[IXX10:%.*]] = fsub double undef, undef
 ; CHECK-NEXT:    [[IXX11:%.*]] = fsub double undef, undef
 ; CHECK-NEXT:    [[IXX12:%.*]] = fsub double undef, undef
@@ -27,16 +27,15 @@ define void @exceed(double %0, double %1) {
 ; CHECK-NEXT:    [[IXX20:%.*]] = fsub double undef, undef
 ; CHECK-NEXT:    [[IXX21:%.*]] = fsub double undef, undef
 ; CHECK-NEXT:    [[IXX22:%.*]] = fsub double undef, undef
-; CHECK-NEXT:    [[TMP6:%.*]] = extractelement <2 x double> [[TMP4]], i32 0
-; CHECK-NEXT:    [[IX2:%.*]] = fmul double [[TMP6]], [[TMP6]]
-; CHECK-NEXT:    [[TMP7:%.*]] = fadd fast <2 x double> [[SHUFFLE]], [[SHUFFLE1]]
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <2 x double> [[TMP2]], double [[TMP1]], i32 1
-; CHECK-NEXT:    [[TMP9:%.*]] = fadd fast <2 x double> [[TMP4]], [[TMP8]]
-; CHECK-NEXT:    [[TMP10:%.*]] = fmul fast <2 x double> [[TMP9]], [[TMP7]]
+; CHECK-NEXT:    [[TMP8:%.*]] = extractelement <2 x double> [[TMP6]], i32 0
+; CHECK-NEXT:    [[IX2:%.*]] = fmul double [[TMP8]], [[TMP8]]
+; CHECK-NEXT:    [[TMP9:%.*]] = fadd fast <2 x double> [[TMP3]], [[TMP5]]
+; CHECK-NEXT:    [[TMP10:%.*]] = shufflevector <2 x double> [[TMP3]], <2 x double> [[TMP5]], <2 x i32> <i32 0, i32 2>
+; CHECK-NEXT:    [[TMP11:%.*]] = fadd fast <2 x double> [[TMP6]], [[TMP10]]
+; CHECK-NEXT:    [[TMP12:%.*]] = fmul fast <2 x double> [[TMP11]], [[TMP9]]
 ; CHECK-NEXT:    [[IXX101:%.*]] = fsub double undef, undef
-; CHECK-NEXT:    [[TMP11:%.*]] = insertelement <2 x double> poison, double [[TMP1]], i32 1
-; CHECK-NEXT:    [[TMP12:%.*]] = shufflevector <2 x double> [[TMP11]], <2 x double> [[TMP4]], <2 x i32> <i32 3, i32 1>
-; CHECK-NEXT:    [[TMP13:%.*]] = fmul fast <2 x double> [[TMP12]], undef
+; CHECK-NEXT:    [[TMP13:%.*]] = shufflevector <2 x double> [[TMP6]], <2 x double> [[TMP5]], <2 x i32> <i32 1, i32 2>
+; CHECK-NEXT:    [[TMP14:%.*]] = fmul fast <2 x double> [[TMP13]], undef
 ; CHECK-NEXT:    switch i32 undef, label [[BB1:%.*]] [
 ; CHECK-NEXT:    i32 0, label [[BB2:%.*]]
 ; CHECK-NEXT:    ]
@@ -45,7 +44,7 @@ define void @exceed(double %0, double %1) {
 ; CHECK:       bb2:
 ; CHECK-NEXT:    br label [[LABEL]]
 ; CHECK:       label:
-; CHECK-NEXT:    [[TMP14:%.*]] = phi <2 x double> [ [[TMP10]], [[BB1]] ], [ [[TMP13]], [[BB2]] ]
+; CHECK-NEXT:    [[TMP15:%.*]] = phi <2 x double> [ [[TMP12]], [[BB1]] ], [ [[TMP14]], [[BB2]] ]
 ; CHECK-NEXT:    ret void
 ;
 entry:
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/crash_netbsd_decompress.ll b/llvm/test/Transforms/SLPVectorizer/X86/crash_netbsd_decompress.ll
index f110f2e760393..12b227cbeb3d3 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/crash_netbsd_decompress.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/crash_netbsd_decompress.ll
@@ -23,7 +23,7 @@ define i32 @fn1() {
 ; CHECK-NEXT:    [[TMP2:%.*]] = load i32, ptr @c, align 4
 ; CHECK-NEXT:    [[AND:%.*]] = and i32 [[TMP2]], 7
 ; CHECK-NEXT:    store i32 [[AND]], ptr @a, align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <2 x i32> <i32 poison, i32 0>, <2 x i32> [[TMP0]], <2 x i32> <i32 2, i32 1>
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> <i32 poison, i32 0>, <2 x i32> <i32 0, i32 3>
 ; CHECK-NEXT:    switch i32 [[AND]], label [[IF_END:%.*]] [
 ; CHECK-NEXT:    i32 7, label [[SAVE_STATE_AND_RETURN]]
 ; CHECK-NEXT:    i32 0, label [[SAVE_STATE_AND_RETURN]]
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/crash_smallpt.ll b/llvm/test/Transforms/SLPVectorizer/X86/crash_smallpt.ll
index 5b7935ba02c32..837b4ffa68260 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/crash_smallpt.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/crash_smallpt.ll
@@ -1,14 +1,10 @@
 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -passes=slp-vectorizer,dce -S -mtriple=x86_64-apple-macosx10.8.0 -mcpu=corei7 | FileCheck %s
 
-target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
-target triple = "x86_64-apple-macosx10.8.0"
+%struct.Ray = type { %struct.Vec, %struct.Vec }
+%struct.Vec = type { double, double, double }
 
-%struct.Ray.5.11.53.113.119.137.149.185.329.389.416 = type { %struct.Vec.0.6.48.108.114.132.144.180.324.384.414, %struct.Vec.0.6.48.108.114.132.144.180.324.384.414 }
-%struct.Vec.0.6.48.108.114.132.144.180.324.384.414 = type { double, double, double }
-
-; Function Attrs: ssp uwtable
-define void @main() #0 {
+define void @main() {
 ; CHECK-LABEL: @main(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br i1 undef, label [[COND_TRUE:%.*]], label [[COND_END:%.*]]
@@ -19,7 +15,7 @@ define void @main() #0 {
 ; CHECK:       invoke.cont:
 ; CHECK-NEXT:    br i1 undef, label [[ARRAYCTOR_CONT:%.*]], label [[INVOKE_CONT]]
 ; CHECK:       arrayctor.cont:
-; CHECK-NEXT:    [[AGG_TMP101211_SROA_0_0_IDX:%.*]] = getelementptr inbounds [[STRUCT_RAY_5_11_53_113_119_137_149_185_329_389_416:%.*]], ptr undef, i64 0, i32 1, i32 0
+; CHECK-NEXT:    [[AGG_TMP101211_SROA_0_0_IDX:%.*]] = getelementptr inbounds [[STRUCT_RAY:%.*]], ptr undef, i64 0, i32 1, i32 0
 ; CHECK-NEXT:    br label [[FOR_COND36_PREHEADER:%.*]]
 ; CHECK:       for.cond36.preheader:
 ; CHECK-NEXT:    br i1 undef, label [[FOR_BODY42_LR_PH_US:%.*]], label [[_Z5CLAMPD_EXIT_1:%.*]]
@@ -28,18 +24,14 @@ define void @main() #0 {
 ; CHECK:       cond.true48.us:
 ; CHECK-NEXT:    br i1 undef, label [[COND_TRUE63_US:%.*]], label [[COND_FALSE66_US:%.*]]
 ; CHECK:       cond.false66.us:
-; CHECK-NEXT:    [[ADD_I276_US:%.*]] = fadd double 0.000000e+00, undef
+; CHECK-NEXT:    [[ADD_I276_US:%.*]] = fadd double 0.000000e+00, 0x3EB0C6F7A0B5ED8D
 ; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <2 x double> <double poison, double 0xBFA5CC2D1960285F>, double [[ADD_I276_US]], i32 0
-; CHECK-NEXT:    [[TMP1:%.*]] = fadd <2 x double> <double 0.000000e+00, double undef>, [[TMP0]]
+; CHECK-NEXT:    [[TMP1:%.*]] = fadd <2 x double> <double 0.000000e+00, double 1.000000e-01>, [[TMP0]]
 ; CHECK-NEXT:    [[TMP2:%.*]] = fmul <2 x double> [[TMP1]], <double 1.400000e+02, double 1.400000e+02>
 ; CHECK-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP2]], <double 5.000000e+01, double 5.200000e+01>
-; CHECK-NEXT:    [[TMP4:%.*]] = extractelement <2 x double> [[TMP1]], i32 0
-; CHECK-NEXT:    [[TMP5:%.*]] = extractelement <2 x double> [[TMP1]], i32 1
 ; CHECK-NEXT:    store <2 x double> [[TMP3]], ptr undef, align 8
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x double> <double poison, double undef>, double [[TMP4]], i32 0
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <2 x double> <double undef, double poison>, double [[TMP5]], i32 1
-; CHECK-NEXT:    [[TMP8:%.*]] = fmul <2 x double> [[TMP6]], [[TMP7]]
-; CHECK-NEXT:    store <2 x double> [[TMP8]], ptr [[AGG_TMP101211_SROA_0_0_IDX]], align 8
+; CHECK-NEXT:    [[TMP4:%.*]] = fmul <2 x double> <double 2.000000e-01, double 3.000000e-01>, [[TMP1]]
+; CHECK-NEXT:    store <2 x double> [[TMP4]], ptr [[AGG_TMP101211_SROA_0_0_IDX]], align 8
 ; CHECK-NEXT:    unreachable
 ; CHECK:       cond.true63.us:
 ; CHECK-NEXT:    unreachable
@@ -51,114 +43,96 @@ define void @main() #0 {
 entry:
   br i1 undef, label %cond.true, label %cond.end
 
-cond.true:                                        ; preds = %entry
+cond.true:
   unreachable
 
-cond.end:                                         ; preds = %entry
+cond.end:
   br label %invoke.cont
 
-invoke.cont:                                      ; preds = %invoke.cont, %cond.end
+invoke.cont:
   br i1 undef, label %arrayctor.cont, label %invoke.cont
 
-arrayctor.cont:                                   ; preds = %invoke.cont
-  %agg.tmp99208.sroa.1.8.idx388 = getelementptr inbounds %struct.Ray.5.11.53.113.119.137.149.185.329.389.416, ptr undef, i64 0, i32 0, i32 1
-  %agg.tmp101211.sroa.0.0.idx = getelementptr inbounds %struct.Ray.5.11.53.113.119.137.149.185.329.389.416, ptr undef, i64 0, i32 1, i32 0
-  %agg.tmp101211.sroa.1.8.idx390 = getelementptr inbounds %struct.Ray.5.11.53.113.119.137.149.185.329.389.416, ptr undef, i64 0, i32 1, i32 1
+arrayctor.cont:
+  %agg.tmp99208.sroa.1.8.idx388 = getelementptr inbounds %struct.Ray, ptr undef, i64 0, i32 0, i32 1
+  %agg.tmp101211.sroa.0.0.idx = getelementptr inbounds %struct.Ray, ptr undef, i64 0, i32 1, i32 0
+  %agg.tmp101211.sroa.1.8.idx390 = getelementptr inbounds %struct.Ray, ptr undef, i64 0, i32 1, i32 1
   br label %for.cond36.preheader
 
-for.cond36.preheader:                             ; preds = %_Z5clampd.exit.1, %arrayctor.cont
+for.cond36.preheader:
   br i1 undef, label %for.body42.lr.ph.us, label %_Z5clampd.exit.1
 
-cond.false51.us:                                  ; preds = %for.body42.lr.ph.us
+cond.false51.us:
   unreachable
 
-cond.true48.us:                                   ; preds = %for.body42.lr.ph.us
+cond.true48.us:
   br i1 undef, label %cond.true63.us, label %cond.false66.us
 
-cond.false66.us:                                  ; preds = %cond.true48.us
-  %add.i276.us = fadd double 0.000000e+00, undef
+cond.false66.us:
+  %add.i276.us = fadd double 0.000000e+00, 0.000001e+00
   %add.i264.us = fadd double %add.i276.us, 0.000000e+00
-  %add4.i267.us = fadd double undef, 0xBFA5CC2D1960285F
+  %add4.i267.us = fadd double 0.1e+00, 0xBFA5CC2D1960285F
   %mul.i254.us = fmul double %add.i264.us, 1.400000e+02
   %mul2.i256.us = fmul double %add4.i267.us, 1.400000e+02
   %add.i243.us = fadd double %mul.i254.us, 5.000000e+01
   %add4.i246.us = fadd double %mul2.i256.us, 5.200000e+01
-  %mul.i.i.us = fmul double undef, %add.i264.us
-  %mul2.i.i.us = fmul double undef, %add4.i267.us
+  %mul.i.i.us = fmul double 0.2e+00, %add.i264.us
+  %mul2.i.i.us = fmul double 0.3e+00, %add4.i267.us
   store double %add.i243.us, ptr undef, align 8
   store double %add4.i246.us, ptr %agg.tmp99208.sroa.1.8.idx388, align 8
   store double %mul.i.i.us, ptr %agg.tmp101211.sroa.0.0.idx, align 8
   store double %mul2.i.i.us, ptr %agg.tmp101211.sroa.1.8.idx390, align 8
   unreachable
 
-cond.true63.us:                                   ; preds = %cond.true48.us
+cond.true63.us:
   unreachable
 
-for.body42.lr.ph.us:                              ; preds = %for.cond36.preheader
+for.body42.lr.ph.us:
   br i1 undef, label %cond.true48.us, label %cond.false51.us
 
-_Z5clampd.exit.1:                                 ; preds = %for.cond36.preheader
+_Z5clampd.exit.1:
   br label %for.cond36.preheader
 }
 
-
-%struct.Ray.5.11.53.95.137.191.197.203.239.257.263.269.275.281.287.293.383.437.443.455.461.599.601 = type { %struct.Vec.0.6.48.90.132.186.192.198.234.252.258.264.270.276.282.288.378.432.438.450.456.594.600, %struct.Vec.0.6.48.90.132.186.192.198.234.252.258.264.270.276.282.288.378.432.438.450.456.594.600 }
-%struct.Vec.0.6.48.90.132.186.192.198.234.252.258.264.270.276.282.288.378.432.438.450.456.594.600 = type { double, double, double }
-
-define void @_Z8radianceRK3RayiPt() #0 {
-; CHECK-LABEL: @_Z8radianceRK3RayiPt(
+define void @test() {
+; CHECK-LABEL: @test(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br i1 undef, label [[IF_THEN78:%.*]], label [[IF_THEN38:%.*]]
 ; CHECK:       if.then38:
-; CHECK-NEXT:    [[AGG_TMP74663_SROA_0_0_IDX:%.*]] = getelementptr inbounds [[STRUCT_RAY_5_11_53_95_137_191_197_203_239_257_263_269_275_281_287_293_383_437_443_455_461_599_601:%.*]], ptr undef, i64 0, i32 1, i32 0
-; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <2 x double> <double undef, double poison>, double undef, i32 1
-; CHECK-NEXT:    [[TMP1:%.*]] = fmul <2 x double> undef, [[TMP0]]
-; CHECK-NEXT:    [[TMP2:%.*]] = fsub <2 x double> undef, [[TMP1]]
-; CHECK-NEXT:    [[TMP3:%.*]] = fmul <2 x double> undef, [[TMP2]]
-; CHECK-NEXT:    [[TMP4:%.*]] = fmul <2 x double> undef, [[TMP3]]
-; CHECK-NEXT:    [[TMP5:%.*]] = fadd <2 x double> undef, [[TMP4]]
-; CHECK-NEXT:    [[TMP6:%.*]] = fadd <2 x double> undef, [[TMP5]]
-; CHECK-NEXT:    [[TMP7:%.*]] = fmul <2 x double> undef, [[TMP6]]
-; CHECK-NEXT:    store <2 x double> [[TMP7]], ptr [[AGG_TMP74663_SROA_0_0_IDX]], align 8
-; CHECK-NEXT:    br label [[RETURN:%.*]]
+; CHECK-NEXT:    [[AGG_TMP74663_SROA_0_0_IDX:%.*]] = getelementptr inbounds [[STRUCT_RAY:%.*]], ptr undef, i64 0, i32 1, i32 0
+; CHECK-NEXT:    store <2 x double> <double 0x3FFA356C1D8A7F76, double 0x3FFDC4F38B38BEF4>, ptr [[AGG_TMP74663_SROA_0_0_IDX]], align 8
+; CHECK-NEXT:    br label [[IF_THEN78]]
 ; CHECK:       if.then78:
-; CHECK-NEXT:    br label [[RETURN]]
-; CHECK:       return:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   br i1 undef, label %if.then78, label %if.then38
 
-if.then38:                                        ; preds = %entry
-  %mul.i.i790 = fmul double undef, undef
-  %mul3.i.i792 = fmul double undef, undef
-  %mul.i764 = fmul double undef, %mul3.i.i792
-  %mul4.i767 = fmul double undef, undef
+if.then38:
+  %mul.i.i790 = fmul double 0.0, 0.1
+  %mul3.i.i792 = fmul double 0.2, 0.3
+  %mul.i764 = fmul double 0.4, %mul3.i.i792
+  %mul4.i767 = fmul double 0.5, 0.6
   %sub.i768 = fsub double %mul.i764, %mul4.i767
-  %mul6.i770 = fmul double undef, %mul.i.i790
-  %mul9.i772 = fmul double undef, %mul3.i.i792
+  %mul6.i770 = fmul double 0.7, %mul.i.i790
+  %mul9.i772 = fmul double 0.8, %mul3.i.i792
   %sub10.i773 = fsub double %mul6.i770, %mul9.i772
-  %mul.i736 = fmul double undef, %sub.i768
-  %mul2.i738 = fmul double undef, %sub10.i773
-  %mul.i727 = fmul double undef, %mul.i736
-  %mul2.i729 = fmul double undef, %mul2.i738
-  %add.i716 = fadd double undef, %mul.i727
-  %add4.i719 = fadd double undef, %mul2.i729
-  %add.i695 = fadd double undef, %add.i716
-  %add4.i698 = fadd double undef, %add4.i719
-  %mul.i.i679 = fmul double undef, %add.i695
-  %mul2.i.i680 = fmul double undef, %add4.i698
-  %agg.tmp74663.sroa.0.0.idx = getelementptr inbounds %struct.Ray.5.11.53.95.137.191.197.203.239.257.263.269.275.281.287.293.383.437.443.455.461.599.601, ptr undef, i64 0, i32 1, i32 0
+  %mul.i736 = fmul double 0.9, %sub.i768
+  %mul2.i738 = fmul double 0.91, %sub10.i773
+  %mul.i727 = fmul double 0.92, %mul.i736
+  %mul2.i729 = fmul double 0.93, %mul2.i738
+  %add.i716 = fadd double 0.94, %mul.i727
+  %add4.i719 = fadd double 0.95, %mul2.i729
+  %add.i695 = fadd double 0.96, %add.i716
+  %add4.i698 = fadd double 0.97, %add4.i719
+  %mul.i.i679 = fmul double 0.98, %add.i695
+  %mul2.i.i680 = fmul double 0.99, %add4.i698
+  %agg.tmp74663.sroa.0.0.idx = getelementptr inbounds %struct.Ray, ptr undef, i64 0, i32 1, i32 0
   store double %mul.i.i679, ptr %agg.tmp74663.sroa.0.0.idx, align 8
-  %agg.tmp74663.sroa.1.8.idx943 = getelementptr inbounds %struct.Ray.5.11.53.95.137.191.197.203.239.257.263.269.275.281.287.293.383.437.443.455.461.599.601, ptr undef, i64 0, i32 1, i32 1
+  %agg.tmp74663.sroa.1.8.idx943 = getelementptr inbounds %struct.Ray, ptr undef, i64 0, i32 1, i32 1
   store double %mul2.i.i680, ptr %agg.tmp74663.sroa.1.8.idx943, align 8
-  br label %return
-
-if.then78:                                        ; preds = %entry
-  br label %return
+  br label %if.then78
 
-return:                                           ; preds = %if.then78, %if.then38
+if.then78:
   ret void
 }
 
-attributes #0 = { ssp uwtable "less-precise-fpmad"="false" "frame-pointer"="none" "no-infs-fp-math"="false" "no-nans-fp-math"="false" "unsafe-fp-math"="false" "use-soft-float"="false" }
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/cse.ll b/llvm/test/Transforms/SLPVectorizer/X86/cse.ll
index a520ddd679dc8..b5602c795f42b 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/cse.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/cse.ll
@@ -19,14 +19,12 @@ define i32 @test(ptr nocapture %G) {
 ; CHECK-NEXT:    [[TMP1:%.*]] = fmul <2 x double> [[TMP0]], <double 4.000000e+00, double 3.000000e+00>
 ; CHECK-NEXT:    [[TMP2:%.*]] = fadd <2 x double> [[TMP1]], <double 1.000000e+00, double 6.000000e+00>
 ; CHECK-NEXT:    store <2 x double> [[TMP2]], ptr [[G]], align 8
-; CHECK-NEXT:    [[TMP3:%.*]] = extractelement <2 x double> [[TMP1]], i32 0
 ; CHECK-NEXT:    [[ARRAYIDX9:%.*]] = getelementptr inbounds double, ptr [[G]], i64 2
-; CHECK-NEXT:    [[TMP4:%.*]] = extractelement <2 x double> [[TMP0]], i32 1
-; CHECK-NEXT:    [[MUL11:%.*]] = fmul double [[TMP4]], 4.000000e+00
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x double> poison, double [[TMP3]], i32 0
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x double> [[TMP5]], double [[MUL11]], i32 1
-; CHECK-NEXT:    [[TMP7:%.*]] = fadd <2 x double> [[TMP6]], <double 7.000000e+00, double 8.000000e+00>
-; CHECK-NEXT:    store <2 x double> [[TMP7]], ptr [[ARRAYIDX9]], align 8
+; CHECK-NEXT:    [[TMP3:%.*]] = extractelement <2 x double> [[TMP0]], i32 1
+; CHECK-NEXT:    [[MUL11:%.*]] = fmul double [[TMP3]], 4.000000e+00
+; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x double> [[TMP1]], double [[MUL11]], i32 1
+; CHECK-NEXT:    [[TMP5:%.*]] = fadd <2 x double> [[TMP4]], <double 7.000000e+00, double 8.000000e+00>
+; CHECK-NEXT:    store <2 x double> [[TMP5]], ptr [[ARRAYIDX9]], align 8
 ; CHECK-NEXT:    ret i32 undef
 ;
 entry:
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/entries-different-vf.ll b/llvm/test/Transforms/SLPVectorizer/X86/entries-different-vf.ll
new file mode 100644
index 0000000000000..d9942887d6c7d
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/X86/entries-different-vf.ll
@@ -0,0 +1,50 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
+; RUN: opt --passes=slp-vectorizer -mtriple=x86_64-unknown-linux-gnu -mcpu=icelake-server -S < %s | FileCheck %s
+
+define i1 @test() {
+; CHECK-LABEL: define i1 @test
+; CHECK-SAME: () #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = shl i64 0, 0
+; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <8 x i64> <i64 poison, i64 poison, i64 0, i64 poison, i64 0, i64 0, i64 0, i64 0>, i64 [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <8 x i64> [[TMP1]], i64 0, i32 1
+; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <8 x i64> [[TMP2]], i64 0, i32 3
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <8 x i64> [[TMP3]], <8 x i64> poison, <4 x i32> <i32 3, i32 undef, i32 1, i32 0>
+; CHECK-NEXT:    [[TMP5:%.*]] = shufflevector <8 x i64> <i64 undef, i64 0, i64 undef, i64 undef, i64 undef, i64 undef, i64 undef, i64 undef>, <8 x i64> [[TMP3]], <8 x i32> <i32 11, i32 11, i32 11, i32 1, i32 9, i32 9, i32 1, i32 8>
+; CHECK-NEXT:    [[TMP6:%.*]] = or <8 x i64> [[TMP3]], [[TMP5]]
+; CHECK-NEXT:    [[TMP7:%.*]] = sub <8 x i64> [[TMP3]], [[TMP5]]
+; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <8 x i64> [[TMP6]], <8 x i64> [[TMP7]], <8 x i32> <i32 0, i32 1, i32 2, i32 11, i32 12, i32 5, i32 6, i32 7>
+; CHECK-NEXT:    [[TMP9:%.*]] = icmp ult <8 x i64> [[TMP8]], zeroinitializer
+; CHECK-NEXT:    [[TMP10:%.*]] = call i1 @llvm.vector.reduce.or.v8i1(<8 x i1> [[TMP9]])
+; CHECK-NEXT:    ret i1 [[TMP10]]
+;
+entry:
+  %0 = shl i64 0, 0
+  %1 = add i64 0, 0
+  %2 = add i64 0, 0
+  %3 = or i64 %2, %1
+  %cmp750 = icmp ult i64 %3, 0
+  %4 = or i64 %0, %1
+  %cmp752 = icmp ult i64 %4, 0
+  %or753 = or i1 %cmp750, %cmp752
+  %5 = or i64 0, %1
+  %cmp754 = icmp ult i64 %5, 0
+  %or755 = or i1 %or753, %cmp754
+  %6 = extractelement <16 x i64> zeroinitializer, i32 0
+  %7 = sub i64 %1, %6
+  %cmp756 = icmp ult i64 %7, 0
+  %or757 = or i1 %or755, %cmp756
+  %8 = sub i64 0, %2
+  %cmp758 = icmp ult i64 %8, 0
+  %or759 = or i1 %or757, %cmp758
+  %9 = or i64 0, %2
+  %cmp760 = icmp ult i64 %9, 0
+  %or761 = or i1 %or759, %cmp760
+  %10 = or i64 0, %6
+  %cmp762 = icmp ult i64 %10, 0
+  %or763 = or i1 %or761, %cmp762
+  %11 = or i64 0, %0
+  %cmp764 = icmp ult i64 %11, 0
+  %or765 = or i1 %or763, %cmp764
+  ret i1 %or765
+}
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/extract-shuffle-inseltpoison.ll b/llvm/test/Transforms/SLPVectorizer/X86/extract-shuffle-inseltpoison.ll
index 04d190bccfea4..74b9f31973d12 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/extract-shuffle-inseltpoison.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/extract-shuffle-inseltpoison.ll
@@ -3,12 +3,9 @@
 
 define <2 x i8> @g(<2 x i8> %x, <2 x i8> %y) {
 ; CHECK-LABEL: @g(
-; CHECK-NEXT:    [[X0:%.*]] = extractelement <2 x i8> [[X:%.*]], i32 0
-; CHECK-NEXT:    [[Y1:%.*]] = extractelement <2 x i8> [[Y:%.*]], i32 1
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x i8> poison, i8 [[X0]], i32 0
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x i8> [[TMP1]], i8 [[Y1]], i32 1
-; CHECK-NEXT:    [[TMP3:%.*]] = mul <2 x i8> [[TMP2]], [[TMP2]]
-; CHECK-NEXT:    ret <2 x i8> [[TMP3]]
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <2 x i8> [[X:%.*]], <2 x i8> [[Y:%.*]], <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    [[TMP2:%.*]] = mul <2 x i8> [[TMP1]], [[TMP1]]
+; CHECK-NEXT:    ret <2 x i8> [[TMP2]]
 ;
   %x0 = extractelement <2 x i8> %x, i32 0
   %y1 = extractelement <2 x i8> %y, i32 1
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/extract-shuffle.ll b/llvm/test/Transforms/SLPVectorizer/X86/extract-shuffle.ll
index b6535f9f2e696..9276efb6e9cdc 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/extract-shuffle.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/extract-shuffle.ll
@@ -3,12 +3,9 @@
 
 define <2 x i8> @g(<2 x i8> %x, <2 x i8> %y) {
 ; CHECK-LABEL: @g(
-; CHECK-NEXT:    [[X0:%.*]] = extractelement <2 x i8> [[X:%.*]], i32 0
-; CHECK-NEXT:    [[Y1:%.*]] = extractelement <2 x i8> [[Y:%.*]], i32 1
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x i8> poison, i8 [[X0]], i32 0
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x i8> [[TMP1]], i8 [[Y1]], i32 1
-; CHECK-NEXT:    [[TMP3:%.*]] = mul <2 x i8> [[TMP2]], [[TMP2]]
-; CHECK-NEXT:    ret <2 x i8> [[TMP3]]
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <2 x i8> [[X:%.*]], <2 x i8> [[Y:%.*]], <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    [[TMP2:%.*]] = mul <2 x i8> [[TMP1]], [[TMP1]]
+; CHECK-NEXT:    ret <2 x i8> [[TMP2]]
 ;
   %x0 = extractelement <2 x i8> %x, i32 0
   %y1 = extractelement <2 x i8> %y, i32 1
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/extract.ll b/llvm/test/Transforms/SLPVectorizer/X86/extract.ll
index 7de9f4a3aee7c..8fde48dd3bb5a 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/extract.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/extract.ll
@@ -28,8 +28,8 @@ define void @fextr1(ptr %ptr) {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[LD:%.*]] = load <2 x double>, ptr undef, align 16
 ; CHECK-NEXT:    [[TMP0:%.*]] = fadd <2 x double> [[LD]], <double 1.200000e+00, double 3.400000e+00>
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x double> [[TMP0]], <2 x double> poison, <2 x i32> <i32 1, i32 0>
-; CHECK-NEXT:    store <2 x double> [[SHUFFLE]], ptr [[PTR:%.*]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <2 x double> [[TMP0]], <2 x double> poison, <2 x i32> <i32 1, i32 0>
+; CHECK-NEXT:    store <2 x double> [[TMP1]], ptr [[PTR:%.*]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
@@ -48,12 +48,9 @@ define void @fextr2(ptr %ptr) {
 ; CHECK-LABEL: @fextr2(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[LD:%.*]] = load <4 x double>, ptr undef, align 32
-; CHECK-NEXT:    [[V0:%.*]] = extractelement <4 x double> [[LD]], i32 0
-; CHECK-NEXT:    [[V1:%.*]] = extractelement <4 x double> [[LD]], i32 1
-; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <2 x double> poison, double [[V0]], i32 0
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x double> [[TMP0]], double [[V1]], i32 1
-; CHECK-NEXT:    [[TMP2:%.*]] = fadd <2 x double> [[TMP1]], <double 5.500000e+00, double 6.600000e+00>
-; CHECK-NEXT:    store <2 x double> [[TMP2]], ptr [[PTR:%.*]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <4 x double> [[LD]], <4 x double> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[TMP1:%.*]] = fadd <2 x double> [[TMP0]], <double 5.500000e+00, double 6.600000e+00>
+; CHECK-NEXT:    store <2 x double> [[TMP1]], ptr [[PTR:%.*]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/extractelement-multiple-uses.ll b/llvm/test/Transforms/SLPVectorizer/X86/extractelement-multiple-uses.ll
index 33959dc48f296..35f521fbe8113 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/extractelement-multiple-uses.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/extractelement-multiple-uses.ll
@@ -14,10 +14,8 @@
 
 define float @multi_uses(<2 x float> %x, <2 x float> %y) {
 ; CHECK-LABEL: @multi_uses(
-; CHECK-NEXT:    [[Y1:%.*]] = extractelement <2 x float> [[Y:%.*]], i32 1
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x float> poison, float [[Y1]], i32 0
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x float> [[TMP1]], <2 x float> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP2:%.*]] = fmul <2 x float> [[X:%.*]], [[SHUFFLE]]
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <2 x float> [[Y:%.*]], <2 x float> poison, <2 x i32> <i32 1, i32 1>
+; CHECK-NEXT:    [[TMP2:%.*]] = fmul <2 x float> [[X:%.*]], [[TMP1]]
 ; CHECK-NEXT:    [[TMP3:%.*]] = extractelement <2 x float> [[TMP2]], i32 0
 ; CHECK-NEXT:    [[TMP4:%.*]] = extractelement <2 x float> [[TMP2]], i32 1
 ; CHECK-NEXT:    [[ADD:%.*]] = fadd float [[TMP3]], [[TMP4]]
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/extractelement.ll b/llvm/test/Transforms/SLPVectorizer/X86/extractelement.ll
index 6bff6d9b45a3d..f5718e04df9ad 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/extractelement.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/extractelement.ll
@@ -82,23 +82,19 @@ define float @f_used_twice_in_tree(<2 x float> %x) {
 ; CHECK-NEXT:    ret float [[ADD]]
 ;
 ; THRESH1-LABEL: @f_used_twice_in_tree(
-; THRESH1-NEXT:    [[TMP1:%.*]] = extractelement <2 x float> [[X:%.*]], i32 1
-; THRESH1-NEXT:    [[TMP2:%.*]] = insertelement <2 x float> poison, float [[TMP1]], i32 0
-; THRESH1-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x float> [[TMP2]], <2 x float> poison, <2 x i32> zeroinitializer
-; THRESH1-NEXT:    [[TMP3:%.*]] = fmul <2 x float> [[SHUFFLE]], [[X]]
-; THRESH1-NEXT:    [[TMP4:%.*]] = extractelement <2 x float> [[TMP3]], i32 0
-; THRESH1-NEXT:    [[TMP5:%.*]] = extractelement <2 x float> [[TMP3]], i32 1
-; THRESH1-NEXT:    [[ADD:%.*]] = fadd float [[TMP4]], [[TMP5]]
+; THRESH1-NEXT:    [[TMP1:%.*]] = shufflevector <2 x float> [[X:%.*]], <2 x float> poison, <2 x i32> <i32 1, i32 1>
+; THRESH1-NEXT:    [[TMP2:%.*]] = fmul <2 x float> [[TMP1]], [[X]]
+; THRESH1-NEXT:    [[TMP3:%.*]] = extractelement <2 x float> [[TMP2]], i32 0
+; THRESH1-NEXT:    [[TMP4:%.*]] = extractelement <2 x float> [[TMP2]], i32 1
+; THRESH1-NEXT:    [[ADD:%.*]] = fadd float [[TMP3]], [[TMP4]]
 ; THRESH1-NEXT:    ret float [[ADD]]
 ;
 ; THRESH2-LABEL: @f_used_twice_in_tree(
-; THRESH2-NEXT:    [[TMP1:%.*]] = extractelement <2 x float> [[X:%.*]], i32 1
-; THRESH2-NEXT:    [[TMP2:%.*]] = insertelement <2 x float> poison, float [[TMP1]], i32 0
-; THRESH2-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x float> [[TMP2]], <2 x float> poison, <2 x i32> zeroinitializer
-; THRESH2-NEXT:    [[TMP3:%.*]] = fmul <2 x float> [[SHUFFLE]], [[X]]
-; THRESH2-NEXT:    [[TMP4:%.*]] = extractelement <2 x float> [[TMP3]], i32 0
-; THRESH2-NEXT:    [[TMP5:%.*]] = extractelement <2 x float> [[TMP3]], i32 1
-; THRESH2-NEXT:    [[ADD:%.*]] = fadd float [[TMP4]], [[TMP5]]
+; THRESH2-NEXT:    [[TMP1:%.*]] = shufflevector <2 x float> [[X:%.*]], <2 x float> poison, <2 x i32> <i32 1, i32 1>
+; THRESH2-NEXT:    [[TMP2:%.*]] = fmul <2 x float> [[TMP1]], [[X]]
+; THRESH2-NEXT:    [[TMP3:%.*]] = extractelement <2 x float> [[TMP2]], i32 0
+; THRESH2-NEXT:    [[TMP4:%.*]] = extractelement <2 x float> [[TMP2]], i32 1
+; THRESH2-NEXT:    [[ADD:%.*]] = fadd float [[TMP3]], [[TMP4]]
 ; THRESH2-NEXT:    ret float [[ADD]]
 ;
   %x0 = extractelement <2 x float> %x, i32 0
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/gather-extractelements-different-bbs.ll b/llvm/test/Transforms/SLPVectorizer/X86/gather-extractelements-different-bbs.ll
index 482334e510f64..97189e4ef2a14 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/gather-extractelements-different-bbs.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/gather-extractelements-different-bbs.ll
@@ -6,32 +6,28 @@ define i32 @foo(i32 %a) {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <2 x i32> <i32 0, i32 poison>, i32 [[A:%.*]], i32 1
 ; CHECK-NEXT:    [[TMP1:%.*]] = sub nsw <2 x i32> zeroinitializer, [[TMP0]]
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x i32> [[TMP1]], <2 x i32> poison, <4 x i32> <i32 0, i32 0, i32 0, i32 1>
-; CHECK-NEXT:    [[TMP2:%.*]] = extractelement <4 x i32> [[SHUFFLE]], i32 0
+; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <2 x i32> [[TMP1]], <2 x i32> poison, <4 x i32> <i32 0, i32 0, i32 0, i32 1>
+; CHECK-NEXT:    [[TMP3:%.*]] = extractelement <4 x i32> [[TMP2]], i32 0
 ; CHECK-NEXT:    br i1 false, label [[BB5:%.*]], label [[BB1:%.*]]
 ; CHECK:       bb1:
-; CHECK-NEXT:    [[TMP3:%.*]] = extractelement <4 x i32> [[SHUFFLE]], i32 3
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x i32> poison, i32 [[TMP2]], i32 0
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x i32> [[TMP4]], i32 [[TMP3]], i32 1
-; CHECK-NEXT:    [[TMP6:%.*]] = mul <2 x i32> [[TMP5]], <i32 3, i32 1>
-; CHECK-NEXT:    [[TMP7:%.*]] = extractelement <2 x i32> [[TMP6]], i32 0
-; CHECK-NEXT:    [[TMP8:%.*]] = extractelement <2 x i32> [[TMP6]], i32 1
-; CHECK-NEXT:    [[OP_RDX11:%.*]] = add i32 [[TMP7]], [[TMP8]]
-; CHECK-NEXT:    [[OP_RDX12:%.*]] = add i32 [[OP_RDX11]], 0
+; CHECK-NEXT:    [[TMP4:%.*]] = mul <2 x i32> [[TMP1]], <i32 3, i32 1>
+; CHECK-NEXT:    [[TMP5:%.*]] = extractelement <2 x i32> [[TMP4]], i32 0
+; CHECK-NEXT:    [[TMP6:%.*]] = extractelement <2 x i32> [[TMP4]], i32 1
+; CHECK-NEXT:    [[OP_RDX10:%.*]] = add i32 [[TMP5]], [[TMP6]]
+; CHECK-NEXT:    [[OP_RDX11:%.*]] = add i32 [[OP_RDX10]], 0
 ; CHECK-NEXT:    br label [[BB3:%.*]]
 ; CHECK:       bb2:
 ; CHECK-NEXT:    br label [[BB3]]
 ; CHECK:       bb3:
-; CHECK-NEXT:    [[P1:%.*]] = phi i32 [ [[OP_RDX12]], [[BB1]] ], [ 0, [[BB2:%.*]] ]
+; CHECK-NEXT:    [[P1:%.*]] = phi i32 [ [[OP_RDX11]], [[BB1]] ], [ 0, [[BB2:%.*]] ]
 ; CHECK-NEXT:    ret i32 0
 ; CHECK:       bb4:
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <4 x i32> poison, i32 [[TMP2]], i32 0
-; CHECK-NEXT:    [[SHUFFLE8:%.*]] = shufflevector <4 x i32> [[TMP9]], <4 x i32> poison, <4 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP10:%.*]] = add <4 x i32> [[SHUFFLE]], [[SHUFFLE8]]
-; CHECK-NEXT:    [[TMP11:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[TMP10]])
-; CHECK-NEXT:    [[OP_RDX9:%.*]] = add i32 [[TMP11]], 0
-; CHECK-NEXT:    [[OP_RDX10:%.*]] = add i32 [[OP_RDX9]], [[TMP2]]
-; CHECK-NEXT:    ret i32 [[OP_RDX10]]
+; CHECK-NEXT:    [[TMP7:%.*]] = shufflevector <2 x i32> [[TMP1]], <2 x i32> poison, <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP8:%.*]] = add <4 x i32> [[TMP2]], [[TMP7]]
+; CHECK-NEXT:    [[TMP9:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[TMP8]])
+; CHECK-NEXT:    [[OP_RDX8:%.*]] = add i32 [[TMP9]], 0
+; CHECK-NEXT:    [[OP_RDX9:%.*]] = add i32 [[OP_RDX8]], [[TMP3]]
+; CHECK-NEXT:    ret i32 [[OP_RDX9]]
 ; CHECK:       bb5:
 ; CHECK-NEXT:    br label [[BB4:%.*]]
 ;
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/hadd-inseltpoison.ll b/llvm/test/Transforms/SLPVectorizer/X86/hadd-inseltpoison.ll
index 822db3a2f6ea5..f8fbcc58295a1 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/hadd-inseltpoison.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/hadd-inseltpoison.ll
@@ -143,8 +143,8 @@ define <8 x i16> @test_v8i16(<8 x i16> %a, <8 x i16> %b) {
 ; PR41892
 define void @test_v4f32_v2f32_store(<4 x float> %f, ptr %p){
 ; CHECK-LABEL: @test_v4f32_v2f32_store(
-; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x float> [[F:%.*]], <4 x float> undef, <2 x i32> <i32 1, i32 2>
-; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <4 x float> [[F]], <4 x float> undef, <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x float> [[F:%.*]], <4 x float> poison, <2 x i32> <i32 1, i32 2>
+; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <4 x float> [[F]], <4 x float> poison, <2 x i32> <i32 0, i32 3>
 ; CHECK-NEXT:    [[TMP3:%.*]] = fadd <2 x float> [[TMP1]], [[TMP2]]
 ; CHECK-NEXT:    store <2 x float> [[TMP3]], ptr [[P:%.*]], align 4
 ; CHECK-NEXT:    ret void
@@ -220,8 +220,8 @@ define <4 x double> @test_v4f64_partial_swizzle(<4 x double> %a, <4 x double> %b
 ; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <4 x double> [[A]], <4 x double> [[B]], <2 x i32> <i32 1, i32 5>
 ; CHECK-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP1]], [[TMP2]]
 ; CHECK-NEXT:    [[R3:%.*]] = fadd double [[B2]], [[B3]]
-; CHECK-NEXT:    [[R021:%.*]] = shufflevector <2 x double> [[TMP3]], <2 x double> poison, <4 x i32> <i32 0, i32 undef, i32 1, i32 undef>
-; CHECK-NEXT:    [[R03:%.*]] = insertelement <4 x double> [[R021]], double [[R3]], i64 3
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <2 x double> [[TMP3]], <2 x double> poison, <4 x i32> <i32 0, i32 undef, i32 1, i32 undef>
+; CHECK-NEXT:    [[R03:%.*]] = insertelement <4 x double> [[TMP4]], double [[R3]], i64 3
 ; CHECK-NEXT:    ret <4 x double> [[R03]]
 ;
   %a0 = extractelement <4 x double> %a, i64 0
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/hadd.ll b/llvm/test/Transforms/SLPVectorizer/X86/hadd.ll
index 8fd490a9e38eb..5507cc9404861 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/hadd.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/hadd.ll
@@ -143,8 +143,8 @@ define <8 x i16> @test_v8i16(<8 x i16> %a, <8 x i16> %b) {
 ; PR41892
 define void @test_v4f32_v2f32_store(<4 x float> %f, ptr %p){
 ; CHECK-LABEL: @test_v4f32_v2f32_store(
-; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x float> [[F:%.*]], <4 x float> undef, <2 x i32> <i32 1, i32 2>
-; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <4 x float> [[F]], <4 x float> undef, <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x float> [[F:%.*]], <4 x float> poison, <2 x i32> <i32 1, i32 2>
+; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <4 x float> [[F]], <4 x float> poison, <2 x i32> <i32 0, i32 3>
 ; CHECK-NEXT:    [[TMP3:%.*]] = fadd <2 x float> [[TMP1]], [[TMP2]]
 ; CHECK-NEXT:    store <2 x float> [[TMP3]], ptr [[P:%.*]], align 4
 ; CHECK-NEXT:    ret void
@@ -220,8 +220,8 @@ define <4 x double> @test_v4f64_partial_swizzle(<4 x double> %a, <4 x double> %b
 ; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <4 x double> [[A]], <4 x double> [[B]], <2 x i32> <i32 1, i32 5>
 ; CHECK-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP1]], [[TMP2]]
 ; CHECK-NEXT:    [[R3:%.*]] = fadd double [[B2]], [[B3]]
-; CHECK-NEXT:    [[R021:%.*]] = shufflevector <2 x double> [[TMP3]], <2 x double> poison, <4 x i32> <i32 0, i32 undef, i32 1, i32 undef>
-; CHECK-NEXT:    [[R03:%.*]] = insertelement <4 x double> [[R021]], double [[R3]], i64 3
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <2 x double> [[TMP3]], <2 x double> poison, <4 x i32> <i32 0, i32 undef, i32 1, i32 undef>
+; CHECK-NEXT:    [[R03:%.*]] = insertelement <4 x double> [[TMP4]], double [[R3]], i64 3
 ; CHECK-NEXT:    ret <4 x double> [[R03]]
 ;
   %a0 = extractelement <4 x double> %a, i64 0
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/insert-element-build-vector-const-undef.ll b/llvm/test/Transforms/SLPVectorizer/X86/insert-element-build-vector-const-undef.ll
index 24b0e545bd687..d1350bc029019 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/insert-element-build-vector-const-undef.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/insert-element-build-vector-const-undef.ll
@@ -3,22 +3,13 @@
 
 define <4 x float> @simple_select(<4 x float> %a, <4 x float> %b, <4 x i32> %c) {
 ; CHECK-LABEL: @simple_select(
-; CHECK-NEXT:    [[C0:%.*]] = extractelement <4 x i32> [[C:%.*]], i32 0
-; CHECK-NEXT:    [[C1:%.*]] = extractelement <4 x i32> [[C]], i32 1
-; CHECK-NEXT:    [[A0:%.*]] = extractelement <4 x float> [[A:%.*]], i32 0
-; CHECK-NEXT:    [[A1:%.*]] = extractelement <4 x float> [[A]], i32 1
-; CHECK-NEXT:    [[B0:%.*]] = extractelement <4 x float> [[B:%.*]], i32 0
-; CHECK-NEXT:    [[B1:%.*]] = extractelement <4 x float> [[B]], i32 1
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x i32> poison, i32 [[C0]], i32 0
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x i32> [[TMP1]], i32 [[C1]], i32 1
-; CHECK-NEXT:    [[TMP3:%.*]] = icmp ne <2 x i32> [[TMP2]], zeroinitializer
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> poison, float [[A0]], i32 0
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x float> [[TMP4]], float [[A1]], i32 1
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x float> poison, float [[B0]], i32 0
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <2 x float> [[TMP6]], float [[B1]], i32 1
-; CHECK-NEXT:    [[TMP8:%.*]] = select <2 x i1> [[TMP3]], <2 x float> [[TMP5]], <2 x float> [[TMP7]]
-; CHECK-NEXT:    [[TMP9:%.*]] = shufflevector <2 x float> [[TMP8]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; CHECK-NEXT:    ret <4 x float> [[TMP9]]
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x i32> [[C:%.*]], <4 x i32> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp ne <2 x i32> [[TMP1]], zeroinitializer
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[A:%.*]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <4 x float> [[B:%.*]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[TMP5:%.*]] = select <2 x i1> [[TMP2]], <2 x float> [[TMP3]], <2 x float> [[TMP4]]
+; CHECK-NEXT:    [[TMP6:%.*]] = shufflevector <2 x float> [[TMP5]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; CHECK-NEXT:    ret <4 x float> [[TMP6]]
 ;
   %c0 = extractelement <4 x i32> %c, i32 0
   %c1 = extractelement <4 x i32> %c, i32 1
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/insert-element-build-vector-inseltpoison.ll b/llvm/test/Transforms/SLPVectorizer/X86/insert-element-build-vector-inseltpoison.ll
index e266ce02962b1..654121553b4cf 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/insert-element-build-vector-inseltpoison.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/insert-element-build-vector-inseltpoison.ll
@@ -146,10 +146,10 @@ define <4 x float> @simple_select_eph(<4 x float> %a, <4 x float> %b, <4 x i32>
 ; MINTREESIZE-NEXT:    [[RD:%.*]] = insertelement <4 x float> [[RC]], float [[S3]], i32 3
 ; MINTREESIZE-NEXT:    [[Q0:%.*]] = extractelement <4 x float> [[RD]], i32 0
 ; MINTREESIZE-NEXT:    [[Q1:%.*]] = extractelement <4 x float> [[RD]], i32 1
-; MINTREESIZE-NEXT:    [[TMP5:%.*]] = shufflevector <4 x float> [[RD]], <4 x float> [[RD]], <2 x i32> <i32 0, i32 1>
+; MINTREESIZE-NEXT:    [[TMP5:%.*]] = shufflevector <4 x float> [[RD]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
 ; MINTREESIZE-NEXT:    [[Q2:%.*]] = extractelement <4 x float> [[RD]], i32 2
 ; MINTREESIZE-NEXT:    [[Q3:%.*]] = extractelement <4 x float> [[RD]], i32 3
-; MINTREESIZE-NEXT:    [[TMP6:%.*]] = shufflevector <4 x float> [[RD]], <4 x float> [[RD]], <2 x i32> <i32 2, i32 3>
+; MINTREESIZE-NEXT:    [[TMP6:%.*]] = shufflevector <4 x float> [[RD]], <4 x float> poison, <2 x i32> <i32 2, i32 3>
 ; MINTREESIZE-NEXT:    [[Q4:%.*]] = fadd float [[Q0]], [[Q1]]
 ; MINTREESIZE-NEXT:    [[Q5:%.*]] = fadd float [[Q2]], [[Q3]]
 ; MINTREESIZE-NEXT:    [[TMP7:%.*]] = insertelement <2 x float> poison, float [[Q4]], i32 0
@@ -273,37 +273,19 @@ define <4 x float> @simple_select_users(<4 x float> %a, <4 x float> %b, <4 x i32
 ; Unused insertelement
 define <4 x float> @simple_select_no_users(<4 x float> %a, <4 x float> %b, <4 x i32> %c) #0 {
 ; CHECK-LABEL: @simple_select_no_users(
-; CHECK-NEXT:    [[C0:%.*]] = extractelement <4 x i32> [[C:%.*]], i32 0
-; CHECK-NEXT:    [[C1:%.*]] = extractelement <4 x i32> [[C]], i32 1
-; CHECK-NEXT:    [[C2:%.*]] = extractelement <4 x i32> [[C]], i32 2
-; CHECK-NEXT:    [[C3:%.*]] = extractelement <4 x i32> [[C]], i32 3
-; CHECK-NEXT:    [[A0:%.*]] = extractelement <4 x float> [[A:%.*]], i32 0
-; CHECK-NEXT:    [[A1:%.*]] = extractelement <4 x float> [[A]], i32 1
-; CHECK-NEXT:    [[A2:%.*]] = extractelement <4 x float> [[A]], i32 2
-; CHECK-NEXT:    [[A3:%.*]] = extractelement <4 x float> [[A]], i32 3
-; CHECK-NEXT:    [[B0:%.*]] = extractelement <4 x float> [[B:%.*]], i32 0
-; CHECK-NEXT:    [[B1:%.*]] = extractelement <4 x float> [[B]], i32 1
-; CHECK-NEXT:    [[B2:%.*]] = extractelement <4 x float> [[B]], i32 2
-; CHECK-NEXT:    [[B3:%.*]] = extractelement <4 x float> [[B]], i32 3
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x i32> poison, i32 [[C0]], i32 0
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x i32> [[TMP1]], i32 [[C1]], i32 1
-; CHECK-NEXT:    [[TMP3:%.*]] = icmp ne <2 x i32> [[TMP2]], zeroinitializer
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> poison, float [[A0]], i32 0
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x float> [[TMP4]], float [[A1]], i32 1
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x float> poison, float [[B0]], i32 0
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <2 x float> [[TMP6]], float [[B1]], i32 1
-; CHECK-NEXT:    [[TMP8:%.*]] = select <2 x i1> [[TMP3]], <2 x float> [[TMP5]], <2 x float> [[TMP7]]
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <2 x i32> poison, i32 [[C2]], i32 0
-; CHECK-NEXT:    [[TMP10:%.*]] = insertelement <2 x i32> [[TMP9]], i32 [[C3]], i32 1
-; CHECK-NEXT:    [[TMP11:%.*]] = icmp ne <2 x i32> [[TMP10]], zeroinitializer
-; CHECK-NEXT:    [[TMP12:%.*]] = insertelement <2 x float> poison, float [[A2]], i32 0
-; CHECK-NEXT:    [[TMP13:%.*]] = insertelement <2 x float> [[TMP12]], float [[A3]], i32 1
-; CHECK-NEXT:    [[TMP14:%.*]] = insertelement <2 x float> poison, float [[B2]], i32 0
-; CHECK-NEXT:    [[TMP15:%.*]] = insertelement <2 x float> [[TMP14]], float [[B3]], i32 1
-; CHECK-NEXT:    [[TMP16:%.*]] = select <2 x i1> [[TMP11]], <2 x float> [[TMP13]], <2 x float> [[TMP15]]
-; CHECK-NEXT:    [[TMP17:%.*]] = shufflevector <2 x float> [[TMP8]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; CHECK-NEXT:    [[TMP18:%.*]] = shufflevector <2 x float> [[TMP16]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; CHECK-NEXT:    [[RD1:%.*]] = shufflevector <4 x float> [[TMP18]], <4 x float> poison, <4 x i32> <i32 undef, i32 undef, i32 0, i32 1>
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x i32> [[C:%.*]], <4 x i32> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp ne <2 x i32> [[TMP1]], zeroinitializer
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[A:%.*]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <4 x float> [[B:%.*]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[TMP5:%.*]] = select <2 x i1> [[TMP2]], <2 x float> [[TMP3]], <2 x float> [[TMP4]]
+; CHECK-NEXT:    [[TMP6:%.*]] = shufflevector <4 x i32> [[C]], <4 x i32> poison, <2 x i32> <i32 2, i32 3>
+; CHECK-NEXT:    [[TMP7:%.*]] = icmp ne <2 x i32> [[TMP6]], zeroinitializer
+; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <4 x float> [[A]], <4 x float> poison, <2 x i32> <i32 2, i32 3>
+; CHECK-NEXT:    [[TMP9:%.*]] = shufflevector <4 x float> [[B]], <4 x float> poison, <2 x i32> <i32 2, i32 3>
+; CHECK-NEXT:    [[TMP10:%.*]] = select <2 x i1> [[TMP7]], <2 x float> [[TMP8]], <2 x float> [[TMP9]]
+; CHECK-NEXT:    [[TMP11:%.*]] = shufflevector <2 x float> [[TMP5]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; CHECK-NEXT:    [[TMP12:%.*]] = shufflevector <2 x float> [[TMP10]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; CHECK-NEXT:    [[RD1:%.*]] = shufflevector <4 x float> [[TMP12]], <4 x float> poison, <4 x i32> <i32 undef, i32 undef, i32 0, i32 1>
 ; CHECK-NEXT:    ret <4 x float> [[RD1]]
 ;
   %c0 = extractelement <4 x i32> %c, i32 0
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/insert-element-build-vector.ll b/llvm/test/Transforms/SLPVectorizer/X86/insert-element-build-vector.ll
index 4be025a7324a7..d855dd6292c00 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/insert-element-build-vector.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/insert-element-build-vector.ll
@@ -180,10 +180,10 @@ define <4 x float> @simple_select_eph(<4 x float> %a, <4 x float> %b, <4 x i32>
 ; MINTREESIZE-NEXT:    [[RD:%.*]] = insertelement <4 x float> [[RC]], float [[S3]], i32 3
 ; MINTREESIZE-NEXT:    [[Q0:%.*]] = extractelement <4 x float> [[RD]], i32 0
 ; MINTREESIZE-NEXT:    [[Q1:%.*]] = extractelement <4 x float> [[RD]], i32 1
-; MINTREESIZE-NEXT:    [[TMP5:%.*]] = shufflevector <4 x float> [[RD]], <4 x float> [[RD]], <2 x i32> <i32 0, i32 1>
+; MINTREESIZE-NEXT:    [[TMP5:%.*]] = shufflevector <4 x float> [[RD]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
 ; MINTREESIZE-NEXT:    [[Q2:%.*]] = extractelement <4 x float> [[RD]], i32 2
 ; MINTREESIZE-NEXT:    [[Q3:%.*]] = extractelement <4 x float> [[RD]], i32 3
-; MINTREESIZE-NEXT:    [[TMP6:%.*]] = shufflevector <4 x float> [[RD]], <4 x float> [[RD]], <2 x i32> <i32 2, i32 3>
+; MINTREESIZE-NEXT:    [[TMP6:%.*]] = shufflevector <4 x float> [[RD]], <4 x float> poison, <2 x i32> <i32 2, i32 3>
 ; MINTREESIZE-NEXT:    [[Q4:%.*]] = fadd float [[Q0]], [[Q1]]
 ; MINTREESIZE-NEXT:    [[Q5:%.*]] = fadd float [[Q2]], [[Q3]]
 ; MINTREESIZE-NEXT:    [[TMP7:%.*]] = insertelement <2 x float> poison, float [[Q4]], i32 0
@@ -307,37 +307,19 @@ define <4 x float> @simple_select_users(<4 x float> %a, <4 x float> %b, <4 x i32
 ; Unused insertelement
 define <4 x float> @simple_select_no_users(<4 x float> %a, <4 x float> %b, <4 x i32> %c) #0 {
 ; CHECK-LABEL: @simple_select_no_users(
-; CHECK-NEXT:    [[C0:%.*]] = extractelement <4 x i32> [[C:%.*]], i32 0
-; CHECK-NEXT:    [[C1:%.*]] = extractelement <4 x i32> [[C]], i32 1
-; CHECK-NEXT:    [[C2:%.*]] = extractelement <4 x i32> [[C]], i32 2
-; CHECK-NEXT:    [[C3:%.*]] = extractelement <4 x i32> [[C]], i32 3
-; CHECK-NEXT:    [[A0:%.*]] = extractelement <4 x float> [[A:%.*]], i32 0
-; CHECK-NEXT:    [[A1:%.*]] = extractelement <4 x float> [[A]], i32 1
-; CHECK-NEXT:    [[A2:%.*]] = extractelement <4 x float> [[A]], i32 2
-; CHECK-NEXT:    [[A3:%.*]] = extractelement <4 x float> [[A]], i32 3
-; CHECK-NEXT:    [[B0:%.*]] = extractelement <4 x float> [[B:%.*]], i32 0
-; CHECK-NEXT:    [[B1:%.*]] = extractelement <4 x float> [[B]], i32 1
-; CHECK-NEXT:    [[B2:%.*]] = extractelement <4 x float> [[B]], i32 2
-; CHECK-NEXT:    [[B3:%.*]] = extractelement <4 x float> [[B]], i32 3
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x i32> poison, i32 [[C0]], i32 0
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x i32> [[TMP1]], i32 [[C1]], i32 1
-; CHECK-NEXT:    [[TMP3:%.*]] = icmp ne <2 x i32> [[TMP2]], zeroinitializer
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> poison, float [[A0]], i32 0
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x float> [[TMP4]], float [[A1]], i32 1
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x float> poison, float [[B0]], i32 0
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <2 x float> [[TMP6]], float [[B1]], i32 1
-; CHECK-NEXT:    [[TMP8:%.*]] = select <2 x i1> [[TMP3]], <2 x float> [[TMP5]], <2 x float> [[TMP7]]
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <2 x i32> poison, i32 [[C2]], i32 0
-; CHECK-NEXT:    [[TMP10:%.*]] = insertelement <2 x i32> [[TMP9]], i32 [[C3]], i32 1
-; CHECK-NEXT:    [[TMP11:%.*]] = icmp ne <2 x i32> [[TMP10]], zeroinitializer
-; CHECK-NEXT:    [[TMP12:%.*]] = insertelement <2 x float> poison, float [[A2]], i32 0
-; CHECK-NEXT:    [[TMP13:%.*]] = insertelement <2 x float> [[TMP12]], float [[A3]], i32 1
-; CHECK-NEXT:    [[TMP14:%.*]] = insertelement <2 x float> poison, float [[B2]], i32 0
-; CHECK-NEXT:    [[TMP15:%.*]] = insertelement <2 x float> [[TMP14]], float [[B3]], i32 1
-; CHECK-NEXT:    [[TMP16:%.*]] = select <2 x i1> [[TMP11]], <2 x float> [[TMP13]], <2 x float> [[TMP15]]
-; CHECK-NEXT:    [[TMP17:%.*]] = shufflevector <2 x float> [[TMP8]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; CHECK-NEXT:    [[TMP18:%.*]] = shufflevector <2 x float> [[TMP16]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; CHECK-NEXT:    [[RD1:%.*]] = shufflevector <4 x float> [[TMP18]], <4 x float> undef, <4 x i32> <i32 4, i32 5, i32 0, i32 1>
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x i32> [[C:%.*]], <4 x i32> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp ne <2 x i32> [[TMP1]], zeroinitializer
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[A:%.*]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <4 x float> [[B:%.*]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[TMP5:%.*]] = select <2 x i1> [[TMP2]], <2 x float> [[TMP3]], <2 x float> [[TMP4]]
+; CHECK-NEXT:    [[TMP6:%.*]] = shufflevector <4 x i32> [[C]], <4 x i32> poison, <2 x i32> <i32 2, i32 3>
+; CHECK-NEXT:    [[TMP7:%.*]] = icmp ne <2 x i32> [[TMP6]], zeroinitializer
+; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <4 x float> [[A]], <4 x float> poison, <2 x i32> <i32 2, i32 3>
+; CHECK-NEXT:    [[TMP9:%.*]] = shufflevector <4 x float> [[B]], <4 x float> poison, <2 x i32> <i32 2, i32 3>
+; CHECK-NEXT:    [[TMP10:%.*]] = select <2 x i1> [[TMP7]], <2 x float> [[TMP8]], <2 x float> [[TMP9]]
+; CHECK-NEXT:    [[TMP11:%.*]] = shufflevector <2 x float> [[TMP5]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; CHECK-NEXT:    [[TMP12:%.*]] = shufflevector <2 x float> [[TMP10]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; CHECK-NEXT:    [[RD1:%.*]] = shufflevector <4 x float> [[TMP12]], <4 x float> undef, <4 x i32> <i32 4, i32 5, i32 0, i32 1>
 ; CHECK-NEXT:    ret <4 x float> [[RD1]]
 ;
   %c0 = extractelement <4 x i32> %c, i32 0
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/insertelement-postpone.ll b/llvm/test/Transforms/SLPVectorizer/X86/insertelement-postpone.ll
index 6756d9aa2101e..3fc6fe9e60611 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/insertelement-postpone.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/insertelement-postpone.ll
@@ -13,11 +13,11 @@ define <4 x double> @test(ptr %p2, double %i1754, double %i1781, double %i1778)
 ; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <4 x double> [[TMP0]], double [[I1778:%.*]], i32 1
 ; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <4 x double> [[TMP1]], double [[I1781]], i32 2
 ; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <4 x double> [[TMP2]], double [[I1772]], i32 3
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <4 x double> [[TMP0]], <4 x double> poison, <4 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP4:%.*]] = fmul fast <4 x double> [[TMP3]], [[SHUFFLE]]
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <4 x double> <double 1.000000e+00, double 1.000000e+00, double 1.000000e+00, double poison>, double [[I1797]], i32 3
-; CHECK-NEXT:    [[TMP6:%.*]] = fadd fast <4 x double> [[TMP4]], [[TMP5]]
-; CHECK-NEXT:    ret <4 x double> [[TMP6]]
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <4 x double> [[TMP3]], <4 x double> poison, <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP5:%.*]] = fmul fast <4 x double> [[TMP3]], [[TMP4]]
+; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <4 x double> <double 1.000000e+00, double 1.000000e+00, double 1.000000e+00, double poison>, double [[I1797]], i32 3
+; CHECK-NEXT:    [[TMP7:%.*]] = fadd fast <4 x double> [[TMP5]], [[TMP6]]
+; CHECK-NEXT:    ret <4 x double> [[TMP7]]
 ;
 entry:
   %i1771 = getelementptr inbounds double, ptr %p2, i64 54
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/jumbled-load-multiuse.ll b/llvm/test/Transforms/SLPVectorizer/X86/jumbled-load-multiuse.ll
index a616fd5f7c197..6adcf21a4bc63 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/jumbled-load-multiuse.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/jumbled-load-multiuse.ll
@@ -9,11 +9,10 @@ define i32 @fn1() {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = load <4 x i32>, ptr @b, align 4
 ; CHECK-NEXT:    [[TMP1:%.*]] = icmp sgt <4 x i32> [[TMP0]], zeroinitializer
-; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <4 x i32> <i32 8, i32 poison, i32 ptrtoint (ptr @fn1 to i32), i32 poison>, <4 x i32> [[TMP0]], <4 x i32> <i32 0, i32 5, i32 2, i32 undef>
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <4 x i32> [[TMP2]], <4 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 2>
-; CHECK-NEXT:    [[TMP3:%.*]] = select <4 x i1> [[TMP1]], <4 x i32> [[SHUFFLE]], <4 x i32> <i32 0, i32 6, i32 0, i32 0>
-; CHECK-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <4 x i32> [[TMP3]], <4 x i32> poison, <4 x i32> <i32 1, i32 2, i32 3, i32 0>
-; CHECK-NEXT:    store <4 x i32> [[SHUFFLE1]], ptr @a, align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> <i32 8, i32 poison, i32 ptrtoint (ptr @fn1 to i32), i32 ptrtoint (ptr @fn1 to i32)>, <4 x i32> <i32 4, i32 1, i32 6, i32 7>
+; CHECK-NEXT:    [[TMP3:%.*]] = select <4 x i1> [[TMP1]], <4 x i32> [[TMP2]], <4 x i32> <i32 0, i32 6, i32 0, i32 0>
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <4 x i32> [[TMP3]], <4 x i32> poison, <4 x i32> <i32 1, i32 2, i32 3, i32 0>
+; CHECK-NEXT:    store <4 x i32> [[TMP4]], ptr @a, align 4
 ; CHECK-NEXT:    ret i32 0
 ;
   entry:
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/landing_pad.ll b/llvm/test/Transforms/SLPVectorizer/X86/landing_pad.ll
index 402e111c98598..9ed021448e020 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/landing_pad.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/landing_pad.ll
@@ -10,10 +10,10 @@ define void @foo() personality ptr @bar {
 ; CHECK:       bb2.loopexit:
 ; CHECK-NEXT:    br label [[BB2:%.*]]
 ; CHECK:       bb2:
-; CHECK-NEXT:    [[TMP0:%.*]] = phi <4 x i32> [ [[TMP10:%.*]], [[BB9:%.*]] ], [ poison, [[BB2_LOOPEXIT:%.*]] ]
+; CHECK-NEXT:    [[TMP0:%.*]] = phi <4 x i32> [ [[TMP8:%.*]], [[BB9:%.*]] ], [ poison, [[BB2_LOOPEXIT:%.*]] ]
 ; CHECK-NEXT:    ret void
 ; CHECK:       bb3:
-; CHECK-NEXT:    [[TMP1:%.*]] = phi <2 x i32> [ [[TMP5:%.*]], [[BB6:%.*]] ], [ poison, [[BB1:%.*]] ]
+; CHECK-NEXT:    [[TMP1:%.*]] = phi <2 x i32> [ [[TMP3:%.*]], [[BB6:%.*]] ], [ poison, [[BB1:%.*]] ]
 ; CHECK-NEXT:    [[TMP2:%.*]] = invoke i32 poison(ptr addrspace(1) nonnull poison, i32 0, i32 0, i32 poison) [ "deopt"() ]
 ; CHECK-NEXT:    to label [[BB4:%.*]] unwind label [[BB10:%.*]]
 ; CHECK:       bb4:
@@ -21,32 +21,30 @@ define void @foo() personality ptr @bar {
 ; CHECK:       bb5:
 ; CHECK-NEXT:    br label [[BB7:%.*]]
 ; CHECK:       bb6:
-; CHECK-NEXT:    [[TMP3:%.*]] = phi <2 x i32> [ <i32 0, i32 poison>, [[BB8:%.*]] ]
-; CHECK-NEXT:    [[TMP4:%.*]] = extractelement <2 x i32> [[TMP3]], i32 1
-; CHECK-NEXT:    [[TMP5]] = insertelement <2 x i32> poison, i32 [[TMP4]], i32 1
+; CHECK-NEXT:    [[TMP3]] = phi <2 x i32> [ <i32 0, i32 poison>, [[BB8:%.*]] ]
 ; CHECK-NEXT:    br label [[BB3]]
 ; CHECK:       bb7:
 ; CHECK-NEXT:    [[LOCAL_5_84111:%.*]] = phi i32 [ poison, [[BB8]] ], [ poison, [[BB5]] ]
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x i32> poison, i32 [[LOCAL_5_84111]], i32 0
-; CHECK-NEXT:    [[TMP7:%.*]] = invoke i32 poison(ptr addrspace(1) nonnull poison, i32 poison, i32 poison, i32 poison) [ "deopt"() ]
+; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x i32> poison, i32 [[LOCAL_5_84111]], i32 0
+; CHECK-NEXT:    [[TMP5:%.*]] = invoke i32 poison(ptr addrspace(1) nonnull poison, i32 poison, i32 poison, i32 poison) [ "deopt"() ]
 ; CHECK-NEXT:    to label [[BB8]] unwind label [[BB12:%.*]]
 ; CHECK:       bb8:
 ; CHECK-NEXT:    br i1 poison, label [[BB7]], label [[BB6]]
 ; CHECK:       bb9:
 ; CHECK-NEXT:    [[INDVARS_IV528799:%.*]] = phi i64 [ poison, [[BB10]] ], [ poison, [[BB12]] ]
-; CHECK-NEXT:    [[TMP8:%.*]] = phi <2 x i32> [ [[TMP11:%.*]], [[BB10]] ], [ [[TMP12:%.*]], [[BB12]] ]
-; CHECK-NEXT:    [[TMP9:%.*]] = shufflevector <2 x i32> [[TMP8]], <2 x i32> poison, <4 x i32> <i32 1, i32 0, i32 undef, i32 undef>
-; CHECK-NEXT:    [[TMP10]] = shufflevector <4 x i32> [[TMP9]], <4 x i32> poison, <4 x i32> <i32 undef, i32 undef, i32 0, i32 1>
+; CHECK-NEXT:    [[TMP6:%.*]] = phi <2 x i32> [ [[TMP9:%.*]], [[BB10]] ], [ [[TMP10:%.*]], [[BB12]] ]
+; CHECK-NEXT:    [[TMP7:%.*]] = shufflevector <2 x i32> [[TMP6]], <2 x i32> poison, <4 x i32> <i32 1, i32 0, i32 undef, i32 undef>
+; CHECK-NEXT:    [[TMP8]] = shufflevector <4 x i32> [[TMP7]], <4 x i32> poison, <4 x i32> <i32 undef, i32 undef, i32 0, i32 1>
 ; CHECK-NEXT:    br label [[BB2]]
 ; CHECK:       bb10:
-; CHECK-NEXT:    [[TMP11]] = phi <2 x i32> [ [[TMP1]], [[BB3]] ]
+; CHECK-NEXT:    [[TMP9]] = phi <2 x i32> [ [[TMP1]], [[BB3]] ]
 ; CHECK-NEXT:    [[LANDING_PAD68:%.*]] = landingpad { ptr, i32 }
 ; CHECK-NEXT:    cleanup
 ; CHECK-NEXT:    br label [[BB9]]
 ; CHECK:       bb11:
 ; CHECK-NEXT:    ret void
 ; CHECK:       bb12:
-; CHECK-NEXT:    [[TMP12]] = phi <2 x i32> [ [[TMP6]], [[BB7]] ]
+; CHECK-NEXT:    [[TMP10]] = phi <2 x i32> [ [[TMP4]], [[BB7]] ]
 ; CHECK-NEXT:    [[LANDING_PAD149:%.*]] = landingpad { ptr, i32 }
 ; CHECK-NEXT:    cleanup
 ; CHECK-NEXT:    br label [[BB9]]
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/lookahead.ll b/llvm/test/Transforms/SLPVectorizer/X86/lookahead.ll
index 1ed2841a770db..52a84e6248f79 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/lookahead.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/lookahead.ll
@@ -23,14 +23,14 @@ define void @lookahead_basic(ptr %array) {
 ; CHECK-NEXT:    [[IDX2:%.*]] = getelementptr inbounds double, ptr [[ARRAY:%.*]], i64 2
 ; CHECK-NEXT:    [[IDX4:%.*]] = getelementptr inbounds double, ptr [[ARRAY]], i64 4
 ; CHECK-NEXT:    [[IDX6:%.*]] = getelementptr inbounds double, ptr [[ARRAY]], i64 6
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[ARRAY]], align 8
-; CHECK-NEXT:    [[TMP3:%.*]] = load <2 x double>, ptr [[IDX2]], align 8
-; CHECK-NEXT:    [[TMP5:%.*]] = load <2 x double>, ptr [[IDX4]], align 8
-; CHECK-NEXT:    [[TMP7:%.*]] = load <2 x double>, ptr [[IDX6]], align 8
-; CHECK-NEXT:    [[TMP8:%.*]] = fsub fast <2 x double> [[TMP1]], [[TMP3]]
-; CHECK-NEXT:    [[TMP9:%.*]] = fsub fast <2 x double> [[TMP5]], [[TMP7]]
-; CHECK-NEXT:    [[TMP10:%.*]] = fadd fast <2 x double> [[TMP9]], [[TMP8]]
-; CHECK-NEXT:    store <2 x double> [[TMP10]], ptr [[ARRAY]], align 8
+; CHECK-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[ARRAY]], align 8
+; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[IDX2]], align 8
+; CHECK-NEXT:    [[TMP2:%.*]] = load <2 x double>, ptr [[IDX4]], align 8
+; CHECK-NEXT:    [[TMP3:%.*]] = load <2 x double>, ptr [[IDX6]], align 8
+; CHECK-NEXT:    [[TMP4:%.*]] = fsub fast <2 x double> [[TMP0]], [[TMP1]]
+; CHECK-NEXT:    [[TMP5:%.*]] = fsub fast <2 x double> [[TMP2]], [[TMP3]]
+; CHECK-NEXT:    [[TMP6:%.*]] = fadd fast <2 x double> [[TMP5]], [[TMP4]]
+; CHECK-NEXT:    store <2 x double> [[TMP6]], ptr [[ARRAY]], align 8
 ; CHECK-NEXT:    ret void
 ;
 entry:
@@ -85,12 +85,12 @@ define void @lookahead_alt1(ptr %array) {
 ; CHECK-NEXT:    [[IDX5:%.*]] = getelementptr inbounds double, ptr [[ARRAY]], i64 5
 ; CHECK-NEXT:    [[IDX6:%.*]] = getelementptr inbounds double, ptr [[ARRAY]], i64 6
 ; CHECK-NEXT:    [[IDX7:%.*]] = getelementptr inbounds double, ptr [[ARRAY]], i64 7
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[ARRAY]], align 8
-; CHECK-NEXT:    [[TMP3:%.*]] = load <2 x double>, ptr [[IDX2]], align 8
-; CHECK-NEXT:    [[TMP4:%.*]] = fsub fast <2 x double> [[TMP1]], [[TMP3]]
-; CHECK-NEXT:    [[TMP5:%.*]] = fadd fast <2 x double> [[TMP1]], [[TMP3]]
-; CHECK-NEXT:    [[TMP6:%.*]] = fadd fast <2 x double> [[TMP5]], [[TMP4]]
-; CHECK-NEXT:    store <2 x double> [[TMP6]], ptr [[ARRAY]], align 8
+; CHECK-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[ARRAY]], align 8
+; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[IDX2]], align 8
+; CHECK-NEXT:    [[TMP2:%.*]] = fsub fast <2 x double> [[TMP0]], [[TMP1]]
+; CHECK-NEXT:    [[TMP3:%.*]] = fadd fast <2 x double> [[TMP0]], [[TMP1]]
+; CHECK-NEXT:    [[TMP4:%.*]] = fadd fast <2 x double> [[TMP3]], [[TMP2]]
+; CHECK-NEXT:    store <2 x double> [[TMP4]], ptr [[ARRAY]], align 8
 ; CHECK-NEXT:    ret void
 ;
 entry:
@@ -139,18 +139,18 @@ define void @lookahead_alt2(ptr %array) {
 ; CHECK-NEXT:    [[IDX2:%.*]] = getelementptr inbounds double, ptr [[ARRAY:%.*]], i64 2
 ; CHECK-NEXT:    [[IDX4:%.*]] = getelementptr inbounds double, ptr [[ARRAY]], i64 4
 ; CHECK-NEXT:    [[IDX6:%.*]] = getelementptr inbounds double, ptr [[ARRAY]], i64 6
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[ARRAY]], align 8
-; CHECK-NEXT:    [[TMP3:%.*]] = load <2 x double>, ptr [[IDX2]], align 8
-; CHECK-NEXT:    [[TMP5:%.*]] = load <2 x double>, ptr [[IDX4]], align 8
-; CHECK-NEXT:    [[TMP7:%.*]] = load <2 x double>, ptr [[IDX6]], align 8
-; CHECK-NEXT:    [[TMP8:%.*]] = fsub fast <2 x double> [[TMP5]], [[TMP7]]
-; CHECK-NEXT:    [[TMP9:%.*]] = fadd fast <2 x double> [[TMP5]], [[TMP7]]
-; CHECK-NEXT:    [[TMP10:%.*]] = shufflevector <2 x double> [[TMP8]], <2 x double> [[TMP9]], <2 x i32> <i32 0, i32 3>
-; CHECK-NEXT:    [[TMP11:%.*]] = fadd fast <2 x double> [[TMP1]], [[TMP3]]
-; CHECK-NEXT:    [[TMP12:%.*]] = fsub fast <2 x double> [[TMP1]], [[TMP3]]
-; CHECK-NEXT:    [[TMP13:%.*]] = shufflevector <2 x double> [[TMP11]], <2 x double> [[TMP12]], <2 x i32> <i32 0, i32 3>
-; CHECK-NEXT:    [[TMP14:%.*]] = fadd fast <2 x double> [[TMP10]], [[TMP13]]
-; CHECK-NEXT:    store <2 x double> [[TMP14]], ptr [[ARRAY]], align 8
+; CHECK-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[ARRAY]], align 8
+; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[IDX2]], align 8
+; CHECK-NEXT:    [[TMP2:%.*]] = load <2 x double>, ptr [[IDX4]], align 8
+; CHECK-NEXT:    [[TMP3:%.*]] = load <2 x double>, ptr [[IDX6]], align 8
+; CHECK-NEXT:    [[TMP4:%.*]] = fsub fast <2 x double> [[TMP2]], [[TMP3]]
+; CHECK-NEXT:    [[TMP5:%.*]] = fadd fast <2 x double> [[TMP2]], [[TMP3]]
+; CHECK-NEXT:    [[TMP6:%.*]] = shufflevector <2 x double> [[TMP4]], <2 x double> [[TMP5]], <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    [[TMP7:%.*]] = fadd fast <2 x double> [[TMP0]], [[TMP1]]
+; CHECK-NEXT:    [[TMP8:%.*]] = fsub fast <2 x double> [[TMP0]], [[TMP1]]
+; CHECK-NEXT:    [[TMP9:%.*]] = shufflevector <2 x double> [[TMP7]], <2 x double> [[TMP8]], <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    [[TMP10:%.*]] = fadd fast <2 x double> [[TMP6]], [[TMP9]]
+; CHECK-NEXT:    store <2 x double> [[TMP10]], ptr [[ARRAY]], align 8
 ; CHECK-NEXT:    ret void
 ;
 entry:
@@ -207,22 +207,22 @@ define void @lookahead_external_uses(ptr %A, ptr %B, ptr %C, ptr %D, ptr %S, ptr
 ; CHECK-NEXT:    [[B0:%.*]] = load double, ptr [[B]], align 8
 ; CHECK-NEXT:    [[C0:%.*]] = load double, ptr [[C:%.*]], align 8
 ; CHECK-NEXT:    [[D0:%.*]] = load double, ptr [[D:%.*]], align 8
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[A]], align 8
+; CHECK-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[A]], align 8
 ; CHECK-NEXT:    [[B2:%.*]] = load double, ptr [[IDXB2]], align 8
 ; CHECK-NEXT:    [[A2:%.*]] = load double, ptr [[IDXA2]], align 8
 ; CHECK-NEXT:    [[B1:%.*]] = load double, ptr [[IDXB1]], align 8
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[B0]], i32 0
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x double> [[TMP2]], double [[B2]], i32 1
-; CHECK-NEXT:    [[TMP4:%.*]] = fsub fast <2 x double> [[TMP1]], [[TMP3]]
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x double> poison, double [[C0]], i32 0
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x double> [[TMP5]], double [[A2]], i32 1
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <2 x double> poison, double [[D0]], i32 0
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <2 x double> [[TMP7]], double [[B1]], i32 1
-; CHECK-NEXT:    [[TMP9:%.*]] = fsub fast <2 x double> [[TMP6]], [[TMP8]]
-; CHECK-NEXT:    [[TMP10:%.*]] = fadd fast <2 x double> [[TMP4]], [[TMP9]]
-; CHECK-NEXT:    store <2 x double> [[TMP10]], ptr [[S:%.*]], align 8
-; CHECK-NEXT:    [[TMP12:%.*]] = extractelement <2 x double> [[TMP1]], i32 1
-; CHECK-NEXT:    store double [[TMP12]], ptr [[EXT1:%.*]], align 8
+; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x double> poison, double [[B0]], i32 0
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP1]], double [[B2]], i32 1
+; CHECK-NEXT:    [[TMP3:%.*]] = fsub fast <2 x double> [[TMP0]], [[TMP2]]
+; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x double> poison, double [[C0]], i32 0
+; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x double> [[TMP4]], double [[A2]], i32 1
+; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x double> poison, double [[D0]], i32 0
+; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <2 x double> [[TMP6]], double [[B1]], i32 1
+; CHECK-NEXT:    [[TMP8:%.*]] = fsub fast <2 x double> [[TMP5]], [[TMP7]]
+; CHECK-NEXT:    [[TMP9:%.*]] = fadd fast <2 x double> [[TMP3]], [[TMP8]]
+; CHECK-NEXT:    store <2 x double> [[TMP9]], ptr [[S:%.*]], align 8
+; CHECK-NEXT:    [[TMP10:%.*]] = extractelement <2 x double> [[TMP0]], i32 1
+; CHECK-NEXT:    store double [[TMP10]], ptr [[EXT1:%.*]], align 8
 ; CHECK-NEXT:    ret void
 ;
 entry:
@@ -285,24 +285,24 @@ define void @lookahead_limit_users_budget(ptr %A, ptr %B, ptr %C, ptr %D, ptr %S
 ; CHECK-NEXT:    [[B0:%.*]] = load double, ptr [[B]], align 8
 ; CHECK-NEXT:    [[C0:%.*]] = load double, ptr [[C:%.*]], align 8
 ; CHECK-NEXT:    [[D0:%.*]] = load double, ptr [[D:%.*]], align 8
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[A]], align 8
+; CHECK-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[A]], align 8
 ; CHECK-NEXT:    [[B2:%.*]] = load double, ptr [[IDXB2]], align 8
 ; CHECK-NEXT:    [[A2:%.*]] = load double, ptr [[IDXA2]], align 8
 ; CHECK-NEXT:    [[B1:%.*]] = load double, ptr [[IDXB1]], align 8
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[B0]], i32 0
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x double> [[TMP2]], double [[B2]], i32 1
-; CHECK-NEXT:    [[TMP4:%.*]] = fsub fast <2 x double> [[TMP1]], [[TMP3]]
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x double> poison, double [[C0]], i32 0
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x double> [[TMP5]], double [[A2]], i32 1
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <2 x double> poison, double [[D0]], i32 0
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <2 x double> [[TMP7]], double [[B1]], i32 1
-; CHECK-NEXT:    [[TMP9:%.*]] = fsub fast <2 x double> [[TMP6]], [[TMP8]]
-; CHECK-NEXT:    [[TMP10:%.*]] = fadd fast <2 x double> [[TMP4]], [[TMP9]]
-; CHECK-NEXT:    store <2 x double> [[TMP10]], ptr [[S:%.*]], align 8
-; CHECK-NEXT:    [[TMP12:%.*]] = extractelement <2 x double> [[TMP1]], i32 1
-; CHECK-NEXT:    store double [[TMP12]], ptr [[EXT1:%.*]], align 8
-; CHECK-NEXT:    store double [[TMP12]], ptr [[EXT2:%.*]], align 8
-; CHECK-NEXT:    store double [[TMP12]], ptr [[EXT3:%.*]], align 8
+; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x double> poison, double [[B0]], i32 0
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP1]], double [[B2]], i32 1
+; CHECK-NEXT:    [[TMP3:%.*]] = fsub fast <2 x double> [[TMP0]], [[TMP2]]
+; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x double> poison, double [[C0]], i32 0
+; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x double> [[TMP4]], double [[A2]], i32 1
+; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x double> poison, double [[D0]], i32 0
+; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <2 x double> [[TMP6]], double [[B1]], i32 1
+; CHECK-NEXT:    [[TMP8:%.*]] = fsub fast <2 x double> [[TMP5]], [[TMP7]]
+; CHECK-NEXT:    [[TMP9:%.*]] = fadd fast <2 x double> [[TMP3]], [[TMP8]]
+; CHECK-NEXT:    store <2 x double> [[TMP9]], ptr [[S:%.*]], align 8
+; CHECK-NEXT:    [[TMP10:%.*]] = extractelement <2 x double> [[TMP0]], i32 1
+; CHECK-NEXT:    store double [[TMP10]], ptr [[EXT1:%.*]], align 8
+; CHECK-NEXT:    store double [[TMP10]], ptr [[EXT2:%.*]], align 8
+; CHECK-NEXT:    store double [[TMP10]], ptr [[EXT3:%.*]], align 8
 ; CHECK-NEXT:    store double [[B1]], ptr [[EXT4:%.*]], align 8
 ; CHECK-NEXT:    store double [[B1]], ptr [[EXT5:%.*]], align 8
 ; CHECK-NEXT:    ret void
@@ -358,13 +358,13 @@ declare double @_ZN1i2axEv()
 
 define void @lookahead_crash(ptr %A, ptr %S, ptr %Arg0) {
 ; CHECK-LABEL: @lookahead_crash(
-; CHECK-NEXT:    [[TMP2:%.*]] = load <2 x double>, ptr [[A:%.*]], align 8
+; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[A:%.*]], align 8
 ; CHECK-NEXT:    [[C0:%.*]] = call double @_ZN1i2ayEv(ptr [[ARG0:%.*]])
 ; CHECK-NEXT:    [[C1:%.*]] = call double @_ZN1i2axEv()
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x double> poison, double [[C0]], i32 0
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x double> [[TMP3]], double [[C1]], i32 1
-; CHECK-NEXT:    [[TMP5:%.*]] = fadd fast <2 x double> [[TMP2]], [[TMP4]]
-; CHECK-NEXT:    store <2 x double> [[TMP5]], ptr [[S:%.*]], align 8
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[C0]], i32 0
+; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x double> [[TMP2]], double [[C1]], i32 1
+; CHECK-NEXT:    [[TMP4:%.*]] = fadd fast <2 x double> [[TMP1]], [[TMP3]]
+; CHECK-NEXT:    store <2 x double> [[TMP4]], ptr [[S:%.*]], align 8
 ; CHECK-NEXT:    ret void
 ;
   %IdxA1 = getelementptr inbounds double, ptr %A, i64 1
@@ -393,13 +393,13 @@ define void @ChecksExtractScores(ptr %storeArray, ptr %array, ptr %vecPtr1, ptr
 ; CHECK-NEXT:    [[LOADVEC:%.*]] = load <2 x double>, ptr [[VECPTR1:%.*]], align 4
 ; CHECK-NEXT:    [[LOADVEC2:%.*]] = load <2 x double>, ptr [[VECPTR2:%.*]], align 4
 ; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x double> poison, double [[LOADA0]], i32 0
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x double> [[TMP1]], <2 x double> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP2:%.*]] = fmul <2 x double> [[LOADVEC]], [[SHUFFLE]]
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x double> poison, double [[LOADA1]], i32 0
-; CHECK-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <2 x double> [[TMP3]], <2 x double> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP4:%.*]] = fmul <2 x double> [[LOADVEC2]], [[SHUFFLE1]]
-; CHECK-NEXT:    [[TMP5:%.*]] = fadd <2 x double> [[TMP2]], [[TMP4]]
-; CHECK-NEXT:    store <2 x double> [[TMP5]], ptr [[STOREARRAY:%.*]], align 8
+; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <2 x double> [[TMP1]], <2 x double> poison, <2 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP3:%.*]] = fmul <2 x double> [[LOADVEC]], [[TMP2]]
+; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x double> poison, double [[LOADA1]], i32 0
+; CHECK-NEXT:    [[TMP5:%.*]] = shufflevector <2 x double> [[TMP4]], <2 x double> poison, <2 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP6:%.*]] = fmul <2 x double> [[LOADVEC2]], [[TMP5]]
+; CHECK-NEXT:    [[TMP7:%.*]] = fadd <2 x double> [[TMP3]], [[TMP6]]
+; CHECK-NEXT:    store <2 x double> [[TMP7]], ptr [[STOREARRAY:%.*]], align 8
 ; CHECK-NEXT:    ret void
 ;
   %idx1 = getelementptr inbounds double, ptr %array, i64 1
@@ -527,11 +527,10 @@ define i1 @foo(float %a, float %b, float %c, <4 x float> %vec, i64 %idx2) {
 ; SSE-LABEL: @foo(
 ; SSE-NEXT:    [[VECEXT_I291_I166:%.*]] = extractelement <4 x float> [[VEC:%.*]], i64 0
 ; SSE-NEXT:    [[SUB14_I167:%.*]] = fsub float undef, [[VECEXT_I291_I166]]
-; SSE-NEXT:    [[VECEXT_I276_I169:%.*]] = extractelement <4 x float> [[VEC]], i64 1
 ; SSE-NEXT:    [[TMP1:%.*]] = insertelement <2 x float> poison, float [[A:%.*]], i32 0
 ; SSE-NEXT:    [[TMP2:%.*]] = insertelement <2 x float> [[TMP1]], float [[C:%.*]], i32 1
-; SSE-NEXT:    [[TMP3:%.*]] = insertelement <2 x float> poison, float [[SUB14_I167]], i32 0
-; SSE-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> [[TMP3]], float [[VECEXT_I276_I169]], i32 1
+; SSE-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[VEC]], <4 x float> poison, <2 x i32> <i32 undef, i32 1>
+; SSE-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> [[TMP3]], float [[SUB14_I167]], i32 0
 ; SSE-NEXT:    [[TMP5:%.*]] = fmul <2 x float> [[TMP2]], [[TMP4]]
 ; SSE-NEXT:    [[TMP6:%.*]] = insertelement <2 x float> <float poison, float 3.000000e+01>, float [[B:%.*]], i32 0
 ; SSE-NEXT:    [[TMP7:%.*]] = fsub <2 x float> [[TMP5]], [[TMP6]]
@@ -576,22 +575,16 @@ define void @ChecksExtractScores_different_vectors(ptr %storeArray, ptr %array,
 ; SSE-LABEL: @ChecksExtractScores_different_vectors(
 ; SSE-NEXT:    [[LOADVEC:%.*]] = load <2 x double>, ptr [[VECPTR1:%.*]], align 4
 ; SSE-NEXT:    [[LOADVEC2:%.*]] = load <2 x double>, ptr [[VECPTR2:%.*]], align 4
-; SSE-NEXT:    [[EXTRA0:%.*]] = extractelement <2 x double> [[LOADVEC]], i32 0
-; SSE-NEXT:    [[EXTRA1:%.*]] = extractelement <2 x double> [[LOADVEC2]], i32 1
 ; SSE-NEXT:    [[LOADVEC3:%.*]] = load <2 x double>, ptr [[VECPTR3:%.*]], align 4
 ; SSE-NEXT:    [[LOADVEC4:%.*]] = load <2 x double>, ptr [[VECPTR4:%.*]], align 4
-; SSE-NEXT:    [[EXTRB0:%.*]] = extractelement <2 x double> [[LOADVEC3]], i32 0
-; SSE-NEXT:    [[EXTRB1:%.*]] = extractelement <2 x double> [[LOADVEC4]], i32 1
-; SSE-NEXT:    [[TMP2:%.*]] = load <2 x double>, ptr [[ARRAY:%.*]], align 4
-; SSE-NEXT:    [[TMP3:%.*]] = insertelement <2 x double> poison, double [[EXTRA1]], i32 0
-; SSE-NEXT:    [[TMP4:%.*]] = insertelement <2 x double> [[TMP3]], double [[EXTRB0]], i32 1
-; SSE-NEXT:    [[TMP5:%.*]] = fmul <2 x double> [[TMP4]], [[TMP2]]
-; SSE-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x double> [[TMP5]], <2 x double> poison, <2 x i32> <i32 1, i32 0>
-; SSE-NEXT:    [[TMP6:%.*]] = insertelement <2 x double> poison, double [[EXTRA0]], i32 0
-; SSE-NEXT:    [[TMP7:%.*]] = insertelement <2 x double> [[TMP6]], double [[EXTRB1]], i32 1
-; SSE-NEXT:    [[TMP8:%.*]] = fmul <2 x double> [[TMP7]], [[TMP2]]
-; SSE-NEXT:    [[TMP9:%.*]] = fadd <2 x double> [[SHUFFLE]], [[TMP8]]
-; SSE-NEXT:    store <2 x double> [[TMP9]], ptr [[STOREARRAY:%.*]], align 8
+; SSE-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[ARRAY:%.*]], align 4
+; SSE-NEXT:    [[TMP2:%.*]] = shufflevector <2 x double> [[LOADVEC2]], <2 x double> [[LOADVEC3]], <2 x i32> <i32 1, i32 2>
+; SSE-NEXT:    [[TMP3:%.*]] = fmul <2 x double> [[TMP2]], [[TMP1]]
+; SSE-NEXT:    [[TMP4:%.*]] = shufflevector <2 x double> [[TMP3]], <2 x double> poison, <2 x i32> <i32 1, i32 0>
+; SSE-NEXT:    [[TMP5:%.*]] = shufflevector <2 x double> [[LOADVEC]], <2 x double> [[LOADVEC4]], <2 x i32> <i32 0, i32 3>
+; SSE-NEXT:    [[TMP6:%.*]] = fmul <2 x double> [[TMP5]], [[TMP1]]
+; SSE-NEXT:    [[TMP7:%.*]] = fadd <2 x double> [[TMP4]], [[TMP6]]
+; SSE-NEXT:    store <2 x double> [[TMP7]], ptr [[STOREARRAY:%.*]], align 8
 ; SSE-NEXT:    ret void
 ;
 ; AVX-LABEL: @ChecksExtractScores_different_vectors(
@@ -600,22 +593,16 @@ define void @ChecksExtractScores_different_vectors(ptr %storeArray, ptr %array,
 ; AVX-NEXT:    [[LOADA1:%.*]] = load double, ptr [[IDX1]], align 4
 ; AVX-NEXT:    [[LOADVEC:%.*]] = load <2 x double>, ptr [[VECPTR1:%.*]], align 4
 ; AVX-NEXT:    [[LOADVEC2:%.*]] = load <2 x double>, ptr [[VECPTR2:%.*]], align 4
-; AVX-NEXT:    [[EXTRA0:%.*]] = extractelement <2 x double> [[LOADVEC]], i32 0
-; AVX-NEXT:    [[EXTRA1:%.*]] = extractelement <2 x double> [[LOADVEC2]], i32 1
 ; AVX-NEXT:    [[LOADVEC3:%.*]] = load <2 x double>, ptr [[VECPTR3:%.*]], align 4
 ; AVX-NEXT:    [[LOADVEC4:%.*]] = load <2 x double>, ptr [[VECPTR4:%.*]], align 4
-; AVX-NEXT:    [[EXTRB0:%.*]] = extractelement <2 x double> [[LOADVEC3]], i32 0
-; AVX-NEXT:    [[EXTRB1:%.*]] = extractelement <2 x double> [[LOADVEC4]], i32 1
-; AVX-NEXT:    [[TMP1:%.*]] = insertelement <2 x double> poison, double [[EXTRA0]], i32 0
-; AVX-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP1]], double [[EXTRA1]], i32 1
-; AVX-NEXT:    [[TMP3:%.*]] = insertelement <2 x double> poison, double [[LOADA0]], i32 0
-; AVX-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x double> [[TMP3]], <2 x double> poison, <2 x i32> zeroinitializer
-; AVX-NEXT:    [[TMP4:%.*]] = fmul <2 x double> [[TMP2]], [[SHUFFLE]]
-; AVX-NEXT:    [[TMP5:%.*]] = insertelement <2 x double> poison, double [[EXTRB0]], i32 0
-; AVX-NEXT:    [[TMP6:%.*]] = insertelement <2 x double> [[TMP5]], double [[EXTRB1]], i32 1
-; AVX-NEXT:    [[TMP7:%.*]] = insertelement <2 x double> poison, double [[LOADA1]], i32 0
-; AVX-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <2 x double> [[TMP7]], <2 x double> poison, <2 x i32> zeroinitializer
-; AVX-NEXT:    [[TMP8:%.*]] = fmul <2 x double> [[TMP6]], [[SHUFFLE1]]
+; AVX-NEXT:    [[TMP1:%.*]] = shufflevector <2 x double> [[LOADVEC]], <2 x double> [[LOADVEC2]], <2 x i32> <i32 0, i32 3>
+; AVX-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[LOADA0]], i32 0
+; AVX-NEXT:    [[TMP3:%.*]] = shufflevector <2 x double> [[TMP2]], <2 x double> poison, <2 x i32> zeroinitializer
+; AVX-NEXT:    [[TMP4:%.*]] = fmul <2 x double> [[TMP1]], [[TMP3]]
+; AVX-NEXT:    [[TMP5:%.*]] = shufflevector <2 x double> [[LOADVEC3]], <2 x double> [[LOADVEC4]], <2 x i32> <i32 0, i32 3>
+; AVX-NEXT:    [[TMP6:%.*]] = insertelement <2 x double> poison, double [[LOADA1]], i32 0
+; AVX-NEXT:    [[TMP7:%.*]] = shufflevector <2 x double> [[TMP6]], <2 x double> poison, <2 x i32> zeroinitializer
+; AVX-NEXT:    [[TMP8:%.*]] = fmul <2 x double> [[TMP5]], [[TMP7]]
 ; AVX-NEXT:    [[TMP9:%.*]] = fadd <2 x double> [[TMP4]], [[TMP8]]
 ; AVX-NEXT:    store <2 x double> [[TMP9]], ptr [[STOREARRAY:%.*]], align 8
 ; AVX-NEXT:    ret void
@@ -651,15 +638,15 @@ define void @ChecksExtractScores_different_vectors(ptr %storeArray, ptr %array,
 define double @splat_loads(ptr %array1, ptr %array2, ptr %ptrA, ptr %ptrB) {
 ; SSE-LABEL: @splat_loads(
 ; SSE-NEXT:  entry:
-; SSE-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[ARRAY1:%.*]], align 8
-; SSE-NEXT:    [[TMP3:%.*]] = load <2 x double>, ptr [[ARRAY2:%.*]], align 8
-; SSE-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x double> [[TMP3]], <2 x double> poison, <2 x i32> <i32 1, i32 0>
-; SSE-NEXT:    [[TMP4:%.*]] = fmul <2 x double> [[TMP1]], [[SHUFFLE]]
-; SSE-NEXT:    [[TMP5:%.*]] = fmul <2 x double> [[TMP1]], [[TMP3]]
-; SSE-NEXT:    [[TMP6:%.*]] = fadd <2 x double> [[TMP4]], [[TMP5]]
-; SSE-NEXT:    [[TMP7:%.*]] = extractelement <2 x double> [[TMP6]], i32 0
-; SSE-NEXT:    [[TMP8:%.*]] = extractelement <2 x double> [[TMP6]], i32 1
-; SSE-NEXT:    [[ADD3:%.*]] = fadd double [[TMP7]], [[TMP8]]
+; SSE-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[ARRAY1:%.*]], align 8
+; SSE-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[ARRAY2:%.*]], align 8
+; SSE-NEXT:    [[TMP2:%.*]] = shufflevector <2 x double> [[TMP1]], <2 x double> poison, <2 x i32> <i32 1, i32 0>
+; SSE-NEXT:    [[TMP3:%.*]] = fmul <2 x double> [[TMP0]], [[TMP2]]
+; SSE-NEXT:    [[TMP4:%.*]] = fmul <2 x double> [[TMP0]], [[TMP1]]
+; SSE-NEXT:    [[TMP5:%.*]] = fadd <2 x double> [[TMP3]], [[TMP4]]
+; SSE-NEXT:    [[TMP6:%.*]] = extractelement <2 x double> [[TMP5]], i32 0
+; SSE-NEXT:    [[TMP7:%.*]] = extractelement <2 x double> [[TMP5]], i32 1
+; SSE-NEXT:    [[ADD3:%.*]] = fadd double [[TMP6]], [[TMP7]]
 ; SSE-NEXT:    ret double [[ADD3]]
 ;
 ; AVX-LABEL: @splat_loads(
@@ -667,17 +654,17 @@ define double @splat_loads(ptr %array1, ptr %array2, ptr %ptrA, ptr %ptrB) {
 ; AVX-NEXT:    [[GEP_2_1:%.*]] = getelementptr inbounds double, ptr [[ARRAY2:%.*]], i64 1
 ; AVX-NEXT:    [[LD_2_0:%.*]] = load double, ptr [[ARRAY2]], align 8
 ; AVX-NEXT:    [[LD_2_1:%.*]] = load double, ptr [[GEP_2_1]], align 8
-; AVX-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[ARRAY1:%.*]], align 8
-; AVX-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[LD_2_0]], i32 0
-; AVX-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x double> [[TMP2]], <2 x double> poison, <2 x i32> zeroinitializer
-; AVX-NEXT:    [[TMP3:%.*]] = fmul <2 x double> [[TMP1]], [[SHUFFLE]]
+; AVX-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[ARRAY1:%.*]], align 8
+; AVX-NEXT:    [[TMP1:%.*]] = insertelement <2 x double> poison, double [[LD_2_0]], i32 0
+; AVX-NEXT:    [[TMP2:%.*]] = shufflevector <2 x double> [[TMP1]], <2 x double> poison, <2 x i32> zeroinitializer
+; AVX-NEXT:    [[TMP3:%.*]] = fmul <2 x double> [[TMP0]], [[TMP2]]
 ; AVX-NEXT:    [[TMP4:%.*]] = insertelement <2 x double> poison, double [[LD_2_1]], i32 0
-; AVX-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <2 x double> [[TMP4]], <2 x double> poison, <2 x i32> zeroinitializer
-; AVX-NEXT:    [[TMP5:%.*]] = fmul <2 x double> [[TMP1]], [[SHUFFLE1]]
-; AVX-NEXT:    [[TMP6:%.*]] = fadd <2 x double> [[TMP3]], [[TMP5]]
-; AVX-NEXT:    [[TMP7:%.*]] = extractelement <2 x double> [[TMP6]], i32 0
-; AVX-NEXT:    [[TMP8:%.*]] = extractelement <2 x double> [[TMP6]], i32 1
-; AVX-NEXT:    [[ADD3:%.*]] = fadd double [[TMP7]], [[TMP8]]
+; AVX-NEXT:    [[TMP5:%.*]] = shufflevector <2 x double> [[TMP4]], <2 x double> poison, <2 x i32> zeroinitializer
+; AVX-NEXT:    [[TMP6:%.*]] = fmul <2 x double> [[TMP0]], [[TMP5]]
+; AVX-NEXT:    [[TMP7:%.*]] = fadd <2 x double> [[TMP3]], [[TMP6]]
+; AVX-NEXT:    [[TMP8:%.*]] = extractelement <2 x double> [[TMP7]], i32 0
+; AVX-NEXT:    [[TMP9:%.*]] = extractelement <2 x double> [[TMP7]], i32 1
+; AVX-NEXT:    [[ADD3:%.*]] = fadd double [[TMP8]], [[TMP9]]
 ; AVX-NEXT:    ret double [[ADD3]]
 ;
 entry:
@@ -707,14 +694,14 @@ entry:
 define double @splat_loads_with_internal_uses(ptr %array1, ptr %array2, ptr %ptrA, ptr %ptrB) {
 ; SSE-LABEL: @splat_loads_with_internal_uses(
 ; SSE-NEXT:  entry:
-; SSE-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[ARRAY1:%.*]], align 8
-; SSE-NEXT:    [[TMP3:%.*]] = load <2 x double>, ptr [[ARRAY2:%.*]], align 8
-; SSE-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x double> [[TMP3]], <2 x double> poison, <2 x i32> <i32 1, i32 0>
-; SSE-NEXT:    [[TMP4:%.*]] = fmul <2 x double> [[TMP1]], [[SHUFFLE]]
-; SSE-NEXT:    [[TMP5:%.*]] = fmul <2 x double> [[TMP1]], [[TMP3]]
-; SSE-NEXT:    [[TMP6:%.*]] = fadd <2 x double> [[TMP4]], [[TMP5]]
-; SSE-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <2 x double> [[TMP3]], <2 x double> poison, <2 x i32> zeroinitializer
-; SSE-NEXT:    [[TMP7:%.*]] = fsub <2 x double> [[TMP6]], [[SHUFFLE1]]
+; SSE-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[ARRAY1:%.*]], align 8
+; SSE-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[ARRAY2:%.*]], align 8
+; SSE-NEXT:    [[TMP2:%.*]] = shufflevector <2 x double> [[TMP1]], <2 x double> poison, <2 x i32> <i32 1, i32 0>
+; SSE-NEXT:    [[TMP3:%.*]] = fmul <2 x double> [[TMP0]], [[TMP2]]
+; SSE-NEXT:    [[TMP4:%.*]] = fmul <2 x double> [[TMP0]], [[TMP1]]
+; SSE-NEXT:    [[TMP5:%.*]] = fadd <2 x double> [[TMP3]], [[TMP4]]
+; SSE-NEXT:    [[TMP6:%.*]] = shufflevector <2 x double> [[TMP1]], <2 x double> poison, <2 x i32> zeroinitializer
+; SSE-NEXT:    [[TMP7:%.*]] = fsub <2 x double> [[TMP5]], [[TMP6]]
 ; SSE-NEXT:    [[TMP8:%.*]] = extractelement <2 x double> [[TMP7]], i32 0
 ; SSE-NEXT:    [[TMP9:%.*]] = extractelement <2 x double> [[TMP7]], i32 1
 ; SSE-NEXT:    [[RES:%.*]] = fadd double [[TMP8]], [[TMP9]]
@@ -725,18 +712,18 @@ define double @splat_loads_with_internal_uses(ptr %array1, ptr %array2, ptr %ptr
 ; AVX-NEXT:    [[GEP_2_1:%.*]] = getelementptr inbounds double, ptr [[ARRAY2:%.*]], i64 1
 ; AVX-NEXT:    [[LD_2_0:%.*]] = load double, ptr [[ARRAY2]], align 8
 ; AVX-NEXT:    [[LD_2_1:%.*]] = load double, ptr [[GEP_2_1]], align 8
-; AVX-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[ARRAY1:%.*]], align 8
-; AVX-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[LD_2_0]], i32 0
-; AVX-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x double> [[TMP2]], <2 x double> poison, <2 x i32> zeroinitializer
-; AVX-NEXT:    [[TMP3:%.*]] = fmul <2 x double> [[TMP1]], [[SHUFFLE]]
+; AVX-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[ARRAY1:%.*]], align 8
+; AVX-NEXT:    [[TMP1:%.*]] = insertelement <2 x double> poison, double [[LD_2_0]], i32 0
+; AVX-NEXT:    [[TMP2:%.*]] = shufflevector <2 x double> [[TMP1]], <2 x double> poison, <2 x i32> zeroinitializer
+; AVX-NEXT:    [[TMP3:%.*]] = fmul <2 x double> [[TMP0]], [[TMP2]]
 ; AVX-NEXT:    [[TMP4:%.*]] = insertelement <2 x double> poison, double [[LD_2_1]], i32 0
-; AVX-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <2 x double> [[TMP4]], <2 x double> poison, <2 x i32> zeroinitializer
-; AVX-NEXT:    [[TMP5:%.*]] = fmul <2 x double> [[TMP1]], [[SHUFFLE1]]
-; AVX-NEXT:    [[TMP6:%.*]] = fadd <2 x double> [[TMP3]], [[TMP5]]
-; AVX-NEXT:    [[TMP7:%.*]] = fsub <2 x double> [[TMP6]], [[SHUFFLE]]
-; AVX-NEXT:    [[TMP8:%.*]] = extractelement <2 x double> [[TMP7]], i32 0
-; AVX-NEXT:    [[TMP9:%.*]] = extractelement <2 x double> [[TMP7]], i32 1
-; AVX-NEXT:    [[RES:%.*]] = fadd double [[TMP8]], [[TMP9]]
+; AVX-NEXT:    [[TMP5:%.*]] = shufflevector <2 x double> [[TMP4]], <2 x double> poison, <2 x i32> zeroinitializer
+; AVX-NEXT:    [[TMP6:%.*]] = fmul <2 x double> [[TMP0]], [[TMP5]]
+; AVX-NEXT:    [[TMP7:%.*]] = fadd <2 x double> [[TMP3]], [[TMP6]]
+; AVX-NEXT:    [[TMP8:%.*]] = fsub <2 x double> [[TMP7]], [[TMP2]]
+; AVX-NEXT:    [[TMP9:%.*]] = extractelement <2 x double> [[TMP8]], i32 0
+; AVX-NEXT:    [[TMP10:%.*]] = extractelement <2 x double> [[TMP8]], i32 1
+; AVX-NEXT:    [[RES:%.*]] = fadd double [[TMP9]], [[TMP10]]
 ; AVX-NEXT:    ret double [[RES]]
 ;
 entry:
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/malformed_phis.ll b/llvm/test/Transforms/SLPVectorizer/X86/malformed_phis.ll
index 6cf4a233c2d2d..6000434ac1e95 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/malformed_phis.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/malformed_phis.ll
@@ -103,57 +103,40 @@ define i64 @test_3() #0 {
 ; CHECK:       bb2:
 ; CHECK-NEXT:    br label [[BB3]]
 ; CHECK:       bb3:
-; CHECK-NEXT:    [[TMP0:%.*]] = phi <2 x i32> [ undef, [[BB1]] ], [ poison, [[BB2:%.*]] ]
-; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <16 x i32> <i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 1>
-; CHECK-NEXT:    [[TMP2:%.*]] = extractelement <16 x i32> [[TMP1]], i32 0
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <32 x i32> poison, i32 [[TMP2]], i32 0
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <32 x i32> [[TMP3]], i32 [[TMP2]], i32 1
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <32 x i32> [[TMP4]], i32 [[TMP2]], i32 2
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <32 x i32> [[TMP5]], i32 [[TMP2]], i32 3
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <32 x i32> [[TMP6]], i32 [[TMP2]], i32 4
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <32 x i32> [[TMP7]], i32 [[TMP2]], i32 5
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <32 x i32> [[TMP8]], i32 [[TMP2]], i32 6
-; CHECK-NEXT:    [[TMP10:%.*]] = insertelement <32 x i32> [[TMP9]], i32 [[TMP2]], i32 7
-; CHECK-NEXT:    [[TMP11:%.*]] = insertelement <32 x i32> [[TMP10]], i32 [[TMP2]], i32 8
-; CHECK-NEXT:    [[TMP12:%.*]] = insertelement <32 x i32> [[TMP11]], i32 [[TMP2]], i32 9
-; CHECK-NEXT:    [[TMP13:%.*]] = insertelement <32 x i32> [[TMP12]], i32 [[TMP2]], i32 10
-; CHECK-NEXT:    [[TMP14:%.*]] = insertelement <32 x i32> [[TMP13]], i32 [[TMP2]], i32 11
-; CHECK-NEXT:    [[TMP15:%.*]] = insertelement <32 x i32> [[TMP14]], i32 [[TMP2]], i32 12
-; CHECK-NEXT:    [[TMP16:%.*]] = insertelement <32 x i32> [[TMP15]], i32 [[TMP2]], i32 13
-; CHECK-NEXT:    [[TMP17:%.*]] = insertelement <32 x i32> [[TMP16]], i32 [[TMP2]], i32 14
-; CHECK-NEXT:    [[TMP18:%.*]] = insertelement <32 x i32> [[TMP17]], i32 [[TMP2]], i32 15
-; CHECK-NEXT:    [[TMP19:%.*]] = insertelement <32 x i32> [[TMP18]], i32 [[TMP2]], i32 16
-; CHECK-NEXT:    [[TMP20:%.*]] = insertelement <32 x i32> [[TMP19]], i32 [[TMP2]], i32 17
-; CHECK-NEXT:    [[TMP21:%.*]] = insertelement <32 x i32> [[TMP20]], i32 [[TMP2]], i32 18
-; CHECK-NEXT:    [[TMP22:%.*]] = insertelement <32 x i32> [[TMP21]], i32 [[TMP2]], i32 19
-; CHECK-NEXT:    [[TMP23:%.*]] = insertelement <32 x i32> [[TMP22]], i32 [[TMP2]], i32 20
-; CHECK-NEXT:    [[TMP24:%.*]] = insertelement <32 x i32> [[TMP23]], i32 [[TMP2]], i32 21
-; CHECK-NEXT:    [[TMP25:%.*]] = insertelement <32 x i32> [[TMP24]], i32 [[TMP2]], i32 22
-; CHECK-NEXT:    [[TMP26:%.*]] = insertelement <32 x i32> [[TMP25]], i32 [[TMP2]], i32 23
-; CHECK-NEXT:    [[TMP27:%.*]] = insertelement <32 x i32> [[TMP26]], i32 [[TMP2]], i32 24
-; CHECK-NEXT:    [[TMP28:%.*]] = insertelement <32 x i32> [[TMP27]], i32 [[TMP2]], i32 25
-; CHECK-NEXT:    [[TMP29:%.*]] = insertelement <32 x i32> [[TMP28]], i32 [[TMP2]], i32 26
-; CHECK-NEXT:    [[TMP30:%.*]] = insertelement <32 x i32> [[TMP29]], i32 [[TMP2]], i32 27
-; CHECK-NEXT:    [[TMP31:%.*]] = insertelement <32 x i32> [[TMP30]], i32 [[TMP2]], i32 28
-; CHECK-NEXT:    [[TMP32:%.*]] = insertelement <32 x i32> [[TMP31]], i32 [[TMP2]], i32 29
-; CHECK-NEXT:    [[TMP33:%.*]] = insertelement <32 x i32> [[TMP32]], i32 [[TMP2]], i32 30
-; CHECK-NEXT:    [[TMP34:%.*]] = insertelement <32 x i32> [[TMP33]], i32 [[TMP2]], i32 31
-; CHECK-NEXT:    [[TMP35:%.*]] = call i32 @llvm.vector.reduce.mul.v32i32(<32 x i32> [[TMP34]])
-; CHECK-NEXT:    [[TMP36:%.*]] = call i32 @llvm.vector.reduce.mul.v16i32(<16 x i32> [[TMP1]])
-; CHECK-NEXT:    [[OP_RDX:%.*]] = mul i32 [[TMP35]], [[TMP36]]
-; CHECK-NEXT:    [[OP_RDX1:%.*]] = mul i32 [[OP_RDX]], [[TMP2]]
-; CHECK-NEXT:    [[OP_RDX2:%.*]] = mul i32 [[TMP2]], [[TMP2]]
-; CHECK-NEXT:    [[OP_RDX3:%.*]] = mul i32 [[TMP2]], [[TMP2]]
-; CHECK-NEXT:    [[OP_RDX4:%.*]] = mul i32 [[TMP2]], [[TMP2]]
-; CHECK-NEXT:    [[OP_RDX5:%.*]] = mul i32 [[TMP2]], [[TMP2]]
-; CHECK-NEXT:    [[OP_RDX6:%.*]] = mul i32 [[TMP2]], [[TMP2]]
-; CHECK-NEXT:    [[OP_RDX7:%.*]] = mul i32 [[OP_RDX1]], [[OP_RDX2]]
-; CHECK-NEXT:    [[OP_RDX8:%.*]] = mul i32 [[OP_RDX3]], [[OP_RDX4]]
-; CHECK-NEXT:    [[OP_RDX9:%.*]] = mul i32 [[OP_RDX5]], [[OP_RDX6]]
-; CHECK-NEXT:    [[OP_RDX10:%.*]] = mul i32 [[OP_RDX7]], [[OP_RDX8]]
-; CHECK-NEXT:    [[OP_RDX11:%.*]] = mul i32 [[OP_RDX9]], [[TMP2]]
-; CHECK-NEXT:    [[OP_RDX12:%.*]] = mul i32 [[OP_RDX10]], [[OP_RDX11]]
-; CHECK-NEXT:    [[VAL64:%.*]] = add i32 undef, [[OP_RDX12]]
+; CHECK-NEXT:    [[VAL:%.*]] = phi i32 [ undef, [[BB1]] ], [ undef, [[BB2:%.*]] ]
+; CHECK-NEXT:    [[VAL4:%.*]] = phi i32 [ undef, [[BB1]] ], [ undef, [[BB2]] ]
+; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <32 x i32> poison, i32 [[VAL4]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <32 x i32> [[TMP0]], <32 x i32> poison, <32 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP2:%.*]] = call i32 @llvm.vector.reduce.mul.v32i32(<32 x i32> [[TMP1]])
+; CHECK-NEXT:    [[OP_RDX:%.*]] = mul i32 [[TMP2]], [[VAL4]]
+; CHECK-NEXT:    [[OP_RDX1:%.*]] = mul i32 [[VAL4]], [[VAL4]]
+; CHECK-NEXT:    [[OP_RDX2:%.*]] = mul i32 [[VAL4]], [[VAL4]]
+; CHECK-NEXT:    [[OP_RDX3:%.*]] = mul i32 [[VAL4]], [[VAL4]]
+; CHECK-NEXT:    [[OP_RDX4:%.*]] = mul i32 [[VAL4]], [[VAL4]]
+; CHECK-NEXT:    [[OP_RDX5:%.*]] = mul i32 [[VAL4]], [[VAL4]]
+; CHECK-NEXT:    [[OP_RDX6:%.*]] = mul i32 [[VAL4]], [[VAL4]]
+; CHECK-NEXT:    [[OP_RDX7:%.*]] = mul i32 [[VAL4]], [[VAL4]]
+; CHECK-NEXT:    [[OP_RDX8:%.*]] = mul i32 [[VAL4]], [[VAL4]]
+; CHECK-NEXT:    [[OP_RDX9:%.*]] = mul i32 [[VAL4]], [[VAL4]]
+; CHECK-NEXT:    [[OP_RDX10:%.*]] = mul i32 [[VAL4]], [[VAL4]]
+; CHECK-NEXT:    [[OP_RDX11:%.*]] = mul i32 [[VAL4]], [[VAL4]]
+; CHECK-NEXT:    [[OP_RDX12:%.*]] = mul i32 [[VAL4]], [[VAL4]]
+; CHECK-NEXT:    [[OP_RDX13:%.*]] = mul i32 [[VAL4]], [[VAL4]]
+; CHECK-NEXT:    [[OP_RDX14:%.*]] = mul i32 [[OP_RDX]], [[OP_RDX1]]
+; CHECK-NEXT:    [[OP_RDX15:%.*]] = mul i32 [[OP_RDX2]], [[OP_RDX3]]
+; CHECK-NEXT:    [[OP_RDX16:%.*]] = mul i32 [[OP_RDX4]], [[OP_RDX5]]
+; CHECK-NEXT:    [[OP_RDX17:%.*]] = mul i32 [[OP_RDX6]], [[OP_RDX7]]
+; CHECK-NEXT:    [[OP_RDX18:%.*]] = mul i32 [[OP_RDX8]], [[OP_RDX9]]
+; CHECK-NEXT:    [[OP_RDX19:%.*]] = mul i32 [[OP_RDX10]], [[OP_RDX11]]
+; CHECK-NEXT:    [[OP_RDX20:%.*]] = mul i32 [[OP_RDX12]], [[OP_RDX13]]
+; CHECK-NEXT:    [[OP_RDX21:%.*]] = mul i32 [[OP_RDX14]], [[OP_RDX15]]
+; CHECK-NEXT:    [[OP_RDX22:%.*]] = mul i32 [[OP_RDX16]], [[OP_RDX17]]
+; CHECK-NEXT:    [[OP_RDX23:%.*]] = mul i32 [[OP_RDX18]], [[OP_RDX19]]
+; CHECK-NEXT:    [[OP_RDX24:%.*]] = mul i32 [[OP_RDX20]], [[VAL]]
+; CHECK-NEXT:    [[OP_RDX25:%.*]] = mul i32 [[OP_RDX21]], [[OP_RDX22]]
+; CHECK-NEXT:    [[OP_RDX26:%.*]] = mul i32 [[OP_RDX23]], [[OP_RDX24]]
+; CHECK-NEXT:    [[OP_RDX27:%.*]] = mul i32 [[OP_RDX25]], [[OP_RDX26]]
+; CHECK-NEXT:    [[VAL64:%.*]] = add i32 undef, [[OP_RDX27]]
 ; CHECK-NEXT:    [[VAL65:%.*]] = sext i32 [[VAL64]] to i64
 ; CHECK-NEXT:    ret i64 [[VAL65]]
 ;
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/matched-shuffled-entries.ll b/llvm/test/Transforms/SLPVectorizer/X86/matched-shuffled-entries.ll
index 118372d2d5898..a117fdc75bfdc 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/matched-shuffled-entries.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/matched-shuffled-entries.ll
@@ -15,23 +15,20 @@ define i32 @bar() local_unnamed_addr {
 ; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <16 x i32> [[TMP1]], i32 [[ADD78_1]], i32 6
 ; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <16 x i32> [[TMP2]], i32 [[SUB86_1]], i32 7
 ; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <16 x i32> [[TMP3]], i32 [[ADD78_2]], i32 9
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <16 x i32> [[TMP4]], <16 x i32> poison, <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 9, i32 11, i32 12, i32 13, i32 14, i32 15>
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <16 x i32> <i32 undef, i32 undef, i32 undef, i32 undef, i32 poison, i32 poison, i32 poison, i32 poison, i32 undef, i32 undef, i32 undef, i32 undef, i32 poison, i32 undef, i32 undef, i32 poison>, i32 [[SUB86_1]], i32 4
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <16 x i32> [[TMP5]], i32 [[ADD78_1]], i32 5
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <16 x i32> [[TMP6]], i32 [[ADD94_1]], i32 6
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <16 x i32> [[TMP7]], i32 [[SUB102_1]], i32 7
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <16 x i32> [[TMP8]], i32 [[SUB102_3]], i32 12
-; CHECK-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <16 x i32> [[TMP9]], <16 x i32> poison, <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 12>
-; CHECK-NEXT:    [[TMP10:%.*]] = add nsw <16 x i32> [[SHUFFLE]], [[SHUFFLE1]]
-; CHECK-NEXT:    [[TMP11:%.*]] = sub nsw <16 x i32> [[SHUFFLE]], [[SHUFFLE1]]
-; CHECK-NEXT:    [[TMP12:%.*]] = shufflevector <16 x i32> [[TMP10]], <16 x i32> [[TMP11]], <16 x i32> <i32 0, i32 1, i32 18, i32 19, i32 4, i32 5, i32 22, i32 23, i32 8, i32 9, i32 26, i32 27, i32 12, i32 13, i32 30, i32 31>
-; CHECK-NEXT:    [[TMP13:%.*]] = lshr <16 x i32> [[TMP12]], <i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15>
-; CHECK-NEXT:    [[TMP14:%.*]] = and <16 x i32> [[TMP13]], <i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537>
-; CHECK-NEXT:    [[TMP15:%.*]] = mul nuw <16 x i32> [[TMP14]], <i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535>
-; CHECK-NEXT:    [[TMP16:%.*]] = add <16 x i32> [[TMP15]], [[TMP12]]
-; CHECK-NEXT:    [[TMP17:%.*]] = xor <16 x i32> [[TMP16]], [[TMP15]]
-; CHECK-NEXT:    [[TMP18:%.*]] = call i32 @llvm.vector.reduce.add.v16i32(<16 x i32> [[TMP17]])
-; CHECK-NEXT:    [[SHR:%.*]] = lshr i32 [[TMP18]], 16
+; CHECK-NEXT:    [[TMP5:%.*]] = shufflevector <16 x i32> [[TMP4]], <16 x i32> poison, <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 9, i32 11, i32 12, i32 13, i32 14, i32 15>
+; CHECK-NEXT:    [[TMP6:%.*]] = shufflevector <16 x i32> [[TMP4]], <16 x i32> poison, <16 x i32> <i32 undef, i32 undef, i32 undef, i32 undef, i32 7, i32 6, i32 5, i32 4, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <16 x i32> [[TMP6]], i32 [[SUB102_3]], i32 12
+; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <16 x i32> [[TMP7]], i32 [[SUB102_3]], i32 15
+; CHECK-NEXT:    [[TMP9:%.*]] = add nsw <16 x i32> [[TMP5]], [[TMP8]]
+; CHECK-NEXT:    [[TMP10:%.*]] = sub nsw <16 x i32> [[TMP5]], [[TMP8]]
+; CHECK-NEXT:    [[TMP11:%.*]] = shufflevector <16 x i32> [[TMP9]], <16 x i32> [[TMP10]], <16 x i32> <i32 0, i32 1, i32 18, i32 19, i32 4, i32 5, i32 22, i32 23, i32 8, i32 9, i32 26, i32 27, i32 12, i32 13, i32 30, i32 31>
+; CHECK-NEXT:    [[TMP12:%.*]] = lshr <16 x i32> [[TMP11]], <i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15, i32 15>
+; CHECK-NEXT:    [[TMP13:%.*]] = and <16 x i32> [[TMP12]], <i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537, i32 65537>
+; CHECK-NEXT:    [[TMP14:%.*]] = mul nuw <16 x i32> [[TMP13]], <i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535>
+; CHECK-NEXT:    [[TMP15:%.*]] = add <16 x i32> [[TMP14]], [[TMP11]]
+; CHECK-NEXT:    [[TMP16:%.*]] = xor <16 x i32> [[TMP15]], [[TMP14]]
+; CHECK-NEXT:    [[TMP17:%.*]] = call i32 @llvm.vector.reduce.add.v16i32(<16 x i32> [[TMP16]])
+; CHECK-NEXT:    [[SHR:%.*]] = lshr i32 [[TMP17]], 16
 ; CHECK-NEXT:    [[ADD119:%.*]] = add nuw nsw i32 undef, [[SHR]]
 ; CHECK-NEXT:    [[SHR120:%.*]] = lshr i32 [[ADD119]], 1
 ; CHECK-NEXT:    ret i32 [[SHR120]]
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll b/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
index d417dd275d62f..e5947dcdd0825 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
@@ -187,9 +187,9 @@ define void @gather_sequence_crash(<2 x float> %arg, ptr %arg1, float %arg2, ptr
 ; CHECK-NEXT:    br i1 [[C_1:%.*]], label [[BB16:%.*]], label [[BB6:%.*]]
 ; CHECK:       bb6:
 ; CHECK-NEXT:    [[TMP8:%.*]] = getelementptr inbounds float, ptr [[ARG1:%.*]], i32 3
-; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <4 x float> <float poison, float poison, float poison, float 0.000000e+00>, float [[ARG2:%.*]], i32 0
-; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <2 x float> [[ARG:%.*]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <4 x float> [[TMP0]], <4 x float> [[TMP1]], <4 x i32> <i32 0, i32 4, i32 5, i32 3>
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <2 x float> [[ARG:%.*]], <2 x float> poison, <4 x i32> <i32 undef, i32 0, i32 1, i32 undef>
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x float> [[TMP0]], <4 x float> <float poison, float poison, float poison, float 0.000000e+00>, <4 x i32> <i32 undef, i32 1, i32 2, i32 7>
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <4 x float> [[TMP1]], float [[ARG2:%.*]], i32 0
 ; CHECK-NEXT:    [[TMP3:%.*]] = fmul <4 x float> [[TMP2]], zeroinitializer
 ; CHECK-NEXT:    store <4 x float> [[TMP3]], ptr [[TMP8]], align 4
 ; CHECK-NEXT:    ret void
@@ -226,9 +226,9 @@ define void @gather_sequence_crash(<2 x float> %arg, ptr %arg1, float %arg2, ptr
 ; CHECK-NEXT:    [[TMP38:%.*]] = fadd float 0.000000e+00, [[TMP37]]
 ; CHECK-NEXT:    store float [[TMP38]], ptr [[TMP35]], align 4
 ; CHECK-NEXT:    [[TMP39:%.*]] = getelementptr float, ptr [[ARG4]], i64 1
-; CHECK-NEXT:    [[TMP7:%.*]] = load <2 x float>, ptr [[TMP39]], align 4
-; CHECK-NEXT:    [[TMP8:%.*]] = fadd <2 x float> zeroinitializer, [[TMP7]]
-; CHECK-NEXT:    store <2 x float> [[TMP8]], ptr [[TMP39]], align 4
+; CHECK-NEXT:    [[TMP4:%.*]] = load <2 x float>, ptr [[TMP39]], align 4
+; CHECK-NEXT:    [[TMP5:%.*]] = fadd <2 x float> zeroinitializer, [[TMP4]]
+; CHECK-NEXT:    store <2 x float> [[TMP5]], ptr [[TMP39]], align 4
 ; CHECK-NEXT:    [[TMP44:%.*]] = load float, ptr [[ARG3:%.*]], align 4
 ; CHECK-NEXT:    [[TMP45:%.*]] = load float, ptr [[ARG4]], align 4
 ; CHECK-NEXT:    [[TMP46:%.*]] = fadd float 0.000000e+00, [[TMP45]]
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/odd_store.ll b/llvm/test/Transforms/SLPVectorizer/X86/odd_store.ll
index 32ad5fd80cf57..4795ac6559203 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/odd_store.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/odd_store.ll
@@ -63,11 +63,8 @@ define i32 @foo(ptr noalias nocapture %A, ptr noalias nocapture %B, float %T) {
 ; PR41892
 define void @test_v4f32_v2f32_store(<4 x float> %f, ptr %p){
 ; CHECK-LABEL: @test_v4f32_v2f32_store(
-; CHECK-NEXT:    [[X0:%.*]] = extractelement <4 x float> [[F:%.*]], i64 0
-; CHECK-NEXT:    [[X1:%.*]] = extractelement <4 x float> [[F]], i64 1
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x float> poison, float [[X0]], i32 0
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x float> [[TMP1]], float [[X1]], i32 1
-; CHECK-NEXT:    store <2 x float> [[TMP2]], ptr [[P:%.*]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x float> [[F:%.*]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    store <2 x float> [[TMP1]], ptr [[P:%.*]], align 4
 ; CHECK-NEXT:    ret void
 ;
   %x0 = extractelement <4 x float> %f, i64 0
@@ -95,13 +92,10 @@ define void @test_v4f32_v2f32_splat_store(<4 x float> %f, ptr %p){
 
 define void @test_v4f32_v3f32_store(<4 x float> %f, ptr %p){
 ; CHECK-LABEL: @test_v4f32_v3f32_store(
-; CHECK-NEXT:    [[X0:%.*]] = extractelement <4 x float> [[F:%.*]], i64 0
-; CHECK-NEXT:    [[X1:%.*]] = extractelement <4 x float> [[F]], i64 1
-; CHECK-NEXT:    [[X2:%.*]] = extractelement <4 x float> [[F]], i64 2
+; CHECK-NEXT:    [[X2:%.*]] = extractelement <4 x float> [[F:%.*]], i64 2
 ; CHECK-NEXT:    [[P2:%.*]] = getelementptr inbounds float, ptr [[P:%.*]], i64 2
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x float> poison, float [[X0]], i32 0
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x float> [[TMP1]], float [[X1]], i32 1
-; CHECK-NEXT:    store <2 x float> [[TMP2]], ptr [[P]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x float> [[F]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    store <2 x float> [[TMP1]], ptr [[P]], align 4
 ; CHECK-NEXT:    store float [[X2]], ptr [[P2]], align 4
 ; CHECK-NEXT:    ret void
 ;
@@ -156,10 +150,8 @@ define void @test_v4f32_v4f32_store(<4 x float> %f, ptr %p){
 
 define void @test_v4f32_v4f32_splat_store(<4 x float> %f, ptr %p){
 ; CHECK-LABEL: @test_v4f32_v4f32_splat_store(
-; CHECK-NEXT:    [[X0:%.*]] = extractelement <4 x float> [[F:%.*]], i64 0
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <4 x float> poison, float [[X0]], i32 0
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <4 x float> [[TMP1]], <4 x float> poison, <4 x i32> zeroinitializer
-; CHECK-NEXT:    store <4 x float> [[SHUFFLE]], ptr [[P:%.*]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x float> [[F:%.*]], <4 x float> poison, <4 x i32> zeroinitializer
+; CHECK-NEXT:    store <4 x float> [[TMP1]], ptr [[P:%.*]], align 4
 ; CHECK-NEXT:    ret void
 ;
   %x0 = extractelement <4 x float> %f, i64 0
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/operandorder.ll b/llvm/test/Transforms/SLPVectorizer/X86/operandorder.ll
index 5bac13d87b48f..359c074d6a211 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/operandorder.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/operandorder.ll
@@ -9,19 +9,19 @@ target datalayout = "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f3
 
 define void @shuffle_operands1(ptr noalias %from, ptr noalias %to, double %v1, double %v2) {
 ; CHECK-LABEL: @shuffle_operands1(
-; CHECK-NEXT:    [[TMP2:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x double> poison, double [[V1:%.*]], i64 0
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x double> [[TMP3]], double [[V2:%.*]], i64 1
-; CHECK-NEXT:    [[TMP5:%.*]] = fadd <2 x double> [[TMP2]], [[TMP4]]
-; CHECK-NEXT:    store <2 x double> [[TMP5]], ptr [[TO:%.*]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[V1:%.*]], i64 0
+; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x double> [[TMP2]], double [[V2:%.*]], i64 1
+; CHECK-NEXT:    [[TMP4:%.*]] = fadd <2 x double> [[TMP1]], [[TMP3]]
+; CHECK-NEXT:    store <2 x double> [[TMP4]], ptr [[TO:%.*]], align 4
 ; CHECK-NEXT:    ret void
 ;
 ; SSE2-LABEL: @shuffle_operands1(
-; SSE2-NEXT:    [[TMP2:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
-; SSE2-NEXT:    [[TMP3:%.*]] = insertelement <2 x double> poison, double [[V1:%.*]], i64 0
-; SSE2-NEXT:    [[TMP4:%.*]] = insertelement <2 x double> [[TMP3]], double [[V2:%.*]], i64 1
-; SSE2-NEXT:    [[TMP5:%.*]] = fadd <2 x double> [[TMP2]], [[TMP4]]
-; SSE2-NEXT:    store <2 x double> [[TMP5]], ptr [[TO:%.*]], align 4
+; SSE2-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
+; SSE2-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[V1:%.*]], i64 0
+; SSE2-NEXT:    [[TMP3:%.*]] = insertelement <2 x double> [[TMP2]], double [[V2:%.*]], i64 1
+; SSE2-NEXT:    [[TMP4:%.*]] = fadd <2 x double> [[TMP1]], [[TMP3]]
+; SSE2-NEXT:    store <2 x double> [[TMP4]], ptr [[TO:%.*]], align 4
 ; SSE2-NEXT:    ret void
 ;
   %from_1 = getelementptr double, ptr %from, i64 1
@@ -41,11 +41,11 @@ define void @vecload_vs_broadcast(ptr noalias %from, ptr noalias %to, double %v1
 ; CHECK-NEXT:    br label [[LP:%.*]]
 ; CHECK:       lp:
 ; CHECK-NEXT:    [[P:%.*]] = phi double [ 1.000000e+00, [[LP]] ], [ 0.000000e+00, [[ENTRY:%.*]] ]
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[P]], i64 0
-; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <2 x double> [[TMP2]], <2 x double> [[TMP1]], <2 x i32> <i32 0, i32 2>
-; CHECK-NEXT:    [[TMP4:%.*]] = fadd <2 x double> [[TMP1]], [[TMP3]]
-; CHECK-NEXT:    store <2 x double> [[TMP4]], ptr [[TO:%.*]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <2 x double> [[TMP0]], <2 x double> poison, <2 x i32> <i32 undef, i32 0>
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP1]], double [[P]], i64 0
+; CHECK-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP0]], [[TMP2]]
+; CHECK-NEXT:    store <2 x double> [[TMP3]], ptr [[TO:%.*]], align 4
 ; CHECK-NEXT:    br i1 undef, label [[LP]], label [[EXT:%.*]]
 ; CHECK:       ext:
 ; CHECK-NEXT:    ret void
@@ -55,11 +55,11 @@ define void @vecload_vs_broadcast(ptr noalias %from, ptr noalias %to, double %v1
 ; SSE2-NEXT:    br label [[LP:%.*]]
 ; SSE2:       lp:
 ; SSE2-NEXT:    [[P:%.*]] = phi double [ 1.000000e+00, [[LP]] ], [ 0.000000e+00, [[ENTRY:%.*]] ]
-; SSE2-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
-; SSE2-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[P]], i64 0
-; SSE2-NEXT:    [[TMP3:%.*]] = shufflevector <2 x double> [[TMP2]], <2 x double> [[TMP1]], <2 x i32> <i32 0, i32 2>
-; SSE2-NEXT:    [[TMP4:%.*]] = fadd <2 x double> [[TMP1]], [[TMP3]]
-; SSE2-NEXT:    store <2 x double> [[TMP4]], ptr [[TO:%.*]], align 4
+; SSE2-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
+; SSE2-NEXT:    [[TMP1:%.*]] = shufflevector <2 x double> [[TMP0]], <2 x double> poison, <2 x i32> <i32 undef, i32 0>
+; SSE2-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP1]], double [[P]], i64 0
+; SSE2-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP0]], [[TMP2]]
+; SSE2-NEXT:    store <2 x double> [[TMP3]], ptr [[TO:%.*]], align 4
 ; SSE2-NEXT:    br i1 undef, label [[LP]], label [[EXT:%.*]]
 ; SSE2:       ext:
 ; SSE2-NEXT:    ret void
@@ -89,11 +89,11 @@ define void @vecload_vs_broadcast2(ptr noalias %from, ptr noalias %to, double %v
 ; CHECK-NEXT:    br label [[LP:%.*]]
 ; CHECK:       lp:
 ; CHECK-NEXT:    [[P:%.*]] = phi double [ 1.000000e+00, [[LP]] ], [ 0.000000e+00, [[ENTRY:%.*]] ]
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[P]], i64 0
-; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <2 x double> [[TMP2]], <2 x double> [[TMP1]], <2 x i32> <i32 0, i32 2>
-; CHECK-NEXT:    [[TMP4:%.*]] = fadd <2 x double> [[TMP3]], [[TMP1]]
-; CHECK-NEXT:    store <2 x double> [[TMP4]], ptr [[TO:%.*]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <2 x double> [[TMP0]], <2 x double> poison, <2 x i32> <i32 undef, i32 0>
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP1]], double [[P]], i64 0
+; CHECK-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP2]], [[TMP0]]
+; CHECK-NEXT:    store <2 x double> [[TMP3]], ptr [[TO:%.*]], align 4
 ; CHECK-NEXT:    br i1 undef, label [[LP]], label [[EXT:%.*]]
 ; CHECK:       ext:
 ; CHECK-NEXT:    ret void
@@ -103,11 +103,11 @@ define void @vecload_vs_broadcast2(ptr noalias %from, ptr noalias %to, double %v
 ; SSE2-NEXT:    br label [[LP:%.*]]
 ; SSE2:       lp:
 ; SSE2-NEXT:    [[P:%.*]] = phi double [ 1.000000e+00, [[LP]] ], [ 0.000000e+00, [[ENTRY:%.*]] ]
-; SSE2-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
-; SSE2-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[P]], i64 0
-; SSE2-NEXT:    [[TMP3:%.*]] = shufflevector <2 x double> [[TMP2]], <2 x double> [[TMP1]], <2 x i32> <i32 0, i32 2>
-; SSE2-NEXT:    [[TMP4:%.*]] = fadd <2 x double> [[TMP3]], [[TMP1]]
-; SSE2-NEXT:    store <2 x double> [[TMP4]], ptr [[TO:%.*]], align 4
+; SSE2-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
+; SSE2-NEXT:    [[TMP1:%.*]] = shufflevector <2 x double> [[TMP0]], <2 x double> poison, <2 x i32> <i32 undef, i32 0>
+; SSE2-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP1]], double [[P]], i64 0
+; SSE2-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP2]], [[TMP0]]
+; SSE2-NEXT:    store <2 x double> [[TMP3]], ptr [[TO:%.*]], align 4
 ; SSE2-NEXT:    br i1 undef, label [[LP]], label [[EXT:%.*]]
 ; SSE2:       ext:
 ; SSE2-NEXT:    ret void
@@ -137,11 +137,11 @@ define void @vecload_vs_broadcast3(ptr noalias %from, ptr noalias %to, double %v
 ; CHECK-NEXT:    br label [[LP:%.*]]
 ; CHECK:       lp:
 ; CHECK-NEXT:    [[P:%.*]] = phi double [ 1.000000e+00, [[LP]] ], [ 0.000000e+00, [[ENTRY:%.*]] ]
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[P]], i64 0
-; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <2 x double> [[TMP2]], <2 x double> [[TMP1]], <2 x i32> <i32 0, i32 2>
-; CHECK-NEXT:    [[TMP4:%.*]] = fadd <2 x double> [[TMP3]], [[TMP1]]
-; CHECK-NEXT:    store <2 x double> [[TMP4]], ptr [[TO:%.*]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <2 x double> [[TMP0]], <2 x double> poison, <2 x i32> <i32 undef, i32 0>
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP1]], double [[P]], i64 0
+; CHECK-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP2]], [[TMP0]]
+; CHECK-NEXT:    store <2 x double> [[TMP3]], ptr [[TO:%.*]], align 4
 ; CHECK-NEXT:    br i1 undef, label [[LP]], label [[EXT:%.*]]
 ; CHECK:       ext:
 ; CHECK-NEXT:    ret void
@@ -151,11 +151,11 @@ define void @vecload_vs_broadcast3(ptr noalias %from, ptr noalias %to, double %v
 ; SSE2-NEXT:    br label [[LP:%.*]]
 ; SSE2:       lp:
 ; SSE2-NEXT:    [[P:%.*]] = phi double [ 1.000000e+00, [[LP]] ], [ 0.000000e+00, [[ENTRY:%.*]] ]
-; SSE2-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
-; SSE2-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[P]], i64 0
-; SSE2-NEXT:    [[TMP3:%.*]] = shufflevector <2 x double> [[TMP2]], <2 x double> [[TMP1]], <2 x i32> <i32 0, i32 2>
-; SSE2-NEXT:    [[TMP4:%.*]] = fadd <2 x double> [[TMP3]], [[TMP1]]
-; SSE2-NEXT:    store <2 x double> [[TMP4]], ptr [[TO:%.*]], align 4
+; SSE2-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
+; SSE2-NEXT:    [[TMP1:%.*]] = shufflevector <2 x double> [[TMP0]], <2 x double> poison, <2 x i32> <i32 undef, i32 0>
+; SSE2-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP1]], double [[P]], i64 0
+; SSE2-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP2]], [[TMP0]]
+; SSE2-NEXT:    store <2 x double> [[TMP3]], ptr [[TO:%.*]], align 4
 ; SSE2-NEXT:    br i1 undef, label [[LP]], label [[EXT:%.*]]
 ; SSE2:       ext:
 ; SSE2-NEXT:    ret void
@@ -185,10 +185,10 @@ define void @shuffle_nodes_match1(ptr noalias %from, ptr noalias %to, double %v1
 ; CHECK-NEXT:    br label [[LP:%.*]]
 ; CHECK:       lp:
 ; CHECK-NEXT:    [[P:%.*]] = phi double [ 1.000000e+00, [[LP]] ], [ 0.000000e+00, [[ENTRY:%.*]] ]
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x double> [[TMP1]], <2 x double> poison, <2 x i32> <i32 1, i32 0>
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP1]], double [[P]], i64 1
-; CHECK-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP2]], [[SHUFFLE]]
+; CHECK-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <2 x double> [[TMP0]], <2 x double> poison, <2 x i32> <i32 1, i32 0>
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP0]], double [[P]], i64 1
+; CHECK-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP2]], [[TMP1]]
 ; CHECK-NEXT:    store <2 x double> [[TMP3]], ptr [[TO:%.*]], align 4
 ; CHECK-NEXT:    br i1 undef, label [[LP]], label [[EXT:%.*]]
 ; CHECK:       ext:
@@ -199,10 +199,10 @@ define void @shuffle_nodes_match1(ptr noalias %from, ptr noalias %to, double %v1
 ; SSE2-NEXT:    br label [[LP:%.*]]
 ; SSE2:       lp:
 ; SSE2-NEXT:    [[P:%.*]] = phi double [ 1.000000e+00, [[LP]] ], [ 0.000000e+00, [[ENTRY:%.*]] ]
-; SSE2-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
-; SSE2-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x double> [[TMP1]], <2 x double> poison, <2 x i32> <i32 1, i32 0>
-; SSE2-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP1]], double [[P]], i64 1
-; SSE2-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP2]], [[SHUFFLE]]
+; SSE2-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
+; SSE2-NEXT:    [[TMP1:%.*]] = shufflevector <2 x double> [[TMP0]], <2 x double> poison, <2 x i32> <i32 1, i32 0>
+; SSE2-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP0]], double [[P]], i64 1
+; SSE2-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP2]], [[TMP1]]
 ; SSE2-NEXT:    store <2 x double> [[TMP3]], ptr [[TO:%.*]], align 4
 ; SSE2-NEXT:    br i1 undef, label [[LP]], label [[EXT:%.*]]
 ; SSE2:       ext:
@@ -251,10 +251,10 @@ define void @vecload_vs_broadcast4(ptr noalias %from, ptr noalias %to, double %v
 ; SSE2-NEXT:    br label [[LP:%.*]]
 ; SSE2:       lp:
 ; SSE2-NEXT:    [[P:%.*]] = phi double [ 1.000000e+00, [[LP]] ], [ 0.000000e+00, [[ENTRY:%.*]] ]
-; SSE2-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
-; SSE2-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x double> [[TMP1]], <2 x double> poison, <2 x i32> <i32 1, i32 0>
-; SSE2-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP1]], double [[P]], i64 1
-; SSE2-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP2]], [[SHUFFLE]]
+; SSE2-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
+; SSE2-NEXT:    [[TMP1:%.*]] = shufflevector <2 x double> [[TMP0]], <2 x double> poison, <2 x i32> <i32 1, i32 0>
+; SSE2-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP0]], double [[P]], i64 1
+; SSE2-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP2]], [[TMP1]]
 ; SSE2-NEXT:    store <2 x double> [[TMP3]], ptr [[TO:%.*]], align 4
 ; SSE2-NEXT:    br i1 undef, label [[LP]], label [[EXT:%.*]]
 ; SSE2:       ext:
@@ -304,10 +304,10 @@ define void @shuffle_nodes_match2(ptr noalias %from, ptr noalias %to, double %v1
 ; SSE2-NEXT:    br label [[LP:%.*]]
 ; SSE2:       lp:
 ; SSE2-NEXT:    [[P:%.*]] = phi double [ 1.000000e+00, [[LP]] ], [ 0.000000e+00, [[ENTRY:%.*]] ]
-; SSE2-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
-; SSE2-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x double> [[TMP1]], <2 x double> poison, <2 x i32> <i32 1, i32 0>
-; SSE2-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP1]], double [[P]], i64 1
-; SSE2-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[SHUFFLE]], [[TMP2]]
+; SSE2-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[FROM:%.*]], align 4
+; SSE2-NEXT:    [[TMP1:%.*]] = shufflevector <2 x double> [[TMP0]], <2 x double> poison, <2 x i32> <i32 1, i32 0>
+; SSE2-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP0]], double [[P]], i64 1
+; SSE2-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP1]], [[TMP2]]
 ; SSE2-NEXT:    store <2 x double> [[TMP3]], ptr [[TO:%.*]], align 4
 ; SSE2-NEXT:    br i1 undef, label [[LP]], label [[EXT:%.*]]
 ; SSE2:       ext:
@@ -345,7 +345,7 @@ define void @good_load_order() {
 ; CHECK-NEXT:    [[TMP0:%.*]] = load float, ptr @a, align 16
 ; CHECK-NEXT:    br label [[FOR_BODY3:%.*]]
 ; CHECK:       for.body3:
-; CHECK-NEXT:    [[TMP1:%.*]] = phi float [ [[TMP0]], [[FOR_COND1_PREHEADER]] ], [ [[TMP14:%.*]], [[FOR_BODY3]] ]
+; CHECK-NEXT:    [[TMP1:%.*]] = phi float [ [[TMP0]], [[FOR_COND1_PREHEADER]] ], [ [[TMP12:%.*]], [[FOR_BODY3]] ]
 ; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[FOR_COND1_PREHEADER]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY3]] ]
 ; CHECK-NEXT:    [[TMP2:%.*]] = trunc i64 [[INDVARS_IV]] to i32
 ; CHECK-NEXT:    [[TMP3:%.*]] = add i32 [[TMP2]], 1
@@ -355,20 +355,20 @@ define void @good_load_order() {
 ; CHECK-NEXT:    [[TMP5:%.*]] = trunc i64 [[INDVARS_IV]] to i32
 ; CHECK-NEXT:    [[TMP6:%.*]] = add i32 [[TMP5]], 4
 ; CHECK-NEXT:    [[ARRAYIDX31:%.*]] = getelementptr inbounds [32000 x float], ptr @a, i32 0, i32 [[TMP6]]
-; CHECK-NEXT:    [[TMP8:%.*]] = load <4 x float>, ptr [[ARRAYIDX]], align 4
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <4 x float> poison, float [[TMP1]], i64 0
-; CHECK-NEXT:    [[TMP10:%.*]] = shufflevector <4 x float> [[TMP9]], <4 x float> [[TMP8]], <4 x i32> <i32 0, i32 4, i32 5, i32 6>
-; CHECK-NEXT:    [[TMP11:%.*]] = fmul <4 x float> [[TMP8]], [[TMP10]]
-; CHECK-NEXT:    store <4 x float> [[TMP11]], ptr [[ARRAYIDX5]], align 4
+; CHECK-NEXT:    [[TMP7:%.*]] = load <4 x float>, ptr [[ARRAYIDX]], align 4
+; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <4 x float> [[TMP7]], <4 x float> poison, <4 x i32> <i32 undef, i32 0, i32 1, i32 2>
+; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <4 x float> [[TMP8]], float [[TMP1]], i64 0
+; CHECK-NEXT:    [[TMP10:%.*]] = fmul <4 x float> [[TMP7]], [[TMP9]]
+; CHECK-NEXT:    store <4 x float> [[TMP10]], ptr [[ARRAYIDX5]], align 4
 ; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nuw nsw i64 [[INDVARS_IV]], 5
-; CHECK-NEXT:    [[TMP13:%.*]] = trunc i64 [[INDVARS_IV_NEXT]] to i32
-; CHECK-NEXT:    [[ARRAYIDX41:%.*]] = getelementptr inbounds [32000 x float], ptr @a, i32 0, i32 [[TMP13]]
-; CHECK-NEXT:    [[TMP14]] = load float, ptr [[ARRAYIDX41]], align 4
-; CHECK-NEXT:    [[TMP15:%.*]] = extractelement <4 x float> [[TMP8]], i64 3
-; CHECK-NEXT:    [[MUL45:%.*]] = fmul float [[TMP14]], [[TMP15]]
+; CHECK-NEXT:    [[TMP11:%.*]] = trunc i64 [[INDVARS_IV_NEXT]] to i32
+; CHECK-NEXT:    [[ARRAYIDX41:%.*]] = getelementptr inbounds [32000 x float], ptr @a, i32 0, i32 [[TMP11]]
+; CHECK-NEXT:    [[TMP12]] = load float, ptr [[ARRAYIDX41]], align 4
+; CHECK-NEXT:    [[TMP13:%.*]] = extractelement <4 x float> [[TMP7]], i64 3
+; CHECK-NEXT:    [[MUL45:%.*]] = fmul float [[TMP12]], [[TMP13]]
 ; CHECK-NEXT:    store float [[MUL45]], ptr [[ARRAYIDX31]], align 4
-; CHECK-NEXT:    [[TMP16:%.*]] = trunc i64 [[INDVARS_IV_NEXT]] to i32
-; CHECK-NEXT:    [[CMP2:%.*]] = icmp slt i32 [[TMP16]], 31995
+; CHECK-NEXT:    [[TMP14:%.*]] = trunc i64 [[INDVARS_IV_NEXT]] to i32
+; CHECK-NEXT:    [[CMP2:%.*]] = icmp slt i32 [[TMP14]], 31995
 ; CHECK-NEXT:    br i1 [[CMP2]], label [[FOR_BODY3]], label [[FOR_END:%.*]]
 ; CHECK:       for.end:
 ; CHECK-NEXT:    ret void
@@ -380,7 +380,7 @@ define void @good_load_order() {
 ; SSE2-NEXT:    [[TMP0:%.*]] = load float, ptr @a, align 16
 ; SSE2-NEXT:    br label [[FOR_BODY3:%.*]]
 ; SSE2:       for.body3:
-; SSE2-NEXT:    [[TMP1:%.*]] = phi float [ [[TMP0]], [[FOR_COND1_PREHEADER]] ], [ [[TMP14:%.*]], [[FOR_BODY3]] ]
+; SSE2-NEXT:    [[TMP1:%.*]] = phi float [ [[TMP0]], [[FOR_COND1_PREHEADER]] ], [ [[TMP12:%.*]], [[FOR_BODY3]] ]
 ; SSE2-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[FOR_COND1_PREHEADER]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY3]] ]
 ; SSE2-NEXT:    [[TMP2:%.*]] = trunc i64 [[INDVARS_IV]] to i32
 ; SSE2-NEXT:    [[TMP3:%.*]] = add i32 [[TMP2]], 1
@@ -390,20 +390,20 @@ define void @good_load_order() {
 ; SSE2-NEXT:    [[TMP5:%.*]] = trunc i64 [[INDVARS_IV]] to i32
 ; SSE2-NEXT:    [[TMP6:%.*]] = add i32 [[TMP5]], 4
 ; SSE2-NEXT:    [[ARRAYIDX31:%.*]] = getelementptr inbounds [32000 x float], ptr @a, i32 0, i32 [[TMP6]]
-; SSE2-NEXT:    [[TMP8:%.*]] = load <4 x float>, ptr [[ARRAYIDX]], align 4
-; SSE2-NEXT:    [[TMP9:%.*]] = insertelement <4 x float> poison, float [[TMP1]], i64 0
-; SSE2-NEXT:    [[TMP10:%.*]] = shufflevector <4 x float> [[TMP9]], <4 x float> [[TMP8]], <4 x i32> <i32 0, i32 4, i32 5, i32 6>
-; SSE2-NEXT:    [[TMP11:%.*]] = fmul <4 x float> [[TMP8]], [[TMP10]]
-; SSE2-NEXT:    store <4 x float> [[TMP11]], ptr [[ARRAYIDX5]], align 4
+; SSE2-NEXT:    [[TMP7:%.*]] = load <4 x float>, ptr [[ARRAYIDX]], align 4
+; SSE2-NEXT:    [[TMP8:%.*]] = shufflevector <4 x float> [[TMP7]], <4 x float> poison, <4 x i32> <i32 undef, i32 0, i32 1, i32 2>
+; SSE2-NEXT:    [[TMP9:%.*]] = insertelement <4 x float> [[TMP8]], float [[TMP1]], i64 0
+; SSE2-NEXT:    [[TMP10:%.*]] = fmul <4 x float> [[TMP7]], [[TMP9]]
+; SSE2-NEXT:    store <4 x float> [[TMP10]], ptr [[ARRAYIDX5]], align 4
 ; SSE2-NEXT:    [[INDVARS_IV_NEXT]] = add nuw nsw i64 [[INDVARS_IV]], 5
-; SSE2-NEXT:    [[TMP13:%.*]] = trunc i64 [[INDVARS_IV_NEXT]] to i32
-; SSE2-NEXT:    [[ARRAYIDX41:%.*]] = getelementptr inbounds [32000 x float], ptr @a, i32 0, i32 [[TMP13]]
-; SSE2-NEXT:    [[TMP14]] = load float, ptr [[ARRAYIDX41]], align 4
-; SSE2-NEXT:    [[TMP15:%.*]] = extractelement <4 x float> [[TMP8]], i64 3
-; SSE2-NEXT:    [[MUL45:%.*]] = fmul float [[TMP14]], [[TMP15]]
+; SSE2-NEXT:    [[TMP11:%.*]] = trunc i64 [[INDVARS_IV_NEXT]] to i32
+; SSE2-NEXT:    [[ARRAYIDX41:%.*]] = getelementptr inbounds [32000 x float], ptr @a, i32 0, i32 [[TMP11]]
+; SSE2-NEXT:    [[TMP12]] = load float, ptr [[ARRAYIDX41]], align 4
+; SSE2-NEXT:    [[TMP13:%.*]] = extractelement <4 x float> [[TMP7]], i64 3
+; SSE2-NEXT:    [[MUL45:%.*]] = fmul float [[TMP12]], [[TMP13]]
 ; SSE2-NEXT:    store float [[MUL45]], ptr [[ARRAYIDX31]], align 4
-; SSE2-NEXT:    [[TMP16:%.*]] = trunc i64 [[INDVARS_IV_NEXT]] to i32
-; SSE2-NEXT:    [[CMP2:%.*]] = icmp slt i32 [[TMP16]], 31995
+; SSE2-NEXT:    [[TMP14:%.*]] = trunc i64 [[INDVARS_IV_NEXT]] to i32
+; SSE2-NEXT:    [[CMP2:%.*]] = icmp slt i32 [[TMP14]], 31995
 ; SSE2-NEXT:    br i1 [[CMP2]], label [[FOR_BODY3]], label [[FOR_END:%.*]]
 ; SSE2:       for.end:
 ; SSE2-NEXT:    ret void
@@ -458,17 +458,17 @@ for.end:
 
 define void @load_reorder_double(ptr nocapture %c, ptr noalias nocapture readonly %a, ptr noalias nocapture readonly %b){
 ; CHECK-LABEL: @load_reorder_double(
-; CHECK-NEXT:    [[TMP2:%.*]] = load <2 x double>, ptr [[B:%.*]], align 4
-; CHECK-NEXT:    [[TMP4:%.*]] = load <2 x double>, ptr [[A:%.*]], align 4
-; CHECK-NEXT:    [[TMP5:%.*]] = fadd <2 x double> [[TMP2]], [[TMP4]]
-; CHECK-NEXT:    store <2 x double> [[TMP5]], ptr [[C:%.*]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[B:%.*]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = load <2 x double>, ptr [[A:%.*]], align 4
+; CHECK-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP1]], [[TMP2]]
+; CHECK-NEXT:    store <2 x double> [[TMP3]], ptr [[C:%.*]], align 4
 ; CHECK-NEXT:    ret void
 ;
 ; SSE2-LABEL: @load_reorder_double(
-; SSE2-NEXT:    [[TMP2:%.*]] = load <2 x double>, ptr [[B:%.*]], align 4
-; SSE2-NEXT:    [[TMP4:%.*]] = load <2 x double>, ptr [[A:%.*]], align 4
-; SSE2-NEXT:    [[TMP5:%.*]] = fadd <2 x double> [[TMP2]], [[TMP4]]
-; SSE2-NEXT:    store <2 x double> [[TMP5]], ptr [[C:%.*]], align 4
+; SSE2-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[B:%.*]], align 4
+; SSE2-NEXT:    [[TMP2:%.*]] = load <2 x double>, ptr [[A:%.*]], align 4
+; SSE2-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP1]], [[TMP2]]
+; SSE2-NEXT:    store <2 x double> [[TMP3]], ptr [[C:%.*]], align 4
 ; SSE2-NEXT:    ret void
 ;
   %1 = load double, ptr %a
@@ -493,17 +493,17 @@ define void @load_reorder_double(ptr nocapture %c, ptr noalias nocapture readonl
 
 define void @load_reorder_float(ptr nocapture %c, ptr noalias nocapture readonly %a, ptr noalias nocapture readonly %b){
 ; CHECK-LABEL: @load_reorder_float(
-; CHECK-NEXT:    [[TMP2:%.*]] = load <4 x float>, ptr [[A:%.*]], align 4
-; CHECK-NEXT:    [[TMP4:%.*]] = load <4 x float>, ptr [[B:%.*]], align 4
-; CHECK-NEXT:    [[TMP5:%.*]] = fadd <4 x float> [[TMP2]], [[TMP4]]
-; CHECK-NEXT:    store <4 x float> [[TMP5]], ptr [[C:%.*]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x float>, ptr [[A:%.*]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = load <4 x float>, ptr [[B:%.*]], align 4
+; CHECK-NEXT:    [[TMP3:%.*]] = fadd <4 x float> [[TMP1]], [[TMP2]]
+; CHECK-NEXT:    store <4 x float> [[TMP3]], ptr [[C:%.*]], align 4
 ; CHECK-NEXT:    ret void
 ;
 ; SSE2-LABEL: @load_reorder_float(
-; SSE2-NEXT:    [[TMP2:%.*]] = load <4 x float>, ptr [[A:%.*]], align 4
-; SSE2-NEXT:    [[TMP4:%.*]] = load <4 x float>, ptr [[B:%.*]], align 4
-; SSE2-NEXT:    [[TMP5:%.*]] = fadd <4 x float> [[TMP2]], [[TMP4]]
-; SSE2-NEXT:    store <4 x float> [[TMP5]], ptr [[C:%.*]], align 4
+; SSE2-NEXT:    [[TMP1:%.*]] = load <4 x float>, ptr [[A:%.*]], align 4
+; SSE2-NEXT:    [[TMP2:%.*]] = load <4 x float>, ptr [[B:%.*]], align 4
+; SSE2-NEXT:    [[TMP3:%.*]] = fadd <4 x float> [[TMP1]], [[TMP2]]
+; SSE2-NEXT:    store <4 x float> [[TMP3]], ptr [[C:%.*]], align 4
 ; SSE2-NEXT:    ret void
 ;
   %1 = load float, ptr %a
@@ -542,21 +542,21 @@ define void @load_reorder_float(ptr nocapture %c, ptr noalias nocapture readonly
 
 define void @opcode_reorder(ptr noalias nocapture %a, ptr noalias nocapture readonly %b, ptr noalias nocapture readonly %c,ptr noalias nocapture readonly %d) {
 ; CHECK-LABEL: @opcode_reorder(
-; CHECK-NEXT:    [[TMP2:%.*]] = load <4 x float>, ptr [[B:%.*]], align 4
-; CHECK-NEXT:    [[TMP4:%.*]] = load <4 x float>, ptr [[C:%.*]], align 4
-; CHECK-NEXT:    [[TMP5:%.*]] = fadd <4 x float> [[TMP2]], [[TMP4]]
-; CHECK-NEXT:    [[TMP7:%.*]] = load <4 x float>, ptr [[D:%.*]], align 4
-; CHECK-NEXT:    [[TMP8:%.*]] = fadd <4 x float> [[TMP7]], [[TMP5]]
-; CHECK-NEXT:    store <4 x float> [[TMP8]], ptr [[A:%.*]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x float>, ptr [[B:%.*]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = load <4 x float>, ptr [[C:%.*]], align 4
+; CHECK-NEXT:    [[TMP3:%.*]] = fadd <4 x float> [[TMP1]], [[TMP2]]
+; CHECK-NEXT:    [[TMP4:%.*]] = load <4 x float>, ptr [[D:%.*]], align 4
+; CHECK-NEXT:    [[TMP5:%.*]] = fadd <4 x float> [[TMP4]], [[TMP3]]
+; CHECK-NEXT:    store <4 x float> [[TMP5]], ptr [[A:%.*]], align 4
 ; CHECK-NEXT:    ret void
 ;
 ; SSE2-LABEL: @opcode_reorder(
-; SSE2-NEXT:    [[TMP2:%.*]] = load <4 x float>, ptr [[B:%.*]], align 4
-; SSE2-NEXT:    [[TMP4:%.*]] = load <4 x float>, ptr [[C:%.*]], align 4
-; SSE2-NEXT:    [[TMP5:%.*]] = fadd <4 x float> [[TMP2]], [[TMP4]]
-; SSE2-NEXT:    [[TMP7:%.*]] = load <4 x float>, ptr [[D:%.*]], align 4
-; SSE2-NEXT:    [[TMP8:%.*]] = fadd <4 x float> [[TMP7]], [[TMP5]]
-; SSE2-NEXT:    store <4 x float> [[TMP8]], ptr [[A:%.*]], align 4
+; SSE2-NEXT:    [[TMP1:%.*]] = load <4 x float>, ptr [[B:%.*]], align 4
+; SSE2-NEXT:    [[TMP2:%.*]] = load <4 x float>, ptr [[C:%.*]], align 4
+; SSE2-NEXT:    [[TMP3:%.*]] = fadd <4 x float> [[TMP1]], [[TMP2]]
+; SSE2-NEXT:    [[TMP4:%.*]] = load <4 x float>, ptr [[D:%.*]], align 4
+; SSE2-NEXT:    [[TMP5:%.*]] = fadd <4 x float> [[TMP4]], [[TMP3]]
+; SSE2-NEXT:    store <4 x float> [[TMP5]], ptr [[A:%.*]], align 4
 ; SSE2-NEXT:    ret void
 ;
   %1 = load float, ptr %b
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/phi.ll b/llvm/test/Transforms/SLPVectorizer/X86/phi.ll
index 4f2c95645a753..57a92385fb125 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/phi.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/phi.ll
@@ -26,11 +26,11 @@ define i32 @foo(ptr nocapture %A, i32 %k) {
 ; CHECK-NEXT:    br i1 [[TOBOOL]], label [[IF_ELSE:%.*]], label [[IF_END:%.*]]
 ; CHECK:       if.else:
 ; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds double, ptr [[A:%.*]], i64 10
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[ARRAYIDX]], align 8
+; CHECK-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[ARRAYIDX]], align 8
 ; CHECK-NEXT:    br label [[IF_END]]
 ; CHECK:       if.end:
-; CHECK-NEXT:    [[TMP2:%.*]] = phi <2 x double> [ [[TMP1]], [[IF_ELSE]] ], [ <double 3.000000e+00, double 5.000000e+00>, [[ENTRY:%.*]] ]
-; CHECK-NEXT:    store <2 x double> [[TMP2]], ptr [[A]], align 8
+; CHECK-NEXT:    [[TMP1:%.*]] = phi <2 x double> [ [[TMP0]], [[IF_ELSE]] ], [ <double 3.000000e+00, double 5.000000e+00>, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    store <2 x double> [[TMP1]], ptr [[A]], align 8
 ; CHECK-NEXT:    ret i32 undef
 ;
 entry:
@@ -73,19 +73,19 @@ if.end:                                           ; preds = %entry, %if.else
 define i32 @foo2(ptr noalias nocapture %B, ptr noalias nocapture %A, i32 %n, i32 %m) #0 {
 ; CHECK-LABEL: @foo2(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, ptr [[A:%.*]], align 8
+; CHECK-NEXT:    [[TMP0:%.*]] = load <2 x double>, ptr [[A:%.*]], align 8
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[I_019:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INC:%.*]], [[FOR_BODY]] ]
-; CHECK-NEXT:    [[TMP2:%.*]] = phi <2 x double> [ [[TMP1]], [[ENTRY]] ], [ [[TMP5:%.*]], [[FOR_BODY]] ]
-; CHECK-NEXT:    [[TMP3:%.*]] = fadd <2 x double> [[TMP2]], <double 1.000000e+01, double 1.000000e+01>
-; CHECK-NEXT:    [[TMP4:%.*]] = fmul <2 x double> [[TMP3]], <double 4.000000e+00, double 4.000000e+00>
-; CHECK-NEXT:    [[TMP5]] = fadd <2 x double> [[TMP4]], <double 4.000000e+00, double 4.000000e+00>
+; CHECK-NEXT:    [[TMP1:%.*]] = phi <2 x double> [ [[TMP0]], [[ENTRY]] ], [ [[TMP4:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[TMP2:%.*]] = fadd <2 x double> [[TMP1]], <double 1.000000e+01, double 1.000000e+01>
+; CHECK-NEXT:    [[TMP3:%.*]] = fmul <2 x double> [[TMP2]], <double 4.000000e+00, double 4.000000e+00>
+; CHECK-NEXT:    [[TMP4]] = fadd <2 x double> [[TMP3]], <double 4.000000e+00, double 4.000000e+00>
 ; CHECK-NEXT:    [[INC]] = add nsw i32 [[I_019]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i32 [[INC]], 100
 ; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_END:%.*]], label [[FOR_BODY]]
 ; CHECK:       for.end:
-; CHECK-NEXT:    store <2 x double> [[TMP5]], ptr [[B:%.*]], align 8
+; CHECK-NEXT:    store <2 x double> [[TMP4]], ptr [[B:%.*]], align 8
 ; CHECK-NEXT:    ret i32 0
 ;
 entry:
@@ -138,41 +138,40 @@ define float @foo3(ptr nocapture readonly %A) #0 {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = load float, ptr [[A:%.*]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds float, ptr [[A]], i64 1
-; CHECK-NEXT:    [[TMP2:%.*]] = load <4 x float>, ptr [[ARRAYIDX1]], align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = extractelement <4 x float> [[TMP2]], i32 0
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> poison, float [[TMP0]], i32 0
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x float> [[TMP4]], float [[TMP3]], i32 1
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x float>, ptr [[ARRAYIDX1]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <4 x float> [[TMP1]], <4 x float> poison, <2 x i32> <i32 undef, i32 0>
+; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x float> [[TMP2]], float [[TMP0]], i32 0
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
 ; CHECK-NEXT:    [[R_052:%.*]] = phi float [ [[TMP0]], [[ENTRY]] ], [ [[ADD6:%.*]], [[FOR_BODY]] ]
-; CHECK-NEXT:    [[TMP6:%.*]] = phi <4 x float> [ [[TMP2]], [[ENTRY]] ], [ [[TMP16:%.*]], [[FOR_BODY]] ]
-; CHECK-NEXT:    [[TMP7:%.*]] = phi <2 x float> [ [[TMP5]], [[ENTRY]] ], [ [[TMP12:%.*]], [[FOR_BODY]] ]
-; CHECK-NEXT:    [[TMP8:%.*]] = extractelement <2 x float> [[TMP7]], i32 0
-; CHECK-NEXT:    [[MUL:%.*]] = fmul float [[TMP8]], 7.000000e+00
+; CHECK-NEXT:    [[TMP4:%.*]] = phi <4 x float> [ [[TMP1]], [[ENTRY]] ], [ [[TMP13:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[TMP5:%.*]] = phi <2 x float> [ [[TMP3]], [[ENTRY]] ], [ [[TMP9:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[TMP6:%.*]] = extractelement <2 x float> [[TMP5]], i32 0
+; CHECK-NEXT:    [[MUL:%.*]] = fmul float [[TMP6]], 7.000000e+00
 ; CHECK-NEXT:    [[ADD6]] = fadd float [[R_052]], [[MUL]]
-; CHECK-NEXT:    [[TMP9:%.*]] = add nsw i64 [[INDVARS_IV]], 2
-; CHECK-NEXT:    [[ARRAYIDX14:%.*]] = getelementptr inbounds float, ptr [[A]], i64 [[TMP9]]
-; CHECK-NEXT:    [[TMP10:%.*]] = load float, ptr [[ARRAYIDX14]], align 4
+; CHECK-NEXT:    [[TMP7:%.*]] = add nsw i64 [[INDVARS_IV]], 2
+; CHECK-NEXT:    [[ARRAYIDX14:%.*]] = getelementptr inbounds float, ptr [[A]], i64 [[TMP7]]
+; CHECK-NEXT:    [[TMP8:%.*]] = load float, ptr [[ARRAYIDX14]], align 4
 ; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nuw nsw i64 [[INDVARS_IV]], 3
 ; CHECK-NEXT:    [[ARRAYIDX19:%.*]] = getelementptr inbounds float, ptr [[A]], i64 [[INDVARS_IV_NEXT]]
-; CHECK-NEXT:    [[TMP12]] = load <2 x float>, ptr [[ARRAYIDX19]], align 4
-; CHECK-NEXT:    [[TMP13:%.*]] = shufflevector <2 x float> [[TMP7]], <2 x float> [[TMP12]], <4 x i32> <i32 1, i32 undef, i32 2, i32 3>
-; CHECK-NEXT:    [[TMP14:%.*]] = insertelement <4 x float> [[TMP13]], float [[TMP10]], i32 1
-; CHECK-NEXT:    [[TMP15:%.*]] = fmul <4 x float> [[TMP14]], <float 8.000000e+00, float 9.000000e+00, float 1.000000e+01, float 1.100000e+01>
-; CHECK-NEXT:    [[TMP16]] = fadd <4 x float> [[TMP6]], [[TMP15]]
-; CHECK-NEXT:    [[TMP17:%.*]] = trunc i64 [[INDVARS_IV_NEXT]] to i32
-; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 [[TMP17]], 121
+; CHECK-NEXT:    [[TMP9]] = load <2 x float>, ptr [[ARRAYIDX19]], align 4
+; CHECK-NEXT:    [[TMP10:%.*]] = shufflevector <2 x float> [[TMP5]], <2 x float> [[TMP9]], <4 x i32> <i32 1, i32 undef, i32 2, i32 3>
+; CHECK-NEXT:    [[TMP11:%.*]] = insertelement <4 x float> [[TMP10]], float [[TMP8]], i32 1
+; CHECK-NEXT:    [[TMP12:%.*]] = fmul <4 x float> [[TMP11]], <float 8.000000e+00, float 9.000000e+00, float 1.000000e+01, float 1.100000e+01>
+; CHECK-NEXT:    [[TMP13]] = fadd <4 x float> [[TMP4]], [[TMP12]]
+; CHECK-NEXT:    [[TMP14:%.*]] = trunc i64 [[INDVARS_IV_NEXT]] to i32
+; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 [[TMP14]], 121
 ; CHECK-NEXT:    br i1 [[CMP]], label [[FOR_BODY]], label [[FOR_END:%.*]]
 ; CHECK:       for.end:
-; CHECK-NEXT:    [[TMP18:%.*]] = extractelement <4 x float> [[TMP16]], i32 0
-; CHECK-NEXT:    [[ADD28:%.*]] = fadd float [[ADD6]], [[TMP18]]
-; CHECK-NEXT:    [[TMP19:%.*]] = extractelement <4 x float> [[TMP16]], i32 1
-; CHECK-NEXT:    [[ADD29:%.*]] = fadd float [[ADD28]], [[TMP19]]
-; CHECK-NEXT:    [[TMP20:%.*]] = extractelement <4 x float> [[TMP16]], i32 2
-; CHECK-NEXT:    [[ADD30:%.*]] = fadd float [[ADD29]], [[TMP20]]
-; CHECK-NEXT:    [[TMP21:%.*]] = extractelement <4 x float> [[TMP16]], i32 3
-; CHECK-NEXT:    [[ADD31:%.*]] = fadd float [[ADD30]], [[TMP21]]
+; CHECK-NEXT:    [[TMP15:%.*]] = extractelement <4 x float> [[TMP13]], i32 0
+; CHECK-NEXT:    [[ADD28:%.*]] = fadd float [[ADD6]], [[TMP15]]
+; CHECK-NEXT:    [[TMP16:%.*]] = extractelement <4 x float> [[TMP13]], i32 1
+; CHECK-NEXT:    [[ADD29:%.*]] = fadd float [[ADD28]], [[TMP16]]
+; CHECK-NEXT:    [[TMP17:%.*]] = extractelement <4 x float> [[TMP13]], i32 2
+; CHECK-NEXT:    [[ADD30:%.*]] = fadd float [[ADD29]], [[TMP17]]
+; CHECK-NEXT:    [[TMP18:%.*]] = extractelement <4 x float> [[TMP13]], i32 3
+; CHECK-NEXT:    [[ADD31:%.*]] = fadd float [[ADD30]], [[TMP18]]
 ; CHECK-NEXT:    ret float [[ADD31]]
 ;
 entry:
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/phis-different-order.ll b/llvm/test/Transforms/SLPVectorizer/X86/phis-different-order.ll
new file mode 100644
index 0000000000000..2a0d4804026fd
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/X86/phis-different-order.ll
@@ -0,0 +1,47 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
+; RUN: opt -passes=slp-vectorizer -S < %s -mcpu=pentium4 -mtriple=i386-pc-windows-msvc19.16.0 | FileCheck %s
+
+define ptr @test()  {
+; CHECK-LABEL: define ptr @test
+; CHECK-SAME: () #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[PC:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ 0, [[BB:%.*]] ]
+; CHECK-NEXT:    br i1 false, label [[FOR_INC:%.*]], label [[BB2:%.*]]
+; CHECK:       for.inc:
+; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <2 x i64> <i64 0, i64 poison>, i64 [[PC]], i32 1
+; CHECK-NEXT:    br i1 false, label [[BB2]], label [[BB]]
+; CHECK:       bb:
+; CHECK-NEXT:    br i1 false, label [[BB2]], label [[FOR_BODY]]
+; CHECK:       bb1:
+; CHECK-NEXT:    br label [[BB2]]
+; CHECK:       bb2:
+; CHECK-NEXT:    [[TMP1:%.*]] = phi <2 x i64> [ poison, [[BB1:%.*]] ], [ [[TMP0]], [[BB]] ], [ [[TMP0]], [[FOR_INC]] ], [ zeroinitializer, [[FOR_BODY]] ]
+; CHECK-NEXT:    store <2 x i64> [[TMP1]], ptr getelementptr inbounds ([7 x i64], ptr null, i32 0, i32 2), align 8
+; CHECK-NEXT:    ret ptr null
+;
+entry:
+  br label %for.body
+
+for.body:
+  %pc = phi i64 [ 0, %entry ], [ 0, %bb ]
+  br i1 false, label %for.inc, label %bb2
+
+for.inc:
+  br i1 false, label %bb2, label %bb
+
+bb:
+  br i1 false, label %bb2, label %for.body
+
+bb1:
+  br label %bb2
+
+bb2:
+  %ph1 = phi i64 [ 0, %bb1 ], [ 0, %bb ], [ 0, %for.inc ], [ 0, %for.body ]
+  %ph2 = phi i64 [ 0, %bb1 ], [ 0, %for.body ], [ %pc, %bb ], [ %pc, %for.inc ]
+  store i64 %ph1, ptr getelementptr inbounds ([7 x i64], ptr null, i32 0, i32 2), align 8
+  store i64 %ph2, ptr getelementptr inbounds ([7 x i64], ptr null, i32 0, i32 3), align 8
+  ret ptr null
+}
+
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/pr35497.ll b/llvm/test/Transforms/SLPVectorizer/X86/pr35497.ll
index 98df52a85248b..9c7e8f66c6c6c 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/pr35497.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/pr35497.ll
@@ -68,14 +68,13 @@ define void @pr35497() local_unnamed_addr #0 {
 ; SSE-NEXT:    [[TMP4:%.*]] = shufflevector <2 x i64> [[TMP3]], <2 x i64> poison, <2 x i32> <i32 1, i32 0>
 ; SSE-NEXT:    [[TMP5:%.*]] = add nuw nsw <2 x i64> [[TMP4]], zeroinitializer
 ; SSE-NEXT:    store <2 x i64> [[TMP5]], ptr undef, align 1
-; SSE-NEXT:    [[TMP6:%.*]] = insertelement <2 x i64> poison, i64 [[ADD]], i32 0
-; SSE-NEXT:    [[TMP7:%.*]] = shufflevector <2 x i64> [[TMP6]], <2 x i64> [[TMP5]], <2 x i32> <i32 0, i32 3>
-; SSE-NEXT:    [[TMP8:%.*]] = shl <2 x i64> [[TMP7]], <i64 2, i64 2>
-; SSE-NEXT:    [[TMP9:%.*]] = and <2 x i64> [[TMP8]], <i64 20, i64 20>
-; SSE-NEXT:    [[TMP10:%.*]] = shufflevector <2 x i64> [[TMP9]], <2 x i64> poison, <2 x i32> <i32 1, i32 0>
-; SSE-NEXT:    [[TMP11:%.*]] = lshr <2 x i64> [[TMP5]], <i64 6, i64 6>
-; SSE-NEXT:    [[TMP12:%.*]] = add nuw nsw <2 x i64> [[TMP10]], [[TMP11]]
-; SSE-NEXT:    store <2 x i64> [[TMP12]], ptr [[ARRAYIDX2_2]], align 1
+; SSE-NEXT:    [[TMP6:%.*]] = insertelement <2 x i64> [[TMP5]], i64 [[ADD]], i32 0
+; SSE-NEXT:    [[TMP7:%.*]] = shl <2 x i64> [[TMP6]], <i64 2, i64 2>
+; SSE-NEXT:    [[TMP8:%.*]] = and <2 x i64> [[TMP7]], <i64 20, i64 20>
+; SSE-NEXT:    [[TMP9:%.*]] = shufflevector <2 x i64> [[TMP8]], <2 x i64> poison, <2 x i32> <i32 1, i32 0>
+; SSE-NEXT:    [[TMP10:%.*]] = lshr <2 x i64> [[TMP5]], <i64 6, i64 6>
+; SSE-NEXT:    [[TMP11:%.*]] = add nuw nsw <2 x i64> [[TMP9]], [[TMP10]]
+; SSE-NEXT:    store <2 x i64> [[TMP11]], ptr [[ARRAYIDX2_2]], align 1
 ; SSE-NEXT:    ret void
 ;
 ; AVX-LABEL: @pr35497(
@@ -89,14 +88,13 @@ define void @pr35497() local_unnamed_addr #0 {
 ; AVX-NEXT:    [[TMP3:%.*]] = and <2 x i64> [[TMP2]], <i64 20, i64 20>
 ; AVX-NEXT:    [[TMP4:%.*]] = add nuw nsw <2 x i64> [[TMP3]], zeroinitializer
 ; AVX-NEXT:    store <2 x i64> [[TMP4]], ptr undef, align 1
-; AVX-NEXT:    [[TMP5:%.*]] = insertelement <2 x i64> poison, i64 [[ADD]], i32 0
-; AVX-NEXT:    [[TMP6:%.*]] = shufflevector <2 x i64> [[TMP5]], <2 x i64> [[TMP4]], <2 x i32> <i32 0, i32 3>
-; AVX-NEXT:    [[TMP7:%.*]] = shl <2 x i64> [[TMP6]], <i64 2, i64 2>
-; AVX-NEXT:    [[TMP8:%.*]] = and <2 x i64> [[TMP7]], <i64 20, i64 20>
-; AVX-NEXT:    [[TMP9:%.*]] = shufflevector <2 x i64> [[TMP8]], <2 x i64> poison, <2 x i32> <i32 1, i32 0>
-; AVX-NEXT:    [[TMP10:%.*]] = lshr <2 x i64> [[TMP4]], <i64 6, i64 6>
-; AVX-NEXT:    [[TMP11:%.*]] = add nuw nsw <2 x i64> [[TMP9]], [[TMP10]]
-; AVX-NEXT:    store <2 x i64> [[TMP11]], ptr [[ARRAYIDX2_2]], align 1
+; AVX-NEXT:    [[TMP5:%.*]] = insertelement <2 x i64> [[TMP4]], i64 [[ADD]], i32 0
+; AVX-NEXT:    [[TMP6:%.*]] = shl <2 x i64> [[TMP5]], <i64 2, i64 2>
+; AVX-NEXT:    [[TMP7:%.*]] = and <2 x i64> [[TMP6]], <i64 20, i64 20>
+; AVX-NEXT:    [[TMP8:%.*]] = shufflevector <2 x i64> [[TMP7]], <2 x i64> poison, <2 x i32> <i32 1, i32 0>
+; AVX-NEXT:    [[TMP9:%.*]] = lshr <2 x i64> [[TMP4]], <i64 6, i64 6>
+; AVX-NEXT:    [[TMP10:%.*]] = add nuw nsw <2 x i64> [[TMP8]], [[TMP9]]
+; AVX-NEXT:    store <2 x i64> [[TMP10]], ptr [[ARRAYIDX2_2]], align 1
 ; AVX-NEXT:    ret void
 ;
 entry:
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/pr47629-inseltpoison.ll b/llvm/test/Transforms/SLPVectorizer/X86/pr47629-inseltpoison.ll
index 3850b42327857..8284fc404ca18 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/pr47629-inseltpoison.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/pr47629-inseltpoison.ll
@@ -71,12 +71,19 @@ define void @gather_load(ptr noalias nocapture %0, ptr noalias nocapture readonl
 ; AVX512F-NEXT:    ret void
 ;
 ; AVX512VL-LABEL: @gather_load(
-; AVX512VL-NEXT:    [[TMP3:%.*]] = insertelement <4 x ptr> poison, ptr [[TMP1:%.*]], i64 0
-; AVX512VL-NEXT:    [[SHUFFLE:%.*]] = shufflevector <4 x ptr> [[TMP3]], <4 x ptr> poison, <4 x i32> zeroinitializer
-; AVX512VL-NEXT:    [[TMP4:%.*]] = getelementptr i32, <4 x ptr> [[SHUFFLE]], <4 x i64> <i64 0, i64 11, i64 4, i64 1>
-; AVX512VL-NEXT:    [[TMP5:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> [[TMP4]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> poison), !tbaa [[TBAA0:![0-9]+]]
-; AVX512VL-NEXT:    [[TMP6:%.*]] = add nsw <4 x i32> [[TMP5]], <i32 1, i32 2, i32 3, i32 4>
-; AVX512VL-NEXT:    store <4 x i32> [[TMP6]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP3:%.*]] = getelementptr inbounds i32, ptr [[TMP1:%.*]], i64 1
+; AVX512VL-NEXT:    [[TMP4:%.*]] = load i32, ptr [[TMP1]], align 4, !tbaa [[TBAA0:![0-9]+]]
+; AVX512VL-NEXT:    [[TMP5:%.*]] = getelementptr inbounds i32, ptr [[TMP1]], i64 11
+; AVX512VL-NEXT:    [[TMP6:%.*]] = load i32, ptr [[TMP5]], align 4, !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP7:%.*]] = getelementptr inbounds i32, ptr [[TMP1]], i64 4
+; AVX512VL-NEXT:    [[TMP8:%.*]] = load i32, ptr [[TMP7]], align 4, !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP9:%.*]] = load i32, ptr [[TMP3]], align 4, !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP10:%.*]] = insertelement <4 x i32> poison, i32 [[TMP4]], i64 0
+; AVX512VL-NEXT:    [[TMP11:%.*]] = insertelement <4 x i32> [[TMP10]], i32 [[TMP6]], i64 1
+; AVX512VL-NEXT:    [[TMP12:%.*]] = insertelement <4 x i32> [[TMP11]], i32 [[TMP8]], i64 2
+; AVX512VL-NEXT:    [[TMP13:%.*]] = insertelement <4 x i32> [[TMP12]], i32 [[TMP9]], i64 3
+; AVX512VL-NEXT:    [[TMP14:%.*]] = add nsw <4 x i32> [[TMP13]], <i32 1, i32 2, i32 3, i32 4>
+; AVX512VL-NEXT:    store <4 x i32> [[TMP14]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
 ; AVX512VL-NEXT:    ret void
 ;
   %3 = getelementptr inbounds i32, ptr %1, i64 1
@@ -174,11 +181,11 @@ define void @gather_load_2(ptr noalias nocapture %0, ptr noalias nocapture reado
 ;
 ; AVX512VL-LABEL: @gather_load_2(
 ; AVX512VL-NEXT:    [[TMP3:%.*]] = insertelement <4 x ptr> poison, ptr [[TMP1:%.*]], i64 0
-; AVX512VL-NEXT:    [[SHUFFLE:%.*]] = shufflevector <4 x ptr> [[TMP3]], <4 x ptr> poison, <4 x i32> zeroinitializer
-; AVX512VL-NEXT:    [[TMP4:%.*]] = getelementptr i32, <4 x ptr> [[SHUFFLE]], <4 x i64> <i64 1, i64 10, i64 3, i64 5>
-; AVX512VL-NEXT:    [[TMP5:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> [[TMP4]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> poison), !tbaa [[TBAA0]]
-; AVX512VL-NEXT:    [[TMP6:%.*]] = add nsw <4 x i32> [[TMP5]], <i32 1, i32 2, i32 3, i32 4>
-; AVX512VL-NEXT:    store <4 x i32> [[TMP6]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP4:%.*]] = shufflevector <4 x ptr> [[TMP3]], <4 x ptr> poison, <4 x i32> zeroinitializer
+; AVX512VL-NEXT:    [[TMP5:%.*]] = getelementptr i32, <4 x ptr> [[TMP4]], <4 x i64> <i64 1, i64 10, i64 3, i64 5>
+; AVX512VL-NEXT:    [[TMP6:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> [[TMP5]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> poison), !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP7:%.*]] = add nsw <4 x i32> [[TMP6]], <i32 1, i32 2, i32 3, i32 4>
+; AVX512VL-NEXT:    store <4 x i32> [[TMP7]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
 ; AVX512VL-NEXT:    ret void
 ;
   %3 = getelementptr inbounds i32, ptr %1, i64 1
@@ -304,20 +311,20 @@ define void @gather_load_3(ptr noalias nocapture %0, ptr noalias nocapture reado
 ;
 ; AVX512F-LABEL: @gather_load_3(
 ; AVX512F-NEXT:    [[TMP3:%.*]] = insertelement <8 x ptr> poison, ptr [[TMP1:%.*]], i64 0
-; AVX512F-NEXT:    [[SHUFFLE:%.*]] = shufflevector <8 x ptr> [[TMP3]], <8 x ptr> poison, <8 x i32> zeroinitializer
-; AVX512F-NEXT:    [[TMP4:%.*]] = getelementptr i32, <8 x ptr> [[SHUFFLE]], <8 x i64> <i64 0, i64 11, i64 4, i64 15, i64 18, i64 9, i64 6, i64 21>
-; AVX512F-NEXT:    [[TMP5:%.*]] = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> [[TMP4]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> poison), !tbaa [[TBAA0]]
-; AVX512F-NEXT:    [[TMP6:%.*]] = add <8 x i32> [[TMP5]], <i32 1, i32 2, i32 3, i32 4, i32 1, i32 2, i32 3, i32 4>
-; AVX512F-NEXT:    store <8 x i32> [[TMP6]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
+; AVX512F-NEXT:    [[TMP4:%.*]] = shufflevector <8 x ptr> [[TMP3]], <8 x ptr> poison, <8 x i32> zeroinitializer
+; AVX512F-NEXT:    [[TMP5:%.*]] = getelementptr i32, <8 x ptr> [[TMP4]], <8 x i64> <i64 0, i64 11, i64 4, i64 15, i64 18, i64 9, i64 6, i64 21>
+; AVX512F-NEXT:    [[TMP6:%.*]] = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> [[TMP5]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> poison), !tbaa [[TBAA0]]
+; AVX512F-NEXT:    [[TMP7:%.*]] = add <8 x i32> [[TMP6]], <i32 1, i32 2, i32 3, i32 4, i32 1, i32 2, i32 3, i32 4>
+; AVX512F-NEXT:    store <8 x i32> [[TMP7]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
 ; AVX512F-NEXT:    ret void
 ;
 ; AVX512VL-LABEL: @gather_load_3(
 ; AVX512VL-NEXT:    [[TMP3:%.*]] = insertelement <8 x ptr> poison, ptr [[TMP1:%.*]], i64 0
-; AVX512VL-NEXT:    [[SHUFFLE:%.*]] = shufflevector <8 x ptr> [[TMP3]], <8 x ptr> poison, <8 x i32> zeroinitializer
-; AVX512VL-NEXT:    [[TMP4:%.*]] = getelementptr i32, <8 x ptr> [[SHUFFLE]], <8 x i64> <i64 0, i64 11, i64 4, i64 15, i64 18, i64 9, i64 6, i64 21>
-; AVX512VL-NEXT:    [[TMP5:%.*]] = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> [[TMP4]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> poison), !tbaa [[TBAA0]]
-; AVX512VL-NEXT:    [[TMP6:%.*]] = add <8 x i32> [[TMP5]], <i32 1, i32 2, i32 3, i32 4, i32 1, i32 2, i32 3, i32 4>
-; AVX512VL-NEXT:    store <8 x i32> [[TMP6]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP4:%.*]] = shufflevector <8 x ptr> [[TMP3]], <8 x ptr> poison, <8 x i32> zeroinitializer
+; AVX512VL-NEXT:    [[TMP5:%.*]] = getelementptr i32, <8 x ptr> [[TMP4]], <8 x i64> <i64 0, i64 11, i64 4, i64 15, i64 18, i64 9, i64 6, i64 21>
+; AVX512VL-NEXT:    [[TMP6:%.*]] = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> [[TMP5]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> poison), !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP7:%.*]] = add <8 x i32> [[TMP6]], <i32 1, i32 2, i32 3, i32 4, i32 1, i32 2, i32 3, i32 4>
+; AVX512VL-NEXT:    store <8 x i32> [[TMP7]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
 ; AVX512VL-NEXT:    ret void
 ;
   %3 = load i32, ptr %1, align 4, !tbaa !2
@@ -461,20 +468,20 @@ define void @gather_load_4(ptr noalias nocapture %t0, ptr noalias nocapture read
 ;
 ; AVX512F-LABEL: @gather_load_4(
 ; AVX512F-NEXT:    [[TMP1:%.*]] = insertelement <8 x ptr> poison, ptr [[T1:%.*]], i64 0
-; AVX512F-NEXT:    [[SHUFFLE:%.*]] = shufflevector <8 x ptr> [[TMP1]], <8 x ptr> poison, <8 x i32> zeroinitializer
-; AVX512F-NEXT:    [[TMP2:%.*]] = getelementptr i32, <8 x ptr> [[SHUFFLE]], <8 x i64> <i64 0, i64 11, i64 4, i64 15, i64 18, i64 9, i64 6, i64 21>
-; AVX512F-NEXT:    [[TMP3:%.*]] = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> [[TMP2]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> poison), !tbaa [[TBAA0]]
-; AVX512F-NEXT:    [[TMP4:%.*]] = add <8 x i32> [[TMP3]], <i32 1, i32 2, i32 3, i32 4, i32 1, i32 2, i32 3, i32 4>
-; AVX512F-NEXT:    store <8 x i32> [[TMP4]], ptr [[T0:%.*]], align 4, !tbaa [[TBAA0]]
+; AVX512F-NEXT:    [[TMP2:%.*]] = shufflevector <8 x ptr> [[TMP1]], <8 x ptr> poison, <8 x i32> zeroinitializer
+; AVX512F-NEXT:    [[TMP3:%.*]] = getelementptr i32, <8 x ptr> [[TMP2]], <8 x i64> <i64 0, i64 11, i64 4, i64 15, i64 18, i64 9, i64 6, i64 21>
+; AVX512F-NEXT:    [[TMP4:%.*]] = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> [[TMP3]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> poison), !tbaa [[TBAA0]]
+; AVX512F-NEXT:    [[TMP5:%.*]] = add <8 x i32> [[TMP4]], <i32 1, i32 2, i32 3, i32 4, i32 1, i32 2, i32 3, i32 4>
+; AVX512F-NEXT:    store <8 x i32> [[TMP5]], ptr [[T0:%.*]], align 4, !tbaa [[TBAA0]]
 ; AVX512F-NEXT:    ret void
 ;
 ; AVX512VL-LABEL: @gather_load_4(
 ; AVX512VL-NEXT:    [[TMP1:%.*]] = insertelement <8 x ptr> poison, ptr [[T1:%.*]], i64 0
-; AVX512VL-NEXT:    [[SHUFFLE:%.*]] = shufflevector <8 x ptr> [[TMP1]], <8 x ptr> poison, <8 x i32> zeroinitializer
-; AVX512VL-NEXT:    [[TMP2:%.*]] = getelementptr i32, <8 x ptr> [[SHUFFLE]], <8 x i64> <i64 0, i64 11, i64 4, i64 15, i64 18, i64 9, i64 6, i64 21>
-; AVX512VL-NEXT:    [[TMP3:%.*]] = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> [[TMP2]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> poison), !tbaa [[TBAA0]]
-; AVX512VL-NEXT:    [[TMP4:%.*]] = add <8 x i32> [[TMP3]], <i32 1, i32 2, i32 3, i32 4, i32 1, i32 2, i32 3, i32 4>
-; AVX512VL-NEXT:    store <8 x i32> [[TMP4]], ptr [[T0:%.*]], align 4, !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP2:%.*]] = shufflevector <8 x ptr> [[TMP1]], <8 x ptr> poison, <8 x i32> zeroinitializer
+; AVX512VL-NEXT:    [[TMP3:%.*]] = getelementptr i32, <8 x ptr> [[TMP2]], <8 x i64> <i64 0, i64 11, i64 4, i64 15, i64 18, i64 9, i64 6, i64 21>
+; AVX512VL-NEXT:    [[TMP4:%.*]] = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> [[TMP3]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> poison), !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP5:%.*]] = add <8 x i32> [[TMP4]], <i32 1, i32 2, i32 3, i32 4, i32 1, i32 2, i32 3, i32 4>
+; AVX512VL-NEXT:    store <8 x i32> [[TMP5]], ptr [[T0:%.*]], align 4, !tbaa [[TBAA0]]
 ; AVX512VL-NEXT:    ret void
 ;
   %t5 = getelementptr inbounds i32, ptr %t0, i64 1
@@ -551,14 +558,15 @@ define void @gather_load_div(ptr noalias nocapture %0, ptr noalias nocapture rea
 ; SSE-NEXT:    [[TMP26:%.*]] = insertelement <4 x float> [[TMP25]], float [[TMP18]], i64 3
 ; SSE-NEXT:    [[TMP27:%.*]] = fdiv <4 x float> [[TMP22]], [[TMP26]]
 ; SSE-NEXT:    store <4 x float> [[TMP27]], ptr [[TMP0]], align 4, !tbaa [[TBAA0]]
-; SSE-NEXT:    [[TMP29:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 17
-; SSE-NEXT:    [[TMP30:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 33
-; SSE-NEXT:    [[TMP31:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 8
-; SSE-NEXT:    [[TMP32:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 30
-; SSE-NEXT:    [[TMP33:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 5
-; SSE-NEXT:    [[TMP34:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 27
-; SSE-NEXT:    [[TMP35:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 20
-; SSE-NEXT:    [[TMP36:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 23
+; SSE-NEXT:    [[TMP28:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 17
+; SSE-NEXT:    [[TMP29:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 33
+; SSE-NEXT:    [[TMP30:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 8
+; SSE-NEXT:    [[TMP31:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 30
+; SSE-NEXT:    [[TMP32:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 5
+; SSE-NEXT:    [[TMP33:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 27
+; SSE-NEXT:    [[TMP34:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 20
+; SSE-NEXT:    [[TMP35:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 23
+; SSE-NEXT:    [[TMP36:%.*]] = load float, ptr [[TMP28]], align 4, !tbaa [[TBAA0]]
 ; SSE-NEXT:    [[TMP37:%.*]] = load float, ptr [[TMP29]], align 4, !tbaa [[TBAA0]]
 ; SSE-NEXT:    [[TMP38:%.*]] = load float, ptr [[TMP30]], align 4, !tbaa [[TBAA0]]
 ; SSE-NEXT:    [[TMP39:%.*]] = load float, ptr [[TMP31]], align 4, !tbaa [[TBAA0]]
@@ -566,17 +574,16 @@ define void @gather_load_div(ptr noalias nocapture %0, ptr noalias nocapture rea
 ; SSE-NEXT:    [[TMP41:%.*]] = load float, ptr [[TMP33]], align 4, !tbaa [[TBAA0]]
 ; SSE-NEXT:    [[TMP42:%.*]] = load float, ptr [[TMP34]], align 4, !tbaa [[TBAA0]]
 ; SSE-NEXT:    [[TMP43:%.*]] = load float, ptr [[TMP35]], align 4, !tbaa [[TBAA0]]
-; SSE-NEXT:    [[TMP44:%.*]] = load float, ptr [[TMP36]], align 4, !tbaa [[TBAA0]]
-; SSE-NEXT:    [[TMP45:%.*]] = insertelement <4 x float> poison, float [[TMP37]], i64 0
-; SSE-NEXT:    [[TMP46:%.*]] = insertelement <4 x float> [[TMP45]], float [[TMP39]], i64 1
-; SSE-NEXT:    [[TMP47:%.*]] = insertelement <4 x float> [[TMP46]], float [[TMP41]], i64 2
-; SSE-NEXT:    [[TMP48:%.*]] = insertelement <4 x float> [[TMP47]], float [[TMP43]], i64 3
-; SSE-NEXT:    [[TMP49:%.*]] = insertelement <4 x float> poison, float [[TMP38]], i64 0
-; SSE-NEXT:    [[TMP50:%.*]] = insertelement <4 x float> [[TMP49]], float [[TMP40]], i64 1
-; SSE-NEXT:    [[TMP51:%.*]] = insertelement <4 x float> [[TMP50]], float [[TMP42]], i64 2
-; SSE-NEXT:    [[TMP52:%.*]] = insertelement <4 x float> [[TMP51]], float [[TMP44]], i64 3
-; SSE-NEXT:    [[TMP53:%.*]] = fdiv <4 x float> [[TMP48]], [[TMP52]]
-; SSE-NEXT:    store <4 x float> [[TMP53]], ptr [[TMP10]], align 4, !tbaa [[TBAA0]]
+; SSE-NEXT:    [[TMP44:%.*]] = insertelement <4 x float> poison, float [[TMP36]], i64 0
+; SSE-NEXT:    [[TMP45:%.*]] = insertelement <4 x float> [[TMP44]], float [[TMP38]], i64 1
+; SSE-NEXT:    [[TMP46:%.*]] = insertelement <4 x float> [[TMP45]], float [[TMP40]], i64 2
+; SSE-NEXT:    [[TMP47:%.*]] = insertelement <4 x float> [[TMP46]], float [[TMP42]], i64 3
+; SSE-NEXT:    [[TMP48:%.*]] = insertelement <4 x float> poison, float [[TMP37]], i64 0
+; SSE-NEXT:    [[TMP49:%.*]] = insertelement <4 x float> [[TMP48]], float [[TMP39]], i64 1
+; SSE-NEXT:    [[TMP50:%.*]] = insertelement <4 x float> [[TMP49]], float [[TMP41]], i64 2
+; SSE-NEXT:    [[TMP51:%.*]] = insertelement <4 x float> [[TMP50]], float [[TMP43]], i64 3
+; SSE-NEXT:    [[TMP52:%.*]] = fdiv <4 x float> [[TMP47]], [[TMP51]]
+; SSE-NEXT:    store <4 x float> [[TMP52]], ptr [[TMP10]], align 4, !tbaa [[TBAA0]]
 ; SSE-NEXT:    ret void
 ;
 ; AVX-LABEL: @gather_load_div(
@@ -685,24 +692,24 @@ define void @gather_load_div(ptr noalias nocapture %0, ptr noalias nocapture rea
 ;
 ; AVX512F-LABEL: @gather_load_div(
 ; AVX512F-NEXT:    [[TMP3:%.*]] = insertelement <8 x ptr> poison, ptr [[TMP1:%.*]], i64 0
-; AVX512F-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <8 x ptr> [[TMP3]], <8 x ptr> poison, <8 x i32> zeroinitializer
-; AVX512F-NEXT:    [[TMP4:%.*]] = getelementptr float, <8 x ptr> [[SHUFFLE1]], <8 x i64> <i64 4, i64 13, i64 11, i64 44, i64 33, i64 30, i64 27, i64 23>
-; AVX512F-NEXT:    [[TMP5:%.*]] = getelementptr float, <8 x ptr> [[SHUFFLE1]], <8 x i64> <i64 0, i64 10, i64 3, i64 14, i64 17, i64 8, i64 5, i64 20>
-; AVX512F-NEXT:    [[TMP6:%.*]] = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> [[TMP5]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x float> poison), !tbaa [[TBAA0]]
-; AVX512F-NEXT:    [[TMP7:%.*]] = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> [[TMP4]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x float> poison), !tbaa [[TBAA0]]
-; AVX512F-NEXT:    [[TMP8:%.*]] = fdiv <8 x float> [[TMP6]], [[TMP7]]
-; AVX512F-NEXT:    store <8 x float> [[TMP8]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
+; AVX512F-NEXT:    [[TMP4:%.*]] = shufflevector <8 x ptr> [[TMP3]], <8 x ptr> poison, <8 x i32> zeroinitializer
+; AVX512F-NEXT:    [[TMP5:%.*]] = getelementptr float, <8 x ptr> [[TMP4]], <8 x i64> <i64 4, i64 13, i64 11, i64 44, i64 33, i64 30, i64 27, i64 23>
+; AVX512F-NEXT:    [[TMP6:%.*]] = getelementptr float, <8 x ptr> [[TMP4]], <8 x i64> <i64 0, i64 10, i64 3, i64 14, i64 17, i64 8, i64 5, i64 20>
+; AVX512F-NEXT:    [[TMP7:%.*]] = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> [[TMP6]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x float> poison), !tbaa [[TBAA0]]
+; AVX512F-NEXT:    [[TMP8:%.*]] = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> [[TMP5]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x float> poison), !tbaa [[TBAA0]]
+; AVX512F-NEXT:    [[TMP9:%.*]] = fdiv <8 x float> [[TMP7]], [[TMP8]]
+; AVX512F-NEXT:    store <8 x float> [[TMP9]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
 ; AVX512F-NEXT:    ret void
 ;
 ; AVX512VL-LABEL: @gather_load_div(
 ; AVX512VL-NEXT:    [[TMP3:%.*]] = insertelement <8 x ptr> poison, ptr [[TMP1:%.*]], i64 0
-; AVX512VL-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <8 x ptr> [[TMP3]], <8 x ptr> poison, <8 x i32> zeroinitializer
-; AVX512VL-NEXT:    [[TMP4:%.*]] = getelementptr float, <8 x ptr> [[SHUFFLE1]], <8 x i64> <i64 4, i64 13, i64 11, i64 44, i64 33, i64 30, i64 27, i64 23>
-; AVX512VL-NEXT:    [[TMP5:%.*]] = getelementptr float, <8 x ptr> [[SHUFFLE1]], <8 x i64> <i64 0, i64 10, i64 3, i64 14, i64 17, i64 8, i64 5, i64 20>
-; AVX512VL-NEXT:    [[TMP6:%.*]] = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> [[TMP5]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x float> poison), !tbaa [[TBAA0]]
-; AVX512VL-NEXT:    [[TMP7:%.*]] = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> [[TMP4]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x float> poison), !tbaa [[TBAA0]]
-; AVX512VL-NEXT:    [[TMP8:%.*]] = fdiv <8 x float> [[TMP6]], [[TMP7]]
-; AVX512VL-NEXT:    store <8 x float> [[TMP8]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP4:%.*]] = shufflevector <8 x ptr> [[TMP3]], <8 x ptr> poison, <8 x i32> zeroinitializer
+; AVX512VL-NEXT:    [[TMP5:%.*]] = getelementptr float, <8 x ptr> [[TMP4]], <8 x i64> <i64 4, i64 13, i64 11, i64 44, i64 33, i64 30, i64 27, i64 23>
+; AVX512VL-NEXT:    [[TMP6:%.*]] = getelementptr float, <8 x ptr> [[TMP4]], <8 x i64> <i64 0, i64 10, i64 3, i64 14, i64 17, i64 8, i64 5, i64 20>
+; AVX512VL-NEXT:    [[TMP7:%.*]] = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> [[TMP6]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x float> poison), !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP8:%.*]] = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> [[TMP5]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x float> poison), !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP9:%.*]] = fdiv <8 x float> [[TMP7]], [[TMP8]]
+; AVX512VL-NEXT:    store <8 x float> [[TMP9]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
 ; AVX512VL-NEXT:    ret void
 ;
   %3 = load float, ptr %1, align 4, !tbaa !2
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/pr47629.ll b/llvm/test/Transforms/SLPVectorizer/X86/pr47629.ll
index a1d08c987541b..7597cf14f7651 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/pr47629.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/pr47629.ll
@@ -71,12 +71,19 @@ define void @gather_load(ptr noalias nocapture %0, ptr noalias nocapture readonl
 ; AVX512F-NEXT:    ret void
 ;
 ; AVX512VL-LABEL: @gather_load(
-; AVX512VL-NEXT:    [[TMP3:%.*]] = insertelement <4 x ptr> poison, ptr [[TMP1:%.*]], i64 0
-; AVX512VL-NEXT:    [[SHUFFLE:%.*]] = shufflevector <4 x ptr> [[TMP3]], <4 x ptr> poison, <4 x i32> zeroinitializer
-; AVX512VL-NEXT:    [[TMP4:%.*]] = getelementptr i32, <4 x ptr> [[SHUFFLE]], <4 x i64> <i64 0, i64 11, i64 4, i64 1>
-; AVX512VL-NEXT:    [[TMP5:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> [[TMP4]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> poison), !tbaa [[TBAA0:![0-9]+]]
-; AVX512VL-NEXT:    [[TMP6:%.*]] = add nsw <4 x i32> [[TMP5]], <i32 1, i32 2, i32 3, i32 4>
-; AVX512VL-NEXT:    store <4 x i32> [[TMP6]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP3:%.*]] = getelementptr inbounds i32, ptr [[TMP1:%.*]], i64 1
+; AVX512VL-NEXT:    [[TMP4:%.*]] = load i32, ptr [[TMP1]], align 4, !tbaa [[TBAA0:![0-9]+]]
+; AVX512VL-NEXT:    [[TMP5:%.*]] = getelementptr inbounds i32, ptr [[TMP1]], i64 11
+; AVX512VL-NEXT:    [[TMP6:%.*]] = load i32, ptr [[TMP5]], align 4, !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP7:%.*]] = getelementptr inbounds i32, ptr [[TMP1]], i64 4
+; AVX512VL-NEXT:    [[TMP8:%.*]] = load i32, ptr [[TMP7]], align 4, !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP9:%.*]] = load i32, ptr [[TMP3]], align 4, !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP10:%.*]] = insertelement <4 x i32> undef, i32 [[TMP4]], i64 0
+; AVX512VL-NEXT:    [[TMP11:%.*]] = insertelement <4 x i32> [[TMP10]], i32 [[TMP6]], i64 1
+; AVX512VL-NEXT:    [[TMP12:%.*]] = insertelement <4 x i32> [[TMP11]], i32 [[TMP8]], i64 2
+; AVX512VL-NEXT:    [[TMP13:%.*]] = insertelement <4 x i32> [[TMP12]], i32 [[TMP9]], i64 3
+; AVX512VL-NEXT:    [[TMP14:%.*]] = add nsw <4 x i32> [[TMP13]], <i32 1, i32 2, i32 3, i32 4>
+; AVX512VL-NEXT:    store <4 x i32> [[TMP14]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
 ; AVX512VL-NEXT:    ret void
 ;
   %3 = getelementptr inbounds i32, ptr %1, i64 1
@@ -174,11 +181,11 @@ define void @gather_load_2(ptr noalias nocapture %0, ptr noalias nocapture reado
 ;
 ; AVX512VL-LABEL: @gather_load_2(
 ; AVX512VL-NEXT:    [[TMP3:%.*]] = insertelement <4 x ptr> poison, ptr [[TMP1:%.*]], i64 0
-; AVX512VL-NEXT:    [[SHUFFLE:%.*]] = shufflevector <4 x ptr> [[TMP3]], <4 x ptr> poison, <4 x i32> zeroinitializer
-; AVX512VL-NEXT:    [[TMP4:%.*]] = getelementptr i32, <4 x ptr> [[SHUFFLE]], <4 x i64> <i64 1, i64 10, i64 3, i64 5>
-; AVX512VL-NEXT:    [[TMP5:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> [[TMP4]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> poison), !tbaa [[TBAA0]]
-; AVX512VL-NEXT:    [[TMP6:%.*]] = add nsw <4 x i32> [[TMP5]], <i32 1, i32 2, i32 3, i32 4>
-; AVX512VL-NEXT:    store <4 x i32> [[TMP6]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP4:%.*]] = shufflevector <4 x ptr> [[TMP3]], <4 x ptr> poison, <4 x i32> zeroinitializer
+; AVX512VL-NEXT:    [[TMP5:%.*]] = getelementptr i32, <4 x ptr> [[TMP4]], <4 x i64> <i64 1, i64 10, i64 3, i64 5>
+; AVX512VL-NEXT:    [[TMP6:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> [[TMP5]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> poison), !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP7:%.*]] = add nsw <4 x i32> [[TMP6]], <i32 1, i32 2, i32 3, i32 4>
+; AVX512VL-NEXT:    store <4 x i32> [[TMP7]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
 ; AVX512VL-NEXT:    ret void
 ;
   %3 = getelementptr inbounds i32, ptr %1, i64 1
@@ -304,20 +311,20 @@ define void @gather_load_3(ptr noalias nocapture %0, ptr noalias nocapture reado
 ;
 ; AVX512F-LABEL: @gather_load_3(
 ; AVX512F-NEXT:    [[TMP3:%.*]] = insertelement <8 x ptr> poison, ptr [[TMP1:%.*]], i64 0
-; AVX512F-NEXT:    [[SHUFFLE:%.*]] = shufflevector <8 x ptr> [[TMP3]], <8 x ptr> poison, <8 x i32> zeroinitializer
-; AVX512F-NEXT:    [[TMP4:%.*]] = getelementptr i32, <8 x ptr> [[SHUFFLE]], <8 x i64> <i64 0, i64 11, i64 4, i64 15, i64 18, i64 9, i64 6, i64 21>
-; AVX512F-NEXT:    [[TMP5:%.*]] = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> [[TMP4]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> poison), !tbaa [[TBAA0]]
-; AVX512F-NEXT:    [[TMP6:%.*]] = add <8 x i32> [[TMP5]], <i32 1, i32 2, i32 3, i32 4, i32 1, i32 2, i32 3, i32 4>
-; AVX512F-NEXT:    store <8 x i32> [[TMP6]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
+; AVX512F-NEXT:    [[TMP4:%.*]] = shufflevector <8 x ptr> [[TMP3]], <8 x ptr> poison, <8 x i32> zeroinitializer
+; AVX512F-NEXT:    [[TMP5:%.*]] = getelementptr i32, <8 x ptr> [[TMP4]], <8 x i64> <i64 0, i64 11, i64 4, i64 15, i64 18, i64 9, i64 6, i64 21>
+; AVX512F-NEXT:    [[TMP6:%.*]] = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> [[TMP5]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> poison), !tbaa [[TBAA0]]
+; AVX512F-NEXT:    [[TMP7:%.*]] = add <8 x i32> [[TMP6]], <i32 1, i32 2, i32 3, i32 4, i32 1, i32 2, i32 3, i32 4>
+; AVX512F-NEXT:    store <8 x i32> [[TMP7]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
 ; AVX512F-NEXT:    ret void
 ;
 ; AVX512VL-LABEL: @gather_load_3(
 ; AVX512VL-NEXT:    [[TMP3:%.*]] = insertelement <8 x ptr> poison, ptr [[TMP1:%.*]], i64 0
-; AVX512VL-NEXT:    [[SHUFFLE:%.*]] = shufflevector <8 x ptr> [[TMP3]], <8 x ptr> poison, <8 x i32> zeroinitializer
-; AVX512VL-NEXT:    [[TMP4:%.*]] = getelementptr i32, <8 x ptr> [[SHUFFLE]], <8 x i64> <i64 0, i64 11, i64 4, i64 15, i64 18, i64 9, i64 6, i64 21>
-; AVX512VL-NEXT:    [[TMP5:%.*]] = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> [[TMP4]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> poison), !tbaa [[TBAA0]]
-; AVX512VL-NEXT:    [[TMP6:%.*]] = add <8 x i32> [[TMP5]], <i32 1, i32 2, i32 3, i32 4, i32 1, i32 2, i32 3, i32 4>
-; AVX512VL-NEXT:    store <8 x i32> [[TMP6]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP4:%.*]] = shufflevector <8 x ptr> [[TMP3]], <8 x ptr> poison, <8 x i32> zeroinitializer
+; AVX512VL-NEXT:    [[TMP5:%.*]] = getelementptr i32, <8 x ptr> [[TMP4]], <8 x i64> <i64 0, i64 11, i64 4, i64 15, i64 18, i64 9, i64 6, i64 21>
+; AVX512VL-NEXT:    [[TMP6:%.*]] = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> [[TMP5]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> poison), !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP7:%.*]] = add <8 x i32> [[TMP6]], <i32 1, i32 2, i32 3, i32 4, i32 1, i32 2, i32 3, i32 4>
+; AVX512VL-NEXT:    store <8 x i32> [[TMP7]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
 ; AVX512VL-NEXT:    ret void
 ;
   %3 = load i32, ptr %1, align 4, !tbaa !2
@@ -461,20 +468,20 @@ define void @gather_load_4(ptr noalias nocapture %t0, ptr noalias nocapture read
 ;
 ; AVX512F-LABEL: @gather_load_4(
 ; AVX512F-NEXT:    [[TMP1:%.*]] = insertelement <8 x ptr> poison, ptr [[T1:%.*]], i64 0
-; AVX512F-NEXT:    [[SHUFFLE:%.*]] = shufflevector <8 x ptr> [[TMP1]], <8 x ptr> poison, <8 x i32> zeroinitializer
-; AVX512F-NEXT:    [[TMP2:%.*]] = getelementptr i32, <8 x ptr> [[SHUFFLE]], <8 x i64> <i64 0, i64 11, i64 4, i64 15, i64 18, i64 9, i64 6, i64 21>
-; AVX512F-NEXT:    [[TMP3:%.*]] = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> [[TMP2]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> poison), !tbaa [[TBAA0]]
-; AVX512F-NEXT:    [[TMP4:%.*]] = add <8 x i32> [[TMP3]], <i32 1, i32 2, i32 3, i32 4, i32 1, i32 2, i32 3, i32 4>
-; AVX512F-NEXT:    store <8 x i32> [[TMP4]], ptr [[T0:%.*]], align 4, !tbaa [[TBAA0]]
+; AVX512F-NEXT:    [[TMP2:%.*]] = shufflevector <8 x ptr> [[TMP1]], <8 x ptr> poison, <8 x i32> zeroinitializer
+; AVX512F-NEXT:    [[TMP3:%.*]] = getelementptr i32, <8 x ptr> [[TMP2]], <8 x i64> <i64 0, i64 11, i64 4, i64 15, i64 18, i64 9, i64 6, i64 21>
+; AVX512F-NEXT:    [[TMP4:%.*]] = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> [[TMP3]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> poison), !tbaa [[TBAA0]]
+; AVX512F-NEXT:    [[TMP5:%.*]] = add <8 x i32> [[TMP4]], <i32 1, i32 2, i32 3, i32 4, i32 1, i32 2, i32 3, i32 4>
+; AVX512F-NEXT:    store <8 x i32> [[TMP5]], ptr [[T0:%.*]], align 4, !tbaa [[TBAA0]]
 ; AVX512F-NEXT:    ret void
 ;
 ; AVX512VL-LABEL: @gather_load_4(
 ; AVX512VL-NEXT:    [[TMP1:%.*]] = insertelement <8 x ptr> poison, ptr [[T1:%.*]], i64 0
-; AVX512VL-NEXT:    [[SHUFFLE:%.*]] = shufflevector <8 x ptr> [[TMP1]], <8 x ptr> poison, <8 x i32> zeroinitializer
-; AVX512VL-NEXT:    [[TMP2:%.*]] = getelementptr i32, <8 x ptr> [[SHUFFLE]], <8 x i64> <i64 0, i64 11, i64 4, i64 15, i64 18, i64 9, i64 6, i64 21>
-; AVX512VL-NEXT:    [[TMP3:%.*]] = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> [[TMP2]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> poison), !tbaa [[TBAA0]]
-; AVX512VL-NEXT:    [[TMP4:%.*]] = add <8 x i32> [[TMP3]], <i32 1, i32 2, i32 3, i32 4, i32 1, i32 2, i32 3, i32 4>
-; AVX512VL-NEXT:    store <8 x i32> [[TMP4]], ptr [[T0:%.*]], align 4, !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP2:%.*]] = shufflevector <8 x ptr> [[TMP1]], <8 x ptr> poison, <8 x i32> zeroinitializer
+; AVX512VL-NEXT:    [[TMP3:%.*]] = getelementptr i32, <8 x ptr> [[TMP2]], <8 x i64> <i64 0, i64 11, i64 4, i64 15, i64 18, i64 9, i64 6, i64 21>
+; AVX512VL-NEXT:    [[TMP4:%.*]] = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> [[TMP3]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> poison), !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP5:%.*]] = add <8 x i32> [[TMP4]], <i32 1, i32 2, i32 3, i32 4, i32 1, i32 2, i32 3, i32 4>
+; AVX512VL-NEXT:    store <8 x i32> [[TMP5]], ptr [[T0:%.*]], align 4, !tbaa [[TBAA0]]
 ; AVX512VL-NEXT:    ret void
 ;
   %t5 = getelementptr inbounds i32, ptr %t0, i64 1
@@ -551,14 +558,15 @@ define void @gather_load_div(ptr noalias nocapture %0, ptr noalias nocapture rea
 ; SSE-NEXT:    [[TMP26:%.*]] = insertelement <4 x float> [[TMP25]], float [[TMP18]], i64 3
 ; SSE-NEXT:    [[TMP27:%.*]] = fdiv <4 x float> [[TMP22]], [[TMP26]]
 ; SSE-NEXT:    store <4 x float> [[TMP27]], ptr [[TMP0]], align 4, !tbaa [[TBAA0]]
-; SSE-NEXT:    [[TMP29:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 17
-; SSE-NEXT:    [[TMP30:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 33
-; SSE-NEXT:    [[TMP31:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 8
-; SSE-NEXT:    [[TMP32:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 30
-; SSE-NEXT:    [[TMP33:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 5
-; SSE-NEXT:    [[TMP34:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 27
-; SSE-NEXT:    [[TMP35:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 20
-; SSE-NEXT:    [[TMP36:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 23
+; SSE-NEXT:    [[TMP28:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 17
+; SSE-NEXT:    [[TMP29:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 33
+; SSE-NEXT:    [[TMP30:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 8
+; SSE-NEXT:    [[TMP31:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 30
+; SSE-NEXT:    [[TMP32:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 5
+; SSE-NEXT:    [[TMP33:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 27
+; SSE-NEXT:    [[TMP34:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 20
+; SSE-NEXT:    [[TMP35:%.*]] = getelementptr inbounds float, ptr [[TMP1]], i64 23
+; SSE-NEXT:    [[TMP36:%.*]] = load float, ptr [[TMP28]], align 4, !tbaa [[TBAA0]]
 ; SSE-NEXT:    [[TMP37:%.*]] = load float, ptr [[TMP29]], align 4, !tbaa [[TBAA0]]
 ; SSE-NEXT:    [[TMP38:%.*]] = load float, ptr [[TMP30]], align 4, !tbaa [[TBAA0]]
 ; SSE-NEXT:    [[TMP39:%.*]] = load float, ptr [[TMP31]], align 4, !tbaa [[TBAA0]]
@@ -566,17 +574,16 @@ define void @gather_load_div(ptr noalias nocapture %0, ptr noalias nocapture rea
 ; SSE-NEXT:    [[TMP41:%.*]] = load float, ptr [[TMP33]], align 4, !tbaa [[TBAA0]]
 ; SSE-NEXT:    [[TMP42:%.*]] = load float, ptr [[TMP34]], align 4, !tbaa [[TBAA0]]
 ; SSE-NEXT:    [[TMP43:%.*]] = load float, ptr [[TMP35]], align 4, !tbaa [[TBAA0]]
-; SSE-NEXT:    [[TMP44:%.*]] = load float, ptr [[TMP36]], align 4, !tbaa [[TBAA0]]
-; SSE-NEXT:    [[TMP45:%.*]] = insertelement <4 x float> poison, float [[TMP37]], i64 0
-; SSE-NEXT:    [[TMP46:%.*]] = insertelement <4 x float> [[TMP45]], float [[TMP39]], i64 1
-; SSE-NEXT:    [[TMP47:%.*]] = insertelement <4 x float> [[TMP46]], float [[TMP41]], i64 2
-; SSE-NEXT:    [[TMP48:%.*]] = insertelement <4 x float> [[TMP47]], float [[TMP43]], i64 3
-; SSE-NEXT:    [[TMP49:%.*]] = insertelement <4 x float> poison, float [[TMP38]], i64 0
-; SSE-NEXT:    [[TMP50:%.*]] = insertelement <4 x float> [[TMP49]], float [[TMP40]], i64 1
-; SSE-NEXT:    [[TMP51:%.*]] = insertelement <4 x float> [[TMP50]], float [[TMP42]], i64 2
-; SSE-NEXT:    [[TMP52:%.*]] = insertelement <4 x float> [[TMP51]], float [[TMP44]], i64 3
-; SSE-NEXT:    [[TMP53:%.*]] = fdiv <4 x float> [[TMP48]], [[TMP52]]
-; SSE-NEXT:    store <4 x float> [[TMP53]], ptr [[TMP10]], align 4, !tbaa [[TBAA0]]
+; SSE-NEXT:    [[TMP44:%.*]] = insertelement <4 x float> poison, float [[TMP36]], i64 0
+; SSE-NEXT:    [[TMP45:%.*]] = insertelement <4 x float> [[TMP44]], float [[TMP38]], i64 1
+; SSE-NEXT:    [[TMP46:%.*]] = insertelement <4 x float> [[TMP45]], float [[TMP40]], i64 2
+; SSE-NEXT:    [[TMP47:%.*]] = insertelement <4 x float> [[TMP46]], float [[TMP42]], i64 3
+; SSE-NEXT:    [[TMP48:%.*]] = insertelement <4 x float> poison, float [[TMP37]], i64 0
+; SSE-NEXT:    [[TMP49:%.*]] = insertelement <4 x float> [[TMP48]], float [[TMP39]], i64 1
+; SSE-NEXT:    [[TMP50:%.*]] = insertelement <4 x float> [[TMP49]], float [[TMP41]], i64 2
+; SSE-NEXT:    [[TMP51:%.*]] = insertelement <4 x float> [[TMP50]], float [[TMP43]], i64 3
+; SSE-NEXT:    [[TMP52:%.*]] = fdiv <4 x float> [[TMP47]], [[TMP51]]
+; SSE-NEXT:    store <4 x float> [[TMP52]], ptr [[TMP10]], align 4, !tbaa [[TBAA0]]
 ; SSE-NEXT:    ret void
 ;
 ; AVX-LABEL: @gather_load_div(
@@ -685,24 +692,24 @@ define void @gather_load_div(ptr noalias nocapture %0, ptr noalias nocapture rea
 ;
 ; AVX512F-LABEL: @gather_load_div(
 ; AVX512F-NEXT:    [[TMP3:%.*]] = insertelement <8 x ptr> poison, ptr [[TMP1:%.*]], i64 0
-; AVX512F-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <8 x ptr> [[TMP3]], <8 x ptr> poison, <8 x i32> zeroinitializer
-; AVX512F-NEXT:    [[TMP4:%.*]] = getelementptr float, <8 x ptr> [[SHUFFLE1]], <8 x i64> <i64 4, i64 13, i64 11, i64 44, i64 33, i64 30, i64 27, i64 23>
-; AVX512F-NEXT:    [[TMP5:%.*]] = getelementptr float, <8 x ptr> [[SHUFFLE1]], <8 x i64> <i64 0, i64 10, i64 3, i64 14, i64 17, i64 8, i64 5, i64 20>
-; AVX512F-NEXT:    [[TMP6:%.*]] = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> [[TMP5]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x float> poison), !tbaa [[TBAA0]]
-; AVX512F-NEXT:    [[TMP7:%.*]] = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> [[TMP4]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x float> poison), !tbaa [[TBAA0]]
-; AVX512F-NEXT:    [[TMP8:%.*]] = fdiv <8 x float> [[TMP6]], [[TMP7]]
-; AVX512F-NEXT:    store <8 x float> [[TMP8]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
+; AVX512F-NEXT:    [[TMP4:%.*]] = shufflevector <8 x ptr> [[TMP3]], <8 x ptr> poison, <8 x i32> zeroinitializer
+; AVX512F-NEXT:    [[TMP5:%.*]] = getelementptr float, <8 x ptr> [[TMP4]], <8 x i64> <i64 4, i64 13, i64 11, i64 44, i64 33, i64 30, i64 27, i64 23>
+; AVX512F-NEXT:    [[TMP6:%.*]] = getelementptr float, <8 x ptr> [[TMP4]], <8 x i64> <i64 0, i64 10, i64 3, i64 14, i64 17, i64 8, i64 5, i64 20>
+; AVX512F-NEXT:    [[TMP7:%.*]] = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> [[TMP6]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x float> poison), !tbaa [[TBAA0]]
+; AVX512F-NEXT:    [[TMP8:%.*]] = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> [[TMP5]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x float> poison), !tbaa [[TBAA0]]
+; AVX512F-NEXT:    [[TMP9:%.*]] = fdiv <8 x float> [[TMP7]], [[TMP8]]
+; AVX512F-NEXT:    store <8 x float> [[TMP9]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
 ; AVX512F-NEXT:    ret void
 ;
 ; AVX512VL-LABEL: @gather_load_div(
 ; AVX512VL-NEXT:    [[TMP3:%.*]] = insertelement <8 x ptr> poison, ptr [[TMP1:%.*]], i64 0
-; AVX512VL-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <8 x ptr> [[TMP3]], <8 x ptr> poison, <8 x i32> zeroinitializer
-; AVX512VL-NEXT:    [[TMP4:%.*]] = getelementptr float, <8 x ptr> [[SHUFFLE1]], <8 x i64> <i64 4, i64 13, i64 11, i64 44, i64 33, i64 30, i64 27, i64 23>
-; AVX512VL-NEXT:    [[TMP5:%.*]] = getelementptr float, <8 x ptr> [[SHUFFLE1]], <8 x i64> <i64 0, i64 10, i64 3, i64 14, i64 17, i64 8, i64 5, i64 20>
-; AVX512VL-NEXT:    [[TMP6:%.*]] = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> [[TMP5]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x float> poison), !tbaa [[TBAA0]]
-; AVX512VL-NEXT:    [[TMP7:%.*]] = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> [[TMP4]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x float> poison), !tbaa [[TBAA0]]
-; AVX512VL-NEXT:    [[TMP8:%.*]] = fdiv <8 x float> [[TMP6]], [[TMP7]]
-; AVX512VL-NEXT:    store <8 x float> [[TMP8]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP4:%.*]] = shufflevector <8 x ptr> [[TMP3]], <8 x ptr> poison, <8 x i32> zeroinitializer
+; AVX512VL-NEXT:    [[TMP5:%.*]] = getelementptr float, <8 x ptr> [[TMP4]], <8 x i64> <i64 4, i64 13, i64 11, i64 44, i64 33, i64 30, i64 27, i64 23>
+; AVX512VL-NEXT:    [[TMP6:%.*]] = getelementptr float, <8 x ptr> [[TMP4]], <8 x i64> <i64 0, i64 10, i64 3, i64 14, i64 17, i64 8, i64 5, i64 20>
+; AVX512VL-NEXT:    [[TMP7:%.*]] = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> [[TMP6]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x float> poison), !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP8:%.*]] = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> [[TMP5]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x float> poison), !tbaa [[TBAA0]]
+; AVX512VL-NEXT:    [[TMP9:%.*]] = fdiv <8 x float> [[TMP7]], [[TMP8]]
+; AVX512VL-NEXT:    store <8 x float> [[TMP9]], ptr [[TMP0:%.*]], align 4, !tbaa [[TBAA0]]
 ; AVX512VL-NEXT:    ret void
 ;
   %3 = load float, ptr %1, align 4, !tbaa !2
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/pr49081.ll b/llvm/test/Transforms/SLPVectorizer/X86/pr49081.ll
index ccc559d30b6b3..d91470c50992e 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/pr49081.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/pr49081.ll
@@ -9,7 +9,7 @@ define dso_local <4 x float> @foo(<4 x i32> %0) {
 ; CHECK-NEXT:    [[TMP3:%.*]] = sitofp i32 [[TMP2]] to float
 ; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <4 x float> undef, float [[TMP3]], i64 0
 ; CHECK-NEXT:    [[TMP5:%.*]] = shufflevector <4 x float> [[TMP4]], <4 x float> poison, <4 x i32> <i32 0, i32 0, i32 undef, i32 undef>
-; CHECK-NEXT:    [[TMP6:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> undef, <2 x i32> <i32 2, i32 3>
+; CHECK-NEXT:    [[TMP6:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <2 x i32> <i32 2, i32 3>
 ; CHECK-NEXT:    [[TMP7:%.*]] = sitofp <2 x i32> [[TMP6]] to <2 x float>
 ; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <2 x float> [[TMP7]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
 ; CHECK-NEXT:    [[TMP9:%.*]] = shufflevector <4 x float> [[TMP5]], <4 x float> [[TMP8]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/reduce-constants.ll b/llvm/test/Transforms/SLPVectorizer/X86/reduce-constants.ll
new file mode 100644
index 0000000000000..c91b9b0004028
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/X86/reduce-constants.ll
@@ -0,0 +1,18 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -S -mtriple=x86_64 -passes=slp-vectorizer < %s | FileCheck %s
+
+define double @test(double %m, double %a) {
+; CHECK-LABEL: @test(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = call fast double @llvm.maxnum.f64(double 0.000000e+00, double [[M:%.*]])
+; CHECK-NEXT:    [[TMP1:%.*]] = call fast double @llvm.maxnum.f64(double [[TMP0]], double [[A:%.*]])
+; CHECK-NEXT:    ret double [[TMP1]]
+;
+entry:
+  %c = tail call fast double @llvm.maxnum.f64(double %m, double 0.000000e+00)
+  %c1 = tail call fast double @llvm.maxnum.f64(double %a, double %c)
+  %c2 = tail call fast double @llvm.maxnum.f64(double %c1, double 0.000000e+00)
+  ret double %c2
+}
+
+declare double @llvm.maxnum.f64(double, double)
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/reduction-logical.ll b/llvm/test/Transforms/SLPVectorizer/X86/reduction-logical.ll
index 94dfb4de871b3..97fb5686c8710 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/reduction-logical.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/reduction-logical.ll
@@ -95,19 +95,16 @@ define i1 @logical_or_fcmp(<4 x float> %x) {
 define i1 @logical_and_icmp_diff_preds(<4 x i32> %x) {
 ; SSE-LABEL: @logical_and_icmp_diff_preds(
 ; SSE-NEXT:    [[X0:%.*]] = extractelement <4 x i32> [[X:%.*]], i32 0
-; SSE-NEXT:    [[X1:%.*]] = extractelement <4 x i32> [[X]], i32 1
 ; SSE-NEXT:    [[X2:%.*]] = extractelement <4 x i32> [[X]], i32 2
-; SSE-NEXT:    [[X3:%.*]] = extractelement <4 x i32> [[X]], i32 3
 ; SSE-NEXT:    [[C0:%.*]] = icmp ult i32 [[X0]], 0
 ; SSE-NEXT:    [[C2:%.*]] = icmp sgt i32 [[X2]], 0
-; SSE-NEXT:    [[TMP1:%.*]] = insertelement <2 x i32> poison, i32 [[X3]], i32 0
-; SSE-NEXT:    [[TMP2:%.*]] = insertelement <2 x i32> [[TMP1]], i32 [[X1]], i32 1
-; SSE-NEXT:    [[TMP3:%.*]] = icmp slt <2 x i32> [[TMP2]], zeroinitializer
-; SSE-NEXT:    [[TMP4:%.*]] = extractelement <2 x i1> [[TMP3]], i32 1
-; SSE-NEXT:    [[S1:%.*]] = select i1 [[C0]], i1 [[TMP4]], i1 false
+; SSE-NEXT:    [[TMP1:%.*]] = shufflevector <4 x i32> [[X]], <4 x i32> poison, <2 x i32> <i32 3, i32 1>
+; SSE-NEXT:    [[TMP2:%.*]] = icmp slt <2 x i32> [[TMP1]], zeroinitializer
+; SSE-NEXT:    [[TMP3:%.*]] = extractelement <2 x i1> [[TMP2]], i32 1
+; SSE-NEXT:    [[S1:%.*]] = select i1 [[C0]], i1 [[TMP3]], i1 false
 ; SSE-NEXT:    [[S2:%.*]] = select i1 [[S1]], i1 [[C2]], i1 false
-; SSE-NEXT:    [[TMP5:%.*]] = extractelement <2 x i1> [[TMP3]], i32 0
-; SSE-NEXT:    [[S3:%.*]] = select i1 [[S2]], i1 [[TMP5]], i1 false
+; SSE-NEXT:    [[TMP4:%.*]] = extractelement <2 x i1> [[TMP2]], i32 0
+; SSE-NEXT:    [[S3:%.*]] = select i1 [[S2]], i1 [[TMP4]], i1 false
 ; SSE-NEXT:    ret i1 [[S3]]
 ;
 ; AVX-LABEL: @logical_and_icmp_diff_preds(
@@ -187,16 +184,13 @@ define i1 @mixed_logical_icmp(<4 x i32> %x) {
 
 define i1 @logical_and_icmp_subvec(<4 x i32> %x) {
 ; SSE-LABEL: @logical_and_icmp_subvec(
-; SSE-NEXT:    [[X0:%.*]] = extractelement <4 x i32> [[X:%.*]], i32 0
-; SSE-NEXT:    [[X1:%.*]] = extractelement <4 x i32> [[X]], i32 1
-; SSE-NEXT:    [[X2:%.*]] = extractelement <4 x i32> [[X]], i32 2
-; SSE-NEXT:    [[TMP1:%.*]] = insertelement <2 x i32> poison, i32 [[X1]], i32 0
-; SSE-NEXT:    [[TMP2:%.*]] = insertelement <2 x i32> [[TMP1]], i32 [[X0]], i32 1
-; SSE-NEXT:    [[TMP3:%.*]] = icmp slt <2 x i32> [[TMP2]], zeroinitializer
+; SSE-NEXT:    [[X2:%.*]] = extractelement <4 x i32> [[X:%.*]], i32 2
+; SSE-NEXT:    [[TMP1:%.*]] = shufflevector <4 x i32> [[X]], <4 x i32> poison, <2 x i32> <i32 1, i32 0>
+; SSE-NEXT:    [[TMP2:%.*]] = icmp slt <2 x i32> [[TMP1]], zeroinitializer
 ; SSE-NEXT:    [[C2:%.*]] = icmp slt i32 [[X2]], 0
-; SSE-NEXT:    [[TMP4:%.*]] = extractelement <2 x i1> [[TMP3]], i32 0
-; SSE-NEXT:    [[TMP5:%.*]] = extractelement <2 x i1> [[TMP3]], i32 1
-; SSE-NEXT:    [[S1:%.*]] = select i1 [[TMP5]], i1 [[TMP4]], i1 false
+; SSE-NEXT:    [[TMP3:%.*]] = extractelement <2 x i1> [[TMP2]], i32 0
+; SSE-NEXT:    [[TMP4:%.*]] = extractelement <2 x i1> [[TMP2]], i32 1
+; SSE-NEXT:    [[S1:%.*]] = select i1 [[TMP4]], i1 [[TMP3]], i1 false
 ; SSE-NEXT:    [[S2:%.*]] = select i1 [[S1]], i1 [[C2]], i1 false
 ; SSE-NEXT:    ret i1 [[S2]]
 ;
@@ -228,13 +222,13 @@ define i1 @logical_and_icmp_subvec(<4 x i32> %x) {
 
 define i1 @logical_and_icmp_clamp(<4 x i32> %x) {
 ; CHECK-LABEL: @logical_and_icmp_clamp(
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <4 x i32> [[X:%.*]], <4 x i32> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 0, i32 1, i32 2, i32 3>
-; CHECK-NEXT:    [[TMP1:%.*]] = icmp sgt <8 x i32> [[SHUFFLE]], <i32 17, i32 17, i32 17, i32 17, i32 42, i32 42, i32 42, i32 42>
-; CHECK-NEXT:    [[TMP2:%.*]] = icmp slt <8 x i32> [[SHUFFLE]], <i32 17, i32 17, i32 17, i32 17, i32 42, i32 42, i32 42, i32 42>
-; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <8 x i1> [[TMP1]], <8 x i1> [[TMP2]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 12, i32 13, i32 14, i32 15>
-; CHECK-NEXT:    [[TMP4:%.*]] = freeze <8 x i1> [[TMP3]]
-; CHECK-NEXT:    [[TMP5:%.*]] = call i1 @llvm.vector.reduce.and.v8i1(<8 x i1> [[TMP4]])
-; CHECK-NEXT:    ret i1 [[TMP5]]
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x i32> [[X:%.*]], <4 x i32> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 0, i32 1, i32 2, i32 3>
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp sgt <8 x i32> [[TMP1]], <i32 17, i32 17, i32 17, i32 17, i32 42, i32 42, i32 42, i32 42>
+; CHECK-NEXT:    [[TMP3:%.*]] = icmp slt <8 x i32> [[TMP1]], <i32 17, i32 17, i32 17, i32 17, i32 42, i32 42, i32 42, i32 42>
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <8 x i1> [[TMP2]], <8 x i1> [[TMP3]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 12, i32 13, i32 14, i32 15>
+; CHECK-NEXT:    [[TMP5:%.*]] = freeze <8 x i1> [[TMP4]]
+; CHECK-NEXT:    [[TMP6:%.*]] = call i1 @llvm.vector.reduce.and.v8i1(<8 x i1> [[TMP5]])
+; CHECK-NEXT:    ret i1 [[TMP6]]
 ;
   %x0 = extractelement <4 x i32> %x, i32 0
   %x1 = extractelement <4 x i32> %x, i32 1
@@ -260,15 +254,15 @@ define i1 @logical_and_icmp_clamp(<4 x i32> %x) {
 
 define i1 @logical_and_icmp_clamp_extra_use_cmp(<4 x i32> %x) {
 ; CHECK-LABEL: @logical_and_icmp_clamp_extra_use_cmp(
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <4 x i32> [[X:%.*]], <4 x i32> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 0, i32 1, i32 2, i32 3>
-; CHECK-NEXT:    [[TMP1:%.*]] = icmp sgt <8 x i32> [[SHUFFLE]], <i32 17, i32 17, i32 17, i32 17, i32 42, i32 42, i32 42, i32 42>
-; CHECK-NEXT:    [[TMP2:%.*]] = icmp slt <8 x i32> [[SHUFFLE]], <i32 17, i32 17, i32 17, i32 17, i32 42, i32 42, i32 42, i32 42>
-; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <8 x i1> [[TMP1]], <8 x i1> [[TMP2]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 12, i32 13, i32 14, i32 15>
-; CHECK-NEXT:    [[TMP4:%.*]] = extractelement <8 x i1> [[TMP3]], i32 6
-; CHECK-NEXT:    call void @use1(i1 [[TMP4]])
-; CHECK-NEXT:    [[TMP5:%.*]] = freeze <8 x i1> [[TMP3]]
-; CHECK-NEXT:    [[TMP6:%.*]] = call i1 @llvm.vector.reduce.and.v8i1(<8 x i1> [[TMP5]])
-; CHECK-NEXT:    ret i1 [[TMP6]]
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <4 x i32> [[X:%.*]], <4 x i32> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 0, i32 1, i32 2, i32 3>
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp sgt <8 x i32> [[TMP1]], <i32 17, i32 17, i32 17, i32 17, i32 42, i32 42, i32 42, i32 42>
+; CHECK-NEXT:    [[TMP3:%.*]] = icmp slt <8 x i32> [[TMP1]], <i32 17, i32 17, i32 17, i32 17, i32 42, i32 42, i32 42, i32 42>
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <8 x i1> [[TMP2]], <8 x i1> [[TMP3]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 12, i32 13, i32 14, i32 15>
+; CHECK-NEXT:    [[TMP5:%.*]] = extractelement <8 x i1> [[TMP4]], i32 6
+; CHECK-NEXT:    call void @use1(i1 [[TMP5]])
+; CHECK-NEXT:    [[TMP6:%.*]] = freeze <8 x i1> [[TMP4]]
+; CHECK-NEXT:    [[TMP7:%.*]] = call i1 @llvm.vector.reduce.and.v8i1(<8 x i1> [[TMP6]])
+; CHECK-NEXT:    ret i1 [[TMP7]]
 ;
   %x0 = extractelement <4 x i32> %x, i32 0
   %x1 = extractelement <4 x i32> %x, i32 1
@@ -335,27 +329,12 @@ define i1 @logical_and_icmp_clamp_extra_use_select(<4 x i32> %x) {
 
 define i1 @logical_and_icmp_clamp_v8i32(<8 x i32> %x, <8 x i32> %y) {
 ; CHECK-LABEL: @logical_and_icmp_clamp_v8i32(
-; CHECK-NEXT:    [[X0:%.*]] = extractelement <8 x i32> [[X:%.*]], i32 0
-; CHECK-NEXT:    [[X1:%.*]] = extractelement <8 x i32> [[X]], i32 1
-; CHECK-NEXT:    [[X2:%.*]] = extractelement <8 x i32> [[X]], i32 2
-; CHECK-NEXT:    [[X3:%.*]] = extractelement <8 x i32> [[X]], i32 3
-; CHECK-NEXT:    [[Y0:%.*]] = extractelement <8 x i32> [[Y:%.*]], i32 0
-; CHECK-NEXT:    [[Y1:%.*]] = extractelement <8 x i32> [[Y]], i32 1
-; CHECK-NEXT:    [[Y2:%.*]] = extractelement <8 x i32> [[Y]], i32 2
-; CHECK-NEXT:    [[Y3:%.*]] = extractelement <8 x i32> [[Y]], i32 3
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <8 x i32> poison, i32 [[X0]], i32 0
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <8 x i32> [[TMP1]], i32 [[X1]], i32 1
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <8 x i32> [[TMP2]], i32 [[X2]], i32 2
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <8 x i32> [[TMP3]], i32 [[X3]], i32 3
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <8 x i32> [[TMP4]], <8 x i32> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 0, i32 1, i32 2, i32 3>
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <8 x i32> <i32 42, i32 42, i32 42, i32 42, i32 poison, i32 poison, i32 poison, i32 poison>, i32 [[Y0]], i32 4
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <8 x i32> [[TMP5]], i32 [[Y1]], i32 5
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <8 x i32> [[TMP6]], i32 [[Y2]], i32 6
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <8 x i32> [[TMP7]], i32 [[Y3]], i32 7
-; CHECK-NEXT:    [[TMP9:%.*]] = icmp slt <8 x i32> [[SHUFFLE]], [[TMP8]]
-; CHECK-NEXT:    [[TMP10:%.*]] = freeze <8 x i1> [[TMP9]]
-; CHECK-NEXT:    [[TMP11:%.*]] = call i1 @llvm.vector.reduce.and.v8i1(<8 x i1> [[TMP10]])
-; CHECK-NEXT:    ret i1 [[TMP11]]
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[X:%.*]], <8 x i32> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 0, i32 1, i32 2, i32 3>
+; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <8 x i32> [[Y:%.*]], <8 x i32> <i32 42, i32 42, i32 42, i32 42, i32 poison, i32 poison, i32 poison, i32 poison>, <8 x i32> <i32 8, i32 9, i32 10, i32 11, i32 0, i32 1, i32 2, i32 3>
+; CHECK-NEXT:    [[TMP3:%.*]] = icmp slt <8 x i32> [[TMP1]], [[TMP2]]
+; CHECK-NEXT:    [[TMP4:%.*]] = freeze <8 x i1> [[TMP3]]
+; CHECK-NEXT:    [[TMP5:%.*]] = call i1 @llvm.vector.reduce.and.v8i1(<8 x i1> [[TMP4]])
+; CHECK-NEXT:    ret i1 [[TMP5]]
 ;
   %x0 = extractelement <8 x i32> %x, i32 0
   %x1 = extractelement <8 x i32> %x, i32 1
@@ -386,21 +365,18 @@ define i1 @logical_and_icmp_clamp_v8i32(<8 x i32> %x, <8 x i32> %y) {
 define i1 @logical_and_icmp_clamp_partial(<4 x i32> %x) {
 ; SSE-LABEL: @logical_and_icmp_clamp_partial(
 ; SSE-NEXT:    [[TMP1:%.*]] = extractelement <4 x i32> [[X:%.*]], i32 2
-; SSE-NEXT:    [[TMP2:%.*]] = extractelement <4 x i32> [[X]], i32 1
-; SSE-NEXT:    [[TMP3:%.*]] = extractelement <4 x i32> [[X]], i32 0
-; SSE-NEXT:    [[TMP4:%.*]] = insertelement <2 x i32> poison, i32 [[TMP2]], i32 0
-; SSE-NEXT:    [[TMP5:%.*]] = insertelement <2 x i32> [[TMP4]], i32 [[TMP3]], i32 1
-; SSE-NEXT:    [[TMP6:%.*]] = icmp slt <2 x i32> [[TMP5]], <i32 42, i32 42>
+; SSE-NEXT:    [[TMP2:%.*]] = shufflevector <4 x i32> [[X]], <4 x i32> poison, <2 x i32> <i32 1, i32 0>
+; SSE-NEXT:    [[TMP3:%.*]] = icmp slt <2 x i32> [[TMP2]], <i32 42, i32 42>
 ; SSE-NEXT:    [[C2:%.*]] = icmp slt i32 [[TMP1]], 42
-; SSE-NEXT:    [[TMP7:%.*]] = icmp sgt <4 x i32> [[X]], <i32 17, i32 17, i32 17, i32 17>
-; SSE-NEXT:    [[TMP8:%.*]] = freeze <4 x i1> [[TMP7]]
-; SSE-NEXT:    [[TMP9:%.*]] = call i1 @llvm.vector.reduce.and.v4i1(<4 x i1> [[TMP8]])
-; SSE-NEXT:    [[TMP10:%.*]] = extractelement <2 x i1> [[TMP6]], i32 0
-; SSE-NEXT:    [[OP_RDX:%.*]] = select i1 [[TMP9]], i1 [[TMP10]], i1 false
-; SSE-NEXT:    [[TMP11:%.*]] = extractelement <2 x i1> [[TMP6]], i32 1
-; SSE-NEXT:    [[OP_RDX1:%.*]] = select i1 [[TMP11]], i1 [[C2]], i1 false
-; SSE-NEXT:    [[TMP12:%.*]] = freeze i1 [[OP_RDX]]
-; SSE-NEXT:    [[OP_RDX2:%.*]] = select i1 [[TMP12]], i1 [[OP_RDX1]], i1 false
+; SSE-NEXT:    [[TMP4:%.*]] = icmp sgt <4 x i32> [[X]], <i32 17, i32 17, i32 17, i32 17>
+; SSE-NEXT:    [[TMP5:%.*]] = freeze <4 x i1> [[TMP4]]
+; SSE-NEXT:    [[TMP6:%.*]] = call i1 @llvm.vector.reduce.and.v4i1(<4 x i1> [[TMP5]])
+; SSE-NEXT:    [[TMP7:%.*]] = extractelement <2 x i1> [[TMP3]], i32 0
+; SSE-NEXT:    [[OP_RDX:%.*]] = select i1 [[TMP6]], i1 [[TMP7]], i1 false
+; SSE-NEXT:    [[TMP8:%.*]] = extractelement <2 x i1> [[TMP3]], i32 1
+; SSE-NEXT:    [[OP_RDX1:%.*]] = select i1 [[TMP8]], i1 [[C2]], i1 false
+; SSE-NEXT:    [[TMP9:%.*]] = freeze i1 [[OP_RDX]]
+; SSE-NEXT:    [[OP_RDX2:%.*]] = select i1 [[TMP9]], i1 [[OP_RDX1]], i1 false
 ; SSE-NEXT:    ret i1 [[OP_RDX2]]
 ;
 ; AVX-LABEL: @logical_and_icmp_clamp_partial(
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/reduction-same-vals.ll b/llvm/test/Transforms/SLPVectorizer/X86/reduction-same-vals.ll
index 94aa4449c6797..3daebe50d724f 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/reduction-same-vals.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/reduction-same-vals.ll
@@ -11,15 +11,9 @@ define i64 @test() {
 ; CHECK-NEXT:    [[TMP:%.*]] = phi i32 [ 0, [[BB2:%.*]] ], [ 0, [[BB1:%.*]] ]
 ; CHECK-NEXT:    [[TMP4:%.*]] = phi i32 [ 0, [[BB2]] ], [ 0, [[BB1]] ]
 ; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <8 x i32> poison, i32 [[TMP4]], i32 0
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <8 x i32> [[TMP0]], i32 [[TMP4]], i32 1
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <8 x i32> [[TMP1]], i32 [[TMP4]], i32 2
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <8 x i32> [[TMP2]], i32 [[TMP4]], i32 3
-; CHECK-NEXT:    [[TMP44:%.*]] = insertelement <8 x i32> [[TMP3]], i32 [[TMP4]], i32 4
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <8 x i32> [[TMP44]], i32 [[TMP4]], i32 5
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <8 x i32> [[TMP5]], i32 [[TMP4]], i32 6
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <8 x i32> [[TMP6]], i32 [[TMP4]], i32 7
-; CHECK-NEXT:    [[TMP8:%.*]] = call i32 @llvm.vector.reduce.mul.v8i32(<8 x i32> [[TMP7]])
-; CHECK-NEXT:    [[OP_RDX:%.*]] = mul i32 [[TMP8]], [[TMP4]]
+; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <8 x i32> [[TMP0]], <8 x i32> poison, <8 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP2:%.*]] = call i32 @llvm.vector.reduce.mul.v8i32(<8 x i32> [[TMP1]])
+; CHECK-NEXT:    [[OP_RDX:%.*]] = mul i32 [[TMP2]], [[TMP4]]
 ; CHECK-NEXT:    [[OP_RDX1:%.*]] = mul i32 [[TMP4]], [[TMP4]]
 ; CHECK-NEXT:    [[OP_RDX2:%.*]] = mul i32 [[OP_RDX]], [[OP_RDX1]]
 ; CHECK-NEXT:    [[OP_RDX3:%.*]] = mul i32 [[OP_RDX2]], [[TMP]]
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/reduction2.ll b/llvm/test/Transforms/SLPVectorizer/X86/reduction2.ll
index ffa052f7cfddd..458f4f98d815f 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/reduction2.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/reduction2.ll
@@ -89,7 +89,7 @@ define i1 @fcmp_lt_gt(double %a, double %b, double %c) {
 ; CHECK-NEXT:    [[MUL:%.*]] = fmul double [[A:%.*]], 2.000000e+00
 ; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <2 x double> poison, double [[C:%.*]], i32 1
 ; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x double> [[TMP0]], double [[FNEG]], i32 0
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[C]], i32 0
+; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <2 x double> [[TMP1]], <2 x double> poison, <2 x i32> <i32 1, i32 undef>
 ; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x double> [[TMP2]], double [[B]], i32 1
 ; CHECK-NEXT:    [[TMP4:%.*]] = fsub <2 x double> [[TMP1]], [[TMP3]]
 ; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x double> poison, double [[MUL]], i32 0
@@ -138,7 +138,7 @@ define i1 @fcmp_lt(double %a, double %b, double %c) {
 ; CHECK-NEXT:    [[MUL:%.*]] = fmul double [[A:%.*]], 2.000000e+00
 ; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x double> poison, double [[C:%.*]], i32 1
 ; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> [[TMP1]], double [[FNEG]], i32 0
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x double> poison, double [[C]], i32 0
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <2 x double> [[TMP2]], <2 x double> poison, <2 x i32> <i32 1, i32 undef>
 ; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x double> [[TMP3]], double [[B]], i32 1
 ; CHECK-NEXT:    [[TMP5:%.*]] = fsub <2 x double> [[TMP2]], [[TMP4]]
 ; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x double> poison, double [[MUL]], i32 0
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/remark_extract_broadcast.ll b/llvm/test/Transforms/SLPVectorizer/X86/remark_extract_broadcast.ll
index e8ca9aeb8147c..d11eb609b12d0 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/remark_extract_broadcast.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/remark_extract_broadcast.ll
@@ -8,9 +8,9 @@ define void @fextr(ptr %ptr) {
 ; CHECK-NEXT:    [[LD:%.*]] = load <8 x i16>, ptr undef, align 16
 ; CHECK-NEXT:    br label [[T:%.*]]
 ; CHECK:       t:
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <8 x i16> [[LD]], <8 x i16> poison, <8 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP0:%.*]] = add <8 x i16> [[LD]], [[SHUFFLE]]
-; CHECK-NEXT:    store <8 x i16> [[TMP0]], ptr [[PTR:%.*]], align 2
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <8 x i16> [[LD]], <8 x i16> poison, <8 x i32> <i32 0, i32 undef, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0>
+; CHECK-NEXT:    [[TMP1:%.*]] = add <8 x i16> [[LD]], [[TMP0]]
+; CHECK-NEXT:    store <8 x i16> [[TMP1]], ptr [[PTR:%.*]], align 2
 ; CHECK-NEXT:    ret void
 ;
 ; YAML:      Pass:            slp-vectorizer
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/remark_horcost.ll b/llvm/test/Transforms/SLPVectorizer/X86/remark_horcost.ll
index d79c08763be8a..f657477ce8e52 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/remark_horcost.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/remark_horcost.ll
@@ -86,7 +86,7 @@ for.body:                                         ; preds = %for.body, %entry
   ; YAML-NEXT: Function:        foo
   ; YAML-NEXT: Args:
   ; YAML-NEXT:   - String:          'Stores SLP vectorized with cost '
-  ; YAML-NEXT:   - Cost:            '-14'
+  ; YAML-NEXT:   - Cost:            '-5'
   ; YAML-NEXT:   - String:          ' and with tree size '
   ; YAML-NEXT:   - TreeSize:        '4'
 
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/remark_not_all_parts.ll b/llvm/test/Transforms/SLPVectorizer/X86/remark_not_all_parts.ll
index 21a8654ef03b0..0a2cc76d2e25d 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/remark_not_all_parts.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/remark_not_all_parts.ll
@@ -62,7 +62,7 @@ for.body:                                         ; preds = %for.body, %entry
   ; YAML-NEXT: Function:        foo
   ; YAML-NEXT: Args:
   ; YAML-NEXT:   - String:          'Stores SLP vectorized with cost '
-  ; YAML-NEXT:   - Cost:            '-4'
+  ; YAML-NEXT:   - Cost:            '-1'
   ; YAML-NEXT:   - String:          ' and with tree size '
   ; YAML-NEXT:   - TreeSize:        '4'
 
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/reorder-clustered-node.ll b/llvm/test/Transforms/SLPVectorizer/X86/reorder-clustered-node.ll
index 72faaa93df02e..16578504de084 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/reorder-clustered-node.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/reorder-clustered-node.ll
@@ -14,17 +14,16 @@ define i1 @test(ptr %arg, ptr %i233, i64 %i241, ptr %i235, ptr %i237, ptr %i227)
 ; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <8 x ptr> <ptr poison, ptr null, ptr poison, ptr null, ptr null, ptr null, ptr null, ptr null>, ptr [[I242]], i32 0
 ; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <8 x ptr> [[TMP2]], ptr [[I250]], i32 2
 ; CHECK-NEXT:    [[TMP4:%.*]] = icmp ult <8 x ptr> [[TMP3]], [[TMP1]]
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <8 x ptr> poison, ptr [[I250]], i32 0
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <8 x ptr> [[TMP5]], ptr [[I242]], i32 1
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <8 x ptr> [[TMP6]], ptr [[I245]], i32 2
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <8 x ptr> [[TMP7]], ptr [[I248]], i32 3
-; CHECK-NEXT:    [[TMP9:%.*]] = shufflevector <8 x ptr> [[TMP8]], <8 x ptr> poison, <8 x i32> <i32 2, i32 0, i32 1, i32 3, i32 0, i32 1, i32 2, i32 3>
-; CHECK-NEXT:    [[TMP10:%.*]] = shufflevector <8 x ptr> <ptr poison, ptr poison, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null>, <8 x ptr> [[TMP1]], <8 x i32> <i32 8, i32 9, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
-; CHECK-NEXT:    [[TMP11:%.*]] = icmp ult <8 x ptr> [[TMP9]], [[TMP10]]
-; CHECK-NEXT:    [[TMP12:%.*]] = shufflevector <8 x i1> [[TMP11]], <8 x i1> poison, <8 x i32> <i32 1, i32 2, i32 0, i32 3, i32 4, i32 5, i32 6, i32 7>
-; CHECK-NEXT:    [[TMP13:%.*]] = or <8 x i1> [[TMP4]], [[TMP12]]
-; CHECK-NEXT:    [[TMP14:%.*]] = call i1 @llvm.vector.reduce.and.v8i1(<8 x i1> [[TMP13]])
-; CHECK-NEXT:    [[OP_RDX:%.*]] = and i1 [[TMP14]], false
+; CHECK-NEXT:    [[TMP5:%.*]] = shufflevector <8 x ptr> [[TMP3]], <8 x ptr> poison, <4 x i32> <i32 2, i32 0, i32 undef, i32 undef>
+; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <4 x ptr> [[TMP5]], ptr [[I245]], i32 2
+; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <4 x ptr> [[TMP6]], ptr [[I248]], i32 3
+; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <4 x ptr> [[TMP7]], <4 x ptr> poison, <8 x i32> <i32 2, i32 0, i32 1, i32 3, i32 0, i32 1, i32 2, i32 3>
+; CHECK-NEXT:    [[TMP9:%.*]] = shufflevector <8 x ptr> [[TMP1]], <8 x ptr> <ptr poison, ptr poison, ptr null, ptr null, ptr null, ptr null, ptr null, ptr null>, <8 x i32> <i32 0, i32 1, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+; CHECK-NEXT:    [[TMP10:%.*]] = icmp ult <8 x ptr> [[TMP8]], [[TMP9]]
+; CHECK-NEXT:    [[TMP11:%.*]] = shufflevector <8 x i1> [[TMP10]], <8 x i1> poison, <8 x i32> <i32 1, i32 2, i32 0, i32 3, i32 4, i32 5, i32 6, i32 7>
+; CHECK-NEXT:    [[TMP12:%.*]] = or <8 x i1> [[TMP4]], [[TMP11]]
+; CHECK-NEXT:    [[TMP13:%.*]] = call i1 @llvm.vector.reduce.and.v8i1(<8 x i1> [[TMP12]])
+; CHECK-NEXT:    [[OP_RDX:%.*]] = and i1 [[TMP13]], false
 ; CHECK-NEXT:    ret i1 [[OP_RDX]]
 ;
 bb:
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/reorder-reused-masked-gather.ll b/llvm/test/Transforms/SLPVectorizer/X86/reorder-reused-masked-gather.ll
index b182df599ce16..035bc3ecc09d4 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/reorder-reused-masked-gather.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/reorder-reused-masked-gather.ll
@@ -10,7 +10,7 @@ define void @test(ptr noalias %0, ptr %p) {
 ; CHECK-NEXT:    [[TMP6:%.*]] = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> [[TMP4]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x float> poison)
 ; CHECK-NEXT:    [[TMP7:%.*]] = shufflevector <8 x float> [[TMP6]], <8 x float> poison, <16 x i32> <i32 4, i32 3, i32 0, i32 1, i32 2, i32 0, i32 1, i32 2, i32 0, i32 2, i32 5, i32 6, i32 7, i32 5, i32 6, i32 7>
 ; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <8 x float> [[TMP6]], <8 x float> poison, <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-; CHECK-NEXT:    [[TMP9:%.*]] = shufflevector <16 x float> <float poison, float poison, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00>, <16 x float> [[TMP8]], <16 x i32> <i32 16, i32 17, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+; CHECK-NEXT:    [[TMP9:%.*]] = shufflevector <16 x float> [[TMP8]], <16 x float> <float poison, float poison, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00>, <16 x i32> <i32 0, i32 1, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31>
 ; CHECK-NEXT:    [[TMP10:%.*]] = fadd reassoc nsz arcp contract afn <16 x float> [[TMP7]], [[TMP9]]
 ; CHECK-NEXT:    [[TMP11:%.*]] = shufflevector <16 x float> [[TMP10]], <16 x float> poison, <16 x i32> <i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 1, i32 9, i32 0, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
 ; CHECK-NEXT:    store <16 x float> [[TMP11]], ptr [[TMP5]], align 4
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/reused-scalars-in-buildvector.ll b/llvm/test/Transforms/SLPVectorizer/X86/reused-scalars-in-buildvector.ll
index 24334c43da684..5cf0dc6a40a81 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/reused-scalars-in-buildvector.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/reused-scalars-in-buildvector.ll
@@ -3,23 +3,19 @@
 
 define void @test(ptr %p, i32 %a, i32 %b, i32 %c, i32 %d, i32 %e, i32 %f) {
 ; CHECK-LABEL: @test(
-; CHECK-NEXT:    [[S1:%.*]] = sub i32 [[A:%.*]], 1
-; CHECK-NEXT:    [[S2:%.*]] = sub i32 [[B:%.*]], 1
-; CHECK-NEXT:    [[M1:%.*]] = mul i32 [[S1]], [[E:%.*]]
-; CHECK-NEXT:    [[M2:%.*]] = mul i32 [[S2]], [[F:%.*]]
-; CHECK-NEXT:    [[M3:%.*]] = mul i32 [[S1]], [[C:%.*]]
-; CHECK-NEXT:    [[M4:%.*]] = mul i32 [[S2]], [[D:%.*]]
-; CHECK-NEXT:    [[A1:%.*]] = add i32 [[A]], [[M1]]
-; CHECK-NEXT:    [[A2:%.*]] = add i32 [[B]], [[M2]]
-; CHECK-NEXT:    [[A3:%.*]] = add i32 [[C]], [[M3]]
-; CHECK-NEXT:    [[A4:%.*]] = add i32 [[D]], [[M4]]
-; CHECK-NEXT:    store i32 [[A1]], ptr [[P:%.*]], align 4
-; CHECK-NEXT:    [[GEP:%.*]] = getelementptr inbounds i32, ptr [[P]], i32 1
-; CHECK-NEXT:    store i32 [[A2]], ptr [[GEP]], align 4
-; CHECK-NEXT:    [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[P]], i32 2
-; CHECK-NEXT:    store i32 [[A3]], ptr [[GEP1]], align 4
-; CHECK-NEXT:    [[GEP2:%.*]] = getelementptr inbounds i32, ptr [[P]], i32 3
-; CHECK-NEXT:    store i32 [[A4]], ptr [[GEP2]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x i32> poison, i32 [[A:%.*]], i32 0
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x i32> [[TMP1]], i32 [[B:%.*]], i32 1
+; CHECK-NEXT:    [[TMP3:%.*]] = sub <2 x i32> [[TMP2]], <i32 1, i32 1>
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <2 x i32> [[TMP3]], <2 x i32> poison, <4 x i32> <i32 0, i32 1, i32 0, i32 1>
+; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <4 x i32> poison, i32 [[E:%.*]], i32 0
+; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <4 x i32> [[TMP5]], i32 [[F:%.*]], i32 1
+; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <4 x i32> [[TMP6]], i32 [[C:%.*]], i32 2
+; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <4 x i32> [[TMP7]], i32 [[D:%.*]], i32 3
+; CHECK-NEXT:    [[TMP9:%.*]] = mul <4 x i32> [[TMP4]], [[TMP8]]
+; CHECK-NEXT:    [[TMP10:%.*]] = shufflevector <2 x i32> [[TMP2]], <2 x i32> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
+; CHECK-NEXT:    [[TMP11:%.*]] = shufflevector <4 x i32> [[TMP10]], <4 x i32> [[TMP8]], <4 x i32> <i32 0, i32 1, i32 6, i32 7>
+; CHECK-NEXT:    [[TMP12:%.*]] = add <4 x i32> [[TMP11]], [[TMP9]]
+; CHECK-NEXT:    store <4 x i32> [[TMP12]], ptr [[P:%.*]], align 4
 ; CHECK-NEXT:    ret void
 ;
   %s1 = sub i32 %a, 1
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/root-trunc-extract-reuse.ll b/llvm/test/Transforms/SLPVectorizer/X86/root-trunc-extract-reuse.ll
index f996040f4d517..ccde383afa3ed 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/root-trunc-extract-reuse.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/root-trunc-extract-reuse.ll
@@ -17,9 +17,8 @@ define i1 @test() {
 ; CHECK-NEXT:    [[T13:%.*]] = and <2 x i32> [[TMP4]], zeroinitializer
 ; CHECK-NEXT:    br label [[ELSE1:%.*]]
 ; CHECK:       else1:
-; CHECK-NEXT:    [[T20:%.*]] = extractelement <2 x i32> [[T13]], i64 0
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x i32> poison, i32 [[BF_CAST162]], i32 0
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x i32> [[TMP5]], i32 [[T20]], i32 1
+; CHECK-NEXT:    [[TMP5:%.*]] = shufflevector <2 x i32> [[T13]], <2 x i32> poison, <2 x i32> <i32 undef, i32 0>
+; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x i32> [[TMP5]], i32 [[BF_CAST162]], i32 0
 ; CHECK-NEXT:    [[TMP7:%.*]] = icmp ugt <2 x i32> [[TMP6]], zeroinitializer
 ; CHECK-NEXT:    [[TMP8:%.*]] = extractelement <2 x i1> [[TMP7]], i32 1
 ; CHECK-NEXT:    ret i1 [[TMP8]]
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/scatter-vectorize-reorder.ll b/llvm/test/Transforms/SLPVectorizer/X86/scatter-vectorize-reorder.ll
index 284737b23a6c7..c79e9b94278cd 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/scatter-vectorize-reorder.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/scatter-vectorize-reorder.ll
@@ -12,10 +12,10 @@ define void @test() {
 ; CHECK-NEXT:    [[TMP1:%.*]] = fsub <2 x float> zeroinitializer, [[TMP0]]
 ; CHECK-NEXT:    [[TMP2:%.*]] = load float, ptr [[ARRAYIDX10_I_I86]], align 4
 ; CHECK-NEXT:    [[TMP3:%.*]] = load float, ptr undef, align 4
-; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <2 x float> <float 0.000000e+00, float poison>, <2 x float> [[TMP0]], <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <2 x float> [[TMP0]], <2 x float> <float 0.000000e+00, float poison>, <2 x i32> <i32 2, i32 1>
 ; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x float> poison, float [[TMP3]], i32 0
 ; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x float> [[TMP5]], float [[TMP2]], i32 1
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <2 x float> <float poison, float 0.000000e+00>, float [[TMP2]], i32 0
+; CHECK-NEXT:    [[TMP7:%.*]] = shufflevector <2 x float> [[TMP6]], <2 x float> <float poison, float 0.000000e+00>, <2 x i32> <i32 1, i32 3>
 ; CHECK-NEXT:    [[TMP8:%.*]] = call <2 x float> @llvm.fmuladd.v2f32(<2 x float> [[TMP4]], <2 x float> [[TMP6]], <2 x float> [[TMP7]])
 ; CHECK-NEXT:    br i1 false, label [[BB2:%.*]], label [[BB3:%.*]]
 ; CHECK:       bb2:
@@ -23,12 +23,12 @@ define void @test() {
 ; CHECK-NEXT:    br label [[BB3]]
 ; CHECK:       bb3:
 ; CHECK-NEXT:    [[TMP10:%.*]] = phi <2 x float> [ [[TMP9]], [[BB2]] ], [ zeroinitializer, [[BB1]] ]
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x float> [[TMP10]], <2 x float> poison, <2 x i32> <i32 1, i32 0>
-; CHECK-NEXT:    [[TMP11:%.*]] = fadd <2 x float> [[TMP1]], [[SHUFFLE]]
-; CHECK-NEXT:    [[TMP12:%.*]] = fadd <2 x float> [[TMP11]], zeroinitializer
-; CHECK-NEXT:    [[TMP13:%.*]] = fsub <2 x float> [[TMP12]], zeroinitializer
+; CHECK-NEXT:    [[TMP11:%.*]] = shufflevector <2 x float> [[TMP10]], <2 x float> poison, <2 x i32> <i32 1, i32 0>
+; CHECK-NEXT:    [[TMP12:%.*]] = fadd <2 x float> [[TMP1]], [[TMP11]]
+; CHECK-NEXT:    [[TMP13:%.*]] = fadd <2 x float> [[TMP12]], zeroinitializer
 ; CHECK-NEXT:    [[TMP14:%.*]] = fsub <2 x float> [[TMP13]], zeroinitializer
-; CHECK-NEXT:    store <2 x float> [[TMP14]], ptr [[ARRAYIDX21_I]], align 16
+; CHECK-NEXT:    [[TMP15:%.*]] = fsub <2 x float> [[TMP14]], zeroinitializer
+; CHECK-NEXT:    store <2 x float> [[TMP15]], ptr [[ARRAYIDX21_I]], align 16
 ; CHECK-NEXT:    ret void
 ;
 entry:
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/scatter-vectorize-reused-pointer.ll b/llvm/test/Transforms/SLPVectorizer/X86/scatter-vectorize-reused-pointer.ll
index ec555f7e7aacc..8dbb79cd81617 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/scatter-vectorize-reused-pointer.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/scatter-vectorize-reused-pointer.ll
@@ -1,5 +1,5 @@
 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -S -passes=slp-vectorizer < %s -mtriple=x86_64-unknown-linux-gnu -mcpu=skylake -slp-threshold=-13 | FileCheck %s
+; RUN: opt -S -passes=slp-vectorizer < %s -mtriple=x86_64-unknown-linux-gnu -mcpu=skylake -slp-threshold=-14 | FileCheck %s
 
 define void @test(i1 %c, ptr %arg) {
 ; CHECK-LABEL: @test(
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/vect-gather-same-nodes.ll b/llvm/test/Transforms/SLPVectorizer/X86/vect-gather-same-nodes.ll
index c34a91f6be7ce..e378909b49b92 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/vect-gather-same-nodes.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/vect-gather-same-nodes.ll
@@ -13,16 +13,15 @@ define void @test(ptr %a, ptr %b) {
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[TMP3:%.*]] = load float, ptr null, align 4
 ; CHECK-NEXT:    [[TMP4:%.*]] = load <2 x float>, ptr [[A:%.*]], align 4
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 0, i32 1>
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <4 x float> [[TMP2]], float [[TMP0]], i32 3
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <4 x float> [[TMP5]], float [[TMP3]], i32 2
-; CHECK-NEXT:    [[TMP7:%.*]] = fmul <4 x float> [[SHUFFLE]], [[TMP6]]
-; CHECK-NEXT:    [[SHUFFLE1:%.*]] = shufflevector <4 x float> [[TMP7]], <4 x float> poison, <4 x i32> <i32 1, i32 2, i32 3, i32 0>
-; CHECK-NEXT:    [[SHUFFLE2:%.*]] = shufflevector <4 x float> [[SHUFFLE]], <4 x float> poison, <4 x i32> <i32 0, i32 1, i32 0, i32 1>
-; CHECK-NEXT:    [[TMP8:%.*]] = fmul <4 x float> [[SHUFFLE2]], zeroinitializer
-; CHECK-NEXT:    [[TMP9:%.*]] = fadd <4 x float> [[SHUFFLE1]], [[TMP8]]
-; CHECK-NEXT:    [[TMP10:%.*]] = fadd <4 x float> [[TMP9]], zeroinitializer
-; CHECK-NEXT:    store <4 x float> [[TMP10]], ptr [[RESULT]], align 4
+; CHECK-NEXT:    [[TMP5:%.*]] = shufflevector <2 x float> [[TMP4]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 0, i32 1>
+; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <4 x float> [[TMP2]], float [[TMP0]], i32 3
+; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <4 x float> [[TMP6]], float [[TMP3]], i32 2
+; CHECK-NEXT:    [[TMP8:%.*]] = fmul <4 x float> [[TMP5]], [[TMP7]]
+; CHECK-NEXT:    [[TMP9:%.*]] = shufflevector <4 x float> [[TMP8]], <4 x float> poison, <4 x i32> <i32 1, i32 2, i32 3, i32 0>
+; CHECK-NEXT:    [[TMP10:%.*]] = fmul <4 x float> [[TMP5]], zeroinitializer
+; CHECK-NEXT:    [[TMP11:%.*]] = fadd <4 x float> [[TMP9]], [[TMP10]]
+; CHECK-NEXT:    [[TMP12:%.*]] = fadd <4 x float> [[TMP11]], zeroinitializer
+; CHECK-NEXT:    store <4 x float> [[TMP12]], ptr [[RESULT]], align 4
 ; CHECK-NEXT:    br label [[FOR_BODY]]
 ;
 entry:
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/vectorize-widest-phis.ll b/llvm/test/Transforms/SLPVectorizer/X86/vectorize-widest-phis.ll
index 666a2ec85c0fa..aa3c2be7dc9c2 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/vectorize-widest-phis.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/vectorize-widest-phis.ll
@@ -12,24 +12,22 @@ define void @foo() {
 ; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <4 x float> [[TMP0]], float [[CONV]], i32 1
 ; CHECK-NEXT:    br label [[BB2:%.*]]
 ; CHECK:       bb2:
-; CHECK-NEXT:    [[TMP2:%.*]] = phi <4 x float> [ [[TMP1]], [[BB1]] ], [ [[TMP14:%.*]], [[BB3:%.*]] ]
+; CHECK-NEXT:    [[TMP2:%.*]] = phi <4 x float> [ [[TMP1]], [[BB1]] ], [ [[TMP10:%.*]], [[BB3:%.*]] ]
 ; CHECK-NEXT:    [[TMP3:%.*]] = load double, ptr undef, align 8
 ; CHECK-NEXT:    br i1 undef, label [[BB3]], label [[BB4:%.*]]
 ; CHECK:       bb4:
 ; CHECK-NEXT:    [[TMP4:%.*]] = fpext <4 x float> [[TMP2]] to <4 x double>
 ; CHECK-NEXT:    [[CONV2:%.*]] = uitofp i16 undef to double
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x double> <double undef, double poison>, double [[TMP3]], i32 1
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x double> <double undef, double poison>, double [[CONV2]], i32 1
-; CHECK-NEXT:    [[TMP7:%.*]] = fsub <2 x double> [[TMP5]], [[TMP6]]
-; CHECK-NEXT:    [[TMP8:%.*]] = fadd <2 x double> [[TMP5]], [[TMP6]]
-; CHECK-NEXT:    [[TMP9:%.*]] = shufflevector <2 x double> [[TMP7]], <2 x double> [[TMP8]], <2 x i32> <i32 0, i32 3>
-; CHECK-NEXT:    [[TMP10:%.*]] = shufflevector <2 x double> [[TMP9]], <2 x double> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; CHECK-NEXT:    [[TMP11:%.*]] = fcmp ogt <4 x double> [[TMP10]], [[TMP4]]
-; CHECK-NEXT:    [[TMP12:%.*]] = fptrunc <4 x double> [[TMP10]] to <4 x float>
-; CHECK-NEXT:    [[TMP13:%.*]] = select <4 x i1> [[TMP11]], <4 x float> [[TMP2]], <4 x float> [[TMP12]]
+; CHECK-NEXT:    [[ADD1:%.*]] = fadd double [[TMP3]], [[CONV2]]
+; CHECK-NEXT:    [[SUB1:%.*]] = fsub double undef, undef
+; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <4 x double> <double poison, double poison, double undef, double undef>, double [[SUB1]], i32 0
+; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <4 x double> [[TMP5]], double [[ADD1]], i32 1
+; CHECK-NEXT:    [[TMP7:%.*]] = fcmp ogt <4 x double> [[TMP6]], [[TMP4]]
+; CHECK-NEXT:    [[TMP8:%.*]] = fptrunc <4 x double> [[TMP6]] to <4 x float>
+; CHECK-NEXT:    [[TMP9:%.*]] = select <4 x i1> [[TMP7]], <4 x float> [[TMP2]], <4 x float> [[TMP8]]
 ; CHECK-NEXT:    br label [[BB3]]
 ; CHECK:       bb3:
-; CHECK-NEXT:    [[TMP14]] = phi <4 x float> [ [[TMP13]], [[BB4]] ], [ [[TMP2]], [[BB2]] ]
+; CHECK-NEXT:    [[TMP10]] = phi <4 x float> [ [[TMP9]], [[BB4]] ], [ [[TMP2]], [[BB2]] ]
 ; CHECK-NEXT:    br label [[BB2]]
 ;
 entry:
diff --git a/llvm/test/Transforms/SROA/pr57796.ll b/llvm/test/Transforms/SROA/pr57796.ll
new file mode 100644
index 0000000000000..1bf1ad7ee934a
--- /dev/null
+++ b/llvm/test/Transforms/SROA/pr57796.ll
@@ -0,0 +1,46 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-PRESERVE-CFG
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-MODIFY-CFG
+
+%struct.Value = type { %union.anon }
+%union.anon = type { <32 x i8> }
+
+@A = dso_local global i64 0, align 8
+
+; Make sure that sroa does not crash when dealing with an invalid vector
+; element type.
+define void @foo() {
+; CHECK-LABEL: @foo(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[REF_TMP_I:%.*]] = alloca [[STRUCT_VALUE:%.*]], align 32
+; CHECK-NEXT:    call void @value_create(ptr sret([[STRUCT_VALUE]]) align 32 [[REF_TMP_I]])
+; CHECK-NEXT:    [[CALL_I:%.*]] = call align 32 ptr @value_set_type(ptr align 32 [[REF_TMP_I]])
+; CHECK-NEXT:    [[TMP0:%.*]] = load <32 x i8>, ptr [[CALL_I]], align 32
+; CHECK-NEXT:    [[REF_TMP_SROA_0_0_VEC_EXTRACT:%.*]] = shufflevector <32 x i8> [[TMP0]], <32 x i8> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <8 x i8> [[REF_TMP_SROA_0_0_VEC_EXTRACT]] to x86_mmx
+; CHECK-NEXT:    [[TMP2:%.*]] = call x86_mmx @llvm.x86.sse.pshuf.w(x86_mmx [[TMP1]], i8 0)
+; CHECK-NEXT:    store x86_mmx [[TMP2]], ptr @A, align 8
+; CHECK-NEXT:    ret void
+;
+entry:
+  %ref.tmp.i = alloca %struct.Value, align 32
+  %ref.tmp = alloca %struct.Value, align 32
+  call void @value_create(ptr sret(%struct.Value) align 32 %ref.tmp.i)
+  %call.i = call align 32 ptr @value_set_type(ptr align 32 %ref.tmp.i)
+  %0 = load <32 x i8>, ptr %call.i, align 32
+  store <32 x i8> %0, ptr %ref.tmp, align 32
+  %1 = load x86_mmx, ptr %ref.tmp, align 32
+  %2 = call x86_mmx @llvm.x86.sse.pshuf.w(x86_mmx %1, i8 0)
+  store x86_mmx %2, ptr @A, align 8
+  ret void
+}
+
+declare x86_mmx @llvm.x86.sse.pshuf.w(x86_mmx, i8 immarg)
+
+declare dso_local void @value_create(ptr sret(%struct.Value) align 32)
+
+declare dso_local align 32 ptr @value_set_type(ptr align 32)
+
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-MODIFY-CFG: {{.*}}
+; CHECK-PRESERVE-CFG: {{.*}}
diff --git a/llvm/test/Transforms/SROA/sroa-common-type-fail-promotion.ll b/llvm/test/Transforms/SROA/sroa-common-type-fail-promotion.ll
index 2734c1a520333..09d98fdaba4c4 100644
--- a/llvm/test/Transforms/SROA/sroa-common-type-fail-promotion.ll
+++ b/llvm/test/Transforms/SROA/sroa-common-type-fail-promotion.ll
@@ -15,12 +15,15 @@ define amdgpu_kernel void @test_zeroinit() #0 {
 ; CHECK-LABEL: @test_zeroinit(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[DATA:%.*]] = load <4 x float>, ptr undef, align 16
-; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x half>
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x i16>
 ; CHECK-NEXT:    br label [[BB:%.*]]
 ; CHECK:       bb:
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 0
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 1
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 2
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT]] to half
 ; CHECK-NEXT:    ret void
 ;
 entry:
@@ -43,12 +46,15 @@ define amdgpu_kernel void @test_memset() #0 {
 ; CHECK-LABEL: @test_memset(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[DATA:%.*]] = load <4 x float>, ptr undef, align 16
-; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x half>
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x i16>
 ; CHECK-NEXT:    br label [[BB:%.*]]
 ; CHECK:       bb:
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 0
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 1
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 2
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT]] to half
 ; CHECK-NEXT:    ret void
 ;
 entry:
@@ -67,24 +73,19 @@ bb:
   ret void
 }
 
-; Initial SROA pass failed to promote alloca and same alloca type was re-used
-; so alloca was not re-added to the worklist after initial SROA pass. This
-; caused it to fail to promote unlike the other tests.
 define amdgpu_kernel void @vector_type_alloca() #0 {
 ; CHECK-LABEL: @vector_type_alloca(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY:%.*]] = alloca <8 x half>, align 16
-; CHECK-NEXT:    store <8 x half> zeroinitializer, ptr [[B_BLOCKWISE_COPY]], align 16
 ; CHECK-NEXT:    [[DATA:%.*]] = load <4 x float>, ptr undef, align 16
-; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x half>
-; CHECK-NEXT:    store <8 x half> [[TMP0]], ptr [[B_BLOCKWISE_COPY]], align 16
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x i16>
 ; CHECK-NEXT:    br label [[BB:%.*]]
 ; CHECK:       bb:
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_0_LOAD1:%.*]] = load half, ptr [[B_BLOCKWISE_COPY]], align 16
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_2_PTR2_SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[B_BLOCKWISE_COPY]], i64 2
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_2_LOAD2:%.*]] = load half, ptr [[B_BLOCKWISE_COPY_2_PTR2_SROA_IDX]], align 2
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_4_PTR3_SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[B_BLOCKWISE_COPY]], i64 4
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_4_LOAD3:%.*]] = load half, ptr [[B_BLOCKWISE_COPY_4_PTR3_SROA_IDX]], align 4
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT]] to half
 ; CHECK-NEXT:    ret void
 ;
 entry:
@@ -107,12 +108,15 @@ define amdgpu_kernel void @test_struct_contain_multiple_types1() #0 {
 ; CHECK-LABEL: @test_struct_contain_multiple_types1(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[DATA:%.*]] = load <4 x float>, ptr undef, align 16
-; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x half>
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x i16>
 ; CHECK-NEXT:    br label [[BB:%.*]]
 ; CHECK:       bb:
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 0
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 1
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 2
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT]] to half
 ; CHECK-NEXT:    ret void
 ;
 entry:
@@ -140,12 +144,15 @@ define amdgpu_kernel void @test_struct_contain_multiple_types2() #0 {
 ; CHECK-NEXT:    [[DATA1_FCA_2_EXTRACT:%.*]] = extractvalue [4 x i32] [[DATA1]], 2
 ; CHECK-NEXT:    [[DATA1_FCA_3_EXTRACT:%.*]] = extractvalue [4 x i32] [[DATA1]], 3
 ; CHECK-NEXT:    [[DATA2:%.*]] = load <4 x float>, ptr undef, align 16
-; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA2]] to <8 x half>
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA2]] to <8 x i16>
 ; CHECK-NEXT:    br label [[BB:%.*]]
 ; CHECK:       bb:
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_5_0_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 0
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_5_2_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 1
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_5_4_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 2
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_5_16_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_5_16_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_5_18_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_5_18_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_5_20_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_5_20_VEC_EXTRACT]] to half
 ; CHECK-NEXT:    ret void
 ;
 entry:
@@ -173,13 +180,15 @@ define amdgpu_kernel void @test_struct_array_vector() #0 {
 ; CHECK-LABEL: @test_struct_array_vector(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[DATA0:%.*]] = load <4 x float>, ptr undef, align 16
-; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA0]] to <8 x half>
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA0]] to <8 x i16>
 ; CHECK-NEXT:    [[DATA1:%.*]] = load <4 x float>, ptr undef, align 16
-; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <4 x float> [[DATA1]] to <8 x half>
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <4 x float> [[DATA1]] to <8 x i16>
 ; CHECK-NEXT:    br label [[BB:%.*]]
 ; CHECK:       bb:
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 0
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_3_0_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP1]], i32 0
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_3_16_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP1]], i32 0
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_3_16_VEC_EXTRACT]] to half
 ; CHECK-NEXT:    ret void
 ;
 entry:
@@ -211,7 +220,7 @@ define amdgpu_kernel void @test_struct_array_vector_i16() #0 {
 ; CHECK:       bb:
 ; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 0
 ; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 1
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_4_0_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP1]], i32 0
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_4_16_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP1]], i32 0
 ; CHECK-NEXT:    ret void
 ;
 entry:
@@ -279,12 +288,15 @@ define amdgpu_kernel void @test_array_vector() #0 {
 ; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_5:%.*]] = alloca <8 x half>, align 16
 ; CHECK-NEXT:    call void @llvm.memset.p0.i32(ptr align 16 [[B_BLOCKWISE_COPY_SROA_5]], i8 0, i32 16, i1 false)
 ; CHECK-NEXT:    [[DATA:%.*]] = load <4 x float>, ptr undef, align 16
-; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x half>
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x i16>
 ; CHECK-NEXT:    br label [[BB:%.*]]
 ; CHECK:       bb:
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 0
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 1
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 2
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT]] to half
 ; CHECK-NEXT:    ret void
 ;
 entry:
@@ -309,12 +321,15 @@ define amdgpu_kernel void @test_array_vector2() #0 {
 ; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_5:%.*]] = alloca <8 x half>, align 16
 ; CHECK-NEXT:    call void @llvm.memset.p0.i32(ptr align 16 [[B_BLOCKWISE_COPY_SROA_5]], i8 0, i32 16, i1 false)
 ; CHECK-NEXT:    [[DATA:%.*]] = load <4 x float>, ptr undef, align 16
-; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x half>
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x i16>
 ; CHECK-NEXT:    br label [[BB:%.*]]
 ; CHECK:       bb:
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 0
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 1
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 2
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT]] to half
 ; CHECK-NEXT:    ret void
 ;
 entry:
diff --git a/llvm/test/Transforms/SROA/vector-promotion.ll b/llvm/test/Transforms/SROA/vector-promotion.ll
index 06e046ea3b663..c748e36a74714 100644
--- a/llvm/test/Transforms/SROA/vector-promotion.ll
+++ b/llvm/test/Transforms/SROA/vector-promotion.ll
@@ -891,34 +891,26 @@ define <2 x i64> @test13(i32 %a, i32 %b, i32 %c, i32 %d) {
 ; cast to a different vector type
 ; CHECK-LABEL: @test13(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[X_SROA_0:%.*]] = alloca <2 x i64>, align 16
-; CHECK-NEXT:    store i32 [[A:%.*]], ptr [[X_SROA_0]], align 16
-; CHECK-NEXT:    [[X_SROA_0_4_X_TMP2_SROA_IDX1:%.*]] = getelementptr inbounds i8, ptr [[X_SROA_0]], i64 4
-; CHECK-NEXT:    store i32 [[B:%.*]], ptr [[X_SROA_0_4_X_TMP2_SROA_IDX1]], align 4
-; CHECK-NEXT:    [[X_SROA_0_8_X_TMP3_SROA_IDX2:%.*]] = getelementptr inbounds i8, ptr [[X_SROA_0]], i64 8
-; CHECK-NEXT:    store i32 [[C:%.*]], ptr [[X_SROA_0_8_X_TMP3_SROA_IDX2]], align 8
-; CHECK-NEXT:    [[X_SROA_0_12_X_TMP4_SROA_IDX3:%.*]] = getelementptr inbounds i8, ptr [[X_SROA_0]], i64 12
-; CHECK-NEXT:    store i32 [[D:%.*]], ptr [[X_SROA_0_12_X_TMP4_SROA_IDX3]], align 4
-; CHECK-NEXT:    [[X_SROA_0_0_X_SROA_0_0_RESULT:%.*]] = load <2 x i64>, ptr [[X_SROA_0]], align 16
-; CHECK-NEXT:    ret <2 x i64> [[X_SROA_0_0_X_SROA_0_0_RESULT]]
+; CHECK-NEXT:    [[X_SROA_0_0_VEC_INSERT:%.*]] = insertelement <4 x i32> undef, i32 [[A:%.*]], i32 0
+; CHECK-NEXT:    [[X_SROA_0_4_VEC_INSERT:%.*]] = insertelement <4 x i32> [[X_SROA_0_0_VEC_INSERT]], i32 [[B:%.*]], i32 1
+; CHECK-NEXT:    [[X_SROA_0_8_VEC_INSERT:%.*]] = insertelement <4 x i32> [[X_SROA_0_4_VEC_INSERT]], i32 [[C:%.*]], i32 2
+; CHECK-NEXT:    [[X_SROA_0_12_VEC_INSERT:%.*]] = insertelement <4 x i32> [[X_SROA_0_8_VEC_INSERT]], i32 [[D:%.*]], i32 3
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x i32> [[X_SROA_0_12_VEC_INSERT]] to <2 x i64>
+; CHECK-NEXT:    ret <2 x i64> [[TMP0]]
 ;
 ; DEBUG-LABEL: @test13(
 ; DEBUG-NEXT:  entry:
-; DEBUG-NEXT:    [[X_SROA_0:%.*]] = alloca <2 x i64>, align 16, !dbg [[DBG354:![0-9]+]]
-; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META349:![0-9]+]], metadata !DIExpression()), !dbg [[DBG354]]
-; DEBUG-NEXT:    store i32 [[A:%.*]], ptr [[X_SROA_0]], align 16, !dbg [[DBG355:![0-9]+]]
+; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META349:![0-9]+]], metadata !DIExpression()), !dbg [[DBG354:![0-9]+]]
+; DEBUG-NEXT:    [[X_SROA_0_0_VEC_INSERT:%.*]] = insertelement <4 x i32> undef, i32 [[A:%.*]], i32 0, !dbg [[DBG355:![0-9]+]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META350:![0-9]+]], metadata !DIExpression()), !dbg [[DBG356:![0-9]+]]
-; DEBUG-NEXT:    [[X_SROA_0_4_X_TMP2_SROA_IDX1:%.*]] = getelementptr inbounds i8, ptr [[X_SROA_0]], i64 4, !dbg [[DBG357:![0-9]+]]
-; DEBUG-NEXT:    store i32 [[B:%.*]], ptr [[X_SROA_0_4_X_TMP2_SROA_IDX1]], align 4, !dbg [[DBG357]]
+; DEBUG-NEXT:    [[X_SROA_0_4_VEC_INSERT:%.*]] = insertelement <4 x i32> [[X_SROA_0_0_VEC_INSERT]], i32 [[B:%.*]], i32 1, !dbg [[DBG357:![0-9]+]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META351:![0-9]+]], metadata !DIExpression()), !dbg [[DBG358:![0-9]+]]
-; DEBUG-NEXT:    [[X_SROA_0_8_X_TMP3_SROA_IDX2:%.*]] = getelementptr inbounds i8, ptr [[X_SROA_0]], i64 8, !dbg [[DBG359:![0-9]+]]
-; DEBUG-NEXT:    store i32 [[C:%.*]], ptr [[X_SROA_0_8_X_TMP3_SROA_IDX2]], align 8, !dbg [[DBG359]]
+; DEBUG-NEXT:    [[X_SROA_0_8_VEC_INSERT:%.*]] = insertelement <4 x i32> [[X_SROA_0_4_VEC_INSERT]], i32 [[C:%.*]], i32 2, !dbg [[DBG359:![0-9]+]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META352:![0-9]+]], metadata !DIExpression()), !dbg [[DBG360:![0-9]+]]
-; DEBUG-NEXT:    [[X_SROA_0_12_X_TMP4_SROA_IDX3:%.*]] = getelementptr inbounds i8, ptr [[X_SROA_0]], i64 12, !dbg [[DBG361:![0-9]+]]
-; DEBUG-NEXT:    store i32 [[D:%.*]], ptr [[X_SROA_0_12_X_TMP4_SROA_IDX3]], align 4, !dbg [[DBG361]]
-; DEBUG-NEXT:    [[X_SROA_0_0_X_SROA_0_0_RESULT:%.*]] = load <2 x i64>, ptr [[X_SROA_0]], align 16, !dbg [[DBG362:![0-9]+]]
-; DEBUG-NEXT:    call void @llvm.dbg.value(metadata <2 x i64> [[X_SROA_0_0_X_SROA_0_0_RESULT]], metadata [[META353:![0-9]+]], metadata !DIExpression()), !dbg [[DBG362]]
-; DEBUG-NEXT:    ret <2 x i64> [[X_SROA_0_0_X_SROA_0_0_RESULT]], !dbg [[DBG363:![0-9]+]]
+; DEBUG-NEXT:    [[X_SROA_0_12_VEC_INSERT:%.*]] = insertelement <4 x i32> [[X_SROA_0_8_VEC_INSERT]], i32 [[D:%.*]], i32 3, !dbg [[DBG361:![0-9]+]]
+; DEBUG-NEXT:    [[TMP0:%.*]] = bitcast <4 x i32> [[X_SROA_0_12_VEC_INSERT]] to <2 x i64>, !dbg [[DBG362:![0-9]+]]
+; DEBUG-NEXT:    call void @llvm.dbg.value(metadata <2 x i64> [[TMP0]], metadata [[META353:![0-9]+]], metadata !DIExpression()), !dbg [[DBG362]]
+; DEBUG-NEXT:    ret <2 x i64> [[TMP0]], !dbg [[DBG363:![0-9]+]]
 ;
 entry:
   %x = alloca [4 x i32]
@@ -938,43 +930,35 @@ define i32 @test14(<2 x i64> %x) {
 ; cast to a different vector type
 ; CHECK-LABEL: @test14(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[X_ADDR:%.*]] = alloca <2 x i64>, align 16
-; CHECK-NEXT:    store <2 x i64> [[X:%.*]], ptr [[X_ADDR]], align 16
-; CHECK-NEXT:    [[X_ADDR_0_A:%.*]] = load i32, ptr [[X_ADDR]], align 16
-; CHECK-NEXT:    [[X_ADDR_4_X_TMP2_SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[X_ADDR]], i64 4
-; CHECK-NEXT:    [[X_ADDR_4_B:%.*]] = load i32, ptr [[X_ADDR_4_X_TMP2_SROA_IDX]], align 4
-; CHECK-NEXT:    [[X_ADDR_8_X_TMP3_SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[X_ADDR]], i64 8
-; CHECK-NEXT:    [[X_ADDR_8_C:%.*]] = load i32, ptr [[X_ADDR_8_X_TMP3_SROA_IDX]], align 8
-; CHECK-NEXT:    [[X_ADDR_12_X_TMP4_SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[X_ADDR]], i64 12
-; CHECK-NEXT:    [[X_ADDR_12_D:%.*]] = load i32, ptr [[X_ADDR_12_X_TMP4_SROA_IDX]], align 4
-; CHECK-NEXT:    [[ADD:%.*]] = add i32 [[X_ADDR_0_A]], [[X_ADDR_4_B]]
-; CHECK-NEXT:    [[ADD1:%.*]] = add i32 [[X_ADDR_8_C]], [[X_ADDR_12_D]]
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <2 x i64> [[X:%.*]] to <4 x i32>
+; CHECK-NEXT:    [[X_ADDR_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <4 x i32> [[TMP0]], i32 0
+; CHECK-NEXT:    [[X_ADDR_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <4 x i32> [[TMP0]], i32 1
+; CHECK-NEXT:    [[X_ADDR_SROA_0_8_VEC_EXTRACT:%.*]] = extractelement <4 x i32> [[TMP0]], i32 2
+; CHECK-NEXT:    [[X_ADDR_SROA_0_12_VEC_EXTRACT:%.*]] = extractelement <4 x i32> [[TMP0]], i32 3
+; CHECK-NEXT:    [[ADD:%.*]] = add i32 [[X_ADDR_SROA_0_0_VEC_EXTRACT]], [[X_ADDR_SROA_0_4_VEC_EXTRACT]]
+; CHECK-NEXT:    [[ADD1:%.*]] = add i32 [[X_ADDR_SROA_0_8_VEC_EXTRACT]], [[X_ADDR_SROA_0_12_VEC_EXTRACT]]
 ; CHECK-NEXT:    [[ADD2:%.*]] = add i32 [[ADD]], [[ADD1]]
 ; CHECK-NEXT:    ret i32 [[ADD2]]
 ;
 ; DEBUG-LABEL: @test14(
 ; DEBUG-NEXT:  entry:
-; DEBUG-NEXT:    [[X_ADDR:%.*]] = alloca <2 x i64>, align 16, !dbg [[DBG378:![0-9]+]]
-; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr [[X_ADDR]], metadata [[META366:![0-9]+]], metadata !DIExpression()), !dbg [[DBG378]]
-; DEBUG-NEXT:    store <2 x i64> [[X:%.*]], ptr [[X_ADDR]], align 16, !dbg [[DBG379:![0-9]+]]
+; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META366:![0-9]+]], metadata !DIExpression()), !dbg [[DBG378:![0-9]+]]
+; DEBUG-NEXT:    [[TMP0:%.*]] = bitcast <2 x i64> [[X:%.*]] to <4 x i32>, !dbg [[DBG379:![0-9]+]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META367:![0-9]+]], metadata !DIExpression()), !dbg [[DBG380:![0-9]+]]
-; DEBUG-NEXT:    [[X_ADDR_0_A:%.*]] = load i32, ptr [[X_ADDR]], align 16, !dbg [[DBG381:![0-9]+]]
-; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[X_ADDR_0_A]], metadata [[META368:![0-9]+]], metadata !DIExpression()), !dbg [[DBG381]]
+; DEBUG-NEXT:    [[X_ADDR_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <4 x i32> [[TMP0]], i32 0, !dbg [[DBG381:![0-9]+]]
+; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[X_ADDR_SROA_0_0_VEC_EXTRACT]], metadata [[META368:![0-9]+]], metadata !DIExpression()), !dbg [[DBG381]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META369:![0-9]+]], metadata !DIExpression()), !dbg [[DBG382:![0-9]+]]
-; DEBUG-NEXT:    [[X_ADDR_4_X_TMP2_SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[X_ADDR]], i64 4, !dbg [[DBG383:![0-9]+]]
-; DEBUG-NEXT:    [[X_ADDR_4_B:%.*]] = load i32, ptr [[X_ADDR_4_X_TMP2_SROA_IDX]], align 4, !dbg [[DBG383]]
-; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[X_ADDR_4_B]], metadata [[META370:![0-9]+]], metadata !DIExpression()), !dbg [[DBG383]]
+; DEBUG-NEXT:    [[X_ADDR_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <4 x i32> [[TMP0]], i32 1, !dbg [[DBG383:![0-9]+]]
+; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[X_ADDR_SROA_0_4_VEC_EXTRACT]], metadata [[META370:![0-9]+]], metadata !DIExpression()), !dbg [[DBG383]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META371:![0-9]+]], metadata !DIExpression()), !dbg [[DBG384:![0-9]+]]
-; DEBUG-NEXT:    [[X_ADDR_8_X_TMP3_SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[X_ADDR]], i64 8, !dbg [[DBG385:![0-9]+]]
-; DEBUG-NEXT:    [[X_ADDR_8_C:%.*]] = load i32, ptr [[X_ADDR_8_X_TMP3_SROA_IDX]], align 8, !dbg [[DBG385]]
-; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[X_ADDR_8_C]], metadata [[META372:![0-9]+]], metadata !DIExpression()), !dbg [[DBG385]]
+; DEBUG-NEXT:    [[X_ADDR_SROA_0_8_VEC_EXTRACT:%.*]] = extractelement <4 x i32> [[TMP0]], i32 2, !dbg [[DBG385:![0-9]+]]
+; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[X_ADDR_SROA_0_8_VEC_EXTRACT]], metadata [[META372:![0-9]+]], metadata !DIExpression()), !dbg [[DBG385]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META373:![0-9]+]], metadata !DIExpression()), !dbg [[DBG386:![0-9]+]]
-; DEBUG-NEXT:    [[X_ADDR_12_X_TMP4_SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[X_ADDR]], i64 12, !dbg [[DBG387:![0-9]+]]
-; DEBUG-NEXT:    [[X_ADDR_12_D:%.*]] = load i32, ptr [[X_ADDR_12_X_TMP4_SROA_IDX]], align 4, !dbg [[DBG387]]
-; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[X_ADDR_12_D]], metadata [[META374:![0-9]+]], metadata !DIExpression()), !dbg [[DBG387]]
-; DEBUG-NEXT:    [[ADD:%.*]] = add i32 [[X_ADDR_0_A]], [[X_ADDR_4_B]], !dbg [[DBG388:![0-9]+]]
+; DEBUG-NEXT:    [[X_ADDR_SROA_0_12_VEC_EXTRACT:%.*]] = extractelement <4 x i32> [[TMP0]], i32 3, !dbg [[DBG387:![0-9]+]]
+; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[X_ADDR_SROA_0_12_VEC_EXTRACT]], metadata [[META374:![0-9]+]], metadata !DIExpression()), !dbg [[DBG387]]
+; DEBUG-NEXT:    [[ADD:%.*]] = add i32 [[X_ADDR_SROA_0_0_VEC_EXTRACT]], [[X_ADDR_SROA_0_4_VEC_EXTRACT]], !dbg [[DBG388:![0-9]+]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[ADD]], metadata [[META375:![0-9]+]], metadata !DIExpression()), !dbg [[DBG388]]
-; DEBUG-NEXT:    [[ADD1:%.*]] = add i32 [[X_ADDR_8_C]], [[X_ADDR_12_D]], !dbg [[DBG389:![0-9]+]]
+; DEBUG-NEXT:    [[ADD1:%.*]] = add i32 [[X_ADDR_SROA_0_8_VEC_EXTRACT]], [[X_ADDR_SROA_0_12_VEC_EXTRACT]], !dbg [[DBG389:![0-9]+]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[ADD1]], metadata [[META376:![0-9]+]], metadata !DIExpression()), !dbg [[DBG389]]
 ; DEBUG-NEXT:    [[ADD2:%.*]] = add i32 [[ADD]], [[ADD1]], !dbg [[DBG390:![0-9]+]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[ADD2]], metadata [[META377:![0-9]+]], metadata !DIExpression()), !dbg [[DBG390]]
diff --git a/llvm/test/Transforms/SampleProfile/inline-mergeprof-dup.ll b/llvm/test/Transforms/SampleProfile/inline-mergeprof-dup.ll
index c345d1017a455..369f1b74e0146 100644
--- a/llvm/test/Transforms/SampleProfile/inline-mergeprof-dup.ll
+++ b/llvm/test/Transforms/SampleProfile/inline-mergeprof-dup.ll
@@ -73,7 +73,7 @@ attributes #0 = { "use-sample-profile" }
 !15 = !DILocation(line: 6, scope: !12)
 
 
-;; Check the profile of funciton sum is only merged once though the original callsite is replicted.
+;; Check the profile of function sum is only merged once though the original callsite is replicted.
 ; CHECK: name: "sum"
 ; CHECK-NEXT: {!"function_entry_count", i64 46}
 ; CHECK: !{!"branch_weights", i32 11, i32 37}
diff --git a/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch-markloopasdeleted.ll b/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch-markloopasdeleted.ll
index b2c7226dba6a9..c8e1291b9cd55 100644
--- a/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch-markloopasdeleted.ll
+++ b/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch-markloopasdeleted.ll
@@ -1,4 +1,4 @@
-; RUN: opt < %s -enable-loop-distribute -passes='loop-distribute,loop-mssa(simple-loop-unswitch<nontrivial>),loop-distribute' -o /dev/null -S -verify-cfg-preserved -debug-pass-manager=verbose 2>&1 | FileCheck %s
+; RUN: opt < %s -enable-loop-distribute -passes='loop-distribute,loop-mssa(simple-loop-unswitch<nontrivial>),loop-distribute' -o /dev/null -S -verify-analysis-invalidation=0 -debug-pass-manager=verbose 2>&1 | FileCheck %s
 
 
 ; Running loop-distribute will result in LoopAccessAnalysis being required and
@@ -29,7 +29,6 @@
 ;
 ; CHECK: Invalidating analysis: LoopAccessAnalysis on test6
 ; CHECK-NEXT: Running pass: LoopDistributePass on test6
-; CHECK-NEXT: Running analysis: PreservedCFGCheckerAnalysis on test6
 ; CHECK-NEXT: Running analysis: LoopAccessAnalysis on test6
 
 
diff --git a/llvm/test/Transforms/SimplifyCFG/UnreachableEliminate.ll b/llvm/test/Transforms/SimplifyCFG/UnreachableEliminate.ll
index 1dee0704b3c30..7fe32dfc244d7 100644
--- a/llvm/test/Transforms/SimplifyCFG/UnreachableEliminate.ll
+++ b/llvm/test/Transforms/SimplifyCFG/UnreachableEliminate.ll
@@ -444,28 +444,6 @@ else:
 define void @test9_gep_inbounds_nonzero(i1 %X, ptr %Y) {
 ; CHECK-LABEL: @test9_gep_inbounds_nonzero(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[TMP0:%.*]] = xor i1 [[X:%.*]], true
-; CHECK-NEXT:    call void @llvm.assume(i1 [[TMP0]])
-; CHECK-NEXT:    [[GEP:%.*]] = getelementptr inbounds i8, ptr [[Y:%.*]], i64 12
-; CHECK-NEXT:    [[TMP1:%.*]] = call ptr @fn_nonnull_noundef_arg(ptr [[GEP]])
-; CHECK-NEXT:    ret void
-;
-entry:
-  br i1 %X, label %if, label %else
-
-if:
-  br label %else
-
-else:
-  %phi = phi ptr [ %Y, %entry ], [ null, %if ]
-  %gep = getelementptr inbounds i8, ptr %phi, i64 12
-  call ptr @fn_nonnull_noundef_arg(ptr %gep)
-  ret void
-}
-
-define void @test9_gep_inbounds_nonzero_null_defined(i1 %X, ptr %Y) #0 {
-; CHECK-LABEL: @test9_gep_inbounds_nonzero_null_defined(
-; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[SPEC_SELECT:%.*]] = select i1 [[X:%.*]], ptr null, ptr [[Y:%.*]]
 ; CHECK-NEXT:    [[GEP:%.*]] = getelementptr inbounds i8, ptr [[SPEC_SELECT]], i64 12
 ; CHECK-NEXT:    [[TMP0:%.*]] = call ptr @fn_nonnull_noundef_arg(ptr [[GEP]])
@@ -484,30 +462,9 @@ else:
   ret void
 }
 
-define void @test9_gep_inbounds_unknown_null(i1 %X, ptr %Y, i64 %I) {
-; CHECK-LABEL: @test9_gep_inbounds_unknown_null(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[TMP0:%.*]] = xor i1 [[X:%.*]], true
-; CHECK-NEXT:    call void @llvm.assume(i1 [[TMP0]])
-; CHECK-NEXT:    [[GEP:%.*]] = getelementptr inbounds i8, ptr [[Y:%.*]], i64 [[I:%.*]]
-; CHECK-NEXT:    [[TMP1:%.*]] = call ptr @fn_nonnull_noundef_arg(ptr [[GEP]])
-; CHECK-NEXT:    ret void
-;
-entry:
-  br i1 %X, label %if, label %else
-
-if:
-  br label %else
-
-else:
-  %phi = phi ptr [ %Y, %entry ], [ null, %if ]
-  %gep = getelementptr inbounds i8, ptr %phi, i64 %I
-  call ptr @fn_nonnull_noundef_arg(ptr %gep)
-  ret void
-}
 
-define void @test9_gep_inbounds_unknown_null_defined(i1 %X, ptr %Y, i64 %I) #0 {
-; CHECK-LABEL: @test9_gep_inbounds_unknown_null_defined(
+define void @test9_gep_inbouds_unknown_null(i1 %X, ptr %Y, i64 %I) {
+; CHECK-LABEL: @test9_gep_inbouds_unknown_null(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[SPEC_SELECT:%.*]] = select i1 [[X:%.*]], ptr null, ptr [[Y:%.*]]
 ; CHECK-NEXT:    [[GEP:%.*]] = getelementptr inbounds i8, ptr [[SPEC_SELECT]], i64 [[I:%.*]]
@@ -527,27 +484,6 @@ else:
   ret void
 }
 
-define void @test9_gep_inbounds_unknown_null_call_noundef(i1 %X, ptr %Y, i64 %I) {
-; CHECK-LABEL: @test9_gep_inbounds_unknown_null_call_noundef(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[SPEC_SELECT:%.*]] = select i1 [[X:%.*]], ptr null, ptr [[Y:%.*]]
-; CHECK-NEXT:    [[GEP:%.*]] = getelementptr inbounds i8, ptr [[SPEC_SELECT]], i64 [[I:%.*]]
-; CHECK-NEXT:    [[TMP0:%.*]] = call ptr @fn_noundef_arg(ptr [[GEP]])
-; CHECK-NEXT:    ret void
-;
-entry:
-  br i1 %X, label %if, label %else
-
-if:
-  br label %else
-
-else:
-  %phi = phi ptr [ %Y, %entry ], [ null, %if ]
-  %gep = getelementptr inbounds i8, ptr %phi, i64 %I
-  call ptr @fn_noundef_arg(ptr %gep)
-  ret void
-}
-
 define void @test9_gep_unknown_null(i1 %X, ptr %Y, i64 %I) {
 ; CHECK-LABEL: @test9_gep_unknown_null(
 ; CHECK-NEXT:  entry:
diff --git a/llvm/test/Transforms/SimplifyCFG/unreachable-eliminate-on-ret.ll b/llvm/test/Transforms/SimplifyCFG/unreachable-eliminate-on-ret.ll
deleted file mode 100644
index 924f98a46ab54..0000000000000
--- a/llvm/test/Transforms/SimplifyCFG/unreachable-eliminate-on-ret.ll
+++ /dev/null
@@ -1,141 +0,0 @@
-; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -passes=simplifycfg -simplifycfg-require-and-preserve-domtree=1 -S < %s | FileCheck %s
-
-define noundef i32 @test_ret_noundef(i1 %cond) {
-; CHECK-LABEL: @test_ret_noundef(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    ret i32 1
-;
-entry:
-  br i1 %cond, label %bb1, label %bb2
-
-bb1:
-  br label %bb2
-
-bb2:
-  %r = phi i32 [ undef, %entry ], [ 1, %bb1 ]
-  ret i32 %r
-}
-
-define i32 @test_ret(i1 %cond) {
-; CHECK-LABEL: @test_ret(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[SPEC_SELECT:%.*]] = select i1 [[COND:%.*]], i32 1, i32 undef
-; CHECK-NEXT:    ret i32 [[SPEC_SELECT]]
-;
-entry:
-  br i1 %cond, label %bb1, label %bb2
-
-bb1:
-  br label %bb2
-
-bb2:
-  %r = phi i32 [ undef, %entry ], [ 1, %bb1 ]
-  ret i32 %r
-}
-
-define nonnull noundef ptr @test_ret_ptr_nonnull_noundef(i1 %cond, ptr %x) {
-; CHECK-LABEL: @test_ret_ptr_nonnull_noundef(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    call void @llvm.assume(i1 [[COND:%.*]])
-; CHECK-NEXT:    ret ptr [[X:%.*]]
-;
-entry:
-  br i1 %cond, label %bb1, label %bb2
-
-bb1:
-  br label %bb2
-
-bb2:
-  %r = phi ptr [ null, %entry ], [ %x, %bb1 ]
-  ret ptr %r
-}
-
-define nonnull noundef ptr @test_ret_ptr_nonnull_noundef_gep_nonzero(i1 %cond, ptr %x) {
-; CHECK-LABEL: @test_ret_ptr_nonnull_noundef_gep_nonzero(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[SPEC_SELECT:%.*]] = select i1 [[COND:%.*]], ptr [[X:%.*]], ptr null
-; CHECK-NEXT:    [[GEP:%.*]] = getelementptr ptr, ptr [[SPEC_SELECT]], i64 12
-; CHECK-NEXT:    ret ptr [[GEP]]
-;
-entry:
-  br i1 %cond, label %bb1, label %bb2
-
-bb1:
-  br label %bb2
-
-bb2:
-  %phi = phi ptr [ null, %entry ], [ %x, %bb1 ]
-  %gep = getelementptr ptr, ptr %phi, i64 12
-  ret ptr %gep
-}
-
-define nonnull noundef ptr @test_ret_ptr_nonnull_noundef_gep_inbounds_nonzero(i1 %cond, ptr %x) {
-; CHECK-LABEL: @test_ret_ptr_nonnull_noundef_gep_inbounds_nonzero(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    call void @llvm.assume(i1 [[COND:%.*]])
-; CHECK-NEXT:    [[GEP:%.*]] = getelementptr inbounds ptr, ptr [[X:%.*]], i64 12
-; CHECK-NEXT:    ret ptr [[GEP]]
-;
-entry:
-  br i1 %cond, label %bb1, label %bb2
-
-bb1:
-  br label %bb2
-
-bb2:
-  %phi = phi ptr [ null, %entry ], [ %x, %bb1 ]
-  %gep = getelementptr inbounds ptr, ptr %phi, i64 12
-  ret ptr %gep
-}
-
-define nonnull ptr @test_ret_ptr_nonnull(i1 %cond, ptr %x) {
-; CHECK-LABEL: @test_ret_ptr_nonnull(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[SPEC_SELECT:%.*]] = select i1 [[COND:%.*]], ptr [[X:%.*]], ptr null
-; CHECK-NEXT:    ret ptr [[SPEC_SELECT]]
-;
-entry:
-  br i1 %cond, label %bb1, label %bb2
-
-bb1:
-  br label %bb2
-
-bb2:
-  %r = phi ptr [ null, %entry ], [ %x, %bb1 ]
-  ret ptr %r
-}
-
-define noundef ptr @test_ret_ptr_noundef(i1 %cond, ptr %x) {
-; CHECK-LABEL: @test_ret_ptr_noundef(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[SPEC_SELECT:%.*]] = select i1 [[COND:%.*]], ptr [[X:%.*]], ptr null
-; CHECK-NEXT:    ret ptr [[SPEC_SELECT]]
-;
-entry:
-  br i1 %cond, label %bb1, label %bb2
-
-bb1:
-  br label %bb2
-
-bb2:
-  %r = phi ptr [ null, %entry ], [ %x, %bb1 ]
-  ret ptr %r
-}
-
-define ptr @test_ret_ptr(i1 %cond, ptr %x) {
-; CHECK-LABEL: @test_ret_ptr(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[SPEC_SELECT:%.*]] = select i1 [[COND:%.*]], ptr [[X:%.*]], ptr null
-; CHECK-NEXT:    ret ptr [[SPEC_SELECT]]
-;
-entry:
-  br i1 %cond, label %bb1, label %bb2
-
-bb1:
-  br label %bb2
-
-bb2:
-  %r = phi ptr [ null, %entry ], [ %x, %bb1 ]
-  ret ptr %r
-}
diff --git a/llvm/test/Transforms/SpeculativeExecution/spec-calls.ll b/llvm/test/Transforms/SpeculativeExecution/spec-calls.ll
index 6a693b44d5380..8b40ac553ceea 100644
--- a/llvm/test/Transforms/SpeculativeExecution/spec-calls.ll
+++ b/llvm/test/Transforms/SpeculativeExecution/spec-calls.ll
@@ -109,3 +109,196 @@ b:
 }
 
 declare half @llvm.fptrunc.round.f16.f32(float, metadata)
+
+; CHECK-LABEL: @ifThen_vector_reduce_fadd(
+; CHECK: %reduce = call float @llvm.vector.reduce.fadd.v2f32(float %x, <2 x float> %y)
+; CHECK-NEXT: br i1 true
+define void @ifThen_vector_reduce_fadd(float %x, <2 x float> %y) {
+  br i1 true, label %a, label %b
+
+a:
+  %reduce = call float @llvm.vector.reduce.fadd.v2f32(float %x, <2 x float> %y)
+  br label %b
+
+b:
+  ret void
+}
+
+declare float @llvm.vector.reduce.fadd.v2f32(float, <2 x float>)
+
+; CHECK-LABEL: @ifThen_vector_reduce_fmul(
+; CHECK: %reduce = call float @llvm.vector.reduce.fmul.v2f32(float %x, <2 x float> %y)
+; CHECK-NEXT: br i1 true
+define void @ifThen_vector_reduce_fmul(float %x, <2 x float> %y) {
+  br i1 true, label %a, label %b
+
+a:
+  %reduce = call float @llvm.vector.reduce.fmul.v2f32(float %x, <2 x float> %y)
+  br label %b
+
+b:
+  ret void
+}
+
+declare float @llvm.vector.reduce.fmul.v2f32(float, <2 x float>)
+
+; CHECK-LABEL: @ifThen_vector_reduce_add(
+; CHECK: %reduce = call i32 @llvm.vector.reduce.add.v2i32(<2 x i32> %x)
+; CHECK-NEXT: br i1 true
+define void @ifThen_vector_reduce_add(<2 x i32> %x) {
+  br i1 true, label %a, label %b
+
+a:
+  %reduce = call i32 @llvm.vector.reduce.add.v2i32(<2 x i32> %x)
+  br label %b
+
+b:
+  ret void
+}
+
+declare i32 @llvm.vector.reduce.add.v2i32(<2 x i32>)
+
+; CHECK-LABEL: @ifThen_vector_reduce_mul(
+; CHECK: %reduce = call i32 @llvm.vector.reduce.mul.v2i32(<2 x i32> %x)
+; CHECK-NEXT: br i1 true
+define void @ifThen_vector_reduce_mul(<2 x i32> %x) {
+  br i1 true, label %a, label %b
+
+a:
+  %reduce = call i32 @llvm.vector.reduce.mul.v2i32(<2 x i32> %x)
+  br label %b
+
+b:
+  ret void
+}
+
+declare i32 @llvm.vector.reduce.mul.v2i32(<2 x i32>)
+
+
+; CHECK-LABEL: @ifThen_vector_reduce_and(
+; CHECK: %reduce = call i32 @llvm.vector.reduce.and.v2i32(<2 x i32> %x)
+; CHECK-NEXT: br i1 true
+define void @ifThen_vector_reduce_and(<2 x i32> %x) {
+  br i1 true, label %a, label %b
+
+a:
+  %reduce = call i32 @llvm.vector.reduce.and.v2i32(<2 x i32> %x)
+  br label %b
+
+b:
+  ret void
+}
+
+declare i32 @llvm.vector.reduce.and.v2i32(<2 x i32>)
+
+; CHECK-LABEL: @ifThen_vector_reduce_or(
+; CHECK: %reduce = call i32 @llvm.vector.reduce.or.v2i32(<2 x i32> %x)
+; CHECK-NEXT: br i1 true
+define void @ifThen_vector_reduce_or(<2 x i32> %x) {
+  br i1 true, label %a, label %b
+
+a:
+  %reduce = call i32 @llvm.vector.reduce.or.v2i32(<2 x i32> %x)
+  br label %b
+
+b:
+  ret void
+}
+
+declare i32 @llvm.vector.reduce.or.v2i32(<2 x i32>)
+
+; CHECK-LABEL: @ifThen_vector_reduce_xor(
+; CHECK: %reduce = call i32 @llvm.vector.reduce.xor.v2i32(<2 x i32> %x)
+; CHECK-NEXT: br i1 true
+define void @ifThen_vector_reduce_xor(<2 x i32> %x) {
+  br i1 true, label %a, label %b
+
+a:
+  %reduce = call i32 @llvm.vector.reduce.xor.v2i32(<2 x i32> %x)
+  br label %b
+
+b:
+  ret void
+}
+
+declare i32 @llvm.vector.reduce.xor.v2i32(<2 x i32>)
+
+; CHECK-LABEL: @ifThen_vector_reduce_smax(
+; CHECK: %reduce = call i32 @llvm.vector.reduce.smax.v2i32(<2 x i32> %x)
+; CHECK-NEXT: br i1 true
+define void @ifThen_vector_reduce_smax(<2 x i32> %x) {
+  br i1 true, label %a, label %b
+
+a:
+  %reduce = call i32 @llvm.vector.reduce.smax.v2i32(<2 x i32> %x)
+  br label %b
+
+b:
+  ret void
+}
+
+declare i32 @llvm.vector.reduce.smax.v2i32(<2 x i32>)
+
+; CHECK-LABEL: @ifThen_vector_reduce_umax(
+; CHECK: %reduce = call i32 @llvm.vector.reduce.umax.v2i32(<2 x i32> %x)
+; CHECK-NEXT: br i1 true
+define void @ifThen_vector_reduce_umax(<2 x i32> %x) {
+  br i1 true, label %a, label %b
+
+a:
+  %reduce = call i32 @llvm.vector.reduce.umax.v2i32(<2 x i32> %x)
+  br label %b
+
+b:
+  ret void
+}
+
+declare i32 @llvm.vector.reduce.umax.v2i32(<2 x i32>)
+
+; CHECK-LABEL: @ifThen_vector_reduce_umin(
+; CHECK: %reduce = call i32 @llvm.vector.reduce.umin.v2i32(<2 x i32> %x)
+; CHECK-NEXT: br i1 true
+define void @ifThen_vector_reduce_umin(<2 x i32> %x) {
+  br i1 true, label %a, label %b
+
+a:
+  %reduce = call i32 @llvm.vector.reduce.umin.v2i32(<2 x i32> %x)
+  br label %b
+
+b:
+  ret void
+}
+
+declare i32 @llvm.vector.reduce.umin.v2i32(<2 x i32>)
+
+; CHECK-LABEL: @ifThen_vector_reduce_fmax(
+; CHECK: %reduce = call float @llvm.vector.reduce.fmax.v2f32(<2 x float> %x)
+; CHECK-NEXT: br i1 true
+define void @ifThen_vector_reduce_fmax(<2 x float> %x) {
+  br i1 true, label %a, label %b
+
+a:
+  %reduce = call float @llvm.vector.reduce.fmax.v2f32(<2 x float> %x)
+  br label %b
+
+b:
+  ret void
+}
+
+declare float @llvm.vector.reduce.fmax.v2f32(<2 x float>)
+
+; CHECK-LABEL: @ifThen_vector_reduce_fmin(
+; CHECK: %reduce = call float @llvm.vector.reduce.fmin.v2f32(<2 x float> %x)
+; CHECK-NEXT: br i1 true
+define void @ifThen_vector_reduce_fmin(<2 x float> %x) {
+  br i1 true, label %a, label %b
+
+a:
+  %reduce = call float @llvm.vector.reduce.fmin.v2f32(<2 x float> %x)
+  br label %b
+
+b:
+  ret void
+}
+
+declare float @llvm.vector.reduce.fmin.v2f32(<2 x float>)
diff --git a/llvm/test/Verifier/range-1.ll b/llvm/test/Verifier/range-1.ll
index 859c7981d599b..011eb3cc98b24 100644
--- a/llvm/test/Verifier/range-1.ll
+++ b/llvm/test/Verifier/range-1.ll
@@ -147,3 +147,10 @@ entry:
 }
 !18 = !{}
 ; CHECK: It should have at least one range!
+
+define <2 x i8> @vector_range_wrong_type(ptr %x) {
+  %y = load <2 x i8>, ptr %x, !range !19
+  ret <2 x i8> %y
+}
+!19 = !{i16 0, i16 10}
+; CHECK: Range types must match instruction type!
diff --git a/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/aarch64_function_name.ll.expected b/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/aarch64_function_name.ll.expected
index 61a02b5df07e0..8367156d9e51b 100644
--- a/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/aarch64_function_name.ll.expected
+++ b/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/aarch64_function_name.ll.expected
@@ -7,12 +7,12 @@
 define hidden i32 @"_Z54bar$ompvariant$bar"() {
 ; LINUX-LABEL: _Z54bar$ompvariant$bar:
 ; LINUX:       // %bb.0: // %entry
-; LINUX-NEXT:    mov w0, #2
+; LINUX-NEXT:    mov w0, #2 // =0x2
 ; LINUX-NEXT:    ret
 ;
 ; DARWIN-LABEL: _Z54bar$ompvariant$bar:
 ; DARWIN:       ; %bb.0: ; %entry
-; DARWIN-NEXT:    mov w0, #2
+; DARWIN-NEXT:    mov w0, #2 ; =0x2
 ; DARWIN-NEXT:    ret
 entry:
   ret i32 2
diff --git a/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/aarch64_generated_funcs.ll.generated.expected b/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/aarch64_generated_funcs.ll.generated.expected
index 92e3485929d48..a75abf44201bd 100644
--- a/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/aarch64_generated_funcs.ll.generated.expected
+++ b/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/aarch64_generated_funcs.ll.generated.expected
@@ -71,7 +71,7 @@ attributes #0 = { noredzone nounwind ssp uwtable "frame-pointer"="all" }
 ; CHECK-NEXT:    .cfi_def_cfa w29, 16
 ; CHECK-NEXT:    .cfi_offset w30, -8
 ; CHECK-NEXT:    .cfi_offset w29, -16
-; CHECK-NEXT:    mov w8, #1
+; CHECK-NEXT:    mov w8, #1 // =0x1
 ; CHECK-NEXT:    stur xzr, [x29, #-8]
 ; CHECK-NEXT:    cbz wzr, .LBB0_3
 ; CHECK-NEXT:  // %bb.1:
@@ -79,7 +79,7 @@ attributes #0 = { noredzone nounwind ssp uwtable "frame-pointer"="all" }
 ; CHECK-NEXT:    ldur w8, [x29, #-8]
 ; CHECK-NEXT:    cbz w8, .LBB0_4
 ; CHECK-NEXT:  .LBB0_2:
-; CHECK-NEXT:    mov w8, #1
+; CHECK-NEXT:    mov w8, #1 // =0x1
 ; CHECK-NEXT:    str w8, [sp, #16]
 ; CHECK-NEXT:    b .LBB0_5
 ; CHECK-NEXT:  .LBB0_3:
@@ -87,7 +87,7 @@ attributes #0 = { noredzone nounwind ssp uwtable "frame-pointer"="all" }
 ; CHECK-NEXT:    ldur w8, [x29, #-8]
 ; CHECK-NEXT:    cbnz w8, .LBB0_2
 ; CHECK-NEXT:  .LBB0_4:
-; CHECK-NEXT:    mov w8, #1
+; CHECK-NEXT:    mov w8, #1 // =0x1
 ; CHECK-NEXT:    bl OUTLINED_FUNCTION_0
 ; CHECK-NEXT:  .LBB0_5:
 ; CHECK-NEXT:    mov w0, wzr
@@ -108,13 +108,13 @@ attributes #0 = { noredzone nounwind ssp uwtable "frame-pointer"="all" }
 ; CHECK-NEXT:    .cfi_def_cfa w29, 16
 ; CHECK-NEXT:    .cfi_offset w30, -8
 ; CHECK-NEXT:    .cfi_offset w29, -16
-; CHECK-NEXT:    mov w8, #1
+; CHECK-NEXT:    mov w8, #1 // =0x1
 ; CHECK-NEXT:    adrp x9, x
-; CHECK-NEXT:    mov w10, #2
-; CHECK-NEXT:    mov w11, #3
+; CHECK-NEXT:    mov w10, #2 // =0x2
+; CHECK-NEXT:    mov w11, #3 // =0x3
 ; CHECK-NEXT:    mov w0, wzr
 ; CHECK-NEXT:    str w8, [x9, :lo12:x]
-; CHECK-NEXT:    mov w9, #4
+; CHECK-NEXT:    mov w9, #4 // =0x4
 ; CHECK-NEXT:    stp w8, wzr, [x29, #-8]
 ; CHECK-NEXT:    stur w10, [x29, #-12]
 ; CHECK-NEXT:    stp w9, w11, [sp, #12]
@@ -132,9 +132,9 @@ attributes #0 = { noredzone nounwind ssp uwtable "frame-pointer"="all" }
 ;
 ; CHECK-LABEL: OUTLINED_FUNCTION_0:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w9, #2
-; CHECK-NEXT:    mov w10, #3
-; CHECK-NEXT:    mov w11, #4
+; CHECK-NEXT:    mov w9, #2 // =0x2
+; CHECK-NEXT:    mov w10, #3 // =0x3
+; CHECK-NEXT:    mov w11, #4 // =0x4
 ; CHECK-NEXT:    stp w9, w8, [x29, #-12]
 ; CHECK-NEXT:    stp w11, w10, [sp, #12]
 ; CHECK-NEXT:    ret
diff --git a/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/aarch64_generated_funcs.ll.nogenerated.expected b/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/aarch64_generated_funcs.ll.nogenerated.expected
index c59b24fbd697a..01dd58dd9a636 100644
--- a/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/aarch64_generated_funcs.ll.nogenerated.expected
+++ b/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/aarch64_generated_funcs.ll.nogenerated.expected
@@ -12,7 +12,7 @@ define dso_local i32 @check_boundaries() #0 {
 ; CHECK-NEXT:    .cfi_def_cfa w29, 16
 ; CHECK-NEXT:    .cfi_offset w30, -8
 ; CHECK-NEXT:    .cfi_offset w29, -16
-; CHECK-NEXT:    mov w8, #1
+; CHECK-NEXT:    mov w8, #1 // =0x1
 ; CHECK-NEXT:    stur xzr, [x29, #-8]
 ; CHECK-NEXT:    cbz wzr, .LBB0_3
 ; CHECK-NEXT:  // %bb.1:
@@ -20,7 +20,7 @@ define dso_local i32 @check_boundaries() #0 {
 ; CHECK-NEXT:    ldur w8, [x29, #-8]
 ; CHECK-NEXT:    cbz w8, .LBB0_4
 ; CHECK-NEXT:  .LBB0_2:
-; CHECK-NEXT:    mov w8, #1
+; CHECK-NEXT:    mov w8, #1 // =0x1
 ; CHECK-NEXT:    str w8, [sp, #16]
 ; CHECK-NEXT:    b .LBB0_5
 ; CHECK-NEXT:  .LBB0_3:
@@ -28,7 +28,7 @@ define dso_local i32 @check_boundaries() #0 {
 ; CHECK-NEXT:    ldur w8, [x29, #-8]
 ; CHECK-NEXT:    cbnz w8, .LBB0_2
 ; CHECK-NEXT:  .LBB0_4:
-; CHECK-NEXT:    mov w8, #1
+; CHECK-NEXT:    mov w8, #1 // =0x1
 ; CHECK-NEXT:    bl OUTLINED_FUNCTION_0
 ; CHECK-NEXT:  .LBB0_5:
 ; CHECK-NEXT:    mov w0, wzr
@@ -85,13 +85,13 @@ define dso_local i32 @main() #0 {
 ; CHECK-NEXT:    .cfi_def_cfa w29, 16
 ; CHECK-NEXT:    .cfi_offset w30, -8
 ; CHECK-NEXT:    .cfi_offset w29, -16
-; CHECK-NEXT:    mov w8, #1
+; CHECK-NEXT:    mov w8, #1 // =0x1
 ; CHECK-NEXT:    adrp x9, x
-; CHECK-NEXT:    mov w10, #2
-; CHECK-NEXT:    mov w11, #3
+; CHECK-NEXT:    mov w10, #2 // =0x2
+; CHECK-NEXT:    mov w11, #3 // =0x3
 ; CHECK-NEXT:    mov w0, wzr
 ; CHECK-NEXT:    str w8, [x9, :lo12:x]
-; CHECK-NEXT:    mov w9, #4
+; CHECK-NEXT:    mov w9, #4 // =0x4
 ; CHECK-NEXT:    stp w8, wzr, [x29, #-8]
 ; CHECK-NEXT:    stur w10, [x29, #-12]
 ; CHECK-NEXT:    stp w9, w11, [sp, #12]
diff --git a/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/sparc_generated_funcs.ll.generated.expected b/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/sparc_generated_funcs.ll.generated.expected
index 071afeefcda40..710ace327013d 100644
--- a/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/sparc_generated_funcs.ll.generated.expected
+++ b/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/sparc_generated_funcs.ll.generated.expected
@@ -74,12 +74,12 @@ attributes #0 = { noredzone nounwind ssp uwtable "frame-pointer"="all" }
 ; CHECK-NEXT:    st %g0, [%fp+-4]
 ; CHECK-NEXT:    cmp %g0, 0
 ; CHECK-NEXT:    st %g0, [%fp+-8]
-; CHECK-NEXT:    be .LBB0_1
+; CHECK-NEXT:    be .LBB0_2
 ; CHECK-NEXT:    mov 1, %i0
-; CHECK-NEXT:  ! %bb.2:
+; CHECK-NEXT:  ! %bb.1:
 ; CHECK-NEXT:    ba .LBB0_3
 ; CHECK-NEXT:    st %i0, [%fp+-16]
-; CHECK-NEXT:  .LBB0_1:
+; CHECK-NEXT:  .LBB0_2:
 ; CHECK-NEXT:    st %i0, [%fp+-8]
 ; CHECK-NEXT:    mov 2, %i0
 ; CHECK-NEXT:    st %i0, [%fp+-12]
@@ -90,13 +90,13 @@ attributes #0 = { noredzone nounwind ssp uwtable "frame-pointer"="all" }
 ; CHECK-NEXT:  .LBB0_3:
 ; CHECK-NEXT:    ld [%fp+-8], %i0
 ; CHECK-NEXT:    cmp %i0, 0
-; CHECK-NEXT:    be .LBB0_4
+; CHECK-NEXT:    be .LBB0_5
 ; CHECK-NEXT:    nop
-; CHECK-NEXT:  ! %bb.5:
+; CHECK-NEXT:  ! %bb.4:
 ; CHECK-NEXT:    mov 1, %i0
 ; CHECK-NEXT:    ba .LBB0_6
 ; CHECK-NEXT:    st %i0, [%fp+-16]
-; CHECK-NEXT:  .LBB0_4:
+; CHECK-NEXT:  .LBB0_5:
 ; CHECK-NEXT:    mov 1, %i0
 ; CHECK-NEXT:    st %i0, [%fp+-8]
 ; CHECK-NEXT:    mov 2, %i0
diff --git a/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/sparc_generated_funcs.ll.nogenerated.expected b/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/sparc_generated_funcs.ll.nogenerated.expected
index 2029df835f013..0eacb85cbe5b8 100644
--- a/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/sparc_generated_funcs.ll.nogenerated.expected
+++ b/llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/sparc_generated_funcs.ll.nogenerated.expected
@@ -15,12 +15,12 @@ define dso_local i32 @check_boundaries() #0 {
 ; CHECK-NEXT:    st %g0, [%fp+-4]
 ; CHECK-NEXT:    cmp %g0, 0
 ; CHECK-NEXT:    st %g0, [%fp+-8]
-; CHECK-NEXT:    be .LBB0_1
+; CHECK-NEXT:    be .LBB0_2
 ; CHECK-NEXT:    mov 1, %i0
-; CHECK-NEXT:  ! %bb.2:
+; CHECK-NEXT:  ! %bb.1:
 ; CHECK-NEXT:    ba .LBB0_3
 ; CHECK-NEXT:    st %i0, [%fp+-16]
-; CHECK-NEXT:  .LBB0_1:
+; CHECK-NEXT:  .LBB0_2:
 ; CHECK-NEXT:    st %i0, [%fp+-8]
 ; CHECK-NEXT:    mov 2, %i0
 ; CHECK-NEXT:    st %i0, [%fp+-12]
@@ -31,13 +31,13 @@ define dso_local i32 @check_boundaries() #0 {
 ; CHECK-NEXT:  .LBB0_3:
 ; CHECK-NEXT:    ld [%fp+-8], %i0
 ; CHECK-NEXT:    cmp %i0, 0
-; CHECK-NEXT:    be .LBB0_4
+; CHECK-NEXT:    be .LBB0_5
 ; CHECK-NEXT:    nop
-; CHECK-NEXT:  ! %bb.5:
+; CHECK-NEXT:  ! %bb.4:
 ; CHECK-NEXT:    mov 1, %i0
 ; CHECK-NEXT:    ba .LBB0_6
 ; CHECK-NEXT:    st %i0, [%fp+-16]
-; CHECK-NEXT:  .LBB0_4:
+; CHECK-NEXT:  .LBB0_5:
 ; CHECK-NEXT:    mov 1, %i0
 ; CHECK-NEXT:    st %i0, [%fp+-8]
 ; CHECK-NEXT:    mov 2, %i0
diff --git a/llvm/test/tools/UpdateTestChecks/update_test_checks/Inputs/basic.ll.v2.expected b/llvm/test/tools/UpdateTestChecks/update_test_checks/Inputs/basic.ll.v2.expected
new file mode 100644
index 0000000000000..991d86a063128
--- /dev/null
+++ b/llvm/test/tools/UpdateTestChecks/update_test_checks/Inputs/basic.ll.v2.expected
@@ -0,0 +1,65 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
+; Example input for update_test_checks (taken from test/Transforms/InstSimplify/add.ll)
+; RUN: opt < %s -passes=instsimplify -S | FileCheck %s
+
+define i32 @common_sub_operand(i32 %X, i32 %Y) {
+; CHECK-LABEL: define i32 @common_sub_operand
+; CHECK-SAME: (i32 [[X:%.*]], i32 [[Y:%.*]]) {
+; CHECK-NEXT:    ret i32 [[X]]
+;
+  %Z = sub i32 %X, %Y
+  %Q = add i32 %Z, %Y
+  ret i32 %Q
+}
+
+define i32 @negated_operand(i32 %x) {
+; CHECK-LABEL: define i32 @negated_operand
+; CHECK-SAME: (i32 [[X:%.*]]) {
+; CHECK-NEXT:    ret i32 0
+;
+  %negx = sub i32 0, %x
+  %r = add i32 %negx, %x
+  ret i32 %r
+}
+
+define <2 x i32> @negated_operand_commute_vec(<2 x i32> %x) {
+; CHECK-LABEL: define <2 x i32> @negated_operand_commute_vec
+; CHECK-SAME: (<2 x i32> [[X:%.*]]) {
+; CHECK-NEXT:    ret <2 x i32> zeroinitializer
+;
+  %negx = sub <2 x i32> zeroinitializer, %x
+  %r = add <2 x i32> %x, %negx
+  ret <2 x i32> %r
+}
+
+define i8 @knownnegation(i8 %x, i8 %y) {
+; CHECK-LABEL: define i8 @knownnegation
+; CHECK-SAME: (i8 [[X:%.*]], i8 [[Y:%.*]]) {
+; CHECK-NEXT:    ret i8 0
+;
+  %xy = sub i8 %x, %y
+  %yx = sub i8 %y, %x
+  %r = add i8 %xy, %yx
+  ret i8 %r
+}
+
+define <2 x i8> @knownnegation_commute_vec(<2 x i8> %x, <2 x i8> %y) {
+; CHECK-LABEL: define <2 x i8> @knownnegation_commute_vec
+; CHECK-SAME: (<2 x i8> [[X:%.*]], <2 x i8> [[Y:%.*]]) {
+; CHECK-NEXT:    ret <2 x i8> zeroinitializer
+;
+  %xy = sub <2 x i8> %x, %y
+  %yx = sub <2 x i8> %y, %x
+  %r = add <2 x i8> %yx, %xy
+  ret <2 x i8> %r
+}
+
+define i32 @nameless_value(i32 %X) {
+; CHECK-LABEL: define i32 @nameless_value
+; CHECK-SAME: (i32 [[X:%.*]]) {
+; CHECK-NEXT:    [[TMP1:%.*]] = sub i32 42, [[X]]
+; CHECK-NEXT:    ret i32 [[TMP1]]
+;
+  %1 = sub i32 42, %X
+  ret i32 %1
+}
diff --git a/llvm/test/tools/UpdateTestChecks/update_test_checks/basic.test b/llvm/test/tools/UpdateTestChecks/update_test_checks/basic.test
index 2227ead654702..2a8ebe51faf2f 100644
--- a/llvm/test/tools/UpdateTestChecks/update_test_checks/basic.test
+++ b/llvm/test/tools/UpdateTestChecks/update_test_checks/basic.test
@@ -14,3 +14,8 @@
 ## added to the update invocation below.
 # RUN: %update_test_checks %t.ll
 # RUN: diff -u %t.ll %S/Inputs/basic.ll.funcsig.expected
+## Restore the original file without --function-signature and check that
+## --version 2 will implicitly enable it and also check the return type.
+# RUN: cp -f %S/Inputs/basic.ll %t.ll
+# RUN: %update_test_checks %t.ll --version 2
+# RUN: diff -u %t.ll %S/Inputs/basic.ll.v2.expected
diff --git a/llvm/test/tools/dsymutil/Inputs/dwarf5-loclists.o b/llvm/test/tools/dsymutil/Inputs/dwarf5-loclists.o
new file mode 100644
index 0000000000000..694346426c8c4
Binary files /dev/null and b/llvm/test/tools/dsymutil/Inputs/dwarf5-loclists.o differ
diff --git a/llvm/test/tools/dsymutil/X86/debug-loc-base-addr.test b/llvm/test/tools/dsymutil/X86/debug-loc-base-addr.test
index 07244c288a13b..a36e9139bcf0b 100644
--- a/llvm/test/tools/dsymutil/X86/debug-loc-base-addr.test
+++ b/llvm/test/tools/dsymutil/X86/debug-loc-base-addr.test
@@ -1,4 +1,4 @@
-RUN: dsymutil -oso-prepend-path %p/../Inputs %p/../Inputs/private/tmp/baseaddr/loc1.x86_64 -f -o - | llvm-dwarfdump -debug-loc - | FileCheck %s
+RUN: dsymutil -oso-prepend-path %p/../Inputs %p/../Inputs/private/tmp/baseaddr/loc1.x86_64 -f -o - | llvm-dwarfdump --debug-info - | FileCheck %s
 
 The test was compiled from a single source:
 $ cat loc1.cpp
@@ -17,10 +17,14 @@ int main() {
   return 0;
 }
 
-CHECK: .debug_loc contents:
-CHECK: [0x0000000100000f94,  0x0000000100000f97): DW_OP_consts +3, DW_OP_stack_value
-CHECK: [0x0000000100000f97,  0x0000000100000f99): DW_OP_consts +4, DW_OP_stack_value
-
-CHECK: [0x0000000100000f94,  0x0000000100000f97): DW_OP_consts +5, DW_OP_stack_value
-
-CHECK: [0x0000000100000f97,  0x0000000100000f99): DW_OP_reg0 RAX
+CHECK: DW_TAG_compile_unit
+CHECK: DW_TAG_variable
+CHECK: DW_AT_location
+CHECK: [0x0000000100000f94, 0x0000000100000f97): DW_OP_consts +3, DW_OP_stack_value
+CHECK: [0x0000000100000f97, 0x0000000100000f99): DW_OP_consts +4, DW_OP_stack_value
+CHECK: DW_TAG_variable
+CHECK: DW_AT_location
+CHECK: [0x0000000100000f94, 0x0000000100000f97): DW_OP_consts +5, DW_OP_stack_value
+CHECK: DW_TAG_variable
+CHECK: DW_AT_location
+CHECK: [0x0000000100000f97, 0x0000000100000f99): DW_OP_reg0 RAX
diff --git a/llvm/test/tools/dsymutil/X86/dwarf5-loclists.test b/llvm/test/tools/dsymutil/X86/dwarf5-loclists.test
new file mode 100644
index 0000000000000..d393cda76af52
--- /dev/null
+++ b/llvm/test/tools/dsymutil/X86/dwarf5-loclists.test
@@ -0,0 +1,107 @@
+## This test checks that .debug_loclists table is correclty handled
+## and transformed into the DW_FORM_sec_offset form in case
+## --no-update. Or correctly preserved in --update case.
+
+## cat dwarf5-loclists.c
+## #include <stdio.h>
+##
+## int main ( int argv, char** argc ) {
+##   printf ("\n argv %d", argv );
+##   argv++;
+##   printf ("\n argv %d", argv );
+##   argv++;
+##   printf ("\n argv %d", &argv );
+##   return 0;
+## }
+
+## $ clang -gdwarf-5 dwarf5-loclists.c -c -O2 -o dwarf5-rnglists.o
+
+#RUN: dsymutil -oso-prepend-path %p/../Inputs -y %s -o %t.dSYM
+#RUN: llvm-dwarfdump --verify  %t.dSYM | FileCheck %s
+#RUN: llvm-dwarfdump -a --verbose  %t.dSYM | FileCheck %s --check-prefix DWARF-CHECK
+
+#RUN: dsymutil --update -oso-prepend-path %p/../Inputs -y %s -o %t.dSYM
+#RUN: llvm-dwarfdump --verify  %t.dSYM | FileCheck %s
+#RUN: llvm-dwarfdump -a --verbose  %t.dSYM | FileCheck %s --check-prefix UPD-DWARF-CHECK
+
+#CHECK: No errors.
+
+#DWARF-CHECK: DW_TAG_formal_parameter
+#DWARF-CHECK: DW_AT_name {{.*}} "dwarf5-loclists.c"
+#DWARF-CHECK-NOT: DW_AT_loclists_base
+#DWARF-CHECK: DW_TAG_formal_parameter
+#DWARF-CHECK:   DW_AT_location [DW_FORM_sec_offset]   (0x0000000c:
+#DWARF-CHECK:     [0x0000000100000f70, 0x0000000100000f87): DW_OP_reg5 RDI
+#DWARF-CHECK:     [0x0000000100000f87, 0x0000000100000f93): DW_OP_reg3 RBX
+#DWARF-CHECK:     [0x0000000100000f93, 0x0000000100000f9d): DW_OP_reg4 RSI
+#DWARF-CHECK:     [0x0000000100000fa0, 0x0000000100000fa3): DW_OP_reg3 RBX
+#DWARF-CHECK:     [0x0000000100000fa3, 0x0000000100000fbc): DW_OP_breg6 RBP-20)
+#DWARF-CHECK:   DW_AT_name {{.*}} "argv"
+#DWARF-CHECK: DW_TAG_formal_parameter
+#DWARF-CHECK: DW_AT_location [DW_FORM_sec_offset]   (0x0000004a:
+#DWARF-CHECK:     [0x0000000100000f70, 0x0000000100000f89): DW_OP_reg4 RSI
+#DWARF-CHECK:     [0x0000000100000f89, 0x0000000100000fbc): DW_OP_entry_value(DW_OP_reg4 RSI), DW_OP_stack_value)
+#DWARF-CHECK:   DW_AT_name {{.*}} "argc"
+#DWARF-CHECK: .debug_loclists contents:
+#DWARF-CHECK: 0x00000000: locations list header: length = 0x00000062, format = DWARF32, version = 0x0005, addr_size = 0x08, seg_size = 0x00, offset_entry_count = 0x00000000
+#DWARF-CHECK: 0x0000000c:
+#DWARF-CHECK:     DW_LLE_start_length    (0x0000000100000f70, 0x0000000000000017)
+#DWARF-CHECK:                => [0x0000000100000f70, 0x0000000100000f87): DW_OP_reg5 RDI
+#DWARF-CHECK:     DW_LLE_start_length    (0x0000000100000f87, 0x000000000000000c)
+#DWARF-CHECK:                => [0x0000000100000f87, 0x0000000100000f93): DW_OP_reg3 RBX
+#DWARF-CHECK:     DW_LLE_start_length    (0x0000000100000f93, 0x000000000000000a)
+#DWARF-CHECK:                => [0x0000000100000f93, 0x0000000100000f9d): DW_OP_reg4 RSI
+#DWARF-CHECK:     DW_LLE_start_length    (0x0000000100000fa0, 0x0000000000000003)
+#DWARF-CHECK:                => [0x0000000100000fa0, 0x0000000100000fa3): DW_OP_reg3 RBX
+#DWARF-CHECK:     DW_LLE_start_length    (0x0000000100000fa3, 0x0000000000000019)
+#DWARF-CHECK:                => [0x0000000100000fa3, 0x0000000100000fbc): DW_OP_breg6 RBP-20
+#DWARF-CHECK:     DW_LLE_end_of_list     ()
+#DWARF-CHECK: 0x0000004a:
+#DWARF-CHECK:     DW_LLE_start_length    (0x0000000100000f70, 0x0000000000000019)
+#DWARF-CHECK:                => [0x0000000100000f70, 0x0000000100000f89): DW_OP_reg4 RSI
+#DWARF-CHECK:     DW_LLE_start_length    (0x0000000100000f89, 0x0000000000000033)
+#DWARF-CHECK:                => [0x0000000100000f89, 0x0000000100000fbc): DW_OP_entry_value(DW_OP_reg4 RSI), DW_OP_stack_value
+#DWARF-CHECK:     DW_LLE_end_of_list     ()
+
+#UPD-DWARF-CHECK: DW_TAG_compile_unit
+#UPD-DWARF-CHECK: DW_AT_name {{.*}} "dwarf5-loclists.c"
+#UPD-DWARF-CHECK: DW_AT_loclists_base [DW_FORM_sec_offset]  (0x0000000c)
+#UPD-DWARF-CHECK: DW_TAG_formal_parameter [8]   (0x00000058)
+#UPD-DWARF-CHECK:   DW_AT_location [DW_FORM_loclistx]     (indexed (0x0) loclist = 0x00000014:
+#UPD-DWARF-CHECK:    [0x0000000000000000, 0x0000000000000017): DW_OP_reg5 RDI
+#UPD-DWARF-CHECK:    [0x0000000000000017, 0x0000000000000023): DW_OP_reg3 RBX
+#UPD-DWARF-CHECK:    [0x0000000000000023, 0x000000000000002d): DW_OP_reg4 RSI
+#UPD-DWARF-CHECK:    [0x0000000000000030, 0x0000000000000033): DW_OP_reg3 RBX
+#UPD-DWARF-CHECK:    [0x0000000000000033, 0x000000000000004c): DW_OP_breg6 RBP-20)
+#UPD-DWARF-CHECK:   DW_AT_name {{.*}} "argv"
+#UPD-DWARF-CHECK: DW_TAG_formal_parameter [8]   (0x00000058)
+#UPD-DWARF-CHECK:   DW_AT_location [DW_FORM_loclistx]     (indexed (0x1) loclist = 0x0000002f:
+#UPD-DWARF-CHECK:    [0x0000000000000000, 0x0000000000000019): DW_OP_reg4 RSI
+#UPD-DWARF-CHECK:    [0x0000000000000019, 0x000000000000004c): DW_OP_entry_value(DW_OP_reg4 RSI), DW_OP_stack_value)
+#UPD-DWARF-CHECK:   DW_AT_name {{.*}} "argc"
+#UPD-DWARF-CHECK: .debug_loclists contents:
+#UPD-DWARF-CHECK: 0x00000000: locations list header: length = 0x00000039, format = DWARF32, version = 0x0005, addr_size = 0x08, seg_size = 0x00, offset_entry_count = 0x00000002
+#UPD-DWARF-CHECK: offsets: [
+#UPD-DWARF-CHECK: 0x00000008 => 0x00000014
+#UPD-DWARF-CHECK: 0x00000023 => 0x0000002f
+#UPD-DWARF-CHECK: ]
+#UPD-DWARF-CHECK: 0x00000014:
+#UPD-DWARF-CHECK:    DW_LLE_offset_pair     (0x0000000000000000, 0x0000000000000017): DW_OP_reg5 RDI
+#UPD-DWARF-CHECK:    DW_LLE_offset_pair     (0x0000000000000017, 0x0000000000000023): DW_OP_reg3 RBX
+#UPD-DWARF-CHECK:    DW_LLE_offset_pair     (0x0000000000000023, 0x000000000000002d): DW_OP_reg4 RSI
+#UPD-DWARF-CHECK:    DW_LLE_offset_pair     (0x0000000000000030, 0x0000000000000033): DW_OP_reg3 RBX
+#UPD-DWARF-CHECK:    DW_LLE_offset_pair     (0x0000000000000033, 0x000000000000004c): DW_OP_breg6 RBP-20
+#UPD-DWARF-CHECK:    DW_LLE_end_of_list     ()
+#UPD-DWARF-CHECK: 0x0000002f:
+#UPD-DWARF-CHECK:    DW_LLE_offset_pair     (0x0000000000000000, 0x0000000000000019): DW_OP_reg4 RSI
+#UPD-DWARF-CHECK:    DW_LLE_offset_pair     (0x0000000000000019, 0x000000000000004c): DW_OP_entry_value(DW_OP_reg4 RSI), DW_OP_stack_value
+#UPD-DWARF-CHECK:    DW_LLE_end_of_list     ()
+
+---
+triple:          'x86_64-apple-darwin'
+objects:
+  - filename:        'dwarf5-loclists.o'
+    timestamp:       1676048242
+    symbols:
+      - { sym: _main, objAddr: 0x0, binAddr: 0x100000F70, size: 0x48 }
+      - { sym: _g1, binAddr: 0x100001000, size: 0x0 }
diff --git a/llvm/test/tools/llvm-cxxfilt/abitag.test b/llvm/test/tools/llvm-cxxfilt/abitag.test
index 172062b8d658b..a617f164320cc 100644
--- a/llvm/test/tools/llvm-cxxfilt/abitag.test
+++ b/llvm/test/tools/llvm-cxxfilt/abitag.test
@@ -1,6 +1,7 @@
-RUN: llvm-cxxfilt -n _Z14returns_stringB5cxx11v _Z6globalB5cxx11 _Z6globalB12a_longer_tag | FileCheck %s
+RUN: llvm-cxxfilt -n _Z14returns_stringB5cxx11v _Z6globalB5cxx11 _Z6globalB12a_longer_tag _ZN6globalB3TagC2Ev _ZN6globalB3TagD2Ev | FileCheck %s
 
 CHECK: returns_string[abi:cxx11]()
 CHECK-NEXT: global[abi:cxx11]
 CHECK-NEXT: global[abi:a_longer_tag]
-
+CHECK-NEXT: global[abi:Tag]::global()
+CHECK-NEXT: global[abi:Tag]::~global()
diff --git a/llvm/test/tools/llvm-debuginfod-find/cache.test b/llvm/test/tools/llvm-debuginfod-find/cache.test
new file mode 100644
index 0000000000000..28359b45220b3
--- /dev/null
+++ b/llvm/test/tools/llvm-debuginfod-find/cache.test
@@ -0,0 +1,38 @@
+REQUIRES: curl
+UNSUPPORTED: system-windows
+
+RUN: rm -rf %t/*
+RUN: mkdir -p %t/buildid/012345678901234{5,6}
+RUN: echo 'f' > %t/buildid/0123456789012345/debuginfo
+RUN: cp %t/buildid/012345678901234{5,6}/debuginfo
+RUN: mkdir %t/cache
+RUN: env DEBUGINFOD_CACHE_PATH=%t/cache DEBUGINFOD_URLS=file://%t \
+RUN:   llvm-debuginfod-find --debuginfo 0123456789012345 > /dev/null
+RUN: ls %t/cache | FileCheck --check-prefix=FIRST --match-full-lines --implicit-check-not {{.}} %s
+
+# Set policy to discard all but one file.
+RUN: env DEBUGINFOD_CACHE_POLICY=prune_interval=0s:cache_size_files=1 \
+RUN:     DEBUGINFOD_CACHE_PATH=%t/cache DEBUGINFOD_URLS=file://%t \
+RUN:     llvm-debuginfod-find --debuginfo 0123456789012346 > /dev/null
+RUN: ls %t/cache | FileCheck --check-prefix=SECOND --match-full-lines --implicit-check-not {{.}} %s
+
+# Add the first file back to the cache.
+RUN: env DEBUGINFOD_CACHE_PATH=%t/cache DEBUGINFOD_URLS=file://%t \
+RUN:     llvm-debuginfod-find --debuginfo 0123456789012345 > /dev/null
+RUN: ls %t/cache | FileCheck --check-prefix=BOTH --match-full-lines --implicit-check-not {{.}} %s
+
+# An invalid cache policy has no effect.
+RUN: env DEBUGINFOD_CACHE_POLICY=invalid:prune_interval=0s:cache_size_files=1 \
+RUN:     DEBUGINFOD_CACHE_PATH=%t/cache DEBUGINFOD_URLS=file://%t \
+RUN:     llvm-debuginfod-find --debuginfo 0123456789012346 > /dev/null
+RUN: ls %t/cache | FileCheck --check-prefix=BOTH --match-full-lines --implicit-check-not {{.}} %s
+
+FIRST: llvmcache-6663022027345102449
+FIRST: llvmcache.timestamp
+
+SECOND: llvmcache-6762748755046480574
+SECOND: llvmcache.timestamp
+
+BOTH: llvmcache-6663022027345102449
+BOTH: llvmcache-6762748755046480574
+BOTH: llvmcache.timestamp
diff --git a/llvm/test/tools/llvm-dwarfutil/ELF/X86/dwarf5-addrx.test b/llvm/test/tools/llvm-dwarfutil/ELF/X86/dwarf5-addrx.test
index 65e5e81ddca78..bf52bac1237c1 100644
--- a/llvm/test/tools/llvm-dwarfutil/ELF/X86/dwarf5-addrx.test
+++ b/llvm/test/tools/llvm-dwarfutil/ELF/X86/dwarf5-addrx.test
@@ -21,7 +21,7 @@
 #DWARF-CHECK: DW_TAG_compile_unit
 #DWARF-CHECK:   DW_AT_name [DW_FORM_strp]  {{.*}}     "CU1"
 #DWARF-CHECK:   DW_AT_low_pc [DW_FORM_addr]     (0x0000000000001130)
-#DWARF-CHECK:   DW_AT_high_pc [DW_FORM_data8]   (0x0000000000000030)
+#DWARF-CHECK:   DW_AT_high_pc [DW_FORM_data8]   (0x0000000000000060)
 #DWARF-CHECK:   DW_TAG_subprogram
 #DWARF-CHECK:     DW_AT_name  [DW_FORM_strp] {{.*}}   "foo1"
 #DWARF-CHECK:     DW_AT_low_pc [DW_FORM_addr]     (0x0000000000001130)
@@ -34,12 +34,24 @@
 #DWARF-CHECK:     DW_AT_name  [DW_FORM_strp] {{.*}}   "foo3"
 #DWARF-CHECK:     DW_AT_low_pc [DW_FORM_addr]     (0x0000000000001150)
 #DWARF-CHECK:     DW_AT_high_pc [DW_FORM_data8]   (0x0000000000000010)
+#DWARF-CHECK:   DW_TAG_subprogram
+#DWARF-CHECK:     DW_AT_name  [DW_FORM_strp] {{.*}}   "foo4"
+#DWARF-CHECK:     DW_AT_low_pc [DW_FORM_addr]     (0x0000000000001160)
+#DWARF-CHECK:     DW_AT_high_pc [DW_FORM_data8]   (0x0000000000000010)
+#DWARF-CHECK:   DW_TAG_subprogram
+#DWARF-CHECK:     DW_AT_name  [DW_FORM_strp] {{.*}}   "foo5"
+#DWARF-CHECK:     DW_AT_low_pc [DW_FORM_addr]     (0x0000000000001170)
+#DWARF-CHECK:     DW_AT_high_pc [DW_FORM_data8]   (0x0000000000000010)
+#DWARF-CHECK:   DW_TAG_subprogram
+#DWARF-CHECK:     DW_AT_name  [DW_FORM_strp] {{.*}}   "foo6"
+#DWARF-CHECK:     DW_AT_low_pc [DW_FORM_addr]     (0x0000000000001180)
+#DWARF-CHECK:     DW_AT_high_pc [DW_FORM_data8]   (0x0000000000000010)
 #DWARF-CHECK=NOT: .debug_addr contents:
 
 #UPD-DWARF-CHECK: DW_TAG_compile_unit
 #UPD-DWARF-CHECK:   DW_AT_name {{.*}}"CU1"
 #UPD-DWARF-CHECK:   DW_AT_low_pc [DW_FORM_addrx]   (indexed (00000000) address = 0x0000000000001130)
-#UPD-DWARF-CHECK:   DW_AT_high_pc [DW_FORM_data8]   (0x0000000000000030)
+#UPD-DWARF-CHECK:   DW_AT_high_pc [DW_FORM_data8]   (0x0000000000000060)
 #UPD-DWARF-CHECK:   DW_TAG_subprogram
 #UPD-DWARF-CHECK:     DW_AT_name  {{.*}}"foo1"
 #UPD-DWARF-CHECK:     DW_AT_low_pc [DW_FORM_addrx]   (indexed (00000000) address = 0x0000000000001130)
@@ -52,12 +64,27 @@
 #UPD-DWARF-CHECK:     DW_AT_name  {{.*}}"foo3"
 #UPD-DWARF-CHECK:     DW_AT_low_pc [DW_FORM_addrx]   (indexed (00000002) address = 0x0000000000001150)
 #UPD-DWARF-CHECK:     DW_AT_high_pc [DW_FORM_data8]   (0x0000000000000010)
+#UPD-DWARF-CHECK:   DW_TAG_subprogram
+#UPD-DWARF-CHECK:     DW_AT_name  {{.*}}"foo4"
+#UPD-DWARF-CHECK:     DW_AT_low_pc [DW_FORM_addrx1]   (indexed (00000003) address = 0x0000000000001160)
+#UPD-DWARF-CHECK:     DW_AT_high_pc [DW_FORM_data8]   (0x0000000000000010)
+#UPD-DWARF-CHECK:   DW_TAG_subprogram
+#UPD-DWARF-CHECK:     DW_AT_name  {{.*}}"foo5"
+#UPD-DWARF-CHECK:     DW_AT_low_pc [DW_FORM_addrx2]   (indexed (00000004) address = 0x0000000000001170)
+#UPD-DWARF-CHECK:     DW_AT_high_pc [DW_FORM_data8]   (0x0000000000000010)
+#UPD-DWARF-CHECK:   DW_TAG_subprogram
+#UPD-DWARF-CHECK:     DW_AT_name  {{.*}}"foo6"
+#UPD-DWARF-CHECK:     DW_AT_low_pc [DW_FORM_addrx4]   (indexed (00000005) address = 0x0000000000001180)
+#UPD-DWARF-CHECK:     DW_AT_high_pc [DW_FORM_data8]   (0x0000000000000010)
 #UPD-DWARF-CHECK: .debug_addr contents:
-#UPD-DWARF-CHECK: 0x00000000: Address table header: length = 0x0000001c, format = DWARF32, version = 0x0005, addr_size = 0x08, seg_size = 0x00
+#UPD-DWARF-CHECK: 0x00000000: Address table header: length = 0x00000034, format = DWARF32, version = 0x0005, addr_size = 0x08, seg_size = 0x00
 #UPD-DWARF-CHECK: Addrs: [
 #UPD-DWARF-CHECK: 0x0000000000001130
 #UPD-DWARF-CHECK: 0x0000000000001140
 #UPD-DWARF-CHECK: 0x0000000000001150
+#UPD-DWARF-CHECK: 0x0000000000001160
+#UPD-DWARF-CHECK: 0x0000000000001170
+#UPD-DWARF-CHECK: 0x0000000000001180
 #UPD-DWARF-CHECK: ]
 
 --- !ELF
@@ -71,7 +98,7 @@ Sections:
     Type:            SHT_PROGBITS
     Flags:           [ SHF_ALLOC, SHF_EXECINSTR ]
     Address:         0x1130
-    Size:            0x30
+    Size:            0x60
 DWARF:
   debug_abbrev:
     - Table:
@@ -101,6 +128,39 @@ DWARF:
             Form:      DW_FORM_data8
           - Attribute: DW_AT_type
             Form:      DW_FORM_ref4
+      - Tag:      DW_TAG_subprogram
+        Children: DW_CHILDREN_yes
+        Attributes:
+          - Attribute: DW_AT_name
+            Form:      DW_FORM_string
+          - Attribute: DW_AT_low_pc
+            Form:      DW_FORM_addrx1
+          - Attribute: DW_AT_high_pc
+            Form:      DW_FORM_data8
+          - Attribute: DW_AT_type
+            Form:      DW_FORM_ref4
+      - Tag:      DW_TAG_subprogram
+        Children: DW_CHILDREN_yes
+        Attributes:
+          - Attribute: DW_AT_name
+            Form:      DW_FORM_string
+          - Attribute: DW_AT_low_pc
+            Form:      DW_FORM_addrx2
+          - Attribute: DW_AT_high_pc
+            Form:      DW_FORM_data8
+          - Attribute: DW_AT_type
+            Form:      DW_FORM_ref4
+      - Tag:      DW_TAG_subprogram
+        Children: DW_CHILDREN_yes
+        Attributes:
+          - Attribute: DW_AT_name
+            Form:      DW_FORM_string
+          - Attribute: DW_AT_low_pc
+            Form:      DW_FORM_addrx4
+          - Attribute: DW_AT_high_pc
+            Form:      DW_FORM_data8
+          - Attribute: DW_AT_type
+            Form:      DW_FORM_ref4
       - Tag:      DW_TAG_base_type
         Children: DW_CHILDREN_no
         Attributes:
@@ -116,30 +176,51 @@ DWARF:
             - Value:  0x04
             - CStr: CU1
             - Value:  0x0
-            - Value:  0x30
+            - Value:  0x60
             - Value:  0x8
         - AbbrCode: 2
           Values:
             - CStr: foo1
             - Value: 0x0
             - Value: 0x10
-            - Value: 0x64
+            - Value: 0xa4
         - AbbrCode: 0
         - AbbrCode: 2
           Values:
             - CStr: foo2
             - Value: 0x01
             - Value: 0x10
-            - Value: 0x64
+            - Value: 0xa4
         - AbbrCode: 0
         - AbbrCode: 2
           Values:
             - CStr: foo3
             - Value: 0x02
             - Value: 0x10
-            - Value: 0x64
+            - Value: 0xa4
         - AbbrCode: 0
         - AbbrCode: 3
+          Values:
+            - CStr: foo4
+            - Value: 0x03
+            - Value: 0x10
+            - Value: 0xa4
+        - AbbrCode: 0
+        - AbbrCode: 4
+          Values:
+            - CStr: foo5
+            - Value: 0x04
+            - Value: 0x10
+            - Value: 0xa4
+        - AbbrCode: 0
+        - AbbrCode: 5
+          Values:
+            - CStr: foo6
+            - Value: 0x05
+            - Value: 0x10
+            - Value: 0xa4
+        - AbbrCode: 0
+        - AbbrCode: 6
           Values:
             - CStr: int
         - AbbrCode: 0
@@ -150,4 +231,7 @@ DWARF:
         - Address: 0x1130
         - Address: 0x1140
         - Address: 0x1150
+        - Address: 0x1160
+        - Address: 0x1170
+        - Address: 0x1180
 ...
diff --git a/llvm/test/tools/llvm-dwarfutil/ELF/X86/dwarf5-attributes.test b/llvm/test/tools/llvm-dwarfutil/ELF/X86/dwarf5-attributes.test
new file mode 100644
index 0000000000000..779e165ab72b8
--- /dev/null
+++ b/llvm/test/tools/llvm-dwarfutil/ELF/X86/dwarf5-attributes.test
@@ -0,0 +1,130 @@
+## Test that DWARFv5 DW_FORM_implicit_const is correctly recognized
+## and copied into the result.
+
+# RUN: yaml2obj %s -o %t.o
+
+# RUN: llvm-dwarfutil %t.o %t1
+# RUN: llvm-dwarfdump -verify %t1 | FileCheck %s --check-prefix VERIFY-CHECK
+# RUN: llvm-dwarfdump -a --verbose %t1 | FileCheck %s
+#
+# RUN: llvm-dwarfutil --no-garbage-collection %t.o %t1
+# RUN: llvm-dwarfdump -verify %t1 | FileCheck %s --check-prefix VERIFY-CHECK
+# RUN: llvm-dwarfdump -a --verbose %t1 | FileCheck %s
+#
+# RUN: llvm-dwarfutil --no-garbage-collection --build-accelerator=DWARF %t.o %t1
+# RUN: llvm-dwarfdump -verify %t1 | FileCheck %s --check-prefix VERIFY-CHECK
+# RUN: llvm-dwarfdump -a --verbose %t1 | FileCheck %s
+
+#VERIFY-CHECK: No errors.
+
+#CHECK: .debug_abbrev
+#CHECK: DW_TAG_compile_unit
+#CHECK: DW_TAG_subprogram
+#CHECK: DW_TAG_base_type
+#CHECK: DW_TAG_variable
+#CHECK: DW_AT_name
+#CHECK: DW_AT_const_value       DW_FORM_implicit_const  33
+#CHECK: DW_AT_type
+#CHECK: DW_TAG_compile_unit
+#CHECK:   DW_AT_name {{.*}}"CU1"
+#CHECK:   DW_AT_low_pc
+#CHECK:   DW_AT_high_pc
+#CHECK:   DW_TAG_subprogram
+#CHECK:     DW_AT_name {{.*}}"foo1"
+#CHECK:     DW_AT_low_pc
+#CHECK:     DW_AT_high_pc
+#CHECK:  DW_TAG_variable
+#CHECK:     DW_AT_name {{.*}}"var1"
+#CHECK:     DW_AT_const_value [DW_FORM_implicit_const]      (33)
+#CHECK:     DW_AT_type {{.*}}"int"
+
+--- !ELF
+FileHeader:
+  Class:    ELFCLASS64
+  Data:     ELFDATA2LSB
+  Type:     ET_REL
+  Machine:  EM_X86_64
+Sections:
+  - Name:            .text
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_ALLOC, SHF_EXECINSTR ]
+    Address:         0x1130
+    Size:            0x60
+DWARF:
+  debug_abbrev:
+    - Table:
+      - Tag:      DW_TAG_compile_unit
+        Children: DW_CHILDREN_yes
+        Attributes:
+          - Attribute: DW_AT_producer
+            Form:      DW_FORM_string
+          - Attribute: DW_AT_language
+            Form:      DW_FORM_data2
+          - Attribute: DW_AT_name
+            Form:      DW_FORM_string
+          - Attribute: DW_AT_low_pc
+            Form:      DW_FORM_addrx
+          - Attribute: DW_AT_high_pc
+            Form:      DW_FORM_data8
+          - Attribute: DW_AT_addr_base
+            Form:      DW_FORM_sec_offset
+      - Tag:      DW_TAG_subprogram
+        Children: DW_CHILDREN_yes
+        Attributes:
+          - Attribute: DW_AT_name
+            Form:      DW_FORM_string
+          - Attribute: DW_AT_low_pc
+            Form:      DW_FORM_addrx
+          - Attribute: DW_AT_high_pc
+            Form:      DW_FORM_data8
+          - Attribute: DW_AT_type
+            Form:      DW_FORM_ref4
+      - Tag:      DW_TAG_base_type
+        Children: DW_CHILDREN_no
+        Attributes:
+          - Attribute: DW_AT_name
+            Form:      DW_FORM_string
+      - Tag:      DW_TAG_variable
+        Children: DW_CHILDREN_no
+        Attributes:
+          - Attribute: DW_AT_name
+            Form:      DW_FORM_string
+          - Attribute: DW_AT_const_value
+            Form:      DW_FORM_implicit_const
+            Value:     33
+          - Attribute: DW_AT_type
+            Form:      DW_FORM_ref4
+  debug_info:
+    - Version: 5
+      UnitType:   DW_UT_compile
+      Entries:
+        - AbbrCode: 1
+          Values:
+            - CStr: by_hand
+            - Value:  0x04
+            - CStr: CU1
+            - Value:  0x0
+            - Value:  0x10
+            - Value:  0x8
+        - AbbrCode: 2
+          Values:
+            - CStr: foo1
+            - Value: 0x0
+            - Value: 0x10
+            - Value: 0x3c
+        - AbbrCode: 0
+        - AbbrCode: 3
+          Values:
+            - CStr: int
+        - AbbrCode: 4
+          Values:
+            - CStr: var1
+            - Value:  0x00000000
+            - Value:  0x0000003c
+        - AbbrCode: 0
+  debug_addr:
+    - Version: 5
+      AddressSize: 0x08
+      Entries:
+        - Address: 0x1130
+...
diff --git a/llvm/test/tools/llvm-dwarfutil/ELF/X86/dwarf5-loclists.test b/llvm/test/tools/llvm-dwarfutil/ELF/X86/dwarf5-loclists.test
new file mode 100644
index 0000000000000..89b01c565aa0a
--- /dev/null
+++ b/llvm/test/tools/llvm-dwarfutil/ELF/X86/dwarf5-loclists.test
@@ -0,0 +1,201 @@
+## Test that DWARFv5 .debug_loclists is correctly recognized
+## and converted into the DW_FORM_sec_offset form in --garbage-collection
+## case or correctly preserved in --no-garbage-collection case.
+
+# RUN: yaml2obj %s -o %t.o
+
+# RUN: llvm-dwarfutil %t.o %t1
+# RUN: llvm-dwarfdump -verify %t1 | FileCheck %s
+# RUN: llvm-dwarfdump -a --verbose %t1 | FileCheck %s --check-prefix DWARF-CHECK
+
+# RUN: llvm-dwarfutil --no-garbage-collection %t.o %t1
+# RUN: llvm-dwarfdump -verify %t1 | FileCheck %s
+# RUN: llvm-dwarfdump -a --verbose %t1 | FileCheck %s --check-prefix UPD-DWARF-CHECK
+
+# RUN: llvm-dwarfutil --no-garbage-collection --build-accelerator=DWARF %t.o %t1
+# RUN: llvm-dwarfdump -verify %t1 | FileCheck %s
+# RUN: llvm-dwarfdump -a --verbose %t1 | FileCheck %s --check-prefix UPD-DWARF-CHECK
+
+#CHECK: No errors.
+
+#DWARF-CHECK: DW_TAG_compile_unit
+#DWARF-CHECK: DW_AT_name {{.*}}"CU1"
+#DWARF-CHECK-NOT: DW_AT_loclists_base
+#DWARF-CHECK: DW_TAG_variable
+#DWARF-CHECK: DW_AT_name {{.*}}"var1"
+#DWARF-CHECK: DW_AT_location [DW_FORM_sec_offset]      (0x0000000c:
+#DWARF-CHECK: [0x0000000000001130, 0x0000000000001140): DW_OP_reg5 RDI)
+#DWARF-CHECK: DW_AT_name {{.*}}"var2"
+#DWARF-CHECK: DW_AT_location [DW_FORM_sec_offset]     (0x00000019:
+#DWARF-CHECK: <default>: DW_OP_reg5 RDI)
+#DWARF-CHECK: DW_TAG_variable
+#DWARF-CHECK: DW_AT_name {{.*}}"var3"
+#DWARF-CHECK: DW_AT_location [DW_FORM_sec_offset]     (0x0000001d:
+#DWARF-CHECK: [0x0000000000001140, 0x0000000000001150): DW_OP_reg5 RDI
+#DWARF-CHECK: [0x0000000000001160, 0x0000000000001170): DW_OP_reg6 RBP)
+
+#UPD-DWARF-CHECK: DW_TAG_compile_unit
+#UPD-DWARF-CHECK:   DW_AT_name {{.*}}"CU1"
+#UPD-DWARF-CHECK:              DW_AT_loclists_base [DW_FORM_sec_offset]  (0x0000000c)
+#UPD-DWARF-CHECK:   DW_TAG_variable
+#UPD-DWARF-CHECK:     DW_AT_name {{.*}}"var1"
+#UPD-DWARF-CHECK:     DW_AT_location [DW_FORM_loclistx]       (indexed (0x0) loclist = 0x00000018:
+#UPD-DWARF-CHECK:     [0x0000000000001130, 0x0000000000001140): DW_OP_reg5 RDI)
+#UPD-DWARF-CHECK:   DW_TAG_variable
+#UPD-DWARF-CHECK:     DW_AT_name {{.*}}"var2"
+#UPD-DWARF-CHECK:     DW_AT_location [DW_FORM_loclistx]       (indexed (0x1) loclist = 0x0000001e:
+#UPD-DWARF-CHECK:        <default>: DW_OP_reg5 RDI)
+#UPD-DWARF-CHECK:   DW_TAG_variable
+#UPD-DWARF-CHECK:     DW_AT_name {{.*}}"var3"
+#UPD-DWARF-CHECK:     DW_AT_location [DW_FORM_loclistx]       (indexed (0x2) loclist = 0x00000022:
+#UPD-DWARF-CHECK:        [0x0000000000001140, 0x0000000000001150): DW_OP_reg5 RDI
+#UPD-DWARF-CHECK:        [0x0000000000001160, 0x0000000000001170): DW_OP_reg6 RBP)
+
+## Following yaml description has Content of the .debug_rnglists exactly like following data vvvvvvvvvvv
+## .debug_rnglists contents:
+## 0x00000000: range list header: length = 0x0000003e, format = DWARF32, version = 0x0005, addr_size = 0x08, seg_size = 0x00, offset_entry_count = 0x00000005
+## offsets: [
+## 0x00000014 => 0x00000020
+## 0x00000018 => 0x00000024
+## 0x0000001c => 0x00000028
+## 0x00000027 => 0x00000033
+## 0x00000032 => 0x0000003e
+## ]
+## ranges:
+## 0x00000020: [DW_RLE_startx_length]:  0x0000000000000000, 0x0000000000000010 => [0x0000000000001130, 0x0000000000001140)
+## 0x00000023: [DW_RLE_end_of_list  ]
+## 0x00000024: [DW_RLE_startx_length]:  0x0000000000000001, 0x0000000000000010 => [0x0000000000001140, 0x0000000000001150)
+## 0x00000027: [DW_RLE_end_of_list  ]
+## 0x00000028: [DW_RLE_start_length ]:  0x0000000000001150, 0x0000000000000010 => [0x0000000000001150, 0x0000000000001160)
+## 0x00000032: [DW_RLE_end_of_list  ]
+## 0x00000033: [DW_RLE_start_length ]:  0x0000000000001160, 0x0000000000000010 => [0x0000000000001160, 0x0000000000001170)
+## 0x0000003d: [DW_RLE_end_of_list  ]
+## 0x0000003e: [DW_RLE_startx_length]:  0x0000000000000000, 0x0000000000000040 => [0x0000000000001130, 0x0000000000001170)
+## 0x00000041: [DW_RLE_end_of_list  ]
+
+
+## Following yaml description has Content of the .debug_loclists exactly like following data vvvvvvvvvvv
+## .debug_loclists contents:
+## 0x00000000: locations list header: length = 0x00000029, format = DWARF32, version = 0x0005, addr_size = 0x08, seg_size = 0x00, offset_entry_count = 0x00000003
+## offsets: [
+## 0x0000000c => 0x00000018
+## 0x00000012 => 0x0000001e
+## 0x00000016 => 0x00000022
+## ]
+## 0x00000018:
+##             DW_LLE_startx_length   (0x0000000000000000, 0x0000000000000010): DW_OP_reg5 RDI
+##             DW_LLE_end_of_list     ()
+##
+## 0x0000001e:
+##             DW_LLE_default_location()
+##                       => <default>: DW_OP_reg5 RDI
+##             DW_LLE_end_of_list     ()
+##
+## 0x00000022:
+##             DW_LLE_startx_length   (0x0000000000000001, 0x0000000000000010): DW_OP_reg5 RDI
+##             DW_LLE_startx_length   (0x0000000000000003, 0x0000000000000010): DW_OP_reg6 RBP
+##             DW_LLE_end_of_list     ()
+
+--- !ELF
+FileHeader:
+  Class:    ELFCLASS64
+  Data:     ELFDATA2LSB
+  Type:     ET_REL
+  Machine:  EM_X86_64
+Sections:
+  - Name:            .text
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_ALLOC, SHF_EXECINSTR ]
+    Address:         0x1130
+    Size:            0x70
+  - Name:            .debug_rnglists
+    Type:            SHT_PROGBITS
+    Flags:           [  ]
+    Content:        "3e000000050008000500000014000000180000001c000000270000003200000003001000030110000750110000000000001000076011000000000000100003004000"
+  - Name:            .debug_loclists
+    Type:            SHT_PROGBITS
+    Flags:           [  ]
+    Content:        "2900000005000800030000000c0000001200000016000000030010015500050155000301100155030310015600"
+DWARF:
+  debug_abbrev:
+    - Table:
+      - Tag:      DW_TAG_compile_unit
+        Children: DW_CHILDREN_yes
+        Attributes:
+          - Attribute: DW_AT_producer
+            Form:      DW_FORM_string
+          - Attribute: DW_AT_language
+            Form:      DW_FORM_data2
+          - Attribute: DW_AT_name
+            Form:      DW_FORM_string
+          - Attribute: DW_AT_low_pc
+            Form:      DW_FORM_addrx
+          - Attribute: DW_AT_ranges
+            Form:      DW_FORM_rnglistx
+          - Attribute: DW_AT_rnglists_base
+            Form:      DW_FORM_sec_offset
+          - Attribute: DW_AT_loclists_base
+            Form:      DW_FORM_sec_offset
+          - Attribute: DW_AT_addr_base
+            Form:      DW_FORM_sec_offset
+      - Tag:      DW_TAG_base_type
+        Children: DW_CHILDREN_no
+        Attributes:
+          - Attribute: DW_AT_name
+            Form:      DW_FORM_string
+      - Tag:      DW_TAG_variable
+        Children: DW_CHILDREN_no
+        Attributes:
+          - Attribute: DW_AT_name
+            Form:      DW_FORM_string
+          - Attribute: DW_AT_const_value
+            Form:      DW_FORM_data4
+          - Attribute: DW_AT_type
+            Form:      DW_FORM_ref_addr
+          - Attribute: DW_AT_location
+            Form:      DW_FORM_loclistx
+  debug_info:
+    - Version: 5
+      UnitType:   DW_UT_compile
+      Entries:
+        - AbbrCode: 1
+          Values:
+            - CStr: by_hand
+            - Value:  0x04
+            - CStr: CU1
+            - Value:  0x0
+            - Value:  0x4
+            - Value:  0xc
+            - Value:  0xc
+            - Value:  0x8
+        - AbbrCode: 2
+          Values:
+            - CStr: int
+        - AbbrCode: 3
+          Values:
+            - CStr: var1
+            - Value:  0x00000000
+            - Value:  0x00000029
+            - Value:  0x0
+        - AbbrCode: 3
+          Values:
+            - CStr: var2
+            - Value:  0x00000000
+            - Value:  0x00000029
+            - Value:  0x1
+        - AbbrCode: 3
+          Values:
+            - CStr: var3
+            - Value:  0x00000000
+            - Value:  0x00000029
+            - Value:  0x2
+        - AbbrCode: 0
+  debug_addr:
+    - Version: 5
+      AddressSize: 0x08
+      Entries:
+        - Address: 0x1130
+        - Address: 0x1140
+        - Address: 0x1150
+        - Address: 0x1160
+...
diff --git a/llvm/test/tools/llvm-dwarfutil/ELF/X86/warning-skipped-loclists.test b/llvm/test/tools/llvm-dwarfutil/ELF/X86/warning-skipped-loclists.test
deleted file mode 100644
index 09e2cea4993da..0000000000000
--- a/llvm/test/tools/llvm-dwarfutil/ELF/X86/warning-skipped-loclists.test
+++ /dev/null
@@ -1,54 +0,0 @@
-## This test checks the warning message displayed if input file
-## contains .debug_loclists section.
-
-# RUN: yaml2obj %s -o %t.o
-
-# RUN: llvm-dwarfutil --garbage-collection %t.o %t1 2>&1 | FileCheck %s -DFILE=%t.o
-
-# CHECK: [[FILE]]: warning: '.debug_loclists' is not currently supported: file will be skipped
-
---- !ELF
-FileHeader:
-  Class:    ELFCLASS64
-  Data:     ELFDATA2LSB
-  Type:     ET_REL
-  Machine:  EM_X86_64
-Sections:
-  - Name:            .text
-    Type:            SHT_PROGBITS
-    Flags:           [ SHF_ALLOC, SHF_EXECINSTR ]
-    Address:         0x1000
-    AddressAlign:    0x0000000000000010
-    Content:        "FFFFFFFF"
-  - Name:            .debug_loclists
-    Type:            SHT_PROGBITS
-    Flags:           [  ]
-    Content:        "0000000000000000000000"
-DWARF:
-  debug_abbrev:
-    - Table:
-      - Tag:      DW_TAG_compile_unit
-        Children: DW_CHILDREN_yes
-        Attributes:
-          - Attribute: DW_AT_producer
-            Form:      DW_FORM_string
-          - Attribute: DW_AT_language
-            Form:      DW_FORM_data2
-          - Attribute: DW_AT_name
-            Form:      DW_FORM_string
-          - Attribute: DW_AT_low_pc
-            Form:      DW_FORM_addr
-          - Attribute: DW_AT_high_pc
-            Form:      DW_FORM_data8
-  debug_info:
-    - Version: 4
-      Entries:
-        - AbbrCode: 1
-          Values:
-            - CStr: by_hand
-            - Value:  0x04
-            - CStr: CU1
-            - Value:  0x1000
-            - Value:  0x4
-        - AbbrCode: 0
-...
diff --git a/llvm/test/tools/llvm-exegesis/PowerPC/lit.local.cfg b/llvm/test/tools/llvm-exegesis/PowerPC/lit.local.cfg
index 135f5796b5837..2fb4364f3721c 100644
--- a/llvm/test/tools/llvm-exegesis/PowerPC/lit.local.cfg
+++ b/llvm/test/tools/llvm-exegesis/PowerPC/lit.local.cfg
@@ -1,3 +1,6 @@
 if not ('PowerPC' in config.root.targets):
     # We need support for PowerPC.
     config.unsupported = True
+elif not ('powerpc' in config.root.host_triple):
+    # We need to be running on an PPC host.
+    config.unsupported = True
diff --git a/llvm/test/tools/llvm-exegesis/X86/lit.local.cfg b/llvm/test/tools/llvm-exegesis/X86/lit.local.cfg
index 7230f35fdb833..7d6abe8807057 100644
--- a/llvm/test/tools/llvm-exegesis/X86/lit.local.cfg
+++ b/llvm/test/tools/llvm-exegesis/X86/lit.local.cfg
@@ -1,3 +1,6 @@
 if not ('X86' in config.root.targets):
     # We need support for X86.
     config.unsupported = True
+elif not ('x86_64' in config.root.host_triple):
+    # We need to be running on an X86 host.
+    config.unsupported = True
diff --git a/llvm/test/tools/llvm-mc/disassembler-options.test b/llvm/test/tools/llvm-mc/disassembler-options.test
index f9dc2eceac67c..a7fd8b1b9f8d3 100644
--- a/llvm/test/tools/llvm-mc/disassembler-options.test
+++ b/llvm/test/tools/llvm-mc/disassembler-options.test
@@ -1,3 +1,4 @@
+# RUN: export LSAN_OPTIONS=detect_leaks=0
 # RUN: not llvm-mc -M invalid /dev/null 2>&1 | FileCheck %s
 
 # CHECK: error: invalid disassembler option 'invalid'
diff --git a/llvm/test/tools/llvm-mca/AArch64/Neoverse/N2-basic-instructions.s b/llvm/test/tools/llvm-mca/AArch64/Neoverse/N2-basic-instructions.s
index 83ffb37af726e..f4c4a20573c4e 100644
--- a/llvm/test/tools/llvm-mca/AArch64/Neoverse/N2-basic-instructions.s
+++ b/llvm/test/tools/llvm-mca/AArch64/Neoverse/N2-basic-instructions.s
@@ -1406,11 +1406,11 @@ drps
 # CHECK-NEXT:  1      1     0.25                        mov	wsp, w20
 # CHECK-NEXT:  1      1     0.25                        mov	x11, sp
 # CHECK-NEXT:  1      1     0.25                        mov	w24, wsp
-# CHECK-NEXT:  1      2     0.50                        add	w3, w5, w7
-# CHECK-NEXT:  1      2     0.50                        add	wzr, w3, w5
-# CHECK-NEXT:  1      2     0.50                        add	w20, wzr, w4
-# CHECK-NEXT:  1      2     0.50                        add	w4, w6, wzr
-# CHECK-NEXT:  1      2     0.50                        add	w11, w13, w15
+# CHECK-NEXT:  1      1     0.25                        add	w3, w5, w7
+# CHECK-NEXT:  1      1     0.25                        add	wzr, w3, w5
+# CHECK-NEXT:  1      1     0.25                        add	w20, wzr, w4
+# CHECK-NEXT:  1      1     0.25                        add	w4, w6, wzr
+# CHECK-NEXT:  1      1     0.25                        add	w11, w13, w15
 # CHECK-NEXT:  1      2     0.50                        add	w9, w3, wzr, lsl #10
 # CHECK-NEXT:  1      2     0.50                        add	w17, w29, w20, lsl #31
 # CHECK-NEXT:  1      2     0.50                        add	w21, w22, w23, lsr #0
@@ -1419,11 +1419,11 @@ drps
 # CHECK-NEXT:  1      2     0.50                        add	w2, w3, w4, asr #0
 # CHECK-NEXT:  1      2     0.50                        add	w5, w6, w7, asr #21
 # CHECK-NEXT:  1      2     0.50                        add	w8, w9, w10, asr #31
-# CHECK-NEXT:  1      2     0.50                        add	x3, x5, x7
-# CHECK-NEXT:  1      2     0.50                        add	xzr, x3, x5
-# CHECK-NEXT:  1      2     0.50                        add	x20, xzr, x4
-# CHECK-NEXT:  1      2     0.50                        add	x4, x6, xzr
-# CHECK-NEXT:  1      2     0.50                        add	x11, x13, x15
+# CHECK-NEXT:  1      1     0.25                        add	x3, x5, x7
+# CHECK-NEXT:  1      1     0.25                        add	xzr, x3, x5
+# CHECK-NEXT:  1      1     0.25                        add	x20, xzr, x4
+# CHECK-NEXT:  1      1     0.25                        add	x4, x6, xzr
+# CHECK-NEXT:  1      1     0.25                        add	x11, x13, x15
 # CHECK-NEXT:  1      2     0.50                        add	x9, x3, xzr, lsl #10
 # CHECK-NEXT:  1      2     0.50                        add	x17, x29, x20, lsl #63
 # CHECK-NEXT:  1      2     0.50                        add	x21, x22, x23, lsr #0
@@ -1432,11 +1432,11 @@ drps
 # CHECK-NEXT:  1      2     0.50                        add	x2, x3, x4, asr #0
 # CHECK-NEXT:  1      2     0.50                        add	x5, x6, x7, asr #21
 # CHECK-NEXT:  1      2     0.50                        add	x8, x9, x10, asr #63
-# CHECK-NEXT:  1      2     0.50                        adds	w3, w5, w7
-# CHECK-NEXT:  1      2     0.50                        cmn	w3, w5
-# CHECK-NEXT:  1      2     0.50                        adds	w20, wzr, w4
-# CHECK-NEXT:  1      2     0.50                        adds	w4, w6, wzr
-# CHECK-NEXT:  1      2     0.50                        adds	w11, w13, w15
+# CHECK-NEXT:  1      1     0.25                        adds	w3, w5, w7
+# CHECK-NEXT:  1      1     0.25                        cmn	w3, w5
+# CHECK-NEXT:  1      1     0.25                        adds	w20, wzr, w4
+# CHECK-NEXT:  1      1     0.25                        adds	w4, w6, wzr
+# CHECK-NEXT:  1      1     0.25                        adds	w11, w13, w15
 # CHECK-NEXT:  1      2     0.50                        adds	w9, w3, wzr, lsl #10
 # CHECK-NEXT:  1      2     0.50                        adds	w17, w29, w20, lsl #31
 # CHECK-NEXT:  1      2     0.50                        adds	w21, w22, w23, lsr #0
@@ -1445,11 +1445,11 @@ drps
 # CHECK-NEXT:  1      2     0.50                        adds	w2, w3, w4, asr #0
 # CHECK-NEXT:  1      2     0.50                        adds	w5, w6, w7, asr #21
 # CHECK-NEXT:  1      2     0.50                        adds	w8, w9, w10, asr #31
-# CHECK-NEXT:  1      2     0.50                        adds	x3, x5, x7
-# CHECK-NEXT:  1      2     0.50                        cmn	x3, x5
-# CHECK-NEXT:  1      2     0.50                        adds	x20, xzr, x4
-# CHECK-NEXT:  1      2     0.50                        adds	x4, x6, xzr
-# CHECK-NEXT:  1      2     0.50                        adds	x11, x13, x15
+# CHECK-NEXT:  1      1     0.25                        adds	x3, x5, x7
+# CHECK-NEXT:  1      1     0.25                        cmn	x3, x5
+# CHECK-NEXT:  1      1     0.25                        adds	x20, xzr, x4
+# CHECK-NEXT:  1      1     0.25                        adds	x4, x6, xzr
+# CHECK-NEXT:  1      1     0.25                        adds	x11, x13, x15
 # CHECK-NEXT:  1      2     0.50                        adds	x9, x3, xzr, lsl #10
 # CHECK-NEXT:  1      2     0.50                        adds	x17, x29, x20, lsl #63
 # CHECK-NEXT:  1      2     0.50                        adds	x21, x22, x23, lsr #0
@@ -1458,10 +1458,10 @@ drps
 # CHECK-NEXT:  1      2     0.50                        adds	x2, x3, x4, asr #0
 # CHECK-NEXT:  1      2     0.50                        adds	x5, x6, x7, asr #21
 # CHECK-NEXT:  1      2     0.50                        adds	x8, x9, x10, asr #63
-# CHECK-NEXT:  1      2     0.50                        sub	w3, w5, w7
-# CHECK-NEXT:  1      2     0.50                        sub	wzr, w3, w5
-# CHECK-NEXT:  1      2     0.50                        sub	w4, w6, wzr
-# CHECK-NEXT:  1      2     0.50                        sub	w11, w13, w15
+# CHECK-NEXT:  1      1     0.25                        sub	w3, w5, w7
+# CHECK-NEXT:  1      1     0.25                        sub	wzr, w3, w5
+# CHECK-NEXT:  1      1     0.25                        sub	w4, w6, wzr
+# CHECK-NEXT:  1      1     0.25                        sub	w11, w13, w15
 # CHECK-NEXT:  1      2     0.50                        sub	w9, w3, wzr, lsl #10
 # CHECK-NEXT:  1      2     0.50                        sub	w17, w29, w20, lsl #31
 # CHECK-NEXT:  1      2     0.50                        sub	w21, w22, w23, lsr #0
@@ -1470,10 +1470,10 @@ drps
 # CHECK-NEXT:  1      2     0.50                        sub	w2, w3, w4, asr #0
 # CHECK-NEXT:  1      2     0.50                        sub	w5, w6, w7, asr #21
 # CHECK-NEXT:  1      2     0.50                        sub	w8, w9, w10, asr #31
-# CHECK-NEXT:  1      2     0.50                        sub	x3, x5, x7
-# CHECK-NEXT:  1      2     0.50                        sub	xzr, x3, x5
-# CHECK-NEXT:  1      2     0.50                        sub	x4, x6, xzr
-# CHECK-NEXT:  1      2     0.50                        sub	x11, x13, x15
+# CHECK-NEXT:  1      1     0.25                        sub	x3, x5, x7
+# CHECK-NEXT:  1      1     0.25                        sub	xzr, x3, x5
+# CHECK-NEXT:  1      1     0.25                        sub	x4, x6, xzr
+# CHECK-NEXT:  1      1     0.25                        sub	x11, x13, x15
 # CHECK-NEXT:  1      2     0.50                        sub	x9, x3, xzr, lsl #10
 # CHECK-NEXT:  1      2     0.50                        sub	x17, x29, x20, lsl #63
 # CHECK-NEXT:  1      2     0.50                        sub	x21, x22, x23, lsr #0
@@ -1482,10 +1482,10 @@ drps
 # CHECK-NEXT:  1      2     0.50                        sub	x2, x3, x4, asr #0
 # CHECK-NEXT:  1      2     0.50                        sub	x5, x6, x7, asr #21
 # CHECK-NEXT:  1      2     0.50                        sub	x8, x9, x10, asr #63
-# CHECK-NEXT:  1      2     0.50                        subs	w3, w5, w7
-# CHECK-NEXT:  1      2     0.50                        cmp	w3, w5
-# CHECK-NEXT:  1      2     0.50                        subs	w4, w6, wzr
-# CHECK-NEXT:  1      2     0.50                        subs	w11, w13, w15
+# CHECK-NEXT:  1      1     0.25                        subs	w3, w5, w7
+# CHECK-NEXT:  1      1     0.25                        cmp	w3, w5
+# CHECK-NEXT:  1      1     0.25                        subs	w4, w6, wzr
+# CHECK-NEXT:  1      1     0.25                        subs	w11, w13, w15
 # CHECK-NEXT:  1      2     0.50                        subs	w9, w3, wzr, lsl #10
 # CHECK-NEXT:  1      2     0.50                        subs	w17, w29, w20, lsl #31
 # CHECK-NEXT:  1      2     0.50                        subs	w21, w22, w23, lsr #0
@@ -1494,10 +1494,10 @@ drps
 # CHECK-NEXT:  1      2     0.50                        subs	w2, w3, w4, asr #0
 # CHECK-NEXT:  1      2     0.50                        subs	w5, w6, w7, asr #21
 # CHECK-NEXT:  1      2     0.50                        subs	w8, w9, w10, asr #31
-# CHECK-NEXT:  1      2     0.50                        subs	x3, x5, x7
-# CHECK-NEXT:  1      2     0.50                        cmp	x3, x5
-# CHECK-NEXT:  1      2     0.50                        subs	x4, x6, xzr
-# CHECK-NEXT:  1      2     0.50                        subs	x11, x13, x15
+# CHECK-NEXT:  1      1     0.25                        subs	x3, x5, x7
+# CHECK-NEXT:  1      1     0.25                        cmp	x3, x5
+# CHECK-NEXT:  1      1     0.25                        subs	x4, x6, xzr
+# CHECK-NEXT:  1      1     0.25                        subs	x11, x13, x15
 # CHECK-NEXT:  1      2     0.50                        subs	x9, x3, xzr, lsl #10
 # CHECK-NEXT:  1      2     0.50                        subs	x17, x29, x20, lsl #63
 # CHECK-NEXT:  1      2     0.50                        subs	x21, x22, x23, lsr #0
@@ -1506,9 +1506,9 @@ drps
 # CHECK-NEXT:  1      2     0.50                        subs	x2, x3, x4, asr #0
 # CHECK-NEXT:  1      2     0.50                        subs	x5, x6, x7, asr #21
 # CHECK-NEXT:  1      2     0.50                        subs	x8, x9, x10, asr #63
-# CHECK-NEXT:  1      2     0.50                        cmn	wzr, w4
-# CHECK-NEXT:  1      2     0.50                        cmn	w5, wzr
-# CHECK-NEXT:  1      2     0.50                        cmn	w6, w7
+# CHECK-NEXT:  1      1     0.25                        cmn	wzr, w4
+# CHECK-NEXT:  1      1     0.25                        cmn	w5, wzr
+# CHECK-NEXT:  1      1     0.25                        cmn	w6, w7
 # CHECK-NEXT:  1      2     0.50                        cmn	w8, w9, lsl #15
 # CHECK-NEXT:  1      2     0.50                        cmn	w10, w11, lsl #31
 # CHECK-NEXT:  1      2     0.50                        cmn	w12, w13, lsr #0
@@ -1517,10 +1517,10 @@ drps
 # CHECK-NEXT:  1      2     0.50                        cmn	w18, w19, asr #0
 # CHECK-NEXT:  1      2     0.50                        cmn	w20, w21, asr #22
 # CHECK-NEXT:  1      2     0.50                        cmn	w22, w23, asr #31
-# CHECK-NEXT:  1      2     0.50                        cmn	x0, x3
-# CHECK-NEXT:  1      2     0.50                        cmn	xzr, x4
-# CHECK-NEXT:  1      2     0.50                        cmn	x5, xzr
-# CHECK-NEXT:  1      2     0.50                        cmn	x6, x7
+# CHECK-NEXT:  1      1     0.25                        cmn	x0, x3
+# CHECK-NEXT:  1      1     0.25                        cmn	xzr, x4
+# CHECK-NEXT:  1      1     0.25                        cmn	x5, xzr
+# CHECK-NEXT:  1      1     0.25                        cmn	x6, x7
 # CHECK-NEXT:  1      2     0.50                        cmn	x8, x9, lsl #15
 # CHECK-NEXT:  1      2     0.50                        cmn	x10, x11, lsl #63
 # CHECK-NEXT:  1      2     0.50                        cmn	x12, x13, lsr #0
@@ -1529,10 +1529,10 @@ drps
 # CHECK-NEXT:  1      2     0.50                        cmn	x18, x19, asr #0
 # CHECK-NEXT:  1      2     0.50                        cmn	x20, x21, asr #55
 # CHECK-NEXT:  1      2     0.50                        cmn	x22, x23, asr #63
-# CHECK-NEXT:  1      2     0.50                        cmp	w0, w3
-# CHECK-NEXT:  1      2     0.50                        cmp	wzr, w4
-# CHECK-NEXT:  1      2     0.50                        cmp	w5, wzr
-# CHECK-NEXT:  1      2     0.50                        cmp	w6, w7
+# CHECK-NEXT:  1      1     0.25                        cmp	w0, w3
+# CHECK-NEXT:  1      1     0.25                        cmp	wzr, w4
+# CHECK-NEXT:  1      1     0.25                        cmp	w5, wzr
+# CHECK-NEXT:  1      1     0.25                        cmp	w6, w7
 # CHECK-NEXT:  1      2     0.50                        cmp	w8, w9, lsl #15
 # CHECK-NEXT:  1      2     0.50                        cmp	w10, w11, lsl #31
 # CHECK-NEXT:  1      2     0.50                        cmp	w12, w13, lsr #0
@@ -1540,10 +1540,10 @@ drps
 # CHECK-NEXT:  1      2     0.50                        cmp	w18, w19, asr #0
 # CHECK-NEXT:  1      2     0.50                        cmp	w20, w21, asr #22
 # CHECK-NEXT:  1      2     0.50                        cmp	w22, w23, asr #31
-# CHECK-NEXT:  1      2     0.50                        cmp	x0, x3
-# CHECK-NEXT:  1      2     0.50                        cmp	xzr, x4
-# CHECK-NEXT:  1      2     0.50                        cmp	x5, xzr
-# CHECK-NEXT:  1      2     0.50                        cmp	x6, x7
+# CHECK-NEXT:  1      1     0.25                        cmp	x0, x3
+# CHECK-NEXT:  1      1     0.25                        cmp	xzr, x4
+# CHECK-NEXT:  1      1     0.25                        cmp	x5, xzr
+# CHECK-NEXT:  1      1     0.25                        cmp	x6, x7
 # CHECK-NEXT:  1      2     0.50                        cmp	x8, x9, lsl #15
 # CHECK-NEXT:  1      2     0.50                        cmp	x10, x11, lsl #63
 # CHECK-NEXT:  1      2     0.50                        cmp	x12, x13, lsr #0
@@ -1552,8 +1552,8 @@ drps
 # CHECK-NEXT:  1      2     0.50                        cmp	x18, x19, asr #0
 # CHECK-NEXT:  1      2     0.50                        cmp	x20, x21, asr #55
 # CHECK-NEXT:  1      2     0.50                        cmp	x22, x23, asr #63
-# CHECK-NEXT:  1      2     0.50                        cmp	wzr, w0
-# CHECK-NEXT:  1      2     0.50                        cmp	xzr, x0
+# CHECK-NEXT:  1      1     0.25                        cmp	wzr, w0
+# CHECK-NEXT:  1      1     0.25                        cmp	xzr, x0
 # CHECK-NEXT:  1      1     0.25                        adc	w29, w27, w25
 # CHECK-NEXT:  1      1     0.25                        adc	wzr, w3, w4
 # CHECK-NEXT:  1      1     0.25                        adc	w9, wzr, w10
@@ -2484,34 +2484,34 @@ drps
 # CHECK-NEXT:  3      6     0.67    *                   ldnp	q23, q29, [x1, #-1024]
 # CHECK-NEXT:  1      1     0.25                        mov	w3, #983055
 # CHECK-NEXT:  1      1     0.25                        mov	x10, #-6148914691236517206
-# CHECK-NEXT:  1      2     0.50                        and	w12, w23, w21
-# CHECK-NEXT:  1      2     0.50                        and	w16, w15, w1, lsl #1
-# CHECK-NEXT:  1      2     0.50                        and	w9, w4, w10, lsl #31
-# CHECK-NEXT:  1      2     0.50                        and	w3, w30, w11
-# CHECK-NEXT:  1      2     0.50                        and	x3, x5, x7, lsl #63
-# CHECK-NEXT:  1      2     0.50                        and	x5, x14, x19, asr #4
-# CHECK-NEXT:  1      2     0.50                        and	w3, w17, w19, ror #31
-# CHECK-NEXT:  1      2     0.50                        and	w0, w2, wzr, lsr #17
-# CHECK-NEXT:  1      2     0.50                        and	w3, w30, w11, asr #2
-# CHECK-NEXT:  1      2     0.50                        and	xzr, x4, x26
-# CHECK-NEXT:  1      2     0.50                        and	w3, wzr, w20, ror #2
-# CHECK-NEXT:  1      2     0.50                        and	x7, x20, xzr, asr #63
-# CHECK-NEXT:  1      2     0.50                        bic	x13, x20, x14, lsl #47
-# CHECK-NEXT:  1      2     0.50                        bic	w2, w7, w9
-# CHECK-NEXT:  1      2     0.50                        orr	w2, w7, w0, asr #31
-# CHECK-NEXT:  1      2     0.50                        orr	x8, x9, x10, lsl #12
-# CHECK-NEXT:  1      2     0.50                        orn	x3, x5, x7, asr #2
-# CHECK-NEXT:  1      2     0.50                        orn	w2, w5, w29
+# CHECK-NEXT:  1      1     0.25                        and	w12, w23, w21
+# CHECK-NEXT:  1      1     0.25                        and	w16, w15, w1, lsl #1
+# CHECK-NEXT:  1      1     0.25                        and	w9, w4, w10, lsl #31
+# CHECK-NEXT:  1      1     0.25                        and	w3, w30, w11
+# CHECK-NEXT:  1      1     0.25                        and	x3, x5, x7, lsl #63
+# CHECK-NEXT:  1      1     0.25                        and	x5, x14, x19, asr #4
+# CHECK-NEXT:  1      1     0.25                        and	w3, w17, w19, ror #31
+# CHECK-NEXT:  1      1     0.25                        and	w0, w2, wzr, lsr #17
+# CHECK-NEXT:  1      1     0.25                        and	w3, w30, w11, asr #2
+# CHECK-NEXT:  1      1     0.25                        and	xzr, x4, x26
+# CHECK-NEXT:  1      1     0.25                        and	w3, wzr, w20, ror #2
+# CHECK-NEXT:  1      1     0.25                        and	x7, x20, xzr, asr #63
+# CHECK-NEXT:  1      1     0.25                        bic	x13, x20, x14, lsl #47
+# CHECK-NEXT:  1      1     0.25                        bic	w2, w7, w9
+# CHECK-NEXT:  1      1     0.25                        orr	w2, w7, w0, asr #31
+# CHECK-NEXT:  1      1     0.25                        orr	x8, x9, x10, lsl #12
+# CHECK-NEXT:  1      1     0.25                        orn	x3, x5, x7, asr #2
+# CHECK-NEXT:  1      1     0.25                        orn	w2, w5, w29
 # CHECK-NEXT:  1      2     0.50                        ands	w7, wzr, w9, lsl #1
 # CHECK-NEXT:  1      2     0.50                        ands	x3, x5, x20, ror #63
-# CHECK-NEXT:  1      2     0.50                        bics	w3, w5, w7
+# CHECK-NEXT:  1      1     0.25                        bics	w3, w5, w7
 # CHECK-NEXT:  1      2     0.50                        bics	x3, xzr, x3, lsl #1
 # CHECK-NEXT:  1      2     0.50                        tst	w3, w7, lsl #31
 # CHECK-NEXT:  1      2     0.50                        tst	x2, x20, asr #2
-# CHECK-NEXT:  1      2     0.50                        mov	x3, x6
-# CHECK-NEXT:  1      2     0.50                        mov	x3, xzr
-# CHECK-NEXT:  1      2     0.50                        mov	wzr, w2
-# CHECK-NEXT:  1      2     0.50                        mov	w3, w5
+# CHECK-NEXT:  1      1     0.25                        mov	x3, x6
+# CHECK-NEXT:  1      1     0.25                        mov	x3, xzr
+# CHECK-NEXT:  1      1     0.25                        mov	wzr, w2
+# CHECK-NEXT:  1      1     0.25                        mov	w3, w5
 # CHECK-NEXT:  1      1     0.25                        movz	w2, #0, lsl #16
 # CHECK-NEXT:  1      1     0.25                        mov	w2, #-1235
 # CHECK-NEXT:  1      1     0.25                        mov	x2, #5299989643264
@@ -2557,7 +2557,7 @@ drps
 
 # CHECK:      Resource pressure per iteration:
 # CHECK-NEXT: [0.0]  [0.1]  [1.0]  [1.1]  [2]    [3.0]  [3.1]  [4]    [5]    [6.0]  [6.1]  [7]    [8]
-# CHECK-NEXT: 11.00  11.00  33.00  33.00  87.33  151.33 151.33 536.00 270.00 143.50 143.50 169.50 85.50
+# CHECK-NEXT: 11.00  11.00  33.00  33.00  87.33  151.33 151.33 517.00 251.00 162.50 162.50 169.50 85.50
 
 # CHECK:      Resource pressure by instruction:
 # CHECK-NEXT: [0.0]  [0.1]  [1.0]  [1.1]  [2]    [3.0]  [3.1]  [4]    [5]    [6.0]  [6.1]  [7]    [8]    Instructions:
@@ -2590,11 +2590,11 @@ drps
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     mov	wsp, w20
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     mov	x11, sp
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     mov	w24, wsp
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	w3, w5, w7
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	wzr, w3, w5
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	w20, wzr, w4
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	w4, w6, wzr
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	w11, w13, w15
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     add	w3, w5, w7
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     add	wzr, w3, w5
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     add	w20, wzr, w4
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     add	w4, w6, wzr
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     add	w11, w13, w15
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	w9, w3, wzr, lsl #10
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	w17, w29, w20, lsl #31
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	w21, w22, w23, lsr #0
@@ -2603,11 +2603,11 @@ drps
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	w2, w3, w4, asr #0
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	w5, w6, w7, asr #21
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	w8, w9, w10, asr #31
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	x3, x5, x7
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	xzr, x3, x5
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	x20, xzr, x4
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	x4, x6, xzr
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	x11, x13, x15
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     add	x3, x5, x7
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     add	xzr, x3, x5
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     add	x20, xzr, x4
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     add	x4, x6, xzr
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     add	x11, x13, x15
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	x9, x3, xzr, lsl #10
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	x17, x29, x20, lsl #63
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	x21, x22, x23, lsr #0
@@ -2616,11 +2616,11 @@ drps
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	x2, x3, x4, asr #0
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	x5, x6, x7, asr #21
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     add	x8, x9, x10, asr #63
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	w3, w5, w7
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	w3, w5
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	w20, wzr, w4
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	w4, w6, wzr
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	w11, w13, w15
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     adds	w3, w5, w7
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmn	w3, w5
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     adds	w20, wzr, w4
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     adds	w4, w6, wzr
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     adds	w11, w13, w15
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	w9, w3, wzr, lsl #10
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	w17, w29, w20, lsl #31
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	w21, w22, w23, lsr #0
@@ -2629,11 +2629,11 @@ drps
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	w2, w3, w4, asr #0
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	w5, w6, w7, asr #21
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	w8, w9, w10, asr #31
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	x3, x5, x7
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	x3, x5
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	x20, xzr, x4
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	x4, x6, xzr
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	x11, x13, x15
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     adds	x3, x5, x7
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmn	x3, x5
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     adds	x20, xzr, x4
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     adds	x4, x6, xzr
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     adds	x11, x13, x15
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	x9, x3, xzr, lsl #10
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	x17, x29, x20, lsl #63
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	x21, x22, x23, lsr #0
@@ -2642,10 +2642,10 @@ drps
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	x2, x3, x4, asr #0
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	x5, x6, x7, asr #21
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     adds	x8, x9, x10, asr #63
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	w3, w5, w7
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	wzr, w3, w5
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	w4, w6, wzr
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	w11, w13, w15
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     sub	w3, w5, w7
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     sub	wzr, w3, w5
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     sub	w4, w6, wzr
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     sub	w11, w13, w15
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	w9, w3, wzr, lsl #10
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	w17, w29, w20, lsl #31
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	w21, w22, w23, lsr #0
@@ -2654,10 +2654,10 @@ drps
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	w2, w3, w4, asr #0
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	w5, w6, w7, asr #21
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	w8, w9, w10, asr #31
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	x3, x5, x7
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	xzr, x3, x5
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	x4, x6, xzr
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	x11, x13, x15
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     sub	x3, x5, x7
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     sub	xzr, x3, x5
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     sub	x4, x6, xzr
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     sub	x11, x13, x15
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	x9, x3, xzr, lsl #10
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	x17, x29, x20, lsl #63
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	x21, x22, x23, lsr #0
@@ -2666,10 +2666,10 @@ drps
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	x2, x3, x4, asr #0
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	x5, x6, x7, asr #21
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     sub	x8, x9, x10, asr #63
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     subs	w3, w5, w7
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	w3, w5
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     subs	w4, w6, wzr
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     subs	w11, w13, w15
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     subs	w3, w5, w7
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmp	w3, w5
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     subs	w4, w6, wzr
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     subs	w11, w13, w15
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     subs	w9, w3, wzr, lsl #10
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     subs	w17, w29, w20, lsl #31
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     subs	w21, w22, w23, lsr #0
@@ -2678,10 +2678,10 @@ drps
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     subs	w2, w3, w4, asr #0
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     subs	w5, w6, w7, asr #21
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     subs	w8, w9, w10, asr #31
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     subs	x3, x5, x7
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	x3, x5
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     subs	x4, x6, xzr
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     subs	x11, x13, x15
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     subs	x3, x5, x7
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmp	x3, x5
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     subs	x4, x6, xzr
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     subs	x11, x13, x15
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     subs	x9, x3, xzr, lsl #10
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     subs	x17, x29, x20, lsl #63
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     subs	x21, x22, x23, lsr #0
@@ -2690,9 +2690,9 @@ drps
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     subs	x2, x3, x4, asr #0
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     subs	x5, x6, x7, asr #21
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     subs	x8, x9, x10, asr #63
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	wzr, w4
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	w5, wzr
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	w6, w7
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmn	wzr, w4
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmn	w5, wzr
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmn	w6, w7
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	w8, w9, lsl #15
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	w10, w11, lsl #31
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	w12, w13, lsr #0
@@ -2701,10 +2701,10 @@ drps
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	w18, w19, asr #0
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	w20, w21, asr #22
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	w22, w23, asr #31
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	x0, x3
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	xzr, x4
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	x5, xzr
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	x6, x7
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmn	x0, x3
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmn	xzr, x4
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmn	x5, xzr
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmn	x6, x7
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	x8, x9, lsl #15
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	x10, x11, lsl #63
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	x12, x13, lsr #0
@@ -2713,10 +2713,10 @@ drps
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	x18, x19, asr #0
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	x20, x21, asr #55
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmn	x22, x23, asr #63
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	w0, w3
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	wzr, w4
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	w5, wzr
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	w6, w7
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmp	w0, w3
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmp	wzr, w4
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmp	w5, wzr
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmp	w6, w7
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	w8, w9, lsl #15
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	w10, w11, lsl #31
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	w12, w13, lsr #0
@@ -2724,10 +2724,10 @@ drps
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	w18, w19, asr #0
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	w20, w21, asr #22
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	w22, w23, asr #31
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	x0, x3
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	xzr, x4
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	x5, xzr
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	x6, x7
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmp	x0, x3
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmp	xzr, x4
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmp	x5, xzr
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmp	x6, x7
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	x8, x9, lsl #15
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	x10, x11, lsl #63
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	x12, x13, lsr #0
@@ -2736,8 +2736,8 @@ drps
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	x18, x19, asr #0
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	x20, x21, asr #55
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	x22, x23, asr #63
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	wzr, w0
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     cmp	xzr, x0
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmp	wzr, w0
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     cmp	xzr, x0
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     adc	w29, w27, w25
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     adc	wzr, w3, w4
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     adc	w9, wzr, w10
@@ -3668,34 +3668,34 @@ drps
 # CHECK-NEXT:  -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -     ldnp	q23, q29, [x1, #-1024]
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     mov	w3, #983055
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     mov	x10, #-6148914691236517206
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     and	w12, w23, w21
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     and	w16, w15, w1, lsl #1
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     and	w9, w4, w10, lsl #31
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     and	w3, w30, w11
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     and	x3, x5, x7, lsl #63
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     and	x5, x14, x19, asr #4
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     and	w3, w17, w19, ror #31
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     and	w0, w2, wzr, lsr #17
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     and	w3, w30, w11, asr #2
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     and	xzr, x4, x26
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     and	w3, wzr, w20, ror #2
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     and	x7, x20, xzr, asr #63
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     bic	x13, x20, x14, lsl #47
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     bic	w2, w7, w9
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     orr	w2, w7, w0, asr #31
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     orr	x8, x9, x10, lsl #12
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     orn	x3, x5, x7, asr #2
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     orn	w2, w5, w29
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     and	w12, w23, w21
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     and	w16, w15, w1, lsl #1
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     and	w9, w4, w10, lsl #31
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     and	w3, w30, w11
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     and	x3, x5, x7, lsl #63
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     and	x5, x14, x19, asr #4
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     and	w3, w17, w19, ror #31
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     and	w0, w2, wzr, lsr #17
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     and	w3, w30, w11, asr #2
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     and	xzr, x4, x26
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     and	w3, wzr, w20, ror #2
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     and	x7, x20, xzr, asr #63
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     bic	x13, x20, x14, lsl #47
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     bic	w2, w7, w9
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     orr	w2, w7, w0, asr #31
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     orr	x8, x9, x10, lsl #12
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     orn	x3, x5, x7, asr #2
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     orn	w2, w5, w29
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     ands	w7, wzr, w9, lsl #1
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     ands	x3, x5, x20, ror #63
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     bics	w3, w5, w7
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     bics	w3, w5, w7
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     bics	x3, xzr, x3, lsl #1
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     tst	w3, w7, lsl #31
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     tst	x2, x20, asr #2
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     mov	x3, x6
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     mov	x3, xzr
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     mov	wzr, w2
-# CHECK-NEXT:  -      -      -      -      -      -      -     0.50   0.50    -      -      -      -     mov	w3, w5
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     mov	x3, x6
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     mov	x3, xzr
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     mov	wzr, w2
+# CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     mov	w3, w5
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     movz	w2, #0, lsl #16
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     mov	w2, #-1235
 # CHECK-NEXT:  -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -     mov	x2, #5299989643264
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/independent-load-stores.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/independent-load-stores.s
index abe470551f419..dd7ac2734318f 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/independent-load-stores.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/independent-load-stores.s
@@ -16,18 +16,18 @@
 # ALL:           Iterations:        100
 # ALL-NEXT:      Instructions:      1000
 
-# NOALIAS-NEXT:  Total Cycles:      1009
-# YESALIAS-NEXT: Total Cycles:      7003
+# NOALIAS-NEXT:  Total Cycles:      1014
+# YESALIAS-NEXT: Total Cycles:      12003
 
 # ALL-NEXT:      Total uOps:        4000
 
 # ALL:           Dispatch Width:    6
 
-# NOALIAS-NEXT:  uOps Per Cycle:    3.96
+# NOALIAS-NEXT:  uOps Per Cycle:    3.94
 # NOALIAS-NEXT:  IPC:               0.99
 
-# YESALIAS-NEXT: uOps Per Cycle:    0.57
-# YESALIAS-NEXT: IPC:               0.14
+# YESALIAS-NEXT: uOps Per Cycle:    0.33
+# YESALIAS-NEXT: IPC:               0.08
 
 # ALL-NEXT:      Block RThroughput: 6.7
 
@@ -40,16 +40,16 @@
 # ALL-NEXT:      [6]: HasSideEffects (U)
 
 # ALL:           [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
-# ALL-NEXT:       4      7     0.50    *      *            addq	$44, 64(%r14)
-# ALL-NEXT:       4      7     0.50    *      *            addq	$44, 128(%r14)
-# ALL-NEXT:       4      7     0.50    *      *            addq	$44, 192(%r14)
-# ALL-NEXT:       4      7     0.50    *      *            addq	$44, 256(%r14)
-# ALL-NEXT:       4      7     0.50    *      *            addq	$44, 320(%r14)
-# ALL-NEXT:       4      7     0.50    *      *            addq	$44, 384(%r14)
-# ALL-NEXT:       4      7     0.50    *      *            addq	$44, 448(%r14)
-# ALL-NEXT:       4      7     0.50    *      *            addq	$44, 512(%r14)
-# ALL-NEXT:       4      7     0.50    *      *            addq	$44, 576(%r14)
-# ALL-NEXT:       4      7     0.50    *      *            addq	$44, 640(%r14)
+# ALL-NEXT:       4      12    0.50    *      *            addq	$44, 64(%r14)
+# ALL-NEXT:       4      12    0.50    *      *            addq	$44, 128(%r14)
+# ALL-NEXT:       4      12    0.50    *      *            addq	$44, 192(%r14)
+# ALL-NEXT:       4      12    0.50    *      *            addq	$44, 256(%r14)
+# ALL-NEXT:       4      12    0.50    *      *            addq	$44, 320(%r14)
+# ALL-NEXT:       4      12    0.50    *      *            addq	$44, 384(%r14)
+# ALL-NEXT:       4      12    0.50    *      *            addq	$44, 448(%r14)
+# ALL-NEXT:       4      12    0.50    *      *            addq	$44, 512(%r14)
+# ALL-NEXT:       4      12    0.50    *      *            addq	$44, 576(%r14)
+# ALL-NEXT:       4      12    0.50    *      *            addq	$44, 640(%r14)
 
 # ALL:           Resources:
 # ALL-NEXT:      [0]   - ADLPPort00
@@ -85,33 +85,30 @@
 
 # ALL:           Timeline view:
 
-# NOALIAS-NEXT:                      012345678
-# NOALIAS-NEXT:  Index     0123456789
+# NOALIAS-NEXT:                      0123456789
+# NOALIAS-NEXT:  Index     0123456789          0123
 
-# YESALIAS-NEXT:                     0123456789          0123456789          0123456789          012
+# YESALIAS-NEXT:                     0123456789          0123456789          0123456789          01234
 # YESALIAS-NEXT: Index     0123456789          0123456789          0123456789          0123456789
 
-# NOALIAS:       [0,0]     DeeeeeeeER.    .  .   addq	$44, 64(%r14)
-# NOALIAS-NEXT:  [0,1]     .DeeeeeeeER    .  .   addq	$44, 128(%r14)
-# NOALIAS-NEXT:  [0,2]     . DeeeeeeeER   .  .   addq	$44, 192(%r14)
-# NOALIAS-NEXT:  [0,3]     .  DeeeeeeeER  .  .   addq	$44, 256(%r14)
-# NOALIAS-NEXT:  [0,4]     .   DeeeeeeeER .  .   addq	$44, 320(%r14)
-# NOALIAS-NEXT:  [0,5]     .    DeeeeeeeER.  .   addq	$44, 384(%r14)
-# NOALIAS-NEXT:  [0,6]     .    .DeeeeeeeER  .   addq	$44, 448(%r14)
-# NOALIAS-NEXT:  [0,7]     .    . DeeeeeeeER .   addq	$44, 512(%r14)
-# NOALIAS-NEXT:  [0,8]     .    .  DeeeeeeeER.   addq	$44, 576(%r14)
-# NOALIAS-NEXT:  [0,9]     .    .   DeeeeeeeER   addq	$44, 640(%r14)
-
-# YESALIAS:      [0,0]     DeeeeeeeER.    .    .    .    .    .    .    .    .    .    .    .    . .   addq	$44, 64(%r14)
-# YESALIAS-NEXT: [0,1]     .D======eeeeeeeER   .    .    .    .    .    .    .    .    .    .    . .   addq	$44, 128(%r14)
-# YESALIAS-NEXT: [0,2]     . D============eeeeeeeER .    .    .    .    .    .    .    .    .    . .   addq	$44, 192(%r14)
-# YESALIAS-NEXT: [0,3]     .  D==================eeeeeeeER    .    .    .    .    .    .    .    . .   addq	$44, 256(%r14)
-# YESALIAS-NEXT: [0,4]     .   D========================eeeeeeeER  .    .    .    .    .    .    . .   addq	$44, 320(%r14)
-# YESALIAS-NEXT: [0,5]     .    D==============================eeeeeeeER.    .    .    .    .    . .   addq	$44, 384(%r14)
-# YESALIAS-NEXT: [0,6]     .    .D====================================eeeeeeeER   .    .    .    . .   addq	$44, 448(%r14)
-# YESALIAS-NEXT: [0,7]     .    . D==========================================eeeeeeeER .    .    . .   addq	$44, 512(%r14)
-# YESALIAS-NEXT: [0,8]     .    .  D================================================eeeeeeeER    . .   addq	$44, 576(%r14)
-# YESALIAS-NEXT: [0,9]     .    .   D======================================================eeeeeeeER   addq	$44, 640(%r14)
+# NOALIAS:       [0,0]     DeeeeeeeeeeeeER.    .  .   addq	$44, 64(%r14)
+# NOALIAS-NEXT:  [0,1]     .DeeeeeeeeeeeeER    .  .   addq	$44, 128(%r14)
+# NOALIAS-NEXT:  [0,2]     . DeeeeeeeeeeeeER   .  .   addq	$44, 192(%r14)
+# NOALIAS-NEXT:  [0,3]     .  DeeeeeeeeeeeeER  .  .   addq	$44, 256(%r14)
+# NOALIAS-NEXT:  [0,4]     .   DeeeeeeeeeeeeER .  .   addq	$44, 320(%r14)
+# NOALIAS-NEXT:  [0,5]     .    DeeeeeeeeeeeeER.  .   addq	$44, 384(%r14)
+# NOALIAS-NEXT:  [0,6]     .    .DeeeeeeeeeeeeER  .   addq	$44, 448(%r14)
+# NOALIAS-NEXT:  [0,7]     .    . DeeeeeeeeeeeeER .   addq	$44, 512(%r14)
+# NOALIAS-NEXT:  [0,8]     .    .  DeeeeeeeeeeeeER.   addq	$44, 576(%r14)
+# NOALIAS-NEXT:  [0,9]     .    .   DeeeeeeeeeeeeER   addq	$44, 640(%r14)
+
+# YESALIAS:      [0,0]     DeeeeeeeeeeeeER.    .    .    .    .    .    .    .    .    .    .    .   .   addq	$44, 64(%r14)
+# YESALIAS-NEXT: [0,1]     .D===========eeeeeeeeeeeeER   .    .    .    .    .    .    .    .    .   .   addq	$44, 128(%r14)
+# YESALIAS-NEXT: [0,2]     . D======================eeeeeeeeeeeeER .    .    .    .    .    .    .   .   addq	$44, 192(%r14)
+# YESALIAS-NEXT: [0,3]     .  D=================================eeeeeeeeeeeeER    .    .    .    .   .   addq	$44, 256(%r14)
+# YESALIAS-NEXT: [0,4]     .   D============================================eeeeeeeeeeeeER  .    .   .   addq	$44, 320(%r14)
+# YESALIAS-NEXT: [0,5]     .    D=======================================================eeeeeeeeeeeeER   addq	$44, 384(%r14)
+# YESALIAS-NEXT: Truncated display due to cycle limit
 
 # ALL:           Average Wait times (based on the timeline view):
 # ALL-NEXT:      [0]: Executions
@@ -133,13 +130,13 @@
 # NOALIAS-NEXT:  9.     1     1.0    1.0    0.0       addq	$44, 640(%r14)
 # NOALIAS-NEXT:         1     1.0    1.0    0.0       <total>
 
-# YESALIAS-NEXT: 1.     1     7.0    0.0    0.0       addq	$44, 128(%r14)
-# YESALIAS-NEXT: 2.     1     13.0   0.0    0.0       addq	$44, 192(%r14)
-# YESALIAS-NEXT: 3.     1     19.0   0.0    0.0       addq	$44, 256(%r14)
-# YESALIAS-NEXT: 4.     1     25.0   0.0    0.0       addq	$44, 320(%r14)
-# YESALIAS-NEXT: 5.     1     31.0   0.0    0.0       addq	$44, 384(%r14)
-# YESALIAS-NEXT: 6.     1     37.0   0.0    0.0       addq	$44, 448(%r14)
-# YESALIAS-NEXT: 7.     1     43.0   0.0    0.0       addq	$44, 512(%r14)
-# YESALIAS-NEXT: 8.     1     49.0   0.0    0.0       addq	$44, 576(%r14)
-# YESALIAS-NEXT: 9.     1     55.0   0.0    0.0       addq	$44, 640(%r14)
-# YESALIAS-NEXT:        1     28.0   0.1    0.0       <total>
+# YESALIAS-NEXT: 1.     1     12.0   0.0    0.0       addq	$44, 128(%r14)
+# YESALIAS-NEXT: 2.     1     23.0   0.0    0.0       addq	$44, 192(%r14)
+# YESALIAS-NEXT: 3.     1     34.0   0.0    0.0       addq	$44, 256(%r14)
+# YESALIAS-NEXT: 4.     1     45.0   0.0    0.0       addq	$44, 320(%r14)
+# YESALIAS-NEXT: 5.     1     56.0   0.0    0.0       addq	$44, 384(%r14)
+# YESALIAS-NEXT: 6.     1     67.0   0.0    0.0       addq	$44, 448(%r14)
+# YESALIAS-NEXT: 7.     1     78.0   0.0    0.0       addq	$44, 512(%r14)
+# YESALIAS-NEXT: 8.     1     89.0   0.0    0.0       addq	$44, 576(%r14)
+# YESALIAS-NEXT: 9.     1     100.0  0.0    0.0       addq	$44, 640(%r14)
+# YESALIAS-NEXT:        1     50.5   0.1    0.0       <total>
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-aes.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-aes.s
index 670c802c8e93d..9384488f06781 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-aes.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-aes.s
@@ -28,16 +28,16 @@ aeskeygenassist $22, (%rax), %xmm2
 # CHECK-NEXT: [6]: HasSideEffects (U)
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
-# CHECK-NEXT:  1      3     0.50                        aesdec	%xmm0, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   aesdec	(%rax), %xmm2
-# CHECK-NEXT:  1      3     0.50                        aesdeclast	%xmm0, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   aesdeclast	(%rax), %xmm2
-# CHECK-NEXT:  1      3     0.50                        aesenc	%xmm0, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   aesenc	(%rax), %xmm2
-# CHECK-NEXT:  1      3     0.50                        aesenclast	%xmm0, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   aesenclast	(%rax), %xmm2
-# CHECK-NEXT:  2      6     1.00                        aesimc	%xmm0, %xmm2
-# CHECK-NEXT:  3      14    1.00    *                   aesimc	(%rax), %xmm2
+# CHECK-NEXT:  1      5     0.50                        aesdec	%xmm0, %xmm2
+# CHECK-NEXT:  2      12    0.50    *                   aesdec	(%rax), %xmm2
+# CHECK-NEXT:  1      5     0.50                        aesdeclast	%xmm0, %xmm2
+# CHECK-NEXT:  2      12    0.50    *                   aesdeclast	(%rax), %xmm2
+# CHECK-NEXT:  1      5     0.50                        aesenc	%xmm0, %xmm2
+# CHECK-NEXT:  2      12    0.50    *                   aesenc	(%rax), %xmm2
+# CHECK-NEXT:  1      5     0.50                        aesenclast	%xmm0, %xmm2
+# CHECK-NEXT:  2      12    0.50    *                   aesenclast	(%rax), %xmm2
+# CHECK-NEXT:  2      8     1.00                        aesimc	%xmm0, %xmm2
+# CHECK-NEXT:  3      15    1.00    *                   aesimc	(%rax), %xmm2
 # CHECK-NEXT:  14     7     4.00                        aeskeygenassist	$22, %xmm0, %xmm2
 # CHECK-NEXT:  14     12    4.00    *                   aeskeygenassist	$22, (%rax), %xmm2
 
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-avx1.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-avx1.s
index fb4e19494adf6..dc8ad8e46a777 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-avx1.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-avx1.s
@@ -1030,145 +1030,145 @@ vzeroupper
 # CHECK-NEXT: [6]: HasSideEffects (U)
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
-# CHECK-NEXT:  1      2     0.50                        vaddpd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      9     0.50    *                   vaddpd	(%rax), %xmm1, %xmm2
-# CHECK-NEXT:  1      2     0.50                        vaddpd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      10    0.50    *                   vaddpd	(%rax), %ymm1, %ymm2
-# CHECK-NEXT:  1      2     0.50                        vaddps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      9     0.50    *                   vaddps	(%rax), %xmm1, %xmm2
-# CHECK-NEXT:  1      2     0.50                        vaddps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      10    0.50    *                   vaddps	(%rax), %ymm1, %ymm2
-# CHECK-NEXT:  1      2     0.50                        vaddsd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      9     0.50    *                   vaddsd	(%rax), %xmm1, %xmm2
-# CHECK-NEXT:  1      2     0.50                        vaddss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      9     0.50    *                   vaddss	(%rax), %xmm1, %xmm2
-# CHECK-NEXT:  1      2     0.50                        vaddsubpd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      9     0.50    *                   vaddsubpd	(%rax), %xmm1, %xmm2
-# CHECK-NEXT:  1      2     0.50                        vaddsubpd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      10    0.50    *                   vaddsubpd	(%rax), %ymm1, %ymm2
-# CHECK-NEXT:  1      2     0.50                        vaddsubps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      9     0.50    *                   vaddsubps	(%rax), %xmm1, %xmm2
-# CHECK-NEXT:  1      2     0.50                        vaddsubps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      10    0.50    *                   vaddsubps	(%rax), %ymm1, %ymm2
-# CHECK-NEXT:  1      3     0.50                        vaesdec	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vaesdec	(%rax), %xmm1, %xmm2
-# CHECK-NEXT:  1      3     0.50                        vaesdeclast	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vaesdeclast	(%rax), %xmm1, %xmm2
-# CHECK-NEXT:  1      3     0.50                        vaesenc	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vaesenc	(%rax), %xmm1, %xmm2
-# CHECK-NEXT:  1      3     0.50                        vaesenclast	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vaesenclast	(%rax), %xmm1, %xmm2
-# CHECK-NEXT:  2      6     1.00                        vaesimc	%xmm0, %xmm2
-# CHECK-NEXT:  3      14    1.00    *                   vaesimc	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        vaddpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  2      10    0.50    *                   vaddpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vaddpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  2      11    0.50    *                   vaddpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vaddps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  2      10    0.50    *                   vaddps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vaddps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  2      11    0.50    *                   vaddps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vaddsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  2      10    0.50    *                   vaddsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vaddss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  2      10    0.50    *                   vaddss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vaddsubpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  2      10    0.50    *                   vaddsubpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vaddsubpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  2      11    0.50    *                   vaddsubpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vaddsubps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  2      10    0.50    *                   vaddsubps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vaddsubps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  2      11    0.50    *                   vaddsubps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      5     0.50                        vaesdec	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  2      12    0.50    *                   vaesdec	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      5     0.50                        vaesdeclast	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  2      12    0.50    *                   vaesdeclast	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      5     0.50                        vaesenc	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  2      12    0.50    *                   vaesenc	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      5     0.50                        vaesenclast	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  2      12    0.50    *                   vaesenclast	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     1.00                        vaesimc	%xmm0, %xmm2
+# CHECK-NEXT:  3      15    1.00    *                   vaesimc	(%rax), %xmm2
 # CHECK-NEXT:  14     7     4.00                        vaeskeygenassist	$22, %xmm0, %xmm2
 # CHECK-NEXT:  14     12    4.00    *                   vaeskeygenassist	$22, (%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        vandnpd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vandnpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vandnpd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vandnpd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vandnpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vandnpd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vandnps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vandnps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vandnps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vandnps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vandnps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vandnps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vandpd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vandpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vandpd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vandpd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vandpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vandpd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vandps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vandps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vandps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vandps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vandps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vandps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vblendpd	$11, %xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vblendpd	$11, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vblendpd	$11, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vblendpd	$11, %ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vblendpd	$11, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vblendpd	$11, (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vblendps	$11, %xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vblendps	$11, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vblendps	$11, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vblendps	$11, %ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vblendps	$11, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vblendps	$11, (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  3      3     1.00                        vblendvpd	%xmm3, %xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  4      8     1.00    *                   vblendvpd	%xmm3, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  4      9     1.00    *                   vblendvpd	%xmm3, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  3      3     1.00                        vblendvpd	%ymm3, %ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  4      9     1.00    *                   vblendvpd	%ymm3, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  4      10    1.00    *                   vblendvpd	%ymm3, (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  3      3     1.00                        vblendvps	%xmm3, %xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  4      8     1.00    *                   vblendvps	%xmm3, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  4      9     1.00    *                   vblendvps	%xmm3, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  3      3     1.00                        vblendvps	%ymm3, %ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  4      9     1.00    *                   vblendvps	%ymm3, (%rax), %ymm1, %ymm2
-# CHECK-NEXT:  1      7     0.33    *                   vbroadcastf128	(%rax), %ymm2
-# CHECK-NEXT:  1      7     0.33    *                   vbroadcastsd	(%rax), %ymm2
-# CHECK-NEXT:  1      6     0.33    *                   vbroadcastss	(%rax), %xmm2
-# CHECK-NEXT:  1      7     0.33    *                   vbroadcastss	(%rax), %ymm2
+# CHECK-NEXT:  4      10    1.00    *                   vblendvps	%ymm3, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.33    *                   vbroadcastf128	(%rax), %ymm2
+# CHECK-NEXT:  1      8     0.33    *                   vbroadcastsd	(%rax), %ymm2
+# CHECK-NEXT:  1      7     0.33    *                   vbroadcastss	(%rax), %xmm2
+# CHECK-NEXT:  1      8     0.33    *                   vbroadcastss	(%rax), %ymm2
 # CHECK-NEXT:  1      4     0.50                        vcmpeqpd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vcmpeqpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vcmpeqpd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vcmpeqpd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vcmpeqpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vcmpeqpd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vcmpeqps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vcmpeqps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vcmpeqps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vcmpeqps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vcmpeqps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vcmpeqps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vcmpeqsd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vcmpeqsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vcmpeqsd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vcmpeqss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vcmpeqss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vcmpeqss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      3     1.00                        vcomisd	%xmm0, %xmm1
-# CHECK-NEXT:  2      9     1.00    *                   vcomisd	(%rax), %xmm1
+# CHECK-NEXT:  2      8     1.00    *                   vcomisd	(%rax), %xmm1
 # CHECK-NEXT:  1      3     1.00                        vcomiss	%xmm0, %xmm1
-# CHECK-NEXT:  2      9     1.00    *                   vcomiss	(%rax), %xmm1
+# CHECK-NEXT:  2      8     1.00    *                   vcomiss	(%rax), %xmm1
 # CHECK-NEXT:  2      5     1.00                        vcvtdq2pd	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vcvtdq2pd	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vcvtdq2pd	(%rax), %xmm2
 # CHECK-NEXT:  2      7     1.00                        vcvtdq2pd	%xmm0, %ymm2
 # CHECK-NEXT:  2      12    0.50    *                   vcvtdq2pd	(%rax), %ymm2
 # CHECK-NEXT:  1      4     0.50                        vcvtdq2ps	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vcvtdq2ps	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vcvtdq2ps	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        vcvtdq2ps	%ymm0, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vcvtdq2ps	(%rax), %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vcvtdq2ps	(%rax), %ymm2
 # CHECK-NEXT:  2      5     1.00                        vcvtpd2dq	%xmm0, %xmm2
-# CHECK-NEXT:  3      11    1.00    *                   vcvtpd2dqx	(%rax), %xmm2
+# CHECK-NEXT:  3      12    1.00    *                   vcvtpd2dqx	(%rax), %xmm2
 # CHECK-NEXT:  2      7     1.00                        vcvtpd2dq	%ymm0, %xmm2
-# CHECK-NEXT:  3      14    1.00    *                   vcvtpd2dqy	(%rax), %xmm2
+# CHECK-NEXT:  3      15    1.00    *                   vcvtpd2dqy	(%rax), %xmm2
 # CHECK-NEXT:  2      5     1.00                        vcvtpd2ps	%xmm0, %xmm2
-# CHECK-NEXT:  3      11    1.00    *                   vcvtpd2psx	(%rax), %xmm2
+# CHECK-NEXT:  3      12    1.00    *                   vcvtpd2psx	(%rax), %xmm2
 # CHECK-NEXT:  2      7     1.00                        vcvtpd2ps	%ymm0, %xmm2
-# CHECK-NEXT:  3      14    1.00    *                   vcvtpd2psy	(%rax), %xmm2
+# CHECK-NEXT:  3      15    1.00    *                   vcvtpd2psy	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        vcvtps2dq	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vcvtps2dq	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vcvtps2dq	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        vcvtps2dq	%ymm0, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vcvtps2dq	(%rax), %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vcvtps2dq	(%rax), %ymm2
 # CHECK-NEXT:  2      5     1.00                        vcvtps2pd	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vcvtps2pd	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vcvtps2pd	(%rax), %xmm2
 # CHECK-NEXT:  2      7     1.00                        vcvtps2pd	%xmm0, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vcvtps2pd	(%rax), %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vcvtps2pd	(%rax), %ymm2
 # CHECK-NEXT:  2      7     1.00                        vcvtsd2si	%xmm0, %ecx
 # CHECK-NEXT:  2      7     1.00                        vcvtsd2si	%xmm0, %rcx
-# CHECK-NEXT:  3      12    1.00    *                   vcvtsd2si	(%rax), %ecx
+# CHECK-NEXT:  3      26    1.00    *                   vcvtsd2si	(%rax), %ecx
 # CHECK-NEXT:  3      12    1.00    *                   vcvtsd2si	(%rax), %rcx
 # CHECK-NEXT:  2      5     1.00                        vcvtsd2ss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  3      11    1.00    *                   vcvtsd2ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  3      12    1.00    *                   vcvtsd2ss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  2      7     1.00                        vcvtsi2sd	%ecx, %xmm0, %xmm2
 # CHECK-NEXT:  2      7     1.00                        vcvtsi2sd	%rcx, %xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vcvtsi2sdl	(%rax), %xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vcvtsi2sdq	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vcvtsi2sdl	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vcvtsi2sdq	(%rax), %xmm0, %xmm2
 # CHECK-NEXT:  2      7     1.00                        vcvtsi2ss	%ecx, %xmm0, %xmm2
 # CHECK-NEXT:  3      8     2.00                        vcvtsi2ss	%rcx, %xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vcvtsi2ssl	(%rax), %xmm0, %xmm2
-# CHECK-NEXT:  3      11    1.00    *                   vcvtsi2ssq	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vcvtsi2ssl	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  3      12    1.00    *                   vcvtsi2ssq	(%rax), %xmm0, %xmm2
 # CHECK-NEXT:  2      5     1.00                        vcvtss2sd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vcvtss2sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vcvtss2sd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  2      7     1.00                        vcvtss2si	%xmm0, %ecx
 # CHECK-NEXT:  3      8     1.00                        vcvtss2si	%xmm0, %rcx
 # CHECK-NEXT:  3      12    1.00    *                   vcvtss2si	(%rax), %ecx
 # CHECK-NEXT:  3      12    1.00    *                   vcvtss2si	(%rax), %rcx
 # CHECK-NEXT:  2      5     1.00                        vcvttpd2dq	%xmm0, %xmm2
-# CHECK-NEXT:  3      11    1.00    *                   vcvttpd2dqx	(%rax), %xmm2
+# CHECK-NEXT:  3      12    1.00    *                   vcvttpd2dqx	(%rax), %xmm2
 # CHECK-NEXT:  2      7     1.00                        vcvttpd2dq	%ymm0, %xmm2
-# CHECK-NEXT:  3      14    1.00    *                   vcvttpd2dqy	(%rax), %xmm2
+# CHECK-NEXT:  3      15    1.00    *                   vcvttpd2dqy	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        vcvttps2dq	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vcvttps2dq	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vcvttps2dq	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        vcvttps2dq	%ymm0, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vcvttps2dq	(%rax), %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vcvttps2dq	(%rax), %ymm2
 # CHECK-NEXT:  2      7     1.00                        vcvttsd2si	%xmm0, %ecx
 # CHECK-NEXT:  2      7     1.00                        vcvttsd2si	%xmm0, %rcx
-# CHECK-NEXT:  3      12    1.00    *                   vcvttsd2si	(%rax), %ecx
+# CHECK-NEXT:  3      26    1.00    *                   vcvttsd2si	(%rax), %ecx
 # CHECK-NEXT:  3      12    1.00    *                   vcvttsd2si	(%rax), %rcx
 # CHECK-NEXT:  2      7     1.00                        vcvttss2si	%xmm0, %ecx
 # CHECK-NEXT:  3      8     1.00                        vcvttss2si	%xmm0, %rcx
@@ -1179,35 +1179,35 @@ vzeroupper
 # CHECK-NEXT:  1      14    1.00                        vdivpd	%ymm0, %ymm1, %ymm2
 # CHECK-NEXT:  2      21    1.00    *                   vdivpd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      11    1.00                        vdivps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      17    1.00    *                   vdivps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      18    1.00    *                   vdivps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      11    1.00                        vdivps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      18    1.00    *                   vdivps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      19    1.00    *                   vdivps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      14    1.00                        vdivsd	%xmm0, %xmm1, %xmm2
 # CHECK-NEXT:  2      20    1.00    *                   vdivsd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      11    1.00                        vdivss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      17    1.00    *                   vdivss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      18    1.00    *                   vdivss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  3      9     1.00                        vdppd	$22, %xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  4      15    1.00    *                   vdppd	$22, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  4      16    1.00    *                   vdppd	$22, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  6      14    1.67                        vdpps	$22, %xmm0, %xmm1, %xmm2
 # CHECK-NEXT:  7      21    1.67    *                   vdpps	$22, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  6      14    1.67                        vdpps	$22, %ymm0, %ymm1, %ymm2
 # CHECK-NEXT:  7      22    1.67    *                   vdpps	$22, (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      3     1.00                        vextractf128	$1, %ymm0, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            vextractf128	$1, %ymm0, (%rax)
+# CHECK-NEXT:  2      12    0.50           *            vextractf128	$1, %ymm0, (%rax)
 # CHECK-NEXT:  2      4     1.00                        vextractps	$1, %xmm0, %ecx
-# CHECK-NEXT:  3      2     1.00           *            vextractps	$1, %xmm0, (%rax)
+# CHECK-NEXT:  3      12    1.00           *            vextractps	$1, %xmm0, (%rax)
 # CHECK-NEXT:  3      5     2.00                        vhaddpd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  4      11    2.00    *                   vhaddpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  4      12    2.00    *                   vhaddpd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  3      5     2.00                        vhaddpd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  4      12    2.00    *                   vhaddpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  4      13    2.00    *                   vhaddpd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  3      5     2.00                        vhaddps	%xmm0, %xmm1, %xmm2
 # CHECK-NEXT:  4      12    2.00    *                   vhaddps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  3      5     2.00                        vhaddps	%ymm0, %ymm1, %ymm2
 # CHECK-NEXT:  4      13    2.00    *                   vhaddps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  3      5     2.00                        vhsubpd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  4      11    2.00    *                   vhsubpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  4      12    2.00    *                   vhsubpd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  3      5     2.00                        vhsubpd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  4      12    2.00    *                   vhsubpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  4      13    2.00    *                   vhsubpd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  3      5     2.00                        vhsubps	%xmm0, %xmm1, %xmm2
 # CHECK-NEXT:  4      12    2.00    *                   vhsubps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  3      5     2.00                        vhsubps	%ymm0, %ymm1, %ymm2
@@ -1215,85 +1215,85 @@ vzeroupper
 # CHECK-NEXT:  1      3     1.00                        vinsertf128	$1, %xmm0, %ymm1, %ymm2
 # CHECK-NEXT:  2      9     0.33    *                   vinsertf128	$1, (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     1.00                        vinsertps	$1, %xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     1.00    *                   vinsertps	$1, (%rax), %xmm1, %xmm2
-# CHECK-NEXT:  1      6     0.33    *                   vlddqu	(%rax), %xmm2
-# CHECK-NEXT:  1      7     0.33    *                   vlddqu	(%rax), %ymm2
+# CHECK-NEXT:  2      8     1.00    *                   vinsertps	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      7     0.33    *                   vlddqu	(%rax), %xmm2
+# CHECK-NEXT:  1      8     0.33    *                   vlddqu	(%rax), %ymm2
 # CHECK-NEXT:  3      7     1.00    *      *      U     vldmxcsr	(%rax)
 # CHECK-NEXT:  2      1     1.00    *      *      U     vmaskmovdqu	%xmm0, %xmm1
-# CHECK-NEXT:  2      7     0.33    *                   vmaskmovpd	(%rax), %xmm0, %xmm2
-# CHECK-NEXT:  2      8     0.33    *                   vmaskmovpd	(%rax), %ymm0, %ymm2
-# CHECK-NEXT:  3      3     1.00    *      *            vmaskmovpd	%xmm0, %xmm1, (%rax)
-# CHECK-NEXT:  3      3     1.00    *      *            vmaskmovpd	%ymm0, %ymm1, (%rax)
-# CHECK-NEXT:  2      7     0.33    *                   vmaskmovps	(%rax), %xmm0, %xmm2
-# CHECK-NEXT:  2      8     0.33    *                   vmaskmovps	(%rax), %ymm0, %ymm2
-# CHECK-NEXT:  3      3     1.00    *      *            vmaskmovps	%xmm0, %xmm1, (%rax)
-# CHECK-NEXT:  3      3     1.00    *      *            vmaskmovps	%ymm0, %ymm1, (%rax)
+# CHECK-NEXT:  2      8     0.33    *                   vmaskmovpd	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  2      9     0.33    *                   vmaskmovpd	(%rax), %ymm0, %ymm2
+# CHECK-NEXT:  3      14    1.00    *      *            vmaskmovpd	%xmm0, %xmm1, (%rax)
+# CHECK-NEXT:  3      14    1.00    *      *            vmaskmovpd	%ymm0, %ymm1, (%rax)
+# CHECK-NEXT:  2      8     0.33    *                   vmaskmovps	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  2      9     0.33    *                   vmaskmovps	(%rax), %ymm0, %ymm2
+# CHECK-NEXT:  3      14    1.00    *      *            vmaskmovps	%xmm0, %xmm1, (%rax)
+# CHECK-NEXT:  3      14    1.00    *      *            vmaskmovps	%ymm0, %ymm1, (%rax)
 # CHECK-NEXT:  1      4     0.50                        vmaxpd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vmaxpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vmaxpd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vmaxpd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vmaxpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vmaxpd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vmaxps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vmaxps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vmaxps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vmaxps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vmaxps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vmaxps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vmaxsd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vmaxsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vmaxsd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vmaxss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vmaxss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vmaxss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vminpd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vminpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vminpd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vminpd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vminpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vminpd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vminps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vminps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vminps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vminps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vminps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vminps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vminsd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vminsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vminsd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vminss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vminss	(%rax), %xmm1, %xmm2
-# CHECK-NEXT:  1      0     0.33                        vmovapd	%xmm0, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            vmovapd	%xmm0, (%rax)
-# CHECK-NEXT:  1      6     0.33    *                   vmovapd	(%rax), %xmm2
-# CHECK-NEXT:  1      0     0.33                        vmovapd	%ymm0, %ymm2
-# CHECK-NEXT:  2      1     0.50           *            vmovapd	%ymm0, (%rax)
-# CHECK-NEXT:  1      7     0.33    *                   vmovapd	(%rax), %ymm2
-# CHECK-NEXT:  1      0     0.33                        vmovaps	%xmm0, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            vmovaps	%xmm0, (%rax)
-# CHECK-NEXT:  1      6     0.33    *                   vmovaps	(%rax), %xmm2
-# CHECK-NEXT:  1      0     0.33                        vmovaps	%ymm0, %ymm2
-# CHECK-NEXT:  2      1     0.50           *            vmovaps	%ymm0, (%rax)
-# CHECK-NEXT:  1      7     0.33    *                   vmovaps	(%rax), %ymm2
-# CHECK-NEXT:  1      1     1.00                        vmovd	%eax, %xmm2
-# CHECK-NEXT:  1      6     0.33    *                   vmovd	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vminss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  0      1     0.00                        vmovapd	%xmm0, %xmm2
+# CHECK-NEXT:  2      12    0.50           *            vmovapd	%xmm0, (%rax)
+# CHECK-NEXT:  1      7     0.33    *                   vmovapd	(%rax), %xmm2
+# CHECK-NEXT:  0      1     0.00                        vmovapd	%ymm0, %ymm2
+# CHECK-NEXT:  2      12    0.50           *            vmovapd	%ymm0, (%rax)
+# CHECK-NEXT:  1      8     0.33    *                   vmovapd	(%rax), %ymm2
+# CHECK-NEXT:  0      1     0.00                        vmovaps	%xmm0, %xmm2
+# CHECK-NEXT:  2      12    0.50           *            vmovaps	%xmm0, (%rax)
+# CHECK-NEXT:  1      7     0.33    *                   vmovaps	(%rax), %xmm2
+# CHECK-NEXT:  0      1     0.00                        vmovaps	%ymm0, %ymm2
+# CHECK-NEXT:  2      12    0.50           *            vmovaps	%ymm0, (%rax)
+# CHECK-NEXT:  1      8     0.33    *                   vmovaps	(%rax), %ymm2
+# CHECK-NEXT:  1      3     1.00                        vmovd	%eax, %xmm2
+# CHECK-NEXT:  1      7     0.33    *                   vmovd	(%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        vmovd	%xmm0, %ecx
-# CHECK-NEXT:  2      1     0.50           *            vmovd	%xmm0, (%rax)
+# CHECK-NEXT:  2      12    0.50           *            vmovd	%xmm0, (%rax)
 # CHECK-NEXT:  1      1     1.00                        vmovddup	%xmm0, %xmm2
 # CHECK-NEXT:  1      7     0.33    *                   vmovddup	(%rax), %xmm2
 # CHECK-NEXT:  1      1     1.00                        vmovddup	%ymm0, %ymm2
-# CHECK-NEXT:  1      7     0.33    *                   vmovddup	(%rax), %ymm2
-# CHECK-NEXT:  1      0     0.33                        vmovdqa	%xmm0, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            vmovdqa	%xmm0, (%rax)
-# CHECK-NEXT:  1      6     0.33    *                   vmovdqa	(%rax), %xmm2
-# CHECK-NEXT:  1      0     0.33                        vmovdqa	%ymm0, %ymm2
-# CHECK-NEXT:  2      1     0.50           *            vmovdqa	%ymm0, (%rax)
-# CHECK-NEXT:  1      7     0.33    *                   vmovdqa	(%rax), %ymm2
-# CHECK-NEXT:  1      0     0.33                        vmovdqu	%xmm0, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            vmovdqu	%xmm0, (%rax)
-# CHECK-NEXT:  1      6     0.33    *                   vmovdqu	(%rax), %xmm2
-# CHECK-NEXT:  1      0     0.33                        vmovdqu	%ymm0, %ymm2
-# CHECK-NEXT:  2      1     0.50           *            vmovdqu	%ymm0, (%rax)
-# CHECK-NEXT:  1      7     0.33    *                   vmovdqu	(%rax), %ymm2
+# CHECK-NEXT:  1      8     0.33    *                   vmovddup	(%rax), %ymm2
+# CHECK-NEXT:  0      1     0.00                        vmovdqa	%xmm0, %xmm2
+# CHECK-NEXT:  2      12    0.50           *            vmovdqa	%xmm0, (%rax)
+# CHECK-NEXT:  1      7     0.33    *                   vmovdqa	(%rax), %xmm2
+# CHECK-NEXT:  0      1     0.00                        vmovdqa	%ymm0, %ymm2
+# CHECK-NEXT:  2      12    0.50           *            vmovdqa	%ymm0, (%rax)
+# CHECK-NEXT:  1      8     0.33    *                   vmovdqa	(%rax), %ymm2
+# CHECK-NEXT:  0      1     0.00                        vmovdqu	%xmm0, %xmm2
+# CHECK-NEXT:  2      12    0.50           *            vmovdqu	%xmm0, (%rax)
+# CHECK-NEXT:  1      7     0.33    *                   vmovdqu	(%rax), %xmm2
+# CHECK-NEXT:  0      1     0.00                        vmovdqu	%ymm0, %ymm2
+# CHECK-NEXT:  2      12    0.50           *            vmovdqu	%ymm0, (%rax)
+# CHECK-NEXT:  1      8     0.33    *                   vmovdqu	(%rax), %ymm2
 # CHECK-NEXT:  1      1     1.00                        vmovhlps	%xmm0, %xmm1, %xmm2
 # CHECK-NEXT:  1      1     1.00                        vmovlhps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            vmovhpd	%xmm0, (%rax)
-# CHECK-NEXT:  2      7     1.00    *                   vmovhpd	(%rax), %xmm1, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            vmovhps	%xmm0, (%rax)
-# CHECK-NEXT:  2      7     1.00    *                   vmovhps	(%rax), %xmm1, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            vmovlpd	%xmm0, (%rax)
-# CHECK-NEXT:  2      7     0.50    *                   vmovlpd	(%rax), %xmm1, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            vmovlps	%xmm0, (%rax)
-# CHECK-NEXT:  2      7     0.50    *                   vmovlps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      12    0.50           *            vmovhpd	%xmm0, (%rax)
+# CHECK-NEXT:  2      8     1.00    *                   vmovhpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      12    0.50           *            vmovhps	%xmm0, (%rax)
+# CHECK-NEXT:  2      8     1.00    *                   vmovhps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      12    0.50           *            vmovlpd	%xmm0, (%rax)
+# CHECK-NEXT:  2      8     0.50    *                   vmovlpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      12    0.50           *            vmovlps	%xmm0, (%rax)
+# CHECK-NEXT:  2      8     0.50    *                   vmovlps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      3     1.00                        vmovmskpd	%xmm0, %ecx
 # CHECK-NEXT:  1      5     1.00                        vmovmskpd	%ymm0, %ecx
 # CHECK-NEXT:  1      3     1.00                        vmovmskps	%xmm0, %ecx
@@ -1307,382 +1307,382 @@ vzeroupper
 # CHECK-NEXT:  2      470   0.50           *            vmovntps	%xmm0, (%rax)
 # CHECK-NEXT:  2      494   0.50           *            vmovntps	%ymm0, (%rax)
 # CHECK-NEXT:  1      1     0.33                        vmovq	%xmm0, %xmm2
-# CHECK-NEXT:  1      1     1.00                        vmovq	%rax, %xmm2
-# CHECK-NEXT:  1      6     0.33    *                   vmovq	(%rax), %xmm2
+# CHECK-NEXT:  1      3     1.00                        vmovq	%rax, %xmm2
+# CHECK-NEXT:  1      7     0.33    *                   vmovq	(%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        vmovq	%xmm0, %rcx
-# CHECK-NEXT:  2      1     0.50           *            vmovq	%xmm0, (%rax)
+# CHECK-NEXT:  2      12    0.50           *            vmovq	%xmm0, (%rax)
 # CHECK-NEXT:  1      1     0.33                        vmovsd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            vmovsd	%xmm0, (%rax)
-# CHECK-NEXT:  1      6     0.33    *                   vmovsd	(%rax), %xmm2
+# CHECK-NEXT:  2      12    0.50           *            vmovsd	%xmm0, (%rax)
+# CHECK-NEXT:  1      7     0.33    *                   vmovsd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vmovshdup	%xmm0, %xmm2
 # CHECK-NEXT:  1      7     0.33    *                   vmovshdup	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vmovshdup	%ymm0, %ymm2
-# CHECK-NEXT:  1      7     0.33    *                   vmovshdup	(%rax), %ymm2
+# CHECK-NEXT:  1      8     0.33    *                   vmovshdup	(%rax), %ymm2
 # CHECK-NEXT:  1      1     0.50                        vmovsldup	%xmm0, %xmm2
 # CHECK-NEXT:  1      7     0.33    *                   vmovsldup	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vmovsldup	%ymm0, %ymm2
-# CHECK-NEXT:  1      7     0.33    *                   vmovsldup	(%rax), %ymm2
+# CHECK-NEXT:  1      8     0.33    *                   vmovsldup	(%rax), %ymm2
 # CHECK-NEXT:  1      1     0.33                        vmovss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            vmovss	%xmm0, (%rax)
-# CHECK-NEXT:  1      6     0.33    *                   vmovss	(%rax), %xmm2
-# CHECK-NEXT:  1      0     0.33                        vmovupd	%xmm0, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            vmovupd	%xmm0, (%rax)
-# CHECK-NEXT:  1      6     0.33    *                   vmovupd	(%rax), %xmm2
-# CHECK-NEXT:  1      0     0.33                        vmovupd	%ymm0, %ymm2
-# CHECK-NEXT:  2      1     0.50           *            vmovupd	%ymm0, (%rax)
-# CHECK-NEXT:  1      7     0.33    *                   vmovupd	(%rax), %ymm2
-# CHECK-NEXT:  1      0     0.33                        vmovups	%xmm0, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            vmovups	%xmm0, (%rax)
-# CHECK-NEXT:  1      6     0.33    *                   vmovups	(%rax), %xmm2
-# CHECK-NEXT:  1      0     0.33                        vmovups	%ymm0, %ymm2
-# CHECK-NEXT:  2      1     0.50           *            vmovups	%ymm0, (%rax)
-# CHECK-NEXT:  1      7     0.33    *                   vmovups	(%rax), %ymm2
+# CHECK-NEXT:  2      12    0.50           *            vmovss	%xmm0, (%rax)
+# CHECK-NEXT:  1      7     0.33    *                   vmovss	(%rax), %xmm2
+# CHECK-NEXT:  0      1     0.00                        vmovupd	%xmm0, %xmm2
+# CHECK-NEXT:  2      12    0.50           *            vmovupd	%xmm0, (%rax)
+# CHECK-NEXT:  1      7     0.33    *                   vmovupd	(%rax), %xmm2
+# CHECK-NEXT:  0      1     0.00                        vmovupd	%ymm0, %ymm2
+# CHECK-NEXT:  2      12    0.50           *            vmovupd	%ymm0, (%rax)
+# CHECK-NEXT:  1      8     0.33    *                   vmovupd	(%rax), %ymm2
+# CHECK-NEXT:  0      1     0.00                        vmovups	%xmm0, %xmm2
+# CHECK-NEXT:  2      12    0.50           *            vmovups	%xmm0, (%rax)
+# CHECK-NEXT:  1      7     0.33    *                   vmovups	(%rax), %xmm2
+# CHECK-NEXT:  0      1     0.00                        vmovups	%ymm0, %ymm2
+# CHECK-NEXT:  2      12    0.50           *            vmovups	%ymm0, (%rax)
+# CHECK-NEXT:  1      8     0.33    *                   vmovups	(%rax), %ymm2
 # CHECK-NEXT:  2      4     1.00                        vmpsadbw	$1, %xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  3      10    1.00    *                   vmpsadbw	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  3      11    1.00    *                   vmpsadbw	$1, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vmulpd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vmulpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vmulpd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vmulpd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vmulpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vmulpd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vmulps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vmulps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vmulps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vmulps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vmulps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vmulps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vmulsd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vmulsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vmulsd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vmulss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vmulss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vmulss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vorpd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vorpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vorpd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vorpd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vorpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vorpd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vorps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vorps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vorps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vorps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vorps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vorps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpabsb	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpabsb	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpabsb	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpabsd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpabsd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpabsd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpabsw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpabsw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpabsw	(%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        vpackssdw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      9     1.00    *                   vpackssdw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      10    1.00    *                   vpackssdw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      3     1.00                        vpacksswb	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      9     1.00    *                   vpacksswb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      10    1.00    *                   vpacksswb	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      3     1.00                        vpackusdw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      9     1.00    *                   vpackusdw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      10    1.00    *                   vpackusdw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      3     1.00                        vpackuswb	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      9     1.00    *                   vpackuswb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      10    1.00    *                   vpackuswb	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vpaddb	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vpaddb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vpaddb	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vpaddd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vpaddd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vpaddd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vpaddq	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vpaddq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vpaddq	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpaddsb	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpaddsb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpaddsb	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpaddsw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpaddsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpaddsw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpaddusb	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpaddusb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpaddusb	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpaddusw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpaddusw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpaddusw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vpaddw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vpaddw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vpaddw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     1.00                        vpalignr	$1, %xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     1.00    *                   vpalignr	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     1.00    *                   vpalignr	$1, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vpand	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vpand	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vpand	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vpandn	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vpandn	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vpandn	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpavgb	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpavgb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpavgb	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpavgw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpavgw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpavgw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  3      3     1.00                        vpblendvb	%xmm3, %xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  4      8     1.00    *                   vpblendvb	%xmm3, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  4      9     1.00    *                   vpblendvb	%xmm3, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpblendw	$11, %xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpblendw	$11, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpblendw	$11, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      3     1.00                        vpclmulqdq	$11, %xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      9     1.00    *                   vpclmulqdq	$11, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      10    1.00    *                   vpclmulqdq	$11, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpcmpeqb	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpcmpeqb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpcmpeqb	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpcmpeqd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpcmpeqd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpcmpeqd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpcmpeqq	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpcmpeqq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpcmpeqq	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpcmpeqw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpcmpeqw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpcmpeqw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  8      16    3.00                        vpcmpestri	$1, %xmm0, %xmm2
 # CHECK-NEXT:  8      31    3.00    *                   vpcmpestri	$1, (%rax), %xmm2
 # CHECK-NEXT:  9      16    3.00                        vpcmpestrm	$1, %xmm0, %xmm2
 # CHECK-NEXT:  9      17    3.00    *                   vpcmpestrm	$1, (%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpcmpgtb	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpcmpgtb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpcmpgtb	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpcmpgtd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpcmpgtd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpcmpgtd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      3     1.00                        vpcmpgtq	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      9     1.00    *                   vpcmpgtq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      10    1.00    *                   vpcmpgtq	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpcmpgtw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpcmpgtw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpcmpgtw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  3      11    3.00                        vpcmpistri	$1, %xmm0, %xmm2
 # CHECK-NEXT:  4      31    3.00    *                   vpcmpistri	$1, (%rax), %xmm2
 # CHECK-NEXT:  3      11    3.00                        vpcmpistrm	$1, %xmm0, %xmm2
 # CHECK-NEXT:  4      16    3.00    *                   vpcmpistrm	$1, (%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        vperm2f128	$1, %ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vperm2f128	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vperm2f128	$1, (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     1.00                        vpermilpd	$1, %xmm0, %xmm2
-# CHECK-NEXT:  2      7     1.00    *                   vpermilpd	$1, (%rax), %xmm2
+# CHECK-NEXT:  2      8     1.00    *                   vpermilpd	$1, (%rax), %xmm2
 # CHECK-NEXT:  1      1     1.00                        vpermilpd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     1.00    *                   vpermilpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     1.00    *                   vpermilpd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     1.00                        vpermilpd	$1, %ymm0, %ymm2
-# CHECK-NEXT:  2      8     1.00    *                   vpermilpd	$1, (%rax), %ymm2
+# CHECK-NEXT:  2      9     1.00    *                   vpermilpd	$1, (%rax), %ymm2
 # CHECK-NEXT:  1      1     1.00                        vpermilpd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     1.00    *                   vpermilpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     1.00    *                   vpermilpd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     1.00                        vpermilps	$1, %xmm0, %xmm2
-# CHECK-NEXT:  2      7     1.00    *                   vpermilps	$1, (%rax), %xmm2
+# CHECK-NEXT:  2      8     1.00    *                   vpermilps	$1, (%rax), %xmm2
 # CHECK-NEXT:  1      1     1.00                        vpermilps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     1.00    *                   vpermilps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     1.00    *                   vpermilps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     1.00                        vpermilps	$1, %ymm0, %ymm2
-# CHECK-NEXT:  2      8     1.00    *                   vpermilps	$1, (%rax), %ymm2
+# CHECK-NEXT:  2      9     1.00    *                   vpermilps	$1, (%rax), %ymm2
 # CHECK-NEXT:  1      1     1.00                        vpermilps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     1.00    *                   vpermilps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     1.00    *                   vpermilps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  2      4     1.00                        vpextrb	$1, %xmm0, %ecx
-# CHECK-NEXT:  3      2     0.50           *            vpextrb	$1, %xmm0, (%rax)
+# CHECK-NEXT:  3      19    0.50           *            vpextrb	$1, %xmm0, (%rax)
 # CHECK-NEXT:  2      4     1.00                        vpextrd	$1, %xmm0, %ecx
-# CHECK-NEXT:  3      2     0.50           *            vpextrd	$1, %xmm0, (%rax)
+# CHECK-NEXT:  3      12    0.50           *            vpextrd	$1, %xmm0, (%rax)
 # CHECK-NEXT:  2      4     1.00                        vpextrq	$1, %xmm0, %rcx
-# CHECK-NEXT:  3      2     0.50           *            vpextrq	$1, %xmm0, (%rax)
+# CHECK-NEXT:  3      12    0.50           *            vpextrq	$1, %xmm0, (%rax)
 # CHECK-NEXT:  2      4     1.00                        vpextrw	$1, %xmm0, %ecx
-# CHECK-NEXT:  3      2     0.50           *            vpextrw	$1, %xmm0, (%rax)
+# CHECK-NEXT:  3      19    0.50           *            vpextrw	$1, %xmm0, (%rax)
 # CHECK-NEXT:  3      2     1.00                        vphaddd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  4      8     1.00    *                   vphaddd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  4      9     1.00    *                   vphaddd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  3      2     1.00                        vphaddsw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  4      8     1.00    *                   vphaddsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  4      9     1.00    *                   vphaddsw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  3      2     1.00                        vphaddw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  4      8     1.00    *                   vphaddw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  4      9     1.00    *                   vphaddw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     1.00                        vphminposuw	%xmm0, %xmm2
 # CHECK-NEXT:  2      11    1.00    *                   vphminposuw	(%rax), %xmm2
 # CHECK-NEXT:  3      2     1.00                        vphsubd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  4      8     1.00    *                   vphsubd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  4      9     1.00    *                   vphsubd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  3      2     1.00                        vphsubsw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  4      8     1.00    *                   vphsubsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  4      9     1.00    *                   vphsubsw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  3      2     1.00                        vphsubw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  4      8     1.00    *                   vphsubw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  4      9     1.00    *                   vphsubw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  2      4     1.00                        vpinsrb	$1, %eax, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpinsrb	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpinsrb	$1, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  2      4     1.00                        vpinsrd	$1, %eax, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpinsrd	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpinsrd	$1, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  2      4     1.00                        vpinsrq	$1, %rax, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpinsrq	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpinsrq	$1, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  2      4     1.00                        vpinsrw	$1, %eax, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpinsrw	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpinsrw	$1, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      5     0.50                        vpmaddubsw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vpmaddubsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   vpmaddubsw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      5     0.50                        vpmaddwd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vpmaddwd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   vpmaddwd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpmaxsb	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpmaxsb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpmaxsb	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpmaxsd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpmaxsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpmaxsd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpmaxsw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpmaxsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpmaxsw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpmaxub	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpmaxub	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpmaxub	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpmaxud	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpmaxud	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpmaxud	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpmaxuw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpmaxuw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpmaxuw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpminsb	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpminsb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpminsb	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpminsd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpminsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpminsd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpminsw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpminsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpminsw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpminub	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpminub	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpminub	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpminud	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpminud	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpminud	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpminuw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpminuw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpminuw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      3     1.00                        vpmovmskb	%xmm0, %ecx
 # CHECK-NEXT:  1      1     0.50                        vpmovsxbd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpmovsxbd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpmovsxbd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpmovsxbq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpmovsxbq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpmovsxbq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpmovsxbw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpmovsxbw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpmovsxbw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpmovsxdq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpmovsxdq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpmovsxdq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpmovsxwd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpmovsxwd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpmovsxwd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpmovsxwq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpmovsxwq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpmovsxwq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpmovzxbd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpmovzxbd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpmovzxbd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpmovzxbq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpmovzxbq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpmovzxbq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpmovzxbw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpmovzxbw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpmovzxbw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpmovzxdq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpmovzxdq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpmovzxdq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpmovzxwd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpmovzxwd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpmovzxwd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpmovzxwq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpmovzxwq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpmovzxwq	(%rax), %xmm2
 # CHECK-NEXT:  1      5     0.50                        vpmuldq	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vpmuldq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   vpmuldq	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      5     0.50                        vpmulhrsw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vpmulhrsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   vpmulhrsw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      5     0.50                        vpmulhuw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vpmulhuw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   vpmulhuw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      5     0.50                        vpmulhw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vpmulhw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   vpmulhw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  2      10    1.00                        vpmulld	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  3      16    1.00    *                   vpmulld	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  3      18    1.00    *                   vpmulld	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      5     0.50                        vpmullw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vpmullw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   vpmullw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      5     0.50                        vpmuludq	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vpmuludq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   vpmuludq	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vpor	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vpor	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vpor	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      3     1.00                        vpsadbw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      9     1.00    *                   vpsadbw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      10    1.00    *                   vpsadbw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpshufb	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpshufb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpshufb	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpshufd	$1, %xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpshufd	$1, (%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpshufd	$1, (%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpshufhw	$1, %xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpshufhw	$1, (%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpshufhw	$1, (%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpshuflw	$1, %xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpshuflw	$1, (%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpshuflw	$1, (%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsignb	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsignb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsignb	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsignd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsignd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsignd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsignw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsignw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsignw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpslld	$1, %xmm0, %xmm2
 # CHECK-NEXT:  2      2     0.67                        vpslld	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpslld	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpslld	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpslldq	$1, %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsllq	$1, %xmm0, %xmm2
 # CHECK-NEXT:  2      2     0.67                        vpsllq	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsllq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsllq	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsllw	$1, %xmm0, %xmm2
 # CHECK-NEXT:  2      2     0.67                        vpsllw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsllw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsllw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsrad	$1, %xmm0, %xmm2
 # CHECK-NEXT:  2      2     0.67                        vpsrad	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsrad	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsrad	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsraw	$1, %xmm0, %xmm2
 # CHECK-NEXT:  2      2     0.67                        vpsraw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsraw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsraw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsrld	$1, %xmm0, %xmm2
 # CHECK-NEXT:  2      2     0.67                        vpsrld	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsrld	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsrld	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsrldq	$1, %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsrlq	$1, %xmm0, %xmm2
 # CHECK-NEXT:  2      2     0.67                        vpsrlq	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsrlq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsrlq	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsrlw	$1, %xmm0, %xmm2
 # CHECK-NEXT:  2      2     0.67                        vpsrlw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsrlw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsrlw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vpsubb	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vpsubb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vpsubb	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vpsubd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vpsubd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vpsubd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vpsubq	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vpsubq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vpsubq	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsubsb	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsubsb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsubsb	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsubsw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsubsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsubsw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsubusb	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsubusb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsubusb	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsubusw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsubusw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsubusw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vpsubw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vpsubw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vpsubw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  2      4     1.00                        vptest	%xmm0, %xmm1
 # CHECK-NEXT:  3      9     1.00    *                   vptest	(%rax), %xmm1
 # CHECK-NEXT:  2      6     1.00                        vptest	%ymm0, %ymm1
 # CHECK-NEXT:  3      12    1.00    *                   vptest	(%rax), %ymm1
 # CHECK-NEXT:  1      1     0.50                        vpunpckhbw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpunpckhbw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpunpckhbw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpunpckhdq	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpunpckhdq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpunpckhdq	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpunpckhqdq	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpunpckhqdq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpunpckhqdq	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpunpckhwd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpunpckhwd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpunpckhwd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpunpcklbw	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpunpcklbw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpunpcklbw	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpunpckldq	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpunpckldq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpunpckldq	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpunpcklqdq	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpunpcklqdq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpunpcklqdq	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpunpcklwd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpunpcklwd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpunpcklwd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vpxor	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vpxor	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vpxor	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     1.00                        vrcpps	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    1.00    *                   vrcpps	(%rax), %xmm2
+# CHECK-NEXT:  2      11    1.00    *                   vrcpps	(%rax), %xmm2
 # CHECK-NEXT:  1      4     1.00                        vrcpps	%ymm0, %ymm2
-# CHECK-NEXT:  2      11    1.00    *                   vrcpps	(%rax), %ymm2
+# CHECK-NEXT:  2      12    1.00    *                   vrcpps	(%rax), %ymm2
 # CHECK-NEXT:  1      4     1.00                        vrcpss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    1.00    *                   vrcpss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    1.00    *                   vrcpss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  2      8     1.00                        vroundpd	$1, %xmm0, %xmm2
-# CHECK-NEXT:  3      14    1.00    *                   vroundpd	$1, (%rax), %xmm2
+# CHECK-NEXT:  3      15    1.00    *                   vroundpd	$1, (%rax), %xmm2
 # CHECK-NEXT:  2      8     1.00                        vroundpd	$1, %ymm0, %ymm2
-# CHECK-NEXT:  3      15    1.00    *                   vroundpd	$1, (%rax), %ymm2
+# CHECK-NEXT:  3      16    1.00    *                   vroundpd	$1, (%rax), %ymm2
 # CHECK-NEXT:  2      8     1.00                        vroundps	$1, %xmm0, %xmm2
-# CHECK-NEXT:  3      14    1.00    *                   vroundps	$1, (%rax), %xmm2
+# CHECK-NEXT:  3      15    1.00    *                   vroundps	$1, (%rax), %xmm2
 # CHECK-NEXT:  2      8     1.00                        vroundps	$1, %ymm0, %ymm2
-# CHECK-NEXT:  3      15    1.00    *                   vroundps	$1, (%rax), %ymm2
+# CHECK-NEXT:  3      16    1.00    *                   vroundps	$1, (%rax), %ymm2
 # CHECK-NEXT:  2      8     1.00                        vroundsd	$1, %xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  3      14    1.00    *                   vroundsd	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  3      15    1.00    *                   vroundsd	$1, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  2      8     1.00                        vroundss	$1, %xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  3      14    1.00    *                   vroundss	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  3      15    1.00    *                   vroundss	$1, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     1.00                        vrsqrtps	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    1.00    *                   vrsqrtps	(%rax), %xmm2
+# CHECK-NEXT:  2      11    1.00    *                   vrsqrtps	(%rax), %xmm2
 # CHECK-NEXT:  1      4     1.00                        vrsqrtps	%ymm0, %ymm2
-# CHECK-NEXT:  2      11    1.00    *                   vrsqrtps	(%rax), %ymm2
+# CHECK-NEXT:  2      12    1.00    *                   vrsqrtps	(%rax), %ymm2
 # CHECK-NEXT:  1      4     1.00                        vrsqrtss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    1.00    *                   vrsqrtss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    1.00    *                   vrsqrtss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vshufpd	$1, %xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vshufpd	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vshufpd	$1, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vshufpd	$1, %ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vshufpd	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vshufpd	$1, (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vshufps	$1, %xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vshufps	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vshufps	$1, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vshufps	$1, %ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vshufps	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vshufps	$1, (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      18    1.00                        vsqrtpd	%xmm0, %xmm2
 # CHECK-NEXT:  2      24    1.00    *                   vsqrtpd	(%rax), %xmm2
 # CHECK-NEXT:  1      18    1.00                        vsqrtpd	%ymm0, %ymm2
 # CHECK-NEXT:  2      25    1.00    *                   vsqrtpd	(%rax), %ymm2
 # CHECK-NEXT:  1      12    1.00                        vsqrtps	%xmm0, %xmm2
-# CHECK-NEXT:  2      18    1.00    *                   vsqrtps	(%rax), %xmm2
+# CHECK-NEXT:  2      19    1.00    *                   vsqrtps	(%rax), %xmm2
 # CHECK-NEXT:  1      12    1.00                        vsqrtps	%ymm0, %ymm2
-# CHECK-NEXT:  2      19    1.00    *                   vsqrtps	(%rax), %ymm2
+# CHECK-NEXT:  2      20    1.00    *                   vsqrtps	(%rax), %ymm2
 # CHECK-NEXT:  1      18    1.00                        vsqrtsd	%xmm0, %xmm1, %xmm2
 # CHECK-NEXT:  2      24    1.00    *                   vsqrtsd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      12    1.00                        vsqrtss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      18    1.00    *                   vsqrtss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      19    1.00    *                   vsqrtss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  4      12    1.00           *      U     vstmxcsr	(%rax)
-# CHECK-NEXT:  1      2     0.50                        vsubpd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      9     0.50    *                   vsubpd	(%rax), %xmm1, %xmm2
-# CHECK-NEXT:  1      2     0.50                        vsubpd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      10    0.50    *                   vsubpd	(%rax), %ymm1, %ymm2
-# CHECK-NEXT:  1      2     0.50                        vsubps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      9     0.50    *                   vsubps	(%rax), %xmm1, %xmm2
-# CHECK-NEXT:  1      2     0.50                        vsubps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      10    0.50    *                   vsubps	(%rax), %ymm1, %ymm2
-# CHECK-NEXT:  1      2     0.50                        vsubsd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      9     0.50    *                   vsubsd	(%rax), %xmm1, %xmm2
-# CHECK-NEXT:  1      2     0.50                        vsubss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      9     0.50    *                   vsubss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vsubpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  2      10    0.50    *                   vsubpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vsubpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  2      11    0.50    *                   vsubpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vsubps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  2      10    0.50    *                   vsubps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vsubps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  2      11    0.50    *                   vsubps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vsubsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  2      10    0.50    *                   vsubsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vsubss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  2      10    0.50    *                   vsubss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      3     1.00                        vtestpd	%xmm0, %xmm1
 # CHECK-NEXT:  2      8     1.00    *                   vtestpd	(%rax), %xmm1
 # CHECK-NEXT:  1      5     1.00                        vtestpd	%ymm0, %ymm1
@@ -1692,33 +1692,33 @@ vzeroupper
 # CHECK-NEXT:  1      5     1.00                        vtestps	%ymm0, %ymm1
 # CHECK-NEXT:  2      11    1.00    *                   vtestps	(%rax), %ymm1
 # CHECK-NEXT:  1      3     1.00                        vucomisd	%xmm0, %xmm1
-# CHECK-NEXT:  2      9     1.00    *                   vucomisd	(%rax), %xmm1
+# CHECK-NEXT:  2      8     1.00    *                   vucomisd	(%rax), %xmm1
 # CHECK-NEXT:  1      3     1.00                        vucomiss	%xmm0, %xmm1
-# CHECK-NEXT:  2      9     1.00    *                   vucomiss	(%rax), %xmm1
+# CHECK-NEXT:  2      8     1.00    *                   vucomiss	(%rax), %xmm1
 # CHECK-NEXT:  1      1     1.00                        vunpckhpd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     1.00    *                   vunpckhpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     1.00    *                   vunpckhpd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     1.00                        vunpckhpd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     1.00    *                   vunpckhpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     1.00    *                   vunpckhpd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     1.00                        vunpckhps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     1.00    *                   vunpckhps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     1.00    *                   vunpckhps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     1.00                        vunpckhps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     1.00    *                   vunpckhps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     1.00    *                   vunpckhps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     1.00                        vunpcklpd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     1.00    *                   vunpcklpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     1.00    *                   vunpcklpd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     1.00                        vunpcklpd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     1.00    *                   vunpcklpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     1.00    *                   vunpcklpd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     1.00                        vunpcklps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     1.00    *                   vunpcklps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     1.00    *                   vunpcklps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     1.00                        vunpcklps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     1.00    *                   vunpcklps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     1.00    *                   vunpcklps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vxorpd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vxorpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vxorpd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vxorpd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vxorpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vxorpd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vxorps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vxorps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vxorps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vxorps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vxorps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vxorps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  10     16    3.00                  U     vzeroall
 # CHECK-NEXT:  0      0     0.00                  U     vzeroupper
 
@@ -1739,7 +1739,7 @@ vzeroupper
 
 # CHECK:      Resource pressure per iteration:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]
-# CHECK-NEXT: 314.40 279.73 107.33 107.33 19.50  281.73 9.40   18.83  18.50  18.50  0.73   107.00  -
+# CHECK-NEXT: 310.90 275.73 107.33 107.33 19.50  277.73 8.90   18.83  18.50  18.50  0.73   107.00  -
 
 # CHECK:      Resource pressure by instruction:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]   Instructions:
@@ -1965,16 +1965,16 @@ vzeroupper
 # CHECK-NEXT: 0.50   0.50   0.33   0.33    -      -      -      -      -      -      -     0.33    -     vminsd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vminss	%xmm0, %xmm1, %xmm2
 # CHECK-NEXT: 0.50   0.50   0.33   0.33    -      -      -      -      -      -      -     0.33    -     vminss	(%rax), %xmm1, %xmm2
-# CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     vmovapd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     vmovapd	%xmm0, %xmm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     vmovapd	%xmm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     vmovapd	(%rax), %xmm2
-# CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     vmovapd	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     vmovapd	%ymm0, %ymm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     vmovapd	%ymm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     vmovapd	(%rax), %ymm2
-# CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     vmovaps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     vmovaps	%xmm0, %xmm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     vmovaps	%xmm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     vmovaps	(%rax), %xmm2
-# CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     vmovaps	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     vmovaps	%ymm0, %ymm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     vmovaps	%ymm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     vmovaps	(%rax), %ymm2
 # CHECK-NEXT:  -      -      -      -      -     1.00    -      -      -      -      -      -      -     vmovd	%eax, %xmm2
@@ -1985,16 +1985,16 @@ vzeroupper
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     vmovddup	(%rax), %xmm2
 # CHECK-NEXT:  -      -      -      -      -     1.00    -      -      -      -      -      -      -     vmovddup	%ymm0, %ymm2
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     vmovddup	(%rax), %ymm2
-# CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     vmovdqa	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa	%xmm0, %xmm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     vmovdqa	%xmm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     vmovdqa	(%rax), %xmm2
-# CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     vmovdqa	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa	%ymm0, %ymm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     vmovdqa	%ymm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     vmovdqa	(%rax), %ymm2
-# CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     vmovdqu	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu	%xmm0, %xmm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     vmovdqu	%xmm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     vmovdqu	(%rax), %xmm2
-# CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     vmovdqu	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu	%ymm0, %ymm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     vmovdqu	%ymm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     vmovdqu	(%rax), %ymm2
 # CHECK-NEXT:  -      -      -      -      -     1.00    -      -      -      -      -      -      -     vmovhlps	%xmm0, %xmm1, %xmm2
@@ -2038,16 +2038,16 @@ vzeroupper
 # CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     vmovss	%xmm0, %xmm1, %xmm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     vmovss	%xmm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     vmovss	(%rax), %xmm2
-# CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     vmovupd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     vmovupd	%xmm0, %xmm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     vmovupd	%xmm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     vmovupd	(%rax), %xmm2
-# CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     vmovupd	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     vmovupd	%ymm0, %ymm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     vmovupd	%ymm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     vmovupd	(%rax), %ymm2
-# CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     vmovups	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     vmovups	%xmm0, %xmm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     vmovups	%xmm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     vmovups	(%rax), %xmm2
-# CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     vmovups	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     vmovups	%ymm0, %ymm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     vmovups	%ymm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     vmovups	(%rax), %ymm2
 # CHECK-NEXT:  -     0.50    -      -      -     1.50    -      -      -      -      -      -      -     vmpsadbw	$1, %xmm0, %xmm1, %xmm2
@@ -2383,7 +2383,7 @@ vzeroupper
 # CHECK-NEXT: 1.00    -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     vsqrtsd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT: 1.00    -      -      -      -      -      -      -      -      -      -      -      -     vsqrtss	%xmm0, %xmm1, %xmm2
 # CHECK-NEXT: 1.00    -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     vsqrtss	(%rax), %xmm1, %xmm2
-# CHECK-NEXT: 1.00    -      -      -     0.50    -     1.00   0.50   0.50   0.50    -      -      -     vstmxcsr	(%rax)
+# CHECK-NEXT: 1.50    -      -      -     0.50    -     0.50   0.50   0.50   0.50    -      -      -     vstmxcsr	(%rax)
 # CHECK-NEXT:  -     0.50    -      -      -     0.50    -      -      -      -      -      -      -     vsubpd	%xmm0, %xmm1, %xmm2
 # CHECK-NEXT:  -     0.50   0.33   0.33    -     0.50    -      -      -      -      -     0.33    -     vsubpd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  -     0.50    -      -      -     0.50    -      -      -      -      -      -      -     vsubpd	%ymm0, %ymm1, %ymm2
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-avx2.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-avx2.s
index eada999e002e2..96f66d1aad6c5 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-avx2.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-avx2.s
@@ -460,11 +460,11 @@ vpxor           (%rax), %ymm1, %ymm2
 # CHECK-NEXT: [6]: HasSideEffects (U)
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
-# CHECK-NEXT:  1      7     0.33    *                   vbroadcasti128	(%rax), %ymm0
+# CHECK-NEXT:  1      8     0.33    *                   vbroadcasti128	(%rax), %ymm0
 # CHECK-NEXT:  1      3     1.00                        vbroadcastsd	%xmm0, %ymm0
 # CHECK-NEXT:  1      3     1.00                        vbroadcastss	%xmm0, %ymm0
 # CHECK-NEXT:  1      3     1.00                        vextracti128	$1, %ymm0, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            vextracti128	$1, %ymm0, (%rax)
+# CHECK-NEXT:  2      12    0.50           *            vextracti128	$1, %ymm0, (%rax)
 # CHECK-NEXT:  5      20    1.00    *                   vgatherdpd	%xmm0, (%rax,%xmm1,2), %xmm2
 # CHECK-NEXT:  8      29    1.33    *                   vgatherdpd	%ymm0, (%rax,%xmm1,2), %ymm2
 # CHECK-NEXT:  8      28    1.33    *                   vgatherdps	%xmm0, (%rax,%xmm1,2), %xmm2
@@ -477,97 +477,97 @@ vpxor           (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  2      9     0.33    *                   vinserti128	$1, (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      8     0.33    *                   vmovntdqa	(%rax), %ymm0
 # CHECK-NEXT:  2      4     1.00                        vmpsadbw	$1, %ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  3      11    1.00    *                   vmpsadbw	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  3      12    1.00    *                   vmpsadbw	$1, (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpabsb	%ymm0, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpabsb	(%rax), %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpabsb	(%rax), %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpabsd	%ymm0, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpabsd	(%rax), %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpabsd	(%rax), %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpabsw	%ymm0, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpabsw	(%rax), %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpabsw	(%rax), %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpackssdw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpackssdw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpackssdw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpacksswb	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpacksswb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpacksswb	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpackusdw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpackusdw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpackusdw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpackuswb	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpackuswb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpackuswb	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vpaddb	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vpaddb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vpaddb	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vpaddd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vpaddd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vpaddd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vpaddq	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vpaddq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vpaddq	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpaddsb	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpaddsb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpaddsb	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpaddsw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpaddsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpaddsw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpaddusb	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpaddusb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpaddusb	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpaddusw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpaddusw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpaddusw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vpaddw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vpaddw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vpaddw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     1.00                        vpalignr	$1, %ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     1.00    *                   vpalignr	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     1.00    *                   vpalignr	$1, (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vpand	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vpand	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vpand	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vpandn	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vpandn	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vpandn	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpavgb	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpavgb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpavgb	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpavgw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpavgw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpavgw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vpblendd	$11, %xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   vpblendd	$11, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   vpblendd	$11, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.33                        vpblendd	$11, %ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vpblendd	$11, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vpblendd	$11, (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  3      3     1.00                        vpblendvb	%ymm3, %ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  4      9     1.00    *                   vpblendvb	%ymm3, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  4      10    1.00    *                   vpblendvb	%ymm3, (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpblendw	$11, %ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpblendw	$11, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpblendw	$11, (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     1.00                        vpbroadcastb	%xmm0, %xmm0
-# CHECK-NEXT:  2      7     1.00    *                   vpbroadcastb	(%rax), %xmm0
+# CHECK-NEXT:  2      8     1.00    *                   vpbroadcastb	(%rax), %xmm0
 # CHECK-NEXT:  1      3     1.00                        vpbroadcastb	%xmm0, %ymm0
-# CHECK-NEXT:  2      8     1.00    *                   vpbroadcastb	(%rax), %ymm0
+# CHECK-NEXT:  2      9     1.00    *                   vpbroadcastb	(%rax), %ymm0
 # CHECK-NEXT:  1      1     1.00                        vpbroadcastd	%xmm0, %xmm0
-# CHECK-NEXT:  1      6     0.33    *                   vpbroadcastd	(%rax), %xmm0
+# CHECK-NEXT:  1      7     0.33    *                   vpbroadcastd	(%rax), %xmm0
 # CHECK-NEXT:  1      3     1.00                        vpbroadcastd	%xmm0, %ymm0
-# CHECK-NEXT:  1      7     0.33    *                   vpbroadcastd	(%rax), %ymm0
+# CHECK-NEXT:  1      8     0.33    *                   vpbroadcastd	(%rax), %ymm0
 # CHECK-NEXT:  1      1     1.00                        vpbroadcastq	%xmm0, %xmm0
-# CHECK-NEXT:  1      6     0.33    *                   vpbroadcastq	(%rax), %xmm0
+# CHECK-NEXT:  1      7     0.33    *                   vpbroadcastq	(%rax), %xmm0
 # CHECK-NEXT:  1      3     1.00                        vpbroadcastq	%xmm0, %ymm0
-# CHECK-NEXT:  1      7     0.33    *                   vpbroadcastq	(%rax), %ymm0
+# CHECK-NEXT:  1      8     0.33    *                   vpbroadcastq	(%rax), %ymm0
 # CHECK-NEXT:  1      1     1.00                        vpbroadcastw	%xmm0, %xmm0
-# CHECK-NEXT:  2      7     1.00    *                   vpbroadcastw	(%rax), %xmm0
+# CHECK-NEXT:  2      8     1.00    *                   vpbroadcastw	(%rax), %xmm0
 # CHECK-NEXT:  1      3     1.00                        vpbroadcastw	%xmm0, %ymm0
-# CHECK-NEXT:  2      8     1.00    *                   vpbroadcastw	(%rax), %ymm0
+# CHECK-NEXT:  2      9     1.00    *                   vpbroadcastw	(%rax), %ymm0
 # CHECK-NEXT:  1      1     0.50                        vpcmpeqb	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpcmpeqb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpcmpeqb	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpcmpeqd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpcmpeqd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpcmpeqd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpcmpeqq	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpcmpeqq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpcmpeqq	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpcmpeqw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpcmpeqw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpcmpeqw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpcmpgtb	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpcmpgtb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpcmpgtb	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpcmpgtd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpcmpgtd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpcmpgtd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpcmpgtq	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpcmpgtq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpcmpgtq	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpcmpgtw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpcmpgtw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpcmpgtw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      3     1.00                        vperm2i128	$1, %ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vperm2i128	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vperm2i128	$1, (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpermd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpermd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpermd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpermpd	$1, %ymm0, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpermpd	$1, (%rax), %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpermpd	$1, (%rax), %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpermps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpermps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpermps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpermq	$1, %ymm0, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpermq	$1, (%rax), %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpermq	$1, (%rax), %ymm2
 # CHECK-NEXT:  8      28    1.33    *                   vpgatherdd	%xmm0, (%rax,%xmm1,2), %xmm2
 # CHECK-NEXT:  12     30    2.67    *                   vpgatherdd	%ymm0, (%rax,%ymm1,2), %ymm2
 # CHECK-NEXT:  5      20    1.00    *                   vpgatherdq	%xmm0, (%rax,%xmm1,2), %xmm2
@@ -577,190 +577,190 @@ vpxor           (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  5      20    1.00    *                   vpgatherqq	%xmm0, (%rax,%xmm1,2), %xmm2
 # CHECK-NEXT:  8      29    1.33    *                   vpgatherqq	%ymm0, (%rax,%ymm1,2), %ymm2
 # CHECK-NEXT:  3      2     1.00                        vphaddd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  4      9     1.00    *                   vphaddd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  4      10    1.00    *                   vphaddd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  3      2     1.00                        vphaddsw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  4      9     1.00    *                   vphaddsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  4      10    1.00    *                   vphaddsw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  3      2     1.00                        vphaddw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  4      9     1.00    *                   vphaddw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  4      10    1.00    *                   vphaddw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  3      2     1.00                        vphsubd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  4      9     1.00    *                   vphsubd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  4      10    1.00    *                   vphsubd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  3      2     1.00                        vphsubsw	%ymm0, %ymm1, %ymm2
 # CHECK-NEXT:  4      10    1.00    *                   vphsubsw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  3      2     1.00                        vphsubw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  4      9     1.00    *                   vphsubw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  4      10    1.00    *                   vphsubw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      5     0.50                        vpmaddubsw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      12    0.50    *                   vpmaddubsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      13    0.50    *                   vpmaddubsw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      5     0.50                        vpmaddwd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      12    0.50    *                   vpmaddwd	(%rax), %ymm1, %ymm2
-# CHECK-NEXT:  2      7     0.33    *                   vpmaskmovd	(%rax), %xmm0, %xmm2
-# CHECK-NEXT:  2      8     0.33    *                   vpmaskmovd	(%rax), %ymm0, %ymm2
-# CHECK-NEXT:  3      3     1.00    *      *            vpmaskmovd	%xmm0, %xmm1, (%rax)
-# CHECK-NEXT:  3      3     1.00    *      *            vpmaskmovd	%ymm0, %ymm1, (%rax)
-# CHECK-NEXT:  2      7     0.33    *                   vpmaskmovq	(%rax), %xmm0, %xmm2
-# CHECK-NEXT:  2      8     0.33    *                   vpmaskmovq	(%rax), %ymm0, %ymm2
-# CHECK-NEXT:  3      3     1.00    *      *            vpmaskmovq	%xmm0, %xmm1, (%rax)
-# CHECK-NEXT:  3      3     1.00    *      *            vpmaskmovq	%ymm0, %ymm1, (%rax)
+# CHECK-NEXT:  2      13    0.50    *                   vpmaddwd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      8     0.33    *                   vpmaskmovd	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  2      9     0.33    *                   vpmaskmovd	(%rax), %ymm0, %ymm2
+# CHECK-NEXT:  3      14    1.00    *      *            vpmaskmovd	%xmm0, %xmm1, (%rax)
+# CHECK-NEXT:  3      14    1.00    *      *            vpmaskmovd	%ymm0, %ymm1, (%rax)
+# CHECK-NEXT:  2      8     0.33    *                   vpmaskmovq	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  2      9     0.33    *                   vpmaskmovq	(%rax), %ymm0, %ymm2
+# CHECK-NEXT:  3      14    1.00    *      *            vpmaskmovq	%xmm0, %xmm1, (%rax)
+# CHECK-NEXT:  3      14    1.00    *      *            vpmaskmovq	%ymm0, %ymm1, (%rax)
 # CHECK-NEXT:  1      1     0.50                        vpmaxsb	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpmaxsb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpmaxsb	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpmaxsd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpmaxsd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpmaxsd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpmaxsw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpmaxsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpmaxsw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpmaxub	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpmaxub	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpmaxub	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpmaxud	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpmaxud	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpmaxud	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpmaxuw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpmaxuw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpmaxuw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpminsb	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpminsb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpminsb	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpminsd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpminsd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpminsd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpminsw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpminsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpminsw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpminub	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpminub	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpminub	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpminud	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpminud	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpminud	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpminuw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpminuw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpminuw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     1.00                        vpmovmskb	%ymm0, %ecx
 # CHECK-NEXT:  1      3     1.00                        vpmovsxbd	%xmm0, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpmovsxbd	(%rax), %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpmovsxbd	(%rax), %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpmovsxbq	%xmm0, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpmovsxbq	(%rax), %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpmovsxbq	(%rax), %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpmovsxbw	%xmm0, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpmovsxbw	(%rax), %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpmovsxbw	(%rax), %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpmovsxdq	%xmm0, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpmovsxdq	(%rax), %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpmovsxdq	(%rax), %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpmovsxwd	%xmm0, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpmovsxwd	(%rax), %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpmovsxwd	(%rax), %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpmovsxwq	%xmm0, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpmovsxwq	(%rax), %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpmovsxwq	(%rax), %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpmovzxbd	%xmm0, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpmovzxbd	(%rax), %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpmovzxbd	(%rax), %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpmovzxbq	%xmm0, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpmovzxbq	(%rax), %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpmovzxbq	(%rax), %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpmovzxbw	%xmm0, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpmovzxbw	(%rax), %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpmovzxbw	(%rax), %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpmovzxdq	%xmm0, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpmovzxdq	(%rax), %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpmovzxdq	(%rax), %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpmovzxwd	%xmm0, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpmovzxwd	(%rax), %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpmovzxwd	(%rax), %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpmovzxwq	%xmm0, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpmovzxwq	(%rax), %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpmovzxwq	(%rax), %ymm2
 # CHECK-NEXT:  1      5     0.50                        vpmuldq	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      12    0.50    *                   vpmuldq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      13    0.50    *                   vpmuldq	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      5     0.50                        vpmulhrsw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      12    0.50    *                   vpmulhrsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      13    0.50    *                   vpmulhrsw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      5     0.50                        vpmulhuw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      12    0.50    *                   vpmulhuw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      13    0.50    *                   vpmulhuw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      5     0.50                        vpmulhw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      12    0.50    *                   vpmulhw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      13    0.50    *                   vpmulhw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  2      10    1.00                        vpmulld	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  3      17    1.00    *                   vpmulld	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  3      18    1.00    *                   vpmulld	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      5     0.50                        vpmullw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      12    0.50    *                   vpmullw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      13    0.50    *                   vpmullw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      5     0.50                        vpmuludq	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      12    0.50    *                   vpmuludq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      13    0.50    *                   vpmuludq	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vpor	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vpor	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vpor	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      3     1.00                        vpsadbw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      10    1.00    *                   vpsadbw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      11    1.00    *                   vpsadbw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpshufb	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpshufb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpshufb	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpshufd	$1, %ymm0, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpshufd	$1, (%rax), %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpshufd	$1, (%rax), %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpshufhw	$1, %ymm0, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpshufhw	$1, (%rax), %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpshufhw	$1, (%rax), %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpshuflw	$1, %ymm0, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpshuflw	$1, (%rax), %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpshuflw	$1, (%rax), %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsignb	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsignb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsignb	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsignd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsignd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsignd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsignw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsignw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsignw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpslld	$1, %ymm0, %ymm2
 # CHECK-NEXT:  2      4     1.00                        vpslld	%xmm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpslld	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpslld	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpslldq	$1, %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsllq	$1, %ymm0, %ymm2
 # CHECK-NEXT:  2      4     1.00                        vpsllq	%xmm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsllq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsllq	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsllvd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsllvd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsllvd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsllvd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsllvd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsllvd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsllvq	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsllvq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsllvq	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsllvq	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsllvq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsllvq	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsllw	$1, %ymm0, %ymm2
 # CHECK-NEXT:  2      4     1.00                        vpsllw	%xmm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsllw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsllw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsrad	$1, %ymm0, %ymm2
 # CHECK-NEXT:  2      4     1.00                        vpsrad	%xmm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsrad	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsrad	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsravd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsravd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsravd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsravd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsravd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsravd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsraw	$1, %ymm0, %ymm2
 # CHECK-NEXT:  2      4     1.00                        vpsraw	%xmm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsraw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsraw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsrld	$1, %ymm0, %ymm2
 # CHECK-NEXT:  2      4     1.00                        vpsrld	%xmm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsrld	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsrld	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsrldq	$1, %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsrlq	$1, %ymm0, %ymm2
 # CHECK-NEXT:  2      4     1.00                        vpsrlq	%xmm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsrlq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsrlq	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsrlvd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsrlvd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsrlvd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsrlvd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsrlvd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsrlvd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsrlvq	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   vpsrlvq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   vpsrlvq	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        vpsrlvq	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsrlvq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsrlvq	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsrlw	$1, %ymm0, %ymm2
 # CHECK-NEXT:  2      4     1.00                        vpsrlw	%xmm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsrlw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsrlw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vpsubb	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vpsubb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vpsubb	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vpsubd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vpsubd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vpsubd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vpsubq	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vpsubq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vpsubq	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsubsb	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsubsb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsubsb	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsubsw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsubsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsubsw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsubusb	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsubusb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsubusb	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpsubusw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpsubusw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpsubusw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vpsubw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vpsubw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vpsubw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpunpckhbw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpunpckhbw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpunpckhbw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpunpckhdq	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpunpckhdq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpunpckhdq	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpunpckhqdq	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpunpckhqdq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpunpckhqdq	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpunpckhwd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpunpckhwd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpunpckhwd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpunpcklbw	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpunpcklbw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpunpcklbw	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpunpckldq	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpunpckldq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpunpckldq	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpunpcklqdq	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpunpcklqdq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpunpcklqdq	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vpunpcklwd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.50    *                   vpunpcklwd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.50    *                   vpunpcklwd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.33                        vpxor	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      8     0.33    *                   vpxor	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      9     0.33    *                   vpxor	(%rax), %ymm1, %ymm2
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - ADLPPort00
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-avxgfni.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-avxgfni.s
index 540c82e87b0cb..2f9fe5dd23a17 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-avxgfni.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-avxgfni.s
@@ -28,18 +28,18 @@ vgf2p8mulb          (%rax), %ymm1, %ymm2
 # CHECK-NEXT: [6]: HasSideEffects (U)
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
-# CHECK-NEXT:  1      3     0.50                        vgf2p8affineinvqb	$0, %xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vgf2p8affineinvqb	$0, (%rax), %xmm1, %xmm2
-# CHECK-NEXT:  1      3     0.50                        vgf2p8affineinvqb	$0, %ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vgf2p8affineinvqb	$0, (%rax), %ymm1, %ymm2
-# CHECK-NEXT:  1      3     0.50                        vgf2p8affineqb	$0, %xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vgf2p8affineqb	$0, (%rax), %xmm1, %xmm2
-# CHECK-NEXT:  1      3     0.50                        vgf2p8affineqb	$0, %ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vgf2p8affineqb	$0, (%rax), %ymm1, %ymm2
-# CHECK-NEXT:  1      3     0.50                        vgf2p8mulb	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vgf2p8mulb	(%rax), %xmm1, %xmm2
-# CHECK-NEXT:  1      3     0.50                        vgf2p8mulb	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vgf2p8mulb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      5     0.50                        vgf2p8affineinvqb	$0, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  2      12    0.50    *                   vgf2p8affineinvqb	$0, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      5     0.50                        vgf2p8affineinvqb	$0, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vgf2p8affineinvqb	$0, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      5     0.50                        vgf2p8affineqb	$0, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  2      12    0.50    *                   vgf2p8affineqb	$0, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      5     0.50                        vgf2p8affineqb	$0, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vgf2p8affineqb	$0, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      5     0.50                        vgf2p8mulb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  2      12    0.50    *                   vgf2p8mulb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      5     0.50                        vgf2p8mulb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vgf2p8mulb	(%rax), %ymm1, %ymm2
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - ADLPPort00
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-avxvnni.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-avxvnni.s
index e586662d71aa3..5c8d5e74e7eda 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-avxvnni.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-avxvnni.s
@@ -35,21 +35,21 @@ vpdpwssds   (%rax), %ymm1, %ymm2
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
 # CHECK-NEXT:  1      5     0.50                        vpdpbusd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vpdpbusd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   vpdpbusd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      5     0.50                        vpdpbusd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      12    0.50    *                   vpdpbusd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      13    0.50    *                   vpdpbusd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      5     0.50                        vpdpbusds	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vpdpbusds	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   vpdpbusds	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      5     0.50                        vpdpbusds	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      12    0.50    *                   vpdpbusds	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      13    0.50    *                   vpdpbusds	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      5     0.50                        vpdpwssd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vpdpwssd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   vpdpwssd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      5     0.50                        vpdpwssd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      12    0.50    *                   vpdpwssd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      13    0.50    *                   vpdpwssd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      5     0.50                        vpdpwssds	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vpdpwssds	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   vpdpwssds	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      5     0.50                        vpdpwssds	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      12    0.50    *                   vpdpwssds	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      13    0.50    *                   vpdpwssds	(%rax), %ymm1, %ymm2
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - ADLPPort00
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-bmi1.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-bmi1.s
index 652c9e3aad450..4ed882a37a68e 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-bmi1.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-bmi1.s
@@ -49,26 +49,26 @@ tzcnt       (%rax), %rcx
 # CHECK-NEXT: [6]: HasSideEffects (U)
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
-# CHECK-NEXT:  1      1     0.33                        andnl	%eax, %ebx, %ecx
-# CHECK-NEXT:  2      6     0.33    *                   andnl	(%rax), %ebx, %ecx
-# CHECK-NEXT:  1      1     0.33                        andnq	%rax, %rbx, %rcx
-# CHECK-NEXT:  2      6     0.33    *                   andnq	(%rax), %rbx, %rcx
-# CHECK-NEXT:  2      2     1.00                        bextrl	%eax, %ebx, %ecx
-# CHECK-NEXT:  3      7     1.00    *                   bextrl	%eax, (%rbx), %ecx
-# CHECK-NEXT:  2      2     1.00                        bextrq	%rax, %rbx, %rcx
-# CHECK-NEXT:  3      7     1.00    *                   bextrq	%rax, (%rbx), %rcx
-# CHECK-NEXT:  1      1     0.33                        blsil	%eax, %ecx
-# CHECK-NEXT:  2      6     0.33    *                   blsil	(%rax), %ecx
-# CHECK-NEXT:  1      1     0.33                        blsiq	%rax, %rcx
-# CHECK-NEXT:  2      6     0.33    *                   blsiq	(%rax), %rcx
-# CHECK-NEXT:  1      1     0.33                        blsmskl	%eax, %ecx
-# CHECK-NEXT:  2      6     0.33    *                   blsmskl	(%rax), %ecx
-# CHECK-NEXT:  1      1     0.33                        blsmskq	%rax, %rcx
-# CHECK-NEXT:  2      6     0.33    *                   blsmskq	(%rax), %rcx
-# CHECK-NEXT:  1      1     0.33                        blsrl	%eax, %ecx
-# CHECK-NEXT:  2      6     0.33    *                   blsrl	(%rax), %ecx
-# CHECK-NEXT:  1      1     0.33                        blsrq	%rax, %rcx
-# CHECK-NEXT:  2      6     0.33    *                   blsrq	(%rax), %rcx
+# CHECK-NEXT:  1      2     0.33                        andnl	%eax, %ebx, %ecx
+# CHECK-NEXT:  2      7     0.33    *                   andnl	(%rax), %ebx, %ecx
+# CHECK-NEXT:  1      2     0.33                        andnq	%rax, %rbx, %rcx
+# CHECK-NEXT:  2      7     0.33    *                   andnq	(%rax), %rbx, %rcx
+# CHECK-NEXT:  2      6     1.00                        bextrl	%eax, %ebx, %ecx
+# CHECK-NEXT:  3      11    1.00    *                   bextrl	%eax, (%rbx), %ecx
+# CHECK-NEXT:  2      6     1.00                        bextrq	%rax, %rbx, %rcx
+# CHECK-NEXT:  3      11    1.00    *                   bextrq	%rax, (%rbx), %rcx
+# CHECK-NEXT:  1      2     0.33                        blsil	%eax, %ecx
+# CHECK-NEXT:  2      7     0.33    *                   blsil	(%rax), %ecx
+# CHECK-NEXT:  1      2     0.33                        blsiq	%rax, %rcx
+# CHECK-NEXT:  2      7     0.33    *                   blsiq	(%rax), %rcx
+# CHECK-NEXT:  1      2     0.33                        blsmskl	%eax, %ecx
+# CHECK-NEXT:  2      7     0.33    *                   blsmskl	(%rax), %ecx
+# CHECK-NEXT:  1      2     0.33                        blsmskq	%rax, %rcx
+# CHECK-NEXT:  2      7     0.33    *                   blsmskq	(%rax), %rcx
+# CHECK-NEXT:  1      2     0.33                        blsrl	%eax, %ecx
+# CHECK-NEXT:  2      7     0.33    *                   blsrl	(%rax), %ecx
+# CHECK-NEXT:  1      2     0.33                        blsrq	%rax, %rcx
+# CHECK-NEXT:  2      7     0.33    *                   blsrq	(%rax), %rcx
 # CHECK-NEXT:  1      3     1.00                        tzcntw	%ax, %cx
 # CHECK-NEXT:  2      8     1.00    *                   tzcntw	(%rax), %cx
 # CHECK-NEXT:  1      3     1.00                        tzcntl	%eax, %ecx
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-bmi2.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-bmi2.s
index 8f955ff6c2d93..559ca83906cb7 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-bmi2.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-bmi2.s
@@ -58,10 +58,10 @@ shrx        %rax, (%rbx), %rcx
 # CHECK-NEXT: [6]: HasSideEffects (U)
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
-# CHECK-NEXT:  1      1     1.00                        bzhil	%eax, %ebx, %ecx
-# CHECK-NEXT:  2      6     1.00    *                   bzhil	%eax, (%rbx), %ecx
-# CHECK-NEXT:  1      1     1.00                        bzhiq	%rax, %rbx, %rcx
-# CHECK-NEXT:  2      6     1.00    *                   bzhiq	%rax, (%rbx), %rcx
+# CHECK-NEXT:  1      3     1.00                        bzhil	%eax, %ebx, %ecx
+# CHECK-NEXT:  2      8     1.00    *                   bzhil	%eax, (%rbx), %ecx
+# CHECK-NEXT:  1      3     1.00                        bzhiq	%rax, %rbx, %rcx
+# CHECK-NEXT:  2      8     1.00    *                   bzhiq	%rax, (%rbx), %rcx
 # CHECK-NEXT:  3      4     1.00                        mulxl	%eax, %ebx, %ecx
 # CHECK-NEXT:  4      9     1.00    *                   mulxl	(%rax), %ebx, %ecx
 # CHECK-NEXT:  2      4     1.00                        mulxq	%rax, %rbx, %rcx
@@ -78,18 +78,18 @@ shrx        %rax, (%rbx), %rcx
 # CHECK-NEXT:  2      6     0.50    *                   rorxl	$1, (%rax), %ecx
 # CHECK-NEXT:  1      1     0.50                        rorxq	$1, %rax, %rcx
 # CHECK-NEXT:  2      6     0.50    *                   rorxq	$1, (%rax), %rcx
-# CHECK-NEXT:  1      1     0.50                        sarxl	%eax, %ebx, %ecx
-# CHECK-NEXT:  2      6     0.50    *                   sarxl	%eax, (%rbx), %ecx
-# CHECK-NEXT:  1      1     0.50                        sarxq	%rax, %rbx, %rcx
-# CHECK-NEXT:  2      6     0.50    *                   sarxq	%rax, (%rbx), %rcx
-# CHECK-NEXT:  1      1     0.50                        shlxl	%eax, %ebx, %ecx
-# CHECK-NEXT:  2      6     0.50    *                   shlxl	%eax, (%rbx), %ecx
-# CHECK-NEXT:  1      1     0.50                        shlxq	%rax, %rbx, %rcx
-# CHECK-NEXT:  2      6     0.50    *                   shlxq	%rax, (%rbx), %rcx
-# CHECK-NEXT:  1      1     0.50                        shrxl	%eax, %ebx, %ecx
-# CHECK-NEXT:  2      6     0.50    *                   shrxl	%eax, (%rbx), %ecx
-# CHECK-NEXT:  1      1     0.50                        shrxq	%rax, %rbx, %rcx
-# CHECK-NEXT:  2      6     0.50    *                   shrxq	%rax, (%rbx), %rcx
+# CHECK-NEXT:  1      3     0.50                        sarxl	%eax, %ebx, %ecx
+# CHECK-NEXT:  2      8     0.50    *                   sarxl	%eax, (%rbx), %ecx
+# CHECK-NEXT:  1      3     0.50                        sarxq	%rax, %rbx, %rcx
+# CHECK-NEXT:  2      8     0.50    *                   sarxq	%rax, (%rbx), %rcx
+# CHECK-NEXT:  1      3     0.50                        shlxl	%eax, %ebx, %ecx
+# CHECK-NEXT:  2      8     0.50    *                   shlxl	%eax, (%rbx), %ecx
+# CHECK-NEXT:  1      3     0.50                        shlxq	%rax, %rbx, %rcx
+# CHECK-NEXT:  2      8     0.50    *                   shlxq	%rax, (%rbx), %rcx
+# CHECK-NEXT:  1      3     0.50                        shrxl	%eax, %ebx, %ecx
+# CHECK-NEXT:  2      8     0.50    *                   shrxl	%eax, (%rbx), %ecx
+# CHECK-NEXT:  1      3     0.50                        shrxq	%rax, %rbx, %rcx
+# CHECK-NEXT:  2      8     0.50    *                   shrxq	%rax, (%rbx), %rcx
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - ADLPPort00
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-clflushopt.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-clflushopt.s
index bd51197554e59..e61cc06951ae5 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-clflushopt.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-clflushopt.s
@@ -31,8 +31,8 @@ clflushopt (%rax)
 
 # CHECK:      Resource pressure per iteration:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]
-# CHECK-NEXT: 0.25    -      -      -     0.50   0.25   0.25   0.50   0.50   0.50   0.25    -      -
+# CHECK-NEXT: 0.20   0.20    -      -     0.50   0.20   0.20   0.50   0.50   0.50   0.20    -      -
 
 # CHECK:      Resource pressure by instruction:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]   Instructions:
-# CHECK-NEXT: 0.25    -      -      -     0.50   0.25   0.25   0.50   0.50   0.50   0.25    -      -     clflushopt	(%rax)
+# CHECK-NEXT: 0.20   0.20    -      -     0.50   0.20   0.20   0.50   0.50   0.50   0.20    -      -     clflushopt	(%rax)
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-clwb.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-clwb.s
index 4d547f247d5db..d35eadcc3f9d0 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-clwb.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-clwb.s
@@ -31,8 +31,8 @@ clwb (%rax)
 
 # CHECK:      Resource pressure per iteration:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]
-# CHECK-NEXT: 0.25    -      -      -     0.50   0.25   0.25   0.50   0.50   0.50   0.25    -      -
+# CHECK-NEXT: 0.20   0.20    -      -     0.50   0.20   0.20   0.50   0.50   0.50   0.20    -      -
 
 # CHECK:      Resource pressure by instruction:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]   Instructions:
-# CHECK-NEXT: 0.25    -      -      -     0.50   0.25   0.25   0.50   0.50   0.50   0.25    -      -     clwb	(%rax)
+# CHECK-NEXT: 0.20   0.20    -      -     0.50   0.20   0.20   0.50   0.50   0.50   0.20    -      -     clwb	(%rax)
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-cmov.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-cmov.s
index 12d157aca298a..87a0e070096c2 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-cmov.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-cmov.s
@@ -112,102 +112,102 @@ cmovgq    (%rax), %rdi
 # CHECK-NEXT: [6]: HasSideEffects (U)
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
-# CHECK-NEXT:  1      2     1.00                        cmovow	%si, %di
-# CHECK-NEXT:  1      2     1.00                        cmovnow	%si, %di
-# CHECK-NEXT:  1      2     1.00                        cmovbw	%si, %di
-# CHECK-NEXT:  1      2     1.00                        cmovaew	%si, %di
-# CHECK-NEXT:  1      2     1.00                        cmovew	%si, %di
-# CHECK-NEXT:  1      2     1.00                        cmovnew	%si, %di
-# CHECK-NEXT:  1      2     1.00                        cmovbew	%si, %di
-# CHECK-NEXT:  1      2     1.00                        cmovaw	%si, %di
-# CHECK-NEXT:  1      2     1.00                        cmovsw	%si, %di
-# CHECK-NEXT:  1      2     1.00                        cmovnsw	%si, %di
-# CHECK-NEXT:  1      2     1.00                        cmovpw	%si, %di
-# CHECK-NEXT:  1      2     1.00                        cmovnpw	%si, %di
-# CHECK-NEXT:  1      2     1.00                        cmovlw	%si, %di
-# CHECK-NEXT:  1      2     1.00                        cmovgew	%si, %di
-# CHECK-NEXT:  1      2     1.00                        cmovlew	%si, %di
-# CHECK-NEXT:  1      2     1.00                        cmovgw	%si, %di
-# CHECK-NEXT:  2      6     1.00    *                   cmovow	(%rax), %di
-# CHECK-NEXT:  2      6     1.00    *                   cmovnow	(%rax), %di
-# CHECK-NEXT:  2      6     1.00    *                   cmovbw	(%rax), %di
-# CHECK-NEXT:  2      6     1.00    *                   cmovaew	(%rax), %di
-# CHECK-NEXT:  2      6     1.00    *                   cmovew	(%rax), %di
-# CHECK-NEXT:  2      6     1.00    *                   cmovnew	(%rax), %di
-# CHECK-NEXT:  2      6     1.00    *                   cmovbew	(%rax), %di
-# CHECK-NEXT:  2      6     1.00    *                   cmovaw	(%rax), %di
-# CHECK-NEXT:  2      6     1.00    *                   cmovsw	(%rax), %di
-# CHECK-NEXT:  2      6     1.00    *                   cmovnsw	(%rax), %di
-# CHECK-NEXT:  2      6     1.00    *                   cmovpw	(%rax), %di
-# CHECK-NEXT:  2      6     1.00    *                   cmovnpw	(%rax), %di
-# CHECK-NEXT:  2      6     1.00    *                   cmovlw	(%rax), %di
-# CHECK-NEXT:  2      6     1.00    *                   cmovgew	(%rax), %di
-# CHECK-NEXT:  2      6     1.00    *                   cmovlew	(%rax), %di
-# CHECK-NEXT:  2      6     1.00    *                   cmovgw	(%rax), %di
-# CHECK-NEXT:  1      2     1.00                        cmovol	%esi, %edi
-# CHECK-NEXT:  1      2     1.00                        cmovnol	%esi, %edi
-# CHECK-NEXT:  1      2     1.00                        cmovbl	%esi, %edi
-# CHECK-NEXT:  1      2     1.00                        cmovael	%esi, %edi
-# CHECK-NEXT:  1      2     1.00                        cmovel	%esi, %edi
-# CHECK-NEXT:  1      2     1.00                        cmovnel	%esi, %edi
-# CHECK-NEXT:  1      2     1.00                        cmovbel	%esi, %edi
-# CHECK-NEXT:  1      2     1.00                        cmoval	%esi, %edi
-# CHECK-NEXT:  1      2     1.00                        cmovsl	%esi, %edi
-# CHECK-NEXT:  1      2     1.00                        cmovnsl	%esi, %edi
-# CHECK-NEXT:  1      2     1.00                        cmovpl	%esi, %edi
-# CHECK-NEXT:  1      2     1.00                        cmovnpl	%esi, %edi
-# CHECK-NEXT:  1      2     1.00                        cmovll	%esi, %edi
-# CHECK-NEXT:  1      2     1.00                        cmovgel	%esi, %edi
-# CHECK-NEXT:  1      2     1.00                        cmovlel	%esi, %edi
-# CHECK-NEXT:  1      2     1.00                        cmovgl	%esi, %edi
-# CHECK-NEXT:  2      6     1.00    *                   cmovol	(%rax), %edi
-# CHECK-NEXT:  2      6     1.00    *                   cmovnol	(%rax), %edi
-# CHECK-NEXT:  2      6     1.00    *                   cmovbl	(%rax), %edi
-# CHECK-NEXT:  2      6     1.00    *                   cmovael	(%rax), %edi
-# CHECK-NEXT:  2      6     1.00    *                   cmovel	(%rax), %edi
-# CHECK-NEXT:  2      6     1.00    *                   cmovnel	(%rax), %edi
-# CHECK-NEXT:  2      6     1.00    *                   cmovbel	(%rax), %edi
-# CHECK-NEXT:  2      6     1.00    *                   cmoval	(%rax), %edi
-# CHECK-NEXT:  2      6     1.00    *                   cmovsl	(%rax), %edi
-# CHECK-NEXT:  2      6     1.00    *                   cmovnsl	(%rax), %edi
-# CHECK-NEXT:  2      6     1.00    *                   cmovpl	(%rax), %edi
-# CHECK-NEXT:  2      6     1.00    *                   cmovnpl	(%rax), %edi
-# CHECK-NEXT:  2      6     1.00    *                   cmovll	(%rax), %edi
-# CHECK-NEXT:  2      6     1.00    *                   cmovgel	(%rax), %edi
-# CHECK-NEXT:  2      6     1.00    *                   cmovlel	(%rax), %edi
-# CHECK-NEXT:  2      6     1.00    *                   cmovgl	(%rax), %edi
-# CHECK-NEXT:  1      2     1.00                        cmovoq	%rsi, %rdi
-# CHECK-NEXT:  1      2     1.00                        cmovnoq	%rsi, %rdi
-# CHECK-NEXT:  1      2     1.00                        cmovbq	%rsi, %rdi
-# CHECK-NEXT:  1      2     1.00                        cmovaeq	%rsi, %rdi
-# CHECK-NEXT:  1      2     1.00                        cmoveq	%rsi, %rdi
-# CHECK-NEXT:  1      2     1.00                        cmovneq	%rsi, %rdi
-# CHECK-NEXT:  1      2     1.00                        cmovbeq	%rsi, %rdi
-# CHECK-NEXT:  1      2     1.00                        cmovaq	%rsi, %rdi
-# CHECK-NEXT:  1      2     1.00                        cmovsq	%rsi, %rdi
-# CHECK-NEXT:  1      2     1.00                        cmovnsq	%rsi, %rdi
-# CHECK-NEXT:  1      2     1.00                        cmovpq	%rsi, %rdi
-# CHECK-NEXT:  1      2     1.00                        cmovnpq	%rsi, %rdi
-# CHECK-NEXT:  1      2     1.00                        cmovlq	%rsi, %rdi
-# CHECK-NEXT:  1      2     1.00                        cmovgeq	%rsi, %rdi
-# CHECK-NEXT:  1      2     1.00                        cmovleq	%rsi, %rdi
-# CHECK-NEXT:  1      2     1.00                        cmovgq	%rsi, %rdi
-# CHECK-NEXT:  2      6     1.00    *                   cmovoq	(%rax), %rdi
-# CHECK-NEXT:  2      6     1.00    *                   cmovnoq	(%rax), %rdi
-# CHECK-NEXT:  2      6     1.00    *                   cmovbq	(%rax), %rdi
-# CHECK-NEXT:  2      6     1.00    *                   cmovaeq	(%rax), %rdi
-# CHECK-NEXT:  2      6     1.00    *                   cmoveq	(%rax), %rdi
-# CHECK-NEXT:  2      6     1.00    *                   cmovneq	(%rax), %rdi
-# CHECK-NEXT:  2      6     1.00    *                   cmovbeq	(%rax), %rdi
-# CHECK-NEXT:  2      6     1.00    *                   cmovaq	(%rax), %rdi
-# CHECK-NEXT:  2      6     1.00    *                   cmovsq	(%rax), %rdi
-# CHECK-NEXT:  2      6     1.00    *                   cmovnsq	(%rax), %rdi
-# CHECK-NEXT:  2      6     1.00    *                   cmovpq	(%rax), %rdi
-# CHECK-NEXT:  2      6     1.00    *                   cmovnpq	(%rax), %rdi
-# CHECK-NEXT:  2      6     1.00    *                   cmovlq	(%rax), %rdi
-# CHECK-NEXT:  2      6     1.00    *                   cmovgeq	(%rax), %rdi
-# CHECK-NEXT:  2      6     1.00    *                   cmovleq	(%rax), %rdi
-# CHECK-NEXT:  2      6     1.00    *                   cmovgq	(%rax), %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovow	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovnow	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovbw	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovaew	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovew	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovnew	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovbew	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovaw	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovsw	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovnsw	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovpw	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovnpw	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovlw	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovgew	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovlew	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovgw	%si, %di
+# CHECK-NEXT:  2      7     0.50    *                   cmovow	(%rax), %di
+# CHECK-NEXT:  2      7     0.50    *                   cmovnow	(%rax), %di
+# CHECK-NEXT:  2      7     0.50    *                   cmovbw	(%rax), %di
+# CHECK-NEXT:  2      7     0.50    *                   cmovaew	(%rax), %di
+# CHECK-NEXT:  2      7     0.50    *                   cmovew	(%rax), %di
+# CHECK-NEXT:  2      7     0.50    *                   cmovnew	(%rax), %di
+# CHECK-NEXT:  2      7     0.50    *                   cmovbew	(%rax), %di
+# CHECK-NEXT:  2      7     0.50    *                   cmovaw	(%rax), %di
+# CHECK-NEXT:  2      7     0.50    *                   cmovsw	(%rax), %di
+# CHECK-NEXT:  2      7     0.50    *                   cmovnsw	(%rax), %di
+# CHECK-NEXT:  2      7     0.50    *                   cmovpw	(%rax), %di
+# CHECK-NEXT:  2      7     0.50    *                   cmovnpw	(%rax), %di
+# CHECK-NEXT:  2      7     0.50    *                   cmovlw	(%rax), %di
+# CHECK-NEXT:  2      7     0.50    *                   cmovgew	(%rax), %di
+# CHECK-NEXT:  2      7     0.50    *                   cmovlew	(%rax), %di
+# CHECK-NEXT:  2      7     0.50    *                   cmovgw	(%rax), %di
+# CHECK-NEXT:  1      1     0.50                        cmovol	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovnol	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovbl	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovael	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovel	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovnel	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovbel	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmoval	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovsl	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovnsl	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovpl	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovnpl	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovll	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovgel	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovlel	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovgl	%esi, %edi
+# CHECK-NEXT:  2      7     0.50    *                   cmovol	(%rax), %edi
+# CHECK-NEXT:  2      7     0.50    *                   cmovnol	(%rax), %edi
+# CHECK-NEXT:  2      7     0.50    *                   cmovbl	(%rax), %edi
+# CHECK-NEXT:  2      7     0.50    *                   cmovael	(%rax), %edi
+# CHECK-NEXT:  2      7     0.50    *                   cmovel	(%rax), %edi
+# CHECK-NEXT:  2      7     0.50    *                   cmovnel	(%rax), %edi
+# CHECK-NEXT:  2      7     0.50    *                   cmovbel	(%rax), %edi
+# CHECK-NEXT:  2      7     0.50    *                   cmoval	(%rax), %edi
+# CHECK-NEXT:  2      7     0.50    *                   cmovsl	(%rax), %edi
+# CHECK-NEXT:  2      7     0.50    *                   cmovnsl	(%rax), %edi
+# CHECK-NEXT:  2      7     0.50    *                   cmovpl	(%rax), %edi
+# CHECK-NEXT:  2      7     0.50    *                   cmovnpl	(%rax), %edi
+# CHECK-NEXT:  2      7     0.50    *                   cmovll	(%rax), %edi
+# CHECK-NEXT:  2      7     0.50    *                   cmovgel	(%rax), %edi
+# CHECK-NEXT:  2      7     0.50    *                   cmovlel	(%rax), %edi
+# CHECK-NEXT:  2      7     0.50    *                   cmovgl	(%rax), %edi
+# CHECK-NEXT:  1      1     0.50                        cmovoq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovnoq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovbq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovaeq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmoveq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovneq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovbeq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovaq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovsq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovnsq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovpq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovnpq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovlq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovgeq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovleq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovgq	%rsi, %rdi
+# CHECK-NEXT:  2      7     0.50    *                   cmovoq	(%rax), %rdi
+# CHECK-NEXT:  2      7     0.50    *                   cmovnoq	(%rax), %rdi
+# CHECK-NEXT:  2      7     0.50    *                   cmovbq	(%rax), %rdi
+# CHECK-NEXT:  2      7     0.50    *                   cmovaeq	(%rax), %rdi
+# CHECK-NEXT:  2      7     0.50    *                   cmoveq	(%rax), %rdi
+# CHECK-NEXT:  2      7     0.50    *                   cmovneq	(%rax), %rdi
+# CHECK-NEXT:  2      7     0.50    *                   cmovbeq	(%rax), %rdi
+# CHECK-NEXT:  2      7     0.50    *                   cmovaq	(%rax), %rdi
+# CHECK-NEXT:  2      7     0.50    *                   cmovsq	(%rax), %rdi
+# CHECK-NEXT:  2      7     0.50    *                   cmovnsq	(%rax), %rdi
+# CHECK-NEXT:  2      7     0.50    *                   cmovpq	(%rax), %rdi
+# CHECK-NEXT:  2      7     0.50    *                   cmovnpq	(%rax), %rdi
+# CHECK-NEXT:  2      7     0.50    *                   cmovlq	(%rax), %rdi
+# CHECK-NEXT:  2      7     0.50    *                   cmovgeq	(%rax), %rdi
+# CHECK-NEXT:  2      7     0.50    *                   cmovleq	(%rax), %rdi
+# CHECK-NEXT:  2      7     0.50    *                   cmovgq	(%rax), %rdi
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - ADLPPort00
@@ -226,103 +226,103 @@ cmovgq    (%rax), %rdi
 
 # CHECK:      Resource pressure per iteration:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]
-# CHECK-NEXT: 96.00   -     16.00  16.00   -      -     96.00   -      -      -      -     16.00   -
+# CHECK-NEXT: 48.00   -     16.00  16.00   -      -     48.00   -      -      -      -     16.00   -
 
 # CHECK:      Resource pressure by instruction:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]   Instructions:
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovow	%si, %di
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovnow	%si, %di
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovbw	%si, %di
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovaew	%si, %di
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovew	%si, %di
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovnew	%si, %di
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovbew	%si, %di
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovaw	%si, %di
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovsw	%si, %di
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovnsw	%si, %di
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovpw	%si, %di
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovnpw	%si, %di
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovlw	%si, %di
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovgew	%si, %di
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovlew	%si, %di
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovgw	%si, %di
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovow	(%rax), %di
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovnow	(%rax), %di
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovbw	(%rax), %di
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovaew	(%rax), %di
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovew	(%rax), %di
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovnew	(%rax), %di
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovbew	(%rax), %di
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovaw	(%rax), %di
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovsw	(%rax), %di
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovnsw	(%rax), %di
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovpw	(%rax), %di
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovnpw	(%rax), %di
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovlw	(%rax), %di
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovgew	(%rax), %di
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovlew	(%rax), %di
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovgw	(%rax), %di
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovol	%esi, %edi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovnol	%esi, %edi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovbl	%esi, %edi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovael	%esi, %edi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovel	%esi, %edi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovnel	%esi, %edi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovbel	%esi, %edi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmoval	%esi, %edi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovsl	%esi, %edi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovnsl	%esi, %edi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovpl	%esi, %edi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovnpl	%esi, %edi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovll	%esi, %edi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovgel	%esi, %edi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovlel	%esi, %edi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovgl	%esi, %edi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovol	(%rax), %edi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovnol	(%rax), %edi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovbl	(%rax), %edi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovael	(%rax), %edi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovel	(%rax), %edi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovnel	(%rax), %edi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovbel	(%rax), %edi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmoval	(%rax), %edi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovsl	(%rax), %edi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovnsl	(%rax), %edi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovpl	(%rax), %edi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovnpl	(%rax), %edi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovll	(%rax), %edi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovgel	(%rax), %edi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovlel	(%rax), %edi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovgl	(%rax), %edi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovoq	%rsi, %rdi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovnoq	%rsi, %rdi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovbq	%rsi, %rdi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovaeq	%rsi, %rdi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmoveq	%rsi, %rdi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovneq	%rsi, %rdi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovbeq	%rsi, %rdi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovaq	%rsi, %rdi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovsq	%rsi, %rdi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovnsq	%rsi, %rdi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovpq	%rsi, %rdi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovnpq	%rsi, %rdi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovlq	%rsi, %rdi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovgeq	%rsi, %rdi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovleq	%rsi, %rdi
-# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     cmovgq	%rsi, %rdi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovoq	(%rax), %rdi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovnoq	(%rax), %rdi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovbq	(%rax), %rdi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovaeq	(%rax), %rdi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmoveq	(%rax), %rdi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovneq	(%rax), %rdi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovbeq	(%rax), %rdi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovaq	(%rax), %rdi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovsq	(%rax), %rdi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovnsq	(%rax), %rdi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovpq	(%rax), %rdi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovnpq	(%rax), %rdi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovlq	(%rax), %rdi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovgeq	(%rax), %rdi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovleq	(%rax), %rdi
-# CHECK-NEXT: 1.00    -     0.33   0.33    -      -     1.00    -      -      -      -     0.33    -     cmovgq	(%rax), %rdi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovow	%si, %di
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovnow	%si, %di
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovbw	%si, %di
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovaew	%si, %di
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovew	%si, %di
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovnew	%si, %di
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovbew	%si, %di
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovaw	%si, %di
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovsw	%si, %di
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovnsw	%si, %di
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovpw	%si, %di
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovnpw	%si, %di
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovlw	%si, %di
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovgew	%si, %di
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovlew	%si, %di
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovgw	%si, %di
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovow	(%rax), %di
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovnow	(%rax), %di
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovbw	(%rax), %di
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovaew	(%rax), %di
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovew	(%rax), %di
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovnew	(%rax), %di
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovbew	(%rax), %di
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovaw	(%rax), %di
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovsw	(%rax), %di
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovnsw	(%rax), %di
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovpw	(%rax), %di
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovnpw	(%rax), %di
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovlw	(%rax), %di
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovgew	(%rax), %di
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovlew	(%rax), %di
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovgw	(%rax), %di
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovol	%esi, %edi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovnol	%esi, %edi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovbl	%esi, %edi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovael	%esi, %edi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovel	%esi, %edi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovnel	%esi, %edi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovbel	%esi, %edi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmoval	%esi, %edi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovsl	%esi, %edi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovnsl	%esi, %edi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovpl	%esi, %edi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovnpl	%esi, %edi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovll	%esi, %edi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovgel	%esi, %edi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovlel	%esi, %edi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovgl	%esi, %edi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovol	(%rax), %edi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovnol	(%rax), %edi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovbl	(%rax), %edi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovael	(%rax), %edi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovel	(%rax), %edi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovnel	(%rax), %edi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovbel	(%rax), %edi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmoval	(%rax), %edi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovsl	(%rax), %edi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovnsl	(%rax), %edi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovpl	(%rax), %edi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovnpl	(%rax), %edi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovll	(%rax), %edi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovgel	(%rax), %edi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovlel	(%rax), %edi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovgl	(%rax), %edi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovoq	%rsi, %rdi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovnoq	%rsi, %rdi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovbq	%rsi, %rdi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovaeq	%rsi, %rdi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmoveq	%rsi, %rdi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovneq	%rsi, %rdi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovbeq	%rsi, %rdi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovaq	%rsi, %rdi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovsq	%rsi, %rdi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovnsq	%rsi, %rdi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovpq	%rsi, %rdi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovnpq	%rsi, %rdi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovlq	%rsi, %rdi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovgeq	%rsi, %rdi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovleq	%rsi, %rdi
+# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cmovgq	%rsi, %rdi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovoq	(%rax), %rdi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovnoq	(%rax), %rdi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovbq	(%rax), %rdi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovaeq	(%rax), %rdi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmoveq	(%rax), %rdi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovneq	(%rax), %rdi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovbeq	(%rax), %rdi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovaq	(%rax), %rdi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovsq	(%rax), %rdi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovnsq	(%rax), %rdi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovpq	(%rax), %rdi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovnpq	(%rax), %rdi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovlq	(%rax), %rdi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovgeq	(%rax), %rdi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovleq	(%rax), %rdi
+# CHECK-NEXT: 0.50    -     0.33   0.33    -      -     0.50    -      -      -      -     0.33    -     cmovgq	(%rax), %rdi
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-f16c.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-f16c.s
index e59bcb413af3f..7e4eeebfdaca1 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-f16c.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-f16c.s
@@ -23,7 +23,7 @@ vcvtps2ph   $0, %ymm0, (%rax)
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
 # CHECK-NEXT:  2      6     1.00                        vcvtph2ps	%xmm0, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   vcvtph2ps	(%rax), %xmm2
+# CHECK-NEXT:  2      12    0.50    *                   vcvtph2ps	(%rax), %xmm2
 # CHECK-NEXT:  2      8     1.00                        vcvtph2ps	%xmm0, %ymm2
 # CHECK-NEXT:  2      12    0.50    *                   vcvtph2ps	(%rax), %ymm2
 # CHECK-NEXT:  2      6     1.00                        vcvtps2ph	$0, %xmm0, %xmm2
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-fma.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-fma.s
index 31419a89a27bf..63fc8dbaa44b2 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-fma.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-fma.s
@@ -299,197 +299,197 @@ vfnmsub231ss (%rax), %xmm1, %xmm2
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
 # CHECK-NEXT:  1      4     0.50                        vfmadd132pd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmadd132pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmadd132pd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmadd132pd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmadd132pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmadd132pd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmadd213pd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmadd213pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmadd213pd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmadd213pd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmadd213pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmadd213pd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmadd231pd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmadd231pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmadd231pd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmadd231pd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmadd231pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmadd231pd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmadd132ps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmadd132ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmadd132ps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmadd132ps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmadd132ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmadd132ps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmadd213ps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmadd213ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmadd213ps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmadd213ps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmadd213ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmadd213ps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmadd231ps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmadd231ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmadd231ps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmadd231ps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmadd231ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmadd231ps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmadd132sd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmadd132sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmadd132sd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmadd213sd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmadd213sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmadd213sd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmadd231sd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmadd231sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmadd231sd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmadd132ss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmadd132ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmadd132ss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmadd213ss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmadd213ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmadd213ss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmadd231ss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmadd231ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmadd231ss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmaddsub132pd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmaddsub132pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmaddsub132pd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmaddsub132pd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmaddsub132pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmaddsub132pd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmaddsub213pd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmaddsub213pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmaddsub213pd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmaddsub213pd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmaddsub213pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmaddsub213pd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmaddsub231pd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmaddsub231pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmaddsub231pd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmaddsub231pd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmaddsub231pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmaddsub231pd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmaddsub132ps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmaddsub132ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmaddsub132ps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmaddsub132ps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmaddsub132ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmaddsub132ps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmaddsub213ps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmaddsub213ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmaddsub213ps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmaddsub213ps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmaddsub213ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmaddsub213ps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmaddsub231ps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmaddsub231ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmaddsub231ps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmaddsub231ps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmaddsub231ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmaddsub231ps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmsub132pd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmsub132pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmsub132pd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmsub132pd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmsub132pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmsub132pd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmsub213pd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmsub213pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmsub213pd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmsub213pd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmsub213pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmsub213pd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmsub231pd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmsub231pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmsub231pd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmsub231pd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmsub231pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmsub231pd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmsub132ps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmsub132ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmsub132ps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmsub132ps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmsub132ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmsub132ps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmsub213ps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmsub213ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmsub213ps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmsub213ps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmsub213ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmsub213ps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmsub231ps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmsub231ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmsub231ps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmsub231ps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmsub231ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmsub231ps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmsub132sd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmsub132sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmsub132sd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmsub213sd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmsub213sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmsub213sd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmsub231sd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmsub231sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmsub231sd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmsub132ss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmsub132ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmsub132ss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmsub213ss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmsub213ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmsub213ss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmsub231ss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmsub231ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmsub231ss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmsubadd132pd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmsubadd132pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmsubadd132pd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmsubadd132pd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmsubadd132pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmsubadd132pd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmsubadd213pd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmsubadd213pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmsubadd213pd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmsubadd213pd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmsubadd213pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmsubadd213pd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmsubadd231pd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmsubadd231pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmsubadd231pd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmsubadd231pd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmsubadd231pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmsubadd231pd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmsubadd132ps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmsubadd132ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmsubadd132ps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmsubadd132ps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmsubadd132ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmsubadd132ps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmsubadd213ps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmsubadd213ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmsubadd213ps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmsubadd213ps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmsubadd213ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmsubadd213ps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfmsubadd231ps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfmsubadd231ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfmsubadd231ps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfmsubadd231ps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfmsubadd231ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfmsubadd231ps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfnmadd132pd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmadd132pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmadd132pd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmadd132pd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfnmadd132pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfnmadd132pd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfnmadd213pd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmadd213pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmadd213pd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmadd213pd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfnmadd213pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfnmadd213pd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfnmadd231pd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmadd231pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmadd231pd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmadd231pd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfnmadd231pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfnmadd231pd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfnmadd132ps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmadd132ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmadd132ps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmadd132ps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfnmadd132ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfnmadd132ps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfnmadd213ps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmadd213ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmadd213ps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmadd213ps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfnmadd213ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfnmadd213ps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfnmadd231ps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmadd231ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmadd231ps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmadd231ps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfnmadd231ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfnmadd231ps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfnmadd132sd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmadd132sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmadd132sd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmadd213sd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmadd213sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmadd213sd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmadd231sd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmadd231sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmadd231sd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmadd132ss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmadd132ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmadd132ss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmadd213ss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmadd213ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmadd213ss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmadd231ss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmadd231ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmadd231ss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmsub132pd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmsub132pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmsub132pd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmsub132pd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfnmsub132pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfnmsub132pd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfnmsub213pd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmsub213pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmsub213pd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmsub213pd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfnmsub213pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfnmsub213pd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfnmsub231pd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmsub231pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmsub231pd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmsub231pd	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfnmsub231pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfnmsub231pd	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfnmsub132ps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmsub132ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmsub132ps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmsub132ps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfnmsub132ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfnmsub132ps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfnmsub213ps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmsub213ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmsub213ps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmsub213ps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfnmsub213ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfnmsub213ps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfnmsub231ps	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmsub231ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmsub231ps	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmsub231ps	%ymm0, %ymm1, %ymm2
-# CHECK-NEXT:  2      11    0.50    *                   vfnmsub231ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      12    0.50    *                   vfnmsub231ps	(%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      4     0.50                        vfnmsub132sd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmsub132sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmsub132sd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmsub213sd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmsub213sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmsub213sd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmsub231sd	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmsub231sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmsub231sd	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmsub132ss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmsub132ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmsub132ss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmsub213ss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmsub213ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmsub213ss	(%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      4     0.50                        vfnmsub231ss	%xmm0, %xmm1, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   vfnmsub231ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   vfnmsub231ss	(%rax), %xmm1, %xmm2
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - ADLPPort00
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-lea.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-lea.s
index 3238a570f3728..762b6d3caef2b 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-lea.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-lea.s
@@ -148,141 +148,141 @@ lea 1024(%rax, %rbx, 2), %rcx
 # CHECK-NEXT: [6]: HasSideEffects (U)
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
-# CHECK-NEXT:  2      2     0.40                        leaw	0, %cx
-# CHECK-NEXT:  1      1     0.20                        leal	0, %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	0, %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	(%eax), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	(%eax), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	(%eax), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	(%rax), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	(%rax), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	(%rax), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	(,%ebx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	(,%ebx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	(,%ebx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	(,%rbx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	(,%rbx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	(,%rbx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	(,%ebx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	(,%ebx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	(,%ebx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	(,%rbx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	(,%rbx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	(,%rbx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	(,%ebx,2), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	(,%ebx,2), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	(,%ebx,2), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	(,%rbx,2), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	(,%rbx,2), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	(,%rbx,2), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	(%eax,%ebx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	(%eax,%ebx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	(%eax,%ebx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	(%rax,%rbx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	(%rax,%rbx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	(%rax,%rbx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	(%eax,%ebx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	(%eax,%ebx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	(%eax,%ebx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	(%rax,%rbx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	(%rax,%rbx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	(%rax,%rbx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	(%eax,%ebx,2), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	(%eax,%ebx,2), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	(%eax,%ebx,2), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	(%rax,%rbx,2), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	(%rax,%rbx,2), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	(%rax,%rbx,2), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	-16, %cx
-# CHECK-NEXT:  1      1     0.20                        leal	-16, %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	-16, %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	-16(%eax), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	-16(%eax), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	-16(%eax), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	-16(%rax), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	-16(%rax), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	-16(%rax), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	-16(,%ebx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	-16(,%ebx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	-16(,%ebx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	-16(,%rbx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	-16(,%rbx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	-16(,%rbx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	-16(,%ebx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	-16(,%ebx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	-16(,%ebx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	-16(,%rbx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	-16(,%rbx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	-16(,%rbx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	-16(,%ebx,2), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	-16(,%ebx,2), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	-16(,%ebx,2), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	-16(,%rbx,2), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	-16(,%rbx,2), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	-16(,%rbx,2), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	-16(%eax,%ebx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	-16(%eax,%ebx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	-16(%eax,%ebx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	-16(%rax,%rbx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	-16(%rax,%rbx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	-16(%rax,%rbx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	-16(%eax,%ebx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	-16(%eax,%ebx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	-16(%eax,%ebx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	-16(%rax,%rbx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	-16(%rax,%rbx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	-16(%rax,%rbx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	-16(%eax,%ebx,2), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	-16(%eax,%ebx,2), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	-16(%eax,%ebx,2), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	-16(%rax,%rbx,2), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	-16(%rax,%rbx,2), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	-16(%rax,%rbx,2), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	1024, %cx
-# CHECK-NEXT:  1      1     0.20                        leal	1024, %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	1024, %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	1024(%eax), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	1024(%eax), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	1024(%eax), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	1024(%rax), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	1024(%rax), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	1024(%rax), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	1024(,%ebx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	1024(,%ebx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	1024(,%ebx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	1024(,%rbx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	1024(,%rbx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	1024(,%rbx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	1024(,%ebx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	1024(,%ebx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	1024(,%ebx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	1024(,%rbx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	1024(,%rbx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	1024(,%rbx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	1024(,%ebx,2), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	1024(,%ebx,2), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	1024(,%ebx,2), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	1024(,%rbx,2), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	1024(,%rbx,2), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	1024(,%rbx,2), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	1024(%eax,%ebx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	1024(%eax,%ebx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	1024(%eax,%ebx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	1024(%rax,%rbx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	1024(%rax,%rbx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	1024(%rax,%rbx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	1024(%eax,%ebx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	1024(%eax,%ebx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	1024(%eax,%ebx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	1024(%rax,%rbx), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	1024(%rax,%rbx), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	1024(%rax,%rbx), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	1024(%eax,%ebx,2), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	1024(%eax,%ebx,2), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	1024(%eax,%ebx,2), %rcx
-# CHECK-NEXT:  2      2     0.40                        leaw	1024(%rax,%rbx,2), %cx
-# CHECK-NEXT:  1      1     0.20                        leal	1024(%rax,%rbx,2), %ecx
-# CHECK-NEXT:  1      1     0.20                        leaq	1024(%rax,%rbx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	0, %cx
+# CHECK-NEXT:  1      1     1.00                        leal	0, %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	0, %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(%eax), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	(%eax), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	(%eax), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(%rax), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	(%rax), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	(%rax), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	(,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	(,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	(,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	(,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	(,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	(,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	(,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	(,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(,%ebx,2), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	(,%ebx,2), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	(,%ebx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(,%rbx,2), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	(,%rbx,2), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	(,%rbx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(%eax,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	(%eax,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	(%eax,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(%rax,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	(%rax,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	(%rax,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(%eax,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	(%eax,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	(%eax,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(%rax,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	(%rax,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	(%rax,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(%eax,%ebx,2), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	(%eax,%ebx,2), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	(%eax,%ebx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(%rax,%rbx,2), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	(%rax,%rbx,2), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	(%rax,%rbx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16, %cx
+# CHECK-NEXT:  1      1     1.00                        leal	-16, %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	-16, %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(%eax), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	-16(%eax), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	-16(%eax), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(%rax), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	-16(%rax), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	-16(%rax), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	-16(,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	-16(,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	-16(,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	-16(,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	-16(,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	-16(,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	-16(,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	-16(,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(,%ebx,2), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	-16(,%ebx,2), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	-16(,%ebx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(,%rbx,2), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	-16(,%rbx,2), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	-16(,%rbx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(%eax,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	-16(%eax,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	-16(%eax,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(%rax,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	-16(%rax,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	-16(%rax,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(%eax,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	-16(%eax,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	-16(%eax,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(%rax,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	-16(%rax,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	-16(%rax,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(%eax,%ebx,2), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	-16(%eax,%ebx,2), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	-16(%eax,%ebx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(%rax,%rbx,2), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	-16(%rax,%rbx,2), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	-16(%rax,%rbx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024, %cx
+# CHECK-NEXT:  1      1     1.00                        leal	1024, %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	1024, %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(%eax), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	1024(%eax), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	1024(%eax), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(%rax), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	1024(%rax), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	1024(%rax), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	1024(,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	1024(,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	1024(,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	1024(,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	1024(,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	1024(,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	1024(,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	1024(,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(,%ebx,2), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	1024(,%ebx,2), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	1024(,%ebx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(,%rbx,2), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	1024(,%rbx,2), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	1024(,%rbx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(%eax,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	1024(%eax,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	1024(%eax,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(%rax,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	1024(%rax,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	1024(%rax,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(%eax,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	1024(%eax,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	1024(%eax,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(%rax,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	1024(%rax,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	1024(%rax,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(%eax,%ebx,2), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	1024(%eax,%ebx,2), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	1024(%eax,%ebx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(%rax,%rbx,2), %cx
+# CHECK-NEXT:  1      1     1.00                        leal	1024(%rax,%rbx,2), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq	1024(%rax,%rbx,2), %rcx
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - ADLPPort00
@@ -301,142 +301,142 @@ lea 1024(%rax, %rbx, 2), %rcx
 
 # CHECK:      Resource pressure per iteration:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]
-# CHECK-NEXT: 36.00  36.00   -      -      -     36.00  36.00   -      -      -     36.00   -      -
+# CHECK-NEXT: 9.00   144.00  -      -      -     9.00   9.00    -      -      -     9.00    -      -
 
 # CHECK:      Resource pressure by instruction:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]   Instructions:
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	0, %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	0, %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	0, %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	(%eax), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	(%eax), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	(%eax), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	(%rax), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	(%rax), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	(%rax), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	(,%ebx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	(,%ebx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	(,%ebx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	(,%rbx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	(,%rbx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	(,%rbx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	(,%ebx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	(,%ebx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	(,%ebx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	(,%rbx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	(,%rbx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	(,%rbx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	(,%ebx,2), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	(,%ebx,2), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	(,%ebx,2), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	(,%rbx,2), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	(,%rbx,2), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	(,%rbx,2), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	(%eax,%ebx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	(%eax,%ebx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	(%eax,%ebx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	(%rax,%rbx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	(%rax,%rbx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	(%rax,%rbx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	(%eax,%ebx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	(%eax,%ebx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	(%eax,%ebx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	(%rax,%rbx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	(%rax,%rbx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	(%rax,%rbx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	(%eax,%ebx,2), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	(%eax,%ebx,2), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	(%eax,%ebx,2), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	(%rax,%rbx,2), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	(%rax,%rbx,2), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	(%rax,%rbx,2), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	-16, %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	-16, %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	-16, %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	-16(%eax), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	-16(%eax), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	-16(%eax), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	-16(%rax), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	-16(%rax), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	-16(%rax), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	-16(,%ebx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	-16(,%ebx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	-16(,%ebx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	-16(,%rbx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	-16(,%rbx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	-16(,%rbx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	-16(,%ebx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	-16(,%ebx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	-16(,%ebx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	-16(,%rbx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	-16(,%rbx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	-16(,%rbx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	-16(,%ebx,2), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	-16(,%ebx,2), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	-16(,%ebx,2), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	-16(,%rbx,2), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	-16(,%rbx,2), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	-16(,%rbx,2), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	-16(%eax,%ebx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	-16(%eax,%ebx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	-16(%eax,%ebx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	-16(%rax,%rbx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	-16(%rax,%rbx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	-16(%rax,%rbx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	-16(%eax,%ebx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	-16(%eax,%ebx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	-16(%eax,%ebx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	-16(%rax,%rbx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	-16(%rax,%rbx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	-16(%rax,%rbx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	-16(%eax,%ebx,2), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	-16(%eax,%ebx,2), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	-16(%eax,%ebx,2), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	-16(%rax,%rbx,2), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	-16(%rax,%rbx,2), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	-16(%rax,%rbx,2), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	1024, %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	1024, %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	1024, %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	1024(%eax), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	1024(%eax), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	1024(%eax), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	1024(%rax), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	1024(%rax), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	1024(%rax), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	1024(,%ebx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	1024(,%ebx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	1024(,%ebx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	1024(,%rbx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	1024(,%rbx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	1024(,%rbx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	1024(,%ebx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	1024(,%ebx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	1024(,%ebx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	1024(,%rbx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	1024(,%rbx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	1024(,%rbx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	1024(,%ebx,2), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	1024(,%ebx,2), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	1024(,%ebx,2), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	1024(,%rbx,2), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	1024(,%rbx,2), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	1024(,%rbx,2), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	1024(%eax,%ebx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	1024(%eax,%ebx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	1024(%eax,%ebx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	1024(%rax,%rbx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	1024(%rax,%rbx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	1024(%rax,%rbx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	1024(%eax,%ebx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	1024(%eax,%ebx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	1024(%eax,%ebx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	1024(%rax,%rbx), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	1024(%rax,%rbx), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	1024(%rax,%rbx), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	1024(%eax,%ebx,2), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	1024(%eax,%ebx,2), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	1024(%eax,%ebx,2), %rcx
-# CHECK-NEXT: 0.40   0.40    -      -      -     0.40   0.40    -      -      -     0.40    -      -     leaw	1024(%rax,%rbx,2), %cx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leal	1024(%rax,%rbx,2), %ecx
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaq	1024(%rax,%rbx,2), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	0, %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	0, %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	0, %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	(%eax), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	(%eax), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	(%eax), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	(%rax), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	(%rax), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	(%rax), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	(,%ebx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	(,%ebx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	(,%ebx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	(,%rbx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	(,%rbx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	(,%rbx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	(,%ebx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	(,%ebx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	(,%ebx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	(,%rbx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	(,%rbx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	(,%rbx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	(,%ebx,2), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	(,%ebx,2), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	(,%ebx,2), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	(,%rbx,2), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	(,%rbx,2), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	(,%rbx,2), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	(%eax,%ebx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	(%eax,%ebx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	(%eax,%ebx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	(%rax,%rbx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	(%rax,%rbx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	(%rax,%rbx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	(%eax,%ebx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	(%eax,%ebx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	(%eax,%ebx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	(%rax,%rbx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	(%rax,%rbx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	(%rax,%rbx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	(%eax,%ebx,2), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	(%eax,%ebx,2), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	(%eax,%ebx,2), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	(%rax,%rbx,2), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	(%rax,%rbx,2), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	(%rax,%rbx,2), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	-16, %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	-16, %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	-16, %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	-16(%eax), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	-16(%eax), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	-16(%eax), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	-16(%rax), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	-16(%rax), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	-16(%rax), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	-16(,%ebx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	-16(,%ebx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	-16(,%ebx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	-16(,%rbx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	-16(,%rbx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	-16(,%rbx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	-16(,%ebx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	-16(,%ebx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	-16(,%ebx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	-16(,%rbx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	-16(,%rbx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	-16(,%rbx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	-16(,%ebx,2), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	-16(,%ebx,2), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	-16(,%ebx,2), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	-16(,%rbx,2), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	-16(,%rbx,2), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	-16(,%rbx,2), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	-16(%eax,%ebx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	-16(%eax,%ebx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	-16(%eax,%ebx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	-16(%rax,%rbx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	-16(%rax,%rbx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	-16(%rax,%rbx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	-16(%eax,%ebx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	-16(%eax,%ebx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	-16(%eax,%ebx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	-16(%rax,%rbx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	-16(%rax,%rbx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	-16(%rax,%rbx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	-16(%eax,%ebx,2), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	-16(%eax,%ebx,2), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	-16(%eax,%ebx,2), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	-16(%rax,%rbx,2), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	-16(%rax,%rbx,2), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	-16(%rax,%rbx,2), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	1024, %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	1024, %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	1024, %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	1024(%eax), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	1024(%eax), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	1024(%eax), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	1024(%rax), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	1024(%rax), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	1024(%rax), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	1024(,%ebx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	1024(,%ebx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	1024(,%ebx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	1024(,%rbx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	1024(,%rbx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	1024(,%rbx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	1024(,%ebx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	1024(,%ebx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	1024(,%ebx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	1024(,%rbx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	1024(,%rbx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	1024(,%rbx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	1024(,%ebx,2), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	1024(,%ebx,2), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	1024(,%ebx,2), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	1024(,%rbx,2), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	1024(,%rbx,2), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	1024(,%rbx,2), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	1024(%eax,%ebx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	1024(%eax,%ebx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	1024(%eax,%ebx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	1024(%rax,%rbx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	1024(%rax,%rbx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	1024(%rax,%rbx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	1024(%eax,%ebx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	1024(%eax,%ebx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	1024(%eax,%ebx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	1024(%rax,%rbx), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	1024(%rax,%rbx), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	1024(%rax,%rbx), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	1024(%eax,%ebx,2), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	1024(%eax,%ebx,2), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	1024(%eax,%ebx,2), %rcx
+# CHECK-NEXT: 0.20   1.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     leaw	1024(%rax,%rbx,2), %cx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leal	1024(%rax,%rbx,2), %ecx
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     leaq	1024(%rax,%rbx,2), %rcx
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-mmx.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-mmx.s
index 6891842939363..c62ea2963323d 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-mmx.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-mmx.s
@@ -165,50 +165,50 @@ pxor        (%rax), %mm2
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
 # CHECK-NEXT:  10     10    4.50    *      *      U     emms
-# CHECK-NEXT:  1      1     1.00                        movd	%eax, %mm2
-# CHECK-NEXT:  1      6     0.33    *                   movd	(%rax), %mm2
+# CHECK-NEXT:  1      3     1.00                        movd	%eax, %mm2
+# CHECK-NEXT:  1      8     0.33    *                   movd	(%rax), %mm2
 # CHECK-NEXT:  1      3     1.00                        movd	%mm0, %ecx
-# CHECK-NEXT:  2      1     0.50           *      U     movd	%mm0, (%rax)
-# CHECK-NEXT:  1      1     1.00                        movq	%rax, %mm2
-# CHECK-NEXT:  1      6     0.33    *                   movq	(%rax), %mm2
+# CHECK-NEXT:  2      18    0.50           *      U     movd	%mm0, (%rax)
+# CHECK-NEXT:  1      3     1.00                        movq	%rax, %mm2
+# CHECK-NEXT:  1      8     0.33    *                   movq	(%rax), %mm2
 # CHECK-NEXT:  1      3     1.00                        movq	%mm0, %rcx
-# CHECK-NEXT:  2      1     0.50           *            movq	%mm0, (%rax)
-# CHECK-NEXT:  3      4     2.00                        packsswb	%mm0, %mm2
-# CHECK-NEXT:  3      10    2.00    *                   packsswb	(%rax), %mm2
-# CHECK-NEXT:  3      4     2.00                        packssdw	%mm0, %mm2
-# CHECK-NEXT:  3      10    2.00    *                   packssdw	(%rax), %mm2
-# CHECK-NEXT:  3      4     2.00                        packuswb	%mm0, %mm2
-# CHECK-NEXT:  3      10    2.00    *                   packuswb	(%rax), %mm2
+# CHECK-NEXT:  2      12    0.50           *            movq	%mm0, (%rax)
+# CHECK-NEXT:  2      4     2.00                        packsswb	%mm0, %mm2
+# CHECK-NEXT:  3      12    2.00    *                   packsswb	(%rax), %mm2
+# CHECK-NEXT:  2      4     2.00                        packssdw	%mm0, %mm2
+# CHECK-NEXT:  3      12    2.00    *                   packssdw	(%rax), %mm2
+# CHECK-NEXT:  2      4     2.00                        packuswb	%mm0, %mm2
+# CHECK-NEXT:  3      12    2.00    *                   packuswb	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.50                        paddb	%mm0, %mm2
-# CHECK-NEXT:  2      7     0.50    *                   paddb	(%rax), %mm2
+# CHECK-NEXT:  2      9     0.50    *                   paddb	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.50                        paddd	%mm0, %mm2
-# CHECK-NEXT:  2      7     0.50    *                   paddd	(%rax), %mm2
+# CHECK-NEXT:  2      9     0.50    *                   paddd	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        paddsb	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   paddsb	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   paddsb	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        paddsw	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   paddsw	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   paddsw	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        paddusb	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   paddusb	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   paddusb	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        paddusw	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   paddusw	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   paddusw	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.50                        paddw	%mm0, %mm2
-# CHECK-NEXT:  2      7     0.50    *                   paddw	(%rax), %mm2
+# CHECK-NEXT:  2      9     0.50    *                   paddw	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.50                        pand	%mm0, %mm2
-# CHECK-NEXT:  2      7     0.50    *                   pand	(%rax), %mm2
+# CHECK-NEXT:  2      9     0.50    *                   pand	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.50                        pandn	%mm0, %mm2
-# CHECK-NEXT:  2      7     0.50    *                   pandn	(%rax), %mm2
+# CHECK-NEXT:  2      9     0.50    *                   pandn	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        pcmpeqb	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   pcmpeqb	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   pcmpeqb	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        pcmpeqd	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   pcmpeqd	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   pcmpeqd	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        pcmpeqw	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   pcmpeqw	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   pcmpeqw	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        pcmpgtb	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   pcmpgtb	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   pcmpgtb	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        pcmpgtd	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   pcmpgtd	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   pcmpgtd	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        pcmpgtw	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   pcmpgtw	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   pcmpgtw	(%rax), %mm2
 # CHECK-NEXT:  1      5     1.00                        pmaddwd	%mm0, %mm2
 # CHECK-NEXT:  2      13    1.00    *                   pmaddwd	(%rax), %mm2
 # CHECK-NEXT:  1      5     1.00                        pmulhw	%mm0, %mm2
@@ -216,59 +216,59 @@ pxor        (%rax), %mm2
 # CHECK-NEXT:  1      5     1.00                        pmullw	%mm0, %mm2
 # CHECK-NEXT:  2      13    1.00    *                   pmullw	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.50                        por	%mm0, %mm2
-# CHECK-NEXT:  2      7     0.50    *                   por	(%rax), %mm2
+# CHECK-NEXT:  2      9     0.50    *                   por	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        pslld	$1, %mm2
 # CHECK-NEXT:  1      1     1.00                        pslld	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   pslld	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   pslld	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        psllq	$1, %mm2
 # CHECK-NEXT:  1      1     1.00                        psllq	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   psllq	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   psllq	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        psllw	$1, %mm2
 # CHECK-NEXT:  1      1     1.00                        psllw	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   psllw	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   psllw	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        psrad	$1, %mm2
 # CHECK-NEXT:  1      1     1.00                        psrad	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   psrad	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   psrad	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        psraw	$1, %mm2
 # CHECK-NEXT:  1      1     1.00                        psraw	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   psraw	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   psraw	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        psrld	$1, %mm2
 # CHECK-NEXT:  1      1     1.00                        psrld	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   psrld	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   psrld	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        psrlq	$1, %mm2
 # CHECK-NEXT:  1      1     1.00                        psrlq	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   psrlq	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   psrlq	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        psrlw	$1, %mm2
 # CHECK-NEXT:  1      1     1.00                        psrlw	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   psrlw	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   psrlw	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.50                        psubb	%mm0, %mm2
-# CHECK-NEXT:  2      7     0.50    *                   psubb	(%rax), %mm2
+# CHECK-NEXT:  2      9     0.50    *                   psubb	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.50                        psubd	%mm0, %mm2
-# CHECK-NEXT:  2      7     0.50    *                   psubd	(%rax), %mm2
+# CHECK-NEXT:  2      9     0.50    *                   psubd	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        psubsb	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   psubsb	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   psubsb	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        psubsw	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   psubsw	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   psubsw	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        psubusb	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   psubusb	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   psubusb	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        psubusw	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   psubusw	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   psubusw	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.50                        psubw	%mm0, %mm2
-# CHECK-NEXT:  2      7     0.50    *                   psubw	(%rax), %mm2
+# CHECK-NEXT:  2      9     0.50    *                   psubw	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        punpckhbw	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   punpckhbw	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   punpckhbw	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        punpckhdq	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   punpckhdq	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   punpckhdq	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        punpckhwd	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   punpckhwd	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   punpckhwd	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        punpcklbw	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   punpcklbw	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   punpcklbw	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        punpckldq	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   punpckldq	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   punpckldq	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        punpcklwd	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   punpcklwd	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   punpcklwd	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.50                        pxor	%mm0, %mm2
-# CHECK-NEXT:  2      7     0.50    *                   pxor	(%rax), %mm2
+# CHECK-NEXT:  2      9     0.50    *                   pxor	(%rax), %mm2
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - ADLPPort00
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-pclmul.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-pclmul.s
index e79f9b7c0e645..871035dbe34a6 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-pclmul.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-pclmul.s
@@ -14,7 +14,7 @@ pclmulqdq     $11, (%rax), %xmm2
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
 # CHECK-NEXT:  1      3     1.00                        pclmulqdq	$11, %xmm0, %xmm2
-# CHECK-NEXT:  2      9     1.00    *                   pclmulqdq	$11, (%rax), %xmm2
+# CHECK-NEXT:  2      10    1.00    *                   pclmulqdq	$11, (%rax), %xmm2
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - ADLPPort00
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse1.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse1.s
index 8ca990b185a17..b292826913d91 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse1.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse1.s
@@ -196,89 +196,89 @@ xorps       (%rax), %xmm2
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
 # CHECK-NEXT:  1      3     0.50                        addps	%xmm0, %xmm2
 # CHECK-NEXT:  2      10    0.50    *                   addps	(%rax), %xmm2
-# CHECK-NEXT:  1      2     0.50                        addss	%xmm0, %xmm2
-# CHECK-NEXT:  2      9     0.50    *                   addss	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        addss	%xmm0, %xmm2
+# CHECK-NEXT:  2      10    0.50    *                   addss	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        andnps	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   andnps	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   andnps	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        andps	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   andps	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   andps	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        cmpeqps	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   cmpeqps	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   cmpeqps	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        cmpeqss	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   cmpeqss	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   cmpeqss	(%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        comiss	%xmm0, %xmm1
-# CHECK-NEXT:  2      9     1.00    *                   comiss	(%rax), %xmm1
-# CHECK-NEXT:  2      6     1.00                        cvtpi2ps	%mm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   cvtpi2ps	(%rax), %xmm2
+# CHECK-NEXT:  2      8     1.00    *                   comiss	(%rax), %xmm1
+# CHECK-NEXT:  2      7     1.00                        cvtpi2ps	%mm0, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   cvtpi2ps	(%rax), %xmm2
 # CHECK-NEXT:  2      9     1.00                        cvtps2pi	%xmm0, %mm2
-# CHECK-NEXT:  2      11    1.00    *                   cvtps2pi	(%rax), %mm2
-# CHECK-NEXT:  2      5     1.00                        cvtsi2ss	%ecx, %xmm2
-# CHECK-NEXT:  3      6     2.00                        cvtsi2ss	%rcx, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   cvtsi2ssl	(%rax), %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   cvtsi2ssl	(%rax), %xmm2
+# CHECK-NEXT:  2      13    1.00    *                   cvtps2pi	(%rax), %mm2
+# CHECK-NEXT:  2      7     1.00                        cvtsi2ss	%ecx, %xmm2
+# CHECK-NEXT:  3      8     2.00                        cvtsi2ss	%rcx, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   cvtsi2ssl	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   cvtsi2ssl	(%rax), %xmm2
 # CHECK-NEXT:  2      7     1.00                        cvtss2si	%xmm0, %ecx
 # CHECK-NEXT:  3      8     1.00                        cvtss2si	%xmm0, %rcx
 # CHECK-NEXT:  3      12    1.00    *                   cvtss2si	(%rax), %ecx
 # CHECK-NEXT:  3      12    1.00    *                   cvtss2si	(%rax), %rcx
 # CHECK-NEXT:  2      9     1.00                        cvttps2pi	%xmm0, %mm2
-# CHECK-NEXT:  2      11    1.00    *                   cvttps2pi	(%rax), %mm2
+# CHECK-NEXT:  2      13    1.00    *                   cvttps2pi	(%rax), %mm2
 # CHECK-NEXT:  2      7     1.00                        cvttss2si	%xmm0, %ecx
 # CHECK-NEXT:  3      8     1.00                        cvttss2si	%xmm0, %rcx
 # CHECK-NEXT:  3      12    1.00    *                   cvttss2si	(%rax), %ecx
 # CHECK-NEXT:  3      12    1.00    *                   cvttss2si	(%rax), %rcx
 # CHECK-NEXT:  1      11    1.00                        divps	%xmm0, %xmm2
-# CHECK-NEXT:  2      17    1.00    *                   divps	(%rax), %xmm2
+# CHECK-NEXT:  2      18    1.00    *                   divps	(%rax), %xmm2
 # CHECK-NEXT:  1      11    1.00                        divss	%xmm0, %xmm2
-# CHECK-NEXT:  2      17    1.00    *                   divss	(%rax), %xmm2
+# CHECK-NEXT:  2      18    1.00    *                   divss	(%rax), %xmm2
 # CHECK-NEXT:  4      7     1.00    *      *      U     ldmxcsr	(%rax)
 # CHECK-NEXT:  4      12    2.00    *      *      U     maskmovq	%mm0, %mm1
 # CHECK-NEXT:  1      4     0.50                        maxps	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   maxps	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   maxps	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        maxss	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   maxss	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   maxss	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        minps	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   minps	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   minps	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        minss	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   minss	(%rax), %xmm2
-# CHECK-NEXT:  1      0     0.33                        movaps	%xmm0, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            movaps	%xmm0, (%rax)
-# CHECK-NEXT:  1      6     0.33    *                   movaps	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   minss	(%rax), %xmm2
+# CHECK-NEXT:  0      1     0.00                        movaps	%xmm0, %xmm2
+# CHECK-NEXT:  2      12    0.50           *            movaps	%xmm0, (%rax)
+# CHECK-NEXT:  1      7     0.33    *                   movaps	(%rax), %xmm2
 # CHECK-NEXT:  1      1     1.00                        movhlps	%xmm0, %xmm2
 # CHECK-NEXT:  1      1     1.00                        movlhps	%xmm0, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            movhps	%xmm0, (%rax)
-# CHECK-NEXT:  2      7     1.00    *                   movhps	(%rax), %xmm2
+# CHECK-NEXT:  2      12    0.50           *            movhps	%xmm0, (%rax)
+# CHECK-NEXT:  2      8     1.00    *                   movhps	(%rax), %xmm2
 # CHECK-NEXT:  2      12    0.50           *            movlps	%xmm0, (%rax)
-# CHECK-NEXT:  2      7     0.50    *                   movlps	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   movlps	(%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        movmskps	%xmm0, %ecx
 # CHECK-NEXT:  2      518   0.50           *            movntps	%xmm0, (%rax)
 # CHECK-NEXT:  2      511   0.50    *      *      U     movntq	%mm0, (%rax)
 # CHECK-NEXT:  1      1     0.33                        movss	%xmm0, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            movss	%xmm0, (%rax)
-# CHECK-NEXT:  1      6     0.33    *                   movss	(%rax), %xmm2
-# CHECK-NEXT:  1      0     0.33                        movups	%xmm0, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            movups	%xmm0, (%rax)
-# CHECK-NEXT:  1      6     0.33    *                   movups	(%rax), %xmm2
+# CHECK-NEXT:  2      12    0.50           *            movss	%xmm0, (%rax)
+# CHECK-NEXT:  1      7     0.33    *                   movss	(%rax), %xmm2
+# CHECK-NEXT:  0      1     0.00                        movups	%xmm0, %xmm2
+# CHECK-NEXT:  2      12    0.50           *            movups	%xmm0, (%rax)
+# CHECK-NEXT:  1      7     0.33    *                   movups	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        mulps	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   mulps	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   mulps	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        mulss	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   mulss	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   mulss	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        orps	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   orps	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   orps	(%rax), %xmm2
 # CHECK-NEXT:  1      1     1.00                        pavgb	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   pavgb	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   pavgb	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        pavgw	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   pavgw	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   pavgw	(%rax), %mm2
 # CHECK-NEXT:  2      4     1.00                        pextrw	$1, %mm0, %ecx
-# CHECK-NEXT:  2      2     2.00                        pinsrw	$1, %eax, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   pinsrw	$1, (%rax), %mm2
+# CHECK-NEXT:  2      4     2.00                        pinsrw	$1, %eax, %mm2
+# CHECK-NEXT:  2      9     1.00    *                   pinsrw	$1, (%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        pmaxsw	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   pmaxsw	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   pmaxsw	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        pmaxub	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   pmaxub	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   pmaxub	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        pminsw	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   pminsw	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   pminsw	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        pminub	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   pminub	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   pminub	(%rax), %mm2
 # CHECK-NEXT:  1      3     1.00                        pmovmskb	%mm0, %ecx
 # CHECK-NEXT:  1      5     1.00                        pmulhuw	%mm0, %mm2
 # CHECK-NEXT:  2      13    1.00    *                   pmulhuw	(%rax), %mm2
@@ -287,37 +287,37 @@ xorps       (%rax), %xmm2
 # CHECK-NEXT:  1      0     0.33    *      *            prefetcht2	(%rax)
 # CHECK-NEXT:  1      0     0.33    *      *            prefetchnta	(%rax)
 # CHECK-NEXT:  1      3     1.00                        psadbw	%mm0, %mm2
-# CHECK-NEXT:  2      9     1.00    *                   psadbw	(%rax), %mm2
+# CHECK-NEXT:  2      11    1.00    *                   psadbw	(%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        pshufw	$1, %mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   pshufw	$1, (%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   pshufw	$1, (%rax), %mm2
 # CHECK-NEXT:  1      4     1.00                        rcpps	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    1.00    *                   rcpps	(%rax), %xmm2
+# CHECK-NEXT:  2      11    1.00    *                   rcpps	(%rax), %xmm2
 # CHECK-NEXT:  1      4     1.00                        rcpss	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    1.00    *                   rcpss	(%rax), %xmm2
+# CHECK-NEXT:  2      11    1.00    *                   rcpss	(%rax), %xmm2
 # CHECK-NEXT:  1      4     1.00                        rsqrtps	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    1.00    *                   rsqrtps	(%rax), %xmm2
+# CHECK-NEXT:  2      11    1.00    *                   rsqrtps	(%rax), %xmm2
 # CHECK-NEXT:  1      4     1.00                        rsqrtss	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    1.00    *                   rsqrtss	(%rax), %xmm2
+# CHECK-NEXT:  2      11    1.00    *                   rsqrtss	(%rax), %xmm2
 # CHECK-NEXT:  2      2     0.50    *      *      U     sfence
 # CHECK-NEXT:  1      1     0.50                        shufps	$1, %xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   shufps	$1, (%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   shufps	$1, (%rax), %xmm2
 # CHECK-NEXT:  1      12    1.00                        sqrtps	%xmm0, %xmm2
-# CHECK-NEXT:  2      18    1.00    *                   sqrtps	(%rax), %xmm2
+# CHECK-NEXT:  2      19    1.00    *                   sqrtps	(%rax), %xmm2
 # CHECK-NEXT:  1      12    1.00                        sqrtss	%xmm0, %xmm2
-# CHECK-NEXT:  2      18    1.00    *                   sqrtss	(%rax), %xmm2
+# CHECK-NEXT:  2      19    1.00    *                   sqrtss	(%rax), %xmm2
 # CHECK-NEXT:  4      12    1.00           *      U     stmxcsr	(%rax)
 # CHECK-NEXT:  1      3     0.50                        subps	%xmm0, %xmm2
 # CHECK-NEXT:  2      10    0.50    *                   subps	(%rax), %xmm2
-# CHECK-NEXT:  1      2     0.50                        subss	%xmm0, %xmm2
-# CHECK-NEXT:  2      9     0.50    *                   subss	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        subss	%xmm0, %xmm2
+# CHECK-NEXT:  2      10    0.50    *                   subss	(%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        ucomiss	%xmm0, %xmm1
-# CHECK-NEXT:  2      9     1.00    *                   ucomiss	(%rax), %xmm1
+# CHECK-NEXT:  2      8     1.00    *                   ucomiss	(%rax), %xmm1
 # CHECK-NEXT:  1      1     1.00                        unpckhps	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     1.00    *                   unpckhps	(%rax), %xmm2
+# CHECK-NEXT:  2      8     1.00    *                   unpckhps	(%rax), %xmm2
 # CHECK-NEXT:  1      1     1.00                        unpcklps	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     1.00    *                   unpcklps	(%rax), %xmm2
+# CHECK-NEXT:  2      8     1.00    *                   unpcklps	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        xorps	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   xorps	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   xorps	(%rax), %xmm2
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - ADLPPort00
@@ -336,7 +336,7 @@ xorps       (%rax), %xmm2
 
 # CHECK:      Resource pressure per iteration:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]
-# CHECK-NEXT: 73.67  25.17  19.33  19.33  5.00   30.17  2.00   5.00   5.00   5.00    -     19.33   -
+# CHECK-NEXT: 74.00  24.50  19.33  19.33  5.00   29.50  1.00   5.00   5.00   5.00    -     19.33   -
 
 # CHECK:      Resource pressure by instruction:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]   Instructions:
@@ -376,7 +376,7 @@ xorps       (%rax), %xmm2
 # CHECK-NEXT: 1.00    -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     divps	(%rax), %xmm2
 # CHECK-NEXT: 1.00    -      -      -      -      -      -      -      -      -      -      -      -     divss	%xmm0, %xmm2
 # CHECK-NEXT: 1.00    -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     divss	(%rax), %xmm2
-# CHECK-NEXT: 1.33   0.33   0.33   0.33    -     0.33   1.00    -      -      -      -     0.33    -     ldmxcsr	(%rax)
+# CHECK-NEXT: 1.83   0.33   0.33   0.33    -     0.33   0.50    -      -      -      -     0.33    -     ldmxcsr	(%rax)
 # CHECK-NEXT: 2.00    -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     maskmovq	%mm0, %mm1
 # CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     maxps	%xmm0, %xmm2
 # CHECK-NEXT: 0.50   0.50   0.33   0.33    -      -      -      -      -      -      -     0.33    -     maxps	(%rax), %xmm2
@@ -386,7 +386,7 @@ xorps       (%rax), %xmm2
 # CHECK-NEXT: 0.50   0.50   0.33   0.33    -      -      -      -      -      -      -     0.33    -     minps	(%rax), %xmm2
 # CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     minss	%xmm0, %xmm2
 # CHECK-NEXT: 0.50   0.50   0.33   0.33    -      -      -      -      -      -      -     0.33    -     minss	(%rax), %xmm2
-# CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     movaps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     movaps	%xmm0, %xmm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     movaps	%xmm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     movaps	(%rax), %xmm2
 # CHECK-NEXT:  -      -      -      -      -     1.00    -      -      -      -      -      -      -     movhlps	%xmm0, %xmm2
@@ -401,7 +401,7 @@ xorps       (%rax), %xmm2
 # CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     movss	%xmm0, %xmm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     movss	%xmm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     movss	(%rax), %xmm2
-# CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     movups	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     movups	%xmm0, %xmm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     movups	%xmm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     movups	(%rax), %xmm2
 # CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     mulps	%xmm0, %xmm2
@@ -451,7 +451,7 @@ xorps       (%rax), %xmm2
 # CHECK-NEXT: 1.00    -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     sqrtps	(%rax), %xmm2
 # CHECK-NEXT: 1.00    -      -      -      -      -      -      -      -      -      -      -      -     sqrtss	%xmm0, %xmm2
 # CHECK-NEXT: 1.00    -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     sqrtss	(%rax), %xmm2
-# CHECK-NEXT: 1.00    -      -      -     0.50    -     1.00   0.50   0.50   0.50    -      -      -     stmxcsr	(%rax)
+# CHECK-NEXT: 1.50    -      -      -     0.50    -     0.50   0.50   0.50   0.50    -      -      -     stmxcsr	(%rax)
 # CHECK-NEXT:  -     0.50    -      -      -     0.50    -      -      -      -      -      -      -     subps	%xmm0, %xmm2
 # CHECK-NEXT:  -     0.50   0.33   0.33    -     0.50    -      -      -      -      -     0.33    -     subps	(%rax), %xmm2
 # CHECK-NEXT:  -     0.50    -      -      -     0.50    -      -      -      -      -      -      -     subss	%xmm0, %xmm2
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse2.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse2.s
index 415089499bd51..964caa1d7f73c 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse2.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse2.s
@@ -407,58 +407,58 @@ xorpd       (%rax), %xmm2
 # CHECK-NEXT: [6]: HasSideEffects (U)
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
-# CHECK-NEXT:  1      2     0.50                        addpd	%xmm0, %xmm2
-# CHECK-NEXT:  2      9     0.50    *                   addpd	(%rax), %xmm2
-# CHECK-NEXT:  1      2     0.50                        addsd	%xmm0, %xmm2
-# CHECK-NEXT:  2      9     0.50    *                   addsd	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        addpd	%xmm0, %xmm2
+# CHECK-NEXT:  2      10    0.50    *                   addpd	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        addsd	%xmm0, %xmm2
+# CHECK-NEXT:  2      10    0.50    *                   addsd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        andnpd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   andnpd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   andnpd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        andpd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   andpd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   andpd	(%rax), %xmm2
 # CHECK-NEXT:  4      2     0.50    *      *      U     clflush	(%rax)
 # CHECK-NEXT:  1      4     0.50                        cmpeqpd	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   cmpeqpd	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   cmpeqpd	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        cmpeqsd	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   cmpeqsd	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   cmpeqsd	(%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        comisd	%xmm0, %xmm1
-# CHECK-NEXT:  2      9     1.00    *                   comisd	(%rax), %xmm1
+# CHECK-NEXT:  2      8     1.00    *                   comisd	(%rax), %xmm1
 # CHECK-NEXT:  2      5     1.00                        cvtdq2pd	%xmm0, %xmm2
 # CHECK-NEXT:  2      11    0.50    *                   cvtdq2pd	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        cvtdq2ps	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   cvtdq2ps	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   cvtdq2ps	(%rax), %xmm2
 # CHECK-NEXT:  2      5     1.00                        cvtpd2dq	%xmm0, %xmm2
 # CHECK-NEXT:  3      12    1.00    *                   cvtpd2dq	(%rax), %xmm2
 # CHECK-NEXT:  2      8     1.00                        cvtpd2pi	%xmm0, %mm2
-# CHECK-NEXT:  3      11    1.00    *                   cvtpd2pi	(%rax), %mm2
+# CHECK-NEXT:  3      24    1.00    *                   cvtpd2pi	(%rax), %mm2
 # CHECK-NEXT:  2      5     1.00                        cvtpd2ps	%xmm0, %xmm2
 # CHECK-NEXT:  3      12    1.00    *                   cvtpd2ps	(%rax), %xmm2
 # CHECK-NEXT:  2      6     1.00                        cvtpi2pd	%mm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   cvtpi2pd	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   cvtpi2pd	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        cvtps2dq	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   cvtps2dq	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   cvtps2dq	(%rax), %xmm2
 # CHECK-NEXT:  2      5     1.00                        cvtps2pd	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   cvtps2pd	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   cvtps2pd	(%rax), %xmm2
 # CHECK-NEXT:  2      7     1.00                        cvtsd2si	%xmm0, %ecx
 # CHECK-NEXT:  2      7     1.00                        cvtsd2si	%xmm0, %rcx
-# CHECK-NEXT:  3      12    1.00    *                   cvtsd2si	(%rax), %ecx
+# CHECK-NEXT:  3      26    1.00    *                   cvtsd2si	(%rax), %ecx
 # CHECK-NEXT:  3      12    1.00    *                   cvtsd2si	(%rax), %rcx
 # CHECK-NEXT:  2      5     1.00                        cvtsd2ss	%xmm0, %xmm2
-# CHECK-NEXT:  3      11    1.00    *                   cvtsd2ss	(%rax), %xmm2
-# CHECK-NEXT:  2      5     1.00                        cvtsi2sd	%ecx, %xmm2
-# CHECK-NEXT:  2      5     1.00                        cvtsi2sd	%rcx, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   cvtsi2sdl	(%rax), %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   cvtsi2sdl	(%rax), %xmm2
+# CHECK-NEXT:  3      12    1.00    *                   cvtsd2ss	(%rax), %xmm2
+# CHECK-NEXT:  2      7     1.00                        cvtsi2sd	%ecx, %xmm2
+# CHECK-NEXT:  2      7     1.00                        cvtsi2sd	%rcx, %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   cvtsi2sdl	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   cvtsi2sdl	(%rax), %xmm2
 # CHECK-NEXT:  2      5     1.00                        cvtss2sd	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   cvtss2sd	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   cvtss2sd	(%rax), %xmm2
 # CHECK-NEXT:  2      5     1.00                        cvttpd2dq	%xmm0, %xmm2
 # CHECK-NEXT:  3      12    1.00    *                   cvttpd2dq	(%rax), %xmm2
 # CHECK-NEXT:  2      8     1.00                        cvttpd2pi	%xmm0, %mm2
-# CHECK-NEXT:  3      11    1.00    *                   cvttpd2pi	(%rax), %mm2
+# CHECK-NEXT:  3      24    1.00    *                   cvttpd2pi	(%rax), %mm2
 # CHECK-NEXT:  1      4     0.50                        cvttps2dq	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   cvttps2dq	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   cvttps2dq	(%rax), %xmm2
 # CHECK-NEXT:  2      7     1.00                        cvttsd2si	%xmm0, %ecx
 # CHECK-NEXT:  2      7     1.00                        cvttsd2si	%xmm0, %rcx
-# CHECK-NEXT:  3      12    1.00    *                   cvttsd2si	(%rax), %ecx
+# CHECK-NEXT:  3      26    1.00    *                   cvttsd2si	(%rax), %ecx
 # CHECK-NEXT:  3      12    1.00    *                   cvttsd2si	(%rax), %rcx
 # CHECK-NEXT:  1      14    1.00                        divpd	%xmm0, %xmm2
 # CHECK-NEXT:  2      20    1.00    *                   divpd	(%rax), %xmm2
@@ -467,213 +467,213 @@ xorpd       (%rax), %xmm2
 # CHECK-NEXT:  1      2     0.50    *      *      U     lfence
 # CHECK-NEXT:  2      1     1.00    *      *      U     maskmovdqu	%xmm0, %xmm1
 # CHECK-NEXT:  1      4     0.50                        maxpd	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   maxpd	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   maxpd	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        maxsd	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   maxsd	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   maxsd	(%rax), %xmm2
 # CHECK-NEXT:  2      3     0.50    *      *      U     mfence
 # CHECK-NEXT:  1      4     0.50                        minpd	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   minpd	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   minpd	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        minsd	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   minsd	(%rax), %xmm2
-# CHECK-NEXT:  1      0     0.33                        movapd	%xmm0, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            movapd	%xmm0, (%rax)
-# CHECK-NEXT:  1      6     0.33    *                   movapd	(%rax), %xmm2
-# CHECK-NEXT:  1      1     1.00                        movd	%eax, %xmm2
-# CHECK-NEXT:  1      6     0.33    *                   movd	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   minsd	(%rax), %xmm2
+# CHECK-NEXT:  0      1     0.00                        movapd	%xmm0, %xmm2
+# CHECK-NEXT:  2      12    0.50           *            movapd	%xmm0, (%rax)
+# CHECK-NEXT:  1      7     0.33    *                   movapd	(%rax), %xmm2
+# CHECK-NEXT:  1      3     1.00                        movd	%eax, %xmm2
+# CHECK-NEXT:  1      7     0.33    *                   movd	(%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        movd	%xmm0, %ecx
-# CHECK-NEXT:  2      1     0.50           *            movd	%xmm0, (%rax)
-# CHECK-NEXT:  1      0     0.33                        movdqa	%xmm0, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            movdqa	%xmm0, (%rax)
-# CHECK-NEXT:  1      6     0.33    *                   movdqa	(%rax), %xmm2
-# CHECK-NEXT:  1      0     0.33                        movdqu	%xmm0, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            movdqu	%xmm0, (%rax)
-# CHECK-NEXT:  1      6     0.33    *                   movdqu	(%rax), %xmm2
-# CHECK-NEXT:  2      1     0.67                        movdq2q	%xmm0, %mm2
-# CHECK-NEXT:  2      1     0.50           *            movhpd	%xmm0, (%rax)
-# CHECK-NEXT:  2      7     1.00    *                   movhpd	(%rax), %xmm2
-# CHECK-NEXT:  2      1     0.50           *            movlpd	%xmm0, (%rax)
-# CHECK-NEXT:  2      7     0.50    *                   movlpd	(%rax), %xmm2
+# CHECK-NEXT:  2      12    0.50           *            movd	%xmm0, (%rax)
+# CHECK-NEXT:  0      1     0.00                        movdqa	%xmm0, %xmm2
+# CHECK-NEXT:  2      12    0.50           *            movdqa	%xmm0, (%rax)
+# CHECK-NEXT:  1      7     0.33    *                   movdqa	(%rax), %xmm2
+# CHECK-NEXT:  0      1     0.00                        movdqu	%xmm0, %xmm2
+# CHECK-NEXT:  2      12    0.50           *            movdqu	%xmm0, (%rax)
+# CHECK-NEXT:  1      7     0.33    *                   movdqu	(%rax), %xmm2
+# CHECK-NEXT:  2      3     0.67                        movdq2q	%xmm0, %mm2
+# CHECK-NEXT:  2      12    0.50           *            movhpd	%xmm0, (%rax)
+# CHECK-NEXT:  2      8     1.00    *                   movhpd	(%rax), %xmm2
+# CHECK-NEXT:  2      12    0.50           *            movlpd	%xmm0, (%rax)
+# CHECK-NEXT:  2      8     0.50    *                   movlpd	(%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        movmskpd	%xmm0, %ecx
 # CHECK-NEXT:  2      518   0.50           *            movntil	%eax, (%rax)
 # CHECK-NEXT:  2      512   0.50           *            movntiq	%rax, (%rax)
 # CHECK-NEXT:  2      512   0.50           *            movntdq	%xmm0, (%rax)
 # CHECK-NEXT:  2      518   0.50           *            movntpd	%xmm0, (%rax)
 # CHECK-NEXT:  1      1     0.33                        movq	%xmm0, %xmm2
-# CHECK-NEXT:  1      1     1.00                        movq	%rax, %xmm2
-# CHECK-NEXT:  1      6     0.33    *                   movq	(%rax), %xmm2
+# CHECK-NEXT:  1      3     1.00                        movq	%rax, %xmm2
+# CHECK-NEXT:  1      7     0.33    *                   movq	(%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        movq	%xmm0, %rcx
-# CHECK-NEXT:  2      1     0.50           *            movq	%xmm0, (%rax)
+# CHECK-NEXT:  2      12    0.50           *            movq	%xmm0, (%rax)
 # CHECK-NEXT:  2      3     1.00                        movq2dq	%mm0, %xmm2
 # CHECK-NEXT:  1      1     0.33                        movsd	%xmm0, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            movsd	%xmm0, (%rax)
-# CHECK-NEXT:  1      6     0.33    *                   movsd	(%rax), %xmm2
-# CHECK-NEXT:  1      0     0.33                        movupd	%xmm0, %xmm2
-# CHECK-NEXT:  2      1     0.50           *            movupd	%xmm0, (%rax)
-# CHECK-NEXT:  1      6     0.33    *                   movupd	(%rax), %xmm2
+# CHECK-NEXT:  2      12    0.50           *            movsd	%xmm0, (%rax)
+# CHECK-NEXT:  1      7     0.33    *                   movsd	(%rax), %xmm2
+# CHECK-NEXT:  0      1     0.00                        movupd	%xmm0, %xmm2
+# CHECK-NEXT:  2      12    0.50           *            movupd	%xmm0, (%rax)
+# CHECK-NEXT:  1      7     0.33    *                   movupd	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        mulpd	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   mulpd	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   mulpd	(%rax), %xmm2
 # CHECK-NEXT:  1      4     0.50                        mulsd	%xmm0, %xmm2
-# CHECK-NEXT:  2      10    0.50    *                   mulsd	(%rax), %xmm2
+# CHECK-NEXT:  2      11    0.50    *                   mulsd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        orpd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   orpd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   orpd	(%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        packssdw	%xmm0, %xmm2
-# CHECK-NEXT:  2      9     1.00    *                   packssdw	(%rax), %xmm2
+# CHECK-NEXT:  2      10    1.00    *                   packssdw	(%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        packsswb	%xmm0, %xmm2
-# CHECK-NEXT:  2      9     1.00    *                   packsswb	(%rax), %xmm2
+# CHECK-NEXT:  2      10    1.00    *                   packsswb	(%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        packuswb	%xmm0, %xmm2
-# CHECK-NEXT:  2      9     1.00    *                   packuswb	(%rax), %xmm2
+# CHECK-NEXT:  2      10    1.00    *                   packuswb	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        paddb	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   paddb	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   paddb	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        paddd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   paddd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   paddd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        paddq	%mm0, %mm2
-# CHECK-NEXT:  2      7     0.50    *                   paddq	(%rax), %mm2
+# CHECK-NEXT:  2      9     0.50    *                   paddq	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.33                        paddq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   paddq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   paddq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        paddsb	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   paddsb	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   paddsb	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        paddsw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   paddsw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   paddsw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        paddusb	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   paddusb	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   paddusb	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        paddusw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   paddusw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   paddusw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        paddw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   paddw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   paddw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        pand	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   pand	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   pand	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        pandn	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   pandn	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   pandn	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pavgb	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pavgb	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pavgb	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pavgw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pavgw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pavgw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pcmpeqb	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pcmpeqb	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pcmpeqb	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pcmpeqd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pcmpeqd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pcmpeqd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pcmpeqw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pcmpeqw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pcmpeqw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pcmpgtb	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pcmpgtb	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pcmpgtb	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pcmpgtd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pcmpgtd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pcmpgtd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pcmpgtw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pcmpgtw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pcmpgtw	(%rax), %xmm2
 # CHECK-NEXT:  2      4     1.00                        pextrw	$1, %xmm0, %ecx
-# CHECK-NEXT:  2      2     1.00                        pinsrw	$1, %eax, %xmm0
+# CHECK-NEXT:  2      4     1.00                        pinsrw	$1, %eax, %xmm0
 # CHECK-NEXT:  2      8     0.50    *                   pinsrw	$1, (%rax), %xmm0
 # CHECK-NEXT:  1      5     0.50                        pmaddwd	%xmm0, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   pmaddwd	(%rax), %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   pmaddwd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pmaxsw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pmaxsw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pmaxsw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pmaxub	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pmaxub	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pmaxub	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pminsw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pminsw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pminsw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pminub	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pminub	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pminub	(%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        pmovmskb	%xmm0, %ecx
 # CHECK-NEXT:  1      5     0.50                        pmulhuw	%xmm0, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   pmulhuw	(%rax), %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   pmulhuw	(%rax), %xmm2
 # CHECK-NEXT:  1      5     0.50                        pmulhw	%xmm0, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   pmulhw	(%rax), %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   pmulhw	(%rax), %xmm2
 # CHECK-NEXT:  1      5     0.50                        pmullw	%xmm0, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   pmullw	(%rax), %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   pmullw	(%rax), %xmm2
 # CHECK-NEXT:  1      5     1.00                        pmuludq	%mm0, %mm2
 # CHECK-NEXT:  2      13    1.00    *                   pmuludq	(%rax), %mm2
 # CHECK-NEXT:  1      5     0.50                        pmuludq	%xmm0, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   pmuludq	(%rax), %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   pmuludq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        por	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   por	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   por	(%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        psadbw	%xmm0, %xmm2
-# CHECK-NEXT:  2      9     1.00    *                   psadbw	(%rax), %xmm2
+# CHECK-NEXT:  2      10    1.00    *                   psadbw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pshufd	$1, %xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pshufd	$1, (%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pshufd	$1, (%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pshufhw	$1, %xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pshufhw	$1, (%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pshufhw	$1, (%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pshuflw	$1, %xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pshuflw	$1, (%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pshuflw	$1, (%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pslld	$1, %xmm2
 # CHECK-NEXT:  2      2     0.67                        pslld	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pslld	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pslld	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pslldq	$1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        psllq	$1, %xmm2
 # CHECK-NEXT:  2      2     0.67                        psllq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   psllq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   psllq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        psllw	$1, %xmm2
 # CHECK-NEXT:  2      2     0.67                        psllw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   psllw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   psllw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        psrad	$1, %xmm2
 # CHECK-NEXT:  2      2     0.67                        psrad	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   psrad	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   psrad	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        psraw	$1, %xmm2
 # CHECK-NEXT:  2      2     0.67                        psraw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   psraw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   psraw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        psrld	$1, %xmm2
 # CHECK-NEXT:  2      2     0.67                        psrld	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   psrld	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   psrld	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        psrldq	$1, %xmm2
 # CHECK-NEXT:  1      1     0.50                        psrlq	$1, %xmm2
 # CHECK-NEXT:  2      2     0.67                        psrlq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   psrlq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   psrlq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        psrlw	$1, %xmm2
 # CHECK-NEXT:  2      2     0.67                        psrlw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   psrlw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   psrlw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        psubb	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   psubb	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   psubb	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        psubd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   psubd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   psubd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        psubq	%mm0, %mm2
-# CHECK-NEXT:  2      7     0.50    *                   psubq	(%rax), %mm2
+# CHECK-NEXT:  2      9     0.50    *                   psubq	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.33                        psubq	%xmm0, %xmm2
 # CHECK-NEXT:  2      8     0.33    *                   psubq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        psubsb	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   psubsb	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   psubsb	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        psubsw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   psubsw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   psubsw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        psubusb	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   psubusb	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   psubusb	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        psubusw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   psubusw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   psubusw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        psubw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   psubw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   psubw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        punpckhbw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   punpckhbw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   punpckhbw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        punpckhdq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   punpckhdq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   punpckhdq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        punpckhqdq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   punpckhqdq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   punpckhqdq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        punpckhwd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   punpckhwd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   punpckhwd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        punpcklbw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   punpcklbw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   punpcklbw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        punpckldq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   punpckldq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   punpckldq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        punpcklqdq	%xmm0, %xmm2
 # CHECK-NEXT:  2      8     0.50    *                   punpcklqdq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        punpcklwd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   punpcklwd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   punpcklwd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        pxor	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   pxor	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   pxor	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        shufpd	$1, %xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   shufpd	$1, (%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   shufpd	$1, (%rax), %xmm2
 # CHECK-NEXT:  1      18    1.00                        sqrtpd	%xmm0, %xmm2
 # CHECK-NEXT:  2      24    1.00    *                   sqrtpd	(%rax), %xmm2
 # CHECK-NEXT:  1      18    1.00                        sqrtsd	%xmm0, %xmm2
 # CHECK-NEXT:  2      24    1.00    *                   sqrtsd	(%rax), %xmm2
-# CHECK-NEXT:  1      2     0.50                        subpd	%xmm0, %xmm2
-# CHECK-NEXT:  2      9     0.50    *                   subpd	(%rax), %xmm2
-# CHECK-NEXT:  1      2     0.50                        subsd	%xmm0, %xmm2
-# CHECK-NEXT:  2      9     0.50    *                   subsd	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        subpd	%xmm0, %xmm2
+# CHECK-NEXT:  2      10    0.50    *                   subpd	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        subsd	%xmm0, %xmm2
+# CHECK-NEXT:  2      10    0.50    *                   subsd	(%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        ucomisd	%xmm0, %xmm1
-# CHECK-NEXT:  2      9     1.00    *                   ucomisd	(%rax), %xmm1
+# CHECK-NEXT:  2      8     1.00    *                   ucomisd	(%rax), %xmm1
 # CHECK-NEXT:  1      1     1.00                        unpckhpd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     1.00    *                   unpckhpd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     1.00    *                   unpckhpd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     1.00                        unpcklpd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     1.00    *                   unpcklpd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     1.00    *                   unpcklpd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        xorpd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   xorpd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   xorpd	(%rax), %xmm2
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - ADLPPort00
@@ -692,7 +692,7 @@ xorpd       (%rax), %xmm2
 
 # CHECK:      Resource pressure per iteration:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]
-# CHECK-NEXT: 109.03 100.53 39.33  39.33  8.50   73.03  1.20   7.83   7.50   7.50   0.20   39.00   -
+# CHECK-NEXT: 107.70 99.20  39.33  39.33  8.50   71.70  1.20   7.83   7.50   7.50   0.20   39.00   -
 
 # CHECK:      Resource pressure by instruction:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]   Instructions:
@@ -764,17 +764,17 @@ xorpd       (%rax), %xmm2
 # CHECK-NEXT: 0.50   0.50   0.33   0.33    -      -      -      -      -      -      -     0.33    -     minpd	(%rax), %xmm2
 # CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     minsd	%xmm0, %xmm2
 # CHECK-NEXT: 0.50   0.50   0.33   0.33    -      -      -      -      -      -      -     0.33    -     minsd	(%rax), %xmm2
-# CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     movapd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     movapd	%xmm0, %xmm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     movapd	%xmm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     movapd	(%rax), %xmm2
 # CHECK-NEXT:  -      -      -      -      -     1.00    -      -      -      -      -      -      -     movd	%eax, %xmm2
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     movd	(%rax), %xmm2
 # CHECK-NEXT: 1.00    -      -      -      -      -      -      -      -      -      -      -      -     movd	%xmm0, %ecx
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     movd	%xmm0, (%rax)
-# CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     movdqa	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     movdqa	%xmm0, %xmm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     movdqa	%xmm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     movdqa	(%rax), %xmm2
-# CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     movdqu	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     movdqu	%xmm0, %xmm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     movdqu	%xmm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     movdqu	(%rax), %xmm2
 # CHECK-NEXT: 0.83   0.33    -      -      -     0.83    -      -      -      -      -      -      -     movdq2q	%xmm0, %mm2
@@ -796,7 +796,7 @@ xorpd       (%rax), %xmm2
 # CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     movsd	%xmm0, %xmm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     movsd	%xmm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     movsd	(%rax), %xmm2
-# CHECK-NEXT: 0.33   0.33    -      -      -     0.33    -      -      -      -      -      -      -     movupd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     movupd	%xmm0, %xmm2
 # CHECK-NEXT:  -      -      -      -     0.50    -      -     0.50   0.50   0.50    -      -      -     movupd	%xmm0, (%rax)
 # CHECK-NEXT:  -      -     0.33   0.33    -      -      -      -      -      -      -     0.33    -     movupd	(%rax), %xmm2
 # CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     mulpd	%xmm0, %xmm2
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse3.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse3.s
index 5b92e7c422f48..15baea9604c74 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse3.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse3.s
@@ -43,19 +43,19 @@ mwait
 # CHECK-NEXT: [6]: HasSideEffects (U)
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
-# CHECK-NEXT:  1      2     0.50                        addsubpd	%xmm0, %xmm2
-# CHECK-NEXT:  2      9     0.50    *                   addsubpd	(%rax), %xmm2
-# CHECK-NEXT:  1      2     0.50                        addsubps	%xmm0, %xmm2
-# CHECK-NEXT:  2      9     0.50    *                   addsubps	(%rax), %xmm2
-# CHECK-NEXT:  3      5     2.00                        haddpd	%xmm0, %xmm2
-# CHECK-NEXT:  4      11    2.00    *                   haddpd	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        addsubpd	%xmm0, %xmm2
+# CHECK-NEXT:  2      10    0.50    *                   addsubpd	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        addsubps	%xmm0, %xmm2
+# CHECK-NEXT:  2      10    0.50    *                   addsubps	(%rax), %xmm2
+# CHECK-NEXT:  3      6     2.00                        haddpd	%xmm0, %xmm2
+# CHECK-NEXT:  4      12    2.00    *                   haddpd	(%rax), %xmm2
 # CHECK-NEXT:  3      6     2.00                        haddps	%xmm0, %xmm2
 # CHECK-NEXT:  4      12    2.00    *                   haddps	(%rax), %xmm2
-# CHECK-NEXT:  3      5     2.00                        hsubpd	%xmm0, %xmm2
-# CHECK-NEXT:  4      11    2.00    *                   hsubpd	(%rax), %xmm2
+# CHECK-NEXT:  3      6     2.00                        hsubpd	%xmm0, %xmm2
+# CHECK-NEXT:  4      12    2.00    *                   hsubpd	(%rax), %xmm2
 # CHECK-NEXT:  3      6     2.00                        hsubps	%xmm0, %xmm2
 # CHECK-NEXT:  4      12    2.00    *                   hsubps	(%rax), %xmm2
-# CHECK-NEXT:  1      6     0.33    *                   lddqu	(%rax), %xmm2
+# CHECK-NEXT:  1      7     0.33    *                   lddqu	(%rax), %xmm2
 # CHECK-NEXT:  1      100   0.25                  U     monitor
 # CHECK-NEXT:  1      1     1.00                        movddup	%xmm0, %xmm2
 # CHECK-NEXT:  1      7     0.33    *                   movddup	(%rax), %xmm2
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse41.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse41.s
index 39daa2aee5611..ffe9150cc5916 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse41.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse41.s
@@ -156,101 +156,101 @@ roundss     $1, (%rax), %xmm2
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
 # CHECK-NEXT:  1      1     0.33                        blendpd	$11, %xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   blendpd	$11, (%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   blendpd	$11, (%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        blendps	$11, %xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.33    *                   blendps	$11, (%rax), %xmm2
+# CHECK-NEXT:  2      8     0.33    *                   blendps	$11, (%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        blendvpd	%xmm0, %xmm0, %xmm2
 # CHECK-NEXT:  2      8     0.33    *                   blendvpd	%xmm0, (%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        blendvps	%xmm0, %xmm0, %xmm2
 # CHECK-NEXT:  2      8     0.33    *                   blendvps	%xmm0, (%rax), %xmm2
 # CHECK-NEXT:  3      9     1.00                        dppd	$22, %xmm0, %xmm2
-# CHECK-NEXT:  4      15    1.00    *                   dppd	$22, (%rax), %xmm2
+# CHECK-NEXT:  4      16    1.00    *                   dppd	$22, (%rax), %xmm2
 # CHECK-NEXT:  6      14    1.67                        dpps	$22, %xmm0, %xmm2
 # CHECK-NEXT:  7      21    1.67    *                   dpps	$22, (%rax), %xmm2
 # CHECK-NEXT:  2      4     1.00                        extractps	$1, %xmm0, %ecx
-# CHECK-NEXT:  3      2     1.00           *            extractps	$1, %xmm0, (%rax)
+# CHECK-NEXT:  3      12    1.00           *            extractps	$1, %xmm0, (%rax)
 # CHECK-NEXT:  1      1     1.00                        insertps	$1, %xmm0, %xmm2
 # CHECK-NEXT:  2      8     1.00    *                   insertps	$1, (%rax), %xmm2
 # CHECK-NEXT:  1      7     0.33    *                   movntdqa	(%rax), %xmm2
 # CHECK-NEXT:  2      4     1.00                        mpsadbw	$1, %xmm0, %xmm2
-# CHECK-NEXT:  3      10    1.00    *                   mpsadbw	$1, (%rax), %xmm2
+# CHECK-NEXT:  3      11    1.00    *                   mpsadbw	$1, (%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        packusdw	%xmm0, %xmm2
-# CHECK-NEXT:  2      9     1.00    *                   packusdw	(%rax), %xmm2
+# CHECK-NEXT:  2      10    1.00    *                   packusdw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.33                        pblendvb	%xmm0, %xmm0, %xmm2
 # CHECK-NEXT:  2      8     0.33    *                   pblendvb	%xmm0, (%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pblendw	$11, %xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pblendw	$11, (%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pblendw	$11, (%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pcmpeqq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pcmpeqq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pcmpeqq	(%rax), %xmm2
 # CHECK-NEXT:  2      4     1.00                        pextrb	$1, %xmm0, %ecx
 # CHECK-NEXT:  3      19    0.50           *            pextrb	$1, %xmm0, (%rax)
 # CHECK-NEXT:  2      4     1.00                        pextrd	$1, %xmm0, %ecx
-# CHECK-NEXT:  3      2     0.50           *            pextrd	$1, %xmm0, (%rax)
+# CHECK-NEXT:  3      12    0.50           *            pextrd	$1, %xmm0, (%rax)
 # CHECK-NEXT:  2      4     1.00                        pextrq	$1, %xmm0, %rcx
-# CHECK-NEXT:  3      2     0.50           *            pextrq	$1, %xmm0, (%rax)
-# CHECK-NEXT:  3      2     0.50           *            pextrw	$1, %xmm0, (%rax)
+# CHECK-NEXT:  3      12    0.50           *            pextrq	$1, %xmm0, (%rax)
+# CHECK-NEXT:  3      19    0.50           *            pextrw	$1, %xmm0, (%rax)
 # CHECK-NEXT:  1      4     1.00                        phminposuw	%xmm0, %xmm2
 # CHECK-NEXT:  2      11    1.00    *                   phminposuw	(%rax), %xmm2
-# CHECK-NEXT:  2      2     1.00                        pinsrb	$1, %eax, %xmm1
-# CHECK-NEXT:  2      7     0.50    *                   pinsrb	$1, (%rax), %xmm1
-# CHECK-NEXT:  2      2     1.00                        pinsrd	$1, %eax, %xmm1
-# CHECK-NEXT:  2      7     0.50    *                   pinsrd	$1, (%rax), %xmm1
-# CHECK-NEXT:  2      2     1.00                        pinsrq	$1, %rax, %xmm1
-# CHECK-NEXT:  2      7     0.50    *                   pinsrq	$1, (%rax), %xmm1
+# CHECK-NEXT:  2      4     1.00                        pinsrb	$1, %eax, %xmm1
+# CHECK-NEXT:  2      8     0.50    *                   pinsrb	$1, (%rax), %xmm1
+# CHECK-NEXT:  2      4     1.00                        pinsrd	$1, %eax, %xmm1
+# CHECK-NEXT:  2      8     0.50    *                   pinsrd	$1, (%rax), %xmm1
+# CHECK-NEXT:  2      4     1.00                        pinsrq	$1, %rax, %xmm1
+# CHECK-NEXT:  2      8     0.50    *                   pinsrq	$1, (%rax), %xmm1
 # CHECK-NEXT:  1      1     0.50                        pmaxsb	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pmaxsb	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pmaxsb	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pmaxsd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pmaxsd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pmaxsd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pmaxud	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pmaxud	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pmaxud	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pmaxuw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pmaxuw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pmaxuw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pminsb	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pminsb	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pminsb	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pminsd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pminsd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pminsd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pminud	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pminud	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pminud	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pminuw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pminuw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pminuw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pmovsxbd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pmovsxbd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pmovsxbd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pmovsxbq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pmovsxbq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pmovsxbq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pmovsxbw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pmovsxbw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pmovsxbw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pmovsxdq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pmovsxdq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pmovsxdq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pmovsxwd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pmovsxwd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pmovsxwd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pmovsxwq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pmovsxwq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pmovsxwq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pmovzxbd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pmovzxbd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pmovzxbd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pmovzxbq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pmovzxbq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pmovzxbq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pmovzxbw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pmovzxbw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pmovzxbw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pmovzxdq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pmovzxdq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pmovzxdq	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pmovzxwd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pmovzxwd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pmovzxwd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     0.50                        pmovzxwq	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pmovzxwq	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pmovzxwq	(%rax), %xmm2
 # CHECK-NEXT:  1      5     0.50                        pmuldq	%xmm0, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   pmuldq	(%rax), %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   pmuldq	(%rax), %xmm2
 # CHECK-NEXT:  2      10    1.00                        pmulld	%xmm0, %xmm2
-# CHECK-NEXT:  3      16    1.00    *                   pmulld	(%rax), %xmm2
+# CHECK-NEXT:  3      18    1.00    *                   pmulld	(%rax), %xmm2
 # CHECK-NEXT:  2      4     1.00                        ptest	%xmm0, %xmm1
 # CHECK-NEXT:  3      9     1.00    *                   ptest	(%rax), %xmm1
 # CHECK-NEXT:  2      8     1.00                        roundpd	$1, %xmm0, %xmm2
-# CHECK-NEXT:  3      14    1.00    *                   roundpd	$1, (%rax), %xmm2
+# CHECK-NEXT:  3      15    1.00    *                   roundpd	$1, (%rax), %xmm2
 # CHECK-NEXT:  2      8     1.00                        roundps	$1, %xmm0, %xmm2
-# CHECK-NEXT:  3      14    1.00    *                   roundps	$1, (%rax), %xmm2
+# CHECK-NEXT:  3      15    1.00    *                   roundps	$1, (%rax), %xmm2
 # CHECK-NEXT:  2      8     1.00                        roundsd	$1, %xmm0, %xmm2
-# CHECK-NEXT:  3      14    1.00    *                   roundsd	$1, (%rax), %xmm2
+# CHECK-NEXT:  3      15    1.00    *                   roundsd	$1, (%rax), %xmm2
 # CHECK-NEXT:  2      8     1.00                        roundss	$1, %xmm0, %xmm2
-# CHECK-NEXT:  3      14    1.00    *                   roundss	$1, (%rax), %xmm2
+# CHECK-NEXT:  3      15    1.00    *                   roundss	$1, (%rax), %xmm2
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - ADLPPort00
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse42.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse42.s
index 63bd12c060207..cb5b34e9b6468 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse42.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-sse42.s
@@ -59,7 +59,7 @@ pcmpgtq     (%rax), %xmm2
 # CHECK-NEXT:  3      11    3.00                        pcmpistrm	$1, %xmm0, %xmm2
 # CHECK-NEXT:  4      16    3.00    *                   pcmpistrm	$1, (%rax), %xmm2
 # CHECK-NEXT:  1      3     1.00                        pcmpgtq	%xmm0, %xmm2
-# CHECK-NEXT:  2      9     1.00    *                   pcmpgtq	(%rax), %xmm2
+# CHECK-NEXT:  2      10    1.00    *                   pcmpgtq	(%rax), %xmm2
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - ADLPPort00
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-ssse3.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-ssse3.s
index fa01796ba382f..33ec9b0fa64d2 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-ssse3.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-ssse3.s
@@ -107,69 +107,69 @@ psignw      (%rax), %xmm2
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
 # CHECK-NEXT:  1      1     1.00                        pabsb	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   pabsb	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   pabsb	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.50                        pabsb	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pabsb	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pabsb	(%rax), %xmm2
 # CHECK-NEXT:  1      1     1.00                        pabsd	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   pabsd	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   pabsd	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.50                        pabsd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pabsd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pabsd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     1.00                        pabsw	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   pabsw	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   pabsw	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.50                        pabsw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pabsw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pabsw	(%rax), %xmm2
 # CHECK-NEXT:  1      1     1.00                        palignr	$1, %mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   palignr	$1, (%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   palignr	$1, (%rax), %mm2
 # CHECK-NEXT:  1      1     1.00                        palignr	$1, %xmm0, %xmm2
-# CHECK-NEXT:  2      7     1.00    *                   palignr	$1, (%rax), %xmm2
+# CHECK-NEXT:  2      8     1.00    *                   palignr	$1, (%rax), %xmm2
 # CHECK-NEXT:  3      3     2.00                        phaddd	%mm0, %mm2
-# CHECK-NEXT:  4      9     2.00    *                   phaddd	(%rax), %mm2
+# CHECK-NEXT:  4      11    2.00    *                   phaddd	(%rax), %mm2
 # CHECK-NEXT:  3      2     1.00                        phaddd	%xmm0, %xmm2
-# CHECK-NEXT:  4      8     1.00    *                   phaddd	(%rax), %xmm2
+# CHECK-NEXT:  4      9     1.00    *                   phaddd	(%rax), %xmm2
 # CHECK-NEXT:  3      3     2.00                        phaddsw	%mm0, %mm2
 # CHECK-NEXT:  4      11    2.00    *                   phaddsw	(%rax), %mm2
 # CHECK-NEXT:  3      2     1.00                        phaddsw	%xmm0, %xmm2
-# CHECK-NEXT:  4      8     1.00    *                   phaddsw	(%rax), %xmm2
+# CHECK-NEXT:  4      9     1.00    *                   phaddsw	(%rax), %xmm2
 # CHECK-NEXT:  3      3     2.00                        phaddw	%mm0, %mm2
-# CHECK-NEXT:  4      9     2.00    *                   phaddw	(%rax), %mm2
+# CHECK-NEXT:  4      11    2.00    *                   phaddw	(%rax), %mm2
 # CHECK-NEXT:  3      2     1.00                        phaddw	%xmm0, %xmm2
-# CHECK-NEXT:  4      8     1.00    *                   phaddw	(%rax), %xmm2
+# CHECK-NEXT:  4      9     1.00    *                   phaddw	(%rax), %xmm2
 # CHECK-NEXT:  3      3     2.00                        phsubd	%mm0, %mm2
-# CHECK-NEXT:  4      9     2.00    *                   phsubd	(%rax), %mm2
+# CHECK-NEXT:  4      11    2.00    *                   phsubd	(%rax), %mm2
 # CHECK-NEXT:  3      2     1.00                        phsubd	%xmm0, %xmm2
-# CHECK-NEXT:  4      8     1.00    *                   phsubd	(%rax), %xmm2
+# CHECK-NEXT:  4      9     1.00    *                   phsubd	(%rax), %xmm2
 # CHECK-NEXT:  3      3     2.00                        phsubsw	%mm0, %mm2
 # CHECK-NEXT:  4      11    2.00    *                   phsubsw	(%rax), %mm2
 # CHECK-NEXT:  3      2     1.00                        phsubsw	%xmm0, %xmm2
-# CHECK-NEXT:  4      8     1.00    *                   phsubsw	(%rax), %xmm2
+# CHECK-NEXT:  4      9     1.00    *                   phsubsw	(%rax), %xmm2
 # CHECK-NEXT:  3      3     2.00                        phsubw	%mm0, %mm2
-# CHECK-NEXT:  4      9     2.00    *                   phsubw	(%rax), %mm2
+# CHECK-NEXT:  4      11    2.00    *                   phsubw	(%rax), %mm2
 # CHECK-NEXT:  3      2     1.00                        phsubw	%xmm0, %xmm2
-# CHECK-NEXT:  4      8     1.00    *                   phsubw	(%rax), %xmm2
+# CHECK-NEXT:  4      9     1.00    *                   phsubw	(%rax), %xmm2
 # CHECK-NEXT:  1      5     1.00                        pmaddubsw	%mm0, %mm2
 # CHECK-NEXT:  2      13    1.00    *                   pmaddubsw	(%rax), %mm2
 # CHECK-NEXT:  1      5     0.50                        pmaddubsw	%xmm0, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   pmaddubsw	(%rax), %xmm2
+# CHECK-NEXT:  2      13    0.50    *                   pmaddubsw	(%rax), %xmm2
 # CHECK-NEXT:  1      5     1.00                        pmulhrsw	%mm0, %mm2
 # CHECK-NEXT:  2      13    1.00    *                   pmulhrsw	(%rax), %mm2
 # CHECK-NEXT:  1      5     0.50                        pmulhrsw	%xmm0, %xmm2
-# CHECK-NEXT:  2      11    0.50    *                   pmulhrsw	(%rax), %xmm2
-# CHECK-NEXT:  3      3     1.00                        pshufb	%mm0, %mm2
-# CHECK-NEXT:  3      9     1.00    *                   pshufb	(%rax), %mm2
+# CHECK-NEXT:  2      13    0.50    *                   pmulhrsw	(%rax), %xmm2
+# CHECK-NEXT:  2      3     1.00                        pshufb	%mm0, %mm2
+# CHECK-NEXT:  3      11    1.00    *                   pshufb	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.50                        pshufb	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   pshufb	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   pshufb	(%rax), %xmm2
 # CHECK-NEXT:  1      1     1.00                        psignb	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   psignb	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   psignb	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.50                        psignb	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   psignb	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   psignb	(%rax), %xmm2
 # CHECK-NEXT:  1      1     1.00                        psignd	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   psignd	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   psignd	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.50                        psignd	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   psignd	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   psignd	(%rax), %xmm2
 # CHECK-NEXT:  1      1     1.00                        psignw	%mm0, %mm2
-# CHECK-NEXT:  2      7     1.00    *                   psignw	(%rax), %mm2
+# CHECK-NEXT:  2      9     1.00    *                   psignw	(%rax), %mm2
 # CHECK-NEXT:  1      1     0.50                        psignw	%xmm0, %xmm2
-# CHECK-NEXT:  2      7     0.50    *                   psignw	(%rax), %xmm2
+# CHECK-NEXT:  2      8     0.50    *                   psignw	(%rax), %xmm2
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - ADLPPort00
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-vaes.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-vaes.s
index 0e33a9af13e32..74b19b968bec5 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-vaes.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-vaes.s
@@ -22,14 +22,14 @@ vaesenclast      (%rax), %ymm1, %ymm3
 # CHECK-NEXT: [6]: HasSideEffects (U)
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
-# CHECK-NEXT:  1      3     0.50                        vaesdec	%ymm0, %ymm1, %ymm3
-# CHECK-NEXT:  2      11    0.50    *                   vaesdec	(%rax), %ymm1, %ymm3
-# CHECK-NEXT:  1      3     0.50                        vaesdeclast	%ymm0, %ymm1, %ymm3
-# CHECK-NEXT:  2      11    0.50    *                   vaesdeclast	(%rax), %ymm1, %ymm3
-# CHECK-NEXT:  1      3     0.50                        vaesenc	%ymm0, %ymm1, %ymm3
-# CHECK-NEXT:  2      11    0.50    *                   vaesenc	(%rax), %ymm1, %ymm3
-# CHECK-NEXT:  1      3     0.50                        vaesenclast	%ymm0, %ymm1, %ymm3
-# CHECK-NEXT:  2      11    0.50    *                   vaesenclast	(%rax), %ymm1, %ymm3
+# CHECK-NEXT:  1      5     0.50                        vaesdec	%ymm0, %ymm1, %ymm3
+# CHECK-NEXT:  2      12    0.50    *                   vaesdec	(%rax), %ymm1, %ymm3
+# CHECK-NEXT:  1      5     0.50                        vaesdeclast	%ymm0, %ymm1, %ymm3
+# CHECK-NEXT:  2      12    0.50    *                   vaesdeclast	(%rax), %ymm1, %ymm3
+# CHECK-NEXT:  1      5     0.50                        vaesenc	%ymm0, %ymm1, %ymm3
+# CHECK-NEXT:  2      12    0.50    *                   vaesenc	(%rax), %ymm1, %ymm3
+# CHECK-NEXT:  1      5     0.50                        vaesenclast	%ymm0, %ymm1, %ymm3
+# CHECK-NEXT:  2      12    0.50    *                   vaesenclast	(%rax), %ymm1, %ymm3
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - ADLPPort00
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-vpclmulqdq.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-vpclmulqdq.s
index 890f977450a72..cd834d35c43d5 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-vpclmulqdq.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-vpclmulqdq.s
@@ -14,7 +14,7 @@ vpclmulqdq    $11, (%rax), %ymm1, %ymm3
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
 # CHECK-NEXT:  1      3     1.00                        vpclmulqdq	$11, %ymm0, %ymm1, %ymm3
-# CHECK-NEXT:  2      10    1.00    *                   vpclmulqdq	$11, (%rax), %ymm1, %ymm3
+# CHECK-NEXT:  2      11    1.00    *                   vpclmulqdq	$11, (%rax), %ymm1, %ymm3
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - ADLPPort00
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-x86_64.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-x86_64.s
index 9d529a344a5e2..b502ab3bf321d 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-x86_64.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/resources-x86_64.s
@@ -1043,8 +1043,8 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  5      13    0.50    *      *            adcb	$7, (%rax)
 # CHECK-NEXT:  5      13    0.50    *      *            lock		adcb	$7, (%rax)
 # CHECK-NEXT:  1      1     0.50                        adcb	%sil, %dil
-# CHECK-NEXT:  6      7     0.60    *      *            adcb	%sil, (%rax)
-# CHECK-NEXT:  6      7     0.60    *      *            lock		adcb	%sil, (%rax)
+# CHECK-NEXT:  6      13    0.60    *      *            adcb	%sil, (%rax)
+# CHECK-NEXT:  6      13    0.60    *      *            lock		adcb	%sil, (%rax)
 # CHECK-NEXT:  2      6     0.50    *                   adcb	(%rax), %dil
 # CHECK-NEXT:  1      1     0.50                        adcw	$0, %ax
 # CHECK-NEXT:  1      1     0.50                        adcw	$0, %di
@@ -1058,8 +1058,8 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  5      12    0.50    *      *            adcw	$7, (%rax)
 # CHECK-NEXT:  5      12    0.50    *      *            lock		adcw	$7, (%rax)
 # CHECK-NEXT:  1      1     0.50                        adcw	%si, %di
-# CHECK-NEXT:  6      7     0.60    *      *            adcw	%si, (%rax)
-# CHECK-NEXT:  6      7     0.60    *      *            lock		adcw	%si, (%rax)
+# CHECK-NEXT:  6      12    0.60    *      *            adcw	%si, (%rax)
+# CHECK-NEXT:  6      12    0.60    *      *            lock		adcw	%si, (%rax)
 # CHECK-NEXT:  2      6     0.50    *                   adcw	(%rax), %di
 # CHECK-NEXT:  1      1     0.50                        adcl	$0, %eax
 # CHECK-NEXT:  1      1     0.50                        adcl	$0, %edi
@@ -1073,8 +1073,8 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  5      12    0.50    *      *            adcl	$7, (%rax)
 # CHECK-NEXT:  5      12    0.50    *      *            lock		adcl	$7, (%rax)
 # CHECK-NEXT:  1      1     0.50                        adcl	%esi, %edi
-# CHECK-NEXT:  6      7     0.60    *      *            adcl	%esi, (%rax)
-# CHECK-NEXT:  6      7     0.60    *      *            lock		adcl	%esi, (%rax)
+# CHECK-NEXT:  6      12    0.60    *      *            adcl	%esi, (%rax)
+# CHECK-NEXT:  6      12    0.60    *      *            lock		adcl	%esi, (%rax)
 # CHECK-NEXT:  2      6     0.50    *                   adcl	(%rax), %edi
 # CHECK-NEXT:  1      1     0.50                        adcq	$0, %rax
 # CHECK-NEXT:  1      1     0.50                        adcq	$0, %rdi
@@ -1088,91 +1088,91 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  5      12    0.50    *      *            adcq	$7, (%rax)
 # CHECK-NEXT:  5      12    0.50    *      *            lock		adcq	$7, (%rax)
 # CHECK-NEXT:  1      1     0.50                        adcq	%rsi, %rdi
-# CHECK-NEXT:  6      7     0.60    *      *            adcq	%rsi, (%rax)
-# CHECK-NEXT:  6      7     0.60    *      *            lock		adcq	%rsi, (%rax)
+# CHECK-NEXT:  6      12    0.60    *      *            adcq	%rsi, (%rax)
+# CHECK-NEXT:  6      12    0.60    *      *            lock		adcq	%rsi, (%rax)
 # CHECK-NEXT:  2      6     0.50    *                   adcq	(%rax), %rdi
 # CHECK-NEXT:  1      1     0.20                        addb	$7, %al
 # CHECK-NEXT:  1      1     0.20                        addb	$7, %dil
-# CHECK-NEXT:  4      7     0.50    *      *            addb	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		addb	$7, (%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            addb	$7, (%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            lock		addb	$7, (%rax)
 # CHECK-NEXT:  1      1     0.20                        addb	%sil, %dil
-# CHECK-NEXT:  4      7     0.50    *      *            addb	%sil, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		addb	%sil, (%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            addb	%sil, (%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            lock		addb	%sil, (%rax)
 # CHECK-NEXT:  2      6     0.33    *                   addb	(%rax), %dil
 # CHECK-NEXT:  1      1     0.20                        addw	$511, %ax
 # CHECK-NEXT:  1      1     0.20                        addw	$511, %di
-# CHECK-NEXT:  4      7     0.50    *      *            addw	$511, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		addw	$511, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            addw	$511, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		addw	$511, (%rax)
 # CHECK-NEXT:  1      1     0.20                        addw	$7, %di
-# CHECK-NEXT:  4      7     0.50    *      *            addw	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		addw	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            addw	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		addw	$7, (%rax)
 # CHECK-NEXT:  1      1     0.20                        addw	%si, %di
-# CHECK-NEXT:  4      7     0.50    *      *            addw	%si, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		addw	%si, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            addw	%si, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		addw	%si, (%rax)
 # CHECK-NEXT:  2      6     0.33    *                   addw	(%rax), %di
 # CHECK-NEXT:  1      1     0.20                        addl	$665536, %eax
 # CHECK-NEXT:  1      1     0.20                        addl	$665536, %edi
-# CHECK-NEXT:  4      7     0.50    *      *            addl	$665536, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		addl	$665536, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            addl	$665536, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		addl	$665536, (%rax)
 # CHECK-NEXT:  1      1     0.20                        addl	$7, %edi
-# CHECK-NEXT:  4      7     0.50    *      *            addl	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		addl	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            addl	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		addl	$7, (%rax)
 # CHECK-NEXT:  1      1     0.20                        addl	%esi, %edi
-# CHECK-NEXT:  4      7     0.50    *      *            addl	%esi, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		addl	%esi, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            addl	%esi, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		addl	%esi, (%rax)
 # CHECK-NEXT:  2      6     0.33    *                   addl	(%rax), %edi
 # CHECK-NEXT:  1      1     0.20                        addq	$665536, %rax
 # CHECK-NEXT:  1      1     0.20                        addq	$665536, %rdi
-# CHECK-NEXT:  4      7     0.50    *      *            addq	$665536, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		addq	$665536, (%rax)
-# CHECK-NEXT:  1      1     0.20                        addq	$7, %rdi
-# CHECK-NEXT:  4      7     0.50    *      *            addq	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		addq	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            addq	$665536, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		addq	$665536, (%rax)
+# CHECK-NEXT:  0      1     0.00                        addq	$7, %rdi
+# CHECK-NEXT:  4      12    0.50    *      *            addq	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		addq	$7, (%rax)
 # CHECK-NEXT:  1      1     0.20                        addq	%rsi, %rdi
-# CHECK-NEXT:  4      7     0.50    *      *            addq	%rsi, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		addq	%rsi, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            addq	%rsi, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		addq	%rsi, (%rax)
 # CHECK-NEXT:  2      6     0.33    *                   addq	(%rax), %rdi
-# CHECK-NEXT:  1      1     0.20                        andb	$7, %al
-# CHECK-NEXT:  1      1     0.20                        andb	$7, %dil
-# CHECK-NEXT:  4      7     0.50    *      *            andb	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		andb	$7, (%rax)
-# CHECK-NEXT:  1      1     0.20                        andb	%sil, %dil
-# CHECK-NEXT:  4      7     0.50    *      *            andb	%sil, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		andb	%sil, (%rax)
+# CHECK-NEXT:  1      2     0.20                        andb	$7, %al
+# CHECK-NEXT:  1      2     0.20                        andb	$7, %dil
+# CHECK-NEXT:  4      13    0.50    *      *            andb	$7, (%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            lock		andb	$7, (%rax)
+# CHECK-NEXT:  1      2     0.20                        andb	%sil, %dil
+# CHECK-NEXT:  4      13    0.50    *      *            andb	%sil, (%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            lock		andb	%sil, (%rax)
 # CHECK-NEXT:  2      6     0.33    *                   andb	(%rax), %dil
 # CHECK-NEXT:  1      1     0.20                        andw	$511, %ax
 # CHECK-NEXT:  1      1     0.20                        andw	$511, %di
-# CHECK-NEXT:  4      7     0.50    *      *            andw	$511, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		andw	$511, (%rax)
-# CHECK-NEXT:  1      1     0.20                        andw	$7, %di
-# CHECK-NEXT:  4      7     0.50    *      *            andw	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		andw	$7, (%rax)
-# CHECK-NEXT:  1      1     0.20                        andw	%si, %di
-# CHECK-NEXT:  4      7     0.50    *      *            andw	%si, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		andw	%si, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            andw	$511, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		andw	$511, (%rax)
+# CHECK-NEXT:  1      2     0.20                        andw	$7, %di
+# CHECK-NEXT:  4      12    0.50    *      *            andw	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		andw	$7, (%rax)
+# CHECK-NEXT:  1      2     0.20                        andw	%si, %di
+# CHECK-NEXT:  4      12    0.50    *      *            andw	%si, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		andw	%si, (%rax)
 # CHECK-NEXT:  2      6     0.33    *                   andw	(%rax), %di
 # CHECK-NEXT:  1      2     0.20                        andl	$665536, %eax
-# CHECK-NEXT:  1      1     0.20                        andl	$665536, %edi
-# CHECK-NEXT:  4      7     0.50    *      *            andl	$665536, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		andl	$665536, (%rax)
-# CHECK-NEXT:  1      1     0.20                        andl	$7, %edi
-# CHECK-NEXT:  4      7     0.50    *      *            andl	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		andl	$7, (%rax)
-# CHECK-NEXT:  1      1     0.20                        andl	%esi, %edi
-# CHECK-NEXT:  4      7     0.50    *      *            andl	%esi, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		andl	%esi, (%rax)
+# CHECK-NEXT:  1      2     0.20                        andl	$665536, %edi
+# CHECK-NEXT:  4      12    0.50    *      *            andl	$665536, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		andl	$665536, (%rax)
+# CHECK-NEXT:  1      2     0.20                        andl	$7, %edi
+# CHECK-NEXT:  4      12    0.50    *      *            andl	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		andl	$7, (%rax)
+# CHECK-NEXT:  1      2     0.20                        andl	%esi, %edi
+# CHECK-NEXT:  4      12    0.50    *      *            andl	%esi, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		andl	%esi, (%rax)
 # CHECK-NEXT:  2      6     0.33    *                   andl	(%rax), %edi
 # CHECK-NEXT:  1      2     0.20                        andq	$665536, %rax
-# CHECK-NEXT:  1      1     0.20                        andq	$665536, %rdi
-# CHECK-NEXT:  4      7     0.50    *      *            andq	$665536, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		andq	$665536, (%rax)
-# CHECK-NEXT:  1      1     0.20                        andq	$7, %rdi
-# CHECK-NEXT:  4      7     0.50    *      *            andq	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		andq	$7, (%rax)
-# CHECK-NEXT:  1      1     0.20                        andq	%rsi, %rdi
-# CHECK-NEXT:  4      7     0.50    *      *            andq	%rsi, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		andq	%rsi, (%rax)
-# CHECK-NEXT:  2      6     0.33    *                   andq	(%rax), %rdi
+# CHECK-NEXT:  1      2     0.20                        andq	$665536, %rdi
+# CHECK-NEXT:  4      12    0.50    *      *            andq	$665536, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		andq	$665536, (%rax)
+# CHECK-NEXT:  1      2     0.20                        andq	$7, %rdi
+# CHECK-NEXT:  4      12    0.50    *      *            andq	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		andq	$7, (%rax)
+# CHECK-NEXT:  1      2     0.20                        andq	%rsi, %rdi
+# CHECK-NEXT:  4      12    0.50    *      *            andq	%rsi, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		andq	%rsi, (%rax)
+# CHECK-NEXT:  2      7     0.33    *                   andq	(%rax), %rdi
 # CHECK-NEXT:  1      3     1.00                        bsfw	%si, %di
 # CHECK-NEXT:  1      3     1.00                        bsrw	%si, %di
 # CHECK-NEXT:  2      8     1.00    *                   bsfw	(%rax), %di
@@ -1203,12 +1203,12 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  1      1     1.00                        btrw	$7, %di
 # CHECK-NEXT:  1      1     1.00                        btsw	$7, %di
 # CHECK-NEXT:  2      6     1.00    *                   btw	$7, (%rax)
-# CHECK-NEXT:  4      7     1.00    *      *            btcw	$7, (%rax)
-# CHECK-NEXT:  4      7     1.00    *      *            btrw	$7, (%rax)
-# CHECK-NEXT:  4      7     1.00    *      *            btsw	$7, (%rax)
-# CHECK-NEXT:  4      7     1.00    *      *            lock		btcw	$7, (%rax)
-# CHECK-NEXT:  4      7     1.00    *      *            lock		btrw	$7, (%rax)
-# CHECK-NEXT:  4      7     1.00    *      *            lock		btsw	$7, (%rax)
+# CHECK-NEXT:  4      12    1.00    *      *            btcw	$7, (%rax)
+# CHECK-NEXT:  4      12    1.00    *      *            btrw	$7, (%rax)
+# CHECK-NEXT:  4      12    1.00    *      *            btsw	$7, (%rax)
+# CHECK-NEXT:  4      12    1.00    *      *            lock		btcw	$7, (%rax)
+# CHECK-NEXT:  4      12    1.00    *      *            lock		btrw	$7, (%rax)
+# CHECK-NEXT:  4      12    1.00    *      *            lock		btsw	$7, (%rax)
 # CHECK-NEXT:  1      1     1.00                        btl	%esi, %edi
 # CHECK-NEXT:  1      1     1.00                        btcl	%esi, %edi
 # CHECK-NEXT:  1      1     1.00                        btrl	%esi, %edi
@@ -1225,16 +1225,16 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  1      1     1.00                        btrl	$7, %edi
 # CHECK-NEXT:  1      1     1.00                        btsl	$7, %edi
 # CHECK-NEXT:  2      6     1.00    *                   btl	$7, (%rax)
-# CHECK-NEXT:  4      7     1.00    *      *            btcl	$7, (%rax)
-# CHECK-NEXT:  4      7     1.00    *      *            btrl	$7, (%rax)
-# CHECK-NEXT:  4      7     1.00    *      *            btsl	$7, (%rax)
-# CHECK-NEXT:  4      7     1.00    *      *            lock		btcl	$7, (%rax)
-# CHECK-NEXT:  4      7     1.00    *      *            lock		btrl	$7, (%rax)
-# CHECK-NEXT:  4      7     1.00    *      *            lock		btsl	$7, (%rax)
-# CHECK-NEXT:  1      1     1.00                        btq	%rsi, %rdi
-# CHECK-NEXT:  1      1     1.00                        btcq	%rsi, %rdi
-# CHECK-NEXT:  1      1     1.00                        btrq	%rsi, %rdi
-# CHECK-NEXT:  1      1     1.00                        btsq	%rsi, %rdi
+# CHECK-NEXT:  4      12    1.00    *      *            btcl	$7, (%rax)
+# CHECK-NEXT:  4      12    1.00    *      *            btrl	$7, (%rax)
+# CHECK-NEXT:  4      12    1.00    *      *            btsl	$7, (%rax)
+# CHECK-NEXT:  4      12    1.00    *      *            lock		btcl	$7, (%rax)
+# CHECK-NEXT:  4      12    1.00    *      *            lock		btrl	$7, (%rax)
+# CHECK-NEXT:  4      12    1.00    *      *            lock		btsl	$7, (%rax)
+# CHECK-NEXT:  1      3     1.00                        btq	%rsi, %rdi
+# CHECK-NEXT:  1      3     1.00                        btcq	%rsi, %rdi
+# CHECK-NEXT:  1      3     1.00                        btrq	%rsi, %rdi
+# CHECK-NEXT:  1      3     1.00                        btsq	%rsi, %rdi
 # CHECK-NEXT:  9      10    1.60    *                   btq	%rsi, (%rax)
 # CHECK-NEXT:  10     17    1.40    *      *            btcq	%rsi, (%rax)
 # CHECK-NEXT:  10     17    1.40    *      *            btrq	%rsi, (%rax)
@@ -1247,15 +1247,15 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  1      1     1.00                        btrq	$7, %rdi
 # CHECK-NEXT:  1      1     1.00                        btsq	$7, %rdi
 # CHECK-NEXT:  2      6     1.00    *                   btq	$7, (%rax)
-# CHECK-NEXT:  4      7     1.00    *      *            btcq	$7, (%rax)
-# CHECK-NEXT:  4      7     1.00    *      *            btrq	$7, (%rax)
-# CHECK-NEXT:  4      7     1.00    *      *            btsq	$7, (%rax)
-# CHECK-NEXT:  4      7     1.00    *      *            lock		btcq	$7, (%rax)
-# CHECK-NEXT:  4      7     1.00    *      *            lock		btrq	$7, (%rax)
-# CHECK-NEXT:  4      7     1.00    *      *            lock		btsq	$7, (%rax)
-# CHECK-NEXT:  1      1     0.33                        cbtw
-# CHECK-NEXT:  1      1     0.33                        cwtl
-# CHECK-NEXT:  1      1     0.33                        cltq
+# CHECK-NEXT:  4      12    1.00    *      *            btcq	$7, (%rax)
+# CHECK-NEXT:  4      12    1.00    *      *            btrq	$7, (%rax)
+# CHECK-NEXT:  4      12    1.00    *      *            btsq	$7, (%rax)
+# CHECK-NEXT:  4      12    1.00    *      *            lock		btcq	$7, (%rax)
+# CHECK-NEXT:  4      12    1.00    *      *            lock		btrq	$7, (%rax)
+# CHECK-NEXT:  4      12    1.00    *      *            lock		btsq	$7, (%rax)
+# CHECK-NEXT:  1      1     0.50                        cbtw
+# CHECK-NEXT:  1      1     0.50                        cwtl
+# CHECK-NEXT:  1      1     0.50                        cltq
 # CHECK-NEXT:  2      2     0.50                        cwtd
 # CHECK-NEXT:  1      1     0.50                        cltd
 # CHECK-NEXT:  1      1     0.50                        cqto
@@ -1310,34 +1310,34 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  6      12    1.00    *      *            lock		cmpxchgq	%rcx, (%rbx)
 # CHECK-NEXT:  26     18    6.00                  U     cpuid
 # CHECK-NEXT:  1      1     0.20                        decb	%dil
-# CHECK-NEXT:  4      7     0.50    *      *            decb	(%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		decb	(%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            decb	(%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            lock		decb	(%rax)
 # CHECK-NEXT:  1      1     0.20                        decw	%di
-# CHECK-NEXT:  4      7     0.50    *      *            decw	(%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		decw	(%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            decw	(%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		decw	(%rax)
 # CHECK-NEXT:  1      1     0.20                        decl	%edi
-# CHECK-NEXT:  4      7     0.50    *      *            decl	(%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		decl	(%rax)
-# CHECK-NEXT:  1      1     0.20                        decq	%rdi
-# CHECK-NEXT:  4      7     0.50    *      *            decq	(%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		decq	(%rax)
-# CHECK-NEXT:  4      17    3.00                  U     divb	%dil
-# CHECK-NEXT:  5      22    3.00    *             U     divb	(%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            decl	(%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		decl	(%rax)
+# CHECK-NEXT:  0      1     0.00                        decq	%rdi
+# CHECK-NEXT:  4      12    0.50    *      *            decq	(%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		decq	(%rax)
+# CHECK-NEXT:  3      17    3.00                  U     divb	%dil
+# CHECK-NEXT:  3      22    3.00    *             U     divb	(%rax)
 # CHECK-NEXT:  4      16    3.00                  U     divw	%si
 # CHECK-NEXT:  5      20    3.00    *             U     divw	(%rax)
 # CHECK-NEXT:  4      15    3.00                  U     divl	%edx
 # CHECK-NEXT:  5      19    3.00    *             U     divl	(%rax)
-# CHECK-NEXT:  5      18    3.00                  U     divq	%rcx
-# CHECK-NEXT:  5      23    3.00    *             U     divq	(%rax)
+# CHECK-NEXT:  3      18    3.00                  U     divq	%rcx
+# CHECK-NEXT:  4      23    3.00    *             U     divq	(%rax)
 # CHECK-NEXT:  57     126   11.50                 U     enter	$7, $4095
-# CHECK-NEXT:  4      17    3.00                  U     idivb	%dil
-# CHECK-NEXT:  5      22    3.00    *             U     idivb	(%rax)
+# CHECK-NEXT:  3      17    3.00                  U     idivb	%dil
+# CHECK-NEXT:  3      22    3.00    *             U     idivb	(%rax)
 # CHECK-NEXT:  4      16    3.00                  U     idivw	%si
 # CHECK-NEXT:  5      20    3.00    *             U     idivw	(%rax)
 # CHECK-NEXT:  4      15    3.00                  U     idivl	%edx
 # CHECK-NEXT:  5      19    3.00    *             U     idivl	(%rax)
-# CHECK-NEXT:  5      18    3.00                  U     idivq	%rcx
-# CHECK-NEXT:  5      23    3.00    *             U     idivq	(%rax)
+# CHECK-NEXT:  3      18    3.00                  U     idivq	%rcx
+# CHECK-NEXT:  4      23    3.00    *             U     idivq	(%rax)
 # CHECK-NEXT:  1      3     1.00                        imulb	%dil
 # CHECK-NEXT:  2      8     1.00    *                   imulb	(%rax)
 # CHECK-NEXT:  4      5     1.00                        imulw	%di
@@ -1369,19 +1369,19 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  87     35    21.00                 U     inw	$7, %ax
 # CHECK-NEXT:  87     35    20.00                 U     inw	%dx, %ax
 # CHECK-NEXT:  94     35    21.00                 U     inl	$7, %eax
-# CHECK-NEXT:  2      1     0.20                  U     inl	%dx, %eax
+# CHECK-NEXT:  99     1     21.00                 U     inl	%dx, %eax
 # CHECK-NEXT:  1      1     0.20                        incb	%dil
-# CHECK-NEXT:  4      7     0.50    *      *            incb	(%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		incb	(%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            incb	(%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            lock		incb	(%rax)
 # CHECK-NEXT:  1      1     0.20                        incw	%di
-# CHECK-NEXT:  4      7     0.50    *      *            incw	(%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		incw	(%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            incw	(%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		incw	(%rax)
 # CHECK-NEXT:  1      1     0.20                        incl	%edi
-# CHECK-NEXT:  4      7     0.50    *      *            incl	(%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		incl	(%rax)
-# CHECK-NEXT:  1      1     0.20                        incq	%rdi
-# CHECK-NEXT:  4      7     0.50    *      *            incq	(%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		incq	(%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            incl	(%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		incl	(%rax)
+# CHECK-NEXT:  0      1     0.00                        incq	%rdi
+# CHECK-NEXT:  4      12    0.50    *      *            incq	(%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		incq	(%rax)
 # CHECK-NEXT:  83     20    19.00                 U     insb	%dx, %es:(%rdi)
 # CHECK-NEXT:  86     20    19.00                 U     insw	%dx, %es:(%rdi)
 # CHECK-NEXT:  92     20    21.00                 U     insl	%dx, %es:(%rdi)
@@ -1432,17 +1432,17 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  2      4     1.00                        mulq	%rcx
 # CHECK-NEXT:  3      9     1.00    *                   mulq	(%rax)
 # CHECK-NEXT:  1      1     0.20                        negb	%dil
-# CHECK-NEXT:  4      7     0.50    *      *            negb	(%r8)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		negb	(%r8)
+# CHECK-NEXT:  4      13    0.50    *      *            negb	(%r8)
+# CHECK-NEXT:  4      13    0.50    *      *            lock		negb	(%r8)
 # CHECK-NEXT:  1      1     0.20                        negw	%si
-# CHECK-NEXT:  4      7     0.50    *      *            negw	(%r9)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		negw	(%r9)
+# CHECK-NEXT:  4      12    0.50    *      *            negw	(%r9)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		negw	(%r9)
 # CHECK-NEXT:  1      1     0.20                        negl	%edx
-# CHECK-NEXT:  4      7     0.50    *      *            negl	(%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		negl	(%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            negl	(%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		negl	(%rax)
 # CHECK-NEXT:  1      1     0.20                        negq	%rcx
-# CHECK-NEXT:  4      7     0.50    *      *            negq	(%r10)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		negq	(%r10)
+# CHECK-NEXT:  4      12    0.50    *      *            negq	(%r10)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		negq	(%r10)
 # CHECK-NEXT:  0      1     0.00                        nop
 # CHECK-NEXT:  0      1     0.00                        nopw	%di
 # CHECK-NEXT:  0      1     0.00                        nopw	(%rcx)
@@ -1451,58 +1451,58 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  0      1     0.00                        nopq	%rdx
 # CHECK-NEXT:  0      1     0.00                        nopq	(%r9)
 # CHECK-NEXT:  1      1     0.20                        notb	%dil
-# CHECK-NEXT:  4      7     0.50    *      *            notb	(%r8)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		notb	(%r8)
+# CHECK-NEXT:  4      13    0.50    *      *            notb	(%r8)
+# CHECK-NEXT:  4      13    0.50    *      *            lock		notb	(%r8)
 # CHECK-NEXT:  1      1     0.20                        notw	%si
-# CHECK-NEXT:  4      7     0.50    *      *            notw	(%r9)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		notw	(%r9)
+# CHECK-NEXT:  4      12    0.50    *      *            notw	(%r9)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		notw	(%r9)
 # CHECK-NEXT:  1      1     0.20                        notl	%edx
-# CHECK-NEXT:  4      7     0.50    *      *            notl	(%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		notl	(%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            notl	(%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		notl	(%rax)
 # CHECK-NEXT:  1      1     0.20                        notq	%rcx
-# CHECK-NEXT:  4      7     0.50    *      *            notq	(%r10)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		notq	(%r10)
-# CHECK-NEXT:  1      1     0.20                        orb	$7, %al
-# CHECK-NEXT:  1      1     0.20                        orb	$7, %dil
-# CHECK-NEXT:  4      7     0.50    *      *            orb	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		orb	$7, (%rax)
-# CHECK-NEXT:  1      1     0.20                        orb	%sil, %dil
-# CHECK-NEXT:  4      7     0.50    *      *            orb	%sil, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		orb	%sil, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            notq	(%r10)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		notq	(%r10)
+# CHECK-NEXT:  1      2     0.20                        orb	$7, %al
+# CHECK-NEXT:  1      2     0.20                        orb	$7, %dil
+# CHECK-NEXT:  4      13    0.50    *      *            orb	$7, (%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            lock		orb	$7, (%rax)
+# CHECK-NEXT:  1      2     0.20                        orb	%sil, %dil
+# CHECK-NEXT:  4      13    0.50    *      *            orb	%sil, (%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            lock		orb	%sil, (%rax)
 # CHECK-NEXT:  2      6     0.33    *                   orb	(%rax), %dil
 # CHECK-NEXT:  1      1     0.20                        orw	$511, %ax
 # CHECK-NEXT:  1      1     0.20                        orw	$511, %di
-# CHECK-NEXT:  4      7     0.50    *      *            orw	$511, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		orw	$511, (%rax)
-# CHECK-NEXT:  1      1     0.20                        orw	$7, %di
-# CHECK-NEXT:  4      7     0.50    *      *            orw	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		orw	$7, (%rax)
-# CHECK-NEXT:  1      1     0.20                        orw	%si, %di
-# CHECK-NEXT:  4      7     0.50    *      *            orw	%si, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		orw	%si, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            orw	$511, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		orw	$511, (%rax)
+# CHECK-NEXT:  1      2     0.20                        orw	$7, %di
+# CHECK-NEXT:  4      12    0.50    *      *            orw	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		orw	$7, (%rax)
+# CHECK-NEXT:  1      2     0.20                        orw	%si, %di
+# CHECK-NEXT:  4      12    0.50    *      *            orw	%si, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		orw	%si, (%rax)
 # CHECK-NEXT:  2      6     0.33    *                   orw	(%rax), %di
 # CHECK-NEXT:  1      2     0.20                        orl	$665536, %eax
-# CHECK-NEXT:  1      1     0.20                        orl	$665536, %edi
-# CHECK-NEXT:  4      7     0.50    *      *            orl	$665536, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		orl	$665536, (%rax)
-# CHECK-NEXT:  1      1     0.20                        orl	$7, %edi
-# CHECK-NEXT:  4      7     0.50    *      *            orl	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		orl	$7, (%rax)
-# CHECK-NEXT:  1      1     0.20                        orl	%esi, %edi
-# CHECK-NEXT:  4      7     0.50    *      *            orl	%esi, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		orl	%esi, (%rax)
+# CHECK-NEXT:  1      2     0.20                        orl	$665536, %edi
+# CHECK-NEXT:  4      12    0.50    *      *            orl	$665536, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		orl	$665536, (%rax)
+# CHECK-NEXT:  1      2     0.20                        orl	$7, %edi
+# CHECK-NEXT:  4      12    0.50    *      *            orl	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		orl	$7, (%rax)
+# CHECK-NEXT:  1      2     0.20                        orl	%esi, %edi
+# CHECK-NEXT:  4      12    0.50    *      *            orl	%esi, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		orl	%esi, (%rax)
 # CHECK-NEXT:  2      6     0.33    *                   orl	(%rax), %edi
 # CHECK-NEXT:  1      2     0.20                        orq	$665536, %rax
-# CHECK-NEXT:  1      1     0.20                        orq	$665536, %rdi
-# CHECK-NEXT:  4      7     0.50    *      *            orq	$665536, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		orq	$665536, (%rax)
-# CHECK-NEXT:  1      1     0.20                        orq	$7, %rdi
-# CHECK-NEXT:  4      7     0.50    *      *            orq	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		orq	$7, (%rax)
-# CHECK-NEXT:  1      1     0.20                        orq	%rsi, %rdi
-# CHECK-NEXT:  4      7     0.50    *      *            orq	%rsi, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		orq	%rsi, (%rax)
-# CHECK-NEXT:  2      6     0.33    *                   orq	(%rax), %rdi
+# CHECK-NEXT:  1      2     0.20                        orq	$665536, %rdi
+# CHECK-NEXT:  4      12    0.50    *      *            orq	$665536, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		orq	$665536, (%rax)
+# CHECK-NEXT:  1      2     0.20                        orq	$7, %rdi
+# CHECK-NEXT:  4      12    0.50    *      *            orq	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		orq	$7, (%rax)
+# CHECK-NEXT:  1      2     0.20                        orq	%rsi, %rdi
+# CHECK-NEXT:  4      12    0.50    *      *            orq	%rsi, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		orq	%rsi, (%rax)
+# CHECK-NEXT:  2      7     0.33    *                   orq	(%rax), %rdi
 # CHECK-NEXT:  73     35    15.33                 U     outb	%al, $7
 # CHECK-NEXT:  73     35    15.50                 U     outb	%al, %dx
 # CHECK-NEXT:  79     35    17.00                 U     outw	%ax, $7
@@ -1569,50 +1569,50 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  2      1     1.00                        rorb	%dil
 # CHECK-NEXT:  5      13    1.00    *      *            rolb	(%rax)
 # CHECK-NEXT:  5      13    1.00    *      *            rorb	(%rax)
-# CHECK-NEXT:  1      1     0.50                        rolb	$7, %dil
-# CHECK-NEXT:  3      1     0.60                        rorb	$7, %dil
-# CHECK-NEXT:  5      7     1.00    *      *            rolb	$7, (%rax)
-# CHECK-NEXT:  5      7     1.00    *      *            rorb	$7, (%rax)
-# CHECK-NEXT:  2      1     1.00                        rolb	%cl, %dil
-# CHECK-NEXT:  2      1     1.00                        rorb	%cl, %dil
-# CHECK-NEXT:  5      7     1.00    *      *            rolb	%cl, (%rax)
-# CHECK-NEXT:  5      7     1.00    *      *            rorb	%cl, (%rax)
+# CHECK-NEXT:  2      1     1.00                        rolb	$7, %dil
+# CHECK-NEXT:  2      1     1.00                        rorb	$7, %dil
+# CHECK-NEXT:  5      13    1.00    *      *            rolb	$7, (%rax)
+# CHECK-NEXT:  5      13    1.00    *      *            rorb	$7, (%rax)
+# CHECK-NEXT:  2      2     1.00                        rolb	%cl, %dil
+# CHECK-NEXT:  2      2     1.00                        rorb	%cl, %dil
+# CHECK-NEXT:  5      13    1.00    *      *            rolb	%cl, (%rax)
+# CHECK-NEXT:  5      13    1.00    *      *            rorb	%cl, (%rax)
 # CHECK-NEXT:  2      1     1.00                        rolw	%di
 # CHECK-NEXT:  2      1     1.00                        rorw	%di
 # CHECK-NEXT:  5      12    1.00    *      *            rolw	(%rax)
 # CHECK-NEXT:  5      12    1.00    *      *            rorw	(%rax)
-# CHECK-NEXT:  1      1     0.50                        rolw	$7, %di
-# CHECK-NEXT:  1      1     0.50                        rorw	$7, %di
-# CHECK-NEXT:  5      7     1.00    *      *            rolw	$7, (%rax)
-# CHECK-NEXT:  5      7     1.00    *      *            rorw	$7, (%rax)
-# CHECK-NEXT:  2      1     1.00                        rolw	%cl, %di
-# CHECK-NEXT:  2      1     1.00                        rorw	%cl, %di
-# CHECK-NEXT:  5      7     1.00    *      *            rolw	%cl, (%rax)
-# CHECK-NEXT:  5      7     1.00    *      *            rorw	%cl, (%rax)
+# CHECK-NEXT:  2      1     1.00                        rolw	$7, %di
+# CHECK-NEXT:  2      1     1.00                        rorw	$7, %di
+# CHECK-NEXT:  5      12    1.00    *      *            rolw	$7, (%rax)
+# CHECK-NEXT:  5      12    1.00    *      *            rorw	$7, (%rax)
+# CHECK-NEXT:  2      2     1.00                        rolw	%cl, %di
+# CHECK-NEXT:  2      2     1.00                        rorw	%cl, %di
+# CHECK-NEXT:  5      12    1.00    *      *            rolw	%cl, (%rax)
+# CHECK-NEXT:  5      12    1.00    *      *            rorw	%cl, (%rax)
 # CHECK-NEXT:  2      1     1.00                        roll	%edi
 # CHECK-NEXT:  2      1     1.00                        rorl	%edi
 # CHECK-NEXT:  5      12    1.00    *      *            roll	(%rax)
 # CHECK-NEXT:  5      12    1.00    *      *            rorl	(%rax)
-# CHECK-NEXT:  1      1     0.50                        roll	$7, %edi
-# CHECK-NEXT:  1      1     0.50                        rorl	$7, %edi
-# CHECK-NEXT:  5      7     1.00    *      *            roll	$7, (%rax)
-# CHECK-NEXT:  5      7     1.00    *      *            rorl	$7, (%rax)
-# CHECK-NEXT:  2      1     1.00                        roll	%cl, %edi
-# CHECK-NEXT:  2      1     1.00                        rorl	%cl, %edi
-# CHECK-NEXT:  5      7     1.00    *      *            roll	%cl, (%rax)
-# CHECK-NEXT:  5      7     1.00    *      *            rorl	%cl, (%rax)
+# CHECK-NEXT:  2      1     1.00                        roll	$7, %edi
+# CHECK-NEXT:  2      1     1.00                        rorl	$7, %edi
+# CHECK-NEXT:  5      12    1.00    *      *            roll	$7, (%rax)
+# CHECK-NEXT:  5      12    1.00    *      *            rorl	$7, (%rax)
+# CHECK-NEXT:  2      2     1.00                        roll	%cl, %edi
+# CHECK-NEXT:  2      2     1.00                        rorl	%cl, %edi
+# CHECK-NEXT:  5      12    1.00    *      *            roll	%cl, (%rax)
+# CHECK-NEXT:  5      12    1.00    *      *            rorl	%cl, (%rax)
 # CHECK-NEXT:  2      1     1.00                        rolq	%rdi
 # CHECK-NEXT:  2      1     1.00                        rorq	%rdi
 # CHECK-NEXT:  5      12    1.00    *      *            rolq	(%rax)
 # CHECK-NEXT:  5      12    1.00    *      *            rorq	(%rax)
-# CHECK-NEXT:  1      1     0.50                        rolq	$7, %rdi
-# CHECK-NEXT:  1      1     0.50                        rorq	$7, %rdi
-# CHECK-NEXT:  5      7     1.00    *      *            rolq	$7, (%rax)
-# CHECK-NEXT:  5      7     1.00    *      *            rorq	$7, (%rax)
-# CHECK-NEXT:  2      1     1.00                        rolq	%cl, %rdi
-# CHECK-NEXT:  2      1     1.00                        rorq	%cl, %rdi
-# CHECK-NEXT:  5      7     1.00    *      *            rolq	%cl, (%rax)
-# CHECK-NEXT:  5      7     1.00    *      *            rorq	%cl, (%rax)
+# CHECK-NEXT:  2      1     1.00                        rolq	$7, %rdi
+# CHECK-NEXT:  2      1     1.00                        rorq	$7, %rdi
+# CHECK-NEXT:  5      12    1.00    *      *            rolq	$7, (%rax)
+# CHECK-NEXT:  5      12    1.00    *      *            rorq	$7, (%rax)
+# CHECK-NEXT:  2      2     1.00                        rolq	%cl, %rdi
+# CHECK-NEXT:  2      2     1.00                        rorq	%cl, %rdi
+# CHECK-NEXT:  5      12    1.00    *      *            rolq	%cl, (%rax)
+# CHECK-NEXT:  5      12    1.00    *      *            rorq	%cl, (%rax)
 # CHECK-NEXT:  2      4     1.00                        sahf
 # CHECK-NEXT:  1      1     0.50                        sarb	%dil
 # CHECK-NEXT:  1      1     0.50                        shlb	%dil
@@ -1623,15 +1623,15 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  1      1     0.50                        sarb	$7, %dil
 # CHECK-NEXT:  1      1     0.50                        shlb	$7, %dil
 # CHECK-NEXT:  1      1     0.50                        shrb	$7, %dil
-# CHECK-NEXT:  4      7     0.50    *      *            sarb	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            shlb	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            shrb	$7, (%rax)
-# CHECK-NEXT:  2      1     1.00                        sarb	%cl, %dil
-# CHECK-NEXT:  2      1     1.00                        shlb	%cl, %dil
-# CHECK-NEXT:  2      1     1.00                        shrb	%cl, %dil
-# CHECK-NEXT:  5      7     1.00    *      *            sarb	%cl, (%rax)
-# CHECK-NEXT:  5      7     1.00    *      *            shlb	%cl, (%rax)
-# CHECK-NEXT:  5      7     1.00    *      *            shrb	%cl, (%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            sarb	$7, (%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            shlb	$7, (%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            shrb	$7, (%rax)
+# CHECK-NEXT:  2      2     1.00                        sarb	%cl, %dil
+# CHECK-NEXT:  2      2     1.00                        shlb	%cl, %dil
+# CHECK-NEXT:  2      2     1.00                        shrb	%cl, %dil
+# CHECK-NEXT:  5      13    1.00    *      *            sarb	%cl, (%rax)
+# CHECK-NEXT:  5      13    1.00    *      *            shlb	%cl, (%rax)
+# CHECK-NEXT:  5      13    1.00    *      *            shrb	%cl, (%rax)
 # CHECK-NEXT:  1      1     0.50                        sarw	%di
 # CHECK-NEXT:  1      1     0.50                        shlw	%di
 # CHECK-NEXT:  1      1     0.50                        shrw	%di
@@ -1641,15 +1641,15 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  1      1     0.50                        sarw	$7, %di
 # CHECK-NEXT:  1      1     0.50                        shlw	$7, %di
 # CHECK-NEXT:  1      1     0.50                        shrw	$7, %di
-# CHECK-NEXT:  4      7     0.50    *      *            sarw	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            shlw	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            shrw	$7, (%rax)
-# CHECK-NEXT:  2      1     1.00                        sarw	%cl, %di
-# CHECK-NEXT:  2      1     1.00                        shlw	%cl, %di
-# CHECK-NEXT:  2      1     1.00                        shrw	%cl, %di
-# CHECK-NEXT:  5      7     1.00    *      *            sarw	%cl, (%rax)
-# CHECK-NEXT:  5      7     1.00    *      *            shlw	%cl, (%rax)
-# CHECK-NEXT:  5      7     1.00    *      *            shrw	%cl, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            sarw	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            shlw	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            shrw	$7, (%rax)
+# CHECK-NEXT:  2      2     1.00                        sarw	%cl, %di
+# CHECK-NEXT:  2      2     1.00                        shlw	%cl, %di
+# CHECK-NEXT:  2      2     1.00                        shrw	%cl, %di
+# CHECK-NEXT:  5      12    1.00    *      *            sarw	%cl, (%rax)
+# CHECK-NEXT:  5      12    1.00    *      *            shlw	%cl, (%rax)
+# CHECK-NEXT:  5      12    1.00    *      *            shrw	%cl, (%rax)
 # CHECK-NEXT:  1      1     0.50                        sarl	%edi
 # CHECK-NEXT:  1      1     0.50                        shll	%edi
 # CHECK-NEXT:  1      1     0.50                        shrl	%edi
@@ -1659,15 +1659,15 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  1      1     0.50                        sarl	$7, %edi
 # CHECK-NEXT:  1      1     0.50                        shll	$7, %edi
 # CHECK-NEXT:  1      1     0.50                        shrl	$7, %edi
-# CHECK-NEXT:  4      7     0.50    *      *            sarl	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            shll	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            shrl	$7, (%rax)
-# CHECK-NEXT:  2      1     1.00                        sarl	%cl, %edi
-# CHECK-NEXT:  2      1     1.00                        shll	%cl, %edi
-# CHECK-NEXT:  2      1     1.00                        shrl	%cl, %edi
-# CHECK-NEXT:  5      7     1.00    *      *            sarl	%cl, (%rax)
-# CHECK-NEXT:  5      7     1.00    *      *            shll	%cl, (%rax)
-# CHECK-NEXT:  5      7     1.00    *      *            shrl	%cl, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            sarl	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            shll	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            shrl	$7, (%rax)
+# CHECK-NEXT:  2      2     1.00                        sarl	%cl, %edi
+# CHECK-NEXT:  2      2     1.00                        shll	%cl, %edi
+# CHECK-NEXT:  2      2     1.00                        shrl	%cl, %edi
+# CHECK-NEXT:  5      12    1.00    *      *            sarl	%cl, (%rax)
+# CHECK-NEXT:  5      12    1.00    *      *            shll	%cl, (%rax)
+# CHECK-NEXT:  5      12    1.00    *      *            shrl	%cl, (%rax)
 # CHECK-NEXT:  1      1     0.50                        sarq	%rdi
 # CHECK-NEXT:  1      1     0.50                        shlq	%rdi
 # CHECK-NEXT:  1      1     0.50                        shrq	%rdi
@@ -1677,15 +1677,15 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  1      1     0.50                        sarq	$7, %rdi
 # CHECK-NEXT:  1      1     0.50                        shlq	$7, %rdi
 # CHECK-NEXT:  1      1     0.50                        shrq	$7, %rdi
-# CHECK-NEXT:  4      7     0.50    *      *            sarq	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            shlq	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            shrq	$7, (%rax)
-# CHECK-NEXT:  2      1     1.00                        sarq	%cl, %rdi
-# CHECK-NEXT:  2      1     1.00                        shlq	%cl, %rdi
-# CHECK-NEXT:  2      1     1.00                        shrq	%cl, %rdi
-# CHECK-NEXT:  5      7     1.00    *      *            sarq	%cl, (%rax)
-# CHECK-NEXT:  5      7     1.00    *      *            shlq	%cl, (%rax)
-# CHECK-NEXT:  5      7     1.00    *      *            shrq	%cl, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            sarq	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            shlq	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            shrq	$7, (%rax)
+# CHECK-NEXT:  2      2     1.00                        sarq	%cl, %rdi
+# CHECK-NEXT:  2      2     1.00                        shlq	%cl, %rdi
+# CHECK-NEXT:  2      2     1.00                        shrq	%cl, %rdi
+# CHECK-NEXT:  5      12    1.00    *      *            sarq	%cl, (%rax)
+# CHECK-NEXT:  5      12    1.00    *      *            shlq	%cl, (%rax)
+# CHECK-NEXT:  5      12    1.00    *      *            shrq	%cl, (%rax)
 # CHECK-NEXT:  1      1     0.50                        sbbb	$0, %al
 # CHECK-NEXT:  1      1     0.50                        sbbb	$0, %dil
 # CHECK-NEXT:  5      13    0.50    *      *            sbbb	$0, (%rax)
@@ -1695,8 +1695,8 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  5      13    0.50    *      *            sbbb	$7, (%rax)
 # CHECK-NEXT:  5      13    0.50    *      *            lock		sbbb	$7, (%rax)
 # CHECK-NEXT:  1      1     0.50                        sbbb	%sil, %dil
-# CHECK-NEXT:  6      7     0.60    *      *            sbbb	%sil, (%rax)
-# CHECK-NEXT:  6      7     0.60    *      *            lock		sbbb	%sil, (%rax)
+# CHECK-NEXT:  6      13    0.60    *      *            sbbb	%sil, (%rax)
+# CHECK-NEXT:  6      13    0.60    *      *            lock		sbbb	%sil, (%rax)
 # CHECK-NEXT:  2      6     0.50    *                   sbbb	(%rax), %dil
 # CHECK-NEXT:  1      1     0.50                        sbbw	$0, %ax
 # CHECK-NEXT:  1      1     0.50                        sbbw	$0, %di
@@ -1710,8 +1710,8 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  5      12    0.50    *      *            sbbw	$7, (%rax)
 # CHECK-NEXT:  5      12    0.50    *      *            lock		sbbw	$7, (%rax)
 # CHECK-NEXT:  1      1     0.50                        sbbw	%si, %di
-# CHECK-NEXT:  6      7     0.60    *      *            sbbw	%si, (%rax)
-# CHECK-NEXT:  6      7     0.60    *      *            lock		sbbw	%si, (%rax)
+# CHECK-NEXT:  6      12    0.60    *      *            sbbw	%si, (%rax)
+# CHECK-NEXT:  6      12    0.60    *      *            lock		sbbw	%si, (%rax)
 # CHECK-NEXT:  2      6     0.50    *                   sbbw	(%rax), %di
 # CHECK-NEXT:  1      1     0.50                        sbbl	$0, %eax
 # CHECK-NEXT:  1      1     0.50                        sbbl	$0, %edi
@@ -1725,8 +1725,8 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  5      12    0.50    *      *            sbbl	$7, (%rax)
 # CHECK-NEXT:  5      12    0.50    *      *            lock		sbbl	$7, (%rax)
 # CHECK-NEXT:  1      1     0.50                        sbbl	%esi, %edi
-# CHECK-NEXT:  6      7     0.60    *      *            sbbl	%esi, (%rax)
-# CHECK-NEXT:  6      7     0.60    *      *            lock		sbbl	%esi, (%rax)
+# CHECK-NEXT:  6      12    0.60    *      *            sbbl	%esi, (%rax)
+# CHECK-NEXT:  6      12    0.60    *      *            lock		sbbl	%esi, (%rax)
 # CHECK-NEXT:  2      6     0.50    *                   sbbl	(%rax), %edi
 # CHECK-NEXT:  1      1     0.50                        sbbq	$0, %rax
 # CHECK-NEXT:  1      1     0.50                        sbbq	$0, %rdi
@@ -1740,46 +1740,46 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  5      12    0.50    *      *            sbbq	$7, (%rax)
 # CHECK-NEXT:  5      12    0.50    *      *            lock		sbbq	$7, (%rax)
 # CHECK-NEXT:  1      1     0.50                        sbbq	%rsi, %rdi
-# CHECK-NEXT:  6      7     0.60    *      *            sbbq	%rsi, (%rax)
-# CHECK-NEXT:  6      7     0.60    *      *            lock		sbbq	%rsi, (%rax)
+# CHECK-NEXT:  6      12    0.60    *      *            sbbq	%rsi, (%rax)
+# CHECK-NEXT:  6      12    0.60    *      *            lock		sbbq	%rsi, (%rax)
 # CHECK-NEXT:  2      6     0.50    *                   sbbq	(%rax), %rdi
 # CHECK-NEXT:  4      6     0.60                  U     scasb	%es:(%rdi), %al
 # CHECK-NEXT:  4      6     0.60                  U     scasw	%es:(%rdi), %ax
 # CHECK-NEXT:  4      6     0.60                  U     scasl	%es:(%rdi), %eax
 # CHECK-NEXT:  4      6     0.60                  U     scasq	%es:(%rdi), %rax
-# CHECK-NEXT:  1      1     0.50                        seto	%al
-# CHECK-NEXT:  3      2     0.50           *            seto	(%rax)
-# CHECK-NEXT:  1      1     0.50                        setno	%al
-# CHECK-NEXT:  3      2     0.50           *            setno	(%rax)
-# CHECK-NEXT:  1      1     0.50                        setb	%al
-# CHECK-NEXT:  3      2     0.50           *            setb	(%rax)
-# CHECK-NEXT:  1      1     0.50                        setae	%al
-# CHECK-NEXT:  3      2     0.50           *            setae	(%rax)
-# CHECK-NEXT:  1      1     0.50                        sete	%al
-# CHECK-NEXT:  3      2     0.50           *            sete	(%rax)
-# CHECK-NEXT:  1      1     0.50                        setne	%al
-# CHECK-NEXT:  3      2     0.50           *            setne	(%rax)
-# CHECK-NEXT:  1      1     0.50                        seta	%al
-# CHECK-NEXT:  3      2     0.50           *            seta	(%rax)
-# CHECK-NEXT:  1      1     0.50                        setbe	%al
-# CHECK-NEXT:  3      2     0.50           *            setbe	(%rax)
-# CHECK-NEXT:  1      1     0.50                        sets	%al
-# CHECK-NEXT:  3      2     0.50           *            sets	(%rax)
-# CHECK-NEXT:  1      1     0.50                        setns	%al
-# CHECK-NEXT:  3      2     0.50           *            setns	(%rax)
-# CHECK-NEXT:  1      1     0.50                        setp	%al
-# CHECK-NEXT:  3      2     0.50           *            setp	(%rax)
-# CHECK-NEXT:  1      1     0.50                        setnp	%al
-# CHECK-NEXT:  3      2     0.50           *            setnp	(%rax)
-# CHECK-NEXT:  1      1     0.50                        setl	%al
-# CHECK-NEXT:  3      2     0.50           *            setl	(%rax)
-# CHECK-NEXT:  1      1     0.50                        setge	%al
-# CHECK-NEXT:  3      2     0.50           *            setge	(%rax)
-# CHECK-NEXT:  1      1     0.50                        setg	%al
-# CHECK-NEXT:  3      2     0.50           *            setg	(%rax)
-# CHECK-NEXT:  1      1     0.50                        setle	%al
-# CHECK-NEXT:  3      2     0.50           *            setle	(%rax)
-# CHECK-NEXT:  3      4     1.00                        shldw	%cl, %si, %di
+# CHECK-NEXT:  2      2     1.00                        seto	%al
+# CHECK-NEXT:  4      13    1.00           *            seto	(%rax)
+# CHECK-NEXT:  2      2     1.00                        setno	%al
+# CHECK-NEXT:  4      13    1.00           *            setno	(%rax)
+# CHECK-NEXT:  2      2     1.00                        setb	%al
+# CHECK-NEXT:  4      13    1.00           *            setb	(%rax)
+# CHECK-NEXT:  2      2     1.00                        setae	%al
+# CHECK-NEXT:  4      13    1.00           *            setae	(%rax)
+# CHECK-NEXT:  2      2     1.00                        sete	%al
+# CHECK-NEXT:  4      13    1.00           *            sete	(%rax)
+# CHECK-NEXT:  2      2     1.00                        setne	%al
+# CHECK-NEXT:  4      13    1.00           *            setne	(%rax)
+# CHECK-NEXT:  2      2     1.00                        seta	%al
+# CHECK-NEXT:  4      13    1.00           *            seta	(%rax)
+# CHECK-NEXT:  2      2     1.00                        setbe	%al
+# CHECK-NEXT:  4      13    1.00           *            setbe	(%rax)
+# CHECK-NEXT:  2      2     1.00                        sets	%al
+# CHECK-NEXT:  4      13    1.00           *            sets	(%rax)
+# CHECK-NEXT:  2      2     1.00                        setns	%al
+# CHECK-NEXT:  4      13    1.00           *            setns	(%rax)
+# CHECK-NEXT:  2      2     1.00                        setp	%al
+# CHECK-NEXT:  4      13    1.00           *            setp	(%rax)
+# CHECK-NEXT:  2      2     1.00                        setnp	%al
+# CHECK-NEXT:  4      13    1.00           *            setnp	(%rax)
+# CHECK-NEXT:  2      2     1.00                        setl	%al
+# CHECK-NEXT:  4      13    1.00           *            setl	(%rax)
+# CHECK-NEXT:  2      2     1.00                        setge	%al
+# CHECK-NEXT:  4      13    1.00           *            setge	(%rax)
+# CHECK-NEXT:  2      2     1.00                        setg	%al
+# CHECK-NEXT:  4      13    1.00           *            setg	(%rax)
+# CHECK-NEXT:  2      2     1.00                        setle	%al
+# CHECK-NEXT:  4      13    1.00           *            setle	(%rax)
+# CHECK-NEXT:  3      5     1.00                        shldw	%cl, %si, %di
 # CHECK-NEXT:  3      5     1.00                        shrdw	%cl, %si, %di
 # CHECK-NEXT:  6      12    1.00    *      *            shldw	%cl, %si, (%rax)
 # CHECK-NEXT:  6      12    1.00    *      *            shrdw	%cl, %si, (%rax)
@@ -1787,16 +1787,16 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  1      3     1.00                        shrdw	$7, %si, %di
 # CHECK-NEXT:  5      12    1.00    *      *            shldw	$7, %si, (%rax)
 # CHECK-NEXT:  5      13    1.00    *      *            shrdw	$7, %si, (%rax)
-# CHECK-NEXT:  3      4     1.00                        shldl	%cl, %esi, %edi
-# CHECK-NEXT:  3      4     1.00                        shrdl	%cl, %esi, %edi
+# CHECK-NEXT:  3      5     1.00                        shldl	%cl, %esi, %edi
+# CHECK-NEXT:  3      5     1.00                        shrdl	%cl, %esi, %edi
 # CHECK-NEXT:  6      12    1.00    *      *            shldl	%cl, %esi, (%rax)
 # CHECK-NEXT:  6      12    1.00    *      *            shrdl	%cl, %esi, (%rax)
 # CHECK-NEXT:  1      3     1.00                        shldl	$7, %esi, %edi
 # CHECK-NEXT:  1      3     1.00                        shrdl	$7, %esi, %edi
 # CHECK-NEXT:  5      12    1.00    *      *            shldl	$7, %esi, (%rax)
 # CHECK-NEXT:  5      12    1.00    *      *            shrdl	$7, %esi, (%rax)
-# CHECK-NEXT:  3      4     1.00                        shldq	%cl, %rsi, %rdi
-# CHECK-NEXT:  3      4     1.00                        shrdq	%cl, %rsi, %rdi
+# CHECK-NEXT:  3      5     1.00                        shldq	%cl, %rsi, %rdi
+# CHECK-NEXT:  3      5     1.00                        shrdq	%cl, %rsi, %rdi
 # CHECK-NEXT:  6      12    1.00    *      *            shldq	%cl, %rsi, (%rax)
 # CHECK-NEXT:  6      12    1.00    *      *            shrdq	%cl, %rsi, (%rax)
 # CHECK-NEXT:  1      3     1.00                        shldq	$7, %rsi, %rdi
@@ -1805,91 +1805,91 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  5      12    1.00    *      *            shrdq	$7, %rsi, (%rax)
 # CHECK-NEXT:  1      1     0.20                  U     stc
 # CHECK-NEXT:  2      6     0.50                  U     std
-# CHECK-NEXT:  1      6     0.33                  U     stosb	%al, %es:(%rdi)
+# CHECK-NEXT:  4      8     0.50                  U     stosb	%al, %es:(%rdi)
 # CHECK-NEXT:  4      7     0.50                  U     stosw	%ax, %es:(%rdi)
 # CHECK-NEXT:  4      7     0.50                  U     stosl	%eax, %es:(%rdi)
 # CHECK-NEXT:  4      7     0.50                  U     stosq	%rax, %es:(%rdi)
 # CHECK-NEXT:  1      1     0.20                        subb	$7, %al
 # CHECK-NEXT:  1      1     0.20                        subb	$7, %dil
-# CHECK-NEXT:  4      7     0.50    *      *            subb	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		subb	$7, (%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            subb	$7, (%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            lock		subb	$7, (%rax)
 # CHECK-NEXT:  1      1     0.20                        subb	%sil, %dil
-# CHECK-NEXT:  4      7     0.50    *      *            subb	%sil, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		subb	%sil, (%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            subb	%sil, (%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            lock		subb	%sil, (%rax)
 # CHECK-NEXT:  2      6     0.33    *                   subb	(%rax), %dil
 # CHECK-NEXT:  1      1     0.20                        subw	$511, %ax
 # CHECK-NEXT:  1      1     0.20                        subw	$511, %di
-# CHECK-NEXT:  4      7     0.50    *      *            subw	$511, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		subw	$511, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            subw	$511, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		subw	$511, (%rax)
 # CHECK-NEXT:  1      1     0.20                        subw	$7, %di
-# CHECK-NEXT:  4      7     0.50    *      *            subw	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		subw	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            subw	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		subw	$7, (%rax)
 # CHECK-NEXT:  1      1     0.20                        subw	%si, %di
-# CHECK-NEXT:  4      7     0.50    *      *            subw	%si, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		subw	%si, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            subw	%si, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		subw	%si, (%rax)
 # CHECK-NEXT:  2      6     0.33    *                   subw	(%rax), %di
 # CHECK-NEXT:  1      1     0.20                        subl	$665536, %eax
 # CHECK-NEXT:  1      1     0.20                        subl	$665536, %edi
-# CHECK-NEXT:  4      7     0.50    *      *            subl	$665536, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		subl	$665536, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            subl	$665536, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		subl	$665536, (%rax)
 # CHECK-NEXT:  1      1     0.20                        subl	$7, %edi
-# CHECK-NEXT:  4      7     0.50    *      *            subl	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		subl	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            subl	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		subl	$7, (%rax)
 # CHECK-NEXT:  1      1     0.20                        subl	%esi, %edi
-# CHECK-NEXT:  4      7     0.50    *      *            subl	%esi, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		subl	%esi, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            subl	%esi, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		subl	%esi, (%rax)
 # CHECK-NEXT:  2      6     0.33    *                   subl	(%rax), %edi
 # CHECK-NEXT:  1      1     0.20                        subq	$665536, %rax
 # CHECK-NEXT:  1      1     0.20                        subq	$665536, %rdi
-# CHECK-NEXT:  4      7     0.50    *      *            subq	$665536, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		subq	$665536, (%rax)
-# CHECK-NEXT:  1      1     0.20                        subq	$7, %rdi
-# CHECK-NEXT:  4      7     0.50    *      *            subq	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		subq	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            subq	$665536, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		subq	$665536, (%rax)
+# CHECK-NEXT:  0      1     0.00                        subq	$7, %rdi
+# CHECK-NEXT:  4      12    0.50    *      *            subq	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		subq	$7, (%rax)
 # CHECK-NEXT:  1      1     0.20                        subq	%rsi, %rdi
-# CHECK-NEXT:  4      7     0.50    *      *            subq	%rsi, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		subq	%rsi, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            subq	%rsi, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		subq	%rsi, (%rax)
 # CHECK-NEXT:  2      6     0.33    *                   subq	(%rax), %rdi
-# CHECK-NEXT:  1      1     0.20                        testb	$7, %al
-# CHECK-NEXT:  1      1     0.20                        testb	$7, %dil
-# CHECK-NEXT:  2      6     0.33    *                   testb	$7, (%rax)
-# CHECK-NEXT:  1      1     0.20                        testb	%sil, %dil
-# CHECK-NEXT:  2      6     0.33    *                   testb	%sil, (%rax)
+# CHECK-NEXT:  1      2     0.20                        testb	$7, %al
+# CHECK-NEXT:  1      2     0.20                        testb	$7, %dil
+# CHECK-NEXT:  2      7     0.33    *                   testb	$7, (%rax)
+# CHECK-NEXT:  1      2     0.20                        testb	%sil, %dil
+# CHECK-NEXT:  2      7     0.33    *                   testb	%sil, (%rax)
 # CHECK-NEXT:  1      1     0.20                        testw	$511, %ax
 # CHECK-NEXT:  1      1     0.20                        testw	$511, %di
-# CHECK-NEXT:  2      6     0.33    *                   testw	$511, (%rax)
+# CHECK-NEXT:  2      7     0.33    *                   testw	$511, (%rax)
 # CHECK-NEXT:  1      1     0.20                        testw	$7, %di
-# CHECK-NEXT:  2      6     0.33    *                   testw	$7, (%rax)
-# CHECK-NEXT:  1      1     0.20                        testw	%si, %di
-# CHECK-NEXT:  2      6     0.33    *                   testw	%si, (%rax)
+# CHECK-NEXT:  2      7     0.33    *                   testw	$7, (%rax)
+# CHECK-NEXT:  1      2     0.20                        testw	%si, %di
+# CHECK-NEXT:  2      7     0.33    *                   testw	%si, (%rax)
 # CHECK-NEXT:  1      2     0.20                        testl	$665536, %eax
-# CHECK-NEXT:  1      1     0.20                        testl	$665536, %edi
-# CHECK-NEXT:  2      6     0.33    *                   testl	$665536, (%rax)
-# CHECK-NEXT:  1      1     0.20                        testl	$7, %edi
-# CHECK-NEXT:  2      6     0.33    *                   testl	$7, (%rax)
-# CHECK-NEXT:  1      1     0.20                        testl	%esi, %edi
-# CHECK-NEXT:  2      6     0.33    *                   testl	%esi, (%rax)
+# CHECK-NEXT:  1      2     0.20                        testl	$665536, %edi
+# CHECK-NEXT:  2      7     0.33    *                   testl	$665536, (%rax)
+# CHECK-NEXT:  1      2     0.20                        testl	$7, %edi
+# CHECK-NEXT:  2      7     0.33    *                   testl	$7, (%rax)
+# CHECK-NEXT:  1      2     0.20                        testl	%esi, %edi
+# CHECK-NEXT:  2      7     0.33    *                   testl	%esi, (%rax)
 # CHECK-NEXT:  1      2     0.20                        testq	$665536, %rax
-# CHECK-NEXT:  1      1     0.20                        testq	$665536, %rdi
-# CHECK-NEXT:  2      6     0.33    *                   testq	$665536, (%rax)
-# CHECK-NEXT:  1      1     0.20                        testq	$7, %rdi
-# CHECK-NEXT:  2      6     0.33    *                   testq	$7, (%rax)
-# CHECK-NEXT:  1      1     0.20                        testq	%rsi, %rdi
-# CHECK-NEXT:  2      6     0.33    *                   testq	%rsi, (%rax)
+# CHECK-NEXT:  1      2     0.20                        testq	$665536, %rdi
+# CHECK-NEXT:  2      7     0.33    *                   testq	$665536, (%rax)
+# CHECK-NEXT:  1      2     0.20                        testq	$7, %rdi
+# CHECK-NEXT:  2      7     0.33    *                   testq	$7, (%rax)
+# CHECK-NEXT:  1      2     0.20                        testq	%rsi, %rdi
+# CHECK-NEXT:  2      7     0.33    *                   testq	%rsi, (%rax)
 # CHECK-NEXT:  1      100   0.25    *             U     ud2
 # CHECK-NEXT:  144    100   35.50                 U     wrmsr
 # CHECK-NEXT:  3      2     0.60                        xaddb	%bl, %cl
-# CHECK-NEXT:  5      7     0.50    *      *            xaddb	%bl, (%rcx)
-# CHECK-NEXT:  5      7     0.50    *      *            lock		xaddb	%bl, (%rcx)
+# CHECK-NEXT:  5      13    0.50    *      *            xaddb	%bl, (%rcx)
+# CHECK-NEXT:  5      13    0.50    *      *            lock		xaddb	%bl, (%rcx)
 # CHECK-NEXT:  3      2     0.60                        xaddw	%bx, %cx
-# CHECK-NEXT:  5      7     0.50    *      *            xaddw	%ax, (%rbx)
-# CHECK-NEXT:  5      7     0.50    *      *            lock		xaddw	%ax, (%rbx)
+# CHECK-NEXT:  5      12    0.50    *      *            xaddw	%ax, (%rbx)
+# CHECK-NEXT:  5      12    0.50    *      *            lock		xaddw	%ax, (%rbx)
 # CHECK-NEXT:  3      2     0.60                        xaddl	%ebx, %ecx
-# CHECK-NEXT:  5      7     0.50    *      *            xaddl	%eax, (%rbx)
-# CHECK-NEXT:  5      7     0.50    *      *            lock		xaddl	%eax, (%rbx)
+# CHECK-NEXT:  5      12    0.50    *      *            xaddl	%eax, (%rbx)
+# CHECK-NEXT:  5      12    0.50    *      *            lock		xaddl	%eax, (%rbx)
 # CHECK-NEXT:  3      2     0.60                        xaddq	%rbx, %rcx
-# CHECK-NEXT:  5      7     0.50    *      *            xaddq	%rax, (%rbx)
-# CHECK-NEXT:  5      7     0.50    *      *            lock		xaddq	%rax, (%rbx)
+# CHECK-NEXT:  5      12    0.50    *      *            xaddq	%rax, (%rbx)
+# CHECK-NEXT:  5      12    0.50    *      *            lock		xaddq	%rax, (%rbx)
 # CHECK-NEXT:  3      2     0.60                        xchgb	%bl, %cl
 # CHECK-NEXT:  8      40    1.00    *      *            xchgb	%bl, (%rbx)
 # CHECK-NEXT:  8      40    1.00    *      *            lock		xchgb	%bl, (%rbx)
@@ -1906,47 +1906,47 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  9      39    1.20    *      *            xchgq	%rax, (%rbx)
 # CHECK-NEXT:  9      39    1.20    *      *            lock		xchgq	%rax, (%rbx)
 # CHECK-NEXT:  3      7     0.40    *                   xlatb
-# CHECK-NEXT:  1      1     0.20                        xorb	$7, %al
-# CHECK-NEXT:  1      1     0.20                        xorb	$7, %dil
-# CHECK-NEXT:  4      7     0.50    *      *            xorb	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		xorb	$7, (%rax)
-# CHECK-NEXT:  1      1     0.20                        xorb	%sil, %dil
-# CHECK-NEXT:  4      7     0.50    *      *            xorb	%sil, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		xorb	%sil, (%rax)
+# CHECK-NEXT:  1      2     0.20                        xorb	$7, %al
+# CHECK-NEXT:  1      2     0.20                        xorb	$7, %dil
+# CHECK-NEXT:  4      13    0.50    *      *            xorb	$7, (%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            lock		xorb	$7, (%rax)
+# CHECK-NEXT:  1      2     0.20                        xorb	%sil, %dil
+# CHECK-NEXT:  4      13    0.50    *      *            xorb	%sil, (%rax)
+# CHECK-NEXT:  4      13    0.50    *      *            lock		xorb	%sil, (%rax)
 # CHECK-NEXT:  2      6     0.33    *                   xorb	(%rax), %dil
 # CHECK-NEXT:  1      1     0.20                        xorw	$511, %ax
 # CHECK-NEXT:  1      1     0.20                        xorw	$511, %di
-# CHECK-NEXT:  4      7     0.50    *      *            xorw	$511, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		xorw	$511, (%rax)
-# CHECK-NEXT:  1      1     0.20                        xorw	$7, %di
-# CHECK-NEXT:  4      7     0.50    *      *            xorw	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		xorw	$7, (%rax)
-# CHECK-NEXT:  1      1     0.20                        xorw	%si, %di
-# CHECK-NEXT:  4      7     0.50    *      *            xorw	%si, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		xorw	%si, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            xorw	$511, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		xorw	$511, (%rax)
+# CHECK-NEXT:  1      2     0.20                        xorw	$7, %di
+# CHECK-NEXT:  4      12    0.50    *      *            xorw	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		xorw	$7, (%rax)
+# CHECK-NEXT:  1      2     0.20                        xorw	%si, %di
+# CHECK-NEXT:  4      12    0.50    *      *            xorw	%si, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		xorw	%si, (%rax)
 # CHECK-NEXT:  2      6     0.33    *                   xorw	(%rax), %di
 # CHECK-NEXT:  1      2     0.20                        xorl	$665536, %eax
-# CHECK-NEXT:  1      1     0.20                        xorl	$665536, %edi
-# CHECK-NEXT:  4      7     0.50    *      *            xorl	$665536, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		xorl	$665536, (%rax)
-# CHECK-NEXT:  1      1     0.20                        xorl	$7, %edi
-# CHECK-NEXT:  4      7     0.50    *      *            xorl	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		xorl	$7, (%rax)
-# CHECK-NEXT:  1      1     0.20                        xorl	%esi, %edi
-# CHECK-NEXT:  4      7     0.50    *      *            xorl	%esi, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		xorl	%esi, (%rax)
+# CHECK-NEXT:  1      2     0.20                        xorl	$665536, %edi
+# CHECK-NEXT:  4      12    0.50    *      *            xorl	$665536, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		xorl	$665536, (%rax)
+# CHECK-NEXT:  1      2     0.20                        xorl	$7, %edi
+# CHECK-NEXT:  4      12    0.50    *      *            xorl	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		xorl	$7, (%rax)
+# CHECK-NEXT:  1      2     0.20                        xorl	%esi, %edi
+# CHECK-NEXT:  4      12    0.50    *      *            xorl	%esi, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		xorl	%esi, (%rax)
 # CHECK-NEXT:  2      6     0.33    *                   xorl	(%rax), %edi
 # CHECK-NEXT:  1      2     0.20                        xorq	$665536, %rax
-# CHECK-NEXT:  1      1     0.20                        xorq	$665536, %rdi
-# CHECK-NEXT:  4      7     0.50    *      *            xorq	$665536, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		xorq	$665536, (%rax)
-# CHECK-NEXT:  1      1     0.20                        xorq	$7, %rdi
-# CHECK-NEXT:  4      7     0.50    *      *            xorq	$7, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		xorq	$7, (%rax)
-# CHECK-NEXT:  1      1     0.20                        xorq	%rsi, %rdi
-# CHECK-NEXT:  4      7     0.50    *      *            xorq	%rsi, (%rax)
-# CHECK-NEXT:  4      7     0.50    *      *            lock		xorq	%rsi, (%rax)
-# CHECK-NEXT:  2      6     0.33    *                   xorq	(%rax), %rdi
+# CHECK-NEXT:  1      2     0.20                        xorq	$665536, %rdi
+# CHECK-NEXT:  4      12    0.50    *      *            xorq	$665536, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		xorq	$665536, (%rax)
+# CHECK-NEXT:  1      2     0.20                        xorq	$7, %rdi
+# CHECK-NEXT:  4      12    0.50    *      *            xorq	$7, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		xorq	$7, (%rax)
+# CHECK-NEXT:  1      2     0.20                        xorq	%rsi, %rdi
+# CHECK-NEXT:  4      12    0.50    *      *            xorq	%rsi, (%rax)
+# CHECK-NEXT:  4      12    0.50    *      *            lock		xorq	%rsi, (%rax)
+# CHECK-NEXT:  2      7     0.33    *                   xorq	(%rax), %rdi
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - ADLPPort00
@@ -1965,7 +1965,7 @@ xorq (%rax), %rdi
 
 # CHECK:      Resource pressure per iteration:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]
-# CHECK-NEXT: 909.92 773.42 210.33 210.33 202.00 578.58 760.42 202.50 202.50 202.00 192.67 210.33  -
+# CHECK-NEXT: 949.92 794.58 213.00 213.00 202.50 599.75 793.42 203.00 203.00 202.50 191.33 213.00  -
 
 # CHECK:      Resource pressure by instruction:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]   Instructions:
@@ -2060,7 +2060,7 @@ xorq (%rax), %rdi
 # CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     addq	$665536, %rdi
 # CHECK-NEXT: 0.20   0.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     addq	$665536, (%rax)
 # CHECK-NEXT: 0.20   0.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     lock		addq	$665536, (%rax)
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     addq	$7, %rdi
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     addq	$7, %rdi
 # CHECK-NEXT: 0.20   0.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     addq	$7, (%rax)
 # CHECK-NEXT: 0.20   0.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     lock		addq	$7, (%rax)
 # CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     addq	%rsi, %rdi
@@ -2188,9 +2188,9 @@ xorq (%rax), %rdi
 # CHECK-NEXT:  -     1.00   0.33   0.33   0.50    -      -     0.50   0.50   0.50    -     0.33    -     lock		btcq	$7, (%rax)
 # CHECK-NEXT:  -     1.00   0.33   0.33   0.50    -      -     0.50   0.50   0.50    -     0.33    -     lock		btrq	$7, (%rax)
 # CHECK-NEXT:  -     1.00   0.33   0.33   0.50    -      -     0.50   0.50   0.50    -     0.33    -     lock		btsq	$7, (%rax)
-# CHECK-NEXT:  -     0.33    -      -      -     0.33    -      -      -      -     0.33    -      -     cbtw
-# CHECK-NEXT:  -     0.33    -      -      -     0.33    -      -      -      -     0.33    -      -     cwtl
-# CHECK-NEXT:  -     0.33    -      -      -     0.33    -      -      -      -     0.33    -      -     cltq
+# CHECK-NEXT:  -     0.50    -      -      -     0.50    -      -      -      -      -      -      -     cbtw
+# CHECK-NEXT:  -     0.50    -      -      -     0.50    -      -      -      -      -      -      -     cwtl
+# CHECK-NEXT:  -     0.50    -      -      -     0.50    -      -      -      -      -      -      -     cltq
 # CHECK-NEXT: 0.70   0.20    -      -      -     0.20   0.70    -      -      -     0.20    -      -     cwtd
 # CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cltd
 # CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     cqto
@@ -2253,26 +2253,26 @@ xorq (%rax), %rdi
 # CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     decl	%edi
 # CHECK-NEXT: 0.20   0.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     decl	(%rax)
 # CHECK-NEXT: 0.20   0.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     lock		decl	(%rax)
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     decq	%rdi
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     decq	%rdi
 # CHECK-NEXT: 0.20   0.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     decq	(%rax)
 # CHECK-NEXT: 0.20   0.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     lock		decq	(%rax)
-# CHECK-NEXT: 0.20   3.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     divb	%dil
-# CHECK-NEXT: 0.20   3.20   0.33   0.33    -     0.20   0.20    -      -      -     0.20   0.33    -     divb	(%rax)
+# CHECK-NEXT:  -     3.00    -      -      -      -      -      -      -      -      -      -      -     divb	%dil
+# CHECK-NEXT:  -     3.00    -      -      -      -      -      -      -      -      -      -      -     divb	(%rax)
 # CHECK-NEXT: 0.20   3.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     divw	%si
 # CHECK-NEXT: 0.20   3.20   0.33   0.33    -     0.20   0.20    -      -      -     0.20   0.33    -     divw	(%rax)
 # CHECK-NEXT: 0.20   3.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     divl	%edx
 # CHECK-NEXT: 0.20   3.20   0.33   0.33    -     0.20   0.20    -      -      -     0.20   0.33    -     divl	(%rax)
-# CHECK-NEXT: 0.20   3.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     divq	%rcx
-# CHECK-NEXT: 0.20   3.20   0.33   0.33    -     0.20   0.20    -      -      -     0.20   0.33    -     divq	(%rax)
+# CHECK-NEXT:  -     3.00    -      -      -      -      -      -      -      -      -      -      -     divq	%rcx
+# CHECK-NEXT:  -     3.00   0.33   0.33    -      -      -      -      -      -      -     0.33    -     divq	(%rax)
 # CHECK-NEXT: 12.50  2.00   4.67   4.67   2.00   9.00   10.50  2.50   2.50   2.00    -     4.67    -     enter	$7, $4095
-# CHECK-NEXT: 0.20   3.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     idivb	%dil
-# CHECK-NEXT: 0.20   3.20   0.33   0.33    -     0.20   0.20    -      -      -     0.20   0.33    -     idivb	(%rax)
+# CHECK-NEXT:  -     3.00    -      -      -      -      -      -      -      -      -      -      -     idivb	%dil
+# CHECK-NEXT:  -     3.00    -      -      -      -      -      -      -      -      -      -      -     idivb	(%rax)
 # CHECK-NEXT: 0.20   3.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     idivw	%si
 # CHECK-NEXT: 0.20   3.20   0.33   0.33    -     0.20   0.20    -      -      -     0.20   0.33    -     idivw	(%rax)
 # CHECK-NEXT: 0.20   3.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     idivl	%edx
 # CHECK-NEXT: 0.20   3.20   0.33   0.33    -     0.20   0.20    -      -      -     0.20   0.33    -     idivl	(%rax)
-# CHECK-NEXT: 0.20   3.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     idivq	%rcx
-# CHECK-NEXT: 0.20   3.20   0.33   0.33    -     0.20   0.20    -      -      -     0.20   0.33    -     idivq	(%rax)
+# CHECK-NEXT:  -     3.00    -      -      -      -      -      -      -      -      -      -      -     idivq	%rcx
+# CHECK-NEXT:  -     3.00   0.33   0.33    -      -      -      -      -      -      -     0.33    -     idivq	(%rax)
 # CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     imulb	%dil
 # CHECK-NEXT:  -     1.00   0.33   0.33    -      -      -      -      -      -      -     0.33    -     imulb	(%rax)
 # CHECK-NEXT: 0.90   1.40    -      -      -     0.40   0.90    -      -      -     0.40    -      -     imulw	%di
@@ -2304,7 +2304,7 @@ xorq (%rax), %rdi
 # CHECK-NEXT: 21.00  20.67  2.33   2.33    -     22.67  14.00   -      -      -     1.67   2.33    -     inw	$7, %ax
 # CHECK-NEXT: 21.30  21.30  2.33   2.33    -     21.80  13.80   -      -      -     1.80   2.33    -     inw	%dx, %ax
 # CHECK-NEXT: 22.20  22.87  3.33   3.33    -     21.87  15.20   -      -      -     1.87   3.33    -     inl	$7, %eax
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     inl	%dx, %eax
+# CHECK-NEXT: 22.80  23.47  3.67   3.67    -     23.47  15.80   -      -      -     2.47   3.67    -     inl	%dx, %eax
 # CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     incb	%dil
 # CHECK-NEXT: 0.20   0.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     incb	(%rax)
 # CHECK-NEXT: 0.20   0.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     lock		incb	(%rax)
@@ -2314,7 +2314,7 @@ xorq (%rax), %rdi
 # CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     incl	%edi
 # CHECK-NEXT: 0.20   0.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     incl	(%rax)
 # CHECK-NEXT: 0.20   0.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     lock		incl	(%rax)
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     incq	%rdi
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     incq	%rdi
 # CHECK-NEXT: 0.20   0.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     incq	(%rax)
 # CHECK-NEXT: 0.20   0.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     lock		incq	(%rax)
 # CHECK-NEXT: 20.20  18.20  2.67   2.67   0.50   20.20  13.20  0.50   0.50   0.50   1.20   2.67    -     insb	%dx, %es:(%rdi)
@@ -2504,8 +2504,8 @@ xorq (%rax), %rdi
 # CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     rorb	%dil
 # CHECK-NEXT: 1.00    -     0.33   0.33   0.50    -     1.00   0.50   0.50   0.50    -     0.33    -     rolb	(%rax)
 # CHECK-NEXT: 1.00    -     0.33   0.33   0.50    -     1.00   0.50   0.50   0.50    -     0.33    -     rorb	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     rolb	$7, %dil
-# CHECK-NEXT: 0.90   0.40    -      -      -     0.40   0.90    -      -      -     0.40    -      -     rorb	$7, %dil
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     rolb	$7, %dil
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     rorb	$7, %dil
 # CHECK-NEXT: 1.00    -     0.33   0.33   0.50    -     1.00   0.50   0.50   0.50    -     0.33    -     rolb	$7, (%rax)
 # CHECK-NEXT: 1.00    -     0.33   0.33   0.50    -     1.00   0.50   0.50   0.50    -     0.33    -     rorb	$7, (%rax)
 # CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     rolb	%cl, %dil
@@ -2516,8 +2516,8 @@ xorq (%rax), %rdi
 # CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     rorw	%di
 # CHECK-NEXT: 1.00    -     0.33   0.33   0.50    -     1.00   0.50   0.50   0.50    -     0.33    -     rolw	(%rax)
 # CHECK-NEXT: 1.00    -     0.33   0.33   0.50    -     1.00   0.50   0.50   0.50    -     0.33    -     rorw	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     rolw	$7, %di
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     rorw	$7, %di
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     rolw	$7, %di
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     rorw	$7, %di
 # CHECK-NEXT: 1.00    -     0.33   0.33   0.50    -     1.00   0.50   0.50   0.50    -     0.33    -     rolw	$7, (%rax)
 # CHECK-NEXT: 1.00    -     0.33   0.33   0.50    -     1.00   0.50   0.50   0.50    -     0.33    -     rorw	$7, (%rax)
 # CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     rolw	%cl, %di
@@ -2528,8 +2528,8 @@ xorq (%rax), %rdi
 # CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     rorl	%edi
 # CHECK-NEXT: 1.00    -     0.33   0.33   0.50    -     1.00   0.50   0.50   0.50    -     0.33    -     roll	(%rax)
 # CHECK-NEXT: 1.00    -     0.33   0.33   0.50    -     1.00   0.50   0.50   0.50    -     0.33    -     rorl	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     roll	$7, %edi
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     rorl	$7, %edi
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     roll	$7, %edi
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     rorl	$7, %edi
 # CHECK-NEXT: 1.00    -     0.33   0.33   0.50    -     1.00   0.50   0.50   0.50    -     0.33    -     roll	$7, (%rax)
 # CHECK-NEXT: 1.00    -     0.33   0.33   0.50    -     1.00   0.50   0.50   0.50    -     0.33    -     rorl	$7, (%rax)
 # CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     roll	%cl, %edi
@@ -2540,8 +2540,8 @@ xorq (%rax), %rdi
 # CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     rorq	%rdi
 # CHECK-NEXT: 1.00    -     0.33   0.33   0.50    -     1.00   0.50   0.50   0.50    -     0.33    -     rolq	(%rax)
 # CHECK-NEXT: 1.00    -     0.33   0.33   0.50    -     1.00   0.50   0.50   0.50    -     0.33    -     rorq	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     rolq	$7, %rdi
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     rorq	$7, %rdi
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     rolq	$7, %rdi
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     rorq	$7, %rdi
 # CHECK-NEXT: 1.00    -     0.33   0.33   0.50    -     1.00   0.50   0.50   0.50    -     0.33    -     rolq	$7, (%rax)
 # CHECK-NEXT: 1.00    -     0.33   0.33   0.50    -     1.00   0.50   0.50   0.50    -     0.33    -     rorq	$7, (%rax)
 # CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     rolq	%cl, %rdi
@@ -2682,38 +2682,38 @@ xorq (%rax), %rdi
 # CHECK-NEXT: 0.60   0.60   0.33   0.33    -     0.60   0.60    -      -      -     0.60   0.33    -     scasw	%es:(%rdi), %ax
 # CHECK-NEXT: 0.60   0.60   0.33   0.33    -     0.60   0.60    -      -      -     0.60   0.33    -     scasl	%es:(%rdi), %eax
 # CHECK-NEXT: 0.60   0.60   0.33   0.33    -     0.60   0.60    -      -      -     0.60   0.33    -     scasq	%es:(%rdi), %rax
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     seto	%al
-# CHECK-NEXT: 0.50    -      -      -     0.50    -     0.50   0.50   0.50   0.50    -      -      -     seto	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     setno	%al
-# CHECK-NEXT: 0.50    -      -      -     0.50    -     0.50   0.50   0.50   0.50    -      -      -     setno	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     setb	%al
-# CHECK-NEXT: 0.50    -      -      -     0.50    -     0.50   0.50   0.50   0.50    -      -      -     setb	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     setae	%al
-# CHECK-NEXT: 0.50    -      -      -     0.50    -     0.50   0.50   0.50   0.50    -      -      -     setae	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     sete	%al
-# CHECK-NEXT: 0.50    -      -      -     0.50    -     0.50   0.50   0.50   0.50    -      -      -     sete	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     setne	%al
-# CHECK-NEXT: 0.50    -      -      -     0.50    -     0.50   0.50   0.50   0.50    -      -      -     setne	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     seta	%al
-# CHECK-NEXT: 0.50    -      -      -     0.50    -     0.50   0.50   0.50   0.50    -      -      -     seta	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     setbe	%al
-# CHECK-NEXT: 0.50    -      -      -     0.50    -     0.50   0.50   0.50   0.50    -      -      -     setbe	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     sets	%al
-# CHECK-NEXT: 0.50    -      -      -     0.50    -     0.50   0.50   0.50   0.50    -      -      -     sets	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     setns	%al
-# CHECK-NEXT: 0.50    -      -      -     0.50    -     0.50   0.50   0.50   0.50    -      -      -     setns	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     setp	%al
-# CHECK-NEXT: 0.50    -      -      -     0.50    -     0.50   0.50   0.50   0.50    -      -      -     setp	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     setnp	%al
-# CHECK-NEXT: 0.50    -      -      -     0.50    -     0.50   0.50   0.50   0.50    -      -      -     setnp	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     setl	%al
-# CHECK-NEXT: 0.50    -      -      -     0.50    -     0.50   0.50   0.50   0.50    -      -      -     setl	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     setge	%al
-# CHECK-NEXT: 0.50    -      -      -     0.50    -     0.50   0.50   0.50   0.50    -      -      -     setge	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     setg	%al
-# CHECK-NEXT: 0.50    -      -      -     0.50    -     0.50   0.50   0.50   0.50    -      -      -     setg	(%rax)
-# CHECK-NEXT: 0.50    -      -      -      -      -     0.50    -      -      -      -      -      -     setle	%al
-# CHECK-NEXT: 0.50    -      -      -     0.50    -     0.50   0.50   0.50   0.50    -      -      -     setle	(%rax)
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     seto	%al
+# CHECK-NEXT: 1.00    -      -      -     0.50    -     1.00   0.50   0.50   0.50    -      -      -     seto	(%rax)
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     setno	%al
+# CHECK-NEXT: 1.00    -      -      -     0.50    -     1.00   0.50   0.50   0.50    -      -      -     setno	(%rax)
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     setb	%al
+# CHECK-NEXT: 1.00    -      -      -     0.50    -     1.00   0.50   0.50   0.50    -      -      -     setb	(%rax)
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     setae	%al
+# CHECK-NEXT: 1.00    -      -      -     0.50    -     1.00   0.50   0.50   0.50    -      -      -     setae	(%rax)
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     sete	%al
+# CHECK-NEXT: 1.00    -      -      -     0.50    -     1.00   0.50   0.50   0.50    -      -      -     sete	(%rax)
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     setne	%al
+# CHECK-NEXT: 1.00    -      -      -     0.50    -     1.00   0.50   0.50   0.50    -      -      -     setne	(%rax)
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     seta	%al
+# CHECK-NEXT: 1.00    -      -      -     0.50    -     1.00   0.50   0.50   0.50    -      -      -     seta	(%rax)
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     setbe	%al
+# CHECK-NEXT: 1.00    -      -      -     0.50    -     1.00   0.50   0.50   0.50    -      -      -     setbe	(%rax)
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     sets	%al
+# CHECK-NEXT: 1.00    -      -      -     0.50    -     1.00   0.50   0.50   0.50    -      -      -     sets	(%rax)
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     setns	%al
+# CHECK-NEXT: 1.00    -      -      -     0.50    -     1.00   0.50   0.50   0.50    -      -      -     setns	(%rax)
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     setp	%al
+# CHECK-NEXT: 1.00    -      -      -     0.50    -     1.00   0.50   0.50   0.50    -      -      -     setp	(%rax)
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     setnp	%al
+# CHECK-NEXT: 1.00    -      -      -     0.50    -     1.00   0.50   0.50   0.50    -      -      -     setnp	(%rax)
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     setl	%al
+# CHECK-NEXT: 1.00    -      -      -     0.50    -     1.00   0.50   0.50   0.50    -      -      -     setl	(%rax)
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     setge	%al
+# CHECK-NEXT: 1.00    -      -      -     0.50    -     1.00   0.50   0.50   0.50    -      -      -     setge	(%rax)
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     setg	%al
+# CHECK-NEXT: 1.00    -      -      -     0.50    -     1.00   0.50   0.50   0.50    -      -      -     setg	(%rax)
+# CHECK-NEXT: 1.00    -      -      -      -      -     1.00    -      -      -      -      -      -     setle	%al
+# CHECK-NEXT: 1.00    -      -      -     0.50    -     1.00   0.50   0.50   0.50    -      -      -     setle	(%rax)
 # CHECK-NEXT: 0.70   1.20    -      -      -     0.20   0.70    -      -      -     0.20    -      -     shldw	%cl, %si, %di
 # CHECK-NEXT: 0.70   1.20    -      -      -     0.20   0.70    -      -      -     0.20    -      -     shrdw	%cl, %si, %di
 # CHECK-NEXT: 0.70   1.20   0.33   0.33   0.50   0.20   0.70   0.50   0.50   0.50   0.20   0.33    -     shldw	%cl, %si, (%rax)
@@ -2740,7 +2740,7 @@ xorq (%rax), %rdi
 # CHECK-NEXT: 0.20   1.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     shrdq	$7, %rsi, (%rax)
 # CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     stc
 # CHECK-NEXT: 0.70   0.20    -      -      -     0.20   0.70    -      -      -     0.20    -      -     std
-# CHECK-NEXT: 0.20   0.20   0.33   0.33    -     0.20   0.20    -      -      -     0.20   0.33    -     stosb	%al, %es:(%rdi)
+# CHECK-NEXT: 0.40   0.40    -      -     0.50   0.40   0.40   0.50   0.50   0.50   0.40    -      -     stosb	%al, %es:(%rdi)
 # CHECK-NEXT: 0.40   0.40    -      -     0.50   0.40   0.40   0.50   0.50   0.50   0.40    -      -     stosw	%ax, %es:(%rdi)
 # CHECK-NEXT: 0.40   0.40    -      -     0.50   0.40   0.40   0.50   0.50   0.50   0.40    -      -     stosl	%eax, %es:(%rdi)
 # CHECK-NEXT: 0.40   0.40    -      -     0.50   0.40   0.40   0.50   0.50   0.50   0.40    -      -     stosq	%rax, %es:(%rdi)
@@ -2778,7 +2778,7 @@ xorq (%rax), %rdi
 # CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     subq	$665536, %rdi
 # CHECK-NEXT: 0.20   0.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     subq	$665536, (%rax)
 # CHECK-NEXT: 0.20   0.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     lock		subq	$665536, (%rax)
-# CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     subq	$7, %rdi
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -     subq	$7, %rdi
 # CHECK-NEXT: 0.20   0.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     subq	$7, (%rax)
 # CHECK-NEXT: 0.20   0.20   0.33   0.33   0.50   0.20   0.20   0.50   0.50   0.50   0.20   0.33    -     lock		subq	$7, (%rax)
 # CHECK-NEXT: 0.20   0.20    -      -      -     0.20   0.20    -      -      -     0.20    -      -     subq	%rsi, %rdi
diff --git a/llvm/test/tools/llvm-mca/X86/AlderlakeP/zero-idioms.s b/llvm/test/tools/llvm-mca/X86/AlderlakeP/zero-idioms.s
index bb0c4bfb4ac2d..90fea632be66b 100644
--- a/llvm/test/tools/llvm-mca/X86/AlderlakeP/zero-idioms.s
+++ b/llvm/test/tools/llvm-mca/X86/AlderlakeP/zero-idioms.s
@@ -124,8 +124,8 @@ vpxor  %ymm3, %ymm3, %ymm5
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
 # CHECK-NEXT:  1      1     0.20                        subl	%eax, %eax
 # CHECK-NEXT:  1      1     0.20                        subq	%rax, %rax
-# CHECK-NEXT:  1      1     0.20                        xorl	%eax, %eax
-# CHECK-NEXT:  1      1     0.20                        xorq	%rax, %rax
+# CHECK-NEXT:  1      2     0.20                        xorl	%eax, %eax
+# CHECK-NEXT:  1      2     0.20                        xorq	%rax, %rax
 # CHECK-NEXT:  1      1     1.00                        pcmpgtb	%mm2, %mm2
 # CHECK-NEXT:  1      1     1.00                        pcmpgtd	%mm2, %mm2
 # CHECK-NEXT:  1      1     1.00                        pcmpgtw	%mm2, %mm2
@@ -321,13 +321,13 @@ vpxor  %ymm3, %ymm3, %ymm5
 
 # CHECK:      [0,0]     DeER .    .    .    .    .    .  .   subl	%eax, %eax
 # CHECK-NEXT: [0,1]     D=eER.    .    .    .    .    .  .   subq	%rax, %rax
-# CHECK-NEXT: [0,2]     D==eER    .    .    .    .    .  .   xorl	%eax, %eax
-# CHECK-NEXT: [0,3]     D===eER   .    .    .    .    .  .   xorq	%rax, %rax
-# CHECK-NEXT: [0,4]     DeE---R   .    .    .    .    .  .   pcmpgtb	%mm2, %mm2
-# CHECK-NEXT: [0,5]     D=eE--R   .    .    .    .    .  .   pcmpgtd	%mm2, %mm2
-# CHECK-NEXT: [0,6]     .D=eE-R   .    .    .    .    .  .   pcmpgtw	%mm2, %mm2
-# CHECK-NEXT: [0,7]     .DeE--R   .    .    .    .    .  .   pcmpgtb	%xmm2, %xmm2
-# CHECK-NEXT: [0,8]     .D=eE-R   .    .    .    .    .  .   pcmpgtd	%xmm2, %xmm2
+# CHECK-NEXT: [0,2]     D==eeER   .    .    .    .    .  .   xorl	%eax, %eax
+# CHECK-NEXT: [0,3]     D====eeER .    .    .    .    .  .   xorq	%rax, %rax
+# CHECK-NEXT: [0,4]     DeE-----R .    .    .    .    .  .   pcmpgtb	%mm2, %mm2
+# CHECK-NEXT: [0,5]     D=eE----R .    .    .    .    .  .   pcmpgtd	%mm2, %mm2
+# CHECK-NEXT: [0,6]     .D=eE---R .    .    .    .    .  .   pcmpgtw	%mm2, %mm2
+# CHECK-NEXT: [0,7]     .DeE----R .    .    .    .    .  .   pcmpgtb	%xmm2, %xmm2
+# CHECK-NEXT: [0,8]     .D=eE---R .    .    .    .    .  .   pcmpgtd	%xmm2, %xmm2
 # CHECK-NEXT: [0,9]     .D==eeeER .    .    .    .    .  .   pcmpgtq	%xmm2, %xmm2
 # CHECK-NEXT: [0,10]    .D=====eER.    .    .    .    .  .   pcmpgtw	%xmm2, %xmm2
 # CHECK-NEXT: [0,11]    .D==eE---R.    .    .    .    .  .   vpcmpgtb	%xmm3, %xmm3, %xmm3
@@ -413,12 +413,12 @@ vpxor  %ymm3, %ymm3, %ymm5
 # CHECK-NEXT: 0.     1     1.0    1.0    0.0       subl	%eax, %eax
 # CHECK-NEXT: 1.     1     2.0    0.0    0.0       subq	%rax, %rax
 # CHECK-NEXT: 2.     1     3.0    0.0    0.0       xorl	%eax, %eax
-# CHECK-NEXT: 3.     1     4.0    0.0    0.0       xorq	%rax, %rax
-# CHECK-NEXT: 4.     1     1.0    1.0    3.0       pcmpgtb	%mm2, %mm2
-# CHECK-NEXT: 5.     1     2.0    0.0    2.0       pcmpgtd	%mm2, %mm2
-# CHECK-NEXT: 6.     1     2.0    0.0    1.0       pcmpgtw	%mm2, %mm2
-# CHECK-NEXT: 7.     1     1.0    1.0    2.0       pcmpgtb	%xmm2, %xmm2
-# CHECK-NEXT: 8.     1     2.0    0.0    1.0       pcmpgtd	%xmm2, %xmm2
+# CHECK-NEXT: 3.     1     5.0    0.0    0.0       xorq	%rax, %rax
+# CHECK-NEXT: 4.     1     1.0    1.0    5.0       pcmpgtb	%mm2, %mm2
+# CHECK-NEXT: 5.     1     2.0    0.0    4.0       pcmpgtd	%mm2, %mm2
+# CHECK-NEXT: 6.     1     2.0    0.0    3.0       pcmpgtw	%mm2, %mm2
+# CHECK-NEXT: 7.     1     1.0    1.0    4.0       pcmpgtb	%xmm2, %xmm2
+# CHECK-NEXT: 8.     1     2.0    0.0    3.0       pcmpgtd	%xmm2, %xmm2
 # CHECK-NEXT: 9.     1     3.0    0.0    0.0       pcmpgtq	%xmm2, %xmm2
 # CHECK-NEXT: 10.    1     6.0    0.0    0.0       pcmpgtw	%xmm2, %xmm2
 # CHECK-NEXT: 11.    1     3.0    3.0    3.0       vpcmpgtb	%xmm3, %xmm3, %xmm3
@@ -493,4 +493,4 @@ vpxor  %ymm3, %ymm3, %ymm5
 # CHECK-NEXT: 80.    1     9.0    0.0    9.0       vxorpd	%ymm1, %ymm1, %ymm3
 # CHECK-NEXT: 81.    1     11.0   1.0    7.0       vpxor	%xmm3, %xmm3, %xmm5
 # CHECK-NEXT: 82.    1     12.0   2.0    6.0       vpxor	%ymm3, %ymm3, %ymm5
-# CHECK-NEXT:        1     9.4    1.0    4.9       <total>
+# CHECK-NEXT:        1     9.4    1.0    5.0       <total>
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/independent-load-stores.s b/llvm/test/tools/llvm-mca/X86/Znver4/independent-load-stores.s
new file mode 100644
index 0000000000000..1b487e6b1b9c7
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/independent-load-stores.s
@@ -0,0 +1,155 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -timeline -timeline-max-iterations=1 < %s | FileCheck %s -check-prefixes=ALL,NOALIAS
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -timeline -timeline-max-iterations=1 -noalias=false < %s | FileCheck %s -check-prefixes=ALL,YESALIAS
+
+  addq	$44, 64(%r14)
+  addq	$44, 128(%r14)
+  addq	$44, 192(%r14)
+  addq	$44, 256(%r14)
+  addq	$44, 320(%r14)
+  addq	$44, 384(%r14)
+  addq	$44, 448(%r14)
+  addq	$44, 512(%r14)
+  addq	$44, 576(%r14)
+  addq	$44, 640(%r14)
+
+# ALL:           Iterations:        100
+# ALL-NEXT:      Instructions:      1000
+
+# NOALIAS-NEXT:  Total Cycles:      675
+# YESALIAS-NEXT: Total Cycles:      6003
+
+# ALL-NEXT:      Total uOps:        1000
+
+# ALL:           Dispatch Width:    6
+
+# NOALIAS-NEXT:  uOps Per Cycle:    1.48
+# NOALIAS-NEXT:  IPC:               1.48
+
+# YESALIAS-NEXT: uOps Per Cycle:    0.17
+# YESALIAS-NEXT: IPC:               0.17
+
+# ALL-NEXT:      Block RThroughput: 6.7
+
+# ALL:           Instruction Info:
+# ALL-NEXT:      [1]: #uOps
+# ALL-NEXT:      [2]: Latency
+# ALL-NEXT:      [3]: RThroughput
+# ALL-NEXT:      [4]: MayLoad
+# ALL-NEXT:      [5]: MayStore
+# ALL-NEXT:      [6]: HasSideEffects (U)
+
+# ALL:           [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# ALL-NEXT:       1      6     0.67    *      *            addq	$44, 64(%r14)
+# ALL-NEXT:       1      6     0.67    *      *            addq	$44, 128(%r14)
+# ALL-NEXT:       1      6     0.67    *      *            addq	$44, 192(%r14)
+# ALL-NEXT:       1      6     0.67    *      *            addq	$44, 256(%r14)
+# ALL-NEXT:       1      6     0.67    *      *            addq	$44, 320(%r14)
+# ALL-NEXT:       1      6     0.67    *      *            addq	$44, 384(%r14)
+# ALL-NEXT:       1      6     0.67    *      *            addq	$44, 448(%r14)
+# ALL-NEXT:       1      6     0.67    *      *            addq	$44, 512(%r14)
+# ALL-NEXT:       1      6     0.67    *      *            addq	$44, 576(%r14)
+# ALL-NEXT:       1      6     0.67    *      *            addq	$44, 640(%r14)
+
+# ALL:           Resources:
+# ALL-NEXT:      [0]   - Zn4AGU0
+# ALL-NEXT:      [1]   - Zn4AGU1
+# ALL-NEXT:      [2]   - Zn4AGU2
+# ALL-NEXT:      [3]   - Zn4ALU0
+# ALL-NEXT:      [4]   - Zn4ALU1
+# ALL-NEXT:      [5]   - Zn4ALU2
+# ALL-NEXT:      [6]   - Zn4ALU3
+# ALL-NEXT:      [7]   - Zn4BRU1
+# ALL-NEXT:      [8]   - Zn4FP0
+# ALL-NEXT:      [9]   - Zn4FP1
+# ALL-NEXT:      [10]  - Zn4FP2
+# ALL-NEXT:      [11]  - Zn4FP3
+# ALL-NEXT:      [12.0] - Zn4FP45
+# ALL-NEXT:      [12.1] - Zn4FP45
+# ALL-NEXT:      [13]  - Zn4FPSt
+# ALL-NEXT:      [14.0] - Zn4LSU
+# ALL-NEXT:      [14.1] - Zn4LSU
+# ALL-NEXT:      [14.2] - Zn4LSU
+# ALL-NEXT:      [15.0] - Zn4Load
+# ALL-NEXT:      [15.1] - Zn4Load
+# ALL-NEXT:      [15.2] - Zn4Load
+# ALL-NEXT:      [16.0] - Zn4Store
+# ALL-NEXT:      [16.1] - Zn4Store
+
+# ALL:           Resource pressure per iteration:
+# ALL-NEXT:      [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# ALL-NEXT:      6.66   6.66   6.68   2.50   2.50   2.50   2.50    -      -      -      -      -      -      -      -     6.66   6.66   6.68   3.33   3.33   3.34   5.00   5.00
+
+# ALL:           Resource pressure by instruction:
+# ALL-NEXT:      [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# ALL-NEXT:      0.66   0.66   0.68    -     0.50    -     0.50    -      -      -      -      -      -      -      -     0.66   0.66   0.68   0.33   0.33   0.34    -     1.00   addq	$44, 64(%r14)
+# ALL-NEXT:      0.66   0.68   0.66   0.50    -     0.50    -      -      -      -      -      -      -      -      -     0.66   0.68   0.66   0.33   0.34   0.33   1.00    -     addq	$44, 128(%r14)
+# ALL-NEXT:      0.68   0.66   0.66    -     0.50    -     0.50    -      -      -      -      -      -      -      -     0.68   0.66   0.66   0.34   0.33   0.33    -     1.00   addq	$44, 192(%r14)
+# ALL-NEXT:      0.66   0.66   0.68   0.50    -     0.50    -      -      -      -      -      -      -      -      -     0.66   0.66   0.68   0.33   0.33   0.34   1.00    -     addq	$44, 256(%r14)
+# ALL-NEXT:      0.66   0.68   0.66    -     0.50    -     0.50    -      -      -      -      -      -      -      -     0.66   0.68   0.66   0.33   0.34   0.33    -     1.00   addq	$44, 320(%r14)
+# ALL-NEXT:      0.68   0.66   0.66   0.50    -     0.50    -      -      -      -      -      -      -      -      -     0.68   0.66   0.66   0.34   0.33   0.33   1.00    -     addq	$44, 384(%r14)
+# ALL-NEXT:      0.66   0.66   0.68    -     0.50    -     0.50    -      -      -      -      -      -      -      -     0.66   0.66   0.68   0.33   0.33   0.34    -     1.00   addq	$44, 448(%r14)
+# ALL-NEXT:      0.66   0.68   0.66   0.50    -     0.50    -      -      -      -      -      -      -      -      -     0.66   0.68   0.66   0.33   0.34   0.33   1.00    -     addq	$44, 512(%r14)
+# ALL-NEXT:      0.68   0.66   0.66    -     0.50    -     0.50    -      -      -      -      -      -      -      -     0.68   0.66   0.66   0.34   0.33   0.33    -     1.00   addq	$44, 576(%r14)
+# ALL-NEXT:      0.66   0.66   0.68   0.50    -     0.50    -      -      -      -      -      -      -      -      -     0.66   0.66   0.68   0.33   0.33   0.34   1.00    -     addq	$44, 640(%r14)
+
+# ALL:           Timeline view:
+
+# NOALIAS-NEXT:                      01234
+# NOALIAS-NEXT:  Index     0123456789
+
+# YESALIAS-NEXT:                     0123456789          0123456789          0123456789
+# YESALIAS-NEXT: Index     0123456789          0123456789          0123456789          012
+
+# NOALIAS:       [0,0]     DeeeeeeER .   .   addq	$44, 64(%r14)
+# NOALIAS-NEXT:  [0,1]     DeeeeeeER .   .   addq	$44, 128(%r14)
+# NOALIAS-NEXT:  [0,2]     D=eeeeeeER.   .   addq	$44, 192(%r14)
+# NOALIAS-NEXT:  [0,3]     D==eeeeeeER   .   addq	$44, 256(%r14)
+# NOALIAS-NEXT:  [0,4]     D==eeeeeeER   .   addq	$44, 320(%r14)
+# NOALIAS-NEXT:  [0,5]     D===eeeeeeER  .   addq	$44, 384(%r14)
+# NOALIAS-NEXT:  [0,6]     .D===eeeeeeER .   addq	$44, 448(%r14)
+# NOALIAS-NEXT:  [0,7]     .D===eeeeeeER .   addq	$44, 512(%r14)
+# NOALIAS-NEXT:  [0,8]     .D====eeeeeeER.   addq	$44, 576(%r14)
+# NOALIAS-NEXT:  [0,9]     .D=====eeeeeeER   addq	$44, 640(%r14)
+
+# YESALIAS:      [0,0]     DeeeeeeER .    .    .    .    .    .    .    .    .    .    . .   addq	$44, 64(%r14)
+# YESALIAS-NEXT: [0,1]     D======eeeeeeER.    .    .    .    .    .    .    .    .    . .   addq	$44, 128(%r14)
+# YESALIAS-NEXT: [0,2]     D============eeeeeeER    .    .    .    .    .    .    .    . .   addq	$44, 192(%r14)
+# YESALIAS-NEXT: [0,3]     D==================eeeeeeER   .    .    .    .    .    .    . .   addq	$44, 256(%r14)
+# YESALIAS-NEXT: [0,4]     D========================eeeeeeER  .    .    .    .    .    . .   addq	$44, 320(%r14)
+# YESALIAS-NEXT: [0,5]     D==============================eeeeeeER .    .    .    .    . .   addq	$44, 384(%r14)
+# YESALIAS-NEXT: [0,6]     .D===================================eeeeeeER.    .    .    . .   addq	$44, 448(%r14)
+# YESALIAS-NEXT: [0,7]     .D=========================================eeeeeeER    .    . .   addq	$44, 512(%r14)
+# YESALIAS-NEXT: [0,8]     .D===============================================eeeeeeER   . .   addq	$44, 576(%r14)
+# YESALIAS-NEXT: [0,9]     .D=====================================================eeeeeeER   addq	$44, 640(%r14)
+
+# ALL:           Average Wait times (based on the timeline view):
+# ALL-NEXT:      [0]: Executions
+# ALL-NEXT:      [1]: Average time spent waiting in a scheduler's queue
+# ALL-NEXT:      [2]: Average time spent waiting in a scheduler's queue while ready
+# ALL-NEXT:      [3]: Average time elapsed from WB until retire stage
+
+# ALL:                 [0]    [1]    [2]    [3]
+# ALL-NEXT:      0.     1     1.0    1.0    0.0       addq	$44, 64(%r14)
+
+# NOALIAS-NEXT:  1.     1     1.0    0.0    0.0       addq	$44, 128(%r14)
+# NOALIAS-NEXT:  2.     1     2.0    1.0    0.0       addq	$44, 192(%r14)
+# NOALIAS-NEXT:  3.     1     3.0    1.0    0.0       addq	$44, 256(%r14)
+# NOALIAS-NEXT:  4.     1     3.0    0.0    0.0       addq	$44, 320(%r14)
+# NOALIAS-NEXT:  5.     1     4.0    1.0    0.0       addq	$44, 384(%r14)
+# NOALIAS-NEXT:  6.     1     4.0    1.0    0.0       addq	$44, 448(%r14)
+# NOALIAS-NEXT:  7.     1     4.0    0.0    0.0       addq	$44, 512(%r14)
+# NOALIAS-NEXT:  8.     1     5.0    1.0    0.0       addq	$44, 576(%r14)
+# NOALIAS-NEXT:  9.     1     6.0    1.0    0.0       addq	$44, 640(%r14)
+# NOALIAS-NEXT:         1     3.3    0.7    0.0       <total>
+
+# YESALIAS-NEXT: 1.     1     7.0    0.0    0.0       addq	$44, 128(%r14)
+# YESALIAS-NEXT: 2.     1     13.0   0.0    0.0       addq	$44, 192(%r14)
+# YESALIAS-NEXT: 3.     1     19.0   0.0    0.0       addq	$44, 256(%r14)
+# YESALIAS-NEXT: 4.     1     25.0   0.0    0.0       addq	$44, 320(%r14)
+# YESALIAS-NEXT: 5.     1     31.0   0.0    0.0       addq	$44, 384(%r14)
+# YESALIAS-NEXT: 6.     1     36.0   0.0    0.0       addq	$44, 448(%r14)
+# YESALIAS-NEXT: 7.     1     42.0   0.0    0.0       addq	$44, 512(%r14)
+# YESALIAS-NEXT: 8.     1     48.0   0.0    0.0       addq	$44, 576(%r14)
+# YESALIAS-NEXT: 9.     1     54.0   0.0    0.0       addq	$44, 640(%r14)
+# YESALIAS-NEXT:        1     27.6   0.1    0.0       <total>
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-adx.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-adx.s
new file mode 100644
index 0000000000000..fdc3662f4db56
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-adx.s
@@ -0,0 +1,70 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+adcx        %ebx, %ecx
+adcx        (%rbx), %ecx
+adcx        %rbx, %rcx
+adcx        (%rbx), %rcx
+
+adox        %ebx, %ecx
+adox        (%rbx), %ecx
+adox        %rbx, %rcx
+adox        (%rbx), %rcx
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     1.00                        adcxl	%ebx, %ecx
+# CHECK-NEXT:  1      5     1.00    *                   adcxl	(%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        adcxq	%rbx, %rcx
+# CHECK-NEXT:  1      5     1.00    *                   adcxq	(%rbx), %rcx
+# CHECK-NEXT:  1      1     1.00                        adoxl	%ebx, %ecx
+# CHECK-NEXT:  1      5     1.00    *                   adoxl	(%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        adoxq	%rbx, %rcx
+# CHECK-NEXT:  1      5     1.00    *                   adoxq	(%rbx), %rcx
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 1.33   1.33   1.33   8.00   8.00   8.00   8.00    -      -      -      -      -      -      -      -     1.33   1.33   1.33   1.33   1.33   1.33    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcxl	%ebx, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     adcxl	(%rbx), %ecx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcxq	%rbx, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     adcxq	(%rbx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adoxl	%ebx, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     adoxl	(%rbx), %ecx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adoxq	%rbx, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     adoxq	(%rbx), %rcx
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-aes.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-aes.s
new file mode 100644
index 0000000000000..7d7c4495588f8
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-aes.s
@@ -0,0 +1,86 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+aesdec          %xmm0, %xmm2
+aesdec          (%rax), %xmm2
+
+aesdeclast      %xmm0, %xmm2
+aesdeclast      (%rax), %xmm2
+
+aesenc          %xmm0, %xmm2
+aesenc          (%rax), %xmm2
+
+aesenclast      %xmm0, %xmm2
+aesenclast      (%rax), %xmm2
+
+aesimc          %xmm0, %xmm2
+aesimc          (%rax), %xmm2
+
+aeskeygenassist $22, %xmm0, %xmm2
+aeskeygenassist $22, (%rax), %xmm2
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      4     0.50                        aesdec	%xmm0, %xmm2
+# CHECK-NEXT:  1      11    0.50    *                   aesdec	(%rax), %xmm2
+# CHECK-NEXT:  1      4     0.50                        aesdeclast	%xmm0, %xmm2
+# CHECK-NEXT:  1      11    0.50    *                   aesdeclast	(%rax), %xmm2
+# CHECK-NEXT:  1      4     0.50                        aesenc	%xmm0, %xmm2
+# CHECK-NEXT:  1      11    0.50    *                   aesenc	(%rax), %xmm2
+# CHECK-NEXT:  1      4     0.50                        aesenclast	%xmm0, %xmm2
+# CHECK-NEXT:  1      11    0.50    *                   aesenclast	(%rax), %xmm2
+# CHECK-NEXT:  1      4     0.50                        aesimc	%xmm0, %xmm2
+# CHECK-NEXT:  1      11    0.50    *                   aesimc	(%rax), %xmm2
+# CHECK-NEXT:  1      4     0.50                        aeskeygenassist	$22, %xmm0, %xmm2
+# CHECK-NEXT:  1      11    0.50    *                   aeskeygenassist	$22, (%rax), %xmm2
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     6.00   6.00    -      -     3.00   3.00    -     2.00   2.00   2.00   2.00   2.00   2.00    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     aesdec	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     aesdec	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     aesdeclast	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     aesdeclast	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     aesenc	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     aesenc	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     aesenclast	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     aesenclast	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     aesimc	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     aesimc	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     aeskeygenassist	$22, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     aeskeygenassist	$22, (%rax), %xmm2
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx1.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx1.s
new file mode 100644
index 0000000000000..ab3d2413c3f6d
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx1.s
@@ -0,0 +1,2446 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vaddpd            %xmm0, %xmm1, %xmm2
+vaddpd            (%rax), %xmm1, %xmm2
+
+vaddpd            %ymm0, %ymm1, %ymm2
+vaddpd            (%rax), %ymm1, %ymm2
+
+vaddps            %xmm0, %xmm1, %xmm2
+vaddps            (%rax), %xmm1, %xmm2
+
+vaddps            %ymm0, %ymm1, %ymm2
+vaddps            (%rax), %ymm1, %ymm2
+
+vaddsd            %xmm0, %xmm1, %xmm2
+vaddsd            (%rax), %xmm1, %xmm2
+
+vaddss            %xmm0, %xmm1, %xmm2
+vaddss            (%rax), %xmm1, %xmm2
+
+vaddsubpd         %xmm0, %xmm1, %xmm2
+vaddsubpd         (%rax), %xmm1, %xmm2
+
+vaddsubpd         %ymm0, %ymm1, %ymm2
+vaddsubpd         (%rax), %ymm1, %ymm2
+
+vaddsubps         %xmm0, %xmm1, %xmm2
+vaddsubps         (%rax), %xmm1, %xmm2
+
+vaddsubps         %ymm0, %ymm1, %ymm2
+vaddsubps         (%rax), %ymm1, %ymm2
+
+vaesdec           %xmm0, %xmm1, %xmm2
+vaesdec           (%rax), %xmm1, %xmm2
+
+vaesdeclast       %xmm0, %xmm1, %xmm2
+vaesdeclast       (%rax), %xmm1, %xmm2
+
+vaesenc           %xmm0, %xmm1, %xmm2
+vaesenc           (%rax), %xmm1, %xmm2
+
+vaesenclast       %xmm0, %xmm1, %xmm2
+vaesenclast       (%rax), %xmm1, %xmm2
+
+vaesimc           %xmm0, %xmm2
+vaesimc           (%rax), %xmm2
+
+vaeskeygenassist  $22, %xmm0, %xmm2
+vaeskeygenassist  $22, (%rax), %xmm2
+
+vandnpd           %xmm0, %xmm1, %xmm2
+vandnpd           (%rax), %xmm1, %xmm2
+
+vandnpd           %ymm0, %ymm1, %ymm2
+vandnpd           (%rax), %ymm1, %ymm2
+
+vandnps           %xmm0, %xmm1, %xmm2
+vandnps           (%rax), %xmm1, %xmm2
+
+vandnps           %ymm0, %ymm1, %ymm2
+vandnps           (%rax), %ymm1, %ymm2
+
+vandpd            %xmm0, %xmm1, %xmm2
+vandpd            (%rax), %xmm1, %xmm2
+
+vandpd            %ymm0, %ymm1, %ymm2
+vandpd            (%rax), %ymm1, %ymm2
+
+vandps            %xmm0, %xmm1, %xmm2
+vandps            (%rax), %xmm1, %xmm2
+
+vandps            %ymm0, %ymm1, %ymm2
+vandps            (%rax), %ymm1, %ymm2
+
+vblendpd          $11, %xmm0, %xmm1, %xmm2
+vblendpd          $11, (%rax), %xmm1, %xmm2
+
+vblendpd          $11, %ymm0, %ymm1, %ymm2
+vblendpd          $11, (%rax), %ymm1, %ymm2
+
+vblendps          $11, %xmm0, %xmm1, %xmm2
+vblendps          $11, (%rax), %xmm1, %xmm2
+
+vblendps          $11, %ymm0, %ymm1, %ymm2
+vblendps          $11, (%rax), %ymm1, %ymm2
+
+vblendvpd         %xmm3, %xmm0, %xmm1, %xmm2
+vblendvpd         %xmm3, (%rax), %xmm1, %xmm2
+
+vblendvpd         %ymm3, %ymm0, %ymm1, %ymm2
+vblendvpd         %ymm3, (%rax), %ymm1, %ymm2
+
+vblendvps         %xmm3, %xmm0, %xmm1, %xmm2
+vblendvps         %xmm3, (%rax), %xmm1, %xmm2
+
+vblendvps         %ymm3, %ymm0, %ymm1, %ymm2
+vblendvps         %ymm3, (%rax), %ymm1, %ymm2
+
+vbroadcastf128    (%rax), %ymm2
+
+vbroadcastsd      (%rax), %ymm2
+
+vbroadcastss      (%rax), %xmm2
+vbroadcastss      (%rax), %ymm2
+
+vcmppd            $0, %xmm0, %xmm1, %xmm2
+vcmppd            $0, (%rax), %xmm1, %xmm2
+
+vcmppd            $0, %ymm0, %ymm1, %ymm2
+vcmppd            $0, (%rax), %ymm1, %ymm2
+
+vcmpps            $0, %xmm0, %xmm1, %xmm2
+vcmpps            $0, (%rax), %xmm1, %xmm2
+
+vcmpps            $0, %ymm0, %ymm1, %ymm2
+vcmpps            $0, (%rax), %ymm1, %ymm2
+
+vcmpsd            $0, %xmm0, %xmm1, %xmm2
+vcmpsd            $0, (%rax), %xmm1, %xmm2
+
+vcmpss            $0, %xmm0, %xmm1, %xmm2
+vcmpss            $0, (%rax), %xmm1, %xmm2
+
+vcomisd           %xmm0, %xmm1
+vcomisd           (%rax), %xmm1
+
+vcomiss           %xmm0, %xmm1
+vcomiss           (%rax), %xmm1
+
+vcvtdq2pd         %xmm0, %xmm2
+vcvtdq2pd         (%rax), %xmm2
+
+vcvtdq2pd         %xmm0, %ymm2
+vcvtdq2pd         (%rax), %ymm2
+
+vcvtdq2ps         %xmm0, %xmm2
+vcvtdq2ps         (%rax), %xmm2
+
+vcvtdq2ps         %ymm0, %ymm2
+vcvtdq2ps         (%rax), %ymm2
+
+vcvtpd2dqx        %xmm0, %xmm2
+vcvtpd2dqx        (%rax), %xmm2
+
+vcvtpd2dqy        %ymm0, %xmm2
+vcvtpd2dqy        (%rax), %xmm2
+
+vcvtpd2psx        %xmm0, %xmm2
+vcvtpd2psx        (%rax), %xmm2
+
+vcvtpd2psy        %ymm0, %xmm2
+vcvtpd2psy        (%rax), %xmm2
+
+vcvtps2dq         %xmm0, %xmm2
+vcvtps2dq         (%rax), %xmm2
+
+vcvtps2dq         %ymm0, %ymm2
+vcvtps2dq         (%rax), %ymm2
+
+vcvtps2pd         %xmm0, %xmm2
+vcvtps2pd         (%rax), %xmm2
+
+vcvtps2pd         %xmm0, %ymm2
+vcvtps2pd         (%rax), %ymm2
+
+vcvtsd2si         %xmm0, %ecx
+vcvtsd2si         %xmm0, %rcx
+vcvtsd2si         (%rax), %ecx
+vcvtsd2si         (%rax), %rcx
+
+vcvtsd2ss         %xmm0, %xmm1, %xmm2
+vcvtsd2ss         (%rax), %xmm1, %xmm2
+
+vcvtsi2sdl        %ecx, %xmm0, %xmm2
+vcvtsi2sdq        %rcx, %xmm0, %xmm2
+vcvtsi2sdl        (%rax), %xmm0, %xmm2
+vcvtsi2sdq        (%rax), %xmm0, %xmm2
+
+vcvtsi2ssl        %ecx, %xmm0, %xmm2
+vcvtsi2ssq        %rcx, %xmm0, %xmm2
+vcvtsi2ssl        (%rax), %xmm0, %xmm2
+vcvtsi2ssq        (%rax), %xmm0, %xmm2
+
+vcvtss2sd         %xmm0, %xmm1, %xmm2
+vcvtss2sd         (%rax), %xmm1, %xmm2
+
+vcvtss2si         %xmm0, %ecx
+vcvtss2si         %xmm0, %rcx
+vcvtss2si         (%rax), %ecx
+vcvtss2si         (%rax), %rcx
+
+vcvttpd2dqx       %xmm0, %xmm2
+vcvttpd2dqx       (%rax), %xmm2
+
+vcvttpd2dqy       %ymm0, %xmm2
+vcvttpd2dqy       (%rax), %xmm2
+
+vcvttps2dq        %xmm0, %xmm2
+vcvttps2dq        (%rax), %xmm2
+
+vcvttps2dq        %ymm0, %ymm2
+vcvttps2dq        (%rax), %ymm2
+
+vcvttsd2si        %xmm0, %ecx
+vcvttsd2si        %xmm0, %rcx
+vcvttsd2si        (%rax), %ecx
+vcvttsd2si        (%rax), %rcx
+
+vcvttss2si        %xmm0, %ecx
+vcvttss2si        %xmm0, %rcx
+vcvttss2si        (%rax), %ecx
+vcvttss2si        (%rax), %rcx
+
+vdivpd            %xmm0, %xmm1, %xmm2
+vdivpd            (%rax), %xmm1, %xmm2
+
+vdivpd            %ymm0, %ymm1, %ymm2
+vdivpd            (%rax), %ymm1, %ymm2
+
+vdivps            %xmm0, %xmm1, %xmm2
+vdivps            (%rax), %xmm1, %xmm2
+
+vdivps            %ymm0, %ymm1, %ymm2
+vdivps            (%rax), %ymm1, %ymm2
+
+vdivsd            %xmm0, %xmm1, %xmm2
+vdivsd            (%rax), %xmm1, %xmm2
+
+vdivss            %xmm0, %xmm1, %xmm2
+vdivss            (%rax), %xmm1, %xmm2
+
+vdppd             $22, %xmm0, %xmm1, %xmm2
+vdppd             $22, (%rax), %xmm1, %xmm2
+
+vdpps             $22, %xmm0, %xmm1, %xmm2
+vdpps             $22, (%rax), %xmm1, %xmm2
+
+vdpps             $22, %ymm0, %ymm1, %ymm2
+vdpps             $22, (%rax), %ymm1, %ymm2
+
+vextractf128      $1, %ymm0, %xmm2
+vextractf128      $1, %ymm0, (%rax)
+
+vextractps        $1, %xmm0, %rcx
+vextractps        $1, %xmm0, (%rax)
+
+vhaddpd           %xmm0, %xmm1, %xmm2
+vhaddpd           (%rax), %xmm1, %xmm2
+
+vhaddpd           %ymm0, %ymm1, %ymm2
+vhaddpd           (%rax), %ymm1, %ymm2
+
+vhaddps           %xmm0, %xmm1, %xmm2
+vhaddps           (%rax), %xmm1, %xmm2
+
+vhaddps           %ymm0, %ymm1, %ymm2
+vhaddps           (%rax), %ymm1, %ymm2
+
+vhsubpd           %xmm0, %xmm1, %xmm2
+vhsubpd           (%rax), %xmm1, %xmm2
+
+vhsubpd           %ymm0, %ymm1, %ymm2
+vhsubpd           (%rax), %ymm1, %ymm2
+
+vhsubps           %xmm0, %xmm1, %xmm2
+vhsubps           (%rax), %xmm1, %xmm2
+
+vhsubps           %ymm0, %ymm1, %ymm2
+vhsubps           (%rax), %ymm1, %ymm2
+
+vinsertf128       $1, %xmm0, %ymm1, %ymm2
+vinsertf128       $1, (%rax), %ymm1, %ymm2
+
+vinsertps         $1, %xmm0, %xmm1, %xmm2
+vinsertps         $1, (%rax), %xmm1, %xmm2
+
+vlddqu            (%rax), %xmm2
+vlddqu            (%rax), %ymm2
+
+vldmxcsr          (%rax)
+
+vmaskmovdqu       %xmm0, %xmm1
+
+vmaskmovpd        (%rax), %xmm0, %xmm2
+vmaskmovpd        (%rax), %ymm0, %ymm2
+
+vmaskmovpd        %xmm0, %xmm1, (%rax)
+vmaskmovpd        %ymm0, %ymm1, (%rax)
+
+vmaskmovps        (%rax), %xmm0, %xmm2
+vmaskmovps        (%rax), %ymm0, %ymm2
+
+vmaskmovps        %xmm0, %xmm1, (%rax)
+vmaskmovps        %ymm0, %ymm1, (%rax)
+
+vmaxpd            %xmm0, %xmm1, %xmm2
+vmaxpd            (%rax), %xmm1, %xmm2
+
+vmaxpd            %ymm0, %ymm1, %ymm2
+vmaxpd            (%rax), %ymm1, %ymm2
+
+vmaxps            %xmm0, %xmm1, %xmm2
+vmaxps            (%rax), %xmm1, %xmm2
+
+vmaxps            %ymm0, %ymm1, %ymm2
+vmaxps            (%rax), %ymm1, %ymm2
+
+vmaxsd            %xmm0, %xmm1, %xmm2
+vmaxsd            (%rax), %xmm1, %xmm2
+
+vmaxss            %xmm0, %xmm1, %xmm2
+vmaxss            (%rax), %xmm1, %xmm2
+
+vminpd            %xmm0, %xmm1, %xmm2
+vminpd            (%rax), %xmm1, %xmm2
+
+vminpd            %ymm0, %ymm1, %ymm2
+vminpd            (%rax), %ymm1, %ymm2
+
+vminps            %xmm0, %xmm1, %xmm2
+vminps            (%rax), %xmm1, %xmm2
+
+vminps            %ymm0, %ymm1, %ymm2
+vminps            (%rax), %ymm1, %ymm2
+
+vminsd            %xmm0, %xmm1, %xmm2
+vminsd            (%rax), %xmm1, %xmm2
+
+vminss            %xmm0, %xmm1, %xmm2
+vminss            (%rax), %xmm1, %xmm2
+
+vmovapd           %xmm0, %xmm2
+vmovapd           %xmm0, (%rax)
+vmovapd           (%rax), %xmm2
+
+vmovapd           %ymm0, %ymm2
+vmovapd           %ymm0, (%rax)
+vmovapd           (%rax), %ymm2
+
+vmovaps           %xmm0, %xmm2
+vmovaps           %xmm0, (%rax)
+vmovaps           (%rax), %xmm2
+
+vmovaps           %ymm0, %ymm2
+vmovaps           %ymm0, (%rax)
+vmovaps           (%rax), %ymm2
+
+vmovd             %eax, %xmm2
+vmovd             (%rax), %xmm2
+
+vmovd             %xmm0, %ecx
+vmovd             %xmm0, (%rax)
+
+vmovddup          %xmm0, %xmm2
+vmovddup          (%rax), %xmm2
+
+vmovddup          %ymm0, %ymm2
+vmovddup          (%rax), %ymm2
+
+vmovdqa           %xmm0, %xmm2
+vmovdqa           %xmm0, (%rax)
+vmovdqa           (%rax), %xmm2
+
+vmovdqa           %ymm0, %ymm2
+vmovdqa           %ymm0, (%rax)
+vmovdqa           (%rax), %ymm2
+
+vmovdqu           %xmm0, %xmm2
+vmovdqu           %xmm0, (%rax)
+vmovdqu           (%rax), %xmm2
+
+vmovdqu           %ymm0, %ymm2
+vmovdqu           %ymm0, (%rax)
+vmovdqu           (%rax), %ymm2
+
+vmovhlps          %xmm0, %xmm1, %xmm2
+vmovlhps          %xmm0, %xmm1, %xmm2
+
+vmovhpd           %xmm0, (%rax)
+vmovhpd           (%rax), %xmm1, %xmm2
+
+vmovhps           %xmm0, (%rax)
+vmovhps           (%rax), %xmm1, %xmm2
+
+vmovlpd           %xmm0, (%rax)
+vmovlpd           (%rax), %xmm1, %xmm2
+
+vmovlps           %xmm0, (%rax)
+vmovlps           (%rax), %xmm1, %xmm2
+
+vmovmskpd         %xmm0, %rcx
+vmovmskpd         %ymm0, %rcx
+
+vmovmskps         %xmm0, %rcx
+vmovmskps         %ymm0, %rcx
+
+vmovntdq          %xmm0, (%rax)
+vmovntdq          %ymm0, (%rax)
+
+vmovntdqa         (%rax), %xmm2
+vmovntdqa         (%rax), %ymm2
+
+vmovntpd          %xmm0, (%rax)
+vmovntpd          %ymm0, (%rax)
+
+vmovntps          %xmm0, (%rax)
+vmovntps          %ymm0, (%rax)
+
+vmovq             %xmm0, %xmm2
+
+vmovq             %rax, %xmm2
+vmovq             (%rax), %xmm2
+
+vmovq             %xmm0, %rcx
+vmovq             %xmm0, (%rax)
+
+vmovsd            %xmm0, %xmm1, %xmm2
+vmovsd            %xmm0, (%rax)
+vmovsd            (%rax), %xmm2
+
+vmovshdup         %xmm0, %xmm2
+vmovshdup         (%rax), %xmm2
+
+vmovshdup         %ymm0, %ymm2
+vmovshdup         (%rax), %ymm2
+
+vmovsldup         %xmm0, %xmm2
+vmovsldup         (%rax), %xmm2
+
+vmovsldup         %ymm0, %ymm2
+vmovsldup         (%rax), %ymm2
+
+vmovss            %xmm0, %xmm1, %xmm2
+vmovss            %xmm0, (%rax)
+vmovss            (%rax), %xmm2
+
+vmovupd           %xmm0, %xmm2
+vmovupd           %xmm0, (%rax)
+vmovupd           (%rax), %xmm2
+
+vmovupd           %ymm0, %ymm2
+vmovupd           %ymm0, (%rax)
+vmovupd           (%rax), %ymm2
+
+vmovups           %xmm0, %xmm2
+vmovups           %xmm0, (%rax)
+vmovups           (%rax), %xmm2
+
+vmovups           %ymm0, %ymm2
+vmovups           %ymm0, (%rax)
+vmovups           (%rax), %ymm2
+
+vmpsadbw          $1, %xmm0, %xmm1, %xmm2
+vmpsadbw          $1, (%rax), %xmm1, %xmm2
+
+vmulpd            %xmm0, %xmm1, %xmm2
+vmulpd            (%rax), %xmm1, %xmm2
+
+vmulpd            %ymm0, %ymm1, %ymm2
+vmulpd            (%rax), %ymm1, %ymm2
+
+vmulps            %xmm0, %xmm1, %xmm2
+vmulps            (%rax), %xmm1, %xmm2
+
+vmulps            %ymm0, %ymm1, %ymm2
+vmulps            (%rax), %ymm1, %ymm2
+
+vmulsd            %xmm0, %xmm1, %xmm2
+vmulsd            (%rax), %xmm1, %xmm2
+
+vmulss            %xmm0, %xmm1, %xmm2
+vmulss            (%rax), %xmm1, %xmm2
+
+vorpd             %xmm0, %xmm1, %xmm2
+vorpd             (%rax), %xmm1, %xmm2
+
+vorpd             %ymm0, %ymm1, %ymm2
+vorpd             (%rax), %ymm1, %ymm2
+
+vorps             %xmm0, %xmm1, %xmm2
+vorps             (%rax), %xmm1, %xmm2
+
+vorps             %ymm0, %ymm1, %ymm2
+vorps             (%rax), %ymm1, %ymm2
+
+vpabsb            %xmm0, %xmm2
+vpabsb            (%rax), %xmm2
+
+vpabsd            %xmm0, %xmm2
+vpabsd            (%rax), %xmm2
+
+vpabsw            %xmm0, %xmm2
+vpabsw            (%rax), %xmm2
+
+vpackssdw         %xmm0, %xmm1, %xmm2
+vpackssdw         (%rax), %xmm1, %xmm2
+
+vpacksswb         %xmm0, %xmm1, %xmm2
+vpacksswb         (%rax), %xmm1, %xmm2
+
+vpackusdw         %xmm0, %xmm1, %xmm2
+vpackusdw         (%rax), %xmm1, %xmm2
+
+vpackuswb         %xmm0, %xmm1, %xmm2
+vpackuswb         (%rax), %xmm1, %xmm2
+
+vpaddb            %xmm0, %xmm1, %xmm2
+vpaddb            (%rax), %xmm1, %xmm2
+
+vpaddd            %xmm0, %xmm1, %xmm2
+vpaddd            (%rax), %xmm1, %xmm2
+
+vpaddq            %xmm0, %xmm1, %xmm2
+vpaddq            (%rax), %xmm1, %xmm2
+
+vpaddsb           %xmm0, %xmm1, %xmm2
+vpaddsb           (%rax), %xmm1, %xmm2
+
+vpaddsw           %xmm0, %xmm1, %xmm2
+vpaddsw           (%rax), %xmm1, %xmm2
+
+vpaddusb          %xmm0, %xmm1, %xmm2
+vpaddusb          (%rax), %xmm1, %xmm2
+
+vpaddusw          %xmm0, %xmm1, %xmm2
+vpaddusw          (%rax), %xmm1, %xmm2
+
+vpaddw            %xmm0, %xmm1, %xmm2
+vpaddw            (%rax), %xmm1, %xmm2
+
+vpalignr          $1, %xmm0, %xmm1, %xmm2
+vpalignr          $1, (%rax), %xmm1, %xmm2
+
+vpand             %xmm0, %xmm1, %xmm2
+vpand             (%rax), %xmm1, %xmm2
+
+vpandn            %xmm0, %xmm1, %xmm2
+vpandn            (%rax), %xmm1, %xmm2
+
+vpavgb            %xmm0, %xmm1, %xmm2
+vpavgb            (%rax), %xmm1, %xmm2
+
+vpavgw            %xmm0, %xmm1, %xmm2
+vpavgw            (%rax), %xmm1, %xmm2
+
+vpblendvb         %xmm3, %xmm0, %xmm1, %xmm2
+vpblendvb         %xmm3, (%rax), %xmm1, %xmm2
+
+vpblendw          $11, %xmm0, %xmm1, %xmm2
+vpblendw          $11, (%rax), %xmm1, %xmm2
+
+vpclmulqdq        $11, %xmm0, %xmm1, %xmm2
+vpclmulqdq        $11, (%rax), %xmm1, %xmm2
+
+vpcmpeqb          %xmm0, %xmm1, %xmm2
+vpcmpeqb          (%rax), %xmm1, %xmm2
+
+vpcmpeqd          %xmm0, %xmm1, %xmm2
+vpcmpeqd          (%rax), %xmm1, %xmm2
+
+vpcmpeqq          %xmm0, %xmm1, %xmm2
+vpcmpeqq          (%rax), %xmm1, %xmm2
+
+vpcmpeqw          %xmm0, %xmm1, %xmm2
+vpcmpeqw          (%rax), %xmm1, %xmm2
+
+vpcmpestri        $1, %xmm0, %xmm2
+vpcmpestri        $1, (%rax), %xmm2
+
+vpcmpestrm        $1, %xmm0, %xmm2
+vpcmpestrm        $1, (%rax), %xmm2
+
+vpcmpgtb          %xmm0, %xmm1, %xmm2
+vpcmpgtb          (%rax), %xmm1, %xmm2
+
+vpcmpgtd          %xmm0, %xmm1, %xmm2
+vpcmpgtd          (%rax), %xmm1, %xmm2
+
+vpcmpgtq          %xmm0, %xmm1, %xmm2
+vpcmpgtq          (%rax), %xmm1, %xmm2
+
+vpcmpgtw          %xmm0, %xmm1, %xmm2
+vpcmpgtw          (%rax), %xmm1, %xmm2
+
+vpcmpistri        $1, %xmm0, %xmm2
+vpcmpistri        $1, (%rax), %xmm2
+
+vpcmpistrm        $1, %xmm0, %xmm2
+vpcmpistrm        $1, (%rax), %xmm2
+
+vperm2f128        $1, %ymm0, %ymm1, %ymm2
+vperm2f128        $1, (%rax), %ymm1, %ymm2
+
+vpermilpd         $1, %xmm0, %xmm2
+vpermilpd         $1, (%rax), %xmm2
+vpermilpd         %xmm0, %xmm1, %xmm2
+vpermilpd         (%rax), %xmm1, %xmm2
+
+vpermilpd         $1, %ymm0, %ymm2
+vpermilpd         $1, (%rax), %ymm2
+vpermilpd         %ymm0, %ymm1, %ymm2
+vpermilpd         (%rax), %ymm1, %ymm2
+
+vpermilps         $1, %xmm0, %xmm2
+vpermilps         $1, (%rax), %xmm2
+vpermilps         %xmm0, %xmm1, %xmm2
+vpermilps         (%rax), %xmm1, %xmm2
+
+vpermilps         $1, %ymm0, %ymm2
+vpermilps         $1, (%rax), %ymm2
+vpermilps         %ymm0, %ymm1, %ymm2
+vpermilps         (%rax), %ymm1, %ymm2
+
+vpextrb           $1, %xmm0, %ecx
+vpextrb           $1, %xmm0, (%rax)
+
+vpextrd           $1, %xmm0, %ecx
+vpextrd           $1, %xmm0, (%rax)
+
+vpextrq           $1, %xmm0, %rcx
+vpextrq           $1, %xmm0, (%rax)
+
+vpextrw           $1, %xmm0, %ecx
+vpextrw           $1, %xmm0, (%rax)
+
+vphaddd           %xmm0, %xmm1, %xmm2
+vphaddd           (%rax), %xmm1, %xmm2
+
+vphaddsw          %xmm0, %xmm1, %xmm2
+vphaddsw          (%rax), %xmm1, %xmm2
+
+vphaddw           %xmm0, %xmm1, %xmm2
+vphaddw           (%rax), %xmm1, %xmm2
+
+vphminposuw       %xmm0, %xmm2
+vphminposuw       (%rax), %xmm2
+
+vphsubd           %xmm0, %xmm1, %xmm2
+vphsubd           (%rax), %xmm1, %xmm2
+
+vphsubsw          %xmm0, %xmm1, %xmm2
+vphsubsw          (%rax), %xmm1, %xmm2
+
+vphsubw           %xmm0, %xmm1, %xmm2
+vphsubw           (%rax), %xmm1, %xmm2
+
+vpinsrb           $1, %eax, %xmm1, %xmm2
+vpinsrb           $1, (%rax), %xmm1, %xmm2
+
+vpinsrd           $1, %eax, %xmm1, %xmm2
+vpinsrd           $1, (%rax), %xmm1, %xmm2
+
+vpinsrq           $1, %rax, %xmm1, %xmm2
+vpinsrq           $1, (%rax), %xmm1, %xmm2
+
+vpinsrw           $1, %eax, %xmm1, %xmm2
+vpinsrw           $1, (%rax), %xmm1, %xmm2
+
+vpmaddubsw        %xmm0, %xmm1, %xmm2
+vpmaddubsw        (%rax), %xmm1, %xmm2
+
+vpmaddwd          %xmm0, %xmm1, %xmm2
+vpmaddwd          (%rax), %xmm1, %xmm2
+
+vpmaxsb           %xmm0, %xmm1, %xmm2
+vpmaxsb           (%rax), %xmm1, %xmm2
+
+vpmaxsd           %xmm0, %xmm1, %xmm2
+vpmaxsd           (%rax), %xmm1, %xmm2
+
+vpmaxsw           %xmm0, %xmm1, %xmm2
+vpmaxsw           (%rax), %xmm1, %xmm2
+
+vpmaxub           %xmm0, %xmm1, %xmm2
+vpmaxub           (%rax), %xmm1, %xmm2
+
+vpmaxud           %xmm0, %xmm1, %xmm2
+vpmaxud           (%rax), %xmm1, %xmm2
+
+vpmaxuw           %xmm0, %xmm1, %xmm2
+vpmaxuw           (%rax), %xmm1, %xmm2
+
+vpminsb           %xmm0, %xmm1, %xmm2
+vpminsb           (%rax), %xmm1, %xmm2
+
+vpminsd           %xmm0, %xmm1, %xmm2
+vpminsd           (%rax), %xmm1, %xmm2
+
+vpminsw           %xmm0, %xmm1, %xmm2
+vpminsw           (%rax), %xmm1, %xmm2
+
+vpminub           %xmm0, %xmm1, %xmm2
+vpminub           (%rax), %xmm1, %xmm2
+
+vpminud           %xmm0, %xmm1, %xmm2
+vpminud           (%rax), %xmm1, %xmm2
+
+vpminuw           %xmm0, %xmm1, %xmm2
+vpminuw           (%rax), %xmm1, %xmm2
+
+vpmovmskb         %xmm0, %rcx
+
+vpmovsxbd         %xmm0, %xmm2
+vpmovsxbd         (%rax), %xmm2
+
+vpmovsxbq         %xmm0, %xmm2
+vpmovsxbq         (%rax), %xmm2
+
+vpmovsxbw         %xmm0, %xmm2
+vpmovsxbw         (%rax), %xmm2
+
+vpmovsxdq         %xmm0, %xmm2
+vpmovsxdq         (%rax), %xmm2
+
+vpmovsxwd         %xmm0, %xmm2
+vpmovsxwd         (%rax), %xmm2
+
+vpmovsxwq         %xmm0, %xmm2
+vpmovsxwq         (%rax), %xmm2
+
+vpmovzxbd         %xmm0, %xmm2
+vpmovzxbd         (%rax), %xmm2
+
+vpmovzxbq         %xmm0, %xmm2
+vpmovzxbq         (%rax), %xmm2
+
+vpmovzxbw         %xmm0, %xmm2
+vpmovzxbw         (%rax), %xmm2
+
+vpmovzxdq         %xmm0, %xmm2
+vpmovzxdq         (%rax), %xmm2
+
+vpmovzxwd         %xmm0, %xmm2
+vpmovzxwd         (%rax), %xmm2
+
+vpmovzxwq         %xmm0, %xmm2
+vpmovzxwq         (%rax), %xmm2
+
+vpmuldq           %xmm0, %xmm1, %xmm2
+vpmuldq           (%rax), %xmm1, %xmm2
+
+vpmulhrsw         %xmm0, %xmm1, %xmm2
+vpmulhrsw         (%rax), %xmm1, %xmm2
+
+vpmulhuw          %xmm0, %xmm1, %xmm2
+vpmulhuw          (%rax), %xmm1, %xmm2
+
+vpmulhw           %xmm0, %xmm1, %xmm2
+vpmulhw           (%rax), %xmm1, %xmm2
+
+vpmulld           %xmm0, %xmm1, %xmm2
+vpmulld           (%rax), %xmm1, %xmm2
+
+vpmullw           %xmm0, %xmm1, %xmm2
+vpmullw           (%rax), %xmm1, %xmm2
+
+vpmuludq          %xmm0, %xmm1, %xmm2
+vpmuludq          (%rax), %xmm1, %xmm2
+
+vpor              %xmm0, %xmm1, %xmm2
+vpor              (%rax), %xmm1, %xmm2
+
+vpsadbw           %xmm0, %xmm1, %xmm2
+vpsadbw           (%rax), %xmm1, %xmm2
+
+vpshufb           %xmm0, %xmm1, %xmm2
+vpshufb           (%rax), %xmm1, %xmm2
+
+vpshufd           $1, %xmm0, %xmm2
+vpshufd           $1, (%rax), %xmm2
+
+vpshufhw          $1, %xmm0, %xmm2
+vpshufhw          $1, (%rax), %xmm2
+
+vpshuflw          $1, %xmm0, %xmm2
+vpshuflw          $1, (%rax), %xmm2
+
+vpsignb           %xmm0, %xmm1, %xmm2
+vpsignb           (%rax), %xmm1, %xmm2
+
+vpsignd           %xmm0, %xmm1, %xmm2
+vpsignd           (%rax), %xmm1, %xmm2
+
+vpsignw           %xmm0, %xmm1, %xmm2
+vpsignw           (%rax), %xmm1, %xmm2
+
+vpslld            $1, %xmm0, %xmm2
+vpslld            %xmm0, %xmm1, %xmm2
+vpslld            (%rax), %xmm1, %xmm2
+
+vpslldq           $1, %xmm1, %xmm2
+
+vpsllq            $1, %xmm0, %xmm2
+vpsllq            %xmm0, %xmm1, %xmm2
+vpsllq            (%rax), %xmm1, %xmm2
+
+vpsllw            $1, %xmm0, %xmm2
+vpsllw            %xmm0, %xmm1, %xmm2
+vpsllw            (%rax), %xmm1, %xmm2
+
+vpsrad            $1, %xmm0, %xmm2
+vpsrad            %xmm0, %xmm1, %xmm2
+vpsrad            (%rax), %xmm1, %xmm2
+
+vpsraw            $1, %xmm0, %xmm2
+vpsraw            %xmm0, %xmm1, %xmm2
+vpsraw            (%rax), %xmm1, %xmm2
+
+vpsrld            $1, %xmm0, %xmm2
+vpsrld            %xmm0, %xmm1, %xmm2
+vpsrld            (%rax), %xmm1, %xmm2
+
+vpsrldq           $1, %xmm1, %xmm2
+
+vpsrlq            $1, %xmm0, %xmm2
+vpsrlq            %xmm0, %xmm1, %xmm2
+vpsrlq            (%rax), %xmm1, %xmm2
+
+vpsrlw            $1, %xmm0, %xmm2
+vpsrlw            %xmm0, %xmm1, %xmm2
+vpsrlw            (%rax), %xmm1, %xmm2
+
+vpsubb            %xmm0, %xmm1, %xmm2
+vpsubb            (%rax), %xmm1, %xmm2
+
+vpsubd            %xmm0, %xmm1, %xmm2
+vpsubd            (%rax), %xmm1, %xmm2
+
+vpsubq            %xmm0, %xmm1, %xmm2
+vpsubq            (%rax), %xmm1, %xmm2
+
+vpsubsb           %xmm0, %xmm1, %xmm2
+vpsubsb           (%rax), %xmm1, %xmm2
+
+vpsubsw           %xmm0, %xmm1, %xmm2
+vpsubsw           (%rax), %xmm1, %xmm2
+
+vpsubusb          %xmm0, %xmm1, %xmm2
+vpsubusb          (%rax), %xmm1, %xmm2
+
+vpsubusw          %xmm0, %xmm1, %xmm2
+vpsubusw          (%rax), %xmm1, %xmm2
+
+vpsubw            %xmm0, %xmm1, %xmm2
+vpsubw            (%rax), %xmm1, %xmm2
+
+vptest            %xmm0, %xmm1
+vptest            (%rax), %xmm1
+
+vptest            %ymm0, %ymm1
+vptest            (%rax), %ymm1
+
+vpunpckhbw        %xmm0, %xmm1, %xmm2
+vpunpckhbw        (%rax), %xmm1, %xmm2
+
+vpunpckhdq        %xmm0, %xmm1, %xmm2
+vpunpckhdq        (%rax), %xmm1, %xmm2
+
+vpunpckhqdq       %xmm0, %xmm1, %xmm2
+vpunpckhqdq       (%rax), %xmm1, %xmm2
+
+vpunpckhwd        %xmm0, %xmm1, %xmm2
+vpunpckhwd        (%rax), %xmm1, %xmm2
+
+vpunpcklbw        %xmm0, %xmm1, %xmm2
+vpunpcklbw        (%rax), %xmm1, %xmm2
+
+vpunpckldq        %xmm0, %xmm1, %xmm2
+vpunpckldq        (%rax), %xmm1, %xmm2
+
+vpunpcklqdq       %xmm0, %xmm1, %xmm2
+vpunpcklqdq       (%rax), %xmm1, %xmm2
+
+vpunpcklwd        %xmm0, %xmm1, %xmm2
+vpunpcklwd        (%rax), %xmm1, %xmm2
+
+vpxor             %xmm0, %xmm1, %xmm2
+vpxor             (%rax), %xmm1, %xmm2
+
+vrcpps            %xmm0, %xmm2
+vrcpps            (%rax), %xmm2
+
+vrcpps            %ymm0, %ymm2
+vrcpps            (%rax), %ymm2
+
+vrcpss            %xmm0, %xmm1, %xmm2
+vrcpss            (%rax), %xmm1, %xmm2
+
+vroundpd          $1, %xmm0, %xmm2
+vroundpd          $1, (%rax), %xmm2
+
+vroundpd          $1, %ymm0, %ymm2
+vroundpd          $1, (%rax), %ymm2
+
+vroundps          $1, %xmm0, %xmm2
+vroundps          $1, (%rax), %xmm2
+
+vroundps          $1, %ymm0, %ymm2
+vroundps          $1, (%rax), %ymm2
+
+vroundsd          $1, %xmm0, %xmm1, %xmm2
+vroundsd          $1, (%rax), %xmm1, %xmm2
+
+vroundss          $1, %xmm0, %xmm1, %xmm2
+vroundss          $1, (%rax), %xmm1, %xmm2
+
+vrsqrtps          %xmm0, %xmm2
+vrsqrtps          (%rax), %xmm2
+
+vrsqrtps          %ymm0, %ymm2
+vrsqrtps          (%rax), %ymm2
+
+vrsqrtss          %xmm0, %xmm1, %xmm2
+vrsqrtss          (%rax), %xmm1, %xmm2
+
+vshufpd           $1, %xmm0, %xmm1, %xmm2
+vshufpd           $1, (%rax), %xmm1, %xmm2
+
+vshufpd           $1, %ymm0, %ymm1, %ymm2
+vshufpd           $1, (%rax), %ymm1, %ymm2
+
+vshufps           $1, %xmm0, %xmm1, %xmm2
+vshufps           $1, (%rax), %xmm1, %xmm2
+
+vshufps           $1, %ymm0, %ymm1, %ymm2
+vshufps           $1, (%rax), %ymm1, %ymm2
+
+vsqrtpd           %xmm0, %xmm2
+vsqrtpd           (%rax), %xmm2
+
+vsqrtpd           %ymm0, %ymm2
+vsqrtpd           (%rax), %ymm2
+
+vsqrtps           %xmm0, %xmm2
+vsqrtps           (%rax), %xmm2
+
+vsqrtps           %ymm0, %ymm2
+vsqrtps           (%rax), %ymm2
+
+vsqrtsd           %xmm0, %xmm1, %xmm2
+vsqrtsd           (%rax), %xmm1, %xmm2
+
+vsqrtss           %xmm0, %xmm1, %xmm2
+vsqrtss           (%rax), %xmm1, %xmm2
+
+vstmxcsr          (%rax)
+
+vsubpd            %xmm0, %xmm1, %xmm2
+vsubpd            (%rax), %xmm1, %xmm2
+
+vsubpd            %ymm0, %ymm1, %ymm2
+vsubpd            (%rax), %ymm1, %ymm2
+
+vsubps            %xmm0, %xmm1, %xmm2
+vsubps            (%rax), %xmm1, %xmm2
+
+vsubps            %ymm0, %ymm1, %ymm2
+vsubps            (%rax), %ymm1, %ymm2
+
+vsubsd            %xmm0, %xmm1, %xmm2
+vsubsd            (%rax), %xmm1, %xmm2
+
+vsubss            %xmm0, %xmm1, %xmm2
+vsubss            (%rax), %xmm1, %xmm2
+
+vtestpd          %xmm0, %xmm1
+vtestpd          (%rax), %xmm1
+
+vtestpd          %ymm0, %ymm1
+vtestpd          (%rax), %ymm1
+
+vtestps          %xmm0, %xmm1
+vtestps          (%rax), %xmm1
+
+vtestps          %ymm0, %ymm1
+vtestps          (%rax), %ymm1
+
+vucomisd          %xmm0, %xmm1
+vucomisd          (%rax), %xmm1
+
+vucomiss          %xmm0, %xmm1
+vucomiss          (%rax), %xmm1
+
+vunpckhpd         %xmm0, %xmm1, %xmm2
+vunpckhpd         (%rax), %xmm1, %xmm2
+
+vunpckhpd         %ymm0, %ymm1, %ymm2
+vunpckhpd         (%rax), %ymm1, %ymm2
+
+vunpckhps         %xmm0, %xmm1, %xmm2
+vunpckhps         (%rax), %xmm1, %xmm2
+
+vunpckhps         %ymm0, %ymm1, %ymm2
+vunpckhps         (%rax), %ymm1, %ymm2
+
+vunpcklpd         %xmm0, %xmm1, %xmm2
+vunpcklpd         (%rax), %xmm1, %xmm2
+
+vunpcklpd         %ymm0, %ymm1, %ymm2
+vunpcklpd         (%rax), %ymm1, %ymm2
+
+vunpcklps         %xmm0, %xmm1, %xmm2
+vunpcklps         (%rax), %xmm1, %xmm2
+
+vunpcklps         %ymm0, %ymm1, %ymm2
+vunpcklps         (%rax), %ymm1, %ymm2
+
+vxorpd            %xmm0, %xmm1, %xmm2
+vxorpd            (%rax), %xmm1, %xmm2
+
+vxorpd            %ymm0, %ymm1, %ymm2
+vxorpd            (%rax), %ymm1, %ymm2
+
+vxorps            %xmm0, %xmm1, %xmm2
+vxorps            (%rax), %xmm1, %xmm2
+
+vxorps            %ymm0, %ymm1, %ymm2
+vxorps            (%rax), %ymm1, %ymm2
+
+vzeroall
+vzeroupper
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      3     0.50                        vaddpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vaddpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vaddpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vaddpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vaddps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vaddps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vaddps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vaddps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vaddsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vaddsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vaddss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vaddss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vaddsubpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vaddsubpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vaddsubpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vaddsubpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vaddsubps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vaddsubps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vaddsubps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vaddsubps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     0.50                        vaesdec	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    0.50    *                   vaesdec	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     0.50                        vaesdeclast	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    0.50    *                   vaesdeclast	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     0.50                        vaesenc	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    0.50    *                   vaesenc	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     0.50                        vaesenclast	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    0.50    *                   vaesenclast	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     0.50                        vaesimc	%xmm0, %xmm2
+# CHECK-NEXT:  1      11    0.50    *                   vaesimc	(%rax), %xmm2
+# CHECK-NEXT:  1      4     0.50                        vaeskeygenassist	$22, %xmm0, %xmm2
+# CHECK-NEXT:  1      11    0.50    *                   vaeskeygenassist	$22, (%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        vandnpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vandnpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vandnps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vandnps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vandpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vandpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vandps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vandps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vblendpd	$11, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vblendpd	$11, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vblendpd	$11, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vblendpd	$11, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vblendps	$11, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vblendps	$11, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vblendps	$11, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vblendps	$11, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vblendvpd	%xmm3, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vblendvpd	%xmm3, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vblendvpd	%ymm3, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vblendvpd	%ymm3, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vblendvps	%xmm3, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vblendvps	%xmm3, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vblendvps	%ymm3, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vblendvps	%ymm3, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf128	(%rax), %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastsd	(%rax), %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastss	(%rax), %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastss	(%rax), %ymm2
+# CHECK-NEXT:  1      2     1.00                        vcmpeqpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vcmpeqpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vcmpeqpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      9     1.00    *                   vcmpeqpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      2     1.00                        vcmpeqps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vcmpeqps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vcmpeqps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      9     1.00    *                   vcmpeqps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      2     1.00                        vcmpeqsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vcmpeqsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vcmpeqss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   vcmpeqss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      4     1.00                        vcomisd	%xmm0, %xmm1
+# CHECK-NEXT:  2      11    1.00    *                   vcomisd	(%rax), %xmm1
+# CHECK-NEXT:  2      4     1.00                        vcomiss	%xmm0, %xmm1
+# CHECK-NEXT:  2      11    1.00    *                   vcomiss	(%rax), %xmm1
+# CHECK-NEXT:  1      3     0.50                        vcvtdq2pd	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2pd	(%rax), %xmm2
+# CHECK-NEXT:  2      3     1.00                        vcvtdq2pd	%xmm0, %ymm2
+# CHECK-NEXT:  1      10    1.00    *                   vcvtdq2pd	(%rax), %ymm2
+# CHECK-NEXT:  1      3     0.50                        vcvtdq2ps	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2ps	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        vcvtdq2ps	%ymm0, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2ps	(%rax), %ymm2
+# CHECK-NEXT:  1      3     1.00                        vcvtpd2dq	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2dqx	(%rax), %xmm2
+# CHECK-NEXT:  2      3     1.00                        vcvtpd2dq	%ymm0, %xmm2
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2dqy	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        vcvtpd2ps	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vcvtpd2psx	(%rax), %xmm2
+# CHECK-NEXT:  2      6     1.00                        vcvtpd2ps	%ymm0, %xmm2
+# CHECK-NEXT:  2      13    1.00    *                   vcvtpd2psy	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        vcvtps2dq	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2dq	(%rax), %xmm2
+# CHECK-NEXT:  1      4     0.50                        vcvtps2dq	%ymm0, %ymm2
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2dq	(%rax), %ymm2
+# CHECK-NEXT:  1      3     0.50                        vcvtps2pd	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2pd	(%rax), %xmm2
+# CHECK-NEXT:  2      4     1.00                        vcvtps2pd	%xmm0, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vcvtps2pd	(%rax), %ymm2
+# CHECK-NEXT:  1      1     0.50                        vcvtsd2si	%xmm0, %ecx
+# CHECK-NEXT:  1      1     0.50                        vcvtsd2si	%xmm0, %rcx
+# CHECK-NEXT:  1      8     0.50    *                   vcvtsd2si	(%rax), %ecx
+# CHECK-NEXT:  1      8     0.50    *                   vcvtsd2si	(%rax), %rcx
+# CHECK-NEXT:  1      3     0.50                        vcvtsd2ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vcvtsd2ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      5     1.00                        vcvtsi2sd	%ecx, %xmm0, %xmm2
+# CHECK-NEXT:  2      5     1.00                        vcvtsi2sd	%rcx, %xmm0, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vcvtsi2sdl	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vcvtsi2sdq	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  2      4     1.00                        vcvtsi2ss	%ecx, %xmm0, %xmm2
+# CHECK-NEXT:  2      4     1.00                        vcvtsi2ss	%rcx, %xmm0, %xmm2
+# CHECK-NEXT:  1      10    1.00    *                   vcvtsi2ssl	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  1      10    1.00    *                   vcvtsi2ssq	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vcvtss2sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vcvtss2sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      5     2.50                        vcvtss2si	%xmm0, %ecx
+# CHECK-NEXT:  2      5     2.50                        vcvtss2si	%xmm0, %rcx
+# CHECK-NEXT:  2      12    2.50    *                   vcvtss2si	(%rax), %ecx
+# CHECK-NEXT:  2      12    2.50    *                   vcvtss2si	(%rax), %rcx
+# CHECK-NEXT:  1      3     1.00                        vcvttpd2dq	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2dqx	(%rax), %xmm2
+# CHECK-NEXT:  2      3     1.00                        vcvttpd2dq	%ymm0, %xmm2
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2dqy	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        vcvttps2dq	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2dq	(%rax), %xmm2
+# CHECK-NEXT:  1      4     0.50                        vcvttps2dq	%ymm0, %ymm2
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2dq	(%rax), %ymm2
+# CHECK-NEXT:  2      5     2.50                        vcvttsd2si	%xmm0, %ecx
+# CHECK-NEXT:  2      5     2.50                        vcvttsd2si	%xmm0, %rcx
+# CHECK-NEXT:  2      12    2.50    *                   vcvttsd2si	(%rax), %ecx
+# CHECK-NEXT:  2      12    2.50    *                   vcvttsd2si	(%rax), %rcx
+# CHECK-NEXT:  2      5     2.50                        vcvttss2si	%xmm0, %ecx
+# CHECK-NEXT:  2      5     2.50                        vcvttss2si	%xmm0, %rcx
+# CHECK-NEXT:  2      12    2.50    *                   vcvttss2si	(%rax), %ecx
+# CHECK-NEXT:  2      12    2.50    *                   vcvttss2si	(%rax), %rcx
+# CHECK-NEXT:  1      13    5.00                        vdivpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      20    5.00    *                   vdivpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      13    5.00                        vdivpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      20    5.00    *                   vdivpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      11    3.00                        vdivps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      18    3.00    *                   vdivps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      11    3.00                        vdivps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      18    3.00    *                   vdivps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      13    5.00                        vdivsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      20    5.00    *                   vdivsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      11    3.00                        vdivss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      18    3.00    *                   vdivss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  3      7     3.00                        vdppd	$22, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  5      14    3.00    *                   vdppd	$22, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  8      11    4.00                        vdpps	$22, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  10     18    4.00    *                   vdpps	$22, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  7      11    4.00                        vdpps	$22, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  8      18    4.00    *                   vdpps	$22, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vextractf128	$1, %ymm0, %xmm2
+# CHECK-NEXT:  2      8     1.00           *            vextractf128	$1, %ymm0, (%rax)
+# CHECK-NEXT:  2      1     1.00                        vextractps	$1, %xmm0, %ecx
+# CHECK-NEXT:  2      2     1.00           *            vextractps	$1, %xmm0, (%rax)
+# CHECK-NEXT:  3      4     2.00                        vhaddpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  3      11    2.00    *                   vhaddpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  3      4     2.00                        vhaddpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  4      11    2.00    *                   vhaddpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  3      4     2.00                        vhaddps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  3      11    2.00    *                   vhaddps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  3      4     2.00                        vhaddps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  4      11    2.00    *                   vhaddps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  3      4     2.00                        vhsubpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  3      11    2.00    *                   vhsubpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  3      4     2.00                        vhsubpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  4      11    2.00    *                   vhsubpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  3      4     2.00                        vhsubps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  3      11    2.00    *                   vhsubps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  3      4     2.00                        vhsubps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  4      11    2.00    *                   vhsubps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      2     1.00                        vinsertf128	$1, %xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     1.00    *                   vinsertf128	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vinsertps	$1, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vinsertps	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vlddqu	(%rax), %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vlddqu	(%rax), %ymm2
+# CHECK-NEXT:  1      5     1.50    *      *      U     vldmxcsr	(%rax)
+# CHECK-NEXT:  1      1     1.00    *      *      U     vmaskmovdqu	%xmm0, %xmm1
+# CHECK-NEXT:  1      8     0.50    *                   vmaskmovpd	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vmaskmovpd	(%rax), %ymm0, %ymm2
+# CHECK-NEXT:  10     1     4.00    *      *            vmaskmovpd	%xmm0, %xmm1, (%rax)
+# CHECK-NEXT:  18     1     6.00    *      *            vmaskmovpd	%ymm0, %ymm1, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmaskmovps	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vmaskmovps	(%rax), %ymm0, %ymm2
+# CHECK-NEXT:  18     1     6.00    *      *            vmaskmovps	%xmm0, %xmm1, (%rax)
+# CHECK-NEXT:  42     1     12.00   *      *            vmaskmovps	%ymm0, %ymm1, (%rax)
+# CHECK-NEXT:  1      1     0.50                        vmaxpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vmaxpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vmaxpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      9     1.00    *                   vmaxpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vmaxps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vmaxps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vmaxps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      9     1.00    *                   vmaxps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      2     1.00                        vmaxsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vmaxsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vmaxss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   vmaxss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vminpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vminpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vminpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      9     1.00    *                   vminpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vminps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vminps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vminps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      9     1.00    *                   vminps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      2     1.00                        vminsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vminsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vminss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   vminss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      0     0.17                        vmovapd	%xmm0, %xmm2
+# CHECK-NEXT:  1      1     1.00           *            vmovapd	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovapd	(%rax), %xmm2
+# CHECK-NEXT:  1      0     0.17                        vmovapd	%ymm0, %ymm2
+# CHECK-NEXT:  1      1     1.00           *            vmovapd	%ymm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovapd	(%rax), %ymm2
+# CHECK-NEXT:  1      0     0.17                        vmovaps	%xmm0, %xmm2
+# CHECK-NEXT:  1      1     1.00           *            vmovaps	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovaps	(%rax), %xmm2
+# CHECK-NEXT:  1      0     0.17                        vmovaps	%ymm0, %ymm2
+# CHECK-NEXT:  1      1     1.00           *            vmovaps	%ymm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovaps	(%rax), %ymm2
+# CHECK-NEXT:  1      1     1.00                        vmovd	%eax, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vmovd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     1.00                        vmovd	%xmm0, %ecx
+# CHECK-NEXT:  1      1     1.00           *            vmovd	%xmm0, (%rax)
+# CHECK-NEXT:  1      1     0.50                        vmovddup	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vmovddup	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        vmovddup	%ymm0, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vmovddup	(%rax), %ymm2
+# CHECK-NEXT:  1      0     0.17                        vmovdqa	%xmm0, %xmm2
+# CHECK-NEXT:  1      1     1.00           *            vmovdqa	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa	(%rax), %xmm2
+# CHECK-NEXT:  1      0     0.17                        vmovdqa	%ymm0, %ymm2
+# CHECK-NEXT:  1      1     1.00           *            vmovdqa	%ymm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa	(%rax), %ymm2
+# CHECK-NEXT:  1      0     0.17                        vmovdqu	%xmm0, %xmm2
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu	(%rax), %xmm2
+# CHECK-NEXT:  1      0     0.17                        vmovdqu	%ymm0, %ymm2
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu	%ymm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu	(%rax), %ymm2
+# CHECK-NEXT:  1      1     0.50                        vmovhlps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vmovlhps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  2      2     1.00           *            vmovhpd	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovhpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      2     1.00           *            vmovhps	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovhps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     1.00           *            vmovlpd	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovlpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     1.00           *            vmovlps	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovlps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     1.00                        vmovmskpd	%xmm0, %ecx
+# CHECK-NEXT:  1      1     1.00                        vmovmskpd	%ymm0, %ecx
+# CHECK-NEXT:  1      1     1.00                        vmovmskps	%xmm0, %ecx
+# CHECK-NEXT:  1      1     1.00                        vmovmskps	%ymm0, %ecx
+# CHECK-NEXT:  1      1     1.00           *            vmovntdq	%xmm0, (%rax)
+# CHECK-NEXT:  1      1     1.00           *            vmovntdq	%ymm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovntdqa	(%rax), %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vmovntdqa	(%rax), %ymm2
+# CHECK-NEXT:  1      1     1.00           *            vmovntpd	%xmm0, (%rax)
+# CHECK-NEXT:  1      1     1.00           *            vmovntpd	%ymm0, (%rax)
+# CHECK-NEXT:  1      1     1.00           *            vmovntps	%xmm0, (%rax)
+# CHECK-NEXT:  1      1     1.00           *            vmovntps	%ymm0, (%rax)
+# CHECK-NEXT:  1      1     0.25                        vmovq	%xmm0, %xmm2
+# CHECK-NEXT:  1      1     1.00                        vmovq	%rax, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vmovq	(%rax), %xmm2
+# CHECK-NEXT:  1      1     1.00                        vmovq	%xmm0, %rcx
+# CHECK-NEXT:  1      1     1.00           *            vmovq	%xmm0, (%rax)
+# CHECK-NEXT:  1      1     0.50                        vmovsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      1     1.00           *            vmovsd	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovsd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        vmovshdup	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vmovshdup	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        vmovshdup	%ymm0, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vmovshdup	(%rax), %ymm2
+# CHECK-NEXT:  1      1     0.50                        vmovsldup	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vmovsldup	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        vmovsldup	%ymm0, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vmovsldup	(%rax), %ymm2
+# CHECK-NEXT:  1      1     0.50                        vmovss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      1     1.00           *            vmovss	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovss	(%rax), %xmm2
+# CHECK-NEXT:  1      0     0.17                        vmovupd	%xmm0, %xmm2
+# CHECK-NEXT:  1      1     1.00           *            vmovupd	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovupd	(%rax), %xmm2
+# CHECK-NEXT:  1      0     0.17                        vmovupd	%ymm0, %ymm2
+# CHECK-NEXT:  1      1     1.00           *            vmovupd	%ymm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovupd	(%rax), %ymm2
+# CHECK-NEXT:  1      0     0.17                        vmovups	%xmm0, %xmm2
+# CHECK-NEXT:  1      1     1.00           *            vmovups	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovups	(%rax), %xmm2
+# CHECK-NEXT:  1      0     0.17                        vmovups	%ymm0, %ymm2
+# CHECK-NEXT:  1      1     1.00           *            vmovups	%ymm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vmovups	(%rax), %ymm2
+# CHECK-NEXT:  4      4     2.00                        vmpsadbw	$1, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  6      11    2.00    *                   vmpsadbw	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vmulpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vmulpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vmulpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vmulpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vmulps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vmulps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vmulps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vmulps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vmulsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vmulsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vmulss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vmulss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vorpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vorpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vorps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vorps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpabsb	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpabsb	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpabsd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpabsd	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpabsw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpabsw	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpackssdw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpackssdw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpacksswb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpacksswb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpackusdw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpackusdw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpackuswb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpackuswb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpaddb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpaddb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpaddd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpaddq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpaddsb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpaddsw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpaddusb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpaddusw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpaddw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpaddw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpalignr	$1, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpalignr	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpand	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpand	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpandn	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpandn	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpavgb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpavgb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpavgw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpavgw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpblendvb	%xmm3, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpblendvb	%xmm3, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpblendw	$11, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpblendw	$11, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  4      4     2.00                        vpclmulqdq	$11, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  4      11    2.00    *                   vpclmulqdq	$11, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpcmpeqq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  8      6     3.00                        vpcmpestri	$1, %xmm0, %xmm2
+# CHECK-NEXT:  12     13    3.00    *                   vpcmpestri	$1, (%rax), %xmm2
+# CHECK-NEXT:  7      6     3.00                        vpcmpestrm	$1, %xmm0, %xmm2
+# CHECK-NEXT:  12     13    3.00    *                   vpcmpestrm	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  4      2     2.00                        vpcmpistri	$1, %xmm0, %xmm2
+# CHECK-NEXT:  4      9     2.00    *                   vpcmpistri	$1, (%rax), %xmm2
+# CHECK-NEXT:  3      6     2.00                        vpcmpistrm	$1, %xmm0, %xmm2
+# CHECK-NEXT:  4      13    2.00    *                   vpcmpistrm	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      3     1.00                        vperm2f128	$1, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      7     1.00    *                   vperm2f128	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpermilpd	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpermilpd	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        vpermilpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vpermilpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpermilpd	$1, %ymm0, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpermilpd	$1, (%rax), %ymm2
+# CHECK-NEXT:  1      2     0.67                        vpermilpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vpermilpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpermilps	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpermilps	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        vpermilps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vpermilps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpermilps	$1, %ymm0, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpermilps	$1, (%rax), %ymm2
+# CHECK-NEXT:  1      2     0.67                        vpermilps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vpermilps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      1     1.00                        vpextrb	$1, %xmm0, %ecx
+# CHECK-NEXT:  2      2     1.00           *            vpextrb	$1, %xmm0, (%rax)
+# CHECK-NEXT:  2      1     1.00                        vpextrd	$1, %xmm0, %ecx
+# CHECK-NEXT:  2      2     1.00           *            vpextrd	$1, %xmm0, (%rax)
+# CHECK-NEXT:  2      1     1.00                        vpextrq	$1, %xmm0, %rcx
+# CHECK-NEXT:  2      2     1.00           *            vpextrq	$1, %xmm0, (%rax)
+# CHECK-NEXT:  2      1     1.00                        vpextrw	$1, %xmm0, %ecx
+# CHECK-NEXT:  2      2     1.00           *            vpextrw	$1, %xmm0, (%rax)
+# CHECK-NEXT:  3      2     2.00                        vphaddd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  3      9     2.00    *                   vphaddd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  3      2     2.00                        vphaddsw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  3      9     2.00    *                   vphaddsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  3      2     2.00                        vphaddw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  3      9     2.00    *                   vphaddw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vphminposuw	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vphminposuw	(%rax), %xmm2
+# CHECK-NEXT:  3      2     2.00                        vphsubd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  3      9     2.00    *                   vphsubd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  3      2     2.00                        vphsubsw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  3      9     2.00    *                   vphsubsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  3      2     2.00                        vphsubw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  3      9     2.00    *                   vphsubw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      2     1.00                        vpinsrb	$1, %eax, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     1.50    *                   vpinsrb	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      2     1.00                        vpinsrd	$1, %eax, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     1.50    *                   vpinsrd	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      2     1.00                        vpinsrq	$1, %rax, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     1.50    *                   vpinsrq	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      2     1.00                        vpinsrw	$1, %eax, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     1.50    *                   vpinsrw	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vpmaddubsw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vpmaddubsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vpmaddwd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vpmaddwd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpmaxsb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpmaxsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpmaxsw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpmaxub	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxub	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpmaxud	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxud	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpmaxuw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxuw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpminsb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpminsb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpminsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpminsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpminsw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpminsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpminub	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpminub	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpminud	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpminud	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpminuw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpminuw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     1.00                        vpmovmskb	%xmm0, %ecx
+# CHECK-NEXT:  1      4     2.00                        vpmovsxbd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxbd	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        vpmovsxbq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxbq	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        vpmovsxbw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxbw	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        vpmovsxdq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxdq	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        vpmovsxwd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxwd	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        vpmovsxwq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxwq	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        vpmovzxbd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxbd	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        vpmovzxbq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxbq	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        vpmovzxbw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxbw	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        vpmovzxdq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxdq	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        vpmovzxwd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxwd	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        vpmovzxwq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxwq	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        vpmuldq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vpmuldq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vpmulhrsw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhrsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vpmulhuw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhuw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vpmulhw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vpmulld	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vpmulld	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vpmullw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vpmullw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vpmuludq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vpmuludq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpor	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpor	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vpsadbw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vpsadbw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpshufb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpshufb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpshufd	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpshufd	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpshufhw	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpshufhw	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpshuflw	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpshuflw	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpsignb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsignb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpsignd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsignd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpsignw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsignw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpslld	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpslld	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   vpslld	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpslldq	$1, %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpsllq	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpsllq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   vpsllq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpsllw	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpsllw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   vpsllw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpsrad	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpsrad	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   vpsrad	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpsraw	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpsraw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   vpsraw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpsrld	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpsrld	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   vpsrld	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpsrldq	$1, %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpsrlq	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpsrlq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   vpsrlq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpsrlw	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpsrlw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   vpsrlw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpsubb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsubb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpsubd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpsubq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpsubsb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpsubsw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpsubusb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        vpsubusw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpsubw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsubw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      1     1.00                        vptest	%xmm0, %xmm1
+# CHECK-NEXT:  2      8     1.00    *                   vptest	(%rax), %xmm1
+# CHECK-NEXT:  2      1     1.00                        vptest	%ymm0, %ymm1
+# CHECK-NEXT:  2      8     1.00    *                   vptest	(%rax), %ymm1
+# CHECK-NEXT:  1      1     0.50                        vpunpckhbw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhbw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpunpckhdq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhdq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpunpckhqdq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhqdq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpunpckhwd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhwd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpunpcklbw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpunpcklbw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpunpckldq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckldq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpunpcklqdq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpunpcklqdq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpunpcklwd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpunpcklwd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpxor	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpxor	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     0.50                        vrcpps	%xmm0, %xmm2
+# CHECK-NEXT:  1      11    0.50    *                   vrcpps	(%rax), %xmm2
+# CHECK-NEXT:  1      5     0.50                        vrcpps	%ymm0, %ymm2
+# CHECK-NEXT:  1      12    0.50    *                   vrcpps	(%rax), %ymm2
+# CHECK-NEXT:  1      4     0.50                        vrcpss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    0.50    *                   vrcpss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vroundpd	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vroundpd	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        vroundpd	$1, %ymm0, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vroundpd	$1, (%rax), %ymm2
+# CHECK-NEXT:  1      3     0.50                        vroundps	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vroundps	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        vroundps	$1, %ymm0, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vroundps	$1, (%rax), %ymm2
+# CHECK-NEXT:  1      3     0.50                        vroundsd	$1, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vroundsd	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vroundss	$1, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vroundss	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vrsqrtps	%xmm0, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vrsqrtps	(%rax), %xmm2
+# CHECK-NEXT:  1      4     1.00                        vrsqrtps	%ymm0, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vrsqrtps	(%rax), %ymm2
+# CHECK-NEXT:  1      4     1.00                        vrsqrtss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vrsqrtss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vshufpd	$1, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vshufpd	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vshufpd	$1, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vshufpd	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vshufps	$1, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vshufps	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vshufps	$1, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vshufps	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      21    9.00                        vsqrtpd	%xmm0, %xmm2
+# CHECK-NEXT:  1      28    9.00    *                   vsqrtpd	(%rax), %xmm2
+# CHECK-NEXT:  1      21    9.00                        vsqrtpd	%ymm0, %ymm2
+# CHECK-NEXT:  1      28    9.00    *                   vsqrtpd	(%rax), %ymm2
+# CHECK-NEXT:  1      15    5.00                        vsqrtps	%xmm0, %xmm2
+# CHECK-NEXT:  1      22    5.00    *                   vsqrtps	(%rax), %xmm2
+# CHECK-NEXT:  1      15    5.00                        vsqrtps	%ymm0, %ymm2
+# CHECK-NEXT:  1      22    5.00    *                   vsqrtps	(%rax), %ymm2
+# CHECK-NEXT:  1      21    9.00                        vsqrtsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      28    9.00    *                   vsqrtsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      15    5.00                        vsqrtss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      22    5.00    *                   vsqrtss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      2     15.00          *      U     vstmxcsr	(%rax)
+# CHECK-NEXT:  1      3     0.50                        vsubpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vsubpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vsubpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vsubpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vsubps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vsubps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vsubps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vsubps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vsubsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vsubsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vsubss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vsubss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  2      1     1.00                        vtestpd	%xmm0, %xmm1
+# CHECK-NEXT:  2      8     1.00    *                   vtestpd	(%rax), %xmm1
+# CHECK-NEXT:  2      1     1.00                        vtestpd	%ymm0, %ymm1
+# CHECK-NEXT:  2      8     1.00    *                   vtestpd	(%rax), %ymm1
+# CHECK-NEXT:  2      1     1.00                        vtestps	%xmm0, %xmm1
+# CHECK-NEXT:  2      8     1.00    *                   vtestps	(%rax), %xmm1
+# CHECK-NEXT:  2      1     1.00                        vtestps	%ymm0, %ymm1
+# CHECK-NEXT:  2      8     1.00    *                   vtestps	(%rax), %ymm1
+# CHECK-NEXT:  2      4     1.00                        vucomisd	%xmm0, %xmm1
+# CHECK-NEXT:  2      11    1.00    *                   vucomisd	(%rax), %xmm1
+# CHECK-NEXT:  2      4     1.00                        vucomiss	%xmm0, %xmm1
+# CHECK-NEXT:  2      11    1.00    *                   vucomiss	(%rax), %xmm1
+# CHECK-NEXT:  1      1     0.50                        vunpckhpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vunpckhpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vunpckhps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vunpckhps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vunpcklpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vunpcklpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vunpcklps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vunpcklps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vxorpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vxorpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vxorps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vxorps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  18     10    6.00                  U     vzeroall
+# CHECK-NEXT:  1      0     0.25                  U     vzeroupper
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 1.33   1.33   1.33   16.50  16.50  16.50  16.50   -     211.25 420.08 288.58 158.08 208.50 208.50 65.00  119.67 119.67 119.67 107.00 107.00 107.00 19.00  19.00
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddsubpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddsubpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddsubpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddsubpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddsubps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddsubps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddsubps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddsubps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesdec	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesdec	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesdeclast	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesdeclast	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesenc	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesenc	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesenclast	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesenclast	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesimc	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesimc	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaeskeygenassist	$22, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaeskeygenassist	$22, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandnpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandnpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandnps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandnps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vblendpd	$11, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vblendpd	$11, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vblendpd	$11, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vblendpd	$11, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vblendps	$11, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vblendps	$11, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vblendps	$11, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vblendps	$11, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vblendvpd	%xmm3, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vblendvpd	%xmm3, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vblendvpd	%ymm3, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vblendvpd	%ymm3, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vblendvps	%xmm3, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vblendvps	%xmm3, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vblendvps	%ymm3, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vblendvps	%ymm3, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf128	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastsd	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastss	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastss	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcomisd	%xmm0, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcomisd	(%rax), %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcomiss	%xmm0, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcomiss	(%rax), %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2pd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2pd	%xmm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2ps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2ps	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2dq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dqx	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2dq	%ymm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dqy	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2ps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2psx	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2ps	%ymm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2psy	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2dq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2dq	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2pd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2pd	%xmm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtsd2si	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtsd2si	%xmm0, %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtsd2si	(%rax), %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtsd2si	(%rax), %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtsd2ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtsd2ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtsi2sd	%ecx, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtsi2sd	%rcx, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtsi2sdl	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtsi2sdq	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtsi2ss	%ecx, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtsi2ss	%rcx, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtsi2ssl	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtsi2ssq	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtss2sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtss2sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -     vcvtss2si	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -     vcvtss2si	%xmm0, %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtss2si	(%rax), %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtss2si	(%rax), %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2dq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dqx	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2dq	%ymm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dqy	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2dq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2dq	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -     vcvttsd2si	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -     vcvttsd2si	%xmm0, %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttsd2si	(%rax), %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttsd2si	(%rax), %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -     vcvttss2si	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -     vcvttss2si	%xmm0, %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttss2si	(%rax), %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttss2si	(%rax), %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdppd	$22, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00   3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdppd	$22, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     4.00   4.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdpps	$22, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     4.00   4.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdpps	$22, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     4.00   4.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdpps	$22, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     4.00   4.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdpps	$22, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vextractf128	$1, %ymm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vextractf128	$1, %ymm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vextractps	$1, %xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vextractps	$1, %xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -     vhaddpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vhaddpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -     vhaddpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vhaddpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -     vhaddps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vhaddps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -     vhaddps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vhaddps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -     vhsubpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vhsubpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -     vhsubpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vhsubpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -     vhsubps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vhsubps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -     vhsubps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vhsubps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinsertf128	$1, %xmm0, %ymm1, %ymm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -     1.00    -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinsertf128	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vinsertps	$1, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinsertps	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vlddqu	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vlddqu	(%rax), %ymm2
+# CHECK-NEXT: 0.33   0.33   0.33   1.50   1.50   1.50   1.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vldmxcsr	(%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmaskmovdqu	%xmm0, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaskmovpd	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaskmovpd	(%rax), %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     2.00   2.00   4.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmaskmovpd	%xmm0, %xmm1, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     3.00   3.00   6.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmaskmovpd	%ymm0, %ymm1, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaskmovps	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaskmovps	(%rax), %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     3.00   3.00   6.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmaskmovps	%xmm0, %xmm1, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     6.00   6.00   12.00  0.33   0.33   0.33    -      -      -     0.50   0.50   vmaskmovps	%ymm0, %ymm1, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vminpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vminps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovapd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovapd	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovapd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovapd	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovapd	%ymm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovapd	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovaps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovaps	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovaps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovaps	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovaps	%ymm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovaps	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vmovd	%eax, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vmovd	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovd	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovddup	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovddup	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovddup	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovddup	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqa	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqa	%ymm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu	%ymm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovhlps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovlhps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovhpd	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovhpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovhps	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovhps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovlpd	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovlpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovlps	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovlps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovmskpd	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovmskpd	%ymm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovmskps	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovmskps	%ymm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovntdq	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovntdq	%ymm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovntdqa	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovntdqa	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovntpd	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovntpd	%ymm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovntps	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovntps	%ymm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vmovq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vmovq	%rax, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vmovq	%xmm0, %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovq	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovsd	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovsd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovshdup	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovshdup	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovshdup	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovshdup	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovsldup	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovsldup	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovsldup	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovsldup	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovss	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovss	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovupd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovupd	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovupd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovupd	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovupd	%ymm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovupd	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovups	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovups	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovups	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovups	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovups	%ymm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovups	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00   2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vmpsadbw	$1, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00   2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmpsadbw	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vorpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vorpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vorps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vorps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpabsb	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsb	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpabsd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpabsw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackssdw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackssdw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpacksswb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpacksswb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackusdw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackusdw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackuswb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackuswb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddsb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddsw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddusb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddusw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpalignr	$1, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpalignr	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpand	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpand	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpandn	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpandn	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpavgb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpavgw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpblendvb	%xmm3, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendvb	%xmm3, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpblendw	$11, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendw	$11, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpclmulqdq	$11, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpclmulqdq	$11, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpcmpeqq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00   3.00   3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vpcmpestri	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00   3.00   3.00   3.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpestri	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00   3.00   3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vpcmpestrm	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00   3.00   3.00   3.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpestrm	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00   2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vpcmpistri	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00   2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpistri	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00   2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vpcmpistrm	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00   2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpistrm	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vperm2f128	$1, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -     1.00    -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vperm2f128	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermilpd	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermilpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermilpd	$1, %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$1, (%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermilps	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermilps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermilps	$1, %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$1, (%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vpextrb	$1, %xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vpextrb	$1, %xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vpextrd	$1, %xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vpextrd	$1, %xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vpextrq	$1, %xmm0, %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vpextrq	$1, %xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vpextrw	$1, %xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vpextrw	$1, %xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vphaddd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vphaddd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vphaddsw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vphaddsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vphaddw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vphaddw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vphminposuw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vphminposuw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vphsubd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vphsubd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vphsubsw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vphsubsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vphsubw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vphsubw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vpinsrb	$1, %eax, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpinsrb	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vpinsrd	$1, %eax, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpinsrd	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vpinsrq	$1, %rax, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpinsrq	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vpinsrw	$1, %eax, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpinsrw	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaddubsw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddubsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaddwd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddwd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxsb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxsw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxub	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxub	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxud	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxud	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxuw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxuw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminsb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminsw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminub	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminub	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminud	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminud	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminuw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminuw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovmskb	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxdq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxdq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxdq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxdq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmuldq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmuldq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhrsw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhrsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhuw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhuw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulld	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmullw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmuludq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmuludq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpor	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpor	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsadbw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsadbw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshufb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshufd	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshufhw	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufhw	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshuflw	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshuflw	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsignb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsignb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsignd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsignd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsignw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsignw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpslld	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpslld	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpslld	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpslldq	$1, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsllq	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrad	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrad	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrad	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrld	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrld	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrld	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrldq	$1, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrlq	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubsb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubsw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubusb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubusw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50   1.00    -      -      -      -      -      -      -      -     vptest	%xmm0, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     1.00   1.00   1.00   0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptest	(%rax), %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50   1.00    -      -      -      -      -      -      -      -     vptest	%ymm0, %ymm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     1.00   1.00   1.00   0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptest	(%rax), %ymm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhbw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhbw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhdq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhqdq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhqdq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhwd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhwd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklbw	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklbw	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckldq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklqdq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklqdq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklwd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklwd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpxor	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpxor	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vrcpps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrcpps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vrcpps	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrcpps	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vrcpss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrcpss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vroundpd	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vroundpd	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vroundpd	$1, %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vroundpd	$1, (%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vroundps	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vroundps	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vroundps	$1, %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vroundps	$1, (%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vroundsd	$1, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vroundsd	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vroundss	$1, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vroundss	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrsqrtps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrsqrtps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrsqrtps	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrsqrtps	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrsqrtss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrsqrtss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vshufpd	$1, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufpd	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vshufpd	$1, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufpd	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vshufps	$1, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufps	$1, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vshufps	$1, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufps	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtpd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtpd	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtps	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT: 0.33   0.33   0.33   15.00  15.00  15.00  15.00   -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   vstmxcsr	(%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vsubpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vsubpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vsubps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vsubps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vsubsd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubsd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vsubss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vtestpd	%xmm0, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vtestpd	(%rax), %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vtestpd	%ymm0, %ymm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vtestpd	(%rax), %ymm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vtestps	%xmm0, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vtestps	(%rax), %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vtestps	%ymm0, %ymm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vtestps	(%rax), %ymm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vucomisd	%xmm0, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vucomisd	(%rax), %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vucomiss	%xmm0, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vucomiss	(%rax), %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vxorpd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vxorpd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vxorps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vxorps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     6.00   6.00   6.00   6.00    -      -      -      -      -      -      -      -      -      -      -     vzeroall
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vzeroupper
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx2.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx2.s
new file mode 100644
index 0000000000000..95895352f26f6
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx2.s
@@ -0,0 +1,1096 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vbroadcasti128  (%rax), %ymm0
+
+vbroadcastsd    %xmm0, %ymm0
+vbroadcastss    %xmm0, %ymm0
+
+vextracti128    $1, %ymm0, %xmm2
+vextracti128    $1, %ymm0, (%rax)
+
+vgatherdpd      %xmm0, (%rax,%xmm1,2), %xmm2
+vgatherdpd      %ymm0, (%rax,%xmm1,2), %ymm2
+
+vgatherdps      %xmm0, (%rax,%xmm1,2), %xmm2
+vgatherdps      %ymm0, (%rax,%ymm1,2), %ymm2
+
+vgatherqpd      %xmm0, (%rax,%xmm1,2), %xmm2
+vgatherqpd      %ymm0, (%rax,%ymm1,2), %ymm2
+
+vgatherqps      %xmm0, (%rax,%xmm1,2), %xmm2
+vgatherqps      %xmm0, (%rax,%ymm1,2), %xmm2
+
+vinserti128     $1, %xmm0, %ymm1, %ymm2
+vinserti128     $1, (%rax), %ymm1, %ymm2
+
+vmovntdqa       (%rax), %ymm0
+
+vmpsadbw        $1, %ymm0, %ymm1, %ymm2
+vmpsadbw        $1, (%rax), %ymm1, %ymm2
+
+vpabsb          %ymm0, %ymm2
+vpabsb          (%rax), %ymm2
+
+vpabsd          %ymm0, %ymm2
+vpabsd          (%rax), %ymm2
+
+vpabsw          %ymm0, %ymm2
+vpabsw          (%rax), %ymm2
+
+vpackssdw       %ymm0, %ymm1, %ymm2
+vpackssdw       (%rax), %ymm1, %ymm2
+
+vpacksswb       %ymm0, %ymm1, %ymm2
+vpacksswb       (%rax), %ymm1, %ymm2
+
+vpackusdw       %ymm0, %ymm1, %ymm2
+vpackusdw       (%rax), %ymm1, %ymm2
+
+vpackuswb       %ymm0, %ymm1, %ymm2
+vpackuswb       (%rax), %ymm1, %ymm2
+
+vpaddb          %ymm0, %ymm1, %ymm2
+vpaddb          (%rax), %ymm1, %ymm2
+
+vpaddd          %ymm0, %ymm1, %ymm2
+vpaddd          (%rax), %ymm1, %ymm2
+
+vpaddq          %ymm0, %ymm1, %ymm2
+vpaddq          (%rax), %ymm1, %ymm2
+
+vpaddsb         %ymm0, %ymm1, %ymm2
+vpaddsb         (%rax), %ymm1, %ymm2
+
+vpaddsw         %ymm0, %ymm1, %ymm2
+vpaddsw         (%rax), %ymm1, %ymm2
+
+vpaddusb        %ymm0, %ymm1, %ymm2
+vpaddusb        (%rax), %ymm1, %ymm2
+
+vpaddusw        %ymm0, %ymm1, %ymm2
+vpaddusw        (%rax), %ymm1, %ymm2
+
+vpaddw          %ymm0, %ymm1, %ymm2
+vpaddw          (%rax), %ymm1, %ymm2
+
+vpalignr        $1, %ymm0, %ymm1, %ymm2
+vpalignr        $1, (%rax), %ymm1, %ymm2
+
+vpand           %ymm0, %ymm1, %ymm2
+vpand           (%rax), %ymm1, %ymm2
+
+vpandn          %ymm0, %ymm1, %ymm2
+vpandn          (%rax), %ymm1, %ymm2
+
+vpavgb          %ymm0, %ymm1, %ymm2
+vpavgb          (%rax), %ymm1, %ymm2
+
+vpavgw          %ymm0, %ymm1, %ymm2
+vpavgw          (%rax), %ymm1, %ymm2
+
+vpblendd        $11, %xmm0, %xmm1, %xmm2
+vpblendd        $11, (%rax), %xmm1, %xmm2
+
+vpblendd        $11, %ymm0, %ymm1, %ymm2
+vpblendd        $11, (%rax), %ymm1, %ymm2
+
+vpblendvb       %ymm3, %ymm0, %ymm1, %ymm2
+vpblendvb       %ymm3, (%rax), %ymm1, %ymm2
+
+vpblendw        $11, %ymm0, %ymm1, %ymm2
+vpblendw        $11, (%rax), %ymm1, %ymm2
+
+vpbroadcastb    %xmm0, %xmm0
+vpbroadcastb    (%rax), %xmm0
+
+vpbroadcastb    %xmm0, %ymm0
+vpbroadcastb    (%rax), %ymm0
+
+vpbroadcastd    %xmm0, %xmm0
+vpbroadcastd    (%rax), %xmm0
+
+vpbroadcastd    %xmm0, %ymm0
+vpbroadcastd    (%rax), %ymm0
+
+vpbroadcastq    %xmm0, %xmm0
+vpbroadcastq    (%rax), %xmm0
+
+vpbroadcastq    %xmm0, %ymm0
+vpbroadcastq    (%rax), %ymm0
+
+vpbroadcastw    %xmm0, %xmm0
+vpbroadcastw    (%rax), %xmm0
+
+vpbroadcastw    %xmm0, %ymm0
+vpbroadcastw    (%rax), %ymm0
+
+vpcmpeqb        %ymm0, %ymm1, %ymm2
+vpcmpeqb        (%rax), %ymm1, %ymm2
+
+vpcmpeqd        %ymm0, %ymm1, %ymm2
+vpcmpeqd        (%rax), %ymm1, %ymm2
+
+vpcmpeqq        %ymm0, %ymm1, %ymm2
+vpcmpeqq        (%rax), %ymm1, %ymm2
+
+vpcmpeqw        %ymm0, %ymm1, %ymm2
+vpcmpeqw        (%rax), %ymm1, %ymm2
+
+vpcmpgtb        %ymm0, %ymm1, %ymm2
+vpcmpgtb        (%rax), %ymm1, %ymm2
+
+vpcmpgtd        %ymm0, %ymm1, %ymm2
+vpcmpgtd        (%rax), %ymm1, %ymm2
+
+vpcmpgtq        %ymm0, %ymm1, %ymm2
+vpcmpgtq        (%rax), %ymm1, %ymm2
+
+vpcmpgtw        %ymm0, %ymm1, %ymm2
+vpcmpgtw        (%rax), %ymm1, %ymm2
+
+vperm2i128      $1, %ymm0, %ymm1, %ymm2
+vperm2i128      $1, (%rax), %ymm1, %ymm2
+
+vpermd          %ymm0, %ymm1, %ymm2
+vpermd          (%rax), %ymm1, %ymm2
+
+vpermpd         $1, %ymm0, %ymm2
+vpermpd         $1, (%rax), %ymm2
+
+vpermps         %ymm0, %ymm1, %ymm2
+vpermps         (%rax), %ymm1, %ymm2
+
+vpermq          $1, %ymm0, %ymm2
+vpermq          $1, (%rax), %ymm2
+
+vpgatherdd      %xmm0, (%rax,%xmm1,2), %xmm2
+vpgatherdd      %ymm0, (%rax,%ymm1,2), %ymm2
+
+vpgatherdq      %xmm0, (%rax,%xmm1,2), %xmm2
+vpgatherdq      %ymm0, (%rax,%xmm1,2), %ymm2
+
+vpgatherqd      %xmm0, (%rax,%xmm1,2), %xmm2
+vpgatherqd      %xmm0, (%rax,%ymm1,2), %xmm2
+
+vpgatherqq      %xmm0, (%rax,%xmm1,2), %xmm2
+vpgatherqq      %ymm0, (%rax,%ymm1,2), %ymm2
+
+vphaddd         %ymm0, %ymm1, %ymm2
+vphaddd         (%rax), %ymm1, %ymm2
+
+vphaddsw        %ymm0, %ymm1, %ymm2
+vphaddsw        (%rax), %ymm1, %ymm2
+
+vphaddw         %ymm0, %ymm1, %ymm2
+vphaddw         (%rax), %ymm1, %ymm2
+
+vphsubd         %ymm0, %ymm1, %ymm2
+vphsubd         (%rax), %ymm1, %ymm2
+
+vphsubsw        %ymm0, %ymm1, %ymm2
+vphsubsw        (%rax), %ymm1, %ymm2
+
+vphsubw         %ymm0, %ymm1, %ymm2
+vphsubw         (%rax), %ymm1, %ymm2
+
+vpmaddubsw      %ymm0, %ymm1, %ymm2
+vpmaddubsw      (%rax), %ymm1, %ymm2
+
+vpmaddwd        %ymm0, %ymm1, %ymm2
+vpmaddwd        (%rax), %ymm1, %ymm2
+
+vpmaskmovd      (%rax), %xmm0, %xmm2
+vpmaskmovd      (%rax), %ymm0, %ymm2
+
+vpmaskmovd      %xmm0, %xmm1, (%rax)
+vpmaskmovd      %ymm0, %ymm1, (%rax)
+
+vpmaskmovq      (%rax), %xmm0, %xmm2
+vpmaskmovq      (%rax), %ymm0, %ymm2
+
+vpmaskmovq      %xmm0, %xmm1, (%rax)
+vpmaskmovq      %ymm0, %ymm1, (%rax)
+
+vpmaxsb         %ymm0, %ymm1, %ymm2
+vpmaxsb         (%rax), %ymm1, %ymm2
+
+vpmaxsd         %ymm0, %ymm1, %ymm2
+vpmaxsd         (%rax), %ymm1, %ymm2
+
+vpmaxsw         %ymm0, %ymm1, %ymm2
+vpmaxsw         (%rax), %ymm1, %ymm2
+
+vpmaxub         %ymm0, %ymm1, %ymm2
+vpmaxub         (%rax), %ymm1, %ymm2
+
+vpmaxud         %ymm0, %ymm1, %ymm2
+vpmaxud         (%rax), %ymm1, %ymm2
+
+vpmaxuw         %ymm0, %ymm1, %ymm2
+vpmaxuw         (%rax), %ymm1, %ymm2
+
+vpminsb         %ymm0, %ymm1, %ymm2
+vpminsb         (%rax), %ymm1, %ymm2
+
+vpminsd         %ymm0, %ymm1, %ymm2
+vpminsd         (%rax), %ymm1, %ymm2
+
+vpminsw         %ymm0, %ymm1, %ymm2
+vpminsw         (%rax), %ymm1, %ymm2
+
+vpminub         %ymm0, %ymm1, %ymm2
+vpminub         (%rax), %ymm1, %ymm2
+
+vpminud         %ymm0, %ymm1, %ymm2
+vpminud         (%rax), %ymm1, %ymm2
+
+vpminuw         %ymm0, %ymm1, %ymm2
+vpminuw         (%rax), %ymm1, %ymm2
+
+vpmovmskb       %ymm0, %rcx
+
+vpmovsxbd       %xmm0, %ymm2
+vpmovsxbd       (%rax), %ymm2
+
+vpmovsxbq       %xmm0, %ymm2
+vpmovsxbq       (%rax), %ymm2
+
+vpmovsxbw       %xmm0, %ymm2
+vpmovsxbw       (%rax), %ymm2
+
+vpmovsxdq       %xmm0, %ymm2
+vpmovsxdq       (%rax), %ymm2
+
+vpmovsxwd       %xmm0, %ymm2
+vpmovsxwd       (%rax), %ymm2
+
+vpmovsxwq       %xmm0, %ymm2
+vpmovsxwq       (%rax), %ymm2
+
+vpmovzxbd       %xmm0, %ymm2
+vpmovzxbd       (%rax), %ymm2
+
+vpmovzxbq       %xmm0, %ymm2
+vpmovzxbq       (%rax), %ymm2
+
+vpmovzxbw       %xmm0, %ymm2
+vpmovzxbw       (%rax), %ymm2
+
+vpmovzxdq       %xmm0, %ymm2
+vpmovzxdq       (%rax), %ymm2
+
+vpmovzxwd       %xmm0, %ymm2
+vpmovzxwd       (%rax), %ymm2
+
+vpmovzxwq       %xmm0, %ymm2
+vpmovzxwq       (%rax), %ymm2
+
+vpmuldq         %ymm0, %ymm1, %ymm2
+vpmuldq         (%rax), %ymm1, %ymm2
+
+vpmulhrsw       %ymm0, %ymm1, %ymm2
+vpmulhrsw       (%rax), %ymm1, %ymm2
+
+vpmulhuw        %ymm0, %ymm1, %ymm2
+vpmulhuw        (%rax), %ymm1, %ymm2
+
+vpmulhw         %ymm0, %ymm1, %ymm2
+vpmulhw         (%rax), %ymm1, %ymm2
+
+vpmulld         %ymm0, %ymm1, %ymm2
+vpmulld         (%rax), %ymm1, %ymm2
+
+vpmullw         %ymm0, %ymm1, %ymm2
+vpmullw         (%rax), %ymm1, %ymm2
+
+vpmuludq        %ymm0, %ymm1, %ymm2
+vpmuludq        (%rax), %ymm1, %ymm2
+
+vpor            %ymm0, %ymm1, %ymm2
+vpor            (%rax), %ymm1, %ymm2
+
+vpsadbw         %ymm0, %ymm1, %ymm2
+vpsadbw         (%rax), %ymm1, %ymm2
+
+vpshufb         %ymm0, %ymm1, %ymm2
+vpshufb         (%rax), %ymm1, %ymm2
+
+vpshufd         $1, %ymm0, %ymm2
+vpshufd         $1, (%rax), %ymm2
+
+vpshufhw        $1, %ymm0, %ymm2
+vpshufhw        $1, (%rax), %ymm2
+
+vpshuflw        $1, %ymm0, %ymm2
+vpshuflw        $1, (%rax), %ymm2
+
+vpsignb         %ymm0, %ymm1, %ymm2
+vpsignb         (%rax), %ymm1, %ymm2
+
+vpsignd         %ymm0, %ymm1, %ymm2
+vpsignd         (%rax), %ymm1, %ymm2
+
+vpsignw         %ymm0, %ymm1, %ymm2
+vpsignw         (%rax), %ymm1, %ymm2
+
+vpslld          $1, %ymm0, %ymm2
+vpslld          %xmm0, %ymm1, %ymm2
+vpslld          (%rax), %ymm1, %ymm2
+
+vpslldq         $1, %ymm1, %ymm2
+
+vpsllq          $1, %ymm0, %ymm2
+vpsllq          %xmm0, %ymm1, %ymm2
+vpsllq          (%rax), %ymm1, %ymm2
+
+vpsllvd         %xmm0, %xmm1, %xmm2
+vpsllvd         (%rax), %xmm1, %xmm2
+
+vpsllvd         %ymm0, %ymm1, %ymm2
+vpsllvd         (%rax), %ymm1, %ymm2
+
+vpsllvq         %xmm0, %xmm1, %xmm2
+vpsllvq         (%rax), %xmm1, %xmm2
+
+vpsllvq         %ymm0, %ymm1, %ymm2
+vpsllvq         (%rax), %ymm1, %ymm2
+
+vpsllw          $1, %ymm0, %ymm2
+vpsllw          %xmm0, %ymm1, %ymm2
+vpsllw          (%rax), %ymm1, %ymm2
+
+vpsrad          $1, %ymm0, %ymm2
+vpsrad          %xmm0, %ymm1, %ymm2
+vpsrad          (%rax), %ymm1, %ymm2
+
+vpsravd         %xmm0, %xmm1, %xmm2
+vpsravd         (%rax), %xmm1, %xmm2
+
+vpsravd         %ymm0, %ymm1, %ymm2
+vpsravd         (%rax), %ymm1, %ymm2
+
+vpsraw          $1, %ymm0, %ymm2
+vpsraw          %xmm0, %ymm1, %ymm2
+vpsraw          (%rax), %ymm1, %ymm2
+
+vpsrld          $1, %ymm0, %ymm2
+vpsrld          %xmm0, %ymm1, %ymm2
+vpsrld          (%rax), %ymm1, %ymm2
+
+vpsrldq         $1, %ymm1, %ymm2
+
+vpsrlq          $1, %ymm0, %ymm2
+vpsrlq          %xmm0, %ymm1, %ymm2
+vpsrlq          (%rax), %ymm1, %ymm2
+
+vpsrlvd         %xmm0, %xmm1, %xmm2
+vpsrlvd         (%rax), %xmm1, %xmm2
+
+vpsrlvd         %ymm0, %ymm1, %ymm2
+vpsrlvd         (%rax), %ymm1, %ymm2
+
+vpsrlvq         %xmm0, %xmm1, %xmm2
+vpsrlvq         (%rax), %xmm1, %xmm2
+
+vpsrlvq         %ymm0, %ymm1, %ymm2
+vpsrlvq         (%rax), %ymm1, %ymm2
+
+vpsrlw          $1, %ymm0, %ymm2
+vpsrlw          %xmm0, %ymm1, %ymm2
+vpsrlw          (%rax), %ymm1, %ymm2
+
+vpsubb          %ymm0, %ymm1, %ymm2
+vpsubb          (%rax), %ymm1, %ymm2
+
+vpsubd          %ymm0, %ymm1, %ymm2
+vpsubd          (%rax), %ymm1, %ymm2
+
+vpsubq          %ymm0, %ymm1, %ymm2
+vpsubq          (%rax), %ymm1, %ymm2
+
+vpsubsb         %ymm0, %ymm1, %ymm2
+vpsubsb         (%rax), %ymm1, %ymm2
+
+vpsubsw         %ymm0, %ymm1, %ymm2
+vpsubsw         (%rax), %ymm1, %ymm2
+
+vpsubusb        %ymm0, %ymm1, %ymm2
+vpsubusb        (%rax), %ymm1, %ymm2
+
+vpsubusw        %ymm0, %ymm1, %ymm2
+vpsubusw        (%rax), %ymm1, %ymm2
+
+vpsubw          %ymm0, %ymm1, %ymm2
+vpsubw          (%rax), %ymm1, %ymm2
+
+vpunpckhbw      %ymm0, %ymm1, %ymm2
+vpunpckhbw      (%rax), %ymm1, %ymm2
+
+vpunpckhdq      %ymm0, %ymm1, %ymm2
+vpunpckhdq      (%rax), %ymm1, %ymm2
+
+vpunpckhqdq     %ymm0, %ymm1, %ymm2
+vpunpckhqdq     (%rax), %ymm1, %ymm2
+
+vpunpckhwd      %ymm0, %ymm1, %ymm2
+vpunpckhwd      (%rax), %ymm1, %ymm2
+
+vpunpcklbw      %ymm0, %ymm1, %ymm2
+vpunpcklbw      (%rax), %ymm1, %ymm2
+
+vpunpckldq      %ymm0, %ymm1, %ymm2
+vpunpckldq      (%rax), %ymm1, %ymm2
+
+vpunpcklqdq     %ymm0, %ymm1, %ymm2
+vpunpcklqdq     (%rax), %ymm1, %ymm2
+
+vpunpcklwd      %ymm0, %ymm1, %ymm2
+vpunpcklwd      (%rax), %ymm1, %ymm2
+
+vpxor           %ymm0, %ymm1, %ymm2
+vpxor           (%rax), %ymm1, %ymm2
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti128	(%rax), %ymm0
+# CHECK-NEXT:  1      2     1.00                        vbroadcastsd	%xmm0, %ymm0
+# CHECK-NEXT:  1      2     1.00                        vbroadcastss	%xmm0, %ymm0
+# CHECK-NEXT:  1      4     1.00                        vextracti128	$1, %ymm0, %xmm2
+# CHECK-NEXT:  2      8     1.00           *            vextracti128	$1, %ymm0, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   vgatherdpd	%xmm0, (%rax,%xmm1,2), %xmm2
+# CHECK-NEXT:  1      5     0.33    *                   vgatherdpd	%ymm0, (%rax,%xmm1,2), %ymm2
+# CHECK-NEXT:  1      5     0.33    *                   vgatherdps	%xmm0, (%rax,%xmm1,2), %xmm2
+# CHECK-NEXT:  1      5     0.33    *                   vgatherdps	%ymm0, (%rax,%ymm1,2), %ymm2
+# CHECK-NEXT:  1      5     0.33    *                   vgatherqpd	%xmm0, (%rax,%xmm1,2), %xmm2
+# CHECK-NEXT:  1      5     0.33    *                   vgatherqpd	%ymm0, (%rax,%ymm1,2), %ymm2
+# CHECK-NEXT:  1      5     0.33    *                   vgatherqps	%xmm0, (%rax,%xmm1,2), %xmm2
+# CHECK-NEXT:  1      5     0.33    *                   vgatherqps	%xmm0, (%rax,%ymm1,2), %xmm2
+# CHECK-NEXT:  1      1     1.00                        vinserti128	$1, %xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     1.00    *                   vinserti128	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vmovntdqa	(%rax), %ymm0
+# CHECK-NEXT:  3      4     2.00                        vmpsadbw	$1, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  4      11    2.00    *                   vmpsadbw	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpabsb	%ymm0, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpabsb	(%rax), %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpabsd	%ymm0, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpabsd	(%rax), %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpabsw	%ymm0, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpabsw	(%rax), %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpackssdw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpackssdw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpacksswb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpacksswb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpackusdw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpackusdw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpackuswb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpackuswb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpaddb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpaddb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpaddd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpaddq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpaddsb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpaddsw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpaddusb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpaddusw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpaddw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpaddw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpalignr	$1, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpalignr	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpand	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpand	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpandn	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpandn	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpavgb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpavgb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpavgw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpavgw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpblendd	$11, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpblendd	$11, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vpblendd	$11, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpblendd	$11, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpblendvb	%ymm3, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpblendvb	%ymm3, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpblendw	$11, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpblendw	$11, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastb	%xmm0, %xmm0
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastb	(%rax), %xmm0
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastb	%xmm0, %ymm0
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastb	(%rax), %ymm0
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastd	%xmm0, %xmm0
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastd	(%rax), %xmm0
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastd	%xmm0, %ymm0
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastd	(%rax), %ymm0
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastq	%xmm0, %xmm0
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastq	(%rax), %xmm0
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastq	%xmm0, %ymm0
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastq	(%rax), %ymm0
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastw	%xmm0, %xmm0
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastw	(%rax), %xmm0
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastw	%xmm0, %ymm0
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastw	(%rax), %ymm0
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpcmpeqq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     1.00                        vperm2i128	$1, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     1.00    *                   vperm2i128	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      5     1.00                        vpermd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  2      9     2.00    *                   vpermd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      6     1.00                        vpermpd	$1, %ymm0, %ymm2
+# CHECK-NEXT:  3      10    2.00    *                   vpermpd	$1, (%rax), %ymm2
+# CHECK-NEXT:  2      7     1.00                        vpermps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  3      11    2.00    *                   vpermps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  2      6     1.00                        vpermq	$1, %ymm0, %ymm2
+# CHECK-NEXT:  2      9     2.00    *                   vpermq	$1, (%rax), %ymm2
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherdd	%xmm0, (%rax,%xmm1,2), %xmm2
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherdd	%ymm0, (%rax,%ymm1,2), %ymm2
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherdq	%xmm0, (%rax,%xmm1,2), %xmm2
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherdq	%ymm0, (%rax,%xmm1,2), %ymm2
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherqd	%xmm0, (%rax,%xmm1,2), %xmm2
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherqd	%xmm0, (%rax,%ymm1,2), %xmm2
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherqq	%xmm0, (%rax,%xmm1,2), %xmm2
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherqq	%ymm0, (%rax,%ymm1,2), %ymm2
+# CHECK-NEXT:  3      3     3.00                        vphaddd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  4      10    3.00    *                   vphaddd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  3      3     3.00                        vphaddsw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  4      10    3.00    *                   vphaddsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  3      3     3.00                        vphaddw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  4      10    3.00    *                   vphaddw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  3      3     3.00                        vphsubd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  4      10    3.00    *                   vphsubd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  3      3     3.00                        vphsubsw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  4      10    3.00    *                   vphsubsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  3      3     3.00                        vphsubw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  4      10    3.00    *                   vphsubw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vpmaddubsw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vpmaddubsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vpmaddwd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vpmaddwd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmaskmovd	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmaskmovd	(%rax), %ymm0, %ymm2
+# CHECK-NEXT:  18     1     6.00    *      *            vpmaskmovd	%xmm0, %xmm1, (%rax)
+# CHECK-NEXT:  42     1     12.00   *      *            vpmaskmovd	%ymm0, %ymm1, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   vpmaskmovq	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmaskmovq	(%rax), %ymm0, %ymm2
+# CHECK-NEXT:  10     1     4.00    *      *            vpmaskmovq	%xmm0, %xmm1, (%rax)
+# CHECK-NEXT:  18     1     6.00    *      *            vpmaskmovq	%ymm0, %ymm1, (%rax)
+# CHECK-NEXT:  1      1     0.25                        vpmaxsb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpmaxsd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpmaxsw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpmaxub	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxub	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpmaxud	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxud	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpmaxuw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxuw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpminsb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpminsb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpminsd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpminsd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpminsw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpminsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpminub	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpminub	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpminud	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpminud	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpminuw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpminuw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     1.00                        vpmovmskb	%ymm0, %ecx
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbd	%xmm0, %ymm2
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbd	(%rax), %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbq	%xmm0, %ymm2
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbq	(%rax), %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbw	%xmm0, %ymm2
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbw	(%rax), %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpmovsxdq	%xmm0, %ymm2
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxdq	(%rax), %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpmovsxwd	%xmm0, %ymm2
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxwd	(%rax), %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpmovsxwq	%xmm0, %ymm2
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxwq	(%rax), %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbd	%xmm0, %ymm2
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbd	(%rax), %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbq	%xmm0, %ymm2
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbq	(%rax), %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbw	%xmm0, %ymm2
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbw	(%rax), %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpmovzxdq	%xmm0, %ymm2
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxdq	(%rax), %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpmovzxwd	%xmm0, %ymm2
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxwd	(%rax), %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpmovzxwq	%xmm0, %ymm2
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxwq	(%rax), %ymm2
+# CHECK-NEXT:  1      3     0.50                        vpmuldq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vpmuldq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vpmulhrsw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhrsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vpmulhuw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhuw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vpmulhw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vpmulld	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vpmulld	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vpmullw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vpmullw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vpmuludq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vpmuludq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpor	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpor	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vpsadbw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vpsadbw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpshufb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpshufb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpshufd	$1, %ymm0, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpshufd	$1, (%rax), %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpshufhw	$1, %ymm0, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpshufhw	$1, (%rax), %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpshuflw	$1, %ymm0, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpshuflw	$1, (%rax), %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsignb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsignb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsignd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsignd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsignw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsignw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpslld	$1, %ymm0, %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpslld	%xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpslld	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpslldq	$1, %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsllq	$1, %ymm0, %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpsllq	%xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsllq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsllvd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsllvd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpsllvd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsllvd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsllvq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsllvq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpsllvq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsllvq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsllw	$1, %ymm0, %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpsllw	%xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsllw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsrad	$1, %ymm0, %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpsrad	%xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsrad	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsravd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsravd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpsravd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsravd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsraw	$1, %ymm0, %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpsraw	%xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsraw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsrld	$1, %ymm0, %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpsrld	%xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsrld	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpsrldq	$1, %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsrlq	$1, %ymm0, %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpsrlq	%xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsrlvd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlvd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpsrlvd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlvd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsrlvq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlvq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        vpsrlvq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlvq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsrlw	$1, %ymm0, %ymm2
+# CHECK-NEXT:  1      2     1.00                        vpsrlw	%xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpsubb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsubb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpsubd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpsubq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsubsb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsubsw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsubusb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpsubusw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpsubw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpsubw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpunpckhbw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhbw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpunpckhdq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhdq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpunpckhqdq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhqdq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpunpckhwd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhwd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpunpcklbw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpunpcklbw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpunpckldq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckldq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpunpcklqdq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpunpcklqdq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.50                        vpunpcklwd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpunpcklwd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vpxor	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vpxor	(%rax), %ymm1, %ymm2
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 6.67   6.67   6.67    -      -      -      -      -     93.75  138.75 98.25  36.25  80.50  80.50  29.00  52.33  52.33  52.33  50.67  50.67  50.67  2.50   2.50
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti128	(%rax), %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastsd	%xmm0, %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastss	%xmm0, %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vextracti128	$1, %ymm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vextracti128	$1, %ymm0, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherdpd	%xmm0, (%rax,%xmm1,2), %xmm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherdpd	%ymm0, (%rax,%xmm1,2), %ymm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherdps	%xmm0, (%rax,%xmm1,2), %xmm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherdps	%ymm0, (%rax,%ymm1,2), %ymm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherqpd	%xmm0, (%rax,%xmm1,2), %xmm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherqpd	%ymm0, (%rax,%ymm1,2), %ymm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherqps	%xmm0, (%rax,%xmm1,2), %xmm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherqps	%xmm0, (%rax,%ymm1,2), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinserti128	$1, %xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinserti128	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovntdqa	(%rax), %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00   2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vmpsadbw	$1, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00   2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmpsadbw	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpabsb	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsb	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpabsd	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsd	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpabsw	%ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsw	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpackssdw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackssdw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpacksswb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpacksswb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpackusdw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackusdw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpackuswb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackuswb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddsb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddsw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddusb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddusw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpalignr	$1, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpalignr	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpand	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpand	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpandn	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpandn	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpavgb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpavgw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpblendd	$11, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendd	$11, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpblendd	$11, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendd	$11, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpblendvb	%ymm3, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendvb	%ymm3, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpblendw	$11, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendw	$11, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%xmm0, %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastb	(%rax), %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%xmm0, %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastb	(%rax), %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastd	%xmm0, %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastd	(%rax), %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastd	%xmm0, %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastd	(%rax), %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastq	%xmm0, %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastq	(%rax), %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastq	%xmm0, %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastq	(%rax), %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%xmm0, %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastw	(%rax), %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%xmm0, %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastw	(%rax), %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpcmpeqq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vperm2i128	$1, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vperm2i128	$1, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpermd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -     2.00    -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpermpd	$1, %ymm0, %ymm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -     2.00    -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	$1, (%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpermps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -     2.00    -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpermq	$1, %ymm0, %ymm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -     2.00    -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	$1, (%rax), %ymm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherdd	%xmm0, (%rax,%xmm1,2), %xmm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherdd	%ymm0, (%rax,%ymm1,2), %ymm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherdq	%xmm0, (%rax,%xmm1,2), %xmm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherdq	%ymm0, (%rax,%xmm1,2), %ymm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherqd	%xmm0, (%rax,%xmm1,2), %xmm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherqd	%xmm0, (%rax,%ymm1,2), %xmm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherqq	%xmm0, (%rax,%xmm1,2), %xmm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherqq	%ymm0, (%rax,%ymm1,2), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vphaddd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vphaddd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vphaddsw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vphaddsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vphaddw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vphaddw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vphsubd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vphsubd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vphsubsw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vphsubsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vphsubw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vphsubw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaddubsw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddubsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaddwd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddwd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaskmovd	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaskmovd	(%rax), %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     3.00   3.00   6.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vpmaskmovd	%xmm0, %xmm1, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     6.00   6.00   12.00  0.33   0.33   0.33    -      -      -     0.50   0.50   vpmaskmovd	%ymm0, %ymm1, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaskmovq	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaskmovq	(%rax), %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     2.00   2.00   4.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vpmaskmovq	%xmm0, %xmm1, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     3.00   3.00   6.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vpmaskmovq	%ymm0, %ymm1, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxsb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxsd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxsw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxub	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxub	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxud	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxud	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxuw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxuw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminsb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminsd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminsw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminub	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminub	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminud	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminud	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminuw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminuw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovmskb	%ymm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbd	%xmm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbd	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbq	%xmm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbq	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbw	%xmm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbw	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxdq	%xmm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxdq	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwd	%xmm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwd	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwq	%xmm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwq	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbd	%xmm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbd	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbq	%xmm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbq	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbw	%xmm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbw	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxdq	%xmm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxdq	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwd	%xmm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwd	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwq	%xmm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwq	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmuldq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmuldq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhrsw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhrsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhuw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhuw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulld	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmullw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmuludq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmuludq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpor	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpor	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsadbw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsadbw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshufb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshufd	$1, %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$1, (%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshufhw	$1, %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufhw	$1, (%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshuflw	$1, %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshuflw	$1, (%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsignb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsignb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsignd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsignd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsignw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsignw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpslld	$1, %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpslld	%xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpslld	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpslldq	$1, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsllq	$1, %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllq	%xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsllvd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllvd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsllvd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllvd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsllvq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllvq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsllvq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllvq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	$1, %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	%xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrad	$1, %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrad	%xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrad	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsravd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsravd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsravd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsravd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	$1, %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	%xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrld	$1, %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrld	%xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrld	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrldq	$1, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrlq	$1, %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlq	%xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrlvd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlvd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrlvd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlvd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrlvq	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlvq	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrlvq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlvq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	$1, %ymm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	%xmm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubsb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubsw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubusb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusb	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubusw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhbw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhbw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhdq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhqdq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhqdq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhwd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhwd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklbw	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklbw	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckldq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklqdq	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklqdq	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklwd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklwd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpxor	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpxor	(%rax), %ymm1, %ymm2
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512.s
new file mode 100644
index 0000000000000..ea1b1da108f11
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512.s
@@ -0,0 +1,2819 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+kandw             %k0, %k1, %k2
+kandnw            %k0, %k1, %k2
+knotw             %k0, %k2
+korw              %k0, %k1, %k2
+kxnorw            %k0, %k1, %k2
+kxorw             %k0, %k1, %k2
+kshiftlw          $2, %k1, %k2
+kshiftrw          $2, %k1, %k2
+kunpckbw          %k0, %k1, %k2
+
+vaddpd            %zmm16, %zmm17, %zmm19
+vaddpd            (%rax), %zmm17, %zmm19
+vaddpd            (%rax){1to8}, %zmm17, %zmm19
+vaddpd            %zmm16, %zmm17, %zmm19 {k1}
+vaddpd            (%rax), %zmm17, %zmm19 {k1}
+vaddpd            (%rax){1to8}, %zmm17, %zmm19 {k1}
+vaddpd            %zmm16, %zmm17, %zmm19 {z}{k1}
+vaddpd            (%rax), %zmm17, %zmm19 {z}{k1}
+vaddpd            (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vaddps            %zmm16, %zmm17, %zmm19
+vaddps            (%rax), %zmm17, %zmm19
+vaddps            (%rax){1to16}, %zmm17, %zmm19
+vaddps            %zmm16, %zmm17, %zmm19 {k1}
+vaddps            (%rax), %zmm17, %zmm19 {k1}
+vaddps            (%rax){1to16}, %zmm17, %zmm19 {k1}
+vaddps            %zmm16, %zmm17, %zmm19 {z}{k1}
+vaddps            (%rax), %zmm17, %zmm19 {z}{k1}
+vaddps            (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+valignd           $1, %zmm16, %zmm17, %zmm19
+valignd           $1, (%rax), %zmm17, %zmm19
+valignd           $1, (%rax){1to16}, %zmm17, %zmm19
+valignd           $1, %zmm16, %zmm17, %zmm19 {k1}
+valignd           $1, (%rax), %zmm17, %zmm19 {k1}
+valignd           $1, (%rax){1to16}, %zmm17, %zmm19 {k1}
+valignd           $1, %zmm16, %zmm17, %zmm19 {z}{k1}
+valignd           $1, (%rax), %zmm17, %zmm19 {z}{k1}
+valignd           $1, (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+valignq           $1, %zmm16, %zmm17, %zmm19
+valignq           $1, (%rax), %zmm17, %zmm19
+valignq           $1, (%rax){1to8}, %zmm17, %zmm19
+valignq           $1, %zmm16, %zmm17, %zmm19 {k1}
+valignq           $1, (%rax), %zmm17, %zmm19 {k1}
+valignq           $1, (%rax){1to8}, %zmm17, %zmm19 {k1}
+valignq           $1, %zmm16, %zmm17, %zmm19 {z}{k1}
+valignq           $1, (%rax), %zmm17, %zmm19 {z}{k1}
+valignq           $1, (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vbroadcastf32x4   (%rax), %zmm19
+vbroadcastf32x4   (%rax), %zmm19 {k1}
+vbroadcastf32x4   (%rax), %zmm19 {z}{k1}
+
+vbroadcastf64x4   (%rax), %zmm19
+vbroadcastf64x4   (%rax), %zmm19 {k1}
+vbroadcastf64x4   (%rax), %zmm19 {z}{k1}
+
+vbroadcasti32x4   (%rax), %zmm19
+vbroadcasti32x4   (%rax), %zmm19 {k1}
+vbroadcasti32x4   (%rax), %zmm19 {z}{k1}
+
+vbroadcasti64x4   (%rax), %zmm19
+vbroadcasti64x4   (%rax), %zmm19 {k1}
+vbroadcasti64x4   (%rax), %zmm19 {z}{k1}
+
+vbroadcastsd      %xmm16, %zmm19
+vbroadcastsd      (%rax), %zmm19
+vbroadcastsd      %xmm16, %zmm19 {k1}
+vbroadcastsd      (%rax), %zmm19 {k1}
+vbroadcastsd      %xmm16, %zmm19 {z}{k1}
+vbroadcastsd      (%rax), %zmm19 {z}{k1}
+
+vbroadcastss      %xmm16, %zmm19
+vbroadcastss      (%rax), %zmm19
+vbroadcastss      %xmm16, %zmm19 {k1}
+vbroadcastss      (%rax), %zmm19 {k1}
+vbroadcastss      %xmm16, %zmm19 {z}{k1}
+vbroadcastss      (%rax), %zmm19 {z}{k1}
+
+vcmppd            $0, %zmm0, %zmm1, %k2
+vcmppd            $0, (%rax), %zmm1, %k2
+vcmppd            $0, (%rax){1to8}, %zmm1, %k2
+vcmppd            $0, %zmm0, %zmm1, %k2 {k3}
+vcmppd            $0, (%rax), %zmm1, %k2 {k3}
+vcmppd            $0, (%rax){1to8}, %zmm1, %k2 {k3}
+
+vcmpps            $0, %zmm0, %zmm1, %k2
+vcmpps            $0, (%rax), %zmm1, %k2
+vcmpps            $0, (%rax){1to16}, %zmm1, %k2
+vcmpps            $0, %zmm0, %zmm1, %k2 {k3}
+vcmpps            $0, (%rax), %zmm1, %k2 {k3}
+vcmpps            $0, (%rax){1to16}, %zmm1, %k2 {k3}
+
+vcmpsd            $0, %xmm0, %xmm1, %k2
+vcmpsd            $0, (%rax), %xmm1, %k2
+vcmpsd            $0, %xmm0, %xmm1, %k2 {k3}
+vcmpsd            $0, (%rax), %xmm1, %k2 {k3}
+
+vcmpss            $0, %xmm0, %xmm1, %k2
+vcmpss            $0, (%rax), %xmm1, %k2
+vcmpss            $0, %xmm0, %xmm1, %k2 {k3}
+vcmpss            $0, (%rax), %xmm1, %k2 {k3}
+
+vcomiss           %xmm16, %xmm17
+vcomiss           (%rax), %xmm17
+
+vcvtdq2pd         %ymm16, %zmm19
+vcvtdq2pd         (%rax), %zmm19
+vcvtdq2pd         (%rax){1to8}, %zmm19
+vcvtdq2pd         %ymm16, %zmm19 {k1}
+vcvtdq2pd         (%rax), %zmm19 {k1}
+vcvtdq2pd         (%rax){1to8}, %zmm19 {k1}
+vcvtdq2pd         %ymm16, %zmm19 {z}{k1}
+vcvtdq2pd         (%rax), %zmm19 {z}{k1}
+vcvtdq2pd         (%rax){1to8}, %zmm19 {z}{k1}
+
+vcvtdq2ps         %zmm16, %zmm19
+vcvtdq2ps         (%rax), %zmm19
+vcvtdq2ps         (%rax){1to16}, %zmm19
+vcvtdq2ps         %zmm16, %zmm19 {k1}
+vcvtdq2ps         (%rax), %zmm19 {k1}
+vcvtdq2ps         (%rax){1to16}, %zmm19 {k1}
+vcvtdq2ps         %zmm16, %zmm19 {z}{k1}
+vcvtdq2ps         (%rax), %zmm19 {z}{k1}
+vcvtdq2ps         (%rax){1to16}, %zmm19 {z}{k1}
+
+vcvtpd2dq         %zmm16, %ymm19
+vcvtpd2dq         (%rax), %ymm19
+vcvtpd2dq         (%rax){1to8}, %ymm19
+vcvtpd2dq         %zmm16, %ymm19 {k1}
+vcvtpd2dq         (%rax), %ymm19 {k1}
+vcvtpd2dq         (%rax){1to8}, %ymm19 {k1}
+vcvtpd2dq         %zmm16, %ymm19 {z}{k1}
+vcvtpd2dq         (%rax), %ymm19 {z}{k1}
+vcvtpd2dq         (%rax){1to8}, %ymm19 {z}{k1}
+
+vcvtpd2udq        %zmm16, %ymm19
+vcvtpd2udq        (%rax), %ymm19
+vcvtpd2udq        (%rax){1to8}, %ymm19
+vcvtpd2udq        %zmm16, %ymm19 {k1}
+vcvtpd2udq        (%rax), %ymm19 {k1}
+vcvtpd2udq        (%rax){1to8}, %ymm19 {k1}
+vcvtpd2udq        %zmm16, %ymm19 {z}{k1}
+vcvtpd2udq        (%rax), %ymm19 {z}{k1}
+vcvtpd2udq        (%rax){1to8}, %ymm19 {z}{k1}
+
+vcvttpd2dq        %zmm16, %ymm19
+vcvttpd2dq        (%rax), %ymm19
+vcvttpd2dq        (%rax){1to8}, %ymm19
+vcvttpd2dq        %zmm16, %ymm19 {k1}
+vcvttpd2dq        (%rax), %ymm19 {k1}
+vcvttpd2dq        (%rax){1to8}, %ymm19 {k1}
+vcvttpd2dq        %zmm16, %ymm19 {z}{k1}
+vcvttpd2dq        (%rax), %ymm19 {z}{k1}
+vcvttpd2dq        (%rax){1to8}, %ymm19 {z}{k1}
+
+vcvttpd2udq       %zmm16, %ymm19
+vcvttpd2udq       (%rax), %ymm19
+vcvttpd2udq       (%rax){1to8}, %ymm19
+vcvttpd2udq       %zmm16, %ymm19 {k1}
+vcvttpd2udq       (%rax), %ymm19 {k1}
+vcvttpd2udq       (%rax){1to8}, %ymm19 {k1}
+vcvttpd2udq       %zmm16, %ymm19 {z}{k1}
+vcvttpd2udq       (%rax), %ymm19 {z}{k1}
+vcvttpd2udq       (%rax){1to8}, %ymm19 {z}{k1}
+
+vcvtpd2ps         %zmm16, %ymm19
+vcvtpd2ps         (%rax), %ymm19
+vcvtpd2ps         (%rax){1to8}, %ymm19
+vcvtpd2ps         %zmm16, %ymm19 {k1}
+vcvtpd2ps         (%rax), %ymm19 {k1}
+vcvtpd2ps         (%rax){1to8}, %ymm19 {k1}
+vcvtpd2ps         %zmm16, %ymm19 {z}{k1}
+vcvtpd2ps         (%rax), %ymm19 {z}{k1}
+vcvtpd2ps         (%rax){1to8}, %ymm19 {z}{k1}
+
+vcvtps2dq         %zmm16, %zmm19
+vcvtps2dq         (%rax), %zmm19
+vcvtps2dq         (%rax){1to16}, %zmm19
+vcvtps2dq         %zmm16, %zmm19 {k1}
+vcvtps2dq         (%rax), %zmm19 {k1}
+vcvtps2dq         (%rax){1to16}, %zmm19 {k1}
+vcvtps2dq         %zmm16, %zmm19 {z}{k1}
+vcvtps2dq         (%rax), %zmm19 {z}{k1}
+vcvtps2dq         (%rax){1to16}, %zmm19 {z}{k1}
+
+vcvttps2dq        %zmm16, %zmm19
+vcvttps2dq        (%rax), %zmm19
+vcvttps2dq        (%rax){1to16}, %zmm19
+vcvttps2dq        %zmm16, %zmm19 {k1}
+vcvttps2dq        (%rax), %zmm19 {k1}
+vcvttps2dq        (%rax){1to16}, %zmm19 {k1}
+vcvttps2dq        %zmm16, %zmm19 {z}{k1}
+vcvttps2dq        (%rax), %zmm19 {z}{k1}
+vcvttps2dq        (%rax){1to16}, %zmm19 {z}{k1}
+
+vcvtps2pd         %ymm16, %zmm19
+vcvtps2pd         (%rax), %zmm19
+vcvtps2pd         (%rax){1to8}, %zmm19
+vcvtps2pd         %ymm16, %zmm19 {k1}
+vcvtps2pd         (%rax), %zmm19 {k1}
+vcvtps2pd         (%rax){1to8}, %zmm19 {k1}
+vcvtps2pd         %ymm16, %zmm19 {z}{k1}
+vcvtps2pd         (%rax), %zmm19 {z}{k1}
+vcvtps2pd         (%rax){1to8}, %zmm19 {z}{k1}
+
+vcvtsd2usi        %xmm0, %ecx
+vcvtsd2usi        %xmm0, %rcx
+vcvtsd2usi        (%rax), %ecx
+vcvtsd2usi        (%rax), %rcx
+
+vcvtss2usi        %xmm0, %ecx
+vcvtss2usi        %xmm0, %rcx
+vcvtss2usi        (%rax), %ecx
+vcvtss2usi        (%rax), %rcx
+
+vcvtps2udq        %zmm16, %zmm19
+vcvtps2udq        (%rax), %zmm19
+vcvtps2udq        (%rax){1to16}, %zmm19
+vcvtps2udq        %zmm16, %zmm19 {k1}
+vcvtps2udq        (%rax), %zmm19 {k1}
+vcvtps2udq        (%rax){1to16}, %zmm19 {k1}
+vcvtps2udq        %zmm16, %zmm19 {z}{k1}
+vcvtps2udq        (%rax), %zmm19 {z}{k1}
+vcvtps2udq        (%rax){1to16}, %zmm19 {z}{k1}
+
+vcvttps2udq       %zmm16, %zmm19
+vcvttps2udq       (%rax), %zmm19
+vcvttps2udq       (%rax){1to16}, %zmm19
+vcvttps2udq       %zmm16, %zmm19 {k1}
+vcvttps2udq       (%rax), %zmm19 {k1}
+vcvttps2udq       (%rax){1to16}, %zmm19 {k1}
+vcvttps2udq       %zmm16, %zmm19 {z}{k1}
+vcvttps2udq       (%rax), %zmm19 {z}{k1}
+vcvttps2udq       (%rax){1to16}, %zmm19 {z}{k1}
+
+vcvttsd2usi       %xmm0, %ecx
+vcvttsd2usi       %xmm0, %rcx
+vcvttsd2usi       (%rax), %ecx
+vcvttsd2usi       (%rax), %rcx
+
+vcvttss2usi       %xmm0, %ecx
+vcvttss2usi       %xmm0, %rcx
+vcvttss2usi       (%rax), %ecx
+vcvttss2usi       (%rax), %rcx
+
+vcvtudq2pd        %ymm16, %zmm19
+vcvtudq2pd        (%rax), %zmm19
+vcvtudq2pd        (%rax){1to8}, %zmm19
+vcvtudq2pd        %ymm16, %zmm19 {k1}
+vcvtudq2pd        (%rax), %zmm19 {k1}
+vcvtudq2pd        (%rax){1to8}, %zmm19 {k1}
+vcvtudq2pd        %ymm16, %zmm19 {z}{k1}
+vcvtudq2pd        (%rax), %zmm19 {z}{k1}
+vcvtudq2pd        (%rax){1to8}, %zmm19 {z}{k1}
+
+vcvtudq2ps        %zmm16, %zmm19
+vcvtudq2ps        (%rax), %zmm19
+vcvtudq2ps        (%rax){1to16}, %zmm19
+vcvtudq2ps        %zmm16, %zmm19 {k1}
+vcvtudq2ps        (%rax), %zmm19 {k1}
+vcvtudq2ps        (%rax){1to16}, %zmm19 {k1}
+vcvtudq2ps        %zmm16, %zmm19 {z}{k1}
+vcvtudq2ps        (%rax), %zmm19 {z}{k1}
+vcvtudq2ps        (%rax){1to16}, %zmm19 {z}{k1}
+
+vcvtusi2sdl       %ecx, %xmm0, %xmm2
+vcvtusi2sdq       %rcx, %xmm0, %xmm2
+vcvtusi2sdl       (%rax), %xmm0, %xmm2
+vcvtusi2sdq       (%rax), %xmm0, %xmm2
+
+vcvtusi2ssl       %ecx, %xmm0, %xmm2
+vcvtusi2ssq       %rcx, %xmm0, %xmm2
+vcvtusi2ssl       (%rax), %xmm0, %xmm2
+vcvtusi2ssq       (%rax), %xmm0, %xmm2
+
+vdivpd            %zmm16, %zmm17, %zmm19
+vdivpd            (%rax), %zmm17, %zmm19
+vdivpd            (%rax){1to8}, %zmm17, %zmm19
+vdivpd            %zmm16, %zmm17, %zmm19 {k1}
+vdivpd            (%rax), %zmm17, %zmm19 {k1}
+vdivpd            (%rax){1to8}, %zmm17, %zmm19 {k1}
+vdivpd            %zmm16, %zmm17, %zmm19 {z}{k1}
+vdivpd            (%rax), %zmm17, %zmm19 {z}{k1}
+vdivpd            (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vdivps            %zmm16, %zmm17, %zmm19
+vdivps            (%rax), %zmm17, %zmm19
+vdivps            (%rax){1to16}, %zmm17, %zmm19
+vdivps            %zmm16, %zmm17, %zmm19 {k1}
+vdivps            (%rax), %zmm17, %zmm19 {k1}
+vdivps            (%rax){1to16}, %zmm17, %zmm19 {k1}
+vdivps            %zmm16, %zmm17, %zmm19 {z}{k1}
+vdivps            (%rax), %zmm17, %zmm19 {z}{k1}
+vdivps            (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vgatherdpd        (%rax,%ymm1,2), %zmm2 {k1}
+vgatherdps        (%rax,%zmm1,2), %zmm2 {k1}
+vgatherqpd        (%rax,%zmm1,2), %zmm2 {k1}
+vgatherqps        (%rax,%zmm1,2), %ymm2 {k1}
+
+vmaxpd            %zmm16, %zmm17, %zmm19
+vmaxpd            (%rax), %zmm17, %zmm19
+vmaxpd            (%rax){1to8}, %zmm17, %zmm19
+vmaxpd            %zmm16, %zmm17, %zmm19 {k1}
+vmaxpd            (%rax), %zmm17, %zmm19 {k1}
+vmaxpd            (%rax){1to8}, %zmm17, %zmm19 {k1}
+vmaxpd            %zmm16, %zmm17, %zmm19 {z}{k1}
+vmaxpd            (%rax), %zmm17, %zmm19 {z}{k1}
+vmaxpd            (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vmaxps            %zmm16, %zmm17, %zmm19
+vmaxps            (%rax), %zmm17, %zmm19
+vmaxps            (%rax){1to16}, %zmm17, %zmm19
+vmaxps            %zmm16, %zmm17, %zmm19 {k1}
+vmaxps            (%rax), %zmm17, %zmm19 {k1}
+vmaxps            (%rax){1to16}, %zmm17, %zmm19 {k1}
+vmaxps            %zmm16, %zmm17, %zmm19 {z}{k1}
+vmaxps            (%rax), %zmm17, %zmm19 {z}{k1}
+vmaxps            (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vminpd            %zmm16, %zmm17, %zmm19
+vminpd            (%rax), %zmm17, %zmm19
+vminpd            (%rax){1to8}, %zmm17, %zmm19
+vminpd            %zmm16, %zmm17, %zmm19 {k1}
+vminpd            (%rax), %zmm17, %zmm19 {k1}
+vminpd            (%rax){1to8}, %zmm17, %zmm19 {k1}
+vminpd            %zmm16, %zmm17, %zmm19 {z}{k1}
+vminpd            (%rax), %zmm17, %zmm19 {z}{k1}
+vminpd            (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vminps            %zmm16, %zmm17, %zmm19
+vminps            (%rax), %zmm17, %zmm19
+vminps            (%rax){1to16}, %zmm17, %zmm19
+vminps            %zmm16, %zmm17, %zmm19 {k1}
+vminps            (%rax), %zmm17, %zmm19 {k1}
+vminps            (%rax){1to16}, %zmm17, %zmm19 {k1}
+vminps            %zmm16, %zmm17, %zmm19 {z}{k1}
+vminps            (%rax), %zmm17, %zmm19 {z}{k1}
+vminps            (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vmovapd           %zmm16, %zmm19
+vmovapd           (%rax), %zmm19
+vmovapd           %zmm16, (%rax)
+vmovapd           %zmm16, %zmm19 {k1}
+vmovapd           (%rax), %zmm19 {k1}
+vmovapd           %zmm16, (%rax) {k1}
+vmovapd           %zmm16, %zmm19 {z}{k1}
+vmovapd           (%rax), %zmm19 {z}{k1}
+
+vmovaps           %zmm16, %zmm19
+vmovaps           (%rax), %zmm19
+vmovaps           %zmm16, (%rax)
+vmovaps           %zmm16, %zmm19 {k1}
+vmovaps           (%rax), %zmm19 {k1}
+vmovaps           %zmm16, (%rax) {k1}
+vmovaps           %zmm16, %zmm19 {z}{k1}
+vmovaps           (%rax), %zmm19 {z}{k1}
+
+vmovddup          %zmm16, %zmm19
+vmovddup          (%rax), %zmm19
+vmovddup          %zmm16, %zmm19 {k1}
+vmovddup          (%rax), %zmm19 {k1}
+vmovddup          %zmm16, %zmm19 {z}{k1}
+vmovddup          (%rax), %zmm19 {z}{k1}
+
+vmovdqa32         %zmm16, %zmm19
+vmovdqa32         (%rax), %zmm19
+vmovdqa32         %zmm16, (%rax)
+vmovdqa32         %zmm16, %zmm19 {k1}
+vmovdqa32         (%rax), %zmm19 {k1}
+vmovdqa32         %zmm16, (%rax) {k1}
+vmovdqa32         %zmm16, %zmm19 {z}{k1}
+vmovdqa32         (%rax), %zmm19 {z}{k1}
+
+vmovdqa64         %zmm16, %zmm19
+vmovdqa64         (%rax), %zmm19
+vmovdqa64         %zmm16, (%rax)
+vmovdqa64         %zmm16, %zmm19 {k1}
+vmovdqa64         (%rax), %zmm19 {k1}
+vmovdqa64         %zmm16, (%rax) {k1}
+vmovdqa64         %zmm16, %zmm19 {z}{k1}
+vmovdqa64         (%rax), %zmm19 {z}{k1}
+
+vmovdqu32         %zmm16, %zmm19
+vmovdqu32         (%rax), %zmm19
+vmovdqu32         %zmm16, (%rax)
+vmovdqu32         %zmm16, %zmm19 {k1}
+vmovdqu32         (%rax), %zmm19 {k1}
+vmovdqu32         %zmm16, (%rax) {k1}
+vmovdqu32         %zmm16, %zmm19 {z}{k1}
+vmovdqu32         (%rax), %zmm19 {z}{k1}
+
+vmovdqu64         %zmm16, %zmm19
+vmovdqu64         (%rax), %zmm19
+vmovdqu64         %zmm16, (%rax)
+vmovdqu64         %zmm16, %zmm19 {k1}
+vmovdqu64         (%rax), %zmm19 {k1}
+vmovdqu64         %zmm16, (%rax) {k1}
+vmovdqu64         %zmm16, %zmm19 {z}{k1}
+vmovdqu64         (%rax), %zmm19 {z}{k1}
+
+vmovshdup         %zmm16, %zmm19
+vmovshdup         (%rax), %zmm19
+vmovshdup         %zmm16, %zmm19 {k1}
+vmovshdup         (%rax), %zmm19 {k1}
+vmovshdup         %zmm16, %zmm19 {z}{k1}
+vmovshdup         (%rax), %zmm19 {z}{k1}
+
+vmovsldup         %zmm16, %zmm19
+vmovsldup         (%rax), %zmm19
+vmovsldup         %zmm16, %zmm19 {k1}
+vmovsldup         (%rax), %zmm19 {k1}
+vmovsldup         %zmm16, %zmm19 {z}{k1}
+vmovsldup         (%rax), %zmm19 {z}{k1}
+
+vmovupd           %zmm16, %zmm19
+vmovupd           (%rax), %zmm19
+vmovupd           %zmm16, (%rax)
+vmovupd           %zmm16, %zmm19 {k1}
+vmovupd           (%rax), %zmm19 {k1}
+vmovupd           %zmm16, (%rax) {k1}
+vmovupd           %zmm16, %zmm19 {z}{k1}
+vmovupd           (%rax), %zmm19 {z}{k1}
+
+vmovups           %zmm16, %zmm19
+vmovups           (%rax), %zmm19
+vmovups           %zmm16, (%rax)
+vmovups           %zmm16, %zmm19 {k1}
+vmovups           (%rax), %zmm19 {k1}
+vmovups           %zmm16, (%rax) {k1}
+vmovups           %zmm16, %zmm19 {z}{k1}
+vmovups           (%rax), %zmm19 {z}{k1}
+
+vmulpd            %zmm16, %zmm17, %zmm19
+vmulpd            (%rax), %zmm17, %zmm19
+vmulpd            (%rax){1to8}, %zmm17, %zmm19
+vmulpd            %zmm16, %zmm17, %zmm19 {k1}
+vmulpd            (%rax), %zmm17, %zmm19 {k1}
+vmulpd            (%rax){1to8}, %zmm17, %zmm19 {k1}
+vmulpd            %zmm16, %zmm17, %zmm19 {z}{k1}
+vmulpd            (%rax), %zmm17, %zmm19 {z}{k1}
+vmulpd            (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vmulps            %zmm16, %zmm17, %zmm19
+vmulps            (%rax), %zmm17, %zmm19
+vmulps            (%rax){1to16}, %zmm17, %zmm19
+vmulps            %zmm16, %zmm17, %zmm19 {k1}
+vmulps            (%rax), %zmm17, %zmm19 {k1}
+vmulps            (%rax){1to16}, %zmm17, %zmm19 {k1}
+vmulps            %zmm16, %zmm17, %zmm19 {z}{k1}
+vmulps            (%rax), %zmm17, %zmm19 {z}{k1}
+vmulps            (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vpabsd            %zmm16, %zmm19
+vpabsd            (%rax), %zmm19
+vpabsd            (%rax){1to16}, %zmm19
+vpabsd            %zmm16, %zmm19 {k1}
+vpabsd            (%rax), %zmm19 {k1}
+vpabsd            (%rax){1to16}, %zmm19 {k1}
+vpabsd            %zmm16, %zmm19 {z}{k1}
+vpabsd            (%rax), %zmm19 {z}{k1}
+vpabsd            (%rax){1to16}, %zmm19 {z}{k1}
+
+vpabsq            %zmm16, %zmm19
+vpabsq            (%rax), %zmm19
+vpabsq            (%rax){1to8}, %zmm19
+vpabsq            %zmm16, %zmm19 {k1}
+vpabsq            (%rax), %zmm19 {k1}
+vpabsq            (%rax){1to8}, %zmm19 {k1}
+vpabsq            %zmm16, %zmm19 {z}{k1}
+vpabsq            (%rax), %zmm19 {z}{k1}
+vpabsq            (%rax){1to8}, %zmm19 {z}{k1}
+
+vpaddd            %zmm16, %zmm17, %zmm19
+vpaddd            (%rax), %zmm17, %zmm19
+vpaddd            (%rax){1to16}, %zmm17, %zmm19
+vpaddd            %zmm16, %zmm17, %zmm19 {k1}
+vpaddd            (%rax), %zmm17, %zmm19 {k1}
+vpaddd            (%rax){1to16}, %zmm17, %zmm19 {k1}
+vpaddd            %zmm16, %zmm17, %zmm19 {z}{k1}
+vpaddd            (%rax), %zmm17, %zmm19 {z}{k1}
+vpaddd            (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vpaddq            %zmm16, %zmm17, %zmm19
+vpaddq            (%rax), %zmm17, %zmm19
+vpaddq            (%rax){1to8}, %zmm17, %zmm19
+vpaddq            %zmm16, %zmm17, %zmm19 {k1}
+vpaddq            (%rax), %zmm17, %zmm19 {k1}
+vpaddq            (%rax){1to8}, %zmm17, %zmm19 {k1}
+vpaddq            %zmm16, %zmm17, %zmm19 {z}{k1}
+vpaddq            (%rax), %zmm17, %zmm19 {z}{k1}
+vpaddq            (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vpbroadcastd      %xmm16, %zmm19
+vpbroadcastd      (%rax), %zmm19
+vpbroadcastd      %xmm16, %zmm19 {k1}
+vpbroadcastd      (%rax), %zmm19 {k1}
+vpbroadcastd      %xmm16, %zmm19 {z}{k1}
+vpbroadcastd      (%rax), %zmm19 {z}{k1}
+
+vpbroadcastq      %xmm16, %zmm19
+vpbroadcastq      (%rax), %zmm19
+vpbroadcastq      %xmm16, %zmm19 {k1}
+vpbroadcastq      (%rax), %zmm19 {k1}
+vpbroadcastq      %xmm16, %zmm19 {z}{k1}
+vpbroadcastq      (%rax), %zmm19 {z}{k1}
+
+vpcmpd            $0, %zmm0, %zmm1, %k2
+vpcmpd            $0, (%rax), %zmm1, %k2
+vpcmpd            $0, (%rax){1to16}, %zmm1, %k2
+vpcmpd            $0, %zmm0, %zmm1, %k2 {k3}
+vpcmpd            $0, (%rax), %zmm1, %k2 {k3}
+vpcmpd            $0, (%rax){1to16}, %zmm1, %k2 {k3}
+
+vpcmpeqd          %zmm0, %zmm1, %k2
+vpcmpeqd          (%rax), %zmm1, %k2
+vpcmpeqd          (%rax){1to16}, %zmm1, %k2
+vpcmpeqd          %zmm0, %zmm1, %k2 {k3}
+vpcmpeqd          (%rax), %zmm1, %k2 {k3}
+vpcmpeqd          (%rax){1to16}, %zmm1, %k2 {k3}
+
+vpcmpeqq          %zmm0, %zmm1, %k2
+vpcmpeqq          (%rax), %zmm1, %k2
+vpcmpeqq          (%rax){1to8}, %zmm1, %k2
+vpcmpeqq          %zmm0, %zmm1, %k2 {k3}
+vpcmpeqq          (%rax), %zmm1, %k2 {k3}
+vpcmpeqq          (%rax){1to8}, %zmm1, %k2 {k3}
+
+vpcmpgtd          %zmm0, %zmm1, %k2
+vpcmpgtd          (%rax), %zmm1, %k2
+vpcmpgtd          (%rax){1to16}, %zmm1, %k2
+vpcmpgtd          %zmm0, %zmm1, %k2 {k3}
+vpcmpgtd          (%rax), %zmm1, %k2 {k3}
+vpcmpgtd          (%rax){1to16}, %zmm1, %k2 {k3}
+
+vpcmpgtq          %zmm0, %zmm1, %k2
+vpcmpgtq          (%rax), %zmm1, %k2
+vpcmpgtq          (%rax){1to8}, %zmm1, %k2
+vpcmpgtq          %zmm0, %zmm1, %k2 {k3}
+vpcmpgtq          (%rax), %zmm1, %k2 {k3}
+vpcmpgtq          (%rax){1to8}, %zmm1, %k2 {k3}
+
+vpcmpq            $0, %zmm0, %zmm1, %k2
+vpcmpq            $0, (%rax), %zmm1, %k2
+vpcmpq            $0, (%rax){1to8}, %zmm1, %k2
+vpcmpq            $0, %zmm0, %zmm1, %k2 {k3}
+vpcmpq            $0, (%rax), %zmm1, %k2 {k3}
+vpcmpq            $0, (%rax){1to8}, %zmm1, %k2 {k3}
+
+vpcmpud           $0, %zmm0, %zmm1, %k2
+vpcmpud           $0, (%rax), %zmm1, %k2
+vpcmpud           $0, (%rax){1to16}, %zmm1, %k2
+vpcmpud           $0, %zmm0, %zmm1, %k2 {k3}
+vpcmpud           $0, (%rax), %zmm1, %k2 {k3}
+vpcmpud           $0, (%rax){1to16}, %zmm1, %k2 {k3}
+
+vpcmpuq           $0, %zmm0, %zmm1, %k2
+vpcmpuq           $0, (%rax), %zmm1, %k2
+vpcmpuq           $0, (%rax){1to8}, %zmm1, %k2
+vpcmpuq           $0, %zmm0, %zmm1, %k2 {k3}
+vpcmpuq           $0, (%rax), %zmm1, %k2 {k3}
+vpcmpuq           $0, (%rax){1to8}, %zmm1, %k2 {k3}
+
+vpgatherdq        (%rax,%ymm1,2), %zmm2 {k1}
+vpgatherdd        (%rax,%zmm1,2), %zmm2 {k1}
+vpgatherqq        (%rax,%zmm1,2), %zmm2 {k1}
+vpgatherqd        (%rax,%zmm1,2), %ymm2 {k1}
+
+vpmovsxbd         %xmm16, %zmm19
+vpmovsxbd         (%rax), %zmm19
+vpmovsxbd         %xmm16, %zmm19 {k1}
+vpmovsxbd         (%rax), %zmm19 {k1}
+vpmovsxbd         %xmm16, %zmm19 {z}{k1}
+vpmovsxbd         (%rax), %zmm19 {z}{k1}
+
+vpmovsxbq         %xmm16, %zmm19
+vpmovsxbq         (%rax), %zmm19
+vpmovsxbq         %xmm16, %zmm19 {k1}
+vpmovsxbq         (%rax), %zmm19 {k1}
+vpmovsxbq         %xmm16, %zmm19 {z}{k1}
+vpmovsxbq         (%rax), %zmm19 {z}{k1}
+
+vpmovsxdq         %ymm16, %zmm19
+vpmovsxdq         (%rax), %zmm19
+vpmovsxdq         %ymm16, %zmm19 {k1}
+vpmovsxdq         (%rax), %zmm19 {k1}
+vpmovsxdq         %ymm16, %zmm19 {z}{k1}
+vpmovsxdq         (%rax), %zmm19 {z}{k1}
+
+vpmovsxwd         %ymm16, %zmm19
+vpmovsxwd         (%rax), %zmm19
+vpmovsxwd         %ymm16, %zmm19 {k1}
+vpmovsxwd         (%rax), %zmm19 {k1}
+vpmovsxwd         %ymm16, %zmm19 {z}{k1}
+vpmovsxwd         (%rax), %zmm19 {z}{k1}
+
+vpmovsxwq         %xmm16, %zmm19
+vpmovsxwq         (%rax), %zmm19
+vpmovsxwq         %xmm16, %zmm19 {k1}
+vpmovsxwq         (%rax), %zmm19 {k1}
+vpmovsxwq         %xmm16, %zmm19 {z}{k1}
+vpmovsxwq         (%rax), %zmm19 {z}{k1}
+
+vpmovzxbd         %xmm16, %zmm19
+vpmovzxbd         (%rax), %zmm19
+vpmovzxbd         %xmm16, %zmm19 {k1}
+vpmovzxbd         (%rax), %zmm19 {k1}
+vpmovzxbd         %xmm16, %zmm19 {z}{k1}
+vpmovzxbd         (%rax), %zmm19 {z}{k1}
+
+vpmovzxbq         %xmm16, %zmm19
+vpmovzxbq         (%rax), %zmm19
+vpmovzxbq         %xmm16, %zmm19 {k1}
+vpmovzxbq         (%rax), %zmm19 {k1}
+vpmovzxbq         %xmm16, %zmm19 {z}{k1}
+vpmovzxbq         (%rax), %zmm19 {z}{k1}
+
+vpmovzxdq         %ymm16, %zmm19
+vpmovzxdq         (%rax), %zmm19
+vpmovzxdq         %ymm16, %zmm19 {k1}
+vpmovzxdq         (%rax), %zmm19 {k1}
+vpmovzxdq         %ymm16, %zmm19 {z}{k1}
+vpmovzxdq         (%rax), %zmm19 {z}{k1}
+
+vpmovzxwd         %ymm16, %zmm19
+vpmovzxwd         (%rax), %zmm19
+vpmovzxwd         %ymm16, %zmm19 {k1}
+vpmovzxwd         (%rax), %zmm19 {k1}
+vpmovzxwd         %ymm16, %zmm19 {z}{k1}
+vpmovzxwd         (%rax), %zmm19 {z}{k1}
+
+vpmovzxwq         %xmm16, %zmm19
+vpmovzxwq         (%rax), %zmm19
+vpmovzxwq         %xmm16, %zmm19 {k1}
+vpmovzxwq         (%rax), %zmm19 {k1}
+vpmovzxwq         %xmm16, %zmm19 {z}{k1}
+vpmovzxwq         (%rax), %zmm19 {z}{k1}
+
+vpmulld           %zmm16, %zmm17, %zmm19
+vpmulld           (%rax), %zmm17, %zmm19
+vpmulld           (%rax){1to16}, %zmm17, %zmm19
+vpmulld           %zmm16, %zmm17, %zmm19 {k1}
+vpmulld           (%rax), %zmm17, %zmm19 {k1}
+vpmulld           (%rax){1to16}, %zmm17, %zmm19 {k1}
+vpmulld           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpmulld           (%rax), %zmm17, %zmm19 {z}{k1}
+vpmulld           (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vpermd            %zmm16, %zmm17, %zmm19
+vpermd            (%rax), %zmm17, %zmm19
+vpermd            (%rax){1to16}, %zmm17, %zmm19
+vpermd            %zmm16, %zmm17, %zmm19 {k1}
+vpermd            (%rax), %zmm17, %zmm19 {k1}
+vpermd            (%rax){1to16}, %zmm17, %zmm19 {k1}
+vpermd            %zmm16, %zmm17, %zmm19 {z}{k1}
+vpermd            (%rax), %zmm17, %zmm19 {z}{k1}
+vpermd            (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vpermilpd         $0, %zmm16, %zmm19
+vpermilpd         $0, (%rax), %zmm19
+vpermilpd         $0, (%rax){1to8}, %zmm19
+vpermilpd         $0, %zmm16, %zmm19 {k1}
+vpermilpd         $0, (%rax), %zmm19 {k1}
+vpermilpd         $0, (%rax){1to8}, %zmm19 {k1}
+vpermilpd         $0, %zmm16, %zmm19 {z}{k1}
+vpermilpd         $0, (%rax), %zmm19 {z}{k1}
+vpermilpd         $0, (%rax){1to8}, %zmm19 {z}{k1}
+
+vpermilpd         %zmm16, %zmm17, %zmm19
+vpermilpd         (%rax), %zmm17, %zmm19
+vpermilpd         (%rax){1to8}, %zmm17, %zmm19
+vpermilpd         %zmm16, %zmm17, %zmm19 {k1}
+vpermilpd         (%rax), %zmm17, %zmm19 {k1}
+vpermilpd         (%rax){1to8}, %zmm17, %zmm19 {k1}
+vpermilpd         %zmm16, %zmm17, %zmm19 {z}{k1}
+vpermilpd         (%rax), %zmm17, %zmm19 {z}{k1}
+vpermilpd         (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vpermilps         $0, %zmm16, %zmm19
+vpermilps         $0, (%rax), %zmm19
+vpermilps         $0, (%rax){1to16}, %zmm19
+vpermilps         $0, %zmm16, %zmm19 {k1}
+vpermilps         $0, (%rax), %zmm19 {k1}
+vpermilps         $0, (%rax){1to16}, %zmm19 {k1}
+vpermilps         $0, %zmm16, %zmm19 {z}{k1}
+vpermilps         $0, (%rax), %zmm19 {z}{k1}
+vpermilps         $0, (%rax){1to16}, %zmm19 {z}{k1}
+
+vpermilps         %zmm16, %zmm17, %zmm19
+vpermilps         (%rax), %zmm17, %zmm19
+vpermilps         (%rax){1to16}, %zmm17, %zmm19
+vpermilps         %zmm16, %zmm17, %zmm19 {k1}
+vpermilps         (%rax), %zmm17, %zmm19 {k1}
+vpermilps         (%rax){1to16}, %zmm17, %zmm19 {k1}
+vpermilps         %zmm16, %zmm17, %zmm19 {z}{k1}
+vpermilps         (%rax), %zmm17, %zmm19 {z}{k1}
+vpermilps         (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vpermpd           $0, %zmm16, %zmm19
+vpermpd           $0, (%rax), %zmm19
+vpermpd           $0, (%rax){1to8}, %zmm19
+vpermpd           $0, %zmm16, %zmm19 {k1}
+vpermpd           $0, (%rax), %zmm19 {k1}
+vpermpd           $0, (%rax){1to8}, %zmm19 {k1}
+vpermpd           $0, %zmm16, %zmm19 {z}{k1}
+vpermpd           $0, (%rax), %zmm19 {z}{k1}
+vpermpd           $0, (%rax){1to8}, %zmm19 {z}{k1}
+
+vpermpd           %zmm16, %zmm17, %zmm19
+vpermpd           (%rax), %zmm17, %zmm19
+vpermpd           (%rax){1to8}, %zmm17, %zmm19
+vpermpd           %zmm16, %zmm17, %zmm19 {k1}
+vpermpd           (%rax), %zmm17, %zmm19 {k1}
+vpermpd           (%rax){1to8}, %zmm17, %zmm19 {k1}
+vpermpd           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpermpd           (%rax), %zmm17, %zmm19 {z}{k1}
+vpermpd           (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vpermps           %zmm16, %zmm17, %zmm19
+vpermps           (%rax), %zmm17, %zmm19
+vpermps           (%rax){1to16}, %zmm17, %zmm19
+vpermps           %zmm16, %zmm17, %zmm19 {k1}
+vpermps           (%rax), %zmm17, %zmm19 {k1}
+vpermps           (%rax){1to16}, %zmm17, %zmm19 {k1}
+vpermps           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpermps           (%rax), %zmm17, %zmm19 {z}{k1}
+vpermps           (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vpermq            $0, %zmm16, %zmm19
+vpermq            $0, (%rax), %zmm19
+vpermq            $0, (%rax){1to8}, %zmm19
+vpermq            $0, %zmm16, %zmm19 {k1}
+vpermq            $0, (%rax), %zmm19 {k1}
+vpermq            $0, (%rax){1to8}, %zmm19 {k1}
+vpermq            $0, %zmm16, %zmm19 {z}{k1}
+vpermq            $0, (%rax), %zmm19 {z}{k1}
+vpermq            $0, (%rax){1to8}, %zmm19 {z}{k1}
+
+vpermq            %zmm16, %zmm17, %zmm19
+vpermq            (%rax), %zmm17, %zmm19
+vpermq            (%rax){1to8}, %zmm17, %zmm19
+vpermq            %zmm16, %zmm17, %zmm19 {k1}
+vpermq            (%rax), %zmm17, %zmm19 {k1}
+vpermq            (%rax){1to8}, %zmm17, %zmm19 {k1}
+vpermq            %zmm16, %zmm17, %zmm19 {z}{k1}
+vpermq            (%rax), %zmm17, %zmm19 {z}{k1}
+vpermq            (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vpshufd           $0, %zmm16, %zmm19
+vpshufd           $0, (%rax), %zmm19
+vpshufd           $0, (%rax){1to16}, %zmm19
+vpshufd           $0, %zmm16, %zmm19 {k1}
+vpshufd           $0, (%rax), %zmm19 {k1}
+vpshufd           $0, (%rax){1to16}, %zmm19 {k1}
+vpshufd           $0, %zmm16, %zmm19 {z}{k1}
+vpshufd           $0, (%rax), %zmm19 {z}{k1}
+vpshufd           $0, (%rax){1to16}, %zmm19 {z}{k1}
+
+vpsubd            %zmm16, %zmm17, %zmm19
+vpsubd            (%rax), %zmm17, %zmm19
+vpsubd            (%rax){1to16}, %zmm17, %zmm19
+vpsubd            %zmm16, %zmm17, %zmm19 {k1}
+vpsubd            (%rax), %zmm17, %zmm19 {k1}
+vpsubd            (%rax){1to16}, %zmm17, %zmm19 {k1}
+vpsubd            %zmm16, %zmm17, %zmm19 {z}{k1}
+vpsubd            (%rax), %zmm17, %zmm19 {z}{k1}
+vpsubd            (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vpsubq            %zmm16, %zmm17, %zmm19
+vpsubq            (%rax), %zmm17, %zmm19
+vpsubq            (%rax){1to8}, %zmm17, %zmm19
+vpsubq            %zmm16, %zmm17, %zmm19 {k1}
+vpsubq            (%rax), %zmm17, %zmm19 {k1}
+vpsubq            (%rax){1to8}, %zmm17, %zmm19 {k1}
+vpsubq            %zmm16, %zmm17, %zmm19 {z}{k1}
+vpsubq            (%rax), %zmm17, %zmm19 {z}{k1}
+vpsubq            (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vpunpckhdq        %zmm16, %zmm17, %zmm19
+vpunpckhdq        (%rax), %zmm17, %zmm19
+vpunpckhdq        (%rax){1to16}, %zmm17, %zmm19
+vpunpckhdq        %zmm16, %zmm17, %zmm19 {k1}
+vpunpckhdq        (%rax), %zmm17, %zmm19 {k1}
+vpunpckhdq        (%rax){1to16}, %zmm17, %zmm19 {k1}
+vpunpckhdq        %zmm16, %zmm17, %zmm19 {z}{k1}
+vpunpckhdq        (%rax), %zmm17, %zmm19 {z}{k1}
+vpunpckhdq        (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vpunpckhqdq       %zmm16, %zmm17, %zmm19
+vpunpckhqdq       (%rax), %zmm17, %zmm19
+vpunpckhqdq       (%rax){1to8}, %zmm17, %zmm19
+vpunpckhqdq       %zmm16, %zmm17, %zmm19 {k1}
+vpunpckhqdq       (%rax), %zmm17, %zmm19 {k1}
+vpunpckhqdq       (%rax){1to8}, %zmm17, %zmm19 {k1}
+vpunpckhqdq       %zmm16, %zmm17, %zmm19 {z}{k1}
+vpunpckhqdq       (%rax), %zmm17, %zmm19 {z}{k1}
+vpunpckhqdq       (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vpunpckldq        %zmm16, %zmm17, %zmm19
+vpunpckldq        (%rax), %zmm17, %zmm19
+vpunpckldq        (%rax){1to16}, %zmm17, %zmm19
+vpunpckldq        %zmm16, %zmm17, %zmm19 {k1}
+vpunpckldq        (%rax), %zmm17, %zmm19 {k1}
+vpunpckldq        (%rax){1to16}, %zmm17, %zmm19 {k1}
+vpunpckldq        %zmm16, %zmm17, %zmm19 {z}{k1}
+vpunpckldq        (%rax), %zmm17, %zmm19 {z}{k1}
+vpunpckldq        (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vpunpcklqdq       %zmm16, %zmm17, %zmm19
+vpunpcklqdq       (%rax), %zmm17, %zmm19
+vpunpcklqdq       (%rax){1to8}, %zmm17, %zmm19
+vpunpcklqdq       %zmm16, %zmm17, %zmm19 {k1}
+vpunpcklqdq       (%rax), %zmm17, %zmm19 {k1}
+vpunpcklqdq       (%rax){1to8}, %zmm17, %zmm19 {k1}
+vpunpcklqdq       %zmm16, %zmm17, %zmm19 {z}{k1}
+vpunpcklqdq       (%rax), %zmm17, %zmm19 {z}{k1}
+vpunpcklqdq       (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vshuff32x4        $0, %zmm16, %zmm17, %zmm19
+vshuff32x4        $0, (%rax), %zmm17, %zmm19
+vshuff32x4        $0, (%rax){1to16}, %zmm17, %zmm19
+vshuff32x4        $0, %zmm16, %zmm17, %zmm19 {k1}
+vshuff32x4        $0, (%rax), %zmm17, %zmm19 {k1}
+vshuff32x4        $0, (%rax){1to16}, %zmm17, %zmm19 {k1}
+vshuff32x4        $0, %zmm16, %zmm17, %zmm19 {z}{k1}
+vshuff32x4        $0, (%rax), %zmm17, %zmm19 {z}{k1}
+vshuff32x4        $0, (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vshuff64x2        $0, %zmm16, %zmm17, %zmm19
+vshuff64x2        $0, (%rax), %zmm17, %zmm19
+vshuff64x2        $0, (%rax){1to8}, %zmm17, %zmm19
+vshuff64x2        $0, %zmm16, %zmm17, %zmm19 {k1}
+vshuff64x2        $0, (%rax), %zmm17, %zmm19 {k1}
+vshuff64x2        $0, (%rax){1to8}, %zmm17, %zmm19 {k1}
+vshuff64x2        $0, %zmm16, %zmm17, %zmm19 {z}{k1}
+vshuff64x2        $0, (%rax), %zmm17, %zmm19 {z}{k1}
+vshuff64x2        $0, (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vshufi32x4        $0, %zmm16, %zmm17, %zmm19
+vshufi32x4        $0, (%rax), %zmm17, %zmm19
+vshufi32x4        $0, (%rax){1to16}, %zmm17, %zmm19
+vshufi32x4        $0, %zmm16, %zmm17, %zmm19 {k1}
+vshufi32x4        $0, (%rax), %zmm17, %zmm19 {k1}
+vshufi32x4        $0, (%rax){1to16}, %zmm17, %zmm19 {k1}
+vshufi32x4        $0, %zmm16, %zmm17, %zmm19 {z}{k1}
+vshufi32x4        $0, (%rax), %zmm17, %zmm19 {z}{k1}
+vshufi32x4        $0, (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vshufi64x2        $0, %zmm16, %zmm17, %zmm19
+vshufi64x2        $0, (%rax), %zmm17, %zmm19
+vshufi64x2        $0, (%rax){1to8}, %zmm17, %zmm19
+vshufi64x2        $0, %zmm16, %zmm17, %zmm19 {k1}
+vshufi64x2        $0, (%rax), %zmm17, %zmm19 {k1}
+vshufi64x2        $0, (%rax){1to8}, %zmm17, %zmm19 {k1}
+vshufi64x2        $0, %zmm16, %zmm17, %zmm19 {z}{k1}
+vshufi64x2        $0, (%rax), %zmm17, %zmm19 {z}{k1}
+vshufi64x2        $0, (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vsqrtpd           %zmm16, %zmm19
+vsqrtpd           (%rax), %zmm19
+vsqrtpd           (%rax){1to8}, %zmm19
+vsqrtpd           %zmm16, %zmm19 {k1}
+vsqrtpd           (%rax), %zmm19 {k1}
+vsqrtpd           (%rax){1to8}, %zmm19 {k1}
+vsqrtpd           %zmm16, %zmm19 {z}{k1}
+vsqrtpd           (%rax), %zmm19 {z}{k1}
+vsqrtpd           (%rax){1to8}, %zmm19 {z}{k1}
+
+vsqrtps           %zmm16, %zmm19
+vsqrtps           (%rax), %zmm19
+vsqrtps           (%rax){1to16}, %zmm19
+vsqrtps           %zmm16, %zmm19 {k1}
+vsqrtps           (%rax), %zmm19 {k1}
+vsqrtps           (%rax){1to16}, %zmm19 {k1}
+vsqrtps           %zmm16, %zmm19 {z}{k1}
+vsqrtps           (%rax), %zmm19 {z}{k1}
+vsqrtps           (%rax){1to16}, %zmm19 {z}{k1}
+
+vsqrtsd           %xmm16, %xmm17, %xmm19
+vsqrtsd           (%rax), %xmm17, %xmm19
+vsqrtsd           %xmm16, %xmm17, %xmm19 {k1}
+vsqrtsd           (%rax), %xmm17, %xmm19 {k1}
+vsqrtsd           %xmm16, %xmm17, %xmm19 {z}{k1}
+vsqrtsd           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vsqrtss           %xmm16, %xmm17, %xmm19
+vsqrtss           (%rax), %xmm17, %xmm19
+vsqrtss           %xmm16, %xmm17, %xmm19 {k1}
+vsqrtss           (%rax), %xmm17, %xmm19 {k1}
+vsqrtss           %xmm16, %xmm17, %xmm19 {z}{k1}
+vsqrtss           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpsubd            %zmm16, %zmm17, %zmm19
+vpsubd            (%rax), %zmm17, %zmm19
+vpsubd            (%rax){1to16}, %zmm17, %zmm19
+vpsubd            %zmm16, %zmm17, %zmm19 {k1}
+vpsubd            (%rax), %zmm17, %zmm19 {k1}
+vpsubd            (%rax){1to16}, %zmm17, %zmm19 {k1}
+vpsubd            %zmm16, %zmm17, %zmm19 {z}{k1}
+vpsubd            (%rax), %zmm17, %zmm19 {z}{k1}
+vpsubd            (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vpsubq            %zmm16, %zmm17, %zmm19
+vpsubq            (%rax), %zmm17, %zmm19
+vpsubq            (%rax){1to8}, %zmm17, %zmm19
+vpsubq            %zmm16, %zmm17, %zmm19 {k1}
+vpsubq            (%rax), %zmm17, %zmm19 {k1}
+vpsubq            (%rax){1to8}, %zmm17, %zmm19 {k1}
+vpsubq            %zmm16, %zmm17, %zmm19 {z}{k1}
+vpsubq            (%rax), %zmm17, %zmm19 {z}{k1}
+vpsubq            (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vptestmd          %zmm0, %zmm1, %k2
+vptestmd          (%rax), %zmm1, %k2
+vptestmd          (%rax){1to16}, %zmm1, %k2
+vptestmd          %zmm0, %zmm1, %k2 {k3}
+vptestmd          (%rax), %zmm1, %k2 {k3}
+vptestmd          (%rax){1to16}, %zmm1, %k2 {k3}
+
+vptestmq          %zmm0, %zmm1, %k2
+vptestmq          (%rax), %zmm1, %k2
+vptestmq          (%rax){1to8}, %zmm1, %k2
+vptestmq          %zmm0, %zmm1, %k2 {k3}
+vptestmq          (%rax), %zmm1, %k2 {k3}
+vptestmq          (%rax){1to8}, %zmm1, %k2 {k3}
+
+vptestnmd         %zmm0, %zmm1, %k2
+vptestnmd         (%rax), %zmm1, %k2
+vptestnmd         (%rax){1to16}, %zmm1, %k2
+vptestnmd         %zmm0, %zmm1, %k2 {k3}
+vptestnmd         (%rax), %zmm1, %k2 {k3}
+vptestnmd         (%rax){1to16}, %zmm1, %k2 {k3}
+
+vptestnmq         %zmm0, %zmm1, %k2
+vptestnmq         (%rax), %zmm1, %k2
+vptestnmq         (%rax){1to8}, %zmm1, %k2
+vptestnmq         %zmm0, %zmm1, %k2 {k3}
+vptestnmq         (%rax), %zmm1, %k2 {k3}
+vptestnmq         (%rax){1to8}, %zmm1, %k2 {k3}
+
+vsubpd            %zmm16, %zmm17, %zmm19
+vsubpd            (%rax), %zmm17, %zmm19
+vsubpd            (%rax){1to8}, %zmm17, %zmm19
+vsubpd            %zmm16, %zmm17, %zmm19 {k1}
+vsubpd            (%rax), %zmm17, %zmm19 {k1}
+vsubpd            (%rax){1to8}, %zmm17, %zmm19 {k1}
+vsubpd            %zmm16, %zmm17, %zmm19 {z}{k1}
+vsubpd            (%rax), %zmm17, %zmm19 {z}{k1}
+vsubpd            (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vsubps            %zmm16, %zmm17, %zmm19
+vsubps            (%rax), %zmm17, %zmm19
+vsubps            (%rax){1to16}, %zmm17, %zmm19
+vsubps            %zmm16, %zmm17, %zmm19 {k1}
+vsubps            (%rax), %zmm17, %zmm19 {k1}
+vsubps            (%rax){1to16}, %zmm17, %zmm19 {k1}
+vsubps            %zmm16, %zmm17, %zmm19 {z}{k1}
+vsubps            (%rax), %zmm17, %zmm19 {z}{k1}
+vsubps            (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vucomiss          %xmm16, %xmm17
+vucomiss          (%rax), %xmm17
+
+vunpckhpd         %zmm16, %zmm17, %zmm19
+vunpckhpd         (%rax), %zmm17, %zmm19
+vunpckhpd         (%rax){1to8}, %zmm17, %zmm19
+vunpckhpd         %zmm16, %zmm17, %zmm19 {k1}
+vunpckhpd         (%rax), %zmm17, %zmm19 {k1}
+vunpckhpd         (%rax){1to8}, %zmm17, %zmm19 {k1}
+vunpckhpd         %zmm16, %zmm17, %zmm19 {z}{k1}
+vunpckhpd         (%rax), %zmm17, %zmm19 {z}{k1}
+vunpckhpd         (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vunpckhps         %zmm16, %zmm17, %zmm19
+vunpckhps         (%rax), %zmm17, %zmm19
+vunpckhps         (%rax){1to16}, %zmm17, %zmm19
+vunpckhps         %zmm16, %zmm17, %zmm19 {k1}
+vunpckhps         (%rax), %zmm17, %zmm19 {k1}
+vunpckhps         (%rax){1to16}, %zmm17, %zmm19 {k1}
+vunpckhps         %zmm16, %zmm17, %zmm19 {z}{k1}
+vunpckhps         (%rax), %zmm17, %zmm19 {z}{k1}
+vunpckhps         (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vunpcklpd         %zmm16, %zmm17, %zmm19
+vunpcklpd         (%rax), %zmm17, %zmm19
+vunpcklpd         (%rax){1to8}, %zmm17, %zmm19
+vunpcklpd         %zmm16, %zmm17, %zmm19 {k1}
+vunpcklpd         (%rax), %zmm17, %zmm19 {k1}
+vunpcklpd         (%rax){1to8}, %zmm17, %zmm19 {k1}
+vunpcklpd         %zmm16, %zmm17, %zmm19 {z}{k1}
+vunpcklpd         (%rax), %zmm17, %zmm19 {z}{k1}
+vunpcklpd         (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vunpcklps         %zmm16, %zmm17, %zmm19
+vunpcklps         (%rax), %zmm17, %zmm19
+vunpcklps         (%rax){1to16}, %zmm17, %zmm19
+vunpcklps         %zmm16, %zmm17, %zmm19 {k1}
+vunpcklps         (%rax), %zmm17, %zmm19 {k1}
+vunpcklps         (%rax){1to16}, %zmm17, %zmm19 {k1}
+vunpcklps         %zmm16, %zmm17, %zmm19 {z}{k1}
+vunpcklps         (%rax), %zmm17, %zmm19 {z}{k1}
+vunpcklps         (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     0.50                        kandw	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kandnw	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        knotw	%k0, %k2
+# CHECK-NEXT:  1      1     0.50                        korw	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kxnorw	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kxorw	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kshiftlw	$2, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kshiftrw	$2, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kunpckbw	%k0, %k1, %k2
+# CHECK-NEXT:  1      3     1.00                        vaddpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vaddpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vaddpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      3     1.00                        vaddpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vaddpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vaddpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vaddpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vaddpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vaddpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vaddps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vaddps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vaddps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      3     1.00                        vaddps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vaddps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vaddps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vaddps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vaddps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vaddps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     0.50                        valignd	$1, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   valignd	$1, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   valignd	$1, (%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     1.00                        valignd	$1, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   valignd	$1, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   valignd	$1, (%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        valignd	$1, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   valignd	$1, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   valignd	$1, (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     0.50                        valignq	$1, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   valignq	$1, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   valignq	$1, (%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     1.00                        valignq	$1, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   valignq	$1, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   valignq	$1, (%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        valignq	$1, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   valignq	$1, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   valignq	$1, (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf32x4	(%rax), %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf32x4	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf32x4	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf64x4	(%rax), %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf64x4	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf64x4	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti32x4	(%rax), %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti32x4	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti32x4	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti64x4	(%rax), %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti64x4	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti64x4	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vbroadcastsd	%xmm16, %zmm19
+# CHECK-NEXT:  3      9     1.00    *                   vbroadcastsd	(%rax), %zmm19
+# CHECK-NEXT:  1      2     1.00                        vbroadcastsd	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vbroadcastsd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vbroadcastsd	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vbroadcastsd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vbroadcastss	%xmm16, %zmm19
+# CHECK-NEXT:  3      9     1.00    *                   vbroadcastss	(%rax), %zmm19
+# CHECK-NEXT:  1      2     1.00                        vbroadcastss	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vbroadcastss	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vbroadcastss	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vbroadcastss	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     2.00                        vcmpeqpd	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      9     2.00    *                   vcmpeqpd	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      9     2.00    *                   vcmpeqpd	(%rax){1to8}, %zmm1, %k2
+# CHECK-NEXT:  1      2     2.00                        vcmpeqpd	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      9     2.00    *                   vcmpeqpd	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      9     2.00    *                   vcmpeqpd	(%rax){1to8}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      2     2.00                        vcmpeqps	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      9     2.00    *                   vcmpeqps	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      9     2.00    *                   vcmpeqps	(%rax){1to16}, %zmm1, %k2
+# CHECK-NEXT:  1      2     2.00                        vcmpeqps	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      9     2.00    *                   vcmpeqps	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      9     2.00    *                   vcmpeqps	(%rax){1to16}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      2     1.00                        vcmpeqsd	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      9     1.00    *                   vcmpeqsd	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      2     1.00                        vcmpeqsd	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      9     1.00    *                   vcmpeqsd	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      2     1.00                        vcmpeqss	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      9     1.00    *                   vcmpeqss	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      2     1.00                        vcmpeqss	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      9     1.00    *                   vcmpeqss	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  2      4     1.00                        vcomiss	%xmm16, %xmm17
+# CHECK-NEXT:  2      11    1.00    *                   vcomiss	(%rax), %xmm17
+# CHECK-NEXT:  4      4     2.00                        vcvtdq2pd	%ymm16, %zmm19
+# CHECK-NEXT:  3      11    2.00    *                   vcvtdq2pd	(%rax), %zmm19
+# CHECK-NEXT:  3      11    2.00    *                   vcvtdq2pd	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  4      4     2.00                        vcvtdq2pd	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  3      11    2.00    *                   vcvtdq2pd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  3      11    2.00    *                   vcvtdq2pd	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  4      4     2.00                        vcvtdq2pd	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      11    2.00    *                   vcvtdq2pd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      11    2.00    *                   vcvtdq2pd	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      3     1.00                        vcvtdq2ps	%zmm16, %zmm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvtdq2ps	(%rax), %zmm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvtdq2ps	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  2      3     1.00                        vcvtdq2ps	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtdq2ps	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtdq2ps	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  2      3     1.00                        vcvtdq2ps	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtdq2ps	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtdq2ps	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      3     2.00                        vcvtpd2dq	%zmm16, %ymm19
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2dq	(%rax), %ymm19
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2dq	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  2      3     2.00                        vcvtpd2dq	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2dq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2dq	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  2      3     2.00                        vcvtpd2dq	%zmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2dq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2dq	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      3     2.00                        vcvtpd2udq	%zmm16, %ymm19
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2udq	(%rax), %ymm19
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2udq	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  2      3     2.00                        vcvtpd2udq	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2udq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2udq	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  2      3     2.00                        vcvtpd2udq	%zmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2udq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2udq	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      3     2.00                        vcvttpd2dq	%zmm16, %ymm19
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2dq	(%rax), %ymm19
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2dq	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  2      3     2.00                        vcvttpd2dq	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2dq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2dq	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  2      3     2.00                        vcvttpd2dq	%zmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2dq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2dq	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      3     2.00                        vcvttpd2udq	%zmm16, %ymm19
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2udq	(%rax), %ymm19
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2udq	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  2      3     2.00                        vcvttpd2udq	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2udq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2udq	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  2      3     2.00                        vcvttpd2udq	%zmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2udq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2udq	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  4      6     2.00                        vcvtpd2ps	%zmm16, %ymm19
+# CHECK-NEXT:  4      13    2.00    *                   vcvtpd2ps	(%rax), %ymm19
+# CHECK-NEXT:  4      13    2.00    *                   vcvtpd2ps	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  4      6     2.00                        vcvtpd2ps	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  4      13    2.00    *                   vcvtpd2ps	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  4      13    2.00    *                   vcvtpd2ps	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  4      6     2.00                        vcvtpd2ps	%zmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  4      13    2.00    *                   vcvtpd2ps	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  4      13    2.00    *                   vcvtpd2ps	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      4     1.00                        vcvtps2dq	%zmm16, %zmm19
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2dq	(%rax), %zmm19
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2dq	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  2      4     1.00                        vcvtps2dq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2dq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2dq	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  2      4     1.00                        vcvtps2dq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2dq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2dq	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      4     1.00                        vcvttps2dq	%zmm16, %zmm19
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2dq	(%rax), %zmm19
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2dq	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  2      4     1.00                        vcvttps2dq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2dq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2dq	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  2      4     1.00                        vcvttps2dq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2dq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2dq	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  4      6     2.00                        vcvtps2pd	%ymm16, %zmm19
+# CHECK-NEXT:  3      13    2.00    *                   vcvtps2pd	(%rax), %zmm19
+# CHECK-NEXT:  3      13    2.00    *                   vcvtps2pd	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  4      6     2.00                        vcvtps2pd	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  3      13    2.00    *                   vcvtps2pd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  3      13    2.00    *                   vcvtps2pd	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  4      6     2.00                        vcvtps2pd	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      13    2.00    *                   vcvtps2pd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      13    2.00    *                   vcvtps2pd	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vcvtsd2usi	%xmm0, %ecx
+# CHECK-NEXT:  1      1     0.50                        vcvtsd2usi	%xmm0, %rcx
+# CHECK-NEXT:  1      8     0.50    *                   vcvtsd2usi	(%rax), %ecx
+# CHECK-NEXT:  1      8     0.50    *                   vcvtsd2usi	(%rax), %rcx
+# CHECK-NEXT:  2      5     2.50                        vcvtss2usi	%xmm0, %ecx
+# CHECK-NEXT:  2      5     2.50                        vcvtss2usi	%xmm0, %rcx
+# CHECK-NEXT:  2      12    2.50    *                   vcvtss2usi	(%rax), %ecx
+# CHECK-NEXT:  2      12    2.50    *                   vcvtss2usi	(%rax), %rcx
+# CHECK-NEXT:  2      4     1.00                        vcvtps2udq	%zmm16, %zmm19
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2udq	(%rax), %zmm19
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2udq	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  2      4     1.00                        vcvtps2udq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2udq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2udq	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  2      4     1.00                        vcvtps2udq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2udq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2udq	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      4     1.00                        vcvttps2udq	%zmm16, %zmm19
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2udq	(%rax), %zmm19
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2udq	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  2      4     1.00                        vcvttps2udq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2udq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2udq	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  2      4     1.00                        vcvttps2udq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2udq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2udq	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vcvttsd2usi	%xmm0, %ecx
+# CHECK-NEXT:  1      1     0.50                        vcvttsd2usi	%xmm0, %rcx
+# CHECK-NEXT:  1      8     0.50    *                   vcvttsd2usi	(%rax), %ecx
+# CHECK-NEXT:  1      8     0.50    *                   vcvttsd2usi	(%rax), %rcx
+# CHECK-NEXT:  2      5     2.50                        vcvttss2usi	%xmm0, %ecx
+# CHECK-NEXT:  2      5     2.50                        vcvttss2usi	%xmm0, %rcx
+# CHECK-NEXT:  2      12    2.50    *                   vcvttss2usi	(%rax), %ecx
+# CHECK-NEXT:  2      12    2.50    *                   vcvttss2usi	(%rax), %rcx
+# CHECK-NEXT:  4      4     2.00                        vcvtudq2pd	%ymm16, %zmm19
+# CHECK-NEXT:  3      11    2.00    *                   vcvtudq2pd	(%rax), %zmm19
+# CHECK-NEXT:  3      11    2.00    *                   vcvtudq2pd	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  4      4     2.00                        vcvtudq2pd	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  3      11    2.00    *                   vcvtudq2pd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  3      11    2.00    *                   vcvtudq2pd	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  4      4     2.00                        vcvtudq2pd	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      11    2.00    *                   vcvtudq2pd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      11    2.00    *                   vcvtudq2pd	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      3     1.00                        vcvtudq2ps	%zmm16, %zmm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvtudq2ps	(%rax), %zmm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvtudq2ps	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  2      3     1.00                        vcvtudq2ps	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtudq2ps	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtudq2ps	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  2      3     1.00                        vcvtudq2ps	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtudq2ps	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtudq2ps	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      5     1.00                        vcvtusi2sd	%ecx, %xmm0, %xmm2
+# CHECK-NEXT:  2      5     1.00                        vcvtusi2sd	%rcx, %xmm0, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vcvtusi2sdl	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vcvtusi2sdq	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  2      4     1.00                        vcvtusi2ss	%ecx, %xmm0, %xmm2
+# CHECK-NEXT:  2      4     1.00                        vcvtusi2ss	%rcx, %xmm0, %xmm2
+# CHECK-NEXT:  1      10    1.00    *                   vcvtusi2ssl	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  1      10    1.00    *                   vcvtusi2ssq	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  1      13    10.00                       vdivpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      20    10.00   *                   vdivpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      20    10.00   *                   vdivpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      13    10.00                       vdivpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      20    10.00   *                   vdivpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      20    10.00   *                   vdivpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      13    10.00                       vdivpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      20    10.00   *                   vdivpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      20    10.00   *                   vdivpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      11    6.00                        vdivps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      18    6.00    *                   vdivps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      18    6.00    *                   vdivps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      11    6.00                        vdivps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      18    6.00    *                   vdivps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      18    6.00    *                   vdivps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      11    6.00                        vdivps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      18    6.00    *                   vdivps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      18    6.00    *                   vdivps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      5     0.33    *                   vgatherdpd	(%rax,%ymm1,2), %zmm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vgatherdps	(%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vgatherqpd	(%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vgatherqps	(%rax,%zmm1,2), %ymm2 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vmaxpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      9     2.00    *                   vmaxpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      9     2.00    *                   vmaxpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vmaxpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      9     2.00    *                   vmaxpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      9     2.00    *                   vmaxpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vmaxpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      9     2.00    *                   vmaxpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      9     2.00    *                   vmaxpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     2.00                        vmaxps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      9     2.00    *                   vmaxps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      9     2.00    *                   vmaxps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     2.00                        vmaxps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      9     2.00    *                   vmaxps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      9     2.00    *                   vmaxps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     2.00                        vmaxps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      9     2.00    *                   vmaxps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      9     2.00    *                   vmaxps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vminpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      9     2.00    *                   vminpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      9     2.00    *                   vminpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vminpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      9     2.00    *                   vminpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      9     2.00    *                   vminpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vminpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      9     2.00    *                   vminpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      9     2.00    *                   vminpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     2.00                        vminps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      9     2.00    *                   vminps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      9     2.00    *                   vminps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     2.00                        vminps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      9     2.00    *                   vminps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      9     2.00    *                   vminps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     2.00                        vminps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      9     2.00    *                   vminps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      9     2.00    *                   vminps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovapd	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovapd	(%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovapd	%zmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovapd	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovapd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovapd	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovapd	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovapd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovaps	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovaps	(%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovaps	%zmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovaps	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovaps	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovaps	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovaps	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovaps	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vmovddup	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vmovddup	(%rax), %zmm19
+# CHECK-NEXT:  1      2     1.00                        vmovddup	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vmovddup	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vmovddup	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vmovddup	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovdqa32	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa32	(%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovdqa32	%zmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovdqa32	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa32	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovdqa32	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovdqa32	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa32	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovdqa64	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa64	(%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovdqa64	%zmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovdqa64	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa64	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovdqa64	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovdqa64	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa64	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu32	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu32	(%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu32	%zmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovdqu32	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu32	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu32	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu32	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu32	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu64	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu64	(%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu64	%zmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovdqu64	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu64	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu64	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu64	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu64	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vmovshdup	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vmovshdup	(%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00                        vmovshdup	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vmovshdup	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vmovshdup	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vmovshdup	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vmovsldup	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vmovsldup	(%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00                        vmovsldup	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vmovsldup	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vmovsldup	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vmovsldup	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovupd	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovupd	(%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovupd	%zmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovupd	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovupd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovupd	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovupd	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovupd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovups	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovups	(%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovups	%zmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovups	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovups	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovups	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovups	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovups	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vmulpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vmulpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vmulpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      3     1.00                        vmulpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vmulpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vmulpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vmulpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vmulpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vmulpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vmulps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vmulps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vmulps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      3     1.00                        vmulps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vmulps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vmulps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vmulps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vmulps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vmulps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpabsd	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpabsd	(%rax), %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpabsd	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpabsd	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsd	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpabsd	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsd	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpabsq	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpabsq	(%rax), %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpabsq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpabsq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpabsq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpaddd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpaddd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpaddd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpaddq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpaddq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpaddq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastd	%xmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastd	(%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastd	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastd	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastq	%xmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastq	(%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastq	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastq	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpcmpeqd	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax){1to16}, %zmm1, %k2
+# CHECK-NEXT:  1      1     0.50                        vpcmpeqd	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax){1to16}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vpcmpeqd	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax){1to16}, %zmm1, %k2
+# CHECK-NEXT:  1      1     0.50                        vpcmpeqd	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax){1to16}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vpcmpeqq	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax){1to8}, %zmm1, %k2
+# CHECK-NEXT:  1      1     0.50                        vpcmpeqq	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax){1to8}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vpcmpgtd	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtd	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtd	(%rax){1to16}, %zmm1, %k2
+# CHECK-NEXT:  1      1     0.50                        vpcmpgtd	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtd	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtd	(%rax){1to16}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vpcmpgtq	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtq	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtq	(%rax){1to8}, %zmm1, %k2
+# CHECK-NEXT:  1      1     0.50                        vpcmpgtq	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtq	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtq	(%rax){1to8}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vpcmpeqq	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax){1to8}, %zmm1, %k2
+# CHECK-NEXT:  1      1     0.50                        vpcmpeqq	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax){1to8}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vpcmpequd	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequd	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequd	(%rax){1to16}, %zmm1, %k2
+# CHECK-NEXT:  1      1     0.50                        vpcmpequd	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequd	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequd	(%rax){1to16}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vpcmpequq	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequq	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequq	(%rax){1to8}, %zmm1, %k2
+# CHECK-NEXT:  1      1     0.50                        vpcmpequq	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequq	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequq	(%rax){1to8}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherdq	(%rax,%ymm1,2), %zmm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherdd	(%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherqq	(%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherqd	(%rax,%zmm1,2), %ymm2 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmovsxbd	%xmm16, %zmm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbd	(%rax), %zmm19
+# CHECK-NEXT:  1      4     2.00                        vpmovsxbd	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmovsxbd	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpmovsxbq	%xmm16, %zmm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbq	(%rax), %zmm19
+# CHECK-NEXT:  1      4     2.00                        vpmovsxbq	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmovsxbq	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpmovsxdq	%ymm16, %zmm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxdq	(%rax), %zmm19
+# CHECK-NEXT:  1      4     2.00                        vpmovsxdq	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxdq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmovsxdq	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxdq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpmovsxwd	%ymm16, %zmm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxwd	(%rax), %zmm19
+# CHECK-NEXT:  1      4     2.00                        vpmovsxwd	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxwd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmovsxwd	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxwd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpmovsxwq	%xmm16, %zmm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxwq	(%rax), %zmm19
+# CHECK-NEXT:  1      4     2.00                        vpmovsxwq	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxwq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmovsxwq	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxwq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpmovzxbd	%xmm16, %zmm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbd	(%rax), %zmm19
+# CHECK-NEXT:  1      4     2.00                        vpmovzxbd	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmovzxbd	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpmovzxbq	%xmm16, %zmm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbq	(%rax), %zmm19
+# CHECK-NEXT:  1      4     2.00                        vpmovzxbq	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmovzxbq	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpmovzxdq	%ymm16, %zmm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxdq	(%rax), %zmm19
+# CHECK-NEXT:  1      4     2.00                        vpmovzxdq	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxdq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmovzxdq	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxdq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpmovzxwd	%ymm16, %zmm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxwd	(%rax), %zmm19
+# CHECK-NEXT:  1      4     2.00                        vpmovzxwd	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxwd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmovzxwd	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxwd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpmovzxwq	%xmm16, %zmm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxwq	(%rax), %zmm19
+# CHECK-NEXT:  1      4     2.00                        vpmovzxwq	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxwq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmovzxwq	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxwq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vpmulld	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpmulld	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpmulld	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      3     1.00                        vpmulld	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpmulld	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpmulld	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vpmulld	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpmulld	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpmulld	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      5     1.00                        vpermd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermd	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      5     1.00                        vpermd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermd	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      5     1.00                        vpermd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermd	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpermilpd	$0, %zmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpermilpd	$0, (%rax), %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpermilpd	$0, (%rax){1to8}, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpermilpd	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpermilpd	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpermilpd	$0, (%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpermilpd	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpermilpd	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpermilpd	$0, (%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     0.67                        vpermilpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpermilpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpermilpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     0.67                        vpermilpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpermilpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpermilpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     0.67                        vpermilpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpermilpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpermilpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpermilps	$0, %zmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpermilps	$0, (%rax), %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpermilps	$0, (%rax){1to16}, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpermilps	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpermilps	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpermilps	$0, (%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpermilps	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpermilps	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpermilps	$0, (%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     0.67                        vpermilps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpermilps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpermilps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     0.67                        vpermilps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpermilps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpermilps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     0.67                        vpermilps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpermilps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpermilps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpermpd	$0, %zmm16, %zmm19
+# CHECK-NEXT:  3      9     1.00    *                   vpermpd	$0, (%rax), %zmm19
+# CHECK-NEXT:  3      9     1.00    *                   vpermpd	$0, (%rax){1to8}, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpermpd	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vpermpd	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vpermpd	$0, (%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpermpd	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vpermpd	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vpermpd	$0, (%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      5     1.00                        vpermpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  3      14    1.00    *                   vpermpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  3      14    1.00    *                   vpermpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      5     1.00                        vpermpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  3      14    1.00    *                   vpermpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  3      14    1.00    *                   vpermpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      5     1.00                        vpermpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      14    1.00    *                   vpermpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      14    1.00    *                   vpermpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      5     1.00                        vpermps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  3      14    1.00    *                   vpermps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  3      14    1.00    *                   vpermps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      5     1.00                        vpermps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  3      14    1.00    *                   vpermps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  3      14    1.00    *                   vpermps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      5     1.00                        vpermps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      14    1.00    *                   vpermps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      14    1.00    *                   vpermps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpermq	$0, %zmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpermq	$0, (%rax), %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpermq	$0, (%rax){1to8}, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpermq	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpermq	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpermq	$0, (%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpermq	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpermq	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpermq	$0, (%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      1     0.50                        vpermq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermq	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  2      1     0.50                        vpermq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermq	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  2      1     0.50                        vpermq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermq	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpshufd	$0, %zmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpshufd	$0, (%rax), %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpshufd	$0, (%rax){1to16}, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpshufd	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpshufd	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpshufd	$0, (%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpshufd	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpshufd	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpshufd	$0, (%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsubd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpsubd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsubd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsubq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpsubq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsubq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpunpckhdq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckhdq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckhdq	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpunpckhdq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckhdq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckhdq	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpunpckhdq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckhdq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckhdq	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpunpckhqdq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckhqdq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckhqdq	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpunpckhqdq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckhqdq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckhqdq	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpunpckhqdq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckhqdq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckhqdq	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpunpckldq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckldq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckldq	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpunpckldq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckldq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckldq	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpunpckldq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckldq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckldq	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpunpcklqdq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpunpcklqdq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpunpcklqdq	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpunpcklqdq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpcklqdq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpcklqdq	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpunpcklqdq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpcklqdq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpcklqdq	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vshuff32x4	$0, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  3      9     1.00    *                   vshuff32x4	$0, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  3      9     1.00    *                   vshuff32x4	$0, (%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vshuff32x4	$0, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vshuff32x4	$0, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vshuff32x4	$0, (%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vshuff32x4	$0, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vshuff32x4	$0, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vshuff32x4	$0, (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vshuff64x2	$0, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  3      9     1.00    *                   vshuff64x2	$0, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  3      9     1.00    *                   vshuff64x2	$0, (%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vshuff64x2	$0, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vshuff64x2	$0, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vshuff64x2	$0, (%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vshuff64x2	$0, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vshuff64x2	$0, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vshuff64x2	$0, (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vshufi32x4	$0, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  3      9     1.00    *                   vshufi32x4	$0, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  3      9     1.00    *                   vshufi32x4	$0, (%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vshufi32x4	$0, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vshufi32x4	$0, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vshufi32x4	$0, (%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vshufi32x4	$0, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vshufi32x4	$0, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vshufi32x4	$0, (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vshufi64x2	$0, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  3      9     1.00    *                   vshufi64x2	$0, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  3      9     1.00    *                   vshufi64x2	$0, (%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vshufi64x2	$0, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vshufi64x2	$0, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vshufi64x2	$0, (%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vshufi64x2	$0, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vshufi64x2	$0, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vshufi64x2	$0, (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      21    18.00                       vsqrtpd	%zmm16, %zmm19
+# CHECK-NEXT:  1      28    18.00   *                   vsqrtpd	(%rax), %zmm19
+# CHECK-NEXT:  1      28    18.00   *                   vsqrtpd	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  1      21    18.00                       vsqrtpd	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      28    18.00   *                   vsqrtpd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      28    18.00   *                   vsqrtpd	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  1      21    18.00                       vsqrtpd	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      28    18.00   *                   vsqrtpd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      28    18.00   *                   vsqrtpd	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      15    10.00                       vsqrtps	%zmm16, %zmm19
+# CHECK-NEXT:  1      22    10.00   *                   vsqrtps	(%rax), %zmm19
+# CHECK-NEXT:  1      22    10.00   *                   vsqrtps	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  1      15    10.00                       vsqrtps	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      22    10.00   *                   vsqrtps	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      22    10.00   *                   vsqrtps	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  1      15    10.00                       vsqrtps	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      22    10.00   *                   vsqrtps	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      22    10.00   *                   vsqrtps	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      21    9.00                        vsqrtsd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      28    9.00    *                   vsqrtsd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      21    9.00                        vsqrtsd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      28    9.00    *                   vsqrtsd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      21    9.00                        vsqrtsd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      28    9.00    *                   vsqrtsd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      15    5.00                        vsqrtss	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      22    5.00    *                   vsqrtss	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      15    5.00                        vsqrtss	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      22    5.00    *                   vsqrtss	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      15    5.00                        vsqrtss	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      22    5.00    *                   vsqrtss	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsubd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpsubd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsubd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsubq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpsubq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsubq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vptestmd	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestmd	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestmd	(%rax){1to16}, %zmm1, %k2
+# CHECK-NEXT:  1      5     2.50                        vptestmd	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestmd	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestmd	(%rax){1to16}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vptestmq	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestmq	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestmq	(%rax){1to8}, %zmm1, %k2
+# CHECK-NEXT:  1      5     2.50                        vptestmq	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestmq	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestmq	(%rax){1to8}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vptestnmd	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmd	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmd	(%rax){1to16}, %zmm1, %k2
+# CHECK-NEXT:  1      5     2.50                        vptestnmd	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmd	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmd	(%rax){1to16}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vptestnmq	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmq	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmq	(%rax){1to8}, %zmm1, %k2
+# CHECK-NEXT:  1      5     2.50                        vptestnmq	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmq	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmq	(%rax){1to8}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      3     1.00                        vsubpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vsubpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vsubpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      3     1.00                        vsubpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vsubpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vsubpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vsubpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vsubpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vsubpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vsubps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vsubps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vsubps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      3     1.00                        vsubps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vsubps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vsubps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vsubps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vsubps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vsubps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      4     1.00                        vucomiss	%xmm16, %xmm17
+# CHECK-NEXT:  2      11    1.00    *                   vucomiss	(%rax), %xmm17
+# CHECK-NEXT:  1      1     1.00                        vunpckhpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vunpckhpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vunpckhpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vunpckhpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vunpckhpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vunpckhpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vunpckhpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vunpckhpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vunpckhpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vunpckhps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vunpckhps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vunpckhps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vunpckhps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vunpckhps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vunpckhps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vunpckhps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vunpckhps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vunpckhps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vunpcklpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vunpcklpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vunpcklpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vunpcklpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vunpcklpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vunpcklpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vunpcklpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vunpcklpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vunpcklpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vunpcklps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vunpcklps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vunpcklps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vunpcklps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vunpcklps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vunpcklps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vunpcklps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vunpcklps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vunpcklps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 2.67   2.67   2.67    -      -      -      -      -     209.00 1048.50 618.00 352.50 277.50 277.50 16.00 187.67 187.67 187.67 182.33 182.33 182.33 8.00   8.00
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kandw	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kandnw	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     knotw	%k0, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     korw	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kxnorw	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kxorw	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     kshiftlw	$2, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     kshiftrw	$2, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kunpckbw	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vaddpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vaddpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vaddpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vaddps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vaddps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vaddps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     valignd	$1, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignd	$1, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignd	$1, (%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     valignd	$1, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignd	$1, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignd	$1, (%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     valignd	$1, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignd	$1, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignd	$1, (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     valignq	$1, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignq	$1, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignq	$1, (%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     valignq	$1, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignq	$1, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignq	$1, (%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     valignq	$1, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignq	$1, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignq	$1, (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf32x4	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf32x4	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf32x4	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf64x4	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf64x4	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf64x4	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti32x4	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti32x4	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti32x4	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti64x4	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti64x4	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti64x4	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastsd	%xmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastsd	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastsd	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastsd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastsd	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastsd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastss	%xmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastss	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastss	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastss	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastss	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastss	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqpd	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqpd	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqpd	(%rax){1to8}, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqpd	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqpd	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqpd	(%rax){1to8}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqps	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqps	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqps	(%rax){1to16}, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqps	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqps	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqps	(%rax){1to16}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqsd	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqsd	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqsd	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqsd	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqss	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqss	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqss	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqss	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcomiss	%xmm16, %xmm17
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcomiss	(%rax), %xmm17
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2pd	%ymm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2pd	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2pd	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2ps	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2ps	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2ps	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2dq	%zmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dq	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2dq	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dq	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2dq	%zmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dq	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2udq	%zmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2udq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2udq	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2udq	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2udq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2udq	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2udq	%zmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2udq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2udq	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2dq	%zmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dq	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2dq	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dq	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2dq	%zmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dq	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2udq	%zmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2udq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2udq	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2udq	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2udq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2udq	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2udq	%zmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2udq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2udq	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2ps	%zmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2ps	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2ps	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2ps	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2ps	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2ps	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2ps	%zmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2ps	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2ps	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2dq	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2dq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2dq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttps2dq	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttps2dq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttps2dq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2pd	%ymm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2pd	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2pd	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtsd2usi	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtsd2usi	%xmm0, %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtsd2usi	(%rax), %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtsd2usi	(%rax), %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -     vcvtss2usi	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -     vcvtss2usi	%xmm0, %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtss2usi	(%rax), %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtss2usi	(%rax), %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2udq	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2udq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2udq	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2udq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2udq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2udq	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2udq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2udq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2udq	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttps2udq	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2udq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2udq	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttps2udq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2udq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2udq	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttps2udq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2udq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2udq	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttsd2usi	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttsd2usi	%xmm0, %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttsd2usi	(%rax), %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttsd2usi	(%rax), %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -     vcvttss2usi	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -     vcvttss2usi	%xmm0, %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttss2usi	(%rax), %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttss2usi	(%rax), %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtudq2pd	%ymm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtudq2pd	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtudq2pd	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtudq2pd	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtudq2pd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtudq2pd	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtudq2pd	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtudq2pd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtudq2pd	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtudq2ps	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtudq2ps	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtudq2ps	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtudq2ps	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtudq2ps	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtudq2ps	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtudq2ps	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtudq2ps	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtudq2ps	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtusi2sd	%ecx, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtusi2sd	%rcx, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtusi2sdl	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtusi2sdq	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtusi2ss	%ecx, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtusi2ss	%rcx, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtusi2ssl	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtusi2ssq	(%rax), %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     10.00   -      -      -      -      -      -      -      -      -      -      -      -      -     vdivpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     10.00   -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     10.00   -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     10.00   -      -      -      -      -      -      -      -      -      -      -      -      -     vdivpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     10.00   -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     10.00   -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     10.00   -      -      -      -      -      -      -      -      -      -      -      -      -     vdivpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     10.00   -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     10.00   -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     6.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     6.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     6.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     6.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     6.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     6.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     6.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     6.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     6.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherdpd	(%rax,%ymm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherdps	(%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherqpd	(%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherqps	(%rax,%zmm1,2), %ymm2 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovapd	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovapd	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovapd	%zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovapd	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovapd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovapd	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovapd	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovapd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovaps	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovaps	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovaps	%zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovaps	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovaps	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovaps	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovaps	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovaps	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovddup	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovddup	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovddup	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovddup	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovddup	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovddup	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa32	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa32	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqa32	%zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa32	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa32	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqa32	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa32	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa32	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa64	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa64	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqa64	%zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa64	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa64	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqa64	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa64	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa64	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu32	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu32	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu32	%zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu32	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu32	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu32	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu32	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu32	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu64	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu64	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu64	%zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu64	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu64	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu64	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu64	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu64	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovshdup	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovshdup	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovshdup	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovshdup	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovshdup	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovshdup	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovsldup	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovsldup	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovsldup	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovsldup	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovsldup	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovsldup	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovupd	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovupd	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovupd	%zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovupd	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovupd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovupd	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovupd	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovupd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovups	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovups	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovups	%zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovups	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovups	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovups	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovups	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovups	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpabsd	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsd	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsd	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpabsd	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsd	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpabsd	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsd	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpabsq	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpabsq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpabsq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastd	%xmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastd	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastd	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastd	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastq	%xmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastq	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastq	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqd	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax){1to16}, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqd	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax){1to16}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqd	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax){1to16}, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqd	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax){1to16}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqq	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax){1to8}, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqq	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax){1to8}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtd	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtd	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtd	(%rax){1to16}, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtd	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtd	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtd	(%rax){1to16}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtq	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtq	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtq	(%rax){1to8}, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtq	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtq	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtq	(%rax){1to8}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqq	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax){1to8}, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqq	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax){1to8}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpequd	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequd	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequd	(%rax){1to16}, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpequd	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequd	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequd	(%rax){1to16}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpequq	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequq	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequq	(%rax){1to8}, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpequq	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequq	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequq	(%rax){1to8}, %zmm1, %k2 {%k3}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherdq	(%rax,%ymm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherdd	(%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherqq	(%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherqd	(%rax,%zmm1,2), %ymm2 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbd	%xmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbd	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbd	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbd	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbq	%xmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbq	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbq	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxdq	%ymm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxdq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxdq	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxdq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxdq	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxdq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwd	%ymm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwd	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwd	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwd	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwq	%xmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwq	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwq	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbd	%xmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbd	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbd	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbd	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbq	%xmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbq	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbq	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxdq	%ymm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxdq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxdq	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxdq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxdq	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxdq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwd	%ymm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwd	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwd	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwd	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwq	%xmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwq	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwq	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmulld	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmulld	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmulld	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermd	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermd	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermd	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermilpd	$0, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$0, (%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$0, (%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermilpd	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$0, (%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermilpd	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$0, (%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermilps	$0, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$0, (%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$0, (%rax){1to16}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermilps	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$0, (%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermilps	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$0, (%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpermpd	$0, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	$0, (%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	$0, (%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpermpd	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	$0, (%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpermpd	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	$0, (%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpermq	$0, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	$0, (%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	$0, (%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpermq	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	$0, (%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpermq	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	$0, (%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpshufd	$0, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$0, (%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$0, (%rax){1to16}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpshufd	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$0, (%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpshufd	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$0, (%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhdq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhdq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhdq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhqdq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhqdq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhqdq	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhqdq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhqdq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhqdq	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhqdq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhqdq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhqdq	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckldq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckldq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckldq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklqdq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklqdq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklqdq	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklqdq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklqdq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklqdq	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklqdq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklqdq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklqdq	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshuff32x4	$0, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff32x4	$0, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff32x4	$0, (%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshuff32x4	$0, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff32x4	$0, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff32x4	$0, (%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshuff32x4	$0, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff32x4	$0, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff32x4	$0, (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshuff64x2	$0, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff64x2	$0, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff64x2	$0, (%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshuff64x2	$0, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff64x2	$0, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff64x2	$0, (%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshuff64x2	$0, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff64x2	$0, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff64x2	$0, (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshufi32x4	$0, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi32x4	$0, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi32x4	$0, (%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshufi32x4	$0, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi32x4	$0, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi32x4	$0, (%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshufi32x4	$0, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi32x4	$0, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi32x4	$0, (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshufi64x2	$0, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi64x2	$0, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi64x2	$0, (%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshufi64x2	$0, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi64x2	$0, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi64x2	$0, (%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshufi64x2	$0, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi64x2	$0, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi64x2	$0, (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     18.00   -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtpd	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     18.00   -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     18.00   -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     18.00   -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtpd	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     18.00   -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     18.00   -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     18.00   -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtpd	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     18.00   -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     18.00   -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     10.00   -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtps	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     10.00   -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     10.00   -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     10.00   -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtps	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     10.00   -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     10.00   -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     10.00   -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtps	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     10.00   -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     10.00   -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtsd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtsd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtsd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtsd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtsd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtsd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtss	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtss	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtss	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtss	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtss	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtss	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vptestmd	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmd	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmd	(%rax){1to16}, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestmd	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmd	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmd	(%rax){1to16}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vptestmq	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmq	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmq	(%rax){1to8}, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestmq	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmq	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmq	(%rax){1to8}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vptestnmd	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmd	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmd	(%rax){1to16}, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestnmd	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmd	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmd	(%rax){1to16}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vptestnmq	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmq	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmq	(%rax){1to8}, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestnmq	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmq	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmq	(%rax){1to8}, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vsubpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vsubpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vsubpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vsubps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vsubps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vsubps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vucomiss	%xmm16, %xmm17
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vucomiss	(%rax), %xmm17
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512bitalg.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512bitalg.s
new file mode 100644
index 0000000000000..83207fc5785eb
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512bitalg.s
@@ -0,0 +1,95 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vpopcntb        %zmm1, %zmm0
+vpopcntb        (%rdi), %zmm0
+vpopcntb        %zmm1, %zmm0 {%k1}
+vpopcntb        (%rdi), %zmm0 {%k1}
+vpopcntb        %zmm1, %zmm0 {%k1} {z}
+vpopcntb        (%rdi), %zmm0 {%k1} {z}
+
+vpopcntw        %zmm1, %zmm0
+vpopcntw        (%rdi), %zmm0
+vpopcntw        %zmm1, %zmm0 {%k1}
+vpopcntw        (%rdi), %zmm0 {%k1}
+vpopcntw        %zmm1, %zmm0 {%k1} {z}
+vpopcntw        (%rdi), %zmm0 {%k1} {z}
+
+vpshufbitqmb    %zmm16, %zmm17, %k2
+vpshufbitqmb    (%rdi), %zmm17, %k2
+vpshufbitqmb    %zmm16, %zmm17, %k2 {%k1}
+vpshufbitqmb    (%rdi), %zmm17, %k2 {%k1}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     0.50                        vpopcntb	%zmm1, %zmm0
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntb	(%rdi), %zmm0
+# CHECK-NEXT:  1      1     0.50                        vpopcntb	%zmm1, %zmm0 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntb	(%rdi), %zmm0 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpopcntb	%zmm1, %zmm0 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntb	(%rdi), %zmm0 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpopcntw	%zmm1, %zmm0
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntw	(%rdi), %zmm0
+# CHECK-NEXT:  1      1     0.50                        vpopcntw	%zmm1, %zmm0 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntw	(%rdi), %zmm0 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpopcntw	%zmm1, %zmm0 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntw	(%rdi), %zmm0 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vpshufbitqmb	%zmm16, %zmm17, %k2
+# CHECK-NEXT:  1      10    1.00    *                   vpshufbitqmb	(%rdi), %zmm17, %k2
+# CHECK-NEXT:  1      3     1.00                        vpshufbitqmb	%zmm16, %zmm17, %k2 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshufbitqmb	(%rdi), %zmm17, %k2 {%k1}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     10.00  6.00   6.00   10.00  4.00   4.00    -     2.67   2.67   2.67   2.67   2.67   2.67    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpopcntb	%zmm1, %zmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntb	(%rdi), %zmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpopcntb	%zmm1, %zmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntb	(%rdi), %zmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpopcntb	%zmm1, %zmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntb	(%rdi), %zmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpopcntw	%zmm1, %zmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntw	(%rdi), %zmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpopcntw	%zmm1, %zmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntw	(%rdi), %zmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpopcntw	%zmm1, %zmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntw	(%rdi), %zmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpshufbitqmb	%zmm16, %zmm17, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufbitqmb	(%rdi), %zmm17, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpshufbitqmb	%zmm16, %zmm17, %k2 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufbitqmb	(%rdi), %zmm17, %k2 {%k1}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512bitalgvl.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512bitalgvl.s
new file mode 100644
index 0000000000000..7242374e58901
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512bitalgvl.s
@@ -0,0 +1,146 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vpopcntb        %xmm1, %xmm0
+vpopcntb        (%rdi), %xmm0
+vpopcntb        %xmm1, %xmm0 {%k1}
+vpopcntb        (%rdi), %xmm0 {%k1}
+vpopcntb        %xmm1, %xmm0 {%k1} {z}
+vpopcntb        (%rdi), %xmm0 {%k1} {z}
+
+vpopcntb        %ymm1, %ymm0
+vpopcntb        (%rdi), %ymm0
+vpopcntb        %ymm1, %ymm0 {%k1}
+vpopcntb        (%rdi), %ymm0 {%k1}
+vpopcntb        %ymm1, %ymm0 {%k1} {z}
+vpopcntb        (%rdi), %ymm0 {%k1} {z}
+
+vpopcntw        %xmm1, %xmm0
+vpopcntw        (%rdi), %xmm0
+vpopcntw        %xmm1, %xmm0 {%k1}
+vpopcntw        (%rdi), %xmm0 {%k1}
+vpopcntw        %xmm1, %xmm0 {%k1} {z}
+vpopcntw        (%rdi), %xmm0 {%k1} {z}
+
+vpopcntw        %ymm1, %ymm0
+vpopcntw        (%rdi), %ymm0
+vpopcntw        %ymm1, %ymm0 {%k1}
+vpopcntw        (%rdi), %ymm0 {%k1}
+vpopcntw        %ymm1, %ymm0 {%k1} {z}
+vpopcntw        (%rdi), %ymm0 {%k1} {z}
+
+vpshufbitqmb    %xmm16, %xmm17, %k2
+vpshufbitqmb    (%rdi), %xmm17, %k2
+vpshufbitqmb    %xmm16, %xmm17, %k2 {%k1}
+vpshufbitqmb    (%rdi), %xmm17, %k2 {%k1}
+
+vpshufbitqmb    %ymm16, %ymm17, %k2
+vpshufbitqmb    (%rdi), %ymm17, %k2
+vpshufbitqmb    %ymm16, %ymm17, %k2 {%k1}
+vpshufbitqmb    (%rdi), %ymm17, %k2 {%k1}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      2     1.00                        vpopcntb	%xmm1, %xmm0
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntb	(%rdi), %xmm0
+# CHECK-NEXT:  1      2     1.00                        vpopcntb	%xmm1, %xmm0 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntb	(%rdi), %xmm0 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpopcntb	%xmm1, %xmm0 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntb	(%rdi), %xmm0 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpopcntb	%ymm1, %ymm0
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntb	(%rdi), %ymm0
+# CHECK-NEXT:  1      1     0.25                        vpopcntb	%ymm1, %ymm0 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntb	(%rdi), %ymm0 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpopcntb	%ymm1, %ymm0 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntb	(%rdi), %ymm0 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpopcntw	%xmm1, %xmm0
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntw	(%rdi), %xmm0
+# CHECK-NEXT:  1      2     1.00                        vpopcntw	%xmm1, %xmm0 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntw	(%rdi), %xmm0 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpopcntw	%xmm1, %xmm0 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntw	(%rdi), %xmm0 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpopcntw	%ymm1, %ymm0
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntw	(%rdi), %ymm0
+# CHECK-NEXT:  1      1     0.25                        vpopcntw	%ymm1, %ymm0 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntw	(%rdi), %ymm0 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpopcntw	%ymm1, %ymm0 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntw	(%rdi), %ymm0 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshufbitqmb	%xmm16, %xmm17, %k2
+# CHECK-NEXT:  1      10    0.50    *                   vpshufbitqmb	(%rdi), %xmm17, %k2
+# CHECK-NEXT:  1      2     1.00                        vpshufbitqmb	%xmm16, %xmm17, %k2 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshufbitqmb	(%rdi), %xmm17, %k2 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vpshufbitqmb	%ymm16, %ymm17, %k2
+# CHECK-NEXT:  1      10    0.50    *                   vpshufbitqmb	(%rdi), %ymm17, %k2
+# CHECK-NEXT:  1      3     0.50                        vpshufbitqmb	%ymm16, %ymm17, %k2 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshufbitqmb	(%rdi), %ymm17, %k2 {%k1}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     15.50  12.50  4.50   7.50   8.00   8.00    -     5.33   5.33   5.33   5.33   5.33   5.33    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpopcntb	%xmm1, %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntb	(%rdi), %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpopcntb	%xmm1, %xmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntb	(%rdi), %xmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpopcntb	%xmm1, %xmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntb	(%rdi), %xmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpopcntb	%ymm1, %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntb	(%rdi), %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpopcntb	%ymm1, %ymm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntb	(%rdi), %ymm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpopcntb	%ymm1, %ymm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntb	(%rdi), %ymm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpopcntw	%xmm1, %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntw	(%rdi), %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpopcntw	%xmm1, %xmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntw	(%rdi), %xmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpopcntw	%xmm1, %xmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntw	(%rdi), %xmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpopcntw	%ymm1, %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntw	(%rdi), %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpopcntw	%ymm1, %ymm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntw	(%rdi), %ymm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpopcntw	%ymm1, %ymm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntw	(%rdi), %ymm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshufbitqmb	%xmm16, %xmm17, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufbitqmb	(%rdi), %xmm17, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshufbitqmb	%xmm16, %xmm17, %k2 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufbitqmb	(%rdi), %xmm17, %k2 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpshufbitqmb	%ymm16, %ymm17, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufbitqmb	(%rdi), %ymm17, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpshufbitqmb	%ymm16, %ymm17, %k2 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufbitqmb	(%rdi), %ymm17, %k2 {%k1}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512bw.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512bw.s
new file mode 100644
index 0000000000000..d2f32608efb42
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512bw.s
@@ -0,0 +1,1645 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+kaddd             %k0, %k1, %k2
+kaddq             %k0, %k1, %k2
+
+kandd             %k0, %k1, %k2
+kandq             %k0, %k1, %k2
+
+kandnd            %k0, %k1, %k2
+kandnq            %k0, %k1, %k2
+
+kmovd             %k0, %k2
+kmovd             (%rax), %k2
+kmovd             %k0, (%rax)
+kmovd             %eax, %k2
+kmovd             %k0, %eax
+
+kmovq             %k0, %k2
+kmovq             (%rax), %k2
+kmovq             %k0, (%rax)
+kmovq             %rax, %k2
+kmovq             %k0, %rax
+
+knotd             %k0, %k2
+knotq             %k0, %k2
+
+kord              %k0, %k1, %k2
+korq              %k0, %k1, %k2
+
+kortestd          %k0, %k2
+kortestq          %k0, %k2
+
+kshiftld          $2, %k1, %k2
+kshiftlq          $2, %k1, %k2
+
+kshiftrd          $2, %k1, %k2
+kshiftrq          $2, %k1, %k2
+
+ktestd            %k0, %k2
+ktestq            %k0, %k2
+
+kunpckdq          %k0, %k1, %k2
+kunpckwd          %k0, %k1, %k2
+
+kxnord            %k0, %k1, %k2
+kxnorq            %k0, %k1, %k2
+
+kxord             %k0, %k1, %k2
+kxorq             %k0, %k1, %k2
+
+vdbpsadbw         $0, %zmm16, %zmm17, %zmm19
+vdbpsadbw         $0, (%rax), %zmm17, %zmm19
+vdbpsadbw         $0, %zmm16, %zmm17, %zmm19 {k1}
+vdbpsadbw         $0, (%rax), %zmm17, %zmm19 {k1}
+vdbpsadbw         $0, %zmm16, %zmm17, %zmm19 {z}{k1}
+vdbpsadbw         $0, (%rax), %zmm17, %zmm19 {z}{k1}
+
+vmovdqu8          %zmm16, %zmm19
+vmovdqu8          (%rax), %zmm19
+vmovdqu8          %zmm16, (%rax)
+vmovdqu8          %zmm16, %zmm19 {k1}
+vmovdqu8          (%rax), %zmm19 {k1}
+vmovdqu8          %zmm16, (%rax) {k1}
+vmovdqu8          %zmm16, %zmm19 {z}{k1}
+vmovdqu8          (%rax), %zmm19 {z}{k1}
+
+vmovdqu16         %zmm16, %zmm19
+vmovdqu16         (%rax), %zmm19
+vmovdqu16         %zmm16, (%rax)
+vmovdqu16         %zmm16, %zmm19 {k1}
+vmovdqu16         (%rax), %zmm19 {k1}
+vmovdqu16         %zmm16, (%rax) {k1}
+vmovdqu16         %zmm16, %zmm19 {z}{k1}
+vmovdqu16         (%rax), %zmm19 {z}{k1}
+
+vpabsb            %zmm16, %zmm19
+vpabsb            (%rax), %zmm19
+vpabsb            %zmm16, %zmm19 {k1}
+vpabsb            (%rax), %zmm19 {k1}
+vpabsb            %zmm16, %zmm19 {z}{k1}
+vpabsb            (%rax), %zmm19 {z}{k1}
+
+vpabsw            %zmm16, %zmm19
+vpabsw            (%rax), %zmm19
+vpabsw            %zmm16, %zmm19 {k1}
+vpabsw            (%rax), %zmm19 {k1}
+vpabsw            %zmm16, %zmm19 {z}{k1}
+vpabsw            (%rax), %zmm19 {z}{k1}
+
+vpackssdw         %zmm16, %zmm17, %zmm19
+vpackssdw         (%rax), %zmm17, %zmm19
+vpackssdw         %zmm16, %zmm17, %zmm19 {k1}
+vpackssdw         (%rax), %zmm17, %zmm19 {k1}
+vpackssdw         %zmm16, %zmm17, %zmm19 {z}{k1}
+vpackssdw         (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpacksswb         %zmm16, %zmm17, %zmm19
+vpacksswb         (%rax), %zmm17, %zmm19
+vpacksswb         %zmm16, %zmm17, %zmm19 {k1}
+vpacksswb         (%rax), %zmm17, %zmm19 {k1}
+vpacksswb         %zmm16, %zmm17, %zmm19 {z}{k1}
+vpacksswb         (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpackusdw         %zmm16, %zmm17, %zmm19
+vpackusdw         (%rax), %zmm17, %zmm19
+vpackusdw         %zmm16, %zmm17, %zmm19 {k1}
+vpackusdw         (%rax), %zmm17, %zmm19 {k1}
+vpackusdw         %zmm16, %zmm17, %zmm19 {z}{k1}
+vpackusdw         (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpackuswb         %zmm16, %zmm17, %zmm19
+vpackuswb         (%rax), %zmm17, %zmm19
+vpackuswb         %zmm16, %zmm17, %zmm19 {k1}
+vpackuswb         (%rax), %zmm17, %zmm19 {k1}
+vpackuswb         %zmm16, %zmm17, %zmm19 {z}{k1}
+vpackuswb         (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpaddb            %zmm16, %zmm17, %zmm19
+vpaddb            (%rax), %zmm17, %zmm19
+vpaddb            %zmm16, %zmm17, %zmm19 {k1}
+vpaddb            (%rax), %zmm17, %zmm19 {k1}
+vpaddb            %zmm16, %zmm17, %zmm19 {z}{k1}
+vpaddb            (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpaddsb           %zmm16, %zmm17, %zmm19
+vpaddsb           (%rax), %zmm17, %zmm19
+vpaddsb           %zmm16, %zmm17, %zmm19 {k1}
+vpaddsb           (%rax), %zmm17, %zmm19 {k1}
+vpaddsb           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpaddsb           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpaddsw           %zmm16, %zmm17, %zmm19
+vpaddsw           (%rax), %zmm17, %zmm19
+vpaddsw           %zmm16, %zmm17, %zmm19 {k1}
+vpaddsw           (%rax), %zmm17, %zmm19 {k1}
+vpaddsw           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpaddsw           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpaddusb          %zmm16, %zmm17, %zmm19
+vpaddusb          (%rax), %zmm17, %zmm19
+vpaddusb          %zmm16, %zmm17, %zmm19 {k1}
+vpaddusb          (%rax), %zmm17, %zmm19 {k1}
+vpaddusb          %zmm16, %zmm17, %zmm19 {z}{k1}
+vpaddusb          (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpaddusw          %zmm16, %zmm17, %zmm19
+vpaddusw          (%rax), %zmm17, %zmm19
+vpaddusw          %zmm16, %zmm17, %zmm19 {k1}
+vpaddusw          (%rax), %zmm17, %zmm19 {k1}
+vpaddusw          %zmm16, %zmm17, %zmm19 {z}{k1}
+vpaddusw          (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpaddw            %zmm16, %zmm17, %zmm19
+vpaddw            (%rax), %zmm17, %zmm19
+vpaddw            %zmm16, %zmm17, %zmm19 {k1}
+vpaddw            (%rax), %zmm17, %zmm19 {k1}
+vpaddw            %zmm16, %zmm17, %zmm19 {z}{k1}
+vpaddw            (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpalignr          $1, %zmm16, %zmm17, %zmm19
+vpalignr          $1, (%rax), %zmm17, %zmm19
+vpalignr          $1, %zmm16, %zmm17, %zmm19 {k1}
+vpalignr          $1, (%rax), %zmm17, %zmm19 {k1}
+vpalignr          $1, %zmm16, %zmm17, %zmm19 {z}{k1}
+vpalignr          $1, (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpavgb            %zmm16, %zmm17, %zmm19
+vpavgb            (%rax), %zmm17, %zmm19
+vpavgb            %zmm16, %zmm17, %zmm19 {k1}
+vpavgb            (%rax), %zmm17, %zmm19 {k1}
+vpavgb            %zmm16, %zmm17, %zmm19 {z}{k1}
+vpavgb            (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpavgw            %zmm16, %zmm17, %zmm19
+vpavgw            (%rax), %zmm17, %zmm19
+vpavgw            %zmm16, %zmm17, %zmm19 {k1}
+vpavgw            (%rax), %zmm17, %zmm19 {k1}
+vpavgw            %zmm16, %zmm17, %zmm19 {z}{k1}
+vpavgw            (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpblendmb         %zmm16, %zmm17, %zmm19
+vpblendmb         (%rax), %zmm17, %zmm19
+vpblendmb         %zmm16, %zmm17, %zmm19 {k1}
+vpblendmb         (%rax), %zmm17, %zmm19 {k1}
+vpblendmb         %zmm16, %zmm17, %zmm19 {z}{k1}
+vpblendmb         (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpblendmw         %zmm16, %zmm17, %zmm19
+vpblendmw         (%rax), %zmm17, %zmm19
+vpblendmw         %zmm16, %zmm17, %zmm19 {k1}
+vpblendmw         (%rax), %zmm17, %zmm19 {k1}
+vpblendmw         %zmm16, %zmm17, %zmm19 {z}{k1}
+vpblendmw         (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpbroadcastb      %xmm16, %zmm19
+vpbroadcastb      (%rax), %zmm19
+vpbroadcastb      %eax, %zmm19
+vpbroadcastb      %xmm16, %zmm19 {k1}
+vpbroadcastb      (%rax), %zmm19 {k1}
+vpbroadcastb      %eax, %zmm19 {k1}
+vpbroadcastb      %xmm16, %zmm19 {z}{k1}
+vpbroadcastb      (%rax), %zmm19 {z}{k1}
+vpbroadcastb      %eax, %zmm19 {z}{k1}
+
+vpbroadcastw      %xmm16, %zmm19
+vpbroadcastw      (%rax), %zmm19
+vpbroadcastw      %eax, %zmm19
+vpbroadcastw      %xmm16, %zmm19 {k1}
+vpbroadcastw      (%rax), %zmm19 {k1}
+vpbroadcastw      %eax, %zmm19 {k1}
+vpbroadcastw      %xmm16, %zmm19 {z}{k1}
+vpbroadcastw      (%rax), %zmm19 {z}{k1}
+vpbroadcastw      %eax, %zmm19 {z}{k1}
+
+vpcmpb            $0, %zmm0, %zmm1, %k2
+vpcmpb            $0, (%rax), %zmm1, %k2
+vpcmpb            $0, %zmm0, %zmm1, %k2 {k3}
+vpcmpb            $0, (%rax), %zmm1, %k2 {k3}
+
+vpcmpeqb          %zmm0, %zmm1, %k2
+vpcmpeqb          (%rax), %zmm1, %k2
+vpcmpeqb          %zmm0, %zmm1, %k2 {k3}
+vpcmpeqb          (%rax), %zmm1, %k2 {k3}
+
+vpcmpeqw          %zmm0, %zmm1, %k2
+vpcmpeqw          (%rax), %zmm1, %k2
+vpcmpeqw          %zmm0, %zmm1, %k2 {k3}
+vpcmpeqw          (%rax), %zmm1, %k2 {k3}
+
+vpcmpgtb          %zmm0, %zmm1, %k2
+vpcmpgtb          (%rax), %zmm1, %k2
+vpcmpgtb          %zmm0, %zmm1, %k2 {k3}
+vpcmpgtb          (%rax), %zmm1, %k2 {k3}
+
+vpcmpgtw          %zmm0, %zmm1, %k2
+vpcmpgtw          (%rax), %zmm1, %k2
+vpcmpgtw          %zmm0, %zmm1, %k2 {k3}
+vpcmpgtw          (%rax), %zmm1, %k2 {k3}
+
+vpcmpub           $0, %zmm0, %zmm1, %k2
+vpcmpub           $0, (%rax), %zmm1, %k2
+vpcmpub           $0, %zmm0, %zmm1, %k2 {k3}
+vpcmpub           $0, (%rax), %zmm1, %k2 {k3}
+
+vpcmpuw           $0, %zmm0, %zmm1, %k2
+vpcmpuw           $0, (%rax), %zmm1, %k2
+vpcmpuw           $0, %zmm0, %zmm1, %k2 {k3}
+vpcmpuw           $0, (%rax), %zmm1, %k2 {k3}
+
+vpcmpw            $0, %zmm0, %zmm1, %k2
+vpcmpw            $0, (%rax), %zmm1, %k2
+vpcmpw            $0, %zmm0, %zmm1, %k2 {k3}
+vpcmpw            $0, (%rax), %zmm1, %k2 {k3}
+
+vpextrb           $0, %xmm16, %rax
+vpextrb           $0, %xmm16, (%rax)
+
+vpextrw           $0, %xmm16, %rax
+vpextrw           $0, %xmm16, (%rax)
+
+vpinsrb           $0, %rax, %xmm16, %xmm19
+vpinsrb           $0, (%rax), %xmm16, %xmm19
+
+vpinsrw           $0, %rax, %xmm16, %xmm19
+vpinsrw           $0, (%rax), %xmm16, %xmm19
+
+vpermw            %zmm16, %zmm17, %zmm19
+vpermw            (%rax), %zmm17, %zmm19
+vpermw            %zmm16, %zmm17, %zmm19 {k1}
+vpermw            (%rax), %zmm17, %zmm19 {k1}
+vpermw            %zmm16, %zmm17, %zmm19 {z}{k1}
+vpermw            (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpermi2w          %zmm16, %zmm17, %zmm19
+vpermi2w          (%rax), %zmm17, %zmm19
+vpermi2w          %zmm16, %zmm17, %zmm19 {k1}
+vpermi2w          (%rax), %zmm17, %zmm19 {k1}
+vpermi2w          %zmm16, %zmm17, %zmm19 {z}{k1}
+vpermi2w          (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpermt2w          %zmm16, %zmm17, %zmm19
+vpermt2w          (%rax), %zmm17, %zmm19
+vpermt2w          %zmm16, %zmm17, %zmm19 {k1}
+vpermt2w          (%rax), %zmm17, %zmm19 {k1}
+vpermt2w          %zmm16, %zmm17, %zmm19 {z}{k1}
+vpermt2w          (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpmaddubsw        %zmm16, %zmm17, %zmm19
+vpmaddubsw        (%rax), %zmm17, %zmm19
+vpmaddubsw        %zmm16, %zmm17, %zmm19 {k1}
+vpmaddubsw        (%rax), %zmm17, %zmm19 {k1}
+vpmaddubsw        %zmm16, %zmm17, %zmm19 {z}{k1}
+vpmaddubsw        (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpmaddwd          %zmm16, %zmm17, %zmm19
+vpmaddwd          (%rax), %zmm17, %zmm19
+vpmaddwd          %zmm16, %zmm17, %zmm19 {k1}
+vpmaddwd          (%rax), %zmm17, %zmm19 {k1}
+vpmaddwd          %zmm16, %zmm17, %zmm19 {z}{k1}
+vpmaddwd          (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpmaxsb           %zmm16, %zmm17, %zmm19
+vpmaxsb           (%rax), %zmm17, %zmm19
+vpmaxsb           %zmm16, %zmm17, %zmm19 {k1}
+vpmaxsb           (%rax), %zmm17, %zmm19 {k1}
+vpmaxsb           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpmaxsb           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpmaxsw           %zmm16, %zmm17, %zmm19
+vpmaxsw           (%rax), %zmm17, %zmm19
+vpmaxsw           %zmm16, %zmm17, %zmm19 {k1}
+vpmaxsw           (%rax), %zmm17, %zmm19 {k1}
+vpmaxsw           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpmaxsw           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpmaxub           %zmm16, %zmm17, %zmm19
+vpmaxub           (%rax), %zmm17, %zmm19
+vpmaxub           %zmm16, %zmm17, %zmm19 {k1}
+vpmaxub           (%rax), %zmm17, %zmm19 {k1}
+vpmaxub           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpmaxub           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpmaxuw           %zmm16, %zmm17, %zmm19
+vpmaxuw           (%rax), %zmm17, %zmm19
+vpmaxuw           %zmm16, %zmm17, %zmm19 {k1}
+vpmaxuw           (%rax), %zmm17, %zmm19 {k1}
+vpmaxuw           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpmaxuw           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpminsb           %zmm16, %zmm17, %zmm19
+vpminsb           (%rax), %zmm17, %zmm19
+vpminsb           %zmm16, %zmm17, %zmm19 {k1}
+vpminsb           (%rax), %zmm17, %zmm19 {k1}
+vpminsb           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpminsb           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpminsw           %zmm16, %zmm17, %zmm19
+vpminsw           (%rax), %zmm17, %zmm19
+vpminsw           %zmm16, %zmm17, %zmm19 {k1}
+vpminsw           (%rax), %zmm17, %zmm19 {k1}
+vpminsw           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpminsw           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpminub           %zmm16, %zmm17, %zmm19
+vpminub           (%rax), %zmm17, %zmm19
+vpminub           %zmm16, %zmm17, %zmm19 {k1}
+vpminub           (%rax), %zmm17, %zmm19 {k1}
+vpminub           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpminub           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpminuw           %zmm16, %zmm17, %zmm19
+vpminuw           (%rax), %zmm17, %zmm19
+vpminuw           %zmm16, %zmm17, %zmm19 {k1}
+vpminuw           (%rax), %zmm17, %zmm19 {k1}
+vpminuw           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpminuw           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpmovb2m          %zmm0, %k0
+vpmovw2m          %zmm0, %k0
+
+vpmovm2b          %k0, %zmm0
+vpmovm2w          %k0, %zmm0
+
+vpmovsxbw         %ymm16, %zmm19
+vpmovsxbw         (%rax), %zmm19
+vpmovsxbw         %ymm16, %zmm19 {k1}
+vpmovsxbw         (%rax), %zmm19 {k1}
+vpmovsxbw         %ymm16, %zmm19 {z}{k1}
+vpmovsxbw         (%rax), %zmm19 {z}{k1}
+
+vpmovswb          %zmm16, %ymm19
+vpmovswb          %zmm16, (%rax)
+vpmovswb          %zmm16, %ymm19 {k1}
+vpmovswb          %zmm16, (%rax) {k1}
+
+vpmovuswb         %zmm16, %ymm19
+vpmovuswb         %zmm16, (%rax)
+vpmovuswb         %zmm16, %ymm19 {k1}
+vpmovuswb         %zmm16, (%rax) {k1}
+
+vpmovwb           %zmm16, %ymm19
+vpmovwb           %zmm16, (%rax)
+vpmovwb           %zmm16, %ymm19 {k1}
+vpmovwb           %zmm16, (%rax) {k1}
+
+vpmovzxbw         %ymm16, %zmm19
+vpmovzxbw         (%rax), %zmm19
+vpmovzxbw         %ymm16, %zmm19 {k1}
+vpmovzxbw         (%rax), %zmm19 {k1}
+vpmovzxbw         %ymm16, %zmm19 {z}{k1}
+vpmovzxbw         (%rax), %zmm19 {z}{k1}
+
+vpmulhrsw         %zmm16, %zmm17, %zmm19
+vpmulhrsw         (%rax), %zmm17, %zmm19
+vpmulhrsw         %zmm16, %zmm17, %zmm19 {k1}
+vpmulhrsw         (%rax), %zmm17, %zmm19 {k1}
+vpmulhrsw         %zmm16, %zmm17, %zmm19 {z}{k1}
+vpmulhrsw         (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpmulhuw          %zmm16, %zmm17, %zmm19
+vpmulhuw          (%rax), %zmm17, %zmm19
+vpmulhuw          %zmm16, %zmm17, %zmm19 {k1}
+vpmulhuw          (%rax), %zmm17, %zmm19 {k1}
+vpmulhuw          %zmm16, %zmm17, %zmm19 {z}{k1}
+vpmulhuw          (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpmulhw           %zmm16, %zmm17, %zmm19
+vpmulhw           (%rax), %zmm17, %zmm19
+vpmulhw           %zmm16, %zmm17, %zmm19 {k1}
+vpmulhw           (%rax), %zmm17, %zmm19 {k1}
+vpmulhw           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpmulhw           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpmullw           %zmm16, %zmm17, %zmm19
+vpmullw           (%rax), %zmm17, %zmm19
+vpmullw           %zmm16, %zmm17, %zmm19 {k1}
+vpmullw           (%rax), %zmm17, %zmm19 {k1}
+vpmullw           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpmullw           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpsadbw           %zmm16, %zmm17, %zmm19
+vpsadbw           (%rax), %zmm17, %zmm19
+
+vpshufb           %zmm16, %zmm17, %zmm19
+vpshufb           (%rax), %zmm17, %zmm19
+vpshufb           %zmm16, %zmm17, %zmm19 {k1}
+vpshufb           (%rax), %zmm17, %zmm19 {k1}
+vpshufb           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpshufb           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpshufhw          $0, %zmm16, %zmm19
+vpshufhw          $0, (%rax), %zmm19
+vpshufhw          $0, %zmm16, %zmm19 {k1}
+vpshufhw          $0, (%rax), %zmm19 {k1}
+vpshufhw          $0, %zmm16, %zmm19 {z}{k1}
+vpshufhw          $0, (%rax), %zmm19 {z}{k1}
+
+vpshuflw          $0, %zmm16, %zmm19
+vpshuflw          $0, (%rax), %zmm19
+vpshuflw          $0, %zmm16, %zmm19 {k1}
+vpshuflw          $0, (%rax), %zmm19 {k1}
+vpshuflw          $0, %zmm16, %zmm19 {z}{k1}
+vpshuflw          $0, (%rax), %zmm19 {z}{k1}
+
+vpslldq           $1, %zmm16, %zmm19
+vpslldq           $1, (%rax), %zmm19
+
+vpsllvw           %zmm16, %zmm17, %zmm19
+vpsllvw           (%rax), %zmm17, %zmm19
+vpsllvw           %zmm16, %zmm17, %zmm19 {k1}
+vpsllvw           (%rax), %zmm17, %zmm19 {k1}
+vpsllvw           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpsllvw           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpsllw            $0, %zmm16, %zmm19
+vpsllw            $0, (%rax), %zmm19
+vpsllw            $0, %zmm16, %zmm19 {k1}
+vpsllw            $0, (%rax), %zmm19 {k1}
+vpsllw            $0, %zmm16, %zmm19 {z}{k1}
+vpsllw            $0, (%rax), %zmm19 {z}{k1}
+
+vpsllw            %xmm16, %zmm17, %zmm19
+vpsllw            (%rax), %zmm17, %zmm19
+vpsllw            %xmm16, %zmm17, %zmm19 {k1}
+vpsllw            (%rax), %zmm17, %zmm19 {k1}
+vpsllw            %xmm16, %zmm17, %zmm19 {z}{k1}
+vpsllw            (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpsravw           %zmm16, %zmm17, %zmm19
+vpsravw           (%rax), %zmm17, %zmm19
+vpsravw           %zmm16, %zmm17, %zmm19 {k1}
+vpsravw           (%rax), %zmm17, %zmm19 {k1}
+vpsravw           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpsravw           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpsraw            $0, %zmm16, %zmm19
+vpsraw            $0, (%rax), %zmm19
+vpsraw            $0, %zmm16, %zmm19 {k1}
+vpsraw            $0, (%rax), %zmm19 {k1}
+vpsraw            $0, %zmm16, %zmm19 {z}{k1}
+vpsraw            $0, (%rax), %zmm19 {z}{k1}
+
+vpsraw            %xmm16, %zmm17, %zmm19
+vpsraw            (%rax), %zmm17, %zmm19
+vpsraw            %xmm16, %zmm17, %zmm19 {k1}
+vpsraw            (%rax), %zmm17, %zmm19 {k1}
+vpsraw            %xmm16, %zmm17, %zmm19 {z}{k1}
+vpsraw            (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpsrldq           $1, %zmm16, %zmm19
+vpsrldq           $1, (%rax), %zmm19
+
+vpsrlvw           %zmm16, %zmm17, %zmm19
+vpsrlvw           (%rax), %zmm17, %zmm19
+vpsrlvw           %zmm16, %zmm17, %zmm19 {k1}
+vpsrlvw           (%rax), %zmm17, %zmm19 {k1}
+vpsrlvw           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpsrlvw           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpsrlw            $0, %zmm16, %zmm19
+vpsrlw            $0, (%rax), %zmm19
+vpsrlw            $0, %zmm16, %zmm19 {k1}
+vpsrlw            $0, (%rax), %zmm19 {k1}
+vpsrlw            $0, %zmm16, %zmm19 {z}{k1}
+vpsrlw            $0, (%rax), %zmm19 {z}{k1}
+
+vpsrlw            %xmm16, %zmm17, %zmm19
+vpsrlw            (%rax), %zmm17, %zmm19
+vpsrlw            %xmm16, %zmm17, %zmm19 {k1}
+vpsrlw            (%rax), %zmm17, %zmm19 {k1}
+vpsrlw            %xmm16, %zmm17, %zmm19 {z}{k1}
+vpsrlw            (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpsubb            %zmm16, %zmm17, %zmm19
+vpsubb            (%rax), %zmm17, %zmm19
+vpsubb            %zmm16, %zmm17, %zmm19 {k1}
+vpsubb            (%rax), %zmm17, %zmm19 {k1}
+vpsubb            %zmm16, %zmm17, %zmm19 {z}{k1}
+vpsubb            (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpsubsb           %zmm16, %zmm17, %zmm19
+vpsubsb           (%rax), %zmm17, %zmm19
+vpsubsb           %zmm16, %zmm17, %zmm19 {k1}
+vpsubsb           (%rax), %zmm17, %zmm19 {k1}
+vpsubsb           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpsubsb           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpsubsw           %zmm16, %zmm17, %zmm19
+vpsubsw           (%rax), %zmm17, %zmm19
+vpsubsw           %zmm16, %zmm17, %zmm19 {k1}
+vpsubsw           (%rax), %zmm17, %zmm19 {k1}
+vpsubsw           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpsubsw           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpsubusb          %zmm16, %zmm17, %zmm19
+vpsubusb          (%rax), %zmm17, %zmm19
+vpsubusb          %zmm16, %zmm17, %zmm19 {k1}
+vpsubusb          (%rax), %zmm17, %zmm19 {k1}
+vpsubusb          %zmm16, %zmm17, %zmm19 {z}{k1}
+vpsubusb          (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpsubusw          %zmm16, %zmm17, %zmm19
+vpsubusw          (%rax), %zmm17, %zmm19
+vpsubusw          %zmm16, %zmm17, %zmm19 {k1}
+vpsubusw          (%rax), %zmm17, %zmm19 {k1}
+vpsubusw          %zmm16, %zmm17, %zmm19 {z}{k1}
+vpsubusw          (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpsubw            %zmm16, %zmm17, %zmm19
+vpsubw            (%rax), %zmm17, %zmm19
+vpsubw            %zmm16, %zmm17, %zmm19 {k1}
+vpsubw            (%rax), %zmm17, %zmm19 {k1}
+vpsubw            %zmm16, %zmm17, %zmm19 {z}{k1}
+vpsubw            (%rax), %zmm17, %zmm19 {z}{k1}
+
+vptestmb          %zmm0, %zmm1, %k2
+vptestmb          (%rax), %zmm1, %k2
+vptestmb          %zmm0, %zmm1, %k2 {k3}
+vptestmb          (%rax), %zmm1, %k2 {k3}
+
+vptestmw          %zmm0, %zmm1, %k2
+vptestmw          (%rax), %zmm1, %k2
+vptestmw          %zmm0, %zmm1, %k2 {k3}
+vptestmw          (%rax), %zmm1, %k2 {k3}
+
+vptestnmb         %zmm0, %zmm1, %k2
+vptestnmb         (%rax), %zmm1, %k2
+vptestnmb         %zmm0, %zmm1, %k2 {k3}
+vptestnmb         (%rax), %zmm1, %k2 {k3}
+
+vptestnmw         %zmm0, %zmm1, %k2
+vptestnmw         (%rax), %zmm1, %k2
+vptestnmw         %zmm0, %zmm1, %k2 {k3}
+vptestnmw         (%rax), %zmm1, %k2 {k3}
+
+vpunpckhbw        %zmm16, %zmm17, %zmm19
+vpunpckhbw        (%rax), %zmm17, %zmm19
+vpunpckhbw        %zmm16, %zmm17, %zmm19 {k1}
+vpunpckhbw        (%rax), %zmm17, %zmm19 {k1}
+vpunpckhbw        %zmm16, %zmm17, %zmm19 {z}{k1}
+vpunpckhbw        (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpunpckhwd        %zmm16, %zmm17, %zmm19
+vpunpckhwd        (%rax), %zmm17, %zmm19
+vpunpckhwd        %zmm16, %zmm17, %zmm19 {k1}
+vpunpckhwd        (%rax), %zmm17, %zmm19 {k1}
+vpunpckhwd        %zmm16, %zmm17, %zmm19 {z}{k1}
+vpunpckhwd        (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpunpcklbw        %zmm16, %zmm17, %zmm19
+vpunpcklbw        (%rax), %zmm17, %zmm19
+vpunpcklbw        %zmm16, %zmm17, %zmm19 {k1}
+vpunpcklbw        (%rax), %zmm17, %zmm19 {k1}
+vpunpcklbw        %zmm16, %zmm17, %zmm19 {z}{k1}
+vpunpcklbw        (%rax), %zmm17, %zmm19 {z}{k1}
+
+vpunpcklwd        %zmm16, %zmm17, %zmm19
+vpunpcklwd        (%rax), %zmm17, %zmm19
+vpunpcklwd        %zmm16, %zmm17, %zmm19 {k1}
+vpunpcklwd        (%rax), %zmm17, %zmm19 {k1}
+vpunpcklwd        %zmm16, %zmm17, %zmm19 {z}{k1}
+vpunpcklwd        (%rax), %zmm17, %zmm19 {z}{k1}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     0.50                        kaddd	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kaddq	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kandd	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kandq	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kandnd	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kandnq	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kmovd	%k0, %k2
+# CHECK-NEXT:  1      5     0.33    *                   kmovd	(%rax), %k2
+# CHECK-NEXT:  1      1     0.50           *            kmovd	%k0, (%rax)
+# CHECK-NEXT:  1      1     0.50                        kmovd	%eax, %k2
+# CHECK-NEXT:  1      1     0.50                        kmovd	%k0, %eax
+# CHECK-NEXT:  1      1     0.50                        kmovq	%k0, %k2
+# CHECK-NEXT:  1      5     0.33    *                   kmovq	(%rax), %k2
+# CHECK-NEXT:  1      1     0.50           *            kmovq	%k0, (%rax)
+# CHECK-NEXT:  1      1     0.50                        kmovq	%rax, %k2
+# CHECK-NEXT:  1      1     0.50                        kmovq	%k0, %rax
+# CHECK-NEXT:  1      1     0.50                        knotd	%k0, %k2
+# CHECK-NEXT:  1      1     0.50                        knotq	%k0, %k2
+# CHECK-NEXT:  1      1     0.50                        kord	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        korq	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kortestd	%k0, %k2
+# CHECK-NEXT:  1      1     0.50                        kortestq	%k0, %k2
+# CHECK-NEXT:  1      1     0.50                        kshiftld	$2, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kshiftlq	$2, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kshiftrd	$2, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kshiftrq	$2, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        ktestd	%k0, %k2
+# CHECK-NEXT:  1      1     0.50                        ktestq	%k0, %k2
+# CHECK-NEXT:  1      1     0.50                        kunpckdq	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kunpckwd	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kxnord	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kxnorq	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kxord	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kxorq	%k0, %k1, %k2
+# CHECK-NEXT:  1      4     1.00                        vdbpsadbw	$0, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      11    1.00    *                   vdbpsadbw	$0, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      4     1.00                        vdbpsadbw	$0, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      11    1.00    *                   vdbpsadbw	$0, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      4     1.00                        vdbpsadbw	$0, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.00    *                   vdbpsadbw	$0, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu8	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu8	(%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu8	%zmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovdqu8	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu8	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu8	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu8	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu8	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu16	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu16	(%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu16	%zmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovdqu16	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu16	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu16	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu16	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu16	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpabsb	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpabsb	(%rax), %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpabsb	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsb	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpabsb	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsb	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpabsw	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpabsw	(%rax), %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpabsw	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsw	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpabsw	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsw	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpackssdw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpackssdw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpackssdw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpackssdw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpackssdw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpackssdw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpacksswb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpacksswb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpacksswb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpacksswb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpacksswb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpacksswb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpackusdw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpackusdw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpackusdw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpackusdw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpackusdw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpackusdw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpackuswb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpackuswb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpackuswb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpackuswb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpackuswb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpackuswb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpaddb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpaddb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpaddb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpaddsb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpaddsb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpaddsb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpaddsw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpaddsw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpaddsw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpaddusb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpaddusb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpaddusb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpaddusw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpaddusw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpaddusw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpaddw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpaddw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpaddw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpalignr	$1, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpalignr	$1, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpalignr	$1, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpalignr	$1, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpalignr	$1, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpalignr	$1, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpavgb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpavgb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpavgb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpavgb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpavgb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpavgb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpavgw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpavgw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpavgw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpavgw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpavgw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpavgw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpblendmb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpblendmb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpblendmb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpblendmb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpblendmb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpblendmb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpblendmw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpblendmw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpblendmw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpblendmw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpblendmw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpblendmw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastb	%xmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastb	(%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastb	%eax, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastb	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastb	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastb	%eax, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastb	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastb	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastb	%eax, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastw	%xmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastw	(%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastw	%eax, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastw	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastw	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastw	%eax, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastw	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastw	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastw	%eax, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpcmpeqb	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqb	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      1     0.50                        vpcmpeqb	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqb	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vpcmpeqb	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqb	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      1     0.50                        vpcmpeqb	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqb	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vpcmpeqw	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqw	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      1     0.50                        vpcmpeqw	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqw	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vpcmpgtb	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtb	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      1     0.50                        vpcmpgtb	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtb	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vpcmpgtw	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtw	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      1     0.50                        vpcmpgtw	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtw	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vpcmpequb	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequb	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      1     0.50                        vpcmpequb	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequb	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vpcmpequw	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequw	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      1     0.50                        vpcmpequw	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequw	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vpcmpeqw	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqw	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      1     0.50                        vpcmpeqw	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqw	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  2      1     1.00                        vpextrb	$0, %xmm16, %eax
+# CHECK-NEXT:  2      2     1.00           *            vpextrb	$0, %xmm16, (%rax)
+# CHECK-NEXT:  2      1     1.00                        vpextrw	$0, %xmm16, %eax
+# CHECK-NEXT:  2      2     1.00           *            vpextrw	$0, %xmm16, (%rax)
+# CHECK-NEXT:  2      1     1.00                        vpinsrb	$0, %eax, %xmm16, %xmm19
+# CHECK-NEXT:  1      8     1.50    *                   vpinsrb	$0, (%rax), %xmm16, %xmm19
+# CHECK-NEXT:  2      1     1.00                        vpinsrw	$0, %eax, %xmm16, %xmm19
+# CHECK-NEXT:  1      8     1.50    *                   vpinsrw	$0, (%rax), %xmm16, %xmm19
+# CHECK-NEXT:  1      5     1.00                        vpermw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      5     1.00                        vpermw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      5     1.00                        vpermw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      5     1.00                        vpermi2w	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermi2w	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      5     1.00                        vpermi2w	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermi2w	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      5     1.00                        vpermi2w	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermi2w	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      5     1.00                        vpermt2w	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermt2w	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      5     1.00                        vpermt2w	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermt2w	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      5     1.00                        vpermt2w	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermt2w	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vpmaddubsw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpmaddubsw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      3     1.00                        vpmaddubsw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpmaddubsw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vpmaddubsw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpmaddubsw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vpmaddwd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpmaddwd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      3     1.00                        vpmaddwd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpmaddwd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vpmaddwd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpmaddwd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpmaxsb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpmaxsb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpmaxsb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpmaxsw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpmaxsw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpmaxsw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpmaxub	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxub	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpmaxub	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxub	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpmaxub	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxub	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpmaxuw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxuw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpmaxuw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxuw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpmaxuw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxuw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpminsb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpminsb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpminsb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpminsb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpminsb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpminsb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpminsw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpminsw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpminsw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpminsw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpminsw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpminsw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpminub	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpminub	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpminub	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpminub	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpminub	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpminub	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpminuw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpminuw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpminuw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpminuw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpminuw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpminuw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpmovb2m	%zmm0, %k0
+# CHECK-NEXT:  1      1     1.00                        vpmovw2m	%zmm0, %k0
+# CHECK-NEXT:  1      0     0.17                        vpmovm2b	%k0, %zmm0
+# CHECK-NEXT:  1      0     0.17                        vpmovm2w	%k0, %zmm0
+# CHECK-NEXT:  1      4     2.00                        vpmovsxbw	%ymm16, %zmm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbw	(%rax), %zmm19
+# CHECK-NEXT:  1      4     2.00                        vpmovsxbw	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbw	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmovsxbw	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbw	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      4     1.50                        vpmovswb	%zmm16, %ymm19
+# CHECK-NEXT:  1      11    1.50           *            vpmovswb	%zmm16, (%rax)
+# CHECK-NEXT:  2      4     1.50                        vpmovswb	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    1.50           *            vpmovswb	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  2      4     1.50                        vpmovuswb	%zmm16, %ymm19
+# CHECK-NEXT:  1      11    1.50           *            vpmovuswb	%zmm16, (%rax)
+# CHECK-NEXT:  2      4     1.50                        vpmovuswb	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    1.50           *            vpmovuswb	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  2      4     1.50                        vpmovwb	%zmm16, %ymm19
+# CHECK-NEXT:  1      11    1.50           *            vpmovwb	%zmm16, (%rax)
+# CHECK-NEXT:  2      4     1.50                        vpmovwb	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    1.50           *            vpmovwb	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmovzxbw	%ymm16, %zmm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbw	(%rax), %zmm19
+# CHECK-NEXT:  1      4     2.00                        vpmovzxbw	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbw	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmovzxbw	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbw	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vpmulhrsw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpmulhrsw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      3     1.00                        vpmulhrsw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpmulhrsw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vpmulhrsw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpmulhrsw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vpmulhuw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpmulhuw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      3     1.00                        vpmulhuw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpmulhuw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vpmulhuw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpmulhuw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vpmulhw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpmulhw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      3     1.00                        vpmulhw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpmulhw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vpmulhw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpmulhw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vpmullw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpmullw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      3     1.00                        vpmullw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpmullw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vpmullw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpmullw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     1.00                        vpsadbw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      11    1.00    *                   vpsadbw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpshufb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpshufb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpshufb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpshufb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshufb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpshufb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpshufhw	$0, %zmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpshufhw	$0, (%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpshufhw	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpshufhw	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpshufhw	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpshufhw	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpshuflw	$0, %zmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpshuflw	$0, (%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpshuflw	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpshuflw	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpshuflw	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpshuflw	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpslldq	$1, %zmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpslldq	$1, (%rax), %zmm19
+# CHECK-NEXT:  2      1     1.00                        vpsllvw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  2      8     1.00    *                   vpsllvw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  2      1     1.00                        vpsllvw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  2      8     1.00    *                   vpsllvw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  2      1     1.00                        vpsllvw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     1.00    *                   vpsllvw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsllw	$0, %zmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpsllw	$0, (%rax), %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpsllw	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpsllw	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsllw	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpsllw	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpsllw	%xmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpsllw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpsllw	%xmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpsllw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpsllw	%xmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpsllw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      1     1.00                        vpsravw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  2      8     1.00    *                   vpsravw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  2      1     1.00                        vpsravw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  2      8     1.00    *                   vpsravw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  2      1     1.00                        vpsravw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     1.00    *                   vpsravw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsraw	$0, %zmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpsraw	$0, (%rax), %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpsraw	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpsraw	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsraw	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpsraw	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpsraw	%xmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpsraw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpsraw	%xmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpsraw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpsraw	%xmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpsraw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpsrldq	$1, %zmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpsrldq	$1, (%rax), %zmm19
+# CHECK-NEXT:  2      1     1.00                        vpsrlvw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  2      8     1.00    *                   vpsrlvw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  2      1     1.00                        vpsrlvw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  2      8     1.00    *                   vpsrlvw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  2      1     1.00                        vpsrlvw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     1.00    *                   vpsrlvw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsrlw	$0, %zmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpsrlw	$0, (%rax), %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpsrlw	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpsrlw	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsrlw	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpsrlw	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpsrlw	%xmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpsrlw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpsrlw	%xmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpsrlw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpsrlw	%xmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpsrlw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsubb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpsubb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsubb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsubsb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpsubsb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsubsb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsubsw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpsubsw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsubsw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsubusb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpsubusb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsubusb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsubusw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpsubusw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsubusw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsubw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpsubw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsubw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vptestmb	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestmb	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      5     2.50                        vptestmb	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestmb	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vptestmw	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestmw	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      5     2.50                        vptestmw	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestmw	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vptestnmb	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmb	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      5     2.50                        vptestnmb	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmb	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vptestnmw	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmw	(%rax), %zmm1, %k2
+# CHECK-NEXT:  1      5     2.50                        vptestnmw	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmw	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     1.00                        vpunpckhbw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckhbw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpunpckhbw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckhbw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpunpckhbw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckhbw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpunpckhwd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckhwd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpunpckhwd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckhwd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpunpckhwd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpckhwd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpunpcklbw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpunpcklbw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpunpcklbw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpcklbw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpunpcklbw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpcklbw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpunpcklwd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vpunpcklwd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vpunpcklwd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpcklwd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpunpcklwd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpunpcklwd	(%rax), %zmm17, %zmm19 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 0.67   0.67   0.67   2.00   2.00   2.00   2.00    -     178.50 310.00 275.50 162.00 123.00 123.00 6.00   77.33  77.33  77.33  75.33  75.33  75.33  3.00   3.00
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kaddd	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kaddq	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kandd	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kandq	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kandnd	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kandnq	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kmovd	%k0, %k2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     kmovd	(%rax), %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -     kmovd	%k0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -     kmovd	%eax, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kmovd	%k0, %eax
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kmovq	%k0, %k2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     kmovq	(%rax), %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -     kmovq	%k0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -     kmovq	%rax, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kmovq	%k0, %rax
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     knotd	%k0, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     knotq	%k0, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kord	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     korq	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kortestd	%k0, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kortestq	%k0, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     kshiftld	$2, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     kshiftlq	$2, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     kshiftrd	$2, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     kshiftrq	$2, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     ktestd	%k0, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     ktestq	%k0, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kunpckdq	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kunpckwd	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kxnord	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kxnorq	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kxord	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kxorq	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vdbpsadbw	$0, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00   1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdbpsadbw	$0, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vdbpsadbw	$0, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00   1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdbpsadbw	$0, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vdbpsadbw	$0, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00   1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdbpsadbw	$0, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu8	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu8	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu8	%zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu8	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu8	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu8	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu8	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu8	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu16	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu16	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu16	%zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu16	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu16	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu16	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu16	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu16	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpabsb	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsb	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpabsb	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsb	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpabsb	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsb	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpabsw	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsw	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpabsw	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsw	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpabsw	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsw	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackssdw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackssdw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackssdw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackssdw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackssdw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackssdw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpacksswb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpacksswb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpacksswb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpacksswb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpacksswb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpacksswb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackusdw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackusdw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackusdw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackusdw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackusdw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackusdw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackuswb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackuswb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackuswb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackuswb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackuswb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackuswb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddsb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddsb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddsb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddsw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddsw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddsw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddusb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddusb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddusb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddusw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddusw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddusw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpaddw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpalignr	$1, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpalignr	$1, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpalignr	$1, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpalignr	$1, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpalignr	$1, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpalignr	$1, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpavgb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpavgb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpavgb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpavgw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpavgw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpavgw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpblendmb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendmb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpblendmb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendmb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpblendmb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendmb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpblendmw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendmw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpblendmw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendmw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpblendmw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendmw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%xmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastb	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%eax, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastb	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%eax, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastb	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%eax, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%xmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastw	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%eax, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastw	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%eax, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastw	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%eax, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqb	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqb	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqb	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqb	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqb	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqb	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqb	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqb	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqw	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqw	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqw	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqw	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtb	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtb	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtb	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtb	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtw	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtw	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtw	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtw	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpequb	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequb	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpequb	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequb	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpequw	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequw	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpequw	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequw	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqw	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqw	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqw	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqw	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vpextrb	$0, %xmm16, %eax
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vpextrb	$0, %xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vpextrw	$0, %xmm16, %eax
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vpextrw	$0, %xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vpinsrb	$0, %eax, %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpinsrb	$0, (%rax), %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vpinsrw	$0, %eax, %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpinsrw	$0, (%rax), %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermi2w	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermi2w	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermi2w	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermi2w	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermi2w	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermi2w	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermt2w	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermt2w	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermt2w	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermt2w	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermt2w	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermt2w	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmaddubsw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddubsw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmaddubsw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddubsw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmaddubsw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddubsw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmaddwd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddwd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmaddwd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddwd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmaddwd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddwd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaxsb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaxsb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaxsb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaxsw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaxsw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaxsw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaxub	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxub	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaxub	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxub	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaxub	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxub	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaxuw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxuw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaxuw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxuw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaxuw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxuw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpminsb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpminsb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpminsb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpminsw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpminsw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpminsw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpminub	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminub	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpminub	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminub	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpminub	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminub	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpminuw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminuw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpminuw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminuw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpminuw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminuw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovb2m	%zmm0, %k0
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovw2m	%zmm0, %k0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovm2b	%k0, %zmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovm2w	%k0, %zmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbw	%ymm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbw	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbw	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbw	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbw	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbw	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -     vpmovswb	%zmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovswb	%zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -     vpmovswb	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovswb	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -     vpmovuswb	%zmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovuswb	%zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -     vpmovuswb	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovuswb	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -     vpmovwb	%zmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovwb	%zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -     vpmovwb	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovwb	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbw	%ymm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbw	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbw	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbw	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbw	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbw	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmulhrsw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhrsw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmulhrsw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhrsw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmulhrsw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhrsw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmulhuw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhuw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmulhuw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhuw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmulhuw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhuw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmulhw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmulhw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmulhw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmullw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmullw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmullw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vpsadbw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00   1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsadbw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshufb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshufb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshufb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpshufhw	$0, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufhw	$0, (%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpshufhw	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufhw	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpshufhw	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufhw	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpshuflw	$0, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshuflw	$0, (%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpshuflw	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshuflw	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpshuflw	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshuflw	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpslldq	$1, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpslldq	$1, (%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpsllvw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllvw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpsllvw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllvw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpsllvw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllvw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	$0, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	$0, (%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	%xmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	%xmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	%xmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpsravw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsravw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpsravw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsravw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpsravw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsravw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	$0, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	$0, (%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	%xmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	%xmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	%xmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrldq	$1, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrldq	$1, (%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpsrlvw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlvw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpsrlvw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlvw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpsrlvw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlvw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	$0, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	$0, (%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	$0, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	$0, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	$0, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	$0, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	%xmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	%xmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	%xmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubsb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubsb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubsb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubsw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubsw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubsw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubusb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubusb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubusb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubusw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubusw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubusw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsubw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vptestmb	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmb	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestmb	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmb	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vptestmw	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmw	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestmw	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmw	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vptestnmb	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmb	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestnmb	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmb	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vptestnmw	%zmm0, %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmw	(%rax), %zmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestnmw	%zmm0, %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmw	(%rax), %zmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhbw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhbw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhbw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhbw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhbw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhbw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhwd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhwd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhwd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhwd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhwd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhwd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklbw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklbw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklbw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklbw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklbw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklbw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklwd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklwd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklwd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklwd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklwd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklwd	(%rax), %zmm17, %zmm19 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512bwvl.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512bwvl.s
new file mode 100644
index 0000000000000..3ac35f55b7ee4
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512bwvl.s
@@ -0,0 +1,2958 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vdbpsadbw         $0, %xmm16, %xmm17, %xmm19
+vdbpsadbw         $0, (%rax), %xmm17, %xmm19
+vdbpsadbw         $0, %xmm16, %xmm17, %xmm19 {k1}
+vdbpsadbw         $0, (%rax), %xmm17, %xmm19 {k1}
+vdbpsadbw         $0, %xmm16, %xmm17, %xmm19 {z}{k1}
+vdbpsadbw         $0, (%rax), %xmm17, %xmm19 {z}{k1}
+
+vdbpsadbw         $0, %ymm16, %ymm17, %ymm19
+vdbpsadbw         $0, (%rax), %ymm17, %ymm19
+vdbpsadbw         $0, %ymm16, %ymm17, %ymm19 {k1}
+vdbpsadbw         $0, (%rax), %ymm17, %ymm19 {k1}
+vdbpsadbw         $0, %ymm16, %ymm17, %ymm19 {z}{k1}
+vdbpsadbw         $0, (%rax), %ymm17, %ymm19 {z}{k1}
+
+vmovdqu8          %xmm16, %xmm19
+vmovdqu8          (%rax), %xmm19
+vmovdqu8          %xmm16, (%rax)
+vmovdqu8          %xmm16, %xmm19 {k1}
+vmovdqu8          (%rax), %xmm19 {k1}
+vmovdqu8          %xmm16, (%rax) {k1}
+vmovdqu8          %xmm16, %xmm19 {z}{k1}
+vmovdqu8          (%rax), %xmm19 {z}{k1}
+
+vmovdqu8          %ymm16, %ymm19
+vmovdqu8          (%rax), %ymm19
+vmovdqu8          %ymm16, (%rax)
+vmovdqu8          %ymm16, %ymm19 {k1}
+vmovdqu8          (%rax), %ymm19 {k1}
+vmovdqu8          %ymm16, (%rax) {k1}
+vmovdqu8          %ymm16, %ymm19 {z}{k1}
+vmovdqu8          (%rax), %ymm19 {z}{k1}
+
+vmovdqu16         %xmm16, %xmm19
+vmovdqu16         (%rax), %xmm19
+vmovdqu16         %xmm16, (%rax)
+vmovdqu16         %xmm16, %xmm19 {k1}
+vmovdqu16         (%rax), %xmm19 {k1}
+vmovdqu16         %xmm16, (%rax) {k1}
+vmovdqu16         %xmm16, %xmm19 {z}{k1}
+vmovdqu16         (%rax), %xmm19 {z}{k1}
+
+vmovdqu16         %ymm16, %ymm19
+vmovdqu16         (%rax), %ymm19
+vmovdqu16         %ymm16, (%rax)
+vmovdqu16         %ymm16, %ymm19 {k1}
+vmovdqu16         (%rax), %ymm19 {k1}
+vmovdqu16         %ymm16, (%rax) {k1}
+vmovdqu16         %ymm16, %ymm19 {z}{k1}
+vmovdqu16         (%rax), %ymm19 {z}{k1}
+
+vpabsb            %xmm16, %xmm19
+vpabsb            (%rax), %xmm19
+vpabsb            %xmm16, %xmm19 {k1}
+vpabsb            (%rax), %xmm19 {k1}
+vpabsb            %xmm16, %xmm19 {z}{k1}
+vpabsb            (%rax), %xmm19 {z}{k1}
+
+vpabsb            %ymm16, %ymm19
+vpabsb            (%rax), %ymm19
+vpabsb            %ymm16, %ymm19 {k1}
+vpabsb            (%rax), %ymm19 {k1}
+vpabsb            %ymm16, %ymm19 {z}{k1}
+vpabsb            (%rax), %ymm19 {z}{k1}
+
+vpabsw            %xmm16, %xmm19
+vpabsw            (%rax), %xmm19
+vpabsw            %xmm16, %xmm19 {k1}
+vpabsw            (%rax), %xmm19 {k1}
+vpabsw            %xmm16, %xmm19 {z}{k1}
+vpabsw            (%rax), %xmm19 {z}{k1}
+
+vpabsw            %ymm16, %ymm19
+vpabsw            (%rax), %ymm19
+vpabsw            %ymm16, %ymm19 {k1}
+vpabsw            (%rax), %ymm19 {k1}
+vpabsw            %ymm16, %ymm19 {z}{k1}
+vpabsw            (%rax), %ymm19 {z}{k1}
+
+vpackssdw         %xmm16, %xmm17, %xmm19
+vpackssdw         (%rax), %xmm17, %xmm19
+vpackssdw         %xmm16, %xmm17, %xmm19 {k1}
+vpackssdw         (%rax), %xmm17, %xmm19 {k1}
+vpackssdw         %xmm16, %xmm17, %xmm19 {z}{k1}
+vpackssdw         (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpackssdw         %ymm16, %ymm17, %ymm19
+vpackssdw         (%rax), %ymm17, %ymm19
+vpackssdw         %ymm16, %ymm17, %ymm19 {k1}
+vpackssdw         (%rax), %ymm17, %ymm19 {k1}
+vpackssdw         %ymm16, %ymm17, %ymm19 {z}{k1}
+vpackssdw         (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpacksswb         %xmm16, %xmm17, %xmm19
+vpacksswb         (%rax), %xmm17, %xmm19
+vpacksswb         %xmm16, %xmm17, %xmm19 {k1}
+vpacksswb         (%rax), %xmm17, %xmm19 {k1}
+vpacksswb         %xmm16, %xmm17, %xmm19 {z}{k1}
+vpacksswb         (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpacksswb         %ymm16, %ymm17, %ymm19
+vpacksswb         (%rax), %ymm17, %ymm19
+vpacksswb         %ymm16, %ymm17, %ymm19 {k1}
+vpacksswb         (%rax), %ymm17, %ymm19 {k1}
+vpacksswb         %ymm16, %ymm17, %ymm19 {z}{k1}
+vpacksswb         (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpackusdw         %xmm16, %xmm17, %xmm19
+vpackusdw         (%rax), %xmm17, %xmm19
+vpackusdw         %xmm16, %xmm17, %xmm19 {k1}
+vpackusdw         (%rax), %xmm17, %xmm19 {k1}
+vpackusdw         %xmm16, %xmm17, %xmm19 {z}{k1}
+vpackusdw         (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpackusdw         %ymm16, %ymm17, %ymm19
+vpackusdw         (%rax), %ymm17, %ymm19
+vpackusdw         %ymm16, %ymm17, %ymm19 {k1}
+vpackusdw         (%rax), %ymm17, %ymm19 {k1}
+vpackusdw         %ymm16, %ymm17, %ymm19 {z}{k1}
+vpackusdw         (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpackuswb         %xmm16, %xmm17, %xmm19
+vpackuswb         (%rax), %xmm17, %xmm19
+vpackuswb         %xmm16, %xmm17, %xmm19 {k1}
+vpackuswb         (%rax), %xmm17, %xmm19 {k1}
+vpackuswb         %xmm16, %xmm17, %xmm19 {z}{k1}
+vpackuswb         (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpackuswb         %ymm16, %ymm17, %ymm19
+vpackuswb         (%rax), %ymm17, %ymm19
+vpackuswb         %ymm16, %ymm17, %ymm19 {k1}
+vpackuswb         (%rax), %ymm17, %ymm19 {k1}
+vpackuswb         %ymm16, %ymm17, %ymm19 {z}{k1}
+vpackuswb         (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpaddb            %xmm16, %xmm17, %xmm19
+vpaddb            (%rax), %xmm17, %xmm19
+vpaddb            %xmm16, %xmm17, %xmm19 {k1}
+vpaddb            (%rax), %xmm17, %xmm19 {k1}
+vpaddb            %xmm16, %xmm17, %xmm19 {z}{k1}
+vpaddb            (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpaddb            %ymm16, %ymm17, %ymm19
+vpaddb            (%rax), %ymm17, %ymm19
+vpaddb            %ymm16, %ymm17, %ymm19 {k1}
+vpaddb            (%rax), %ymm17, %ymm19 {k1}
+vpaddb            %ymm16, %ymm17, %ymm19 {z}{k1}
+vpaddb            (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpaddsb           %xmm16, %xmm17, %xmm19
+vpaddsb           (%rax), %xmm17, %xmm19
+vpaddsb           %xmm16, %xmm17, %xmm19 {k1}
+vpaddsb           (%rax), %xmm17, %xmm19 {k1}
+vpaddsb           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpaddsb           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpaddsb           %ymm16, %ymm17, %ymm19
+vpaddsb           (%rax), %ymm17, %ymm19
+vpaddsb           %ymm16, %ymm17, %ymm19 {k1}
+vpaddsb           (%rax), %ymm17, %ymm19 {k1}
+vpaddsb           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpaddsb           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpaddsw           %xmm16, %xmm17, %xmm19
+vpaddsw           (%rax), %xmm17, %xmm19
+vpaddsw           %xmm16, %xmm17, %xmm19 {k1}
+vpaddsw           (%rax), %xmm17, %xmm19 {k1}
+vpaddsw           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpaddsw           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpaddsw           %ymm16, %ymm17, %ymm19
+vpaddsw           (%rax), %ymm17, %ymm19
+vpaddsw           %ymm16, %ymm17, %ymm19 {k1}
+vpaddsw           (%rax), %ymm17, %ymm19 {k1}
+vpaddsw           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpaddsw           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpaddusb          %xmm16, %xmm17, %xmm19
+vpaddusb          (%rax), %xmm17, %xmm19
+vpaddusb          %xmm16, %xmm17, %xmm19 {k1}
+vpaddusb          (%rax), %xmm17, %xmm19 {k1}
+vpaddusb          %xmm16, %xmm17, %xmm19 {z}{k1}
+vpaddusb          (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpaddusb          %ymm16, %ymm17, %ymm19
+vpaddusb          (%rax), %ymm17, %ymm19
+vpaddusb          %ymm16, %ymm17, %ymm19 {k1}
+vpaddusb          (%rax), %ymm17, %ymm19 {k1}
+vpaddusb          %ymm16, %ymm17, %ymm19 {z}{k1}
+vpaddusb          (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpaddusw          %xmm16, %xmm17, %xmm19
+vpaddusw          (%rax), %xmm17, %xmm19
+vpaddusw          %xmm16, %xmm17, %xmm19 {k1}
+vpaddusw          (%rax), %xmm17, %xmm19 {k1}
+vpaddusw          %xmm16, %xmm17, %xmm19 {z}{k1}
+vpaddusw          (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpaddusw          %ymm16, %ymm17, %ymm19
+vpaddusw          (%rax), %ymm17, %ymm19
+vpaddusw          %ymm16, %ymm17, %ymm19 {k1}
+vpaddusw          (%rax), %ymm17, %ymm19 {k1}
+vpaddusw          %ymm16, %ymm17, %ymm19 {z}{k1}
+vpaddusw          (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpaddw            %xmm16, %xmm17, %xmm19
+vpaddw            (%rax), %xmm17, %xmm19
+vpaddw            %xmm16, %xmm17, %xmm19 {k1}
+vpaddw            (%rax), %xmm17, %xmm19 {k1}
+vpaddw            %xmm16, %xmm17, %xmm19 {z}{k1}
+vpaddw            (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpaddw            %ymm16, %ymm17, %ymm19
+vpaddw            (%rax), %ymm17, %ymm19
+vpaddw            %ymm16, %ymm17, %ymm19 {k1}
+vpaddw            (%rax), %ymm17, %ymm19 {k1}
+vpaddw            %ymm16, %ymm17, %ymm19 {z}{k1}
+vpaddw            (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpalignr          $1, %xmm16, %xmm17, %xmm19
+vpalignr          $1, (%rax), %xmm17, %xmm19
+vpalignr          $1, %xmm16, %xmm17, %xmm19 {k1}
+vpalignr          $1, (%rax), %xmm17, %xmm19 {k1}
+vpalignr          $1, %xmm16, %xmm17, %xmm19 {z}{k1}
+vpalignr          $1, (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpalignr          $1, %ymm16, %ymm17, %ymm19
+vpalignr          $1, (%rax), %ymm17, %ymm19
+vpalignr          $1, %ymm16, %ymm17, %ymm19 {k1}
+vpalignr          $1, (%rax), %ymm17, %ymm19 {k1}
+vpalignr          $1, %ymm16, %ymm17, %ymm19 {z}{k1}
+vpalignr          $1, (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpavgb            %xmm16, %xmm17, %xmm19
+vpavgb            (%rax), %xmm17, %xmm19
+vpavgb            %xmm16, %xmm17, %xmm19 {k1}
+vpavgb            (%rax), %xmm17, %xmm19 {k1}
+vpavgb            %xmm16, %xmm17, %xmm19 {z}{k1}
+vpavgb            (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpavgb            %ymm16, %ymm17, %ymm19
+vpavgb            (%rax), %ymm17, %ymm19
+vpavgb            %ymm16, %ymm17, %ymm19 {k1}
+vpavgb            (%rax), %ymm17, %ymm19 {k1}
+vpavgb            %ymm16, %ymm17, %ymm19 {z}{k1}
+vpavgb            (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpavgw            %xmm16, %xmm17, %xmm19
+vpavgw            (%rax), %xmm17, %xmm19
+vpavgw            %xmm16, %xmm17, %xmm19 {k1}
+vpavgw            (%rax), %xmm17, %xmm19 {k1}
+vpavgw            %xmm16, %xmm17, %xmm19 {z}{k1}
+vpavgw            (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpavgw            %ymm16, %ymm17, %ymm19
+vpavgw            (%rax), %ymm17, %ymm19
+vpavgw            %ymm16, %ymm17, %ymm19 {k1}
+vpavgw            (%rax), %ymm17, %ymm19 {k1}
+vpavgw            %ymm16, %ymm17, %ymm19 {z}{k1}
+vpavgw            (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpblendmb         %xmm16, %xmm17, %xmm19
+vpblendmb         (%rax), %xmm17, %xmm19
+vpblendmb         %xmm16, %xmm17, %xmm19 {k1}
+vpblendmb         (%rax), %xmm17, %xmm19 {k1}
+vpblendmb         %xmm16, %xmm17, %xmm19 {z}{k1}
+vpblendmb         (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpblendmb         %ymm16, %ymm17, %ymm19
+vpblendmb         (%rax), %ymm17, %ymm19
+vpblendmb         %ymm16, %ymm17, %ymm19 {k1}
+vpblendmb         (%rax), %ymm17, %ymm19 {k1}
+vpblendmb         %ymm16, %ymm17, %ymm19 {z}{k1}
+vpblendmb         (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpblendmw         %xmm16, %xmm17, %xmm19
+vpblendmw         (%rax), %xmm17, %xmm19
+vpblendmw         %xmm16, %xmm17, %xmm19 {k1}
+vpblendmw         (%rax), %xmm17, %xmm19 {k1}
+vpblendmw         %xmm16, %xmm17, %xmm19 {z}{k1}
+vpblendmw         (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpblendmw         %ymm16, %ymm17, %ymm19
+vpblendmw         (%rax), %ymm17, %ymm19
+vpblendmw         %ymm16, %ymm17, %ymm19 {k1}
+vpblendmw         (%rax), %ymm17, %ymm19 {k1}
+vpblendmw         %ymm16, %ymm17, %ymm19 {z}{k1}
+vpblendmw         (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpbroadcastb      %xmm16, %xmm19
+vpbroadcastb      (%rax), %xmm19
+vpbroadcastb      %eax, %xmm19
+vpbroadcastb      %xmm16, %xmm19 {k1}
+vpbroadcastb      (%rax), %xmm19 {k1}
+vpbroadcastb      %eax, %xmm19 {k1}
+vpbroadcastb      %xmm16, %xmm19 {z}{k1}
+vpbroadcastb      (%rax), %xmm19 {z}{k1}
+vpbroadcastb      %eax, %xmm19 {z}{k1}
+
+vpbroadcastb      %xmm16, %ymm19
+vpbroadcastb      (%rax), %ymm19
+vpbroadcastb      %eax, %ymm19
+vpbroadcastb      %xmm16, %ymm19 {k1}
+vpbroadcastb      (%rax), %ymm19 {k1}
+vpbroadcastb      %eax, %ymm19 {k1}
+vpbroadcastb      %xmm16, %ymm19 {z}{k1}
+vpbroadcastb      (%rax), %ymm19 {z}{k1}
+vpbroadcastb      %eax, %ymm19 {z}{k1}
+
+vpbroadcastw      %xmm16, %xmm19
+vpbroadcastw      (%rax), %xmm19
+vpbroadcastw      %eax, %xmm19
+vpbroadcastw      %xmm16, %xmm19 {k1}
+vpbroadcastw      (%rax), %xmm19 {k1}
+vpbroadcastw      %eax, %xmm19 {k1}
+vpbroadcastw      %xmm16, %xmm19 {z}{k1}
+vpbroadcastw      (%rax), %xmm19 {z}{k1}
+vpbroadcastw      %eax, %xmm19 {z}{k1}
+
+vpbroadcastw      %xmm16, %ymm19
+vpbroadcastw      (%rax), %ymm19
+vpbroadcastw      %eax, %ymm19
+vpbroadcastw      %xmm16, %ymm19 {k1}
+vpbroadcastw      (%rax), %ymm19 {k1}
+vpbroadcastw      %eax, %ymm19 {k1}
+vpbroadcastw      %xmm16, %ymm19 {z}{k1}
+vpbroadcastw      (%rax), %ymm19 {z}{k1}
+vpbroadcastw      %eax, %ymm19 {z}{k1}
+
+vpcmpb            $0, %xmm0, %xmm1, %k2
+vpcmpb            $0, (%rax), %xmm1, %k2
+vpcmpb            $0, %xmm0, %xmm1, %k2 {k3}
+vpcmpb            $0, (%rax), %xmm1, %k2 {k3}
+
+vpcmpb            $0, %ymm0, %ymm1, %k2
+vpcmpb            $0, (%rax), %ymm1, %k2
+vpcmpb            $0, %ymm0, %ymm1, %k2 {k3}
+vpcmpb            $0, (%rax), %ymm1, %k2 {k3}
+
+vpcmpeqb          %xmm0, %xmm1, %k2
+vpcmpeqb          (%rax), %xmm1, %k2
+vpcmpeqb          %xmm0, %xmm1, %k2 {k3}
+vpcmpeqb          (%rax), %xmm1, %k2 {k3}
+
+vpcmpeqb          %ymm0, %ymm1, %k2
+vpcmpeqb          (%rax), %ymm1, %k2
+vpcmpeqb          %ymm0, %ymm1, %k2 {k3}
+vpcmpeqb          (%rax), %ymm1, %k2 {k3}
+
+vpcmpeqw          %xmm0, %xmm1, %k2
+vpcmpeqw          (%rax), %xmm1, %k2
+vpcmpeqw          %xmm0, %xmm1, %k2 {k3}
+vpcmpeqw          (%rax), %xmm1, %k2 {k3}
+
+vpcmpeqw          %ymm0, %ymm1, %k2
+vpcmpeqw          (%rax), %ymm1, %k2
+vpcmpeqw          %ymm0, %ymm1, %k2 {k3}
+vpcmpeqw          (%rax), %ymm1, %k2 {k3}
+
+vpcmpgtb          %xmm0, %xmm1, %k2
+vpcmpgtb          (%rax), %xmm1, %k2
+vpcmpgtb          %xmm0, %xmm1, %k2 {k3}
+vpcmpgtb          (%rax), %xmm1, %k2 {k3}
+
+vpcmpgtb          %ymm0, %ymm1, %k2
+vpcmpgtb          (%rax), %ymm1, %k2
+vpcmpgtb          %ymm0, %ymm1, %k2 {k3}
+vpcmpgtb          (%rax), %ymm1, %k2 {k3}
+
+vpcmpgtw          %xmm0, %xmm1, %k2
+vpcmpgtw          (%rax), %xmm1, %k2
+vpcmpgtw          %xmm0, %xmm1, %k2 {k3}
+vpcmpgtw          (%rax), %xmm1, %k2 {k3}
+
+vpcmpgtw          %ymm0, %ymm1, %k2
+vpcmpgtw          (%rax), %ymm1, %k2
+vpcmpgtw          %ymm0, %ymm1, %k2 {k3}
+vpcmpgtw          (%rax), %ymm1, %k2 {k3}
+
+vpcmpub           $0, %xmm0, %xmm1, %k2
+vpcmpub           $0, (%rax), %xmm1, %k2
+vpcmpub           $0, %xmm0, %xmm1, %k2 {k3}
+vpcmpub           $0, (%rax), %xmm1, %k2 {k3}
+
+vpcmpub           $0, %ymm0, %ymm1, %k2
+vpcmpub           $0, (%rax), %ymm1, %k2
+vpcmpub           $0, %ymm0, %ymm1, %k2 {k3}
+vpcmpub           $0, (%rax), %ymm1, %k2 {k3}
+
+vpcmpuw           $0, %xmm0, %xmm1, %k2
+vpcmpuw           $0, (%rax), %xmm1, %k2
+vpcmpuw           $0, %xmm0, %xmm1, %k2 {k3}
+vpcmpuw           $0, (%rax), %xmm1, %k2 {k3}
+
+vpcmpuw           $0, %ymm0, %ymm1, %k2
+vpcmpuw           $0, (%rax), %ymm1, %k2
+vpcmpuw           $0, %ymm0, %ymm1, %k2 {k3}
+vpcmpuw           $0, (%rax), %ymm1, %k2 {k3}
+
+vpcmpw            $0, %xmm0, %xmm1, %k2
+vpcmpw            $0, (%rax), %xmm1, %k2
+vpcmpw            $0, %xmm0, %xmm1, %k2 {k3}
+vpcmpw            $0, (%rax), %xmm1, %k2 {k3}
+
+vpcmpw            $0, %ymm0, %ymm1, %k2
+vpcmpw            $0, (%rax), %ymm1, %k2
+vpcmpw            $0, %ymm0, %ymm1, %k2 {k3}
+vpcmpw            $0, (%rax), %ymm1, %k2 {k3}
+
+vpermw            %xmm16, %xmm17, %xmm19
+vpermw            (%rax), %xmm17, %xmm19
+vpermw            %xmm16, %xmm17, %xmm19 {k1}
+vpermw            (%rax), %xmm17, %xmm19 {k1}
+vpermw            %xmm16, %xmm17, %xmm19 {z}{k1}
+vpermw            (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpermw            %ymm16, %ymm17, %ymm19
+vpermw            (%rax), %ymm17, %ymm19
+vpermw            %ymm16, %ymm17, %ymm19 {k1}
+vpermw            (%rax), %ymm17, %ymm19 {k1}
+vpermw            %ymm16, %ymm17, %ymm19 {z}{k1}
+vpermw            (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpermi2w          %xmm16, %xmm17, %xmm19
+vpermi2w          (%rax), %xmm17, %xmm19
+vpermi2w          %xmm16, %xmm17, %xmm19 {k1}
+vpermi2w          (%rax), %xmm17, %xmm19 {k1}
+vpermi2w          %xmm16, %xmm17, %xmm19 {z}{k1}
+vpermi2w          (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpermi2w          %ymm16, %ymm17, %ymm19
+vpermi2w          (%rax), %ymm17, %ymm19
+vpermi2w          %ymm16, %ymm17, %ymm19 {k1}
+vpermi2w          (%rax), %ymm17, %ymm19 {k1}
+vpermi2w          %ymm16, %ymm17, %ymm19 {z}{k1}
+vpermi2w          (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpermt2w          %xmm16, %xmm17, %xmm19
+vpermt2w          (%rax), %xmm17, %xmm19
+vpermt2w          %xmm16, %xmm17, %xmm19 {k1}
+vpermt2w          (%rax), %xmm17, %xmm19 {k1}
+vpermt2w          %xmm16, %xmm17, %xmm19 {z}{k1}
+vpermt2w          (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpermt2w          %ymm16, %ymm17, %ymm19
+vpermt2w          (%rax), %ymm17, %ymm19
+vpermt2w          %ymm16, %ymm17, %ymm19 {k1}
+vpermt2w          (%rax), %ymm17, %ymm19 {k1}
+vpermt2w          %ymm16, %ymm17, %ymm19 {z}{k1}
+vpermt2w          (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpmaddubsw        %xmm16, %xmm17, %xmm19
+vpmaddubsw        (%rax), %xmm17, %xmm19
+vpmaddubsw        %xmm16, %xmm17, %xmm19 {k1}
+vpmaddubsw        (%rax), %xmm17, %xmm19 {k1}
+vpmaddubsw        %xmm16, %xmm17, %xmm19 {z}{k1}
+vpmaddubsw        (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpmaddubsw        %ymm16, %ymm17, %ymm19
+vpmaddubsw        (%rax), %ymm17, %ymm19
+vpmaddubsw        %ymm16, %ymm17, %ymm19 {k1}
+vpmaddubsw        (%rax), %ymm17, %ymm19 {k1}
+vpmaddubsw        %ymm16, %ymm17, %ymm19 {z}{k1}
+vpmaddubsw        (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpmaddwd          %xmm16, %xmm17, %xmm19
+vpmaddwd          (%rax), %xmm17, %xmm19
+vpmaddwd          %xmm16, %xmm17, %xmm19 {k1}
+vpmaddwd          (%rax), %xmm17, %xmm19 {k1}
+vpmaddwd          %xmm16, %xmm17, %xmm19 {z}{k1}
+vpmaddwd          (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpmaddwd          %ymm16, %ymm17, %ymm19
+vpmaddwd          (%rax), %ymm17, %ymm19
+vpmaddwd          %ymm16, %ymm17, %ymm19 {k1}
+vpmaddwd          (%rax), %ymm17, %ymm19 {k1}
+vpmaddwd          %ymm16, %ymm17, %ymm19 {z}{k1}
+vpmaddwd          (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpmaxsb           %xmm16, %xmm17, %xmm19
+vpmaxsb           (%rax), %xmm17, %xmm19
+vpmaxsb           %xmm16, %xmm17, %xmm19 {k1}
+vpmaxsb           (%rax), %xmm17, %xmm19 {k1}
+vpmaxsb           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpmaxsb           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpmaxsb           %ymm16, %ymm17, %ymm19
+vpmaxsb           (%rax), %ymm17, %ymm19
+vpmaxsb           %ymm16, %ymm17, %ymm19 {k1}
+vpmaxsb           (%rax), %ymm17, %ymm19 {k1}
+vpmaxsb           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpmaxsb           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpmaxsw           %xmm16, %xmm17, %xmm19
+vpmaxsw           (%rax), %xmm17, %xmm19
+vpmaxsw           %xmm16, %xmm17, %xmm19 {k1}
+vpmaxsw           (%rax), %xmm17, %xmm19 {k1}
+vpmaxsw           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpmaxsw           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpmaxsw           %ymm16, %ymm17, %ymm19
+vpmaxsw           (%rax), %ymm17, %ymm19
+vpmaxsw           %ymm16, %ymm17, %ymm19 {k1}
+vpmaxsw           (%rax), %ymm17, %ymm19 {k1}
+vpmaxsw           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpmaxsw           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpmaxub           %xmm16, %xmm17, %xmm19
+vpmaxub           (%rax), %xmm17, %xmm19
+vpmaxub           %xmm16, %xmm17, %xmm19 {k1}
+vpmaxub           (%rax), %xmm17, %xmm19 {k1}
+vpmaxub           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpmaxub           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpmaxub           %ymm16, %ymm17, %ymm19
+vpmaxub           (%rax), %ymm17, %ymm19
+vpmaxub           %ymm16, %ymm17, %ymm19 {k1}
+vpmaxub           (%rax), %ymm17, %ymm19 {k1}
+vpmaxub           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpmaxub           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpmaxuw           %xmm16, %xmm17, %xmm19
+vpmaxuw           (%rax), %xmm17, %xmm19
+vpmaxuw           %xmm16, %xmm17, %xmm19 {k1}
+vpmaxuw           (%rax), %xmm17, %xmm19 {k1}
+vpmaxuw           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpmaxuw           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpmaxuw           %ymm16, %ymm17, %ymm19
+vpmaxuw           (%rax), %ymm17, %ymm19
+vpmaxuw           %ymm16, %ymm17, %ymm19 {k1}
+vpmaxuw           (%rax), %ymm17, %ymm19 {k1}
+vpmaxuw           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpmaxuw           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpminsb           %xmm16, %xmm17, %xmm19
+vpminsb           (%rax), %xmm17, %xmm19
+vpminsb           %xmm16, %xmm17, %xmm19 {k1}
+vpminsb           (%rax), %xmm17, %xmm19 {k1}
+vpminsb           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpminsb           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpminsb           %ymm16, %ymm17, %ymm19
+vpminsb           (%rax), %ymm17, %ymm19
+vpminsb           %ymm16, %ymm17, %ymm19 {k1}
+vpminsb           (%rax), %ymm17, %ymm19 {k1}
+vpminsb           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpminsb           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpminsw           %xmm16, %xmm17, %xmm19
+vpminsw           (%rax), %xmm17, %xmm19
+vpminsw           %xmm16, %xmm17, %xmm19 {k1}
+vpminsw           (%rax), %xmm17, %xmm19 {k1}
+vpminsw           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpminsw           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpminsw           %ymm16, %ymm17, %ymm19
+vpminsw           (%rax), %ymm17, %ymm19
+vpminsw           %ymm16, %ymm17, %ymm19 {k1}
+vpminsw           (%rax), %ymm17, %ymm19 {k1}
+vpminsw           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpminsw           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpminub           %xmm16, %xmm17, %xmm19
+vpminub           (%rax), %xmm17, %xmm19
+vpminub           %xmm16, %xmm17, %xmm19 {k1}
+vpminub           (%rax), %xmm17, %xmm19 {k1}
+vpminub           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpminub           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpminub           %ymm16, %ymm17, %ymm19
+vpminub           (%rax), %ymm17, %ymm19
+vpminub           %ymm16, %ymm17, %ymm19 {k1}
+vpminub           (%rax), %ymm17, %ymm19 {k1}
+vpminub           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpminub           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpminuw           %xmm16, %xmm17, %xmm19
+vpminuw           (%rax), %xmm17, %xmm19
+vpminuw           %xmm16, %xmm17, %xmm19 {k1}
+vpminuw           (%rax), %xmm17, %xmm19 {k1}
+vpminuw           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpminuw           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpminuw           %ymm16, %ymm17, %ymm19
+vpminuw           (%rax), %ymm17, %ymm19
+vpminuw           %ymm16, %ymm17, %ymm19 {k1}
+vpminuw           (%rax), %ymm17, %ymm19 {k1}
+vpminuw           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpminuw           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpmovb2m          %xmm0, %k0
+vpmovw2m          %xmm0, %k0
+
+vpmovb2m          %ymm0, %k0
+vpmovw2m          %ymm0, %k0
+
+vpmovm2b          %k0, %xmm0
+vpmovm2w          %k0, %xmm0
+
+vpmovm2b          %k0, %ymm0
+vpmovm2w          %k0, %ymm0
+
+vpmovsxbw         %xmm16, %xmm19
+vpmovsxbw         (%rax), %xmm19
+vpmovsxbw         %xmm16, %xmm19 {k1}
+vpmovsxbw         (%rax), %xmm19 {k1}
+vpmovsxbw         %xmm16, %xmm19 {z}{k1}
+vpmovsxbw         (%rax), %xmm19 {z}{k1}
+
+vpmovsxbw         %xmm16, %ymm19
+vpmovsxbw         (%rax), %ymm19
+vpmovsxbw         %xmm16, %ymm19 {k1}
+vpmovsxbw         (%rax), %ymm19 {k1}
+vpmovsxbw         %xmm16, %ymm19 {z}{k1}
+vpmovsxbw         (%rax), %ymm19 {z}{k1}
+
+vpmovswb          %xmm16, %xmm19
+vpmovswb          %xmm16, (%rax)
+vpmovswb          %xmm16, %xmm19 {k1}
+vpmovswb          %xmm16, (%rax) {k1}
+
+vpmovswb          %ymm16, %xmm19
+vpmovswb          %ymm16, (%rax)
+vpmovswb          %ymm16, %xmm19 {k1}
+vpmovswb          %ymm16, (%rax) {k1}
+
+vpmovuswb         %xmm16, %xmm19
+vpmovuswb         %xmm16, (%rax)
+vpmovuswb         %xmm16, %xmm19 {k1}
+vpmovuswb         %xmm16, (%rax) {k1}
+
+vpmovuswb         %ymm16, %xmm19
+vpmovuswb         %ymm16, (%rax)
+vpmovuswb         %ymm16, %xmm19 {k1}
+vpmovuswb         %ymm16, (%rax) {k1}
+
+vpmovwb           %xmm16, %xmm19
+vpmovwb           %xmm16, (%rax)
+vpmovwb           %xmm16, %xmm19 {k1}
+vpmovwb           %xmm16, (%rax) {k1}
+
+vpmovwb           %ymm16, %xmm19
+vpmovwb           %ymm16, (%rax)
+vpmovwb           %ymm16, %xmm19 {k1}
+vpmovwb           %ymm16, (%rax) {k1}
+
+vpmovzxbw         %xmm16, %xmm19
+vpmovzxbw         (%rax), %xmm19
+vpmovzxbw         %xmm16, %xmm19 {k1}
+vpmovzxbw         (%rax), %xmm19 {k1}
+vpmovzxbw         %xmm16, %xmm19 {z}{k1}
+vpmovzxbw         (%rax), %xmm19 {z}{k1}
+
+vpmovzxbw         %xmm16, %ymm19
+vpmovzxbw         (%rax), %ymm19
+vpmovzxbw         %xmm16, %ymm19 {k1}
+vpmovzxbw         (%rax), %ymm19 {k1}
+vpmovzxbw         %xmm16, %ymm19 {z}{k1}
+vpmovzxbw         (%rax), %ymm19 {z}{k1}
+
+vpmulhrsw         %xmm16, %xmm17, %xmm19
+vpmulhrsw         (%rax), %xmm17, %xmm19
+vpmulhrsw         %xmm16, %xmm17, %xmm19 {k1}
+vpmulhrsw         (%rax), %xmm17, %xmm19 {k1}
+vpmulhrsw         %xmm16, %xmm17, %xmm19 {z}{k1}
+vpmulhrsw         (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpmulhrsw         %ymm16, %ymm17, %ymm19
+vpmulhrsw         (%rax), %ymm17, %ymm19
+vpmulhrsw         %ymm16, %ymm17, %ymm19 {k1}
+vpmulhrsw         (%rax), %ymm17, %ymm19 {k1}
+vpmulhrsw         %ymm16, %ymm17, %ymm19 {z}{k1}
+vpmulhrsw         (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpmulhuw          %xmm16, %xmm17, %xmm19
+vpmulhuw          (%rax), %xmm17, %xmm19
+vpmulhuw          %xmm16, %xmm17, %xmm19 {k1}
+vpmulhuw          (%rax), %xmm17, %xmm19 {k1}
+vpmulhuw          %xmm16, %xmm17, %xmm19 {z}{k1}
+vpmulhuw          (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpmulhuw          %ymm16, %ymm17, %ymm19
+vpmulhuw          (%rax), %ymm17, %ymm19
+vpmulhuw          %ymm16, %ymm17, %ymm19 {k1}
+vpmulhuw          (%rax), %ymm17, %ymm19 {k1}
+vpmulhuw          %ymm16, %ymm17, %ymm19 {z}{k1}
+vpmulhuw          (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpmulhw           %xmm16, %xmm17, %xmm19
+vpmulhw           (%rax), %xmm17, %xmm19
+vpmulhw           %xmm16, %xmm17, %xmm19 {k1}
+vpmulhw           (%rax), %xmm17, %xmm19 {k1}
+vpmulhw           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpmulhw           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpmulhw           %ymm16, %ymm17, %ymm19
+vpmulhw           (%rax), %ymm17, %ymm19
+vpmulhw           %ymm16, %ymm17, %ymm19 {k1}
+vpmulhw           (%rax), %ymm17, %ymm19 {k1}
+vpmulhw           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpmulhw           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpmullw           %xmm16, %xmm17, %xmm19
+vpmullw           (%rax), %xmm17, %xmm19
+vpmullw           %xmm16, %xmm17, %xmm19 {k1}
+vpmullw           (%rax), %xmm17, %xmm19 {k1}
+vpmullw           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpmullw           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpmullw           %ymm16, %ymm17, %ymm19
+vpmullw           (%rax), %ymm17, %ymm19
+vpmullw           %ymm16, %ymm17, %ymm19 {k1}
+vpmullw           (%rax), %ymm17, %ymm19 {k1}
+vpmullw           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpmullw           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpsadbw           %xmm16, %xmm17, %xmm19
+vpsadbw           (%rax), %xmm17, %xmm19
+
+vpsadbw           %ymm16, %ymm17, %ymm19
+vpsadbw           (%rax), %ymm17, %ymm19
+
+vpshufb           %xmm16, %xmm17, %xmm19
+vpshufb           (%rax), %xmm17, %xmm19
+vpshufb           %xmm16, %xmm17, %xmm19 {k1}
+vpshufb           (%rax), %xmm17, %xmm19 {k1}
+vpshufb           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpshufb           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpshufb           %ymm16, %ymm17, %ymm19
+vpshufb           (%rax), %ymm17, %ymm19
+vpshufb           %ymm16, %ymm17, %ymm19 {k1}
+vpshufb           (%rax), %ymm17, %ymm19 {k1}
+vpshufb           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpshufb           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpshufhw          $0, %xmm16, %xmm19
+vpshufhw          $0, (%rax), %xmm19
+vpshufhw          $0, %xmm16, %xmm19 {k1}
+vpshufhw          $0, (%rax), %xmm19 {k1}
+vpshufhw          $0, %xmm16, %xmm19 {z}{k1}
+vpshufhw          $0, (%rax), %xmm19 {z}{k1}
+
+vpshufhw          $0, %ymm16, %ymm19
+vpshufhw          $0, (%rax), %ymm19
+vpshufhw          $0, %ymm16, %ymm19 {k1}
+vpshufhw          $0, (%rax), %ymm19 {k1}
+vpshufhw          $0, %ymm16, %ymm19 {z}{k1}
+vpshufhw          $0, (%rax), %ymm19 {z}{k1}
+
+vpshuflw          $0, %xmm16, %xmm19
+vpshuflw          $0, (%rax), %xmm19
+vpshuflw          $0, %xmm16, %xmm19 {k1}
+vpshuflw          $0, (%rax), %xmm19 {k1}
+vpshuflw          $0, %xmm16, %xmm19 {z}{k1}
+vpshuflw          $0, (%rax), %xmm19 {z}{k1}
+
+vpshuflw          $0, %ymm16, %ymm19
+vpshuflw          $0, (%rax), %ymm19
+vpshuflw          $0, %ymm16, %ymm19 {k1}
+vpshuflw          $0, (%rax), %ymm19 {k1}
+vpshuflw          $0, %ymm16, %ymm19 {z}{k1}
+vpshuflw          $0, (%rax), %ymm19 {z}{k1}
+
+vpslldq           $1, %xmm16, %xmm19
+vpslldq           $1, (%rax), %xmm19
+
+vpslldq           $1, %ymm16, %ymm19
+vpslldq           $1, (%rax), %ymm19
+
+vpsllvw           %xmm16, %xmm17, %xmm19
+vpsllvw           (%rax), %xmm17, %xmm19
+vpsllvw           %xmm16, %xmm17, %xmm19 {k1}
+vpsllvw           (%rax), %xmm17, %xmm19 {k1}
+vpsllvw           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpsllvw           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpsllvw           %ymm16, %ymm17, %ymm19
+vpsllvw           (%rax), %ymm17, %ymm19
+vpsllvw           %ymm16, %ymm17, %ymm19 {k1}
+vpsllvw           (%rax), %ymm17, %ymm19 {k1}
+vpsllvw           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpsllvw           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpsllw            $0, %xmm16, %xmm19
+vpsllw            $0, (%rax), %xmm19
+vpsllw            $0, %xmm16, %xmm19 {k1}
+vpsllw            $0, (%rax), %xmm19 {k1}
+vpsllw            $0, %xmm16, %xmm19 {z}{k1}
+vpsllw            $0, (%rax), %xmm19 {z}{k1}
+
+vpsllw            $0, %ymm16, %ymm19
+vpsllw            $0, (%rax), %ymm19
+vpsllw            $0, %ymm16, %ymm19 {k1}
+vpsllw            $0, (%rax), %ymm19 {k1}
+vpsllw            $0, %ymm16, %ymm19 {z}{k1}
+vpsllw            $0, (%rax), %ymm19 {z}{k1}
+
+vpsllw            %xmm16, %xmm17, %xmm19
+vpsllw            (%rax), %xmm17, %xmm19
+vpsllw            %xmm16, %xmm17, %xmm19 {k1}
+vpsllw            (%rax), %xmm17, %xmm19 {k1}
+vpsllw            %xmm16, %xmm17, %xmm19 {z}{k1}
+vpsllw            (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpsllw            %xmm16, %ymm17, %ymm19
+vpsllw            (%rax), %ymm17, %ymm19
+vpsllw            %xmm16, %ymm17, %ymm19 {k1}
+vpsllw            (%rax), %ymm17, %ymm19 {k1}
+vpsllw            %xmm16, %ymm17, %ymm19 {z}{k1}
+vpsllw            (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpsravw           %xmm16, %xmm17, %xmm19
+vpsravw           (%rax), %xmm17, %xmm19
+vpsravw           %xmm16, %xmm17, %xmm19 {k1}
+vpsravw           (%rax), %xmm17, %xmm19 {k1}
+vpsravw           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpsravw           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpsravw           %ymm16, %ymm17, %ymm19
+vpsravw           (%rax), %ymm17, %ymm19
+vpsravw           %ymm16, %ymm17, %ymm19 {k1}
+vpsravw           (%rax), %ymm17, %ymm19 {k1}
+vpsravw           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpsravw           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpsraw            $0, %xmm16, %xmm19
+vpsraw            $0, (%rax), %xmm19
+vpsraw            $0, %xmm16, %xmm19 {k1}
+vpsraw            $0, (%rax), %xmm19 {k1}
+vpsraw            $0, %xmm16, %xmm19 {z}{k1}
+vpsraw            $0, (%rax), %xmm19 {z}{k1}
+
+vpsraw            $0, %ymm16, %ymm19
+vpsraw            $0, (%rax), %ymm19
+vpsraw            $0, %ymm16, %ymm19 {k1}
+vpsraw            $0, (%rax), %ymm19 {k1}
+vpsraw            $0, %ymm16, %ymm19 {z}{k1}
+vpsraw            $0, (%rax), %ymm19 {z}{k1}
+
+vpsraw            %xmm16, %xmm17, %xmm19
+vpsraw            (%rax), %xmm17, %xmm19
+vpsraw            %xmm16, %xmm17, %xmm19 {k1}
+vpsraw            (%rax), %xmm17, %xmm19 {k1}
+vpsraw            %xmm16, %xmm17, %xmm19 {z}{k1}
+vpsraw            (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpsraw            %xmm16, %ymm17, %ymm19
+vpsraw            (%rax), %ymm17, %ymm19
+vpsraw            %xmm16, %ymm17, %ymm19 {k1}
+vpsraw            (%rax), %ymm17, %ymm19 {k1}
+vpsraw            %xmm16, %ymm17, %ymm19 {z}{k1}
+vpsraw            (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpsrldq           $1, %xmm16, %xmm19
+vpsrldq           $1, (%rax), %xmm19
+
+vpsrldq           $1, %ymm16, %ymm19
+vpsrldq           $1, (%rax), %ymm19
+
+vpsrlvw           %xmm16, %xmm17, %xmm19
+vpsrlvw           (%rax), %xmm17, %xmm19
+vpsrlvw           %xmm16, %xmm17, %xmm19 {k1}
+vpsrlvw           (%rax), %xmm17, %xmm19 {k1}
+vpsrlvw           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpsrlvw           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpsrlvw           %ymm16, %ymm17, %ymm19
+vpsrlvw           (%rax), %ymm17, %ymm19
+vpsrlvw           %ymm16, %ymm17, %ymm19 {k1}
+vpsrlvw           (%rax), %ymm17, %ymm19 {k1}
+vpsrlvw           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpsrlvw           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpsrlw            $0, %xmm16, %xmm19
+vpsrlw            $0, (%rax), %xmm19
+vpsrlw            $0, %xmm16, %xmm19 {k1}
+vpsrlw            $0, (%rax), %xmm19 {k1}
+vpsrlw            $0, %xmm16, %xmm19 {z}{k1}
+vpsrlw            $0, (%rax), %xmm19 {z}{k1}
+
+vpsrlw            $0, %ymm16, %ymm19
+vpsrlw            $0, (%rax), %ymm19
+vpsrlw            $0, %ymm16, %ymm19 {k1}
+vpsrlw            $0, (%rax), %ymm19 {k1}
+vpsrlw            $0, %ymm16, %ymm19 {z}{k1}
+vpsrlw            $0, (%rax), %ymm19 {z}{k1}
+
+vpsrlw            %xmm16, %xmm17, %xmm19
+vpsrlw            (%rax), %xmm17, %xmm19
+vpsrlw            %xmm16, %xmm17, %xmm19 {k1}
+vpsrlw            (%rax), %xmm17, %xmm19 {k1}
+vpsrlw            %xmm16, %xmm17, %xmm19 {z}{k1}
+vpsrlw            (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpsrlw            %xmm16, %ymm17, %ymm19
+vpsrlw            (%rax), %ymm17, %ymm19
+vpsrlw            %xmm16, %ymm17, %ymm19 {k1}
+vpsrlw            (%rax), %ymm17, %ymm19 {k1}
+vpsrlw            %xmm16, %ymm17, %ymm19 {z}{k1}
+vpsrlw            (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpsubb            %xmm16, %xmm17, %xmm19
+vpsubb            (%rax), %xmm17, %xmm19
+vpsubb            %xmm16, %xmm17, %xmm19 {k1}
+vpsubb            (%rax), %xmm17, %xmm19 {k1}
+vpsubb            %xmm16, %xmm17, %xmm19 {z}{k1}
+vpsubb            (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpsubb            %ymm16, %ymm17, %ymm19
+vpsubb            (%rax), %ymm17, %ymm19
+vpsubb            %ymm16, %ymm17, %ymm19 {k1}
+vpsubb            (%rax), %ymm17, %ymm19 {k1}
+vpsubb            %ymm16, %ymm17, %ymm19 {z}{k1}
+vpsubb            (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpsubsb           %xmm16, %xmm17, %xmm19
+vpsubsb           (%rax), %xmm17, %xmm19
+vpsubsb           %xmm16, %xmm17, %xmm19 {k1}
+vpsubsb           (%rax), %xmm17, %xmm19 {k1}
+vpsubsb           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpsubsb           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpsubsb           %ymm16, %ymm17, %ymm19
+vpsubsb           (%rax), %ymm17, %ymm19
+vpsubsb           %ymm16, %ymm17, %ymm19 {k1}
+vpsubsb           (%rax), %ymm17, %ymm19 {k1}
+vpsubsb           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpsubsb           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpsubsw           %xmm16, %xmm17, %xmm19
+vpsubsw           (%rax), %xmm17, %xmm19
+vpsubsw           %xmm16, %xmm17, %xmm19 {k1}
+vpsubsw           (%rax), %xmm17, %xmm19 {k1}
+vpsubsw           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpsubsw           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpsubsw           %ymm16, %ymm17, %ymm19
+vpsubsw           (%rax), %ymm17, %ymm19
+vpsubsw           %ymm16, %ymm17, %ymm19 {k1}
+vpsubsw           (%rax), %ymm17, %ymm19 {k1}
+vpsubsw           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpsubsw           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpsubusb          %xmm16, %xmm17, %xmm19
+vpsubusb          (%rax), %xmm17, %xmm19
+vpsubusb          %xmm16, %xmm17, %xmm19 {k1}
+vpsubusb          (%rax), %xmm17, %xmm19 {k1}
+vpsubusb          %xmm16, %xmm17, %xmm19 {z}{k1}
+vpsubusb          (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpsubusb          %ymm16, %ymm17, %ymm19
+vpsubusb          (%rax), %ymm17, %ymm19
+vpsubusb          %ymm16, %ymm17, %ymm19 {k1}
+vpsubusb          (%rax), %ymm17, %ymm19 {k1}
+vpsubusb          %ymm16, %ymm17, %ymm19 {z}{k1}
+vpsubusb          (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpsubusw          %xmm16, %xmm17, %xmm19
+vpsubusw          (%rax), %xmm17, %xmm19
+vpsubusw          %xmm16, %xmm17, %xmm19 {k1}
+vpsubusw          (%rax), %xmm17, %xmm19 {k1}
+vpsubusw          %xmm16, %xmm17, %xmm19 {z}{k1}
+vpsubusw          (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpsubusw          %ymm16, %ymm17, %ymm19
+vpsubusw          (%rax), %ymm17, %ymm19
+vpsubusw          %ymm16, %ymm17, %ymm19 {k1}
+vpsubusw          (%rax), %ymm17, %ymm19 {k1}
+vpsubusw          %ymm16, %ymm17, %ymm19 {z}{k1}
+vpsubusw          (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpsubw            %xmm16, %xmm17, %xmm19
+vpsubw            (%rax), %xmm17, %xmm19
+vpsubw            %xmm16, %xmm17, %xmm19 {k1}
+vpsubw            (%rax), %xmm17, %xmm19 {k1}
+vpsubw            %xmm16, %xmm17, %xmm19 {z}{k1}
+vpsubw            (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpsubw            %ymm16, %ymm17, %ymm19
+vpsubw            (%rax), %ymm17, %ymm19
+vpsubw            %ymm16, %ymm17, %ymm19 {k1}
+vpsubw            (%rax), %ymm17, %ymm19 {k1}
+vpsubw            %ymm16, %ymm17, %ymm19 {z}{k1}
+vpsubw            (%rax), %ymm17, %ymm19 {z}{k1}
+
+vptestmb          %xmm0, %xmm1, %k2
+vptestmb          (%rax), %xmm1, %k2
+vptestmb          %xmm0, %xmm1, %k2 {k3}
+vptestmb          (%rax), %xmm1, %k2 {k3}
+
+vptestmb          %ymm0, %ymm1, %k2
+vptestmb          (%rax), %ymm1, %k2
+vptestmb          %ymm0, %ymm1, %k2 {k3}
+vptestmb          (%rax), %ymm1, %k2 {k3}
+
+vptestmw          %xmm0, %xmm1, %k2
+vptestmw          (%rax), %xmm1, %k2
+vptestmw          %xmm0, %xmm1, %k2 {k3}
+vptestmw          (%rax), %xmm1, %k2 {k3}
+
+vptestmw          %ymm0, %ymm1, %k2
+vptestmw          (%rax), %ymm1, %k2
+vptestmw          %ymm0, %ymm1, %k2 {k3}
+vptestmw          (%rax), %ymm1, %k2 {k3}
+
+vptestnmb         %xmm0, %xmm1, %k2
+vptestnmb         (%rax), %xmm1, %k2
+vptestnmb         %xmm0, %xmm1, %k2 {k3}
+vptestnmb         (%rax), %xmm1, %k2 {k3}
+
+vptestnmb         %ymm0, %ymm1, %k2
+vptestnmb         (%rax), %ymm1, %k2
+vptestnmb         %ymm0, %ymm1, %k2 {k3}
+vptestnmb         (%rax), %ymm1, %k2 {k3}
+
+vptestnmw         %xmm0, %xmm1, %k2
+vptestnmw         (%rax), %xmm1, %k2
+vptestnmw         %xmm0, %xmm1, %k2 {k3}
+vptestnmw         (%rax), %xmm1, %k2 {k3}
+
+vptestnmw         %ymm0, %ymm1, %k2
+vptestnmw         (%rax), %ymm1, %k2
+vptestnmw         %ymm0, %ymm1, %k2 {k3}
+vptestnmw         (%rax), %ymm1, %k2 {k3}
+
+vpunpckhbw         %xmm16, %xmm17, %xmm19
+vpunpckhbw         (%rax), %xmm17, %xmm19
+vpunpckhbw         %xmm16, %xmm17, %xmm19 {k1}
+vpunpckhbw         (%rax), %xmm17, %xmm19 {k1}
+vpunpckhbw         %xmm16, %xmm17, %xmm19 {z}{k1}
+vpunpckhbw         (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpunpckhbw         %ymm16, %ymm17, %ymm19
+vpunpckhbw         (%rax), %ymm17, %ymm19
+vpunpckhbw         %ymm16, %ymm17, %ymm19 {k1}
+vpunpckhbw         (%rax), %ymm17, %ymm19 {k1}
+vpunpckhbw         %ymm16, %ymm17, %ymm19 {z}{k1}
+vpunpckhbw         (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpunpckhwd         %xmm16, %xmm17, %xmm19
+vpunpckhwd         (%rax), %xmm17, %xmm19
+vpunpckhwd         %xmm16, %xmm17, %xmm19 {k1}
+vpunpckhwd         (%rax), %xmm17, %xmm19 {k1}
+vpunpckhwd         %xmm16, %xmm17, %xmm19 {z}{k1}
+vpunpckhwd         (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpunpckhwd         %ymm16, %ymm17, %ymm19
+vpunpckhwd         (%rax), %ymm17, %ymm19
+vpunpckhwd         %ymm16, %ymm17, %ymm19 {k1}
+vpunpckhwd         (%rax), %ymm17, %ymm19 {k1}
+vpunpckhwd         %ymm16, %ymm17, %ymm19 {z}{k1}
+vpunpckhwd         (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpunpcklbw         %xmm16, %xmm17, %xmm19
+vpunpcklbw         (%rax), %xmm17, %xmm19
+vpunpcklbw         %xmm16, %xmm17, %xmm19 {k1}
+vpunpcklbw         (%rax), %xmm17, %xmm19 {k1}
+vpunpcklbw         %xmm16, %xmm17, %xmm19 {z}{k1}
+vpunpcklbw         (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpunpcklbw         %ymm16, %ymm17, %ymm19
+vpunpcklbw         (%rax), %ymm17, %ymm19
+vpunpcklbw         %ymm16, %ymm17, %ymm19 {k1}
+vpunpcklbw         (%rax), %ymm17, %ymm19 {k1}
+vpunpcklbw         %ymm16, %ymm17, %ymm19 {z}{k1}
+vpunpcklbw         (%rax), %ymm17, %ymm19 {z}{k1}
+
+vpunpcklwd         %xmm16, %xmm17, %xmm19
+vpunpcklwd         (%rax), %xmm17, %xmm19
+vpunpcklwd         %xmm16, %xmm17, %xmm19 {k1}
+vpunpcklwd         (%rax), %xmm17, %xmm19 {k1}
+vpunpcklwd         %xmm16, %xmm17, %xmm19 {z}{k1}
+vpunpcklwd         (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpunpcklwd         %ymm16, %ymm17, %ymm19
+vpunpcklwd         (%rax), %ymm17, %ymm19
+vpunpcklwd         %ymm16, %ymm17, %ymm19 {k1}
+vpunpcklwd         (%rax), %ymm17, %ymm19 {k1}
+vpunpcklwd         %ymm16, %ymm17, %ymm19 {z}{k1}
+vpunpcklwd         (%rax), %ymm17, %ymm19 {z}{k1}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      3     0.50                        vdbpsadbw	$0, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vdbpsadbw	$0, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vdbpsadbw	$0, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vdbpsadbw	$0, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vdbpsadbw	$0, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vdbpsadbw	$0, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vdbpsadbw	$0, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vdbpsadbw	$0, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vdbpsadbw	$0, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vdbpsadbw	$0, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vdbpsadbw	$0, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vdbpsadbw	$0, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu8	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu8	(%rax), %xmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu8	%xmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovdqu8	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu8	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu8	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu8	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu8	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu8	%ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu8	(%rax), %ymm19
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu8	%ymm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovdqu8	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu8	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu8	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu8	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu8	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu16	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu16	(%rax), %xmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu16	%xmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovdqu16	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu16	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu16	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu16	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu16	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu16	%ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu16	(%rax), %ymm19
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu16	%ymm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovdqu16	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu16	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu16	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu16	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu16	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpabsb	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpabsb	(%rax), %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpabsb	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsb	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpabsb	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsb	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpabsb	%ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpabsb	(%rax), %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpabsb	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsb	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpabsb	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsb	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpabsw	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpabsw	(%rax), %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpabsw	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsw	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpabsw	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsw	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpabsw	%ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpabsw	(%rax), %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpabsw	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsw	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpabsw	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpabsw	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpackssdw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpackssdw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpackssdw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpackssdw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpackssdw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpackssdw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpackssdw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpackssdw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpackssdw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpackssdw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpackssdw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpackssdw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpacksswb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpacksswb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpacksswb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpacksswb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpacksswb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpacksswb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpacksswb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpacksswb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpacksswb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpacksswb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpacksswb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpacksswb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpackusdw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpackusdw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpackusdw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpackusdw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpackusdw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpackusdw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpackusdw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpackusdw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpackusdw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpackusdw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpackusdw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpackusdw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpackuswb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpackuswb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpackuswb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpackuswb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpackuswb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpackuswb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpackuswb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpackuswb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpackuswb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpackuswb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpackuswb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpackuswb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpaddb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpaddb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpaddb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpaddb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpaddb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpaddb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpaddsb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpaddsb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpaddsb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpaddsb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpaddsb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpaddsb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpaddsw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpaddsw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpaddsw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpaddsw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpaddsw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpaddsw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddsw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpaddusb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpaddusb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpaddusb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpaddusb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpaddusb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpaddusb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpaddusw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpaddusw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpaddusw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpaddusw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpaddusw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpaddusw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddusw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpaddw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpaddw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpaddw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpaddw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpaddw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpaddw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpalignr	$1, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpalignr	$1, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpalignr	$1, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpalignr	$1, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpalignr	$1, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpalignr	$1, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpalignr	$1, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpalignr	$1, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpalignr	$1, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpalignr	$1, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpalignr	$1, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpalignr	$1, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpavgb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpavgb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpavgb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpavgb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpavgb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpavgb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpavgb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpavgb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpavgb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpavgb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpavgb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpavgb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpavgw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpavgw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpavgw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpavgw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpavgw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpavgw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpavgw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpavgw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpavgw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpavgw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpavgw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpavgw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpblendmb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpblendmb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpblendmb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpblendmb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpblendmb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpblendmb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpblendmb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpblendmb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpblendmb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpblendmb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpblendmb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpblendmb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpblendmw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpblendmw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpblendmw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpblendmw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpblendmw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpblendmw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpblendmw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpblendmw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpblendmw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpblendmw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpblendmw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpblendmw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastb	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastb	(%rax), %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastb	%eax, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastb	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastb	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastb	%eax, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastb	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastb	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastb	%eax, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastb	%xmm16, %ymm19
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastb	(%rax), %ymm19
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastb	%eax, %ymm19
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastb	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastb	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastb	%eax, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastb	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastb	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastb	%eax, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastw	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastw	(%rax), %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastw	%eax, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastw	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastw	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastw	%eax, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastw	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastw	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastw	%eax, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastw	%xmm16, %ymm19
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastw	(%rax), %ymm19
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastw	%eax, %ymm19
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastw	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastw	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastw	%eax, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastw	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastw	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastw	%eax, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqb	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqb	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqb	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqb	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqb	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqb	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqb	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqb	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqb	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqb	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqb	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqb	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqb	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqb	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqb	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqb	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqw	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqw	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqw	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqw	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqw	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqw	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqw	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqw	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtb	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtb	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtb	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtb	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtb	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtb	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtb	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtb	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtw	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtw	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtw	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtw	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtw	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtw	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtw	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtw	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpequb	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequb	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpequb	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequb	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpequb	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequb	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpequb	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequb	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpequw	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequw	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpequw	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequw	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpequw	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequw	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpequw	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequw	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqw	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqw	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqw	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqw	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqw	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqw	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqw	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqw	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      2     1.00                        vpermw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpermw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpermw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     1.00                        vpermw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      4     1.00                        vpermw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     1.00                        vpermw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vpermi2w	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermi2w	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      3     1.00                        vpermi2w	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermi2w	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vpermi2w	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermi2w	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     1.00                        vpermi2w	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermi2w	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      4     1.00                        vpermi2w	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermi2w	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     1.00                        vpermi2w	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermi2w	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vpermt2w	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermt2w	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      3     1.00                        vpermt2w	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermt2w	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vpermt2w	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermt2w	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     1.00                        vpermt2w	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermt2w	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      4     1.00                        vpermt2w	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermt2w	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     1.00                        vpermt2w	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermt2w	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpmaddubsw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmaddubsw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vpmaddubsw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmaddubsw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vpmaddubsw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmaddubsw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpmaddubsw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmaddubsw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vpmaddubsw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmaddubsw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vpmaddubsw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmaddubsw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpmaddwd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmaddwd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vpmaddwd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmaddwd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vpmaddwd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmaddwd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpmaddwd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmaddwd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vpmaddwd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmaddwd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vpmaddwd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmaddwd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpmaxsb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpmaxsb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpmaxsb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpmaxsb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpmaxsb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpmaxsb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpmaxsw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpmaxsw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpmaxsw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpmaxsw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpmaxsw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpmaxsw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxsw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpmaxub	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxub	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpmaxub	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxub	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpmaxub	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxub	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpmaxub	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxub	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpmaxub	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxub	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpmaxub	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxub	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpmaxuw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxuw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpmaxuw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxuw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpmaxuw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxuw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpmaxuw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxuw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpmaxuw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxuw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpmaxuw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmaxuw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpminsb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpminsb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpminsb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpminsb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpminsb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpminsb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpminsb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpminsb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpminsb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpminsb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpminsb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpminsb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpminsw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpminsw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpminsw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpminsw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpminsw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpminsw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpminsw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpminsw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpminsw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpminsw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpminsw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpminsw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpminub	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpminub	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpminub	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpminub	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpminub	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpminub	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpminub	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpminub	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpminub	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpminub	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpminub	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpminub	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpminuw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpminuw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpminuw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpminuw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpminuw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpminuw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpminuw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpminuw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpminuw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpminuw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpminuw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpminuw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpmovb2m	%xmm0, %k0
+# CHECK-NEXT:  1      1     1.00                        vpmovw2m	%xmm0, %k0
+# CHECK-NEXT:  1      1     1.00                        vpmovb2m	%ymm0, %k0
+# CHECK-NEXT:  1      1     1.00                        vpmovw2m	%ymm0, %k0
+# CHECK-NEXT:  1      0     0.17                        vpmovm2b	%k0, %xmm0
+# CHECK-NEXT:  1      0     0.17                        vpmovm2w	%k0, %xmm0
+# CHECK-NEXT:  1      0     0.17                        vpmovm2b	%k0, %ymm0
+# CHECK-NEXT:  1      0     0.17                        vpmovm2w	%k0, %ymm0
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbw	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxbw	(%rax), %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbw	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxbw	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbw	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxbw	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbw	%xmm16, %ymm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbw	(%rax), %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbw	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbw	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbw	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbw	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovswb	%xmm16, %xmm19
+# CHECK-NEXT:  1      11    1.50           *            vpmovswb	%xmm16, (%rax)
+# CHECK-NEXT:  1      2     1.00                        vpmovswb	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      11    1.50           *            vpmovswb	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  2      4     1.50                        vpmovswb	%ymm16, %xmm19
+# CHECK-NEXT:  1      11    1.50           *            vpmovswb	%ymm16, (%rax)
+# CHECK-NEXT:  2      4     1.50                        vpmovswb	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      11    1.50           *            vpmovswb	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovuswb	%xmm16, %xmm19
+# CHECK-NEXT:  1      11    1.50           *            vpmovuswb	%xmm16, (%rax)
+# CHECK-NEXT:  1      2     1.00                        vpmovuswb	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      11    1.50           *            vpmovuswb	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  2      4     1.50                        vpmovuswb	%ymm16, %xmm19
+# CHECK-NEXT:  1      11    1.50           *            vpmovuswb	%ymm16, (%rax)
+# CHECK-NEXT:  2      4     1.50                        vpmovuswb	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      11    1.50           *            vpmovuswb	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovwb	%xmm16, %xmm19
+# CHECK-NEXT:  1      11    1.50           *            vpmovwb	%xmm16, (%rax)
+# CHECK-NEXT:  1      2     1.00                        vpmovwb	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      11    1.50           *            vpmovwb	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  2      4     1.50                        vpmovwb	%ymm16, %xmm19
+# CHECK-NEXT:  1      11    1.50           *            vpmovwb	%ymm16, (%rax)
+# CHECK-NEXT:  2      4     1.50                        vpmovwb	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      11    1.50           *            vpmovwb	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbw	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxbw	(%rax), %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbw	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxbw	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbw	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxbw	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbw	%xmm16, %ymm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbw	(%rax), %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbw	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbw	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbw	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbw	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpmulhrsw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhrsw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vpmulhrsw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhrsw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vpmulhrsw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhrsw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpmulhrsw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhrsw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vpmulhrsw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhrsw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vpmulhrsw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhrsw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpmulhuw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhuw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vpmulhuw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhuw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vpmulhuw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhuw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpmulhuw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhuw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vpmulhuw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhuw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vpmulhuw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhuw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpmulhw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vpmulhw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vpmulhw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpmulhw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vpmulhw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vpmulhw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulhw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpmullw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmullw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vpmullw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmullw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vpmullw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmullw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpmullw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmullw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vpmullw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmullw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vpmullw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmullw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpsadbw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpsadbw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vpsadbw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpsadbw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpshufb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpshufb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpshufb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpshufb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshufb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpshufb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshufb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpshufb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpshufb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpshufb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshufb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpshufb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpshufhw	$0, %xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpshufhw	$0, (%rax), %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpshufhw	$0, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpshufhw	$0, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpshufhw	$0, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpshufhw	$0, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpshufhw	$0, %ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpshufhw	$0, (%rax), %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpshufhw	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpshufhw	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpshufhw	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpshufhw	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpshuflw	$0, %xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpshuflw	$0, (%rax), %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpshuflw	$0, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpshuflw	$0, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpshuflw	$0, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpshuflw	$0, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpshuflw	$0, %ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpshuflw	$0, (%rax), %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpshuflw	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpshuflw	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpshuflw	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpshuflw	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpslldq	$1, %xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpslldq	$1, (%rax), %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpslldq	$1, %ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpslldq	$1, (%rax), %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpsllvw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsllvw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpsllvw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsllvw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsllvw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsllvw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsllvw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsllvw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpsllvw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsllvw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsllvw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsllvw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsllw	$0, %xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsllw	$0, (%rax), %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpsllw	$0, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsllw	$0, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsllw	$0, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsllw	$0, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsllw	$0, %ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsllw	$0, (%rax), %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpsllw	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsllw	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsllw	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsllw	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpsllw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      9     1.00    *                   vpsllw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpsllw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      9     1.00    *                   vpsllw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpsllw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      9     1.00    *                   vpsllw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpsllw	%xmm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsllw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpsllw	%xmm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsllw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpsllw	%xmm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsllw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsravw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsravw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpsravw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsravw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsravw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsravw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsravw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsravw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpsravw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsravw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsravw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsravw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsraw	$0, %xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsraw	$0, (%rax), %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpsraw	$0, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsraw	$0, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsraw	$0, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsraw	$0, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsraw	$0, %ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsraw	$0, (%rax), %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpsraw	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsraw	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsraw	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsraw	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpsraw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      9     1.00    *                   vpsraw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpsraw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      9     1.00    *                   vpsraw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpsraw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      9     1.00    *                   vpsraw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpsraw	%xmm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsraw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpsraw	%xmm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsraw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpsraw	%xmm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsraw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsrldq	$1, %xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsrldq	$1, (%rax), %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpsrldq	$1, %ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsrldq	$1, (%rax), %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpsrlvw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlvw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpsrlvw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlvw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsrlvw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlvw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsrlvw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlvw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpsrlvw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlvw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsrlvw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlvw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsrlw	$0, %xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlw	$0, (%rax), %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpsrlw	$0, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlw	$0, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsrlw	$0, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlw	$0, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpsrlw	$0, %ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlw	$0, (%rax), %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpsrlw	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlw	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpsrlw	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlw	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpsrlw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      9     1.00    *                   vpsrlw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpsrlw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      9     1.00    *                   vpsrlw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpsrlw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      9     1.00    *                   vpsrlw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpsrlw	%xmm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpsrlw	%xmm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpsrlw	%xmm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsrlw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpsubb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpsubb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpsubb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpsubb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpsubb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpsubb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpsubsb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpsubsb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpsubsb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpsubsb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpsubsb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpsubsb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpsubsw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpsubsw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpsubsw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpsubsw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpsubsw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpsubsw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubsw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpsubusb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpsubusb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpsubusb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpsubusb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpsubusb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpsubusb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpsubusw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpsubusw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpsubusw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpsubusw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpsubusw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpsubusw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubusw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpsubw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpsubw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpsubw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpsubw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpsubw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpsubw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vptestmb	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestmb	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      3     1.50                        vptestmb	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestmb	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      4     2.00                        vptestmb	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestmb	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      4     2.00                        vptestmb	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestmb	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vptestmw	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestmw	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      3     1.50                        vptestmw	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestmw	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      4     2.00                        vptestmw	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestmw	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      4     2.00                        vptestmw	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestmw	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vptestnmb	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmb	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      3     1.50                        vptestnmb	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmb	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      4     2.00                        vptestnmb	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmb	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      4     2.00                        vptestnmb	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmb	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vptestnmw	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmw	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      3     1.50                        vptestnmw	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmw	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      4     2.00                        vptestnmw	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmw	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      4     2.00                        vptestnmw	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmw	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vpunpckhbw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhbw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpunpckhbw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhbw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpunpckhbw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhbw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpunpckhbw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhbw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpunpckhbw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhbw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpunpckhbw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhbw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpunpckhwd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhwd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpunpckhwd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhwd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpunpckhwd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhwd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpunpckhwd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhwd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpunpckhwd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhwd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpunpckhwd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhwd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpunpcklbw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpunpcklbw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpunpcklbw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpcklbw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpunpcklbw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpcklbw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpunpcklbw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpunpcklbw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpunpcklbw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpcklbw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpunpcklbw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpcklbw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpunpcklwd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpunpcklwd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpunpcklwd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpcklwd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpunpcklwd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpcklwd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpunpcklwd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpunpcklwd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpunpcklwd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpcklwd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpunpcklwd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpcklwd	(%rax), %ymm17, %ymm19 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -     4.00   4.00   4.00   4.00    -     233.00 423.50 312.50 140.00 226.00 226.00 8.00   150.67 150.67 150.67 148.00 148.00 148.00 4.00   4.00
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vdbpsadbw	$0, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdbpsadbw	$0, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vdbpsadbw	$0, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdbpsadbw	$0, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vdbpsadbw	$0, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdbpsadbw	$0, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vdbpsadbw	$0, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdbpsadbw	$0, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vdbpsadbw	$0, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdbpsadbw	$0, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vdbpsadbw	$0, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdbpsadbw	$0, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu8	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu8	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu8	%xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu8	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu8	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu8	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu8	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu8	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu8	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu8	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu8	%ymm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu8	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu8	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu8	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu8	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu8	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu16	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu16	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu16	%xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu16	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu16	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu16	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu16	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu16	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu16	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu16	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu16	%ymm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu16	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu16	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu16	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu16	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu16	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpabsb	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsb	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpabsb	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsb	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpabsb	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsb	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpabsb	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsb	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpabsb	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsb	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpabsb	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsb	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpabsw	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsw	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpabsw	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsw	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpabsw	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsw	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpabsw	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsw	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpabsw	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsw	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpabsw	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpabsw	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackssdw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackssdw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackssdw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackssdw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackssdw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackssdw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackssdw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackssdw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackssdw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackssdw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackssdw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackssdw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpacksswb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpacksswb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpacksswb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpacksswb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpacksswb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpacksswb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpacksswb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpacksswb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpacksswb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpacksswb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpacksswb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpacksswb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackusdw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackusdw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackusdw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackusdw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackusdw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackusdw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackusdw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackusdw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackusdw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackusdw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackusdw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackusdw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackuswb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackuswb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackuswb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackuswb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackuswb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackuswb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackuswb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackuswb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackuswb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackuswb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpackuswb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpackuswb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddsb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddsb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddsb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddsb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddsb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddsb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddsw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddsw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddsw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddsw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddsw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddsw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddsw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddusb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddusb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddusb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddusb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddusb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddusb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddusw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddusw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpaddusw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddusw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddusw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddusw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddusw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpalignr	$1, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpalignr	$1, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpalignr	$1, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpalignr	$1, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpalignr	$1, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpalignr	$1, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpalignr	$1, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpalignr	$1, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpalignr	$1, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpalignr	$1, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpalignr	$1, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpalignr	$1, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpavgb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpavgb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpavgb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpavgb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpavgb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpavgb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpavgw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpavgw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpavgw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpavgw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpavgw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpavgw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpavgw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpblendmb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendmb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpblendmb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendmb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpblendmb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendmb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpblendmb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendmb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpblendmb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendmb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpblendmb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendmb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpblendmw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendmw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpblendmw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendmw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpblendmw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendmw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpblendmw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendmw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpblendmw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendmw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpblendmw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpblendmw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastb	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%eax, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastb	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%eax, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastb	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%eax, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastb	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%eax, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastb	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%eax, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastb	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastb	%eax, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastw	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%eax, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastw	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%eax, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastw	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%eax, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastw	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%eax, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastw	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%eax, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastw	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastw	%eax, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqb	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqb	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqb	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqb	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqb	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqb	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqb	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqb	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqb	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqb	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqb	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqb	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqb	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqb	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqb	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqb	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqw	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqw	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqw	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqw	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqw	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqw	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqw	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqw	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtb	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtb	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtb	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtb	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtb	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtb	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtb	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtb	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtw	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtw	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtw	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtw	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtw	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtw	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtw	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtw	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpequb	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequb	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpequb	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequb	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpequb	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequb	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpequb	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequb	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpequw	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequw	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpequw	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequw	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpequw	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequw	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpequw	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequw	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqw	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqw	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqw	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqw	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqw	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqw	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqw	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqw	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermi2w	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermi2w	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermi2w	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermi2w	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermi2w	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermi2w	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermi2w	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermi2w	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermi2w	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermi2w	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermi2w	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermi2w	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermt2w	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermt2w	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermt2w	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermt2w	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermt2w	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermt2w	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermt2w	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermt2w	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermt2w	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermt2w	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermt2w	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermt2w	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaddubsw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddubsw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaddubsw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddubsw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaddubsw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddubsw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaddubsw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddubsw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaddubsw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddubsw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaddubsw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddubsw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaddwd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddwd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaddwd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddwd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaddwd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddwd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaddwd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddwd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaddwd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddwd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmaddwd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaddwd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxsb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxsb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxsb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxsb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxsb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxsb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxsw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxsw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxsw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxsw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxsw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxsw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxsw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxub	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxub	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxub	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxub	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxub	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxub	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxub	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxub	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxub	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxub	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxub	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxub	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxuw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxuw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxuw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxuw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxuw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxuw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxuw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxuw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxuw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxuw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmaxuw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmaxuw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminsb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminsb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminsb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminsb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminsb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminsb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminsw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminsw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminsw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminsw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminsw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminsw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminsw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminub	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminub	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminub	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminub	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminub	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminub	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminub	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminub	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminub	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminub	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminub	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminub	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminuw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminuw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminuw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminuw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminuw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminuw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminuw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminuw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminuw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminuw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpminuw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpminuw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovb2m	%xmm0, %k0
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovw2m	%xmm0, %k0
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovb2m	%ymm0, %k0
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovw2m	%ymm0, %k0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovm2b	%k0, %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovm2w	%k0, %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovm2b	%k0, %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovm2w	%k0, %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbw	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbw	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbw	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbw	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbw	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbw	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbw	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbw	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbw	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbw	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbw	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbw	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovswb	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovswb	%xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovswb	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovswb	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -     vpmovswb	%ymm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovswb	%ymm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -     vpmovswb	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovswb	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovuswb	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovuswb	%xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovuswb	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovuswb	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -     vpmovuswb	%ymm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovuswb	%ymm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -     vpmovuswb	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovuswb	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovwb	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovwb	%xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovwb	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovwb	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -     vpmovwb	%ymm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovwb	%ymm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -     vpmovwb	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovwb	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbw	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbw	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbw	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbw	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbw	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbw	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbw	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbw	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbw	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbw	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbw	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbw	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhrsw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhrsw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhrsw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhrsw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhrsw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhrsw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhrsw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhrsw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhrsw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhrsw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhrsw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhrsw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhuw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhuw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhuw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhuw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhuw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhuw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhuw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhuw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhuw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhuw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhuw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhuw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulhw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulhw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmullw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmullw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmullw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmullw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmullw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmullw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsadbw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsadbw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpsadbw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsadbw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshufb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshufb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshufb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshufb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshufb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshufb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshufhw	$0, %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufhw	$0, (%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshufhw	$0, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufhw	$0, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshufhw	$0, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufhw	$0, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshufhw	$0, %ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufhw	$0, (%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshufhw	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufhw	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshufhw	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufhw	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshuflw	$0, %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshuflw	$0, (%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshuflw	$0, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshuflw	$0, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshuflw	$0, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshuflw	$0, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshuflw	$0, %ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshuflw	$0, (%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshuflw	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshuflw	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshuflw	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshuflw	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpslldq	$1, %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpslldq	$1, (%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpslldq	$1, %ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpslldq	$1, (%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsllvw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllvw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsllvw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllvw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsllvw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllvw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsllvw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllvw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsllvw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllvw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsllvw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllvw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	$0, %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	$0, (%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	$0, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	$0, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	$0, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	$0, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	$0, %ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	$0, (%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	%xmm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	%xmm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsllw	%xmm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsllw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsravw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsravw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsravw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsravw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsravw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsravw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsravw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsravw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsravw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsravw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsravw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsravw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	$0, %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	$0, (%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	$0, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	$0, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	$0, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	$0, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	$0, %ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	$0, (%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	%xmm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	%xmm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsraw	%xmm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsraw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrldq	$1, %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrldq	$1, (%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrldq	$1, %ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrldq	$1, (%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrlvw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlvw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrlvw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlvw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrlvw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlvw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrlvw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlvw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrlvw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlvw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpsrlvw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlvw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	$0, %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	$0, (%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	$0, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	$0, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	$0, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	$0, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	$0, %ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	$0, (%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	%xmm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	%xmm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsrlw	%xmm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsrlw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubsb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubsb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubsb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubsb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubsb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubsb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubsw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubsw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubsw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubsw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubsw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubsw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubsw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubusb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubusb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubusb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubusb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubusb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubusb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubusw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubusw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubusw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubusw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubusw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubusw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubusw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vptestmb	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmb	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestmb	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmb	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestmb	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmb	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestmb	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmb	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vptestmw	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmw	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestmw	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmw	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestmw	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmw	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestmw	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmw	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vptestnmb	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmb	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestnmb	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmb	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestnmb	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmb	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestnmb	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmb	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vptestnmw	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmw	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestnmw	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmw	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestnmw	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmw	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestnmw	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmw	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhbw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhbw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhbw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhbw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhbw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhbw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhbw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhbw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhbw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhbw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhbw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhbw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhwd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhwd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhwd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhwd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhwd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhwd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhwd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhwd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhwd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhwd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhwd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhwd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklbw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklbw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklbw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklbw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklbw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklbw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklbw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklbw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklbw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklbw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklbw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklbw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklwd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklwd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklwd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklwd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklwd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklwd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklwd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklwd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklwd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklwd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpcklwd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpcklwd	(%rax), %ymm17, %ymm19 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512cd.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512cd.s
new file mode 100644
index 0000000000000..96590638b5b37
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512cd.s
@@ -0,0 +1,164 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vpbroadcastmb2q   %k0, %zmm16
+
+vpbroadcastmw2d   %k0, %zmm16
+
+vpconflictd       %zmm16, %zmm19
+vpconflictd       (%rax), %zmm19
+vpconflictd       (%rax){1to16}, %zmm19
+vpconflictd       %zmm16, %zmm19 {k1}
+vpconflictd       (%rax), %zmm19 {k1}
+vpconflictd       (%rax){1to16}, %zmm19 {k1}
+vpconflictd       %zmm16, %zmm19 {z}{k1}
+vpconflictd       (%rax), %zmm19 {z}{k1}
+vpconflictd       (%rax){1to16}, %zmm19 {z}{k1}
+
+vpconflictq       %zmm16, %zmm19
+vpconflictq       (%rax), %zmm19
+vpconflictq       (%rax){1to8}, %zmm19
+vpconflictq       %zmm16, %zmm19 {k1}
+vpconflictq       (%rax), %zmm19 {k1}
+vpconflictq       (%rax){1to8}, %zmm19 {k1}
+vpconflictq       %zmm16, %zmm19 {z}{k1}
+vpconflictq       (%rax), %zmm19 {z}{k1}
+vpconflictq       (%rax){1to8}, %zmm19 {z}{k1}
+
+vplzcntd          %zmm16, %zmm19
+vplzcntd          (%rax), %zmm19
+vplzcntd          (%rax){1to16}, %zmm19
+vplzcntd          %zmm16, %zmm19 {k1}
+vplzcntd          (%rax), %zmm19 {k1}
+vplzcntd          (%rax){1to16}, %zmm19 {k1}
+vplzcntd          %zmm16, %zmm19 {z}{k1}
+vplzcntd          (%rax), %zmm19 {z}{k1}
+vplzcntd          (%rax){1to16}, %zmm19 {z}{k1}
+
+vplzcntq          %zmm16, %zmm19
+vplzcntq          (%rax), %zmm19
+vplzcntq          (%rax){1to8}, %zmm19
+vplzcntq          %zmm16, %zmm19 {k1}
+vplzcntq          (%rax), %zmm19 {k1}
+vplzcntq          (%rax){1to8}, %zmm19 {k1}
+vplzcntq          %zmm16, %zmm19 {z}{k1}
+vplzcntq          (%rax), %zmm19 {z}{k1}
+vplzcntq          (%rax){1to8}, %zmm19 {z}{k1}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastmb2q	%k0, %zmm16
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastmw2d	%k0, %zmm16
+# CHECK-NEXT:  4      6     1.50                        vpconflictd	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictd	(%rax), %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictd	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  4      6     1.50                        vpconflictd	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictd	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  4      6     1.50                        vpconflictd	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictd	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  4      6     1.50                        vpconflictq	%zmm16, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictq	(%rax), %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  4      6     1.50                        vpconflictq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  4      6     1.50                        vpconflictq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vplzcntd	%zmm16, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vplzcntd	(%rax), %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vplzcntd	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vplzcntd	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vplzcntd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vplzcntd	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vplzcntd	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vplzcntd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vplzcntd	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vplzcntq	%zmm16, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vplzcntq	(%rax), %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vplzcntq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vplzcntq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vplzcntq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vplzcntq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vplzcntq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vplzcntq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vplzcntq	(%rax){1to8}, %zmm19 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     30.00  25.00  19.00  24.00  12.00  12.00   -     8.00   8.00   8.00   8.00   8.00   8.00    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastmb2q	%k0, %zmm16
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastmw2d	%k0, %zmm16
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   2.00   2.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vpconflictd	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictd	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictd	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   2.00   2.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vpconflictd	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictd	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   2.00   2.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vpconflictd	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictd	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   2.00   2.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vpconflictq	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   2.00   2.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vpconflictq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   2.00   2.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vpconflictq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vplzcntd	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntd	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntd	(%rax){1to16}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vplzcntd	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntd	(%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vplzcntd	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntd	(%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vplzcntq	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vplzcntq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vplzcntq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntq	(%rax){1to8}, %zmm19 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512cdvl.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512cdvl.s
new file mode 100644
index 0000000000000..800716d23492a
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512cdvl.s
@@ -0,0 +1,282 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vpbroadcastmb2q   %k0, %xmm16
+vpbroadcastmb2q   %k0, %ymm16
+
+vpbroadcastmw2d   %k0, %xmm16
+vpbroadcastmw2d   %k0, %ymm16
+
+vpconflictd       %xmm16, %xmm19
+vpconflictd       (%rax), %xmm19
+vpconflictd       (%rax){1to4}, %xmm19
+vpconflictd       %xmm16, %xmm19 {k1}
+vpconflictd       (%rax), %xmm19 {k1}
+vpconflictd       (%rax){1to4}, %xmm19 {k1}
+vpconflictd       %xmm16, %xmm19 {z}{k1}
+vpconflictd       (%rax), %xmm19 {z}{k1}
+vpconflictd       (%rax){1to4}, %xmm19 {z}{k1}
+
+vpconflictd       %ymm16, %ymm19
+vpconflictd       (%rax), %ymm19
+vpconflictd       (%rax){1to8}, %ymm19
+vpconflictd       %ymm16, %ymm19 {k1}
+vpconflictd       (%rax), %ymm19 {k1}
+vpconflictd       (%rax){1to8}, %ymm19 {k1}
+vpconflictd       %ymm16, %ymm19 {z}{k1}
+vpconflictd       (%rax), %ymm19 {z}{k1}
+vpconflictd       (%rax){1to8}, %ymm19 {z}{k1}
+
+vpconflictq       %xmm16, %xmm19
+vpconflictq       (%rax), %xmm19
+vpconflictq       (%rax){1to2}, %xmm19
+vpconflictq       %xmm16, %xmm19 {k1}
+vpconflictq       (%rax), %xmm19 {k1}
+vpconflictq       (%rax){1to2}, %xmm19 {k1}
+vpconflictq       %xmm16, %xmm19 {z}{k1}
+vpconflictq       (%rax), %xmm19 {z}{k1}
+vpconflictq       (%rax){1to2}, %xmm19 {z}{k1}
+
+vpconflictq       %ymm16, %ymm19
+vpconflictq       (%rax), %ymm19
+vpconflictq       (%rax){1to4}, %ymm19
+vpconflictq       %ymm16, %ymm19 {k1}
+vpconflictq       (%rax), %ymm19 {k1}
+vpconflictq       (%rax){1to4}, %ymm19 {k1}
+vpconflictq       %ymm16, %ymm19 {z}{k1}
+vpconflictq       (%rax), %ymm19 {z}{k1}
+vpconflictq       (%rax){1to4}, %ymm19 {z}{k1}
+
+vplzcntd          %xmm16, %xmm19
+vplzcntd          (%rax), %xmm19
+vplzcntd          (%rax){1to4}, %xmm19
+vplzcntd          %xmm16, %xmm19 {k1}
+vplzcntd          (%rax), %xmm19 {k1}
+vplzcntd          (%rax){1to4}, %xmm19 {k1}
+vplzcntd          %xmm16, %xmm19 {z}{k1}
+vplzcntd          (%rax), %xmm19 {z}{k1}
+vplzcntd          (%rax){1to4}, %xmm19 {z}{k1}
+
+vplzcntd          %ymm16, %ymm19
+vplzcntd          (%rax), %ymm19
+vplzcntd          (%rax){1to8}, %ymm19
+vplzcntd          %ymm16, %ymm19 {k1}
+vplzcntd          (%rax), %ymm19 {k1}
+vplzcntd          (%rax){1to8}, %ymm19 {k1}
+vplzcntd          %ymm16, %ymm19 {z}{k1}
+vplzcntd          (%rax), %ymm19 {z}{k1}
+vplzcntd          (%rax){1to8}, %ymm19 {z}{k1}
+
+vplzcntq          %xmm16, %xmm19
+vplzcntq          (%rax), %xmm19
+vplzcntq          (%rax){1to2}, %xmm19
+vplzcntq          %xmm16, %xmm19 {k1}
+vplzcntq          (%rax), %xmm19 {k1}
+vplzcntq          (%rax){1to2}, %xmm19 {k1}
+vplzcntq          %xmm16, %xmm19 {z}{k1}
+vplzcntq          (%rax), %xmm19 {z}{k1}
+vplzcntq          (%rax){1to2}, %xmm19 {z}{k1}
+
+vplzcntq          %ymm16, %ymm19
+vplzcntq          (%rax), %ymm19
+vplzcntq          (%rax){1to4}, %ymm19
+vplzcntq          %ymm16, %ymm19 {k1}
+vplzcntq          (%rax), %ymm19 {k1}
+vplzcntq          (%rax){1to4}, %ymm19 {k1}
+vplzcntq          %ymm16, %ymm19 {z}{k1}
+vplzcntq          (%rax), %ymm19 {z}{k1}
+vplzcntq          (%rax){1to4}, %ymm19 {z}{k1}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastmb2q	%k0, %xmm16
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastmb2q	%k0, %ymm16
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastmw2d	%k0, %xmm16
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastmw2d	%k0, %ymm16
+# CHECK-NEXT:  1      2     1.00                        vpconflictd	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictd	(%rax), %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictd	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpconflictd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictd	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpconflictd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictd	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  4      6     1.50                        vpconflictd	%ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictd	(%rax), %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictd	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  4      6     1.50                        vpconflictd	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictd	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  4      6     1.50                        vpconflictd	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictd	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpconflictq	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictq	(%rax), %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpconflictq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpconflictq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  4      6     1.50                        vpconflictq	%ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictq	(%rax), %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  4      6     1.50                        vpconflictq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  4      6     1.50                        vpconflictq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpconflictq	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vplzcntd	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntd	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntd	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vplzcntd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntd	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vplzcntd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntd	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vplzcntd	%ymm16, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntd	(%rax), %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntd	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vplzcntd	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntd	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vplzcntd	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntd	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vplzcntq	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntq	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vplzcntq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vplzcntq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vplzcntq	%ymm16, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntq	(%rax), %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vplzcntq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vplzcntq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vplzcntq	(%rax){1to4}, %ymm19 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     42.00  38.00  20.00  24.00  24.00  24.00   -     16.00  16.00  16.00  16.00  16.00  16.00   -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastmb2q	%k0, %xmm16
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastmb2q	%k0, %ymm16
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastmw2d	%k0, %xmm16
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastmw2d	%k0, %ymm16
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpconflictd	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictd	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictd	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpconflictd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictd	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpconflictd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictd	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   2.00   2.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vpconflictd	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictd	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictd	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   2.00   2.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vpconflictd	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictd	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   2.00   2.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vpconflictd	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictd	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpconflictq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpconflictq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpconflictq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   2.00   2.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vpconflictq	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   2.00   2.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vpconflictq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   2.00   2.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vpconflictq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpconflictq	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vplzcntd	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntd	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntd	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vplzcntd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntd	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vplzcntd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntd	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vplzcntd	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntd	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntd	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vplzcntd	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntd	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vplzcntd	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntd	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vplzcntq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vplzcntq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vplzcntq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vplzcntq	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vplzcntq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vplzcntq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vplzcntq	(%rax){1to4}, %ymm19 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512dq.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512dq.s
new file mode 100644
index 0000000000000..6eee1e4c2e3ff
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512dq.s
@@ -0,0 +1,1277 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+kaddb             %k0, %k1, %k2
+kaddw             %k0, %k1, %k2
+kandb             %k0, %k1, %k2
+kandnb            %k0, %k1, %k2
+
+kmovb             %k0, %k2
+kmovb             (%rax), %k2
+kmovb             %k0, (%rax)
+kmovb             %eax, %k2
+kmovb             %k0, %eax
+
+knotb             %k0, %k2
+
+korb              %k0, %k1, %k2
+
+kortestb          %k0, %k2
+
+kshiftlb          $2, %k1, %k2
+kshiftrb          $2, %k1, %k2
+
+ktestb            %k0, %k2
+ktestw           %k0, %k2
+
+kxnorb            %k0, %k1, %k2
+kxorb             %k0, %k1, %k2
+
+vandnpd           %zmm16, %zmm17, %zmm19
+vandnpd           (%rax), %zmm17, %zmm19
+vandnpd           (%rax){1to8}, %zmm17, %zmm19
+vandnpd           %zmm16, %zmm17, %zmm19 {k1}
+vandnpd           (%rax), %zmm17, %zmm19 {k1}
+vandnpd           (%rax){1to8}, %zmm17, %zmm19 {k1}
+vandnpd           %zmm16, %zmm17, %zmm19 {z}{k1}
+vandnpd           (%rax), %zmm17, %zmm19 {z}{k1}
+vandnpd           (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vandnps           %zmm16, %zmm17, %zmm19
+vandnps           (%rax), %zmm17, %zmm19
+vandnps           (%rax){1to16}, %zmm17, %zmm19
+vandnps           %zmm16, %zmm17, %zmm19 {k1}
+vandnps           (%rax), %zmm17, %zmm19 {k1}
+vandnps           (%rax){1to16}, %zmm17, %zmm19 {k1}
+vandnps           %zmm16, %zmm17, %zmm19 {z}{k1}
+vandnps           (%rax), %zmm17, %zmm19 {z}{k1}
+vandnps           (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vandpd            %zmm16, %zmm17, %zmm19
+vandpd            (%rax), %zmm17, %zmm19
+vandpd            (%rax){1to8}, %zmm17, %zmm19
+vandpd            %zmm16, %zmm17, %zmm19 {k1}
+vandpd            (%rax), %zmm17, %zmm19 {k1}
+vandpd            (%rax){1to8}, %zmm17, %zmm19 {k1}
+vandpd            %zmm16, %zmm17, %zmm19 {z}{k1}
+vandpd            (%rax), %zmm17, %zmm19 {z}{k1}
+vandpd            (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vandps            %zmm16, %zmm17, %zmm19
+vandps            (%rax), %zmm17, %zmm19
+vandps            (%rax){1to16}, %zmm17, %zmm19
+vandps            %zmm16, %zmm17, %zmm19 {k1}
+vandps            (%rax), %zmm17, %zmm19 {k1}
+vandps            (%rax){1to16}, %zmm17, %zmm19 {k1}
+vandps            %zmm16, %zmm17, %zmm19 {z}{k1}
+vandps            (%rax), %zmm17, %zmm19 {z}{k1}
+vandps            (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vbroadcastf32x2   %xmm16, %zmm19
+vbroadcastf32x2   (%rax), %zmm19
+vbroadcastf32x2   %xmm16, %zmm19 {k1}
+vbroadcastf32x2   (%rax), %zmm19 {k1}
+vbroadcastf32x2   %xmm16, %zmm19 {z}{k1}
+vbroadcastf32x2   (%rax), %zmm19 {z}{k1}
+
+vbroadcastf32x8   (%rax), %zmm19
+vbroadcastf32x8   (%rax), %zmm19 {k1}
+vbroadcastf32x8   (%rax), %zmm19 {z}{k1}
+
+vbroadcastf64x2   (%rax), %zmm19
+vbroadcastf64x2   (%rax), %zmm19 {k1}
+vbroadcastf64x2   (%rax), %zmm19 {z}{k1}
+
+vbroadcasti32x2   %xmm16, %zmm19
+vbroadcasti32x2   (%rax), %zmm19
+vbroadcasti32x2   %xmm16, %zmm19 {k1}
+vbroadcasti32x2   (%rax), %zmm19 {k1}
+vbroadcasti32x2   %xmm16, %zmm19 {z}{k1}
+vbroadcasti32x2   (%rax), %zmm19 {z}{k1}
+
+vbroadcasti32x8   (%rax), %zmm19
+vbroadcasti32x8   (%rax), %zmm19 {k1}
+vbroadcasti32x8   (%rax), %zmm19 {z}{k1}
+
+vbroadcasti64x2   (%rax), %zmm19
+vbroadcasti64x2   (%rax), %zmm19 {k1}
+vbroadcasti64x2   (%rax), %zmm19 {z}{k1}
+
+vcvtpd2qq         %zmm16, %zmm19
+vcvtpd2qq         (%rax), %zmm19
+vcvtpd2qq         (%rax){1to8}, %zmm19
+vcvtpd2qq         %zmm16, %zmm19 {k1}
+vcvtpd2qq         (%rax), %zmm19 {k1}
+vcvtpd2qq         (%rax){1to8}, %zmm19 {k1}
+vcvtpd2qq         %zmm16, %zmm19 {z}{k1}
+vcvtpd2qq         (%rax), %zmm19 {z}{k1}
+vcvtpd2qq         (%rax){1to8}, %zmm19 {z}{k1}
+
+vcvtpd2uqq        %zmm16, %zmm19
+vcvtpd2uqq        (%rax), %zmm19
+vcvtpd2uqq        (%rax){1to8}, %zmm19
+vcvtpd2uqq        %zmm16, %zmm19 {k1}
+vcvtpd2uqq        (%rax), %zmm19 {k1}
+vcvtpd2uqq        (%rax){1to8}, %zmm19 {k1}
+vcvtpd2uqq        %zmm16, %zmm19 {z}{k1}
+vcvtpd2uqq        (%rax), %zmm19 {z}{k1}
+vcvtpd2uqq        (%rax){1to8}, %zmm19 {z}{k1}
+
+vcvtps2qq         %ymm16, %zmm19
+vcvtps2qq         (%rax), %zmm19
+vcvtps2qq         (%rax){1to8}, %zmm19
+vcvtps2qq         %ymm16, %zmm19 {k1}
+vcvtps2qq         (%rax), %zmm19 {k1}
+vcvtps2qq         (%rax){1to8}, %zmm19 {k1}
+vcvtps2qq         %ymm16, %zmm19 {z}{k1}
+vcvtps2qq         (%rax), %zmm19 {z}{k1}
+vcvtps2qq         (%rax){1to8}, %zmm19 {z}{k1}
+
+vcvtps2uqq        %ymm16, %zmm19
+vcvtps2uqq        (%rax), %zmm19
+vcvtps2uqq        (%rax){1to8}, %zmm19
+vcvtps2uqq        %ymm16, %zmm19 {k1}
+vcvtps2uqq        (%rax), %zmm19 {k1}
+vcvtps2uqq        (%rax){1to8}, %zmm19 {k1}
+vcvtps2uqq        %ymm16, %zmm19 {z}{k1}
+vcvtps2uqq        (%rax), %zmm19 {z}{k1}
+vcvtps2uqq        (%rax){1to8}, %zmm19 {z}{k1}
+
+vcvtqq2pd         %zmm16, %zmm19
+vcvtqq2pd         (%rax), %zmm19
+vcvtqq2pd         (%rax){1to8}, %zmm19
+vcvtqq2pd         %zmm16, %zmm19 {k1}
+vcvtqq2pd         (%rax), %zmm19 {k1}
+vcvtqq2pd         (%rax){1to8}, %zmm19 {k1}
+vcvtqq2pd         %zmm16, %zmm19 {z}{k1}
+vcvtqq2pd         (%rax), %zmm19 {z}{k1}
+vcvtqq2pd         (%rax){1to8}, %zmm19 {z}{k1}
+
+vcvtqq2ps         %zmm16, %ymm19
+vcvtqq2ps         (%rax), %ymm19
+vcvtqq2ps         (%rax){1to8}, %ymm19
+vcvtqq2ps         %zmm16, %ymm19 {k1}
+vcvtqq2ps         (%rax), %ymm19 {k1}
+vcvtqq2ps         (%rax){1to8}, %ymm19 {k1}
+vcvtqq2ps         %zmm16, %ymm19 {z}{k1}
+vcvtqq2ps         (%rax), %ymm19 {z}{k1}
+vcvtqq2ps         (%rax){1to8}, %ymm19 {z}{k1}
+
+vcvttpd2qq        %zmm16, %zmm19
+vcvttpd2qq        (%rax), %zmm19
+vcvttpd2qq        (%rax){1to8}, %zmm19
+vcvttpd2qq        %zmm16, %zmm19 {k1}
+vcvttpd2qq        (%rax), %zmm19 {k1}
+vcvttpd2qq        (%rax){1to8}, %zmm19 {k1}
+vcvttpd2qq        %zmm16, %zmm19 {z}{k1}
+vcvttpd2qq        (%rax), %zmm19 {z}{k1}
+vcvttpd2qq        (%rax){1to8}, %zmm19 {z}{k1}
+
+vcvttpd2uqq       %zmm16, %zmm19
+vcvttpd2uqq       (%rax), %zmm19
+vcvttpd2uqq       (%rax){1to8}, %zmm19
+vcvttpd2uqq       %zmm16, %zmm19 {k1}
+vcvttpd2uqq       (%rax), %zmm19 {k1}
+vcvttpd2uqq       (%rax){1to8}, %zmm19 {k1}
+vcvttpd2uqq       %zmm16, %zmm19 {z}{k1}
+vcvttpd2uqq       (%rax), %zmm19 {z}{k1}
+vcvttpd2uqq       (%rax){1to8}, %zmm19 {z}{k1}
+
+vcvttps2qq        %ymm16, %zmm19
+vcvttps2qq        (%rax), %zmm19
+vcvttps2qq        (%rax){1to8}, %zmm19
+vcvttps2qq        %ymm16, %zmm19 {k1}
+vcvttps2qq        (%rax), %zmm19 {k1}
+vcvttps2qq        (%rax){1to8}, %zmm19 {k1}
+vcvttps2qq        %ymm16, %zmm19 {z}{k1}
+vcvttps2qq        (%rax), %zmm19 {z}{k1}
+vcvttps2qq        (%rax){1to8}, %zmm19 {z}{k1}
+
+vcvttps2uqq       %ymm16, %zmm19
+vcvttps2uqq       (%rax), %zmm19
+vcvttps2uqq       (%rax){1to8}, %zmm19
+vcvttps2uqq       %ymm16, %zmm19 {k1}
+vcvttps2uqq       (%rax), %zmm19 {k1}
+vcvttps2uqq       (%rax){1to8}, %zmm19 {k1}
+vcvttps2uqq       %ymm16, %zmm19 {z}{k1}
+vcvttps2uqq       (%rax), %zmm19 {z}{k1}
+vcvttps2uqq       (%rax){1to8}, %zmm19 {z}{k1}
+
+vcvtuqq2pd        %zmm16, %zmm19
+vcvtuqq2pd        (%rax), %zmm19
+vcvtuqq2pd        (%rax){1to8}, %zmm19
+vcvtuqq2pd        %zmm16, %zmm19 {k1}
+vcvtuqq2pd        (%rax), %zmm19 {k1}
+vcvtuqq2pd        (%rax){1to8}, %zmm19 {k1}
+vcvtuqq2pd        %zmm16, %zmm19 {z}{k1}
+vcvtuqq2pd        (%rax), %zmm19 {z}{k1}
+vcvtuqq2pd        (%rax){1to8}, %zmm19 {z}{k1}
+
+vcvtuqq2ps        %zmm16, %ymm19
+vcvtuqq2ps        (%rax), %ymm19
+vcvtuqq2ps        (%rax){1to8}, %ymm19
+vcvtuqq2ps        %zmm16, %ymm19 {k1}
+vcvtuqq2ps        (%rax), %ymm19 {k1}
+vcvtuqq2ps        (%rax){1to8}, %ymm19 {k1}
+vcvtuqq2ps        %zmm16, %ymm19 {z}{k1}
+vcvtuqq2ps        (%rax), %ymm19 {z}{k1}
+vcvtuqq2ps        (%rax){1to8}, %ymm19 {z}{k1}
+
+vextractf32x8     $1, %zmm16, %ymm19
+vextractf32x8     $1, %zmm16, (%rax)
+vextractf32x8     $1, %zmm16, %ymm19 {k1}
+vextractf32x8     $1, %zmm16, (%rax) {k1}
+vextractf32x8     $1, %zmm16, %ymm19 {z}{k1}
+
+vextractf64x2     $1, %zmm16, %xmm19
+vextractf64x2     $1, %zmm16, (%rax)
+vextractf64x2     $1, %zmm16, %xmm19 {k1}
+vextractf64x2     $1, %zmm16, (%rax) {k1}
+vextractf64x2     $1, %zmm16, %xmm19 {z}{k1}
+
+vextracti32x8     $1, %zmm16, %ymm19
+vextracti32x8     $1, %zmm16, (%rax)
+vextracti32x8     $1, %zmm16, %ymm19 {k1}
+vextracti32x8     $1, %zmm16, (%rax) {k1}
+vextracti32x8     $1, %zmm16, %ymm19 {z}{k1}
+
+vextracti64x2     $1, %zmm16, %xmm19
+vextracti64x2     $1, %zmm16, (%rax)
+vextracti64x2     $1, %zmm16, %xmm19 {k1}
+vextracti64x2     $1, %zmm16, (%rax) {k1}
+vextracti64x2     $1, %zmm16, %xmm19 {z}{k1}
+
+vfpclasspd        $0xab, %zmm16, %k1
+vfpclasspdz       $0xab, (%rax), %k1
+vfpclasspdz       $0xab, (%rax){1to8}, %k1
+vfpclasspd        $0xab, %zmm16, %k1 {k2}
+vfpclasspdz       $0xab, (%rax), %k1 {k2}
+vfpclasspdz       $0xab, (%rax){1to8}, %k1 {k2}
+
+vfpclassps        $0xab, %zmm16, %k1
+vfpclasspsz       $0xab, (%rax), %k1
+vfpclasspsz       $0xab, (%rax){1to16}, %k1
+vfpclassps        $0xab, %zmm16, %k1 {k2}
+vfpclasspsz       $0xab, (%rax), %k1 {k2}
+vfpclasspsz       $0xab, (%rax){1to16}, %k1 {k2}
+
+vfpclasssd        $0xab, %xmm16, %k1
+vfpclasssd        $0xab, (%rax), %k1
+vfpclasssd        $0xab, %xmm16, %k1 {k2}
+vfpclasssd        $0xab, (%rax), %k1 {k2}
+
+vfpclassss        $0xab, %xmm16, %k1
+vfpclassss        $0xab, (%rax), %k1
+vfpclassss        $0xab, %xmm16, %k1 {k2}
+vfpclassss        $0xab, (%rax), %k1 {k2}
+
+vinsertf32x8      $1, %ymm16, %zmm16, %zmm19
+vinsertf32x8      $1, (%rax), %zmm16, %zmm19
+vinsertf32x8      $1, %ymm16, %zmm16, %zmm19 {k1}
+vinsertf32x8      $1, (%rax), %zmm16, %zmm19 {k1}
+vinsertf32x8      $1, %ymm16, %zmm16, %zmm19 {z}{k1}
+vinsertf32x8      $1, (%rax), %zmm16, %zmm19 {z}{k1}
+
+vinsertf64x2      $1, %xmm16, %zmm16, %zmm19
+vinsertf64x2      $1, (%rax), %zmm16, %zmm19
+vinsertf64x2      $1, %xmm16, %zmm16, %zmm19 {k1}
+vinsertf64x2      $1, (%rax), %zmm16, %zmm19 {k1}
+vinsertf64x2      $1, %xmm16, %zmm16, %zmm19 {z}{k1}
+vinsertf64x2      $1, (%rax), %zmm16, %zmm19 {z}{k1}
+
+vinserti32x8      $1, %ymm16, %zmm16, %zmm19
+vinserti32x8      $1, (%rax), %zmm16, %zmm19
+vinserti32x8      $1, %ymm16, %zmm16, %zmm19 {k1}
+vinserti32x8      $1, (%rax), %zmm16, %zmm19 {k1}
+vinserti32x8      $1, %ymm16, %zmm16, %zmm19 {z}{k1}
+vinserti32x8      $1, (%rax), %zmm16, %zmm19 {z}{k1}
+
+vinserti64x2      $1, %xmm16, %zmm16, %zmm19
+vinserti64x2      $1, (%rax), %zmm16, %zmm19
+vinserti64x2      $1, %xmm16, %zmm16, %zmm19 {k1}
+vinserti64x2      $1, (%rax), %zmm16, %zmm19 {k1}
+vinserti64x2      $1, %xmm16, %zmm16, %zmm19 {z}{k1}
+vinserti64x2      $1, (%rax), %zmm16, %zmm19 {z}{k1}
+
+vorpd             %zmm16, %zmm17, %zmm19
+vorpd             (%rax), %zmm17, %zmm19
+vorpd             (%rax){1to8}, %zmm17, %zmm19
+vorpd             %zmm16, %zmm17, %zmm19 {k1}
+vorpd             (%rax), %zmm17, %zmm19 {k1}
+vorpd             (%rax){1to8}, %zmm17, %zmm19 {k1}
+vorpd             %zmm16, %zmm17, %zmm19 {z}{k1}
+vorpd             (%rax), %zmm17, %zmm19 {z}{k1}
+vorpd             (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vorps             %zmm16, %zmm17, %zmm19
+vorps             (%rax), %zmm17, %zmm19
+vorps             (%rax){1to16}, %zmm17, %zmm19
+vorps             %zmm16, %zmm17, %zmm19 {k1}
+vorps             (%rax), %zmm17, %zmm19 {k1}
+vorps             (%rax){1to16}, %zmm17, %zmm19 {k1}
+vorps             %zmm16, %zmm17, %zmm19 {z}{k1}
+vorps             (%rax), %zmm17, %zmm19 {z}{k1}
+vorps             (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vpextrd           $1, %xmm16, %ecx
+vpextrd           $1, %xmm16, (%rax)
+
+vpextrq           $1, %xmm16, %rcx
+vpextrq           $1, %xmm16, (%rax)
+
+vpinsrd           $1, %ecx, %xmm16, %xmm19
+vpinsrd           $1, (%rax), %xmm16, %xmm19
+
+vpinsrq           $1, %rcx, %xmm16, %xmm19
+vpinsrq           $1, (%rax), %xmm16, %xmm19
+
+vpmovm2d          %k0, %zmm0
+vpmovm2q          %k0, %zmm0
+
+vpmovd2m          %zmm0, %k0
+vpmovq2m          %zmm0, %k0
+
+vpmullq           %zmm16, %zmm17, %zmm19
+vpmullq           (%rax), %zmm17, %zmm19
+vpmullq           %zmm16, %zmm17, %zmm19 {k1}
+vpmullq           (%rax), %zmm17, %zmm19 {k1}
+vpmullq           %zmm16, %zmm17, %zmm19 {z}{k1}
+vpmullq           (%rax), %zmm17, %zmm19 {z}{k1}
+
+vrangepd          $ab, %zmm16, %zmm17, %zmm19
+vrangepd          $ab, (%rax), %zmm17, %zmm19
+vrangepd          $ab, (%rax){1to8}, %zmm17, %zmm19
+vrangepd          $ab, %zmm16, %zmm17, %zmm19 {k1}
+vrangepd          $ab, (%rax), %zmm17, %zmm19 {k1}
+vrangepd          $ab, (%rax){1to8}, %zmm17, %zmm19 {k1}
+vrangepd          $ab, %zmm16, %zmm17, %zmm19 {z}{k1}
+vrangepd          $ab, (%rax), %zmm17, %zmm19 {z}{k1}
+vrangepd          $ab, (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vrangepd          $ab, {sae}, %zmm16, %zmm17, %zmm19
+vrangepd          $ab, {sae}, %zmm16, %zmm17, %zmm19 {k1}
+vrangepd          $ab, {sae}, %zmm16, %zmm17, %zmm19 {z}{k1}
+
+vrangeps          $ab, %zmm16, %zmm17, %zmm19
+vrangeps          $ab, (%rax), %zmm17, %zmm19
+vrangeps          $ab, (%rax){1to16}, %zmm17, %zmm19
+vrangeps          $ab, %zmm16, %zmm17, %zmm19 {k1}
+vrangeps          $ab, (%rax), %zmm17, %zmm19 {k1}
+vrangeps          $ab, (%rax){1to16}, %zmm17, %zmm19 {k1}
+vrangeps          $ab, %zmm16, %zmm17, %zmm19 {z}{k1}
+vrangeps          $ab, (%rax), %zmm17, %zmm19 {z}{k1}
+vrangeps          $ab, (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vrangeps          $ab, {sae}, %zmm16, %zmm17, %zmm19
+vrangeps          $ab, {sae}, %zmm16, %zmm17, %zmm19 {k1}
+vrangeps          $ab, {sae}, %zmm16, %zmm17, %zmm19 {z}{k1}
+
+vrangesd          $ab, %xmm16, %xmm17, %xmm19
+vrangesd          $ab, (%rax), %xmm17, %xmm19
+vrangesd          $ab, %xmm16, %xmm17, %xmm19 {k1}
+vrangesd          $ab, (%rax), %xmm17, %xmm19 {k1}
+vrangesd          $ab, %xmm16, %xmm17, %xmm19 {z}{k1}
+vrangesd          $ab, (%rax), %xmm17, %xmm19 {z}{k1}
+
+vrangesd          $ab, {sae}, %xmm16, %xmm17, %xmm19
+vrangesd          $ab, {sae}, %xmm16, %xmm17, %xmm19 {k1}
+vrangesd          $ab, {sae}, %xmm16, %xmm17, %xmm19 {z}{k1}
+
+vrangess          $ab, %xmm16, %xmm17, %xmm19
+vrangess          $ab, (%rax), %xmm17, %xmm19
+vrangess          $ab, %xmm16, %xmm17, %xmm19 {k1}
+vrangess          $ab, (%rax), %xmm17, %xmm19 {k1}
+vrangess          $ab, %xmm16, %xmm17, %xmm19 {z}{k1}
+vrangess          $ab, (%rax), %xmm17, %xmm19 {z}{k1}
+
+vrangess          $ab, {sae}, %xmm16, %xmm17, %xmm19
+vrangess          $ab, {sae}, %xmm16, %xmm17, %xmm19 {k1}
+vrangess          $ab, {sae}, %xmm16, %xmm17, %xmm19 {z}{k1}
+
+vreducepd         $ab, %zmm16, %zmm19
+vreducepd         $ab, (%rax), %zmm19
+vreducepd         $ab, (%rax){1to8}, %zmm19
+vreducepd         $ab, %zmm16, %zmm19 {k1}
+vreducepd         $ab, (%rax), %zmm19 {k1}
+vreducepd         $ab, (%rax){1to8}, %zmm19 {k1}
+vreducepd         $ab, %zmm16, %zmm19 {z}{k1}
+vreducepd         $ab, (%rax), %zmm19 {z}{k1}
+vreducepd         $ab, (%rax){1to8}, %zmm19 {z}{k1}
+
+vreducepd         $ab, {sae}, %zmm16, %zmm19
+vreducepd         $ab, {sae}, %zmm16, %zmm19 {k1}
+vreducepd         $ab, {sae}, %zmm16, %zmm19 {z}{k1}
+
+vreduceps         $ab, %zmm16, %zmm19
+vreduceps         $ab, (%rax), %zmm19
+vreduceps         $ab, (%rax){1to16}, %zmm19
+vreduceps         $ab, %zmm16, %zmm19 {k1}
+vreduceps         $ab, (%rax), %zmm19 {k1}
+vreduceps         $ab, (%rax){1to16}, %zmm19 {k1}
+vreduceps         $ab, %zmm16, %zmm19 {z}{k1}
+vreduceps         $ab, (%rax), %zmm19 {z}{k1}
+vreduceps         $ab, (%rax){1to16}, %zmm19 {z}{k1}
+
+vreduceps         $ab, {sae}, %zmm16, %zmm19
+vreduceps         $ab, {sae}, %zmm16, %zmm19 {k1}
+vreduceps         $ab, {sae}, %zmm16, %zmm19 {z}{k1}
+
+vreducesd         $ab, %xmm16, %xmm17, %xmm19
+vreducesd         $ab, (%rax), %xmm17, %xmm19
+vreducesd         $ab, %xmm16, %xmm17, %xmm19 {k1}
+vreducesd         $ab, (%rax), %xmm17, %xmm19 {k1}
+vreducesd         $ab, %xmm16, %xmm17, %xmm19 {z}{k1}
+vreducesd         $ab, (%rax), %xmm17, %xmm19 {z}{k1}
+
+vreducesd         $ab, {sae}, %xmm16, %xmm17, %xmm19
+vreducesd         $ab, {sae}, %xmm16, %xmm17, %xmm19 {k1}
+vreducesd         $ab, {sae}, %xmm16, %xmm17, %xmm19 {z}{k1}
+
+vreducess         $ab, %xmm16, %xmm17, %xmm19
+vreducess         $ab, (%rax), %xmm17, %xmm19
+vreducess         $ab, %xmm16, %xmm17, %xmm19 {k1}
+vreducess         $ab, (%rax), %xmm17, %xmm19 {k1}
+vreducess         $ab, %xmm16, %xmm17, %xmm19 {z}{k1}
+vreducess         $ab, (%rax), %xmm17, %xmm19 {z}{k1}
+
+vreducess         $ab, {sae}, %xmm16, %xmm17, %xmm19
+vreducess         $ab, {sae}, %xmm16, %xmm17, %xmm19 {k1}
+vreducess         $ab, {sae}, %xmm16, %xmm17, %xmm19 {z}{k1}
+
+vxorpd            %zmm16, %zmm17, %zmm19
+vxorpd            (%rax), %zmm17, %zmm19
+vxorpd            (%rax){1to8}, %zmm17, %zmm19
+vxorpd            %zmm16, %zmm17, %zmm19 {k1}
+vxorpd            (%rax), %zmm17, %zmm19 {k1}
+vxorpd            (%rax){1to8}, %zmm17, %zmm19 {k1}
+vxorpd            %zmm16, %zmm17, %zmm19 {z}{k1}
+vxorpd            (%rax), %zmm17, %zmm19 {z}{k1}
+vxorpd            (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vxorps            %zmm16, %zmm17, %zmm19
+vxorps            (%rax), %zmm17, %zmm19
+vxorps            (%rax){1to16}, %zmm17, %zmm19
+vxorps            %zmm16, %zmm17, %zmm19 {k1}
+vxorps            (%rax), %zmm17, %zmm19 {k1}
+vxorps            (%rax){1to16}, %zmm17, %zmm19 {k1}
+vxorps            %zmm16, %zmm17, %zmm19 {z}{k1}
+vxorps            (%rax), %zmm17, %zmm19 {z}{k1}
+vxorps            (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     0.50                        kaddb	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kaddw	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kandb	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kandnb	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kmovb	%k0, %k2
+# CHECK-NEXT:  1      5     0.33    *                   kmovb	(%rax), %k2
+# CHECK-NEXT:  1      1     0.50           *            kmovb	%k0, (%rax)
+# CHECK-NEXT:  1      1     0.50                        kmovb	%eax, %k2
+# CHECK-NEXT:  1      1     0.50                        kmovb	%k0, %eax
+# CHECK-NEXT:  1      1     0.50                        knotb	%k0, %k2
+# CHECK-NEXT:  1      1     0.50                        korb	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kortestb	%k0, %k2
+# CHECK-NEXT:  1      1     0.50                        kshiftlb	$2, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kshiftrb	$2, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        ktestb	%k0, %k2
+# CHECK-NEXT:  1      1     0.50                        ktestw	%k0, %k2
+# CHECK-NEXT:  1      1     0.50                        kxnorb	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        kxorb	%k0, %k1, %k2
+# CHECK-NEXT:  1      1     0.50                        vandnpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vandnpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vandnpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vandnps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vandnps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vandnps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vandpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vandpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vandpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vandps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vandps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vandps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vbroadcastf32x2	%xmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vbroadcastf32x2	(%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00                        vbroadcastf32x2	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vbroadcastf32x2	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vbroadcastf32x2	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vbroadcastf32x2	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf32x8	(%rax), %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf32x8	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf32x8	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf64x2	(%rax), %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf64x2	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf64x2	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vbroadcasti32x2	%xmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vbroadcasti32x2	(%rax), %zmm19
+# CHECK-NEXT:  1      1     1.00                        vbroadcasti32x2	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vbroadcasti32x2	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vbroadcasti32x2	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vbroadcasti32x2	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti32x8	(%rax), %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti32x8	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti32x8	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti64x2	(%rax), %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti64x2	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti64x2	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      3     2.00                        vcvtpd2qq	%zmm16, %zmm19
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2qq	(%rax), %zmm19
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2qq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  2      3     2.00                        vcvtpd2qq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2qq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2qq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  2      3     2.00                        vcvtpd2qq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2qq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2qq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      3     2.00                        vcvtpd2uqq	%zmm16, %zmm19
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2uqq	(%rax), %zmm19
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2uqq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  2      3     2.00                        vcvtpd2uqq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2uqq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2uqq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  2      3     2.00                        vcvtpd2uqq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2uqq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    2.00    *                   vcvtpd2uqq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      4     1.00                        vcvtps2qq	%ymm16, %zmm19
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2qq	(%rax), %zmm19
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2qq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  2      4     1.00                        vcvtps2qq	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2qq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2qq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  2      4     1.00                        vcvtps2qq	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2qq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2qq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      4     1.00                        vcvtps2uqq	%ymm16, %zmm19
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2uqq	(%rax), %zmm19
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2uqq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  2      4     1.00                        vcvtps2uqq	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2uqq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2uqq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  2      4     1.00                        vcvtps2uqq	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2uqq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      11    1.00    *                   vcvtps2uqq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  4      4     2.00                        vcvtqq2pd	%zmm16, %zmm19
+# CHECK-NEXT:  3      11    2.00    *                   vcvtqq2pd	(%rax), %zmm19
+# CHECK-NEXT:  3      11    2.00    *                   vcvtqq2pd	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  4      4     2.00                        vcvtqq2pd	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  3      11    2.00    *                   vcvtqq2pd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  3      11    2.00    *                   vcvtqq2pd	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  4      4     2.00                        vcvtqq2pd	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      11    2.00    *                   vcvtqq2pd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      11    2.00    *                   vcvtqq2pd	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      3     1.00                        vcvtqq2ps	%zmm16, %ymm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvtqq2ps	(%rax), %ymm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvtqq2ps	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  2      3     1.00                        vcvtqq2ps	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtqq2ps	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtqq2ps	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  2      3     1.00                        vcvtqq2ps	%zmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtqq2ps	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtqq2ps	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      3     2.00                        vcvttpd2qq	%zmm16, %zmm19
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2qq	(%rax), %zmm19
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2qq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  2      3     2.00                        vcvttpd2qq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2qq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2qq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  2      3     2.00                        vcvttpd2qq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2qq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2qq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      3     2.00                        vcvttpd2uqq	%zmm16, %zmm19
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2uqq	(%rax), %zmm19
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2uqq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  2      3     2.00                        vcvttpd2uqq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2uqq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2uqq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  2      3     2.00                        vcvttpd2uqq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2uqq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    2.00    *                   vcvttpd2uqq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      4     1.00                        vcvttps2qq	%ymm16, %zmm19
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2qq	(%rax), %zmm19
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2qq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  2      4     1.00                        vcvttps2qq	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2qq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2qq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  2      4     1.00                        vcvttps2qq	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2qq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2qq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      4     1.00                        vcvttps2uqq	%ymm16, %zmm19
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2uqq	(%rax), %zmm19
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2uqq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  2      4     1.00                        vcvttps2uqq	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2uqq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2uqq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  2      4     1.00                        vcvttps2uqq	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2uqq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      11    1.00    *                   vcvttps2uqq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  4      4     2.00                        vcvtuqq2pd	%zmm16, %zmm19
+# CHECK-NEXT:  3      11    2.00    *                   vcvtuqq2pd	(%rax), %zmm19
+# CHECK-NEXT:  3      11    2.00    *                   vcvtuqq2pd	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  4      4     2.00                        vcvtuqq2pd	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  3      11    2.00    *                   vcvtuqq2pd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  3      11    2.00    *                   vcvtuqq2pd	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  4      4     2.00                        vcvtuqq2pd	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      11    2.00    *                   vcvtuqq2pd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      11    2.00    *                   vcvtuqq2pd	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      3     1.00                        vcvtuqq2ps	%zmm16, %ymm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvtuqq2ps	(%rax), %ymm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvtuqq2ps	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  2      3     1.00                        vcvtuqq2ps	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtuqq2ps	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtuqq2ps	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  2      3     1.00                        vcvtuqq2ps	%zmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtuqq2ps	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtuqq2ps	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vextractf32x8	$1, %zmm16, %ymm19
+# CHECK-NEXT:  1      1     1.00                        vextractf32x8	$1, %zmm16, (%rax)
+# CHECK-NEXT:  1      2     1.00                        vextractf32x8	$1, %zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vextractf32x8	$1, %zmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      2     1.00                        vextractf32x8	$1, %zmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vextractf64x2	$1, %zmm16, %xmm19
+# CHECK-NEXT:  1      1     1.00                        vextractf64x2	$1, %zmm16, (%rax)
+# CHECK-NEXT:  1      2     1.00                        vextractf64x2	$1, %zmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vextractf64x2	$1, %zmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      2     1.00                        vextractf64x2	$1, %zmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vextracti32x8	$1, %zmm16, %ymm19
+# CHECK-NEXT:  1      1     1.00                        vextracti32x8	$1, %zmm16, (%rax)
+# CHECK-NEXT:  1      1     1.00                        vextracti32x8	$1, %zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vextracti32x8	$1, %zmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      1     1.00                        vextracti32x8	$1, %zmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vextracti64x2	$1, %zmm16, %xmm19
+# CHECK-NEXT:  1      1     1.00                        vextracti64x2	$1, %zmm16, (%rax)
+# CHECK-NEXT:  1      1     1.00                        vextracti64x2	$1, %zmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vextracti64x2	$1, %zmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      1     1.00                        vextracti64x2	$1, %zmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     2.00                        vfpclasspd	$171, %zmm16, %k1
+# CHECK-NEXT:  1      9     2.00    *                   vfpclasspdz	$171, (%rax), %k1
+# CHECK-NEXT:  1      9     2.00    *                   vfpclasspd	$171, (%rax){1to8}, %k1
+# CHECK-NEXT:  1      2     2.00                        vfpclasspd	$171, %zmm16, %k1 {%k2}
+# CHECK-NEXT:  1      9     2.00    *                   vfpclasspdz	$171, (%rax), %k1 {%k2}
+# CHECK-NEXT:  1      9     2.00    *                   vfpclasspd	$171, (%rax){1to8}, %k1 {%k2}
+# CHECK-NEXT:  1      2     2.00                        vfpclassps	$171, %zmm16, %k1
+# CHECK-NEXT:  1      9     2.00    *                   vfpclasspsz	$171, (%rax), %k1
+# CHECK-NEXT:  1      9     2.00    *                   vfpclassps	$171, (%rax){1to16}, %k1
+# CHECK-NEXT:  1      2     2.00                        vfpclassps	$171, %zmm16, %k1 {%k2}
+# CHECK-NEXT:  1      9     2.00    *                   vfpclasspsz	$171, (%rax), %k1 {%k2}
+# CHECK-NEXT:  1      9     2.00    *                   vfpclassps	$171, (%rax){1to16}, %k1 {%k2}
+# CHECK-NEXT:  1      2     1.00                        vfpclasssd	$171, %xmm16, %k1
+# CHECK-NEXT:  1      9     1.00    *                   vfpclasssd	$171, (%rax), %k1
+# CHECK-NEXT:  1      2     1.00                        vfpclasssd	$171, %xmm16, %k1 {%k2}
+# CHECK-NEXT:  1      9     1.00    *                   vfpclasssd	$171, (%rax), %k1 {%k2}
+# CHECK-NEXT:  1      2     1.00                        vfpclassss	$171, %xmm16, %k1
+# CHECK-NEXT:  1      9     1.00    *                   vfpclassss	$171, (%rax), %k1
+# CHECK-NEXT:  1      2     1.00                        vfpclassss	$171, %xmm16, %k1 {%k2}
+# CHECK-NEXT:  1      9     1.00    *                   vfpclassss	$171, (%rax), %k1 {%k2}
+# CHECK-NEXT:  1      2     1.00                        vinsertf32x8	$1, %ymm16, %zmm16, %zmm19
+# CHECK-NEXT:  3      9     1.00    *                   vinsertf32x8	$1, (%rax), %zmm16, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vinsertf32x8	$1, %ymm16, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vinsertf32x8	$1, (%rax), %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vinsertf32x8	$1, %ymm16, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vinsertf32x8	$1, (%rax), %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vinsertf64x2	$1, %xmm16, %zmm16, %zmm19
+# CHECK-NEXT:  3      9     1.00    *                   vinsertf64x2	$1, (%rax), %zmm16, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vinsertf64x2	$1, %xmm16, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vinsertf64x2	$1, (%rax), %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vinsertf64x2	$1, %xmm16, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vinsertf64x2	$1, (%rax), %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vinserti32x8	$1, %ymm16, %zmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vinserti32x8	$1, (%rax), %zmm16, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vinserti32x8	$1, %ymm16, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vinserti32x8	$1, (%rax), %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vinserti32x8	$1, %ymm16, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vinserti32x8	$1, (%rax), %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vinserti64x2	$1, %xmm16, %zmm16, %zmm19
+# CHECK-NEXT:  1      8     1.00    *                   vinserti64x2	$1, (%rax), %zmm16, %zmm19
+# CHECK-NEXT:  1      1     1.00                        vinserti64x2	$1, %xmm16, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vinserti64x2	$1, (%rax), %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vinserti64x2	$1, %xmm16, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vinserti64x2	$1, (%rax), %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vorpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vorpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vorpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vorps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vorps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vorps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      1     1.00                        vpextrd	$1, %xmm16, %ecx
+# CHECK-NEXT:  2      2     1.00           *            vpextrd	$1, %xmm16, (%rax)
+# CHECK-NEXT:  2      1     1.00                        vpextrq	$1, %xmm16, %rcx
+# CHECK-NEXT:  2      2     1.00           *            vpextrq	$1, %xmm16, (%rax)
+# CHECK-NEXT:  2      1     1.00                        vpinsrd	$1, %ecx, %xmm16, %xmm19
+# CHECK-NEXT:  1      8     1.50    *                   vpinsrd	$1, (%rax), %xmm16, %xmm19
+# CHECK-NEXT:  2      1     1.00                        vpinsrq	$1, %rcx, %xmm16, %xmm19
+# CHECK-NEXT:  1      8     1.50    *                   vpinsrq	$1, (%rax), %xmm16, %xmm19
+# CHECK-NEXT:  1      0     0.17                        vpmovm2d	%k0, %zmm0
+# CHECK-NEXT:  1      0     0.17                        vpmovm2q	%k0, %zmm0
+# CHECK-NEXT:  1      1     1.00                        vpmovd2m	%zmm0, %k0
+# CHECK-NEXT:  1      1     1.00                        vpmovq2m	%zmm0, %k0
+# CHECK-NEXT:  1      3     1.00                        vpmullq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpmullq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      3     1.00                        vpmullq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpmullq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vpmullq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpmullq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vrangepd	$ab, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vrangepd	$ab, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vrangepd	$ab, (%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vrangepd	$ab, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vrangepd	$ab, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vrangepd	$ab, (%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vrangepd	$ab, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vrangepd	$ab, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vrangepd	$ab, (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vrangepd	$ab, {sae}, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vrangepd	$ab, {sae}, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vrangepd	$ab, {sae}, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vrangeps	$ab, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vrangeps	$ab, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vrangeps	$ab, (%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vrangeps	$ab, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vrangeps	$ab, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vrangeps	$ab, (%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vrangeps	$ab, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vrangeps	$ab, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vrangeps	$ab, (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vrangeps	$ab, {sae}, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vrangeps	$ab, {sae}, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vrangeps	$ab, {sae}, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vrangesd	$ab, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vrangesd	$ab, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vrangesd	$ab, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vrangesd	$ab, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vrangesd	$ab, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vrangesd	$ab, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vrangesd	$ab, {sae}, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vrangesd	$ab, {sae}, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vrangesd	$ab, {sae}, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vrangess	$ab, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vrangess	$ab, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vrangess	$ab, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vrangess	$ab, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vrangess	$ab, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vrangess	$ab, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vrangess	$ab, {sae}, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vrangess	$ab, {sae}, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vrangess	$ab, {sae}, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      6     3.00                        vreducepd	$ab, %zmm16, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vreducepd	$ab, (%rax), %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vreducepd	$ab, (%rax){1to8}, %zmm19
+# CHECK-NEXT:  2      6     3.00                        vreducepd	$ab, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vreducepd	$ab, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vreducepd	$ab, (%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  2      6     3.00                        vreducepd	$ab, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vreducepd	$ab, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vreducepd	$ab, (%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      6     3.00                        vreducepd	$ab, {sae}, %zmm16, %zmm19
+# CHECK-NEXT:  2      6     3.00                        vreducepd	$ab, {sae}, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      6     3.00                        vreducepd	$ab, {sae}, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      6     3.00                        vreduceps	$ab, %zmm16, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vreduceps	$ab, (%rax), %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vreduceps	$ab, (%rax){1to16}, %zmm19
+# CHECK-NEXT:  2      6     3.00                        vreduceps	$ab, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vreduceps	$ab, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vreduceps	$ab, (%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  2      6     3.00                        vreduceps	$ab, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vreduceps	$ab, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vreduceps	$ab, (%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      6     3.00                        vreduceps	$ab, {sae}, %zmm16, %zmm19
+# CHECK-NEXT:  2      6     3.00                        vreduceps	$ab, {sae}, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      6     3.00                        vreduceps	$ab, {sae}, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      6     3.00                        vreducesd	$ab, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vreducesd	$ab, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  2      6     3.00                        vreducesd	$ab, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vreducesd	$ab, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  2      6     3.00                        vreducesd	$ab, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vreducesd	$ab, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      6     3.00                        vreducesd	$ab, {sae}, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  2      6     3.00                        vreducesd	$ab, {sae}, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  2      6     3.00                        vreducesd	$ab, {sae}, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      6     3.00                        vreducess	$ab, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vreducess	$ab, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  2      6     3.00                        vreducess	$ab, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vreducess	$ab, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  2      6     3.00                        vreducess	$ab, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vreducess	$ab, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      6     3.00                        vreducess	$ab, {sae}, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  2      6     3.00                        vreducess	$ab, {sae}, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  2      6     3.00                        vreducess	$ab, {sae}, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vxorpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vxorpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vxorpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vxorps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vxorps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vxorps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 0.33   0.33   0.33   2.00   2.00   2.00   2.00    -     98.00  147.00 313.50 312.50 113.50 113.50 10.00  71.33  71.33  71.33  68.00  68.00  68.00  5.00   5.00
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kaddb	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kaddw	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kandb	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kandnb	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kmovb	%k0, %k2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     kmovb	(%rax), %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -     kmovb	%k0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -     kmovb	%eax, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kmovb	%k0, %eax
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     knotb	%k0, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     korb	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kortestb	%k0, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     kshiftlb	$2, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     kshiftrb	$2, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     ktestb	%k0, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     ktestw	%k0, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kxnorb	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     kxorb	%k0, %k1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vandnpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vandnpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vandnpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vandnps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vandnps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vandnps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vandpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vandpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vandpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vandps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vandps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vandps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastf32x2	%xmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf32x2	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastf32x2	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf32x2	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastf32x2	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf32x2	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf32x8	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf32x8	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf32x8	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf64x2	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf64x2	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf64x2	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcasti32x2	%xmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti32x2	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcasti32x2	%xmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti32x2	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcasti32x2	%xmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti32x2	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti32x8	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti32x8	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti32x8	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti64x2	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti64x2	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti64x2	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2qq	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2qq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2qq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2qq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2qq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2qq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2qq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2qq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2qq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2uqq	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2uqq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2uqq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2uqq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2uqq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2uqq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2uqq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2uqq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2uqq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2qq	%ymm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2qq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2qq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2qq	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2qq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2qq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2qq	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2qq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2qq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2uqq	%ymm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2uqq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2uqq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2uqq	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2uqq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2uqq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2uqq	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2uqq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2uqq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtqq2pd	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2pd	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2pd	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtqq2pd	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2pd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2pd	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtqq2pd	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2pd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2pd	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtqq2ps	%zmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2ps	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2ps	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtqq2ps	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2ps	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2ps	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtqq2ps	%zmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2ps	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2ps	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2qq	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2qq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2qq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2qq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2qq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2qq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2qq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2qq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2qq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2uqq	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2uqq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2uqq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2uqq	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2uqq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2uqq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2uqq	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2uqq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2uqq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttps2qq	%ymm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2qq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2qq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttps2qq	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2qq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2qq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttps2qq	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2qq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2qq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttps2uqq	%ymm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2uqq	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2uqq	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttps2uqq	%ymm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2uqq	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2uqq	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttps2uqq	%ymm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2uqq	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2uqq	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtuqq2pd	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2pd	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2pd	(%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtuqq2pd	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2pd	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2pd	(%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     vcvtuqq2pd	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2pd	(%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2pd	(%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtuqq2ps	%zmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2ps	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2ps	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtuqq2ps	%zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2ps	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2ps	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtuqq2ps	%zmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2ps	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2ps	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vextractf32x8	$1, %zmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vextractf32x8	$1, %zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vextractf32x8	$1, %zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vextractf32x8	$1, %zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vextractf32x8	$1, %zmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vextractf64x2	$1, %zmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vextractf64x2	$1, %zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vextractf64x2	$1, %zmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vextractf64x2	$1, %zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vextractf64x2	$1, %zmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vextracti32x8	$1, %zmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vextracti32x8	$1, %zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vextracti32x8	$1, %zmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vextracti32x8	$1, %zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vextracti32x8	$1, %zmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vextracti64x2	$1, %zmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vextracti64x2	$1, %zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vextracti64x2	$1, %zmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vextracti64x2	$1, %zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vextracti64x2	$1, %zmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfpclasspd	$171, %zmm16, %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasspdz	$171, (%rax), %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasspd	$171, (%rax){1to8}, %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfpclasspd	$171, %zmm16, %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasspdz	$171, (%rax), %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasspd	$171, (%rax){1to8}, %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfpclassps	$171, %zmm16, %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasspsz	$171, (%rax), %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclassps	$171, (%rax){1to16}, %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfpclassps	$171, %zmm16, %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasspsz	$171, (%rax), %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclassps	$171, (%rax){1to16}, %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfpclasssd	$171, %xmm16, %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasssd	$171, (%rax), %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfpclasssd	$171, %xmm16, %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasssd	$171, (%rax), %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfpclassss	$171, %xmm16, %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclassss	$171, (%rax), %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfpclassss	$171, %xmm16, %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclassss	$171, (%rax), %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinsertf32x8	$1, %ymm16, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinsertf32x8	$1, (%rax), %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinsertf32x8	$1, %ymm16, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinsertf32x8	$1, (%rax), %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinsertf32x8	$1, %ymm16, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinsertf32x8	$1, (%rax), %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinsertf64x2	$1, %xmm16, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinsertf64x2	$1, (%rax), %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinsertf64x2	$1, %xmm16, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinsertf64x2	$1, (%rax), %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinsertf64x2	$1, %xmm16, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinsertf64x2	$1, (%rax), %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinserti32x8	$1, %ymm16, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinserti32x8	$1, (%rax), %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinserti32x8	$1, %ymm16, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinserti32x8	$1, (%rax), %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinserti32x8	$1, %ymm16, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinserti32x8	$1, (%rax), %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinserti64x2	$1, %xmm16, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinserti64x2	$1, (%rax), %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinserti64x2	$1, %xmm16, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinserti64x2	$1, (%rax), %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinserti64x2	$1, %xmm16, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinserti64x2	$1, (%rax), %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vorpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vorpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vorpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vorps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vorps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vorps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vpextrd	$1, %xmm16, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vpextrd	$1, %xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vpextrq	$1, %xmm16, %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vpextrq	$1, %xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vpinsrd	$1, %ecx, %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpinsrd	$1, (%rax), %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     vpinsrq	$1, %rcx, %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpinsrq	$1, (%rax), %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovm2d	%k0, %zmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovm2q	%k0, %zmm0
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovd2m	%zmm0, %k0
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovq2m	%zmm0, %k0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmullq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmullq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpmullq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangepd	$ab, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangepd	$ab, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangepd	$ab, (%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangepd	$ab, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangepd	$ab, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangepd	$ab, (%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangepd	$ab, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangepd	$ab, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangepd	$ab, (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangepd	$ab, {sae}, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangepd	$ab, {sae}, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangepd	$ab, {sae}, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangeps	$ab, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangeps	$ab, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangeps	$ab, (%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangeps	$ab, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangeps	$ab, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangeps	$ab, (%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangeps	$ab, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangeps	$ab, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangeps	$ab, (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangeps	$ab, {sae}, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangeps	$ab, {sae}, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangeps	$ab, {sae}, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangesd	$ab, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangesd	$ab, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangesd	$ab, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangesd	$ab, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangesd	$ab, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangesd	$ab, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangesd	$ab, {sae}, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangesd	$ab, {sae}, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangesd	$ab, {sae}, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangess	$ab, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangess	$ab, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangess	$ab, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangess	$ab, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangess	$ab, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangess	$ab, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangess	$ab, {sae}, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangess	$ab, {sae}, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangess	$ab, {sae}, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducepd	$ab, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducepd	$ab, (%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducepd	$ab, (%rax){1to8}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducepd	$ab, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducepd	$ab, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducepd	$ab, (%rax){1to8}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducepd	$ab, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducepd	$ab, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducepd	$ab, (%rax){1to8}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducepd	$ab, {sae}, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducepd	$ab, {sae}, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducepd	$ab, {sae}, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreduceps	$ab, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreduceps	$ab, (%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreduceps	$ab, (%rax){1to16}, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreduceps	$ab, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreduceps	$ab, (%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreduceps	$ab, (%rax){1to16}, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreduceps	$ab, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreduceps	$ab, (%rax), %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreduceps	$ab, (%rax){1to16}, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreduceps	$ab, {sae}, %zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreduceps	$ab, {sae}, %zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreduceps	$ab, {sae}, %zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducesd	$ab, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducesd	$ab, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducesd	$ab, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducesd	$ab, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducesd	$ab, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducesd	$ab, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducesd	$ab, {sae}, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducesd	$ab, {sae}, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducesd	$ab, {sae}, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducess	$ab, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducess	$ab, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducess	$ab, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducess	$ab, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducess	$ab, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducess	$ab, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducess	$ab, {sae}, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducess	$ab, {sae}, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducess	$ab, {sae}, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vxorpd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vxorpd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vxorpd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vxorps	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vxorps	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vxorps	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512dqvl.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512dqvl.s
new file mode 100644
index 0000000000000..37e42daa955f2
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512dqvl.s
@@ -0,0 +1,1677 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vandnpd           %xmm16, %xmm17, %xmm19
+vandnpd           (%rax), %xmm17, %xmm19
+vandnpd           (%rax){1to2}, %xmm17, %xmm19
+vandnpd           %xmm16, %xmm17, %xmm19 {k1}
+vandnpd           (%rax), %xmm17, %xmm19 {k1}
+vandnpd           (%rax){1to2}, %xmm17, %xmm19 {k1}
+vandnpd           %xmm16, %xmm17, %xmm19 {z}{k1}
+vandnpd           (%rax), %xmm17, %xmm19 {z}{k1}
+vandnpd           (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+vandnpd           %ymm16, %ymm17, %ymm19
+vandnpd           (%rax), %ymm17, %ymm19
+vandnpd           (%rax){1to4}, %ymm17, %ymm19
+vandnpd           %ymm16, %ymm17, %ymm19 {k1}
+vandnpd           (%rax), %ymm17, %ymm19 {k1}
+vandnpd           (%rax){1to4}, %ymm17, %ymm19 {k1}
+vandnpd           %ymm16, %ymm17, %ymm19 {z}{k1}
+vandnpd           (%rax), %ymm17, %ymm19 {z}{k1}
+vandnpd           (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vandnps           %xmm16, %xmm17, %xmm19
+vandnps           (%rax), %xmm17, %xmm19
+vandnps           (%rax){1to4}, %xmm17, %xmm19
+vandnps           %xmm16, %xmm17, %xmm19 {k1}
+vandnps           (%rax), %xmm17, %xmm19 {k1}
+vandnps           (%rax){1to4}, %xmm17, %xmm19 {k1}
+vandnps           %xmm16, %xmm17, %xmm19 {z}{k1}
+vandnps           (%rax), %xmm17, %xmm19 {z}{k1}
+vandnps           (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vandnps           %ymm16, %ymm17, %ymm19
+vandnps           (%rax), %ymm17, %ymm19
+vandnps           (%rax){1to8}, %ymm17, %ymm19
+vandnps           %ymm16, %ymm17, %ymm19 {k1}
+vandnps           (%rax), %ymm17, %ymm19 {k1}
+vandnps           (%rax){1to8}, %ymm17, %ymm19 {k1}
+vandnps           %ymm16, %ymm17, %ymm19 {z}{k1}
+vandnps           (%rax), %ymm17, %ymm19 {z}{k1}
+vandnps           (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vandpd            %xmm16, %xmm17, %xmm19
+vandpd            (%rax), %xmm17, %xmm19
+vandpd            (%rax){1to2}, %xmm17, %xmm19
+vandpd            %xmm16, %xmm17, %xmm19 {k1}
+vandpd            (%rax), %xmm17, %xmm19 {k1}
+vandpd            (%rax){1to2}, %xmm17, %xmm19 {k1}
+vandpd            %xmm16, %xmm17, %xmm19 {z}{k1}
+vandpd            (%rax), %xmm17, %xmm19 {z}{k1}
+vandpd            (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+vandpd            %ymm16, %ymm17, %ymm19
+vandpd            (%rax), %ymm17, %ymm19
+vandpd            (%rax){1to4}, %ymm17, %ymm19
+vandpd            %ymm16, %ymm17, %ymm19 {k1}
+vandpd            (%rax), %ymm17, %ymm19 {k1}
+vandpd            (%rax){1to4}, %ymm17, %ymm19 {k1}
+vandpd            %ymm16, %ymm17, %ymm19 {z}{k1}
+vandpd            (%rax), %ymm17, %ymm19 {z}{k1}
+vandpd            (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vandps            %xmm16, %xmm17, %xmm19
+vandps            (%rax), %xmm17, %xmm19
+vandps            (%rax){1to4}, %xmm17, %xmm19
+vandps            %xmm16, %xmm17, %xmm19 {k1}
+vandps            (%rax), %xmm17, %xmm19 {k1}
+vandps            (%rax){1to4}, %xmm17, %xmm19 {k1}
+vandps            %xmm16, %xmm17, %xmm19 {z}{k1}
+vandps            (%rax), %xmm17, %xmm19 {z}{k1}
+vandps            (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vandps            %ymm16, %ymm17, %ymm19
+vandps            (%rax), %ymm17, %ymm19
+vandps            (%rax){1to8}, %ymm17, %ymm19
+vandps            %ymm16, %ymm17, %ymm19 {k1}
+vandps            (%rax), %ymm17, %ymm19 {k1}
+vandps            (%rax){1to8}, %ymm17, %ymm19 {k1}
+vandps            %ymm16, %ymm17, %ymm19 {z}{k1}
+vandps            (%rax), %ymm17, %ymm19 {z}{k1}
+vandps            (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vbroadcastf32x2   %xmm16, %ymm19
+vbroadcastf32x2   (%rax), %ymm19
+vbroadcastf32x2   %xmm16, %ymm19 {k1}
+vbroadcastf32x2   (%rax), %ymm19 {k1}
+vbroadcastf32x2   %xmm16, %ymm19 {z}{k1}
+vbroadcastf32x2   (%rax), %ymm19 {z}{k1}
+
+vbroadcastf64x2   (%rax), %ymm19
+vbroadcastf64x2   (%rax), %ymm19 {k1}
+vbroadcastf64x2   (%rax), %ymm19 {z}{k1}
+
+vbroadcasti32x2   %xmm16, %xmm19
+vbroadcasti32x2   (%rax), %xmm19
+vbroadcasti32x2   %xmm16, %xmm19 {k1}
+vbroadcasti32x2   (%rax), %xmm19 {k1}
+vbroadcasti32x2   %xmm16, %xmm19 {z}{k1}
+vbroadcasti32x2   (%rax), %xmm19 {z}{k1}
+
+vbroadcasti32x2   %xmm16, %ymm19
+vbroadcasti32x2   (%rax), %ymm19
+vbroadcasti32x2   %xmm16, %ymm19 {k1}
+vbroadcasti32x2   (%rax), %ymm19 {k1}
+vbroadcasti32x2   %xmm16, %ymm19 {z}{k1}
+vbroadcasti32x2   (%rax), %ymm19 {z}{k1}
+
+vbroadcasti64x2   (%rax), %ymm19
+vbroadcasti64x2   (%rax), %ymm19 {k1}
+vbroadcasti64x2   (%rax), %ymm19 {z}{k1}
+
+vcvtpd2qq         %xmm16, %xmm19
+vcvtpd2qq         (%rax), %xmm19
+vcvtpd2qq         (%rax){1to2}, %xmm19
+vcvtpd2qq         %xmm16, %xmm19 {k1}
+vcvtpd2qq         (%rax), %xmm19 {k1}
+vcvtpd2qq         (%rax){1to2}, %xmm19 {k1}
+vcvtpd2qq         %xmm16, %xmm19 {z}{k1}
+vcvtpd2qq         (%rax), %xmm19 {z}{k1}
+vcvtpd2qq         (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvtpd2qq         %ymm16, %ymm19
+vcvtpd2qq         (%rax), %ymm19
+vcvtpd2qq         (%rax){1to4}, %ymm19
+vcvtpd2qq         %ymm16, %ymm19 {k1}
+vcvtpd2qq         (%rax), %ymm19 {k1}
+vcvtpd2qq         (%rax){1to4}, %ymm19 {k1}
+vcvtpd2qq         %ymm16, %ymm19 {z}{k1}
+vcvtpd2qq         (%rax), %ymm19 {z}{k1}
+vcvtpd2qq         (%rax){1to4}, %ymm19 {z}{k1}
+
+vcvtpd2uqq        %xmm16, %xmm19
+vcvtpd2uqq        (%rax), %xmm19
+vcvtpd2uqq        (%rax){1to2}, %xmm19
+vcvtpd2uqq        %xmm16, %xmm19 {k1}
+vcvtpd2uqq        (%rax), %xmm19 {k1}
+vcvtpd2uqq        (%rax){1to2}, %xmm19 {k1}
+vcvtpd2uqq        %xmm16, %xmm19 {z}{k1}
+vcvtpd2uqq        (%rax), %xmm19 {z}{k1}
+vcvtpd2uqq        (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvtpd2uqq        %ymm16, %ymm19
+vcvtpd2uqq        (%rax), %ymm19
+vcvtpd2uqq        (%rax){1to4}, %ymm19
+vcvtpd2uqq        %ymm16, %ymm19 {k1}
+vcvtpd2uqq        (%rax), %ymm19 {k1}
+vcvtpd2uqq        (%rax){1to4}, %ymm19 {k1}
+vcvtpd2uqq        %ymm16, %ymm19 {z}{k1}
+vcvtpd2uqq        (%rax), %ymm19 {z}{k1}
+vcvtpd2uqq        (%rax){1to4}, %ymm19 {z}{k1}
+
+vcvtps2qq         %xmm16, %xmm19
+vcvtps2qq         (%rax), %xmm19
+vcvtps2qq         (%rax){1to2}, %xmm19
+vcvtps2qq         %xmm16, %xmm19 {k1}
+vcvtps2qq         (%rax), %xmm19 {k1}
+vcvtps2qq         (%rax){1to2}, %xmm19 {k1}
+vcvtps2qq         %xmm16, %xmm19 {z}{k1}
+vcvtps2qq         (%rax), %xmm19 {z}{k1}
+vcvtps2qq         (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvtps2qq         %xmm16, %ymm19
+vcvtps2qq         (%rax), %ymm19
+vcvtps2qq         (%rax){1to4}, %ymm19
+vcvtps2qq         %xmm16, %ymm19 {k1}
+vcvtps2qq         (%rax), %ymm19 {k1}
+vcvtps2qq         (%rax){1to4}, %ymm19 {k1}
+vcvtps2qq         %xmm16, %ymm19 {z}{k1}
+vcvtps2qq         (%rax), %ymm19 {z}{k1}
+vcvtps2qq         (%rax){1to4}, %ymm19 {z}{k1}
+
+vcvtps2uqq        %xmm16, %xmm19
+vcvtps2uqq        (%rax), %xmm19
+vcvtps2uqq        (%rax){1to2}, %xmm19
+vcvtps2uqq        %xmm16, %xmm19 {k1}
+vcvtps2uqq        (%rax), %xmm19 {k1}
+vcvtps2uqq        (%rax){1to2}, %xmm19 {k1}
+vcvtps2uqq        %xmm16, %xmm19 {z}{k1}
+vcvtps2uqq        (%rax), %xmm19 {z}{k1}
+vcvtps2uqq        (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvtps2uqq        %xmm16, %ymm19
+vcvtps2uqq        (%rax), %ymm19
+vcvtps2uqq        (%rax){1to4}, %ymm19
+vcvtps2uqq        %xmm16, %ymm19 {k1}
+vcvtps2uqq        (%rax), %ymm19 {k1}
+vcvtps2uqq        (%rax){1to4}, %ymm19 {k1}
+vcvtps2uqq        %xmm16, %ymm19 {z}{k1}
+vcvtps2uqq        (%rax), %ymm19 {z}{k1}
+vcvtps2uqq        (%rax){1to4}, %ymm19 {z}{k1}
+
+vcvtqq2pd         %xmm16, %xmm19
+vcvtqq2pd         (%rax), %xmm19
+vcvtqq2pd         (%rax){1to2}, %xmm19
+vcvtqq2pd         %xmm16, %xmm19 {k1}
+vcvtqq2pd         (%rax), %xmm19 {k1}
+vcvtqq2pd         (%rax){1to2}, %xmm19 {k1}
+vcvtqq2pd         %xmm16, %xmm19 {z}{k1}
+vcvtqq2pd         (%rax), %xmm19 {z}{k1}
+vcvtqq2pd         (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvtqq2pd         %ymm16, %ymm19
+vcvtqq2pd         (%rax), %ymm19
+vcvtqq2pd         (%rax){1to4}, %ymm19
+vcvtqq2pd         %ymm16, %ymm19 {k1}
+vcvtqq2pd         (%rax), %ymm19 {k1}
+vcvtqq2pd         (%rax){1to4}, %ymm19 {k1}
+vcvtqq2pd         %ymm16, %ymm19 {z}{k1}
+vcvtqq2pd         (%rax), %ymm19 {z}{k1}
+vcvtqq2pd         (%rax){1to4}, %ymm19 {z}{k1}
+
+vcvtqq2ps         %xmm16, %xmm19
+vcvtqq2psx        (%rax), %xmm19
+vcvtqq2ps         (%rax){1to2}, %xmm19
+vcvtqq2ps         %xmm16, %xmm19 {k1}
+vcvtqq2psx        (%rax), %xmm19 {k1}
+vcvtqq2ps         (%rax){1to2}, %xmm19 {k1}
+vcvtqq2ps         %xmm16, %xmm19 {z}{k1}
+vcvtqq2psx        (%rax), %xmm19 {z}{k1}
+vcvtqq2ps         (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvtqq2ps         %ymm16, %xmm19
+vcvtqq2psx        (%rax), %xmm19
+vcvtqq2ps         (%rax){1to4}, %xmm19
+vcvtqq2ps         %ymm16, %xmm19 {k1}
+vcvtqq2psx        (%rax), %xmm19 {k1}
+vcvtqq2ps         (%rax){1to4}, %xmm19 {k1}
+vcvtqq2ps         %ymm16, %xmm19 {z}{k1}
+vcvtqq2psx        (%rax), %xmm19 {z}{k1}
+vcvtqq2ps         (%rax){1to4}, %xmm19 {z}{k1}
+
+vcvttpd2qq        %xmm16, %xmm19
+vcvttpd2qq        (%rax), %xmm19
+vcvttpd2qq        (%rax){1to2}, %xmm19
+vcvttpd2qq        %xmm16, %xmm19 {k1}
+vcvttpd2qq        (%rax), %xmm19 {k1}
+vcvttpd2qq        (%rax){1to2}, %xmm19 {k1}
+vcvttpd2qq        %xmm16, %xmm19 {z}{k1}
+vcvttpd2qq        (%rax), %xmm19 {z}{k1}
+vcvttpd2qq        (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvttpd2qq        %ymm16, %ymm19
+vcvttpd2qq        (%rax), %ymm19
+vcvttpd2qq        (%rax){1to4}, %ymm19
+vcvttpd2qq        %ymm16, %ymm19 {k1}
+vcvttpd2qq        (%rax), %ymm19 {k1}
+vcvttpd2qq        (%rax){1to4}, %ymm19 {k1}
+vcvttpd2qq        %ymm16, %ymm19 {z}{k1}
+vcvttpd2qq        (%rax), %ymm19 {z}{k1}
+vcvttpd2qq        (%rax){1to4}, %ymm19 {z}{k1}
+
+vcvttpd2uqq       %xmm16, %xmm19
+vcvttpd2uqq       (%rax), %xmm19
+vcvttpd2uqq       (%rax){1to2}, %xmm19
+vcvttpd2uqq       %xmm16, %xmm19 {k1}
+vcvttpd2uqq       (%rax), %xmm19 {k1}
+vcvttpd2uqq       (%rax){1to2}, %xmm19 {k1}
+vcvttpd2uqq       %xmm16, %xmm19 {z}{k1}
+vcvttpd2uqq       (%rax), %xmm19 {z}{k1}
+vcvttpd2uqq       (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvttpd2uqq       %ymm16, %ymm19
+vcvttpd2uqq       (%rax), %ymm19
+vcvttpd2uqq       (%rax){1to4}, %ymm19
+vcvttpd2uqq       %ymm16, %ymm19 {k1}
+vcvttpd2uqq       (%rax), %ymm19 {k1}
+vcvttpd2uqq       (%rax){1to4}, %ymm19 {k1}
+vcvttpd2uqq       %ymm16, %ymm19 {z}{k1}
+vcvttpd2uqq       (%rax), %ymm19 {z}{k1}
+vcvttpd2uqq       (%rax){1to4}, %ymm19 {z}{k1}
+
+vcvttps2qq        %xmm16, %xmm19
+vcvttps2qq        (%rax), %xmm19
+vcvttps2qq        (%rax){1to2}, %xmm19
+vcvttps2qq        %xmm16, %xmm19 {k1}
+vcvttps2qq        (%rax), %xmm19 {k1}
+vcvttps2qq        (%rax){1to2}, %xmm19 {k1}
+vcvttps2qq        %xmm16, %xmm19 {z}{k1}
+vcvttps2qq        (%rax), %xmm19 {z}{k1}
+vcvttps2qq        (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvttps2qq        %xmm16, %ymm19
+vcvttps2qq        (%rax), %ymm19
+vcvttps2qq        (%rax){1to4}, %ymm19
+vcvttps2qq        %xmm16, %ymm19 {k1}
+vcvttps2qq        (%rax), %ymm19 {k1}
+vcvttps2qq        (%rax){1to4}, %ymm19 {k1}
+vcvttps2qq        %xmm16, %ymm19 {z}{k1}
+vcvttps2qq        (%rax), %ymm19 {z}{k1}
+vcvttps2qq        (%rax){1to4}, %ymm19 {z}{k1}
+
+vcvttps2uqq       %xmm16, %xmm19
+vcvttps2uqq       (%rax), %xmm19
+vcvttps2uqq       (%rax){1to2}, %xmm19
+vcvttps2uqq       %xmm16, %xmm19 {k1}
+vcvttps2uqq       (%rax), %xmm19 {k1}
+vcvttps2uqq       (%rax){1to2}, %xmm19 {k1}
+vcvttps2uqq       %xmm16, %xmm19 {z}{k1}
+vcvttps2uqq       (%rax), %xmm19 {z}{k1}
+vcvttps2uqq       (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvttps2uqq       %xmm16, %ymm19
+vcvttps2uqq       (%rax), %ymm19
+vcvttps2uqq       (%rax){1to4}, %ymm19
+vcvttps2uqq       %xmm16, %ymm19 {k1}
+vcvttps2uqq       (%rax), %ymm19 {k1}
+vcvttps2uqq       (%rax){1to4}, %ymm19 {k1}
+vcvttps2uqq       %xmm16, %ymm19 {z}{k1}
+vcvttps2uqq       (%rax), %ymm19 {z}{k1}
+vcvttps2uqq       (%rax){1to4}, %ymm19 {z}{k1}
+
+vcvtuqq2pd        %xmm16, %xmm19
+vcvtuqq2pd        (%rax), %xmm19
+vcvtuqq2pd        (%rax){1to2}, %xmm19
+vcvtuqq2pd        %xmm16, %xmm19 {k1}
+vcvtuqq2pd        (%rax), %xmm19 {k1}
+vcvtuqq2pd        (%rax){1to2}, %xmm19 {k1}
+vcvtuqq2pd        %xmm16, %xmm19 {z}{k1}
+vcvtuqq2pd        (%rax), %xmm19 {z}{k1}
+vcvtuqq2pd        (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvtuqq2pd        %ymm16, %ymm19
+vcvtuqq2pd        (%rax), %ymm19
+vcvtuqq2pd        (%rax){1to4}, %ymm19
+vcvtuqq2pd        %ymm16, %ymm19 {k1}
+vcvtuqq2pd        (%rax), %ymm19 {k1}
+vcvtuqq2pd        (%rax){1to4}, %ymm19 {k1}
+vcvtuqq2pd        %ymm16, %ymm19 {z}{k1}
+vcvtuqq2pd        (%rax), %ymm19 {z}{k1}
+vcvtuqq2pd        (%rax){1to4}, %ymm19 {z}{k1}
+
+vcvtuqq2ps        %xmm16, %xmm19
+vcvtuqq2psx       (%rax), %xmm19
+vcvtuqq2ps        (%rax){1to2}, %xmm19
+vcvtuqq2ps        %xmm16, %xmm19 {k1}
+vcvtuqq2psx       (%rax), %xmm19 {k1}
+vcvtuqq2ps        (%rax){1to2}, %xmm19 {k1}
+vcvtuqq2ps        %xmm16, %xmm19 {z}{k1}
+vcvtuqq2psx       (%rax), %xmm19 {z}{k1}
+vcvtuqq2ps        (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvtuqq2ps        %ymm16, %xmm19
+vcvtuqq2psx       (%rax), %xmm19
+vcvtuqq2ps        (%rax){1to4}, %xmm19
+vcvtuqq2ps        %ymm16, %xmm19 {k1}
+vcvtuqq2psx       (%rax), %xmm19 {k1}
+vcvtuqq2ps        (%rax){1to4}, %xmm19 {k1}
+vcvtuqq2ps        %ymm16, %xmm19 {z}{k1}
+vcvtuqq2psx       (%rax), %xmm19 {z}{k1}
+vcvtuqq2ps        (%rax){1to4}, %xmm19 {z}{k1}
+
+vextractf64x2     $1, %ymm16, %xmm19
+vextractf64x2     $1, %ymm16, (%rax)
+vextractf64x2     $1, %ymm16, %xmm19 {k1}
+vextractf64x2     $1, %ymm16, (%rax) {k1}
+vextractf64x2     $1, %ymm16, %xmm19 {z}{k1}
+
+vextracti64x2     $1, %ymm16, %xmm19
+vextracti64x2     $1, %ymm16, (%rax)
+vextracti64x2     $1, %ymm16, %xmm19 {k1}
+vextracti64x2     $1, %ymm16, (%rax) {k1}
+vextracti64x2     $1, %ymm16, %xmm19 {z}{k1}
+
+vfpclasspd        $0xab, %xmm16, %k1
+vfpclasspdx       $0xab, (%rax), %k1
+vfpclasspdx       $0xab, (%rax){1to2}, %k1
+vfpclasspd        $0xab, %xmm16, %k1 {k2}
+vfpclasspdx       $0xab, (%rax), %k1 {k2}
+vfpclasspdx       $0xab, (%rax){1to2}, %k1 {k2}
+
+vfpclasspd        $0xab, %ymm16, %k1
+vfpclasspdy       $0xab, (%rax), %k1
+vfpclasspdy       $0xab, (%rax){1to4}, %k1
+vfpclasspd        $0xab, %ymm16, %k1 {k2}
+vfpclasspdy       $0xab, (%rax), %k1 {k2}
+vfpclasspdy       $0xab, (%rax){1to4}, %k1 {k2}
+
+vfpclassps        $0xab, %xmm16, %k1
+vfpclasspsx       $0xab, (%rax), %k1
+vfpclasspsx       $0xab, (%rax){1to4}, %k1
+vfpclassps        $0xab, %xmm16, %k1 {k2}
+vfpclasspsx       $0xab, (%rax), %k1 {k2}
+vfpclasspsx       $0xab, (%rax){1to4}, %k1 {k2}
+
+vfpclassps        $0xab, %ymm16, %k1
+vfpclasspsy       $0xab, (%rax), %k1
+vfpclasspsy       $0xab, (%rax){1to8}, %k1
+vfpclassps        $0xab, %ymm16, %k1 {k2}
+vfpclasspsy       $0xab, (%rax), %k1 {k2}
+vfpclasspsy       $0xab, (%rax){1to8}, %k1 {k2}
+
+vinsertf64x2      $1, %xmm16, %ymm16, %ymm19
+vinsertf64x2      $1, (%rax), %ymm16, %ymm19
+vinsertf64x2      $1, %xmm16, %ymm16, %ymm19 {k1}
+vinsertf64x2      $1, (%rax), %ymm16, %ymm19 {k1}
+vinsertf64x2      $1, %xmm16, %ymm16, %ymm19 {z}{k1}
+vinsertf64x2      $1, (%rax), %ymm16, %ymm19 {z}{k1}
+
+vinserti64x2      $1, %xmm16, %ymm16, %ymm19
+vinserti64x2      $1, (%rax), %ymm16, %ymm19
+vinserti64x2      $1, %xmm16, %ymm16, %ymm19 {k1}
+vinserti64x2      $1, (%rax), %ymm16, %ymm19 {k1}
+vinserti64x2      $1, %xmm16, %ymm16, %ymm19 {z}{k1}
+vinserti64x2      $1, (%rax), %ymm16, %ymm19 {z}{k1}
+
+vpmovm2d          %k0, %xmm0
+vpmovm2q          %k0, %xmm0
+
+vpmovm2d          %k0, %ymm0
+vpmovm2q          %k0, %ymm0
+
+vpmovd2m          %xmm0, %k0
+vpmovq2m          %xmm0, %k0
+
+vpmovd2m          %ymm0, %k0
+vpmovq2m          %ymm0, %k0
+
+vorpd             %xmm16, %xmm17, %xmm19
+vorpd             (%rax), %xmm17, %xmm19
+vorpd             (%rax){1to2}, %xmm17, %xmm19
+vorpd             %xmm16, %xmm17, %xmm19 {k1}
+vorpd             (%rax), %xmm17, %xmm19 {k1}
+vorpd             (%rax){1to2}, %xmm17, %xmm19 {k1}
+vorpd             %xmm16, %xmm17, %xmm19 {z}{k1}
+vorpd             (%rax), %xmm17, %xmm19 {z}{k1}
+vorpd             (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+vorpd             %ymm16, %ymm17, %ymm19
+vorpd             (%rax), %ymm17, %ymm19
+vorpd             (%rax){1to4}, %ymm17, %ymm19
+vorpd             %ymm16, %ymm17, %ymm19 {k1}
+vorpd             (%rax), %ymm17, %ymm19 {k1}
+vorpd             (%rax){1to4}, %ymm17, %ymm19 {k1}
+vorpd             %ymm16, %ymm17, %ymm19 {z}{k1}
+vorpd             (%rax), %ymm17, %ymm19 {z}{k1}
+vorpd             (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vorps             %xmm16, %xmm17, %xmm19
+vorps             (%rax), %xmm17, %xmm19
+vorps             (%rax){1to4}, %xmm17, %xmm19
+vorps             %xmm16, %xmm17, %xmm19 {k1}
+vorps             (%rax), %xmm17, %xmm19 {k1}
+vorps             (%rax){1to4}, %xmm17, %xmm19 {k1}
+vorps             %xmm16, %xmm17, %xmm19 {z}{k1}
+vorps             (%rax), %xmm17, %xmm19 {z}{k1}
+vorps             (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vorps             %ymm16, %ymm17, %ymm19
+vorps             (%rax), %ymm17, %ymm19
+vorps             (%rax){1to8}, %ymm17, %ymm19
+vorps             %ymm16, %ymm17, %ymm19 {k1}
+vorps             (%rax), %ymm17, %ymm19 {k1}
+vorps             (%rax){1to8}, %ymm17, %ymm19 {k1}
+vorps             %ymm16, %ymm17, %ymm19 {z}{k1}
+vorps             (%rax), %ymm17, %ymm19 {z}{k1}
+vorps             (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vpmullq           %xmm16, %xmm17, %xmm19
+vpmullq           (%rax), %xmm17, %xmm19
+vpmullq           %xmm16, %xmm17, %xmm19 {k1}
+vpmullq           (%rax), %xmm17, %xmm19 {k1}
+vpmullq           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpmullq           (%rax), %xmm17, %xmm19 {z}{k1}
+
+vpmullq           %ymm16, %ymm17, %ymm19
+vpmullq           (%rax), %ymm17, %ymm19
+vpmullq           %ymm16, %ymm17, %ymm19 {k1}
+vpmullq           (%rax), %ymm17, %ymm19 {k1}
+vpmullq           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpmullq           (%rax), %ymm17, %ymm19 {z}{k1}
+
+vrangepd          $ab, %xmm16, %xmm17, %xmm19
+vrangepd          $ab, (%rax), %xmm17, %xmm19
+vrangepd          $ab, (%rax){1to2}, %xmm17, %xmm19
+vrangepd          $ab, %xmm16, %xmm17, %xmm19 {k1}
+vrangepd          $ab, (%rax), %xmm17, %xmm19 {k1}
+vrangepd          $ab, (%rax){1to2}, %xmm17, %xmm19 {k1}
+vrangepd          $ab, %xmm16, %xmm17, %xmm19 {z}{k1}
+vrangepd          $ab, (%rax), %xmm17, %xmm19 {z}{k1}
+vrangepd          $ab, (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+vrangepd          $ab, %ymm16, %ymm17, %ymm19
+vrangepd          $ab, (%rax), %ymm17, %ymm19
+vrangepd          $ab, (%rax){1to4}, %ymm17, %ymm19
+vrangepd          $ab, %ymm16, %ymm17, %ymm19 {k1}
+vrangepd          $ab, (%rax), %ymm17, %ymm19 {k1}
+vrangepd          $ab, (%rax){1to4}, %ymm17, %ymm19 {k1}
+vrangepd          $ab, %ymm16, %ymm17, %ymm19 {z}{k1}
+vrangepd          $ab, (%rax), %ymm17, %ymm19 {z}{k1}
+vrangepd          $ab, (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vrangeps          $ab, %xmm16, %xmm17, %xmm19
+vrangeps          $ab, (%rax), %xmm17, %xmm19
+vrangeps          $ab, (%rax){1to4}, %xmm17, %xmm19
+vrangeps          $ab, %xmm16, %xmm17, %xmm19 {k1}
+vrangeps          $ab, (%rax), %xmm17, %xmm19 {k1}
+vrangeps          $ab, (%rax){1to4}, %xmm17, %xmm19 {k1}
+vrangeps          $ab, %xmm16, %xmm17, %xmm19 {z}{k1}
+vrangeps          $ab, (%rax), %xmm17, %xmm19 {z}{k1}
+vrangeps          $ab, (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vrangeps          $ab, %ymm16, %ymm17, %ymm19
+vrangeps          $ab, (%rax), %ymm17, %ymm19
+vrangeps          $ab, (%rax){1to8}, %ymm17, %ymm19
+vrangeps          $ab, %ymm16, %ymm17, %ymm19 {k1}
+vrangeps          $ab, (%rax), %ymm17, %ymm19 {k1}
+vrangeps          $ab, (%rax){1to8}, %ymm17, %ymm19 {k1}
+vrangeps          $ab, %ymm16, %ymm17, %ymm19 {z}{k1}
+vrangeps          $ab, (%rax), %ymm17, %ymm19 {z}{k1}
+vrangeps          $ab, (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vreducepd         $ab, %xmm16, %xmm19
+vreducepd         $ab, (%rax), %xmm19
+vreducepd         $ab, (%rax){1to2}, %xmm19
+vreducepd         $ab, %xmm16, %xmm19 {k1}
+vreducepd         $ab, (%rax), %xmm19 {k1}
+vreducepd         $ab, (%rax){1to2}, %xmm19 {k1}
+vreducepd         $ab, %xmm16, %xmm19 {z}{k1}
+vreducepd         $ab, (%rax), %xmm19 {z}{k1}
+vreducepd         $ab, (%rax){1to2}, %xmm19 {z}{k1}
+
+vreducepd         $ab, %ymm16, %ymm19
+vreducepd         $ab, (%rax), %ymm19
+vreducepd         $ab, (%rax){1to4}, %ymm19
+vreducepd         $ab, %ymm16, %ymm19 {k1}
+vreducepd         $ab, (%rax), %ymm19 {k1}
+vreducepd         $ab, (%rax){1to4}, %ymm19 {k1}
+vreducepd         $ab, %ymm16, %ymm19 {z}{k1}
+vreducepd         $ab, (%rax), %ymm19 {z}{k1}
+vreducepd         $ab, (%rax){1to4}, %ymm19 {z}{k1}
+
+vreduceps         $ab, %xmm16, %xmm19
+vreduceps         $ab, (%rax), %xmm19
+vreduceps         $ab, (%rax){1to4}, %xmm19
+vreduceps         $ab, %xmm16, %xmm19 {k1}
+vreduceps         $ab, (%rax), %xmm19 {k1}
+vreduceps         $ab, (%rax){1to4}, %xmm19 {k1}
+vreduceps         $ab, %xmm16, %xmm19 {z}{k1}
+vreduceps         $ab, (%rax), %xmm19 {z}{k1}
+vreduceps         $ab, (%rax){1to4}, %xmm19 {z}{k1}
+
+vreduceps         $ab, %ymm16, %ymm19
+vreduceps         $ab, (%rax), %ymm19
+vreduceps         $ab, (%rax){1to8}, %ymm19
+vreduceps         $ab, %ymm16, %ymm19 {k1}
+vreduceps         $ab, (%rax), %ymm19 {k1}
+vreduceps         $ab, (%rax){1to8}, %ymm19 {k1}
+vreduceps         $ab, %ymm16, %ymm19 {z}{k1}
+vreduceps         $ab, (%rax), %ymm19 {z}{k1}
+vreduceps         $ab, (%rax){1to8}, %ymm19 {z}{k1}
+
+vxorpd            %xmm16, %xmm17, %xmm19
+vxorpd            (%rax), %xmm17, %xmm19
+vxorpd            (%rax){1to2}, %xmm17, %xmm19
+vxorpd            %xmm16, %xmm17, %xmm19 {k1}
+vxorpd            (%rax), %xmm17, %xmm19 {k1}
+vxorpd            (%rax){1to2}, %xmm17, %xmm19 {k1}
+vxorpd            %xmm16, %xmm17, %xmm19 {z}{k1}
+vxorpd            (%rax), %xmm17, %xmm19 {z}{k1}
+vxorpd            (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+vxorpd            %ymm16, %ymm17, %ymm19
+vxorpd            (%rax), %ymm17, %ymm19
+vxorpd            (%rax){1to4}, %ymm17, %ymm19
+vxorpd            %ymm16, %ymm17, %ymm19 {k1}
+vxorpd            (%rax), %ymm17, %ymm19 {k1}
+vxorpd            (%rax){1to4}, %ymm17, %ymm19 {k1}
+vxorpd            %ymm16, %ymm17, %ymm19 {z}{k1}
+vxorpd            (%rax), %ymm17, %ymm19 {z}{k1}
+vxorpd            (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vxorps            %xmm16, %xmm17, %xmm19
+vxorps            (%rax), %xmm17, %xmm19
+vxorps            (%rax){1to4}, %xmm17, %xmm19
+vxorps            %xmm16, %xmm17, %xmm19 {k1}
+vxorps            (%rax), %xmm17, %xmm19 {k1}
+vxorps            (%rax){1to4}, %xmm17, %xmm19 {k1}
+vxorps            %xmm16, %xmm17, %xmm19 {z}{k1}
+vxorps            (%rax), %xmm17, %xmm19 {z}{k1}
+vxorps            (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vxorps            %ymm16, %ymm17, %ymm19
+vxorps            (%rax), %ymm17, %ymm19
+vxorps            (%rax){1to8}, %ymm17, %ymm19
+vxorps            %ymm16, %ymm17, %ymm19 {k1}
+vxorps            (%rax), %ymm17, %ymm19 {k1}
+vxorps            (%rax){1to8}, %ymm17, %ymm19 {k1}
+vxorps            %ymm16, %ymm17, %ymm19 {z}{k1}
+vxorps            (%rax), %ymm17, %ymm19 {z}{k1}
+vxorps            (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     0.25                        vandnpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vandnpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vandnpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vandnpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vandnpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vandnpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandnpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vandnps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vandnps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vandnps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vandnps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vandnps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vandnps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandnps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vandpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vandpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vandpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vandpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vandpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vandpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vandps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vandps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vandps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vandps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vandps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vandps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vandps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vbroadcastf32x2	%xmm16, %ymm19
+# CHECK-NEXT:  1      8     1.00    *                   vbroadcastf32x2	(%rax), %ymm19
+# CHECK-NEXT:  1      1     1.00                        vbroadcastf32x2	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vbroadcastf32x2	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vbroadcastf32x2	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vbroadcastf32x2	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf64x2	(%rax), %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf64x2	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf64x2	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vbroadcasti32x2	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti32x2	(%rax), %xmm19
+# CHECK-NEXT:  1      1     0.50                        vbroadcasti32x2	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti32x2	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vbroadcasti32x2	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti32x2	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vbroadcasti32x2	%xmm16, %ymm19
+# CHECK-NEXT:  1      8     1.00    *                   vbroadcasti32x2	(%rax), %ymm19
+# CHECK-NEXT:  1      1     1.00                        vbroadcasti32x2	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vbroadcasti32x2	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vbroadcasti32x2	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vbroadcasti32x2	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti64x2	(%rax), %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti64x2	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti64x2	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vcvtpd2qq	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2qq	(%rax), %xmm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2qq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     1.00                        vcvtpd2qq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2qq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2qq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vcvtpd2qq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2qq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2qq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      3     1.00                        vcvtpd2qq	%ymm16, %ymm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2qq	(%rax), %ymm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2qq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  2      3     1.00                        vcvtpd2qq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2qq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2qq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  2      3     1.00                        vcvtpd2qq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2qq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2qq	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vcvtpd2uqq	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2uqq	(%rax), %xmm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2uqq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     1.00                        vcvtpd2uqq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2uqq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2uqq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vcvtpd2uqq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2uqq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2uqq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      3     1.00                        vcvtpd2uqq	%ymm16, %ymm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2uqq	(%rax), %ymm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2uqq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  2      3     1.00                        vcvtpd2uqq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2uqq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2uqq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  2      3     1.00                        vcvtpd2uqq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2uqq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2uqq	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vcvtps2qq	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2qq	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2qq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vcvtps2qq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2qq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2qq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvtps2qq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2qq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2qq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     0.50                        vcvtps2qq	%xmm16, %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2qq	(%rax), %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2qq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  1      4     0.50                        vcvtps2qq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2qq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2qq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     0.50                        vcvtps2qq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2qq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2qq	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vcvtps2uqq	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2uqq	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2uqq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vcvtps2uqq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2uqq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2uqq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvtps2uqq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2uqq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2uqq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     0.50                        vcvtps2uqq	%xmm16, %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2uqq	(%rax), %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2uqq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  1      4     0.50                        vcvtps2uqq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2uqq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2uqq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     0.50                        vcvtps2uqq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2uqq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2uqq	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vcvtqq2pd	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtqq2pd	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtqq2pd	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vcvtqq2pd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtqq2pd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtqq2pd	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvtqq2pd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtqq2pd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtqq2pd	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      3     1.00                        vcvtqq2pd	%ymm16, %ymm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvtqq2pd	(%rax), %ymm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvtqq2pd	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  2      3     1.00                        vcvtqq2pd	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtqq2pd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtqq2pd	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  2      3     1.00                        vcvtqq2pd	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtqq2pd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtqq2pd	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vcvtqq2ps	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtqq2psx	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtqq2ps	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vcvtqq2ps	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtqq2psx	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtqq2ps	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvtqq2ps	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtqq2psx	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtqq2ps	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vcvtqq2ps	%ymm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtqq2psx	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtqq2ps	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vcvtqq2ps	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtqq2psx	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtqq2ps	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvtqq2ps	%ymm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtqq2psx	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtqq2ps	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vcvttpd2qq	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2qq	(%rax), %xmm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2qq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     1.00                        vcvttpd2qq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2qq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2qq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vcvttpd2qq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2qq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2qq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      3     1.00                        vcvttpd2qq	%ymm16, %ymm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2qq	(%rax), %ymm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2qq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  2      3     1.00                        vcvttpd2qq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2qq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2qq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  2      3     1.00                        vcvttpd2qq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2qq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2qq	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vcvttpd2uqq	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2uqq	(%rax), %xmm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2uqq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     1.00                        vcvttpd2uqq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2uqq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2uqq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vcvttpd2uqq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2uqq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2uqq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      3     1.00                        vcvttpd2uqq	%ymm16, %ymm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2uqq	(%rax), %ymm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2uqq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  2      3     1.00                        vcvttpd2uqq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2uqq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2uqq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  2      3     1.00                        vcvttpd2uqq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2uqq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2uqq	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vcvttps2qq	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2qq	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2qq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vcvttps2qq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2qq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2qq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvttps2qq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2qq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2qq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     0.50                        vcvttps2qq	%xmm16, %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2qq	(%rax), %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2qq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  1      4     0.50                        vcvttps2qq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2qq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2qq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     0.50                        vcvttps2qq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2qq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2qq	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vcvttps2uqq	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2uqq	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2uqq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vcvttps2uqq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2uqq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2uqq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvttps2uqq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2uqq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2uqq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     0.50                        vcvttps2uqq	%xmm16, %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2uqq	(%rax), %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2uqq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  1      4     0.50                        vcvttps2uqq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2uqq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2uqq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     0.50                        vcvttps2uqq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2uqq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2uqq	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vcvtuqq2pd	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtuqq2pd	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtuqq2pd	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vcvtuqq2pd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtuqq2pd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtuqq2pd	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvtuqq2pd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtuqq2pd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtuqq2pd	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      3     1.00                        vcvtuqq2pd	%ymm16, %ymm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvtuqq2pd	(%rax), %ymm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvtuqq2pd	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  2      3     1.00                        vcvtuqq2pd	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtuqq2pd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtuqq2pd	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  2      3     1.00                        vcvtuqq2pd	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtuqq2pd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtuqq2pd	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vcvtuqq2ps	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtuqq2psx	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtuqq2ps	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vcvtuqq2ps	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtuqq2psx	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtuqq2ps	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvtuqq2ps	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtuqq2psx	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtuqq2ps	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vcvtuqq2ps	%ymm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtuqq2psx	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtuqq2ps	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vcvtuqq2ps	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtuqq2psx	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtuqq2ps	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvtuqq2ps	%ymm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtuqq2psx	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtuqq2ps	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vextractf64x2	$1, %ymm16, %xmm19
+# CHECK-NEXT:  1      1     1.00                        vextractf64x2	$1, %ymm16, (%rax)
+# CHECK-NEXT:  1      2     1.00                        vextractf64x2	$1, %ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vextractf64x2	$1, %ymm16, (%rax) {%k1}
+# CHECK-NEXT:  1      2     1.00                        vextractf64x2	$1, %ymm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vextracti64x2	$1, %ymm16, %xmm19
+# CHECK-NEXT:  1      1     1.00                        vextracti64x2	$1, %ymm16, (%rax)
+# CHECK-NEXT:  1      1     1.00                        vextracti64x2	$1, %ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vextracti64x2	$1, %ymm16, (%rax) {%k1}
+# CHECK-NEXT:  1      1     1.00                        vextracti64x2	$1, %ymm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vfpclasspd	$171, %xmm16, %k1
+# CHECK-NEXT:  1      8     0.50    *                   vfpclasspdx	$171, (%rax), %k1
+# CHECK-NEXT:  1      8     0.50    *                   vfpclasspd	$171, (%rax){1to2}, %k1
+# CHECK-NEXT:  1      1     0.50                        vfpclasspd	$171, %xmm16, %k1 {%k2}
+# CHECK-NEXT:  1      8     0.50    *                   vfpclasspdx	$171, (%rax), %k1 {%k2}
+# CHECK-NEXT:  1      8     0.50    *                   vfpclasspd	$171, (%rax){1to2}, %k1 {%k2}
+# CHECK-NEXT:  1      2     1.00                        vfpclasspd	$171, %ymm16, %k1
+# CHECK-NEXT:  1      9     1.00    *                   vfpclasspdy	$171, (%rax), %k1
+# CHECK-NEXT:  1      9     1.00    *                   vfpclasspd	$171, (%rax){1to4}, %k1
+# CHECK-NEXT:  1      2     1.00                        vfpclasspd	$171, %ymm16, %k1 {%k2}
+# CHECK-NEXT:  1      9     1.00    *                   vfpclasspdy	$171, (%rax), %k1 {%k2}
+# CHECK-NEXT:  1      9     1.00    *                   vfpclasspd	$171, (%rax){1to4}, %k1 {%k2}
+# CHECK-NEXT:  1      1     0.50                        vfpclassps	$171, %xmm16, %k1
+# CHECK-NEXT:  1      8     0.50    *                   vfpclasspsx	$171, (%rax), %k1
+# CHECK-NEXT:  1      8     0.50    *                   vfpclassps	$171, (%rax){1to4}, %k1
+# CHECK-NEXT:  1      1     0.50                        vfpclassps	$171, %xmm16, %k1 {%k2}
+# CHECK-NEXT:  1      8     0.50    *                   vfpclasspsx	$171, (%rax), %k1 {%k2}
+# CHECK-NEXT:  1      8     0.50    *                   vfpclassps	$171, (%rax){1to4}, %k1 {%k2}
+# CHECK-NEXT:  1      2     1.00                        vfpclassps	$171, %ymm16, %k1
+# CHECK-NEXT:  1      9     1.00    *                   vfpclasspsy	$171, (%rax), %k1
+# CHECK-NEXT:  1      9     1.00    *                   vfpclassps	$171, (%rax){1to8}, %k1
+# CHECK-NEXT:  1      2     1.00                        vfpclassps	$171, %ymm16, %k1 {%k2}
+# CHECK-NEXT:  1      9     1.00    *                   vfpclasspsy	$171, (%rax), %k1 {%k2}
+# CHECK-NEXT:  1      9     1.00    *                   vfpclassps	$171, (%rax){1to8}, %k1 {%k2}
+# CHECK-NEXT:  1      2     1.00                        vinsertf64x2	$1, %xmm16, %ymm16, %ymm19
+# CHECK-NEXT:  3      9     1.00    *                   vinsertf64x2	$1, (%rax), %ymm16, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vinsertf64x2	$1, %xmm16, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vinsertf64x2	$1, (%rax), %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vinsertf64x2	$1, %xmm16, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vinsertf64x2	$1, (%rax), %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vinserti64x2	$1, %xmm16, %ymm16, %ymm19
+# CHECK-NEXT:  1      8     1.00    *                   vinserti64x2	$1, (%rax), %ymm16, %ymm19
+# CHECK-NEXT:  1      1     1.00                        vinserti64x2	$1, %xmm16, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vinserti64x2	$1, (%rax), %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vinserti64x2	$1, %xmm16, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vinserti64x2	$1, (%rax), %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vpmovm2d	%k0, %xmm0
+# CHECK-NEXT:  1      0     0.17                        vpmovm2q	%k0, %xmm0
+# CHECK-NEXT:  1      0     0.17                        vpmovm2d	%k0, %ymm0
+# CHECK-NEXT:  1      0     0.17                        vpmovm2q	%k0, %ymm0
+# CHECK-NEXT:  1      1     1.00                        vpmovd2m	%xmm0, %k0
+# CHECK-NEXT:  1      1     1.00                        vpmovq2m	%xmm0, %k0
+# CHECK-NEXT:  1      1     1.00                        vpmovd2m	%ymm0, %k0
+# CHECK-NEXT:  1      1     1.00                        vpmovq2m	%ymm0, %k0
+# CHECK-NEXT:  1      1     0.25                        vorpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vorpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vorpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vorpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vorpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vorpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vorpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vorps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vorps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vorps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vorps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vorps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vorps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vorps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpmullq	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmullq	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vpmullq	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmullq	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vpmullq	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmullq	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpmullq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmullq	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vpmullq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmullq	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vpmullq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmullq	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vrangepd	$ab, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vrangepd	$ab, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vrangepd	$ab, (%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vrangepd	$ab, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vrangepd	$ab, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vrangepd	$ab, (%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vrangepd	$ab, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vrangepd	$ab, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vrangepd	$ab, (%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vrangepd	$ab, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vrangepd	$ab, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vrangepd	$ab, (%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vrangepd	$ab, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vrangepd	$ab, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vrangepd	$ab, (%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vrangepd	$ab, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vrangepd	$ab, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vrangepd	$ab, (%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vrangeps	$ab, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vrangeps	$ab, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vrangeps	$ab, (%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vrangeps	$ab, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vrangeps	$ab, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vrangeps	$ab, (%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vrangeps	$ab, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vrangeps	$ab, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vrangeps	$ab, (%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vrangeps	$ab, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vrangeps	$ab, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vrangeps	$ab, (%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vrangeps	$ab, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vrangeps	$ab, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vrangeps	$ab, (%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vrangeps	$ab, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vrangeps	$ab, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vrangeps	$ab, (%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      6     3.00                        vreducepd	$ab, %xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vreducepd	$ab, (%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vreducepd	$ab, (%rax){1to2}, %xmm19
+# CHECK-NEXT:  2      6     3.00                        vreducepd	$ab, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vreducepd	$ab, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vreducepd	$ab, (%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  2      6     3.00                        vreducepd	$ab, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vreducepd	$ab, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vreducepd	$ab, (%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vreducepd	$ab, %ymm16, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vreducepd	$ab, (%rax), %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vreducepd	$ab, (%rax){1to4}, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vreducepd	$ab, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vreducepd	$ab, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vreducepd	$ab, (%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vreducepd	$ab, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vreducepd	$ab, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vreducepd	$ab, (%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      6     3.00                        vreduceps	$ab, %xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vreduceps	$ab, (%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vreduceps	$ab, (%rax){1to4}, %xmm19
+# CHECK-NEXT:  2      6     3.00                        vreduceps	$ab, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vreduceps	$ab, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vreduceps	$ab, (%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  2      6     3.00                        vreduceps	$ab, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vreduceps	$ab, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vreduceps	$ab, (%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vreduceps	$ab, %ymm16, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vreduceps	$ab, (%rax), %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vreduceps	$ab, (%rax){1to8}, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vreduceps	$ab, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vreduceps	$ab, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vreduceps	$ab, (%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vreduceps	$ab, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vreduceps	$ab, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vreduceps	$ab, (%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vxorpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vxorpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vxorpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vxorpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vxorpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vxorpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vxorpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vxorps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vxorps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vxorps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vxorps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vxorps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vxorps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vxorps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -     4.00   4.00   4.00   4.00    -     72.00  102.00 240.00 240.00 167.50 167.50 4.00   111.67 111.67 111.67 110.33 110.33 110.33 2.00   2.00
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandnpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandnpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandnpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandnpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandnpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandnpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandnps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandnps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandnps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandnps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandnps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandnps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandnps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vandps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vandps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastf32x2	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf32x2	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastf32x2	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf32x2	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastf32x2	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf32x2	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf64x2	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf64x2	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf64x2	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vbroadcasti32x2	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti32x2	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vbroadcasti32x2	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti32x2	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vbroadcasti32x2	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti32x2	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcasti32x2	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti32x2	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcasti32x2	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti32x2	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcasti32x2	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti32x2	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti64x2	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti64x2	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti64x2	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2qq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2qq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2qq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2qq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2qq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2qq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2qq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2qq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2qq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2qq	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2qq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2qq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2qq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2qq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2qq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2qq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2qq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2qq	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2uqq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2uqq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2uqq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2uqq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2uqq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2uqq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2uqq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2uqq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2uqq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2uqq	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2uqq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2uqq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2uqq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2uqq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2uqq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2uqq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2uqq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2uqq	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2qq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2qq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2qq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2qq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2qq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2qq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2qq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2qq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2qq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2qq	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2qq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2qq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2qq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2qq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2qq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2qq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2qq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2qq	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2uqq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2uqq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2uqq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2uqq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2uqq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2uqq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2uqq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2uqq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2uqq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2uqq	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2uqq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2uqq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2uqq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2uqq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2uqq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2uqq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2uqq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2uqq	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtqq2pd	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2pd	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2pd	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtqq2pd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2pd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2pd	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtqq2pd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2pd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2pd	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtqq2pd	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2pd	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2pd	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtqq2pd	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2pd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2pd	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtqq2pd	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2pd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2pd	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtqq2ps	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2psx	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2ps	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtqq2ps	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2psx	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2ps	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtqq2ps	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2psx	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2ps	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtqq2ps	%ymm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2psx	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2ps	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtqq2ps	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2psx	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2ps	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtqq2ps	%ymm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2psx	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtqq2ps	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2qq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2qq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2qq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2qq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2qq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2qq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2qq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2qq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2qq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2qq	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2qq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2qq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2qq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2qq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2qq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2qq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2qq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2qq	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2uqq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2uqq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2uqq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2uqq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2uqq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2uqq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2uqq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2uqq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2uqq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2uqq	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2uqq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2uqq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2uqq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2uqq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2uqq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2uqq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2uqq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2uqq	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2qq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2qq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2qq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2qq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2qq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2qq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2qq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2qq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2qq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2qq	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2qq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2qq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2qq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2qq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2qq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2qq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2qq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2qq	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2uqq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2uqq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2uqq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2uqq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2uqq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2uqq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2uqq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2uqq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2uqq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2uqq	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2uqq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2uqq	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2uqq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2uqq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2uqq	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2uqq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2uqq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2uqq	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtuqq2pd	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2pd	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2pd	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtuqq2pd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2pd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2pd	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtuqq2pd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2pd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2pd	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtuqq2pd	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2pd	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2pd	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtuqq2pd	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2pd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2pd	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtuqq2pd	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2pd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2pd	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtuqq2ps	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2psx	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2ps	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtuqq2ps	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2psx	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2ps	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtuqq2ps	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2psx	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2ps	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtuqq2ps	%ymm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2psx	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2ps	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtuqq2ps	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2psx	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2ps	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtuqq2ps	%ymm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2psx	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtuqq2ps	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vextractf64x2	$1, %ymm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vextractf64x2	$1, %ymm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vextractf64x2	$1, %ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vextractf64x2	$1, %ymm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vextractf64x2	$1, %ymm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vextracti64x2	$1, %ymm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vextracti64x2	$1, %ymm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vextracti64x2	$1, %ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vextracti64x2	$1, %ymm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vextracti64x2	$1, %ymm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vfpclasspd	$171, %xmm16, %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasspdx	$171, (%rax), %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasspd	$171, (%rax){1to2}, %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vfpclasspd	$171, %xmm16, %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasspdx	$171, (%rax), %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasspd	$171, (%rax){1to2}, %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfpclasspd	$171, %ymm16, %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasspdy	$171, (%rax), %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasspd	$171, (%rax){1to4}, %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfpclasspd	$171, %ymm16, %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasspdy	$171, (%rax), %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasspd	$171, (%rax){1to4}, %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vfpclassps	$171, %xmm16, %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasspsx	$171, (%rax), %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclassps	$171, (%rax){1to4}, %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vfpclassps	$171, %xmm16, %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasspsx	$171, (%rax), %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclassps	$171, (%rax){1to4}, %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfpclassps	$171, %ymm16, %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasspsy	$171, (%rax), %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclassps	$171, (%rax){1to8}, %k1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfpclassps	$171, %ymm16, %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclasspsy	$171, (%rax), %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfpclassps	$171, (%rax){1to8}, %k1 {%k2}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinsertf64x2	$1, %xmm16, %ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinsertf64x2	$1, (%rax), %ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinsertf64x2	$1, %xmm16, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinsertf64x2	$1, (%rax), %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinsertf64x2	$1, %xmm16, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinsertf64x2	$1, (%rax), %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinserti64x2	$1, %xmm16, %ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinserti64x2	$1, (%rax), %ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinserti64x2	$1, %xmm16, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinserti64x2	$1, (%rax), %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vinserti64x2	$1, %xmm16, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vinserti64x2	$1, (%rax), %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovm2d	%k0, %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovm2q	%k0, %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovm2d	%k0, %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovm2q	%k0, %ymm0
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovd2m	%xmm0, %k0
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovq2m	%xmm0, %k0
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovd2m	%ymm0, %k0
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpmovq2m	%ymm0, %k0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vorpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vorpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vorpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vorpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vorpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vorpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vorps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vorps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vorps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vorps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vorps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vorps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vorps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmullq	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullq	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmullq	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullq	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmullq	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullq	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmullq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullq	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmullq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullq	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmullq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmullq	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangepd	$ab, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangepd	$ab, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangepd	$ab, (%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangepd	$ab, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangepd	$ab, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangepd	$ab, (%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangepd	$ab, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangepd	$ab, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangepd	$ab, (%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangepd	$ab, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangepd	$ab, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangepd	$ab, (%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangepd	$ab, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangepd	$ab, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangepd	$ab, (%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangepd	$ab, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangepd	$ab, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangepd	$ab, (%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangeps	$ab, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangeps	$ab, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangeps	$ab, (%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangeps	$ab, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangeps	$ab, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangeps	$ab, (%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangeps	$ab, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangeps	$ab, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangeps	$ab, (%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangeps	$ab, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangeps	$ab, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangeps	$ab, (%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangeps	$ab, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangeps	$ab, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangeps	$ab, (%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vrangeps	$ab, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangeps	$ab, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vrangeps	$ab, (%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducepd	$ab, %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducepd	$ab, (%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducepd	$ab, (%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducepd	$ab, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducepd	$ab, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducepd	$ab, (%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreducepd	$ab, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducepd	$ab, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducepd	$ab, (%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vreducepd	$ab, %ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducepd	$ab, (%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducepd	$ab, (%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vreducepd	$ab, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducepd	$ab, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducepd	$ab, (%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vreducepd	$ab, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducepd	$ab, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreducepd	$ab, (%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreduceps	$ab, %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreduceps	$ab, (%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreduceps	$ab, (%rax){1to4}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreduceps	$ab, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreduceps	$ab, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreduceps	$ab, (%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     vreduceps	$ab, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreduceps	$ab, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreduceps	$ab, (%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vreduceps	$ab, %ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreduceps	$ab, (%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreduceps	$ab, (%rax){1to8}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vreduceps	$ab, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreduceps	$ab, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreduceps	$ab, (%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vreduceps	$ab, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreduceps	$ab, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vreduceps	$ab, (%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vxorpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vxorpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vxorpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vxorpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vxorpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vxorpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vxorps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vxorps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vxorps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vxorps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vxorps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vxorps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vxorps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512gfni.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512gfni.s
new file mode 100644
index 0000000000000..501713d373cb8
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512gfni.s
@@ -0,0 +1,119 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vgf2p8affineinvqb   $0, %zmm16, %zmm17, %zmm19
+vgf2p8affineinvqb   $0, (%rax), %zmm17, %zmm19
+vgf2p8affineinvqb   $0, (%rax){1to8}, %zmm17, %zmm19
+vgf2p8affineinvqb   $0, %zmm16, %zmm17, %zmm19 {k1}
+vgf2p8affineinvqb   $0, (%rax), %zmm17, %zmm19 {k1}
+vgf2p8affineinvqb   $0, (%rax){1to8}, %zmm17, %zmm19 {k1}
+vgf2p8affineinvqb   $0, %zmm16, %zmm17, %zmm19 {z}{k1}
+vgf2p8affineinvqb   $0, (%rax), %zmm17, %zmm19 {z}{k1}
+vgf2p8affineinvqb   $0, (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vgf2p8affineqb      $0, %zmm16, %zmm17, %zmm19
+vgf2p8affineqb      $0, (%rax), %zmm17, %zmm19
+vgf2p8affineqb      $0, (%rax){1to8}, %zmm17, %zmm19
+vgf2p8affineqb      $0, %zmm16, %zmm17, %zmm19
+vgf2p8affineqb      $0, (%rax), %zmm17, %zmm19
+vgf2p8affineqb      $0, (%rax){1to8}, %zmm17, %zmm19
+vgf2p8affineqb      $0, %zmm16, %zmm17, %zmm19 {z}{k1}
+vgf2p8affineqb      $0, (%rax), %zmm17, %zmm19 {z}{k1}
+vgf2p8affineqb      $0, (%rax){1to8}, %zmm17, %zmm19 {z}{k1}
+
+vgf2p8mulb          %zmm16, %zmm17, %zmm19
+vgf2p8mulb          (%rax), %zmm17, %zmm19
+vgf2p8mulb          %zmm16, %zmm17, %zmm19 {k1}
+vgf2p8mulb          (%rax), %zmm17, %zmm19 {k1}
+vgf2p8mulb          %zmm16, %zmm17, %zmm19 {z}{k1}
+vgf2p8mulb          (%rax), %zmm17, %zmm19 {z}{k1}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      3     1.00                        vgf2p8affineinvqb	$0, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vgf2p8affineinvqb	$0, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vgf2p8affineinvqb	$0, (%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      3     1.00                        vgf2p8affineinvqb	$0, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vgf2p8affineinvqb	$0, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vgf2p8affineinvqb	$0, (%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vgf2p8affineinvqb	$0, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vgf2p8affineinvqb	$0, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vgf2p8affineinvqb	$0, (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vgf2p8affineqb	$0, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vgf2p8affineqb	$0, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vgf2p8affineqb	$0, (%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      3     1.00                        vgf2p8affineqb	$0, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vgf2p8affineqb	$0, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vgf2p8affineqb	$0, (%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      3     1.00                        vgf2p8affineqb	$0, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vgf2p8affineqb	$0, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vgf2p8affineqb	$0, (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vgf2p8mulb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vgf2p8mulb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vgf2p8mulb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vgf2p8mulb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vgf2p8mulb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vgf2p8mulb	(%rax), %zmm17, %zmm19 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     21.00  3.00   3.00   21.00  7.50   7.50    -     5.00   5.00   5.00   5.00   5.00   5.00    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineinvqb	$0, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineinvqb	$0, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineinvqb	$0, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineqb	$0, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineqb	$0, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineqb	$0, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8mulb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8mulb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8mulb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8mulb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8mulb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8mulb	(%rax), %zmm17, %zmm19 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512gfnivl.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512gfnivl.s
new file mode 100644
index 0000000000000..47dc617796034
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512gfnivl.s
@@ -0,0 +1,194 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vgf2p8affineinvqb   $0, %xmm16, %xmm17, %xmm19
+vgf2p8affineinvqb   $0, (%rax), %xmm17, %xmm19
+vgf2p8affineinvqb   $0, (%rax){1to2}, %xmm17, %xmm19
+vgf2p8affineinvqb   $0, %xmm16, %xmm17, %xmm19 {k1}
+vgf2p8affineinvqb   $0, (%rax), %xmm17, %xmm19 {k1}
+vgf2p8affineinvqb   $0, (%rax){1to2}, %xmm17, %xmm19 {k1}
+vgf2p8affineinvqb   $0, %xmm16, %xmm17, %xmm19 {z}{k1}
+vgf2p8affineinvqb   $0, (%rax), %xmm17, %xmm19 {z}{k1}
+vgf2p8affineinvqb   $0, (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+vgf2p8affineinvqb   $0, %ymm16, %ymm17, %ymm19
+vgf2p8affineinvqb   $0, (%rax), %ymm17, %ymm19
+vgf2p8affineinvqb   $0, (%rax){1to4}, %ymm17, %ymm19
+vgf2p8affineinvqb   $0, %ymm16, %ymm17, %ymm19 {k1}
+vgf2p8affineinvqb   $0, (%rax), %ymm17, %ymm19 {k1}
+vgf2p8affineinvqb   $0, (%rax){1to4}, %ymm17, %ymm19 {k1}
+vgf2p8affineinvqb   $0, %ymm16, %ymm17, %ymm19 {z}{k1}
+vgf2p8affineinvqb   $0, (%rax), %ymm17, %ymm19 {z}{k1}
+vgf2p8affineinvqb   $0, (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vgf2p8affineqb      $0, %xmm16, %xmm17, %xmm19
+vgf2p8affineqb      $0, (%rax), %xmm17, %xmm19
+vgf2p8affineqb      $0, (%rax){1to2}, %xmm17, %xmm19
+vgf2p8affineqb      $0, %xmm16, %xmm17, %xmm19
+vgf2p8affineqb      $0, (%rax), %xmm17, %xmm19
+vgf2p8affineqb      $0, (%rax){1to2}, %xmm17, %xmm19
+vgf2p8affineqb      $0, %xmm16, %xmm17, %xmm19 {z}{k1}
+vgf2p8affineqb      $0, (%rax), %xmm17, %xmm19 {z}{k1}
+vgf2p8affineqb      $0, (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+vgf2p8affineqb      $0, %ymm16, %ymm17, %ymm19
+vgf2p8affineqb      $0, (%rax), %ymm17, %ymm19
+vgf2p8affineqb      $0, (%rax){1to4}, %ymm17, %ymm19
+vgf2p8affineqb      $0, %ymm16, %ymm17, %ymm19
+vgf2p8affineqb      $0, (%rax), %ymm17, %ymm19
+vgf2p8affineqb      $0, (%rax){1to4}, %ymm17, %ymm19
+vgf2p8affineqb      $0, %ymm16, %ymm17, %ymm19 {z}{k1}
+vgf2p8affineqb      $0, (%rax), %ymm17, %ymm19 {z}{k1}
+vgf2p8affineqb      $0, (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vgf2p8mulb          %xmm16, %xmm17, %xmm19
+vgf2p8mulb          (%rax), %xmm17, %xmm19
+vgf2p8mulb          %xmm16, %xmm17, %xmm19 {k1}
+vgf2p8mulb          (%rax), %xmm17, %xmm19 {k1}
+vgf2p8mulb          %xmm16, %xmm17, %xmm19 {z}{k1}
+vgf2p8mulb          (%rax), %xmm17, %xmm19 {z}{k1}
+
+vgf2p8mulb          %ymm16, %ymm17, %ymm19
+vgf2p8mulb          (%rax), %ymm17, %ymm19
+vgf2p8mulb          %ymm16, %ymm17, %ymm19 {k1}
+vgf2p8mulb          (%rax), %ymm17, %ymm19 {k1}
+vgf2p8mulb          %ymm16, %ymm17, %ymm19 {z}{k1}
+vgf2p8mulb          (%rax), %ymm17, %ymm19 {z}{k1}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      3     0.50                        vgf2p8affineinvqb	$0, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineinvqb	$0, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineinvqb	$0, (%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vgf2p8affineinvqb	$0, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineinvqb	$0, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineinvqb	$0, (%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vgf2p8affineinvqb	$0, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineinvqb	$0, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineinvqb	$0, (%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vgf2p8affineinvqb	$0, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineinvqb	$0, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineinvqb	$0, (%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vgf2p8affineinvqb	$0, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineinvqb	$0, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineinvqb	$0, (%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vgf2p8affineinvqb	$0, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineinvqb	$0, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineinvqb	$0, (%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vgf2p8affineqb	$0, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineqb	$0, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineqb	$0, (%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vgf2p8affineqb	$0, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineqb	$0, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineqb	$0, (%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vgf2p8affineqb	$0, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineqb	$0, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineqb	$0, (%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vgf2p8affineqb	$0, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineqb	$0, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineqb	$0, (%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vgf2p8affineqb	$0, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineqb	$0, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineqb	$0, (%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vgf2p8affineqb	$0, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineqb	$0, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineqb	$0, (%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vgf2p8mulb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vgf2p8mulb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vgf2p8mulb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vgf2p8mulb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vgf2p8mulb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vgf2p8mulb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vgf2p8mulb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vgf2p8mulb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vgf2p8mulb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vgf2p8mulb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vgf2p8mulb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vgf2p8mulb	(%rax), %ymm17, %ymm19 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     21.00  3.00   3.00   21.00  15.00  15.00   -     10.00  10.00  10.00  10.00  10.00  10.00   -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineinvqb	$0, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineinvqb	$0, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineinvqb	$0, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineinvqb	$0, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineinvqb	$0, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineinvqb	$0, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineqb	$0, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineqb	$0, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineqb	$0, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineqb	$0, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineqb	$0, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineqb	$0, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vgf2p8mulb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8mulb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vgf2p8mulb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8mulb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vgf2p8mulb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8mulb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vgf2p8mulb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8mulb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vgf2p8mulb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8mulb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vgf2p8mulb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8mulb	(%rax), %ymm17, %ymm19 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512ifma.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512ifma.s
new file mode 100644
index 0000000000000..a71589edf84e6
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512ifma.s
@@ -0,0 +1,100 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vpmadd52huq     %zmm16, %zmm17, %zmm19
+vpmadd52huq     (%rdi), %zmm17, %zmm19
+vpmadd52huq     (%rdi){1to8}, %zmm17, %zmm19
+vpmadd52huq     %zmm16, %zmm17, %zmm19 {%k1}
+vpmadd52huq     (%rdi), %zmm17, %zmm19 {%k1}
+vpmadd52huq     (%rdi){1to8}, %zmm17, %zmm19 {%k1}
+vpmadd52huq     %zmm16, %zmm17, %zmm19 {%k1} {z}
+vpmadd52huq     (%rdi), %zmm17, %zmm19 {%k1} {z}
+vpmadd52huq     (%rdi){1to8}, %zmm17, %zmm19 {%k1} {z}
+
+vpmadd52luq     %zmm16, %zmm17, %zmm19
+vpmadd52luq     (%rdi), %zmm17, %zmm19
+vpmadd52luq     (%rdi){1to8}, %zmm17, %zmm19
+vpmadd52luq     %zmm16, %zmm17, %zmm19 {%k1}
+vpmadd52luq     (%rdi), %zmm17, %zmm19 {%k1}
+vpmadd52luq     (%rdi){1to8}, %zmm17, %zmm19 {%k1}
+vpmadd52luq     %zmm16, %zmm17, %zmm19 {%k1} {z}
+vpmadd52luq     (%rdi), %zmm17, %zmm19 {%k1} {z}
+vpmadd52luq     (%rdi){1to8}, %zmm17, %zmm19 {%k1} {z}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      4     2.00                        vpmadd52huq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpmadd52huq	(%rdi), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpmadd52huq	(%rdi){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      4     2.00                        vpmadd52huq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpmadd52huq	(%rdi), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpmadd52huq	(%rdi){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmadd52huq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpmadd52huq	(%rdi), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpmadd52huq	(%rdi){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpmadd52luq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpmadd52luq	(%rdi), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpmadd52luq	(%rdi){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      4     2.00                        vpmadd52luq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpmadd52luq	(%rdi), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpmadd52luq	(%rdi){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmadd52luq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpmadd52luq	(%rdi), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpmadd52luq	(%rdi){1to8}, %zmm17, %zmm19 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     24.00  12.00   -     12.00  6.00   6.00    -     4.00   4.00   4.00   4.00   4.00   4.00    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpmadd52huq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52huq	(%rdi), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52huq	(%rdi){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpmadd52huq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52huq	(%rdi), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52huq	(%rdi){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpmadd52huq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52huq	(%rdi), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52huq	(%rdi){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpmadd52luq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52luq	(%rdi), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52luq	(%rdi){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpmadd52luq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52luq	(%rdi), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52luq	(%rdi){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpmadd52luq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52luq	(%rdi), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52luq	(%rdi){1to8}, %zmm17, %zmm19 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512ifmavl.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512ifmavl.s
new file mode 100644
index 0000000000000..9b7393d74ba0e
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512ifmavl.s
@@ -0,0 +1,156 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vpmadd52huq     %xmm16, %xmm17, %xmm19
+vpmadd52huq     (%rdi), %xmm17, %xmm19
+vpmadd52huq     (%rdi){1to2}, %xmm17, %xmm19
+vpmadd52huq     %xmm16, %xmm17, %xmm19 {%k1}
+vpmadd52huq     (%rdi), %xmm17, %xmm19 {%k1}
+vpmadd52huq     (%rdi){1to2}, %xmm17, %xmm19 {%k1}
+vpmadd52huq     %xmm16, %xmm17, %xmm19 {%k1} {z}
+vpmadd52huq     (%rdi), %xmm17, %xmm19 {%k1} {z}
+vpmadd52huq     (%rdi){1to2}, %xmm17, %xmm19 {%k1} {z}
+
+vpmadd52huq     %ymm16, %ymm17, %ymm19
+vpmadd52huq     (%rdi), %ymm17, %ymm19
+vpmadd52huq     (%rdi){1to4}, %ymm17, %ymm19
+vpmadd52huq     %ymm16, %ymm17, %ymm19 {%k1}
+vpmadd52huq     (%rdi), %ymm17, %ymm19 {%k1}
+vpmadd52huq     (%rdi){1to4}, %ymm17, %ymm19 {%k1}
+vpmadd52huq     %ymm16, %ymm17, %ymm19 {%k1} {z}
+vpmadd52huq     (%rdi), %ymm17, %ymm19 {%k1} {z}
+vpmadd52huq     (%rdi){1to4}, %ymm17, %ymm19 {%k1} {z}
+
+vpmadd52luq     %xmm16, %xmm17, %xmm19
+vpmadd52luq     (%rdi), %xmm17, %xmm19
+vpmadd52luq     (%rdi){1to2}, %xmm17, %xmm19
+vpmadd52luq     %xmm16, %xmm17, %xmm19 {%k1}
+vpmadd52luq     (%rdi), %xmm17, %xmm19 {%k1}
+vpmadd52luq     (%rdi){1to2}, %xmm17, %xmm19 {%k1}
+vpmadd52luq     %xmm16, %xmm17, %xmm19 {%k1} {z}
+vpmadd52luq     (%rdi), %xmm17, %xmm19 {%k1} {z}
+vpmadd52luq     (%rdi){1to2}, %xmm17, %xmm19 {%k1} {z}
+
+vpmadd52luq     %ymm16, %ymm17, %ymm19
+vpmadd52luq     (%rdi), %ymm17, %ymm19
+vpmadd52luq     (%rdi){1to4}, %ymm17, %ymm19
+vpmadd52luq     %ymm16, %ymm17, %ymm19 {%k1}
+vpmadd52luq     (%rdi), %ymm17, %ymm19 {%k1}
+vpmadd52luq     (%rdi){1to4}, %ymm17, %ymm19 {%k1}
+vpmadd52luq     %ymm16, %ymm17, %ymm19 {%k1} {z}
+vpmadd52luq     (%rdi), %ymm17, %ymm19 {%k1} {z}
+vpmadd52luq     (%rdi){1to4}, %ymm17, %ymm19 {%k1} {z}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      4     2.00                        vpmadd52huq	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52huq	(%rdi), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52huq	(%rdi){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      4     2.00                        vpmadd52huq	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52huq	(%rdi), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52huq	(%rdi){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmadd52huq	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52huq	(%rdi), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52huq	(%rdi){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpmadd52huq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52huq	(%rdi), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52huq	(%rdi){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      4     2.00                        vpmadd52huq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52huq	(%rdi), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52huq	(%rdi){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmadd52huq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52huq	(%rdi), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52huq	(%rdi){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpmadd52luq	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52luq	(%rdi), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52luq	(%rdi){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      4     2.00                        vpmadd52luq	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52luq	(%rdi), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52luq	(%rdi){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmadd52luq	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52luq	(%rdi), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52luq	(%rdi){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpmadd52luq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52luq	(%rdi), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52luq	(%rdi){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      4     2.00                        vpmadd52luq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52luq	(%rdi), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52luq	(%rdi){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpmadd52luq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52luq	(%rdi), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmadd52luq	(%rdi){1to4}, %ymm17, %ymm19 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     36.00  24.00   -     12.00  12.00  12.00   -     8.00   8.00   8.00   8.00   8.00   8.00    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpmadd52huq	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52huq	(%rdi), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52huq	(%rdi){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpmadd52huq	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52huq	(%rdi), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52huq	(%rdi){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpmadd52huq	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52huq	(%rdi), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52huq	(%rdi){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpmadd52huq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52huq	(%rdi), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52huq	(%rdi){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpmadd52huq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52huq	(%rdi), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52huq	(%rdi){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpmadd52huq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52huq	(%rdi), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52huq	(%rdi){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpmadd52luq	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52luq	(%rdi), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52luq	(%rdi){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpmadd52luq	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52luq	(%rdi), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52luq	(%rdi){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpmadd52luq	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52luq	(%rdi), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52luq	(%rdi){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpmadd52luq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52luq	(%rdi), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52luq	(%rdi){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpmadd52luq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52luq	(%rdi), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52luq	(%rdi){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpmadd52luq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52luq	(%rdi), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmadd52luq	(%rdi){1to4}, %ymm17, %ymm19 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vaes.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vaes.s
new file mode 100644
index 0000000000000..462233c88190c
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vaes.s
@@ -0,0 +1,72 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vaesdec          %zmm16, %zmm17, %zmm19
+vaesdec          (%rax), %zmm17, %zmm19
+
+vaesdeclast      %zmm16, %zmm17, %zmm19
+vaesdeclast      (%rax), %zmm17, %zmm19
+
+vaesenc          %zmm16, %zmm17, %zmm19
+vaesenc          (%rax), %zmm17, %zmm19
+
+vaesenclast      %zmm16, %zmm17, %zmm19
+vaesenclast      (%rax), %zmm17, %zmm19
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      4     0.50                        vaesdec	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      11    0.50    *                   vaesdec	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      4     0.50                        vaesdeclast	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      11    0.50    *                   vaesdeclast	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      4     0.50                        vaesenc	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      11    0.50    *                   vaesenc	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      4     0.50                        vaesenclast	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      11    0.50    *                   vaesenclast	(%rax), %zmm17, %zmm19
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     4.00   4.00    -      -     2.00   2.00    -     1.33   1.33   1.33   1.33   1.33   1.33    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesdec	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesdec	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesdeclast	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesdeclast	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesenc	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesenc	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesenclast	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesenclast	(%rax), %zmm17, %zmm19
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vaesvl.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vaesvl.s
new file mode 100644
index 0000000000000..59fceb8ee0894
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vaesvl.s
@@ -0,0 +1,100 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vaesdec          %xmm16, %xmm17, %xmm19
+vaesdec          (%rax), %xmm17, %xmm19
+
+vaesdec          %ymm16, %ymm17, %ymm19
+vaesdec          (%rax), %ymm17, %ymm19
+
+vaesdeclast      %xmm16, %xmm17, %xmm19
+vaesdeclast      (%rax), %xmm17, %xmm19
+
+vaesdeclast      %ymm16, %ymm17, %ymm19
+vaesdeclast      (%rax), %ymm17, %ymm19
+
+vaesenc          %xmm16, %xmm17, %xmm19
+vaesenc          (%rax), %xmm17, %xmm19
+
+vaesenc          %ymm16, %ymm17, %ymm19
+vaesenc          (%rax), %ymm17, %ymm19
+
+vaesenclast      %xmm16, %xmm17, %xmm19
+vaesenclast      (%rax), %xmm17, %xmm19
+
+vaesenclast      %ymm16, %ymm17, %ymm19
+vaesenclast      (%rax), %ymm17, %ymm19
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      4     0.50                        vaesdec	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      11    0.50    *                   vaesdec	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      4     0.50                        vaesdec	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vaesdec	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      4     0.50                        vaesdeclast	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      11    0.50    *                   vaesdeclast	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      4     0.50                        vaesdeclast	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vaesdeclast	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      4     0.50                        vaesenc	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      11    0.50    *                   vaesenc	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      4     0.50                        vaesenc	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vaesenc	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      4     0.50                        vaesenclast	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      11    0.50    *                   vaesenclast	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      4     0.50                        vaesenclast	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vaesenclast	(%rax), %ymm17, %ymm19
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     8.00   8.00    -      -     4.00   4.00    -     2.67   2.67   2.67   2.67   2.67   2.67    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesdec	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesdec	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesdec	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesdec	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesdeclast	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesdeclast	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesdeclast	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesdeclast	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesenc	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesenc	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesenc	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesenc	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesenclast	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesenclast	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesenclast	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesenclast	(%rax), %ymm17, %ymm19
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vbmi.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vbmi.s
new file mode 100644
index 0000000000000..845faf8e2f942
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vbmi.s
@@ -0,0 +1,129 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vpermb            %zmm16, %zmm17, %zmm19
+vpermb            (%rax), %zmm17, %zmm19
+vpermb            %zmm16, %zmm17, %zmm19 {k1}
+vpermb            (%rax), %zmm17, %zmm19 {k1}
+vpermb            %zmm16, %zmm17, %zmm19 {k1}{z}
+vpermb            (%rax), %zmm17, %zmm19 {k1}{z}
+
+vpermi2b          %zmm16, %zmm17, %zmm19
+vpermi2b          (%rax), %zmm17, %zmm19
+vpermi2b          %zmm16, %zmm17, %zmm19 {k1}
+vpermi2b          (%rax), %zmm17, %zmm19 {k1}
+vpermi2b          %zmm16, %zmm17, %zmm19 {k1}{z}
+vpermi2b          (%rax), %zmm17, %zmm19 {k1}{z}
+
+vpermt2b          %zmm16, %zmm17, %zmm19
+vpermt2b          (%rax), %zmm17, %zmm19
+vpermt2b          %zmm16, %zmm17, %zmm19 {k1}
+vpermt2b          (%rax), %zmm17, %zmm19 {k1}
+vpermt2b          %zmm16, %zmm17, %zmm19 {k1}{z}
+vpermt2b          (%rax), %zmm17, %zmm19 {k1}{z}
+
+vpmultishiftqb    %zmm16, %zmm17, %zmm19
+vpmultishiftqb    (%rax), %zmm17, %zmm19
+vpmultishiftqb    (%rax){1to8}, %zmm17, %zmm19
+vpmultishiftqb    %zmm16, %zmm17, %zmm19 {k1}
+vpmultishiftqb    (%rax), %zmm17, %zmm19 {k1}
+vpmultishiftqb    (%rax){1to8}, %zmm17, %zmm19 {k1}
+vpmultishiftqb    %zmm16, %zmm17, %zmm19 {k1}{z}
+vpmultishiftqb    (%rax), %zmm17, %zmm19 {k1}{z}
+vpmultishiftqb    (%rax){1to8}, %zmm17, %zmm19 {k1}{z}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      5     1.00                        vpermb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      5     1.00                        vpermb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      5     1.00                        vpermb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      5     1.00                        vpermi2b	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermi2b	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      5     1.00                        vpermi2b	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermi2b	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      5     1.00                        vpermi2b	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermi2b	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      5     1.00                        vpermt2b	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermt2b	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      5     1.00                        vpermt2b	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermt2b	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      5     1.00                        vpermt2b	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermt2b	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpmultishiftqb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmultishiftqb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmultishiftqb	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpmultishiftqb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmultishiftqb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmultishiftqb	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpmultishiftqb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmultishiftqb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmultishiftqb	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     4.50   18.00  18.00  4.50   7.50   7.50    -     5.00   5.00   5.00   5.00   5.00   5.00    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermi2b	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermi2b	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermi2b	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermi2b	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermi2b	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermi2b	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermt2b	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermt2b	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermt2b	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermt2b	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermt2b	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermt2b	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpmultishiftqb	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmultishiftqb	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmultishiftqb	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpmultishiftqb	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmultishiftqb	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmultishiftqb	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpmultishiftqb	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmultishiftqb	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmultishiftqb	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vbmi2.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vbmi2.s
new file mode 100644
index 0000000000000..594518dcc775a
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vbmi2.s
@@ -0,0 +1,408 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vpcompressb       %zmm16, %zmm19
+vpcompressb       %zmm16, (%rax)
+vpcompressb       %zmm16, %zmm19 {k1}
+vpcompressb       %zmm16, (%rax) {k1}
+vpcompressb       %zmm16, %zmm19 {k1}{z}
+
+vpcompressw       %zmm16, %zmm19
+vpcompressw       %zmm16, (%rax)
+vpcompressw       %zmm16, %zmm19 {k1}
+vpcompressw       %zmm16, (%rax) {k1}
+vpcompressw       %zmm16, %zmm19 {k1}{z}
+
+vpexpandb         %zmm16, %zmm19
+vpexpandb         (%rax), %zmm19
+vpexpandb         %zmm16, %zmm19 {k1}
+vpexpandb         (%rax), %zmm19 {k1}
+vpexpandb         %zmm16, %zmm19 {k1}{z}
+
+vpexpandw         %zmm16, %zmm19
+vpexpandw         (%rax), %zmm19
+vpexpandw         %zmm16, %zmm19 {k1}
+vpexpandw         (%rax), %zmm19 {k1}
+vpexpandw         %zmm16, %zmm19 {k1}{z}
+
+vpshldd           $1, %zmm16, %zmm17, %zmm19
+vpshldd           $1, (%rax), %zmm17, %zmm19
+vpshldd           $1, (%rax){1to16}, %zmm17, %zmm19
+vpshldd           $1, %zmm16, %zmm17, %zmm19 {k1}
+vpshldd           $1, (%rax), %zmm17, %zmm19 {k1}
+vpshldd           $1, (%rax){1to16}, %zmm17, %zmm19 {k1}
+vpshldd           $1, %zmm16, %zmm17, %zmm19 {k1}{z}
+vpshldd           $1, (%rax), %zmm17, %zmm19 {k1}{z}
+vpshldd           $1, (%rax){1to16}, %zmm17, %zmm19 {k1}{z}
+
+vpshldq           $1, %zmm16, %zmm17, %zmm19
+vpshldq           $1, (%rax), %zmm17, %zmm19
+vpshldq           $1, (%rax){1to8}, %zmm17, %zmm19
+vpshldq           $1, %zmm16, %zmm17, %zmm19 {k1}
+vpshldq           $1, (%rax), %zmm17, %zmm19 {k1}
+vpshldq           $1, (%rax){1to8}, %zmm17, %zmm19 {k1}
+vpshldq           $1, %zmm16, %zmm17, %zmm19 {k1}{z}
+vpshldq           $1, (%rax), %zmm17, %zmm19 {k1}{z}
+vpshldq           $1, (%rax){1to8}, %zmm17, %zmm19 {k1}{z}
+
+vpshldvd          %zmm16, %zmm17, %zmm19
+vpshldvd          (%rax), %zmm17, %zmm19
+vpshldvd          (%rax){1to16}, %zmm17, %zmm19
+vpshldvd          %zmm16, %zmm17, %zmm19 {k1}
+vpshldvd          (%rax), %zmm17, %zmm19 {k1}
+vpshldvd          (%rax){1to16}, %zmm17, %zmm19 {k1}
+vpshldvd          %zmm16, %zmm17, %zmm19 {k1}{z}
+vpshldvd          (%rax), %zmm17, %zmm19 {k1}{z}
+vpshldvd          (%rax){1to16}, %zmm17, %zmm19 {k1}{z}
+
+vpshldvq          %zmm16, %zmm17, %zmm19
+vpshldvq          (%rax), %zmm17, %zmm19
+vpshldvq          (%rax){1to8}, %zmm17, %zmm19
+vpshldvq          %zmm16, %zmm17, %zmm19 {k1}
+vpshldvq          (%rax), %zmm17, %zmm19 {k1}
+vpshldvq          (%rax){1to8}, %zmm17, %zmm19 {k1}
+vpshldvq          %zmm16, %zmm17, %zmm19 {k1}{z}
+vpshldvq          (%rax), %zmm17, %zmm19 {k1}{z}
+vpshldvq          (%rax){1to8}, %zmm17, %zmm19 {k1}{z}
+
+vpshldvw          %zmm16, %zmm17, %zmm19
+vpshldvw          (%rax), %zmm17, %zmm19
+vpshldvw          %zmm16, %zmm17, %zmm19 {k1}
+vpshldvw          (%rax), %zmm17, %zmm19 {k1}
+vpshldvw          %zmm16, %zmm17, %zmm19 {k1}{z}
+vpshldvw          (%rax), %zmm17, %zmm19 {k1}{z}
+
+vpshldw           $1, %zmm16, %zmm17, %zmm19
+vpshldw           $1, (%rax), %zmm17, %zmm19
+vpshldw           $1, %zmm16, %zmm17, %zmm19 {k1}
+vpshldw           $1, (%rax), %zmm17, %zmm19 {k1}
+vpshldw           $1, %zmm16, %zmm17, %zmm19 {k1}{z}
+vpshldw           $1, (%rax), %zmm17, %zmm19 {k1}{z}
+
+vpshrdd           $1, %zmm16, %zmm17, %zmm19
+vpshrdd           $1, (%rax), %zmm17, %zmm19
+vpshrdd           $1, (%rax){1to16}, %zmm17, %zmm19
+vpshrdd           $1, %zmm16, %zmm17, %zmm19 {k1}
+vpshrdd           $1, (%rax), %zmm17, %zmm19 {k1}
+vpshrdd           $1, (%rax){1to16}, %zmm17, %zmm19 {k1}
+vpshrdd           $1, %zmm16, %zmm17, %zmm19 {k1}{z}
+vpshrdd           $1, (%rax), %zmm17, %zmm19 {k1}{z}
+vpshrdd           $1, (%rax){1to16}, %zmm17, %zmm19 {k1}{z}
+
+vpshrdq           $1, %zmm16, %zmm17, %zmm19
+vpshrdq           $1, (%rax), %zmm17, %zmm19
+vpshrdq           $1, (%rax){1to8}, %zmm17, %zmm19
+vpshrdq           $1, %zmm16, %zmm17, %zmm19 {k1}
+vpshrdq           $1, (%rax), %zmm17, %zmm19 {k1}
+vpshrdq           $1, (%rax){1to8}, %zmm17, %zmm19 {k1}
+vpshrdq           $1, %zmm16, %zmm17, %zmm19 {k1}{z}
+vpshrdq           $1, (%rax), %zmm17, %zmm19 {k1}{z}
+vpshrdq           $1, (%rax){1to8}, %zmm17, %zmm19 {k1}{z}
+
+vpshrdvd          %zmm16, %zmm17, %zmm19
+vpshrdvd          (%rax), %zmm17, %zmm19
+vpshrdvd          (%rax){1to16}, %zmm17, %zmm19
+vpshrdvd          %zmm16, %zmm17, %zmm19 {k1}
+vpshrdvd          (%rax), %zmm17, %zmm19 {k1}
+vpshrdvd          (%rax){1to16}, %zmm17, %zmm19 {k1}
+vpshrdvd          %zmm16, %zmm17, %zmm19 {k1}{z}
+vpshrdvd          (%rax), %zmm17, %zmm19 {k1}{z}
+vpshrdvd          (%rax){1to16}, %zmm17, %zmm19 {k1}{z}
+
+vpshrdvq          %zmm16, %zmm17, %zmm19
+vpshrdvq          (%rax), %zmm17, %zmm19
+vpshrdvq          (%rax){1to8}, %zmm17, %zmm19
+vpshrdvq          %zmm16, %zmm17, %zmm19 {k1}
+vpshrdvq          (%rax), %zmm17, %zmm19 {k1}
+vpshrdvq          (%rax){1to8}, %zmm17, %zmm19 {k1}
+vpshrdvq          %zmm16, %zmm17, %zmm19 {k1}{z}
+vpshrdvq          (%rax), %zmm17, %zmm19 {k1}{z}
+vpshrdvq          (%rax){1to8}, %zmm17, %zmm19 {k1}{z}
+
+vpshrdvw          %zmm16, %zmm17, %zmm19
+vpshrdvw          (%rax), %zmm17, %zmm19
+vpshrdvw          %zmm16, %zmm17, %zmm19 {k1}
+vpshrdvw          (%rax), %zmm17, %zmm19 {k1}
+vpshrdvw          %zmm16, %zmm17, %zmm19 {k1}{z}
+vpshrdvw          (%rax), %zmm17, %zmm19 {k1}{z}
+
+vpshrdw           $1, %zmm16, %zmm17, %zmm19
+vpshrdw           $1, (%rax), %zmm17, %zmm19
+vpshrdw           $1, %zmm16, %zmm17, %zmm19 {k1}
+vpshrdw           $1, (%rax), %zmm17, %zmm19 {k1}
+vpshrdw           $1, %zmm16, %zmm17, %zmm19 {k1}{z}
+vpshrdw           $1, (%rax), %zmm17, %zmm19 {k1}{z}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      5     1.00                  U     vpcompressb	%zmm16, %zmm19
+# CHECK-NEXT:  2      8     0.50           *            vpcompressb	%zmm16, (%rax)
+# CHECK-NEXT:  1      5     1.00                        vpcompressb	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50           *            vpcompressb	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      5     1.00                        vpcompressb	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      5     1.00                  U     vpcompressw	%zmm16, %zmm19
+# CHECK-NEXT:  2      8     0.50           *            vpcompressw	%zmm16, (%rax)
+# CHECK-NEXT:  1      5     1.00                        vpcompressw	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50           *            vpcompressw	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      5     1.00                        vpcompressw	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      5     1.00                  U     vpexpandb	%zmm16, %zmm19
+# CHECK-NEXT:  2      8     0.50                  U     vpexpandb	(%rax), %zmm19
+# CHECK-NEXT:  1      5     1.00                        vpexpandb	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpexpandb	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      5     1.00                        vpexpandb	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      5     1.00                  U     vpexpandw	%zmm16, %zmm19
+# CHECK-NEXT:  2      8     0.50                  U     vpexpandw	(%rax), %zmm19
+# CHECK-NEXT:  1      5     1.00                        vpexpandw	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpexpandw	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  1      5     1.00                        vpexpandw	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshldd	$1, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshldd	$1, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshldd	$1, (%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpshldd	$1, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldd	$1, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldd	$1, (%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshldd	$1, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldd	$1, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldd	$1, (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshldq	$1, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshldq	$1, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshldq	$1, (%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpshldq	$1, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldq	$1, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldq	$1, (%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshldq	$1, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldq	$1, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldq	$1, (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vpshldvd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshldvd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshldvd	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpshldvd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldvd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldvd	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshldvd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldvd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldvd	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vpshldvq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshldvq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshldvq	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpshldvq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldvq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldvq	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshldvq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldvq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldvq	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vpshldvw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshldvw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpshldvw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldvw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshldvw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldvw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshldw	$1, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshldw	$1, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpshldw	$1, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldw	$1, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshldw	$1, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshldw	$1, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshrdd	$1, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdd	$1, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdd	$1, (%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpshrdd	$1, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdd	$1, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdd	$1, (%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshrdd	$1, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdd	$1, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdd	$1, (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshrdq	$1, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdq	$1, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdq	$1, (%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpshrdq	$1, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdq	$1, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdq	$1, (%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshrdq	$1, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdq	$1, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdq	$1, (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vpshrdvd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdvd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdvd	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpshrdvd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdvd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdvd	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshrdvd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdvd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdvd	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vpshrdvq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdvq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdvq	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpshrdvq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdvq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdvq	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshrdvq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdvq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdvq	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vpshrdvw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdvw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpshrdvw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdvw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshrdvw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdvw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshrdw	$1, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdw	$1, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      2     1.00                        vpshrdw	$1, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdw	$1, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshrdw	$1, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpshrdw	$1, (%rax), %zmm17, %zmm19 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     96.00  46.00  16.00  66.00  34.00  34.00   -     22.67  22.67  22.67  22.67  22.67  22.67   -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpcompressb	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcompressb	%zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpcompressb	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcompressb	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpcompressb	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpcompressw	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcompressw	%zmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpcompressw	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcompressw	%zmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpcompressw	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpexpandb	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpexpandb	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpexpandb	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpexpandb	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpexpandb	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpexpandw	%zmm16, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpexpandw	(%rax), %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpexpandw	%zmm16, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpexpandw	(%rax), %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpexpandw	%zmm16, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldd	$1, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldd	$1, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldd	$1, (%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldd	$1, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldd	$1, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldd	$1, (%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldd	$1, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldd	$1, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldd	$1, (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldq	$1, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldq	$1, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldq	$1, (%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldq	$1, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldq	$1, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldq	$1, (%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldq	$1, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldq	$1, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldq	$1, (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpshldvd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvd	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldvd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvd	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldvd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvd	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpshldvq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvq	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldvq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvq	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldvq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvq	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpshldvw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldvw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldvw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldw	$1, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldw	$1, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldw	$1, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldw	$1, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldw	$1, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldw	$1, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdd	$1, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdd	$1, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdd	$1, (%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdd	$1, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdd	$1, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdd	$1, (%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdd	$1, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdd	$1, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdd	$1, (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdq	$1, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdq	$1, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdq	$1, (%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdq	$1, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdq	$1, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdq	$1, (%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdq	$1, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdq	$1, (%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdq	$1, (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpshrdvd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvd	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdvd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvd	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdvd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvd	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpshrdvq	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvq	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvq	(%rax){1to8}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdvq	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvq	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvq	(%rax){1to8}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdvq	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvq	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvq	(%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpshrdvw	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvw	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdvw	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvw	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdvw	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvw	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdw	$1, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdw	$1, (%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdw	$1, %zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdw	$1, (%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdw	$1, %zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdw	$1, (%rax), %zmm17, %zmm19 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vbmi2vl.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vbmi2vl.s
new file mode 100644
index 0000000000000..7b9c2516929b8
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vbmi2vl.s
@@ -0,0 +1,772 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vpcompressb       %xmm16, %xmm19
+vpcompressb       %xmm16, (%rax)
+vpcompressb       %xmm16, %xmm19 {k1}
+vpcompressb       %xmm16, (%rax) {k1}
+vpcompressb       %xmm16, %xmm19 {k1}{z}
+
+vpcompressb       %ymm16, %ymm19
+vpcompressb       %ymm16, (%rax)
+vpcompressb       %ymm16, %ymm19 {k1}
+vpcompressb       %ymm16, (%rax) {k1}
+vpcompressb       %ymm16, %ymm19 {k1}{z}
+
+vpcompressw       %xmm16, %xmm19
+vpcompressw       %xmm16, (%rax)
+vpcompressw       %xmm16, %xmm19 {k1}
+vpcompressw       %xmm16, (%rax) {k1}
+vpcompressw       %xmm16, %xmm19 {k1}{z}
+
+vpcompressw       %ymm16, %ymm19
+vpcompressw       %ymm16, (%rax)
+vpcompressw       %ymm16, %ymm19 {k1}
+vpcompressw       %ymm16, (%rax) {k1}
+vpcompressw       %ymm16, %ymm19 {k1}{z}
+
+vpexpandb         %xmm16, %xmm19
+vpexpandb         (%rax), %xmm19
+vpexpandb         %xmm16, %xmm19 {k1}
+vpexpandb         (%rax), %xmm19 {k1}
+vpexpandb         %xmm16, %xmm19 {k1}{z}
+
+vpexpandb         %ymm16, %ymm19
+vpexpandb         (%rax), %ymm19
+vpexpandb         %ymm16, %ymm19 {k1}
+vpexpandb         (%rax), %ymm19 {k1}
+vpexpandb         %ymm16, %ymm19 {k1}{z}
+
+vpexpandw         %xmm16, %xmm19
+vpexpandw         (%rax), %xmm19
+vpexpandw         %xmm16, %xmm19 {k1}
+vpexpandw         (%rax), %xmm19 {k1}
+vpexpandw         %xmm16, %xmm19 {k1}{z}
+
+vpexpandw         %ymm16, %ymm19
+vpexpandw         (%rax), %ymm19
+vpexpandw         %ymm16, %ymm19 {k1}
+vpexpandw         (%rax), %ymm19 {k1}
+vpexpandw         %ymm16, %ymm19 {k1}{z}
+
+vpshldd           $1, %xmm16, %xmm17, %xmm19
+vpshldd           $1, (%rax), %xmm17, %xmm19
+vpshldd           $1, (%rax){1to4}, %xmm17, %xmm19
+vpshldd           $1, %xmm16, %xmm17, %xmm19 {k1}
+vpshldd           $1, (%rax), %xmm17, %xmm19 {k1}
+vpshldd           $1, (%rax){1to4}, %xmm17, %xmm19 {k1}
+vpshldd           $1, %xmm16, %xmm17, %xmm19 {k1}{z}
+vpshldd           $1, (%rax), %xmm17, %xmm19 {k1}{z}
+vpshldd           $1, (%rax){1to4}, %xmm17, %xmm19 {k1}{z}
+
+vpshldd           $1, %ymm16, %ymm17, %ymm19
+vpshldd           $1, (%rax), %ymm17, %ymm19
+vpshldd           $1, (%rax){1to8}, %ymm17, %ymm19
+vpshldd           $1, %ymm16, %ymm17, %ymm19 {k1}
+vpshldd           $1, (%rax), %ymm17, %ymm19 {k1}
+vpshldd           $1, (%rax){1to8}, %ymm17, %ymm19 {k1}
+vpshldd           $1, %ymm16, %ymm17, %ymm19 {k1}{z}
+vpshldd           $1, (%rax), %ymm17, %ymm19 {k1}{z}
+vpshldd           $1, (%rax){1to8}, %ymm17, %ymm19 {k1}{z}
+
+vpshldq           $1, %xmm16, %xmm17, %xmm19
+vpshldq           $1, (%rax), %xmm17, %xmm19
+vpshldq           $1, (%rax){1to2}, %xmm17, %xmm19
+vpshldq           $1, %xmm16, %xmm17, %xmm19 {k1}
+vpshldq           $1, (%rax), %xmm17, %xmm19 {k1}
+vpshldq           $1, (%rax){1to2}, %xmm17, %xmm19 {k1}
+vpshldq           $1, %xmm16, %xmm17, %xmm19 {k1}{z}
+vpshldq           $1, (%rax), %xmm17, %xmm19 {k1}{z}
+vpshldq           $1, (%rax){1to2}, %xmm17, %xmm19 {k1}{z}
+
+vpshldq           $1, %ymm16, %ymm17, %ymm19
+vpshldq           $1, (%rax), %ymm17, %ymm19
+vpshldq           $1, (%rax){1to4}, %ymm17, %ymm19
+vpshldq           $1, %ymm16, %ymm17, %ymm19 {k1}
+vpshldq           $1, (%rax), %ymm17, %ymm19 {k1}
+vpshldq           $1, (%rax){1to4}, %ymm17, %ymm19 {k1}
+vpshldq           $1, %ymm16, %ymm17, %ymm19 {k1}{z}
+vpshldq           $1, (%rax), %ymm17, %ymm19 {k1}{z}
+vpshldq           $1, (%rax){1to4}, %ymm17, %ymm19 {k1}{z}
+
+vpshldvd          %xmm16, %xmm17, %xmm19
+vpshldvd          (%rax), %xmm17, %xmm19
+vpshldvd          (%rax){1to4}, %xmm17, %xmm19
+vpshldvd          %xmm16, %xmm17, %xmm19 {k1}
+vpshldvd          (%rax), %xmm17, %xmm19 {k1}
+vpshldvd          (%rax){1to4}, %xmm17, %xmm19 {k1}
+vpshldvd          %xmm16, %xmm17, %xmm19 {k1}{z}
+vpshldvd          (%rax), %xmm17, %xmm19 {k1}{z}
+vpshldvd          (%rax){1to4}, %xmm17, %xmm19 {k1}{z}
+
+vpshldvd          %ymm16, %ymm17, %ymm19
+vpshldvd          (%rax), %ymm17, %ymm19
+vpshldvd          (%rax){1to8}, %ymm17, %ymm19
+vpshldvd          %ymm16, %ymm17, %ymm19 {k1}
+vpshldvd          (%rax), %ymm17, %ymm19 {k1}
+vpshldvd          (%rax){1to8}, %ymm17, %ymm19 {k1}
+vpshldvd          %ymm16, %ymm17, %ymm19 {k1}{z}
+vpshldvd          (%rax), %ymm17, %ymm19 {k1}{z}
+vpshldvd          (%rax){1to8}, %ymm17, %ymm19 {k1}{z}
+
+vpshldvq          %xmm16, %xmm17, %xmm19
+vpshldvq          (%rax), %xmm17, %xmm19
+vpshldvq          (%rax){1to2}, %xmm17, %xmm19
+vpshldvq          %xmm16, %xmm17, %xmm19 {k1}
+vpshldvq          (%rax), %xmm17, %xmm19 {k1}
+vpshldvq          (%rax){1to2}, %xmm17, %xmm19 {k1}
+vpshldvq          %xmm16, %xmm17, %xmm19 {k1}{z}
+vpshldvq          (%rax), %xmm17, %xmm19 {k1}{z}
+vpshldvq          (%rax){1to2}, %xmm17, %xmm19 {k1}{z}
+
+vpshldvq          %ymm16, %ymm17, %ymm19
+vpshldvq          (%rax), %ymm17, %ymm19
+vpshldvq          (%rax){1to4}, %ymm17, %ymm19
+vpshldvq          %ymm16, %ymm17, %ymm19 {k1}
+vpshldvq          (%rax), %ymm17, %ymm19 {k1}
+vpshldvq          (%rax){1to4}, %ymm17, %ymm19 {k1}
+vpshldvq          %ymm16, %ymm17, %ymm19 {k1}{z}
+vpshldvq          (%rax), %ymm17, %ymm19 {k1}{z}
+vpshldvq          (%rax){1to4}, %ymm17, %ymm19 {k1}{z}
+
+vpshldvw          %xmm16, %xmm17, %xmm19
+vpshldvw          (%rax), %xmm17, %xmm19
+vpshldvw          %xmm16, %xmm17, %xmm19 {k1}
+vpshldvw          (%rax), %xmm17, %xmm19 {k1}
+vpshldvw          %xmm16, %xmm17, %xmm19 {k1}{z}
+vpshldvw          (%rax), %xmm17, %xmm19 {k1}{z}
+
+vpshldvw          %ymm16, %ymm17, %ymm19
+vpshldvw          (%rax), %ymm17, %ymm19
+vpshldvw          %ymm16, %ymm17, %ymm19 {k1}
+vpshldvw          (%rax), %ymm17, %ymm19 {k1}
+vpshldvw          %ymm16, %ymm17, %ymm19 {k1}{z}
+vpshldvw          (%rax), %ymm17, %ymm19 {k1}{z}
+
+vpshldw           $1, %xmm16, %xmm17, %xmm19
+vpshldw           $1, (%rax), %xmm17, %xmm19
+vpshldw           $1, %xmm16, %xmm17, %xmm19 {k1}
+vpshldw           $1, (%rax), %xmm17, %xmm19 {k1}
+vpshldw           $1, %xmm16, %xmm17, %xmm19 {k1}{z}
+vpshldw           $1, (%rax), %xmm17, %xmm19 {k1}{z}
+
+vpshldw           $1, %ymm16, %ymm17, %ymm19
+vpshldw           $1, (%rax), %ymm17, %ymm19
+vpshldw           $1, %ymm16, %ymm17, %ymm19 {k1}
+vpshldw           $1, (%rax), %ymm17, %ymm19 {k1}
+vpshldw           $1, %ymm16, %ymm17, %ymm19 {k1}{z}
+vpshldw           $1, (%rax), %ymm17, %ymm19 {k1}{z}
+
+vpshrdd           $1, %xmm16, %xmm17, %xmm19
+vpshrdd           $1, (%rax), %xmm17, %xmm19
+vpshrdd           $1, (%rax){1to4}, %xmm17, %xmm19
+vpshrdd           $1, %xmm16, %xmm17, %xmm19 {k1}
+vpshrdd           $1, (%rax), %xmm17, %xmm19 {k1}
+vpshrdd           $1, (%rax){1to4}, %xmm17, %xmm19 {k1}
+vpshrdd           $1, %xmm16, %xmm17, %xmm19 {k1}{z}
+vpshrdd           $1, (%rax), %xmm17, %xmm19 {k1}{z}
+vpshrdd           $1, (%rax){1to4}, %xmm17, %xmm19 {k1}{z}
+
+vpshrdd           $1, %ymm16, %ymm17, %ymm19
+vpshrdd           $1, (%rax), %ymm17, %ymm19
+vpshrdd           $1, (%rax){1to8}, %ymm17, %ymm19
+vpshrdd           $1, %ymm16, %ymm17, %ymm19 {k1}
+vpshrdd           $1, (%rax), %ymm17, %ymm19 {k1}
+vpshrdd           $1, (%rax){1to8}, %ymm17, %ymm19 {k1}
+vpshrdd           $1, %ymm16, %ymm17, %ymm19 {k1}{z}
+vpshrdd           $1, (%rax), %ymm17, %ymm19 {k1}{z}
+vpshrdd           $1, (%rax){1to8}, %ymm17, %ymm19 {k1}{z}
+
+vpshrdq           $1, %xmm16, %xmm17, %xmm19
+vpshrdq           $1, (%rax), %xmm17, %xmm19
+vpshrdq           $1, (%rax){1to2}, %xmm17, %xmm19
+vpshrdq           $1, %xmm16, %xmm17, %xmm19 {k1}
+vpshrdq           $1, (%rax), %xmm17, %xmm19 {k1}
+vpshrdq           $1, (%rax){1to2}, %xmm17, %xmm19 {k1}
+vpshrdq           $1, %xmm16, %xmm17, %xmm19 {k1}{z}
+vpshrdq           $1, (%rax), %xmm17, %xmm19 {k1}{z}
+vpshrdq           $1, (%rax){1to2}, %xmm17, %xmm19 {k1}{z}
+
+vpshrdq           $1, %ymm16, %ymm17, %ymm19
+vpshrdq           $1, (%rax), %ymm17, %ymm19
+vpshrdq           $1, (%rax){1to4}, %ymm17, %ymm19
+vpshrdq           $1, %ymm16, %ymm17, %ymm19 {k1}
+vpshrdq           $1, (%rax), %ymm17, %ymm19 {k1}
+vpshrdq           $1, (%rax){1to4}, %ymm17, %ymm19 {k1}
+vpshrdq           $1, %ymm16, %ymm17, %ymm19 {k1}{z}
+vpshrdq           $1, (%rax), %ymm17, %ymm19 {k1}{z}
+vpshrdq           $1, (%rax){1to4}, %ymm17, %ymm19 {k1}{z}
+
+vpshrdvd          %xmm16, %xmm17, %xmm19
+vpshrdvd          (%rax), %xmm17, %xmm19
+vpshrdvd          (%rax){1to4}, %xmm17, %xmm19
+vpshrdvd          %xmm16, %xmm17, %xmm19 {k1}
+vpshrdvd          (%rax), %xmm17, %xmm19 {k1}
+vpshrdvd          (%rax){1to4}, %xmm17, %xmm19 {k1}
+vpshrdvd          %xmm16, %xmm17, %xmm19 {k1}{z}
+vpshrdvd          (%rax), %xmm17, %xmm19 {k1}{z}
+vpshrdvd          (%rax){1to4}, %xmm17, %xmm19 {k1}{z}
+
+vpshrdvd          %ymm16, %ymm17, %ymm19
+vpshrdvd          (%rax), %ymm17, %ymm19
+vpshrdvd          (%rax){1to8}, %ymm17, %ymm19
+vpshrdvd          %ymm16, %ymm17, %ymm19 {k1}
+vpshrdvd          (%rax), %ymm17, %ymm19 {k1}
+vpshrdvd          (%rax){1to8}, %ymm17, %ymm19 {k1}
+vpshrdvd          %ymm16, %ymm17, %ymm19 {k1}{z}
+vpshrdvd          (%rax), %ymm17, %ymm19 {k1}{z}
+vpshrdvd          (%rax){1to8}, %ymm17, %ymm19 {k1}{z}
+
+vpshrdvq          %xmm16, %xmm17, %xmm19
+vpshrdvq          (%rax), %xmm17, %xmm19
+vpshrdvq          (%rax){1to2}, %xmm17, %xmm19
+vpshrdvq          %xmm16, %xmm17, %xmm19 {k1}
+vpshrdvq          (%rax), %xmm17, %xmm19 {k1}
+vpshrdvq          (%rax){1to2}, %xmm17, %xmm19 {k1}
+vpshrdvq          %xmm16, %xmm17, %xmm19 {k1}{z}
+vpshrdvq          (%rax), %xmm17, %xmm19 {k1}{z}
+vpshrdvq          (%rax){1to2}, %xmm17, %xmm19 {k1}{z}
+
+vpshrdvq          %ymm16, %ymm17, %ymm19
+vpshrdvq          (%rax), %ymm17, %ymm19
+vpshrdvq          (%rax){1to4}, %ymm17, %ymm19
+vpshrdvq          %ymm16, %ymm17, %ymm19 {k1}
+vpshrdvq          (%rax), %ymm17, %ymm19 {k1}
+vpshrdvq          (%rax){1to4}, %ymm17, %ymm19 {k1}
+vpshrdvq          %ymm16, %ymm17, %ymm19 {k1}{z}
+vpshrdvq          (%rax), %ymm17, %ymm19 {k1}{z}
+vpshrdvq          (%rax){1to4}, %ymm17, %ymm19 {k1}{z}
+
+vpshrdvw          %xmm16, %xmm17, %xmm19
+vpshrdvw          (%rax), %xmm17, %xmm19
+vpshrdvw          %xmm16, %xmm17, %xmm19 {k1}
+vpshrdvw          (%rax), %xmm17, %xmm19 {k1}
+vpshrdvw          %xmm16, %xmm17, %xmm19 {k1}{z}
+vpshrdvw          (%rax), %xmm17, %xmm19 {k1}{z}
+
+vpshrdvw          %ymm16, %ymm17, %ymm19
+vpshrdvw          (%rax), %ymm17, %ymm19
+vpshrdvw          %ymm16, %ymm17, %ymm19 {k1}
+vpshrdvw          (%rax), %ymm17, %ymm19 {k1}
+vpshrdvw          %ymm16, %ymm17, %ymm19 {k1}{z}
+vpshrdvw          (%rax), %ymm17, %ymm19 {k1}{z}
+
+vpshrdw           $1, %xmm16, %xmm17, %xmm19
+vpshrdw           $1, (%rax), %xmm17, %xmm19
+vpshrdw           $1, %xmm16, %xmm17, %xmm19 {k1}
+vpshrdw           $1, (%rax), %xmm17, %xmm19 {k1}
+vpshrdw           $1, %xmm16, %xmm17, %xmm19 {k1}{z}
+vpshrdw           $1, (%rax), %xmm17, %xmm19 {k1}{z}
+
+vpshrdw           $1, %ymm16, %ymm17, %ymm19
+vpshrdw           $1, (%rax), %ymm17, %ymm19
+vpshrdw           $1, %ymm16, %ymm17, %ymm19 {k1}
+vpshrdw           $1, (%rax), %ymm17, %ymm19 {k1}
+vpshrdw           $1, %ymm16, %ymm17, %ymm19 {k1}{z}
+vpshrdw           $1, (%rax), %ymm17, %ymm19 {k1}{z}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      2     1.00                  U     vpcompressb	%xmm16, %xmm19
+# CHECK-NEXT:  2      8     0.50           *            vpcompressb	%xmm16, (%rax)
+# CHECK-NEXT:  1      2     1.00                        vpcompressb	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50           *            vpcompressb	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpcompressb	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     1.00                  U     vpcompressb	%ymm16, %ymm19
+# CHECK-NEXT:  2      8     0.50           *            vpcompressb	%ymm16, (%rax)
+# CHECK-NEXT:  1      4     1.00                        vpcompressb	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  2      8     0.50           *            vpcompressb	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  1      4     1.00                        vpcompressb	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                  U     vpcompressw	%xmm16, %xmm19
+# CHECK-NEXT:  2      8     0.50           *            vpcompressw	%xmm16, (%rax)
+# CHECK-NEXT:  1      2     1.00                        vpcompressw	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50           *            vpcompressw	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpcompressw	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     1.00                  U     vpcompressw	%ymm16, %ymm19
+# CHECK-NEXT:  2      8     0.50           *            vpcompressw	%ymm16, (%rax)
+# CHECK-NEXT:  1      4     1.00                        vpcompressw	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  2      8     0.50           *            vpcompressw	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  1      4     1.00                        vpcompressw	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      1     0.50                  U     vpexpandb	%xmm16, %xmm19
+# CHECK-NEXT:  2      8     0.50                  U     vpexpandb	(%rax), %xmm19
+# CHECK-NEXT:  2      1     0.50                        vpexpandb	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpexpandb	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  2      1     0.50                        vpexpandb	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     1.00                  U     vpexpandb	%ymm16, %ymm19
+# CHECK-NEXT:  2      8     0.50                  U     vpexpandb	(%rax), %ymm19
+# CHECK-NEXT:  1      4     1.00                        vpexpandb	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpexpandb	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      4     1.00                        vpexpandb	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      1     0.50                  U     vpexpandw	%xmm16, %xmm19
+# CHECK-NEXT:  2      8     0.50                  U     vpexpandw	(%rax), %xmm19
+# CHECK-NEXT:  2      1     0.50                        vpexpandw	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpexpandw	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  2      1     0.50                        vpexpandw	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     1.00                  U     vpexpandw	%ymm16, %ymm19
+# CHECK-NEXT:  2      8     0.50                  U     vpexpandw	(%rax), %ymm19
+# CHECK-NEXT:  1      4     1.00                        vpexpandw	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpexpandw	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      4     1.00                        vpexpandw	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshldd	$1, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldd	$1, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldd	$1, (%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpshldd	$1, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldd	$1, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldd	$1, (%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshldd	$1, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldd	$1, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldd	$1, (%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshldd	$1, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldd	$1, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldd	$1, (%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpshldd	$1, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldd	$1, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldd	$1, (%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshldd	$1, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldd	$1, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldd	$1, (%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshldq	$1, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldq	$1, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldq	$1, (%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpshldq	$1, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldq	$1, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldq	$1, (%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshldq	$1, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldq	$1, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldq	$1, (%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshldq	$1, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldq	$1, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldq	$1, (%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpshldq	$1, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldq	$1, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldq	$1, (%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshldq	$1, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldq	$1, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldq	$1, (%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpshldvd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvd	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpshldvd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvd	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshldvd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvd	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpshldvd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvd	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpshldvd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvd	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshldvd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvd	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpshldvq	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvq	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvq	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpshldvq	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvq	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvq	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshldvq	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvq	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvq	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpshldvq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvq	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvq	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpshldvq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvq	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvq	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshldvq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvq	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvq	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpshldvw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpshldvw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshldvw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpshldvw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpshldvw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshldvw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldvw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshldw	$1, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldw	$1, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpshldw	$1, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldw	$1, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshldw	$1, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldw	$1, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshldw	$1, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshldw	$1, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpshldw	$1, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldw	$1, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshldw	$1, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshldw	$1, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshrdd	$1, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdd	$1, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdd	$1, (%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpshrdd	$1, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdd	$1, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdd	$1, (%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshrdd	$1, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdd	$1, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdd	$1, (%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshrdd	$1, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdd	$1, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdd	$1, (%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpshrdd	$1, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdd	$1, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdd	$1, (%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshrdd	$1, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdd	$1, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdd	$1, (%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshrdq	$1, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdq	$1, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdq	$1, (%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpshrdq	$1, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdq	$1, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdq	$1, (%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshrdq	$1, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdq	$1, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdq	$1, (%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshrdq	$1, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdq	$1, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdq	$1, (%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpshrdq	$1, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdq	$1, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdq	$1, (%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshrdq	$1, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdq	$1, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdq	$1, (%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpshrdvd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvd	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpshrdvd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvd	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshrdvd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvd	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpshrdvd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvd	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpshrdvd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvd	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshrdvd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvd	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpshrdvq	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvq	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvq	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpshrdvq	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvq	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvq	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshrdvq	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvq	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvq	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpshrdvq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvq	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvq	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpshrdvq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvq	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvq	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshrdvq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvq	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvq	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpshrdvw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpshrdvw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshrdvw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpshrdvw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpshrdvw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshrdvw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdvw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshrdw	$1, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdw	$1, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpshrdw	$1, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdw	$1, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshrdw	$1, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdw	$1, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpshrdw	$1, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdw	$1, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpshrdw	$1, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdw	$1, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpshrdw	$1, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpshrdw	$1, (%rax), %ymm17, %ymm19 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     126.00 89.00  29.00  66.00  68.00  68.00   -     45.33  45.33  45.33  45.33  45.33  45.33   -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpcompressb	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcompressb	%xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpcompressb	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcompressb	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpcompressb	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpcompressb	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcompressb	%ymm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpcompressb	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcompressb	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpcompressb	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpcompressw	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcompressw	%xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpcompressw	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcompressw	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpcompressw	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpcompressw	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcompressw	%ymm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpcompressw	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcompressw	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpcompressw	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpexpandb	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpexpandb	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpexpandb	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpexpandb	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpexpandb	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpexpandb	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpexpandb	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpexpandb	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpexpandb	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpexpandb	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpexpandw	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpexpandw	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpexpandw	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpexpandw	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpexpandw	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpexpandw	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpexpandw	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpexpandw	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpexpandw	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpexpandw	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldd	$1, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldd	$1, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldd	$1, (%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldd	$1, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldd	$1, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldd	$1, (%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldd	$1, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldd	$1, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldd	$1, (%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldd	$1, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldd	$1, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldd	$1, (%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldd	$1, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldd	$1, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldd	$1, (%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldd	$1, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldd	$1, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldd	$1, (%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldq	$1, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldq	$1, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldq	$1, (%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldq	$1, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldq	$1, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldq	$1, (%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldq	$1, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldq	$1, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldq	$1, (%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldq	$1, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldq	$1, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldq	$1, (%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldq	$1, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldq	$1, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldq	$1, (%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldq	$1, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldq	$1, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldq	$1, (%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpshldvd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvd	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldvd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvd	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldvd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvd	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpshldvd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvd	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldvd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvd	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldvd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvd	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpshldvq	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvq	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvq	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldvq	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvq	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvq	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldvq	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvq	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvq	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpshldvq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvq	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvq	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldvq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvq	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvq	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldvq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvq	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvq	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpshldvw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldvw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldvw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpshldvw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldvw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldvw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldvw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldw	$1, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldw	$1, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldw	$1, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldw	$1, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldw	$1, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldw	$1, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldw	$1, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldw	$1, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldw	$1, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldw	$1, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshldw	$1, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshldw	$1, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdd	$1, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdd	$1, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdd	$1, (%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdd	$1, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdd	$1, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdd	$1, (%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdd	$1, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdd	$1, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdd	$1, (%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdd	$1, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdd	$1, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdd	$1, (%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdd	$1, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdd	$1, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdd	$1, (%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdd	$1, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdd	$1, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdd	$1, (%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdq	$1, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdq	$1, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdq	$1, (%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdq	$1, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdq	$1, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdq	$1, (%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdq	$1, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdq	$1, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdq	$1, (%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdq	$1, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdq	$1, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdq	$1, (%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdq	$1, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdq	$1, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdq	$1, (%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdq	$1, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdq	$1, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdq	$1, (%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpshrdvd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvd	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdvd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvd	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdvd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvd	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpshrdvd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvd	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdvd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvd	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdvd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvd	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpshrdvq	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvq	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvq	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdvq	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvq	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvq	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdvq	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvq	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvq	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpshrdvq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvq	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvq	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdvq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvq	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvq	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdvq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvq	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvq	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpshrdvw	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvw	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdvw	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvw	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdvw	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvw	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpshrdvw	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvw	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdvw	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvw	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdvw	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdvw	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdw	$1, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdw	$1, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdw	$1, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdw	$1, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdw	$1, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdw	$1, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdw	$1, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdw	$1, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdw	$1, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdw	$1, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpshrdw	$1, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshrdw	$1, (%rax), %ymm17, %ymm19 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vbmivl.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vbmivl.s
new file mode 100644
index 0000000000000..887f2f2157865
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vbmivl.s
@@ -0,0 +1,214 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vpermb            %xmm16, %xmm17, %xmm19
+vpermb            (%rax), %xmm17, %xmm19
+vpermb            %xmm16, %xmm17, %xmm19 {k1}
+vpermb            (%rax), %xmm17, %xmm19 {k1}
+vpermb            %xmm16, %xmm17, %xmm19 {k1}{z}
+vpermb            (%rax), %xmm17, %xmm19 {k1}{z}
+
+vpermb            %ymm16, %ymm17, %ymm19
+vpermb            (%rax), %ymm17, %ymm19
+vpermb            %ymm16, %ymm17, %ymm19 {k1}
+vpermb            (%rax), %ymm17, %ymm19 {k1}
+vpermb            %ymm16, %ymm17, %ymm19 {k1}{z}
+vpermb            (%rax), %ymm17, %ymm19 {k1}{z}
+
+vpermi2b          %xmm16, %xmm17, %xmm19
+vpermi2b          (%rax), %xmm17, %xmm19
+vpermi2b          %xmm16, %xmm17, %xmm19 {k1}
+vpermi2b          (%rax), %xmm17, %xmm19 {k1}
+vpermi2b          %xmm16, %xmm17, %xmm19 {k1}{z}
+vpermi2b          (%rax), %xmm17, %xmm19 {k1}{z}
+
+vpermi2b          %ymm16, %ymm17, %ymm19
+vpermi2b          (%rax), %ymm17, %ymm19
+vpermi2b          %ymm16, %ymm17, %ymm19 {k1}
+vpermi2b          (%rax), %ymm17, %ymm19 {k1}
+vpermi2b          %ymm16, %ymm17, %ymm19 {k1}{z}
+vpermi2b          (%rax), %ymm17, %ymm19 {k1}{z}
+
+vpermt2b          %xmm16, %xmm17, %xmm19
+vpermt2b          (%rax), %xmm17, %xmm19
+vpermt2b          %xmm16, %xmm17, %xmm19 {k1}
+vpermt2b          (%rax), %xmm17, %xmm19 {k1}
+vpermt2b          %xmm16, %xmm17, %xmm19 {k1}{z}
+vpermt2b          (%rax), %xmm17, %xmm19 {k1}{z}
+
+vpermt2b          %ymm16, %ymm17, %ymm19
+vpermt2b          (%rax), %ymm17, %ymm19
+vpermt2b          %ymm16, %ymm17, %ymm19 {k1}
+vpermt2b          (%rax), %ymm17, %ymm19 {k1}
+vpermt2b          %ymm16, %ymm17, %ymm19 {k1}{z}
+vpermt2b          (%rax), %ymm17, %ymm19 {k1}{z}
+
+vpmultishiftqb    %xmm16, %xmm17, %xmm19
+vpmultishiftqb    (%rax), %xmm17, %xmm19
+vpmultishiftqb    (%rax){1to2}, %xmm17, %xmm19
+vpmultishiftqb    %xmm16, %xmm17, %xmm19 {k1}
+vpmultishiftqb    (%rax), %xmm17, %xmm19 {k1}
+vpmultishiftqb    (%rax){1to2}, %xmm17, %xmm19 {k1}
+vpmultishiftqb    %xmm16, %xmm17, %xmm19 {k1}{z}
+vpmultishiftqb    (%rax), %xmm17, %xmm19 {k1}{z}
+vpmultishiftqb    (%rax){1to2}, %xmm17, %xmm19 {k1}{z}
+
+vpmultishiftqb    %ymm16, %ymm17, %ymm19
+vpmultishiftqb    (%rax), %ymm17, %ymm19
+vpmultishiftqb    (%rax){1to4}, %ymm17, %ymm19
+vpmultishiftqb    %ymm16, %ymm17, %ymm19 {k1}
+vpmultishiftqb    (%rax), %ymm17, %ymm19 {k1}
+vpmultishiftqb    (%rax){1to4}, %ymm17, %ymm19 {k1}
+vpmultishiftqb    %ymm16, %ymm17, %ymm19 {k1}{z}
+vpmultishiftqb    (%rax), %ymm17, %ymm19 {k1}{z}
+vpmultishiftqb    (%rax){1to4}, %ymm17, %ymm19 {k1}{z}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      2     1.00                        vpermb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpermb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpermb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     1.00                        vpermb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      4     1.00                        vpermb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     1.00                        vpermb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vpermi2b	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermi2b	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      3     1.00                        vpermi2b	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermi2b	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vpermi2b	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermi2b	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     1.00                        vpermi2b	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermi2b	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      4     1.00                        vpermi2b	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermi2b	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     1.00                        vpermi2b	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermi2b	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vpermt2b	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermt2b	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      3     1.00                        vpermt2b	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermt2b	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vpermt2b	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermt2b	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     1.00                        vpermt2b	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermt2b	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      4     1.00                        vpermt2b	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermt2b	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     1.00                        vpermt2b	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermt2b	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpmultishiftqb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmultishiftqb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmultishiftqb	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpmultishiftqb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmultishiftqb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmultishiftqb	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpmultishiftqb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmultishiftqb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmultishiftqb	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpmultishiftqb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmultishiftqb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmultishiftqb	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpmultishiftqb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmultishiftqb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmultishiftqb	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpmultishiftqb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmultishiftqb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmultishiftqb	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     4.50   31.50  31.50  4.50   15.00  15.00   -     10.00  10.00  10.00  10.00  10.00  10.00   -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermi2b	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermi2b	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermi2b	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermi2b	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermi2b	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermi2b	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermi2b	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermi2b	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermi2b	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermi2b	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermi2b	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermi2b	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermt2b	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermt2b	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermt2b	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermt2b	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermt2b	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermt2b	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermt2b	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermt2b	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermt2b	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermt2b	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermt2b	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermt2b	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmultishiftqb	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmultishiftqb	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmultishiftqb	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmultishiftqb	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmultishiftqb	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmultishiftqb	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmultishiftqb	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmultishiftqb	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmultishiftqb	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmultishiftqb	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmultishiftqb	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmultishiftqb	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmultishiftqb	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmultishiftqb	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmultishiftqb	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpmultishiftqb	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmultishiftqb	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmultishiftqb	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vl.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vl.s
new file mode 100644
index 0000000000000..1408037ef99ae
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vl.s
@@ -0,0 +1,4419 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vaddpd            %xmm16, %xmm17, %xmm19
+vaddpd            (%rax), %xmm17, %xmm19
+vaddpd            (%rax){1to2}, %xmm17, %xmm19
+vaddpd            %xmm16, %xmm17, %xmm19 {k1}
+vaddpd            (%rax), %xmm17, %xmm19 {k1}
+vaddpd            (%rax){1to2}, %xmm17, %xmm19 {k1}
+vaddpd            %xmm16, %xmm17, %xmm19 {z}{k1}
+vaddpd            (%rax), %xmm17, %xmm19 {z}{k1}
+vaddpd            (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+vaddpd            %ymm16, %ymm17, %ymm19
+vaddpd            (%rax), %ymm17, %ymm19
+vaddpd            (%rax){1to4}, %ymm17, %ymm19
+vaddpd            %ymm16, %ymm17, %ymm19 {k1}
+vaddpd            (%rax), %ymm17, %ymm19 {k1}
+vaddpd            (%rax){1to4}, %ymm17, %ymm19 {k1}
+vaddpd            %ymm16, %ymm17, %ymm19 {z}{k1}
+vaddpd            (%rax), %ymm17, %ymm19 {z}{k1}
+vaddpd            (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vaddps            %xmm16, %xmm17, %xmm19
+vaddps            (%rax), %xmm17, %xmm19
+vaddps            (%rax){1to4}, %xmm17, %xmm19
+vaddps            %xmm16, %xmm17, %xmm19 {k1}
+vaddps            (%rax), %xmm17, %xmm19 {k1}
+vaddps            (%rax){1to4}, %xmm17, %xmm19 {k1}
+vaddps            %xmm16, %xmm17, %xmm19 {z}{k1}
+vaddps            (%rax), %xmm17, %xmm19 {z}{k1}
+vaddps            (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vaddps            %ymm16, %ymm17, %ymm19
+vaddps            (%rax), %ymm17, %ymm19
+vaddps            (%rax){1to8}, %ymm17, %ymm19
+vaddps            %ymm16, %ymm17, %ymm19 {k1}
+vaddps            (%rax), %ymm17, %ymm19 {k1}
+vaddps            (%rax){1to8}, %ymm17, %ymm19 {k1}
+vaddps            %ymm16, %ymm17, %ymm19 {z}{k1}
+vaddps            (%rax), %ymm17, %ymm19 {z}{k1}
+vaddps            (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+valignd           $1, %xmm16, %xmm17, %xmm19
+valignd           $1, (%rax), %xmm17, %xmm19
+valignd           $1, (%rax){1to4}, %xmm17, %xmm19
+valignd           $1, %xmm16, %xmm17, %xmm19 {k1}
+valignd           $1, (%rax), %xmm17, %xmm19 {k1}
+valignd           $1, (%rax){1to4}, %xmm17, %xmm19 {k1}
+valignd           $1, %xmm16, %xmm17, %xmm19 {z}{k1}
+valignd           $1, (%rax), %xmm17, %xmm19 {z}{k1}
+valignd           $1, (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+valignd           $1, %ymm16, %ymm17, %ymm19
+valignd           $1, (%rax), %ymm17, %ymm19
+valignd           $1, (%rax){1to8}, %ymm17, %ymm19
+valignd           $1, %ymm16, %ymm17, %ymm19 {k1}
+valignd           $1, (%rax), %ymm17, %ymm19 {k1}
+valignd           $1, (%rax){1to8}, %ymm17, %ymm19 {k1}
+valignd           $1, %ymm16, %ymm17, %ymm19 {z}{k1}
+valignd           $1, (%rax), %ymm17, %ymm19 {z}{k1}
+valignd           $1, (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+valignq           $1, %xmm16, %xmm17, %xmm19
+valignq           $1, (%rax), %xmm17, %xmm19
+valignq           $1, (%rax){1to2}, %xmm17, %xmm19
+valignq           $1, %xmm16, %xmm17, %xmm19 {k1}
+valignq           $1, (%rax), %xmm17, %xmm19 {k1}
+valignq           $1, (%rax){1to2}, %xmm17, %xmm19 {k1}
+valignq           $1, %xmm16, %xmm17, %xmm19 {z}{k1}
+valignq           $1, (%rax), %xmm17, %xmm19 {z}{k1}
+valignq           $1, (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+valignq           $1, %ymm16, %ymm17, %ymm19
+valignq           $1, (%rax), %ymm17, %ymm19
+valignq           $1, (%rax){1to4}, %ymm17, %ymm19
+valignq           $1, %ymm16, %ymm17, %ymm19 {k1}
+valignq           $1, (%rax), %ymm17, %ymm19 {k1}
+valignq           $1, (%rax){1to4}, %ymm17, %ymm19 {k1}
+valignq           $1, %ymm16, %ymm17, %ymm19 {z}{k1}
+valignq           $1, (%rax), %ymm17, %ymm19 {z}{k1}
+valignq           $1, (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vbroadcastf32x4   (%rax), %ymm19
+vbroadcastf32x4   (%rax), %ymm19 {k1}
+vbroadcastf32x4   (%rax), %ymm19 {z}{k1}
+
+vbroadcasti32x4   (%rax), %ymm19
+vbroadcasti32x4   (%rax), %ymm19 {k1}
+vbroadcasti32x4   (%rax), %ymm19 {z}{k1}
+
+vbroadcastsd      %xmm16, %ymm19
+vbroadcastsd      (%rax), %ymm19
+vbroadcastsd      %xmm16, %ymm19 {k1}
+vbroadcastsd      (%rax), %ymm19 {k1}
+vbroadcastsd      %xmm16, %ymm19 {z}{k1}
+vbroadcastsd      (%rax), %ymm19 {z}{k1}
+
+vbroadcastss      %xmm16, %xmm19
+vbroadcastss      (%rax), %xmm19
+vbroadcastss      %xmm16, %xmm19 {k1}
+vbroadcastss      (%rax), %xmm19 {k1}
+vbroadcastss      %xmm16, %xmm19 {z}{k1}
+vbroadcastss      (%rax), %xmm19 {z}{k1}
+
+vbroadcastss      %xmm16, %ymm19
+vbroadcastss      (%rax), %ymm19
+vbroadcastss      %xmm16, %ymm19 {k1}
+vbroadcastss      (%rax), %ymm19 {k1}
+vbroadcastss      %xmm16, %ymm19 {z}{k1}
+vbroadcastss      (%rax), %ymm19 {z}{k1}
+
+vcmppd            $0, %xmm0, %xmm1, %k2
+vcmppd            $0, (%rax), %xmm1, %k2
+vcmppd            $0, (%rax){1to2}, %xmm1, %k2
+vcmppd            $0, %xmm0, %xmm1, %k2 {k3}
+vcmppd            $0, (%rax), %xmm1, %k2 {k3}
+vcmppd            $0, (%rax){1to2}, %xmm1, %k2 {k3}
+
+vcmppd            $0, %ymm0, %ymm1, %k2
+vcmppd            $0, (%rax), %ymm1, %k2
+vcmppd            $0, (%rax){1to4}, %ymm1, %k2
+vcmppd            $0, %ymm0, %ymm1, %k2 {k3}
+vcmppd            $0, (%rax), %ymm1, %k2 {k3}
+vcmppd            $0, (%rax){1to4}, %ymm1, %k2 {k3}
+
+vcmpps            $0, %xmm0, %xmm1, %k2
+vcmpps            $0, (%rax), %xmm1, %k2
+vcmpps            $0, (%rax){1to4}, %xmm1, %k2
+vcmpps            $0, %xmm0, %xmm1, %k2 {k3}
+vcmpps            $0, (%rax), %xmm1, %k2 {k3}
+vcmpps            $0, (%rax){1to4}, %xmm1, %k2 {k3}
+
+vcmpps            $0, %ymm0, %ymm1, %k2
+vcmpps            $0, (%rax), %ymm1, %k2
+vcmpps            $0, (%rax){1to8}, %ymm1, %k2
+vcmpps            $0, %ymm0, %ymm1, %k2 {k3}
+vcmpps            $0, (%rax), %ymm1, %k2 {k3}
+vcmpps            $0, (%rax){1to8}, %ymm1, %k2 {k3}
+
+vcvtdq2pd         %xmm16, %xmm19
+vcvtdq2pd         (%rax), %xmm19
+vcvtdq2pd         (%rax){1to2}, %xmm19
+vcvtdq2pd         %xmm16, %xmm19 {k1}
+vcvtdq2pd         (%rax), %xmm19 {k1}
+vcvtdq2pd         (%rax){1to2}, %xmm19 {k1}
+vcvtdq2pd         %xmm16, %xmm19 {z}{k1}
+vcvtdq2pd         (%rax), %xmm19 {z}{k1}
+vcvtdq2pd         (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvtdq2pd         %xmm16, %ymm19
+vcvtdq2pd         (%rax), %ymm19
+vcvtdq2pd         (%rax){1to4}, %ymm19
+vcvtdq2pd         %xmm16, %ymm19 {k1}
+vcvtdq2pd         (%rax), %ymm19 {k1}
+vcvtdq2pd         (%rax){1to4}, %ymm19 {k1}
+vcvtdq2pd         %xmm16, %ymm19 {z}{k1}
+vcvtdq2pd         (%rax), %ymm19 {z}{k1}
+vcvtdq2pd         (%rax){1to4}, %ymm19 {z}{k1}
+
+vcvtdq2ps         %xmm16, %xmm19
+vcvtdq2ps         (%rax), %xmm19
+vcvtdq2ps         (%rax){1to4}, %xmm19
+vcvtdq2ps         %xmm16, %xmm19 {k1}
+vcvtdq2ps         (%rax), %xmm19 {k1}
+vcvtdq2ps         (%rax){1to4},%xmm19 {k1}
+vcvtdq2ps         %xmm16, %xmm19 {z}{k1}
+vcvtdq2ps         (%rax), %xmm19 {z}{k1}
+vcvtdq2ps         (%rax){1to4}, %xmm19 {z}{k1}
+
+vcvtdq2ps         %ymm16, %ymm19
+vcvtdq2ps         (%rax), %ymm19
+vcvtdq2ps         (%rax){1to8}, %ymm19
+vcvtdq2ps         %ymm16,%ymm19 {k1}
+vcvtdq2ps         (%rax),%ymm19 {k1}
+vcvtdq2ps         (%rax){1to8}, %ymm19 {k1}
+vcvtdq2ps         %ymm16, %ymm19 {z}{k1}
+vcvtdq2ps         (%rax), %ymm19 {z}{k1}
+vcvtdq2ps         (%rax){1to8}, %ymm19 {z}{k1}
+
+vcvtpd2dqy        %ymm16, %xmm19
+vcvtpd2dqy        (%rax), %xmm19
+vcvtpd2dqy        (%rax){1to4}, %xmm19
+vcvtpd2dqy        %ymm16, %xmm19 {k1}
+vcvtpd2dqy        (%rax), %xmm19 {k1}
+vcvtpd2dqy        (%rax){1to4}, %xmm19 {k1}
+vcvtpd2dqy        %ymm16, %xmm19 {z}{k1}
+vcvtpd2dqy        (%rax), %xmm19 {z}{k1}
+vcvtpd2dqy        (%rax){1to4}, %xmm19 {z}{k1}
+
+vcvtpd2dqx        %xmm16, %xmm19
+vcvtpd2dqx        (%rax), %xmm19
+vcvtpd2dqx        (%rax){1to2}, %xmm19
+vcvtpd2dqx        %xmm16, %xmm19 {k1}
+vcvtpd2dqx        (%rax), %xmm19 {k1}
+vcvtpd2dqx        (%rax){1to2},%xmm19 {k1}
+vcvtpd2dqx        %xmm16, %xmm19 {z}{k1}
+vcvtpd2dqx        (%rax), %xmm19 {z}{k1}
+vcvtpd2dqx        (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvtpd2psy        %ymm16, %xmm19
+vcvtpd2psy        (%rax), %xmm19
+vcvtpd2psy        (%rax){1to4}, %xmm19
+vcvtpd2psy        %ymm16, %xmm19 {k1}
+vcvtpd2psy        (%rax), %xmm19 {k1}
+vcvtpd2psy        (%rax){1to4}, %xmm19 {k1}
+vcvtpd2psy        %ymm16, %xmm19 {z}{k1}
+vcvtpd2psy        (%rax), %xmm19 {z}{k1}
+vcvtpd2psy        (%rax){1to4}, %xmm19 {z}{k1}
+
+vcvtpd2psx        %xmm16, %xmm19
+vcvtpd2psx        (%rax), %xmm19
+vcvtpd2psx        (%rax){1to2}, %xmm19
+vcvtpd2psx        %xmm16, %xmm19 {k1}
+vcvtpd2psx        (%rax), %xmm19 {k1}
+vcvtpd2psx        (%rax){1to2},%xmm19 {k1}
+vcvtpd2psx        %xmm16, %xmm19 {z}{k1}
+vcvtpd2psx        (%rax), %xmm19 {z}{k1}
+vcvtpd2psx        (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvtpd2udqy       %ymm16, %xmm19
+vcvtpd2udqy       (%rax), %xmm19
+vcvtpd2udqy       (%rax){1to4}, %xmm19
+vcvtpd2udqy       %ymm16, %xmm19 {k1}
+vcvtpd2udqy       (%rax), %xmm19 {k1}
+vcvtpd2udqy       (%rax){1to4}, %xmm19 {k1}
+vcvtpd2udqy       %ymm16, %xmm19 {z}{k1}
+vcvtpd2udqy       (%rax), %xmm19 {z}{k1}
+vcvtpd2udqy       (%rax){1to4}, %xmm19 {z}{k1}
+
+vcvtpd2udqx       %xmm16, %xmm19
+vcvtpd2udqx       (%rax), %xmm19
+vcvtpd2udqx       (%rax){1to2}, %xmm19
+vcvtpd2udqx       %xmm16, %xmm19 {k1}
+vcvtpd2udqx       (%rax), %xmm19 {k1}
+vcvtpd2udqx       (%rax){1to2},%xmm19 {k1}
+vcvtpd2udqx       %xmm16, %xmm19 {z}{k1}
+vcvtpd2udqx       (%rax), %xmm19 {z}{k1}
+vcvtpd2udqx       (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvtps2dq         %xmm16, %xmm19
+vcvtps2dq         (%rax), %xmm19
+vcvtps2dq         (%rax){1to4}, %xmm19
+vcvtps2dq         %xmm16, %xmm19 {k1}
+vcvtps2dq         (%rax), %xmm19 {k1}
+vcvtps2dq         (%rax){1to4},%xmm19 {k1}
+vcvtps2dq         %xmm16, %xmm19 {z}{k1}
+vcvtps2dq         (%rax), %xmm19 {z}{k1}
+vcvtps2dq         (%rax){1to4}, %xmm19 {z}{k1}
+
+vcvtps2dq         %ymm16, %ymm19
+vcvtps2dq         (%rax), %ymm19
+vcvtps2dq         (%rax){1to8}, %ymm19
+vcvtps2dq         %ymm16,%ymm19 {k1}
+vcvtps2dq         (%rax),%ymm19 {k1}
+vcvtps2dq         (%rax){1to8}, %ymm19 {k1}
+vcvtps2dq         %ymm16, %ymm19 {z}{k1}
+vcvtps2dq         (%rax), %ymm19 {z}{k1}
+vcvtps2dq         (%rax){1to8}, %ymm19 {z}{k1}
+
+vcvtps2pd         %xmm16, %xmm19
+vcvtps2pd         (%rax), %xmm19
+vcvtps2pd         (%rax){1to2}, %xmm19
+vcvtps2pd         %xmm16, %xmm19 {k1}
+vcvtps2pd         (%rax), %xmm19 {k1}
+vcvtps2pd         (%rax){1to2},%xmm19 {k1}
+vcvtps2pd         %xmm16, %xmm19 {z}{k1}
+vcvtps2pd         (%rax), %xmm19 {z}{k1}
+vcvtps2pd         (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvtps2pd         %xmm16, %ymm19
+vcvtps2pd         (%rax), %ymm19
+vcvtps2pd         (%rax){1to4}, %ymm19
+vcvtps2pd         %xmm16,%ymm19 {k1}
+vcvtps2pd         (%rax),%ymm19 {k1}
+vcvtps2pd         (%rax){1to4}, %ymm19 {k1}
+vcvtps2pd         %xmm16, %ymm19 {z}{k1}
+vcvtps2pd         (%rax), %ymm19 {z}{k1}
+vcvtps2pd         (%rax){1to4}, %ymm19 {z}{k1}
+
+vcvtps2udq        %xmm16, %xmm19
+vcvtps2udq        (%rax), %xmm19
+vcvtps2udq        (%rax){1to4}, %xmm19
+vcvtps2udq        %xmm16, %xmm19 {k1}
+vcvtps2udq        (%rax), %xmm19 {k1}
+vcvtps2udq        (%rax){1to4},%xmm19 {k1}
+vcvtps2udq        %xmm16, %xmm19 {z}{k1}
+vcvtps2udq        (%rax), %xmm19 {z}{k1}
+vcvtps2udq        (%rax){1to4}, %xmm19 {z}{k1}
+
+vcvtps2udq        %ymm16, %ymm19
+vcvtps2udq        (%rax), %ymm19
+vcvtps2udq        (%rax){1to8}, %ymm19
+vcvtps2udq        %ymm16,%ymm19 {k1}
+vcvtps2udq        (%rax),%ymm19 {k1}
+vcvtps2udq        (%rax){1to8}, %ymm19 {k1}
+vcvtps2udq        %ymm16, %ymm19 {z}{k1}
+vcvtps2udq        (%rax), %ymm19 {z}{k1}
+vcvtps2udq        (%rax){1to8}, %ymm19 {z}{k1}
+
+vcvttpd2dqy       %ymm16, %xmm19
+vcvttpd2dqy       (%rax), %xmm19
+vcvttpd2dqy       (%rax){1to4}, %xmm19
+vcvttpd2dqy       %ymm16, %xmm19 {k1}
+vcvttpd2dqy       (%rax), %xmm19 {k1}
+vcvttpd2dqy       (%rax){1to4}, %xmm19 {k1}
+vcvttpd2dqy       %ymm16, %xmm19 {z}{k1}
+vcvttpd2dqy       (%rax), %xmm19 {z}{k1}
+vcvttpd2dqy       (%rax){1to4}, %xmm19 {z}{k1}
+
+vcvttpd2dqx       %xmm16, %xmm19
+vcvttpd2dqx       (%rax), %xmm19
+vcvttpd2dqx       (%rax){1to2}, %xmm19
+vcvttpd2dqx       %xmm16, %xmm19 {k1}
+vcvttpd2dqx       (%rax), %xmm19 {k1}
+vcvttpd2dqx       (%rax){1to2},%xmm19 {k1}
+vcvttpd2dqx       %xmm16, %xmm19 {z}{k1}
+vcvttpd2dqx       (%rax), %xmm19 {z}{k1}
+vcvttpd2dqx       (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvttps2dq        %xmm16, %xmm19
+vcvttps2dq        (%rax), %xmm19
+vcvttps2dq        (%rax){1to4}, %xmm19
+vcvttps2dq        %xmm16, %xmm19 {k1}
+vcvttps2dq        (%rax), %xmm19 {k1}
+vcvttps2dq        (%rax){1to4},%xmm19 {k1}
+vcvttps2dq        %xmm16, %xmm19 {z}{k1}
+vcvttps2dq        (%rax), %xmm19 {z}{k1}
+vcvttps2dq        (%rax){1to4}, %xmm19 {z}{k1}
+
+vcvttps2dq        %ymm16, %ymm19
+vcvttps2dq        (%rax), %ymm19
+vcvttps2dq        (%rax){1to8}, %ymm19
+vcvttps2dq        %ymm16,%ymm19 {k1}
+vcvttps2dq        (%rax),%ymm19 {k1}
+vcvttps2dq        (%rax){1to8}, %ymm19 {k1}
+vcvttps2dq        %ymm16, %ymm19 {z}{k1}
+vcvttps2dq        (%rax), %ymm19 {z}{k1}
+vcvttps2dq        (%rax){1to8}, %ymm19 {z}{k1}
+
+vcvttpd2udqy      %ymm16, %xmm19
+vcvttpd2udqy      (%rax), %xmm19
+vcvttpd2udqy      (%rax){1to4}, %xmm19
+vcvttpd2udqy      %ymm16, %xmm19 {k1}
+vcvttpd2udqy      (%rax), %xmm19 {k1}
+vcvttpd2udqy      (%rax){1to4}, %xmm19 {k1}
+vcvttpd2udqy      %ymm16, %xmm19 {z}{k1}
+vcvttpd2udqy      (%rax), %xmm19 {z}{k1}
+vcvttpd2udqy      (%rax){1to4}, %xmm19 {z}{k1}
+
+vcvttpd2udqx      %xmm16, %xmm19
+vcvttpd2udqx      (%rax), %xmm19
+vcvttpd2udqx      (%rax){1to2}, %xmm19
+vcvttpd2udqx      %xmm16, %xmm19 {k1}
+vcvttpd2udqx      (%rax), %xmm19 {k1}
+vcvttpd2udqx      (%rax){1to2},%xmm19 {k1}
+vcvttpd2udqx      %xmm16, %xmm19 {z}{k1}
+vcvttpd2udqx      (%rax), %xmm19 {z}{k1}
+vcvttpd2udqx      (%rax){1to2}, %xmm19 {z}{k1}
+
+vcvttps2udq       %xmm16, %xmm19
+vcvttps2udq       (%rax), %xmm19
+vcvttps2udq       (%rax){1to4}, %xmm19
+vcvttps2udq       %xmm16, %xmm19 {k1}
+vcvttps2udq       (%rax), %xmm19 {k1}
+vcvttps2udq       (%rax){1to4},%xmm19 {k1}
+vcvttps2udq       %xmm16, %xmm19 {z}{k1}
+vcvttps2udq       (%rax), %xmm19 {z}{k1}
+vcvttps2udq       (%rax){1to4}, %xmm19 {z}{k1}
+
+vcvttps2udq       %ymm16, %ymm19
+vcvttps2udq       (%rax), %ymm19
+vcvttps2udq       (%rax){1to8}, %ymm19
+vcvttps2udq       %ymm16,%ymm19 {k1}
+vcvttps2udq       (%rax),%ymm19 {k1}
+vcvttps2udq       (%rax){1to8}, %ymm19 {k1}
+vcvttps2udq       %ymm16, %ymm19 {z}{k1}
+vcvttps2udq       (%rax), %ymm19 {z}{k1}
+vcvttps2udq       (%rax){1to8}, %ymm19 {z}{k1}
+
+vdivpd            %xmm16, %xmm17, %xmm19
+vdivpd            (%rax), %xmm17, %xmm19
+vdivpd            (%rax){1to2}, %xmm17, %xmm19
+vdivpd            %xmm16, %xmm17, %xmm19 {k1}
+vdivpd            (%rax), %xmm17, %xmm19 {k1}
+vdivpd            (%rax){1to2}, %xmm17, %xmm19 {k1}
+vdivpd            %xmm16, %xmm17, %xmm19 {z}{k1}
+vdivpd            (%rax), %xmm17, %xmm19 {z}{k1}
+vdivpd            (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+vdivpd            %ymm16, %ymm17, %ymm19
+vdivpd            (%rax), %ymm17, %ymm19
+vdivpd            (%rax){1to4}, %ymm17, %ymm19
+vdivpd            %ymm16, %ymm17, %ymm19 {k1}
+vdivpd            (%rax), %ymm17, %ymm19 {k1}
+vdivpd            (%rax){1to4}, %ymm17, %ymm19 {k1}
+vdivpd            %ymm16, %ymm17, %ymm19 {z}{k1}
+vdivpd            (%rax), %ymm17, %ymm19 {z}{k1}
+vdivpd            (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vdivps            %xmm16, %xmm17, %xmm19
+vdivps            (%rax), %xmm17, %xmm19
+vdivps            (%rax){1to4}, %xmm17, %xmm19
+vdivps            %xmm16, %xmm17, %xmm19 {k1}
+vdivps            (%rax), %xmm17, %xmm19 {k1}
+vdivps            (%rax){1to4}, %xmm17, %xmm19 {k1}
+vdivps            %xmm16, %xmm17, %xmm19 {z}{k1}
+vdivps            (%rax), %xmm17, %xmm19 {z}{k1}
+vdivps            (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vdivps            %ymm16, %ymm17, %ymm19
+vdivps            (%rax), %ymm17, %ymm19
+vdivps            (%rax){1to8}, %ymm17, %ymm19
+vdivps            %ymm16, %ymm17, %ymm19 {k1}
+vdivps            (%rax), %ymm17, %ymm19 {k1}
+vdivps            (%rax){1to8}, %ymm17, %ymm19 {k1}
+vdivps            %ymm16, %ymm17, %ymm19 {z}{k1}
+vdivps            (%rax), %ymm17, %ymm19 {z}{k1}
+vdivps            (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vgatherdpd        (%rax,%xmm1,2), %ymm2 {k1}
+vgatherdps        (%rax,%ymm1,2), %ymm2 {k1}
+vgatherqpd        (%rax,%ymm1,2), %ymm2 {k1}
+vgatherqps        (%rax,%ymm1,2), %xmm2 {k1}
+
+vgatherdpd        (%rax,%xmm1,2), %xmm2 {k1}
+vgatherdps        (%rax,%xmm1,2), %xmm2 {k1}
+vgatherqpd        (%rax,%xmm1,2), %xmm2 {k1}
+vgatherqps        (%rax,%xmm1,2), %xmm2 {k1}
+
+vmaxpd            %xmm16, %xmm17, %xmm19
+vmaxpd            (%rax), %xmm17, %xmm19
+vmaxpd            (%rax){1to2}, %xmm17, %xmm19
+vmaxpd            %xmm16, %xmm17, %xmm19 {k1}
+vmaxpd            (%rax), %xmm17, %xmm19 {k1}
+vmaxpd            (%rax){1to2}, %xmm17, %xmm19 {k1}
+vmaxpd            %xmm16, %xmm17, %xmm19 {z}{k1}
+vmaxpd            (%rax), %xmm17, %xmm19 {z}{k1}
+vmaxpd            (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+vmaxpd            %ymm16, %ymm17, %ymm19
+vmaxpd            (%rax), %ymm17, %ymm19
+vmaxpd            (%rax){1to4}, %ymm17, %ymm19
+vmaxpd            %ymm16, %ymm17, %ymm19 {k1}
+vmaxpd            (%rax), %ymm17, %ymm19 {k1}
+vmaxpd            (%rax){1to4}, %ymm17, %ymm19 {k1}
+vmaxpd            %ymm16, %ymm17, %ymm19 {z}{k1}
+vmaxpd            (%rax), %ymm17, %ymm19 {z}{k1}
+vmaxpd            (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vmaxps            %xmm16, %xmm17, %xmm19
+vmaxps            (%rax), %xmm17, %xmm19
+vmaxps            (%rax){1to4}, %xmm17, %xmm19
+vmaxps            %xmm16, %xmm17, %xmm19 {k1}
+vmaxps            (%rax), %xmm17, %xmm19 {k1}
+vmaxps            (%rax){1to4}, %xmm17, %xmm19 {k1}
+vmaxps            %xmm16, %xmm17, %xmm19 {z}{k1}
+vmaxps            (%rax), %xmm17, %xmm19 {z}{k1}
+vmaxps            (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vmaxps            %ymm16, %ymm17, %ymm19
+vmaxps            (%rax), %ymm17, %ymm19
+vmaxps            (%rax){1to8}, %ymm17, %ymm19
+vmaxps            %ymm16, %ymm17, %ymm19 {k1}
+vmaxps            (%rax), %ymm17, %ymm19 {k1}
+vmaxps            (%rax){1to8}, %ymm17, %ymm19 {k1}
+vmaxps            %ymm16, %ymm17, %ymm19 {z}{k1}
+vmaxps            (%rax), %ymm17, %ymm19 {z}{k1}
+vmaxps            (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vminpd            %xmm16, %xmm17, %xmm19
+vminpd            (%rax), %xmm17, %xmm19
+vminpd            (%rax){1to2}, %xmm17, %xmm19
+vminpd            %xmm16, %xmm17, %xmm19 {k1}
+vminpd            (%rax), %xmm17, %xmm19 {k1}
+vminpd            (%rax){1to2}, %xmm17, %xmm19 {k1}
+vminpd            %xmm16, %xmm17, %xmm19 {z}{k1}
+vminpd            (%rax), %xmm17, %xmm19 {z}{k1}
+vminpd            (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+vminpd            %ymm16, %ymm17, %ymm19
+vminpd            (%rax), %ymm17, %ymm19
+vminpd            (%rax){1to4}, %ymm17, %ymm19
+vminpd            %ymm16, %ymm17, %ymm19 {k1}
+vminpd            (%rax), %ymm17, %ymm19 {k1}
+vminpd            (%rax){1to4}, %ymm17, %ymm19 {k1}
+vminpd            %ymm16, %ymm17, %ymm19 {z}{k1}
+vminpd            (%rax), %ymm17, %ymm19 {z}{k1}
+vminpd            (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vminps            %xmm16, %xmm17, %xmm19
+vminps            (%rax), %xmm17, %xmm19
+vminps            (%rax){1to4}, %xmm17, %xmm19
+vminps            %xmm16, %xmm17, %xmm19 {k1}
+vminps            (%rax), %xmm17, %xmm19 {k1}
+vminps            (%rax){1to4}, %xmm17, %xmm19 {k1}
+vminps            %xmm16, %xmm17, %xmm19 {z}{k1}
+vminps            (%rax), %xmm17, %xmm19 {z}{k1}
+vminps            (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vminps            %ymm16, %ymm17, %ymm19
+vminps            (%rax), %ymm17, %ymm19
+vminps            (%rax){1to8}, %ymm17, %ymm19
+vminps            %ymm16, %ymm17, %ymm19 {k1}
+vminps            (%rax), %ymm17, %ymm19 {k1}
+vminps            (%rax){1to8}, %ymm17, %ymm19 {k1}
+vminps            %ymm16, %ymm17, %ymm19 {z}{k1}
+vminps            (%rax), %ymm17, %ymm19 {z}{k1}
+vminps            (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vmovapd           %xmm16, %xmm19
+vmovapd           (%rax), %xmm19
+vmovapd           %xmm16, (%rax)
+vmovapd           %xmm16, %xmm19 {k1}
+vmovapd           (%rax), %xmm19 {k1}
+vmovapd           %xmm16, (%rax) {k1}
+vmovapd           %xmm16, %xmm19 {z}{k1}
+vmovapd           (%rax), %xmm19 {z}{k1}
+
+vmovapd           %ymm16, %ymm19
+vmovapd           (%rax), %ymm19
+vmovapd           %ymm16, (%rax)
+vmovapd           %ymm16, %ymm19 {k1}
+vmovapd           (%rax), %ymm19 {k1}
+vmovapd           %ymm16, (%rax) {k1}
+vmovapd           %ymm16, %ymm19 {z}{k1}
+vmovapd           (%rax), %ymm19 {z}{k1}
+
+vmovaps           %xmm16, %xmm19
+vmovaps           (%rax), %xmm19
+vmovaps           %xmm16, (%rax)
+vmovaps           %xmm16, %xmm19 {k1}
+vmovaps           (%rax), %xmm19 {k1}
+vmovaps           %xmm16, (%rax) {k1}
+vmovaps           %xmm16, %xmm19 {z}{k1}
+vmovaps           (%rax), %xmm19 {z}{k1}
+
+vmovaps           %ymm16, %ymm19
+vmovaps           (%rax), %ymm19
+vmovaps           %ymm16, (%rax)
+vmovaps           %ymm16, %ymm19 {k1}
+vmovaps           (%rax), %ymm19 {k1}
+vmovaps           %ymm16, (%rax) {k1}
+vmovaps           %ymm16, %ymm19 {z}{k1}
+vmovaps           (%rax), %ymm19 {z}{k1}
+
+vmovddup          %xmm16, %xmm19
+vmovddup          (%rax), %xmm19
+vmovddup          %xmm16, %xmm19 {k1}
+vmovddup          (%rax), %xmm19 {k1}
+vmovddup          %xmm16, %xmm19 {z}{k1}
+vmovddup          (%rax), %xmm19 {z}{k1}
+
+vmovdqa32         %xmm16, %xmm19
+vmovdqa32         (%rax), %xmm19
+vmovdqa32         %xmm16, (%rax)
+vmovdqa32         %xmm16, %xmm19 {k1}
+vmovdqa32         (%rax), %xmm19 {k1}
+vmovdqa32         %xmm16, (%rax) {k1}
+vmovdqa32         %xmm16, %xmm19 {z}{k1}
+vmovdqa32         (%rax), %xmm19 {z}{k1}
+
+vmovdqa32         %ymm16, %ymm19
+vmovdqa32         (%rax), %ymm19
+vmovdqa32         %ymm16, (%rax)
+vmovdqa32         %ymm16, %ymm19 {k1}
+vmovdqa32         (%rax), %ymm19 {k1}
+vmovdqa32         %ymm16, (%rax) {k1}
+vmovdqa32         %ymm16, %ymm19 {z}{k1}
+vmovdqa32         (%rax), %ymm19 {z}{k1}
+
+vmovdqa64         %xmm16, %xmm19
+vmovdqa64         (%rax), %xmm19
+vmovdqa64         %xmm16, (%rax)
+vmovdqa64         %xmm16, %xmm19 {k1}
+vmovdqa64         (%rax), %xmm19 {k1}
+vmovdqa64         %xmm16, (%rax) {k1}
+vmovdqa64         %xmm16, %xmm19 {z}{k1}
+vmovdqa64         (%rax), %xmm19 {z}{k1}
+
+vmovdqa64         %ymm16, %ymm19
+vmovdqa64         (%rax), %ymm19
+vmovdqa64         %ymm16, (%rax)
+vmovdqa64         %ymm16, %ymm19 {k1}
+vmovdqa64         (%rax), %ymm19 {k1}
+vmovdqa64         %ymm16, (%rax) {k1}
+vmovdqa64         %ymm16, %ymm19 {z}{k1}
+vmovdqa64         (%rax), %ymm19 {z}{k1}
+
+vmovdqu32         %xmm16, %xmm19
+vmovdqu32         (%rax), %xmm19
+vmovdqu32         %xmm16, (%rax)
+vmovdqu32         %xmm16, %xmm19 {k1}
+vmovdqu32         (%rax), %xmm19 {k1}
+vmovdqu32         %xmm16, (%rax) {k1}
+vmovdqu32         %xmm16, %xmm19 {z}{k1}
+vmovdqu32         (%rax), %xmm19 {z}{k1}
+
+vmovdqu32         %ymm16, %ymm19
+vmovdqu32         (%rax), %ymm19
+vmovdqu32         %ymm16, (%rax)
+vmovdqu32         %ymm16, %ymm19 {k1}
+vmovdqu32         (%rax), %ymm19 {k1}
+vmovdqu32         %ymm16, (%rax) {k1}
+vmovdqu32         %ymm16, %ymm19 {z}{k1}
+vmovdqu32         (%rax), %ymm19 {z}{k1}
+
+vmovdqu64         %xmm16, %xmm19
+vmovdqu64         (%rax), %xmm19
+vmovdqu64         %xmm16, (%rax)
+vmovdqu64         %xmm16, %xmm19 {k1}
+vmovdqu64         (%rax), %xmm19 {k1}
+vmovdqu64         %xmm16, (%rax) {k1}
+vmovdqu64         %xmm16, %xmm19 {z}{k1}
+vmovdqu64         (%rax), %xmm19 {z}{k1}
+
+vmovdqu64         %ymm16, %ymm19
+vmovdqu64         (%rax), %ymm19
+vmovdqu64         %ymm16, (%rax)
+vmovdqu64         %ymm16, %ymm19 {k1}
+vmovdqu64         (%rax), %ymm19 {k1}
+vmovdqu64         %ymm16, (%rax) {k1}
+vmovdqu64         %ymm16, %ymm19 {z}{k1}
+vmovdqu64         (%rax), %ymm19 {z}{k1}
+
+vmovddup          %ymm16, %ymm19
+vmovddup          (%rax), %ymm19
+vmovddup          %ymm16, %ymm19 {k1}
+vmovddup          (%rax), %ymm19 {k1}
+vmovddup          %ymm16, %ymm19 {z}{k1}
+vmovddup          (%rax), %ymm19 {z}{k1}
+
+vmovshdup         %xmm16, %xmm19
+vmovshdup         (%rax), %xmm19
+vmovshdup         %xmm16, %xmm19 {k1}
+vmovshdup         (%rax), %xmm19 {k1}
+vmovshdup         %xmm16, %xmm19 {z}{k1}
+vmovshdup         (%rax), %xmm19 {z}{k1}
+
+vmovshdup         %ymm16, %ymm19
+vmovshdup         (%rax), %ymm19
+vmovshdup         %ymm16, %ymm19 {k1}
+vmovshdup         (%rax), %ymm19 {k1}
+vmovshdup         %ymm16, %ymm19 {z}{k1}
+vmovshdup         (%rax), %ymm19 {z}{k1}
+
+vmovsldup         %xmm16, %xmm19
+vmovsldup         (%rax), %xmm19
+vmovsldup         %xmm16, %xmm19 {k1}
+vmovsldup         (%rax), %xmm19 {k1}
+vmovsldup         %xmm16, %xmm19 {z}{k1}
+vmovsldup         (%rax), %xmm19 {z}{k1}
+
+vmovsldup         %ymm16, %ymm19
+vmovsldup         (%rax), %ymm19
+vmovsldup         %ymm16, %ymm19 {k1}
+vmovsldup         (%rax), %ymm19 {k1}
+vmovsldup         %ymm16, %ymm19 {z}{k1}
+vmovsldup         (%rax), %ymm19 {z}{k1}
+
+vmovupd           %xmm16, %xmm19
+vmovupd           (%rax), %xmm19
+vmovupd           %xmm16, (%rax)
+vmovupd           %xmm16, %xmm19 {k1}
+vmovupd           (%rax), %xmm19 {k1}
+vmovupd           %xmm16, (%rax) {k1}
+vmovupd           %xmm16, %xmm19 {z}{k1}
+vmovupd           (%rax), %xmm19 {z}{k1}
+
+vmovupd           %ymm16, %ymm19
+vmovupd           (%rax), %ymm19
+vmovupd           %ymm16, (%rax)
+vmovupd           %ymm16, %ymm19 {k1}
+vmovupd           (%rax), %ymm19 {k1}
+vmovupd           %ymm16, (%rax) {k1}
+vmovupd           %ymm16, %ymm19 {z}{k1}
+vmovupd           (%rax), %ymm19 {z}{k1}
+
+vmovups           %xmm16, %xmm19
+vmovups           (%rax), %xmm19
+vmovups           %xmm16, (%rax)
+vmovups           %xmm16, %xmm19 {k1}
+vmovups           (%rax), %xmm19 {k1}
+vmovups           %xmm16, (%rax) {k1}
+vmovups           %xmm16, %xmm19 {z}{k1}
+vmovups           (%rax), %xmm19 {z}{k1}
+
+vmovups           %ymm16, %ymm19
+vmovups           (%rax), %ymm19
+vmovups           %ymm16, (%rax)
+vmovups           %ymm16, %ymm19 {k1}
+vmovups           (%rax), %ymm19 {k1}
+vmovups           %ymm16, (%rax) {k1}
+vmovups           %ymm16, %ymm19 {z}{k1}
+vmovups           (%rax), %ymm19 {z}{k1}
+
+vmulpd            %xmm16, %xmm17, %xmm19
+vmulpd            (%rax), %xmm17, %xmm19
+vmulpd            (%rax){1to2}, %xmm17, %xmm19
+vmulpd            %xmm16, %xmm17, %xmm19 {k1}
+vmulpd            (%rax), %xmm17, %xmm19 {k1}
+vmulpd            (%rax){1to2}, %xmm17, %xmm19 {k1}
+vmulpd            %xmm16, %xmm17, %xmm19 {z}{k1}
+vmulpd            (%rax), %xmm17, %xmm19 {z}{k1}
+vmulpd            (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+vmulpd            %ymm16, %ymm17, %ymm19
+vmulpd            (%rax), %ymm17, %ymm19
+vmulpd            (%rax){1to4}, %ymm17, %ymm19
+vmulpd            %ymm16, %ymm17, %ymm19 {k1}
+vmulpd            (%rax), %ymm17, %ymm19 {k1}
+vmulpd            (%rax){1to4}, %ymm17, %ymm19 {k1}
+vmulpd            %ymm16, %ymm17, %ymm19 {z}{k1}
+vmulpd            (%rax), %ymm17, %ymm19 {z}{k1}
+vmulpd            (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vmulps            %xmm16, %xmm17, %xmm19
+vmulps            (%rax), %xmm17, %xmm19
+vmulps            (%rax){1to4}, %xmm17, %xmm19
+vmulps            %xmm16, %xmm17, %xmm19 {k1}
+vmulps            (%rax), %xmm17, %xmm19 {k1}
+vmulps            (%rax){1to4}, %xmm17, %xmm19 {k1}
+vmulps            %xmm16, %xmm17, %xmm19 {z}{k1}
+vmulps            (%rax), %xmm17, %xmm19 {z}{k1}
+vmulps            (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vmulps            %ymm16, %ymm17, %ymm19
+vmulps            (%rax), %ymm17, %ymm19
+vmulps            (%rax){1to8}, %ymm17, %ymm19
+vmulps            %ymm16, %ymm17, %ymm19 {k1}
+vmulps            (%rax), %ymm17, %ymm19 {k1}
+vmulps            (%rax){1to8}, %ymm17, %ymm19 {k1}
+vmulps            %ymm16, %ymm17, %ymm19 {z}{k1}
+vmulps            (%rax), %ymm17, %ymm19 {z}{k1}
+vmulps            (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vpaddd            %xmm16, %xmm17, %xmm19
+vpaddd            (%rax), %xmm17, %xmm19
+vpaddd            (%rax){1to4}, %xmm17, %xmm19
+vpaddd            %xmm16, %xmm17, %xmm19 {k1}
+vpaddd            (%rax), %xmm17, %xmm19 {k1}
+vpaddd            (%rax){1to4}, %xmm17, %xmm19 {k1}
+vpaddd            %xmm16, %xmm17, %xmm19 {z}{k1}
+vpaddd            (%rax), %xmm17, %xmm19 {z}{k1}
+vpaddd            (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vpaddd            %ymm16, %ymm17, %ymm19
+vpaddd            (%rax), %ymm17, %ymm19
+vpaddd            (%rax){1to8}, %ymm17, %ymm19
+vpaddd            %ymm16, %ymm17, %ymm19 {k1}
+vpaddd            (%rax), %ymm17, %ymm19 {k1}
+vpaddd            (%rax){1to8}, %ymm17, %ymm19 {k1}
+vpaddd            %ymm16, %ymm17, %ymm19 {z}{k1}
+vpaddd            (%rax), %ymm17, %ymm19 {z}{k1}
+vpaddd            (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vpaddq            %xmm16, %xmm17, %xmm19
+vpaddq            (%rax), %xmm17, %xmm19
+vpaddq            (%rax){1to2}, %xmm17, %xmm19
+vpaddq            %xmm16, %xmm17, %xmm19 {k1}
+vpaddq            (%rax), %xmm17, %xmm19 {k1}
+vpaddq            (%rax){1to2}, %xmm17, %xmm19 {k1}
+vpaddq            %xmm16, %xmm17, %xmm19 {z}{k1}
+vpaddq            (%rax), %xmm17, %xmm19 {z}{k1}
+vpaddq            (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+vpaddq            %ymm16, %ymm17, %ymm19
+vpaddq            (%rax), %ymm17, %ymm19
+vpaddq            (%rax){1to4}, %ymm17, %ymm19
+vpaddq            %ymm16, %ymm17, %ymm19 {k1}
+vpaddq            (%rax), %ymm17, %ymm19 {k1}
+vpaddq            (%rax){1to4}, %ymm17, %ymm19 {k1}
+vpaddq            %ymm16, %ymm17, %ymm19 {z}{k1}
+vpaddq            (%rax), %ymm17, %ymm19 {z}{k1}
+vpaddq            (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vpbroadcastd      %xmm16, %xmm19
+vpbroadcastd      (%rax), %xmm19
+vpbroadcastd      %xmm16, %xmm19 {k1}
+vpbroadcastd      (%rax), %xmm19 {k1}
+vpbroadcastd      %xmm16, %xmm19 {z}{k1}
+vpbroadcastd      (%rax), %xmm19 {z}{k1}
+
+vpbroadcastd      %xmm16, %ymm19
+vpbroadcastd      (%rax), %ymm19
+vpbroadcastd      %xmm16, %ymm19 {k1}
+vpbroadcastd      (%rax), %ymm19 {k1}
+vpbroadcastd      %xmm16, %ymm19 {z}{k1}
+vpbroadcastd      (%rax), %ymm19 {z}{k1}
+
+vpbroadcastq      %xmm16, %xmm19
+vpbroadcastq      (%rax), %xmm19
+vpbroadcastq      %xmm16, %xmm19 {k1}
+vpbroadcastq      (%rax), %xmm19 {k1}
+vpbroadcastq      %xmm16, %xmm19 {z}{k1}
+vpbroadcastq      (%rax), %xmm19 {z}{k1}
+
+vpbroadcastq      %xmm16, %ymm19
+vpbroadcastq      (%rax), %ymm19
+vpbroadcastq      %xmm16, %ymm19 {k1}
+vpbroadcastq      (%rax), %ymm19 {k1}
+vpbroadcastq      %xmm16, %ymm19 {z}{k1}
+vpbroadcastq      (%rax), %ymm19 {z}{k1}
+
+vpcmpd            $0, %xmm0, %xmm1, %k2
+vpcmpd            $0, (%rax), %xmm1, %k2
+vpcmpd            $0, (%rax){1to4}, %xmm1, %k2
+vpcmpd            $0, %xmm0, %xmm1, %k2 {k3}
+vpcmpd            $0, (%rax), %xmm1, %k2 {k3}
+vpcmpd            $0, (%rax){1to4}, %xmm1, %k2 {k3}
+
+vpcmpd            $0, %ymm0, %ymm1, %k2
+vpcmpd            $0, (%rax), %ymm1, %k2
+vpcmpd            $0, (%rax){1to8}, %ymm1, %k2
+vpcmpd            $0, %ymm0, %ymm1, %k2 {k3}
+vpcmpd            $0, (%rax), %ymm1, %k2 {k3}
+vpcmpd            $0, (%rax){1to8}, %ymm1, %k2 {k3}
+
+vpcmpeqd          %xmm0, %xmm1, %k2
+vpcmpeqd          (%rax), %xmm1, %k2
+vpcmpeqd          (%rax){1to4}, %xmm1, %k2
+vpcmpeqd          %xmm0, %xmm1, %k2 {k3}
+vpcmpeqd          (%rax), %xmm1, %k2 {k3}
+vpcmpeqd          (%rax){1to4}, %xmm1, %k2 {k3}
+
+vpcmpeqd          %ymm0, %ymm1, %k2
+vpcmpeqd          (%rax), %ymm1, %k2
+vpcmpeqd          (%rax){1to8}, %ymm1, %k2
+vpcmpeqd          %ymm0, %ymm1, %k2 {k3}
+vpcmpeqd          (%rax), %ymm1, %k2 {k3}
+vpcmpeqd          (%rax){1to8}, %ymm1, %k2 {k3}
+
+vpcmpeqq          %xmm0, %xmm1, %k2
+vpcmpeqq          (%rax), %xmm1, %k2
+vpcmpeqq          (%rax){1to2}, %xmm1, %k2
+vpcmpeqq          %xmm0, %xmm1, %k2 {k3}
+vpcmpeqq          (%rax), %xmm1, %k2 {k3}
+vpcmpeqq          (%rax){1to2}, %xmm1, %k2 {k3}
+
+vpcmpeqq          %ymm0, %ymm1, %k2
+vpcmpeqq          (%rax), %ymm1, %k2
+vpcmpeqq          (%rax){1to4}, %ymm1, %k2
+vpcmpeqq          %ymm0, %ymm1, %k2 {k3}
+vpcmpeqq          (%rax), %ymm1, %k2 {k3}
+vpcmpeqq          (%rax){1to4}, %ymm1, %k2 {k3}
+
+vpcmpgtd          %xmm0, %xmm1, %k2
+vpcmpgtd          (%rax), %xmm1, %k2
+vpcmpgtd          (%rax){1to4}, %xmm1, %k2
+vpcmpgtd          %xmm0, %xmm1, %k2 {k3}
+vpcmpgtd          (%rax), %xmm1, %k2 {k3}
+vpcmpgtd          (%rax){1to4}, %xmm1, %k2 {k3}
+
+vpcmpgtd          %ymm0, %ymm1, %k2
+vpcmpgtd          (%rax), %ymm1, %k2
+vpcmpgtd          (%rax){1to8}, %ymm1, %k2
+vpcmpgtd          %ymm0, %ymm1, %k2 {k3}
+vpcmpgtd          (%rax), %ymm1, %k2 {k3}
+vpcmpgtd          (%rax){1to8}, %ymm1, %k2 {k3}
+
+vpcmpgtq          %xmm0, %xmm1, %k2
+vpcmpgtq          (%rax), %xmm1, %k2
+vpcmpgtq          (%rax){1to2}, %xmm1, %k2
+vpcmpgtq          %xmm0, %xmm1, %k2 {k3}
+vpcmpgtq          (%rax), %xmm1, %k2 {k3}
+vpcmpgtq          (%rax){1to2}, %xmm1, %k2 {k3}
+
+vpcmpgtq          %ymm0, %ymm1, %k2
+vpcmpgtq          (%rax), %ymm1, %k2
+vpcmpgtq          (%rax){1to4}, %ymm1, %k2
+vpcmpgtq          %ymm0, %ymm1, %k2 {k3}
+vpcmpgtq          (%rax), %ymm1, %k2 {k3}
+vpcmpgtq          (%rax){1to4}, %ymm1, %k2 {k3}
+
+vpcmpq            $0, %xmm0, %xmm1, %k2
+vpcmpq            $0, (%rax), %xmm1, %k2
+vpcmpq            $0, (%rax){1to2}, %xmm1, %k2
+vpcmpq            $0, %xmm0, %xmm1, %k2 {k3}
+vpcmpq            $0, (%rax), %xmm1, %k2 {k3}
+vpcmpq            $0, (%rax){1to2}, %xmm1, %k2 {k3}
+
+vpcmpq            $0, %ymm0, %ymm1, %k2
+vpcmpq            $0, (%rax), %ymm1, %k2
+vpcmpq            $0, (%rax){1to4}, %ymm1, %k2
+vpcmpq            $0, %ymm0, %ymm1, %k2 {k3}
+vpcmpq            $0, (%rax), %ymm1, %k2 {k3}
+vpcmpq            $0, (%rax){1to4}, %ymm1, %k2 {k3}
+
+vpcmpud           $0, %xmm0, %xmm1, %k2
+vpcmpud           $0, (%rax), %xmm1, %k2
+vpcmpud           $0, (%rax){1to4}, %xmm1, %k2
+vpcmpud           $0, %xmm0, %xmm1, %k2 {k3}
+vpcmpud           $0, (%rax), %xmm1, %k2 {k3}
+vpcmpud           $0, (%rax){1to4}, %xmm1, %k2 {k3}
+
+vpcmpud           $0, %ymm0, %ymm1, %k2
+vpcmpud           $0, (%rax), %ymm1, %k2
+vpcmpud           $0, (%rax){1to8}, %ymm1, %k2
+vpcmpud           $0, %ymm0, %ymm1, %k2 {k3}
+vpcmpud           $0, (%rax), %ymm1, %k2 {k3}
+vpcmpud           $0, (%rax){1to8}, %ymm1, %k2 {k3}
+
+vpcmpuq           $0, %xmm0, %xmm1, %k2
+vpcmpuq           $0, (%rax), %xmm1, %k2
+vpcmpuq           $0, (%rax){1to2}, %xmm1, %k2
+vpcmpuq           $0, %xmm0, %xmm1, %k2 {k3}
+vpcmpuq           $0, (%rax), %xmm1, %k2 {k3}
+vpcmpuq           $0, (%rax){1to2}, %xmm1, %k2 {k3}
+
+vpcmpuq           $0, %ymm0, %ymm1, %k2
+vpcmpuq           $0, (%rax), %ymm1, %k2
+vpcmpuq           $0, (%rax){1to4}, %ymm1, %k2
+vpcmpuq           $0, %ymm0, %ymm1, %k2 {k3}
+vpcmpuq           $0, (%rax), %ymm1, %k2 {k3}
+vpcmpuq           $0, (%rax){1to4}, %ymm1, %k2 {k3}
+
+vpermd            %ymm16, %ymm17, %ymm19
+vpermd            (%rax), %ymm17, %ymm19
+vpermd            (%rax){1to8}, %ymm17, %ymm19
+vpermd            %ymm16, %ymm17, %ymm19 {k1}
+vpermd            (%rax), %ymm17, %ymm19 {k1}
+vpermd            (%rax){1to8}, %ymm17, %ymm19 {k1}
+vpermd            %ymm16, %ymm17, %ymm19 {z}{k1}
+vpermd            (%rax), %ymm17, %ymm19 {z}{k1}
+vpermd            (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vpermilpd         $0, %xmm16, %xmm19
+vpermilpd         $0, (%rax), %xmm19
+vpermilpd         $0, (%rax){1to2}, %xmm19
+vpermilpd         $0, %xmm16, %xmm19 {k1}
+vpermilpd         $0, (%rax), %xmm19 {k1}
+vpermilpd         $0, (%rax){1to2}, %xmm19 {k1}
+vpermilpd         $0, %xmm16, %xmm19 {z}{k1}
+vpermilpd         $0, (%rax), %xmm19 {z}{k1}
+vpermilpd         $0, (%rax){1to2}, %xmm19 {z}{k1}
+
+vpermilpd         $0, %ymm16, %ymm19
+vpermilpd         $0, (%rax), %ymm19
+vpermilpd         $0, (%rax){1to4}, %ymm19
+vpermilpd         $0, %ymm16, %ymm19 {k1}
+vpermilpd         $0, (%rax), %ymm19 {k1}
+vpermilpd         $0, (%rax){1to4}, %ymm19 {k1}
+vpermilpd         $0, %ymm16, %ymm19 {z}{k1}
+vpermilpd         $0, (%rax), %ymm19 {z}{k1}
+vpermilpd         $0, (%rax){1to4}, %ymm19 {z}{k1}
+
+vpermilpd         %xmm16, %xmm17, %xmm19
+vpermilpd         (%rax), %xmm17, %xmm19
+vpermilpd         (%rax){1to2}, %xmm17, %xmm19
+vpermilpd         %xmm16, %xmm17, %xmm19 {k1}
+vpermilpd         (%rax), %xmm17, %xmm19 {k1}
+vpermilpd         (%rax){1to2}, %xmm17, %xmm19 {k1}
+vpermilpd         %xmm16, %xmm17, %xmm19 {z}{k1}
+vpermilpd         (%rax), %xmm17, %xmm19 {z}{k1}
+vpermilpd         (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+vpermilpd         %ymm16, %ymm17, %ymm19
+vpermilpd         (%rax), %ymm17, %ymm19
+vpermilpd         (%rax){1to4}, %ymm17, %ymm19
+vpermilpd         %ymm16, %ymm17, %ymm19 {k1}
+vpermilpd         (%rax), %ymm17, %ymm19 {k1}
+vpermilpd         (%rax){1to4}, %ymm17, %ymm19 {k1}
+vpermilpd         %ymm16, %ymm17, %ymm19 {z}{k1}
+vpermilpd         (%rax), %ymm17, %ymm19 {z}{k1}
+vpermilpd         (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vpermilps         $0, %xmm16, %xmm19
+vpermilps         $0, (%rax), %xmm19
+vpermilps         $0, (%rax){1to4}, %xmm19
+vpermilps         $0, %xmm16, %xmm19 {k1}
+vpermilps         $0, (%rax), %xmm19 {k1}
+vpermilps         $0, (%rax){1to4}, %xmm19 {k1}
+vpermilps         $0, %xmm16, %xmm19 {z}{k1}
+vpermilps         $0, (%rax), %xmm19 {z}{k1}
+vpermilps         $0, (%rax){1to4}, %xmm19 {z}{k1}
+
+vpermilps         $0, %ymm16, %ymm19
+vpermilps         $0, (%rax), %ymm19
+vpermilps         $0, (%rax){1to8}, %ymm19
+vpermilps         $0, %ymm16, %ymm19 {k1}
+vpermilps         $0, (%rax), %ymm19 {k1}
+vpermilps         $0, (%rax){1to8}, %ymm19 {k1}
+vpermilps         $0, %ymm16, %ymm19 {z}{k1}
+vpermilps         $0, (%rax), %ymm19 {z}{k1}
+vpermilps         $0, (%rax){1to8}, %ymm19 {z}{k1}
+
+vpermilps         %xmm16, %xmm17, %xmm19
+vpermilps         (%rax), %xmm17, %xmm19
+vpermilps         (%rax){1to4}, %xmm17, %xmm19
+vpermilps         %xmm16, %xmm17, %xmm19 {k1}
+vpermilps         (%rax), %xmm17, %xmm19 {k1}
+vpermilps         (%rax){1to4}, %xmm17, %xmm19 {k1}
+vpermilps         %xmm16, %xmm17, %xmm19 {z}{k1}
+vpermilps         (%rax), %xmm17, %xmm19 {z}{k1}
+vpermilps         (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vpermilps         %ymm16, %ymm17, %ymm19
+vpermilps         (%rax), %ymm17, %ymm19
+vpermilps         (%rax){1to8}, %ymm17, %ymm19
+vpermilps         %ymm16, %ymm17, %ymm19 {k1}
+vpermilps         (%rax), %ymm17, %ymm19 {k1}
+vpermilps         (%rax){1to8}, %ymm17, %ymm19 {k1}
+vpermilps         %ymm16, %ymm17, %ymm19 {z}{k1}
+vpermilps         (%rax), %ymm17, %ymm19 {z}{k1}
+vpermilps         (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vpermpd           $0, %ymm16, %ymm19
+vpermpd           $0, (%rax), %ymm19
+vpermpd           $0, (%rax){1to4}, %ymm19
+vpermpd           $0, %ymm16, %ymm19 {k1}
+vpermpd           $0, (%rax), %ymm19 {k1}
+vpermpd           $0, (%rax){1to4}, %ymm19 {k1}
+vpermpd           $0, %ymm16, %ymm19 {z}{k1}
+vpermpd           $0, (%rax), %ymm19 {z}{k1}
+vpermpd           $0, (%rax){1to4}, %ymm19 {z}{k1}
+
+vpermpd           %ymm16, %ymm17, %ymm19
+vpermpd           (%rax), %ymm17, %ymm19
+vpermpd           (%rax){1to4}, %ymm17, %ymm19
+vpermpd           %ymm16, %ymm17, %ymm19 {k1}
+vpermpd           (%rax), %ymm17, %ymm19 {k1}
+vpermpd           (%rax){1to4}, %ymm17, %ymm19 {k1}
+vpermpd           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpermpd           (%rax), %ymm17, %ymm19 {z}{k1}
+vpermpd           (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vpermps           %ymm16, %ymm17, %ymm19
+vpermps           (%rax), %ymm17, %ymm19
+vpermps           (%rax){1to8}, %ymm17, %ymm19
+vpermps           %ymm16, %ymm17, %ymm19 {k1}
+vpermps           (%rax), %ymm17, %ymm19 {k1}
+vpermps           (%rax){1to8}, %ymm17, %ymm19 {k1}
+vpermps           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpermps           (%rax), %ymm17, %ymm19 {z}{k1}
+vpermps           (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vpermq            $0, %ymm16, %ymm19
+vpermq            $0, (%rax), %ymm19
+vpermq            $0, (%rax){1to4}, %ymm19
+vpermq            $0, %ymm16, %ymm19 {k1}
+vpermq            $0, (%rax), %ymm19 {k1}
+vpermq            $0, (%rax){1to4}, %ymm19 {k1}
+vpermq            $0, %ymm16, %ymm19 {z}{k1}
+vpermq            $0, (%rax), %ymm19 {z}{k1}
+vpermq            $0, (%rax){1to4}, %ymm19 {z}{k1}
+
+vpermq            %ymm16, %ymm17, %ymm19
+vpermq            (%rax), %ymm17, %ymm19
+vpermq            (%rax){1to4}, %ymm17, %ymm19
+vpermq            %ymm16, %ymm17, %ymm19 {k1}
+vpermq            (%rax), %ymm17, %ymm19 {k1}
+vpermq            (%rax){1to4}, %ymm17, %ymm19 {k1}
+vpermq            %ymm16, %ymm17, %ymm19 {z}{k1}
+vpermq            (%rax), %ymm17, %ymm19 {z}{k1}
+vpermq            (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vpgatherdq        (%rax,%xmm1,2), %ymm2 {k1}
+vpgatherdd        (%rax,%ymm1,2), %ymm2 {k1}
+vpgatherqq        (%rax,%ymm1,2), %ymm2 {k1}
+vpgatherqd        (%rax,%ymm1,2), %xmm2 {k1}
+
+vpgatherdq        (%rax,%xmm1,2), %xmm2 {k1}
+vpgatherdd        (%rax,%xmm1,2), %xmm2 {k1}
+vpgatherqq        (%rax,%xmm1,2), %xmm2 {k1}
+vpgatherqd        (%rax,%xmm1,2), %xmm2 {k1}
+
+vpmovsxbd         %xmm16, %xmm19
+vpmovsxbd         (%rax), %xmm19
+vpmovsxbd         %xmm16, %xmm19 {k1}
+vpmovsxbd         (%rax), %xmm19 {k1}
+vpmovsxbd         %xmm16, %xmm19 {z}{k1}
+vpmovsxbd         (%rax), %xmm19 {z}{k1}
+
+vpmovsxbd         %xmm16, %ymm19
+vpmovsxbd         (%rax), %ymm19
+vpmovsxbd         %xmm16, %ymm19 {k1}
+vpmovsxbd         (%rax), %ymm19 {k1}
+vpmovsxbd         %xmm16, %ymm19 {z}{k1}
+vpmovsxbd         (%rax), %ymm19 {z}{k1}
+
+vpmovsxbq         %xmm16, %xmm19
+vpmovsxbq         (%rax), %xmm19
+vpmovsxbq         %xmm16, %xmm19 {k1}
+vpmovsxbq         (%rax), %xmm19 {k1}
+vpmovsxbq         %xmm16, %xmm19 {z}{k1}
+vpmovsxbq         (%rax), %xmm19 {z}{k1}
+
+vpmovsxbq         %xmm16, %ymm19
+vpmovsxbq         (%rax), %ymm19
+vpmovsxbq         %xmm16, %ymm19 {k1}
+vpmovsxbq         (%rax), %ymm19 {k1}
+vpmovsxbq         %xmm16, %ymm19 {z}{k1}
+vpmovsxbq         (%rax), %ymm19 {z}{k1}
+
+vpmovsxdq         %xmm16, %xmm19
+vpmovsxdq         (%rax), %xmm19
+vpmovsxdq         %xmm16, %xmm19 {k1}
+vpmovsxdq         (%rax), %xmm19 {k1}
+vpmovsxdq         %xmm16, %xmm19 {z}{k1}
+vpmovsxdq         (%rax), %xmm19 {z}{k1}
+
+vpmovsxdq         %xmm16, %ymm19
+vpmovsxdq         (%rax), %ymm19
+vpmovsxdq         %xmm16, %ymm19 {k1}
+vpmovsxdq         (%rax), %ymm19 {k1}
+vpmovsxdq         %xmm16, %ymm19 {z}{k1}
+vpmovsxdq         (%rax), %ymm19 {z}{k1}
+
+vpmovsxwd         %xmm16, %xmm19
+vpmovsxwd         (%rax), %xmm19
+vpmovsxwd         %xmm16, %xmm19 {k1}
+vpmovsxwd         (%rax), %xmm19 {k1}
+vpmovsxwd         %xmm16, %xmm19 {z}{k1}
+vpmovsxwd         (%rax), %xmm19 {z}{k1}
+
+vpmovsxwd         %xmm16, %ymm19
+vpmovsxwd         (%rax), %ymm19
+vpmovsxwd         %xmm16, %ymm19 {k1}
+vpmovsxwd         (%rax), %ymm19 {k1}
+vpmovsxwd         %xmm16, %ymm19 {z}{k1}
+vpmovsxwd         (%rax), %ymm19 {z}{k1}
+
+vpmovsxwq         %xmm16, %xmm19
+vpmovsxwq         (%rax), %xmm19
+vpmovsxwq         %xmm16, %xmm19 {k1}
+vpmovsxwq         (%rax), %xmm19 {k1}
+vpmovsxwq         %xmm16, %xmm19 {z}{k1}
+vpmovsxwq         (%rax), %xmm19 {z}{k1}
+
+vpmovsxwq         %xmm16, %ymm19
+vpmovsxwq         (%rax), %ymm19
+vpmovsxwq         %xmm16, %ymm19 {k1}
+vpmovsxwq         (%rax), %ymm19 {k1}
+vpmovsxwq         %xmm16, %ymm19 {z}{k1}
+vpmovsxwq         (%rax), %ymm19 {z}{k1}
+
+vpmovzxbd         %xmm16, %xmm19
+vpmovzxbd         (%rax), %xmm19
+vpmovzxbd         %xmm16, %xmm19 {k1}
+vpmovzxbd         (%rax), %xmm19 {k1}
+vpmovzxbd         %xmm16, %xmm19 {z}{k1}
+vpmovzxbd         (%rax), %xmm19 {z}{k1}
+
+vpmovzxbd         %xmm16, %ymm19
+vpmovzxbd         (%rax), %ymm19
+vpmovzxbd         %xmm16, %ymm19 {k1}
+vpmovzxbd         (%rax), %ymm19 {k1}
+vpmovzxbd         %xmm16, %ymm19 {z}{k1}
+vpmovzxbd         (%rax), %ymm19 {z}{k1}
+
+vpmovzxbq         %xmm16, %xmm19
+vpmovzxbq         (%rax), %xmm19
+vpmovzxbq         %xmm16, %xmm19 {k1}
+vpmovzxbq         (%rax), %xmm19 {k1}
+vpmovzxbq         %xmm16, %xmm19 {z}{k1}
+vpmovzxbq         (%rax), %xmm19 {z}{k1}
+
+vpmovzxbq         %xmm16, %ymm19
+vpmovzxbq         (%rax), %ymm19
+vpmovzxbq         %xmm16, %ymm19 {k1}
+vpmovzxbq         (%rax), %ymm19 {k1}
+vpmovzxbq         %xmm16, %ymm19 {z}{k1}
+vpmovzxbq         (%rax), %ymm19 {z}{k1}
+
+vpmovzxdq         %xmm16, %xmm19
+vpmovzxdq         (%rax), %xmm19
+vpmovzxdq         %xmm16, %xmm19 {k1}
+vpmovzxdq         (%rax), %xmm19 {k1}
+vpmovzxdq         %xmm16, %xmm19 {z}{k1}
+vpmovzxdq         (%rax), %xmm19 {z}{k1}
+
+vpmovzxdq         %xmm16, %ymm19
+vpmovzxdq         (%rax), %ymm19
+vpmovzxdq         %xmm16, %ymm19 {k1}
+vpmovzxdq         (%rax), %ymm19 {k1}
+vpmovzxdq         %xmm16, %ymm19 {z}{k1}
+vpmovzxdq         (%rax), %ymm19 {z}{k1}
+
+vpmovzxwd         %xmm16, %xmm19
+vpmovzxwd         (%rax), %xmm19
+vpmovzxwd         %xmm16, %xmm19 {k1}
+vpmovzxwd         (%rax), %xmm19 {k1}
+vpmovzxwd         %xmm16, %xmm19 {z}{k1}
+vpmovzxwd         (%rax), %xmm19 {z}{k1}
+
+vpmovzxwd         %xmm16, %ymm19
+vpmovzxwd         (%rax), %ymm19
+vpmovzxwd         %xmm16, %ymm19 {k1}
+vpmovzxwd         (%rax), %ymm19 {k1}
+vpmovzxwd         %xmm16, %ymm19 {z}{k1}
+vpmovzxwd         (%rax), %ymm19 {z}{k1}
+
+vpmovzxwq         %xmm16, %xmm19
+vpmovzxwq         (%rax), %xmm19
+vpmovzxwq         %xmm16, %xmm19 {k1}
+vpmovzxwq         (%rax), %xmm19 {k1}
+vpmovzxwq         %xmm16, %xmm19 {z}{k1}
+vpmovzxwq         (%rax), %xmm19 {z}{k1}
+
+vpmovzxwq         %xmm16, %ymm19
+vpmovzxwq         (%rax), %ymm19
+vpmovzxwq         %xmm16, %ymm19 {k1}
+vpmovzxwq         (%rax), %ymm19 {k1}
+vpmovzxwq         %xmm16, %ymm19 {z}{k1}
+vpmovzxwq         (%rax), %ymm19 {z}{k1}
+
+vpmulld           %xmm16, %xmm17, %xmm19
+vpmulld           (%rax), %xmm17, %xmm19
+vpmulld           (%rax){1to4}, %xmm17, %xmm19
+vpmulld           %xmm16, %xmm17, %xmm19 {k1}
+vpmulld           (%rax), %xmm17, %xmm19 {k1}
+vpmulld           (%rax){1to4}, %xmm17, %xmm19 {k1}
+vpmulld           %xmm16, %xmm17, %xmm19 {z}{k1}
+vpmulld           (%rax), %xmm17, %xmm19 {z}{k1}
+vpmulld           (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vpmulld           %ymm16, %ymm17, %ymm19
+vpmulld           (%rax), %ymm17, %ymm19
+vpmulld           (%rax){1to8}, %ymm17, %ymm19
+vpmulld           %ymm16, %ymm17, %ymm19 {k1}
+vpmulld           (%rax), %ymm17, %ymm19 {k1}
+vpmulld           (%rax){1to8}, %ymm17, %ymm19 {k1}
+vpmulld           %ymm16, %ymm17, %ymm19 {z}{k1}
+vpmulld           (%rax), %ymm17, %ymm19 {z}{k1}
+vpmulld           (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vpshufd           $0, %xmm16, %xmm19
+vpshufd           $0, (%rax), %xmm19
+vpshufd           $0, (%rax){1to4}, %xmm19
+vpshufd           $0, %xmm16, %xmm19 {k1}
+vpshufd           $0, (%rax), %xmm19 {k1}
+vpshufd           $0, (%rax){1to4}, %xmm19 {k1}
+vpshufd           $0, %xmm16, %xmm19 {z}{k1}
+vpshufd           $0, (%rax), %xmm19 {z}{k1}
+vpshufd           $0, (%rax){1to4}, %xmm19 {z}{k1}
+
+vpshufd           $0, %ymm16, %ymm19
+vpshufd           $0, (%rax), %ymm19
+vpshufd           $0, (%rax){1to8}, %ymm19
+vpshufd           $0, %ymm16, %ymm19 {k1}
+vpshufd           $0, (%rax), %ymm19 {k1}
+vpshufd           $0, (%rax){1to8}, %ymm19 {k1}
+vpshufd           $0, %ymm16, %ymm19 {z}{k1}
+vpshufd           $0, (%rax), %ymm19 {z}{k1}
+vpshufd           $0, (%rax){1to8}, %ymm19 {z}{k1}
+
+vpsubd            %xmm16, %xmm17, %xmm19
+vpsubd            (%rax), %xmm17, %xmm19
+vpsubd            (%rax){1to4}, %xmm17, %xmm19
+vpsubd            %xmm16, %xmm17, %xmm19 {k1}
+vpsubd            (%rax), %xmm17, %xmm19 {k1}
+vpsubd            (%rax){1to4}, %xmm17, %xmm19 {k1}
+vpsubd            %xmm16, %xmm17, %xmm19 {z}{k1}
+vpsubd            (%rax), %xmm17, %xmm19 {z}{k1}
+vpsubd            (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vpsubd            %ymm16, %ymm17, %ymm19
+vpsubd            (%rax), %ymm17, %ymm19
+vpsubd            (%rax){1to8}, %ymm17, %ymm19
+vpsubd            %ymm16, %ymm17, %ymm19 {k1}
+vpsubd            (%rax), %ymm17, %ymm19 {k1}
+vpsubd            (%rax){1to8}, %ymm17, %ymm19 {k1}
+vpsubd            %ymm16, %ymm17, %ymm19 {z}{k1}
+vpsubd            (%rax), %ymm17, %ymm19 {z}{k1}
+vpsubd            (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vpsubq            %xmm16, %xmm17, %xmm19
+vpsubq            (%rax), %xmm17, %xmm19
+vpsubq            (%rax){1to2}, %xmm17, %xmm19
+vpsubq            %xmm16, %xmm17, %xmm19 {k1}
+vpsubq            (%rax), %xmm17, %xmm19 {k1}
+vpsubq            (%rax){1to2}, %xmm17, %xmm19 {k1}
+vpsubq            %xmm16, %xmm17, %xmm19 {z}{k1}
+vpsubq            (%rax), %xmm17, %xmm19 {z}{k1}
+vpsubq            (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+vpsubq            %ymm16, %ymm17, %ymm19
+vpsubq            (%rax), %ymm17, %ymm19
+vpsubq            (%rax){1to4}, %ymm17, %ymm19
+vpsubq            %ymm16, %ymm17, %ymm19 {k1}
+vpsubq            (%rax), %ymm17, %ymm19 {k1}
+vpsubq            (%rax){1to4}, %ymm17, %ymm19 {k1}
+vpsubq            %ymm16, %ymm17, %ymm19 {z}{k1}
+vpsubq            (%rax), %ymm17, %ymm19 {z}{k1}
+vpsubq            (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vptestmd          %xmm0, %xmm1, %k2
+vptestmd          (%rax), %xmm1, %k2
+vptestmd          (%rax){1to4}, %xmm1, %k2
+vptestmd          %xmm0, %xmm1, %k2 {k3}
+vptestmd          (%rax), %xmm1, %k2 {k3}
+vptestmd          (%rax){1to4}, %xmm1, %k2 {k3}
+
+vptestmd          %ymm0, %ymm1, %k2
+vptestmd          (%rax), %ymm1, %k2
+vptestmd          (%rax){1to8}, %ymm1, %k2
+vptestmd          %ymm0, %ymm1, %k2 {k3}
+vptestmd          (%rax), %ymm1, %k2 {k3}
+vptestmd          (%rax){1to8}, %ymm1, %k2 {k3}
+
+vptestmq          %xmm0, %xmm1, %k2
+vptestmq          (%rax), %xmm1, %k2
+vptestmq          (%rax){1to2}, %xmm1, %k2
+vptestmq          %xmm0, %xmm1, %k2 {k3}
+vptestmq          (%rax), %xmm1, %k2 {k3}
+vptestmq          (%rax){1to2}, %xmm1, %k2 {k3}
+
+vptestmq          %ymm0, %ymm1, %k2
+vptestmq          (%rax), %ymm1, %k2
+vptestmq          (%rax){1to4}, %ymm1, %k2
+vptestmq          %ymm0, %ymm1, %k2 {k3}
+vptestmq          (%rax), %ymm1, %k2 {k3}
+vptestmq          (%rax){1to4}, %ymm1, %k2 {k3}
+
+vptestnmd         %xmm0, %xmm1, %k2
+vptestnmd         (%rax), %xmm1, %k2
+vptestnmd         (%rax){1to4}, %xmm1, %k2
+vptestnmd         %xmm0, %xmm1, %k2 {k3}
+vptestnmd         (%rax), %xmm1, %k2 {k3}
+vptestnmd         (%rax){1to4}, %xmm1, %k2 {k3}
+
+vptestnmd         %ymm0, %ymm1, %k2
+vptestnmd         (%rax), %ymm1, %k2
+vptestnmd         (%rax){1to8}, %ymm1, %k2
+vptestnmd         %ymm0, %ymm1, %k2 {k3}
+vptestnmd         (%rax), %ymm1, %k2 {k3}
+vptestnmd         (%rax){1to8}, %ymm1, %k2 {k3}
+
+vptestnmq         %xmm0, %xmm1, %k2
+vptestnmq         (%rax), %xmm1, %k2
+vptestnmq         (%rax){1to2}, %xmm1, %k2
+vptestnmq         %xmm0, %xmm1, %k2 {k3}
+vptestnmq         (%rax), %xmm1, %k2 {k3}
+vptestnmq         (%rax){1to2}, %xmm1, %k2 {k3}
+
+vptestnmq         %ymm0, %ymm1, %k2
+vptestnmq         (%rax), %ymm1, %k2
+vptestnmq         (%rax){1to4}, %ymm1, %k2
+vptestnmq         %ymm0, %ymm1, %k2 {k3}
+vptestnmq         (%rax), %ymm1, %k2 {k3}
+vptestnmq         (%rax){1to4}, %ymm1, %k2 {k3}
+
+vpunpckhdq        %xmm16, %xmm17, %xmm19
+vpunpckhdq        (%rax), %xmm17, %xmm19
+vpunpckhdq        (%rax){1to4}, %xmm17, %xmm19
+vpunpckhdq        %xmm16, %xmm17, %xmm19 {k1}
+vpunpckhdq        (%rax), %xmm17, %xmm19 {k1}
+vpunpckhdq        (%rax){1to4}, %xmm17, %xmm19 {k1}
+vpunpckhdq        %xmm16, %xmm17, %xmm19 {z}{k1}
+vpunpckhdq        (%rax), %xmm17, %xmm19 {z}{k1}
+vpunpckhdq        (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vpunpckhdq        %ymm16, %ymm17, %ymm19
+vpunpckhdq        (%rax), %ymm17, %ymm19
+vpunpckhdq        (%rax){1to8}, %ymm17, %ymm19
+vpunpckhdq        %ymm16, %ymm17, %ymm19 {k1}
+vpunpckhdq        (%rax), %ymm17, %ymm19 {k1}
+vpunpckhdq        (%rax){1to8}, %ymm17, %ymm19 {k1}
+vpunpckhdq        %ymm16, %ymm17, %ymm19 {z}{k1}
+vpunpckhdq        (%rax), %ymm17, %ymm19 {z}{k1}
+vpunpckhdq        (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vpunpckldq        %xmm16, %xmm17, %xmm19
+vpunpckldq        (%rax), %xmm17, %xmm19
+vpunpckldq        (%rax){1to4}, %xmm17, %xmm19
+vpunpckldq        %xmm16, %xmm17, %xmm19 {k1}
+vpunpckldq        (%rax), %xmm17, %xmm19 {k1}
+vpunpckldq        (%rax){1to4}, %xmm17, %xmm19 {k1}
+vpunpckldq        %xmm16, %xmm17, %xmm19 {z}{k1}
+vpunpckldq        (%rax), %xmm17, %xmm19 {z}{k1}
+vpunpckldq        (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vpunpckldq        %ymm16, %ymm17, %ymm19
+vpunpckldq        (%rax), %ymm17, %ymm19
+vpunpckldq        (%rax){1to8}, %ymm17, %ymm19
+vpunpckldq        %ymm16, %ymm17, %ymm19 {k1}
+vpunpckldq        (%rax), %ymm17, %ymm19 {k1}
+vpunpckldq        (%rax){1to8}, %ymm17, %ymm19 {k1}
+vpunpckldq        %ymm16, %ymm17, %ymm19 {z}{k1}
+vpunpckldq        (%rax), %ymm17, %ymm19 {z}{k1}
+vpunpckldq        (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vshuff32x4        $0, %ymm16, %ymm17, %ymm19
+vshuff32x4        $0, (%rax), %ymm17, %ymm19
+vshuff32x4        $0, (%rax){1to8}, %ymm17, %ymm19
+vshuff32x4        $0, %ymm16, %ymm17, %ymm19 {k1}
+vshuff32x4        $0, (%rax), %ymm17, %ymm19 {k1}
+vshuff32x4        $0, (%rax){1to8}, %ymm17, %ymm19 {k1}
+vshuff32x4        $0, %ymm16, %ymm17, %ymm19 {z}{k1}
+vshuff32x4        $0, (%rax), %ymm17, %ymm19 {z}{k1}
+vshuff32x4        $0, (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vshuff64x2        $0, %ymm16, %ymm17, %ymm19
+vshuff64x2        $0, (%rax), %ymm17, %ymm19
+vshuff64x2        $0, (%rax){1to4}, %ymm17, %ymm19
+vshuff64x2        $0, %ymm16, %ymm17, %ymm19 {k1}
+vshuff64x2        $0, (%rax), %ymm17, %ymm19 {k1}
+vshuff64x2        $0, (%rax){1to4}, %ymm17, %ymm19 {k1}
+vshuff64x2        $0, %ymm16, %ymm17, %ymm19 {z}{k1}
+vshuff64x2        $0, (%rax), %ymm17, %ymm19 {z}{k1}
+vshuff64x2        $0, (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vshufi32x4        $0, %ymm16, %ymm17, %ymm19
+vshufi32x4        $0, (%rax), %ymm17, %ymm19
+vshufi32x4        $0, (%rax){1to8}, %ymm17, %ymm19
+vshufi32x4        $0, %ymm16, %ymm17, %ymm19 {k1}
+vshufi32x4        $0, (%rax), %ymm17, %ymm19 {k1}
+vshufi32x4        $0, (%rax){1to8}, %ymm17, %ymm19 {k1}
+vshufi32x4        $0, %ymm16, %ymm17, %ymm19 {z}{k1}
+vshufi32x4        $0, (%rax), %ymm17, %ymm19 {z}{k1}
+vshufi32x4        $0, (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vshufi64x2        $0, %ymm16, %ymm17, %ymm19
+vshufi64x2        $0, (%rax), %ymm17, %ymm19
+vshufi64x2        $0, (%rax){1to4}, %ymm17, %ymm19
+vshufi64x2        $0, %ymm16, %ymm17, %ymm19 {k1}
+vshufi64x2        $0, (%rax), %ymm17, %ymm19 {k1}
+vshufi64x2        $0, (%rax){1to4}, %ymm17, %ymm19 {k1}
+vshufi64x2        $0, %ymm16, %ymm17, %ymm19 {z}{k1}
+vshufi64x2        $0, (%rax), %ymm17, %ymm19 {z}{k1}
+vshufi64x2        $0, (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vsqrtpd           %xmm16, %xmm19
+vsqrtpd           (%rax), %xmm19
+vsqrtpd           (%rax){1to2}, %xmm19
+vsqrtpd           %xmm16, %xmm19 {k1}
+vsqrtpd           (%rax), %xmm19 {k1}
+vsqrtpd           (%rax){1to2}, %xmm19 {k1}
+vsqrtpd           %xmm16, %xmm19 {z}{k1}
+vsqrtpd           (%rax), %xmm19 {z}{k1}
+vsqrtpd           (%rax){1to2}, %xmm19 {z}{k1}
+
+vsqrtpd           %ymm16, %ymm19
+vsqrtpd           (%rax), %ymm19
+vsqrtpd           (%rax){1to4}, %ymm19
+vsqrtpd           %ymm16, %ymm19 {k1}
+vsqrtpd           (%rax), %ymm19 {k1}
+vsqrtpd           (%rax){1to4}, %ymm19 {k1}
+vsqrtpd           %ymm16, %ymm19 {z}{k1}
+vsqrtpd           (%rax), %ymm19 {z}{k1}
+vsqrtpd           (%rax){1to4}, %ymm19 {z}{k1}
+
+vsqrtps           %xmm16, %xmm19
+vsqrtps           (%rax), %xmm19
+vsqrtps           (%rax){1to4}, %xmm19
+vsqrtps           %xmm16, %xmm19 {k1}
+vsqrtps           (%rax), %xmm19 {k1}
+vsqrtps           (%rax){1to4}, %xmm19 {k1}
+vsqrtps           %xmm16, %xmm19 {z}{k1}
+vsqrtps           (%rax), %xmm19 {z}{k1}
+vsqrtps           (%rax){1to4}, %xmm19 {z}{k1}
+
+vsqrtps           %ymm16, %ymm19
+vsqrtps           (%rax), %ymm19
+vsqrtps           (%rax){1to8}, %ymm19
+vsqrtps           %ymm16, %ymm19 {k1}
+vsqrtps           (%rax), %ymm19 {k1}
+vsqrtps           (%rax){1to8}, %ymm19 {k1}
+vsqrtps           %ymm16, %ymm19 {z}{k1}
+vsqrtps           (%rax), %ymm19 {z}{k1}
+vsqrtps           (%rax){1to8}, %ymm19 {z}{k1}
+
+vsubpd            %xmm16, %xmm17, %xmm19
+vsubpd            (%rax), %xmm17, %xmm19
+vsubpd            (%rax){1to2}, %xmm17, %xmm19
+vsubpd            %xmm16, %xmm17, %xmm19 {k1}
+vsubpd            (%rax), %xmm17, %xmm19 {k1}
+vsubpd            (%rax){1to2}, %xmm17, %xmm19 {k1}
+vsubpd            %xmm16, %xmm17, %xmm19 {z}{k1}
+vsubpd            (%rax), %xmm17, %xmm19 {z}{k1}
+vsubpd            (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+vsubpd            %ymm16, %ymm17, %ymm19
+vsubpd            (%rax), %ymm17, %ymm19
+vsubpd            (%rax){1to4}, %ymm17, %ymm19
+vsubpd            %ymm16, %ymm17, %ymm19 {k1}
+vsubpd            (%rax), %ymm17, %ymm19 {k1}
+vsubpd            (%rax){1to4}, %ymm17, %ymm19 {k1}
+vsubpd            %ymm16, %ymm17, %ymm19 {z}{k1}
+vsubpd            (%rax), %ymm17, %ymm19 {z}{k1}
+vsubpd            (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vsubps            %xmm16, %xmm17, %xmm19
+vsubps            (%rax), %xmm17, %xmm19
+vsubps            (%rax){1to4}, %xmm17, %xmm19
+vsubps            %xmm16, %xmm17, %xmm19 {k1}
+vsubps            (%rax), %xmm17, %xmm19 {k1}
+vsubps            (%rax){1to4}, %xmm17, %xmm19 {k1}
+vsubps            %xmm16, %xmm17, %xmm19 {z}{k1}
+vsubps            (%rax), %xmm17, %xmm19 {z}{k1}
+vsubps            (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vsubps            %ymm16, %ymm17, %ymm19
+vsubps            (%rax), %ymm17, %ymm19
+vsubps            (%rax){1to8}, %ymm17, %ymm19
+vsubps            %ymm16, %ymm17, %ymm19 {k1}
+vsubps            (%rax), %ymm17, %ymm19 {k1}
+vsubps            (%rax){1to8}, %ymm17, %ymm19 {k1}
+vsubps            %ymm16, %ymm17, %ymm19 {z}{k1}
+vsubps            (%rax), %ymm17, %ymm19 {z}{k1}
+vsubps            (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vunpckhpd         %xmm16, %xmm17, %xmm19
+vunpckhpd         (%rax), %xmm17, %xmm19
+vunpckhpd         (%rax){1to2}, %xmm17, %xmm19
+vunpckhpd         %xmm16, %xmm17, %xmm19 {k1}
+vunpckhpd         (%rax), %xmm17, %xmm19 {k1}
+vunpckhpd         (%rax){1to2}, %xmm17, %xmm19 {k1}
+vunpckhpd         %xmm16, %xmm17, %xmm19 {z}{k1}
+vunpckhpd         (%rax), %xmm17, %xmm19 {z}{k1}
+vunpckhpd         (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+vunpckhpd         %ymm16, %ymm17, %ymm19
+vunpckhpd         (%rax), %ymm17, %ymm19
+vunpckhpd         (%rax){1to4}, %ymm17, %ymm19
+vunpckhpd         %ymm16, %ymm17, %ymm19 {k1}
+vunpckhpd         (%rax), %ymm17, %ymm19 {k1}
+vunpckhpd         (%rax){1to4}, %ymm17, %ymm19 {k1}
+vunpckhpd         %ymm16, %ymm17, %ymm19 {z}{k1}
+vunpckhpd         (%rax), %ymm17, %ymm19 {z}{k1}
+vunpckhpd         (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vunpckhps         %xmm16, %xmm17, %xmm19
+vunpckhps         (%rax), %xmm17, %xmm19
+vunpckhps         (%rax){1to4}, %xmm17, %xmm19
+vunpckhps         %xmm16, %xmm17, %xmm19 {k1}
+vunpckhps         (%rax), %xmm17, %xmm19 {k1}
+vunpckhps         (%rax){1to4}, %xmm17, %xmm19 {k1}
+vunpckhps         %xmm16, %xmm17, %xmm19 {z}{k1}
+vunpckhps         (%rax), %xmm17, %xmm19 {z}{k1}
+vunpckhps         (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vunpckhps         %ymm16, %ymm17, %ymm19
+vunpckhps         (%rax), %ymm17, %ymm19
+vunpckhps         (%rax){1to8}, %ymm17, %ymm19
+vunpckhps         %ymm16, %ymm17, %ymm19 {k1}
+vunpckhps         (%rax), %ymm17, %ymm19 {k1}
+vunpckhps         (%rax){1to8}, %ymm17, %ymm19 {k1}
+vunpckhps         %ymm16, %ymm17, %ymm19 {z}{k1}
+vunpckhps         (%rax), %ymm17, %ymm19 {z}{k1}
+vunpckhps         (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vunpcklpd         %xmm16, %xmm17, %xmm19
+vunpcklpd         (%rax), %xmm17, %xmm19
+vunpcklpd         (%rax){1to2}, %xmm17, %xmm19
+vunpcklpd         %xmm16, %xmm17, %xmm19 {k1}
+vunpcklpd         (%rax), %xmm17, %xmm19 {k1}
+vunpcklpd         (%rax){1to2}, %xmm17, %xmm19 {k1}
+vunpcklpd         %xmm16, %xmm17, %xmm19 {z}{k1}
+vunpcklpd         (%rax), %xmm17, %xmm19 {z}{k1}
+vunpcklpd         (%rax){1to2}, %xmm17, %xmm19 {z}{k1}
+
+vunpcklpd         %ymm16, %ymm17, %ymm19
+vunpcklpd         (%rax), %ymm17, %ymm19
+vunpcklpd         (%rax){1to4}, %ymm17, %ymm19
+vunpcklpd         %ymm16, %ymm17, %ymm19 {k1}
+vunpcklpd         (%rax), %ymm17, %ymm19 {k1}
+vunpcklpd         (%rax){1to4}, %ymm17, %ymm19 {k1}
+vunpcklpd         %ymm16, %ymm17, %ymm19 {z}{k1}
+vunpcklpd         (%rax), %ymm17, %ymm19 {z}{k1}
+vunpcklpd         (%rax){1to4}, %ymm17, %ymm19 {z}{k1}
+
+vunpcklps         %xmm16, %xmm17, %xmm19
+vunpcklps         (%rax), %xmm17, %xmm19
+vunpcklps         (%rax){1to4}, %xmm17, %xmm19
+vunpcklps         %xmm16, %xmm17, %xmm19 {k1}
+vunpcklps         (%rax), %xmm17, %xmm19 {k1}
+vunpcklps         (%rax){1to4}, %xmm17, %xmm19 {k1}
+vunpcklps         %xmm16, %xmm17, %xmm19 {z}{k1}
+vunpcklps         (%rax), %xmm17, %xmm19 {z}{k1}
+vunpcklps         (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vunpcklps         %ymm16, %ymm17, %ymm19
+vunpcklps         (%rax), %ymm17, %ymm19
+vunpcklps         (%rax){1to8}, %ymm17, %ymm19
+vunpcklps         %ymm16, %ymm17, %ymm19 {k1}
+vunpcklps         (%rax), %ymm17, %ymm19 {k1}
+vunpcklps         (%rax){1to8}, %ymm17, %ymm19 {k1}
+vunpcklps         %ymm16, %ymm17, %ymm19 {z}{k1}
+vunpcklps         (%rax), %ymm17, %ymm19 {z}{k1}
+vunpcklps         (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      3     0.50                        vaddpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vaddpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vaddpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vaddpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vaddpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vaddpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vaddpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vaddpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vaddpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vaddpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vaddpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vaddpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vaddpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vaddpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vaddpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vaddpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vaddpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vaddpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vaddps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vaddps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vaddps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vaddps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vaddps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vaddps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vaddps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vaddps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vaddps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vaddps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vaddps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vaddps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vaddps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vaddps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vaddps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vaddps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vaddps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vaddps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      4     0.50                        valignd	$1, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   valignd	$1, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   valignd	$1, (%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        valignd	$1, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   valignd	$1, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   valignd	$1, (%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        valignd	$1, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   valignd	$1, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   valignd	$1, (%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     0.50                        valignd	$1, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   valignd	$1, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   valignd	$1, (%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.50                        valignd	$1, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   valignd	$1, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   valignd	$1, (%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        valignd	$1, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   valignd	$1, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   valignd	$1, (%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      4     0.50                        valignq	$1, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   valignq	$1, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   valignq	$1, (%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        valignq	$1, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   valignq	$1, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   valignq	$1, (%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        valignq	$1, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   valignq	$1, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   valignq	$1, (%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     0.50                        valignq	$1, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   valignq	$1, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   valignq	$1, (%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.50                        valignq	$1, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   valignq	$1, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   valignq	$1, (%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        valignq	$1, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   valignq	$1, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   valignq	$1, (%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf32x4	(%rax), %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf32x4	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcastf32x4	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti32x4	(%rax), %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti32x4	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vbroadcasti32x4	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vbroadcastsd	%xmm16, %ymm19
+# CHECK-NEXT:  3      9     1.00    *                   vbroadcastsd	(%rax), %ymm19
+# CHECK-NEXT:  1      2     1.00                        vbroadcastsd	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vbroadcastsd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vbroadcastsd	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vbroadcastsd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vbroadcastss	%xmm16, %xmm19
+# CHECK-NEXT:  3      9     1.00    *                   vbroadcastss	(%rax), %xmm19
+# CHECK-NEXT:  1      2     1.00                        vbroadcastss	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vbroadcastss	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vbroadcastss	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vbroadcastss	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vbroadcastss	%xmm16, %ymm19
+# CHECK-NEXT:  3      9     1.00    *                   vbroadcastss	(%rax), %ymm19
+# CHECK-NEXT:  1      2     1.00                        vbroadcastss	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vbroadcastss	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vbroadcastss	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vbroadcastss	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vcmpeqpd	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vcmpeqpd	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vcmpeqpd	(%rax){1to2}, %xmm1, %k2
+# CHECK-NEXT:  1      1     0.50                        vcmpeqpd	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vcmpeqpd	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vcmpeqpd	(%rax){1to2}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      2     1.00                        vcmpeqpd	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      9     1.00    *                   vcmpeqpd	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      9     1.00    *                   vcmpeqpd	(%rax){1to4}, %ymm1, %k2
+# CHECK-NEXT:  1      2     1.00                        vcmpeqpd	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      9     1.00    *                   vcmpeqpd	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      9     1.00    *                   vcmpeqpd	(%rax){1to4}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vcmpeqps	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vcmpeqps	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vcmpeqps	(%rax){1to4}, %xmm1, %k2
+# CHECK-NEXT:  1      1     0.50                        vcmpeqps	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vcmpeqps	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vcmpeqps	(%rax){1to4}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      2     1.00                        vcmpeqps	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      9     1.00    *                   vcmpeqps	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      9     1.00    *                   vcmpeqps	(%rax){1to8}, %ymm1, %k2
+# CHECK-NEXT:  1      2     1.00                        vcmpeqps	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      9     1.00    *                   vcmpeqps	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      9     1.00    *                   vcmpeqps	(%rax){1to8}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      3     0.50                        vcvtdq2pd	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2pd	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2pd	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vcvtdq2pd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2pd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2pd	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvtdq2pd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2pd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2pd	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      3     1.00                        vcvtdq2pd	%xmm16, %ymm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvtdq2pd	(%rax), %ymm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvtdq2pd	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  2      3     1.00                        vcvtdq2pd	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtdq2pd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtdq2pd	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  2      3     1.00                        vcvtdq2pd	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtdq2pd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtdq2pd	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vcvtdq2ps	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2ps	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2ps	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vcvtdq2ps	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2ps	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2ps	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvtdq2ps	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2ps	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2ps	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vcvtdq2ps	%ymm16, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2ps	(%rax), %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2ps	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vcvtdq2ps	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2ps	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2ps	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvtdq2ps	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2ps	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtdq2ps	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      3     1.00                        vcvtpd2dq	%ymm16, %xmm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2dqy	(%rax), %xmm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2dq	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  2      3     1.00                        vcvtpd2dq	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2dqy	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2dq	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  2      3     1.00                        vcvtpd2dq	%ymm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2dqy	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2dq	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vcvtpd2dq	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2dqx	(%rax), %xmm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2dq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     1.00                        vcvtpd2dq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2dqx	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2dq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vcvtpd2dq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2dqx	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2dq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      6     1.00                        vcvtpd2ps	%ymm16, %xmm19
+# CHECK-NEXT:  2      13    1.00    *                   vcvtpd2psy	(%rax), %xmm19
+# CHECK-NEXT:  2      13    1.00    *                   vcvtpd2ps	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  2      6     1.00                        vcvtpd2ps	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  2      13    1.00    *                   vcvtpd2psy	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  2      13    1.00    *                   vcvtpd2ps	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  2      6     1.00                        vcvtpd2ps	%ymm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      13    1.00    *                   vcvtpd2psy	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      13    1.00    *                   vcvtpd2ps	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vcvtpd2ps	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtpd2psx	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtpd2ps	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vcvtpd2ps	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtpd2psx	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtpd2ps	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvtpd2ps	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtpd2psx	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtpd2ps	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      3     1.00                        vcvtpd2udq	%ymm16, %xmm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2udqy	(%rax), %xmm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2udq	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  2      3     1.00                        vcvtpd2udq	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2udqy	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2udq	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  2      3     1.00                        vcvtpd2udq	%ymm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2udqy	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvtpd2udq	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vcvtpd2udq	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2udqx	(%rax), %xmm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2udq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     1.00                        vcvtpd2udq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2udqx	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2udq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vcvtpd2udq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2udqx	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvtpd2udq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vcvtps2dq	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2dq	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2dq	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vcvtps2dq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2dq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2dq	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvtps2dq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2dq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2dq	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     0.50                        vcvtps2dq	%ymm16, %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2dq	(%rax), %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2dq	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  1      4     0.50                        vcvtps2dq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2dq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2dq	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     0.50                        vcvtps2dq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2dq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2dq	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vcvtps2pd	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2pd	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2pd	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vcvtps2pd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2pd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2pd	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvtps2pd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2pd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2pd	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      4     1.00                        vcvtps2pd	%xmm16, %ymm19
+# CHECK-NEXT:  1      11    1.00    *                   vcvtps2pd	(%rax), %ymm19
+# CHECK-NEXT:  1      11    1.00    *                   vcvtps2pd	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  2      4     1.00                        vcvtps2pd	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    1.00    *                   vcvtps2pd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      11    1.00    *                   vcvtps2pd	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  2      4     1.00                        vcvtps2pd	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.00    *                   vcvtps2pd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.00    *                   vcvtps2pd	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vcvtps2udq	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2udq	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2udq	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vcvtps2udq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2udq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2udq	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvtps2udq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2udq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvtps2udq	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     0.50                        vcvtps2udq	%ymm16, %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2udq	(%rax), %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2udq	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  1      4     0.50                        vcvtps2udq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2udq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2udq	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     0.50                        vcvtps2udq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2udq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    0.50    *                   vcvtps2udq	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      3     1.00                        vcvttpd2dq	%ymm16, %xmm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2dqy	(%rax), %xmm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2dq	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  2      3     1.00                        vcvttpd2dq	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2dqy	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2dq	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  2      3     1.00                        vcvttpd2dq	%ymm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2dqy	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2dq	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vcvttpd2dq	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2dqx	(%rax), %xmm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2dq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     1.00                        vcvttpd2dq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2dqx	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2dq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vcvttpd2dq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2dqx	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2dq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vcvttps2dq	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2dq	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2dq	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vcvttps2dq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2dq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2dq	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvttps2dq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2dq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2dq	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     0.50                        vcvttps2dq	%ymm16, %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2dq	(%rax), %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2dq	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  1      4     0.50                        vcvttps2dq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2dq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2dq	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     0.50                        vcvttps2dq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2dq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2dq	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      3     1.00                        vcvttpd2udq	%ymm16, %xmm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2udqy	(%rax), %xmm19
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2udq	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  2      3     1.00                        vcvttpd2udq	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2udqy	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2udq	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  2      3     1.00                        vcvttpd2udq	%ymm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2udqy	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  2      10    1.00    *                   vcvttpd2udq	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     1.00                        vcvttpd2udq	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2udqx	(%rax), %xmm19
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2udq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      3     1.00                        vcvttpd2udq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2udqx	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2udq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     1.00                        vcvttpd2udq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2udqx	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vcvttpd2udq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vcvttps2udq	%xmm16, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2udq	(%rax), %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2udq	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vcvttps2udq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2udq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2udq	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vcvttps2udq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2udq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vcvttps2udq	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     0.50                        vcvttps2udq	%ymm16, %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2udq	(%rax), %ymm19
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2udq	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  1      4     0.50                        vcvttps2udq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2udq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2udq	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     0.50                        vcvttps2udq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2udq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    0.50    *                   vcvttps2udq	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      13    5.00                        vdivpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      20    5.00    *                   vdivpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      20    5.00    *                   vdivpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      13    5.00                        vdivpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      20    5.00    *                   vdivpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      20    5.00    *                   vdivpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      13    5.00                        vdivpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      20    5.00    *                   vdivpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      20    5.00    *                   vdivpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      13    5.00                        vdivpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      20    5.00    *                   vdivpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      20    5.00    *                   vdivpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      13    5.00                        vdivpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      20    5.00    *                   vdivpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      20    5.00    *                   vdivpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      13    5.00                        vdivpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      20    5.00    *                   vdivpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      20    5.00    *                   vdivpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    3.00                        vdivps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      18    3.00    *                   vdivps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      18    3.00    *                   vdivps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      11    3.00                        vdivps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      18    3.00    *                   vdivps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      18    3.00    *                   vdivps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      11    3.00                        vdivps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      18    3.00    *                   vdivps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      18    3.00    *                   vdivps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      11    3.00                        vdivps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      18    3.00    *                   vdivps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      18    3.00    *                   vdivps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      11    3.00                        vdivps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      18    3.00    *                   vdivps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      18    3.00    *                   vdivps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    3.00                        vdivps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      18    3.00    *                   vdivps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      18    3.00    *                   vdivps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      5     0.33    *                   vgatherdpd	(%rax,%xmm1,2), %ymm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vgatherdps	(%rax,%ymm1,2), %ymm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vgatherqpd	(%rax,%ymm1,2), %ymm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vgatherqps	(%rax,%ymm1,2), %xmm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vgatherdpd	(%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vgatherdps	(%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vgatherqpd	(%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vgatherqps	(%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vmaxpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmaxpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmaxpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vmaxpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmaxpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmaxpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vmaxpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmaxpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmaxpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vmaxpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      9     1.00    *                   vmaxpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      9     1.00    *                   vmaxpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vmaxpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      9     1.00    *                   vmaxpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      9     1.00    *                   vmaxpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vmaxpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      9     1.00    *                   vmaxpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      9     1.00    *                   vmaxpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vmaxps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmaxps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmaxps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vmaxps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmaxps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmaxps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vmaxps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmaxps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmaxps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vmaxps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      9     1.00    *                   vmaxps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      9     1.00    *                   vmaxps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vmaxps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      9     1.00    *                   vmaxps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      9     1.00    *                   vmaxps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vmaxps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      9     1.00    *                   vmaxps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      9     1.00    *                   vmaxps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vminpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vminpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vminpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      2     1.00                        vminpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vminpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vminpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vminpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vminpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vminpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vminpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      9     1.00    *                   vminpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      9     1.00    *                   vminpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vminpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      9     1.00    *                   vminpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      9     1.00    *                   vminpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vminpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      9     1.00    *                   vminpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      9     1.00    *                   vminpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vminps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vminps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vminps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vminps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vminps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vminps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vminps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vminps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vminps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vminps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      9     1.00    *                   vminps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      9     1.00    *                   vminps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vminps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      9     1.00    *                   vminps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      9     1.00    *                   vminps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vminps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      9     1.00    *                   vminps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      9     1.00    *                   vminps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovapd	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovapd	(%rax), %xmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovapd	%xmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovapd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovapd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovapd	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovapd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovapd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovapd	%ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovapd	(%rax), %ymm19
+# CHECK-NEXT:  1      1     1.00           *            vmovapd	%ymm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovapd	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovapd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovapd	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovapd	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovapd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovaps	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovaps	(%rax), %xmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovaps	%xmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovaps	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovaps	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovaps	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovaps	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovaps	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovaps	%ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovaps	(%rax), %ymm19
+# CHECK-NEXT:  1      1     1.00           *            vmovaps	%ymm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovaps	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovaps	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovaps	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovaps	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovaps	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vmovddup	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovddup	(%rax), %xmm19
+# CHECK-NEXT:  1      2     1.00                        vmovddup	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovddup	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vmovddup	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovddup	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovdqa32	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa32	(%rax), %xmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovdqa32	%xmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovdqa32	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa32	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovdqa32	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovdqa32	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa32	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovdqa32	%ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa32	(%rax), %ymm19
+# CHECK-NEXT:  1      1     1.00           *            vmovdqa32	%ymm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovdqa32	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa32	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovdqa32	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovdqa32	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa32	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovdqa64	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa64	(%rax), %xmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovdqa64	%xmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovdqa64	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa64	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovdqa64	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovdqa64	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa64	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovdqa64	%ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa64	(%rax), %ymm19
+# CHECK-NEXT:  1      1     1.00           *            vmovdqa64	%ymm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovdqa64	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa64	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovdqa64	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovdqa64	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqa64	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu32	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu32	(%rax), %xmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu32	%xmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovdqu32	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu32	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu32	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu32	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu32	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu32	%ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu32	(%rax), %ymm19
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu32	%ymm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovdqu32	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu32	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu32	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu32	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu32	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu64	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu64	(%rax), %xmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu64	%xmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovdqu64	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu64	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu64	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu64	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu64	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu64	%ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu64	(%rax), %ymm19
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu64	%ymm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovdqu64	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu64	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovdqu64	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovdqu64	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovdqu64	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vmovddup	%ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovddup	(%rax), %ymm19
+# CHECK-NEXT:  1      2     1.00                        vmovddup	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovddup	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vmovddup	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovddup	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vmovshdup	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovshdup	(%rax), %xmm19
+# CHECK-NEXT:  1      1     0.50                        vmovshdup	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovshdup	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vmovshdup	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovshdup	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vmovshdup	%ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovshdup	(%rax), %ymm19
+# CHECK-NEXT:  1      1     0.50                        vmovshdup	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovshdup	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vmovshdup	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovshdup	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vmovsldup	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovsldup	(%rax), %xmm19
+# CHECK-NEXT:  1      1     0.50                        vmovsldup	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovsldup	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vmovsldup	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovsldup	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vmovsldup	%ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovsldup	(%rax), %ymm19
+# CHECK-NEXT:  1      1     0.50                        vmovsldup	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovsldup	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vmovsldup	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovsldup	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovupd	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovupd	(%rax), %xmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovupd	%xmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovupd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovupd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovupd	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovupd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovupd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovupd	%ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovupd	(%rax), %ymm19
+# CHECK-NEXT:  1      1     1.00           *            vmovupd	%ymm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovupd	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovupd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovupd	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovupd	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovupd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovups	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovups	(%rax), %xmm19
+# CHECK-NEXT:  1      1     1.00           *            vmovups	%xmm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovups	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovups	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovups	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovups	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovups	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      0     0.17                        vmovups	%ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vmovups	(%rax), %ymm19
+# CHECK-NEXT:  1      1     1.00           *            vmovups	%ymm16, (%rax)
+# CHECK-NEXT:  1      0     0.17                        vmovups	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vmovups	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00           *            vmovups	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  1      0     0.17                        vmovups	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vmovups	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vmulpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vmulpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vmulpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vmulpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vmulpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vmulpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vmulpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vmulpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vmulpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vmulpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vmulpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vmulpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vmulpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vmulpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vmulpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vmulpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vmulpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vmulpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vmulps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vmulps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vmulps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vmulps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vmulps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vmulps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vmulps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vmulps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vmulps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vmulps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vmulps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vmulps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vmulps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vmulps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vmulps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vmulps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vmulps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vmulps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpaddd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpaddd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpaddd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpaddd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpaddd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpaddd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddd	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpaddq	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpaddq	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpaddq	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpaddq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpaddq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpaddq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpaddq	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastd	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastd	(%rax), %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastd	%xmm16, %ymm19
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastd	(%rax), %ymm19
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastd	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastd	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastq	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastq	(%rax), %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpbroadcastq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpbroadcastq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastq	%xmm16, %ymm19
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastq	(%rax), %ymm19
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpbroadcastq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpbroadcastq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqd	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax){1to4}, %xmm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqd	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax){1to4}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqd	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax){1to8}, %ymm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqd	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax){1to8}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqd	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax){1to4}, %xmm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqd	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax){1to4}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqd	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax){1to8}, %ymm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqd	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqd	(%rax){1to8}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqq	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax){1to2}, %xmm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqq	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax){1to2}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqq	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax){1to4}, %ymm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqq	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax){1to4}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtd	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtd	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtd	(%rax){1to4}, %xmm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtd	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtd	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtd	(%rax){1to4}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtd	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtd	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtd	(%rax){1to8}, %ymm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtd	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtd	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtd	(%rax){1to8}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtq	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtq	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtq	(%rax){1to2}, %xmm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtq	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtq	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtq	(%rax){1to2}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtq	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtq	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtq	(%rax){1to4}, %ymm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpgtq	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtq	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpgtq	(%rax){1to4}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqq	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax){1to2}, %xmm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqq	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax){1to2}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqq	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax){1to4}, %ymm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpeqq	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpeqq	(%rax){1to4}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpequd	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequd	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequd	(%rax){1to4}, %xmm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpequd	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequd	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequd	(%rax){1to4}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpequd	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequd	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequd	(%rax){1to8}, %ymm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpequd	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequd	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequd	(%rax){1to8}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpequq	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequq	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequq	(%rax){1to2}, %xmm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpequq	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequq	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequq	(%rax){1to2}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vpcmpequq	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequq	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequq	(%rax){1to4}, %ymm1, %k2
+# CHECK-NEXT:  1      1     0.25                        vpcmpequq	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequq	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vpcmpequq	(%rax){1to4}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      4     1.00                        vpermd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermd	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      4     1.00                        vpermd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermd	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     1.00                        vpermd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermd	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpermilpd	$0, %xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpermilpd	$0, (%rax), %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpermilpd	$0, (%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpermilpd	$0, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpermilpd	$0, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpermilpd	$0, (%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpermilpd	$0, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpermilpd	$0, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpermilpd	$0, (%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpermilpd	$0, %ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpermilpd	$0, (%rax), %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpermilpd	$0, (%rax){1to4}, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpermilpd	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpermilpd	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpermilpd	$0, (%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpermilpd	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpermilpd	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpermilpd	$0, (%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     0.67                        vpermilpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpermilpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpermilpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      2     0.67                        vpermilpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpermilpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpermilpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     0.67                        vpermilpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpermilpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpermilpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     0.67                        vpermilpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpermilpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpermilpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     0.67                        vpermilpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpermilpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpermilpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     0.67                        vpermilpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpermilpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpermilpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpermilps	$0, %xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpermilps	$0, (%rax), %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpermilps	$0, (%rax){1to4}, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpermilps	$0, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpermilps	$0, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpermilps	$0, (%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpermilps	$0, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpermilps	$0, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpermilps	$0, (%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpermilps	$0, %ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpermilps	$0, (%rax), %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpermilps	$0, (%rax){1to8}, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpermilps	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpermilps	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpermilps	$0, (%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpermilps	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpermilps	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpermilps	$0, (%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     0.67                        vpermilps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpermilps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpermilps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      2     0.67                        vpermilps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpermilps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpermilps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      2     0.67                        vpermilps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpermilps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpermilps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     0.67                        vpermilps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpermilps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpermilps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     0.67                        vpermilps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpermilps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpermilps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     0.67                        vpermilps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpermilps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpermilps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpermpd	$0, %ymm16, %ymm19
+# CHECK-NEXT:  3      9     1.00    *                   vpermpd	$0, (%rax), %ymm19
+# CHECK-NEXT:  3      9     1.00    *                   vpermpd	$0, (%rax){1to4}, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpermpd	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vpermpd	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vpermpd	$0, (%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpermpd	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vpermpd	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vpermpd	$0, (%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      4     1.00                        vpermpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  3      14    1.00    *                   vpermpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  3      14    1.00    *                   vpermpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      4     1.00                        vpermpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  3      14    1.00    *                   vpermpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  3      14    1.00    *                   vpermpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     1.00                        vpermpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  3      14    1.00    *                   vpermpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  3      14    1.00    *                   vpermpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      4     1.00                        vpermps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  3      14    1.00    *                   vpermps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  3      14    1.00    *                   vpermps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      4     1.00                        vpermps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  3      14    1.00    *                   vpermps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  3      14    1.00    *                   vpermps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     1.00                        vpermps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  3      14    1.00    *                   vpermps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  3      14    1.00    *                   vpermps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     1.00                        vpermq	$0, %ymm16, %ymm19
+# CHECK-NEXT:  1      8     1.00    *                   vpermq	$0, (%rax), %ymm19
+# CHECK-NEXT:  1      8     1.00    *                   vpermq	$0, (%rax){1to4}, %ymm19
+# CHECK-NEXT:  1      1     1.00                        vpermq	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpermq	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      8     1.00    *                   vpermq	$0, (%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     1.00                        vpermq	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpermq	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     1.00    *                   vpermq	$0, (%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      4     1.00                        vpermq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermq	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  2      8     0.50    *                   vpermq	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      4     1.00                        vpermq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermq	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  2      8     0.50    *                   vpermq	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     1.00                        vpermq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermq	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  2      8     0.50    *                   vpermq	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherdq	(%rax,%xmm1,2), %ymm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherdd	(%rax,%ymm1,2), %ymm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherqq	(%rax,%ymm1,2), %ymm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherqd	(%rax,%ymm1,2), %xmm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherdq	(%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherdd	(%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherqq	(%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT:  1      5     0.33    *                   vpgatherqd	(%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbd	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxbd	(%rax), %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxbd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxbd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbd	%xmm16, %ymm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbd	(%rax), %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbd	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbd	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbq	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxbq	(%rax), %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxbq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxbq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbq	%xmm16, %ymm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbq	(%rax), %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxbq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxbq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxdq	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxdq	(%rax), %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpmovsxdq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxdq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxdq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxdq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxdq	%xmm16, %ymm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxdq	(%rax), %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpmovsxdq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxdq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxdq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxdq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxwd	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxwd	(%rax), %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpmovsxwd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxwd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxwd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxwd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxwd	%xmm16, %ymm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxwd	(%rax), %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpmovsxwd	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxwd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxwd	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxwd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxwq	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxwq	(%rax), %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpmovsxwq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxwq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxwq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovsxwq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxwq	%xmm16, %ymm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxwq	(%rax), %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpmovsxwq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxwq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovsxwq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovsxwq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbd	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxbd	(%rax), %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxbd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxbd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbd	%xmm16, %ymm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbd	(%rax), %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbd	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbd	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbq	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxbq	(%rax), %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxbq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxbq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbq	%xmm16, %ymm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbq	(%rax), %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxbq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxbq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxdq	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxdq	(%rax), %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpmovzxdq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxdq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxdq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxdq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxdq	%xmm16, %ymm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxdq	(%rax), %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpmovzxdq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxdq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxdq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxdq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxwd	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxwd	(%rax), %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpmovzxwd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxwd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxwd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxwd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxwd	%xmm16, %ymm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxwd	(%rax), %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpmovzxwd	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxwd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxwd	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxwd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxwq	%xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxwq	(%rax), %xmm19
+# CHECK-NEXT:  1      2     1.00                        vpmovzxwq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxwq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxwq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpmovzxwq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxwq	%xmm16, %ymm19
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxwq	(%rax), %ymm19
+# CHECK-NEXT:  1      2     1.00                        vpmovzxwq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxwq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpmovzxwq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      11    1.50    *                   vpmovzxwq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpmulld	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmulld	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmulld	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vpmulld	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulld	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulld	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vpmulld	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulld	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulld	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vpmulld	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmulld	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpmulld	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vpmulld	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulld	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulld	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vpmulld	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulld	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpmulld	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpshufd	$0, %xmm16, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpshufd	$0, (%rax), %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpshufd	$0, (%rax){1to4}, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpshufd	$0, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpshufd	$0, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpshufd	$0, (%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpshufd	$0, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpshufd	$0, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpshufd	$0, (%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpshufd	$0, %ymm16, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpshufd	$0, (%rax), %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpshufd	$0, (%rax){1to8}, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpshufd	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpshufd	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpshufd	$0, (%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpshufd	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpshufd	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpshufd	$0, (%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpsubd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpsubd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpsubd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpsubd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpsubd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpsubd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubd	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpsubq	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpsubq	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpsubq	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpsubq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpsubq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpsubq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpsubq	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vptestmd	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestmd	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestmd	(%rax){1to4}, %xmm1, %k2
+# CHECK-NEXT:  1      3     1.50                        vptestmd	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestmd	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestmd	(%rax){1to4}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      4     2.00                        vptestmd	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestmd	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestmd	(%rax){1to8}, %ymm1, %k2
+# CHECK-NEXT:  1      4     2.00                        vptestmd	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestmd	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestmd	(%rax){1to8}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vptestmq	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestmq	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestmq	(%rax){1to2}, %xmm1, %k2
+# CHECK-NEXT:  1      3     1.50                        vptestmq	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestmq	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestmq	(%rax){1to2}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      4     2.00                        vptestmq	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestmq	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestmq	(%rax){1to4}, %ymm1, %k2
+# CHECK-NEXT:  1      4     2.00                        vptestmq	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestmq	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestmq	(%rax){1to4}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vptestnmd	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmd	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmd	(%rax){1to4}, %xmm1, %k2
+# CHECK-NEXT:  1      3     1.50                        vptestnmd	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmd	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmd	(%rax){1to4}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      4     2.00                        vptestnmd	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmd	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmd	(%rax){1to8}, %ymm1, %k2
+# CHECK-NEXT:  1      4     2.00                        vptestnmd	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmd	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmd	(%rax){1to8}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.25                        vptestnmq	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmq	(%rax), %xmm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmq	(%rax){1to2}, %xmm1, %k2
+# CHECK-NEXT:  1      3     1.50                        vptestnmq	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmq	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmq	(%rax){1to2}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  1      4     2.00                        vptestnmq	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmq	(%rax), %ymm1, %k2
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmq	(%rax){1to4}, %ymm1, %k2
+# CHECK-NEXT:  1      4     2.00                        vptestnmq	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmq	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      8     0.50    *                   vptestnmq	(%rax){1to4}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  1      1     0.50                        vpunpckhdq	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhdq	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhdq	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpunpckhdq	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhdq	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhdq	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpunpckhdq	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhdq	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhdq	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpunpckhdq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhdq	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhdq	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpunpckhdq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhdq	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhdq	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpunpckhdq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhdq	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckhdq	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpunpckldq	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckldq	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckldq	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vpunpckldq	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckldq	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckldq	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpunpckldq	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckldq	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckldq	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpunpckldq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckldq	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckldq	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpunpckldq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckldq	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckldq	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpunpckldq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckldq	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpunpckldq	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vshuff32x4	$0, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  3      9     1.00    *                   vshuff32x4	$0, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  3      9     1.00    *                   vshuff32x4	$0, (%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vshuff32x4	$0, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vshuff32x4	$0, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vshuff32x4	$0, (%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vshuff32x4	$0, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vshuff32x4	$0, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vshuff32x4	$0, (%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vshuff64x2	$0, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  3      9     1.00    *                   vshuff64x2	$0, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  3      9     1.00    *                   vshuff64x2	$0, (%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vshuff64x2	$0, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vshuff64x2	$0, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vshuff64x2	$0, (%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vshuff64x2	$0, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vshuff64x2	$0, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vshuff64x2	$0, (%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vshufi32x4	$0, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  3      9     1.00    *                   vshufi32x4	$0, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  3      9     1.00    *                   vshufi32x4	$0, (%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vshufi32x4	$0, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vshufi32x4	$0, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vshufi32x4	$0, (%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vshufi32x4	$0, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vshufi32x4	$0, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vshufi32x4	$0, (%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vshufi64x2	$0, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  3      9     1.00    *                   vshufi64x2	$0, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  3      9     1.00    *                   vshufi64x2	$0, (%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      2     1.00                        vshufi64x2	$0, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vshufi64x2	$0, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  3      9     1.00    *                   vshufi64x2	$0, (%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vshufi64x2	$0, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vshufi64x2	$0, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  3      9     1.00    *                   vshufi64x2	$0, (%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      21    9.00                        vsqrtpd	%xmm16, %xmm19
+# CHECK-NEXT:  1      28    9.00    *                   vsqrtpd	(%rax), %xmm19
+# CHECK-NEXT:  1      28    9.00    *                   vsqrtpd	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  1      21    9.00                        vsqrtpd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      28    9.00    *                   vsqrtpd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      28    9.00    *                   vsqrtpd	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  1      21    9.00                        vsqrtpd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      28    9.00    *                   vsqrtpd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      28    9.00    *                   vsqrtpd	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      21    9.00                        vsqrtpd	%ymm16, %ymm19
+# CHECK-NEXT:  1      28    9.00    *                   vsqrtpd	(%rax), %ymm19
+# CHECK-NEXT:  1      28    9.00    *                   vsqrtpd	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  1      21    9.00                        vsqrtpd	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      28    9.00    *                   vsqrtpd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      28    9.00    *                   vsqrtpd	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  1      21    9.00                        vsqrtpd	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      28    9.00    *                   vsqrtpd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      28    9.00    *                   vsqrtpd	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      15    5.00                        vsqrtps	%xmm16, %xmm19
+# CHECK-NEXT:  1      22    5.00    *                   vsqrtps	(%rax), %xmm19
+# CHECK-NEXT:  1      22    5.00    *                   vsqrtps	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  1      15    5.00                        vsqrtps	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  1      22    5.00    *                   vsqrtps	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  1      22    5.00    *                   vsqrtps	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  1      15    5.00                        vsqrtps	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      22    5.00    *                   vsqrtps	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      22    5.00    *                   vsqrtps	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      15    5.00                        vsqrtps	%ymm16, %ymm19
+# CHECK-NEXT:  1      22    5.00    *                   vsqrtps	(%rax), %ymm19
+# CHECK-NEXT:  1      22    5.00    *                   vsqrtps	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  1      15    5.00                        vsqrtps	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  1      22    5.00    *                   vsqrtps	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  1      22    5.00    *                   vsqrtps	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  1      15    5.00                        vsqrtps	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      22    5.00    *                   vsqrtps	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      22    5.00    *                   vsqrtps	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vsubpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vsubpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vsubpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vsubpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vsubpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vsubpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vsubpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vsubpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vsubpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vsubpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vsubpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vsubpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vsubpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vsubpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vsubpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vsubpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vsubpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vsubpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vsubps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vsubps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vsubps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      3     0.50                        vsubps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vsubps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vsubps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vsubps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vsubps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vsubps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      3     0.50                        vsubps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vsubps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vsubps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      3     0.50                        vsubps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vsubps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vsubps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      3     0.50                        vsubps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vsubps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vsubps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vunpckhpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vunpckhpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vunpckhpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vunpckhpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vunpckhpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vunpckhpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vunpckhps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vunpckhps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vunpckhps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vunpckhps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vunpckhps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vunpckhps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vunpckhps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vunpcklpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vunpcklpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vunpcklpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vunpcklpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vunpcklpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vunpcklpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vunpcklps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      1     0.50                        vunpcklps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vunpcklps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vunpcklps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vunpcklps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vunpcklps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vunpcklps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 5.33   5.33   5.33    -      -      -      -      -     175.00 945.00 531.50 261.50 442.50 442.50 32.00  300.33 300.33 300.33 289.67 289.67 289.67 16.00  16.00
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vaddps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaddps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     valignd	$1, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignd	$1, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignd	$1, (%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     valignd	$1, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignd	$1, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignd	$1, (%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     valignd	$1, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignd	$1, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignd	$1, (%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     valignd	$1, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignd	$1, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignd	$1, (%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     valignd	$1, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignd	$1, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignd	$1, (%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     valignd	$1, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignd	$1, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignd	$1, (%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     valignq	$1, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignq	$1, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignq	$1, (%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     valignq	$1, %xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignq	$1, (%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignq	$1, (%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     valignq	$1, %xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignq	$1, (%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignq	$1, (%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     valignq	$1, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignq	$1, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignq	$1, (%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     valignq	$1, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignq	$1, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignq	$1, (%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     valignq	$1, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignq	$1, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     valignq	$1, (%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf32x4	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf32x4	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastf32x4	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti32x4	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti32x4	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcasti32x4	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastsd	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastsd	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastsd	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastsd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastsd	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastsd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastss	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastss	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastss	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastss	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastss	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastss	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastss	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastss	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastss	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastss	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vbroadcastss	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vbroadcastss	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqpd	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqpd	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqpd	(%rax){1to2}, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqpd	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqpd	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqpd	(%rax){1to2}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqpd	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqpd	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqpd	(%rax){1to4}, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqpd	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqpd	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqpd	(%rax){1to4}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqps	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqps	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqps	(%rax){1to4}, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqps	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqps	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqps	(%rax){1to4}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqps	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqps	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqps	(%rax){1to8}, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vcmpeqps	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqps	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcmpeqps	(%rax){1to8}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2pd	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2pd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2pd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2pd	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2pd	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2pd	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2pd	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2ps	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2ps	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2ps	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2ps	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2ps	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtdq2ps	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtdq2ps	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2dq	%ymm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dqy	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dq	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2dq	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dqy	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dq	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2dq	%ymm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dqy	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dq	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2dq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dqx	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2dq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dqx	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2dq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dqx	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2dq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2ps	%ymm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2psy	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2ps	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2ps	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2psy	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2ps	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2ps	%ymm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2psy	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2ps	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2ps	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2psx	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2ps	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2ps	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2psx	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2ps	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2ps	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2psx	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2ps	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2udq	%ymm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2udqy	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2udq	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2udq	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2udqy	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2udq	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2udq	%ymm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2udqy	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2udq	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2udq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2udqx	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2udq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2udq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2udqx	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2udq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtpd2udq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2udqx	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtpd2udq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2dq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2dq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2dq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2dq	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2dq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2dq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2dq	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2pd	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2pd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2pd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2pd	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2pd	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2pd	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2pd	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2udq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2udq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2udq	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2udq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2udq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2udq	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2udq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2udq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2udq	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2udq	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2udq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2udq	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2udq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2udq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2udq	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtps2udq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2udq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtps2udq	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2dq	%ymm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dqy	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dq	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2dq	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dqy	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dq	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2dq	%ymm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dqy	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dq	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2dq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dqx	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2dq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dqx	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2dq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dqx	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2dq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2dq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2dq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2dq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2dq	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2dq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2dq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2dq	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2udq	%ymm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2udqy	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2udq	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2udq	%ymm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2udqy	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2udq	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2udq	%ymm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2udqy	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2udq	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2udq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2udqx	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2udq	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2udq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2udqx	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2udq	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvttpd2udq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2udqx	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttpd2udq	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2udq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2udq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2udq	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2udq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2udq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2udq	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2udq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2udq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2udq	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2udq	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2udq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2udq	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2udq	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2udq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2udq	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvttps2udq	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2udq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvttps2udq	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vdivps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vdivps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherdpd	(%rax,%xmm1,2), %ymm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherdps	(%rax,%ymm1,2), %ymm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherqpd	(%rax,%ymm1,2), %ymm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherqps	(%rax,%ymm1,2), %xmm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherdpd	(%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherdps	(%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherqpd	(%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgatherqps	(%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vmaxps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmaxps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vminps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vminps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vminps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vminps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vminps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovapd	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovapd	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovapd	%xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovapd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovapd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovapd	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovapd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovapd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovapd	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovapd	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovapd	%ymm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovapd	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovapd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovapd	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovapd	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovapd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovaps	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovaps	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovaps	%xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovaps	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovaps	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovaps	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovaps	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovaps	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovaps	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovaps	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovaps	%ymm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovaps	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovaps	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovaps	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovaps	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovaps	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovddup	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovddup	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovddup	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovddup	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovddup	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovddup	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa32	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa32	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqa32	%xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa32	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa32	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqa32	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa32	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa32	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa32	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa32	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqa32	%ymm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa32	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa32	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqa32	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa32	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa32	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa64	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa64	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqa64	%xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa64	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa64	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqa64	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa64	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa64	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa64	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa64	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqa64	%ymm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa64	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa64	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqa64	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqa64	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqa64	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu32	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu32	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu32	%xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu32	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu32	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu32	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu32	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu32	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu32	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu32	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu32	%ymm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu32	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu32	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu32	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu32	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu32	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu64	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu64	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu64	%xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu64	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu64	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu64	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu64	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu64	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu64	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu64	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu64	%ymm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu64	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu64	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovdqu64	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovdqu64	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovdqu64	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovddup	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovddup	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovddup	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovddup	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vmovddup	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovddup	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovshdup	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovshdup	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovshdup	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovshdup	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovshdup	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovshdup	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovshdup	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovshdup	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovshdup	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovshdup	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovshdup	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovshdup	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovsldup	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovsldup	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovsldup	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovsldup	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovsldup	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovsldup	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovsldup	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovsldup	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovsldup	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovsldup	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vmovsldup	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovsldup	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovupd	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovupd	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovupd	%xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovupd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovupd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovupd	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovupd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovupd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovupd	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovupd	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovupd	%ymm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovupd	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovupd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovupd	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovupd	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovupd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovups	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovups	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovups	%xmm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovups	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovups	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovups	%xmm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovups	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovups	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovups	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovups	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovups	%ymm16, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovups	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovups	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vmovups	%ymm16, (%rax) {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vmovups	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmovups	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vmulps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vmulps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddd	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddq	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddq	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddq	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpaddq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpaddq	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastd	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastd	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastd	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastd	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastd	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastd	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastq	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpbroadcastq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpbroadcastq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqd	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax){1to4}, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqd	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax){1to4}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqd	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax){1to8}, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqd	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax){1to8}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqd	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax){1to4}, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqd	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax){1to4}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqd	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax){1to8}, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqd	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqd	(%rax){1to8}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqq	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax){1to2}, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqq	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax){1to2}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqq	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax){1to4}, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqq	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax){1to4}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtd	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtd	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtd	(%rax){1to4}, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtd	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtd	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtd	(%rax){1to4}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtd	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtd	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtd	(%rax){1to8}, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtd	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtd	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtd	(%rax){1to8}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtq	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtq	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtq	(%rax){1to2}, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtq	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtq	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtq	(%rax){1to2}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtq	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtq	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtq	(%rax){1to4}, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpgtq	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtq	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpgtq	(%rax){1to4}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqq	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax){1to2}, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqq	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax){1to2}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqq	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax){1to4}, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpeqq	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpeqq	(%rax){1to4}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpequd	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequd	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequd	(%rax){1to4}, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpequd	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequd	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequd	(%rax){1to4}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpequd	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequd	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequd	(%rax){1to8}, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpequd	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequd	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequd	(%rax){1to8}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpequq	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequq	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequq	(%rax){1to2}, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpequq	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequq	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequq	(%rax){1to2}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpequq	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequq	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequq	(%rax){1to4}, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpcmpequq	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequq	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpcmpequq	(%rax){1to4}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermd	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermd	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermd	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermilpd	$0, %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$0, (%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$0, (%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermilpd	$0, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$0, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$0, (%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermilpd	$0, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$0, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$0, (%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermilpd	$0, %ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$0, (%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$0, (%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermilpd	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$0, (%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermilpd	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	$0, (%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermilps	$0, %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$0, (%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$0, (%rax){1to4}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermilps	$0, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$0, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$0, (%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermilps	$0, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$0, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$0, (%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermilps	$0, %ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$0, (%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$0, (%rax){1to8}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermilps	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$0, (%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpermilps	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	$0, (%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -     vpermilps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermilps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpermpd	$0, %ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	$0, (%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	$0, (%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpermpd	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	$0, (%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpermpd	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	$0, (%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpermq	$0, %ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	$0, (%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	$0, (%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpermq	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	$0, (%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpermq	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	$0, (%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpermq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpermq	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherdq	(%rax,%xmm1,2), %ymm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherdd	(%rax,%ymm1,2), %ymm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherqq	(%rax,%ymm1,2), %ymm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherqd	(%rax,%ymm1,2), %xmm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherdq	(%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherdd	(%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherqq	(%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpgatherqd	(%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbd	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbd	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbd	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbd	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbd	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbd	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbq	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxbq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxbq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxdq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxdq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxdq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxdq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxdq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxdq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxdq	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxdq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxdq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxdq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxdq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxdq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwd	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwd	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwd	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwd	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwd	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwd	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwq	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovsxwq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovsxwq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbd	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbd	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbd	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbd	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbd	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbd	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbq	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxbq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxbq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxdq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxdq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxdq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxdq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxdq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxdq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxdq	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxdq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxdq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxdq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxdq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxdq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwd	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwd	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwd	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwd	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwd	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwd	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwq	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwq	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwq	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwq	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwq	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwq	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwq	%xmm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwq	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwq	%xmm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwq	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpmovzxwq	%xmm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmovzxwq	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulld	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulld	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulld	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulld	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulld	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vpmulld	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpmulld	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshufd	$0, %xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$0, (%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$0, (%rax){1to4}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshufd	$0, %xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$0, (%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$0, (%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshufd	$0, %xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$0, (%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$0, (%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshufd	$0, %ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$0, (%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$0, (%rax){1to8}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshufd	$0, %ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$0, (%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$0, (%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpshufd	$0, %ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$0, (%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpshufd	$0, (%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubd	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubq	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubq	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubq	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpsubq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpsubq	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vptestmd	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmd	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmd	(%rax){1to4}, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestmd	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmd	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmd	(%rax){1to4}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestmd	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmd	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmd	(%rax){1to8}, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestmd	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmd	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmd	(%rax){1to8}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vptestmq	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmq	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmq	(%rax){1to2}, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestmq	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmq	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmq	(%rax){1to2}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestmq	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmq	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmq	(%rax){1to4}, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestmq	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmq	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestmq	(%rax){1to4}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vptestnmd	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmd	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmd	(%rax){1to4}, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestnmd	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmd	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmd	(%rax){1to4}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestnmd	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmd	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmd	(%rax){1to8}, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestnmd	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmd	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmd	(%rax){1to8}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vptestnmq	%xmm0, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmq	(%rax), %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmq	(%rax){1to2}, %xmm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestnmq	%xmm0, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmq	(%rax), %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmq	(%rax){1to2}, %xmm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestnmq	%ymm0, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmq	(%rax), %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmq	(%rax){1to4}, %ymm1, %k2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vptestnmq	%ymm0, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmq	(%rax), %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vptestnmq	(%rax){1to4}, %ymm1, %k2 {%k3}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhdq	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhdq	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhdq	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhdq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhdq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckhdq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckhdq	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckldq	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckldq	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckldq	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckldq	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckldq	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vpunpckldq	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpunpckldq	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshuff32x4	$0, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff32x4	$0, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff32x4	$0, (%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshuff32x4	$0, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff32x4	$0, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff32x4	$0, (%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshuff32x4	$0, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff32x4	$0, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff32x4	$0, (%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshuff64x2	$0, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff64x2	$0, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff64x2	$0, (%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshuff64x2	$0, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff64x2	$0, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff64x2	$0, (%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshuff64x2	$0, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff64x2	$0, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshuff64x2	$0, (%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshufi32x4	$0, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi32x4	$0, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi32x4	$0, (%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshufi32x4	$0, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi32x4	$0, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi32x4	$0, (%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshufi32x4	$0, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi32x4	$0, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi32x4	$0, (%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshufi64x2	$0, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi64x2	$0, (%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi64x2	$0, (%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshufi64x2	$0, %ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi64x2	$0, (%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi64x2	$0, (%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vshufi64x2	$0, %ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi64x2	$0, (%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vshufi64x2	$0, (%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtpd	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax){1to2}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtpd	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax){1to2}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtpd	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax){1to2}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtpd	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax){1to4}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtpd	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax){1to4}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtpd	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtpd	(%rax){1to4}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtps	%xmm16, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax), %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax){1to4}, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtps	%xmm16, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax), %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax){1to4}, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtps	%xmm16, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax), %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax){1to4}, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtps	%ymm16, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax), %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax){1to8}, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtps	%ymm16, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax), %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax){1to8}, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vsqrtps	%ymm16, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax), %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsqrtps	(%rax){1to8}, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vsubpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vsubpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vsubpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vsubpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vsubpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vsubpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vsubps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vsubps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vsubps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vsubps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vsubps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vsubps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vsubps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpckhps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpckhps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklpd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax){1to2}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklpd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax){1to2}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklpd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax){1to2}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklpd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax){1to4}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklpd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax){1to4}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklpd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklpd	(%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklps	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklps	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklps	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklps	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklps	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     vunpcklps	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vunpcklps	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vnni.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vnni.s
new file mode 100644
index 0000000000000..ecb64c5c4e993
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vnni.s
@@ -0,0 +1,156 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vpdpbusd    %zmm16, %zmm17, %zmm19
+vpdpbusd    (%rax), %zmm17, %zmm19
+vpdpbusd    (%rax){1to16}, %zmm17, %zmm19
+vpdpbusd    %zmm16, %zmm17, %zmm19 {k1}
+vpdpbusd    (%rax), %zmm17, %zmm19 {k1}
+vpdpbusd    (%rax){1to16}, %zmm17, %zmm19 {k1}
+vpdpbusd    %zmm16, %zmm17, %zmm19 {z}{k1}
+vpdpbusd    (%rax), %zmm17, %zmm19 {z}{k1}
+vpdpbusd    (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vpdpbusds   %zmm16, %zmm17, %zmm19
+vpdpbusds   (%rax), %zmm17, %zmm19
+vpdpbusds   (%rax){1to16}, %zmm17, %zmm19
+vpdpbusds   %zmm16, %zmm17, %zmm19 {k1}
+vpdpbusds   (%rax), %zmm17, %zmm19 {k1}
+vpdpbusds   (%rax){1to16}, %zmm17, %zmm19 {k1}
+vpdpbusds   %zmm16, %zmm17, %zmm19 {z}{k1}
+vpdpbusds   (%rax), %zmm17, %zmm19 {z}{k1}
+vpdpbusds   (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vpdpwssd    %zmm16, %zmm17, %zmm19
+vpdpwssd    (%rax), %zmm17, %zmm19
+vpdpwssd    (%rax){1to16}, %zmm17, %zmm19
+vpdpwssd    %zmm16, %zmm17, %zmm19 {k1}
+vpdpwssd    (%rax), %zmm17, %zmm19 {k1}
+vpdpwssd    (%rax){1to16}, %zmm17, %zmm19 {k1}
+vpdpwssd    %zmm16, %zmm17, %zmm19 {z}{k1}
+vpdpwssd    (%rax), %zmm17, %zmm19 {z}{k1}
+vpdpwssd    (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+vpdpwssds   %zmm16, %zmm17, %zmm19
+vpdpwssds   (%rax), %zmm17, %zmm19
+vpdpwssds   (%rax){1to16}, %zmm17, %zmm19
+vpdpwssds   %zmm16, %zmm17, %zmm19 {k1}
+vpdpwssds   (%rax), %zmm17, %zmm19 {k1}
+vpdpwssds   (%rax){1to16}, %zmm17, %zmm19 {k1}
+vpdpwssds   %zmm16, %zmm17, %zmm19 {z}{k1}
+vpdpwssds   (%rax), %zmm17, %zmm19 {z}{k1}
+vpdpwssds   (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      4     2.00                        vpdpbusd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpdpbusd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpdpbusd	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      4     2.00                        vpdpbusd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpdpbusd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpdpbusd	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpdpbusd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpdpbusd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpdpbusd	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpdpbusds	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpdpbusds	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpdpbusds	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      4     2.00                        vpdpbusds	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpdpbusds	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpdpbusds	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpdpbusds	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpdpbusds	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpdpbusds	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpdpwssd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpdpwssd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpdpwssd	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      4     2.00                        vpdpwssd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpdpwssd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpdpwssd	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpdpwssd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpdpwssd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpdpwssd	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpdpwssds	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpdpwssds	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  1      10    1.00    *                   vpdpwssds	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  1      4     2.00                        vpdpwssds	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpdpwssds	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      10    1.00    *                   vpdpwssds	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpdpwssds	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpdpwssds	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    1.00    *                   vpdpwssds	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     48.00  24.00   -     24.00  12.00  12.00   -     8.00   8.00   8.00   8.00   8.00   8.00    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusds	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusds	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusds	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssd	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssd	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssd	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssds	%zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax), %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax){1to16}, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssds	%zmm16, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax), %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax){1to16}, %zmm17, %zmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssds	%zmm16, %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax), %zmm17, %zmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -     1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vnnivl.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vnnivl.s
new file mode 100644
index 0000000000000..f101bc14dc33f
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vnnivl.s
@@ -0,0 +1,268 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vpdpbusd    %xmm16, %xmm17, %xmm19
+vpdpbusd    (%rax), %xmm17, %xmm19
+vpdpbusd    (%rax){1to4}, %xmm17, %xmm19
+vpdpbusd    %xmm16, %xmm17, %xmm19 {k1}
+vpdpbusd    (%rax), %xmm17, %xmm19 {k1}
+vpdpbusd    (%rax){1to4}, %xmm17, %xmm19 {k1}
+vpdpbusd    %xmm16, %xmm17, %xmm19 {z}{k1}
+vpdpbusd    (%rax), %xmm17, %xmm19 {z}{k1}
+vpdpbusd    (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vpdpbusd    %ymm16, %ymm17, %ymm19
+vpdpbusd    (%rax), %ymm17, %ymm19
+vpdpbusd    (%rax){1to8}, %ymm17, %ymm19
+vpdpbusd    %ymm16, %ymm17, %ymm19 {k1}
+vpdpbusd    (%rax), %ymm17, %ymm19 {k1}
+vpdpbusd    (%rax){1to8}, %ymm17, %ymm19 {k1}
+vpdpbusd    %ymm16, %ymm17, %ymm19 {z}{k1}
+vpdpbusd    (%rax), %ymm17, %ymm19 {z}{k1}
+vpdpbusd    (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vpdpbusds   %xmm16, %xmm17, %xmm19
+vpdpbusds   (%rax), %xmm17, %xmm19
+vpdpbusds   (%rax){1to4}, %xmm17, %xmm19
+vpdpbusds   %xmm16, %xmm17, %xmm19 {k1}
+vpdpbusds   (%rax), %xmm17, %xmm19 {k1}
+vpdpbusds   (%rax){1to4}, %xmm17, %xmm19 {k1}
+vpdpbusds   %xmm16, %xmm17, %xmm19 {z}{k1}
+vpdpbusds   (%rax), %xmm17, %xmm19 {z}{k1}
+vpdpbusds   (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vpdpbusds   %ymm16, %ymm17, %ymm19
+vpdpbusds   (%rax), %ymm17, %ymm19
+vpdpbusds   (%rax){1to8}, %ymm17, %ymm19
+vpdpbusds   %ymm16, %ymm17, %ymm19 {k1}
+vpdpbusds   (%rax), %ymm17, %ymm19 {k1}
+vpdpbusds   (%rax){1to8}, %ymm17, %ymm19 {k1}
+vpdpbusds   %ymm16, %ymm17, %ymm19 {z}{k1}
+vpdpbusds   (%rax), %ymm17, %ymm19 {z}{k1}
+vpdpbusds   (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vpdpwssd    %xmm16, %xmm17, %xmm19
+vpdpwssd    (%rax), %xmm17, %xmm19
+vpdpwssd    (%rax){1to4}, %xmm17, %xmm19
+vpdpwssd    %xmm16, %xmm17, %xmm19 {k1}
+vpdpwssd    (%rax), %xmm17, %xmm19 {k1}
+vpdpwssd    (%rax){1to4}, %xmm17, %xmm19 {k1}
+vpdpwssd    %xmm16, %xmm17, %xmm19 {z}{k1}
+vpdpwssd    (%rax), %xmm17, %xmm19 {z}{k1}
+vpdpwssd    (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vpdpwssd    %ymm16, %ymm17, %ymm19
+vpdpwssd    (%rax), %ymm17, %ymm19
+vpdpwssd    (%rax){1to8}, %ymm17, %ymm19
+vpdpwssd    %ymm16, %ymm17, %ymm19 {k1}
+vpdpwssd    (%rax), %ymm17, %ymm19 {k1}
+vpdpwssd    (%rax){1to8}, %ymm17, %ymm19 {k1}
+vpdpwssd    %ymm16, %ymm17, %ymm19 {z}{k1}
+vpdpwssd    (%rax), %ymm17, %ymm19 {z}{k1}
+vpdpwssd    (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+vpdpwssds   %xmm16, %xmm17, %xmm19
+vpdpwssds   (%rax), %xmm17, %xmm19
+vpdpwssds   (%rax){1to4}, %xmm17, %xmm19
+vpdpwssds   %xmm16, %xmm17, %xmm19 {k1}
+vpdpwssds   (%rax), %xmm17, %xmm19 {k1}
+vpdpwssds   (%rax){1to4}, %xmm17, %xmm19 {k1}
+vpdpwssds   %xmm16, %xmm17, %xmm19 {z}{k1}
+vpdpwssds   (%rax), %xmm17, %xmm19 {z}{k1}
+vpdpwssds   (%rax){1to4}, %xmm17, %xmm19 {z}{k1}
+
+vpdpwssds   %ymm16, %ymm17, %ymm19
+vpdpwssds   (%rax), %ymm17, %ymm19
+vpdpwssds   (%rax){1to8}, %ymm17, %ymm19
+vpdpwssds   %ymm16, %ymm17, %ymm19 {k1}
+vpdpwssds   (%rax), %ymm17, %ymm19 {k1}
+vpdpwssds   (%rax){1to8}, %ymm17, %ymm19 {k1}
+vpdpwssds   %ymm16, %ymm17, %ymm19 {z}{k1}
+vpdpwssds   (%rax), %ymm17, %ymm19 {z}{k1}
+vpdpwssds   (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      4     2.00                        vpdpbusd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusd	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      4     2.00                        vpdpbusd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusd	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpdpbusd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusd	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpdpbusd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusd	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      4     2.00                        vpdpbusd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusd	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpdpbusd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusd	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpdpbusds	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusds	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusds	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      4     2.00                        vpdpbusds	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusds	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusds	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpdpbusds	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusds	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusds	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpdpbusds	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusds	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusds	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      4     2.00                        vpdpbusds	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusds	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusds	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpdpbusds	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusds	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusds	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpdpwssd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssd	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      4     2.00                        vpdpwssd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssd	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpdpwssd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssd	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpdpwssd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssd	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      4     2.00                        vpdpwssd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssd	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpdpwssd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssd	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpdpwssds	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssds	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssds	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  1      4     2.00                        vpdpwssds	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssds	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssds	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpdpwssds	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssds	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssds	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  1      4     2.00                        vpdpwssds	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssds	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssds	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  1      4     2.00                        vpdpwssds	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssds	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssds	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  1      4     2.00                        vpdpwssds	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssds	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssds	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     72.00  48.00   -     24.00  24.00  24.00   -     16.00  16.00  16.00  16.00  16.00  16.00   -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusds	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusds	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusds	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusds	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusds	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusds	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssd	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssd	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssd	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssd	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssd	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssd	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssds	%xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax){1to4}, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssds	%xmm16, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax), %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax){1to4}, %xmm17, %xmm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssds	%xmm16, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax), %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax){1to4}, %xmm17, %xmm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssds	%ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax), %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax){1to8}, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssds	%ymm16, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax), %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax){1to8}, %ymm17, %ymm19 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssds	%ymm16, %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax), %ymm17, %ymm19 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vp2intersect.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vp2intersect.s
new file mode 100644
index 0000000000000..3230ae9d473c5
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vp2intersect.s
@@ -0,0 +1,53 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=tigerlake -instruction-tables < %s | FileCheck %s
+
+vp2intersectd    %zmm16, %zmm19, %k0
+vp2intersectd    (%rax), %zmm19, %k0
+vp2intersectd    (%rax){1to16}, %zmm19, %k0
+
+vp2intersectq    %zmm16, %zmm19, %k0
+vp2intersectq    (%rax), %zmm19, %k0
+vp2intersectq    (%rax){1to8}, %zmm19, %k0
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     1.00                        vp2intersectd	%zmm16, %zmm19, %k0
+# CHECK-NEXT:  2      8     1.00    *                   vp2intersectd	(%rax), %zmm19, %k0
+# CHECK-NEXT:  2      8     1.00    *                   vp2intersectd	(%rax){1to16}, %zmm19, %k0
+# CHECK-NEXT:  1      1     1.00                        vp2intersectq	%zmm16, %zmm19, %k0
+# CHECK-NEXT:  2      8     1.00    *                   vp2intersectq	(%rax), %zmm19, %k0
+# CHECK-NEXT:  2      8     1.00    *                   vp2intersectq	(%rax){1to8}, %zmm19, %k0
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - ICXDivider
+# CHECK-NEXT: [1]   - ICXFPDivider
+# CHECK-NEXT: [2]   - ICXPort0
+# CHECK-NEXT: [3]   - ICXPort1
+# CHECK-NEXT: [4]   - ICXPort2
+# CHECK-NEXT: [5]   - ICXPort3
+# CHECK-NEXT: [6]   - ICXPort4
+# CHECK-NEXT: [7]   - ICXPort5
+# CHECK-NEXT: [8]   - ICXPort6
+# CHECK-NEXT: [9]   - ICXPort7
+# CHECK-NEXT: [10]  - ICXPort8
+# CHECK-NEXT: [11]  - ICXPort9
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]
+# CHECK-NEXT:  -      -     6.00    -     2.00   2.00    -      -      -      -      -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   Instructions:
+# CHECK-NEXT:  -      -     1.00    -      -      -      -      -      -      -      -      -     vp2intersectd	%zmm16, %zmm19, %k0
+# CHECK-NEXT:  -      -     1.00    -     0.50   0.50    -      -      -      -      -      -     vp2intersectd	(%rax), %zmm19, %k0
+# CHECK-NEXT:  -      -     1.00    -     0.50   0.50    -      -      -      -      -      -     vp2intersectd	(%rax){1to16}, %zmm19, %k0
+# CHECK-NEXT:  -      -     1.00    -      -      -      -      -      -      -      -      -     vp2intersectq	%zmm16, %zmm19, %k0
+# CHECK-NEXT:  -      -     1.00    -     0.50   0.50    -      -      -      -      -      -     vp2intersectq	(%rax), %zmm19, %k0
+# CHECK-NEXT:  -      -     1.00    -     0.50   0.50    -      -      -      -      -      -     vp2intersectq	(%rax){1to8}, %zmm19, %k0
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vp2intersectvl.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vp2intersectvl.s
new file mode 100644
index 0000000000000..b7e81f560fb8a
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vp2intersectvl.s
@@ -0,0 +1,73 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=tigerlake -instruction-tables < %s | FileCheck %s
+
+vp2intersectd    %xmm16, %xmm19, %k0
+vp2intersectd    (%rax), %xmm19, %k0
+vp2intersectd    (%rax){1to4}, %xmm19, %k0
+
+vp2intersectd    %ymm16, %ymm19, %k0
+vp2intersectd    (%rax), %ymm19, %k0
+vp2intersectd    (%rax){1to8}, %ymm19, %k0
+
+vp2intersectq    %xmm16, %xmm19, %k0
+vp2intersectq    (%rax), %xmm19, %k0
+vp2intersectq    (%rax){1to2}, %xmm19, %k0
+
+vp2intersectq    %ymm16, %ymm19, %k0
+vp2intersectq    (%rax), %ymm19, %k0
+vp2intersectq    (%rax){1to4}, %ymm19, %k0
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     0.50                        vp2intersectd	%xmm16, %xmm19, %k0
+# CHECK-NEXT:  2      7     0.50    *                   vp2intersectd	(%rax), %xmm19, %k0
+# CHECK-NEXT:  2      7     0.50    *                   vp2intersectd	(%rax){1to4}, %xmm19, %k0
+# CHECK-NEXT:  1      1     0.50                        vp2intersectd	%ymm16, %ymm19, %k0
+# CHECK-NEXT:  2      8     0.50    *                   vp2intersectd	(%rax), %ymm19, %k0
+# CHECK-NEXT:  2      8     0.50    *                   vp2intersectd	(%rax){1to8}, %ymm19, %k0
+# CHECK-NEXT:  1      1     0.50                        vp2intersectq	%xmm16, %xmm19, %k0
+# CHECK-NEXT:  2      7     0.50    *                   vp2intersectq	(%rax), %xmm19, %k0
+# CHECK-NEXT:  2      7     0.50    *                   vp2intersectq	(%rax){1to2}, %xmm19, %k0
+# CHECK-NEXT:  1      1     0.50                        vp2intersectq	%ymm16, %ymm19, %k0
+# CHECK-NEXT:  2      8     0.50    *                   vp2intersectq	(%rax), %ymm19, %k0
+# CHECK-NEXT:  2      8     0.50    *                   vp2intersectq	(%rax){1to4}, %ymm19, %k0
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - ICXDivider
+# CHECK-NEXT: [1]   - ICXFPDivider
+# CHECK-NEXT: [2]   - ICXPort0
+# CHECK-NEXT: [3]   - ICXPort1
+# CHECK-NEXT: [4]   - ICXPort2
+# CHECK-NEXT: [5]   - ICXPort3
+# CHECK-NEXT: [6]   - ICXPort4
+# CHECK-NEXT: [7]   - ICXPort5
+# CHECK-NEXT: [8]   - ICXPort6
+# CHECK-NEXT: [9]   - ICXPort7
+# CHECK-NEXT: [10]  - ICXPort8
+# CHECK-NEXT: [11]  - ICXPort9
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]
+# CHECK-NEXT:  -      -     6.00   6.00   4.00   4.00    -      -      -      -      -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   Instructions:
+# CHECK-NEXT:  -      -     0.50   0.50    -      -      -      -      -      -      -      -     vp2intersectd	%xmm16, %xmm19, %k0
+# CHECK-NEXT:  -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -     vp2intersectd	(%rax), %xmm19, %k0
+# CHECK-NEXT:  -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -     vp2intersectd	(%rax){1to4}, %xmm19, %k0
+# CHECK-NEXT:  -      -     0.50   0.50    -      -      -      -      -      -      -      -     vp2intersectd	%ymm16, %ymm19, %k0
+# CHECK-NEXT:  -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -     vp2intersectd	(%rax), %ymm19, %k0
+# CHECK-NEXT:  -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -     vp2intersectd	(%rax){1to8}, %ymm19, %k0
+# CHECK-NEXT:  -      -     0.50   0.50    -      -      -      -      -      -      -      -     vp2intersectq	%xmm16, %xmm19, %k0
+# CHECK-NEXT:  -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -     vp2intersectq	(%rax), %xmm19, %k0
+# CHECK-NEXT:  -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -     vp2intersectq	(%rax){1to2}, %xmm19, %k0
+# CHECK-NEXT:  -      -     0.50   0.50    -      -      -      -      -      -      -      -     vp2intersectq	%ymm16, %ymm19, %k0
+# CHECK-NEXT:  -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -     vp2intersectq	(%rax), %ymm19, %k0
+# CHECK-NEXT:  -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -     vp2intersectq	(%rax){1to4}, %ymm19, %k0
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vpclmulqdq.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vpclmulqdq.s
new file mode 100644
index 0000000000000..87ba0607e71d1
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vpclmulqdq.s
@@ -0,0 +1,51 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vpclmulqdq    $11, %zmm16, %zmm17, %zmm19
+vpclmulqdq    $11, (%rax), %zmm17, %zmm19
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  4      4     2.00                        vpclmulqdq	$11, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  4      11    2.00    *                   vpclmulqdq	$11, (%rax), %zmm17, %zmm19
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     4.00   4.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpclmulqdq	$11, %zmm16, %zmm17, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpclmulqdq	$11, (%rax), %zmm17, %zmm19
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vpclmulqdqvl.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vpclmulqdqvl.s
new file mode 100644
index 0000000000000..3c80c567227c5
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vpclmulqdqvl.s
@@ -0,0 +1,58 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vpclmulqdq    $11, %xmm16, %xmm17, %xmm19
+vpclmulqdq    $11, (%rax), %xmm17, %xmm19
+
+vpclmulqdq    $11, %ymm16, %ymm17, %ymm19
+vpclmulqdq    $11, (%rax), %ymm17, %ymm19
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  4      4     2.00                        vpclmulqdq	$11, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  4      11    2.00    *                   vpclmulqdq	$11, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  4      4     2.00                        vpclmulqdq	$11, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  4      11    2.00    *                   vpclmulqdq	$11, (%rax), %ymm17, %ymm19
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     8.00   8.00    -      -     1.00   1.00    -     0.67   0.67   0.67   0.67   0.67   0.67    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpclmulqdq	$11, %xmm16, %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpclmulqdq	$11, (%rax), %xmm17, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpclmulqdq	$11, %ymm16, %ymm17, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpclmulqdq	$11, (%rax), %ymm17, %ymm19
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vpopcntdq.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vpopcntdq.s
new file mode 100644
index 0000000000000..43b5165e628e3
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vpopcntdq.s
@@ -0,0 +1,104 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vpopcntd    %zmm1, %zmm0
+vpopcntd    (%rdi), %zmm0
+vpopcntd    (%rdi){1to16}, %zmm0
+
+vpopcntd    %zmm1, %zmm0 {%k1}
+vpopcntd    (%rdi), %zmm0 {%k1}
+vpopcntd    (%rdi){1to16}, %zmm0 {%k1}
+
+vpopcntd    %zmm1, %zmm0 {%k1} {z}
+vpopcntd    (%rdi), %zmm0 {%k1} {z}
+vpopcntd    (%rdi){1to16}, %zmm0 {%k1} {z}
+
+vpopcntq    %zmm1, %zmm0
+vpopcntq    (%rdi), %zmm0
+vpopcntq    (%rdi){1to8}, %zmm0
+
+vpopcntq    %zmm1, %zmm0 {%k1}
+vpopcntq    (%rdi), %zmm0 {%k1}
+vpopcntq    (%rdi){1to8}, %zmm0 {%k1}
+
+vpopcntq    %zmm1, %zmm0 {%k1} {z}
+vpopcntq    (%rdi), %zmm0 {%k1} {z}
+vpopcntq    (%rdi){1to8}, %zmm0 {%k1} {z}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     0.50                        vpopcntd	%zmm1, %zmm0
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntd	(%rdi), %zmm0
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntd	(%rdi){1to16}, %zmm0
+# CHECK-NEXT:  1      1     0.50                        vpopcntd	%zmm1, %zmm0 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntd	(%rdi), %zmm0 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntd	(%rdi){1to16}, %zmm0 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpopcntd	%zmm1, %zmm0 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntd	(%rdi), %zmm0 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntd	(%rdi){1to16}, %zmm0 {%k1} {z}
+# CHECK-NEXT:  1      1     0.50                        vpopcntq	%zmm1, %zmm0
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntq	(%rdi), %zmm0
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntq	(%rdi){1to8}, %zmm0
+# CHECK-NEXT:  1      1     0.50                        vpopcntq	%zmm1, %zmm0 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntq	(%rdi), %zmm0 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntq	(%rdi){1to8}, %zmm0 {%k1}
+# CHECK-NEXT:  1      1     0.50                        vpopcntq	%zmm1, %zmm0 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntq	(%rdi), %zmm0 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntq	(%rdi){1to8}, %zmm0 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     9.00   9.00   9.00   9.00   6.00   6.00    -     4.00   4.00   4.00   4.00   4.00   4.00    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpopcntd	%zmm1, %zmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntd	(%rdi), %zmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntd	(%rdi){1to16}, %zmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpopcntd	%zmm1, %zmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntd	(%rdi), %zmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntd	(%rdi){1to16}, %zmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpopcntd	%zmm1, %zmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntd	(%rdi), %zmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntd	(%rdi){1to16}, %zmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpopcntq	%zmm1, %zmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntq	(%rdi), %zmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntq	(%rdi){1to8}, %zmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpopcntq	%zmm1, %zmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntq	(%rdi), %zmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntq	(%rdi){1to8}, %zmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vpopcntq	%zmm1, %zmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntq	(%rdi), %zmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntq	(%rdi){1to8}, %zmm0 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vpopcntdqvl.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vpopcntdqvl.s
new file mode 100644
index 0000000000000..cfb25692d0ad3
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vpopcntdqvl.s
@@ -0,0 +1,164 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vpopcntd    %xmm1, %xmm0
+vpopcntd    (%rdi), %xmm0
+vpopcntd    (%rdi){1to4}, %xmm0
+
+vpopcntd    %xmm1, %xmm0 {%k1}
+vpopcntd    (%rdi), %xmm0 {%k1}
+vpopcntd    (%rdi){1to4}, %xmm0 {%k1}
+
+vpopcntd    %xmm1, %xmm0 {%k1} {z}
+vpopcntd    (%rdi), %xmm0 {%k1} {z}
+vpopcntd    (%rdi){1to4}, %xmm0 {%k1} {z}
+
+vpopcntd    %ymm1, %ymm0
+vpopcntd    (%rdi), %ymm0
+vpopcntd    (%rdi){1to8}, %ymm0
+
+vpopcntd    %ymm1, %ymm0 {%k1}
+vpopcntd    (%rdi), %ymm0 {%k1}
+vpopcntd    (%rdi){1to8}, %ymm0 {%k1}
+
+vpopcntd    %ymm1, %ymm0 {%k1} {z}
+vpopcntd    (%rdi), %ymm0 {%k1} {z}
+vpopcntd    (%rdi){1to8}, %ymm0 {%k1} {z}
+
+vpopcntq    %xmm1, %xmm0
+vpopcntq    (%rdi), %xmm0
+vpopcntq    (%rdi){1to2}, %xmm0
+
+vpopcntq    %xmm1, %xmm0 {%k1}
+vpopcntq    (%rdi), %xmm0 {%k1}
+vpopcntq    (%rdi){1to2}, %xmm0 {%k1}
+
+vpopcntq    %xmm1, %xmm0 {%k1} {z}
+vpopcntq    (%rdi), %xmm0 {%k1} {z}
+vpopcntq    (%rdi){1to2}, %xmm0 {%k1} {z}
+
+vpopcntq    %ymm1, %ymm0
+vpopcntq    (%rdi), %ymm0
+vpopcntq    (%rdi){1to4}, %ymm0
+
+vpopcntq    %ymm1, %ymm0 {%k1}
+vpopcntq    (%rdi), %ymm0 {%k1}
+vpopcntq    (%rdi){1to4}, %ymm0 {%k1}
+
+vpopcntq    %ymm1, %ymm0 {%k1} {z}
+vpopcntq    (%rdi), %ymm0 {%k1} {z}
+vpopcntq    (%rdi){1to4}, %ymm0 {%k1} {z}
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      2     1.00                        vpopcntd	%xmm1, %xmm0
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntd	(%rdi), %xmm0
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntd	(%rdi){1to4}, %xmm0
+# CHECK-NEXT:  1      2     1.00                        vpopcntd	%xmm1, %xmm0 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntd	(%rdi), %xmm0 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntd	(%rdi){1to4}, %xmm0 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpopcntd	%xmm1, %xmm0 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntd	(%rdi), %xmm0 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntd	(%rdi){1to4}, %xmm0 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpopcntd	%ymm1, %ymm0
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntd	(%rdi), %ymm0
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntd	(%rdi){1to8}, %ymm0
+# CHECK-NEXT:  1      1     0.25                        vpopcntd	%ymm1, %ymm0 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntd	(%rdi), %ymm0 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntd	(%rdi){1to8}, %ymm0 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpopcntd	%ymm1, %ymm0 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntd	(%rdi), %ymm0 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntd	(%rdi){1to8}, %ymm0 {%k1} {z}
+# CHECK-NEXT:  1      2     1.00                        vpopcntq	%xmm1, %xmm0
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntq	(%rdi), %xmm0
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntq	(%rdi){1to2}, %xmm0
+# CHECK-NEXT:  1      2     1.00                        vpopcntq	%xmm1, %xmm0 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntq	(%rdi), %xmm0 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntq	(%rdi){1to2}, %xmm0 {%k1}
+# CHECK-NEXT:  1      2     1.00                        vpopcntq	%xmm1, %xmm0 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntq	(%rdi), %xmm0 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntq	(%rdi){1to2}, %xmm0 {%k1} {z}
+# CHECK-NEXT:  1      1     0.25                        vpopcntq	%ymm1, %ymm0
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntq	(%rdi), %ymm0
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntq	(%rdi){1to4}, %ymm0
+# CHECK-NEXT:  1      1     0.25                        vpopcntq	%ymm1, %ymm0 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntq	(%rdi), %ymm0 {%k1}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntq	(%rdi){1to4}, %ymm0 {%k1}
+# CHECK-NEXT:  1      1     0.25                        vpopcntq	%ymm1, %ymm0 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntq	(%rdi), %ymm0 {%k1} {z}
+# CHECK-NEXT:  1      8     0.50    *                   vpopcntq	(%rdi){1to4}, %ymm0 {%k1} {z}
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     13.50  13.50  7.50   7.50   12.00  12.00   -     8.00   8.00   8.00   8.00   8.00   8.00    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpopcntd	%xmm1, %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntd	(%rdi), %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntd	(%rdi){1to4}, %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpopcntd	%xmm1, %xmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntd	(%rdi), %xmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntd	(%rdi){1to4}, %xmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpopcntd	%xmm1, %xmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntd	(%rdi), %xmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntd	(%rdi){1to4}, %xmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpopcntd	%ymm1, %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntd	(%rdi), %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntd	(%rdi){1to8}, %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpopcntd	%ymm1, %ymm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntd	(%rdi), %ymm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntd	(%rdi){1to8}, %ymm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpopcntd	%ymm1, %ymm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntd	(%rdi), %ymm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntd	(%rdi){1to8}, %ymm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpopcntq	%xmm1, %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntq	(%rdi), %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntq	(%rdi){1to2}, %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpopcntq	%xmm1, %xmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntq	(%rdi), %xmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntq	(%rdi){1to2}, %xmm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpopcntq	%xmm1, %xmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntq	(%rdi), %xmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntq	(%rdi){1to2}, %xmm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpopcntq	%ymm1, %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntq	(%rdi), %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntq	(%rdi){1to4}, %ymm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpopcntq	%ymm1, %ymm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntq	(%rdi), %ymm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntq	(%rdi){1to4}, %ymm0 {%k1}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vpopcntq	%ymm1, %ymm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntq	(%rdi), %ymm0 {%k1} {z}
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpopcntq	(%rdi){1to4}, %ymm0 {%k1} {z}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avxgfni.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avxgfni.s
new file mode 100644
index 0000000000000..f1aa550042dad
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avxgfni.s
@@ -0,0 +1,86 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vgf2p8affineinvqb   $0, %xmm0, %xmm1, %xmm2
+vgf2p8affineinvqb   $0, (%rax), %xmm1, %xmm2
+
+vgf2p8affineinvqb   $0, %ymm0, %ymm1, %ymm2
+vgf2p8affineinvqb   $0, (%rax), %ymm1, %ymm2
+
+vgf2p8affineqb      $0, %xmm0, %xmm1, %xmm2
+vgf2p8affineqb      $0, (%rax), %xmm1, %xmm2
+
+vgf2p8affineqb      $0, %ymm0, %ymm1, %ymm2
+vgf2p8affineqb      $0, (%rax), %ymm1, %ymm2
+
+vgf2p8mulb          %xmm0, %xmm1, %xmm2
+vgf2p8mulb          (%rax), %xmm1, %xmm2
+
+vgf2p8mulb          %ymm0, %ymm1, %ymm2
+vgf2p8mulb          (%rax), %ymm1, %ymm2
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      3     0.50                        vgf2p8affineinvqb	$0, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineinvqb	$0, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vgf2p8affineinvqb	$0, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineinvqb	$0, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      3     0.50                        vgf2p8affineqb	$0, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineqb	$0, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      3     0.50                        vgf2p8affineqb	$0, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vgf2p8affineqb	$0, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      1     0.25                        vgf2p8mulb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   vgf2p8mulb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      1     0.25                        vgf2p8mulb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      8     0.50    *                   vgf2p8mulb	(%rax), %ymm1, %ymm2
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     5.00   1.00   1.00   5.00   3.00   3.00    -     2.00   2.00   2.00   2.00   2.00   2.00    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineinvqb	$0, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineinvqb	$0, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineinvqb	$0, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineqb	$0, %xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     vgf2p8affineqb	$0, %ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8affineqb	$0, (%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vgf2p8mulb	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8mulb	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     vgf2p8mulb	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vgf2p8mulb	(%rax), %ymm1, %ymm2
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avxvnni.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avxvnni.s
new file mode 100644
index 0000000000000..f0d426dbd3a63
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avxvnni.s
@@ -0,0 +1,100 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vpdpbusd    %xmm0, %xmm1, %xmm2
+vpdpbusd    (%rax), %xmm1, %xmm2
+
+vpdpbusd    %ymm0, %ymm1, %ymm2
+vpdpbusd    (%rax), %ymm1, %ymm2
+
+vpdpbusds   %xmm0, %xmm1, %xmm2
+vpdpbusds   (%rax), %xmm1, %xmm2
+
+vpdpbusds   %ymm0, %ymm1, %ymm2
+vpdpbusds   (%rax), %ymm1, %ymm2
+
+vpdpwssd    %xmm0, %xmm1, %xmm2
+vpdpwssd    (%rax), %xmm1, %xmm2
+
+vpdpwssd    %ymm0, %ymm1, %ymm2
+vpdpwssd    (%rax), %ymm1, %ymm2
+
+vpdpwssds   %xmm0, %xmm1, %xmm2
+vpdpwssds   (%rax), %xmm1, %xmm2
+
+vpdpwssds   %ymm0, %ymm1, %ymm2
+vpdpwssds   (%rax), %ymm1, %ymm2
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      4     2.00                        vpdpbusd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     2.00                        vpdpbusd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     2.00                        vpdpbusds	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusds	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     2.00                        vpdpbusds	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vpdpbusds	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     2.00                        vpdpwssd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     2.00                        vpdpwssd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     2.00                        vpdpwssds	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssds	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     2.00                        vpdpwssds	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      10    0.50    *                   vpdpwssds	(%rax), %ymm1, %ymm2
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     20.00  16.00   -     4.00   4.00   4.00    -     2.67   2.67   2.67   2.67   2.67   2.67    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusds	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpbusds	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpbusds	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssds	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpdpwssds	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpdpwssds	(%rax), %ymm1, %ymm2
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-bmi1.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-bmi1.s
new file mode 100644
index 0000000000000..f4888cf81523f
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-bmi1.s
@@ -0,0 +1,135 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+andn        %eax, %ebx, %ecx
+andn        (%rax), %ebx, %ecx
+
+andn        %rax, %rbx, %rcx
+andn        (%rax), %rbx, %rcx
+
+bextr       %eax, %ebx, %ecx
+bextr       %eax, (%rbx), %ecx
+
+bextr       %rax, %rbx, %rcx
+bextr       %rax, (%rbx), %rcx
+
+blsi        %eax, %ecx
+blsi        (%rax), %ecx
+
+blsi        %rax, %rcx
+blsi        (%rax), %rcx
+
+blsmsk      %eax, %ecx
+blsmsk      (%rax), %ecx
+
+blsmsk      %rax, %rcx
+blsmsk      (%rax), %rcx
+
+blsr        %eax, %ecx
+blsr        (%rax), %ecx
+
+blsr        %rax, %rcx
+blsr        (%rax), %rcx
+
+tzcnt       %ax, %cx
+tzcnt       (%rax), %cx
+
+tzcnt       %eax, %ecx
+tzcnt       (%rax), %ecx
+
+tzcnt       %rax, %rcx
+tzcnt       (%rax), %rcx
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     0.25                        andnl	%eax, %ebx, %ecx
+# CHECK-NEXT:  1      5     0.33    *                   andnl	(%rax), %ebx, %ecx
+# CHECK-NEXT:  1      1     0.25                        andnq	%rax, %rbx, %rcx
+# CHECK-NEXT:  1      5     0.33    *                   andnq	(%rax), %rbx, %rcx
+# CHECK-NEXT:  1      1     0.50                        bextrl	%eax, %ebx, %ecx
+# CHECK-NEXT:  2      5     0.50    *                   bextrl	%eax, (%rbx), %ecx
+# CHECK-NEXT:  1      1     0.50                        bextrq	%rax, %rbx, %rcx
+# CHECK-NEXT:  2      5     0.50    *                   bextrq	%rax, (%rbx), %rcx
+# CHECK-NEXT:  1      1     0.25                        blsil	%eax, %ecx
+# CHECK-NEXT:  2      5     0.33    *                   blsil	(%rax), %ecx
+# CHECK-NEXT:  1      1     0.25                        blsiq	%rax, %rcx
+# CHECK-NEXT:  2      5     0.33    *                   blsiq	(%rax), %rcx
+# CHECK-NEXT:  1      1     0.25                        blsmskl	%eax, %ecx
+# CHECK-NEXT:  2      5     0.33    *                   blsmskl	(%rax), %ecx
+# CHECK-NEXT:  1      1     0.25                        blsmskq	%rax, %rcx
+# CHECK-NEXT:  2      5     0.33    *                   blsmskq	(%rax), %rcx
+# CHECK-NEXT:  1      1     0.25                        blsrl	%eax, %ecx
+# CHECK-NEXT:  2      5     0.33    *                   blsrl	(%rax), %ecx
+# CHECK-NEXT:  1      1     0.25                        blsrq	%rax, %rcx
+# CHECK-NEXT:  2      5     0.33    *                   blsrq	(%rax), %rcx
+# CHECK-NEXT:  2      2     1.00                        tzcntw	%ax, %cx
+# CHECK-NEXT:  2      6     0.50    *                   tzcntw	(%rax), %cx
+# CHECK-NEXT:  2      2     0.50                        tzcntl	%eax, %ecx
+# CHECK-NEXT:  2      6     0.50    *                   tzcntl	(%rax), %ecx
+# CHECK-NEXT:  2      2     0.50                        tzcntq	%rax, %rcx
+# CHECK-NEXT:  2      6     0.50    *                   tzcntq	(%rax), %rcx
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 4.33   4.33   4.33   5.00   9.50   9.50   5.00    -      -      -      -      -      -      -      -     4.33   4.33   4.33   4.33   4.33   4.33    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     andnl	%eax, %ebx, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     andnl	(%rax), %ebx, %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     andnq	%rax, %rbx, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     andnq	(%rax), %rbx, %rcx
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     bextrl	%eax, %ebx, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     bextrl	%eax, (%rbx), %ecx
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     bextrq	%rax, %rbx, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     bextrq	%rax, (%rbx), %rcx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     blsil	%eax, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     blsil	(%rax), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     blsiq	%rax, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     blsiq	(%rax), %rcx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     blsmskl	%eax, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     blsmskl	(%rax), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     blsmskq	%rax, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     blsmskq	(%rax), %rcx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     blsrl	%eax, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     blsrl	(%rax), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     blsrq	%rax, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     blsrq	(%rax), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     tzcntw	%ax, %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     tzcntw	(%rax), %cx
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     tzcntl	%eax, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     tzcntl	(%rax), %ecx
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     tzcntq	%rax, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     tzcntq	(%rax), %rcx
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-bmi2.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-bmi2.s
new file mode 100644
index 0000000000000..2e38ed4d8d0d7
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-bmi2.s
@@ -0,0 +1,156 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+bzhi        %eax, %ebx, %ecx
+bzhi        %eax, (%rbx), %ecx
+
+bzhi        %rax, %rbx, %rcx
+bzhi        %rax, (%rbx), %rcx
+
+mulx        %eax, %ebx, %ecx
+mulx        (%rax), %ebx, %ecx
+
+mulx        %rax, %rbx, %rcx
+mulx        (%rax), %rbx, %rcx
+
+pdep        %eax, %ebx, %ecx
+pdep        (%rax), %ebx, %ecx
+
+pdep        %rax, %rbx, %rcx
+pdep        (%rax), %rbx, %rcx
+
+pext        %eax, %ebx, %ecx
+pext        (%rax), %ebx, %ecx
+
+pext        %rax, %rbx, %rcx
+pext        (%rax), %rbx, %rcx
+
+rorx        $1, %eax, %ecx
+rorx        $1, (%rax), %ecx
+
+rorx        $1, %rax, %rcx
+rorx        $1, (%rax), %rcx
+
+sarx        %eax, %ebx, %ecx
+sarx        %eax, (%rbx), %ecx
+
+sarx        %rax, %rbx, %rcx
+sarx        %rax, (%rbx), %rcx
+
+shlx        %eax, %ebx, %ecx
+shlx        %eax, (%rbx), %ecx
+
+shlx        %rax, %rbx, %rcx
+shlx        %rax, (%rbx), %rcx
+
+shrx        %eax, %ebx, %ecx
+shrx        %eax, (%rbx), %ecx
+
+shrx        %rax, %rbx, %rcx
+shrx        %rax, (%rbx), %rcx
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     0.50                        bzhil	%eax, %ebx, %ecx
+# CHECK-NEXT:  2      5     0.50    *                   bzhil	%eax, (%rbx), %ecx
+# CHECK-NEXT:  1      1     0.50                        bzhiq	%rax, %rbx, %rcx
+# CHECK-NEXT:  2      5     0.50    *                   bzhiq	%rax, (%rbx), %rcx
+# CHECK-NEXT:  2      4     1.00                        mulxl	%eax, %ebx, %ecx
+# CHECK-NEXT:  2      8     1.00    *                   mulxl	(%rax), %ebx, %ecx
+# CHECK-NEXT:  2      4     1.00                        mulxq	%rax, %rbx, %rcx
+# CHECK-NEXT:  2      8     1.00    *                   mulxq	(%rax), %rbx, %rcx
+# CHECK-NEXT:  1      3     1.00                        pdepl	%eax, %ebx, %ecx
+# CHECK-NEXT:  1      5     0.33    *                   pdepl	(%rax), %ebx, %ecx
+# CHECK-NEXT:  1      3     1.00                        pdepq	%rax, %rbx, %rcx
+# CHECK-NEXT:  1      5     0.33    *                   pdepq	(%rax), %rbx, %rcx
+# CHECK-NEXT:  1      3     1.00                        pextl	%eax, %ebx, %ecx
+# CHECK-NEXT:  1      5     0.33    *                   pextl	(%rax), %ebx, %ecx
+# CHECK-NEXT:  1      3     1.00                        pextq	%rax, %rbx, %rcx
+# CHECK-NEXT:  1      5     0.33    *                   pextq	(%rax), %rbx, %rcx
+# CHECK-NEXT:  1      1     0.50                        rorxl	$1, %eax, %ecx
+# CHECK-NEXT:  2      5     0.50    *                   rorxl	$1, (%rax), %ecx
+# CHECK-NEXT:  1      1     0.50                        rorxq	$1, %rax, %rcx
+# CHECK-NEXT:  2      5     0.50    *                   rorxq	$1, (%rax), %rcx
+# CHECK-NEXT:  1      1     0.50                        sarxl	%eax, %ebx, %ecx
+# CHECK-NEXT:  2      5     0.50    *                   sarxl	%eax, (%rbx), %ecx
+# CHECK-NEXT:  1      1     0.50                        sarxq	%rax, %rbx, %rcx
+# CHECK-NEXT:  2      5     0.50    *                   sarxq	%rax, (%rbx), %rcx
+# CHECK-NEXT:  1      1     0.50                        shlxl	%eax, %ebx, %ecx
+# CHECK-NEXT:  2      5     0.50    *                   shlxl	%eax, (%rbx), %ecx
+# CHECK-NEXT:  1      1     0.50                        shlxq	%rax, %rbx, %rcx
+# CHECK-NEXT:  2      5     0.50    *                   shlxq	%rax, (%rbx), %rcx
+# CHECK-NEXT:  1      1     0.50                        shrxl	%eax, %ebx, %ecx
+# CHECK-NEXT:  2      5     0.50    *                   shrxl	%eax, (%rbx), %ecx
+# CHECK-NEXT:  1      1     0.50                        shrxq	%rax, %rbx, %rcx
+# CHECK-NEXT:  2      5     0.50    *                   shrxq	%rax, (%rbx), %rcx
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 5.33   5.33   5.33   1.00   19.00  11.00  1.00    -      -      -      -      -      -      -      -     5.33   5.33   5.33   5.33   5.33   5.33    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     bzhil	%eax, %ebx, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     bzhil	%eax, (%rbx), %ecx
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     bzhiq	%rax, %rbx, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     bzhiq	%rax, (%rbx), %rcx
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     mulxl	%eax, %ebx, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     mulxl	(%rax), %ebx, %ecx
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     mulxq	%rax, %rbx, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     mulxq	(%rax), %rbx, %rcx
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     pdepl	%eax, %ebx, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pdepl	(%rax), %ebx, %ecx
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     pdepq	%rax, %rbx, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pdepq	(%rax), %rbx, %rcx
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     pextl	%eax, %ebx, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pextl	(%rax), %ebx, %ecx
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     pextq	%rax, %rbx, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pextq	(%rax), %rbx, %rcx
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rorxl	$1, %eax, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rorxl	$1, (%rax), %ecx
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rorxq	$1, %rax, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rorxq	$1, (%rax), %rcx
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sarxl	%eax, %ebx, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     sarxl	%eax, (%rbx), %ecx
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sarxq	%rax, %rbx, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     sarxq	%rax, (%rbx), %rcx
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shlxl	%eax, %ebx, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     shlxl	%eax, (%rbx), %ecx
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shlxq	%rax, %rbx, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     shlxq	%rax, (%rbx), %rcx
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrxl	%eax, %ebx, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     shrxl	%eax, (%rbx), %ecx
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrxq	%rax, %rbx, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     shrxq	%rax, (%rbx), %rcx
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-clflushopt.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-clflushopt.s
new file mode 100644
index 0000000000000..a78a70a6ae149
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-clflushopt.s
@@ -0,0 +1,48 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+clflushopt (%rax)
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      5     0.33    *      *      U     clflushopt	(%rax)
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     clflushopt	(%rax)
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-clwb.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-clwb.s
new file mode 100644
index 0000000000000..97c05c138b898
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-clwb.s
@@ -0,0 +1,48 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+clwb (%rax)
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      5     0.33    *      *      U     clwb	(%rax)
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     clwb	(%rax)
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-cmov.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-cmov.s
new file mode 100644
index 0000000000000..ad74d00579111
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-cmov.s
@@ -0,0 +1,338 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+cmovow    %si, %di
+cmovnow   %si, %di
+cmovbw    %si, %di
+cmovaew   %si, %di
+cmovew    %si, %di
+cmovnew   %si, %di
+cmovbew   %si, %di
+cmovaw    %si, %di
+cmovsw    %si, %di
+cmovnsw   %si, %di
+cmovpw    %si, %di
+cmovnpw   %si, %di
+cmovlw    %si, %di
+cmovgew   %si, %di
+cmovlew   %si, %di
+cmovgw    %si, %di
+
+cmovow    (%rax), %di
+cmovnow   (%rax), %di
+cmovbw    (%rax), %di
+cmovaew   (%rax), %di
+cmovew    (%rax), %di
+cmovnew   (%rax), %di
+cmovbew   (%rax), %di
+cmovaw    (%rax), %di
+cmovsw    (%rax), %di
+cmovnsw   (%rax), %di
+cmovpw    (%rax), %di
+cmovnpw   (%rax), %di
+cmovlw    (%rax), %di
+cmovgew   (%rax), %di
+cmovlew   (%rax), %di
+cmovgw    (%rax), %di
+
+cmovol    %esi, %edi
+cmovnol   %esi, %edi
+cmovbl    %esi, %edi
+cmovael   %esi, %edi
+cmovel    %esi, %edi
+cmovnel   %esi, %edi
+cmovbel   %esi, %edi
+cmoval    %esi, %edi
+cmovsl    %esi, %edi
+cmovnsl   %esi, %edi
+cmovpl    %esi, %edi
+cmovnpl   %esi, %edi
+cmovll    %esi, %edi
+cmovgel   %esi, %edi
+cmovlel   %esi, %edi
+cmovgl    %esi, %edi
+
+cmovol    (%rax), %edi
+cmovnol   (%rax), %edi
+cmovbl    (%rax), %edi
+cmovael   (%rax), %edi
+cmovel    (%rax), %edi
+cmovnel   (%rax), %edi
+cmovbel   (%rax), %edi
+cmoval    (%rax), %edi
+cmovsl    (%rax), %edi
+cmovnsl   (%rax), %edi
+cmovpl    (%rax), %edi
+cmovnpl   (%rax), %edi
+cmovll    (%rax), %edi
+cmovgel   (%rax), %edi
+cmovlel   (%rax), %edi
+cmovgl    (%rax), %edi
+
+cmovoq    %rsi, %rdi
+cmovnoq   %rsi, %rdi
+cmovbq    %rsi, %rdi
+cmovaeq   %rsi, %rdi
+cmoveq    %rsi, %rdi
+cmovneq   %rsi, %rdi
+cmovbeq   %rsi, %rdi
+cmovaq    %rsi, %rdi
+cmovsq    %rsi, %rdi
+cmovnsq   %rsi, %rdi
+cmovpq    %rsi, %rdi
+cmovnpq   %rsi, %rdi
+cmovlq    %rsi, %rdi
+cmovgeq   %rsi, %rdi
+cmovleq   %rsi, %rdi
+cmovgq    %rsi, %rdi
+
+cmovoq    (%rax), %rdi
+cmovnoq   (%rax), %rdi
+cmovbq    (%rax), %rdi
+cmovaeq   (%rax), %rdi
+cmoveq    (%rax), %rdi
+cmovneq   (%rax), %rdi
+cmovbeq   (%rax), %rdi
+cmovaq    (%rax), %rdi
+cmovsq    (%rax), %rdi
+cmovnsq   (%rax), %rdi
+cmovpq    (%rax), %rdi
+cmovnpq   (%rax), %rdi
+cmovlq    (%rax), %rdi
+cmovgeq   (%rax), %rdi
+cmovleq   (%rax), %rdi
+cmovgq    (%rax), %rdi
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     0.50                        cmovow	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovnow	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovbw	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovaew	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovew	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovnew	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovbew	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovaw	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovsw	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovnsw	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovpw	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovnpw	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovlw	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovgew	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovlew	%si, %di
+# CHECK-NEXT:  1      1     0.50                        cmovgw	%si, %di
+# CHECK-NEXT:  1      5     0.50    *                   cmovow	(%rax), %di
+# CHECK-NEXT:  1      5     0.50    *                   cmovnow	(%rax), %di
+# CHECK-NEXT:  1      5     0.50    *                   cmovbw	(%rax), %di
+# CHECK-NEXT:  1      5     0.50    *                   cmovaew	(%rax), %di
+# CHECK-NEXT:  1      5     0.50    *                   cmovew	(%rax), %di
+# CHECK-NEXT:  1      5     0.50    *                   cmovnew	(%rax), %di
+# CHECK-NEXT:  1      5     0.50    *                   cmovbew	(%rax), %di
+# CHECK-NEXT:  1      5     0.50    *                   cmovaw	(%rax), %di
+# CHECK-NEXT:  1      5     0.50    *                   cmovsw	(%rax), %di
+# CHECK-NEXT:  1      5     0.50    *                   cmovnsw	(%rax), %di
+# CHECK-NEXT:  1      5     0.50    *                   cmovpw	(%rax), %di
+# CHECK-NEXT:  1      5     0.50    *                   cmovnpw	(%rax), %di
+# CHECK-NEXT:  1      5     0.50    *                   cmovlw	(%rax), %di
+# CHECK-NEXT:  1      5     0.50    *                   cmovgew	(%rax), %di
+# CHECK-NEXT:  1      5     0.50    *                   cmovlew	(%rax), %di
+# CHECK-NEXT:  1      5     0.50    *                   cmovgw	(%rax), %di
+# CHECK-NEXT:  1      1     0.50                        cmovol	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovnol	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovbl	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovael	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovel	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovnel	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovbel	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmoval	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovsl	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovnsl	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovpl	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovnpl	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovll	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovgel	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovlel	%esi, %edi
+# CHECK-NEXT:  1      1     0.50                        cmovgl	%esi, %edi
+# CHECK-NEXT:  1      5     0.50    *                   cmovol	(%rax), %edi
+# CHECK-NEXT:  1      5     0.50    *                   cmovnol	(%rax), %edi
+# CHECK-NEXT:  1      5     0.50    *                   cmovbl	(%rax), %edi
+# CHECK-NEXT:  1      5     0.50    *                   cmovael	(%rax), %edi
+# CHECK-NEXT:  1      5     0.50    *                   cmovel	(%rax), %edi
+# CHECK-NEXT:  1      5     0.50    *                   cmovnel	(%rax), %edi
+# CHECK-NEXT:  1      5     0.50    *                   cmovbel	(%rax), %edi
+# CHECK-NEXT:  1      5     0.50    *                   cmoval	(%rax), %edi
+# CHECK-NEXT:  1      5     0.50    *                   cmovsl	(%rax), %edi
+# CHECK-NEXT:  1      5     0.50    *                   cmovnsl	(%rax), %edi
+# CHECK-NEXT:  1      5     0.50    *                   cmovpl	(%rax), %edi
+# CHECK-NEXT:  1      5     0.50    *                   cmovnpl	(%rax), %edi
+# CHECK-NEXT:  1      5     0.50    *                   cmovll	(%rax), %edi
+# CHECK-NEXT:  1      5     0.50    *                   cmovgel	(%rax), %edi
+# CHECK-NEXT:  1      5     0.50    *                   cmovlel	(%rax), %edi
+# CHECK-NEXT:  1      5     0.50    *                   cmovgl	(%rax), %edi
+# CHECK-NEXT:  1      1     0.50                        cmovoq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovnoq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovbq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovaeq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmoveq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovneq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovbeq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovaq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovsq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovnsq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovpq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovnpq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovlq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovgeq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovleq	%rsi, %rdi
+# CHECK-NEXT:  1      1     0.50                        cmovgq	%rsi, %rdi
+# CHECK-NEXT:  1      5     0.50    *                   cmovoq	(%rax), %rdi
+# CHECK-NEXT:  1      5     0.50    *                   cmovnoq	(%rax), %rdi
+# CHECK-NEXT:  1      5     0.50    *                   cmovbq	(%rax), %rdi
+# CHECK-NEXT:  1      5     0.50    *                   cmovaeq	(%rax), %rdi
+# CHECK-NEXT:  1      5     0.50    *                   cmoveq	(%rax), %rdi
+# CHECK-NEXT:  1      5     0.50    *                   cmovneq	(%rax), %rdi
+# CHECK-NEXT:  1      5     0.50    *                   cmovbeq	(%rax), %rdi
+# CHECK-NEXT:  1      5     0.50    *                   cmovaq	(%rax), %rdi
+# CHECK-NEXT:  1      5     0.50    *                   cmovsq	(%rax), %rdi
+# CHECK-NEXT:  1      5     0.50    *                   cmovnsq	(%rax), %rdi
+# CHECK-NEXT:  1      5     0.50    *                   cmovpq	(%rax), %rdi
+# CHECK-NEXT:  1      5     0.50    *                   cmovnpq	(%rax), %rdi
+# CHECK-NEXT:  1      5     0.50    *                   cmovlq	(%rax), %rdi
+# CHECK-NEXT:  1      5     0.50    *                   cmovgeq	(%rax), %rdi
+# CHECK-NEXT:  1      5     0.50    *                   cmovleq	(%rax), %rdi
+# CHECK-NEXT:  1      5     0.50    *                   cmovgq	(%rax), %rdi
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 16.00  16.00  16.00  48.00   -      -     48.00   -      -      -      -      -      -      -      -     16.00  16.00  16.00  16.00  16.00  16.00   -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovow	%si, %di
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovnow	%si, %di
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovbw	%si, %di
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovaew	%si, %di
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovew	%si, %di
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovnew	%si, %di
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovbew	%si, %di
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovaw	%si, %di
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovsw	%si, %di
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovnsw	%si, %di
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovpw	%si, %di
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovnpw	%si, %di
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovlw	%si, %di
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovgew	%si, %di
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovlew	%si, %di
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovgw	%si, %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovow	(%rax), %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovnow	(%rax), %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovbw	(%rax), %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovaew	(%rax), %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovew	(%rax), %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovnew	(%rax), %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovbew	(%rax), %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovaw	(%rax), %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovsw	(%rax), %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovnsw	(%rax), %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovpw	(%rax), %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovnpw	(%rax), %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovlw	(%rax), %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovgew	(%rax), %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovlew	(%rax), %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovgw	(%rax), %di
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovol	%esi, %edi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovnol	%esi, %edi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovbl	%esi, %edi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovael	%esi, %edi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovel	%esi, %edi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovnel	%esi, %edi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovbel	%esi, %edi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmoval	%esi, %edi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovsl	%esi, %edi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovnsl	%esi, %edi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovpl	%esi, %edi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovnpl	%esi, %edi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovll	%esi, %edi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovgel	%esi, %edi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovlel	%esi, %edi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovgl	%esi, %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovol	(%rax), %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovnol	(%rax), %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovbl	(%rax), %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovael	(%rax), %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovel	(%rax), %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovnel	(%rax), %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovbel	(%rax), %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmoval	(%rax), %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovsl	(%rax), %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovnsl	(%rax), %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovpl	(%rax), %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovnpl	(%rax), %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovll	(%rax), %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovgel	(%rax), %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovlel	(%rax), %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovgl	(%rax), %edi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovoq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovnoq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovbq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovaeq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmoveq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovneq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovbeq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovaq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovsq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovnsq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovpq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovnpq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovlq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovgeq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovleq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmovgq	%rsi, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovoq	(%rax), %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovnoq	(%rax), %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovbq	(%rax), %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovaeq	(%rax), %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmoveq	(%rax), %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovneq	(%rax), %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovbeq	(%rax), %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovaq	(%rax), %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovsq	(%rax), %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovnsq	(%rax), %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovpq	(%rax), %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovnpq	(%rax), %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovlq	(%rax), %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovgeq	(%rax), %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovleq	(%rax), %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.50    -      -     0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmovgq	(%rax), %rdi
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-cmpxchg.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-cmpxchg.s
new file mode 100644
index 0000000000000..64feeaf6d4ad8
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-cmpxchg.s
@@ -0,0 +1,57 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+cmpxchg8b  (%rax)
+cmpxchg16b (%rax)
+lock cmpxchg8b  (%rax)
+lock cmpxchg16b (%rax)
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  19     3     6.00    *      *            cmpxchg8b	(%rax)
+# CHECK-NEXT:  28     4     14.75   *      *            cmpxchg16b	(%rax)
+# CHECK-NEXT:  19     3     6.00    *      *            lock		cmpxchg8b	(%rax)
+# CHECK-NEXT:  28     4     14.75   *      *            lock		cmpxchg16b	(%rax)
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -     41.50  41.50  41.50  41.50   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -     6.00   6.00   6.00   6.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpxchg8b	(%rax)
+# CHECK-NEXT:  -      -      -     14.75  14.75  14.75  14.75   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpxchg16b	(%rax)
+# CHECK-NEXT:  -      -      -     6.00   6.00   6.00   6.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     lock		cmpxchg8b	(%rax)
+# CHECK-NEXT:  -      -      -     14.75  14.75  14.75  14.75   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     lock		cmpxchg16b	(%rax)
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-f16c.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-f16c.s
new file mode 100644
index 0000000000000..f8badb55a521c
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-f16c.s
@@ -0,0 +1,72 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vcvtph2ps   %xmm0, %xmm2
+vcvtph2ps   (%rax), %xmm2
+
+vcvtph2ps   %xmm0, %ymm2
+vcvtph2ps   (%rax), %ymm2
+
+vcvtps2ph   $0, %xmm0, %xmm2
+vcvtps2ph   $0, %xmm0, (%rax)
+
+vcvtps2ph   $0, %ymm0, %xmm2
+vcvtps2ph   $0, %ymm0, (%rax)
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      3     0.50                        vcvtph2ps	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   vcvtph2ps	(%rax), %xmm2
+# CHECK-NEXT:  2      4     1.00                        vcvtph2ps	%xmm0, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vcvtph2ps	(%rax), %ymm2
+# CHECK-NEXT:  1      3     1.00                        vcvtps2ph	$0, %xmm0, %xmm2
+# CHECK-NEXT:  2      4     1.00           *            vcvtps2ph	$0, %xmm0, (%rax)
+# CHECK-NEXT:  2      6     1.00                        vcvtps2ph	$0, %ymm0, %xmm2
+# CHECK-NEXT:  3      7     1.00           *            vcvtps2ph	$0, %ymm0, (%rax)
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     6.50   6.50   2.00   2.00   2.00   1.33   1.33   1.33   0.67   0.67   0.67   1.00   1.00
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     vcvtph2ps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtph2ps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtph2ps	%xmm0, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vcvtph2ps	(%rax), %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2ph	$0, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vcvtps2ph	$0, %xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     vcvtps2ph	$0, %ymm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   vcvtps2ph	$0, %ymm0, (%rax)
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-fma.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-fma.s
new file mode 100644
index 0000000000000..3d2dd4c695f2e
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-fma.s
@@ -0,0 +1,716 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vfmadd132pd %xmm0, %xmm1, %xmm2
+vfmadd132pd (%rax), %xmm1, %xmm2
+
+vfmadd132pd %ymm0, %ymm1, %ymm2
+vfmadd132pd (%rax), %ymm1, %ymm2
+
+vfmadd213pd %xmm0, %xmm1, %xmm2
+vfmadd213pd (%rax), %xmm1, %xmm2
+
+vfmadd213pd %ymm0, %ymm1, %ymm2
+vfmadd213pd (%rax), %ymm1, %ymm2
+
+vfmadd231pd %xmm0, %xmm1, %xmm2
+vfmadd231pd (%rax), %xmm1, %xmm2
+
+vfmadd231pd %ymm0, %ymm1, %ymm2
+vfmadd231pd (%rax), %ymm1, %ymm2
+
+vfmadd132ps %xmm0, %xmm1, %xmm2
+vfmadd132ps (%rax), %xmm1, %xmm2
+
+vfmadd132ps %ymm0, %ymm1, %ymm2
+vfmadd132ps (%rax), %ymm1, %ymm2
+
+vfmadd213ps %xmm0, %xmm1, %xmm2
+vfmadd213ps (%rax), %xmm1, %xmm2
+
+vfmadd213ps %ymm0, %ymm1, %ymm2
+vfmadd213ps (%rax), %ymm1, %ymm2
+
+vfmadd231ps %xmm0, %xmm1, %xmm2
+vfmadd231ps (%rax), %xmm1, %xmm2
+
+vfmadd231ps %ymm0, %ymm1, %ymm2
+vfmadd231ps (%rax), %ymm1, %ymm2
+
+vfmadd132sd %xmm0, %xmm1, %xmm2
+vfmadd132sd (%rax), %xmm1, %xmm2
+
+vfmadd213sd %xmm0, %xmm1, %xmm2
+vfmadd213sd (%rax), %xmm1, %xmm2
+
+vfmadd231sd %xmm0, %xmm1, %xmm2
+vfmadd231sd (%rax), %xmm1, %xmm2
+
+vfmadd132ss %xmm0, %xmm1, %xmm2
+vfmadd132ss (%rax), %xmm1, %xmm2
+
+vfmadd213ss %xmm0, %xmm1, %xmm2
+vfmadd213ss (%rax), %xmm1, %xmm2
+
+vfmadd231ss %xmm0, %xmm1, %xmm2
+vfmadd231ss (%rax), %xmm1, %xmm2
+
+vfmaddsub132pd %xmm0, %xmm1, %xmm2
+vfmaddsub132pd (%rax), %xmm1, %xmm2
+
+vfmaddsub132pd %ymm0, %ymm1, %ymm2
+vfmaddsub132pd (%rax), %ymm1, %ymm2
+
+vfmaddsub213pd %xmm0, %xmm1, %xmm2
+vfmaddsub213pd (%rax), %xmm1, %xmm2
+
+vfmaddsub213pd %ymm0, %ymm1, %ymm2
+vfmaddsub213pd (%rax), %ymm1, %ymm2
+
+vfmaddsub231pd %xmm0, %xmm1, %xmm2
+vfmaddsub231pd (%rax), %xmm1, %xmm2
+
+vfmaddsub231pd %ymm0, %ymm1, %ymm2
+vfmaddsub231pd (%rax), %ymm1, %ymm2
+
+vfmaddsub132ps %xmm0, %xmm1, %xmm2
+vfmaddsub132ps (%rax), %xmm1, %xmm2
+
+vfmaddsub132ps %ymm0, %ymm1, %ymm2
+vfmaddsub132ps (%rax), %ymm1, %ymm2
+
+vfmaddsub213ps %xmm0, %xmm1, %xmm2
+vfmaddsub213ps (%rax), %xmm1, %xmm2
+
+vfmaddsub213ps %ymm0, %ymm1, %ymm2
+vfmaddsub213ps (%rax), %ymm1, %ymm2
+
+vfmaddsub231ps %xmm0, %xmm1, %xmm2
+vfmaddsub231ps (%rax), %xmm1, %xmm2
+
+vfmaddsub231ps %ymm0, %ymm1, %ymm2
+vfmaddsub231ps (%rax), %ymm1, %ymm2
+
+vfmsub132pd %xmm0, %xmm1, %xmm2
+vfmsub132pd (%rax), %xmm1, %xmm2
+
+vfmsub132pd %ymm0, %ymm1, %ymm2
+vfmsub132pd (%rax), %ymm1, %ymm2
+
+vfmsub213pd %xmm0, %xmm1, %xmm2
+vfmsub213pd (%rax), %xmm1, %xmm2
+
+vfmsub213pd %ymm0, %ymm1, %ymm2
+vfmsub213pd (%rax), %ymm1, %ymm2
+
+vfmsub231pd %xmm0, %xmm1, %xmm2
+vfmsub231pd (%rax), %xmm1, %xmm2
+
+vfmsub231pd %ymm0, %ymm1, %ymm2
+vfmsub231pd (%rax), %ymm1, %ymm2
+
+vfmsub132ps %xmm0, %xmm1, %xmm2
+vfmsub132ps (%rax), %xmm1, %xmm2
+
+vfmsub132ps %ymm0, %ymm1, %ymm2
+vfmsub132ps (%rax), %ymm1, %ymm2
+
+vfmsub213ps %xmm0, %xmm1, %xmm2
+vfmsub213ps (%rax), %xmm1, %xmm2
+
+vfmsub213ps %ymm0, %ymm1, %ymm2
+vfmsub213ps (%rax), %ymm1, %ymm2
+
+vfmsub231ps %xmm0, %xmm1, %xmm2
+vfmsub231ps (%rax), %xmm1, %xmm2
+
+vfmsub231ps %ymm0, %ymm1, %ymm2
+vfmsub231ps (%rax), %ymm1, %ymm2
+
+vfmsub132sd %xmm0, %xmm1, %xmm2
+vfmsub132sd (%rax), %xmm1, %xmm2
+
+vfmsub213sd %xmm0, %xmm1, %xmm2
+vfmsub213sd (%rax), %xmm1, %xmm2
+
+vfmsub231sd %xmm0, %xmm1, %xmm2
+vfmsub231sd (%rax), %xmm1, %xmm2
+
+vfmsub132ss %xmm0, %xmm1, %xmm2
+vfmsub132ss (%rax), %xmm1, %xmm2
+
+vfmsub213ss %xmm0, %xmm1, %xmm2
+vfmsub213ss (%rax), %xmm1, %xmm2
+
+vfmsub231ss %xmm0, %xmm1, %xmm2
+vfmsub231ss (%rax), %xmm1, %xmm2
+
+vfmsubadd132pd %xmm0, %xmm1, %xmm2
+vfmsubadd132pd (%rax), %xmm1, %xmm2
+
+vfmsubadd132pd %ymm0, %ymm1, %ymm2
+vfmsubadd132pd (%rax), %ymm1, %ymm2
+
+vfmsubadd213pd %xmm0, %xmm1, %xmm2
+vfmsubadd213pd (%rax), %xmm1, %xmm2
+
+vfmsubadd213pd %ymm0, %ymm1, %ymm2
+vfmsubadd213pd (%rax), %ymm1, %ymm2
+
+vfmsubadd231pd %xmm0, %xmm1, %xmm2
+vfmsubadd231pd (%rax), %xmm1, %xmm2
+
+vfmsubadd231pd %ymm0, %ymm1, %ymm2
+vfmsubadd231pd (%rax), %ymm1, %ymm2
+
+vfmsubadd132ps %xmm0, %xmm1, %xmm2
+vfmsubadd132ps (%rax), %xmm1, %xmm2
+
+vfmsubadd132ps %ymm0, %ymm1, %ymm2
+vfmsubadd132ps (%rax), %ymm1, %ymm2
+
+vfmsubadd213ps %xmm0, %xmm1, %xmm2
+vfmsubadd213ps (%rax), %xmm1, %xmm2
+
+vfmsubadd213ps %ymm0, %ymm1, %ymm2
+vfmsubadd213ps (%rax), %ymm1, %ymm2
+
+vfmsubadd231ps %xmm0, %xmm1, %xmm2
+vfmsubadd231ps (%rax), %xmm1, %xmm2
+
+vfmsubadd231ps %ymm0, %ymm1, %ymm2
+vfmsubadd231ps (%rax), %ymm1, %ymm2
+
+vfnmadd132pd %xmm0, %xmm1, %xmm2
+vfnmadd132pd (%rax), %xmm1, %xmm2
+
+vfnmadd132pd %ymm0, %ymm1, %ymm2
+vfnmadd132pd (%rax), %ymm1, %ymm2
+
+vfnmadd213pd %xmm0, %xmm1, %xmm2
+vfnmadd213pd (%rax), %xmm1, %xmm2
+
+vfnmadd213pd %ymm0, %ymm1, %ymm2
+vfnmadd213pd (%rax), %ymm1, %ymm2
+
+vfnmadd231pd %xmm0, %xmm1, %xmm2
+vfnmadd231pd (%rax), %xmm1, %xmm2
+
+vfnmadd231pd %ymm0, %ymm1, %ymm2
+vfnmadd231pd (%rax), %ymm1, %ymm2
+
+vfnmadd132ps %xmm0, %xmm1, %xmm2
+vfnmadd132ps (%rax), %xmm1, %xmm2
+
+vfnmadd132ps %ymm0, %ymm1, %ymm2
+vfnmadd132ps (%rax), %ymm1, %ymm2
+
+vfnmadd213ps %xmm0, %xmm1, %xmm2
+vfnmadd213ps (%rax), %xmm1, %xmm2
+
+vfnmadd213ps %ymm0, %ymm1, %ymm2
+vfnmadd213ps (%rax), %ymm1, %ymm2
+
+vfnmadd231ps %xmm0, %xmm1, %xmm2
+vfnmadd231ps (%rax), %xmm1, %xmm2
+
+vfnmadd231ps %ymm0, %ymm1, %ymm2
+vfnmadd231ps (%rax), %ymm1, %ymm2
+
+vfnmadd132sd %xmm0, %xmm1, %xmm2
+vfnmadd132sd (%rax), %xmm1, %xmm2
+
+vfnmadd213sd %xmm0, %xmm1, %xmm2
+vfnmadd213sd (%rax), %xmm1, %xmm2
+
+vfnmadd231sd %xmm0, %xmm1, %xmm2
+vfnmadd231sd (%rax), %xmm1, %xmm2
+
+vfnmadd132ss %xmm0, %xmm1, %xmm2
+vfnmadd132ss (%rax), %xmm1, %xmm2
+
+vfnmadd213ss %xmm0, %xmm1, %xmm2
+vfnmadd213ss (%rax), %xmm1, %xmm2
+
+vfnmadd231ss %xmm0, %xmm1, %xmm2
+vfnmadd231ss (%rax), %xmm1, %xmm2
+
+vfnmsub132pd %xmm0, %xmm1, %xmm2
+vfnmsub132pd (%rax), %xmm1, %xmm2
+
+vfnmsub132pd %ymm0, %ymm1, %ymm2
+vfnmsub132pd (%rax), %ymm1, %ymm2
+
+vfnmsub213pd %xmm0, %xmm1, %xmm2
+vfnmsub213pd (%rax), %xmm1, %xmm2
+
+vfnmsub213pd %ymm0, %ymm1, %ymm2
+vfnmsub213pd (%rax), %ymm1, %ymm2
+
+vfnmsub231pd %xmm0, %xmm1, %xmm2
+vfnmsub231pd (%rax), %xmm1, %xmm2
+
+vfnmsub231pd %ymm0, %ymm1, %ymm2
+vfnmsub231pd (%rax), %ymm1, %ymm2
+
+vfnmsub132ps %xmm0, %xmm1, %xmm2
+vfnmsub132ps (%rax), %xmm1, %xmm2
+
+vfnmsub132ps %ymm0, %ymm1, %ymm2
+vfnmsub132ps (%rax), %ymm1, %ymm2
+
+vfnmsub213ps %xmm0, %xmm1, %xmm2
+vfnmsub213ps (%rax), %xmm1, %xmm2
+
+vfnmsub213ps %ymm0, %ymm1, %ymm2
+vfnmsub213ps (%rax), %ymm1, %ymm2
+
+vfnmsub231ps %xmm0, %xmm1, %xmm2
+vfnmsub231ps (%rax), %xmm1, %xmm2
+
+vfnmsub231ps %ymm0, %ymm1, %ymm2
+vfnmsub231ps (%rax), %ymm1, %ymm2
+
+vfnmsub132sd %xmm0, %xmm1, %xmm2
+vfnmsub132sd (%rax), %xmm1, %xmm2
+
+vfnmsub213sd %xmm0, %xmm1, %xmm2
+vfnmsub213sd (%rax), %xmm1, %xmm2
+
+vfnmsub231sd %xmm0, %xmm1, %xmm2
+vfnmsub231sd (%rax), %xmm1, %xmm2
+
+vfnmsub132ss %xmm0, %xmm1, %xmm2
+vfnmsub132ss (%rax), %xmm1, %xmm2
+
+vfnmsub213ss %xmm0, %xmm1, %xmm2
+vfnmsub213ss (%rax), %xmm1, %xmm2
+
+vfnmsub231ss %xmm0, %xmm1, %xmm2
+vfnmsub231ss (%rax), %xmm1, %xmm2
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      4     1.00                        vfmadd132pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmadd132pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmadd132pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmadd132pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmadd213pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmadd213pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmadd213pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmadd213pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmadd231pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmadd231pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmadd231pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmadd231pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmadd132ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmadd132ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmadd132ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmadd132ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmadd213ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmadd213ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmadd213ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmadd213ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmadd231ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmadd231ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmadd231ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmadd231ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmadd132sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmadd132sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmadd213sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmadd213sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmadd231sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmadd231sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmadd132ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmadd132ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmadd213ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmadd213ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmadd231ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmadd231ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmaddsub132pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmaddsub132pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmaddsub132pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmaddsub132pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmaddsub213pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmaddsub213pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmaddsub213pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmaddsub213pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmaddsub231pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmaddsub231pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmaddsub231pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmaddsub231pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmaddsub132ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmaddsub132ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmaddsub132ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmaddsub132ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmaddsub213ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmaddsub213ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmaddsub213ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmaddsub213ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmaddsub231ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmaddsub231ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmaddsub231ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmaddsub231ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmsub132pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsub132pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmsub132pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsub132pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmsub213pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsub213pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmsub213pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsub213pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmsub231pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsub231pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmsub231pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsub231pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmsub132ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsub132ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmsub132ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsub132ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmsub213ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsub213ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmsub213ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsub213ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmsub231ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsub231ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmsub231ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsub231ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmsub132sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsub132sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmsub213sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsub213sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmsub231sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsub231sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmsub132ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsub132ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmsub213ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsub213ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmsub231ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsub231ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmsubadd132pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsubadd132pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmsubadd132pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsubadd132pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmsubadd213pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsubadd213pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmsubadd213pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsubadd213pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmsubadd231pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsubadd231pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmsubadd231pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsubadd231pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmsubadd132ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsubadd132ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmsubadd132ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsubadd132ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmsubadd213ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsubadd213ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmsubadd213ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsubadd213ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfmsubadd231ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsubadd231ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfmsubadd231ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfmsubadd231ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfnmadd132pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmadd132pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmadd132pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmadd132pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfnmadd213pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmadd213pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmadd213pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmadd213pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfnmadd231pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmadd231pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmadd231pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmadd231pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfnmadd132ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmadd132ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmadd132ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmadd132ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfnmadd213ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmadd213ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmadd213ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmadd213ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfnmadd231ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmadd231ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmadd231ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmadd231ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfnmadd132sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmadd132sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmadd213sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmadd213sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmadd231sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmadd231sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmadd132ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmadd132ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmadd213ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmadd213ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmadd231ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmadd231ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmsub132pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmsub132pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmsub132pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmsub132pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfnmsub213pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmsub213pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmsub213pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmsub213pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfnmsub231pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmsub231pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmsub231pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmsub231pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfnmsub132ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmsub132ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmsub132ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmsub132ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfnmsub213ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmsub213ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmsub213ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmsub213ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfnmsub231ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmsub231ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmsub231ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmsub231ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  1      4     1.00                        vfnmsub132sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmsub132sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmsub213sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmsub213sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmsub231sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmsub231sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmsub132ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmsub132ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmsub213ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmsub213ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  1      4     1.00                        vfnmsub231ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   vfnmsub231ss	(%rax), %xmm1, %xmm2
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     192.00 192.00  -      -     48.00  48.00   -     32.00  32.00  32.00  32.00  32.00  32.00   -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmadd132pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmadd132pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmadd132pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmadd132pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmadd213pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmadd213pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmadd213pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmadd213pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmadd231pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmadd231pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmadd231pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmadd231pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmadd132ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmadd132ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmadd132ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmadd132ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmadd213ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmadd213ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmadd213ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmadd213ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmadd231ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmadd231ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmadd231ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmadd231ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmadd132sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmadd132sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmadd213sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmadd213sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmadd231sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmadd231sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmadd132ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmadd132ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmadd213ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmadd213ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmadd231ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmadd231ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmaddsub132pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmaddsub132pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmaddsub132pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmaddsub132pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmaddsub213pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmaddsub213pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmaddsub213pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmaddsub213pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmaddsub231pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmaddsub231pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmaddsub231pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmaddsub231pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmaddsub132ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmaddsub132ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmaddsub132ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmaddsub132ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmaddsub213ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmaddsub213ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmaddsub213ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmaddsub213ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmaddsub231ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmaddsub231ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmaddsub231ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmaddsub231ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsub132pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsub132pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsub132pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsub132pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsub213pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsub213pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsub213pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsub213pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsub231pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsub231pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsub231pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsub231pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsub132ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsub132ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsub132ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsub132ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsub213ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsub213ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsub213ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsub213ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsub231ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsub231ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsub231ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsub231ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsub132sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsub132sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsub213sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsub213sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsub231sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsub231sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsub132ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsub132ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsub213ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsub213ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsub231ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsub231ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsubadd132pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsubadd132pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsubadd132pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsubadd132pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsubadd213pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsubadd213pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsubadd213pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsubadd213pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsubadd231pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsubadd231pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsubadd231pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsubadd231pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsubadd132ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsubadd132ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsubadd132ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsubadd132ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsubadd213ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsubadd213ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsubadd213ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsubadd213ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsubadd231ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsubadd231ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfmsubadd231ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfmsubadd231ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmadd132pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmadd132pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmadd132pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmadd132pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmadd213pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmadd213pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmadd213pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmadd213pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmadd231pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmadd231pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmadd231pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmadd231pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmadd132ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmadd132ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmadd132ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmadd132ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmadd213ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmadd213ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmadd213ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmadd213ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmadd231ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmadd231ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmadd231ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmadd231ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmadd132sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmadd132sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmadd213sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmadd213sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmadd231sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmadd231sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmadd132ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmadd132ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmadd213ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmadd213ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmadd231ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmadd231ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmsub132pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmsub132pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmsub132pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmsub132pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmsub213pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmsub213pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmsub213pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmsub213pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmsub231pd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmsub231pd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmsub231pd	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmsub231pd	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmsub132ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmsub132ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmsub132ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmsub132ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmsub213ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmsub213ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmsub213ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmsub213ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmsub231ps	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmsub231ps	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmsub231ps	%ymm0, %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmsub231ps	(%rax), %ymm1, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmsub132sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmsub132sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmsub213sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmsub213sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmsub231sd	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmsub231sd	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmsub132ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmsub132ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmsub213ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmsub213ss	(%rax), %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vfnmsub231ss	%xmm0, %xmm1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vfnmsub231ss	(%rax), %xmm1, %xmm2
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-fsgsbase.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-fsgsbase.s
new file mode 100644
index 0000000000000..ee00cdbc10bff
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-fsgsbase.s
@@ -0,0 +1,72 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+rdfsbase %eax
+rdfsbase %rax
+
+rdgsbase %eax
+rdgsbase %rax
+
+wrfsbase %edi
+wrfsbase %rdi
+
+wrgsbase %edi
+wrgsbase %rdi
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  100    100   25.00   *      *      U     rdfsbasel	%eax
+# CHECK-NEXT:  100    100   25.00   *      *      U     rdfsbaseq	%rax
+# CHECK-NEXT:  100    100   25.00   *      *      U     rdgsbasel	%eax
+# CHECK-NEXT:  100    100   25.00   *      *      U     rdgsbaseq	%rax
+# CHECK-NEXT:  100    100   25.00   *      *      U     wrfsbasel	%edi
+# CHECK-NEXT:  100    100   25.00   *      *      U     wrfsbaseq	%rdi
+# CHECK-NEXT:  100    100   25.00   *      *      U     wrgsbasel	%edi
+# CHECK-NEXT:  100    100   25.00   *      *      U     wrgsbaseq	%rdi
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -     200.00 200.00 200.00 200.00  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rdfsbasel	%eax
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rdfsbaseq	%rax
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rdgsbasel	%eax
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rdgsbaseq	%rax
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     wrfsbasel	%edi
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     wrfsbaseq	%rdi
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     wrgsbasel	%edi
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     wrgsbaseq	%rdi
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-gfni.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-gfni.s
new file mode 100644
index 0000000000000..7cbc22c5183be
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-gfni.s
@@ -0,0 +1,65 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+gf2p8affineinvqb    $0, %xmm0, %xmm1
+gf2p8affineinvqb    $0, (%rax), %xmm1
+
+gf2p8affineqb       $0, %xmm0, %xmm1
+gf2p8affineqb       $0, (%rax), %xmm1
+
+gf2p8mulb           %xmm0, %xmm1
+gf2p8mulb           (%rax), %xmm1
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      3     0.50                        gf2p8affineinvqb	$0, %xmm0, %xmm1
+# CHECK-NEXT:  1      10    0.50    *                   gf2p8affineinvqb	$0, (%rax), %xmm1
+# CHECK-NEXT:  1      3     0.50                        gf2p8affineqb	$0, %xmm0, %xmm1
+# CHECK-NEXT:  1      10    0.50    *                   gf2p8affineqb	$0, (%rax), %xmm1
+# CHECK-NEXT:  1      1     0.25                        gf2p8mulb	%xmm0, %xmm1
+# CHECK-NEXT:  1      8     0.50    *                   gf2p8mulb	(%rax), %xmm1
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.50   0.50   0.50   2.50   1.50   1.50    -     1.00   1.00   1.00   1.00   1.00   1.00    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     gf2p8affineinvqb	$0, %xmm0, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     gf2p8affineinvqb	$0, (%rax), %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     gf2p8affineqb	$0, %xmm0, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     gf2p8affineqb	$0, (%rax), %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     gf2p8mulb	%xmm0, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     gf2p8mulb	(%rax), %xmm1
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-lea.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-lea.s
new file mode 100644
index 0000000000000..d259949af3846
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-lea.s
@@ -0,0 +1,452 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+lea 0(), %cx
+lea 0(), %ecx
+lea 0(), %rcx
+lea (%eax), %cx
+lea (%eax), %ecx
+lea (%eax), %rcx
+lea (%rax), %cx
+lea (%rax), %ecx
+lea (%rax), %rcx
+lea (, %ebx), %cx
+lea (, %ebx), %ecx
+lea (, %ebx), %rcx
+lea (, %rbx), %cx
+lea (, %rbx), %ecx
+lea (, %rbx), %rcx
+lea (, %ebx, 1), %cx
+lea (, %ebx, 1), %ecx
+lea (, %ebx, 1), %rcx
+lea (, %rbx, 1), %cx
+lea (, %rbx, 1), %ecx
+lea (, %rbx, 1), %rcx
+lea (, %ebx, 2), %cx
+lea (, %ebx, 2), %ecx
+lea (, %ebx, 2), %rcx
+lea (, %rbx, 2), %cx
+lea (, %rbx, 2), %ecx
+lea (, %rbx, 2), %rcx
+lea (%eax, %ebx), %cx
+lea (%eax, %ebx), %ecx
+lea (%eax, %ebx), %rcx
+lea (%rax, %rbx), %cx
+lea (%rax, %rbx), %ecx
+lea (%rax, %rbx), %rcx
+lea (%eax, %ebx, 1), %cx
+lea (%eax, %ebx, 1), %ecx
+lea (%eax, %ebx, 1), %rcx
+lea (%rax, %rbx, 1), %cx
+lea (%rax, %rbx, 1), %ecx
+lea (%rax, %rbx, 1), %rcx
+lea (%eax, %ebx, 2), %cx
+lea (%eax, %ebx, 2), %ecx
+lea (%eax, %ebx, 2), %rcx
+lea (%rax, %rbx, 2), %cx
+lea (%rax, %rbx, 2), %ecx
+lea (%rax, %rbx, 2), %rcx
+
+lea -16(), %cx
+lea -16(), %ecx
+lea -16(), %rcx
+lea -16(%eax), %cx
+lea -16(%eax), %ecx
+lea -16(%eax), %rcx
+lea -16(%rax), %cx
+lea -16(%rax), %ecx
+lea -16(%rax), %rcx
+lea -16(, %ebx), %cx
+lea -16(, %ebx), %ecx
+lea -16(, %ebx), %rcx
+lea -16(, %rbx), %cx
+lea -16(, %rbx), %ecx
+lea -16(, %rbx), %rcx
+lea -16(, %ebx, 1), %cx
+lea -16(, %ebx, 1), %ecx
+lea -16(, %ebx, 1), %rcx
+lea -16(, %rbx, 1), %cx
+lea -16(, %rbx, 1), %ecx
+lea -16(, %rbx, 1), %rcx
+lea -16(, %ebx, 2), %cx
+lea -16(, %ebx, 2), %ecx
+lea -16(, %ebx, 2), %rcx
+lea -16(, %rbx, 2), %cx
+lea -16(, %rbx, 2), %ecx
+lea -16(, %rbx, 2), %rcx
+lea -16(%eax, %ebx), %cx
+lea -16(%eax, %ebx), %ecx
+lea -16(%eax, %ebx), %rcx
+lea -16(%rax, %rbx), %cx
+lea -16(%rax, %rbx), %ecx
+lea -16(%rax, %rbx), %rcx
+lea -16(%eax, %ebx, 1), %cx
+lea -16(%eax, %ebx, 1), %ecx
+lea -16(%eax, %ebx, 1), %rcx
+lea -16(%rax, %rbx, 1), %cx
+lea -16(%rax, %rbx, 1), %ecx
+lea -16(%rax, %rbx, 1), %rcx
+lea -16(%eax, %ebx, 2), %cx
+lea -16(%eax, %ebx, 2), %ecx
+lea -16(%eax, %ebx, 2), %rcx
+lea -16(%rax, %rbx, 2), %cx
+lea -16(%rax, %rbx, 2), %ecx
+lea -16(%rax, %rbx, 2), %rcx
+
+lea 1024(), %cx
+lea 1024(), %ecx
+lea 1024(), %rcx
+lea 1024(%eax), %cx
+lea 1024(%eax), %ecx
+lea 1024(%eax), %rcx
+lea 1024(%rax), %cx
+lea 1024(%rax), %ecx
+lea 1024(%rax), %rcx
+lea 1024(, %ebx), %cx
+lea 1024(, %ebx), %ecx
+lea 1024(, %ebx), %rcx
+lea 1024(, %rbx), %cx
+lea 1024(, %rbx), %ecx
+lea 1024(, %rbx), %rcx
+lea 1024(, %ebx, 1), %cx
+lea 1024(, %ebx, 1), %ecx
+lea 1024(, %ebx, 1), %rcx
+lea 1024(, %rbx, 1), %cx
+lea 1024(, %rbx, 1), %ecx
+lea 1024(, %rbx, 1), %rcx
+lea 1024(, %ebx, 2), %cx
+lea 1024(, %ebx, 2), %ecx
+lea 1024(, %ebx, 2), %rcx
+lea 1024(, %rbx, 2), %cx
+lea 1024(, %rbx, 2), %ecx
+lea 1024(, %rbx, 2), %rcx
+lea 1024(%eax, %ebx), %cx
+lea 1024(%eax, %ebx), %ecx
+lea 1024(%eax, %ebx), %rcx
+lea 1024(%rax, %rbx), %cx
+lea 1024(%rax, %rbx), %ecx
+lea 1024(%rax, %rbx), %rcx
+lea 1024(%eax, %ebx, 1), %cx
+lea 1024(%eax, %ebx, 1), %ecx
+lea 1024(%eax, %ebx, 1), %rcx
+lea 1024(%rax, %rbx, 1), %cx
+lea 1024(%rax, %rbx, 1), %ecx
+lea 1024(%rax, %rbx, 1), %rcx
+lea 1024(%eax, %ebx, 2), %cx
+lea 1024(%eax, %ebx, 2), %ecx
+lea 1024(%eax, %ebx, 2), %rcx
+lea 1024(%rax, %rbx, 2), %cx
+lea 1024(%rax, %rbx, 2), %ecx
+lea 1024(%rax, %rbx, 2), %rcx
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  2      2     1.00                        leaw	0, %cx
+# CHECK-NEXT:  1      1     0.33                        leal	0, %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	0, %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(%eax), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	(%eax), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	(%eax), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(%rax), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	(%rax), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	(%rax), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(,%ebx), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	(,%ebx), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	(,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(,%rbx), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	(,%rbx), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	(,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(,%ebx), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	(,%ebx), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	(,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(,%rbx), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	(,%rbx), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	(,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(,%ebx,2), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	(,%ebx,2), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	(,%ebx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(,%rbx,2), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	(,%rbx,2), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	(,%rbx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(%eax,%ebx), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	(%eax,%ebx), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	(%eax,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(%rax,%rbx), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	(%rax,%rbx), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	(%rax,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(%eax,%ebx), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	(%eax,%ebx), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	(%eax,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(%rax,%rbx), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	(%rax,%rbx), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	(%rax,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(%eax,%ebx,2), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	(%eax,%ebx,2), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	(%eax,%ebx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	(%rax,%rbx,2), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	(%rax,%rbx,2), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	(%rax,%rbx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16, %cx
+# CHECK-NEXT:  1      1     0.33                        leal	-16, %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	-16, %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(%eax), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	-16(%eax), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	-16(%eax), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(%rax), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	-16(%rax), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	-16(%rax), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(,%ebx), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	-16(,%ebx), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	-16(,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(,%rbx), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	-16(,%rbx), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	-16(,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(,%ebx), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	-16(,%ebx), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	-16(,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(,%rbx), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	-16(,%rbx), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	-16(,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(,%ebx,2), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	-16(,%ebx,2), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	-16(,%ebx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(,%rbx,2), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	-16(,%rbx,2), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	-16(,%rbx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(%eax,%ebx), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	-16(%eax,%ebx), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	-16(%eax,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(%rax,%rbx), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	-16(%rax,%rbx), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	-16(%rax,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(%eax,%ebx), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	-16(%eax,%ebx), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	-16(%eax,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(%rax,%rbx), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	-16(%rax,%rbx), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	-16(%rax,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(%eax,%ebx,2), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	-16(%eax,%ebx,2), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	-16(%eax,%ebx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	-16(%rax,%rbx,2), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	-16(%rax,%rbx,2), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	-16(%rax,%rbx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024, %cx
+# CHECK-NEXT:  1      1     0.33                        leal	1024, %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	1024, %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(%eax), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	1024(%eax), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	1024(%eax), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(%rax), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	1024(%rax), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	1024(%rax), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(,%ebx), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	1024(,%ebx), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	1024(,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(,%rbx), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	1024(,%rbx), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	1024(,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(,%ebx), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	1024(,%ebx), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	1024(,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(,%rbx), %cx
+# CHECK-NEXT:  1      1     0.33                        leal	1024(,%rbx), %ecx
+# CHECK-NEXT:  1      1     0.33                        leaq	1024(,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(,%ebx,2), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	1024(,%ebx,2), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	1024(,%ebx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(,%rbx,2), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	1024(,%rbx,2), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	1024(,%rbx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(%eax,%ebx), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	1024(%eax,%ebx), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	1024(%eax,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(%rax,%rbx), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	1024(%rax,%rbx), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	1024(%rax,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(%eax,%ebx), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	1024(%eax,%ebx), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	1024(%eax,%ebx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(%rax,%rbx), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	1024(%rax,%rbx), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	1024(%rax,%rbx), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(%eax,%ebx,2), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	1024(%eax,%ebx,2), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	1024(%eax,%ebx,2), %rcx
+# CHECK-NEXT:  2      2     1.00                        leaw	1024(%rax,%rbx,2), %cx
+# CHECK-NEXT:  2      2     0.25                        leal	1024(%rax,%rbx,2), %ecx
+# CHECK-NEXT:  2      2     0.25                        leaq	1024(%rax,%rbx,2), %rcx
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 16.67  16.67  16.67  55.00  55.00  55.00  55.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	0, %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	0, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	0, %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	(%eax), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	(%eax), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	(%eax), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	(%rax), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	(%rax), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	(%rax), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	(,%ebx), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	(,%ebx), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	(,%ebx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	(,%rbx), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	(,%rbx), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	(,%rbx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	(,%ebx), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	(,%ebx), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	(,%ebx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	(,%rbx), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	(,%rbx), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	(,%rbx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	(,%ebx,2), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	(,%ebx,2), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	(,%ebx,2), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	(,%rbx,2), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	(,%rbx,2), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	(,%rbx,2), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	(%eax,%ebx), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	(%eax,%ebx), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	(%eax,%ebx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	(%rax,%rbx), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	(%rax,%rbx), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	(%rax,%rbx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	(%eax,%ebx), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	(%eax,%ebx), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	(%eax,%ebx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	(%rax,%rbx), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	(%rax,%rbx), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	(%rax,%rbx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	(%eax,%ebx,2), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	(%eax,%ebx,2), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	(%eax,%ebx,2), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	(%rax,%rbx,2), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	(%rax,%rbx,2), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	(%rax,%rbx,2), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	-16, %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	-16, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	-16, %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	-16(%eax), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	-16(%eax), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	-16(%eax), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	-16(%rax), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	-16(%rax), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	-16(%rax), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	-16(,%ebx), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	-16(,%ebx), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	-16(,%ebx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	-16(,%rbx), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	-16(,%rbx), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	-16(,%rbx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	-16(,%ebx), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	-16(,%ebx), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	-16(,%ebx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	-16(,%rbx), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	-16(,%rbx), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	-16(,%rbx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	-16(,%ebx,2), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	-16(,%ebx,2), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	-16(,%ebx,2), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	-16(,%rbx,2), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	-16(,%rbx,2), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	-16(,%rbx,2), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	-16(%eax,%ebx), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	-16(%eax,%ebx), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	-16(%eax,%ebx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	-16(%rax,%rbx), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	-16(%rax,%rbx), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	-16(%rax,%rbx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	-16(%eax,%ebx), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	-16(%eax,%ebx), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	-16(%eax,%ebx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	-16(%rax,%rbx), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	-16(%rax,%rbx), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	-16(%rax,%rbx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	-16(%eax,%ebx,2), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	-16(%eax,%ebx,2), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	-16(%eax,%ebx,2), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	-16(%rax,%rbx,2), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	-16(%rax,%rbx,2), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	-16(%rax,%rbx,2), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	1024, %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	1024, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	1024, %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	1024(%eax), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	1024(%eax), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	1024(%eax), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	1024(%rax), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	1024(%rax), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	1024(%rax), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	1024(,%ebx), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	1024(,%ebx), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	1024(,%ebx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	1024(,%rbx), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	1024(,%rbx), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	1024(,%rbx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	1024(,%ebx), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	1024(,%ebx), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	1024(,%ebx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	1024(,%rbx), %cx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	1024(,%rbx), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	1024(,%rbx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	1024(,%ebx,2), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	1024(,%ebx,2), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	1024(,%ebx,2), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	1024(,%rbx,2), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	1024(,%rbx,2), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	1024(,%rbx,2), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	1024(%eax,%ebx), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	1024(%eax,%ebx), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	1024(%eax,%ebx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	1024(%rax,%rbx), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	1024(%rax,%rbx), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	1024(%rax,%rbx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	1024(%eax,%ebx), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	1024(%eax,%ebx), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	1024(%eax,%ebx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	1024(%rax,%rbx), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	1024(%rax,%rbx), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	1024(%rax,%rbx), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	1024(%eax,%ebx,2), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	1024(%eax,%ebx,2), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	1024(%eax,%ebx,2), %rcx
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw	1024(%rax,%rbx,2), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal	1024(%rax,%rbx,2), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq	1024(%rax,%rbx,2), %rcx
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-lzcnt.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-lzcnt.s
new file mode 100644
index 0000000000000..1339ad5fbb30d
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-lzcnt.s
@@ -0,0 +1,65 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+lzcntw      %cx, %cx
+lzcntw      (%rax), %cx
+
+lzcntl      %eax, %ecx
+lzcntl      (%rax), %ecx
+
+lzcntq      %rax, %rcx
+lzcntq      (%rax), %rcx
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     1.00                        lzcntw	%cx, %cx
+# CHECK-NEXT:  1      5     0.33    *                   lzcntw	(%rax), %cx
+# CHECK-NEXT:  1      1     0.25                        lzcntl	%eax, %ecx
+# CHECK-NEXT:  1      5     0.33    *                   lzcntl	(%rax), %ecx
+# CHECK-NEXT:  1      1     0.25                        lzcntq	%rax, %rcx
+# CHECK-NEXT:  1      5     0.33    *                   lzcntq	(%rax), %rcx
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 1.00   1.00   1.00   2.25   2.25   2.25   2.25    -      -      -      -      -      -      -      -     1.00   1.00   1.00   1.00   1.00   1.00    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     lzcntw	%cx, %cx
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     lzcntw	(%rax), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     lzcntl	%eax, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     lzcntl	(%rax), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     lzcntq	%rax, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     lzcntq	(%rax), %rcx
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-mmx.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-mmx.s
new file mode 100644
index 0000000000000..23d352c95adf6
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-mmx.s
@@ -0,0 +1,408 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+emms
+
+movd        %eax, %mm2
+movd        (%rax), %mm2
+
+movd        %mm0, %ecx
+movd        %mm0, (%rax)
+
+movq        %rax, %mm2
+movq        (%rax), %mm2
+
+movq        %mm0, %rcx
+movq        %mm0, (%rax)
+
+packsswb    %mm0, %mm2
+packsswb    (%rax), %mm2
+
+packssdw    %mm0, %mm2
+packssdw    (%rax), %mm2
+
+packuswb    %mm0, %mm2
+packuswb    (%rax), %mm2
+
+paddb       %mm0, %mm2
+paddb       (%rax), %mm2
+
+paddd       %mm0, %mm2
+paddd       (%rax), %mm2
+
+paddsb      %mm0, %mm2
+paddsb      (%rax), %mm2
+
+paddsw      %mm0, %mm2
+paddsw      (%rax), %mm2
+
+paddusb     %mm0, %mm2
+paddusb     (%rax), %mm2
+
+paddusw     %mm0, %mm2
+paddusw     (%rax), %mm2
+
+paddw       %mm0, %mm2
+paddw       (%rax), %mm2
+
+pand        %mm0, %mm2
+pand        (%rax), %mm2
+
+pandn       %mm0, %mm2
+pandn       (%rax), %mm2
+
+pcmpeqb     %mm0, %mm2
+pcmpeqb     (%rax), %mm2
+
+pcmpeqd     %mm0, %mm2
+pcmpeqd     (%rax), %mm2
+
+pcmpeqw     %mm0, %mm2
+pcmpeqw     (%rax), %mm2
+
+pcmpgtb     %mm0, %mm2
+pcmpgtb     (%rax), %mm2
+
+pcmpgtd     %mm0, %mm2
+pcmpgtd     (%rax), %mm2
+
+pcmpgtw     %mm0, %mm2
+pcmpgtw     (%rax), %mm2
+
+pmaddwd     %mm0, %mm2
+pmaddwd     (%rax), %mm2
+
+pmulhw      %mm0, %mm2
+pmulhw      (%rax), %mm2
+
+pmullw      %mm0, %mm2
+pmullw      (%rax), %mm2
+
+por         %mm0, %mm2
+por         (%rax), %mm2
+
+pslld       $1, %mm2
+pslld       %mm0, %mm2
+pslld       (%rax), %mm2
+
+psllq       $1, %mm2
+psllq       %mm0, %mm2
+psllq       (%rax), %mm2
+
+psllw       $1, %mm2
+psllw       %mm0, %mm2
+psllw       (%rax), %mm2
+
+psrad       $1, %mm2
+psrad       %mm0, %mm2
+psrad       (%rax), %mm2
+
+psraw       $1, %mm2
+psraw       %mm0, %mm2
+psraw       (%rax), %mm2
+
+psrld       $1, %mm2
+psrld       %mm0, %mm2
+psrld       (%rax), %mm2
+
+psrlq       $1, %mm2
+psrlq       %mm0, %mm2
+psrlq       (%rax), %mm2
+
+psrlw       $1, %mm2
+psrlw       %mm0, %mm2
+psrlw       (%rax), %mm2
+
+psubb       %mm0, %mm2
+psubb       (%rax), %mm2
+
+psubd       %mm0, %mm2
+psubd       (%rax), %mm2
+
+psubsb      %mm0, %mm2
+psubsb      (%rax), %mm2
+
+psubsw      %mm0, %mm2
+psubsw      (%rax), %mm2
+
+psubusb     %mm0, %mm2
+psubusb     (%rax), %mm2
+
+psubusw     %mm0, %mm2
+psubusw     (%rax), %mm2
+
+psubw       %mm0, %mm2
+psubw       (%rax), %mm2
+
+punpckhbw   %mm0, %mm2
+punpckhbw   (%rax), %mm2
+
+punpckhdq   %mm0, %mm2
+punpckhdq   (%rax), %mm2
+
+punpckhwd   %mm0, %mm2
+punpckhwd   (%rax), %mm2
+
+punpcklbw   %mm0, %mm2
+punpcklbw   (%rax), %mm2
+
+punpckldq   %mm0, %mm2
+punpckldq   (%rax), %mm2
+
+punpcklwd   %mm0, %mm2
+punpcklwd   (%rax), %mm2
+
+pxor        %mm0, %mm2
+pxor        (%rax), %mm2
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      2     0.25    *      *      U     emms
+# CHECK-NEXT:  2      1     1.00                        movd	%eax, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   movd	(%rax), %mm2
+# CHECK-NEXT:  1      1     1.00                        movd	%mm0, %ecx
+# CHECK-NEXT:  1      1     1.00           *      U     movd	%mm0, (%rax)
+# CHECK-NEXT:  2      1     1.00                        movq	%rax, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   movq	(%rax), %mm2
+# CHECK-NEXT:  1      1     1.00                        movq	%mm0, %rcx
+# CHECK-NEXT:  1      1     1.00           *            movq	%mm0, (%rax)
+# CHECK-NEXT:  1      1     0.50                        packsswb	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   packsswb	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.50                        packssdw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   packssdw	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.50                        packuswb	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   packuswb	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        paddb	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   paddb	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        paddd	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   paddd	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        paddsb	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   paddsb	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        paddsw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   paddsw	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        paddusb	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   paddusb	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        paddusw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   paddusw	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        paddw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   paddw	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        pand	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pand	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        pandn	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pandn	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        pcmpeqb	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pcmpeqb	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        pcmpeqd	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pcmpeqd	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        pcmpeqw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pcmpeqw	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        pcmpgtb	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pcmpgtb	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        pcmpgtd	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pcmpgtd	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        pcmpgtw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pcmpgtw	(%rax), %mm2
+# CHECK-NEXT:  1      3     0.50                        pmaddwd	%mm0, %mm2
+# CHECK-NEXT:  1      10    0.50    *                   pmaddwd	(%rax), %mm2
+# CHECK-NEXT:  1      3     0.50                        pmulhw	%mm0, %mm2
+# CHECK-NEXT:  1      10    0.50    *                   pmulhw	(%rax), %mm2
+# CHECK-NEXT:  1      3     0.50                        pmullw	%mm0, %mm2
+# CHECK-NEXT:  1      10    0.50    *                   pmullw	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        por	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   por	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.50                        pslld	$1, %mm2
+# CHECK-NEXT:  1      1     0.50                        pslld	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pslld	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.50                        psllq	$1, %mm2
+# CHECK-NEXT:  1      1     0.50                        psllq	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   psllq	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.50                        psllw	$1, %mm2
+# CHECK-NEXT:  1      1     0.50                        psllw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   psllw	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.50                        psrad	$1, %mm2
+# CHECK-NEXT:  1      1     0.50                        psrad	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   psrad	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.50                        psraw	$1, %mm2
+# CHECK-NEXT:  1      1     0.50                        psraw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   psraw	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.50                        psrld	$1, %mm2
+# CHECK-NEXT:  1      1     0.50                        psrld	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   psrld	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.50                        psrlq	$1, %mm2
+# CHECK-NEXT:  1      1     0.50                        psrlq	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   psrlq	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.50                        psrlw	$1, %mm2
+# CHECK-NEXT:  1      1     0.50                        psrlw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   psrlw	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        psubb	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   psubb	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        psubd	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   psubd	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        psubsb	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   psubsb	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        psubsw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   psubsw	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        psubusb	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   psubusb	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        psubusw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   psubusw	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        psubw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   psubw	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.50                        punpckhbw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   punpckhbw	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.50                        punpckhdq	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   punpckhdq	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.50                        punpckhwd	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   punpckhwd	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.50                        punpcklbw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   punpcklbw	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.50                        punpckldq	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   punpckldq	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.50                        punpcklwd	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   punpcklwd	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        pxor	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pxor	(%rax), %mm2
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -     17.00  35.00  35.00  17.00  27.00  27.00  2.00   16.00  16.00  16.00  15.33  15.33  15.33  1.00   1.00
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     emms
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00   1.00   1.00   0.50   0.50    -      -      -      -      -      -      -      -      -     movd	%eax, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movd	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     movd	%mm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movd	%mm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00   1.00   1.00   0.50   0.50    -      -      -      -      -      -      -      -      -     movq	%rax, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movq	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     movq	%mm0, %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movq	%mm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     packsswb	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     packsswb	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     packssdw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     packssdw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     packuswb	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     packuswb	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     paddb	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     paddb	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     paddd	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     paddd	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     paddsb	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     paddsb	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     paddsw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     paddsw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     paddusb	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     paddusb	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     paddusw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     paddusw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     paddw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     paddw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pand	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pand	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pandn	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pandn	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pcmpeqb	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pcmpeqb	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pcmpeqd	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pcmpeqd	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pcmpeqw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pcmpeqw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pcmpgtb	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pcmpgtb	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pcmpgtd	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pcmpgtd	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pcmpgtw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pcmpgtw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     pmaddwd	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmaddwd	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     pmulhw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmulhw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     pmullw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmullw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     por	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     por	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     pslld	$1, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     pslld	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pslld	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psllq	$1, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psllq	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psllq	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psllw	$1, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psllw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psllw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psrad	$1, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psrad	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psrad	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psraw	$1, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psraw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psraw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psrld	$1, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psrld	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psrld	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psrlq	$1, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psrlq	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psrlq	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psrlw	$1, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psrlw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psrlw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     psubb	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psubb	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     psubd	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psubd	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     psubsb	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psubsb	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     psubsw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psubsw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     psubusb	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psubusb	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     psubusw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psubusw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     psubw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psubw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     punpckhbw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     punpckhbw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     punpckhdq	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     punpckhdq	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     punpckhwd	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     punpckhwd	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     punpcklbw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     punpcklbw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     punpckldq	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     punpckldq	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     punpcklwd	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     punpcklwd	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pxor	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pxor	(%rax), %mm2
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-movbe.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-movbe.s
new file mode 100644
index 0000000000000..5e180f3b4342d
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-movbe.s
@@ -0,0 +1,65 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+movbe  %cx, (%rax)
+movbe  (%rax), %cx
+
+movbe  %ecx, (%rax)
+movbe  (%rax), %ecx
+
+movbe  %rcx, (%rax)
+movbe  (%rax), %rcx
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  2      1     1.00           *            movbew	%cx, (%rax)
+# CHECK-NEXT:  1      4     1.00    *                   movbew	(%rax), %cx
+# CHECK-NEXT:  2      1     1.00           *            movbel	%ecx, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   movbel	(%rax), %ecx
+# CHECK-NEXT:  2      1     1.00           *            movbeq	%rcx, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   movbeq	(%rax), %rcx
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 2.00   2.00   2.00   4.50   4.50   4.50   4.50    -      -      -      -      -      -      -      -     2.00   2.00   2.00   1.00   1.00   1.00   1.50   1.50
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT: 0.33   0.33   0.33   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   movbew	%cx, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movbew	(%rax), %cx
+# CHECK-NEXT: 0.33   0.33   0.33   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   movbel	%ecx, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movbel	(%rax), %ecx
+# CHECK-NEXT: 0.33   0.33   0.33   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   movbeq	%rcx, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movbeq	(%rax), %rcx
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-pclmul.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-pclmul.s
new file mode 100644
index 0000000000000..a36fb2aabe486
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-pclmul.s
@@ -0,0 +1,51 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+pclmulqdq     $11, %xmm0, %xmm2
+pclmulqdq     $11, (%rax), %xmm2
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  4      4     2.00                        pclmulqdq	$11, %xmm0, %xmm2
+# CHECK-NEXT:  4      11    2.00    *                   pclmulqdq	$11, (%rax), %xmm2
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     4.00   4.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     pclmulqdq	$11, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pclmulqdq	$11, (%rax), %xmm2
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-popcnt.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-popcnt.s
new file mode 100644
index 0000000000000..aa77de9bb18b4
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-popcnt.s
@@ -0,0 +1,65 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+popcntw     %cx, %cx
+popcntw     (%rax), %cx
+
+popcntl     %eax, %ecx
+popcntl     (%rax), %ecx
+
+popcntq     %rax, %rcx
+popcntq     (%rax), %rcx
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     1.00                        popcntw	%cx, %cx
+# CHECK-NEXT:  1      5     0.33    *                   popcntw	(%rax), %cx
+# CHECK-NEXT:  1      1     0.25                        popcntl	%eax, %ecx
+# CHECK-NEXT:  1      5     0.33    *                   popcntl	(%rax), %ecx
+# CHECK-NEXT:  1      1     0.25                        popcntq	%rax, %rcx
+# CHECK-NEXT:  1      5     0.33    *                   popcntq	(%rax), %rcx
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 1.00   1.00   1.00   2.25   2.25   2.25   2.25    -      -      -      -      -      -      -      -     1.00   1.00   1.00   1.00   1.00   1.00    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     popcntw	%cx, %cx
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     popcntw	(%rax), %cx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     popcntl	%eax, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     popcntl	(%rax), %ecx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     popcntq	%rax, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     popcntq	(%rax), %rcx
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-prefetchw.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-prefetchw.s
new file mode 100644
index 0000000000000..20d211cfeccad
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-prefetchw.s
@@ -0,0 +1,51 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+prefetch    (%rax)
+prefetchw   (%rax)
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      5     0.33    *      *            prefetch	(%rax)
+# CHECK-NEXT:  1      5     0.33    *      *            prefetchw	(%rax)
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 0.67   0.67   0.67    -      -      -      -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.67   0.67   0.67    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     prefetch	(%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     prefetchw	(%rax)
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-rdrand.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-rdrand.s
new file mode 100644
index 0000000000000..755238693ea7c
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-rdrand.s
@@ -0,0 +1,54 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+rdrand   %ax
+rdrand   %eax
+rdrand   %rax
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  100    100   25.00                 U     rdrandw	%ax
+# CHECK-NEXT:  100    100   25.00                 U     rdrandl	%eax
+# CHECK-NEXT:  100    100   25.00                 U     rdrandq	%rax
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -     75.00  75.00  75.00  75.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rdrandw	%ax
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rdrandl	%eax
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rdrandq	%rax
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-rdseed.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-rdseed.s
new file mode 100644
index 0000000000000..e2b028c4c47ed
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-rdseed.s
@@ -0,0 +1,54 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+rdseed   %ax
+rdseed   %eax
+rdseed   %rax
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  100    100   25.00                 U     rdseedw	%ax
+# CHECK-NEXT:  100    100   25.00                 U     rdseedl	%eax
+# CHECK-NEXT:  100    100   25.00                 U     rdseedq	%rax
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -     75.00  75.00  75.00  75.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rdseedw	%ax
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rdseedl	%eax
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rdseedq	%rax
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-sse1.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-sse1.s
new file mode 100644
index 0000000000000..6879fb28ffdf9
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-sse1.s
@@ -0,0 +1,476 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+addps       %xmm0, %xmm2
+addps       (%rax), %xmm2
+
+addss       %xmm0, %xmm2
+addss       (%rax), %xmm2
+
+andnps      %xmm0, %xmm2
+andnps      (%rax), %xmm2
+
+andps       %xmm0, %xmm2
+andps       (%rax), %xmm2
+
+cmpps       $0, %xmm0, %xmm2
+cmpps       $0, (%rax), %xmm2
+
+cmpss       $0, %xmm0, %xmm2
+cmpss       $0, (%rax), %xmm2
+
+comiss      %xmm0, %xmm1
+comiss      (%rax), %xmm1
+
+cvtpi2ps    %mm0, %xmm2
+cvtpi2ps    (%rax), %xmm2
+
+cvtps2pi    %xmm0, %mm2
+cvtps2pi    (%rax), %mm2
+
+cvtsi2ss    %ecx, %xmm2
+cvtsi2ss    %rcx, %xmm2
+cvtsi2ssl   (%rax), %xmm2
+cvtsi2ssq   (%rax), %xmm2
+
+cvtss2si    %xmm0, %ecx
+cvtss2si    %xmm0, %rcx
+cvtss2si    (%rax), %ecx
+cvtss2si    (%rax), %rcx
+
+cvttps2pi   %xmm0, %mm2
+cvttps2pi   (%rax), %mm2
+
+cvttss2si   %xmm0, %ecx
+cvttss2si   %xmm0, %rcx
+cvttss2si   (%rax), %ecx
+cvttss2si   (%rax), %rcx
+
+divps       %xmm0, %xmm2
+divps       (%rax), %xmm2
+
+divss       %xmm0, %xmm2
+divss       (%rax), %xmm2
+
+ldmxcsr     (%rax)
+
+maskmovq    %mm0, %mm1
+
+maxps       %xmm0, %xmm2
+maxps       (%rax), %xmm2
+
+maxss       %xmm0, %xmm2
+maxss       (%rax), %xmm2
+
+minps       %xmm0, %xmm2
+minps       (%rax), %xmm2
+
+minss       %xmm0, %xmm2
+minss       (%rax), %xmm2
+
+movaps      %xmm0, %xmm2
+movaps      %xmm0, (%rax)
+movaps      (%rax), %xmm2
+
+movhlps     %xmm0, %xmm2
+movlhps     %xmm0, %xmm2
+
+movhps      %xmm0, (%rax)
+movhps      (%rax), %xmm2
+
+movlps      %xmm0, (%rax)
+movlps      (%rax), %xmm2
+
+movmskps    %xmm0, %rcx
+
+movntps     %xmm0, (%rax)
+movntq      %mm0, (%rax)
+
+movss       %xmm0, %xmm2
+movss       %xmm0, (%rax)
+movss       (%rax), %xmm2
+
+movups      %xmm0, %xmm2
+movups      %xmm0, (%rax)
+movups      (%rax), %xmm2
+
+mulps       %xmm0, %xmm2
+mulps       (%rax), %xmm2
+
+mulss       %xmm0, %xmm2
+mulss       (%rax), %xmm2
+
+orps        %xmm0, %xmm2
+orps        (%rax), %xmm2
+
+pavgb       %mm0, %mm2
+pavgb       (%rax), %mm2
+
+pavgw       %mm0, %mm2
+pavgw       (%rax), %mm2
+
+pextrw      $1, %mm0, %rcx
+
+pinsrw      $1, %rax, %mm2
+pinsrw      $1, (%rax), %mm2
+
+pmaxsw      %mm0, %mm2
+pmaxsw      (%rax), %mm2
+
+pmaxub      %mm0, %mm2
+pmaxub      (%rax), %mm2
+
+pminsw      %mm0, %mm2
+pminsw      (%rax), %mm2
+
+pminub      %mm0, %mm2
+pminub      (%rax), %mm2
+
+pmovmskb    %mm0, %rcx
+
+pmulhuw     %mm0, %mm2
+pmulhuw     (%rax), %mm2
+
+prefetcht0  (%rax)
+prefetcht1  (%rax)
+prefetcht2  (%rax)
+prefetchnta (%rax)
+
+psadbw      %mm0, %mm2
+psadbw      (%rax), %mm2
+
+pshufw      $1, %mm0, %mm2
+pshufw      $1, (%rax), %mm2
+
+rcpps       %xmm0, %xmm2
+rcpps       (%rax), %xmm2
+
+rcpss       %xmm0, %xmm2
+rcpss       (%rax), %xmm2
+
+rsqrtps     %xmm0, %xmm2
+rsqrtps     (%rax), %xmm2
+
+rsqrtss     %xmm0, %xmm2
+rsqrtss     (%rax), %xmm2
+
+sfence
+
+shufps      $1, %xmm0, %xmm2
+shufps      $1, (%rax), %xmm2
+
+sqrtps      %xmm0, %xmm2
+sqrtps      (%rax), %xmm2
+
+sqrtss      %xmm0, %xmm2
+sqrtss      (%rax), %xmm2
+
+stmxcsr     (%rax)
+
+subps       %xmm0, %xmm2
+subps       (%rax), %xmm2
+
+subss       %xmm0, %xmm2
+subss       (%rax), %xmm2
+
+ucomiss     %xmm0, %xmm1
+ucomiss     (%rax), %xmm1
+
+unpckhps    %xmm0, %xmm2
+unpckhps    (%rax), %xmm2
+
+unpcklps    %xmm0, %xmm2
+unpcklps    (%rax), %xmm2
+
+xorps       %xmm0, %xmm2
+xorps       (%rax), %xmm2
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      3     0.50                        addps	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   addps	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        addss	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   addss	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        andnps	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   andnps	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        andps	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   andps	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        cmpeqps	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   cmpeqps	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        cmpeqss	%xmm0, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   cmpeqss	(%rax), %xmm2
+# CHECK-NEXT:  2      4     1.00                        comiss	%xmm0, %xmm1
+# CHECK-NEXT:  2      11    1.00    *                   comiss	(%rax), %xmm1
+# CHECK-NEXT:  1      3     0.50                        cvtpi2ps	%mm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   cvtpi2ps	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        cvtps2pi	%xmm0, %mm2
+# CHECK-NEXT:  1      10    0.50    *                   cvtps2pi	(%rax), %mm2
+# CHECK-NEXT:  2      4     1.00                        cvtsi2ss	%ecx, %xmm2
+# CHECK-NEXT:  2      4     1.00                        cvtsi2ss	%rcx, %xmm2
+# CHECK-NEXT:  1      10    1.00    *                   cvtsi2ssl	(%rax), %xmm2
+# CHECK-NEXT:  1      10    1.00    *                   cvtsi2ssq	(%rax), %xmm2
+# CHECK-NEXT:  2      5     2.50                        cvtss2si	%xmm0, %ecx
+# CHECK-NEXT:  2      5     2.50                        cvtss2si	%xmm0, %rcx
+# CHECK-NEXT:  2      12    2.50    *                   cvtss2si	(%rax), %ecx
+# CHECK-NEXT:  2      12    2.50    *                   cvtss2si	(%rax), %rcx
+# CHECK-NEXT:  1      3     0.50                        cvttps2pi	%xmm0, %mm2
+# CHECK-NEXT:  1      10    0.50    *                   cvttps2pi	(%rax), %mm2
+# CHECK-NEXT:  2      5     2.50                        cvttss2si	%xmm0, %ecx
+# CHECK-NEXT:  2      5     2.50                        cvttss2si	%xmm0, %rcx
+# CHECK-NEXT:  2      12    2.50    *                   cvttss2si	(%rax), %ecx
+# CHECK-NEXT:  2      12    2.50    *                   cvttss2si	(%rax), %rcx
+# CHECK-NEXT:  1      11    3.00                        divps	%xmm0, %xmm2
+# CHECK-NEXT:  1      18    3.00    *                   divps	(%rax), %xmm2
+# CHECK-NEXT:  1      11    3.00                        divss	%xmm0, %xmm2
+# CHECK-NEXT:  1      18    3.00    *                   divss	(%rax), %xmm2
+# CHECK-NEXT:  1      5     1.50    *      *      U     ldmxcsr	(%rax)
+# CHECK-NEXT:  1      1     0.50    *      *      U     maskmovq	%mm0, %mm1
+# CHECK-NEXT:  1      1     0.50                        maxps	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   maxps	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        maxss	%xmm0, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   maxss	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        minps	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   minps	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        minss	%xmm0, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   minss	(%rax), %xmm2
+# CHECK-NEXT:  1      0     0.17                        movaps	%xmm0, %xmm2
+# CHECK-NEXT:  1      1     1.00           *            movaps	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   movaps	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        movhlps	%xmm0, %xmm2
+# CHECK-NEXT:  1      1     0.50                        movlhps	%xmm0, %xmm2
+# CHECK-NEXT:  2      2     1.00           *            movhps	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   movhps	(%rax), %xmm2
+# CHECK-NEXT:  1      1     1.00           *            movlps	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   movlps	(%rax), %xmm2
+# CHECK-NEXT:  1      1     1.00                        movmskps	%xmm0, %ecx
+# CHECK-NEXT:  1      1     1.00           *            movntps	%xmm0, (%rax)
+# CHECK-NEXT:  1      1     1.00    *      *      U     movntq	%mm0, (%rax)
+# CHECK-NEXT:  1      1     0.50                        movss	%xmm0, %xmm2
+# CHECK-NEXT:  1      1     1.00           *            movss	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   movss	(%rax), %xmm2
+# CHECK-NEXT:  1      0     0.17                        movups	%xmm0, %xmm2
+# CHECK-NEXT:  1      1     1.00           *            movups	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   movups	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        mulps	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   mulps	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        mulss	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   mulss	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        orps	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   orps	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pavgb	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pavgb	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        pavgw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pavgw	(%rax), %mm2
+# CHECK-NEXT:  2      1     1.00                        pextrw	$1, %mm0, %ecx
+# CHECK-NEXT:  2      2     1.00                        pinsrw	$1, %eax, %mm2
+# CHECK-NEXT:  1      8     1.50    *                   pinsrw	$1, (%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        pmaxsw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pmaxsw	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        pmaxub	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pmaxub	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        pminsw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pminsw	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        pminub	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pminub	(%rax), %mm2
+# CHECK-NEXT:  1      1     1.00                        pmovmskb	%mm0, %ecx
+# CHECK-NEXT:  1      3     0.50                        pmulhuw	%mm0, %mm2
+# CHECK-NEXT:  1      10    0.50    *                   pmulhuw	(%rax), %mm2
+# CHECK-NEXT:  1      5     0.33    *      *            prefetcht0	(%rax)
+# CHECK-NEXT:  1      5     0.33    *      *            prefetcht1	(%rax)
+# CHECK-NEXT:  1      5     0.33    *      *            prefetcht2	(%rax)
+# CHECK-NEXT:  1      5     0.33    *      *            prefetchnta	(%rax)
+# CHECK-NEXT:  1      3     0.50                        psadbw	%mm0, %mm2
+# CHECK-NEXT:  1      10    0.50    *                   psadbw	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.50                        pshufw	$1, %mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pshufw	$1, (%rax), %mm2
+# CHECK-NEXT:  1      4     0.50                        rcpps	%xmm0, %xmm2
+# CHECK-NEXT:  1      11    0.50    *                   rcpps	(%rax), %xmm2
+# CHECK-NEXT:  1      4     0.50                        rcpss	%xmm0, %xmm2
+# CHECK-NEXT:  1      11    0.50    *                   rcpss	(%rax), %xmm2
+# CHECK-NEXT:  1      4     1.00                        rsqrtps	%xmm0, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   rsqrtps	(%rax), %xmm2
+# CHECK-NEXT:  1      4     1.00                        rsqrtss	%xmm0, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   rsqrtss	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.33    *      *      U     sfence
+# CHECK-NEXT:  1      1     0.50                        shufps	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   shufps	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      15    5.00                        sqrtps	%xmm0, %xmm2
+# CHECK-NEXT:  1      22    5.00    *                   sqrtps	(%rax), %xmm2
+# CHECK-NEXT:  1      15    5.00                        sqrtss	%xmm0, %xmm2
+# CHECK-NEXT:  1      22    5.00    *                   sqrtss	(%rax), %xmm2
+# CHECK-NEXT:  2      2     15.00          *      U     stmxcsr	(%rax)
+# CHECK-NEXT:  1      3     0.50                        subps	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   subps	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        subss	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   subss	(%rax), %xmm2
+# CHECK-NEXT:  2      4     1.00                        ucomiss	%xmm0, %xmm1
+# CHECK-NEXT:  2      11    1.00    *                   ucomiss	(%rax), %xmm1
+# CHECK-NEXT:  1      1     0.50                        unpckhps	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   unpckhps	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        unpcklps	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   unpcklps	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        xorps	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   xorps	(%rax), %xmm2
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 2.00   2.00   2.00   16.50  16.50  16.50  16.50   -     24.50  66.50  46.00  38.00  33.00  33.00  7.00   22.33  22.33  22.33  19.33  19.33  19.33  4.00   4.00
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     addps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     addps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     addss	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     addss	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     andnps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     andnps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     andps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     andps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     cmpeqps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpeqps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     cmpeqss	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpeqss	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     comiss	%xmm0, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     comiss	(%rax), %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     cvtpi2ps	%mm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtpi2ps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     cvtps2pi	%xmm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtps2pi	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     cvtsi2ss	%ecx, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     cvtsi2ss	%rcx, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtsi2ssl	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtsi2ssq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -     cvtss2si	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -     cvtss2si	%xmm0, %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtss2si	(%rax), %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtss2si	(%rax), %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     cvttps2pi	%xmm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvttps2pi	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -     cvttss2si	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50    -      -      -      -      -      -      -      -      -      -      -     cvttss2si	%xmm0, %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvttss2si	(%rax), %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.50   2.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvttss2si	(%rax), %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     divps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     divps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     divss	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     divss	(%rax), %xmm2
+# CHECK-NEXT: 0.33   0.33   0.33   1.50   1.50   1.50   1.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     ldmxcsr	(%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     maskmovq	%mm0, %mm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     maxps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     maxps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     maxss	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     maxss	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     minps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     minps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     minss	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     minss	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movaps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movaps	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movaps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     movhlps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     movlhps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movhps	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movhps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movlps	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movlps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     movmskps	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movntps	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movntq	%mm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     movss	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movss	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movss	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movups	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movups	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movups	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     mulps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     mulps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     mulss	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     mulss	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     orps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     orps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pavgb	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pavgb	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pavgw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pavgw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     pextrw	$1, %mm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     pinsrw	$1, %eax, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pinsrw	$1, (%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pmaxsw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmaxsw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pmaxub	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmaxub	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pminsw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pminsw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pminub	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pminub	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     pmovmskb	%mm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     pmulhuw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmulhuw	(%rax), %mm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     prefetcht0	(%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     prefetcht1	(%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     prefetcht2	(%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     prefetchnta	(%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     psadbw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psadbw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     pshufw	$1, %mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pshufw	$1, (%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     rcpps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rcpps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     rcpss	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rcpss	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     rsqrtps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rsqrtps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     rsqrtss	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rsqrtss	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -      -      -     sfence
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     shufps	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     shufps	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     sqrtps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     sqrtps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     sqrtss	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     sqrtss	(%rax), %xmm2
+# CHECK-NEXT: 0.33   0.33   0.33   15.00  15.00  15.00  15.00   -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   stmxcsr	(%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     subps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     subps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     subss	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     subss	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     ucomiss	%xmm0, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     ucomiss	(%rax), %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     unpckhps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     unpckhps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     unpcklps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     unpcklps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     xorps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     xorps	(%rax), %xmm2
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-sse2.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-sse2.s
new file mode 100644
index 0000000000000..02013b85f417a
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-sse2.s
@@ -0,0 +1,975 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+addpd       %xmm0, %xmm2
+addpd       (%rax), %xmm2
+
+addsd       %xmm0, %xmm2
+addsd       (%rax), %xmm2
+
+andnpd      %xmm0, %xmm2
+andnpd      (%rax), %xmm2
+
+andpd       %xmm0, %xmm2
+andpd       (%rax), %xmm2
+
+clflush     (%rax)
+
+cmppd       $0, %xmm0, %xmm2
+cmppd       $0, (%rax), %xmm2
+
+cmpsd       $0, %xmm0, %xmm2
+cmpsd       $0, (%rax), %xmm2
+
+comisd      %xmm0, %xmm1
+comisd      (%rax), %xmm1
+
+cvtdq2pd    %xmm0, %xmm2
+cvtdq2pd    (%rax), %xmm2
+
+cvtdq2ps    %xmm0, %xmm2
+cvtdq2ps    (%rax), %xmm2
+
+cvtpd2dq    %xmm0, %xmm2
+cvtpd2dq    (%rax), %xmm2
+
+cvtpd2pi    %xmm0, %mm2
+cvtpd2pi    (%rax), %mm2
+
+cvtpd2ps    %xmm0, %xmm2
+cvtpd2ps    (%rax), %xmm2
+
+cvtpi2pd    %mm0, %xmm2
+cvtpi2pd    (%rax), %xmm2
+
+cvtps2dq    %xmm0, %xmm2
+cvtps2dq    (%rax), %xmm2
+
+cvtps2pd    %xmm0, %xmm2
+cvtps2pd    (%rax), %xmm2
+
+cvtsd2si    %xmm0, %ecx
+cvtsd2si    %xmm0, %rcx
+cvtsd2si    (%rax), %ecx
+cvtsd2si    (%rax), %rcx
+
+cvtsd2ss    %xmm0, %xmm2
+cvtsd2ss    (%rax), %xmm2
+
+cvtsi2sd    %ecx, %xmm2
+cvtsi2sd    %rcx, %xmm2
+cvtsi2sdl   (%rax), %xmm2
+cvtsi2sdq   (%rax), %xmm2
+
+cvtss2sd    %xmm0, %xmm2
+cvtss2sd    (%rax), %xmm2
+
+cvttpd2dq   %xmm0, %xmm2
+cvttpd2dq   (%rax), %xmm2
+
+cvttpd2pi   %xmm0, %mm2
+cvttpd2pi   (%rax), %mm2
+
+cvttps2dq   %xmm0, %xmm2
+cvttps2dq   (%rax), %xmm2
+
+cvttsd2si   %xmm0, %ecx
+cvttsd2si   %xmm0, %rcx
+cvttsd2si   (%rax), %ecx
+cvttsd2si   (%rax), %rcx
+
+divpd       %xmm0, %xmm2
+divpd       (%rax), %xmm2
+
+divsd       %xmm0, %xmm2
+divsd       (%rax), %xmm2
+
+lfence
+
+maskmovdqu  %xmm0, %xmm1
+
+maxpd       %xmm0, %xmm2
+maxpd       (%rax), %xmm2
+
+maxsd       %xmm0, %xmm2
+maxsd       (%rax), %xmm2
+
+mfence
+
+minpd       %xmm0, %xmm2
+minpd       (%rax), %xmm2
+
+minsd       %xmm0, %xmm2
+minsd       (%rax), %xmm2
+
+movapd      %xmm0, %xmm2
+movapd      %xmm0, (%rax)
+movapd      (%rax), %xmm2
+
+movd        %eax, %xmm2
+movd        (%rax), %xmm2
+
+movd        %xmm0, %ecx
+movd        %xmm0, (%rax)
+
+movdqa      %xmm0, %xmm2
+movdqa      %xmm0, (%rax)
+movdqa      (%rax), %xmm2
+
+movdqu      %xmm0, %xmm2
+movdqu      %xmm0, (%rax)
+movdqu      (%rax), %xmm2
+
+movdq2q     %xmm0, %mm2
+
+movhpd      %xmm0, (%rax)
+movhpd      (%rax), %xmm2
+
+movlpd      %xmm0, (%rax)
+movlpd      (%rax), %xmm2
+
+movmskpd    %xmm0, %rcx
+
+movntil     %eax, (%rax)
+movntiq     %rax, (%rax)
+
+movntdq     %xmm0, (%rax)
+movntpd     %xmm0, (%rax)
+
+movq        %xmm0, %xmm2
+
+movq        %rax, %xmm2
+movq        (%rax), %xmm2
+
+movq        %xmm0, %rcx
+movq        %xmm0, (%rax)
+
+movq2dq     %mm0, %xmm2
+
+movsd       %xmm0, %xmm2
+movsd       %xmm0, (%rax)
+movsd       (%rax), %xmm2
+
+movupd      %xmm0, %xmm2
+movupd      %xmm0, (%rax)
+movupd      (%rax), %xmm2
+
+mulpd       %xmm0, %xmm2
+mulpd       (%rax), %xmm2
+
+mulsd       %xmm0, %xmm2
+mulsd       (%rax), %xmm2
+
+orpd        %xmm0, %xmm2
+orpd        (%rax), %xmm2
+
+packssdw    %xmm0, %xmm2
+packssdw    (%rax), %xmm2
+
+packsswb    %xmm0, %xmm2
+packsswb    (%rax), %xmm2
+
+packuswb    %xmm0, %xmm2
+packuswb    (%rax), %xmm2
+
+paddb       %xmm0, %xmm2
+paddb       (%rax), %xmm2
+
+paddd       %xmm0, %xmm2
+paddd       (%rax), %xmm2
+
+paddq       %mm0, %mm2
+paddq       (%rax), %mm2
+
+paddq       %xmm0, %xmm2
+paddq       (%rax), %xmm2
+
+paddsb      %xmm0, %xmm2
+paddsb      (%rax), %xmm2
+
+paddsw      %xmm0, %xmm2
+paddsw      (%rax), %xmm2
+
+paddusb     %xmm0, %xmm2
+paddusb     (%rax), %xmm2
+
+paddusw     %xmm0, %xmm2
+paddusw     (%rax), %xmm2
+
+paddw       %xmm0, %xmm2
+paddw       (%rax), %xmm2
+
+pand        %xmm0, %xmm2
+pand        (%rax), %xmm2
+
+pandn       %xmm0, %xmm2
+pandn       (%rax), %xmm2
+
+pavgb       %xmm0, %xmm2
+pavgb       (%rax), %xmm2
+
+pavgw       %xmm0, %xmm2
+pavgw       (%rax), %xmm2
+
+pcmpeqb     %xmm0, %xmm2
+pcmpeqb     (%rax), %xmm2
+
+pcmpeqd     %xmm0, %xmm2
+pcmpeqd     (%rax), %xmm2
+
+pcmpeqw     %xmm0, %xmm2
+pcmpeqw     (%rax), %xmm2
+
+pcmpgtb     %xmm0, %xmm2
+pcmpgtb     (%rax), %xmm2
+
+pcmpgtd     %xmm0, %xmm2
+pcmpgtd     (%rax), %xmm2
+
+pcmpgtw     %xmm0, %xmm2
+pcmpgtw     (%rax), %xmm2
+
+pextrw      $1, %xmm0, %rcx
+
+pinsrw      $1, %rax, %xmm0
+pinsrw      $1, (%rax), %xmm0
+
+pmaddwd     %xmm0, %xmm2
+pmaddwd     (%rax), %xmm2
+
+pmaxsw      %xmm0, %xmm2
+pmaxsw      (%rax), %xmm2
+
+pmaxub      %xmm0, %xmm2
+pmaxub      (%rax), %xmm2
+
+pminsw      %xmm0, %xmm2
+pminsw      (%rax), %xmm2
+
+pminub      %xmm0, %xmm2
+pminub      (%rax), %xmm2
+
+pmovmskb    %xmm0, %rcx
+
+pmulhuw     %xmm0, %xmm2
+pmulhuw     (%rax), %xmm2
+
+pmulhw      %xmm0, %xmm2
+pmulhw      (%rax), %xmm2
+
+pmullw      %xmm0, %xmm2
+pmullw      (%rax), %xmm2
+
+pmuludq     %mm0, %mm2
+pmuludq     (%rax), %mm2
+
+pmuludq     %xmm0, %xmm2
+pmuludq     (%rax), %xmm2
+
+por         %xmm0, %xmm2
+por         (%rax), %xmm2
+
+psadbw      %xmm0, %xmm2
+psadbw      (%rax), %xmm2
+
+pshufd      $1, %xmm0, %xmm2
+pshufd      $1, (%rax), %xmm2
+
+pshufhw     $1, %xmm0, %xmm2
+pshufhw     $1, (%rax), %xmm2
+
+pshuflw     $1, %xmm0, %xmm2
+pshuflw     $1, (%rax), %xmm2
+
+pslld       $1, %xmm2
+pslld       %xmm0, %xmm2
+pslld       (%rax), %xmm2
+
+pslldq      $1, %xmm2
+
+psllq       $1, %xmm2
+psllq       %xmm0, %xmm2
+psllq       (%rax), %xmm2
+
+psllw       $1, %xmm2
+psllw       %xmm0, %xmm2
+psllw       (%rax), %xmm2
+
+psrad       $1, %xmm2
+psrad       %xmm0, %xmm2
+psrad       (%rax), %xmm2
+
+psraw       $1, %xmm2
+psraw       %xmm0, %xmm2
+psraw       (%rax), %xmm2
+
+psrld       $1, %xmm2
+psrld       %xmm0, %xmm2
+psrld       (%rax), %xmm2
+
+psrldq      $1, %xmm2
+
+psrlq       $1, %xmm2
+psrlq       %xmm0, %xmm2
+psrlq       (%rax), %xmm2
+
+psrlw       $1, %xmm2
+psrlw       %xmm0, %xmm2
+psrlw       (%rax), %xmm2
+
+psubb       %xmm0, %xmm2
+psubb       (%rax), %xmm2
+
+psubd       %xmm0, %xmm2
+psubd       (%rax), %xmm2
+
+psubq       %mm0, %mm2
+psubq       (%rax), %mm2
+
+psubq       %xmm0, %xmm2
+psubq       (%rax), %xmm2
+
+psubsb      %xmm0, %xmm2
+psubsb      (%rax), %xmm2
+
+psubsw      %xmm0, %xmm2
+psubsw      (%rax), %xmm2
+
+psubusb     %xmm0, %xmm2
+psubusb     (%rax), %xmm2
+
+psubusw     %xmm0, %xmm2
+psubusw     (%rax), %xmm2
+
+psubw       %xmm0, %xmm2
+psubw       (%rax), %xmm2
+
+punpckhbw   %xmm0, %xmm2
+punpckhbw   (%rax), %xmm2
+
+punpckhdq   %xmm0, %xmm2
+punpckhdq   (%rax), %xmm2
+
+punpckhqdq  %xmm0, %xmm2
+punpckhqdq  (%rax), %xmm2
+
+punpckhwd   %xmm0, %xmm2
+punpckhwd   (%rax), %xmm2
+
+punpcklbw   %xmm0, %xmm2
+punpcklbw   (%rax), %xmm2
+
+punpckldq   %xmm0, %xmm2
+punpckldq   (%rax), %xmm2
+
+punpcklqdq  %xmm0, %xmm2
+punpcklqdq  (%rax), %xmm2
+
+punpcklwd   %xmm0, %xmm2
+punpcklwd   (%rax), %xmm2
+
+pxor        %xmm0, %xmm2
+pxor        (%rax), %xmm2
+
+shufpd      $1, %xmm0, %xmm2
+shufpd      $1, (%rax), %xmm2
+
+sqrtpd      %xmm0, %xmm2
+sqrtpd      (%rax), %xmm2
+
+sqrtsd      %xmm0, %xmm2
+sqrtsd      (%rax), %xmm2
+
+subpd       %xmm0, %xmm2
+subpd       (%rax), %xmm2
+
+subsd       %xmm0, %xmm2
+subsd       (%rax), %xmm2
+
+ucomisd     %xmm0, %xmm1
+ucomisd     (%rax), %xmm1
+
+unpckhpd    %xmm0, %xmm2
+unpckhpd    (%rax), %xmm2
+
+unpcklpd    %xmm0, %xmm2
+unpcklpd    (%rax), %xmm2
+
+xorpd       %xmm0, %xmm2
+xorpd       (%rax), %xmm2
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      3     0.50                        addpd	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   addpd	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        addsd	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   addsd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        andnpd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   andnpd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        andpd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   andpd	(%rax), %xmm2
+# CHECK-NEXT:  1      5     0.33    *      *      U     clflush	(%rax)
+# CHECK-NEXT:  1      2     1.00                        cmpeqpd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   cmpeqpd	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        cmpeqsd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   cmpeqsd	(%rax), %xmm2
+# CHECK-NEXT:  2      4     1.00                        comisd	%xmm0, %xmm1
+# CHECK-NEXT:  2      11    1.00    *                   comisd	(%rax), %xmm1
+# CHECK-NEXT:  1      3     0.50                        cvtdq2pd	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   cvtdq2pd	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        cvtdq2ps	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   cvtdq2ps	(%rax), %xmm2
+# CHECK-NEXT:  1      3     1.00                        cvtpd2dq	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    1.00    *                   cvtpd2dq	(%rax), %xmm2
+# CHECK-NEXT:  1      3     1.00                        cvtpd2pi	%xmm0, %mm2
+# CHECK-NEXT:  1      10    1.00    *                   cvtpd2pi	(%rax), %mm2
+# CHECK-NEXT:  1      3     0.50                        cvtpd2ps	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   cvtpd2ps	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        cvtpi2pd	%mm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   cvtpi2pd	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        cvtps2dq	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   cvtps2dq	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        cvtps2pd	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   cvtps2pd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        cvtsd2si	%xmm0, %ecx
+# CHECK-NEXT:  1      1     0.50                        cvtsd2si	%xmm0, %rcx
+# CHECK-NEXT:  1      8     0.50    *                   cvtsd2si	(%rax), %ecx
+# CHECK-NEXT:  1      8     0.50    *                   cvtsd2si	(%rax), %rcx
+# CHECK-NEXT:  1      3     0.50                        cvtsd2ss	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   cvtsd2ss	(%rax), %xmm2
+# CHECK-NEXT:  2      5     1.00                        cvtsi2sd	%ecx, %xmm2
+# CHECK-NEXT:  2      5     1.00                        cvtsi2sd	%rcx, %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   cvtsi2sdl	(%rax), %xmm2
+# CHECK-NEXT:  1      11    1.00    *                   cvtsi2sdq	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        cvtss2sd	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   cvtss2sd	(%rax), %xmm2
+# CHECK-NEXT:  1      3     1.00                        cvttpd2dq	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    1.00    *                   cvttpd2dq	(%rax), %xmm2
+# CHECK-NEXT:  1      3     1.00                        cvttpd2pi	%xmm0, %mm2
+# CHECK-NEXT:  1      10    1.00    *                   cvttpd2pi	(%rax), %mm2
+# CHECK-NEXT:  1      3     0.50                        cvttps2dq	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   cvttps2dq	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        cvttsd2si	%xmm0, %ecx
+# CHECK-NEXT:  1      1     0.50                        cvttsd2si	%xmm0, %rcx
+# CHECK-NEXT:  1      8     0.50    *                   cvttsd2si	(%rax), %ecx
+# CHECK-NEXT:  1      8     0.50    *                   cvttsd2si	(%rax), %rcx
+# CHECK-NEXT:  1      13    5.00                        divpd	%xmm0, %xmm2
+# CHECK-NEXT:  1      20    5.00    *                   divpd	(%rax), %xmm2
+# CHECK-NEXT:  1      13    5.00                        divsd	%xmm0, %xmm2
+# CHECK-NEXT:  1      20    5.00    *                   divsd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     10.00   *      *      U     lfence
+# CHECK-NEXT:  1      1     1.00    *      *      U     maskmovdqu	%xmm0, %xmm1
+# CHECK-NEXT:  1      1     0.50                        maxpd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   maxpd	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        maxsd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   maxsd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     25.00   *      *      U     mfence
+# CHECK-NEXT:  1      1     0.50                        minpd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   minpd	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        minsd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   minsd	(%rax), %xmm2
+# CHECK-NEXT:  1      0     0.17                        movapd	%xmm0, %xmm2
+# CHECK-NEXT:  1      1     1.00           *            movapd	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   movapd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     1.00                        movd	%eax, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   movd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     1.00                        movd	%xmm0, %ecx
+# CHECK-NEXT:  1      1     1.00           *            movd	%xmm0, (%rax)
+# CHECK-NEXT:  1      0     0.17                        movdqa	%xmm0, %xmm2
+# CHECK-NEXT:  1      1     1.00           *            movdqa	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   movdqa	(%rax), %xmm2
+# CHECK-NEXT:  1      0     0.17                        movdqu	%xmm0, %xmm2
+# CHECK-NEXT:  1      1     1.00           *            movdqu	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   movdqu	(%rax), %xmm2
+# CHECK-NEXT:  1      0     0.17                        movdq2q	%xmm0, %mm2
+# CHECK-NEXT:  2      2     1.00           *            movhpd	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   movhpd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     1.00           *            movlpd	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   movlpd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     1.00                        movmskpd	%xmm0, %ecx
+# CHECK-NEXT:  1      1     1.00           *            movntil	%eax, (%rax)
+# CHECK-NEXT:  1      1     1.00           *            movntiq	%rax, (%rax)
+# CHECK-NEXT:  1      1     1.00           *            movntdq	%xmm0, (%rax)
+# CHECK-NEXT:  1      1     1.00           *            movntpd	%xmm0, (%rax)
+# CHECK-NEXT:  1      1     0.25                        movq	%xmm0, %xmm2
+# CHECK-NEXT:  1      1     1.00                        movq	%rax, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   movq	(%rax), %xmm2
+# CHECK-NEXT:  1      1     1.00                        movq	%xmm0, %rcx
+# CHECK-NEXT:  1      1     1.00           *            movq	%xmm0, (%rax)
+# CHECK-NEXT:  2      1     0.50                        movq2dq	%mm0, %xmm2
+# CHECK-NEXT:  1      1     0.50                        movsd	%xmm0, %xmm2
+# CHECK-NEXT:  1      1     1.00           *            movsd	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   movsd	(%rax), %xmm2
+# CHECK-NEXT:  1      0     0.17                        movupd	%xmm0, %xmm2
+# CHECK-NEXT:  1      1     1.00           *            movupd	%xmm0, (%rax)
+# CHECK-NEXT:  1      8     0.50    *                   movupd	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        mulpd	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   mulpd	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        mulsd	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   mulsd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        orpd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   orpd	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        packssdw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   packssdw	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        packsswb	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   packsswb	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        packuswb	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   packuswb	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        paddb	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   paddb	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        paddd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   paddd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        paddq	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   paddq	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        paddq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   paddq	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        paddsb	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   paddsb	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        paddsw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   paddsw	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        paddusb	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   paddusb	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        paddusw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   paddusw	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        paddw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   paddw	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pand	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pand	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pandn	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pandn	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        pavgb	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pavgb	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        pavgw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pavgw	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pcmpeqb	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pcmpeqb	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pcmpeqd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pcmpeqd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pcmpeqw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pcmpeqw	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pcmpgtb	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pcmpgtb	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pcmpgtd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pcmpgtd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pcmpgtw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pcmpgtw	(%rax), %xmm2
+# CHECK-NEXT:  2      1     1.00                        pextrw	$1, %xmm0, %ecx
+# CHECK-NEXT:  2      2     1.00                        pinsrw	$1, %eax, %xmm0
+# CHECK-NEXT:  1      8     1.50    *                   pinsrw	$1, (%rax), %xmm0
+# CHECK-NEXT:  1      3     0.50                        pmaddwd	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   pmaddwd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pmaxsw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pmaxsw	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pmaxub	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pmaxub	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pminsw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pminsw	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pminub	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pminub	(%rax), %xmm2
+# CHECK-NEXT:  1      1     1.00                        pmovmskb	%xmm0, %ecx
+# CHECK-NEXT:  1      3     0.50                        pmulhuw	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   pmulhuw	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        pmulhw	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   pmulhw	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        pmullw	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   pmullw	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        pmuludq	%mm0, %mm2
+# CHECK-NEXT:  1      10    0.50    *                   pmuludq	(%rax), %mm2
+# CHECK-NEXT:  1      3     0.50                        pmuludq	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   pmuludq	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        por	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   por	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        psadbw	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   psadbw	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        pshufd	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pshufd	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        pshufhw	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pshufhw	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        pshuflw	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pshuflw	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        pslld	$1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        pslld	%xmm0, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   pslld	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        pslldq	$1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        psllq	$1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        psllq	%xmm0, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   psllq	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        psllw	$1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        psllw	%xmm0, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   psllw	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        psrad	$1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        psrad	%xmm0, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   psrad	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        psraw	$1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        psraw	%xmm0, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   psraw	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        psrld	$1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        psrld	%xmm0, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   psrld	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        psrldq	$1, %xmm2
+# CHECK-NEXT:  1      1     0.50                        psrlq	$1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        psrlq	%xmm0, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   psrlq	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        psrlw	$1, %xmm2
+# CHECK-NEXT:  1      2     1.00                        psrlw	%xmm0, %xmm2
+# CHECK-NEXT:  1      9     1.00    *                   psrlw	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        psubb	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   psubb	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        psubd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   psubd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        psubq	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   psubq	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.25                        psubq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   psubq	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        psubsb	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   psubsb	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        psubsw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   psubsw	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        psubusb	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   psubusb	(%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        psubusw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   psubusw	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        psubw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   psubw	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        punpckhbw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   punpckhbw	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        punpckhdq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   punpckhdq	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        punpckhqdq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   punpckhqdq	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        punpckhwd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   punpckhwd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        punpcklbw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   punpcklbw	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        punpckldq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   punpckldq	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        punpcklqdq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   punpcklqdq	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        punpcklwd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   punpcklwd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pxor	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pxor	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        shufpd	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   shufpd	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      21    9.00                        sqrtpd	%xmm0, %xmm2
+# CHECK-NEXT:  1      28    9.00    *                   sqrtpd	(%rax), %xmm2
+# CHECK-NEXT:  1      21    9.00                        sqrtsd	%xmm0, %xmm2
+# CHECK-NEXT:  1      28    9.00    *                   sqrtsd	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        subpd	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   subpd	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        subsd	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   subsd	(%rax), %xmm2
+# CHECK-NEXT:  2      4     1.00                        ucomisd	%xmm0, %xmm1
+# CHECK-NEXT:  2      11    1.00    *                   ucomisd	(%rax), %xmm1
+# CHECK-NEXT:  1      1     0.50                        unpckhpd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   unpckhpd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        unpcklpd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   unpcklpd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        xorpd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   xorpd	(%rax), %xmm2
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 1.00   1.00   1.00   25.00  25.00  25.00  25.00   -     58.25  140.25 81.25  53.25  72.00  72.00  12.00  54.67  54.67  54.67  39.33  39.33  39.33  8.00   8.00
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     addpd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     addpd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     addsd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     addsd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     andnpd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     andnpd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     andpd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     andpd	(%rax), %xmm2
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     clflush	(%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     cmpeqpd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpeqpd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     cmpeqsd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpeqsd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     comisd	%xmm0, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     comisd	(%rax), %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     cvtdq2pd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtdq2pd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     cvtdq2ps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtdq2ps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     cvtpd2dq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtpd2dq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     cvtpd2pi	%xmm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtpd2pi	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     cvtpd2ps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtpd2ps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     cvtpi2pd	%mm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtpi2pd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     cvtps2dq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtps2dq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     cvtps2pd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtps2pd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     cvtsd2si	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     cvtsd2si	%xmm0, %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtsd2si	(%rax), %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtsd2si	(%rax), %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     cvtsd2ss	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtsd2ss	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     cvtsi2sd	%ecx, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     cvtsi2sd	%rcx, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtsi2sdl	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtsi2sdq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     cvtss2sd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvtss2sd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     cvttpd2dq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvttpd2dq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -     cvttpd2pi	%xmm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00   1.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvttpd2pi	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     cvttps2dq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvttps2dq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     cvttsd2si	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     cvttsd2si	%xmm0, %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvttsd2si	(%rax), %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cvttsd2si	(%rax), %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     divpd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     divpd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -      -      -      -      -      -      -      -      -      -      -      -     divsd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     5.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     divsd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     10.00  10.00  10.00   -      -      -      -      -     lfence
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   maskmovdqu	%xmm0, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     maxpd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     maxpd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     maxsd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     maxsd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     mfence
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     minpd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     minpd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     minsd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     minsd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movapd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movapd	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movapd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     movd	%eax, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     movd	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movd	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movdqa	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movdqa	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movdqa	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movdqu	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movdqu	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movdqu	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movdq2q	%xmm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movhpd	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movhpd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movlpd	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movlpd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     movmskpd	%xmm0, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -     1.00   1.00   movntil	%eax, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -     1.00   1.00   movntiq	%rax, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movntdq	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movntpd	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     movq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     movq	%rax, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     movq	%xmm0, %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movq	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -     movq2dq	%mm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     movsd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movsd	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movsd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movupd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   movupd	%xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movupd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     mulpd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     mulpd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     mulsd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     mulsd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     orpd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     orpd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     packssdw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     packssdw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     packsswb	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     packsswb	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     packuswb	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     packuswb	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     paddb	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     paddb	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     paddd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     paddd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     paddq	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     paddq	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     paddq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     paddq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     paddsb	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     paddsb	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     paddsw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     paddsw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     paddusb	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     paddusb	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     paddusw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     paddusw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     paddw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     paddw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pand	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pand	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pandn	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pandn	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     pavgb	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pavgb	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     pavgw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pavgw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pcmpeqb	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pcmpeqb	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pcmpeqd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pcmpeqd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pcmpeqw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pcmpeqw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pcmpgtb	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pcmpgtb	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pcmpgtd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pcmpgtd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pcmpgtw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pcmpgtw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     pextrw	$1, %xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     pinsrw	$1, %eax, %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pinsrw	$1, (%rax), %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     pmaddwd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmaddwd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pmaxsw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmaxsw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pmaxub	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmaxub	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pminsw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pminsw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pminub	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pminub	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     pmovmskb	%xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     pmulhuw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmulhuw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     pmulhw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmulhw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     pmullw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmullw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     pmuludq	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmuludq	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     pmuludq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmuludq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     por	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     por	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     psadbw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psadbw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     pshufd	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pshufd	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     pshufhw	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pshufhw	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     pshuflw	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pshuflw	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     pslld	$1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     pslld	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pslld	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     pslldq	$1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psllq	$1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     psllq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psllq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psllw	$1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     psllw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psllw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psrad	$1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     psrad	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psrad	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psraw	$1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     psraw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psraw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psrld	$1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     psrld	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psrld	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     psrldq	$1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psrlq	$1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     psrlq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psrlq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     psrlw	$1, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     psrlw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psrlw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     psubb	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psubb	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     psubd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psubd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     psubq	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psubq	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     psubq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psubq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     psubsb	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psubsb	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     psubsw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psubsw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     psubusb	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psubusb	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     psubusw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psubusw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     psubw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psubw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     punpckhbw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     punpckhbw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     punpckhdq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     punpckhdq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     punpckhqdq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     punpckhqdq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     punpckhwd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     punpckhwd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     punpcklbw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     punpcklbw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     punpckldq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     punpckldq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     punpcklqdq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     punpcklqdq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     punpcklwd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     punpcklwd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pxor	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pxor	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     shufpd	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     shufpd	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -      -      -      -      -      -      -      -      -      -      -      -     sqrtpd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     sqrtpd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -      -      -      -      -      -      -      -      -      -      -      -     sqrtsd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     9.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     sqrtsd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     subpd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     subpd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     subsd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     subsd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     ucomisd	%xmm0, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     ucomisd	(%rax), %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     unpckhpd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     unpckhpd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     unpcklpd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     unpcklpd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     xorpd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     xorpd	(%rax), %xmm2
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-sse3.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-sse3.s
new file mode 100644
index 0000000000000..51ffdc62664c2
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-sse3.s
@@ -0,0 +1,119 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+addsubpd  %xmm0, %xmm2
+addsubpd  (%rax),  %xmm2
+
+addsubps  %xmm0, %xmm2
+addsubps  (%rax), %xmm2
+
+haddpd    %xmm0, %xmm2
+haddpd    (%rax), %xmm2
+
+haddps    %xmm0, %xmm2
+haddps    (%rax), %xmm2
+
+hsubpd    %xmm0, %xmm2
+hsubpd    (%rax), %xmm2
+
+hsubps    %xmm0, %xmm2
+hsubps    (%rax), %xmm2
+
+lddqu     (%rax), %xmm2
+
+monitor
+
+movddup   %xmm0, %xmm2
+movddup   (%rax), %xmm2
+
+movshdup  %xmm0, %xmm2
+movshdup  (%rax), %xmm2
+
+movsldup  %xmm0, %xmm2
+movsldup  (%rax), %xmm2
+
+mwait
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      3     0.50                        addsubpd	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   addsubpd	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        addsubps	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   addsubps	(%rax), %xmm2
+# CHECK-NEXT:  3      4     2.00                        haddpd	%xmm0, %xmm2
+# CHECK-NEXT:  3      11    2.00    *                   haddpd	(%rax), %xmm2
+# CHECK-NEXT:  3      4     2.00                        haddps	%xmm0, %xmm2
+# CHECK-NEXT:  3      11    2.00    *                   haddps	(%rax), %xmm2
+# CHECK-NEXT:  3      4     2.00                        hsubpd	%xmm0, %xmm2
+# CHECK-NEXT:  3      11    2.00    *                   hsubpd	(%rax), %xmm2
+# CHECK-NEXT:  3      4     2.00                        hsubps	%xmm0, %xmm2
+# CHECK-NEXT:  3      11    2.00    *                   hsubps	(%rax), %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   lddqu	(%rax), %xmm2
+# CHECK-NEXT:  100    100   25.00                 U     monitor
+# CHECK-NEXT:  1      1     0.50                        movddup	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   movddup	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        movshdup	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   movshdup	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        movsldup	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   movsldup	(%rax), %xmm2
+# CHECK-NEXT:  100    100   25.00   *      *      U     mwait
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -     50.00  50.00  50.00  50.00   -      -     3.00   21.00  2.00   5.00   5.00    -     3.33   3.33   3.33   3.33   3.33   3.33    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     addsubpd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     addsubpd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     addsubps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     addsubps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -     haddpd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     haddpd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -     haddps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     haddps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -     hsubpd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     hsubpd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -     hsubps	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     hsubps	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     lddqu	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     monitor
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     movddup	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movddup	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     movshdup	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movshdup	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     movsldup	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movsldup	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     mwait
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-sse41.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-sse41.s
new file mode 100644
index 0000000000000..5521c1eae6dfd
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-sse41.s
@@ -0,0 +1,381 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+blendpd     $11, %xmm0, %xmm2
+blendpd     $11, (%rax), %xmm2
+
+blendps     $11, %xmm0, %xmm2
+blendps     $11, (%rax), %xmm2
+
+blendvpd    %xmm0, %xmm2
+blendvpd    (%rax), %xmm2
+
+blendvps    %xmm0, %xmm2
+blendvps    (%rax), %xmm2
+
+dppd        $22, %xmm0, %xmm2
+dppd        $22, (%rax), %xmm2
+
+dpps        $22, %xmm0, %xmm2
+dpps        $22, (%rax), %xmm2
+
+extractps   $1, %xmm0, %rcx
+extractps   $1, %xmm0, (%rax)
+
+insertps    $1, %xmm0, %xmm2
+insertps    $1, (%rax), %xmm2
+
+movntdqa    (%rax), %xmm2
+
+mpsadbw     $1, %xmm0, %xmm2
+mpsadbw     $1, (%rax), %xmm2
+
+packusdw    %xmm0, %xmm2
+packusdw    (%rax), %xmm2
+
+pblendvb    %xmm0, %xmm2
+pblendvb    (%rax), %xmm2
+
+pblendw     $11, %xmm0, %xmm2
+pblendw     $11, (%rax), %xmm2
+
+pcmpeqq     %xmm0, %xmm2
+pcmpeqq     (%rax), %xmm2
+
+pextrb      $1, %xmm0, %ecx
+pextrb      $1, %xmm0, (%rax)
+
+pextrd      $1, %xmm0, %ecx
+pextrd      $1, %xmm0, (%rax)
+
+pextrq      $1, %xmm0, %rcx
+pextrq      $1, %xmm0, (%rax)
+
+pextrw      $1, %xmm0, (%rax)
+
+phminposuw  %xmm0, %xmm2
+phminposuw  (%rax), %xmm2
+
+pinsrb      $1, %eax, %xmm1
+pinsrb      $1, (%rax), %xmm1
+
+pinsrd      $1, %eax, %xmm1
+pinsrd      $1, (%rax), %xmm1
+
+pinsrq      $1, %rax, %xmm1
+pinsrq      $1, (%rax), %xmm1
+
+pmaxsb      %xmm0, %xmm2
+pmaxsb      (%rax), %xmm2
+
+pmaxsd      %xmm0, %xmm2
+pmaxsd      (%rax), %xmm2
+
+pmaxud      %xmm0, %xmm2
+pmaxud      (%rax), %xmm2
+
+pmaxuw      %xmm0, %xmm2
+pmaxuw      (%rax), %xmm2
+
+pminsb      %xmm0, %xmm2
+pminsb      (%rax), %xmm2
+
+pminsd      %xmm0, %xmm2
+pminsd      (%rax), %xmm2
+
+pminud      %xmm0, %xmm2
+pminud      (%rax), %xmm2
+
+pminuw      %xmm0, %xmm2
+pminuw      (%rax), %xmm2
+
+pmovsxbd    %xmm0, %xmm2
+pmovsxbd    (%rax), %xmm2
+
+pmovsxbq    %xmm0, %xmm2
+pmovsxbq    (%rax), %xmm2
+
+pmovsxbw    %xmm0, %xmm2
+pmovsxbw    (%rax), %xmm2
+
+pmovsxdq    %xmm0, %xmm2
+pmovsxdq    (%rax), %xmm2
+
+pmovsxwd    %xmm0, %xmm2
+pmovsxwd    (%rax), %xmm2
+
+pmovsxwq    %xmm0, %xmm2
+pmovsxwq    (%rax), %xmm2
+
+pmovzxbd    %xmm0, %xmm2
+pmovzxbd    (%rax), %xmm2
+
+pmovzxbq    %xmm0, %xmm2
+pmovzxbq    (%rax), %xmm2
+
+pmovzxbw    %xmm0, %xmm2
+pmovzxbw    (%rax), %xmm2
+
+pmovzxdq    %xmm0, %xmm2
+pmovzxdq    (%rax), %xmm2
+
+pmovzxwd    %xmm0, %xmm2
+pmovzxwd    (%rax), %xmm2
+
+pmovzxwq    %xmm0, %xmm2
+pmovzxwq    (%rax), %xmm2
+
+pmuldq      %xmm0, %xmm2
+pmuldq      (%rax), %xmm2
+
+pmulld      %xmm0, %xmm2
+pmulld      (%rax), %xmm2
+
+ptest       %xmm0, %xmm1
+ptest       (%rax), %xmm1
+
+roundpd     $1, %xmm0, %xmm2
+roundpd     $1, (%rax), %xmm2
+
+roundps     $1, %xmm0, %xmm2
+roundps     $1, (%rax), %xmm2
+
+roundsd     $1, %xmm0, %xmm2
+roundsd     $1, (%rax), %xmm2
+
+roundss     $1, %xmm0, %xmm2
+roundss     $1, (%rax), %xmm2
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     0.50                        blendpd	$11, %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   blendpd	$11, (%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        blendps	$11, %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   blendps	$11, (%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        blendvpd	%xmm0, %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   blendvpd	%xmm0, (%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        blendvps	%xmm0, %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   blendvps	%xmm0, (%rax), %xmm2
+# CHECK-NEXT:  3      7     3.00                        dppd	$22, %xmm0, %xmm2
+# CHECK-NEXT:  5      14    3.00    *                   dppd	$22, (%rax), %xmm2
+# CHECK-NEXT:  8      11    4.00                        dpps	$22, %xmm0, %xmm2
+# CHECK-NEXT:  10     18    4.00    *                   dpps	$22, (%rax), %xmm2
+# CHECK-NEXT:  2      1     1.00                        extractps	$1, %xmm0, %ecx
+# CHECK-NEXT:  2      2     1.00           *            extractps	$1, %xmm0, (%rax)
+# CHECK-NEXT:  1      1     0.50                        insertps	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   insertps	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   movntdqa	(%rax), %xmm2
+# CHECK-NEXT:  4      4     2.00                        mpsadbw	$1, %xmm0, %xmm2
+# CHECK-NEXT:  6      11    2.00    *                   mpsadbw	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      2     1.00                        packusdw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   packusdw	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        pblendvb	%xmm0, %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pblendvb	%xmm0, (%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pblendw	$11, %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pblendw	$11, (%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pcmpeqq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pcmpeqq	(%rax), %xmm2
+# CHECK-NEXT:  2      1     1.00                        pextrb	$1, %xmm0, %ecx
+# CHECK-NEXT:  2      2     1.00           *            pextrb	$1, %xmm0, (%rax)
+# CHECK-NEXT:  2      1     1.00                        pextrd	$1, %xmm0, %ecx
+# CHECK-NEXT:  2      2     1.00           *            pextrd	$1, %xmm0, (%rax)
+# CHECK-NEXT:  2      1     1.00                        pextrq	$1, %xmm0, %rcx
+# CHECK-NEXT:  2      2     1.00           *            pextrq	$1, %xmm0, (%rax)
+# CHECK-NEXT:  2      2     1.00           *            pextrw	$1, %xmm0, (%rax)
+# CHECK-NEXT:  1      3     0.50                        phminposuw	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   phminposuw	(%rax), %xmm2
+# CHECK-NEXT:  2      2     1.00                        pinsrb	$1, %eax, %xmm1
+# CHECK-NEXT:  1      8     1.50    *                   pinsrb	$1, (%rax), %xmm1
+# CHECK-NEXT:  2      2     1.00                        pinsrd	$1, %eax, %xmm1
+# CHECK-NEXT:  1      8     1.50    *                   pinsrd	$1, (%rax), %xmm1
+# CHECK-NEXT:  2      2     1.00                        pinsrq	$1, %rax, %xmm1
+# CHECK-NEXT:  1      8     1.50    *                   pinsrq	$1, (%rax), %xmm1
+# CHECK-NEXT:  1      1     0.25                        pmaxsb	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pmaxsb	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pmaxsd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pmaxsd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pmaxud	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pmaxud	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pmaxuw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pmaxuw	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pminsb	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pminsb	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pminsd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pminsd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pminud	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pminud	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pminuw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pminuw	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        pmovsxbd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pmovsxbd	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        pmovsxbq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pmovsxbq	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        pmovsxbw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pmovsxbw	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        pmovsxdq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pmovsxdq	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        pmovsxwd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pmovsxwd	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        pmovsxwq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pmovsxwq	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        pmovzxbd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pmovzxbd	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        pmovzxbq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pmovzxbq	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        pmovzxbw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pmovzxbw	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        pmovzxdq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pmovzxdq	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        pmovzxwd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pmovzxwd	(%rax), %xmm2
+# CHECK-NEXT:  1      4     2.00                        pmovzxwq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pmovzxwq	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        pmuldq	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   pmuldq	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        pmulld	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   pmulld	(%rax), %xmm2
+# CHECK-NEXT:  2      1     1.00                        ptest	%xmm0, %xmm1
+# CHECK-NEXT:  2      8     1.00    *                   ptest	(%rax), %xmm1
+# CHECK-NEXT:  1      3     0.50                        roundpd	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   roundpd	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        roundps	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   roundps	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        roundsd	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   roundsd	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        roundss	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   roundss	$1, (%rax), %xmm2
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     31.00  61.50  46.50  16.00  35.50  35.50  7.00   16.33  16.33  16.33  14.67  14.67  14.67  2.50   2.50
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     blendpd	$11, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     blendpd	$11, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     blendps	$11, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     blendps	$11, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     blendvpd	%xmm0, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     blendvpd	%xmm0, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     blendvps	%xmm0, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     blendvps	%xmm0, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     dppd	$22, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00   3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     dppd	$22, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     4.00   4.00    -      -      -      -      -      -      -      -      -      -      -      -      -     dpps	$22, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     4.00   4.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     dpps	$22, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     extractps	$1, %xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   extractps	$1, %xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     insertps	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     insertps	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movntdqa	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00   2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     mpsadbw	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00   2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     mpsadbw	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     packusdw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     packusdw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     pblendvb	%xmm0, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pblendvb	%xmm0, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pblendw	$11, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pblendw	$11, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pcmpeqq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pcmpeqq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     pextrb	$1, %xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   pextrb	$1, %xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     pextrd	$1, %xmm0, %ecx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   pextrd	$1, %xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     pextrq	$1, %xmm0, %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   pextrq	$1, %xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     0.50   0.50   1.00   0.33   0.33   0.33    -      -      -     0.50   0.50   pextrw	$1, %xmm0, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     phminposuw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     phminposuw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     pinsrb	$1, %eax, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pinsrb	$1, (%rax), %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     pinsrd	$1, %eax, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pinsrd	$1, (%rax), %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -     pinsrq	$1, %rax, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -     1.50   1.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pinsrq	$1, (%rax), %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pmaxsb	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmaxsb	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pmaxsd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmaxsd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pmaxud	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmaxud	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pmaxuw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmaxuw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pminsb	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pminsb	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pminsd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pminsd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pminud	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pminud	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pminuw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pminuw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     pmovsxbd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmovsxbd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     pmovsxbq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmovsxbq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     pmovsxbw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmovsxbw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     pmovsxdq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmovsxdq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     pmovsxwd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmovsxwd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     pmovsxwq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmovsxwq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     pmovzxbd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmovzxbd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     pmovzxbq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmovzxbq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     pmovzxbw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmovzxbw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     pmovzxdq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmovzxdq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     pmovzxwd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmovzxwd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -     pmovzxwq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmovzxwq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     pmuldq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmuldq	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     pmulld	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmulld	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50   1.00    -      -      -      -      -      -      -      -     ptest	%xmm0, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     1.00   1.00   1.00   0.33   0.33   0.33   0.33   0.33   0.33    -      -     ptest	(%rax), %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     roundpd	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     roundpd	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     roundps	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     roundps	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     roundsd	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     roundsd	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     roundss	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     roundss	$1, (%rax), %xmm2
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-sse42.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-sse42.s
new file mode 100644
index 0000000000000..015d37e3e6296
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-sse42.s
@@ -0,0 +1,114 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+crc32b      %al, %ecx
+crc32b      (%rax), %ecx
+
+crc32l      %eax, %ecx
+crc32l      (%rax), %ecx
+
+crc32w      %ax, %ecx
+crc32w      (%rax), %ecx
+
+crc32b      %al, %rcx
+crc32b      (%rax), %rcx
+
+crc32q      %rax, %rcx
+crc32q      (%rax), %rcx
+
+pcmpestri   $1, %xmm0, %xmm2
+pcmpestri   $1, (%rax), %xmm2
+
+pcmpestrm   $1, %xmm0, %xmm2
+pcmpestrm   $1, (%rax), %xmm2
+
+pcmpistri   $1, %xmm0, %xmm2
+pcmpistri   $1, (%rax), %xmm2
+
+pcmpistrm   $1, %xmm0, %xmm2
+pcmpistrm   $1, (%rax), %xmm2
+
+pcmpgtq     %xmm0, %xmm2
+pcmpgtq     (%rax), %xmm2
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      3     1.00                        crc32b	%al, %ecx
+# CHECK-NEXT:  1      7     1.00    *                   crc32b	(%rax), %ecx
+# CHECK-NEXT:  1      3     1.00                        crc32l	%eax, %ecx
+# CHECK-NEXT:  1      7     1.00    *                   crc32l	(%rax), %ecx
+# CHECK-NEXT:  1      3     1.00                        crc32w	%ax, %ecx
+# CHECK-NEXT:  1      7     1.00    *                   crc32w	(%rax), %ecx
+# CHECK-NEXT:  1      3     1.00                        crc32b	%al, %rcx
+# CHECK-NEXT:  1      7     1.00    *                   crc32b	(%rax), %rcx
+# CHECK-NEXT:  1      3     1.00                        crc32q	%rax, %rcx
+# CHECK-NEXT:  1      7     1.00    *                   crc32q	(%rax), %rcx
+# CHECK-NEXT:  8      6     3.00                        pcmpestri	$1, %xmm0, %xmm2
+# CHECK-NEXT:  12     13    3.00    *                   pcmpestri	$1, (%rax), %xmm2
+# CHECK-NEXT:  7      6     3.00                        pcmpestrm	$1, %xmm0, %xmm2
+# CHECK-NEXT:  12     13    3.00    *                   pcmpestrm	$1, (%rax), %xmm2
+# CHECK-NEXT:  4      2     2.00                        pcmpistri	$1, %xmm0, %xmm2
+# CHECK-NEXT:  4      9     2.00    *                   pcmpistri	$1, (%rax), %xmm2
+# CHECK-NEXT:  3      6     2.00                        pcmpistrm	$1, %xmm0, %xmm2
+# CHECK-NEXT:  4      13    2.00    *                   pcmpistrm	$1, (%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pcmpgtq	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pcmpgtq	(%rax), %xmm2
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 1.67   1.67   1.67    -     10.00   -      -      -     20.50  20.50  20.50  20.50  2.50   2.50    -     3.33   3.33   3.33   3.33   3.33   3.33    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     crc32b	%al, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     crc32b	(%rax), %ecx
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     crc32l	%eax, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     crc32l	(%rax), %ecx
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     crc32w	%ax, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     crc32w	(%rax), %ecx
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     crc32b	%al, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     crc32b	(%rax), %rcx
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     crc32q	%rax, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     crc32q	(%rax), %rcx
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00   3.00   3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     pcmpestri	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00   3.00   3.00   3.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pcmpestri	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00   3.00   3.00   3.00    -      -      -      -      -      -      -      -      -      -      -     pcmpestrm	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     3.00   3.00   3.00   3.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pcmpestrm	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00   2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     pcmpistri	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00   2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pcmpistri	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00   2.00   2.00    -      -      -      -      -      -      -      -      -      -      -     pcmpistrm	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00   2.00   2.00   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pcmpistrm	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pcmpgtq	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pcmpgtq	(%rax), %xmm2
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-ssse3.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-ssse3.s
new file mode 100644
index 0000000000000..173c72171aafe
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-ssse3.s
@@ -0,0 +1,268 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+pabsb       %mm0, %mm2
+pabsb       (%rax), %mm2
+
+pabsb       %xmm0, %xmm2
+pabsb       (%rax), %xmm2
+
+pabsd       %mm0, %mm2
+pabsd       (%rax), %mm2
+
+pabsd       %xmm0, %xmm2
+pabsd       (%rax), %xmm2
+
+pabsw       %mm0, %mm2
+pabsw       (%rax), %mm2
+
+pabsw       %xmm0, %xmm2
+pabsw       (%rax), %xmm2
+
+palignr     $1, %mm0, %mm2
+palignr     $1, (%rax), %mm2
+
+palignr     $1, %xmm0, %xmm2
+palignr     $1, (%rax), %xmm2
+
+phaddd      %mm0, %mm2
+phaddd      (%rax), %mm2
+
+phaddd      %xmm0, %xmm2
+phaddd      (%rax), %xmm2
+
+phaddsw     %mm0, %mm2
+phaddsw     (%rax), %mm2
+
+phaddsw     %xmm0, %xmm2
+phaddsw     (%rax), %xmm2
+
+phaddw      %mm0, %mm2
+phaddw      (%rax), %mm2
+
+phaddw      %xmm0, %xmm2
+phaddw      (%rax), %xmm2
+
+phsubd      %mm0, %mm2
+phsubd      (%rax), %mm2
+
+phsubd      %xmm0, %xmm2
+phsubd      (%rax), %xmm2
+
+phsubsw     %mm0, %mm2
+phsubsw     (%rax), %mm2
+
+phsubsw     %xmm0, %xmm2
+phsubsw     (%rax), %xmm2
+
+phsubw      %mm0, %mm2
+phsubw      (%rax), %mm2
+
+phsubw      %xmm0, %xmm2
+phsubw      (%rax), %xmm2
+
+pmaddubsw   %mm0, %mm2
+pmaddubsw   (%rax), %mm2
+
+pmaddubsw   %xmm0, %xmm2
+pmaddubsw   (%rax), %xmm2
+
+pmulhrsw    %mm0, %mm2
+pmulhrsw    (%rax), %mm2
+
+pmulhrsw    %xmm0, %xmm2
+pmulhrsw    (%rax), %xmm2
+
+pshufb      %mm0, %mm2
+pshufb      (%rax), %mm2
+
+pshufb      %xmm0, %xmm2
+pshufb      (%rax), %xmm2
+
+psignb      %mm0, %mm2
+psignb      (%rax), %mm2
+
+psignb      %xmm0, %xmm2
+psignb      (%rax), %xmm2
+
+psignd      %mm0, %mm2
+psignd      (%rax), %mm2
+
+psignd      %xmm0, %xmm2
+psignd      (%rax), %xmm2
+
+psignw      %mm0, %mm2
+psignw      (%rax), %mm2
+
+psignw      %xmm0, %xmm2
+psignw      (%rax), %xmm2
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     0.25                        pabsb	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pabsb	(%rax), %mm2
+# CHECK-NEXT:  1      2     1.00                        pabsb	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pabsb	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pabsd	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pabsd	(%rax), %mm2
+# CHECK-NEXT:  1      2     1.00                        pabsd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pabsd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        pabsw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pabsw	(%rax), %mm2
+# CHECK-NEXT:  1      2     1.00                        pabsw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pabsw	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        palignr	$1, %mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   palignr	$1, (%rax), %mm2
+# CHECK-NEXT:  1      2     1.00                        palignr	$1, %xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   palignr	$1, (%rax), %xmm2
+# CHECK-NEXT:  3      2     2.00                        phaddd	%mm0, %mm2
+# CHECK-NEXT:  4      9     2.00    *                   phaddd	(%rax), %mm2
+# CHECK-NEXT:  3      2     2.00                        phaddd	%xmm0, %xmm2
+# CHECK-NEXT:  3      9     2.00    *                   phaddd	(%rax), %xmm2
+# CHECK-NEXT:  3      2     2.00                        phaddsw	%mm0, %mm2
+# CHECK-NEXT:  4      9     2.00    *                   phaddsw	(%rax), %mm2
+# CHECK-NEXT:  3      2     2.00                        phaddsw	%xmm0, %xmm2
+# CHECK-NEXT:  3      9     2.00    *                   phaddsw	(%rax), %xmm2
+# CHECK-NEXT:  3      2     2.00                        phaddw	%mm0, %mm2
+# CHECK-NEXT:  4      9     2.00    *                   phaddw	(%rax), %mm2
+# CHECK-NEXT:  3      2     2.00                        phaddw	%xmm0, %xmm2
+# CHECK-NEXT:  3      9     2.00    *                   phaddw	(%rax), %xmm2
+# CHECK-NEXT:  3      2     2.00                        phsubd	%mm0, %mm2
+# CHECK-NEXT:  4      9     2.00    *                   phsubd	(%rax), %mm2
+# CHECK-NEXT:  3      2     2.00                        phsubd	%xmm0, %xmm2
+# CHECK-NEXT:  3      9     2.00    *                   phsubd	(%rax), %xmm2
+# CHECK-NEXT:  3      2     2.00                        phsubsw	%mm0, %mm2
+# CHECK-NEXT:  4      9     2.00    *                   phsubsw	(%rax), %mm2
+# CHECK-NEXT:  3      2     2.00                        phsubsw	%xmm0, %xmm2
+# CHECK-NEXT:  3      9     2.00    *                   phsubsw	(%rax), %xmm2
+# CHECK-NEXT:  3      2     2.00                        phsubw	%mm0, %mm2
+# CHECK-NEXT:  4      9     2.00    *                   phsubw	(%rax), %mm2
+# CHECK-NEXT:  3      2     2.00                        phsubw	%xmm0, %xmm2
+# CHECK-NEXT:  3      9     2.00    *                   phsubw	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        pmaddubsw	%mm0, %mm2
+# CHECK-NEXT:  1      10    0.50    *                   pmaddubsw	(%rax), %mm2
+# CHECK-NEXT:  1      3     0.50                        pmaddubsw	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   pmaddubsw	(%rax), %xmm2
+# CHECK-NEXT:  1      3     0.50                        pmulhrsw	%mm0, %mm2
+# CHECK-NEXT:  1      10    0.50    *                   pmulhrsw	(%rax), %mm2
+# CHECK-NEXT:  1      3     0.50                        pmulhrsw	%xmm0, %xmm2
+# CHECK-NEXT:  1      10    0.50    *                   pmulhrsw	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.50                        pshufb	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   pshufb	(%rax), %mm2
+# CHECK-NEXT:  1      1     0.50                        pshufb	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   pshufb	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        psignb	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   psignb	(%rax), %mm2
+# CHECK-NEXT:  1      2     1.00                        psignb	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   psignb	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        psignd	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   psignd	(%rax), %mm2
+# CHECK-NEXT:  1      2     1.00                        psignd	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   psignd	(%rax), %xmm2
+# CHECK-NEXT:  1      1     0.25                        psignw	%mm0, %mm2
+# CHECK-NEXT:  1      8     0.50    *                   psignw	(%rax), %mm2
+# CHECK-NEXT:  1      2     1.00                        psignw	%xmm0, %xmm2
+# CHECK-NEXT:  1      8     0.50    *                   psignw	(%rax), %xmm2
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     62.50  15.00  9.00   8.50   16.00  16.00   -     10.67  10.67  10.67  10.67  10.67  10.67   -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pabsb	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pabsb	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     pabsb	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pabsb	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pabsd	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pabsd	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     pabsd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pabsd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     pabsw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pabsw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     pabsw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pabsw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     palignr	$1, %mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     palignr	$1, (%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     palignr	$1, %xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     palignr	$1, (%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     phaddd	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     phaddd	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     phaddd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     phaddd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     phaddsw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     phaddsw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     phaddsw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     phaddsw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     phaddw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     phaddw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     phaddw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     phaddw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     phsubd	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     phsubd	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     phsubd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     phsubd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     phsubsw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     phsubsw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     phsubsw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     phsubsw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     phsubw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     phsubw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     phsubw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     phsubw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     pmaddubsw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmaddubsw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     pmaddubsw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmaddubsw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     pmulhrsw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmulhrsw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50    -      -      -      -      -      -      -      -      -      -      -     pmulhrsw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50    -      -     0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pmulhrsw	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     pshufb	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pshufb	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -     pshufb	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     0.50   0.50    -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     pshufb	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     psignb	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psignb	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     psignb	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psignb	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     psignd	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psignd	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     psignd	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psignd	(%rax), %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -     psignw	%mm0, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psignw	(%rax), %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     psignw	%xmm0, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.25   0.25   0.25   0.25   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     psignw	(%rax), %xmm2
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-vaes.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-vaes.s
new file mode 100644
index 0000000000000..5389413436f2c
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-vaes.s
@@ -0,0 +1,72 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vaesdec          %ymm0, %ymm1, %ymm3
+vaesdec          (%rax), %ymm1, %ymm3
+
+vaesdeclast      %ymm0, %ymm1, %ymm3
+vaesdeclast      (%rax), %ymm1, %ymm3
+
+vaesenc          %ymm0, %ymm1, %ymm3
+vaesenc          (%rax), %ymm1, %ymm3
+
+vaesenclast      %ymm0, %ymm1, %ymm3
+vaesenclast      (%rax), %ymm1, %ymm3
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      4     0.50                        vaesdec	%ymm0, %ymm1, %ymm3
+# CHECK-NEXT:  1      11    0.50    *                   vaesdec	(%rax), %ymm1, %ymm3
+# CHECK-NEXT:  1      4     0.50                        vaesdeclast	%ymm0, %ymm1, %ymm3
+# CHECK-NEXT:  1      11    0.50    *                   vaesdeclast	(%rax), %ymm1, %ymm3
+# CHECK-NEXT:  1      4     0.50                        vaesenc	%ymm0, %ymm1, %ymm3
+# CHECK-NEXT:  1      11    0.50    *                   vaesenc	(%rax), %ymm1, %ymm3
+# CHECK-NEXT:  1      4     0.50                        vaesenclast	%ymm0, %ymm1, %ymm3
+# CHECK-NEXT:  1      11    0.50    *                   vaesenclast	(%rax), %ymm1, %ymm3
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     4.00   4.00    -      -     2.00   2.00    -     1.33   1.33   1.33   1.33   1.33   1.33    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesdec	%ymm0, %ymm1, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesdec	(%rax), %ymm1, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesdeclast	%ymm0, %ymm1, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesdeclast	(%rax), %ymm1, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesenc	%ymm0, %ymm1, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesenc	(%rax), %ymm1, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     vaesenclast	%ymm0, %ymm1, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vaesenclast	(%rax), %ymm1, %ymm3
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-vpclmulqdq.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-vpclmulqdq.s
new file mode 100644
index 0000000000000..55a36d0f1ea09
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-vpclmulqdq.s
@@ -0,0 +1,51 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+vpclmulqdq    $11,  %ymm0, %ymm1, %ymm3
+vpclmulqdq    $11, (%rax), %ymm1, %ymm3
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  4      4     2.00                        vpclmulqdq	$11, %ymm0, %ymm1, %ymm3
+# CHECK-NEXT:  4      11    2.00    *                   vpclmulqdq	$11, (%rax), %ymm1, %ymm3
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     4.00   4.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpclmulqdq	$11, %ymm0, %ymm1, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00   2.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     vpclmulqdq	$11, (%rax), %ymm1, %ymm3
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-x86_32.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-x86_32.s
new file mode 100644
index 0000000000000..894a221ce7bc1
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-x86_32.s
@@ -0,0 +1,93 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=i686-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+aaa
+
+aad
+aad $7
+
+aam
+aam $7
+
+aas
+
+bound %bx, (%eax)
+bound %ebx, (%eax)
+
+daa
+
+das
+
+into
+
+leave
+
+salc
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  100    100   25.00                       aaa
+# CHECK-NEXT:  100    100   25.00                       aad
+# CHECK-NEXT:  100    100   25.00                       aad	$7
+# CHECK-NEXT:  100    100   25.00                       aam
+# CHECK-NEXT:  100    100   25.00                       aam	$7
+# CHECK-NEXT:  100    100   25.00                       aas
+# CHECK-NEXT:  100    100   25.00                 U     bound	%bx, (%eax)
+# CHECK-NEXT:  100    100   25.00                 U     bound	%ebx, (%eax)
+# CHECK-NEXT:  100    100   25.00                       daa
+# CHECK-NEXT:  100    100   25.00                       das
+# CHECK-NEXT:  100    100   25.00                 U     into
+# CHECK-NEXT:  1      1     0.25    *                   leave
+# CHECK-NEXT:  1      1     0.25                  U     salc
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -     275.50 275.50 275.50 275.50  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     aaa
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     aad
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     aad	$7
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     aam
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     aam	$7
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     aas
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     bound	%bx, (%eax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     bound	%ebx, (%eax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     daa
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     das
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     into
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leave
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     salc
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-x86_64.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-x86_64.s
new file mode 100644
index 0000000000000..bc06a60e1c3d8
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-x86_64.s
@@ -0,0 +1,2894 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+adcb $0, %al
+adcb $0, %dil
+adcb $0, (%rax)
+lock adcb $0, (%rax)
+adcb $7, %al
+adcb $7, %dil
+adcb $7, (%rax)
+lock adcb $7, (%rax)
+adcb %sil, %dil
+adcb %sil, (%rax)
+lock adcb %sil, (%rax)
+adcb (%rax), %dil
+
+adcw $0, %ax
+adcw $0, %di
+adcw $0, (%rax)
+lock adcw $0, (%rax)
+adcw $511, %ax
+adcw $511, %di
+adcw $511, (%rax)
+lock adcw $511, (%rax)
+adcw $7, %di
+adcw $7, (%rax)
+lock adcw $7, (%rax)
+adcw %si, %di
+adcw %si, (%rax)
+lock adcw %si, (%rax)
+adcw (%rax), %di
+
+adcl $0, %eax
+adcl $0, %edi
+adcl $0, (%rax)
+lock adcl $0, (%rax)
+adcl $665536, %eax
+adcl $665536, %edi
+adcl $665536, (%rax)
+lock adcl $665536, (%rax)
+adcl $7, %edi
+adcl $7, (%rax)
+lock adcl $7, (%rax)
+adcl %esi, %edi
+adcl %esi, (%rax)
+lock adcl %esi, (%rax)
+adcl (%rax), %edi
+
+adcq $0, %rax
+adcq $0, %rdi
+adcq $0, (%rax)
+lock adcq $0, (%rax)
+adcq $665536, %rax
+adcq $665536, %rdi
+adcq $665536, (%rax)
+lock adcq $665536, (%rax)
+adcq $7, %rdi
+adcq $7, (%rax)
+lock adcq $7, (%rax)
+adcq %rsi, %rdi
+adcq %rsi, (%rax)
+lock adcq %rsi, (%rax)
+adcq (%rax), %rdi
+
+addb $7, %al
+addb $7, %dil
+addb $7, (%rax)
+lock addb $7, (%rax)
+addb %sil, %dil
+addb %sil, (%rax)
+lock addb %sil, (%rax)
+addb (%rax), %dil
+
+addw $511, %ax
+addw $511, %di
+addw $511, (%rax)
+lock addw $511, (%rax)
+addw $7, %di
+addw $7, (%rax)
+lock addw $7, (%rax)
+addw %si, %di
+addw %si, (%rax)
+lock addw %si, (%rax)
+addw (%rax), %di
+
+addl $665536, %eax
+addl $665536, %edi
+addl $665536, (%rax)
+lock addl $665536, (%rax)
+addl $7, %edi
+addl $7, (%rax)
+lock addl $7, (%rax)
+addl %esi, %edi
+addl %esi, (%rax)
+lock addl %esi, (%rax)
+addl (%rax), %edi
+
+addq $665536, %rax
+addq $665536, %rdi
+addq $665536, (%rax)
+lock addq $665536, (%rax)
+addq $7, %rdi
+addq $7, (%rax)
+lock addq $7, (%rax)
+addq %rsi, %rdi
+addq %rsi, (%rax)
+lock addq %rsi, (%rax)
+addq (%rax), %rdi
+
+andb $7, %al
+andb $7, %dil
+andb $7, (%rax)
+lock andb $7, (%rax)
+andb %sil, %dil
+andb %sil, (%rax)
+lock andb %sil, (%rax)
+andb (%rax), %dil
+
+andw $511, %ax
+andw $511, %di
+andw $511, (%rax)
+lock andw $511, (%rax)
+andw $7, %di
+andw $7, (%rax)
+lock andw $7, (%rax)
+andw %si, %di
+andw %si, (%rax)
+lock andw %si, (%rax)
+andw (%rax), %di
+
+andl $665536, %eax
+andl $665536, %edi
+andl $665536, (%rax)
+lock andl $665536, (%rax)
+andl $7, %edi
+andl $7, (%rax)
+lock andl $7, (%rax)
+andl %esi, %edi
+andl %esi, (%rax)
+lock andl %esi, (%rax)
+andl (%rax), %edi
+
+andq $665536, %rax
+andq $665536, %rdi
+andq $665536, (%rax)
+lock andq $665536, (%rax)
+andq $7, %rdi
+andq $7, (%rax)
+lock andq $7, (%rax)
+andq %rsi, %rdi
+andq %rsi, (%rax)
+lock andq %rsi, (%rax)
+andq (%rax), %rdi
+
+bsfw %si, %di
+bsrw %si, %di
+bsfw (%rax), %di
+bsrw (%rax), %di
+
+bsfl %esi, %edi
+bsrl %esi, %edi
+bsfl (%rax), %edi
+bsrl (%rax), %edi
+
+bsfq %rsi, %rdi
+bsrq %rsi, %rdi
+bsfq (%rax), %rdi
+bsrq (%rax), %rdi
+
+bswap %eax
+bswap %rax
+
+btw  %si, %di
+btcw %si, %di
+btrw %si, %di
+btsw %si, %di
+btw  %si, (%rax)
+btcw %si, (%rax)
+btrw %si, (%rax)
+btsw %si, (%rax)
+lock btcw %si, (%rax)
+lock btrw %si, (%rax)
+lock btsw %si, (%rax)
+btw  $7, %di
+btcw $7, %di
+btrw $7, %di
+btsw $7, %di
+btw  $7, (%rax)
+btcw $7, (%rax)
+btrw $7, (%rax)
+btsw $7, (%rax)
+lock btcw $7, (%rax)
+lock btrw $7, (%rax)
+lock btsw $7, (%rax)
+
+btl  %esi, %edi
+btcl %esi, %edi
+btrl %esi, %edi
+btsl %esi, %edi
+btl  %esi, (%rax)
+btcl %esi, (%rax)
+btrl %esi, (%rax)
+btsl %esi, (%rax)
+lock btcl %esi, (%rax)
+lock btrl %esi, (%rax)
+lock btsl %esi, (%rax)
+btl  $7, %edi
+btcl $7, %edi
+btrl $7, %edi
+btsl $7, %edi
+btl  $7, (%rax)
+btcl $7, (%rax)
+btrl $7, (%rax)
+btsl $7, (%rax)
+lock btcl $7, (%rax)
+lock btrl $7, (%rax)
+lock btsl $7, (%rax)
+
+btq  %rsi, %rdi
+btcq %rsi, %rdi
+btrq %rsi, %rdi
+btsq %rsi, %rdi
+btq  %rsi, (%rax)
+btcq %rsi, (%rax)
+btrq %rsi, (%rax)
+btsq %rsi, (%rax)
+lock btcq %rsi, (%rax)
+lock btrq %rsi, (%rax)
+lock btsq %rsi, (%rax)
+btq  $7, %rdi
+btcq $7, %rdi
+btrq $7, %rdi
+btsq $7, %rdi
+btq  $7, (%rax)
+btcq $7, (%rax)
+btrq $7, (%rax)
+btsq $7, (%rax)
+lock btcq $7, (%rax)
+lock btrq $7, (%rax)
+lock btsq $7, (%rax)
+
+cbw
+cwde
+cdqe
+cwd
+cdq
+cqo
+
+clc
+cld
+cmc
+
+cmpb $7, %al
+cmpb $7, %dil
+cmpb $7, (%rax)
+cmpb %sil, %dil
+cmpb %sil, (%rax)
+cmpb (%rax), %dil
+
+cmpw $511, %ax
+cmpw $511, %di
+cmpw $511, (%rax)
+cmpw $7, %di
+cmpw $7, (%rax)
+cmpw %si, %di
+cmpw %si, (%rax)
+cmpw (%rax), %di
+
+cmpl $665536, %eax
+cmpl $665536, %edi
+cmpl $665536, (%rax)
+cmpl $7, %edi
+cmpl $7, (%rax)
+cmpl %esi, %edi
+cmpl %esi, (%rax)
+cmpl (%rax), %edi
+
+cmpq $665536, %rax
+cmpq $665536, %rdi
+cmpq $665536, (%rax)
+cmpq $7, %rdi
+cmpq $7, (%rax)
+cmpq %rsi, %rdi
+cmpq %rsi, (%rax)
+cmpq (%rax), %rdi
+
+cmpsb
+cmpsw
+cmpsl
+cmpsq
+
+cmpxchgb %cl, %bl
+cmpxchgb %cl, (%rbx)
+lock cmpxchgb %cl, (%rbx)
+
+cmpxchgw %cx, %bx
+cmpxchgw %cx, (%rbx)
+lock cmpxchgw %cx, (%rbx)
+
+cmpxchgl %ecx, %ebx
+cmpxchgl %ecx, (%rbx)
+lock cmpxchgl %ecx, (%rbx)
+
+cmpxchgq %rcx, %rbx
+cmpxchgq %rcx, (%rbx)
+lock cmpxchgq %rcx, (%rbx)
+
+cpuid
+
+decb %dil
+decb (%rax)
+lock decb (%rax)
+decw %di
+decw (%rax)
+lock decw (%rax)
+decl %edi
+decl (%rax)
+lock decl (%rax)
+decq %rdi
+decq (%rax)
+lock decq (%rax)
+
+divb %dil
+divb (%rax)
+divw %si
+divw (%rax)
+divl %edx
+divl (%rax)
+divq %rcx
+divq (%rax)
+
+enter $7, $4095
+
+idivb %dil
+idivb (%rax)
+idivw %si
+idivw (%rax)
+idivl %edx
+idivl (%rax)
+idivq %rcx
+idivq (%rax)
+
+imulb %dil
+imulb (%rax)
+
+imulw %di
+imulw (%rax)
+imulw %si, %di
+imulw (%rax), %di
+imulw $511, %si, %di
+imulw $511, (%rax), %di
+imulw $7, %si, %di
+imulw $7, (%rax), %di
+
+imull %edi
+imull (%rax)
+imull %esi, %edi
+imull (%rax), %edi
+imull $665536, %esi, %edi
+imull $665536, (%rax), %edi
+imull $7, %esi, %edi
+imull $7, (%rax), %edi
+
+imulq %rdi
+imulq (%rax)
+imulq %rsi, %rdi
+imulq (%rax), %rdi
+imulq $665536, %rsi, %rdi
+imulq $665536, (%rax), %rdi
+imulq $7, %rsi, %rdi
+imulq $7, (%rax), %rdi
+
+inb $7,  %al
+inb %dx, %al
+inw $7,  %ax
+inw %dx, %ax
+inl $7,  %eax
+inl %dx, %eax
+
+incb %dil
+incb (%rax)
+lock incb (%rax)
+incw %di
+incw (%rax)
+lock incw (%rax)
+incl %edi
+incl (%rax)
+lock incl (%rax)
+incq %rdi
+incq (%rax)
+lock incq (%rax)
+
+insb
+insw
+insl
+
+int $7
+
+invlpg  (%rax)
+invlpga %rax, %ecx
+
+lahf
+
+leave
+
+lodsb
+lodsw
+lodsl
+lodsq
+
+loop 0
+loope 0
+loopne 0
+
+movsb
+movsw
+movsl
+movsq
+
+movsbw %al, %di
+movzbw %al, %di
+movsbw (%rax), %di
+movzbw (%rax), %di
+movsbl %al, %edi
+movzbl %al, %edi
+movsbl (%rax), %edi
+movzbl (%rax), %edi
+movsbq %al, %rdi
+movzbq %al, %rdi
+movsbq (%rax), %rdi
+movzbq (%rax), %rdi
+
+movswl %ax, %edi
+movzwl %ax, %edi
+movswl (%rax), %edi
+movzwl (%rax), %edi
+movswq %ax, %rdi
+movzwq %ax, %rdi
+movswq (%rax), %rdi
+movzwq (%rax), %rdi
+
+movslq %eax, %rdi
+movslq (%rax), %rdi
+
+mulb %dil
+mulb (%rax)
+mulw %si
+mulw (%rax)
+mull %edx
+mull (%rax)
+mulq %rcx
+mulq (%rax)
+
+negb %dil
+negb (%r8)
+lock negb (%r8)
+negw %si
+negw (%r9)
+lock negw (%r9)
+negl %edx
+negl (%rax)
+lock negl (%rax)
+negq %rcx
+negq (%r10)
+lock negq (%r10)
+
+nop
+nopw %di
+nopw (%rcx)
+nopl %esi
+nopl (%r8)
+nopq %rdx
+nopq (%r9)
+
+notb %dil
+notb (%r8)
+lock notb (%r8)
+notw %si
+notw (%r9)
+lock notw (%r9)
+notl %edx
+notl (%rax)
+lock notl (%rax)
+notq %rcx
+notq (%r10)
+lock notq (%r10)
+
+orb $7, %al
+orb $7, %dil
+orb $7, (%rax)
+lock orb $7, (%rax)
+orb %sil, %dil
+orb %sil, (%rax)
+lock orb %sil, (%rax)
+orb (%rax), %dil
+
+orw $511, %ax
+orw $511, %di
+orw $511, (%rax)
+lock orw $511, (%rax)
+orw $7, %di
+orw $7, (%rax)
+lock orw $7, (%rax)
+orw %si, %di
+orw %si, (%rax)
+lock orw %si, (%rax)
+orw (%rax), %di
+
+orl $665536, %eax
+orl $665536, %edi
+orl $665536, (%rax)
+lock orl $665536, (%rax)
+orl $7, %edi
+orl $7, (%rax)
+lock orl $7, (%rax)
+orl %esi, %edi
+orl %esi, (%rax)
+lock orl %esi, (%rax)
+orl (%rax), %edi
+
+orq $665536, %rax
+orq $665536, %rdi
+orq $665536, (%rax)
+lock orq $665536, (%rax)
+orq $7, %rdi
+orq $7, (%rax)
+lock orq $7, (%rax)
+orq %rsi, %rdi
+orq %rsi, (%rax)
+lock orq %rsi, (%rax)
+orq (%rax), %rdi
+
+outb %al,  $7
+outb %al,  %dx
+outw %ax,  $7
+outw %ax,  %dx
+outl %eax, $7
+outl %eax, %dx
+
+outsb
+outsw
+outsl
+
+pause
+
+rclb %dil
+rcrb %dil
+rclb (%rax)
+rcrb (%rax)
+rclb $7, %dil
+rcrb $7, %dil
+rclb $7, (%rax)
+rcrb $7, (%rax)
+rclb %cl, %dil
+rcrb %cl, %dil
+rclb %cl, (%rax)
+rcrb %cl, (%rax)
+
+rclw %di
+rcrw %di
+rclw (%rax)
+rcrw (%rax)
+rclw $7, %di
+rcrw $7, %di
+rclw $7, (%rax)
+rcrw $7, (%rax)
+rclw %cl, %di
+rcrw %cl, %di
+rclw %cl, (%rax)
+rcrw %cl, (%rax)
+
+rcll %edi
+rcrl %edi
+rcll (%rax)
+rcrl (%rax)
+rcll $7, %edi
+rcrl $7, %edi
+rcll $7, (%rax)
+rcrl $7, (%rax)
+rcll %cl, %edi
+rcrl %cl, %edi
+rcll %cl, (%rax)
+rcrl %cl, (%rax)
+
+rclq %rdi
+rcrq %rdi
+rclq (%rax)
+rcrq (%rax)
+rclq $7, %rdi
+rcrq $7, %rdi
+rclq $7, (%rax)
+rcrq $7, (%rax)
+rclq %cl, %rdi
+rcrq %cl, %rdi
+rclq %cl, (%rax)
+rcrq %cl, (%rax)
+
+rdmsr
+rdpmc
+rdtsc
+rdtscp
+
+rolb %dil
+rorb %dil
+rolb (%rax)
+rorb (%rax)
+rolb $7, %dil
+rorb $7, %dil
+rolb $7, (%rax)
+rorb $7, (%rax)
+rolb %cl, %dil
+rorb %cl, %dil
+rolb %cl, (%rax)
+rorb %cl, (%rax)
+
+rolw %di
+rorw %di
+rolw (%rax)
+rorw (%rax)
+rolw $7, %di
+rorw $7, %di
+rolw $7, (%rax)
+rorw $7, (%rax)
+rolw %cl, %di
+rorw %cl, %di
+rolw %cl, (%rax)
+rorw %cl, (%rax)
+
+roll %edi
+rorl %edi
+roll (%rax)
+rorl (%rax)
+roll $7, %edi
+rorl $7, %edi
+roll $7, (%rax)
+rorl $7, (%rax)
+roll %cl, %edi
+rorl %cl, %edi
+roll %cl, (%rax)
+rorl %cl, (%rax)
+
+rolq %rdi
+rorq %rdi
+rolq (%rax)
+rorq (%rax)
+rolq $7, %rdi
+rorq $7, %rdi
+rolq $7, (%rax)
+rorq $7, (%rax)
+rolq %cl, %rdi
+rorq %cl, %rdi
+rolq %cl, (%rax)
+rorq %cl, (%rax)
+
+sahf
+
+sarb %dil
+shlb %dil
+shrb %dil
+sarb (%rax)
+shlb (%rax)
+shrb (%rax)
+sarb $7, %dil
+shlb $7, %dil
+shrb $7, %dil
+sarb $7, (%rax)
+shlb $7, (%rax)
+shrb $7, (%rax)
+sarb %cl, %dil
+shlb %cl, %dil
+shrb %cl, %dil
+sarb %cl, (%rax)
+shlb %cl, (%rax)
+shrb %cl, (%rax)
+
+sarw %di
+shlw %di
+shrw %di
+sarw (%rax)
+shlw (%rax)
+shrw (%rax)
+sarw $7, %di
+shlw $7, %di
+shrw $7, %di
+sarw $7, (%rax)
+shlw $7, (%rax)
+shrw $7, (%rax)
+sarw %cl, %di
+shlw %cl, %di
+shrw %cl, %di
+sarw %cl, (%rax)
+shlw %cl, (%rax)
+shrw %cl, (%rax)
+
+sarl %edi
+shll %edi
+shrl %edi
+sarl (%rax)
+shll (%rax)
+shrl (%rax)
+sarl $7, %edi
+shll $7, %edi
+shrl $7, %edi
+sarl $7, (%rax)
+shll $7, (%rax)
+shrl $7, (%rax)
+sarl %cl, %edi
+shll %cl, %edi
+shrl %cl, %edi
+sarl %cl, (%rax)
+shll %cl, (%rax)
+shrl %cl, (%rax)
+
+sarq %rdi
+shlq %rdi
+shrq %rdi
+sarq (%rax)
+shlq (%rax)
+shrq (%rax)
+sarq $7, %rdi
+shlq $7, %rdi
+shrq $7, %rdi
+sarq $7, (%rax)
+shlq $7, (%rax)
+shrq $7, (%rax)
+sarq %cl, %rdi
+shlq %cl, %rdi
+shrq %cl, %rdi
+sarq %cl, (%rax)
+shlq %cl, (%rax)
+shrq %cl, (%rax)
+
+sbbb $0, %al
+sbbb $0, %dil
+sbbb $0, (%rax)
+lock sbbb $0, (%rax)
+sbbb $7, %al
+sbbb $7, %dil
+sbbb $7, (%rax)
+lock sbbb $7, (%rax)
+sbbb %sil, %dil
+sbbb %sil, (%rax)
+lock sbbb %sil, (%rax)
+sbbb (%rax), %dil
+
+sbbw $0, %ax
+sbbw $0, %di
+sbbw $0, (%rax)
+lock sbbw $0, (%rax)
+sbbw $511, %ax
+sbbw $511, %di
+sbbw $511, (%rax)
+lock sbbw $511, (%rax)
+sbbw $7, %di
+sbbw $7, (%rax)
+lock sbbw $7, (%rax)
+sbbw %si, %di
+sbbw %si, (%rax)
+lock sbbw %si, (%rax)
+sbbw (%rax), %di
+
+sbbl $0, %eax
+sbbl $0, %edi
+sbbl $0, (%rax)
+lock sbbl $0, (%rax)
+sbbl $665536, %eax
+sbbl $665536, %edi
+sbbl $665536, (%rax)
+lock sbbl $665536, (%rax)
+sbbl $7, %edi
+sbbl $7, (%rax)
+lock sbbl $7, (%rax)
+sbbl %esi, %edi
+sbbl %esi, (%rax)
+lock sbbl %esi, (%rax)
+sbbl (%rax), %edi
+
+sbbq $0, %rax
+sbbq $0, %rdi
+sbbq $0, (%rax)
+lock sbbq $0, (%rax)
+sbbq $665536, %rax
+sbbq $665536, %rdi
+sbbq $665536, (%rax)
+lock sbbq $665536, (%rax)
+sbbq $7, %rdi
+sbbq $7, (%rax)
+lock sbbq $7, (%rax)
+sbbq %rsi, %rdi
+sbbq %rsi, (%rax)
+lock sbbq %rsi, (%rax)
+sbbq (%rax), %rdi
+
+scasb
+scasw
+scasl
+scasq
+
+seto  %al
+seto  (%rax)
+setno %al
+setno (%rax)
+setb  %al
+setb  (%rax)
+setnb %al
+setnb (%rax)
+setz  %al
+setz  (%rax)
+setnz %al
+setnz (%rax)
+seta  %al
+seta  (%rax)
+setna %al
+setna (%rax)
+sets  %al
+sets  (%rax)
+setns %al
+setns (%rax)
+setp  %al
+setp  (%rax)
+setnp %al
+setnp (%rax)
+setl  %al
+setl  (%rax)
+setnl %al
+setnl (%rax)
+setg  %al
+setg  (%rax)
+setng %al
+setng (%rax)
+
+shldw %cl, %si, %di
+shrdw %cl, %si, %di
+shldw %cl, %si, (%rax)
+shrdw %cl, %si, (%rax)
+shldw $7, %si, %di
+shrdw $7, %si, %di
+shldw $7, %si, (%rax)
+shrdw $7, %si, (%rax)
+
+shldl %cl, %esi, %edi
+shrdl %cl, %esi, %edi
+shldl %cl, %esi, (%rax)
+shrdl %cl, %esi, (%rax)
+shldl $7, %esi, %edi
+shrdl $7, %esi, %edi
+shldl $7, %esi, (%rax)
+shrdl $7, %esi, (%rax)
+
+shldq %cl, %rsi, %rdi
+shrdq %cl, %rsi, %rdi
+shldq %cl, %rsi, (%rax)
+shrdq %cl, %rsi, (%rax)
+shldq $7, %rsi, %rdi
+shrdq $7, %rsi, %rdi
+shldq $7, %rsi, (%rax)
+shrdq $7, %rsi, (%rax)
+
+stc
+std
+
+stosb
+stosw
+stosl
+stosq
+
+subb $7, %al
+subb $7, %dil
+subb $7, (%rax)
+lock subb $7, (%rax)
+subb %sil, %dil
+subb %sil, (%rax)
+lock subb %sil, (%rax)
+subb (%rax), %dil
+
+subw $511, %ax
+subw $511, %di
+subw $511, (%rax)
+lock subw $511, (%rax)
+subw $7, %di
+subw $7, (%rax)
+lock subw $7, (%rax)
+subw %si, %di
+subw %si, (%rax)
+lock subw %si, (%rax)
+subw (%rax), %di
+
+subl $665536, %eax
+subl $665536, %edi
+subl $665536, (%rax)
+lock subl $665536, (%rax)
+subl $7, %edi
+subl $7, (%rax)
+lock subl $7, (%rax)
+subl %esi, %edi
+subl %esi, (%rax)
+lock subl %esi, (%rax)
+subl (%rax), %edi
+
+subq $665536, %rax
+subq $665536, %rdi
+subq $665536, (%rax)
+lock subq $665536, (%rax)
+subq $7, %rdi
+subq $7, (%rax)
+lock subq $7, (%rax)
+subq %rsi, %rdi
+subq %rsi, (%rax)
+lock subq %rsi, (%rax)
+subq (%rax), %rdi
+
+testb $7, %al
+testb $7, %dil
+testb $7, (%rax)
+testb %sil, %dil
+testb %sil, (%rax)
+
+testw $511, %ax
+testw $511, %di
+testw $511, (%rax)
+testw $7, %di
+testw $7, (%rax)
+testw %si, %di
+testw %si, (%rax)
+
+testl $665536, %eax
+testl $665536, %edi
+testl $665536, (%rax)
+testl $7, %edi
+testl $7, (%rax)
+testl %esi, %edi
+testl %esi, (%rax)
+
+testq $665536, %rax
+testq $665536, %rdi
+testq $665536, (%rax)
+testq $7, %rdi
+testq $7, (%rax)
+testq %rsi, %rdi
+testq %rsi, (%rax)
+
+ud2
+
+wrmsr
+
+xaddb %bl, %cl
+xaddb %bl, (%rcx)
+lock xaddb %bl, (%rcx)
+
+xaddw %bx, %cx
+xaddw %ax, (%rbx)
+lock xaddw %ax, (%rbx)
+
+xaddl %ebx, %ecx
+xaddl %eax, (%rbx)
+lock xaddl %eax, (%rbx)
+
+xaddq %rbx, %rcx
+xaddq %rax, (%rbx)
+lock xaddq %rax, (%rbx)
+
+xchgb %bl, %cl
+xchgb %bl, (%rbx)
+lock xchgb %bl, (%rbx)
+
+xchgw %ax, %bx
+xchgw %bx, %cx
+xchgw %ax, (%rbx)
+lock xchgw %ax, (%rbx)
+
+xchgl %eax, %ebx
+xchgl %ebx, %ecx
+xchgl %eax, (%rbx)
+lock xchgl %eax, (%rbx)
+
+xchgq %rax, %rbx
+xchgq %rbx, %rcx
+xchgq %rax, (%rbx)
+lock xchgq %rax, (%rbx)
+
+xlatb
+
+xorb $7, %al
+xorb $7, %dil
+xorb $7, (%rax)
+lock xorb $7, (%rax)
+xorb %sil, %dil
+xorb %sil, (%rax)
+lock xorb %sil, (%rax)
+xorb (%rax), %dil
+
+xorw $511, %ax
+xorw $511, %di
+xorw $511, (%rax)
+lock xorw $511, (%rax)
+xorw $7, %di
+xorw $7, (%rax)
+lock xorw $7, (%rax)
+xorw %si, %di
+xorw %si, (%rax)
+lock xorw %si, (%rax)
+xorw (%rax), %di
+
+xorl $665536, %eax
+xorl $665536, %edi
+xorl $665536, (%rax)
+lock xorl $665536, (%rax)
+xorl $7, %edi
+xorl $7, (%rax)
+lock xorl $7, (%rax)
+xorl %esi, %edi
+xorl %esi, (%rax)
+lock xorl %esi, (%rax)
+xorl (%rax), %edi
+
+xorq $665536, %rax
+xorq $665536, %rdi
+xorq $665536, (%rax)
+lock xorq $665536, (%rax)
+xorq $7, %rdi
+xorq $7, (%rax)
+lock xorq $7, (%rax)
+xorq %rsi, %rdi
+xorq %rsi, (%rax)
+lock xorq %rsi, (%rax)
+xorq (%rax), %rdi
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     1.00                        adcb	$0, %al
+# CHECK-NEXT:  1      1     1.00                        adcb	$0, %dil
+# CHECK-NEXT:  1      6     1.00    *      *            adcb	$0, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		adcb	$0, (%rax)
+# CHECK-NEXT:  1      1     1.00                        adcb	$7, %al
+# CHECK-NEXT:  1      1     1.00                        adcb	$7, %dil
+# CHECK-NEXT:  1      6     1.00    *      *            adcb	$7, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		adcb	$7, (%rax)
+# CHECK-NEXT:  1      1     1.00                        adcb	%sil, %dil
+# CHECK-NEXT:  1      1     1.75    *      *            adcb	%sil, (%rax)
+# CHECK-NEXT:  1      1     1.75    *      *            lock		adcb	%sil, (%rax)
+# CHECK-NEXT:  1      5     1.00    *                   adcb	(%rax), %dil
+# CHECK-NEXT:  1      1     1.00                        adcw	$0, %ax
+# CHECK-NEXT:  1      1     1.00                        adcw	$0, %di
+# CHECK-NEXT:  1      6     1.00    *      *            adcw	$0, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		adcw	$0, (%rax)
+# CHECK-NEXT:  1      1     1.00                        adcw	$511, %ax
+# CHECK-NEXT:  1      1     1.00                        adcw	$511, %di
+# CHECK-NEXT:  1      6     1.00    *      *            adcw	$511, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		adcw	$511, (%rax)
+# CHECK-NEXT:  1      1     1.00                        adcw	$7, %di
+# CHECK-NEXT:  1      6     1.00    *      *            adcw	$7, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		adcw	$7, (%rax)
+# CHECK-NEXT:  1      1     1.00                        adcw	%si, %di
+# CHECK-NEXT:  1      6     1.00    *      *            adcw	%si, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		adcw	%si, (%rax)
+# CHECK-NEXT:  1      5     1.00    *                   adcw	(%rax), %di
+# CHECK-NEXT:  1      1     1.00                        adcl	$0, %eax
+# CHECK-NEXT:  1      1     1.00                        adcl	$0, %edi
+# CHECK-NEXT:  1      6     1.00    *      *            adcl	$0, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		adcl	$0, (%rax)
+# CHECK-NEXT:  1      1     1.00                        adcl	$665536, %eax
+# CHECK-NEXT:  1      1     1.00                        adcl	$665536, %edi
+# CHECK-NEXT:  1      6     1.00    *      *            adcl	$665536, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		adcl	$665536, (%rax)
+# CHECK-NEXT:  1      1     1.00                        adcl	$7, %edi
+# CHECK-NEXT:  1      6     1.00    *      *            adcl	$7, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		adcl	$7, (%rax)
+# CHECK-NEXT:  1      1     1.00                        adcl	%esi, %edi
+# CHECK-NEXT:  1      6     1.00    *      *            adcl	%esi, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		adcl	%esi, (%rax)
+# CHECK-NEXT:  1      5     1.00    *                   adcl	(%rax), %edi
+# CHECK-NEXT:  1      1     1.00                        adcq	$0, %rax
+# CHECK-NEXT:  1      1     1.00                        adcq	$0, %rdi
+# CHECK-NEXT:  1      6     1.00    *      *            adcq	$0, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		adcq	$0, (%rax)
+# CHECK-NEXT:  1      1     1.00                        adcq	$665536, %rax
+# CHECK-NEXT:  1      1     1.00                        adcq	$665536, %rdi
+# CHECK-NEXT:  1      6     1.00    *      *            adcq	$665536, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		adcq	$665536, (%rax)
+# CHECK-NEXT:  1      1     1.00                        adcq	$7, %rdi
+# CHECK-NEXT:  1      6     1.00    *      *            adcq	$7, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		adcq	$7, (%rax)
+# CHECK-NEXT:  1      1     1.00                        adcq	%rsi, %rdi
+# CHECK-NEXT:  1      6     1.00    *      *            adcq	%rsi, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		adcq	%rsi, (%rax)
+# CHECK-NEXT:  1      5     1.00    *                   adcq	(%rax), %rdi
+# CHECK-NEXT:  1      1     1.00                        addb	$7, %al
+# CHECK-NEXT:  1      1     0.25                        addb	$7, %dil
+# CHECK-NEXT:  1      6     0.67    *      *            addb	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		addb	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        addb	%sil, %dil
+# CHECK-NEXT:  1      6     0.67    *      *            addb	%sil, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		addb	%sil, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   addb	(%rax), %dil
+# CHECK-NEXT:  1      1     1.00                        addw	$511, %ax
+# CHECK-NEXT:  1      1     0.25                        addw	$511, %di
+# CHECK-NEXT:  1      6     0.67    *      *            addw	$511, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		addw	$511, (%rax)
+# CHECK-NEXT:  1      1     0.25                        addw	$7, %di
+# CHECK-NEXT:  1      6     0.67    *      *            addw	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		addw	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        addw	%si, %di
+# CHECK-NEXT:  1      6     0.67    *      *            addw	%si, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		addw	%si, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   addw	(%rax), %di
+# CHECK-NEXT:  1      1     1.00                        addl	$665536, %eax
+# CHECK-NEXT:  1      1     0.25                        addl	$665536, %edi
+# CHECK-NEXT:  1      6     0.67    *      *            addl	$665536, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		addl	$665536, (%rax)
+# CHECK-NEXT:  1      1     0.25                        addl	$7, %edi
+# CHECK-NEXT:  1      6     0.67    *      *            addl	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		addl	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        addl	%esi, %edi
+# CHECK-NEXT:  1      6     0.67    *      *            addl	%esi, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		addl	%esi, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   addl	(%rax), %edi
+# CHECK-NEXT:  1      1     1.00                        addq	$665536, %rax
+# CHECK-NEXT:  1      1     0.25                        addq	$665536, %rdi
+# CHECK-NEXT:  1      6     0.67    *      *            addq	$665536, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		addq	$665536, (%rax)
+# CHECK-NEXT:  1      1     0.25                        addq	$7, %rdi
+# CHECK-NEXT:  1      6     0.67    *      *            addq	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		addq	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        addq	%rsi, %rdi
+# CHECK-NEXT:  1      6     0.67    *      *            addq	%rsi, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		addq	%rsi, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   addq	(%rax), %rdi
+# CHECK-NEXT:  1      1     1.00                        andb	$7, %al
+# CHECK-NEXT:  1      1     0.25                        andb	$7, %dil
+# CHECK-NEXT:  1      6     0.67    *      *            andb	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		andb	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        andb	%sil, %dil
+# CHECK-NEXT:  1      6     0.67    *      *            andb	%sil, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		andb	%sil, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   andb	(%rax), %dil
+# CHECK-NEXT:  1      1     1.00                        andw	$511, %ax
+# CHECK-NEXT:  1      1     0.25                        andw	$511, %di
+# CHECK-NEXT:  1      6     0.67    *      *            andw	$511, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		andw	$511, (%rax)
+# CHECK-NEXT:  1      1     0.25                        andw	$7, %di
+# CHECK-NEXT:  1      6     0.67    *      *            andw	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		andw	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        andw	%si, %di
+# CHECK-NEXT:  1      6     0.67    *      *            andw	%si, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		andw	%si, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   andw	(%rax), %di
+# CHECK-NEXT:  1      1     1.00                        andl	$665536, %eax
+# CHECK-NEXT:  1      1     0.25                        andl	$665536, %edi
+# CHECK-NEXT:  1      6     0.67    *      *            andl	$665536, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		andl	$665536, (%rax)
+# CHECK-NEXT:  1      1     0.25                        andl	$7, %edi
+# CHECK-NEXT:  1      6     0.67    *      *            andl	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		andl	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        andl	%esi, %edi
+# CHECK-NEXT:  1      6     0.67    *      *            andl	%esi, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		andl	%esi, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   andl	(%rax), %edi
+# CHECK-NEXT:  1      1     1.00                        andq	$665536, %rax
+# CHECK-NEXT:  1      1     0.25                        andq	$665536, %rdi
+# CHECK-NEXT:  1      6     0.67    *      *            andq	$665536, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		andq	$665536, (%rax)
+# CHECK-NEXT:  1      1     0.25                        andq	$7, %rdi
+# CHECK-NEXT:  1      6     0.67    *      *            andq	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		andq	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        andq	%rsi, %rdi
+# CHECK-NEXT:  1      6     0.67    *      *            andq	%rsi, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		andq	%rsi, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   andq	(%rax), %rdi
+# CHECK-NEXT:  6      1     1.00                        bsfw	%si, %di
+# CHECK-NEXT:  6      1     1.00                        bsrw	%si, %di
+# CHECK-NEXT:  7      5     1.00    *                   bsfw	(%rax), %di
+# CHECK-NEXT:  7      5     1.00    *                   bsrw	(%rax), %di
+# CHECK-NEXT:  6      1     1.00                        bsfl	%esi, %edi
+# CHECK-NEXT:  6      1     1.00                        bsrl	%esi, %edi
+# CHECK-NEXT:  7      5     1.00    *                   bsfl	(%rax), %edi
+# CHECK-NEXT:  7      5     1.00    *                   bsrl	(%rax), %edi
+# CHECK-NEXT:  6      1     1.00                        bsfq	%rsi, %rdi
+# CHECK-NEXT:  6      1     1.00                        bsrq	%rsi, %rdi
+# CHECK-NEXT:  7      5     1.00    *                   bsfq	(%rax), %rdi
+# CHECK-NEXT:  7      5     1.00    *                   bsrq	(%rax), %rdi
+# CHECK-NEXT:  1      1     0.25                        bswapl	%eax
+# CHECK-NEXT:  1      1     0.25                        bswapq	%rax
+# CHECK-NEXT:  1      1     0.50                        btw	%si, %di
+# CHECK-NEXT:  2      2     1.00                        btcw	%si, %di
+# CHECK-NEXT:  2      2     1.00                        btrw	%si, %di
+# CHECK-NEXT:  2      2     1.00                        btsw	%si, %di
+# CHECK-NEXT:  7      5     0.50    *                   btw	%si, (%rax)
+# CHECK-NEXT:  9      7     0.67    *      *            btcw	%si, (%rax)
+# CHECK-NEXT:  9      7     0.67    *      *            btrw	%si, (%rax)
+# CHECK-NEXT:  9      7     0.67    *      *            btsw	%si, (%rax)
+# CHECK-NEXT:  9      7     0.67    *      *            lock		btcw	%si, (%rax)
+# CHECK-NEXT:  9      7     0.67    *      *            lock		btrw	%si, (%rax)
+# CHECK-NEXT:  9      7     0.67    *      *            lock		btsw	%si, (%rax)
+# CHECK-NEXT:  1      1     0.50                        btw	$7, %di
+# CHECK-NEXT:  2      2     1.00                        btcw	$7, %di
+# CHECK-NEXT:  2      2     1.00                        btrw	$7, %di
+# CHECK-NEXT:  2      2     1.00                        btsw	$7, %di
+# CHECK-NEXT:  2      5     0.50    *                   btw	$7, (%rax)
+# CHECK-NEXT:  4      7     0.67    *      *            btcw	$7, (%rax)
+# CHECK-NEXT:  4      7     0.67    *      *            btrw	$7, (%rax)
+# CHECK-NEXT:  4      7     0.67    *      *            btsw	$7, (%rax)
+# CHECK-NEXT:  4      7     0.67    *      *            lock		btcw	$7, (%rax)
+# CHECK-NEXT:  4      7     0.67    *      *            lock		btrw	$7, (%rax)
+# CHECK-NEXT:  4      7     0.67    *      *            lock		btsw	$7, (%rax)
+# CHECK-NEXT:  1      1     0.50                        btl	%esi, %edi
+# CHECK-NEXT:  2      2     1.00                        btcl	%esi, %edi
+# CHECK-NEXT:  2      2     1.00                        btrl	%esi, %edi
+# CHECK-NEXT:  2      2     1.00                        btsl	%esi, %edi
+# CHECK-NEXT:  7      5     0.50    *                   btl	%esi, (%rax)
+# CHECK-NEXT:  9      7     0.67    *      *            btcl	%esi, (%rax)
+# CHECK-NEXT:  9      7     0.67    *      *            btrl	%esi, (%rax)
+# CHECK-NEXT:  9      7     0.67    *      *            btsl	%esi, (%rax)
+# CHECK-NEXT:  9      7     0.67    *      *            lock		btcl	%esi, (%rax)
+# CHECK-NEXT:  9      7     0.67    *      *            lock		btrl	%esi, (%rax)
+# CHECK-NEXT:  9      7     0.67    *      *            lock		btsl	%esi, (%rax)
+# CHECK-NEXT:  1      1     0.50                        btl	$7, %edi
+# CHECK-NEXT:  2      2     1.00                        btcl	$7, %edi
+# CHECK-NEXT:  2      2     1.00                        btrl	$7, %edi
+# CHECK-NEXT:  2      2     1.00                        btsl	$7, %edi
+# CHECK-NEXT:  2      5     0.50    *                   btl	$7, (%rax)
+# CHECK-NEXT:  4      7     0.67    *      *            btcl	$7, (%rax)
+# CHECK-NEXT:  4      7     0.67    *      *            btrl	$7, (%rax)
+# CHECK-NEXT:  4      7     0.67    *      *            btsl	$7, (%rax)
+# CHECK-NEXT:  4      7     0.67    *      *            lock		btcl	$7, (%rax)
+# CHECK-NEXT:  4      7     0.67    *      *            lock		btrl	$7, (%rax)
+# CHECK-NEXT:  4      7     0.67    *      *            lock		btsl	$7, (%rax)
+# CHECK-NEXT:  1      1     0.50                        btq	%rsi, %rdi
+# CHECK-NEXT:  2      2     1.00                        btcq	%rsi, %rdi
+# CHECK-NEXT:  2      2     1.00                        btrq	%rsi, %rdi
+# CHECK-NEXT:  2      2     1.00                        btsq	%rsi, %rdi
+# CHECK-NEXT:  7      5     0.50    *                   btq	%rsi, (%rax)
+# CHECK-NEXT:  9      7     0.67    *      *            btcq	%rsi, (%rax)
+# CHECK-NEXT:  9      7     0.67    *      *            btrq	%rsi, (%rax)
+# CHECK-NEXT:  9      7     0.67    *      *            btsq	%rsi, (%rax)
+# CHECK-NEXT:  9      7     0.67    *      *            lock		btcq	%rsi, (%rax)
+# CHECK-NEXT:  9      7     0.67    *      *            lock		btrq	%rsi, (%rax)
+# CHECK-NEXT:  9      7     0.67    *      *            lock		btsq	%rsi, (%rax)
+# CHECK-NEXT:  1      1     0.50                        btq	$7, %rdi
+# CHECK-NEXT:  2      2     1.00                        btcq	$7, %rdi
+# CHECK-NEXT:  2      2     1.00                        btrq	$7, %rdi
+# CHECK-NEXT:  2      2     1.00                        btsq	$7, %rdi
+# CHECK-NEXT:  2      5     0.50    *                   btq	$7, (%rax)
+# CHECK-NEXT:  4      7     0.67    *      *            btcq	$7, (%rax)
+# CHECK-NEXT:  4      7     0.67    *      *            btrq	$7, (%rax)
+# CHECK-NEXT:  4      7     0.67    *      *            btsq	$7, (%rax)
+# CHECK-NEXT:  4      7     0.67    *      *            lock		btcq	$7, (%rax)
+# CHECK-NEXT:  4      7     0.67    *      *            lock		btrq	$7, (%rax)
+# CHECK-NEXT:  4      7     0.67    *      *            lock		btsq	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        cbtw
+# CHECK-NEXT:  1      1     0.25                        cwtl
+# CHECK-NEXT:  1      1     0.25                        cltq
+# CHECK-NEXT:  1      1     0.25                        cwtd
+# CHECK-NEXT:  1      1     0.25                        cltd
+# CHECK-NEXT:  1      1     0.25                        cqto
+# CHECK-NEXT:  1      1     0.25                  U     clc
+# CHECK-NEXT:  1      1     0.25                  U     cld
+# CHECK-NEXT:  1      1     0.25                  U     cmc
+# CHECK-NEXT:  1      1     0.25                        cmpb	$7, %al
+# CHECK-NEXT:  1      1     0.25                        cmpb	$7, %dil
+# CHECK-NEXT:  1      5     0.33    *                   cmpb	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        cmpb	%sil, %dil
+# CHECK-NEXT:  1      5     0.33    *                   cmpb	%sil, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   cmpb	(%rax), %dil
+# CHECK-NEXT:  1      1     0.25                        cmpw	$511, %ax
+# CHECK-NEXT:  1      1     0.25                        cmpw	$511, %di
+# CHECK-NEXT:  1      5     0.33    *                   cmpw	$511, (%rax)
+# CHECK-NEXT:  1      1     0.25                        cmpw	$7, %di
+# CHECK-NEXT:  1      5     0.33    *                   cmpw	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        cmpw	%si, %di
+# CHECK-NEXT:  1      5     0.33    *                   cmpw	%si, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   cmpw	(%rax), %di
+# CHECK-NEXT:  1      1     0.25                        cmpl	$665536, %eax
+# CHECK-NEXT:  1      1     0.25                        cmpl	$665536, %edi
+# CHECK-NEXT:  1      5     0.33    *                   cmpl	$665536, (%rax)
+# CHECK-NEXT:  1      1     0.25                        cmpl	$7, %edi
+# CHECK-NEXT:  1      5     0.33    *                   cmpl	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        cmpl	%esi, %edi
+# CHECK-NEXT:  1      5     0.33    *                   cmpl	%esi, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   cmpl	(%rax), %edi
+# CHECK-NEXT:  1      1     0.25                        cmpq	$665536, %rax
+# CHECK-NEXT:  1      1     0.25                        cmpq	$665536, %rdi
+# CHECK-NEXT:  1      5     0.33    *                   cmpq	$665536, (%rax)
+# CHECK-NEXT:  1      1     0.25                        cmpq	$7, %rdi
+# CHECK-NEXT:  1      5     0.33    *                   cmpq	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        cmpq	%rsi, %rdi
+# CHECK-NEXT:  1      5     0.33    *                   cmpq	%rsi, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   cmpq	(%rax), %rdi
+# CHECK-NEXT:  100    100   25.00                 U     cmpsb	%es:(%rdi), (%rsi)
+# CHECK-NEXT:  100    100   25.00                 U     cmpsw	%es:(%rdi), (%rsi)
+# CHECK-NEXT:  100    100   25.00                 U     cmpsl	%es:(%rdi), (%rsi)
+# CHECK-NEXT:  100    100   25.00                 U     cmpsq	%es:(%rdi), (%rsi)
+# CHECK-NEXT:  3      3     3.00                        cmpxchgb	%cl, %bl
+# CHECK-NEXT:  5      7     3.00    *      *            cmpxchgb	%cl, (%rbx)
+# CHECK-NEXT:  5      7     3.00    *      *            lock		cmpxchgb	%cl, (%rbx)
+# CHECK-NEXT:  5      3     3.00                        cmpxchgw	%cx, %bx
+# CHECK-NEXT:  6      3     3.00    *      *            cmpxchgw	%cx, (%rbx)
+# CHECK-NEXT:  6      3     3.00    *      *            lock		cmpxchgw	%cx, (%rbx)
+# CHECK-NEXT:  5      3     3.00                        cmpxchgl	%ecx, %ebx
+# CHECK-NEXT:  6      3     3.00    *      *            cmpxchgl	%ecx, (%rbx)
+# CHECK-NEXT:  6      3     3.00    *      *            lock		cmpxchgl	%ecx, (%rbx)
+# CHECK-NEXT:  5      3     3.00                        cmpxchgq	%rcx, %rbx
+# CHECK-NEXT:  6      3     3.00    *      *            cmpxchgq	%rcx, (%rbx)
+# CHECK-NEXT:  6      3     3.00    *      *            lock		cmpxchgq	%rcx, (%rbx)
+# CHECK-NEXT:  100    100   25.00                 U     cpuid
+# CHECK-NEXT:  1      1     0.25                        decb	%dil
+# CHECK-NEXT:  1      6     0.67    *      *            decb	(%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		decb	(%rax)
+# CHECK-NEXT:  1      1     0.25                        decw	%di
+# CHECK-NEXT:  1      6     0.67    *      *            decw	(%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		decw	(%rax)
+# CHECK-NEXT:  1      1     0.25                        decl	%edi
+# CHECK-NEXT:  1      6     0.67    *      *            decl	(%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		decl	(%rax)
+# CHECK-NEXT:  1      1     0.25                        decq	%rdi
+# CHECK-NEXT:  1      6     0.67    *      *            decq	(%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		decq	(%rax)
+# CHECK-NEXT:  2      10    10.00                 U     divb	%dil
+# CHECK-NEXT:  2      14    10.00   *             U     divb	(%rax)
+# CHECK-NEXT:  2      11    11.00                 U     divw	%si
+# CHECK-NEXT:  2      15    11.00   *             U     divw	(%rax)
+# CHECK-NEXT:  2      13    13.00                 U     divl	%edx
+# CHECK-NEXT:  2      17    13.00   *             U     divl	(%rax)
+# CHECK-NEXT:  2      17    17.00                 U     divq	%rcx
+# CHECK-NEXT:  2      21    17.00   *             U     divq	(%rax)
+# CHECK-NEXT:  100    100   25.00                 U     enter	$7, $4095
+# CHECK-NEXT:  2      10    10.00                 U     idivb	%dil
+# CHECK-NEXT:  2      14    10.00   *             U     idivb	(%rax)
+# CHECK-NEXT:  2      11    11.00                 U     idivw	%si
+# CHECK-NEXT:  2      15    11.00   *             U     idivw	(%rax)
+# CHECK-NEXT:  2      13    13.00                 U     idivl	%edx
+# CHECK-NEXT:  2      17    13.00   *             U     idivl	(%rax)
+# CHECK-NEXT:  2      17    17.00                 U     idivq	%rcx
+# CHECK-NEXT:  2      21    17.00   *             U     idivq	(%rax)
+# CHECK-NEXT:  1      3     3.00                        imulb	%dil
+# CHECK-NEXT:  1      7     3.00    *                   imulb	(%rax)
+# CHECK-NEXT:  3      3     3.00                        imulw	%di
+# CHECK-NEXT:  4      7     3.00    *                   imulw	(%rax)
+# CHECK-NEXT:  1      3     1.00                        imulw	%si, %di
+# CHECK-NEXT:  1      7     1.00    *                   imulw	(%rax), %di
+# CHECK-NEXT:  2      4     4.00                        imulw	$511, %si, %di
+# CHECK-NEXT:  2      8     4.00    *                   imulw	$511, (%rax), %di
+# CHECK-NEXT:  2      4     4.00                        imulw	$7, %si, %di
+# CHECK-NEXT:  2      8     4.00    *                   imulw	$7, (%rax), %di
+# CHECK-NEXT:  2      3     3.00                        imull	%edi
+# CHECK-NEXT:  2      7     3.00    *                   imull	(%rax)
+# CHECK-NEXT:  1      3     1.00                        imull	%esi, %edi
+# CHECK-NEXT:  1      7     1.00    *                   imull	(%rax), %edi
+# CHECK-NEXT:  1      3     1.00                        imull	$665536, %esi, %edi
+# CHECK-NEXT:  1      7     1.00    *                   imull	$665536, (%rax), %edi
+# CHECK-NEXT:  1      3     1.00                        imull	$7, %esi, %edi
+# CHECK-NEXT:  1      7     1.00    *                   imull	$7, (%rax), %edi
+# CHECK-NEXT:  2      3     3.00                        imulq	%rdi
+# CHECK-NEXT:  2      7     3.00    *                   imulq	(%rax)
+# CHECK-NEXT:  1      3     1.00                        imulq	%rsi, %rdi
+# CHECK-NEXT:  1      7     1.00    *                   imulq	(%rax), %rdi
+# CHECK-NEXT:  1      3     1.00                        imulq	$665536, %rsi, %rdi
+# CHECK-NEXT:  1      7     1.00    *                   imulq	$665536, (%rax), %rdi
+# CHECK-NEXT:  1      3     1.00                        imulq	$7, %rsi, %rdi
+# CHECK-NEXT:  1      7     1.00    *                   imulq	$7, (%rax), %rdi
+# CHECK-NEXT:  100    100   25.00                 U     inb	$7, %al
+# CHECK-NEXT:  100    100   25.00                 U     inb	%dx, %al
+# CHECK-NEXT:  100    100   25.00                 U     inw	$7, %ax
+# CHECK-NEXT:  100    100   25.00                 U     inw	%dx, %ax
+# CHECK-NEXT:  100    100   25.00                 U     inl	$7, %eax
+# CHECK-NEXT:  100    100   25.00                 U     inl	%dx, %eax
+# CHECK-NEXT:  1      1     0.25                        incb	%dil
+# CHECK-NEXT:  1      6     0.67    *      *            incb	(%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		incb	(%rax)
+# CHECK-NEXT:  1      1     0.25                        incw	%di
+# CHECK-NEXT:  1      6     0.67    *      *            incw	(%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		incw	(%rax)
+# CHECK-NEXT:  1      1     0.25                        incl	%edi
+# CHECK-NEXT:  1      6     0.67    *      *            incl	(%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		incl	(%rax)
+# CHECK-NEXT:  1      1     0.25                        incq	%rdi
+# CHECK-NEXT:  1      6     0.67    *      *            incq	(%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		incq	(%rax)
+# CHECK-NEXT:  100    100   25.00                 U     insb	%dx, %es:(%rdi)
+# CHECK-NEXT:  100    100   25.00                 U     insw	%dx, %es:(%rdi)
+# CHECK-NEXT:  100    100   25.00                 U     insl	%dx, %es:(%rdi)
+# CHECK-NEXT:  100    100   25.00   *      *      U     int	$7
+# CHECK-NEXT:  100    100   25.00                 U     invlpg	(%rax)
+# CHECK-NEXT:  100    100   25.00                 U     invlpga
+# CHECK-NEXT:  1      1     1.00                        lahf
+# CHECK-NEXT:  1      1     0.25    *                   leave
+# CHECK-NEXT:  100    100   25.00                 U     lodsb	(%rsi), %al
+# CHECK-NEXT:  100    100   25.00                 U     lodsw	(%rsi), %ax
+# CHECK-NEXT:  100    100   25.00                 U     lodsl	(%rsi), %eax
+# CHECK-NEXT:  100    100   25.00                 U     lodsq	(%rsi), %rax
+# CHECK-NEXT:  1      1     0.50                  U     loop	0
+# CHECK-NEXT:  1      1     0.50                  U     loope	0
+# CHECK-NEXT:  1      1     0.50                  U     loopne	0
+# CHECK-NEXT:  100    100   25.00                 U     movsb	(%rsi), %es:(%rdi)
+# CHECK-NEXT:  100    100   25.00                 U     movsw	(%rsi), %es:(%rdi)
+# CHECK-NEXT:  100    100   25.00                 U     movsl	(%rsi), %es:(%rdi)
+# CHECK-NEXT:  100    100   25.00                 U     movsq	(%rsi), %es:(%rdi)
+# CHECK-NEXT:  1      1     1.00                        movsbw	%al, %di
+# CHECK-NEXT:  1      1     1.00                        movzbw	%al, %di
+# CHECK-NEXT:  1      5     1.00    *                   movsbw	(%rax), %di
+# CHECK-NEXT:  1      5     1.00    *                   movzbw	(%rax), %di
+# CHECK-NEXT:  1      1     0.25                        movsbl	%al, %edi
+# CHECK-NEXT:  1      1     0.25                        movzbl	%al, %edi
+# CHECK-NEXT:  1      5     0.33    *                   movsbl	(%rax), %edi
+# CHECK-NEXT:  1      5     0.33    *                   movzbl	(%rax), %edi
+# CHECK-NEXT:  1      1     0.25                        movsbq	%al, %rdi
+# CHECK-NEXT:  1      1     0.25                        movzbq	%al, %rdi
+# CHECK-NEXT:  1      5     0.33    *                   movsbq	(%rax), %rdi
+# CHECK-NEXT:  1      5     0.33    *                   movzbq	(%rax), %rdi
+# CHECK-NEXT:  1      1     0.25                        movswl	%ax, %edi
+# CHECK-NEXT:  1      1     0.25                        movzwl	%ax, %edi
+# CHECK-NEXT:  1      5     0.33    *                   movswl	(%rax), %edi
+# CHECK-NEXT:  1      5     0.33    *                   movzwl	(%rax), %edi
+# CHECK-NEXT:  1      1     0.25                        movswq	%ax, %rdi
+# CHECK-NEXT:  1      1     0.25                        movzwq	%ax, %rdi
+# CHECK-NEXT:  1      5     0.33    *                   movswq	(%rax), %rdi
+# CHECK-NEXT:  1      5     0.33    *                   movzwq	(%rax), %rdi
+# CHECK-NEXT:  1      1     0.25                        movslq	%eax, %rdi
+# CHECK-NEXT:  1      5     0.33    *                   movslq	(%rax), %rdi
+# CHECK-NEXT:  1      3     3.00                        mulb	%dil
+# CHECK-NEXT:  1      7     3.00    *                   mulb	(%rax)
+# CHECK-NEXT:  3      3     3.00                        mulw	%si
+# CHECK-NEXT:  4      7     3.00    *                   mulw	(%rax)
+# CHECK-NEXT:  2      3     3.00                        mull	%edx
+# CHECK-NEXT:  2      7     3.00    *                   mull	(%rax)
+# CHECK-NEXT:  2      3     3.00                        mulq	%rcx
+# CHECK-NEXT:  2      7     3.00    *                   mulq	(%rax)
+# CHECK-NEXT:  1      1     0.25                        negb	%dil
+# CHECK-NEXT:  1      6     0.67    *      *            negb	(%r8)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		negb	(%r8)
+# CHECK-NEXT:  1      1     0.25                        negw	%si
+# CHECK-NEXT:  1      6     0.67    *      *            negw	(%r9)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		negw	(%r9)
+# CHECK-NEXT:  1      1     0.25                        negl	%edx
+# CHECK-NEXT:  1      6     0.67    *      *            negl	(%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		negl	(%rax)
+# CHECK-NEXT:  1      1     0.25                        negq	%rcx
+# CHECK-NEXT:  1      6     0.67    *      *            negq	(%r10)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		negq	(%r10)
+# CHECK-NEXT:  1      0     0.25                        nop
+# CHECK-NEXT:  1      0     0.25                        nopw	%di
+# CHECK-NEXT:  1      0     0.25                        nopw	(%rcx)
+# CHECK-NEXT:  1      0     0.25                        nopl	%esi
+# CHECK-NEXT:  1      0     0.25                        nopl	(%r8)
+# CHECK-NEXT:  1      0     0.25                        nopq	%rdx
+# CHECK-NEXT:  1      0     0.25                        nopq	(%r9)
+# CHECK-NEXT:  1      1     0.25                        notb	%dil
+# CHECK-NEXT:  1      6     0.67    *      *            notb	(%r8)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		notb	(%r8)
+# CHECK-NEXT:  1      1     0.25                        notw	%si
+# CHECK-NEXT:  1      6     0.67    *      *            notw	(%r9)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		notw	(%r9)
+# CHECK-NEXT:  1      1     0.25                        notl	%edx
+# CHECK-NEXT:  1      6     0.67    *      *            notl	(%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		notl	(%rax)
+# CHECK-NEXT:  1      1     0.25                        notq	%rcx
+# CHECK-NEXT:  1      6     0.67    *      *            notq	(%r10)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		notq	(%r10)
+# CHECK-NEXT:  1      1     1.00                        orb	$7, %al
+# CHECK-NEXT:  1      1     0.25                        orb	$7, %dil
+# CHECK-NEXT:  1      6     0.67    *      *            orb	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		orb	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        orb	%sil, %dil
+# CHECK-NEXT:  1      6     0.67    *      *            orb	%sil, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		orb	%sil, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   orb	(%rax), %dil
+# CHECK-NEXT:  1      1     1.00                        orw	$511, %ax
+# CHECK-NEXT:  1      1     0.25                        orw	$511, %di
+# CHECK-NEXT:  1      6     0.67    *      *            orw	$511, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		orw	$511, (%rax)
+# CHECK-NEXT:  1      1     0.25                        orw	$7, %di
+# CHECK-NEXT:  1      6     0.67    *      *            orw	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		orw	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        orw	%si, %di
+# CHECK-NEXT:  1      6     0.67    *      *            orw	%si, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		orw	%si, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   orw	(%rax), %di
+# CHECK-NEXT:  1      1     1.00                        orl	$665536, %eax
+# CHECK-NEXT:  1      1     0.25                        orl	$665536, %edi
+# CHECK-NEXT:  1      6     0.67    *      *            orl	$665536, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		orl	$665536, (%rax)
+# CHECK-NEXT:  1      1     0.25                        orl	$7, %edi
+# CHECK-NEXT:  1      6     0.67    *      *            orl	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		orl	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        orl	%esi, %edi
+# CHECK-NEXT:  1      6     0.67    *      *            orl	%esi, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		orl	%esi, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   orl	(%rax), %edi
+# CHECK-NEXT:  1      1     1.00                        orq	$665536, %rax
+# CHECK-NEXT:  1      1     0.25                        orq	$665536, %rdi
+# CHECK-NEXT:  1      6     0.67    *      *            orq	$665536, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		orq	$665536, (%rax)
+# CHECK-NEXT:  1      1     0.25                        orq	$7, %rdi
+# CHECK-NEXT:  1      6     0.67    *      *            orq	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		orq	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        orq	%rsi, %rdi
+# CHECK-NEXT:  1      6     0.67    *      *            orq	%rsi, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		orq	%rsi, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   orq	(%rax), %rdi
+# CHECK-NEXT:  100    100   25.00                 U     outb	%al, $7
+# CHECK-NEXT:  100    100   25.00                 U     outb	%al, %dx
+# CHECK-NEXT:  100    100   25.00                 U     outw	%ax, $7
+# CHECK-NEXT:  100    100   25.00                 U     outw	%ax, %dx
+# CHECK-NEXT:  100    100   25.00                 U     outl	%eax, $7
+# CHECK-NEXT:  100    100   25.00                 U     outl	%eax, %dx
+# CHECK-NEXT:  100    100   25.00                 U     outsb	(%rsi), %dx
+# CHECK-NEXT:  100    100   25.00                 U     outsw	(%rsi), %dx
+# CHECK-NEXT:  100    100   25.00                 U     outsl	(%rsi), %dx
+# CHECK-NEXT:  1      0     0.25    *      *      U     pause
+# CHECK-NEXT:  1      1     1.00                        rclb	%dil
+# CHECK-NEXT:  1      1     1.00                        rcrb	%dil
+# CHECK-NEXT:  2      5     1.00           *            rclb	(%rax)
+# CHECK-NEXT:  2      5     1.00           *            rcrb	(%rax)
+# CHECK-NEXT:  9      4     4.00                        rclb	$7, %dil
+# CHECK-NEXT:  7      3     3.00                        rcrb	$7, %dil
+# CHECK-NEXT:  11     8     4.00           *            rclb	$7, (%rax)
+# CHECK-NEXT:  10     7     4.00           *            rcrb	$7, (%rax)
+# CHECK-NEXT:  9      4     4.00                        rclb	%cl, %dil
+# CHECK-NEXT:  7      3     3.00                        rcrb	%cl, %dil
+# CHECK-NEXT:  11     8     4.00           *            rclb	%cl, (%rax)
+# CHECK-NEXT:  9      7     4.00           *            rcrb	%cl, (%rax)
+# CHECK-NEXT:  1      1     1.00                        rclw	%di
+# CHECK-NEXT:  1      1     1.00                        rcrw	%di
+# CHECK-NEXT:  2      5     1.00           *            rclw	(%rax)
+# CHECK-NEXT:  2      5     1.00           *            rcrw	(%rax)
+# CHECK-NEXT:  9      4     4.00                        rclw	$7, %di
+# CHECK-NEXT:  7      3     3.00                        rcrw	$7, %di
+# CHECK-NEXT:  11     8     4.00           *            rclw	$7, (%rax)
+# CHECK-NEXT:  10     7     4.00           *            rcrw	$7, (%rax)
+# CHECK-NEXT:  9      4     4.00                        rclw	%cl, %di
+# CHECK-NEXT:  7      3     3.00                        rcrw	%cl, %di
+# CHECK-NEXT:  11     8     4.00           *            rclw	%cl, (%rax)
+# CHECK-NEXT:  9      7     4.00           *            rcrw	%cl, (%rax)
+# CHECK-NEXT:  1      1     1.00                        rcll	%edi
+# CHECK-NEXT:  1      1     1.00                        rcrl	%edi
+# CHECK-NEXT:  2      5     1.00           *            rcll	(%rax)
+# CHECK-NEXT:  2      5     1.00           *            rcrl	(%rax)
+# CHECK-NEXT:  9      4     4.00                        rcll	$7, %edi
+# CHECK-NEXT:  7      3     3.00                        rcrl	$7, %edi
+# CHECK-NEXT:  11     8     4.00           *            rcll	$7, (%rax)
+# CHECK-NEXT:  10     7     4.00           *            rcrl	$7, (%rax)
+# CHECK-NEXT:  9      4     4.00                        rcll	%cl, %edi
+# CHECK-NEXT:  7      3     3.00                        rcrl	%cl, %edi
+# CHECK-NEXT:  11     8     4.00           *            rcll	%cl, (%rax)
+# CHECK-NEXT:  9      7     4.00           *            rcrl	%cl, (%rax)
+# CHECK-NEXT:  1      1     1.00                        rclq	%rdi
+# CHECK-NEXT:  1      1     1.00                        rcrq	%rdi
+# CHECK-NEXT:  2      5     1.00           *            rclq	(%rax)
+# CHECK-NEXT:  2      5     1.00           *            rcrq	(%rax)
+# CHECK-NEXT:  9      4     4.00                        rclq	$7, %rdi
+# CHECK-NEXT:  7      3     3.00                        rcrq	$7, %rdi
+# CHECK-NEXT:  11     8     4.00           *            rclq	$7, (%rax)
+# CHECK-NEXT:  10     7     4.00           *            rcrq	$7, (%rax)
+# CHECK-NEXT:  9      4     4.00                        rclq	%cl, %rdi
+# CHECK-NEXT:  7      3     3.00                        rcrq	%cl, %rdi
+# CHECK-NEXT:  11     8     4.00           *            rclq	%cl, (%rax)
+# CHECK-NEXT:  9      7     4.00           *            rcrq	%cl, (%rax)
+# CHECK-NEXT:  100    100   25.00                 U     rdmsr
+# CHECK-NEXT:  100    100   25.00                 U     rdpmc
+# CHECK-NEXT:  100    100   25.00                 U     rdtsc
+# CHECK-NEXT:  100    100   25.00                 U     rdtscp
+# CHECK-NEXT:  1      1     0.50                        rolb	%dil
+# CHECK-NEXT:  1      1     0.50                        rorb	%dil
+# CHECK-NEXT:  2      5     0.67    *      *            rolb	(%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            rorb	(%rax)
+# CHECK-NEXT:  1      1     0.50                        rolb	$7, %dil
+# CHECK-NEXT:  1      1     0.50                        rorb	$7, %dil
+# CHECK-NEXT:  2      5     0.67    *      *            rolb	$7, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            rorb	$7, (%rax)
+# CHECK-NEXT:  1      1     0.50                        rolb	%cl, %dil
+# CHECK-NEXT:  1      1     0.50                        rorb	%cl, %dil
+# CHECK-NEXT:  2      5     0.67    *      *            rolb	%cl, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            rorb	%cl, (%rax)
+# CHECK-NEXT:  1      1     0.50                        rolw	%di
+# CHECK-NEXT:  1      1     0.50                        rorw	%di
+# CHECK-NEXT:  2      5     0.67    *      *            rolw	(%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            rorw	(%rax)
+# CHECK-NEXT:  1      1     0.50                        rolw	$7, %di
+# CHECK-NEXT:  1      1     0.50                        rorw	$7, %di
+# CHECK-NEXT:  2      5     0.67    *      *            rolw	$7, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            rorw	$7, (%rax)
+# CHECK-NEXT:  1      1     0.50                        rolw	%cl, %di
+# CHECK-NEXT:  1      1     0.50                        rorw	%cl, %di
+# CHECK-NEXT:  2      5     0.67    *      *            rolw	%cl, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            rorw	%cl, (%rax)
+# CHECK-NEXT:  1      1     0.50                        roll	%edi
+# CHECK-NEXT:  1      1     0.50                        rorl	%edi
+# CHECK-NEXT:  2      5     0.67    *      *            roll	(%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            rorl	(%rax)
+# CHECK-NEXT:  1      1     0.50                        roll	$7, %edi
+# CHECK-NEXT:  1      1     0.50                        rorl	$7, %edi
+# CHECK-NEXT:  2      5     0.67    *      *            roll	$7, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            rorl	$7, (%rax)
+# CHECK-NEXT:  1      1     0.50                        roll	%cl, %edi
+# CHECK-NEXT:  1      1     0.50                        rorl	%cl, %edi
+# CHECK-NEXT:  2      5     0.67    *      *            roll	%cl, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            rorl	%cl, (%rax)
+# CHECK-NEXT:  1      1     0.50                        rolq	%rdi
+# CHECK-NEXT:  1      1     0.50                        rorq	%rdi
+# CHECK-NEXT:  2      5     0.67    *      *            rolq	(%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            rorq	(%rax)
+# CHECK-NEXT:  1      1     0.50                        rolq	$7, %rdi
+# CHECK-NEXT:  1      1     0.50                        rorq	$7, %rdi
+# CHECK-NEXT:  2      5     0.67    *      *            rolq	$7, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            rorq	$7, (%rax)
+# CHECK-NEXT:  1      1     0.50                        rolq	%cl, %rdi
+# CHECK-NEXT:  1      1     0.50                        rorq	%cl, %rdi
+# CHECK-NEXT:  2      5     0.67    *      *            rolq	%cl, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            rorq	%cl, (%rax)
+# CHECK-NEXT:  1      1     1.00                        sahf
+# CHECK-NEXT:  1      1     0.50                        sarb	%dil
+# CHECK-NEXT:  1      1     0.50                        shlb	%dil
+# CHECK-NEXT:  1      1     0.50                        shrb	%dil
+# CHECK-NEXT:  2      5     0.67    *      *            sarb	(%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shlb	(%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shrb	(%rax)
+# CHECK-NEXT:  1      1     0.50                        sarb	$7, %dil
+# CHECK-NEXT:  1      1     0.50                        shlb	$7, %dil
+# CHECK-NEXT:  1      1     0.50                        shrb	$7, %dil
+# CHECK-NEXT:  2      5     0.67    *      *            sarb	$7, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shlb	$7, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shrb	$7, (%rax)
+# CHECK-NEXT:  1      1     0.50                        sarb	%cl, %dil
+# CHECK-NEXT:  1      1     0.50                        shlb	%cl, %dil
+# CHECK-NEXT:  1      1     0.50                        shrb	%cl, %dil
+# CHECK-NEXT:  2      5     0.67    *      *            sarb	%cl, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shlb	%cl, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shrb	%cl, (%rax)
+# CHECK-NEXT:  1      1     0.50                        sarw	%di
+# CHECK-NEXT:  1      1     0.50                        shlw	%di
+# CHECK-NEXT:  1      1     0.50                        shrw	%di
+# CHECK-NEXT:  2      5     0.67    *      *            sarw	(%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shlw	(%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shrw	(%rax)
+# CHECK-NEXT:  1      1     0.50                        sarw	$7, %di
+# CHECK-NEXT:  1      1     0.50                        shlw	$7, %di
+# CHECK-NEXT:  1      1     0.50                        shrw	$7, %di
+# CHECK-NEXT:  2      5     0.67    *      *            sarw	$7, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shlw	$7, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shrw	$7, (%rax)
+# CHECK-NEXT:  1      1     0.50                        sarw	%cl, %di
+# CHECK-NEXT:  1      1     0.50                        shlw	%cl, %di
+# CHECK-NEXT:  1      1     0.50                        shrw	%cl, %di
+# CHECK-NEXT:  2      5     0.67    *      *            sarw	%cl, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shlw	%cl, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shrw	%cl, (%rax)
+# CHECK-NEXT:  1      1     0.50                        sarl	%edi
+# CHECK-NEXT:  1      1     0.50                        shll	%edi
+# CHECK-NEXT:  1      1     0.50                        shrl	%edi
+# CHECK-NEXT:  2      5     0.67    *      *            sarl	(%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shll	(%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shrl	(%rax)
+# CHECK-NEXT:  1      1     0.50                        sarl	$7, %edi
+# CHECK-NEXT:  1      1     0.50                        shll	$7, %edi
+# CHECK-NEXT:  1      1     0.50                        shrl	$7, %edi
+# CHECK-NEXT:  2      5     0.67    *      *            sarl	$7, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shll	$7, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shrl	$7, (%rax)
+# CHECK-NEXT:  1      1     0.50                        sarl	%cl, %edi
+# CHECK-NEXT:  1      1     0.50                        shll	%cl, %edi
+# CHECK-NEXT:  1      1     0.50                        shrl	%cl, %edi
+# CHECK-NEXT:  2      5     0.67    *      *            sarl	%cl, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shll	%cl, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shrl	%cl, (%rax)
+# CHECK-NEXT:  1      1     0.50                        sarq	%rdi
+# CHECK-NEXT:  1      1     0.50                        shlq	%rdi
+# CHECK-NEXT:  1      1     0.50                        shrq	%rdi
+# CHECK-NEXT:  2      5     0.67    *      *            sarq	(%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shlq	(%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shrq	(%rax)
+# CHECK-NEXT:  1      1     0.50                        sarq	$7, %rdi
+# CHECK-NEXT:  1      1     0.50                        shlq	$7, %rdi
+# CHECK-NEXT:  1      1     0.50                        shrq	$7, %rdi
+# CHECK-NEXT:  2      5     0.67    *      *            sarq	$7, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shlq	$7, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shrq	$7, (%rax)
+# CHECK-NEXT:  1      1     0.50                        sarq	%cl, %rdi
+# CHECK-NEXT:  1      1     0.50                        shlq	%cl, %rdi
+# CHECK-NEXT:  1      1     0.50                        shrq	%cl, %rdi
+# CHECK-NEXT:  2      5     0.67    *      *            sarq	%cl, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shlq	%cl, (%rax)
+# CHECK-NEXT:  2      5     0.67    *      *            shrq	%cl, (%rax)
+# CHECK-NEXT:  1      1     1.00                        sbbb	$0, %al
+# CHECK-NEXT:  1      1     1.00                        sbbb	$0, %dil
+# CHECK-NEXT:  1      6     1.00    *      *            sbbb	$0, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		sbbb	$0, (%rax)
+# CHECK-NEXT:  1      1     1.00                        sbbb	$7, %al
+# CHECK-NEXT:  1      1     1.00                        sbbb	$7, %dil
+# CHECK-NEXT:  1      6     1.00    *      *            sbbb	$7, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		sbbb	$7, (%rax)
+# CHECK-NEXT:  1      1     1.00                        sbbb	%sil, %dil
+# CHECK-NEXT:  1      1     1.75    *      *            sbbb	%sil, (%rax)
+# CHECK-NEXT:  1      1     1.75    *      *            lock		sbbb	%sil, (%rax)
+# CHECK-NEXT:  1      5     1.00    *                   sbbb	(%rax), %dil
+# CHECK-NEXT:  1      1     1.00                        sbbw	$0, %ax
+# CHECK-NEXT:  1      1     1.00                        sbbw	$0, %di
+# CHECK-NEXT:  1      6     1.00    *      *            sbbw	$0, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		sbbw	$0, (%rax)
+# CHECK-NEXT:  1      1     1.00                        sbbw	$511, %ax
+# CHECK-NEXT:  1      1     1.00                        sbbw	$511, %di
+# CHECK-NEXT:  1      6     1.00    *      *            sbbw	$511, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		sbbw	$511, (%rax)
+# CHECK-NEXT:  1      1     1.00                        sbbw	$7, %di
+# CHECK-NEXT:  1      6     1.00    *      *            sbbw	$7, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		sbbw	$7, (%rax)
+# CHECK-NEXT:  1      1     1.00                        sbbw	%si, %di
+# CHECK-NEXT:  1      6     1.00    *      *            sbbw	%si, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		sbbw	%si, (%rax)
+# CHECK-NEXT:  1      5     1.00    *                   sbbw	(%rax), %di
+# CHECK-NEXT:  1      1     1.00                        sbbl	$0, %eax
+# CHECK-NEXT:  1      1     1.00                        sbbl	$0, %edi
+# CHECK-NEXT:  1      6     1.00    *      *            sbbl	$0, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		sbbl	$0, (%rax)
+# CHECK-NEXT:  1      1     1.00                        sbbl	$665536, %eax
+# CHECK-NEXT:  1      1     1.00                        sbbl	$665536, %edi
+# CHECK-NEXT:  1      6     1.00    *      *            sbbl	$665536, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		sbbl	$665536, (%rax)
+# CHECK-NEXT:  1      1     1.00                        sbbl	$7, %edi
+# CHECK-NEXT:  1      6     1.00    *      *            sbbl	$7, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		sbbl	$7, (%rax)
+# CHECK-NEXT:  1      1     1.00                        sbbl	%esi, %edi
+# CHECK-NEXT:  1      6     1.00    *      *            sbbl	%esi, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		sbbl	%esi, (%rax)
+# CHECK-NEXT:  1      5     1.00    *                   sbbl	(%rax), %edi
+# CHECK-NEXT:  1      1     1.00                        sbbq	$0, %rax
+# CHECK-NEXT:  1      1     1.00                        sbbq	$0, %rdi
+# CHECK-NEXT:  1      6     1.00    *      *            sbbq	$0, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		sbbq	$0, (%rax)
+# CHECK-NEXT:  1      1     1.00                        sbbq	$665536, %rax
+# CHECK-NEXT:  1      1     1.00                        sbbq	$665536, %rdi
+# CHECK-NEXT:  1      6     1.00    *      *            sbbq	$665536, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		sbbq	$665536, (%rax)
+# CHECK-NEXT:  1      1     1.00                        sbbq	$7, %rdi
+# CHECK-NEXT:  1      6     1.00    *      *            sbbq	$7, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		sbbq	$7, (%rax)
+# CHECK-NEXT:  1      1     1.00                        sbbq	%rsi, %rdi
+# CHECK-NEXT:  1      6     1.00    *      *            sbbq	%rsi, (%rax)
+# CHECK-NEXT:  1      6     1.00    *      *            lock		sbbq	%rsi, (%rax)
+# CHECK-NEXT:  1      5     1.00    *                   sbbq	(%rax), %rdi
+# CHECK-NEXT:  100    100   25.00                 U     scasb	%es:(%rdi), %al
+# CHECK-NEXT:  100    100   25.00                 U     scasw	%es:(%rdi), %ax
+# CHECK-NEXT:  100    100   25.00                 U     scasl	%es:(%rdi), %eax
+# CHECK-NEXT:  100    100   25.00                 U     scasq	%es:(%rdi), %rax
+# CHECK-NEXT:  1      1     1.00                        seto	%al
+# CHECK-NEXT:  2      2     1.00           *            seto	(%rax)
+# CHECK-NEXT:  1      1     1.00                        setno	%al
+# CHECK-NEXT:  2      2     1.00           *            setno	(%rax)
+# CHECK-NEXT:  1      1     1.00                        setb	%al
+# CHECK-NEXT:  2      2     1.00           *            setb	(%rax)
+# CHECK-NEXT:  1      1     1.00                        setae	%al
+# CHECK-NEXT:  2      2     1.00           *            setae	(%rax)
+# CHECK-NEXT:  1      1     1.00                        sete	%al
+# CHECK-NEXT:  2      2     1.00           *            sete	(%rax)
+# CHECK-NEXT:  1      1     1.00                        setne	%al
+# CHECK-NEXT:  2      2     1.00           *            setne	(%rax)
+# CHECK-NEXT:  1      1     1.00                        seta	%al
+# CHECK-NEXT:  2      2     1.00           *            seta	(%rax)
+# CHECK-NEXT:  1      1     1.00                        setbe	%al
+# CHECK-NEXT:  2      2     1.00           *            setbe	(%rax)
+# CHECK-NEXT:  1      1     1.00                        sets	%al
+# CHECK-NEXT:  2      2     1.00           *            sets	(%rax)
+# CHECK-NEXT:  1      1     1.00                        setns	%al
+# CHECK-NEXT:  2      2     1.00           *            setns	(%rax)
+# CHECK-NEXT:  1      1     1.00                        setp	%al
+# CHECK-NEXT:  2      2     1.00           *            setp	(%rax)
+# CHECK-NEXT:  1      1     1.00                        setnp	%al
+# CHECK-NEXT:  2      2     1.00           *            setnp	(%rax)
+# CHECK-NEXT:  1      1     1.00                        setl	%al
+# CHECK-NEXT:  2      2     1.00           *            setl	(%rax)
+# CHECK-NEXT:  1      1     1.00                        setge	%al
+# CHECK-NEXT:  2      2     1.00           *            setge	(%rax)
+# CHECK-NEXT:  1      1     1.00                        setg	%al
+# CHECK-NEXT:  2      2     1.00           *            setg	(%rax)
+# CHECK-NEXT:  1      1     1.00                        setle	%al
+# CHECK-NEXT:  2      2     1.00           *            setle	(%rax)
+# CHECK-NEXT:  5      2     1.50                        shldw	%cl, %si, %di
+# CHECK-NEXT:  5      2     1.50                        shrdw	%cl, %si, %di
+# CHECK-NEXT:  6      6     2.00    *      *            shldw	%cl, %si, (%rax)
+# CHECK-NEXT:  6      6     2.00    *      *            shrdw	%cl, %si, (%rax)
+# CHECK-NEXT:  4      2     1.50                        shldw	$7, %si, %di
+# CHECK-NEXT:  4      2     1.50                        shrdw	$7, %si, %di
+# CHECK-NEXT:  6      6     2.00    *      *            shldw	$7, %si, (%rax)
+# CHECK-NEXT:  6      6     2.00    *      *            shrdw	$7, %si, (%rax)
+# CHECK-NEXT:  5      2     1.50                        shldl	%cl, %esi, %edi
+# CHECK-NEXT:  5      2     1.50                        shrdl	%cl, %esi, %edi
+# CHECK-NEXT:  6      6     2.00    *      *            shldl	%cl, %esi, (%rax)
+# CHECK-NEXT:  6      6     2.00    *      *            shrdl	%cl, %esi, (%rax)
+# CHECK-NEXT:  4      2     1.50                        shldl	$7, %esi, %edi
+# CHECK-NEXT:  4      2     1.50                        shrdl	$7, %esi, %edi
+# CHECK-NEXT:  6      6     2.00    *      *            shldl	$7, %esi, (%rax)
+# CHECK-NEXT:  6      6     2.00    *      *            shrdl	$7, %esi, (%rax)
+# CHECK-NEXT:  5      2     1.50                        shldq	%cl, %rsi, %rdi
+# CHECK-NEXT:  5      2     1.50                        shrdq	%cl, %rsi, %rdi
+# CHECK-NEXT:  6      6     2.00    *      *            shldq	%cl, %rsi, (%rax)
+# CHECK-NEXT:  6      6     2.00    *      *            shrdq	%cl, %rsi, (%rax)
+# CHECK-NEXT:  4      2     1.50                        shldq	$7, %rsi, %rdi
+# CHECK-NEXT:  4      2     1.50                        shrdq	$7, %rsi, %rdi
+# CHECK-NEXT:  6      6     2.00    *      *            shldq	$7, %rsi, (%rax)
+# CHECK-NEXT:  6      6     2.00    *      *            shrdq	$7, %rsi, (%rax)
+# CHECK-NEXT:  1      1     0.25                  U     stc
+# CHECK-NEXT:  1      1     0.25                  U     std
+# CHECK-NEXT:  100    100   25.00                 U     stosb	%al, %es:(%rdi)
+# CHECK-NEXT:  100    100   25.00                 U     stosw	%ax, %es:(%rdi)
+# CHECK-NEXT:  100    100   25.00                 U     stosl	%eax, %es:(%rdi)
+# CHECK-NEXT:  100    100   25.00                 U     stosq	%rax, %es:(%rdi)
+# CHECK-NEXT:  1      1     1.00                        subb	$7, %al
+# CHECK-NEXT:  1      1     0.25                        subb	$7, %dil
+# CHECK-NEXT:  1      6     0.67    *      *            subb	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		subb	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        subb	%sil, %dil
+# CHECK-NEXT:  1      6     0.67    *      *            subb	%sil, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		subb	%sil, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   subb	(%rax), %dil
+# CHECK-NEXT:  1      1     1.00                        subw	$511, %ax
+# CHECK-NEXT:  1      1     0.25                        subw	$511, %di
+# CHECK-NEXT:  1      6     0.67    *      *            subw	$511, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		subw	$511, (%rax)
+# CHECK-NEXT:  1      1     0.25                        subw	$7, %di
+# CHECK-NEXT:  1      6     0.67    *      *            subw	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		subw	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        subw	%si, %di
+# CHECK-NEXT:  1      6     0.67    *      *            subw	%si, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		subw	%si, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   subw	(%rax), %di
+# CHECK-NEXT:  1      1     1.00                        subl	$665536, %eax
+# CHECK-NEXT:  1      1     0.25                        subl	$665536, %edi
+# CHECK-NEXT:  1      6     0.67    *      *            subl	$665536, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		subl	$665536, (%rax)
+# CHECK-NEXT:  1      1     0.25                        subl	$7, %edi
+# CHECK-NEXT:  1      6     0.67    *      *            subl	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		subl	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        subl	%esi, %edi
+# CHECK-NEXT:  1      6     0.67    *      *            subl	%esi, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		subl	%esi, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   subl	(%rax), %edi
+# CHECK-NEXT:  1      1     1.00                        subq	$665536, %rax
+# CHECK-NEXT:  1      1     0.25                        subq	$665536, %rdi
+# CHECK-NEXT:  1      6     0.67    *      *            subq	$665536, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		subq	$665536, (%rax)
+# CHECK-NEXT:  1      1     0.25                        subq	$7, %rdi
+# CHECK-NEXT:  1      6     0.67    *      *            subq	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		subq	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        subq	%rsi, %rdi
+# CHECK-NEXT:  1      6     0.67    *      *            subq	%rsi, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		subq	%rsi, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   subq	(%rax), %rdi
+# CHECK-NEXT:  1      1     0.25                        testb	$7, %al
+# CHECK-NEXT:  1      1     0.25                        testb	$7, %dil
+# CHECK-NEXT:  1      5     0.33    *                   testb	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        testb	%sil, %dil
+# CHECK-NEXT:  1      5     0.33    *                   testb	%sil, (%rax)
+# CHECK-NEXT:  1      1     0.25                        testw	$511, %ax
+# CHECK-NEXT:  1      1     0.25                        testw	$511, %di
+# CHECK-NEXT:  1      5     0.33    *                   testw	$511, (%rax)
+# CHECK-NEXT:  1      1     0.25                        testw	$7, %di
+# CHECK-NEXT:  1      5     0.33    *                   testw	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        testw	%si, %di
+# CHECK-NEXT:  1      5     0.33    *                   testw	%si, (%rax)
+# CHECK-NEXT:  1      1     0.25                        testl	$665536, %eax
+# CHECK-NEXT:  1      1     0.25                        testl	$665536, %edi
+# CHECK-NEXT:  1      5     0.33    *                   testl	$665536, (%rax)
+# CHECK-NEXT:  1      1     0.25                        testl	$7, %edi
+# CHECK-NEXT:  1      5     0.33    *                   testl	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        testl	%esi, %edi
+# CHECK-NEXT:  1      5     0.33    *                   testl	%esi, (%rax)
+# CHECK-NEXT:  1      1     0.25                        testq	$665536, %rax
+# CHECK-NEXT:  1      1     0.25                        testq	$665536, %rdi
+# CHECK-NEXT:  1      5     0.33    *                   testq	$665536, (%rax)
+# CHECK-NEXT:  1      1     0.25                        testq	$7, %rdi
+# CHECK-NEXT:  1      5     0.33    *                   testq	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        testq	%rsi, %rdi
+# CHECK-NEXT:  1      5     0.33    *                   testq	%rsi, (%rax)
+# CHECK-NEXT:  100    100   25.00   *             U     ud2
+# CHECK-NEXT:  100    100   25.00                 U     wrmsr
+# CHECK-NEXT:  2      0     2.00                        xaddb	%bl, %cl
+# CHECK-NEXT:  1      5     0.67    *      *            xaddb	%bl, (%rcx)
+# CHECK-NEXT:  1      5     0.67    *      *            lock		xaddb	%bl, (%rcx)
+# CHECK-NEXT:  2      0     2.00                        xaddw	%bx, %cx
+# CHECK-NEXT:  1      5     0.67    *      *            xaddw	%ax, (%rbx)
+# CHECK-NEXT:  1      5     0.67    *      *            lock		xaddw	%ax, (%rbx)
+# CHECK-NEXT:  2      0     2.00                        xaddl	%ebx, %ecx
+# CHECK-NEXT:  1      5     0.67    *      *            xaddl	%eax, (%rbx)
+# CHECK-NEXT:  1      5     0.67    *      *            lock		xaddl	%eax, (%rbx)
+# CHECK-NEXT:  2      0     2.00                        xaddq	%rbx, %rcx
+# CHECK-NEXT:  1      5     0.67    *      *            xaddq	%rax, (%rbx)
+# CHECK-NEXT:  1      5     0.67    *      *            lock		xaddq	%rax, (%rbx)
+# CHECK-NEXT:  2      1     0.50                        xchgb	%bl, %cl
+# CHECK-NEXT:  5      7     0.50    *      *            xchgb	%bl, (%rbx)
+# CHECK-NEXT:  5      7     0.50    *      *            lock		xchgb	%bl, (%rbx)
+# CHECK-NEXT:  2      1     0.50                        xchgw	%bx, %ax
+# CHECK-NEXT:  2      1     0.50                        xchgw	%bx, %cx
+# CHECK-NEXT:  5      7     0.50    *      *            xchgw	%ax, (%rbx)
+# CHECK-NEXT:  5      7     0.50    *      *            lock		xchgw	%ax, (%rbx)
+# CHECK-NEXT:  2      0     0.33                        xchgl	%ebx, %eax
+# CHECK-NEXT:  2      0     0.33                        xchgl	%ebx, %ecx
+# CHECK-NEXT:  2      6     0.50    *      *            xchgl	%eax, (%rbx)
+# CHECK-NEXT:  2      6     0.50    *      *            lock		xchgl	%eax, (%rbx)
+# CHECK-NEXT:  2      0     0.33                        xchgq	%rbx, %rax
+# CHECK-NEXT:  2      0     0.33                        xchgq	%rbx, %rcx
+# CHECK-NEXT:  2      6     0.50    *      *            xchgq	%rax, (%rbx)
+# CHECK-NEXT:  2      6     0.50    *      *            lock		xchgq	%rax, (%rbx)
+# CHECK-NEXT:  1      5     0.33    *                   xlatb
+# CHECK-NEXT:  1      1     1.00                        xorb	$7, %al
+# CHECK-NEXT:  1      1     0.25                        xorb	$7, %dil
+# CHECK-NEXT:  1      6     0.67    *      *            xorb	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		xorb	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        xorb	%sil, %dil
+# CHECK-NEXT:  1      6     0.67    *      *            xorb	%sil, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		xorb	%sil, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   xorb	(%rax), %dil
+# CHECK-NEXT:  1      1     1.00                        xorw	$511, %ax
+# CHECK-NEXT:  1      1     0.25                        xorw	$511, %di
+# CHECK-NEXT:  1      6     0.67    *      *            xorw	$511, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		xorw	$511, (%rax)
+# CHECK-NEXT:  1      1     0.25                        xorw	$7, %di
+# CHECK-NEXT:  1      6     0.67    *      *            xorw	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		xorw	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        xorw	%si, %di
+# CHECK-NEXT:  1      6     0.67    *      *            xorw	%si, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		xorw	%si, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   xorw	(%rax), %di
+# CHECK-NEXT:  1      1     1.00                        xorl	$665536, %eax
+# CHECK-NEXT:  1      1     0.25                        xorl	$665536, %edi
+# CHECK-NEXT:  1      6     0.67    *      *            xorl	$665536, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		xorl	$665536, (%rax)
+# CHECK-NEXT:  1      1     0.25                        xorl	$7, %edi
+# CHECK-NEXT:  1      6     0.67    *      *            xorl	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		xorl	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        xorl	%esi, %edi
+# CHECK-NEXT:  1      6     0.67    *      *            xorl	%esi, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		xorl	%esi, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   xorl	(%rax), %edi
+# CHECK-NEXT:  1      1     1.00                        xorq	$665536, %rax
+# CHECK-NEXT:  1      1     0.25                        xorq	$665536, %rdi
+# CHECK-NEXT:  1      6     0.67    *      *            xorq	$665536, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		xorq	$665536, (%rax)
+# CHECK-NEXT:  1      1     0.25                        xorq	$7, %rdi
+# CHECK-NEXT:  1      6     0.67    *      *            xorq	$7, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		xorq	$7, (%rax)
+# CHECK-NEXT:  1      1     0.25                        xorq	%rsi, %rdi
+# CHECK-NEXT:  1      6     0.67    *      *            xorq	%rsi, (%rax)
+# CHECK-NEXT:  1      6     0.67    *      *            lock		xorq	%rsi, (%rax)
+# CHECK-NEXT:  1      5     0.33    *                   xorq	(%rax), %rdi
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 259.00 259.00 259.00 1733.00 1865.50 1775.50 1529.50 1.50  -    -      -      -      -      -      -     259.00 259.00 259.00 151.67 151.67 151.67 161.00 161.00
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcb	$0, %al
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcb	$0, %dil
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   adcb	$0, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		adcb	$0, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcb	$7, %al
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcb	$7, %dil
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   adcb	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		adcb	$7, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcb	%sil, %dil
+# CHECK-NEXT: 0.33   0.33   0.33   1.75   1.75   1.75   1.75    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   adcb	%sil, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   1.75   1.75   1.75   1.75    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		adcb	%sil, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     adcb	(%rax), %dil
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcw	$0, %ax
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcw	$0, %di
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   adcw	$0, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		adcw	$0, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcw	$511, %ax
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcw	$511, %di
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   adcw	$511, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		adcw	$511, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcw	$7, %di
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   adcw	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		adcw	$7, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcw	%si, %di
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   adcw	%si, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		adcw	%si, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     adcw	(%rax), %di
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcl	$0, %eax
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcl	$0, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   adcl	$0, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		adcl	$0, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcl	$665536, %eax
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcl	$665536, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   adcl	$665536, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		adcl	$665536, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcl	$7, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   adcl	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		adcl	$7, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcl	%esi, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   adcl	%esi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		adcl	%esi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     adcl	(%rax), %edi
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcq	$0, %rax
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcq	$0, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   adcq	$0, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		adcq	$0, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcq	$665536, %rax
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcq	$665536, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   adcq	$665536, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		adcq	$665536, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcq	$7, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   adcq	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		adcq	$7, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     adcq	%rsi, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   adcq	%rsi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		adcq	%rsi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     adcq	(%rax), %rdi
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     addb	$7, %al
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     addb	$7, %dil
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   addb	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		addb	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     addb	%sil, %dil
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   addb	%sil, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		addb	%sil, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     addb	(%rax), %dil
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     addw	$511, %ax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     addw	$511, %di
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   addw	$511, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		addw	$511, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     addw	$7, %di
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   addw	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		addw	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     addw	%si, %di
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   addw	%si, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		addw	%si, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     addw	(%rax), %di
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     addl	$665536, %eax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     addl	$665536, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   addl	$665536, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		addl	$665536, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     addl	$7, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   addl	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		addl	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     addl	%esi, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   addl	%esi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		addl	%esi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     addl	(%rax), %edi
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     addq	$665536, %rax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     addq	$665536, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   addq	$665536, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		addq	$665536, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     addq	$7, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   addq	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		addq	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     addq	%rsi, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   addq	%rsi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		addq	%rsi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     addq	(%rax), %rdi
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     andb	$7, %al
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     andb	$7, %dil
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   andb	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		andb	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     andb	%sil, %dil
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   andb	%sil, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		andb	%sil, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     andb	(%rax), %dil
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     andw	$511, %ax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     andw	$511, %di
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   andw	$511, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		andw	$511, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     andw	$7, %di
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   andw	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		andw	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     andw	%si, %di
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   andw	%si, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		andw	%si, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     andw	(%rax), %di
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     andl	$665536, %eax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     andl	$665536, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   andl	$665536, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		andl	$665536, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     andl	$7, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   andl	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		andl	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     andl	%esi, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   andl	%esi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		andl	%esi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     andl	(%rax), %edi
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     andq	$665536, %rax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     andq	$665536, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   andq	$665536, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		andq	$665536, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     andq	$7, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   andq	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		andq	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     andq	%rsi, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   andq	%rsi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		andq	%rsi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     andq	(%rax), %rdi
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     bsfw	%si, %di
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     bsrw	%si, %di
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     bsfw	(%rax), %di
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     bsrw	(%rax), %di
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     bsfl	%esi, %edi
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     bsrl	%esi, %edi
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     bsfl	(%rax), %edi
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     bsrl	(%rax), %edi
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     bsfq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     bsrq	%rsi, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     bsfq	(%rax), %rdi
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     bsrq	(%rax), %rdi
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     bswapl	%eax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     bswapq	%rax
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btw	%si, %di
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btcw	%si, %di
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btrw	%si, %di
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btsw	%si, %di
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     btw	%si, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   btcw	%si, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   btrw	%si, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   btsw	%si, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		btcw	%si, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		btrw	%si, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		btsw	%si, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btw	$7, %di
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btcw	$7, %di
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btrw	$7, %di
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btsw	$7, %di
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     btw	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   btcw	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   btrw	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   btsw	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		btcw	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		btrw	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		btsw	$7, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btl	%esi, %edi
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btcl	%esi, %edi
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btrl	%esi, %edi
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btsl	%esi, %edi
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     btl	%esi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   btcl	%esi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   btrl	%esi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   btsl	%esi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		btcl	%esi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		btrl	%esi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		btsl	%esi, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btl	$7, %edi
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btcl	$7, %edi
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btrl	$7, %edi
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btsl	$7, %edi
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     btl	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   btcl	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   btrl	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   btsl	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		btcl	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		btrl	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		btsl	$7, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btcq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btrq	%rsi, %rdi
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btsq	%rsi, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     btq	%rsi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   btcq	%rsi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   btrq	%rsi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   btsq	%rsi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		btcq	%rsi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		btrq	%rsi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		btsq	%rsi, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btq	$7, %rdi
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btcq	$7, %rdi
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btrq	$7, %rdi
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     btsq	$7, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     btq	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   btcq	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   btrq	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   btsq	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		btcq	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		btrq	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		btsq	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cbtw
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cwtl
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cltq
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cwtd
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cltd
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cqto
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     clc
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cld
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmc
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpb	$7, %al
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpb	$7, %dil
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpb	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpb	%sil, %dil
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpb	%sil, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpb	(%rax), %dil
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpw	$511, %ax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpw	$511, %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpw	$511, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpw	$7, %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpw	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpw	%si, %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpw	%si, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpw	(%rax), %di
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpl	$665536, %eax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpl	$665536, %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpl	$665536, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpl	$7, %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpl	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpl	%esi, %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpl	%esi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpl	(%rax), %edi
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpq	$665536, %rax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpq	$665536, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpq	$665536, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpq	$7, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpq	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpq	%rsi, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpq	%rsi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpq	(%rax), %rdi
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpsb	%es:(%rdi), (%rsi)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpsw	%es:(%rdi), (%rsi)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpsl	%es:(%rdi), (%rsi)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpsq	%es:(%rdi), (%rsi)
+# CHECK-NEXT:  -      -      -     3.00   3.00   3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpxchgb	%cl, %bl
+# CHECK-NEXT: 0.33   0.33   0.33   3.00   3.00   3.00   3.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     cmpxchgb	%cl, (%rbx)
+# CHECK-NEXT: 0.33   0.33   0.33   3.00   3.00   3.00   3.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     lock		cmpxchgb	%cl, (%rbx)
+# CHECK-NEXT:  -      -      -     3.00   3.00   3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpxchgw	%cx, %bx
+# CHECK-NEXT:  -      -      -     3.00   3.00   3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpxchgw	%cx, (%rbx)
+# CHECK-NEXT:  -      -      -     3.00   3.00   3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     lock		cmpxchgw	%cx, (%rbx)
+# CHECK-NEXT:  -      -      -     3.00   3.00   3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpxchgl	%ecx, %ebx
+# CHECK-NEXT:  -      -      -     3.00   3.00   3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpxchgl	%ecx, (%rbx)
+# CHECK-NEXT:  -      -      -     3.00   3.00   3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     lock		cmpxchgl	%ecx, (%rbx)
+# CHECK-NEXT:  -      -      -     3.00   3.00   3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpxchgq	%rcx, %rbx
+# CHECK-NEXT:  -      -      -     3.00   3.00   3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cmpxchgq	%rcx, (%rbx)
+# CHECK-NEXT:  -      -      -     3.00   3.00   3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     lock		cmpxchgq	%rcx, (%rbx)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     cpuid
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     decb	%dil
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   decb	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		decb	(%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     decw	%di
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   decw	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		decw	(%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     decl	%edi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   decl	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		decl	(%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     decq	%rdi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   decq	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		decq	(%rax)
+# CHECK-NEXT:  -      -      -     10.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     divb	%dil
+# CHECK-NEXT: 0.33   0.33   0.33   10.00   -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     divb	(%rax)
+# CHECK-NEXT:  -      -      -     11.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     divw	%si
+# CHECK-NEXT: 0.33   0.33   0.33   11.00   -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     divw	(%rax)
+# CHECK-NEXT:  -      -      -     13.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     divl	%edx
+# CHECK-NEXT: 0.33   0.33   0.33   13.00   -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     divl	(%rax)
+# CHECK-NEXT:  -      -      -     17.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     divq	%rcx
+# CHECK-NEXT: 0.33   0.33   0.33   17.00   -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     divq	(%rax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     enter	$7, $4095
+# CHECK-NEXT:  -      -      -     10.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     idivb	%dil
+# CHECK-NEXT: 0.33   0.33   0.33   10.00   -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     idivb	(%rax)
+# CHECK-NEXT:  -      -      -     11.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     idivw	%si
+# CHECK-NEXT: 0.33   0.33   0.33   11.00   -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     idivw	(%rax)
+# CHECK-NEXT:  -      -      -     13.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     idivl	%edx
+# CHECK-NEXT: 0.33   0.33   0.33   13.00   -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     idivl	(%rax)
+# CHECK-NEXT:  -      -      -     17.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     idivq	%rcx
+# CHECK-NEXT: 0.33   0.33   0.33   17.00   -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     idivq	(%rax)
+# CHECK-NEXT:  -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     imulb	%dil
+# CHECK-NEXT: 0.33   0.33   0.33    -     3.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     imulb	(%rax)
+# CHECK-NEXT:  -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     imulw	%di
+# CHECK-NEXT: 0.33   0.33   0.33    -     3.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     imulw	(%rax)
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     imulw	%si, %di
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     imulw	(%rax), %di
+# CHECK-NEXT:  -      -      -      -     4.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     imulw	$511, %si, %di
+# CHECK-NEXT: 0.33   0.33   0.33    -     4.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     imulw	$511, (%rax), %di
+# CHECK-NEXT:  -      -      -      -     4.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     imulw	$7, %si, %di
+# CHECK-NEXT: 0.33   0.33   0.33    -     4.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     imulw	$7, (%rax), %di
+# CHECK-NEXT:  -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     imull	%edi
+# CHECK-NEXT: 0.33   0.33   0.33    -     3.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     imull	(%rax)
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     imull	%esi, %edi
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     imull	(%rax), %edi
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     imull	$665536, %esi, %edi
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     imull	$665536, (%rax), %edi
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     imull	$7, %esi, %edi
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     imull	$7, (%rax), %edi
+# CHECK-NEXT:  -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     imulq	%rdi
+# CHECK-NEXT: 0.33   0.33   0.33    -     3.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     imulq	(%rax)
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     imulq	%rsi, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     imulq	(%rax), %rdi
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     imulq	$665536, %rsi, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     imulq	$665536, (%rax), %rdi
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     imulq	$7, %rsi, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     imulq	$7, (%rax), %rdi
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     inb	$7, %al
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     inb	%dx, %al
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     inw	$7, %ax
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     inw	%dx, %ax
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     inl	$7, %eax
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     inl	%dx, %eax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     incb	%dil
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   incb	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		incb	(%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     incw	%di
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   incw	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		incw	(%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     incl	%edi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   incl	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		incl	(%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     incq	%rdi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   incq	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		incq	(%rax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     insb	%dx, %es:(%rdi)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     insw	%dx, %es:(%rdi)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     insl	%dx, %es:(%rdi)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     int	$7
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     invlpg	(%rax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     invlpga
+# CHECK-NEXT:  -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     lahf
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leave
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     lodsb	(%rsi), %al
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     lodsw	(%rsi), %ax
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     lodsl	(%rsi), %eax
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     lodsq	(%rsi), %rax
+# CHECK-NEXT:  -      -      -     0.50    -      -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     loop	0
+# CHECK-NEXT:  -      -      -     0.50    -      -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     loope	0
+# CHECK-NEXT:  -      -      -     0.50    -      -      -     0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     loopne	0
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movsb	(%rsi), %es:(%rdi)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movsw	(%rsi), %es:(%rdi)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movsl	(%rsi), %es:(%rdi)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movsq	(%rsi), %es:(%rdi)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movsbw	%al, %di
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movzbw	%al, %di
+# CHECK-NEXT: 1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movsbw	(%rax), %di
+# CHECK-NEXT: 1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movzbw	(%rax), %di
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movsbl	%al, %edi
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movzbl	%al, %edi
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movsbl	(%rax), %edi
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movzbl	(%rax), %edi
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movsbq	%al, %rdi
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movzbq	%al, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movsbq	(%rax), %rdi
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movzbq	(%rax), %rdi
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movswl	%ax, %edi
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movzwl	%ax, %edi
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movswl	(%rax), %edi
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movzwl	(%rax), %edi
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movswq	%ax, %rdi
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movzwq	%ax, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movswq	(%rax), %rdi
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movzwq	(%rax), %rdi
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     movslq	%eax, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     movslq	(%rax), %rdi
+# CHECK-NEXT:  -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     mulb	%dil
+# CHECK-NEXT: 0.33   0.33   0.33    -     3.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     mulb	(%rax)
+# CHECK-NEXT:  -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     mulw	%si
+# CHECK-NEXT: 0.33   0.33   0.33    -     3.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     mulw	(%rax)
+# CHECK-NEXT:  -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     mull	%edx
+# CHECK-NEXT: 0.33   0.33   0.33    -     3.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     mull	(%rax)
+# CHECK-NEXT:  -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     mulq	%rcx
+# CHECK-NEXT: 0.33   0.33   0.33    -     3.00    -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     mulq	(%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     negb	%dil
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   negb	(%r8)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		negb	(%r8)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     negw	%si
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   negw	(%r9)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		negw	(%r9)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     negl	%edx
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   negl	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		negl	(%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     negq	%rcx
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   negq	(%r10)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		negq	(%r10)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     nop
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     nopw	%di
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     nopw	(%rcx)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     nopl	%esi
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     nopl	(%r8)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     nopq	%rdx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     nopq	(%r9)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     notb	%dil
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   notb	(%r8)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		notb	(%r8)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     notw	%si
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   notw	(%r9)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		notw	(%r9)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     notl	%edx
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   notl	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		notl	(%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     notq	%rcx
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   notq	(%r10)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		notq	(%r10)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     orb	$7, %al
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     orb	$7, %dil
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   orb	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		orb	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     orb	%sil, %dil
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   orb	%sil, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		orb	%sil, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     orb	(%rax), %dil
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     orw	$511, %ax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     orw	$511, %di
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   orw	$511, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		orw	$511, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     orw	$7, %di
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   orw	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		orw	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     orw	%si, %di
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   orw	%si, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		orw	%si, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     orw	(%rax), %di
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     orl	$665536, %eax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     orl	$665536, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   orl	$665536, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		orl	$665536, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     orl	$7, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   orl	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		orl	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     orl	%esi, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   orl	%esi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		orl	%esi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     orl	(%rax), %edi
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     orq	$665536, %rax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     orq	$665536, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   orq	$665536, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		orq	$665536, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     orq	$7, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   orq	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		orq	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     orq	%rsi, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   orq	%rsi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		orq	%rsi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     orq	(%rax), %rdi
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     outb	%al, $7
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     outb	%al, %dx
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     outw	%ax, $7
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     outw	%ax, %dx
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     outl	%eax, $7
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     outl	%eax, %dx
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     outsb	(%rsi), %dx
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     outsw	(%rsi), %dx
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     outsl	(%rsi), %dx
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     pause
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rclb	%dil
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rcrb	%dil
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00   1.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rclb	(%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00   1.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rcrb	(%rax)
+# CHECK-NEXT:  -      -      -      -     4.00   4.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rclb	$7, %dil
+# CHECK-NEXT:  -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rcrb	$7, %dil
+# CHECK-NEXT: 0.33   0.33   0.33    -     4.00   4.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rclb	$7, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -     4.00   4.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rcrb	$7, (%rax)
+# CHECK-NEXT:  -      -      -      -     4.00   4.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rclb	%cl, %dil
+# CHECK-NEXT:  -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rcrb	%cl, %dil
+# CHECK-NEXT: 0.33   0.33   0.33    -     4.00   4.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rclb	%cl, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -     4.00   4.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rcrb	%cl, (%rax)
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rclw	%di
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rcrw	%di
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00   1.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rclw	(%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00   1.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rcrw	(%rax)
+# CHECK-NEXT:  -      -      -      -     4.00   4.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rclw	$7, %di
+# CHECK-NEXT:  -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rcrw	$7, %di
+# CHECK-NEXT: 0.33   0.33   0.33    -     4.00   4.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rclw	$7, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -     4.00   4.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rcrw	$7, (%rax)
+# CHECK-NEXT:  -      -      -      -     4.00   4.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rclw	%cl, %di
+# CHECK-NEXT:  -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rcrw	%cl, %di
+# CHECK-NEXT: 0.33   0.33   0.33    -     4.00   4.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rclw	%cl, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -     4.00   4.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rcrw	%cl, (%rax)
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rcll	%edi
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rcrl	%edi
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00   1.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rcll	(%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00   1.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rcrl	(%rax)
+# CHECK-NEXT:  -      -      -      -     4.00   4.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rcll	$7, %edi
+# CHECK-NEXT:  -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rcrl	$7, %edi
+# CHECK-NEXT: 0.33   0.33   0.33    -     4.00   4.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rcll	$7, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -     4.00   4.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rcrl	$7, (%rax)
+# CHECK-NEXT:  -      -      -      -     4.00   4.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rcll	%cl, %edi
+# CHECK-NEXT:  -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rcrl	%cl, %edi
+# CHECK-NEXT: 0.33   0.33   0.33    -     4.00   4.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rcll	%cl, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -     4.00   4.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rcrl	%cl, (%rax)
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rclq	%rdi
+# CHECK-NEXT:  -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rcrq	%rdi
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00   1.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rclq	(%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -     1.00   1.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rcrq	(%rax)
+# CHECK-NEXT:  -      -      -      -     4.00   4.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rclq	$7, %rdi
+# CHECK-NEXT:  -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rcrq	$7, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33    -     4.00   4.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rclq	$7, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -     4.00   4.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rcrq	$7, (%rax)
+# CHECK-NEXT:  -      -      -      -     4.00   4.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rclq	%cl, %rdi
+# CHECK-NEXT:  -      -      -      -     3.00   3.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rcrq	%cl, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33    -     4.00   4.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rclq	%cl, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -     4.00   4.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     rcrq	%cl, (%rax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rdmsr
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rdpmc
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rdtsc
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rdtscp
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rolb	%dil
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rorb	%dil
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rolb	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rorb	(%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rolb	$7, %dil
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rorb	$7, %dil
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rolb	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rorb	$7, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rolb	%cl, %dil
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rorb	%cl, %dil
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rolb	%cl, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rorb	%cl, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rolw	%di
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rorw	%di
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rolw	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rorw	(%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rolw	$7, %di
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rorw	$7, %di
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rolw	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rorw	$7, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rolw	%cl, %di
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rorw	%cl, %di
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rolw	%cl, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rorw	%cl, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     roll	%edi
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rorl	%edi
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   roll	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rorl	(%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     roll	$7, %edi
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rorl	$7, %edi
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   roll	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rorl	$7, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     roll	%cl, %edi
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rorl	%cl, %edi
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   roll	%cl, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rorl	%cl, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rolq	%rdi
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rorq	%rdi
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rolq	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rorq	(%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rolq	$7, %rdi
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rorq	$7, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rolq	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rorq	$7, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rolq	%cl, %rdi
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     rorq	%cl, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rolq	%cl, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   rorq	%cl, (%rax)
+# CHECK-NEXT:  -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sahf
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sarb	%dil
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shlb	%dil
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrb	%dil
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sarb	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shlb	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shrb	(%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sarb	$7, %dil
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shlb	$7, %dil
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrb	$7, %dil
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sarb	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shlb	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shrb	$7, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sarb	%cl, %dil
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shlb	%cl, %dil
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrb	%cl, %dil
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sarb	%cl, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shlb	%cl, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shrb	%cl, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sarw	%di
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shlw	%di
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrw	%di
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sarw	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shlw	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shrw	(%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sarw	$7, %di
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shlw	$7, %di
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrw	$7, %di
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sarw	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shlw	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shrw	$7, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sarw	%cl, %di
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shlw	%cl, %di
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrw	%cl, %di
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sarw	%cl, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shlw	%cl, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shrw	%cl, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sarl	%edi
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shll	%edi
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrl	%edi
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sarl	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shll	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shrl	(%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sarl	$7, %edi
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shll	$7, %edi
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrl	$7, %edi
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sarl	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shll	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shrl	$7, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sarl	%cl, %edi
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shll	%cl, %edi
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrl	%cl, %edi
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sarl	%cl, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shll	%cl, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shrl	%cl, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sarq	%rdi
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shlq	%rdi
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrq	%rdi
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sarq	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shlq	(%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shrq	(%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sarq	$7, %rdi
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shlq	$7, %rdi
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrq	$7, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sarq	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shlq	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shrq	$7, (%rax)
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sarq	%cl, %rdi
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shlq	%cl, %rdi
+# CHECK-NEXT:  -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrq	%cl, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sarq	%cl, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shlq	%cl, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67    -     0.50   0.50    -      -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   shrq	%cl, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbb	$0, %al
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbb	$0, %dil
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sbbb	$0, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		sbbb	$0, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbb	$7, %al
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbb	$7, %dil
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sbbb	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		sbbb	$7, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbb	%sil, %dil
+# CHECK-NEXT: 0.33   0.33   0.33   1.75   1.75   1.75   1.75    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sbbb	%sil, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   1.75   1.75   1.75   1.75    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		sbbb	%sil, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     sbbb	(%rax), %dil
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbw	$0, %ax
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbw	$0, %di
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sbbw	$0, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		sbbw	$0, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbw	$511, %ax
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbw	$511, %di
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sbbw	$511, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		sbbw	$511, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbw	$7, %di
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sbbw	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		sbbw	$7, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbw	%si, %di
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sbbw	%si, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		sbbw	%si, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     sbbw	(%rax), %di
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbl	$0, %eax
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbl	$0, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sbbl	$0, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		sbbl	$0, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbl	$665536, %eax
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbl	$665536, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sbbl	$665536, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		sbbl	$665536, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbl	$7, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sbbl	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		sbbl	$7, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbl	%esi, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sbbl	%esi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		sbbl	%esi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     sbbl	(%rax), %edi
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbq	$0, %rax
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbq	$0, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sbbq	$0, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		sbbq	$0, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbq	$665536, %rax
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbq	$665536, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sbbq	$665536, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		sbbq	$665536, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbq	$7, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sbbq	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		sbbq	$7, (%rax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sbbq	%rsi, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   sbbq	%rsi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		sbbq	%rsi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     sbbq	(%rax), %rdi
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     scasb	%es:(%rdi), %al
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     scasw	%es:(%rdi), %ax
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     scasl	%es:(%rdi), %eax
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     scasq	%es:(%rdi), %rax
+# CHECK-NEXT:  -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     seto	%al
+# CHECK-NEXT: 0.33   0.33   0.33   1.00    -      -     1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   seto	(%rax)
+# CHECK-NEXT:  -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     setno	%al
+# CHECK-NEXT: 0.33   0.33   0.33   1.00    -      -     1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   setno	(%rax)
+# CHECK-NEXT:  -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     setb	%al
+# CHECK-NEXT: 0.33   0.33   0.33   1.00    -      -     1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   setb	(%rax)
+# CHECK-NEXT:  -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     setae	%al
+# CHECK-NEXT: 0.33   0.33   0.33   1.00    -      -     1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   setae	(%rax)
+# CHECK-NEXT:  -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sete	%al
+# CHECK-NEXT: 0.33   0.33   0.33   1.00    -      -     1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   sete	(%rax)
+# CHECK-NEXT:  -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     setne	%al
+# CHECK-NEXT: 0.33   0.33   0.33   1.00    -      -     1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   setne	(%rax)
+# CHECK-NEXT:  -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     seta	%al
+# CHECK-NEXT: 0.33   0.33   0.33   1.00    -      -     1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   seta	(%rax)
+# CHECK-NEXT:  -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     setbe	%al
+# CHECK-NEXT: 0.33   0.33   0.33   1.00    -      -     1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   setbe	(%rax)
+# CHECK-NEXT:  -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     sets	%al
+# CHECK-NEXT: 0.33   0.33   0.33   1.00    -      -     1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   sets	(%rax)
+# CHECK-NEXT:  -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     setns	%al
+# CHECK-NEXT: 0.33   0.33   0.33   1.00    -      -     1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   setns	(%rax)
+# CHECK-NEXT:  -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     setp	%al
+# CHECK-NEXT: 0.33   0.33   0.33   1.00    -      -     1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   setp	(%rax)
+# CHECK-NEXT:  -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     setnp	%al
+# CHECK-NEXT: 0.33   0.33   0.33   1.00    -      -     1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   setnp	(%rax)
+# CHECK-NEXT:  -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     setl	%al
+# CHECK-NEXT: 0.33   0.33   0.33   1.00    -      -     1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   setl	(%rax)
+# CHECK-NEXT:  -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     setge	%al
+# CHECK-NEXT: 0.33   0.33   0.33   1.00    -      -     1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   setge	(%rax)
+# CHECK-NEXT:  -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     setg	%al
+# CHECK-NEXT: 0.33   0.33   0.33   1.00    -      -     1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   setg	(%rax)
+# CHECK-NEXT:  -      -      -     1.00    -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     setle	%al
+# CHECK-NEXT: 0.33   0.33   0.33   1.00    -      -     1.00    -      -      -      -      -      -      -      -     0.33   0.33   0.33    -      -      -     0.50   0.50   setle	(%rax)
+# CHECK-NEXT:  -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shldw	%cl, %si, %di
+# CHECK-NEXT:  -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrdw	%cl, %si, %di
+# CHECK-NEXT: 0.33   0.33   0.33    -     2.00   2.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     shldw	%cl, %si, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -     2.00   2.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     shrdw	%cl, %si, (%rax)
+# CHECK-NEXT:  -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shldw	$7, %si, %di
+# CHECK-NEXT:  -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrdw	$7, %si, %di
+# CHECK-NEXT: 0.33   0.33   0.33    -     2.00   2.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     shldw	$7, %si, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -     2.00   2.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     shrdw	$7, %si, (%rax)
+# CHECK-NEXT:  -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shldl	%cl, %esi, %edi
+# CHECK-NEXT:  -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrdl	%cl, %esi, %edi
+# CHECK-NEXT: 0.33   0.33   0.33    -     2.00   2.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     shldl	%cl, %esi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -     2.00   2.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     shrdl	%cl, %esi, (%rax)
+# CHECK-NEXT:  -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shldl	$7, %esi, %edi
+# CHECK-NEXT:  -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrdl	$7, %esi, %edi
+# CHECK-NEXT: 0.33   0.33   0.33    -     2.00   2.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     shldl	$7, %esi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -     2.00   2.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     shrdl	$7, %esi, (%rax)
+# CHECK-NEXT:  -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shldq	%cl, %rsi, %rdi
+# CHECK-NEXT:  -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrdq	%cl, %rsi, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33    -     2.00   2.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     shldq	%cl, %rsi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -     2.00   2.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     shrdq	%cl, %rsi, (%rax)
+# CHECK-NEXT:  -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shldq	$7, %rsi, %rdi
+# CHECK-NEXT:  -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     shrdq	$7, %rsi, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33    -     2.00   2.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     shldq	$7, %rsi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33    -     2.00   2.00    -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     shrdq	$7, %rsi, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     stc
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     std
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     stosb	%al, %es:(%rdi)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     stosw	%ax, %es:(%rdi)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     stosl	%eax, %es:(%rdi)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     stosq	%rax, %es:(%rdi)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     subb	$7, %al
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     subb	$7, %dil
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   subb	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		subb	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     subb	%sil, %dil
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   subb	%sil, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		subb	%sil, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     subb	(%rax), %dil
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     subw	$511, %ax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     subw	$511, %di
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   subw	$511, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		subw	$511, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     subw	$7, %di
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   subw	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		subw	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     subw	%si, %di
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   subw	%si, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		subw	%si, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     subw	(%rax), %di
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     subl	$665536, %eax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     subl	$665536, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   subl	$665536, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		subl	$665536, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     subl	$7, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   subl	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		subl	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     subl	%esi, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   subl	%esi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		subl	%esi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     subl	(%rax), %edi
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     subq	$665536, %rax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     subq	$665536, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   subq	$665536, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		subq	$665536, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     subq	$7, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   subq	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		subq	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     subq	%rsi, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   subq	%rsi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		subq	%rsi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     subq	(%rax), %rdi
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     testb	$7, %al
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     testb	$7, %dil
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     testb	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     testb	%sil, %dil
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     testb	%sil, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     testw	$511, %ax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     testw	$511, %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     testw	$511, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     testw	$7, %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     testw	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     testw	%si, %di
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     testw	%si, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     testl	$665536, %eax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     testl	$665536, %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     testl	$665536, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     testl	$7, %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     testl	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     testl	%esi, %edi
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     testl	%esi, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     testq	$665536, %rax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     testq	$665536, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     testq	$665536, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     testq	$7, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     testq	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     testq	%rsi, %rdi
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     testq	%rsi, (%rax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     ud2
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     wrmsr
+# CHECK-NEXT:  -      -      -     2.00   2.00   2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xaddb	%bl, %cl
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   xaddb	%bl, (%rcx)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		xaddb	%bl, (%rcx)
+# CHECK-NEXT:  -      -      -     2.00   2.00   2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xaddw	%bx, %cx
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   xaddw	%ax, (%rbx)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		xaddw	%ax, (%rbx)
+# CHECK-NEXT:  -      -      -     2.00   2.00   2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xaddl	%ebx, %ecx
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   xaddl	%eax, (%rbx)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		xaddl	%eax, (%rbx)
+# CHECK-NEXT:  -      -      -     2.00   2.00   2.00   2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xaddq	%rbx, %rcx
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   xaddq	%rax, (%rbx)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		xaddq	%rax, (%rbx)
+# CHECK-NEXT:  -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xchgb	%bl, %cl
+# CHECK-NEXT: 0.33   0.33   0.33   0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     xchgb	%bl, (%rbx)
+# CHECK-NEXT: 0.33   0.33   0.33   0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     lock		xchgb	%bl, (%rbx)
+# CHECK-NEXT:  -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xchgw	%bx, %ax
+# CHECK-NEXT:  -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xchgw	%bx, %cx
+# CHECK-NEXT: 0.33   0.33   0.33   0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     xchgw	%ax, (%rbx)
+# CHECK-NEXT: 0.33   0.33   0.33   0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     lock		xchgw	%ax, (%rbx)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xchgl	%ebx, %eax
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xchgl	%ebx, %ecx
+# CHECK-NEXT: 0.33   0.33   0.33   0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     xchgl	%eax, (%rbx)
+# CHECK-NEXT: 0.33   0.33   0.33   0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     lock		xchgl	%eax, (%rbx)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xchgq	%rbx, %rax
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xchgq	%rbx, %rcx
+# CHECK-NEXT: 0.33   0.33   0.33   0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     xchgq	%rax, (%rbx)
+# CHECK-NEXT: 0.33   0.33   0.33   0.50   0.50   0.50   0.50    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     lock		xchgq	%rax, (%rbx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     xlatb
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xorb	$7, %al
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xorb	$7, %dil
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   xorb	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		xorb	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xorb	%sil, %dil
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   xorb	%sil, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		xorb	%sil, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     xorb	(%rax), %dil
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xorw	$511, %ax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xorw	$511, %di
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   xorw	$511, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		xorw	$511, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xorw	$7, %di
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   xorw	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		xorw	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xorw	%si, %di
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   xorw	%si, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		xorw	%si, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     xorw	(%rax), %di
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xorl	$665536, %eax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xorl	$665536, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   xorl	$665536, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		xorl	$665536, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xorl	$7, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   xorl	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		xorl	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xorl	%esi, %edi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   xorl	%esi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		xorl	%esi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     xorl	(%rax), %edi
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xorq	$665536, %rax
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xorq	$665536, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   xorq	$665536, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		xorq	$665536, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xorq	$7, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   xorq	$7, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		xorq	$7, (%rax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xorq	%rsi, %rdi
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   xorq	%rsi, (%rax)
+# CHECK-NEXT: 0.67   0.67   0.67   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.67   0.67   0.67   0.33   0.33   0.33   0.50   0.50   lock		xorq	%rsi, (%rax)
+# CHECK-NEXT: 0.33   0.33   0.33   0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     xorq	(%rax), %rdi
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-x87.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-x87.s
new file mode 100644
index 0000000000000..f366a4798e6bb
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-x87.s
@@ -0,0 +1,536 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+f2xm1
+
+fabs
+
+fadd %st, %st(1)
+fadd %st(2)
+fadds (%ecx)
+faddl (%ecx)
+faddp %st(1)
+faddp %st(2)
+fiadds (%ecx)
+fiaddl (%ecx)
+
+fbld (%ecx)
+fbstp (%eax)
+
+fchs
+
+fnclex
+
+fcmovb %st(1), %st
+fcmovbe %st(1), %st
+fcmove %st(1), %st
+fcmovnb %st(1), %st
+fcmovnbe %st(1), %st
+fcmovne %st(1), %st
+fcmovnu %st(1), %st
+fcmovu %st(1), %st
+
+fcom %st(1)
+fcom %st(3)
+fcoms (%ecx)
+fcoml (%eax)
+fcomp %st(1)
+fcomp %st(3)
+fcomps (%ecx)
+fcompl (%eax)
+fcompp
+
+fcomi %st(3)
+fcompi %st(3)
+
+fcos
+
+fdecstp
+
+fdiv %st, %st(1)
+fdiv %st(2)
+fdivs (%ecx)
+fdivl (%eax)
+fdivp %st(1)
+fdivp %st(2)
+fidivs (%ecx)
+fidivl (%eax)
+
+fdivr %st, %st(1)
+fdivr %st(2)
+fdivrs (%ecx)
+fdivrl (%eax)
+fdivrp %st(1)
+fdivrp %st(2)
+fidivrs (%ecx)
+fidivrl (%eax)
+
+ffree %st(0)
+
+ficoms (%ecx)
+ficoml (%eax)
+ficomps (%ecx)
+ficompl (%eax)
+
+filds (%edx)
+fildl (%ecx)
+fildll (%eax)
+
+fincstp
+
+fninit
+
+fists (%edx)
+fistl (%ecx)
+fistps (%edx)
+fistpl (%ecx)
+fistpll (%eax)
+
+fisttps (%edx)
+fisttpl (%ecx)
+fisttpll (%eax)
+
+fld %st(0)
+flds (%edx)
+fldl (%ecx)
+fldt (%eax)
+
+fldcw (%eax)
+fldenv (%eax)
+
+fld1
+fldl2e
+fldl2t
+fldlg2
+fldln2
+fldpi
+fldz
+
+fmul %st, %st(1)
+fmul %st(2)
+fmuls (%ecx)
+fmull (%eax)
+fmulp %st(1)
+fmulp %st(2)
+fimuls (%ecx)
+fimull (%eax)
+
+fnop
+
+fpatan
+
+fprem
+fprem1
+
+fptan
+
+frndint
+
+frstor (%eax)
+
+fnsave (%eax)
+
+fscale
+
+fsin
+
+fsincos
+
+fsqrt
+
+fst %st(0)
+fsts (%edx)
+fstl (%ecx)
+fstp %st(0)
+fstpl (%edx)
+fstpl (%ecx)
+fstpt (%eax)
+
+fnstcw (%eax)
+fnstenv (%eax)
+fnstsw (%eax)
+
+frstor (%eax)
+fsave (%eax)
+
+fsub %st, %st(1)
+fsub %st(2)
+fsubs (%ecx)
+fsubl (%eax)
+fsubp %st(1)
+fsubp %st(2)
+fisubs (%ecx)
+fisubl (%eax)
+
+fsubr %st, %st(1)
+fsubr %st(2)
+fsubrs (%ecx)
+fsubrl (%eax)
+fsubrp %st(1)
+fsubrp %st(2)
+fisubrs (%ecx)
+fisubrl (%eax)
+
+ftst
+
+fucom %st(1)
+fucom %st(3)
+fucomp %st(1)
+fucomp %st(3)
+fucompp
+
+fucomi %st(3)
+fucompi %st(3)
+
+fwait
+
+fxam
+
+fxch %st(1)
+fxch %st(3)
+
+fxrstor (%eax)
+fxsave (%eax)
+
+fxtract
+
+fyl2x
+fyl2xp1
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  100    100   25.00                 U     f2xm1
+# CHECK-NEXT:  1      1     1.00                  U     fabs
+# CHECK-NEXT:  1      3     0.50                  U     fadd	%st, %st(1)
+# CHECK-NEXT:  1      3     0.50                  U     fadd	%st(2), %st
+# CHECK-NEXT:  1      10    0.50    *             U     fadds	(%ecx)
+# CHECK-NEXT:  1      10    0.50    *             U     faddl	(%ecx)
+# CHECK-NEXT:  1      3     0.50                  U     faddp	%st, %st(1)
+# CHECK-NEXT:  1      3     0.50                  U     faddp	%st, %st(2)
+# CHECK-NEXT:  2      5     6.00    *             U     fiadds	(%ecx)
+# CHECK-NEXT:  2      5     6.00    *             U     fiaddl	(%ecx)
+# CHECK-NEXT:  100    100   25.00   *             U     fbld	(%ecx)
+# CHECK-NEXT:  100    100   25.00          *      U     fbstp	(%eax)
+# CHECK-NEXT:  1      1     1.00                  U     fchs
+# CHECK-NEXT:  100    100   25.00                 U     fnclex
+# CHECK-NEXT:  7      7     7.00                  U     fcmovb	%st(1), %st
+# CHECK-NEXT:  7      7     7.00                  U     fcmovbe	%st(1), %st
+# CHECK-NEXT:  7      7     7.00                  U     fcmove	%st(1), %st
+# CHECK-NEXT:  7      7     7.00                  U     fcmovnb	%st(1), %st
+# CHECK-NEXT:  7      7     7.00                  U     fcmovnbe	%st(1), %st
+# CHECK-NEXT:  7      7     7.00                  U     fcmovne	%st(1), %st
+# CHECK-NEXT:  7      7     7.00                  U     fcmovnu	%st(1), %st
+# CHECK-NEXT:  7      7     7.00                  U     fcmovu	%st(1), %st
+# CHECK-NEXT:  1      3     1.00                  U     fcom	%st(1)
+# CHECK-NEXT:  1      3     1.00                  U     fcom	%st(3)
+# CHECK-NEXT:  1      10    1.00    *             U     fcoms	(%ecx)
+# CHECK-NEXT:  1      10    1.00    *             U     fcoml	(%eax)
+# CHECK-NEXT:  1      3     1.00                  U     fcomp	%st(1)
+# CHECK-NEXT:  1      3     1.00                  U     fcomp	%st(3)
+# CHECK-NEXT:  1      10    1.00    *             U     fcomps	(%ecx)
+# CHECK-NEXT:  1      10    1.00    *             U     fcompl	(%eax)
+# CHECK-NEXT:  100    100   25.00                 U     fcompp
+# CHECK-NEXT:  1      3     1.00                  U     fcomi	%st(3), %st
+# CHECK-NEXT:  1      3     1.00                  U     fcompi	%st(3), %st
+# CHECK-NEXT:  100    100   25.00                 U     fcos
+# CHECK-NEXT:  100    100   25.00                 U     fdecstp
+# CHECK-NEXT:  1      11    3.00                  U     fdiv	%st, %st(1)
+# CHECK-NEXT:  1      11    3.00                  U     fdiv	%st(2), %st
+# CHECK-NEXT:  1      18    3.00    *             U     fdivs	(%ecx)
+# CHECK-NEXT:  1      18    3.00    *             U     fdivl	(%eax)
+# CHECK-NEXT:  1      11    3.00                  U     fdivp	%st, %st(1)
+# CHECK-NEXT:  1      11    3.00                  U     fdivp	%st, %st(2)
+# CHECK-NEXT:  2      5     15.50   *             U     fidivs	(%ecx)
+# CHECK-NEXT:  2      5     15.50   *             U     fidivl	(%eax)
+# CHECK-NEXT:  1      11    3.00                  U     fdivr	%st, %st(1)
+# CHECK-NEXT:  1      11    3.00                  U     fdivr	%st(2), %st
+# CHECK-NEXT:  1      18    3.00    *             U     fdivrs	(%ecx)
+# CHECK-NEXT:  1      18    3.00    *             U     fdivrl	(%eax)
+# CHECK-NEXT:  1      11    3.00                  U     fdivrp	%st, %st(1)
+# CHECK-NEXT:  1      11    3.00                  U     fdivrp	%st, %st(2)
+# CHECK-NEXT:  2      5     15.50   *             U     fidivrs	(%ecx)
+# CHECK-NEXT:  2      5     15.50   *             U     fidivrl	(%eax)
+# CHECK-NEXT:  100    100   25.00                 U     ffree	%st(0)
+# CHECK-NEXT:  1      10    1.00    *             U     ficoms	(%ecx)
+# CHECK-NEXT:  1      10    1.00    *             U     ficoml	(%eax)
+# CHECK-NEXT:  1      10    1.00    *             U     ficomps	(%ecx)
+# CHECK-NEXT:  1      10    1.00    *             U     ficompl	(%eax)
+# CHECK-NEXT:  1      5     0.33    *             U     filds	(%edx)
+# CHECK-NEXT:  1      5     0.33    *             U     fildl	(%ecx)
+# CHECK-NEXT:  1      5     0.33    *             U     fildll	(%eax)
+# CHECK-NEXT:  100    100   25.00                 U     fincstp
+# CHECK-NEXT:  100    100   25.00                 U     fninit
+# CHECK-NEXT:  1      1     1.00           *      U     fists	(%edx)
+# CHECK-NEXT:  1      1     1.00           *      U     fistl	(%ecx)
+# CHECK-NEXT:  1      1     1.00           *      U     fistps	(%edx)
+# CHECK-NEXT:  1      1     1.00           *      U     fistpl	(%ecx)
+# CHECK-NEXT:  1      1     1.00           *      U     fistpll	(%eax)
+# CHECK-NEXT:  1      1     1.00           *      U     fisttps	(%edx)
+# CHECK-NEXT:  1      1     1.00           *      U     fisttpl	(%ecx)
+# CHECK-NEXT:  1      1     1.00           *      U     fisttpll	(%eax)
+# CHECK-NEXT:  1      1     1.00                  U     fld	%st(0)
+# CHECK-NEXT:  1      5     0.33    *             U     flds	(%edx)
+# CHECK-NEXT:  1      5     0.33    *             U     fldl	(%ecx)
+# CHECK-NEXT:  1      5     0.33    *             U     fldt	(%eax)
+# CHECK-NEXT:  1      5     0.33    *             U     fldcw	(%eax)
+# CHECK-NEXT:  100    100   25.00   *             U     fldenv	(%eax)
+# CHECK-NEXT:  1      11    1.00                  U     fld1
+# CHECK-NEXT:  1      11    1.00                  U     fldl2e
+# CHECK-NEXT:  1      11    1.00                  U     fldl2t
+# CHECK-NEXT:  1      11    1.00                  U     fldlg2
+# CHECK-NEXT:  1      11    1.00                  U     fldln2
+# CHECK-NEXT:  1      11    1.00                  U     fldpi
+# CHECK-NEXT:  1      8     1.00                  U     fldz
+# CHECK-NEXT:  1      3     0.50                  U     fmul	%st, %st(1)
+# CHECK-NEXT:  1      3     0.50                  U     fmul	%st(2), %st
+# CHECK-NEXT:  1      10    0.50    *             U     fmuls	(%ecx)
+# CHECK-NEXT:  1      10    0.50    *             U     fmull	(%eax)
+# CHECK-NEXT:  1      3     0.50                  U     fmulp	%st, %st(1)
+# CHECK-NEXT:  1      3     0.50                  U     fmulp	%st, %st(2)
+# CHECK-NEXT:  2      5     6.00    *             U     fimuls	(%ecx)
+# CHECK-NEXT:  2      5     6.00    *             U     fimull	(%eax)
+# CHECK-NEXT:  1      0     0.25                  U     fnop
+# CHECK-NEXT:  100    100   25.00                 U     fpatan
+# CHECK-NEXT:  100    100   25.00                 U     fprem
+# CHECK-NEXT:  100    100   25.00                 U     fprem1
+# CHECK-NEXT:  100    100   25.00                 U     fptan
+# CHECK-NEXT:  100    100   25.00                 U     frndint
+# CHECK-NEXT:  100    100   25.00   *             U     frstor	(%eax)
+# CHECK-NEXT:  100    100   25.00          *      U     fnsave	(%eax)
+# CHECK-NEXT:  100    100   25.00                 U     fscale
+# CHECK-NEXT:  100    100   25.00                 U     fsin
+# CHECK-NEXT:  100    100   25.00                 U     fsincos
+# CHECK-NEXT:  1      22    23.00                 U     fsqrt
+# CHECK-NEXT:  1      1     1.00                  U     fst	%st(0)
+# CHECK-NEXT:  1      1     1.00           *      U     fsts	(%edx)
+# CHECK-NEXT:  1      1     1.00           *      U     fstl	(%ecx)
+# CHECK-NEXT:  1      1     1.00                  U     fstp	%st(0)
+# CHECK-NEXT:  1      1     1.00           *      U     fstpl	(%edx)
+# CHECK-NEXT:  1      1     1.00           *      U     fstpl	(%ecx)
+# CHECK-NEXT:  1      1     1.00           *      U     fstpt	(%eax)
+# CHECK-NEXT:  1      1     0.25           *      U     fnstcw	(%eax)
+# CHECK-NEXT:  100    100   25.00          *      U     fnstenv	(%eax)
+# CHECK-NEXT:  100    100   25.00          *      U     fnstsw	(%eax)
+# CHECK-NEXT:  100    100   25.00   *             U     frstor	(%eax)
+# CHECK-NEXT:  100    100   25.00                 U     wait
+# CHECK-NEXT:  100    100   25.00          *      U     fnsave	(%eax)
+# CHECK-NEXT:  1      3     0.50                  U     fsub	%st, %st(1)
+# CHECK-NEXT:  1      3     0.50                  U     fsub	%st(2), %st
+# CHECK-NEXT:  1      10    0.50    *             U     fsubs	(%ecx)
+# CHECK-NEXT:  1      10    0.50    *             U     fsubl	(%eax)
+# CHECK-NEXT:  1      3     0.50                  U     fsubp	%st, %st(1)
+# CHECK-NEXT:  1      3     0.50                  U     fsubp	%st, %st(2)
+# CHECK-NEXT:  2      5     6.00    *             U     fisubs	(%ecx)
+# CHECK-NEXT:  2      5     6.00    *             U     fisubl	(%eax)
+# CHECK-NEXT:  1      3     0.50                  U     fsubr	%st, %st(1)
+# CHECK-NEXT:  1      3     0.50                  U     fsubr	%st(2), %st
+# CHECK-NEXT:  1      10    0.50    *             U     fsubrs	(%ecx)
+# CHECK-NEXT:  1      10    0.50    *             U     fsubrl	(%eax)
+# CHECK-NEXT:  1      3     0.50                  U     fsubrp	%st, %st(1)
+# CHECK-NEXT:  1      3     0.50                  U     fsubrp	%st, %st(2)
+# CHECK-NEXT:  2      5     6.00    *             U     fisubrs	(%ecx)
+# CHECK-NEXT:  2      5     6.00    *             U     fisubrl	(%eax)
+# CHECK-NEXT:  1      3     1.00                  U     ftst
+# CHECK-NEXT:  1      3     1.00                  U     fucom	%st(1)
+# CHECK-NEXT:  1      3     1.00                  U     fucom	%st(3)
+# CHECK-NEXT:  1      3     1.00                  U     fucomp	%st(1)
+# CHECK-NEXT:  1      3     1.00                  U     fucomp	%st(3)
+# CHECK-NEXT:  1      3     1.00                  U     fucompp
+# CHECK-NEXT:  1      3     1.00                  U     fucomi	%st(3), %st
+# CHECK-NEXT:  1      3     1.00                  U     fucompi	%st(3), %st
+# CHECK-NEXT:  100    100   25.00                 U     wait
+# CHECK-NEXT:  2      1     1.00                  U     fxam
+# CHECK-NEXT:  1      1     1.00                  U     fxch	%st(1)
+# CHECK-NEXT:  1      1     1.00                  U     fxch	%st(3)
+# CHECK-NEXT:  100    100   25.00   *      *      U     fxrstor	(%eax)
+# CHECK-NEXT:  100    100   25.00   *      *      U     fxsave	(%eax)
+# CHECK-NEXT:  100    100   25.00                 U     fxtract
+# CHECK-NEXT:  100    100   25.00                 U     fyl2x
+# CHECK-NEXT:  100    100   25.00                 U     fyl2xp1
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT: 10.67  10.67  10.67  861.50 861.50 861.50 861.50  -     137.00 204.00 120.00 119.00 13.50  13.50   -     24.00  24.00  24.00  15.33  15.33  15.33  13.00  13.00
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     f2xm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fabs
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     fadd	%st, %st(1)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     fadd	%st(2), %st
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fadds	(%ecx)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     faddl	(%ecx)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     faddp	%st, %st(1)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     faddp	%st, %st(2)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -     6.00   6.00   6.00   6.00    -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fiadds	(%ecx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -     6.00   6.00   6.00   6.00    -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fiaddl	(%ecx)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fbld	(%ecx)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fbstp	(%eax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fchs
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fnclex
+# CHECK-NEXT:  -      -      -     7.00   7.00   7.00   7.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fcmovb	%st(1), %st
+# CHECK-NEXT:  -      -      -     7.00   7.00   7.00   7.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fcmovbe	%st(1), %st
+# CHECK-NEXT:  -      -      -     7.00   7.00   7.00   7.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fcmove	%st(1), %st
+# CHECK-NEXT:  -      -      -     7.00   7.00   7.00   7.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fcmovnb	%st(1), %st
+# CHECK-NEXT:  -      -      -     7.00   7.00   7.00   7.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fcmovnbe	%st(1), %st
+# CHECK-NEXT:  -      -      -     7.00   7.00   7.00   7.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fcmovne	%st(1), %st
+# CHECK-NEXT:  -      -      -     7.00   7.00   7.00   7.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fcmovnu	%st(1), %st
+# CHECK-NEXT:  -      -      -     7.00   7.00   7.00   7.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fcmovu	%st(1), %st
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fcom	%st(1)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fcom	%st(3)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fcoms	(%ecx)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fcoml	(%eax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fcomp	%st(1)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fcomp	%st(3)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fcomps	(%ecx)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fcompl	(%eax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fcompp
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fcomi	%st(3), %st
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fcompi	%st(3), %st
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fcos
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fdecstp
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fdiv	%st, %st(1)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fdiv	%st(2), %st
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fdivs	(%ecx)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fdivl	(%eax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fdivp	%st, %st(1)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fdivp	%st, %st(2)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -     15.50  15.50  15.50  15.50   -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fidivs	(%ecx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -     15.50  15.50  15.50  15.50   -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fidivl	(%eax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fdivr	%st, %st(1)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fdivr	%st(2), %st
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fdivrs	(%ecx)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fdivrl	(%eax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fdivrp	%st, %st(1)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     3.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fdivrp	%st, %st(2)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -     15.50  15.50  15.50  15.50   -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fidivrs	(%ecx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -     15.50  15.50  15.50  15.50   -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fidivrl	(%eax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     ffree	%st(0)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     ficoms	(%ecx)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     ficoml	(%eax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     ficomps	(%ecx)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     ficompl	(%eax)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     filds	(%edx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fildl	(%ecx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fildll	(%eax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fincstp
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fninit
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -     1.00   1.00   fists	(%edx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -     1.00   1.00   fistl	(%ecx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -     1.00   1.00   fistps	(%edx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -     1.00   1.00   fistpl	(%ecx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -     1.00   1.00   fistpll	(%eax)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -     1.00   1.00   fisttps	(%edx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -     1.00   1.00   fisttpl	(%ecx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -     1.00   1.00   fisttpll	(%eax)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fld	%st(0)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     flds	(%edx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fldl	(%ecx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fldt	(%eax)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fldcw	(%eax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fldenv	(%eax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fld1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fldl2e
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fldl2t
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fldlg2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fldln2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fldpi
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fldz
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     fmul	%st, %st(1)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     fmul	%st(2), %st
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fmuls	(%ecx)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -     0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fmull	(%eax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     fmulp	%st, %st(1)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -     fmulp	%st, %st(2)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -     6.00   6.00   6.00   6.00    -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fimuls	(%ecx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -     6.00   6.00   6.00   6.00    -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fimull	(%eax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fnop
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fpatan
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fprem
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fprem1
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fptan
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     frndint
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     frstor	(%eax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fnsave	(%eax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fscale
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fsin
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fsincos
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     23.00   -      -      -      -      -      -      -      -      -      -      -      -      -     fsqrt
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fst	%st(0)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -     1.00   1.00   fsts	(%edx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -     1.00   1.00   fstl	(%ecx)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fstp	%st(0)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -     1.00   1.00   fstpl	(%edx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -     1.00   1.00   fstpl	(%ecx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -      -      -      -      -      -      -      -     0.67   0.67   0.67    -      -      -     1.00   1.00   fstpt	(%eax)
+# CHECK-NEXT:  -      -      -     0.25   0.25   0.25   0.25    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fnstcw	(%eax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fnstenv	(%eax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fnstsw	(%eax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     frstor	(%eax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     wait
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fnsave	(%eax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     fsub	%st, %st(1)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     fsub	%st(2), %st
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fsubs	(%ecx)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fsubl	(%eax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     fsubp	%st, %st(1)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     fsubp	%st, %st(2)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -     6.00   6.00   6.00   6.00    -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fisubs	(%ecx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -     6.00   6.00   6.00   6.00    -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fisubl	(%eax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     fsubr	%st, %st(1)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     fsubr	%st(2), %st
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fsubrs	(%ecx)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50   0.50   0.50    -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fsubrl	(%eax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     fsubrp	%st, %st(1)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -     fsubrp	%st, %st(2)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -     6.00   6.00   6.00   6.00    -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fisubrs	(%ecx)
+# CHECK-NEXT: 0.33   0.33   0.33    -      -      -      -      -     6.00   6.00   6.00   6.00    -      -      -     0.33   0.33   0.33   0.33   0.33   0.33    -      -     fisubrl	(%eax)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     ftst
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fucom	%st(1)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fucom	%st(3)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fucomp	%st(1)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fucomp	%st(3)
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fucompp
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fucomi	%st(3), %st
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     fucompi	%st(3), %st
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     wait
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -     fxam
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fxch	%st(1)
+# CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fxch	%st(3)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fxrstor	(%eax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fxsave	(%eax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fxtract
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fyl2x
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     fyl2xp1
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-xsave.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-xsave.s
new file mode 100644
index 0000000000000..9c2f777c29181
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-xsave.s
@@ -0,0 +1,64 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -instruction-tables < %s | FileCheck %s
+
+xgetbv
+
+xrstor  (%rax)
+
+xrstors (%rax)
+
+xsave   (%rax)
+
+xsetbv
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  100    100   25.00                 U     xgetbv
+# CHECK-NEXT:  100    100   25.00   *      *      U     xrstor	(%rax)
+# CHECK-NEXT:  100    100   25.00   *      *      U     xrstors	(%rax)
+# CHECK-NEXT:  100    100   25.00   *      *      U     xsave	(%rax)
+# CHECK-NEXT:  100    100   25.00   *      *      U     xsetbv
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -     125.00 125.00 125.00 125.00  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xgetbv
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xrstor	(%rax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xrstors	(%rax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xsave	(%rax)
+# CHECK-NEXT:  -      -      -     25.00  25.00  25.00  25.00   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xsetbv
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/zero-idioms.s b/llvm/test/tools/llvm-mca/X86/Znver4/zero-idioms.s
new file mode 100644
index 0000000000000..cc3c286a2c638
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/zero-idioms.s
@@ -0,0 +1,796 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -timeline -register-file-stats -iterations=1 < %s | FileCheck %s
+
+subl  %eax, %eax
+subq  %rax, %rax
+xorl  %eax, %eax
+xorq  %rax, %rax
+
+pcmpgtb   %mm2, %mm2
+pcmpgtd   %mm2, %mm2
+pcmpgtw   %mm2, %mm2
+
+pcmpgtb   %xmm2, %xmm2
+pcmpgtd   %xmm2, %xmm2
+pcmpgtq   %xmm2, %xmm2
+pcmpgtw   %xmm2, %xmm2
+
+vpcmpgtb  %xmm3, %xmm3, %xmm3
+vpcmpgtd  %xmm3, %xmm3, %xmm3
+vpcmpgtq  %xmm3, %xmm3, %xmm3
+vpcmpgtw  %xmm3, %xmm3, %xmm3
+
+vpcmpgtb  %xmm3, %xmm3, %xmm5
+vpcmpgtd  %xmm3, %xmm3, %xmm5
+vpcmpgtq  %xmm3, %xmm3, %xmm5
+vpcmpgtw  %xmm3, %xmm3, %xmm5
+
+vpcmpgtb  %ymm3, %ymm3, %ymm3
+vpcmpgtd  %ymm3, %ymm3, %ymm3
+vpcmpgtq  %ymm3, %ymm3, %ymm3
+vpcmpgtw  %ymm3, %ymm3, %ymm3
+
+vpcmpgtb  %ymm3, %ymm3, %ymm5
+vpcmpgtd  %ymm3, %ymm3, %ymm5
+vpcmpgtq  %ymm3, %ymm3, %ymm5
+vpcmpgtw  %ymm3, %ymm3, %ymm5
+
+psubb   %mm2, %mm2
+psubd   %mm2, %mm2
+psubq   %mm2, %mm2
+psubw   %mm2, %mm2
+psubb   %xmm2, %xmm2
+psubd   %xmm2, %xmm2
+psubq   %xmm2, %xmm2
+psubw   %xmm2, %xmm2
+vpsubb  %xmm3, %xmm3, %xmm3
+vpsubd  %xmm3, %xmm3, %xmm3
+vpsubq  %xmm3, %xmm3, %xmm3
+vpsubw  %xmm3, %xmm3, %xmm3
+vpsubb  %ymm3, %ymm3, %ymm3
+vpsubd  %ymm3, %ymm3, %ymm3
+vpsubq  %ymm3, %ymm3, %ymm3
+vpsubw  %ymm3, %ymm3, %ymm3
+
+vpsubb  %xmm3, %xmm3, %xmm5
+vpsubd  %xmm3, %xmm3, %xmm5
+vpsubq  %xmm3, %xmm3, %xmm5
+vpsubw  %xmm3, %xmm3, %xmm5
+vpsubb  %ymm3, %ymm3, %ymm5
+vpsubd  %ymm3, %ymm3, %ymm5
+vpsubq  %ymm3, %ymm3, %ymm5
+vpsubw  %ymm3, %ymm3, %ymm5
+
+vpsubb  %xmm19, %xmm19, %xmm19
+vpsubd  %xmm19, %xmm19, %xmm19
+vpsubq  %xmm19, %xmm19, %xmm19
+vpsubw  %xmm19, %xmm19, %xmm19
+vpsubb  %ymm19, %ymm19, %ymm19
+vpsubd  %ymm19, %ymm19, %ymm19
+vpsubq  %ymm19, %ymm19, %ymm19
+vpsubw  %ymm19, %ymm19, %ymm19
+vpsubb  %zmm19, %zmm19, %zmm19
+vpsubd  %zmm19, %zmm19, %zmm19
+vpsubq  %zmm19, %zmm19, %zmm19
+vpsubw  %zmm19, %zmm19, %zmm19
+
+vpsubb  %xmm19, %xmm19, %xmm21
+vpsubd  %xmm19, %xmm19, %xmm21
+vpsubq  %xmm19, %xmm19, %xmm21
+vpsubw  %xmm19, %xmm19, %xmm21
+vpsubb  %ymm19, %ymm19, %ymm21
+vpsubd  %ymm19, %ymm19, %ymm21
+vpsubq  %ymm19, %ymm19, %ymm21
+vpsubw  %ymm19, %ymm19, %ymm21
+vpsubb  %zmm19, %zmm19, %zmm21
+vpsubd  %zmm19, %zmm19, %zmm21
+vpsubq  %zmm19, %zmm19, %zmm21
+vpsubw  %zmm19, %zmm19, %zmm21
+
+andnps  %xmm0, %xmm0
+andnpd  %xmm1, %xmm1
+vandnps %xmm2, %xmm2, %xmm2
+vandnpd %xmm1, %xmm1, %xmm1
+vandnps %ymm2, %ymm2, %ymm2
+vandnpd %ymm1, %ymm1, %ymm1
+vandnps %zmm2, %zmm2, %zmm2
+vandnpd %zmm1, %zmm1, %zmm1
+pandn   %mm2, %mm2
+pandn   %xmm2, %xmm2
+vpandn  %xmm3, %xmm3, %xmm3
+vpandn  %ymm3, %ymm3, %ymm3
+
+vpandnd  %xmm19, %xmm19, %xmm19
+vpandnq  %xmm19, %xmm19, %xmm19
+vpandnd  %ymm19, %ymm19, %ymm19
+vpandnq  %ymm19, %ymm19, %ymm19
+vpandnd  %zmm19, %zmm19, %zmm19
+vpandnq  %zmm19, %zmm19, %zmm19
+
+vandnps %xmm2, %xmm2, %xmm5
+vandnpd %xmm1, %xmm1, %xmm5
+vpandn  %xmm3, %xmm3, %xmm5
+vandnps %ymm2, %ymm2, %ymm5
+vandnpd %ymm1, %ymm1, %ymm5
+vpandn  %ymm3, %ymm3, %ymm5
+vandnps %zmm2, %zmm2, %zmm5
+vandnpd %zmm1, %zmm1, %zmm5
+
+vpandnd  %xmm19, %xmm19, %xmm21
+vpandnq  %xmm19, %xmm19, %xmm21
+vpandnd  %ymm19, %ymm19, %ymm21
+vpandnq  %ymm19, %ymm19, %ymm21
+vpandnd  %zmm19, %zmm19, %zmm21
+vpandnq  %zmm19, %zmm19, %zmm21
+
+xorps  %xmm0, %xmm0
+xorpd  %xmm1, %xmm1
+vxorps %xmm2, %xmm2, %xmm2
+vxorpd %xmm1, %xmm1, %xmm1
+vxorps %ymm2, %ymm2, %ymm2
+vxorpd %ymm1, %ymm1, %ymm1
+vxorps %zmm2, %zmm2, %zmm2
+vxorpd %zmm1, %zmm1, %zmm1
+pxor   %mm2, %mm2
+pxor   %xmm2, %xmm2
+vpxor  %xmm3, %xmm3, %xmm3
+vpxor  %ymm3, %ymm3, %ymm3
+
+vpxord  %xmm19, %xmm19, %xmm19
+vpxorq  %xmm19, %xmm19, %xmm19
+vpxord  %ymm19, %ymm19, %ymm19
+vpxorq  %ymm19, %ymm19, %ymm19
+vpxord  %zmm19, %zmm19, %zmm19
+vpxorq  %zmm19, %zmm19, %zmm19
+
+vxorps %xmm4, %xmm4, %xmm5
+vxorpd %xmm1, %xmm1, %xmm3
+vxorps %ymm4, %ymm4, %ymm5
+vxorpd %ymm1, %ymm1, %ymm3
+vxorps %zmm4, %zmm4, %zmm5
+vxorpd %zmm1, %zmm1, %zmm3
+vpxor  %xmm3, %xmm3, %xmm5
+vpxor  %ymm3, %ymm3, %ymm5
+
+vpxord  %xmm19, %xmm19, %xmm21
+vpxorq  %xmm19, %xmm19, %xmm21
+vpxord  %ymm19, %ymm19, %ymm21
+vpxorq  %ymm19, %ymm19, %ymm21
+vpxord  %zmm19, %zmm19, %zmm21
+vpxorq  %zmm19, %zmm19, %zmm21
+
+# CHECK:      Iterations:        1
+# CHECK-NEXT: Instructions:      139
+# CHECK-NEXT: Total Cycles:      42
+# CHECK-NEXT: Total uOps:        139
+
+# CHECK:      Dispatch Width:    6
+# CHECK-NEXT: uOps Per Cycle:    3.31
+# CHECK-NEXT: IPC:               3.31
+# CHECK-NEXT: Block RThroughput: 25.8
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      0     0.17                        subl	%eax, %eax
+# CHECK-NEXT:  1      0     0.17                        subq	%rax, %rax
+# CHECK-NEXT:  1      0     0.17                        xorl	%eax, %eax
+# CHECK-NEXT:  1      0     0.17                        xorq	%rax, %rax
+# CHECK-NEXT:  1      1     0.25                        pcmpgtb	%mm2, %mm2
+# CHECK-NEXT:  1      1     0.25                        pcmpgtd	%mm2, %mm2
+# CHECK-NEXT:  1      1     0.25                        pcmpgtw	%mm2, %mm2
+# CHECK-NEXT:  1      1     0.25                        pcmpgtb	%xmm2, %xmm2
+# CHECK-NEXT:  1      1     0.25                        pcmpgtd	%xmm2, %xmm2
+# CHECK-NEXT:  1      1     0.25                        pcmpgtq	%xmm2, %xmm2
+# CHECK-NEXT:  1      1     0.25                        pcmpgtw	%xmm2, %xmm2
+# CHECK-NEXT:  1      0     0.17                        vpcmpgtb	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  1      0     0.17                        vpcmpgtd	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  1      0     0.17                        vpcmpgtq	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  1      0     0.17                        vpcmpgtw	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  1      0     0.17                        vpcmpgtb	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  1      0     0.17                        vpcmpgtd	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  1      0     0.17                        vpcmpgtq	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  1      0     0.17                        vpcmpgtw	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  1      0     0.17                        vpcmpgtb	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  1      0     0.17                        vpcmpgtd	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  1      0     0.17                        vpcmpgtq	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  1      0     0.17                        vpcmpgtw	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  1      0     0.17                        vpcmpgtb	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  1      0     0.17                        vpcmpgtd	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  1      0     0.17                        vpcmpgtq	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  1      0     0.17                        vpcmpgtw	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  1      1     0.25                        psubb	%mm2, %mm2
+# CHECK-NEXT:  1      1     0.25                        psubd	%mm2, %mm2
+# CHECK-NEXT:  1      1     0.25                        psubq	%mm2, %mm2
+# CHECK-NEXT:  1      1     0.25                        psubw	%mm2, %mm2
+# CHECK-NEXT:  1      1     0.25                        psubb	%xmm2, %xmm2
+# CHECK-NEXT:  1      1     0.25                        psubd	%xmm2, %xmm2
+# CHECK-NEXT:  1      1     0.25                        psubq	%xmm2, %xmm2
+# CHECK-NEXT:  1      1     0.25                        psubw	%xmm2, %xmm2
+# CHECK-NEXT:  1      0     0.17                        vpsubb	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  1      0     0.17                        vpsubd	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  1      0     0.17                        vpsubq	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  1      0     0.17                        vpsubw	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  1      0     0.17                        vpsubb	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  1      0     0.17                        vpsubd	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  1      0     0.17                        vpsubq	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  1      0     0.17                        vpsubw	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  1      0     0.17                        vpsubb	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  1      0     0.17                        vpsubd	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  1      0     0.17                        vpsubq	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  1      0     0.17                        vpsubw	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  1      0     0.17                        vpsubb	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  1      0     0.17                        vpsubd	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  1      0     0.17                        vpsubq	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  1      0     0.17                        vpsubw	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  1      1     0.25                        vpsubb	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpsubd	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpsubq	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpsubw	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpsubb	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpsubd	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpsubq	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpsubw	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpsubb	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpsubd	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpsubq	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpsubw	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT:  1      1     0.25                        vpsubb	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT:  1      1     0.25                        vpsubd	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT:  1      1     0.25                        vpsubq	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT:  1      1     0.25                        vpsubw	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT:  1      1     0.25                        vpsubb	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT:  1      1     0.25                        vpsubd	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT:  1      1     0.25                        vpsubq	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT:  1      1     0.25                        vpsubw	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT:  1      1     0.50                        vpsubb	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT:  1      1     0.50                        vpsubd	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT:  1      1     0.50                        vpsubq	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT:  1      1     0.50                        vpsubw	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT:  1      1     0.25                        andnps	%xmm0, %xmm0
+# CHECK-NEXT:  1      1     0.25                        andnpd	%xmm1, %xmm1
+# CHECK-NEXT:  1      0     0.17                        vandnps	%xmm2, %xmm2, %xmm2
+# CHECK-NEXT:  1      0     0.17                        vandnpd	%xmm1, %xmm1, %xmm1
+# CHECK-NEXT:  1      0     0.17                        vandnps	%ymm2, %ymm2, %ymm2
+# CHECK-NEXT:  1      0     0.17                        vandnpd	%ymm1, %ymm1, %ymm1
+# CHECK-NEXT:  1      1     0.50                        vandnps	%zmm2, %zmm2, %zmm2
+# CHECK-NEXT:  1      1     0.50                        vandnpd	%zmm1, %zmm1, %zmm1
+# CHECK-NEXT:  1      1     0.25                        pandn	%mm2, %mm2
+# CHECK-NEXT:  1      1     0.25                        pandn	%xmm2, %xmm2
+# CHECK-NEXT:  1      0     0.17                        vpandn	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  1      0     0.17                        vpandn	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  1      1     0.25                        vpandnd	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpandnq	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpandnd	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpandnq	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpandnd	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpandnq	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT:  1      0     0.17                        vandnps	%xmm2, %xmm2, %xmm5
+# CHECK-NEXT:  1      0     0.17                        vandnpd	%xmm1, %xmm1, %xmm5
+# CHECK-NEXT:  1      0     0.17                        vpandn	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  1      0     0.17                        vandnps	%ymm2, %ymm2, %ymm5
+# CHECK-NEXT:  1      0     0.17                        vandnpd	%ymm1, %ymm1, %ymm5
+# CHECK-NEXT:  1      0     0.17                        vpandn	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  1      1     0.50                        vandnps	%zmm2, %zmm2, %zmm5
+# CHECK-NEXT:  1      1     0.50                        vandnpd	%zmm1, %zmm1, %zmm5
+# CHECK-NEXT:  1      1     0.25                        vpandnd	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT:  1      1     0.25                        vpandnq	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT:  1      1     0.25                        vpandnd	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT:  1      1     0.25                        vpandnq	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT:  1      1     0.50                        vpandnd	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT:  1      1     0.50                        vpandnq	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT:  1      1     0.25                        xorps	%xmm0, %xmm0
+# CHECK-NEXT:  1      1     0.25                        xorpd	%xmm1, %xmm1
+# CHECK-NEXT:  1      0     0.17                        vxorps	%xmm2, %xmm2, %xmm2
+# CHECK-NEXT:  1      0     0.17                        vxorpd	%xmm1, %xmm1, %xmm1
+# CHECK-NEXT:  1      0     0.17                        vxorps	%ymm2, %ymm2, %ymm2
+# CHECK-NEXT:  1      0     0.17                        vxorpd	%ymm1, %ymm1, %ymm1
+# CHECK-NEXT:  1      1     0.50                        vxorps	%zmm2, %zmm2, %zmm2
+# CHECK-NEXT:  1      1     0.50                        vxorpd	%zmm1, %zmm1, %zmm1
+# CHECK-NEXT:  1      1     0.25                        pxor	%mm2, %mm2
+# CHECK-NEXT:  1      1     0.25                        pxor	%xmm2, %xmm2
+# CHECK-NEXT:  1      0     0.17                        vpxor	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  1      0     0.17                        vpxor	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  1      1     0.25                        vpxord	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpxorq	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT:  1      1     0.25                        vpxord	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT:  1      1     0.25                        vpxorq	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT:  1      1     0.50                        vpxord	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT:  1      1     0.50                        vpxorq	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT:  1      0     0.17                        vxorps	%xmm4, %xmm4, %xmm5
+# CHECK-NEXT:  1      0     0.17                        vxorpd	%xmm1, %xmm1, %xmm3
+# CHECK-NEXT:  1      0     0.17                        vxorps	%ymm4, %ymm4, %ymm5
+# CHECK-NEXT:  1      0     0.17                        vxorpd	%ymm1, %ymm1, %ymm3
+# CHECK-NEXT:  1      1     0.50                        vxorps	%zmm4, %zmm4, %zmm5
+# CHECK-NEXT:  1      1     0.50                        vxorpd	%zmm1, %zmm1, %zmm3
+# CHECK-NEXT:  1      0     0.17                        vpxor	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  1      0     0.17                        vpxor	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  1      1     0.25                        vpxord	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT:  1      1     0.25                        vpxorq	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT:  1      1     0.25                        vpxord	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT:  1      1     0.25                        vpxorq	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT:  1      1     0.50                        vpxord	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT:  1      1     0.50                        vpxorq	%zmm19, %zmm19, %zmm21
+
+# CHECK:      Register File statistics:
+# CHECK-NEXT: Total number of mappings created:    65
+# CHECK-NEXT: Max number of mappings used:         45
+
+# CHECK:      *  Register File #1 -- Zn4FpPRF:
+# CHECK-NEXT:    Number of physical registers:     192
+# CHECK-NEXT:    Total number of mappings created: 65
+# CHECK-NEXT:    Max number of mappings used:      45
+
+# CHECK:      *  Register File #2 -- Zn4IntegerPRF:
+# CHECK-NEXT:    Number of physical registers:     224
+# CHECK-NEXT:    Total number of mappings created: 0
+# CHECK-NEXT:    Max number of mappings used:      0
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0]   - Zn4AGU0
+# CHECK-NEXT: [1]   - Zn4AGU1
+# CHECK-NEXT: [2]   - Zn4AGU2
+# CHECK-NEXT: [3]   - Zn4ALU0
+# CHECK-NEXT: [4]   - Zn4ALU1
+# CHECK-NEXT: [5]   - Zn4ALU2
+# CHECK-NEXT: [6]   - Zn4ALU3
+# CHECK-NEXT: [7]   - Zn4BRU1
+# CHECK-NEXT: [8]   - Zn4FP0
+# CHECK-NEXT: [9]   - Zn4FP1
+# CHECK-NEXT: [10]  - Zn4FP2
+# CHECK-NEXT: [11]  - Zn4FP3
+# CHECK-NEXT: [12.0] - Zn4FP45
+# CHECK-NEXT: [12.1] - Zn4FP45
+# CHECK-NEXT: [13]  - Zn4FPSt
+# CHECK-NEXT: [14.0] - Zn4LSU
+# CHECK-NEXT: [14.1] - Zn4LSU
+# CHECK-NEXT: [14.2] - Zn4LSU
+# CHECK-NEXT: [15.0] - Zn4Load
+# CHECK-NEXT: [15.1] - Zn4Load
+# CHECK-NEXT: [15.2] - Zn4Load
+# CHECK-NEXT: [16.0] - Zn4Store
+# CHECK-NEXT: [16.1] - Zn4Store
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     25.00  25.00  27.00  26.00   -      -      -      -      -      -      -      -      -      -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12.0] [12.1] [13]   [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     subl	%eax, %eax
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     subq	%rax, %rax
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xorl	%eax, %eax
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     xorq	%rax, %rax
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     pcmpgtb	%mm2, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     pcmpgtd	%mm2, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     pcmpgtw	%mm2, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     pcmpgtb	%xmm2, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     pcmpgtd	%xmm2, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     pcmpgtq	%xmm2, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     pcmpgtw	%xmm2, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpcmpgtb	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpcmpgtd	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpcmpgtq	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpcmpgtw	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpcmpgtb	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpcmpgtd	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpcmpgtq	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpcmpgtw	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpcmpgtb	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpcmpgtd	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpcmpgtq	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpcmpgtw	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpcmpgtb	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpcmpgtd	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpcmpgtq	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpcmpgtw	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     psubb	%mm2, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     psubd	%mm2, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     psubq	%mm2, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     psubw	%mm2, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     psubb	%xmm2, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     psubd	%xmm2, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     psubq	%xmm2, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     psubw	%xmm2, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubb	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubd	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubq	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubw	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubb	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubd	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubq	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubw	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubb	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubd	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubq	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubw	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubb	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubd	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubq	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubw	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubb	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpsubd	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpsubq	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubw	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubb	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubd	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubq	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubw	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubb	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubd	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubq	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubw	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubb	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpsubd	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpsubq	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubw	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubb	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpsubd	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpsubq	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubw	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubb	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -     vpsubd	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpsubq	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpsubw	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     andnps	%xmm0, %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     andnpd	%xmm1, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vandnps	%xmm2, %xmm2, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vandnpd	%xmm1, %xmm1, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vandnps	%ymm2, %ymm2, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vandnpd	%ymm1, %ymm1, %ymm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vandnps	%zmm2, %zmm2, %zmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -     vandnpd	%zmm1, %zmm1, %zmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     pandn	%mm2, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     pandn	%xmm2, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpandn	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpandn	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpandnd	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpandnq	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpandnd	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpandnq	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpandnd	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -     vpandnq	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vandnps	%xmm2, %xmm2, %xmm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vandnpd	%xmm1, %xmm1, %xmm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpandn	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vandnps	%ymm2, %ymm2, %ymm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vandnpd	%ymm1, %ymm1, %ymm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpandn	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -     vandnps	%zmm2, %zmm2, %zmm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -     vandnpd	%zmm1, %zmm1, %zmm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpandnd	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpandnq	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpandnd	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpandnq	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpandnd	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpandnq	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     xorps	%xmm0, %xmm0
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     xorpd	%xmm1, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vxorps	%xmm2, %xmm2, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vxorpd	%xmm1, %xmm1, %xmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vxorps	%ymm2, %ymm2, %ymm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vxorpd	%ymm1, %ymm1, %ymm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -     vxorps	%zmm2, %zmm2, %zmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -     vxorpd	%zmm1, %zmm1, %zmm1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     pxor	%mm2, %mm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     pxor	%xmm2, %xmm2
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpxor	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpxor	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpxord	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpxorq	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpxord	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpxorq	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpxord	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -     vpxorq	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vxorps	%xmm4, %xmm4, %xmm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vxorpd	%xmm1, %xmm1, %xmm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vxorps	%ymm4, %ymm4, %ymm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vxorpd	%ymm1, %ymm1, %ymm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vxorps	%zmm4, %zmm4, %zmm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -     vxorpd	%zmm1, %zmm1, %zmm3
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpxor	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpxor	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -     vpxord	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpxorq	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -     vpxord	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     1.00    -      -      -      -      -      -      -      -      -      -      -     vpxorq	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -     vpxord	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     2.00    -      -      -      -      -      -      -      -      -      -      -      -      -     vpxorq	%zmm19, %zmm19, %zmm21
+
+# CHECK:      Timeline view:
+# CHECK-NEXT:                     0123456789          0123456789
+# CHECK-NEXT: Index     0123456789          0123456789          01
+
+# CHECK:      [0,0]     DR   .    .    .    .    .    .    .    ..   subl	%eax, %eax
+# CHECK-NEXT: [0,1]     DR   .    .    .    .    .    .    .    ..   subq	%rax, %rax
+# CHECK-NEXT: [0,2]     DR   .    .    .    .    .    .    .    ..   xorl	%eax, %eax
+# CHECK-NEXT: [0,3]     DR   .    .    .    .    .    .    .    ..   xorq	%rax, %rax
+# CHECK-NEXT: [0,4]     DeER .    .    .    .    .    .    .    ..   pcmpgtb	%mm2, %mm2
+# CHECK-NEXT: [0,5]     D=eER.    .    .    .    .    .    .    ..   pcmpgtd	%mm2, %mm2
+# CHECK-NEXT: [0,6]     .D=eER    .    .    .    .    .    .    ..   pcmpgtw	%mm2, %mm2
+# CHECK-NEXT: [0,7]     .DeE-R    .    .    .    .    .    .    ..   pcmpgtb	%xmm2, %xmm2
+# CHECK-NEXT: [0,8]     .DeE-R    .    .    .    .    .    .    ..   pcmpgtd	%xmm2, %xmm2
+# CHECK-NEXT: [0,9]     .DeE-R    .    .    .    .    .    .    ..   pcmpgtq	%xmm2, %xmm2
+# CHECK-NEXT: [0,10]    .D=eER    .    .    .    .    .    .    ..   pcmpgtw	%xmm2, %xmm2
+# CHECK-NEXT: [0,11]    .D---R    .    .    .    .    .    .    ..   vpcmpgtb	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: [0,12]    . D--R    .    .    .    .    .    .    ..   vpcmpgtd	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: [0,13]    . D--R    .    .    .    .    .    .    ..   vpcmpgtq	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: [0,14]    . D--R    .    .    .    .    .    .    ..   vpcmpgtw	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: [0,15]    . D---R   .    .    .    .    .    .    ..   vpcmpgtb	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: [0,16]    . D---R   .    .    .    .    .    .    ..   vpcmpgtd	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: [0,17]    . D---R   .    .    .    .    .    .    ..   vpcmpgtq	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: [0,18]    .  D--R   .    .    .    .    .    .    ..   vpcmpgtw	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: [0,19]    .  D--R   .    .    .    .    .    .    ..   vpcmpgtb	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: [0,20]    .  D--R   .    .    .    .    .    .    ..   vpcmpgtd	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: [0,21]    .  D--R   .    .    .    .    .    .    ..   vpcmpgtq	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: [0,22]    .  D--R   .    .    .    .    .    .    ..   vpcmpgtw	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: [0,23]    .  D--R   .    .    .    .    .    .    ..   vpcmpgtb	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: [0,24]    .   D--R  .    .    .    .    .    .    ..   vpcmpgtd	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: [0,25]    .   D--R  .    .    .    .    .    .    ..   vpcmpgtq	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: [0,26]    .   D--R  .    .    .    .    .    .    ..   vpcmpgtw	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: [0,27]    .   DeER  .    .    .    .    .    .    ..   psubb	%mm2, %mm2
+# CHECK-NEXT: [0,28]    .   D=eER .    .    .    .    .    .    ..   psubd	%mm2, %mm2
+# CHECK-NEXT: [0,29]    .   D==eER.    .    .    .    .    .    ..   psubq	%mm2, %mm2
+# CHECK-NEXT: [0,30]    .    D==eER    .    .    .    .    .    ..   psubw	%mm2, %mm2
+# CHECK-NEXT: [0,31]    .    DeE--R    .    .    .    .    .    ..   psubb	%xmm2, %xmm2
+# CHECK-NEXT: [0,32]    .    DeE--R    .    .    .    .    .    ..   psubd	%xmm2, %xmm2
+# CHECK-NEXT: [0,33]    .    DeE--R    .    .    .    .    .    ..   psubq	%xmm2, %xmm2
+# CHECK-NEXT: [0,34]    .    D=eE-R    .    .    .    .    .    ..   psubw	%xmm2, %xmm2
+# CHECK-NEXT: [0,35]    .    D----R    .    .    .    .    .    ..   vpsubb	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: [0,36]    .    .D---R    .    .    .    .    .    ..   vpsubd	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: [0,37]    .    .D---R    .    .    .    .    .    ..   vpsubq	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: [0,38]    .    .D---R    .    .    .    .    .    ..   vpsubw	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: [0,39]    .    .D----R   .    .    .    .    .    ..   vpsubb	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: [0,40]    .    .D----R   .    .    .    .    .    ..   vpsubd	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: [0,41]    .    .D----R   .    .    .    .    .    ..   vpsubq	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: [0,42]    .    . D---R   .    .    .    .    .    ..   vpsubw	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: [0,43]    .    . D---R   .    .    .    .    .    ..   vpsubb	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: [0,44]    .    . D---R   .    .    .    .    .    ..   vpsubd	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: [0,45]    .    . D---R   .    .    .    .    .    ..   vpsubq	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: [0,46]    .    . D---R   .    .    .    .    .    ..   vpsubw	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: [0,47]    .    . D---R   .    .    .    .    .    ..   vpsubb	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: [0,48]    .    .  D---R  .    .    .    .    .    ..   vpsubd	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: [0,49]    .    .  D---R  .    .    .    .    .    ..   vpsubq	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: [0,50]    .    .  D---R  .    .    .    .    .    ..   vpsubw	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: [0,51]    .    .  DeE-R  .    .    .    .    .    ..   vpsubb	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT: [0,52]    .    .  D=eER  .    .    .    .    .    ..   vpsubd	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT: [0,53]    .    .  D==eER .    .    .    .    .    ..   vpsubq	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT: [0,54]    .    .   D==eER.    .    .    .    .    ..   vpsubw	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT: [0,55]    .    .   D===eER    .    .    .    .    ..   vpsubb	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT: [0,56]    .    .   D====eER   .    .    .    .    ..   vpsubd	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT: [0,57]    .    .   D=====eER  .    .    .    .    ..   vpsubq	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT: [0,58]    .    .   D======eER .    .    .    .    ..   vpsubw	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT: [0,59]    .    .   D=======eER.    .    .    .    ..   vpsubb	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT: [0,60]    .    .    D=======eER    .    .    .    ..   vpsubd	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT: [0,61]    .    .    D========eER   .    .    .    ..   vpsubq	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT: [0,62]    .    .    D=========eER  .    .    .    ..   vpsubw	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT: [0,63]    .    .    D==========eER .    .    .    ..   vpsubb	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT: [0,64]    .    .    D===========eER.    .    .    ..   vpsubd	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT: [0,65]    .    .    D===========eER.    .    .    ..   vpsubq	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT: [0,66]    .    .    .D==========eER.    .    .    ..   vpsubw	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT: [0,67]    .    .    .D==========eER.    .    .    ..   vpsubb	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT: [0,68]    .    .    .D===========eER    .    .    ..   vpsubd	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT: [0,69]    .    .    .D===========eER    .    .    ..   vpsubq	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT: [0,70]    .    .    .D===========eER    .    .    ..   vpsubw	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT: [0,71]    .    .    .D===========eER    .    .    ..   vpsubb	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT: [0,72]    .    .    . D===========eER   .    .    ..   vpsubd	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT: [0,73]    .    .    . D===========eER   .    .    ..   vpsubq	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT: [0,74]    .    .    . D===========eER   .    .    ..   vpsubw	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT: [0,75]    .    .    . DeE-----------R   .    .    ..   andnps	%xmm0, %xmm0
+# CHECK-NEXT: [0,76]    .    .    . DeE-----------R   .    .    ..   andnpd	%xmm1, %xmm1
+# CHECK-NEXT: [0,77]    .    .    . D-------------R   .    .    ..   vandnps	%xmm2, %xmm2, %xmm2
+# CHECK-NEXT: [0,78]    .    .    .  D------------R   .    .    ..   vandnpd	%xmm1, %xmm1, %xmm1
+# CHECK-NEXT: [0,79]    .    .    .  D------------R   .    .    ..   vandnps	%ymm2, %ymm2, %ymm2
+# CHECK-NEXT: [0,80]    .    .    .  D------------R   .    .    ..   vandnpd	%ymm1, %ymm1, %ymm1
+# CHECK-NEXT: [0,81]    .    .    .  DeE-----------R  .    .    ..   vandnps	%zmm2, %zmm2, %zmm2
+# CHECK-NEXT: [0,82]    .    .    .  DeE-----------R  .    .    ..   vandnpd	%zmm1, %zmm1, %zmm1
+# CHECK-NEXT: [0,83]    .    .    .  DeE-----------R  .    .    ..   pandn	%mm2, %mm2
+# CHECK-NEXT: [0,84]    .    .    .   DeE----------R  .    .    ..   pandn	%xmm2, %xmm2
+# CHECK-NEXT: [0,85]    .    .    .   D------------R  .    .    ..   vpandn	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: [0,86]    .    .    .   D------------R  .    .    ..   vpandn	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: [0,87]    .    .    .   D==========eER  .    .    ..   vpandnd	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT: [0,88]    .    .    .   D===========eER .    .    ..   vpandnq	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT: [0,89]    .    .    .   D============eER.    .    ..   vpandnd	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT: [0,90]    .    .    .    D============eER    .    ..   vpandnq	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT: [0,91]    .    .    .    D=============eER   .    ..   vpandnd	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT: [0,92]    .    .    .    D==============eER  .    ..   vpandnq	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT: [0,93]    .    .    .    D----------------R  .    ..   vandnps	%xmm2, %xmm2, %xmm5
+# CHECK-NEXT: [0,94]    .    .    .    D----------------R  .    ..   vandnpd	%xmm1, %xmm1, %xmm5
+# CHECK-NEXT: [0,95]    .    .    .    D----------------R  .    ..   vpandn	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: [0,96]    .    .    .    .D---------------R  .    ..   vandnps	%ymm2, %ymm2, %ymm5
+# CHECK-NEXT: [0,97]    .    .    .    .D---------------R  .    ..   vandnpd	%ymm1, %ymm1, %ymm5
+# CHECK-NEXT: [0,98]    .    .    .    .D---------------R  .    ..   vpandn	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: [0,99]    .    .    .    .DeE-------------R  .    ..   vandnps	%zmm2, %zmm2, %zmm5
+# CHECK-NEXT: [0,100]   .    .    .    .DeE-------------R  .    ..   vandnpd	%zmm1, %zmm1, %zmm5
+# CHECK-NEXT: [0,101]   .    .    .    .D==============eER .    ..   vpandnd	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT: [0,102]   .    .    .    . D=============eER .    ..   vpandnq	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT: [0,103]   .    .    .    . D=============eER .    ..   vpandnd	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT: [0,104]   .    .    .    . D==============eER.    ..   vpandnq	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT: [0,105]   .    .    .    . D==============eER.    ..   vpandnd	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT: [0,106]   .    .    .    . D==============eER.    ..   vpandnq	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT: [0,107]   .    .    .    . D=eE-------------R.    ..   xorps	%xmm0, %xmm0
+# CHECK-NEXT: [0,108]   .    .    .    .  DeE-------------R.    ..   xorpd	%xmm1, %xmm1
+# CHECK-NEXT: [0,109]   .    .    .    .  D---------------R.    ..   vxorps	%xmm2, %xmm2, %xmm2
+# CHECK-NEXT: [0,110]   .    .    .    .  D---------------R.    ..   vxorpd	%xmm1, %xmm1, %xmm1
+# CHECK-NEXT: [0,111]   .    .    .    .  D---------------R.    ..   vxorps	%ymm2, %ymm2, %ymm2
+# CHECK-NEXT: [0,112]   .    .    .    .  D---------------R.    ..   vxorpd	%ymm1, %ymm1, %ymm1
+# CHECK-NEXT: [0,113]   .    .    .    .  D=eE-------------R    ..   vxorps	%zmm2, %zmm2, %zmm2
+# CHECK-NEXT: [0,114]   .    .    .    .   DeE-------------R    ..   vxorpd	%zmm1, %zmm1, %zmm1
+# CHECK-NEXT: [0,115]   .    .    .    .   D======eE-------R    ..   pxor	%mm2, %mm2
+# CHECK-NEXT: [0,116]   .    .    .    .   D======eE-------R    ..   pxor	%xmm2, %xmm2
+# CHECK-NEXT: [0,117]   .    .    .    .   D---------------R    ..   vpxor	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: [0,118]   .    .    .    .   D---------------R    ..   vpxor	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: [0,119]   .    .    .    .   D============eE-R    ..   vpxord	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT: [0,120]   .    .    .    .    D============eER    ..   vpxorq	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT: [0,121]   .    .    .    .    D=============eER   ..   vpxord	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT: [0,122]   .    .    .    .    D==============eER  ..   vpxorq	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT: [0,123]   .    .    .    .    D===============eER ..   vpxord	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT: [0,124]   .    .    .    .    D================eER..   vpxorq	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT: [0,125]   .    .    .    .    D------------------R..   vxorps	%xmm4, %xmm4, %xmm5
+# CHECK-NEXT: [0,126]   .    .    .    .    .D-----------------R..   vxorpd	%xmm1, %xmm1, %xmm3
+# CHECK-NEXT: [0,127]   .    .    .    .    .D-----------------R..   vxorps	%ymm4, %ymm4, %ymm5
+# CHECK-NEXT: [0,128]   .    .    .    .    .D-----------------R..   vxorpd	%ymm1, %ymm1, %ymm3
+# CHECK-NEXT: [0,129]   .    .    .    .    .D====eE-----------R..   vxorps	%zmm4, %zmm4, %zmm5
+# CHECK-NEXT: [0,130]   .    .    .    .    .D=====eE----------R..   vxorpd	%zmm1, %zmm1, %zmm3
+# CHECK-NEXT: [0,131]   .    .    .    .    .D-----------------R..   vpxor	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: [0,132]   .    .    .    .    . D----------------R..   vpxor	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: [0,133]   .    .    .    .    . D===============eER.   vpxord	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT: [0,134]   .    .    .    .    . D===============eER.   vpxorq	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT: [0,135]   .    .    .    .    . D===============eER.   vpxord	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT: [0,136]   .    .    .    .    . D================eER   vpxorq	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT: [0,137]   .    .    .    .    . D================eER   vpxord	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT: [0,138]   .    .    .    .    .  D===============eER   vpxorq	%zmm19, %zmm19, %zmm21
+
+# CHECK:      Average Wait times (based on the timeline view):
+# CHECK-NEXT: [0]: Executions
+# CHECK-NEXT: [1]: Average time spent waiting in a scheduler's queue
+# CHECK-NEXT: [2]: Average time spent waiting in a scheduler's queue while ready
+# CHECK-NEXT: [3]: Average time elapsed from WB until retire stage
+
+# CHECK:            [0]    [1]    [2]    [3]
+# CHECK-NEXT: 0.     1     0.0    0.0    0.0       subl	%eax, %eax
+# CHECK-NEXT: 1.     1     0.0    0.0    0.0       subq	%rax, %rax
+# CHECK-NEXT: 2.     1     0.0    0.0    0.0       xorl	%eax, %eax
+# CHECK-NEXT: 3.     1     0.0    0.0    0.0       xorq	%rax, %rax
+# CHECK-NEXT: 4.     1     1.0    1.0    0.0       pcmpgtb	%mm2, %mm2
+# CHECK-NEXT: 5.     1     2.0    0.0    0.0       pcmpgtd	%mm2, %mm2
+# CHECK-NEXT: 6.     1     2.0    0.0    0.0       pcmpgtw	%mm2, %mm2
+# CHECK-NEXT: 7.     1     1.0    1.0    1.0       pcmpgtb	%xmm2, %xmm2
+# CHECK-NEXT: 8.     1     1.0    1.0    1.0       pcmpgtd	%xmm2, %xmm2
+# CHECK-NEXT: 9.     1     1.0    1.0    1.0       pcmpgtq	%xmm2, %xmm2
+# CHECK-NEXT: 10.    1     2.0    2.0    0.0       pcmpgtw	%xmm2, %xmm2
+# CHECK-NEXT: 11.    1     0.0    0.0    3.0       vpcmpgtb	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: 12.    1     0.0    0.0    2.0       vpcmpgtd	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: 13.    1     0.0    0.0    2.0       vpcmpgtq	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: 14.    1     0.0    0.0    2.0       vpcmpgtw	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: 15.    1     0.0    0.0    3.0       vpcmpgtb	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: 16.    1     0.0    0.0    3.0       vpcmpgtd	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: 17.    1     0.0    0.0    3.0       vpcmpgtq	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: 18.    1     0.0    0.0    2.0       vpcmpgtw	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: 19.    1     0.0    0.0    2.0       vpcmpgtb	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: 20.    1     0.0    0.0    2.0       vpcmpgtd	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: 21.    1     0.0    0.0    2.0       vpcmpgtq	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: 22.    1     0.0    0.0    2.0       vpcmpgtw	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: 23.    1     0.0    0.0    2.0       vpcmpgtb	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: 24.    1     0.0    0.0    2.0       vpcmpgtd	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: 25.    1     0.0    0.0    2.0       vpcmpgtq	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: 26.    1     0.0    0.0    2.0       vpcmpgtw	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: 27.    1     1.0    1.0    0.0       psubb	%mm2, %mm2
+# CHECK-NEXT: 28.    1     2.0    0.0    0.0       psubd	%mm2, %mm2
+# CHECK-NEXT: 29.    1     3.0    0.0    0.0       psubq	%mm2, %mm2
+# CHECK-NEXT: 30.    1     3.0    0.0    0.0       psubw	%mm2, %mm2
+# CHECK-NEXT: 31.    1     1.0    1.0    2.0       psubb	%xmm2, %xmm2
+# CHECK-NEXT: 32.    1     1.0    1.0    2.0       psubd	%xmm2, %xmm2
+# CHECK-NEXT: 33.    1     1.0    1.0    2.0       psubq	%xmm2, %xmm2
+# CHECK-NEXT: 34.    1     2.0    2.0    1.0       psubw	%xmm2, %xmm2
+# CHECK-NEXT: 35.    1     0.0    0.0    4.0       vpsubb	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: 36.    1     0.0    0.0    3.0       vpsubd	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: 37.    1     0.0    0.0    3.0       vpsubq	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: 38.    1     0.0    0.0    3.0       vpsubw	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: 39.    1     0.0    0.0    4.0       vpsubb	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: 40.    1     0.0    0.0    4.0       vpsubd	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: 41.    1     0.0    0.0    4.0       vpsubq	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: 42.    1     0.0    0.0    3.0       vpsubw	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: 43.    1     0.0    0.0    3.0       vpsubb	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: 44.    1     0.0    0.0    3.0       vpsubd	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: 45.    1     0.0    0.0    3.0       vpsubq	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: 46.    1     0.0    0.0    3.0       vpsubw	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: 47.    1     0.0    0.0    3.0       vpsubb	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: 48.    1     0.0    0.0    3.0       vpsubd	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: 49.    1     0.0    0.0    3.0       vpsubq	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: 50.    1     0.0    0.0    3.0       vpsubw	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: 51.    1     1.0    1.0    1.0       vpsubb	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT: 52.    1     2.0    0.0    0.0       vpsubd	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT: 53.    1     3.0    0.0    0.0       vpsubq	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT: 54.    1     3.0    0.0    0.0       vpsubw	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT: 55.    1     4.0    0.0    0.0       vpsubb	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT: 56.    1     5.0    0.0    0.0       vpsubd	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT: 57.    1     6.0    0.0    0.0       vpsubq	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT: 58.    1     7.0    0.0    0.0       vpsubw	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT: 59.    1     8.0    0.0    0.0       vpsubb	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT: 60.    1     8.0    0.0    0.0       vpsubd	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT: 61.    1     9.0    0.0    0.0       vpsubq	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT: 62.    1     10.0   0.0    0.0       vpsubw	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT: 63.    1     11.0   0.0    0.0       vpsubb	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT: 64.    1     12.0   1.0    0.0       vpsubd	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT: 65.    1     12.0   1.0    0.0       vpsubq	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT: 66.    1     11.0   1.0    0.0       vpsubw	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT: 67.    1     11.0   1.0    0.0       vpsubb	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT: 68.    1     12.0   2.0    0.0       vpsubd	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT: 69.    1     12.0   2.0    0.0       vpsubq	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT: 70.    1     12.0   2.0    0.0       vpsubw	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT: 71.    1     12.0   2.0    0.0       vpsubb	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT: 72.    1     12.0   3.0    0.0       vpsubd	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT: 73.    1     12.0   3.0    0.0       vpsubq	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT: 74.    1     12.0   3.0    0.0       vpsubw	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT: 75.    1     1.0    1.0    11.0      andnps	%xmm0, %xmm0
+# CHECK-NEXT: 76.    1     1.0    1.0    11.0      andnpd	%xmm1, %xmm1
+# CHECK-NEXT: 77.    1     0.0    0.0    13.0      vandnps	%xmm2, %xmm2, %xmm2
+# CHECK-NEXT: 78.    1     0.0    0.0    12.0      vandnpd	%xmm1, %xmm1, %xmm1
+# CHECK-NEXT: 79.    1     0.0    0.0    12.0      vandnps	%ymm2, %ymm2, %ymm2
+# CHECK-NEXT: 80.    1     0.0    0.0    12.0      vandnpd	%ymm1, %ymm1, %ymm1
+# CHECK-NEXT: 81.    1     1.0    1.0    11.0      vandnps	%zmm2, %zmm2, %zmm2
+# CHECK-NEXT: 82.    1     1.0    1.0    11.0      vandnpd	%zmm1, %zmm1, %zmm1
+# CHECK-NEXT: 83.    1     1.0    1.0    11.0      pandn	%mm2, %mm2
+# CHECK-NEXT: 84.    1     1.0    1.0    10.0      pandn	%xmm2, %xmm2
+# CHECK-NEXT: 85.    1     0.0    0.0    12.0      vpandn	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: 86.    1     0.0    0.0    12.0      vpandn	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: 87.    1     11.0   4.0    0.0       vpandnd	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT: 88.    1     12.0   0.0    0.0       vpandnq	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT: 89.    1     13.0   0.0    0.0       vpandnd	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT: 90.    1     13.0   0.0    0.0       vpandnq	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT: 91.    1     14.0   0.0    0.0       vpandnd	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT: 92.    1     15.0   0.0    0.0       vpandnq	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT: 93.    1     0.0    0.0    16.0      vandnps	%xmm2, %xmm2, %xmm5
+# CHECK-NEXT: 94.    1     0.0    0.0    16.0      vandnpd	%xmm1, %xmm1, %xmm5
+# CHECK-NEXT: 95.    1     0.0    0.0    16.0      vpandn	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: 96.    1     0.0    0.0    15.0      vandnps	%ymm2, %ymm2, %ymm5
+# CHECK-NEXT: 97.    1     0.0    0.0    15.0      vandnpd	%ymm1, %ymm1, %ymm5
+# CHECK-NEXT: 98.    1     0.0    0.0    15.0      vpandn	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: 99.    1     1.0    1.0    13.0      vandnps	%zmm2, %zmm2, %zmm5
+# CHECK-NEXT: 100.   1     1.0    1.0    13.0      vandnpd	%zmm1, %zmm1, %zmm5
+# CHECK-NEXT: 101.   1     15.0   0.0    0.0       vpandnd	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT: 102.   1     14.0   0.0    0.0       vpandnq	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT: 103.   1     14.0   0.0    0.0       vpandnd	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT: 104.   1     15.0   1.0    0.0       vpandnq	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT: 105.   1     15.0   1.0    0.0       vpandnd	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT: 106.   1     15.0   1.0    0.0       vpandnq	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT: 107.   1     2.0    2.0    13.0      xorps	%xmm0, %xmm0
+# CHECK-NEXT: 108.   1     1.0    1.0    13.0      xorpd	%xmm1, %xmm1
+# CHECK-NEXT: 109.   1     0.0    0.0    15.0      vxorps	%xmm2, %xmm2, %xmm2
+# CHECK-NEXT: 110.   1     0.0    0.0    15.0      vxorpd	%xmm1, %xmm1, %xmm1
+# CHECK-NEXT: 111.   1     0.0    0.0    15.0      vxorps	%ymm2, %ymm2, %ymm2
+# CHECK-NEXT: 112.   1     0.0    0.0    15.0      vxorpd	%ymm1, %ymm1, %ymm1
+# CHECK-NEXT: 113.   1     2.0    2.0    13.0      vxorps	%zmm2, %zmm2, %zmm2
+# CHECK-NEXT: 114.   1     1.0    1.0    13.0      vxorpd	%zmm1, %zmm1, %zmm1
+# CHECK-NEXT: 115.   1     7.0    7.0    7.0       pxor	%mm2, %mm2
+# CHECK-NEXT: 116.   1     7.0    7.0    7.0       pxor	%xmm2, %xmm2
+# CHECK-NEXT: 117.   1     0.0    0.0    15.0      vpxor	%xmm3, %xmm3, %xmm3
+# CHECK-NEXT: 118.   1     0.0    0.0    15.0      vpxor	%ymm3, %ymm3, %ymm3
+# CHECK-NEXT: 119.   1     13.0   1.0    1.0       vpxord	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT: 120.   1     13.0   0.0    0.0       vpxorq	%xmm19, %xmm19, %xmm19
+# CHECK-NEXT: 121.   1     14.0   0.0    0.0       vpxord	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT: 122.   1     15.0   0.0    0.0       vpxorq	%ymm19, %ymm19, %ymm19
+# CHECK-NEXT: 123.   1     16.0   0.0    0.0       vpxord	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT: 124.   1     17.0   0.0    0.0       vpxorq	%zmm19, %zmm19, %zmm19
+# CHECK-NEXT: 125.   1     0.0    0.0    18.0      vxorps	%xmm4, %xmm4, %xmm5
+# CHECK-NEXT: 126.   1     0.0    0.0    17.0      vxorpd	%xmm1, %xmm1, %xmm3
+# CHECK-NEXT: 127.   1     0.0    0.0    17.0      vxorps	%ymm4, %ymm4, %ymm5
+# CHECK-NEXT: 128.   1     0.0    0.0    17.0      vxorpd	%ymm1, %ymm1, %ymm3
+# CHECK-NEXT: 129.   1     5.0    5.0    11.0      vxorps	%zmm4, %zmm4, %zmm5
+# CHECK-NEXT: 130.   1     6.0    6.0    10.0      vxorpd	%zmm1, %zmm1, %zmm3
+# CHECK-NEXT: 131.   1     0.0    0.0    17.0      vpxor	%xmm3, %xmm3, %xmm5
+# CHECK-NEXT: 132.   1     0.0    0.0    16.0      vpxor	%ymm3, %ymm3, %ymm5
+# CHECK-NEXT: 133.   1     16.0   0.0    0.0       vpxord	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT: 134.   1     16.0   0.0    0.0       vpxorq	%xmm19, %xmm19, %xmm21
+# CHECK-NEXT: 135.   1     16.0   0.0    0.0       vpxord	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT: 136.   1     17.0   1.0    0.0       vpxorq	%ymm19, %ymm19, %ymm21
+# CHECK-NEXT: 137.   1     17.0   1.0    0.0       vpxord	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT: 138.   1     16.0   1.0    0.0       vpxorq	%zmm19, %zmm19, %zmm21
+# CHECK-NEXT:        1     4.5    0.6    4.6       <total>
diff --git a/llvm/test/tools/llvm-mca/X86/cpus.s b/llvm/test/tools/llvm-mca/X86/cpus.s
index 568df4644996d..38e0365dc80eb 100644
--- a/llvm/test/tools/llvm-mca/X86/cpus.s
+++ b/llvm/test/tools/llvm-mca/X86/cpus.s
@@ -5,6 +5,7 @@
 # RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=znver1 -resource-pressure=false -instruction-info=false < %s | FileCheck --check-prefixes=ALL,ZNVER1 %s
 # RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=znver2 -resource-pressure=false -instruction-info=false < %s | FileCheck --check-prefixes=ALL,ZNVER2 %s
 # RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=znver3 -resource-pressure=false -instruction-info=false < %s | FileCheck --check-prefixes=ALL,ZNVER3 %s
+# RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=znver4 -resource-pressure=false -instruction-info=false < %s | FileCheck --check-prefixes=ALL,ZNVER4 %s
 # RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=sandybridge -resource-pressure=false -instruction-info=false < %s | FileCheck --check-prefixes=ALL,SANDYBRIDGE %s
 # RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=ivybridge -resource-pressure=false -instruction-info=false < %s | FileCheck --check-prefixes=ALL,IVYBRIDGE %s
 # RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=haswell -resource-pressure=false -instruction-info=false < %s | FileCheck --check-prefixes=ALL,HASWELL %s
@@ -99,3 +100,8 @@ add %edi, %eax
 # ZNVER3-NEXT:      uOps Per Cycle:    0.97
 # ZNVER3-NEXT:      IPC:               0.97
 # ZNVER3-NEXT:      Block RThroughput: 0.3
+
+# ZNVER4:           Dispatch Width:    6
+# ZNVER4-NEXT:      uOps Per Cycle:    0.97
+# ZNVER4-NEXT:      IPC:               0.97
+# ZNVER4-NEXT:      Block RThroughput: 0.3
diff --git a/llvm/test/tools/llvm-mca/X86/read-after-ld-1.s b/llvm/test/tools/llvm-mca/X86/read-after-ld-1.s
index 9efa6a480a3cc..7b9fb758320ec 100644
--- a/llvm/test/tools/llvm-mca/X86/read-after-ld-1.s
+++ b/llvm/test/tools/llvm-mca/X86/read-after-ld-1.s
@@ -10,6 +10,7 @@
 # RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver1 -iterations=1 -resource-pressure=false -instruction-info=false -timeline < %s | FileCheck %s -check-prefixes=ALL,ZNVER1
 # RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver2 -iterations=1 -resource-pressure=false -instruction-info=false -timeline < %s | FileCheck %s -check-prefixes=ALL,ZNVER2
 # RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver3 -iterations=1 -resource-pressure=false -instruction-info=false -timeline < %s | FileCheck %s -check-prefixes=ALL,ZNVER3
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver4 -iterations=1 -resource-pressure=false -instruction-info=false -timeline < %s | FileCheck %s -check-prefixes=ALL,ZNVER4
 
 vdivps  %xmm0, %xmm1, %xmm1
 vaddps  (%rax), %xmm1, %xmm1
@@ -47,6 +48,9 @@ vaddps  (%rax), %xmm1, %xmm1
 # ZNVER3-NEXT:    Total Cycles:      17
 # ZNVER3-NEXT:    Total uOps:        2
 
+# ZNVER4-NEXT:    Total Cycles:      17
+# ZNVER4-NEXT:    Total uOps:        2
+
 # BARCELONA:      Dispatch Width:    4
 # BARCELONA-NEXT: uOps Per Cycle:    0.15
 # BARCELONA-NEXT: IPC:               0.10
@@ -97,6 +101,11 @@ vaddps  (%rax), %xmm1, %xmm1
 # ZNVER3-NEXT:    IPC:               0.12
 # ZNVER3-NEXT:    Block RThroughput: 3.0
 
+# ZNVER4:         Dispatch Width:    6
+# ZNVER4-NEXT:    uOps Per Cycle:    0.12
+# ZNVER4-NEXT:    IPC:               0.12
+# ZNVER4-NEXT:    Block RThroughput: 3.0
+
 # ALL:            Timeline view:
 
 # BARCELONA-NEXT:                     0123456789
@@ -129,6 +138,9 @@ vaddps  (%rax), %xmm1, %xmm1
 # ZNVER3-NEXT:                        0123456
 # ZNVER3-NEXT:    Index     0123456789
 
+# ZNVER4-NEXT:                        0123456
+# ZNVER4-NEXT:    Index     0123456789
+
 # BARCELONA:      [0,0]     DeeeeeeeeeeeeeeER  .   vdivps	%xmm0, %xmm1, %xmm1
 # BARCELONA-NEXT: [0,1]     D========eeeeeeeeeER   vaddps	(%rax), %xmm1, %xmm1
 
@@ -159,6 +171,9 @@ vaddps  (%rax), %xmm1, %xmm1
 # ZNVER3:         [0,0]     DeeeeeeeeeeeER ..   vdivps	%xmm0, %xmm1, %xmm1
 # ZNVER3-NEXT:    [0,1]     D====eeeeeeeeeeER   vaddps	(%rax), %xmm1, %xmm1
 
+# ZNVER4:         [0,0]     DeeeeeeeeeeeER ..   vdivps	%xmm0, %xmm1, %xmm1
+# ZNVER4-NEXT:    [0,1]     D====eeeeeeeeeeER   vaddps	(%rax), %xmm1, %xmm1
+
 # ALL:            Average Wait times (based on the timeline view):
 # ALL-NEXT:       [0]: Executions
 # ALL-NEXT:       [1]: Average time spent waiting in a scheduler's queue
@@ -197,3 +212,6 @@ vaddps  (%rax), %xmm1, %xmm1
 
 # ZNVER3-NEXT:    1.     1     5.0    0.0    0.0       vaddps	(%rax), %xmm1, %xmm1
 # ZNVER3-NEXT:           1     3.0    0.5    0.0       <total>
+
+# ZNVER4-NEXT:    1.     1     5.0    0.0    0.0       vaddps	(%rax), %xmm1, %xmm1
+# ZNVER4-NEXT:           1     3.0    0.5    0.0       <total>
diff --git a/llvm/test/tools/llvm-mca/X86/register-file-statistics.s b/llvm/test/tools/llvm-mca/X86/register-file-statistics.s
index 79c2de8fe3a0e..fd90a6ec59977 100644
--- a/llvm/test/tools/llvm-mca/X86/register-file-statistics.s
+++ b/llvm/test/tools/llvm-mca/X86/register-file-statistics.s
@@ -6,6 +6,7 @@
 # RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=znver1 -iterations=1 -all-stats=false -all-views=false -register-file-stats < %s | FileCheck --check-prefixes=ALL,ZNVER1 %s
 # RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=znver2 -iterations=1 -all-stats=false -all-views=false -register-file-stats < %s | FileCheck --check-prefixes=ALL,ZNVER2 %s
 # RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=znver3 -iterations=1 -all-stats=false -all-views=false -register-file-stats < %s | FileCheck --check-prefixes=ALL,ZNVER3 %s
+# RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=znver4 -iterations=1 -all-stats=false -all-views=false -register-file-stats < %s | FileCheck --check-prefixes=ALL,ZNVER4 %s
 # RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=sandybridge -iterations=1 -all-stats=false -all-views=false -register-file-stats < %s | FileCheck --check-prefixes=ALL %s
 # RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=ivybridge -iterations=1 -all-stats=false -all-views=false -register-file-stats < %s | FileCheck --check-prefixes=ALL %s
 # RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=haswell -iterations=1 -all-stats=false -all-views=false -register-file-stats < %s | FileCheck --check-prefixes=ALL %s
@@ -54,6 +55,11 @@ xor %eax, %ebx
 # ZNVER3-NEXT:       Total number of mappings created: 0
 # ZNVER3-NEXT:       Max number of mappings used:      0
 
+# ZNVER4:         *  Register File #1 -- Zn4FpPRF:
+# ZNVER4-NEXT:       Number of physical registers:     192
+# ZNVER4-NEXT:       Total number of mappings created: 0
+# ZNVER4-NEXT:       Max number of mappings used:      0
+
 # BDVER2:         *  Register File #2 -- PdIntegerPRF:
 # BDVER2-NEXT:       Number of physical registers:     96
 # BDVER2-NEXT:       Total number of mappings created: 2
@@ -78,3 +84,8 @@ xor %eax, %ebx
 # ZNVER3-NEXT:       Number of physical registers:     192
 # ZNVER3-NEXT:       Total number of mappings created: 2
 # ZNVER3-NEXT:       Max number of mappings used:      2
+
+# ZNVER4:         *  Register File #2 -- Zn4IntegerPRF:
+# ZNVER4-NEXT:       Number of physical registers:     224
+# ZNVER4-NEXT:       Total number of mappings created: 2
+# ZNVER4-NEXT:       Max number of mappings used:      2
diff --git a/llvm/test/tools/llvm-mca/X86/scheduler-queue-usage.s b/llvm/test/tools/llvm-mca/X86/scheduler-queue-usage.s
index 0dcfdaa1acfd7..2d5c0483de7df 100644
--- a/llvm/test/tools/llvm-mca/X86/scheduler-queue-usage.s
+++ b/llvm/test/tools/llvm-mca/X86/scheduler-queue-usage.s
@@ -5,6 +5,7 @@
 # RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=znver1 -iterations=1 -all-stats=false -all-views=false -scheduler-stats < %s | FileCheck --check-prefixes=ALL,ZNVER1 %s
 # RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=znver2 -iterations=1 -all-stats=false -all-views=false -scheduler-stats < %s | FileCheck --check-prefixes=ALL,ZNVER2 %s
 # RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=znver3 -iterations=1 -all-stats=false -all-views=false -scheduler-stats < %s | FileCheck --check-prefixes=ALL,ZNVER3 %s
+# RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=znver4 -iterations=1 -all-stats=false -all-views=false -scheduler-stats < %s | FileCheck --check-prefixes=ALL,ZNVER4 %s
 # RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=sandybridge -iterations=1 -all-stats=false -all-views=false -scheduler-stats < %s | FileCheck --check-prefixes=ALL,SNB %s
 # RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=ivybridge -iterations=1 -all-stats=false -all-views=false -scheduler-stats < %s | FileCheck --check-prefixes=ALL,IVB %s
 # RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=haswell -iterations=1 -all-stats=false -all-views=false -scheduler-stats < %s | FileCheck --check-prefixes=ALL,HSW %s
@@ -112,6 +113,12 @@ xor %eax, %ebx
 # ZNVER3-NEXT:     [3] Maximum number of used buffer entries.
 # ZNVER3-NEXT:     [4] Total number of buffer entries.
 
+# ZNVER4:          Scheduler's queue usage:
+# ZNVER4-NEXT:     [1] Resource name.
+# ZNVER4-NEXT:     [2] Average number of used buffer entries.
+# ZNVER4-NEXT:     [3] Maximum number of used buffer entries.
+# ZNVER4-NEXT:     [4] Total number of buffer entries.
+
 # BARCELONA:        [1]            [2]        [3]        [4]
 # BARCELONA-NEXT:  SBPortAny        0          1          54
 
@@ -165,3 +172,9 @@ xor %eax, %ebx
 # ZNVER3-NEXT:     Zn3Int           0          1          96
 # ZNVER3-NEXT:     Zn3Load          0          0          72
 # ZNVER3-NEXT:     Zn3Store         0          0          64
+
+# ZNVER4:           [1]            [2]        [3]        [4]
+# ZNVER4-NEXT:     Zn4FP            0          0          64
+# ZNVER4-NEXT:     Zn4Int           0          1          96
+# ZNVER4-NEXT:     Zn4Load          0          0          72
+# ZNVER4-NEXT:     Zn4Store         0          0          64
diff --git a/llvm/test/tools/llvm-objdump/ELF/RISCV/riscv-attributes.s b/llvm/test/tools/llvm-objdump/ELF/RISCV/riscv-attributes.s
new file mode 100644
index 0000000000000..7c41b63dcbf86
--- /dev/null
+++ b/llvm/test/tools/llvm-objdump/ELF/RISCV/riscv-attributes.s
@@ -0,0 +1,91 @@
+# RUN: rm -rf %t && split-file %s %t && cd %t
+
+# RUN: llvm-mc -filetype=obj -triple=riscv64 -mattr=+m,+f,+d,+v noncanonicalized_arch.s -o noncanonicalized_arch.o
+# RUN: llvm-objdump -d noncanonicalized_arch.o | FileCheck %s --check-prefix=NONCANON
+
+# RUN: llvm-mc -filetype=obj -triple=riscv64 invalid_arch.s -o invalid_arch.o
+# RUN: not llvm-objdump -d invalid_arch.o 2>&1 | FileCheck %s --check-prefix=INVALID
+
+# RUN: llvm-mc -filetype=obj -triple=riscv32 unknown_i_version.s -o unknown_i_version.o
+# RUN: not llvm-objdump -d unknown_i_version.o 2>&1 | FileCheck %s --check-prefix=UNKNOWN-I-VERSION
+
+# RUN: llvm-mc -filetype=obj -triple=riscv32 -mattr=+zicbom unknown_ext_version.s -o unknown_ext_version.o
+# RUN: llvm-objdump -d unknown_ext_version.o 2>&1 | FileCheck %s --check-prefix=UNKNOWN-EXT-VERSION
+
+# RUN: llvm-mc -filetype=obj -triple=riscv64 unknown_ext_name.s -o unknown_ext_name.o
+# RUN: llvm-objdump -d unknown_ext_name.o 2>&1 | FileCheck %s --check-prefix=UNKNOWN-EXT-NAME
+
+#--- noncanonicalized_arch.s
+# NONCANON: vsetvli a3, a2, e8, m8, tu, mu
+vsetvli a3, a2, e8, m8, tu, mu
+
+.section .riscv.attributes,"",@0x70000003
+.byte 0x41
+.long .Lend-.riscv.attributes-1
+.asciz "riscv"  # vendor
+.Lbegin:
+.byte 1  # Tag_File
+.long .Lend-.Lbegin
+.byte 5  # Tag_RISCV_arch
+.asciz "rv64gcv"
+.Lend:
+
+#--- invalid_arch.s
+# INVALID: string must begin with rv32{i,e,g} or rv64{i,g} 
+nop
+
+.section .riscv.attributes,"",@0x70000003
+.byte 0x41
+.long .Lend-.riscv.attributes-1
+.asciz "riscv"  # vendor
+.Lbegin:
+.byte 1  # Tag_File
+.long .Lend-.Lbegin
+.byte 5  # Tag_RISCV_arch
+.asciz "nonsense"
+.Lend:
+
+#--- unknown_i_version.s
+# UNKNOWN-I-VERSION: unsupported version number 99.99 for extension 'i'
+nop
+
+.section .riscv.attributes,"",@0x70000003
+.byte 0x41
+.long .Lend-.riscv.attributes-1
+.asciz "riscv"  # vendor
+.Lbegin:
+.byte 1  # Tag_File
+.long .Lend-.Lbegin
+.byte 5  # Tag_RISCV_arch
+.asciz "rv32i99p99"
+.Lend:
+
+#--- unknown_ext_version.s
+# UNKNOWN-EXT-VERSION: <unknown>
+cbo.clean (t0)
+
+.section .riscv.attributes,"",@0x70000003
+.byte 0x41
+.long .Lend-.riscv.attributes-1
+.asciz "riscv"  # vendor
+.Lbegin:
+.byte 1  # Tag_File
+.long .Lend-.Lbegin
+.byte 5  # Tag_RISCV_arch
+.asciz "rv32i2p0_zicbom0p1"
+.Lend:
+
+#--- unknown_ext_name.s
+# UNKNOWN-EXT-NAME: nop
+nop
+
+.section .riscv.attributes,"",@0x70000003
+.byte 0x41
+.long .Lend-.riscv.attributes-1
+.asciz "riscv"  # vendor
+.Lbegin:
+.byte 1  # Tag_File
+.long .Lend-.Lbegin
+.byte 5  # Tag_RISCV_arch
+.asciz "rv32i2p0_zmadeup1p0_smadeup1p0_xmadeup1p0_sxmadeup1p0"
+.Lend:
diff --git a/llvm/test/tools/llvm-objdump/X86/elf-bbaddrmap-disassemble-symbolize-operands.yaml b/llvm/test/tools/llvm-objdump/X86/elf-bbaddrmap-disassemble-symbolize-operands.yaml
index 2d4a52211f812..cf4ac77b3d154 100644
--- a/llvm/test/tools/llvm-objdump/X86/elf-bbaddrmap-disassemble-symbolize-operands.yaml
+++ b/llvm/test/tools/llvm-objdump/X86/elf-bbaddrmap-disassemble-symbolize-operands.yaml
@@ -5,19 +5,12 @@
 # UNSUPPORTED: system-windows
 
 ## Executable object file.
-# RUN: yaml2obj --docnum=1 -DTYPE=ET_EXEC -DFOO_ADDR=0x4000 -DBAR_ADDR=0x5000 %s -o %t1
+# RUN: yaml2obj --docnum=1 -DFOO_ADDR=0x4000 -DBAR_ADDR=0x5000 %s -o %t1
 # RUN: llvm-objdump %t1 -d --symbolize-operands -M intel --no-show-raw-insn --no-leading-addr | \
 # RUN:   FileCheck %s -DSYM=symbol --match-full-lines --check-prefixes=INTEL
 # RUN: llvm-objdump %t1 -d --symbolize-operands -M att --no-show-raw-insn --no-leading-addr | \
 # RUN:   FileCheck %s -DSYM=symbol --match-full-lines --check-prefixes=ATT
 
-## Relocatable object file.
-# RUN: yaml2obj --docnum=1 -DTYPE=ET_REL -DFOO_ADDR=0x0 -DBAR_ADDR=0x0 %s -o %t2
-# RUN: llvm-objdump %t2 -d --symbolize-operands -M intel --no-show-raw-insn --no-leading-addr | \
-# RUN:   FileCheck %s -DSYM=foo+0x200c --match-full-lines --check-prefix=INTEL
-# RUN: llvm-objdump %t2 -d --symbolize-operands -M att --no-show-raw-insn --no-leading-addr | \
-# RUN:   FileCheck %s -DSYM=foo+0x200c --match-full-lines --check-prefix=ATT
-
 ## Executable object file with a single SHT_LLVM_BB_ADDR_MAP for multiple text sections.
 # RUN: yaml2obj --docnum=2 %s -o %t3
 # RUN: llvm-objdump %t3 -d --symbolize-operands -M intel --no-show-raw-insn --no-leading-addr | \
@@ -78,7 +71,7 @@
 FileHeader:
   Class:   ELFCLASS64
   Data:    ELFDATA2LSB
-  Type:    [[TYPE]]
+  Type:    ET_EXEC
   Machine: EM_X86_64
 Sections:
   - Name:    .text.foo
diff --git a/llvm/test/tools/llvm-objdump/X86/elf-bbaddrmap-symbolize-relocatable.yaml b/llvm/test/tools/llvm-objdump/X86/elf-bbaddrmap-symbolize-relocatable.yaml
new file mode 100644
index 0000000000000..d5a618467bd2c
--- /dev/null
+++ b/llvm/test/tools/llvm-objdump/X86/elf-bbaddrmap-symbolize-relocatable.yaml
@@ -0,0 +1,91 @@
+## Test that in the presence of SHT_LLVM_BB_ADDR_MAP sections,
+## --symbolize-operands can display <BB*> labels properly in a relocatable
+## object file.
+
+## Fails on windows (https://github.com/llvm/llvm-project/issues/60013).
+# UNSUPPORTED: system-windows
+
+## Relocatable Object file.
+# RUN: yaml2obj %s -o %t1
+# RUN: llvm-objdump %t1 -d --symbolize-operands -M att --no-show-raw-insn --no-leading-addr | \
+# RUN:   FileCheck %s -DSYM=symbol --match-full-lines
+
+--- !ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_REL
+  Machine:         EM_X86_64
+  SectionHeaderStringTable: .strtab
+Sections:
+  - Name:    .text
+    Type:    SHT_PROGBITS
+    Flags:   [ SHF_ALLOC, SHF_EXECINSTR ]
+    Content: FFFF
+  - Name:    .llvm_bb_addr_map
+    Type:    SHT_LLVM_BB_ADDR_MAP
+    Link:    .text
+    Entries:
+      - Version: 2
+        BBEntries:
+          - ID:              0
+            AddressOffset:   0x0
+            Size:            0x1
+            Metadata:        0x1
+      - Version: 2
+        BBEntries:
+          - ID:              0
+            AddressOffset:   0x0
+            Size:            0x1
+            Metadata:        0x1
+  - Name:  .rela.llvm_bb_addr_map
+    Type:  SHT_RELA
+    Flags: [ SHF_INFO_LINK ]
+    Link:  .symtab
+    Info:  .llvm_bb_addr_map
+    Relocations:
+      - Offset: 0x2
+        Symbol: .text
+        Type:   R_X86_64_64
+      - Offset: 0x11
+        Symbol: .text
+        Type:   R_X86_64_64
+        Addend: 1
+Symbols:
+  - Name:    a
+    Section: .text
+    Value:   0x0
+  - Name:    c
+    Section: .text
+    Value:   0x1
+  - Name:    .text
+    Type:    STT_SECTION
+    Section: .text
+
+# CHECK:      <a>:
+# CHECK-NEXT: <BB0>:
+# CHECK:      <c>:
+# CHECK-NEXT: <BB0>:
+
+## Check that if we're missing a relocation section we get the appropriate
+## warning.
+
+# RUN: yaml2obj %s --docnum=2 -o %t2
+# RUN: llvm-readobj %t2 --bb-addr-map 2>&1 | FileCheck %s --check-prefix=NO-RELA-SECTION -DFILE=%t2
+
+--- !ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_REL
+  Machine:         EM_X86_64
+Sections:
+  - Name:    .text
+    Type:    SHT_PROGBITS
+    Flags:   [ SHF_ALLOC, SHF_EXECINSTR ]
+    Content: FF
+  - Name:    .llvm_bb_addr_map
+    Type:    SHT_LLVM_BB_ADDR_MAP
+    Link:    .text
+
+# NO-RELA-SECTION: warning: '[[FILE]]': unable to get relocation section for SHT_LLVM_BB_ADDR_MAP section with index 2
diff --git a/llvm/test/tools/llvm-profdata/Inputs/basic.memprofexe b/llvm/test/tools/llvm-profdata/Inputs/basic.memprofexe
index 4d3e4798e4791..d03894b6d8e40 100755
Binary files a/llvm/test/tools/llvm-profdata/Inputs/basic.memprofexe and b/llvm/test/tools/llvm-profdata/Inputs/basic.memprofexe differ
diff --git a/llvm/test/tools/llvm-profdata/Inputs/basic.memprofraw b/llvm/test/tools/llvm-profdata/Inputs/basic.memprofraw
index 88da4a76e47f0..62b7d299d3aa1 100644
Binary files a/llvm/test/tools/llvm-profdata/Inputs/basic.memprofraw and b/llvm/test/tools/llvm-profdata/Inputs/basic.memprofraw differ
diff --git a/llvm/test/tools/llvm-profdata/Inputs/buildid.memprofexe b/llvm/test/tools/llvm-profdata/Inputs/buildid.memprofexe
new file mode 100755
index 0000000000000..5687d065d035d
Binary files /dev/null and b/llvm/test/tools/llvm-profdata/Inputs/buildid.memprofexe differ
diff --git a/llvm/test/tools/llvm-profdata/Inputs/buildid.memprofraw b/llvm/test/tools/llvm-profdata/Inputs/buildid.memprofraw
new file mode 100644
index 0000000000000..43f9ca99c28bb
Binary files /dev/null and b/llvm/test/tools/llvm-profdata/Inputs/buildid.memprofraw differ
diff --git a/llvm/test/tools/llvm-profdata/Inputs/inline.memprofexe b/llvm/test/tools/llvm-profdata/Inputs/inline.memprofexe
index 82aad1948982d..766d2c4713136 100755
Binary files a/llvm/test/tools/llvm-profdata/Inputs/inline.memprofexe and b/llvm/test/tools/llvm-profdata/Inputs/inline.memprofexe differ
diff --git a/llvm/test/tools/llvm-profdata/Inputs/inline.memprofraw b/llvm/test/tools/llvm-profdata/Inputs/inline.memprofraw
old mode 100755
new mode 100644
index f9f84e227b220..26254ae67c7c2
Binary files a/llvm/test/tools/llvm-profdata/Inputs/inline.memprofraw and b/llvm/test/tools/llvm-profdata/Inputs/inline.memprofraw differ
diff --git a/llvm/test/tools/llvm-profdata/Inputs/memprof-inline.exe b/llvm/test/tools/llvm-profdata/Inputs/memprof-inline.exe
deleted file mode 100755
index 9b6fd16e9a272..0000000000000
Binary files a/llvm/test/tools/llvm-profdata/Inputs/memprof-inline.exe and /dev/null differ
diff --git a/llvm/test/tools/llvm-profdata/Inputs/multi.memprofexe b/llvm/test/tools/llvm-profdata/Inputs/multi.memprofexe
index 507cf982993e7..d5c3ec65b3ac7 100755
Binary files a/llvm/test/tools/llvm-profdata/Inputs/multi.memprofexe and b/llvm/test/tools/llvm-profdata/Inputs/multi.memprofexe differ
diff --git a/llvm/test/tools/llvm-profdata/Inputs/multi.memprofraw b/llvm/test/tools/llvm-profdata/Inputs/multi.memprofraw
index b6b895ff96fa9..a1018aae2fba4 100644
Binary files a/llvm/test/tools/llvm-profdata/Inputs/multi.memprofraw and b/llvm/test/tools/llvm-profdata/Inputs/multi.memprofraw differ
diff --git a/llvm/test/tools/llvm-profdata/Inputs/pic.memprofexe b/llvm/test/tools/llvm-profdata/Inputs/pic.memprofexe
index eb1cd9b4bb778..d854186380e73 100755
Binary files a/llvm/test/tools/llvm-profdata/Inputs/pic.memprofexe and b/llvm/test/tools/llvm-profdata/Inputs/pic.memprofexe differ
diff --git a/llvm/test/tools/llvm-profdata/Inputs/pic.memprofraw b/llvm/test/tools/llvm-profdata/Inputs/pic.memprofraw
index 9b16c8f5dd0b5..992022fa9129c 100644
Binary files a/llvm/test/tools/llvm-profdata/Inputs/pic.memprofraw and b/llvm/test/tools/llvm-profdata/Inputs/pic.memprofraw differ
diff --git a/llvm/test/tools/llvm-profdata/Inputs/update_memprof_inputs.sh b/llvm/test/tools/llvm-profdata/Inputs/update_memprof_inputs.sh
new file mode 100755
index 0000000000000..5365a0bb6cc3a
--- /dev/null
+++ b/llvm/test/tools/llvm-profdata/Inputs/update_memprof_inputs.sh
@@ -0,0 +1,83 @@
+#!/bin/bash
+
+if [ -z $1 ]; then
+  echo "Path to clang required!"
+  echo "Usage: update_memprof_inputs.sh /path/to/updated/clang"
+  exit 1
+else
+  CLANG=$1
+fi
+
+# Allows the script to be invoked from other directories.
+OUTDIR=$(dirname $(realpath -s $0))
+
+read -r -d '' BASIC << EOF
+#include <stdlib.h>
+#include <string.h>
+int main(int argc, char **argv) {
+  char *x = (char *)malloc(10);
+  memset(x, 0, 10);
+  free(x);
+  x = (char *)malloc(10);
+  memset(x, 0, 10);
+  free(x);
+  return 0;
+}
+EOF
+
+read -r -d '' INLINE << EOF
+#include <stdlib.h>
+#include <string.h>
+
+__attribute__((always_inline))
+void qux(int x) {
+  char *ptr = malloc(x);
+  memset(ptr, 0, x);
+  free(ptr);
+}
+
+__attribute__((noinline))
+void foo(int x){ qux(x); }
+
+__attribute__((noinline))
+void bar(int x) { foo(x); }
+
+int main(int argc, char **argv) {
+  bar(argc);
+  return 0;
+}
+EOF
+
+read -r -d '' MULTI << EOF
+#include <sanitizer/memprof_interface.h>
+#include <stdlib.h>
+#include <string.h>
+int main(int argc, char **argv) {
+  char *x = (char *)malloc(10);
+  memset(x, 0, 10);
+  free(x);
+  __memprof_profile_dump();
+  x = (char *)malloc(10);
+  memset(x, 0, 10);
+  free(x);
+  return 0;
+}
+EOF
+
+DEFAULT_MEMPROF_FLAGS="-fuse-ld=lld -Wl,--no-rosegment -gmlt -fdebug-info-for-profiling -fmemory-profile -mno-omit-leaf-frame-pointer -fno-omit-frame-pointer -fno-optimize-sibling-calls -m64 -Wl,-build-id -no-pie"
+
+# Map each test to their source and any additional flags separated by ; 
+declare -A INPUTS
+INPUTS["basic"]="BASIC"
+INPUTS["inline"]="INLINE"
+INPUTS["multi"]="MULTI"
+INPUTS["pic"]="BASIC;-pie"
+INPUTS["buildid"]="BASIC;-Wl,-build-id=sha1"
+
+for name in "${!INPUTS[@]}"; do
+  IFS=";" read -r src flags <<< "${INPUTS[$name]}"
+  echo "${!src}" > ${OUTDIR}/${name}.c
+  ${CLANG} ${DEFAULT_MEMPROF_FLAGS} ${flags} ${OUTDIR}/${name}.c -o ${OUTDIR}/${name}.memprofexe
+  env MEMPROF_OPTIONS=log_path=stdout ${OUTDIR}/${name}.memprofexe > ${OUTDIR}/${name}.memprofraw
+  rm ${OUTDIR}/${name}.c
+done
diff --git a/llvm/test/tools/llvm-profdata/memprof-basic.test b/llvm/test/tools/llvm-profdata/memprof-basic.test
index a070845b5ab06..8eaa2fa1013f4 100644
--- a/llvm/test/tools/llvm-profdata/memprof-basic.test
+++ b/llvm/test/tools/llvm-profdata/memprof-basic.test
@@ -1,37 +1,6 @@
 REQUIRES: x86_64-linux
 
-The input raw profile test has been generated from the following source code:
-
-```
-#include <stdlib.h>
-#include <string.h>
-int main(int argc, char **argv) {
-  char *x = (char *)malloc(10);
-  memset(x, 0, 10);
-  free(x);
-  x = (char *)malloc(10);
-  memset(x, 0, 10);
-  free(x);
-  return 0;
-}
-```
-
-The following commands were used to compile the source to a memprof instrumented
-executable and collect a raw binary format profile. Since the profile contains
-virtual addresses for the callstack, we do not expect the raw binary profile to
-be deterministic. The summary should be deterministic apart from changes to
-the shared libraries linked in which could change the number of segments
-recorded.
-
-```
-clang -fuse-ld=lld -Wl,--no-rosegment -gmlt -fdebug-info-for-profiling \
-      -fmemory-profile -mno-omit-leaf-frame-pointer -fno-omit-frame-pointer \
-      -fno-optimize-sibling-calls -m64 -Wl,-build-id -no-pie \
-      source.c -o basic.memprofexe
-
-env MEMPROF_OPTIONS=log_path=stdout ./basic.memprofexe > basic.memprofraw
-```
-
+To update the inputs used below run Inputs/update_memprof_inputs.sh /path/to/updated/clang
 RUN: llvm-profdata show --memory %p/Inputs/basic.memprofraw --profiled-binary %p/Inputs/basic.memprofexe -o - | FileCheck %s
 
 We expect 2 MIB entries, 1 each for the malloc calls in the program. Any
@@ -39,17 +8,17 @@ additional allocations which do not originate from the main binary are pruned.
 
 CHECK:  MemprofProfile:
 CHECK-NEXT:   Summary:
-CHECK-NEXT:     Version: 2
+CHECK-NEXT:     Version: 3
 CHECK-NEXT:     NumSegments: {{[0-9]+}}
 CHECK-NEXT:     NumMibInfo: 2
 CHECK-NEXT:     NumAllocFunctions: 1
 CHECK-NEXT:     NumStackOffsets: 2
 CHECK-NEXT:   Segments:
 CHECK-NEXT:   -
-CHECK-NEXT:     BuildId: <None>
-CHECK-NEXT:     Start: 0x{{[0-9]+}}
-CHECK-NEXT:     End: 0x{{[0-9]+}}
-CHECK-NEXT:     Offset: 0x{{[0-9]+}}
+CHECK-NEXT:     BuildId: {{[[:xdigit:]]+}}
+CHECK-NEXT:     Start: 0x{{[[:xdigit:]]+}}
+CHECK-NEXT:     End: 0x{{[[:xdigit:]]+}}
+CHECK-NEXT:     Offset: 0x{{[[:xdigit:]]+}}
 CHECK-NEXT:   -
 
 CHECK:   Records:
diff --git a/llvm/test/tools/llvm-profdata/memprof-buildid.test b/llvm/test/tools/llvm-profdata/memprof-buildid.test
new file mode 100644
index 0000000000000..9b055d3a70bbe
--- /dev/null
+++ b/llvm/test/tools/llvm-profdata/memprof-buildid.test
@@ -0,0 +1,12 @@
+REQUIRES: x86_64-linux
+
+To update the inputs used below run Inputs/update_memprof_inputs.sh /path/to/updated/clang
+RUN: llvm-readelf --notes %p/Inputs/buildid.memprofexe > %t1.txt
+RUN: llvm-profdata show --memory %p/Inputs/buildid.memprofraw --profiled-binary %p/Inputs/buildid.memprofexe -o -  > %t2.txt
+RUN: cat %t1.txt %t2.txt | FileCheck %s
+
+COM: First extract the id from the llvm-readelf output.
+CHECK: Build ID: [[ID:[[:xdigit:]]+]]
+
+COM: Then match it with the profdata output.
+CHECK: BuildId: {{.*}}[[ID]]
diff --git a/llvm/test/tools/llvm-profdata/memprof-inline.test b/llvm/test/tools/llvm-profdata/memprof-inline.test
index 87eaa8364d308..dd842c0542083 100644
--- a/llvm/test/tools/llvm-profdata/memprof-inline.test
+++ b/llvm/test/tools/llvm-profdata/memprof-inline.test
@@ -1,56 +1,21 @@
 REQUIRES: x86_64-linux
 
-The input raw profile test has been generated from the following source code:
-
-```
-#include <stdlib.h>
-#include <string.h>
-
-__attribute__((always_inline))
-void qux(int x) {
-  char *ptr = malloc(x);
-  memset(ptr, 0, x);
-  free(ptr);
-}
-
-__attribute__((noinline))
-void foo(int x){ qux(x); }
-
-__attribute__((noinline))
-void bar(int x) { foo(x); }
-
-int main(int argc, char **argv) {
-  bar(argc);
-  return 0;
-}
-```
-
-Compile and run with the following commands:
-
-```
-bin/clang -fuse-ld=lld -Wl,--no-rosegment -gmlt -fdebug-info-for-profiling \
-          -fmemory-profile -mno-omit-leaf-frame-pointer -fno-omit-frame-pointer \
-          -fno-optimize-sibling-calls -m64 -Wl,-build-id -no-pie \
-          inline.c -o inline.memprofexe
-
-env MEMPROF_OPTIONS=log_path=stdout ./inline.memprofexe > inline.memprofraw
-```
-
+To update the inputs used below run Inputs/update_memprof_inputs.sh /path/to/updated/clang
 RUN: llvm-profdata show --memory %p/Inputs/inline.memprofraw --profiled-binary %p/Inputs/inline.memprofexe | FileCheck %s
 
 CHECK:  MemprofProfile:
 CHECK-NEXT:  Summary:
-CHECK-NEXT:    Version: 2
+CHECK-NEXT:    Version: 3
 CHECK-NEXT:    NumSegments: {{[0-9]+}}
 CHECK-NEXT:    NumMibInfo: 2
 CHECK-NEXT:    NumAllocFunctions: 2
 CHECK-NEXT:    NumStackOffsets: 1
 CHECK-NEXT:  Segments:
 CHECK-NEXT:  -
-CHECK-NEXT:    BuildId: <None>
-CHECK-NEXT:    Start: 0x{{[0-9]+}}
-CHECK-NEXT:    End: 0x{{[0-9]+}}
-CHECK-NEXT:    Offset: 0x{{[0-9]+}}
+CHECK-NEXT:    BuildId: {{[[:xdigit:]]+}}
+CHECK-NEXT:    Start: 0x{{[[:xdigit:]]+}}
+CHECK-NEXT:    End: 0x{{[[:xdigit:]]+}}
+CHECK-NEXT:    Offset: 0x{{[[:xdigit:]]+}}
 CHECK-NEXT:  -
 
 CHECK:  Records:
diff --git a/llvm/test/tools/llvm-profdata/memprof-merge.test b/llvm/test/tools/llvm-profdata/memprof-merge.test
index 46de2adc9941e..46a07957f863a 100644
--- a/llvm/test/tools/llvm-profdata/memprof-merge.test
+++ b/llvm/test/tools/llvm-profdata/memprof-merge.test
@@ -1,40 +1,13 @@
 REQUIRES: x86_64-linux
 
-The input memprof and instrumented raw profiles were generated from the following source code:
-
-```
-#include <stdlib.h>
-#include <string.h>
-int main(int argc, char **argv) {
-  char *x = (char *)malloc(10);
-  memset(x, 0, 10);
-  free(x);
-  x = (char *)malloc(10);
-  memset(x, 0, 10);
-  free(x);
-  return 0;
-}
-```
-
-Steps to collect the memprof raw profile and the instrprof raw profile:
-
-```
-# Collect instrprof profile with name compression disabled since some buildbots
-# do not have zlib.
-clang -mllvm -enable-name-compression=false -fprofile-generate source.c -o instr.out
-./instr.out
-mv *.profraw basic.profraw
-
-# Collect memprof profile.
-clang -fuse-ld=lld -Wl,--no-rosegment -gmlt -fdebug-info-for-profiling \
-      -fmemory-profile -mno-omit-leaf-frame-pointer -fno-omit-frame-pointer \
-      -fno-optimize-sibling-calls -m64 -Wl,-build-id -no-pie \
-      source.c -o basic.memprofexe
-
-env MEMPROF_OPTIONS=log_path=stdout ./basic.memprofexe > basic.memprofraw
-```
-
-RUN: llvm-profdata merge %p/Inputs/basic.profraw %p/Inputs/basic.memprofraw --profiled-binary %p/Inputs/basic.memprofexe -o %t.prof
+RUN: echo ":ir" > %t.proftext 
+RUN: echo "main" >> %t.proftext
+RUN: echo "742261418966908927" >> %t.proftext
+RUN: echo "1" >> %t.proftext
+RUN: echo "1" >> %t.proftext
+
+To update the inputs used below run Inputs/update_memprof_inputs.sh /path/to/updated/clang
+RUN: llvm-profdata merge %t.proftext %p/Inputs/basic.memprofraw --profiled-binary %p/Inputs/basic.memprofexe -o %t.prof
 RUN: llvm-profdata show %t.prof | FileCheck %s
 
 For now we only check the validity of the instrumented profile since we don't
diff --git a/llvm/test/tools/llvm-profdata/memprof-multi.test b/llvm/test/tools/llvm-profdata/memprof-multi.test
index cb5b7dfaeb0a7..f3cdbd1f5266c 100644
--- a/llvm/test/tools/llvm-profdata/memprof-multi.test
+++ b/llvm/test/tools/llvm-profdata/memprof-multi.test
@@ -1,46 +1,13 @@
 REQUIRES: x86_64-linux
 
-The input raw profile test has been generated from the following source code:
-
-```
-#include <sanitizer/memprof_interface.h>
-#include <stdlib.h>
-#include <string.h>
-int main(int argc, char **argv) {
-  char *x = (char *)malloc(10);
-  memset(x, 0, 10);
-  free(x);
-  __memprof_profile_dump();
-  x = (char *)malloc(10);
-  memset(x, 0, 10);
-  free(x);
-  return 0;
-}
-```
-
-The following commands were used to compile the source to a memprof instrumented
-executable and collect a raw binary format profile. Since the profile contains
-virtual addresses for the callstack, we do not expect the raw binary profile to
-be deterministic. The summary should be deterministic apart from changes to
-the shared libraries linked in which could change the number of segments
-recorded.
-
-```
-clang -fuse-ld=lld -Wl,--no-rosegment -gmlt -fdebug-info-for-profiling \
-      -fmemory-profile -mno-omit-leaf-frame-pointer -fno-omit-frame-pointer \
-      -fno-optimize-sibling-calls -m64 -Wl,-build-id -no-pie \
-      source.c -o multi.memprofexe
-
-env MEMPROF_OPTIONS=log_path=stdout ./multi.memprofexe > multi.memprofraw
-```
-
+To update the inputs used below run Inputs/update_memprof_inputs.sh /path/to/updated/clang
 RUN: llvm-profdata show --memory %p/Inputs/multi.memprofraw --profiled-binary %p/Inputs/multi.memprofexe -o - | FileCheck %s
 
 We expect 2 MIB entries, 1 each for the malloc calls in the program.
 
 CHECK:  MemprofProfile:
 CHECK-NEXT:  Summary:
-CHECK-NEXT:    Version: 2
+CHECK-NEXT:    Version: 3
 CHECK-NEXT:    NumSegments: {{[0-9]+}}
 CHECK-NEXT:    NumMibInfo: 2
 CHECK-NEXT:    NumAllocFunctions: 1
diff --git a/llvm/test/tools/llvm-profdata/memprof-pic.test b/llvm/test/tools/llvm-profdata/memprof-pic.test
index f993c441ceaa5..fe55578b80731 100644
--- a/llvm/test/tools/llvm-profdata/memprof-pic.test
+++ b/llvm/test/tools/llvm-profdata/memprof-pic.test
@@ -1,40 +1,12 @@
 REQUIRES: x86_64-linux
 
-This test ensures that llvm-profdata fails with a descriptive error message
-when invoked on a memprof profiled binary which was built with position
-independent code.
-
-The input raw profile test has been generated from the following source code:
-
-```
-#include <stdlib.h>
-#include <string.h>
-int main(int argc, char **argv) {
-  char *x = (char *)malloc(10);
-  memset(x, 0, 10);
-  free(x);
-  x = (char *)malloc(10);
-  memset(x, 0, 10);
-  free(x);
-  return 0;
-}
-```
-
-The following commands were used to compile the source to a memprof instrumented
-executable and collect a raw binary format profile. Since the profile contains
-virtual addresses for the callstack, we do not expect the raw binary profile to
-be deterministic. The summary should be deterministic apart from changes to
-the shared libraries linked in which could change the number of segments
+Since the profile contains virtual addresses for the callstack,
+we do not expect the raw binary profile to be deterministic. The
+summary should be deterministic apart from changes to the shared
+libraries linked in which could change the number of segments
 recorded.
 
-```
-clang -fuse-ld=lld -Wl,--no-rosegment -gmlt -fdebug-info-for-profiling \
-      -fmemory-profile -mno-omit-leaf-frame-pointer -fno-omit-frame-pointer \
-      -fno-optimize-sibling-calls -m64 -Wl,-build-id -pie \
-      source.c -o pic.memprofexe
-
-env MEMPROF_OPTIONS=log_path=stdout ./pic.memprofexe > pic.memprofraw
-```
-
+To update the inputs used below run Inputs/update_memprof_inputs.sh /path/to/updated/clang
 RUN: not llvm-profdata show --memory %p/Inputs/pic.memprofraw --profiled-binary %p/Inputs/pic.memprofexe -o - 2>&1 | FileCheck %s
+
 CHECK: Unsupported position independent code
diff --git a/llvm/test/tools/llvm-profdata/merge-incompatible.test b/llvm/test/tools/llvm-profdata/merge-incompatible.test
index db77af9a8a54a..fca72a452069e 100644
--- a/llvm/test/tools/llvm-profdata/merge-incompatible.test
+++ b/llvm/test/tools/llvm-profdata/merge-incompatible.test
@@ -1,3 +1,4 @@
+RUN: export LSAN_OPTIONS=detect_leaks=0
 RUN: not llvm-profdata merge %p/Inputs/fe-basic.proftext %p/Inputs/ir-basic.proftext -o /dev/null 2>&1 | FileCheck %s
 CHECK: ir-basic.proftext: Merge IR generated profile with Clang generated profile.
 
diff --git a/llvm/test/tools/llvm-readobj/ELF/aarch64-symbols-stother.test b/llvm/test/tools/llvm-readobj/ELF/aarch64-symbols-stother.test
index bc9d1286e0a0d..a223c519c819e 100644
--- a/llvm/test/tools/llvm-readobj/ELF/aarch64-symbols-stother.test
+++ b/llvm/test/tools/llvm-readobj/ELF/aarch64-symbols-stother.test
@@ -2,6 +2,7 @@
 
 # RUN: yaml2obj %s -o %t.o
 # RUN: llvm-readobj --symbols %t.o | FileCheck %s --check-prefix=LLVM
+# RUN: llvm-readobj --symbols %t.o --elf-output-style=JSON --pretty-print | FileCheck %s --check-prefix=JSON
 # RUN: llvm-readelf --symbols %t.o | FileCheck %s --check-prefix=GNU
 
 # LLVM:      Name: foo1
@@ -28,6 +29,54 @@
 # GNU-NEXT: 3: 0000000000000000 0 NOTYPE LOCAL PROTECTED [VARIANT_PCS]      UND foo3
 # GNU-NEXT: 4: 0000000000000000 0 NOTYPE LOCAL PROTECTED                    UND foo4
 
+# JSON:     "Name": "foo1",
+# JSON:     "Other": {
+# JSON-NEXT:  "Value": 128,
+# JSON-NEXT:  "Flags": [
+# JSON-NEXT:    {
+# JSON-NEXT:      "Name": "STO_AARCH64_VARIANT_PCS",
+# JSON-NEXT:      "Value": 128
+# JSON-NEXT:    }
+# JSON-NEXT:  ]
+# JSON-NEXT:},
+
+# JSON:     "Name": "foo2",
+# JSON:     "Other": {
+# JSON-NEXT:  "Value": 192,
+# JSON-NEXT:  "Flags": [
+# JSON-NEXT:    {
+# JSON-NEXT:      "Name": "STO_AARCH64_VARIANT_PCS",
+# JSON-NEXT:      "Value": 128
+# JSON-NEXT:    }
+# JSON-NEXT:  ]
+# JSON-NEXT:},
+
+# JSON:     "Name": "foo3",
+# JSON:     "Other": {
+# JSON-NEXT:  "Value": 131,
+# JSON-NEXT:  "Flags": [
+# JSON-NEXT:    {
+# JSON-NEXT:      "Name": "STO_AARCH64_VARIANT_PCS",
+# JSON-NEXT:      "Value": 128
+# JSON-NEXT:    },
+# JSON-NEXT:    {
+# JSON-NEXT:      "Name": "STV_PROTECTED",
+# JSON-NEXT:      "Value": 3
+# JSON-NEXT:    }
+# JSON-NEXT:  ]
+# JSON-NEXT:},
+
+# JSON:     "Name": "foo4",
+# JSON:     "Other": {
+# JSON-NEXT:  "Value": 3,
+# JSON-NEXT:  "Flags": [
+# JSON-NEXT:    {
+# JSON-NEXT:      "Name": "STV_PROTECTED",
+# JSON-NEXT:      "Value": 3
+# JSON-NEXT:    }
+# JSON-NEXT:  ]
+# JSON-NEXT:},
+
 --- !ELF
 FileHeader:
   Class:   ELFCLASS64
diff --git a/llvm/test/tools/llvm-readobj/ELF/bb-addr-map-relocatable.test b/llvm/test/tools/llvm-readobj/ELF/bb-addr-map-relocatable.test
new file mode 100644
index 0000000000000..f59a694ae52e4
--- /dev/null
+++ b/llvm/test/tools/llvm-readobj/ELF/bb-addr-map-relocatable.test
@@ -0,0 +1,211 @@
+## This test checks how we handle the --bb-addr-map option on relocatable
+## object files.
+
+## Fails on windows (https://github.com/llvm/llvm-project/issues/60013).
+# UNSUPPORTED: system-windows
+
+# RUN: yaml2obj %s -o %t1.o
+# RUN: llvm-readobj %t1.o --bb-addr-map | FileCheck %s
+
+# CHECK:      BBAddrMap [
+# CHECK-NEXT:   Function {
+# CHECK-NEXT:     At: 0x0
+# CHECK-NEXT:     Name: <?>
+# CHECK-NEXT:     BB entries [
+# CHECK-NEXT:       {
+# CHECK-NEXT:         ID: 0
+# CHECK-NEXT:         Offset: 0x0
+# CHECK-NEXT:         Size: 0xF
+# CHECK-NEXT:         HasReturn: Yes
+# CHECK-NEXT:         HasTailCall: No
+# CHECK-NEXT:         IsEHPad: No
+# CHECK-NEXT:         CanFallThrough: No
+# CHECK-NEXT:       }
+# CHECK-NEXT:     ]
+# CHECK-NEXT:   }
+# CHECK-NEXT:   Function {
+# CHECK-NEXT:     At: 0x10
+# CHECK-NEXT:     Name: <?>
+# CHECK-NEXT:     BB entries [
+# CHECK-NEXT:       {
+# CHECK-NEXT:         ID: 0
+# CHECK-NEXT:         Offset: 0x0
+# CHECK-NEXT:         Size: 0x11
+# CHECK-NEXT:         HasReturn: No
+# CHECK-NEXT:         HasTailCall: No
+# CHECK-NEXT:         IsEHPad: No
+# CHECK-NEXT:         CanFallThrough: Yes
+# CHECK-NEXT:       }
+# CHECK-NEXT:     ]
+# CHECK-NEXT:   }
+# CHECK-NEXT: ]
+
+--- !ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_REL
+  Machine:         EM_X86_64
+Sections:
+  - Name:    .text
+    Type:    SHT_PROGBITS
+    Flags:   [ SHF_ALLOC, SHF_EXECINSTR ]
+  - Name:    .llvm_bb_addr_map
+    Type:    SHT_LLVM_BB_ADDR_MAP
+    Link:    .text
+    Entries:
+      - Version: 2
+        BBEntries:
+          - ID:              0
+            AddressOffset:   0x0
+            Size:            0xF
+            Metadata:        0x1
+      - Version: 2
+        BBEntries:
+          - ID:              0
+            AddressOffset:   0x0
+            Size:            0x11
+            Metadata:        0x8
+  - Name:  .rela.llvm_bb_addr_map
+    Type:  SHT_RELA
+    Flags: [ SHF_INFO_LINK ]
+    Link:  .symtab
+    Info:  .llvm_bb_addr_map
+    Relocations:
+      - Offset: 0x2
+        Symbol: .text
+        Type:   R_X86_64_64
+      - Offset: 0x11
+        Symbol: .text
+        Type:   R_X86_64_64
+        Addend: 16
+Symbols:
+  - Name:    a
+    Section: .text
+    Value:   0x0
+  - Name:    c
+    Section: .text
+    Value:   0x10
+  - Name:    .text
+    Type:    STT_SECTION
+    Section: .text
+
+## Check that if we have a relocatable file and no relocation section for
+## a SHT_LLVM_BB_ADDR_MAP section, we give the appropriate warnings.
+
+--- !ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_REL
+  Machine:         EM_X86_64
+Sections:
+  - Name:    .text
+    Type:    SHT_PROGBITS
+    Flags:   [ SHF_ALLOC, SHF_EXECINSTR ]
+  - Name:    .llvm_bb_addr_map
+    Type:    SHT_LLVM_BB_ADDR_MAP
+    Link:    .text
+
+# RUN: yaml2obj %s --docnum=2 -o %t2.o
+# RUN: llvm-readobj %t2.o --bb-addr-map 2>&1 | FileCheck %s --check-prefix=NO-RELA -DFILE=%t2.o
+
+# NO-RELA: warning: '[[FILE]]': unable to get relocation section for SHT_LLVM_BB_ADDR_MAP section with index 2
+
+## Check that we get a warning if we expect a relocation and it is not present.
+
+--- !ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_REL
+  Machine:         EM_X86_64
+Sections:
+  - Name:    .text
+    Type:    SHT_PROGBITS
+    Flags:   [ SHF_ALLOC, SHF_EXECINSTR ]
+  - Name:    .llvm_bb_addr_map
+    Type:    SHT_LLVM_BB_ADDR_MAP
+    Link:    .text
+    Entries:
+      - Version: 2
+        BBEntries:
+          - ID:              0
+            AddressOffset:   0x0
+            Size:            0xF
+            Metadata:        0x1
+  - Name:  .rela.llvm_bb_addr_map
+    Type:  SHT_RELA
+    Flags: [ SHF_INFO_LINK ]
+    Info:  .llvm_bb_addr_map
+
+# RUN: yaml2obj %s --docnum=3 -o %t3.o
+# RUN: llvm-readobj %t3.o --bb-addr-map 2>&1 | FileCheck %s --check-prefix=MISSING-RELOCATION -DFILE=%t3.o
+
+# MISSING-RELOCATION: warning: '[[FILE]]': unable to dump SHT_LLVM_BB_ADDR_MAP section with index 2: failed to get relocation data for offset: 2 in section SHT_LLVM_BB_ADDR_MAP section with index 2
+
+## Check that if we have a missing relocated section, we fail and give the
+## appropriate warning.
+
+--- !ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_REL
+  Machine:         EM_X86_64
+Sections:
+  - Name:  .rela.llvm_bb_addr_map
+    Type:  SHT_RELA
+    Flags: [ SHF_INFO_LINK ]
+    Info:  0xFF
+
+# RUN: yaml2obj %s --docnum=4 -o %t4.o
+# RUN: llvm-readobj %t4.o --bb-addr-map 2>&1 | FileCheck %s --check-prefix=NO-RELOCATED-SECTION -DFILE=%t4.o
+
+# NO-RELOCATED-SECTION: warning: '[[FILE]]': failed to get SHT_LLVM_BB_ADDR_MAP section(s): SHT_RELA section with index 1: failed to get a relocated section: invalid section index: 255
+
+## Check that if we have an ET_DYN file with a .rela.dyn section, we don't get
+## a warning about a missing relocation section and can get the baddrmap data.
+
+--- !ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_DYN
+  Machine:         EM_X86_64
+Sections:
+  - Name:            .rela.dyn
+    Type:            SHT_RELA
+    Flags:           [ SHF_ALLOC ]
+  - Name:            .llvm_bb_addr_map
+    Type:            SHT_LLVM_BB_ADDR_MAP
+    Flags:           [ SHF_LINK_ORDER ]
+    Entries:
+      - Version:         2
+        Address:         0xF
+        BBEntries:
+          - ID:              0
+            AddressOffset:   0x0
+            Size:            0xF
+            Metadata:        0x1
+
+# RUN: yaml2obj %s --docnum=5 -o %t5.o
+# RUN: llvm-readobj %t5.o --bb-addr-map 2>&1 | FileCheck %s --check-prefix=ET-DYN-NO-WARNING -DFILE=%t5.o
+
+# ET-DYN-NO-WARNING: BBAddrMap [
+# ET-DYN-NO-WARNING:   Function {
+# ET-DYN-NO-WARNING:     At: 0xF
+# ET-DYN-NO-WARNING:     Name: <?>
+# ET-DYN-NO-WARNING:     BB entries [
+# ET-DYN-NO-WARNING:       {
+# ET-DYN-NO-WARNING:         ID: 0
+# ET-DYN-NO-WARNING:         Offset: 0x0
+# ET-DYN-NO-WARNING:         Size: 0xF
+# ET-DYN-NO-WARNING:         HasReturn: Yes
+# ET-DYN-NO-WARNING:         HasTailCall: No
+# ET-DYN-NO-WARNING:         IsEHPad: No
+# ET-DYN-NO-WARNING:         CanFallThrough: No
+# ET-DYN-NO-WARNING:       }
+# ET-DYN-NO-WARNING:     ]
+# ET-DYN-NO-WARNING:   }
+# ET-DYN-NO-WARNING: ]
diff --git a/llvm/test/tools/llvm-readobj/ELF/call-graph-profile.test b/llvm/test/tools/llvm-readobj/ELF/call-graph-profile.test
index 99532a62957fc..5ca5a6682f549 100644
--- a/llvm/test/tools/llvm-readobj/ELF/call-graph-profile.test
+++ b/llvm/test/tools/llvm-readobj/ELF/call-graph-profile.test
@@ -350,3 +350,23 @@ Sections:
 Symbols:
   - Name: foo
   - Name: bar
+
+## Check that we report a warning when we fail to get a section associated with
+## a relocation section.
+
+# RUN: yaml2obj %s --docnum=8 -o %t9.o
+# RUN: llvm-readobj %t9.o --cg-profile 2>&1 | FileCheck %s -DFILE=%t9.o --check-prefix=LLVM-RELOC-NO-SECTIONS
+# RUN: llvm-readobj %t9.o --elf-cg-profile 2>&1 | FileCheck %s -DFILE=%t9.o --check-prefix=LLVM-RELOC-NO-SECTIONS
+
+# LLVM-RELOC-NO-SECTIONS: warning: '[[FILE]]': unable to get CG Profile section(s): SHT_RELA section with index 1: failed to get a relocated section: invalid section index: 255
+
+--- !ELF
+FileHeader:
+  Class: ELFCLASS64
+  Data:  ELFDATA2LSB
+  Type:  ET_DYN
+  Machine: EM_X86_64
+Sections:
+  - Name: .rela.llvm.call-graph-profile
+    Type: SHT_RELA
+    Info: 0xFF
diff --git a/llvm/test/tools/llvm-readobj/ELF/groups.test b/llvm/test/tools/llvm-readobj/ELF/groups.test
index afa5fb05b91ea..8bb80c019ee7f 100644
--- a/llvm/test/tools/llvm-readobj/ELF/groups.test
+++ b/llvm/test/tools/llvm-readobj/ELF/groups.test
@@ -4,6 +4,7 @@
 # RUN: yaml2obj %s -o %t.o
 # RUN: llvm-readobj -g %t.o | FileCheck %s
 # RUN: llvm-readobj --elf-section-groups %t.o | FileCheck %s
+# RUN: llvm-readobj --elf-section-groups %t.o --elf-output-style=JSON --pretty-print | FileCheck %s --check-prefix=JSON
 # RUN: llvm-readelf -g %t.o | FileCheck --check-prefix=GNU %s
 
 # CHECK:      Groups {
@@ -41,6 +42,57 @@
 # GNU-NEXT:  [    5] .text.bar
 # GNU-NEXT:  [    6] .rela.text.bar
 
+# JSON:      "Groups": {
+# JSON-NEXT:   "Group": {
+# JSON-NEXT:     "Name": {
+# JSON-NEXT:       "Name": ".group",
+# JSON-NEXT:       "Value": 16
+# JSON-NEXT:     },
+# JSON-NEXT:     "Index": 1,
+# JSON-NEXT:     "Link": 7,
+# JSON-NEXT:     "Info": 1,
+# JSON-NEXT:     "Type": {
+# JSON-NEXT:       "Name": "COMDAT",
+# JSON-NEXT:       "Value": 1
+# JSON-NEXT:     },
+# JSON-NEXT:     "Signature": "foo",
+# JSON-NEXT:     "GroupSections": [
+# JSON-NEXT:       {
+# JSON-NEXT:         "Name": ".text.foo",
+# JSON-NEXT:         "Index": 3
+# JSON-NEXT:       },
+# JSON-NEXT:       {
+# JSON-NEXT:         "Name": ".rela.text.foo",
+# JSON-NEXT:         "Index": 4
+# JSON-NEXT:       }
+# JSON-NEXT:     ]
+# JSON-NEXT:   },
+# JSON-NEXT:   "Group": {
+# JSON-NEXT:     "Name": {
+# JSON-NEXT:       "Name": ".group1",
+# JSON-NEXT:       "Value": 64
+# JSON-NEXT:     },
+# JSON-NEXT:     "Index": 2,
+# JSON-NEXT:     "Link": 7,
+# JSON-NEXT:     "Info": 2,
+# JSON-NEXT:     "Type": {
+# JSON-NEXT:       "Name": "COMDAT",
+# JSON-NEXT:       "Value": 1
+# JSON-NEXT:     },
+# JSON-NEXT:     "Signature": "bar",
+# JSON-NEXT:     "GroupSections": [
+# JSON-NEXT:       {
+# JSON-NEXT:         "Name": ".text.bar",
+# JSON-NEXT:         "Index": 5
+# JSON-NEXT:       },
+# JSON-NEXT:       {
+# JSON-NEXT:         "Name": ".rela.text.bar",
+# JSON-NEXT:         "Index": 6
+# JSON-NEXT:       }
+# JSON-NEXT:     ]
+# JSON-NEXT:   }
+# JSON-NEXT: }
+
 --- !ELF
 FileHeader:
   Class: ELFCLASS64
diff --git a/llvm/test/tools/llvm-readobj/ELF/mips-symbols-stother.test b/llvm/test/tools/llvm-readobj/ELF/mips-symbols-stother.test
index 37415faea8abc..cc6a13928203b 100644
--- a/llvm/test/tools/llvm-readobj/ELF/mips-symbols-stother.test
+++ b/llvm/test/tools/llvm-readobj/ELF/mips-symbols-stother.test
@@ -2,6 +2,7 @@
 
 # RUN: yaml2obj %s -o %t.o
 # RUN: llvm-readobj --symbols %t.o | FileCheck %s --strict-whitespace --check-prefix=MIPS-LLVM
+# RUN: llvm-readobj --symbols %t.o --elf-output-style=JSON --pretty-print | FileCheck %s --check-prefix=MIPS-JSON
 # RUN: llvm-readelf --symbols %t.o | FileCheck %s --strict-whitespace --check-prefix=MIPS-GNU
 
 #      MIPS-LLVM:Name: foo
@@ -23,6 +24,39 @@
 # MIPS-GNU-NEXT:    1: 00000000     0 NOTYPE  LOCAL  DEFAULT [<other: 0xac>] UND foo
 # MIPS-GNU-NEXT:    2: 00000000     0 NOTYPE  LOCAL  DEFAULT [<other: 0xf0>] UND bar
 
+# MIPS-JSON:      "Name": "foo",
+# MIPS-JSON:      "Other": {
+# MIPS-JSON-NEXT:   "Value": 172,
+# MIPS-JSON-NEXT:   "Flags": [
+# MIPS-JSON-NEXT:     {
+# MIPS-JSON-NEXT:       "Name": "STO_MIPS_MICROMIPS",
+# MIPS-JSON-NEXT:       "Value": 128
+# MIPS-JSON-NEXT:     },
+# MIPS-JSON-NEXT:     {
+# MIPS-JSON-NEXT:       "Name": "STO_MIPS_OPTIONAL",
+# MIPS-JSON-NEXT:       "Value": 4
+# MIPS-JSON-NEXT:     },
+# MIPS-JSON-NEXT:     {
+# MIPS-JSON-NEXT:       "Name": "STO_MIPS_PIC",
+# MIPS-JSON-NEXT:       "Value": 32
+# MIPS-JSON-NEXT:     },
+# MIPS-JSON-NEXT:     {
+# MIPS-JSON-NEXT:       "Name": "STO_MIPS_PLT",
+# MIPS-JSON-NEXT:       "Value": 8
+# MIPS-JSON-NEXT:     }
+# MIPS-JSON-NEXT:   ]
+# MIPS-JSON-NEXT: },
+# MIPS-JSON:      "Name": "bar",
+# MIPS-JSON:      "Other": {
+# MIPS-JSON-NEXT:   "Value": 240,
+# MIPS-JSON-NEXT:   "Flags": [
+# MIPS-JSON-NEXT:     {
+# MIPS-JSON-NEXT:       "Name": "STO_MIPS_MIPS16",
+# MIPS-JSON-NEXT:       "Value": 240
+# MIPS-JSON-NEXT:     }
+# MIPS-JSON-NEXT:   ]
+# MIPS-JSON-NEXT: },
+
 --- !ELF
 FileHeader:
   Class:   ELFCLASS32
diff --git a/llvm/test/tools/llvm-readobj/ELF/relocations.test b/llvm/test/tools/llvm-readobj/ELF/relocations.test
index 360d3689d6651..34cff4098514d 100644
--- a/llvm/test/tools/llvm-readobj/ELF/relocations.test
+++ b/llvm/test/tools/llvm-readobj/ELF/relocations.test
@@ -47,8 +47,7 @@
 # GNU-64-NEXT:ffffffffffffffff  0000000800000001 R_X86_64_64            0000000000000003 rela_minneg - 8000000000000000
 # GNU-64-NEXT:0000000000000009  000000090000000b R_X86_64_32S           ffffffffffffffff rela_maxpos + 7fffffffffffffff
 
-## Show that --expand-relocs expands the relocation dump for LLVM style only
-## (and not GNU).
+## Show that --expand-relocs expands the relocation dump for LLVM style and not GNU.
 # RUN: llvm-readobj -r --expand-relocs %t64 \
 # RUN:   | FileCheck %s --check-prefix=LLVM-EXPAND-64 --match-full-lines --strict-whitespace
 # RUN: llvm-readelf -r --expand-relocs %t64 \
@@ -111,6 +110,147 @@
 # LLVM-EXPAND-64-NEXT:  }
 # LLVM-EXPAND-64-NEXT:]
 
+## Show that JSON relocations are always expanded.
+# RUN: llvm-readobj -r %t64 --elf-output-style=JSON --pretty-print \
+# RUN:   | FileCheck %s --check-prefix=JSON-EXPAND-64
+# RUN: llvm-readobj -r --expand-relocs %t64 --elf-output-style=JSON --pretty-print \
+# RUN:   | FileCheck %s --check-prefix=JSON-EXPAND-64
+
+# JSON-EXPAND-64:      "Relocations": [
+# JSON-EXPAND-64-NEXT:   {
+# JSON-EXPAND-64-NEXT:   "SectionIndex": 2
+# JSON-EXPAND-64-NEXT:   "Relocs": [
+# JSON-EXPAND-64-NEXT:     {
+# JSON-EXPAND-64-NEXT:       "Relocation": {
+# JSON-EXPAND-64-NEXT:         "Offset": 0,
+# JSON-EXPAND-64-NEXT:         "Type": {
+# JSON-EXPAND-64-NEXT:           "Name": "R_X86_64_NONE",
+# JSON-EXPAND-64-NEXT:           "Value": 0
+# JSON-EXPAND-64-NEXT:         },
+# JSON-EXPAND-64-NEXT:         "Symbol": {
+# JSON-EXPAND-64-NEXT:           "Name": "rel_0",
+# JSON-EXPAND-64-NEXT:           "Value": 1
+# JSON-EXPAND-64-NEXT:         }
+# JSON-EXPAND-64-NEXT:       }
+# JSON-EXPAND-64-NEXT:     },
+# JSON-EXPAND-64-NEXT:     {
+# JSON-EXPAND-64-NEXT:       "Relocation": {
+# JSON-EXPAND-64-NEXT:         "Offset": 1,
+# JSON-EXPAND-64-NEXT:         "Type": {
+# JSON-EXPAND-64-NEXT:           "Name": "R_X86_64_PC32",
+# JSON-EXPAND-64-NEXT:           "Value": 2
+# JSON-EXPAND-64-NEXT:         },
+# JSON-EXPAND-64-NEXT:         "Symbol": {
+# JSON-EXPAND-64-NEXT:           "Name": "rel_neg",
+# JSON-EXPAND-64-NEXT:           "Value": 2
+# JSON-EXPAND-64-NEXT:         }
+# JSON-EXPAND-64-NEXT:       }
+# JSON-EXPAND-64-NEXT:     },
+# JSON-EXPAND-64-NEXT:     {
+# JSON-EXPAND-64-NEXT:       "Relocation": {
+# JSON-EXPAND-64-NEXT:         "Offset": 5,
+# JSON-EXPAND-64-NEXT:         "Type": {
+# JSON-EXPAND-64-NEXT:           "Name": "R_X86_64_PLT32",
+# JSON-EXPAND-64-NEXT:           "Value": 4
+# JSON-EXPAND-64-NEXT:         },
+# JSON-EXPAND-64-NEXT:         "Symbol": {
+# JSON-EXPAND-64-NEXT:           "Name": "rel_pos",
+# JSON-EXPAND-64-NEXT:           "Value": 3
+# JSON-EXPAND-64-NEXT:         }
+# JSON-EXPAND-64-NEXT:       }
+# JSON-EXPAND-64-NEXT:     },
+# JSON-EXPAND-64-NEXT:     {
+# JSON-EXPAND-64-NEXT:       "Relocation": {
+# JSON-EXPAND-64-NEXT:         "Offset": 9,
+# JSON-EXPAND-64-NEXT:         "Type": {
+# JSON-EXPAND-64-NEXT:           "Name": "R_X86_64_64",
+# JSON-EXPAND-64-NEXT:           "Value": 1
+# JSON-EXPAND-64-NEXT:         },
+# JSON-EXPAND-64-NEXT:         "Symbol": {
+# JSON-EXPAND-64-NEXT:           "Name": "rel_64",
+# JSON-EXPAND-64-NEXT:           "Value": 4
+# JSON-EXPAND-64-NEXT:         }
+# JSON-EXPAND-64-NEXT:       }
+# JSON-EXPAND-64-NEXT:     }
+# JSON-EXPAND-64-NEXT:    ]
+# JSON-EXPAND-64-NEXT:   },
+# JSON-EXPAND-64-NEXT:   {
+# JSON-EXPAND-64-NEXT:   "SectionIndex": 3,
+# JSON-EXPAND-64-NEXT:   "Relocs": [
+# JSON-EXPAND-64-NEXT:     {
+# JSON-EXPAND-64-NEXT:       "Relocation": {
+# JSON-EXPAND-64-NEXT:         "Offset": 0,
+# JSON-EXPAND-64-NEXT:         "Type": {
+# JSON-EXPAND-64-NEXT:           "Name": "R_X86_64_NONE",
+# JSON-EXPAND-64-NEXT:           "Value": 0
+# JSON-EXPAND-64-NEXT:         },
+# JSON-EXPAND-64-NEXT:         "Symbol": {
+# JSON-EXPAND-64-NEXT:           "Name": "rela_0",
+# JSON-EXPAND-64-NEXT:           "Value": 5
+# JSON-EXPAND-64-NEXT:         },
+# JSON-EXPAND-64-NEXT:         "Addend": 0
+# JSON-EXPAND-64-NEXT:       }
+# JSON-EXPAND-64-NEXT:     },
+# JSON-EXPAND-64-NEXT:     {
+# JSON-EXPAND-64-NEXT:       "Relocation": {
+# JSON-EXPAND-64-NEXT:         "Offset": 1,
+# JSON-EXPAND-64-NEXT:         "Type": {
+# JSON-EXPAND-64-NEXT:           "Name": "R_X86_64_PC32",
+# JSON-EXPAND-64-NEXT:           "Value": 2
+# JSON-EXPAND-64-NEXT:         },
+# JSON-EXPAND-64-NEXT:         "Symbol": {
+# JSON-EXPAND-64-NEXT:           "Name": "rela_neg",
+# JSON-EXPAND-64-NEXT:           "Value": 6
+# JSON-EXPAND-64-NEXT:         },
+# JSON-EXPAND-64-NEXT:         "Addend": 18446744073709551615
+# JSON-EXPAND-64-NEXT:       }
+# JSON-EXPAND-64-NEXT:     },
+# JSON-EXPAND-64-NEXT:     {
+# JSON-EXPAND-64-NEXT:       "Relocation": {
+# JSON-EXPAND-64-NEXT:         "Offset": 5,
+# JSON-EXPAND-64-NEXT:         "Type": {
+# JSON-EXPAND-64-NEXT:           "Name": "R_X86_64_PLT32",
+# JSON-EXPAND-64-NEXT:           "Value": 4
+# JSON-EXPAND-64-NEXT:         },
+# JSON-EXPAND-64-NEXT:         "Symbol": {
+# JSON-EXPAND-64-NEXT:           "Name": "rela_pos",
+# JSON-EXPAND-64-NEXT:           "Value": 7
+# JSON-EXPAND-64-NEXT:         },
+# JSON-EXPAND-64-NEXT:         "Addend": 2
+# JSON-EXPAND-64-NEXT:       }
+# JSON-EXPAND-64-NEXT:     },
+# JSON-EXPAND-64-NEXT:     {
+# JSON-EXPAND-64-NEXT:       "Relocation": {
+# JSON-EXPAND-64-NEXT:         "Offset": 18446744073709551615,
+# JSON-EXPAND-64-NEXT:         "Type": {
+# JSON-EXPAND-64-NEXT:           "Name": "R_X86_64_64",
+# JSON-EXPAND-64-NEXT:           "Value": 1
+# JSON-EXPAND-64-NEXT:         },
+# JSON-EXPAND-64-NEXT:         "Symbol": {
+# JSON-EXPAND-64-NEXT:           "Name": "rela_minneg",
+# JSON-EXPAND-64-NEXT:           "Value": 8
+# JSON-EXPAND-64-NEXT:         },
+# JSON-EXPAND-64-NEXT:         "Addend": 9223372036854775808
+# JSON-EXPAND-64-NEXT:       }
+# JSON-EXPAND-64-NEXT:     },
+# JSON-EXPAND-64-NEXT:     {
+# JSON-EXPAND-64-NEXT:       "Relocation": {
+# JSON-EXPAND-64-NEXT:         "Offset": 9,
+# JSON-EXPAND-64-NEXT:         "Type": {
+# JSON-EXPAND-64-NEXT:           "Name": "R_X86_64_32S",
+# JSON-EXPAND-64-NEXT:           "Value": 11
+# JSON-EXPAND-64-NEXT:         },
+# JSON-EXPAND-64-NEXT:         "Symbol": {
+# JSON-EXPAND-64-NEXT:           "Name": "rela_maxpos",
+# JSON-EXPAND-64-NEXT:           "Value": 9
+# JSON-EXPAND-64-NEXT:         },
+# JSON-EXPAND-64-NEXT:         "Addend": 9223372036854775807
+# JSON-EXPAND-64-NEXT:       }
+# JSON-EXPAND-64-NEXT:     }
+# JSON-EXPAND-64-NEXT:    ]
+# JSON-EXPAND-64-NEXT:  }
+# JSON-EXPAND-64-NEXT: ]
+
 --- !ELF
 FileHeader:
   Class:   ELFCLASS64
@@ -330,8 +470,7 @@ Symbols:
 # GNU-32-NEXT:ffffffff  00000701 R_386_32               00000003   rela_minneg - 80000000
 # GNU-32-NEXT:00000009  00000803 R_386_GOT32            ffffffff   rela_maxpos + 7fffffff
 
-## Show that --expand-relocs expands the relocation dump for LLVM style only
-## (and not GNU).
+## Show that --expand-relocs expands the relocation dump for LLVM style and not GNU.
 # RUN: llvm-readobj -r --expand-relocs %t32 \
 # RUN:   | FileCheck %s --check-prefix=LLVM-EXPAND-32 --strict-whitespace --match-full-lines
 # RUN: llvm-readelf -r --expand-relocs %t32 \
diff --git a/llvm/test/tools/llvm-readobj/ELF/stack-sizes.test b/llvm/test/tools/llvm-readobj/ELF/stack-sizes.test
index 1023509713678..c2f25f0d1800c 100644
--- a/llvm/test/tools/llvm-readobj/ELF/stack-sizes.test
+++ b/llvm/test/tools/llvm-readobj/ELF/stack-sizes.test
@@ -870,8 +870,8 @@ Sections:
 # RUN: llvm-readobj --stack-sizes %t18 2>&1 | \
 # RUN:   FileCheck %s --implicit-check-not="warning:" -DFILE=%t18 --check-prefix=INVALID-TARGET
 
-# INVALID-TARGET: warning: '[[FILE]]': SHT_RELA section with index 1: failed to get a relocated section: invalid section index: 255
-# INVALID-TARGET: warning: '[[FILE]]': SHT_RELA section with index 2: failed to get a relocated section: invalid section index: 255
+# INVALID-TARGET: warning: '[[FILE]]': unable to get stack size map section(s): SHT_RELA section with index 1: failed to get a relocated section: invalid section index: 255
+# INVALID-TARGET: SHT_RELA section with index 2: failed to get a relocated section: invalid section index: 255
 
 --- !ELF
 FileHeader:
diff --git a/llvm/test/tools/llvm-readobj/ELF/symbol-visibility.test b/llvm/test/tools/llvm-readobj/ELF/symbol-visibility.test
index 4c298b1797a7b..a25e1e0a253ae 100644
--- a/llvm/test/tools/llvm-readobj/ELF/symbol-visibility.test
+++ b/llvm/test/tools/llvm-readobj/ELF/symbol-visibility.test
@@ -7,6 +7,7 @@
 # RUN: yaml2obj --docnum=1 %s -o %t1.o
 # RUN: llvm-readobj --symbols %t1.o | FileCheck %s --check-prefix=LLVM
 # RUN: llvm-readelf --symbols %t1.o | FileCheck %s --strict-whitespace --check-prefix=GNU
+# RUN: llvm-readobj --symbols --pretty-print --elf-output-style=JSON %t1.o | FileCheck %s --check-prefix=JSON
 
 # LLVM:      Name: default
 # LLVM:      Other: 0
@@ -30,6 +31,45 @@
 # GNU-NEXT: HIDDEN    UND hidden
 # GNU-NEXT: PROTECTED UND protected
 
+# JSON:      "Name": "default",
+# JSON:      "Other": {
+# JSON-NEXT:   "Value": 0,
+# JSON-NEXT:   "Flags": []
+# JSON-NEXT: },
+
+# JSON:      "Name": "internal",
+# JSON:      "Other": {
+# JSON-NEXT:    "Value": 1,
+# JSON-NEXT:    "Flags": [
+# JSON-NEXT:      {
+# JSON-NEXT:        "Name": "STV_INTERNAL",
+# JSON-NEXT:        "Value": 1
+# JSON-NEXT:      }
+# JSON-NEXT:    ]
+# JSON-NEXT:  },
+
+# JSON:      "Name": "hidden",
+# JSON:      "Other": {
+# JSON-NEXT:    "Value": 2,
+# JSON-NEXT:    "Flags": [
+# JSON-NEXT:      {
+# JSON-NEXT:        "Name": "STV_HIDDEN",
+# JSON-NEXT:        "Value": 2
+# JSON-NEXT:      }
+# JSON-NEXT:    ]
+# JSON-NEXT:  },
+
+# JSON:      "Name": "protected",
+# JSON:      "Other": {
+# JSON-NEXT:    "Value": 3,
+# JSON-NEXT:    "Flags": [
+# JSON-NEXT:      {
+# JSON-NEXT:        "Name": "STV_PROTECTED",
+# JSON-NEXT:        "Value": 3
+# JSON-NEXT:      }
+# JSON-NEXT:    ]
+# JSON-NEXT:  },
+
 --- !ELF
 FileHeader:
   Class: ELFCLASS32
diff --git a/llvm/test/tools/llvm-reduce/fail-execute-test.test b/llvm/test/tools/llvm-reduce/fail-execute-test.test
index 1582ece88e9cb..f60107f97327f 100644
--- a/llvm/test/tools/llvm-reduce/fail-execute-test.test
+++ b/llvm/test/tools/llvm-reduce/fail-execute-test.test
@@ -1,3 +1,4 @@
+# RUN: export LSAN_OPTIONS=detect_leaks=0
 # RUN: not llvm-reduce --test=%s.NotAFileInTestingDir %p/Inputs/test-output-format.ll 2>&1 | FileCheck -DFILENAME=%s.NotAFileInTestingDir --strict-whitespace %s
 
-# CHECK: Error running interesting-ness test: {{(Executable "[[FILENAME]]" doesn't exist$)?(program not executable$)?}}
\ No newline at end of file
+# CHECK: Error running interesting-ness test: {{(Executable "[[FILENAME]]" doesn't exist$)?(program not executable$)?}}
diff --git a/llvm/test/tools/llvm-symbolizer/Inputs/split-dwarf-dwp-invalid b/llvm/test/tools/llvm-symbolizer/Inputs/split-dwarf-dwp-invalid
new file mode 100755
index 0000000000000..38464cc7671e2
Binary files /dev/null and b/llvm/test/tools/llvm-symbolizer/Inputs/split-dwarf-dwp-invalid differ
diff --git a/llvm/test/tools/llvm-symbolizer/Inputs/split-dwarf-dwp-invalid.dwp b/llvm/test/tools/llvm-symbolizer/Inputs/split-dwarf-dwp-invalid.dwp
new file mode 100644
index 0000000000000..06b7196408355
Binary files /dev/null and b/llvm/test/tools/llvm-symbolizer/Inputs/split-dwarf-dwp-invalid.dwp differ
diff --git a/llvm/test/tools/llvm-symbolizer/split-dwarf-dwp-invalid.test b/llvm/test/tools/llvm-symbolizer/split-dwarf-dwp-invalid.test
new file mode 100644
index 0000000000000..46b244b1315a5
--- /dev/null
+++ b/llvm/test/tools/llvm-symbolizer/split-dwarf-dwp-invalid.test
@@ -0,0 +1,33 @@
+RUN: llvm-symbolizer --dwp=%p/Inputs/split-dwarf-dwp-invalid.dwp \
+RUN:     --obj=%p/Inputs/split-dwarf-dwp-invalid \
+RUN:     0x0000000000001140 0x0000000000001130 0x0000000000001130 \
+RUN:     2>&1 | FileCheck %s
+
+The test contains a hand-modified dwp file containing an invalid DWARF version
+in the second-queried CU.
+
+This ensures llvm-symbolizer doesn't crash when trying to access the invalid CU
+a second time (the first time works fine, the second fails the sorted search
+because of a null entry that can't be sorted/compared)
+
+One warning rather than two would be good here, but we put the CUs in an
+offset-sorted list, so a null entry in that list makes it impossible to sort
+(since it has no offset), so on failure we don't put anything in the list,
+which means next time it's queried it is parsed again and fails and warns
+again. We could make the list a pair of {offset, pointer} but that'd require
+fixing up a bunch of clients probably (possibly by providing a filter iterator
+that skips the null entries, keeping those only as an implementation detail?) -
+or perhaps just have a separate list of offsets that have failed to parse
+previously?
+
+CHECK: warning: DWARF unit at offset 0x00000000 has unsupported version 255, supported are 2-5
+CHECK: warning: DWARF unit at offset 0x00000000 has unsupported version 255, supported are 2-5
+
+CHECK: other()
+CHECK: /usr/local/google/home/blaikie/dev/scratch{{[/\\]}}other.cpp:1:16
+
+CHECK: main
+CHECK: /usr/local/google/home/blaikie/dev/scratch{{[/\\]}}test.cpp:5:0
+
+CHECK: main
+CHECK: /usr/local/google/home/blaikie/dev/scratch{{[/\\]}}test.cpp:5:0
diff --git a/llvm/tools/bugpoint/OptimizerDriver.cpp b/llvm/tools/bugpoint/OptimizerDriver.cpp
index 1197528d0dd30..f7239f5dc61bd 100644
--- a/llvm/tools/bugpoint/OptimizerDriver.cpp
+++ b/llvm/tools/bugpoint/OptimizerDriver.cpp
@@ -207,7 +207,7 @@ bool BugDriver::runPasses(Module &Program,
     Args.push_back(OptArgs[i]);
   // Pin to legacy PM since bugpoint has lots of infra and hacks revolving
   // around the legacy PM.
-  Args.push_back("-enable-new-pm=0");
+  Args.push_back("-bugpoint-enable-legacy-pm");
   Args.push_back("-disable-symbolication");
   Args.push_back("-o");
   Args.push_back(OutputFilename);
diff --git a/llvm/tools/bugpoint/bugpoint.cpp b/llvm/tools/bugpoint/bugpoint.cpp
index 42d3395c73b15..e49efdfe7c8e0 100644
--- a/llvm/tools/bugpoint/bugpoint.cpp
+++ b/llvm/tools/bugpoint/bugpoint.cpp
@@ -30,7 +30,6 @@
 #include "llvm/Support/TargetSelect.h"
 #include "llvm/Support/Valgrind.h"
 #include "llvm/Transforms/IPO/AlwaysInliner.h"
-#include "llvm/Transforms/IPO/PassManagerBuilder.h"
 
 // Enable this macro to debug bugpoint itself.
 //#define DEBUG_BUGPOINT 1
@@ -66,24 +65,6 @@ static cl::opt<bool>
 static cl::list<const PassInfo *, bool, PassNameParser>
     PassList(cl::desc("Passes available:"));
 
-static cl::opt<bool>
-    OptLevelO1("O1", cl::desc("Optimization level 1. Identical to 'opt -O1'"));
-
-static cl::opt<bool>
-    OptLevelO2("O2", cl::desc("Optimization level 2. Identical to 'opt -O2'"));
-
-static cl::opt<bool> OptLevelOs(
-    "Os",
-    cl::desc(
-        "Like -O2 with extra optimizations for size. Similar to clang -Os"));
-
-static cl::opt<bool>
-OptLevelOz("Oz",
-           cl::desc("Like -Os but reduces code size further. Similar to clang -Oz"));
-
-static cl::opt<bool>
-    OptLevelO3("O3", cl::desc("Optimization level 3. Identical to 'opt -O3'"));
-
 static cl::opt<std::string>
     OverrideTriple("mtriple", cl::desc("Override target triple for module"));
 
@@ -110,23 +91,6 @@ class AddToDriver : public legacy::FunctionPassManager {
 };
 }
 
-// This routine adds optimization passes based on selected optimization level,
-// OptLevel.
-//
-// OptLevel - Optimization Level
-static void AddOptimizationPasses(legacy::FunctionPassManager &FPM,
-                                  unsigned OptLevel,
-                                  unsigned SizeLevel) {
-  PassManagerBuilder Builder;
-  Builder.OptLevel = OptLevel;
-  Builder.SizeLevel = SizeLevel;
-
-  Builder.Inliner = createAlwaysInlinerLegacyPass();
-
-  Builder.populateFunctionPassManager(FPM);
-  Builder.populateModulePassManager(FPM);
-}
-
 #define HANDLE_EXTENSION(Ext)                                                  \
   llvm::PassPluginLibraryInfo get##Ext##PluginInfo();
 #include "llvm/Support/Extension.def"
@@ -192,17 +156,6 @@ int main(int argc, char **argv) {
 
   AddToDriver PM(D);
 
-  if (OptLevelO1)
-    AddOptimizationPasses(PM, 1, 0);
-  else if (OptLevelO2)
-    AddOptimizationPasses(PM, 2, 0);
-  else if (OptLevelO3)
-    AddOptimizationPasses(PM, 3, 0);
-  else if (OptLevelOs)
-    AddOptimizationPasses(PM, 2, 1);
-  else if (OptLevelOz)
-    AddOptimizationPasses(PM, 2, 2);
-
   for (const PassInfo *PI : PassList)
     D.addPass(std::string(PI->getPassArgument()));
 
diff --git a/llvm/tools/dsymutil/DwarfLinkerForBinary.cpp b/llvm/tools/dsymutil/DwarfLinkerForBinary.cpp
index 413cbec5c51db..c5e1bf1eb91ac 100644
--- a/llvm/tools/dsymutil/DwarfLinkerForBinary.cpp
+++ b/llvm/tools/dsymutil/DwarfLinkerForBinary.cpp
@@ -1004,7 +1004,8 @@ bool DwarfLinkerForBinary::AddressManager::isLiveSubprogram(
 
   dwarf::Form Form = Abbrev->getFormByIndex(*LowPcIdx);
 
-  if (Form == dwarf::DW_FORM_addr) {
+  switch (Form) {
+  case dwarf::DW_FORM_addr: {
     uint64_t Offset = DIE.getOffset() + getULEB128Size(Abbrev->getCode());
     uint64_t LowPcOffset, LowPcEndOffset;
     std::tie(LowPcOffset, LowPcEndOffset) =
@@ -1012,8 +1013,11 @@ bool DwarfLinkerForBinary::AddressManager::isLiveSubprogram(
     return hasValidRelocationAt(ValidDebugInfoRelocs, LowPcOffset,
                                 LowPcEndOffset, MyInfo);
   }
-
-  if (Form == dwarf::DW_FORM_addrx) {
+  case dwarf::DW_FORM_addrx:
+  case dwarf::DW_FORM_addrx1:
+  case dwarf::DW_FORM_addrx2:
+  case dwarf::DW_FORM_addrx3:
+  case dwarf::DW_FORM_addrx4: {
     std::optional<DWARFFormValue> AddrValue = DIE.find(dwarf::DW_AT_low_pc);
     if (std::optional<uint64_t> AddrOffsetSectionBase =
             DIE.getDwarfUnit()->getAddrOffsetSectionBase()) {
@@ -1022,11 +1026,14 @@ bool DwarfLinkerForBinary::AddressManager::isLiveSubprogram(
           StartOffset + DIE.getDwarfUnit()->getAddressByteSize();
       return hasValidRelocationAt(ValidDebugAddrRelocs, StartOffset, EndOffset,
                                   MyInfo);
-    } else
-      Linker.reportWarning("no base offset for address table", SrcFileName);
-  }
+    }
 
-  return false;
+    Linker.reportWarning("no base offset for address table", SrcFileName);
+    return false;
+  }
+  default:
+    return false;
+  }
 }
 
 uint64_t
diff --git a/llvm/tools/dsymutil/dsymutil.cpp b/llvm/tools/dsymutil/dsymutil.cpp
index 1c11297f940a6..1da434c0227e5 100644
--- a/llvm/tools/dsymutil/dsymutil.cpp
+++ b/llvm/tools/dsymutil/dsymutil.cpp
@@ -35,6 +35,7 @@
 #include "llvm/Support/CrashRecoveryContext.h"
 #include "llvm/Support/FileCollector.h"
 #include "llvm/Support/FileSystem.h"
+#include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/InitLLVM.h"
 #include "llvm/Support/LLVMDriver.h"
 #include "llvm/Support/Path.h"
@@ -801,12 +802,14 @@ int main(int argc, char **argv) {
         ErrorOr<vfs::Status> stat = Options.LinkOpts.VFS->status(File.path());
         if (!stat)
           break;
-        FileOffset += stat->getSize();
         if (FileOffset > UINT32_MAX) {
-          WithColor::error() << "the universal binary has a slice with an "
-              "offset exceeds 4GB and will produce an invalid Mach-O file.";
+          WithColor::error() << formatv(
+              "the universal binary has a slice with a starting offset ({0:x}) "
+              "that exceeds 4GB and will produce an invalid Mach-O file.",
+              FileOffset);
           return EXIT_FAILURE;
         }
+        FileOffset += stat->getSize();
       }
       if (!MachOUtils::generateUniversalBinary(TempFiles,
                                                OutputLocationOrErr->DWARFFile,
diff --git a/llvm/tools/lli/lli.cpp b/llvm/tools/lli/lli.cpp
index 808ec88d1c2f4..ff73ac18f5eee 100644
--- a/llvm/tools/lli/lli.cpp
+++ b/llvm/tools/lli/lli.cpp
@@ -946,9 +946,22 @@ int runOrcJIT(const char *ProgName) {
   auto J = ExitOnErr(Builder.create());
 
   auto *ObjLayer = &J->getObjLinkingLayer();
-  if (auto *RTDyldObjLayer = dyn_cast<orc::RTDyldObjectLinkingLayer>(ObjLayer))
+  if (auto *RTDyldObjLayer = dyn_cast<orc::RTDyldObjectLinkingLayer>(ObjLayer)) {
     RTDyldObjLayer->registerJITEventListener(
         *JITEventListener::createGDBRegistrationListener());
+#if LLVM_USE_OPROFILE
+    RTDyldObjLayer->registerJITEventListener(
+        *JITEventListener::createOProfileJITEventListener());
+#endif
+#if LLVM_USE_INTEL_JITEVENTS
+    RTDyldObjLayer->registerJITEventListener(
+        *JITEventListener::createIntelJITEventListener());
+#endif
+#if LLVM_USE_PERF
+    RTDyldObjLayer->registerJITEventListener(
+        *JITEventListener::createPerfJITEventListener());
+#endif
+  }
 
   if (PerModuleLazy)
     J->setPartitionFunction(orc::CompileOnDemandLayer::compileWholeModule);
diff --git a/llvm/tools/llvm-c-test/include-all.c b/llvm/tools/llvm-c-test/include-all.c
index eada03caf5e07..d5d07ff0b8915 100644
--- a/llvm/tools/llvm-c-test/include-all.c
+++ b/llvm/tools/llvm-c-test/include-all.c
@@ -36,6 +36,5 @@
 #include "llvm-c/Target.h"
 #include "llvm-c/TargetMachine.h"
 #include "llvm-c/Transforms/PassBuilder.h"
-#include "llvm-c/Transforms/PassManagerBuilder.h"
 #include "llvm-c/Types.h"
 #include "llvm-c/lto.h"
diff --git a/llvm/tools/llvm-debuginfo-analyzer/README.txt b/llvm/tools/llvm-debuginfo-analyzer/README.txt
new file mode 100644
index 0000000000000..e6c20db7cd712
--- /dev/null
+++ b/llvm/tools/llvm-debuginfo-analyzer/README.txt
@@ -0,0 +1,224 @@
+//===- llvm/tools/llvm-debuginfo-analyzer/README.txt ----------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file contains notes collected during the development, review and test.
+// It describes limitations, know issues and future work.
+//
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// Remove the use of macros in 'LVReader.h' that describe the bumpallocators.
+//===----------------------------------------------------------------------===//
+https://reviews.llvm.org/D137933#inline-1389904
+
+Use a standard (or LLVM) map with typeinfo (would need a specialization
+to expose equality and hasher) for the allocators and the creation
+functions could be a function template.
+
+//===----------------------------------------------------------------------===//
+// Use a lit test instead of a unit test for the logical readers.
+//===----------------------------------------------------------------------===//
+https://reviews.llvm.org/D125783#inline-1324376
+
+As the DebugInfoLogicalView library is sufficiently exposed via the
+llvm-debuginfo-analyzer tool, follow the LLVM general approach and
+use LIT tests to validate the logical readers.
+
+Convert the unitests:
+  llvm-project/llvm/unittests/DebugInfo/LogicalView/CodeViewReaderTest.cpp
+  llvm-project/llvm/unittests/DebugInfo/LogicalView/ELFReaderTest.cpp
+
+into LIT tests:
+  llvm-project/llvm/test/DebugInfo/LogicalView/CodeViewReader.test
+  llvm-project/llvm/test/DebugInfo/LogicalView/ELFReader.test
+
+//===----------------------------------------------------------------------===//
+// Eliminate calls to 'getInputFileDirectory()' in the unit tests.
+//===----------------------------------------------------------------------===//
+https://reviews.llvm.org/D125783#inline-1324359
+
+Rewrite the unittests 'LFReaderTest' and 'CodeViewReaderTest'to eliminate
+the call:
+
+  getInputFileDirectory()
+
+as use of that call is discouraged.
+
+See: Use a lit test instead of a unit test for the logical readers.
+
+//===----------------------------------------------------------------------===//
+// Fix mismatch between %d/%x format strings and uint64_t type.
+//===----------------------------------------------------------------------===//
+https://reviews.llvm.org/D137400
+https://github.com/llvm/llvm-project/issues/58758
+
+Incorrect printing of uint64_t on 32-bit platforms.
+Add the PRIx64 specifier to the printing code (format()).
+
+//===----------------------------------------------------------------------===//
+// Remove 'LVScope::Children' container.
+//===----------------------------------------------------------------------===//
+https://reviews.llvm.org/D137933#inline-1373902
+
+Use a chaining iterator over the other containers rather than keep a
+separate container 'Children' that mirrors their contents.
+
+//===----------------------------------------------------------------------===//
+// Use TableGen for command line options.
+//===----------------------------------------------------------------------===//
+https://reviews.llvm.org/D125777#inline-1291801
+
+The current trend is to use TableGen for command-line options in tools.
+Change command line options to use tablegen as many other LLVM tools.
+
+//===----------------------------------------------------------------------===//
+// LVDoubleMap to return optional<ValueType> instead of null pointer.
+//===----------------------------------------------------------------------===//
+https://reviews.llvm.org/D125783#inline-1294164
+
+The more idiomatic LLVM way to handle this would be to have 'find '
+return Optional<ValueType>.
+
+//===----------------------------------------------------------------------===//
+// Pass references instead of pointers (Comparison functions).
+//===----------------------------------------------------------------------===//
+https://reviews.llvm.org/D125782#inline-1293920
+
+In the comparison functions, pass references instead of pointers (when
+pointers cannot be null).
+
+//===----------------------------------------------------------------------===//
+// Use StringMap where possible.
+//===----------------------------------------------------------------------===//
+https://reviews.llvm.org/D125783#inline-1294211
+
+LLVM has a StringMap class that is advertised as more efficient than
+std::map<std::string, ValueType>. Mainly it does fewer allocations
+because the key is not a std::string.
+
+Replace the use of std::map<std::string, ValueType> with String Map.
+One specific case is the LVSymbolNames definitions.
+
+//===----------------------------------------------------------------------===//
+// Calculate unique offset for CodeView elements.
+//===----------------------------------------------------------------------===//
+In order to have the same logical functionality as the ELF Reader, such
+as:
+
+- find scopes contribution to debug info
+- sort by its physical location
+
+The logical elements must have an unique offset (similar like the DWARF
+DIE offset).
+
+//===----------------------------------------------------------------------===//
+// Move 'initializeFileAndStringTables' to the COFF Library.
+//===----------------------------------------------------------------------===//
+There is some code in the CodeView reader that was extracted/adapted
+from 'tools/llvm-readobj/COFFDumper.cpp' that can be moved to the COFF
+library.
+
+We had a similar case with code shared with llvm-pdbutil that was moved
+to the PDB library: https://reviews.llvm.org/D122226
+
+//===----------------------------------------------------------------------===//
+// Move 'getSymbolKindName'/'formatRegisterId' to the CodeView Library.
+//===----------------------------------------------------------------------===//
+There is some code in the CodeView reader that was extracted/adapted
+from 'lib/DebugInfo/CodeView/SymbolDumper.cpp' that can be used.
+
+//===----------------------------------------------------------------------===//
+// Use of std::unordered_set instead of std::set.
+//===----------------------------------------------------------------------===//
+https://reviews.llvm.org/D125784#inline-1221421
+
+Replace the std::set usage for DeducedScopes, UnresolvedScopes and
+IdentifiedNamespaces with std::unordered_set and get the benefit
+of the O(1) while inserting/searching, as the order is not important.
+
+//===----------------------------------------------------------------------===//
+// Optimize 'LVNamespaceDeduction::find' funtion.
+//===----------------------------------------------------------------------===//
+https://reviews.llvm.org/D125784#inline-1296195
+
+Optimize the 'find' method to use the proposed code:
+
+  LVStringRefs::iterator Iter = std::find_if(Components.begin(), Components.end(),
+    [](StringRef Name) {
+        return IdentifiedNamespaces.find(Name) == IdentifiedNamespaces.end();
+    });
+  LVStringRefs::size_type FirstNonNamespace = std::distance(Components.begin(), Iter);
+
+//===----------------------------------------------------------------------===//
+// Move all the printing support to a common module.
+//===----------------------------------------------------------------------===//
+Factor out printing functionality from the logical elements into a
+common module.
+
+//===----------------------------------------------------------------------===//
+// Refactor 'LVBinaryReader::processLines'.
+//===----------------------------------------------------------------------===//
+https://reviews.llvm.org/D125783#inline-1246155
+https://reviews.llvm.org/D137156
+
+During the traversal of the debug information sections, we created the
+logical lines representing the disassembled instructions from the text
+section and the logical lines representing the line records from the
+debug line section. Using the ranges associated with the logical scopes,
+we will allocate those logical lines to their logical scopes.
+
+Consider the case when any of those lines become orphans, causing
+incorrect scope parent for disassembly or line records.
+
+//===----------------------------------------------------------------------===//
+// Add support for '-ffunction-sections'.
+//===----------------------------------------------------------------------===//
+https://reviews.llvm.org/D125783#inline-1295012
+
+Only linked executables are handled. It does not support relocatable
+files compiled with -ffunction-sections.
+
+//===----------------------------------------------------------------------===//
+// Add support for DWARF v5 .debug_names section.
+// Add support for CodeView public symbols stream.
+//===----------------------------------------------------------------------===//
+https://reviews.llvm.org/D125783#inline-1294142
+
+The ELF and CodeView readers use the public names information to create
+the instructions (LVLineAssembler). Instead of relying on DWARF section
+names (.debug_pubnames, .debug_names) and CodeView public symbol stream
+(S_PUB32), the readers collects the needed information while processing
+the debug information.
+
+If the object file supports the above section names and stream, use them
+to create the public names.
+
+//===----------------------------------------------------------------------===//
+// Add support for some extra DWARF locations.
+//===----------------------------------------------------------------------===//
+The following DWARF debug location operands are not supported:
+
+- DW_OP_const_type
+- DW_OP_entry_value
+- DW_OP_implicit_value
+
+//===----------------------------------------------------------------------===//
+// Add support for additional binary formats.
+//===----------------------------------------------------------------------===//
+- WebAssembly (Wasm).
+  https://github.com/llvm/llvm-project/issues/57040#issuecomment-1211336680
+
+- Extended COFF (XCOFF)
+
+//===----------------------------------------------------------------------===//
+// Add support for JSON or YAML.
+//===----------------------------------------------------------------------===//
+The logical view uses its own and non-standard free form text when
+displaying information on logical elements.
+
+//===----------------------------------------------------------------------===//
diff --git a/llvm/tools/llvm-exegesis/lib/Analysis.cpp b/llvm/tools/llvm-exegesis/lib/Analysis.cpp
index d1ec125f82ae7..2d7e17fc7399f 100644
--- a/llvm/tools/llvm-exegesis/lib/Analysis.cpp
+++ b/llvm/tools/llvm-exegesis/lib/Analysis.cpp
@@ -28,7 +28,7 @@ enum EscapeTag { kEscapeCsv, kEscapeHtml, kEscapeHtmlString };
 template <EscapeTag Tag> void writeEscaped(raw_ostream &OS, const StringRef S);
 
 template <> void writeEscaped<kEscapeCsv>(raw_ostream &OS, const StringRef S) {
-  if (!llvm::is_contained(S, kCsvSep)) {
+  if (!S.contains(kCsvSep)) {
     OS << S;
   } else {
     // Needs escaping.
diff --git a/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp b/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp
index dbd1dc2e4a716..faefc7db1bd1b 100644
--- a/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp
+++ b/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp
@@ -30,6 +30,7 @@
 #include <inttypes.h>
 #include <iostream>
 #include <map>
+#include <optional>
 #include <string>
 #include <system_error>
 #include <vector>
@@ -107,6 +108,13 @@ static opt<unsigned>
                     "number of cores on the current machine."),
                cl::value_desc("n"), cat(ConversionOptions));
 
+static opt<uint64_t>
+    SegmentSize("segment-size",
+               desc("Specify the size in bytes of the size the final GSYM file "
+                    "should be segmented into. This allows GSYM files to be "
+                    "split across multiple files."),
+               cl::value_desc("s"), cat(ConversionOptions));
+
 static opt<bool>
     Quiet("quiet", desc("Do not output warnings about the debug information"),
           cat(ConversionOptions));
@@ -310,7 +318,11 @@ static llvm::Error handleObjectFile(ObjectFile &Obj,
   // Save the GSYM file to disk.
   support::endianness Endian =
       Obj.makeTriple().isLittleEndian() ? support::little : support::big;
-  if (auto Err = Gsym.save(OutFile, Endian))
+
+  std::optional<uint64_t> OptSegmentSize;
+  if (SegmentSize > 0)
+    OptSegmentSize = SegmentSize;
+  if (auto Err = Gsym.save(OutFile, Endian, OptSegmentSize))
     return Err;
 
   // Verify the DWARF if requested. This will ensure all the info in the DWARF
diff --git a/llvm/tools/llvm-jitlink/llvm-jitlink.cpp b/llvm/tools/llvm-jitlink/llvm-jitlink.cpp
index be0fe1750a718..5e91ad068cdbc 100644
--- a/llvm/tools/llvm-jitlink/llvm-jitlink.cpp
+++ b/llvm/tools/llvm-jitlink/llvm-jitlink.cpp
@@ -511,8 +511,7 @@ class InProcessDeltaMapper final : public InProcessMemoryMapper {
     auto FixedAI = std::move(AI);
     FixedAI.MappingBase -= DeltaAddr;
     for (auto &Seg : FixedAI.Segments)
-      Seg.AG = AllocGroup(MemProt::Read | MemProt::Write,
-                          Seg.AG.getMemDeallocPolicy());
+      Seg.AG = {MemProt::Read | MemProt::Write, Seg.AG.getMemLifetimePolicy()};
     FixedAI.Actions.clear();
     InProcessMemoryMapper::initialize(
         FixedAI, [this, OnInitialized = std::move(OnInitialized)](
diff --git a/llvm/tools/llvm-mca/CodeRegion.h b/llvm/tools/llvm-mca/CodeRegion.h
index b5b2f3a0d1189..f9b999f645d3b 100644
--- a/llvm/tools/llvm-mca/CodeRegion.h
+++ b/llvm/tools/llvm-mca/CodeRegion.h
@@ -167,7 +167,7 @@ class CodeRegions {
   bool isValid() const { return !FoundErrors; }
 
   bool isRegionActive(llvm::StringRef Description) const {
-    return ActiveRegions.find(Description) != ActiveRegions.end();
+    return ActiveRegions.contains(Description);
   }
 };
 
diff --git a/llvm/tools/llvm-mca/Views/BottleneckAnalysis.cpp b/llvm/tools/llvm-mca/Views/BottleneckAnalysis.cpp
index c2d6a70317b63..b254ccd6670ff 100644
--- a/llvm/tools/llvm-mca/Views/BottleneckAnalysis.cpp
+++ b/llvm/tools/llvm-mca/Views/BottleneckAnalysis.cpp
@@ -52,7 +52,7 @@ void PressureTracker::getResourceUsers(uint64_t ResourceMask,
   const MCProcResourceDesc &PRDesc = *SM.getProcResource(ProcResID);
   for (unsigned I = 0, E = PRDesc.NumUnits; I < E; ++I) {
     const User U = getResourceUser(ProcResID, I);
-    if (U.second && IPI.find(U.first) != IPI.end())
+    if (U.second && IPI.contains(U.first))
       Users.emplace_back(U);
   }
 }
diff --git a/llvm/tools/llvm-mca/Views/BottleneckAnalysis.h b/llvm/tools/llvm-mca/Views/BottleneckAnalysis.h
index cd5af0afcf5b6..e709b25c3f768 100644
--- a/llvm/tools/llvm-mca/Views/BottleneckAnalysis.h
+++ b/llvm/tools/llvm-mca/Views/BottleneckAnalysis.h
@@ -146,19 +146,19 @@ class PressureTracker {
                         SmallVectorImpl<User> &Users) const;
 
   unsigned getRegisterPressureCycles(unsigned IID) const {
-    assert(IPI.find(IID) != IPI.end() && "Instruction is not tracked!");
+    assert(IPI.contains(IID) && "Instruction is not tracked!");
     const InstructionPressureInfo &Info = IPI.find(IID)->second;
     return Info.RegisterPressureCycles;
   }
 
   unsigned getMemoryPressureCycles(unsigned IID) const {
-    assert(IPI.find(IID) != IPI.end() && "Instruction is not tracked!");
+    assert(IPI.contains(IID) && "Instruction is not tracked!");
     const InstructionPressureInfo &Info = IPI.find(IID)->second;
     return Info.MemoryPressureCycles;
   }
 
   unsigned getResourcePressureCycles(unsigned IID) const {
-    assert(IPI.find(IID) != IPI.end() && "Instruction is not tracked!");
+    assert(IPI.contains(IID) && "Instruction is not tracked!");
     const InstructionPressureInfo &Info = IPI.find(IID)->second;
     return Info.ResourcePressureCycles;
   }
diff --git a/llvm/tools/llvm-mca/Views/InstructionInfoView.cpp b/llvm/tools/llvm-mca/Views/InstructionInfoView.cpp
index 257fdca8cb366..3262de48d41e3 100644
--- a/llvm/tools/llvm-mca/Views/InstructionInfoView.cpp
+++ b/llvm/tools/llvm-mca/Views/InstructionInfoView.cpp
@@ -55,10 +55,7 @@ void InstructionInfoView::printView(raw_ostream &OS) const {
     }
   }
 
-  int Index = 0;
-  for (const auto &I : enumerate(zip(IIVD, Source))) {
-    const InstructionInfoViewData &IIVDEntry = std::get<0>(I.value());
-
+  for (const auto &[Index, IIVDEntry, Inst] : enumerate(IIVD, Source)) {
     TempStream << ' ' << IIVDEntry.NumMicroOpcodes << "    ";
     if (IIVDEntry.NumMicroOpcodes < 10)
       TempStream << "  ";
@@ -92,7 +89,7 @@ void InstructionInfoView::printView(raw_ostream &OS) const {
     }
 
     if (PrintEncodings) {
-      StringRef Encoding(CE.getEncoding(I.index()));
+      StringRef Encoding(CE.getEncoding(Index));
       unsigned EncodingSize = Encoding.size();
       TempStream << " " << EncodingSize
                  << (EncodingSize < 10 ? "     " : "    ");
@@ -104,9 +101,7 @@ void InstructionInfoView::printView(raw_ostream &OS) const {
       FOS.flush();
     }
 
-    const MCInst &Inst = std::get<1>(I.value());
     TempStream << printInstructionString(Inst) << '\n';
-    ++Index;
   }
 
   TempStream.flush();
diff --git a/llvm/tools/llvm-mca/Views/ResourcePressureView.cpp b/llvm/tools/llvm-mca/Views/ResourcePressureView.cpp
index 1aa00775b5307..0f059bcc0a069 100644
--- a/llvm/tools/llvm-mca/Views/ResourcePressureView.cpp
+++ b/llvm/tools/llvm-mca/Views/ResourcePressureView.cpp
@@ -57,7 +57,7 @@ void ResourcePressureView::onEvent(const HWInstructionEvent &Event) {
   for (const std::pair<ResourceRef, ResourceCycles> &Use :
        IssueEvent.UsedResources) {
     const ResourceRef &RR = Use.first;
-    assert(Resource2VecIndex.find(RR.first) != Resource2VecIndex.end());
+    assert(Resource2VecIndex.contains(RR.first));
     unsigned R2VIndex = Resource2VecIndex[RR.first];
     R2VIndex += llvm::countr_zero(RR.second);
     ResourceUsage[R2VIndex + NumResourceUnits * SourceIdx] += Use.second;
diff --git a/llvm/tools/llvm-objdump/llvm-objdump.cpp b/llvm/tools/llvm-objdump/llvm-objdump.cpp
index 44f685f615c39..e3d14192c0625 100644
--- a/llvm/tools/llvm-objdump/llvm-objdump.cpp
+++ b/llvm/tools/llvm-objdump/llvm-objdump.cpp
@@ -1439,8 +1439,10 @@ static void disassembleObject(const Target *TheTarget, ObjectFile &Obj,
     AddrToBBAddrMap.clear();
     if (const auto *Elf = dyn_cast<ELFObjectFileBase>(&Obj)) {
       auto BBAddrMapsOrErr = Elf->readBBAddrMap(SectionIndex);
-      if (!BBAddrMapsOrErr)
+      if (!BBAddrMapsOrErr) {
         reportWarning(toString(BBAddrMapsOrErr.takeError()), Obj.getFileName());
+        return;
+      }
       for (auto &FunctionBBAddrMap : *BBAddrMapsOrErr)
         AddrToBBAddrMap.emplace(FunctionBBAddrMap.Addr,
                                 std::move(FunctionBBAddrMap));
diff --git a/llvm/tools/llvm-pdbutil/MinimalSymbolDumper.cpp b/llvm/tools/llvm-pdbutil/MinimalSymbolDumper.cpp
index 8e17284871a9a..46481d3786299 100644
--- a/llvm/tools/llvm-pdbutil/MinimalSymbolDumper.cpp
+++ b/llvm/tools/llvm-pdbutil/MinimalSymbolDumper.cpp
@@ -210,6 +210,8 @@ static std::string formatSourceLanguage(SourceLanguage Lang) {
     RETURN_CASE(SourceLanguage, D, "d");
     RETURN_CASE(SourceLanguage, Swift, "swift");
     RETURN_CASE(SourceLanguage, Rust, "rust");
+    RETURN_CASE(SourceLanguage, ObjC, "objc");
+    RETURN_CASE(SourceLanguage, ObjCpp, "objc++");
   }
   return formatUnknownEnum(Lang);
 }
diff --git a/llvm/tools/llvm-profdata/llvm-profdata.cpp b/llvm/tools/llvm-profdata/llvm-profdata.cpp
index e27d0becad2fe..28c4db6b5c4a2 100644
--- a/llvm/tools/llvm-profdata/llvm-profdata.cpp
+++ b/llvm/tools/llvm-profdata/llvm-profdata.cpp
@@ -635,7 +635,7 @@ adjustInstrProfile(std::unique_ptr<WriterContext> &WC,
       }
     }
 
-    if (StaticFuncMap.find(NewName) == StaticFuncMap.end()) {
+    if (!StaticFuncMap.contains(NewName)) {
       StaticFuncMap[NewName] = Name;
     } else {
       StaticFuncMap[NewName] = DuplicateNameStr;
diff --git a/llvm/tools/llvm-readobj/ELFDumper.cpp b/llvm/tools/llvm-readobj/ELFDumper.cpp
index 5615d7579a074..2434aa7b59cfa 100644
--- a/llvm/tools/llvm-readobj/ELFDumper.cpp
+++ b/llvm/tools/llvm-readobj/ELFDumper.cpp
@@ -235,6 +235,9 @@ template <typename ELFT> class ELFDumper : public ObjDumper {
     return 4;
   }
 
+  std::vector<EnumEntry<unsigned>>
+  getOtherFlagsFromSymbol(const Elf_Ehdr &Header, const Elf_Sym &Symbol) const;
+
   Elf_Dyn_Range dynamic_table() const {
     // A valid .dynamic section contains an array of entries terminated
     // with a DT_NULL entry. However, sometimes the section content may
@@ -328,12 +331,6 @@ template <typename ELFT> class ELFDumper : public ObjDumper {
   void printRelocatableStackSizes(std::function<void()> PrintHeader);
   void printNonRelocatableStackSizes(std::function<void()> PrintHeader);
 
-  /// Retrieves sections with corresponding relocation sections based on
-  /// IsMatch.
-  void getSectionAndRelocations(
-      std::function<bool(const Elf_Shdr &)> IsMatch,
-      llvm::MapVector<const Elf_Shdr *, const Elf_Shdr *> &SecToRelocMap);
-
   const object::ELFObjectFile<ELFT> &ObjF;
   const ELFFile<ELFT> &Obj;
   StringRef FileName;
@@ -692,14 +689,14 @@ template <typename ELFT> class LLVMELFDumper : public ELFDumper<ELFT> {
   void printMemtag(
       const ArrayRef<std::pair<std::string, std::string>> DynamicEntries,
       const ArrayRef<uint8_t> AndroidNoteDesc) override;
+  void printSymbolSection(const Elf_Sym &Symbol, unsigned SymIndex,
+                          DataRegion<Elf_Word> ShndxTable) const;
 
 private:
   void printRelrReloc(const Elf_Relr &R) override;
   void printRelRelaReloc(const Relocation<ELFT> &R,
                          const RelSymbol<ELFT> &RelSym) override;
 
-  void printSymbolSection(const Elf_Sym &Symbol, unsigned SymIndex,
-                          DataRegion<Elf_Word> ShndxTable) const;
   void printSymbol(const Elf_Sym &Symbol, unsigned SymIndex,
                    DataRegion<Elf_Word> ShndxTable,
                    std::optional<StringRef> StrTable, bool IsDynamic,
@@ -712,8 +709,22 @@ template <typename ELFT> class LLVMELFDumper : public ELFDumper<ELFT> {
   void printMipsGOT(const MipsGOTParser<ELFT> &Parser) override;
   void printMipsPLT(const MipsGOTParser<ELFT> &Parser) override;
   void printMipsABIFlags() override;
+  virtual void printZeroSymbolOtherField(const Elf_Sym &Symbol) const;
 
 protected:
+  virtual std::string getGroupSectionHeaderName() const;
+  void printSymbolOtherField(const Elf_Sym &Symbol) const;
+  virtual void printExpandedRelRelaReloc(const Relocation<ELFT> &R,
+                                         StringRef SymbolName,
+                                         StringRef RelocName);
+  virtual void printDefaultRelRelaReloc(const Relocation<ELFT> &R,
+                                        StringRef SymbolName,
+                                        StringRef RelocName);
+  virtual void printRelocationSectionInfo(const Elf_Shdr &Sec, StringRef Name,
+                                          const unsigned SecNdx);
+  virtual void printSectionGroupMembers(StringRef Name, uint64_t Idx) const;
+  virtual void printEmptyGroupMessage() const;
+
   ScopedPrinter &W;
 };
 
@@ -726,9 +737,23 @@ template <typename ELFT> class JSONELFDumper : public LLVMELFDumper<ELFT> {
   JSONELFDumper(const object::ELFObjectFile<ELFT> &ObjF, ScopedPrinter &Writer)
       : LLVMELFDumper<ELFT>(ObjF, Writer) {}
 
+  std::string getGroupSectionHeaderName() const override;
+
   void printFileSummary(StringRef FileStr, ObjectFile &Obj,
                         ArrayRef<std::string> InputFilenames,
                         const Archive *A) override;
+  virtual void printZeroSymbolOtherField(const Elf_Sym &Symbol) const override;
+
+  void printDefaultRelRelaReloc(const Relocation<ELFT> &R,
+                                StringRef SymbolName,
+                                StringRef RelocName) override;
+
+  void printRelocationSectionInfo(const Elf_Shdr &Sec, StringRef Name,
+                                  const unsigned SecNdx) override;
+
+  void printSectionGroupMembers(StringRef Name, uint64_t Idx) const override;
+
+  void printEmptyGroupMessage() const override;
 
 private:
   std::unique_ptr<DictScope> FileScope;
@@ -3276,6 +3301,35 @@ void ELFDumper<ELFT>::printReloc(const Relocation<ELFT> &R, unsigned RelIndex,
     printRelRelaReloc(R, *Target);
 }
 
+template <class ELFT>
+std::vector<EnumEntry<unsigned>>
+ELFDumper<ELFT>::getOtherFlagsFromSymbol(const Elf_Ehdr &Header,
+                                         const Elf_Sym &Symbol) const {
+  std::vector<EnumEntry<unsigned>> SymOtherFlags(std::begin(ElfSymOtherFlags),
+                                                 std::end(ElfSymOtherFlags));
+  if (Header.e_machine == EM_MIPS) {
+    // Someone in their infinite wisdom decided to make STO_MIPS_MIPS16
+    // flag overlap with other ST_MIPS_xxx flags. So consider both
+    // cases separately.
+    if ((Symbol.st_other & STO_MIPS_MIPS16) == STO_MIPS_MIPS16)
+      SymOtherFlags.insert(SymOtherFlags.end(),
+                           std::begin(ElfMips16SymOtherFlags),
+                           std::end(ElfMips16SymOtherFlags));
+    else
+      SymOtherFlags.insert(SymOtherFlags.end(),
+                           std::begin(ElfMipsSymOtherFlags),
+                           std::end(ElfMipsSymOtherFlags));
+  } else if (Header.e_machine == EM_AARCH64) {
+    SymOtherFlags.insert(SymOtherFlags.end(),
+                         std::begin(ElfAArch64SymOtherFlags),
+                         std::end(ElfAArch64SymOtherFlags));
+  } else if (Header.e_machine == EM_RISCV) {
+    SymOtherFlags.insert(SymOtherFlags.end(), std::begin(ElfRISCVSymOtherFlags),
+                         std::end(ElfRISCVSymOtherFlags));
+  }
+  return SymOtherFlags;
+}
+
 static inline void printFields(formatted_raw_ostream &OS, StringRef Str1,
                                StringRef Str2) {
   OS.PadToColumn(2u);
@@ -6281,38 +6335,11 @@ void ELFDumper<ELFT>::printNonRelocatableStackSizes(
   }
 }
 
-template <class ELFT>
-void ELFDumper<ELFT>::getSectionAndRelocations(
-    std::function<bool(const Elf_Shdr &)> IsMatch,
-    llvm::MapVector<const Elf_Shdr *, const Elf_Shdr *> &SecToRelocMap) {
-  for (const Elf_Shdr &Sec : cantFail(Obj.sections())) {
-    if (IsMatch(Sec))
-      if (SecToRelocMap.insert(std::make_pair(&Sec, (const Elf_Shdr *)nullptr))
-              .second)
-        continue;
-
-    if (Sec.sh_type != ELF::SHT_RELA && Sec.sh_type != ELF::SHT_REL)
-      continue;
-
-    Expected<const Elf_Shdr *> RelSecOrErr = Obj.getSection(Sec.sh_info);
-    if (!RelSecOrErr) {
-      reportUniqueWarning(describe(Sec) +
-                          ": failed to get a relocated section: " +
-                          toString(RelSecOrErr.takeError()));
-      continue;
-    }
-    const Elf_Shdr *ContentsSec = *RelSecOrErr;
-    if (IsMatch(*ContentsSec))
-      SecToRelocMap[ContentsSec] = &Sec;
-  }
-}
-
 template <class ELFT>
 void ELFDumper<ELFT>::printRelocatableStackSizes(
     std::function<void()> PrintHeader) {
   // Build a map between stack size sections and their corresponding relocation
   // sections.
-  llvm::MapVector<const Elf_Shdr *, const Elf_Shdr *> StackSizeRelocMap;
   auto IsMatch = [&](const Elf_Shdr &Sec) -> bool {
     StringRef SectionName;
     if (Expected<StringRef> NameOrErr = Obj.getSectionName(Sec))
@@ -6322,9 +6349,16 @@ void ELFDumper<ELFT>::printRelocatableStackSizes(
 
     return SectionName == ".stack_sizes";
   };
-  getSectionAndRelocations(IsMatch, StackSizeRelocMap);
 
-  for (const auto &StackSizeMapEntry : StackSizeRelocMap) {
+  Expected<MapVector<const Elf_Shdr *, const Elf_Shdr *>>
+      StackSizeRelocMapOrErr = Obj.getSectionAndRelocations(IsMatch);
+  if (!StackSizeRelocMapOrErr) {
+    reportUniqueWarning("unable to get stack size map section(s): " +
+                        toString(StackSizeRelocMapOrErr.takeError()));
+    return;
+  }
+
+  for (const auto &StackSizeMapEntry : *StackSizeRelocMapOrErr) {
     PrintHeader();
     const Elf_Shdr *StackSizesELFSec = StackSizeMapEntry.first;
     const Elf_Shdr *RelocSec = StackSizeMapEntry.second;
@@ -6692,9 +6726,9 @@ template <class ELFT> void LLVMELFDumper<ELFT>::printGroupSections() {
     W.printNumber("Link", G.Link);
     W.printNumber("Info", G.Info);
     W.printHex("Type", getGroupType(G.Type), G.Type);
-    W.startLine() << "Signature: " << G.Signature << "\n";
+    W.printString("Signature", G.Signature);
 
-    ListScope L(W, "Section(s) in group");
+    ListScope L(W, getGroupSectionHeaderName());
     for (const GroupMember &GM : G.Members) {
       const GroupSection *MainGroup = Map[GM.Index];
       if (MainGroup != &G)
@@ -6704,12 +6738,23 @@ template <class ELFT> void LLVMELFDumper<ELFT>::printGroupSections() {
             Twine(MainGroup->Index) +
             ", was also found in the group section with index " +
             Twine(G.Index));
-      W.startLine() << GM.Name << " (" << GM.Index << ")\n";
+      printSectionGroupMembers(GM.Name, GM.Index);
     }
   }
 
   if (V.empty())
-    W.startLine() << "There are no group sections in the file.\n";
+    printEmptyGroupMessage();
+}
+
+template <class ELFT>
+std::string LLVMELFDumper<ELFT>::getGroupSectionHeaderName() const {
+  return "Section(s) in group";
+}
+
+template <class ELFT>
+void LLVMELFDumper<ELFT>::printSectionGroupMembers(StringRef Name,
+                                                   uint64_t Idx) const {
+  W.startLine() << Name << " (" << Idx << ")\n";
 }
 
 template <class ELFT> void LLVMELFDumper<ELFT>::printRelocations() {
@@ -6721,11 +6766,7 @@ template <class ELFT> void LLVMELFDumper<ELFT>::printRelocations() {
 
     StringRef Name = this->getPrintableSectionName(Sec);
     unsigned SecNdx = &Sec - &cantFail(this->Obj.sections()).front();
-    W.startLine() << "Section (" << SecNdx << ") " << Name << " {\n";
-    W.indent();
-    this->printRelocationsHelper(Sec);
-    W.unindent();
-    W.startLine() << "}\n";
+    printRelocationSectionInfo(Sec, Name, SecNdx);
   }
 }
 
@@ -6734,6 +6775,42 @@ void LLVMELFDumper<ELFT>::printRelrReloc(const Elf_Relr &R) {
   W.startLine() << W.hex(R) << "\n";
 }
 
+template <class ELFT>
+void LLVMELFDumper<ELFT>::printExpandedRelRelaReloc(const Relocation<ELFT> &R,
+                                                    StringRef SymbolName,
+                                                    StringRef RelocName) {
+  DictScope Group(W, "Relocation");
+  W.printHex("Offset", R.Offset);
+  W.printNumber("Type", RelocName, R.Type);
+  W.printNumber("Symbol", !SymbolName.empty() ? SymbolName : "-", R.Symbol);
+  if (R.Addend)
+    W.printHex("Addend", (uintX_t)*R.Addend);
+}
+
+template <class ELFT>
+void LLVMELFDumper<ELFT>::printDefaultRelRelaReloc(const Relocation<ELFT> &R,
+                                                   StringRef SymbolName,
+                                                   StringRef RelocName) {
+  raw_ostream &OS = W.startLine();
+  OS << W.hex(R.Offset) << " " << RelocName << " "
+     << (!SymbolName.empty() ? SymbolName : "-");
+  if (R.Addend)
+    OS << " " << W.hex((uintX_t)*R.Addend);
+  OS << "\n";
+}
+
+template <class ELFT>
+void LLVMELFDumper<ELFT>::printRelocationSectionInfo(const Elf_Shdr &Sec,
+                                                     StringRef Name,
+                                                     const unsigned SecNdx) {
+  DictScope D(W, (Twine("Section (") + Twine(SecNdx) + ") " + Name).str());
+  this->printRelocationsHelper(Sec);
+}
+
+template <class ELFT> void LLVMELFDumper<ELFT>::printEmptyGroupMessage() const {
+  W.startLine() << "There are no group sections in the file.\n";
+}
+
 template <class ELFT>
 void LLVMELFDumper<ELFT>::printRelRelaReloc(const Relocation<ELFT> &R,
                                             const RelSymbol<ELFT> &RelSym) {
@@ -6742,19 +6819,9 @@ void LLVMELFDumper<ELFT>::printRelRelaReloc(const Relocation<ELFT> &R,
   this->Obj.getRelocationTypeName(R.Type, RelocName);
 
   if (opts::ExpandRelocs) {
-    DictScope Group(W, "Relocation");
-    W.printHex("Offset", R.Offset);
-    W.printNumber("Type", RelocName, R.Type);
-    W.printNumber("Symbol", !SymbolName.empty() ? SymbolName : "-", R.Symbol);
-    if (R.Addend)
-      W.printHex("Addend", (uintX_t)*R.Addend);
+    printExpandedRelRelaReloc(R, SymbolName, RelocName);
   } else {
-    raw_ostream &OS = W.startLine();
-    OS << W.hex(R.Offset) << " " << RelocName << " "
-       << (!SymbolName.empty() ? SymbolName : "-");
-    if (R.Addend)
-      OS << " " << W.hex((uintX_t)*R.Addend);
-    OS << "\n";
+    printDefaultRelRelaReloc(R, SymbolName, RelocName);
   }
 }
 
@@ -6869,6 +6936,22 @@ void LLVMELFDumper<ELFT>::printSymbolSection(
   }
 }
 
+template <class ELFT>
+void LLVMELFDumper<ELFT>::printSymbolOtherField(const Elf_Sym &Symbol) const {
+  std::vector<EnumEntry<unsigned>> SymOtherFlags =
+      this->getOtherFlagsFromSymbol(this->Obj.getHeader(), Symbol);
+  W.printFlags("Other", Symbol.st_other, ArrayRef(SymOtherFlags), 0x3u);
+}
+
+template <class ELFT>
+void LLVMELFDumper<ELFT>::printZeroSymbolOtherField(
+    const Elf_Sym &Symbol) const {
+  assert(Symbol.st_other == 0 && "non-zero Other Field");
+  // Usually st_other flag is zero. Do not pollute the output
+  // by flags enumeration in that case.
+  W.printNumber("Other", 0);
+}
+
 template <class ELFT>
 void LLVMELFDumper<ELFT>::printSymbol(const Elf_Sym &Symbol, unsigned SymIndex,
                                       DataRegion<Elf_Word> ShndxTable,
@@ -6890,35 +6973,9 @@ void LLVMELFDumper<ELFT>::printSymbol(const Elf_Sym &Symbol, unsigned SymIndex,
   else
     W.printEnum("Type", SymbolType, ArrayRef(ElfSymbolTypes));
   if (Symbol.st_other == 0)
-    // Usually st_other flag is zero. Do not pollute the output
-    // by flags enumeration in that case.
-    W.printNumber("Other", 0);
-  else {
-    std::vector<EnumEntry<unsigned>> SymOtherFlags(std::begin(ElfSymOtherFlags),
-                                                   std::end(ElfSymOtherFlags));
-    if (this->Obj.getHeader().e_machine == EM_MIPS) {
-      // Someones in their infinite wisdom decided to make STO_MIPS_MIPS16
-      // flag overlapped with other ST_MIPS_xxx flags. So consider both
-      // cases separately.
-      if ((Symbol.st_other & STO_MIPS_MIPS16) == STO_MIPS_MIPS16)
-        SymOtherFlags.insert(SymOtherFlags.end(),
-                             std::begin(ElfMips16SymOtherFlags),
-                             std::end(ElfMips16SymOtherFlags));
-      else
-        SymOtherFlags.insert(SymOtherFlags.end(),
-                             std::begin(ElfMipsSymOtherFlags),
-                             std::end(ElfMipsSymOtherFlags));
-    } else if (this->Obj.getHeader().e_machine == EM_AARCH64) {
-      SymOtherFlags.insert(SymOtherFlags.end(),
-                           std::begin(ElfAArch64SymOtherFlags),
-                           std::end(ElfAArch64SymOtherFlags));
-    } else if (this->Obj.getHeader().e_machine == EM_RISCV) {
-      SymOtherFlags.insert(SymOtherFlags.end(),
-                           std::begin(ElfRISCVSymOtherFlags),
-                           std::end(ElfRISCVSymOtherFlags));
-    }
-    W.printFlags("Other", Symbol.st_other, ArrayRef(SymOtherFlags), 0x3u);
-  }
+    printZeroSymbolOtherField(Symbol);
+  else
+    printSymbolOtherField(Symbol);
   printSymbolSection(Symbol, SymIndex, ShndxTable);
 }
 
@@ -7148,14 +7205,19 @@ static bool getSymbolIndices(const typename ELFT::Shdr *CGRelSection,
 }
 
 template <class ELFT> void LLVMELFDumper<ELFT>::printCGProfile() {
-  llvm::MapVector<const Elf_Shdr *, const Elf_Shdr *> SecToRelocMap;
-
   auto IsMatch = [](const Elf_Shdr &Sec) -> bool {
     return Sec.sh_type == ELF::SHT_LLVM_CALL_GRAPH_PROFILE;
   };
-  this->getSectionAndRelocations(IsMatch, SecToRelocMap);
 
-  for (const auto &CGMapEntry : SecToRelocMap) {
+  Expected<MapVector<const Elf_Shdr *, const Elf_Shdr *>> SecToRelocMapOrErr =
+      this->Obj.getSectionAndRelocations(IsMatch);
+  if (!SecToRelocMapOrErr) {
+    this->reportUniqueWarning("unable to get CG Profile section(s): " +
+                              toString(SecToRelocMapOrErr.takeError()));
+    return;
+  }
+
+  for (const auto &CGMapEntry : *SecToRelocMapOrErr) {
     const Elf_Shdr *CGSection = CGMapEntry.first;
     const Elf_Shdr *CGRelSection = CGMapEntry.second;
 
@@ -7194,21 +7256,35 @@ template <class ELFT> void LLVMELFDumper<ELFT>::printCGProfile() {
 
 template <class ELFT> void LLVMELFDumper<ELFT>::printBBAddrMaps() {
   bool IsRelocatable = this->Obj.getHeader().e_type == ELF::ET_REL;
-  for (const Elf_Shdr &Sec : cantFail(this->Obj.sections())) {
-    if (Sec.sh_type != SHT_LLVM_BB_ADDR_MAP &&
-        Sec.sh_type != SHT_LLVM_BB_ADDR_MAP_V0) {
-      continue;
-    }
+  using Elf_Shdr = typename ELFT::Shdr;
+  auto IsMatch = [](const Elf_Shdr &Sec) -> bool {
+    return Sec.sh_type == ELF::SHT_LLVM_BB_ADDR_MAP ||
+           Sec.sh_type == ELF::SHT_LLVM_BB_ADDR_MAP_V0;
+  };
+  Expected<MapVector<const Elf_Shdr *, const Elf_Shdr *>> SecRelocMapOrErr =
+      this->Obj.getSectionAndRelocations(IsMatch);
+  if (!SecRelocMapOrErr) {
+    this->reportUniqueWarning(
+        "failed to get SHT_LLVM_BB_ADDR_MAP section(s): " +
+        toString(SecRelocMapOrErr.takeError()));
+    return;
+  }
+  for (auto const &[Sec, RelocSec] : *SecRelocMapOrErr) {
     std::optional<const Elf_Shdr *> FunctionSec;
     if (IsRelocatable)
       FunctionSec =
-          unwrapOrError(this->FileName, this->Obj.getSection(Sec.sh_link));
+          unwrapOrError(this->FileName, this->Obj.getSection(Sec->sh_link));
     ListScope L(W, "BBAddrMap");
+    if (IsRelocatable && !RelocSec) {
+      this->reportUniqueWarning("unable to get relocation section for " +
+                                this->describe(*Sec));
+      continue;
+    }
     Expected<std::vector<BBAddrMap>> BBAddrMapOrErr =
-        this->Obj.decodeBBAddrMap(Sec);
+        this->Obj.decodeBBAddrMap(*Sec, RelocSec);
     if (!BBAddrMapOrErr) {
-      this->reportUniqueWarning("unable to dump " + this->describe(Sec) + ": " +
-                                toString(BBAddrMapOrErr.takeError()));
+      this->reportUniqueWarning("unable to dump " + this->describe(*Sec) +
+                                ": " + toString(BBAddrMapOrErr.takeError()));
       continue;
     }
     for (const BBAddrMap &AM : *BBAddrMapOrErr) {
@@ -7220,7 +7296,7 @@ template <class ELFT> void LLVMELFDumper<ELFT>::printBBAddrMaps() {
       if (FuncSymIndex.empty())
         this->reportUniqueWarning(
             "could not identify function symbol for address (0x" +
-            Twine::utohexstr(AM.Addr) + ") in " + this->describe(Sec));
+            Twine::utohexstr(AM.Addr) + ") in " + this->describe(*Sec));
       else
         FuncName = this->getStaticSymbolName(FuncSymIndex.front());
       W.printString("Name", FuncName);
@@ -7656,3 +7732,45 @@ void JSONELFDumper<ELFT>::printFileSummary(StringRef FileStr, ObjectFile &Obj,
       std::string(formatv("{0}bit", 8 * Obj.getBytesInAddress())));
   this->printLoadName();
 }
+
+template <class ELFT>
+void JSONELFDumper<ELFT>::printZeroSymbolOtherField(
+    const Elf_Sym &Symbol) const {
+  // We want the JSON format to be uniform, since it is machine readable, so
+  // always print the `Other` field the same way.
+  this->printSymbolOtherField(Symbol);
+}
+
+template <class ELFT>
+void JSONELFDumper<ELFT>::printDefaultRelRelaReloc(const Relocation<ELFT> &R,
+                                                   StringRef SymbolName,
+                                                   StringRef RelocName) {
+  this->printExpandedRelRelaReloc(R, SymbolName, RelocName);
+}
+
+template <class ELFT>
+void JSONELFDumper<ELFT>::printRelocationSectionInfo(const Elf_Shdr &Sec,
+                                                     StringRef Name,
+                                                     const unsigned SecNdx) {
+  DictScope Group(this->W);
+  this->W.printNumber("SectionIndex", SecNdx);
+  ListScope D(this->W, "Relocs");
+  this->printRelocationsHelper(Sec);
+}
+
+template <class ELFT>
+std::string JSONELFDumper<ELFT>::getGroupSectionHeaderName() const {
+  return "GroupSections";
+}
+
+template <class ELFT>
+void JSONELFDumper<ELFT>::printSectionGroupMembers(StringRef Name,
+                                                   uint64_t Idx) const {
+  DictScope Grp(this->W);
+  this->W.printString("Name", Name);
+  this->W.printNumber("Index", Idx);
+}
+
+template <class ELFT> void JSONELFDumper<ELFT>::printEmptyGroupMessage() const {
+  // JSON output does not need to print anything for empty groups
+}
diff --git a/llvm/tools/llvm-reduce/deltas/ReduceArguments.cpp b/llvm/tools/llvm-reduce/deltas/ReduceArguments.cpp
index 10bbc64340325..d3572e2f5abf7 100644
--- a/llvm/tools/llvm-reduce/deltas/ReduceArguments.cpp
+++ b/llvm/tools/llvm-reduce/deltas/ReduceArguments.cpp
@@ -101,9 +101,9 @@ static void extractArgumentsFromModule(Oracle &O, ReducerWorkItem &WorkItem) {
       continue;
 
     std::set<int> ArgIndexesToKeep;
-    for (auto &Arg : enumerate(F->args()))
-      if (ArgsToKeep.count(&Arg.value()))
-        ArgIndexesToKeep.insert(Arg.index());
+    for (const auto &[Index, Arg] : enumerate(F->args()))
+      if (ArgsToKeep.count(&Arg))
+        ArgIndexesToKeep.insert(Index);
 
     auto *ClonedFunc = CloneFunction(F, VMap);
     // In order to preserve function order, we move Clone after old Function
diff --git a/llvm/tools/llvm-xray/xray-account.cpp b/llvm/tools/llvm-xray/xray-account.cpp
index a9d91297b4e18..24a3552cfb91e 100644
--- a/llvm/tools/llvm-xray/xray-account.cpp
+++ b/llvm/tools/llvm-xray/xray-account.cpp
@@ -80,7 +80,7 @@ static cl::opt<SortField> AccountSortOutput(
     "sort", cl::desc("sort output by this field"), cl::value_desc("field"),
     cl::sub(Account), cl::init(SortField::FUNCID),
     cl::values(clEnumValN(SortField::FUNCID, "funcid", "function id"),
-               clEnumValN(SortField::COUNT, "count", "funciton call counts"),
+               clEnumValN(SortField::COUNT, "count", "function call counts"),
                clEnumValN(SortField::MIN, "min", "minimum function durations"),
                clEnumValN(SortField::MED, "med", "median function durations"),
                clEnumValN(SortField::PCT90, "90p", "90th percentile durations"),
diff --git a/llvm/tools/llvm-xray/xray-graph.cpp b/llvm/tools/llvm-xray/xray-graph.cpp
index b8328052f4731..de67993d75901 100644
--- a/llvm/tools/llvm-xray/xray-graph.cpp
+++ b/llvm/tools/llvm-xray/xray-graph.cpp
@@ -200,7 +200,7 @@ static std::string escapeString(StringRef Label) {
 // example caused by tail call elimination and if the option is enabled then
 // then tries to recover from this.
 //
-// This funciton will also error if the records are out of order, as the trace
+// This function will also error if the records are out of order, as the trace
 // is expected to be sorted.
 //
 // The graph generated has an immaginary root for functions called by no-one at
diff --git a/llvm/tools/opt/NewPMDriver.cpp b/llvm/tools/opt/NewPMDriver.cpp
index bb9711e7aa65a..57d3d2e86aa30 100644
--- a/llvm/tools/opt/NewPMDriver.cpp
+++ b/llvm/tools/opt/NewPMDriver.cpp
@@ -395,20 +395,20 @@ bool llvm::runPassPipeline(StringRef Arg0, Module &M, TargetMachine *TM,
   PrintPassOpts.SkipAnalyses = DebugPM == DebugLogging::Quiet;
   StandardInstrumentations SI(M.getContext(), DebugPM != DebugLogging::None,
                               VerifyEachPass, PrintPassOpts);
-  SI.registerCallbacks(PIC, &FAM);
+  SI.registerCallbacks(PIC, &MAM);
   DebugifyEachInstrumentation Debugify;
   DebugifyStatsMap DIStatsMap;
   DebugInfoPerPass DebugInfoBeforePass;
   if (DebugifyEach) {
     Debugify.setDIStatsMap(DIStatsMap);
     Debugify.setDebugifyMode(DebugifyMode::SyntheticDebugInfo);
-    Debugify.registerCallbacks(PIC);
+    Debugify.registerCallbacks(PIC, MAM);
   } else if (VerifyEachDebugInfoPreserve) {
     Debugify.setDebugInfoBeforePass(DebugInfoBeforePass);
     Debugify.setDebugifyMode(DebugifyMode::OriginalDebugInfo);
     Debugify.setOrigDIVerifyBugsReportFilePath(
       VerifyDIPreserveExport);
-    Debugify.registerCallbacks(PIC);
+    Debugify.registerCallbacks(PIC, MAM);
   }
 
   PipelineTuningOptions PTO;
diff --git a/llvm/tools/opt/opt.cpp b/llvm/tools/opt/opt.cpp
index 1b07bbe094163..6a582fb0d425b 100644
--- a/llvm/tools/opt/opt.cpp
+++ b/llvm/tools/opt/opt.cpp
@@ -68,12 +68,12 @@ static codegen::RegisterCodeGenFlags CFG;
 static cl::list<const PassInfo *, bool, PassNameParser> PassList(cl::desc(
     "Optimizations available (use '-passes=' for the new pass manager)"));
 
-static cl::opt<bool> EnableNewPassManager(
-    "enable-new-pm",
-    cl::desc("Enable the new pass manager, translating "
-             "'opt -foo' to 'opt -passes=foo'. This is strictly for the new PM "
-             "migration, use '-passes=' when possible."),
-    cl::init(true));
+static cl::opt<bool> EnableLegacyPassManager(
+    "bugpoint-enable-legacy-pm",
+    cl::desc(
+        "Enable the legacy pass manager. This is strictly for bugpoint "
+        "due to it not working with the new PM, please do not use otherwise."),
+    cl::init(false));
 
 // This flag specifies a textual description of the optimization pass pipeline
 // to run over the module. This flag switches opt to use the new pass manager
@@ -485,11 +485,8 @@ int main(int argc, char **argv) {
 
   LLVMContext Context;
 
-  // If `-passes=` is specified, use NPM.
-  // If `-enable-new-pm` is specified and there are no codegen passes, use NPM.
-  // e.g. `-enable-new-pm -sroa` will use NPM.
-  // but `-enable-new-pm -codegenprepare` will still revert to legacy PM.
-  const bool UseNPM = (EnableNewPassManager && !shouldForceLegacyPM()) ||
+  // TODO: remove shouldForceLegacyPM().
+  const bool UseNPM = (!EnableLegacyPassManager && !shouldForceLegacyPM()) ||
                       PassPipeline.getNumOccurrences() > 0;
 
   if (UseNPM && !PassList.empty()) {
diff --git a/llvm/tools/sancov/coverage-report-server.py b/llvm/tools/sancov/coverage-report-server.py
index 5ea978fae642a..4c666dbc111d8 100755
--- a/llvm/tools/sancov/coverage-report-server.py
+++ b/llvm/tools/sancov/coverage-report-server.py
@@ -71,6 +71,8 @@
 </html>
 """
 
+FILE_URI_PREFIX = "/file/"
+
 class SymcovData:
     def __init__(self, symcov_json):
         self.covered_points = frozenset(symcov_json['covered-points'])
@@ -129,7 +131,7 @@ class ServerHandler(http.server.BaseHTTPRequestHandler):
     src_path = None
 
     def do_GET(self):
-        norm_path = os.path.normpath(urllib.parse.unquote(self.path[1:]))
+        norm_path = os.path.normpath(urllib.parse.unquote(self.path[len(FILE_URI_PREFIX):]))
         if self.path == '/':
             self.send_response(200)
             self.send_header("Content-type", "text/html; charset=utf-8")
@@ -141,8 +143,9 @@ def do_GET(self):
                 if not file_coverage:
                     continue
                 filelist.append(
-                        "<tr><td><a href=\"./{name}\">{name}</a></td>"
+                        "<tr><td><a href=\"{prefix}{name}\">{name}</a></td>"
                         "<td>{coverage}%</td></tr>".format(
+                            prefix=FILE_URI_PREFIX,
                             name=html.escape(filename, quote=True), 
                             coverage=format_pct(file_coverage)))
 
diff --git a/llvm/unittests/ADT/DenseMapTest.cpp b/llvm/unittests/ADT/DenseMapTest.cpp
index 69bf4e19e4eea..79d57d35265c2 100644
--- a/llvm/unittests/ADT/DenseMapTest.cpp
+++ b/llvm/unittests/ADT/DenseMapTest.cpp
@@ -122,6 +122,7 @@ TYPED_TEST(DenseMapTest, EmptyIntMapTest) {
 
   // Lookup tests
   EXPECT_FALSE(this->Map.count(this->getKey()));
+  EXPECT_FALSE(this->Map.contains(this->getKey()));
   EXPECT_TRUE(this->Map.find(this->getKey()) == this->Map.end());
   EXPECT_EQ(typename TypeParam::mapped_type(),
             this->Map.lookup(this->getKey()));
@@ -153,6 +154,7 @@ TYPED_TEST(DenseMapTest, SingleEntryMapTest) {
 
   // Lookup tests
   EXPECT_TRUE(this->Map.count(this->getKey()));
+  EXPECT_TRUE(this->Map.contains(this->getKey()));
   EXPECT_TRUE(this->Map.find(this->getKey()) == this->Map.begin());
   EXPECT_EQ(this->getValue(), this->Map.lookup(this->getKey()));
   EXPECT_EQ(this->getValue(), this->Map[this->getKey()]);
diff --git a/llvm/unittests/ADT/IteratorTest.cpp b/llvm/unittests/ADT/IteratorTest.cpp
index b2a11c4c6bd7d..7d10729c2dd9f 100644
--- a/llvm/unittests/ADT/IteratorTest.cpp
+++ b/llvm/unittests/ADT/IteratorTest.cpp
@@ -692,6 +692,49 @@ TEST(ZipIteratorTest, Reverse) {
   EXPECT_TRUE(all_of(ascending, [](unsigned n) { return (n & 0x01) == 0; }));
 }
 
+// Int iterator that keeps track of the number of its copies.
+struct CountingIntIterator : IntIterator {
+  unsigned *cnt;
+
+  CountingIntIterator(int *it, unsigned &counter)
+      : IntIterator(it), cnt(&counter) {}
+
+  CountingIntIterator(const CountingIntIterator &other)
+      : IntIterator(other.I), cnt(other.cnt) {
+    ++(*cnt);
+  }
+  CountingIntIterator &operator=(const CountingIntIterator &other) {
+    this->I = other.I;
+    this->cnt = other.cnt;
+    ++(*cnt);
+    return *this;
+  }
+};
+
+// Check that the iterators do not get copied with each `zippy` iterator
+// increment.
+TEST(ZipIteratorTest, IteratorCopies) {
+  std::vector<int> ints(1000, 42);
+  unsigned total_copy_count = 0;
+  CountingIntIterator begin(ints.data(), total_copy_count);
+  CountingIntIterator end(ints.data() + ints.size(), total_copy_count);
+
+  size_t iters = 0;
+  auto zippy = zip_equal(ints, llvm::make_range(begin, end));
+  const unsigned creation_copy_count = total_copy_count;
+
+  for (auto [a, b] : zippy) {
+    EXPECT_EQ(a, b);
+    ++iters;
+  }
+  EXPECT_EQ(iters, ints.size());
+
+  // We expect the number of copies to be much smaller than the number of loop
+  // iterations.
+  unsigned loop_copy_count = total_copy_count - creation_copy_count;
+  EXPECT_LT(loop_copy_count, 10u);
+}
+
 TEST(RangeTest, Distance) {
   std::vector<int> v1;
   std::vector<int> v2{1, 2, 3};
diff --git a/llvm/unittests/ADT/MapVectorTest.cpp b/llvm/unittests/ADT/MapVectorTest.cpp
index 552f9956bdc2c..1a371cbfba81e 100644
--- a/llvm/unittests/ADT/MapVectorTest.cpp
+++ b/llvm/unittests/ADT/MapVectorTest.cpp
@@ -87,8 +87,10 @@ TEST(MapVectorTest, erase) {
   MV.insert(std::make_pair(5, 6));
   ASSERT_EQ(MV.size(), 3u);
 
+  ASSERT_TRUE(MV.contains(1));
   MV.erase(MV.find(1));
   ASSERT_EQ(MV.size(), 2u);
+  ASSERT_FALSE(MV.contains(1));
   ASSERT_EQ(MV.find(1), MV.end());
   ASSERT_EQ(MV[3], 4);
   ASSERT_EQ(MV[5], 6);
diff --git a/llvm/unittests/ADT/RangeAdapterTest.cpp b/llvm/unittests/ADT/RangeAdapterTest.cpp
index 3c68103e17452..5c27225360248 100644
--- a/llvm/unittests/ADT/RangeAdapterTest.cpp
+++ b/llvm/unittests/ADT/RangeAdapterTest.cpp
@@ -157,14 +157,12 @@ TYPED_TEST(RangeAdapterRValueTest, HasRbegin) {
 
 TYPED_TEST(RangeAdapterRValueTest, RangeType) {
   static_assert(
-      std::is_same<
-          decltype(reverse(*static_cast<TypeParam *>(nullptr)).begin()),
-          decltype(static_cast<TypeParam *>(nullptr)->rbegin())>::value,
+      std::is_same_v<decltype(reverse(std::declval<TypeParam>()).begin()),
+                     decltype(std::declval<TypeParam>().rbegin())>,
       "reverse().begin() should have the same type as rbegin()");
   static_assert(
-      std::is_same<
-          decltype(reverse(*static_cast<const TypeParam *>(nullptr)).begin()),
-          decltype(static_cast<const TypeParam *>(nullptr)->rbegin())>::value,
+      std::is_same_v<decltype(reverse(std::declval<const TypeParam>()).begin()),
+                     decltype(std::declval<const TypeParam>().rbegin())>,
       "reverse().begin() should have the same type as rbegin() [const]");
 }
 
diff --git a/llvm/unittests/ADT/STLExtrasTest.cpp b/llvm/unittests/ADT/STLExtrasTest.cpp
index a02b1dc8af22c..bb602bb6c39f7 100644
--- a/llvm/unittests/ADT/STLExtrasTest.cpp
+++ b/llvm/unittests/ADT/STLExtrasTest.cpp
@@ -13,7 +13,12 @@
 
 #include <array>
 #include <climits>
+#include <cstddef>
+#include <initializer_list>
 #include <list>
+#include <tuple>
+#include <type_traits>
+#include <utility>
 #include <vector>
 
 using namespace llvm;
@@ -151,6 +156,131 @@ TEST(STLExtrasTest, EnumerateModifyRValue) {
                                    PairType(2u, '4')));
 }
 
+TEST(STLExtrasTest, EnumerateTwoRanges) {
+  using Tuple = std::tuple<size_t, int, bool>;
+
+  std::vector<int> Ints = {1, 2};
+  std::vector<bool> Bools = {true, false};
+  EXPECT_THAT(llvm::enumerate(Ints, Bools),
+              ElementsAre(Tuple(0, 1, true), Tuple(1, 2, false)));
+
+  // Check that we can modify the values when the temporary is a const
+  // reference.
+  for (const auto &[Idx, Int, Bool] : llvm::enumerate(Ints, Bools)) {
+    (void)Idx;
+    Bool = false;
+    Int = -1;
+  }
+
+  EXPECT_THAT(Ints, ElementsAre(-1, -1));
+  EXPECT_THAT(Bools, ElementsAre(false, false));
+
+  // Check that we can modify the values when the result gets copied.
+  for (auto [Idx, Bool, Int] : llvm::enumerate(Bools, Ints)) {
+    (void)Idx;
+    Int = 3;
+    Bool = true;
+  }
+
+  EXPECT_THAT(Ints, ElementsAre(3, 3));
+  EXPECT_THAT(Bools, ElementsAre(true, true));
+
+  // Check that we can modify the values through `.value()`.
+  size_t Iters = 0;
+  for (auto It : llvm::enumerate(Bools, Ints)) {
+    EXPECT_EQ(It.index(), Iters);
+    ++Iters;
+
+    std::get<0>(It.value()) = false;
+    std::get<1>(It.value()) = 4;
+  }
+
+  EXPECT_THAT(Ints, ElementsAre(4, 4));
+  EXPECT_THAT(Bools, ElementsAre(false, false));
+}
+
+TEST(STLExtrasTest, EnumerateThreeRanges) {
+  using Tuple = std::tuple<size_t, int, bool, char>;
+
+  std::vector<int> Ints = {1, 2};
+  std::vector<bool> Bools = {true, false};
+  char Chars[] = {'X', 'D'};
+  EXPECT_THAT(llvm::enumerate(Ints, Bools, Chars),
+              ElementsAre(Tuple(0, 1, true, 'X'), Tuple(1, 2, false, 'D')));
+
+  for (auto [Idx, Int, Bool, Char] : llvm::enumerate(Ints, Bools, Chars)) {
+    (void)Idx;
+    Int = 0;
+    Bool = true;
+    Char = '!';
+  }
+
+  EXPECT_THAT(Ints, ElementsAre(0, 0));
+  EXPECT_THAT(Bools, ElementsAre(true, true));
+  EXPECT_THAT(Chars, ElementsAre('!', '!'));
+
+  // Check that we can modify the values through `.values()`.
+  size_t Iters = 0;
+  for (auto It : llvm::enumerate(Ints, Bools, Chars)) {
+    EXPECT_EQ(It.index(), Iters);
+    ++Iters;
+    auto [Int, Bool, Char] = It.value();
+    Int = 42;
+    Bool = false;
+    Char = '$';
+  }
+
+  EXPECT_THAT(Ints, ElementsAre(42, 42));
+  EXPECT_THAT(Bools, ElementsAre(false, false));
+  EXPECT_THAT(Chars, ElementsAre('$', '$'));
+}
+
+TEST(STLExtrasTest, EnumerateTemporaries) {
+  using Tuple = std::tuple<size_t, int, bool>;
+
+  EXPECT_THAT(
+      llvm::enumerate(llvm::SmallVector<int>({1, 2, 3}),
+                      std::vector<bool>({true, false, true})),
+      ElementsAre(Tuple(0, 1, true), Tuple(1, 2, false), Tuple(2, 3, true)));
+
+  size_t Iters = 0;
+  // This is fine from the point of view of range lifetimes because `zippy` will
+  // move all temporaries into its storage. No lifetime extension is necessary.
+  for (auto [Idx, Int, Bool] :
+       llvm::enumerate(llvm::SmallVector<int>({1, 2, 3}),
+                       std::vector<bool>({true, false, true}))) {
+    EXPECT_EQ(Idx, Iters);
+    ++Iters;
+    Int = 0;
+    Bool = true;
+  }
+
+  Iters = 0;
+  // The same thing but with the result as a const reference.
+  for (const auto &[Idx, Int, Bool] :
+       llvm::enumerate(llvm::SmallVector<int>({1, 2, 3}),
+                       std::vector<bool>({true, false, true}))) {
+    EXPECT_EQ(Idx, Iters);
+    ++Iters;
+    Int = 0;
+    Bool = true;
+  }
+}
+
+#if defined(GTEST_HAS_DEATH_TEST) && !defined(NDEBUG)
+TEST(STLExtrasTest, EnumerateDifferentLengths) {
+  std::vector<int> Ints = {0, 1};
+  bool Bools[] = {true, false, true};
+  std::string Chars = "abc";
+  EXPECT_DEATH(llvm::enumerate(Ints, Bools, Chars),
+               "Ranges have different length");
+  EXPECT_DEATH(llvm::enumerate(Bools, Ints, Chars),
+               "Ranges have different length");
+  EXPECT_DEATH(llvm::enumerate(Bools, Chars, Ints),
+               "Ranges have different length");
+}
+#endif
+
 template <bool B> struct CanMove {};
 template <> struct CanMove<false> {
   CanMove(CanMove &&) = delete;
@@ -188,8 +318,8 @@ class Counted : CanMove<Moveable>, CanCopy<Copyable> {
 template <bool Moveable, bool Copyable>
 struct Range : Counted<Moveable, Copyable> {
   using Counted<Moveable, Copyable>::Counted;
-  int *begin() { return nullptr; }
-  int *end() { return nullptr; }
+  int *begin() const { return nullptr; }
+  int *end() const { return nullptr; }
 };
 
 TEST(STLExtrasTest, EnumerateLifetimeSemanticsPRValue) {
@@ -1027,6 +1157,56 @@ TEST(STLExtrasTest, IsContainedInitializerList) {
   static_assert(!is_contained({1, 2, 3, 4}, 5), "It's not there :(");
 }
 
+TEST(STLExtrasTest, IsContainedMemberContains) {
+  // Check that `llvm::is_contained` uses the member `.contains()` when
+  // available. Check that `.contains()` is preferred over `.find()`.
+  struct Foo {
+    bool contains(int) const {
+      ++NumContainsCalls;
+      return ContainsResult;
+    }
+    int *begin() { return nullptr; }
+    int *end() { return nullptr; }
+    int *find(int) { return nullptr; }
+
+    bool ContainsResult = false;
+    mutable unsigned NumContainsCalls = 0;
+  } Container;
+
+  EXPECT_EQ(Container.NumContainsCalls, 0u);
+  EXPECT_FALSE(is_contained(Container, 1));
+  EXPECT_EQ(Container.NumContainsCalls, 1u);
+
+  Container.ContainsResult = true;
+  EXPECT_TRUE(is_contained(Container, 1));
+  EXPECT_EQ(Container.NumContainsCalls, 2u);
+}
+
+TEST(STLExtrasTest, IsContainedMemberFind) {
+  // Check that `llvm::is_contained` uses the member `.find(x)` when available.
+  struct Foo {
+    auto begin() { return Data.begin(); }
+    auto end() { return Data.end(); }
+    auto find(int X) {
+      ++NumFindCalls;
+      return std::find(begin(), end(), X);
+    }
+
+    std::vector<int> Data;
+    mutable unsigned NumFindCalls = 0;
+  } Container;
+
+  Container.Data = {1, 2, 3};
+
+  EXPECT_EQ(Container.NumFindCalls, 0u);
+  EXPECT_TRUE(is_contained(Container, 1));
+  EXPECT_TRUE(is_contained(Container, 3));
+  EXPECT_EQ(Container.NumFindCalls, 2u);
+
+  EXPECT_FALSE(is_contained(Container, 4));
+  EXPECT_EQ(Container.NumFindCalls, 3u);
+}
+
 TEST(STLExtrasTest, addEnumValues) {
   enum A { Zero = 0, One = 1 };
   enum B { IntMax = INT_MAX, ULongLongMax = ULLONG_MAX };
@@ -1055,4 +1235,36 @@ TEST(STLExtrasTest, addEnumValues) {
                 "addEnumValues(ULongLongMax, C::Two) failed.");
 }
 
+TEST(STLExtrasTest, LessFirst) {
+  {
+    std::pair<int, int> A(0, 1);
+    std::pair<int, int> B(1, 0);
+    EXPECT_TRUE(less_first()(A, B));
+    EXPECT_FALSE(less_first()(B, A));
+  }
+
+  {
+    std::tuple<int, int> A(0, 1);
+    std::tuple<int, int> B(1, 0);
+    EXPECT_TRUE(less_first()(A, B));
+    EXPECT_FALSE(less_first()(B, A));
+  }
+}
+
+TEST(STLExtrasTest, LessSecond) {
+  {
+    std::pair<int, int> A(0, 1);
+    std::pair<int, int> B(1, 0);
+    EXPECT_FALSE(less_second()(A, B));
+    EXPECT_TRUE(less_second()(B, A));
+  }
+
+  {
+    std::tuple<int, int> A(0, 1);
+    std::tuple<int, int> B(1, 0);
+    EXPECT_FALSE(less_second()(A, B));
+    EXPECT_TRUE(less_second()(B, A));
+  }
+}
+
 } // namespace
diff --git a/llvm/unittests/ADT/SequenceTest.cpp b/llvm/unittests/ADT/SequenceTest.cpp
index aec1ea457aeb3..acc15bf4daae6 100644
--- a/llvm/unittests/ADT/SequenceTest.cpp
+++ b/llvm/unittests/ADT/SequenceTest.cpp
@@ -296,4 +296,13 @@ TEST(SequenceTest, NonIterableEnums) {
               ElementsAre(UntypedEnum::A));
 }
 
+// Reproducer for https://github.com/llvm/llvm-project/issues/61122
+TEST(SequenceTest, CorrectReferenceType) {
+  std::vector<int> vals = {1, 2, 3};
+  detail::SafeIntIterator<int, false> begin(4);
+  detail::SafeIntIterator<int, false> end(6);
+  vals.insert(vals.end(), begin, end);
+  EXPECT_THAT(vals, ElementsAre(1, 2, 3, 4, 5));
+}
+
 } // namespace
diff --git a/llvm/unittests/ADT/StringMapTest.cpp b/llvm/unittests/ADT/StringMapTest.cpp
index 431b3973d6d6c..f9b138e9a4721 100644
--- a/llvm/unittests/ADT/StringMapTest.cpp
+++ b/llvm/unittests/ADT/StringMapTest.cpp
@@ -41,6 +41,7 @@ class StringMapTest : public testing::Test {
     EXPECT_TRUE(testMap.begin() == testMap.end());
 
     // Lookup tests
+    EXPECT_FALSE(testMap.contains(testKey));
     EXPECT_EQ(0u, testMap.count(testKey));
     EXPECT_EQ(0u, testMap.count(StringRef(testKeyFirst, testKeyLength)));
     EXPECT_EQ(0u, testMap.count(testKeyStr));
@@ -64,6 +65,7 @@ class StringMapTest : public testing::Test {
     EXPECT_TRUE(it == testMap.end());
 
     // Lookup tests
+    EXPECT_TRUE(testMap.contains(testKey));
     EXPECT_EQ(1u, testMap.count(testKey));
     EXPECT_EQ(1u, testMap.count(StringRef(testKeyFirst, testKeyLength)));
     EXPECT_EQ(1u, testMap.count(testKeyStr));
diff --git a/llvm/unittests/Analysis/AssumeBundleQueriesTest.cpp b/llvm/unittests/Analysis/AssumeBundleQueriesTest.cpp
index 1286ec9e34b0e..ef84ed5d99d68 100644
--- a/llvm/unittests/Analysis/AssumeBundleQueriesTest.cpp
+++ b/llvm/unittests/Analysis/AssumeBundleQueriesTest.cpp
@@ -232,7 +232,8 @@ static bool FindExactlyAttributes(RetainedKnowledgeMap &Map, Value *WasOn,
        }) {
     bool ShouldHaveAttr = Reg.match(Attr, &Matches) && Matches[0] == Attr;
 
-    if (ShouldHaveAttr != (Map.find(RetainedKnowledgeKey{WasOn, Attribute::getAttrKindFromName(Attr)}) != Map.end()))
+    if (ShouldHaveAttr != (Map.contains(RetainedKnowledgeKey{
+                              WasOn, Attribute::getAttrKindFromName(Attr)})))
       return false;
   }
   return true;
diff --git a/llvm/unittests/Analysis/CMakeLists.txt b/llvm/unittests/Analysis/CMakeLists.txt
index f7f17b394faad..91a129edbc145 100644
--- a/llvm/unittests/Analysis/CMakeLists.txt
+++ b/llvm/unittests/Analysis/CMakeLists.txt
@@ -39,6 +39,7 @@ set(ANALYSIS_TEST_SOURCES
   MLModelRunnerTest.cpp
   PhiValuesTest.cpp
   PluginInlineAdvisorAnalysisTest.cpp
+  PluginInlineOrderAnalysisTest.cpp
   ProfileSummaryInfoTest.cpp
   ScalarEvolutionTest.cpp
   VectorFunctionABITest.cpp
@@ -72,4 +73,10 @@ if(CMAKE_SYSTEM_NAME STREQUAL "AIX")
   set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,-brtl")
 endif()
 
+# Export symbols from the plugins shared objects.
+if(NOT WIN32)
+  export_executable_symbols_for_plugins(AnalysisTests)
+endif()
+
 add_subdirectory(InlineAdvisorPlugin)
+add_subdirectory(InlineOrderPlugin)
diff --git a/llvm/unittests/Analysis/InlineAdvisorPlugin/CMakeLists.txt b/llvm/unittests/Analysis/InlineAdvisorPlugin/CMakeLists.txt
index aa319abcb9e8c..b0faf19268fa5 100644
--- a/llvm/unittests/Analysis/InlineAdvisorPlugin/CMakeLists.txt
+++ b/llvm/unittests/Analysis/InlineAdvisorPlugin/CMakeLists.txt
@@ -14,7 +14,6 @@ if (NOT WIN32)
       )
   set_target_properties(InlineAdvisorPlugin PROPERTIES FOLDER "Tests")
 
-  export_executable_symbols_for_plugins(AnalysisTests)
   # The plugin depends on some of the output files of intrinsics_gen, so make sure
   # it is built before the plugin.
   add_dependencies(InlineAdvisorPlugin intrinsics_gen)
diff --git a/llvm/unittests/Analysis/InlineOrderPlugin/CMakeLists.txt b/llvm/unittests/Analysis/InlineOrderPlugin/CMakeLists.txt
new file mode 100644
index 0000000000000..0ed3f04a79b46
--- /dev/null
+++ b/llvm/unittests/Analysis/InlineOrderPlugin/CMakeLists.txt
@@ -0,0 +1,22 @@
+# The order plugin expects to not link against the Analysis, Support and Core 
+# libraries, but expects them to exist in the process loading the plugin. This 
+# doesn't work with DLLs on Windows (where a shared library can't have undefined
+# references), so just skip this testcase on Windows.
+if (NOT WIN32)
+  unset(LLVM_LINK_COMPONENTS)
+  add_llvm_library(InlineOrderPlugin MODULE BUILDTREE_ONLY
+    InlineOrderPlugin.cpp
+    )
+  # Put PLUGIN next to the unit test executable.
+  set_output_directory(InlineOrderPlugin
+      BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR}/${CMAKE_CFG_INTDIR}/../
+      LIBRARY_DIR ${CMAKE_CURRENT_BINARY_DIR}/${CMAKE_CFG_INTDIR}/../
+      )
+  set_target_properties(InlineOrderPlugin PROPERTIES FOLDER "Tests")
+
+  # The plugin depends on some of the output files of intrinsics_gen, so make sure
+  # it is built before the plugin.
+  add_dependencies(InlineOrderPlugin intrinsics_gen)
+  add_dependencies(AnalysisTests InlineOrderPlugin)
+  set_property(TARGET InlineOrderPlugin PROPERTY FOLDER "Tests/UnitTests/AnalysisTests")
+endif()
diff --git a/llvm/unittests/Analysis/InlineOrderPlugin/InlineOrderPlugin.cpp b/llvm/unittests/Analysis/InlineOrderPlugin/InlineOrderPlugin.cpp
new file mode 100644
index 0000000000000..9c46c1b8e22ba
--- /dev/null
+++ b/llvm/unittests/Analysis/InlineOrderPlugin/InlineOrderPlugin.cpp
@@ -0,0 +1,70 @@
+#include "llvm/IR/Function.h"
+#include "llvm/IR/Module.h"
+#include "llvm/Pass.h"
+#include "llvm/Passes/PassBuilder.h"
+#include "llvm/Passes/PassPlugin.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/raw_ostream.h"
+
+#include "llvm/Analysis/InlineOrder.h"
+
+using namespace llvm;
+
+namespace {
+
+class NoFooInlineOrder : public InlineOrder<std::pair<CallBase *, int>> {
+public:
+  NoFooInlineOrder(FunctionAnalysisManager &FAM, const InlineParams &Params,
+                   ModuleAnalysisManager &MAM, Module &M) {
+    DefaultInlineOrder = getDefaultInlineOrder(FAM, Params, MAM, M);
+  }
+  size_t size() override { return DefaultInlineOrder->size(); }
+  void push(const std::pair<CallBase *, int> &Elt) override {
+    // We ignore calles named "foo"
+    if (Elt.first->getCalledFunction()->getName() == "foo") {
+      DefaultInlineOrder->push(Elt);
+    }
+  }
+  std::pair<CallBase *, int> pop() override {
+    return DefaultInlineOrder->pop();
+  }
+  void erase_if(function_ref<bool(std::pair<CallBase *, int>)> Pred) override {
+    DefaultInlineOrder->erase_if(Pred);
+  }
+
+private:
+  std::unique_ptr<InlineOrder<std::pair<CallBase *, int>>> DefaultInlineOrder;
+};
+
+std::unique_ptr<InlineOrder<std::pair<CallBase *, int>>>
+NoFooInlineOrderFactory(FunctionAnalysisManager &FAM,
+                        const InlineParams &Params, ModuleAnalysisManager &MAM,
+                        Module &M) {
+  return std::make_unique<NoFooInlineOrder>(FAM, Params, MAM, M);
+}
+
+} // namespace
+
+/* New PM Registration */
+llvm::PassPluginLibraryInfo getDefaultDynamicInlineOrderPluginInfo() {
+  return {LLVM_PLUGIN_API_VERSION, "DynamicDefaultInlineOrder",
+          LLVM_VERSION_STRING, [](PassBuilder &PB) {
+            // We use the PassBuilder's callback mechanism
+            // to register our Analysis: this will register
+            // our PluginInlineOrderAnalysis instance with
+            // the ModuleAnalysisManager
+            PB.registerAnalysisRegistrationCallback(
+                [](ModuleAnalysisManager &MAM) {
+                  MAM.registerPass([] {
+                    // defaultInlineOrderFactory will be
+                    // used to create an InlineOrder
+                    return PluginInlineOrderAnalysis(NoFooInlineOrderFactory);
+                  });
+                });
+          }};
+}
+
+extern "C" LLVM_ATTRIBUTE_WEAK ::llvm::PassPluginLibraryInfo
+llvmGetPassPluginInfo() {
+  return getDefaultDynamicInlineOrderPluginInfo();
+}
diff --git a/llvm/unittests/Analysis/PluginInlineAdvisorAnalysisTest.cpp b/llvm/unittests/Analysis/PluginInlineAdvisorAnalysisTest.cpp
index 55ffa0ea622f3..9e160aae689f3 100644
--- a/llvm/unittests/Analysis/PluginInlineAdvisorAnalysisTest.cpp
+++ b/llvm/unittests/Analysis/PluginInlineAdvisorAnalysisTest.cpp
@@ -10,6 +10,8 @@
 
 namespace llvm {
 
+namespace {
+
 void anchor() {}
 
 static std::string libPath(const std::string Name = "InlineAdvisorPlugin") {
@@ -84,6 +86,12 @@ struct CompilerInstance {
                                   ThinOrFullLTOPhase::None));
   }
 
+  ~CompilerInstance() {
+    // Reset the static variable that tracks if the plugin has been registered.
+    // This is needed to allow the test to run multiple times.
+    PluginInlineAdvisorAnalysis::HasBeenRegistered = false;
+  }
+
   std::string output;
   std::unique_ptr<Module> outputM;
 
@@ -256,6 +264,8 @@ define i32 @fib_check(){
 }
   )"};
 
+} // namespace
+
 // check that loading a plugin works
 // the plugin being loaded acts identically to the default inliner
 TEST(PluginInlineAdvisorTest, PluginLoad) {
diff --git a/llvm/unittests/Analysis/PluginInlineOrderAnalysisTest.cpp b/llvm/unittests/Analysis/PluginInlineOrderAnalysisTest.cpp
new file mode 100644
index 0000000000000..46957af823fce
--- /dev/null
+++ b/llvm/unittests/Analysis/PluginInlineOrderAnalysisTest.cpp
@@ -0,0 +1,251 @@
+#include "llvm/Analysis/CallGraph.h"
+#include "llvm/AsmParser/Parser.h"
+#include "llvm/Config/config.h"
+#include "llvm/Passes/PassBuilder.h"
+#include "llvm/Passes/PassPlugin.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/raw_ostream.h"
+#include "llvm/Testing/Support/Error.h"
+#include "gtest/gtest.h"
+
+#include "llvm/Analysis/InlineOrder.h"
+
+namespace llvm {
+
+namespace {
+
+void anchor() {}
+
+std::string libPath(const std::string Name = "InlineOrderPlugin") {
+  const auto &Argvs = testing::internal::GetArgvs();
+  const char *Argv0 =
+      Argvs.size() > 0 ? Argvs[0].c_str() : "PluginInlineOrderAnalysisTest";
+  void *Ptr = (void *)(intptr_t)anchor;
+  std::string Path = sys::fs::getMainExecutable(Argv0, Ptr);
+  llvm::SmallString<256> Buf{sys::path::parent_path(Path)};
+  sys::path::append(Buf, (Name + LLVM_PLUGIN_EXT).c_str());
+  return std::string(Buf.str());
+}
+
+struct CompilerInstance {
+  LLVMContext Ctx;
+  ModulePassManager MPM;
+  InlineParams IP;
+
+  PassBuilder PB;
+  LoopAnalysisManager LAM;
+  FunctionAnalysisManager FAM;
+  CGSCCAnalysisManager CGAM;
+  ModuleAnalysisManager MAM;
+
+  SMDiagnostic Error;
+
+  // Connect the plugin to our compiler instance.
+  void setupPlugin() {
+    auto PluginPath = libPath();
+    ASSERT_NE("", PluginPath);
+    Expected<PassPlugin> Plugin = PassPlugin::Load(PluginPath);
+    ASSERT_TRUE(!!Plugin) << "Plugin path: " << PluginPath;
+    Plugin->registerPassBuilderCallbacks(PB);
+  }
+
+  CompilerInstance() {
+    IP = getInlineParams(3, 0);
+    PB.registerModuleAnalyses(MAM);
+    PB.registerCGSCCAnalyses(CGAM);
+    PB.registerFunctionAnalyses(FAM);
+    PB.registerLoopAnalyses(LAM);
+    PB.crossRegisterProxies(LAM, FAM, CGAM, MAM);
+    MPM.addPass(ModuleInlinerPass(IP, InliningAdvisorMode::Default,
+                                  ThinOrFullLTOPhase::None));
+  }
+
+  ~CompilerInstance() {
+    // Reset the static variable that tracks if the plugin has been registered.
+    // This is needed to allow the test to run multiple times.
+    PluginInlineOrderAnalysis::unregister();
+  }
+
+  std::string Output;
+  std::unique_ptr<Module> OutputM;
+
+  // Run with the dynamic inline order.
+  auto run(StringRef IR) {
+    OutputM = parseAssemblyString(IR, Error, Ctx);
+    MPM.run(*OutputM, MAM);
+    ASSERT_TRUE(OutputM);
+    Output.clear();
+    raw_string_ostream OStream{Output};
+    OutputM->print(OStream, nullptr);
+    ASSERT_TRUE(true);
+  }
+};
+
+StringRef TestIRS[] = {
+    // Simple 3 function inline case.
+    R"(
+define void @f1() {
+  call void @foo()
+  ret void
+}
+define void @foo() {
+  call void @f3()
+  ret void
+}
+define void @f3() {
+  ret void
+}
+  )",
+    // Test that has 5 functions of which 2 are recursive.
+    R"(
+define void @f1() {
+  call void @foo()
+  ret void
+}
+define void @f2() {
+  call void @foo()
+  ret void
+}
+define void @foo() {
+  call void @f4()
+  call void @f5()
+  ret void
+}
+define void @f4() {
+  ret void
+}
+define void @f5() {
+  call void @foo()
+  ret void
+}
+  )",
+    // Test with 2 mutually recursive functions and 1 function with a loop.
+    R"(
+define void @f1() {
+  call void @f2()
+  ret void
+}
+define void @f2() {
+  call void @foo()
+  ret void
+}
+define void @foo() {
+  call void @f1()
+  ret void
+}
+define void @f4() {
+  br label %loop
+loop:
+  call void @f5()
+  br label %loop
+}
+define void @f5() {
+  ret void
+}
+  )",
+    // Test that has a function that computes fibonacci in a loop, one in a
+    // recursive manner, and one that calls both and compares them.
+    R"(
+define i32 @fib_loop(i32 %n){
+    %curr = alloca i32
+    %last = alloca i32
+    %i = alloca i32
+    store i32 1, i32* %curr
+    store i32 1, i32* %last
+    store i32 2, i32* %i
+    br label %loop_cond
+  loop_cond:
+    %i_val = load i32, i32* %i
+    %cmp = icmp slt i32 %i_val, %n
+    br i1 %cmp, label %loop_body, label %loop_end
+  loop_body:
+    %curr_val = load i32, i32* %curr
+    %last_val = load i32, i32* %last
+    %add = add i32 %curr_val, %last_val
+    store i32 %add, i32* %last
+    store i32 %curr_val, i32* %curr
+    %i_val2 = load i32, i32* %i
+    %add2 = add i32 %i_val2, 1
+    store i32 %add2, i32* %i
+    br label %loop_cond
+  loop_end:
+    %curr_val3 = load i32, i32* %curr
+    ret i32 %curr_val3
+}
+
+define i32 @foo(i32 %n){
+    %cmp = icmp eq i32 %n, 0
+    %cmp2 = icmp eq i32 %n, 1
+    %or = or i1 %cmp, %cmp2
+    br i1 %or, label %if_true, label %if_false
+  if_true:
+    ret i32 1
+  if_false:
+    %sub = sub i32 %n, 1
+    %call = call i32 @foo(i32 %sub)
+    %sub2 = sub i32 %n, 2
+    %call2 = call i32 @foo(i32 %sub2)
+    %add = add i32 %call, %call2
+    ret i32 %add
+}
+
+define i32 @fib_check(){
+    %correct = alloca i32
+    %i = alloca i32
+    store i32 1, i32* %correct
+    store i32 0, i32* %i
+    br label %loop_cond
+  loop_cond:
+    %i_val = load i32, i32* %i
+    %cmp = icmp slt i32 %i_val, 10
+    br i1 %cmp, label %loop_body, label %loop_end
+  loop_body:
+    %i_val2 = load i32, i32* %i
+    %call = call i32 @fib_loop(i32 %i_val2)
+    %i_val3 = load i32, i32* %i
+    %call2 = call i32 @foo(i32 %i_val3)
+    %cmp2 = icmp ne i32 %call, %call2
+    br i1 %cmp2, label %if_true, label %if_false
+  if_true:
+    store i32 0, i32* %correct
+    br label %if_end
+  if_false:
+    br label %if_end
+  if_end:
+    %i_val4 = load i32, i32* %i
+    %add = add i32 %i_val4, 1
+    store i32 %add, i32* %i
+    br label %loop_cond
+  loop_end:
+    %correct_val = load i32, i32* %correct
+    ret i32 %correct_val
+}
+  )"};
+
+} // namespace
+
+// Check that the behaviour of a custom inline order is correct.
+// The custom order drops any functions named "foo" so all tests
+// should contain at least one function named foo.
+TEST(PluginInlineOrderTest, NoInlineFoo) {
+#if !defined(LLVM_ENABLE_PLUGINS)
+  // Skip the test if plugins are disabled.
+  GTEST_SKIP();
+#endif
+  CompilerInstance CI{};
+  CI.setupPlugin();
+
+  for (StringRef IR : TestIRS) {
+    bool FoundFoo = false;
+    CI.run(IR);
+    CallGraph CGraph = CallGraph(*CI.OutputM);
+    for (auto &Node : CGraph) {
+      for (auto &Edge : *Node.second) {
+        FoundFoo |= Edge.second->getFunction()->getName() == "foo";
+      }
+    }
+    ASSERT_TRUE(FoundFoo);
+  }
+}
+
+} // namespace llvm
diff --git a/llvm/unittests/Analysis/ValueTrackingTest.cpp b/llvm/unittests/Analysis/ValueTrackingTest.cpp
index ef130522f10f4..4e6b1ef0c93bf 100644
--- a/llvm/unittests/Analysis/ValueTrackingTest.cpp
+++ b/llvm/unittests/Analysis/ValueTrackingTest.cpp
@@ -110,6 +110,18 @@ class ComputeKnownBitsTest : public ValueTrackingTest {
   }
 };
 
+class ComputeKnownFPClassTest : public ValueTrackingTest {
+protected:
+  void expectKnownFPClass(unsigned KnownTrue, std::optional<bool> SignBitKnown,
+                          Instruction *TestVal = nullptr) {
+    if (!TestVal)
+      TestVal = A;
+
+    KnownFPClass Known = computeKnownFPClass(TestVal, M->getDataLayout());
+    EXPECT_EQ(KnownTrue, Known.KnownFPClasses);
+    EXPECT_EQ(SignBitKnown, Known.SignBit);
+  }
+};
 }
 
 TEST_F(MatchSelectPatternTest, SimpleFMin) {
@@ -1157,10 +1169,7 @@ TEST(ValueTracking, canCreatePoisonOrUndef) {
       {{false, false}, "call noundef i32 @g(i32 %x)"},
       {{true, false}, "fcmp nnan oeq float %fx, %fy"},
       {{false, false}, "fcmp oeq float %fx, %fy"},
-      {{true, false},
-       "ashr <4 x i32> %vx, select (i1 icmp sgt (i32 ptrtoint (i32* @s to "
-       "i32), i32 1), <4 x i32> zeroinitializer, <4 x i32> <i32 0, i32 1, i32 "
-       "2, i32 3>)"},
+      {{true, false}, "ashr i32 %x, ptrtoint (i32* @s to i32)"},
       {{false, false},
        "call {i32, i1} @llvm.sadd.with.overflow.i32(i32 %x, i32 %y)"},
       {{false, false},
@@ -1261,6 +1270,276 @@ TEST_F(ComputeKnownBitsTest, ComputeKnownMulBits) {
   expectKnownBits(/*zero*/ 95u, /*one*/ 32u);
 }
 
+TEST_F(ComputeKnownFPClassTest, SelectPos0) {
+  parseAssembly(
+      "define float @test(i1 %cond) {\n"
+      "  %A = select i1 %cond, float 0.0, float 0.0"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(fcPosZero, false);
+}
+
+TEST_F(ComputeKnownFPClassTest, SelectNeg0) {
+  parseAssembly(
+      "define float @test(i1 %cond) {\n"
+      "  %A = select i1 %cond, float -0.0, float -0.0"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(fcNegZero, true);
+}
+
+TEST_F(ComputeKnownFPClassTest, SelectPosOrNeg0) {
+  parseAssembly(
+      "define float @test(i1 %cond) {\n"
+      "  %A = select i1 %cond, float 0.0, float -0.0"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(fcZero, std::nullopt);
+}
+
+TEST_F(ComputeKnownFPClassTest, SelectPosInf) {
+  parseAssembly(
+      "define float @test(i1 %cond) {\n"
+      "  %A = select i1 %cond, float 0x7FF0000000000000, float 0x7FF0000000000000"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(fcPosInf, false);
+}
+
+TEST_F(ComputeKnownFPClassTest, SelectNegInf) {
+  parseAssembly(
+      "define float @test(i1 %cond) {\n"
+      "  %A = select i1 %cond, float 0xFFF0000000000000, float 0xFFF0000000000000"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(fcNegInf, true);
+}
+
+TEST_F(ComputeKnownFPClassTest, SelectPosOrNegInf) {
+  parseAssembly(
+      "define float @test(i1 %cond) {\n"
+      "  %A = select i1 %cond, float 0x7FF0000000000000, float 0xFFF0000000000000"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(fcInf, std::nullopt);
+}
+
+TEST_F(ComputeKnownFPClassTest, SelectNNaN) {
+  parseAssembly(
+      "define float @test(i1 %cond, float %arg0, float %arg1) {\n"
+      "  %A = select nnan i1 %cond, float %arg0, float %arg1"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(~fcNan, std::nullopt);
+}
+
+TEST_F(ComputeKnownFPClassTest, SelectNInf) {
+  parseAssembly(
+      "define float @test(i1 %cond, float %arg0, float %arg1) {\n"
+      "  %A = select ninf i1 %cond, float %arg0, float %arg1"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(~fcInf, std::nullopt);
+}
+
+TEST_F(ComputeKnownFPClassTest, SelectNNaNNInf) {
+  parseAssembly(
+      "define float @test(i1 %cond, float %arg0, float %arg1) {\n"
+      "  %A = select nnan ninf i1 %cond, float %arg0, float %arg1"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(~(fcNan | fcInf), std::nullopt);
+}
+
+TEST_F(ComputeKnownFPClassTest, SelectNoFPClassArgUnionAll) {
+  parseAssembly(
+      "define float @test(i1 %cond, float nofpclass(snan ninf nsub pzero pnorm) %arg0, float nofpclass(qnan nnorm nzero psub pinf) %arg1) {\n"
+      "  %A = select i1 %cond, float %arg0, float %arg1"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(fcAllFlags, std::nullopt);
+}
+
+TEST_F(ComputeKnownFPClassTest, SelectNoFPClassArgNoNan) {
+  parseAssembly(
+      "define float @test(i1 %cond, float nofpclass(nan) %arg0, float nofpclass(nan) %arg1) {\n"
+      "  %A = select i1 %cond, float %arg0, float %arg1"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(~fcNan, std::nullopt);
+}
+
+TEST_F(ComputeKnownFPClassTest, SelectNoFPClassArgNoPInf) {
+  parseAssembly(
+      "define float @test(i1 %cond, float nofpclass(inf) %arg0, float nofpclass(pinf) %arg1) {\n"
+      "  %A = select i1 %cond, float %arg0, float %arg1"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(~fcPosInf, std::nullopt);
+}
+
+TEST_F(ComputeKnownFPClassTest, SelectNoFPClassArgNoNInf) {
+  parseAssembly(
+      "define float @test(i1 %cond, float nofpclass(ninf) %arg0, float nofpclass(inf) %arg1) {\n"
+      "  %A = select i1 %cond, float %arg0, float %arg1"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(~fcNegInf, std::nullopt);
+}
+
+TEST_F(ComputeKnownFPClassTest, SelectNoFPClassCallSiteNoNan) {
+  parseAssembly(
+      "declare float @func()\n"
+      "define float @test() {\n"
+      "  %A = call nofpclass(nan) float @func()\n"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(~fcNan, std::nullopt);
+}
+
+TEST_F(ComputeKnownFPClassTest, SelectNoFPClassCallSiteNoZeros) {
+  parseAssembly(
+      "declare float @func()\n"
+      "define float @test() {\n"
+      "  %A = call nofpclass(zero) float @func()\n"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(~fcZero, std::nullopt);
+}
+
+TEST_F(ComputeKnownFPClassTest, SelectNoFPClassDeclarationNoNan) {
+  parseAssembly(
+      "declare nofpclass(nan) float @no_nans()\n"
+      "define float @test() {\n"
+      "  %A = call float @no_nans()\n"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(~fcNan, std::nullopt);
+}
+
+// Check nofpclass + ninf works on a callsite
+TEST_F(ComputeKnownFPClassTest, SelectNoFPClassCallSiteNoZerosNInfFlags) {
+  parseAssembly(
+      "declare float @func()\n"
+      "define float @test() {\n"
+      "  %A = call ninf nofpclass(zero) float @func()\n"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(~(fcZero | fcInf), std::nullopt);
+}
+
+TEST_F(ComputeKnownFPClassTest, FNegNInf) {
+  parseAssembly(
+      "define float @test(float %arg) {\n"
+      "  %A = fneg ninf float %arg"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(~fcInf, std::nullopt);
+}
+
+TEST_F(ComputeKnownFPClassTest, FabsUnknown) {
+  parseAssembly(
+      "declare float @llvm.fabs.f32(float)"
+      "define float @test(float %arg) {\n"
+      "  %A = call float @llvm.fabs.f32(float %arg)"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(fcAllFlags, false);
+}
+
+TEST_F(ComputeKnownFPClassTest, FNegFabsUnknown) {
+  parseAssembly(
+      "declare float @llvm.fabs.f32(float)"
+      "define float @test(float %arg) {\n"
+      "  %fabs = call float @llvm.fabs.f32(float %arg)"
+      "  %A = fneg float %fabs"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(fcAllFlags, true);
+}
+
+TEST_F(ComputeKnownFPClassTest, NegFabsNInf) {
+  parseAssembly(
+      "declare float @llvm.fabs.f32(float)"
+      "define float @test(float %arg) {\n"
+      "  %fabs = call ninf float @llvm.fabs.f32(float %arg)"
+      "  %A = fneg float %fabs"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(~fcInf, true);
+}
+
+TEST_F(ComputeKnownFPClassTest, FNegFabsNNaN) {
+  parseAssembly(
+      "declare float @llvm.fabs.f32(float)"
+      "define float @test(float %arg) {\n"
+      "  %fabs = call nnan float @llvm.fabs.f32(float %arg)"
+      "  %A = fneg float %fabs"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(~fcNan, true);
+}
+
+TEST_F(ComputeKnownFPClassTest, CopySignNNanSrc0) {
+  parseAssembly(
+      "declare float @llvm.fabs.f32(float)\n"
+      "declare float @llvm.copysign.f32(float, float)\n"
+      "define float @test(float %arg0, float %arg1) {\n"
+      "  %fabs = call nnan float @llvm.fabs.f32(float %arg0)"
+      "  %A = call float @llvm.copysign.f32(float %fabs, float %arg1)"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(~fcNan, std::nullopt);
+}
+
+TEST_F(ComputeKnownFPClassTest, CopySignNInfSrc0_NegSign) {
+  parseAssembly(
+      "declare float @llvm.sqrt.f32(float)\n"
+      "declare float @llvm.copysign.f32(float, float)\n"
+      "define float @test(float %arg0, float %arg1) {\n"
+      "  %ninf = call ninf float @llvm.sqrt.f32(float %arg0)"
+      "  %A = call float @llvm.copysign.f32(float %ninf, float -1.0)"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(fcNegFinite, true);
+}
+
+TEST_F(ComputeKnownFPClassTest, CopySignNInfSrc0_PosSign) {
+  parseAssembly(
+      "declare float @llvm.sqrt.f32(float)\n"
+      "declare float @llvm.copysign.f32(float, float)\n"
+      "define float @test(float %arg0, float %arg1) {\n"
+      "  %ninf = call ninf float @llvm.sqrt.f32(float %arg0)"
+      "  %A = call float @llvm.copysign.f32(float %ninf, float 1.0)"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(fcPosFinite, false);
+}
+
+TEST_F(ComputeKnownFPClassTest, UIToFP) {
+  parseAssembly(
+      "define float @test(i32 %arg0, i16 %arg1) {\n"
+      "  %A = uitofp i32 %arg0 to float"
+      "  %A2 = uitofp i16 %arg1 to half"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(fcPosFinite, false, A);
+  expectKnownFPClass(fcPositive, false, A2);
+}
+
+TEST_F(ComputeKnownFPClassTest, SIToFP) {
+  parseAssembly(
+      "define float @test(i32 %arg0, i16 %arg1, i17 %arg2) {\n"
+      "  %A = sitofp i32 %arg0 to float"
+      "  %A2 = sitofp i16 %arg1 to half"
+      "  %A3 = sitofp i17 %arg2 to half"
+      "  ret float %A\n"
+      "}\n");
+  expectKnownFPClass(fcFinite, std::nullopt, A);
+  expectKnownFPClass(fcFinite, std::nullopt, A2);
+  expectKnownFPClass(~fcNan, std::nullopt, A3);
+}
+
 TEST_F(ValueTrackingTest, isNonZeroRecurrence) {
   parseAssembly(R"(
     define i1 @test(i8 %n, i8 %r) {
diff --git a/llvm/unittests/DebugInfo/CodeView/RandomAccessVisitorTest.cpp b/llvm/unittests/DebugInfo/CodeView/RandomAccessVisitorTest.cpp
index 135e7f854ecef..0686b47c69d52 100644
--- a/llvm/unittests/DebugInfo/CodeView/RandomAccessVisitorTest.cpp
+++ b/llvm/unittests/DebugInfo/CodeView/RandomAccessVisitorTest.cpp
@@ -222,7 +222,7 @@ TEST_F(RandomAccessVisitorTest, MultipleVisits) {
 
   // 5, 5, 5
   EXPECT_EQ(3u, TestState->Callbacks.count());
-  for (auto I : enumerate(IndicesToVisit))
+  for (const auto &I : enumerate(IndicesToVisit))
     EXPECT_TRUE(ValidateVisitedRecord(I.index(), I.value()));
 }
 
@@ -251,7 +251,7 @@ TEST_F(RandomAccessVisitorTest, DescendingWithinChunk) {
 
   // 2, 4, 7
   EXPECT_EQ(3u, TestState->Callbacks.count());
-  for (auto I : enumerate(IndicesToVisit))
+  for (const auto &I : enumerate(IndicesToVisit))
     EXPECT_TRUE(ValidateVisitedRecord(I.index(), I.value()));
 }
 
@@ -280,7 +280,7 @@ TEST_F(RandomAccessVisitorTest, AscendingWithinChunk) {
 
   // 2, 4, 7
   EXPECT_EQ(3u, TestState->Callbacks.count());
-  for (auto &I : enumerate(IndicesToVisit))
+  for (const auto &I : enumerate(IndicesToVisit))
     EXPECT_TRUE(ValidateVisitedRecord(I.index(), I.value()));
 }
 
@@ -311,7 +311,7 @@ TEST_F(RandomAccessVisitorTest, StopPrematurelyInChunk) {
 
   // [0, 2]
   EXPECT_EQ(3u, TestState->Callbacks.count());
-  for (auto I : enumerate(IndicesToVisit))
+  for (const auto &I : enumerate(IndicesToVisit))
     EXPECT_TRUE(ValidateVisitedRecord(I.index(), I.value()));
 }
 
@@ -341,7 +341,7 @@ TEST_F(RandomAccessVisitorTest, InnerChunk) {
 
   // 5, 7
   EXPECT_EQ(2u, TestState->Callbacks.count());
-  for (auto &I : enumerate(IndicesToVisit))
+  for (const auto &I : enumerate(IndicesToVisit))
     EXPECT_TRUE(ValidateVisitedRecord(I.index(), I.value()));
 }
 
diff --git a/llvm/unittests/DebugInfo/DWARF/DWARFFormValueTest.cpp b/llvm/unittests/DebugInfo/DWARF/DWARFFormValueTest.cpp
index a7f272249f583..c79daa6e29410 100644
--- a/llvm/unittests/DebugInfo/DWARF/DWARFFormValueTest.cpp
+++ b/llvm/unittests/DebugInfo/DWARF/DWARFFormValueTest.cpp
@@ -30,6 +30,10 @@ TEST(DWARFFormValue, FormClass) {
   EXPECT_FALSE(isFormClass(DW_FORM_data8, DWARFFormValue::FC_Address));
   EXPECT_TRUE(isFormClass(DW_FORM_data8, DWARFFormValue::FC_Constant));
   EXPECT_TRUE(isFormClass(DW_FORM_data8, DWARFFormValue::FC_SectionOffset));
+  EXPECT_TRUE(doesFormBelongToClass(DW_FORM_data8,
+                                    DWARFFormValue::FC_SectionOffset, 3));
+  EXPECT_FALSE(doesFormBelongToClass(DW_FORM_data8,
+                                     DWARFFormValue::FC_SectionOffset, 5));
   EXPECT_TRUE(
       isFormClass(DW_FORM_sec_offset, DWARFFormValue::FC_SectionOffset));
   EXPECT_TRUE(isFormClass(DW_FORM_GNU_str_index, DWARFFormValue::FC_String));
diff --git a/llvm/unittests/DebugInfo/GSYM/GSYMTest.cpp b/llvm/unittests/DebugInfo/GSYM/GSYMTest.cpp
index 366c6a5ff74db..7897f529b93ad 100644
--- a/llvm/unittests/DebugInfo/GSYM/GSYMTest.cpp
+++ b/llvm/unittests/DebugInfo/GSYM/GSYMTest.cpp
@@ -2443,3 +2443,211 @@ TEST(GSYMTest, TestGsymCreatorMultipleSymbolsWithNoSize) {
                    1, // NumAddresses
                    ArrayRef<uint8_t>(UUID));
 }
+
+// Helper function to quickly create a FunctionInfo in a GsymCreator for testing.
+static void AddFunctionInfo(GsymCreator &GC, const char *FuncName,
+                            uint64_t FuncAddr, const char *SourcePath,
+                            const char *HeaderPath) {
+  FunctionInfo FI(FuncAddr, 0x30, GC.insertString(FuncName));
+  FI.OptLineTable = LineTable();
+  const uint32_t SourceFileIdx = GC.insertFile(SourcePath);
+  const uint32_t HeaderFileIdx = GC.insertFile(HeaderPath);
+  FI.OptLineTable->push(LineEntry(FuncAddr+0x00, SourceFileIdx, 5));
+  FI.OptLineTable->push(LineEntry(FuncAddr+0x10, HeaderFileIdx, 10));
+  FI.OptLineTable->push(LineEntry(FuncAddr+0x12, HeaderFileIdx, 20));
+  FI.OptLineTable->push(LineEntry(FuncAddr+0x14, HeaderFileIdx, 11));
+  FI.OptLineTable->push(LineEntry(FuncAddr+0x16, HeaderFileIdx, 30));
+  FI.OptLineTable->push(LineEntry(FuncAddr+0x18, HeaderFileIdx, 12));
+  FI.OptLineTable->push(LineEntry(FuncAddr+0x20, SourceFileIdx, 8));
+  FI.Inline = InlineInfo();
+
+  std::string InlineName1(FuncName); InlineName1.append("1");
+  std::string InlineName2(FuncName); InlineName2.append("2");
+  std::string InlineName3(FuncName); InlineName3.append("3");
+
+  FI.Inline->Name = GC.insertString(InlineName1);
+  FI.Inline->CallFile = SourceFileIdx;
+  FI.Inline->CallLine = 6;
+  FI.Inline->Ranges.insert(AddressRange(FuncAddr + 0x10, FuncAddr + 0x20));
+  InlineInfo Inline2;
+  Inline2.Name = GC.insertString(InlineName2);
+  Inline2.CallFile = HeaderFileIdx;
+  Inline2.CallLine = 33;
+  Inline2.Ranges.insert(AddressRange(FuncAddr + 0x12, FuncAddr + 0x14));
+  FI.Inline->Children.emplace_back(Inline2);
+  InlineInfo Inline3;
+  Inline3.Name = GC.insertString(InlineName3);
+  Inline3.CallFile = HeaderFileIdx;
+  Inline3.CallLine = 35;
+  Inline3.Ranges.insert(AddressRange(FuncAddr + 0x16, FuncAddr + 0x18));
+  FI.Inline->Children.emplace_back(Inline3);
+  GC.addFunctionInfo(std::move(FI));
+}
+
+// Finalize a GsymCreator, encode it and decode it and return the error or
+// GsymReader that was successfully decoded.
+static Expected<GsymReader> FinalizeEncodeAndDecode(GsymCreator &GC) {
+  Error FinalizeErr = GC.finalize(llvm::nulls());
+  if (FinalizeErr)
+    return std::move(FinalizeErr);
+  SmallString<1024> Str;
+  raw_svector_ostream OutStrm(Str);
+  const auto ByteOrder = support::endian::system_endianness();
+  FileWriter FW(OutStrm, ByteOrder);
+  llvm::Error Err = GC.encode(FW);
+  if (Err)
+    return std::move(Err);
+  return GsymReader::copyBuffer(OutStrm.str());
+}
+
+TEST(GSYMTest, TestGsymSegmenting) {
+  // Test creating a GSYM file with function infos and segment the information.
+  // We verify segmenting is working by creating a full GSYM and also by
+  // encoding multiple segments, then we verify that we get the same information
+  // when doing lookups on the full GSYM that was decoded from encoding the
+  // entire GSYM and also by decoding information from the segments themselves.
+  GsymCreator GC;
+  GC.setBaseAddress(0);
+  AddFunctionInfo(GC, "main", 0x1000, "/tmp/main.c", "/tmp/main.h");
+  AddFunctionInfo(GC, "foo", 0x2000, "/tmp/foo.c", "/tmp/foo.h");
+  AddFunctionInfo(GC, "bar", 0x3000, "/tmp/bar.c", "/tmp/bar.h");
+  AddFunctionInfo(GC, "baz", 0x4000, "/tmp/baz.c", "/tmp/baz.h");
+  Expected<GsymReader> GR = FinalizeEncodeAndDecode(GC);
+  ASSERT_THAT_EXPECTED(GR, Succeeded());
+  //GR->dump(outs());
+
+  // Create segmented GSYM files where each file contains 1 function. We will
+  // then test doing lookups on the "GR", or the full GSYM file and then test
+  // doing lookups on the GsymReader objects for each segment to ensure we get
+  // the exact same information. So after all of the code below we will have
+  // GsymReader objects that each contain one function. We name the creators
+  // and readers to match the one and only address they contain.
+  // GC1000 and GR1000 are for [0x1000-0x1030)
+  // GC2000 and GR2000 are for [0x2000-0x2030)
+  // GC3000 and GR3000 are for [0x3000-0x3030)
+  // GC4000 and GR4000 are for [0x4000-0x4030)
+
+  // Create the segments and verify that FuncIdx, an in/out parameter, gets
+  // updated as expected.
+  size_t FuncIdx = 0;
+  // Make sure we get an error if the segment size is too small to encode a
+  // single function info.
+  llvm::Expected<std::unique_ptr<GsymCreator>> GCError =
+      GC.createSegment(57, FuncIdx);
+  ASSERT_FALSE((bool)GCError);
+  checkError("a segment size of 57 is to small to fit any function infos, "
+             "specify a larger value", GCError.takeError());
+  // Make sure that the function index didn't get incremented when we didn't
+  // encode any values into the segmented GsymCreator.
+  ASSERT_EQ(FuncIdx, (size_t)0);
+
+  llvm::Expected<std::unique_ptr<GsymCreator>> GC1000 =
+      GC.createSegment(128, FuncIdx);
+  ASSERT_THAT_EXPECTED(GC1000, Succeeded());
+  ASSERT_EQ(FuncIdx, (size_t)1);
+  llvm::Expected<std::unique_ptr<GsymCreator>> GC2000 =
+      GC.createSegment(128, FuncIdx);
+  ASSERT_THAT_EXPECTED(GC2000, Succeeded());
+  ASSERT_EQ(FuncIdx, (size_t)2);
+  llvm::Expected<std::unique_ptr<GsymCreator>> GC3000 =
+      GC.createSegment(128, FuncIdx);
+  ASSERT_THAT_EXPECTED(GC3000, Succeeded());
+  ASSERT_EQ(FuncIdx, (size_t)3);
+  llvm::Expected<std::unique_ptr<GsymCreator>> GC4000 =
+      GC.createSegment(128, FuncIdx);
+  ASSERT_THAT_EXPECTED(GC4000, Succeeded());
+  ASSERT_EQ(FuncIdx, (size_t)4);
+  // When there are no function infos left to encode we expect to get  no error
+  // and get a NULL GsymCreator in the return value from createSegment.
+  llvm::Expected<std::unique_ptr<GsymCreator>> GCNull =
+      GC.createSegment(128, FuncIdx);
+  ASSERT_THAT_EXPECTED(GCNull, Succeeded());
+  ASSERT_TRUE(GC1000.get() != nullptr);
+  ASSERT_TRUE(GC2000.get() != nullptr);
+  ASSERT_TRUE(GC3000.get() != nullptr);
+  ASSERT_TRUE(GC4000.get() != nullptr);
+  ASSERT_TRUE(GCNull.get() == nullptr);
+  // Encode and decode the GsymReader for each segment and verify they succeed.
+  Expected<GsymReader> GR1000 = FinalizeEncodeAndDecode(*GC1000.get());
+  ASSERT_THAT_EXPECTED(GR1000, Succeeded());
+  Expected<GsymReader> GR2000 = FinalizeEncodeAndDecode(*GC2000.get());
+  ASSERT_THAT_EXPECTED(GR2000, Succeeded());
+  Expected<GsymReader> GR3000 = FinalizeEncodeAndDecode(*GC3000.get());
+  ASSERT_THAT_EXPECTED(GR3000, Succeeded());
+  Expected<GsymReader> GR4000 = FinalizeEncodeAndDecode(*GC4000.get());
+  ASSERT_THAT_EXPECTED(GR4000, Succeeded());
+
+  // Verify that all lookups match the range [0x1000-0x1030) when doing lookups
+  // in the GsymReader that contains all functions and from the segmented
+  // GsymReader in GR1000.
+  for (uint64_t Addr = 0x1000; Addr < 0x1030; ++Addr) {
+    // Lookup in the main GsymReader that contains all function infos
+    auto MainLR = GR->lookup(Addr);
+    ASSERT_THAT_EXPECTED(MainLR, Succeeded());
+    auto SegmentLR = GR1000->lookup(Addr);
+    ASSERT_THAT_EXPECTED(SegmentLR, Succeeded());
+    // Make sure the lookup results match.
+    EXPECT_EQ(MainLR.get(), SegmentLR.get());
+    // Make sure that the lookups on the functions that are not in the segment
+    // fail as expected.
+    ASSERT_THAT_EXPECTED(GR1000->lookup(0x2000), Failed());
+    ASSERT_THAT_EXPECTED(GR1000->lookup(0x3000), Failed());
+    ASSERT_THAT_EXPECTED(GR1000->lookup(0x4000), Failed());
+  }
+
+  // Verify that all lookups match the range [0x2000-0x2030) when doing lookups
+  // in the GsymReader that contains all functions and from the segmented
+  // GsymReader in GR2000.
+  for (uint64_t Addr = 0x2000; Addr < 0x2030; ++Addr) {
+    // Lookup in the main GsymReader that contains all function infos
+    auto MainLR = GR->lookup(Addr);
+    ASSERT_THAT_EXPECTED(MainLR, Succeeded());
+    auto SegmentLR = GR2000->lookup(Addr);
+    ASSERT_THAT_EXPECTED(SegmentLR, Succeeded());
+    // Make sure the lookup results match.
+    EXPECT_EQ(MainLR.get(), SegmentLR.get());
+    // Make sure that the lookups on the functions that are not in the segment
+    // fail as expected.
+    ASSERT_THAT_EXPECTED(GR2000->lookup(0x1000), Failed());
+    ASSERT_THAT_EXPECTED(GR2000->lookup(0x3000), Failed());
+    ASSERT_THAT_EXPECTED(GR2000->lookup(0x4000), Failed());
+
+  }
+
+  // Verify that all lookups match the range [0x3000-0x3030) when doing lookups
+  // in the GsymReader that contains all functions and from the segmented
+  // GsymReader in GR3000.
+  for (uint64_t Addr = 0x3000; Addr < 0x3030; ++Addr) {
+    // Lookup in the main GsymReader that contains all function infos
+    auto MainLR = GR->lookup(Addr);
+    ASSERT_THAT_EXPECTED(MainLR, Succeeded());
+    auto SegmentLR = GR3000->lookup(Addr);
+    ASSERT_THAT_EXPECTED(SegmentLR, Succeeded());
+    // Make sure the lookup results match.
+    EXPECT_EQ(MainLR.get(), SegmentLR.get());
+    // Make sure that the lookups on the functions that are not in the segment
+    // fail as expected.
+    ASSERT_THAT_EXPECTED(GR3000->lookup(0x1000), Failed());
+    ASSERT_THAT_EXPECTED(GR3000->lookup(0x2000), Failed());
+    ASSERT_THAT_EXPECTED(GR3000->lookup(0x4000), Failed());
+}
+
+  // Verify that all lookups match the range [0x4000-0x4030) when doing lookups
+  // in the GsymReader that contains all functions and from the segmented
+  // GsymReader in GR4000.
+  for (uint64_t Addr = 0x4000; Addr < 0x4030; ++Addr) {
+    // Lookup in the main GsymReader that contains all function infos
+    auto MainLR = GR->lookup(Addr);
+    ASSERT_THAT_EXPECTED(MainLR, Succeeded());
+    // Lookup in the GsymReader for that contains 0x4000
+    auto SegmentLR = GR4000->lookup(Addr);
+    ASSERT_THAT_EXPECTED(SegmentLR, Succeeded());
+    // Make sure the lookup results match.
+    EXPECT_EQ(MainLR.get(), SegmentLR.get());
+    // Make sure that the lookups on the functions that are not in the segment
+    // fail as expected.
+    ASSERT_THAT_EXPECTED(GR4000->lookup(0x1000), Failed());
+    ASSERT_THAT_EXPECTED(GR4000->lookup(0x2000), Failed());
+    ASSERT_THAT_EXPECTED(GR4000->lookup(0x3000), Failed());
+  }
+}
diff --git a/llvm/unittests/ExecutionEngine/JITLink/LinkGraphTests.cpp b/llvm/unittests/ExecutionEngine/JITLink/LinkGraphTests.cpp
index 80f557122ec20..0146c3b4cf6e0 100644
--- a/llvm/unittests/ExecutionEngine/JITLink/LinkGraphTests.cpp
+++ b/llvm/unittests/ExecutionEngine/JITLink/LinkGraphTests.cpp
@@ -756,3 +756,31 @@ TEST(LinkGraphTest, IsCStringBlockTest) {
   EXPECT_TRUE(isCStringBlock(SizeOneZeroFillBlock));
   EXPECT_FALSE(isCStringBlock(LargerZeroFillBlock));
 }
+
+TEST(LinkGraphTest, BasicLayoutHonorsNoAlloc) {
+  // Check that BasicLayout honors NoAlloc.
+  LinkGraph G("foo", Triple("x86_64-apple-darwin"), 8, support::little,
+              getGenericEdgeKindName);
+
+  // Create a regular section and block.
+  auto &Sec1 =
+      G.createSection("__data", orc::MemProt::Read | orc::MemProt::Write);
+  G.createContentBlock(Sec1, BlockContent.slice(0, 8), orc::ExecutorAddr(), 8,
+                       0);
+
+  // Create a NoAlloc section and block.
+  auto &Sec2 =
+      G.createSection("__metadata", orc::MemProt::Read | orc::MemProt::Write);
+  Sec2.setMemLifetimePolicy(orc::MemLifetimePolicy::NoAlloc);
+  G.createContentBlock(Sec2, BlockContent.slice(0, 8), orc::ExecutorAddr(), 8,
+                       0);
+
+  BasicLayout BL(G);
+
+  EXPECT_EQ(std::distance(BL.segments().begin(), BL.segments().end()), 1U);
+  EXPECT_EQ(BL.segments().begin()->first,
+            orc::MemProt::Read | orc::MemProt::Write);
+  auto &SegInfo = BL.segments().begin()->second;
+  EXPECT_EQ(SegInfo.Alignment, 8U);
+  EXPECT_EQ(SegInfo.ContentSize, 8U);
+}
diff --git a/llvm/unittests/ExecutionEngine/MCJIT/MCJITObjectCacheTest.cpp b/llvm/unittests/ExecutionEngine/MCJIT/MCJITObjectCacheTest.cpp
index b19fe55ffde8d..896315638634f 100644
--- a/llvm/unittests/ExecutionEngine/MCJIT/MCJITObjectCacheTest.cpp
+++ b/llvm/unittests/ExecutionEngine/MCJIT/MCJITObjectCacheTest.cpp
@@ -27,7 +27,7 @@ class TestObjectCache : public ObjectCache {
   void notifyObjectCompiled(const Module *M, MemoryBufferRef Obj) override {
     // If we've seen this module before, note that.
     const std::string ModuleID = M->getModuleIdentifier();
-    if (ObjMap.find(ModuleID) != ObjMap.end())
+    if (ObjMap.contains(ModuleID))
       DuplicateInserted = true;
     // Store a copy of the buffer in our map.
     ObjMap[ModuleID] = copyBuffer(Obj);
@@ -47,8 +47,7 @@ class TestObjectCache : public ObjectCache {
   bool wereDuplicatesInserted() { return DuplicateInserted; }
 
   bool wasModuleLookedUp(const Module *M) {
-    return ModulesLookedUp.find(M->getModuleIdentifier())
-                                      != ModulesLookedUp.end();
+    return ModulesLookedUp.contains(M->getModuleIdentifier());
   }
 
   const MemoryBuffer* getObjectInternal(const Module* M) {
diff --git a/llvm/unittests/ExecutionEngine/Orc/EPCGenericJITLinkMemoryManagerTest.cpp b/llvm/unittests/ExecutionEngine/Orc/EPCGenericJITLinkMemoryManagerTest.cpp
index 7a34dfc9e2304..d3a66294bd571 100644
--- a/llvm/unittests/ExecutionEngine/Orc/EPCGenericJITLinkMemoryManagerTest.cpp
+++ b/llvm/unittests/ExecutionEngine/Orc/EPCGenericJITLinkMemoryManagerTest.cpp
@@ -46,9 +46,9 @@ class SimpleAllocator {
       assert(Seg.Size <= std::numeric_limits<size_t>::max());
       if (auto EC = sys::Memory::protectMappedMemory(
               {Mem, static_cast<size_t>(Seg.Size)},
-              toSysMemoryProtectionFlags(Seg.AG.getMemProt())))
+              toSysMemoryProtectionFlags(Seg.RAG.Prot)))
         return errorCodeToError(EC);
-      if ((Seg.AG.getMemProt() & MemProt::Exec) != MemProt::Exec)
+      if ((Seg.RAG.Prot & MemProt::Exec) != MemProt::Exec)
         sys::Memory::InvalidateInstructionCache(Mem, Seg.Size);
     }
     return Error::success();
diff --git a/llvm/unittests/IR/ConstantsTest.cpp b/llvm/unittests/IR/ConstantsTest.cpp
index 71ff311e7693f..577d9e9e697a8 100644
--- a/llvm/unittests/IR/ConstantsTest.cpp
+++ b/llvm/unittests/IR/ConstantsTest.cpp
@@ -207,7 +207,6 @@ TEST(ConstantsTest, AsInstructionsTest) {
   Type *Int64Ty = Type::getInt64Ty(Context);
   Type *Int32Ty = Type::getInt32Ty(Context);
   Type *Int16Ty = Type::getInt16Ty(Context);
-  Type *Int1Ty = Type::getInt1Ty(Context);
   Type *FloatTy = Type::getFloatTy(Context);
   Type *DoubleTy = Type::getDoubleTy(Context);
 
@@ -219,7 +218,6 @@ TEST(ConstantsTest, AsInstructionsTest) {
   Constant *P0 = ConstantExpr::getPtrToInt(Global, Int32Ty);
   Constant *P1 = ConstantExpr::getUIToFP(P0, FloatTy);
   Constant *P2 = ConstantExpr::getUIToFP(P0, DoubleTy);
-  Constant *P3 = ConstantExpr::getTrunc(P0, Int1Ty);
   Constant *P4 = ConstantExpr::getPtrToInt(Global2, Int32Ty);
   Constant *P5 = ConstantExpr::getUIToFP(P4, FloatTy);
   Constant *P6 = ConstantExpr::getBitCast(P4, FixedVectorType::get(Int16Ty, 2));
@@ -270,8 +268,6 @@ TEST(ConstantsTest, AsInstructionsTest) {
   CHECK(ConstantExpr::getFPExtend(P1, DoubleTy),
         "fpext float " P1STR " to double");
 
-  CHECK(ConstantExpr::getSelect(P3, P0, P4),
-        "select i1 " P3STR ", i32 " P0STR ", i32 " P4STR);
   CHECK(ConstantExpr::getICmp(CmpInst::ICMP_EQ, P0, P4),
         "icmp eq i32 " P0STR ", " P4STR);
   CHECK(ConstantExpr::getFCmp(CmpInst::FCMP_ULT, P1, P5),
diff --git a/llvm/unittests/IR/MetadataTest.cpp b/llvm/unittests/IR/MetadataTest.cpp
index 85d394a9fbb85..04d9443ef1857 100644
--- a/llvm/unittests/IR/MetadataTest.cpp
+++ b/llvm/unittests/IR/MetadataTest.cpp
@@ -1089,6 +1089,159 @@ TEST_F(DILocationTest, Merge) {
     EXPECT_EQ(SPI, M->getScope());
     EXPECT_EQ(nullptr, M->getInlinedAt());
   }
+
+  // Merge a location in C, which is inlined-at in B that is inlined in A,
+  // with a location in A that has the same scope, line and column as B's
+  // inlined-at location.
+  {
+    auto *FA = getFile();
+    auto *FB = getFile();
+    auto *FC = getFile();
+
+    auto *SPA = DISubprogram::getDistinct(Context, FA, "a", "a", FA, 0, nullptr,
+                                          0, nullptr, 0, 0, DINode::FlagZero,
+                                          DISubprogram::SPFlagZero, nullptr);
+
+    auto *SPB = DISubprogram::getDistinct(Context, FB, "b", "b", FB, 0, nullptr,
+                                          0, nullptr, 0, 0, DINode::FlagZero,
+                                          DISubprogram::SPFlagZero, nullptr);
+
+    auto *SPC = DISubprogram::getDistinct(Context, FC, "c", "c", FC, 0, nullptr,
+                                          0, nullptr, 0, 0, DINode::FlagZero,
+                                          DISubprogram::SPFlagZero, nullptr);
+
+    auto *A = DILocation::get(Context, 3, 2, SPA);
+    auto *B = DILocation::get(Context, 2, 4, SPB, A);
+    auto *C = DILocation::get(Context, 13, 2, SPC, B);
+    auto *M = DILocation::getMergedLocation(A, C);
+    EXPECT_EQ(3u, M->getLine());
+    EXPECT_EQ(2u, M->getColumn());
+    EXPECT_TRUE(isa<DILocalScope>(M->getScope()));
+    EXPECT_EQ(SPA, M->getScope());
+    EXPECT_EQ(nullptr, M->getInlinedAt());
+  }
+
+  // Two inlined locations with the same scope, line and column
+  // in the same inlined-at function at different line and column.
+  {
+    auto *FA = getFile();
+    auto *FB = getFile();
+    auto *FC = getFile();
+
+    auto *SPA = DISubprogram::getDistinct(Context, FA, "a", "a", FA, 0, nullptr,
+                                          0, nullptr, 0, 0, DINode::FlagZero,
+                                          DISubprogram::SPFlagZero, nullptr);
+
+    auto *SPB = DISubprogram::getDistinct(Context, FB, "b", "b", FB, 0, nullptr,
+                                          0, nullptr, 0, 0, DINode::FlagZero,
+                                          DISubprogram::SPFlagZero, nullptr);
+
+    auto *SPC = DISubprogram::getDistinct(Context, FC, "c", "c", FC, 0, nullptr,
+                                          0, nullptr, 0, 0, DINode::FlagZero,
+                                          DISubprogram::SPFlagZero, nullptr);
+
+    auto *A = DILocation::get(Context, 10, 20, SPA);
+    auto *B1 = DILocation::get(Context, 3, 2, SPB, A);
+    auto *B2 = DILocation::get(Context, 4, 5, SPB, A);
+    auto *C1 = DILocation::get(Context, 2, 4, SPC, B1);
+    auto *C2 = DILocation::get(Context, 2, 4, SPC, B2);
+
+    auto *M = DILocation::getMergedLocation(C1, C2);
+    EXPECT_EQ(2u, M->getLine());
+    EXPECT_EQ(4u, M->getColumn());
+    EXPECT_EQ(SPC, M->getScope());
+    ASSERT_NE(nullptr, M->getInlinedAt());
+
+    auto *I1 = M->getInlinedAt();
+    EXPECT_EQ(0u, I1->getLine());
+    EXPECT_EQ(0u, I1->getColumn());
+    EXPECT_EQ(SPB, I1->getScope());
+    EXPECT_EQ(A, I1->getInlinedAt());
+  }
+
+  // Two locations, different line/column and scope in the same subprogram,
+  // inlined at the same place. This should result in a 0:0 location with
+  // the nearest common scope in the inlined function.
+  {
+    auto *FA = getFile();
+    auto *FI = getFile();
+
+    auto *SPA = DISubprogram::getDistinct(Context, FA, "a", "a", FA, 0, nullptr,
+                                          0, nullptr, 0, 0, DINode::FlagZero,
+                                          DISubprogram::SPFlagZero, nullptr);
+
+    auto *SPI = DISubprogram::getDistinct(Context, FI, "i", "i", FI, 0, nullptr,
+                                          0, nullptr, 0, 0, DINode::FlagZero,
+                                          DISubprogram::SPFlagZero, nullptr);
+
+    // Nearest common scope for the two locations in a.
+    auto *SPAScope1 = DILexicalBlock::getDistinct(Context, SPA, FA, 4, 9);
+
+    // Scope for the first location in a.
+    auto *SPAScope2 =
+        DILexicalBlock::getDistinct(Context, SPAScope1, FA, 10, 12);
+
+    // Scope for the second location in a.
+    auto *SPAScope3 =
+        DILexicalBlock::getDistinct(Context, SPAScope1, FA, 20, 8);
+    auto *SPAScope4 =
+        DILexicalBlock::getDistinct(Context, SPAScope3, FA, 21, 12);
+
+    auto *I = DILocation::get(Context, 3, 8, SPI);
+    auto *A1 = DILocation::get(Context, 12, 7, SPAScope2, I);
+    auto *A2 = DILocation::get(Context, 21, 15, SPAScope4, I);
+    auto *M = DILocation::getMergedLocation(A1, A2);
+    EXPECT_EQ(0u, M->getLine());
+    EXPECT_EQ(0u, M->getColumn());
+    EXPECT_TRUE(isa<DILocalScope>(M->getScope()));
+    EXPECT_EQ(SPAScope1, M->getScope());
+    EXPECT_EQ(I, M->getInlinedAt());
+  }
+
+  // Regression test to catch a case where an iterator was invalidated due to
+  // handling the chain of inlined-at locations after the nearest common
+  // location for the two arguments were found.
+  {
+    auto *FA = getFile();
+    auto *FB = getFile();
+    auto *FI = getFile();
+
+    auto *SPA = DISubprogram::getDistinct(Context, FA, "a", "a", FA, 0, nullptr,
+                                          0, nullptr, 0, 0, DINode::FlagZero,
+                                          DISubprogram::SPFlagZero, nullptr);
+
+    auto *SPB = DISubprogram::getDistinct(Context, FB, "b", "b", FB, 0, nullptr,
+                                          0, nullptr, 0, 0, DINode::FlagZero,
+                                          DISubprogram::SPFlagZero, nullptr);
+
+    auto *SPI = DISubprogram::getDistinct(Context, FI, "i", "i", FI, 0, nullptr,
+                                          0, nullptr, 0, 0, DINode::FlagZero,
+                                          DISubprogram::SPFlagZero, nullptr);
+
+    auto *SPAScope1 = DILexicalBlock::getDistinct(Context, SPA, FA, 4, 9);
+    auto *SPAScope2 = DILexicalBlock::getDistinct(Context, SPA, FA, 8, 3);
+
+    DILocation *InlinedAt = nullptr;
+
+    // Create a chain of inlined-at locations.
+    for (int i = 0; i < 256; i++) {
+      InlinedAt = DILocation::get(Context, 3 + i, 8 + i, SPI, InlinedAt);
+    }
+
+    auto *A1 = DILocation::get(Context, 5, 9, SPAScope1, InlinedAt);
+    auto *A2 = DILocation::get(Context, 9, 8, SPAScope2, InlinedAt);
+    auto *B = DILocation::get(Context, 10, 3, SPB, A1);
+    auto *M1 = DILocation::getMergedLocation(B, A2);
+    EXPECT_EQ(0u, M1->getLine());
+    EXPECT_EQ(0u, M1->getColumn());
+    EXPECT_TRUE(isa<DILocalScope>(M1->getScope()));
+    EXPECT_EQ(SPA, M1->getScope());
+    EXPECT_EQ(InlinedAt, M1->getInlinedAt());
+
+    // Test the other argument order for good measure.
+    auto *M2 = DILocation::getMergedLocation(A2, B);
+    EXPECT_EQ(M1, M2);
+  }
 }
 
 TEST_F(DILocationTest, getDistinct) {
diff --git a/llvm/unittests/IR/PassManagerTest.cpp b/llvm/unittests/IR/PassManagerTest.cpp
index bae5d46ecd11c..a6487169224c2 100644
--- a/llvm/unittests/IR/PassManagerTest.cpp
+++ b/llvm/unittests/IR/PassManagerTest.cpp
@@ -824,10 +824,13 @@ TEST_F(PassManagerTest, FunctionPassCFGChecker) {
 
   auto *F = M->getFunction("foo");
   FunctionAnalysisManager FAM;
+  ModuleAnalysisManager MAM;
   FunctionPassManager FPM;
   PassInstrumentationCallbacks PIC;
   StandardInstrumentations SI(M->getContext(), /*DebugLogging*/ true);
-  SI.registerCallbacks(PIC, &FAM);
+  SI.registerCallbacks(PIC, &MAM);
+  MAM.registerPass([&] { return PassInstrumentationAnalysis(&PIC); });
+  MAM.registerPass([&] { return FunctionAnalysisManagerModuleProxy(FAM); });
   FAM.registerPass([&] { return PassInstrumentationAnalysis(&PIC); });
   FAM.registerPass([&] { return DominatorTreeAnalysis(); });
   FAM.registerPass([&] { return AssumptionAnalysis(); });
@@ -870,10 +873,13 @@ TEST_F(PassManagerTest, FunctionPassCFGCheckerInvalidateAnalysis) {
 
   auto *F = M->getFunction("foo");
   FunctionAnalysisManager FAM;
+  ModuleAnalysisManager MAM;
   FunctionPassManager FPM;
   PassInstrumentationCallbacks PIC;
   StandardInstrumentations SI(M->getContext(), /*DebugLogging*/ true);
-  SI.registerCallbacks(PIC, &FAM);
+  SI.registerCallbacks(PIC, &MAM);
+  MAM.registerPass([&] { return FunctionAnalysisManagerModuleProxy(FAM); });
+  MAM.registerPass([&] { return PassInstrumentationAnalysis(&PIC); });
   FAM.registerPass([&] { return PassInstrumentationAnalysis(&PIC); });
   FAM.registerPass([&] { return DominatorTreeAnalysis(); });
   FAM.registerPass([&] { return AssumptionAnalysis(); });
@@ -935,10 +941,13 @@ TEST_F(PassManagerTest, FunctionPassCFGCheckerWrapped) {
 
   auto *F = M->getFunction("foo");
   FunctionAnalysisManager FAM;
+  ModuleAnalysisManager MAM;
   FunctionPassManager FPM;
   PassInstrumentationCallbacks PIC;
   StandardInstrumentations SI(M->getContext(), /*DebugLogging*/ true);
-  SI.registerCallbacks(PIC, &FAM);
+  SI.registerCallbacks(PIC, &MAM);
+  MAM.registerPass([&] { return FunctionAnalysisManagerModuleProxy(FAM); });
+  MAM.registerPass([&] { return PassInstrumentationAnalysis(&PIC); });
   FAM.registerPass([&] { return PassInstrumentationAnalysis(&PIC); });
   FAM.registerPass([&] { return DominatorTreeAnalysis(); });
   FAM.registerPass([&] { return AssumptionAnalysis(); });
@@ -950,4 +959,116 @@ TEST_F(PassManagerTest, FunctionPassCFGCheckerWrapped) {
   FPM.addPass(TestSimplifyCFGWrapperPass(InnerFPM));
   FPM.run(*F, FAM);
 }
+
+#ifdef EXPENSIVE_CHECKS
+
+struct WrongFunctionPass : PassInfoMixin<WrongFunctionPass> {
+  PreservedAnalyses run(Function &F, FunctionAnalysisManager &FAM) {
+    F.getEntryBlock().begin()->eraseFromParent();
+    return PreservedAnalyses::all();
+  }
+  static StringRef name() { return "WrongFunctionPass"; }
+};
+
+TEST_F(PassManagerTest, FunctionPassMissedFunctionAnalysisInvalidation) {
+  LLVMContext Context;
+  auto M = parseIR(Context, "define void @foo() {\n"
+                            "  %a = add i32 0, 0\n"
+                            "  ret void\n"
+                            "}\n");
+
+  FunctionAnalysisManager FAM;
+  ModuleAnalysisManager MAM;
+  PassInstrumentationCallbacks PIC;
+  StandardInstrumentations SI(M->getContext(), /*DebugLogging*/ false);
+  SI.registerCallbacks(PIC, &MAM);
+  MAM.registerPass([&] { return FunctionAnalysisManagerModuleProxy(FAM); });
+  MAM.registerPass([&] { return PassInstrumentationAnalysis(&PIC); });
+  FAM.registerPass([&] { return PassInstrumentationAnalysis(&PIC); });
+  FAM.registerPass([&] { return ModuleAnalysisManagerFunctionProxy(MAM); });
+
+  FunctionPassManager FPM;
+  FPM.addPass(WrongFunctionPass());
+
+  auto *F = M->getFunction("foo");
+  EXPECT_DEATH(FPM.run(*F, FAM), "Function @foo changed by WrongFunctionPass without invalidating analyses");
+}
+
+struct WrongModulePass : PassInfoMixin<WrongModulePass> {
+  PreservedAnalyses run(Module &M, ModuleAnalysisManager &MAM) {
+    for (Function &F : M)
+      F.getEntryBlock().begin()->eraseFromParent();
+
+    PreservedAnalyses PA;
+    PA.preserveSet<AllAnalysesOn<Function>>();
+    PA.preserve<FunctionAnalysisManagerModuleProxy>();
+    return PA;
+  }
+  static StringRef name() { return "WrongModulePass"; }
+};
+
+TEST_F(PassManagerTest, ModulePassMissedFunctionAnalysisInvalidation) {
+  LLVMContext Context;
+  auto M = parseIR(Context, "define void @foo() {\n"
+                            "  %a = add i32 0, 0\n"
+                            "  ret void\n"
+                            "}\n");
+
+  FunctionAnalysisManager FAM;
+  ModuleAnalysisManager MAM;
+  PassInstrumentationCallbacks PIC;
+  StandardInstrumentations SI(M->getContext(), /*DebugLogging*/ false);
+  SI.registerCallbacks(PIC, &MAM);
+  MAM.registerPass([&] { return FunctionAnalysisManagerModuleProxy(FAM); });
+  MAM.registerPass([&] { return PassInstrumentationAnalysis(&PIC); });
+  FAM.registerPass([&] { return PassInstrumentationAnalysis(&PIC); });
+  FAM.registerPass([&] { return ModuleAnalysisManagerFunctionProxy(MAM); });
+
+  ModulePassManager MPM;
+  MPM.addPass(WrongModulePass());
+
+  EXPECT_DEATH(
+      MPM.run(*M, MAM),
+      "Function @foo changed by WrongModulePass without invalidating analyses");
+}
+
+struct WrongModulePass2 : PassInfoMixin<WrongModulePass2> {
+  PreservedAnalyses run(Module &M, ModuleAnalysisManager &MAM) {
+    for (Function &F : M)
+      F.getEntryBlock().begin()->eraseFromParent();
+
+    PreservedAnalyses PA;
+    PA.preserveSet<AllAnalysesOn<Module>>();
+    PA.abandon<FunctionAnalysisManagerModuleProxy>();
+    return PA;
+  }
+  static StringRef name() { return "WrongModulePass2"; }
+};
+
+TEST_F(PassManagerTest, ModulePassMissedModuleAnalysisInvalidation) {
+  LLVMContext Context;
+  auto M = parseIR(Context, "define void @foo() {\n"
+                            "  %a = add i32 0, 0\n"
+                            "  ret void\n"
+                            "}\n");
+
+  FunctionAnalysisManager FAM;
+  ModuleAnalysisManager MAM;
+  PassInstrumentationCallbacks PIC;
+  StandardInstrumentations SI(M->getContext(), /*DebugLogging*/ false);
+  SI.registerCallbacks(PIC, &MAM);
+  MAM.registerPass([&] { return FunctionAnalysisManagerModuleProxy(FAM); });
+  MAM.registerPass([&] { return PassInstrumentationAnalysis(&PIC); });
+  FAM.registerPass([&] { return PassInstrumentationAnalysis(&PIC); });
+  FAM.registerPass([&] { return ModuleAnalysisManagerFunctionProxy(MAM); });
+
+  ModulePassManager MPM;
+  MPM.addPass(WrongModulePass2());
+
+  EXPECT_DEATH(
+      MPM.run(*M, MAM),
+      "Module changed by WrongModulePass2 without invalidating analyses");
+}
+
+#endif
 }
diff --git a/llvm/unittests/IR/ShuffleVectorInstTest.cpp b/llvm/unittests/IR/ShuffleVectorInstTest.cpp
index cf4f91cd2a84d..ba1b9807cffb4 100644
--- a/llvm/unittests/IR/ShuffleVectorInstTest.cpp
+++ b/llvm/unittests/IR/ShuffleVectorInstTest.cpp
@@ -116,4 +116,33 @@ TEST(ShuffleVectorInst, isOneUseSingleSourceMask) {
       ShuffleVectorInst::isOneUseSingleSourceMask({0, 1, 2, 3, 3, 3, 1, 0}, 4));
 }
 
+TEST(ShuffleVectorInst, isInterleaveMask) {
+  SmallVector<unsigned> StartIndexes;
+  ASSERT_TRUE(ShuffleVectorInst::isInterleaveMask({0, 4, 1, 5, 2, 6, 3, 7}, 2,
+                                                  8, StartIndexes));
+  ASSERT_EQ(StartIndexes, SmallVector<unsigned>({0, 4}));
+
+  ASSERT_FALSE(
+      ShuffleVectorInst::isInterleaveMask({0, 4, 1, 6, 2, 6, 3, 7}, 2, 8));
+
+  ASSERT_TRUE(ShuffleVectorInst::isInterleaveMask({4, 0, 5, 1, 6, 2, 7, 3}, 2,
+                                                  8, StartIndexes));
+  ASSERT_EQ(StartIndexes, SmallVector<unsigned>({4, 0}));
+
+  ASSERT_TRUE(ShuffleVectorInst::isInterleaveMask({4, 0, -1, 1, -1, 2, 7, 3}, 2,
+                                                  8, StartIndexes));
+  ASSERT_EQ(StartIndexes, SmallVector<unsigned>({4, 0}));
+
+  ASSERT_TRUE(ShuffleVectorInst::isInterleaveMask({0, 2, 4, 1, 3, 5}, 3, 6,
+                                                  StartIndexes));
+  ASSERT_EQ(StartIndexes, SmallVector<unsigned>({0, 2, 4}));
+
+  ASSERT_TRUE(ShuffleVectorInst::isInterleaveMask({4, -1, 0, 5, 3, 1}, 3, 6,
+                                                  StartIndexes));
+  ASSERT_EQ(StartIndexes, SmallVector<unsigned>({4, 2, 0}));
+
+  ASSERT_FALSE(
+      ShuffleVectorInst::isInterleaveMask({8, 2, 12, 4, 9, 3, 13, 5}, 4, 8));
+}
+
 } // end anonymous namespace
diff --git a/llvm/unittests/IR/ValueHandleTest.cpp b/llvm/unittests/IR/ValueHandleTest.cpp
index 8eb6b5f89a409..e11c545520d5a 100644
--- a/llvm/unittests/IR/ValueHandleTest.cpp
+++ b/llvm/unittests/IR/ValueHandleTest.cpp
@@ -496,8 +496,8 @@ TEST_F(ValueHandle, AssertingVH_DenseMap) {
   Map.insert({BitcastV.get(), 1});
   Map.insert({ConstantV, 2});
   // These will create a temporary AssertingVH during lookup.
-  EXPECT_TRUE(Map.find(BitcastV.get()) != Map.end());
-  EXPECT_TRUE(Map.find(ConstantV) != Map.end());
+  EXPECT_TRUE(Map.contains(BitcastV.get()));
+  EXPECT_TRUE(Map.contains(ConstantV));
   // These will not create a temporary AssertingVH.
   EXPECT_TRUE(Map.find_as(BitcastV.get()) != Map.end());
   EXPECT_TRUE(Map.find_as(ConstantV) != Map.end());
@@ -508,8 +508,8 @@ TEST_F(ValueHandle, PoisoningVH_DenseMap) {
   Map.insert({BitcastV.get(), 1});
   Map.insert({ConstantV, 2});
   // These will create a temporary PoisoningVH during lookup.
-  EXPECT_TRUE(Map.find(BitcastV.get()) != Map.end());
-  EXPECT_TRUE(Map.find(ConstantV) != Map.end());
+  EXPECT_TRUE(Map.contains(BitcastV.get()));
+  EXPECT_TRUE(Map.contains(ConstantV));
   // These will not create a temporary PoisoningVH.
   EXPECT_TRUE(Map.find_as(BitcastV.get()) != Map.end());
   EXPECT_TRUE(Map.find_as(ConstantV) != Map.end());
diff --git a/llvm/unittests/Object/ELFObjectFileTest.cpp b/llvm/unittests/Object/ELFObjectFileTest.cpp
index 99595eba0962e..74d6d72cc8bae 100644
--- a/llvm/unittests/Object/ELFObjectFileTest.cpp
+++ b/llvm/unittests/Object/ELFObjectFileTest.cpp
@@ -808,3 +808,104 @@ TEST(ELFObjectFileTest, ExecutableWithRelocs) {
   RelocatableFileYamlString += ContentsString;
   DoCheck(RelocatableFileYamlString);
 }
+
+TEST(ELFObjectFileTest, GetSectionAndRelocations) {
+  StringRef HeaderString(R"(
+--- !ELF
+FileHeader:
+  Class: ELFCLASS64
+  Data:  ELFDATA2LSB
+  Type:  ET_EXEC
+)");
+
+  using Elf_Shdr = Elf_Shdr_Impl<ELF64LE>;
+
+  auto DoCheckSucceeds = [&](StringRef ContentsString,
+                             std::function<Expected<bool>(const Elf_Shdr &)>
+                                 Matcher) {
+    SmallString<0> Storage;
+    SmallString<128> FullYamlString(HeaderString);
+    FullYamlString += ContentsString;
+    Expected<ELFObjectFile<ELF64LE>> ElfOrErr =
+        toBinary<ELF64LE>(Storage, FullYamlString);
+    ASSERT_THAT_EXPECTED(ElfOrErr, Succeeded());
+
+    Expected<MapVector<const Elf_Shdr *, const Elf_Shdr *>> SecToRelocMapOrErr =
+        ElfOrErr->getELFFile().getSectionAndRelocations(Matcher);
+    ASSERT_THAT_EXPECTED(SecToRelocMapOrErr, Succeeded());
+
+    // Basic verification to make sure we have the correct section types.
+    for (auto const &[Sec, RelaSec] : *SecToRelocMapOrErr) {
+      ASSERT_EQ(Sec->sh_type, ELF::SHT_PROGBITS);
+      ASSERT_EQ(RelaSec->sh_type, ELF::SHT_RELA);
+    }
+  };
+
+  auto DoCheckFails = [&](StringRef ContentsString,
+                          std::function<Expected<bool>(const Elf_Shdr &)>
+                              Matcher,
+                          const char *ErrorMessage) {
+    SmallString<0> Storage;
+    SmallString<128> FullYamlString(HeaderString);
+    FullYamlString += ContentsString;
+    Expected<ELFObjectFile<ELF64LE>> ElfOrErr =
+        toBinary<ELF64LE>(Storage, FullYamlString);
+    ASSERT_THAT_EXPECTED(ElfOrErr, Succeeded());
+
+    Expected<MapVector<const Elf_Shdr *, const Elf_Shdr *>> SecToRelocMapOrErr =
+        ElfOrErr->getELFFile().getSectionAndRelocations(Matcher);
+    ASSERT_THAT_ERROR(SecToRelocMapOrErr.takeError(),
+                      FailedWithMessage(ErrorMessage));
+  };
+
+  auto DefaultMatcher = [](const Elf_Shdr &Sec) -> bool {
+    return Sec.sh_type == ELF::SHT_PROGBITS;
+  };
+
+  StringRef TwoTextSections = R"(
+Sections:
+  - Name: .text
+    Type: SHT_PROGBITS
+    Flags: [ SHF_ALLOC, SHF_EXECINSTR ]
+  - Name: .rela.text
+    Type: SHT_RELA
+    Flags: [ SHF_INFO_LINK ]
+    Info: .text
+  - Name: .text2
+    Type: SHT_PROGBITS
+    Flags: [ SHF_ALLOC, SHF_EXECINSTR ]
+  - Name: .rela.text2
+    Type: SHT_RELA
+    Flags: [ SHF_INFO_LINK ]
+    Info: .text2
+)";
+  DoCheckSucceeds(TwoTextSections, DefaultMatcher);
+
+  StringRef OneTextSection = R"(
+Sections:
+  - Name: .text
+    Type: SHT_PROGBITS
+    Flags: [ SHF_ALLOC, SHF_EXECINSTR ]
+)";
+
+  auto ErroringMatcher = [](const Elf_Shdr &Sec) -> Expected<bool> {
+    if(Sec.sh_type == ELF::SHT_PROGBITS)
+      return createError("This was supposed to fail.");
+    return false;
+  };
+
+  DoCheckFails(OneTextSection, ErroringMatcher,
+               "This was supposed to fail.");
+
+  StringRef MissingRelocatableContent = R"(
+Sections:
+  - Name: .rela.text
+    Type: SHT_RELA
+    Flags: [ SHF_INFO_LINK ]
+    Info: 0xFF
+)";
+
+  DoCheckFails(MissingRelocatableContent, DefaultMatcher,
+               "SHT_RELA section with index 1: failed to get a "
+               "relocated section: invalid section index: 255");
+}
diff --git a/llvm/unittests/ProfileData/MemProfTest.cpp b/llvm/unittests/ProfileData/MemProfTest.cpp
index 0908d3b12af90..9d79385ad5de7 100644
--- a/llvm/unittests/ProfileData/MemProfTest.cpp
+++ b/llvm/unittests/ProfileData/MemProfTest.cpp
@@ -321,14 +321,23 @@ TEST(MemProf, SymbolizationFilter) {
                                                 specifier(), false))
       .Times(1)
       .WillRepeatedly(Return(makeInliningInfo({
-          {"foo", 10, 5, 30},
+          {"foo", 10, 5, 30, "memprof/memprof_test_file.cpp"},
+      })));
+
+  EXPECT_CALL(*Symbolizer, symbolizeInlinedCode(SectionedAddress{0x5000},
+                                                specifier(), false))
+      .Times(1)
+      .WillRepeatedly(Return(makeInliningInfo({
+          // Depending on how the runtime was compiled, only the filename
+          // may be present in the debug information.
+          {"malloc", 70, 57, 3, "memprof_malloc_linux.cpp"},
       })));
 
   CallStackMap CSM;
   CSM[0x1] = {0x1000, 0x2000, 0x3000, 0x4000};
   // This entry should be dropped since all PCs are either not
   // symbolizable or belong to the runtime.
-  CSM[0x2] = {0x1000, 0x2000};
+  CSM[0x2] = {0x1000, 0x2000, 0x5000};
 
   llvm::MapVector<uint64_t, MemInfoBlock> Prof;
   Prof[0x1].AllocCount = 1;
diff --git a/llvm/unittests/Support/FileCollectorTest.cpp b/llvm/unittests/Support/FileCollectorTest.cpp
index 0e8e450386532..184d0e3fdfd17 100644
--- a/llvm/unittests/Support/FileCollectorTest.cpp
+++ b/llvm/unittests/Support/FileCollectorTest.cpp
@@ -35,9 +35,7 @@ class TestingFileCollector : public FileCollector {
   using FileCollector::Seen;
   using FileCollector::VFSWriter;
 
-  bool hasSeen(StringRef fs) {
-    return Seen.find(fs) != Seen.end();
-  }
+  bool hasSeen(StringRef fs) { return Seen.contains(fs); }
 };
 
 } // end anonymous namespace
diff --git a/llvm/unittests/Support/MemoryBufferTest.cpp b/llvm/unittests/Support/MemoryBufferTest.cpp
index d3969fb106d2b..bfabd734f02ef 100644
--- a/llvm/unittests/Support/MemoryBufferTest.cpp
+++ b/llvm/unittests/Support/MemoryBufferTest.cpp
@@ -161,6 +161,10 @@ TEST_F(MemoryBufferTest, copy) {
 
   // verify the two copies do not point to the same place
   EXPECT_NE(MBC1->getBufferStart(), MBC2->getBufferStart());
+
+  // check that copies from defaulted stringrefs don't trigger UB.
+  OwningBuffer MBC3(MemoryBuffer::getMemBufferCopy(StringRef{}));
+  EXPECT_NE(nullptr, MBC3.get());
 }
 
 #if LLVM_ENABLE_THREADS
diff --git a/llvm/unittests/Support/RISCVISAInfoTest.cpp b/llvm/unittests/Support/RISCVISAInfoTest.cpp
index 18620180e87f1..2f623a94ce2a8 100644
--- a/llvm/unittests/Support/RISCVISAInfoTest.cpp
+++ b/llvm/unittests/Support/RISCVISAInfoTest.cpp
@@ -10,12 +10,13 @@
 #include "llvm/Testing/Support/Error.h"
 #include "gtest/gtest.h"
 
+using ::testing::ElementsAre;
+
 using namespace llvm;
 
 bool operator==(const llvm::RISCVExtensionInfo &A,
                 const llvm::RISCVExtensionInfo &B) {
-  return A.ExtName == B.ExtName && A.MajorVersion == B.MajorVersion &&
-         A.MinorVersion == B.MinorVersion;
+  return A.MajorVersion == B.MajorVersion && A.MinorVersion == B.MinorVersion;
 }
 
 TEST(ParseNormalizedArchString, RejectsUpperCase) {
@@ -48,33 +49,29 @@ TEST(ParseNormalizedArchString, AcceptsValidBaseISAsAndSetsXLen) {
   ASSERT_THAT_EXPECTED(MaybeRV32I, Succeeded());
   RISCVISAInfo &InfoRV32I = **MaybeRV32I;
   EXPECT_EQ(InfoRV32I.getExtensions().size(), 1UL);
-  EXPECT_TRUE(InfoRV32I.getExtensions().at("i") ==
-              (RISCVExtensionInfo{"i", 2, 0}));
+  EXPECT_TRUE(InfoRV32I.getExtensions().at("i") == (RISCVExtensionInfo{2, 0}));
   EXPECT_EQ(InfoRV32I.getXLen(), 32U);
 
   auto MaybeRV32E = RISCVISAInfo::parseNormalizedArchString("rv32e2p0");
   ASSERT_THAT_EXPECTED(MaybeRV32E, Succeeded());
   RISCVISAInfo &InfoRV32E = **MaybeRV32E;
   EXPECT_EQ(InfoRV32E.getExtensions().size(), 1UL);
-  EXPECT_TRUE(InfoRV32E.getExtensions().at("e") ==
-              (RISCVExtensionInfo{"e", 2, 0}));
-  EXPECT_EQ(InfoRV32I.getXLen(), 32U);
+  EXPECT_TRUE(InfoRV32E.getExtensions().at("e") == (RISCVExtensionInfo{2, 0}));
+  EXPECT_EQ(InfoRV32E.getXLen(), 32U);
 
   auto MaybeRV64I = RISCVISAInfo::parseNormalizedArchString("rv64i2p0");
   ASSERT_THAT_EXPECTED(MaybeRV64I, Succeeded());
   RISCVISAInfo &InfoRV64I = **MaybeRV64I;
   EXPECT_EQ(InfoRV64I.getExtensions().size(), 1UL);
-  EXPECT_TRUE(InfoRV64I.getExtensions().at("i") ==
-              (RISCVExtensionInfo{"i", 2, 0}));
+  EXPECT_TRUE(InfoRV64I.getExtensions().at("i") == (RISCVExtensionInfo{2, 0}));
   EXPECT_EQ(InfoRV64I.getXLen(), 64U);
 
   auto MaybeRV64E = RISCVISAInfo::parseNormalizedArchString("rv64e2p0");
   ASSERT_THAT_EXPECTED(MaybeRV64E, Succeeded());
   RISCVISAInfo &InfoRV64E = **MaybeRV64E;
   EXPECT_EQ(InfoRV64E.getExtensions().size(), 1UL);
-  EXPECT_TRUE(InfoRV64E.getExtensions().at("e") ==
-              (RISCVExtensionInfo{"e", 2, 0}));
-  EXPECT_EQ(InfoRV64I.getXLen(), 64U);
+  EXPECT_TRUE(InfoRV64E.getExtensions().at("e") == (RISCVExtensionInfo{2, 0}));
+  EXPECT_EQ(InfoRV64E.getXLen(), 64U);
 }
 
 TEST(ParseNormalizedArchString, AcceptsArbitraryExtensionsAndVersions) {
@@ -83,14 +80,12 @@ TEST(ParseNormalizedArchString, AcceptsArbitraryExtensionsAndVersions) {
   ASSERT_THAT_EXPECTED(MaybeISAInfo, Succeeded());
   RISCVISAInfo &Info = **MaybeISAInfo;
   EXPECT_EQ(Info.getExtensions().size(), 5UL);
-  EXPECT_TRUE(Info.getExtensions().at("i") == (RISCVExtensionInfo{"i", 5, 1}));
-  EXPECT_TRUE(Info.getExtensions().at("m") == (RISCVExtensionInfo{"m", 3, 2}));
+  EXPECT_TRUE(Info.getExtensions().at("i") == (RISCVExtensionInfo{5, 1}));
+  EXPECT_TRUE(Info.getExtensions().at("m") == (RISCVExtensionInfo{3, 2}));
   EXPECT_TRUE(Info.getExtensions().at("zmadeup") ==
-              (RISCVExtensionInfo{"zmadeup", 11, 12}));
-  EXPECT_TRUE(Info.getExtensions().at("sfoo") ==
-              (RISCVExtensionInfo{"sfoo", 2, 0}));
-  EXPECT_TRUE(Info.getExtensions().at("xbar") ==
-              (RISCVExtensionInfo{"xbar", 3, 0}));
+              (RISCVExtensionInfo{11, 12}));
+  EXPECT_TRUE(Info.getExtensions().at("sfoo") == (RISCVExtensionInfo{2, 0}));
+  EXPECT_TRUE(Info.getExtensions().at("xbar") == (RISCVExtensionInfo{3, 0}));
 }
 
 TEST(ParseNormalizedArchString, UpdatesFLenMinVLenMaxELen) {
@@ -103,3 +98,344 @@ TEST(ParseNormalizedArchString, UpdatesFLenMinVLenMaxELen) {
   EXPECT_EQ(Info.getMinVLen(), 64U);
   EXPECT_EQ(Info.getMaxELen(), 64U);
 }
+
+TEST(ParseArchString, RejectsUpperCase) {
+  for (StringRef Input : {"RV32", "rV64", "rv32i2P0", "rv64i2p0_A2p0"}) {
+    EXPECT_EQ(toString(RISCVISAInfo::parseArchString(Input, true).takeError()),
+              "string must be lowercase");
+  }
+}
+
+TEST(ParseArchString, RejectsInvalidBaseISA) {
+  for (StringRef Input : {"rv32", "rv64", "rv65i"}) {
+    EXPECT_EQ(toString(RISCVISAInfo::parseArchString(Input, true).takeError()),
+              "string must begin with rv32{i,e,g} or rv64{i,g}");
+  }
+  for (StringRef Input : {"rv32j", "rv64k", "rv32_i"}) {
+    EXPECT_EQ(toString(RISCVISAInfo::parseArchString(Input, true).takeError()),
+              "first letter should be 'e', 'i' or 'g'");
+  }
+}
+
+TEST(ParseArchString, RejectsUnsupportedBaseISA) {
+  EXPECT_EQ(toString(RISCVISAInfo::parseArchString("rv64e", true).takeError()),
+            "standard user-level extension 'e' requires 'rv32'");
+  for (StringRef Input : {"rv128i", "rv128g"}) {
+    EXPECT_EQ(toString(RISCVISAInfo::parseArchString(Input, true).takeError()),
+              "string must begin with rv32{i,e,g} or rv64{i,g}");
+  }
+}
+
+TEST(ParseArchString, AcceptsSupportedBaseISAsAndSetsXLenAndFLen) {
+  auto MaybeRV32I = RISCVISAInfo::parseArchString("rv32i", true);
+  ASSERT_THAT_EXPECTED(MaybeRV32I, Succeeded());
+  RISCVISAInfo &InfoRV32I = **MaybeRV32I;
+  RISCVISAInfo::OrderedExtensionMap ExtsRV32I = InfoRV32I.getExtensions();
+  EXPECT_EQ(ExtsRV32I.size(), 1UL);
+  EXPECT_TRUE(ExtsRV32I.at("i") == (RISCVExtensionInfo{2, 0}));
+  EXPECT_EQ(InfoRV32I.getXLen(), 32U);
+  EXPECT_EQ(InfoRV32I.getFLen(), 0U);
+
+  auto MaybeRV32E = RISCVISAInfo::parseArchString("rv32e", true);
+  ASSERT_THAT_EXPECTED(MaybeRV32E, Succeeded());
+  RISCVISAInfo &InfoRV32E = **MaybeRV32E;
+  RISCVISAInfo::OrderedExtensionMap ExtsRV32E = InfoRV32E.getExtensions();
+  EXPECT_EQ(ExtsRV32E.size(), 1UL);
+  EXPECT_TRUE(ExtsRV32E.at("e") == (RISCVExtensionInfo{1, 9}));
+  EXPECT_EQ(InfoRV32E.getXLen(), 32U);
+  EXPECT_EQ(InfoRV32E.getFLen(), 0U);
+
+  auto MaybeRV32G = RISCVISAInfo::parseArchString("rv32g", true);
+  ASSERT_THAT_EXPECTED(MaybeRV32G, Succeeded());
+  RISCVISAInfo &InfoRV32G = **MaybeRV32G;
+  RISCVISAInfo::OrderedExtensionMap ExtsRV32G = InfoRV32G.getExtensions();
+  EXPECT_EQ(ExtsRV32G.size(), 5UL);
+  EXPECT_TRUE(ExtsRV32G.at("i") == (RISCVExtensionInfo{2, 0}));
+  EXPECT_TRUE(ExtsRV32G.at("m") == (RISCVExtensionInfo{2, 0}));
+  EXPECT_TRUE(ExtsRV32G.at("a") == (RISCVExtensionInfo{2, 0}));
+  EXPECT_TRUE(ExtsRV32G.at("f") == (RISCVExtensionInfo{2, 0}));
+  EXPECT_TRUE(ExtsRV32G.at("d") == (RISCVExtensionInfo{2, 0}));
+  EXPECT_EQ(InfoRV32G.getXLen(), 32U);
+  EXPECT_EQ(InfoRV32G.getFLen(), 64U);
+
+  auto MaybeRV64I = RISCVISAInfo::parseArchString("rv64i", true);
+  ASSERT_THAT_EXPECTED(MaybeRV64I, Succeeded());
+  RISCVISAInfo &InfoRV64I = **MaybeRV64I;
+  RISCVISAInfo::OrderedExtensionMap ExtsRV64I = InfoRV64I.getExtensions();
+  EXPECT_EQ(ExtsRV64I.size(), 1UL);
+  EXPECT_TRUE(ExtsRV64I.at("i") == (RISCVExtensionInfo{2, 0}));
+  EXPECT_EQ(InfoRV64I.getXLen(), 64U);
+  EXPECT_EQ(InfoRV64I.getFLen(), 0U);
+
+  auto MaybeRV64G = RISCVISAInfo::parseArchString("rv64g", true);
+  ASSERT_THAT_EXPECTED(MaybeRV64G, Succeeded());
+  RISCVISAInfo &InfoRV64G = **MaybeRV64G;
+  RISCVISAInfo::OrderedExtensionMap ExtsRV64G = InfoRV64G.getExtensions();
+  EXPECT_EQ(ExtsRV64G.size(), 5UL);
+  EXPECT_TRUE(ExtsRV64G.at("i") == (RISCVExtensionInfo{2, 0}));
+  EXPECT_TRUE(ExtsRV64G.at("m") == (RISCVExtensionInfo{2, 0}));
+  EXPECT_TRUE(ExtsRV64G.at("a") == (RISCVExtensionInfo{2, 0}));
+  EXPECT_TRUE(ExtsRV64G.at("f") == (RISCVExtensionInfo{2, 0}));
+  EXPECT_TRUE(ExtsRV64G.at("d") == (RISCVExtensionInfo{2, 0}));
+  EXPECT_EQ(InfoRV64G.getXLen(), 64U);
+  EXPECT_EQ(InfoRV64G.getFLen(), 64U);
+}
+
+TEST(ParseArchString, RequiresCanonicalOrderForExtensions) {
+  EXPECT_EQ(
+      toString(RISCVISAInfo::parseArchString("rv64idf", true).takeError()),
+      "standard user-level extension not given in canonical order 'f'");
+  EXPECT_EQ(
+      toString(RISCVISAInfo::parseArchString("rv32iam", true).takeError()),
+      "standard user-level extension not given in canonical order 'm'");
+  EXPECT_EQ(
+      toString(
+          RISCVISAInfo::parseArchString("rv32i_zfinx_a", true).takeError()),
+      "invalid extension prefix 'a'");
+  EXPECT_EQ(
+      toString(RISCVISAInfo::parseArchString("rv64i_svnapot_zicsr", true)
+                   .takeError()),
+      "standard user-level extension not given in canonical order 'zicsr'");
+  EXPECT_THAT_EXPECTED(
+      RISCVISAInfo::parseArchString("rv64imafdc_zicsr_svnapot", true),
+      Succeeded());
+}
+
+TEST(ParseArchString, RejectsUnrecognizedExtensionNamesByDefault) {
+  EXPECT_EQ(toString(RISCVISAInfo::parseArchString("rv64ib", true).takeError()),
+            "unsupported standard user-level extension 'b'");
+  EXPECT_EQ(
+      toString(
+          RISCVISAInfo::parseArchString("rv32i_zmadeup", true).takeError()),
+      "unsupported standard user-level extension 'zmadeup'");
+  EXPECT_EQ(
+      toString(
+          RISCVISAInfo::parseArchString("rv64g_smadeup", true).takeError()),
+      "unsupported standard supervisor-level extension 'smadeup'");
+  EXPECT_EQ(
+      toString(
+          RISCVISAInfo::parseArchString("rv64g_sxmadeup", true).takeError()),
+      "unsupported non-standard supervisor-level extension 'sxmadeup'");
+  EXPECT_EQ(
+      toString(
+          RISCVISAInfo::parseArchString("rv64g_xmadeup", true).takeError()),
+      "unsupported non-standard user-level extension 'xmadeup'");
+}
+
+TEST(ParseArchString, IgnoresUnrecognizedExtensionNamesWithIgnoreUnknown) {
+  for (StringRef Input : {"rv32ib", "rv32i_zmadeup", "rv64i_smadeup",
+                          "rv32i_sxmadeup", "rv64i_xmadeup"}) {
+    auto MaybeISAInfo = RISCVISAInfo::parseArchString(Input, true, false, true);
+    ASSERT_THAT_EXPECTED(MaybeISAInfo, Succeeded());
+    RISCVISAInfo &Info = **MaybeISAInfo;
+    RISCVISAInfo::OrderedExtensionMap Exts = Info.getExtensions();
+    EXPECT_EQ(Exts.size(), 1UL);
+    EXPECT_TRUE(Exts.at("i") == (RISCVExtensionInfo{2, 0}));
+  }
+
+  // Checks that supported extensions aren't incorrectly ignored when a
+  // version is present (an early version of the patch had this mistake).
+  auto MaybeISAInfo =
+      RISCVISAInfo::parseArchString("rv32i_zbc1p0_xmadeup", true, false, true);
+  ASSERT_THAT_EXPECTED(MaybeISAInfo, Succeeded());
+  RISCVISAInfo::OrderedExtensionMap Exts = (*MaybeISAInfo)->getExtensions();
+  EXPECT_TRUE(Exts.at("zbc") == (RISCVExtensionInfo{1, 0}));
+}
+
+TEST(ParseArchString, AcceptsVersionInLongOrShortForm) {
+  for (StringRef Input : {"rv64i2", "rv64i2p0"}) {
+    auto MaybeISAInfo = RISCVISAInfo::parseArchString(Input, true);
+    ASSERT_THAT_EXPECTED(MaybeISAInfo, Succeeded());
+    RISCVISAInfo::OrderedExtensionMap Exts = (*MaybeISAInfo)->getExtensions();
+    EXPECT_TRUE(Exts.at("i") == (RISCVExtensionInfo{2, 0}));
+  }
+  for (StringRef Input : {"rv32i_zfinx1", "rv32i_zfinx1p0"}) {
+    auto MaybeISAInfo = RISCVISAInfo::parseArchString(Input, true);
+    ASSERT_THAT_EXPECTED(MaybeISAInfo, Succeeded());
+    RISCVISAInfo::OrderedExtensionMap Exts = (*MaybeISAInfo)->getExtensions();
+    EXPECT_TRUE(Exts.at("zfinx") == (RISCVExtensionInfo{1, 0}));
+  }
+}
+
+TEST(ParseArchString, RejectsUnrecognizedExtensionVersionsByDefault) {
+  EXPECT_EQ(
+      toString(RISCVISAInfo::parseArchString("rv64i1p0", true).takeError()),
+      "unsupported version number 1.0 for extension 'i'");
+  EXPECT_EQ(
+      toString(RISCVISAInfo::parseArchString("rv64i9p9", true).takeError()),
+      "unsupported version number 9.9 for extension 'i'");
+  EXPECT_EQ(
+      toString(RISCVISAInfo::parseArchString("rv32im0p1", true).takeError()),
+      "unsupported version number 0.1 for extension 'm'");
+  EXPECT_EQ(toString(RISCVISAInfo::parseArchString("rv32izifencei10p10", true)
+                         .takeError()),
+            "unsupported version number 10.10 for extension 'zifencei'");
+}
+
+TEST(ParseArchString, RejectsUnrecognisedBaseISAVersionEvenWithIgnoreUnknown) {
+  EXPECT_EQ(
+      toString(RISCVISAInfo::parseArchString("rv64i1p0", true, false, true)
+                   .takeError()),
+      "unsupported version number 1.0 for extension 'i'");
+}
+
+TEST(ParseArchString,
+     IgnoresExtensionsWithUnrecognizedVersionsWithIgnoreUnknown) {
+  for (StringRef Input : {"rv32im1p1", "rv64i_svnapot10p9", "rv32i_zicsr0p5"}) {
+    auto MaybeISAInfo = RISCVISAInfo::parseArchString(Input, true, false, true);
+    ASSERT_THAT_EXPECTED(MaybeISAInfo, Succeeded());
+    RISCVISAInfo::OrderedExtensionMap Exts = (*MaybeISAInfo)->getExtensions();
+    EXPECT_EQ(Exts.size(), 1UL);
+    EXPECT_TRUE(Exts.at("i") == (RISCVExtensionInfo{2, 0}));
+  }
+}
+
+TEST(ParseArchString, AcceptsUnderscoreSplittingExtensions) {
+  for (StringRef Input : {"rv32imafdczifencei", "rv32i_m_a_f_d_c_zifencei"}) {
+    auto MaybeISAInfo = RISCVISAInfo::parseArchString(Input, true);
+    ASSERT_THAT_EXPECTED(MaybeISAInfo, Succeeded());
+    RISCVISAInfo::OrderedExtensionMap Exts = (*MaybeISAInfo)->getExtensions();
+    EXPECT_EQ(Exts.size(), 7UL);
+    EXPECT_EQ(Exts.count("i"), 1U);
+    EXPECT_EQ(Exts.count("m"), 1U);
+    EXPECT_EQ(Exts.count("a"), 1U);
+    EXPECT_EQ(Exts.count("f"), 1U);
+    EXPECT_EQ(Exts.count("d"), 1U);
+    EXPECT_EQ(Exts.count("c"), 1U);
+    EXPECT_EQ(Exts.count("zifencei"), 1U);
+  }
+}
+
+TEST(ParseArchString, RejectsDoubleOrTrailingUnderscore) {
+  EXPECT_EQ(
+      toString(RISCVISAInfo::parseArchString("rv64i__m", true).takeError()),
+      "invalid standard user-level extension '_'");
+
+  for (StringRef Input :
+       {"rv32ezicsr__zifencei", "rv32i_", "rv32izicsr_", "rv64im_"}) {
+    EXPECT_EQ(toString(RISCVISAInfo::parseArchString(Input, true).takeError()),
+              "extension name missing after separator '_'");
+  }
+}
+
+TEST(ParseArchString, RejectsDuplicateExtensionNames) {
+  EXPECT_EQ(toString(RISCVISAInfo::parseArchString("rv64ii", true).takeError()),
+            "invalid standard user-level extension 'i'");
+  EXPECT_EQ(toString(RISCVISAInfo::parseArchString("rv32ee", true).takeError()),
+            "invalid standard user-level extension 'e'");
+  EXPECT_EQ(
+      toString(RISCVISAInfo::parseArchString("rv64imm", true).takeError()),
+      "standard user-level extension not given in canonical order 'm'");
+  EXPECT_EQ(
+      toString(
+          RISCVISAInfo::parseArchString("rv32i_zicsr_zicsr", true).takeError()),
+      "duplicated standard user-level extension 'zicsr'");
+}
+
+TEST(ParseArchString,
+     RejectsExperimentalExtensionsIfNotEnableExperimentalExtension) {
+  EXPECT_EQ(
+      toString(
+          RISCVISAInfo::parseArchString("rv64izihintntl", false).takeError()),
+      "requires '-menable-experimental-extensions' for experimental extension "
+      "'zihintntl'");
+}
+
+TEST(ParseArchString,
+     AcceptsExperimentalExtensionsIfEnableExperimentalExtension) {
+  // Note: If zihintntl becomes none-experimental, this test will need
+  // updating (and unfortunately, it will still pass). The failure of
+  // RejectsExperimentalExtensionsIfNotEnableExperimentalExtension will
+  // hopefully serve as a reminder to update.
+  auto MaybeISAInfo =
+      RISCVISAInfo::parseArchString("rv64izihintntl", true, false);
+  ASSERT_THAT_EXPECTED(MaybeISAInfo, Succeeded());
+  RISCVISAInfo::OrderedExtensionMap Exts = (*MaybeISAInfo)->getExtensions();
+  EXPECT_EQ(Exts.size(), 2UL);
+  EXPECT_EQ(Exts.count("zihintntl"), 1U);
+  auto MaybeISAInfo2 = RISCVISAInfo::parseArchString("rv64izihintntl0p2", true);
+  ASSERT_THAT_EXPECTED(MaybeISAInfo2, Succeeded());
+  RISCVISAInfo::OrderedExtensionMap Exts2 = (*MaybeISAInfo2)->getExtensions();
+  EXPECT_EQ(Exts2.size(), 2UL);
+  EXPECT_EQ(Exts2.count("zihintntl"), 1U);
+}
+
+TEST(ParseArchString,
+     RequiresExplicitVersionNumberForExperimentalExtensionByDefault) {
+  EXPECT_EQ(
+      toString(
+          RISCVISAInfo::parseArchString("rv64izihintntl", true).takeError()),
+      "experimental extension requires explicit version number `zihintntl`");
+}
+
+TEST(ParseArchString,
+     AcceptsUnrecognizedVersionIfNotExperimentalExtensionVersionCheck) {
+  auto MaybeISAInfo =
+      RISCVISAInfo::parseArchString("rv64izihintntl9p9", true, false);
+  ASSERT_THAT_EXPECTED(MaybeISAInfo, Succeeded());
+  RISCVISAInfo::OrderedExtensionMap Exts = (*MaybeISAInfo)->getExtensions();
+  EXPECT_EQ(Exts.size(), 2UL);
+  EXPECT_TRUE(Exts.at("zihintntl") == (RISCVExtensionInfo{9, 9}));
+}
+
+TEST(ParseArchString, RejectsUnrecognizedVersionForExperimentalExtension) {
+  EXPECT_EQ(
+      toString(
+          RISCVISAInfo::parseArchString("rv64izihintntl9p9", true).takeError()),
+      "unsupported version number 9.9 for experimental extension 'zihintntl' "
+      "(this compiler supports 0.2)");
+}
+
+TEST(ParseArchString, RejectsExtensionVersionForG) {
+  for (StringRef Input : {"rv32g1c", "rv64g2p0"}) {
+    EXPECT_EQ(toString(RISCVISAInfo::parseArchString(Input, true).takeError()),
+              "version not supported for 'g'");
+  }
+}
+
+TEST(ParseArchString, AddsImpliedExtensions) {
+  // Does not attempt to exhaustively test all implications.
+  auto MaybeRV64ID = RISCVISAInfo::parseArchString("rv64id", true);
+  ASSERT_THAT_EXPECTED(MaybeRV64ID, Succeeded());
+  RISCVISAInfo::OrderedExtensionMap ExtsRV64ID =
+      (*MaybeRV64ID)->getExtensions();
+  EXPECT_EQ(ExtsRV64ID.size(), 3UL);
+  EXPECT_EQ(ExtsRV64ID.count("i"), 1U);
+  EXPECT_EQ(ExtsRV64ID.count("f"), 1U);
+  EXPECT_EQ(ExtsRV64ID.count("d"), 1U);
+
+  auto MaybeRV32IZKN = RISCVISAInfo::parseArchString("rv64izkn", true);
+  ASSERT_THAT_EXPECTED(MaybeRV32IZKN, Succeeded());
+  RISCVISAInfo::OrderedExtensionMap ExtsRV32IZKN =
+      (*MaybeRV32IZKN)->getExtensions();
+  EXPECT_EQ(ExtsRV32IZKN.size(), 8UL);
+  EXPECT_EQ(ExtsRV32IZKN.count("i"), 1U);
+  EXPECT_EQ(ExtsRV32IZKN.count("zbkb"), 1U);
+  EXPECT_EQ(ExtsRV32IZKN.count("zbkc"), 1U);
+  EXPECT_EQ(ExtsRV32IZKN.count("zbkx"), 1U);
+  EXPECT_EQ(ExtsRV32IZKN.count("zkne"), 1U);
+  EXPECT_EQ(ExtsRV32IZKN.count("zknd"), 1U);
+  EXPECT_EQ(ExtsRV32IZKN.count("zknh"), 1U);
+  EXPECT_EQ(ExtsRV32IZKN.count("zkn"), 1U);
+}
+
+TEST(ParseArchString, RejectsConflictingExtensions) {
+  for (StringRef Input : {"rv32ifzfinx", "rv64gzdinx"}) {
+    EXPECT_EQ(toString(RISCVISAInfo::parseArchString(Input, true).takeError()),
+              "'f' and 'zfinx' extensions are incompatible");
+  }
+}
+
+TEST(ToFeatureVector, IIsDroppedAndExperimentalExtensionsArePrefixed) {
+  auto MaybeISAInfo1 =
+      RISCVISAInfo::parseArchString("rv64im_zihintntl", true, false);
+  ASSERT_THAT_EXPECTED(MaybeISAInfo1, Succeeded());
+  EXPECT_THAT((*MaybeISAInfo1)->toFeatureVector(),
+              ElementsAre("+m", "+experimental-zihintntl"));
+
+  auto MaybeISAInfo2 = RISCVISAInfo::parseArchString(
+      "rv32e_zihintntl_xventanacondops", true, false);
+  ASSERT_THAT_EXPECTED(MaybeISAInfo2, Succeeded());
+  EXPECT_THAT((*MaybeISAInfo2)->toFeatureVector(),
+              ElementsAre("+e", "+experimental-zihintntl", "+xventanacondops"));
+}
diff --git a/llvm/unittests/Support/ScopedPrinterTest.cpp b/llvm/unittests/Support/ScopedPrinterTest.cpp
index 5afb796aeacff..b98fe5361481a 100644
--- a/llvm/unittests/Support/ScopedPrinterTest.cpp
+++ b/llvm/unittests/Support/ScopedPrinterTest.cpp
@@ -236,8 +236,8 @@ DoesNotExist: 0x5
 
   const char *JSONExpectedOut = R"({
   "Exists": {
-    "Value": "Name2",
-    "RawValue": 2
+    "Name": "Name2",
+    "Value": 2
   },
   "DoesNotExist": 5
 })";
@@ -345,15 +345,15 @@ FirstSecondThirdByteMask [ (0x333)
 
   const char *JSONExpectedOut = R"({
   "ZeroFlag": {
-    "RawFlags": 0,
+    "Value": 0,
     "Flags": []
   },
   "NoFlag": {
-    "RawFlags": 8,
+    "Value": 8,
     "Flags": []
   },
   "Flag1": {
-    "RawFlags": 1,
+    "Value": 1,
     "Flags": [
       {
         "Name": "Name1",
@@ -362,7 +362,7 @@ FirstSecondThirdByteMask [ (0x333)
     ]
   },
   "Flag1&3": {
-    "RawFlags": 5,
+    "Value": 5,
     "Flags": [
       {
         "Name": "Name1",
@@ -375,30 +375,30 @@ FirstSecondThirdByteMask [ (0x333)
     ]
   },
   "ZeroFlagRaw": {
-    "RawFlags": 0,
+    "Value": 0,
     "Flags": []
   },
   "NoFlagRaw": {
-    "RawFlags": 8,
+    "Value": 8,
     "Flags": [
       8
     ]
   },
   "Flag1Raw": {
-    "RawFlags": 1,
+    "Value": 1,
     "Flags": [
       1
     ]
   },
   "Flag1&3Raw": {
-    "RawFlags": 5,
+    "Value": 5,
     "Flags": [
       1,
       4
     ]
   },
   "FlagSorted": {
-    "RawFlags": 7,
+    "Value": 7,
     "Flags": [
       {
         "Name": "A",
@@ -415,7 +415,7 @@ FirstSecondThirdByteMask [ (0x333)
     ]
   },
   "NoBitMask": {
-    "RawFlags": 4095,
+    "Value": 4095,
     "Flags": [
       {
         "Name": "FirstByte1",
@@ -456,7 +456,7 @@ FirstSecondThirdByteMask [ (0x333)
     ]
   },
   "FirstByteMask": {
-    "RawFlags": 3,
+    "Value": 3,
     "Flags": [
       {
         "Name": "FirstByte3",
@@ -465,7 +465,7 @@ FirstSecondThirdByteMask [ (0x333)
     ]
   },
   "SecondByteMask": {
-    "RawFlags": 48,
+    "Value": 48,
     "Flags": [
       {
         "Name": "SecondByte3",
@@ -474,7 +474,7 @@ FirstSecondThirdByteMask [ (0x333)
     ]
   },
   "ValueOutsideMask": {
-    "RawFlags": 1,
+    "Value": 1,
     "Flags": [
       {
         "Name": "FirstByte1",
@@ -483,11 +483,11 @@ FirstSecondThirdByteMask [ (0x333)
     ]
   },
   "FirstSecondByteMask": {
-    "RawFlags": 255,
+    "Value": 255,
     "Flags": []
   },
   "FirstSecondThirdByteMask": {
-    "RawFlags": 819,
+    "Value": 819,
     "Flags": [
       {
         "Name": "FirstByte3",
@@ -594,8 +594,8 @@ label: value (0)
   "int8_t-min": -128,
   "apsint": 9999999999999999999999,
   "label": {
-    "Value": "value",
-    "RawValue": 0
+    "Name": "value",
+    "Value": 0
   }
 })";
   verifyAll(ExpectedOut, JSONExpectedOut, PrintFunc);
@@ -761,8 +761,8 @@ HexLabel: Name (0x10)
   const char *JSONExpectedOut = R"({
   "HexNumber": 16,
   "HexLabel": {
-    "Value": "Name",
-    "RawValue": 16
+    "Name": "Name",
+    "Value": 16
   }
 })";
   verifyAll(ExpectedOut, JSONExpectedOut, PrintFunc);
diff --git a/llvm/unittests/TargetParser/TargetParserTest.cpp b/llvm/unittests/TargetParser/TargetParserTest.cpp
index ef5aa82c169a7..741d5a2d4b480 100644
--- a/llvm/unittests/TargetParser/TargetParserTest.cpp
+++ b/llvm/unittests/TargetParser/TargetParserTest.cpp
@@ -120,7 +120,7 @@ TEST_P(ARMCPUTestFixture, ARMCPUTests) {
   ARM::ArchKind AK = ARM::parseCPUArch(params.CPUName);
   EXPECT_EQ(params.ExpectedArch, ARM::getArchName(AK));
 
-  unsigned FPUKind = ARM::getDefaultFPU(params.CPUName, AK);
+  ARM::FPUKind FPUKind = ARM::getDefaultFPU(params.CPUName, AK);
   EXPECT_EQ(params.ExpectedFPU, ARM::getFPUName(FPUKind));
 
   uint64_t default_extensions = ARM::getDefaultExtensions(params.CPUName, AK);
@@ -765,10 +765,10 @@ static bool
 testArchExtDependency(const char *ArchExt,
                       const std::initializer_list<const char *> &Expected) {
   std::vector<StringRef> Features;
-  unsigned FPUID;
+  ARM::FPUKind FPUKind;
 
   if (!ARM::appendArchExtFeatures("", ARM::ArchKind::ARMV8_1MMainline, ArchExt,
-                                  Features, FPUID))
+                                  Features, FPUKind))
     return false;
 
   return llvm::all_of(Expected, [&](StringRef Ext) {
@@ -1600,7 +1600,7 @@ TEST(TargetParserTest, AArch64ExtensionFeatures) {
       AArch64::AEK_B16B16,  AArch64::AEK_SMEF16F16, AArch64::AEK_CSSC,
       AArch64::AEK_RCPC3,   AArch64::AEK_THE,       AArch64::AEK_D128,
       AArch64::AEK_LSE128,  AArch64::AEK_SPECRES2,  AArch64::AEK_RASv2,
-      AArch64::AEK_ITE,
+      AArch64::AEK_ITE,     AArch64::AEK_GCS,
   };
 
   std::vector<StringRef> Features;
@@ -1671,6 +1671,7 @@ TEST(TargetParserTest, AArch64ExtensionFeatures) {
   EXPECT_TRUE(llvm::is_contained(Features, "+lse128"));
   EXPECT_TRUE(llvm::is_contained(Features, "+specres2"));
   EXPECT_TRUE(llvm::is_contained(Features, "+ite"));
+  EXPECT_TRUE(llvm::is_contained(Features, "+gcs"));
 
   // Assuming we listed every extension above, this should produce the same
   // result. (note that AEK_NONE doesn't have a name so it won't be in the
@@ -1793,6 +1794,7 @@ TEST(TargetParserTest, AArch64ArchExtFeature) {
       {"pmuv3", "nopmuv3", "+perfmon", "-perfmon"},
       {"predres2", "nopredres2", "+specres2", "-specres2"},
       {"rasv2", "norasv2", "+rasv2", "-rasv2"},
+      {"gcs", "nogcs", "+gcs", "-gcs"},
   };
 
   for (unsigned i = 0; i < std::size(ArchExt); i++) {
diff --git a/llvm/unittests/Transforms/Vectorize/VPlanHCFGTest.cpp b/llvm/unittests/Transforms/Vectorize/VPlanHCFGTest.cpp
index 4b4a5173aa5ca..e8c84cc2b62e8 100644
--- a/llvm/unittests/Transforms/Vectorize/VPlanHCFGTest.cpp
+++ b/llvm/unittests/Transforms/Vectorize/VPlanHCFGTest.cpp
@@ -139,8 +139,7 @@ compound=true
   TargetLibraryInfo TLI(TLII);
   SmallPtrSet<Instruction *, 1> DeadInstructions;
   VPlanTransforms::VPInstructionsToVPRecipes(
-      LI->getLoopFor(LoopHeader), Plan, [](PHINode *P) { return nullptr; },
-      DeadInstructions, *SE, TLI);
+      Plan, [](PHINode *P) { return nullptr; }, DeadInstructions, *SE, TLI);
 }
 
 TEST_F(VPlanHCFGTest, testVPInstructionToVPRecipesInner) {
@@ -171,8 +170,7 @@ TEST_F(VPlanHCFGTest, testVPInstructionToVPRecipesInner) {
   TargetLibraryInfoImpl TLII(Triple(M.getTargetTriple()));
   TargetLibraryInfo TLI(TLII);
   VPlanTransforms::VPInstructionsToVPRecipes(
-      LI->getLoopFor(LoopHeader), Plan, [](PHINode *P) { return nullptr; },
-      DeadInstructions, *SE, TLI);
+      Plan, [](PHINode *P) { return nullptr; }, DeadInstructions, *SE, TLI);
 
   VPBlockBase *Entry = Plan->getEntry()->getEntryBasicBlock();
   EXPECT_NE(nullptr, Entry->getSingleSuccessor());
diff --git a/llvm/unittests/Transforms/Vectorize/VPlanTest.cpp b/llvm/unittests/Transforms/Vectorize/VPlanTest.cpp
index 430cb290fe0f8..9a150cffae502 100644
--- a/llvm/unittests/Transforms/Vectorize/VPlanTest.cpp
+++ b/llvm/unittests/Transforms/Vectorize/VPlanTest.cpp
@@ -899,7 +899,7 @@ TEST(VPRecipeTest, CastVPWidenSelectRecipeToVPUserAndVPDef) {
   Args.push_back(&Op2);
   Args.push_back(&Op3);
   VPWidenSelectRecipe WidenSelectR(*SelectI,
-                                   make_range(Args.begin(), Args.end()), false);
+                                   make_range(Args.begin(), Args.end()));
   EXPECT_TRUE(isa<VPUser>(&WidenSelectR));
   VPRecipeBase *BaseR = &WidenSelectR;
   EXPECT_TRUE(isa<VPUser>(BaseR));
@@ -976,8 +976,7 @@ TEST(VPRecipeTest, CastVPReplicateRecipeToVPUser) {
   Args.push_back(&Op1);
   Args.push_back(&Op2);
 
-  VPReplicateRecipe Recipe(nullptr, make_range(Args.begin(), Args.end()), true,
-                           false);
+  VPReplicateRecipe Recipe(nullptr, make_range(Args.begin(), Args.end()), true);
   EXPECT_TRUE(isa<VPUser>(&Recipe));
   VPRecipeBase *BaseR = &Recipe;
   EXPECT_TRUE(isa<VPUser>(BaseR));
@@ -1048,8 +1047,7 @@ TEST(VPRecipeTest, MayHaveSideEffectsAndMayReadWriteMemory) {
     Args.push_back(&Op1);
     Args.push_back(&Op2);
     Args.push_back(&Op3);
-    VPWidenSelectRecipe Recipe(*SelectI, make_range(Args.begin(), Args.end()),
-                               false);
+    VPWidenSelectRecipe Recipe(*SelectI, make_range(Args.begin(), Args.end()));
     EXPECT_FALSE(Recipe.mayHaveSideEffects());
     EXPECT_FALSE(Recipe.mayReadFromMemory());
     EXPECT_FALSE(Recipe.mayWriteToMemory());
diff --git a/llvm/unittests/tools/llvm-profdata/OutputSizeLimitTest.cpp b/llvm/unittests/tools/llvm-profdata/OutputSizeLimitTest.cpp
index c4f8248358969..25da748865ec7 100644
--- a/llvm/unittests/tools/llvm-profdata/OutputSizeLimitTest.cpp
+++ b/llvm/unittests/tools/llvm-profdata/OutputSizeLimitTest.cpp
@@ -77,7 +77,8 @@ template <typename T> class ExpectedErrorOr : public Expected<T> {
 /// Note that even though by default samples with fewest total count are dropped
 /// first, this is not a requirement. Samples can be dropped by any order.
 static ExpectedErrorOr<void *> RunTest(StringRef Input, size_t SizeLimit,
-                                       SampleProfileFormat Format) {
+                                       SampleProfileFormat Format,
+                                       bool Compress = false) {
   // Read Input profile.
   auto FS = vfs::getRealFileSystem();
   LLVMContext Context;
@@ -93,6 +94,8 @@ static ExpectedErrorOr<void *> RunTest(StringRef Input, size_t SizeLimit,
   {
     DEF_VAR_RETURN_IF_ERROR(Writer,
                             SampleProfileWriter::create(Temp.path(), Format));
+    if (Compress)
+      Writer->setToCompressAllSections();
     std::error_code EC = Writer->writeWithSizeLimit(OldProfiles, SizeLimit);
     // too_large means no sample could be written because SizeLimit is too
     // small. Otherwise any other error code indicates unexpected failure.
@@ -180,3 +183,13 @@ TEST(TestOutputSizeLimit, TestOutputSizeLimitText) {
         RunTest(Input1, OutputSizeLimit, llvm::sampleprof::SPF_Text),
         Succeeded());
 }
+
+#if LLVM_ENABLE_ZLIB
+TEST(TestOutputSizeLimit, TestOutputSizeLimitExtBinaryCompressed) {
+  for (size_t OutputSizeLimit :
+       {507, 506, 505, 494, 493, 492, 483, 482, 481, 480})
+    ASSERT_THAT_EXPECTED(RunTest(Input1, OutputSizeLimit,
+                                 llvm::sampleprof::SPF_Ext_Binary, true),
+                         Succeeded());
+}
+#endif
diff --git a/llvm/utils/TableGen/CMakeLists.txt b/llvm/utils/TableGen/CMakeLists.txt
index 44ed5549c8ea3..42f20a6b7164d 100644
--- a/llvm/utils/TableGen/CMakeLists.txt
+++ b/llvm/utils/TableGen/CMakeLists.txt
@@ -15,6 +15,7 @@ add_tablegen(llvm-tblgen LLVM
   CodeGenHwModes.cpp
   CodeGenInstAlias.cpp
   CodeGenInstruction.cpp
+  CodeGenIntrinsics.cpp
   CodeGenMapTable.cpp
   CodeGenRegisters.cpp
   CodeGenSchedule.cpp
diff --git a/llvm/utils/TableGen/CodeEmitterGen.cpp b/llvm/utils/TableGen/CodeEmitterGen.cpp
index 1dd076a437766..11a57bb1bc310 100644
--- a/llvm/utils/TableGen/CodeEmitterGen.cpp
+++ b/llvm/utils/TableGen/CodeEmitterGen.cpp
@@ -51,8 +51,7 @@ class CodeEmitterGen {
   std::string getInstructionCaseForEncoding(Record *R, Record *EncodingDef,
                                             CodeGenTarget &Target);
   bool addCodeToMergeInOperand(Record *R, BitsInit *BI,
-                               const std::string &VarName, unsigned &NumberedOp,
-                               std::set<unsigned> &NamedOpIndices,
+                               const std::string &VarName,
                                std::string &Case, CodeGenTarget &Target);
 
   void emitInstructionBaseValues(
@@ -81,8 +80,6 @@ int CodeEmitterGen::getVariableBit(const std::string &VarName,
 // Returns true if it succeeds, false if an error.
 bool CodeEmitterGen::addCodeToMergeInOperand(Record *R, BitsInit *BI,
                                              const std::string &VarName,
-                                             unsigned &NumberedOp,
-                                             std::set<unsigned> &NamedOpIndices,
                                              std::string &Case,
                                              CodeGenTarget &Target) {
   CodeGenInstruction &CGI = Target.getInstruction(R);
@@ -114,52 +111,8 @@ bool CodeEmitterGen::addCodeToMergeInOperand(Record *R, BitsInit *BI,
     // Get the machine operand number for the indicated operand.
     OpIdx = CGI.Operands[OpIdx].MIOperandNo;
   } else {
-    // Fall back to positional lookup. By default, we now disable positional
-    // lookup (and print an error, below), but even so, we'll do the lookup to
-    // help print a helpful diagnostic message.
-    //
-    // TODO: When we remove useDeprecatedPositionallyEncodedOperands, delete all
-    // this code, just leaving a "no operand named X in record Y" error.
-
-    unsigned NumberOps = CGI.Operands.size();
-    /// If this operand is not supposed to be emitted by the
-    /// generated emitter, skip it.
-    while (NumberedOp < NumberOps &&
-           (CGI.Operands.isFlatOperandNotEmitted(NumberedOp) ||
-              (!NamedOpIndices.empty() && NamedOpIndices.count(
-                CGI.Operands.getSubOperandNumber(NumberedOp).first)))) {
-      ++NumberedOp;
-    }
-
-    if (NumberedOp >=
-        CGI.Operands.back().MIOperandNo + CGI.Operands.back().MINumOperands) {
-      if (!Target.getInstructionSet()->getValueAsBit(
-              "useDeprecatedPositionallyEncodedOperands")) {
-        PrintError(R, Twine("No operand named ") + VarName + " in record " +
-                          R->getName() +
-                          " (would've given 'too few operands' error with "
-                          "useDeprecatedPositionallyEncodedOperands=true)");
-      } else {
-        PrintError(R, "Too few operands in record " + R->getName() +
-                          " (no match for variable " + VarName + ")");
-      }
-      return false;
-    }
-
-    OpIdx = NumberedOp++;
-
-    if (!Target.getInstructionSet()->getValueAsBit(
-            "useDeprecatedPositionallyEncodedOperands")) {
-      std::pair<unsigned, unsigned> SO =
-          CGI.Operands.getSubOperandNumber(OpIdx);
-      std::string OpName = CGI.Operands[SO.first].Name;
-      PrintError(R, Twine("No operand named ") + VarName + " in record " +
-                        R->getName() + " (would've used positional operand #" +
-                        Twine(SO.first) + " ('" + OpName + "') sub-op #" +
-                        Twine(SO.second) +
-                        " with useDeprecatedPositionallyEncodedOperands=true)");
-      return false;
-    }
+    PrintError(R, Twine("No operand named ") + VarName + " in record " + R->getName());
+    return false;
   }
 
   if (CGI.Operands.isFlatOperandNotEmitted(OpIdx)) {
@@ -320,22 +273,6 @@ std::string CodeEmitterGen::getInstructionCaseForEncoding(Record *R, Record *Enc
                                                           CodeGenTarget &Target) {
   std::string Case;
   BitsInit *BI = EncodingDef->getValueAsBitsInit("Inst");
-  unsigned NumberedOp = 0;
-  std::set<unsigned> NamedOpIndices;
-
-  // Collect the set of operand indices that might correspond to named
-  // operand, and skip these when assigning operands based on position.
-  if (Target.getInstructionSet()->
-       getValueAsBit("noNamedPositionallyEncodedOperands")) {
-    CodeGenInstruction &CGI = Target.getInstruction(R);
-    for (const RecordVal &RV : R->getValues()) {
-      unsigned OpIdx;
-      if (!CGI.Operands.hasOperandNamed(RV.getName(), OpIdx))
-        continue;
-
-      NamedOpIndices.insert(OpIdx);
-    }
-  }
 
   // Loop over all of the fields in the instruction, determining which are the
   // operands to the instruction.
@@ -347,8 +284,8 @@ std::string CodeEmitterGen::getInstructionCaseForEncoding(Record *R, Record *Enc
       continue;
 
     Success &=
-        addCodeToMergeInOperand(R, BI, std::string(RV.getName()), NumberedOp,
-                                NamedOpIndices, Case, Target);
+        addCodeToMergeInOperand(R, BI, std::string(RV.getName()),
+                                Case, Target);
   }
 
   if (!Success) {
diff --git a/llvm/utils/TableGen/CodeGenDAGPatterns.cpp b/llvm/utils/TableGen/CodeGenDAGPatterns.cpp
index 56cf9c797c7aa..8b9cfee36a679 100644
--- a/llvm/utils/TableGen/CodeGenDAGPatterns.cpp
+++ b/llvm/utils/TableGen/CodeGenDAGPatterns.cpp
@@ -2000,9 +2000,8 @@ bool TreePatternNode::isIsomorphicTo(const TreePatternNode *N,
   if (isLeaf()) {
     if (DefInit *DI = dyn_cast<DefInit>(getLeafValue())) {
       if (DefInit *NDI = dyn_cast<DefInit>(N->getLeafValue())) {
-        return ((DI->getDef() == NDI->getDef())
-                && (DepVars.find(getName()) == DepVars.end()
-                    || getName() == N->getName()));
+        return ((DI->getDef() == NDI->getDef()) &&
+                (!DepVars.contains(getName()) || getName() == N->getName()));
       }
     }
     return getLeafValue() == N->getLeafValue();
@@ -4451,14 +4450,14 @@ void CodeGenDAGPatterns::ExpandHwModeBasedTypes() {
 
       // Fill the map entry for this mode.
       const HwMode &HM = CGH.getMode(M);
-      AppendPattern(P, M, "(MF->getSubtarget().checkFeatures(\"" + HM.Features + "\"))");
+      AppendPattern(P, M, HM.Predicates);
 
       // Add negations of the HM's predicates to the default predicate.
       if (!DefaultCheck.empty())
         DefaultCheck += " && ";
-      DefaultCheck += "(!(MF->getSubtarget().checkFeatures(\"";
-      DefaultCheck += HM.Features;
-      DefaultCheck += "\")))";
+      DefaultCheck += "!(";
+      DefaultCheck += HM.Predicates;
+      DefaultCheck += ")";
     }
 
     bool HasDefault = Modes.count(DefaultMode);
diff --git a/llvm/utils/TableGen/CodeGenHwModes.cpp b/llvm/utils/TableGen/CodeGenHwModes.cpp
index 9c5d5268516c3..99a97e89e60c5 100644
--- a/llvm/utils/TableGen/CodeGenHwModes.cpp
+++ b/llvm/utils/TableGen/CodeGenHwModes.cpp
@@ -21,6 +21,19 @@ StringRef CodeGenHwModes::DefaultModeName = "DefaultMode";
 HwMode::HwMode(Record *R) {
   Name = R->getName();
   Features = std::string(R->getValueAsString("Features"));
+
+  std::vector<Record *> PredicateRecs = R->getValueAsListOfDefs("Predicates");
+  SmallString<128> PredicateCheck;
+  raw_svector_ostream OS(PredicateCheck);
+  ListSeparator LS(" && ");
+  for (Record *Pred : PredicateRecs) {
+    StringRef CondString = Pred->getValueAsString("CondString");
+    if (CondString.empty())
+      continue;
+    OS << LS << '(' << CondString << ')';
+  }
+
+  Predicates = std::string(PredicateCheck);
 }
 
 LLVM_DUMP_METHOD
diff --git a/llvm/utils/TableGen/CodeGenHwModes.h b/llvm/utils/TableGen/CodeGenHwModes.h
index 7923df1fc0a5f..09d20ad85c5ec 100644
--- a/llvm/utils/TableGen/CodeGenHwModes.h
+++ b/llvm/utils/TableGen/CodeGenHwModes.h
@@ -31,6 +31,7 @@ namespace llvm {
     HwMode(Record *R);
     StringRef Name;
     std::string Features;
+    std::string Predicates;
     void dump() const;
   };
 
diff --git a/llvm/utils/TableGen/CodeGenIntrinsics.cpp b/llvm/utils/TableGen/CodeGenIntrinsics.cpp
new file mode 100644
index 0000000000000..73e40c7dd6b80
--- /dev/null
+++ b/llvm/utils/TableGen/CodeGenIntrinsics.cpp
@@ -0,0 +1,345 @@
+//===- CodeGenIntrinsics.cpp - Intrinsic Class Wrapper --------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines a wrapper class for the 'Intrinsic' TableGen class.
+//
+//===----------------------------------------------------------------------===//
+
+#include "CodeGenIntrinsics.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/Twine.h"
+#include "llvm/Support/ErrorHandling.h"
+#include "llvm/TableGen/Error.h"
+#include "llvm/TableGen/Record.h"
+#include <algorithm>
+#include <cassert>
+using namespace llvm;
+using namespace llvm::tmp;
+
+/// getValueType - Return the MVT::SimpleValueType that the specified TableGen
+/// record corresponds to.
+MVT::SimpleValueType llvm::tmp::getValueType(Record *Rec) {
+  return (MVT::SimpleValueType)Rec->getValueAsInt("Value");
+}
+
+//===----------------------------------------------------------------------===//
+// CodeGenIntrinsic Implementation
+//===----------------------------------------------------------------------===//
+
+CodeGenIntrinsicTable::CodeGenIntrinsicTable(const RecordKeeper &RC) {
+  std::vector<Record *> IntrProperties =
+      RC.getAllDerivedDefinitions("IntrinsicProperty");
+
+  std::vector<Record *> DefaultProperties;
+  for (Record *Rec : IntrProperties)
+    if (Rec->getValueAsBit("IsDefault"))
+      DefaultProperties.push_back(Rec);
+
+  std::vector<Record *> Defs = RC.getAllDerivedDefinitions("Intrinsic");
+  Intrinsics.reserve(Defs.size());
+
+  for (unsigned I = 0, e = Defs.size(); I != e; ++I)
+    Intrinsics.push_back(CodeGenIntrinsic(Defs[I], DefaultProperties));
+
+  llvm::sort(Intrinsics,
+             [](const CodeGenIntrinsic &LHS, const CodeGenIntrinsic &RHS) {
+               return std::tie(LHS.TargetPrefix, LHS.Name) <
+                      std::tie(RHS.TargetPrefix, RHS.Name);
+             });
+  Targets.push_back({"", 0, 0});
+  for (size_t I = 0, E = Intrinsics.size(); I < E; ++I)
+    if (Intrinsics[I].TargetPrefix != Targets.back().Name) {
+      Targets.back().Count = I - Targets.back().Offset;
+      Targets.push_back({Intrinsics[I].TargetPrefix, I, 0});
+    }
+  Targets.back().Count = Intrinsics.size() - Targets.back().Offset;
+}
+
+CodeGenIntrinsic::CodeGenIntrinsic(Record *R,
+                                   std::vector<Record *> DefaultProperties) {
+  TheDef = R;
+  std::string DefName = std::string(R->getName());
+  ArrayRef<SMLoc> DefLoc = R->getLoc();
+  Properties = 0;
+  isOverloaded = false;
+  isCommutative = false;
+  canThrow = false;
+  isNoReturn = false;
+  isNoCallback = false;
+  isNoSync = false;
+  isNoFree = false;
+  isWillReturn = false;
+  isCold = false;
+  isNoDuplicate = false;
+  isNoMerge = false;
+  isConvergent = false;
+  isSpeculatable = false;
+  hasSideEffects = false;
+
+  if (DefName.size() <= 4 || DefName.substr(0, 4) != "int_")
+    PrintFatalError(DefLoc,
+                    "Intrinsic '" + DefName + "' does not start with 'int_'!");
+
+  EnumName = DefName.substr(4);
+
+  if (R->getValue(
+          "ClangBuiltinName")) // Ignore a missing ClangBuiltinName field.
+    ClangBuiltinName = std::string(R->getValueAsString("ClangBuiltinName"));
+  if (R->getValue("MSBuiltinName")) // Ignore a missing MSBuiltinName field.
+    MSBuiltinName = std::string(R->getValueAsString("MSBuiltinName"));
+
+  TargetPrefix = std::string(R->getValueAsString("TargetPrefix"));
+  Name = std::string(R->getValueAsString("LLVMName"));
+
+  if (Name == "") {
+    // If an explicit name isn't specified, derive one from the DefName.
+    Name = "llvm.";
+
+    for (unsigned i = 0, e = EnumName.size(); i != e; ++i)
+      Name += (EnumName[i] == '_') ? '.' : EnumName[i];
+  } else {
+    // Verify it starts with "llvm.".
+    if (Name.size() <= 5 || Name.substr(0, 5) != "llvm.")
+      PrintFatalError(DefLoc, "Intrinsic '" + DefName +
+                                  "'s name does not start with 'llvm.'!");
+  }
+
+  // If TargetPrefix is specified, make sure that Name starts with
+  // "llvm.<targetprefix>.".
+  if (!TargetPrefix.empty()) {
+    if (Name.size() < 6 + TargetPrefix.size() ||
+        Name.substr(5, 1 + TargetPrefix.size()) != (TargetPrefix + "."))
+      PrintFatalError(DefLoc, "Intrinsic '" + DefName +
+                                  "' does not start with 'llvm." +
+                                  TargetPrefix + ".'!");
+  }
+
+  ListInit *RetTypes = R->getValueAsListInit("RetTypes");
+  ListInit *ParamTypes = R->getValueAsListInit("ParamTypes");
+
+  // First collate a list of overloaded types.
+  std::vector<MVT::SimpleValueType> OverloadedVTs;
+  for (ListInit *TypeList : {RetTypes, ParamTypes}) {
+    for (unsigned i = 0, e = TypeList->size(); i != e; ++i) {
+      Record *TyEl = TypeList->getElementAsRecord(i);
+      assert(TyEl->isSubClassOf("LLVMType") && "Expected a type!");
+
+      if (TyEl->isSubClassOf("LLVMMatchType"))
+        continue;
+
+      MVT::SimpleValueType VT = getValueType(TyEl->getValueAsDef("VT"));
+      if (MVT(VT).isOverloaded()) {
+        OverloadedVTs.push_back(VT);
+        isOverloaded = true;
+      }
+    }
+  }
+
+  // Parse the list of return types.
+  ListInit *TypeList = RetTypes;
+  for (unsigned i = 0, e = TypeList->size(); i != e; ++i) {
+    Record *TyEl = TypeList->getElementAsRecord(i);
+    assert(TyEl->isSubClassOf("LLVMType") && "Expected a type!");
+    MVT::SimpleValueType VT;
+    if (TyEl->isSubClassOf("LLVMMatchType")) {
+      unsigned MatchTy = TyEl->getValueAsInt("Number");
+      assert(MatchTy < OverloadedVTs.size() && "Invalid matching number!");
+      VT = OverloadedVTs[MatchTy];
+      // It only makes sense to use the extended and truncated vector element
+      // variants with iAny types; otherwise, if the intrinsic is not
+      // overloaded, all the types can be specified directly.
+      assert(((!TyEl->isSubClassOf("LLVMExtendedType") &&
+               !TyEl->isSubClassOf("LLVMTruncatedType")) ||
+              VT == MVT::iAny || VT == MVT::vAny) &&
+             "Expected iAny or vAny type");
+    } else {
+      VT = getValueType(TyEl->getValueAsDef("VT"));
+    }
+
+    // Reject invalid types.
+    if (VT == MVT::isVoid)
+      PrintFatalError(DefLoc, "Intrinsic '" + DefName +
+                                  " has void in result type list!");
+
+    IS.RetVTs.push_back(VT);
+    IS.RetTypeDefs.push_back(TyEl);
+  }
+
+  // Parse the list of parameter types.
+  TypeList = ParamTypes;
+  for (unsigned i = 0, e = TypeList->size(); i != e; ++i) {
+    Record *TyEl = TypeList->getElementAsRecord(i);
+    assert(TyEl->isSubClassOf("LLVMType") && "Expected a type!");
+    MVT::SimpleValueType VT;
+    if (TyEl->isSubClassOf("LLVMMatchType")) {
+      unsigned MatchTy = TyEl->getValueAsInt("Number");
+      if (MatchTy >= OverloadedVTs.size()) {
+        PrintError(R->getLoc(), "Parameter #" + Twine(i) +
+                                    " has out of bounds matching "
+                                    "number " +
+                                    Twine(MatchTy));
+        PrintFatalError(DefLoc,
+                        Twine("ParamTypes is ") + TypeList->getAsString());
+      }
+      VT = OverloadedVTs[MatchTy];
+      // It only makes sense to use the extended and truncated vector element
+      // variants with iAny types; otherwise, if the intrinsic is not
+      // overloaded, all the types can be specified directly.
+      assert(((!TyEl->isSubClassOf("LLVMExtendedType") &&
+               !TyEl->isSubClassOf("LLVMTruncatedType")) ||
+              VT == MVT::iAny || VT == MVT::vAny) &&
+             "Expected iAny or vAny type");
+    } else
+      VT = getValueType(TyEl->getValueAsDef("VT"));
+
+    // Reject invalid types.
+    if (VT == MVT::isVoid && i != e - 1 /*void at end means varargs*/)
+      PrintFatalError(DefLoc, "Intrinsic '" + DefName +
+                                  " has void in result type list!");
+
+    IS.ParamVTs.push_back(VT);
+    IS.ParamTypeDefs.push_back(TyEl);
+  }
+
+  // Parse the intrinsic properties.
+  ListInit *PropList = R->getValueAsListInit("IntrProperties");
+  for (unsigned i = 0, e = PropList->size(); i != e; ++i) {
+    Record *Property = PropList->getElementAsRecord(i);
+    assert(Property->isSubClassOf("IntrinsicProperty") &&
+           "Expected a property!");
+
+    setProperty(Property);
+  }
+
+  // Set default properties to true.
+  setDefaultProperties(R, DefaultProperties);
+
+  // Also record the SDPatternOperator Properties.
+  Properties = parseSDPatternOperatorProperties(R);
+
+  // Sort the argument attributes for later benefit.
+  for (auto &Attrs : ArgumentAttributes)
+    llvm::sort(Attrs);
+}
+
+void CodeGenIntrinsic::setDefaultProperties(
+    Record *R, std::vector<Record *> DefaultProperties) {
+  // opt-out of using default attributes.
+  if (R->getValueAsBit("DisableDefaultAttributes"))
+    return;
+
+  for (Record *Rec : DefaultProperties)
+    setProperty(Rec);
+}
+
+void CodeGenIntrinsic::setProperty(Record *R) {
+  if (R->getName() == "IntrNoMem")
+    ME = MemoryEffects::none();
+  else if (R->getName() == "IntrReadMem") {
+    if (ME.onlyWritesMemory())
+      PrintFatalError(TheDef->getLoc(),
+                      Twine("IntrReadMem cannot be used after IntrNoMem or "
+                            "IntrWriteMem. Default is ReadWrite"));
+    ME &= MemoryEffects::readOnly();
+  } else if (R->getName() == "IntrWriteMem") {
+    if (ME.onlyReadsMemory())
+      PrintFatalError(TheDef->getLoc(),
+                      Twine("IntrWriteMem cannot be used after IntrNoMem or "
+                            "IntrReadMem. Default is ReadWrite"));
+    ME &= MemoryEffects::writeOnly();
+  } else if (R->getName() == "IntrArgMemOnly")
+    ME &= MemoryEffects::argMemOnly();
+  else if (R->getName() == "IntrInaccessibleMemOnly")
+    ME &= MemoryEffects::inaccessibleMemOnly();
+  else if (R->getName() == "IntrInaccessibleMemOrArgMemOnly")
+    ME &= MemoryEffects::inaccessibleOrArgMemOnly();
+  else if (R->getName() == "Commutative")
+    isCommutative = true;
+  else if (R->getName() == "Throws")
+    canThrow = true;
+  else if (R->getName() == "IntrNoDuplicate")
+    isNoDuplicate = true;
+  else if (R->getName() == "IntrNoMerge")
+    isNoMerge = true;
+  else if (R->getName() == "IntrConvergent")
+    isConvergent = true;
+  else if (R->getName() == "IntrNoReturn")
+    isNoReturn = true;
+  else if (R->getName() == "IntrNoCallback")
+    isNoCallback = true;
+  else if (R->getName() == "IntrNoSync")
+    isNoSync = true;
+  else if (R->getName() == "IntrNoFree")
+    isNoFree = true;
+  else if (R->getName() == "IntrWillReturn")
+    isWillReturn = !isNoReturn;
+  else if (R->getName() == "IntrCold")
+    isCold = true;
+  else if (R->getName() == "IntrSpeculatable")
+    isSpeculatable = true;
+  else if (R->getName() == "IntrHasSideEffects")
+    hasSideEffects = true;
+  else if (R->isSubClassOf("NoCapture")) {
+    unsigned ArgNo = R->getValueAsInt("ArgNo");
+    addArgAttribute(ArgNo, NoCapture);
+  } else if (R->isSubClassOf("NoAlias")) {
+    unsigned ArgNo = R->getValueAsInt("ArgNo");
+    addArgAttribute(ArgNo, NoAlias);
+  } else if (R->isSubClassOf("NoUndef")) {
+    unsigned ArgNo = R->getValueAsInt("ArgNo");
+    addArgAttribute(ArgNo, NoUndef);
+  } else if (R->isSubClassOf("NonNull")) {
+    unsigned ArgNo = R->getValueAsInt("ArgNo");
+    addArgAttribute(ArgNo, NonNull);
+  } else if (R->isSubClassOf("Returned")) {
+    unsigned ArgNo = R->getValueAsInt("ArgNo");
+    addArgAttribute(ArgNo, Returned);
+  } else if (R->isSubClassOf("ReadOnly")) {
+    unsigned ArgNo = R->getValueAsInt("ArgNo");
+    addArgAttribute(ArgNo, ReadOnly);
+  } else if (R->isSubClassOf("WriteOnly")) {
+    unsigned ArgNo = R->getValueAsInt("ArgNo");
+    addArgAttribute(ArgNo, WriteOnly);
+  } else if (R->isSubClassOf("ReadNone")) {
+    unsigned ArgNo = R->getValueAsInt("ArgNo");
+    addArgAttribute(ArgNo, ReadNone);
+  } else if (R->isSubClassOf("ImmArg")) {
+    unsigned ArgNo = R->getValueAsInt("ArgNo");
+    addArgAttribute(ArgNo, ImmArg);
+  } else if (R->isSubClassOf("Align")) {
+    unsigned ArgNo = R->getValueAsInt("ArgNo");
+    uint64_t Align = R->getValueAsInt("Align");
+    addArgAttribute(ArgNo, Alignment, Align);
+  } else
+    llvm_unreachable("Unknown property!");
+}
+
+bool CodeGenIntrinsic::isParamAPointer(unsigned ParamIdx) const {
+  if (ParamIdx >= IS.ParamVTs.size())
+    return false;
+  MVT ParamType = MVT(IS.ParamVTs[ParamIdx]);
+  return ParamType == MVT::iPTR || ParamType == MVT::iPTRAny;
+}
+
+bool CodeGenIntrinsic::isParamImmArg(unsigned ParamIdx) const {
+  // Convert argument index to attribute index starting from `FirstArgIndex`.
+  ++ParamIdx;
+  if (ParamIdx >= ArgumentAttributes.size())
+    return false;
+  ArgAttribute Val{ImmArg, 0};
+  return std::binary_search(ArgumentAttributes[ParamIdx].begin(),
+                            ArgumentAttributes[ParamIdx].end(), Val);
+}
+
+void CodeGenIntrinsic::addArgAttribute(unsigned Idx, ArgAttrKind AK,
+                                       uint64_t V) {
+  if (Idx >= ArgumentAttributes.size())
+    ArgumentAttributes.resize(Idx + 1);
+  ArgumentAttributes[Idx].emplace_back(AK, V);
+}
diff --git a/llvm/utils/TableGen/CodeGenIntrinsics.h b/llvm/utils/TableGen/CodeGenIntrinsics.h
index 2278216de137e..d704b0e531d3c 100644
--- a/llvm/utils/TableGen/CodeGenIntrinsics.h
+++ b/llvm/utils/TableGen/CodeGenIntrinsics.h
@@ -25,6 +25,13 @@ namespace llvm {
 class Record;
 class RecordKeeper;
 
+// FIXME: Sweep this in the near future.
+namespace tmp {
+/// getValueType - Return the MVT::SimpleValueType that the specified TableGen
+/// record corresponds to.
+MVT::SimpleValueType getValueType(Record *Rec);
+} // namespace tmp
+
 struct CodeGenIntrinsic {
   Record *TheDef;             // The actual record defining this intrinsic.
   std::string Name;           // The name of the LLVM function "llvm.bswap.i32"
diff --git a/llvm/utils/TableGen/CodeGenSchedule.cpp b/llvm/utils/TableGen/CodeGenSchedule.cpp
index 8c0dac30df4d3..04219a6e54d95 100644
--- a/llvm/utils/TableGen/CodeGenSchedule.cpp
+++ b/llvm/utils/TableGen/CodeGenSchedule.cpp
@@ -298,12 +298,12 @@ processSTIPredicate(STIPredicateFunction &Fn,
     RecVec Classes = Def->getValueAsListOfDefs("Classes");
     for (const Record *EC : Classes) {
       const Record *Pred = EC->getValueAsDef("Predicate");
-      if (Predicate2Index.find(Pred) == Predicate2Index.end())
+      if (!Predicate2Index.contains(Pred))
         Predicate2Index[Pred] = NumUniquePredicates++;
 
       RecVec Opcodes = EC->getValueAsListOfDefs("Opcodes");
       for (const Record *Opcode : Opcodes) {
-        if (Opcode2Index.find(Opcode) == Opcode2Index.end()) {
+        if (!Opcode2Index.contains(Opcode)) {
           Opcode2Index[Opcode] = OpcodeMappings.size();
           OpcodeMappings.emplace_back(Opcode, OpcodeInfo());
         }
diff --git a/llvm/utils/TableGen/CodeGenTarget.cpp b/llvm/utils/TableGen/CodeGenTarget.cpp
index 672728ae16c20..ca8ce48c31034 100644
--- a/llvm/utils/TableGen/CodeGenTarget.cpp
+++ b/llvm/utils/TableGen/CodeGenTarget.cpp
@@ -15,13 +15,11 @@
 
 #include "CodeGenTarget.h"
 #include "CodeGenInstruction.h"
-#include "CodeGenIntrinsics.h"
 #include "CodeGenSchedule.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ErrorHandling.h"
-#include "llvm/Support/ModRef.h"
 #include "llvm/TableGen/Error.h"
 #include "llvm/TableGen/Record.h"
 #include <algorithm>
@@ -641,319 +639,3 @@ ComplexPattern::ComplexPattern(Record *R) {
                                        "'!");
     }
 }
-
-//===----------------------------------------------------------------------===//
-// CodeGenIntrinsic Implementation
-//===----------------------------------------------------------------------===//
-
-CodeGenIntrinsicTable::CodeGenIntrinsicTable(const RecordKeeper &RC) {
-  std::vector<Record *> IntrProperties =
-      RC.getAllDerivedDefinitions("IntrinsicProperty");
-
-  std::vector<Record *> DefaultProperties;
-  for (Record *Rec : IntrProperties)
-    if (Rec->getValueAsBit("IsDefault"))
-      DefaultProperties.push_back(Rec);
-
-  std::vector<Record *> Defs = RC.getAllDerivedDefinitions("Intrinsic");
-  Intrinsics.reserve(Defs.size());
-
-  for (unsigned I = 0, e = Defs.size(); I != e; ++I)
-    Intrinsics.push_back(CodeGenIntrinsic(Defs[I], DefaultProperties));
-
-  llvm::sort(Intrinsics,
-             [](const CodeGenIntrinsic &LHS, const CodeGenIntrinsic &RHS) {
-               return std::tie(LHS.TargetPrefix, LHS.Name) <
-                      std::tie(RHS.TargetPrefix, RHS.Name);
-             });
-  Targets.push_back({"", 0, 0});
-  for (size_t I = 0, E = Intrinsics.size(); I < E; ++I)
-    if (Intrinsics[I].TargetPrefix != Targets.back().Name) {
-      Targets.back().Count = I - Targets.back().Offset;
-      Targets.push_back({Intrinsics[I].TargetPrefix, I, 0});
-    }
-  Targets.back().Count = Intrinsics.size() - Targets.back().Offset;
-}
-
-CodeGenIntrinsic::CodeGenIntrinsic(Record *R,
-                                   std::vector<Record *> DefaultProperties) {
-  TheDef = R;
-  std::string DefName = std::string(R->getName());
-  ArrayRef<SMLoc> DefLoc = R->getLoc();
-  Properties = 0;
-  isOverloaded = false;
-  isCommutative = false;
-  canThrow = false;
-  isNoReturn = false;
-  isNoCallback = false;
-  isNoSync = false;
-  isNoFree = false;
-  isWillReturn = false;
-  isCold = false;
-  isNoDuplicate = false;
-  isNoMerge = false;
-  isConvergent = false;
-  isSpeculatable = false;
-  hasSideEffects = false;
-
-  if (DefName.size() <= 4 || DefName.substr(0, 4) != "int_")
-    PrintFatalError(DefLoc,
-                    "Intrinsic '" + DefName + "' does not start with 'int_'!");
-
-  EnumName = DefName.substr(4);
-
-  if (R->getValue(
-          "ClangBuiltinName")) // Ignore a missing ClangBuiltinName field.
-    ClangBuiltinName = std::string(R->getValueAsString("ClangBuiltinName"));
-  if (R->getValue("MSBuiltinName")) // Ignore a missing MSBuiltinName field.
-    MSBuiltinName = std::string(R->getValueAsString("MSBuiltinName"));
-
-  TargetPrefix = std::string(R->getValueAsString("TargetPrefix"));
-  Name = std::string(R->getValueAsString("LLVMName"));
-
-  if (Name == "") {
-    // If an explicit name isn't specified, derive one from the DefName.
-    Name = "llvm.";
-
-    for (unsigned i = 0, e = EnumName.size(); i != e; ++i)
-      Name += (EnumName[i] == '_') ? '.' : EnumName[i];
-  } else {
-    // Verify it starts with "llvm.".
-    if (Name.size() <= 5 || Name.substr(0, 5) != "llvm.")
-      PrintFatalError(DefLoc, "Intrinsic '" + DefName +
-                                  "'s name does not start with 'llvm.'!");
-  }
-
-  // If TargetPrefix is specified, make sure that Name starts with
-  // "llvm.<targetprefix>.".
-  if (!TargetPrefix.empty()) {
-    if (Name.size() < 6 + TargetPrefix.size() ||
-        Name.substr(5, 1 + TargetPrefix.size()) != (TargetPrefix + "."))
-      PrintFatalError(DefLoc, "Intrinsic '" + DefName +
-                                  "' does not start with 'llvm." +
-                                  TargetPrefix + ".'!");
-  }
-
-  ListInit *RetTypes = R->getValueAsListInit("RetTypes");
-  ListInit *ParamTypes = R->getValueAsListInit("ParamTypes");
-
-  // First collate a list of overloaded types.
-  std::vector<MVT::SimpleValueType> OverloadedVTs;
-  for (ListInit *TypeList : {RetTypes, ParamTypes}) {
-    for (unsigned i = 0, e = TypeList->size(); i != e; ++i) {
-      Record *TyEl = TypeList->getElementAsRecord(i);
-      assert(TyEl->isSubClassOf("LLVMType") && "Expected a type!");
-
-      if (TyEl->isSubClassOf("LLVMMatchType"))
-        continue;
-
-      MVT::SimpleValueType VT = getValueType(TyEl->getValueAsDef("VT"));
-      if (MVT(VT).isOverloaded()) {
-        OverloadedVTs.push_back(VT);
-        isOverloaded = true;
-      }
-    }
-  }
-
-  // Parse the list of return types.
-  ListInit *TypeList = RetTypes;
-  for (unsigned i = 0, e = TypeList->size(); i != e; ++i) {
-    Record *TyEl = TypeList->getElementAsRecord(i);
-    assert(TyEl->isSubClassOf("LLVMType") && "Expected a type!");
-    MVT::SimpleValueType VT;
-    if (TyEl->isSubClassOf("LLVMMatchType")) {
-      unsigned MatchTy = TyEl->getValueAsInt("Number");
-      assert(MatchTy < OverloadedVTs.size() && "Invalid matching number!");
-      VT = OverloadedVTs[MatchTy];
-      // It only makes sense to use the extended and truncated vector element
-      // variants with iAny types; otherwise, if the intrinsic is not
-      // overloaded, all the types can be specified directly.
-      assert(((!TyEl->isSubClassOf("LLVMExtendedType") &&
-               !TyEl->isSubClassOf("LLVMTruncatedType")) ||
-              VT == MVT::iAny || VT == MVT::vAny) &&
-             "Expected iAny or vAny type");
-    } else {
-      VT = getValueType(TyEl->getValueAsDef("VT"));
-    }
-
-    // Reject invalid types.
-    if (VT == MVT::isVoid)
-      PrintFatalError(DefLoc, "Intrinsic '" + DefName +
-                                  " has void in result type list!");
-
-    IS.RetVTs.push_back(VT);
-    IS.RetTypeDefs.push_back(TyEl);
-  }
-
-  // Parse the list of parameter types.
-  TypeList = ParamTypes;
-  for (unsigned i = 0, e = TypeList->size(); i != e; ++i) {
-    Record *TyEl = TypeList->getElementAsRecord(i);
-    assert(TyEl->isSubClassOf("LLVMType") && "Expected a type!");
-    MVT::SimpleValueType VT;
-    if (TyEl->isSubClassOf("LLVMMatchType")) {
-      unsigned MatchTy = TyEl->getValueAsInt("Number");
-      if (MatchTy >= OverloadedVTs.size()) {
-        PrintError(R->getLoc(), "Parameter #" + Twine(i) +
-                                    " has out of bounds matching "
-                                    "number " +
-                                    Twine(MatchTy));
-        PrintFatalError(DefLoc,
-                        Twine("ParamTypes is ") + TypeList->getAsString());
-      }
-      VT = OverloadedVTs[MatchTy];
-      // It only makes sense to use the extended and truncated vector element
-      // variants with iAny types; otherwise, if the intrinsic is not
-      // overloaded, all the types can be specified directly.
-      assert(((!TyEl->isSubClassOf("LLVMExtendedType") &&
-               !TyEl->isSubClassOf("LLVMTruncatedType")) ||
-              VT == MVT::iAny || VT == MVT::vAny) &&
-             "Expected iAny or vAny type");
-    } else
-      VT = getValueType(TyEl->getValueAsDef("VT"));
-
-    // Reject invalid types.
-    if (VT == MVT::isVoid && i != e - 1 /*void at end means varargs*/)
-      PrintFatalError(DefLoc, "Intrinsic '" + DefName +
-                                  " has void in result type list!");
-
-    IS.ParamVTs.push_back(VT);
-    IS.ParamTypeDefs.push_back(TyEl);
-  }
-
-  // Parse the intrinsic properties.
-  ListInit *PropList = R->getValueAsListInit("IntrProperties");
-  for (unsigned i = 0, e = PropList->size(); i != e; ++i) {
-    Record *Property = PropList->getElementAsRecord(i);
-    assert(Property->isSubClassOf("IntrinsicProperty") &&
-           "Expected a property!");
-
-    setProperty(Property);
-  }
-
-  // Set default properties to true.
-  setDefaultProperties(R, DefaultProperties);
-
-  // Also record the SDPatternOperator Properties.
-  Properties = parseSDPatternOperatorProperties(R);
-
-  // Sort the argument attributes for later benefit.
-  for (auto &Attrs : ArgumentAttributes)
-    llvm::sort(Attrs);
-}
-
-void CodeGenIntrinsic::setDefaultProperties(
-    Record *R, std::vector<Record *> DefaultProperties) {
-  // opt-out of using default attributes.
-  if (R->getValueAsBit("DisableDefaultAttributes"))
-    return;
-
-  for (Record *Rec : DefaultProperties)
-    setProperty(Rec);
-}
-
-void CodeGenIntrinsic::setProperty(Record *R) {
-  if (R->getName() == "IntrNoMem")
-    ME = MemoryEffects::none();
-  else if (R->getName() == "IntrReadMem") {
-    if (ME.onlyWritesMemory())
-      PrintFatalError(TheDef->getLoc(),
-                      Twine("IntrReadMem cannot be used after IntrNoMem or "
-                            "IntrWriteMem. Default is ReadWrite"));
-    ME &= MemoryEffects::readOnly();
-  } else if (R->getName() == "IntrWriteMem") {
-    if (ME.onlyReadsMemory())
-      PrintFatalError(TheDef->getLoc(),
-                      Twine("IntrWriteMem cannot be used after IntrNoMem or "
-                            "IntrReadMem. Default is ReadWrite"));
-    ME &= MemoryEffects::writeOnly();
-  } else if (R->getName() == "IntrArgMemOnly")
-    ME &= MemoryEffects::argMemOnly();
-  else if (R->getName() == "IntrInaccessibleMemOnly")
-    ME &= MemoryEffects::inaccessibleMemOnly();
-  else if (R->getName() == "IntrInaccessibleMemOrArgMemOnly")
-    ME &= MemoryEffects::inaccessibleOrArgMemOnly();
-  else if (R->getName() == "Commutative")
-    isCommutative = true;
-  else if (R->getName() == "Throws")
-    canThrow = true;
-  else if (R->getName() == "IntrNoDuplicate")
-    isNoDuplicate = true;
-  else if (R->getName() == "IntrNoMerge")
-    isNoMerge = true;
-  else if (R->getName() == "IntrConvergent")
-    isConvergent = true;
-  else if (R->getName() == "IntrNoReturn")
-    isNoReturn = true;
-  else if (R->getName() == "IntrNoCallback")
-    isNoCallback = true;
-  else if (R->getName() == "IntrNoSync")
-    isNoSync = true;
-  else if (R->getName() == "IntrNoFree")
-    isNoFree = true;
-  else if (R->getName() == "IntrWillReturn")
-    isWillReturn = !isNoReturn;
-  else if (R->getName() == "IntrCold")
-    isCold = true;
-  else if (R->getName() == "IntrSpeculatable")
-    isSpeculatable = true;
-  else if (R->getName() == "IntrHasSideEffects")
-    hasSideEffects = true;
-  else if (R->isSubClassOf("NoCapture")) {
-    unsigned ArgNo = R->getValueAsInt("ArgNo");
-    addArgAttribute(ArgNo, NoCapture);
-  } else if (R->isSubClassOf("NoAlias")) {
-    unsigned ArgNo = R->getValueAsInt("ArgNo");
-    addArgAttribute(ArgNo, NoAlias);
-  } else if (R->isSubClassOf("NoUndef")) {
-    unsigned ArgNo = R->getValueAsInt("ArgNo");
-    addArgAttribute(ArgNo, NoUndef);
-  } else if (R->isSubClassOf("NonNull")) {
-    unsigned ArgNo = R->getValueAsInt("ArgNo");
-    addArgAttribute(ArgNo, NonNull);
-  } else if (R->isSubClassOf("Returned")) {
-    unsigned ArgNo = R->getValueAsInt("ArgNo");
-    addArgAttribute(ArgNo, Returned);
-  } else if (R->isSubClassOf("ReadOnly")) {
-    unsigned ArgNo = R->getValueAsInt("ArgNo");
-    addArgAttribute(ArgNo, ReadOnly);
-  } else if (R->isSubClassOf("WriteOnly")) {
-    unsigned ArgNo = R->getValueAsInt("ArgNo");
-    addArgAttribute(ArgNo, WriteOnly);
-  } else if (R->isSubClassOf("ReadNone")) {
-    unsigned ArgNo = R->getValueAsInt("ArgNo");
-    addArgAttribute(ArgNo, ReadNone);
-  } else if (R->isSubClassOf("ImmArg")) {
-    unsigned ArgNo = R->getValueAsInt("ArgNo");
-    addArgAttribute(ArgNo, ImmArg);
-  } else if (R->isSubClassOf("Align")) {
-    unsigned ArgNo = R->getValueAsInt("ArgNo");
-    uint64_t Align = R->getValueAsInt("Align");
-    addArgAttribute(ArgNo, Alignment, Align);
-  } else
-    llvm_unreachable("Unknown property!");
-}
-
-bool CodeGenIntrinsic::isParamAPointer(unsigned ParamIdx) const {
-  if (ParamIdx >= IS.ParamVTs.size())
-    return false;
-  MVT ParamType = MVT(IS.ParamVTs[ParamIdx]);
-  return ParamType == MVT::iPTR || ParamType == MVT::iPTRAny;
-}
-
-bool CodeGenIntrinsic::isParamImmArg(unsigned ParamIdx) const {
-  // Convert argument index to attribute index starting from `FirstArgIndex`.
-  ++ParamIdx;
-  if (ParamIdx >= ArgumentAttributes.size())
-    return false;
-  ArgAttribute Val{ImmArg, 0};
-  return std::binary_search(ArgumentAttributes[ParamIdx].begin(),
-                            ArgumentAttributes[ParamIdx].end(), Val);
-}
-
-void CodeGenIntrinsic::addArgAttribute(unsigned Idx, ArgAttrKind AK,
-                                       uint64_t V) {
-  if (Idx >= ArgumentAttributes.size())
-    ArgumentAttributes.resize(Idx + 1);
-  ArgumentAttributes[Idx].emplace_back(AK, V);
-}
diff --git a/llvm/utils/TableGen/DXILEmitter.cpp b/llvm/utils/TableGen/DXILEmitter.cpp
index 33a2f73ed00e4..5239cb74d2aaa 100644
--- a/llvm/utils/TableGen/DXILEmitter.cpp
+++ b/llvm/utils/TableGen/DXILEmitter.cpp
@@ -322,7 +322,7 @@ static void emitDXILOperationTable(std::vector<DXILOperationData> &DXILOps,
   for (auto &DXILOp : DXILOps) {
     OpStrings.add(DXILOp.DXILOp.str());
 
-    if (ClassSet.find(DXILOp.DXILClass) != ClassSet.end())
+    if (ClassSet.contains(DXILOp.DXILClass))
       continue;
     ClassSet.insert(DXILOp.DXILClass);
     OpClassStrings.add(getDXILOpClassName(DXILOp.DXILClass));
diff --git a/llvm/utils/TableGen/DecoderEmitter.cpp b/llvm/utils/TableGen/DecoderEmitter.cpp
index eabc158ab91ed..b3f696654fc5f 100644
--- a/llvm/utils/TableGen/DecoderEmitter.cpp
+++ b/llvm/utils/TableGen/DecoderEmitter.cpp
@@ -2029,193 +2029,11 @@ populateInstruction(CodeGenTarget &Target, const Record &EncodingDef,
   if (IsVarLenInst) {
     parseVarLenInstOperand(EncodingDef, InsnOperands, CGI);
   } else {
-    std::map<std::string, std::vector<OperandInfo>> NumberedInsnOperands;
-    std::set<std::string> NumberedInsnOperandsNoTie;
-    bool SupportPositionalDecoding =
-        Target.getInstructionSet()->getValueAsBit(
-            "useDeprecatedPositionallyEncodedOperands") &&
-        Target.getInstructionSet()->getValueAsBit(
-            "decodePositionallyEncodedOperands");
-    if (SupportPositionalDecoding) {
-      const std::vector<RecordVal> &Vals = Def.getValues();
-      unsigned NumberedOp = 0;
-
-      std::set<unsigned> NamedOpIndices;
-      if (Target.getInstructionSet()->getValueAsBit(
-              "noNamedPositionallyEncodedOperands"))
-        // Collect the set of operand indices that might correspond to named
-        // operand, and skip these when assigning operands based on position.
-        for (unsigned i = 0, e = Vals.size(); i != e; ++i) {
-          unsigned OpIdx;
-          if (!CGI.Operands.hasOperandNamed(Vals[i].getName(), OpIdx))
-            continue;
-
-          NamedOpIndices.insert(OpIdx);
-        }
-
-      for (unsigned i = 0, e = Vals.size(); i != e; ++i) {
-        // Ignore fixed fields in the record, we're looking for values like:
-        //    bits<5> RST = { ?, ?, ?, ?, ? };
-        if (Vals[i].isNonconcreteOK() || Vals[i].getValue()->isComplete())
-          continue;
-
-        // Determine if Vals[i] actually contributes to the Inst encoding.
-        unsigned bi = 0;
-        for (; bi < Bits.getNumBits(); ++bi) {
-          VarInit *Var = nullptr;
-          VarBitInit *BI = dyn_cast<VarBitInit>(Bits.getBit(bi));
-          if (BI)
-            Var = dyn_cast<VarInit>(BI->getBitVar());
-          else
-            Var = dyn_cast<VarInit>(Bits.getBit(bi));
-
-          if (Var && Var->getName() == Vals[i].getName())
-            break;
-        }
-
-        if (bi == Bits.getNumBits())
-          continue;
-
-        // Skip variables that correspond to explicitly-named operands.
-        unsigned OpIdx;
-        std::pair<unsigned, unsigned> SubOp;
-        if (CGI.Operands.hasSubOperandAlias(Vals[i].getName(), SubOp) ||
-            CGI.Operands.hasOperandNamed(Vals[i].getName(), OpIdx))
-          continue;
-
-        // Get the bit range for this operand:
-        unsigned bitStart = bi++, bitWidth = 1;
-        for (; bi < Bits.getNumBits(); ++bi) {
-          VarInit *Var = nullptr;
-          VarBitInit *BI = dyn_cast<VarBitInit>(Bits.getBit(bi));
-          if (BI)
-            Var = dyn_cast<VarInit>(BI->getBitVar());
-          else
-            Var = dyn_cast<VarInit>(Bits.getBit(bi));
-
-          if (!Var)
-            break;
-
-          if (Var->getName() != Vals[i].getName())
-            break;
-
-          ++bitWidth;
-        }
-
-        unsigned NumberOps = CGI.Operands.size();
-        while (NumberedOp < NumberOps &&
-               (CGI.Operands.isFlatOperandNotEmitted(NumberedOp) ||
-                (!NamedOpIndices.empty() &&
-                 NamedOpIndices.count(
-                     CGI.Operands.getSubOperandNumber(NumberedOp).first))))
-          ++NumberedOp;
-
-        OpIdx = NumberedOp++;
-
-        // OpIdx now holds the ordered operand number of Vals[i].
-        std::pair<unsigned, unsigned> SO =
-            CGI.Operands.getSubOperandNumber(OpIdx);
-        const std::string &Name = CGI.Operands[SO.first].Name;
-
-        LLVM_DEBUG(dbgs() << "Numbered operand mapping for " << Def.getName()
-                          << ": " << Name << "(" << SO.first << ", "
-                          << SO.second << ") => " << Vals[i].getName() << "\n");
-
-        std::string Decoder;
-        Record *TypeRecord = CGI.Operands[SO.first].Rec;
-
-        RecordVal *DecoderString = TypeRecord->getValue("DecoderMethod");
-        StringInit *String =
-            DecoderString ? dyn_cast<StringInit>(DecoderString->getValue())
-                          : nullptr;
-        if (String && String->getValue() != "")
-          Decoder = std::string(String->getValue());
-
-        if (Decoder == "" && CGI.Operands[SO.first].MIOperandInfo &&
-            CGI.Operands[SO.first].MIOperandInfo->getNumArgs()) {
-          Init *Arg = CGI.Operands[SO.first].MIOperandInfo->getArg(SO.second);
-          if (DefInit *DI = cast<DefInit>(Arg))
-            TypeRecord = DI->getDef();
-        }
-
-        bool isReg = false;
-        if (TypeRecord->isSubClassOf("RegisterOperand"))
-          TypeRecord = TypeRecord->getValueAsDef("RegClass");
-        if (TypeRecord->isSubClassOf("RegisterClass")) {
-          Decoder = "Decode" + TypeRecord->getName().str() + "RegisterClass";
-          isReg = true;
-        } else if (TypeRecord->isSubClassOf("PointerLikeRegClass")) {
-          Decoder = "DecodePointerLikeRegClass" +
-                    utostr(TypeRecord->getValueAsInt("RegClassKind"));
-          isReg = true;
-        }
-
-        DecoderString = TypeRecord->getValue("DecoderMethod");
-        String = DecoderString ? dyn_cast<StringInit>(DecoderString->getValue())
-                               : nullptr;
-        if (!isReg && String && String->getValue() != "")
-          Decoder = std::string(String->getValue());
-
-        RecordVal *HasCompleteDecoderVal =
-            TypeRecord->getValue("hasCompleteDecoder");
-        BitInit *HasCompleteDecoderBit =
-            HasCompleteDecoderVal
-                ? dyn_cast<BitInit>(HasCompleteDecoderVal->getValue())
-                : nullptr;
-        bool HasCompleteDecoder =
-            HasCompleteDecoderBit ? HasCompleteDecoderBit->getValue() : true;
-
-        OperandInfo OpInfo(Decoder, HasCompleteDecoder);
-        OpInfo.addField(bitStart, bitWidth, 0);
-
-        NumberedInsnOperands[Name].push_back(OpInfo);
-
-        // FIXME: For complex operands with custom decoders we can't handle tied
-        // sub-operands automatically. Skip those here and assume that this is
-        // fixed up elsewhere.
-        if (CGI.Operands[SO.first].MIOperandInfo &&
-            CGI.Operands[SO.first].MIOperandInfo->getNumArgs() > 1 && String &&
-            String->getValue() != "")
-          NumberedInsnOperandsNoTie.insert(Name);
-      }
-    }
-
     // For each operand, see if we can figure out where it is encoded.
     for (const auto &Op : InOutOperands) {
       Init *OpInit = Op.first;
       StringRef OpName = Op.second;
 
-      if (SupportPositionalDecoding) {
-        if (!NumberedInsnOperands[std::string(OpName)].empty()) {
-          llvm::append_range(InsnOperands,
-                             NumberedInsnOperands[std::string(OpName)]);
-          continue;
-        }
-        if (!NumberedInsnOperands[TiedNames[std::string(OpName)]].empty()) {
-          if (!NumberedInsnOperandsNoTie.count(
-                  TiedNames[std::string(OpName)])) {
-            // Figure out to which (sub)operand we're tied.
-            unsigned i =
-                CGI.Operands.getOperandNamed(TiedNames[std::string(OpName)]);
-            int tiedTo = CGI.Operands[i].getTiedRegister();
-            if (tiedTo == -1) {
-              i = CGI.Operands.getOperandNamed(OpName);
-              tiedTo = CGI.Operands[i].getTiedRegister();
-            }
-
-            if (tiedTo != -1) {
-              std::pair<unsigned, unsigned> SO =
-                  CGI.Operands.getSubOperandNumber(tiedTo);
-
-              InsnOperands.push_back(
-                  NumberedInsnOperands[TiedNames[std::string(OpName)]]
-                                      [SO.second]);
-            }
-          }
-          continue;
-        }
-      }
-
       // We're ready to find the instruction encoding locations for this operand.
 
       // First, find the operand type ("OpInit"), and sub-op names
diff --git a/llvm/utils/TableGen/GlobalISel/GIMatchTree.cpp b/llvm/utils/TableGen/GlobalISel/GIMatchTree.cpp
index d98884493e840..fcf9c25ae74a3 100644
--- a/llvm/utils/TableGen/GlobalISel/GIMatchTree.cpp
+++ b/llvm/utils/TableGen/GlobalISel/GIMatchTree.cpp
@@ -89,20 +89,20 @@ GIMatchTreeBuilderLeafInfo::GIMatchTreeBuilderLeafInfo(
       TraversableEdges(MatchDag.getNumEdges()),
       TestablePredicates(MatchDag.getNumPredicates()) {
   // Number all the predicates in this DAG
-  for (auto &P : enumerate(MatchDag.predicates())) {
-    PredicateIDs.insert(std::make_pair(P.value(), P.index()));
+  for (const auto &[Idx, P] : enumerate(MatchDag.predicates())) {
+    PredicateIDs.insert(std::make_pair(P, Idx));
   }
 
   // Number all the predicate dependencies in this DAG and set up a bitvector
   // for each predicate indicating the unsatisfied dependencies.
-  for (auto &Dep : enumerate(MatchDag.predicate_edges())) {
-    PredicateDepIDs.insert(std::make_pair(Dep.value(), Dep.index()));
+  for (const auto &[Idx, Dep] : enumerate(MatchDag.predicate_edges())) {
+    PredicateDepIDs.insert(std::make_pair(Dep, Idx));
   }
   UnsatisfiedPredDepsForPred.resize(MatchDag.getNumPredicates(),
                                     BitVector(PredicateDepIDs.size()));
-  for (auto &Dep : enumerate(MatchDag.predicate_edges())) {
-    unsigned ID = PredicateIDs.lookup(Dep.value()->getPredicate());
-    UnsatisfiedPredDepsForPred[ID].set(Dep.index());
+  for (const auto &[Idx, Dep] : enumerate(MatchDag.predicate_edges())) {
+    unsigned ID = PredicateIDs.lookup(Dep->getPredicate());
+    UnsatisfiedPredDepsForPred[ID].set(Idx);
   }
 }
 
@@ -134,10 +134,10 @@ void GIMatchTreeBuilderLeafInfo::declareInstr(const GIMatchDagInstr *Instr, unsi
   // Mark the dependencies that are now satisfied as a result of this
   // instruction and mark any predicates whose dependencies are fully
   // satisfied.
-  for (auto &Dep : enumerate(MatchDag.predicate_edges())) {
+  for (const auto &Dep : enumerate(MatchDag.predicate_edges())) {
     if (Dep.value()->getRequiredMI() == Instr &&
         Dep.value()->getRequiredMO() == nullptr) {
-      for (auto &DepsFor : enumerate(UnsatisfiedPredDepsForPred)) {
+      for (const auto &DepsFor : enumerate(UnsatisfiedPredDepsForPred)) {
         DepsFor.value().reset(Dep.index());
         if (DepsFor.value().none())
           TestablePredicates.set(DepsFor.index());
@@ -157,10 +157,9 @@ void GIMatchTreeBuilderLeafInfo::declareOperand(unsigned InstrID,
   // When an operand becomes reachable, we potentially activate some traversals.
   // Record the edges that can now be followed as a result of this
   // instruction.
-  for (auto &E : enumerate(MatchDag.edges())) {
-    if (E.value()->getFromMI() == Instr &&
-        E.value()->getFromMO()->getIdx() == OpIdx) {
-      TraversableEdges.set(E.index());
+  for (const auto &[Idx, E] : enumerate(MatchDag.edges())) {
+    if (E->getFromMI() == Instr && E->getFromMO()->getIdx() == OpIdx) {
+      TraversableEdges.set(Idx);
     }
   }
 
@@ -168,10 +167,10 @@ void GIMatchTreeBuilderLeafInfo::declareOperand(unsigned InstrID,
   // Clear the dependencies that are now satisfied as a result of this
   // operand and activate any predicates whose dependencies are fully
   // satisfied.
-  for (auto &Dep : enumerate(MatchDag.predicate_edges())) {
+  for (const auto &Dep : enumerate(MatchDag.predicate_edges())) {
     if (Dep.value()->getRequiredMI() == Instr && Dep.value()->getRequiredMO() &&
         Dep.value()->getRequiredMO()->getIdx() == OpIdx) {
-      for (auto &DepsFor : enumerate(UnsatisfiedPredDepsForPred)) {
+      for (const auto &DepsFor : enumerate(UnsatisfiedPredDepsForPred)) {
         DepsFor.value().reset(Dep.index());
         if (DepsFor.value().none())
           TestablePredicates.set(DepsFor.index());
@@ -339,9 +338,9 @@ void GIMatchTreeBuilder::runStep() {
          "Must always partition into at least one partition");
 
   TreeNode->setNumChildren(Partitioner->getNumPartitions());
-  for (auto &C : enumerate(TreeNode->children())) {
-    SubtreeBuilders.emplace_back(&C.value(), NextInstrID);
-    Partitioner->applyForPartition(C.index(), *this, SubtreeBuilders.back());
+  for (const auto &[Idx, Child] : enumerate(TreeNode->children())) {
+    SubtreeBuilders.emplace_back(&Child, NextInstrID);
+    Partitioner->applyForPartition(Idx, *this, SubtreeBuilders.back());
   }
 
   TreeNode->setPartitioner(std::move(Partitioner));
@@ -536,22 +535,22 @@ void GIMatchTreeOpcodePartitioner::applyForPartition(
 
   BitVector PossibleLeaves = getPossibleLeavesForPartition(PartitionIdx);
   // Consume any predicates we handled.
-  for (auto &EnumeratedLeaf : enumerate(Builder.getPossibleLeaves())) {
-    if (!PossibleLeaves[EnumeratedLeaf.index()])
+  for (const auto &[Index, EnumeratedLeaf] :
+       enumerate(Builder.getPossibleLeaves())) {
+    if (!PossibleLeaves[Index])
       continue;
 
-    auto &Leaf = EnumeratedLeaf.value();
-    const auto &TestedPredicatesForLeaf =
-        TestedPredicates[EnumeratedLeaf.index()];
+    const auto &TestedPredicatesForLeaf = TestedPredicates[Index];
 
     for (unsigned PredIdx : TestedPredicatesForLeaf.set_bits()) {
-      LLVM_DEBUG(dbgs() << "    " << Leaf.getName() << " tested predicate #"
-                        << PredIdx << " of " << TestedPredicatesForLeaf.size()
-                        << " " << *Leaf.getPredicate(PredIdx) << "\n");
-      Leaf.RemainingPredicates.reset(PredIdx);
-      Leaf.TestablePredicates.reset(PredIdx);
+      LLVM_DEBUG(dbgs() << "    " << EnumeratedLeaf.getName()
+                        << " tested predicate #" << PredIdx << " of "
+                        << TestedPredicatesForLeaf.size() << " "
+                        << *EnumeratedLeaf.getPredicate(PredIdx) << "\n");
+      EnumeratedLeaf.RemainingPredicates.reset(PredIdx);
+      EnumeratedLeaf.TestablePredicates.reset(PredIdx);
     }
-    SubBuilder.addLeaf(Leaf);
+    SubBuilder.addLeaf(EnumeratedLeaf);
   }
 
   // Nothing to do, we don't know anything about this instruction as a result
@@ -571,11 +570,11 @@ void GIMatchTreeOpcodePartitioner::applyForPartition(
     if (!InstrInfo)
       continue;
     const GIMatchDagInstr *Instr = InstrInfo->getInstrNode();
-    for (auto &E : enumerate(Leaf.getMatchDag().edges())) {
-      if (E.value()->getFromMI() == Instr &&
-          E.value()->getFromMO()->getIdx() < CGI->Operands.size()) {
-        ReferencedOperands.resize(E.value()->getFromMO()->getIdx() + 1);
-        ReferencedOperands.set(E.value()->getFromMO()->getIdx());
+    for (const auto &E : Leaf.getMatchDag().edges()) {
+      if (E->getFromMI() == Instr &&
+          E->getFromMO()->getIdx() < CGI->Operands.size()) {
+        ReferencedOperands.resize(E->getFromMO()->getIdx() + 1);
+        ReferencedOperands.set(E->getFromMO()->getIdx());
       }
     }
   }
@@ -715,16 +714,16 @@ void GIMatchTreeVRegDefPartitioner::applyForPartition(
 
   std::vector<BitVector> TraversedEdgesByNewLeaves;
   // Consume any edges we handled.
-  for (auto &EnumeratedLeaf : enumerate(Builder.getPossibleLeaves())) {
-    if (!PossibleLeaves[EnumeratedLeaf.index()])
+  for (const auto &[Index, EnumeratedLeaf] :
+       enumerate(Builder.getPossibleLeaves())) {
+    if (!PossibleLeaves[Index])
       continue;
 
-    auto &Leaf = EnumeratedLeaf.value();
-    const auto &TraversedEdgesForLeaf = TraversedEdges[EnumeratedLeaf.index()];
+    const auto &TraversedEdgesForLeaf = TraversedEdges[Index];
     TraversedEdgesByNewLeaves.push_back(TraversedEdgesForLeaf);
-    Leaf.RemainingEdges.reset(TraversedEdgesForLeaf);
-    Leaf.TraversableEdges.reset(TraversedEdgesForLeaf);
-    SubBuilder.addLeaf(Leaf);
+    EnumeratedLeaf.RemainingEdges.reset(TraversedEdgesForLeaf);
+    EnumeratedLeaf.TraversableEdges.reset(TraversedEdgesForLeaf);
+    SubBuilder.addLeaf(EnumeratedLeaf);
   }
 
   // Nothing to do. The only thing we know is that it isn't a vreg-def.
diff --git a/llvm/utils/TableGen/GlobalISelEmitter.cpp b/llvm/utils/TableGen/GlobalISelEmitter.cpp
index 70f38ae504de0..dfeb7aaeeda9e 100644
--- a/llvm/utils/TableGen/GlobalISelEmitter.cpp
+++ b/llvm/utils/TableGen/GlobalISelEmitter.cpp
@@ -3369,7 +3369,7 @@ unsigned RuleMatcher::getInsnVarID(InstructionMatcher &InsnMatcher) const {
 }
 
 void RuleMatcher::defineOperand(StringRef SymbolicName, OperandMatcher &OM) {
-  if (DefinedOperands.find(SymbolicName) == DefinedOperands.end()) {
+  if (!DefinedOperands.contains(SymbolicName)) {
     DefinedOperands[SymbolicName] = &OM;
     return;
   }
@@ -3383,7 +3383,7 @@ void RuleMatcher::defineOperand(StringRef SymbolicName, OperandMatcher &OM) {
 }
 
 void RuleMatcher::definePhysRegOperand(Record *Reg, OperandMatcher &OM) {
-  if (PhysRegOperands.find(Reg) == PhysRegOperands.end()) {
+  if (!PhysRegOperands.contains(Reg)) {
     PhysRegOperands[Reg] = &OM;
     return;
   }
diff --git a/llvm/utils/TableGen/IntrinsicEmitter.cpp b/llvm/utils/TableGen/IntrinsicEmitter.cpp
index 9fce53e8dbd03..b759c71811124 100644
--- a/llvm/utils/TableGen/IntrinsicEmitter.cpp
+++ b/llvm/utils/TableGen/IntrinsicEmitter.cpp
@@ -11,7 +11,6 @@
 //===----------------------------------------------------------------------===//
 
 #include "CodeGenIntrinsics.h"
-#include "CodeGenTarget.h"
 #include "SequenceToOffsetTable.h"
 #include "TableGenBackends.h"
 #include "llvm/ADT/ArrayRef.h"
@@ -37,6 +36,7 @@
 #include <utility>
 #include <vector>
 using namespace llvm;
+using namespace llvm::tmp;
 
 cl::OptionCategory GenIntrinsicCat("Options for -gen-intrinsic-enums");
 cl::opt<std::string>
diff --git a/llvm/utils/TableGen/RISCVTargetDefEmitter.cpp b/llvm/utils/TableGen/RISCVTargetDefEmitter.cpp
index fa6508cbfc696..b534ce4b3b785 100644
--- a/llvm/utils/TableGen/RISCVTargetDefEmitter.cpp
+++ b/llvm/utils/TableGen/RISCVTargetDefEmitter.cpp
@@ -27,7 +27,7 @@ using ISAInfoTy = llvm::Expected<std::unique_ptr<RISCVISAInfo>>;
 // get feature name from feature records instead of feature bits.
 static std::string getMArch(const Record &Rec) {
   std::vector<std::string> FeatureVector;
-  int XLen = 32;
+  unsigned XLen = 32;
 
   // Convert features to FeatureVector.
   for (auto *Feature : Rec.getValueAsListOfDefs("Features")) {
diff --git a/llvm/utils/TableGen/X86FoldTablesEmitter.cpp b/llvm/utils/TableGen/X86FoldTablesEmitter.cpp
index d4b2d06d8b999..052151dfb007e 100644
--- a/llvm/utils/TableGen/X86FoldTablesEmitter.cpp
+++ b/llvm/utils/TableGen/X86FoldTablesEmitter.cpp
@@ -15,7 +15,9 @@
 #include "CodeGenTarget.h"
 #include "TableGenBackends.h"
 #include "X86RecognizableInstr.h"
+#include "llvm/ADT/DenseMap.h"
 #include "llvm/Support/FormattedStream.h"
+#include "llvm/Support/X86FoldTablesUtils.h"
 #include "llvm/TableGen/Record.h"
 #include "llvm/TableGen/TableGenBackend.h"
 
@@ -23,23 +25,14 @@ using namespace llvm;
 using namespace X86Disassembler;
 
 namespace {
-
-// 3 possible strategies for the unfolding flag (TB_NO_REVERSE) of the
-// manual added entries.
-enum UnfoldStrategy {
-  UNFOLD,     // Allow unfolding
-  NO_UNFOLD,  // Prevent unfolding
-  NO_STRATEGY // Make decision according to operands' sizes
-};
-
 // Represents an entry in the manual mapped instructions set.
 struct ManualMapEntry {
   const char *RegInstStr;
   const char *MemInstStr;
-  UnfoldStrategy Strategy;
+  uint16_t Strategy;
 
   ManualMapEntry(const char *RegInstStr, const char *MemInstStr,
-                 UnfoldStrategy Strategy = NO_STRATEGY)
+                 uint16_t Strategy = 0)
       : RegInstStr(RegInstStr), MemInstStr(MemInstStr), Strategy(Strategy) {}
 };
 
@@ -52,37 +45,7 @@ const char *ExplicitUnalign[] = {"MOVDQU", "MOVUPS", "MOVUPD",
                                  "PCMPESTRM", "PCMPESTRI",
                                  "PCMPISTRM", "PCMPISTRI" };
 
-// For manually mapping instructions that do not match by their encoding.
-const ManualMapEntry ManualMapSet[] = {
-    { "ADD16ri_DB",         "ADD16mi",         NO_UNFOLD  },
-    { "ADD16ri8_DB",        "ADD16mi8",        NO_UNFOLD  },
-    { "ADD16rr_DB",         "ADD16mr",         NO_UNFOLD  },
-    { "ADD32ri_DB",         "ADD32mi",         NO_UNFOLD  },
-    { "ADD32ri8_DB",        "ADD32mi8",        NO_UNFOLD  },
-    { "ADD32rr_DB",         "ADD32mr",         NO_UNFOLD  },
-    { "ADD64ri32_DB",       "ADD64mi32",       NO_UNFOLD  },
-    { "ADD64ri8_DB",        "ADD64mi8",        NO_UNFOLD  },
-    { "ADD64rr_DB",         "ADD64mr",         NO_UNFOLD  },
-    { "ADD8ri_DB",          "ADD8mi",          NO_UNFOLD  },
-    { "ADD8rr_DB",          "ADD8mr",          NO_UNFOLD  },
-    { "ADD16rr_DB",         "ADD16rm",         NO_UNFOLD  },
-    { "ADD32rr_DB",         "ADD32rm",         NO_UNFOLD  },
-    { "ADD64rr_DB",         "ADD64rm",         NO_UNFOLD  },
-    { "ADD8rr_DB",          "ADD8rm",          NO_UNFOLD  },
-    { "MMX_MOVD64from64rr", "MMX_MOVQ64mr",    UNFOLD },
-    { "MMX_MOVD64grr",      "MMX_MOVD64mr",    UNFOLD },
-    { "MOVLHPSrr",          "MOVHPSrm",        NO_UNFOLD  },
-    { "PUSH16r",            "PUSH16rmm",       UNFOLD },
-    { "PUSH32r",            "PUSH32rmm",       UNFOLD },
-    { "PUSH64r",            "PUSH64rmm",       UNFOLD },
-    { "TAILJMPr",           "TAILJMPm",        UNFOLD },
-    { "TAILJMPr64",         "TAILJMPm64",      UNFOLD },
-    { "TAILJMPr64_REX",     "TAILJMPm64_REX",  UNFOLD },
-    { "VMOVLHPSZrr",        "VMOVHPSZ128rm",   NO_UNFOLD  },
-    { "VMOVLHPSrr",         "VMOVHPSrm",       NO_UNFOLD  },
-};
-
-
+#include "X86FoldTablesEmitterManualMapSet.inc"
 static bool isExplicitAlign(const CodeGenInstruction *Inst) {
   return any_of(ExplicitAlign, [Inst](const char *InstStr) {
     return Inst->TheDef->getName().contains(InstStr);
@@ -106,50 +69,63 @@ class X86FoldTablesEmitter {
 
   public:
     bool CannotUnfold = false;
+    bool CannotFold = false;
     bool IsLoad = false;
     bool IsStore = false;
     bool IsAligned = false;
     unsigned int Alignment = 0;
 
+    X86FoldTableEntry() = default;
     X86FoldTableEntry(const CodeGenInstruction *RegInst,
                       const CodeGenInstruction *MemInst)
         : RegInst(RegInst), MemInst(MemInst) {}
 
     void print(formatted_raw_ostream &OS) const {
+      // Stop printing record if it can't fold and unfold.
+      if(CannotUnfold && CannotFold)
+        return;
       OS.indent(2);
-      OS << "{ X86::" << RegInst->TheDef->getName() << ",";
-      OS.PadToColumn(40);
-      OS  << "X86::" << MemInst->TheDef->getName() << ",";
-      OS.PadToColumn(75);
+      OS << "{X86::" << RegInst->TheDef->getName() << ", ";
+      OS  << "X86::" << MemInst->TheDef->getName() << ", ";
 
       std::string Attrs;
       if (IsLoad)
-        Attrs += "TB_FOLDED_LOAD | ";
+        Attrs += "TB_FOLDED_LOAD|";
       if (IsStore)
-        Attrs += "TB_FOLDED_STORE | ";
+        Attrs += "TB_FOLDED_STORE|";
       if (CannotUnfold)
-        Attrs += "TB_NO_REVERSE | ";
+        Attrs += "TB_NO_REVERSE|";
+      if (CannotFold)
+        Attrs += "TB_NO_FORWARD|";
       if (IsAligned)
-        Attrs += "TB_ALIGN_" + std::to_string(Alignment) + " | ";
+        Attrs += "TB_ALIGN_" + std::to_string(Alignment) + "|";
 
-      StringRef SimplifiedAttrs = StringRef(Attrs).rtrim("| ");
+      StringRef SimplifiedAttrs = StringRef(Attrs).rtrim("|");
       if (SimplifiedAttrs.empty())
         SimplifiedAttrs = "0";
 
-      OS << SimplifiedAttrs << " },\n";
+      OS << SimplifiedAttrs << "},\n";
     }
 
-    bool operator<(const X86FoldTableEntry &RHS) const {
-      bool LHSpseudo = RegInst->TheDef->getValueAsBit("isPseudo");
-      bool RHSpseudo = RHS.RegInst->TheDef->getValueAsBit("isPseudo");
+  };
+
+  struct CodeGenInstructionComparator {
+    // Comparator function
+    bool operator()(const CodeGenInstruction *LHS,
+                    const CodeGenInstruction *RHS) const {
+      assert(LHS && RHS && "LHS and RHS shouldn't be nullptr");
+      bool LHSpseudo = LHS->TheDef->getValueAsBit("isPseudo");
+      bool RHSpseudo = RHS->TheDef->getValueAsBit("isPseudo");
       if (LHSpseudo != RHSpseudo)
         return LHSpseudo;
 
-      return RegInst->TheDef->getName() < RHS.RegInst->TheDef->getName();
+      return LHS->TheDef->getName() < RHS->TheDef->getName();
     }
   };
 
-  typedef std::vector<X86FoldTableEntry> FoldTable;
+  typedef std::map<const CodeGenInstruction *, X86FoldTableEntry,
+                   CodeGenInstructionComparator>
+      FoldTable;
   // std::vector for each folding table.
   // Table2Addr - Holds instructions which their memory form performs load+store
   // Table#i - Holds instructions which the their memory form perform a load OR
@@ -171,14 +147,14 @@ class X86FoldTablesEmitter {
   // Decides to which table to add the entry with the given instructions.
   // S sets the strategy of adding the TB_NO_REVERSE flag.
   void updateTables(const CodeGenInstruction *RegInstr,
-                    const CodeGenInstruction *MemInstr,
-                    const UnfoldStrategy S = NO_STRATEGY);
+                    const CodeGenInstruction *MemInstr, const uint16_t S = 0,
+                    bool IsManual = false);
 
   // Generates X86FoldTableEntry with the given instructions and fill it with
   // the appropriate flags - then adds it to Table.
   void addEntryWithFlags(FoldTable &Table, const CodeGenInstruction *RegInstr,
-                         const CodeGenInstruction *MemInstr,
-                         const UnfoldStrategy S, const unsigned int FoldedInd);
+                         const CodeGenInstruction *MemInstr, const uint16_t S,
+                         const unsigned int FoldedInd, bool isManual);
 
   // Print the given table as a static const C++ array of type
   // X86MemoryFoldTableEntry.
@@ -187,8 +163,8 @@ class X86FoldTablesEmitter {
     OS << "static const X86MemoryFoldTableEntry MemoryFold" << TableName
        << "[] = {\n";
 
-    for (const X86FoldTableEntry &E : Table)
-      E.print(OS);
+    for (auto &E : Table)
+      E.second.print(OS);
 
     OS << "};\n\n";
   }
@@ -393,13 +369,26 @@ class IsMatch {
 void X86FoldTablesEmitter::addEntryWithFlags(FoldTable &Table,
                                              const CodeGenInstruction *RegInstr,
                                              const CodeGenInstruction *MemInstr,
-                                             const UnfoldStrategy S,
-                                             const unsigned int FoldedInd) {
+                                             const uint16_t S,
+                                             const unsigned int FoldedInd,
+                                             bool isManual) {
 
   X86FoldTableEntry Result = X86FoldTableEntry(RegInstr, MemInstr);
   Record *RegRec = RegInstr->TheDef;
   Record *MemRec = MemInstr->TheDef;
 
+  if (isManual) {
+    Result.CannotUnfold = (S & TB_NO_REVERSE) != 0;
+    Result.CannotFold = (S & TB_NO_FORWARD) != 0;
+    Result.IsLoad = (S & TB_FOLDED_LOAD) != 0;
+    Result.IsStore = (S & TB_FOLDED_STORE) != 0;
+    Result.IsAligned = (S & TB_ALIGN_MASK) != 0;
+    auto AlignValue = (S & TB_ALIGN_MASK) >> TB_ALIGN_SHIFT;
+    Result.Alignment = AlignValue > 0 ? (1 << (AlignValue - 1)) : 0;
+    Table[RegInstr] = Result;
+    return;
+  }
+
   // Only table0 entries should explicitly specify a load or store flag.
   if (&Table == &Table0) {
     unsigned MemInOpsNum = MemRec->getValueAsDag("InOperandList")->getNumArgs();
@@ -422,12 +411,27 @@ void X86FoldTablesEmitter::addEntryWithFlags(FoldTable &Table,
   // the register in the register form instruction.
   // If the register's size is greater than the memory's operand size, do not
   // allow unfolding.
-  if (S == UNFOLD)
-    Result.CannotUnfold = false;
-  else if (S == NO_UNFOLD)
+
+  // the unfolded load size will be based on the register size. If that’s bigger
+  // than the memory operand size, the unfolded load will load more memory and
+  // potentially cause a memory fault.
+  if (getRegOperandSize(RegOpRec) > getMemOperandSize(MemOpRec))
+    Result.CannotUnfold = true;
+
+  // Check no-kz version's isMoveReg
+  Record *BaseDef = nullptr;
+  if (RegRec->getName().ends_with("rkz") &&
+      (BaseDef = Records.getDef(
+           RegRec->getName().substr(0, RegRec->getName().size() - 2)))) {
+    Result.CannotUnfold =
+        Target.getInstruction(BaseDef).isMoveReg ? true : Result.CannotUnfold;
+  } else if (RegRec->getName().ends_with("rk") &&
+             (BaseDef = Records.getDef(
+                  RegRec->getName().substr(0, RegRec->getName().size() - 1)))) {
+    Result.CannotUnfold =
+        Target.getInstruction(BaseDef).isMoveReg ? true : Result.CannotUnfold;
+  } else if (RegInstr->isMoveReg && Result.IsStore)
     Result.CannotUnfold = true;
-  else if (getRegOperandSize(RegOpRec) > getMemOperandSize(MemOpRec))
-    Result.CannotUnfold = true; // S == NO_STRATEGY
 
   uint64_t Enc = getValueFromBitsInit(RegRec->getValueAsBitsInit("OpEncBits"));
   if (isExplicitAlign(RegInstr)) {
@@ -445,13 +449,19 @@ void X86FoldTablesEmitter::addEntryWithFlags(FoldTable &Table,
       Result.Alignment = 16;
     }
   }
+  // Expand is only ever created as a masked instruction. It is not safe to
+  // unfold a masked expand because we don't know if it came from an expand load
+  // intrinsic or folding a plain load. If it is from a expand load intrinsic,
+  // Unfolding to plain load would read more elements and could trigger a fault.
+  if (RegRec->getName().contains("EXPAND"))
+    Result.CannotUnfold = true;
 
-  Table.push_back(Result);
+  Table[RegInstr] = Result;
 }
 
 void X86FoldTablesEmitter::updateTables(const CodeGenInstruction *RegInstr,
                                         const CodeGenInstruction *MemInstr,
-                                        const UnfoldStrategy S) {
+                                        const uint16_t S, bool IsManual) {
 
   Record *RegRec = RegInstr->TheDef;
   Record *MemRec = MemInstr->TheDef;
@@ -462,7 +472,7 @@ void X86FoldTablesEmitter::updateTables(const CodeGenInstruction *RegInstr,
 
   // Instructions which Read-Modify-Write should be added to Table2Addr.
   if (MemOutSize != RegOutSize && MemInSize == RegInSize) {
-    addEntryWithFlags(Table2Addr, RegInstr, MemInstr, S, 0);
+    addEntryWithFlags(Table2Addr, RegInstr, MemInstr, S, 0, IsManual);
     return;
   }
 
@@ -479,19 +489,19 @@ void X86FoldTablesEmitter::updateTables(const CodeGenInstruction *RegInstr,
           isMemoryOperand(MemOpRec)) {
         switch (i) {
         case 0:
-          addEntryWithFlags(Table0, RegInstr, MemInstr, S, 0);
+          addEntryWithFlags(Table0, RegInstr, MemInstr, S, 0, IsManual);
           return;
         case 1:
-          addEntryWithFlags(Table1, RegInstr, MemInstr, S, 1);
+          addEntryWithFlags(Table1, RegInstr, MemInstr, S, 1, IsManual);
           return;
         case 2:
-          addEntryWithFlags(Table2, RegInstr, MemInstr, S, 2);
+          addEntryWithFlags(Table2, RegInstr, MemInstr, S, 2, IsManual);
           return;
         case 3:
-          addEntryWithFlags(Table3, RegInstr, MemInstr, S, 3);
+          addEntryWithFlags(Table3, RegInstr, MemInstr, S, 3, IsManual);
           return;
         case 4:
-          addEntryWithFlags(Table4, RegInstr, MemInstr, S, 4);
+          addEntryWithFlags(Table4, RegInstr, MemInstr, S, 4, IsManual);
           return;
         }
       }
@@ -508,7 +518,7 @@ void X86FoldTablesEmitter::updateTables(const CodeGenInstruction *RegInstr,
     Record *MemOpRec = MemInstr->Operands[RegOutSize - 1].Rec;
     if (isRegisterOperand(RegOpRec) && isMemoryOperand(MemOpRec) &&
         getRegOperandSize(RegOpRec) == getMemOperandSize(MemOpRec))
-      addEntryWithFlags(Table0, RegInstr, MemInstr, S, 0);
+      addEntryWithFlags(Table0, RegInstr, MemInstr, S, 0, IsManual);
   }
 }
 
@@ -591,17 +601,9 @@ void X86FoldTablesEmitter::run(formatted_raw_ostream &OS) {
     Record *MemInstIter = Records.getDef(Entry.MemInstStr);
 
     updateTables(&(Target.getInstruction(RegInstIter)),
-                 &(Target.getInstruction(MemInstIter)), Entry.Strategy);
+                 &(Target.getInstruction(MemInstIter)), Entry.Strategy, true);
   }
 
-  // Sort the tables before printing.
-  llvm::sort(Table2Addr);
-  llvm::sort(Table0);
-  llvm::sort(Table1);
-  llvm::sort(Table2);
-  llvm::sort(Table3);
-  llvm::sort(Table4);
-
   // Print all tables.
   printTable(Table2Addr, "Table2Addr", OS);
   printTable(Table0, "Table0", OS);
diff --git a/llvm/utils/TableGen/X86FoldTablesEmitterManualMapSet.inc b/llvm/utils/TableGen/X86FoldTablesEmitterManualMapSet.inc
new file mode 100644
index 0000000000000..8b7f3878f6f59
--- /dev/null
+++ b/llvm/utils/TableGen/X86FoldTablesEmitterManualMapSet.inc
@@ -0,0 +1,83 @@
+const ManualMapEntry ManualMapSet[] = {
+    // Part1: These following records are for manually mapping instructions that
+    // do not match by their encoding.
+    { "ADD16ri_DB",         "ADD16mi",         TB_NO_REVERSE  },
+    { "ADD16ri8_DB",        "ADD16mi8",        TB_NO_REVERSE  },
+    { "ADD16rr_DB",         "ADD16mr",         TB_NO_REVERSE  },
+    { "ADD32ri_DB",         "ADD32mi",         TB_NO_REVERSE  },
+    { "ADD32ri8_DB",        "ADD32mi8",        TB_NO_REVERSE  },
+    { "ADD32rr_DB",         "ADD32mr",         TB_NO_REVERSE  },
+    { "ADD64ri32_DB",       "ADD64mi32",       TB_NO_REVERSE  },
+    { "ADD64ri8_DB",        "ADD64mi8",        TB_NO_REVERSE  },
+    { "ADD64rr_DB",         "ADD64mr",         TB_NO_REVERSE  },
+    { "ADD8ri_DB",          "ADD8mi",          TB_NO_REVERSE  },
+    { "ADD8rr_DB",          "ADD8mr",          TB_NO_REVERSE  },
+    { "ADD16rr_DB",         "ADD16rm",         TB_NO_REVERSE  },
+    { "ADD32rr_DB",         "ADD32rm",         TB_NO_REVERSE  },
+    { "ADD64rr_DB",         "ADD64rm",         TB_NO_REVERSE  },
+    { "ADD8rr_DB",          "ADD8rm",          TB_NO_REVERSE  },
+    { "MMX_MOVD64from64rr", "MMX_MOVQ64mr",    TB_FOLDED_STORE },
+    { "MMX_MOVD64grr",      "MMX_MOVD64mr",    TB_FOLDED_STORE },
+    { "MOV64toSDrr",        "MOV64mr",         TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "MOVDI2SSrr",         "MOV32mr",         TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "MOVPQIto64rr",       "MOVPQI2QImr",     TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "MOVSDto64rr",        "MOVSDmr",         TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "MOVSS2DIrr",         "MOVSSmr",         TB_FOLDED_STORE },
+    { "MOVLHPSrr",          "MOVHPSrm",        TB_NO_REVERSE  },
+    { "PUSH16r",            "PUSH16rmm",       TB_FOLDED_LOAD },
+    { "PUSH32r",            "PUSH32rmm",       TB_FOLDED_LOAD },
+    { "PUSH64r",            "PUSH64rmm",       TB_FOLDED_LOAD },
+    { "TAILJMPr",           "TAILJMPm",        TB_FOLDED_LOAD },
+    { "TAILJMPr64",         "TAILJMPm64",      TB_FOLDED_LOAD },
+    { "TAILJMPr64_REX",     "TAILJMPm64_REX",  TB_FOLDED_LOAD },
+    { "TCRETURNri",         "TCRETURNmi",      TB_FOLDED_LOAD | TB_NO_FORWARD },
+    { "TCRETURNri64",       "TCRETURNmi64",    TB_FOLDED_LOAD | TB_NO_FORWARD },
+    { "VMOVLHPSZrr",        "VMOVHPSZ128rm",   TB_NO_REVERSE  },
+    { "VMOVLHPSrr",         "VMOVHPSrm",       TB_NO_REVERSE  },
+    { "VMOV64toSDZrr",      "MOV64mr",         TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "VMOV64toSDrr",       "MOV64mr",         TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "VMOVDI2SSZrr",       "MOV32mr",         TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "VMOVDI2SSrr",        "MOV32mr",         TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "VMOVPQIto64Zrr",     "VMOVPQI2QIZmr",   TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "VMOVPQIto64rr",      "VMOVPQI2QImr",    TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "VMOVSDto64Zrr",      "VMOVSDZmr",       TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "VMOVSDto64rr",       "VMOVSDmr",        TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "VMOVSS2DIZrr",       "VMOVSSZmr",       TB_FOLDED_STORE },
+    { "VMOVSS2DIrr",        "VMOVSSmr",        TB_FOLDED_STORE },
+    { "MMX_MOVD64to64rr",   "MMX_MOVQ64rm",    0 },
+    { "MOV64toPQIrr",       "MOVQI2PQIrm",     TB_NO_REVERSE },
+    { "MOV64toSDrr",        "MOVSDrm_alt",     TB_NO_REVERSE },
+    { "MOVDI2SSrr",         "MOVSSrm_alt",     0 },
+    { "VMOV64toPQIZrr",     "VMOVQI2PQIZrm",    TB_NO_REVERSE },
+    { "VMOV64toPQIrr",      "VMOVQI2PQIrm",     TB_NO_REVERSE },
+    { "VMOV64toSDZrr",      "VMOVSDZrm_alt",   TB_NO_REVERSE },
+    { "VMOV64toSDrr",       "VMOVSDrm_alt",    TB_NO_REVERSE },
+    { "VMOVDI2SSZrr",       "VMOVSSZrm_alt",   0 },
+    { "VMOVDI2SSrr",        "VMOVSSrm_alt",    0 },
+    { "MOVSDrr",            "MOVLPDrm",         TB_NO_REVERSE },
+    { "VMOVSDZrr",          "VMOVLPDZ128rm",    TB_NO_REVERSE },
+    { "VMOVSDrr",           "VMOVLPDrm",        TB_NO_REVERSE },
+
+    // Part2: These following records are for manually mapping instructions that
+    // have same opcode.
+    // INSERTPSrm has no count_s while INSERTPSrr has count_s.
+    // count_s is to indicate which element in dst vector is inserted.
+    // if count_s!=0, we can't fold INSERTPSrr into INSERTPSrm
+    //
+    // the following folding can happen when count_s==0
+    // load xmm0, m32
+    // insertpsrr xmm1, xmm0, imm
+    // =>
+    // insertpsrm xmm1, m32, imm
+    { "INSERTPSrr",         "INSERTPSrm",      TB_NO_REVERSE | TB_NO_FORWARD },
+    { "UD1Lr",              "UD1Lm",           TB_NO_REVERSE | TB_NO_FORWARD },
+    { "UD1Qr",              "UD1Qm",           TB_NO_REVERSE | TB_NO_FORWARD },
+    { "UD1Wr",              "UD1Wm",           TB_NO_REVERSE | TB_NO_FORWARD },
+    // Remove {"MMX_MOVQ64rr", "MMX_MOVQ64mr"} since it will create duplicate in
+    // unfolding table due to the {"MMX_MOVD64from64rr", "MMX_MOVQ64mr"}
+    { "MMX_MOVQ64rr",       "MMX_MOVQ64mr",    TB_NO_FORWARD | TB_NO_REVERSE },
+    // Remove {"MMX_MOVQ64rr", "MMX_MOVQ64rm"} since it will create duplicate in
+    // unfolding table due to the {"MMX_MOVD64from64rr", "MMX_MOVQ64rm"}
+    { "MMX_MOVQ64rr",       "MMX_MOVQ64rm",    TB_NO_FORWARD | TB_NO_REVERSE },
+};
+
diff --git a/llvm/utils/TableGen/X86RecognizableInstr.cpp b/llvm/utils/TableGen/X86RecognizableInstr.cpp
index e5c1e53936f6d..6aeb200862dbd 100644
--- a/llvm/utils/TableGen/X86RecognizableInstr.cpp
+++ b/llvm/utils/TableGen/X86RecognizableInstr.cpp
@@ -955,6 +955,9 @@ OperandType RecognizableInstr::typeFromString(const std::string &s,
   TYPE("i128mem",             TYPE_M)
   TYPE("i256mem",             TYPE_M)
   TYPE("i512mem",             TYPE_M)
+  TYPE("i512mem_GR16",        TYPE_M)
+  TYPE("i512mem_GR32",        TYPE_M)
+  TYPE("i512mem_GR64",        TYPE_M)
   TYPE("i64i32imm_brtarget",  TYPE_REL)
   TYPE("i16imm_brtarget",     TYPE_REL)
   TYPE("i32imm_brtarget",     TYPE_REL)
@@ -1221,6 +1224,9 @@ RecognizableInstr::memoryEncodingFromString(const std::string &s,
   ENCODING("i128mem",         ENCODING_RM)
   ENCODING("i256mem",         ENCODING_RM)
   ENCODING("i512mem",         ENCODING_RM)
+  ENCODING("i512mem_GR16",    ENCODING_RM)
+  ENCODING("i512mem_GR32",    ENCODING_RM)
+  ENCODING("i512mem_GR64",    ENCODING_RM)
   ENCODING("f80mem",          ENCODING_RM)
   ENCODING("lea64_32mem",     ENCODING_RM)
   ENCODING("lea64mem",        ENCODING_RM)
diff --git a/llvm/utils/TableGen/jupyter/README.md b/llvm/utils/TableGen/jupyter/README.md
index 581662a069f9c..660b5a7015feb 100644
--- a/llvm/utils/TableGen/jupyter/README.md
+++ b/llvm/utils/TableGen/jupyter/README.md
@@ -32,7 +32,7 @@ Then run one of:
     # Then in the notebook interface, select 'LLVM TableGen' from the 'New' menu.
 
     # To run the example notebook in this folder.
-    jupyter notebook LLVM_Tablegen.ipynb
+    jupyter notebook LLVM_TableGen.ipynb
 
     # To use the kernel from the command line.
     jupyter console --kernel tablegen
diff --git a/llvm/utils/UnicodeData/UnicodeNameMappingGenerator.cpp b/llvm/utils/UnicodeData/UnicodeNameMappingGenerator.cpp
index 83770e845d408..7bf9c8b03b418 100644
--- a/llvm/utils/UnicodeData/UnicodeNameMappingGenerator.cpp
+++ b/llvm/utils/UnicodeData/UnicodeNameMappingGenerator.cpp
@@ -361,9 +361,8 @@ int main(int argc, char **argv) {
     char32_t Codepoint = Entry.first;
     const std::string &Name = Entry.second;
     // Ignore names which are not valid.
-    if (Name.empty() || !llvm::all_of(Name, [](char C) {
-          return llvm::is_contained(Letters, C);
-        })) {
+    if (Name.empty() ||
+        !llvm::all_of(Name, [](char C) { return Letters.contains(C); })) {
       continue;
     }
     printf("%06x: %s\n", static_cast<unsigned int>(Codepoint), Name.c_str());
diff --git a/llvm/utils/UpdateTestChecks/common.py b/llvm/utils/UpdateTestChecks/common.py
index b52306a1c67f9..aa0b12812e205 100644
--- a/llvm/utils/UpdateTestChecks/common.py
+++ b/llvm/utils/UpdateTestChecks/common.py
@@ -22,9 +22,10 @@
 Version changelog:
 
 1: Initial version, used by tests that don't specify --version explicitly.
-2: --function-signature now also checks return type/attributes.
+2: --function-signature is now enabled by default and also checks return
+   type/attributes.
 """
-DEFAULT_VERSION = 1
+DEFAULT_VERSION = 2
 
 class Regex(object):
   """Wrap a compiled regular expression object to allow deep copy of a regexp.
@@ -157,6 +158,11 @@ def __call__(self, parser, namespace, values, option_string=None):
   _global_hex_value_regex = args.global_hex_value_regex
   return args
 
+def parse_args(parser, argv):
+  args = parser.parse_args(argv)
+  if args.version >= 2:
+    args.function_signature = True
+  return args
 
 class InputLineInfo(object):
   def __init__(self, line, line_number, args, argv):
@@ -246,7 +252,7 @@ def itertests(test_patterns, parser, script_name, comment_prefix=None, argparse_
       if not is_regenerate:
         argv.insert(1, '--version=' + str(DEFAULT_VERSION))
 
-      args = parser.parse_args(argv[1:])
+      args = parse_args(parser, argv[1:])
       if argparse_callback is not None:
         argparse_callback(args)
       if is_regenerate:
@@ -1199,6 +1205,8 @@ def get_autogennote_suffix(parser, args):
         continue
     if parser.get_default(action.dest) == value:
       continue  # Don't add default values
+    if action.dest == 'function_signature' and args.version >= 2:
+      continue # Enabled by default in version 2
     if action.dest == 'filters':
       # Create a separate option for each filter element.  The value is a list
       # of Filter objects.
@@ -1225,7 +1233,7 @@ def check_for_command(line, parser, args, argv, argparse_callback):
     for option in shlex.split(cmd_m.group('cmd').strip()):
       if option:
         argv.append(option)
-    args = parser.parse_args(filter(lambda arg: arg not in args.tests, argv))
+    args = parse_args(parser, filter(lambda arg: arg not in args.tests, argv))
     if argparse_callback is not None:
       argparse_callback(args)
   return args, argv
diff --git a/llvm/utils/chunk-print-before-all.py b/llvm/utils/chunk-print-before-all.py
index 251c789295727..b5756c59c2719 100755
--- a/llvm/utils/chunk-print-before-all.py
+++ b/llvm/utils/chunk-print-before-all.py
@@ -19,7 +19,7 @@ def get_pass_name(line, prefix):
 
 def print_chunk(lines, prefix, pass_name):
     global chunk_id
-    fname = str(chunk_id) + "-" + prefix + "-" + pass_name + ".ll"
+    fname = str(chunk_id).zfill(4) + "-" + prefix + "-" + pass_name + ".ll"
     chunk_id = chunk_id + 1
     print("writing chunk " + fname + " (" + str(len(lines)) + " lines)")
     with open(fname, "w") as f:
diff --git a/llvm/utils/gn/build/toolchain/target_flags.gni b/llvm/utils/gn/build/toolchain/target_flags.gni
index 92da8774e935d..a8521feeee371 100644
--- a/llvm/utils/gn/build/toolchain/target_flags.gni
+++ b/llvm/utils/gn/build/toolchain/target_flags.gni
@@ -14,6 +14,7 @@ if (current_os == "android") {
     "--target=$llvm_current_triple",
     "--sysroot=$android_ndk_path/toolchains/llvm/prebuilt/linux-x86_64/sysroot",
     "--gcc-toolchain=$android_ndk_path/toolchains/llvm/prebuilt/linux-x86_64",
+    "-fno-emulated-tls",
   ]
   target_ldflags += [
     "-static-libstdc++",
diff --git a/llvm/utils/gn/secondary/clang-tools-extra/clang-tidy/cppcoreguidelines/BUILD.gn b/llvm/utils/gn/secondary/clang-tools-extra/clang-tidy/cppcoreguidelines/BUILD.gn
index f8cc60c21b9f1..37a33fb3ef721 100644
--- a/llvm/utils/gn/secondary/clang-tools-extra/clang-tidy/cppcoreguidelines/BUILD.gn
+++ b/llvm/utils/gn/secondary/clang-tools-extra/clang-tidy/cppcoreguidelines/BUILD.gn
@@ -17,6 +17,7 @@ static_library("cppcoreguidelines") {
   ]
   sources = [
     "AvoidCaptureDefaultWhenCapturingThisCheck.cpp",
+    "AvoidCapturingLambdaCoroutinesCheck.cpp",
     "AvoidConstOrRefDataMembersCheck.cpp",
     "AvoidDoWhileCheck.cpp",
     "AvoidGotoCheck.cpp",
@@ -40,6 +41,7 @@ static_library("cppcoreguidelines") {
     "ProTypeStaticCastDowncastCheck.cpp",
     "ProTypeUnionAccessCheck.cpp",
     "ProTypeVarargCheck.cpp",
+    "RvalueReferenceParamNotMovedCheck.cpp",
     "SlicingCheck.cpp",
     "SpecialMemberFunctionsCheck.cpp",
     "VirtualClassDestructorCheck.cpp",
diff --git a/llvm/utils/gn/secondary/clang-tools-extra/clangd/unittests/BUILD.gn b/llvm/utils/gn/secondary/clang-tools-extra/clangd/unittests/BUILD.gn
index e65b2e73ad2cc..65fb1659f70fb 100644
--- a/llvm/utils/gn/secondary/clang-tools-extra/clangd/unittests/BUILD.gn
+++ b/llvm/utils/gn/secondary/clang-tools-extra/clangd/unittests/BUILD.gn
@@ -23,6 +23,7 @@ unittest("ClangdTests") {
     "//clang/lib/Lex",
     "//clang/lib/Sema",
     "//clang/lib/Serialization",
+    "//clang/lib/Testing",
     "//clang/lib/Tooling",
     "//clang/lib/Tooling/Core",
     "//clang/lib/Tooling/Inclusions",
@@ -119,6 +120,7 @@ unittest("ClangdTests") {
     "XRefsTests.cpp",
     "support/CancellationTests.cpp",
     "support/ContextTests.cpp",
+    "support/FileCacheTests.cpp",
     "support/FunctionTests.cpp",
     "support/MarkupTests.cpp",
     "support/MemoryTreeTests.cpp",
@@ -144,6 +146,7 @@ unittest("ClangdTests") {
     "tweaks/RawStringLiteralTests.cpp",
     "tweaks/RemoveUsingNamespaceTests.cpp",
     "tweaks/ShowSelectionTreeTests.cpp",
+    "tweaks/SpecialMembersTests.cpp",
     "tweaks/SwapIfBranchesTests.cpp",
     "tweaks/TweakTesting.cpp",
     "tweaks/TweakTests.cpp",
diff --git a/llvm/utils/gn/secondary/clang/lib/Basic/BUILD.gn b/llvm/utils/gn/secondary/clang/lib/Basic/BUILD.gn
index 664dabfb4eac4..639276f48c84b 100644
--- a/llvm/utils/gn/secondary/clang/lib/Basic/BUILD.gn
+++ b/llvm/utils/gn/secondary/clang/lib/Basic/BUILD.gn
@@ -78,6 +78,7 @@ static_library("Basic") {
     "OpenCLOptions.cpp",
     "OpenMPKinds.cpp",
     "OperatorPrecedence.cpp",
+    "ParsedAttrInfo.cpp",
     "ProfileList.cpp",
     "SanitizerSpecialCaseList.cpp",
     "Sanitizers.cpp",
diff --git a/llvm/utils/gn/secondary/compiler-rt/lib/profile/BUILD.gn b/llvm/utils/gn/secondary/compiler-rt/lib/profile/BUILD.gn
index 21e3ae8d20284..f59c49fdbbdf0 100644
--- a/llvm/utils/gn/secondary/compiler-rt/lib/profile/BUILD.gn
+++ b/llvm/utils/gn/secondary/compiler-rt/lib/profile/BUILD.gn
@@ -38,6 +38,7 @@ static_library("profile") {
     "InstrProfilingMerge.c",
     "InstrProfilingMergeFile.c",
     "InstrProfilingNameVar.c",
+    "InstrProfilingPlatformAIX.c",
     "InstrProfilingPlatformDarwin.c",
     "InstrProfilingPlatformFuchsia.c",
     "InstrProfilingPlatformLinux.c",
diff --git a/llvm/utils/gn/secondary/compiler-rt/lib/scudo/BUILD.gn b/llvm/utils/gn/secondary/compiler-rt/lib/scudo/BUILD.gn
index f928f0bbb35c4..19ddd4ea89c00 100644
--- a/llvm/utils/gn/secondary/compiler-rt/lib/scudo/BUILD.gn
+++ b/llvm/utils/gn/secondary/compiler-rt/lib/scudo/BUILD.gn
@@ -14,8 +14,8 @@ group("scudo") {
       "standalone/benchmarks:ScudoBenchmarks($toolchain)",
       "standalone/tests:ScudoCUnitTest($toolchain)",
       "standalone/tests:ScudoCxxUnitTest($toolchain)",
-      "standalone/tests:ScudoUnitTest($toolchain)",
       "standalone/tests:ScudoHooksUnitTest($toolchain)",
+      "standalone/tests:ScudoUnitTest($toolchain)",
     ]
   }
   testonly = true
diff --git a/llvm/utils/gn/secondary/libcxx/include/BUILD.gn b/llvm/utils/gn/secondary/libcxx/include/BUILD.gn
index 1c8424ad2c90a..764f12ce21ec2 100644
--- a/llvm/utils/gn/secondary/libcxx/include/BUILD.gn
+++ b/llvm/utils/gn/secondary/libcxx/include/BUILD.gn
@@ -300,10 +300,15 @@ if (current_toolchain == default_toolchain) {
       "__bsd_locale_defaults.h",
       "__bsd_locale_fallbacks.h",
       "__charconv/chars_format.h",
+      "__charconv/from_chars_integral.h",
       "__charconv/from_chars_result.h",
       "__charconv/tables.h",
+      "__charconv/to_chars.h",
       "__charconv/to_chars_base_10.h",
+      "__charconv/to_chars_floating_point.h",
+      "__charconv/to_chars_integral.h",
       "__charconv/to_chars_result.h",
+      "__charconv/traits.h",
       "__chrono/calendar.h",
       "__chrono/concepts.h",
       "__chrono/convert_to_timespec.h",
@@ -372,6 +377,11 @@ if (current_toolchain == default_toolchain) {
       "__debug",
       "__debug_utils/randomize_range.h",
       "__errc",
+      "__exception/exception.h",
+      "__exception/exception_ptr.h",
+      "__exception/nested_exception.h",
+      "__exception/operations.h",
+      "__exception/terminate.h",
       "__expected/bad_expected_access.h",
       "__expected/expected.h",
       "__expected/unexpect.h",
diff --git a/llvm/utils/gn/secondary/libunwind/src/BUILD.gn b/llvm/utils/gn/secondary/libunwind/src/BUILD.gn
index 87bde59010913..f4aa67afbe13c 100644
--- a/llvm/utils/gn/secondary/libunwind/src/BUILD.gn
+++ b/llvm/utils/gn/secondary/libunwind/src/BUILD.gn
@@ -91,6 +91,7 @@ if (libunwind_enable_shared && current_os != "android") {
       ]
     }
     if (current_os == "mac") {
+      cflags += [ "-fno-stack-protector" ]
       ldflags += [
         "-Wl,-compatibility_version,1",
         "-Wl,-install_name,/usr/lib/libunwind.1.dylib",
diff --git a/llvm/utils/gn/secondary/lldb/source/Plugins/BUILD.gn b/llvm/utils/gn/secondary/lldb/source/Plugins/BUILD.gn
index ca5e55453365e..83d4898cf53f3 100644
--- a/llvm/utils/gn/secondary/lldb/source/Plugins/BUILD.gn
+++ b/llvm/utils/gn/secondary/lldb/source/Plugins/BUILD.gn
@@ -104,6 +104,10 @@ lldb_plugins += [
     "//lldb/source/Plugins/ObjectFile/PECOFF",
     "ObjectFilePECOFF",
   ],
+  [
+    "//lldb/source/Plugins/ObjectFile/Placeholder",
+    "ObjectFilePlaceholder",
+  ],
   [
     "//lldb/source/Plugins/ObjectFile/wasm",
     "ObjectFileWasm",
diff --git a/llvm/utils/gn/secondary/lldb/source/Plugins/ObjectFile/Placeholder/BUILD.gn b/llvm/utils/gn/secondary/lldb/source/Plugins/ObjectFile/Placeholder/BUILD.gn
new file mode 100644
index 0000000000000..60d42c4c7297e
--- /dev/null
+++ b/llvm/utils/gn/secondary/lldb/source/Plugins/ObjectFile/Placeholder/BUILD.gn
@@ -0,0 +1,13 @@
+static_library("Placeholder") {
+  output_name = "lldbPluginObjectFilePlaceholder"
+  configs += [ "//llvm/utils/gn/build:lldb_code" ]
+  deps = [
+    "//lldb/source/Core",
+    "//lldb/source/Host",
+    "//lldb/source/Symbol",
+    "//lldb/source/Target",
+    "//llvm/lib/Support",
+    "//llvm/lib/TargetParser",
+  ]
+  sources = [ "ObjectFilePlaceholder.cpp" ]
+}
diff --git a/llvm/utils/gn/secondary/lldb/source/Plugins/Process/POSIX/BUILD.gn b/llvm/utils/gn/secondary/lldb/source/Plugins/Process/POSIX/BUILD.gn
index dad3128ce224a..c830b7c804b04 100644
--- a/llvm/utils/gn/secondary/lldb/source/Plugins/Process/POSIX/BUILD.gn
+++ b/llvm/utils/gn/secondary/lldb/source/Plugins/Process/POSIX/BUILD.gn
@@ -12,7 +12,6 @@ static_library("POSIX") {
   sources = [
     "CrashReason.cpp",
     "NativeProcessELF.cpp",
-    "ProcessMessage.cpp",
     "ProcessPOSIXLog.cpp",
   ]
 }
diff --git a/llvm/utils/gn/secondary/lldb/source/Plugins/Process/Utility/BUILD.gn b/llvm/utils/gn/secondary/lldb/source/Plugins/Process/Utility/BUILD.gn
index db4600e315a7a..4f50b8fa5964e 100644
--- a/llvm/utils/gn/secondary/lldb/source/Plugins/Process/Utility/BUILD.gn
+++ b/llvm/utils/gn/secondary/lldb/source/Plugins/Process/Utility/BUILD.gn
@@ -27,7 +27,6 @@ static_library("Utility") {
     "LinuxProcMaps.cpp",
     "LinuxSignals.cpp",
     "MemoryTagManagerAArch64MTE.cpp",
-    "MipsLinuxSignals.cpp",
     "NativeProcessSoftwareSingleStep.cpp",
     "NativeRegisterContextDBReg_arm64.cpp",
     "NativeRegisterContextDBReg_x86.cpp",
diff --git a/llvm/utils/gn/secondary/llvm/lib/Analysis/BUILD.gn b/llvm/utils/gn/secondary/llvm/lib/Analysis/BUILD.gn
index 940df6e0b76bd..23dd1cbd79c36 100644
--- a/llvm/utils/gn/secondary/llvm/lib/Analysis/BUILD.gn
+++ b/llvm/utils/gn/secondary/llvm/lib/Analysis/BUILD.gn
@@ -76,7 +76,6 @@ static_library("Analysis") {
     "Lint.cpp",
     "Loads.cpp",
     "Local.cpp",
-    "LogicCombine.cpp",
     "LoopAccessAnalysis.cpp",
     "LoopAnalysisManager.cpp",
     "LoopCacheAnalysis.cpp",
diff --git a/llvm/utils/gn/secondary/llvm/lib/Target/X86/BUILD.gn b/llvm/utils/gn/secondary/llvm/lib/Target/X86/BUILD.gn
index 74358e484131d..e2351f36ed1b1 100644
--- a/llvm/utils/gn/secondary/llvm/lib/Target/X86/BUILD.gn
+++ b/llvm/utils/gn/secondary/llvm/lib/Target/X86/BUILD.gn
@@ -93,8 +93,8 @@ static_library("LLVMX86CodeGen") {
     "X86FastPreTileConfig.cpp",
     "X86FastTileConfig.cpp",
     "X86FixupBWInsts.cpp",
-    "X86FixupLEAs.cpp",
     "X86FixupInstTuning.cpp",
+    "X86FixupLEAs.cpp",
     "X86FixupSetCC.cpp",
     "X86FlagsCopyLowering.cpp",
     "X86FloatingPoint.cpp",
diff --git a/llvm/utils/gn/secondary/llvm/lib/Transforms/IPO/BUILD.gn b/llvm/utils/gn/secondary/llvm/lib/Transforms/IPO/BUILD.gn
index e960be0f3fd65..644d30f10854e 100644
--- a/llvm/utils/gn/secondary/llvm/lib/Transforms/IPO/BUILD.gn
+++ b/llvm/utils/gn/secondary/llvm/lib/Transforms/IPO/BUILD.gn
@@ -52,7 +52,6 @@ static_library("IPO") {
     "ModuleInliner.cpp",
     "OpenMPOpt.cpp",
     "PartialInlining.cpp",
-    "PassManagerBuilder.cpp",
     "SCCP.cpp",
     "SampleContextTracker.cpp",
     "SampleProfile.cpp",
diff --git a/llvm/utils/gn/secondary/llvm/lib/Transforms/Utils/BUILD.gn b/llvm/utils/gn/secondary/llvm/lib/Transforms/Utils/BUILD.gn
index b3d84fb813036..035ed661dce39 100644
--- a/llvm/utils/gn/secondary/llvm/lib/Transforms/Utils/BUILD.gn
+++ b/llvm/utils/gn/secondary/llvm/lib/Transforms/Utils/BUILD.gn
@@ -11,7 +11,6 @@ static_library("Utils") {
     "ASanStackFrameLayout.cpp",
     "AddDiscriminators.cpp",
     "AssumeBundleBuilder.cpp",
-    "CountVisits.cpp",
     "BasicBlockUtils.cpp",
     "BreakCriticalEdges.cpp",
     "BuildLibCalls.cpp",
@@ -25,6 +24,7 @@ static_library("Utils") {
     "CodeExtractor.cpp",
     "CodeLayout.cpp",
     "CodeMoverUtils.cpp",
+    "CountVisits.cpp",
     "CtorUtils.cpp",
     "Debugify.cpp",
     "DemoteRegToStack.cpp",
diff --git a/llvm/utils/gn/secondary/llvm/unittests/Analysis/BUILD.gn b/llvm/utils/gn/secondary/llvm/unittests/Analysis/BUILD.gn
index d946c57e338fc..dbeed697ce497 100644
--- a/llvm/utils/gn/secondary/llvm/unittests/Analysis/BUILD.gn
+++ b/llvm/utils/gn/secondary/llvm/unittests/Analysis/BUILD.gn
@@ -41,6 +41,7 @@ unittest("AnalysisTests") {
     "MemorySSATest.cpp",
     "PhiValuesTest.cpp",
     "PluginInlineAdvisorAnalysisTest.cpp",
+    "PluginInlineOrderAnalysisTest.cpp",
     "ProfileSummaryInfoTest.cpp",
     "ScalarEvolutionTest.cpp",
     "SparsePropagation.cpp",
@@ -58,7 +59,10 @@ unittest("AnalysisTests") {
   # Otherwise, reconfiguring with plugins disabled will leave behind a stale
   # executable.
   if (host_os != "win") {
-    deps += [ "InlineAdvisorPlugin" ]
+    deps += [
+      "InlineAdvisorPlugin",
+      "InlineOrderPlugin",
+    ]
   }
 
   # Support plugins.
diff --git a/llvm/utils/gn/secondary/llvm/unittests/Analysis/InlineOrderPlugin/BUILD.gn b/llvm/utils/gn/secondary/llvm/unittests/Analysis/InlineOrderPlugin/BUILD.gn
new file mode 100644
index 0000000000000..0c5c0c5c6619e
--- /dev/null
+++ b/llvm/utils/gn/secondary/llvm/unittests/Analysis/InlineOrderPlugin/BUILD.gn
@@ -0,0 +1,23 @@
+if (host_os != "win") {
+  loadable_module("InlineOrderPlugin") {
+    # Put plugin next to the unit test executable.
+    output_dir =
+        rebase_path(get_label_info("//llvm/unittests/Analysis:AnalysisTests",
+                                   "target_out_dir"))
+
+    sources = [ "InlineOrderPlugin.cpp" ]
+
+    deps = [
+      # TestPlugin doesn't want to link in any LLVM code, it just needs its
+      # headers.
+      "//llvm/include/llvm/IR:public_tablegen",
+    ]
+
+    if (host_os != "mac" && host_os != "win") {
+      # The GN build currently doesn't globally pass -fPIC, but that's
+      # needed for building .so files on ELF.  Just pass it manually
+      # for loadable_modules for now.
+      cflags = [ "-fPIC" ]
+    }
+  }
+}
diff --git a/llvm/utils/gn/secondary/llvm/unittests/Target/AMDGPU/BUILD.gn b/llvm/utils/gn/secondary/llvm/unittests/Target/AMDGPU/BUILD.gn
index 75ecb26f4c7e1..be5ddad1caa2d 100644
--- a/llvm/utils/gn/secondary/llvm/unittests/Target/AMDGPU/BUILD.gn
+++ b/llvm/utils/gn/secondary/llvm/unittests/Target/AMDGPU/BUILD.gn
@@ -7,8 +7,8 @@ unittest("AMDGPUTests") {
 
     # AMDGPUBaseInfo.cpp includes GCNSubtarget.h which after 490e348e679
     # includes the generated AMDGPUGenRegisterBank.inc file :/
-    "//llvm/lib/Target/AMDGPU/:AMDGPUGenRegisterBank",
     "//llvm/lib/Target/AMDGPU:LLVMAMDGPUCodeGen",
+    "//llvm/lib/Target/AMDGPU/:AMDGPUGenRegisterBank",
     "//llvm/lib/Target/AMDGPU/MCTargetDesc",
     "//llvm/lib/Target/AMDGPU/TargetInfo",
     "//llvm/lib/Target/AMDGPU/Utils",
diff --git a/llvm/utils/gn/secondary/llvm/utils/TableGen/BUILD.gn b/llvm/utils/gn/secondary/llvm/utils/TableGen/BUILD.gn
index b0abe638c528a..a9352fd128df0 100644
--- a/llvm/utils/gn/secondary/llvm/utils/TableGen/BUILD.gn
+++ b/llvm/utils/gn/secondary/llvm/utils/TableGen/BUILD.gn
@@ -17,6 +17,7 @@ executable("llvm-tblgen") {
     "CodeGenHwModes.cpp",
     "CodeGenInstAlias.cpp",
     "CodeGenInstruction.cpp",
+    "CodeGenIntrinsics.cpp",
     "CodeGenMapTable.cpp",
     "CodeGenRegisters.cpp",
     "CodeGenSchedule.cpp",
diff --git a/llvm/utils/kate/llvm-tablegen.xml b/llvm/utils/kate/llvm-tablegen.xml
index 38bb3083a0daf..486d373a1186d 100644
--- a/llvm/utils/kate/llvm-tablegen.xml
+++ b/llvm/utils/kate/llvm-tablegen.xml
@@ -42,6 +42,8 @@
       <item> !ge </item>
       <item> !gt </item>
       <item> !ne </item>
+      <item> !tolower </item>
+      <item> !toupper </item>
     </list>
     <list name="objects">
       <item> class </item>
diff --git a/llvm/utils/lit/README.rst b/llvm/utils/lit/README.rst
index b8befef4fbe00..3a1457057ac79 100644
--- a/llvm/utils/lit/README.rst
+++ b/llvm/utils/lit/README.rst
@@ -41,7 +41,7 @@ Please browse the issues labeled *tools:llvm-lit* in LLVM's issue tracker for
 ideas on what to work on:
 https://github.com/llvm/llvm-project/labels/tools%3Allvm-lit
 
-Before submitting patches, run the test suite to ensure nothing has regressed:
+Before submitting patches, run the test suite to ensure nothing has regressed::
 
     # From within your LLVM source directory.
     utils/lit/lit.py \
diff --git a/mlir/cmake/modules/AddMLIR.cmake b/mlir/cmake/modules/AddMLIR.cmake
index bbc5c2a42358f..2b12530bda985 100644
--- a/mlir/cmake/modules/AddMLIR.cmake
+++ b/mlir/cmake/modules/AddMLIR.cmake
@@ -495,7 +495,9 @@ function(add_mlir_aggregate name)
   # many other platforms are more strict. We want these libraries to be
   # self contained, and we want any undefined symbols to be reported at
   # library construction time, not at library use, so make Linux strict too.
-  if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
+  # We make an exception for sanitizer builds, since the AddressSanitizer
+  # run-time doesn't get linked into shared libraries.
+  if((CMAKE_SYSTEM_NAME STREQUAL "Linux") AND (NOT LLVM_USE_SANITIZER))
     target_link_options(${name} PRIVATE
       "LINKER:-z,defs"
     )
diff --git a/mlir/docs/BytecodeFormat.md b/mlir/docs/BytecodeFormat.md
index 4d061e9f08d36..b4f7400274f43 100644
--- a/mlir/docs/BytecodeFormat.md
+++ b/mlir/docs/BytecodeFormat.md
@@ -6,7 +6,8 @@ This documents describes the MLIR bytecode format and its encoding.
 
 ## Magic Number
 
-MLIR uses the following four-byte magic number to indicate bytecode files:
+MLIR uses the following four-byte magic number to
+indicate bytecode files:
 
 '\[‘M’<sub>8</sub>, ‘L’<sub>8</sub>, ‘ï’<sub>8</sub>, ‘R’<sub>8</sub>\]'
 
@@ -157,16 +158,25 @@ dialect_section {
 }
 
 op_name_group {
-  dialect: varint,
+  dialect: varint // (dialectID << 1) | (hasVersion),
+  version : dialect_version_section
   numOpNames: varint,
   opNames: varint[]
 }
+
+dialect_version_section {
+  size: varint,
+  version: byte[]
+}
+
 ```
 
-Dialects are encoded as indexes to the name string within the string section.
-Operation names are encoded in groups by dialect, with each group containing the
-dialect, the number of operation names, and the array of indexes to each name
-within the string section.
+Dialects are encoded as a `varint` containing the index to the name string
+within the string section, plus a flag indicating whether the dialect is
+versioned. Operation names are encoded in groups by dialect, with each group
+containing the dialect, the number of operation names, and the array of indexes
+to each name within the string section. The version is encoded as a nested
+section.
 
 ### Attribute/Type Sections
 
diff --git a/mlir/docs/LangRef.md b/mlir/docs/LangRef.md
index 07f804018c2aa..467c1987ce1ac 100644
--- a/mlir/docs/LangRef.md
+++ b/mlir/docs/LangRef.md
@@ -845,3 +845,18 @@ The [builtin dialect](Dialects/Builtin.md) defines a set of attribute values
 that are directly usable by any other dialect in MLIR. These types cover a range
 from primitive integer and floating-point values, attribute dictionaries, dense
 multi-dimensional arrays, and more.
+
+### IR Versionning
+
+A dialect can opt-in to handle versioning through the
+`BytecodeDialectInterface`. Few hooks are exposed to the dialect to allow
+managing a version encoded into the bytecode file. The version is loaded lazily
+and allows to retrieve the version information while parsing the input IR, and
+gives an opportunity to each dialect for which a version is present to perform
+IR upgrades post-parsing through the `upgradeFromVersion` method. Custom
+Attribute and Type encodings can also be upgraded according to the dialect
+version using readAttribute and readType methods.
+
+There is no restriction on what kind of information a dialect is allowed to
+encode to model its versioning. Currently, versioning is supported only for
+bytecode formats.
diff --git a/mlir/examples/standalone/test/Standalone/standalone-opt.mlir b/mlir/examples/standalone/test/Standalone/standalone-opt.mlir
index 1a78a9d8cb9ec..101f9015f393d 100644
--- a/mlir/examples/standalone/test/Standalone/standalone-opt.mlir
+++ b/mlir/examples/standalone/test/Standalone/standalone-opt.mlir
@@ -1,3 +1,3 @@
 // RUN: standalone-opt --show-dialects | FileCheck %s
 // CHECK: Available Dialects:
-// CHECK: standalone
+// CHECK-SAME: standalone
diff --git a/mlir/include/mlir-c/Dialect/SparseTensor.h b/mlir/include/mlir-c/Dialect/SparseTensor.h
index 8027f319b28cf..7d560dd80a90a 100644
--- a/mlir/include/mlir-c/Dialect/SparseTensor.h
+++ b/mlir/include/mlir-c/Dialect/SparseTensor.h
@@ -41,40 +41,40 @@ enum MlirSparseTensorDimLevelType {
 // SparseTensorEncodingAttr
 //===----------------------------------------------------------------------===//
 
-/// Checks whether the given attribute is a sparse_tensor.encoding attribute.
+/// Checks whether the given attribute is a `sparse_tensor.encoding` attribute.
 MLIR_CAPI_EXPORTED bool
 mlirAttributeIsASparseTensorEncodingAttr(MlirAttribute attr);
 
-/// Creates a sparse_tensor.encoding attribute with the given parameters.
+/// Creates a `sparse_tensor.encoding` attribute with the given parameters.
 MLIR_CAPI_EXPORTED MlirAttribute mlirSparseTensorEncodingAttrGet(
-    MlirContext ctx, intptr_t numDimLevelTypes,
+    MlirContext ctx, intptr_t lvlRank,
     enum MlirSparseTensorDimLevelType const *dimLevelTypes,
-    MlirAffineMap dimOrdering, MlirAffineMap higherOrdering,
-    int pointerBitWidth, int indexBitWidth);
+    MlirAffineMap dimOrdering, MlirAffineMap higherOrdering, int posWidth,
+    int crdWidth);
 
-/// Returns the number of dim level types in a sparse_tensor.encoding attribute.
+/// Returns the level-rank of the `sparse_tensor.encoding` attribute.
 MLIR_CAPI_EXPORTED intptr_t
-mlirSparseTensorEncodingGetNumDimLevelTypes(MlirAttribute attr);
+mlirSparseTensorEncodingGetLvlRank(MlirAttribute attr);
 
-/// Returns a specified dim level type in a sparse_tensor.encoding attribute.
+/// Returns a specified level-type of the `sparse_tensor.encoding` attribute.
 MLIR_CAPI_EXPORTED enum MlirSparseTensorDimLevelType
-mlirSparseTensorEncodingAttrGetDimLevelType(MlirAttribute attr, intptr_t pos);
+mlirSparseTensorEncodingAttrGetDimLevelType(MlirAttribute attr, intptr_t lvl);
 
-/// Returns the dimension ordering in a sparse_tensor.encoding attribute.
+/// Returns the dimension-ordering of the `sparse_tensor.encoding` attribute.
 MLIR_CAPI_EXPORTED MlirAffineMap
 mlirSparseTensorEncodingAttrGetDimOrdering(MlirAttribute attr);
 
-/// Returns the higher ordering in a sparse_tensor.encoding attribute.
+/// Returns the higher-ordering of the `sparse_tensor.encoding` attribute.
 MLIR_CAPI_EXPORTED MlirAffineMap
 mlirSparseTensorEncodingAttrGetHigherOrdering(MlirAttribute attr);
 
-/// Returns the pointer bit width in a sparse_tensor.encoding attribute.
+/// Returns the position bitwidth of the `sparse_tensor.encoding` attribute.
 MLIR_CAPI_EXPORTED int
-mlirSparseTensorEncodingAttrGetPointerBitWidth(MlirAttribute attr);
+mlirSparseTensorEncodingAttrGetPosWidth(MlirAttribute attr);
 
-/// Returns the index bit width in a sparse_tensor.encoding attribute.
+/// Returns the coordinate bitwidth of the `sparse_tensor.encoding` attribute.
 MLIR_CAPI_EXPORTED int
-mlirSparseTensorEncodingAttrGetIndexBitWidth(MlirAttribute attr);
+mlirSparseTensorEncodingAttrGetCrdWidth(MlirAttribute attr);
 
 #ifdef __cplusplus
 }
diff --git a/mlir/include/mlir/Analysis/Presburger/IntegerRelation.h b/mlir/include/mlir/Analysis/Presburger/IntegerRelation.h
index 527faf3d41812..347be26325e5a 100644
--- a/mlir/include/mlir/Analysis/Presburger/IntegerRelation.h
+++ b/mlir/include/mlir/Analysis/Presburger/IntegerRelation.h
@@ -203,27 +203,27 @@ class IntegerRelation {
   /// Get the number of vars of the specified kind.
   unsigned getNumVarKind(VarKind kind) const {
     return space.getNumVarKind(kind);
-  };
+  }
 
   /// Return the index at which the specified kind of vars starts.
   unsigned getVarKindOffset(VarKind kind) const {
     return space.getVarKindOffset(kind);
-  };
+  }
 
   /// Return the index at Which the specified kind of vars ends.
   unsigned getVarKindEnd(VarKind kind) const {
     return space.getVarKindEnd(kind);
-  };
+  }
 
   /// Get the number of elements of the specified kind in the range
   /// [varStart, varLimit).
   unsigned getVarKindOverlap(VarKind kind, unsigned varStart,
                              unsigned varLimit) const {
     return space.getVarKindOverlap(kind, varStart, varLimit);
-  };
+  }
 
   /// Return the VarKind of the var at the specified position.
-  VarKind getVarKindAt(unsigned pos) const { return space.getVarKindAt(pos); };
+  VarKind getVarKindAt(unsigned pos) const { return space.getVarKindAt(pos); }
 
   /// The struct CountsSnapshot stores the count of each VarKind, and also of
   /// each constraint type. getCounts() returns a CountsSnapshot object
@@ -493,7 +493,7 @@ class IntegerRelation {
     if (ub)
       *ub = getInt64Vec(ubMPInt);
     if (boundFloorDivisor)
-      *boundFloorDivisor = int64_t(boundFloorDivisorMPInt);
+      *boundFloorDivisor = static_cast<int64_t>(boundFloorDivisorMPInt);
     return llvm::transformOptional(result, int64FromMPInt);
   }
 
diff --git a/mlir/include/mlir/Bytecode/BytecodeImplementation.h b/mlir/include/mlir/Bytecode/BytecodeImplementation.h
index 1ae839d5f40e7..ea9bcad735b36 100644
--- a/mlir/include/mlir/Bytecode/BytecodeImplementation.h
+++ b/mlir/include/mlir/Bytecode/BytecodeImplementation.h
@@ -235,6 +235,17 @@ class DialectBytecodeWriter {
   virtual void writeOwnedBlob(ArrayRef<char> blob) = 0;
 };
 
+//===--------------------------------------------------------------------===//
+// Dialect Version Interface.
+//===--------------------------------------------------------------------===//
+
+/// This class is used to represent the version of a dialect, for the purpose
+/// of polymorphic destruction.
+class DialectVersion {
+public:
+  virtual ~DialectVersion() = default;
+};
+
 //===----------------------------------------------------------------------===//
 // BytecodeDialectInterface
 //===----------------------------------------------------------------------===//
@@ -256,6 +267,19 @@ class BytecodeDialectInterface
     return Attribute();
   }
 
+  /// Read a versioned attribute encoding belonging to this dialect from the
+  /// given reader. This method should return null in the case of failure, and
+  /// falls back to the non-versioned reader in case the dialect implements
+  /// versioning but it does not support versioned custom encodings for the
+  /// attributes.
+  virtual Attribute readAttribute(DialectBytecodeReader &reader,
+                                  const DialectVersion &version) const {
+    reader.emitError()
+        << "dialect " << getDialect()->getNamespace()
+        << " does not support reading versioned attributes from bytecode";
+    return Attribute();
+  }
+
   /// Read a type belonging to this dialect from the given reader. This method
   /// should return null in the case of failure.
   virtual Type readType(DialectBytecodeReader &reader) const {
@@ -264,6 +288,19 @@ class BytecodeDialectInterface
     return Type();
   }
 
+  /// Read a versioned type encoding belonging to this dialect from the given
+  /// reader. This method should return null in the case of failure, and
+  /// falls back to the non-versioned reader in case the dialect implements
+  /// versioning but it does not support versioned custom encodings for the
+  /// types.
+  virtual Type readType(DialectBytecodeReader &reader,
+                        const DialectVersion &version) const {
+    reader.emitError()
+        << "dialect " << getDialect()->getNamespace()
+        << " does not support reading versioned types from bytecode";
+    return Type();
+  }
+
   //===--------------------------------------------------------------------===//
   // Writing
   //===--------------------------------------------------------------------===//
@@ -285,6 +322,27 @@ class BytecodeDialectInterface
                                   DialectBytecodeWriter &writer) const {
     return failure();
   }
+
+  /// Write the version of this dialect to the given writer.
+  virtual void writeVersion(DialectBytecodeWriter &writer) const {}
+
+  // Read the version of this dialect from the provided reader and return it as
+  // a `unique_ptr` to a dialect version object.
+  virtual std::unique_ptr<DialectVersion>
+  readVersion(DialectBytecodeReader &reader) const {
+    reader.emitError("Dialect does not support versioning");
+    return nullptr;
+  }
+
+  /// Hook invoked after parsing completed, if a version directive was present
+  /// and included an entry for the current dialect. This hook offers the
+  /// opportunity to the dialect to visit the IR and upgrades constructs emitted
+  /// by the version of the dialect corresponding to the provided version.
+  virtual LogicalResult
+  upgradeFromVersion(Operation *topLevelOp,
+                     const DialectVersion &version) const {
+    return success();
+  }
 };
 
 } // namespace mlir
diff --git a/mlir/include/mlir/Conversion/LLVMCommon/LoweringOptions.h b/mlir/include/mlir/Conversion/LLVMCommon/LoweringOptions.h
index a3338ebf982c2..bc45fcd84e098 100644
--- a/mlir/include/mlir/Conversion/LLVMCommon/LoweringOptions.h
+++ b/mlir/include/mlir/Conversion/LLVMCommon/LoweringOptions.h
@@ -33,7 +33,7 @@ class LowerToLLVMOptions {
   LowerToLLVMOptions(MLIRContext *ctx, const DataLayout &dl);
 
   bool useBarePtrCallConv = false;
-  bool useOpaquePointers = false;
+  bool useOpaquePointers = true;
 
   enum class AllocLowering {
     /// Use malloc for for heap allocations.
diff --git a/mlir/include/mlir/Conversion/Passes.td b/mlir/include/mlir/Conversion/Passes.td
index 700ed6109ae34..a958b967bb4b6 100644
--- a/mlir/include/mlir/Conversion/Passes.td
+++ b/mlir/include/mlir/Conversion/Passes.td
@@ -158,7 +158,7 @@ def ConvertAsyncToLLVMPass : Pass<"convert-async-to-llvm", "ModuleOp"> {
   ];
   let options = [
     Option<"useOpaquePointers", "use-opaque-pointers", "bool",
-           /*default=*/"false", "Generate LLVM IR using opaque pointers "
+           /*default=*/"true", "Generate LLVM IR using opaque pointers "
            "instead of typed pointers">,
   ];
 }
@@ -251,7 +251,7 @@ def ConvertControlFlowToLLVMPass : Pass<"convert-cf-to-llvm", "ModuleOp"> {
            /*default=kDeriveIndexBitwidthFromDataLayout*/"0",
            "Bitwidth of the index type, 0 to use size of machine word">,
     Option<"useOpaquePointers", "use-opaque-pointers", "bool",
-                   /*default=*/"false", "Generate LLVM IR using opaque pointers "
+                   /*default=*/"true", "Generate LLVM IR using opaque pointers "
                    "instead of typed pointers">,
   ];
 }
@@ -314,7 +314,7 @@ def ConvertFuncToLLVMPass : Pass<"convert-func-to-llvm", "ModuleOp"> {
            "String description (LLVM format) of the data layout that is "
            "expected on the produced module">,
     Option<"useOpaquePointers", "use-opaque-pointers", "bool",
-                       /*default=*/"false", "Generate LLVM IR using opaque pointers "
+                       /*default=*/"true", "Generate LLVM IR using opaque pointers "
                        "instead of typed pointers">,
   ];
 }
@@ -362,7 +362,7 @@ def GpuToLLVMConversionPass : Pass<"gpu-to-llvm", "ModuleOp"> {
                "Annotation attribute string for GPU binary"
                >,
     Option<"useOpaquePointers", "use-opaque-pointers", "bool",
-               /*default=*/"false", "Generate LLVM IR using opaque pointers "
+               /*default=*/"true", "Generate LLVM IR using opaque pointers "
                "instead of typed pointers">,
   ];
 
@@ -388,7 +388,7 @@ def LowerHostCodeToLLVMPass : Pass<"lower-host-to-llvm", "ModuleOp"> {
 
   let options = [
     Option<"useOpaquePointers", "use-opaque-pointers", "bool",
-                 /*default=*/"false", "Generate LLVM IR using opaque pointers "
+                 /*default=*/"true", "Generate LLVM IR using opaque pointers "
                  "instead of typed pointers">
   ];
 
@@ -414,7 +414,7 @@ def ConvertGpuOpsToNVVMOps : Pass<"convert-gpu-to-nvvm", "gpu::GPUModuleOp"> {
     Option<"hasRedux", "has-redux", "bool", /*default=*/"false",
            "Target gpu supports redux">,
     Option<"useOpaquePointers", "use-opaque-pointers", "bool",
-                   /*default=*/"false", "Generate LLVM IR using opaque pointers "
+                   /*default=*/"true", "Generate LLVM IR using opaque pointers "
                    "instead of typed pointers">,
   ];
 }
@@ -451,7 +451,7 @@ def ConvertGpuOpsToROCDLOps : Pass<"convert-gpu-to-rocdl", "gpu::GPUModuleOp"> {
             clEnumValN(::mlir::gpu::amd::Runtime::OpenCL, "OpenCL", "OpenCL")
           )}]>,
     Option<"useOpaquePointers", "use-opaque-pointers", "bool",
-               /*default=*/"false", "Generate LLVM IR using opaque pointers "
+               /*default=*/"true", "Generate LLVM IR using opaque pointers "
                "instead of typed pointers">,
   ];
 }
@@ -507,7 +507,7 @@ def ConvertVulkanLaunchFuncToVulkanCallsPass
 
   let options = [
      Option<"useOpaquePointers", "use-opaque-pointers", "bool",
-            /*default=*/"false", "Generate LLVM IR using opaque pointers "
+            /*default=*/"true", "Generate LLVM IR using opaque pointers "
             "instead of typed pointers">
   ];
 
@@ -545,11 +545,10 @@ def ConvertLinalgToLLVMPass : Pass<"convert-linalg-to-llvm", "ModuleOp"> {
   let summary = "Convert the operations from the linalg dialect into the LLVM "
                 "dialect";
   let dependentDialects = ["scf::SCFDialect", "LLVM::LLVMDialect"];
-
   let options = [
      Option<"useOpaquePointers", "use-opaque-pointers", "bool",
-            /*default=*/"false", "Generate LLVM IR using opaque pointers "
-            "instead of typed pointers">
+                /*default=*/"true", "Generate LLVM IR using opaque pointers "
+                "instead of typed pointers">
   ];
 }
 
@@ -657,7 +656,7 @@ def FinalizeMemRefToLLVMConversionPass :
            "Use generic allocation and deallocation functions instead of the "
            "classic 'malloc', 'aligned_alloc' and 'free' functions">,
     Option<"useOpaquePointers", "use-opaque-pointers", "bool",
-               /*default=*/"false", "Generate LLVM IR using opaque pointers "
+               /*default=*/"true", "Generate LLVM IR using opaque pointers "
                "instead of typed pointers">
   ];
 }
@@ -702,7 +701,7 @@ def ConvertNVGPUToNVVMPass : Pass<"convert-nvgpu-to-nvvm"> {
   ];
   let options = [
     Option<"useOpaquePointers", "use-opaque-pointers", "bool",
-              /*default=*/"false", "Generate LLVM IR using opaque pointers "
+              /*default=*/"true", "Generate LLVM IR using opaque pointers "
               "instead of typed pointers">
   ];
 }
@@ -727,7 +726,7 @@ def ConvertOpenACCToLLVMPass : Pass<"convert-openacc-to-llvm", "ModuleOp"> {
   let dependentDialects = ["LLVM::LLVMDialect"];
   let options = [
     Option<"useOpaquePointers", "use-opaque-pointers", "bool",
-           /*default=*/"false", "Generate LLVM IR using opaque pointers "
+           /*default=*/"true", "Generate LLVM IR using opaque pointers "
            "instead of typed pointers">,
   ];
 }
@@ -798,7 +797,7 @@ def ConvertSCFToOpenMPPass : Pass<"convert-scf-to-openmp", "ModuleOp"> {
 
   let options = [
     Option<"useOpaquePointers", "use-opaque-pointers", "bool",
-                 /*default=*/"false", "Generate LLVM IR using opaque pointers "
+                 /*default=*/"true", "Generate LLVM IR using opaque pointers "
                  "instead of typed pointers">
   ];
 
@@ -813,13 +812,12 @@ def ConvertSCFToOpenMPPass : Pass<"convert-scf-to-openmp", "ModuleOp"> {
 def SCFToSPIRV : Pass<"convert-scf-to-spirv"> {
   let summary = "Convert SCF dialect to SPIR-V dialect.";
   let description = [{
-    This pass converts SCF ops into SPIR-V structured control flow ops.
-    SPIR-V structured control flow ops does not support yielding values.
+    Converts SCF ops into SPIR-V structured control flow ops.
+    SPIR-V structured control flow ops do not support yielding values.
     So for SCF ops yielding values, SPIR-V variables are created for
     holding the values and load/store operations are emitted for updating
     them.
   }];
-  let constructor = "mlir::createConvertSCFToSPIRVPass()";
   let dependentDialects = ["spirv::SPIRVDialect"];
 }
 
@@ -888,7 +886,7 @@ def ConvertSPIRVToLLVMPass : Pass<"convert-spirv-to-llvm", "ModuleOp"> {
 
   let options = [
     Option<"useOpaquePointers", "use-opaque-pointers", "bool",
-                 /*default=*/"false", "Generate LLVM IR using opaque pointers "
+                 /*default=*/"true", "Generate LLVM IR using opaque pointers "
                  "instead of typed pointers">
   ];
 }
@@ -1098,7 +1096,7 @@ def ConvertVectorToLLVMPass : Pass<"convert-vector-to-llvm", "ModuleOp"> {
            "Enables the use of X86Vector dialect while lowering the vector "
 	   "dialect.">,
 	  Option<"useOpaquePointers", "use-opaque-pointers", "bool",
-               /*default=*/"false", "Generate LLVM IR using opaque pointers "
+               /*default=*/"true", "Generate LLVM IR using opaque pointers "
                "instead of typed pointers">
   ];
 }
diff --git a/mlir/include/mlir/Conversion/VectorToSPIRV/VectorToSPIRV.h b/mlir/include/mlir/Conversion/VectorToSPIRV/VectorToSPIRV.h
index 7b8882b36cd5d..f8c02c54066b8 100644
--- a/mlir/include/mlir/Conversion/VectorToSPIRV/VectorToSPIRV.h
+++ b/mlir/include/mlir/Conversion/VectorToSPIRV/VectorToSPIRV.h
@@ -23,6 +23,15 @@ class SPIRVTypeConverter;
 void populateVectorToSPIRVPatterns(SPIRVTypeConverter &typeConverter,
                                    RewritePatternSet &patterns);
 
+/// Appends patterns to convert vector reduction of the form:
+/// ```
+///   vector.reduction <add>, (muli (ext %lhs), (ext %rhs)), [%acc]
+/// ```
+///
+/// to SPIR-V integer dot product ops.
+void populateVectorReductionToSPIRVDotProductPatterns(
+    RewritePatternSet &patterns);
+
 } // namespace mlir
 
 #endif // MLIR_CONVERSION_VECTORTOSPIRV_VECTORTOSPIRV_H
diff --git a/mlir/include/mlir/Debug/BreakpointManager.h b/mlir/include/mlir/Debug/BreakpointManager.h
new file mode 100644
index 0000000000000..cc4be60cb69a3
--- /dev/null
+++ b/mlir/include/mlir/Debug/BreakpointManager.h
@@ -0,0 +1,95 @@
+//===- BreakpointManager.h - Breakpoint Manager Support ----*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_TRACING_BREAKPOINTMANAGER_H
+#define MLIR_TRACING_BREAKPOINTMANAGER_H
+
+#include "mlir/IR/Action.h"
+#include "llvm/ADT/MapVector.h"
+
+namespace mlir {
+namespace tracing {
+
+/// This abstract class represents a breakpoint.
+class Breakpoint {
+public:
+  virtual ~Breakpoint() = default;
+
+  /// TypeID for the subclass, used for casting purpose.
+  TypeID getTypeID() const { return typeID; }
+
+  bool isEnabled() const { return enableStatus; }
+  void enable() { enableStatus = true; }
+  void disable() { enableStatus = false; }
+  virtual void print(raw_ostream &os) const = 0;
+
+protected:
+  Breakpoint(TypeID typeID) : enableStatus(true), typeID(typeID) {}
+
+private:
+  /// The current state of the breakpoint. A breakpoint can be either enabled
+  /// or disabled.
+  bool enableStatus;
+  TypeID typeID;
+};
+
+inline raw_ostream &operator<<(raw_ostream &os, const Breakpoint &breakpoint) {
+  breakpoint.print(os);
+  return os;
+}
+
+/// This class provides a CRTP wrapper around a base breakpoint class to define
+/// a few necessary utility methods.
+template <typename Derived>
+class BreakpointBase : public Breakpoint {
+public:
+  /// Support isa/dyn_cast functionality for the derived pass class.
+  static bool classof(const Breakpoint *breakpoint) {
+    return breakpoint->getTypeID() == TypeID::get<Derived>();
+  }
+
+protected:
+  BreakpointBase() : Breakpoint(TypeID::get<Derived>()) {}
+};
+
+/// A breakpoint manager is responsible for managing a set of breakpoints and
+/// matching them to a given action.
+class BreakpointManager {
+public:
+  virtual ~BreakpointManager() = default;
+
+  /// TypeID for the subclass, used for casting purpose.
+  TypeID getTypeID() const { return typeID; }
+
+  /// Try to match a Breakpoint to a given Action. If there is a match and
+  /// the breakpoint is enabled, return the breakpoint. Otherwise, return
+  /// nullptr.
+  virtual Breakpoint *match(const Action &action) const = 0;
+
+protected:
+  BreakpointManager(TypeID typeID) : typeID(typeID) {}
+
+  TypeID typeID;
+};
+
+/// CRTP base class for BreakpointManager implementations.
+template <typename Derived>
+class BreakpointManagerBase : public BreakpointManager {
+public:
+  BreakpointManagerBase() : BreakpointManager(TypeID::get<Derived>()) {}
+
+  /// Provide classof to allow casting between breakpoint manager types.
+  static bool classof(const BreakpointManager *breakpointManager) {
+    return breakpointManager->getTypeID() == TypeID::get<Derived>();
+  }
+};
+
+} // namespace tracing
+} // namespace mlir
+
+#endif // MLIR_TRACING_BREAKPOINTMANAGER_H
diff --git a/mlir/include/mlir/Debug/BreakpointManagers/TagBreakpointManager.h b/mlir/include/mlir/Debug/BreakpointManagers/TagBreakpointManager.h
new file mode 100644
index 0000000000000..85fdb9a63286a
--- /dev/null
+++ b/mlir/include/mlir/Debug/BreakpointManagers/TagBreakpointManager.h
@@ -0,0 +1,65 @@
+//===- TagBreakpointManager.h - Simple breakpoint Support -------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DEBUG_BREAKPOINTMANAGERS_TAGBREAKPOINTMANAGER_H
+#define MLIR_DEBUG_BREAKPOINTMANAGERS_TAGBREAKPOINTMANAGER_H
+
+#include "mlir/Debug/BreakpointManager.h"
+#include "mlir/Debug/ExecutionContext.h"
+#include "mlir/IR/Action.h"
+#include "llvm/ADT/MapVector.h"
+
+namespace mlir {
+namespace tracing {
+
+/// Simple breakpoint matching an action "tag".
+class TagBreakpoint : public BreakpointBase<TagBreakpoint> {
+public:
+  TagBreakpoint(StringRef tag) : tag(tag) {}
+
+  void print(raw_ostream &os) const override { os << "Tag: `" << tag << '`'; }
+
+private:
+  /// A tag to associate the TagBreakpoint with.
+  std::string tag;
+
+  /// Allow access to `tag`.
+  friend class TagBreakpointManager;
+};
+
+/// This is a manager to store a collection of breakpoints that trigger
+/// on tags.
+class TagBreakpointManager
+    : public BreakpointManagerBase<TagBreakpointManager> {
+public:
+  Breakpoint *match(const Action &action) const override {
+    auto it = breakpoints.find(action.getTag());
+    if (it != breakpoints.end() && it->second->isEnabled())
+      return it->second.get();
+    return {};
+  }
+
+  /// Add a breakpoint to the manager for the given tag and return it.
+  /// If a breakpoint already exists for the given tag, return the existing
+  /// instance.
+  TagBreakpoint *addBreakpoint(StringRef tag) {
+    auto result = breakpoints.insert({tag, nullptr});
+    auto &it = result.first;
+    if (result.second)
+      it->second = std::make_unique<TagBreakpoint>(tag.str());
+    return it->second.get();
+  }
+
+private:
+  llvm::StringMap<std::unique_ptr<TagBreakpoint>> breakpoints;
+};
+
+} // namespace tracing
+} // namespace mlir
+
+#endif // MLIR_DEBUG_BREAKPOINTMANAGERS_TAGBREAKPOINTMANAGER_H
diff --git a/mlir/include/mlir/Support/DebugCounter.h b/mlir/include/mlir/Debug/Counter.h
similarity index 72%
rename from mlir/include/mlir/Support/DebugCounter.h
rename to mlir/include/mlir/Debug/Counter.h
index 5b6c015aadda5..622a87c9f6066 100644
--- a/mlir/include/mlir/Support/DebugCounter.h
+++ b/mlir/include/mlir/Debug/Counter.h
@@ -6,39 +6,40 @@
 //
 //===----------------------------------------------------------------------===//
 
-#ifndef MLIR_SUPPORT_DEBUGCOUNTER_H
-#define MLIR_SUPPORT_DEBUGCOUNTER_H
+#ifndef MLIR_TRACING_DEBUGCOUNTER_H
+#define MLIR_TRACING_DEBUGCOUNTER_H
 
-#include "mlir/Support/DebugAction.h"
+#include "mlir/IR/Action.h"
 #include "llvm/ADT/StringMap.h"
 #include <string>
 
 namespace mlir {
+namespace tracing {
 
-/// This class implements a debug action handler that attaches a counter value
+/// This class implements an action handler that attaches a counter value
 /// to debug actions and enables/disables execution of these action based on the
 /// value of the counter. The counter controls the execution of the action with
 /// a "skip" and "count" value. The "skip" value is used to skip a certain
-/// number of initial executions of a debug action. The "count" value is used to
-/// prevent a debug action from executing after it has executed for a set number
+/// number of initial executions of an action. The "count" value is used to
+/// prevent an action from executing after it has executed for a set number
 /// of times (not including any executions that have been skipped). For example,
-/// a counter for a debug action with `skip=47` and `count=2`, would skip the
+/// a counter for an action with `skip=47` and `count=2`, would skip the
 /// first 47 executions, then execute twice, and finally prevent any further
 /// executions.
-class DebugCounter : public DebugActionManager::GenericHandler {
+class DebugCounter {
 public:
   DebugCounter();
-  ~DebugCounter() override;
+  ~DebugCounter();
 
-  /// Add a counter for the given debug action tag. `countToSkip` is the number
+  /// Add a counter for the given action tag. `countToSkip` is the number
   /// of counter executions to skip before enabling execution of the action.
   /// `countToStopAfter` is the number of executions of the counter to allow
   /// before preventing the action from executing any more.
   void addCounter(StringRef actionTag, int64_t countToSkip,
                   int64_t countToStopAfter);
 
-  /// Register a counter with the specified name.
-  FailureOr<bool> shouldExecute(StringRef tag, StringRef description) final;
+  /// Entry point for handling actions.
+  void operator()(llvm::function_ref<void()> transform, const Action &action);
 
   /// Print the counters that have been registered with this instance to the
   /// provided output stream.
@@ -48,6 +49,9 @@ class DebugCounter : public DebugActionManager::GenericHandler {
   static void registerCLOptions();
 
 private:
+  // Returns true if the next action matching this tag should be executed.
+  bool shouldExecute(StringRef tag);
+
   /// Apply the registered CL options to this debug counter instance.
   void applyCLOptions();
 
@@ -68,6 +72,7 @@ class DebugCounter : public DebugActionManager::GenericHandler {
   llvm::StringMap<Counter> counters;
 };
 
+} // namespace tracing
 } // namespace mlir
 
-#endif
+#endif // MLIR_TRACING_DEBUGCOUNTER_H
diff --git a/mlir/include/mlir/Debug/ExecutionContext.h b/mlir/include/mlir/Debug/ExecutionContext.h
new file mode 100644
index 0000000000000..097ae372ca35b
--- /dev/null
+++ b/mlir/include/mlir/Debug/ExecutionContext.h
@@ -0,0 +1,132 @@
+//===- ExecutionContext.h -  Execution Context Support *- C++ -*-=============//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_TRACING_EXECUTIONCONTEXT_H
+#define MLIR_TRACING_EXECUTIONCONTEXT_H
+
+#include "mlir/Debug/BreakpointManager.h"
+#include "mlir/IR/Action.h"
+#include "llvm/ADT/SmallVector.h"
+
+namespace mlir {
+namespace tracing {
+
+/// This class is used to keep track of the active actions in the stack.
+/// It provides the current action but also access to the parent entry in the
+/// stack. This allows to keep track of the nested nature in which actions may
+/// be executed.
+struct ActionActiveStack {
+public:
+  ActionActiveStack(const ActionActiveStack *parent, const Action &action,
+                    int depth)
+      : parent(parent), action(action), depth(depth) {}
+  const ActionActiveStack *getParent() const { return parent; }
+  const Action &getAction() const { return action; }
+  int getDepth() const { return depth; }
+
+private:
+  const ActionActiveStack *parent;
+  const Action &action;
+  int depth;
+};
+
+/// The ExecutionContext is the main orchestration of the infrastructure, it
+/// acts as a handler in the MLIRContext for executing an Action. When an action
+/// is dispatched, it'll query its set of Breakpoints managers for a breakpoint
+/// matching this action. If a breakpoint is hit, it passes the action and the
+/// breakpoint information to a callback. The callback is responsible for
+/// controlling the execution of the action through an enum value it returns.
+/// Optionally, observers can be registered to be notified before and after the
+/// callback is executed.
+class ExecutionContext {
+public:
+  /// Enum that allows the client of the context to control the execution of the
+  /// action.
+  /// - Apply: The action is executed.
+  /// - Skip: The action is skipped.
+  /// - Step: The action is executed and the execution is paused before the next
+  ///         action, including for nested actions encountered before the
+  ///         current action finishes.
+  /// - Next: The action is executed and the execution is paused after the
+  ///         current action finishes before the next action.
+  /// - Finish: The action is executed and the execution is paused only when we
+  ///           reach the parent/enclosing operation. If there are no enclosing
+  ///           operation, the execution continues without stopping.
+  enum Control { Apply = 1, Skip = 2, Step = 3, Next = 4, Finish = 5 };
+
+  /// The type of the callback that is used to control the execution.
+  /// The callback is passed the current action.
+  using CallbackTy = function_ref<Control(const ActionActiveStack *)>;
+
+  /// Create an ExecutionContext with a callback that is used to control the
+  /// execution.
+  ExecutionContext(CallbackTy callback) { setCallback(callback); }
+  ExecutionContext() = default;
+
+  /// Set the callback that is used to control the execution.
+  void setCallback(CallbackTy callback);
+
+  /// This abstract class defines the interface used to observe an Action
+  /// execution. It allows to be notified before and after the callback is
+  /// processed, but can't affect the execution.
+  struct Observer {
+    virtual ~Observer() = default;
+    /// This method is called before the Action is executed
+    /// If a breakpoint was hit, it is passed as an argument to the callback.
+    /// The `willExecute` argument indicates whether the action will be executed
+    /// or not.
+    /// Note that this method will be called from multiple threads concurrently
+    /// when MLIR multi-threading is enabled.
+    virtual void beforeExecute(const ActionActiveStack *action,
+                               Breakpoint *breakpoint, bool willExecute) {}
+
+    /// This method is called after the Action is executed, if it was executed.
+    /// It is not called if the action is skipped.
+    /// Note that this method will be called from multiple threads concurrently
+    /// when MLIR multi-threading is enabled.
+    virtual void afterExecute(const ActionActiveStack *action) {}
+  };
+
+  /// Register a new `Observer` on this context. It'll be notified before and
+  /// after executing an action. Note that this method is not thread-safe: it
+  /// isn't supported to add a new observer while actions may be executed.
+  void registerObserver(Observer *observer);
+
+  /// Register a new `BreakpointManager` on this context. It'll have a chance to
+  /// match an action before it gets executed. Note that this method is not
+  /// thread-safe: it isn't supported to add a new manager while actions may be
+  /// executed.
+  void addBreakpointManager(BreakpointManager *manager) {
+    breakpoints.push_back(manager);
+  }
+
+  /// Process the given action. This is the operator called by MLIRContext on
+  /// `executeAction()`.
+  void operator()(function_ref<void()> transform, const Action &action);
+
+private:
+  /// Callback that is executed when a breakpoint is hit and allows the client
+  /// to control the execution.
+  CallbackTy onBreakpointControlExecutionCallback;
+
+  /// Next point to stop execution as describe by `Control` enum.
+  /// This is handle by indicating at which levels of depth the next
+  /// break should happen.
+  std::optional<int> depthToBreak;
+
+  /// Observers that are notified before and after the callback is executed.
+  SmallVector<Observer *> observers;
+
+  /// The list of managers that are queried for breakpoints.
+  SmallVector<BreakpointManager *> breakpoints;
+};
+
+} // namespace tracing
+} // namespace mlir
+
+#endif // MLIR_TRACING_EXECUTIONCONTEXT_H
diff --git a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
index 5f728516ac390..826c740c90cff 100644
--- a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
+++ b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
@@ -167,8 +167,9 @@ struct DependenceResult {
 
 DependenceResult checkMemrefAccessDependence(
     const MemRefAccess &srcAccess, const MemRefAccess &dstAccess,
-    unsigned loopDepth, FlatAffineValueConstraints *dependenceConstraints,
-    SmallVector<DependenceComponent, 2> *dependenceComponents,
+    unsigned loopDepth,
+    FlatAffineValueConstraints *dependenceConstraints = nullptr,
+    SmallVector<DependenceComponent, 2> *dependenceComponents = nullptr,
     bool allowRAR = false);
 
 /// Utility function that returns true if the provided DependenceResult
diff --git a/mlir/include/mlir/Dialect/Affine/Analysis/AffineStructures.h b/mlir/include/mlir/Dialect/Affine/Analysis/AffineStructures.h
index 0282d2ea7fff1..1b302f55422d8 100644
--- a/mlir/include/mlir/Dialect/Affine/Analysis/AffineStructures.h
+++ b/mlir/include/mlir/Dialect/Affine/Analysis/AffineStructures.h
@@ -43,12 +43,14 @@ class MultiAffineFunction;
 class FlatAffineValueConstraints : public presburger::IntegerPolyhedron {
 public:
   /// Constructs a constraint system reserving memory for the specified number
-  /// of constraints and variables.
+  /// of constraints and variables. `valArgs` are the optional SSA values
+  /// associated with each dimension/symbol. These must either be empty or match
+  /// the number of dimensions and symbols.
   FlatAffineValueConstraints(unsigned numReservedInequalities,
                              unsigned numReservedEqualities,
                              unsigned numReservedCols, unsigned numDims,
                              unsigned numSymbols, unsigned numLocals,
-                             ArrayRef<std::optional<Value>> valArgs = {})
+                             ArrayRef<std::optional<Value>> valArgs)
       : IntegerPolyhedron(numReservedInequalities, numReservedEqualities,
                           numReservedCols,
                           presburger::PresburgerSpace::getSetSpace(
@@ -62,11 +64,48 @@ class FlatAffineValueConstraints : public presburger::IntegerPolyhedron {
       values.append(valArgs.begin(), valArgs.end());
   }
 
-  /// Constructs a constraint system with the specified number of
+  /// Constructs a constraint system reserving memory for the specified number
+  /// of constraints and variables. `valArgs` are the optional SSA values
+  /// associated with each dimension/symbol. These must either be empty or match
+  /// the number of dimensions and symbols.
+  FlatAffineValueConstraints(unsigned numReservedInequalities,
+                             unsigned numReservedEqualities,
+                             unsigned numReservedCols, unsigned numDims,
+                             unsigned numSymbols, unsigned numLocals,
+                             ArrayRef<Value> valArgs = {})
+      : IntegerPolyhedron(numReservedInequalities, numReservedEqualities,
+                          numReservedCols,
+                          presburger::PresburgerSpace::getSetSpace(
+                              numDims, numSymbols, numLocals)) {
+    assert(numReservedCols >= getNumVars() + 1);
+    assert(valArgs.empty() || valArgs.size() == getNumDimAndSymbolVars());
+    values.reserve(numReservedCols);
+    if (valArgs.empty())
+      values.resize(getNumDimAndSymbolVars(), std::nullopt);
+    else
+      values.append(valArgs.begin(), valArgs.end());
+  }
+
+  /// Constructs a constraint system with the specified number of dimensions
+  /// and symbols. `valArgs` are the optional SSA values associated with each
+  /// dimension/symbol. These must either be empty or match the number of
+  /// dimensions and symbols.
+  FlatAffineValueConstraints(unsigned numDims, unsigned numSymbols,
+                             unsigned numLocals,
+                             ArrayRef<std::optional<Value>> valArgs)
+      : FlatAffineValueConstraints(/*numReservedInequalities=*/0,
+                                   /*numReservedEqualities=*/0,
+                                   /*numReservedCols=*/numDims + numSymbols +
+                                       numLocals + 1,
+                                   numDims, numSymbols, numLocals, valArgs) {}
+
+  /// Constructs a constraint system with the specified number of dimensions
+  /// and symbols. `valArgs` are the optional SSA values associated with each
+  /// dimension/symbol. These must either be empty or match the number of
   /// dimensions and symbols.
   FlatAffineValueConstraints(unsigned numDims = 0, unsigned numSymbols = 0,
                              unsigned numLocals = 0,
-                             ArrayRef<std::optional<Value>> valArgs = {})
+                             ArrayRef<Value> valArgs = {})
       : FlatAffineValueConstraints(/*numReservedInequalities=*/0,
                                    /*numReservedEqualities=*/0,
                                    /*numReservedCols=*/numDims + numSymbols +
@@ -83,11 +122,6 @@ class FlatAffineValueConstraints : public presburger::IntegerPolyhedron {
       values.append(valArgs.begin(), valArgs.end());
   }
 
-  /// Create a flat affine constraint system from an AffineValueMap or a list of
-  /// these. The constructed system will only include equalities.
-  explicit FlatAffineValueConstraints(const AffineValueMap &avm);
-  explicit FlatAffineValueConstraints(ArrayRef<const AffineValueMap *> avmRef);
-
   /// Creates an affine constraint system from an IntegerSet.
   explicit FlatAffineValueConstraints(IntegerSet set, ValueRange operands = {});
 
@@ -112,19 +146,6 @@ class FlatAffineValueConstraints : public presburger::IntegerPolyhedron {
     return cst->getKind() == Kind::FlatAffineValueConstraints;
   }
 
-  /// Clears any existing data and reserves memory for the specified
-  /// constraints.
-  void reset(unsigned numReservedInequalities, unsigned numReservedEqualities,
-             unsigned numReservedCols, unsigned numDims, unsigned numSymbols,
-             unsigned numLocals = 0);
-  void reset(unsigned numDims = 0, unsigned numSymbols = 0,
-             unsigned numLocals = 0);
-  void reset(unsigned numReservedInequalities, unsigned numReservedEqualities,
-             unsigned numReservedCols, unsigned numDims, unsigned numSymbols,
-             unsigned numLocals, ArrayRef<Value> valArgs);
-  void reset(unsigned numDims, unsigned numSymbols, unsigned numLocals,
-             ArrayRef<Value> valArgs);
-
   /// Clones this object.
   std::unique_ptr<FlatAffineValueConstraints> clone() const;
 
@@ -225,11 +246,11 @@ class FlatAffineValueConstraints : public presburger::IntegerPolyhedron {
   /// null AffineMap if such a bound can't be found (or yet unimplemented).
   ///
   /// By default the returned lower bounds are closed and upper bounds are open.
-  /// This can be changed by getClosedUB.
+  /// If `closedUb` is true, the upper bound is closed.
   void getSliceBounds(unsigned offset, unsigned num, MLIRContext *context,
                       SmallVectorImpl<AffineMap> *lbMaps,
                       SmallVectorImpl<AffineMap> *ubMaps,
-                      bool getClosedUB = false);
+                      bool closedUB = false);
 
   /// Composes an affine map whose dimensions and symbols match one to one with
   /// the dimensions and symbols of this FlatAffineConstraints. The results of
@@ -241,13 +262,15 @@ class FlatAffineValueConstraints : public presburger::IntegerPolyhedron {
   /// treating [0, offset) U [offset + num, symStartPos) as dimensions and
   /// [symStartPos, getNumDimAndSymbolVars) as symbols, and `pos` lies in
   /// [0, num). The multi-dimensional maps in the returned pair represent the
-  /// max and min of potentially multiple affine expressions. The upper bound is
-  /// exclusive. `localExprs` holds pre-computed AffineExpr's for all local
-  /// variables in the system.
+  /// max and min of potentially multiple affine expressions. `localExprs` holds
+  /// pre-computed AffineExpr's for all local variables in the system.
+  ///
+  /// By default the returned lower bounds are closed and upper bounds are open.
+  /// If `closedUb` is true, the upper bound is closed.
   std::pair<AffineMap, AffineMap>
   getLowerAndUpperBound(unsigned pos, unsigned offset, unsigned num,
                         unsigned symStartPos, ArrayRef<AffineExpr> localExprs,
-                        MLIRContext *context) const;
+                        MLIRContext *context, bool closedUB = false) const;
 
   /// Returns the bound for the variable at `pos` from the inequality at
   /// `ineqPos` as a 1-d affine value map (affine map + operands). The returned
diff --git a/mlir/include/mlir/Dialect/Affine/Analysis/Utils.h b/mlir/include/mlir/Dialect/Affine/Analysis/Utils.h
index 72850bf9ab517..7f6eced071b15 100644
--- a/mlir/include/mlir/Dialect/Affine/Analysis/Utils.h
+++ b/mlir/include/mlir/Dialect/Affine/Analysis/Utils.h
@@ -39,6 +39,10 @@ class Value;
 /// from the outermost 'affine.for' operation to the innermost one.
 void getAffineForIVs(Operation &op, SmallVectorImpl<AffineForOp> *loops);
 
+/// Populates 'ivs' with IVs of the surrounding affine.for and affine.parallel
+/// ops ordered from the outermost one to the innermost.
+void getAffineIVs(Operation &op, SmallVectorImpl<Value> &ivs);
+
 /// Populates 'ops' with affine operations enclosing `op` ordered from outermost
 /// to innermost. affine.for, affine.if, or affine.parallel ops comprise such
 /// surrounding affine ops.
diff --git a/mlir/include/mlir/Dialect/Affine/IR/AffineOps.h b/mlir/include/mlir/Dialect/Affine/IR/AffineOps.h
index 7bf78cd369f22..cda38a5580645 100644
--- a/mlir/include/mlir/Dialect/Affine/IR/AffineOps.h
+++ b/mlir/include/mlir/Dialect/Affine/IR/AffineOps.h
@@ -441,10 +441,21 @@ namespace mlir {
 /// AffineForOp.
 bool isAffineForInductionVar(Value val);
 
+/// Returns true if `val` is the induction variable of an AffineParallelOp.
+bool isAffineParallelInductionVar(Value val);
+
+/// Returns true if the provided value is the induction variable of an
+/// AffineForOp or AffineParallelOp.
+bool isAffineInductionVar(Value val);
+
 /// Returns the loop parent of an induction variable. If the provided value is
 /// not an induction variable, then return nullptr.
 AffineForOp getForInductionVarOwner(Value val);
 
+/// Returns true if the provided value is among the induction variables of an
+/// AffineParallelOp.
+AffineParallelOp getAffineParallelInductionVarOwner(Value val);
+
 /// Extracts the induction variables from a list of AffineForOps and places them
 /// in the output argument `ivs`.
 void extractForInductionVars(ArrayRef<AffineForOp> forInsts,
diff --git a/mlir/include/mlir/Dialect/Affine/Passes.h b/mlir/include/mlir/Dialect/Affine/Passes.h
index 7102626db0f60..9d253fb7660d0 100644
--- a/mlir/include/mlir/Dialect/Affine/Passes.h
+++ b/mlir/include/mlir/Dialect/Affine/Passes.h
@@ -18,6 +18,7 @@
 #include <limits>
 
 namespace mlir {
+
 namespace func {
 class FuncOp;
 } // namespace func
@@ -110,10 +111,6 @@ createLoopUnrollAndJamPass(int unrollJamFactor = -1);
 /// memory hierarchy.
 std::unique_ptr<OperationPass<func::FuncOp>> createPipelineDataTransferPass();
 
-/// Populate patterns that expand affine index operations into more fundamental
-/// operations (not necessarily restricted to Affine dialect).
-void populateAffineExpandIndexOpsPatterns(RewritePatternSet &patterns);
-
 /// Creates a pass to expand affine index operations into more fundamental
 /// operations (not necessarily restricted to Affine dialect).
 std::unique_ptr<Pass> createAffineExpandIndexOpsPass();
diff --git a/mlir/include/mlir/Dialect/Affine/Transforms/Transforms.h b/mlir/include/mlir/Dialect/Affine/Transforms/Transforms.h
new file mode 100644
index 0000000000000..dbba3b7549ff1
--- /dev/null
+++ b/mlir/include/mlir/Dialect/Affine/Transforms/Transforms.h
@@ -0,0 +1,45 @@
+//===- Transforms.h - Transforms Entrypoints --------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This header file defines a set of transforms specific for the AffineOps
+// dialect.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_AFFINE_TRANSFORMS_TRANSFORMS_H
+#define MLIR_DIALECT_AFFINE_TRANSFORMS_TRANSFORMS_H
+
+#include "mlir/Support/LogicalResult.h"
+
+namespace mlir {
+class RewritePatternSet;
+class RewriterBase;
+class AffineApplyOp;
+
+/// Populate patterns that expand affine index operations into more fundamental
+/// operations (not necessarily restricted to Affine dialect).
+void populateAffineExpandIndexOpsPatterns(RewritePatternSet &patterns);
+
+/// Helper function to rewrite `op`'s affine map and reorder its operands such
+/// that they are in increasing order of hoistability (i.e. the least hoistable)
+/// operands come first in the operand list.
+void reorderOperandsByHoistability(RewriterBase &rewriter, AffineApplyOp op);
+
+/// Split an "affine.apply" operation into smaller ops.
+/// This reassociates a large AffineApplyOp into an ordered list of smaller
+/// AffineApplyOps. This can be used right before lowering affine ops to arith
+/// to exhibit more opportunities for CSE and LICM.
+/// Return the sink AffineApplyOp on success or failure if `op` does not
+/// decompose into smaller AffineApplyOps.
+/// Note that this can be undone by canonicalization which tries to
+/// maximally compose chains of AffineApplyOps.
+FailureOr<AffineApplyOp> decompose(RewriterBase &rewriter, AffineApplyOp op);
+
+} // namespace mlir
+
+#endif // MLIR_DIALECT_AFFINE_TRANSFORMS_TRANSFORMS_H
diff --git a/mlir/include/mlir/Dialect/GPU/TransformOps/GPUDeviceMappingAttr.td b/mlir/include/mlir/Dialect/GPU/TransformOps/GPUDeviceMappingAttr.td
index 3b261acdee83a..699390c2f2959 100644
--- a/mlir/include/mlir/Dialect/GPU/TransformOps/GPUDeviceMappingAttr.td
+++ b/mlir/include/mlir/Dialect/GPU/TransformOps/GPUDeviceMappingAttr.td
@@ -64,6 +64,41 @@ def GPUWarpMappingAttr : GPU_Attr<"GPUWarpMapping", "warp", [
   }];
 }
 
+def LinearIdEnum : I64EnumAttr<"LinearId", "linear ids for loop mapping", [
+    DimX, DimY, DimZ]> {
+  let cppNamespace = "::mlir::gpu";
+}
+
+def GPULinearIdMapping : GPU_Attr<"GPULinearIdMapping", "linear", [
+  DeclareAttrInterfaceMethods<DeviceMappingAttrInterface> ] >  {
+  let parameters = (ins
+    EnumParameter<LinearIdEnum>:$linear_id
+  );
+  let assemblyFormat = "`<` params `>`";
+  let description = [{
+    An attribute to allow re-interpreting the linear mapping for threads in GPU
+    devices.
+
+    Threads (aka work item) are grouped into a thread block where block may be
+    described by a 1-, 2-, or 3-dimensional rectangular basis.
+    The linear thread id is obtained by linearizing the 1-, 2- or 3-dimensional
+    index. For instance, if the basis is denoted as (BX, BY, BZ) and the thread
+    id is denoted by (tx, ty, tz), the linear thread id is:
+      `linear_id = tx + ty * BX + tz * BX * BY)`.
+    The linear thread id is fixed for the duration of a GPU kernel.
+    
+    This linear id mapping attribute indicates a different linearization relation
+    is applied locally to a loop nest. 
+    
+    For instance, if the new basis is denoted as (LBX, LBY, LBZ) the thread id
+    in the new basis is:
+      `(linear_id mod LBX , (linear_id / LBX) mod * LBY, linear_id / (LBX * LBY))`.
+    This reinterpretation is only fixe for the duration of a loop nest.
+    
+    It can be consumed by lowering to generate GPU code.
+  }];
+}
+
 def BlocksEnum : I64EnumAttr<"Blocks", "threads for loop mapping", [
     DimX, DimY, DimZ]> {
   let cppNamespace = "::mlir::gpu";
diff --git a/mlir/include/mlir/Dialect/GPU/TransformOps/GPUTransformOps.h b/mlir/include/mlir/Dialect/GPU/TransformOps/GPUTransformOps.h
index 91352b28fc7b7..57d74d856cba7 100644
--- a/mlir/include/mlir/Dialect/GPU/TransformOps/GPUTransformOps.h
+++ b/mlir/include/mlir/Dialect/GPU/TransformOps/GPUTransformOps.h
@@ -33,34 +33,96 @@ class DialectRegistry;
 namespace transform {
 namespace gpu {
 
-/// Searches `scf.forall` ops nested under `target` and maps each such
-/// op to GPU threads. Mapping is one-to-one and the induction variables of
-/// `scf.forall` are rewritten to gpu.thread_id according to the
-/// thread_dim_apping attribute. Sibling `scf.forall` are supported in
-/// which case, the union of the number of threads is computed and may result in
-/// predication. Dynamic, `scf.forall` trip counts are currently not
-/// supported. Dynamic block dim sizes are currently not supported.
-DiagnosedSilenceableFailure mapNestedForeachToThreadsImpl(
-    RewriterBase &rewriter, Operation *target,
-    const SmallVectorImpl<int64_t> &blockDim,
-    function_ref<void(RewriterBase &, scf::ForallOp, SmallVectorImpl<Value> &)>
-        threadIdGenerator,
-    bool syncAfterDistribute, std::optional<TransformOpInterface> transformOp,
-    const ArrayRef<DeviceMappingAttrInterface> &threadMappingAttributes);
-
-/// Maps the top level `scf.forall` op to GPU Thread Blocks. Mapping is
-/// one-to-one and the induction variables of `scf.forall` are rewritten
-/// to gpu.block_id according to the thread_dim_apping attribute. Dynamic,
-/// `scf.forall` trip counts are currently not supported. Dynamic block
-/// dim sizes are currently not supported.
-DiagnosedSilenceableFailure mapForeachToBlocksImpl(
-    RewriterBase &rewriter, scf::ForallOp forallOp,
-    function_ref<void(RewriterBase &, scf::ForallOp, SmallVectorImpl<Value> &)>
-        blockIdGenerator,
-    SmallVectorImpl<int64_t> &gridDims, TransformOpInterface transformOp,
-    const ArrayRef<DeviceMappingAttrInterface> &mappingAttributes);
-
-/// Finds the top level scf::ForallOp of given target.
+/// Helper type for functions that generate ids for the mapping of a
+/// scf.forall.
+struct IdBuilderResult {
+  // Ops used to replace the forall induction variables.
+  SmallVector<Value> mappingIdOps;
+  // Actual mapping sizes used to predicate the forall body when they are
+  // smaller than the available mapping sizes.
+  SmallVector<int64_t> predicateMappingSizes;
+  // Ops used to predicate the forall body when predicateMappingSizes is smaller
+  // than the available mapping sizes.
+  SmallVector<Value> predicateIdOps;
+};
+
+/// Common gpu id builder type, allows the configuration of lowering for various
+/// mapping schemes. Takes:
+///   - A rewriter with insertion point set before the forall op to rewrite.
+///   - The loc of the forall op to rewrite.
+///   - A list of positive integers carrying the mapping sizes for the current
+///     forall op to rewrite.
+using GpuIdBuilderFnType =
+    std::function<IdBuilderResult(RewriterBase &, Location, ArrayRef<int64_t>)>;
+
+/// Helper struct for configuring the rewrite of mapped scf.forall ops to
+/// various gpu id configurations.
+struct GpuIdBuilder {
+  GpuIdBuilder(ArrayRef<OpFoldResult> blockDims, ArrayRef<int64_t> mappingSizes)
+      : blockDimsOfr(blockDims), availableMappingSizes(mappingSizes),
+        mappingAttributes(), idBuilder() {}
+
+  /// List of OpFoldResult carrying the  multi-dimensional number of
+  /// threads available in the current kernel (i.e. the current blockDims in
+  /// CUDA parlance).
+  ArrayRef<OpFoldResult> blockDimsOfr;
+
+  /// A list of positive integers carrying the number of available mapping
+  /// resources that can trigger predication,
+  ArrayRef<int64_t> availableMappingSizes;
+
+  /// The mapping attributes targeted by this generator.
+  SmallVector<DeviceMappingAttrInterface> mappingAttributes;
+
+  /// The constructor that builds the concrete IR for mapping ids.
+  GpuIdBuilderFnType idBuilder;
+};
+
+/// Map the top level `scf.forall` op to GPU Thread Blocks.
+/// Mapping is one-to-one and the induction variables of `scf.forall` are
+/// rewritten to gpu.block_id according to the thread_dim_mapping attribute.
+///
+/// Dynamic, `scf.forall` trip counts are currently not supported.
+/// Dynamic block dim sizes are currently not supported.
+DiagnosedSilenceableFailure
+mapForallToBlocksImpl(RewriterBase &rewriter, TransformOpInterface transformOp,
+                      scf::ForallOp forallOp,
+                      SmallVectorImpl<int64_t> &gridDims,
+                      const GpuIdBuilder &gpuIdBuilder);
+
+/// Search `scf.forall` ops nested under `target` and map each such op to an
+/// explicit GPU implementation along `availableMappingSizes`.
+/// The mapping is one-to-one and the induction variables of `scf.forall` are
+/// rewritten to gpuIdBuilder.idBuilder according to the
+/// gpuIdBuilder.mappingAttributes attribute.
+///
+/// Dynamic, `scf.forall` trip counts are currently not supported.
+/// Dynamic `availableMappingSizes` sizes are currently not supported.
+/// `availableMappingSizes` is expected to be of size 3.
+DiagnosedSilenceableFailure mapOneForallToThreadsImpl(
+    RewriterBase &rewriter, std::optional<TransformOpInterface> transformOp,
+    scf::ForallOp forallOp, ArrayRef<int64_t> availableMappingSizes,
+    bool syncAfterDistribute, const GpuIdBuilder &gpuIdBuilder);
+
+/// Search `scf.forall` ops nested under `target` and map each such op to an
+/// explicit GPU implementation along blockDims and warpDims.
+/// The mapping is one-to-one and the induction variables of `scf.forall` are
+/// rewritten to threads and warps ids according to the mapping attribute.
+///
+/// Dynamic, `scf.forall` trip counts are currently not supported.
+/// Dynamic `blockDims` or `warpDims` or `linearDims` sizes are currently not
+/// supported.
+/// `blockDims` is expected to be of size 3.
+/// `warpDims` is expected to be empty or of size 3.
+///
+/// The insertion point of the `rewriter` is expected to be set at the
+/// beginning of the `target` body block and dominate all other blocks.
+DiagnosedSilenceableFailure mapNestedForallToThreadsImpl(
+    RewriterBase &rewriter, std::optional<TransformOpInterface> transformOp,
+    Operation *target, ArrayRef<int64_t> blockDimsOfr,
+    ArrayRef<int64_t> warpDims, bool syncAfterDistribute);
+
+/// Find the unique top level scf::ForallOp within a given target op.
 DiagnosedSilenceableFailure
 findTopLevelForallOp(Operation *target, scf::ForallOp &topLevelForallOp,
                      TransformOpInterface transformOp);
diff --git a/mlir/include/mlir/Dialect/GPU/TransformOps/GPUTransformOps.td b/mlir/include/mlir/Dialect/GPU/TransformOps/GPUTransformOps.td
index 76292b4683380..c719fedc90e33 100644
--- a/mlir/include/mlir/Dialect/GPU/TransformOps/GPUTransformOps.td
+++ b/mlir/include/mlir/Dialect/GPU/TransformOps/GPUTransformOps.td
@@ -15,28 +15,33 @@ include "mlir/Dialect/PDL/IR/PDLTypes.td"
 include "mlir/Interfaces/SideEffectInterfaces.td"
 include "mlir/IR/OpBase.td"
 
-def MapNestedForeachToThreads :
-  Op<Transform_Dialect, "gpu.map_nested_foreach_to_threads",
+def MapNestedForallToThreads :
+  Op<Transform_Dialect, "gpu.map_nested_forall_to_threads",
     [FunctionalStyleTransformOpTrait,
      MemoryEffectsOpInterface,
      TransformEachOpTrait,
      TransformOpInterface]> {
   let description = [{
-      Target the `gpu.launch op` and rewrite all `scf.forall`
-      nested in it to distributed `gpu.thread_id` attribute.
-
-      The operation searches for `scf.forall` ops nested under `target`
-      and maps each such op to GPU threads. Mapping is one-to-one and the
-      induction variables of `scf.forall` are rewritten to
-      `gpu.thread_id` according to the `mapping` attribute.
-
-      Sibling `scf.forall` are supported in which case, the union of
-      the number of threads is computed and may result in predication.
-
-      Multiple scf.forall are supported per `gpu.launch` in which case,
-      the max of all the threads is computed and taken for the global
-      `gpu.thread_id`. If necessary, `scf.forall` that do not use the
-      whole thread range result in predicated computations.
+      Target the `gpu.launch op` and rewrite all `scf.forall` nested in it to 
+      distributed `gpu.thread_id` attribute.
+
+      The operation searches for `scf.forall` ops nested under `target` and maps
+      each such op to GPU threads. 
+      
+      `scf.forall` induction variables are rewritten to `gpu.thread_id` according
+      to the `mapping` attribute.
+
+      Different types of mappings attributes are supported:
+        - the block_dims is a list of integers that specifies the number of
+          threads in each dimension. This is a mandatory attribute that is used
+          to constrain the number of threads in each dimension. If an 
+          `scf.forall` op is mapped to fewer threads, predication occurs.
+        - the warp_dims is a list of integers that specifies the number of
+          warps in each dimension. This is an optional attribute that is used
+          to constrain the number of warps in each dimension. When present, this
+          attribute must be specified in a way that is compatible with the 
+          block_dims attribute. If an `scf.forall` op is mapped to fewer warps,
+          predicaiton occurs.
 
       Dynamic `scf.forall` trip counts are currently not supported.
       Dynamic block dim sizes are currently not supported.
@@ -45,10 +50,12 @@ def MapNestedForeachToThreads :
       Only `scf.forall` distributed to **at most 3 dimensions** are
       currently supported.
 
-      Barriers are inserted after each scf.forall op for now.
+      The `sync_after_distribute`attribute controls whether a `gpu.barrier` is
+      inserted after each scf.forall op. At this time, this is an all or nothing
+      choice. This will need to be tightened in the future.
 
-      The operation alters the block size of the given gpu_launch using
-      blockDim argument.
+      The operation alters the block size of the given gpu_launch using the 
+      mandatory block_dims argument.
 
       #### Return modes:
 
@@ -83,6 +90,7 @@ def MapNestedForeachToThreads :
         gpu.terminator
       }
       ```
+
       is translated to:
 
       ```
@@ -104,11 +112,18 @@ def MapNestedForeachToThreads :
     }];
 
   let arguments = (ins PDL_Operation:$target,
-                   DefaultValuedAttr<I64ArrayAttr, "{}">:$blockDim,
-                   DefaultValuedAttr<BoolAttr, "true">:$syncAfterDistribute);
+                   DefaultValuedAttr<DenseI64ArrayAttr, "{}">:$block_dims,
+                   DefaultValuedOptionalAttr<DenseI64ArrayAttr, "{}">:$warp_dims,
+                   DefaultValuedAttr<BoolAttr, "true">:$sync_after_distribute);
   let results = (outs PDL_Operation:$result);
 
-  let assemblyFormat = "$target attr-dict";
+  let assemblyFormat = [{
+    $target
+    `block_dims` `=` $block_dims
+    (`warp_dims` `=` $warp_dims^)?
+    (`sync_after_distribute` `=` $sync_after_distribute^)?
+    attr-dict
+  }];
   let extraClassDeclaration = [{
     ::mlir::DiagnosedSilenceableFailure applyToOne(
         ::mlir::Operation *target,
@@ -117,9 +132,8 @@ def MapNestedForeachToThreads :
   }];
 }
 
-
-def MapForeachToBlocks :
-  Op<Transform_Dialect, "gpu.map_foreach_to_blocks",
+def MapForallToBlocks :
+  Op<Transform_Dialect, "gpu.map_forall_to_blocks",
     [FunctionalStyleTransformOpTrait,
      MemoryEffectsOpInterface,
      TransformOpInterface,
@@ -142,8 +156,8 @@ def MapForeachToBlocks :
     Only scf.forall distributed to **at most 3 dimensions** are
     currently supported.
 
-    The operation alters the block size of the given gpu_launch using
-    gridDim argument.
+    The operation alters the block size of the given gpu_launch using the 
+    grid_dims argument.
 
     #### Return modes:
 
@@ -162,11 +176,16 @@ def MapForeachToBlocks :
   }];
 
   let arguments = (ins PDL_Operation:$target,
-                   DefaultValuedAttr<I64ArrayAttr, "{}">:$gridDim,
+                   DefaultValuedOptionalAttr<DenseI64ArrayAttr, "{}">:$grid_dims,
                    UnitAttr:$generate_gpu_launch);
   let results = (outs PDL_Operation:$result);
 
-  let assemblyFormat = "$target attr-dict";
+  let assemblyFormat = [{
+    $target
+    (`generate_gpu_launch` $generate_gpu_launch^)?
+    (`grid_dims` `=` $grid_dims^)?
+    attr-dict
+  }];
   let extraClassDeclaration = [{
     ::mlir::DiagnosedSilenceableFailure applyToOne(
         ::mlir::Operation *target,
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMEnums.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMEnums.td
index 654e120b6cc67..5c1d72a047cd9 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMEnums.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMEnums.td
@@ -546,6 +546,25 @@ def UnnamedAddr : LLVM_EnumAttr<
   let cppNamespace = "::mlir::LLVM";
 }
 
+//===----------------------------------------------------------------------===//
+// Visibility
+//===----------------------------------------------------------------------===//
+
+def VisibilityDefault
+    : LLVM_EnumAttrCase<"Default", "", "DefaultVisibility", 0>;
+def VisibilityHidden
+    : LLVM_EnumAttrCase<"Hidden", "hidden", "HiddenVisibility", 1>;
+def VisibilityProtected
+    : LLVM_EnumAttrCase<"Protected", "protected", "ProtectedVisibility", 2>;
+
+def Visibility : LLVM_EnumAttr<
+    "Visibility",
+    "::llvm::GlobalValue::VisibilityTypes",
+    "LLVM GlobalValue Visibility",
+    [VisibilityDefault, VisibilityHidden, VisibilityProtected]> {
+  let cppNamespace = "::mlir::LLVM";
+}
+
 //===----------------------------------------------------------------------===//
 // ModRefInfo
 //===----------------------------------------------------------------------===//
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
index d12564b6f245a..1b62ce0ca3e6e 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
@@ -381,16 +381,27 @@ def LLVM_StackRestoreOp : LLVM_ZeroResultIntrOp<"stackrestore"> {
 //
 
 // LLVM vector reduction over a single vector.
-class LLVM_VecReductionBase<string mnem, Type element>
+class LLVM_VecReductionBase<string mnem, Type element, bit requiresFastmath=0>
     : LLVM_OneResultIntrOp<"vector.reduce." # mnem, [], [0],
-                           [Pure, SameOperandsAndResultElementType]>,
-      Arguments<(ins LLVM_VectorOf<element>)>;
+                           [Pure, SameOperandsAndResultElementType],
+                           requiresFastmath> {
+      dag commonArgs = (ins LLVM_VectorOf<element>:$in);
+}
 
 class LLVM_VecReductionF<string mnem>
-    : LLVM_VecReductionBase<mnem, AnyFloat>;
+    : LLVM_VecReductionBase<mnem, AnyFloat, /*requiresFastmath=*/1> {
+  dag fmfArg = (
+    ins DefaultValuedAttr<LLVM_FastmathFlagsAttr, "{}">:$fastmathFlags);
+  let arguments = !con(commonArgs, fmfArg);
+
+  let assemblyFormat = "`(` operands `)` custom<LLVMOpAttrs>(attr-dict) `:` "
+      "functional-type(operands, results)";
+}
 
 class LLVM_VecReductionI<string mnem>
-    : LLVM_VecReductionBase<mnem, AnySignlessInteger>;
+    : LLVM_VecReductionBase<mnem, AnySignlessInteger> {
+      let arguments = commonArgs;
+}
 
 // LLVM vector reduction over a single vector, with an initial value,
 // and with permission to reassociate the reduction operations.
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMOps.td
index 8660d63929429..c2bb2f34a463a 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMOps.td
@@ -173,7 +173,8 @@ def LLVM_FNegOp : LLVM_UnaryFloatArithmeticOp<
 def LLVM_AllocaOp : LLVM_Op<"alloca">, LLVM_MemOpPatterns {
   let arguments = (ins AnyInteger:$arraySize,
                    OptionalAttr<I64Attr>:$alignment,
-                   OptionalAttr<TypeAttr>:$elem_type);
+                   OptionalAttr<TypeAttr>:$elem_type,
+                   UnitAttr:$inalloca);
   let results = (outs Res<LLVM_AnyPointer, "",
                           [MemAlloc<AutomaticAllocationScopeResource>]>:$res);
   string llvmInstName = "Alloca";
@@ -184,16 +185,18 @@ def LLVM_AllocaOp : LLVM_Op<"alloca">, LLVM_MemOpPatterns {
                    : op.getType().cast<LLVMPointerType>().getElementType());
     auto *inst = builder.CreateAlloca(elementType, addrSpace, $arraySize);
     }] # setAlignmentCode # [{
+    inst->setUsedWithInAlloca($inalloca);
     $res = inst;
   }];
-  // FIXME: Import attributes.
   string mlirBuilder = [{
     auto *allocaInst = cast<llvm::AllocaInst>(inst);
     Type allocatedType =
       moduleImport.convertType(allocaInst->getAllocatedType());
     unsigned alignment = allocaInst->getAlign().value();
     $res = $_builder.create<LLVM::AllocaOp>(
-      $_location, $_resultType, allocatedType, $arraySize, alignment);
+      $_location, $_resultType, $arraySize,
+      alignment == 0 ? IntegerAttr() : $_builder.getI64IntegerAttr(alignment),
+      TypeAttr::get(allocatedType), allocaInst->isUsedWithInAlloca());
   }];
   let builders = [
     OpBuilder<(ins "Type":$resultType, "Value":$arraySize,
@@ -203,9 +206,9 @@ def LLVM_AllocaOp : LLVM_Op<"alloca">, LLVM_MemOpPatterns {
              "pass the allocated type explicitly if opaque pointers are used");
       if (alignment == 0)
         return build($_builder, $_state, resultType, arraySize, IntegerAttr(),
-                     TypeAttr());
+                     TypeAttr(), false);
       build($_builder, $_state, resultType, arraySize,
-        $_builder.getI64IntegerAttr(alignment), TypeAttr());
+        $_builder.getI64IntegerAttr(alignment), TypeAttr(), false);
     }]>,
     OpBuilder<(ins "Type":$resultType, "Type":$elementType, "Value":$arraySize,
                CArg<"unsigned", "0">:$alignment),
@@ -216,7 +219,7 @@ def LLVM_AllocaOp : LLVM_Op<"alloca">, LLVM_MemOpPatterns {
       build($_builder, $_state, resultType, arraySize,
             alignment == 0 ? IntegerAttr()
                            : $_builder.getI64IntegerAttr(alignment),
-            elemTypeAttr);
+            elemTypeAttr, false);
 
     }]>
     ];
@@ -768,17 +771,22 @@ def LLVM_ShuffleVectorOp : LLVM_Op<"shufflevector",
 // Misc operations.
 def LLVM_SelectOp
     : LLVM_Op<"select",
-          [Pure, AllTypesMatch<["trueValue", "falseValue", "res"]>]>,
+          [Pure, AllTypesMatch<["trueValue", "falseValue", "res"]>,
+           DeclareOpInterfaceMethods<FastmathFlagsInterface>]>,
       LLVM_Builder<
           "$res = builder.CreateSelect($condition, $trueValue, $falseValue);"> {
   let arguments = (ins LLVM_ScalarOrVectorOf<I1>:$condition,
-                   LLVM_Type:$trueValue, LLVM_Type:$falseValue);
+                   LLVM_Type:$trueValue, LLVM_Type:$falseValue,
+                   DefaultValuedAttr<LLVM_FastmathFlagsAttr,
+                                     "{}">:$fastmathFlags);
   let results = (outs LLVM_Type:$res);
   let assemblyFormat = "operands attr-dict `:` type($condition) `,` type($res)";
   string llvmInstName = "Select";
   string mlirBuilder = [{
-    $res = $_builder.create<LLVM::SelectOp>(
+    auto op = $_builder.create<LLVM::SelectOp>(
       $_location, $_resultType, $condition, $trueValue, $falseValue);
+    moduleImport.setFastmathFlagsAttr(inst, op);
+    $res = op;
   }];
 }
 def LLVM_FreezeOp : LLVM_Op<"freeze", [SameOperandsAndResultType]> {
@@ -1298,7 +1306,8 @@ def LLVM_GlobalOp : LLVM_Op<"mlir.global",
     OptionalAttr<I64Attr>:$alignment,
     DefaultValuedAttr<ConfinedAttr<I32Attr, [IntNonNegative]>, "0">:$addr_space,
     OptionalAttr<UnnamedAddr>:$unnamed_addr,
-    OptionalAttr<StrAttr>:$section
+    OptionalAttr<StrAttr>:$section,
+    DefaultValuedAttr<Visibility, "mlir::LLVM::Visibility::Default">:$visibility_
   );
   let summary = "LLVM dialect global.";
   let description = [{
@@ -1543,7 +1552,8 @@ def LLVM_LLVMFuncOp : LLVM_Op<"func", [
     OptionalAttr<DictArrayAttr>:$arg_attrs,
     OptionalAttr<DictArrayAttr>:$res_attrs,
     OptionalAttr<I64Attr>:$function_entry_count,
-    OptionalAttr<LLVM_MemoryEffectsAttr>:$memory
+    OptionalAttr<LLVM_MemoryEffectsAttr>:$memory,
+    DefaultValuedAttr<Visibility, "mlir::LLVM::Visibility::Default">:$visibility_
   );
 
   let regions = (region AnyRegion:$body);
diff --git a/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h b/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
index 4dd7641e8193b..8e38de4844e2b 100644
--- a/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
@@ -570,15 +570,15 @@ LogicalResult vectorize(RewriterBase &rewriter, LinalgOp linalgOp,
 LogicalResult vectorizeCopy(RewriterBase &builder, memref::CopyOp copyOp);
 
 /// Emit a loop nest of `scf.for` with the proper body for `linalgOp`.
-FailureOr<LinalgLoops> linalgOpToLoops(PatternRewriter &rewriter,
+FailureOr<LinalgLoops> linalgOpToLoops(RewriterBase &rewriter,
                                        LinalgOp linalgOp);
 
 /// Emit a loop nest of `scf.parallel` with the proper body for `linalgOp`.
-FailureOr<LinalgLoops> linalgOpToParallelLoops(PatternRewriter &rewriter,
+FailureOr<LinalgLoops> linalgOpToParallelLoops(RewriterBase &rewriter,
                                                LinalgOp linalgOp);
 
 /// Emit a loop nest of `affine.for` with the proper body for `linalgOp`.
-FailureOr<LinalgLoops> linalgOpToAffineLoops(PatternRewriter &rewriter,
+FailureOr<LinalgLoops> linalgOpToAffineLoops(RewriterBase &rewriter,
                                              LinalgOp linalgOp);
 
 /// Creates a number of ranges equal to the number of non-zero in `tileSizes`.
@@ -818,7 +818,7 @@ struct SplitReductionResult {
   LinalgOp resultCombiningLinalgOp;
 };
 FailureOr<SplitReductionResult>
-splitReduction(PatternRewriter &b, LinalgOp op,
+splitReduction(RewriterBase &b, LinalgOp op,
                const ControlSplitReductionFn &controlSplitReductionFn,
                bool useAlloc = false);
 
@@ -870,7 +870,7 @@ splitReduction(PatternRewriter &b, LinalgOp op,
 ///  return %4 : tensor<16x32xf32>
 /// ```
 FailureOr<SplitReductionResult>
-splitReductionByScaling(PatternRewriter &b, LinalgOp op,
+splitReductionByScaling(RewriterBase &b, LinalgOp op,
                         const ControlSplitReductionFn &controlSplitReductionFn,
                         bool useAlloc = false);
 
@@ -1041,6 +1041,19 @@ struct DownscaleDepthwiseConv2DNhwcHwcOp final
   }
 };
 
+struct DownscaleConv2DOp final : public OpRewritePattern<Conv2DOp> {
+  DownscaleConv2DOp(MLIRContext *context, PatternBenefit benefit = 1)
+      : OpRewritePattern<Conv2DOp>(context, benefit) {}
+
+  FailureOr<Conv1DOp> returningMatchAndRewrite(Conv2DOp convOp,
+                                               PatternRewriter &rewriter) const;
+
+  LogicalResult matchAndRewrite(Conv2DOp convOp,
+                                PatternRewriter &rewriter) const override {
+    return returningMatchAndRewrite(convOp, rewriter);
+  }
+};
+
 ///
 /// Linalg generalization pattern.
 ///
@@ -1084,7 +1097,7 @@ struct PadOpTransformationPattern : public OpRewritePattern<tensor::PadOp> {
 };
 
 using OptimizeCopyFn =
-    std::function<LogicalResult(PatternRewriter &, tensor::PadOp, Value)>;
+    std::function<LogicalResult(RewriterBase &, tensor::PadOp, Value)>;
 
 /// Rewrite a tensor::PadOp into a sequence of EmptyOp, FillOp and
 /// InsertSliceOp. For now, only constant padding values are supported.
@@ -1100,7 +1113,7 @@ struct GeneralizePadOpPattern : public OpRewritePattern<tensor::PadOp> {
 
 protected:
   OptimizeCopyFn optimizeCopyFn;
-  Value createFillOrGenerateOp(PatternRewriter &rewriter, tensor::PadOp padOp,
+  Value createFillOrGenerateOp(RewriterBase &rewriter, tensor::PadOp padOp,
                                Value dest,
                                const SmallVector<Value> &dynSizes) const;
 };
diff --git a/mlir/include/mlir/Dialect/NVGPU/Utils/MMAUtils.h b/mlir/include/mlir/Dialect/NVGPU/Utils/MMAUtils.h
index 5880b09161c7c..003a160985ee1 100644
--- a/mlir/include/mlir/Dialect/NVGPU/Utils/MMAUtils.h
+++ b/mlir/include/mlir/Dialect/NVGPU/Utils/MMAUtils.h
@@ -93,18 +93,6 @@ FailureOr<AffineMap>
 getLaneIdToLdMatrixMatrixCoord(OpBuilder &builder, Location loc,
                                const LdMatrixParams &params);
 
-/// Transform `vector.contract` into (m,k)x(n,k)x(m,n) form so that it can be
-/// converted to `nvgpu.mma.sync`. This specific form is meant to indicate that
-/// the vector operands are organized such that the reduction dimension is
-/// contiguous.
-struct PrepareContractToGPUMMASync
-    : public OpRewritePattern<vector::ContractionOp> {
-  using OpRewritePattern<vector::ContractionOp>::OpRewritePattern;
-
-  LogicalResult matchAndRewrite(vector::ContractionOp op,
-                                PatternRewriter &rewriter) const override;
-};
-
 } // namespace nvgpu
 } // namespace mlir
 
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 7dec1c8126b3c..37bb80a03ea00 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -28,6 +28,15 @@ def OpenMP_Dialect : Dialect {
   let cppNamespace = "::mlir::omp";
   let dependentDialects = ["::mlir::LLVM::LLVMDialect"];
   let useDefaultAttributePrinterParser = 1;
+
+  let extraClassDeclaration = [{
+    // Set the omp.is_device attribute on the module with the specified boolean
+    static void setIsDevice(Operation* module, bool isDevice);
+
+    // Return the value of the omp.is_device attribute stored in the module if it
+    // exists, otherwise return false by default
+    static bool getIsDevice(Operation* module);
+  }];
 }
 
 // OmpCommon requires definition of OpenACC_Dialect.
diff --git a/mlir/include/mlir/Dialect/SCF/IR/SCFOps.td b/mlir/include/mlir/Dialect/SCF/IR/SCFOps.td
index 7d129caa3084a..9396aa16f1392 100644
--- a/mlir/include/mlir/Dialect/SCF/IR/SCFOps.td
+++ b/mlir/include/mlir/Dialect/SCF/IR/SCFOps.td
@@ -652,15 +652,6 @@ def ForallOp : SCF_Op<"forall", [
     /// Checks if the lbs are zeros and steps are ones.
     bool isNormalized();
 
-    /// Helper to sort `values` according to matching `keys`.
-    /// Take a custom `compare` binary comparator which returns true if the first
-    /// element is smaller than the second (i.e. compatible with std::sort).
-    /// This is a helper typically used to sort numThreads values before they are
-    /// mapped to concrete physical dimensions of hardware.
-    static SmallVector<Value> getValuesSortedByKey(
-      ArrayRef<Attribute> keys, ValueRange values,
-      llvm::function_ref<bool(Attribute, Attribute)> compare);
-
     // The ensureTerminator method generated by SingleBlockImplicitTerminator is
     // unaware of the fact that our terminator also needs a region to be
     // well-formed. We override it here to ensure that we do the right thing.
diff --git a/mlir/include/mlir/Dialect/SCF/Transforms/Patterns.h b/mlir/include/mlir/Dialect/SCF/Transforms/Patterns.h
index 4cb5befd3053c..598e8ba6d3af1 100644
--- a/mlir/include/mlir/Dialect/SCF/Transforms/Patterns.h
+++ b/mlir/include/mlir/Dialect/SCF/Transforms/Patterns.h
@@ -31,6 +31,10 @@ namespace scf {
 /// }
 /// S1(N) S2(N-1)                // Epilogue
 /// S2(N)                        // Epilogue
+FailureOr<ForOp> pipelineForLoop(RewriterBase &rewriter, ForOp forOp,
+                                 const PipeliningOption &options);
+
+// TODO: such patterns should be auto-generated.
 class ForLoopPipeliningPattern : public OpRewritePattern<ForOp> {
 public:
   ForLoopPipeliningPattern(const PipeliningOption &options,
@@ -42,7 +46,9 @@ class ForLoopPipeliningPattern : public OpRewritePattern<ForOp> {
   }
 
   FailureOr<ForOp> returningMatchAndRewrite(ForOp forOp,
-                                            PatternRewriter &rewriter) const;
+                                            PatternRewriter &rewriter) const {
+    return pipelineForLoop(rewriter, forOp, options);
+  }
 
 protected:
   PipeliningOption options;
diff --git a/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h b/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
index 1ed850faa17a9..5e03eccfc2f3a 100644
--- a/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
+++ b/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
@@ -18,7 +18,7 @@
 
 namespace mlir {
 class Operation;
-class PatternRewriter;
+class RewriterBase;
 class TilingInterface;
 } // namespace mlir
 
@@ -243,7 +243,7 @@ struct SCFReductionTilingResult {
 ///   : tensor<7x4xf32> -> tensor<7xf32>
 /// ```
 FailureOr<scf::SCFReductionTilingResult>
-tileReductionUsingScf(PatternRewriter &b, PartialReductionOpInterface op,
+tileReductionUsingScf(RewriterBase &b, PartialReductionOpInterface op,
                       ArrayRef<OpFoldResult> tileSize);
 
 } // namespace scf
diff --git a/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h b/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h
index 1314d8628dc55..bfeab9da7632f 100644
--- a/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h
@@ -152,7 +152,7 @@ struct PipeliningOption {
   // peeled. This takes the original operation, an i1 predicate value and the
   // pattern rewriter.
   using PredicateOpFn =
-      std::function<Operation *(Operation *, Value, PatternRewriter &)>;
+      std::function<Operation *(RewriterBase &, Operation *, Value)>;
   PredicateOpFn predicateFn = nullptr;
 
   // TODO: add option to decide if the prologue should be peeled.
diff --git a/mlir/include/mlir/Dialect/SparseTensor/IR/Enums.h b/mlir/include/mlir/Dialect/SparseTensor/IR/Enums.h
index 165b3af74ab4e..89065b15bf230 100644
--- a/mlir/include/mlir/Dialect/SparseTensor/IR/Enums.h
+++ b/mlir/include/mlir/Dialect/SparseTensor/IR/Enums.h
@@ -40,12 +40,12 @@ namespace sparse_tensor {
 
 /// This type is used in the public API at all places where MLIR expects
 /// values with the built-in type "index".  For now, we simply assume that
-/// type is 64-bit, but targets with different "index" bit widths should
+/// type is 64-bit, but targets with different "index" bitwidths should
 /// link with an alternatively built runtime support library.
 // TODO: support such targets?
 using index_type = uint64_t;
 
-/// Encoding of overhead types (both pointer overhead and indices
+/// Encoding of overhead types (both position overhead and coordinate
 /// overhead), for "overloading" @newSparseTensor.
 enum class OverheadType : uint32_t {
   kIndex = 0,
diff --git a/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorAttrDefs.td b/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorAttrDefs.td
index 44a2b44ada694..a5b96a86596eb 100644
--- a/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorAttrDefs.td
+++ b/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorAttrDefs.td
@@ -19,6 +19,41 @@ class SparseTensor_Attr<string name,
                         list<Trait> traits = []>
     : AttrDef<SparseTensor_Dialect, name, traits>;
 
+//===----------------------------------------------------------------------===//
+// Type aliases.
+//
+// These attributes are just like `IndexAttr` (include/mlir/IR/OpBase.td),
+// except that:
+// (1) the `summary` is more specific (i.e., the fourth parameter to
+//     `TypedAttrBase`), which helps tablegen provide better error messages.
+// (2) tablegen-generated getters will have the given `returnType`, in
+//     lieu of the `APInt` that `IndexAttr` uses.  This avoids the boilerplate
+//     of needing to say `get{FOO}().getZExtValue()`, as well as using
+//     C++ types which better document intent.
+//===----------------------------------------------------------------------===//
+
+def DimensionAttr :
+    TypedAttrBase<
+      Index, "IntegerAttr",
+      And<[CPred<"$_self.isa<::mlir::IntegerAttr>()">,
+           CPred<"$_self.cast<::mlir::IntegerAttr>().getType()"
+                 ".isa<::mlir::IndexType>()">]>,
+      "dimension attribute"> {
+  let returnType = [{::mlir::sparse_tensor::Dimension}];
+  let convertFromStorage = [{$_self.getValue().getZExtValue()}];
+}
+
+def LevelAttr :
+    TypedAttrBase<
+      Index, "IntegerAttr",
+      And<[CPred<"$_self.isa<::mlir::IntegerAttr>()">,
+           CPred<"$_self.cast<::mlir::IntegerAttr>().getType()"
+                 ".isa<::mlir::IndexType>()">]>,
+      "level attribute"> {
+  let returnType = [{::mlir::sparse_tensor::Level}];
+  let convertFromStorage = [{$_self.getValue().getZExtValue()}];
+}
+
 //===----------------------------------------------------------------------===//
 // Sparse Tensor Dimension Slice Attribute.
 //===----------------------------------------------------------------------===//
@@ -92,73 +127,86 @@ def SparseTensorEncodingAttr : SparseTensor_Attr<"SparseTensorEncoding",
 
     The attribute consists of the following fields.
 
-    - Dimension level type for each dimension of a tensor type:
-        - **dense** : dimension is dense, all entries along this dimension
-          are stored
-        - **compressed** : dimension is sparse, only nonzeros along this dimensions
-          are stored
-        - **singleton** : dimension stores individual indices with no siblings
-      By default, each dimension level types has the property of being unique
-      (no duplicates at that level) and ordered (indices appear sorted at that
-      level). The following two suffixes can be used to make the last two
-      dimension level types not-unique (duplicates may appear) and not-ordered
-      (indices may appear unsorted).
+    - Level-type for each level of a tensor type:
+        - **dense** : all entries along this level are stored.
+        - **compressed** : only nonzeros along this level are stored.
+        - **singleton** : a variant of the compressed level-format,
+          for when coordinates are guaranteed to have no siblings at this level.
+      By default, each level-type has the property of being unique (no
+      duplicates at that level) and ordered (coordinates appear sorted
+      at that level).  The following two suffixes can be used to specify
+      that the level should instead be non-unique (duplicates may appear)
+      and/or non-ordered (coordinates may appear unsorted).
         - **-nu** : not unique
         - **-no** : not ordered
-      Currently, these suffixes, is present, should appear in this order.
-      In the future, we may introduce many more dimension level types and
-      properties, and separate specifying the two completely rather than
-      using this suffix mechanism.
-
-    - An optional dimension ordering on the indices of this tensor type. Unlike
-      dense storage, most sparse storage schemes do not provide fast random
-      access.  This affine map specifies the order of dimensions that should be
-      supported by the sparse storage scheme. For example, for a 2-d tensor,
-      `(i, j) -> (i, j)` requests row-wise storage and `(i, j) -> (j, i)`
-      requests column-wise storage.  By default, an identify mapping is used,
-      which implies that the original indices directly correspond to stored
-      indices.
-
-    - An optional higher-ordering mapping from the original index space of
-      the tensor to a higher-order index space, used to define block-sparse
-      storage or ELL (jagged diagonal) storage. For example, for a 2-d tensor,
-      the mapping `(i, j) -> (i floordiv 2, j floordiv 3, i mod 2, j mod 3)`
-      imposes an higher-order partitioning into 2x3 blocks along the matrix
-      layout. A dimension ordering can be used to define a desired ordering
-      on this higher-order index space. Likewise, the dimension level types
-      define dense or compressed storage along this higher-order index space.
-      For block-sparse, blocks are typically stored with compression while
-      dense storage is used within each block (although hybrid schemes are
-      possible as well). The higher-order mapping also provides a notion of
-      "counting a dimension", where every stored element with the same index
-      is mapped to a new slice. For instance, ELL storage of a 2-d tensor can
-      be defined with the mapping `(i, j) -> (#i, i, j)` using the notation
-      of [Chou20]. Lacking the `#` symbol in MLIR's affine mapping, we use
-      a free symbol `c` to define such counting, together with a constant
-      that denotes the number of resulting slices. For example, the mapping
-      `(i, j)[c] -> (c * 3 * i, i, j)` with the first two higher-order indices
-      stored dense and the innermost compressed denotes ELL storage with
-      three jagged diagonals that count the dimension `i`.
-
-      TODO: introduce a real counting symbol to MLIR's mapping, since an
-            expression like 3*c*i has no direct interpretation?
-
-    - The required bit width for "pointer" storage (integral offsets into
-      the sparse storage scheme). A narrow width reduces the memory footprint
-      of overhead storage, as long as the width suffices to define the total
-      required range (viz. the maximum number of stored entries over all indirection
-      dimensions). The choices are `8`, `16`, `32`, `64`, or, the default, `0` to
-      indicate the native bit width.
-
-    - The required bit width for "index" storage (elements of the coordinates of
-      stored entries). A narrow width reduces the memory footprint of overhead
-      storage, as long as the width suffices to define the total required range
-      (viz. the maximum value of each tensor index over all dimensions). The
-      choices are `8`, `16`, `32`, `64`, or, the default, `0` to indicate a
-      native bit width.
-
-    - An optional array of SparseTensorDimSliceAttr, which specifies how the sparse
-      tensor is partitioned on each level.
+      Currently, these suffixes (if present) must appear in this order.
+      In the future, we may introduce additional level-types and
+      properties, and split up how the level-format and properties are
+      specified rather than using this suffix mechanism.
+
+      TODO: This field is called "dimLevelType" for historical reasons,
+      even though the types are per-level rather than per-dimension.
+      (This will be corrected in an upcoming change that completely
+      overhauls the syntax of this attribute.)
+
+    - An optional permutation which maps (higher-ordering)-coordinates
+      to level-coordinates; defaulting to the identity permutation.
+      For example, given a 2-d tensor with the default higher-ordering,
+      `(i, j) -> (i, j)` specifies row-wise storage and `(i, j) ->
+      (j, i)` specifies column-wise storage.
+
+      TODO: this field is called "dimOrdering" for historical reasons,
+      even though it actually operates on level-coordinates rather than
+      dimension-coordinates.
+      (This will be corrected in an upcoming change that completely
+      overhauls the syntax of this attribute.)
+
+    - An optional higher-order mapping from dimension-coordinates to
+      a higher-order coordinate space; defaulting to the identity map.
+      This is applied before the `dimOrdering`, thus we have the composite:
+      dimCoords --higherOrdering--> hoCoords --dimOrdering--> lvlCoords.
+      The higher-order mapping is used to define block-sparse storage,
+      jagged-diagonal (JDS/ELL/ITPACK) storage, etc.
+
+      For example, given a 2-d tensor, the mapping
+      `(i, j) -> (i floordiv 2, j floordiv 3, i mod 2, j mod 3)`
+      imposes an higher-order partitioning into 2x3 blocks along the
+      matrix layout.  For block-sparsity, blocks are typically stored
+      with compression while dense storage is used within each block
+      (although hybrid schemes are possible as well).
+
+      TODO: the following example is out-of-date and will be implemented
+      in a different manner than described here.
+      (This will be corrected in an upcoming change that completely
+      overhauls the syntax of this attribute.)
+
+      The higher-order mapping also provides a notion of "counting a
+      dimension", where every stored element with the same coordinate
+      is mapped to a new slice.  For instance, ELL storage of a 2-d
+      tensor can be defined with the mapping `(i, j) -> (#i, i, j)`
+      using the notation of [Chou20].  Lacking the `#` symbol in MLIR's
+      affine mapping, we use a free symbol `c` to define such counting,
+      together with a constant that denotes the number of resulting
+      slices.  For example, the mapping `(i, j)[c] -> (c * 3 * i, i, j)`
+      with the level-types `["dense", "dense", "compressed"]` denotes ELL
+      storage with three jagged diagonals that count the dimension `i`.
+
+    - The required bitwidth for "position" storage (integral offsets
+      into the sparse storage scheme).  A narrow width reduces the memory
+      footprint of overhead storage, as long as the width suffices to
+      define the total required range (viz. the maximum number of stored
+      entries over all indirection levels).  The choices are `8`, `16`,
+      `32`, `64`, or, the default, `0` to indicate the native bitwidth.
+
+    - The required bitwidth for "coordinate" storage (the coordinates
+      of stored entries).  A narrow width reduces the memory footprint
+      of overhead storage, as long as the width suffices to define
+      the total required range (viz. the maximum value of each tensor
+      coordinate over all levels).  The choices are `8`, `16`, `32`,
+      `64`, or, the default, `0` to indicate a native bitwidth.
+
+    - An optional array of `SparseTensorDimSliceAttr`, which specifies
+      how the sparse tensor is partitioned on each dimension.
 
     Examples:
 
@@ -179,8 +227,8 @@ def SparseTensorEncodingAttr : SparseTensor_Attr<"SparseTensorEncoding",
     #DCSC = #sparse_tensor.encoding<{
       dimLevelType = [ "compressed", "compressed" ],
       dimOrdering = affine_map<(i, j) -> (j, i)>,
-      pointerBitWidth = 32,
-      indexBitWidth = 8
+      posWidth = 32,
+      crdWidth = 8
     }>
     ... tensor<8x8xf64, #DCSC> ...
 
@@ -214,20 +262,20 @@ def SparseTensorEncodingAttr : SparseTensor_Attr<"SparseTensorEncoding",
   // Data in sparse tensor encoding.
   let parameters = (
     ins
-    // A dimension level type for each dimension of the tensor type.
+    // A level-type for each level of the sparse storage.
     ArrayRefParameter<
       "::mlir::sparse_tensor::DimLevelType",
-      "per dimension level type"
+      "level-types"
       >: $dimLevelType,
-    // A dimension order on the indices of this tensor type.
+    // A permutation from (higher-ordering)-coordinates to level-coordinates.
     "AffineMap":$dimOrdering,
-    // A mapping between the original and higher-ordering index space.
+    // A mapping from dimension-coordinates to (higher-ordering)-coordinates.
     "AffineMap":$higherOrdering,
-    // The required bit width for pointer storage.
-    "unsigned":$pointerBitWidth,
-    // The required bit width for index storage.
-    "unsigned":$indexBitWidth,
-    // A dimension level type for each dimension of the tensor type.
+    // The required bitwidth for position storage.
+    "unsigned":$posWidth,
+    // The required bitwidth for coordinate storage.
+    "unsigned":$crdWidth,
+    // A slice attribute for each dimension of the tensor type.
     ArrayRefParameter<
       "::mlir::sparse_tensor::SparseTensorDimSliceAttr",
       "per dimension slice metadata"
@@ -238,23 +286,23 @@ def SparseTensorEncodingAttr : SparseTensor_Attr<"SparseTensorEncoding",
     AttrBuilder<(ins "ArrayRef<::mlir::sparse_tensor::DimLevelType>":$dimLevelType,
                      "AffineMap":$dimOrdering,
                      "AffineMap":$higherOrdering,
-                     "unsigned":$pointerBitWidth,
-                     "unsigned":$indexBitWidth), [{
+                     "unsigned":$posWidth,
+                     "unsigned":$crdWidth), [{
       return $_get($_ctxt, dimLevelType,
                          dimOrdering,
                          higherOrdering,
-                         pointerBitWidth,
-                         indexBitWidth,
+                         posWidth,
+                         crdWidth,
                          ArrayRef<::mlir::sparse_tensor::SparseTensorDimSliceAttr>{});
     }]>
   ];
 
   let extraClassDeclaration = [{
-    /// Returns the type for pointer storage based on pointerBitWidth
-    Type getPointerType() const;
+    /// Returns the type for position storage based on posWidth
+    Type getPosType() const;
 
-    /// Returns the type for index storage based on indexBitWidth
-    Type getIndexType() const;
+    /// Returns the type for coordinate storage based on crdWidth
+    Type getCrdType() const;
 
     /// Constructs a new encoding with the dimOrdering and higherOrdering
     /// reset to the default/identity.
@@ -316,10 +364,12 @@ def SparseTensorEncodingAttr : SparseTensor_Attr<"SparseTensorEncoding",
 // The C++ enum for Storage Specifier kind.
 def SparseTensorStorageSpecifierKindEnum
     : I32EnumAttr<"StorageSpecifierKind", "sparse tensor storage specifier kind", [
-        I32EnumAttrCase<"DimSize",    0, "dim_sz">,
-        I32EnumAttrCase<"PtrMemSize", 1, "ptr_mem_sz">,
-        I32EnumAttrCase<"IdxMemSize", 2, "idx_mem_sz">,
+        I32EnumAttrCase<"LvlSize",    0, "lvl_sz">,
+        I32EnumAttrCase<"PosMemSize", 1, "pos_mem_sz">,
+        I32EnumAttrCase<"CrdMemSize", 2, "crd_mem_sz">,
         I32EnumAttrCase<"ValMemSize", 3, "val_mem_sz">,
+        I32EnumAttrCase<"DimOffset",  4, "dim_offset">,
+        I32EnumAttrCase<"DimStride",  5, "dim_stride">,
       ]> {
   let genSpecializedAttr = 0;
   let cppNamespace = SparseTensor_Dialect.cppNamespace;
diff --git a/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorBase.td b/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorBase.td
index 2506da4042bd3..b0e9089c3230e 100644
--- a/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorBase.td
+++ b/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorBase.td
@@ -20,7 +20,7 @@ def SparseTensor_Dialect : Dialect {
     types first class citizens within the MLIR compiler infrastructure.
     The dialect forms a bridge between high-level operations on sparse
     tensors types and lower-level operations on the actual sparse storage
-    schemes consisting of pointers, indices, and values. Lower-level
+    schemes consisting of positions, coordinates, and values. Lower-level
     support may consist of fully generated code or may be provided by
     means of a small sparse runtime support library.
 
@@ -31,27 +31,29 @@ def SparseTensor_Dialect : Dialect {
     to tensor algebra by [Kjolstad17,Kjolstad20] in the Sparse Tensor
     Algebra Compiler (TACO) project (see http://tensor-compiler.org).
 
-    The MLIR implementation [Biketal22] closely follows the "sparse iteration
-    theory" that forms the foundation of TACO. A rewriting rule is applied to
-    each tensor expression in the Linalg dialect (MLIR's tensor index notation)
-    where the sparsity of tensors is indicated using the per-dimension level
-    types dense/compressed together with a specification of the order on the
-    dimensions (see [Chou18] for an in-depth discussions and possible
-    extensions to these level types). Subsequently, a topologically sorted
-    iteration graph, reflecting the required order on indices with respect
-    to the dimensions of each tensor, is constructed to ensure that all tensors
-    are visited in natural index order. Next, iteration lattices are
-    constructed for the tensor expression for every index in topological
-    order. Each iteration lattice point consists of a conjunction of tensor
-    indices together with a tensor (sub)expression that needs to be evaluated
-    for that conjunction.  Within the lattice, iteration points are ordered
-    according to the way indices are exhausted. As such these iteration
-    lattices drive actual sparse code generation, which consists of a
-    relatively straightforward one-to-one mapping from iteration lattices
-    to combinations of for-loops, while-loops, and if-statements. Sparse
-    tensor outputs that materialize uninitialized are handled with direct
-    insertions if all parallel loops are outermost or insertions that
-    indirectly go through a 1-dimensional access pattern expansion
+    The MLIR implementation [Biketal22] closely follows the "sparse
+    iteration theory" that forms the foundation of TACO.  A rewriting
+    rule is applied to each tensor expression in the Linalg dialect
+    (MLIR's tensor index notation) where the sparsity of tensors is
+    indicated using the per-level level-types (e.g., dense, compressed,
+    singleton) together with a specification of the order on the levels
+    (see [Chou18] for an in-depth discussions and possible extensions
+    to these level-types).  Subsequently, a topologically sorted
+    iteration graph, reflecting the required order on coordinates with
+    respect to the levels of each tensor, is constructed to ensure
+    that all tensors are visited in natural level-coordinate order.
+    Next, iteration lattices are constructed for the tensor expression
+    for every index in topological order.  Each iteration lattice point
+    consists of a conjunction of tensor coordinates together with a tensor
+    (sub)expression that needs to be evaluated for that conjunction.
+    Within the lattice, iteration points are ordered according to
+    the way coordinates are exhausted.  As such these iteration
+    lattices drive actual sparse code generation, which consists of
+    a relatively straightforward one-to-one mapping from iteration
+    lattices to combinations of for-loops, while-loops, and if-statements.
+    Sparse tensor outputs that materialize uninitialized are handled with
+    direct insertions if all parallel loops are outermost or insertions
+    that indirectly go through a 1-dimensional access pattern expansion
     (a.k.a. workspace) where feasible [Gustavson72,Bik96,Kjolstad19].
 
     * [Bik96] Aart J.C. Bik. Compiler Support for Sparse Matrix Computations.
diff --git a/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorOps.td b/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorOps.td
index 9d0b124ff5b47..336f19686a1c5 100644
--- a/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorOps.td
+++ b/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorOps.td
@@ -54,82 +54,94 @@ def SparseTensor_NewOp : SparseTensor_Op<"new", [Pure]>,
 }
 
 def SparseTensor_PackOp : SparseTensor_Op<"pack", [Pure]>,
-    Arguments<(ins 1DTensorOf<[AnyType]>:$data,
-                   2DTensorOf<[AnySignlessIntegerOrIndex]>:$indices)>,
+    Arguments<(ins 1DTensorOf<[AnyType]>:$values,
+                   2DTensorOf<[AnySignlessIntegerOrIndex]>:$coordinates)>,
     Results<(outs AnySparseTensor: $result)> {
-  let summary = "Returns a sparse tensor from the given (data, indices) pair";
+  let summary = "Returns a sparse tensor from the given (values, coordinates) pair";
 
   let description = [{
-    Packs the data/indices into a COO sparse tensor. The coordinates in `indices`
-    shall not exceed the dimension sizes of the returned sparse tensor. Note
-    that the returned tensor must be statically shaped because it is impossible
-    to infer the shape from sparse coordinates.
-
-    `$indices`: stored via a 2-D tensor of integer elements with shape [N, ndims],
-    which specifies the indices of the elements in the sparse tensor that contains
-    non-zero values.
-
-    `$data`: stored via a 1-D tensor with shape [N], that supplies the corresponding
-    values for the indices.
-
-    The operation can be used to materialize a sparse tensor from external sources.
-    E.g., when passing from Python as two numpy arrays for data and indices.
+    Packs the values/coordinates into a COO sparse tensor.  The length
+    of `values` must match the outer-length of `coordinates`, since these
+    two tensors are "zipped" together.  The `coordinates` argument provides
+    level-coords for each value, therefore, the inner-length of `coordinates`
+    must match the level-rank of the returned tensor, and each level-coords
+    must be valid for the level-sizes of the returned tensor.  Note that
+    the returned tensor must be statically shaped because it is impossible
+    to infer the dimension-shape from level-coordinates alone.
+
+    TODO: The returned tensor is allowed (in principle) to have non-identity
+    dimOrdering/higherOrdering mappings.  However, the current implementation
+    does not yet support them.
+
+    - `coordinates : tensor<NSE x lvlRank x iType>`
+      supplies the level-coords for each element in `values`.
+    - `values : tensor<NSE x V>`
+      supplies the corresponding values for each entry in `coordinates`.
+
+    This operation can be used to materialize a sparse tensor from external
+    sources; e.g., when passing two numpy arrays from Python.
 
     Example:
-    ```mlir
-    %data    = arith.constant dense<[ 1.1,   2.2,   3.3 ]> : tensor<3xf64>
-    %indices = arith.constant dense<[[0,0], [1,2], [1,3]]> : tensor<3x2xindex>
 
-    %st = sparse_tensor.pack %data, %indices : tensor<3xf64>, tensor<3x2xindex
-                                            to tensor<3x4xf64, #COO>
+    ```mlir
+    %values      = arith.constant dense<[ 1.1,   2.2,   3.3 ]> : tensor<3xf64>
+    %coordinates = arith.constant dense<[[0,0], [1,2], [1,3]]> : tensor<3x2xindex>
+    %st = sparse_tensor.pack %values, %coordinates
+        : tensor<3xf64>, tensor<3x2xindex> to tensor<3x4xf64, #COO>
     // yields COO format |1.1, 0.0, 0.0, 0.0|
     //     of 3x4 matrix |0.0, 0.0, 2.2, 3.3|
     //                   |0.0, 0.0, 0.0, 0.0|
     ```
   }];
 
-  let assemblyFormat = "$data `,` $indices attr-dict `:` type($data) `,` type($indices)"
-                                                   "`to` type($result)";
+  let assemblyFormat =
+    "$values `,` $coordinates attr-dict"
+    "`:` type($values) `,` type($coordinates) `to` type($result)";
 
   let hasVerifier = 1;
 }
 
 def SparseTensor_UnpackOp : SparseTensor_Op<"unpack", [Pure]>,
     Arguments<(ins AnySparseTensor:$tensor)>,
-    Results<(outs 1DTensorOf<[AnyType]>:$data,
-                  2DTensorOf<[AnySignlessIntegerOrIndex]>:$indices,
-                  AnySignlessIntegerOrIndex:$nnz)> {
-  let summary = "Returns the (data, indices) pair unpacked from the input tensor";
+    Results<(outs 1DTensorOf<[AnyType]>:$values,
+                  2DTensorOf<[AnySignlessIntegerOrIndex]>:$coordinates,
+                  AnySignlessIntegerOrIndex:$nse)> {
+  let summary = "Returns the (values, coordinates) pair unpacked from the input tensor";
 
   let description = [{
-    Unpack is the inverse operation of `sparse_tensor::pack`. It returns the data/indices
-    extracted from a COO sparse tensor. Additionally, it also returns an integer value
-    indicating the number of entries in the source tensor.
+    The unpack operation is the inverse of `sparse_tensor::pack`.  It returns
+    the values, level-coordinates, and number-of-stored-entries extracted
+    from the sparse tensor.  The source tensor is allowed (in principle)
+    to have non-identity dimOrdering/higherOrdering mappings.  Regardless
+    of the mappings, the returned `coordinates` are always level-coordinates,
+    because this is what we mean by "unpacking" as opposed to other forms
+    of exposing sparse tensors to external clients.  This operation can be
+    used for returning an unpacked MLIR sparse tensor to frontend; e.g.,
+    returning two numpy arrays to Python.
 
-    The operation can be used to return an unpacked MLIR sparse tensor to frontend.
-    E.g., returning two numpy arrays for data and indices.
+    TODO: the current implementation does not yet support non-identity mappings.
 
-    The unpack operation ends the life time of the sparse tensor, and using this
-    after the unpack is undefined behavior.
+    This operation ends the lifetime of the sparse tensor, and using
+    the tensor after the unpack is undefined behavior.
 
     Example:
+
     ```mlir
     // input COO format |1.1, 0.0, 0.0, 0.0|
     //    of 3x4 matrix |0.0, 0.0, 2.2, 3.3|
     //                  |0.0, 0.0, 0.0, 0.0|
-    %data, %indices, %nnz = sparse_tensor.unpack %st
-                          : tensor<3x4xf64, #COO>
-                         to tensor<2xf64>, tensor<2x2xindex>, index
-
-    // %data    = arith.constant dense<[ 1.1,   2.2,   3.3 ]> : tensor<3xf64>
-    // %indices = arith.constant dense<[[0,0], [1,2], [1,3]]> : tensor<3x2xindex>
-    // %nnz = 2
-
+    %values, %coordinates, %nse
+      = sparse_tensor.unpack %st
+      : tensor<3x4xf64, #COO> to tensor<2xf64>, tensor<2x2xindex>, index
+    // %values      = arith.constant dense<[ 1.1,   2.2,   3.3 ]> : tensor<3xf64>
+    // %coordinates = arith.constant dense<[[0,0], [1,2], [1,3]]> : tensor<3x2xindex>
+    // %nse = 3
     ```
   }];
 
-  let assemblyFormat = "$tensor attr-dict `:` type($tensor) "
-                                        "`to` type($data) `,` type($indices)`,` type($nnz)";
+  let assemblyFormat =
+    "$tensor attr-dict `:` type($tensor)"
+    "`to` type($values) `,` type($coordinates) `,` type($nse)";
 
   let hasVerifier = 1;
 }
@@ -182,25 +194,25 @@ def SparseTensor_ConvertOp : SparseTensor_Op<"convert",
   let hasVerifier = 1;
 }
 
-def SparseTensor_ToPointersOp : SparseTensor_Op<"pointers", [Pure]>,
-    Arguments<(ins AnySparseTensor:$tensor, IndexAttr:$dimension)>,
+def SparseTensor_ToPositionsOp : SparseTensor_Op<"positions", [Pure]>,
+    Arguments<(ins AnySparseTensor:$tensor, LevelAttr:$level)>,
     Results<(outs AnyStridedMemRefOfRank<1>:$result)> {
-  let summary = "Extracts pointers array at given dimension from a tensor";
+  let summary = "Extracts the `level`-th positions array of the `tensor`";
   let description = [{
-    Returns the pointers array of the sparse storage format at the
-    given dimension for the given sparse tensor. This is similar to the
-    `bufferization.to_memref` operation in the sense that it provides a bridge
-    between a tensor world view and a bufferized world view. Unlike the
-    `bufferization.to_memref` operation, however, this sparse operation actually
-    lowers into code that extracts the pointers array from the sparse storage
-    scheme (either by calling a support library or through direct code).
+    Returns the positions array of the tensor's storage at the given
+    level.  This is similar to the `bufferization.to_memref` operation
+    in the sense that it provides a bridge between a tensor world view
+    and a bufferized world view.  Unlike the `bufferization.to_memref`
+    operation, however, this sparse operation actually lowers into code
+    that extracts the positions array from the sparse storage itself
+    (either by calling a support library or through direct code).
 
     Writing into the result of this operation is undefined behavior.
 
     Example:
 
     ```mlir
-    %1 = sparse_tensor.pointers %0 { dimension = 1 : index }
+    %1 = sparse_tensor.positions %0 { level = 1 : index }
        : tensor<64x64xf64, #CSR> to memref<?xindex>
     ```
   }];
@@ -208,25 +220,25 @@ def SparseTensor_ToPointersOp : SparseTensor_Op<"pointers", [Pure]>,
   let hasVerifier = 1;
 }
 
-def SparseTensor_ToIndicesOp : SparseTensor_Op<"indices", [Pure]>,
-    Arguments<(ins AnySparseTensor:$tensor, IndexAttr:$dimension)>,
+def SparseTensor_ToCoordinatesOp : SparseTensor_Op<"coordinates", [Pure]>,
+    Arguments<(ins AnySparseTensor:$tensor, LevelAttr:$level)>,
     Results<(outs AnyStridedMemRefOfRank<1>:$result)> {
-  let summary = "Extracts indices array at given dimension from a tensor";
+  let summary = "Extracts the `level`-th coordinates array of the `tensor`";
   let description = [{
-    Returns the indices array of the sparse storage format at the
-    given dimension for the given sparse tensor. This is similar to the
-    `bufferization.to_memref` operation in the sense that it provides a bridge
-    between a tensor world view and a bufferized world view. Unlike the
-    `bufferization.to_memref` operation, however, this sparse operation actually
-    lowers into code that extracts the indices array from the sparse storage
-    scheme (either by calling a support library or through direct code).
+    Returns the coordinates array of the tensor's storage at the given
+    level.  This is similar to the `bufferization.to_memref` operation
+    in the sense that it provides a bridge between a tensor world view
+    and a bufferized world view.  Unlike the `bufferization.to_memref`
+    operation, however, this sparse operation actually lowers into code
+    that extracts the coordinates array from the sparse storage itself
+    (either by calling a support library or through direct code).
 
     Writing into the result of this operation is undefined behavior.
 
     Example:
 
     ```mlir
-    %1 = sparse_tensor.indices %0 { dimension = 1 : index }
+    %1 = sparse_tensor.coordinates %0 { level = 1 : index }
        : tensor<64x64xf64, #CSR> to memref<?xindex>
     ```
   }];
@@ -234,30 +246,31 @@ def SparseTensor_ToIndicesOp : SparseTensor_Op<"indices", [Pure]>,
   let hasVerifier = 1;
 }
 
-def SparseTensor_ToIndicesBufferOp : SparseTensor_Op<"indices_buffer", [Pure]>,
+def SparseTensor_ToCoordinatesBufferOp : SparseTensor_Op<"coordinates_buffer", [Pure]>,
     Arguments<(ins AnySparseTensor:$tensor)>,
     Results<(outs AnyStridedMemRefOfRank<1>:$result)> {
-  let summary = "Extracts the linear indices array from a tensor";
+  let summary = "Extracts the linear coordinates array from a tensor";
   let description = [{
-    Returns the linear indices array for a sparse tensor with a trailing COO
-    region with at least two dimensions. It is an error if the tensor doesn't
-    contain such a COO region. This is similar to the `bufferization.to_memref`
-    operation in the sense that it provides a bridge between a tensor world view
-    and a bufferized world view. Unlike the `bufferization.to_memref` operation,
-    however, this sparse operation actually lowers into code that extracts the
-    linear indices array from the sparse storage scheme that stores the indices
-    for the COO region as an array of structures. For example, a 2D COO sparse
-    tensor with two non-zero elements at coordinates (1, 3) and (4, 6) are
-    stored in a linear buffer as (1, 4, 3, 6) instead of two buffer as (1, 4)
-    and (3, 6).
+    Returns the linear coordinates array for a sparse tensor with
+    a trailing COO region with at least two levels.  It is an error
+    if the tensor doesn't contain such a COO region.  This is similar
+    to the `bufferization.to_memref` operation in the sense that it
+    provides a bridge between a tensor world view and a bufferized
+    world view.  Unlike the `bufferization.to_memref` operation,
+    however, this operation actually lowers into code that extracts
+    the linear coordinates array from the sparse storage scheme that
+    stores the coordinates for the COO region as an array of structures.
+    For example, a 2D COO sparse tensor with two non-zero elements at
+    coordinates (1, 3) and (4, 6) are stored in a linear buffer as
+    (1, 4, 3, 6) instead of two buffer as (1, 4) and (3, 6).
 
     Writing into the result of this operation is undefined behavior.
 
     Example:
 
     ```mlir
-    %1 = sparse_tensor.indices_buffer %0
-    : tensor<64x64xf64, #COO> to memref<?xindex>
+    %1 = sparse_tensor.coordinates_buffer %0
+       : tensor<64x64xf64, #COO> to memref<?xindex>
     ```
   }];
   let assemblyFormat = "$tensor attr-dict `:` type($tensor) `to` type($result)";
@@ -345,41 +358,65 @@ def SparseTensor_ToSliceStrideOp : SparseTensor_Op<"slice.stride", [Pure]>,
 }
 
 def SparseTensor_StorageSpecifierInitOp : SparseTensor_Op<"storage_specifier.init", [Pure]>,
+    Arguments<(ins Optional<SparseTensorStorageSpecifier>:$source)>,
     Results<(outs SparseTensorStorageSpecifier:$result)> {
   let summary = "";
   let description = [{
-    Returns an initial storage specifier value. A storage specifier value holds
-    the sizes for tensor dimensions, pointer arrays, index arrays, and the value array.
+    Returns an initial storage specifier value.  A storage specifier
+    value holds the level-sizes, position arrays, coordinate arrays,
+    and the value array.
+    If this is a specifier for slices, it also holds the extra strides/offsets
+    for each tensor dimension.
+
+    TODO: The sparse tensor slice support is currently in a unstable state, and
+    is subject to change in the future.
 
     Example:
 
     ```mlir
+    #CSR = #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ]}>
+    #CSR_SLICE = #sparse_tensor.encoding<{
+      dimLevelType = [ "dense", "compressed" ],
+      slice = [ (1, 4, 1), (1, 4, 2) ]
+    }>
+
     %0 = sparse_tensor.storage_specifier.init :  !sparse_tensor.storage_specifier<#CSR>
+    %1 = sparse_tensor.storage_specifier.init with %src
+         : !sparse_tensor.storage_specifier<#CSR> to
+           !sparse_tensor.storage_specifier<#CSR_SLICE>
     ```
   }];
 
-  let assemblyFormat = "attr-dict `:` qualified(type($result))";
+  let builders = [
+    OpBuilder<(ins "Type":$result),
+    [{
+      build($_builder, $_state, result, Value());
+    }]>
+  ];
+
+  let assemblyFormat = "attr-dict (`with` $source^)? `:` (`from` qualified(type($source))^ `to`)?"
+                                                        " qualified(type($result))";
 }
 
 def SparseTensor_GetStorageSpecifierOp : SparseTensor_Op<"storage_specifier.get", [Pure]>,
     Arguments<(ins SparseTensorStorageSpecifier:$specifier,
                    SparseTensorStorageSpecifierKindAttr:$specifierKind,
-                   OptionalAttr<IndexAttr>:$dim)>,
+                   OptionalAttr<LevelAttr>:$level)>,
     Results<(outs Index:$result)> {
   let summary = "";
   let description = [{
     Returns the requested field of the given storage_specifier.
 
-    Example of querying the size of the index array for level 0:
+    Example of querying the size of the coordinates array for level 0:
 
     ```mlir
-    %0 = sparse_tensor.storage_specifier.get %arg0 idx_mem_sz at 0
+    %0 = sparse_tensor.storage_specifier.get %arg0 crd_mem_sz at 0
          : !sparse_tensor.storage_specifier<#COO>
     ```
   }];
 
-  let assemblyFormat = "$specifier $specifierKind (`at` $dim^)? attr-dict `:` "
-                       "qualified(type($specifier))";
+  let assemblyFormat = "$specifier $specifierKind (`at` $level^)? attr-dict"
+                       "`:` qualified(type($specifier))";
   let hasVerifier = 1;
   let hasFolder = 1;
 }
@@ -388,7 +425,7 @@ def SparseTensor_SetStorageSpecifierOp : SparseTensor_Op<"storage_specifier.set"
     [Pure, AllTypesMatch<["result", "specifier"]>]>,
     Arguments<(ins SparseTensorStorageSpecifier:$specifier,
                    SparseTensorStorageSpecifierKindAttr:$specifierKind,
-                   OptionalAttr<IndexAttr>:$dim,
+                   OptionalAttr<LevelAttr>:$level,
                    Index:$value)>,
     Results<(outs SparseTensorStorageSpecifier:$result)> {
   let summary = "";
@@ -396,16 +433,15 @@ def SparseTensor_SetStorageSpecifierOp : SparseTensor_Op<"storage_specifier.set"
     Set the field of the storage specifier to the given input value. Returns
     the updated storage_specifier as a new SSA value.
 
-    Example of updating the sizes of the index array for level 0:
+    Example of updating the sizes of the coordinates array for level 0:
 
     ```mlir
-    %0 = sparse_tensor.storage_specifier.set %arg0 idx_mem_sz at 0 with %new_sz
+    %0 = sparse_tensor.storage_specifier.set %arg0 crd_mem_sz at 0 with %new_sz
        : !sparse_tensor.storage_specifier<#COO>
-
     ```
   }];
-  let assemblyFormat = "$specifier $specifierKind (`at` $dim^)? `with` $value attr-dict `:` "
-                       "qualified(type($result))";
+  let assemblyFormat = "$specifier $specifierKind (`at` $level^)? `with` $value"
+                       " attr-dict `:` qualified(type($result))";
   let hasVerifier = 1;
 }
 
@@ -429,16 +465,16 @@ def SparseTensor_NumberOfEntriesOp : SparseTensor_Op<"number_of_entries", [Pure]
 }
 
 def SparseTensor_ConcatenateOp : SparseTensor_Op<"concatenate", [Pure]>,
-    Arguments<(ins Variadic<AnyRankedTensor>:$inputs, IndexAttr:$dimension)>,
+    Arguments<(ins Variadic<AnyRankedTensor>:$inputs, DimensionAttr:$dimension)>,
     Results<(outs AnyRankedTensor:$result)> {
 
   let summary = "Concatenates a list of tensors into a single tensor.";
   let description = [{
-     Concatenates a list input tensors and the output tensor with the same rank.
-     The concatenation happens on the specified `dimension` (0<= dimension < rank).
-     The resulting `dimension` size is the sum of all the input dimension sizes,
-     while all the other dimensions should have the same size in the input and
-     output tensors.
+     Concatenates a list input tensors and the output tensor with the same
+     dimension-rank.  The concatenation happens on the specified `dimension`
+     (0 <= dimension < dimRank).  The resulting `dimension` size is the
+     sum of all the input sizes for that dimension, while all the other
+     dimensions should have the same size in the input and output tensors.
 
      Only statically-sized input tensors are accepted, while the output tensor
      can be dynamically-sized.
@@ -470,24 +506,23 @@ def SparseTensor_InsertOp : SparseTensor_Op<"insert",
      AllTypesMatch<["tensor", "result"]>]>,
     Arguments<(ins AnyType:$value,
                AnySparseTensor:$tensor,
-               Variadic<Index>:$indices)>,
+               Variadic<Index>:$lvlCoords)>,
     Results<(outs AnySparseTensor:$result)> {
-  string summary = "Inserts a value into given sparse tensor";
+  string summary = "Inserts a value into the sparse tensor";
   string description = [{
-    Inserts the given value at given indices into the underlying
-    sparse storage format of the given tensor with the given indices.
-    The arity of indices must match the rank of the tensor. This
-    operation can only be applied when a tensor materializes unintialized
-    with a `bufferization.alloc_tensor` operation and the final tensor
-    is constructed with a `load` operation that has the `hasInserts`
-    attribute set.
-
-    Properties in the sparse tensor type fully describe what kind
-    of insertion order is allowed. When all dimensions have "unique"
+    Inserts the value into the underlying storage of the tensor at the
+    given level-coordinates.  The arity of `lvlCoords` must match the
+    level-rank of the tensor.  This operation can only be applied when
+    the tensor materializes unintialized from a `bufferization.alloc_tensor`
+    operation and the final tensor is constructed with a `load` operation
+    which has the `hasInserts` attribute set.
+
+    The level-properties of the sparse tensor type fully describe what
+    kind of insertion order is allowed.  When all levels have "unique"
     and "ordered" properties, for example, insertions should occur in
-    strict lexicographical index order. Other properties define
-    different insertion regimens. Inserting in a way contrary to
-    these properties results in undefined behavior.
+    strict lexicographical level-coordinate order.  Other properties
+    define different insertion regimens.  Inserting in a way contrary
+    to these properties results in undefined behavior.
 
     Note that this operation is "impure" in the sense that even though
     the result is modeled through an SSA value, the insertion is eventually
@@ -501,7 +536,8 @@ def SparseTensor_InsertOp : SparseTensor_Op<"insert",
     %result = sparse_tensor.insert %val into %tensor[%i,%j] : tensor<1024x1024xf64, #CSR>
     ```
   }];
-  let assemblyFormat = "$value `into` $tensor `[` $indices `]` attr-dict `:` type($tensor)";
+  let assemblyFormat = "$value `into` $tensor `[` $lvlCoords `]` attr-dict"
+                       "`:` type($tensor)";
   let hasVerifier = 1;
 }
 
@@ -580,7 +616,7 @@ def SparseTensor_ExpandOp : SparseTensor_Op<"expand", []>,
                   Index:$count)> {
   string summary = "Expands an access pattern for insertion";
   string description = [{
-    Performs an access pattern expansion for the innermost dimensions of the
+    Performs an access pattern expansion for the innermost levels of the
     given tensor. This operation is useful to implement kernels in which a
     sparse tensor appears as output. This technique is known under several
     different names and using several alternative implementations,
@@ -588,15 +624,17 @@ def SparseTensor_ExpandOp : SparseTensor_Op<"expand", []>,
     [Pissanetzky84], in phase scan [Duff90], access pattern expansion [Bik96],
     and workspaces [Kjolstad19].
 
-    The values and filled array have sizes that suffice for a *dense* innermost
-    dimension (e.g. a full row for matrices). The added array and count are used
-    to store new indices when a false value is encountered in the filled array.
-    All arrays should be allocated before the loop (possibly even shared between
-    loops in a future optimization) so that their *dense* initialization can be
-    amortized over many iterations. Setting and resetting the dense arrays in
-    the loop nest itself is kept *sparse* by only iterating over set elements
-    through an indirection using the added array, so that the operations are
-    kept proportional to the number of nonzeros.
+    The `values` and `filled` arrays must have lengths equal to the
+    level-size of the innermost level (i.e., as if the innermost level
+    were *dense*).  The `added` array and `count` are used to store new
+    level-coordinates when a false value is encountered in the `filled`
+    array.  All arrays should be allocated before the loop (possibly even
+    shared between loops in a future optimization) so that their *dense*
+    initialization can be amortized over many iterations.  Setting and
+    resetting the dense arrays in the loop nest itself is kept *sparse*
+    by only iterating over set elements through an indirection using
+    the added array, so that the operations are kept proportional to
+    the number of nonzeros.
 
     Note that this operation is "impure" in the sense that even though the
     results are modeled through SSA values, the operation relies on a proper
@@ -620,18 +658,19 @@ def SparseTensor_CompressOp : SparseTensor_Op<"compress",
                    StridedMemRefRankOf<[Index],[1]>:$added,
                    Index:$count,
                    AnySparseTensor:$tensor,
-                   Variadic<Index>:$indices)>,
+                   Variadic<Index>:$lvlCoords)>,
     Results<(outs AnySparseTensor:$result)> {
   string summary = "Compressed an access pattern for insertion";
   string description = [{
     Finishes a single access pattern expansion by moving inserted elements
     into the sparse storage scheme of the given tensor with the given
-    indices. The arity of indices is one less than the rank of the tensor,
-    with the remainder innermost indices defined through the added array.
-    The values and filled array are reset in a *sparse* fashion by only
-    iterating over set elements through an indirection using the added
-    array, so that the operations are kept proportional to the number of
-    nonzeros. See the `sparse_tensor.expand` operation for more details.
+    level-coordinates.  The arity of `lvlCoords` is one less than the
+    level-rank of the tensor, with the coordinate of the innermost
+    level defined through the `added` array.  The `values` and `filled`
+    arrays are reset in a *sparse* fashion by only iterating over set
+    elements through an indirection using the `added` array, so that
+    the operations are kept proportional to the number of nonzeros.
+    See the `sparse_tensor.expand` operation for more details.
 
     Note that this operation is "impure" in the sense that even though
     the result is modeled through an SSA value, the insertion is eventually
@@ -645,7 +684,7 @@ def SparseTensor_CompressOp : SparseTensor_Op<"compress",
     ```
   }];
   let assemblyFormat = "$values `,` $filled `,` $added `,` $count"
-                       " `into` $tensor `[` $indices `]` attr-dict"
+                       " `into` $tensor `[` $lvlCoords `]` attr-dict"
                        " `:` type($values) `,` type($filled) `,` type($added)"
                        " `,` type($tensor)";
   let hasVerifier = 1;
@@ -1161,17 +1200,16 @@ def SparseTensor_ForeachOp : SparseTensor_Op<"foreach",
      the dimensions are visited (e.g., row first or column first). This is only
      applicable when the input tensor is a non-annotated dense tensor.
 
-     For an input tensor with rank n, the block must take n + 1 (and additional loop
-     carried variables as described below) arguments. The first n arguments must be
-     Index type, together indicating the current coordinates of the element being visited.
-     The last argument must have the same type as the
-     tensor's element type, representing the actual value loaded from the input
-     tensor at the given coordinates.
+     For an input tensor with dim-rank `n`, the block must take `n + 1`
+     arguments (plus additional loop-carried variables as described below).
+     The first `n` arguments provide the dimension-coordinates of the element
+     being visited, and must all have `index` type.  The `(n+1)`-th argument
+     provides the element's value, and must have the tensor's element type.
 
      `sparse_tensor.foreach` can also operate on loop-carried variables and returns
      the final values after loop termination. The initial values of the variables are
      passed as additional SSA operands to the "sparse_tensor.foreach" following the n + 1
-     SSA values mentioned above (n coordinate and 1 value).
+     SSA values mentioned above (n coordinates and 1 value).
 
      The region must terminate with a "sparse_tensor.yield" that passes the current
      values of all loop-carried variables to the next iteration, or to the
@@ -1188,9 +1226,10 @@ def SparseTensor_ForeachOp : SparseTensor_Op<"foreach",
      }
      ```
 
-     It is important to note that foreach generated loop iterates over the stored elements
-     in the storage order. However, no matter what storage order is used, the indices passed
-     to the block always obey the original dimension order.
+     It is important to note that the generated loop iterates over
+     elements in their storage order.  However, regardless of the
+     storage scheme used by the tensor, the block is always given
+     the dimension-coordinates.
 
      For example:
      ```mlir
diff --git a/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorType.h b/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorType.h
index c2adc2694e3ef..bdcd3632b5d38 100644
--- a/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorType.h
+++ b/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorType.h
@@ -58,6 +58,12 @@ class SparseTensorType {
       : SparseTensorType(
             RankedTensorType::get(stp.getShape(), stp.getElementType(), enc)) {}
 
+  // Copy-assignment would be implicitly deleted (because our fields
+  // are const), so we explicitly delete it for clarity.
+  SparseTensorType &operator=(const SparseTensorType &) = delete;
+  // So we must explicitly define the copy-ctor to silence -Wdeprecated-copy.
+  SparseTensorType(const SparseTensorType &) = default;
+
   /// Constructs a new `SparseTensorType` with the same dimension-shape
   /// and element type, but with the encoding replaced by the given encoding.
   SparseTensorType withEncoding(SparseTensorEncodingAttr newEnc) const {
@@ -226,22 +232,20 @@ class SparseTensorType {
   bool isOrderedLvl(Level l) const { return isOrderedDLT(getLvlType(l)); }
   bool isUniqueLvl(Level l) const { return isUniqueDLT(getLvlType(l)); }
 
-  /// Returns the index-overhead bitwidth, defaulting to zero.
-  unsigned getIndexBitWidth() const { return enc ? enc.getIndexBitWidth() : 0; }
+  /// Returns the coordinate-overhead bitwidth, defaulting to zero.
+  unsigned getCrdWidth() const { return enc ? enc.getCrdWidth() : 0; }
 
-  /// Returns the pointer-overhead bitwidth, defaulting to zero.
-  unsigned getPointerBitWidth() const {
-    return enc ? enc.getPointerBitWidth() : 0;
-  }
+  /// Returns the position-overhead bitwidth, defaulting to zero.
+  unsigned getPosWidth() const { return enc ? enc.getPosWidth() : 0; }
 
-  /// Returns the index-overhead MLIR type, defaulting to `IndexType`.
-  Type getIndexType() const {
-    return detail::getIntegerOrIndexType(getContext(), getIndexBitWidth());
+  /// Returns the coordinate-overhead MLIR type, defaulting to `IndexType`.
+  Type getCrdType() const {
+    return detail::getIntegerOrIndexType(getContext(), getCrdWidth());
   }
 
-  /// Returns the pointer-overhead MLIR type, defaulting to `IndexType`.
-  Type getPointerType() const {
-    return detail::getIntegerOrIndexType(getContext(), getPointerBitWidth());
+  /// Returns the position-overhead MLIR type, defaulting to `IndexType`.
+  Type getPosType() const {
+    return detail::getIntegerOrIndexType(getContext(), getPosWidth());
   }
 
 private:
diff --git a/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorTypes.td b/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorTypes.td
index 3ae40d625c8a2..9a8c13ea810c2 100644
--- a/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorTypes.td
+++ b/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorTypes.td
@@ -37,11 +37,11 @@ def SparseTensor_StorageSpecifier : SparseTensor_Type<"StorageSpecifier"> {
     encoding ::= attribute-value
     ```
 
-    Values with storage_specifier types represent aggregated storage scheme metadata
-    for the given sparse tensor encoding.
-    It currently holds a set of values for sizes of sparse tensor dimension, index array,
-    pointer array and value array.
-    Note that the type is not yet stable and subject to change in the near future.
+    Values with storage_specifier types represent aggregated storage scheme
+    metadata for the given sparse tensor encoding.  It currently holds
+    a set of values for level-sizes, coordinate arrays, position arrays,
+    and value array.  Note that the type is not yet stable and subject to
+    change in the near future.
 
     Examples:
 
diff --git a/mlir/include/mlir/Dialect/SparseTensor/Pipelines/Passes.h b/mlir/include/mlir/Dialect/SparseTensor/Pipelines/Passes.h
index d3a16de8c5519..09758c92da237 100644
--- a/mlir/include/mlir/Dialect/SparseTensor/Pipelines/Passes.h
+++ b/mlir/include/mlir/Dialect/SparseTensor/Pipelines/Passes.h
@@ -53,6 +53,13 @@ struct SparseCompilerOptions
               "any-storage-any-loop",
               "Enable sparse parallelization for any storage and loop."))};
 
+  PassOptions::Option<bool> enableIndexReduction{
+      *this, "enable-index-reduction",
+      desc("Enable dependent index reduction based algorithm to handle "
+           "non-trivial index expressions on sparse inputs (experimental "
+           "features)"),
+      init(false)};
+
   PassOptions::Option<bool> enableRuntimeLibrary{
       *this, "enable-runtime-library",
       desc("Enable runtime library for manipulating sparse tensors"),
@@ -108,7 +115,7 @@ struct SparseCompilerOptions
 
   /// Projects out the options for `createSparsificationPass`.
   SparsificationOptions sparsificationOptions() const {
-    return SparsificationOptions(parallelization);
+    return SparsificationOptions(parallelization, enableIndexReduction);
   }
 
   /// Projects out the options for `createSparseTensorConversionPass`.
diff --git a/mlir/include/mlir/Dialect/SparseTensor/Transforms/Passes.h b/mlir/include/mlir/Dialect/SparseTensor/Transforms/Passes.h
index df465e443d36a..734ffa0d1e803 100644
--- a/mlir/include/mlir/Dialect/SparseTensor/Transforms/Passes.h
+++ b/mlir/include/mlir/Dialect/SparseTensor/Transforms/Passes.h
@@ -49,11 +49,12 @@ enum class SparseParallelizationStrategy {
 
 /// Options for the Sparsification pass.
 struct SparsificationOptions {
-  SparsificationOptions(SparseParallelizationStrategy p)
-      : parallelizationStrategy(p) {}
+  SparsificationOptions(SparseParallelizationStrategy p, bool idxReduc)
+      : parallelizationStrategy(p), enableIndexReduction(idxReduc) {}
   SparsificationOptions()
-      : SparsificationOptions(SparseParallelizationStrategy::kNone) {}
+      : SparsificationOptions(SparseParallelizationStrategy::kNone, false) {}
   SparseParallelizationStrategy parallelizationStrategy;
+  bool enableIndexReduction;
 };
 
 /// Sets up sparsification rewriting rules with the given options.
diff --git a/mlir/include/mlir/Dialect/SparseTensor/Transforms/Passes.td b/mlir/include/mlir/Dialect/SparseTensor/Transforms/Passes.td
index 6ec9d4eb9e46e..0df55d918e3d4 100644
--- a/mlir/include/mlir/Dialect/SparseTensor/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/SparseTensor/Transforms/Passes.td
@@ -81,6 +81,9 @@ def SparsificationPass : Pass<"sparsification", "ModuleOp"> {
   ];
   // TODO(57514): These enum options are duplicated in Passes.h.
   let options = [
+    Option<"enableIndexReduction", "enable-index-reduction", "bool",
+           "false",
+           "Enable dependent index reduction based algorithm to handle non-trivial index expressions on sparse inputs (experimental features)">,
     Option<"parallelization", "parallelization-strategy", "mlir::SparseParallelizationStrategy",
            "mlir::SparseParallelizationStrategy::kNone",
            "Set the parallelization strategy", [{llvm::cl::values(
diff --git a/mlir/include/mlir/Dialect/SparseTensor/Utils/Merger.h b/mlir/include/mlir/Dialect/SparseTensor/Utils/Merger.h
index c96f76ee1c094..59c5b78fda7b8 100644
--- a/mlir/include/mlir/Dialect/SparseTensor/Utils/Merger.h
+++ b/mlir/include/mlir/Dialect/SparseTensor/Utils/Merger.h
@@ -15,6 +15,7 @@
 
 #include "mlir/Dialect/Linalg/IR/Linalg.h"
 #include "mlir/Dialect/SparseTensor/IR/Enums.h"
+#include "mlir/Dialect/SparseTensor/IR/SparseTensor.h"
 #include "mlir/IR/Value.h"
 #include "llvm/ADT/BitVector.h"
 #include <optional>
@@ -23,11 +24,27 @@ namespace mlir {
 namespace sparse_tensor {
 
 /// Tensor expression kind.
+///
+/// The `kLoopVar` leaf kind is for representing `linalg::IndexOp`.
+/// That is, its argument is a `LoopId` identifying the loop-variable
+/// in question, and its value will be the current iteration's value
+/// of that loop-variable.  See the `LoopId` documentation for more details.
+//
+// TODO: make this an `enum class` nested in the `TensorExp` class;
+// to improve namespacing, and match the pattern used by other "Kind"
+// enums in MLIR.
+//
+// TODO: Modify this definition so that the numeric values already encode
+// the `ExpArity` (while extending the notion of "arity" to include not
+// just the number of `ExprId` children the node has, but also whether the
+// node has a `Value` and/or `Operation*`).  Doing this will avoid needing
+// to enumerate all the kinds in `getExpArity` and in the `TensorExp` ctor,
+// and should help clean up a few other places as well.
 enum Kind {
   // Leaf.
   kTensor = 0,
   kInvariant,
-  kIndex,
+  kLoopVar,
   // Unary operations.
   kAbsF,
   kAbsC,
@@ -87,27 +104,94 @@ enum Kind {
   kReduce, // semiring reduction op
 };
 
+// TODO: These type aliases currently only serve to make the code more
+// self-documenting, however because they are not type-checked they can
+// do nothing to prevent mixups.  We should really change them from mere
+// aliases to actual struct definitions, so that we can type-check them.
+
+/// Tensor identifiers.  The valid set of identifiers is defined by the
+/// first argument passed to the `Merger` ctor.
+using TensorId = unsigned;
+
+/// Loop identifiers.  The valid set of identifiers is defined by the
+/// second two arguments to the `Merger` ctor.
+///
+/// These identifiers serve as proxies for the `$dim` argument to
+/// `linalg::IndexOp`, however the numerical value of a `LoopId` should
+/// not necessarily be equated with the numerical value of the corresponding
+/// `$dim` argument.  The `$dim` arguments are De Bruijn indices: that
+/// is, they identify the loop which binds the loop-variable by counting
+/// the enclosing loops from innermost to outermost, starting from zero.
+/// Whereas `LoopId` are considered to be arbitrary names for identifying
+/// loops; since the `Merger` does not care about the actual ordering of
+/// loops, and leaves it up to the `LoopEmitter` to specify the actual
+/// loop ordering (`LoopOrd`).
+///
+/// TODO: Despite the above claim that `$dim` and `LoopId` need not be
+/// numerically equal, some code in the `Merger` class does equate them
+/// (e.g., `buildTensorExp`).  So we need to explicate the exact relationship
+/// between `$dim`, `LoopId`, and `LoopOrd`; especially with regards to their
+/// providence.  If `LoopId` really is supposed to be equated with `$dim`,
+/// then we should change the name to `LoopIdx` or similar, to capture the
+/// fact that its numerical value is not invariant when entering/exiting
+/// loops (unlike `TensorId`, `ExprId`, `LatPointId`, and `LatSetId` which
+/// are invariant identifiers).
+using LoopId = unsigned;
+
+/// A compressed representation of `std::pair<TensorId, LoopId>`.
+/// The compression scheme is such that this also serves as an index
+/// into the bitvector stored in `LatPoint` (since that bitvector is
+/// just the implementation for a set of `TensorLoopId` values).
+using TensorLoopId = unsigned;
+
+/// `TensorExp` identifiers.  These are allocated by `Merger::addExp`,
+/// and serve as unique identifiers for the corresponding `TensorExp` object.
+using ExprId = unsigned;
+
+/// `LatPoint` identifiers.  These are allocated by `Merger::addLat`,
+/// and serve as unique identifiers for the corresponding `LatPoint` object.
+using LatPointId = unsigned;
+
+/// `LatSet` identifiers.  These are allocated by `Merger::addSet` (and
+/// by other methods calling that one), and serve as unique identifiers
+/// for the corresponding `SmallVector<LatPointId>` object.
+using LatSetId = unsigned;
+
+/// A constant serving as the canonically invalid identifier, regardless
+/// of the identifier type.
+static constexpr unsigned kInvalidId = -1u;
+
 /// Children subexpressions of tensor operations.
 struct Children {
-  unsigned e0;
-  unsigned e1;
+  ExprId e0;
+  ExprId e1;
 };
 
 /// Tensor expression. Represents a MLIR expression in tensor index notation.
 struct TensorExp {
-  TensorExp(Kind k, unsigned x, unsigned y, Value v, Operation *operation);
+  // The `x` parameter has different types depending on the value of the
+  // `k` parameter.  The correspondences are:
+  // * `kTensor`    -> `TensorId`
+  // * `kInvariant` -> `kInvalidId`
+  // * `kLoopVar`   -> `LoopId`
+  // * else         -> `ExprId`
+  //
+  // The `y`, `v`, and `op` parameters either must or must not be
+  // `kInvalidId`/`nullptr`, depending on the value of the `k` parameter;
+  // however, they have uniform C++ types regardless of the value of `k`.
+  TensorExp(Kind k, unsigned x, ExprId y, Value v, Operation *op);
 
   /// Tensor expression kind.
   Kind kind;
 
   union {
-    /// Expressions representing tensors simply have a tensor number.
-    unsigned tensor;
+    /// `kTensor` expressions simply have a tensor identifier.
+    TensorId tensor;
 
-    /// Indices hold the index number.
-    unsigned index;
+    /// `kLoopVar` expressions simply have a loop identifier.
+    LoopId loop;
 
-    /// Tensor operations hold the indices of their children.
+    /// All other expressions hold the `ExprId`s of their children.
     Children children;
   };
 
@@ -123,24 +207,29 @@ struct TensorExp {
   Operation *op;
 };
 
-/// Lattice point. Each lattice point consists of a conjunction of tensor
-/// loop indices (encoded in a bitvector) and the index of the corresponding
-/// tensor expression.
+/// Lattice point.  Each lattice point consists of a formal conjunction
+/// of `TensorLoopId`s, together with the identifier of the corresponding
+/// tensor expression.  The formal conjunction is represented as a set of
+/// `TensorLoopId`, where that set is implemented as a `BitVector`.
 struct LatPoint {
-  LatPoint(unsigned n, unsigned e, unsigned b);
-  LatPoint(const BitVector &b, unsigned e);
+  /// Construct the lattice point from a given set of `TensorLoopId`s.
+  LatPoint(const BitVector &bits, ExprId e);
+
+  /// Construct a lattice point with `(t,i)` as the only `TensorLoopId`,
+  /// where `(t,i) < (numTensors,numLoops)`.
+  LatPoint(unsigned numTensors, unsigned numLoops, TensorId t, LoopId i,
+           ExprId e);
 
-  /// Conjunction of tensor loop indices as bitvector. This represents
-  /// all indices involved in the tensor expression
+  /// Conjunction of all `TensorLoopId`s involved in the tensor expression.
   BitVector bits;
 
-  /// Simplified conjunction of tensor loop indices as bitvector. This
+  /// Simplified conjunction of `TensorLoopId` as bitvector.  This
   /// represents a simplified condition under which this tensor expression
   /// must execute. Pre-computed during codegen to avoid repeated eval.
   BitVector simple;
 
-  /// Index of the tensor expression.
-  unsigned exp;
+  /// Identifier of the tensor expression.
+  ExprId exp;
 };
 
 /// A class to handle all iteration lattice operations. This class abstracts
@@ -157,246 +246,274 @@ class Merger {
   ///
   /// In addition to natives loops (which are specified by the GenericOp),
   /// extra filter loops are needed in order to handle affine expressions on
-  /// sparse dimensions. E.g., (d0, d1, d2) => (d0 + d1, d2), a naive
+  /// sparse levels.  E.g., (d0, d1, d2) => (d0 + d1, d2), a naive
   /// implementation of the filter loop could be generated as
   ///
-  /// for (coord : sparse_dim[0])
-  ///   if (coord == d0 + d1) {
+  /// for (const auto c0 : coordinates[0]) {
+  ///   if (c0 == d0 + d1) {
   ///      generated_code;
   ///   }
   /// }
   ///
   /// to filter out coordinates that are not equal to the affine expression.
-  ///
-  /// TODO: we want to make the filter loop more efficient in the future, e.g.,
-  /// by avoiding scanning the full stored index sparse (keeping the last
-  /// position in ordered list) or even apply binary search to find the index.
-  ///
-  Merger(unsigned t, unsigned l, unsigned fl);
-
-  /// Adds a tensor expression. Returns its index.
-  unsigned addExp(Kind k, unsigned e0, unsigned e1 = -1u, Value v = Value(),
-                  Operation *op = nullptr);
-  unsigned addExp(Kind k, unsigned e, Value v, Operation *op = nullptr) {
-    return addExp(k, e, -1u, v, op);
+  //
+  // TODO: we want to make the filter loop more efficient in the future,
+  // e.g., by avoiding scanning the full list of stored coordinates (keeping
+  // the last position in ordered list) or even apply binary search to find
+  // the coordinate.
+  //
+  // TODO: would be cleaner to understand/document if the first argument
+  // gave the number of input tensors, instead of the current number of
+  // input+output tensors.
+  Merger(unsigned numInputOutputTensors, unsigned numNativeLoops,
+         unsigned numFilterLoops);
+
+  /// Constructs a new tensor expression, and returns its identifier.
+  /// The type of the `e0` argument varies according to the value of the
+  /// `k` argument, as described by the `TensorExp` ctor.
+  ExprId addExp(Kind k, unsigned e0, ExprId e1 = kInvalidId, Value v = Value(),
+                Operation *op = nullptr);
+  ExprId addExp(Kind k, ExprId e, Value v, Operation *op = nullptr) {
+    return addExp(k, e, kInvalidId, v, op);
   }
-  unsigned addExp(Kind k, Value v, Operation *op = nullptr) {
-    return addExp(k, -1u, -1u, v, op);
+  ExprId addExp(Kind k, Value v, Operation *op = nullptr) {
+    return addExp(k, kInvalidId, kInvalidId, v, op);
   }
 
-  /// Adds an iteration lattice point. Returns its index.
-  unsigned addLat(unsigned t, unsigned i, unsigned e);
+  /// Constructs a new iteration lattice point, and returns its identifier.
+  LatPointId addLat(TensorId t, LoopId i, ExprId e);
 
-  /// Adds a new, initially empty, set. Returns its index.
-  unsigned addSet();
+  /// Constructs a new (initially empty) set, and returns its identifier.
+  LatSetId addSet();
 
   /// Computes a single conjunction of two lattice points by taking the "union"
-  /// of loop indices (effectively constructing a larger "intersection" of those
-  /// indices) with a newly constructed tensor (sub)expression of given kind.
-  /// Returns the index of the new lattice point.
-  unsigned conjLatPoint(Kind kind, unsigned p0, unsigned p1,
-                        Operation *op = nullptr);
-
-  /// Conjunctive merge of two lattice sets L0 and L1 is conjunction of
-  /// cartesian product. Returns the index of the new set.
-  unsigned takeConj(Kind kind, unsigned s0, unsigned s1,
-                    Operation *op = nullptr);
-
-  /// Disjunctive merge of two lattice sets L0 and L1 is (L0 /\_op L1, L0, L1).
-  /// Returns the index of the new set.
-  unsigned takeDisj(Kind kind, unsigned s0, unsigned s1,
-                    Operation *op = nullptr);
-
-  /// Disjunctive merge of two lattice sets L0 and L1 with custom handling of
-  /// the overlap, left, and right regions. Any region may be left missing in
-  /// the output. Returns the index of the new set.
-  unsigned takeCombi(Kind kind, unsigned s0, unsigned s1, Operation *orig,
-                     bool includeLeft, Kind ltrans, Operation *opleft,
-                     bool includeRight, Kind rtrans, Operation *opright);
+  /// of `LoopId` (effectively constructing a larger "intersection" of those
+  /// loops) with a newly constructed tensor (sub)expression of given kind.
+  /// Returns the identifier of the new lattice point.
+  LatPointId conjLat(Kind kind, LatPointId p0, LatPointId p1,
+                     Operation *op = nullptr);
+
+  /// Conjunctive merge of two lattice sets: `(s0 /\_op s1)`.
+  /// Returns the identifier of the new set.
+  LatSetId conjSet(Kind kind, LatSetId s0, LatSetId s1,
+                   Operation *op = nullptr);
+
+  /// Disjunctive merge of two lattice sets: `(s0 /\_op s1, s0, s1)`.
+  /// Returns the identifier of the new set.
+  LatSetId disjSet(Kind kind, LatSetId s0, LatSetId s1,
+                   Operation *op = nullptr);
+
+  /// Disjunctive merge of two lattice sets with custom handling of the
+  /// overlap, left, and right regions.  Any region may be left missing
+  /// in the output.  Returns the identifier of the new set.
+  LatSetId combiSet(Kind kind, LatSetId s0, LatSetId s1, Operation *orig,
+                    bool includeLeft, Kind ltrans, Operation *opleft,
+                    bool includeRight, Kind rtrans, Operation *opright);
 
   /// Maps the unary operator over the lattice set of the operand, i.e. each
   /// lattice point on an expression E is simply copied over, but with OP E
-  /// as new expression. Returns the index of the new set.
-  unsigned mapSet(Kind kind, unsigned s0, Value v = Value(),
+  /// as new expression. Returns the identifier of the new set.
+  LatSetId mapSet(Kind kind, LatSetId s, Value v = Value(),
                   Operation *op = nullptr);
 
   /// Optimizes the iteration lattice points in the given set. This
   /// method should be called right before code generation to avoid
   /// generating redundant loops and conditions.
-  unsigned optimizeSet(unsigned s0);
+  LatSetId optimizeSet(LatSetId s);
 
   /// Simplifies the conditions in a conjunction of a given lattice point
   /// within the given set using just two basic rules:
   /// (1) multiple dense conditions are reduced to single dense, and
   /// (2) a *singleton* sparse/dense is reduced to sparse/random access.
-  BitVector simplifyCond(unsigned s0, unsigned p0);
-
-  /// Returns true if Li > Lj.
-  bool latGT(unsigned i, unsigned j) const;
-
-  /// Returns true if Li and Lj only differ in dense.
-  bool onlyDenseDiff(unsigned i, unsigned j);
-
-  /// Bit translation (get tensor ID).
-  unsigned tensor(unsigned b) const { return b % numTensors; }
-  /// Bit translation (get loop index).
-  unsigned index(unsigned b) const { return b / numTensors; }
-
-  /// Get the number of total loops (native loops + filter loops).
-  unsigned getNumLoops() const { return numLoops; }
-  /// Get the number of native loops.
-  unsigned getNumNativeLoops() const { return numNativeLoops; }
-  /// Get the number of filter loops.
-  unsigned getNumFilterLoops() const { return numLoops - numNativeLoops; }
-  /// Get the starting filter loop index.
-  unsigned getFilterLoopStartingIdx() const { return getNumNativeLoops(); }
-
-  /// Returns true if bit corresponds to index of output tensor.
-  bool isOutTensor(unsigned b, unsigned i) const {
-    return tensor(b) == outTensor && index(b) == i;
+  BitVector simplifyCond(LatSetId s, LatPointId p);
+
+  /// Returns true if p0 > p1.
+  bool latGT(LatPointId p0, LatPointId p1) const;
+
+  /// Returns true if p0 and p1 only differ in dense.
+  bool onlyDenseDiff(LatPointId p0, LatPointId p1) const;
+
+  /// Gets the tensor-identifier of the `TensorLoopId`.
+  TensorId tensor(TensorLoopId b) const { return b % numTensors; }
+  /// Gets the loop-identifier of the `TensorLoopId`.
+  LoopId loop(TensorLoopId b) const { return b / numTensors; }
+
+  /// Get the total number of tensors (including the output-tensor and
+  /// synthetic-tensor).  The result is given the type `TensorId` since
+  /// the result is primarily used as an upper bound for `TensorId`s.
+  TensorId getNumTensors() const { return numTensors; }
+
+  /// Get the total number of loops (native loops + filter loops).
+  /// The result is given the type `LoopId` since the result will
+  /// generally be used as a for-loop upper bound.
+  LoopId getNumLoops() const { return numLoops; }
+  /// Get the number of native loops.  The result is given the type
+  /// `LoopId` since the result will generally be used as a for-loop
+  /// upper bound.
+  LoopId getNumNativeLoops() const { return numNativeLoops; }
+  /// Get the number of filter loops.  The result is given the type
+  /// `LoopId` since the result will generally be used as a for-loop
+  /// upper bound.
+  LoopId getNumFilterLoops() const { return numLoops - numNativeLoops; }
+  /// Get the identifier of the first filter-loop.
+  LoopId getStartingFilterLoopId() const { return getNumNativeLoops(); }
+
+  /// Returns true if `b` is the `i`th loop of the output tensor.
+  bool isOutTensor(TensorLoopId b, LoopId i) const {
+    assert(i < numLoops);
+    return b == numTensors * i + outTensor;
   }
 
-  /// Gets tensor ID for the output tensor.
-  unsigned getOutTensorID() const { return outTensor; }
-  /// Gets tensor ID for the synthetic tensor (used for all invariant tensor
-  /// expressions).
-  unsigned getSynTensorID() const { return syntheticTensor; }
+  /// Get the output tensor's identifier.
+  TensorId getOutTensorID() const { return outTensor; }
+  /// Get the synthetic tensor's identifier (used for all invariant
+  /// tensor expressions).
+  TensorId getSynTensorID() const { return syntheticTensor; }
+
+  bool isFilterLoop(LoopId i) const {
+    assert(i < numLoops);
+    return i >= numNativeLoops;
+  }
 
-  bool isFilterLoop(unsigned ldx) const {
-    assert(ldx < numLoops);
-    return ldx >= numNativeLoops;
+  /// Returns true if the expression is `(kTensor t)`.
+  bool expIsTensor(ExprId e, TensorId t) const {
+    return tensorExps[e].kind == kTensor && tensorExps[e].tensor == t;
   }
 
-  /// Returns true if the expression contains the `t` as an operand.
-  bool expContainsTensor(unsigned e, unsigned t) const;
+  /// Returns true if the expression contains the tensor as an operand.
+  bool expContainsTensor(ExprId e, TensorId t) const;
 
   /// Returns true if the expression contains a negation on output tensor.
   /// I.e., `- outTensor` or `exp - outputTensor`
   /// NOTE: this is an trivial tests in that it does not handle recursive
   /// negation, i.e., it returns true when the expression is `-(-tensor)`.
-  bool hasNegateOnOut(unsigned e) const;
+  bool hasNegateOnOut(ExprId e) const;
 
   /// Returns true if given tensor iterates *only* in the given tensor
   /// expression. For the output tensor, this defines a "simply dynamic"
   /// operation [Bik96]. For instance: a(i) *= 2.0 or a(i) += a(i) for
   /// sparse vector a.
-  bool isSingleCondition(unsigned t, unsigned e) const;
+  bool isSingleCondition(TensorId t, ExprId e) const;
 
-  /// Returns true if any set bit corresponds to sparse dimension level type.
+  /// Returns true if any `TensorLoopId` in the bitvector corresponds
+  /// to sparse level-type.
   bool hasAnySparse(const BitVector &bits) const;
 
-  /// Gets the dimension level type of the `t`th tensor on `i`th loop.
-  DimLevelType getDimLevelType(unsigned t, unsigned i) const {
+  /// Gets the level-type of the `t`th tensor on `i`th loop.
+  DimLevelType getDimLevelType(TensorId t, LoopId i) const {
     assert(t < numTensors && i < numLoops);
-    return dimTypes[t][i];
+    return lvlTypes[t][i];
   }
-
-  /// Gets the dimension level type of `b`.
-  DimLevelType getDimLevelType(unsigned b) const {
-    return getDimLevelType(tensor(b), index(b));
+  DimLevelType getDimLevelType(TensorLoopId b) const {
+    return getDimLevelType(tensor(b), loop(b));
   }
 
-  std::optional<unsigned> getLoopIdx(unsigned t, unsigned dim) const {
-    assert(t < numTensors && dim < numLoops);
-    return dimToLoopIdx[t][dim];
+  /// Gets the loop identifier for the `lvl`th level of the `t`th tensor.
+  std::optional<LoopId> getLoopId(TensorId t, Level lvl) const {
+    assert(t < numTensors && lvl < lvlToLoop[t].size());
+    return lvlToLoop[t][lvl];
   }
 
-  /// Gets the dimension number of the the `t`th tensor on `i`th loop.
-  std::optional<unsigned> getDimNum(unsigned t, unsigned i) const {
+  /// Gets the level number of the the `t`th tensor on `i`th loop.
+  std::optional<Level> getLvl(TensorId t, LoopId i) const {
     assert(t < numTensors && i < numLoops);
-    return loopIdxToDim[t][i];
+    return loopToLvl[t][i];
   }
-
-  /// Gets the dimension number of `b`.
-  std::optional<unsigned> getDimNum(unsigned b) const {
-    return getDimNum(tensor(b), index(b));
+  std::optional<Level> getLvl(TensorLoopId b) const {
+    return getLvl(tensor(b), loop(b));
   }
 
-  /// Sets the dimension and dimension level type of the `t`th tensor on `i`th
-  /// loop.
-  void setDimAndDimLevelType(unsigned t, unsigned i, unsigned dim,
-                             DimLevelType dlt) {
-    assert(isValidDLT(dlt));
-    dimTypes[t][i] = dlt;
-    loopIdxToDim[t][i] = dim;
-    assert(dim < numLoops);
-    dimToLoopIdx[t][dim] = i;
+  /// Sets the level number and level-type of the `t`th tensor on
+  /// `i`th loop.
+  void setLevelAndType(TensorId t, LoopId i, Level lvl, DimLevelType dlt) {
+    assert(t < numTensors && i < numLoops && lvl < lvlToLoop[t].size() &&
+           isValidDLT(dlt));
+    lvlTypes[t][i] = dlt;
+    loopToLvl[t][i] = lvl;
+    lvlToLoop[t][lvl] = i;
   }
 
-  // Iterates the bits of a lattice, for each set bit, converts it into the
-  // corresponding tensor dimension and invokes the callback.
-  void foreachTidDimPairInBits(
-      const BitVector &bits,
-      function_ref<void(unsigned b, unsigned tid, std::optional<unsigned> dim,
-                        DimLevelType dlt)>
-          cb) {
-    for (unsigned b : bits.set_bits())
-      cb(b, tensor(b), getDimNum(b), getDimLevelType(b));
+  /// Iterates over a set of `TensorLoopId`s, invoking the callback
+  /// for each `TensorLoopId` and passing it the corresponding tensor
+  /// identifier, level, and level-type.
+  void
+  foreachTensorLoopId(LatPointId p,
+                      function_ref<void(TensorLoopId, TensorId,
+                                        std::optional<Level>, DimLevelType)>
+                          callback) const {
+    for (const TensorLoopId b : latPoints[p].bits.set_bits())
+      callback(b, tensor(b), getLvl(b), getDimLevelType(b));
   }
 
-  // Has sparse output tensor setter.
+  /// Sets whether the output tensor is sparse or not.
   void setHasSparseOut(bool s) { hasSparseOut = s; }
 
   /// Convenience getters to immediately access the stored nodes.
   /// Typically it is inadvisible to keep the reference around, as in
-  /// "TensorExpr &te = merger.exp(e))", since insertions into the merger
+  /// `TensorExpr &te = merger.exp(e)`, since insertions into the merger
   /// may cause data movement and invalidate the underlying memory address.
-  TensorExp &exp(unsigned e) { return tensorExps[e]; }
-  LatPoint &lat(unsigned l) { return latPoints[l]; }
-  SmallVector<unsigned> &set(unsigned s) { return latSets[s]; }
+  TensorExp &exp(ExprId e) { return tensorExps[e]; }
+  LatPoint &lat(LatPointId p) { return latPoints[p]; }
+  SmallVector<LatPointId> &set(LatSetId s) { return latSets[s]; }
 
 #ifndef NDEBUG
   /// Print methods (for debugging).
-  void dumpExp(unsigned e) const;
-  void dumpLat(unsigned p) const;
-  void dumpSet(unsigned s) const;
+  void dumpExp(ExprId e) const;
+  void dumpLat(LatPointId p) const;
+  void dumpSet(LatSetId s) const;
   void dumpBits(const BitVector &bits) const;
 #endif
 
   /// Builds the iteration lattices in a bottom-up traversal given the
-  /// remaining tensor (sub)expression and the next loop index in the
-  /// iteration graph. Returns index of the root expression.
-  unsigned buildLattices(unsigned e, unsigned i);
+  /// remaining tensor (sub)expression and the next loop in the iteration
+  /// graph.  Returns the identifier of the root set.
+  LatSetId buildLattices(ExprId e, LoopId i);
 
   /// Builds a tensor expression from the given Linalg operation.
-  /// Returns index of the root expression on success.
-  std::optional<unsigned> buildTensorExpFromLinalg(linalg::GenericOp op);
+  /// On success, returns the identifier of the root expression.
+  std::optional<ExprId> buildTensorExpFromLinalg(linalg::GenericOp op);
 
   /// Rebuilds SSA format from a tensor expression.
-  Value buildExp(RewriterBase &rewriter, Location loc, unsigned e, Value v0,
-                 Value v1);
+  Value buildExp(RewriterBase &rewriter, Location loc, ExprId e, Value v0,
+                 Value v1) const;
 
 private:
   /// Private helpers.
-  bool maybeZero(unsigned e) const;
-  bool isInvariant(unsigned e) const;
-  Type inferType(unsigned e, Value src);
+  bool maybeZero(ExprId e) const;
+  bool isInvariant(ExprId e) const;
+  Type inferType(ExprId e, Value src) const;
 
   /// Traverses the SSA tree (possibly a DAG) to build a tensor expression.
-  std::optional<unsigned> buildTensorExp(linalg::GenericOp op, Value v);
+  std::optional<ExprId> buildTensorExp(linalg::GenericOp op, Value v);
 
   /// Merger data structures.
-  const unsigned outTensor;
-  const unsigned syntheticTensor;
+  const TensorId outTensor;
+  const TensorId syntheticTensor;
   const unsigned numTensors;
   const unsigned numNativeLoops;
   const unsigned numLoops;
   bool hasSparseOut;
 
-  // Map that converts pair<tensor id, loop id> to the corresponding dimension
-  // level type.
-  std::vector<std::vector<DimLevelType>> dimTypes;
+  // Below we use `std::vector` for things which have a priori fixed
+  // sizes, whereas we use `llvm::SmallVector` for things with variable
+  // size.  Do beware that these two classes differ in the semantics of
+  // `operator[]`: `SmallVector` performs OOB checks, whereas `std::vector`
+  // does not.
+
+  // Map that converts pair<TensorId, LoopId> to the corresponding
+  // level-type.
+  std::vector<std::vector<DimLevelType>> lvlTypes;
 
-  // Map that converts pair<tensor id, loop id> to the corresponding
-  // dimension.
-  std::vector<std::vector<std::optional<unsigned>>> loopIdxToDim;
+  // Map that converts pair<TensorId, LoopId> to the corresponding
+  // level.
+  std::vector<std::vector<std::optional<Level>>> loopToLvl;
 
-  // Map that converts pair<tensor id, dim> to the corresponding loop id.
-  std::vector<std::vector<std::optional<unsigned>>> dimToLoopIdx;
+  // Map that converts pair<TensorId, Level> to the corresponding LoopId.
+  std::vector<std::vector<std::optional<LoopId>>> lvlToLoop;
 
   llvm::SmallVector<TensorExp> tensorExps;
   llvm::SmallVector<LatPoint> latPoints;
-  llvm::SmallVector<SmallVector<unsigned>> latSets;
+  llvm::SmallVector<SmallVector<LatPointId>> latSets;
 };
 
 } // namespace sparse_tensor
diff --git a/mlir/include/mlir/Dialect/Tensor/IR/TensorOps.td b/mlir/include/mlir/Dialect/Tensor/IR/TensorOps.td
index 80c0ba5e754a9..09b7775dcaae4 100644
--- a/mlir/include/mlir/Dialect/Tensor/IR/TensorOps.td
+++ b/mlir/include/mlir/Dialect/Tensor/IR/TensorOps.td
@@ -1788,6 +1788,13 @@ def Tensor_PackOp : Tensor_RelayoutOp<"pack", [
         ArrayRef<int64_t> innerTileSizes, ArrayRef<int64_t> innerDimsPos,
         ArrayRef<int64_t> outerDimsPerm = {});
 
+    // Returns true if we have enough static information to catch undefined
+    // behavior when the tile size does not divide perfectly the dimension of
+    // the input tensor.
+    static bool requirePaddingValue(ArrayRef<int64_t> inputShape,
+                                    ArrayRef<int64_t> innerDimsPos,
+                                    ArrayRef<OpFoldResult> innerTiles);
+
     static Value createDestinationTensor(OpBuilder &b, Location loc,
         Value source, ArrayRef<OpFoldResult> innerTileSizes,
         ArrayRef<int64_t> innerDimsPos, ArrayRef<int64_t> outerDimsPerm);
diff --git a/mlir/include/mlir/Dialect/Tensor/IR/TensorTilingInterfaceImpl.h b/mlir/include/mlir/Dialect/Tensor/IR/TensorTilingInterfaceImpl.h
index 30a5026cd68b3..7228a5a297ad8 100644
--- a/mlir/include/mlir/Dialect/Tensor/IR/TensorTilingInterfaceImpl.h
+++ b/mlir/include/mlir/Dialect/Tensor/IR/TensorTilingInterfaceImpl.h
@@ -16,6 +16,9 @@
 #include "mlir/IR/Dialect.h"
 
 namespace mlir {
+
+struct TilingResult;
+
 namespace tensor {
 
 class PadOp;
@@ -39,10 +42,10 @@ class PadOp;
 /// to guard against the case that we might take a zero-sized slice from the
 /// original source. For such cases, we `tensor.generate` to generate the
 /// full tensor.
-Operation *bubbleUpPadSlice(OpBuilder &b, tensor::PadOp padOp,
-                            ArrayRef<OpFoldResult> offsets,
-                            ArrayRef<OpFoldResult> sizes,
-                            bool generateZeroSliceGuard = true);
+FailureOr<TilingResult> bubbleUpPadSlice(OpBuilder &b, tensor::PadOp padOp,
+                                         ArrayRef<OpFoldResult> offsets,
+                                         ArrayRef<OpFoldResult> sizes,
+                                         bool generateZeroSliceGuard = true);
 
 /// Registers external models for Tiling interface for tensor ops.
 /// Currently, it registers:
diff --git a/mlir/include/mlir/Dialect/Tensor/Transforms/Transforms.h b/mlir/include/mlir/Dialect/Tensor/Transforms/Transforms.h
index 01985c943527c..4cdf360c51d72 100644
--- a/mlir/include/mlir/Dialect/Tensor/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/Tensor/Transforms/Transforms.h
@@ -13,6 +13,9 @@
 #include "mlir/IR/PatternMatch.h"
 
 namespace mlir {
+
+struct TilingResult;
+
 namespace tensor {
 
 /// Populates `patterns` with patterns to wrap a tensor.pad op with an scf.if op
@@ -26,7 +29,7 @@ void populateSplitPaddingPatterns(RewritePatternSet &patterns,
 /// provide a mechanism to control where the application happens. With use of
 /// transform dialect that control is done within the transform dialect. Other
 /// use cases can inherit from this pattern and add necessary controls.
-FailureOr<Value> replaceExtractSliceWithTiledProducer(
+FailureOr<TilingResult> replaceExtractSliceWithTiledProducer(
     OpBuilder &builder, tensor::ExtractSliceOp sliceOp, OpResult producerOp);
 
 /// Collects patterns to merge consecutive tensor.insert_slice/extract_slice
diff --git a/mlir/include/mlir/Dialect/Tensor/Utils/Utils.h b/mlir/include/mlir/Dialect/Tensor/Utils/Utils.h
index c1460c9c79d63..c0f33d15cb518 100644
--- a/mlir/include/mlir/Dialect/Tensor/Utils/Utils.h
+++ b/mlir/include/mlir/Dialect/Tensor/Utils/Utils.h
@@ -26,6 +26,11 @@ PadOp createPadHighOp(RankedTensorType type, Value source, Value pad,
 SmallVector<Value> createDynamicDimValues(OpBuilder &b, Location loc,
                                           Value rankedTensor);
 
+// Returns the tensor extent along dimension `dim` if `rankedTensor` is of
+// `RankedTensorType`. Returns `failure()` otherwise.
+FailureOr<OpFoldResult> createDimValue(OpBuilder &b, Location loc,
+                                       Value rankedTensor, int64_t dim);
+
 // Creates dim ops or constant ops for each dimension of the ranked tensor
 // argument and returns these as values.
 SmallVector<OpFoldResult> createDimValues(OpBuilder &b, Location loc,
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index fc4cf7d82174c..be5720caeb0de 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -183,6 +183,36 @@ def Tosa_DepthwiseConv2DOp : Tosa_Op<"depthwise_conv2d", [
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// Operator: fft2d
+//===----------------------------------------------------------------------===//
+def Tosa_FFT2dOp : Tosa_Op<"fft2d", [
+    DeclareOpInterfaceMethods<InferShapedTypeOpInterface,
+                              ["inferReturnTypeComponents"]>,
+    Pure]> {
+  let summary = "Performs FFT2D operation on the input.";
+
+  let description = [{
+    Performs a batched complex 2D Fast Fourier Transform over the input. The
+    complex input values are constructed from the corresponding values in the
+    input_real and input_imag tensors. The resulting values in the output are
+    split into the output_real and output_imag tensors. No normalization is
+    applied on either the forward or inverse versions of the operation.
+  }];
+
+  let arguments = (ins
+    Tosa_Tensor3D:$input_real,
+    Tosa_Tensor3D:$input_imag,
+
+    BoolAttr:$inverse
+  );
+
+  let results = (outs
+    Tosa_Tensor3D:$output_real,
+    Tosa_Tensor3D:$output_imag
+  );
+}
+
 //===----------------------------------------------------------------------===//
 // Operator: fully_connected
 //===----------------------------------------------------------------------===//
diff --git a/mlir/include/mlir/Dialect/Transform/IR/TransformUtils.h b/mlir/include/mlir/Dialect/Transform/IR/TransformUtils.h
deleted file mode 100644
index ca3da2b682e70..0000000000000
--- a/mlir/include/mlir/Dialect/Transform/IR/TransformUtils.h
+++ /dev/null
@@ -1,28 +0,0 @@
-//===- TransformUtils.h - Transform Dialect Utils ---------------*- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef MLIR_DIALECT_TRANSFORM_IR_TRANSFORMUTILS_H
-#define MLIR_DIALECT_TRANSFORM_IR_TRANSFORMUTILS_H
-
-#include "mlir/IR/PatternMatch.h"
-
-namespace mlir {
-namespace transform {
-
-/// A simple pattern rewriter that can be constructed from a context. This is
-/// necessary to apply patterns to a specific op locally.
-class TrivialPatternRewriter : public PatternRewriter {
-public:
-  explicit TrivialPatternRewriter(MLIRContext *context)
-      : PatternRewriter(context) {}
-};
-
-} // namespace transform
-} // namespace mlir
-
-#endif // MLIR_DIALECT_TRANSFORM_IR_TRANSFORMUTILS_H
diff --git a/mlir/include/mlir/Dialect/Transform/Utils/RaggedArray.h b/mlir/include/mlir/Dialect/Transform/Utils/RaggedArray.h
index c3c38d61bef3e..0ee23914fa4e1 100644
--- a/mlir/include/mlir/Dialect/Transform/Utils/RaggedArray.h
+++ b/mlir/include/mlir/Dialect/Transform/Utils/RaggedArray.h
@@ -9,6 +9,7 @@
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallVector.h"
+#include <iterator>
 
 namespace mlir {
 /// A 2D array where each row may have different length. Elements of each row
@@ -25,15 +26,92 @@ class RaggedArray {
 
   /// Accesses `pos`-th row.
   ArrayRef<T> operator[](size_t pos) const { return at(pos); }
-  ArrayRef<T> at(size_t pos) const { return slices[pos]; }
+  ArrayRef<T> at(size_t pos) const {
+    if (slices[pos].first == static_cast<size_t>(-1))
+      return ArrayRef<T>();
+    return ArrayRef<T>(storage).slice(slices[pos].first, slices[pos].second);
+  }
   MutableArrayRef<T> operator[](size_t pos) { return at(pos); }
-  MutableArrayRef<T> at(size_t pos) { return slices[pos]; }
+  MutableArrayRef<T> at(size_t pos) {
+    if (slices[pos].first == static_cast<size_t>(-1))
+      return MutableArrayRef<T>();
+    return MutableArrayRef<T>(storage).slice(slices[pos].first,
+                                             slices[pos].second);
+  }
+
+  /// Iterator over the rows.
+  class iterator
+      : public llvm::iterator_facade_base<
+            iterator, std::forward_iterator_tag, MutableArrayRef<T>,
+            std::ptrdiff_t, MutableArrayRef<T> *, MutableArrayRef<T>> {
+  public:
+    /// Creates the start iterator.
+    explicit iterator(RaggedArray &ragged) : ragged(ragged), pos(0) {}
+
+    /// Creates the end iterator.
+    iterator(RaggedArray &ragged, size_t pos) : ragged(ragged), pos(pos) {}
+
+    /// Dereferences the current iterator. Assumes in-bounds.
+    MutableArrayRef<T> operator*() const { return ragged[pos]; }
+
+    /// Increments the iterator.
+    iterator &operator++() {
+      if (pos < ragged.slices.size())
+        ++pos;
+      return *this;
+    }
+
+    /// Compares the two iterators. Iterators into different ragged arrays
+    /// compare not equal.
+    bool operator==(const iterator &other) const {
+      return &ragged == &other.ragged && pos == other.pos;
+    }
+
+  private:
+    RaggedArray &ragged;
+    size_t pos;
+  };
+
+  /// Constant iterator over the rows.
+  class const_iterator
+      : public llvm::iterator_facade_base<
+            const_iterator, std::forward_iterator_tag, ArrayRef<T>,
+            std::ptrdiff_t, ArrayRef<T> *, ArrayRef<T>> {
+  public:
+    /// Creates the start iterator.
+    explicit const_iterator(const RaggedArray &ragged)
+        : ragged(ragged), pos(0) {}
+
+    /// Creates the end iterator.
+    const_iterator(const RaggedArray &ragged, size_t pos)
+        : ragged(ragged), pos(pos) {}
+
+    /// Dereferences the current iterator. Assumes in-bounds.
+    ArrayRef<T> operator*() const { return ragged[pos]; }
+
+    /// Increments the iterator.
+    const_iterator &operator++() {
+      if (pos < ragged.slices.size())
+        ++pos;
+      return *this;
+    }
+
+    /// Compares the two iterators. Iterators into different ragged arrays
+    /// compare not equal.
+    bool operator==(const const_iterator &other) const {
+      return &ragged == &other.ragged && pos == other.pos;
+    }
+
+  private:
+    const RaggedArray &ragged;
+    size_t pos;
+  };
 
   /// Iterator over rows.
-  auto begin() { return slices.begin(); }
-  auto begin() const { return slices.begin(); }
-  auto end() { return slices.end(); }
-  auto end() const { return slices.end(); }
+  const_iterator begin() const { return const_iterator(*this); }
+  const_iterator end() const { return const_iterator(*this, slices.size()); }
+  iterator begin() { return iterator(*this); }
+  iterator end() { return iterator(*this, slices.size()); }
 
   /// Reserve space to store `size` rows with `nestedSize` elements each.
   void reserve(size_t size, size_t nestedSize = 0) {
@@ -53,38 +131,41 @@ class RaggedArray {
   /// succeeding rows.
   template <typename Range>
   void replace(size_t pos, Range &&elements) {
-    auto from = slices[pos].data();
-    if (from != nullptr) {
-      auto to = std::next(from, slices[pos].size());
+    if (slices[pos].first != static_cast<size_t>(-1)) {
+      auto from = std::next(storage.begin(), slices[pos].first);
+      auto to = std::next(from, slices[pos].second);
       auto newFrom = storage.erase(from, to);
       // Update the array refs after the underlying storage was shifted.
       for (size_t i = pos + 1, e = size(); i < e; ++i) {
-        slices[i] = MutableArrayRef<T>(newFrom, slices[i].size());
-        std::advance(newFrom, slices[i].size());
+        slices[i] = std::make_pair(std::distance(storage.begin(), newFrom),
+                                   slices[i].second);
+        std::advance(newFrom, slices[i].second);
       }
     }
     slices[pos] = appendToStorage(std::forward<Range>(elements));
   }
 
   /// Appends `num` empty rows to the array.
-  void appendEmptyRows(size_t num) { slices.resize(slices.size() + num); }
+  void appendEmptyRows(size_t num) {
+    slices.resize(slices.size() + num, std::pair<size_t, size_t>(-1, 0));
+  }
 
 private:
-  /// Appends the given elements to the storage and returns an ArrayRef pointing
-  /// to them in the storage.
+  /// Appends the given elements to the storage and returns an ArrayRef
+  /// pointing to them in the storage.
   template <typename Range>
-  MutableArrayRef<T> appendToStorage(Range &&elements) {
+  std::pair<size_t, size_t> appendToStorage(Range &&elements) {
     size_t start = storage.size();
     llvm::append_range(storage, std::forward<Range>(elements));
-    return MutableArrayRef<T>(storage).drop_front(start);
+    return std::make_pair(start, storage.size() - start);
   }
 
-  /// Outer elements of the ragged array. Each entry is a reference to a
-  /// contiguous segment in the `storage` list that contains the actual
-  /// elements. This allows for elements to be stored contiguously without
-  /// nested vectors and for different segments to be set or replaced in any
-  /// order.
-  SmallVector<MutableArrayRef<T>> slices;
+  /// Outer elements of the ragged array. Each entry is an (offset, length)
+  /// pair identifying a contiguous segment in the `storage` list that
+  /// contains the actual elements. This allows for elements to be stored
+  /// contiguously without nested vectors and for different segments to be set
+  /// or replaced in any order.
+  SmallVector<std::pair<size_t, size_t>> slices;
 
   /// Dense storage for ragged array elements.
   SmallVector<T> storage;
diff --git a/mlir/include/mlir/Dialect/Utils/IndexingUtils.h b/mlir/include/mlir/Dialect/Utils/IndexingUtils.h
index 64eea7eded92b..1969bd3a33121 100644
--- a/mlir/include/mlir/Dialect/Utils/IndexingUtils.h
+++ b/mlir/include/mlir/Dialect/Utils/IndexingUtils.h
@@ -23,35 +23,68 @@
 namespace mlir {
 class ArrayAttr;
 
-/// Computes and returns the linearized index of 'offsets' w.r.t. 'basis'.
-int64_t linearize(ArrayRef<int64_t> offsets, ArrayRef<int64_t> basis);
-
-/// Given the strides together with a linear index in the dimension
-/// space, returns the vector-space offsets in each dimension for a
-/// de-linearized index.
-SmallVector<int64_t> delinearize(ArrayRef<int64_t> strides,
-                                 int64_t linearIndex);
+//===----------------------------------------------------------------------===//
+// Utils that operate on static integer values.
+//===----------------------------------------------------------------------===//
 
-/// Given a set of sizes, compute and return the strides (i.e. the number of
-/// linear incides to skip along the (k-1) most minor dimensions to get the next
-/// k-slice). This is also the basis that one can use to linearize an n-D offset
-/// confined to `[0 .. sizes]`.
-SmallVector<int64_t> computeStrides(ArrayRef<int64_t> sizes);
+/// Given a set of sizes, return the suffix product.
+///
+/// When applied to slicing, this is the calculation needed to derive the
+/// strides (i.e. the number of linear indices to skip along the (k-1) most
+/// minor dimensions to get the next k-slice).
+///
+/// This is the basis to linearize an n-D offset confined to `[0 ... sizes]`.
+///
+/// Assuming `sizes` is `[s0, .. sn]`, return the vector<int64_t>
+///   `[s1 * ... * sn, s2 * ... * sn, ..., sn, 1]`.
+///
+/// `sizes` elements are asserted to be non-negative.
+///
+/// Return an empty vector if `sizes` is empty.
+SmallVector<int64_t> computeSuffixProduct(ArrayRef<int64_t> sizes);
+inline SmallVector<int64_t> computeStrides(ArrayRef<int64_t> sizes) {
+  return computeSuffixProduct(sizes);
+}
 
-/// Return a vector containing llvm::zip of v1 and v2 multiplied elementwise.
+/// Return a vector containing llvm::zip_equal(v1, v2) multiplied elementwise.
+///
+/// Return an empty vector if `v1` and `v2` are empty.
 SmallVector<int64_t> computeElementwiseMul(ArrayRef<int64_t> v1,
                                            ArrayRef<int64_t> v2);
 
-/// Compute and return the multi-dimensional integral ratio of `subShape` to
-/// the trailing dimensions of `shape`. This represents how many times
-/// `subShape` fits within `shape`.
-/// If integral division is not possible, return std::nullopt.
+/// Return the number of elements of basis (i.e. the max linear index).
+/// Return `0` if `basis` is empty.
+///
+/// `basis` elements are asserted to be non-negative.
+///
+/// Return `0` if `basis` is empty.
+int64_t computeMaxLinearIndex(ArrayRef<int64_t> basis);
+
+/// Return the linearized index of 'offsets' w.r.t. 'basis'.
+///
+/// `basis` elements are asserted to be non-negative.
+int64_t linearize(ArrayRef<int64_t> offsets, ArrayRef<int64_t> basis);
+
+/// Given the strides together with a linear index in the dimension space,
+/// return the vector-space offsets in each dimension for a de-linearized index.
+/// `strides` elements are asserted to be non-negative.
+///
+/// Let `li = linearIndex`, assuming `strides` are `[s0, .. sn]`, return the
+/// vector of int64_t
+///   `[li % s0, (li / s0) % s1, ..., (li / s0 / .. / sn-1) % sn]`
+SmallVector<int64_t> delinearize(int64_t linearIndex,
+                                 ArrayRef<int64_t> strides);
+
+/// Return the multi-dimensional integral ratio of `subShape` to the trailing
+/// dimensions of `shape`. This represents how many times `subShape` fits
+/// within `shape`. If integral division is not possible, return std::nullopt.
 /// The trailing `subShape.size()` entries of both shapes are assumed (and
-/// enforced) to only contain noonnegative values.
+/// enforced) to only contain non-negative values.
 ///
 /// Examples:
 ///   - shapeRatio({3, 5, 8}, {2, 5, 2}) returns {3, 2, 1}.
-///   - shapeRatio({3, 8}, {2, 5, 2}) returns std::nullopt (subshape has higher
+///   - shapeRatio({3, 8}, {2, 5, 2}) returns std::nullopt (subshape has
+///   higher
 ///     rank).
 ///   - shapeRatio({42, 2, 10, 32}, {2, 5, 2}) returns {42, 1, 2, 16} which is
 ///     derived as {42(leading shape dim), 2/2, 10/5, 32/2}.
@@ -60,14 +93,96 @@ SmallVector<int64_t> computeElementwiseMul(ArrayRef<int64_t> v1,
 std::optional<SmallVector<int64_t>>
 computeShapeRatio(ArrayRef<int64_t> shape, ArrayRef<int64_t> subShape);
 
+//===----------------------------------------------------------------------===//
+// Utils that operate on AffineExpr.
+//===----------------------------------------------------------------------===//
+
+/// Given a set of sizes, return the suffix product.
+///
+/// When applied to slicing, this is the calculation needed to derive the
+/// strides (i.e. the number of linear indices to skip along the (k-1) most
+/// minor dimensions to get the next k-slice).
+///
+/// This is the basis to linearize an n-D offset confined to `[0 ... sizes]`.
+///
+/// Assuming `sizes` is `[s0, .. sn]`, return the vector<AffineExpr>
+///   `[s1 * ... * sn, s2 * ... * sn, ..., sn, 1]`.
+///
+/// It is the caller's responsibility to pass proper AffineExpr kind that
+/// result in valid AffineExpr (i.e. cannot multiply 2 AffineDimExpr or divide
+/// by an AffineDimExpr).
+///
+/// `sizes` elements are expected to bind to non-negative values.
+///
+/// Return an empty vector if `sizes` is empty.
+SmallVector<AffineExpr> computeSuffixProduct(ArrayRef<AffineExpr> sizes);
+inline SmallVector<AffineExpr> computeStrides(ArrayRef<AffineExpr> sizes) {
+  return computeSuffixProduct(sizes);
+}
+
+/// Return a vector containing llvm::zip_equal(v1, v2) multiplied elementwise.
+///
+/// It is the caller's responsibility to pass proper AffineExpr kind that
+/// result in valid AffineExpr (i.e. cannot multiply 2 AffineDimExpr or divide
+/// by an AffineDimExpr).
+///
+/// Return an empty vector if `v1` and `v2` are empty.
+SmallVector<AffineExpr> computeElementwiseMul(ArrayRef<AffineExpr> v1,
+                                              ArrayRef<AffineExpr> v2);
+
 /// Return the number of elements of basis (i.e. the max linear index).
 /// Return `0` if `basis` is empty.
-int64_t computeMaxLinearIndex(ArrayRef<int64_t> basis);
+///
+/// It is the caller's responsibility to pass proper AffineExpr kind that
+/// result in valid AffineExpr (i.e. cannot multiply 2 AffineDimExpr or divide
+/// by an AffineDimExpr).
+///
+/// `basis` elements are expected to bind to non-negative values.
+///
+/// Return the `0` AffineConstantExpr if `basis` is empty.
+AffineExpr computeMaxLinearIndex(MLIRContext *ctx, ArrayRef<AffineExpr> basis);
+
+/// Return the linearized index of 'offsets' w.r.t. 'basis'.
+///
+/// Assuming `offsets` is `[o0, .. on]` and `basis` is `[b0, .. bn]`, return the
+/// AffineExpr `o0 * b0 + .. + on * bn`.
+///
+/// It is the caller's responsibility to pass proper AffineExpr kind that result
+/// in valid AffineExpr (i.e. cannot multiply 2 AffineDimExpr or divide by an
+/// AffineDimExpr).
+///
+/// `basis` elements are expected to bind to non-negative values.
+AffineExpr linearize(MLIRContext *ctx, ArrayRef<AffineExpr> offsets,
+                     ArrayRef<AffineExpr> basis);
+AffineExpr linearize(MLIRContext *ctx, ArrayRef<AffineExpr> offsets,
+                     ArrayRef<int64_t> basis);
+
+/// Given the strides together with a linear index in the dimension space,
+/// return the vector-space offsets in each dimension for a de-linearized index.
+///
+/// Let `li = linearIndex`, assuming `strides` are `[s0, .. sn]`, return the
+/// vector of AffineExpr
+///   `[li % s0, (li / s0) % s1, ..., (li / s0 / .. / sn-1) % sn]`
+///
+/// It is the caller's responsibility to pass proper AffineExpr kind that result
+/// in valid AffineExpr (i.e. cannot multiply 2 AffineDimExpr or divide by an
+/// AffineDimExpr).
+///
+/// `strides` elements are expected to bind to non-negative values.
+SmallVector<AffineExpr> delinearize(AffineExpr linearIndex,
+                                    ArrayRef<AffineExpr> strides);
+SmallVector<AffineExpr> delinearize(AffineExpr linearIndex,
+                                    ArrayRef<int64_t> strides);
+
+//===----------------------------------------------------------------------===//
+// Permutation utils.
+//===----------------------------------------------------------------------===//
 
 /// Apply the permutation defined by `permutation` to `inVec`.
 /// Element `i` in `inVec` is mapped to location `j = permutation[i]`.
-/// E.g.: for an input vector `inVec = ['a', 'b', 'c']` and a permutation vector
-/// `permutation = [2, 0, 1]`, this function leaves `inVec = ['c', 'a', 'b']`.
+/// E.g.: for an input vector `inVec = ['a', 'b', 'c']` and a permutation
+/// vector `permutation = [2, 0, 1]`, this function leaves `inVec = ['c', 'a',
+/// 'b']`.
 template <typename T, unsigned N>
 void applyPermutationToVector(SmallVector<T, N> &inVec,
                               ArrayRef<int64_t> permutation) {
@@ -83,18 +198,11 @@ SmallVector<int64_t> invertPermutationVector(ArrayRef<int64_t> permutation);
 /// Method to check if an interchange vector is a permutation.
 bool isPermutationVector(ArrayRef<int64_t> interchange);
 
-/// Helper that returns a subset of `arrayAttr` as a vector of int64_t.
+/// Helper to return a subset of `arrayAttr` as a vector of int64_t.
+// TODO: Port everything relevant to DenseArrayAttr and drop this util.
 SmallVector<int64_t> getI64SubArray(ArrayAttr arrayAttr, unsigned dropFront = 0,
                                     unsigned dropBack = 0);
 
-/// Computes and returns linearized affine expression w.r.t. `basis`.
-mlir::AffineExpr getLinearAffineExpr(ArrayRef<int64_t> basis, mlir::Builder &b);
-
-/// Given the strides in the dimension space, returns the affine expressions for
-/// vector-space offsets in each dimension for a de-linearized index.
-SmallVector<mlir::AffineExpr>
-getDelinearizedAffineExpr(ArrayRef<int64_t> strides, mlir::Builder &b);
-
 } // namespace mlir
 
 #endif // MLIR_DIALECT_UTILS_INDEXINGUTILS_H
diff --git a/mlir/include/mlir/Dialect/Utils/StaticValueUtils.h b/mlir/include/mlir/Dialect/Utils/StaticValueUtils.h
index c97c0834f588d..27c27756b3918 100644
--- a/mlir/include/mlir/Dialect/Utils/StaticValueUtils.h
+++ b/mlir/include/mlir/Dialect/Utils/StaticValueUtils.h
@@ -63,14 +63,18 @@ SmallVector<int64_t, 4> extractFromI64ArrayAttr(Attribute attr);
 /// Given a value, try to extract a constant Attribute. If this fails, return
 /// the original value.
 OpFoldResult getAsOpFoldResult(Value val);
-
 /// Given an array of values, try to extract a constant Attribute from each
 /// value. If this fails, return the original value.
 SmallVector<OpFoldResult> getAsOpFoldResult(ValueRange values);
-
 /// Convert `arrayAttr` to a vector of OpFoldResult.
 SmallVector<OpFoldResult> getAsOpFoldResult(ArrayAttr arrayAttr);
 
+/// Convert int64_t to integer attributes of index type and return them as
+/// OpFoldResult.
+OpFoldResult getAsIndexOpFoldResult(MLIRContext *ctx, int64_t val);
+SmallVector<OpFoldResult> getAsIndexOpFoldResult(MLIRContext *ctx,
+                                                 ArrayRef<int64_t> values);
+
 /// If ofr is a constant integer or an IntegerAttr, return the integer.
 std::optional<int64_t> getConstantIntValue(OpFoldResult ofr);
 
@@ -113,6 +117,17 @@ std::pair<ArrayAttr, SmallVector<Value>>
 decomposeMixedValues(Builder &b,
                      const SmallVectorImpl<OpFoldResult> &mixedValues);
 
+/// Helper to sort `values` according to matching `keys`.
+SmallVector<Value>
+getValuesSortedByKey(ArrayRef<Attribute> keys, ArrayRef<Value> values,
+                     llvm::function_ref<bool(Attribute, Attribute)> compare);
+SmallVector<OpFoldResult>
+getValuesSortedByKey(ArrayRef<Attribute> keys, ArrayRef<OpFoldResult> values,
+                     llvm::function_ref<bool(Attribute, Attribute)> compare);
+SmallVector<int64_t>
+getValuesSortedByKey(ArrayRef<Attribute> keys, ArrayRef<int64_t> values,
+                     llvm::function_ref<bool(Attribute, Attribute)> compare);
+
 } // namespace mlir
 
 #endif // MLIR_DIALECT_UTILS_STATICVALUEUTILS_H
diff --git a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
index 6a5913e512388..539486f48d49d 100644
--- a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
+++ b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
@@ -1751,10 +1751,16 @@ def Vector_MaskedLoadOp :
     element is read from memory. Otherwise, the corresponding element is taken
     from a 1-D pass-through vector. Informally the semantics are:
     ```
-    result[0] := mask[0] ? base[i+0] : pass_thru[0]
-    result[1] := mask[1] ? base[i+1] : pass_thru[1]
+    result[0] := if mask[0] then base[i + 0] else pass_thru[0]
+    result[1] := if mask[1] then base[i + 1] else pass_thru[1]
     etc.
     ```
+
+    If a mask bit is set and the corresponding index is out-of-bounds for the
+    given base, the behavior is undefined. If a mask bit is not set, the value
+    comes from the pass-through vector regardless of the index, and the index is
+    allowed to be out-of-bounds.
+
     The masked load can be used directly where applicable, or can be used
     during progressively lowering to bring other memory operations closer to
     hardware ISA support for a masked load. The semantics of the operation
@@ -1811,6 +1817,12 @@ def Vector_MaskedStoreOp :
     if (mask[1]) base[i+1] = value[1]
     etc.
     ```
+
+    If a mask bit is set and the corresponding index is out-of-bounds for the
+    given base, the behavior is undefined. If a mask bit is not set, no value
+    is stored regardless of the index, and the index is allowed to be
+    out-of-bounds.
+
     The masked store can be used directly where applicable, or can be used
     during progressively lowering to bring other memory operations closer to
     hardware ISA support for a masked store. The semantics of the operation
@@ -1857,21 +1869,27 @@ def Vector_GatherOp :
 
   let summary = [{
     gathers elements from memory or ranked tensor into a vector as defined by an
-    index vector and mask
+    index vector and a mask vector
   }];
 
   let description = [{
-    The gather operation gathers elements from memory or ranked tensor into a
-    n-D vector as defined by a base with indices and an additional n-D index
-    vector (each index is a 1-D offset on the base), but only if the
-    corresponding bit is set in a n-D mask vector. Otherwise, the element is
-    taken from a n-D pass-through vector. Informally the semantics are:
-    ```
-    result[0] := mask[0] ? base[index[0]] : pass_thru[0]
-    result[1] := mask[1] ? base[index[1]] : pass_thru[1]
+    The gather operation returns an n-D vector whose elements are either loaded
+    from memory or ranked tensor, or taken from a pass-through vector, depending
+    on the values of an n-D mask vector.
+    If a mask bit is set, the corresponding result element is defined by the base
+    with indices and the n-D index vector (each index is a 1-D offset on the base).
+    Otherwise, the corresponding element is taken from the n-D pass-through vector.
+    Informally the semantics are:
+    ```
+    result[0] := if mask[0] then base[index[0]] else pass_thru[0]
+    result[1] := if mask[1] then base[index[1]] else pass_thru[1]
     etc.
     ```
-    The vector dialect leaves out-of-bounds behavior undefined.
+
+    If a mask bit is set and the corresponding index is out-of-bounds for the
+    given base, the behavior is undefined. If a mask bit is not set, the value
+    comes from the pass-through vector regardless of the index, and the index is
+    allowed to be out-of-bounds.
 
     The gather operation can be used directly where applicable, or can be used
     during progressively lowering to bring other memory operations closer to
@@ -1921,10 +1939,13 @@ def Vector_ScatterOp :
                VectorOfRankAndType<[1], [I1]>:$mask,
                VectorOfRank<[1]>:$valueToStore)> {
 
-  let summary = "scatters elements from a vector into memory as defined by an index vector and mask";
+  let summary = [{
+    scatters elements from a vector into memory as defined by an index vector
+    and a mask vector
+  }];
 
   let description = [{
-    The scatter operation scatters elements from a 1-D vector into memory as
+    The scatter operation stores elements from a 1-D vector into memory as
     defined by a base with indices and an additional 1-D index vector, but
     only if the corresponding bit in a 1-D mask vector is set. Otherwise, no
     action is taken for that element. Informally the semantics are:
@@ -1933,10 +1954,16 @@ def Vector_ScatterOp :
     if (mask[1]) base[index[1]] = value[1]
     etc.
     ```
-    The vector dialect leaves out-of-bounds and repeated index behavior
-    undefined. Underlying implementations may enforce strict sequential
-    semantics for the latter, though.
-    TODO: enforce the latter always?
+
+    If a mask bit is set and the corresponding index is out-of-bounds for the
+    given base, the behavior is undefined. If a mask bit is not set, no value
+    is stored regardless of the index, and the index is allowed to be
+    out-of-bounds.
+
+    If the index vector contains two or more duplicate indices, the behavior is
+    undefined. Underlying implementation may enforce strict sequential
+    semantics.
+    TODO: always enforce strict sequential semantics?
 
     The scatter operation can be used directly where applicable, or can be used
     during progressively lowering to bring other memory operations closer to
@@ -1993,12 +2020,17 @@ def Vector_ExpandLoadOp :
     is taken from a 1-D pass-through vector. Informally the semantics are:
     ```
     index = i
-    result[0] := mask[0] ? base[index++] : pass_thru[0]
-    result[1] := mask[1] ? base[index++] : pass_thru[1]
+    result[0] := if mask[0] then base[index++] else pass_thru[0]
+    result[1] := if mask[1] then base[index++] else pass_thru[1]
     etc.
     ```
     Note that the index increment is done conditionally.
 
+    If a mask bit is set and the corresponding index is out-of-bounds for the
+    given base, the behavior is undefined. If a mask bit is not set, the value
+    comes from the pass-through vector regardless of the index, and the index is
+    allowed to be out-of-bounds.
+
     The expand load can be used directly where applicable, or can be used
     during progressively lowering to bring other memory operations closer to
     hardware ISA support for an expand. The semantics of the operation closely
@@ -2057,6 +2089,11 @@ def Vector_CompressStoreOp :
     ```
     Note that the index increment is done conditionally.
 
+    If a mask bit is set and the corresponding index is out-of-bounds for the
+    given base, the behavior is undefined. If a mask bit is not set, no value
+    is stored regardless of the index, and the index is allowed to be
+    out-of-bounds.
+
     The compress store can be used directly where applicable, or can be used
     during progressively lowering to bring other memory operations closer to
     hardware ISA support for a compress. The semantics of the operation closely
diff --git a/mlir/include/mlir/Dialect/Vector/Transforms/VectorRewritePatterns.h b/mlir/include/mlir/Dialect/Vector/Transforms/VectorRewritePatterns.h
index 775bfbf0241bf..af68de7e0051e 100644
--- a/mlir/include/mlir/Dialect/Vector/Transforms/VectorRewritePatterns.h
+++ b/mlir/include/mlir/Dialect/Vector/Transforms/VectorRewritePatterns.h
@@ -17,6 +17,7 @@
 #include "mlir/Dialect/Vector/Utils/VectorUtils.h"
 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/PatternMatch.h"
+#include "mlir/Support/LogicalResult.h"
 
 namespace mlir {
 class RewritePatternSet;
@@ -147,6 +148,27 @@ void populateVectorContractLoweringPatterns(
     VectorTransformsOptions options = VectorTransformsOptions(),
     PatternBenefit benefit = 1);
 
+/// Canonicalization of a `vector.contraction %a, %b, %c` with row-major matmul
+/// semantics to a contraction with MMT semantics (matrix matrix multiplication
+/// with the RHS transposed). This specific form is meant to have the vector
+/// operands are organized such that the reduction dimension is contiguous.
+/// Example:
+/// ```
+/// vector.contract {indexing_maps = [affine_map<(m, n, k) -> (m, k)>,
+///                                   affine_map<(m, n, k) -> (n, k)>,
+///                                   affine_map<(m, n, k) -> (m, n)>],
+///                  iterator_types = ["parallel", "parallel", "reduction"],
+///                  kind = #vector.kind<add>} %a, %b, %c : ...
+/// ```
+///
+///  The `constraint` predicate is used to decide which `vector.contraction` ops
+///  to filter out.
+void populateVectorContractCanonicalizeMatmulToMMT(
+    RewritePatternSet &patterns,
+    std::function<LogicalResult(vector::ContractionOp)> constraint =
+        [](vector::ContractionOp) { return success(); },
+    PatternBenefit = 1);
+
 /// Collect patterns to convert reduction op to vector.contract and fold
 /// transpose/broadcast ops into the contract.
 void populateVectorReductionToContractPatterns(RewritePatternSet &patterns,
@@ -312,6 +334,14 @@ void populateVectorUnrollPatterns(RewritePatternSet &patterns,
                                   const UnrollVectorOptions &options,
                                   PatternBenefit benefit = 1);
 
+/// Expands `vector.gather` ops into a series of conditional scalar loads
+/// (`vector.load` for memrefs or `tensor.extract` for tensors). These loads are
+/// conditional to avoid out-of-bounds memory accesses and guarded with `scf.if`
+/// ops. This lowering path is intended for targets that do not feature
+/// dedicated gather ops.
+void populateVectorGatherLoweringPatterns(RewritePatternSet &patterns,
+                                          PatternBenefit benefit = 1);
+
 //===----------------------------------------------------------------------===//
 // Finer-grained patterns exposed for more control over individual lowerings.
 //===----------------------------------------------------------------------===//
diff --git a/mlir/include/mlir/ExecutionEngine/SparseTensor/COO.h b/mlir/include/mlir/ExecutionEngine/SparseTensor/COO.h
index d004fe3e6ebb8..6f66a3884d649 100644
--- a/mlir/include/mlir/ExecutionEngine/SparseTensor/COO.h
+++ b/mlir/include/mlir/ExecutionEngine/SparseTensor/COO.h
@@ -27,24 +27,24 @@ namespace mlir {
 namespace sparse_tensor {
 
 /// An element of a sparse tensor in coordinate-scheme representation
-/// (i.e., a pair of indices and value).  For example, a rank-1 vector
-/// element would look like
+/// (i.e., a pair of coordinates and value).  For example, a rank-1
+/// vector element would look like
 ///   ({i}, a[i])
 /// and a rank-5 tensor element would look like
 ///   ({i,j,k,l,m}, a[i,j,k,l,m])
 ///
-/// The indices are represented as a (non-owning) pointer into a shared
-/// pool of indices, rather than being stored directly in this object.
-/// This significantly improves performance because it: (1) reduces
-/// the per-element memory footprint, and (2) centralizes the memory
-/// management for indices.  The only downside is that the indices
-/// themselves cannot be retrieved without knowing the rank of the
-/// tensor to which this element belongs (and that rank is not stored
-/// in this object).
+/// The coordinates are represented as a (non-owning) pointer into
+/// a shared pool of coordinates, rather than being stored directly in
+/// this object.  This significantly improves performance because it:
+/// (1) reduces the per-element memory footprint, and (2) centralizes
+/// the memory management for coordinates.  The only downside is that
+/// the coordinates themselves cannot be retrieved without knowing the
+/// rank of the tensor to which this element belongs (and that rank is
+/// not stored in this object).
 template <typename V>
 struct Element final {
-  Element(const uint64_t *ind, V val) : indices(ind), value(val){};
-  const uint64_t *indices; // pointer into shared index pool
+  Element(const uint64_t *coords, V val) : coords(coords), value(val){};
+  const uint64_t *coords; // pointer into shared coordinates pool
   V value;
 };
 
@@ -58,9 +58,9 @@ struct ElementLT final {
   /// Precondition: the elements must both be valid for `rank`.
   bool operator()(const Element<V> &e1, const Element<V> &e2) const {
     for (uint64_t d = 0; d < rank; ++d) {
-      if (e1.indices[d] == e2.indices[d])
+      if (e1.coords[d] == e2.coords[d])
         continue;
-      return e1.indices[d] < e2.indices[d];
+      return e1.coords[d] < e2.coords[d];
     }
     return false;
   }
@@ -132,11 +132,11 @@ class SparseTensorCOO final {
       assert(dimSizes[d] > 0 && "Dimension size zero has trivial storage");
     if (capacity) {
       elements.reserve(capacity);
-      indices.reserve(capacity * dimRank);
+      coordinates.reserve(capacity * dimRank);
     }
   }
 
-  /// Gets the rank of the tensor.
+  /// Gets the dimension-rank of the tensor.
   uint64_t getRank() const { return dimSizes.size(); }
 
   /// Gets the dimension-sizes array.
@@ -149,37 +149,38 @@ class SparseTensorCOO final {
   ElementLT<V> getElementLT() const { return ElementLT<V>(getRank()); }
 
   /// Adds an element to the tensor.  This method does not check whether
-  /// `ind` is already associated with a value, it adds it regardless.
+  /// `dimCoords` is already associated with a value, it adds it regardless.
   /// Resolving such conflicts is left up to clients of the iterator
   /// interface.
   ///
   /// This method invalidates all iterators.
   ///
   /// Asserts:
-  /// * the `ind` is valid for `rank`
-  /// * the elements of `ind` are valid for `dimSizes`.
-  void add(const std::vector<uint64_t> &ind, V val) {
-    const uint64_t *base = indices.data();
-    uint64_t size = indices.size();
-    uint64_t rank = getRank();
-    assert(ind.size() == rank && "Element rank mismatch");
-    for (uint64_t r = 0; r < rank; ++r) {
-      assert(ind[r] < dimSizes[r] && "Index is too large for the dimension");
-      indices.push_back(ind[r]);
+  /// * the `dimCoords` is valid for `getRank`.
+  /// * the components of `dimCoords` are valid for `getDimSizes`.
+  void add(const std::vector<uint64_t> &dimCoords, V val) {
+    const uint64_t *base = coordinates.data();
+    const uint64_t size = coordinates.size();
+    const uint64_t dimRank = getRank();
+    assert(dimCoords.size() == dimRank && "Element rank mismatch");
+    for (uint64_t d = 0; d < dimRank; ++d) {
+      assert(dimCoords[d] < dimSizes[d] &&
+             "Coordinate is too large for the dimension");
+      coordinates.push_back(dimCoords[d]);
     }
-    // This base only changes if indices were reallocated. In that case, we
-    // need to correct all previous pointers into the vector. Note that this
-    // only happens if we did not set the initial capacity right, and then only
-    // for every internal vector reallocation (which with the doubling rule
-    // should only incur an amortized linear overhead).
-    const uint64_t *newBase = indices.data();
+    // This base only changes if `coordinates` was reallocated.  In which
+    // case, we need to correct all previous pointers into the vector.
+    // Note that this only happens if we did not set the initial capacity
+    // right, and then only for every internal vector reallocation (which
+    // with the doubling rule should only incur an amortized linear overhead).
+    const uint64_t *const newBase = coordinates.data();
     if (newBase != base) {
       for (uint64_t i = 0, n = elements.size(); i < n; ++i)
-        elements[i].indices = newBase + (elements[i].indices - base);
+        elements[i].coords = newBase + (elements[i].coords - base);
       base = newBase;
     }
     // Add the new element and update the sorted bit.
-    Element<V> addedElem(base + size, val);
+    const Element<V> addedElem(base + size, val);
     if (!elements.empty() && isSorted)
       isSorted = getElementLT()(elements.back(), addedElem);
     elements.push_back(addedElem);
@@ -188,8 +189,9 @@ class SparseTensorCOO final {
   const_iterator begin() const { return elements.cbegin(); }
   const_iterator end() const { return elements.cend(); }
 
-  /// Sorts elements lexicographically by index.  If an index is mapped to
-  /// multiple values, then the relative order of those values is unspecified.
+  /// Sorts elements lexicographically by coordinates.  If a coordinate
+  /// is mapped to multiple values, then the relative order of those
+  /// values is unspecified.
   ///
   /// This method invalidates all iterators.
   void sort() {
@@ -202,7 +204,7 @@ class SparseTensorCOO final {
 private:
   const std::vector<uint64_t> dimSizes; // per-dimension sizes
   std::vector<Element<V>> elements;     // all COO elements
-  std::vector<uint64_t> indices;        // shared index pool
+  std::vector<uint64_t> coordinates;    // shared coordinate pool
   bool isSorted;
 };
 
diff --git a/mlir/include/mlir/ExecutionEngine/SparseTensor/File.h b/mlir/include/mlir/ExecutionEngine/SparseTensor/File.h
index d8382de72630e..8d5eb48f38048 100644
--- a/mlir/include/mlir/ExecutionEngine/SparseTensor/File.h
+++ b/mlir/include/mlir/ExecutionEngine/SparseTensor/File.h
@@ -46,7 +46,7 @@ struct is_complex<std::complex<T>> final : public std::true_type {};
 /// then returns an arbitrary value.  If `IsPattern` is false, then
 /// reads the value from the current line buffer beginning at `linePtr`.
 template <typename V, bool IsPattern>
-inline std::enable_if_t<!is_complex<V>::value, V> readCOOValue(char **linePtr) {
+inline std::enable_if_t<!is_complex<V>::value, V> readValue(char **linePtr) {
   // The external formats always store these numerical values with the type
   // double, but we cast these values to the sparse tensor object type.
   // For a pattern tensor, we arbitrarily pick the value 1 for all entries.
@@ -59,7 +59,7 @@ inline std::enable_if_t<!is_complex<V>::value, V> readCOOValue(char **linePtr) {
 /// then returns an arbitrary value.  If `IsPattern` is false, then reads
 /// the value from the current line buffer beginning at `linePtr`.
 template <typename V, bool IsPattern>
-inline std::enable_if_t<is_complex<V>::value, V> readCOOValue(char **linePtr) {
+inline std::enable_if_t<is_complex<V>::value, V> readValue(char **linePtr) {
   // Read two values to make a complex. The external formats always store
   // numerical values with the type double, but we cast these values to the
   // sparse tensor object type. For a pattern tensor, we arbitrarily pick the
@@ -72,14 +72,12 @@ inline std::enable_if_t<is_complex<V>::value, V> readCOOValue(char **linePtr) {
   return V(re, im);
 }
 
-/// Returns an element-value.  If `is_pattern` is true, then returns an
-/// arbitrary value.  If `is_pattern` is false, then reads the value from
+/// Returns an element-value.  If `isPattern` is true, then returns an
+/// arbitrary value.  If `isPattern` is false, then reads the value from
 /// the current line buffer beginning at `linePtr`.
 template <typename V>
-inline V readCOOValue(char **linePtr, bool is_pattern) {
-  if (is_pattern)
-    return readCOOValue<V, true>(linePtr);
-  return readCOOValue<V, false>(linePtr);
+inline V readValue(char **linePtr, bool isPattern) {
+  return isPattern ? readValue<V, true>(linePtr) : readValue<V, false>(linePtr);
 }
 
 } // namespace detail
@@ -171,15 +169,17 @@ class SparseTensorReader final {
     return isSymmetric_;
   }
 
-  /// Gets the rank of the tensor.  Is only valid after parsing the header.
+  /// Gets the dimension-rank of the tensor.  Is only valid after parsing
+  /// the header.
   uint64_t getRank() const {
     assert(isValid() && "Attempt to getRank() before readHeader()");
     return idata[0];
   }
 
-  /// Gets the number of non-zeros.  Is only valid after parsing the header.
-  uint64_t getNNZ() const {
-    assert(isValid() && "Attempt to getNNZ() before readHeader()");
+  /// Gets the number of stored elements.  Is only valid after parsing
+  /// the header.
+  uint64_t getNSE() const {
+    assert(isValid() && "Attempt to getNSE() before readHeader()");
     return idata[1];
   }
 
@@ -201,23 +201,22 @@ class SparseTensorReader final {
 
   /// Reads a sparse tensor element from the next line in the input file and
   /// returns the value of the element. Stores the coordinates of the element
-  /// to the `indices` array.
+  /// to the `dimCoords` array.
   template <typename V>
-  V readCOOElement(uint64_t rank, uint64_t *indices) {
-    assert(rank == getRank() && "rank mismatch");
-    char *linePtr = readCOOIndices(indices);
-    return detail::readCOOValue<V>(&linePtr, isPattern());
+  V readElement(uint64_t dimRank, uint64_t *dimCoords) {
+    assert(dimRank == getRank() && "rank mismatch");
+    char *linePtr = readCoords(dimCoords);
+    return detail::readValue<V>(&linePtr, isPattern());
   }
 
   /// Allocates a new COO object for `lvlSizes`, initializes it by reading
-  /// all the elements from the file and applying `dim2lvl` to their indices,
-  /// and then closes the file.
+  /// all the elements from the file and applying `dim2lvl` to their
+  /// dim-coordinates, and then closes the file.
   ///
   /// Preconditions:
   /// * `lvlSizes` must be valid for `lvlRank`.
   /// * `dim2lvl` must be valid for `getRank()`.
-  /// * `dim2lvl` maps indices valid for `getDimSizes()` to indices
-  ///   valid for `lvlSizes`.
+  /// * `dim2lvl` maps `getDimSizes()`-coordinates to `lvlSizes`-coordinates.
   /// * the file's actual value type can be read as `V`.
   ///
   /// Asserts:
@@ -263,22 +262,22 @@ class SparseTensorReader final {
   void readLine();
 
   /// Reads the next line of the input file and parses the coordinates
-  /// into the `indices` argument.  Returns the position in the `line`
+  /// into the `dimCoords` argument.  Returns the position in the `line`
   /// buffer where the element's value should be parsed from.  This method
-  /// has been factored out from `readCOOElement` to minimize code bloat
+  /// has been factored out from `readElement` to minimize code bloat
   /// for the generated library.
   ///
-  /// Precondition: `indices` is valid for `getRank()`.
-  template <typename I>
-  char *readCOOIndices(I *indices) {
+  /// Precondition: `dimCoords` is valid for `getRank()`.
+  template <typename C>
+  char *readCoords(C *dimCoords) {
     readLine();
     // Local variable for tracking the parser's position in the `line` buffer.
     char *linePtr = line;
     for (uint64_t dimRank = getRank(), d = 0; d < dimRank; ++d) {
-      // Parse the 1-based index.
-      uint64_t idx = strtoul(linePtr, &linePtr, 10);
-      // Store the 0-based index.
-      indices[d] = static_cast<I>(idx - 1);
+      // Parse the 1-based coordinate.
+      uint64_t c = strtoul(linePtr, &linePtr, 10);
+      // Store the 0-based coordinate.
+      dimCoords[d] = static_cast<C>(c - 1);
     }
     return linePtr;
   }
@@ -334,8 +333,8 @@ SparseTensorCOO<V> *SparseTensorReader::readCOO(uint64_t lvlRank,
   const uint64_t dimRank = getRank();
   assert(lvlRank == dimRank && "Rank mismatch");
   detail::PermutationRef d2l(dimRank, dim2lvl);
-  // Prepare a COO object with the number of nonzeros as initial capacity.
-  auto *lvlCOO = new SparseTensorCOO<V>(lvlRank, lvlSizes, getNNZ());
+  // Prepare a COO object with the number of stored elems as initial capacity.
+  auto *lvlCOO = new SparseTensorCOO<V>(lvlRank, lvlSizes, getNSE());
   // Do some manual LICM, to avoid assertions in the for-loop.
   const bool IsPattern = isPattern();
   if (IsPattern)
@@ -352,17 +351,17 @@ void SparseTensorReader::readCOOLoop(uint64_t lvlRank,
                                      detail::PermutationRef dim2lvl,
                                      SparseTensorCOO<V> *lvlCOO) {
   const uint64_t dimRank = getRank();
-  std::vector<uint64_t> dimInd(dimRank);
-  std::vector<uint64_t> lvlInd(lvlRank);
-  for (uint64_t nnz = getNNZ(), k = 0; k < nnz; ++k) {
-    // We inline `readCOOElement` here in order to avoid redundant
+  std::vector<uint64_t> dimCoords(dimRank);
+  std::vector<uint64_t> lvlCoords(lvlRank);
+  for (uint64_t nse = getNSE(), k = 0; k < nse; ++k) {
+    // We inline `readElement` here in order to avoid redundant
     // assertions, since they're guaranteed by the call to `isValid()`
-    // and the construction of `dimInd` above.
-    char *linePtr = readCOOIndices(dimInd.data());
-    const V value = detail::readCOOValue<V, IsPattern>(&linePtr);
-    dim2lvl.pushforward(dimRank, dimInd.data(), lvlInd.data());
+    // and the construction of `dimCoords` above.
+    char *linePtr = readCoords(dimCoords.data());
+    const V value = detail::readValue<V, IsPattern>(&linePtr);
+    dim2lvl.pushforward(dimRank, dimCoords.data(), lvlCoords.data());
     // TODO: <https://github.com/llvm/llvm-project/issues/54179>
-    lvlCOO->add(lvlInd, value);
+    lvlCOO->add(lvlCoords, value);
   }
 }
 
@@ -371,6 +370,10 @@ bool SparseTensorReader::readToBuffers(uint64_t lvlRank,
                                        const uint64_t *dim2lvl,
                                        C *lvlCoordinates, V *values) {
   assert(isValid() && "Attempt to readCOO() before readHeader()");
+  // Construct a `PermutationRef` for the `pushforward` below.
+  // TODO: This specific implementation does not generalize to arbitrary
+  // mappings, but once we functionalize the `dim2lvl` argument we can
+  // simply use that function instead.
   const uint64_t dimRank = getRank();
   assert(lvlRank == dimRank && "Rank mismatch");
   detail::PermutationRef d2l(dimRank, dim2lvl);
@@ -391,19 +394,19 @@ bool SparseTensorReader::readToBuffersLoop(uint64_t lvlRank,
                                            detail::PermutationRef dim2lvl,
                                            C *lvlCoordinates, V *values) {
   const uint64_t dimRank = getRank();
-  const uint64_t nse = getNNZ();
+  const uint64_t nse = getNSE();
   std::vector<C> dimCoords(dimRank);
   // Read the first element with isSorted=false as a way to avoid accessing its
   // previous element.
   bool isSorted = false;
   char *linePtr;
-  // We inline `readCOOElement` here in order to avoid redundant assertions,
+  // We inline `readElement` here in order to avoid redundant assertions,
   // since they're guaranteed by the call to `isValid()` and the construction
   // of `dimCoords` above.
-  auto readElement = [&]() {
-    linePtr = readCOOIndices<C>(dimCoords.data());
+  const auto readNextElement = [&]() {
+    linePtr = readCoords<C>(dimCoords.data());
     dim2lvl.pushforward(dimRank, dimCoords.data(), lvlCoordinates);
-    *values = detail::readCOOValue<V, IsPattern>(&linePtr);
+    *values = detail::readValue<V, IsPattern>(&linePtr);
     if (isSorted) {
       // Note that isSorted was set to false while reading the first element,
       // to guarantee the safeness of using prevLvlCoords.
@@ -421,10 +424,10 @@ bool SparseTensorReader::readToBuffersLoop(uint64_t lvlRank,
     lvlCoordinates += lvlRank;
     ++values;
   };
-  readElement();
+  readNextElement();
   isSorted = true;
   for (uint64_t n = 1; n < nse; ++n)
-    readElement();
+    readNextElement();
 
   return isSorted;
 }
@@ -434,21 +437,21 @@ template <typename V>
 inline void writeExtFROSTT(const SparseTensorCOO<V> &coo,
                            const char *filename) {
   assert(filename && "Got nullptr for filename");
-  auto &dimSizes = coo.getDimSizes();
-  auto &elements = coo.getElements();
-  const uint64_t rank = coo.getRank();
-  const uint64_t nnz = elements.size();
+  const auto &dimSizes = coo.getDimSizes();
+  const auto &elements = coo.getElements();
+  const uint64_t dimRank = coo.getRank();
+  const uint64_t nse = elements.size();
   std::fstream file;
   file.open(filename, std::ios_base::out | std::ios_base::trunc);
   assert(file.is_open());
-  file << "; extended FROSTT format\n" << rank << " " << nnz << std::endl;
-  for (uint64_t r = 0; r < rank - 1; ++r)
-    file << dimSizes[r] << " ";
-  file << dimSizes[rank - 1] << std::endl;
-  for (uint64_t i = 0; i < nnz; ++i) {
-    auto &idx = elements[i].indices;
-    for (uint64_t r = 0; r < rank; ++r)
-      file << (idx[r] + 1) << " ";
+  file << "; extended FROSTT format\n" << dimRank << " " << nse << std::endl;
+  for (uint64_t d = 0; d < dimRank - 1; ++d)
+    file << dimSizes[d] << " ";
+  file << dimSizes[dimRank - 1] << std::endl;
+  for (uint64_t i = 0; i < nse; ++i) {
+    const auto &coords = elements[i].coords;
+    for (uint64_t d = 0; d < dimRank; ++d)
+      file << (coords[d] + 1) << " ";
     file << elements[i].value << std::endl;
   }
   file.flush();
diff --git a/mlir/include/mlir/ExecutionEngine/SparseTensor/Storage.h b/mlir/include/mlir/ExecutionEngine/SparseTensor/Storage.h
index c5e310937efe4..c38ed59068be5 100644
--- a/mlir/include/mlir/ExecutionEngine/SparseTensor/Storage.h
+++ b/mlir/include/mlir/ExecutionEngine/SparseTensor/Storage.h
@@ -15,9 +15,9 @@
 // This file contains definitions for the following classes:
 //
 // * `SparseTensorStorageBase`
-// * `SparseTensorStorage<P, I, V>`
+// * `SparseTensorStorage<P, C, V>`
 // * `SparseTensorEnumeratorBase<V>`
-// * `SparseTensorEnumerator<P, I, V>`
+// * `SparseTensorEnumerator<P, C, V>`
 // * `SparseTensorNNZ`
 //
 // Ideally we would split the storage classes and enumerator classes
@@ -52,9 +52,9 @@ class SparseTensorEnumeratorBase;
 // These macros ensure consistent error messages, without risk of incuring
 // an additional method call to do so.
 #define ASSERT_VALID_DIM(d)                                                    \
-  assert(d < getDimRank() && "Dimension index is out of bounds");
+  assert(d < getDimRank() && "Dimension is out of bounds");
 #define ASSERT_VALID_LVL(l)                                                    \
-  assert(l < getLvlRank() && "Level index is out of bounds");
+  assert(l < getLvlRank() && "Level is out of bounds");
 #define ASSERT_COMPRESSED_LVL(l)                                               \
   assert(isCompressedLvl(l) && "Level is not compressed");
 #define ASSERT_COMPRESSED_OR_SINGLETON_LVL(l)                                  \
@@ -70,8 +70,8 @@ class SparseTensorEnumeratorBase;
 // retained for the sake of future-proofing.
 #define ASSERT_DENSE_DLT(dlt) assert(isDenseDLT(dlt) && "Level is not dense");
 
-/// Abstract base class for `SparseTensorStorage<P,I,V>`.  This class
-/// takes responsibility for all the `<P,I,V>`-independent aspects
+/// Abstract base class for `SparseTensorStorage<P,C,V>`.  This class
+/// takes responsibility for all the `<P,C,V>`-independent aspects
 /// of the tensor (e.g., shape, sparsity, permutation).  In addition,
 /// we use function overloading to implement "partial" method
 /// specialization, which the C-API relies on to catch type errors
@@ -89,7 +89,7 @@ class SparseTensorEnumeratorBase;
 /// and prone to introduce confusion whenever the qualifiers are dropped.
 /// Where necessary, we use "axis" as the generic term.
 ///
-/// The *size* of an axis is the cardinality of possible coordinate/index
+/// The *size* of an axis is the cardinality of possible coordinate
 /// values along that axis (regardless of which coordinates have stored
 /// element values).  As such, each size must be non-zero since if any
 /// axis has size-zero then the whole tensor would have trivial storage
@@ -129,8 +129,7 @@ class SparseTensorStorageBase {
   /// * `dimSizes`, `lvlSizes`, `lvlTypes`, and `lvl2dim` must be nonnull.
   /// * `dimSizes` must be valid for `dimRank`.
   /// * `lvlSizes`, `lvlTypes`, and `lvl2dim` must be valid for `lvlRank`.
-  /// * `lvl2dim` must map indices valid for `lvlSizes` to indices valid
-  ///   for `dimSizes`.
+  /// * `lvl2dim` must map `lvlSizes`-coordinates to `dimSizes`-coordinates.
   ///
   /// Asserts:
   /// * `dimRank` and `lvlRank` are nonzero.
@@ -213,28 +212,29 @@ class SparseTensorStorageBase {
   MLIR_SPARSETENSOR_FOREVERY_V(DECL_NEWENUMERATOR)
 #undef DECL_NEWENUMERATOR
 
-  /// Gets pointers-overhead storage.
-#define DECL_GETPOINTERS(PNAME, P)                                             \
-  virtual void getPointers(std::vector<P> **, uint64_t);
-  MLIR_SPARSETENSOR_FOREVERY_FIXED_O(DECL_GETPOINTERS)
-#undef DECL_GETPOINTERS
+  /// Gets positions-overhead storage for the given level.
+#define DECL_GETPOSITIONS(PNAME, P)                                            \
+  virtual void getPositions(std::vector<P> **, uint64_t);
+  MLIR_SPARSETENSOR_FOREVERY_FIXED_O(DECL_GETPOSITIONS)
+#undef DECL_GETPOSITIONS
 
-  /// Gets indices-overhead storage.
-#define DECL_GETINDICES(INAME, I)                                              \
-  virtual void getIndices(std::vector<I> **, uint64_t);
-  MLIR_SPARSETENSOR_FOREVERY_FIXED_O(DECL_GETINDICES)
-#undef DECL_GETINDICES
-  virtual uint64_t getIndex(uint64_t l, uint64_t pos) const = 0;
+  /// Gets coordinates-overhead storage for the given level.
+#define DECL_GETCOORDINATES(INAME, C)                                          \
+  virtual void getCoordinates(std::vector<C> **, uint64_t);
+  MLIR_SPARSETENSOR_FOREVERY_FIXED_O(DECL_GETCOORDINATES)
+#undef DECL_GETCOORDINATES
+  /// Gets the coordinate-value stored at the given level and position.
+  virtual uint64_t getCrd(uint64_t lvl, uint64_t pos) const = 0;
 
   /// Gets primary storage.
 #define DECL_GETVALUES(VNAME, V) virtual void getValues(std::vector<V> **);
   MLIR_SPARSETENSOR_FOREVERY_V(DECL_GETVALUES)
 #undef DECL_GETVALUES
 
-  /// Element-wise insertion in lexicographic index order.  The first
-  /// argument is the level-indices for the value being inserted.
+  /// Element-wise insertion in lexicographic coordinate order.  The first
+  /// argument is the level-coordinates for the value being inserted.
   // TODO: For better safety, this should take a parameter for the
-  // length of `lvlInd` and check that against `getLvlRank()`.
+  // length of `lvlCoords` and check that against `getLvlRank()`.
 #define DECL_LEXINSERT(VNAME, V) virtual void lexInsert(const uint64_t *, V);
   MLIR_SPARSETENSOR_FOREVERY_V(DECL_LEXINSERT)
 #undef DECL_LEXINSERT
@@ -244,7 +244,7 @@ class SparseTensorStorageBase {
   /// iterating over the nonzero elements.
   ///
   /// Arguments:
-  /// * `lvlInd` the level-indices shared by the values being inserted.
+  /// * `lvlCoords` the level-coordinates shared by the values being inserted.
   /// * `values` a map from last-level coordinates to their associated value.
   /// * `filled` a map from last-level coordinates to bool, indicating
   ///   whether `values` contains a valid value to be inserted.
@@ -269,22 +269,22 @@ class SparseTensorStorageBase {
 //===----------------------------------------------------------------------===//
 // This forward decl is necessary for defining `SparseTensorStorage`,
 // but isn't sufficient for splitting it off.
-template <typename P, typename I, typename V>
+template <typename P, typename C, typename V>
 class SparseTensorEnumerator;
 
 /// A memory-resident sparse tensor using a storage scheme based on
-/// per-dimension sparse/dense annotations.  This data structure provides
+/// per-level sparse/dense annotations.  This data structure provides
 /// a bufferized form of a sparse tensor type.  In contrast to generating
 /// setup methods for each differently annotated sparse tensor, this
 /// method provides a convenient "one-size-fits-all" solution that simply
 /// takes an input tensor and annotations to implement all required setup
 /// in a general manner.
-template <typename P, typename I, typename V>
+template <typename P, typename C, typename V>
 class SparseTensorStorage final : public SparseTensorStorageBase {
   /// Private constructor to share code between the other constructors.
   /// Beware that the object is not necessarily guaranteed to be in a
   /// valid state after this constructor alone; e.g., `isCompressedLvl(l)`
-  /// doesn't entail `!(pointers[l].empty())`.
+  /// doesn't entail `!(positions[l].empty())`.
   ///
   /// Preconditions/assertions are as per the `SparseTensorStorageBase` ctor.
   SparseTensorStorage(uint64_t dimRank, const uint64_t *dimSizes,
@@ -292,7 +292,7 @@ class SparseTensorStorage final : public SparseTensorStorageBase {
                       const DimLevelType *lvlTypes, const uint64_t *lvl2dim)
       : SparseTensorStorageBase(dimRank, dimSizes, lvlRank, lvlSizes, lvlTypes,
                                 lvl2dim),
-        pointers(lvlRank), indices(lvlRank), lvlCursor(lvlRank) {}
+        positions(lvlRank), coordinates(lvlRank), lvlCursor(lvlRank) {}
 
 public:
   /// Constructs a sparse tensor with the given encoding, and allocates
@@ -339,11 +339,11 @@ class SparseTensorStorage final : public SparseTensorStorageBase {
   /// are as per the `SparseTensorStorageBase` ctor; which is to say,
   /// the `dimSizes` and `lvlSizes` must both be "sizes" not "shapes",
   /// since there's nowhere to reconstruct dynamic sizes from.
-  static SparseTensorStorage<P, I, V> *
+  static SparseTensorStorage<P, C, V> *
   newEmpty(uint64_t dimRank, const uint64_t *dimSizes, uint64_t lvlRank,
            const uint64_t *lvlSizes, const DimLevelType *lvlTypes,
            const uint64_t *lvl2dim) {
-    return new SparseTensorStorage<P, I, V>(dimRank, dimSizes, lvlRank,
+    return new SparseTensorStorage<P, C, V>(dimRank, dimSizes, lvlRank,
                                             lvlSizes, lvlTypes, lvl2dim, true);
   }
 
@@ -365,7 +365,7 @@ class SparseTensorStorage final : public SparseTensorStorageBase {
   // that there's a feasible way to do so from within the library itself.
   // Therefore, when we functionalize the `lvl2dim` mapping we'll have
   // to update the type/preconditions of this factory too.
-  static SparseTensorStorage<P, I, V> *
+  static SparseTensorStorage<P, C, V> *
   newFromCOO(uint64_t dimRank, const uint64_t *dimShape, uint64_t lvlRank,
              const DimLevelType *lvlTypes, const uint64_t *lvl2dim,
              SparseTensorCOO<V> &lvlCOO);
@@ -375,9 +375,9 @@ class SparseTensorStorage final : public SparseTensorStorageBase {
   ///
   /// Preconditions:
   /// * as per the `SparseTensorStorageBase` ctor.
-  /// * `src2lvl` must be valid for `srcRank`, must map indices valid for
-  ///   `source.getDimSizes()` to indices valid for `lvlSizes`, and therefore
-  ///   must be the inverse of `lvl2dim`.
+  /// * `src2lvl` must be valid for `srcRank`, must map coordinates valid
+  ///    for `source.getDimSizes()` to coordinates valid for `lvlSizes`,
+  ///    and therefore must be the inverse of `lvl2dim`.
   /// * `source` must have the same value type `V`.
   ///
   /// Asserts:
@@ -396,7 +396,7 @@ class SparseTensorStorage final : public SparseTensorStorageBase {
   // enable this factory to be used for performing a much larger class of
   // transformations (which can already be handled by the `SparseTensorNNZ`
   // implementation).
-  static SparseTensorStorage<P, I, V> *
+  static SparseTensorStorage<P, C, V> *
   newFromSparseTensor(uint64_t dimRank, const uint64_t *dimShape,
                       uint64_t lvlRank, const uint64_t *lvlSizes,
                       const DimLevelType *lvlTypes, const uint64_t *lvl2dim,
@@ -406,67 +406,67 @@ class SparseTensorStorage final : public SparseTensorStorageBase {
   ~SparseTensorStorage() final = default;
 
   /// Partially specialize these getter methods based on template types.
-  void getPointers(std::vector<P> **out, uint64_t l) final {
+  void getPositions(std::vector<P> **out, uint64_t lvl) final {
     assert(out && "Received nullptr for out parameter");
-    ASSERT_VALID_LVL(l);
-    *out = &pointers[l];
+    ASSERT_VALID_LVL(lvl);
+    *out = &positions[lvl];
   }
-  void getIndices(std::vector<I> **out, uint64_t l) final {
+  void getCoordinates(std::vector<C> **out, uint64_t lvl) final {
     assert(out && "Received nullptr for out parameter");
-    ASSERT_VALID_LVL(l);
-    *out = &indices[l];
+    ASSERT_VALID_LVL(lvl);
+    *out = &coordinates[lvl];
   }
   void getValues(std::vector<V> **out) final {
     assert(out && "Received nullptr for out parameter");
     *out = &values;
   }
 
-  uint64_t getIndex(uint64_t l, uint64_t pos) const final {
-    ASSERT_COMPRESSED_OR_SINGLETON_LVL(l);
-    assert(pos < indices[l].size() && "Index position is out of bounds");
-    return indices[l][pos]; // Converts the stored `I` into `uint64_t`.
+  uint64_t getCrd(uint64_t lvl, uint64_t pos) const final {
+    ASSERT_COMPRESSED_OR_SINGLETON_LVL(lvl);
+    assert(pos < coordinates[lvl].size() && "Position is out of bounds");
+    return coordinates[lvl][pos]; // Converts the stored `C` into `uint64_t`.
   }
 
   /// Partially specialize lexicographical insertions based on template types.
-  void lexInsert(const uint64_t *lvlInd, V val) final {
-    assert(lvlInd && "Received nullptr for level-indices");
+  void lexInsert(const uint64_t *lvlCoords, V val) final {
+    assert(lvlCoords && "Received nullptr for level-coordinates");
     // First, wrap up pending insertion path.
     uint64_t diffLvl = 0;
-    uint64_t topIdx = 0;
+    uint64_t full = 0;
     if (!values.empty()) {
-      diffLvl = lexDiff(lvlInd);
+      diffLvl = lexDiff(lvlCoords);
       endPath(diffLvl + 1);
-      topIdx = lvlCursor[diffLvl] + 1;
+      full = lvlCursor[diffLvl] + 1;
     }
     // Then continue with insertion path.
-    insPath(lvlInd, diffLvl, topIdx, val);
+    insPath(lvlCoords, diffLvl, full, val);
   }
 
   /// Partially specialize expanded insertions based on template types.
-  void expInsert(uint64_t *lvlInd, V *values, bool *filled, uint64_t *added,
+  void expInsert(uint64_t *lvlCoords, V *values, bool *filled, uint64_t *added,
                  uint64_t count) final {
-    assert((lvlInd && values && filled && added) && "Received nullptr");
+    assert((lvlCoords && values && filled && added) && "Received nullptr");
     if (count == 0)
       return;
     // Sort.
     std::sort(added, added + count);
     // Restore insertion path for first insert.
     const uint64_t lastLvl = getLvlRank() - 1;
-    uint64_t index = added[0];
-    assert(filled[index] && "added index is not filled");
-    lvlInd[lastLvl] = index;
-    lexInsert(lvlInd, values[index]);
-    values[index] = 0;
-    filled[index] = false;
+    uint64_t c = added[0];
+    assert(filled[c] && "added coordinate is not filled");
+    lvlCoords[lastLvl] = c;
+    lexInsert(lvlCoords, values[c]);
+    values[c] = 0;
+    filled[c] = false;
     // Subsequent insertions are quick.
     for (uint64_t i = 1; i < count; ++i) {
-      assert(index < added[i] && "non-lexicographic insertion");
-      index = added[i];
-      assert(filled[index] && "added index is not filled");
-      lvlInd[lastLvl] = index;
-      insPath(lvlInd, lastLvl, added[i - 1] + 1, values[index]);
-      values[index] = 0;
-      filled[index] = false;
+      assert(c < added[i] && "non-lexicographic insertion");
+      c = added[i];
+      assert(filled[c] && "added coordinate is not filled");
+      lvlCoords[lastLvl] = c;
+      insPath(lvlCoords, lastLvl, added[i - 1] + 1, values[c]);
+      values[c] = 0;
+      filled[c] = false;
     }
   }
 
@@ -478,14 +478,14 @@ class SparseTensorStorage final : public SparseTensorStorageBase {
       endPath(0);
   }
 
-  /// Allocates a new enumerator for this class's `<P,I,V>` types and
-  /// erase the `<P,I>` parts from the type.  Callers must make sure to
+  /// Allocates a new enumerator for this class's `<P,C,V>` types and
+  /// erase the `<P,C>` parts from the type.  Callers must make sure to
   /// delete the enumerator when they're done with it.
   void newEnumerator(SparseTensorEnumeratorBase<V> **out, uint64_t trgRank,
                      const uint64_t *trgSizes, uint64_t srcRank,
                      const uint64_t *src2trg) const final {
     assert(out && "Received nullptr for out parameter");
-    *out = new SparseTensorEnumerator<P, I, V>(*this, trgRank, trgSizes,
+    *out = new SparseTensorEnumerator<P, C, V>(*this, trgRank, trgSizes,
                                                srcRank, src2trg);
   }
 
@@ -498,11 +498,11 @@ class SparseTensorStorage final : public SparseTensorStorageBase {
   SparseTensorCOO<V> *toCOO(uint64_t trgRank, const uint64_t *trgSizes,
                             uint64_t srcRank, const uint64_t *src2trg) const {
     // We inline `newEnumerator` to avoid virtual dispatch and allocation.
-    SparseTensorEnumerator<P, I, V> enumerator(*this, trgRank, trgSizes,
+    SparseTensorEnumerator<P, C, V> enumerator(*this, trgRank, trgSizes,
                                                srcRank, src2trg);
     auto *coo = new SparseTensorCOO<V>(trgRank, trgSizes, values.size());
     enumerator.forallElements(
-        [&coo](const auto &trgInd, V val) { coo->add(trgInd, val); });
+        [&coo](const auto &trgCoords, V val) { coo->add(trgCoords, val); });
     // TODO: This assertion assumes there are no stored zeros,
     // or if there are then that we don't filter them out.
     // Cf., <https://github.com/llvm/llvm-project/issues/54179>
@@ -511,58 +511,58 @@ class SparseTensorStorage final : public SparseTensorStorageBase {
   }
 
 private:
-  /// Appends an arbitrary new position to `pointers[l]`.  This method
+  /// Appends an arbitrary new position to `positions[lvl]`.  This method
   /// checks that `pos` is representable in the `P` type; however, it
   /// does not check that `pos` is semantically valid (i.e., larger than
-  /// the previous position and smaller than `indices[l].capacity()`).
-  void appendPointer(uint64_t l, uint64_t pos, uint64_t count = 1) {
-    ASSERT_COMPRESSED_LVL(l);
+  /// the previous position and smaller than `coordinates[lvl].capacity()`).
+  void appendPos(uint64_t lvl, uint64_t pos, uint64_t count = 1) {
+    ASSERT_COMPRESSED_LVL(lvl);
     // TODO: we'd like to recover the nicer error message:
-    // "Pointer value is too large for the P-type"
-    pointers[l].insert(pointers[l].end(), count,
-                       detail::checkOverflowCast<P>(pos));
+    // "Position value is too large for the P-type"
+    positions[lvl].insert(positions[lvl].end(), count,
+                          detail::checkOverflowCast<P>(pos));
   }
 
-  /// Appends index `i` to level `l`, in the semantically general sense.
-  /// For non-dense levels, that means appending to the `indices[l]` array,
-  /// checking that `i` is representable in the `I` type; however, we do
-  /// not verify other semantic requirements (e.g., that `i` is in bounds
-  /// for `lvlSizes[l]`, and not previously occurring in the same segment).
-  /// For dense levels, this method instead appends the appropriate number
-  /// of zeros to the `values` array, where `full` is the number of "entries"
-  /// already written to `values` for this segment (aka one after the highest
-  /// index previously appended).
-  void appendIndex(uint64_t l, uint64_t full, uint64_t i) {
-    const auto dlt = getLvlType(l); // Avoid redundant bounds checking.
+  /// Appends coordinate `crd` to level `lvl`, in the semantically
+  /// general sense.  For non-dense levels, that means appending to the
+  /// `coordinates[lvl]` array, checking that `crd` is representable in
+  /// the `C` type; however, we do not verify other semantic requirements
+  /// (e.g., that `crd` is in bounds for `lvlSizes[lvl]`, and not previously
+  /// occurring in the same segment).  For dense levels, this method instead
+  /// appends the appropriate number of zeros to the `values` array, where
+  /// `full` is the number of "entries" already written to `values` for this
+  /// segment (aka one after the highest coordinate previously appended).
+  void appendCrd(uint64_t lvl, uint64_t full, uint64_t crd) {
+    const auto dlt = getLvlType(lvl); // Avoid redundant bounds checking.
     if (isCompressedDLT(dlt) || isSingletonDLT(dlt)) {
       // TODO: we'd like to recover the nicer error message:
-      // "Index value is too large for the I-type"
-      indices[l].push_back(detail::checkOverflowCast<I>(i));
-    } else { // Dense dimension.
+      // "Coordinate value is too large for the C-type"
+      coordinates[lvl].push_back(detail::checkOverflowCast<C>(crd));
+    } else { // Dense level.
       ASSERT_DENSE_DLT(dlt);
-      assert(i >= full && "Index was already filled");
-      if (i == full)
+      assert(crd >= full && "Coordinate was already filled");
+      if (crd == full)
         return; // Short-circuit, since it'll be a nop.
-      if (l + 1 == getLvlRank())
-        values.insert(values.end(), i - full, 0);
+      if (lvl + 1 == getLvlRank())
+        values.insert(values.end(), crd - full, 0);
       else
-        finalizeSegment(l + 1, 0, i - full);
+        finalizeSegment(lvl + 1, 0, crd - full);
     }
   }
 
-  /// Writes the given coordinate to `indices[l][pos]`.  This method
-  /// checks that `i` is representable in the `I` type; however, it
-  /// does not check that `i` is semantically valid (i.e., in bounds
-  /// for `dimSizes[l]` and not elsewhere occurring in the same segment).
-  void writeIndex(uint64_t l, uint64_t pos, uint64_t i) {
-    ASSERT_COMPRESSED_OR_SINGLETON_LVL(l);
+  /// Writes the given coordinate to `coordinates[lvl][pos]`.  This method
+  /// checks that `crd` is representable in the `C` type; however, it
+  /// does not check that `crd` is semantically valid (i.e., in bounds
+  /// for `dimSizes[lvl]` and not elsewhere occurring in the same segment).
+  void writeCrd(uint64_t lvl, uint64_t pos, uint64_t crd) {
+    ASSERT_COMPRESSED_OR_SINGLETON_LVL(lvl);
     // Subscript assignment to `std::vector` requires that the `pos`-th
     // entry has been initialized; thus we must be sure to check `size()`
     // here, instead of `capacity()` as would be ideal.
-    assert(pos < indices[l].size() && "Index position is out of bounds");
+    assert(pos < coordinates[lvl].size() && "Position is out of bounds");
     // TODO: we'd like to recover the nicer error message:
-    // "Index value is too large for the I-type"
-    indices[l][pos] = detail::checkOverflowCast<I>(i);
+    // "Coordinate value is too large for the C-type"
+    coordinates[lvl][pos] = detail::checkOverflowCast<C>(crd);
   }
 
   /// Computes the assembled-size associated with the `l`-th level,
@@ -571,12 +571,12 @@ class SparseTensorStorage final : public SparseTensorStorageBase {
   /// storage, as opposed to "level-sizes" which are the cardinality
   /// of possible coordinates for that level.
   ///
-  /// Precondition: the `pointers[l]` array must be fully initialized
+  /// Precondition: the `positions[l]` array must be fully initialized
   /// before calling this method.
   uint64_t assembledSize(uint64_t parentSz, uint64_t l) const {
     const auto dlt = getLvlType(l); // Avoid redundant bounds checking.
     if (isCompressedDLT(dlt))
-      return pointers[l][parentSz];
+      return positions[l][parentSz];
     if (isSingletonDLT(dlt))
       return parentSz; // New size is same as the parent.
     if (isDenseDLT(dlt))
@@ -586,18 +586,18 @@ class SparseTensorStorage final : public SparseTensorStorageBase {
   }
 
   /// Initializes sparse tensor storage scheme from a memory-resident sparse
-  /// tensor in coordinate scheme. This method prepares the pointers and
-  /// indices arrays under the given per-dimension dense/sparse annotations.
+  /// tensor in coordinate scheme. This method prepares the positions and
+  /// coordinates arrays under the given per-level dense/sparse annotations.
   ///
   /// Preconditions:
   /// * the `lvlElements` must be lexicographically sorted.
-  /// * the indices of every element are valid for `getLvlSizes()`
+  /// * the coordinates of every element are valid for `getLvlSizes()`
   ///   (i.e., equal rank and pointwise less-than).
   void fromCOO(const std::vector<Element<V>> &lvlElements, uint64_t lo,
                uint64_t hi, uint64_t l) {
     const uint64_t lvlRank = getLvlRank();
     assert(l <= lvlRank && hi <= lvlElements.size());
-    // Once dimensions are exhausted, insert the numerical values.
+    // Once levels are exhausted, insert the numerical values.
     if (l == lvlRank) {
       assert(lo < hi);
       values.push_back(lvlElements[lo].value);
@@ -606,30 +606,30 @@ class SparseTensorStorage final : public SparseTensorStorageBase {
     // Visit all elements in this interval.
     uint64_t full = 0;
     while (lo < hi) { // If `hi` is unchanged, then `lo < lvlElements.size()`.
-      // Find segment in interval with same index elements in this level.
-      const uint64_t i = lvlElements[lo].indices[l];
+      // Find segment in interval with same coordinate at this level.
+      const uint64_t c = lvlElements[lo].coords[l];
       uint64_t seg = lo + 1;
       if (isUniqueLvl(l))
-        while (seg < hi && lvlElements[seg].indices[l] == i)
+        while (seg < hi && lvlElements[seg].coords[l] == c)
           ++seg;
       // Handle segment in interval for sparse or dense level.
-      appendIndex(l, full, i);
-      full = i + 1;
+      appendCrd(l, full, c);
+      full = c + 1;
       fromCOO(lvlElements, lo, seg, l + 1);
       // And move on to next segment in interval.
       lo = seg;
     }
-    // Finalize the sparse pointer structure at this level.
+    // Finalize the sparse position structure at this level.
     finalizeSegment(l, full);
   }
 
-  /// Finalizes the sparse pointer structure at this level.
+  /// Finalizes the sparse position structure at this level.
   void finalizeSegment(uint64_t l, uint64_t full = 0, uint64_t count = 1) {
     if (count == 0)
       return; // Short-circuit, since it'll be a nop.
     const auto dlt = getLvlType(l); // Avoid redundant bounds checking.
     if (isCompressedDLT(dlt)) {
-      appendPointer(l, indices[l].size(), count);
+      appendPos(l, coordinates[l].size(), count);
     } else if (isSingletonDLT(dlt)) {
       return; // Nothing to finalize.
     } else {  // Dense dimension.
@@ -661,29 +661,29 @@ class SparseTensorStorage final : public SparseTensorStorageBase {
   }
 
   /// Continues a single insertion path, outer to inner.  The first
-  /// argument is the storage-level indices for the value being inserted.
-  void insPath(const uint64_t *lvlInd, uint64_t diffLvl, uint64_t topIdx,
+  /// argument is the level-coordinates for the value being inserted.
+  void insPath(const uint64_t *lvlCoords, uint64_t diffLvl, uint64_t full,
                V val) {
     const uint64_t lvlRank = getLvlRank();
     assert(diffLvl <= lvlRank && "Level-diff is out of bounds");
     for (uint64_t l = diffLvl; l < lvlRank; ++l) {
-      const uint64_t i = lvlInd[l];
-      appendIndex(l, topIdx, i);
-      topIdx = 0;
-      lvlCursor[l] = i;
+      const uint64_t c = lvlCoords[l];
+      appendCrd(l, full, c);
+      full = 0;
+      lvlCursor[l] = c;
     }
     values.push_back(val);
   }
 
-  /// Finds the lexicographically first level where the level-indices
+  /// Finds the lexicographically first level where the level-coordinates
   /// in the argument differ from those in the current cursor.
-  uint64_t lexDiff(const uint64_t *lvlInd) const {
+  uint64_t lexDiff(const uint64_t *lvlCoords) const {
     const uint64_t lvlRank = getLvlRank();
     for (uint64_t l = 0; l < lvlRank; ++l)
-      if (lvlInd[l] > lvlCursor[l])
+      if (lvlCoords[l] > lvlCursor[l])
         return l;
       else
-        assert(lvlInd[l] == lvlCursor[l] && "non-lexicographic insertion");
+        assert(lvlCoords[l] == lvlCursor[l] && "non-lexicographic insertion");
     assert(0 && "duplicate insertion");
     return -1u;
   }
@@ -691,10 +691,10 @@ class SparseTensorStorage final : public SparseTensorStorageBase {
   // Allow `SparseTensorEnumerator` to access the data-members (to avoid
   // the cost of virtual-function dispatch in inner loops), without
   // making them public to other client code.
-  friend class SparseTensorEnumerator<P, I, V>;
+  friend class SparseTensorEnumerator<P, C, V>;
 
-  std::vector<std::vector<P>> pointers;
-  std::vector<std::vector<I>> indices;
+  std::vector<std::vector<P>> positions;
+  std::vector<std::vector<C>> coordinates;
   std::vector<V> values;
   std::vector<uint64_t> lvlCursor; // cursor for lexicographic insertion.
 };
@@ -709,7 +709,7 @@ class SparseTensorStorage final : public SparseTensorStorageBase {
 /// `SparseTensorStorage` under a permutation.  That is, the `forallElements`
 /// method encapsulates the loop-nest for enumerating the elements of
 /// the source tensor (in whatever order is best for the source tensor),
-/// and applies a permutation to the coordinates/indices before handing
+/// and applies a permutation to the coordinates before handing
 /// each element to the callback.  A single enumerator object can be
 /// freely reused for several calls to `forallElements`, just so long
 /// as each call is sequential with respect to one another.
@@ -719,10 +719,10 @@ class SparseTensorStorage final : public SparseTensorStorageBase {
 /// outlive the sparse tensor they depend on.
 ///
 /// Design Note: The reason we define this class instead of simply using
-/// `SparseTensorEnumerator<P,I,V>` is because we need to hide/generalize
-/// the `<P,I>` template parameters from MLIR client code (to simplify the
+/// `SparseTensorEnumerator<P,C,V>` is because we need to hide/generalize
+/// the `<P,C>` template parameters from MLIR client code (to simplify the
 /// type parameters used for direct sparse-to-sparse conversion).  And the
-/// reason we define the `SparseTensorEnumerator<P,I,V>` subclasses rather
+/// reason we define the `SparseTensorEnumerator<P,C,V>` subclasses rather
 /// than simply using this class, is to avoid the cost of virtual-method
 /// dispatch within the loop-nest.
 template <typename V>
@@ -735,8 +735,8 @@ class MLIR_SPARSETENSOR_GSL_POINTER [[nodiscard]] SparseTensorEnumeratorBase {
   /// Preconditions:
   /// * the `src` must have the same `V` value type.
   /// * `trgSizes` must be valid for `trgRank`.
-  /// * `src2trg` must be valid for `srcRank`, and must map indices
-  ///   valid for `src.getDimSizes()` to indices valid for `trgSizes`.
+  /// * `src2trg` must be valid for `srcRank`, and must map coordinates
+  ///   valid for `src.getDimSizes()` to coordinates valid for `trgSizes`.
   ///
   /// Asserts:
   /// * `trgSizes` must be nonnull and must contain only nonzero sizes.
@@ -779,7 +779,7 @@ class MLIR_SPARSETENSOR_GSL_POINTER [[nodiscard]] SparseTensorEnumeratorBase {
   const std::vector<uint64_t> &getTrgSizes() const { return trgSizes; }
 
   /// Enumerates all elements of the source tensor, permutes their
-  /// indices, and passes the permuted element to the callback.
+  /// coordinates, and passes the permuted element to the callback.
   /// The callback must not store the cursor reference directly,
   /// since this function reuses the storage.  Instead, the callback
   /// must copy it if they want to keep it.
@@ -793,11 +793,11 @@ class MLIR_SPARSETENSOR_GSL_POINTER [[nodiscard]] SparseTensorEnumeratorBase {
 };
 
 //===----------------------------------------------------------------------===//
-template <typename P, typename I, typename V>
+template <typename P, typename C, typename V>
 class MLIR_SPARSETENSOR_GSL_POINTER [[nodiscard]] SparseTensorEnumerator final
     : public SparseTensorEnumeratorBase<V> {
   using Base = SparseTensorEnumeratorBase<V>;
-  using StorageImpl = SparseTensorStorage<P, I, V>;
+  using StorageImpl = SparseTensorStorage<P, C, V>;
 
 public:
   /// Constructs an enumerator which automatically applies the given
@@ -830,7 +830,7 @@ class MLIR_SPARSETENSOR_GSL_POINTER [[nodiscard]] SparseTensorEnumerator final
   /// The recursive component of the public `forallElements`.
   void forallElements(ElementConsumer<V> yield, uint64_t parentPos,
                       uint64_t l) {
-    // Recover the `<P,I,V>` type parameters of `src`.
+    // Recover the `<P,C,V>` type parameters of `src`.
     const auto &src = static_cast<const StorageImpl &>(this->src);
     if (l == src.getLvlRank()) {
       assert(parentPos < src.values.size() &&
@@ -844,28 +844,28 @@ class MLIR_SPARSETENSOR_GSL_POINTER [[nodiscard]] SparseTensorEnumerator final
     if (isCompressedDLT(dlt)) {
       // Look up the bounds of the `l`-level segment determined by the
       // `(l - 1)`-level position `parentPos`.
-      const std::vector<P> &pointersL = src.pointers[l];
-      assert(parentPos + 1 < pointersL.size() &&
-             "Parent pointer position is out of bounds");
-      const uint64_t pstart = static_cast<uint64_t>(pointersL[parentPos]);
-      const uint64_t pstop = static_cast<uint64_t>(pointersL[parentPos + 1]);
-      // Loop-invariant code for looking up the `l`-level coordinates/indices.
-      const std::vector<I> &indicesL = src.indices[l];
-      assert(pstop <= indicesL.size() && "Index position is out of bounds");
+      const std::vector<P> &positionsL = src.positions[l];
+      assert(parentPos + 1 < positionsL.size() &&
+             "Parent position is out of bounds");
+      const uint64_t pstart = static_cast<uint64_t>(positionsL[parentPos]);
+      const uint64_t pstop = static_cast<uint64_t>(positionsL[parentPos + 1]);
+      // Loop-invariant code for looking up the `l`-level coordinates.
+      const std::vector<C> &coordinatesL = src.coordinates[l];
+      assert(pstop <= coordinatesL.size() && "Stop position is out of bounds");
       for (uint64_t pos = pstart; pos < pstop; ++pos) {
-        cursorL = static_cast<uint64_t>(indicesL[pos]);
+        cursorL = static_cast<uint64_t>(coordinatesL[pos]);
         forallElements(yield, pos, l + 1);
       }
     } else if (isSingletonDLT(dlt)) {
-      cursorL = src.getIndex(l, parentPos);
+      cursorL = src.getCrd(l, parentPos);
       forallElements(yield, parentPos, l + 1);
-    } else { // Dense dimension.
+    } else { // Dense level.
       ASSERT_DENSE_DLT(dlt);
       const uint64_t sz = src.getLvlSizes()[l];
       const uint64_t pstart = parentPos * sz;
-      for (uint64_t i = 0; i < sz; ++i) {
-        cursorL = i;
-        forallElements(yield, pstart + i, l + 1);
+      for (uint64_t c = 0; c < sz; ++c) {
+        cursorL = c;
+        forallElements(yield, pstart + c, l + 1);
       }
     }
   }
@@ -913,29 +913,29 @@ class MLIR_SPARSETENSOR_GSL_POINTER [[nodiscard]] SparseTensorNNZ final {
     assert(enumerator.getTrgRank() == getLvlRank() && "Tensor rank mismatch");
     assert(enumerator.getTrgSizes() == lvlSizes && "Tensor size mismatch");
     enumerator.forallElements(
-        [this](const std::vector<uint64_t> &ind, V) { add(ind); });
+        [this](const std::vector<uint64_t> &lvlCoords, V) { add(lvlCoords); });
   }
 
   /// The type of callback functions which receive an nnz-statistic.
   using NNZConsumer = const std::function<void(uint64_t)> &;
 
-  /// Lexicographically enumerates all indicies for levels strictly
+  /// Lexicographically enumerates all coordinates for levels strictly
   /// less than `stopLvl`, and passes their nnz statistic to the callback.
   /// Since our use-case only requires the statistic not the coordinates
   /// themselves, we do not bother to construct those coordinates.
-  void forallIndices(uint64_t stopLvl, NNZConsumer yield) const;
+  void forallCoords(uint64_t stopLvl, NNZConsumer yield) const;
 
 private:
   /// Adds a new element (i.e., increment its statistics).  We use
   /// a method rather than inlining into the lambda in `initialize`,
   /// to avoid spurious templating over `V`.  And this method is private
-  /// to avoid needing to re-assert validity of `lvlInd` (which is
+  /// to avoid needing to re-assert validity of `lvlCoords` (which is
   /// guaranteed by `forallElements`).
-  void add(const std::vector<uint64_t> &lvlInd);
+  void add(const std::vector<uint64_t> &lvlCoords);
 
-  /// Recursive component of the public `forallIndices`.
-  void forallIndices(NNZConsumer yield, uint64_t stopLvl, uint64_t parentPos,
-                     uint64_t l) const;
+  /// Recursive component of the public `forallCoords`.
+  void forallCoords(NNZConsumer yield, uint64_t stopLvl, uint64_t parentPos,
+                    uint64_t l) const;
 
   // All of these are in the target storage-order.
   const std::vector<uint64_t> &lvlSizes;
@@ -944,10 +944,10 @@ class MLIR_SPARSETENSOR_GSL_POINTER [[nodiscard]] SparseTensorNNZ final {
 };
 
 //===----------------------------------------------------------------------===//
-// Definitions of the ctors and factories of `SparseTensorStorage<P,I,V>`.
+// Definitions of the ctors and factories of `SparseTensorStorage<P,C,V>`.
 
-template <typename P, typename I, typename V>
-SparseTensorStorage<P, I, V> *SparseTensorStorage<P, I, V>::newFromCOO(
+template <typename P, typename C, typename V>
+SparseTensorStorage<P, C, V> *SparseTensorStorage<P, C, V>::newFromCOO(
     uint64_t dimRank, const uint64_t *dimShape, uint64_t lvlRank,
     const DimLevelType *lvlTypes, const uint64_t *lvl2dim,
     SparseTensorCOO<V> &lvlCOO) {
@@ -966,12 +966,12 @@ SparseTensorStorage<P, I, V> *SparseTensorStorage<P, I, V>::newFromCOO(
            "Dimension sizes do not match expected shape");
     dimSizes[d] = lvlSizes[l];
   }
-  return new SparseTensorStorage<P, I, V>(dimRank, dimSizes.data(), lvlRank,
+  return new SparseTensorStorage<P, C, V>(dimRank, dimSizes.data(), lvlRank,
                                           lvlTypes, lvl2dim, lvlCOO);
 }
 
-template <typename P, typename I, typename V>
-SparseTensorStorage<P, I, V> *SparseTensorStorage<P, I, V>::newFromSparseTensor(
+template <typename P, typename C, typename V>
+SparseTensorStorage<P, C, V> *SparseTensorStorage<P, C, V>::newFromSparseTensor(
     uint64_t dimRank, const uint64_t *dimShape, uint64_t lvlRank,
     const uint64_t *lvlSizes, const DimLevelType *lvlTypes,
     const uint64_t *lvl2dim, uint64_t srcRank, const uint64_t *src2lvl,
@@ -989,24 +989,24 @@ SparseTensorStorage<P, I, V> *SparseTensorStorage<P, I, V>::newFromSparseTensor(
 #endif
   SparseTensorEnumeratorBase<V> *lvlEnumerator;
   source.newEnumerator(&lvlEnumerator, lvlRank, lvlSizes, srcRank, src2lvl);
-  auto *tensor = new SparseTensorStorage<P, I, V>(
+  auto *tensor = new SparseTensorStorage<P, C, V>(
       dimRank, dimSizes.data(), lvlRank, lvlTypes, lvl2dim, *lvlEnumerator);
   delete lvlEnumerator;
   return tensor;
 }
 
-template <typename P, typename I, typename V>
-SparseTensorStorage<P, I, V>::SparseTensorStorage(
+template <typename P, typename C, typename V>
+SparseTensorStorage<P, C, V>::SparseTensorStorage(
     uint64_t dimRank, const uint64_t *dimSizes, uint64_t lvlRank,
     const uint64_t *lvlSizes, const DimLevelType *lvlTypes,
     const uint64_t *lvl2dim, bool initializeValuesIfAllDense)
     : SparseTensorStorage(dimRank, dimSizes, lvlRank, lvlSizes, lvlTypes,
                           lvl2dim) {
-  // Provide hints on capacity of pointers and indices.
+  // Provide hints on capacity of positions and coordinates.
   // TODO: needs much fine-tuning based on actual sparsity; currently
-  //       we reserve pointer/index space based on all previous dense
-  //       dimensions, which works well up to first sparse dim; but
-  //       we should really use nnz and dense/sparse distribution.
+  // we reserve position/coordinate space based on all previous dense
+  // levels, which works well up to first sparse level; but we should
+  // really use nnz and dense/sparse distribution.
   bool allDense = true;
   uint64_t sz = 1;
   for (uint64_t l = 0; l < lvlRank; ++l) {
@@ -1014,16 +1014,16 @@ SparseTensorStorage<P, I, V>::SparseTensorStorage(
     if (isCompressedDLT(dlt)) {
       // TODO: Take a parameter between 1 and `lvlSizes[l]`, and multiply
       // `sz` by that before reserving. (For now we just use 1.)
-      pointers[l].reserve(sz + 1);
-      pointers[l].push_back(0);
-      indices[l].reserve(sz);
+      positions[l].reserve(sz + 1);
+      positions[l].push_back(0);
+      coordinates[l].reserve(sz);
       sz = 1;
       allDense = false;
     } else if (isSingletonDLT(dlt)) {
-      indices[l].reserve(sz);
+      coordinates[l].reserve(sz);
       sz = 1;
       allDense = false;
-    } else { // Dense dimension.
+    } else { // Dense level.
       ASSERT_DENSE_DLT(dlt);
       sz = detail::checkedMul(sz, lvlSizes[l]);
     }
@@ -1032,8 +1032,8 @@ SparseTensorStorage<P, I, V>::SparseTensorStorage(
     values.resize(sz, 0);
 }
 
-template <typename P, typename I, typename V>
-SparseTensorStorage<P, I, V>::SparseTensorStorage( // NOLINT
+template <typename P, typename C, typename V>
+SparseTensorStorage<P, C, V>::SparseTensorStorage( // NOLINT
     uint64_t dimRank, const uint64_t *dimSizes, uint64_t lvlRank,
     const DimLevelType *lvlTypes, const uint64_t *lvl2dim,
     SparseTensorCOO<V> &lvlCOO)
@@ -1046,13 +1046,13 @@ SparseTensorStorage<P, I, V>::SparseTensorStorage( // NOLINT
   lvlCOO.sort();
   // Now actually insert the `elements`.
   const auto &elements = lvlCOO.getElements();
-  uint64_t nnz = elements.size();
-  values.reserve(nnz);
-  fromCOO(elements, 0, nnz, 0);
+  const uint64_t nse = elements.size();
+  values.reserve(nse);
+  fromCOO(elements, 0, nse, 0);
 }
 
-template <typename P, typename I, typename V>
-SparseTensorStorage<P, I, V>::SparseTensorStorage(
+template <typename P, typename C, typename V>
+SparseTensorStorage<P, C, V>::SparseTensorStorage(
     uint64_t dimRank, const uint64_t *dimSizes, uint64_t lvlRank,
     const DimLevelType *lvlTypes, const uint64_t *lvl2dim,
     SparseTensorEnumeratorBase<V> &lvlEnumerator)
@@ -1064,64 +1064,64 @@ SparseTensorStorage<P, I, V>::SparseTensorStorage(
     // Initialize the statistics structure.
     SparseTensorNNZ nnz(getLvlSizes(), getLvlTypes());
     nnz.initialize(lvlEnumerator);
-    // Initialize "pointers" overhead (and allocate "indices", "values").
+    // Initialize "positions" overhead (and allocate "coordinates", "values").
     uint64_t parentSz = 1; // assembled-size of the `(l - 1)`-level.
     for (uint64_t l = 0; l < lvlRank; ++l) {
       const auto dlt = lvlTypes[l]; // Avoid redundant bounds checking.
       if (isCompressedDLT(dlt)) {
-        pointers[l].reserve(parentSz + 1);
-        pointers[l].push_back(0);
+        positions[l].reserve(parentSz + 1);
+        positions[l].push_back(0);
         uint64_t currentPos = 0;
-        nnz.forallIndices(l, [this, &currentPos, l](uint64_t n) {
+        nnz.forallCoords(l, [this, &currentPos, l](uint64_t n) {
           currentPos += n;
-          appendPointer(l, currentPos);
+          appendPos(l, currentPos);
         });
-        assert(pointers[l].size() == parentSz + 1 &&
-               "Final pointers size doesn't match allocated size");
+        assert(positions[l].size() == parentSz + 1 &&
+               "Final positions size doesn't match allocated size");
         // That assertion entails `assembledSize(parentSz, l)`
-        // is now in a valid state.  That is, `pointers[l][parentSz]`
+        // is now in a valid state.  That is, `positions[l][parentSz]`
         // equals the present value of `currentPos`, which is the
-        // correct assembled-size for `indices[l]`.
+        // correct assembled-size for `coordinates[l]`.
       }
       // Update assembled-size for the next iteration.
       parentSz = assembledSize(parentSz, l);
-      // Ideally we need only `indices[l].reserve(parentSz)`, however
+      // Ideally we need only `coordinates[l].reserve(parentSz)`, however
       // the `std::vector` implementation forces us to initialize it too.
       // That is, in the yieldPos loop we need random-access assignment
-      // to `indices[l]`; however, `std::vector`'s subscript-assignment
+      // to `coordinates[l]`; however, `std::vector`'s subscript-assignment
       // only allows assigning to already-initialized positions.
       if (isCompressedDLT(dlt) || isSingletonDLT(dlt))
-        indices[l].resize(parentSz, 0);
+        coordinates[l].resize(parentSz, 0);
       else
         ASSERT_DENSE_DLT(dlt); // Future-proofing.
     }
     values.resize(parentSz, 0); // Both allocate and zero-initialize.
   }
   // The yieldPos loop
-  lvlEnumerator.forallElements([this](const auto &lvlInd, V val) {
+  lvlEnumerator.forallElements([this](const auto &lvlCoords, V val) {
     uint64_t parentSz = 1, parentPos = 0;
     for (uint64_t lvlRank = getLvlRank(), l = 0; l < lvlRank; ++l) {
       const auto dlt = getLvlTypes()[l]; // Avoid redundant bounds checking.
       if (isCompressedDLT(dlt)) {
         // If `parentPos == parentSz` then it's valid as an array-lookup;
         // however, it's semantically invalid here since that entry
-        // does not represent a segment of `indices[l]`.  Moreover, that
+        // does not represent a segment of `coordinates[l]`.  Moreover, that
         // entry must be immutable for `assembledSize` to remain valid.
-        assert(parentPos < parentSz && "Pointers position is out of bounds");
-        const uint64_t currentPos = pointers[l][parentPos];
+        assert(parentPos < parentSz && "Parent position is out of bounds");
+        const uint64_t currentPos = positions[l][parentPos];
         // This increment won't overflow the `P` type, since it can't
-        // exceed the original value of `pointers[l][parentPos+1]`
+        // exceed the original value of `positions[l][parentPos+1]`
         // which was already verified to be within bounds for `P`
         // when it was written to the array.
-        pointers[l][parentPos]++;
-        writeIndex(l, currentPos, lvlInd[l]);
+        positions[l][parentPos]++;
+        writeCrd(l, currentPos, lvlCoords[l]);
         parentPos = currentPos;
       } else if (isSingletonDLT(dlt)) {
-        writeIndex(l, parentPos, lvlInd[l]);
+        writeCrd(l, parentPos, lvlCoords[l]);
         // the new parentPos equals the old parentPos.
-      } else { // Dense dimension.
+      } else { // Dense level.
         ASSERT_DENSE_DLT(dlt);
-        parentPos = parentPos * getLvlSizes()[l] + lvlInd[l];
+        parentPos = parentPos * getLvlSizes()[l] + lvlCoords[l];
       }
       parentSz = assembledSize(parentSz, l);
     }
@@ -1132,17 +1132,17 @@ SparseTensorStorage<P, I, V>::SparseTensorStorage(
   for (uint64_t parentSz = 1, l = 0; l < lvlRank; ++l) {
     const auto dlt = lvlTypes[l]; // Avoid redundant bounds checking.
     if (isCompressedDLT(dlt)) {
-      assert(parentSz == pointers[l].size() - 1 &&
-             "Actual pointers size doesn't match the expected size");
+      assert(parentSz == positions[l].size() - 1 &&
+             "Actual positions size doesn't match the expected size");
       // Can't check all of them, but at least we can check the last one.
-      assert(pointers[l][parentSz - 1] == pointers[l][parentSz] &&
-             "Pointers got corrupted");
+      assert(positions[l][parentSz - 1] == positions[l][parentSz] &&
+             "Positions got corrupted");
       // TODO: optimize this by using `memmove` or similar.
       for (uint64_t n = 0; n < parentSz; ++n) {
         const uint64_t parentPos = parentSz - n;
-        pointers[l][parentPos] = pointers[l][parentPos - 1];
+        positions[l][parentPos] = positions[l][parentPos - 1];
       }
-      pointers[l][0] = 0;
+      positions[l][0] = 0;
     } else {
       // Both dense and singleton are no-ops for the finalizeYieldPos loop.
       // This assertion is for future-proofing.
diff --git a/mlir/include/mlir/ExecutionEngine/SparseTensorRuntime.h b/mlir/include/mlir/ExecutionEngine/SparseTensorRuntime.h
index 0c5c8c2db5bcc..21a02502e4a9e 100644
--- a/mlir/include/mlir/ExecutionEngine/SparseTensorRuntime.h
+++ b/mlir/include/mlir/ExecutionEngine/SparseTensorRuntime.h
@@ -61,8 +61,8 @@ MLIR_CRUNNERUTILS_EXPORT void *_mlir_ciface_newSparseTensor( // NOLINT
     StridedMemRefType<index_type, 1> *lvlSizesRef,
     StridedMemRefType<DimLevelType, 1> *lvlTypesRef,
     StridedMemRefType<index_type, 1> *lvl2dimRef,
-    StridedMemRefType<index_type, 1> *dim2lvlRef, OverheadType ptrTp,
-    OverheadType indTp, PrimaryType valTp, Action action, void *ptr);
+    StridedMemRefType<index_type, 1> *dim2lvlRef, OverheadType posTp,
+    OverheadType crdTp, PrimaryType valTp, Action action, void *ptr);
 
 // TODO: document what all the arguments are/mean for the functions below,
 // especially with regards to "dim"-vs-"lvl" and mappings/permutations.
@@ -74,43 +74,46 @@ MLIR_CRUNNERUTILS_EXPORT void *_mlir_ciface_newSparseTensor( // NOLINT
 MLIR_SPARSETENSOR_FOREVERY_V(DECL_SPARSEVALUES)
 #undef DECL_SPARSEVALUES
 
-/// Tensor-storage method to obtain direct access to the pointers array
-/// for the given dimension.
-#define DECL_SPARSEPOINTERS(PNAME, P)                                          \
-  MLIR_CRUNNERUTILS_EXPORT void _mlir_ciface_sparsePointers##PNAME(            \
-      StridedMemRefType<P, 1> *out, void *tensor, index_type d);
-MLIR_SPARSETENSOR_FOREVERY_O(DECL_SPARSEPOINTERS)
-#undef DECL_SPARSEPOINTERS
-
-/// Tensor-storage method to obtain direct access to the indices array
-/// for the given dimension.
-#define DECL_SPARSEINDICES(INAME, I)                                           \
-  MLIR_CRUNNERUTILS_EXPORT void _mlir_ciface_sparseIndices##INAME(             \
-      StridedMemRefType<I, 1> *out, void *tensor, index_type d);
-MLIR_SPARSETENSOR_FOREVERY_O(DECL_SPARSEINDICES)
-#undef DECL_SPARSEINDICES
+/// Tensor-storage method to obtain direct access to the positions array
+/// for the given level.
+#define DECL_SPARSEPOSITIONS(PNAME, P)                                         \
+  MLIR_CRUNNERUTILS_EXPORT void _mlir_ciface_sparsePositions##PNAME(           \
+      StridedMemRefType<P, 1> *out, void *tensor, index_type lvl);
+MLIR_SPARSETENSOR_FOREVERY_O(DECL_SPARSEPOSITIONS)
+#undef DECL_SPARSEPOSITIONS
+
+/// Tensor-storage method to obtain direct access to the coordinates array
+/// for the given level.
+#define DECL_SPARSECOORDINATES(CNAME, C)                                       \
+  MLIR_CRUNNERUTILS_EXPORT void _mlir_ciface_sparseCoordinates##CNAME(         \
+      StridedMemRefType<C, 1> *out, void *tensor, index_type lvl);
+MLIR_SPARSETENSOR_FOREVERY_O(DECL_SPARSECOORDINATES)
+#undef DECL_SPARSECOORDINATES
 
 /// Coordinate-scheme method for adding a new element.
 #define DECL_ADDELT(VNAME, V)                                                  \
   MLIR_CRUNNERUTILS_EXPORT void *_mlir_ciface_addElt##VNAME(                   \
-      void *coo, StridedMemRefType<V, 0> *vref,                                \
-      StridedMemRefType<index_type, 1> *iref,                                  \
-      StridedMemRefType<index_type, 1> *pref);
+      void *lvlCOO, StridedMemRefType<V, 0> *vref,                             \
+      StridedMemRefType<index_type, 1> *dimCoordsRef,                          \
+      StridedMemRefType<index_type, 1> *dim2lvlRef);
 MLIR_SPARSETENSOR_FOREVERY_V(DECL_ADDELT)
 #undef DECL_ADDELT
 
 /// Coordinate-scheme method for getting the next element while iterating.
+/// The `cref` argument uses the same coordinate-space as the `iter` (which
+/// can be either dim- or lvl-coords, depending on context).
 #define DECL_GETNEXT(VNAME, V)                                                 \
   MLIR_CRUNNERUTILS_EXPORT bool _mlir_ciface_getNext##VNAME(                   \
-      void *coo, StridedMemRefType<index_type, 1> *iref,                       \
+      void *iter, StridedMemRefType<index_type, 1> *cref,                      \
       StridedMemRefType<V, 0> *vref);
 MLIR_SPARSETENSOR_FOREVERY_V(DECL_GETNEXT)
 #undef DECL_GETNEXT
 
-/// Tensor-storage method to insert elements in lexicographical index order.
+/// Tensor-storage method to insert elements in lexicographical
+/// level-coordinate order.
 #define DECL_LEXINSERT(VNAME, V)                                               \
   MLIR_CRUNNERUTILS_EXPORT void _mlir_ciface_lexInsert##VNAME(                 \
-      void *tensor, StridedMemRefType<index_type, 1> *cref,                    \
+      void *tensor, StridedMemRefType<index_type, 1> *lvlCoordsRef,            \
       StridedMemRefType<V, 0> *vref);
 MLIR_SPARSETENSOR_FOREVERY_V(DECL_LEXINSERT)
 #undef DECL_LEXINSERT
@@ -118,7 +121,7 @@ MLIR_SPARSETENSOR_FOREVERY_V(DECL_LEXINSERT)
 /// Tensor-storage method to insert using expansion.
 #define DECL_EXPINSERT(VNAME, V)                                               \
   MLIR_CRUNNERUTILS_EXPORT void _mlir_ciface_expInsert##VNAME(                 \
-      void *tensor, StridedMemRefType<index_type, 1> *cref,                    \
+      void *tensor, StridedMemRefType<index_type, 1> *lvlCoordsRef,            \
       StridedMemRefType<V, 1> *vref, StridedMemRefType<bool, 1> *fref,         \
       StridedMemRefType<index_type, 1> *aref, index_type count);
 MLIR_SPARSETENSOR_FOREVERY_V(DECL_EXPINSERT)
@@ -181,9 +184,10 @@ MLIR_CRUNNERUTILS_EXPORT void readSparseTensorShape(char *filename,
 ///   nse:     number of specified elements (usually the nonzeros)
 ///   shape:   array with dimension size for each rank
 ///   values:  a "nse" array with values for all specified elements
-///   indices: a flat "nse * rank" array with indices for all specified elements
-///   perm:    the permutation of the dimensions in the storage
-///   sparse:  the sparsity for the dimensions
+///   coordinates: a flat "nse * rank" array with coordinates for all
+///            specified elements
+///   perm:    the permutation of the levels in the storage
+///   sparse:  the sparsity for the levels
 ///
 /// For example, the sparse matrix
 ///     | 1.0 0.0 0.0 |
@@ -193,11 +197,11 @@ MLIR_CRUNNERUTILS_EXPORT void readSparseTensorShape(char *filename,
 ///      nse     = 3
 ///      shape   = [2, 3]
 ///      values  = [1.0, 5.0, 3.0]
-///      indices = [ 0, 0,  1, 1,  1, 2]
+///      coordinates = [ 0, 0,  1, 1,  1, 2]
 #define DECL_CONVERTTOMLIRSPARSETENSOR(VNAME, V)                               \
   MLIR_CRUNNERUTILS_EXPORT void *convertToMLIRSparseTensor##VNAME(             \
-      uint64_t rank, uint64_t nse, uint64_t *shape, V *values,                 \
-      uint64_t *indices, uint64_t *perm, uint8_t *sparse);
+      uint64_t rank, uint64_t nse, uint64_t *dimSizes, V *values,              \
+      uint64_t *dimCoordinates, uint64_t *dim2lvl, uint8_t *lvlTypes);
 MLIR_SPARSETENSOR_FOREVERY_V(DECL_CONVERTTOMLIRSPARSETENSOR)
 #undef DECL_CONVERTTOMLIRSPARSETENSOR
 
@@ -207,16 +211,17 @@ MLIR_SPARSETENSOR_FOREVERY_V(DECL_CONVERTTOMLIRSPARSETENSOR)
 ///
 ///   rank:    rank of tensor
 ///   nse:     number of specified elements (usually the nonzeros)
-///   shape:   array with dimension size for each rank
+///   shape:   array with size for each dimension
 ///   values:  a "nse" array with values for all specified elements
-///   indices: a flat "nse * rank" array with indices for all specified elements
+///   coordinates: a flat "nse * rank" array with coordinates for all
+///            specified elements
 ///
-/// The input is a pointer to `SparseTensorStorage<P, I, V>`, typically
+/// The input is a pointer to `SparseTensorStorage<P, C, V>`, typically
 /// returned from `convertToMLIRSparseTensor`.
 #define DECL_CONVERTFROMMLIRSPARSETENSOR(VNAME, V)                             \
   MLIR_CRUNNERUTILS_EXPORT void convertFromMLIRSparseTensor##VNAME(            \
       void *tensor, uint64_t *pRank, uint64_t *pNse, uint64_t **pShape,        \
-      V **pValues, uint64_t **pIndices);
+      V **pValues, uint64_t **pCoordinates);
 MLIR_SPARSETENSOR_FOREVERY_V(DECL_CONVERTFROMMLIRSPARSETENSOR)
 #undef DECL_CONVERTFROMMLIRSPARSETENSOR
 
@@ -242,8 +247,8 @@ MLIR_CRUNNERUTILS_EXPORT void *_mlir_ciface_newSparseTensorFromReader(
     void *p, StridedMemRefType<index_type, 1> *lvlSizesRef,
     StridedMemRefType<DimLevelType, 1> *lvlTypesRef,
     StridedMemRefType<index_type, 1> *lvl2dimRef,
-    StridedMemRefType<index_type, 1> *dim2lvlRef, OverheadType ptrTp,
-    OverheadType indTp, PrimaryType valTp);
+    StridedMemRefType<index_type, 1> *dim2lvlRef, OverheadType posTp,
+    OverheadType crdTp, PrimaryType valTp);
 
 /// Returns the rank of the sparse tensor being read.
 MLIR_CRUNNERUTILS_EXPORT index_type getSparseTensorReaderRank(void *p);
@@ -251,8 +256,8 @@ MLIR_CRUNNERUTILS_EXPORT index_type getSparseTensorReaderRank(void *p);
 /// Returns the is_symmetric bit for the sparse tensor being read.
 MLIR_CRUNNERUTILS_EXPORT bool getSparseTensorReaderIsSymmetric(void *p);
 
-/// Returns the number of non-zero values for the sparse tensor being read.
-MLIR_CRUNNERUTILS_EXPORT index_type getSparseTensorReaderNNZ(void *p);
+/// Returns the number of stored elements for the sparse tensor being read.
+MLIR_CRUNNERUTILS_EXPORT index_type getSparseTensorReaderNSE(void *p);
 
 /// Returns the size of a dimension for the sparse tensor being read.
 MLIR_CRUNNERUTILS_EXPORT index_type getSparseTensorReaderDimSize(void *p,
@@ -278,7 +283,7 @@ MLIR_CRUNNERUTILS_EXPORT void delSparseTensorReader(void *p);
 /// Returns the next element for the sparse tensor being read.
 #define DECL_GETNEXT(VNAME, V)                                                 \
   MLIR_CRUNNERUTILS_EXPORT void _mlir_ciface_getSparseTensorReaderNext##VNAME( \
-      void *p, StridedMemRefType<index_type, 1> *iref,                         \
+      void *p, StridedMemRefType<index_type, 1> *dimCoordsRef,                 \
       StridedMemRefType<V, 0> *vref);
 MLIR_SPARSETENSOR_FOREVERY_V(DECL_GETNEXT)
 #undef DECL_GETNEXT
@@ -306,15 +311,16 @@ MLIR_CRUNNERUTILS_EXPORT void *createSparseTensorWriter(char *filename);
 /// SparseTensorWriter.
 MLIR_CRUNNERUTILS_EXPORT void delSparseTensorWriter(void *p);
 
-/// Outputs the sparse tensor rank, nnz and shape.
+/// Outputs the sparse tensor dim-rank, nse, and dim-shape.
 MLIR_CRUNNERUTILS_EXPORT void _mlir_ciface_outSparseTensorWriterMetaData(
-    void *p, index_type rank, index_type nnz,
-    StridedMemRefType<index_type, 1> *dref);
+    void *p, index_type dimRank, index_type nse,
+    StridedMemRefType<index_type, 1> *dimSizesRef);
 
 /// Outputs an element for the sparse tensor.
 #define DECL_OUTNEXT(VNAME, V)                                                 \
   MLIR_CRUNNERUTILS_EXPORT void _mlir_ciface_outSparseTensorWriterNext##VNAME( \
-      void *p, index_type rank, StridedMemRefType<index_type, 1> *iref,        \
+      void *p, index_type dimRank,                                             \
+      StridedMemRefType<index_type, 1> *dimCoordsRef,                          \
       StridedMemRefType<V, 0> *vref);
 MLIR_SPARSETENSOR_FOREVERY_V(DECL_OUTNEXT)
 #undef DECL_OUTNEXT
diff --git a/mlir/include/mlir/IR/Action.h b/mlir/include/mlir/IR/Action.h
new file mode 100644
index 0000000000000..569d4288f2086
--- /dev/null
+++ b/mlir/include/mlir/IR/Action.h
@@ -0,0 +1,84 @@
+//===- Action.h -  Action Support ---------------------*- C++ -*-=============//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file contains definitions for the action framework. This framework
+// allows for external entities to control certain actions taken by the compiler
+// by registering handler functions.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_IR_ACTION_H
+#define MLIR_IR_ACTION_H
+
+#include "mlir/Support/LogicalResult.h"
+#include "mlir/Support/TypeID.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/Sequence.h"
+#include "llvm/ADT/StringMap.h"
+#include "llvm/Support/TypeName.h"
+#include "llvm/Support/raw_ostream.h"
+#include <functional>
+#include <type_traits>
+
+namespace mlir {
+namespace tracing {
+
+/// An action is a specific action that is to be taken by the compiler,
+/// that can be toggled and controlled by an external user. There are no
+/// constraints on the granularity of an action, it could be as simple as
+/// "perform this fold" and as complex as "run this pass pipeline".
+///
+/// This class represents the base class of the ActionImpl class (see below).
+/// This holds the template-invariant elements of the Action class.
+class Action {
+public:
+  virtual ~Action() = default;
+
+  /// Return the unique action id of this action, use for casting
+  /// functionality.
+  TypeID getActionID() const { return actionID; }
+
+  /// Return a string "tag" which intends to uniquely identify this type of
+  /// action. For example "pass-application" or "pattern-rewrite".
+  virtual StringRef getTag() const = 0;
+
+  virtual void print(raw_ostream &os) const {
+    os << "Action \"" << getTag() << "\"";
+  }
+
+protected:
+  Action(TypeID actionID) : actionID(actionID) {}
+
+  /// The type of the derived action class, used for `isa`/`dyn_cast`.
+  TypeID actionID;
+};
+
+/// CRTP Implementation of an action. This class provides a base class for
+/// implementing specific actions.
+///  Derived classes are expected to provide the following:
+///   * static constexpr StringLiteral tag = "...";
+///     - This method returns a unique string identifier, similar to a command
+///       line flag or DEBUG_TYPE.
+template <typename Derived>
+class ActionImpl : public Action {
+public:
+  ActionImpl() : Action(TypeID::get<Derived>()) {}
+
+  /// Provide classof to allow casting between action types.
+  static bool classof(const Action *action) {
+    return action->getActionID() == TypeID::get<Derived>();
+  }
+
+  /// Forward tag access to the derived class.
+  StringRef getTag() const final { return Derived::tag; }
+};
+
+} // namespace tracing
+} // namespace mlir
+
+#endif // MLIR_IR_ACTION_H
diff --git a/mlir/include/mlir/IR/AffineExpr.h b/mlir/include/mlir/IR/AffineExpr.h
index 1d0688a38003f..74cd5e1e557d5 100644
--- a/mlir/include/mlir/IR/AffineExpr.h
+++ b/mlir/include/mlir/IR/AffineExpr.h
@@ -321,13 +321,6 @@ void bindSymbols(MLIRContext *ctx, AffineExprTy &e, AffineExprTy2 &...exprs) {
   bindSymbols<N + 1, AffineExprTy2 &...>(ctx, exprs...);
 }
 
-template <typename AffineExprTy>
-void bindSymbolsList(MLIRContext *ctx, SmallVectorImpl<AffineExprTy> &exprs) {
-  int idx = 0;
-  for (AffineExprTy &e : exprs)
-    e = getAffineSymbolExpr(idx++, ctx);
-}
-
 } // namespace detail
 
 /// Bind a list of AffineExpr references to DimExpr at positions:
@@ -337,6 +330,13 @@ void bindDims(MLIRContext *ctx, AffineExprTy &...exprs) {
   detail::bindDims<0>(ctx, exprs...);
 }
 
+template <typename AffineExprTy>
+void bindDimsList(MLIRContext *ctx, MutableArrayRef<AffineExprTy> exprs) {
+  int idx = 0;
+  for (AffineExprTy &e : exprs)
+    e = getAffineDimExpr(idx++, ctx);
+}
+
 /// Bind a list of AffineExpr references to SymbolExpr at positions:
 ///   [0 .. sizeof...(exprs)]
 template <typename... AffineExprTy>
@@ -344,6 +344,13 @@ void bindSymbols(MLIRContext *ctx, AffineExprTy &...exprs) {
   detail::bindSymbols<0>(ctx, exprs...);
 }
 
+template <typename AffineExprTy>
+void bindSymbolsList(MLIRContext *ctx, MutableArrayRef<AffineExprTy> exprs) {
+  int idx = 0;
+  for (AffineExprTy &e : exprs)
+    e = getAffineSymbolExpr(idx++, ctx);
+}
+
 } // namespace mlir
 
 namespace llvm {
diff --git a/mlir/include/mlir/IR/ExtensibleDialect.h b/mlir/include/mlir/IR/ExtensibleDialect.h
index 304ff96e0f737..cb8f4fcfa7224 100644
--- a/mlir/include/mlir/IR/ExtensibleDialect.h
+++ b/mlir/include/mlir/IR/ExtensibleDialect.h
@@ -69,6 +69,17 @@ class DynamicAttrDefinition : public SelfOwningTypeID {
   get(StringRef name, ExtensibleDialect *dialect, VerifierFn &&verifier,
       ParserFn &&parser, PrinterFn &&printer);
 
+  /// Sets the verifier function for this attribute. It should emits an error
+  /// message and returns failure if a problem is detected, or returns success
+  /// if everything is ok.
+  void setVerifyFn(VerifierFn &&verify) { verifier = std::move(verify); }
+
+  /// Sets the static hook for parsing this attribute assembly.
+  void setParseFn(ParserFn &&parse) { parser = std::move(parse); }
+
+  /// Sets the static hook for printing this attribute assembly.
+  void setPrintFn(PrinterFn &&print) { printer = std::move(print); }
+
   /// Check that the attribute parameters are valid.
   LogicalResult verify(function_ref<InFlightDiagnostic()> emitError,
                        ArrayRef<Attribute> params) const {
@@ -214,6 +225,17 @@ class DynamicTypeDefinition : public SelfOwningTypeID {
   get(StringRef name, ExtensibleDialect *dialect, VerifierFn &&verifier,
       ParserFn &&parser, PrinterFn &&printer);
 
+  /// Sets the verifier function for this type. It should emits an error
+  /// message and returns failure if a problem is detected, or returns success
+  /// if everything is ok.
+  void setVerifyFn(VerifierFn &&verify) { verifier = std::move(verify); }
+
+  /// Sets the static hook for parsing this type assembly.
+  void setParseFn(ParserFn &&parse) { parser = std::move(parse); }
+
+  /// Sets the static hook for printing this type assembly.
+  void setPrintFn(PrinterFn &&print) { printer = std::move(print); }
+
   /// Check that the type parameters are valid.
   LogicalResult verify(function_ref<InFlightDiagnostic()> emitError,
                        ArrayRef<Attribute> params) const {
diff --git a/mlir/include/mlir/IR/Iterators.h b/mlir/include/mlir/IR/Iterators.h
new file mode 100644
index 0000000000000..c16f7117f3dc9
--- /dev/null
+++ b/mlir/include/mlir/IR/Iterators.h
@@ -0,0 +1,75 @@
+//===- Iterators.h - IR iterators for IR visitors ---------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// The iterators defined in this file can be used together with IR visitors.
+// Note: These iterators cannot be defined in Visitors.h because that would
+// introduce a cyclic header dependency due to Operation.h.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_IR_ITERATORS_H
+#define MLIR_IR_ITERATORS_H
+
+#include "mlir/IR/Operation.h"
+#include "mlir/IR/RegionGraphTraits.h"
+#include "mlir/IR/RegionKindInterface.h"
+#include "mlir/Support/LLVM.h"
+
+#include "llvm/ADT/DepthFirstIterator.h"
+
+namespace mlir {
+/// This iterator enumerates elements in "reverse" order. It is a wrapper around
+/// llvm::reverse.
+struct ReverseIterator {
+  // llvm::reverse uses RangeT::rbegin and RangeT::rend.
+  template <typename RangeT>
+  static constexpr auto makeIterable(RangeT &&range) {
+    return llvm::reverse(
+        ForwardIterator::makeIterable(std::forward<RangeT>(range)));
+  }
+};
+
+/// This iterator enumerates elements according to their dominance relationship.
+/// Operations and regions are enumerated in "forward" order. Blocks are
+/// enumerated according to their successor relationship. Unreachable blocks are
+/// not enumerated.
+///
+/// Note: If `NoGraphRegions` is set to "true", this iterator asserts that each
+/// visited region has SSA dominance. In either case, the ops in such regions
+/// are visited in forward order, but for regions without SSA dominance this
+/// does not guarantee that defining ops are visited before their users.
+template <bool NoGraphRegions = false>
+struct ForwardDominanceIterator {
+  static Block &makeIterable(Block &range) {
+    return ForwardIterator::makeIterable(range);
+  }
+
+  static auto makeIterable(Region &region) {
+    if (NoGraphRegions) {
+      // Only regions with SSA dominance are allowed.
+      assert(mayHaveSSADominance(region) && "graph regions are not allowed");
+    }
+
+    // Create DFS iterator. Blocks are enumerated according to their successor
+    // relationship.
+    Block *null = nullptr;
+    auto it = region.empty()
+                  ? llvm::make_range(llvm::df_end(null), llvm::df_end(null))
+                  : llvm::depth_first(&region.front());
+
+    // Walk API expects Block references instead of pointers.
+    return llvm::make_pointee_range(it);
+  }
+
+  static MutableArrayRef<Region> makeIterable(Operation &range) {
+    return ForwardIterator::makeIterable(range);
+  }
+};
+} // namespace mlir
+
+#endif // MLIR_IR_ITERATORS_H
diff --git a/mlir/include/mlir/IR/MLIRContext.h b/mlir/include/mlir/IR/MLIRContext.h
index 5750dfccbb49f..cc13447d9d584 100644
--- a/mlir/include/mlir/IR/MLIRContext.h
+++ b/mlir/include/mlir/IR/MLIRContext.h
@@ -20,7 +20,9 @@ class ThreadPool;
 } // namespace llvm
 
 namespace mlir {
-class DebugActionManager;
+namespace tracing {
+class Action;
+}
 class DiagnosticEngine;
 class Dialect;
 class DialectRegistry;
@@ -30,6 +32,7 @@ class Location;
 class MLIRContextImpl;
 class RegisteredOperationName;
 class StorageUniquer;
+class IRUnit;
 
 /// MLIRContext is the top-level object for a collection of MLIR operations. It
 /// holds immortal uniqued objects like types, and the tables used to unique
@@ -214,9 +217,6 @@ class MLIRContext {
   /// instances. This should not be used directly.
   StorageUniquer &getAttributeUniquer();
 
-  /// Returns the manager of debug actions within the context.
-  DebugActionManager &getDebugActionManager();
-
   /// These APIs are tracking whether the context will be used in a
   /// multithreading environment: this has no effect other than enabling
   /// assertions on misuses of some APIs.
@@ -239,10 +239,57 @@ class MLIRContext {
   /// (attributes, operations, types, etc.).
   llvm::hash_code getRegistryHash();
 
+  //===--------------------------------------------------------------------===//
+  // Action API
+  //===--------------------------------------------------------------------===//
+
+  /// Signatures for the action handler that can be registered with the context.
+  using HandlerTy =
+      std::function<void(function_ref<void()>, const tracing::Action &)>;
+
+  /// Register a handler for handling actions that are dispatched through this
+  /// context. A nullptr handler can be set to disable a previously set handler.
+  void registerActionHandler(HandlerTy handler);
+
+  /// Return true if a valid ActionHandler is set.
+  bool hasActionHandler();
+
+  /// Dispatch the provided action to the handler if any, or just execute it.
+  void executeAction(function_ref<void()> actionFn,
+                     const tracing::Action &action) {
+    if (LLVM_UNLIKELY(hasActionHandler()))
+      executeActionInternal(actionFn, action);
+    else
+      actionFn();
+  }
+
+  /// Dispatch the provided action to the handler if any, or just execute it.
+  template <typename ActionTy, typename... Args>
+  void executeAction(function_ref<void()> actionFn, Args &&...args) {
+    if (LLVM_UNLIKELY(hasActionHandler()))
+      executeActionInternal<ActionTy, Args...>(actionFn,
+                                               std::forward<Args>(args)...);
+    else
+      actionFn();
+  }
+
 private:
   /// Return true if the given dialect is currently loading.
   bool isDialectLoading(StringRef dialectNamespace);
 
+  /// Internal helper for the dispatch method.
+  void executeActionInternal(function_ref<void()> actionFn,
+                             const tracing::Action &action);
+
+  /// Internal helper for the dispatch method. We get here after checking that
+  /// there is a handler, for the purpose of keeping this code out-of-line. and
+  /// avoid calling the ctor for the Action unnecessarily.
+  template <typename ActionTy, typename... Args>
+  LLVM_ATTRIBUTE_NOINLINE void
+  executeActionInternal(function_ref<void()> actionFn, Args &&...args) {
+    executeActionInternal(actionFn, ActionTy(std::forward<Args>(args)...));
+  }
+
   const std::unique_ptr<MLIRContextImpl> impl;
 
   MLIRContext(const MLIRContext &) = delete;
diff --git a/mlir/include/mlir/IR/PatternMatch.h b/mlir/include/mlir/IR/PatternMatch.h
index ed431badf05b0..9c4790c031201 100644
--- a/mlir/include/mlir/IR/PatternMatch.h
+++ b/mlir/include/mlir/IR/PatternMatch.h
@@ -491,17 +491,36 @@ class RewriterBase : public OpBuilder {
   /// This method erases all operations in a block.
   virtual void eraseBlock(Block *block);
 
-  /// Merge the operations of block 'source' into the end of block 'dest'.
-  /// 'source's predecessors must either be empty or only contain 'dest`.
-  /// 'argValues' is used to replace the block arguments of 'source' after
-  /// merging.
-  virtual void mergeBlocks(Block *source, Block *dest,
-                           ValueRange argValues = std::nullopt);
-
-  // Merge the operations of block 'source' before the operation 'op'. Source
-  // block should not have existing predecessors or successors.
-  void mergeBlockBefore(Block *source, Operation *op,
-                        ValueRange argValues = std::nullopt);
+  /// Inline the operations of block 'source' into block 'dest' before the given
+  /// position. The source block will be deleted and must have no uses.
+  /// 'argValues' is used to replace the block arguments of 'source'.
+  ///
+  /// If the source block is inserted at the end of the dest block, the dest
+  /// block must have no successors. Similarly, if the source block is inserted
+  /// somewhere in the middle (or beginning) of the dest block, the source block
+  /// must have no successors. Otherwise, the resulting IR would have
+  /// unreachable operations.
+  virtual void inlineBlockBefore(Block *source, Block *dest,
+                                 Block::iterator before,
+                                 ValueRange argValues = std::nullopt);
+
+  /// Inline the operations of block 'source' before the operation 'op'. The
+  /// source block will be deleted and must have no uses. 'argValues' is used to
+  /// replace the block arguments of 'source'
+  ///
+  /// The source block must have no successors. Otherwise, the resulting IR
+  /// would have unreachable operations.
+  void inlineBlockBefore(Block *source, Operation *op,
+                         ValueRange argValues = std::nullopt);
+
+  /// Inline the operations of block 'source' into the end of block 'dest'. The
+  /// source block will be deleted and must have no uses. 'argValues' is used to
+  /// replace the block arguments of 'source'
+  ///
+  /// The dest block must have no successors. Otherwise, the resulting IR would
+  /// have unreachable operation.
+  void mergeBlocks(Block *source, Block *dest,
+                   ValueRange argValues = std::nullopt);
 
   /// Split the operations starting at "before" (inclusive) out of the given
   /// block into a new block, and return it.
diff --git a/mlir/include/mlir/IR/RegionKindInterface.h b/mlir/include/mlir/IR/RegionKindInterface.h
index a4b77d65cd654..46bfe717533a8 100644
--- a/mlir/include/mlir/IR/RegionKindInterface.h
+++ b/mlir/include/mlir/IR/RegionKindInterface.h
@@ -38,6 +38,11 @@ class HasOnlyGraphRegion : public TraitBase<ConcreteType, HasOnlyGraphRegion> {
 };
 } // namespace OpTrait
 
+/// Return "true" if the given region may have SSA dominance. This function also
+/// returns "true" in case the owner op is an unregistered op or an op that does
+/// not implement the RegionKindInterface.
+bool mayHaveSSADominance(Region &region);
+
 } // namespace mlir
 
 #include "mlir/IR/RegionKindInterface.h.inc"
diff --git a/mlir/include/mlir/IR/Visitors.h b/mlir/include/mlir/IR/Visitors.h
index 871c19ff5c9b0..fe987741aa907 100644
--- a/mlir/include/mlir/IR/Visitors.h
+++ b/mlir/include/mlir/IR/Visitors.h
@@ -74,17 +74,6 @@ struct ForwardIterator {
   }
 };
 
-/// This iterator enumerates elements in "reverse" order. It is a wrapper around
-/// llvm::reverse.
-struct ReverseIterator {
-  template <typename RangeT>
-  static constexpr auto makeIterable(RangeT &&range) {
-    // llvm::reverse uses RangeT::rbegin and RangeT::rend.
-    return llvm::reverse(
-        ForwardIterator::makeIterable(std::forward<RangeT>(range)));
-  }
-};
-
 /// A utility class to encode the current walk stage for "generic" walkers.
 /// When walking an operation, we can either choose a Pre/Post order walker
 /// which invokes the callback on an operation before/after all its attached
diff --git a/mlir/include/mlir/Interfaces/InferTypeOpInterface.h b/mlir/include/mlir/Interfaces/InferTypeOpInterface.h
index 42f5ec47fcdf3..b63d8b6be6739 100644
--- a/mlir/include/mlir/Interfaces/InferTypeOpInterface.h
+++ b/mlir/include/mlir/Interfaces/InferTypeOpInterface.h
@@ -28,6 +28,12 @@ namespace mlir {
 class ShapedTypeComponents;
 using ReifiedRankedShapedTypeDims = SmallVector<SmallVector<OpFoldResult>>;
 
+/// Reify the shape of the result of an operation (typically in terms of the
+/// shape of its operands).
+LogicalResult
+reifyResultShapes(OpBuilder &b, Operation *op,
+                  ReifiedRankedShapedTypeDims &reifiedReturnShapes);
+
 /// Adaptor class to abstract the differences between whether value is from
 /// a ShapedType or ShapedTypeComponents or DenseIntElementsAttribute.
 class ShapeAdaptor {
diff --git a/mlir/include/mlir/Interfaces/TilingInterface.h b/mlir/include/mlir/Interfaces/TilingInterface.h
index 99cbe21b178ca..ca570490ccf5b 100644
--- a/mlir/include/mlir/Interfaces/TilingInterface.h
+++ b/mlir/include/mlir/Interfaces/TilingInterface.h
@@ -21,6 +21,20 @@
 #include "mlir/Interfaces/ViewLikeInterface.h"
 #include "mlir/Support/LLVM.h"
 
+namespace mlir {
+
+/// Container for result values of tiling.
+/// - `tiledOps` contains operations created by the tiling implementation that
+/// are returned to the caller for further transformations.
+/// - `tiledValues` contains the tiled value corresponding to the result of the
+/// untiled operation.
+struct TilingResult {
+  SmallVector<Operation *> tiledOps;
+  SmallVector<Value> tiledValues;
+};
+
+} // namespace mlir
+
 /// Include the ODS generated interface header files.
 #include "mlir/Interfaces/TilingInterface.h.inc"
 
diff --git a/mlir/include/mlir/Interfaces/TilingInterface.td b/mlir/include/mlir/Interfaces/TilingInterface.td
index 6cc0685bdae41..66382f29c2424 100644
--- a/mlir/include/mlir/Interfaces/TilingInterface.td
+++ b/mlir/include/mlir/Interfaces/TilingInterface.td
@@ -63,7 +63,7 @@ def TilingInterface : OpInterface<"TilingInterface"> {
           The method returns the operation that is the tiled
           implementation.
         }],
-        /*retType=*/"SmallVector<Operation *>",
+        /*retType=*/"FailureOr<TilingResult>",
         /*methodName=*/"getTiledImplementation",
         /*args=*/(ins
             "OpBuilder &":$b,
@@ -119,7 +119,7 @@ def TilingInterface : OpInterface<"TilingInterface"> {
             iteration space).
           - `sizes` provides the size of the tile.
         }],
-        /*retType=*/"FailureOr<Value>",
+        /*retType=*/"FailureOr<TilingResult>",
         /*methodName=*/"generateResultTileValue",
         /*args=*/(ins
           "OpBuilder &":$b,
diff --git a/mlir/include/mlir/Support/DebugAction.h b/mlir/include/mlir/Support/DebugAction.h
deleted file mode 100644
index e1dc25e7792f8..0000000000000
--- a/mlir/include/mlir/Support/DebugAction.h
+++ /dev/null
@@ -1,229 +0,0 @@
-//===- DebugAction.h - Debug Action Support ---------------------*- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// This file contains definitions for the debug action framework. This framework
-// allows for external entities to control certain actions taken by the compiler
-// by registering handler functions. A debug action handler provides the
-// internal implementation for the various queries on a debug action, such as
-// whether it should execute or not.
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef MLIR_SUPPORT_DEBUGACTION_H
-#define MLIR_SUPPORT_DEBUGACTION_H
-
-#include "mlir/Support/LogicalResult.h"
-#include "mlir/Support/TypeID.h"
-#include "llvm/ADT/ArrayRef.h"
-#include "llvm/ADT/Sequence.h"
-#include "llvm/ADT/StringMap.h"
-#include "llvm/Support/TypeName.h"
-#include "llvm/Support/raw_ostream.h"
-#include <functional>
-#include <type_traits>
-
-namespace mlir {
-
-//===----------------------------------------------------------------------===//
-// DebugActionManager
-//===----------------------------------------------------------------------===//
-
-/// This class represents manages debug actions, and orchestrates the
-/// communication between action queries and action handlers. An action handler
-/// is either an action specific handler, i.e. a derived class of
-/// `MyActionType::Handler`, or a generic handler, i.e. a derived class of
-/// `DebugActionManager::GenericHandler`. For more details on action specific
-/// handlers, see the definition of `DebugAction::Handler` below. For more
-/// details on generic handlers, see `DebugActionManager::GenericHandler` below.
-class DebugActionManager {
-public:
-  //===--------------------------------------------------------------------===//
-  // Handlers
-  //===--------------------------------------------------------------------===//
-
-  /// This class represents the base class of a debug action handler.
-  class HandlerBase {
-  public:
-    virtual ~HandlerBase() = default;
-
-    /// Return the unique handler id of this handler, use for casting
-    /// functionality.
-    TypeID getHandlerID() const { return handlerID; }
-
-  protected:
-    HandlerBase(TypeID handlerID) : handlerID(handlerID) {}
-
-    /// The type of the derived handler class. This allows for detecting if a
-    /// handler can handle a given action type.
-    TypeID handlerID;
-  };
-
-  /// This class represents a generic action handler. A generic handler allows
-  /// for handling any action type. Handlers of this type are useful for
-  /// implementing general functionality that doesn't necessarily need to
-  /// interpret the exact action parameters, or can rely on an external
-  /// interpreter (such as the user). Given that these handlers are generic,
-  /// they take a set of opaque parameters that try to map the context of the
-  /// action type in a generic way.
-  class GenericHandler : public HandlerBase {
-  public:
-    GenericHandler() : HandlerBase(TypeID::get<GenericHandler>()) {}
-
-    /// This hook allows for controlling whether an action should execute or
-    /// not. It should return failure if the handler could not process the
-    /// action, passing it to the next registered handler.
-    virtual FailureOr<bool> shouldExecute(StringRef actionTag,
-                                          StringRef description) {
-      return failure();
-    }
-
-    /// Provide classof to allow casting between handler types.
-    static bool classof(const DebugActionManager::HandlerBase *handler) {
-      return handler->getHandlerID() == TypeID::get<GenericHandler>();
-    }
-  };
-
-  /// Register the given action handler with the manager.
-  void registerActionHandler(std::unique_ptr<HandlerBase> handler) {
-    // The manager is always disabled if built without debug.
-#if LLVM_ENABLE_ABI_BREAKING_CHECKS
-    actionHandlers.emplace_back(std::move(handler));
-#endif
-  }
-  template <typename T>
-  void registerActionHandler() {
-    registerActionHandler(std::make_unique<T>());
-  }
-
-  //===--------------------------------------------------------------------===//
-  // Action Queries
-  //===--------------------------------------------------------------------===//
-
-  /// Returns true if the given action type should be executed, false otherwise.
-  /// `Args` are a set of parameters used by handlers of `ActionType` to
-  /// determine if the action should be executed.
-  template <typename ActionType, typename... Args>
-  bool shouldExecute(Args &&...args) {
-    // The manager is always disabled if built without debug.
-#if !LLVM_ENABLE_ABI_BREAKING_CHECKS
-    return true;
-#else
-    // Invoke the `shouldExecute` method on the provided handler.
-    auto shouldExecuteFn = [&](auto *handler, auto &&...handlerParams) {
-      return handler->shouldExecute(
-          std::forward<decltype(handlerParams)>(handlerParams)...);
-    };
-    FailureOr<bool> result = dispatchToHandler<ActionType, bool>(
-        shouldExecuteFn, std::forward<Args>(args)...);
-
-    // If the action wasn't handled, execute the action by default.
-    return succeeded(result) ? *result : true;
-#endif
-  }
-
-private:
-// The manager is always disabled if built without debug.
-#if LLVM_ENABLE_ABI_BREAKING_CHECKS
-  //===--------------------------------------------------------------------===//
-  // Query to Handler Dispatch
-  //===--------------------------------------------------------------------===//
-
-  /// Dispath a given callback on any handlers that are able to process queries
-  /// on the given action type. This method returns failure if no handlers could
-  /// process the action, or success(with a result) if a handler processed the
-  /// action.
-  template <typename ActionType, typename ResultT, typename HandlerCallbackT,
-            typename... Args>
-  FailureOr<ResultT> dispatchToHandler(HandlerCallbackT &&handlerCallback,
-                                       Args &&...args) {
-    static_assert(ActionType::template canHandleWith<Args...>(),
-                  "cannot execute action with the given set of parameters");
-
-    // Process any generic or action specific handlers.
-    // TODO: We currently just pick the first handler that gives us a result,
-    // but in the future we may want to employ a reduction over all of the
-    // values returned.
-    for (std::unique_ptr<HandlerBase> &it : llvm::reverse(actionHandlers)) {
-      FailureOr<ResultT> result = failure();
-      if (auto *handler = dyn_cast<typename ActionType::Handler>(&*it)) {
-        result = handlerCallback(handler, std::forward<Args>(args)...);
-      } else if (auto *genericHandler = dyn_cast<GenericHandler>(&*it)) {
-        result = handlerCallback(genericHandler, ActionType::getTag(),
-                                 ActionType::getDescription());
-      }
-
-      // If the handler succeeded, return the result. Otherwise, try a new
-      // handler.
-      if (succeeded(result))
-        return result;
-    }
-    return failure();
-  }
-
-  /// The set of action handlers that have been registered with the manager.
-  SmallVector<std::unique_ptr<HandlerBase>> actionHandlers;
-#endif
-};
-
-//===----------------------------------------------------------------------===//
-// DebugAction
-//===----------------------------------------------------------------------===//
-
-/// A debug action is a specific action that is to be taken by the compiler,
-/// that can be toggled and controlled by an external user. There are no
-/// constraints on the granularity of an action, it could be as simple as
-/// "perform this fold" and as complex as "run this pass pipeline". Via template
-/// parameters `ParameterTs`, a user may provide the set of argument types that
-/// are provided when handling a query on this action. Derived classes are
-/// expected to provide the following:
-///   * static llvm::StringRef getTag()
-///     - This method returns a unique string identifier, similar to a command
-///       line flag or DEBUG_TYPE.
-///   * static llvm::StringRef getDescription()
-///     - This method returns a short description of what the action represents.
-///
-/// This class provides a handler class that can be derived from to handle
-/// instances of this action. The parameters to its query methods map 1-1 to the
-/// types on the action type.
-template <typename Derived, typename... ParameterTs>
-class DebugAction {
-public:
-  class Handler : public DebugActionManager::HandlerBase {
-  public:
-    Handler() : HandlerBase(TypeID::get<Derived>()) {}
-
-    /// This hook allows for controlling whether an action should execute or
-    /// not. `parameters` correspond to the set of values provided by the
-    /// action as context. It should return failure if the handler could not
-    /// process the action, passing it to the next registered handler.
-    virtual FailureOr<bool> shouldExecute(ParameterTs... parameters) {
-      return failure();
-    }
-
-    /// Provide classof to allow casting between handler types.
-    static bool classof(const DebugActionManager::HandlerBase *handler) {
-      return handler->getHandlerID() == TypeID::get<Derived>();
-    }
-  };
-
-private:
-  /// Returns true if the action can be handled within the given set of
-  /// parameter types.
-  template <typename... CallerParameterTs>
-  static constexpr bool canHandleWith() {
-    return std::is_invocable_v<function_ref<void(ParameterTs...)>,
-                               CallerParameterTs...>;
-  }
-
-  /// Allow access to `canHandleWith`.
-  friend class DebugActionManager;
-};
-
-} // namespace mlir
-
-#endif // MLIR_SUPPORT_DEBUGACTION_H
diff --git a/mlir/include/mlir/Tools/mlir-opt/MlirOptMain.h b/mlir/include/mlir/Tools/mlir-opt/MlirOptMain.h
index caa76934d1625..7cfcff129e100 100644
--- a/mlir/include/mlir/Tools/mlir-opt/MlirOptMain.h
+++ b/mlir/include/mlir/Tools/mlir-opt/MlirOptMain.h
@@ -34,8 +34,21 @@ class PassManager;
 /// This is intended to help building tools like mlir-opt by collecting the
 /// supported options.
 /// The API is fluent, and the options are sorted in alphabetical order below.
+/// The options can be exposed to the LLVM command line by registering them
+/// with `MlirOptMainConfig::registerCLOptions();` and creating a config using
+/// `auto config = MlirOptMainConfig::createFromCLOptions();`.
 class MlirOptMainConfig {
 public:
+  /// Register the options as global LLVM command line options.
+  static void registerCLOptions();
+
+  /// Create a new config with the default set from the CL options.
+  static MlirOptMainConfig createFromCLOptions();
+
+  ///
+  /// Options.
+  ///
+
   /// Allow operation with no registered dialects.
   /// This option is for convenience during testing only and discouraged in
   /// general.
@@ -124,7 +137,7 @@ class MlirOptMainConfig {
   }
   bool shouldVerifyPasses() const { return verifyPassesFlag; }
 
-private:
+protected:
   /// Allow operation with no registered dialects.
   /// This option is for convenience during testing only and discouraged in
   /// general.
diff --git a/mlir/include/mlir/Transforms/DialectConversion.h b/mlir/include/mlir/Transforms/DialectConversion.h
index 229dc016957c6..b5127c099a366 100644
--- a/mlir/include/mlir/Transforms/DialectConversion.h
+++ b/mlir/include/mlir/Transforms/DialectConversion.h
@@ -702,8 +702,10 @@ class ConversionPatternRewriter final : public PatternRewriter,
   /// PatternRewriter hook for splitting a block into two parts.
   Block *splitBlock(Block *block, Block::iterator before) override;
 
-  /// PatternRewriter hook for merging a block into another.
-  void mergeBlocks(Block *source, Block *dest, ValueRange argValues) override;
+  /// PatternRewriter hook for inlining the ops of a block into another block.
+  void inlineBlockBefore(Block *source, Block *dest, Block::iterator before,
+                         ValueRange argValues = std::nullopt) override;
+  using PatternRewriter::inlineBlockBefore;
 
   /// PatternRewriter hook for moving blocks out of a region.
   void inlineRegionBefore(Region &region, Region &parent,
diff --git a/mlir/include/mlir/Transforms/FoldUtils.h b/mlir/include/mlir/Transforms/FoldUtils.h
index ff1083724e59c..a6dc18369e77a 100644
--- a/mlir/include/mlir/Transforms/FoldUtils.h
+++ b/mlir/include/mlir/Transforms/FoldUtils.h
@@ -58,50 +58,6 @@ class OperationFolder {
   /// externally to this OperationFolder. `op` must be a constant op.
   void notifyRemoval(Operation *op);
 
-  /// Create an operation of specific op type with the given builder,
-  /// and immediately try to fold it. This function populates 'results' with
-  /// the results after folding the operation.
-  template <typename OpTy, typename... Args>
-  void create(OpBuilder &builder, SmallVectorImpl<Value> &results,
-              Location location, Args &&...args) {
-    // The op needs to be inserted only if the fold (below) fails, or the number
-    // of results produced by the successful folding is zero (which is treated
-    // as an in-place fold). Using create methods of the builder will insert the
-    // op, so not using it here.
-    OperationState state(location, OpTy::getOperationName());
-    OpTy::build(builder, state, std::forward<Args>(args)...);
-    Operation *op = Operation::create(state);
-
-    if (failed(tryToFold(builder, op, results)) || results.empty()) {
-      builder.insert(op);
-      results.assign(op->result_begin(), op->result_end());
-      return;
-    }
-    op->destroy();
-  }
-
-  /// Overload to create or fold a single result operation.
-  template <typename OpTy, typename... Args>
-  std::enable_if_t<OpTy::template hasTrait<OpTrait::OneResult>(), Value>
-  create(OpBuilder &builder, Location location, Args &&...args) {
-    SmallVector<Value, 1> results;
-    create<OpTy>(builder, results, location, std::forward<Args>(args)...);
-    return results.front();
-  }
-
-  /// Overload to create or fold a zero result operation.
-  template <typename OpTy, typename... Args>
-  std::enable_if_t<OpTy::template hasTrait<OpTrait::ZeroResults>(), OpTy>
-  create(OpBuilder &builder, Location location, Args &&...args) {
-    auto op = builder.create<OpTy>(location, std::forward<Args>(args)...);
-    SmallVector<Value, 0> unused;
-    (void)tryToFold(op.getOperation(), unused);
-
-    // Folding cannot remove a zero-result operation, so for convenience we
-    // continue to return it.
-    return op;
-  }
-
   /// Clear out any constants cached inside of the folder.
   void clear();
 
diff --git a/mlir/include/mlir/Transforms/Passes.h b/mlir/include/mlir/Transforms/Passes.h
index 12986c9bd1ffe..1d95e256b9161 100644
--- a/mlir/include/mlir/Transforms/Passes.h
+++ b/mlir/include/mlir/Transforms/Passes.h
@@ -31,10 +31,11 @@ class GreedyRewriteConfig;
 #define GEN_PASS_DECL_CANONICALIZER
 #define GEN_PASS_DECL_CONTROLFLOWSINK
 #define GEN_PASS_DECL_CSEPASS
+#define GEN_PASS_DECL_INLINER
 #define GEN_PASS_DECL_LOOPINVARIANTCODEMOTION
-#define GEN_PASS_DECL_STRIPDEBUGINFO
+#define GEN_PASS_DECL_PRINTIRPASS
 #define GEN_PASS_DECL_PRINTOPSTATS
-#define GEN_PASS_DECL_INLINER
+#define GEN_PASS_DECL_STRIPDEBUGINFO
 #define GEN_PASS_DECL_SCCP
 #define GEN_PASS_DECL_SYMBOLDCE
 #define GEN_PASS_DECL_SYMBOLPRIVATIZE
@@ -65,7 +66,7 @@ std::unique_ptr<Pass> createControlFlowSinkPass();
 std::unique_ptr<Pass> createCSEPass();
 
 /// Creates a pass to print IR on the debug stream.
-std::unique_ptr<Pass> createPrintIRPass();
+std::unique_ptr<Pass> createPrintIRPass(const PrintIRPassOptions & = {});
 
 /// Creates a pass that generates IR to verify ops at runtime.
 std::unique_ptr<Pass> createGenerateRuntimeVerificationPass();
diff --git a/mlir/include/mlir/Transforms/Passes.td b/mlir/include/mlir/Transforms/Passes.td
index d667ceaa37f47..cecefb35466bc 100644
--- a/mlir/include/mlir/Transforms/Passes.td
+++ b/mlir/include/mlir/Transforms/Passes.td
@@ -92,6 +92,9 @@ def PrintIRPass : Pass<"print-ir"> {
     purposes to inspect the IR at a specific point in the pipeline.
   }];
   let constructor = "mlir::createPrintIRPass()";
+  let options = [
+    Option<"label", "label", "std::string", /*default=*/"", "Label">,
+  ];
 }
 
 def GenerateRuntimeVerification : Pass<"generate-runtime-verification"> {
@@ -109,7 +112,7 @@ def Inliner : Pass<"inline"> {
   let constructor = "mlir::createInlinerPass()";
   let options = [
     Option<"defaultPipelineStr", "default-pipeline", "std::string",
-           /*default=*/"", "The default optimizer pipeline used for callables">,
+           /*default=*/"\"canonicalize\"", "The default optimizer pipeline used for callables">,
     ListOption<"opPipelineList", "op-pipelines", "OpPassManager",
                "Callable operation specific optimizer pipelines (in the form "
                "of `dialect.op(pipeline)`)">,
diff --git a/mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp b/mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp
index 478ff6d6f75ca..d3bc80603ad18 100644
--- a/mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp
+++ b/mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp
@@ -191,13 +191,13 @@ void AbstractSparseDataFlowAnalysis::visitBlock(Block *block) {
             dyn_cast<BranchOpInterface>(predecessor->getTerminator())) {
       SuccessorOperands operands =
           branch.getSuccessorOperands(it.getSuccessorIndex());
-      for (auto &it : llvm::enumerate(argLattices)) {
-        if (Value operand = operands[it.index()]) {
-          join(it.value(), *getLatticeElementFor(block, operand));
+      for (auto [idx, lattice] : llvm::enumerate(argLattices)) {
+        if (Value operand = operands[idx]) {
+          join(lattice, *getLatticeElementFor(block, operand));
         } else {
           // Conservatively consider internally produced arguments as entry
           // points.
-          setAllToEntryStates(it.value());
+          setAllToEntryStates(lattice);
         }
       }
     } else {
diff --git a/mlir/lib/AsmParser/Parser.cpp b/mlir/lib/AsmParser/Parser.cpp
index c44190b7b4a61..46c1ae976d9d4 100644
--- a/mlir/lib/AsmParser/Parser.cpp
+++ b/mlir/lib/AsmParser/Parser.cpp
@@ -389,7 +389,7 @@ ParseResult Parser::codeCompleteDialectOrElidedOpName(SMLoc loc) {
     const char *bufBegin = state.lex.getBufferBegin();
     const char *it = loc.getPointer() - 1;
     for (; it > bufBegin && *it != '\n'; --it)
-      if (!llvm::is_contained(StringRef(" \t\r"), *it))
+      if (!StringRef(" \t\r").contains(*it))
         return true;
     return false;
   };
diff --git a/mlir/lib/Bindings/Python/DialectSparseTensor.cpp b/mlir/lib/Bindings/Python/DialectSparseTensor.cpp
index da44141e28a2a..e84937df89ba1 100644
--- a/mlir/lib/Bindings/Python/DialectSparseTensor.cpp
+++ b/mlir/lib/Bindings/Python/DialectSparseTensor.cpp
@@ -35,27 +35,27 @@ static void populateDialectSparseTensorSubmodule(const py::module &m) {
           [](py::object cls,
              std::vector<MlirSparseTensorDimLevelType> dimLevelTypes,
              std::optional<MlirAffineMap> dimOrdering,
-             std::optional<MlirAffineMap> higherOrdering, int pointerBitWidth,
-             int indexBitWidth, MlirContext context) {
+             std::optional<MlirAffineMap> higherOrdering, int posWidth,
+             int crdWidth, MlirContext context) {
             return cls(mlirSparseTensorEncodingAttrGet(
                 context, dimLevelTypes.size(), dimLevelTypes.data(),
                 dimOrdering ? *dimOrdering : MlirAffineMap{nullptr},
                 higherOrdering ? *higherOrdering : MlirAffineMap{nullptr},
-                pointerBitWidth, indexBitWidth));
+                posWidth, crdWidth));
           },
           py::arg("cls"), py::arg("dim_level_types"), py::arg("dim_ordering"),
-          py::arg("higher_ordering"), py::arg("pointer_bit_width"),
-          py::arg("index_bit_width"), py::arg("context") = py::none(),
+          py::arg("higher_ordering"), py::arg("pos_width"),
+          py::arg("crd_width"), py::arg("context") = py::none(),
           "Gets a sparse_tensor.encoding from parameters.")
       .def_property_readonly(
           "dim_level_types",
           [](MlirAttribute self) {
+            const int lvlRank = mlirSparseTensorEncodingGetLvlRank(self);
             std::vector<MlirSparseTensorDimLevelType> ret;
-            for (int i = 0,
-                     e = mlirSparseTensorEncodingGetNumDimLevelTypes(self);
-                 i < e; ++i)
+            ret.reserve(lvlRank);
+            for (int l = 0; l < lvlRank; ++l)
               ret.push_back(
-                  mlirSparseTensorEncodingAttrGetDimLevelType(self, i));
+                  mlirSparseTensorEncodingAttrGetDimLevelType(self, l));
             return ret;
           })
       .def_property_readonly(
@@ -76,14 +76,10 @@ static void populateDialectSparseTensorSubmodule(const py::module &m) {
               return {};
             return ret;
           })
-      .def_property_readonly(
-          "pointer_bit_width",
-          [](MlirAttribute self) {
-            return mlirSparseTensorEncodingAttrGetPointerBitWidth(self);
-          })
-      .def_property_readonly("index_bit_width", [](MlirAttribute self) {
-        return mlirSparseTensorEncodingAttrGetIndexBitWidth(self);
-      });
+      .def_property_readonly("pos_width",
+                             mlirSparseTensorEncodingAttrGetPosWidth)
+      .def_property_readonly("crd_width",
+                             mlirSparseTensorEncodingAttrGetCrdWidth);
 }
 
 PYBIND11_MODULE(_mlirDialectsSparseTensor, m) {
diff --git a/mlir/lib/Bindings/Python/IRAttributes.cpp b/mlir/lib/Bindings/Python/IRAttributes.cpp
index c8ede8b06e1e3..c59a54b6699a7 100644
--- a/mlir/lib/Bindings/Python/IRAttributes.cpp
+++ b/mlir/lib/Bindings/Python/IRAttributes.cpp
@@ -344,15 +344,10 @@ class PyFloatAttribute : public PyConcreteAttribute<PyFloatAttribute> {
     c.def_static(
         "get",
         [](PyType &type, double value, DefaultingPyLocation loc) {
+          PyMlirContext::ErrorCapture errors(loc->getContext());
           MlirAttribute attr = mlirFloatAttrDoubleGetChecked(loc, type, value);
-          // TODO: Rework error reporting once diagnostic engine is exposed
-          // in C API.
-          if (mlirAttributeIsNull(attr)) {
-            throw SetPyError(PyExc_ValueError,
-                             Twine("invalid '") +
-                                 py::repr(py::cast(type)).cast<std::string>() +
-                                 "' and expected floating point type.");
-          }
+          if (mlirAttributeIsNull(attr))
+            throw MLIRError("Invalid attribute", errors.take());
           return PyFloatAttribute(type.getContext(), attr);
         },
         py::arg("type"), py::arg("value"), py::arg("loc") = py::none(),
@@ -629,8 +624,17 @@ class PyDenseElementsAttribute
       }
     }
     if (bulkLoadElementType) {
-      auto shapedType = mlirRankedTensorTypeGet(
-          shape.size(), shape.data(), *bulkLoadElementType, encodingAttr);
+      MlirType shapedType;
+      if (mlirTypeIsAShaped(*bulkLoadElementType)) {
+        if (explicitShape) {
+          throw std::invalid_argument("Shape can only be specified explicitly "
+                                      "when the type is not a shaped type.");
+        }
+        shapedType = *bulkLoadElementType;
+      } else {
+        shapedType = mlirRankedTensorTypeGet(
+            shape.size(), shape.data(), *bulkLoadElementType, encodingAttr);
+      }
       size_t rawBufferSize = arrayInfo.size * arrayInfo.itemsize;
       MlirAttribute attr = mlirDenseElementsAttrRawBufferGet(
           shapedType, rawBufferSize, arrayInfo.ptr);
diff --git a/mlir/lib/Bindings/Python/IRCore.cpp b/mlir/lib/Bindings/Python/IRCore.cpp
index e03b6470c4dbd..8d637ea2b4abf 100644
--- a/mlir/lib/Bindings/Python/IRCore.cpp
+++ b/mlir/lib/Bindings/Python/IRCore.cpp
@@ -15,6 +15,7 @@
 #include "mlir-c/BuiltinAttributes.h"
 #include "mlir-c/BuiltinTypes.h"
 #include "mlir-c/Debug.h"
+#include "mlir-c/Diagnostics.h"
 #include "mlir-c/IR.h"
 //#include "mlir-c/Registration.h"
 #include "llvm/ADT/ArrayRef.h"
@@ -38,7 +39,7 @@ using llvm::Twine;
 static const char kContextParseTypeDocstring[] =
     R"(Parses the assembly form of a type.
 
-Returns a Type object or raises a ValueError if the type cannot be parsed.
+Returns a Type object or raises an MLIRError if the type cannot be parsed.
 
 See also: https://mlir.llvm.org/docs/LangRef/#type-system
 )";
@@ -58,7 +59,7 @@ static const char kContextGetNameLocationDocString[] =
 static const char kModuleParseDocstring[] =
     R"(Parses a module's assembly format from a string.
 
-Returns a new MlirModule or raises a ValueError if the parsing fails.
+Returns a new MlirModule or raises an MLIRError if the parsing fails.
 
 See also: https://mlir.llvm.org/docs/LangRef/
 )";
@@ -654,6 +655,20 @@ py::object PyMlirContext::attachDiagnosticHandler(py::object callback) {
   return pyHandlerObject;
 }
 
+MlirLogicalResult PyMlirContext::ErrorCapture::handler(MlirDiagnostic diag,
+                                                       void *userData) {
+  auto *self = static_cast<ErrorCapture *>(userData);
+  // Check if the context requested we emit errors instead of capturing them.
+  if (self->ctx->emitErrorDiagnostics)
+    return mlirLogicalResultFailure();
+
+  if (mlirDiagnosticGetSeverity(diag) != MlirDiagnosticError)
+    return mlirLogicalResultFailure();
+
+  self->errors.emplace_back(PyDiagnostic(diag).getInfo());
+  return mlirLogicalResultSuccess();
+}
+
 PyMlirContext &DefaultingPyMlirContext::resolve() {
   PyMlirContext *context = PyThreadContextEntry::getDefaultContext();
   if (!context) {
@@ -870,6 +885,13 @@ py::tuple PyDiagnostic::getNotes() {
   return *materializedNotes;
 }
 
+PyDiagnostic::DiagnosticInfo PyDiagnostic::getInfo() {
+  std::vector<DiagnosticInfo> notes;
+  for (py::handle n : getNotes())
+    notes.emplace_back(n.cast<PyDiagnostic>().getInfo());
+  return {getSeverity(), getLocation(), getMessage(), std::move(notes)};
+}
+
 //------------------------------------------------------------------------------
 // PyDialect, PyDialectDescriptor, PyDialects, PyDialectRegistry
 //------------------------------------------------------------------------------
@@ -1062,13 +1084,12 @@ PyOperationRef PyOperation::createDetached(PyMlirContextRef contextRef,
 PyOperationRef PyOperation::parse(PyMlirContextRef contextRef,
                                   const std::string &sourceStr,
                                   const std::string &sourceName) {
+  PyMlirContext::ErrorCapture errors(contextRef);
   MlirOperation op =
       mlirOperationCreateParse(contextRef->get(), toMlirStringRef(sourceStr),
                                toMlirStringRef(sourceName));
-  // TODO: Include error diagnostic messages in the exception message
   if (mlirOperationIsNull(op))
-    throw py::value_error(
-        "Unable to parse operation assembly (see diagnostics)");
+    throw MLIRError("Unable to parse operation assembly", errors.take());
   return PyOperation::createDetached(std::move(contextRef), op);
 }
 
@@ -1155,6 +1176,14 @@ void PyOperationBase::moveBefore(PyOperationBase &other) {
   operation.parentKeepAlive = otherOp.parentKeepAlive;
 }
 
+bool PyOperationBase::verify() {
+  PyOperation &op = getOperation();
+  PyMlirContext::ErrorCapture errors(op.getContext());
+  if (!mlirOperationVerify(op.get()))
+    throw MLIRError("Verification failed", errors.take());
+  return true;
+}
+
 std::optional<PyOperationRef> PyOperation::getParentOperation() {
   checkValid();
   if (!isAttached())
@@ -2287,6 +2316,16 @@ void mlir::python::populateIRCore(py::module &m) {
         return self.getMessage();
       });
 
+  py::class_<PyDiagnostic::DiagnosticInfo>(m, "DiagnosticInfo",
+                                           py::module_local())
+      .def(py::init<>([](PyDiagnostic diag) { return diag.getInfo(); }))
+      .def_readonly("severity", &PyDiagnostic::DiagnosticInfo::severity)
+      .def_readonly("location", &PyDiagnostic::DiagnosticInfo::location)
+      .def_readonly("message", &PyDiagnostic::DiagnosticInfo::message)
+      .def_readonly("notes", &PyDiagnostic::DiagnosticInfo::notes)
+      .def("__str__",
+           [](PyDiagnostic::DiagnosticInfo &self) { return self.message; });
+
   py::class_<PyDiagnosticHandler>(m, "DiagnosticHandler", py::module_local())
       .def("detach", &PyDiagnosticHandler::detach)
       .def_property_readonly("attached", &PyDiagnosticHandler::isAttached)
@@ -2375,6 +2414,11 @@ void mlir::python::populateIRCore(py::module &m) {
             mlirContextAppendDialectRegistry(self.get(), registry);
           },
           py::arg("registry"))
+      .def_property("emit_error_diagnostics", nullptr,
+                    &PyMlirContext::setEmitErrorDiagnostics,
+                    "Emit error diagnostics to diagnostic handlers. By default "
+                    "error diagnostics are captured and reported through "
+                    "MLIRError exceptions.")
       .def("load_all_available_dialects", [](PyMlirContext &self) {
         mlirContextLoadAllAvailableDialects(self.get());
       });
@@ -2566,16 +2610,12 @@ void mlir::python::populateIRCore(py::module &m) {
       .def(MLIR_PYTHON_CAPI_FACTORY_ATTR, &PyModule::createFromCapsule)
       .def_static(
           "parse",
-          [](const std::string moduleAsm, DefaultingPyMlirContext context) {
+          [](const std::string &moduleAsm, DefaultingPyMlirContext context) {
+            PyMlirContext::ErrorCapture errors(context->getRef());
             MlirModule module = mlirModuleCreateParse(
                 context->get(), toMlirStringRef(moduleAsm));
-            // TODO: Rework error reporting once diagnostic engine is exposed
-            // in C API.
-            if (mlirModuleIsNull(module)) {
-              throw SetPyError(
-                  PyExc_ValueError,
-                  "Unable to parse module assembly (see diagnostics)");
-            }
+            if (mlirModuleIsNull(module))
+              throw MLIRError("Unable to parse module assembly", errors.take());
             return PyModule::forModule(module).releaseObject();
           },
           py::arg("asm"), py::arg("context") = py::none(),
@@ -2724,13 +2764,9 @@ void mlir::python::populateIRCore(py::module &m) {
            py::arg("print_generic_op_form") = false,
            py::arg("use_local_scope") = false,
            py::arg("assume_verified") = false, kOperationGetAsmDocstring)
-      .def(
-          "verify",
-          [](PyOperationBase &self) {
-            return mlirOperationVerify(self.getOperation());
-          },
-          "Verify the operation and return true if it passes, false if it "
-          "fails.")
+      .def("verify", &PyOperationBase::verify,
+           "Verify the operation. Raises MLIRError if verification fails, and "
+           "returns true otherwise.")
       .def("move_after", &PyOperationBase::moveAfter, py::arg("other"),
            "Puts self immediately after the other operation in its parent "
            "block.")
@@ -2833,12 +2869,12 @@ void mlir::python::populateIRCore(py::module &m) {
         // directly.
         std::string clsOpName =
             py::cast<std::string>(cls.attr("OPERATION_NAME"));
-        MlirStringRef parsedOpName =
+        MlirStringRef identifier =
             mlirIdentifierStr(mlirOperationGetName(*parsed.get()));
-        if (!mlirStringRefEqual(parsedOpName, toMlirStringRef(clsOpName)))
-          throw py::value_error(
-              "Expected a '" + clsOpName + "' op, got: '" +
-              std::string(parsedOpName.data, parsedOpName.length) + "'");
+        std::string_view parsedOpName(identifier.data, identifier.length);
+        if (clsOpName != parsedOpName)
+          throw MLIRError(Twine("Expected a '") + clsOpName + "' op, got: '" +
+                          parsedOpName + "'");
         return PyOpView::constructDerived(cls, *parsed.get());
       },
       py::arg("cls"), py::arg("source"), py::kw_only(),
@@ -3071,19 +3107,16 @@ void mlir::python::populateIRCore(py::module &m) {
       .def_static(
           "parse",
           [](std::string attrSpec, DefaultingPyMlirContext context) {
+            PyMlirContext::ErrorCapture errors(context->getRef());
             MlirAttribute type = mlirAttributeParseGet(
                 context->get(), toMlirStringRef(attrSpec));
-            // TODO: Rework error reporting once diagnostic engine is exposed
-            // in C API.
-            if (mlirAttributeIsNull(type)) {
-              throw SetPyError(PyExc_ValueError,
-                               Twine("Unable to parse attribute: '") +
-                                   attrSpec + "'");
-            }
+            if (mlirAttributeIsNull(type))
+              throw MLIRError("Unable to parse attribute", errors.take());
             return PyAttribute(context->getRef(), type);
           },
           py::arg("asm"), py::arg("context") = py::none(),
-          "Parses an attribute from an assembly form")
+          "Parses an attribute from an assembly form. Raises an MLIRError on "
+          "failure.")
       .def_property_readonly(
           "context",
           [](PyAttribute &self) { return self.getContext().getObject(); },
@@ -3182,15 +3215,11 @@ void mlir::python::populateIRCore(py::module &m) {
       .def_static(
           "parse",
           [](std::string typeSpec, DefaultingPyMlirContext context) {
+            PyMlirContext::ErrorCapture errors(context->getRef());
             MlirType type =
                 mlirTypeParseGet(context->get(), toMlirStringRef(typeSpec));
-            // TODO: Rework error reporting once diagnostic engine is exposed
-            // in C API.
-            if (mlirTypeIsNull(type)) {
-              throw SetPyError(PyExc_ValueError,
-                               Twine("Unable to parse type: '") + typeSpec +
-                                   "'");
-            }
+            if (mlirTypeIsNull(type))
+              throw MLIRError("Unable to parse type", errors.take());
             return PyType(context->getRef(), type);
           },
           py::arg("asm"), py::arg("context") = py::none(),
@@ -3342,4 +3371,17 @@ void mlir::python::populateIRCore(py::module &m) {
 
   // Attribute builder getter.
   PyAttrBuilderMap::bind(m);
+
+  py::register_local_exception_translator([](std::exception_ptr p) {
+    // We can't define exceptions with custom fields through pybind, so instead
+    // the exception class is defined in python and imported here.
+    try {
+      if (p)
+        std::rethrow_exception(p);
+    } catch (const MLIRError &e) {
+      py::object obj = py::module_::import(MAKE_MLIR_PYTHON_QUALNAME("ir"))
+                           .attr("MLIRError")(e.message, e.errorDiagnostics);
+      PyErr_SetObject(PyExc_Exception, obj.ptr());
+    }
+  });
 }
diff --git a/mlir/lib/Bindings/Python/IRModule.h b/mlir/lib/Bindings/Python/IRModule.h
index 4aced3639127a..fc236b1c68ecb 100644
--- a/mlir/lib/Bindings/Python/IRModule.h
+++ b/mlir/lib/Bindings/Python/IRModule.h
@@ -221,6 +221,11 @@ class PyMlirContext {
   /// registration object (internally a PyDiagnosticHandler).
   pybind11::object attachDiagnosticHandler(pybind11::object callback);
 
+  /// Controls whether error diagnostics should be propagated to diagnostic
+  /// handlers, instead of being captured by `ErrorCapture`.
+  void setEmitErrorDiagnostics(bool value) { emitErrorDiagnostics = value; }
+  struct ErrorCapture;
+
 private:
   PyMlirContext(MlirContext context);
   // Interns the mapping of live MlirContext::ptr to PyMlirContext instances,
@@ -248,6 +253,8 @@ class PyMlirContext {
       llvm::DenseMap<void *, std::pair<pybind11::handle, PyOperation *>>;
   LiveOperationMap liveOperations;
 
+  bool emitErrorDiagnostics = false;
+
   MlirContext context;
   friend class PyModule;
   friend class PyOperation;
@@ -281,6 +288,34 @@ class BaseContextObject {
   PyMlirContextRef contextRef;
 };
 
+/// Wrapper around an MlirLocation.
+class PyLocation : public BaseContextObject {
+public:
+  PyLocation(PyMlirContextRef contextRef, MlirLocation loc)
+      : BaseContextObject(std::move(contextRef)), loc(loc) {}
+
+  operator MlirLocation() const { return loc; }
+  MlirLocation get() const { return loc; }
+
+  /// Enter and exit the context manager.
+  pybind11::object contextEnter();
+  void contextExit(const pybind11::object &excType,
+                   const pybind11::object &excVal,
+                   const pybind11::object &excTb);
+
+  /// Gets a capsule wrapping the void* within the MlirLocation.
+  pybind11::object getCapsule();
+
+  /// Creates a PyLocation from the MlirLocation wrapped by a capsule.
+  /// Note that PyLocation instances are uniqued, so the returned object
+  /// may be a pre-existing object. Ownership of the underlying MlirLocation
+  /// is taken by calling this function.
+  static PyLocation createFromCapsule(pybind11::object capsule);
+
+private:
+  MlirLocation loc;
+};
+
 /// Python class mirroring the C MlirDiagnostic struct. Note that these structs
 /// are only valid for the duration of a diagnostic callback and attempting
 /// to access them outside of that will raise an exception. This applies to
@@ -295,6 +330,16 @@ class PyDiagnostic {
   pybind11::str getMessage();
   pybind11::tuple getNotes();
 
+  /// Materialized diagnostic information. This is safe to access outside the
+  /// diagnostic callback.
+  struct DiagnosticInfo {
+    MlirDiagnosticSeverity severity;
+    PyLocation location;
+    std::string message;
+    std::vector<DiagnosticInfo> notes;
+  };
+  DiagnosticInfo getInfo();
+
 private:
   MlirDiagnostic diagnostic;
 
@@ -351,6 +396,30 @@ class PyDiagnosticHandler {
   friend class PyMlirContext;
 };
 
+/// RAII object that captures any error diagnostics emitted to the provided
+/// context.
+struct PyMlirContext::ErrorCapture {
+  ErrorCapture(PyMlirContextRef ctx)
+      : ctx(ctx), handlerID(mlirContextAttachDiagnosticHandler(
+                      ctx->get(), handler, /*userData=*/this,
+                      /*deleteUserData=*/nullptr)) {}
+  ~ErrorCapture() {
+    mlirContextDetachDiagnosticHandler(ctx->get(), handlerID);
+    assert(errors.empty() && "unhandled captured errors");
+  }
+
+  std::vector<PyDiagnostic::DiagnosticInfo> take() {
+    return std::move(errors);
+  };
+
+private:
+  PyMlirContextRef ctx;
+  MlirDiagnosticHandlerID handlerID;
+  std::vector<PyDiagnostic::DiagnosticInfo> errors;
+
+  static MlirLogicalResult handler(MlirDiagnostic diag, void *userData);
+};
+
 /// Wrapper around an MlirDialect. This is exported as `DialectDescriptor` in
 /// order to differentiate it from the `Dialect` base class which is extended by
 /// plugins which extend dialect functionality through extension python code.
@@ -416,34 +485,6 @@ class PyDialectRegistry {
   MlirDialectRegistry registry;
 };
 
-/// Wrapper around an MlirLocation.
-class PyLocation : public BaseContextObject {
-public:
-  PyLocation(PyMlirContextRef contextRef, MlirLocation loc)
-      : BaseContextObject(std::move(contextRef)), loc(loc) {}
-
-  operator MlirLocation() const { return loc; }
-  MlirLocation get() const { return loc; }
-
-  /// Enter and exit the context manager.
-  pybind11::object contextEnter();
-  void contextExit(const pybind11::object &excType,
-                   const pybind11::object &excVal,
-                   const pybind11::object &excTb);
-
-  /// Gets a capsule wrapping the void* within the MlirLocation.
-  pybind11::object getCapsule();
-
-  /// Creates a PyLocation from the MlirLocation wrapped by a capsule.
-  /// Note that PyLocation instances are uniqued, so the returned object
-  /// may be a pre-existing object. Ownership of the underlying MlirLocation
-  /// is taken by calling this function.
-  static PyLocation createFromCapsule(pybind11::object capsule);
-
-private:
-  MlirLocation loc;
-};
-
 /// Used in function arguments when None should resolve to the current context
 /// manager set instance.
 class DefaultingPyLocation
@@ -519,6 +560,10 @@ class PyOperationBase {
   void moveAfter(PyOperationBase &other);
   void moveBefore(PyOperationBase &other);
 
+  /// Verify the operation. Throws `MLIRError` if verification fails, and
+  /// returns `true` otherwise.
+  bool verify();
+
   /// Each must provide access to the raw Operation.
   virtual PyOperation &getOperation() = 0;
 };
@@ -1073,6 +1118,16 @@ class PySymbolTable {
   MlirSymbolTable symbolTable;
 };
 
+/// Custom exception that allows access to error diagnostic information. This is
+/// converted to the `ir.MLIRError` python exception when thrown.
+struct MLIRError {
+  MLIRError(llvm::Twine message,
+            std::vector<PyDiagnostic::DiagnosticInfo> &&errorDiagnostics = {})
+      : message(message.str()), errorDiagnostics(std::move(errorDiagnostics)) {}
+  std::string message;
+  std::vector<PyDiagnostic::DiagnosticInfo> errorDiagnostics;
+};
+
 void populateIRAffine(pybind11::module &m);
 void populateIRAttributes(pybind11::module &m);
 void populateIRCore(pybind11::module &m);
diff --git a/mlir/lib/Bindings/Python/IRTypes.cpp b/mlir/lib/Bindings/Python/IRTypes.cpp
index 87ffe593655b2..2166bab902a13 100644
--- a/mlir/lib/Bindings/Python/IRTypes.cpp
+++ b/mlir/lib/Bindings/Python/IRTypes.cpp
@@ -407,17 +407,11 @@ class PyVectorType : public PyConcreteType<PyVectorType, PyShapedType> {
         "get",
         [](std::vector<int64_t> shape, PyType &elementType,
            DefaultingPyLocation loc) {
+          PyMlirContext::ErrorCapture errors(loc->getContext());
           MlirType t = mlirVectorTypeGetChecked(loc, shape.size(), shape.data(),
                                                 elementType);
-          // TODO: Rework error reporting once diagnostic engine is exposed
-          // in C API.
-          if (mlirTypeIsNull(t)) {
-            throw SetPyError(
-                PyExc_ValueError,
-                Twine("invalid '") +
-                    py::repr(py::cast(elementType)).cast<std::string>() +
-                    "' and expected floating point or integer type.");
-          }
+          if (mlirTypeIsNull(t))
+            throw MLIRError("Invalid type", errors.take());
           return PyVectorType(elementType.getContext(), t);
         },
         py::arg("shape"), py::arg("elementType"), py::arg("loc") = py::none(),
@@ -438,20 +432,12 @@ class PyRankedTensorType
         "get",
         [](std::vector<int64_t> shape, PyType &elementType,
            std::optional<PyAttribute> &encodingAttr, DefaultingPyLocation loc) {
+          PyMlirContext::ErrorCapture errors(loc->getContext());
           MlirType t = mlirRankedTensorTypeGetChecked(
               loc, shape.size(), shape.data(), elementType,
               encodingAttr ? encodingAttr->get() : mlirAttributeGetNull());
-          // TODO: Rework error reporting once diagnostic engine is exposed
-          // in C API.
-          if (mlirTypeIsNull(t)) {
-            throw SetPyError(
-                PyExc_ValueError,
-                Twine("invalid '") +
-                    py::repr(py::cast(elementType)).cast<std::string>() +
-                    "' and expected floating point, integer, vector or "
-                    "complex "
-                    "type.");
-          }
+          if (mlirTypeIsNull(t))
+            throw MLIRError("Invalid type", errors.take());
           return PyRankedTensorType(elementType.getContext(), t);
         },
         py::arg("shape"), py::arg("element_type"),
@@ -479,18 +465,10 @@ class PyUnrankedTensorType
     c.def_static(
         "get",
         [](PyType &elementType, DefaultingPyLocation loc) {
+          PyMlirContext::ErrorCapture errors(loc->getContext());
           MlirType t = mlirUnrankedTensorTypeGetChecked(loc, elementType);
-          // TODO: Rework error reporting once diagnostic engine is exposed
-          // in C API.
-          if (mlirTypeIsNull(t)) {
-            throw SetPyError(
-                PyExc_ValueError,
-                Twine("invalid '") +
-                    py::repr(py::cast(elementType)).cast<std::string>() +
-                    "' and expected floating point, integer, vector or "
-                    "complex "
-                    "type.");
-          }
+          if (mlirTypeIsNull(t))
+            throw MLIRError("Invalid type", errors.take());
           return PyUnrankedTensorType(elementType.getContext(), t);
         },
         py::arg("element_type"), py::arg("loc") = py::none(),
@@ -511,23 +489,15 @@ class PyMemRefType : public PyConcreteType<PyMemRefType, PyShapedType> {
          [](std::vector<int64_t> shape, PyType &elementType,
             PyAttribute *layout, PyAttribute *memorySpace,
             DefaultingPyLocation loc) {
+           PyMlirContext::ErrorCapture errors(loc->getContext());
            MlirAttribute layoutAttr = layout ? *layout : mlirAttributeGetNull();
            MlirAttribute memSpaceAttr =
                memorySpace ? *memorySpace : mlirAttributeGetNull();
            MlirType t =
                mlirMemRefTypeGetChecked(loc, elementType, shape.size(),
                                         shape.data(), layoutAttr, memSpaceAttr);
-           // TODO: Rework error reporting once diagnostic engine is exposed
-           // in C API.
-           if (mlirTypeIsNull(t)) {
-             throw SetPyError(
-                 PyExc_ValueError,
-                 Twine("invalid '") +
-                     py::repr(py::cast(elementType)).cast<std::string>() +
-                     "' and expected floating point, integer, vector or "
-                     "complex "
-                     "type.");
-           }
+           if (mlirTypeIsNull(t))
+             throw MLIRError("Invalid type", errors.take());
            return PyMemRefType(elementType.getContext(), t);
          },
          py::arg("shape"), py::arg("element_type"),
@@ -570,23 +540,15 @@ class PyUnrankedMemRefType
          "get",
          [](PyType &elementType, PyAttribute *memorySpace,
             DefaultingPyLocation loc) {
+           PyMlirContext::ErrorCapture errors(loc->getContext());
            MlirAttribute memSpaceAttr = {};
            if (memorySpace)
              memSpaceAttr = *memorySpace;
 
            MlirType t =
                mlirUnrankedMemRefTypeGetChecked(loc, elementType, memSpaceAttr);
-           // TODO: Rework error reporting once diagnostic engine is exposed
-           // in C API.
-           if (mlirTypeIsNull(t)) {
-             throw SetPyError(
-                 PyExc_ValueError,
-                 Twine("invalid '") +
-                     py::repr(py::cast(elementType)).cast<std::string>() +
-                     "' and expected floating point, integer, vector or "
-                     "complex "
-                     "type.");
-           }
+           if (mlirTypeIsNull(t))
+             throw MLIRError("Invalid type", errors.take());
            return PyUnrankedMemRefType(elementType.getContext(), t);
          },
          py::arg("element_type"), py::arg("memory_space"),
diff --git a/mlir/lib/Bindings/Python/Pass.cpp b/mlir/lib/Bindings/Python/Pass.cpp
index 7e90d8be66cb6..79c53084e9260 100644
--- a/mlir/lib/Bindings/Python/Pass.cpp
+++ b/mlir/lib/Bindings/Python/Pass.cpp
@@ -117,15 +117,16 @@ void mlir::python::populatePassManagerSubmodule(py::module &m) {
       .def(
           "run",
           [](PyPassManager &passManager, PyOperationBase &op) {
+            PyMlirContext::ErrorCapture errors(op.getOperation().getContext());
             MlirLogicalResult status = mlirPassManagerRunOnOp(
                 passManager.get(), op.getOperation().get());
             if (mlirLogicalResultIsFailure(status))
-              throw SetPyError(PyExc_RuntimeError,
-                               "Failure while executing pass pipeline.");
+              throw MLIRError("Failure while executing pass pipeline",
+                              errors.take());
           },
           py::arg("operation"),
-          "Run the pass manager on the provided operation, throw a "
-          "RuntimeError on failure.")
+          "Run the pass manager on the provided operation, raising an "
+          "MLIRError on failure.")
       .def(
           "__str__",
           [](PyPassManager &self) {
diff --git a/mlir/lib/Bytecode/Encoding.h b/mlir/lib/Bytecode/Encoding.h
index ee1789fdcaf4c..0072538154806 100644
--- a/mlir/lib/Bytecode/Encoding.h
+++ b/mlir/lib/Bytecode/Encoding.h
@@ -23,8 +23,11 @@ namespace bytecode {
 //===----------------------------------------------------------------------===//
 
 enum {
+  /// The minimum supported version of the bytecode.
+  kMinSupportedVersion = 0,
+
   /// The current bytecode version.
-  kVersion = 0,
+  kVersion = 1,
 
   /// An arbitrary value used to fill alignment padding.
   kAlignmentByte = 0xCB,
@@ -61,8 +64,11 @@ enum ID : uint8_t {
   /// section.
   kResourceOffset = 6,
 
+  /// This section contains the versions of each dialect.
+  kDialectVersions = 7,
+
   /// The total number of section types.
-  kNumSections = 7,
+  kNumSections = 8,
 };
 } // namespace Section
 
diff --git a/mlir/lib/Bytecode/Reader/BytecodeReader.cpp b/mlir/lib/Bytecode/Reader/BytecodeReader.cpp
index 5e71c3a9e5f45..d6f1e18d35ae9 100644
--- a/mlir/lib/Bytecode/Reader/BytecodeReader.cpp
+++ b/mlir/lib/Bytecode/Reader/BytecodeReader.cpp
@@ -47,6 +47,8 @@ static std::string toString(bytecode::Section::ID sectionID) {
     return "Resource (5)";
   case bytecode::Section::kResourceOffset:
     return "ResourceOffset (6)";
+  case bytecode::Section::kDialectVersions:
+    return "DialectVersions (7)";
   default:
     return ("Unknown (" + Twine(static_cast<unsigned>(sectionID)) + ")").str();
   }
@@ -63,6 +65,7 @@ static bool isSectionOptional(bytecode::Section::ID sectionID) {
     return false;
   case bytecode::Section::kResource:
   case bytecode::Section::kResourceOffset:
+  case bytecode::Section::kDialectVersions:
     return true;
   default:
     llvm_unreachable("unknown section ID");
@@ -350,6 +353,13 @@ class StringSectionReader {
     return parseEntry(reader, strings, result, "string");
   }
 
+  /// Parse a shared string from the string section. The shared string is
+  /// encoded using an index to a corresponding string in the string section.
+  LogicalResult parseStringAtIndex(EncodingReader &reader, uint64_t index,
+                                   StringRef &result) {
+    return resolveEntry(reader, strings, index, result, "string");
+  }
+
 private:
   /// The table of strings referenced within the bytecode file.
   SmallVector<StringRef> strings;
@@ -400,31 +410,15 @@ LogicalResult StringSectionReader::initialize(Location fileLoc,
 //===----------------------------------------------------------------------===//
 
 namespace {
+class DialectReader;
+
 /// This struct represents a dialect entry within the bytecode.
 struct BytecodeDialect {
   /// Load the dialect into the provided context if it hasn't been loaded yet.
   /// Returns failure if the dialect couldn't be loaded *and* the provided
   /// context does not allow unregistered dialects. The provided reader is used
   /// for error emission if necessary.
-  LogicalResult load(EncodingReader &reader, MLIRContext *ctx) {
-    if (dialect)
-      return success();
-    Dialect *loadedDialect = ctx->getOrLoadDialect(name);
-    if (!loadedDialect && !ctx->allowsUnregisteredDialects()) {
-      return reader.emitError(
-          "dialect '", name,
-          "' is unknown. If this is intended, please call "
-          "allowUnregisteredDialects() on the MLIRContext, or use "
-          "-allow-unregistered-dialect with the MLIR tool used.");
-    }
-    dialect = loadedDialect;
-
-    // If the dialect was actually loaded, check to see if it has a bytecode
-    // interface.
-    if (loadedDialect)
-      interface = dyn_cast<BytecodeDialectInterface>(loadedDialect);
-    return success();
-  }
+  LogicalResult load(DialectReader &reader, MLIRContext *ctx);
 
   /// Return the loaded dialect, or nullptr if the dialect is unknown. This can
   /// only be called after `load`.
@@ -446,6 +440,12 @@ struct BytecodeDialect {
 
   /// The name of the dialect.
   StringRef name;
+
+  /// A buffer containing the encoding of the dialect version parsed.
+  ArrayRef<uint8_t> versionBuffer;
+
+  /// Lazy loaded dialect version from the handle above.
+  std::unique_ptr<DialectVersion> loadedVersion;
 };
 
 /// This struct represents an operation name entry within the bytecode.
@@ -496,7 +496,7 @@ class ResourceSectionReader {
   initialize(Location fileLoc, const ParserConfig &config,
              MutableArrayRef<BytecodeDialect> dialects,
              StringSectionReader &stringReader, ArrayRef<uint8_t> sectionData,
-             ArrayRef<uint8_t> offsetSectionData,
+             ArrayRef<uint8_t> offsetSectionData, DialectReader &dialectReader,
              const std::shared_ptr<llvm::SourceMgr> &bufferOwnerRef);
 
   /// Parse a dialect resource handle from the resource section.
@@ -643,7 +643,7 @@ LogicalResult ResourceSectionReader::initialize(
     Location fileLoc, const ParserConfig &config,
     MutableArrayRef<BytecodeDialect> dialects,
     StringSectionReader &stringReader, ArrayRef<uint8_t> sectionData,
-    ArrayRef<uint8_t> offsetSectionData,
+    ArrayRef<uint8_t> offsetSectionData, DialectReader &dialectReader,
     const std::shared_ptr<llvm::SourceMgr> &bufferOwnerRef) {
   EncodingReader resourceReader(sectionData, fileLoc);
   EncodingReader offsetReader(offsetSectionData, fileLoc);
@@ -684,7 +684,7 @@ LogicalResult ResourceSectionReader::initialize(
   while (!offsetReader.empty()) {
     BytecodeDialect *dialect;
     if (failed(parseEntry(offsetReader, dialects, dialect, "dialect")) ||
-        failed(dialect->load(resourceReader, ctx)))
+        failed(dialect->load(dialectReader, ctx)))
       return failure();
     Dialect *loadedDialect = dialect->getLoadedDialect();
     if (!loadedDialect) {
@@ -1051,7 +1051,8 @@ template <typename T>
 LogicalResult AttrTypeReader::parseCustomEntry(Entry<T> &entry,
                                                EncodingReader &reader,
                                                StringRef entryType) {
-  if (failed(entry.dialect->load(reader, fileLoc.getContext())))
+  DialectReader dialectReader(*this, stringReader, resourceReader, reader);
+  if (failed(entry.dialect->load(dialectReader, fileLoc.getContext())))
     return failure();
 
   // Ensure that the dialect implements the bytecode interface.
@@ -1060,12 +1061,22 @@ LogicalResult AttrTypeReader::parseCustomEntry(Entry<T> &entry,
                             "' does not implement the bytecode interface");
   }
 
-  // Ask the dialect to parse the entry.
-  DialectReader dialectReader(*this, stringReader, resourceReader, reader);
-  if constexpr (std::is_same_v<T, Type>)
-    entry.entry = entry.dialect->interface->readType(dialectReader);
-  else
-    entry.entry = entry.dialect->interface->readAttribute(dialectReader);
+  // Ask the dialect to parse the entry. If the dialect is versioned, parse
+  // using the versioned encoding readers.
+  if (entry.dialect->loadedVersion.get()) {
+    if constexpr (std::is_same_v<T, Type>)
+      entry.entry = entry.dialect->interface->readType(
+          dialectReader, *entry.dialect->loadedVersion);
+    else
+      entry.entry = entry.dialect->interface->readAttribute(
+          dialectReader, *entry.dialect->loadedVersion);
+
+  } else {
+    if constexpr (std::is_same_v<T, Type>)
+      entry.entry = entry.dialect->interface->readType(dialectReader);
+    else
+      entry.entry = entry.dialect->interface->readAttribute(dialectReader);
+  }
   return success(!!entry.entry);
 }
 
@@ -1122,7 +1133,8 @@ class BytecodeReader {
   // Resource Section
 
   LogicalResult
-  parseResourceSection(std::optional<ArrayRef<uint8_t>> resourceData,
+  parseResourceSection(EncodingReader &reader,
+                       std::optional<ArrayRef<uint8_t>> resourceData,
                        std::optional<ArrayRef<uint8_t>> resourceOffsetData);
 
   //===--------------------------------------------------------------------===//
@@ -1306,7 +1318,7 @@ LogicalResult BytecodeReader::read(llvm::MemoryBufferRef buffer, Block *block) {
 
   // Process the resource section if present.
   if (failed(parseResourceSection(
-          sectionDatas[bytecode::Section::kResource],
+          reader, sectionDatas[bytecode::Section::kResource],
           sectionDatas[bytecode::Section::kResourceOffset])))
     return failure();
 
@@ -1326,7 +1338,8 @@ LogicalResult BytecodeReader::parseVersion(EncodingReader &reader) {
 
   // Validate the bytecode version.
   uint64_t currentVersion = bytecode::kVersion;
-  if (version < currentVersion) {
+  uint64_t minSupportedVersion = bytecode::kMinSupportedVersion;
+  if (version < minSupportedVersion) {
     return reader.emitError("bytecode version ", version,
                             " is older than the current version of ",
                             currentVersion, ", and upgrade is not supported");
@@ -1342,6 +1355,36 @@ LogicalResult BytecodeReader::parseVersion(EncodingReader &reader) {
 //===----------------------------------------------------------------------===//
 // Dialect Section
 
+LogicalResult BytecodeDialect::load(DialectReader &reader, MLIRContext *ctx) {
+  if (dialect)
+    return success();
+  Dialect *loadedDialect = ctx->getOrLoadDialect(name);
+  if (!loadedDialect && !ctx->allowsUnregisteredDialects()) {
+    return reader.emitError("dialect '")
+           << name
+           << "' is unknown. If this is intended, please call "
+              "allowUnregisteredDialects() on the MLIRContext, or use "
+              "-allow-unregistered-dialect with the MLIR tool used.";
+  }
+  dialect = loadedDialect;
+
+  // If the dialect was actually loaded, check to see if it has a bytecode
+  // interface.
+  if (loadedDialect)
+    interface = dyn_cast<BytecodeDialectInterface>(loadedDialect);
+  if (!versionBuffer.empty()) {
+    if (!interface)
+      return reader.emitError("dialect '")
+             << name
+             << "' does not implement the bytecode interface, "
+                "but found a version entry";
+    loadedVersion = interface->readVersion(reader);
+    if (!loadedVersion)
+      return failure();
+  }
+  return success();
+}
+
 LogicalResult
 BytecodeReader::parseDialectSection(ArrayRef<uint8_t> sectionData) {
   EncodingReader sectionReader(sectionData, fileLoc);
@@ -1353,9 +1396,34 @@ BytecodeReader::parseDialectSection(ArrayRef<uint8_t> sectionData) {
   dialects.resize(numDialects);
 
   // Parse each of the dialects.
-  for (uint64_t i = 0; i < numDialects; ++i)
-    if (failed(stringReader.parseString(sectionReader, dialects[i].name)))
+  for (uint64_t i = 0; i < numDialects; ++i) {
+    /// Before version 1, there wasn't any versioning available for dialects,
+    /// and the entryIdx represent the string itself.
+    if (version == 0) {
+      if (failed(stringReader.parseString(sectionReader, dialects[i].name)))
+        return failure();
+      continue;
+    }
+    // Parse ID representing dialect and version.
+    uint64_t dialectNameIdx;
+    bool versionAvailable;
+    if (failed(sectionReader.parseVarIntWithFlag(dialectNameIdx,
+                                                 versionAvailable)))
+      return failure();
+    if (failed(stringReader.parseStringAtIndex(sectionReader, dialectNameIdx,
+                                               dialects[i].name)))
       return failure();
+    if (versionAvailable) {
+      bytecode::Section::ID sectionID;
+      if (failed(
+              sectionReader.parseSection(sectionID, dialects[i].versionBuffer)))
+        return failure();
+      if (sectionID != bytecode::Section::kDialectVersions) {
+        emitError(fileLoc, "expected dialect version section");
+        return failure();
+      }
+    }
+  }
 
   // Parse the operation names, which are grouped by dialect.
   auto parseOpName = [&](BytecodeDialect *dialect) {
@@ -1379,7 +1447,11 @@ FailureOr<OperationName> BytecodeReader::parseOpName(EncodingReader &reader) {
   // Check to see if this operation name has already been resolved. If we
   // haven't, load the dialect and build the operation name.
   if (!opName->opName) {
-    if (failed(opName->dialect->load(reader, getContext())))
+    // Load the dialect and its version.
+    EncodingReader versionReader(opName->dialect->versionBuffer, fileLoc);
+    DialectReader dialectReader(attrTypeReader, stringReader, resourceReader,
+                                versionReader);
+    if (failed(opName->dialect->load(dialectReader, getContext())))
       return failure();
     opName->opName.emplace((opName->dialect->name + "." + opName->name).str(),
                            getContext());
@@ -1391,7 +1463,7 @@ FailureOr<OperationName> BytecodeReader::parseOpName(EncodingReader &reader) {
 // Resource Section
 
 LogicalResult BytecodeReader::parseResourceSection(
-    std::optional<ArrayRef<uint8_t>> resourceData,
+    EncodingReader &reader, std::optional<ArrayRef<uint8_t>> resourceData,
     std::optional<ArrayRef<uint8_t>> resourceOffsetData) {
   // Ensure both sections are either present or not.
   if (resourceData.has_value() != resourceOffsetData.has_value()) {
@@ -1408,9 +1480,11 @@ LogicalResult BytecodeReader::parseResourceSection(
     return success();
 
   // Initialize the resource reader with the resource sections.
+  DialectReader dialectReader(attrTypeReader, stringReader, resourceReader,
+                              reader);
   return resourceReader.initialize(fileLoc, config, dialects, stringReader,
                                    *resourceData, *resourceOffsetData,
-                                   bufferOwnerRef);
+                                   dialectReader, bufferOwnerRef);
 }
 
 //===----------------------------------------------------------------------===//
@@ -1442,6 +1516,18 @@ LogicalResult BytecodeReader::parseIRSection(ArrayRef<uint8_t> sectionData,
         "not all forward unresolved forward operand references");
   }
 
+  // Resolve dialect version.
+  for (const BytecodeDialect &byteCodeDialect : dialects) {
+    // Parsing is complete, give an opportunity to each dialect to visit the
+    // IR and perform upgrades.
+    if (!byteCodeDialect.loadedVersion)
+      continue;
+    if (byteCodeDialect.interface &&
+        failed(byteCodeDialect.interface->upgradeFromVersion(
+            *moduleOp, *byteCodeDialect.loadedVersion)))
+      return failure();
+  }
+
   // Verify that the parsed operations are valid.
   if (config.shouldVerifyAfterParse() && failed(verify(*moduleOp)))
     return failure();
diff --git a/mlir/lib/Bytecode/Writer/BytecodeWriter.cpp b/mlir/lib/Bytecode/Writer/BytecodeWriter.cpp
index 1fd313453fb24..8433f3d84b852 100644
--- a/mlir/lib/Bytecode/Writer/BytecodeWriter.cpp
+++ b/mlir/lib/Bytecode/Writer/BytecodeWriter.cpp
@@ -10,13 +10,10 @@
 #include "../Encoding.h"
 #include "IRNumbering.h"
 #include "mlir/Bytecode/BytecodeImplementation.h"
-#include "mlir/IR/BuiltinDialect.h"
 #include "mlir/IR/OpImplementation.h"
 #include "llvm/ADT/CachedHashString.h"
 #include "llvm/ADT/MapVector.h"
 #include "llvm/ADT/SmallString.h"
-#include "llvm/Support/Debug.h"
-#include <random>
 
 #define DEBUG_TYPE "mlir-bytecode-writer"
 
@@ -261,6 +258,116 @@ class EncodingEmitter {
   unsigned requiredAlignment = 1;
 };
 
+//===----------------------------------------------------------------------===//
+// StringSectionBuilder
+//===----------------------------------------------------------------------===//
+
+namespace {
+/// This class is used to simplify the process of emitting the string section.
+class StringSectionBuilder {
+public:
+  /// Add the given string to the string section, and return the index of the
+  /// string within the section.
+  size_t insert(StringRef str) {
+    auto it = strings.insert({llvm::CachedHashStringRef(str), strings.size()});
+    return it.first->second;
+  }
+
+  /// Write the current set of strings to the given emitter.
+  void write(EncodingEmitter &emitter) {
+    emitter.emitVarInt(strings.size());
+
+    // Emit the sizes in reverse order, so that we don't need to backpatch an
+    // offset to the string data or have a separate section.
+    for (const auto &it : llvm::reverse(strings))
+      emitter.emitVarInt(it.first.size() + 1);
+    // Emit the string data itself.
+    for (const auto &it : strings)
+      emitter.emitNulTerminatedString(it.first.val());
+  }
+
+private:
+  /// A set of strings referenced within the bytecode. The value of the map is
+  /// unused.
+  llvm::MapVector<llvm::CachedHashStringRef, size_t> strings;
+};
+} // namespace
+
+class DialectWriter : public DialectBytecodeWriter {
+public:
+  DialectWriter(EncodingEmitter &emitter, IRNumberingState &numberingState,
+                StringSectionBuilder &stringSection)
+      : emitter(emitter), numberingState(numberingState),
+        stringSection(stringSection) {}
+
+  //===--------------------------------------------------------------------===//
+  // IR
+  //===--------------------------------------------------------------------===//
+
+  void writeAttribute(Attribute attr) override {
+    emitter.emitVarInt(numberingState.getNumber(attr));
+  }
+  void writeType(Type type) override {
+    emitter.emitVarInt(numberingState.getNumber(type));
+  }
+
+  void writeResourceHandle(const AsmDialectResourceHandle &resource) override {
+    emitter.emitVarInt(numberingState.getNumber(resource));
+  }
+
+  //===--------------------------------------------------------------------===//
+  // Primitives
+  //===--------------------------------------------------------------------===//
+
+  void writeVarInt(uint64_t value) override { emitter.emitVarInt(value); }
+
+  void writeSignedVarInt(int64_t value) override {
+    emitter.emitSignedVarInt(value);
+  }
+
+  void writeAPIntWithKnownWidth(const APInt &value) override {
+    size_t bitWidth = value.getBitWidth();
+
+    // If the value is a single byte, just emit it directly without going
+    // through a varint.
+    if (bitWidth <= 8)
+      return emitter.emitByte(value.getLimitedValue());
+
+    // If the value fits within a single varint, emit it directly.
+    if (bitWidth <= 64)
+      return emitter.emitSignedVarInt(value.getLimitedValue());
+
+    // Otherwise, we need to encode a variable number of active words. We use
+    // active words instead of the number of total words under the observation
+    // that smaller values will be more common.
+    unsigned numActiveWords = value.getActiveWords();
+    emitter.emitVarInt(numActiveWords);
+
+    const uint64_t *rawValueData = value.getRawData();
+    for (unsigned i = 0; i < numActiveWords; ++i)
+      emitter.emitSignedVarInt(rawValueData[i]);
+  }
+
+  void writeAPFloatWithKnownSemantics(const APFloat &value) override {
+    writeAPIntWithKnownWidth(value.bitcastToAPInt());
+  }
+
+  void writeOwnedString(StringRef str) override {
+    emitter.emitVarInt(stringSection.insert(str));
+  }
+
+  void writeOwnedBlob(ArrayRef<char> blob) override {
+    emitter.emitVarInt(blob.size());
+    emitter.emitOwnedBlob(ArrayRef<uint8_t>(
+        reinterpret_cast<const uint8_t *>(blob.data()), blob.size()));
+  }
+
+private:
+  EncodingEmitter &emitter;
+  IRNumberingState &numberingState;
+  StringSectionBuilder &stringSection;
+};
+
 /// A simple raw_ostream wrapper around a EncodingEmitter. This removes the need
 /// to go through an intermediate buffer when interacting with code that wants a
 /// raw_ostream.
@@ -307,41 +414,6 @@ void EncodingEmitter::emitMultiByteVarInt(uint64_t value) {
   emitBytes({reinterpret_cast<uint8_t *>(&value), sizeof(value)});
 }
 
-//===----------------------------------------------------------------------===//
-// StringSectionBuilder
-//===----------------------------------------------------------------------===//
-
-namespace {
-/// This class is used to simplify the process of emitting the string section.
-class StringSectionBuilder {
-public:
-  /// Add the given string to the string section, and return the index of the
-  /// string within the section.
-  size_t insert(StringRef str) {
-    auto it = strings.insert({llvm::CachedHashStringRef(str), strings.size()});
-    return it.first->second;
-  }
-
-  /// Write the current set of strings to the given emitter.
-  void write(EncodingEmitter &emitter) {
-    emitter.emitVarInt(strings.size());
-
-    // Emit the sizes in reverse order, so that we don't need to backpatch an
-    // offset to the string data or have a separate section.
-    for (const auto &it : llvm::reverse(strings))
-      emitter.emitVarInt(it.first.size() + 1);
-    // Emit the string data itself.
-    for (const auto &it : strings)
-      emitter.emitNulTerminatedString(it.first.val());
-  }
-
-private:
-  /// A set of strings referenced within the bytecode. The value of the map is
-  /// unused.
-  llvm::MapVector<llvm::CachedHashStringRef, size_t> strings;
-};
-} // namespace
-
 //===----------------------------------------------------------------------===//
 // Bytecode Writer
 //===----------------------------------------------------------------------===//
@@ -464,8 +536,28 @@ void BytecodeWriter::writeDialectSection(EncodingEmitter &emitter) {
   // Emit the referenced dialects.
   auto dialects = numberingState.getDialects();
   dialectEmitter.emitVarInt(llvm::size(dialects));
-  for (DialectNumbering &dialect : dialects)
-    dialectEmitter.emitVarInt(stringSection.insert(dialect.name));
+  for (DialectNumbering &dialect : dialects) {
+    // Write the string section and get the ID.
+    size_t nameID = stringSection.insert(dialect.name);
+
+    // Try writing the version to the versionEmitter.
+    EncodingEmitter versionEmitter;
+    if (dialect.interface) {
+      // The writer used when emitting using a custom bytecode encoding.
+      DialectWriter versionWriter(versionEmitter, numberingState,
+                                  stringSection);
+      dialect.interface->writeVersion(versionWriter);
+    }
+
+    // If the version emitter is empty, version is not available. We can encode
+    // this in the dialect ID, so if there is no version, we don't write the
+    // section.
+    size_t versionAvailable = versionEmitter.size() > 0;
+    dialectEmitter.emitVarIntWithFlag(nameID, versionAvailable);
+    if (versionAvailable)
+      dialectEmitter.emitSection(bytecode::Section::kDialectVersions,
+                                 std::move(versionEmitter));
+  }
 
   // Emit the referenced operation names grouped by dialect.
   auto emitOpName = [&](OpNameNumbering &name) {
@@ -479,83 +571,6 @@ void BytecodeWriter::writeDialectSection(EncodingEmitter &emitter) {
 //===----------------------------------------------------------------------===//
 // Attributes and Types
 
-namespace {
-class DialectWriter : public DialectBytecodeWriter {
-public:
-  DialectWriter(EncodingEmitter &emitter, IRNumberingState &numberingState,
-                StringSectionBuilder &stringSection)
-      : emitter(emitter), numberingState(numberingState),
-        stringSection(stringSection) {}
-
-  //===--------------------------------------------------------------------===//
-  // IR
-  //===--------------------------------------------------------------------===//
-
-  void writeAttribute(Attribute attr) override {
-    emitter.emitVarInt(numberingState.getNumber(attr));
-  }
-  void writeType(Type type) override {
-    emitter.emitVarInt(numberingState.getNumber(type));
-  }
-
-  void writeResourceHandle(const AsmDialectResourceHandle &resource) override {
-    emitter.emitVarInt(numberingState.getNumber(resource));
-  }
-
-  //===--------------------------------------------------------------------===//
-  // Primitives
-  //===--------------------------------------------------------------------===//
-
-  void writeVarInt(uint64_t value) override { emitter.emitVarInt(value); }
-
-  void writeSignedVarInt(int64_t value) override {
-    emitter.emitSignedVarInt(value);
-  }
-
-  void writeAPIntWithKnownWidth(const APInt &value) override {
-    size_t bitWidth = value.getBitWidth();
-
-    // If the value is a single byte, just emit it directly without going
-    // through a varint.
-    if (bitWidth <= 8)
-      return emitter.emitByte(value.getLimitedValue());
-
-    // If the value fits within a single varint, emit it directly.
-    if (bitWidth <= 64)
-      return emitter.emitSignedVarInt(value.getLimitedValue());
-
-    // Otherwise, we need to encode a variable number of active words. We use
-    // active words instead of the number of total words under the observation
-    // that smaller values will be more common.
-    unsigned numActiveWords = value.getActiveWords();
-    emitter.emitVarInt(numActiveWords);
-
-    const uint64_t *rawValueData = value.getRawData();
-    for (unsigned i = 0; i < numActiveWords; ++i)
-      emitter.emitSignedVarInt(rawValueData[i]);
-  }
-
-  void writeAPFloatWithKnownSemantics(const APFloat &value) override {
-    writeAPIntWithKnownWidth(value.bitcastToAPInt());
-  }
-
-  void writeOwnedString(StringRef str) override {
-    emitter.emitVarInt(stringSection.insert(str));
-  }
-
-  void writeOwnedBlob(ArrayRef<char> blob) override {
-    emitter.emitVarInt(blob.size());
-    emitter.emitOwnedBlob(ArrayRef<uint8_t>(
-        reinterpret_cast<const uint8_t *>(blob.data()), blob.size()));
-  }
-
-private:
-  EncodingEmitter &emitter;
-  IRNumberingState &numberingState;
-  StringSectionBuilder &stringSection;
-};
-} // namespace
-
 void BytecodeWriter::writeAttrTypeSection(EncodingEmitter &emitter) {
   EncodingEmitter attrTypeEmitter;
   EncodingEmitter offsetEmitter;
diff --git a/mlir/lib/Bytecode/Writer/IRNumbering.cpp b/mlir/lib/Bytecode/Writer/IRNumbering.cpp
index 1ae9c92c09c4b..06778b9c7c75e 100644
--- a/mlir/lib/Bytecode/Writer/IRNumbering.cpp
+++ b/mlir/lib/Bytecode/Writer/IRNumbering.cpp
@@ -98,8 +98,8 @@ static void groupByDialectPerByte(T range) {
   }
 
   // Assign the entry numbers based on the sort order.
-  for (auto &entry : llvm::enumerate(range))
-    entry.value()->number = entry.index();
+  for (auto [idx, value] : llvm::enumerate(range))
+    value->number = idx;
 }
 
 IRNumberingState::IRNumberingState(Operation *op) {
@@ -132,8 +132,8 @@ IRNumberingState::IRNumberingState(Operation *op) {
   // found, given that the number of dialects on average is small enough to fit
   // within a singly byte (128). If we ever have real world use cases that have
   // a huge number of dialects, this could be made more intelligent.
-  for (auto &it : llvm::enumerate(dialects))
-    it.value().second->number = it.index();
+  for (auto [idx, dialect] : llvm::enumerate(dialects))
+    dialect.second->number = idx;
 
   // Number each of the recorded components within each dialect.
 
@@ -245,7 +245,7 @@ void IRNumberingState::number(Region &region) {
 
   // Number the blocks within this region.
   size_t blockCount = 0;
-  for (auto &it : llvm::enumerate(region)) {
+  for (auto it : llvm::enumerate(region)) {
     blockIDs.try_emplace(&it.value(), it.index());
     number(it.value());
     ++blockCount;
diff --git a/mlir/lib/CAPI/Dialect/SparseTensor.cpp b/mlir/lib/CAPI/Dialect/SparseTensor.cpp
index 831cdd8a486da..1aa6d329d41ca 100644
--- a/mlir/lib/CAPI/Dialect/SparseTensor.cpp
+++ b/mlir/lib/CAPI/Dialect/SparseTensor.cpp
@@ -46,17 +46,17 @@ bool mlirAttributeIsASparseTensorEncodingAttr(MlirAttribute attr) {
 }
 
 MlirAttribute mlirSparseTensorEncodingAttrGet(
-    MlirContext ctx, intptr_t numDimLevelTypes,
+    MlirContext ctx, intptr_t lvlRank,
     MlirSparseTensorDimLevelType const *dimLevelTypes,
-    MlirAffineMap dimOrdering, MlirAffineMap higherOrdering,
-    int pointerBitWidth, int indexBitWidth) {
+    MlirAffineMap dimOrdering, MlirAffineMap higherOrdering, int posWidth,
+    int crdWidth) {
   SmallVector<DimLevelType> cppDimLevelTypes;
-  cppDimLevelTypes.resize(numDimLevelTypes);
-  for (intptr_t i = 0; i < numDimLevelTypes; ++i)
-    cppDimLevelTypes[i] = static_cast<DimLevelType>(dimLevelTypes[i]);
+  cppDimLevelTypes.reserve(lvlRank);
+  for (intptr_t l = 0; l < lvlRank; ++l)
+    cppDimLevelTypes.push_back(static_cast<DimLevelType>(dimLevelTypes[l]));
   return wrap(SparseTensorEncodingAttr::get(
       unwrap(ctx), cppDimLevelTypes, unwrap(dimOrdering),
-      unwrap(higherOrdering), pointerBitWidth, indexBitWidth));
+      unwrap(higherOrdering), posWidth, crdWidth));
 }
 
 MlirAffineMap mlirSparseTensorEncodingAttrGetDimOrdering(MlirAttribute attr) {
@@ -69,7 +69,7 @@ mlirSparseTensorEncodingAttrGetHigherOrdering(MlirAttribute attr) {
       unwrap(attr).cast<SparseTensorEncodingAttr>().getHigherOrdering());
 }
 
-intptr_t mlirSparseTensorEncodingGetNumDimLevelTypes(MlirAttribute attr) {
+intptr_t mlirSparseTensorEncodingGetLvlRank(MlirAttribute attr) {
   return unwrap(attr).cast<SparseTensorEncodingAttr>().getLvlRank();
 }
 
@@ -79,10 +79,10 @@ mlirSparseTensorEncodingAttrGetDimLevelType(MlirAttribute attr, intptr_t lvl) {
       unwrap(attr).cast<SparseTensorEncodingAttr>().getLvlType(lvl));
 }
 
-int mlirSparseTensorEncodingAttrGetPointerBitWidth(MlirAttribute attr) {
-  return unwrap(attr).cast<SparseTensorEncodingAttr>().getPointerBitWidth();
+int mlirSparseTensorEncodingAttrGetPosWidth(MlirAttribute attr) {
+  return unwrap(attr).cast<SparseTensorEncodingAttr>().getPosWidth();
 }
 
-int mlirSparseTensorEncodingAttrGetIndexBitWidth(MlirAttribute attr) {
-  return unwrap(attr).cast<SparseTensorEncodingAttr>().getIndexBitWidth();
+int mlirSparseTensorEncodingAttrGetCrdWidth(MlirAttribute attr) {
+  return unwrap(attr).cast<SparseTensorEncodingAttr>().getCrdWidth();
 }
diff --git a/mlir/lib/CMakeLists.txt b/mlir/lib/CMakeLists.txt
index 3fb13a2c20021..c71664a3f0063 100644
--- a/mlir/lib/CMakeLists.txt
+++ b/mlir/lib/CMakeLists.txt
@@ -5,6 +5,7 @@ add_subdirectory(Analysis)
 add_subdirectory(AsmParser)
 add_subdirectory(Bytecode)
 add_subdirectory(Conversion)
+add_subdirectory(Debug)
 add_subdirectory(Dialect)
 add_subdirectory(IR)
 add_subdirectory(Interfaces)
diff --git a/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp b/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
index 3a1785c0edffa..749f569fa7868 100644
--- a/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
+++ b/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
@@ -200,7 +200,7 @@ struct RawBufferOpLowering : public ConvertOpToLLVMPattern<GpuOp> {
 
     // Indexing (voffset)
     Value voffset;
-    for (auto &pair : llvm::enumerate(adaptor.getIndices())) {
+    for (auto pair : llvm::enumerate(adaptor.getIndices())) {
       size_t i = pair.index();
       Value index = pair.value();
       Value strideOp;
diff --git a/mlir/lib/Conversion/ArithToSPIRV/ArithToSPIRV.cpp b/mlir/lib/Conversion/ArithToSPIRV/ArithToSPIRV.cpp
index be46a06ba53ab..b6ed24490bb0e 100644
--- a/mlir/lib/Conversion/ArithToSPIRV/ArithToSPIRV.cpp
+++ b/mlir/lib/Conversion/ArithToSPIRV/ArithToSPIRV.cpp
@@ -845,7 +845,8 @@ CmpIOpPattern::matchAndRewrite(arith::CmpIOp op, OpAdaptor adaptor,
 #define DISPATCH(cmpPredicate, spirvOp)                                        \
   case cmpPredicate:                                                           \
     if (spirvOp::template hasTrait<OpTrait::spirv::UnsignedOp>() &&            \
-        srcType != dstType && !hasSameBitwidth(srcType, dstType)) {            \
+        !getElementTypeOrSelf(srcType).isIndex() && srcType != dstType &&      \
+        !hasSameBitwidth(srcType, dstType)) {                                  \
       return op.emitError(                                                     \
           "bitwidth emulation is not implemented yet on unsigned op");         \
     }                                                                          \
diff --git a/mlir/lib/Conversion/FuncToLLVM/FuncToLLVM.cpp b/mlir/lib/Conversion/FuncToLLVM/FuncToLLVM.cpp
index 80f0bbbfb46ae..bbfb6281163be 100644
--- a/mlir/lib/Conversion/FuncToLLVM/FuncToLLVM.cpp
+++ b/mlir/lib/Conversion/FuncToLLVM/FuncToLLVM.cpp
@@ -134,7 +134,7 @@ static void wrapForExternalCallers(OpBuilder &rewriter, Location loc,
 
   SmallVector<Value, 8> args;
   size_t argOffset = resultStructType ? 1 : 0;
-  for (auto &[index, argType] : llvm::enumerate(type.getInputs())) {
+  for (auto [index, argType] : llvm::enumerate(type.getInputs())) {
     Value arg = wrapperFuncOp.getArgument(index + argOffset);
     if (auto memrefType = argType.dyn_cast<MemRefType>()) {
       Value loaded = rewriter.create<LLVM::LoadOp>(
@@ -222,11 +222,11 @@ static void wrapExternalFunction(OpBuilder &builder, Location loc,
 
   // Iterate over the inputs of the original function and pack values into
   // memref descriptors if the original type is a memref.
-  for (auto &en : llvm::enumerate(type.getInputs())) {
+  for (Type input : type.getInputs()) {
     Value arg;
     int numToDrop = 1;
-    auto memRefType = en.value().dyn_cast<MemRefType>();
-    auto unrankedMemRefType = en.value().dyn_cast<UnrankedMemRefType>();
+    auto memRefType = input.dyn_cast<MemRefType>();
+    auto unrankedMemRefType = input.dyn_cast<UnrankedMemRefType>();
     if (memRefType || unrankedMemRefType) {
       numToDrop = memRefType
                       ? MemRefDescriptor::getNumUnpackedValues(memRefType)
@@ -677,9 +677,8 @@ struct ReturnOpLowering : public ConvertOpToLLVMPattern<func::ReturnOp> {
         getTypeConverter()->packFunctionResults(op.getOperandTypes());
 
     Value packed = rewriter.create<LLVM::UndefOp>(loc, packedType);
-    for (auto &it : llvm::enumerate(updatedOperands)) {
-      packed = rewriter.create<LLVM::InsertValueOp>(loc, packed, it.value(),
-                                                    it.index());
+    for (auto [idx, operand] : llvm::enumerate(updatedOperands)) {
+      packed = rewriter.create<LLVM::InsertValueOp>(loc, packed, operand, idx);
     }
     rewriter.replaceOpWithNewOp<LLVM::ReturnOp>(op, TypeRange(), packed,
                                                 op->getAttrs());
diff --git a/mlir/lib/Conversion/GPUCommon/GPUToLLVMConversion.cpp b/mlir/lib/Conversion/GPUCommon/GPUToLLVMConversion.cpp
index ec85b99df10ff..55a5e46839558 100644
--- a/mlir/lib/Conversion/GPUCommon/GPUToLLVMConversion.cpp
+++ b/mlir/lib/Conversion/GPUCommon/GPUToLLVMConversion.cpp
@@ -680,8 +680,7 @@ Value ConvertLaunchFuncOpToGpuRuntimeCallPattern::generateParamsArray(
   for (const auto &en : llvm::enumerate(arguments)) {
     Value fieldPtr = builder.create<LLVM::GEPOp>(
         loc, getTypeConverter()->getPointerType(argumentTypes[en.index()]),
-        argumentTypes[en.index()], structPtr,
-        ArrayRef<LLVM::GEPArg>{0, en.index()});
+        structType, structPtr, ArrayRef<LLVM::GEPArg>{0, en.index()});
     builder.create<LLVM::StoreOp>(loc, en.value(), fieldPtr);
     auto elementPtr = builder.create<LLVM::GEPOp>(
         loc, llvmPointerPointerType, llvmPointerType, arrayPtr,
diff --git a/mlir/lib/Conversion/LLVMCommon/TypeConverter.cpp b/mlir/lib/Conversion/LLVMCommon/TypeConverter.cpp
index 48ba15849cc2e..e24be1dfdf6b9 100644
--- a/mlir/lib/Conversion/LLVMCommon/TypeConverter.cpp
+++ b/mlir/lib/Conversion/LLVMCommon/TypeConverter.cpp
@@ -228,12 +228,11 @@ Type LLVMTypeConverter::convertFunctionSignature(
                               ? barePtrFuncArgTypeConverter
                               : structFuncArgTypeConverter;
   // Convert argument types one by one and check for errors.
-  for (auto &en : llvm::enumerate(funcTy.getInputs())) {
-    Type type = en.value();
+  for (auto [idx, type] : llvm::enumerate(funcTy.getInputs())) {
     SmallVector<Type, 8> converted;
     if (failed(funcArgConverter(*this, type, converted)))
       return {};
-    result.addInputs(en.index(), converted);
+    result.addInputs(idx, converted);
   }
 
   // If function does not return anything, create the void result type,
diff --git a/mlir/lib/Conversion/MathToFuncs/MathToFuncs.cpp b/mlir/lib/Conversion/MathToFuncs/MathToFuncs.cpp
index d24a54c7cbc88..c177e50e2c3c9 100644
--- a/mlir/lib/Conversion/MathToFuncs/MathToFuncs.cpp
+++ b/mlir/lib/Conversion/MathToFuncs/MathToFuncs.cpp
@@ -103,7 +103,7 @@ VecOpToScalarOp<Op>::matchAndRewrite(Op op, PatternRewriter &rewriter) const {
       loc, DenseElementsAttr::get(vecType, initValueAttr));
   SmallVector<int64_t> strides = computeStrides(shape);
   for (int64_t linearIndex = 0; linearIndex < numElements; ++linearIndex) {
-    SmallVector<int64_t> positions = delinearize(strides, linearIndex);
+    SmallVector<int64_t> positions = delinearize(linearIndex, strides);
     SmallVector<Value> operands;
     for (Value input : op->getOperands())
       operands.push_back(
diff --git a/mlir/lib/Conversion/MathToLibm/MathToLibm.cpp b/mlir/lib/Conversion/MathToLibm/MathToLibm.cpp
index 1e5b317caa941..d6834441e92bb 100644
--- a/mlir/lib/Conversion/MathToLibm/MathToLibm.cpp
+++ b/mlir/lib/Conversion/MathToLibm/MathToLibm.cpp
@@ -89,7 +89,7 @@ VecOpToScalarOp<Op>::matchAndRewrite(Op op, PatternRewriter &rewriter) const {
                vecType, FloatAttr::get(vecType.getElementType(), 0.0)));
   SmallVector<int64_t> strides = computeStrides(shape);
   for (auto linearIndex = 0; linearIndex < numElements; ++linearIndex) {
-    SmallVector<int64_t> positions = delinearize(strides, linearIndex);
+    SmallVector<int64_t> positions = delinearize(linearIndex, strides);
     SmallVector<Value> operands;
     for (auto input : op->getOperands())
       operands.push_back(
diff --git a/mlir/lib/Conversion/MemRefToLLVM/MemRefToLLVM.cpp b/mlir/lib/Conversion/MemRefToLLVM/MemRefToLLVM.cpp
index 7da7b66ab1f35..2af5a2522566d 100644
--- a/mlir/lib/Conversion/MemRefToLLVM/MemRefToLLVM.cpp
+++ b/mlir/lib/Conversion/MemRefToLLVM/MemRefToLLVM.cpp
@@ -580,15 +580,11 @@ struct GenericAtomicRMWOpLowering
 
     // Split the block into initial, loop, and ending parts.
     auto *initBlock = rewriter.getInsertionBlock();
-    auto *loopBlock = rewriter.createBlock(
-        initBlock->getParent(), std::next(Region::iterator(initBlock)),
-        valueType, loc);
-    auto *endBlock = rewriter.createBlock(
-        loopBlock->getParent(), std::next(Region::iterator(loopBlock)));
+    auto *loopBlock = rewriter.splitBlock(initBlock, Block::iterator(atomicOp));
+    loopBlock->addArgument(valueType, loc);
 
-    // Operations range to be moved to `endBlock`.
-    auto opsToMoveStart = atomicOp->getIterator();
-    auto opsToMoveEnd = initBlock->back().getIterator();
+    auto *endBlock =
+        rewriter.splitBlock(loopBlock, Block::iterator(atomicOp)++);
 
     // Compute the loaded value and branch to the loop block.
     rewriter.setInsertionPointToEnd(initBlock);
@@ -628,30 +624,12 @@ struct GenericAtomicRMWOpLowering
                                     loopBlock, newLoaded);
 
     rewriter.setInsertionPointToEnd(endBlock);
-    moveOpsRange(atomicOp.getResult(), newLoaded, std::next(opsToMoveStart),
-                 std::next(opsToMoveEnd), rewriter);
 
     // The 'result' of the atomic_rmw op is the newly loaded value.
     rewriter.replaceOp(atomicOp, {newLoaded});
 
     return success();
   }
-
-private:
-  // Clones a segment of ops [start, end) and erases the original.
-  void moveOpsRange(ValueRange oldResult, ValueRange newResult,
-                    Block::iterator start, Block::iterator end,
-                    ConversionPatternRewriter &rewriter) const {
-    IRMapping mapping;
-    mapping.map(oldResult, newResult);
-    SmallVector<Operation *, 2> opsToErase;
-    for (auto it = start; it != end; ++it) {
-      rewriter.clone(*it, mapping);
-      opsToErase.push_back(&*it);
-    }
-    for (auto *it : opsToErase)
-      rewriter.eraseOp(it);
-  }
 };
 
 /// Returns the LLVM type of the global variable given the memref type `type`.
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index d27af456f41b9..e88bb05d0b0b9 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -401,22 +401,22 @@ static std::string buildMmaSparseAsmConstraintString(unsigned matASize,
     ss << "=r,";
   for (unsigned i = 0; i < matASize + matBSize + matCSize; i++)
     ss << "r,";
-  // The final two operands are for the sparsity metadata and sparsity selector.
-  ss << "r,r";
+  // The final operand is for the sparsity metadata.
+  // The sparsity selector appears as direct literal.
+  ss << "r";
   ss.flush();
   return str;
 }
 
 /// Returns the string for the `mma.sp.sync` instruction that corresponds to
-/// the give parameters. Note that this function doesn't do any validation,
+/// the given parameters. Note that this function doesn't do any validation,
 /// it's expected that the provided parameters correspond to a valid
 /// instruction.
-static std::string
-buildMmaSparseAsmString(const std::array<int64_t, 3> &shape, unsigned matASize,
-                        unsigned matBSize, unsigned matCSize,
-                        NVVM::MMATypes ptxTypeA, NVVM::MMATypes ptxTypeB,
-                        NVVM::MMATypes ptxTypeC, NVVM::MMATypes ptxTypeD,
-                        std::optional<NVVM::MMAIntOverflow> overflow) {
+static std::string buildMmaSparseAsmString(
+    const std::array<int64_t, 3> &shape, unsigned matASize, unsigned matBSize,
+    unsigned matCSize, NVVM::MMATypes ptxTypeA, NVVM::MMATypes ptxTypeB,
+    NVVM::MMATypes ptxTypeC, NVVM::MMATypes ptxTypeD,
+    std::optional<NVVM::MMAIntOverflow> overflow, unsigned metaDataSelector) {
   auto ptxTypeStr = [](NVVM::MMATypes ptxType) {
     return NVVM::stringifyMMATypes(ptxType);
   };
@@ -442,7 +442,8 @@ buildMmaSparseAsmString(const std::array<int64_t, 3> &shape, unsigned matASize,
     ss << "},";
   }
   ss << "$" << asmArgIdx++ << ",";
-  ss << "$" << asmArgIdx++ << ";";
+  assert(metaDataSelector <= 1);
+  ss << "0x" << metaDataSelector << ";";
   ss.flush();
   return asmStr;
 }
@@ -459,22 +460,21 @@ static FailureOr<LLVM::InlineAsmOp> emitMmaSparseSyncOpAsm(
   auto asmDialectAttr = LLVM::AsmDialectAttr::get(rewriter.getContext(),
                                                   LLVM::AsmDialect::AD_ATT);
 
-  std::string asmStr = buildMmaSparseAsmString(
-      shape, unpackedAData.size(), unpackedB.size(), unpackedC.size(), ptxTypeA,
-      ptxTypeB, ptxTypeC, ptxTypeD, overflow);
-  std::string constraintStr = buildMmaSparseAsmConstraintString(
-      unpackedAData.size(), unpackedB.size(), unpackedC.size());
+  const unsigned matASize = unpackedAData.size();
+  const unsigned matBSize = unpackedB.size();
+  const unsigned matCSize = unpackedC.size();
 
-  Value selectorVal = rewriter.create<LLVM::ConstantOp>(
-      loc, rewriter.getI32Type(), rewriter.getI32IntegerAttr(metadataSelector));
+  std::string asmStr = buildMmaSparseAsmString(
+      shape, matASize, matBSize, matCSize, ptxTypeA, ptxTypeB, ptxTypeC,
+      ptxTypeD, overflow, metadataSelector);
+  std::string constraintStr =
+      buildMmaSparseAsmConstraintString(matASize, matBSize, matCSize);
 
   SmallVector<Value> asmVals;
-  asmVals.reserve(unpackedAData.size() + unpackedB.size() + unpackedC.size() +
-                  2);
+  asmVals.reserve(matASize + matBSize + matCSize + 1);
   for (ArrayRef<Value> args : {unpackedAData, unpackedB, unpackedC})
     llvm::append_range(asmVals, args);
   asmVals.push_back(indexData);
-  asmVals.push_back(selectorVal);
 
   return rewriter.create<LLVM::InlineAsmOp>(loc,
                                             /*resultTypes=*/intrinsicResultType,
diff --git a/mlir/lib/Conversion/OpenMPToLLVM/OpenMPToLLVM.cpp b/mlir/lib/Conversion/OpenMPToLLVM/OpenMPToLLVM.cpp
index 4e38aaf53aa39..6d66149b7dcea 100644
--- a/mlir/lib/Conversion/OpenMPToLLVM/OpenMPToLLVM.cpp
+++ b/mlir/lib/Conversion/OpenMPToLLVM/OpenMPToLLVM.cpp
@@ -121,6 +121,23 @@ struct RegionOpWithVarOperandsConversion : public ConvertOpToLLVMPattern<T> {
   }
 };
 
+template <typename T>
+struct RegionLessOpConversion : public ConvertOpToLLVMPattern<T> {
+  using ConvertOpToLLVMPattern<T>::ConvertOpToLLVMPattern;
+  LogicalResult
+  matchAndRewrite(T curOp, typename T::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    TypeConverter *converter = ConvertToLLVMPattern::getTypeConverter();
+    SmallVector<Type> resTypes;
+    if (failed(converter->convertTypes(curOp->getResultTypes(), resTypes)))
+      return failure();
+
+    rewriter.replaceOpWithNewOp<T>(curOp, resTypes, adaptor.getOperands(),
+                                   curOp->getAttrs());
+    return success();
+  }
+};
+
 struct ReductionOpConversion : public ConvertOpToLLVMPattern<omp::ReductionOp> {
   using ConvertOpToLLVMPattern<omp::ReductionOp>::ConvertOpToLLVMPattern;
   LogicalResult
@@ -147,6 +164,29 @@ struct LegalizeDataOpForLLVMTranslation : public ConvertOpToLLVMPattern<Op> {
     return success();
   }
 };
+
+struct ReductionDeclareOpConversion
+    : public ConvertOpToLLVMPattern<omp::ReductionDeclareOp> {
+  using ConvertOpToLLVMPattern<omp::ReductionDeclareOp>::ConvertOpToLLVMPattern;
+  LogicalResult
+  matchAndRewrite(omp::ReductionDeclareOp curOp, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto newOp = rewriter.create<omp::ReductionDeclareOp>(
+        curOp.getLoc(), TypeRange(), curOp.getSymNameAttr(),
+        TypeAttr::get(this->getTypeConverter()->convertType(
+            curOp.getTypeAttr().getValue())));
+    for (unsigned idx = 0; idx < curOp.getNumRegions(); idx++) {
+      rewriter.inlineRegionBefore(curOp.getRegion(idx), newOp.getRegion(idx),
+                                  newOp.getRegion(idx).end());
+      if (failed(rewriter.convertRegionTypes(&newOp.getRegion(idx),
+                                             *this->getTypeConverter())))
+        return failure();
+    }
+
+    rewriter.eraseOp(curOp);
+    return success();
+  }
+};
 } // namespace
 
 void mlir::configureOpenMPToLLVMConversionLegality(
@@ -160,17 +200,25 @@ void mlir::configureOpenMPToLLVMConversionLegality(
            typeConverter.isLegal(op->getOperandTypes()) &&
            typeConverter.isLegal(op->getResultTypes());
   });
-  target
-      .addDynamicallyLegalOp<mlir::omp::AtomicReadOp, mlir::omp::AtomicWriteOp,
-                             mlir::omp::FlushOp, mlir::omp::ThreadprivateOp,
-                             mlir::omp::EnterDataOp, mlir::omp::ExitDataOp>(
-          [&](Operation *op) {
-            return typeConverter.isLegal(op->getOperandTypes()) &&
-                   typeConverter.isLegal(op->getResultTypes());
-          });
+  target.addDynamicallyLegalOp<mlir::omp::AtomicReadOp,
+                               mlir::omp::AtomicWriteOp, mlir::omp::FlushOp,
+                               mlir::omp::ThreadprivateOp, mlir::omp::YieldOp,
+                               mlir::omp::EnterDataOp, mlir::omp::ExitDataOp>(
+      [&](Operation *op) {
+        return typeConverter.isLegal(op->getOperandTypes()) &&
+               typeConverter.isLegal(op->getResultTypes());
+      });
   target.addDynamicallyLegalOp<mlir::omp::ReductionOp>([&](Operation *op) {
     return typeConverter.isLegal(op->getOperandTypes());
   });
+  target.addDynamicallyLegalOp<mlir::omp::ReductionDeclareOp>(
+      [&](Operation *op) {
+        return typeConverter.isLegal(&op->getRegion(0)) &&
+               typeConverter.isLegal(&op->getRegion(1)) &&
+               typeConverter.isLegal(&op->getRegion(2)) &&
+               typeConverter.isLegal(op->getOperandTypes()) &&
+               typeConverter.isLegal(op->getResultTypes());
+      });
 }
 
 void mlir::populateOpenMPToLLVMConversionPatterns(LLVMTypeConverter &converter,
@@ -179,8 +227,8 @@ void mlir::populateOpenMPToLLVMConversionPatterns(LLVMTypeConverter &converter,
       LegalizeDataOpForLLVMTranslation<omp::DataOp>,
       LegalizeDataOpForLLVMTranslation<omp::EnterDataOp>,
       LegalizeDataOpForLLVMTranslation<omp::ExitDataOp>, ReductionOpConversion,
-      RegionOpConversion<omp::CriticalOp>, RegionOpConversion<omp::MasterOp>,
-      ReductionOpConversion, RegionOpConversion<omp::MasterOp>,
+      ReductionDeclareOpConversion, RegionOpConversion<omp::CriticalOp>,
+      RegionOpConversion<omp::MasterOp>, ReductionOpConversion,
       RegionOpConversion<omp::ParallelOp>, RegionOpConversion<omp::WsLoopOp>,
       RegionOpConversion<omp::SectionsOp>, RegionOpConversion<omp::SectionOp>,
       RegionOpConversion<omp::SimdLoopOp>, RegionOpConversion<omp::SingleOp>,
@@ -189,7 +237,8 @@ void mlir::populateOpenMPToLLVMConversionPatterns(LLVMTypeConverter &converter,
       RegionLessOpWithVarOperandsConversion<omp::AtomicWriteOp>,
       RegionOpWithVarOperandsConversion<omp::AtomicUpdateOp>,
       RegionLessOpWithVarOperandsConversion<omp::FlushOp>,
-      RegionLessOpWithVarOperandsConversion<omp::ThreadprivateOp>>(converter);
+      RegionLessOpWithVarOperandsConversion<omp::ThreadprivateOp>,
+      RegionLessOpConversion<omp::YieldOp>>(converter);
 }
 
 namespace {
diff --git a/mlir/lib/Conversion/PDLToPDLInterp/PredicateTree.cpp b/mlir/lib/Conversion/PDLToPDLInterp/PredicateTree.cpp
index 076e2a247d1a7..034291440ad2c 100644
--- a/mlir/lib/Conversion/PDLToPDLInterp/PredicateTree.cpp
+++ b/mlir/lib/Conversion/PDLToPDLInterp/PredicateTree.cpp
@@ -185,20 +185,20 @@ getTreePredicates(std::vector<PositionalPredicate> &predList, Value val,
   if (types.size() == 1 && types[0].getType().isa<pdl::RangeType>()) {
     getTreePredicates(predList, types.front(), builder, inputs,
                       builder.getType(builder.getAllResults(opPos)));
-  } else {
-    bool foundVariableLength = false;
-    for (auto &resultIt : llvm::enumerate(types)) {
-      bool isVariadic = resultIt.value().getType().isa<pdl::RangeType>();
-      foundVariableLength |= isVariadic;
+    return;
+  }
 
-      auto *resultPos =
-          foundVariableLength
-              ? builder.getResultGroup(pos, resultIt.index(), isVariadic)
-              : builder.getResult(pos, resultIt.index());
-      predList.emplace_back(resultPos, builder.getIsNotNull());
-      getTreePredicates(predList, resultIt.value(), builder, inputs,
-                        builder.getType(resultPos));
-    }
+  bool foundVariableLength = false;
+  for (auto [idx, typeValue] : llvm::enumerate(types)) {
+    bool isVariadic = typeValue.getType().isa<pdl::RangeType>();
+    foundVariableLength |= isVariadic;
+
+    auto *resultPos = foundVariableLength
+                          ? builder.getResultGroup(pos, idx, isVariadic)
+                          : builder.getResult(pos, idx);
+    predList.emplace_back(resultPos, builder.getIsNotNull());
+    getTreePredicates(predList, typeValue, builder, inputs,
+                      builder.getType(resultPos));
   }
 }
 
@@ -815,7 +815,7 @@ static void propagatePattern(std::unique_ptr<MatcherNode> &node,
         std::make_unique<SuccessNode>(list.pattern, list.root, std::move(node));
 
     // If the pattern doesn't contain this predicate, ignore it.
-  } else if (list.predicates.find(*current) == list.predicates.end()) {
+  } else if (!list.predicates.contains(*current)) {
     propagatePattern(node, list, std::next(current), end);
 
     // If the current matcher node is invalid, create a new one for this
diff --git a/mlir/lib/Conversion/SCFToControlFlow/SCFToControlFlow.cpp b/mlir/lib/Conversion/SCFToControlFlow/SCFToControlFlow.cpp
index 2f9ad2b7ad9ba..57ba18bd53f1b 100644
--- a/mlir/lib/Conversion/SCFToControlFlow/SCFToControlFlow.cpp
+++ b/mlir/lib/Conversion/SCFToControlFlow/SCFToControlFlow.cpp
@@ -506,7 +506,7 @@ ParallelLowering::matchAndRewrite(ParallelOp parallelOp,
     Value arg = iterArgs[yieldOperands.size()];
     yieldOperands.push_back(reduceBlock.getTerminator()->getOperand(0));
     rewriter.eraseOp(reduceBlock.getTerminator());
-    rewriter.mergeBlockBefore(&reduceBlock, &op, {arg, reduce.getOperand()});
+    rewriter.inlineBlockBefore(&reduceBlock, &op, {arg, reduce.getOperand()});
     rewriter.eraseOp(reduce);
   }
 
@@ -516,8 +516,8 @@ ParallelLowering::matchAndRewrite(ParallelOp parallelOp,
   if (newBody->empty())
     rewriter.mergeBlocks(parallelOp.getBody(), newBody, ivs);
   else
-    rewriter.mergeBlockBefore(parallelOp.getBody(), newBody->getTerminator(),
-                              ivs);
+    rewriter.inlineBlockBefore(parallelOp.getBody(), newBody->getTerminator(),
+                               ivs);
 
   // Finally, create the terminator if required (for loops with no results, it
   // has been already created in loop construction).
diff --git a/mlir/lib/Conversion/SCFToGPU/SCFToGPU.cpp b/mlir/lib/Conversion/SCFToGPU/SCFToGPU.cpp
index 03cf661c8899e..54ae158983fa7 100644
--- a/mlir/lib/Conversion/SCFToGPU/SCFToGPU.cpp
+++ b/mlir/lib/Conversion/SCFToGPU/SCFToGPU.cpp
@@ -513,7 +513,7 @@ static LogicalResult processParallelLoop(
                   ensureLaunchIndependent(cloningMap.lookupOrDefault(step))});
           // todo(herhut,ravishankarm): Update the behavior of setMappingAttr
           // when this condition is relaxed.
-          if (bounds.find(processor) != bounds.end()) {
+          if (bounds.contains(processor)) {
             return rewriter.notifyMatchFailure(
                 parallelOp, "cannot redefine the bound for processor " +
                                 Twine(static_cast<int64_t>(processor)));
diff --git a/mlir/lib/Conversion/SCFToOpenMP/SCFToOpenMP.cpp b/mlir/lib/Conversion/SCFToOpenMP/SCFToOpenMP.cpp
index f843c3d815ab0..e91cd0c24171e 100644
--- a/mlir/lib/Conversion/SCFToOpenMP/SCFToOpenMP.cpp
+++ b/mlir/lib/Conversion/SCFToOpenMP/SCFToOpenMP.cpp
@@ -310,6 +310,10 @@ static omp::ReductionDeclareOp declareReduction(PatternRewriter &builder,
     return createDecl(builder, symbolTable, reduce,
                       builder.getFloatAttr(type, 1.0));
   }
+  if (matchSimpleReduction<arith::MulIOp, LLVM::MulOp>(reduction)) {
+    return createDecl(builder, symbolTable, reduce,
+                      builder.getIntegerAttr(type, 1));
+  }
 
   // Match select-based min/max reductions.
   bool isMin;
diff --git a/mlir/lib/Conversion/SCFToSPIRV/SCFToSPIRV.cpp b/mlir/lib/Conversion/SCFToSPIRV/SCFToSPIRV.cpp
index 00062ef4a7a91..81a6378fc7e49 100644
--- a/mlir/lib/Conversion/SCFToSPIRV/SCFToSPIRV.cpp
+++ b/mlir/lib/Conversion/SCFToSPIRV/SCFToSPIRV.cpp
@@ -41,16 +41,57 @@ struct ScfToSPIRVContextImpl {
 /// StoreOp cannot be created earlier as they may use a different type than
 /// yield operands.
 ScfToSPIRVContext::ScfToSPIRVContext() {
-  impl = std::make_unique<ScfToSPIRVContextImpl>();
+  impl = std::make_unique<::ScfToSPIRVContextImpl>();
 }
 
 ScfToSPIRVContext::~ScfToSPIRVContext() = default;
 
+namespace {
+
 //===----------------------------------------------------------------------===//
-// Pattern Declarations
+// Helper Functions
+//===----------------------------------------------------------------------===//
+
+/// Replaces SCF op outputs with SPIR-V variable loads.
+/// We create VariableOp to handle the results value of the control flow region.
+/// spirv.mlir.loop/spirv.mlir.selection currently don't yield value. Right
+/// after the loop we load the value from the allocation and use it as the SCF
+/// op result.
+template <typename ScfOp, typename OpTy>
+void replaceSCFOutputValue(ScfOp scfOp, OpTy newOp,
+                           ConversionPatternRewriter &rewriter,
+                           ScfToSPIRVContextImpl *scfToSPIRVContext,
+                           ArrayRef<Type> returnTypes) {
+
+  Location loc = scfOp.getLoc();
+  auto &allocas = scfToSPIRVContext->outputVars[newOp];
+  // Clearing the allocas is necessary in case a dialect conversion path failed
+  // previously, and this is the second attempt of this conversion.
+  allocas.clear();
+  SmallVector<Value, 8> resultValue;
+  for (Type convertedType : returnTypes) {
+    auto pointerType =
+        spirv::PointerType::get(convertedType, spirv::StorageClass::Function);
+    rewriter.setInsertionPoint(newOp);
+    auto alloc = rewriter.create<spirv::VariableOp>(
+        loc, pointerType, spirv::StorageClass::Function,
+        /*initializer=*/nullptr);
+    allocas.push_back(alloc);
+    rewriter.setInsertionPointAfter(newOp);
+    Value loadResult = rewriter.create<spirv::LoadOp>(loc, alloc);
+    resultValue.push_back(loadResult);
+  }
+  rewriter.replaceOp(scfOp, resultValue);
+}
+
+Region::iterator getBlockIt(Region &region, unsigned index) {
+  return std::next(region.begin(), index);
+}
+
+//===----------------------------------------------------------------------===//
+// Conversion Patterns
 //===----------------------------------------------------------------------===//
 
-namespace {
 /// Common class for all vector to GPU patterns.
 template <typename OpTy>
 class SCFToSPIRVPattern : public OpConversionPattern<OpTy> {
@@ -79,356 +120,316 @@ class SCFToSPIRVPattern : public OpConversionPattern<OpTy> {
   SPIRVTypeConverter &typeConverter;
 };
 
+//===----------------------------------------------------------------------===//
+// scf::ForOp
+//===----------------------------------------------------------------------===//
+
 /// Pattern to convert a scf::ForOp within kernel functions into spirv::LoopOp.
-class ForOpConversion final : public SCFToSPIRVPattern<scf::ForOp> {
-public:
-  using SCFToSPIRVPattern<scf::ForOp>::SCFToSPIRVPattern;
+struct ForOpConversion final : SCFToSPIRVPattern<scf::ForOp> {
+  using SCFToSPIRVPattern::SCFToSPIRVPattern;
 
   LogicalResult
   matchAndRewrite(scf::ForOp forOp, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override;
+                  ConversionPatternRewriter &rewriter) const override {
+    // scf::ForOp can be lowered to the structured control flow represented by
+    // spirv::LoopOp by making the continue block of the spirv::LoopOp the loop
+    // latch and the merge block the exit block. The resulting spirv::LoopOp has
+    // a single back edge from the continue to header block, and a single exit
+    // from header to merge.
+    auto loc = forOp.getLoc();
+    auto loopOp = rewriter.create<spirv::LoopOp>(loc, spirv::LoopControl::None);
+    loopOp.addEntryAndMergeBlock();
+
+    OpBuilder::InsertionGuard guard(rewriter);
+    // Create the block for the header.
+    auto *header = new Block();
+    // Insert the header.
+    loopOp.getBody().getBlocks().insert(getBlockIt(loopOp.getBody(), 1),
+                                        header);
+
+    // Create the new induction variable to use.
+    Value adapLowerBound = adaptor.getLowerBound();
+    BlockArgument newIndVar =
+        header->addArgument(adapLowerBound.getType(), adapLowerBound.getLoc());
+    for (Value arg : adaptor.getInitArgs())
+      header->addArgument(arg.getType(), arg.getLoc());
+    Block *body = forOp.getBody();
+
+    // Apply signature conversion to the body of the forOp. It has a single
+    // block, with argument which is the induction variable. That has to be
+    // replaced with the new induction variable.
+    TypeConverter::SignatureConversion signatureConverter(
+        body->getNumArguments());
+    signatureConverter.remapInput(0, newIndVar);
+    for (unsigned i = 1, e = body->getNumArguments(); i < e; i++)
+      signatureConverter.remapInput(i, header->getArgument(i));
+    body = rewriter.applySignatureConversion(&forOp.getLoopBody(),
+                                             signatureConverter);
+
+    // Move the blocks from the forOp into the loopOp. This is the body of the
+    // loopOp.
+    rewriter.inlineRegionBefore(forOp->getRegion(0), loopOp.getBody(),
+                                getBlockIt(loopOp.getBody(), 2));
+
+    SmallVector<Value, 8> args(1, adaptor.getLowerBound());
+    args.append(adaptor.getInitArgs().begin(), adaptor.getInitArgs().end());
+    // Branch into it from the entry.
+    rewriter.setInsertionPointToEnd(&(loopOp.getBody().front()));
+    rewriter.create<spirv::BranchOp>(loc, header, args);
+
+    // Generate the rest of the loop header.
+    rewriter.setInsertionPointToEnd(header);
+    auto *mergeBlock = loopOp.getMergeBlock();
+    auto cmpOp = rewriter.create<spirv::SLessThanOp>(
+        loc, rewriter.getI1Type(), newIndVar, adaptor.getUpperBound());
+
+    rewriter.create<spirv::BranchConditionalOp>(
+        loc, cmpOp, body, ArrayRef<Value>(), mergeBlock, ArrayRef<Value>());
+
+    // Generate instructions to increment the step of the induction variable and
+    // branch to the header.
+    Block *continueBlock = loopOp.getContinueBlock();
+    rewriter.setInsertionPointToEnd(continueBlock);
+
+    // Add the step to the induction variable and branch to the header.
+    Value updatedIndVar = rewriter.create<spirv::IAddOp>(
+        loc, newIndVar.getType(), newIndVar, adaptor.getStep());
+    rewriter.create<spirv::BranchOp>(loc, header, updatedIndVar);
+
+    // Infer the return types from the init operands. Vector type may get
+    // converted to CooperativeMatrix or to Vector type, to avoid having complex
+    // extra logic to figure out the right type we just infer it from the Init
+    // operands.
+    SmallVector<Type, 8> initTypes;
+    for (auto arg : adaptor.getInitArgs())
+      initTypes.push_back(arg.getType());
+    replaceSCFOutputValue(forOp, loopOp, rewriter, scfToSPIRVContext,
+                          initTypes);
+    return success();
+  }
 };
 
+//===----------------------------------------------------------------------===//
+// scf::IfOp
+//===----------------------------------------------------------------------===//
+
 /// Pattern to convert a scf::IfOp within kernel functions into
 /// spirv::SelectionOp.
-class IfOpConversion final : public SCFToSPIRVPattern<scf::IfOp> {
-public:
-  using SCFToSPIRVPattern<scf::IfOp>::SCFToSPIRVPattern;
+struct IfOpConversion : SCFToSPIRVPattern<scf::IfOp> {
+  using SCFToSPIRVPattern::SCFToSPIRVPattern;
 
   LogicalResult
   matchAndRewrite(scf::IfOp ifOp, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override;
-};
-
-class TerminatorOpConversion final : public SCFToSPIRVPattern<scf::YieldOp> {
-public:
-  using SCFToSPIRVPattern<scf::YieldOp>::SCFToSPIRVPattern;
+                  ConversionPatternRewriter &rewriter) const override {
+    // When lowering `scf::IfOp` we explicitly create a selection header block
+    // before the control flow diverges and a merge block where control flow
+    // subsequently converges.
+    auto loc = ifOp.getLoc();
+
+    // Create `spirv.selection` operation, selection header block and merge
+    // block.
+    auto selectionOp =
+        rewriter.create<spirv::SelectionOp>(loc, spirv::SelectionControl::None);
+    auto *mergeBlock = rewriter.createBlock(&selectionOp.getBody(),
+                                            selectionOp.getBody().end());
+    rewriter.create<spirv::MergeOp>(loc);
+
+    OpBuilder::InsertionGuard guard(rewriter);
+    auto *selectionHeaderBlock =
+        rewriter.createBlock(&selectionOp.getBody().front());
+
+    // Inline `then` region before the merge block and branch to it.
+    auto &thenRegion = ifOp.getThenRegion();
+    auto *thenBlock = &thenRegion.front();
+    rewriter.setInsertionPointToEnd(&thenRegion.back());
+    rewriter.create<spirv::BranchOp>(loc, mergeBlock);
+    rewriter.inlineRegionBefore(thenRegion, mergeBlock);
+
+    auto *elseBlock = mergeBlock;
+    // If `else` region is not empty, inline that region before the merge block
+    // and branch to it.
+    if (!ifOp.getElseRegion().empty()) {
+      auto &elseRegion = ifOp.getElseRegion();
+      elseBlock = &elseRegion.front();
+      rewriter.setInsertionPointToEnd(&elseRegion.back());
+      rewriter.create<spirv::BranchOp>(loc, mergeBlock);
+      rewriter.inlineRegionBefore(elseRegion, mergeBlock);
+    }
 
-  LogicalResult
-  matchAndRewrite(scf::YieldOp terminatorOp, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override;
+    // Create a `spirv.BranchConditional` operation for selection header block.
+    rewriter.setInsertionPointToEnd(selectionHeaderBlock);
+    rewriter.create<spirv::BranchConditionalOp>(loc, adaptor.getCondition(),
+                                                thenBlock, ArrayRef<Value>(),
+                                                elseBlock, ArrayRef<Value>());
+
+    SmallVector<Type, 8> returnTypes;
+    for (auto result : ifOp.getResults()) {
+      auto convertedType = typeConverter.convertType(result.getType());
+      if (!convertedType)
+        return rewriter.notifyMatchFailure(
+            loc,
+            llvm::formatv("failed to convert type '{0}'", result.getType()));
+
+      returnTypes.push_back(convertedType);
+    }
+    replaceSCFOutputValue(ifOp, selectionOp, rewriter, scfToSPIRVContext,
+                          returnTypes);
+    return success();
+  }
 };
 
-class WhileOpConversion final : public SCFToSPIRVPattern<scf::WhileOp> {
+//===----------------------------------------------------------------------===//
+// scf::YieldOp
+//===----------------------------------------------------------------------===//
+
+struct TerminatorOpConversion final : SCFToSPIRVPattern<scf::YieldOp> {
 public:
-  using SCFToSPIRVPattern<scf::WhileOp>::SCFToSPIRVPattern;
+  using SCFToSPIRVPattern::SCFToSPIRVPattern;
 
   LogicalResult
-  matchAndRewrite(scf::WhileOp forOp, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override;
-};
-} // namespace
+  matchAndRewrite(scf::YieldOp terminatorOp, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    ValueRange operands = adaptor.getOperands();
 
-/// Helper function to replaces SCF op outputs with SPIR-V variable loads.
-/// We create VariableOp to handle the results value of the control flow region.
-/// spirv.mlir.loop/spirv.mlir.selection currently don't yield value. Right
-/// after the loop we load the value from the allocation and use it as the SCF
-/// op result.
-template <typename ScfOp, typename OpTy>
-static void replaceSCFOutputValue(ScfOp scfOp, OpTy newOp,
-                                  ConversionPatternRewriter &rewriter,
-                                  ScfToSPIRVContextImpl *scfToSPIRVContext,
-                                  ArrayRef<Type> returnTypes) {
+    Operation *parent = terminatorOp->getParentOp();
 
-  Location loc = scfOp.getLoc();
-  auto &allocas = scfToSPIRVContext->outputVars[newOp];
-  // Clearing the allocas is necessary in case a dialect conversion path failed
-  // previously, and this is the second attempt of this conversion.
-  allocas.clear();
-  SmallVector<Value, 8> resultValue;
-  for (Type convertedType : returnTypes) {
-    auto pointerType =
-        spirv::PointerType::get(convertedType, spirv::StorageClass::Function);
-    rewriter.setInsertionPoint(newOp);
-    auto alloc = rewriter.create<spirv::VariableOp>(
-        loc, pointerType, spirv::StorageClass::Function,
-        /*initializer=*/nullptr);
-    allocas.push_back(alloc);
-    rewriter.setInsertionPointAfter(newOp);
-    Value loadResult = rewriter.create<spirv::LoadOp>(loc, alloc);
-    resultValue.push_back(loadResult);
+    // TODO: Implement conversion for the remaining `scf` ops.
+    if (parent->getDialect()->getNamespace() ==
+            scf::SCFDialect::getDialectNamespace() &&
+        !isa<scf::IfOp, scf::ForOp, scf::WhileOp>(parent))
+      return rewriter.notifyMatchFailure(
+          terminatorOp,
+          llvm::formatv("conversion not supported for parent op: '{0}'",
+                        parent->getName()));
+
+    // If the region return values, store each value into the associated
+    // VariableOp created during lowering of the parent region.
+    if (!operands.empty()) {
+      auto &allocas = scfToSPIRVContext->outputVars[parent];
+      if (allocas.size() != operands.size())
+        return failure();
+
+      auto loc = terminatorOp.getLoc();
+      for (unsigned i = 0, e = operands.size(); i < e; i++)
+        rewriter.create<spirv::StoreOp>(loc, allocas[i], operands[i]);
+      if (isa<spirv::LoopOp>(parent)) {
+        // For loops we also need to update the branch jumping back to the
+        // header.
+        auto br = cast<spirv::BranchOp>(
+            rewriter.getInsertionBlock()->getTerminator());
+        SmallVector<Value, 8> args(br.getBlockArguments());
+        args.append(operands.begin(), operands.end());
+        rewriter.setInsertionPoint(br);
+        rewriter.create<spirv::BranchOp>(terminatorOp.getLoc(), br.getTarget(),
+                                         args);
+        rewriter.eraseOp(br);
+      }
+    }
+    rewriter.eraseOp(terminatorOp);
+    return success();
   }
-  rewriter.replaceOp(scfOp, resultValue);
-}
-
-static Region::iterator getBlockIt(Region &region, unsigned index) {
-  return std::next(region.begin(), index);
-}
+};
 
 //===----------------------------------------------------------------------===//
-// scf::ForOp
+// scf::WhileOp
 //===----------------------------------------------------------------------===//
 
-LogicalResult
-ForOpConversion::matchAndRewrite(scf::ForOp forOp, OpAdaptor adaptor,
-                                 ConversionPatternRewriter &rewriter) const {
-  // scf::ForOp can be lowered to the structured control flow represented by
-  // spirv::LoopOp by making the continue block of the spirv::LoopOp the loop
-  // latch and the merge block the exit block. The resulting spirv::LoopOp has a
-  // single back edge from the continue to header block, and a single exit from
-  // header to merge.
-  auto loc = forOp.getLoc();
-  auto loopOp = rewriter.create<spirv::LoopOp>(loc, spirv::LoopControl::None);
-  loopOp.addEntryAndMergeBlock();
-
-  OpBuilder::InsertionGuard guard(rewriter);
-  // Create the block for the header.
-  auto *header = new Block();
-  // Insert the header.
-  loopOp.getBody().getBlocks().insert(getBlockIt(loopOp.getBody(), 1), header);
-
-  // Create the new induction variable to use.
-  Value adapLowerBound = adaptor.getLowerBound();
-  BlockArgument newIndVar =
-      header->addArgument(adapLowerBound.getType(), adapLowerBound.getLoc());
-  for (Value arg : adaptor.getInitArgs())
-    header->addArgument(arg.getType(), arg.getLoc());
-  Block *body = forOp.getBody();
-
-  // Apply signature conversion to the body of the forOp. It has a single block,
-  // with argument which is the induction variable. That has to be replaced with
-  // the new induction variable.
-  TypeConverter::SignatureConversion signatureConverter(
-      body->getNumArguments());
-  signatureConverter.remapInput(0, newIndVar);
-  for (unsigned i = 1, e = body->getNumArguments(); i < e; i++)
-    signatureConverter.remapInput(i, header->getArgument(i));
-  body = rewriter.applySignatureConversion(&forOp.getLoopBody(),
-                                           signatureConverter);
-
-  // Move the blocks from the forOp into the loopOp. This is the body of the
-  // loopOp.
-  rewriter.inlineRegionBefore(forOp->getRegion(0), loopOp.getBody(),
-                              getBlockIt(loopOp.getBody(), 2));
-
-  SmallVector<Value, 8> args(1, adaptor.getLowerBound());
-  args.append(adaptor.getInitArgs().begin(), adaptor.getInitArgs().end());
-  // Branch into it from the entry.
-  rewriter.setInsertionPointToEnd(&(loopOp.getBody().front()));
-  rewriter.create<spirv::BranchOp>(loc, header, args);
-
-  // Generate the rest of the loop header.
-  rewriter.setInsertionPointToEnd(header);
-  auto *mergeBlock = loopOp.getMergeBlock();
-  auto cmpOp = rewriter.create<spirv::SLessThanOp>(
-      loc, rewriter.getI1Type(), newIndVar, adaptor.getUpperBound());
-
-  rewriter.create<spirv::BranchConditionalOp>(
-      loc, cmpOp, body, ArrayRef<Value>(), mergeBlock, ArrayRef<Value>());
-
-  // Generate instructions to increment the step of the induction variable and
-  // branch to the header.
-  Block *continueBlock = loopOp.getContinueBlock();
-  rewriter.setInsertionPointToEnd(continueBlock);
-
-  // Add the step to the induction variable and branch to the header.
-  Value updatedIndVar = rewriter.create<spirv::IAddOp>(
-      loc, newIndVar.getType(), newIndVar, adaptor.getStep());
-  rewriter.create<spirv::BranchOp>(loc, header, updatedIndVar);
-
-  // Infer the return types from the init operands. Vector type may get
-  // converted to CooperativeMatrix or to Vector type, to avoid having complex
-  // extra logic to figure out the right type we just infer it from the Init
-  // operands.
-  SmallVector<Type, 8> initTypes;
-  for (auto arg : adaptor.getInitArgs())
-    initTypes.push_back(arg.getType());
-  replaceSCFOutputValue(forOp, loopOp, rewriter, scfToSPIRVContext, initTypes);
-  return success();
-}
+struct WhileOpConversion final : SCFToSPIRVPattern<scf::WhileOp> {
+  using SCFToSPIRVPattern::SCFToSPIRVPattern;
 
-//===----------------------------------------------------------------------===//
-// scf::IfOp
-//===----------------------------------------------------------------------===//
+  LogicalResult
+  matchAndRewrite(scf::WhileOp whileOp, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto loc = whileOp.getLoc();
+    auto loopOp = rewriter.create<spirv::LoopOp>(loc, spirv::LoopControl::None);
+    loopOp.addEntryAndMergeBlock();
 
-LogicalResult
-IfOpConversion::matchAndRewrite(scf::IfOp ifOp, OpAdaptor adaptor,
-                                ConversionPatternRewriter &rewriter) const {
-  // When lowering `scf::IfOp` we explicitly create a selection header block
-  // before the control flow diverges and a merge block where control flow
-  // subsequently converges.
-  auto loc = ifOp.getLoc();
-
-  // Create `spirv.selection` operation, selection header block and merge block.
-  auto selectionOp =
-      rewriter.create<spirv::SelectionOp>(loc, spirv::SelectionControl::None);
-  auto *mergeBlock =
-      rewriter.createBlock(&selectionOp.getBody(), selectionOp.getBody().end());
-  rewriter.create<spirv::MergeOp>(loc);
-
-  OpBuilder::InsertionGuard guard(rewriter);
-  auto *selectionHeaderBlock =
-      rewriter.createBlock(&selectionOp.getBody().front());
-
-  // Inline `then` region before the merge block and branch to it.
-  auto &thenRegion = ifOp.getThenRegion();
-  auto *thenBlock = &thenRegion.front();
-  rewriter.setInsertionPointToEnd(&thenRegion.back());
-  rewriter.create<spirv::BranchOp>(loc, mergeBlock);
-  rewriter.inlineRegionBefore(thenRegion, mergeBlock);
-
-  auto *elseBlock = mergeBlock;
-  // If `else` region is not empty, inline that region before the merge block
-  // and branch to it.
-  if (!ifOp.getElseRegion().empty()) {
-    auto &elseRegion = ifOp.getElseRegion();
-    elseBlock = &elseRegion.front();
-    rewriter.setInsertionPointToEnd(&elseRegion.back());
-    rewriter.create<spirv::BranchOp>(loc, mergeBlock);
-    rewriter.inlineRegionBefore(elseRegion, mergeBlock);
-  }
+    OpBuilder::InsertionGuard guard(rewriter);
 
-  // Create a `spirv.BranchConditional` operation for selection header block.
-  rewriter.setInsertionPointToEnd(selectionHeaderBlock);
-  rewriter.create<spirv::BranchConditionalOp>(loc, adaptor.getCondition(),
-                                              thenBlock, ArrayRef<Value>(),
-                                              elseBlock, ArrayRef<Value>());
+    Region &beforeRegion = whileOp.getBefore();
+    Region &afterRegion = whileOp.getAfter();
 
-  SmallVector<Type, 8> returnTypes;
-  for (auto result : ifOp.getResults()) {
-    auto convertedType = typeConverter.convertType(result.getType());
-    if (!convertedType)
-      return rewriter.notifyMatchFailure(
-          loc, llvm::formatv("failed to convert type '{0}'", result.getType()));
+    Block &entryBlock = *loopOp.getEntryBlock();
+    Block &beforeBlock = beforeRegion.front();
+    Block &afterBlock = afterRegion.front();
+    Block &mergeBlock = *loopOp.getMergeBlock();
 
-    returnTypes.push_back(convertedType);
-  }
-  replaceSCFOutputValue(ifOp, selectionOp, rewriter, scfToSPIRVContext,
-                        returnTypes);
-  return success();
-}
+    auto cond = cast<scf::ConditionOp>(beforeBlock.getTerminator());
+    SmallVector<Value> condArgs;
+    if (failed(rewriter.getRemappedValues(cond.getArgs(), condArgs)))
+      return failure();
 
-//===----------------------------------------------------------------------===//
-// scf::YieldOp
-//===----------------------------------------------------------------------===//
+    Value conditionVal = rewriter.getRemappedValue(cond.getCondition());
+    if (!conditionVal)
+      return failure();
 
-/// Yield is lowered to stores to the VariableOp created during lowering of the
-/// parent region. For loops we also need to update the branch looping back to
-/// the header with the loop carried values.
-LogicalResult TerminatorOpConversion::matchAndRewrite(
-    scf::YieldOp terminatorOp, OpAdaptor adaptor,
-    ConversionPatternRewriter &rewriter) const {
-  ValueRange operands = adaptor.getOperands();
-
-  // If the region is return values, store each value into the associated
-  // VariableOp created during lowering of the parent region.
-  if (!operands.empty()) {
-    auto &allocas = scfToSPIRVContext->outputVars[terminatorOp->getParentOp()];
-    if (allocas.size() != operands.size())
+    auto yield = cast<scf::YieldOp>(afterBlock.getTerminator());
+    SmallVector<Value> yieldArgs;
+    if (failed(rewriter.getRemappedValues(yield.getResults(), yieldArgs)))
       return failure();
 
-    auto loc = terminatorOp.getLoc();
-    for (unsigned i = 0, e = operands.size(); i < e; i++)
-      rewriter.create<spirv::StoreOp>(loc, allocas[i], operands[i]);
-    if (isa<spirv::LoopOp>(terminatorOp->getParentOp())) {
-      // For loops we also need to update the branch jumping back to the header.
-      auto br =
-          cast<spirv::BranchOp>(rewriter.getInsertionBlock()->getTerminator());
-      SmallVector<Value, 8> args(br.getBlockArguments());
-      args.append(operands.begin(), operands.end());
-      rewriter.setInsertionPoint(br);
-      rewriter.create<spirv::BranchOp>(terminatorOp.getLoc(), br.getTarget(),
-                                       args);
-      rewriter.eraseOp(br);
+    // Move the while before block as the initial loop header block.
+    rewriter.inlineRegionBefore(beforeRegion, loopOp.getBody(),
+                                getBlockIt(loopOp.getBody(), 1));
+
+    // Move the while after block as the initial loop body block.
+    rewriter.inlineRegionBefore(afterRegion, loopOp.getBody(),
+                                getBlockIt(loopOp.getBody(), 2));
+
+    // Jump from the loop entry block to the loop header block.
+    rewriter.setInsertionPointToEnd(&entryBlock);
+    rewriter.create<spirv::BranchOp>(loc, &beforeBlock, adaptor.getInits());
+
+    auto condLoc = cond.getLoc();
+
+    SmallVector<Value> resultValues(condArgs.size());
+
+    // For other SCF ops, the scf.yield op yields the value for the whole SCF
+    // op. So we use the scf.yield op as the anchor to create/load/store SPIR-V
+    // local variables. But for the scf.while op, the scf.yield op yields a
+    // value for the before region, which may not matching the whole op's
+    // result. Instead, the scf.condition op returns values matching the whole
+    // op's results. So we need to create/load/store variables according to
+    // that.
+    for (const auto &it : llvm::enumerate(condArgs)) {
+      auto res = it.value();
+      auto i = it.index();
+      auto pointerType =
+          spirv::PointerType::get(res.getType(), spirv::StorageClass::Function);
+
+      // Create local variables before the scf.while op.
+      rewriter.setInsertionPoint(loopOp);
+      auto alloc = rewriter.create<spirv::VariableOp>(
+          condLoc, pointerType, spirv::StorageClass::Function,
+          /*initializer=*/nullptr);
+
+      // Load the final result values after the scf.while op.
+      rewriter.setInsertionPointAfter(loopOp);
+      auto loadResult = rewriter.create<spirv::LoadOp>(condLoc, alloc);
+      resultValues[i] = loadResult;
+
+      // Store the current iteration's result value.
+      rewriter.setInsertionPointToEnd(&beforeBlock);
+      rewriter.create<spirv::StoreOp>(condLoc, alloc, res);
     }
-  }
-  rewriter.eraseOp(terminatorOp);
-  return success();
-}
 
-//===----------------------------------------------------------------------===//
-// scf::WhileOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult
-WhileOpConversion::matchAndRewrite(scf::WhileOp whileOp, OpAdaptor adaptor,
-                                   ConversionPatternRewriter &rewriter) const {
-  auto loc = whileOp.getLoc();
-  auto loopOp = rewriter.create<spirv::LoopOp>(loc, spirv::LoopControl::None);
-  loopOp.addEntryAndMergeBlock();
-
-  OpBuilder::InsertionGuard guard(rewriter);
-
-  Region &beforeRegion = whileOp.getBefore();
-  Region &afterRegion = whileOp.getAfter();
-
-  Block &entryBlock = *loopOp.getEntryBlock();
-  Block &beforeBlock = beforeRegion.front();
-  Block &afterBlock = afterRegion.front();
-  Block &mergeBlock = *loopOp.getMergeBlock();
-
-  auto cond = cast<scf::ConditionOp>(beforeBlock.getTerminator());
-  SmallVector<Value> condArgs;
-  if (failed(rewriter.getRemappedValues(cond.getArgs(), condArgs)))
-    return failure();
-
-  Value conditionVal = rewriter.getRemappedValue(cond.getCondition());
-  if (!conditionVal)
-    return failure();
-
-  auto yield = cast<scf::YieldOp>(afterBlock.getTerminator());
-  SmallVector<Value> yieldArgs;
-  if (failed(rewriter.getRemappedValues(yield.getResults(), yieldArgs)))
-    return failure();
-
-  // Move the while before block as the initial loop header block.
-  rewriter.inlineRegionBefore(beforeRegion, loopOp.getBody(),
-                              getBlockIt(loopOp.getBody(), 1));
-
-  // Move the while after block as the initial loop body block.
-  rewriter.inlineRegionBefore(afterRegion, loopOp.getBody(),
-                              getBlockIt(loopOp.getBody(), 2));
-
-  // Jump from the loop entry block to the loop header block.
-  rewriter.setInsertionPointToEnd(&entryBlock);
-  rewriter.create<spirv::BranchOp>(loc, &beforeBlock, adaptor.getInits());
-
-  auto condLoc = cond.getLoc();
-
-  SmallVector<Value> resultValues(condArgs.size());
-
-  // For other SCF ops, the scf.yield op yields the value for the whole SCF op.
-  // So we use the scf.yield op as the anchor to create/load/store SPIR-V local
-  // variables. But for the scf.while op, the scf.yield op yields a value for
-  // the before region, which may not matching the whole op's result. Instead,
-  // the scf.condition op returns values matching the whole op's results. So we
-  // need to create/load/store variables according to that.
-  for (const auto &it : llvm::enumerate(condArgs)) {
-    auto res = it.value();
-    auto i = it.index();
-    auto pointerType =
-        spirv::PointerType::get(res.getType(), spirv::StorageClass::Function);
-
-    // Create local variables before the scf.while op.
-    rewriter.setInsertionPoint(loopOp);
-    auto alloc = rewriter.create<spirv::VariableOp>(
-        condLoc, pointerType, spirv::StorageClass::Function,
-        /*initializer=*/nullptr);
-
-    // Load the final result values after the scf.while op.
-    rewriter.setInsertionPointAfter(loopOp);
-    auto loadResult = rewriter.create<spirv::LoadOp>(condLoc, alloc);
-    resultValues[i] = loadResult;
-
-    // Store the current iteration's result value.
     rewriter.setInsertionPointToEnd(&beforeBlock);
-    rewriter.create<spirv::StoreOp>(condLoc, alloc, res);
-  }
-
-  rewriter.setInsertionPointToEnd(&beforeBlock);
-  rewriter.replaceOpWithNewOp<spirv::BranchConditionalOp>(
-      cond, conditionVal, &afterBlock, condArgs, &mergeBlock, std::nullopt);
+    rewriter.replaceOpWithNewOp<spirv::BranchConditionalOp>(
+        cond, conditionVal, &afterBlock, condArgs, &mergeBlock, std::nullopt);
 
-  // Convert the scf.yield op to a branch back to the header block.
-  rewriter.setInsertionPointToEnd(&afterBlock);
-  rewriter.replaceOpWithNewOp<spirv::BranchOp>(yield, &beforeBlock, yieldArgs);
+    // Convert the scf.yield op to a branch back to the header block.
+    rewriter.setInsertionPointToEnd(&afterBlock);
+    rewriter.replaceOpWithNewOp<spirv::BranchOp>(yield, &beforeBlock,
+                                                 yieldArgs);
 
-  rewriter.replaceOp(whileOp, resultValues);
-  return success();
-}
+    rewriter.replaceOp(whileOp, resultValues);
+    return success();
+  }
+};
+} // namespace
 
 //===----------------------------------------------------------------------===//
-// Hooks
+// Public API
 //===----------------------------------------------------------------------===//
 
 void mlir::populateSCFToSPIRVPatterns(SPIRVTypeConverter &typeConverter,
diff --git a/mlir/lib/Conversion/SCFToSPIRV/SCFToSPIRVPass.cpp b/mlir/lib/Conversion/SCFToSPIRV/SCFToSPIRVPass.cpp
index a6ce99284db96..1e8fe4423a422 100644
--- a/mlir/lib/Conversion/SCFToSPIRV/SCFToSPIRVPass.cpp
+++ b/mlir/lib/Conversion/SCFToSPIRV/SCFToSPIRVPass.cpp
@@ -56,7 +56,3 @@ void SCFToSPIRVPass::runOnOperation() {
   if (failed(applyPartialConversion(op, *target, std::move(patterns))))
     return signalPassFailure();
 }
-
-std::unique_ptr<OperationPass<>> mlir::createConvertSCFToSPIRVPass() {
-  return std::make_unique<SCFToSPIRVPass>();
-}
diff --git a/mlir/lib/Conversion/SPIRVCommon/Pattern.h b/mlir/lib/Conversion/SPIRVCommon/Pattern.h
index 4da3e197ca3ed..7425f4b5311ce 100644
--- a/mlir/lib/Conversion/SPIRVCommon/Pattern.h
+++ b/mlir/lib/Conversion/SPIRVCommon/Pattern.h
@@ -10,6 +10,7 @@
 #define MLIR_CONVERSION_SPIRVCOMMON_PATTERN_H
 
 #include "mlir/Dialect/SPIRV/IR/SPIRVOpTraits.h"
+#include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
 #include "llvm/Support/FormatVariadic.h"
 
@@ -34,9 +35,11 @@ struct ElementwiseOpPattern : public OpConversionPattern<Op> {
     }
 
     if (SPIRVOp::template hasTrait<OpTrait::spirv::UnsignedOp>() &&
-        !op.getType().isIndex() && dstType != op.getType()) {
-      return op.emitError(
-          "bitwidth emulation is not implemented yet on unsigned op");
+        !getElementTypeOrSelf(op.getType()).isIndex() &&
+        dstType != op.getType()) {
+      op.dump();
+      return op.emitError("bitwidth emulation is not implemented yet on "
+                          "unsigned op pattern version");
     }
     rewriter.template replaceOpWithNewOp<SPIRVOp>(op, dstType,
                                                   adaptor.getOperands());
diff --git a/mlir/lib/Conversion/TosaToLinalg/TosaToLinalg.cpp b/mlir/lib/Conversion/TosaToLinalg/TosaToLinalg.cpp
index 46f5292a9ad1a..271a09539e46e 100644
--- a/mlir/lib/Conversion/TosaToLinalg/TosaToLinalg.cpp
+++ b/mlir/lib/Conversion/TosaToLinalg/TosaToLinalg.cpp
@@ -823,117 +823,15 @@ static LogicalResult reduceMatchAndRewriteHelper(Operation *op, uint64_t axis,
         rewriter.getAffineDimExpr(expandedDim + 1));
   }
 
+  // Lower directly to `tensor::ExpandShapeOp` instead of `tosa::ReshapeOp`,
+  // since here we know which dimension to expand, and `tosa::ReshapeOp` would
+  // not have access to such information. This matters when handling dynamically
+  // sized tensors.
   rewriter.replaceOpWithNewOp<tensor::ExpandShapeOp>(
       op, resultTy, linalgOp.getResults()[0], reassociationMap);
   return success();
 }
 
-static bool findIntermediateShape(ArrayRef<int64_t> lhsShape,
-                                  ArrayRef<int64_t> rhsShape,
-                                  SmallVector<int64_t> &intermediateShape,
-                                  bool isDynamic) {
-  if (isDynamic) {
-    // TODO (natashaknk): Make dynamic intermediate shape not always be rank-1
-    intermediateShape = {ShapedType::kDynamic};
-    return true;
-  }
-
-  if (lhsShape.empty() || rhsShape.empty()) {
-    intermediateShape = {};
-    return true;
-  }
-
-  unsigned currLhsDim = 0, currRhsDim = 0;
-  while (currLhsDim < lhsShape.size() && currRhsDim < rhsShape.size()) {
-    int64_t rhsSize = rhsShape[currRhsDim];
-    int64_t lhsSize = lhsShape[currLhsDim];
-    while (lhsSize != rhsSize && currLhsDim < lhsShape.size() &&
-           currRhsDim < rhsShape.size()) {
-      if (lhsSize < rhsSize) {
-        currLhsDim++;
-        if (currLhsDim < lhsShape.size()) {
-          lhsSize *= lhsShape[currLhsDim];
-        }
-      } else {
-        currRhsDim++;
-        if (currRhsDim < rhsShape.size()) {
-          rhsSize *= rhsShape[currRhsDim];
-        }
-      }
-    }
-    if (lhsSize == rhsSize) {
-      intermediateShape.push_back(lhsSize);
-    }
-    currRhsDim++;
-    currLhsDim++;
-  }
-
-  // If the iterators didn't reach the end and their leftover dimensions are not
-  // equal to 1 an intermediate shape was not found.
-  while (currLhsDim < lhsShape.size()) {
-    if (lhsShape[currLhsDim++] != 1) {
-      return false;
-    }
-  }
-
-  while (currRhsDim < rhsShape.size()) {
-    if (rhsShape[currRhsDim++] != 1) {
-      return false;
-    }
-  }
-
-  return true;
-}
-
-static bool createReassociationMapsForCollapse(
-    PatternRewriter &rewriter, ArrayRef<int64_t> srcShape,
-    ArrayRef<int64_t> dstShape,
-    SmallVector<ReassociationExprs, 4> &reassociationMap, bool isDynamic) {
-
-  // If the shape is dynamic, create a map for collapsing into one dimension.
-  if (isDynamic) {
-    SmallVector<AffineExpr, 2> exprs;
-    for (int i = 0, s = srcShape.size(); i < s; ++i)
-      exprs.push_back(rewriter.getAffineDimExpr(i));
-    reassociationMap = {exprs};
-    return true;
-  }
-
-  if (dstShape.empty()) {
-    reassociationMap = {};
-    return true;
-  }
-
-  reassociationMap.resize(dstShape.size());
-  unsigned currSrcDim = 0, currDstDim = 0;
-  while (currSrcDim < srcShape.size() && currDstDim < dstShape.size()) {
-    int64_t dstSize = dstShape[currDstDim];
-    int64_t srcSize = srcShape[currSrcDim];
-    while (srcSize < dstSize && currSrcDim < srcShape.size()) {
-      reassociationMap[currDstDim].push_back(
-          rewriter.getAffineDimExpr(currSrcDim++));
-      srcSize *= srcShape[currSrcDim];
-    }
-    if (srcSize == dstSize) {
-      reassociationMap[currDstDim].push_back(
-          rewriter.getAffineDimExpr(currSrcDim++));
-      // If the next dim in collapsedShape is not 1, treat subsequent dims in
-      // expandedShape which are 1 to be collapsed.
-      if (currDstDim == dstShape.size() - 1 || dstShape[currDstDim + 1] != 1) {
-        while (currSrcDim < srcShape.size() && srcShape[currSrcDim] == 1) {
-          reassociationMap[currDstDim].push_back(
-              rewriter.getAffineDimExpr(currSrcDim++));
-        }
-      }
-    }
-    currDstDim++;
-  }
-
-  // If both iterators didn't reach the end, we have leftover dimentions which
-  // implies that we have a mismatch in shape.
-  return currSrcDim == srcShape.size() && currDstDim == dstShape.size();
-}
-
 namespace {
 
 template <typename SrcOp>
@@ -947,115 +845,6 @@ class PointwiseConverter : public OpRewritePattern<SrcOp> {
   }
 };
 
-class ReshapeConverterCollapse : public OpConversionPattern<tosa::ReshapeOp> {
-public:
-  using OpConversionPattern<tosa::ReshapeOp>::OpConversionPattern;
-
-  LogicalResult
-  matchAndRewrite(tosa::ReshapeOp reshape, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const final {
-    ShapedType operandTy = adaptor.getInput1().getType().cast<ShapedType>();
-    ShapedType resultTy = reshape.getType().template cast<ShapedType>();
-    bool isDynamic = !operandTy.hasStaticShape();
-
-    if (isDynamic && resultTy.getRank() != 1) {
-      return rewriter.notifyMatchFailure(
-          reshape, "Cannot collapse dynamic dims to more than one dimension");
-    }
-
-    SmallVector<ReassociationExprs, 4> reassociationMap;
-    if (!createReassociationMapsForCollapse(rewriter, operandTy.getShape(),
-                                            resultTy.getShape(),
-                                            reassociationMap, isDynamic)) {
-      return rewriter.notifyMatchFailure(
-          reshape,
-          "tosa.reshape Attempting to collapse into an incompatible shape");
-    }
-
-    SmallVector<int64_t> intermediateShape;
-    if (!findIntermediateShape(operandTy.getShape(), resultTy.getShape(),
-                               intermediateShape, isDynamic)) {
-      return rewriter.notifyMatchFailure(
-          reshape, "tosa.reshape Cannot collapse into given shape");
-    }
-
-    rewriter.replaceOpWithNewOp<tensor::CollapseShapeOp>(
-        reshape, resultTy, adaptor.getOperands()[0], reassociationMap);
-    return success();
-  }
-};
-
-class ReshapeConverterExpand : public OpConversionPattern<tosa::ReshapeOp> {
-public:
-  using OpConversionPattern<tosa::ReshapeOp>::OpConversionPattern;
-
-  LogicalResult
-  matchAndRewrite(tosa::ReshapeOp reshape, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const final {
-    ShapedType operandTy = adaptor.getInput1().getType().cast<ShapedType>();
-    ShapedType resultTy = reshape.getType().template cast<ShapedType>();
-    bool isDynamic = !operandTy.hasStaticShape();
-
-    if (isDynamic && operandTy.getRank() != 1) {
-      return rewriter.notifyMatchFailure(
-          reshape, "Cannot expand dynamic dims from more than one dimension");
-    }
-
-    SmallVector<ReassociationExprs, 4> reassociationMap;
-    if (!createReassociationMapsForCollapse(rewriter, resultTy.getShape(),
-                                            operandTy.getShape(),
-                                            reassociationMap, isDynamic)) {
-      return rewriter.notifyMatchFailure(
-          reshape,
-          "tosa.reshape Attempting to expand into an incompatible shape");
-    }
-
-    SmallVector<int64_t> intermediateShape;
-    if (!findIntermediateShape(operandTy.getShape(), resultTy.getShape(),
-                               intermediateShape, isDynamic) ||
-        intermediateShape != operandTy.getShape()) {
-      return rewriter.notifyMatchFailure(
-          reshape, "tosa.reshape Cannot expand into given shape");
-    }
-    rewriter.replaceOpWithNewOp<tensor::ExpandShapeOp>(
-        reshape, resultTy, adaptor.getOperands()[0], reassociationMap);
-    return success();
-  }
-};
-
-class ReshapeConverterCollapseExpand
-    : public OpConversionPattern<tosa::ReshapeOp> {
-public:
-  using OpConversionPattern<tosa::ReshapeOp>::OpConversionPattern;
-
-  LogicalResult
-  matchAndRewrite(tosa::ReshapeOp reshape, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const final {
-    ShapedType operandTy = adaptor.getInput1().getType().cast<ShapedType>();
-    ShapedType resultTy = reshape.getType().template cast<ShapedType>();
-    bool isDynamic = !operandTy.hasStaticShape();
-
-    SmallVector<int64_t> intermediateShape;
-    if (!findIntermediateShape(resultTy.getShape(), operandTy.getShape(),
-                               intermediateShape, isDynamic)) {
-      return rewriter.notifyMatchFailure(
-          reshape, "tosa.reshape Cannot identify an intermediate shape between "
-                   "the given two shapes");
-    }
-
-    Value collapse = rewriter.create<tosa::ReshapeOp>(
-        reshape.getLoc(),
-        RankedTensorType::get(intermediateShape,
-                              reshape.getType().getElementType()),
-        adaptor.getInput1());
-    Value expand =
-        rewriter.create<tosa::ReshapeOp>(reshape.getLoc(), resultTy, collapse);
-    rewriter.replaceOp(reshape, expand);
-
-    return success();
-  }
-};
-
 class TransposeConverter : public OpRewritePattern<tosa::TransposeOp> {
 public:
   using OpRewritePattern<tosa::TransposeOp>::OpRewritePattern;
@@ -1786,68 +1575,6 @@ class ReduceConverter : public OpRewritePattern<SrcOp> {
   }
 };
 
-struct ConcatConverter : public OpConversionPattern<tosa::ConcatOp> {
-  using OpConversionPattern<tosa::ConcatOp>::OpConversionPattern;
-
-  LogicalResult
-  matchAndRewrite(tosa::ConcatOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    auto inputType = op.getOperand(0).getType().template cast<ShapedType>();
-    auto resultType = op.getType().dyn_cast<RankedTensorType>();
-
-    Location loc = op.getLoc();
-    int axis = op.getAxis();
-    Value axisValue = rewriter.createOrFold<arith::ConstantOp>(
-        loc, rewriter.getIndexAttr(axis));
-    int rank = resultType.getRank();
-    SmallVector<Value, 3> offsets, sizes, strides;
-    sizes.reserve(rank);
-    strides.resize(rank, rewriter.create<arith::ConstantIndexOp>(loc, 1));
-    offsets.resize(rank, rewriter.create<arith::ConstantIndexOp>(loc, 0));
-
-    SmallVector<Value> dynDims;
-    for (int i = 0; i < rank; ++i) {
-      sizes.push_back(rewriter.createOrFold<tensor::DimOp>(
-          loc, adaptor.getOperands()[0], i));
-      if (inputType.isDynamicDim(i)) {
-        dynDims.push_back(
-            rewriter.create<tensor::DimOp>(loc, op.getOperand(0), i));
-      }
-    }
-
-    Value resultDimSize = sizes[axis];
-    for (auto arg : adaptor.getOperands().drop_front()) {
-      auto size = rewriter.createOrFold<tensor::DimOp>(loc, arg, axisValue);
-      resultDimSize =
-          rewriter.createOrFold<arith::AddIOp>(loc, resultDimSize, size);
-    }
-    sizes[axis] = resultDimSize;
-
-    Value emptyTensor = rewriter.create<tensor::EmptyOp>(
-        loc, resultType.getShape(), resultType.getElementType(), dynDims);
-
-    auto toOpFoldResult = [](Value v) -> OpFoldResult {
-      auto op = v.getDefiningOp<arith::ConstantIndexOp>();
-      if (!op)
-        return v;
-      return op.getValue();
-    };
-    Value result = emptyTensor;
-    for (auto arg : adaptor.getOperands()) {
-      sizes[axis] = rewriter.createOrFold<tensor::DimOp>(loc, arg, axisValue);
-      result = rewriter.createOrFold<tensor::InsertSliceOp>(
-          loc, arg, result,
-          llvm::to_vector(llvm::map_range(offsets, toOpFoldResult)),
-          llvm::to_vector(llvm::map_range(sizes, toOpFoldResult)),
-          llvm::to_vector(llvm::map_range(strides, toOpFoldResult)));
-      offsets[axis] =
-          rewriter.createOrFold<arith::AddIOp>(loc, offsets[axis], sizes[axis]);
-    }
-    rewriter.replaceOp(op, result);
-    return success();
-  }
-};
-
 class ReverseConverter : public OpRewritePattern<tosa::ReverseOp> {
 public:
   using OpRewritePattern<tosa::ReverseOp>::OpRewritePattern;
@@ -2295,13 +2022,6 @@ void mlir::tosa::populateTosaToLinalgConversionPatterns(
   patterns->add<MaterializeResizeBroadcast>(patterns->getContext(),
                                             /*benefit=*/300);
 
-  patterns->add<ReshapeConverterCollapse>(patterns->getContext(),
-                                          /*benefit=*/100);
-  patterns->add<ReshapeConverterExpand>(patterns->getContext(),
-                                        /*benefit=*/200);
-  patterns->add<ReshapeConverterCollapseExpand>(patterns->getContext(),
-                                                /*benefit=*/300);
-
   patterns->add<
       // clang-format off
       PointwiseConverter<tosa::AddOp>,
@@ -2347,7 +2067,6 @@ void mlir::tosa::populateTosaToLinalgConversionPatterns(
       ReduceConverter<tosa::ReduceSumOp>,
       ReduceConverter<tosa::ReduceProdOp>,
       ArgMaxConverter,
-      ConcatConverter,
       GatherConverter,
       RescaleConverter,
       ReverseConverter,
diff --git a/mlir/lib/Conversion/TosaToLinalg/TosaToLinalgPass.cpp b/mlir/lib/Conversion/TosaToLinalg/TosaToLinalgPass.cpp
index 4b1e351e9746e..820cadbf62b56 100644
--- a/mlir/lib/Conversion/TosaToLinalg/TosaToLinalgPass.cpp
+++ b/mlir/lib/Conversion/TosaToLinalg/TosaToLinalgPass.cpp
@@ -55,7 +55,9 @@ struct TosaToLinalg : public impl::TosaToLinalgBase<TosaToLinalg> {
     target.addLegalOp<tosa::IfOp>();
     target.addLegalOp<tosa::ConstOp>();
     target.addLegalOp<tosa::WhileOp>();
+    target.addLegalOp<tosa::ConcatOp>();
     target.addLegalOp<tosa::SliceOp>();
+    target.addLegalOp<tosa::ReshapeOp>();
     target.addLegalOp<tosa::PadOp>();
 
     target.markUnknownOpDynamicallyLegal([](Operation *) { return true; });
diff --git a/mlir/lib/Conversion/TosaToTensor/TosaToTensor.cpp b/mlir/lib/Conversion/TosaToTensor/TosaToTensor.cpp
index 55acad6d0cf21..1ef31df2defd7 100644
--- a/mlir/lib/Conversion/TosaToTensor/TosaToTensor.cpp
+++ b/mlir/lib/Conversion/TosaToTensor/TosaToTensor.cpp
@@ -15,21 +15,236 @@
 #include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Tosa/IR/TosaOps.h"
 #include "mlir/IR/PatternMatch.h"
-#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+#include "mlir/Transforms/DialectConversion.h"
 
 using namespace mlir;
 using namespace tosa;
 
+static bool findIntermediateShape(ArrayRef<int64_t> lhsShape,
+                                  ArrayRef<int64_t> rhsShape,
+                                  SmallVector<int64_t> &intermediateShape,
+                                  bool isDynamic) {
+  if (isDynamic) {
+    // TODO (natashaknk): Make dynamic intermediate shape not always be rank-1
+    intermediateShape = {ShapedType::kDynamic};
+    return true;
+  }
+
+  if (lhsShape.empty() || rhsShape.empty()) {
+    intermediateShape = {};
+    return true;
+  }
+
+  unsigned currLhsDim = 0, currRhsDim = 0;
+  while (currLhsDim < lhsShape.size() && currRhsDim < rhsShape.size()) {
+    int64_t rhsSize = rhsShape[currRhsDim];
+    int64_t lhsSize = lhsShape[currLhsDim];
+    while (lhsSize != rhsSize && currLhsDim < lhsShape.size() &&
+           currRhsDim < rhsShape.size()) {
+      if (lhsSize < rhsSize) {
+        currLhsDim++;
+        if (currLhsDim < lhsShape.size()) {
+          lhsSize *= lhsShape[currLhsDim];
+        }
+      } else {
+        currRhsDim++;
+        if (currRhsDim < rhsShape.size()) {
+          rhsSize *= rhsShape[currRhsDim];
+        }
+      }
+    }
+    if (lhsSize == rhsSize) {
+      intermediateShape.push_back(lhsSize);
+    }
+    currRhsDim++;
+    currLhsDim++;
+  }
+
+  // If the iterators didn't reach the end and their leftover dimensions are not
+  // equal to 1 an intermediate shape was not found.
+  while (currLhsDim < lhsShape.size()) {
+    if (lhsShape[currLhsDim++] != 1) {
+      return false;
+    }
+  }
+
+  while (currRhsDim < rhsShape.size()) {
+    if (rhsShape[currRhsDim++] != 1) {
+      return false;
+    }
+  }
+
+  return true;
+}
+
+static bool createReassociationMapsForCollapse(
+    PatternRewriter &rewriter, ArrayRef<int64_t> srcShape,
+    ArrayRef<int64_t> dstShape,
+    SmallVector<ReassociationExprs, 4> &reassociationMap, bool isDynamic) {
+
+  // If the shape is dynamic, create a map for collapsing into one dimension.
+  if (isDynamic) {
+    SmallVector<AffineExpr, 2> exprs;
+    for (int i = 0, s = srcShape.size(); i < s; ++i)
+      exprs.push_back(rewriter.getAffineDimExpr(i));
+    reassociationMap = {exprs};
+    return true;
+  }
+
+  if (dstShape.empty()) {
+    reassociationMap = {};
+    return true;
+  }
+
+  reassociationMap.resize(dstShape.size());
+  unsigned currSrcDim = 0, currDstDim = 0;
+  while (currSrcDim < srcShape.size() && currDstDim < dstShape.size()) {
+    int64_t dstSize = dstShape[currDstDim];
+    int64_t srcSize = srcShape[currSrcDim];
+    while (srcSize < dstSize && currSrcDim < srcShape.size()) {
+      reassociationMap[currDstDim].push_back(
+          rewriter.getAffineDimExpr(currSrcDim++));
+      srcSize *= srcShape[currSrcDim];
+    }
+    if (srcSize == dstSize) {
+      reassociationMap[currDstDim].push_back(
+          rewriter.getAffineDimExpr(currSrcDim++));
+      // If the next dim in collapsedShape is not 1, treat subsequent dims in
+      // expandedShape which are 1 to be collapsed.
+      if (currDstDim == dstShape.size() - 1 || dstShape[currDstDim + 1] != 1) {
+        while (currSrcDim < srcShape.size() && srcShape[currSrcDim] == 1) {
+          reassociationMap[currDstDim].push_back(
+              rewriter.getAffineDimExpr(currSrcDim++));
+        }
+      }
+    }
+    currDstDim++;
+  }
+
+  // If both iterators didn't reach the end, we have leftover dimentions which
+  // implies that we have a mismatch in shape.
+  return currSrcDim == srcShape.size() && currDstDim == dstShape.size();
+}
+
 namespace {
+class ReshapeConverterCollapse : public OpConversionPattern<tosa::ReshapeOp> {
+public:
+  using OpConversionPattern<tosa::ReshapeOp>::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(tosa::ReshapeOp reshape, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const final {
+    ShapedType operandTy = adaptor.getInput1().getType().cast<ShapedType>();
+    ShapedType resultTy = reshape.getType().template cast<ShapedType>();
+    bool isDynamic = !operandTy.hasStaticShape();
 
-class SliceConverter : public OpRewritePattern<tosa::SliceOp> {
+    if (isDynamic && resultTy.getRank() != 1) {
+      return rewriter.notifyMatchFailure(
+          reshape, "Cannot collapse dynamic dims to more than one dimension");
+    }
+
+    SmallVector<ReassociationExprs, 4> reassociationMap;
+    if (!createReassociationMapsForCollapse(rewriter, operandTy.getShape(),
+                                            resultTy.getShape(),
+                                            reassociationMap, isDynamic)) {
+      return rewriter.notifyMatchFailure(
+          reshape,
+          "tosa.reshape Attempting to collapse into an incompatible shape");
+    }
+
+    SmallVector<int64_t> intermediateShape;
+    if (!findIntermediateShape(operandTy.getShape(), resultTy.getShape(),
+                               intermediateShape, isDynamic)) {
+      return rewriter.notifyMatchFailure(
+          reshape, "tosa.reshape Cannot collapse into given shape");
+    }
+
+    rewriter.replaceOpWithNewOp<tensor::CollapseShapeOp>(
+        reshape, resultTy, adaptor.getOperands()[0], reassociationMap);
+    return success();
+  }
+};
+
+class ReshapeConverterExpand : public OpConversionPattern<tosa::ReshapeOp> {
 public:
-  using OpRewritePattern<tosa::SliceOp>::OpRewritePattern;
+  using OpConversionPattern<tosa::ReshapeOp>::OpConversionPattern;
 
-  LogicalResult matchAndRewrite(tosa::SliceOp sliceOp,
-                                PatternRewriter &rewriter) const final {
+  LogicalResult
+  matchAndRewrite(tosa::ReshapeOp reshape, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const final {
+    ShapedType operandTy = adaptor.getInput1().getType().cast<ShapedType>();
+    ShapedType resultTy = reshape.getType().template cast<ShapedType>();
+    bool isDynamic = !operandTy.hasStaticShape();
+
+    if (isDynamic && operandTy.getRank() != 1) {
+      return rewriter.notifyMatchFailure(
+          reshape, "Cannot expand dynamic dims from more than one dimension");
+    }
+
+    SmallVector<ReassociationExprs, 4> reassociationMap;
+    if (!createReassociationMapsForCollapse(rewriter, resultTy.getShape(),
+                                            operandTy.getShape(),
+                                            reassociationMap, isDynamic)) {
+      return rewriter.notifyMatchFailure(
+          reshape,
+          "tosa.reshape Attempting to expand into an incompatible shape");
+    }
+
+    SmallVector<int64_t> intermediateShape;
+    if (!findIntermediateShape(operandTy.getShape(), resultTy.getShape(),
+                               intermediateShape, isDynamic) ||
+        intermediateShape != operandTy.getShape()) {
+      return rewriter.notifyMatchFailure(
+          reshape, "tosa.reshape Cannot expand into given shape");
+    }
+    rewriter.replaceOpWithNewOp<tensor::ExpandShapeOp>(
+        reshape, resultTy, adaptor.getOperands()[0], reassociationMap);
+    return success();
+  }
+};
+
+class ReshapeConverterCollapseExpand
+    : public OpConversionPattern<tosa::ReshapeOp> {
+public:
+  using OpConversionPattern<tosa::ReshapeOp>::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(tosa::ReshapeOp reshape, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const final {
+    ShapedType operandTy = adaptor.getInput1().getType().cast<ShapedType>();
+    ShapedType resultTy = reshape.getType().template cast<ShapedType>();
+    bool isDynamic = !operandTy.hasStaticShape();
+
+    SmallVector<int64_t> intermediateShape;
+    if (!findIntermediateShape(resultTy.getShape(), operandTy.getShape(),
+                               intermediateShape, isDynamic)) {
+      return rewriter.notifyMatchFailure(
+          reshape, "tosa.reshape Cannot identify an intermediate shape between "
+                   "the given two shapes");
+    }
+
+    Value collapse = rewriter.create<tosa::ReshapeOp>(
+        reshape.getLoc(),
+        RankedTensorType::get(intermediateShape,
+                              reshape.getType().getElementType()),
+        adaptor.getInput1());
+    Value expand =
+        rewriter.create<tosa::ReshapeOp>(reshape.getLoc(), resultTy, collapse);
+    rewriter.replaceOp(reshape, expand);
+
+    return success();
+  }
+};
+
+class SliceConverter : public OpConversionPattern<tosa::SliceOp> {
+public:
+  using OpConversionPattern<tosa::SliceOp>::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(tosa::SliceOp sliceOp, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const final {
     Location loc = sliceOp.getLoc();
-    Value input = sliceOp.getInput();
+    Value input = adaptor.getInput();
     SmallVector<int64_t> strides, sizes;
     ArrayRef<int64_t> starts = sliceOp.getStart();
     strides.resize(sliceOp.getType().template cast<ShapedType>().getRank(), 1);
@@ -134,9 +349,78 @@ class PadConverter : public OpRewritePattern<tosa::PadOp> {
   }
 };
 
+struct ConcatConverter : public OpConversionPattern<tosa::ConcatOp> {
+  using OpConversionPattern<tosa::ConcatOp>::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(tosa::ConcatOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto inputType = op.getOperand(0).getType().template cast<ShapedType>();
+    auto resultType = op.getType().dyn_cast<RankedTensorType>();
+
+    Location loc = op.getLoc();
+    int axis = op.getAxis();
+    Value axisValue = rewriter.createOrFold<arith::ConstantOp>(
+        loc, rewriter.getIndexAttr(axis));
+    int rank = resultType.getRank();
+    SmallVector<Value, 3> offsets, sizes, strides;
+    sizes.reserve(rank);
+    strides.resize(rank, rewriter.create<arith::ConstantIndexOp>(loc, 1));
+    offsets.resize(rank, rewriter.create<arith::ConstantIndexOp>(loc, 0));
+
+    SmallVector<Value> dynDims;
+    for (int i = 0; i < rank; ++i) {
+      sizes.push_back(rewriter.createOrFold<tensor::DimOp>(
+          loc, adaptor.getOperands()[0], i));
+      if (inputType.isDynamicDim(i)) {
+        dynDims.push_back(
+            rewriter.create<tensor::DimOp>(loc, op.getOperand(0), i));
+      }
+    }
+
+    Value resultDimSize = sizes[axis];
+    for (auto arg : adaptor.getOperands().drop_front()) {
+      auto size = rewriter.createOrFold<tensor::DimOp>(loc, arg, axisValue);
+      resultDimSize =
+          rewriter.createOrFold<arith::AddIOp>(loc, resultDimSize, size);
+    }
+    sizes[axis] = resultDimSize;
+
+    Value emptyTensor = rewriter.create<tensor::EmptyOp>(
+        loc, resultType.getShape(), resultType.getElementType(), dynDims);
+
+    auto toOpFoldResult = [](Value v) -> OpFoldResult {
+      auto op = v.getDefiningOp<arith::ConstantIndexOp>();
+      if (!op)
+        return v;
+      return op.getValue();
+    };
+    Value result = emptyTensor;
+    for (auto arg : adaptor.getOperands()) {
+      sizes[axis] = rewriter.createOrFold<tensor::DimOp>(loc, arg, axisValue);
+      result = rewriter.createOrFold<tensor::InsertSliceOp>(
+          loc, arg, result,
+          llvm::to_vector(llvm::map_range(offsets, toOpFoldResult)),
+          llvm::to_vector(llvm::map_range(sizes, toOpFoldResult)),
+          llvm::to_vector(llvm::map_range(strides, toOpFoldResult)));
+      offsets[axis] =
+          rewriter.createOrFold<arith::AddIOp>(loc, offsets[axis], sizes[axis]);
+    }
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
 } // namespace
 
 void mlir::tosa::populateTosaToTensorConversionPatterns(
     RewritePatternSet *patterns) {
-  patterns->add<SliceConverter, PadConverter>(patterns->getContext());
+  patterns->add<SliceConverter, PadConverter, ConcatConverter>(
+      patterns->getContext());
+  patterns->add<ReshapeConverterCollapse>(patterns->getContext(),
+                                          /*benefit=*/100);
+  patterns->add<ReshapeConverterExpand>(patterns->getContext(),
+                                        /*benefit=*/200);
+  patterns->add<ReshapeConverterCollapseExpand>(patterns->getContext(),
+                                                /*benefit=*/300);
 }
diff --git a/mlir/lib/Conversion/TosaToTensor/TosaToTensorPass.cpp b/mlir/lib/Conversion/TosaToTensor/TosaToTensorPass.cpp
index af6a08e7bcf14..50dc55667fb94 100644
--- a/mlir/lib/Conversion/TosaToTensor/TosaToTensorPass.cpp
+++ b/mlir/lib/Conversion/TosaToTensor/TosaToTensorPass.cpp
@@ -35,6 +35,8 @@ struct TosaToTensor : public impl::TosaToTensorBase<TosaToTensor> {
   void runOnOperation() override {
     RewritePatternSet patterns(&getContext());
     ConversionTarget target(getContext());
+    target.addIllegalOp<tosa::ConcatOp>();
+    target.addIllegalOp<tosa::ReshapeOp>();
     target.addIllegalOp<tosa::SliceOp>();
     target.addIllegalOp<tosa::PadOp>();
     target.addLegalDialect<arith::ArithDialect>();
diff --git a/mlir/lib/Conversion/VectorToGPU/CMakeLists.txt b/mlir/lib/Conversion/VectorToGPU/CMakeLists.txt
index d9dbe349a0d63..42f1cd7c98e6c 100644
--- a/mlir/lib/Conversion/VectorToGPU/CMakeLists.txt
+++ b/mlir/lib/Conversion/VectorToGPU/CMakeLists.txt
@@ -16,5 +16,6 @@ add_mlir_conversion_library(MLIRVectorToGPU
   MLIRNVGPUUtils
   MLIRTransforms
   MLIRVectorDialect
+  MLIRVectorTransforms
   MLIRVectorUtils
   )
diff --git a/mlir/lib/Conversion/VectorToGPU/VectorToGPU.cpp b/mlir/lib/Conversion/VectorToGPU/VectorToGPU.cpp
index d9533b4e16b44..cc9813648bcc1 100644
--- a/mlir/lib/Conversion/VectorToGPU/VectorToGPU.cpp
+++ b/mlir/lib/Conversion/VectorToGPU/VectorToGPU.cpp
@@ -24,6 +24,7 @@
 #include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/Dialect/Utils/StructuredOpsUtils.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Dialect/Vector/Transforms/VectorRewritePatterns.h"
 #include "mlir/Dialect/Vector/Utils/VectorUtils.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/IR/BuiltinOps.h"
@@ -1173,9 +1174,8 @@ void mlir::populatePrepareVectorToMMAPatterns(RewritePatternSet &patterns,
         patterns.getContext());
     return;
   }
-  patterns
-      .add<nvgpu::PrepareContractToGPUMMASync, CombineTransferReadOpTranspose>(
-          patterns.getContext());
+  vector::populateVectorContractCanonicalizeMatmulToMMT(patterns);
+  patterns.add<CombineTransferReadOpTranspose>(patterns.getContext());
 }
 
 LogicalResult mlir::convertVectorToMMAOps(RewriterBase &rewriter,
diff --git a/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp b/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp
index f705284845a2d..c56d03f6f31d7 100644
--- a/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp
+++ b/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp
@@ -819,7 +819,7 @@ class VectorMaskOpConversionBase
 
   LogicalResult
   matchAndRewrite(vector::MaskOp maskOp, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override final {
+                  ConversionPatternRewriter &rewriter) const final {
     // Match against the maskable operation kind.
     auto maskedOp = llvm::dyn_cast_or_null<MaskedOp>(maskOp.getMaskableOp());
     if (!maskedOp)
@@ -841,7 +841,7 @@ class MaskedReductionOpConversion
   using VectorMaskOpConversionBase<
       vector::ReductionOp>::VectorMaskOpConversionBase;
 
-  virtual LogicalResult matchAndRewriteMaskableOp(
+  LogicalResult matchAndRewriteMaskableOp(
       vector::MaskOp maskOp, MaskableOpInterface maskableOp,
       ConversionPatternRewriter &rewriter) const override {
     auto reductionOp = cast<ReductionOp>(maskableOp.getOperation());
diff --git a/mlir/lib/Conversion/VectorToSPIRV/CMakeLists.txt b/mlir/lib/Conversion/VectorToSPIRV/CMakeLists.txt
index 7794ce71b024d..bb9f793d7fe0f 100644
--- a/mlir/lib/Conversion/VectorToSPIRV/CMakeLists.txt
+++ b/mlir/lib/Conversion/VectorToSPIRV/CMakeLists.txt
@@ -10,6 +10,7 @@ add_mlir_conversion_library(MLIRVectorToSPIRV
   intrinsics_gen
 
   LINK_LIBS PUBLIC
+  MLIRArithDialect
   MLIRSPIRVDialect
   MLIRSPIRVConversion
   MLIRVectorDialect
diff --git a/mlir/lib/Conversion/VectorToSPIRV/VectorToSPIRV.cpp b/mlir/lib/Conversion/VectorToSPIRV/VectorToSPIRV.cpp
index a3a3a612ed147..20c52f536a23f 100644
--- a/mlir/lib/Conversion/VectorToSPIRV/VectorToSPIRV.cpp
+++ b/mlir/lib/Conversion/VectorToSPIRV/VectorToSPIRV.cpp
@@ -12,6 +12,7 @@
 
 #include "mlir/Conversion/VectorToSPIRV/VectorToSPIRV.h"
 
+#include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/SPIRV/IR/SPIRVDialect.h"
 #include "mlir/Dialect/SPIRV/IR/SPIRVOps.h"
 #include "mlir/Dialect/SPIRV/IR/SPIRVTypes.h"
@@ -20,6 +21,9 @@
 #include "mlir/IR/BuiltinAttributes.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/Matchers.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/TypeUtilities.h"
+#include "mlir/Support/LogicalResult.h"
 #include "mlir/Transforms/DialectConversion.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/STLExtras.h"
@@ -436,6 +440,84 @@ struct VectorShuffleOpConvert final
   }
 };
 
+struct VectorReductionToDotProd final : OpRewritePattern<vector::ReductionOp> {
+  using OpRewritePattern::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::ReductionOp op,
+                                PatternRewriter &rewriter) const override {
+    if (op.getKind() != vector::CombiningKind::ADD)
+      return rewriter.notifyMatchFailure(op, "combining kind is not 'add'");
+
+    auto resultType = dyn_cast<IntegerType>(op.getType());
+    if (!resultType)
+      return rewriter.notifyMatchFailure(op, "result is not an integer");
+
+    int64_t resultBitwidth = resultType.getIntOrFloatBitWidth();
+    if (!llvm::is_contained({32, 64}, resultBitwidth))
+      return rewriter.notifyMatchFailure(op, "unsupported integer bitwidth");
+
+    VectorType inVecTy = op.getSourceVectorType();
+    if (inVecTy.getNumElements() != 4 || inVecTy.getShape().size() != 1 ||
+        inVecTy.isScalable())
+      return rewriter.notifyMatchFailure(op, "unsupported vector shape");
+
+    auto mul = op.getVector().getDefiningOp<arith::MulIOp>();
+    if (!mul)
+      return rewriter.notifyMatchFailure(
+          op, "reduction operand is not 'arith.muli'");
+
+    if (succeeded(handleCase<arith::ExtSIOp, arith::ExtSIOp, spirv::SDotOp,
+                             spirv::SDotAccSatOp, false>(op, mul, rewriter)))
+      return success();
+
+    if (succeeded(handleCase<arith::ExtUIOp, arith::ExtUIOp, spirv::UDotOp,
+                             spirv::UDotAccSatOp, false>(op, mul, rewriter)))
+      return success();
+
+    if (succeeded(handleCase<arith::ExtSIOp, arith::ExtUIOp, spirv::SUDotOp,
+                             spirv::SUDotAccSatOp, false>(op, mul, rewriter)))
+      return success();
+
+    if (succeeded(handleCase<arith::ExtUIOp, arith::ExtSIOp, spirv::SUDotOp,
+                             spirv::SUDotAccSatOp, true>(op, mul, rewriter)))
+      return success();
+
+    return failure();
+  }
+
+private:
+  template <typename LhsExtensionOp, typename RhsExtensionOp, typename DotOp,
+            typename DotAccOp, bool SwapOperands>
+  static LogicalResult handleCase(vector::ReductionOp op, arith::MulIOp mul,
+                                  PatternRewriter &rewriter) {
+    auto lhs = mul.getLhs().getDefiningOp<LhsExtensionOp>();
+    if (!lhs || !getElementTypeOrSelf(lhs.getIn().getType()).isInteger(8))
+      return failure();
+
+    auto rhs = mul.getRhs().getDefiningOp<RhsExtensionOp>();
+    if (!rhs || !getElementTypeOrSelf(rhs.getIn().getType()).isInteger(8))
+      return failure();
+
+    Value lhsIn = lhs.getIn();
+    Value rhsIn = rhs.getIn();
+
+    // There's no variant of dot prod ops for unsigned LHS and signed RHS, so
+    // we have to swap operands instead in that case.
+    if (SwapOperands)
+      std::swap(lhsIn, rhsIn);
+
+    if (Value acc = op.getAcc()) {
+      rewriter.replaceOpWithNewOp<DotAccOp>(op, op.getType(), lhsIn, rhsIn, acc,
+                                            nullptr);
+    } else {
+      rewriter.replaceOpWithNewOp<DotOp>(op, op.getType(), lhsIn, rhsIn,
+                                         nullptr);
+    }
+
+    return success();
+  }
+};
+
 } // namespace
 #define CL_MAX_MIN_OPS                                                         \
   spirv::CLFMaxOp, spirv::CLFMinOp, spirv::CLUMaxOp, spirv::CLUMinOp,          \
@@ -457,3 +539,8 @@ void mlir::populateVectorToSPIRVPatterns(SPIRVTypeConverter &typeConverter,
       VectorShuffleOpConvert, VectorSplatPattern>(typeConverter,
                                                   patterns.getContext());
 }
+
+void mlir::populateVectorReductionToSPIRVDotProductPatterns(
+    RewritePatternSet &patterns) {
+  patterns.add<VectorReductionToDotProd>(patterns.getContext());
+}
diff --git a/mlir/lib/Debug/CMakeLists.txt b/mlir/lib/Debug/CMakeLists.txt
new file mode 100644
index 0000000000000..336749078dae4
--- /dev/null
+++ b/mlir/lib/Debug/CMakeLists.txt
@@ -0,0 +1,12 @@
+add_mlir_library(MLIRDebug
+  DebugCounter.cpp
+  ExecutionContext.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Debug
+
+  LINK_LIBS PUBLIC
+  ${LLVM_PTHREAD_LIB}
+  MLIRIR
+  MLIRSupport
+)
diff --git a/mlir/lib/Support/DebugCounter.cpp b/mlir/lib/Debug/DebugCounter.cpp
similarity index 95%
rename from mlir/lib/Support/DebugCounter.cpp
rename to mlir/lib/Debug/DebugCounter.cpp
index 44bcdf4ce1b75..a8747f9137c9d 100644
--- a/mlir/lib/Support/DebugCounter.cpp
+++ b/mlir/lib/Debug/DebugCounter.cpp
@@ -6,13 +6,14 @@
 //
 //===----------------------------------------------------------------------===//
 
-#include "mlir/Support/DebugCounter.h"
+#include "mlir/Debug/Counter.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/Format.h"
 #include "llvm/Support/ManagedStatic.h"
 
 using namespace mlir;
+using namespace mlir::tracing;
 
 //===----------------------------------------------------------------------===//
 // DebugCounter CommandLine Options
@@ -61,9 +62,13 @@ void DebugCounter::addCounter(StringRef actionTag, int64_t countToSkip,
   counters.try_emplace(actionTag, countToSkip, countToStopAfter);
 }
 
-// Register a counter with the specified name.
-FailureOr<bool> DebugCounter::shouldExecute(StringRef tag,
-                                            StringRef description) {
+void DebugCounter::operator()(llvm::function_ref<void()> transform,
+                              const Action &action) {
+  if (shouldExecute(action.getTag()))
+    transform();
+}
+
+bool DebugCounter::shouldExecute(StringRef tag) {
   auto counterIt = counters.find(tag);
   if (counterIt == counters.end())
     return true;
diff --git a/mlir/lib/Debug/ExecutionContext.cpp b/mlir/lib/Debug/ExecutionContext.cpp
new file mode 100644
index 0000000000000..ee7c33a6b3f14
--- /dev/null
+++ b/mlir/lib/Debug/ExecutionContext.cpp
@@ -0,0 +1,97 @@
+//===- ExecutionContext.cpp - Debug Execution Context Support -------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Debug/ExecutionContext.h"
+
+#include "llvm/ADT/ScopeExit.h"
+
+#include <cstddef>
+
+using namespace mlir;
+using namespace mlir::tracing;
+
+//===----------------------------------------------------------------------===//
+// ExecutionContext
+//===----------------------------------------------------------------------===//
+
+static const thread_local ActionActiveStack *actionStack = nullptr;
+
+void ExecutionContext::setCallback(CallbackTy callback) {
+  onBreakpointControlExecutionCallback = callback;
+}
+
+void ExecutionContext::registerObserver(Observer *observer) {
+  observers.push_back(observer);
+}
+
+void ExecutionContext::operator()(llvm::function_ref<void()> transform,
+                                  const Action &action) {
+  // Update the top of the stack with the current action.
+  int depth = 0;
+  if (actionStack)
+    depth = actionStack->getDepth() + 1;
+  ActionActiveStack info{actionStack, action, depth};
+  actionStack = &info;
+  auto raii = llvm::make_scope_exit([&]() { actionStack = info.getParent(); });
+  Breakpoint *breakpoint = nullptr;
+
+  // Invoke the callback here and handles control requests here.
+  auto handleUserInput = [&]() -> bool {
+    if (!onBreakpointControlExecutionCallback)
+      return true;
+    auto todoNext = onBreakpointControlExecutionCallback(actionStack);
+    switch (todoNext) {
+    case ExecutionContext::Apply:
+      depthToBreak = std::nullopt;
+      return true;
+    case ExecutionContext::Skip:
+      depthToBreak = std::nullopt;
+      return false;
+    case ExecutionContext::Step:
+      depthToBreak = depth + 1;
+      return true;
+    case ExecutionContext::Next:
+      depthToBreak = depth;
+      return true;
+    case ExecutionContext::Finish:
+      depthToBreak = depth - 1;
+      return true;
+    }
+    llvm::report_fatal_error("Unknown control request");
+  };
+
+  // Try to find a breakpoint that would hit on this action.
+  // Right now there is no way to collect them all, we stop at the first one.
+  for (auto *breakpointManager : breakpoints) {
+    breakpoint = breakpointManager->match(action);
+    if (breakpoint)
+      break;
+  }
+
+  bool shouldExecuteAction = true;
+  // If we have a breakpoint, or if `depthToBreak` was previously set and the
+  // current depth matches, we invoke the user-provided callback.
+  if (breakpoint || (depthToBreak && depth <= depthToBreak))
+    shouldExecuteAction = handleUserInput();
+
+  // Notify the observers about the current action.
+  for (auto *observer : observers)
+    observer->beforeExecute(actionStack, breakpoint, shouldExecuteAction);
+
+  if (shouldExecuteAction) {
+    // Execute the action here.
+    transform();
+
+    // Notify the observers about completion of the action.
+    for (auto *observer : observers)
+      observer->afterExecute(actionStack);
+  }
+
+  if (depthToBreak && depth <= depthToBreak)
+    handleUserInput();
+}
diff --git a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
index 6e730140a3b94..d7720a052e0dd 100644
--- a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
@@ -172,10 +172,8 @@ bool mlir::isLoopMemoryParallel(AffineForOp forOp) {
     MemRefAccess srcAccess(srcOp);
     for (auto *dstOp : loadAndStoreOps) {
       MemRefAccess dstAccess(dstOp);
-      FlatAffineValueConstraints dependenceConstraints;
-      DependenceResult result = checkMemrefAccessDependence(
-          srcAccess, dstAccess, depth, &dependenceConstraints,
-          /*dependenceComponents=*/nullptr);
+      DependenceResult result =
+          checkMemrefAccessDependence(srcAccess, dstAccess, depth);
       if (result.value != DependenceResult::NoDependence)
         return false;
     }
@@ -260,7 +258,8 @@ LogicalResult mlir::getIndexSet(MutableArrayRef<Operation *> ops,
   }
   extractInductionVars(loopOps, indices);
   // Reset while associating Values in 'indices' to the domain.
-  domain->reset(numDims, /*numSymbols=*/0, /*numLocals=*/0, indices);
+  *domain = FlatAffineValueConstraints(numDims, /*numSymbols=*/0,
+                                       /*numLocals=*/0, indices);
   for (Operation *op : ops) {
     // Add constraints from forOp's bounds.
     if (AffineForOp forOp = dyn_cast<AffineForOp>(op)) {
@@ -287,14 +286,6 @@ static LogicalResult getOpIndexSet(Operation *op,
   return getIndexSet(ops, indexSet);
 }
 
-/// Returns true if `val` is an induction of an affine.parallel op.
-static bool isAffineParallelInductionVar(Value val) {
-  auto ivArg = val.dyn_cast<BlockArgument>();
-  if (!ivArg)
-    return false;
-  return isa<AffineParallelOp>(ivArg.getOwner()->getParentOp());
-}
-
 // Returns the number of outer loop common to 'src/dstDomain'.
 // Loops common to 'src/dst' domains are added to 'commonLoops' if non-null.
 static unsigned
@@ -322,9 +313,10 @@ getNumCommonLoops(const FlatAffineValueConstraints &srcDomain,
 }
 
 /// Returns the closest surrounding block common to `opA` and `opB`. `opA` and
-/// `opB` should be in the same affine scope and thus such a block is guaranteed
-/// to exist.
-static Block *getCommonBlock(Operation *opA, Operation *opB) {
+/// `opB` should be in the same affine scope. Returns nullptr if such a block
+/// does not exist (when the two ops are in different blocks of an op starting
+/// an `AffineScope`).
+static Block *getCommonBlockInAffineScope(Operation *opA, Operation *opB) {
   // Get the chain of ancestor blocks for the given `MemRefAccess` instance. The
   // chain extends up to and includnig an op that starts an affine scope.
   auto getChainOfAncestorBlocks =
@@ -342,7 +334,7 @@ static Block *getCommonBlock(Operation *opA, Operation *opB) {
         ancestorBlocks.push_back(currBlock);
       };
 
-  // Find the closest common block including those in AffineIf.
+  // Find the closest common block.
   SmallVector<Block *, 4> srcAncestorBlocks, dstAncestorBlocks;
   getChainOfAncestorBlocks(opA, srcAncestorBlocks);
   getChainOfAncestorBlocks(opB, dstAncestorBlocks);
@@ -352,28 +344,31 @@ static Block *getCommonBlock(Operation *opA, Operation *opB) {
        i >= 0 && j >= 0 && srcAncestorBlocks[i] == dstAncestorBlocks[j];
        i--, j--)
     commonBlock = srcAncestorBlocks[i];
-  // This is guaranteed since both ops are from the same affine scope.
-  assert(commonBlock && "ops expected to have a common surrounding block");
+
   return commonBlock;
 }
 
 /// Returns true if the ancestor operation of 'srcAccess' appears before the
 /// ancestor operation of 'dstAccess' in their common ancestral block. The
 /// operations for `srcAccess` and `dstAccess` are expected to be in the same
-/// affine scope.
+/// affine scope and have a common surrounding block within it.
 static bool srcAppearsBeforeDstInAncestralBlock(const MemRefAccess &srcAccess,
                                                 const MemRefAccess &dstAccess) {
   // Get Block common to 'srcAccess.opInst' and 'dstAccess.opInst'.
-  auto *commonBlock = getCommonBlock(srcAccess.opInst, dstAccess.opInst);
+  Block *commonBlock =
+      getCommonBlockInAffineScope(srcAccess.opInst, dstAccess.opInst);
+  assert(commonBlock &&
+         "ops expected to have a common surrounding block in affine scope");
+
   // Check the dominance relationship between the respective ancestors of the
   // src and dst in the Block of the innermost among the common loops.
-  auto *srcInst = commonBlock->findAncestorOpInBlock(*srcAccess.opInst);
-  assert(srcInst && "src access op must lie in common block");
-  auto *dstInst = commonBlock->findAncestorOpInBlock(*dstAccess.opInst);
-  assert(dstInst && "dest access op must lie in common block");
+  Operation *srcOp = commonBlock->findAncestorOpInBlock(*srcAccess.opInst);
+  assert(srcOp && "src access op must lie in common block");
+  Operation *dstOp = commonBlock->findAncestorOpInBlock(*dstAccess.opInst);
+  assert(dstOp && "dest access op must lie in common block");
 
-  // Determine whether dstInst comes after srcInst.
-  return srcInst->isBeforeInBlock(dstInst);
+  // Determine whether dstOp comes after srcOp.
+  return srcOp->isBeforeInBlock(dstOp);
 }
 
 // Adds ordering constraints to 'dependenceDomain' based on number of loops
@@ -607,8 +602,8 @@ DependenceResult mlir::checkMemrefAccessDependence(
     SmallVector<DependenceComponent, 2> *dependenceComponents, bool allowRAR) {
   LLVM_DEBUG(llvm::dbgs() << "Checking for dependence at depth: "
                           << Twine(loopDepth) << " between:\n";);
-  LLVM_DEBUG(srcAccess.opInst->dump(););
-  LLVM_DEBUG(dstAccess.opInst->dump(););
+  LLVM_DEBUG(srcAccess.opInst->dump());
+  LLVM_DEBUG(dstAccess.opInst->dump());
 
   // Return 'NoDependence' if these accesses do not access the same memref.
   if (srcAccess.memref != dstAccess.memref)
@@ -620,9 +615,12 @@ DependenceResult mlir::checkMemrefAccessDependence(
       !isa<AffineWriteOpInterface>(dstAccess.opInst))
     return DependenceResult::NoDependence;
 
-  // We can't analyze further if the ops lie in different affine scopes.
+  // We can't analyze further if the ops lie in different affine scopes or have
+  // no common block in an affine scope.
   if (getAffineScope(srcAccess.opInst) != getAffineScope(dstAccess.opInst))
     return DependenceResult::Failure;
+  if (!getCommonBlockInAffineScope(srcAccess.opInst, dstAccess.opInst))
+    return DependenceResult::Failure;
 
   // Create access relation from each MemRefAccess.
   FlatAffineRelation srcRel, dstRel;
@@ -652,23 +650,24 @@ DependenceResult mlir::checkMemrefAccessDependence(
   // memory locations.
   dstRel.inverse();
   dstRel.compose(srcRel);
-  *dependenceConstraints = dstRel;
 
   // Add 'src' happens before 'dst' ordering constraints.
-  addOrderingConstraints(srcDomain, dstDomain, loopDepth,
-                         dependenceConstraints);
+  addOrderingConstraints(srcDomain, dstDomain, loopDepth, &dstRel);
 
   // Return 'NoDependence' if the solution space is empty: no dependence.
-  if (dependenceConstraints->isEmpty())
+  if (dstRel.isEmpty())
     return DependenceResult::NoDependence;
 
   // Compute dependence direction vector and return true.
   if (dependenceComponents != nullptr)
-    computeDirectionVector(srcDomain, dstDomain, loopDepth,
-                           dependenceConstraints, dependenceComponents);
+    computeDirectionVector(srcDomain, dstDomain, loopDepth, &dstRel,
+                           dependenceComponents);
 
   LLVM_DEBUG(llvm::dbgs() << "Dependence polyhedron:\n");
-  LLVM_DEBUG(dependenceConstraints->dump());
+  LLVM_DEBUG(dstRel.dump());
+
+  if (dependenceConstraints)
+    *dependenceConstraints = dstRel;
   return DependenceResult::HasDependence;
 }
 
@@ -693,12 +692,12 @@ void mlir::getDependenceComponents(
         auto *dstOp = loadAndStoreOps[j];
         MemRefAccess dstAccess(dstOp);
 
-        FlatAffineValueConstraints dependenceConstraints;
         SmallVector<DependenceComponent, 2> depComps;
         // TODO: Explore whether it would be profitable to pre-compute and store
         // deps instead of repeatedly checking.
         DependenceResult result = checkMemrefAccessDependence(
-            srcAccess, dstAccess, d, &dependenceConstraints, &depComps);
+            srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
+            &depComps);
         if (hasDependence(result))
           depCompsVec->push_back(depComps);
       }
diff --git a/mlir/lib/Dialect/Affine/Analysis/AffineStructures.cpp b/mlir/lib/Dialect/Affine/Analysis/AffineStructures.cpp
index 3e7d81113bcba..9c9d089416c5e 100644
--- a/mlir/lib/Dialect/Affine/Analysis/AffineStructures.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/AffineStructures.cpp
@@ -75,7 +75,8 @@ getFlattenedAffineExprs(ArrayRef<AffineExpr> exprs, unsigned numDims,
                         std::vector<SmallVector<int64_t, 8>> *flattenedExprs,
                         FlatAffineValueConstraints *localVarCst) {
   if (exprs.empty()) {
-    localVarCst->reset(numDims, numSymbols);
+    if (localVarCst)
+      *localVarCst = FlatAffineValueConstraints(numDims, numSymbols);
     return success();
   }
 
@@ -120,7 +121,9 @@ LogicalResult mlir::getFlattenedAffineExprs(
     AffineMap map, std::vector<SmallVector<int64_t, 8>> *flattenedExprs,
     FlatAffineValueConstraints *localVarCst) {
   if (map.getNumResults() == 0) {
-    localVarCst->reset(map.getNumDims(), map.getNumSymbols());
+    if (localVarCst)
+      *localVarCst =
+          FlatAffineValueConstraints(map.getNumDims(), map.getNumSymbols());
     return success();
   }
   return ::getFlattenedAffineExprs(map.getResults(), map.getNumDims(),
@@ -132,7 +135,9 @@ LogicalResult mlir::getFlattenedAffineExprs(
     IntegerSet set, std::vector<SmallVector<int64_t, 8>> *flattenedExprs,
     FlatAffineValueConstraints *localVarCst) {
   if (set.getNumConstraints() == 0) {
-    localVarCst->reset(set.getNumDims(), set.getNumSymbols());
+    if (localVarCst)
+      *localVarCst =
+          FlatAffineValueConstraints(set.getNumDims(), set.getNumSymbols());
     return success();
   }
   return ::getFlattenedAffineExprs(set.getConstraints(), set.getNumDims(),
@@ -231,50 +236,6 @@ FlatAffineValueConstraints::getHyperrectangular(ValueRange ivs, ValueRange lbs,
   return res;
 }
 
-void FlatAffineValueConstraints::reset(unsigned numReservedInequalities,
-                                       unsigned numReservedEqualities,
-                                       unsigned newNumReservedCols,
-                                       unsigned newNumDims,
-                                       unsigned newNumSymbols,
-                                       unsigned newNumLocals) {
-  assert(newNumReservedCols >= newNumDims + newNumSymbols + newNumLocals + 1 &&
-         "minimum 1 column");
-  *this = FlatAffineValueConstraints(numReservedInequalities,
-                                     numReservedEqualities, newNumReservedCols,
-                                     newNumDims, newNumSymbols, newNumLocals);
-}
-
-void FlatAffineValueConstraints::reset(unsigned newNumDims,
-                                       unsigned newNumSymbols,
-                                       unsigned newNumLocals) {
-  reset(/*numReservedInequalities=*/0, /*numReservedEqualities=*/0,
-        /*numReservedCols=*/newNumDims + newNumSymbols + newNumLocals + 1,
-        newNumDims, newNumSymbols, newNumLocals);
-}
-
-void FlatAffineValueConstraints::reset(
-    unsigned numReservedInequalities, unsigned numReservedEqualities,
-    unsigned newNumReservedCols, unsigned newNumDims, unsigned newNumSymbols,
-    unsigned newNumLocals, ArrayRef<Value> valArgs) {
-  assert(newNumReservedCols >= newNumDims + newNumSymbols + newNumLocals + 1 &&
-         "minimum 1 column");
-  SmallVector<std::optional<Value>, 8> newVals;
-  if (!valArgs.empty())
-    newVals.assign(valArgs.begin(), valArgs.end());
-
-  *this = FlatAffineValueConstraints(
-      numReservedInequalities, numReservedEqualities, newNumReservedCols,
-      newNumDims, newNumSymbols, newNumLocals, newVals);
-}
-
-void FlatAffineValueConstraints::reset(unsigned newNumDims,
-                                       unsigned newNumSymbols,
-                                       unsigned newNumLocals,
-                                       ArrayRef<Value> valArgs) {
-  reset(0, 0, newNumDims + newNumSymbols + newNumLocals + 1, newNumDims,
-        newNumSymbols, newNumLocals, valArgs);
-}
-
 unsigned FlatAffineValueConstraints::appendDimVar(ValueRange vals) {
   unsigned pos = getNumDimVars();
   return insertVar(VarKind::SetDim, pos, vals);
@@ -920,7 +881,8 @@ static bool detectAsFloorDiv(const FlatAffineValueConstraints &cst,
 std::pair<AffineMap, AffineMap>
 FlatAffineValueConstraints::getLowerAndUpperBound(
     unsigned pos, unsigned offset, unsigned num, unsigned symStartPos,
-    ArrayRef<AffineExpr> localExprs, MLIRContext *context) const {
+    ArrayRef<AffineExpr> localExprs, MLIRContext *context,
+    bool closedUB) const {
   assert(pos + offset < getNumDimVars() && "invalid dim start pos");
   assert(symStartPos >= (pos + offset) && "invalid sym start pos");
   assert(getNumLocalVars() == localExprs.size() &&
@@ -970,8 +932,8 @@ FlatAffineValueConstraints::getLowerAndUpperBound(
     auto expr =
         getAffineExprFromFlatForm(ub, dimCount, symCount, localExprs, context);
     expr = expr.floorDiv(std::abs(ineq[pos + offset]));
-    // Upper bound is exclusive.
-    ubExprs.push_back(expr + 1);
+    int64_t ubAdjustment = closedUB ? 0 : 1;
+    ubExprs.push_back(expr + ubAdjustment);
   }
 
   // Equalities. It's both a lower and a upper bound.
@@ -1009,7 +971,7 @@ FlatAffineValueConstraints::getLowerAndUpperBound(
 void FlatAffineValueConstraints::getSliceBounds(
     unsigned offset, unsigned num, MLIRContext *context,
     SmallVectorImpl<AffineMap> *lbMaps, SmallVectorImpl<AffineMap> *ubMaps,
-    bool getClosedUB) {
+    bool closedUB) {
   assert(num < getNumDimVars() && "invalid range");
 
   // Basic simplification.
@@ -1108,7 +1070,7 @@ void FlatAffineValueConstraints::getSliceBounds(
     // again.
   } while (changed);
 
-  int64_t ubAdjustment = getClosedUB ? 0 : 1;
+  int64_t ubAdjustment = closedUB ? 0 : 1;
 
   // Set the lower and upper bound maps for all the variables that were
   // computed as affine expressions of the rest as the "detected expr" and
@@ -1140,7 +1102,8 @@ void FlatAffineValueConstraints::getSliceBounds(
           tmpClone->removeRedundantInequalities();
         }
         std::tie(lbMap, ubMap) = tmpClone->getLowerAndUpperBound(
-            pos, offset, num, getNumDimVars(), /*localExprs=*/{}, context);
+            pos, offset, num, getNumDimVars(), /*localExprs=*/{}, context,
+            closedUB);
       }
 
       // If the above fails, we'll just use the constant lower bound and the
diff --git a/mlir/lib/Dialect/Affine/Analysis/Utils.cpp b/mlir/lib/Dialect/Affine/Analysis/Utils.cpp
index 581ffb0ad2f66..8ab219af98c9d 100644
--- a/mlir/lib/Dialect/Affine/Analysis/Utils.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/Utils.cpp
@@ -65,7 +65,8 @@ void mlir::getEnclosingAffineOps(Operation &op,
 LogicalResult
 ComputationSliceState::getSourceAsConstraints(FlatAffineValueConstraints &cst) {
   assert(!ivs.empty() && "Cannot have a slice without its IVs");
-  cst.reset(/*numDims=*/ivs.size(), /*numSymbols=*/0, /*numLocals=*/0, ivs);
+  cst = FlatAffineValueConstraints(/*numDims=*/ivs.size(), /*numSymbols=*/0,
+                                   /*numLocals=*/0, ivs);
   for (Value iv : ivs) {
     AffineForOp loop = getForInductionVarOwner(iv);
     assert(loop && "Expected affine for");
@@ -87,7 +88,7 @@ ComputationSliceState::getAsConstraints(FlatAffineValueConstraints *cst) {
   SmallVector<Value, 4> values(ivs);
   // Append 'ivs' then 'operands' to 'values'.
   values.append(lbOperands[0].begin(), lbOperands[0].end());
-  cst->reset(numDims, numSymbols, 0, values);
+  *cst = FlatAffineValueConstraints(numDims, numSymbols, 0, values);
 
   // Add loop bound constraints for values which are loop IVs of the destination
   // of fusion and equality constraints for symbols which are constants.
@@ -126,7 +127,7 @@ void ComputationSliceState::dump() const {
     llvm::errs() << "\t\t" << iv << "\n";
 
   llvm::errs() << "\tLBs:\n";
-  for (auto &en : llvm::enumerate(lbs)) {
+  for (auto en : llvm::enumerate(lbs)) {
     llvm::errs() << "\t\t" << en.value() << "\n";
     llvm::errs() << "\t\tOperands:\n";
     for (Value lbOp : lbOperands[en.index()])
@@ -134,7 +135,7 @@ void ComputationSliceState::dump() const {
   }
 
   llvm::errs() << "\tUBs:\n";
-  for (auto &en : llvm::enumerate(ubs)) {
+  for (auto en : llvm::enumerate(ubs)) {
     llvm::errs() << "\t\t" << en.value() << "\n";
     llvm::errs() << "\t\tOperands:\n";
     for (Value ubOp : ubOperands[en.index()])
@@ -293,9 +294,9 @@ std::optional<bool> ComputationSliceState::isMaximal() const {
     return isMaximalFastCheck;
 
   // Create constraints for the src loop nest being sliced.
-  FlatAffineValueConstraints srcConstraints;
-  srcConstraints.reset(/*numDims=*/ivs.size(), /*numSymbols=*/0,
-                       /*numLocals=*/0, ivs);
+  FlatAffineValueConstraints srcConstraints(/*numDims=*/ivs.size(),
+                                            /*numSymbols=*/0,
+                                            /*numLocals=*/0, ivs);
   for (Value iv : ivs) {
     AffineForOp loop = getForInductionVarOwner(iv);
     assert(loop && "Expected affine for");
@@ -305,7 +306,7 @@ std::optional<bool> ComputationSliceState::isMaximal() const {
 
   // Create constraints for the slice using the dst loop nest information. We
   // retrieve existing dst loops from the lbOperands.
-  SmallVector<Value, 8> consumerIVs;
+  SmallVector<Value> consumerIVs;
   for (Value lbOp : lbOperands[0])
     if (getForInductionVarOwner(lbOp))
       consumerIVs.push_back(lbOp);
@@ -315,9 +316,9 @@ std::optional<bool> ComputationSliceState::isMaximal() const {
   for (int i = consumerIVs.size(), end = ivs.size(); i < end; ++i)
     consumerIVs.push_back(Value());
 
-  FlatAffineValueConstraints sliceConstraints;
-  sliceConstraints.reset(/*numDims=*/consumerIVs.size(), /*numSymbols=*/0,
-                         /*numLocals=*/0, consumerIVs);
+  FlatAffineValueConstraints sliceConstraints(/*numDims=*/consumerIVs.size(),
+                                              /*numSymbols=*/0,
+                                              /*numLocals=*/0, consumerIVs);
 
   if (failed(sliceConstraints.addDomainFromSliceMaps(lbs, ubs, lbOperands[0])))
     return std::nullopt;
@@ -454,19 +455,17 @@ LogicalResult MemRefRegion::compute(Operation *op, unsigned loopDepth,
   unsigned rank = access.getRank();
 
   LLVM_DEBUG(llvm::dbgs() << "MemRefRegion::compute: " << *op
-                          << "depth: " << loopDepth << "\n";);
+                          << "\ndepth: " << loopDepth << "\n";);
 
   // 0-d memrefs.
   if (rank == 0) {
-    SmallVector<AffineForOp, 4> ivs;
-    getAffineForIVs(*op, &ivs);
+    SmallVector<Value, 4> ivs;
+    getAffineIVs(*op, ivs);
     assert(loopDepth <= ivs.size() && "invalid 'loopDepth'");
     // The first 'loopDepth' IVs are symbols for this region.
     ivs.resize(loopDepth);
-    SmallVector<Value, 4> regionSymbols;
-    extractForInductionVars(ivs, &regionSymbols);
     // A 0-d memref has a 0-d region.
-    cst.reset(rank, loopDepth, /*numLocals=*/0, regionSymbols);
+    cst = FlatAffineValueConstraints(rank, loopDepth, /*numLocals=*/0, ivs);
     return success();
   }
 
@@ -497,29 +496,30 @@ LogicalResult MemRefRegion::compute(Operation *op, unsigned loopDepth,
   // We'll first associate the dims and symbols of the access map to the dims
   // and symbols resp. of cst. This will change below once cst is
   // fully constructed out.
-  cst.reset(numDims, numSymbols, 0, operands);
+  cst = FlatAffineValueConstraints(numDims, numSymbols, 0, operands);
 
   // Add equality constraints.
   // Add inequalities for loop lower/upper bounds.
   for (unsigned i = 0; i < numDims + numSymbols; ++i) {
     auto operand = operands[i];
-    if (auto loop = getForInductionVarOwner(operand)) {
+    if (auto affineFor = getForInductionVarOwner(operand)) {
       // Note that cst can now have more dimensions than accessMap if the
       // bounds expressions involve outer loops or other symbols.
       // TODO: rewrite this to use getInstIndexSet; this way
       // conditionals will be handled when the latter supports it.
-      if (failed(cst.addAffineForOpDomain(loop)))
+      if (failed(cst.addAffineForOpDomain(affineFor)))
         return failure();
-    } else {
-      // Has to be a valid symbol.
-      auto symbol = operand;
-      assert(isValidSymbol(symbol));
+    } else if (auto parallelOp = getAffineParallelInductionVarOwner(operand)) {
+      if (failed(cst.addAffineParallelOpDomain(parallelOp)))
+        return failure();
+    } else if (isValidSymbol(operand)) {
       // Check if the symbol is a constant.
-      if (auto *op = symbol.getDefiningOp()) {
-        if (auto constOp = dyn_cast<arith::ConstantIndexOp>(op)) {
-          cst.addBound(FlatAffineValueConstraints::EQ, symbol, constOp.value());
-        }
-      }
+      Value symbol = operand;
+      if (auto constVal = getConstantIntValue(symbol))
+        cst.addBound(FlatAffineValueConstraints::EQ, symbol, constVal.value());
+    } else {
+      LLVM_DEBUG(llvm::dbgs() << "unknown affine dimensional value");
+      return failure();
     }
   }
 
@@ -552,16 +552,14 @@ LogicalResult MemRefRegion::compute(Operation *op, unsigned loopDepth,
 
   // Eliminate any loop IVs other than the outermost 'loopDepth' IVs, on which
   // this memref region is symbolic.
-  SmallVector<AffineForOp, 4> enclosingIVs;
-  getAffineForIVs(*op, &enclosingIVs);
+  SmallVector<Value, 4> enclosingIVs;
+  getAffineIVs(*op, enclosingIVs);
   assert(loopDepth <= enclosingIVs.size() && "invalid loop depth");
   enclosingIVs.resize(loopDepth);
   SmallVector<Value, 4> vars;
   cst.getValues(cst.getNumDimVars(), cst.getNumDimAndSymbolVars(), &vars);
-  for (auto var : vars) {
-    AffineForOp iv;
-    if ((iv = getForInductionVarOwner(var)) &&
-        !llvm::is_contained(enclosingIVs, iv)) {
+  for (Value var : vars) {
+    if ((isAffineInductionVar(var)) && !llvm::is_contained(enclosingIVs, var)) {
       cst.projectOut(var);
     }
   }
@@ -1264,21 +1262,19 @@ bool MemRefAccess::operator==(const MemRefAccess &rhs) const {
                       [](AffineExpr e) { return e == 0; });
 }
 
-/// Populates 'loops' with IVs of the surrounding affine.for and affine.parallel
-/// ops ordered from the outermost one to the innermost.
-static void getAffineIVs(Operation &op, SmallVectorImpl<Value> &loops) {
+void mlir::getAffineIVs(Operation &op, SmallVectorImpl<Value> &ivs) {
   auto *currOp = op.getParentOp();
   AffineForOp currAffineForOp;
-  // Traverse up the hierarchy collecting all 'affine.for' operation while
-  // skipping over 'affine.if' operations.
+  // Traverse up the hierarchy collecting all 'affine.for' and affine.parallel
+  // operation while skipping over 'affine.if' operations.
   while (currOp) {
     if (AffineForOp currAffineForOp = dyn_cast<AffineForOp>(currOp))
-      loops.push_back(currAffineForOp.getInductionVar());
+      ivs.push_back(currAffineForOp.getInductionVar());
     else if (auto parOp = dyn_cast<AffineParallelOp>(currOp))
-      llvm::append_range(loops, parOp.getIVs());
+      llvm::append_range(ivs, parOp.getIVs());
     currOp = currOp->getParentOp();
   }
-  std::reverse(loops.begin(), loops.end());
+  std::reverse(ivs.begin(), ivs.end());
 }
 
 /// Returns the number of surrounding loops common to 'loopsA' and 'loopsB',
diff --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index e74829ac7758d..5ed434fa1e809 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -2504,8 +2504,14 @@ bool mlir::isAffineForInductionVar(Value val) {
   return getForInductionVarOwner(val) != AffineForOp();
 }
 
-/// Returns the loop parent of an induction variable. If the provided value is
-/// not an induction variable, then return nullptr.
+bool mlir::isAffineParallelInductionVar(Value val) {
+  return getAffineParallelInductionVarOwner(val) != nullptr;
+}
+
+bool mlir::isAffineInductionVar(Value val) {
+  return isAffineForInductionVar(val) || isAffineParallelInductionVar(val);
+}
+
 AffineForOp mlir::getForInductionVarOwner(Value val) {
   auto ivArg = val.dyn_cast<BlockArgument>();
   if (!ivArg || !ivArg.getOwner())
@@ -2517,6 +2523,17 @@ AffineForOp mlir::getForInductionVarOwner(Value val) {
   return AffineForOp();
 }
 
+AffineParallelOp mlir::getAffineParallelInductionVarOwner(Value val) {
+  auto ivArg = val.dyn_cast<BlockArgument>();
+  if (!ivArg || !ivArg.getOwner())
+    return nullptr;
+  Operation *containingOp = ivArg.getOwner()->getParentOp();
+  auto parallelOp = dyn_cast<AffineParallelOp>(containingOp);
+  if (parallelOp && llvm::is_contained(parallelOp.getIVs(), val))
+    return parallelOp;
+  return nullptr;
+}
+
 /// Extracts the induction variables from a list of AffineForOps and returns
 /// them.
 void mlir::extractForInductionVars(ArrayRef<AffineForOp> forInsts,
@@ -2724,7 +2741,7 @@ struct AlwaysTrueOrFalseIf : public OpRewritePattern<AffineIfOp> {
     Operation *blockToMoveTerminator = blockToMove->getTerminator();
     // Promote the "blockToMove" block to the parent operation block between the
     // prologue and epilogue of "op".
-    rewriter.mergeBlockBefore(blockToMove, op);
+    rewriter.inlineBlockBefore(blockToMove, op);
     // Replace the "op" operation with the operands of the
     // "blockToMoveTerminator" operation. Note that "blockToMoveTerminator" is
     // the affine.yield operation present in the "blockToMove" block. It has no
diff --git a/mlir/lib/Dialect/Affine/TransformOps/AffineTransformOps.cpp b/mlir/lib/Dialect/Affine/TransformOps/AffineTransformOps.cpp
index 19ee04d10b19a..99dfaa9dee1d2 100644
--- a/mlir/lib/Dialect/Affine/TransformOps/AffineTransformOps.cpp
+++ b/mlir/lib/Dialect/Affine/TransformOps/AffineTransformOps.cpp
@@ -16,7 +16,6 @@
 #include "mlir/Dialect/PDL/IR/PDLTypes.h"
 #include "mlir/Dialect/Transform/IR/TransformDialect.h"
 #include "mlir/Dialect/Transform/IR/TransformInterfaces.h"
-#include "mlir/Dialect/Transform/IR/TransformUtils.h"
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 
 using namespace mlir;
diff --git a/mlir/lib/Dialect/Affine/Transforms/AffineExpandIndexOps.cpp b/mlir/lib/Dialect/Affine/Transforms/AffineExpandIndexOps.cpp
index 0e30b8b22cc15..dfb33d79f8bc6 100644
--- a/mlir/lib/Dialect/Affine/Transforms/AffineExpandIndexOps.cpp
+++ b/mlir/lib/Dialect/Affine/Transforms/AffineExpandIndexOps.cpp
@@ -13,6 +13,7 @@
 #include "mlir/Dialect/Affine/Passes.h"
 
 #include "mlir/Dialect/Affine/IR/AffineOps.h"
+#include "mlir/Dialect/Affine/Transforms/Transforms.h"
 #include "mlir/Dialect/Affine/Utils.h"
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 
diff --git a/mlir/lib/Dialect/Affine/Transforms/CMakeLists.txt b/mlir/lib/Dialect/Affine/Transforms/CMakeLists.txt
index 02f4a51f6e97e..0535b8b5d8f52 100644
--- a/mlir/lib/Dialect/Affine/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/Affine/Transforms/CMakeLists.txt
@@ -5,6 +5,7 @@ add_mlir_dialect_library(MLIRAffineTransforms
   AffineLoopNormalize.cpp
   AffineParallelize.cpp
   AffineScalarReplacement.cpp
+  DecomposeAffineOps.cpp
   LoopCoalescing.cpp
   LoopFusion.cpp
   LoopTiling.cpp
diff --git a/mlir/lib/Dialect/Affine/Transforms/DecomposeAffineOps.cpp b/mlir/lib/Dialect/Affine/Transforms/DecomposeAffineOps.cpp
new file mode 100644
index 0000000000000..daa58b5d8986b
--- /dev/null
+++ b/mlir/lib/Dialect/Affine/Transforms/DecomposeAffineOps.cpp
@@ -0,0 +1,171 @@
+//===- DecomposeAffineOps.cpp - Decompose affine ops into finer-grained ---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements functionality to progressively decompose coarse-grained
+// affine ops into finer-grained ops.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Affine/IR/AffineOps.h"
+#include "mlir/Dialect/Affine/Transforms/Transforms.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+#include "llvm/Support/Debug.h"
+
+using namespace mlir;
+
+#define DEBUG_TYPE "decompose-affine-ops"
+#define DBGS() (llvm::dbgs() << "[" DEBUG_TYPE "]: ")
+#define DBGSNL() (llvm::dbgs() << "\n")
+
+/// Count the number of loops surrounding `operand` such that operand could be
+/// hoisted above.
+/// Stop counting at the first loop over which the operand cannot be hoisted.
+static int64_t numEnclosingInvariantLoops(OpOperand &operand) {
+  int64_t count = 0;
+  Operation *currentOp = operand.getOwner();
+  while (auto loopOp = currentOp->getParentOfType<LoopLikeOpInterface>()) {
+    if (!loopOp.isDefinedOutsideOfLoop(operand.get()))
+      break;
+    currentOp = loopOp;
+    count++;
+  }
+  return count;
+}
+
+void mlir::reorderOperandsByHoistability(RewriterBase &rewriter,
+                                         AffineApplyOp op) {
+  SmallVector<int64_t> numInvariant = llvm::to_vector(
+      llvm::map_range(op->getOpOperands(), [&](OpOperand &operand) {
+        return numEnclosingInvariantLoops(operand);
+      }));
+
+  int64_t numOperands = op.getNumOperands();
+  SmallVector<int64_t> operandPositions =
+      llvm::to_vector(llvm::seq<int64_t>(0, numOperands));
+  llvm::stable_sort(operandPositions, [&numInvariant](size_t i1, size_t i2) {
+    return numInvariant[i1] > numInvariant[i2];
+  });
+
+  SmallVector<AffineExpr> replacements(numOperands);
+  SmallVector<Value> operands(numOperands);
+  for (int64_t i = 0; i < numOperands; ++i) {
+    operands[i] = op.getOperand(operandPositions[i]);
+    replacements[operandPositions[i]] = getAffineSymbolExpr(i, op.getContext());
+  }
+
+  AffineMap map = op.getAffineMap();
+  ArrayRef<AffineExpr> repls{replacements};
+  map = map.replaceDimsAndSymbols(repls.take_front(map.getNumDims()),
+                                  repls.drop_front(map.getNumDims()),
+                                  /*numResultDims=*/0,
+                                  /*numResultSyms=*/numOperands);
+  map = AffineMap::get(0, numOperands,
+                       simplifyAffineExpr(map.getResult(0), 0, numOperands),
+                       op->getContext());
+  canonicalizeMapAndOperands(&map, &operands);
+
+  rewriter.startRootUpdate(op);
+  op.setMap(map);
+  op->setOperands(operands);
+  rewriter.finalizeRootUpdate(op);
+}
+
+/// Build an affine.apply that is a subexpression `expr` of `originalOp`s affine
+/// map and with the same operands.
+/// Canonicalize the map and operands to deduplicate and drop dead operands
+/// before returning but do not perform maximal composition of AffineApplyOp
+/// which would defeat the purpose.
+static AffineApplyOp createSubApply(RewriterBase &rewriter,
+                                    AffineApplyOp originalOp, AffineExpr expr) {
+  MLIRContext *ctx = originalOp->getContext();
+  AffineMap m = originalOp.getAffineMap();
+  auto rhsMap = AffineMap::get(m.getNumDims(), m.getNumSymbols(), expr, ctx);
+  SmallVector<Value> rhsOperands = originalOp->getOperands();
+  canonicalizeMapAndOperands(&rhsMap, &rhsOperands);
+  return rewriter.create<AffineApplyOp>(originalOp.getLoc(), rhsMap,
+                                        rhsOperands);
+}
+
+FailureOr<AffineApplyOp> mlir::decompose(RewriterBase &rewriter,
+                                         AffineApplyOp op) {
+  // 1. Preconditions: only handle dimensionless AffineApplyOp maps with a
+  // top-level binary expression that we can reassociate (i.e. add or mul).
+  AffineMap m = op.getAffineMap();
+  if (m.getNumDims() > 0)
+    return rewriter.notifyMatchFailure(op, "expected no dims");
+
+  AffineExpr remainingExp = m.getResult(0);
+  auto binExpr = remainingExp.dyn_cast<AffineBinaryOpExpr>();
+  if (!binExpr)
+    return rewriter.notifyMatchFailure(op, "terminal affine.apply");
+
+  if (!binExpr.getLHS().isa<AffineBinaryOpExpr>() &&
+      !binExpr.getRHS().isa<AffineBinaryOpExpr>())
+    return rewriter.notifyMatchFailure(op, "terminal affine.apply");
+
+  bool supportedKind = ((binExpr.getKind() == AffineExprKind::Add) ||
+                        (binExpr.getKind() == AffineExprKind::Mul));
+  if (!supportedKind)
+    return rewriter.notifyMatchFailure(
+        op, "only add or mul binary expr can be reassociated");
+
+  LLVM_DEBUG(DBGS() << "Start decomposeIntoFinerGrainedOps: " << op << "\n");
+
+  // 2. Iteratively extract the RHS subexpressions while the top-level binary
+  // expr kind remains the same.
+  MLIRContext *ctx = op->getContext();
+  SmallVector<AffineExpr> subExpressions;
+  while (true) {
+    auto currentBinExpr = remainingExp.dyn_cast<AffineBinaryOpExpr>();
+    if (!currentBinExpr || currentBinExpr.getKind() != binExpr.getKind()) {
+      subExpressions.push_back(remainingExp);
+      LLVM_DEBUG(DBGS() << "--terminal: " << subExpressions.back() << "\n");
+      break;
+    }
+    subExpressions.push_back(currentBinExpr.getRHS());
+    LLVM_DEBUG(DBGS() << "--subExpr: " << subExpressions.back() << "\n");
+    remainingExp = currentBinExpr.getLHS();
+  }
+
+  // 3. Reorder subExpressions by the min symbol they are a function of.
+  // This also takes care of properly reordering local variables.
+  // This however won't be able to split expression that cannot be reassociated
+  // such as ones that involve divs and multiple symbols.
+  auto getMaxSymbol = [&](AffineExpr e) -> int64_t {
+    for (int64_t i = m.getNumSymbols(); i >= 0; --i)
+      if (e.isFunctionOfSymbol(i))
+        return i;
+    return -1;
+  };
+  llvm::stable_sort(subExpressions, [&](AffineExpr e1, AffineExpr e2) {
+    return getMaxSymbol(e1) < getMaxSymbol(e2);
+  });
+  LLVM_DEBUG(
+      llvm::interleaveComma(subExpressions, DBGS() << "--sorted subexprs: ");
+      llvm::dbgs() << "\n");
+
+  // 4. Merge sorted subExpressions iteratively, thus achieving reassociation.
+  auto s0 = getAffineSymbolExpr(0, ctx);
+  auto s1 = getAffineSymbolExpr(1, ctx);
+  AffineMap binMap = AffineMap::get(
+      /*dimCount=*/0, /*symbolCount=*/2,
+      getAffineBinaryOpExpr(binExpr.getKind(), s0, s1), ctx);
+
+  auto current = createSubApply(rewriter, op, subExpressions[0]);
+  for (int64_t i = 1, e = subExpressions.size(); i < e; ++i) {
+    Value tmp = createSubApply(rewriter, op, subExpressions[i]);
+    current = rewriter.create<AffineApplyOp>(op.getLoc(), binMap,
+                                             ValueRange{current, tmp});
+    LLVM_DEBUG(DBGS() << "--reassociate into: " << current << "\n");
+  }
+
+  // 5. Replace original op.
+  rewriter.replaceOp(op, current.getResult());
+  return current;
+}
diff --git a/mlir/lib/Dialect/Affine/Transforms/PipelineDataTransfer.cpp b/mlir/lib/Dialect/Affine/Transforms/PipelineDataTransfer.cpp
index cd07b27e236d7..9b2bd6b7c2a1e 100644
--- a/mlir/lib/Dialect/Affine/Transforms/PipelineDataTransfer.cpp
+++ b/mlir/lib/Dialect/Affine/Transforms/PipelineDataTransfer.cpp
@@ -346,14 +346,14 @@ void PipelineDataTransfer::runOnAffineForOp(AffineForOp forOp) {
   }
   // Everything else (including compute ops and dma finish) are shifted by one.
   for (auto &op : forOp.getBody()->without_terminator())
-    if (instShiftMap.find(&op) == instShiftMap.end())
+    if (!instShiftMap.contains(&op))
       instShiftMap[&op] = 1;
 
   // Get shifts stored in map.
   SmallVector<uint64_t, 8> shifts(forOp.getBody()->getOperations().size());
   unsigned s = 0;
   for (auto &op : forOp.getBody()->without_terminator()) {
-    assert(instShiftMap.find(&op) != instShiftMap.end());
+    assert(instShiftMap.contains(&op));
     shifts[s++] = instShiftMap[&op];
 
     // Tagging operations with shifts for debugging purposes.
diff --git a/mlir/lib/Dialect/Affine/Utils/LoopFusionUtils.cpp b/mlir/lib/Dialect/Affine/Utils/LoopFusionUtils.cpp
index 366c090d0dcf4..624b79f65d429 100644
--- a/mlir/lib/Dialect/Affine/Utils/LoopFusionUtils.cpp
+++ b/mlir/lib/Dialect/Affine/Utils/LoopFusionUtils.cpp
@@ -225,11 +225,9 @@ static unsigned getMaxLoopDepth(ArrayRef<Operation *> srcOps,
       unsigned numCommonLoops =
           getNumCommonSurroundingLoops(*srcOpInst, *dstOpInst);
       for (unsigned d = 1; d <= numCommonLoops + 1; ++d) {
-        FlatAffineValueConstraints dependenceConstraints;
         // TODO: Cache dependence analysis results, check cache here.
-        DependenceResult result = checkMemrefAccessDependence(
-            srcAccess, dstAccess, d, &dependenceConstraints,
-            /*dependenceComponents=*/nullptr);
+        DependenceResult result =
+            checkMemrefAccessDependence(srcAccess, dstAccess, d);
         if (hasDependence(result)) {
           // Store minimum loop depth and break because we want the min 'd' at
           // which there is a dependence.
diff --git a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
index e7550d246e392..bcd87fcc570a3 100644
--- a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
+++ b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
@@ -378,7 +378,6 @@ checkTilingLegalityImpl(MutableArrayRef<mlir::AffineForOp> origLoops) {
 
   unsigned numOps = loadAndStoreOps.size();
   unsigned numLoops = origLoops.size();
-  FlatAffineValueConstraints dependenceConstraints;
   for (unsigned d = 1; d <= numLoops + 1; ++d) {
     for (unsigned i = 0; i < numOps; ++i) {
       Operation *srcOp = loadAndStoreOps[i];
@@ -388,9 +387,9 @@ checkTilingLegalityImpl(MutableArrayRef<mlir::AffineForOp> origLoops) {
         MemRefAccess dstAccess(dstOp);
 
         SmallVector<DependenceComponent, 2> depComps;
-        dependenceConstraints.reset();
         DependenceResult result = checkMemrefAccessDependence(
-            srcAccess, dstAccess, d, &dependenceConstraints, &depComps);
+            srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
+            &depComps);
 
         // Skip if there is no dependence in this case.
         if (!hasDependence(result))
@@ -2362,7 +2361,7 @@ static bool getFullMemRefAsRegion(Operation *op, unsigned numParamLoopIVs,
   ivs.resize(numParamLoopIVs);
   SmallVector<Value, 4> symbols;
   extractForInductionVars(ivs, &symbols);
-  regionCst->reset(rank, numParamLoopIVs, 0);
+  *regionCst = FlatAffineValueConstraints(rank, numParamLoopIVs, 0);
   regionCst->setValues(rank, rank + numParamLoopIVs, symbols);
 
   // Memref dim sizes provide the bounds.
diff --git a/mlir/lib/Dialect/Affine/Utils/Utils.cpp b/mlir/lib/Dialect/Affine/Utils/Utils.cpp
index 5830e8cef99a0..50405953e05bd 100644
--- a/mlir/lib/Dialect/Affine/Utils/Utils.cpp
+++ b/mlir/lib/Dialect/Affine/Utils/Utils.cpp
@@ -660,10 +660,8 @@ static bool mustReachAtInnermost(const MemRefAccess &srcAccess,
 
   unsigned nsLoops =
       getNumCommonSurroundingLoops(*srcAccess.opInst, *destAccess.opInst);
-  FlatAffineValueConstraints dependenceConstraints;
-  DependenceResult result = checkMemrefAccessDependence(
-      srcAccess, destAccess, nsLoops + 1, &dependenceConstraints,
-      /*dependenceComponents=*/nullptr);
+  DependenceResult result =
+      checkMemrefAccessDependence(srcAccess, destAccess, nsLoops + 1);
   return hasDependence(result);
 }
 
diff --git a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
index abf3db1728dcf..7c687142247a6 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
+++ b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
@@ -49,6 +49,27 @@ def AddISubConstantLHS :
           (ConstantLikeMatcher APIntAttr:$c1)),
         (Arith_SubIOp (Arith_ConstantOp (AddIntAttrs $res, $c0, $c1)), $x)>;
 
+def IsScalarOrSplatNegativeOne :
+    Constraint<And<[
+      CPred<"succeeded(getIntOrSplatIntValue($0))">,
+      CPred<"getIntOrSplatIntValue($0)->isAllOnes()">]>>;
+
+// addi(x, muli(y, -1)) -> subi(x, y)
+def AddIMulNegativeOneRhs :
+    Pat<(Arith_AddIOp
+           $x,
+           (Arith_MulIOp $y, (ConstantLikeMatcher AnyAttr:$c0))),
+        (Arith_SubIOp $x, $y),
+        [(IsScalarOrSplatNegativeOne $c0)]>;
+
+// addi(muli(x, -1), y) -> subi(y, x)
+def AddIMulNegativeOneLhs :
+    Pat<(Arith_AddIOp
+           (Arith_MulIOp $x, (ConstantLikeMatcher AnyAttr:$c0)),
+           $y),
+        (Arith_SubIOp $y, $x),
+        [(IsScalarOrSplatNegativeOne $c0)]>;
+
 //===----------------------------------------------------------------------===//
 // AddUIExtendedOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
index f6308a6b000b0..e56f4526291aa 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
@@ -258,8 +258,8 @@ OpFoldResult arith::AddIOp::fold(FoldAdaptor adaptor) {
 
 void arith::AddIOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
                                                 MLIRContext *context) {
-  patterns.add<AddIAddConstant, AddISubConstantRHS, AddISubConstantLHS>(
-      context);
+  patterns.add<AddIAddConstant, AddISubConstantRHS, AddISubConstantLHS,
+               AddIMulNegativeOneRhs, AddIMulNegativeOneLhs>(context);
 }
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
index 0f119d501c241..3b965cf732086 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
@@ -138,17 +138,15 @@ FailureOr<Value> bufferization::allocateTensorForShapedValue(
     bool reifiedShapes = false;
     if (shapedValue.getType().isa<RankedTensorType>() &&
         shapedValue.isa<OpResult>()) {
-      if (auto rankedOp = dyn_cast_or_null<ReifyRankedShapedTypeOpInterface>(
-              shapedValue.getDefiningOp())) {
-        ReifiedRankedShapedTypeDims resultDims;
-        if (succeeded(rankedOp.reifyResultShapes(b, resultDims))) {
-          reifiedShapes = true;
-          auto &shape =
-              resultDims[shapedValue.cast<OpResult>().getResultNumber()];
-          for (const auto &dim : enumerate(tensorType.getShape()))
-            if (ShapedType::isDynamic(dim.value()))
-              dynamicSizes.push_back(shape[dim.index()].get<Value>());
-        }
+      ReifiedRankedShapedTypeDims resultDims;
+      if (succeeded(
+              reifyResultShapes(b, shapedValue.getDefiningOp(), resultDims))) {
+        reifiedShapes = true;
+        auto &shape =
+            resultDims[shapedValue.cast<OpResult>().getResultNumber()];
+        for (const auto &dim : enumerate(tensorType.getShape()))
+          if (ShapedType::isDynamic(dim.value()))
+            dynamicSizes.push_back(shape[dim.index()].get<Value>());
       }
     }
 
diff --git a/mlir/lib/Dialect/ControlFlow/IR/ControlFlowOps.cpp b/mlir/lib/Dialect/ControlFlow/IR/ControlFlowOps.cpp
index eaf69f734bae3..6bdf42e527887 100644
--- a/mlir/lib/Dialect/ControlFlow/IR/ControlFlowOps.cpp
+++ b/mlir/lib/Dialect/ControlFlow/IR/ControlFlowOps.cpp
@@ -147,8 +147,9 @@ simplifyBrToBlockWithSinglePred(BranchOp op, PatternRewriter &rewriter) {
     return failure();
 
   // Merge the successor into the current block and erase the branch.
-  rewriter.mergeBlocks(succ, opParent, op.getOperands());
+  SmallVector<Value> brOperands(op.getOperands());
   rewriter.eraseOp(op);
+  rewriter.mergeBlocks(succ, opParent, brOperands);
   return success();
 }
 
diff --git a/mlir/lib/Dialect/GPU/IR/GPUDialect.cpp b/mlir/lib/Dialect/GPU/IR/GPUDialect.cpp
index 0ec5877f80361..f9d929d163445 100644
--- a/mlir/lib/Dialect/GPU/IR/GPUDialect.cpp
+++ b/mlir/lib/Dialect/GPU/IR/GPUDialect.cpp
@@ -46,6 +46,10 @@ int64_t GPUWarpMappingAttr::getMappingId() const {
   return static_cast<int64_t>(getWarp());
 }
 
+int64_t GPULinearIdMappingAttr::getMappingId() const {
+  return static_cast<int64_t>(getLinearId());
+}
+
 int64_t GPUThreadMappingAttr::getMappingId() const {
   return static_cast<int64_t>(getThread());
 }
diff --git a/mlir/lib/Dialect/GPU/TransformOps/GPUTransformOps.cpp b/mlir/lib/Dialect/GPU/TransformOps/GPUTransformOps.cpp
index ffe5c2a67e05c..f1559970d36d9 100644
--- a/mlir/lib/Dialect/GPU/TransformOps/GPUTransformOps.cpp
+++ b/mlir/lib/Dialect/GPU/TransformOps/GPUTransformOps.cpp
@@ -8,7 +8,9 @@
 
 #include "mlir/Dialect/GPU/TransformOps/GPUTransformOps.h"
 
+#include "mlir/Dialect/Affine/IR/AffineOps.h"
 #include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/GPU/TransformOps/GPUTransformOps.h"
 #include "mlir/Dialect/PDL/IR/PDL.h"
@@ -16,32 +18,269 @@
 #include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/Dialect/Transform/IR/TransformDialect.h"
 #include "mlir/Dialect/Transform/IR/TransformInterfaces.h"
-#include "mlir/Dialect/Transform/IR/TransformUtils.h"
+#include "mlir/Dialect/Utils/IndexingUtils.h"
+#include "mlir/IR/AffineExpr.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/BuiltinAttributes.h"
 #include "mlir/IR/IRMapping.h"
+#include "mlir/IR/MLIRContext.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/Visitors.h"
+#include "mlir/Support/LLVM.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/Support/Debug.h"
 
 using namespace mlir;
 using namespace mlir::gpu;
 using namespace mlir::transform;
+using namespace mlir::transform::gpu;
+
+#define DEBUG_TYPE "gpu-transforms"
+
+#define DBGS() (llvm::dbgs() << '[' << DEBUG_TYPE << "] ")
+#define LDBG(X) LLVM_DEBUG(DBGS() << X << "\n")
+
+namespace {
+
+/// Return a flattened thread id for the workgroup with given sizes.
+static Value buildLinearThreadId(RewriterBase &rewriter, Location loc,
+                                 ArrayRef<OpFoldResult> blockDimsOfr) {
+  LLVM_DEBUG(llvm::interleaveComma(
+                 blockDimsOfr,
+                 DBGS() << "----buildLinearThreadId with blockDimsOfr:  ");
+             llvm::dbgs() << "\n");
+  assert(blockDimsOfr.size() == 3 && "expected 3 workgroup sizes");
+  AffineExpr tx, ty, tz, BDX, BDY;
+  bindDims(rewriter.getContext(), tx, ty, tz);
+  bindSymbols(rewriter.getContext(), BDX, BDY);
+  IndexType indexType = rewriter.getIndexType();
+  SmallVector<OpFoldResult> threadsAndWorkGroups{
+      rewriter.create<ThreadIdOp>(loc, indexType, Dimension::x).getResult(),
+      rewriter.create<ThreadIdOp>(loc, indexType, Dimension::y).getResult(),
+      rewriter.create<ThreadIdOp>(loc, indexType, Dimension::z).getResult()};
+  threadsAndWorkGroups.push_back(blockDimsOfr[0]);
+  threadsAndWorkGroups.push_back(blockDimsOfr[1]);
+  OpFoldResult ofr = makeComposedFoldedAffineApply(
+      rewriter, loc, tx + ty * BDX + tz * BDX * BDY, threadsAndWorkGroups);
+  return getValueOrCreateConstantIndexOp(rewriter, loc, ofr);
+}
+
+/// Builder for gpu::BlockIdOps used in mapping scf.forall to blocks.
+/// The `idBuilder` method returns 3-D values used for indexing rewrites as well
+/// as 3-D sizes for predicate generation.
+struct GpuBlockIdBuilder : public GpuIdBuilder {
+
+  GpuBlockIdBuilder(MLIRContext *ctx, ArrayRef<OpFoldResult> blockDims,
+                    ArrayRef<int64_t> mappingSizes)
+      : GpuIdBuilder(blockDims, mappingSizes) {
+    mappingAttributes = {GPUBlockMappingAttr::get(ctx, Blocks::DimX),
+                         GPUBlockMappingAttr::get(ctx, Blocks::DimY),
+                         GPUBlockMappingAttr::get(ctx, Blocks::DimZ)},
+    idBuilder = [](RewriterBase &rewriter, Location loc,
+                   ArrayRef<int64_t> forallMappingSizes) {
+      IndexType indexType = rewriter.getIndexType();
+      SmallVector<Value> ids{
+          rewriter.create<BlockIdOp>(loc, indexType, Dimension::x),
+          rewriter.create<BlockIdOp>(loc, indexType, Dimension::y),
+          rewriter.create<BlockIdOp>(loc, indexType, Dimension::z)};
+      // Return 3-D ids for indexing rewrite and 3-D sizes and ids for
+      // predicate generation.
+      return IdBuilderResult{ids, SmallVector<int64_t>{forallMappingSizes},
+                             ids};
+    };
+  }
+};
+
+/// Builder for gpu::ThreadIdOp used in mapping scf.forall to thread ids without
+/// any reindexing.
+/// The `idBuilder` method returns 3-D values used for indexing rewrites as well
+/// as 3-D sizes for predicate generation.
+struct GpuThreadIdBuilder : public GpuIdBuilder {
+  GpuThreadIdBuilder(MLIRContext *ctx, ArrayRef<OpFoldResult> blockDims,
+                     ArrayRef<int64_t> mappingSizes)
+      : GpuIdBuilder(blockDims, mappingSizes) {
+    mappingAttributes = {GPUThreadMappingAttr::get(ctx, Threads::DimX),
+                         GPUThreadMappingAttr::get(ctx, Threads::DimY),
+                         GPUThreadMappingAttr::get(ctx, Threads::DimZ)};
+    idBuilder = [](RewriterBase &rewriter, Location loc,
+                   ArrayRef<int64_t> forallMappingSizes) {
+      IndexType indexType = rewriter.getIndexType();
+      SmallVector<Value> ids{
+          rewriter.create<ThreadIdOp>(loc, indexType, Dimension::x),
+          rewriter.create<ThreadIdOp>(loc, indexType, Dimension::y),
+          rewriter.create<ThreadIdOp>(loc, indexType, Dimension::z)};
+      // Return 3-D ids for indexing rewrite and 3-D sizes and ids for
+      // predicate generation.
+      return IdBuilderResult{ids, SmallVector<int64_t>{forallMappingSizes},
+                             ids};
+    };
+  }
+};
+
+/// Builder for warp ids used in mapping scf.forall to warps.
+/// This builder requires a specification of the number of warps along each
+/// dimension to more finely control mapping to warps as well a predication than
+/// by solely analyzing the IR.
+/// The `idBuilder` method returns 3-D values used for indexing rewrites as well
+/// as 3-D sizes for predicate generation.
+struct GpuWarpIdBuilder : public GpuIdBuilder {
+  GpuWarpIdBuilder(MLIRContext *ctx, ArrayRef<OpFoldResult> blockDims,
+                   ArrayRef<int64_t> mappingSizes)
+      : GpuIdBuilder(blockDims, mappingSizes) {
+    mappingAttributes = {GPUWarpMappingAttr::get(ctx, Warps::DimX),
+                         GPUWarpMappingAttr::get(ctx, Warps::DimY),
+                         GPUWarpMappingAttr::get(ctx, Warps::DimZ)};
+    idBuilder = [this](RewriterBase &rewriter, Location loc,
+                       ArrayRef<int64_t> forallMappingSizes) {
+      // Build the linear warp id and decompose it in the basis of
+      // `forallMappingSizes`.
+      Value linearId = buildLinearThreadId(rewriter, loc, this->blockDimsOfr);
+      AffineExpr d0 = getAffineDimExpr(0, rewriter.getContext());
+      OpFoldResult warpIdOfr = makeComposedFoldedAffineApply(
+          rewriter, loc, d0.floorDiv(kWarpSize), {linearId});
+      Value warpId = getValueOrCreateConstantIndexOp(rewriter, loc, warpIdOfr);
+      SmallVector<int64_t> reverseBasisSizes(
+          llvm::reverse(this->availableMappingSizes));
+      SmallVector<int64_t> strides = computeStrides(reverseBasisSizes);
+      SmallVector<AffineExpr> delinearizingExprs = delinearize(d0, strides);
+      SmallVector<Value> ids;
+      for (AffineExpr e : delinearizingExprs)
+        ids.push_back(makeComposedAffineApply(rewriter, loc, e, warpId));
+
+      // clang-format off
+      LDBG("----linearId: " << linearId);
+          LDBG("----warpId: " << warpId);
+      LLVM_DEBUG(llvm::interleaveComma(reverseBasisSizes,
+                                       DBGS() << "--delinearization basis: ");
+                 llvm::dbgs() << "\n";
+                 llvm::interleaveComma(strides,
+                                       DBGS() << "--delinearization strides: ");
+                 llvm::dbgs() << "\n";
+                 llvm::interleaveComma(delinearizingExprs,
+                                       DBGS() << "--delinearization exprs: ");
+                 llvm::dbgs() << "\n";
+                 llvm::interleaveComma(ids, DBGS() << "--ids: ");
+                 llvm::dbgs() << "\n";);
+      // clang-format on
+
+      // Return 3-D ids for indexing rewrite and 3-D sizes and ids for
+      // predicate generation.
+      return IdBuilderResult{ids, SmallVector<int64_t>{forallMappingSizes},
+                             ids};
+    };
+  }
+
+  /// Static specification of the warp size.
+  /// In the future this may be configured by the transformation.
+  static constexpr int64_t kWarpSize = 32;
+};
+
+/// Builder for linear ids used in mapping scf.forall to reindexed threads.
+/// The `idBuilder` method returns 3-D values used for indexing rewrites as well
+/// as 1-D sizes for predicate generation.
+struct GpuLinearIdBuilder : public GpuIdBuilder {
+  GpuLinearIdBuilder(MLIRContext *ctx, ArrayRef<OpFoldResult> blockDims,
+                     ArrayRef<int64_t> mappingSizes)
+      : GpuIdBuilder(blockDims, mappingSizes) {
+    mappingAttributes = {GPULinearIdMappingAttr::get(ctx, LinearId::DimX),
+                         GPULinearIdMappingAttr::get(ctx, LinearId::DimY),
+                         GPULinearIdMappingAttr::get(ctx, LinearId::DimZ)};
+    idBuilder = [this](RewriterBase &rewriter, Location loc,
+                       ArrayRef<int64_t> forallMappingSizes) {
+      // Build the linear thread id and decompose it in the basis of
+      // `forallMappingSizes`.
+      Value linearId = buildLinearThreadId(rewriter, loc, this->blockDimsOfr);
+      SmallVector<int64_t> reverseBasisSizes(llvm::reverse(forallMappingSizes));
+      SmallVector<int64_t> strides = computeStrides(reverseBasisSizes);
+      AffineExpr d0;
+      bindDims(rewriter.getContext(), d0);
+      SmallVector<AffineExpr> delinearizingExprs = delinearize(d0, strides);
+      SmallVector<Value> ids;
+      for (AffineExpr e : delinearizingExprs)
+        ids.push_back(makeComposedAffineApply(rewriter, loc, e, linearId));
+
+      // clang-format off
+      LLVM_DEBUG(llvm::interleaveComma(reverseBasisSizes,
+                                       DBGS() << "--delinearization basis: ");
+                 llvm::dbgs() << "\n";
+                 llvm::interleaveComma(strides,
+                                       DBGS() << "--delinearization strides: ");
+                 llvm::dbgs() << "\n";
+                 llvm::interleaveComma(delinearizingExprs,
+                                       DBGS() << "--delinearization exprs: ");
+                 llvm::dbgs() << "\n";
+                 llvm::interleaveComma(ids, DBGS() << "--ids: ");
+                 llvm::dbgs() << "\n";);
+      // clang-format on
+
+      // Compute and return the 1-D actual mapping size spanned by the linearId,
+      // it will be used to predicate against the linearized total number of
+      // threads.
+      int64_t actualMappingSize = 1;
+      for (int64_t s : forallMappingSizes)
+        actualMappingSize *= s;
+
+      // Return 3-D ids for indexing rewrite and 1-D size and id for
+      // predicate generation.
+      return IdBuilderResult{ids, SmallVector<int64_t>{actualMappingSize},
+                             SmallVector<Value>{linearId}};
+    };
+  }
+};
+
+} // namespace
+
+static DiagnosedSilenceableFailure
+definiteFailureHelper(std::optional<TransformOpInterface> transformOp,
+                      Operation *target, const Twine &message) {
+  if (transformOp.has_value())
+    return transformOp->emitDefiniteFailure() << message;
+  return emitDefiniteFailure(target, message);
+}
 
 /// Check if given mapping attributes are one of the desired attributes
 static DiagnosedSilenceableFailure
-checkAttributeType(ArrayRef<DeviceMappingAttrInterface> threadMappingAttributes,
-                   const std::optional<ArrayAttr> &foreachMapping,
-                   std::optional<TransformOpInterface> transformOp) {
-  if (!foreachMapping.has_value())
-    return transformOp->emitSilenceableError() << "mapping must be present";
+checkMappingAttributeTypes(std::optional<TransformOpInterface> transformOp,
+                           scf::ForallOp forallOp) {
+  if (!forallOp.getMapping().has_value())
+    return definiteFailureHelper(transformOp, forallOp,
+                                 "mapping must be present");
+
+  bool hasBlockMapping =
+      llvm::any_of(forallOp.getMapping().value(), [](Attribute attr) {
+        return attr.isa<GPUBlockMappingAttr>();
+      });
+  bool hasThreadMapping =
+      llvm::any_of(forallOp.getMapping().value(), [](Attribute attr) {
+        return attr.isa<GPUThreadMappingAttr>();
+      });
+  bool hasWarpMapping =
+      llvm::any_of(forallOp.getMapping().value(), [](Attribute attr) {
+        return attr.isa<GPUWarpMappingAttr>();
+      });
+  bool hasLinearMapping =
+      llvm::any_of(forallOp.getMapping().value(), [](Attribute attr) {
+        return attr.isa<GPULinearIdMappingAttr>();
+      });
+  int64_t countMappingTypes = 0;
+  countMappingTypes += hasBlockMapping ? 1 : 0;
+  countMappingTypes += hasThreadMapping ? 1 : 0;
+  countMappingTypes += hasWarpMapping ? 1 : 0;
+  countMappingTypes += hasLinearMapping ? 1 : 0;
+  if (countMappingTypes > 1) {
+    return definiteFailureHelper(
+        transformOp, forallOp,
+        "cannot mix different mapping types, use nesting");
+  }
 
   DenseSet<Attribute> seen;
-  for (Attribute map : foreachMapping->getValue()) {
-    if (!llvm::is_contained(threadMappingAttributes, map)) {
-      return transformOp->emitDefiniteFailure()
-             << "mapping must be one of " << threadMappingAttributes;
-    }
-    if (llvm::is_contained(seen, map)) {
-      return transformOp->emitDefiniteFailure()
-             << map
-             << " is duplicated, cannot map different "
-                "loops to the same processor";
+  for (Attribute map : forallOp.getMapping()->getValue()) {
+    if (seen.contains(map)) {
+      return definiteFailureHelper(
+          transformOp, forallOp,
+          "duplicated attribute, cannot map different loops "
+          "to the same processor");
     }
     seen.insert(map);
   }
@@ -49,9 +288,37 @@ checkAttributeType(ArrayRef<DeviceMappingAttrInterface> threadMappingAttributes,
   return DiagnosedSilenceableFailure::success();
 }
 
-/// Determines if the size of the kernel configuration is supported by the GPU
-/// architecture being used. It presently makes use of CUDA limitations, however
-/// that aspect may be enhanced for other GPUs.
+static DiagnosedSilenceableFailure
+verifyGpuMapping(std::optional<TransformOpInterface> transformOp,
+                 scf::ForallOp forallOp) {
+  // Check the types of the mapping attributes match.
+  DiagnosedSilenceableFailure typeRes =
+      checkMappingAttributeTypes(transformOp, forallOp);
+  if (!typeRes.succeeded())
+    return typeRes;
+
+  // Perform other non-types verifications.
+  if (!forallOp.isNormalized())
+    return definiteFailureHelper(transformOp, forallOp,
+                                 "unsupported non-normalized loops");
+  if (forallOp.getNumResults() > 0)
+    return definiteFailureHelper(transformOp, forallOp,
+                                 "only bufferized scf.forall can be mapped");
+  if (forallOp.getRank() > 3)
+    return definiteFailureHelper(transformOp, forallOp,
+                                 "scf.forall with rank > 3 does not lower");
+  if (llvm::any_of(forallOp.getMixedUpperBound(), [&](OpFoldResult ofr) {
+        return !getConstantIntValue(ofr).has_value();
+      })) {
+    return definiteFailureHelper(transformOp, forallOp,
+                                 "unsupported dynamic sizes in forall op");
+  }
+  return DiagnosedSilenceableFailure::success();
+}
+
+/// Determines if the size of the kernel configuration is supported by the
+/// GPU architecture being used. It presently makes use of CUDA limitations,
+/// however that aspect may be enhanced for other GPUs.
 static DiagnosedSilenceableFailure checkGpuLimits(
     TransformOpInterface transformOp, std::optional<int64_t> gridDimX,
     std::optional<int64_t> gridDimY, std::optional<int64_t> gridDimZ,
@@ -78,17 +345,17 @@ static DiagnosedSilenceableFailure checkGpuLimits(
       gridDimZ.value_or(1) > maxGriddimz ||
       gridDimX.value_or(1) > maxGriddimx) {
     return transformOp.emitSilenceableError()
-           << "Trying to launch a GPU kernel with gridDim = ("
+           << "Trying to launch a GPU kernel with grid_dims = ("
            << gridDimX.value_or(1) << ", " << gridDimY.value_or(1) << ", "
-           << gridDimZ.value_or(1) << ") blockDim = (" << blockDimX.value_or(1)
-           << ", " << blockDimY.value_or(1) << ", " << blockDimZ.value_or(1)
-           << "). It is larger than the limits.";
+           << gridDimZ.value_or(1) << ") block_dims = ("
+           << blockDimX.value_or(1) << ", " << blockDimY.value_or(1) << ", "
+           << blockDimZ.value_or(1) << "). It is larger than the limits.";
   }
   return DiagnosedSilenceableFailure::success();
 }
 
-/// Creates an empty-body gpu::LaunchOp using the provided kernel settings and
-/// put a terminator within.
+/// Creates an empty-body gpu::LaunchOp using the provided kernel settings
+/// and put a terminator within.
 static DiagnosedSilenceableFailure
 createGpuLaunch(RewriterBase &rewriter, Location loc,
                 TransformOpInterface transformOp, LaunchOp &launchOp,
@@ -124,7 +391,7 @@ createGpuLaunch(RewriterBase &rewriter, Location loc,
 
 /// Alter kernel configuration of the given kernel.
 static DiagnosedSilenceableFailure
-alterGpuLaunch(TrivialPatternRewriter &rewriter, LaunchOp gpuLaunch,
+alterGpuLaunch(IRRewriter &rewriter, LaunchOp gpuLaunch,
                TransformOpInterface transformOp,
                std::optional<int64_t> gridDimX = std::nullopt,
                std::optional<int64_t> gridDimY = std::nullopt,
@@ -164,94 +431,206 @@ alterGpuLaunch(TrivialPatternRewriter &rewriter, LaunchOp gpuLaunch,
   return DiagnosedSilenceableFailure::success();
 }
 
-//===----------------------------------------------------------------------===//
-// MapForeachToBlocks
-//===----------------------------------------------------------------------===//
+/// Struct to return the result of the rewrite of a forall operation.
+struct ForallRewriteResult {
+  SmallVector<int64_t> mappingSizes;
+  SmallVector<Value> mappingIds;
+};
 
-DiagnosedSilenceableFailure mlir::transform::gpu::mapForeachToBlocksImpl(
-    RewriterBase &rewriter, scf::ForallOp forallOp,
-    function_ref<void(RewriterBase &, scf::ForallOp, SmallVectorImpl<Value> &)>
-        blockIdGenerator,
-    SmallVectorImpl<int64_t> &gridDims, TransformOpInterface transformOp,
-    const ArrayRef<DeviceMappingAttrInterface> &mappingAttributes) {
-  // Step 0. Target-specific verifications. There is no good place to anchor
-  // those right now: the ForallOp is target-independent and the
-  // transform op does not apply to individual ForallOp.
-  Location loc = forallOp->getLoc();
+/// Helper to replace ids of dimensions known to be 1 by 0 to simplify the IR.
+template <typename OpTy, typename OperationOrBlock>
+static void
+replaceUnitMappingIdsHelper(RewriterBase &rewriter, Location loc,
+                            OperationOrBlock *parent, Value replacement,
+                            ArrayRef<int64_t> availableMappingSizes) {
+  parent->walk([&](OpTy idOp) {
+    if (availableMappingSizes[static_cast<int64_t>(idOp.getDimension())] == 1)
+      rewriter.replaceAllUsesWith(idOp.getResult(), replacement);
+  });
+}
 
-  if (!forallOp.isNormalized())
-    return transformOp.emitSilenceableError()
-           << "unsupported non-normalized loops";
-  if (forallOp.getNumResults() > 0)
-    return transformOp.emitSilenceableError()
-           << "only bufferized scf.forall lowers to "
-              "gpu.block_id";
-  if (forallOp.getRank() > 3)
-    return transformOp.emitSilenceableError()
-           << "scf.forall with rank > 3 does not lower to "
-              "gpu.block_id";
-  if (llvm::any_of(forallOp.getMixedUpperBound(), [](OpFoldResult ofr) {
-        return !getConstantIntValue(ofr).has_value();
-      })) {
-    return transformOp.emitSilenceableError()
-           << "unsupported dynamic griddim size";
-  }
-  SmallVector<Attribute> blockMapping =
-      llvm::to_vector(forallOp.getMapping()->getValue());
+static DiagnosedSilenceableFailure rewriteOneForallCommonImpl(
+    RewriterBase &rewriter, std::optional<TransformOpInterface> transformOp,
+    scf::ForallOp forallOp, ForallRewriteResult &result,
+    ArrayRef<int64_t> availableMappingSizes, const GpuIdBuilder &gpuIdBuilder) {
+  LDBG("--start rewriteOneForallCommonImpl");
 
-  // Step 1. Complete the blockMapping to a full mapping (with 1s) if necessary.
-  SmallVector<Value> numBlocks = forallOp.getUpperBound(rewriter);
-  // Ensure we have 3 block sizes, one for each id.
-  Value one;
-  for (auto attr : mappingAttributes) {
-    if (std::find(blockMapping.begin(), blockMapping.end(), attr) ==
-        blockMapping.end()) {
-      blockMapping.push_back(attr);
-      one = one ? one : rewriter.create<arith::ConstantIndexOp>(loc, 1);
-      numBlocks.push_back(one);
-    }
+  // Step 0. GPU-specific verifications. There is no better place to anchor
+  // those right now: the ForallOp is target-independent and the transform
+  // op does not apply to individual ForallOp.
+  DiagnosedSilenceableFailure diag = verifyGpuMapping(transformOp, forallOp);
+  if (!diag.succeeded())
+    return diag;
+
+  // Step 1. Complete the mapping to a full mapping (with 1s) if necessary.
+  SmallVector<int64_t> tmpMappingSizes = llvm::to_vector(
+      llvm::map_range(forallOp.getMixedUpperBound(), [](OpFoldResult ofr) {
+        auto maybeStaticValue = getConstantIntValue(ofr);
+        assert(maybeStaticValue && "expected static value");
+        return maybeStaticValue.value();
+      }));
+  SmallVector<Attribute> forallMappingAttrs =
+      llvm::to_vector(forallOp.getMapping()->getValue());
+  for (auto attr : gpuIdBuilder.mappingAttributes) {
+    if (llvm::is_contained(forallMappingAttrs, attr))
+      continue;
+    forallMappingAttrs.push_back(attr);
+    tmpMappingSizes.push_back(1);
   }
+  LLVM_DEBUG(
+      llvm::interleaveComma(
+          tmpMappingSizes,
+          DBGS() << "----tmpMappingSizes extracted from scf.forall op: ");
+      llvm::dbgs() << "\n");
 
   // Step 2. sort the values by the corresponding DeviceMappingAttrInterface.
   auto comparator = [&](DeviceMappingAttrInterface a,
                         DeviceMappingAttrInterface b) -> bool {
     return a.getMappingId() < b.getMappingId();
   };
-  SmallVector<Value> gridDimValues =
-      scf::ForallOp::getValuesSortedByKey(blockMapping, numBlocks, comparator);
-  for (Value v : gridDimValues)
-    gridDims.push_back(v.getDefiningOp<arith::ConstantIndexOp>().value());
-
-  // Step 3. Generate the blockIds using the provided generator and map the
-  // induction variables to the newly created ops.
-  SmallVector<Value> blockOps;
-  blockIdGenerator(rewriter, forallOp, blockOps);
+  SmallVector<int64_t> forallMappingSizes =
+      getValuesSortedByKey(forallMappingAttrs, tmpMappingSizes, comparator);
+  LLVM_DEBUG(llvm::interleaveComma(forallMappingSizes,
+                                   DBGS() << "----forallMappingSizes: ");
+             llvm::dbgs() << "\n"; llvm::interleaveComma(
+                 forallMappingAttrs, DBGS() << "----mappingAttrs: ");
+             llvm::dbgs() << "\n");
+
+  // Step 3. Generate the mappingIdOps using the provided generator.
+  Location loc = forallOp.getLoc();
+  OpBuilder::InsertionGuard guard(rewriter);
+  rewriter.setInsertionPoint(forallOp);
+  IdBuilderResult builderResult =
+      gpuIdBuilder.idBuilder(rewriter, loc, forallMappingSizes);
+
+  // Step 4. Map the induction variables to the mappingIdOps, this may involve a
+  // permutation.
+  SmallVector<Value> mappingIdOps = builderResult.mappingIdOps;
   IRMapping bvm;
-  for (auto [blockIdx, blockDim] :
-       llvm::zip(forallOp.getInductionVars(), blockMapping)) {
-    bvm.map(blockIdx,
-            blockOps[static_cast<int64_t>(
-                blockDim.cast<DeviceMappingAttrInterface>().getMappingId())]);
+  for (auto [iv, dim] :
+       llvm::zip_equal(forallOp.getInductionVars(),
+                       ArrayRef<Attribute>{forallMappingAttrs}.take_front(
+                           forallOp.getInductionVars().size()))) {
+    Value peIdOp = mappingIdOps[static_cast<int64_t>(
+        dim.cast<DeviceMappingAttrInterface>().getMappingId())];
+    bvm.map(iv, peIdOp);
   }
 
-  // Step 4. Move the body of forallOp.
-  // Erase the terminator first, it will not be used since we are on buffers.
+  // Step 5. If the availableMappingSizes are already known, create conditionals
+  // to predicate the region. Otherwise, the current forall determines the
+  // availableMappingSizes and no predication occurs.
+  Value predicate;
+  if (!availableMappingSizes.empty()) {
+    SmallVector<int64_t> predicateMappingSizes =
+        builderResult.predicateMappingSizes;
+    SmallVector<Value> predicateIdOps = builderResult.predicateIdOps;
+    // clang-format off
+    LLVM_DEBUG(
+        llvm::interleaveComma(
+          predicateMappingSizes, DBGS() << "----predicateMappingSizes: ");
+        llvm::dbgs() << "\n"; 
+        llvm::interleaveComma(
+          availableMappingSizes, DBGS() << "----availableMappingSizes: ");
+        llvm::dbgs() << "\n";
+        llvm::interleaveComma(predicateIdOps, DBGS() << "----predicateIdOps: ");
+        llvm::dbgs() << "\n");
+    // clang-format on
+    for (auto [id, mappingSize, availableMappingSize] : llvm::zip_equal(
+             predicateIdOps, predicateMappingSizes, availableMappingSizes)) {
+      if (mappingSize > availableMappingSize) {
+        return definiteFailureHelper(
+            transformOp, forallOp,
+            "Trying to map to fewer GPU threads than loop iterations but "
+            "overprovisioning is not yet supported. "
+            "Try additional tiling of the before mapping or map to more "
+            "threads.");
+      }
+      if (mappingSize == availableMappingSize)
+        continue;
+      Value idx = rewriter.create<arith::ConstantIndexOp>(loc, mappingSize);
+      Value tmpPredicate = rewriter.create<arith::CmpIOp>(
+          loc, arith::CmpIPredicate::ult, id, idx);
+      LDBG("----predicate: " << tmpPredicate);
+      predicate = predicate ? rewriter.create<arith::AndIOp>(loc, predicate,
+                                                             tmpPredicate)
+                            : tmpPredicate;
+    }
+  }
+
+  // Step 6. Move the body of forallOp.
+  // Erase the terminator first, it will not be used.
   rewriter.eraseOp(forallOp.getTerminator());
-  Block *targetBlock = forallOp->getBlock();
-  Block::iterator insertionPoint = Block::iterator(forallOp);
+  Block *targetBlock;
+  Block::iterator insertionPoint;
+  if (predicate) {
+    // Step 6.a. If predicated, move at the beginning.
+    auto ifOp = rewriter.create<scf::IfOp>(loc, predicate,
+                                           /*withElseRegion=*/false);
+    targetBlock = ifOp.thenBlock();
+    insertionPoint = ifOp.thenBlock()->begin();
+  } else {
+    // Step 6.b. Otherwise, move inline just at the rewriter insertion
+    // point.
+    targetBlock = forallOp->getBlock();
+    insertionPoint = rewriter.getInsertionPoint();
+  }
   Block &sourceBlock = forallOp.getRegion().front();
   targetBlock->getOperations().splice(insertionPoint,
                                       sourceBlock.getOperations());
 
-  // Step 5. RAUW thread indices to thread ops.
+  // Step 7. RAUW indices.
   for (Value loopIndex : forallOp.getInductionVars()) {
-    Value blockIdx = bvm.lookup(loopIndex);
-    rewriter.replaceAllUsesWith(loopIndex, blockIdx);
+    Value threadIdx = bvm.lookup(loopIndex);
+    rewriter.replaceAllUsesWith(loopIndex, threadIdx);
   }
 
-  // Step 6. Erase old op.
+  // Step 8. Erase old op.
   rewriter.eraseOp(forallOp);
 
+  result = ForallRewriteResult{forallMappingSizes, mappingIdOps};
+  return DiagnosedSilenceableFailure::success();
+}
+
+//===----------------------------------------------------------------------===//
+// MapForallToBlocks
+//===----------------------------------------------------------------------===//
+
+DiagnosedSilenceableFailure mlir::transform::gpu::mapForallToBlocksImpl(
+    RewriterBase &rewriter, TransformOpInterface transformOp,
+    scf::ForallOp forallOp, SmallVectorImpl<int64_t> &gridDims,
+    const GpuIdBuilder &gpuIdBuilder) {
+  LDBG("Start mapForallToBlocksImpl");
+
+  Location loc = forallOp.getLoc();
+  Block *parentBlock = forallOp->getBlock();
+  Value zero;
+  {
+    // Create an early zero index value for replacements and immediately reset
+    // the insertion point.
+    OpBuilder::InsertionGuard guard(rewriter);
+    rewriter.setInsertionPointToStart(parentBlock);
+    zero = rewriter.create<arith::ConstantIndexOp>(loc, 0);
+  }
+
+  SmallVector<int64_t> anyAvailableMappingSizes;
+  ForallRewriteResult rewriteResult;
+  // Pass an empty anyAvailableMappingSizes.
+  DiagnosedSilenceableFailure diag =
+      rewriteOneForallCommonImpl(rewriter, transformOp, forallOp, rewriteResult,
+                                 anyAvailableMappingSizes, gpuIdBuilder);
+
+  // Return if anything goes wrong, use silenceable failure as a match failure.
+  if (!diag.succeeded())
+    return diag;
+
+  // Set the gridDims that act as a return.
+  gridDims = rewriteResult.mappingSizes;
+
+  // Replace ids of dimensions known to be 1 by 0 to simplify the IR.
+  // Here, the result of mapping determines the available mapping sizes.
+  replaceUnitMappingIdsHelper<BlockDimOp>(rewriter, loc, parentBlock, zero,
+                                          gridDims);
+
   return DiagnosedSilenceableFailure::success();
 }
 
@@ -263,7 +642,7 @@ mlir::transform::gpu::findTopLevelForallOp(Operation *target,
     if (forallOp->getParentOfType<scf::ForallOp>())
       return WalkResult::advance();
     if (topLevelForallOp)
-      // TODO: Handle multiple foreach if there is no dependences between them
+      // TODO: Handle multiple forall if they are independent.
       return WalkResult::interrupt();
     topLevelForallOp = forallOp;
     return WalkResult::advance();
@@ -275,27 +654,12 @@ mlir::transform::gpu::findTopLevelForallOp(Operation *target,
   return DiagnosedSilenceableFailure::success();
 }
 
-/// This is a helper that is only used in
-/// rewriteTopLevelForallToGpuBlocks. It generates GPU dialects
-/// block_id.
-static void generateGpuBlockIds(RewriterBase &rewriter, scf::ForallOp foreachOp,
-                                SmallVectorImpl<Value> &blockOps) {
-  Location loc = foreachOp->getLoc();
-  OpBuilder::InsertionGuard guard(rewriter);
-  rewriter.setInsertionPoint(foreachOp);
-  IndexType indexType = rewriter.getIndexType();
-  blockOps = SmallVector<Value>{
-      rewriter.create<BlockIdOp>(loc, indexType, Dimension::x),
-      rewriter.create<BlockIdOp>(loc, indexType, Dimension::y),
-      rewriter.create<BlockIdOp>(loc, indexType, Dimension::z)};
-}
-
 DiagnosedSilenceableFailure
-transform::MapForeachToBlocks::applyToOne(Operation *target,
-                                          ApplyToEachResultList &results,
-                                          transform::TransformState &state) {
+transform::MapForallToBlocks::applyToOne(Operation *target,
+                                         ApplyToEachResultList &results,
+                                         transform::TransformState &state) {
   LaunchOp gpuLaunch = dyn_cast<LaunchOp>(target);
-  TrivialPatternRewriter rewriter(getContext());
+  IRRewriter rewriter(getContext());
   auto transformOp = cast<TransformOpInterface>(getOperation());
 
   if (!getGenerateGpuLaunch() && !gpuLaunch) {
@@ -315,6 +679,10 @@ transform::MapForeachToBlocks::applyToOne(Operation *target,
     return diag;
   }
 
+  SmallVector<int64_t> gridDims{getGridDims()};
+  if (!getGenerateGpuLaunch() && gridDims.size() != 3)
+    return transformOp.emitDefiniteFailure("transform require size-3 mapping");
+
   OpBuilder::InsertionGuard guard(rewriter);
   rewriter.setInsertionPoint(topLevelForallOp);
 
@@ -331,251 +699,186 @@ transform::MapForeachToBlocks::applyToOne(Operation *target,
     topLevelForallOp = cast<scf::ForallOp>(newForallOp);
   }
 
-  SmallVector<int64_t> gridDim = extractFromI64ArrayAttr(getGridDim());
-  SmallVector<DeviceMappingAttrInterface> blockMappingAttributes = {
-      GPUBlockMappingAttr::get(getContext(), Blocks::DimX),
-      GPUBlockMappingAttr::get(getContext(), Blocks::DimY),
-      GPUBlockMappingAttr::get(getContext(), Blocks::DimZ)};
-
-  diag = checkAttributeType(blockMappingAttributes,
-                            topLevelForallOp.getMapping(), transformOp);
-  if (diag.succeeded())
-    diag = mlir::transform::gpu::mapForeachToBlocksImpl(
-        rewriter, topLevelForallOp, generateGpuBlockIds, gridDim, transformOp,
-        blockMappingAttributes);
-  if (diag.succeeded()) {
-    diag = alterGpuLaunch(rewriter, gpuLaunch,
-                          cast<TransformOpInterface>(getOperation()),
-                          gridDim[0], gridDim[1], gridDim[2]);
-  }
+  GpuBlockIdBuilder gpuBlockIdBuilder(getContext(), {}, {});
+  diag = mlir::transform::gpu::mapForallToBlocksImpl(
+      rewriter, transformOp, topLevelForallOp, gridDims, gpuBlockIdBuilder);
+  if (!diag.succeeded())
+    return diag;
+
+  // Set the GPU launch configuration for the grid dims late, this is subject to
+  // IR inspection.
+  diag = alterGpuLaunch(rewriter, gpuLaunch,
+                        cast<TransformOpInterface>(getOperation()), gridDims[0],
+                        gridDims[1], gridDims[2]);
 
   results.push_back(gpuLaunch);
   return diag;
 }
 
 //===----------------------------------------------------------------------===//
-// MapNestedForeachToThreads
+// MapNestedForallToThreads
 //===----------------------------------------------------------------------===//
 
-/// Searches `scf.forall` ops nested under `target` and maps each such
-/// op to GPU threads. Mapping is one-to-one and the induction variables of
-/// `scf.forall` are rewritten to gpu.thread_id according to the
-/// thread_dim_mapping attribute. Sibling `scf.forall` are supported in
-/// which case, the union of the number of threads is computed and may result
-/// in predication. Dynamic, `scf.forall` trip counts are currently
-/// not supported. Dynamic block dim sizes are currently not supported.
-static DiagnosedSilenceableFailure rewriteOneForallToGpuThreads(
-    RewriterBase &rewriter, scf::ForallOp forallOp,
-    const SmallVectorImpl<int64_t> &globalBlockDims,
-    const SmallVectorImpl<Value> &threadOps, bool syncAfterDistribute,
-    std::optional<TransformOpInterface> transformOp,
-    const ArrayRef<DeviceMappingAttrInterface> &threadMappingAttributes) {
-  // Step 0. Target-specific verifications. There is no good place to anchor
-  // those right now: the ForallOp is target-independent and the
-  // transform op does not apply to individual ForallOp.
-  auto failureHelper =
-      [&](const Twine &message) -> DiagnosedSilenceableFailure {
-    if (transformOp.has_value()) {
-      return transformOp->emitSilenceableError() << message;
-    }
-    return emitDefiniteFailure(forallOp, message);
-  };
-  Location loc = forallOp->getLoc();
-  if (!forallOp.isNormalized())
-    return failureHelper("unsupported non-normalized loops");
-  if (forallOp.getNumResults() > 0)
-    return failureHelper("only bufferized scf.forall lowers to gpu.thread_id");
-  if (forallOp.getRank() > 3)
-    return failureHelper(
-        "scf.forall with rank > 3 does not lower to gpu.thread_id");
-  if (llvm::any_of(forallOp.getMixedUpperBound(), [](OpFoldResult ofr) {
-        return !getConstantIntValue(ofr).has_value();
-      })) {
-    return failureHelper("unsupported dynamic blockdim size");
-  }
-  if (!forallOp.getMapping().has_value())
-    return failureHelper("mapping must be present");
-  SmallVector<Attribute> threadMapping =
-      llvm::to_vector(forallOp.getMapping()->getValue());
-
-  // Step 1. Complete the threadMapping to a full mapping (with 1s) if
-  // necessary.
-  SmallVector<Value> numThreads = forallOp.getUpperBound(rewriter);
-  // Ensure we have 3 block sizes, one for each id.
-  Value one;
-  for (auto attr : threadMappingAttributes) {
-    if (std::find(threadMapping.begin(), threadMapping.end(), attr) ==
-        threadMapping.end()) {
-      threadMapping.push_back(attr);
-      one = one ? one : rewriter.create<arith::ConstantIndexOp>(loc, 1);
-      numThreads.push_back(one);
+DiagnosedSilenceableFailure mlir::transform::gpu::mapOneForallToThreadsImpl(
+    RewriterBase &rewriter, std::optional<TransformOpInterface> transformOp,
+    scf::ForallOp forallOp, ArrayRef<int64_t> availableMappingSizes,
+    bool syncAfterDistribute, const GpuIdBuilder &gpuIdBuilder) {
+  // Ignore cases with different attributes than this builder supports.
+  for (Attribute map : forallOp.getMapping()->getValue()) {
+    if (!llvm::is_contained(gpuIdBuilder.mappingAttributes, map)) {
+      LDBG("--skip " << map);
+      LLVM_DEBUG(llvm::interleaveComma(gpuIdBuilder.mappingAttributes,
+                                       DBGS() << "----not in: ");
+                 llvm::dbgs() << "\n";);
+      return emitSilenceableFailure(forallOp);
     }
   }
 
-  // Step 2. sort the values by the corresponding DeviceMappingAttrInterface.
-  auto comparator = [&](DeviceMappingAttrInterface a,
-                        DeviceMappingAttrInterface b) -> bool {
-    return a.getMappingId() < b.getMappingId();
-  };
-  SmallVector<Value> blockDimValues = scf::ForallOp::getValuesSortedByKey(
-      threadMapping, numThreads, comparator);
-  SmallVector<int64_t> blockDims =
-      llvm::to_vector(llvm::map_range(blockDimValues, [](Value v) {
-        return v.getDefiningOp<arith::ConstantIndexOp>().value();
-      }));
-
-  // Step 3. Create the gpu.thread ops and map the induction variables to the
-  // newly created ops.
-  // Replace ids of dimension size 1 by zero to simplify the IR.
-  SmallVector<Value> threadOpsUpdated(threadOps.begin(), threadOps.end());
-  assert(threadOps.size() == globalBlockDims.size());
-  Value zero = rewriter.create<arith::ConstantIndexOp>(loc, 0);
-  for (size_t i : llvm::seq(size_t(0), globalBlockDims.size())) {
-    if (globalBlockDims[i] == 1)
-      threadOpsUpdated[i] = zero;
-  }
-  IRMapping bvm;
-  for (auto [blockIdx, blockDim] :
-       llvm::zip(forallOp.getInductionVars(), threadMapping)) {
-    bvm.map(blockIdx,
-            threadOpsUpdated[blockDim.cast<DeviceMappingAttrInterface>()
-                                 .getMappingId()]);
-  }
-
-  // Step 4. Maybe create conditionals to predicate the region.
-  Value predicate;
-  for (auto [threadId, blockDim, globalBlockDim] :
-       llvm::zip(threadOpsUpdated, blockDims, globalBlockDims)) {
-    if (blockDim > globalBlockDim) {
-      return failureHelper(
-          "The requested GPU threads are fewer than the number of loop trip "
-          "counts. Try to tile scf.forall before mapping or set "
-          "small blockDim.");
-    }
-    if (blockDim == globalBlockDim)
-      continue;
-    Value blockIdx = rewriter.create<arith::ConstantIndexOp>(loc, blockDim);
-    Value tmpPredicate = rewriter.create<arith::CmpIOp>(
-        loc, arith::CmpIPredicate::ult, threadId, blockIdx);
-    predicate =
-        predicate ? rewriter.create<arith::AndIOp>(loc, predicate, tmpPredicate)
-                  : tmpPredicate;
-  }
-
-  // Step 5. Move the body of forallOp.
-  // Erase the terminator first, it will not be used.
-  rewriter.eraseOp(forallOp.getTerminator());
-  Block *targetBlock;
-  Block::iterator insertionPoint;
-  if (predicate) {
-    // Step 5.a. If predicated, move at the beginning.
-    auto ifOp =
-        rewriter.create<scf::IfOp>(loc, predicate, /*withElseRegion=*/false);
-    targetBlock = ifOp.thenBlock();
-    insertionPoint = ifOp.thenBlock()->begin();
-  } else {
-    // Step 5.b. Otherwise, move inline just before forallOp.
-    targetBlock = forallOp->getBlock();
-    insertionPoint = Block::iterator(forallOp);
-  }
-  Block &sourceBlock = forallOp.getRegion().front();
-  targetBlock->getOperations().splice(insertionPoint,
-                                      sourceBlock.getOperations());
+  Location loc = forallOp.getLoc();
+  OpBuilder::InsertionGuard g(rewriter);
+  // Insert after to allow for syncthreads after `forall` is erased.
+  rewriter.setInsertionPointAfter(forallOp);
+  ForallRewriteResult rewriteResult;
+  DiagnosedSilenceableFailure diag =
+      rewriteOneForallCommonImpl(rewriter, transformOp, forallOp, rewriteResult,
+                                 availableMappingSizes, gpuIdBuilder);
 
-  // Step 6. RAUW thread indices to thread ops.
-  for (Value loopIndex : forallOp.getInductionVars()) {
-    Value threadIdx = bvm.lookup(loopIndex);
-    rewriter.replaceAllUsesWith(loopIndex, threadIdx);
-  }
+  // Return if anything goes wrong, use silenceable failure as a match failure.
+  if (!diag.succeeded())
+    return diag;
 
-  // Step 7. syncthreads.
-  // TODO: Need warpsync
+  // Add a syncthreads if needed. TODO: warpsync
   if (syncAfterDistribute)
     rewriter.create<BarrierOp>(loc);
 
-  // Step 8. Erase old op.
-  rewriter.eraseOp(forallOp);
-
   return DiagnosedSilenceableFailure::success();
 }
 
-DiagnosedSilenceableFailure mlir::transform::gpu::mapNestedForeachToThreadsImpl(
-    RewriterBase &rewriter, Operation *target,
-    const SmallVectorImpl<int64_t> &blockDim,
-    function_ref<void(RewriterBase &, scf::ForallOp, SmallVectorImpl<Value> &)>
-        threadIdGenerator,
-    bool syncAfterDistribute, std::optional<TransformOpInterface> transformOp,
-    const ArrayRef<DeviceMappingAttrInterface> &threadMappingAttributes) {
+DiagnosedSilenceableFailure mlir::transform::gpu::mapNestedForallToThreadsImpl(
+    RewriterBase &rewriter, std::optional<TransformOpInterface> transformOp,
+    Operation *target, ArrayRef<int64_t> blockDims, ArrayRef<int64_t> warpDims,
+    bool syncAfterDistribute) {
+  LDBG("Start mapNestedForallToThreadsImpl");
+  MLIRContext *ctx = rewriter.getContext();
+  SmallVector<OpFoldResult> blockDimsOfr =
+      getAsIndexOpFoldResult(ctx, blockDims);
+
+  if (blockDims.size() != 3)
+    return definiteFailureHelper(transformOp, target,
+                                 "requires size-3 thread mapping");
+  if (!warpDims.empty()) {
+    if (warpDims.size() != 3)
+      return definiteFailureHelper(transformOp, target,
+                                   "requires empty or size-3 warp mapping");
+  }
+
+  // Create an early zero index value for replacements.
+  Location loc = target->getLoc();
+  Value zero = rewriter.create<arith::ConstantIndexOp>(loc, 0);
   DiagnosedSilenceableFailure diag = DiagnosedSilenceableFailure::success();
-  target->walk([&](scf::ForallOp forallOp) {
-    // Ignore cases with different attributes.
-    for (Attribute map : forallOp.getMapping()->getValue()) {
-      if (!llvm::is_contained(threadMappingAttributes, map)) {
+  WalkResult walkResult = target->walk([&](scf::ForallOp forallOp) {
+    //===--------------------------------------------------------------------===//
+    // Mapping to warp ids.
+    //===--------------------------------------------------------------------===//
+    if (!warpDims.empty()) {
+      LLVM_DEBUG(
+          llvm::interleaveComma(
+              warpDims, DBGS() << "+mapNestedForallToThreadsImpl warpDims: ");
+          llvm::dbgs() << "\n");
+      LLVM_DEBUG(llvm::interleaveComma(
+                     blockDimsOfr, DBGS() << "--warpDims with blockDimsOfr:  ");
+                 llvm::dbgs() << "\n");
+      GpuWarpIdBuilder gpuWarpIdBuilder(ctx, blockDimsOfr, warpDims);
+      diag = mlir::transform::gpu::mapOneForallToThreadsImpl(
+          rewriter, transformOp, forallOp, warpDims, syncAfterDistribute,
+          gpuWarpIdBuilder);
+      // Use silenceable failure to encode "failure to match" and pass
+      // through.
+      if (diag.isDefiniteFailure())
+        return WalkResult::interrupt();
+      if (diag.succeeded())
         return WalkResult::skip();
-      }
     }
-    diag = checkAttributeType(threadMappingAttributes, forallOp.getMapping(),
-                              transformOp);
-    if (diag.succeeded()) {
-      rewriter.setInsertionPoint(forallOp);
-      SmallVector<Value> threadOps;
-      threadIdGenerator(rewriter, forallOp, threadOps);
-      diag = rewriteOneForallToGpuThreads(rewriter, forallOp, blockDim,
-                                          threadOps, syncAfterDistribute,
-                                          transformOp, threadMappingAttributes);
-    }
-    return diag.succeeded() ? WalkResult::advance() : WalkResult::interrupt();
+
+    //===--------------------------------------------------------------------===//
+    // Mapping to linear ids.
+    //===--------------------------------------------------------------------===//
+    LDBG("+mapNestedForallToThreadsImpl linearDims");
+    LLVM_DEBUG(llvm::interleaveComma(
+                   blockDimsOfr, DBGS() << "--linearDims with blockDimsOfr:  ");
+               llvm::dbgs() << "\n");
+    int64_t numThreads = 1;
+    for (int64_t b : blockDims)
+      numThreads *= b;
+    GpuLinearIdBuilder gpuLinearIdBuilder(ctx, blockDimsOfr, numThreads);
+    diag = mlir::transform::gpu::mapOneForallToThreadsImpl(
+        rewriter, transformOp, forallOp, numThreads, syncAfterDistribute,
+        gpuLinearIdBuilder);
+    // Use silenceable failure to encode "failure to match" and pass through.
+    if (diag.isDefiniteFailure())
+      return WalkResult::interrupt();
+    if (diag.succeeded())
+      return WalkResult::skip();
+
+    //===--------------------------------------------------------------------===//
+    // Mapping to block ids (happens last so we can replay ThreadIdOp).
+    //===--------------------------------------------------------------------===//
+    LLVM_DEBUG(
+        llvm::interleaveComma(
+            blockDimsOfr, DBGS() << "mapNestedForallToThreadsImpl blockDims: ");
+        llvm::dbgs() << "\n");
+    GpuThreadIdBuilder gpuThreadIdBuilder(ctx, blockDimsOfr, blockDims);
+    diag = mlir::transform::gpu::mapOneForallToThreadsImpl(
+        rewriter, transformOp, forallOp, blockDims, syncAfterDistribute,
+        gpuThreadIdBuilder);
+    // Use silenceable failure to encode "failure to match" and pass through.
+    if (diag.isDefiniteFailure())
+      return WalkResult::interrupt();
+
+    return WalkResult::advance();
   });
-  return diag;
+  if (walkResult.wasInterrupted())
+    return diag;
+
+  // Replace ids of dimensions known to be 1 by 0 to simplify the IR.
+  // Here, the result of mapping determines the available mapping sizes.
+  replaceUnitMappingIdsHelper<ThreadIdOp>(rewriter, loc, target, zero,
+                                          blockDims);
+
+  return DiagnosedSilenceableFailure::success();
 }
 
-DiagnosedSilenceableFailure transform::MapNestedForeachToThreads::applyToOne(
+DiagnosedSilenceableFailure transform::MapNestedForallToThreads::applyToOne(
     Operation *target, ApplyToEachResultList &results, TransformState &state) {
   LaunchOp gpuLaunch = dyn_cast<LaunchOp>(target);
   auto transformOp = cast<TransformOpInterface>(getOperation());
 
-  if (!gpuLaunch) {
-    return emitSilenceableError() << "Given target is not gpu.launch";
-  }
+  // Basic high-level verifications.
+  if (!gpuLaunch)
+    return emitSilenceableError() << "Given target is not a gpu.launch";
 
-  SmallVector<int64_t> blockDim = extractFromI64ArrayAttr(getBlockDim());
-  blockDim.resize(/*size=*/3, /*value=*/1);
+  // Mapping to block ids.
+  SmallVector<int64_t> blockDims{getBlockDims()};
 
   DiagnosedSilenceableFailure diag =
       checkGpuLimits(transformOp, std::nullopt, std::nullopt, std::nullopt,
-                     blockDim[0], blockDim[1], blockDim[2]);
+                     blockDims[0], blockDims[1], blockDims[2]);
   if (diag.isSilenceableFailure()) {
-    diag.attachNote(getLoc()) << getBlockDimAttrName() << " is very large";
+    diag.attachNote(getLoc()) << getBlockDimsAttrName() << " is too large";
     return diag;
   }
 
-  MLIRContext *ctx = getContext();
-  TrivialPatternRewriter rewriter(ctx);
-  rewriter.setInsertionPoint(target);
-
-  SmallVector<DeviceMappingAttrInterface> threadMappingAttributes = {
-      GPUThreadMappingAttr::get(ctx, Threads::DimX),
-      GPUThreadMappingAttr::get(ctx, Threads::DimY),
-      GPUThreadMappingAttr::get(ctx, Threads::DimZ)};
-  auto threadIdGenerator = [](RewriterBase &rewriter, scf::ForallOp forallOp,
-                              SmallVectorImpl<Value> &threadIds) {
-    IndexType indexType = rewriter.getIndexType();
-    threadIds.assign({rewriter.create<ThreadIdOp>(forallOp->getLoc(), indexType,
-                                                  Dimension::x),
-                      rewriter.create<ThreadIdOp>(forallOp->getLoc(), indexType,
-                                                  Dimension::y),
-                      rewriter.create<ThreadIdOp>(forallOp->getLoc(), indexType,
-                                                  Dimension::z)});
-  };
-  diag = mlir::transform::gpu::mapNestedForeachToThreadsImpl(
-      rewriter, target, blockDim, threadIdGenerator, getSyncAfterDistribute(),
-      transformOp, threadMappingAttributes);
-
-  if (diag.succeeded()) {
-    diag = alterGpuLaunch(rewriter, gpuLaunch, transformOp, std::nullopt,
-                          std::nullopt, std::nullopt, blockDim[0], blockDim[1],
-                          blockDim[2]);
-  }
+  // Set the GPU launch configuration for the block dims early, this is not
+  // subject to IR inspection.
+  IRRewriter rewriter(getContext());
+  diag = alterGpuLaunch(rewriter, gpuLaunch, transformOp, std::nullopt,
+                        std::nullopt, std::nullopt, blockDims[0], blockDims[1],
+                        blockDims[2]);
+
+  rewriter.setInsertionPointToStart(&gpuLaunch.getBody().front());
+  diag =
+      mapNestedForallToThreadsImpl(rewriter, transformOp, gpuLaunch, blockDims,
+                                   getWarpDims(), getSyncAfterDistribute());
 
   results.push_back(gpuLaunch.getOperation());
   return diag;
diff --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
index c776fdd7ed5c6..1336ad1169c12 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
@@ -185,21 +185,31 @@ void AllocaOp::print(OpAsmPrinter &p) {
   auto funcTy =
       FunctionType::get(getContext(), {getArraySize().getType()}, {getType()});
 
+  if (getInalloca())
+    p << " inalloca";
+
   p << ' ' << getArraySize() << " x " << elemTy;
   if (getAlignment() && *getAlignment() != 0)
-    p.printOptionalAttrDict((*this)->getAttrs(), {kElemTypeAttrName});
-  else
     p.printOptionalAttrDict((*this)->getAttrs(),
-                            {"alignment", kElemTypeAttrName});
+                            {kElemTypeAttrName, getInallocaAttrName()});
+  else
+    p.printOptionalAttrDict(
+        (*this)->getAttrs(),
+        {getAlignmentAttrName(), kElemTypeAttrName, getInallocaAttrName()});
   p << " : " << funcTy;
 }
 
-// <operation> ::= `llvm.alloca` ssa-use `x` type attribute-dict?
-//                 `:` type `,` type
+// <operation> ::= `llvm.alloca` `inalloca`? ssa-use `x` type
+//                  attribute-dict? `:` type `,` type
 ParseResult AllocaOp::parse(OpAsmParser &parser, OperationState &result) {
   OpAsmParser::UnresolvedOperand arraySize;
   Type type, elemType;
   SMLoc trailingTypeLoc;
+
+  if (succeeded(parser.parseOptionalKeyword("inalloca")))
+    result.addAttribute(getInallocaAttrName(result.name),
+                        UnitAttr::get(parser.getContext()));
+
   if (parser.parseOperand(arraySize) || parser.parseKeyword("x") ||
       parser.parseType(elemType) ||
       parser.parseOptionalAttrDict(result.attributes) || parser.parseColon() ||
@@ -1613,13 +1623,16 @@ void GlobalOp::build(OpBuilder &builder, OperationState &result, Type type,
 
 void GlobalOp::print(OpAsmPrinter &p) {
   p << ' ' << stringifyLinkage(getLinkage()) << ' ';
+  StringRef visibility = stringifyVisibility(getVisibility_());
+  if (!visibility.empty())
+    p << visibility << ' ';
+  if (getThreadLocal_())
+    p << "thread_local ";
   if (auto unnamedAddr = getUnnamedAddr()) {
     StringRef str = stringifyUnnamedAddr(*unnamedAddr);
     if (!str.empty())
       p << str << ' ';
   }
-  if (getThreadLocal_())
-    p << "thread_local ";
   if (getConstant())
     p << "constant ";
   p.printSymbolName(getSymName());
@@ -1630,11 +1643,12 @@ void GlobalOp::print(OpAsmPrinter &p) {
   // Note that the alignment attribute is printed using the
   // default syntax here, even though it is an inherent attribute
   // (as defined in https://mlir.llvm.org/docs/LangRef/#attributes)
-  p.printOptionalAttrDict(
-      (*this)->getAttrs(),
-      {SymbolTable::getSymbolAttrName(), getGlobalTypeAttrName(),
-       getConstantAttrName(), getValueAttrName(), getLinkageAttrName(),
-       getUnnamedAddrAttrName(), getThreadLocal_AttrName()});
+  p.printOptionalAttrDict((*this)->getAttrs(),
+                          {SymbolTable::getSymbolAttrName(),
+                           getGlobalTypeAttrName(), getConstantAttrName(),
+                           getValueAttrName(), getLinkageAttrName(),
+                           getUnnamedAddrAttrName(), getThreadLocal_AttrName(),
+                           getVisibility_AttrName()});
 
   // Print the trailing type unless it's a string global.
   if (getValueOrNull().dyn_cast_or_null<StringAttr>())
@@ -1674,6 +1688,7 @@ struct EnumTraits {};
 REGISTER_ENUM_TYPE(Linkage);
 REGISTER_ENUM_TYPE(UnnamedAddr);
 REGISTER_ENUM_TYPE(CConv);
+REGISTER_ENUM_TYPE(Visibility);
 } // namespace
 
 /// Parse an enum from the keyword, or default to the provided default value.
@@ -1707,9 +1722,11 @@ ParseResult GlobalOp::parse(OpAsmParser &parser, OperationState &result) {
                           ctx, parseOptionalLLVMKeyword<Linkage>(
                                    parser, result, LLVM::Linkage::External)));
 
-  if (succeeded(parser.parseOptionalKeyword("thread_local")))
-    result.addAttribute(getThreadLocal_AttrName(result.name),
-                        parser.getBuilder().getUnitAttr());
+  // Parse optional visibility, default to Default.
+  result.addAttribute(getVisibility_AttrName(result.name),
+                      parser.getBuilder().getI64IntegerAttr(
+                          parseOptionalLLVMKeyword<LLVM::Visibility, int64_t>(
+                              parser, result, LLVM::Visibility::Default)));
 
   // Parse optional UnnamedAddr, default to None.
   result.addAttribute(getUnnamedAddrAttrName(result.name),
@@ -1717,6 +1734,10 @@ ParseResult GlobalOp::parse(OpAsmParser &parser, OperationState &result) {
                           parseOptionalLLVMKeyword<UnnamedAddr, int64_t>(
                               parser, result, LLVM::UnnamedAddr::None)));
 
+  if (succeeded(parser.parseOptionalKeyword("thread_local")))
+    result.addAttribute(getThreadLocal_AttrName(result.name),
+                        parser.getBuilder().getUnitAttr());
+
   if (succeeded(parser.parseOptionalKeyword("constant")))
     result.addAttribute(getConstantAttrName(result.name),
                         parser.getBuilder().getUnitAttr());
@@ -2037,6 +2058,12 @@ ParseResult LLVMFuncOp::parse(OpAsmParser &parser, OperationState &result) {
                        parseOptionalLLVMKeyword<Linkage>(
                            parser, result, LLVM::Linkage::External)));
 
+  // Parse optional visibility, default to Default.
+  result.addAttribute(getVisibility_AttrName(result.name),
+                      parser.getBuilder().getI64IntegerAttr(
+                          parseOptionalLLVMKeyword<LLVM::Visibility, int64_t>(
+                              parser, result, LLVM::Visibility::Default)));
+
   // Default to C Calling Convention if no keyword is provided.
   result.addAttribute(
       getCConvAttrName(result.name),
@@ -2087,6 +2114,9 @@ void LLVMFuncOp::print(OpAsmPrinter &p) {
   p << ' ';
   if (getLinkage() != LLVM::Linkage::External)
     p << stringifyLinkage(getLinkage()) << ' ';
+  StringRef visibility = stringifyVisibility(getVisibility_());
+  if (!visibility.empty())
+    p << visibility << ' ';
   if (getCConv() != LLVM::CConv::C)
     p << stringifyCConv(getCConv()) << ' ';
 
@@ -2108,7 +2138,7 @@ void LLVMFuncOp::print(OpAsmPrinter &p) {
   function_interface_impl::printFunctionAttributes(
       p, *this,
       {getFunctionTypeAttrName(), getArgAttrsAttrName(), getResAttrsAttrName(),
-       getLinkageAttrName(), getCConvAttrName()});
+       getLinkageAttrName(), getCConvAttrName(), getVisibility_AttrName()});
 
   // Print the body if this is not an external function.
   Region &body = getBody();
@@ -2408,7 +2438,7 @@ OpFoldResult LLVM::GEPOp::fold(FoldAdaptor adaptor) {
   // Canonicalize any dynamic indices of constant value to constant indices.
   bool changed = false;
   SmallVector<GEPArg> gepArgs;
-  for (auto &iter : llvm::enumerate(indices)) {
+  for (auto iter : llvm::enumerate(indices)) {
     auto integer = iter.value().dyn_cast_or_null<IntegerAttr>();
     // Constant indices can only be int32_t, so if integer does not fit we
     // are forced to keep it dynamic, despite being a constant.
diff --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMTypes.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMTypes.cpp
index 2e6bf86022fa3..4b3b1e183903f 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMTypes.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMTypes.cpp
@@ -731,6 +731,8 @@ bool mlir::LLVM::isCompatibleOuterType(Type type) {
       Float64Type,
       Float80Type,
       Float128Type,
+      Float8E4M3FNType,
+      Float8E5M2Type,
       LLVMArrayType,
       LLVMFunctionType,
       LLVMLabelType,
@@ -801,6 +803,8 @@ static bool isCompatibleImpl(Type type, DenseSet<Type> &compatibleTypes) {
             Float64Type,
             Float80Type,
             Float128Type,
+            Float8E4M3FNType,
+            Float8E5M2Type,
             LLVMLabelType,
             LLVMMetadataType,
             LLVMPPCFP128Type,
diff --git a/mlir/lib/Dialect/Linalg/IR/LinalgInterfaces.cpp b/mlir/lib/Dialect/Linalg/IR/LinalgInterfaces.cpp
index 8e8b08d40da65..393dba20e5fe5 100644
--- a/mlir/lib/Dialect/Linalg/IR/LinalgInterfaces.cpp
+++ b/mlir/lib/Dialect/Linalg/IR/LinalgInterfaces.cpp
@@ -650,10 +650,18 @@ LinalgOp::reifyResultShapes(OpBuilder &b,
   for (OpOperand *opOperand : getDpsInitOperands()) {
     SmallVector<OpFoldResult> shapes;
     for (int64_t dim : llvm::seq<int64_t>(0, getRank(opOperand))) {
-      if (checkDimExpr.visit(shapeExprs[pos]))
-        shapes.push_back(createOrFoldDimOp(b, loc, opOperand->get(), dim));
-      else
-        shapes.push_back(allResultDimValues[pos]);
+      auto shapedType = opOperand->get().getType().cast<ShapedType>();
+      if (!shapedType.isDynamicDim(dim)) {
+        // Static dim: Return IntegerAttr.
+        shapes.push_back(b.getIndexAttr(shapedType.getDimSize(dim)));
+      } else {
+        // Dynamic dim: Return Value.
+        OpFoldResult ofr =
+            checkDimExpr.visit(shapeExprs[pos])
+                ? createOrFoldDimOp(b, loc, opOperand->get(), dim)
+                : allResultDimValues[pos];
+        shapes.push_back(getValueOrCreateConstantIndexOp(b, loc, ofr));
+      }
       pos++;
     }
     reifiedReturnShapes.emplace_back(std::move(shapes));
diff --git a/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp b/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
index 9011a8174854e..5b53e9925c881 100644
--- a/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
+++ b/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
@@ -482,9 +482,7 @@ struct FoldFillWithPad final : public OpRewritePattern<tensor::PadOp> {
       return failure();
 
     ReifiedRankedShapedTypeDims reifiedShape;
-    ReifyRankedShapedTypeOpInterface interface =
-        cast<ReifyRankedShapedTypeOpInterface>(padOp.getOperation());
-    if (failed(interface.reifyResultShapes(rewriter, reifiedShape)))
+    if (failed(reifyResultShapes(rewriter, padOp, reifiedShape)))
       return rewriter.notifyMatchFailure(
           padOp, "failed to reify tensor.pad op result shape");
 
@@ -2341,7 +2339,7 @@ static void createNewOperandWithStaticSizes(
   for (unsigned i = 0; i < sourceShape.size(); i++) {
     int64_t dimShape = sourceShape[i];
     AffineExpr dimExpr = sourceMap.getResult(i);
-    if (affineExprToSize.find(dimExpr) == affineExprToSize.end() ||
+    if (!affineExprToSize.contains(dimExpr) ||
         !sourceType.isDynamicDim(i)) {
       newShape.push_back(dimShape);
       continue;
diff --git a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
index 600cddec47d17..407b8d213de1c 100644
--- a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
+++ b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
@@ -24,7 +24,6 @@
 #include "mlir/Dialect/Transform/IR/TransformDialect.h"
 #include "mlir/Dialect/Transform/IR/TransformInterfaces.h"
 #include "mlir/Dialect/Transform/IR/TransformTypes.h"
-#include "mlir/Dialect/Transform/IR/TransformUtils.h"
 #include "mlir/Dialect/Transform/Utils/Utils.h"
 #include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/IR/AffineMap.h"
@@ -68,6 +67,13 @@ static FailureOr<LinalgOp> tryApply(Operation *operation, Args &&...args) {
 
   // Apply the pattern directly to the op.
   PatternTy pattern(operation->getContext(), std::forward<Args>(args)...);
+  // We want to discourage direct use of PatternRewriter in APIs but In this
+  // very specific case, an IRRewriter is not enough.
+  struct TrivialPatternRewriter : public PatternRewriter {
+  public:
+    explicit TrivialPatternRewriter(MLIRContext *context)
+        : PatternRewriter(context) {}
+  };
   TrivialPatternRewriter rewriter(operation->getContext());
   rewriter.setInsertionPoint(operation);
   auto result = pattern.returningMatchAndRewrite(op, rewriter);
@@ -266,6 +272,7 @@ transform::DecomposeOp::applyToOne(LinalgOp target,
   DOWNSCALE_NORMAL(PoolingNhwcMinUnsignedOp, PoolingNwcMinUnsignedOp)
   DOWNSCALE_NORMAL(PoolingNchwMaxOp, PoolingNcwMaxOp)
   DOWNSCALE(DownscaleDepthwiseConv2DNhwcHwcOp)
+  DOWNSCALE(DownscaleConv2DOp)
 #undef DOWNSCALE_NORMAL
 #undef DOWNSCALE_CALL
 #undef DOWNSCALE
@@ -292,7 +299,7 @@ static LogicalResult applyTilingToAll(
     if (!tilingInterfaceOp)
       return transformOp->emitError("only TilingInterface ops are supported");
 
-    TrivialPatternRewriter rewriter(target->getContext());
+    IRRewriter rewriter(target->getContext());
     rewriter.setInsertionPoint(target);
     FailureOr<scf::SCFTileAndFuseResult> tiledResults =
         applyFn(tilingInterfaceOp);
@@ -376,7 +383,7 @@ transform::FuseOp::apply(mlir::transform::TransformResults &transformResults,
       tileSizes.size() - llvm::count(tileSizes, 0), transformResults,
       [&](TilingInterface tilingInterfaceOp)
           -> FailureOr<scf::SCFTileAndFuseResult> {
-        TrivialPatternRewriter rewriter(getContext());
+        IRRewriter rewriter(getContext());
         return tileConsumerAndFuseProducerGreedilyUsingSCFForOp(
             rewriter, tilingInterfaceOp, tileAndFuseOptions);
       });
@@ -424,16 +431,15 @@ void transform::FuseIntoContainingOp::build(OpBuilder &builder,
 /// Find the first "extract" user of `producerOp` and tile it right before its
 /// use. The tiled op is fused under the `containingOp`.
 /// Return this fused op on success or nullptr if anything fails.
-static Operation *tileAndFuseFirstExtractUse(RewriterBase &rewriter,
-                                             Diagnostic &diag,
-                                             Operation *producerOp,
-                                             Operation *containingOp) {
+static SmallVector<Operation *>
+tileAndFuseFirstExtractUse(RewriterBase &rewriter, Diagnostic &diag,
+                           Operation *producerOp, Operation *containingOp) {
   LLVM_DEBUG(DBGS() << "Try to fuse a direct extract use\n");
   auto tileableProducer = dyn_cast<TilingInterface>(producerOp);
   if (!tileableProducer) {
     diag.attachNote(producerOp->getLoc())
         << "producer is not a TileableInterface: " << *producerOp;
-    return nullptr;
+    return {};
   }
 
   // Search the producer slices accessed within the containing operation.
@@ -448,7 +454,7 @@ static Operation *tileAndFuseFirstExtractUse(RewriterBase &rewriter,
   if (it == tileableProducer->getUsers().end()) {
     diag.attachNote(tileableProducer->getLoc())
         << "could not find fusion opportunity for: " << *tileableProducer;
-    return nullptr;
+    return {};
   }
   auto sliceOpToTile = cast<tensor::ExtractSliceOp>(*it);
 
@@ -461,27 +467,32 @@ static Operation *tileAndFuseFirstExtractUse(RewriterBase &rewriter,
       sliceOpToTile.getSource().cast<OpResult>().getResultNumber();
   LLVM_DEBUG(DBGS() << "resultNumber: " << resultNumber << "\n");
 
-  FailureOr<Value> tiledProducer = tileableProducer.generateResultTileValue(
-      rewriter, resultNumber, sliceOpToTile.getMixedOffsets(),
-      sliceOpToTile.getMixedSizes());
-  if (failed(tiledProducer)) {
+  FailureOr<TilingResult> tileAndFuseResult =
+      tileableProducer.generateResultTileValue(rewriter, resultNumber,
+                                               sliceOpToTile.getMixedOffsets(),
+                                               sliceOpToTile.getMixedSizes());
+  if (failed(tileAndFuseResult)) {
     diag.attachNote(tileableProducer->getLoc())
         << "failed to tile producer op: " << *tileableProducer;
-    return nullptr;
+    return {};
+  }
+
+#ifndef NDEBUG
+  for (auto tiledOp : tileAndFuseResult->tiledOps) {
+    LLVM_DEBUG(DBGS() << "tiledProducer: " << *tiledOp << "\n");
   }
-  LLVM_DEBUG(DBGS() << "tiledProducer: " << *tiledProducer << "\n");
+#endif
 
   // Replace the extract op.
-  Operation *fusedOp = tiledProducer->getDefiningOp();
   auto maybeRankReduced = tensor::ExtractSliceOp::rankReduceIfNeeded(
-      rewriter, sliceOpToTile->getLoc(), fusedOp->getResult(resultNumber),
+      rewriter, sliceOpToTile->getLoc(), tileAndFuseResult->tiledValues[0],
       sliceOpToTile->getResult(0)
           .getType()
           .cast<RankedTensorType>()
           .getShape());
   assert(succeeded(maybeRankReduced) && "unexpected shape");
   rewriter.replaceOp(sliceOpToTile, *maybeRankReduced);
-  return fusedOp;
+  return tileAndFuseResult->tiledOps;
 }
 
 /// First, find the first "scf::ForallOp" user of `producerOp` and ensure
@@ -490,7 +501,8 @@ static Operation *tileAndFuseFirstExtractUse(RewriterBase &rewriter,
 /// right before its "extract" use. The tiled op is fused under the
 /// `containingOp`.
 /// Return this fused op on success or nullptr if anything fails.
-static Operation *tileAndFuseFirstExtractUseThroughContainingOpBlockArgument(
+static SmallVector<Operation *>
+tileAndFuseFirstExtractUseThroughContainingOpBlockArgument(
     RewriterBase &rewriter, Diagnostic &diag, Operation *producerOp,
     Operation *containingOp) {
   LLVM_DEBUG(DBGS() << "Try to fuse an extract use through block argument\n");
@@ -499,7 +511,7 @@ static Operation *tileAndFuseFirstExtractUseThroughContainingOpBlockArgument(
   if (!tileableProducer) {
     diag.attachNote(producerOp->getLoc())
         << "producer is not a TileableInterface: " << *producerOp;
-    return nullptr;
+    return {};
   }
 
   // Search the first use by a "scf::ForallOp" user.
@@ -513,7 +525,7 @@ static Operation *tileAndFuseFirstExtractUseThroughContainingOpBlockArgument(
   if (!forallOp || forallOp != containingOp) {
     diag.attachNote(tileableProducer->getLoc())
         << "could not find a use by the containing op: " << *tileableProducer;
-    return nullptr;
+    return {};
   }
 
   // Search the producer slices accessed within the containing
@@ -535,7 +547,7 @@ static Operation *tileAndFuseFirstExtractUseThroughContainingOpBlockArgument(
   if (itBBArgUsers == bbArg.getUsers().end()) {
     diag.attachNote(containingOp->getLoc())
         << "could not find fusion opportunity for bbArg: " << bbArg;
-    return nullptr;
+    return {};
   }
   auto sliceOpToTile = cast<tensor::ExtractSliceOp>(*itBBArgUsers);
 
@@ -555,7 +567,7 @@ static Operation *tileAndFuseFirstExtractUseThroughContainingOpBlockArgument(
           destinationTensors))) {
     diag.attachNote(tileableProducer->getLoc())
         << "failed to get destination tensors for: " << *tileableProducer;
-    return nullptr;
+    return {};
   }
 
   IRMapping bvm;
@@ -566,21 +578,19 @@ static Operation *tileAndFuseFirstExtractUseThroughContainingOpBlockArgument(
       llvm::make_scope_exit([&]() { rewriter.eraseOp(tileableProducerClone); });
 
   // Tile the producer.
-  FailureOr<Value> tiledProducer =
+  FailureOr<TilingResult> tileAndFuseResult =
       tileableProducerClone.generateResultTileValue(
           rewriter, resultNumber, sliceOpToTile.getMixedOffsets(),
           sliceOpToTile.getMixedSizes());
-  if (failed(tiledProducer)) {
+  if (failed(tileAndFuseResult)) {
     diag.attachNote(tileableProducer->getLoc())
         << "failed to tile producer op: " << *tileableProducer;
-    return nullptr;
+    return {};
   }
-  LLVM_DEBUG(DBGS() << "tiledProducer: " << *tiledProducer << "\n");
 
   // Replace the extract op.
-  Operation *fusedOp = tiledProducer->getDefiningOp();
   auto maybeRankReduced = tensor::ExtractSliceOp::rankReduceIfNeeded(
-      rewriter, sliceOpToTile->getLoc(), fusedOp->getResult(resultNumber),
+      rewriter, sliceOpToTile->getLoc(), tileAndFuseResult->tiledValues[0],
       sliceOpToTile->getResult(0)
           .getType()
           .cast<RankedTensorType>()
@@ -594,7 +604,7 @@ static Operation *tileAndFuseFirstExtractUseThroughContainingOpBlockArgument(
                              destinationTensors.front());
   });
 
-  return fusedOp;
+  return tileAndFuseResult->tiledOps;
 }
 
 static Operation *cloneAndFuseFirstUse(RewriterBase &rewriter, Diagnostic &diag,
@@ -707,21 +717,21 @@ transform::FuseIntoContainingOp::apply(transform::TransformResults &results,
     // cases, we can tile/clone once and reuse the value for each use.
     // Futhermore, producers should then be traversed according to a
     // topological sorting.
-    Operation *tiled =
+    SmallVector<Operation *> tiledOps =
         tileAndFuseFirstExtractUse(rewriter, diag, producerOp, containingOp);
-    if (tiled) {
+    if (!tiledOps.empty()) {
       LLVM_DEBUG(DBGS() << "\nFused a direct extract use\n" << *containingOp);
-      fusedOps.push_back(tiled);
+      fusedOps.append(tiledOps);
       continue;
     }
 
-    Operation *tiledContainingOpOperand =
+    SmallVector<Operation *> tiledContainingOpOperand =
         tileAndFuseFirstExtractUseThroughContainingOpBlockArgument(
             rewriter, diag, producerOp, containingOp);
-    if (tiledContainingOpOperand) {
+    if (!tiledContainingOpOperand.empty()) {
       LLVM_DEBUG(DBGS() << "\nFused an extract use through block argument\n"
                         << *containingOp);
-      fusedOps.push_back(tiledContainingOpOperand);
+      fusedOps.append(tiledContainingOpOperand);
       continue;
     }
 
@@ -760,7 +770,9 @@ transform::GeneralizeOp::applyToOne(LinalgOp target,
     results.push_back(target);
     return DiagnosedSilenceableFailure::success();
   }
-  FailureOr<LinalgOp> generic = tryApply<LinalgGeneralizationPattern>(target);
+  IRRewriter rewriter(getContext());
+  rewriter.setInsertionPoint(target);
+  FailureOr<LinalgOp> generic = generalizeNamedOp(rewriter, target);
   if (succeeded(generic)) {
     results.push_back(generic->getOperation());
     return DiagnosedSilenceableFailure::success();
@@ -782,7 +794,7 @@ transform::InterchangeOp::applyToOne(GenericOp target,
     results.push_back(target);
     return DiagnosedSilenceableFailure::success();
   }
-  TrivialPatternRewriter rewriter(target->getContext());
+  IRRewriter rewriter(target->getContext());
   FailureOr<GenericOp> res =
       interchangeGenericOp(rewriter, target,
                            SmallVector<unsigned>(interchangeVector.begin(),
@@ -1866,7 +1878,7 @@ transform::PromoteOp::applyToOne(LinalgOp target,
   if (failed(promoteSubviewsPrecondition(target, promotionOptions)))
     return emitDefaultDefiniteFailure(target);
 
-  TrivialPatternRewriter rewriter(target->getContext());
+  IRRewriter rewriter(target->getContext());
   rewriter.setInsertionPoint(target);
   FailureOr<LinalgOp> res = promoteSubViews(rewriter, target, promotionOptions);
   if (failed(res))
@@ -1965,7 +1977,7 @@ transform::ScalarizeOp::applyToOne(LinalgOp target,
     return tileSizes;
   });
   SmallVector<int64_t> emptyTileSizes;
-  TrivialPatternRewriter rewriter(getContext());
+  IRRewriter rewriter(getContext());
   rewriter.setInsertionPoint(target);
   FailureOr<scf::SCFTilingResult> maybeTilingResult = tileUsingSCFForOp(
       rewriter, cast<TilingInterface>(target.getOperation()), tilingOptions);
@@ -2017,7 +2029,7 @@ DiagnosedSilenceableFailure SplitOp::apply(TransformResults &results,
                                            TransformState &state) {
   // Collect the dynamic split points if provided.
   ArrayRef<Operation *> payload = state.getPayloadOps(getTarget());
-  TrivialPatternRewriter rewriter(getContext());
+  IRRewriter rewriter(getContext());
   SmallVector<OpFoldResult> splitPoints;
   splitPoints.reserve(payload.size());
   if (getDynamicSplitPoint()) {
@@ -2224,7 +2236,7 @@ DiagnosedSilenceableFailure transform::SplitReductionOp::applyToOne(
                                          unsigned(getInsertSplitDimension()),
                                          bool(getInnerParallel())};
   };
-  TrivialPatternRewriter rewriter(getContext());
+  IRRewriter rewriter(getContext());
   rewriter.setInsertionPoint(target);
   FailureOr<SplitReductionResult> splitResult =
       (getUseScalingAlgorithm())
@@ -2264,7 +2276,7 @@ void transform::TileReductionUsingScfOp::build(
 DiagnosedSilenceableFailure transform::TileReductionUsingScfOp::applyToOne(
     LinalgOp target, transform::ApplyToEachResultList &results,
     transform::TransformState &state) {
-  TrivialPatternRewriter rewriter(getContext());
+  IRRewriter rewriter(getContext());
   rewriter.setInsertionPoint(target);
   FailureOr<scf::SCFReductionTilingResult> result = scf::tileReductionUsingScf(
       rewriter, cast<PartialReductionOpInterface>(target.getOperation()),
@@ -2307,7 +2319,7 @@ void transform::TileReductionUsingForallOp::build(
 DiagnosedSilenceableFailure transform::TileReductionUsingForallOp::applyToOne(
     LinalgOp target, transform::ApplyToEachResultList &results,
     transform::TransformState &state) {
-  TrivialPatternRewriter rewriter(getContext());
+  IRRewriter rewriter(getContext());
   rewriter.setInsertionPoint(target);
   SmallVector<OpFoldResult> numThreads =
       getAsOpFoldResult(rewriter.getI64ArrayAttr(getNumThreads()));
@@ -2453,7 +2465,7 @@ transform::TileOp::apply(TransformResults &transformResults,
   SmallVector<Operation *> tiled;
   SmallVector<SmallVector<Operation *, 4>, 4> loops;
   loops.resize(getLoops().size());
-  for (auto &[i, op] : llvm::enumerate(targets)) {
+  for (auto [i, op] : llvm::enumerate(targets)) {
     auto tilingInterface = dyn_cast<TilingInterface>(op);
     auto dpsInterface = dyn_cast<DestinationStyleOpInterface>(op);
     if (!tilingInterface || !dpsInterface) {
@@ -2494,7 +2506,7 @@ transform::TileOp::apply(TransformResults &transformResults,
     }
 
     tilingOptions.setInterchange(getInterchange());
-    TrivialPatternRewriter rewriter(op->getContext());
+    IRRewriter rewriter(op->getContext());
     FailureOr<scf::SCFTilingResult> maybeTilingResult =
         tileUsingSCFForOp(rewriter, tilingInterface, tilingOptions);
     if (failed(maybeTilingResult))
@@ -2856,7 +2868,7 @@ transform::TileToScfForOp::apply(TransformResults &transformResults,
   SmallVector<Operation *> tiled;
   SmallVector<SmallVector<Operation *, 4>, 4> loops;
   loops.resize(getLoops().size());
-  for (auto &en : llvm::enumerate(targets)) {
+  for (auto en : llvm::enumerate(targets)) {
     auto tilingInterfaceOp = dyn_cast<TilingInterface>(en.value());
     if (!tilingInterfaceOp) {
       DiagnosedSilenceableFailure diag =
@@ -2887,7 +2899,7 @@ transform::TileToScfForOp::apply(TransformResults &transformResults,
     }
 
     tilingOptions.setInterchange(getInterchange());
-    TrivialPatternRewriter rewriter(tilingInterfaceOp.getContext());
+    IRRewriter rewriter(tilingInterfaceOp.getContext());
     FailureOr<scf::SCFTilingResult> tilingResult =
         tileUsingSCFForOp(rewriter, tilingInterfaceOp, tilingOptions);
     if (failed(tilingResult))
diff --git a/mlir/lib/Dialect/Linalg/Transforms/ConvertToDestinationStyle.cpp b/mlir/lib/Dialect/Linalg/Transforms/ConvertToDestinationStyle.cpp
index 700f873d19729..3ec5094ed90b5 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/ConvertToDestinationStyle.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/ConvertToDestinationStyle.cpp
@@ -125,19 +125,17 @@ static SmallVector<Value> reifyOrComputeDynamicSizes(OpBuilder &b,
     return {};
 
   // Try to reify dynamic sizes.
-  if (auto reifiableOp =
-          value.getDefiningOp<ReifyRankedShapedTypeOpInterface>()) {
-    ReifiedRankedShapedTypeDims reifiedShape;
-    if (succeeded(reifiableOp.reifyResultShapes(b, reifiedShape))) {
-      SmallVector<Value> dynSizes;
-      for (int64_t i = 0; i < tensorType.getRank(); ++i) {
-        if (tensorType.isDynamicDim(i))
-          dynSizes.push_back(
-              reifiedShape[value.cast<OpResult>().getResultNumber()][i]
-                  .get<Value>());
-      }
-      return dynSizes;
+  ReifiedRankedShapedTypeDims reifiedShape;
+  if (value.isa<OpResult>() &&
+      succeeded(reifyResultShapes(b, value.getDefiningOp(), reifiedShape))) {
+    SmallVector<Value> dynSizes;
+    for (int64_t i = 0; i < tensorType.getRank(); ++i) {
+      if (tensorType.isDynamicDim(i))
+        dynSizes.push_back(
+            reifiedShape[value.cast<OpResult>().getResultNumber()][i]
+                .get<Value>());
     }
+    return dynSizes;
   }
 
   // Create tensor.dim ops.
@@ -293,8 +291,7 @@ mlir::linalg::rewriteInDestinationPassingStyle(RewriterBase &rewriter,
   Location loc = padOp.getLoc();
   RankedTensorType resultType = padOp.getResultType();
   ReifiedRankedShapedTypeDims reifiedShape;
-  if (failed(cast<ReifyRankedShapedTypeOpInterface>(padOp.getOperation())
-                 .reifyResultShapes(rewriter, reifiedShape)))
+  if (failed(reifyResultShapes(rewriter, padOp, reifiedShape)))
     return rewriter.notifyMatchFailure(
         padOp, "failed to reify tensor.pad op result shape");
   SmallVector<Value> dynamicSizes;
diff --git a/mlir/lib/Dialect/Linalg/Transforms/DataLayoutPropagation.cpp b/mlir/lib/Dialect/Linalg/Transforms/DataLayoutPropagation.cpp
index 2d1b544e468c8..e1b929c8b121e 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/DataLayoutPropagation.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/DataLayoutPropagation.cpp
@@ -157,7 +157,7 @@ getOrCreatePackedViewOfOperand(OpBuilder &b, Location loc, PackInfo packInfo,
   AffineMap origIndexingMap = genericOp.getMatchingIndexingMap(opOperand);
   llvm::DenseMap<int64_t, int64_t> domainDimToOperandDim;
   SmallVector<AffineExpr> exprs(origIndexingMap.getResults());
-  if (genericOp.isScalar(opOperand) || exprs.size() == 0)
+  if (genericOp.isScalar(opOperand) || exprs.empty())
     return std::make_tuple(opOperand->get(),
                            AffineMap::get(numLoops, 0, exprs, b.getContext()));
 
diff --git a/mlir/lib/Dialect/Linalg/Transforms/ElementwiseOpFusion.cpp b/mlir/lib/Dialect/Linalg/Transforms/ElementwiseOpFusion.cpp
index 6e2aa2c2755bc..4df55ddd62e0f 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/ElementwiseOpFusion.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/ElementwiseOpFusion.cpp
@@ -1394,7 +1394,7 @@ void generateCollapsedIndexingRegion(Location loc, Block *block,
   //   i1 = (i_{folded} / d2) % d1
   //   i0 = i_{folded} / (d1 * d2)
   llvm::DenseMap<unsigned, Value> indexReplacementVals;
-  for (auto &foldedDims :
+  for (auto foldedDims :
        enumerate(collapsingInfo.getCollapsedOpToOrigOpMapping())) {
     ReassociationIndicesRef foldedDimsRef(foldedDims.value());
     Value newIndexVal =
diff --git a/mlir/lib/Dialect/Linalg/Transforms/FusePadOpWithLinalgProducer.cpp b/mlir/lib/Dialect/Linalg/Transforms/FusePadOpWithLinalgProducer.cpp
index 760b14edb7943..b6e2ffcbba368 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/FusePadOpWithLinalgProducer.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/FusePadOpWithLinalgProducer.cpp
@@ -62,10 +62,7 @@ struct FusePadOp : OpRewritePattern<tensor::PadOp> {
           padOp, "only supported for ops with all parallel iterator types");
     }
     ReifiedRankedShapedTypeDims resultShape;
-    ReifyRankedShapedTypeOpInterface reifyShapedTypeInterface =
-        dyn_cast<ReifyRankedShapedTypeOpInterface>(padOp.getOperation());
-    if (failed(reifyShapedTypeInterface.reifyResultShapes(rewriter,
-                                                          resultShape)) ||
+    if (failed(reifyResultShapes(rewriter, padOp, resultShape)) ||
         resultShape.size() != 1) {
       return rewriter.notifyMatchFailure(
           padOp, "failed to get shape of pad op result");
diff --git a/mlir/lib/Dialect/Linalg/Transforms/Loops.cpp b/mlir/lib/Dialect/Linalg/Transforms/Loops.cpp
index ae680a1f2c837..583128a418125 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/Loops.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/Loops.cpp
@@ -175,8 +175,8 @@ static void emitScalarImplementation(OpBuilder &b, Location loc,
 
 /// Replace the index operations in the body of the loop nest by the matching
 /// induction variables.
-static void replaceIndexOpsByInductionVariables(LinalgOp linalgOp,
-                                                PatternRewriter &rewriter,
+static void replaceIndexOpsByInductionVariables(RewriterBase &rewriter,
+                                                LinalgOp linalgOp,
                                                 ArrayRef<Operation *> loopOps) {
   // Extract the induction variables of the loop nest from outer to inner.
   SmallVector<Value> allIvs;
@@ -206,7 +206,7 @@ static void replaceIndexOpsByInductionVariables(LinalgOp linalgOp,
 }
 
 template <typename LoopTy>
-static FailureOr<LinalgLoops> linalgOpToLoopsImpl(PatternRewriter &rewriter,
+static FailureOr<LinalgLoops> linalgOpToLoopsImpl(RewriterBase &rewriter,
                                                   LinalgOp linalgOp) {
   using LoadOpTy = std::conditional_t<std::is_same<LoopTy, AffineForOp>::value,
                                       AffineLoadOp, memref::LoadOp>;
@@ -247,7 +247,7 @@ static FailureOr<LinalgLoops> linalgOpToLoopsImpl(PatternRewriter &rewriter,
   }
   LinalgLoops loops(loopSet.begin(), loopSet.end());
   // Replace all index operations in the loop body.
-  replaceIndexOpsByInductionVariables(linalgOp, rewriter, loops);
+  replaceIndexOpsByInductionVariables(rewriter, linalgOp, loops);
   return loops;
 }
 
@@ -367,20 +367,19 @@ mlir::createConvertLinalgToAffineLoopsPass() {
 
 /// Emits a loop nest of `affine.for` with the proper body for `linalgOp`.
 FailureOr<LinalgLoops>
-mlir::linalg::linalgOpToAffineLoops(PatternRewriter &rewriter,
-                                    LinalgOp linalgOp) {
+mlir::linalg::linalgOpToAffineLoops(RewriterBase &rewriter, LinalgOp linalgOp) {
   return linalgOpToLoopsImpl<AffineForOp>(rewriter, linalgOp);
 }
 
 /// Emits a loop nest of `scf.for` with the proper body for `linalgOp`.
-FailureOr<LinalgLoops> mlir::linalg::linalgOpToLoops(PatternRewriter &rewriter,
+FailureOr<LinalgLoops> mlir::linalg::linalgOpToLoops(RewriterBase &rewriter,
                                                      LinalgOp linalgOp) {
   return linalgOpToLoopsImpl<scf::ForOp>(rewriter, linalgOp);
 }
 
 /// Emits a loop nest of `scf.parallel` with the proper body for `linalgOp`.
 FailureOr<LinalgLoops>
-mlir::linalg::linalgOpToParallelLoops(PatternRewriter &rewriter,
+mlir::linalg::linalgOpToParallelLoops(RewriterBase &rewriter,
                                       LinalgOp linalgOp) {
   return linalgOpToLoopsImpl<scf::ParallelOp>(rewriter, linalgOp);
 }
diff --git a/mlir/lib/Dialect/Linalg/Transforms/Split.cpp b/mlir/lib/Dialect/Linalg/Transforms/Split.cpp
index c8c9c0bd4af89..e6fce56d4140b 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/Split.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/Split.cpp
@@ -41,26 +41,26 @@ createSplitPart(RewriterBase &b, Location loc, TilingInterface op,
   offsetsCopy[dimension] = offset;
 
   // Create the part as it it were a single tile.
-  SmallVector<Operation *> tiled =
+  FailureOr<TilingResult> tilingResult =
       op.getTiledImplementation(b, offsetsCopy, sizesCopy);
-  assert(tiled.size() == 1 && "expected a single result from tiling");
-  auto part = cast<TilingInterface>(tiled.front());
 
   // Insert the results back and populate the `results` list.
-  for (auto i : llvm::seq<unsigned>(0, part->getNumResults())) {
+  for (auto [index, result] : llvm::enumerate(tilingResult->tiledValues)) {
     SmallVector<OpFoldResult> resultOffsets, resultSizes;
-    if (failed(op.getResultTilePosition(b, i, offsetsCopy, sizesCopy,
+    if (failed(op.getResultTilePosition(b, index, offsetsCopy, sizesCopy,
                                         resultOffsets, resultSizes)))
       return nullptr;
     SmallVector<OpFoldResult> resultStrides(resultOffsets.size(),
                                             b.getIndexAttr(1));
     Value inserted = b.create<tensor::InsertSliceOp>(
-        loc, part->getResult(i), resultOperands[i], resultOffsets, resultSizes,
+        loc, result, resultOperands[index], resultOffsets, resultSizes,
         resultStrides);
     results.push_back(inserted);
   }
-
-  return part;
+  // TODO: this part can be generalized maybe to not expect a single op.
+  assert(tilingResult->tiledOps.size() == 1 &&
+         "expected split part to return a single tiled operation");
+  return cast<TilingInterface>(tilingResult->tiledOps[0]);
 }
 
 std::pair<TilingInterface, TilingInterface>
diff --git a/mlir/lib/Dialect/Linalg/Transforms/SplitReduction.cpp b/mlir/lib/Dialect/Linalg/Transforms/SplitReduction.cpp
index 26dee47b8926a..d0e865271a39a 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/SplitReduction.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/SplitReduction.cpp
@@ -28,7 +28,7 @@ using namespace mlir;
 using namespace mlir::linalg;
 
 FailureOr<SplitReductionResult> mlir::linalg::splitReduction(
-    PatternRewriter &b, LinalgOp op,
+    RewriterBase &b, LinalgOp op,
     const ControlSplitReductionFn &controlSplitReductionFn, bool useAlloc) {
   OpBuilder::InsertionGuard guard(b);
   b.setInsertionPoint(op);
@@ -156,10 +156,11 @@ FailureOr<SplitReductionResult> mlir::linalg::splitReduction(
   newMaps.push_back(AffineMap::get(oldOutputMap.getNumDims() + 1, 0, outputExpr,
                                    op.getContext()));
   SmallVector<utils::IteratorType> newIteratorTypes;
-  for (auto &it : llvm::enumerate(op.getIteratorTypesArray())) {
-    if (insertSplitDimension == it.index())
+  for (auto [index, iteratorType] :
+       llvm::enumerate(op.getIteratorTypesArray())) {
+    if (insertSplitDimension == index)
       newIteratorTypes.push_back(utils::IteratorType::parallel);
-    newIteratorTypes.push_back(it.value());
+    newIteratorTypes.push_back(iteratorType);
   }
   if (insertSplitDimension == op.getIteratorTypesArray().size()) {
     newIteratorTypes.push_back(utils::IteratorType::parallel);
@@ -238,7 +239,7 @@ static AffineMap insertParallelDim(LinalgOp op, OpOperand &opOperand,
 
 /// Core rewrite implementation.
 FailureOr<SplitReductionResult> mlir::linalg::splitReductionByScaling(
-    PatternRewriter &b, LinalgOp op,
+    RewriterBase &b, LinalgOp op,
     const ControlSplitReductionFn &controlSplitReductionFn, bool useAlloc) {
   OpBuilder::InsertionGuard guard(b);
   b.setInsertionPoint(op);
diff --git a/mlir/lib/Dialect/Linalg/Transforms/Tiling.cpp b/mlir/lib/Dialect/Linalg/Transforms/Tiling.cpp
index dda0f491c6aaa..453fd85f19bb6 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/Tiling.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/Tiling.cpp
@@ -89,7 +89,7 @@ void mlir::linalg::transformIndexOps(
     RewriterBase &b, LinalgOp op, SmallVectorImpl<Value> &ivs,
     const LoopIndexToRangeIndexMap &loopIndexToRangeIndex) {
   SmallVector<Value> allIvs(op.getNumLoops(), nullptr);
-  for (auto &en : enumerate(allIvs)) {
+  for (auto en : enumerate(allIvs)) {
     auto rangeIndex = loopIndexToRangeIndex.find(en.index());
     if (rangeIndex == loopIndexToRangeIndex.end())
       continue;
@@ -388,12 +388,13 @@ static FailureOr<ForallTilingResult> tileToForallOpImpl(
     }
 
     // 4. Tile the cloned op and delete the clone.
-    SmallVector<Operation *> tiledOps =
+    FailureOr<TilingResult> tilingResult =
         cast<TilingInterface>(clonedOp).getTiledImplementation(b, tiledOffsets,
                                                                tiledSizes);
     b.eraseOp(clonedOp);
-    assert(tiledOps.size() == 1 && "expected a single produced tiled op");
-    tiledOp = tiledOps.front();
+    assert(tilingResult->tiledOps.size() == 1 &&
+           "expected a single produced tiled op");
+    tiledOp = tilingResult->tiledOps.front();
   }
 
   // 5. Parallel insert back into the result tensor.
@@ -691,7 +692,7 @@ FailureOr<linalg::ForallReductionTilingResult> linalg::tileReductionUsingForall(
 
   // 4. Clone the tileable op and update its destination operands to use the
   // output bbArgs of the ForallOp.
-  ValueRange tilingResults;
+  SmallVector<Value> tilingResults;
   ArrayRef<BlockArgument> destBbArgs = forallOp.getOutputBlockArguments();
   {
     // 4.a. RAII guard, inserting within forallOp, before terminator.
@@ -729,12 +730,13 @@ FailureOr<linalg::ForallReductionTilingResult> linalg::tileReductionUsingForall(
 
     // 5. Tile the cloned op and delete the clone.
     if (tileSizes.empty()) {
-      SmallVector<Operation *> tiledOps =
+      FailureOr<TilingResult> tilingResult =
           cast<TilingInterface>(clonedOp).getTiledImplementation(
               b, tiledOffsets, tiledSizes);
-      assert(tiledOps.size() == 1 && "expected a single produced tiled op");
-      tiledOp = tiledOps.front();
-      tilingResults = tiledOp->getResults();
+      assert(tilingResult->tiledOps.size() == 1 &&
+             "expected a single produced tiled op");
+      tiledOp = tilingResult->tiledOps.front();
+      tilingResults = tilingResult->tiledValues;
     } else {
       LinalgTilingOptions options;
       FailureOr<TiledLinalgOp> maybeTiled = tileLinalgOpImpl<scf::ForOp>(
diff --git a/mlir/lib/Dialect/Linalg/Transforms/TilingInterfaceImpl.cpp b/mlir/lib/Dialect/Linalg/Transforms/TilingInterfaceImpl.cpp
index 4e6777ecfcb10..676d6330cde3e 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/TilingInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/TilingInterfaceImpl.cpp
@@ -111,7 +111,7 @@ struct LinalgOpTilingInterface
   }
 
   // Instantiate the tiled implementation of the operation.
-  SmallVector<Operation *>
+  FailureOr<TilingResult>
   getTiledImplementation(Operation *op, OpBuilder &b,
                          ArrayRef<OpFoldResult> offsets,
                          ArrayRef<OpFoldResult> sizes) const {
@@ -129,7 +129,7 @@ struct LinalgOpTilingInterface
     Operation *tiledOp = clone(b, linalgOp, resultTensorTypes, tiledOperands);
     offsetIndices(b, cast<LinalgOp>(tiledOp), offsets);
 
-    return {tiledOp};
+    return TilingResult{{tiledOp}, SmallVector<Value>(tiledOp->getResults())};
   }
 
   // Return the details of the output tile generated by the tiled
@@ -160,10 +160,10 @@ struct LinalgOpTilingInterface
     return success();
   }
 
-  FailureOr<Value> generateResultTileValue(Operation *op, OpBuilder &b,
-                                           unsigned resultNumber,
-                                           ArrayRef<OpFoldResult> offsets,
-                                           ArrayRef<OpFoldResult> sizes) const {
+  FailureOr<TilingResult>
+  generateResultTileValue(Operation *op, OpBuilder &b, unsigned resultNumber,
+                          ArrayRef<OpFoldResult> offsets,
+                          ArrayRef<OpFoldResult> sizes) const {
     auto linalgOp = cast<LinalgOp>(op);
 
     // Check that the indexing map used for the output is a projected
@@ -192,17 +192,20 @@ struct LinalgOpTilingInterface
     }
     for (const auto &resultExpr : llvm::enumerate(indexingMap.getResults())) {
       unsigned dimPosition =
-          resultExpr.value().cast<AffineDimExpr>().getPosition();
+          resultExpr.value().template cast<AffineDimExpr>().getPosition();
       iterationTileOffsets[dimPosition] = offsets[resultExpr.index()];
       iterationTileSizes[dimPosition] = sizes[resultExpr.index()];
     }
 
-    SmallVector<Operation *> tiledOp = tilingInterfaceOp.getTiledImplementation(
-        b, iterationTileOffsets, iterationTileSizes);
-    if (tiledOp.size() != 1)
+    FailureOr<TilingResult> tilingResult =
+        tilingInterfaceOp.getTiledImplementation(b, iterationTileOffsets,
+                                                 iterationTileSizes);
+    if (tilingResult->tiledOps.size() != 1)
       return op->emitOpError("failed to generate tiled implementation");
 
-    return tiledOp[0]->getResult(resultNumber);
+    return TilingResult{
+        tilingResult->tiledOps,
+        SmallVector<Value>{tilingResult->tiledValues[resultNumber]}};
   }
 
   LogicalResult generateScalarImplementation(Operation *op, OpBuilder &builder,
@@ -314,7 +317,7 @@ struct LinalgOpPartialReductionInterface
     AffineMap oldOutputMap =
         linalgOp.getMatchingIndexingMap(linalgOp.getDpsInitOperand(0));
     SmallVector<AffineExpr> outputExpr;
-    for (auto &[idx, expr] : llvm::enumerate(oldOutputMap.getResults())) {
+    for (auto [idx, expr] : llvm::enumerate(oldOutputMap.getResults())) {
       if (static_cast<int64_t>(idx) == insertSplitDimension) {
         outputExpr.push_back(b.getAffineDimExpr(reductionDims[0]));
       }
diff --git a/mlir/lib/Dialect/Linalg/Transforms/Transforms.cpp b/mlir/lib/Dialect/Linalg/Transforms/Transforms.cpp
index 01f2c17c39cda..e001f59b21e93 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/Transforms.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/Transforms.cpp
@@ -205,8 +205,7 @@ linalg::rewriteAsPaddedOp(RewriterBase &rewriter, LinalgOp opToPad,
   }
 
   ReifiedRankedShapedTypeDims reifiedResultShapes;
-  if (failed(cast<ReifyRankedShapedTypeOpInterface>(opToPad.getOperation())
-                 .reifyResultShapes(rewriter, reifiedResultShapes))) {
+  if (failed(reifyResultShapes(rewriter, opToPad, reifiedResultShapes))) {
     LLVM_DEBUG(DBGS() << "--failed to reify result shapes -> FAIL\n");
     return rewriter.notifyMatchFailure(opToPad,
                                        "failed to reify result shapes");
@@ -858,7 +857,7 @@ PadOpTransformationPattern::matchAndRewrite(tensor::PadOp padOp,
 /// Filling `dest` using FillOp constant padding value if possible.
 /// Otherwise, generate a tensor::GenerateOp.
 Value GeneralizePadOpPattern::createFillOrGenerateOp(
-    PatternRewriter &rewriter, tensor::PadOp padOp, Value dest,
+    RewriterBase &rewriter, tensor::PadOp padOp, Value dest,
     const SmallVector<Value> &dynSizes) const {
   auto padValue = padOp.getConstantPaddingValue();
   if (padValue)
@@ -953,12 +952,14 @@ LogicalResult ExtractSliceOfPadTensorSwapPattern::matchAndRewrite(
       return failure();
   }
 
-  Operation *tiledPadOp =
+  FailureOr<TilingResult> tilingResult =
       tensor::bubbleUpPadSlice(rewriter, padOp, sliceOp.getMixedOffsets(),
                                sliceOp.getMixedSizes(), zeroSliceGuard);
+  if (failed(tilingResult))
+    return failure();
   // All shapes are static and the data source is actually used. Rewrite into
   // pad(extract_slice(x)).
-  rewriter.replaceOp(sliceOp, tiledPadOp->getResults());
+  rewriter.replaceOp(sliceOp, tilingResult->tiledValues);
   return success();
 }
 
@@ -1361,14 +1362,71 @@ DownscaleDepthwiseConv2DNhwcHwcOp::returningMatchAndRewrite(
   return conv1DOp;
 }
 
+FailureOr<Conv1DOp>
+DownscaleConv2DOp::returningMatchAndRewrite(Conv2DOp convOp,
+                                            PatternRewriter &rewriter) const {
+  if (convOp.hasBufferSemantics())
+    return failure(); // To be implemented.
+
+  Value input = convOp.getInputs().front();
+  Value kernel = convOp.getInputs().back();
+  Value output = convOp.getOutputs().front();
+
+  auto inputType = input.getType().dyn_cast<RankedTensorType>();
+  auto kernelType = kernel.getType().dyn_cast<RankedTensorType>();
+  auto outputType = output.getType().dyn_cast<RankedTensorType>();
+
+  auto kernelShape = kernelType.getShape();
+  auto outputShape = outputType.getShape();
+
+  // Only handle the case where at least one of the window dimensions is
+  // of size 1. Other cases can rely on tiling to reduce to such cases.
+  int64_t khSize = kernelShape[0], kwSize = kernelShape[1];
+  int64_t ohSize = outputShape[0], owSize = outputShape[1];
+  bool removeH = (khSize == 1 && ohSize == 1);
+  bool removeW = (kwSize == 1 && owSize == 1);
+  if (!removeH && !removeW)
+    return failure();
+
+  // Get new shapes and types for all operands by removing the size-1
+  // dimension.
+  using RTTBuilder = RankedTensorType::Builder;
+  RankedTensorType newInputType =
+      RTTBuilder(inputType).dropDim((removeH ? 0 : 1));
+  RankedTensorType newKernelType =
+      RTTBuilder(kernelType).dropDim((removeH ? 0 : 1));
+  RankedTensorType newOutputType =
+      RTTBuilder(outputType).dropDim(removeH ? 0 : 1);
+
+  // Rank-reduce operands.
+  Location loc = convOp.getLoc();
+  Value newInput = tensor::createCanonicalRankReducingExtractSliceOp(
+      rewriter, loc, input, newInputType);
+  Value newKernel = tensor::createCanonicalRankReducingExtractSliceOp(
+      rewriter, loc, kernel, newKernelType);
+  Value newOutput = tensor::createCanonicalRankReducingExtractSliceOp(
+      rewriter, loc, output, newOutputType);
+
+  auto conv1DOp = rewriter.create<Conv1DOp>(loc, newOutputType,
+                                            ValueRange{newInput, newKernel},
+                                            ValueRange{newOutput});
+
+  // Insert back.
+  Value inserted = tensor::createCanonicalRankReducingInsertSliceOp(
+      rewriter, loc, conv1DOp.getResult(0), output);
+  rewriter.replaceOp(convOp, inserted);
+
+  return conv1DOp;
+}
+
 void linalg::populateDecomposeConvolutionPatterns(RewritePatternSet &patterns,
                                                   PatternBenefit benefit) {
   patterns.add<DownscaleSizeOneWindowed2DConvolution<linalg::Conv2DNhwcHwcfOp,
                                                      Conv1DNwcWcfOp>,
                DownscaleSizeOneWindowed2DConvolution<linalg::Conv2DNchwFchwOp,
                                                      Conv1DNcwFcwOp>,
-               DownscaleDepthwiseConv2DNhwcHwcOp>(patterns.getContext(),
-                                                  benefit);
+               DownscaleDepthwiseConv2DNhwcHwcOp, DownscaleConv2DOp>(
+      patterns.getContext(), benefit);
   patterns.add<
       DownscaleSizeOneWindowed2DConvolution<PoolingNhwcSumOp, PoolingNwcSumOp>,
       DownscaleSizeOneWindowed2DConvolution<PoolingNchwSumOp, PoolingNcwSumOp>,
diff --git a/mlir/lib/Dialect/Linalg/Transforms/Vectorization.cpp b/mlir/lib/Dialect/Linalg/Transforms/Vectorization.cpp
index 29cabd42d3da9..6b27b412f183e 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/Vectorization.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/Vectorization.cpp
@@ -31,8 +31,8 @@
 #include "llvm/ADT/TypeSwitch.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/raw_ostream.h"
-#include <type_traits>
 #include <optional>
+#include <type_traits>
 
 using namespace mlir;
 using namespace mlir::linalg;
@@ -62,6 +62,112 @@ static OpType getSingleOpOfType(Block &block) {
   return res;
 }
 
+/// Helper function to extract the input slices after filter is unrolled along
+/// kw.
+static SmallVector<Value>
+extractConvInputSlices(RewriterBase &rewriter, Location loc, Value input,
+                       int64_t nSize, int64_t wSize, int64_t cSize,
+                       int64_t kwSize, int strideW, int dilationW,
+                       int64_t wSizeStep, bool isSingleChanneled) {
+  SmallVector<Value> result;
+  if (isSingleChanneled) {
+    // Extract input slice of size {wSizeStep} @ [w + kw] for non-channeled
+    // convolution.
+    SmallVector<int64_t> sizes{wSizeStep};
+    SmallVector<int64_t> strides{1};
+    for (int64_t kw = 0; kw < kwSize; ++kw) {
+      for (int64_t w = 0; w < wSize; w += wSizeStep) {
+        result.push_back(rewriter.create<vector::ExtractStridedSliceOp>(
+            loc, input, /*offsets=*/ArrayRef<int64_t>{w + kw}, sizes, strides));
+      }
+    }
+  } else {
+    // Extract lhs slice of size {n, wSizeStep, c} @ [0, sw * w + dw * kw, 0]
+    // for channeled convolution.
+    SmallVector<int64_t> sizes{nSize, wSizeStep, cSize};
+    SmallVector<int64_t> strides{1, 1, 1};
+    for (int64_t kw = 0; kw < kwSize; ++kw) {
+      for (int64_t w = 0; w < wSize; w += wSizeStep) {
+        result.push_back(rewriter.create<vector::ExtractStridedSliceOp>(
+            loc, input,
+            /*offsets=*/ArrayRef<int64_t>{0, w * strideW + kw * dilationW, 0},
+            sizes, strides));
+      }
+    }
+  }
+  return result;
+}
+
+/// Helper function to extract the filter slices after filter is unrolled along
+/// kw.
+static SmallVector<Value> extractConvFilterSlices(RewriterBase &rewriter,
+                                                  Location loc, Value filter,
+                                                  int64_t kwSize) {
+  SmallVector<Value> result;
+  // Extract rhs slice of size [{c, f} for channeled convolutions and {1} for
+  // non-chanelled convolution] @ [kw].
+  for (int64_t kw = 0; kw < kwSize; ++kw) {
+    result.push_back(rewriter.create<vector::ExtractOp>(
+        loc, filter, /*offsets=*/ArrayRef<int64_t>{kw}));
+  }
+  return result;
+}
+
+/// Helper function to extract the result slices after filter is unrolled along
+/// kw.
+static SmallVector<Value>
+extractConvResultSlices(RewriterBase &rewriter, Location loc, Value res,
+                        int64_t nSize, int64_t wSize, int64_t fSize,
+                        int64_t wSizeStep, bool isSingleChanneled) {
+  SmallVector<Value> result;
+  if (isSingleChanneled) {
+    // Extract res slice: {wSizeStep} @ [w] for non-channeled convolution.
+    SmallVector<int64_t> sizes{wSizeStep};
+    SmallVector<int64_t> strides{1};
+    for (int64_t w = 0; w < wSize; w += wSizeStep) {
+      result.push_back(rewriter.create<vector::ExtractStridedSliceOp>(
+          loc, res, /*offsets=*/ArrayRef<int64_t>{w}, sizes, strides));
+    }
+  } else {
+    // Extract res slice: {n, wSizeStep, f} @ [0, w, 0] for channeled
+    // convolution.
+    SmallVector<int64_t> sizes{nSize, wSizeStep, fSize};
+    SmallVector<int64_t> strides{1, 1, 1};
+    for (int64_t w = 0; w < wSize; w += wSizeStep) {
+      result.push_back(rewriter.create<vector::ExtractStridedSliceOp>(
+          loc, res, /*offsets=*/ArrayRef<int64_t>{0, w, 0}, sizes, strides));
+    }
+  }
+  return result;
+}
+
+/// Helper function to insert the computed result slices.
+static Value insertConvResultSlices(RewriterBase &rewriter, Location loc,
+                                    Value res, int64_t wSize, int64_t wSizeStep,
+                                    SmallVectorImpl<Value> &resVals,
+                                    bool isSingleChanneled) {
+
+  if (isSingleChanneled) {
+    // Write back res slice: {wSizeStep} @ [w] for non-channeled convolution.
+    // This does not depend on kw.
+    SmallVector<int64_t> strides{1};
+    for (int64_t w = 0; w < wSize; w += wSizeStep) {
+      res = rewriter.create<vector::InsertStridedSliceOp>(
+          loc, resVals[w], res, /*offsets=*/ArrayRef<int64_t>{w}, strides);
+    }
+  } else {
+    // Write back res slice: {n, wSizeStep, f} @ [0, w, 0] for channeled
+    // convolution. This does not depend on kw.
+    SmallVector<int64_t> strides{1, 1, 1};
+    for (int64_t w = 0; w < wSize; w += wSizeStep) {
+      res = rewriter.create<vector::InsertStridedSliceOp>(
+          loc, resVals[w], res, /*offsets=*/ArrayRef<int64_t>{0, w, 0},
+          strides);
+    }
+  }
+  return res;
+}
+
 /// Contains the vectorization state and related methods used across the
 /// vectorization process of a given operation.
 struct VectorizationState {
@@ -334,6 +440,7 @@ static AffineMap reindexIndexingMap(AffineMap map) {
 
 /// Helper enum to represent conv1d input traversal order.
 enum class Conv1DOpOrder {
+  W,   // Corresponds to non-channeled 1D convolution operation.
   Ncw, // Corresponds to operation that traverses the input in (n, c, w) order.
   Nwc  // Corresponds to operation that traverses the input in (n, w, c) order.
 };
@@ -636,81 +743,112 @@ enum VectorMemoryAccessKind {
   Gather
 };
 
-/// Check whether /p val can be used for calculating an index for a contiguous
-/// load operation. This means that /p val should either:
-///   * be invariant with respect to /p linalgOp, or
-///   * increment by 1 with every loop iterator (when /p shouldBeConstant is
-///     false).
-/// Parameters /p trailingLoopDim and /p shouldBeConstant are used to analyze
-/// `linalg.index` ops.
-static bool isContiguousLoadIdx(LinalgOp &linalgOp, Value &val,
-                                size_t trailingLoopDim, bool shouldBeConstant) {
-  auto *block = linalgOp.getBlock();
+/// Checks whether /p val can be used for calculating a loop invariant index.
+static bool isLoopInvariantIdx(LinalgOp &linalgOp, Value &val) {
 
-  // Bail out if this is a block argument for this linalg.generic Op.
+  auto targetShape = linalgOp.getStaticLoopRanges();
+  assert(((llvm::count_if(targetShape,
+                          [](int64_t dimSize) { return dimSize > 1; }) == 1)) &&
+         "n-D vectors are not yet supported");
+  assert(targetShape.back() != 1 &&
+         "1-D vectors with the trailing dim eqaual 1 are not yet supported");
+
+  // Blocks outside _this_ linalg.generic are effectively loop invariant.
+  // However, analysing block arguments for _this_ linalg.generic Op is a bit
+  // tricky. Just bail out in the latter case.
   // TODO: We could try analysing the corresponding affine map here.
-  if (val.dyn_cast<BlockArgument>())
+  auto *block = linalgOp.getBlock();
+  if (isa<BlockArgument>(val))
     return llvm::all_of(block->getArguments(),
                         [&val](Value v) { return (v != val); });
 
   Operation *defOp = val.getDefiningOp();
   assert(defOp && "This is neither a block argument nor an operation result");
 
-  // We know that we are reading into a 1-D tensor like this:
-  // `tensor<1x1x4xi32`. Given this assumption, the following Op:
-  //    * `%idx = `linalg.index dim : index`,
-  // will either:
-  //    1. produce a constant when `dim` _is not_ the trailing loop dim, or
-  //    2. increment with stride one when `dim` _is_ the trailing loop dim.
+  // IndexOp is loop invariant as long as its result remains constant across
+  // iterations. Given the assumptions on the loop ranges above, only the
+  // trailing loop dim ever changes.
+  auto trailingLoopDim = linalgOp.getStaticLoopRanges().size() - 1;
   if (auto indexOp = dyn_cast<linalg::IndexOp>(defOp))
-    return shouldBeConstant ? (indexOp.getDim() != trailingLoopDim)
-                            : (indexOp.getDim() == trailingLoopDim);
+    return (indexOp.getDim() != trailingLoopDim);
 
   auto *ancestor = block->findAncestorOpInBlock(*defOp);
 
-  // Values define outside `linalgOp`.
+  // Values define outside `linalgOp` are loop invariant.
   if (!ancestor)
     return true;
 
-  // Values defined inside `linalgOp`, which are constant.
-  if (dyn_cast<arith::ConstantOp>(ancestor))
+  // Values defined inside `linalgOp`, which are constant, are loop invariant.
+  if (isa<arith::ConstantOp>(ancestor))
     return true;
 
-  // Conservatively reject Ops that could lead to non-contiguous accesses.
-  if (!isa<arith::AddIOp, arith::SubIOp, linalg::IndexOp>(ancestor))
-    return false;
-
   bool result = true;
   for (auto op : ancestor->getOperands())
-    result &=
-        isContiguousLoadIdx(linalgOp, op, trailingLoopDim, shouldBeConstant);
+    result &= isLoopInvariantIdx(linalgOp, op);
 
   return result;
 }
 
-/// Check whether the calculation of \p val is based on linalg.index Op with
-/// the dim attribute matching \p dim.
-static bool isBasedOnIndexOp(LinalgOp &linalgOp, Value &val, size_t dim) {
-  auto *block = linalgOp.getBlock();
-  auto targetShape = linalgOp.getStaticLoopRanges();
+/// Check whether \p val could be used for calculating the trailing index for a
+/// contiguous load operation.
+///
+/// There are currently 3 types of values that are allowed here:
+///   1. loop-invariant values,
+///   2. values that increment by 1 with every loop iteration,
+///   3. results of basic arithmetic operations (linear and continuous)
+///      involving 1., 2. and 3.
+/// This method returns True if indeed only such values are used in calculating
+/// \p val.
+///
+/// Additionally, the trailing index for a contiguous load operation should
+/// increment by 1 with every loop iteration, i.e. be based on:
+///   * `linalg.index <dim>` ,
+/// where <dim> is the trailing dim of the iteration space. \p foundIndexOp is
+/// updated to `true` when such an op is found.
+static bool isContiguousLoadIdx(LinalgOp &linalgOp, Value &val,
+                                bool &foundIndexOp) {
 
-  if (val.isa<BlockArgument>())
-    return false;
+  auto targetShape = linalgOp.getStaticLoopRanges();
+  assert(((llvm::count_if(targetShape,
+                          [](int64_t dimSize) { return dimSize > 1; }) == 1)) &&
+         "n-D vectors are not yet supported");
+  assert(targetShape.back() != 1 &&
+         "1-D vectors with the trailing dim 1 are not yet supported");
+
+  // Blocks outside _this_ linalg.generic are effectively loop invariant.
+  // However, analysing block arguments for _this_ linalg.generic Op is a bit
+  // tricky. Just bail out in the latter case.
+  // TODO: We could try analysing the corresponding affine map here.
+  auto *block = linalgOp.getBlock();
+  if (isa<BlockArgument>(val))
+    return llvm::all_of(block->getArguments(),
+                        [&val](Value v) { return (v != val); });
 
   Operation *defOp = val.getDefiningOp();
   assert(defOp && "This is neither a block argument nor an operation result");
 
-  if (auto indexOp = dyn_cast<linalg::IndexOp>(defOp))
-    return (indexOp.getDim() == dim);
+  // Given the assumption on the loop ranges above, only the trailing loop
+  // index is not constant.
+  auto trailingLoopDim = linalgOp.getStaticLoopRanges().size() - 1;
+  if (auto indexOp = dyn_cast<linalg::IndexOp>(defOp)) {
+    foundIndexOp = (indexOp.getDim() == trailingLoopDim);
+    return true;
+  }
 
   auto *ancestor = block->findAncestorOpInBlock(*defOp);
 
   if (!ancestor)
     return false;
 
+  // Conservatively reject Ops that could lead to indices with stride other
+  // than 1.
+  if (!isa<arith::AddIOp, arith::SubIOp, arith::ConstantOp, linalg::IndexOp>(
+          ancestor))
+    return false;
+
   bool result = false;
   for (auto op : ancestor->getOperands())
-    result |= isBasedOnIndexOp(linalgOp, op, dim);
+    result |= isContiguousLoadIdx(linalgOp, op, foundIndexOp);
 
   return result;
 }
@@ -725,7 +863,7 @@ getTensorExtractMemoryAccessPattern(tensor::ExtractOp extractOp,
 
   auto targetShape = linalgOp.getStaticLoopRanges();
 
-  // Assume that it's a gather load when reading _into_:
+  // 1. Assume that it's a gather load when reading _into_:
   //    * an n-D vector, like`tensor<1x2x4xi32` or`tensor<2x1x4xi32>`, or
   //    * a 1-D vector with the trailing dim equal 1, e.g. `tensor<1x4x1xi32`.
   // TODO: Relax these conditions.
@@ -736,44 +874,36 @@ getTensorExtractMemoryAccessPattern(tensor::ExtractOp extractOp,
 
   auto inputShape = extractOp.getTensor().getType().cast<ShapedType>();
 
-  // Assume that it's a gather load when reading _from_ a tensor for which the
-  // trailing dimension is 1, e.g. `tensor<1x4x1xi32>`.
+  // 2. Assume that it's a gather load when reading _from_ a tensor for which
+  // the trailing dimension is 1, e.g. `tensor<1x4x1xi32>`.
   // TODO: Relax this condition.
   if (inputShape.getShape().back() == 1)
     return VectorMemoryAccessKind::Gather;
 
-  // The trailing loop dim is needed when analyzing ops like:
-  //     * %idx = `linalg.index <dim> : index`.
-  auto trailingLoopDim = targetShape.size() - 1;
-
   bool isContiguous = true;
 
-  // Iterate over all indices. Analyze the way each index is calculated and
-  // decide whether it is suitable for a contiguous load (e.g. loop invariant).
+  // 3a. Analyze the leading indices of `extractOp`.
+  // Look at the way each index is calculated and decide whether it is suitable
+  // for a contiguous load, i.e. whether it's loop invariant.
   auto indices = extractOp.getIndices();
-  for (auto [i, indexVal] : llvm::enumerate(indices)) {
-    if (inputShape.getShape()[i] == 1) {
-      // This index will always be equal 0, so it is a loop-invariant constant.
-      continue;
-    }
+  auto leadIndices = ValueRange(indices.drop_back(1));
 
-    // Should this index be loop invariant?
-    //  * _no_ if this is the trailing index,
-    //  * _yes_ otherwise.
-    auto extractOpBottomIdx = indices.size() - 1;
-    bool loopInvariantIndex = (i != extractOpBottomIdx);
+  for (auto [i, indexVal] : llvm::enumerate(leadIndices)) {
+    if (inputShape.getShape()[i] == 1)
+      continue;
 
-    isContiguous &= isContiguousLoadIdx(linalgOp, indexVal, trailingLoopDim,
-                                        loopInvariantIndex);
+    isContiguous &= isLoopInvariantIdx(linalgOp, indexVal);
   }
 
-  // The trailing index in the extract Op must increment with every iteration,
-  // which means that it must be based on a loop index. Given the assumption
-  // on the output tensor, only the trailing loop index is not constant, so
-  // that's what we need to check against.
+  // 3b. Analyze the trailing index for `extractOp`.
   auto extractOpTrailingIdx = indices.back();
+  // For contiguous loads, the trailing `extractOp` index should increment with
+  // every loop iteration. This effectively means that it must be based on the
+  // trailing loop index. This is what the following bool captures.
+  bool foundIndexOp = false;
   isContiguous &=
-      isBasedOnIndexOp(linalgOp, extractOpTrailingIdx, trailingLoopDim);
+      isContiguousLoadIdx(linalgOp, extractOpTrailingIdx, foundIndexOp);
+  isContiguous &= foundIndexOp;
 
   if (isContiguous) {
     LDBG("Found contigous load: " << extractOp);
@@ -1405,7 +1535,7 @@ struct GenericPadOpVectorizationPattern : public GeneralizePadOpPattern {
   /// Vectorize the copying of a tensor::PadOp's source. This is possible if
   /// each dimension size is statically know in the source type or the result
   /// type (or both).
-  static LogicalResult tryVectorizeCopy(PatternRewriter &rewriter,
+  static LogicalResult tryVectorizeCopy(RewriterBase &rewriter,
                                         tensor::PadOp padOp, Value dest) {
     auto sourceType = padOp.getSourceType();
     auto resultType = padOp.getResultType();
@@ -2032,6 +2162,15 @@ bool isSupportedPoolKind(vector::CombiningKind kind) {
 
 /// Generate a vector implementation for either:
 /// ```
+///   Op def: (     w,     kw  )
+///    Iters: ({Par(), Red()})
+///   Layout: {{w + kw}, {kw}, {w}}
+/// ```
+/// kw is unrolled.
+///
+/// or
+///
+/// ```
 ///   Op def: (     n,     w,     c,    kw,    f  )
 ///    Iters: ({Par(), Par(), Par(), Red(), Red()})
 ///   Layout: {{n, strideW * w + dilationW * kw, c}, {kw, c, f}, {n, w, f}}
@@ -2072,8 +2211,10 @@ struct Conv1DGenerator
     resShapedType = resShaped.getType().dyn_cast<ShapedType>();
     if (!lhsShapedType || !rhsShapedType || !resShapedType)
       return;
-    // LHS has dimension NCW/NWC and RES has dimension NFW/NCW/NWF/NWC.
-    if (lhsShapedType.getRank() != 3 || resShapedType.getRank() != 3)
+    // (LHS has dimension NCW/NWC and RES has dimension NFW/NCW/NWF/NWC) OR
+    // (non-channeled convolution -> LHS and RHS both have single dimensions).
+    if (!((lhsShapedType.getRank() == 3 && resShapedType.getRank() == 3) ||
+          (lhsShapedType.getRank() == 1 && resShapedType.getRank() == 1)))
       return;
 
     Operation *reduceOp = matchLinalgReduction(linalgOp.getDpsInitOperand(0));
@@ -2092,7 +2233,7 @@ struct Conv1DGenerator
     auto rhsRank = rhsShapedType.getRank();
     switch (oper) {
     case Conv:
-      if (rhsRank != 2 && rhsRank!= 3)
+      if (rhsRank != 1 && rhsRank != 2 && rhsRank != 3)
         return;
       break;
     case Pool:
@@ -2106,6 +2247,15 @@ struct Conv1DGenerator
 
   /// Generate a vector implementation for:
   /// ```
+  ///   Op def: (     w,     kw  )
+  ///    Iters: ({Par(), Red()})
+  ///   Layout: {{w + kw}, {kw}, {w}}
+  /// ```
+  /// kw is always unrolled.
+  ///
+  /// or
+  ///
+  /// ```
   ///   Op def: (     n,     w,     c,    kw,    f  )
   ///    Iters: ({Par(), Par(), Par(), Red(), Red()})
   ///   Layout: {{n, strideW * w + dilationW * kw, c}, {kw, c, f}, {n, w, f}}
@@ -2119,7 +2269,21 @@ struct Conv1DGenerator
 
     int64_t nSize, wSize, cSize, kwSize, fSize;
     SmallVector<int64_t, 3> lhsShape, rhsShape, resShape;
+    bool isSingleChanneled = (conv1DOpOrder == Conv1DOpOrder::W);
     switch (conv1DOpOrder) {
+    case Conv1DOpOrder::W:
+      // Initialize unused dimensions
+      nSize = fSize = cSize = 0;
+      // out{W}
+      bindShapeDims(resShapedType, wSize);
+      // kernel{kw}
+      bindShapeDims(rhsShapedType, kwSize);
+      lhsShape = {// iw = ow + kw - 1
+                  //   (i.e. 16 convolved with 3 -> 14)
+                  (wSize + kwSize - 1)};
+      rhsShape = {kwSize};
+      resShape = {wSize};
+      break;
     case Conv1DOpOrder::Nwc:
       // out{n, w, f}
       bindShapeDims(resShapedType, nSize, wSize, fSize);
@@ -2197,24 +2361,27 @@ struct Conv1DGenerator
     auto lhsType = VectorType::get(lhsShape, lhsEltType);
     auto rhsType = VectorType::get(rhsShape, rhsEltType);
     auto resType = VectorType::get(resShape, resEltType);
-    // Read lhs slice of size {w * strideW + kw * dilationW, c, f} @ [0, 0,
-    // 0].
-    Value lhs = rewriter.create<vector::TransferReadOp>(
-        loc, lhsType, lhsShaped, ValueRange{zero, zero, zero});
-    // Read rhs slice of size {kw, c, f} @ [0, 0, 0].
+    // Zero padding with the corresponding dimensions for lhs, rhs and res.
+    SmallVector<Value> lhsPadding(lhsShape.size(), zero);
+    SmallVector<Value> rhsPadding(rhsShape.size(), zero);
+    SmallVector<Value> resPadding(resShape.size(), zero);
+
+    // Read the whole lhs, rhs and res in one shot (with zero padding).
+    Value lhs = rewriter.create<vector::TransferReadOp>(loc, lhsType, lhsShaped,
+                                                        lhsPadding);
     // This is needed only for Conv.
     Value rhs = nullptr;
     if (oper == Conv)
-      rhs = rewriter.create<vector::TransferReadOp>(
-          loc, rhsType, rhsShaped, ValueRange{zero, zero, zero});
-    // Read res slice of size {n, w, f} @ [0, 0, 0].
-    Value res = rewriter.create<vector::TransferReadOp>(
-        loc, resType, resShaped, ValueRange{zero, zero, zero});
-
-    // The base vectorization case is input: {n,w,c}, weight: {kw,c,f}, output:
-    // {n,w,f}. To reuse the base pattern vectorization case, we do pre
-    // transpose on input, weight, and output.
+      rhs = rewriter.create<vector::TransferReadOp>(loc, rhsType, rhsShaped,
+                                                    rhsPadding);
+    Value res = rewriter.create<vector::TransferReadOp>(loc, resType, resShaped,
+                                                        resPadding);
+
+    // The base vectorization case for channeled convolution is input: {n,w,c},
+    // weight: {kw,c,f}, output: {n,w,f}. To reuse the base pattern
+    // vectorization case, we do pre transpose on input, weight, and output.
     switch (conv1DOpOrder) {
+    case Conv1DOpOrder::W:
     case Conv1DOpOrder::Nwc:
       // Base case, so no transposes necessary.
       break;
@@ -2241,45 +2408,35 @@ struct Conv1DGenerator
     //===------------------------------------------------------------------===//
     // Unroll along kw and read slices of lhs and rhs.
     SmallVector<Value> lhsVals, rhsVals, resVals;
-    // Extract lhs slice of size {n, wSizeStep, c} @ [0, sw * w + dw * kw, 0].
-    for (int64_t kw = 0; kw < kwSize; ++kw) {
-      for (int64_t w = 0; w < wSize; w += wSizeStep) {
-        lhsVals.push_back(rewriter.create<vector::ExtractStridedSliceOp>(
-            loc, lhs,
-            /*offsets=*/ArrayRef<int64_t>{0, w * strideW + kw * dilationW, 0},
-            /*sizes=*/ArrayRef<int64_t>{nSize, wSizeStep, cSize},
-            /*strides=*/ArrayRef<int64_t>{1, 1, 1}));
-      }
-    }
-    // Extract rhs slice of size {c, f} @ [kw].
+    lhsVals = extractConvInputSlices(rewriter, loc, lhs, nSize, wSize, cSize,
+                                     kwSize, strideW, dilationW, wSizeStep,
+                                     isSingleChanneled);
     // Do not do for pooling.
     if (oper == Conv)
-      for (int64_t kw = 0; kw < kwSize; ++kw) {
-        rhsVals.push_back(rewriter.create<vector::ExtractOp>(
-            loc, rhs, /*offsets=*/ArrayRef<int64_t>{kw}));
-      }
-    // Extract res slice: {n, wSizeStep, f} @ [0, w, 0].
-    for (int64_t w = 0; w < wSize; w += wSizeStep) {
-      resVals.push_back(rewriter.create<vector::ExtractStridedSliceOp>(
-          loc, res,
-          /*offsets=*/ArrayRef<int64_t>{0, w, 0},
-          /*sizes=*/ArrayRef<int64_t>{nSize, wSizeStep, fSize},
-          /*strides=*/ArrayRef<int64_t>{1, 1, 1}));
-    }
+      rhsVals = extractConvFilterSlices(rewriter, loc, rhs, kwSize);
+    resVals = extractConvResultSlices(rewriter, loc, res, nSize, wSize, fSize,
+                                      wSizeStep, isSingleChanneled);
 
     auto linearIndex = [&](int64_t kw, int64_t w) {
       return kw * (wSize / wSizeStep) + w;
     };
 
     // Compute contraction: O{n, w, f} += I{n, sw * w + dw * kw, c} * F{c, f} or
+    // perform outerproduct for non-channeled convolution or
     // perform simple arith operation for pooling
     for (int64_t kw = 0; kw < kwSize; ++kw) {
       for (int64_t w = 0; w < wSize; w += wSizeStep) {
         switch (oper) {
         case Conv:
-          resVals[w] = conv1dSliceAsContraction(rewriter, loc,
-                                                lhsVals[linearIndex(kw, w)],
-                                                rhsVals[kw], resVals[w]);
+          if (isSingleChanneled) {
+            resVals[w] = conv1dSliceAsOuterProduct(rewriter, loc,
+                                                   lhsVals[linearIndex(kw, w)],
+                                                   rhsVals[kw], resVals[w]);
+          } else {
+            resVals[w] = conv1dSliceAsContraction(rewriter, loc,
+                                                  lhsVals[linearIndex(kw, w)],
+                                                  rhsVals[kw], resVals[w]);
+          }
           break;
         case Pool:
           resVals[w] = pool1dSlice(rewriter, loc, lhsVals[linearIndex(kw, w)],
@@ -2289,22 +2446,17 @@ struct Conv1DGenerator
       }
     }
 
-    // Write back res slice: {n, wSizeStep, f} @ [0, w, 0].
-    // This does not depend on kw.
-    for (int64_t w = 0; w < wSize; w += wSizeStep) {
-      res = rewriter.create<vector::InsertStridedSliceOp>(
-          loc, resVals[w], res,
-          /*offsets=*/ArrayRef<int64_t>{0, w, 0},
-          /*strides=*/ArrayRef<int64_t>{1, 1, 1});
-    }
+    res = insertConvResultSlices(rewriter, loc, res, wSize, wSizeStep, resVals,
+                                 isSingleChanneled);
     //===------------------------------------------------------------------===//
     // End vector-only rewrite part
     //===------------------------------------------------------------------===//
 
-    // The base vectorization case is output: {n,w,f}
+    // The base vectorization case for channeled convolution is output: {n,w,f}
     // To reuse the result from base pattern vectorization case, we post
     // transpose the base case result.
     switch (conv1DOpOrder) {
+    case Conv1DOpOrder::W:
     case Conv1DOpOrder::Nwc:
       // Base case, so no transposes necessary.
       break;
@@ -2316,10 +2468,8 @@ struct Conv1DGenerator
     }
     }
 
-    // Write back res slice of size {n, w, f} @ [0, 0, 0].
     return rewriter
-        .create<vector::TransferWriteOp>(loc, res, resShaped,
-                                         ValueRange{zero, zero, zero})
+        .create<vector::TransferWriteOp>(loc, res, resShaped, resPadding)
         .getOperation();
   }
 
@@ -2336,6 +2486,14 @@ struct Conv1DGenerator
         /*iteratorTypes=*/ArrayRef<vector::IteratorType>{par, par, par, red});
   }
 
+  // Create an outerproduct: lhs{w} * rhs{1} -> res{w} for single channel
+  // convolution.
+  Value conv1dSliceAsOuterProduct(RewriterBase &rewriter, Location loc,
+                                  Value lhs, Value rhs, Value res) {
+    return rewriter.create<vector::OuterProductOp>(
+        loc, res.getType(), lhs, rhs, res, vector::CombiningKind::ADD);
+  }
+
   // Create a reduction: lhs{n, w, c} -> res{n, w, c}
   Value pool1dSlice(RewriterBase &rewriter, Location loc, Value lhs,
                     Value res) {
@@ -2434,15 +2592,17 @@ struct Conv1DGenerator
     // Compute contraction: O{n, w, c} += I{n, sw * w + dw * kw, c} * F{c}
     for (int64_t kw = 0; kw < kwSize; ++kw) {
       for (int64_t w = 0; w < wSize; w += wSizeStep) {
-        resVals[w] = depthwiseConv1dSliceAsMulAcc(
-            rewriter, loc, lhsVals[linearIndex(kw, w)], rhsVals[kw], resVals[w]);
+        resVals[w] = depthwiseConv1dSliceAsMulAcc(rewriter, loc,
+                                                  lhsVals[linearIndex(kw, w)],
+                                                  rhsVals[kw], resVals[w]);
       }
     }
 
     // Its possible we failed to create the Fma.
     if (!llvm::all_of(resVals, [](Value v) { return v; })) {
       // Manually revert (in reverse order) to avoid leaving a bad IR state.
-      for (auto &collection : {resVals, rhsVals, lhsVals, {res, rhs, lhs, zero}})
+      for (auto &collection :
+           {resVals, rhsVals, lhsVals, {res, rhs, lhs, zero}})
         for (Value v : collection)
           rewriter.eraseOp(v.getDefiningOp());
       return rewriter.notifyMatchFailure(op, "failed to create FMA");
@@ -2508,6 +2668,24 @@ struct Conv1DGenerator
     return rewriter.create<arith::AddIOp>(loc, mul, res);
   }
 
+  /// Entry point for non-channeled convolution:
+  ///   {{w + kw}, {kw}, {w}}
+  FailureOr<Operation *> generateNonChanneledConv() {
+    AffineExpr w, kw;
+    bindDims(ctx, w, kw);
+    if (!iters({Par(), Red()}))
+      return rewriter.notifyMatchFailure(op,
+                                         "failed to match conv::W 1-par 1-red");
+
+    // No transposition needed.
+    if (layout({/*lhsIndex*/ {w + kw},
+                /*rhsIndex*/ {kw},
+                /*resIndex*/ {w}}))
+      return conv(Conv1DOpOrder::W);
+
+    return rewriter.notifyMatchFailure(op, "not a conv::W layout");
+  }
+
   /// Entry point that transposes into the common form:
   ///   {{n, strideW * w + dilationW * kw, c}, {kw, c, f}, {n, w, f}}
   FailureOr<Operation *> generateNwcConv() {
@@ -2668,7 +2846,10 @@ static FailureOr<Operation *> vectorizeConvolution(RewriterBase &rewriter,
   auto stride = strides ? *strides.getValues<uint64_t>().begin() : 1;
   auto dilation = dilations ? *dilations.getValues<uint64_t>().begin() : 1;
   Conv1DGenerator e(rewriter, op, stride, dilation);
-  auto res = e.generateNwcConv();
+  auto res = e.generateNonChanneledConv();
+  if (succeeded(res))
+    return res;
+  res = e.generateNwcConv();
   if (succeeded(res))
     return res;
   res = e.generateNcwConv();
diff --git a/mlir/lib/Dialect/Math/Transforms/PolynomialApproximation.cpp b/mlir/lib/Dialect/Math/Transforms/PolynomialApproximation.cpp
index c0f30283ca8e6..6cbb50b0425b3 100644
--- a/mlir/lib/Dialect/Math/Transforms/PolynomialApproximation.cpp
+++ b/mlir/lib/Dialect/Math/Transforms/PolynomialApproximation.cpp
@@ -134,7 +134,7 @@ handleMultidimensionalVectors(ImplicitLocOpBuilder &builder,
   SmallVector<Value> results(maxIndex);
 
   for (int64_t i = 0; i < maxIndex; ++i) {
-    auto offsets = delinearize(strides, i);
+    auto offsets = delinearize(i, strides);
 
     SmallVector<Value> extracted(expandedOperands.size());
     for (const auto &tuple : llvm::enumerate(expandedOperands))
@@ -152,7 +152,7 @@ handleMultidimensionalVectors(ImplicitLocOpBuilder &builder,
 
   for (int64_t i = 0; i < maxIndex; ++i)
     result = builder.create<vector::InsertOp>(results[i], result,
-                                              delinearize(strides, i));
+                                              delinearize(i, strides));
 
   // Reshape back to the original vector shape.
   return builder.create<vector::ShapeCastOp>(
@@ -1212,6 +1212,99 @@ LogicalResult SinAndCosApproximation<isSine, OpTy>::matchAndRewrite(
   return success();
 }
 
+//----------------------------------------------------------------------------//
+// Cbrt approximation.
+//----------------------------------------------------------------------------//
+
+namespace {
+struct CbrtApproximation : public OpRewritePattern<math::CbrtOp> {
+  using OpRewritePattern::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(math::CbrtOp op,
+                                PatternRewriter &rewriter) const final;
+};
+} // namespace
+
+// Estimation of cube-root using an algorithm defined in
+// Hacker's Delight 2nd Edition.
+LogicalResult
+CbrtApproximation::matchAndRewrite(math::CbrtOp op,
+                                   PatternRewriter &rewriter) const {
+  auto operand = op.getOperand();
+  if (!getElementTypeOrSelf(operand).isF32())
+    return rewriter.notifyMatchFailure(op, "unsupported operand type");
+
+  ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+  ArrayRef<int64_t> shape = vectorShape(operand);
+
+  Type floatTy = getElementTypeOrSelf(operand.getType());
+  Type intTy = b.getIntegerType(floatTy.getIntOrFloatBitWidth());
+
+  // Convert to vector types if necessary.
+  floatTy = broadcast(floatTy, shape);
+  intTy = broadcast(intTy, shape);
+
+  auto bconst = [&](Attribute attr) -> Value {
+    Value value = b.create<arith::ConstantOp>(attr);
+    return broadcast(b, value, shape);
+  };
+
+  // Declare the initial values:
+  Value intTwo = bconst(b.getI32IntegerAttr(2));
+  Value intFour = bconst(b.getI32IntegerAttr(4));
+  Value intEight = bconst(b.getI32IntegerAttr(8));
+  Value intMagic = bconst(b.getI32IntegerAttr(0x2a5137a0));
+  Value fpThird = bconst(b.getF32FloatAttr(0.33333333f));
+  Value fpTwo = bconst(b.getF32FloatAttr(2.0f));
+  Value fpZero = bconst(b.getF32FloatAttr(0.0f));
+
+  // Compute an approximation of one third:
+  // union {int ix; float x;};
+  // x = x0;
+  // ix = ix/4 + ix/16;
+  Value absValue = b.create<math::AbsFOp>(operand);
+  Value intValue = b.create<arith::BitcastOp>(intTy, absValue);
+  Value divideBy4 = b.create<arith::ShRSIOp>(intValue, intTwo);
+  Value divideBy16 = b.create<arith::ShRSIOp>(intValue, intFour);
+  intValue = b.create<arith::AddIOp>(divideBy4, divideBy16);
+
+  // ix = ix + ix/16;
+  divideBy16 = b.create<arith::ShRSIOp>(intValue, intFour);
+  intValue = b.create<arith::AddIOp>(intValue, divideBy16);
+
+  // ix = ix + ix/256;
+  Value divideBy256 = b.create<arith::ShRSIOp>(intValue, intEight);
+  intValue = b.create<arith::AddIOp>(intValue, divideBy256);
+
+  // ix = 0x2a5137a0 + ix;
+  intValue = b.create<arith::AddIOp>(intValue, intMagic);
+
+  // Perform one newtons step:
+  // x = 0.33333333f*(2.0f*x + x0/(x*x));
+  Value floatValue = b.create<arith::BitcastOp>(floatTy, intValue);
+  Value squared = b.create<arith::MulFOp>(floatValue, floatValue);
+  Value mulTwo = b.create<arith::MulFOp>(floatValue, fpTwo);
+  Value divSquared = b.create<arith::DivFOp>(absValue, squared);
+  floatValue = b.create<arith::AddFOp>(mulTwo, divSquared);
+  floatValue = b.create<arith::MulFOp>(floatValue, fpThird);
+
+  // x = 0.33333333f*(2.0f*x + x0/(x*x));
+  squared = b.create<arith::MulFOp>(floatValue, floatValue);
+  mulTwo = b.create<arith::MulFOp>(floatValue, fpTwo);
+  divSquared = b.create<arith::DivFOp>(absValue, squared);
+  floatValue = b.create<arith::AddFOp>(mulTwo, divSquared);
+  floatValue = b.create<arith::MulFOp>(floatValue, fpThird);
+
+  // Check for zero and restore sign.
+  Value isZero =
+      b.create<arith::CmpFOp>(arith::CmpFPredicate::OEQ, absValue, fpZero);
+  floatValue = b.create<arith::SelectOp>(isZero, fpZero, floatValue);
+  floatValue = b.create<math::CopySignOp>(floatValue, operand);
+
+  rewriter.replaceOp(op, floatValue);
+  return success();
+}
+
 //----------------------------------------------------------------------------//
 // Rsqrt approximation.
 //----------------------------------------------------------------------------//
@@ -1291,7 +1384,7 @@ void mlir::populateMathPolynomialApproximationPatterns(
   patterns.add<AtanApproximation, Atan2Approximation, TanhApproximation,
                LogApproximation, Log2Approximation, Log1pApproximation,
                ErfPolynomialApproximation, ExpApproximation, ExpM1Approximation,
-               ReuseF32Expansion<math::Atan2Op>,
+               CbrtApproximation, ReuseF32Expansion<math::Atan2Op>,
                SinAndCosApproximation<true, math::SinOp>,
                SinAndCosApproximation<false, math::CosOp>>(
       patterns.getContext());
diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefOps.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefOps.cpp
index 45fac0ce82829..27f1eab09eec6 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefOps.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefOps.cpp
@@ -553,7 +553,7 @@ struct AllocaScopeInliner : public OpRewritePattern<AllocaScopeOp> {
     Block *block = &op.getRegion().front();
     Operation *terminator = block->getTerminator();
     ValueRange results = terminator->getOperands();
-    rewriter.mergeBlockBefore(block, op);
+    rewriter.inlineBlockBefore(block, op);
     rewriter.replaceOp(op, results);
     rewriter.eraseOp(terminator);
     return success();
@@ -1871,15 +1871,13 @@ LogicalResult ReinterpretCastOp::verify() {
            << srcType << " and result memref type " << resultType;
 
   // Match sizes in result memref type and in static_sizes attribute.
-  for (auto &en :
-       llvm::enumerate(llvm::zip(resultType.getShape(), getStaticSizes()))) {
-    int64_t resultSize = std::get<0>(en.value());
-    int64_t expectedSize = std::get<1>(en.value());
+  for (auto [idx, resultSize, expectedSize] :
+       llvm::enumerate(resultType.getShape(), getStaticSizes())) {
     if (!ShapedType::isDynamic(resultSize) &&
         !ShapedType::isDynamic(expectedSize) && resultSize != expectedSize)
       return emitError("expected result type with size = ")
              << expectedSize << " instead of " << resultSize
-             << " in dim = " << en.index();
+             << " in dim = " << idx;
   }
 
   // Match offset and strides in static_offset and static_strides attributes. If
@@ -1900,16 +1898,14 @@ LogicalResult ReinterpretCastOp::verify() {
            << resultOffset << " instead of " << expectedOffset;
 
   // Match strides in result memref type and in static_strides attribute.
-  for (auto &en :
-       llvm::enumerate(llvm::zip(resultStrides, getStaticStrides()))) {
-    int64_t resultStride = std::get<0>(en.value());
-    int64_t expectedStride = std::get<1>(en.value());
+  for (auto [idx, resultStride, expectedStride] :
+       llvm::enumerate(resultStrides, getStaticStrides())) {
     if (!ShapedType::isDynamic(resultStride) &&
         !ShapedType::isDynamic(expectedStride) &&
         resultStride != expectedStride)
       return emitError("expected result type with stride = ")
              << expectedStride << " instead of " << resultStride
-             << " in dim = " << en.index();
+             << " in dim = " << idx;
   }
 
   return success();
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 3cffe05995f0c..744f5c647b228 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -23,6 +23,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   MLIRArithTransforms
   MLIRBufferizationDialect
   MLIRFuncDialect
+  MLIRGPUOps
   MLIRInferTypeOpInterface
   MLIRLoopLikeInterface
   MLIRMemRefDialect
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ExpandStridedMetadata.cpp b/mlir/lib/Dialect/MemRef/Transforms/ExpandStridedMetadata.cpp
index 8ef16d5eeaec4..72e6a2925e83f 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ExpandStridedMetadata.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ExpandStridedMetadata.cpp
@@ -75,7 +75,7 @@ struct SubviewFolder : public OpRewritePattern<memref::SubViewOp> {
     SmallVector<OpFoldResult> values(2 * sourceRank + 1);
     SmallVector<AffineExpr> symbols(2 * sourceRank + 1);
 
-    detail::bindSymbolsList(rewriter.getContext(), symbols);
+    bindSymbolsList(rewriter.getContext(), MutableArrayRef{symbols});
     AffineExpr expr = symbols.front();
     values[0] = ShapedType::isDynamic(sourceOffset)
                     ? getAsOpFoldResult(newExtractStridedMetadata.getOffset())
@@ -262,10 +262,9 @@ SmallVector<OpFoldResult> getExpandedStrides(memref::ExpandShapeOp expandShape,
   auto sourceType = source.getType().cast<MemRefType>();
   auto [strides, offset] = getStridesAndOffset(sourceType);
 
-  OpFoldResult origStride =
-      ShapedType::isDynamic(strides[groupId])
-          ? origStrides[groupId]
-          : builder.getIndexAttr(strides[groupId]);
+  OpFoldResult origStride = ShapedType::isDynamic(strides[groupId])
+                                ? origStrides[groupId]
+                                : builder.getIndexAttr(strides[groupId]);
 
   // Apply the original stride to all the strides.
   int64_t doneStrideIdx = 0;
diff --git a/mlir/lib/Dialect/MemRef/Transforms/FoldMemRefAliasOps.cpp b/mlir/lib/Dialect/MemRef/Transforms/FoldMemRefAliasOps.cpp
index 129ac41233058..89b34fc14fe42 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/FoldMemRefAliasOps.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/FoldMemRefAliasOps.cpp
@@ -11,6 +11,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/MemRef/Transforms/Passes.h"
 
 #include "mlir/Dialect/Affine/IR/AffineOps.h"
@@ -18,6 +19,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/IR/AffineExpr.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 #include "llvm/ADT/SmallBitVector.h"
@@ -54,24 +56,25 @@ resolveSourceIndicesExpandShape(Location loc, PatternRewriter &rewriter,
                                 memref::ExpandShapeOp expandShapeOp,
                                 ValueRange indices,
                                 SmallVectorImpl<Value> &sourceIndices) {
-  for (SmallVector<int64_t, 2> groups :
-       expandShapeOp.getReassociationIndices()) {
+  MLIRContext *ctx = rewriter.getContext();
+  for (ArrayRef<int64_t> groups : expandShapeOp.getReassociationIndices()) {
     assert(!groups.empty() && "association indices groups cannot be empty");
-    unsigned groupSize = groups.size();
-    SmallVector<int64_t> suffixProduct(groupSize);
-    // Calculate suffix product of dimension sizes for all dimensions of expand
-    // shape op result.
-    suffixProduct[groupSize - 1] = 1;
-    for (unsigned i = groupSize - 1; i > 0; i--)
-      suffixProduct[i - 1] =
-          suffixProduct[i] *
-          expandShapeOp.getType().cast<MemRefType>().getDimSize(groups[i]);
-    SmallVector<Value> dynamicIndices(groupSize);
-    for (unsigned i = 0; i < groupSize; i++)
-      dynamicIndices[i] = indices[groups[i]];
+    int64_t groupSize = groups.size();
+
     // Construct the expression for the index value w.r.t to expand shape op
     // source corresponding the indices wrt to expand shape op result.
-    AffineExpr srcIndexExpr = getLinearAffineExpr(suffixProduct, rewriter);
+    SmallVector<int64_t> sizes(groupSize);
+    for (int64_t i = 0; i < groupSize; ++i)
+      sizes[i] = expandShapeOp.getResultType().getDimSize(groups[i]);
+    SmallVector<int64_t> suffixProduct = computeSuffixProduct(sizes);
+    SmallVector<AffineExpr> dims(groupSize);
+    bindDimsList(ctx, MutableArrayRef{dims});
+    AffineExpr srcIndexExpr = linearize(ctx, dims, suffixProduct);
+
+    /// Apply permutation and create AffineApplyOp.
+    SmallVector<Value> dynamicIndices(groupSize);
+    for (int64_t i = 0; i < groupSize; i++)
+      dynamicIndices[i] = indices[groups[i]];
     sourceIndices.push_back(rewriter.create<AffineApplyOp>(
         loc,
         AffineMap::get(/*numDims=*/groupSize, /*numSymbols=*/0, srcIndexExpr),
@@ -98,35 +101,39 @@ resolveSourceIndicesCollapseShape(Location loc, PatternRewriter &rewriter,
                                   memref::CollapseShapeOp collapseShapeOp,
                                   ValueRange indices,
                                   SmallVectorImpl<Value> &sourceIndices) {
-  unsigned cnt = 0;
+  int64_t cnt = 0;
   SmallVector<Value> tmp(indices.size());
   SmallVector<Value> dynamicIndices;
-  for (SmallVector<int64_t, 2> groups :
-       collapseShapeOp.getReassociationIndices()) {
+  for (ArrayRef<int64_t> groups : collapseShapeOp.getReassociationIndices()) {
     assert(!groups.empty() && "association indices groups cannot be empty");
     dynamicIndices.push_back(indices[cnt++]);
-    unsigned groupSize = groups.size();
-    SmallVector<int64_t> suffixProduct(groupSize);
+    int64_t groupSize = groups.size();
+
     // Calculate suffix product for all collapse op source dimension sizes.
-    suffixProduct[groupSize - 1] = 1;
-    for (unsigned i = groupSize - 1; i > 0; i--)
-      suffixProduct[i - 1] =
-          suffixProduct[i] * collapseShapeOp.getSrcType().getDimSize(groups[i]);
+    SmallVector<int64_t> sizes(groupSize);
+    for (int64_t i = 0; i < groupSize; ++i)
+      sizes[i] = collapseShapeOp.getSrcType().getDimSize(groups[i]);
+    SmallVector<int64_t> suffixProduct = computeSuffixProduct(sizes);
+
     // Derive the index values along all dimensions of the source corresponding
     // to the index wrt to collapsed shape op output.
-    SmallVector<AffineExpr, 4> srcIndexExpr =
-        getDelinearizedAffineExpr(suffixProduct, rewriter);
-    for (unsigned i = 0; i < groupSize; i++)
+    auto d0 = rewriter.getAffineDimExpr(0);
+    SmallVector<AffineExpr> delinearizingExprs = delinearize(d0, suffixProduct);
+
+    // Construct the AffineApplyOp for each delinearizingExpr.
+    for (int64_t i = 0; i < groupSize; i++)
       sourceIndices.push_back(rewriter.create<AffineApplyOp>(
-          loc, AffineMap::get(/*numDims=*/1, /*numSymbols=*/0, srcIndexExpr[i]),
+          loc,
+          AffineMap::get(/*numDims=*/1, /*numSymbols=*/0,
+                         delinearizingExprs[i]),
           dynamicIndices));
     dynamicIndices.clear();
   }
   if (collapseShapeOp.getReassociationIndices().empty()) {
     auto zeroAffineMap = rewriter.getConstantAffineMap(0);
-    unsigned srcRank =
+    int64_t srcRank =
         collapseShapeOp.getViewSource().getType().cast<MemRefType>().getRank();
-    for (unsigned i = 0; i < srcRank; i++)
+    for (int64_t i = 0; i < srcRank; i++)
       sourceIndices.push_back(
           rewriter.create<AffineApplyOp>(loc, zeroAffineMap, dynamicIndices));
   }
@@ -157,9 +164,9 @@ resolveSourceIndicesSubView(Location loc, PatternRewriter &rewriter,
   SmallVector<Value> useIndices;
   // Check if this is rank-reducing case. Then for every unit-dim size add a
   // zero to the indices.
-  unsigned resultDim = 0;
+  int64_t resultDim = 0;
   llvm::SmallBitVector unusedDims = subViewOp.getDroppedDims();
-  for (auto dim : llvm::seq<unsigned>(0, subViewOp.getSourceType().getRank())) {
+  for (auto dim : llvm::seq<int64_t>(0, subViewOp.getSourceType().getRank())) {
     if (unusedDims.test(dim))
       useIndices.push_back(rewriter.create<arith::ConstantIndexOp>(loc, 0));
     else
@@ -171,7 +178,7 @@ resolveSourceIndicesSubView(Location loc, PatternRewriter &rewriter,
   for (auto index : llvm::seq<size_t>(0, mixedOffsets.size())) {
     SmallVector<Value> dynamicOperands;
     AffineExpr expr = rewriter.getAffineDimExpr(0);
-    unsigned numSymbols = 0;
+    int64_t numSymbols = 0;
     dynamicOperands.push_back(useIndices[index]);
 
     // Multiply the stride;
@@ -210,6 +217,14 @@ static Value getMemRefOperand(vector::TransferWriteOp op) {
   return op.getSource();
 }
 
+static Value getMemRefOperand(gpu::SubgroupMmaLoadMatrixOp op) {
+  return op.getSrcMemref();
+}
+
+static Value getMemRefOperand(gpu::SubgroupMmaStoreMatrixOp op) {
+  return op.getDstMemref();
+}
+
 /// Given the permutation map of the original
 /// `vector.transfer_read`/`vector.transfer_write` operations compute the
 /// permutation map to use after the subview is folded with it.
@@ -378,7 +393,7 @@ LogicalResult LoadOpOfSubViewOpFolder<OpTy>::matchAndRewrite(
         affineMap, indices, loadOp.getLoc(), rewriter);
     indices.assign(expandedIndices.begin(), expandedIndices.end());
   }
-  SmallVector<Value, 4> sourceIndices;
+  SmallVector<Value> sourceIndices;
   if (failed(resolveSourceIndicesSubView(loadOp.getLoc(), rewriter, subViewOp,
                                          indices, sourceIndices)))
     return failure();
@@ -401,6 +416,11 @@ LogicalResult LoadOpOfSubViewOpFolder<OpTy>::matchAndRewrite(
             transferReadOp.getPadding(),
             /*mask=*/Value(), transferReadOp.getInBoundsAttr());
       })
+      .Case([&](gpu::SubgroupMmaLoadMatrixOp op) {
+        rewriter.replaceOpWithNewOp<gpu::SubgroupMmaLoadMatrixOp>(
+            op, op.getType(), subViewOp.getSource(), sourceIndices,
+            op.getLeadDimension(), op.getTransposeAttr());
+      })
       .Default([](Operation *) { llvm_unreachable("unexpected operation."); });
   return success();
 }
@@ -424,7 +444,7 @@ LogicalResult LoadOpOfExpandShapeOpFolder<OpTy>::matchAndRewrite(
         affineMap, indices, loadOp.getLoc(), rewriter);
     indices.assign(expandedIndices.begin(), expandedIndices.end());
   }
-  SmallVector<Value, 4> sourceIndices;
+  SmallVector<Value> sourceIndices;
   if (failed(resolveSourceIndicesExpandShape(
           loadOp.getLoc(), rewriter, expandShapeOp, indices, sourceIndices)))
     return failure();
@@ -456,7 +476,7 @@ LogicalResult LoadOpOfCollapseShapeOpFolder<OpTy>::matchAndRewrite(
         affineMap, indices, loadOp.getLoc(), rewriter);
     indices.assign(expandedIndices.begin(), expandedIndices.end());
   }
-  SmallVector<Value, 4> sourceIndices;
+  SmallVector<Value> sourceIndices;
   if (failed(resolveSourceIndicesCollapseShape(
           loadOp.getLoc(), rewriter, collapseShapeOp, indices, sourceIndices)))
     return failure();
@@ -488,7 +508,7 @@ LogicalResult StoreOpOfSubViewOpFolder<OpTy>::matchAndRewrite(
         affineMap, indices, storeOp.getLoc(), rewriter);
     indices.assign(expandedIndices.begin(), expandedIndices.end());
   }
-  SmallVector<Value, 4> sourceIndices;
+  SmallVector<Value> sourceIndices;
   if (failed(resolveSourceIndicesSubView(storeOp.getLoc(), rewriter, subViewOp,
                                          indices, sourceIndices)))
     return failure();
@@ -496,11 +516,11 @@ LogicalResult StoreOpOfSubViewOpFolder<OpTy>::matchAndRewrite(
   llvm::TypeSwitch<Operation *, void>(storeOp)
       .Case([&](AffineStoreOp op) {
         rewriter.replaceOpWithNewOp<AffineStoreOp>(
-            storeOp, storeOp.getValue(), subViewOp.getSource(), sourceIndices);
+            op, op.getValue(), subViewOp.getSource(), sourceIndices);
       })
       .Case([&](memref::StoreOp op) {
         rewriter.replaceOpWithNewOp<memref::StoreOp>(
-            storeOp, storeOp.getValue(), subViewOp.getSource(), sourceIndices,
+            op, op.getValue(), subViewOp.getSource(), sourceIndices,
             op.getNontemporal());
       })
       .Case([&](vector::TransferWriteOp op) {
@@ -510,6 +530,11 @@ LogicalResult StoreOpOfSubViewOpFolder<OpTy>::matchAndRewrite(
                                   op.getPermutationMap()),
             op.getInBoundsAttr());
       })
+      .Case([&](gpu::SubgroupMmaStoreMatrixOp op) {
+        rewriter.replaceOpWithNewOp<gpu::SubgroupMmaStoreMatrixOp>(
+            op, op.getSrc(), subViewOp.getSource(), sourceIndices,
+            op.getLeadDimension(), op.getTransposeAttr());
+      })
       .Default([](Operation *) { llvm_unreachable("unexpected operation."); });
   return success();
 }
@@ -533,7 +558,7 @@ LogicalResult StoreOpOfExpandShapeOpFolder<OpTy>::matchAndRewrite(
         affineMap, indices, storeOp.getLoc(), rewriter);
     indices.assign(expandedIndices.begin(), expandedIndices.end());
   }
-  SmallVector<Value, 4> sourceIndices;
+  SmallVector<Value> sourceIndices;
   if (failed(resolveSourceIndicesExpandShape(
           storeOp.getLoc(), rewriter, expandShapeOp, indices, sourceIndices)))
     return failure();
@@ -566,7 +591,7 @@ LogicalResult StoreOpOfCollapseShapeOpFolder<OpTy>::matchAndRewrite(
         affineMap, indices, storeOp.getLoc(), rewriter);
     indices.assign(expandedIndices.begin(), expandedIndices.end());
   }
-  SmallVector<Value, 4> sourceIndices;
+  SmallVector<Value> sourceIndices;
   if (failed(resolveSourceIndicesCollapseShape(
           storeOp.getLoc(), rewriter, collapseShapeOp, indices, sourceIndices)))
     return failure();
@@ -584,9 +609,11 @@ void memref::populateFoldMemRefAliasOpPatterns(RewritePatternSet &patterns) {
   patterns.add<LoadOpOfSubViewOpFolder<AffineLoadOp>,
                LoadOpOfSubViewOpFolder<memref::LoadOp>,
                LoadOpOfSubViewOpFolder<vector::TransferReadOp>,
+               LoadOpOfSubViewOpFolder<gpu::SubgroupMmaLoadMatrixOp>,
                StoreOpOfSubViewOpFolder<AffineStoreOp>,
                StoreOpOfSubViewOpFolder<memref::StoreOp>,
                StoreOpOfSubViewOpFolder<vector::TransferWriteOp>,
+               StoreOpOfSubViewOpFolder<gpu::SubgroupMmaStoreMatrixOp>,
                LoadOpOfExpandShapeOpFolder<AffineLoadOp>,
                LoadOpOfExpandShapeOpFolder<memref::LoadOp>,
                StoreOpOfExpandShapeOpFolder<AffineStoreOp>,
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ResolveShapedTypeResultDims.cpp b/mlir/lib/Dialect/MemRef/Transforms/ResolveShapedTypeResultDims.cpp
index 2a18c559307b3..50ac04d6d45cc 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ResolveShapedTypeResultDims.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ResolveShapedTypeResultDims.cpp
@@ -84,33 +84,18 @@ struct DimOfReifyRankedShapedTypeOpInterface : public OpRewritePattern<OpTy> {
     OpResult dimValue = dimOp.getSource().template dyn_cast<OpResult>();
     if (!dimValue)
       return failure();
-    auto rankedShapeTypeOp =
-        dyn_cast<ReifyRankedShapedTypeOpInterface>(dimValue.getOwner());
-    if (!rankedShapeTypeOp)
-      return failure();
-
     std::optional<int64_t> dimIndex = dimOp.getConstantIndex();
     if (!dimIndex)
       return failure();
 
     ReifiedRankedShapedTypeDims reifiedResultShapes;
-    if (failed(
-            rankedShapeTypeOp.reifyResultShapes(rewriter, reifiedResultShapes)))
-      return failure();
-
-    if (reifiedResultShapes.size() != rankedShapeTypeOp->getNumResults())
+    if (failed(reifyResultShapes(rewriter, dimValue.getOwner(),
+                                 reifiedResultShapes)))
       return failure();
-
     unsigned resultNumber = dimValue.getResultNumber();
-    auto sourceType = dimValue.getType().dyn_cast<RankedTensorType>();
-    if (reifiedResultShapes[resultNumber].size() !=
-        static_cast<size_t>(sourceType.getRank()))
-      return failure();
-
-    rewriter.replaceOp(dimOp,
-                       getValueOrCreateConstantIndexOp(
-                           rewriter, dimOp.getLoc(),
-                           reifiedResultShapes[resultNumber][*dimIndex]));
+    Value replacement = getValueOrCreateConstantIndexOp(
+        rewriter, dimOp.getLoc(), reifiedResultShapes[resultNumber][*dimIndex]);
+    rewriter.replaceOp(dimOp, replacement);
     return success();
   }
 };
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 99623cdc2b340..32f76b6cd5154 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -231,6 +231,9 @@ void MmaSparseSyncOp::build(::mlir::OpBuilder &odsBuilder,
 }
 
 LogicalResult MmaSparseSyncOp::verify() {
+  unsigned sparsitySelector = getSparsitySelector();
+  if (sparsitySelector > 1)
+    return emitOpError() << "sparsity selector should be 0 or 1";
   return verifyMmaSyncOp(this->getOperation(), getMatrixA(), getMatrixB(),
                          getMatrixC(), getMmaShapeAsArray(),
                          getOperation()->hasAttr(getTf32EnabledAttrName()),
diff --git a/mlir/lib/Dialect/NVGPU/Utils/MMAUtils.cpp b/mlir/lib/Dialect/NVGPU/Utils/MMAUtils.cpp
index 44f9b6d4ea012..7525f9f57bc5f 100644
--- a/mlir/lib/Dialect/NVGPU/Utils/MMAUtils.cpp
+++ b/mlir/lib/Dialect/NVGPU/Utils/MMAUtils.cpp
@@ -272,60 +272,3 @@ nvgpu::getLaneIdToLdMatrixMatrixCoord(OpBuilder &builder, Location loc,
 
   return failure();
 }
-
-LogicalResult nvgpu::PrepareContractToGPUMMASync::matchAndRewrite(
-    vector::ContractionOp op, PatternRewriter &rewriter) const {
-  Location loc = op.getLoc();
-  Value lhs = op.getLhs();
-  Value rhs = op.getRhs();
-  Value res = op.getAcc();
-
-  // Set up the parallel/reduction structure in right form.
-  using MapList = ArrayRef<ArrayRef<AffineExpr>>;
-  auto infer = [](MapList m) { return AffineMap::inferFromExprList(m); };
-  AffineExpr m;
-  AffineExpr n;
-  AffineExpr k;
-  bindDims(rewriter.getContext(), m, n, k);
-  static constexpr std::array<int64_t, 2> perm = {1, 0};
-  auto iteratorTypes = op.getIteratorTypes().getValue();
-  SmallVector<AffineMap, 4> maps = op.getIndexingMapsArray();
-  if (iteratorTypes.size() != 3)
-    return failure();
-  if (!(vector::isParallelIterator(iteratorTypes[0]) &&
-        vector::isParallelIterator(iteratorTypes[1]) &&
-        vector::isReductionIterator(iteratorTypes[2])))
-    return failure();
-
-  // The canonical form is "TNT" = A row-major, B col-major, C row-major.
-  const auto canonicalForm = infer({{m, k}, {n, k}, {m, n}});
-  if (maps == canonicalForm) {
-    return failure();
-  }
-  if (maps == infer({{m, k}, {k, n}, {m, n}})) {
-    rhs = rewriter.create<vector::TransposeOp>(loc, rhs, perm);
-  } else if (maps == infer({{k, m}, {k, n}, {m, n}})) {
-    lhs = rewriter.create<vector::TransposeOp>(loc, lhs, perm);
-  } else if (maps == infer({{k, m}, {k, n}, {m, n}})) {
-    rhs = rewriter.create<vector::TransposeOp>(loc, rhs, perm);
-    lhs = rewriter.create<vector::TransposeOp>(loc, lhs, perm);
-  } else if (maps == infer({{k, m}, {k, n}, {n, m}})) {
-    std::swap(rhs, lhs);
-    rhs = rewriter.create<vector::TransposeOp>(loc, rhs, perm);
-    lhs = rewriter.create<vector::TransposeOp>(loc, lhs, perm);
-  } else if (maps == infer({{k, m}, {n, k}, {n, m}})) {
-    std::swap(rhs, lhs);
-    rhs = rewriter.create<vector::TransposeOp>(loc, rhs, perm);
-  } else if (maps == infer({{m, k}, {k, n}, {n, m}})) {
-    std::swap(lhs, rhs);
-    lhs = rewriter.create<vector::TransposeOp>(loc, lhs, perm);
-  } else if (maps == infer({{m, k}, {n, k}, {n, m}})) {
-    std::swap(lhs, rhs);
-  } else {
-    return failure();
-  }
-  rewriter.replaceOpWithNewOp<vector::ContractionOp>(
-      op, lhs, rhs, res, rewriter.getAffineMapArrayAttr(canonicalForm),
-      op.getIteratorTypes());
-  return success();
-}
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index 38d91f62df1a1..d3f79c3711d87 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -1417,6 +1417,26 @@ LogicalResult CancellationPointOp::verify() {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// OpenMPDialect helper functions
+//===----------------------------------------------------------------------===//
+
+// Set the omp.is_device attribute on the module with the specified boolean
+void OpenMPDialect::setIsDevice(Operation* module, bool isDevice) {
+  module->setAttr(
+      mlir::StringAttr::get(module->getContext(), llvm::Twine{"omp.is_device"}),
+      mlir::BoolAttr::get(module->getContext(), isDevice));
+}
+
+// Return the value of the omp.is_device attribute stored in the module if it
+// exists, otherwise return false by default
+bool OpenMPDialect::getIsDevice(Operation* module) {
+  if (Attribute isDevice = module->getAttr("omp.is_device"))
+    if (isDevice.isa<mlir::BoolAttr>())
+      return isDevice.dyn_cast<BoolAttr>().getValue();
+  return false;
+}
+
 #define GET_ATTRDEF_CLASSES
 #include "mlir/Dialect/OpenMP/OpenMPOpsAttributes.cpp.inc"
 
diff --git a/mlir/lib/Dialect/SCF/IR/SCF.cpp b/mlir/lib/Dialect/SCF/IR/SCF.cpp
index 8852b79b7b7ee..4e7bcc499be3d 100644
--- a/mlir/lib/Dialect/SCF/IR/SCF.cpp
+++ b/mlir/lib/Dialect/SCF/IR/SCF.cpp
@@ -107,7 +107,7 @@ static void replaceOpWithRegion(PatternRewriter &rewriter, Operation *op,
   Block *block = &region.front();
   Operation *terminator = block->getTerminator();
   ValueRange results = terminator->getOperands();
-  rewriter.mergeBlockBefore(block, op, blockArgs);
+  rewriter.inlineBlockBefore(block, op, blockArgs);
   rewriter.replaceOp(op, results);
   rewriter.eraseOp(terminator);
 }
@@ -434,29 +434,38 @@ ParseResult ForOp::parse(OpAsmParser &parser, OperationState &result) {
       parser.parseOperand(ub) || parser.parseKeyword("step") ||
       parser.parseOperand(step))
     return failure();
-  // Parse optional type, else assume Index.
-  if (parser.parseOptionalColon())
-    type = builder.getIndexType();
-  else if (parser.parseType(type))
-    return failure();
-  inductionVariable.type = type;
-  if (parser.resolveOperand(lb, type, result.operands) ||
-      parser.resolveOperand(ub, type, result.operands) ||
-      parser.resolveOperand(step, type, result.operands))
-    return failure();
 
   // Parse the optional initial iteration arguments.
   SmallVector<OpAsmParser::Argument, 4> regionArgs;
   SmallVector<OpAsmParser::UnresolvedOperand, 4> operands;
   regionArgs.push_back(inductionVariable);
 
-  if (succeeded(parser.parseOptionalKeyword("iter_args"))) {
+  bool hasIterArgs = succeeded(parser.parseOptionalKeyword("iter_args"));
+  if (hasIterArgs) {
     // Parse assignment list and results type list.
     if (parser.parseAssignmentList(regionArgs, operands) ||
         parser.parseArrowTypeList(result.types))
       return failure();
+  }
 
-    // Resolve input operands.
+  if (regionArgs.size() != result.types.size() + 1)
+    return parser.emitError(
+        parser.getNameLoc(),
+        "mismatch in number of loop-carried values and defined values");
+
+  // Parse optional type, else assume Index.
+  if (parser.parseOptionalColon())
+    type = builder.getIndexType();
+  else if (parser.parseType(type))
+    return failure();
+
+  // Resolve input operands.
+  regionArgs.front().type = type;
+  if (parser.resolveOperand(lb, type, result.operands) ||
+      parser.resolveOperand(ub, type, result.operands) ||
+      parser.resolveOperand(step, type, result.operands))
+    return failure();
+  if (hasIterArgs) {
     for (auto argOperandType :
          llvm::zip(llvm::drop_begin(regionArgs), operands, result.types)) {
       Type type = std::get<2>(argOperandType);
@@ -467,11 +476,6 @@ ParseResult ForOp::parse(OpAsmParser &parser, OperationState &result) {
     }
   }
 
-  if (regionArgs.size() != result.types.size() + 1)
-    return parser.emitError(
-        parser.getNameLoc(),
-        "mismatch in number of loop-carried values and defined values");
-
   // Parse the body region.
   Region *body = result.addRegion();
   if (parser.parseRegion(*body, regionArgs))
@@ -630,7 +634,7 @@ namespace {
 // the ForOp region and can just be forwarded after simplifying the op inits,
 // yields and returns.
 //
-// The implementation uses `mergeBlockBefore` to steal the content of the
+// The implementation uses `inlineBlockBefore` to steal the content of the
 // original ForOp and avoid cloning.
 struct ForOpIterArgsFolder : public OpRewritePattern<scf::ForOp> {
   using OpRewritePattern<scf::ForOp>::OpRewritePattern;
@@ -645,7 +649,7 @@ struct ForOpIterArgsFolder : public OpRewritePattern<scf::ForOp> {
     // arguments `newBlockTransferArgs` keeps the 1-1 mapping of original to
     // transformed block argument mappings. This plays the role of a
     // IRMapping for the particular use case of calling into
-    // `mergeBlockBefore`.
+    // `inlineBlockBefore`.
     SmallVector<bool, 4> keepMask;
     keepMask.reserve(yieldOp.getNumOperands());
     SmallVector<Value, 4> newBlockTransferArgs, newIterArgs, newYieldValues,
@@ -715,7 +719,7 @@ struct ForOpIterArgsFolder : public OpRewritePattern<scf::ForOp> {
     // original terminator that has been merged in.
     if (newIterArgs.empty()) {
       auto newYieldOp = cast<scf::YieldOp>(newBlock.getTerminator());
-      rewriter.mergeBlockBefore(&oldBlock, newYieldOp, newBlockTransferArgs);
+      rewriter.inlineBlockBefore(&oldBlock, newYieldOp, newBlockTransferArgs);
       rewriter.eraseOp(newBlock.getTerminator()->getPrevNode());
       rewriter.replaceOp(forOp, newResultValues);
       return success();
@@ -1391,23 +1395,6 @@ InParallelOp ForallOp::getTerminator() {
   return cast<InParallelOp>(getBody()->getTerminator());
 }
 
-/// Helper to sort `values` according to matching `keys`.
-SmallVector<Value> ForallOp::getValuesSortedByKey(
-    ArrayRef<Attribute> keys, ValueRange values,
-    llvm::function_ref<bool(Attribute, Attribute)> compare) {
-  if (keys.empty())
-    return values;
-  assert(keys.size() == values.size() && "unexpected mismatching sizes");
-  auto indices = llvm::to_vector(llvm::seq<int64_t>(0, values.size()));
-  std::sort(indices.begin(), indices.end(),
-            [&](int64_t i, int64_t j) { return compare(keys[i], keys[j]); });
-  SmallVector<Value> res;
-  res.reserve(values.size());
-  for (int64_t i = 0, e = indices.size(); i < e; ++i)
-    res.push_back(values[indices[i]]);
-  return res;
-}
-
 ForallOp mlir::scf::getForallOpThreadIndexOwner(Value val) {
   auto tidxArg = val.dyn_cast<BlockArgument>();
   if (!tidxArg)
@@ -1908,10 +1895,7 @@ struct ConvertTrivialIfToSelect : public OpRewritePattern<IfOp> {
     auto elseYieldArgs = op.elseYield().getOperands();
 
     SmallVector<Type> nonHoistable;
-    for (const auto &it :
-         llvm::enumerate(llvm::zip(thenYieldArgs, elseYieldArgs))) {
-      Value trueVal = std::get<0>(it.value());
-      Value falseVal = std::get<1>(it.value());
+    for (auto [trueVal, falseVal] : llvm::zip(thenYieldArgs, elseYieldArgs)) {
       if (&op.getThenRegion() == trueVal.getParentRegion() ||
           &op.getElseRegion() == falseVal.getParentRegion())
         nonHoistable.push_back(trueVal.getType());
@@ -3680,7 +3664,7 @@ LogicalResult scf::IndexSwitchOp::verify() {
 
   if (failed(verifyRegion(getDefaultRegion(), "default region")))
     return failure();
-  for (auto &[idx, caseRegion] : llvm::enumerate(getCaseRegions()))
+  for (auto [idx, caseRegion] : llvm::enumerate(getCaseRegions()))
     if (failed(verifyRegion(caseRegion, "case region #" + Twine(idx))))
       return failure();
 
diff --git a/mlir/lib/Dialect/SCF/TransformOps/SCFTransformOps.cpp b/mlir/lib/Dialect/SCF/TransformOps/SCFTransformOps.cpp
index 3dafd3d739bf5..b35e1045b2ee0 100644
--- a/mlir/lib/Dialect/SCF/TransformOps/SCFTransformOps.cpp
+++ b/mlir/lib/Dialect/SCF/TransformOps/SCFTransformOps.cpp
@@ -16,7 +16,6 @@
 #include "mlir/Dialect/SCF/Utils/Utils.h"
 #include "mlir/Dialect/Transform/IR/TransformDialect.h"
 #include "mlir/Dialect/Transform/IR/TransformInterfaces.h"
-#include "mlir/Dialect/Transform/IR/TransformUtils.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 
 using namespace mlir;
@@ -89,7 +88,7 @@ transform::LoopOutlineOp::apply(transform::TransformResults &results,
   for (Operation *target : state.getPayloadOps(getTarget())) {
     Location location = target->getLoc();
     Operation *symbolTableOp = SymbolTable::getNearestSymbolTable(target);
-    TrivialPatternRewriter rewriter(getContext());
+    IRRewriter rewriter(getContext());
     scf::ExecuteRegionOp exec = wrapInExecuteRegion(rewriter, target);
     if (!exec) {
       DiagnosedSilenceableFailure diag = emitSilenceableError()
@@ -190,10 +189,10 @@ transform::LoopPipelineOp::applyToOne(scf::ForOp target,
                        getReadLatency());
       };
   scf::ForLoopPipeliningPattern pattern(options, target->getContext());
-  TrivialPatternRewriter rewriter(getContext());
+  IRRewriter rewriter(getContext());
   rewriter.setInsertionPoint(target);
   FailureOr<scf::ForOp> patternResult =
-      pattern.returningMatchAndRewrite(target, rewriter);
+      scf::pipelineForLoop(rewriter, target, options);
   if (succeeded(patternResult)) {
     results.push_back(*patternResult);
     return DiagnosedSilenceableFailure::success();
diff --git a/mlir/lib/Dialect/SCF/Transforms/LoopPipelining.cpp b/mlir/lib/Dialect/SCF/Transforms/LoopPipelining.cpp
index b9182f5a073ed..6a9f725211427 100644
--- a/mlir/lib/Dialect/SCF/Transforms/LoopPipelining.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/LoopPipelining.cpp
@@ -62,13 +62,13 @@ struct LoopPipelinerInternal {
   bool initializeLoopInfo(ForOp op, const PipeliningOption &options);
   /// Emits the prologue, this creates `maxStage - 1` part which will contain
   /// operations from stages [0; i], where i is the part index.
-  void emitPrologue(PatternRewriter &rewriter);
+  void emitPrologue(RewriterBase &rewriter);
   /// Gather liverange information for Values that are used in a different stage
   /// than its definition.
   llvm::MapVector<Value, LiverangeInfo> analyzeCrossStageValues();
   scf::ForOp createKernelLoop(
       const llvm::MapVector<Value, LiverangeInfo> &crossStageValues,
-      PatternRewriter &rewriter,
+      RewriterBase &rewriter,
       llvm::DenseMap<std::pair<Value, unsigned>, unsigned> &loopArgMap);
   /// Emits the pipelined kernel. This clones loop operations following user
   /// order and remaps operands defined in a different stage as their use.
@@ -76,10 +76,10 @@ struct LoopPipelinerInternal {
       scf::ForOp newForOp,
       const llvm::MapVector<Value, LiverangeInfo> &crossStageValues,
       const llvm::DenseMap<std::pair<Value, unsigned>, unsigned> &loopArgMap,
-      PatternRewriter &rewriter);
+      RewriterBase &rewriter);
   /// Emits the epilogue, this creates `maxStage - 1` part which will contain
   /// operations from stages [i; maxStage], where i is the part index.
-  llvm::SmallVector<Value> emitEpilogue(PatternRewriter &rewriter);
+  llvm::SmallVector<Value> emitEpilogue(RewriterBase &rewriter);
 };
 
 bool LoopPipelinerInternal::initializeLoopInfo(
@@ -116,7 +116,7 @@ bool LoopPipelinerInternal::initializeLoopInfo(
 
   // All operations need to have a stage.
   for (Operation &op : forOp.getBody()->without_terminator()) {
-    if (stages.find(&op) == stages.end()) {
+    if (!stages.contains(&op)) {
       op.emitOpError("not assigned a pipeline stage");
       return false;
     }
@@ -144,7 +144,7 @@ bool LoopPipelinerInternal::initializeLoopInfo(
   if (llvm::any_of(forOp.getBody()->getTerminator()->getOperands(),
                    [this](Value operand) {
                      Operation *def = operand.getDefiningOp();
-                     return !def || stages.find(def) == stages.end();
+                     return !def || !stages.contains(def);
                    }))
     return false;
   annotateFn = options.annotateFn;
@@ -172,7 +172,7 @@ cloneAndUpdateOperands(RewriterBase &rewriter, Operation *op,
   return clone;
 }
 
-void LoopPipelinerInternal::emitPrologue(PatternRewriter &rewriter) {
+void LoopPipelinerInternal::emitPrologue(RewriterBase &rewriter) {
   // Initialize the iteration argument to the loop initiale values.
   for (BlockArgument &arg : forOp.getRegionIterArgs()) {
     OpOperand &operand = forOp.getOpOperandForRegionIterArg(arg);
@@ -244,7 +244,7 @@ LoopPipelinerInternal::analyzeCrossStageValues() {
 scf::ForOp LoopPipelinerInternal::createKernelLoop(
     const llvm::MapVector<Value, LoopPipelinerInternal::LiverangeInfo>
         &crossStageValues,
-    PatternRewriter &rewriter,
+    RewriterBase &rewriter,
     llvm::DenseMap<std::pair<Value, unsigned>, unsigned> &loopArgMap) {
   // Creates the list of initial values associated to values used across
   // stages. The initial values come from the prologue created above.
@@ -294,87 +294,12 @@ scf::ForOp LoopPipelinerInternal::createKernelLoop(
   return newForOp;
 }
 
-/// Replace any use of `target` with `replacement` in `op`'s operands or within
-/// `op`'s nested regions.
-static void replaceInOp(Operation *op, Value target, Value replacement) {
-  for (auto &use : llvm::make_early_inc_range(target.getUses())) {
-    if (op->isAncestor(use.getOwner()))
-      use.set(replacement);
-  }
-}
-
-/// Given a cloned op in the new kernel body, updates induction variable uses.
-/// We replace it with a version incremented based on the stage where it is
-/// used.
-static void updateInductionVariableUses(RewriterBase &rewriter, Location loc,
-                                        Operation *newOp, Value newForIv,
-                                        unsigned maxStage, unsigned useStage,
-                                        unsigned step) {
-  rewriter.setInsertionPoint(newOp);
-  Value offset = rewriter.create<arith::ConstantIndexOp>(
-      loc, (maxStage - useStage) * step);
-  Value iv = rewriter.create<arith::AddIOp>(loc, newForIv, offset);
-  replaceInOp(newOp, newForIv, iv);
-  rewriter.setInsertionPointAfter(newOp);
-}
-
-/// If the value is a loop carried value coming from stage N + 1 remap, it will
-/// become a direct use.
-static void updateIterArgUses(RewriterBase &rewriter, IRMapping &bvm,
-                              Operation *newOp, ForOp oldForOp, ForOp newForOp,
-                              unsigned useStage,
-                              const DenseMap<Operation *, unsigned> &stages) {
-
-  for (unsigned i = 0; i < oldForOp.getNumRegionIterArgs(); i++) {
-    Value yieldedVal = oldForOp.getBody()->getTerminator()->getOperand(i);
-    Operation *dep = yieldedVal.getDefiningOp();
-    if (!dep)
-      continue;
-    auto stageDep = stages.find(dep);
-    if (stageDep == stages.end() || stageDep->second == useStage)
-      continue;
-    if (stageDep->second != useStage + 1)
-      continue;
-    Value replacement = bvm.lookup(yieldedVal);
-    replaceInOp(newOp, newForOp.getRegionIterArg(i), replacement);
-  }
-}
-
-/// For operands defined in a previous stage we need to remap it to use the
-/// correct region argument. We look for the right version of the Value based
-/// on the stage where it is used.
-static void updateCrossStageUses(
-    RewriterBase &rewriter, Operation *newOp, IRMapping &bvm, ForOp newForOp,
-    unsigned useStage, const DenseMap<Operation *, unsigned> &stages,
-    const llvm::DenseMap<std::pair<Value, unsigned>, unsigned> &loopArgMap) {
-  // Because we automatically cloned the sub-regions, there's no simple way
-  // to walk the nested regions in pairs of (oldOps, newOps), so we just
-  // traverse the set of remapped loop arguments, filter which ones are
-  // relevant, and replace any uses.
-  for (auto [remapPair, newIterIdx] : loopArgMap) {
-    auto [crossArgValue, stageIdx] = remapPair;
-    Operation *def = crossArgValue.getDefiningOp();
-    assert(def);
-    unsigned stageDef = stages.lookup(def);
-    if (useStage <= stageDef || useStage - stageDef != stageIdx)
-      continue;
-
-    // Use "lookupOrDefault" for the target value because some operations
-    // are remapped, while in other cases the original will be present.
-    Value target = bvm.lookupOrDefault(crossArgValue);
-    Value replacement = newForOp.getRegionIterArg(newIterIdx);
-
-    // Replace uses in the new op's operands and any nested uses.
-    replaceInOp(newOp, target, replacement);
-  }
-}
-
 void LoopPipelinerInternal::createKernel(
     scf::ForOp newForOp,
     const llvm::MapVector<Value, LoopPipelinerInternal::LiverangeInfo>
         &crossStageValues,
     const llvm::DenseMap<std::pair<Value, unsigned>, unsigned> &loopArgMap,
-    PatternRewriter &rewriter) {
+    RewriterBase &rewriter) {
   valueMapping.clear();
 
   // Create the kernel, we clone instruction based on the order given by
@@ -400,19 +325,62 @@ void LoopPipelinerInternal::createKernel(
   for (Operation *op : opOrder) {
     int64_t useStage = stages[op];
     auto *newOp = rewriter.clone(*op, mapping);
-
-    // Within the kernel body, update uses of the induction variable, uses of
-    // the original iter args, and uses of cross stage values.
-    updateInductionVariableUses(rewriter, forOp.getLoc(), newOp,
-                                newForOp.getInductionVar(), maxStage,
-                                stages[op], step);
-    updateIterArgUses(rewriter, mapping, newOp, forOp, newForOp, useStage,
-                      stages);
-    updateCrossStageUses(rewriter, newOp, mapping, newForOp, useStage, stages,
-                         loopArgMap);
+    SmallVector<OpOperand *> operands;
+    // Collect all the operands for the cloned op and its nested ops.
+    op->walk([&operands](Operation *nestedOp) {
+      for (OpOperand &operand : nestedOp->getOpOperands()) {
+        operands.push_back(&operand);
+      }
+    });
+    for (OpOperand *operand : operands) {
+      Operation *nestedNewOp = mapping.lookup(operand->getOwner());
+      // Special case for the induction variable uses. We replace it with a
+      // version incremented based on the stage where it is used.
+      if (operand->get() == forOp.getInductionVar()) {
+        rewriter.setInsertionPoint(newOp);
+        Value offset = rewriter.create<arith::ConstantIndexOp>(
+            forOp.getLoc(), (maxStage - stages[op]) * step);
+        Value iv = rewriter.create<arith::AddIOp>(
+            forOp.getLoc(), newForOp.getInductionVar(), offset);
+        nestedNewOp->setOperand(operand->getOperandNumber(), iv);
+        rewriter.setInsertionPointAfter(newOp);
+        continue;
+      }
+      auto arg = operand->get().dyn_cast<BlockArgument>();
+      if (arg && arg.getOwner() == forOp.getBody()) {
+        // If the value is a loop carried value coming from stage N + 1 remap,
+        // it will become a direct use.
+        Value ret = forOp.getBody()->getTerminator()->getOperand(
+            arg.getArgNumber() - 1);
+        Operation *dep = ret.getDefiningOp();
+        if (!dep)
+          continue;
+        auto stageDep = stages.find(dep);
+        if (stageDep == stages.end() || stageDep->second == useStage)
+          continue;
+        assert(stageDep->second == useStage + 1);
+        nestedNewOp->setOperand(operand->getOperandNumber(),
+                                mapping.lookupOrDefault(ret));
+        continue;
+      }
+      // For operands defined in a previous stage we need to remap it to use
+      // the correct region argument. We look for the right version of the
+      // Value based on the stage where it is used.
+      Operation *def = operand->get().getDefiningOp();
+      if (!def)
+        continue;
+      auto stageDef = stages.find(def);
+      if (stageDef == stages.end() || stageDef->second == useStage)
+        continue;
+      auto remap = loopArgMap.find(
+          std::make_pair(operand->get(), useStage - stageDef->second));
+      assert(remap != loopArgMap.end());
+      nestedNewOp->setOperand(operand->getOperandNumber(),
+                              newForOp.getRegionIterArgs()[remap->second]);
+    }
 
     if (predicates[useStage]) {
-      newOp = predicateFn(newOp, predicates[useStage], rewriter);
+      newOp = predicateFn(rewriter, newOp, predicates[useStage]);
       // Remap the results to the new predicated one.
       for (auto values : llvm::zip(op->getResults(), newOp->getResults()))
         mapping.map(std::get<0>(values), std::get<1>(values));
@@ -462,7 +430,7 @@ void LoopPipelinerInternal::createKernel(
 }
 
 llvm::SmallVector<Value>
-LoopPipelinerInternal::emitEpilogue(PatternRewriter &rewriter) {
+LoopPipelinerInternal::emitEpilogue(RewriterBase &rewriter) {
   llvm::SmallVector<Value> returnValues(forOp->getNumResults());
   // Emit different versions of the induction variable. They will be
   // removed by dead code if not used.
@@ -527,9 +495,8 @@ void LoopPipelinerInternal::setValueMapping(Value key, Value el, int64_t idx) {
 
 } // namespace
 
-FailureOr<ForOp> ForLoopPipeliningPattern::returningMatchAndRewrite(
-    ForOp forOp, PatternRewriter &rewriter) const {
-
+FailureOr<ForOp> mlir::scf::pipelineForLoop(RewriterBase &rewriter, ForOp forOp,
+                                            const PipeliningOption &options) {
   LoopPipelinerInternal pipeliner;
   if (!pipeliner.initializeLoopInfo(forOp, options))
     return failure();
diff --git a/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp b/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
index 003d2d055da2c..6706f54662839 100644
--- a/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
@@ -251,18 +251,20 @@ updateDestinationOperandsForTiledOp(OpBuilder &builder,
 /// a destination passing style op.
 static SmallVector<Value>
 yieldTiledValues(RewriterBase &rewriter, ArrayRef<Value> initValues,
-                 Operation *tiledOp,
+                 TilingResult tilingResult,
                  ArrayRef<SmallVector<OpFoldResult>> tileOffsetsList,
                  ArrayRef<SmallVector<OpFoldResult>> tileSizesList,
                  MutableArrayRef<scf::ForOp> loops) {
   SmallVector<Value> replacements =
-      yieldTiledValues(rewriter, initValues, tiledOp->getResults(),
+      yieldTiledValues(rewriter, initValues, tilingResult.tiledValues,
                        tileOffsetsList, tileSizesList, loops);
-  if (auto dstOp = dyn_cast<DestinationStyleOpInterface>(tiledOp)) {
-    auto innerMostLoop = loops.back();
-    SmallVector<Value> tiledOpDestinationTensors = dstOp.getDpsInitOperands();
-    updateDestinationOperandsForTiledOp(rewriter, tiledOpDestinationTensors,
-                                        innerMostLoop.getRegionIterArgs());
+  for (auto tiledOp : tilingResult.tiledOps) {
+    if (auto dstOp = dyn_cast<DestinationStyleOpInterface>(tiledOp)) {
+      auto innerMostLoop = loops.back();
+      SmallVector<Value> tiledOpDestinationTensors = dstOp.getDpsInitOperands();
+      updateDestinationOperandsForTiledOp(rewriter, tiledOpDestinationTensors,
+                                          innerMostLoop.getRegionIterArgs());
+    }
   }
   return replacements;
 }
@@ -345,9 +347,9 @@ mlir::scf::tileUsingSCFForOp(RewriterBase &rewriter, TilingInterface op,
   if (!tilingResult.loops.empty())
     rewriter.setInsertionPoint(
         tilingResult.loops.back().getBody()->getTerminator());
-  SmallVector<Operation *> tiledImplementation =
+  FailureOr<TilingResult> tiledImplementation =
       op.getTiledImplementation(rewriter, offsets, sizes);
-  tilingResult.tiledOps.append(tiledImplementation);
+  tilingResult.tiledOps.append(tiledImplementation->tiledOps);
   if (op->getNumResults() == 0) {
     // nothing more to do.
     return tilingResult;
@@ -356,9 +358,7 @@ mlir::scf::tileUsingSCFForOp(RewriterBase &rewriter, TilingInterface op,
   // If loops are empty, the tiled op is used as the replacement for the untiled
   // op.
   if (tilingResult.loops.empty()) {
-    tilingResult.replacements = llvm::to_vector(
-        llvm::map_range(tiledImplementation[0]->getResults(),
-                        [](OpResult result) -> Value { return result; }));
+    tilingResult.replacements = tiledImplementation->tiledValues;
     return tilingResult;
   }
 
@@ -384,7 +384,7 @@ mlir::scf::tileUsingSCFForOp(RewriterBase &rewriter, TilingInterface op,
     return rewriter.notifyMatchFailure(op, "failed to get destinations");
 
   tilingResult.replacements = yieldTiledValues(
-      rewriter, destinationTensors, tilingResult.tiledOps.back(),
+      rewriter, destinationTensors, tiledImplementation.value(),
       resultOffsetsList, resultSizesList, tilingResult.loops);
 
   LLVM_DEBUG({
@@ -398,7 +398,7 @@ mlir::scf::tileUsingSCFForOp(RewriterBase &rewriter, TilingInterface op,
 }
 
 FailureOr<scf::SCFReductionTilingResult>
-mlir::scf::tileReductionUsingScf(PatternRewriter &b,
+mlir::scf::tileReductionUsingScf(RewriterBase &b,
                                  PartialReductionOpInterface op,
                                  ArrayRef<OpFoldResult> tileSize) {
   Location loc = op.getLoc();
@@ -423,7 +423,7 @@ mlir::scf::tileReductionUsingScf(PatternRewriter &b,
     return b.notifyMatchFailure(
         op, "only support ops with one reduction dimension.");
   int reductionDim;
-  for (auto &[idx, iteratorType] :
+  for (auto [idx, iteratorType] :
        llvm::enumerate(tilingInterfaceOp.getLoopIteratorTypes())) {
     if (iteratorType == utils::IteratorType::reduction) {
       reductionDim = idx;
@@ -523,12 +523,13 @@ mlir::scf::tileAndFuseProducerOfSlice(RewriterBase &rewriter,
   // 2. Generate the tiled implementation of the producer of the source
   OpBuilder::InsertionGuard g(rewriter);
   rewriter.setInsertionPoint(candidateSliceOp);
-  FailureOr<Value> fusedProducerValue =
+  FailureOr<TilingResult> tileAndFuseResult =
       tensor::replaceExtractSliceWithTiledProducer(rewriter, candidateSliceOp,
                                                    fusableProducer);
-  if (failed(fusedProducerValue))
+  if (failed(tileAndFuseResult))
     return std::nullopt;
-  rewriter.replaceAllUsesWith(candidateSliceOp, fusedProducerValue.value());
+  rewriter.replaceAllUsesWith(candidateSliceOp,
+                              tileAndFuseResult->tiledValues[0]);
 
   // 3. If the slice is for a destination operand, for example,
   //
@@ -592,8 +593,10 @@ mlir::scf::tileAndFuseProducerOfSlice(RewriterBase &rewriter,
       outerMostLoop.setIterArg(iterArgNumber.value(),
                                dstOp.getTiedOpOperand(fusableProducer)->get());
     }
-    if (auto dstOp = fusedProducerValue.value()
-                         .getDefiningOp<DestinationStyleOpInterface>()) {
+    for (auto tileAndFusedOp : tileAndFuseResult->tiledOps) {
+      auto dstOp = dyn_cast<DestinationStyleOpInterface>(tileAndFusedOp);
+      if (!dstOp)
+        continue;
       scf::ForOp innerMostLoop = loops.back();
       updateDestinationOperandsForTiledOp(
           rewriter, dstOp.getDpsInitOperand(resultNumber)->get(),
@@ -601,7 +604,7 @@ mlir::scf::tileAndFuseProducerOfSlice(RewriterBase &rewriter,
     }
   }
   return scf::SCFFuseProducerOfSliceResult{fusableProducer,
-                                           fusedProducerValue.value()};
+                                           tileAndFuseResult->tiledValues[0]};
 }
 
 /// Reconstruct the fused producer from within the tiled-and-fused code.
diff --git a/mlir/lib/Dialect/SCF/Utils/AffineCanonicalizationUtils.cpp b/mlir/lib/Dialect/SCF/Utils/AffineCanonicalizationUtils.cpp
index 7799fa9231c62..6964747cdebb6 100644
--- a/mlir/lib/Dialect/SCF/Utils/AffineCanonicalizationUtils.cpp
+++ b/mlir/lib/Dialect/SCF/Utils/AffineCanonicalizationUtils.cpp
@@ -152,7 +152,7 @@ LogicalResult scf::canonicalizeMinMaxOpInLoop(RewriterBase &rewriter,
   // Find all iteration variables among `minOp`'s operands add constrain them.
   for (Value operand : op->getOperands()) {
     // Skip duplicate ivs.
-    if (llvm::is_contained(allIvs, operand))
+    if (allIvs.contains(operand))
       continue;
 
     // If `operand` is an iteration variable: Find corresponding loop
diff --git a/mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp b/mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp
index 3acfba23ddfd1..1713c4490f960 100644
--- a/mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp
+++ b/mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp
@@ -366,7 +366,6 @@ struct ConvertAccessChain : public ConvertAliasResource<spirv::AccessChainOp> {
     }
 
     Location loc = acOp.getLoc();
-    auto i32Type = rewriter.getI32Type();
 
     if (srcElemType.isIntOrFloat() && dstElemType.isa<VectorType>()) {
       // The source indices are for a buffer with scalar element types. Rewrite
@@ -376,16 +375,19 @@ struct ConvertAccessChain : public ConvertAliasResource<spirv::AccessChainOp> {
       int srcNumBytes = *srcElemType.getSizeInBytes();
       int dstNumBytes = *dstElemType.getSizeInBytes();
       assert(dstNumBytes >= srcNumBytes && dstNumBytes % srcNumBytes == 0);
-      int ratio = dstNumBytes / srcNumBytes;
-      auto ratioValue = rewriter.create<spirv::ConstantOp>(
-          loc, i32Type, rewriter.getI32IntegerAttr(ratio));
 
       auto indices = llvm::to_vector<4>(acOp.getIndices());
       Value oldIndex = indices.back();
+      Type indexType = oldIndex.getType();
+
+      int ratio = dstNumBytes / srcNumBytes;
+      auto ratioValue = rewriter.create<spirv::ConstantOp>(
+          loc, indexType, rewriter.getIntegerAttr(indexType, ratio));
+
       indices.back() =
-          rewriter.create<spirv::SDivOp>(loc, i32Type, oldIndex, ratioValue);
+          rewriter.create<spirv::SDivOp>(loc, indexType, oldIndex, ratioValue);
       indices.push_back(
-          rewriter.create<spirv::SModOp>(loc, i32Type, oldIndex, ratioValue));
+          rewriter.create<spirv::SModOp>(loc, indexType, oldIndex, ratioValue));
 
       rewriter.replaceOpWithNewOp<spirv::AccessChainOp>(
           acOp, adaptor.getBasePtr(), indices);
@@ -400,14 +402,17 @@ struct ConvertAccessChain : public ConvertAliasResource<spirv::AccessChainOp> {
       int srcNumBytes = *srcElemType.getSizeInBytes();
       int dstNumBytes = *dstElemType.getSizeInBytes();
       assert(srcNumBytes >= dstNumBytes && srcNumBytes % dstNumBytes == 0);
-      int ratio = srcNumBytes / dstNumBytes;
-      auto ratioValue = rewriter.create<spirv::ConstantOp>(
-          loc, i32Type, rewriter.getI32IntegerAttr(ratio));
 
       auto indices = llvm::to_vector<4>(acOp.getIndices());
       Value oldIndex = indices.back();
+      Type indexType = oldIndex.getType();
+
+      int ratio = srcNumBytes / dstNumBytes;
+      auto ratioValue = rewriter.create<spirv::ConstantOp>(
+          loc, indexType, rewriter.getIntegerAttr(indexType, ratio));
+
       indices.back() =
-          rewriter.create<spirv::IMulOp>(loc, i32Type, oldIndex, ratioValue);
+          rewriter.create<spirv::IMulOp>(loc, indexType, oldIndex, ratioValue);
 
       rewriter.replaceOpWithNewOp<spirv::AccessChainOp>(
           acOp, adaptor.getBasePtr(), indices);
diff --git a/mlir/lib/Dialect/SparseTensor/IR/SparseTensorDialect.cpp b/mlir/lib/Dialect/SparseTensor/IR/SparseTensorDialect.cpp
index ec185c316a42a..3bf11189d3805 100644
--- a/mlir/lib/Dialect/SparseTensor/IR/SparseTensorDialect.cpp
+++ b/mlir/lib/Dialect/SparseTensor/IR/SparseTensorDialect.cpp
@@ -121,18 +121,18 @@ Type mlir::sparse_tensor::detail::getIntegerOrIndexType(MLIRContext *ctx,
   return IndexType::get(ctx);
 }
 
-Type SparseTensorEncodingAttr::getPointerType() const {
-  return detail::getIntegerOrIndexType(getContext(), getPointerBitWidth());
+Type SparseTensorEncodingAttr::getPosType() const {
+  return detail::getIntegerOrIndexType(getContext(), getPosWidth());
 }
 
-Type SparseTensorEncodingAttr::getIndexType() const {
-  return detail::getIntegerOrIndexType(getContext(), getIndexBitWidth());
+Type SparseTensorEncodingAttr::getCrdType() const {
+  return detail::getIntegerOrIndexType(getContext(), getCrdWidth());
 }
 
 SparseTensorEncodingAttr SparseTensorEncodingAttr::withoutOrdering() const {
-  return SparseTensorEncodingAttr::get(
-      getContext(), getDimLevelType(), AffineMap(), AffineMap(),
-      getPointerBitWidth(), getIndexBitWidth());
+  return SparseTensorEncodingAttr::get(getContext(), getDimLevelType(),
+                                       AffineMap(), AffineMap(), getPosWidth(),
+                                       getCrdWidth());
 }
 
 SparseTensorEncodingAttr SparseTensorEncodingAttr::withoutBitWidths() const {
@@ -227,18 +227,18 @@ Attribute SparseTensorEncodingAttr::parse(AsmParser &parser, Type type) {
   RETURN_ON_FAIL(parser.parseLBrace())
 
   // Process the data from the parsed dictionary value into struct-like data.
-  SmallVector<DimLevelType> dlt;
+  SmallVector<DimLevelType> lvlTypes;
   SmallVector<SparseTensorDimSliceAttr> slices;
   AffineMap dimOrd = {};
   AffineMap higherOrd = {};
-  unsigned ptr = 0;
-  unsigned ind = 0;
+  unsigned posWidth = 0;
+  unsigned crdWidth = 0;
 
   StringRef attrName;
   // Exactly 6 keys.
   SmallVector<StringRef, 6> keys = {"dimLevelType",   "dimOrdering",
-                                    "higherOrdering", "pointerBitWidth",
-                                    "indexBitWidth",  "slice"};
+                                    "higherOrdering", "posWidth",
+                                    "crdWidth",       "slice"};
   while (succeeded(parser.parseOptionalKeyword(&attrName))) {
     if (!llvm::is_contained(keys, attrName)) {
       parser.emitError(parser.getNameLoc(), "unexpected key: ") << attrName;
@@ -247,6 +247,10 @@ Attribute SparseTensorEncodingAttr::parse(AsmParser &parser, Type type) {
 
     // Consume the `=` after keys
     RETURN_ON_FAIL(parser.parseEqual())
+    // FIXME: using `operator==` below duplicates the string comparison
+    // cost of the `is_contained` check above. Should instead use some
+    // "find" function that returns the index into `keys` so that we can
+    // dispatch on that instead.
     if (attrName == "dimLevelType") {
       Attribute attr;
       RETURN_ON_FAIL(parser.parseAttribute(attr));
@@ -257,7 +261,7 @@ Attribute SparseTensorEncodingAttr::parse(AsmParser &parser, Type type) {
         ERROR_IF(!strAttr, "expected a string value in dimension level types")
         auto strVal = strAttr.getValue();
         if (auto optDLT = parseDLT(strVal)) {
-          dlt.push_back(optDLT.value());
+          lvlTypes.push_back(optDLT.value());
         } else {
           parser.emitError(parser.getNameLoc(),
                            "unexpected dimension level type: ")
@@ -277,18 +281,18 @@ Attribute SparseTensorEncodingAttr::parse(AsmParser &parser, Type type) {
       auto affineAttr = attr.dyn_cast<AffineMapAttr>();
       ERROR_IF(!affineAttr, "expected an affine map for higher ordering")
       higherOrd = affineAttr.getValue();
-    } else if (attrName == "pointerBitWidth") {
+    } else if (attrName == "posWidth") {
       Attribute attr;
       RETURN_ON_FAIL(parser.parseAttribute(attr))
       auto intAttr = attr.dyn_cast<IntegerAttr>();
-      ERROR_IF(!intAttr, "expected an integral pointer bitwidth")
-      ptr = intAttr.getInt();
-    } else if (attrName == "indexBitWidth") {
+      ERROR_IF(!intAttr, "expected an integral position bitwidth")
+      posWidth = intAttr.getInt();
+    } else if (attrName == "crdWidth") {
       Attribute attr;
       RETURN_ON_FAIL(parser.parseAttribute(attr))
       auto intAttr = attr.dyn_cast<IntegerAttr>();
       ERROR_IF(!intAttr, "expected an integral index bitwidth")
-      ind = intAttr.getInt();
+      crdWidth = intAttr.getInt();
     } else if (attrName == "slice") {
       RETURN_ON_FAIL(parser.parseLSquare())
       // Dispatches to DimSliceAttr to skip mnemonic
@@ -319,7 +323,8 @@ Attribute SparseTensorEncodingAttr::parse(AsmParser &parser, Type type) {
 
   // Construct struct-like storage for attribute.
   return parser.getChecked<SparseTensorEncodingAttr>(
-      parser.getContext(), dlt, dimOrd, higherOrd, ptr, ind, slices);
+      parser.getContext(), lvlTypes, dimOrd, higherOrd, posWidth, crdWidth,
+      slices);
 }
 
 void SparseTensorEncodingAttr::print(AsmPrinter &printer) const {
@@ -334,10 +339,10 @@ void SparseTensorEncodingAttr::print(AsmPrinter &printer) const {
     printer << ", dimOrdering = affine_map<" << getDimOrdering() << ">";
   if (getHigherOrdering())
     printer << ", higherOrdering = affine_map<" << getHigherOrdering() << ">";
-  if (getPointerBitWidth())
-    printer << ", pointerBitWidth = " << getPointerBitWidth();
-  if (getIndexBitWidth())
-    printer << ", indexBitWidth = " << getIndexBitWidth();
+  if (getPosWidth())
+    printer << ", posWidth = " << getPosWidth();
+  if (getCrdWidth())
+    printer << ", crdWidth = " << getCrdWidth();
   if (!getDimSlices().empty()) {
     printer << ", slice = [ ";
     llvm::interleaveComma(getDimSlices(), printer,
@@ -355,12 +360,12 @@ void SparseTensorEncodingAttr::print(AsmPrinter &printer) const {
 LogicalResult SparseTensorEncodingAttr::verify(
     function_ref<InFlightDiagnostic()> emitError,
     ArrayRef<DimLevelType> dimLevelType, AffineMap dimOrdering,
-    AffineMap higherOrdering, unsigned pointerBitWidth, unsigned indexBitWidth,
+    AffineMap higherOrdering, unsigned posWidth, unsigned crdWidth,
     ArrayRef<SparseTensorDimSliceAttr> dimSlices) {
-  if (!acceptBitWidth(pointerBitWidth))
-    return emitError() << "unexpected pointer bitwidth: " << pointerBitWidth;
-  if (!acceptBitWidth(indexBitWidth))
-    return emitError() << "unexpected index bitwidth: " << indexBitWidth;
+  if (!acceptBitWidth(posWidth))
+    return emitError() << "unexpected position bitwidth: " << posWidth;
+  if (!acceptBitWidth(crdWidth))
+    return emitError() << "unexpected coordinate bitwidth: " << crdWidth;
   // Before we can check that the level-rank is consistent/coherent
   // across all fields, we need to define it.  The source-of-truth for
   // the `getLvlRank` method is the length of the level-types array,
@@ -403,9 +408,9 @@ LogicalResult SparseTensorEncodingAttr::verifyEncoding(
     function_ref<InFlightDiagnostic()> emitError) const {
   // Check structural integrity.  In particular, this ensures that the
   // level-rank is coherent across all the fields.
-  RETURN_FAILURE_IF_FAILED(verify(
-      emitError, getDimLevelType(), getDimOrdering(), getHigherOrdering(),
-      getPointerBitWidth(), getIndexBitWidth(), getDimSlices()))
+  RETURN_FAILURE_IF_FAILED(verify(emitError, getDimLevelType(),
+                                  getDimOrdering(), getHigherOrdering(),
+                                  getPosWidth(), getCrdWidth(), getDimSlices()))
   // Check integrity with tensor type specifics.  In particular, we
   // need only check that the dimension-rank of the tensor agrees with
   // the dimension-rank of the encoding.
@@ -498,11 +503,10 @@ RankedTensorType sparse_tensor::getCOOFromTypeWithOrdering(RankedTensorType rtt,
   // TODO: Maybe pick the bitwidth based on input/output tensors (probably the
   // largest one among them) in the original operation instead of using the
   // default value.
-  unsigned pointerBitWidth = src.getPointerBitWidth();
-  unsigned indexBitWidth = src.getIndexBitWidth();
+  unsigned posWidth = src.getPosWidth();
+  unsigned crdWidth = src.getCrdWidth();
   auto enc = SparseTensorEncodingAttr::get(src.getContext(), lvlTypes, lvlPerm,
-                                           AffineMap(), pointerBitWidth,
-                                           indexBitWidth);
+                                           AffineMap(), posWidth, crdWidth);
   return RankedTensorType::get(src.getDimShape(), src.getElementType(), enc);
 }
 
@@ -566,7 +570,7 @@ Level mlir::sparse_tensor::toStoredDim(RankedTensorType type, Dimension d) {
 /// We normalized sparse tensor encoding attribute by always using
 /// ordered/unique DLT such that "compressed-nu-no" and "compressed-nu" (as well
 /// as other variants) lead to the same storage specifier type, and stripping
-/// irrelevant fields that does not alter the sparse tensor memory layout.
+/// irrelevant fields that do not alter the sparse tensor memory layout.
 static SparseTensorEncodingAttr
 getNormalizedEncodingForSpecifier(SparseTensorEncodingAttr enc) {
   SmallVector<DimLevelType> dlts;
@@ -577,14 +581,11 @@ getNormalizedEncodingForSpecifier(SparseTensorEncodingAttr enc) {
       enc.getContext(), dlts,
       AffineMap(), // dimOrdering (irrelavant to storage speicifer)
       AffineMap(), // highLvlOrdering (irrelavant to storage specifer)
-      // Always use index for memSize, dimSize instead of reusing
-      // getBitwidth from pointers/indices.
-      // It allows us to reuse the same SSA value for different bitwidth,
-      // It also avoids casting between index/integer (returned by DimOp)
-      0, 0,
-      // FIXME: we should keep the slice information, for now it is okay as only
-      // constant can be used for slice
-      ArrayRef<SparseTensorDimSliceAttr>{} /*enc.getDimSlices()*/);
+      // Always use `index` for memSize and lvlSize instead of reusing
+      // `getPosWidth` and `getCrdWidth`. It allows us to reuse the same SSA
+      // value for different bitwidth, it also avoids casting between index and
+      // integer (returned by DimOp)
+      0, 0, enc.getDimSlices());
 }
 
 StorageSpecifierType
@@ -596,17 +597,17 @@ StorageSpecifierType::get(MLIRContext *ctx, SparseTensorEncodingAttr encoding) {
 // SparseTensorDialect Operations.
 //===----------------------------------------------------------------------===//
 
-static LogicalResult dimIsInBounds(Dimension dim, Value tensor) {
-  return success(dim < getDimRank(tensor));
+static LogicalResult lvlIsInBounds(Level lvl, Value tensor) {
+  return success(lvl < getSparseTensorType(tensor).getLvlRank());
 }
 
-static LogicalResult isMatchingWidth(Value result, unsigned width) {
-  const Type etp = getMemRefType(result).getElementType();
+static LogicalResult isMatchingWidth(Value mem, unsigned width) {
+  const Type etp = getMemRefType(mem).getElementType();
   return success(width == 0 ? etp.isIndex() : etp.isInteger(width));
 }
 
 static LogicalResult verifySparsifierGetterSetter(
-    StorageSpecifierKind mdKind, std::optional<APInt> lvl,
+    StorageSpecifierKind mdKind, std::optional<Level> lvl,
     TypedValue<StorageSpecifierType> md, Operation *op) {
   if (mdKind == StorageSpecifierKind::ValMemSize && lvl) {
     return op->emitError(
@@ -616,62 +617,83 @@ static LogicalResult verifySparsifierGetterSetter(
   const auto enc = md.getType().getEncoding();
   const Level lvlRank = enc.getLvlRank();
 
+  if (mdKind == StorageSpecifierKind::DimOffset ||
+      mdKind == StorageSpecifierKind::DimStride)
+    if (!enc.isSlice())
+      return op->emitError("requested slice data on non-slice tensor");
+
   if (mdKind != StorageSpecifierKind::ValMemSize) {
     if (!lvl)
       return op->emitError("missing level argument");
 
-    const Level l = lvl.value().getZExtValue();
+    const Level l = lvl.value();
     if (l >= lvlRank)
-      return op->emitError("requested level out of bound");
+      return op->emitError("requested level is out of bounds");
 
-    if (mdKind == StorageSpecifierKind::PtrMemSize && enc.isSingletonLvl(l))
+    if (mdKind == StorageSpecifierKind::PosMemSize && enc.isSingletonLvl(l))
       return op->emitError(
-          "requested pointer memory size on a singleton level");
+          "requested position memory size on a singleton level");
   }
   return success();
 }
 
-static LogicalResult verifyPackUnPack(Operation *op, TensorType cooTp,
-                                      TensorType dataTp, TensorType idxTp) {
-  if (!isUniqueCOOType(cooTp))
-    return op->emitError("must operate on a COO tensor");
-
-  auto enc = getSparseTensorEncoding(cooTp);
-  if (idxTp.getElementType() != enc.getIndexType() ||
-      dataTp.getElementType() != cooTp.getElementType())
-    return op->emitError("unmatched type between input and output");
-
-  auto dNOE = dataTp.getShape()[0];
-  auto iNOE = idxTp.getShape()[0];
-  if (!ShapedType::isDynamic(dNOE) && !ShapedType::isDynamic(iNOE) &&
-      dNOE != iNOE)
-    return op->emitError("unmatched number of elements in data and indices");
-
-  // A tensor<?xNxi32> for indices means the input COO is rank N
-  auto inRank = idxTp.getShape()[1];
-  auto ouRank = cooTp.getRank();
-  if (!ShapedType::isDynamic(inRank) && inRank != ouRank)
-    return op->emitError("unmatched rank between input and output");
+static LogicalResult verifyPackUnPack(Operation *op, bool requiresStaticShape,
+                                      SparseTensorType tensorTp,
+                                      RankedTensorType valuesTp,
+                                      RankedTensorType coordinatesTp) {
+  if (requiresStaticShape && !tensorTp.hasStaticDimShape())
+    return op->emitError("the sparse-tensor must have static shape");
+  if (!tensorTp.hasEncoding())
+    return op->emitError("the sparse-tensor must have an encoding attribute");
+  if (!tensorTp.isIdentity())
+    return op->emitError("the sparse-tensor must have the identity mapping");
+  if (!isUniqueCOOType(tensorTp))
+    return op->emitError("the sparse-tensor must have a COO type");
+
+  if (coordinatesTp.getRank() != 2)
+    return op->emitError("coordinates must have rank 2");
+  if (requiresStaticShape && !coordinatesTp.hasStaticShape())
+    return op->emitError("coordinates must have static shape");
+  if (coordinatesTp.getElementType() != tensorTp.getCrdType())
+    return op->emitError("input/output coordinate-types don't match");
+
+  if (valuesTp.getRank() != 1)
+    return op->emitError("values must have rank 1");
+  if (requiresStaticShape && !valuesTp.hasStaticShape())
+    return op->emitError("values must have static shape");
+  if (valuesTp.getElementType() != tensorTp.getElementType())
+    return op->emitError("input/output element-types don't match");
+
+  const auto valuesNSE = valuesTp.getShape()[0];
+  const auto coordsNSE = coordinatesTp.getShape()[0];
+  if (!ShapedType::isDynamic(valuesNSE) && !ShapedType::isDynamic(coordsNSE) &&
+      valuesNSE != coordsNSE)
+    return op->emitError("values/coordinates number-of-elements don't match");
+
+  // NOTE: We use `getLvlRank` because the `coordinatesTp` is for
+  // level-coordinates (cf., the op documentation).
+  const DynSize coordsRank = coordinatesTp.getShape()[1];
+  const Level tensorRank = tensorTp.getLvlRank();
+  // FIXME: replace the `operator!=` with our backported `safelyNE`.
+  if (!ShapedType::isDynamic(coordsRank) &&
+      coordsRank != static_cast<DynSize>(tensorRank))
+    return op->emitError("input/output level-ranks don't match");
 
   return success();
 }
 
 LogicalResult PackOp::verify() {
-  TensorType dataTp = getData().getType(), idxTp = getIndices().getType();
-  TensorType retTp = getResult().getType();
-
-  if (!retTp.hasStaticShape() || !dataTp.hasStaticShape() ||
-      !idxTp.hasStaticShape())
-    return emitError("all input types must be statically shaped");
-
-  return verifyPackUnPack(*this, retTp, dataTp, idxTp);
+  const auto valuesTp = getRankedTensorType(getValues());
+  const auto coordinatesTp = getRankedTensorType(getCoordinates());
+  const auto resTp = getSparseTensorType(getResult());
+  return verifyPackUnPack(*this, true, resTp, valuesTp, coordinatesTp);
 }
 
 LogicalResult UnpackOp::verify() {
-  TensorType dataTp = getData().getType(), idxTp = getIndices().getType();
-  TensorType srcTp = getTensor().getType();
-
-  return verifyPackUnPack(*this, srcTp, dataTp, idxTp);
+  const auto valuesTp = getRankedTensorType(getValues());
+  const auto coordinatesTp = getRankedTensorType(getCoordinates());
+  const auto srcTp = getSparseTensorType(getTensor());
+  return verifyPackUnPack(*this, false, srcTp, valuesTp, coordinatesTp);
 }
 
 LogicalResult ConvertOp::verify() {
@@ -704,27 +726,25 @@ OpFoldResult ConvertOp::fold(FoldAdaptor adaptor) {
   return {};
 }
 
-LogicalResult ToPointersOp::verify() {
+LogicalResult ToPositionsOp::verify() {
   auto e = getSparseTensorEncoding(getTensor().getType());
-  // FIXME: there seems to be some dim/lvl confusion here.
-  if (failed(dimIsInBounds(getDimension().getZExtValue(), getTensor())))
-    return emitError("requested pointers dimension out of bounds");
-  if (failed(isMatchingWidth(getResult(), e.getPointerBitWidth())))
-    return emitError("unexpected type for pointers");
+  if (failed(lvlIsInBounds(getLevel(), getTensor())))
+    return emitError("requested level is out of bounds");
+  if (failed(isMatchingWidth(getResult(), e.getPosWidth())))
+    return emitError("unexpected type for positions");
   return success();
 }
 
-LogicalResult ToIndicesOp::verify() {
+LogicalResult ToCoordinatesOp::verify() {
   auto e = getSparseTensorEncoding(getTensor().getType());
-  // FIXME: there seems to be some dim/lvl confusion here.
-  if (failed(dimIsInBounds(getDimension().getZExtValue(), getTensor())))
-    return emitError("requested indices dimension out of bounds");
-  if (failed(isMatchingWidth(getResult(), e.getIndexBitWidth())))
-    return emitError("unexpected type for indices");
+  if (failed(lvlIsInBounds(getLevel(), getTensor())))
+    return emitError("requested level is out of bounds");
+  if (failed(isMatchingWidth(getResult(), e.getCrdWidth())))
+    return emitError("unexpected type for coordinates");
   return success();
 }
 
-LogicalResult ToIndicesBufferOp::verify() {
+LogicalResult ToCoordinatesBufferOp::verify() {
   auto e = getSparseTensorEncoding(getTensor().getType());
   if (getCOOStart(e) >= e.getLvlRank())
     return emitError("expected sparse tensor with a COO region");
@@ -755,7 +775,7 @@ LogicalResult ToSliceStrideOp::verify() {
 
 LogicalResult GetStorageSpecifierOp::verify() {
   RETURN_FAILURE_IF_FAILED(verifySparsifierGetterSetter(
-      getSpecifierKind(), getDim(), getSpecifier(), getOperation()))
+      getSpecifierKind(), getLevel(), getSpecifier(), getOperation()))
   return success();
 }
 
@@ -765,17 +785,17 @@ static SetStorageSpecifierOp getSpecifierSetDef(SpecifierOp op) {
 }
 
 OpFoldResult GetStorageSpecifierOp::fold(FoldAdaptor adaptor) {
-  StorageSpecifierKind kind = getSpecifierKind();
-  std::optional<APInt> dim = getDim();
+  const StorageSpecifierKind kind = getSpecifierKind();
+  const auto lvl = getLevel();
   for (auto op = getSpecifierSetDef(*this); op; op = getSpecifierSetDef(op))
-    if (kind == op.getSpecifierKind() && dim == op.getDim())
+    if (kind == op.getSpecifierKind() && lvl == op.getLevel())
       return op.getValue();
   return {};
 }
 
 LogicalResult SetStorageSpecifierOp::verify() {
   RETURN_FAILURE_IF_FAILED(verifySparsifierGetterSetter(
-      getSpecifierKind(), getDim(), getSpecifier(), getOperation()))
+      getSpecifierKind(), getLevel(), getSpecifier(), getOperation()))
   return success();
 }
 
@@ -865,7 +885,7 @@ LogicalResult UnaryOp::verify() {
 
 LogicalResult ConcatenateOp::verify() {
   const auto dstTp = getSparseTensorType(*this);
-  const Dimension concatDim = getDimension().getZExtValue();
+  const Dimension concatDim = getDimension();
   const Dimension dimRank = dstTp.getDimRank();
 
   if (getInputs().size() <= 1)
@@ -922,8 +942,9 @@ LogicalResult ConcatenateOp::verify() {
 }
 
 LogicalResult InsertOp::verify() {
-  if (getDimRank(getTensor()) != static_cast<Dimension>(getIndices().size()))
-    return emitOpError("incorrect number of indices");
+  const auto stt = getSparseTensorType(getTensor());
+  if (stt.getLvlRank() != static_cast<Level>(getLvlCoords().size()))
+    return emitOpError("incorrect number of coordinates");
   return success();
 }
 
@@ -942,9 +963,9 @@ LogicalResult PushBackOp::verify() {
 }
 
 LogicalResult CompressOp::verify() {
-  if (getDimRank(getTensor()) !=
-      1 + static_cast<Dimension>(getIndices().size()))
-    return emitOpError("incorrect number of indices");
+  const auto stt = getSparseTensorType(getTensor());
+  if (stt.getLvlRank() != 1 + static_cast<Level>(getLvlCoords().size()))
+    return emitOpError("incorrect number of coordinates");
   return success();
 }
 
@@ -960,7 +981,7 @@ void ForeachOp::build(
   const auto stt = getSparseTensorType(tensor);
   const Dimension dimRank = stt.getDimRank();
 
-  // Starts with `dimRank`-many indices.
+  // Starts with `dimRank`-many coordinates.
   SmallVector<Type> blockArgTypes(dimRank, builder.getIndexType());
   // Followed by one value.
   blockArgTypes.push_back(stt.getElementType());
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/BufferizableOpInterfaceImpl.cpp
index 641b429f300a4..6fbdc7384f69f 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -139,8 +139,8 @@ struct PackOpInterface
                                             const AnalysisState &state) const {
     assert(op->getNumResults() == 1);
     assert(isUniqueCOOType(op->getResultTypes()[0].cast<RankedTensorType>()));
-    // PackOp reuses the input tensors as data/indices instead of creating new
-    // ones when packing into a COO format.
+    // PackOp reuses the input tensors as values/coordinates instead of
+    // creating new ones when packing into a COO format.
     return {{op->getOpResult(0), BufferRelation::Equivalent}};
   }
 
@@ -171,7 +171,7 @@ struct UnpackOpInterface
 
   AliasingOpResultList getAliasingOpResults(Operation *op, OpOperand &opOperand,
                                             const AnalysisState &state) const {
-    // Conceptually, UnpackOp equals to a list of toIndices/toValueOp
+    // Conceptually, UnpackOp equals to a list of toCoordinates/toValueOp
     return {};
   }
 };
@@ -217,9 +217,10 @@ struct NumberOfEntriesOpInterface
   }
 };
 
-struct ToIndicesBufferOpInterface
+struct ToCoordinatesBufferOpInterface
     : public BufferizableOpInterface::ExternalModel<
-          ToIndicesBufferOpInterface, sparse_tensor::ToIndicesBufferOp> {
+          ToCoordinatesBufferOpInterface,
+          sparse_tensor::ToCoordinatesBufferOp> {
   bool bufferizesToMemoryRead(Operation *op, OpOperand &opOperand,
                               const AnalysisState &state) const {
     return true;
@@ -227,8 +228,8 @@ struct ToIndicesBufferOpInterface
 
   bool bufferizesToMemoryWrite(Operation *op, OpOperand &opOperand,
                                const AnalysisState &state) const {
-    // Potential writes into memory through the result of sparse_tensor.indices
-    // are not considered.
+    // Potential writes into memory through the result of
+    // `sparse_tensor.coordinates` are not considered.
     return false;
   }
 
@@ -238,9 +239,9 @@ struct ToIndicesBufferOpInterface
   }
 };
 
-struct ToIndicesOpInterface
+struct ToCoordinatesOpInterface
     : public BufferizableOpInterface::ExternalModel<
-          ToIndicesOpInterface, sparse_tensor::ToIndicesOp> {
+          ToCoordinatesOpInterface, sparse_tensor::ToCoordinatesOp> {
   bool bufferizesToMemoryRead(Operation *op, OpOperand &opOperand,
                               const AnalysisState &state) const {
     return true;
@@ -248,8 +249,8 @@ struct ToIndicesOpInterface
 
   bool bufferizesToMemoryWrite(Operation *op, OpOperand &opOperand,
                                const AnalysisState &state) const {
-    // Potential writes into memory through the result of sparse_tensor.indices
-    // are not considered.
+    // Potential writes into memory through the result of
+    // `sparse_tensor.coordinates` are not considered.
     return false;
   }
 
@@ -259,9 +260,9 @@ struct ToIndicesOpInterface
   }
 };
 
-struct ToPointersOpInterface
+struct ToPositionsOpInterface
     : public BufferizableOpInterface::ExternalModel<
-          ToPointersOpInterface, sparse_tensor::ToPointersOp> {
+          ToPositionsOpInterface, sparse_tensor::ToPositionsOp> {
   bool bufferizesToMemoryRead(Operation *op, OpOperand &opOperand,
                               const AnalysisState &state) const {
     return true;
@@ -269,8 +270,8 @@ struct ToPointersOpInterface
 
   bool bufferizesToMemoryWrite(Operation *op, OpOperand &opOperand,
                                const AnalysisState &state) const {
-    // Potential writes into memory through the result of sparse_tensor.pointers
-    // are not considered.
+    // Potential writes into memory through the result of
+    // `sparse_tensor.positions` are not considered.
     return false;
   }
 
@@ -318,10 +319,11 @@ void mlir::sparse_tensor::registerBufferizableOpInterfaceExternalModels(
         NumberOfEntriesOpInterface>(*ctx);
     sparse_tensor::PackOp::attachInterface<PackOpInterface>(*ctx);
     sparse_tensor::UnpackOp::attachInterface<UnpackOpInterface>(*ctx);
-    sparse_tensor::ToIndicesBufferOp::attachInterface<
-        ToIndicesBufferOpInterface>(*ctx);
-    sparse_tensor::ToIndicesOp::attachInterface<ToIndicesOpInterface>(*ctx);
-    sparse_tensor::ToPointersOp::attachInterface<ToPointersOpInterface>(*ctx);
+    sparse_tensor::ToCoordinatesBufferOp::attachInterface<
+        ToCoordinatesBufferOpInterface>(*ctx);
+    sparse_tensor::ToCoordinatesOp::attachInterface<ToCoordinatesOpInterface>(
+        *ctx);
+    sparse_tensor::ToPositionsOp::attachInterface<ToPositionsOpInterface>(*ctx);
     sparse_tensor::ToValuesOp::attachInterface<ToValuesOpInterface>(*ctx);
   });
 }
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/CodegenEnv.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/CodegenEnv.cpp
index 62acac20cd967..8e4904ad3a592 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/CodegenEnv.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/CodegenEnv.cpp
@@ -38,12 +38,12 @@ CodegenEnv::CodegenEnv(linalg::GenericOp linop, SparsificationOptions opts,
     : linalgOp(linop), sparseOptions(opts),
       latticeMerger(numTensors, numLoops, numFilterLoops), loopEmitter(),
       topSort(), sparseOut(nullptr), outerParNest(-1u), insChain(), expValues(),
-      expFilled(), expAdded(), expCount(), redVal(), redExp(-1u),
-      redCustom(-1u), redValidLexInsert() {}
+      expFilled(), expAdded(), expCount(), redVal(), redExp(kInvalidId),
+      redCustom(kInvalidId), redValidLexInsert() {}
 
 LogicalResult CodegenEnv::initTensorExp() {
   // Builds the tensor expression for the Linalg operation in SSA form.
-  std::optional<unsigned> optExp = latticeMerger.buildTensorExpFromLinalg(op());
+  std::optional<ExprId> optExp = latticeMerger.buildTensorExpFromLinalg(op());
   if (!optExp || !isAdmissibleTensorExp(*optExp))
     return failure();
 
@@ -101,7 +101,7 @@ std::optional<Operation *> CodegenEnv::genLoopBoundary(
 // Code generation environment verify functions.
 //===----------------------------------------------------------------------===//
 
-bool CodegenEnv::isAdmissibleTensorExp(unsigned exp) {
+bool CodegenEnv::isAdmissibleTensorExp(ExprId exp) {
   // We reject any expression that makes a reduction from `-outTensor`, as those
   // expressions create a dependency between the current iteration (i) and the
   // previous iteration (i-1). It would require iterating over the whole
@@ -115,7 +115,10 @@ bool CodegenEnv::isAdmissibleTensorExp(unsigned exp) {
   }
 
   OpOperand *lhs = linalgOp.getDpsInitOperand(0);
-  unsigned tensor = lhs->getOperandNumber();
+  // That the operand number is a valid `TensorId` will be verified
+  // by the call to `isSingleCondition` below; though we may want to add
+  // assertions to check it here, in order to give better error messages.
+  const TensorId tensor = lhs->getOperandNumber();
   // An non-annotated output tensor is assumed dense, and becomes a random
   // access n-dim memref. Admissible since insertions cannot occur.
   if (getSparseTensorType(lhs->get()).isAllDense())
@@ -140,13 +143,14 @@ bool CodegenEnv::isAdmissibleTopoOrder() {
   OpOperand *lhs = linalgOp.getDpsInitOperand(0);
   // Accept "truly dynamic" if the output tensor materializes uninitialized
   // into the computation and insertions occur in lexicographic index order.
-  unsigned nest = 0;
-  auto iteratorTypes = linalgOp.getIteratorTypesArray();
-  for (unsigned i = 0, e = latticeMerger.getNumLoops(); i < e; i++) {
-    if (!latticeMerger.isFilterLoop(topSortAt(i))) {
+  LoopOrd nest = 0;
+  const auto iteratorTypes = linalgOp.getIteratorTypesArray();
+  assert(topSortSize() == latticeMerger.getNumLoops());
+  for (const LoopId i : topSort) {
+    if (!latticeMerger.isFilterLoop(i)) {
       // We only count non-filter loops as filter loops should be considered
-      // as a special type of parallel loops.
-      if (linalg::isReductionIterator(iteratorTypes[topSortAt(i)]))
+      // a special type of parallel loops.
+      if (linalg::isReductionIterator(iteratorTypes[i]))
         break; // terminate at first reduction
       nest++;
     }
@@ -154,7 +158,7 @@ bool CodegenEnv::isAdmissibleTopoOrder() {
   // Determine admissible dynamic insertion situations:
   // (1) fully injective, since there are no reductions,
   // (2) admissible 1-d expansion in innermost dimension.
-  if (nest >= linalgOp.getRank(lhs) - 1) {
+  if (static_cast<int64_t>(nest) >= linalgOp.getRank(lhs) - 1) {
     outerParNest = nest;
     return true;
   }
@@ -165,19 +169,26 @@ bool CodegenEnv::isAdmissibleTopoOrder() {
 // Code generation environment topological sort methods
 //===----------------------------------------------------------------------===//
 
-ArrayRef<unsigned> CodegenEnv::getTopSortSlice(size_t n, size_t m) const {
-  return ArrayRef<unsigned>(topSort).slice(n, m);
+ArrayRef<LoopId> CodegenEnv::getTopSortSlice(LoopOrd n, LoopOrd m) const {
+  return ArrayRef<LoopId>(topSort).slice(n, m);
 }
 
-ArrayRef<unsigned> CodegenEnv::getLoopCurStack() const {
-  return getTopSortSlice(0, loopEmitter.getCurrentDepth());
+ArrayRef<LoopId> CodegenEnv::getLoopStackUpTo(LoopOrd n) const {
+  return ArrayRef<LoopId>(topSort).take_front(n);
 }
 
-Value CodegenEnv::getLoopIdxValue(size_t loopIdx) const {
-  for (unsigned lv = 0, lve = topSort.size(); lv < lve; lv++)
-    if (topSort[lv] == loopIdx)
-      return loopEmitter.getLoopIV(lv);
-  llvm_unreachable("invalid loop index");
+ArrayRef<LoopId> CodegenEnv::getCurrentLoopStack() const {
+  return getLoopStackUpTo(loopEmitter.getCurrentDepth());
+}
+
+Value CodegenEnv::getLoopVar(LoopId i) const {
+  // TODO: this class should store the inverse of `topSort` so that
+  // it can do this conversion directly, instead of searching through
+  // `topSort` every time.  (Or else, `LoopEmitter` should handle this.)
+  for (LoopOrd n = 0, numLoops = topSortSize(); n < numLoops; n++)
+    if (topSort[n] == i)
+      return loopEmitter.getLoopIV(n);
+  llvm_unreachable("invalid loop identifier");
 }
 
 //===----------------------------------------------------------------------===//
@@ -189,8 +200,10 @@ void CodegenEnv::updateInsertionChain(Value chain) {
   insChain = chain;
 }
 
-bool CodegenEnv::atExpandLevel(OpOperand *o, unsigned rank, unsigned lv) const {
-  return sparseOut == o && outerParNest == rank - 1 && outerParNest == lv;
+// FIXME: clarify what this "rank" is really supposed to mean/be.
+bool CodegenEnv::atExpandLevel(OpOperand *o, unsigned rank, LoopOrd n) const {
+  return sparseOut == o && outerParNest == static_cast<LoopOrd>(rank - 1) &&
+         outerParNest == n;
 }
 
 void CodegenEnv::startExpand(Value values, Value filled, Value added,
@@ -216,21 +229,21 @@ void CodegenEnv::endExpand() {
 // Code generation environment reduction methods
 //===----------------------------------------------------------------------===//
 
-void CodegenEnv::startReduc(unsigned exp, Value val) {
-  assert(redExp == -1u && exp != -1u);
+void CodegenEnv::startReduc(ExprId exp, Value val) {
+  assert(!isReduc() && exp != kInvalidId);
   redExp = exp;
   updateReduc(val);
 }
 
 void CodegenEnv::updateReduc(Value val) {
-  assert(redExp != -1u);
+  assert(isReduc());
   redVal = exp(redExp).val = val;
 }
 
 Value CodegenEnv::endReduc() {
   Value val = redVal;
   updateReduc(Value());
-  redExp = -1u;
+  redExp = kInvalidId;
   return val;
 }
 
@@ -244,17 +257,17 @@ void CodegenEnv::clearValidLexInsert() {
   redValidLexInsert = Value();
 }
 
-void CodegenEnv::startCustomReduc(unsigned exp) {
-  assert(redCustom == -1u && exp != -1u);
+void CodegenEnv::startCustomReduc(ExprId exp) {
+  assert(!isCustomReduc() && exp != kInvalidId);
   redCustom = exp;
 }
 
 Value CodegenEnv::getCustomRedId() {
-  assert(redCustom != -1u);
+  assert(isCustomReduc());
   return dyn_cast<sparse_tensor::ReduceOp>(exp(redCustom).op).getIdentity();
 }
 
 void CodegenEnv::endCustomReduc() {
-  assert(redCustom != -1u);
-  redCustom = -1u;
+  assert(isCustomReduc());
+  redCustom = kInvalidId;
 }
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/CodegenEnv.h b/mlir/lib/Dialect/SparseTensor/Transforms/CodegenEnv.h
index b210ca873520f..8c6a7bd6433db 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/CodegenEnv.h
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/CodegenEnv.h
@@ -45,7 +45,7 @@ class CodegenEnv {
   //
 
   LogicalResult initTensorExp();
-  unsigned getTensorExp() const { return tensorExp; }
+  ExprId getExprId() const { return tensorExp; }
 
   linalg::GenericOp op() const { return linalgOp; }
   const SparsificationOptions &options() const { return sparseOptions; }
@@ -65,13 +65,13 @@ class CodegenEnv {
   // Merger delegates.
   //
 
-  TensorExp &exp(unsigned e) { return latticeMerger.exp(e); }
-  LatPoint &lat(unsigned l) { return latticeMerger.lat(l); }
-  SmallVector<unsigned> &set(unsigned s) { return latticeMerger.set(s); }
-  DimLevelType dlt(unsigned t, unsigned i) const {
+  TensorExp &exp(ExprId e) { return latticeMerger.exp(e); }
+  LatPoint &lat(LatPointId l) { return latticeMerger.lat(l); }
+  SmallVector<LatPointId> &set(LatSetId s) { return latticeMerger.set(s); }
+  DimLevelType dlt(TensorId t, LoopId i) const {
     return latticeMerger.getDimLevelType(t, i);
   }
-  DimLevelType dlt(unsigned b) const {
+  DimLevelType dlt(TensorLoopId b) const {
     return latticeMerger.getDimLevelType(b);
   }
 
@@ -81,7 +81,7 @@ class CodegenEnv {
 
   /// Whether the tensor expression is admissible for codegen.
   /// It also sets the sparseOut if the output tensor is sparse.
-  bool isAdmissibleTensorExp(unsigned exp);
+  bool isAdmissibleTensorExp(ExprId e);
 
   /// Whether the iteration graph is sorted in admissible topoOrder.
   /// Sets outerParNest on success with sparse output
@@ -91,17 +91,21 @@ class CodegenEnv {
   // Topological delegate and sort methods.
   //
 
-  size_t topSortSize() const { return topSort.size(); }
-  unsigned topSortAt(unsigned i) const { return topSort.at(i); }
-  void topSortPushBack(unsigned i) { topSort.push_back(i); }
-  void topSortClear(unsigned capacity = 0) {
+  LoopOrd topSortSize() const { return topSort.size(); }
+  LoopId topSortAt(LoopOrd n) const { return topSort.at(n); }
+  void topSortPushBack(LoopId i) { topSort.push_back(i); }
+  void topSortClear(size_t capacity = 0) {
     topSort.clear();
     topSort.reserve(capacity);
   }
 
-  ArrayRef<unsigned> getTopSortSlice(size_t n, size_t m) const;
-  ArrayRef<unsigned> getLoopCurStack() const;
-  Value getLoopIdxValue(size_t loopIdx) const;
+  ArrayRef<LoopId> getTopSortSlice(LoopOrd n, LoopOrd m) const;
+  ArrayRef<LoopId> getLoopStackUpTo(LoopOrd n) const;
+  ArrayRef<LoopId> getCurrentLoopStack() const;
+  /// Returns the induction-variable for the loop identified by the given
+  /// `LoopId`.  This method handles application of the topological sort
+  /// in order to convert the `LoopId` into the corresponding `LoopOrd`.
+  Value getLoopVar(LoopId i) const;
 
   //
   // Sparse tensor output and expansion methods.
@@ -113,7 +117,8 @@ class CodegenEnv {
   Value getInsertionChain() const { return insChain; }
   void updateInsertionChain(Value chain);
 
-  bool atExpandLevel(OpOperand *o, unsigned rank, unsigned lv) const;
+  // FIXME: clarify what this "rank" is really supposed to mean/be.
+  bool atExpandLevel(OpOperand *o, unsigned rank, LoopOrd n) const;
   void startExpand(Value values, Value filled, Value added, Value count);
   bool isExpand() const { return expValues != nullptr; }
   void updateExpandCount(Value count);
@@ -127,8 +132,8 @@ class CodegenEnv {
   // Reduction methods.
   //
 
-  void startReduc(unsigned exp, Value val);
-  bool isReduc() const { return redExp != -1u; }
+  void startReduc(ExprId exp, Value val);
+  bool isReduc() const { return redExp != kInvalidId; }
   void updateReduc(Value val);
   Value getReduc() const { return redVal; }
   Value endReduc();
@@ -136,8 +141,8 @@ class CodegenEnv {
   void clearValidLexInsert();
   Value getValidLexInsert() const { return redValidLexInsert; }
 
-  void startCustomReduc(unsigned exp);
-  bool isCustomReduc() const { return redCustom != -1u; }
+  void startCustomReduc(ExprId exp);
+  bool isCustomReduc() const { return redCustom != kInvalidId; }
   Value getCustomRedId();
   void endCustomReduc();
 
@@ -154,14 +159,16 @@ class CodegenEnv {
   // Loop emitter helper class.
   LoopEmitter loopEmitter;
 
-  // Topological sort.
-  std::vector<unsigned> topSort;
+  // Topological sort.  This serves as a mapping from `LoopOrd` to `LoopId`
+  // (cf., `getLoopVar` and `topSortAt`).
+  std::vector<LoopId> topSort;
 
   // Sparse tensor as output. Implemented either through direct injective
   // insertion in lexicographic index order or through access pattern
   // expansion in the innermost loop nest (`expValues` through `expCount`).
   OpOperand *sparseOut;
-  unsigned outerParNest;
+  // The count of outer non-filter loops, as defined by `isAdmissibleTopoOrder`.
+  LoopOrd outerParNest;
   Value insChain;
   Value expValues;
   Value expFilled;
@@ -172,8 +179,8 @@ class CodegenEnv {
   // into the merger's expression tree. When the indices of a tensor reduction
   // expression are exhausted, all inner loops can use a scalarized reduction.
   Value redVal;
-  unsigned redExp;
-  unsigned redCustom;
+  ExprId redExp;
+  ExprId redCustom;
 
   // Bookkeeping for lex insertion during reductions. Holds the runtime boolean
   // value of whether any reduction occurred. This is only set during a
@@ -181,7 +188,7 @@ class CodegenEnv {
   Value redValidLexInsert;
 
   // The root tensor expression of the kernel.
-  unsigned tensorExp;
+  ExprId tensorExp;
 };
 
 } // namespace sparse_tensor
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/CodegenUtils.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/CodegenUtils.cpp
index cb4eda192d9a9..bdd6020d9d0ac 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/CodegenUtils.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/CodegenUtils.cpp
@@ -24,52 +24,50 @@ using namespace mlir;
 using namespace mlir::sparse_tensor;
 
 /// If the tensor is a sparse constant, generates and returns the pair of
-/// the constants for the indices and the values.
+/// the constants for the coordinates and the values.
 static std::optional<std::pair<Value, Value>>
 genSplitSparseConstant(OpBuilder &builder, Location loc, Value tensor) {
   if (auto constOp = tensor.getDefiningOp<arith::ConstantOp>()) {
-    if (auto attr = constOp.getValue().dyn_cast<SparseElementsAttr>()) {
-      DenseElementsAttr indicesAttr = attr.getIndices();
-      Value indices = builder.create<arith::ConstantOp>(loc, indicesAttr);
-      DenseElementsAttr valuesAttr = attr.getValues();
-      Value values = builder.create<arith::ConstantOp>(loc, valuesAttr);
-      return std::make_pair(indices, values);
+    if (auto a = constOp.getValue().dyn_cast<SparseElementsAttr>()) {
+      auto coordinates = builder.create<arith::ConstantOp>(loc, a.getIndices());
+      auto values = builder.create<arith::ConstantOp>(loc, a.getValues());
+      return std::make_pair(coordinates, values);
     }
   }
   return {};
 }
 
-/// Generates the code to copy the index at indices[ivs] to ind, and return
-/// the value at value[ivs].
-static Value genIndexAndValueForSparse(OpBuilder &builder, Location loc,
-                                       Value indices, Value values,
-                                       SmallVectorImpl<Value> &indicesArray,
-                                       ValueRange ivs, unsigned rank) {
-  for (unsigned i = 0; i < rank; i++) {
-    Value idx = constantIndex(builder, loc, i);
-    Value val = builder.create<tensor::ExtractOp>(loc, indices,
-                                                  ValueRange{ivs[0], idx});
-    val = builder.create<arith::IndexCastOp>(loc, builder.getIndexType(), val);
-    // builder.create<memref::StoreOp>(loc, val, ind, idx);
-    indicesArray.push_back(val);
+/// Reads `coordinates[k][0..rank-1]` and `value[k]`, appending the
+/// former onto `cvs` and returning the latter.
+// FIXME: Change the `rank` argument to `Dimension dimRank` or `Level lvlRank`,
+// to clarify its intended meaning.
+static Value genCoordsAndValueForSparse(OpBuilder &builder, Location loc,
+                                        Value coordinates, Value values,
+                                        SmallVectorImpl<Value> &cvs, Value k,
+                                        unsigned rank) {
+  for (unsigned d = 0; d < rank; d++) {
+    Value dim = constantIndex(builder, loc, d);
+    Value crd =
+        builder.create<tensor::ExtractOp>(loc, coordinates, ValueRange{k, dim});
+    crd = builder.create<arith::IndexCastOp>(loc, builder.getIndexType(), crd);
+    // builder.create<memref::StoreOp>(loc, crd, cvs, dim);
+    cvs.push_back(crd);
   }
-  return builder.create<tensor::ExtractOp>(loc, values, ivs[0]);
+  return builder.create<tensor::ExtractOp>(loc, values, k);
 }
 
-/// Generates the code to read the value from tensor[ivs], and conditionally
-/// stores the indices ivs to the memory in ind. The generated code looks like
-/// the following and the insertion point after this routine is inside the
-/// if-then branch behind the assignment to ind. This is to ensure that the
-/// code that uses the ind, such as an addEltX call generated after, is inside
-/// the if-then branch.
+/// Generates code to read the value from `tensor[ivs]`, and open
+/// a conditional for whether the value is non-zero.  The generated code
+/// looks like the following and the insertion point after this routine
+/// is inside the then-branch.
 ///    if (tensor[ivs] != 0)
-///      ind = ivs
-static Value genIndexAndValueForDense(OpBuilder &builder, Location loc,
-                                      Value tensor,
-                                      SmallVectorImpl<Value> &indicesArray,
-                                      ValueRange ivs) {
+///      insert_point
+static Value genCoordsAndValueForDense(OpBuilder &builder, Location loc,
+                                       Value tensor,
+                                       SmallVectorImpl<Value> &cvs,
+                                       ValueRange ivs) {
   Value val = genValueForDense(builder, loc, tensor, ivs);
-  indicesArray.append(ivs.begin(), ivs.end());
+  cvs.append(ivs.begin(), ivs.end());
   return val;
 }
 
@@ -101,6 +99,8 @@ OverheadType mlir::sparse_tensor::overheadTypeEncoding(Type tp) {
   llvm_unreachable("Unknown overhead type");
 }
 
+// TODO: should offer an overload of this that takes a `MLIRContext*`
+// instead of the builder, similar to `detail::getIntegerOrIndexType`.
 Type mlir::sparse_tensor::getOverheadType(Builder &builder, OverheadType ot) {
   switch (ot) {
   case OverheadType::kIndex:
@@ -118,24 +118,18 @@ Type mlir::sparse_tensor::getOverheadType(Builder &builder, OverheadType ot) {
 }
 
 OverheadType
-mlir::sparse_tensor::pointerOverheadTypeEncoding(SparseTensorEncodingAttr enc) {
-  return overheadTypeEncoding(enc.getPointerBitWidth());
+mlir::sparse_tensor::posTypeEncoding(SparseTensorEncodingAttr enc) {
+  return overheadTypeEncoding(enc.getPosWidth());
 }
 
 OverheadType
-mlir::sparse_tensor::indexOverheadTypeEncoding(SparseTensorEncodingAttr enc) {
-  return overheadTypeEncoding(enc.getIndexBitWidth());
-}
-
-Type mlir::sparse_tensor::getPointerOverheadType(Builder &builder,
-                                                 SparseTensorEncodingAttr enc) {
-  return getOverheadType(builder, pointerOverheadTypeEncoding(enc));
+mlir::sparse_tensor::crdTypeEncoding(SparseTensorEncodingAttr enc) {
+  return overheadTypeEncoding(enc.getCrdWidth());
 }
 
-Type mlir::sparse_tensor::getIndexOverheadType(Builder &builder,
-                                               SparseTensorEncodingAttr enc) {
-  return getOverheadType(builder, indexOverheadTypeEncoding(enc));
-}
+// TODO: we ought to add some `static_assert` tests to ensure that the
+// `STEA::get{Pos,Crd}Type` methods agree with `getOverheadType(builder,
+// {pos,crd}OverheadTypeEncoding(enc))`
 
 // TODO: Adjust the naming convention for the constructors of
 // `OverheadType` so we can use the `MLIR_SPARSETENSOR_FOREVERY_O` x-macro
@@ -205,36 +199,37 @@ StringRef mlir::sparse_tensor::primaryTypeFunctionSuffix(Type elemTp) {
 //===----------------------------------------------------------------------===//
 
 Value sparse_tensor::genCast(OpBuilder &builder, Location loc, Value value,
-                             Type dstTy) {
-  Type srcTy = value.getType();
-  if (srcTy != dstTy) {
-    // int <=> index
-    if (dstTy.isa<IndexType>() || srcTy.isa<IndexType>())
-      return builder.create<arith::IndexCastOp>(loc, dstTy, value);
-
-    bool ext = srcTy.getIntOrFloatBitWidth() < dstTy.getIntOrFloatBitWidth();
-
-    // float => float.
-    if (srcTy.isa<FloatType>() && dstTy.isa<FloatType>() && ext)
-      return builder.create<arith::ExtFOp>(loc, dstTy, value);
-
-    if (srcTy.isa<FloatType>() && dstTy.isa<FloatType>() && !ext)
-      return builder.create<arith::TruncFOp>(loc, dstTy, value);
-
-    // int => int
-    if (srcTy.isUnsignedInteger() && dstTy.isa<IntegerType>() && ext)
-      return builder.create<arith::ExtUIOp>(loc, dstTy, value);
-
-    if (srcTy.isSignedInteger() && dstTy.isa<IntegerType>() && ext)
-      return builder.create<arith::ExtSIOp>(loc, dstTy, value);
-
-    if (srcTy.isa<IntegerType>() && dstTy.isa<IntegerType>() && !ext)
-      return builder.create<arith::TruncIOp>(loc, dstTy, value);
+                             Type dstTp) {
+  const Type srcTp = value.getType();
+  if (srcTp == dstTp)
+    return value;
+
+  // int <=> index
+  if (srcTp.isa<IndexType>() || dstTp.isa<IndexType>())
+    return builder.create<arith::IndexCastOp>(loc, dstTp, value);
+
+  const bool ext =
+      srcTp.getIntOrFloatBitWidth() < dstTp.getIntOrFloatBitWidth();
+
+  // float => float.
+  if (srcTp.isa<FloatType>() && dstTp.isa<FloatType>()) {
+    if (ext)
+      return builder.create<arith::ExtFOp>(loc, dstTp, value);
+    return builder.create<arith::TruncFOp>(loc, dstTp, value);
+  }
 
-    llvm_unreachable("unhandled type casting");
+  // int => int
+  const auto srcIntTp = srcTp.dyn_cast<IntegerType>();
+  if (srcIntTp && dstTp.isa<IntegerType>()) {
+    if (!ext)
+      return builder.create<arith::TruncIOp>(loc, dstTp, value);
+    if (srcIntTp.isUnsigned())
+      return builder.create<arith::ExtUIOp>(loc, dstTp, value);
+    if (srcIntTp.isSigned())
+      return builder.create<arith::ExtSIOp>(loc, dstTp, value);
   }
 
-  return value;
+  llvm_unreachable("unhandled type casting");
 }
 
 mlir::Attribute mlir::sparse_tensor::getOneAttr(Builder &builder, Type tp) {
@@ -323,53 +318,52 @@ void mlir::sparse_tensor::genReshapeDstShape(
   assert(start == staticDstShape.size());
 }
 
-void mlir::sparse_tensor::translateIndicesArray(
+void mlir::sparse_tensor::reshapeCvs(
     OpBuilder &builder, Location loc,
-    ArrayRef<ReassociationIndices> reassociation, ValueRange srcIndices,
-    ArrayRef<Value> srcShape, ArrayRef<Value> dstShape,
-    SmallVectorImpl<Value> &dstIndices) {
+    ArrayRef<ReassociationIndices> reassociation, // NOLINT
+    ValueRange srcSizes, ValueRange srcCvs,       // NOLINT
+    ValueRange dstSizes, SmallVectorImpl<Value> &dstCvs) {
+  const unsigned srcRank = srcSizes.size();
+  const unsigned dstRank = dstSizes.size();
+  assert(srcRank == srcCvs.size() && "Source rank mismatch");
+  const bool isCollapse = srcRank > dstRank;
+  const ValueRange sizes = isCollapse ? srcSizes : dstSizes;
+  // Iterate over reassociation map.
   unsigned i = 0;
   unsigned start = 0;
-  unsigned dstRank = dstShape.size();
-  unsigned srcRank = srcShape.size();
-  assert(srcRank == srcIndices.size());
-  bool isCollapse = srcRank > dstRank;
-  ArrayRef<Value> shape = isCollapse ? srcShape : dstShape;
-  // Iterate over reassociation map.
   for (const auto &map : llvm::enumerate(reassociation)) {
     // Prepare strides information in dimension slice.
     Value linear = constantIndex(builder, loc, 1);
     for (unsigned j = start, end = start + map.value().size(); j < end; j++) {
-      linear = builder.create<arith::MulIOp>(loc, linear, shape[j]);
+      linear = builder.create<arith::MulIOp>(loc, linear, sizes[j]);
     }
     // Start expansion.
     Value val;
     if (!isCollapse)
-      val = srcIndices[i];
+      val = srcCvs[i];
     // Iterate over dimension slice.
     for (unsigned j = start, end = start + map.value().size(); j < end; j++) {
-      linear = builder.create<arith::DivUIOp>(loc, linear, shape[j]);
+      linear = builder.create<arith::DivUIOp>(loc, linear, sizes[j]);
       if (isCollapse) {
-        Value old = srcIndices[j];
-        Value mul = builder.create<arith::MulIOp>(loc, old, linear);
+        const Value mul = builder.create<arith::MulIOp>(loc, srcCvs[j], linear);
         val = val ? builder.create<arith::AddIOp>(loc, val, mul) : mul;
       } else {
-        Value old = val;
+        const Value old = val;
         val = builder.create<arith::DivUIOp>(loc, val, linear);
-        assert(dstIndices.size() == j);
-        dstIndices.push_back(val);
+        assert(dstCvs.size() == j);
+        dstCvs.push_back(val);
         val = builder.create<arith::RemUIOp>(loc, old, linear);
       }
     }
     // Finalize collapse.
     if (isCollapse) {
-      assert(dstIndices.size() == i);
-      dstIndices.push_back(val);
+      assert(dstCvs.size() == i);
+      dstCvs.push_back(val);
     }
     start += map.value().size();
     i++;
   }
-  assert(dstIndices.size() == dstRank);
+  assert(dstCvs.size() == dstRank);
 }
 
 FlatSymbolRefAttr mlir::sparse_tensor::getFunc(ModuleOp module, StringRef name,
@@ -471,46 +465,51 @@ Value mlir::sparse_tensor::genValueForDense(OpBuilder &builder, Location loc,
 // FIXME:
 // 1. Dense tensors loop should be generated by loop emitter.
 // 2. Support reduction variables to propagate SSA chains properly.
+// 3. Change the `rank` argument to `Dimension dimRank` or `Level lvlRank`,
+//    to clarify its meaning.
 void mlir::sparse_tensor::genDenseTensorOrSparseConstantIterLoop(
     OpBuilder &builder, Location loc, Value src, unsigned rank,
     function_ref<void(OpBuilder &, Location, Value, ValueRange)> bodyBuilder) {
-  SmallVector<Value> indicesArray;
+  // `cvs` is actually the flattened coordinates array for all elements,
+  // not just for one element (since we do not `SmallVector::clear` after
+  // each iteration of the body of the loopnest.
+  SmallVector<Value> cvs;
   SmallVector<Value> lo;
   SmallVector<Value> hi;
   SmallVector<Value> st;
-  Value zero = constantIndex(builder, loc, 0);
-  Value one = constantIndex(builder, loc, 1);
-  auto indicesValues = genSplitSparseConstant(builder, loc, src);
-  bool isCOOConstant = indicesValues.has_value();
-  Value indices;
-  Value values;
-  if (isCOOConstant) {
-    indices = indicesValues->first;
-    values = indicesValues->second;
+  const Value zero = constantIndex(builder, loc, 0);
+  const Value one = constantIndex(builder, loc, 1);
+  const auto splitSrc = genSplitSparseConstant(builder, loc, src);
+  if (splitSrc.has_value()) {
+    const Value srcCoordinates = splitSrc->first;
+    const Value srcValues = splitSrc->second;
     lo.push_back(zero);
-    hi.push_back(linalg::createOrFoldDimOp(builder, loc, values, 0));
+    hi.push_back(linalg::createOrFoldDimOp(builder, loc, srcValues, 0));
     st.push_back(one);
+    scf::buildLoopNest(builder, loc, lo, hi, st, {},
+                       [&](OpBuilder &builder, Location loc, ValueRange ivs,
+                           ValueRange /*args*/) -> scf::ValueVector {
+                         Value val = genCoordsAndValueForSparse(
+                             builder, loc, srcCoordinates, srcValues, cvs,
+                             ivs[0], rank);
+                         bodyBuilder(builder, loc, val, cvs);
+                         return {};
+                       });
   } else {
     for (unsigned i = 0; i < rank; i++) {
       lo.push_back(zero);
       hi.push_back(linalg::createOrFoldDimOp(builder, loc, src, i));
       st.push_back(one);
     }
+    scf::buildLoopNest(builder, loc, lo, hi, st, {},
+                       [&](OpBuilder &builder, Location loc, ValueRange ivs,
+                           ValueRange /*args*/) -> scf::ValueVector {
+                         Value val = genCoordsAndValueForDense(builder, loc,
+                                                               src, cvs, ivs);
+                         bodyBuilder(builder, loc, val, cvs);
+                         return {};
+                       });
   }
-
-  scf::buildLoopNest(
-      builder, loc, lo, hi, st, {},
-      [&](OpBuilder &builder, Location loc, ValueRange ivs,
-          ValueRange args) -> scf::ValueVector {
-        Value val;
-        if (isCOOConstant)
-          val = genIndexAndValueForSparse(builder, loc, indices, values,
-                                          indicesArray, ivs, rank);
-        else
-          val = genIndexAndValueForDense(builder, loc, src, indicesArray, ivs);
-        bodyBuilder(builder, loc, val, indicesArray);
-        return {};
-      });
 }
 
 void mlir::sparse_tensor::sizesFromSrc(OpBuilder &builder,
@@ -534,85 +533,108 @@ Operation *mlir::sparse_tensor::getTop(Operation *op) {
 void sparse_tensor::foreachInSparseConstant(
     Location loc, RewriterBase &rewriter, SparseElementsAttr attr,
     AffineMap order, function_ref<void(ArrayRef<Value>, Value)> callback) {
-  Dimension dimRank = getSparseTensorType(attr).getDimRank();
-  // Foreach on constant.
-  DenseElementsAttr indicesAttr = attr.getIndices();
-  DenseElementsAttr valuesAttr = attr.getValues();
-
-  using CooValuePair = std::pair<SmallVector<IntegerAttr>, Attribute>;
-  SmallVector<CooValuePair> cooV;
-  for (size_t i = 0, nse = valuesAttr.size(); i < nse; i++) {
-    cooV.emplace_back();
-    for (Dimension j = 0; j < dimRank; j++) {
-      auto coordAttr = indicesAttr.getValues<IntegerAttr>()[i * dimRank + j];
-      cooV.back().first.push_back(coordAttr);
-    }
-    auto valAttr = valuesAttr.getValues<Attribute>()[i];
-    cooV.back().second = valAttr;
+  const Dimension dimRank = getSparseTensorType(attr).getDimRank();
+  const auto coordinates = attr.getIndices().getValues<IntegerAttr>();
+  const auto values = attr.getValues().getValues<Attribute>();
+
+  // This is like the `Element<V>` class in the runtime library, but for
+  // MLIR attributes.  In the future we may want to move this out into
+  // a proper class definition to help improve code legibility (e.g.,
+  // `first` -> `coords`, `second` -> `value`) as well as being able
+  // to factor out analogues of `ElementLT<V>` for the sort below, etc.
+  using ElementAttr = std::pair<SmallVector<IntegerAttr>, Attribute>;
+
+  // Construct the COO from the SparseElementsAttr.
+  SmallVector<ElementAttr> elems;
+  for (size_t i = 0, nse = values.size(); i < nse; i++) {
+    elems.emplace_back();
+    elems.back().second = values[i];
+    auto &coords = elems.back().first;
+    coords.reserve(dimRank);
+    for (Dimension d = 0; d < dimRank; d++)
+      coords.push_back(coordinates[i * dimRank + d]);
   }
 
   // Sorts the sparse element attribute based on coordinates.
-  std::sort(cooV.begin(), cooV.end(),
-            [order](const CooValuePair &lhs, const CooValuePair &rhs) {
-              const SmallVectorImpl<IntegerAttr> &lc = lhs.first;
-              const SmallVectorImpl<IntegerAttr> &rc = rhs.first;
-              for (size_t i = 0, e = lc.size(); i < e; i++) {
-                auto l =
-                    order
-                        ? order.getResult(i).cast<AffineDimExpr>().getPosition()
-                        : i;
-                if (lc[l].getInt() == rc[l].getInt())
+  std::sort(elems.begin(), elems.end(),
+            [order, dimRank](const ElementAttr &lhs, const ElementAttr &rhs) {
+              const auto &lhsCoords = lhs.first;
+              const auto &rhsCoords = rhs.first;
+              for (Dimension d = 0; d < dimRank; d++) {
+                // FIXME: This only makes sense for permutations.
+                // And since we don't check that `order` is a permutation,
+                // it can also cause OOB errors when we use `l`.
+                const Level l = order ? order.getDimPosition(d) : d;
+                if (lhsCoords[l].getInt() == rhsCoords[l].getInt())
                   continue;
-                return lc[l].getInt() < rc[l].getInt();
+                return lhsCoords[l].getInt() < rhsCoords[l].getInt();
               }
               llvm_unreachable("no equal coordinate in sparse element attr");
             });
 
-  SmallVector<Value> coords;
-  for (size_t i = 0, nse = valuesAttr.size(); i < nse; i++) {
-    coords.clear();
-    for (Dimension j = 0; j < dimRank; j++) {
-      auto coordAttr = cooV[i].first[j];
-      auto coord =
-          rewriter.create<arith::ConstantIndexOp>(loc, coordAttr.getInt());
-      // Remaps coordinates.
-      coords.push_back(coord);
+  SmallVector<Value> cvs;
+  cvs.reserve(dimRank);
+  for (size_t i = 0, nse = values.size(); i < nse; i++) {
+    // Remap coordinates.
+    cvs.clear();
+    for (Dimension d = 0; d < dimRank; d++) {
+      auto crd = elems[i].first[d].getInt();
+      cvs.push_back(rewriter.create<arith::ConstantIndexOp>(loc, crd));
     }
+    // Remap value.
     Value val;
     if (attr.getElementType().isa<ComplexType>()) {
-      auto valAttr = cooV[i].second.cast<ArrayAttr>();
+      auto valAttr = elems[i].second.cast<ArrayAttr>();
       val = rewriter.create<complex::ConstantOp>(loc, attr.getElementType(),
                                                  valAttr);
     } else {
-      auto valAttr = cooV[i].second.cast<TypedAttr>();
-      // Remaps value.
+      auto valAttr = elems[i].second.cast<TypedAttr>();
       val = rewriter.create<arith::ConstantOp>(loc, valAttr);
     }
     assert(val);
-    callback(coords, val);
+    callback(cvs, val);
   }
 }
 
-void sparse_tensor::storeIndices(OpBuilder &builder, Location loc,
-                                 unsigned size, Value ind, ValueRange ivs,
-                                 unsigned offsetDim, Value offset) {
+SmallVector<Value> sparse_tensor::loadAll(OpBuilder &builder, Location loc,
+                                          size_t size, Value mem,
+                                          size_t offsetIdx, Value offsetVal) {
 #ifndef NDEBUG
-  const auto memTp = ind.getType().cast<MemRefType>();
-  (void)memTp;
+  const auto memTp = mem.getType().cast<MemRefType>();
   assert(memTp.getRank() == 1);
   const DynSize memSh = memTp.getDimSize(0);
-  (void)memSh;
-  assert(ShapedType::isDynamic(memSh) || memSh == static_cast<DynSize>(size));
-  assert(ivs.size() == static_cast<size_t>(size));
-  assert(offsetDim < size);
+  assert(ShapedType::isDynamic(memSh) || memSh >= static_cast<DynSize>(size));
+  assert(offsetIdx == 0 || offsetIdx < size);
 #endif // NDEBUG
-
+  SmallVector<Value> vs;
+  vs.reserve(size);
   for (unsigned i = 0; i < size; i++) {
-    Value idx = ivs[i];
-    if (offsetDim == i && offset)
-      idx = builder.create<arith::AddIOp>(loc, idx, offset);
-    builder.create<memref::StoreOp>(loc, idx, ind,
-                                    constantIndex(builder, loc, i));
+    Value v = builder.create<memref::LoadOp>(loc, mem,
+                                             constantIndex(builder, loc, i));
+    if (i == offsetIdx && offsetVal)
+      v = builder.create<arith::AddIOp>(loc, v, offsetVal);
+    vs.push_back(v);
+  }
+  return vs;
+}
+
+void sparse_tensor::storeAll(OpBuilder &builder, Location loc, Value mem,
+                             ValueRange vs, size_t offsetIdx, Value offsetVal) {
+#ifndef NDEBUG
+  const size_t vsize = vs.size();
+  const auto memTp = mem.getType().cast<MemRefType>();
+  assert(memTp.getRank() == 1);
+  const DynSize memSh = memTp.getDimSize(0);
+  assert(ShapedType::isDynamic(memSh) || memSh >= static_cast<DynSize>(vsize));
+  assert(offsetIdx == 0 || offsetIdx < vsize);
+#endif // NDEBUG
+  for (const auto &v : llvm::enumerate(vs)) {
+    const Value w =
+        (offsetIdx == v.index() && offsetVal)
+            ? builder.create<arith::AddIOp>(loc, v.value(), offsetVal)
+            : v.value();
+    builder.create<memref::StoreOp>(loc, w, mem,
+                                    constantIndex(builder, loc, v.index()));
   }
 }
 
@@ -620,43 +642,44 @@ Value sparse_tensor::reshapeValuesToLevels(OpBuilder &builder, Location loc,
                                            SparseTensorEncodingAttr enc,
                                            ValueRange dimSizes,
                                            Value valuesBuffer,
-                                           Value idxBuffer) {
-  // Use the `idxBuffer` to store the level sizes.
+                                           Value lvlCoords) {
+  // Reuse the `lvlCoords` buffer to store the level-sizes.
   const Level lvlRank = enc.getLvlRank();
   SmallVector<Value> lvlSizes;
   lvlSizes.reserve(lvlRank);
   for (Level l = 0; l < lvlRank; l++)
     // FIXME: `toOrigDim` is deprecated.
     lvlSizes.push_back(dimSizes[toOrigDim(enc, l)]);
-  storeIndices(builder, loc, lvlRank, idxBuffer, lvlSizes);
+  storeAll(builder, loc, lvlCoords, lvlSizes);
   // The memref ReshapeOp requires the sizes buffer to have a static
   // shape.
   const auto iTp = builder.getIndexType();
-  const SmallVector<DynSize, 1> idxBufferShape{static_cast<DynSize>(lvlRank)};
-  const auto idxBufferTp = MemRefType::get(idxBufferShape, iTp);
-  idxBuffer = builder.create<memref::CastOp>(loc, idxBufferTp, idxBuffer);
+  const SmallVector<DynSize, 1> lvlSizesShape{static_cast<DynSize>(lvlRank)};
+  const auto lvlSizesTp = MemRefType::get(lvlSizesShape, iTp);
+  lvlCoords = builder.create<memref::CastOp>(loc, lvlSizesTp, lvlCoords);
+  // Finally, create the ReshapeOp.
   const SmallVector<DynSize> resShape(lvlRank, ShapedType::kDynamic);
   const Type elemTp = getMemRefType(valuesBuffer).getElementType();
   const auto resTp = MemRefType::get(resShape, elemTp);
-  return builder.create<memref::ReshapeOp>(loc, resTp, valuesBuffer, idxBuffer);
+  return builder.create<memref::ReshapeOp>(loc, resTp, valuesBuffer, lvlCoords);
 }
 
-Value sparse_tensor::genToPointers(OpBuilder &builder, Location loc,
-                                   Value tensor, Level lvl) {
+Value sparse_tensor::genToPositions(OpBuilder &builder, Location loc,
+                                    Value tensor, Level lvl) {
   const auto srcTp = getSparseTensorType(tensor);
-  const Type ptrTp = getPointerOverheadType(builder, srcTp.getEncoding());
-  const Type memTp = get1DMemRefType(ptrTp, /*withLayout=*/false);
-  return builder.create<ToPointersOp>(loc, memTp, tensor,
-                                      builder.getIndexAttr(lvl));
+  const Type posTp = srcTp.getEncoding().getPosType();
+  const Type memTp = get1DMemRefType(posTp, /*withLayout=*/false);
+  return builder.create<ToPositionsOp>(loc, memTp, tensor,
+                                       builder.getIndexAttr(lvl));
 }
 
-Value sparse_tensor::genToIndices(OpBuilder &builder, Location loc,
-                                  Value tensor, Level lvl, Level cooStart) {
+Value sparse_tensor::genToCoordinates(OpBuilder &builder, Location loc,
+                                      Value tensor, Level lvl, Level cooStart) {
   const auto srcTp = getSparseTensorType(tensor);
-  const Type idxTp = getIndexOverheadType(builder, srcTp.getEncoding());
-  const Type memTp = get1DMemRefType(idxTp, /*withLayout=*/lvl >= cooStart);
-  return builder.create<ToIndicesOp>(loc, memTp, tensor,
-                                     builder.getIndexAttr(lvl));
+  const Type crdTp = srcTp.getEncoding().getCrdType();
+  const Type memTp = get1DMemRefType(crdTp, /*withLayout=*/lvl >= cooStart);
+  return builder.create<ToCoordinatesOp>(loc, memTp, tensor,
+                                         builder.getIndexAttr(lvl));
 }
 
 Value sparse_tensor::genToValues(OpBuilder &builder, Location loc,
@@ -671,3 +694,23 @@ Value sparse_tensor::genValMemSize(OpBuilder &builder, Location loc,
                                    Value tensor) {
   return getDescriptorFromTensorTuple(tensor).getValMemSize(builder, loc);
 }
+
+Value sparse_tensor::createOrFoldSliceOffsetOp(OpBuilder &builder, Location loc,
+                                               Value tensor, Dimension dim) {
+  auto enc = getSparseTensorEncoding(tensor.getType());
+  assert(enc && enc.isSlice());
+  std::optional<unsigned> offset = enc.getStaticDimSliceOffset(dim);
+  if (offset.has_value())
+    return constantIndex(builder, loc, *offset);
+  return builder.create<ToSliceOffsetOp>(loc, tensor, APInt(64, dim));
+}
+
+Value sparse_tensor::createOrFoldSliceStrideOp(OpBuilder &builder, Location loc,
+                                               Value tensor, Dimension dim) {
+  auto enc = getSparseTensorEncoding(tensor.getType());
+  assert(enc && enc.isSlice());
+  std::optional<unsigned> stride = enc.getStaticDimSliceStride(dim);
+  if (stride.has_value())
+    return constantIndex(builder, loc, *stride);
+  return builder.create<ToSliceStrideOp>(loc, tensor, APInt(64, dim));
+}
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/CodegenUtils.h b/mlir/lib/Dialect/SparseTensor/Transforms/CodegenUtils.h
index de78010e57f22..6d6351cce6dba 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/CodegenUtils.h
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/CodegenUtils.h
@@ -47,17 +47,11 @@ OverheadType overheadTypeEncoding(Type tp);
 /// Converts the internal type-encoding for overhead storage to an mlir::Type.
 Type getOverheadType(Builder &builder, OverheadType ot);
 
-/// Returns the OverheadType for pointer overhead storage.
-OverheadType pointerOverheadTypeEncoding(SparseTensorEncodingAttr enc);
+/// Returns the OverheadType for position overhead storage.
+OverheadType posTypeEncoding(SparseTensorEncodingAttr enc);
 
-/// Returns the OverheadType for index overhead storage.
-OverheadType indexOverheadTypeEncoding(SparseTensorEncodingAttr enc);
-
-/// Returns the mlir::Type for pointer overhead storage.
-Type getPointerOverheadType(Builder &builder, SparseTensorEncodingAttr enc);
-
-/// Returns the mlir::Type for index overhead storage.
-Type getIndexOverheadType(Builder &builder, SparseTensorEncodingAttr enc);
+/// Returns the OverheadType for coordinate overhead storage.
+OverheadType crdTypeEncoding(SparseTensorEncodingAttr enc);
 
 /// Convert OverheadType to its function-name suffix.
 StringRef overheadTypeFunctionSuffix(OverheadType ot);
@@ -98,15 +92,14 @@ Value genIsNonzero(OpBuilder &builder, Location loc, Value v);
 void genReshapeDstShape(Location loc, PatternRewriter &rewriter,
                         SmallVectorImpl<Value> &dstShape,
                         ArrayRef<Value> srcShape,
-                        ArrayRef<int64_t> staticDstShape,
+                        ArrayRef<StaticSize> staticDstShape,
                         ArrayRef<ReassociationIndices> reassociation);
 
-/// Translate indices during a reshaping operation.
-void translateIndicesArray(OpBuilder &builder, Location loc,
-                           ArrayRef<ReassociationIndices> reassociation,
-                           ValueRange srcIndices, ArrayRef<Value> srcShape,
-                           ArrayRef<Value> dstShape,
-                           SmallVectorImpl<Value> &dstIndices);
+/// Reshape coordinates during a reshaping operation.
+void reshapeCvs(OpBuilder &builder, Location loc,
+                ArrayRef<ReassociationIndices> reassociation,
+                ValueRange srcSizes, ValueRange srcCvs, // NOLINT
+                ValueRange dstSizes, SmallVectorImpl<Value> &dstCvs);
 
 /// Returns a function reference (first hit also inserts into module). Sets
 /// the "_emit_c_interface" on the function declaration when requested,
@@ -155,9 +148,9 @@ Value allocDenseTensor(OpBuilder &builder, Location loc,
 /// Generates code to deallocate a dense buffer.
 void deallocDenseTensor(OpBuilder &builder, Location loc, Value buffer);
 
-/// Generates the code to read the value from tensor[ivs]. The generated code
+/// Generates code to read the value from `tensor[ivs]`. The generated code
 /// looks like the following and the insertion point after this routine is
-/// inside the if-then branch behind the assignment to ind.
+/// inside the then-branch.
 ///    if (tensor[ivs] != 0)
 ///      insert_point
 Value genValueForDense(OpBuilder &builder, Location loc, Value tensor,
@@ -177,7 +170,7 @@ Value genValueForDense(OpBuilder &builder, Location loc, Value tensor,
 // The loop to iterate a sparse tensor constant:
 //   for i in range(NNZ)
 //     val = values[i]
-//     [i1,..,ik] = indices[i]
+//     [i1,..,ik] = coordinates[i]
 //     loop-body
 void genDenseTensorOrSparseConstantIterLoop(
     OpBuilder &builder, Location loc, Value src, unsigned rank,
@@ -201,8 +194,8 @@ inline MemRefType get1DMemRefType(Type etp, bool withLayout) {
 /// Scans to top of generated loop.
 Operation *getTop(Operation *op);
 
-/// Iterate over a sparse constant, generates constantOp for value and indices.
-/// E.g.,
+/// Iterate over a sparse constant, generates constantOp for value
+/// and coordinates.  E.g.,
 /// sparse<[ [0], [28], [31] ],
 ///          [ (-5.13, 2.0), (3.0, 4.0), (5.0, 6.0) ] >
 /// =>
@@ -221,18 +214,27 @@ void foreachInSparseConstant(
     Location loc, RewriterBase &rewriter, SparseElementsAttr attr,
     AffineMap order, function_ref<void(ArrayRef<Value>, Value)> callback);
 
-/// Converts the vector indices and store it into the memory pointed by
-/// `ind`, apply (optional) `offset` on `offsetDim`.
-void storeIndices(OpBuilder &builder, Location loc, unsigned rank, Value ind,
-                  ValueRange ivs, unsigned offsetDim = 0,
-                  Value offset = Value());
+/// Loads `size`-many values from the memref, which must have rank-1 and
+/// size greater-or-equal to `size`.  If the optional `(offsetIdx,offsetVal)`
+/// arguments are provided, then the `offsetVal` will be added to the
+/// `offsetIdx`-th value after loading.
+SmallVector<Value> loadAll(OpBuilder &builder, Location loc, size_t size,
+                           Value mem, size_t offsetIdx = 0,
+                           Value offsetVal = Value());
+
+/// Stores all the values of `vs` into the memref `mem`, which must have
+/// rank-1 and size greater-or-equal to `vs.size()`.  If the optional
+/// `(offsetIdx,offsetVal)` arguments are provided, then the `offsetVal`
+/// will be added to the `offsetIdx`-th value before storing.
+void storeAll(OpBuilder &builder, Location loc, Value mem, ValueRange vs,
+              size_t offsetIdx = 0, Value offsetVal = Value());
 
 /// Reshapes the linear values buffer for an annotated all dense sparse tensor
 /// to match the shape of the corresponding dense tensor to support direct
-/// access of the buffer through indices.
+/// access of the buffer through `lvlCoords`.
 Value reshapeValuesToLevels(OpBuilder &builder, Location loc,
                             SparseTensorEncodingAttr enc, ValueRange dimSizes,
-                            Value valuesBuffer, Value idxBuffer);
+                            Value valuesBuffer, Value lvlCoords);
 
 //===----------------------------------------------------------------------===//
 // Inlined constant generators.
@@ -248,7 +250,7 @@ Value reshapeValuesToLevels(OpBuilder &builder, Location loc,
 //===----------------------------------------------------------------------===//
 
 /// Generates a 0-valued constant of the given type.  In addition to
-/// the scalar types (`ComplexType`, ``FloatType`, `IndexType`,
+/// the scalar types (`ComplexType`, `FloatType`, `IndexType`,
 /// `IntegerType`), this also works for `RankedTensorType` and `VectorType`
 /// (for which it generates a constant `DenseElementsAttr` of zeros).
 inline Value constantZero(OpBuilder &builder, Location loc, Type tp) {
@@ -314,18 +316,18 @@ inline Value constantOverheadTypeEncoding(OpBuilder &builder, Location loc,
                      static_cast<uint32_t>(overheadTypeEncoding(width)));
 }
 
-/// Generates a constant of the internal type-encoding for pointer
+/// Generates a constant of the internal type-encoding for position
 /// overhead storage.
-inline Value constantPointerTypeEncoding(OpBuilder &builder, Location loc,
-                                         SparseTensorEncodingAttr enc) {
-  return constantOverheadTypeEncoding(builder, loc, enc.getPointerBitWidth());
+inline Value constantPosTypeEncoding(OpBuilder &builder, Location loc,
+                                     SparseTensorEncodingAttr enc) {
+  return constantOverheadTypeEncoding(builder, loc, enc.getPosWidth());
 }
 
-/// Generates a constant of the internal type-encoding for index overhead
-/// storage.
-inline Value constantIndexTypeEncoding(OpBuilder &builder, Location loc,
-                                       SparseTensorEncodingAttr enc) {
-  return constantOverheadTypeEncoding(builder, loc, enc.getIndexBitWidth());
+/// Generates a constant of the internal type-encoding for coordinate
+/// overhead storage.
+inline Value constantCrdTypeEncoding(OpBuilder &builder, Location loc,
+                                     SparseTensorEncodingAttr enc) {
+  return constantOverheadTypeEncoding(builder, loc, enc.getCrdWidth());
 }
 
 /// Generates a constant of the internal type-encoding for primary storage.
@@ -346,21 +348,31 @@ inline bool isZeroRankedTensorOrScalar(Type type) {
   return !rtp || rtp.getRank() == 0;
 }
 
-/// Infers the result type and generates ToPointersOp.
-Value genToPointers(OpBuilder &builder, Location loc, Value tensor, Level lvl);
+/// Infers the result type and generates `ToPositionsOp`.
+Value genToPositions(OpBuilder &builder, Location loc, Value tensor, Level lvl);
 
-/// Infers the result type and generates ToIndicesOp. If the lvl is within a COO
-/// region, the result type is a memref with unknown stride and offset.
-/// Otherwise, the result type is a memref without any specified layout.
-Value genToIndices(OpBuilder &builder, Location loc, Value tensor, Level lvl,
-                   Level cooStart);
+/// Infers the result type and generates `ToCoordinatesOp`.  If the
+/// level is within a COO region, the result type is a memref with unknown
+/// stride and offset.  Otherwise, the result type is a memref without
+/// any specified layout.
+Value genToCoordinates(OpBuilder &builder, Location loc, Value tensor,
+                       Level lvl, Level cooStart);
 
-/// Infers the result type and generates ToValuesOp.
+/// Infers the result type and generates `ToValuesOp`.
 Value genToValues(OpBuilder &builder, Location loc, Value tensor);
 
 /// Generates code to retrieve the values size for the sparse tensor.
 Value genValMemSize(OpBuilder &builder, Location loc, Value tensor);
 
+/// Generates code to retrieve the slice offset for the sparse tensor slice,
+/// return a constant if the offset is statically known.
+Value createOrFoldSliceOffsetOp(OpBuilder &builder, Location loc, Value tensor,
+                                Dimension dim);
+
+/// Generates code to retrieve the slice slice for the sparse tensor slice,
+/// return a constant if the offset is statically known.
+Value createOrFoldSliceStrideOp(OpBuilder &builder, Location loc, Value tensor,
+                                Dimension dim);
 } // namespace sparse_tensor
 } // namespace mlir
 
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/LoopEmitter.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/LoopEmitter.cpp
index 28f68622e3e95..c3823c0f204d9 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/LoopEmitter.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/LoopEmitter.cpp
@@ -15,6 +15,7 @@
 #include "mlir/Dialect/Linalg/Utils/Utils.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/SparseTensor/IR/SparseTensorType.h"
 #include "mlir/Dialect/Tensor/IR/Tensor.h"
 
 using namespace mlir;
@@ -24,16 +25,16 @@ using namespace mlir::sparse_tensor;
 // File local helper functions.
 //===----------------------------------------------------------------------===//
 
-/// Generates a pointer/index load from the sparse storage scheme. Narrower
-/// data types need to be zero extended before casting the value into the
-/// index type used for looping and indexing.
-static Value genIndexLoad(OpBuilder &builder, Location loc, Value ptr,
+/// Generates a position/coordinate load from the sparse storage scheme.
+/// Narrower data types need to be zero extended before casting the
+/// value into the `Index` type used for looping and indexing.
+static Value genIndexLoad(OpBuilder &builder, Location loc, Value mem,
                           Value s) {
   // For the scalar case, we simply zero extend narrower indices into 64-bit
   // values before casting to index without a performance penalty. Here too,
   // however, indices that already are 64-bit, in theory, cannot express the
   // full range as explained above.
-  Value load = builder.create<memref::LoadOp>(loc, ptr, s);
+  Value load = builder.create<memref::LoadOp>(loc, mem, s);
   if (!load.getType().isa<IndexType>()) {
     if (load.getType().getIntOrFloatBitWidth() < 64)
       load = builder.create<arith::ExtUIOp>(loc, builder.getI64Type(), load);
@@ -43,136 +44,207 @@ static Value genIndexLoad(OpBuilder &builder, Location loc, Value ptr,
   return load;
 }
 
-// TODO: Support dynamic sized slice.
-static Value getSliceOffset(OpBuilder &builder, Location loc,
-                            SparseTensorEncodingAttr enc, unsigned lvl) {
-  return constantIndex(builder, loc, *enc.getStaticLvlSliceOffset(lvl));
+static Value genSliceOffset(OpBuilder &builder, Location loc, Value tensor,
+                            Level lvl) {
+  auto enc = getSparseTensorEncoding(tensor.getType());
+  // FIXME: `toOrigDim` is deprecated
+  return createOrFoldSliceOffsetOp(builder, loc, tensor, toOrigDim(enc, lvl));
 }
 
-static Value getSliceSize(OpBuilder &builder, Location loc,
-                          SparseTensorEncodingAttr enc, unsigned lvl) {
-  return constantIndex(builder, loc, *enc.getStaticLvlSliceSize(lvl));
+static Value genSliceStride(OpBuilder &builder, Location loc, Value tensor,
+                            Level lvl) {
+  auto enc = getSparseTensorEncoding(tensor.getType());
+  // FIXME: `toOrigDim` is deprecated
+  return createOrFoldSliceStrideOp(builder, loc, tensor, toOrigDim(enc, lvl));
 }
 
-static Value getSliceStride(OpBuilder &builder, Location loc,
-                            SparseTensorEncodingAttr enc, unsigned lvl) {
-  return constantIndex(builder, loc, *enc.getStaticLvlSliceStride(lvl));
+/// Converts a coordinate relative to the slice to the coordinate relative
+/// to the underlying tensor.
+// FIXME: that description says "sliceCrd -> tensorCrd"; but the function
+// name suggests it should be "tensorCrd -> sliceCrd".
+static Value toSliceCrd(OpBuilder &builder, Location loc, Value crd,
+                        Value offset, Value stride, Value tensor, Level lvl) {
+  // tensorCrd = sliceCrd * stride + offset
+  crd = builder.create<arith::MulIOp>(loc, crd, stride);
+  crd = builder.create<arith::AddIOp>(loc, crd, offset);
+  return crd;
 }
 
-// Converts a coordinate relative to the slice to the coordinate relative
-// to the underlying tensor.
-static Value toSliceCoord(OpBuilder &builder, Location loc, Value v,
-                          SparseTensorEncodingAttr enc, unsigned lvl) {
-
-  Value stride = getSliceStride(builder, loc, enc, lvl);
-  Value offset = getSliceOffset(builder, loc, enc, lvl);
-  // iv = iv * stride + offset
-  v = builder.create<arith::MulIOp>(loc, v, stride);
-  v = builder.create<arith::AddIOp>(loc, v, offset);
-  return v;
+/// Converts a coordinate relative to the underlying tensor to the coordinate
+/// relative to the slice, returns a extra reminder value
+// FIXME: that description says "tensorCrd -> sliceCrd"; but the function
+// name suggests it should be "sliceCrd -> tensorCrd".
+static std::pair<Value, Value> fromSliceCrd(OpBuilder &builder, Location loc,
+                                            Value crd, Value offset,
+                                            Value stride, Value tensor,
+                                            Level lvl) {
+  // sliceCrd = (tensorCrd - offset) / stride
+  crd = builder.create<arith::SubIOp>(loc, crd, offset);
+  Value rem = builder.create<arith::RemUIOp>(loc, crd, stride);
+  crd = builder.create<arith::DivUIOp>(loc, crd, stride);
+  return std::make_pair(crd, rem);
 }
 
-// Converts a coordinate relative to the underlying tensor to the coordinate
-// relative to the slice, returns a extra reminder value
-static std::pair<Value, Value> fromSliceCoord(OpBuilder &builder, Location loc,
-                                              Value v,
-                                              SparseTensorEncodingAttr enc,
-                                              unsigned lvl) {
-  Value stride = getSliceStride(builder, loc, enc, lvl);
-  Value offset = getSliceOffset(builder, loc, enc, lvl);
-  // iv = (iv - offset) / stride
-  v = builder.create<arith::SubIOp>(loc, v, offset);
-  Value rem = builder.create<arith::RemUIOp>(loc, v, stride);
-  v = builder.create<arith::DivUIOp>(loc, v, stride);
-  return std::make_pair(v, rem);
-}
+std::pair<Value, Value>
+LoopEmitter::genSliceLegitPredicate(OpBuilder &builder, Location loc, Value crd,
+                                    TensorId tid, Level lvl) {
+  assert(isSparseSlices[tid]);
+  Value slice = tensors[tid];
+  Value offset = sliceOffsets[tid][lvl];
+  Value stride = sliceStrides[tid][lvl];
+  auto enc = getSparseTensorEncoding(slice.getType());
+
+  const auto [newCrd, crdRem] =
+      fromSliceCrd(builder, loc, crd, offset, stride, slice, lvl);
+
+  SmallVector<Value, 3> conds; // at most 3 conditions
+
+  // First, coord >= offset (skip the check if offset is known to be 0).
+  if (auto staticOffset = enc.getStaticLvlSliceOffset(lvl);
+      !(staticOffset.has_value() && *staticOffset == 0)) {
+    auto geOffset = builder.create<arith::CmpIOp>(
+        loc, arith::CmpIPredicate::uge, crd, offset);
+    conds.push_back(geOffset);
+  }
 
-static std::pair<Value, Value>
-genSliceLegitPredicate(OpBuilder &builder, Location loc, Value coord,
-                       SparseTensorEncodingAttr enc, unsigned lvl) {
-  std::pair<Value, Value> trans = fromSliceCoord(builder, loc, coord, enc, lvl);
-  // First, coord >= offset (TODO: seems unsigned >= 0 won't be folded, skip
-  // the check if the offset is zero).
-  auto geOffset =
-      builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::uge, coord,
-                                    getSliceOffset(builder, loc, enc, lvl));
   // Second, coord_in_slice < length
-  auto ltLength =
-      builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ult, trans.first,
-                                    getSliceSize(builder, loc, enc, lvl));
-
-  // Third, rem == 0; confirmed that (a % 1) will be folded to 0
-  auto fitStride =
-      builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::eq, trans.second,
-                                    constantIndex(builder, loc, 0));
-
-  auto pred = builder.create<arith::AndIOp>(loc, geOffset, ltLength);
-  pred = builder.create<arith::AndIOp>(loc, pred, fitStride);
-  return {trans.first, pred};
+  auto ltLength = builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ult,
+                                                newCrd, lvlSizes[tid][lvl]);
+  conds.push_back(ltLength);
+
+  // Third, rem == 0 (skip the check if stride is known to be 1).
+  if (auto staticStride = enc.getStaticLvlSliceStride(lvl);
+      !(staticStride.has_value() && *staticStride == 1)) {
+    auto fitStride = builder.create<arith::CmpIOp>(
+        loc, arith::CmpIPredicate::eq, crdRem, constantIndex(builder, loc, 0));
+    conds.push_back(fitStride);
+  }
+
+  // Must meet all condition to be a valid coordinate in slice.
+  auto pred = conds.front();
+  for (auto cond : ValueRange(conds).drop_front())
+    pred = builder.create<arith::AndIOp>(loc, pred, cond);
+
+  return {newCrd, pred};
 }
 
 //===----------------------------------------------------------------------===//
 // Sparse tensor loop emitter class implementations
 //===----------------------------------------------------------------------===//
 
-Value LoopEmitter::genAddress(OpBuilder &builder, Location loc, size_t tid,
-                              size_t dim, Value iv) {
-  Value p = dim == 0 ? constantIndex(builder, loc, 0) : pidxs[tid][dim - 1];
-  Value mul = builder.create<arith::MulIOp>(loc, highs[tid][dim], p);
-  if (isSparseSlices[tid]) {
-    auto enc = getSparseTensorEncoding(tensors[tid].getType());
-    iv = toSliceCoord(builder, loc, iv, enc, dim);
-  }
-  Value add = builder.create<arith::AddIOp>(loc, mul, iv);
+Value LoopEmitter::genAddress(OpBuilder &builder, Location loc, TensorId tid,
+                              Level lvl, Value crd) {
+  Value pos = lvl == 0 ? constantIndex(builder, loc, 0) : posits[tid][lvl - 1];
+  Value mul = builder.create<arith::MulIOp>(loc, highs[tid][lvl], pos);
+  if (isSparseSlices[tid])
+    crd = toSliceCrd(builder, loc, crd, sliceOffsets[tid][lvl],
+                     sliceStrides[tid][lvl], tensors[tid], lvl);
+  Value add = builder.create<arith::AddIOp>(loc, mul, crd);
   return add;
 }
 
-Value LoopEmitter::genSparseCoord(OpBuilder &builder, Location loc, size_t tid,
-                                  size_t l) {
-  Value c = constantIndex(builder, loc, 0);
-  auto reass = getCollapseReassociation(tid, l);
-  for (unsigned i = 0; i < reass.size(); i++) {
-    auto lvl = reass[i];
-    // A load on the indices array yields the coordinate.
-    Value ptr = idxBuffer[tid][lvl];
-    Value off = genIndexLoad(builder, loc, ptr, pidxs[tid][l]);
+Value LoopEmitter::genSegmentHigh(OpBuilder &builder, Location loc,
+                                  TensorId tid, Level lvl, Value pLo,
+                                  Value pHi) {
+  const auto coordinates = coordinatesBuffers[tid][lvl];
+  const auto sameCrd = genIndexLoad(builder, loc, coordinates, pLo);
+  auto whileOp = builder.create<scf::WhileOp>(
+      loc, builder.getIndexType(), pLo,
+      /*beforeBuilder=*/
+      [pHi, coordinates, sameCrd](OpBuilder &builder, Location loc,
+                                  ValueRange ivs) {
+        const auto pos = ivs[0];
+        Value inBound = builder.create<arith::CmpIOp>(
+            loc, arith::CmpIPredicate::ult, pos, pHi);
+        auto ifInBound =
+            builder.create<scf::IfOp>(loc, builder.getI1Type(), inBound, true);
+        {
+          OpBuilder::InsertionGuard guard(builder);
+          // Load the next coordinates only when inbound (to avoid OOB
+          // acccesses).
+          builder.setInsertionPointToStart(ifInBound.thenBlock());
+          Value crd = genIndexLoad(builder, loc, coordinates, pos);
+          Value isSameCrd = builder.create<arith::CmpIOp>(
+              loc, arith::CmpIPredicate::eq, crd, sameCrd);
+          builder.create<scf::YieldOp>(loc, isSameCrd);
+          // Else, the position is out of bound, yield false to terminate the
+          // loop.
+          builder.setInsertionPointToStart(ifInBound.elseBlock());
+          builder.create<scf::YieldOp>(loc, constantI1(builder, loc, false));
+        }
+        builder.create<scf::ConditionOp>(loc, ifInBound.getResults()[0], ivs);
+      },
+      /*afterBuilder=*/
+      [](OpBuilder &builder, Location loc, ValueRange ivs) {
+        // pos ++
+        Value nextPos = builder.create<arith::AddIOp>(
+            loc, ivs[0], constantIndex(builder, loc, 1));
+        builder.create<scf::YieldOp>(loc, nextPos);
+      });
+  // Return the segment high.
+  return whileOp.getResult(0);
+}
+
+Value LoopEmitter::genSparseCrd(OpBuilder &builder, Location loc, TensorId tid,
+                                Level dstLvl) {
+  Value crd = constantIndex(builder, loc, 0);
+  const auto reassoc = getCollapseReassociation(tid, dstLvl);
+  const unsigned reassocSize = reassoc.size();
+  for (unsigned i = 0; i < reassocSize; i++) {
+    const Level srcLvl = reassoc[i];
+    // A load on the coordinates array yields the coordinate.
+    const Value mem = coordinatesBuffers[tid][srcLvl];
+    /// FIXME: See the [CLARIFY_POSITS_LVL] note in the header.
+    const Value pos = posits[tid][dstLvl];
+    const Value off = genIndexLoad(builder, loc, mem, pos);
     // Linearized the coordinates within the same collapse reassociation.
-    c = builder.create<arith::AddIOp>(loc, c, off);
-    if (i != reass.size() - 1) {
-      c = builder.create<arith::MulIOp>(loc, c,
-                                        this->lvlSizes[tid][reass[i + 1]]);
+    crd = builder.create<arith::AddIOp>(loc, crd, off);
+    if (i != reassocSize - 1) {
+      crd = builder.create<arith::MulIOp>(loc, crd,
+                                          this->lvlSizes[tid][reassoc[i + 1]]);
     }
   }
-  return c;
+  return crd;
 }
 
 LoopEmitter::LoopEmitter(ValueRange tensors, StringAttr loopTag, bool hasOutput,
-                         bool isSparseOut, ArrayRef<unsigned> topSort) {
+                         bool isSparseOut, ArrayRef<LoopId> topSort) {
   initialize(tensors, loopTag, hasOutput, isSparseOut, topSort);
 }
 
 void LoopEmitter::initialize(ValueRange ts, StringAttr loopTag, bool hasOutput,
-                             bool isSparseOut, ArrayRef<unsigned> topSort) {
-  // First initializes fields.
+                             bool isSparseOut, ArrayRef<LoopId> topSort) {
+  // First initialize the top-level type of the fields.
   this->loopTag = loopTag;
   this->hasOutput = hasOutput;
   this->isSparseOut = isSparseOut;
+
+  const TensorId numTensors = ts.size();
   this->tensors.assign(ts.begin(), ts.end());
-  this->isSparseSlices.assign(tensors.size(), false);
-  this->dimTypes.assign(tensors.size(), std::vector<DimLevelType>());
-  this->pidxs.assign(tensors.size(), std::vector<Value>());
-  this->coord.assign(tensors.size(), std::vector<Value>());
-  this->highs.assign(tensors.size(), std::vector<Value>());
-  this->lvlSizes.assign(tensors.size(), std::vector<Value>());
-  this->ptrBuffer.assign(tensors.size(), std::vector<Value>());
-  this->idxBuffer.assign(tensors.size(), std::vector<Value>());
-  this->valBuffer.assign(tensors.size(), nullptr);
-  this->loopStack.reserve(topSort.size());
-  this->sparsiferLoopLvlMap.assign(topSort.size(), 0);
-  this->collapseReassoc.assign(tensors.size(), nullptr);
-
-  for (size_t tid = 0, e = tensors.size(); tid < e; tid++) {
-    auto t = tensors[tid];
+  this->lvlTypes.assign(numTensors, std::vector<DimLevelType>());
+  this->lvlSizes.assign(numTensors, std::vector<Value>());
+  this->highs.assign(numTensors, std::vector<Value>());
+  this->segHi.assign(numTensors, std::vector<Value>());
+  this->posits.assign(numTensors, std::vector<Value>());
+  this->coords.assign(numTensors, std::vector<Value>());
+  this->positionsBuffers.assign(numTensors, std::vector<Value>());
+  this->coordinatesBuffers.assign(numTensors, std::vector<Value>());
+  this->valBuffer.assign(numTensors, nullptr);
+  this->collapseReassoc.assign(numTensors, nullptr);
+  this->isSparseSlices.assign(numTensors, false);
+  this->sliceOffsets.assign(numTensors, std::vector<Value>());
+  this->sliceStrides.assign(numTensors, std::vector<Value>());
+
+  const LoopOrd numLoops = topSort.size();
+  // These zeros will be overwritten below, but we need to initialize
+  // them to something since we'll need random-access assignment.
+  this->loopIdToOrd.assign(numLoops, 0);
+  this->loopStack.reserve(numLoops);
+  this->loopSeqStack.reserve(numLoops);
+
+  // Initialize nested types of `TensorId`-indexed fields.
+  for (TensorId tid = 0; tid < numTensors; tid++) {
+    const Value t = tensors[tid];
     // a scalar or 0-dimension tensors
     if (isZeroRankedTensorOrScalar(t.getType()))
       continue;
@@ -186,43 +258,51 @@ void LoopEmitter::initialize(ValueRange ts, StringAttr loopTag, bool hasOutput,
       collapseReassoc[tid] = reshape.getReassociation();
       rtp = reshape.getSrcType();
       // Overwrites the tensor to the source tensor of reshape operations.
-      tensors[tid] = t = reshape.getSrc();
+      tensors[tid] = reshape.getSrc();
     }
-    auto rank = static_cast<size_t>(rtp.getRank());
-    auto enc = getSparseTensorEncoding(rtp);
+    const SparseTensorType stt(rtp);
+    const Level lvlRank = stt.getLvlRank();
     // We always treat sparse output tensor as dense so that we always iterate
-    // it based on dim size.
-    if (enc && !(isOutputTensor(tid) && isSparseOut)) {
+    // it based on lvl size.
+    if (stt.hasEncoding() && !(isOutputTensor(tid) && isSparseOut)) {
+      const auto enc = stt.getEncoding();
       isSparseSlices[tid] = enc.isSlice();
-      for (auto dimTp : enc.getDimLevelType())
-        dimTypes[tid].push_back(dimTp);
-    } else
-      dimTypes[tid].assign(rank, DimLevelType::Dense);
+      for (auto lvlTp : enc.getDimLevelType())
+        lvlTypes[tid].push_back(lvlTp);
+    } else {
+      lvlTypes[tid].assign(lvlRank, DimLevelType::Dense);
+    }
 
     // Initialize using empty value.
-    pidxs[tid].assign(rank, Value());
-    coord[tid].assign(rank, Value());
-    highs[tid].assign(rank, Value());
-    lvlSizes[tid].assign(rank, Value());
-    ptrBuffer[tid].assign(rank, Value());
-    idxBuffer[tid].assign(rank, Value());
+    lvlSizes[tid].assign(lvlRank, Value());
+    highs[tid].assign(lvlRank, Value());
+    segHi[tid].assign(lvlRank, Value());
+    posits[tid].assign(lvlRank, Value());
+    coords[tid].assign(lvlRank, Value());
+    positionsBuffers[tid].assign(lvlRank, Value());
+    coordinatesBuffers[tid].assign(lvlRank, Value());
+    sliceOffsets[tid].assign(lvlRank, Value());
+    sliceStrides[tid].assign(lvlRank, Value());
   }
 
+  // Construct the inverse of the `topSort` from the sparsifier.
+  // This is needed to map `AffineDimExpr`s back to the `LoopOrd`
+  // used in loop emitter.
   // FIXME: This map should be maintained outside loop emitter.
-  for (unsigned i = 0, e = topSort.size(); i < e; i++) {
-    // This is an inverse map of the topologically sorted loop index from
-    // sparsifier. This is needed to map the AffineDimExpr back to the loopStack
-    // index used in loop emitter.
-    sparsiferLoopLvlMap[topSort[i]] = i;
-  }
+  for (LoopOrd n = 0; n < numLoops; n++)
+    loopIdToOrd[topSort[n]] = n;
 }
 
 void LoopEmitter::initializeLoopEmit(OpBuilder &builder, Location loc,
                                      LoopEmitter::OutputUpdater updater) {
-  // For every tensor, find lower and upper bound on dimensions, set the
-  // same bounds on loop indices, and obtain dense or sparse buffer(s).
-  for (size_t t = 0, e = tensors.size(); t < e; t++) {
-    const auto tensor = tensors[t];
+  // For every tensor:
+  // * get the values buffer.
+  // * For every level:
+  //   * get the positions and coordinates buffers
+  //   * get/compute the level-size, which is also used as the upper-bound
+  //     on positions.
+  for (TensorId t = 0, numTensors = getNumTensors(); t < numTensors; t++) {
+    const Value tensor = tensors[t];
     const auto rtp = tensor.getType().dyn_cast<RankedTensorType>();
     if (!rtp)
       // Skips only scalar, zero ranked tensor still need to be bufferized and
@@ -238,26 +318,35 @@ void LoopEmitter::initializeLoopEmit(OpBuilder &builder, Location loc,
     // Scan all levels of current tensor.
     for (Level l = 0; l < lvlRank; l++) {
       // This should be called only once at beginning.
-      assert(!ptrBuffer[t][l] && !idxBuffer[t][l] && !highs[t][l]);
-      const auto dlt = dimTypes[t][l];
+      assert(!positionsBuffers[t][l] && !coordinatesBuffers[t][l] &&
+             !highs[t][l]);
+      const auto lvlTp = lvlTypes[t][l];
       // Handle sparse storage schemes.
-      if (isCompressedDLT(dlt)) {
-        // Generate sparse primitives to obtains pointer and indices.
-        ptrBuffer[t][l] = genToPointers(builder, loc, tensor, l);
-        idxBuffer[t][l] = genToIndices(builder, loc, tensor, l, cooStart);
-      } else if (isSingletonDLT(dlt)) {
-        // Singleton dimension, fetch indices.
-        idxBuffer[t][l] = genToIndices(builder, loc, tensor, l, cooStart);
+      if (isCompressedDLT(lvlTp)) {
+        // Generate sparse primitives to obtain positions and coordinates.
+        positionsBuffers[t][l] = genToPositions(builder, loc, tensor, l);
+        coordinatesBuffers[t][l] =
+            genToCoordinates(builder, loc, tensor, l, cooStart);
+      } else if (isSingletonDLT(lvlTp)) {
+        // Singleton level, fetch coordinates.
+        coordinatesBuffers[t][l] =
+            genToCoordinates(builder, loc, tensor, l, cooStart);
       } else {
-        // Dense dimension, nothing to fetch.
-        assert(isDenseDLT(dlt));
+        // Dense level, nothing to fetch.
+        assert(isDenseDLT(lvlTp));
       }
 
+      // FIXME: `toOrigDim` is deprecated.  For now this relies on the
+      // 1:1 mapping between levels and dimensions, since nowhere else
+      // in the code supports HigherOrdering yet either.
+      Value lvlSz = mlir::linalg::createOrFoldDimOp(builder, loc, tensor,
+                                                    toOrigDim(enc, l));
       // Find upper bound in current dimension.
-      // FIXME: `toOrigDim` is deprecated
-      const Dimension d = toOrigDim(enc, l);
-      lvlSizes[t][l] = highs[t][l] =
-          mlir::linalg::createOrFoldDimOp(builder, loc, tensor, d);
+      highs[t][l] = lvlSizes[t][l] = lvlSz;
+      if (isSparseSlices[t]) {
+        sliceOffsets[t][l] = genSliceOffset(builder, loc, tensors[t], l);
+        sliceStrides[t][l] = genSliceStride(builder, loc, tensors[t], l);
+      }
     }
 
     // Perform the required bufferization. Dense inputs materialize
@@ -285,43 +374,49 @@ void LoopEmitter::initializeLoopEmit(OpBuilder &builder, Location loc,
       valBuffer[t] = denseVal;
     } else {
       // Annotated sparse tensors.
-      // We also need the value buffer for annotated all dense `sparse` tensor.
+      // We also need the value buffer for all-dense annotated "sparse" tensors.
       valBuffer[t] = genToValues(builder, loc, tensor);
     }
-    // NOTE: we can also prepare for 0 dim here in advance, this will hosit
+    // NOTE: we can also prepare for 0 lvl here in advance, this will hoist
     // some loop preparation from tensor iteration, but will also (undesirably)
-    // hosit the code ouside if conditions.
+    // hoist the code ouside if-conditions.
   }
 }
 
 void LoopEmitter::enterNewLoopSeq(OpBuilder &builder, Location loc,
-                                  ArrayRef<size_t> tids,
-                                  ArrayRef<size_t> dims) {
+                                  ArrayRef<TensorId> tids,
+                                  ArrayRef<Level> lvls) {
+  // TODO: sort
   assert(loopSeqStack.size() == loopStack.size());
   // Universal Index starts from 0.
   loopSeqStack.emplace_back(constantIndex(builder, loc, 0));
   // Prepares for all the tensors used in the current loop sequence.
-  for (auto [tid, dim] : llvm::zip(tids, dims))
-    prepareLoopOverTensorAtDim(builder, loc, tid, dim);
+  assert(tids.size() == lvls.size());
+  for (auto [tid, lvl] : llvm::zip(tids, lvls))
+    prepareLoopOverTensorAtLvl(builder, loc, tid, lvl);
 }
 
-Value LoopEmitter::genAffine(OpBuilder &builder, AffineExpr a, Location loc) {
+Value LoopEmitter::genAffine(OpBuilder &builder, Location loc, AffineExpr a) {
   switch (a.getKind()) {
   case AffineExprKind::DimId: {
-    unsigned idx = a.cast<AffineDimExpr>().getPosition();
-    return loopStack[sparsiferLoopLvlMap[idx]].iv;
+    // FIXME: since the one callsite in Sparsification passes in a
+    // level-expression, the `getPosition` must in fact be a `Dimension`.
+    // However, elsewhere we have been lead to expect that `loopIdToOrd`
+    // should be indexed by `LoopId`...
+    const LoopId i = a.cast<AffineDimExpr>().getPosition();
+    return loopStack[loopIdToOrd[i]].iv;
   }
   case AffineExprKind::Add: {
     auto binOp = a.cast<AffineBinaryOpExpr>();
     return builder.create<arith::AddIOp>(
-        loc, genAffine(builder, binOp.getLHS(), loc),
-        genAffine(builder, binOp.getRHS(), loc));
+        loc, genAffine(builder, loc, binOp.getLHS()),
+        genAffine(builder, loc, binOp.getRHS()));
   }
   case AffineExprKind::Mul: {
     auto binOp = a.cast<AffineBinaryOpExpr>();
     return builder.create<arith::MulIOp>(
-        loc, genAffine(builder, binOp.getLHS(), loc),
-        genAffine(builder, binOp.getRHS(), loc));
+        loc, genAffine(builder, loc, binOp.getLHS()),
+        genAffine(builder, loc, binOp.getRHS()));
   }
   case AffineExprKind::Constant: {
     int64_t c = a.cast<AffineConstantExpr>().getValue();
@@ -332,39 +427,44 @@ Value LoopEmitter::genAffine(OpBuilder &builder, AffineExpr a, Location loc) {
   }
 }
 
-Operation *LoopEmitter::enterLoopOverTensorAtDim(
-    OpBuilder &builder, Location loc, ArrayRef<size_t> tids,
-    ArrayRef<size_t> dims, MutableArrayRef<Value> reduc, bool isParallel) {
+Operation *LoopEmitter::enterLoopOverTensorAtLvl(
+    OpBuilder &builder, Location loc, ArrayRef<TensorId> tids,
+    ArrayRef<Level> lvls, MutableArrayRef<Value> reduc, bool isParallel) {
   // TODO: support multiple return on parallel for?
   assert(!isParallel || reduc.size() <= 1);
   bool isSparseInput = false;
-  size_t tid = tids.front(), dim = dims.front();
-  for (auto [t, d] : llvm::zip(tids, dims)) {
-    assert(dimTypes[t].size() > d); // Must be a valid tid, dim pair
-    assert(!coord[t][d]);           // We cannot re-enter the same level
-    auto dimType = dimTypes[t][d];
-    // Must be a recognizable DLT.
-    assert(isDenseDLT(dimType) || isCompressedDLT(dimType) ||
-           isSingletonDLT(dimType));
-    bool isSparse = isCompressedDLT(dimType) || isSingletonDLT(dimType);
+  TensorId tid = tids.front();
+  Level dstLvl = lvls.front();
+  assert(tids.size() == lvls.size());
+  for (auto [t, l] : llvm::zip(tids, lvls)) {
+    // TODO: this check for validity of the (t,l) pairs should be
+    // checked/enforced at the callsites, if possible.
+    assert(t < lvlTypes.size() && l < lvlTypes[t].size());
+    assert(!coords[t][l]); // We cannot re-enter the same level
+    const auto lvlTp = lvlTypes[t][l];
+    const bool isSparse = isCompressedDLT(lvlTp) || isSingletonDLT(lvlTp);
+    // Must be a recognizable level-type.
+    assert(isSparse || isDenseDLT(lvlTp));
     // We can at most have one sparse input, otherwise, a while loop is required
     // to co-iterate multiple sparse tensors.
     assert(!isSparseInput || !isSparse);
     if (isSparse) {
       tid = t;
-      dim = d;
+      dstLvl = l;
     }
     isSparseInput = isSparseInput || isSparse;
   }
 
-  auto enc = getSparseTensorEncoding(tensors[tid].getType());
-  auto reass = getCollapseReassociation(tid, dim);
-  dim = reass.front();
+  const auto reassoc = getCollapseReassociation(tid, dstLvl);
   // TODO: support dynamic slices.
-  Value step = constantIndex(builder, loc, 1);
-  Value lo = isSparseInput ? pidxs[tid][dim]      // current offset
-                           : loopSeqStack.back(); // universal index
-  Value hi = highs[tid][dim];
+  // Use the first source-level here to build the loop bound (which is
+  // also the biggest range).
+  const Level srcLvl = reassoc.front();
+  const Value step = constantIndex(builder, loc, 1);
+  /// FIXME: See the [CLARIFY_POSITS_LVL] note in the header.
+  const Value lo = isSparseInput ? posits[tid][srcLvl]  // current position
+                                 : loopSeqStack.back(); // universal index
+  const Value hi = highs[tid][srcLvl];
 
   Operation *loop = nullptr;
   Value iv;
@@ -378,7 +478,7 @@ Operation *LoopEmitter::enterLoopOverTensorAtDim(
 
     // In-place update on the reduction variable vector.
     // Note that the init vals is not the actual reduction variables but instead
-    // used as a `special handle` to (temporarily) represent them. The
+    // used as a "special handle" to (temporarily) represent them. The
     // expression on init vals will be moved into scf.reduce and replaced with
     // the block arguments when exiting the loop (see exitForLoop). This is
     // needed as we can not build the actual reduction block and get the actual
@@ -399,15 +499,17 @@ Operation *LoopEmitter::enterLoopOverTensorAtDim(
   }
   assert(loop && iv);
 
-  Value c;
+  Value crd;
   if (isSparseInput) {
-    assert(reass.size() == 1 || isUniqueCOOType(tensors[tid].getType()));
+    assert(reassoc.size() == 1 || isUniqueCOOType(tensors[tid].getType()));
     // For COO, the position is the same across consecutive levels.
-    llvm::for_each(reass, [this, tid, iv](int lvl) { pidxs[tid][lvl] = iv; });
-    c = genSparseCoord(builder, loc, tid, dim);
+    /// FIXME: See the [CLARIFY_POSITS_LVL] note in the header.
+    llvm::for_each(reassoc,
+                   [this, tid, iv](Level srcLvl) { posits[tid][srcLvl] = iv; });
+    crd = genSparseCrd(builder, loc, tid, dstLvl);
   } else {
-    // Dense tensor, the coordinates is the inducation variable.
-    c = iv;
+    // Dense tensor, the coordinate is the inducation variable.
+    crd = iv;
   }
 
   if (isSparseSlices[tid] && isSparseInput) {
@@ -417,7 +519,7 @@ Operation *LoopEmitter::enterLoopOverTensorAtDim(
     for (Value red : reduc)
       types.push_back(red.getType());
 
-    auto [trans, pred] = genSliceLegitPredicate(builder, loc, c, enc, dim);
+    auto [trans, pred] = genSliceLegitPredicate(builder, loc, crd, tid, srcLvl);
     bool hasReduc = !types.empty();
     scf::IfOp ifOp = builder.create<scf::IfOp>(loc, types, pred,
                                                /*else*/ hasReduc);
@@ -435,39 +537,37 @@ Operation *LoopEmitter::enterLoopOverTensorAtDim(
     }
     // Set the insertion point to matched branch.
     builder.setInsertionPointToStart(&ifOp.getThenRegion().front());
-    c = trans;
+    crd = trans;
   }
 
-  assert(c);
-  coord[tid][dim] = c;
-  // NOTE: we can also prepare for next dim here in advance
+  assert(crd);
+  coords[tid][srcLvl] = crd;
+  // NOTE: we can also prepare for next level here in advance
   // Push the loop into stack
-  loopStack.emplace_back(ArrayRef<size_t>(tid), ArrayRef<size_t>(dim), loop,
-                         builder.getInsertionBlock(), coord[tid][dim], loopTag);
+  loopStack.emplace_back(ArrayRef<TensorId>(tid), ArrayRef<Level>(srcLvl), loop,
+                         builder.getInsertionBlock(), crd, loopTag);
   // Emit extra locals.
-  emitExtraLocalsForTensorsAtDenseDims(builder, loc, tids, dims);
+  emitExtraLocalsForTensorsAtDenseLvls(builder, loc, tids, lvls);
 
   return loop;
 }
 
-Operation *LoopEmitter::enterFilterLoopOverTensorAtDim(
-    OpBuilder &builder, Location loc, size_t tid, size_t dim, AffineExpr affine,
-    MutableArrayRef<Value> reduc) {
-  assert(!affine.isa<AffineDimExpr>() && !isDenseDLT(dimTypes[tid][dim]));
-  assert(dimTypes[tid].size() > dim);
+Operation *LoopEmitter::enterFilterLoopOverTensorAtLvl(
+    OpBuilder &builder, Location loc, TensorId tid, Level lvl,
+    AffineExpr affine, MutableArrayRef<Value> reduc) {
+  assert(tid < lvlTypes.size() && lvl < lvlTypes[tid].size());
+  assert(!affine.isa<AffineDimExpr>() && !isDenseDLT(lvlTypes[tid][lvl]));
   // We can not re-enter the same level.
-  assert(!coord[tid][dim]);
-
-  Value step = constantIndex(builder, loc, 1);
-
-  Value lo = pidxs[tid][dim];
-  Value hi = highs[tid][dim];
+  assert(!coords[tid][lvl]);
 
   // TODO: We should instead use a whileOp for filter loop to allow early
-  // break when exceeding (for ordered dimensions).
+  // break when exceeding (for ordered levels).
   // TODO: There are many other potiential opportunities that we might apply in
-  // the future. E.g., we could use binary search to located the pointer index.
-  scf::ForOp forOp = builder.create<scf::ForOp>(loc, lo, hi, step, reduc);
+  // the future. E.g., we could use binary search to locate positions.
+  const Value step = constantIndex(builder, loc, 1);
+  const Value pLo = posits[tid][lvl];
+  const Value pHi = highs[tid][lvl];
+  scf::ForOp forOp = builder.create<scf::ForOp>(loc, pLo, pHi, step, reduc);
 
   // In-place update on the reduction variable vector.
   assert(forOp.getNumRegionIterArgs() == reduc.size());
@@ -475,18 +575,19 @@ Operation *LoopEmitter::enterFilterLoopOverTensorAtDim(
     reduc[i] = forOp.getRegionIterArg(i);
 
   builder.setInsertionPointToStart(forOp.getBody());
-  Value iv = forOp.getInductionVar();
-
-  pidxs[tid][dim] = iv;
-  // Generating a load on the indices array yields the coordinate.
-  Value ptr = idxBuffer[tid][dim];
-  coord[tid][dim] = genIndexLoad(builder, loc, ptr, iv);
-
-  // Generate an if condition to filter out indices that is not equal to the
-  // result of the affine expression.
-  Value expected = genAffine(builder, affine, loc);
-  auto pred = builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::eq,
-                                            coord[tid][dim], expected);
+  // The induction variable gives the position.
+  const Value pos = forOp.getInductionVar();
+  posits[tid][lvl] = pos;
+  // Generating a load on the coordinates array yields the crd.
+  const Value mem = coordinatesBuffers[tid][lvl];
+  const Value crd = genIndexLoad(builder, loc, mem, pos);
+  coords[tid][lvl] = crd;
+
+  // Generate an if-condition to filter out coordinates that are not
+  // equal to the result of the affine expression.
+  Value expected = genAffine(builder, loc, affine);
+  auto pred = builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::eq, crd,
+                                            expected);
   SmallVector<Type> types;
   for (Value red : reduc) {
     types.push_back(red.getType());
@@ -510,35 +611,45 @@ Operation *LoopEmitter::enterFilterLoopOverTensorAtDim(
   // Set the insert point to matched branch.
   builder.setInsertionPointToStart(&ifOp.getThenRegion().front());
 
-  // NOTE: we can also prepare for next dim here in advance
+  // NOTE: we can also prepare for next lvl here in advance
   // Push the loop into stack
-  loopStack.emplace_back(ArrayRef<size_t>(tid), ArrayRef<size_t>(dim), forOp,
-                         builder.getInsertionBlock(), coord[tid][dim], nullptr);
+  loopStack.emplace_back(ArrayRef<TensorId>(tid), ArrayRef<Level>(lvl), forOp,
+                         builder.getInsertionBlock(), crd, nullptr);
   return forOp;
 }
 
-void LoopEmitter::genDenseAffineAddressAtCurLevel(OpBuilder &builder,
-                                                  Location loc, size_t tid,
-                                                  size_t dim,
-                                                  AffineExpr affine) {
-  Value affineV = genAffine(builder, affine, loc);
-  pidxs[tid][dim] = genAddress(builder, loc, tid, dim, affineV);
+void LoopEmitter::genDenseAffineAddress(OpBuilder &builder, Location loc,
+                                        TensorId tid, Level lvl,
+                                        AffineExpr lvlExpr) {
+  assert(isDenseDLT(lvlTypes[tid][lvl]));
+  // For dense levels, the level-coordinate also serves as the position.
+  Value lvlCrd = genAffine(builder, loc, lvlExpr);
+  posits[tid][lvl] = genAddress(builder, loc, tid, lvl, lvlCrd);
 }
 
-Operation *LoopEmitter::enterCoIterationOverTensorsAtDims(
-    OpBuilder &builder, Location loc, ArrayRef<size_t> tids,
-    ArrayRef<size_t> dims, bool needsUniv, MutableArrayRef<Value> reduc) {
-  assert(tids.size() == dims.size());
+Operation *LoopEmitter::enterCoIterationOverTensorsAtLvls(
+    OpBuilder &builder, Location loc, ArrayRef<TensorId> tids,
+    ArrayRef<Level> lvls, bool needsUniv, MutableArrayRef<Value> reduc) {
+  assert(tids.size() == lvls.size());
   SmallVector<Type> types;
   SmallVector<Value> operands;
-  // Construct the while-loop with a parameter for each index.
-  Type indexType = builder.getIndexType();
-  for (auto [tid, dim] : llvm::zip(tids, dims)) {
-    if (isCompressedDLT(dimTypes[tid][dim]) ||
-        isSingletonDLT(dimTypes[tid][dim])) {
-      assert(pidxs[tid][dim]);
+  // Construct the while-loop with a parameter for each coordinate.
+  const Type indexType = builder.getIndexType();
+  for (auto [tid, lvl] : llvm::zip(tids, lvls)) {
+    const auto lvlTp = lvlTypes[tid][lvl];
+    if (isCompressedDLT(lvlTp) || isSingletonDLT(lvlTp)) {
+      const auto reassoc = getCollapseReassociation(tid, lvl);
+      for (unsigned i = 0, e = reassoc.size() - 1; i < e; i++) {
+        if (!isUniqueDLT(lvlTypes[tid][reassoc[i]])) {
+          // This is the segment high for each non-unique levels.
+          types.push_back(indexType);
+          operands.push_back(constantIndex(builder, loc, 0));
+        }
+      }
+      const auto pos = posits[tid][reassoc.front()];
+      assert(pos);
       types.push_back(indexType);
-      operands.push_back(pidxs[tid][dim]);
+      operands.push_back(pos);
     }
   }
   // The position where user-supplied reduction variable starts.
@@ -563,22 +674,31 @@ Operation *LoopEmitter::enterCoIterationOverTensorsAtDims(
   builder.setInsertionPointToStart(&whileOp.getBefore().front());
   Value cond;
   unsigned o = 0;
-  for (auto [t, lvl] : llvm::zip(tids, dims)) {
+  for (auto [t, lvl] : llvm::zip(tids, lvls)) {
     unsigned tid = t; // Why `t` can not be captured by lambda?
-    if (isCompressedDLT(dimTypes[tid][lvl]) ||
-        isSingletonDLT(dimTypes[tid][lvl])) {
+    const auto lvlTp = lvlTypes[tid][lvl];
+    if (isCompressedDLT(lvlTp) || isSingletonDLT(lvlTp)) {
+      const auto reassoc = getCollapseReassociation(tid, lvl);
+      assert(reassoc.size() == 1 || isUniqueCOOType(tensors[tid].getType()));
+      for (unsigned i = 0, e = reassoc.size() - 1; i < e; i++) {
+        if (!isUniqueDLT(lvlTypes[tid][reassoc[i]])) {
+          // Links the SSA chain for segHi.
+          segHi[tid][reassoc[i]] = after->getArgument(o++);
+        }
+      }
       Value op1 = before->getArgument(o);
-      Value op2 = highs[tid][lvl];
+      // We used the first level bound as the bound the collapsed set of levels.
+      Value op2 = highs[tid][reassoc.front()];
       Value opc = builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ult,
                                                 op1, op2);
       cond = cond ? builder.create<arith::AndIOp>(loc, cond, opc) : opc;
       // Update positions
       Value pos = after->getArgument(o++);
-      auto reass = getCollapseReassociation(tid, lvl);
-      assert(reass.size() == 1 || isUniqueCOOType(tensors[tid].getType()));
       // For COO, the position is the same across consecutive levels.
-      llvm::for_each(reass,
-                     [this, tid, pos](int lvl) { pidxs[tid][lvl] = pos; });
+      /// FIXME: See the [CLARIFY_POSITS_LVL] note in the header.
+      llvm::for_each(reassoc, [this, tid, pos](Level srcLvl) {
+        posits[tid][srcLvl] = pos;
+      });
     }
   }
   builder.create<scf::ConditionOp>(loc, cond, before->getArguments());
@@ -588,20 +708,17 @@ Operation *LoopEmitter::enterCoIterationOverTensorsAtDims(
 
   SmallVector<std::pair<Value, unsigned>> slicesPreds;
   unsigned i = 0;
-  for (auto [tid, dim] : llvm::zip(tids, dims)) {
+  for (auto [tid, lvl] : llvm::zip(tids, lvls)) {
     // Prepares for next level.
-    if (isCompressedDLT(dimTypes[tid][dim]) ||
-        isSingletonDLT(dimTypes[tid][dim])) {
-      coord[tid][dim] = genSparseCoord(builder, loc, tid, dim);
+    const auto lvlTp = lvlTypes[tid][lvl];
+    if (isCompressedDLT(lvlTp) || isSingletonDLT(lvlTp)) {
+      coords[tid][lvl] = genSparseCrd(builder, loc, tid, lvl);
       if (isSparseSlices[tid]) {
-        Value load =
-            genIndexLoad(builder, loc, idxBuffer[tid][dim], pidxs[tid][dim]);
-        auto enc = getSparseTensorEncoding(tensors[tid].getType());
         auto [trans, pred] =
-            genSliceLegitPredicate(builder, loc, load, enc, dim);
+            genSliceLegitPredicate(builder, loc, coords[tid][lvl], tid, lvl);
         slicesPreds.emplace_back(pred, i);
         // Updates to the relative coordinate to the slice.
-        coord[tid][dim] = trans;
+        coords[tid][lvl] = trans;
       }
       i++;
     }
@@ -611,14 +728,16 @@ Operation *LoopEmitter::enterCoIterationOverTensorsAtDims(
     // Skips invalid loop iteration when slice coordinate is inapplicable.
     SmallVector<Value> yields(after->getArguments());
     // Generates a list of if statments
-    //  pidx = in_slice ? pidx : pidx + 1
-    // TODO: instead of always picking pidx + 1, we should set pidx = high to
-    // break to loop the coordinates is larger than the slice size.
+    //  pos = in_slice ? pos : pos + 1
+    // TODO: instead of always picking pos + 1, we should set pos = high to
+    // break to loop if the coordinates are larger than the slice size.
+    //
+    // This "idx" is the index into `llvm::zip(tids, lvls)`
     for (auto [pred, idx] : slicesPreds) {
-      Value nextPidx = builder.create<arith::AddIOp>(
+      Value nextPos = builder.create<arith::AddIOp>(
           loc, yields[idx], constantIndex(builder, loc, 1));
       yields[idx] =
-          builder.create<arith::SelectOp>(loc, pred, yields[idx], nextPidx);
+          builder.create<arith::SelectOp>(loc, pred, yields[idx], nextPos);
     }
 
     Value pred = slicesPreds.front().first;
@@ -641,31 +760,79 @@ Operation *LoopEmitter::enterCoIterationOverTensorsAtDims(
   Value min;
   // Finds the minimum coordinate
   if (!needsUniv) {
-    for (auto [tid, dim] : llvm::zip(tids, dims)) {
-      if (isCompressedDLT(dimTypes[tid][dim]) ||
-          isSingletonDLT(dimTypes[tid][dim])) {
+    for (auto [tid, lvl] : llvm::zip(tids, lvls)) {
+      const auto lvlTp = lvlTypes[tid][lvl];
+      if (isCompressedDLT(lvlTp) || isSingletonDLT(lvlTp)) {
+        const auto crd = coords[tid][lvl];
         if (min) {
           Value cmp = builder.create<arith::CmpIOp>(
-              loc, arith::CmpIPredicate::ult, coord[tid][dim], min);
-          min = builder.create<arith::SelectOp>(loc, cmp, coord[tid][dim], min);
+              loc, arith::CmpIPredicate::ult, crd, min);
+          min = builder.create<arith::SelectOp>(loc, cmp, crd, min);
         } else {
-          min = coord[tid][dim];
+          min = crd;
         }
       }
     }
   } else {
     assert(!min);
-    // Otherwise, universal index is the minimal pidx.
+    // Otherwise, universal index is the minimal pos.
     min = after->getArguments().back();
   }
 
   // Sets up the loop stack.
-  loopStack.emplace_back(tids, dims, whileOp, builder.getInsertionBlock(), min,
+  loopStack.emplace_back(tids, lvls, whileOp, builder.getInsertionBlock(), min,
                          loopTag);
   assert(loopStack.size() == loopSeqStack.size());
 
+  for (auto [tid, dstLvl] : llvm::zip(tids, lvls)) {
+    const auto reassoc = getCollapseReassociation(tid, dstLvl);
+    assert(reassoc.size() == 1 || isUniqueCOOType(tensors[tid].getType()));
+    // TODO: Refactors this into smaller functions.
+    // NOTE: For all the collapsed level (except for the last one, that is why
+    // the loop ends with `reassoc.size() - 1`), as each iteration is advanced
+    // by the segment size of the last level, which does not always invalidate
+    // the segment size for the previous levels, thus we need to propagate the
+    // segment sizes across loop iterations and only forward if needed.
+    //
+    // E.g., for a COO tensor with the following coordinates array.
+    // (0, 0, 1),
+    // (0, 0, 2),
+    // (1, 1, 1),
+    // segHi[lvl=0] = segHi[lvl=1] = 2
+    // segHi[lvl=2] = 1,
+    // the first iteration does not invalidate segHi[0] and segHi[1]
+    for (unsigned i = 0, e = reassoc.size() - 1; i < e; i++) {
+      const Level srcLvl = reassoc[i];
+      if (!isUniqueDLT(lvlTypes[tid][srcLvl])) {
+        const Value pos = posits[tid][srcLvl];
+        const auto oldSegHi = segHi[tid][srcLvl];
+        assert(oldSegHi);
+        Value newSegHi = builder.create<arith::CmpIOp>(
+            loc, arith::CmpIPredicate::uge, pos, oldSegHi);
+        auto ifNewSegHi = builder.create<scf::IfOp>(loc, builder.getIndexType(),
+                                                    newSegHi, true);
+        {
+          OpBuilder::InsertionGuard guard(builder);
+          builder.setInsertionPointToStart(ifNewSegHi.thenBlock());
+          builder.create<scf::YieldOp>(loc,
+                                       genSegmentHigh(builder, loc, tid, srcLvl,
+                                                      pos, highs[tid][srcLvl]));
+          // Else, resues the same segment high.
+          builder.setInsertionPointToStart(ifNewSegHi.elseBlock());
+          builder.create<scf::YieldOp>(loc, oldSegHi);
+        }
+        highs[tid][srcLvl + 1] = segHi[tid][srcLvl] = ifNewSegHi.getResult(0);
+      }
+    };
+    const auto srcLvl = reassoc.back();
+    if (!isUniqueDLT(lvlTypes[tid][srcLvl])) {
+      segHi[tid][srcLvl] = genSegmentHigh(
+          builder, loc, tid, srcLvl, posits[tid][srcLvl], highs[tid][srcLvl]);
+    }
+  }
+
   // Emits extra locals
-  emitExtraLocalsForTensorsAtDenseDims(builder, loc, tids, dims);
+  emitExtraLocalsForTensorsAtDenseLvls(builder, loc, tids, lvls);
 
   // Updates reduction variables
   assert(after->getNumArguments() == o + reduc.size() + (needsUniv ? 1 : 0));
@@ -676,64 +843,75 @@ Operation *LoopEmitter::enterCoIterationOverTensorsAtDims(
   return whileOp;
 }
 
-void LoopEmitter::prepareLoopOverTensorAtDim(OpBuilder &builder, Location loc,
-                                             size_t tid, size_t dim) {
-  assert(dimTypes[tid].size() > dim);
-  auto dimType = dimTypes[tid][dim];
+void LoopEmitter::prepareLoopOverTensorAtLvl(OpBuilder &builder, Location loc,
+                                             TensorId tid, Level dstLvl) {
+  assert(tid < lvlTypes.size() && dstLvl < lvlTypes[tid].size());
+  const auto lvlTp = lvlTypes[tid][dstLvl];
 
-  if (isDenseDLT(dimType))
+  if (isDenseDLT(lvlTp))
     return;
 
-  auto reassoc = getCollapseReassociation(tid, dim);
-  for (auto lvl : reassoc) {
-    // Either the first dimension, or the previous dimension has been set.
-    assert(lvl == 0 || pidxs[tid][lvl - 1]);
-    Value c0 = constantIndex(builder, loc, 0);
-    Value c1 = constantIndex(builder, loc, 1);
-    if (isCompressedDLT(dimType)) {
-      Value ptr = ptrBuffer[tid][lvl];
+  const Value c0 = constantIndex(builder, loc, 0);
+  const Value c1 = constantIndex(builder, loc, 1);
+  for (const Level srcLvl : getCollapseReassociation(tid, dstLvl)) {
+    // Either the first level, or the previous level has been set.
+    /// FIXME: See the [CLARIFY_POSITS_LVL] note in the header.
+    assert(srcLvl == 0 || posits[tid][srcLvl - 1]);
+    if (!isCompressedDLT(lvlTp) && !isSingletonDLT(lvlTp))
+      continue;
+    if (isCompressedDLT(lvlTp)) {
+      const Value mem = positionsBuffers[tid][srcLvl];
 
-      Value pLo = lvl == 0 ? c0 : pidxs[tid][lvl - 1];
-      pidxs[tid][lvl] = genIndexLoad(builder, loc, ptr, pLo);
+      const Value pLo = srcLvl == 0 ? c0 : posits[tid][srcLvl - 1];
+      posits[tid][srcLvl] = genIndexLoad(builder, loc, mem, pLo);
 
-      Value pHi = builder.create<arith::AddIOp>(loc, pLo, c1);
-      highs[tid][lvl] = genIndexLoad(builder, loc, ptr, pHi);
+      const Value pHi = builder.create<arith::AddIOp>(loc, pLo, c1);
+      highs[tid][srcLvl] = genIndexLoad(builder, loc, mem, pHi);
       return;
     }
-    if (isSingletonDLT(dimType)) {
-      Value pLo = lvl == 0 ? c0 : pidxs[tid][lvl - 1];
-      Value pHi = builder.create<arith::AddIOp>(loc, pLo, c1);
-
-      pidxs[tid][lvl] = pLo;
-      highs[tid][lvl] = pHi;
+    if (isSingletonDLT(lvlTp)) {
+      const Value pLo = srcLvl == 0 ? c0 : posits[tid][srcLvl - 1];
+      posits[tid][srcLvl] = pLo;
+
+      // If we are coiterating non-unique levels, then use pHi=segHi;
+      // otherwise use pHi=pLo+1.
+      // NOTE: Just because the level is non-unique, that does not
+      // guarantee that segHi is defined: because we only generate segHi
+      // whenever coiterating, in order to improve code quality for the
+      // non-coiterating cases.
+      const auto theSegHi = segHi[tid][srcLvl - 1];
+      highs[tid][srcLvl] = (!isUniqueDLT(lvlTypes[tid][srcLvl - 1]) && theSegHi)
+                               ? theSegHi
+                               : builder.create<arith::AddIOp>(loc, pLo, c1);
       return;
     }
   }
 
-  llvm_unreachable("Unrecognizable dimesion type!");
+  llvm_unreachable("Unrecognized level-type!");
 }
 
-void LoopEmitter::emitExtraLocalsForTensorsAtDenseDims(OpBuilder &builder,
+void LoopEmitter::emitExtraLocalsForTensorsAtDenseLvls(OpBuilder &builder,
                                                        Location loc,
-                                                       ArrayRef<size_t> tids,
-                                                       ArrayRef<size_t> dims) {
-  // Initialize dense positions. Note that we generate dense indices of the
+                                                       ArrayRef<TensorId> tids,
+                                                       ArrayRef<Level> lvls) {
+  // Initialize dense positions. Note that we generate dense coordinates of the
   // output tensor unconditionally, since they may not appear in the lattice,
   // but may be needed for linearized codegen.
-  for (auto [tid, dim] : llvm::zip(tids, dims)) {
-    if (isDenseDLT(dimTypes[tid][dim])) {
+  assert(tids.size() == lvls.size());
+  for (auto [tid, lvl] : llvm::zip(tids, lvls)) {
+    if (isDenseDLT(lvlTypes[tid][lvl])) {
       auto enc = getSparseTensorEncoding(tensors[tid].getType());
       if (enc && !isSparseOutput(tid)) {
-        bool validPidx = dim == 0 || pidxs[tid][dim - 1];
-        if (!validPidx) {
-          // We might not find the pidx for the sparse output tensor as it is
+        bool validPos = lvl == 0 || posits[tid][lvl - 1];
+        if (!validPos) {
+          // We might not find the pos for the sparse output tensor as it is
           // unconditionally required by the sparsification.
           assert(isOutputTensor(tid));
           continue;
         }
-        pidxs[tid][dim] =
-            genAddress(builder, loc, tid, dim, loopStack.back().iv);
-        // NOTE: we can also prepare for next dim here in advance
+        posits[tid][lvl] =
+            genAddress(builder, loc, tid, lvl, loopStack.back().iv);
+        // NOTE: we can also prepare for next lvl here in advance
       }
     }
   }
@@ -741,12 +919,9 @@ void LoopEmitter::emitExtraLocalsForTensorsAtDenseDims(OpBuilder &builder,
 
 void LoopEmitter::exitForLoop(RewriterBase &rewriter, Location loc,
                               MutableArrayRef<Value> reduc) {
-  LoopLevelInfo &loopInfo = loopStack.back();
+  const LoopInfo &loopInfo = loopStack.back();
   rewriter.setInsertionPointToEnd(loopInfo.userCodeBlock);
-  auto &dims = loopStack.back().dims;
-  auto &tids = loopStack.back().tids;
-  auto forOp = llvm::dyn_cast<scf::ForOp>(loopInfo.loop);
-  if (forOp) {
+  if (auto forOp = llvm::dyn_cast<scf::ForOp>(loopInfo.loop)) {
     if (!reduc.empty()) {
       assert(reduc.size() == forOp.getNumResults());
       rewriter.create<scf::YieldOp>(loc, reduc);
@@ -777,6 +952,7 @@ void LoopEmitter::exitForLoop(RewriterBase &rewriter, Location loc,
       // One of the operands must be the init value (which is also the
       // previous reduction value).
       assert(curVal);
+#ifndef NDEBUG
       // The reduction expression should be the only user of the reduction val
       // inside the parallel for.
       unsigned numUsers = 0;
@@ -785,7 +961,7 @@ void LoopEmitter::exitForLoop(RewriterBase &rewriter, Location loc,
           numUsers++;
       }
       assert(numUsers == 1);
-      (void)numUsers; // to silence unused variable warning in release build
+#endif // NDEBUG
 
       rewriter.setInsertionPointAfter(redExp);
       auto redOp = rewriter.create<scf::ReduceOp>(loc, curVal);
@@ -811,23 +987,21 @@ void LoopEmitter::exitForLoop(RewriterBase &rewriter, Location loc,
   // Finished iterating a tensor, clean up
   // We only do the clean up on for loop as while loops do not necessarily
   // finish the iteration on a sparse tensor
-  for (auto [tid, dim] : llvm::zip(tids, dims)) {
+  for (auto [tid, lvl] : llvm::zip(loopInfo.tids, loopInfo.lvls)) {
     // Reset to null.
-    coord[tid][dim] = Value();
-    pidxs[tid][dim] = Value();
-    // Dense dimension, high is fixed.
-    if (!isDenseDLT(dimTypes[tid][dim]))
-      highs[tid][dim] = Value();
+    coords[tid][lvl] = Value();
+    posits[tid][lvl] = Value();
+    // Dense level, high is fixed.
+    if (!isDenseDLT(lvlTypes[tid][lvl]))
+      highs[tid][lvl] = Value();
   }
 }
 
 void LoopEmitter::exitCoIterationLoop(OpBuilder &builder, Location loc,
                                       MutableArrayRef<Value> reduc) {
-  const LoopLevelInfo &loopInfo = loopStack.back();
+  const LoopInfo &loopInfo = loopStack.back();
   auto whileOp = llvm::cast<scf::WhileOp>(loopInfo.loop);
   builder.setInsertionPointToEnd(loopInfo.userCodeBlock);
-  auto &dims = loopInfo.dims;
-  auto &tids = loopInfo.tids;
   Value iv = loopInfo.iv;
   // Finalize the induction. Note that the induction could be performed
   // in the individual if-branches to avoid re-evaluating the conditions.
@@ -837,20 +1011,44 @@ void LoopEmitter::exitCoIterationLoop(OpBuilder &builder, Location loc,
   unsigned o = 0;
   SmallVector<Value> operands;
   Value one = constantIndex(builder, loc, 1);
-  for (auto [tid, dim] : llvm::zip(tids, dims)) {
-    if (isCompressedDLT(dimTypes[tid][dim]) ||
-        isSingletonDLT(dimTypes[tid][dim])) {
-      Value op1 = coord[tid][dim];
-      Value op3 = pidxs[tid][dim];
+  for (auto [tid, dstLvl] : llvm::zip(loopInfo.tids, loopInfo.lvls)) {
+    const auto lvlTp = lvlTypes[tid][dstLvl];
+    if (isCompressedDLT(lvlTp) || isSingletonDLT(lvlTp)) {
+      const auto reassoc = getCollapseReassociation(tid, dstLvl);
+      assert(reassoc.size() == 1 || isUniqueCOOType(tensors[tid].getType()));
+      for (unsigned i = 0, e = reassoc.size() - 1; i < e; i++) {
+        const Level srcLvl = reassoc[i];
+        if (!isUniqueDLT(lvlTypes[tid][srcLvl])) {
+          operands.push_back(segHi[tid][srcLvl]);
+          o++;
+        }
+      }
+      const Value crd = coords[tid][dstLvl];
+      const Value pos = posits[tid][dstLvl];
       Value cmp =
-          builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::eq, op1, iv);
-      Value add = builder.create<arith::AddIOp>(loc, op3, one);
-      operands.push_back(builder.create<arith::SelectOp>(loc, cmp, add, op3));
+          builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::eq, crd, iv);
+      // If the loop contains a coiteration with non-unique level, we fast
+      // forward all the duplicated coords by setting the position to the
+      // segment high.
+      Value add = !isUniqueDLT(lvlTypes[tid][reassoc.back()])
+                      ? segHi[tid][reassoc.back()]
+                      : builder.create<arith::AddIOp>(loc, pos, one);
+
+      operands.push_back(builder.create<arith::SelectOp>(loc, cmp, add, pos));
       // Following loops continue iteration from the break point of the
       // current while loop.
-      pidxs[tid][dim] = whileOp->getResult(o++);
-      // The coordinates are invalid now.
-      coord[tid][dim] = nullptr;
+      const Value newPos = whileOp->getResult(o++);
+      // We need to define a new local variable for `tid` to avoid
+      // warnings about "captured structured bindings are a C++20 extension".
+      // FIXME(wrengr): define a helper function to capture this idiom!
+      const TensorId newTid = tid;
+      llvm::for_each(reassoc, [this, newTid, newPos](Level srcLvl) {
+        posits[newTid][srcLvl] = newPos;
+      });
+      // The coordinate is invalid now.
+      coords[tid][dstLvl] = nullptr;
+      // The segment high is invalid now.
+      segHi[tid][dstLvl] = nullptr;
       // highs remains unchanged.
     }
   }
@@ -880,8 +1078,8 @@ void LoopEmitter::exitCurrentLoop(RewriterBase &rewriter, Location loc,
                                   MutableArrayRef<Value> reduc) {
   // Clean up the values, it would help use to discover potential bug at a
   // earlier stage (instead of silently using a wrong value).
-  LoopLevelInfo &loopInfo = loopStack.back();
-  assert(loopInfo.tids.size() == loopInfo.dims.size());
+  const LoopInfo &loopInfo = loopStack.back();
+  assert(loopInfo.tids.size() == loopInfo.lvls.size());
   SmallVector<Value> red;
   if (llvm::isa<scf::WhileOp>(loopInfo.loop)) {
     exitCoIterationLoop(rewriter, loc, reduc);
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/LoopEmitter.h b/mlir/lib/Dialect/SparseTensor/Transforms/LoopEmitter.h
index 09d4afbf5dc96..8e6c65fd96c92 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/LoopEmitter.h
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/LoopEmitter.h
@@ -12,11 +12,36 @@
 #include <vector>
 
 #include "mlir/Dialect/SparseTensor/IR/Enums.h"
+#include "mlir/Dialect/SparseTensor/IR/SparseTensor.h"
+#include "mlir/Dialect/SparseTensor/Utils/Merger.h"
 #include "mlir/IR/PatternMatch.h"
 
 namespace mlir {
 namespace sparse_tensor {
 
+//===----------------------------------------------------------------------===//
+/// The position of a loop in the loop-stack, or the position of a
+/// `LoopId` in a topologically-sorted list of `LoopId`s.
+///
+/// Although this type may have the same cardinality as `LoopId`, it must
+/// not be confused with that type.  The `LoopId` type is used by the `Merger`
+/// as a unique identifier for loop-variables, regardless of the ordering
+/// of those loops.  Whereas the `LoopOrd` type is used by the `LoopEmitter`
+/// (and `CodegenEnv`) to refer to the actual order in which loops are
+/// generated.
+///
+/// TODO: further explicate the correspondences between these various
+/// types.  In particular, since the `$dim` argument to `linalg::IndexOp`
+/// is a De Bruijn index, it seems like that should correspond to `LoopOrd`,
+/// and yet the `Merger` has that correspond with `LoopId` instead.
+/// In addition `LoopEmitter::genAffine` has `AffineDimExpr::position`
+/// correspond to `LoopId`, however it is unclear what the providence
+/// of those `AffineDimExpr` is.
+//
+// TODO: use a struct/class rather than a typedef, so that we can actually
+// typecheck this to avoid mixups in the code.
+using LoopOrd = unsigned;
+
 //===----------------------------------------------------------------------===//
 // SparseTensorLoopEmiter class, manages sparse tensors and helps to
 // generate loop structure to (co)-iterate sparse tensors.
@@ -33,13 +58,13 @@ namespace sparse_tensor {
 //
 // One can use
 //
-// SparseTensorLoopEmiter loopEmiter({T1, T1});
+// LoopEmiter loopEmiter({T1, T1});
 // loopEmiter.initializeLoopEmit();
-// loopEmiter.enterLoopOverTensorAtDim(T1, 0);
-// loopEmiter.enterLoopOverTensorAtDim(T2, 0);
-// loopEmiter.enterLoopOverTensorAtDim(T1, 1);
+// loopEmiter.enterLoopOverTensorAtLvl(T1, 0);
+// loopEmiter.enterLoopOverTensorAtLvl(T2, 0);
+// loopEmiter.enterLoopOverTensorAtLvl(T1, 1);
 // loopEmiter.exitCurrentLoop();
-// loopEmiter.enterLoopOverTensorAtDim(T2, 1);
+// loopEmiter.enterLoopOverTensorAtLvl(T2, 1);
 // loopEmiter.exitCurrentLoop(); // exit k
 // loopEmiter.exitCurrentLoop(); // exit j
 // loopEmiter.exitCurrentLoop(); // exit i
@@ -54,30 +79,31 @@ class LoopEmitter {
 
   LoopEmitter() = default;
 
-  /// Takes an array of tensors inputs, on which the generated loops will
-  /// iterate on. The index of the tensor in the array is also the tensor id
-  /// (tid) used in related functions. If isSparseOut is set, loop emitter
-  /// assume that the sparse output tensor is empty, and will always generate
-  /// loops on it based on the dim sizes. An optional array could be provided
-  /// (by sparsification) to indicate the loop id sequence that will be
-  /// generated. It is used to establish the mapping between affineDimExpr to
-  /// the corresponding loop index in the loop stack that are maintained by the
-  /// loop emitter.
+  /// Takes an array of input tensors, which the generated loops will
+  /// iterate over.  Each tensor is given a `TensorId` (numerically equal
+  /// to the position of that tensor `Value` in the array).  Setting
+  /// `isSparseOut` indicates that the sparse output tensor is empty,
+  /// so the loop emitter will generate loops over it according to the
+  /// level-sizes.  The `topSort` array specifies the actual order in
+  /// which loops are generated, thus providing a mapping from `LoopOrd`
+  /// to `LoopId`.
   void initialize(ValueRange tensors, StringAttr loopTag = nullptr,
                   bool hasOutput = false, bool isSparseOut = false,
-                  ArrayRef<unsigned> topSort = {});
+                  ArrayRef<LoopId> topSort = {});
 
   explicit LoopEmitter(ValueRange tensors, StringAttr loopTag = nullptr,
                        bool hasOutput = false, bool isSparseOut = false,
-                       ArrayRef<unsigned> topSort = {});
+                       ArrayRef<LoopId> topSort = {});
 
-  /// Starts a loop emitting session by generating all the buffers needed to
-  /// iterate tensors.
+  /// Starts a loop emitting session by generating all the buffers needed
+  /// for iterating over the tensors.
   void initializeLoopEmit(OpBuilder &builder, Location loc,
                           OutputUpdater updater = nullptr);
 
-  /// Generates a list of operations to compute the affine expression.
-  Value genAffine(OpBuilder &builder, AffineExpr a, Location loc);
+  /// Generates code to compute an affine expression whose variables are
+  /// `LoopId`s (i.e., `a.cast<AffineDimExpr>().getPosition()` is a valid
+  /// `LoopId`).
+  Value genAffine(OpBuilder &builder, Location loc, AffineExpr a);
 
   /// Enters a new loop sequence, the loops within the same sequence starts
   /// from the break points of previous loop instead of starting over from 0.
@@ -93,73 +119,77 @@ class LoopEmitter {
   ///     ...
   ///   // loop sequence end.
   /// }
-  void enterNewLoopSeq(OpBuilder &builder, Location loc, ArrayRef<size_t> tids,
-                       ArrayRef<size_t> dims);
+  void enterNewLoopSeq(OpBuilder &builder, Location loc,
+                       ArrayRef<TensorId> tids, ArrayRef<Level> lvls);
 
-  // exit the current loop sequence, this will reset universal index to 0.
+  /// Exits the current loop sequence, this will reset universal index to 0.
   void exitCurrentLoopSeq() {
     assert(loopSeqStack.size() == loopStack.size() + 1);
     loopSeqStack.pop_back();
   }
 
-  // TODO: Gets rid of `dim` in the argument list? Track the dimension we
-  // are currently at internally. Then it would be enterNextDimForTensor.
-  // Still need a way to specify the dim for non annoated dense tensor though,
-  // as it can be accessed out of order.
-  /// Emits loop over tensor_tid_dim, it assumes that loops between
-  /// tensor_tid_[0, dim - 1] have already been generated.
+  // TODO: Get rid of `lvls` in the argument list? Track the level we
+  // are currently at internally. Then it would be enterNextLvlForTensor.
+  // Still need a way to specify the lvl for non-annotated tensors though,
+  // as those can be accessed out of order.
+  //
+  /// Emits loop over tensor_tid_lvl, it assumes that loops between
+  /// tensor_tid_[0, lvl - 1] have already been generated.
   /// The function will also perform in-place update on the `reduc` vector to
   /// return the reduction variable used inside the generated loop.
-  Operation *enterLoopOverTensorAtDim(OpBuilder &builder, Location loc,
-                                      ArrayRef<size_t> tids,
-                                      ArrayRef<size_t> dims,
+  Operation *enterLoopOverTensorAtLvl(OpBuilder &builder, Location loc,
+                                      ArrayRef<TensorId> tids,
+                                      ArrayRef<Level> lvls,
                                       MutableArrayRef<Value> reduc = {},
                                       bool isParallel = false);
 
-  Operation *enterFilterLoopOverTensorAtDim(OpBuilder &builder, Location loc,
-                                            size_t tid, size_t dim,
+  Operation *enterFilterLoopOverTensorAtLvl(OpBuilder &builder, Location loc,
+                                            TensorId tid, Level lvl,
                                             AffineExpr affine,
                                             MutableArrayRef<Value> reduc = {});
 
-  void genDenseAffineAddressAtCurLevel(OpBuilder &builder, Location loc,
-                                       size_t tid, size_t dim,
-                                       AffineExpr affine);
+  void genDenseAffineAddress(OpBuilder &builder, Location loc, TensorId tid,
+                             Level lvl, AffineExpr lvlExpr);
 
   /// Emits a co-iteration loop over a set of tensors.
-  Operation *enterCoIterationOverTensorsAtDims(
-      OpBuilder &builder, Location loc, ArrayRef<size_t> tids,
-      ArrayRef<size_t> dims, bool needsUniv, MutableArrayRef<Value> reduc = {});
+  Operation *enterCoIterationOverTensorsAtLvls(
+      OpBuilder &builder, Location loc, ArrayRef<TensorId> tids,
+      ArrayRef<Level> lvls, bool needsUniv, MutableArrayRef<Value> reduc = {});
 
   void exitCurrentLoop(RewriterBase &rewriter, Location loc,
                        MutableArrayRef<Value> reduc = {});
 
-  /// Returns the array of coordinate for all the loop generated till now.
-  void getCoordinateArray(SmallVectorImpl<Value> &coords) const {
+  /// Fills the out-parameter with the loop induction variables for all
+  /// loops in the current loop-stack.  The variables are given in the
+  /// same order as the loop-stack, hence `ivs` should be indexed into
+  /// by `LoopOrd` (not `LoopId`).
+  void getLoopIVs(SmallVectorImpl<Value> &ivs) const {
+    ivs.clear();
+    ivs.reserve(getCurrentDepth());
     for (auto &l : loopStack)
-      coords.push_back(l.iv);
+      ivs.push_back(l.iv);
   }
 
-  /// Gets loop induction variable at the given level.
-  unsigned getCurrentDepth() const { return loopStack.size(); }
+  /// Gets the current depth of the loop-stack.  The result is given
+  /// the type `LoopOrd` for the same reason as one-past-the-end iterators.
+  LoopOrd getCurrentDepth() const { return loopStack.size(); }
 
-  /// Gets loop induction variable at the given level.
-  Value getLoopIV(size_t level) const {
-    if (level < loopStack.size())
-      return loopStack[level].iv;
-    return nullptr;
+  /// Gets loop induction variable for the given `LoopOrd`.
+  Value getLoopIV(LoopOrd n) const {
+    return n < getCurrentDepth() ? loopStack[n].iv : Value();
   }
 
   ///
   /// Getters.
   ///
-  const std::vector<std::vector<Value>> &getPidxs() const { return pidxs; };
-  const std::vector<std::vector<Value>> &getCoord() const { return coord; };
+  const std::vector<std::vector<Value>> &getPosits() const { return posits; };
+  const std::vector<std::vector<Value>> &getCoords() const { return coords; };
   const std::vector<std::vector<Value>> &getHighs() const { return highs; };
-  const std::vector<std::vector<Value>> &getPtrBuffer() const {
-    return ptrBuffer;
+  const std::vector<std::vector<Value>> &getPositionBuffers() const {
+    return positionsBuffers;
   };
-  const std::vector<std::vector<Value>> &getIdxBuffer() const {
-    return idxBuffer;
+  const std::vector<std::vector<Value>> &getCoordinateBuffers() const {
+    return coordinatesBuffers;
   };
   const std::vector<Value> &getValBuffer() const { return valBuffer; };
 
@@ -168,49 +198,74 @@ class LoopEmitter {
   }
 
 private:
-  struct LoopLevelInfo {
-    LoopLevelInfo(ArrayRef<size_t> tids, ArrayRef<size_t> dims, Operation *loop,
-                  Block *userBlock, Value iv, StringAttr loopTag)
-        : tids(tids), dims(dims), loop(loop), userCodeBlock(userBlock), iv(iv) {
+  struct LoopInfo {
+    LoopInfo(ArrayRef<TensorId> tids, ArrayRef<Level> lvls, Operation *loop,
+             Block *userBlock, Value iv, StringAttr loopTag)
+        : tids(tids), lvls(lvls), loop(loop), userCodeBlock(userBlock), iv(iv) {
       // Attached a special tag to loop emitter generated loop.
       if (loopTag)
         loop->setAttr(LoopEmitter::getLoopEmitterLoopAttrName(), loopTag);
     }
-    // TODO: maybe use a vector<pair> for tid and dim?
+    // TODO: maybe use a vector<pair> for tid and lvl?
+    //       (Better yet, compress them together a la `TensorLoopId`.)
     // The set of tensors that the loop is operating on
-    const llvm::SmallVector<size_t> tids;
-    // The corresponding dims for the tensors
-    const llvm::SmallVector<size_t> dims;
+    const llvm::SmallVector<TensorId> tids;
+    // The corresponding levels for the tensors
+    const llvm::SmallVector<Level> lvls;
     const Operation *loop;      // the loop operation
     Block *const userCodeBlock; // the block holding users' generated code.
     const Value iv;             // the induction variable for the loop
   };
 
-  /// Linearizes address for dense dimension (i.e., p = (i * d0) + j).
-  Value genAddress(OpBuilder &builder, Location loc, size_t tid, size_t dim,
+  /// Linearizes address for dense level (i.e., p = (i * d0) + j).
+  Value genAddress(OpBuilder &builder, Location loc, TensorId tid, Level lvl,
                    Value iv);
 
-  /// Generates instructions to compute the coordinate of tesnors[tid] on `l`
-  /// under the current loop context.
-  Value genSparseCoord(OpBuilder &builder, Location loc, size_t tid, size_t l);
-
-  bool isOutputTensor(size_t tid) {
-    return hasOutput && tid == tensors.size() - 1;
+  /// Generates the segment high for a non-unique level (to fast forward
+  /// duplicated coordinates).  That is, it generates the code:
+  ///
+  ///   crd = coordinates_tid_lvl[pos]
+  ///   while (pos < pHi && coordinates_tid_lvl[pos] == crd)
+  ///      pos++;
+  ///   <return pos>;
+  Value genSegmentHigh(OpBuilder &builder, Location loc, TensorId tid,
+                       Level lvl, Value pos, Value pHi);
+
+  /// Generates instructions to compute the coordinate of tensors[tid][lvl]
+  /// under the current loop context.  The final argument is the
+  /// collapsed-output level, whereas this function handles converting
+  /// that to the uncollapsed-input level
+  Value genSparseCrd(OpBuilder &builder, Location loc, TensorId tid,
+                     Level dstLvl);
+
+  /// Generates a predicate to determine whether the tranformed coordinates are
+  /// in the given slice.
+  /// Returns std::pair<Transformed coordinates, Predicate>
+  std::pair<Value, Value> genSliceLegitPredicate(OpBuilder &builder,
+                                                 Location loc, Value crd,
+                                                 TensorId tid, Level lvl);
+
+  TensorId getNumTensors() const { return tensors.size(); }
+
+  bool isOutputTensor(TensorId tid) const {
+    return hasOutput && tid == static_cast<TensorId>(getNumTensors() - 1);
   }
 
-  bool isSparseOutput(size_t tid) { return isOutputTensor(tid) && isSparseOut; }
+  bool isSparseOutput(TensorId tid) const {
+    return isOutputTensor(tid) && isSparseOut;
+  }
 
-  /// Setups [lo, hi] for iterating tensor[dim], it assumes that tensor[0
-  /// ...dims-1] has already been setup.
-  void prepareLoopOverTensorAtDim(OpBuilder &builder, Location loc, size_t tid,
-                                  size_t dim);
+  /// Prepares loop for iterating over `tensor[lvl]`, under the assumption
+  /// that `tensor[0...lvl-1]` loops have already been set up.
+  void prepareLoopOverTensorAtLvl(OpBuilder &builder, Location loc,
+                                  TensorId tid, Level lvl);
 
   /// Emits extra locals, since the locals might not be in simplified lattices
-  /// point used to generate the loops, but are still required to generates
+  /// point used to generate the loops, but are still required to generate
   /// expressions.
-  void emitExtraLocalsForTensorsAtDenseDims(OpBuilder &builder, Location loc,
-                                            ArrayRef<size_t> tids,
-                                            ArrayRef<size_t> dims);
+  void emitExtraLocalsForTensorsAtDenseLvls(OpBuilder &builder, Location loc,
+                                            ArrayRef<TensorId> tids,
+                                            ArrayRef<Level> lvls);
 
   /// Exits a for loop, returns the reduction results, e.g.,
   /// For sequential for loops:
@@ -243,6 +298,38 @@ class LoopEmitter {
   void exitCoIterationLoop(OpBuilder &builder, Location loc,
                            MutableArrayRef<Value> reduc);
 
+  //
+  // View-based-reshape methods.
+  //
+
+  /// Get the collapse reassociation for `tensors[tid][dstLvl]`.
+  /// For unreshaped operands, the reassociation is simply an identity
+  /// transformation.
+  ///
+  /// NOTE: the result uses `Level` rather than the `int64_t` of
+  /// `ReassociationIndices`, since the former gives clarity to what
+  /// the values actually mean.
+  ///
+  /// TODO: why not do this computation when we first store the reassoc,
+  /// instead of doing it every time we look it up?
+  SmallVector<Level, 2> getCollapseReassociation(TensorId tid, Level dstLvl) {
+    assert(tid < getNumTensors() && "Invalid TensorId");
+    assert(collapseReassoc.size() == getNumTensors());
+    if (const auto reassoc = collapseReassoc[tid]) {
+      // TODO: store the dstLvlRank in the LoopEmitter so that we can
+      // check `dstLvl < dstLvlRank` at the top; and only here need to
+      // assert that `reassoc.size() == dstLvlRank`.
+      assert(dstLvl < reassoc.size() && "Level is out-of-bounds");
+      const auto srcLvls = reassoc[dstLvl].cast<ArrayAttr>();
+      return llvm::to_vector<2>(
+          llvm::map_range(srcLvls, [&](Attribute srcLvl) -> Level {
+            // TODO: replace this with the converter for `LevelAttr`.
+            return srcLvl.cast<IntegerAttr>().getValue().getZExtValue();
+          }));
+    }
+    return {dstLvl};
+  }
+
   /// A optional string attribute that should be attached to the loop
   /// generated by loop emitter, it might help following passes to identify
   /// loops that operates on sparse tensors more easily.
@@ -251,61 +338,72 @@ class LoopEmitter {
   /// tensor.
   bool hasOutput;
   bool isSparseOut;
+
+  //
+  // Fields which have `numTensor` many entries.
+  //
+  // TODO: switch to an AOS style to avoid any possible mismatches.
+  //
+
   /// Input and (optional) output tensors.
   std::vector<Value> tensors;
-  /// The dim type array for each tensor.
-  std::vector<std::vector<DimLevelType>> dimTypes;
-  /// Sparse iteration information (by tensor and dim). These arrays
-  /// are updated to remain current within the current loop.
-  std::vector<std::vector<Value>> pidxs;
-  std::vector<std::vector<Value>> coord;
+  /// Level-types for each `(TensorId, Level)` pair.
+  std::vector<std::vector<DimLevelType>> lvlTypes;
+  // Sparse iteration information for each `(TensorId, Level)` pair.
+  // These arrays are updated to remain current within the current loop.
+  // TODO: Clarify which of these are indexed by dstLvl vs srcLvl.
+  //
+  /// The collection of positions for a given element (one such collection
+  /// for each tensor).  This is the position analogue of the "coords"
+  /// naming convention.
+  ///
+  /// FIXME: [CLARIFY_POSITS_LVL] It's unclear which levels are used
+  /// to index the `posits` array.  On the one hand `genSparseCrd`
+  /// uses dstLvl; on the other hand `enterLoopOverTensorAtLvl`,
+  /// `prepareLoopOverTensorAtLvl`, and `enterCoIterationOverTensorsAtLvls`
+  /// uses srcLvl.  So which is it?
+  std::vector<std::vector<Value>> posits;
+  /// The collection of coordinates for a given element (one such
+  /// collection for each tensor).
+  std::vector<std::vector<Value>> coords;
+  // The segment upper bound for non-uniques level after de-duplication.
+  std::vector<std::vector<Value>> segHi;
   std::vector<std::vector<Value>> highs;
   std::vector<std::vector<Value>> lvlSizes;
-  std::vector<std::vector<Value>> ptrBuffer; // to_pointers
-  std::vector<std::vector<Value>> idxBuffer; // to_indices
-  std::vector<Value> valBuffer;              // to_value
+  std::vector<std::vector<Value>> positionsBuffers;   // to_positions
+  std::vector<std::vector<Value>> coordinatesBuffers; // to_coordinates
+  std::vector<Value> valBuffer;                       // to_value
 
   /// Whether the sparse input is a slice.
   std::vector<bool> isSparseSlices;
+  /// Values related to slices.
+  std::vector<std::vector<Value>> sliceOffsets;
+  std::vector<std::vector<Value>> sliceStrides;
+
+  /// Collapse Reassociations related to a specific tensor
+  // TODO: support expand.
+  std::vector<ArrayAttr> collapseReassoc;
+
+  /// TODO: not yet used, it should track the current level for each tensor
+  /// to help eliminate `lvls` paramters from above APIs.
+  /// std::vector<Level> curLvl;
+
+  //
+  // Fields which have at most `numLoops` many entries.
+  //
 
   /// Loop Stack, stores the information of all the nested loops that are
   /// alive.
-  std::vector<LoopLevelInfo> loopStack;
+  std::vector<LoopInfo> loopStack;
 
-  /// Loop Sequence Stack, stores the unversial index for the current loop
+  /// Loop Sequence Stack, stores the universal index for the current loop
   /// sequence.
   std::vector<Value> loopSeqStack;
 
-  /// Maps AffineDimExpr to the index of the loop in loopStack.
+  /// Maps `LoopId` (used by `AffineDimExpr`) to `LoopOrd` (in the `loopStack`).
   /// TODO: We should probably use a callback function here to make it more
   /// general.
-  std::vector<unsigned> sparsiferLoopLvlMap;
-
-  //
-  // View based reshape related-fields and methods
-  //
-
-  /// Collapse Reassociations related to a specific tensor
-  // TODO: support expand.
-  std::vector<ArrayAttr> collapseReassoc;
-
-  /// Get the collapse reassociation for tensors[tid] on l. For unreshaped
-  /// operands, the reassociation is simply an identity transformation.
-  SmallVector<int64_t, 2> getCollapseReassociation(unsigned tid, unsigned l) {
-    // Returns for SmallVector<int64_t, 2> just like `ReassociaionIndices`
-    if (auto reass = collapseReassoc[tid]) {
-      auto attr = reass[l];
-      return llvm::to_vector<2>(
-          llvm::map_range(attr.cast<ArrayAttr>(), [&](Attribute indexAttr) {
-            return indexAttr.cast<IntegerAttr>().getInt();
-          }));
-    }
-    return {l};
-  }
-
-  /// TODO: not yet used, it should track the current level for each tensor
-  /// to help eliminate `dim` paramters from above APIs.
-  /// std::vector<size_t> curLv;
+  std::vector<LoopOrd> loopIdToOrd;
 };
 
 } // namespace sparse_tensor
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseBufferRewriting.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseBufferRewriting.cpp
index 107d9ef7569b0..5e53dbe1cc283 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseBufferRewriting.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseBufferRewriting.cpp
@@ -34,8 +34,6 @@ static constexpr uint64_t loIdx = 0;
 static constexpr uint64_t hiIdx = 1;
 static constexpr uint64_t xStartIdx = 2;
 
-static constexpr const char kLessThanFuncNamePrefix[] = "_sparse_less_than_";
-static constexpr const char kCompareEqFuncNamePrefix[] = "_sparse_compare_eq_";
 static constexpr const char kPartitionFuncNamePrefix[] = "_sparse_partition_";
 static constexpr const char kBinarySearchFuncNamePrefix[] =
     "_sparse_binary_search_";
@@ -181,27 +179,24 @@ static void createSwap(OpBuilder &builder, Location loc, ValueRange args,
   forEachIJPairInAllBuffers(builder, loc, args, nx, ny, isCoo, swapOnePair);
 }
 
-/// Creates a function to compare all the (xs[i], xs[j]) pairs. The method to
-/// compare each pair is create via `compareBuilder`.
-static void createCompareFuncImplementation(
-    OpBuilder &builder, ModuleOp unused, func::FuncOp func, uint64_t nx,
-    uint64_t ny, bool isCoo,
-    function_ref<scf::IfOp(OpBuilder &, Location, Value, Value, Value, bool)>
+/// Creates code to compare all the (xs[i], xs[j]) pairs. The method to compare
+/// each pair is create via `compareBuilder`.
+static Value createInlinedCompareImplementation(
+    OpBuilder &builder, Location loc, ValueRange args, uint64_t nx, uint64_t ny,
+    bool isCoo,
+    function_ref<Value(OpBuilder &, Location, Value, Value, Value, bool, bool)>
         compareBuilder) {
-  OpBuilder::InsertionGuard insertionGuard(builder);
-
-  Block *entryBlock = func.addEntryBlock();
-  builder.setInsertionPointToStart(entryBlock);
-  Location loc = func.getLoc();
-  ValueRange args = entryBlock->getArguments();
-
-  scf::IfOp topIfOp;
+  Value result;
   auto bodyBuilder = [&](uint64_t k, Value i, Value j, Value buffer) {
-    scf::IfOp ifOp = compareBuilder(builder, loc, i, j, buffer, (k == nx - 1));
-    if (k == 0) {
-      topIfOp = ifOp;
-    } else {
+    bool isFirstDim = (k == 0);
+    bool isLastDim = (k == nx - 1);
+    Value val =
+        compareBuilder(builder, loc, i, j, buffer, isFirstDim, isLastDim);
+    if (isFirstDim) {
+      result = val;
+    } else if (!isLastDim) {
       OpBuilder::InsertionGuard insertionGuard(builder);
+      auto ifOp = cast<scf::IfOp>(val.getDefiningOp());
       builder.setInsertionPointAfter(ifOp);
       builder.create<scf::YieldOp>(loc, ifOp.getResult(0));
     }
@@ -209,38 +204,44 @@ static void createCompareFuncImplementation(
 
   forEachIJPairInXs(builder, loc, args, nx, ny, isCoo, bodyBuilder);
 
-  builder.setInsertionPointAfter(topIfOp);
-  builder.create<func::ReturnOp>(loc, topIfOp.getResult(0));
+  builder.setInsertionPointAfterValue(result);
+  return result;
 }
 
-/// Generates an if-statement to compare whether x[i] is equal to x[j].
-static scf::IfOp createEqCompare(OpBuilder &builder, Location loc, Value i,
-                                 Value j, Value x, bool isLastDim) {
-  Value f = constantI1(builder, loc, false);
-  Value t = constantI1(builder, loc, true);
+/// Generates code to compare whether x[i] is equal to x[j] and returns the
+/// result of the comparison.
+static Value createEqCompare(OpBuilder &builder, Location loc, Value i, Value j,
+                             Value x, bool isFirstDim, bool isLastDim) {
   Value vi = builder.create<memref::LoadOp>(loc, x, i);
   Value vj = builder.create<memref::LoadOp>(loc, x, j);
 
-  Value cond =
-      builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::eq, vi, vj);
-  scf::IfOp ifOp =
-      builder.create<scf::IfOp>(loc, f.getType(), cond, /*else=*/true);
-
-  // x[1] != x[j]:
-  builder.setInsertionPointToStart(&ifOp.getElseRegion().front());
-  builder.create<scf::YieldOp>(loc, f);
+  Value res;
+  if (isLastDim) {
+    res = builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::eq, vi, vj);
+    // For 1D, we create a compare without any control flow. Otherwise, we
+    // create YieldOp to return the result in the nested if-stmt.
+    if (!isFirstDim)
+      builder.create<scf::YieldOp>(loc, res);
+  } else {
+    Value ne =
+        builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ne, vi, vj);
+    scf::IfOp ifOp = builder.create<scf::IfOp>(loc, builder.getIntegerType(1),
+                                               ne, /*else=*/true);
+    // If (x[i] != x[j]).
+    builder.setInsertionPointToStart(&ifOp.getThenRegion().front());
+    Value f = constantI1(builder, loc, false);
+    builder.create<scf::YieldOp>(loc, f);
 
-  // x[i] == x[j]:
-  builder.setInsertionPointToStart(&ifOp.getThenRegion().front());
-  if (isLastDim == 1) {
-    // Finish checking all dimensions.
-    builder.create<scf::YieldOp>(loc, t);
+    // If (x[i] == x[j]). Set up the insertion point for the nested if-stmt that
+    // checks the remaining dimensions.
+    builder.setInsertionPointToStart(&ifOp.getElseRegion().front());
+    res = ifOp.getResult(0);
   }
 
-  return ifOp;
+  return res;
 }
 
-/// Creates a function to compare whether xs[i] is equal to xs[j].
+/// Creates code to compare whether xs[i] is equal to xs[j].
 //
 // The generate IR corresponds to this C like algorithm:
 //   if (x0[i] != x0[j])
@@ -250,77 +251,68 @@ static scf::IfOp createEqCompare(OpBuilder &builder, Location loc, Value i,
 //       return false;
 //     else if (x2[2] != x2[j]))
 //       and so on ...
-static void createEqCompareFunc(OpBuilder &builder, ModuleOp unused,
-                                func::FuncOp func, uint64_t nx, uint64_t ny,
-                                bool isCoo, uint32_t nTrailingP = 0) {
+static Value createInlinedEqCompare(OpBuilder &builder, Location loc,
+                                    ValueRange args, uint64_t nx, uint64_t ny,
+                                    bool isCoo, uint32_t nTrailingP = 0) {
   // Compare functions don't use trailing parameters.
   (void)nTrailingP;
   assert(nTrailingP == 0);
-  createCompareFuncImplementation(builder, unused, func, nx, ny, isCoo,
-                                  createEqCompare);
+  return createInlinedCompareImplementation(builder, loc, args, nx, ny, isCoo,
+                                            createEqCompare);
 }
 
-/// Generates an if-statement to compare whether x[i] is less than x[j].
-static scf::IfOp createLessThanCompare(OpBuilder &builder, Location loc,
-                                       Value i, Value j, Value x,
-                                       bool isLastDim) {
-  Value f = constantI1(builder, loc, false);
-  Value t = constantI1(builder, loc, true);
+/// Generates code to compare whether x[i] is less than x[j] and returns the
+/// result of the comparison.
+static Value createLessThanCompare(OpBuilder &builder, Location loc, Value i,
+                                   Value j, Value x, bool isFirstDim,
+                                   bool isLastDim) {
   Value vi = builder.create<memref::LoadOp>(loc, x, i);
   Value vj = builder.create<memref::LoadOp>(loc, x, j);
 
-  Value cond =
-      builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ult, vi, vj);
-  scf::IfOp ifOp =
-      builder.create<scf::IfOp>(loc, f.getType(), cond, /*else=*/true);
-  // If (x[i] < x[j]).
-  builder.setInsertionPointToStart(&ifOp.getThenRegion().front());
-  builder.create<scf::YieldOp>(loc, t);
-
-  builder.setInsertionPointToStart(&ifOp.getElseRegion().front());
-  if (isLastDim == 1) {
-    // Finish checking all dimensions.
-    builder.create<scf::YieldOp>(loc, f);
+  Value res;
+  if (isLastDim) {
+    res = builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ult, vi, vj);
+    // For 1D, we create a compare without any control flow. Otherwise, we
+    // create YieldOp to return the result in the nested if-stmt.
+    if (!isFirstDim)
+      builder.create<scf::YieldOp>(loc, res);
   } else {
-    cond =
-        builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ult, vj, vi);
-    scf::IfOp ifOp2 =
-        builder.create<scf::IfOp>(loc, f.getType(), cond, /*else=*/true);
-    // Otherwise if (x[j] < x[i]).
-    builder.setInsertionPointToStart(&ifOp2.getThenRegion().front());
-    builder.create<scf::YieldOp>(loc, f);
-
-    // Otherwise check the remaining dimensions.
-    builder.setInsertionPointAfter(ifOp2);
-    builder.create<scf::YieldOp>(loc, ifOp2.getResult(0));
-    // Set up the insertion point for the nested if-stmt that checks the
-    // remaining dimensions.
-    builder.setInsertionPointToStart(&ifOp2.getElseRegion().front());
+    Value ne =
+        builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ne, vi, vj);
+    scf::IfOp ifOp = builder.create<scf::IfOp>(loc, builder.getIntegerType(1),
+                                               ne, /*else=*/true);
+    // If (x[i] != x[j]).
+    builder.setInsertionPointToStart(&ifOp.getThenRegion().front());
+    Value lt =
+        builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ult, vi, vj);
+    builder.create<scf::YieldOp>(loc, lt);
+
+    // If (x[i] == x[j]). Set up the insertion point for the nested if-stmt that
+    // checks the remaining dimensions.
+    builder.setInsertionPointToStart(&ifOp.getElseRegion().front());
+    res = ifOp.getResult(0);
   }
 
-  return ifOp;
+  return res;
 }
 
-/// Creates a function to compare whether xs[i] is less than xs[j].
+/// Creates code to compare whether xs[i] is less than xs[j].
 //
 // The generate IR corresponds to this C like algorithm:
-//   if (x0[i] < x0[j])
-//     return true;
-//   else if (x0[j] < x0[i])
-//     return false;
+//   if (x0[i] != x0[j])
+//     return x0[i] < x0[j];
+//   else if (x1[j] != x1[i])
+//     return x1[i] < x1[j];
 //   else
-//     if (x1[i] < x1[j])
-//       return true;
-//     else if (x1[j] < x1[i]))
 //       and so on ...
-static void createLessThanFunc(OpBuilder &builder, ModuleOp unused,
-                               func::FuncOp func, uint64_t nx, uint64_t ny,
-                               bool isCoo, uint32_t nTrailingP = 0) {
+static Value createInlinedLessThan(OpBuilder &builder, Location loc,
+                                   ValueRange args, uint64_t nx, uint64_t ny,
+                                   bool isCoo, uint32_t nTrailingP = 0) {
   // Compare functions don't use trailing parameters.
   (void)nTrailingP;
   assert(nTrailingP == 0);
-  createCompareFuncImplementation(builder, unused, func, nx, ny, isCoo,
-                                  createLessThanCompare);
+  return createInlinedCompareImplementation(builder, loc, args, nx, ny, isCoo,
+                                            createLessThanCompare);
 }
 
 /// Creates a function to use a binary search to find the insertion point for
@@ -379,15 +371,8 @@ static void createBinarySearchFunc(OpBuilder &builder, ModuleOp module,
   uint64_t numXBuffers = isCoo ? 1 : nx;
   compareOperands.append(args.begin() + xStartIdx,
                          args.begin() + xStartIdx + numXBuffers);
-  Type i1Type = IntegerType::get(module.getContext(), 1, IntegerType::Signless);
-  FlatSymbolRefAttr lessThanFunc = getMangledSortHelperFunc(
-      builder, func, {i1Type}, kLessThanFuncNamePrefix, nx, ny, isCoo,
-      compareOperands, createLessThanFunc, nTrailingP);
-  Value cond2 = builder
-                    .create<func::CallOp>(loc, lessThanFunc, TypeRange{i1Type},
-                                          compareOperands)
-                    .getResult(0);
-
+  Value cond2 =
+      createInlinedLessThan(builder, loc, compareOperands, nx, ny, isCoo);
   // Update lo and hi for the WhileOp as follows:
   //   if (xs[p] < xs[mid]))
   //     hi = mid;
@@ -428,15 +413,8 @@ createScanLoop(OpBuilder &builder, ModuleOp module, func::FuncOp func,
     compareOperands.push_back(before->getArgument(0));
   }
   compareOperands.append(xs.begin(), xs.end());
-  MLIRContext *context = module.getContext();
-  Type i1Type = IntegerType::get(context, 1, IntegerType::Signless);
-  FlatSymbolRefAttr lessThanFunc = getMangledSortHelperFunc(
-      builder, func, {i1Type}, kLessThanFuncNamePrefix, nx, ny, isCoo,
-      compareOperands, createLessThanFunc);
-  Value cond = builder
-                   .create<func::CallOp>(loc, lessThanFunc, TypeRange{i1Type},
-                                         compareOperands)
-                   .getResult(0);
+  Value cond =
+      createInlinedLessThan(builder, loc, compareOperands, nx, ny, isCoo);
   builder.create<scf::ConditionOp>(loc, cond, before->getArguments());
 
   Block *after =
@@ -450,31 +428,94 @@ createScanLoop(OpBuilder &builder, ModuleOp module, func::FuncOp func,
   builder.setInsertionPointAfter(whileOp);
   compareOperands[0] = i;
   compareOperands[1] = p;
-  FlatSymbolRefAttr compareEqFunc = getMangledSortHelperFunc(
-      builder, func, {i1Type}, kCompareEqFuncNamePrefix, nx, ny, isCoo,
-      compareOperands, createEqCompareFunc);
   Value compareEq =
-      builder
-          .create<func::CallOp>(loc, compareEqFunc, TypeRange{i1Type},
-                                compareOperands)
-          .getResult(0);
+      createInlinedEqCompare(builder, loc, compareOperands, nx, ny, isCoo);
 
   return std::make_pair(whileOp.getResult(0), compareEq);
 }
 
-/// Creates a code block to swap the values so that data[mi] is the median among
-/// data[lo], data[hi], and data[mi].
-//  The generated code corresponds to this C-like algorithm:
-//  median = mi
-//  if (data[mi] < data[lo]).                               (if1)
-//    if (data[hi] < data[lo])                              (if2)
-//       median = data[hi] < data[mi] ? mi : hi
-//    else
-//       median = lo
-//  else
-//    if data[hi] < data[mi]                                (if3)
-//      median = data[hi] < data[lo] ? lo : hi
-//  if median != mi swap data[median] with data[mi]
+/// Creates and returns an IfOp to compare two elements and swap the elements
+/// if compareFunc(data[b], data[a]) returns true. The new insertion point is
+/// right after the swap instructions.
+static scf::IfOp createCompareThenSwap(OpBuilder &builder, Location loc,
+                                       uint64_t nx, uint64_t ny, bool isCoo,
+                                       SmallVectorImpl<Value> &swapOperands,
+                                       SmallVectorImpl<Value> &compareOperands,
+                                       Value a, Value b) {
+  // Compare(data[b], data[a]).
+  compareOperands[0] = b;
+  compareOperands[1] = a;
+  Value cond =
+      createInlinedLessThan(builder, loc, compareOperands, nx, ny, isCoo);
+  scf::IfOp ifOp = builder.create<scf::IfOp>(loc, cond, /*else=*/false);
+  builder.setInsertionPointToStart(&ifOp.getThenRegion().front());
+  swapOperands[0] = b;
+  swapOperands[1] = a;
+  createSwap(builder, loc, swapOperands, nx, ny, isCoo);
+  return ifOp;
+}
+
+/// Creates code to insert the 3rd element to a list of two sorted elements.
+static void createInsert3rd(OpBuilder &builder, Location loc, uint64_t nx,
+                            uint64_t ny, bool isCoo,
+                            SmallVectorImpl<Value> &swapOperands,
+                            SmallVectorImpl<Value> &compareOperands, Value v0,
+                            Value v1, Value v2) {
+  scf::IfOp ifOp = createCompareThenSwap(builder, loc, nx, ny, isCoo,
+                                         swapOperands, compareOperands, v1, v2);
+  createCompareThenSwap(builder, loc, nx, ny, isCoo, swapOperands,
+                        compareOperands, v0, v1);
+  builder.setInsertionPointAfter(ifOp);
+}
+
+/// Creates code to sort 3 elements.
+static void createSort3(OpBuilder &builder, Location loc, uint64_t nx,
+                        uint64_t ny, bool isCoo,
+                        SmallVectorImpl<Value> &swapOperands,
+                        SmallVectorImpl<Value> &compareOperands, Value v0,
+                        Value v1, Value v2) {
+  // Sort the first 2 elements.
+  scf::IfOp ifOp1 = createCompareThenSwap(
+      builder, loc, nx, ny, isCoo, swapOperands, compareOperands, v0, v1);
+  builder.setInsertionPointAfter(ifOp1);
+
+  // Insert the 3th element.
+  createInsert3rd(builder, loc, nx, ny, isCoo, swapOperands, compareOperands,
+                  v0, v1, v2);
+}
+
+/// Creates code to sort 5 elements.
+static void createSort5(OpBuilder &builder, Location loc, uint64_t nx,
+                        uint64_t ny, bool isCoo,
+                        SmallVectorImpl<Value> &swapOperands,
+                        SmallVectorImpl<Value> &compareOperands, Value v0,
+                        Value v1, Value v2, Value v3, Value v4) {
+  // Sort the first 3 elements.
+  createSort3(builder, loc, nx, ny, isCoo, swapOperands, compareOperands, v0,
+              v1, v2);
+
+  auto insert4th = [&]() {
+    scf::IfOp ifOp = createCompareThenSwap(
+        builder, loc, nx, ny, isCoo, swapOperands, compareOperands, v2, v3);
+    createInsert3rd(builder, loc, nx, ny, isCoo, swapOperands, compareOperands,
+                    v0, v1, v2);
+    builder.setInsertionPointAfter(ifOp);
+  };
+
+  // Insert the 4th element.
+  insert4th();
+
+  // Insert the 5th element.
+  scf::IfOp ifOp = createCompareThenSwap(builder, loc, nx, ny, isCoo,
+                                         swapOperands, compareOperands, v3, v4);
+  insert4th();
+  builder.setInsertionPointAfter(ifOp);
+}
+
+/// Creates a code block to swap the values in indices lo, mi, and hi so that
+/// data[lo], data[mi] and data[hi] are sorted in non-decreasing values. When
+/// the number of values in range [lo, hi) is more than a threshold, we also
+/// include the middle of [lo, mi) and [mi, hi) and sort a total of five values.
 static void createChoosePivot(OpBuilder &builder, ModuleOp module,
                               func::FuncOp func, uint64_t nx, uint64_t ny,
                               bool isCoo, Value lo, Value hi, Value mi,
@@ -483,70 +524,35 @@ static void createChoosePivot(OpBuilder &builder, ModuleOp module,
   uint64_t numXBuffers = isCoo ? 1 : nx;
   compareOperands.append(args.begin() + xStartIdx,
                          args.begin() + xStartIdx + numXBuffers);
-  Type i1Type = IntegerType::get(module.getContext(), 1, IntegerType::Signless);
-  SmallVector<Type, 1> cmpTypes{i1Type};
-  FlatSymbolRefAttr lessThanFunc = getMangledSortHelperFunc(
-      builder, func, cmpTypes, kLessThanFuncNamePrefix, nx, ny, isCoo,
-      compareOperands, createLessThanFunc);
-  Location loc = func.getLoc();
-  // Compare data[mi] < data[lo].
-  Value cond1 =
-      builder.create<func::CallOp>(loc, lessThanFunc, cmpTypes, compareOperands)
-          .getResult(0);
-  SmallVector<Type, 1> ifTypes{lo.getType()};
-  scf::IfOp ifOp1 =
-      builder.create<scf::IfOp>(loc, ifTypes, cond1, /*else=*/true);
-
-  // Generate an if-stmt to find the median value, assuming we already know that
-  // data[b] < data[a] and we haven't compare data[c] yet.
-  auto createFindMedian = [&](Value a, Value b, Value c) -> scf::IfOp {
-    compareOperands[0] = c;
-    compareOperands[1] = a;
-    // Compare data[c]] < data[a].
-    Value cond2 =
-        builder
-            .create<func::CallOp>(loc, lessThanFunc, cmpTypes, compareOperands)
-            .getResult(0);
-    scf::IfOp ifOp2 =
-        builder.create<scf::IfOp>(loc, ifTypes, cond2, /*else=*/true);
-    builder.setInsertionPointToStart(&ifOp2.getThenRegion().front());
-    compareOperands[0] = c;
-    compareOperands[1] = b;
-    // Compare data[c] < data[b].
-    Value cond3 =
-        builder
-            .create<func::CallOp>(loc, lessThanFunc, cmpTypes, compareOperands)
-            .getResult(0);
-    builder.create<scf::YieldOp>(
-        loc, ValueRange{builder.create<arith::SelectOp>(loc, cond3, b, c)});
-    builder.setInsertionPointToStart(&ifOp2.getElseRegion().front());
-    builder.create<scf::YieldOp>(loc, ValueRange{a});
-    return ifOp2;
-  };
-
-  builder.setInsertionPointToStart(&ifOp1.getThenRegion().front());
-  scf::IfOp ifOp2 = createFindMedian(lo, mi, hi);
-  builder.setInsertionPointAfter(ifOp2);
-  builder.create<scf::YieldOp>(loc, ValueRange{ifOp2.getResult(0)});
-
-  builder.setInsertionPointToStart(&ifOp1.getElseRegion().front());
-  scf::IfOp ifOp3 = createFindMedian(mi, lo, hi);
-
-  builder.setInsertionPointAfter(ifOp3);
-  builder.create<scf::YieldOp>(loc, ValueRange{ifOp3.getResult(0)});
-
-  builder.setInsertionPointAfter(ifOp1);
-  Value median = ifOp1.getResult(0);
-  Value cond =
-      builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ne, mi, median);
-  scf::IfOp ifOp =
-      builder.create<scf::IfOp>(loc, TypeRange(), cond, /*else=*/false);
-
-  SmallVector<Value> swapOperands{median, mi};
+  SmallVector<Value> swapOperands{mi, lo};
   swapOperands.append(args.begin() + xStartIdx, args.end());
-  builder.setInsertionPointToStart(&ifOp.getThenRegion().front());
-  createSwap(builder, loc, swapOperands, nx, ny, isCoo);
-  builder.setInsertionPointAfter(ifOp);
+  Location loc = func.getLoc();
+  Value c1 = constantIndex(builder, loc, 1);
+  Value hiP1 = builder.create<arith::AddIOp>(loc, hi, c1);
+  Value len = builder.create<arith::SubIOp>(loc, hiP1, lo);
+  Value lenThreshold = constantIndex(builder, loc, 1000);
+  Value lenCond = builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ult,
+                                                len, lenThreshold);
+  scf::IfOp lenIf = builder.create<scf::IfOp>(loc, lenCond, /*else=*/true);
+
+  // When len < 1000, choose pivot from median of 3 values.
+  builder.setInsertionPointToStart(&lenIf.getThenRegion().front());
+  createSort3(builder, loc, nx, ny, isCoo, swapOperands, compareOperands, lo,
+              mi, hi);
+
+  // When len >= 1000, choose pivot from median of 5 values.
+  builder.setInsertionPointToStart(&lenIf.getElseRegion().front());
+  Value miP1 = builder.create<arith::AddIOp>(loc, hi, c1);
+  Value a = builder.create<arith::AddIOp>(loc, lo, miP1);
+  // Value a is the middle between [loc, mi].
+  a = builder.create<arith::ShRUIOp>(loc, a, c1);
+  Value b = builder.create<arith::AddIOp>(loc, mi, hiP1);
+  // Value b is the middle between [mi, hi].
+  b = builder.create<arith::ShRUIOp>(loc, b, c1);
+  createSort5(builder, loc, nx, ny, isCoo, swapOperands, compareOperands, lo, a,
+              mi, b, hi);
+
+  builder.setInsertionPointAfter(lenIf);
 }
 
 /// Creates a function to perform quick sort partition on the values in the
@@ -758,10 +764,6 @@ static void createShiftDownFunc(OpBuilder &builder, ModuleOp module,
   uint64_t numXBuffers = isCoo ? 1 : nx;
   compareOperands.append(args.begin() + xStartIdx,
                          args.begin() + xStartIdx + numXBuffers);
-  Type i1Type = IntegerType::get(module.getContext(), 1, IntegerType::Signless);
-  FlatSymbolRefAttr lessThanFunc = getMangledSortHelperFunc(
-      builder, func, {i1Type}, kLessThanFuncNamePrefix, nx, ny, isCoo,
-      compareOperands, createLessThanFunc);
 
   // Generate code to inspect the children of 'r' and return the larger child
   // as follows:
@@ -784,10 +786,8 @@ static void createShiftDownFunc(OpBuilder &builder, ModuleOp module,
     // Compare data[left] < data[right].
     compareOperands[0] = lChildIdx;
     compareOperands[1] = rChildIdx;
-    Value cond2 = builder
-                      .create<func::CallOp>(loc, lessThanFunc,
-                                            TypeRange{i1Type}, compareOperands)
-                      .getResult(0);
+    Value cond2 =
+        createInlinedLessThan(builder, loc, compareOperands, nx, ny, isCoo);
     scf::IfOp if2 =
         builder.create<scf::IfOp>(loc, ifTypes, cond2, /*else=*/true);
     builder.setInsertionPointToStart(&if2.getThenRegion().front());
@@ -818,10 +818,8 @@ static void createShiftDownFunc(OpBuilder &builder, ModuleOp module,
   childIdx = before->getArgument(2);
   compareOperands[0] = start;
   compareOperands[1] = childIdx;
-  Value cond = builder
-                   .create<func::CallOp>(loc, lessThanFunc, TypeRange{i1Type},
-                                         compareOperands)
-                   .getResult(0);
+  Value cond =
+      createInlinedLessThan(builder, loc, compareOperands, nx, ny, isCoo);
   builder.create<scf::ConditionOp>(loc, cond, before->getArguments());
 
   // The after-region of the WhileOp.
@@ -918,9 +916,13 @@ static void createHeapSortFunc(OpBuilder &builder, ModuleOp module,
   builder.create<func::ReturnOp>(loc);
 }
 
-static void createQuickSort(OpBuilder &builder, ModuleOp module,
-                            func::FuncOp func, ValueRange args, uint64_t nx,
-                            uint64_t ny, bool isCoo, uint32_t nTrailingP) {
+/// A helper for generating code to perform quick sort. It partitions [lo, hi),
+/// recursively calls quick sort to process the smaller partition and returns
+/// the bigger partition to be processed by the enclosed while-loop.
+static std::pair<Value, Value>
+createQuickSort(OpBuilder &builder, ModuleOp module, func::FuncOp func,
+                ValueRange args, uint64_t nx, uint64_t ny, bool isCoo,
+                uint32_t nTrailingP) {
   MLIRContext *context = module.getContext();
   Location loc = func.getLoc();
   Value lo = args[loIdx];
@@ -928,20 +930,45 @@ static void createQuickSort(OpBuilder &builder, ModuleOp module,
   FlatSymbolRefAttr partitionFunc = getMangledSortHelperFunc(
       builder, func, {IndexType::get(context)}, kPartitionFuncNamePrefix, nx,
       ny, isCoo, args.drop_back(nTrailingP), createPartitionFunc);
-  auto p = builder.create<func::CallOp>(loc, partitionFunc,
-                                        TypeRange{IndexType::get(context)},
-                                        args.drop_back(nTrailingP));
-
-  SmallVector<Value> lowOperands{lo, p.getResult(0)};
-  lowOperands.append(args.begin() + xStartIdx, args.end());
-  builder.create<func::CallOp>(loc, func, lowOperands);
-
-  SmallVector<Value> highOperands{
-      builder.create<arith::AddIOp>(loc, p.getResult(0),
-                                    constantIndex(builder, loc, 1)),
-      hi};
-  highOperands.append(args.begin() + xStartIdx, args.end());
-  builder.create<func::CallOp>(loc, func, highOperands);
+  Value p = builder
+                .create<func::CallOp>(loc, partitionFunc,
+                                      TypeRange{IndexType::get(context)},
+                                      args.drop_back(nTrailingP))
+                .getResult(0);
+  Value pP1 =
+      builder.create<arith::AddIOp>(loc, p, constantIndex(builder, loc, 1));
+  Value lenLow = builder.create<arith::SubIOp>(loc, p, lo);
+  Value lenHigh = builder.create<arith::SubIOp>(loc, hi, p);
+  Value cond = builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ule,
+                                             lenLow, lenHigh);
+
+  SmallVector<Type, 2> types(2, lo.getType()); // Only two types.
+  scf::IfOp ifOp = builder.create<scf::IfOp>(loc, types, cond, /*else=*/true);
+
+  Value c0 = constantIndex(builder, loc, 0);
+  auto mayRecursion = [&](Value low, Value high, Value len) {
+    Value cond =
+        builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ne, len, c0);
+    scf::IfOp ifOp = builder.create<scf::IfOp>(loc, cond, /*else=*/false);
+    builder.setInsertionPointToStart(&ifOp.getThenRegion().front());
+    SmallVector<Value> operands{low, high};
+    operands.append(args.begin() + xStartIdx, args.end());
+    builder.create<func::CallOp>(loc, func, operands);
+    builder.setInsertionPointAfter(ifOp);
+  };
+
+  // Recursively call quickSort to process the smaller partition and return
+  // the bigger partition to be processed by the enclosed while-loop.
+  builder.setInsertionPointToStart(&ifOp.getThenRegion().front());
+  mayRecursion(lo, p, lenLow);
+  builder.create<scf::YieldOp>(loc, ValueRange{pP1, hi});
+
+  builder.setInsertionPointToStart(&ifOp.getElseRegion().front());
+  mayRecursion(pP1, hi, lenHigh);
+  builder.create<scf::YieldOp>(loc, ValueRange{lo, p});
+
+  builder.setInsertionPointAfter(ifOp);
+  return std::make_pair(ifOp.getResult(0), ifOp.getResult(1));
 }
 
 /// Creates a function to perform insertion sort on the values in the range of
@@ -1036,16 +1063,21 @@ static void createSortStableFunc(OpBuilder &builder, ModuleOp module,
 //
 // When nTrailingP == 0, the generated IR corresponds to this C like algorithm:
 // void quickSort(lo, hi, data) {
-//   if (lo + 1 < hi) {
+//   while (lo + 1 < hi) {
 //        p = partition(low, high, data);
-//        quickSort(lo, p, data);
-//        quickSort(p + 1, hi, data);
+//        if (len(lo, p) < len(p+1, hi)) {
+//          quickSort(lo, p, data);
+//          lo = p+1;
+//        } else {
+//          quickSort(p + 1, hi, data);
+//          hi = p;
+//        }
 //   }
 // }
 //
 // When nTrailingP == 1, the generated IR corresponds to this C like algorithm:
 // void hybridQuickSort(lo, hi, data, depthLimit) {
-//   if (lo + 1 < hi) {
+//   while (lo + 1 < hi) {
 //     len = hi - lo;
 //     if (len <= limit) {
 //       insertionSort(lo, hi, data);
@@ -1055,10 +1087,14 @@ static void createSortStableFunc(OpBuilder &builder, ModuleOp module,
 //         heapSort(lo, hi, data);
 //       } else {
 //          p = partition(low, high, data);
-//          quickSort(lo, p, data);
-//          quickSort(p + 1, hi, data);
+//          if (len(lo, p) < len(p+1, hi)) {
+//            quickSort(lo, p, data, depthLimit);
+//            lo = p+1;
+//          } else {
+//            quickSort(p + 1, hi, data, depthLimit);
+//            hi = p;
+//          }
 //       }
-//       depthLimit ++;
 //     }
 //   }
 // }
@@ -1073,70 +1109,98 @@ static void createQuickSortFunc(OpBuilder &builder, ModuleOp module,
   builder.setInsertionPointToStart(entryBlock);
 
   Location loc = func.getLoc();
-  ValueRange args = entryBlock->getArguments();
+  SmallVector<Value> args;
+  args.append(entryBlock->getArguments().begin(),
+              entryBlock->getArguments().end());
   Value lo = args[loIdx];
   Value hi = args[hiIdx];
-  Value loCmp =
+  SmallVector<Type, 2> types(2, lo.getType()); // Only two types.
+  scf::WhileOp whileOp =
+      builder.create<scf::WhileOp>(loc, types, SmallVector<Value, 2>{lo, hi});
+
+  // The before-region of the WhileOp.
+  Block *before =
+      builder.createBlock(&whileOp.getBefore(), {}, types, {loc, loc});
+  builder.setInsertionPointToEnd(before);
+  lo = before->getArgument(0);
+  hi = before->getArgument(1);
+  Value loP1 =
       builder.create<arith::AddIOp>(loc, lo, constantIndex(builder, loc, 1));
-  Value cond =
-      builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ult, loCmp, hi);
-  scf::IfOp ifOp = builder.create<scf::IfOp>(loc, cond, /*else=*/false);
+  Value needSort =
+      builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ult, loP1, hi);
+  builder.create<scf::ConditionOp>(loc, needSort, before->getArguments());
 
-  // The if-stmt true branch.
-  builder.setInsertionPointToStart(&ifOp.getThenRegion().front());
-  Value pDepthLimit;
-  Value savedDepthLimit;
-  scf::IfOp depthIf;
+  // The after-region of the WhileOp.
+  Block *after =
+      builder.createBlock(&whileOp.getAfter(), {}, types, {loc, loc});
+  builder.setInsertionPointToEnd(after);
+  lo = after->getArgument(0);
+  hi = after->getArgument(1);
+  args[0] = lo;
+  args[1] = hi;
 
   if (isHybrid) {
     Value len = builder.create<arith::SubIOp>(loc, hi, lo);
     Value lenLimit = constantIndex(builder, loc, 30);
     Value lenCond = builder.create<arith::CmpIOp>(
         loc, arith::CmpIPredicate::ule, len, lenLimit);
-    scf::IfOp lenIf = builder.create<scf::IfOp>(loc, lenCond, /*else=*/true);
+    scf::IfOp lenIf =
+        builder.create<scf::IfOp>(loc, types, lenCond, /*else=*/true);
 
     // When len <= limit.
     builder.setInsertionPointToStart(&lenIf.getThenRegion().front());
     FlatSymbolRefAttr insertionSortFunc = getMangledSortHelperFunc(
         builder, func, TypeRange(), kSortStableFuncNamePrefix, nx, ny, isCoo,
-        args.drop_back(nTrailingP), createSortStableFunc);
+        ValueRange(args).drop_back(nTrailingP), createSortStableFunc);
     builder.create<func::CallOp>(loc, insertionSortFunc, TypeRange(),
-                                 ValueRange(args.drop_back(nTrailingP)));
+                                 ValueRange(args).drop_back(nTrailingP));
+    builder.create<scf::YieldOp>(loc, ValueRange{lo, lo});
 
     // When len > limit.
     builder.setInsertionPointToStart(&lenIf.getElseRegion().front());
-    pDepthLimit = args.back();
-    savedDepthLimit = builder.create<memref::LoadOp>(loc, pDepthLimit);
-    Value depthLimit = builder.create<arith::SubIOp>(
-        loc, savedDepthLimit, constantI64(builder, loc, 1));
-    builder.create<memref::StoreOp>(loc, depthLimit, pDepthLimit);
+    Value depthLimit = args.back();
+    depthLimit = builder.create<arith::SubIOp>(loc, depthLimit,
+                                               constantI64(builder, loc, 1));
     Value depthCond =
         builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ule,
                                       depthLimit, constantI64(builder, loc, 0));
-    depthIf = builder.create<scf::IfOp>(loc, depthCond, /*else=*/true);
+    scf::IfOp depthIf =
+        builder.create<scf::IfOp>(loc, types, depthCond, /*else=*/true);
 
     // When depth exceeds limit.
     builder.setInsertionPointToStart(&depthIf.getThenRegion().front());
     FlatSymbolRefAttr heapSortFunc = getMangledSortHelperFunc(
         builder, func, TypeRange(), kHeapSortFuncNamePrefix, nx, ny, isCoo,
-        args.drop_back(nTrailingP), createHeapSortFunc);
+        ValueRange(args).drop_back(nTrailingP), createHeapSortFunc);
     builder.create<func::CallOp>(loc, heapSortFunc, TypeRange(),
-                                 ValueRange(args.drop_back(nTrailingP)));
+                                 ValueRange(args).drop_back(nTrailingP));
+    builder.create<scf::YieldOp>(loc, ValueRange{lo, lo});
 
     // When depth doesn't exceed limit.
     builder.setInsertionPointToStart(&depthIf.getElseRegion().front());
-  }
+    args.back() = depthLimit;
+    std::tie(lo, hi) =
+        createQuickSort(builder, module, func, args, nx, ny, isCoo, nTrailingP);
+    builder.create<scf::YieldOp>(loc, ValueRange{lo, hi});
 
-  createQuickSort(builder, module, func, args, nx, ny, isCoo, nTrailingP);
-
-  if (isHybrid) {
-    // Restore depthLimit.
     builder.setInsertionPointAfter(depthIf);
-    builder.create<memref::StoreOp>(loc, savedDepthLimit, pDepthLimit);
+    lo = depthIf.getResult(0);
+    hi = depthIf.getResult(1);
+    builder.create<scf::YieldOp>(loc, ValueRange{lo, hi});
+
+    builder.setInsertionPointAfter(lenIf);
+    lo = lenIf.getResult(0);
+    hi = lenIf.getResult(1);
+  } else {
+    std::tie(lo, hi) =
+        createQuickSort(builder, module, func, args, nx, ny, isCoo, nTrailingP);
   }
 
-  // After the if-stmt.
-  builder.setInsertionPointAfter(ifOp);
+  // New [lo, hi) for the next while-loop iteration.
+  builder.create<scf::YieldOp>(loc, ValueRange{lo, hi});
+
+  // After the while-loop.
+  builder.setInsertionPointAfter(whileOp);
   builder.create<func::ReturnOp>(loc);
 }
 
@@ -1171,9 +1235,6 @@ LogicalResult matchAndRewriteSortOp(OpTy op, ValueRange xys, uint64_t nx,
     funcName = kHybridQuickSortFuncNamePrefix;
     funcGenerator = createQuickSortFunc;
     nTrailingP = 1;
-    Value pDepthLimit = rewriter.create<memref::AllocaOp>(
-        loc, MemRefType::get({}, rewriter.getI64Type()));
-    operands.push_back(pDepthLimit);
     // As a heuristics, set depthLimit = 2 * log2(n).
     Value lo = operands[loIdx];
     Value hi = operands[hiIdx];
@@ -1183,9 +1244,7 @@ LogicalResult matchAndRewriteSortOp(OpTy op, ValueRange xys, uint64_t nx,
     Value depthLimit = rewriter.create<arith::SubIOp>(
         loc, constantI64(rewriter, loc, 64),
         rewriter.create<math::CountLeadingZerosOp>(loc, len));
-    depthLimit = rewriter.create<arith::ShLIOp>(loc, depthLimit,
-                                                constantI64(rewriter, loc, 1));
-    rewriter.create<memref::StoreOp>(loc, depthLimit, pDepthLimit);
+    operands.push_back(depthLimit);
     break;
   }
   case SparseTensorSortKind::QuickSort:
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseStorageSpecifierToLLVM.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseStorageSpecifierToLLVM.cpp
index caa4dd5d722f6..0c68c4db4fe95 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseStorageSpecifierToLLVM.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseStorageSpecifierToLLVM.cpp
@@ -21,20 +21,30 @@ namespace {
 // Helper methods.
 //===----------------------------------------------------------------------===//
 
-static SmallVector<Type, 2> getSpecifierFields(StorageSpecifierType tp) {
+static SmallVector<Type, 4> getSpecifierFields(StorageSpecifierType tp) {
   MLIRContext *ctx = tp.getContext();
   auto enc = tp.getEncoding();
   const Level lvlRank = enc.getLvlRank();
 
-  SmallVector<Type, 2> result;
+  SmallVector<Type, 4> result;
   // TODO: how can we get the lowering type for index type in the later pipeline
   // to be consistent? LLVM::StructureType does not allow index fields.
-  auto indexType = IntegerType::get(tp.getContext(), 64);
-  auto dimSizes = LLVM::LLVMArrayType::get(ctx, indexType, lvlRank);
-  auto memSizes = LLVM::LLVMArrayType::get(ctx, indexType,
+  auto sizeType = IntegerType::get(tp.getContext(), 64);
+  auto lvlSizes = LLVM::LLVMArrayType::get(ctx, sizeType, lvlRank);
+  auto memSizes = LLVM::LLVMArrayType::get(ctx, sizeType,
                                            getNumDataFieldsFromEncoding(enc));
-  result.push_back(dimSizes);
+  result.push_back(lvlSizes);
   result.push_back(memSizes);
+
+  if (enc.isSlice()) {
+    // Extra fields are required for the slice information.
+    auto dimOffset = LLVM::LLVMArrayType::get(ctx, sizeType, lvlRank);
+    auto dimStride = LLVM::LLVMArrayType::get(ctx, sizeType, lvlRank);
+
+    result.push_back(dimOffset);
+    result.push_back(dimStride);
+  }
+
   return result;
 }
 
@@ -47,13 +57,15 @@ static Type convertSpecifier(StorageSpecifierType tp) {
 // Specifier struct builder.
 //===----------------------------------------------------------------------===//
 
-constexpr uint64_t kDimSizePosInSpecifier = 0;
+constexpr uint64_t kLvlSizePosInSpecifier = 0;
 constexpr uint64_t kMemSizePosInSpecifier = 1;
+constexpr uint64_t kDimOffsetPosInSpecifier = 2;
+constexpr uint64_t kDimStridePosInSpecifier = 3;
 
 class SpecifierStructBuilder : public StructBuilder {
 private:
   Value extractField(OpBuilder &builder, Location loc,
-                     ArrayRef<int64_t> indices) {
+                     ArrayRef<int64_t> indices) const {
     return genCast(builder, loc,
                    builder.create<LLVM::ExtractValueOp>(loc, value, indices),
                    builder.getIndexType());
@@ -72,58 +84,133 @@ class SpecifierStructBuilder : public StructBuilder {
   }
 
   // Undef value for dimension sizes, all zero value for memory sizes.
-  static Value getInitValue(OpBuilder &builder, Location loc, Type structType);
+  static Value getInitValue(OpBuilder &builder, Location loc, Type structType,
+                            Value source);
+
+  Value lvlSize(OpBuilder &builder, Location loc, Level lvl) const;
+  void setLvlSize(OpBuilder &builder, Location loc, Level lvl, Value size);
 
-  Value dimSize(OpBuilder &builder, Location loc, unsigned dim);
-  void setDimSize(OpBuilder &builder, Location loc, unsigned dim, Value size);
+  Value dimOffset(OpBuilder &builder, Location loc, Dimension dim) const;
+  void setDimOffset(OpBuilder &builder, Location loc, Dimension dim,
+                    Value size);
 
-  Value memSize(OpBuilder &builder, Location loc, unsigned pos);
-  void setMemSize(OpBuilder &builder, Location loc, unsigned pos, Value size);
+  Value dimStride(OpBuilder &builder, Location loc, Dimension dim) const;
+  void setDimStride(OpBuilder &builder, Location loc, Dimension dim,
+                    Value size);
+
+  Value memSize(OpBuilder &builder, Location loc, FieldIndex fidx) const;
+  void setMemSize(OpBuilder &builder, Location loc, FieldIndex fidx,
+                  Value size);
+
+  Value memSizeArray(OpBuilder &builder, Location loc) const;
+  void setMemSizeArray(OpBuilder &builder, Location loc, Value array);
 };
 
 Value SpecifierStructBuilder::getInitValue(OpBuilder &builder, Location loc,
-                                           Type structType) {
+                                           Type structType, Value source) {
   Value metaData = builder.create<LLVM::UndefOp>(loc, structType);
   SpecifierStructBuilder md(metaData);
-  auto memSizeArrayType = structType.cast<LLVM::LLVMStructType>()
-                              .getBody()[kMemSizePosInSpecifier]
-                              .cast<LLVM::LLVMArrayType>();
+  if (!source) {
+    auto memSizeArrayType = structType.cast<LLVM::LLVMStructType>()
+                                .getBody()[kMemSizePosInSpecifier]
+                                .cast<LLVM::LLVMArrayType>();
+
+    Value zero = constantZero(builder, loc, memSizeArrayType.getElementType());
+    // Fill memSizes array with zero.
+    for (int i = 0, e = memSizeArrayType.getNumElements(); i < e; i++)
+      md.setMemSize(builder, loc, i, zero);
+  } else {
+    // We copy non-slice information (memory sizes array) from source
+    SpecifierStructBuilder sourceMd(source);
+    md.setMemSizeArray(builder, loc, sourceMd.memSizeArray(builder, loc));
+  }
+  return md;
+}
 
-  Value zero = constantZero(builder, loc, memSizeArrayType.getElementType());
-  // Fill memSizes array with zero.
-  for (int i = 0, e = memSizeArrayType.getNumElements(); i < e; i++)
-    md.setMemSize(builder, loc, i, zero);
+/// Builds IR extracting the pos-th offset from the descriptor.
+Value SpecifierStructBuilder::dimOffset(OpBuilder &builder, Location loc,
+                                        Dimension dim) const {
+  return extractField(
+      builder, loc,
+      ArrayRef<int64_t>{kDimOffsetPosInSpecifier, static_cast<int64_t>(dim)});
+}
 
-  return md;
+/// Builds IR inserting the pos-th offset into the descriptor.
+void SpecifierStructBuilder::setDimOffset(OpBuilder &builder, Location loc,
+                                          Dimension dim, Value size) {
+  insertField(
+      builder, loc,
+      ArrayRef<int64_t>{kDimOffsetPosInSpecifier, static_cast<int64_t>(dim)},
+      size);
 }
 
-/// Builds IR inserting the pos-th size into the descriptor.
-Value SpecifierStructBuilder::dimSize(OpBuilder &builder, Location loc,
-                                      unsigned dim) {
-  return extractField(builder, loc,
-                      ArrayRef<int64_t>{kDimSizePosInSpecifier, dim});
+/// Builds IR extracting the `lvl`-th level-size from the descriptor.
+Value SpecifierStructBuilder::lvlSize(OpBuilder &builder, Location loc,
+                                      Level lvl) const {
+  // This static_cast makes the narrowing of `lvl` explicit, as required
+  // by the braces notation for the ctor.
+  return extractField(
+      builder, loc,
+      ArrayRef<int64_t>{kLvlSizePosInSpecifier, static_cast<int64_t>(lvl)});
 }
 
-/// Builds IR inserting the pos-th size into the descriptor.
-void SpecifierStructBuilder::setDimSize(OpBuilder &builder, Location loc,
-                                        unsigned dim, Value size) {
+/// Builds IR inserting the `lvl`-th level-size into the descriptor.
+void SpecifierStructBuilder::setLvlSize(OpBuilder &builder, Location loc,
+                                        Level lvl, Value size) {
+  // This static_cast makes the narrowing of `lvl` explicit, as required
+  // by the braces notation for the ctor.
+  insertField(
+      builder, loc,
+      ArrayRef<int64_t>{kLvlSizePosInSpecifier, static_cast<int64_t>(lvl)},
+      size);
+}
+
+/// Builds IR extracting the pos-th stride from the descriptor.
+Value SpecifierStructBuilder::dimStride(OpBuilder &builder, Location loc,
+                                        Dimension dim) const {
+  return extractField(
+      builder, loc,
+      ArrayRef<int64_t>{kDimStridePosInSpecifier, static_cast<int64_t>(dim)});
+}
 
-  insertField(builder, loc, ArrayRef<int64_t>{kDimSizePosInSpecifier, dim},
-              size);
+/// Builds IR inserting the pos-th stride into the descriptor.
+void SpecifierStructBuilder::setDimStride(OpBuilder &builder, Location loc,
+                                          Dimension dim, Value size) {
+  insertField(
+      builder, loc,
+      ArrayRef<int64_t>{kDimStridePosInSpecifier, static_cast<int64_t>(dim)},
+      size);
 }
 
 /// Builds IR extracting the pos-th memory size into the descriptor.
 Value SpecifierStructBuilder::memSize(OpBuilder &builder, Location loc,
-                                      unsigned pos) {
-  return extractField(builder, loc,
-                      ArrayRef<int64_t>{kMemSizePosInSpecifier, pos});
+                                      FieldIndex fidx) const {
+  return extractField(
+      builder, loc,
+      ArrayRef<int64_t>{kMemSizePosInSpecifier, static_cast<int64_t>(fidx)});
 }
 
-/// Builds IR inserting the pos-th memory size into the descriptor.
+/// Builds IR inserting the `fidx`-th memory-size into the descriptor.
 void SpecifierStructBuilder::setMemSize(OpBuilder &builder, Location loc,
-                                        unsigned pos, Value size) {
-  insertField(builder, loc, ArrayRef<int64_t>{kMemSizePosInSpecifier, pos},
-              size);
+                                        FieldIndex fidx, Value size) {
+  insertField(
+      builder, loc,
+      ArrayRef<int64_t>{kMemSizePosInSpecifier, static_cast<int64_t>(fidx)},
+      size);
+}
+
+/// Builds IR extracting the memory size array from the descriptor.
+Value SpecifierStructBuilder::memSizeArray(OpBuilder &builder,
+                                           Location loc) const {
+  return builder.create<LLVM::ExtractValueOp>(loc, value,
+                                              kMemSizePosInSpecifier);
+}
+
+/// Builds IR inserting the memory size array into the descriptor.
+void SpecifierStructBuilder::setMemSizeArray(OpBuilder &builder, Location loc,
+                                             Value array) {
+  value = builder.create<LLVM::InsertValueOp>(loc, value, array,
+                                              kMemSizePosInSpecifier);
 }
 
 } // namespace
@@ -134,7 +221,7 @@ void SpecifierStructBuilder::setMemSize(OpBuilder &builder, Location loc,
 
 StorageSpecifierToLLVMTypeConverter::StorageSpecifierToLLVMTypeConverter() {
   addConversion([](Type type) { return type; });
-  addConversion([](StorageSpecifierType tp) { return convertSpecifier(tp); });
+  addConversion(convertSpecifier);
 }
 
 //===----------------------------------------------------------------------===//
@@ -151,22 +238,37 @@ class SpecifierGetterSetterOpConverter : public OpConversionPattern<SourceOp> {
   matchAndRewrite(SourceOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     SpecifierStructBuilder spec(adaptor.getSpecifier());
-    Value v;
-    if (op.getSpecifierKind() == StorageSpecifierKind::DimSize) {
-      v = Base::onDimSize(rewriter, op, spec,
-                          op.getDim().value().getZExtValue());
-    } else {
+    switch (op.getSpecifierKind()) {
+    case StorageSpecifierKind::LvlSize: {
+      Value v = Base::onLvlSize(rewriter, op, spec, (*op.getLevel()));
+      rewriter.replaceOp(op, v);
+      return success();
+    }
+    case StorageSpecifierKind::DimOffset: {
+      Value v = Base::onDimOffset(rewriter, op, spec, (*op.getLevel()));
+      rewriter.replaceOp(op, v);
+      return success();
+    }
+    case StorageSpecifierKind::DimStride: {
+      Value v = Base::onDimStride(rewriter, op, spec, (*op.getLevel()));
+      rewriter.replaceOp(op, v);
+      return success();
+    }
+    case StorageSpecifierKind::CrdMemSize:
+    case StorageSpecifierKind::PosMemSize:
+    case StorageSpecifierKind::ValMemSize: {
       auto enc = op.getSpecifier().getType().getEncoding();
       StorageLayout layout(enc);
-      std::optional<unsigned> dim;
-      if (op.getDim())
-        dim = op.getDim().value().getZExtValue();
-      unsigned idx = layout.getMemRefFieldIndex(op.getSpecifierKind(), dim);
-      v = Base::onMemSize(rewriter, op, spec, idx);
+      std::optional<unsigned> lvl;
+      if (op.getLevel())
+        lvl = (*op.getLevel());
+      unsigned idx = layout.getMemRefFieldIndex(op.getSpecifierKind(), lvl);
+      Value v = Base::onMemSize(rewriter, op, spec, idx);
+      rewriter.replaceOp(op, v);
+      return success();
     }
-
-    rewriter.replaceOp(op, v);
-    return success();
+    }
+    llvm_unreachable("unrecognized specifer kind");
   }
 };
 
@@ -174,15 +276,28 @@ struct StorageSpecifierSetOpConverter
     : public SpecifierGetterSetterOpConverter<StorageSpecifierSetOpConverter,
                                               SetStorageSpecifierOp> {
   using SpecifierGetterSetterOpConverter::SpecifierGetterSetterOpConverter;
-  static Value onDimSize(OpBuilder &builder, SetStorageSpecifierOp op,
-                         SpecifierStructBuilder &spec, unsigned d) {
-    spec.setDimSize(builder, op.getLoc(), d, op.getValue());
+
+  static Value onLvlSize(OpBuilder &builder, SetStorageSpecifierOp op,
+                         SpecifierStructBuilder &spec, Level lvl) {
+    spec.setLvlSize(builder, op.getLoc(), lvl, op.getValue());
+    return spec;
+  }
+
+  static Value onDimOffset(OpBuilder &builder, SetStorageSpecifierOp op,
+                           SpecifierStructBuilder &spec, Dimension d) {
+    spec.setDimOffset(builder, op.getLoc(), d, op.getValue());
+    return spec;
+  }
+
+  static Value onDimStride(OpBuilder &builder, SetStorageSpecifierOp op,
+                           SpecifierStructBuilder &spec, Dimension d) {
+    spec.setDimStride(builder, op.getLoc(), d, op.getValue());
     return spec;
   }
 
   static Value onMemSize(OpBuilder &builder, SetStorageSpecifierOp op,
-                         SpecifierStructBuilder &spec, unsigned i) {
-    spec.setMemSize(builder, op.getLoc(), i, op.getValue());
+                         SpecifierStructBuilder &spec, FieldIndex fidx) {
+    spec.setMemSize(builder, op.getLoc(), fidx, op.getValue());
     return spec;
   }
 };
@@ -191,13 +306,25 @@ struct StorageSpecifierGetOpConverter
     : public SpecifierGetterSetterOpConverter<StorageSpecifierGetOpConverter,
                                               GetStorageSpecifierOp> {
   using SpecifierGetterSetterOpConverter::SpecifierGetterSetterOpConverter;
-  static Value onDimSize(OpBuilder &builder, GetStorageSpecifierOp op,
-                         SpecifierStructBuilder &spec, unsigned d) {
-    return spec.dimSize(builder, op.getLoc(), d);
+
+  static Value onLvlSize(OpBuilder &builder, GetStorageSpecifierOp op,
+                         SpecifierStructBuilder &spec, Level lvl) {
+    return spec.lvlSize(builder, op.getLoc(), lvl);
   }
+
+  static Value onDimOffset(OpBuilder &builder, GetStorageSpecifierOp op,
+                           const SpecifierStructBuilder &spec, Dimension d) {
+    return spec.dimOffset(builder, op.getLoc(), d);
+  }
+
+  static Value onDimStride(OpBuilder &builder, GetStorageSpecifierOp op,
+                           const SpecifierStructBuilder &spec, Dimension d) {
+    return spec.dimStride(builder, op.getLoc(), d);
+  }
+
   static Value onMemSize(OpBuilder &builder, GetStorageSpecifierOp op,
-                         SpecifierStructBuilder &spec, unsigned i) {
-    return spec.memSize(builder, op.getLoc(), i);
+                         SpecifierStructBuilder &spec, FieldIndex fidx) {
+    return spec.memSize(builder, op.getLoc(), fidx);
   }
 };
 
@@ -209,8 +336,9 @@ struct StorageSpecifierInitOpConverter
   matchAndRewrite(StorageSpecifierInitOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     Type llvmType = getTypeConverter()->convertType(op.getResult().getType());
-    rewriter.replaceOp(op, SpecifierStructBuilder::getInitValue(
-                               rewriter, op.getLoc(), llvmType));
+    rewriter.replaceOp(
+        op, SpecifierStructBuilder::getInitValue(
+                rewriter, op.getLoc(), llvmType, adaptor.getSource()));
     return success();
   }
 };
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorCodegen.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorCodegen.cpp
index 7da6e0f996f92..ae6e40f1c19cb 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorCodegen.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorCodegen.cpp
@@ -18,6 +18,9 @@
 #include "CodegenUtils.h"
 #include "SparseTensorStorageLayout.h"
 
+#include "llvm/Support/FormatVariadic.h"
+
+#include "mlir/Dialect/Arith/Utils/Utils.h"
 #include "mlir/Dialect/Bufferization/IR/Bufferization.h"
 #include "mlir/Dialect/Func/IR/FuncOps.h"
 #include "mlir/Dialect/Linalg/Utils/Utils.h"
@@ -28,7 +31,6 @@
 #include "mlir/Dialect/SparseTensor/Transforms/Passes.h"
 #include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Transforms/DialectConversion.h"
-#include "llvm/Support/FormatVariadic.h"
 
 #include <optional>
 
@@ -67,13 +69,13 @@ static void flattenOperands(ValueRange operands,
   }
 }
 
-/// Generates a load with proper index typing.
+/// Generates a load with proper `index` typing.
 static Value genLoad(OpBuilder &builder, Location loc, Value mem, Value idx) {
   idx = genCast(builder, loc, idx, builder.getIndexType());
   return builder.create<memref::LoadOp>(loc, mem, idx);
 }
 
-/// Generates a store with proper index typing and (for indices) proper value.
+/// Generates a store with proper `index` typing and proper value.
 static void genStore(OpBuilder &builder, Location loc, Value val, Value mem,
                      Value idx) {
   idx = genCast(builder, loc, idx, builder.getIndexType());
@@ -111,8 +113,7 @@ static Value sizeFromTensorAtDim(OpBuilder &builder, Location loc,
   // accounting for the reordering applied to the sparse storage.
   // FIXME: `toStoredDim` is deprecated.
   const Level lvl = toStoredDim(stt, dim);
-  // FIXME: this method seems to get *level* sizes, but the name is confusing
-  return desc.getDimSize(builder, loc, lvl);
+  return desc.getLvlSize(builder, loc, lvl);
 }
 
 // Gets the dimension size at the given stored level 'lvl', either as a
@@ -150,13 +151,12 @@ static void allocSchemeForRank(OpBuilder &builder, Location loc,
   for (Level l = startLvl; l < lvlRank; l++) {
     const auto dlt = stt.getLvlType(l);
     if (isCompressedDLT(dlt)) {
-      // Append linear x pointers, initialized to zero. Since each compressed
+      // Append linear x positions, initialized to zero. Since each compressed
       // dimension initially already has a single zero entry, this maintains
       // the desired "linear + 1" length property at all times.
-      Type ptrType = stt.getPointerType();
-      Value ptrZero = constantZero(builder, loc, ptrType);
-      createPushback(builder, loc, desc, SparseTensorFieldKind::PtrMemRef, l,
-                     ptrZero, linear);
+      Value posZero = constantZero(builder, loc, stt.getPosType());
+      createPushback(builder, loc, desc, SparseTensorFieldKind::PosMemRef, l,
+                     posZero, linear);
       return;
     }
     if (isSingletonDLT(dlt)) {
@@ -215,32 +215,32 @@ static void createAllocFields(OpBuilder &builder, Location loc,
   // size based on available information. Otherwise we just
   // initialize a few elements to start the reallocation chain.
   // TODO: refine this
-  Value ptrHeuristic, idxHeuristic, valHeuristic;
+  Value posHeuristic, crdHeuristic, valHeuristic;
   if (stt.isAllDense()) {
     valHeuristic = dimSizes[0];
     for (const Value sz : ArrayRef<Value>{dimSizes}.drop_front())
       valHeuristic = builder.create<arith::MulIOp>(loc, valHeuristic, sz);
   } else if (sizeHint) {
     if (getCOOStart(stt.getEncoding()) == 0) {
-      ptrHeuristic = constantIndex(builder, loc, 2);
-      idxHeuristic = builder.create<arith::MulIOp>(
+      posHeuristic = constantIndex(builder, loc, 2);
+      crdHeuristic = builder.create<arith::MulIOp>(
           loc, constantIndex(builder, loc, dimRank), sizeHint); // AOS
     } else if (dimRank == 2 && stt.isDenseLvl(0) && stt.isCompressedLvl(1)) {
-      ptrHeuristic = builder.create<arith::AddIOp>(
+      posHeuristic = builder.create<arith::AddIOp>(
           loc, sizeHint, constantIndex(builder, loc, 1));
-      idxHeuristic = sizeHint;
+      crdHeuristic = sizeHint;
     } else {
-      ptrHeuristic = idxHeuristic = constantIndex(builder, loc, 16);
+      posHeuristic = crdHeuristic = constantIndex(builder, loc, 16);
     }
     valHeuristic = sizeHint;
   } else {
-    ptrHeuristic = idxHeuristic = valHeuristic =
+    posHeuristic = crdHeuristic = valHeuristic =
         constantIndex(builder, loc, 16);
   }
 
   foreachFieldAndTypeInSparseTensor(
       stt,
-      [&builder, &fields, stt, loc, ptrHeuristic, idxHeuristic, valHeuristic,
+      [&builder, &fields, stt, loc, posHeuristic, crdHeuristic, valHeuristic,
        enableInit](Type fType, FieldIndex fIdx, SparseTensorFieldKind fKind,
                    Level /*lvl*/, DimLevelType /*dlt*/) -> bool {
         assert(fields.size() == fIdx);
@@ -249,13 +249,13 @@ static void createAllocFields(OpBuilder &builder, Location loc,
         case SparseTensorFieldKind::StorageSpec:
           field = SparseTensorSpecifier::getInitValue(builder, loc, stt);
           break;
-        case SparseTensorFieldKind::PtrMemRef:
-        case SparseTensorFieldKind::IdxMemRef:
+        case SparseTensorFieldKind::PosMemRef:
+        case SparseTensorFieldKind::CrdMemRef:
         case SparseTensorFieldKind::ValMemRef:
           field = createAllocation(
               builder, loc, fType.cast<MemRefType>(),
-              (fKind == SparseTensorFieldKind::PtrMemRef)   ? ptrHeuristic
-              : (fKind == SparseTensorFieldKind::IdxMemRef) ? idxHeuristic
+              (fKind == SparseTensorFieldKind::PosMemRef)   ? posHeuristic
+              : (fKind == SparseTensorFieldKind::CrdMemRef) ? crdHeuristic
                                                             : valHeuristic,
               enableInit);
           break;
@@ -269,87 +269,89 @@ static void createAllocFields(OpBuilder &builder, Location loc,
   MutSparseTensorDescriptor desc(stt, fields);
 
   // Initialize the storage scheme to an empty tensor. Initialized memSizes
-  // to all zeros, sets the dimSizes to known values and gives all pointer
+  // to all zeros, sets the dimSizes to known values and gives all position
   // fields an initial zero entry, so that it is easier to maintain the
   // "linear + 1" length property.
-  Value ptrZero = constantZero(builder, loc, stt.getPointerType());
+  Value posZero = constantZero(builder, loc, stt.getPosType());
   for (Level lvlRank = stt.getLvlRank(), l = 0; l < lvlRank; l++) {
     // Fills dim sizes array.
-    // FIXME: this method seems to set *level* sizes, but the name is confusing
     // FIXME: `toOrigDim` is deprecated.
-    desc.setDimSize(builder, loc, l, dimSizes[toOrigDim(stt, l)]);
-    // Pushes a leading zero to pointers memref.
+    desc.setLvlSize(builder, loc, l, dimSizes[toOrigDim(stt, l)]);
+    // Pushes a leading zero to positions memref.
     if (stt.isCompressedLvl(l))
-      createPushback(builder, loc, desc, SparseTensorFieldKind::PtrMemRef, l,
-                     ptrZero);
+      createPushback(builder, loc, desc, SparseTensorFieldKind::PosMemRef, l,
+                     posZero);
   }
   allocSchemeForRank(builder, loc, desc, /*rank=*/0);
 }
 
 /// Helper method that generates block specific to compressed case:
 ///
-///  plo = pointers[l][pos[l-1]]
-///  phi = pointers[l][pos[l-1]+1]
-///  msz = indices[l].size()
-///  if (plo < phi) {
-///    present = indices[l][phi-1] == i[l]
+///  // given: parentPos = posCursor[lvl-1]
+///  pstart = desc.positions[lvl][parentPos]
+///  pstop = desc.positions[lvl][parentPos+1]
+///  plast = pstop - 1
+///  msz = desc.coordinates[lvl].size()
+///  if (pstart < pstop) {
+///    isPresent = (desc.coordinates[lvl][plast] == lvlCoords[lvl])
 ///  } else { // first insertion
-///    present = false
-///    pointers[l][pos[l-1]] = msz
+///    isPresent = false
+///    desc.positions[lvl][parentPos] = msz
 ///  }
-///  if (present) { // index already present
-///    next = phi-1
+///  if (isPresent) { // coordinate is already present
+///    pnext = plast
 ///  } else {
-///    indices[l].push_back(i[l])
-///    pointers[l][pos[l-1]+1] = msz+1
-///    next = msz
-///    <prepare level l + 1>
+///    desc.coordinates[lvl].push_back(lvlCoords[lvl])
+///    desc.positions[lvl][parentPos+1] = msz+1
+///    pnext = msz
+///    <prepare level lvl+1>
 ///  }
-///  pos[l] = next
+///  posCursor[lvl] = pnext
 static Value genCompressed(OpBuilder &builder, Location loc,
-                           MutSparseTensorDescriptor desc, ValueRange indices,
-                           Value value, Value pos, Level lvl) {
+                           MutSparseTensorDescriptor desc, ValueRange lvlCoords,
+                           Value /*unused*/, Value parentPos, Level lvl) {
   const SparseTensorType stt(desc.getRankedTensorType());
   const Level lvlRank = stt.getLvlRank();
   assert(lvl < lvlRank && "Level is out of bounds");
-  assert(indices.size() == static_cast<size_t>(lvlRank) &&
+  assert(lvlCoords.size() == static_cast<size_t>(lvlRank) &&
          "Level-rank mismatch");
   SmallVector<Type> types;
   Type indexType = builder.getIndexType();
   Type boolType = builder.getIntegerType(1);
-  unsigned idxIndex;
-  unsigned idxStride;
-  std::tie(idxIndex, idxStride) = desc.getIdxMemRefIndexAndStride(lvl);
-  Value one = constantIndex(builder, loc, 1);
-  Value pp1 = builder.create<arith::AddIOp>(loc, pos, one);
-  Value plo = genLoad(builder, loc, desc.getPtrMemRef(lvl), pos);
-  Value phi = genLoad(builder, loc, desc.getPtrMemRef(lvl), pp1);
-  Value msz = desc.getIdxMemSize(builder, loc, lvl);
-  Value idxStrideC;
-  if (idxStride > 1) {
-    idxStrideC = constantIndex(builder, loc, idxStride);
-    msz = builder.create<arith::DivUIOp>(loc, msz, idxStrideC);
-  }
-  Value phim1 = builder.create<arith::SubIOp>(
-      loc, genCast(builder, loc, phi, indexType), one);
+  unsigned crdFidx;
+  unsigned crdStride;
+  std::tie(crdFidx, crdStride) = desc.getCrdMemRefIndexAndStride(lvl);
+  const Value one = constantIndex(builder, loc, 1);
+  const Value pp1 = builder.create<arith::AddIOp>(loc, parentPos, one);
+  const Value positionsAtLvl = desc.getPosMemRef(lvl);
+  const Value pstart = genLoad(builder, loc, positionsAtLvl, parentPos);
+  const Value pstop = genLoad(builder, loc, positionsAtLvl, pp1);
+  const Value crdMsz = desc.getCrdMemSize(builder, loc, lvl);
+  const Value crdStrideC =
+      crdStride > 1 ? constantIndex(builder, loc, crdStride) : Value();
+  const Value msz =
+      crdStrideC ? builder.create<arith::DivUIOp>(loc, crdMsz, crdStrideC)
+                 : crdMsz;
+  const Value plast = builder.create<arith::SubIOp>(
+      loc, genCast(builder, loc, pstop, indexType), one);
   // Conditional expression.
-  Value lt =
-      builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ult, plo, phi);
+  Value lt = builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ult,
+                                           pstart, pstop);
   types.push_back(boolType);
   scf::IfOp ifOp1 = builder.create<scf::IfOp>(loc, types, lt, /*else*/ true);
   types.pop_back();
   builder.setInsertionPointToStart(&ifOp1.getThenRegion().front());
-  Value crd = genLoad(
-      builder, loc, desc.getMemRefField(idxIndex),
-      idxStride > 1 ? builder.create<arith::MulIOp>(loc, phim1, idxStrideC)
-                    : phim1);
+  Value crd =
+      genLoad(builder, loc, desc.getMemRefField(crdFidx),
+              crdStrideC ? builder.create<arith::MulIOp>(loc, plast, crdStrideC)
+                         : plast);
   Value eq = builder.create<arith::CmpIOp>(
       loc, arith::CmpIPredicate::eq, genCast(builder, loc, crd, indexType),
-      indices[lvl]);
+      lvlCoords[lvl]);
   builder.create<scf::YieldOp>(loc, eq);
   builder.setInsertionPointToStart(&ifOp1.getElseRegion().front());
   if (lvl > 0)
-    genStore(builder, loc, msz, desc.getPtrMemRef(lvl), pos);
+    genStore(builder, loc, msz, positionsAtLvl, parentPos);
   builder.create<scf::YieldOp>(loc, constantI1(builder, loc, false));
   builder.setInsertionPointAfter(ifOp1);
   // If present construct. Note that for a non-unique dimension level, we
@@ -363,22 +365,22 @@ static Value genCompressed(OpBuilder &builder, Location loc,
   const Value p = stt.isUniqueLvl(lvl) ? ifOp1.getResult(0)
                                        : constantI1(builder, loc, false);
   scf::IfOp ifOp2 = builder.create<scf::IfOp>(loc, types, p, /*else*/ true);
-  // If present (fields unaffected, update next to phim1).
+  // If present (fields unaffected, update pnext to plast).
   builder.setInsertionPointToStart(&ifOp2.getThenRegion().front());
 
   // FIXME: This does not looks like a clean way, but probably the most
   // efficient way.
-  desc.getFields().push_back(phim1);
+  desc.getFields().push_back(plast);
   builder.create<scf::YieldOp>(loc, desc.getFields());
   desc.getFields().pop_back();
 
-  // If !present (changes fields, update next).
+  // If !present (changes fields, update pnext).
   builder.setInsertionPointToStart(&ifOp2.getElseRegion().front());
   Value mszp1 = builder.create<arith::AddIOp>(loc, msz, one);
-  genStore(builder, loc, mszp1, desc.getPtrMemRef(lvl), pp1);
-  createPushback(builder, loc, desc, SparseTensorFieldKind::IdxMemRef, lvl,
-                 indices[lvl]);
-  // Prepare the next dimension "as needed".
+  genStore(builder, loc, mszp1, positionsAtLvl, pp1);
+  createPushback(builder, loc, desc, SparseTensorFieldKind::CrdMemRef, lvl,
+                 lvlCoords[lvl]);
+  // Prepare the next level "as needed".
   if ((lvl + 1) < lvlRank)
     allocSchemeForRank(builder, loc, desc, lvl + 1);
 
@@ -415,40 +417,41 @@ static void genInsertBody(OpBuilder &builder, ModuleOp module,
   const SparseTensorType stt(rtp);
   const Level lvlRank = stt.getLvlRank();
 
-  // Construct fields and indices arrays from parameters.
+  // Extract fields and coordinates from args.
   SmallVector<Value> fields = llvm::to_vector(args.drop_back(lvlRank + 1));
   MutSparseTensorDescriptor desc(rtp, fields);
-  const SmallVector<Value> indices =
+  const SmallVector<Value> coordinates =
       llvm::to_vector(args.take_back(lvlRank + 1).drop_back());
   Value value = args.back();
-  Value pos = constantZero(builder, loc, builder.getIndexType());
+  Value parentPos = constantZero(builder, loc, builder.getIndexType());
   // Generate code for every level.
   for (Level l = 0; l < lvlRank; l++) {
     const auto dlt = stt.getLvlType(l);
     if (isCompressedDLT(dlt)) {
       // Create:
       //   if (!present) {
-      //     indices[l].push_back(i[l])
-      //     <update pointers and prepare level l + 1>
+      //     coordinates[l].push_back(coords[l])
+      //     <update positions and prepare level l + 1>
       //   }
-      //   pos[l] = indices.size() - 1
-      //   <insert @ pos[l] at next level l + 1>
-      pos = genCompressed(builder, loc, desc, indices, value, pos, l);
+      //   positions[l] = coordinates.size() - 1
+      //   <insert @ positions[l] at next level l + 1>
+      parentPos =
+          genCompressed(builder, loc, desc, coordinates, value, parentPos, l);
     } else if (isSingletonDLT(dlt)) {
       // Create:
-      //   indices[l].push_back(i[l])
-      //   pos[l] = pos[l-1]
-      //   <insert @ pos[l] at next level l + 1>
-      createPushback(builder, loc, desc, SparseTensorFieldKind::IdxMemRef, l,
-                     indices[l]);
+      //   coordinates[l].push_back(coords[l])
+      //   positions[l] = positions[l-1]
+      //   <insert @ positions[l] at next level l + 1>
+      createPushback(builder, loc, desc, SparseTensorFieldKind::CrdMemRef, l,
+                     coordinates[l]);
     } else {
       assert(isDenseDLT(dlt));
       // Construct the new position as:
-      //   pos[l] = size * pos[l-1] + i[l]
-      //   <insert @ pos[l] at next level l + 1>
+      //   positions[l] = size * positions[l-1] + coords[l]
+      //   <insert @ positions[l] at next level l + 1>
       Value size = sizeFromTensorAtLvl(builder, loc, desc, l);
-      Value mult = builder.create<arith::MulIOp>(loc, size, pos);
-      pos = builder.create<arith::AddIOp>(loc, mult, indices[l]);
+      Value mult = builder.create<arith::MulIOp>(loc, size, parentPos);
+      parentPos = builder.create<arith::AddIOp>(loc, mult, coordinates[l]);
     }
   }
   // Reached the actual value append/insert.
@@ -456,7 +459,7 @@ static void genInsertBody(OpBuilder &builder, ModuleOp module,
     createPushback(builder, loc, desc, SparseTensorFieldKind::ValMemRef,
                    std::nullopt, value);
   else
-    genStore(builder, loc, value, desc.getValMemRef(), pos);
+    genStore(builder, loc, value, desc.getValMemRef(), parentPos);
   builder.create<func::ReturnOp>(loc, fields);
 }
 
@@ -464,19 +467,18 @@ static void genInsertBody(OpBuilder &builder, ModuleOp module,
 /// function doesn't exist yet, call `createFunc` to generate the function.
 static void genInsertionCallHelper(OpBuilder &builder,
                                    MutSparseTensorDescriptor desc,
-                                   SmallVectorImpl<Value> &indices, Value value,
+                                   SmallVectorImpl<Value> &lcvs, Value value,
                                    func::FuncOp insertPoint,
                                    StringRef namePrefix,
                                    FuncGeneratorType createFunc) {
   // The mangled name of the function has this format:
-  //   <namePrefix>_<DLT>_<shape>_<ordering>_<eltType>
-  //     _<indexBitWidth>_<pointerBitWidth>
+  //   <namePrefix>_<DLT>_<shape>_<ordering>_<eltType>_<crdWidth>_<posWidth>
   const SparseTensorType stt(desc.getRankedTensorType());
   SmallString<32> nameBuffer;
   llvm::raw_svector_ostream nameOstream(nameBuffer);
   nameOstream << namePrefix;
-  assert(static_cast<size_t>(stt.getLvlRank()) == indices.size());
   const Level lvlRank = stt.getLvlRank();
+  assert(lcvs.size() == static_cast<size_t>(lvlRank));
   for (Level l = 0; l < lvlRank; l++)
     nameOstream << toMLIRString(stt.getLvlType(l)) << "_";
   // Static dim sizes are used in the generated code while dynamic sizes are
@@ -488,7 +490,7 @@ static void genInsertionCallHelper(OpBuilder &builder,
   if (!stt.isIdentity())
     nameOstream << stt.getDimToLvlMap() << "_";
   nameOstream << stt.getElementType() << "_";
-  nameOstream << stt.getIndexBitWidth() << "_" << stt.getPointerBitWidth();
+  nameOstream << stt.getCrdWidth() << "_" << stt.getPosWidth();
 
   // Look up the function.
   ModuleOp module = insertPoint->getParentOfType<ModuleOp>();
@@ -496,9 +498,9 @@ static void genInsertionCallHelper(OpBuilder &builder,
   auto result = SymbolRefAttr::get(context, nameOstream.str());
   auto func = module.lookupSymbol<func::FuncOp>(result.getAttr());
 
-  // Construct parameters for fields and indices.
+  // Construct operands: fields, coords, and value.
   SmallVector<Value> operands = llvm::to_vector(desc.getFields());
-  operands.append(indices);
+  operands.append(lcvs);
   operands.push_back(value);
   Location loc = insertPoint.getLoc();
 
@@ -531,31 +533,31 @@ static void genEndInsert(OpBuilder &builder, Location loc,
   for (Level l = 0; l < lvlRank; l++) {
     const auto dlt = stt.getLvlType(l);
     if (isCompressedDLT(dlt)) {
-      // Compressed dimensions need a pointer cleanup for all entries
+      // Compressed dimensions need a position cleanup for all entries
       // that were not visited during the insertion pass.
       //
       // TODO: avoid cleanup and keep compressed scheme consistent at all
       // times?
       //
       if (l > 0) {
-        Type ptrType = stt.getPointerType();
-        Value ptrMemRef = desc.getPtrMemRef(l);
-        Value hi = desc.getPtrMemSize(builder, loc, l);
+        Type posType = stt.getPosType();
+        Value posMemRef = desc.getPosMemRef(l);
+        Value hi = desc.getPosMemSize(builder, loc, l);
         Value zero = constantIndex(builder, loc, 0);
         Value one = constantIndex(builder, loc, 1);
         // Vector of only one, but needed by createFor's prototype.
-        SmallVector<Value, 1> inits{genLoad(builder, loc, ptrMemRef, zero)};
+        SmallVector<Value, 1> inits{genLoad(builder, loc, posMemRef, zero)};
         scf::ForOp loop = createFor(builder, loc, hi, inits, one);
         Value i = loop.getInductionVar();
         Value oldv = loop.getRegionIterArg(0);
-        Value newv = genLoad(builder, loc, ptrMemRef, i);
-        Value ptrZero = constantZero(builder, loc, ptrType);
+        Value newv = genLoad(builder, loc, posMemRef, i);
+        Value posZero = constantZero(builder, loc, posType);
         Value cond = builder.create<arith::CmpIOp>(
-            loc, arith::CmpIPredicate::eq, newv, ptrZero);
-        scf::IfOp ifOp = builder.create<scf::IfOp>(loc, TypeRange(ptrType),
+            loc, arith::CmpIPredicate::eq, newv, posZero);
+        scf::IfOp ifOp = builder.create<scf::IfOp>(loc, TypeRange(posType),
                                                    cond, /*else*/ true);
         builder.setInsertionPointToStart(&ifOp.getThenRegion().front());
-        genStore(builder, loc, oldv, ptrMemRef, i);
+        genStore(builder, loc, oldv, posMemRef, i);
         builder.create<scf::YieldOp>(loc, oldv);
         builder.setInsertionPointToStart(&ifOp.getElseRegion().front());
         builder.create<scf::YieldOp>(loc, newv);
@@ -578,7 +580,8 @@ static Value reallocOrSubView(OpBuilder &builder, Location loc, int64_t len,
 
   Value targetLen = constantIndex(builder, loc, len);
   Value bufferLen = linalg::createOrFoldDimOp(builder, loc, buffer, 0);
-  Value reallocP = builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ult,
+  // Reallocates if target length is greater than the actual buffer len.
+  Value reallocP = builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::ugt,
                                                  targetLen, bufferLen);
   scf::IfOp ifOp = builder.create<scf::IfOp>(loc, retTp, reallocP, true);
   // If targetLen > bufferLen, reallocate to get enough sparse to return.
@@ -684,18 +687,35 @@ class SparseDimOpConverter : public OpConversionPattern<tensor::DimOp> {
   LogicalResult
   matchAndRewrite(tensor::DimOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    std::optional<int64_t> index = op.getConstantIndex();
-    if (!index || !getSparseTensorEncoding(adaptor.getSource().getType()))
+    std::optional<int64_t> dim = op.getConstantIndex();
+    if (!dim || !getSparseTensorEncoding(adaptor.getSource().getType()))
       return failure();
 
     auto desc = getDescriptorFromTensorTuple(adaptor.getSource());
-    auto sz = sizeFromTensorAtDim(rewriter, op.getLoc(), desc, *index);
+    auto sz = sizeFromTensorAtDim(rewriter, op.getLoc(), desc, *dim);
 
     rewriter.replaceOp(op, sz);
     return success();
   }
 };
 
+template <typename Op, StorageSpecifierKind kind>
+class SparseSliceGetterOpConverter : public OpConversionPattern<Op> {
+public:
+  using OpConversionPattern<Op>::OpConversionPattern;
+  LogicalResult
+  matchAndRewrite(Op op, typename Op::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // Simply lowers to specifer.get <field> operation.
+    auto desc = getDescriptorFromTensorTuple(adaptor.getSlice());
+    auto v = desc.getSpecifierField(rewriter, op.getLoc(), kind,
+                                    op.getDim().getZExtValue());
+
+    rewriter.replaceOp(op, v);
+    return success();
+  }
+};
+
 /// Sparse codegen rule for trivial tensor casts.
 class SparseCastConverter : public OpConversionPattern<tensor::CastOp> {
 public:
@@ -843,7 +863,7 @@ class SparseExpandConverter : public OpConversionPattern<ExpandOp> {
     rewriter.create<linalg::FillOp>(
         loc, ValueRange{constantZero(rewriter, loc, boolType)},
         ValueRange{filled});
-    // Replace expansion op with these buffers and initial index.
+    // Replace expansion op with these buffers and initial coordinate.
     assert(op.getNumResults() == 4);
     rewriter.replaceOp(op, {values, filled, added, zero});
     return success();
@@ -866,9 +886,9 @@ class SparseCompressConverter : public OpConversionPattern<CompressOp> {
     Value count = adaptor.getCount();
     const SparseTensorType dstType(desc.getRankedTensorType());
     Type eltType = dstType.getElementType();
-    // Prepare indices.
-    SmallVector<Value> indices(adaptor.getIndices());
-    // If the innermost level is ordered, we need to sort the indices
+    // Prepare level-coords.
+    SmallVector<Value> lcvs(adaptor.getLvlCoords());
+    // If the innermost level is ordered, we need to sort the coordinates
     // in the "added" array prior to applying the compression.
     if (dstType.isOrderedLvl(dstType.getLvlRank() - 1))
       rewriter.create<SortOp>(loc, count, ValueRange{added}, ValueRange{},
@@ -880,26 +900,25 @@ class SparseCompressConverter : public OpConversionPattern<CompressOp> {
     //
     // Generate
     //    out_memrefs = for (i = 0; i < count; i++)(in_memrefs) {
-    //      index = added[i];
-    //      value = values[index];
-    //      insert({prev_indices, index}, value);
-    //      new_memrefs = insert(in_memrefs, {prev_indices, index}, value);
-    //      values[index] = 0;
-    //      filled[index] = false;
+    //      crd = added[i];
+    //      value = values[crd];
+    //      insert({lvlCoords, crd}, value);
+    //      new_memrefs = insert(in_memrefs, {lvlCoords, crd}, value);
+    //      values[crd] = 0;
+    //      filled[crd] = false;
     //      yield new_memrefs
     //    }
     scf::ForOp loop = createFor(rewriter, loc, count, desc.getFields());
     Value i = loop.getInductionVar();
-    Value index = genLoad(rewriter, loc, added, i);
-    Value value = genLoad(rewriter, loc, values, index);
-    indices.push_back(index);
+    Value crd = genLoad(rewriter, loc, added, i);
+    Value value = genLoad(rewriter, loc, values, crd);
+    lcvs.push_back(crd);
     // TODO: faster for subsequent insertions?
     auto insertPoint = op->template getParentOfType<func::FuncOp>();
-    genInsertionCallHelper(rewriter, desc, indices, value, insertPoint,
+    genInsertionCallHelper(rewriter, desc, lcvs, value, insertPoint,
                            kInsertFuncNamePrefix, genInsertBody);
-    genStore(rewriter, loc, constantZero(rewriter, loc, eltType), values,
-             index);
-    genStore(rewriter, loc, constantI1(rewriter, loc, false), filled, index);
+    genStore(rewriter, loc, constantZero(rewriter, loc, eltType), values, crd);
+    genStore(rewriter, loc, constantI1(rewriter, loc, false), filled, crd);
     rewriter.create<scf::YieldOp>(loc, desc.getFields());
     rewriter.setInsertionPointAfter(loop);
     Value result = genTuple(rewriter, loc, dstType, loop->getResults());
@@ -924,12 +943,11 @@ class SparseInsertConverter : public OpConversionPattern<InsertOp> {
                   ConversionPatternRewriter &rewriter) const override {
     SmallVector<Value> fields;
     auto desc = getMutDescriptorFromTensorTuple(adaptor.getTensor(), fields);
-    // Prepare and indices.
-    SmallVector<Value> indices(adaptor.getIndices());
+    SmallVector<Value> lcvs(adaptor.getLvlCoords());
     // Generate insertion.
     Value value = adaptor.getValue();
     auto insertPoint = op->template getParentOfType<func::FuncOp>();
-    genInsertionCallHelper(rewriter, desc, indices, value, insertPoint,
+    genInsertionCallHelper(rewriter, desc, lcvs, value, insertPoint,
                            kInsertFuncNamePrefix, genInsertBody);
 
     // Replace operation with resulting memrefs.
@@ -938,39 +956,38 @@ class SparseInsertConverter : public OpConversionPattern<InsertOp> {
   }
 };
 
-/// Sparse codegen rule for pointer accesses.
-class SparseToPointersConverter : public OpConversionPattern<ToPointersOp> {
+/// Sparse codegen rule for position accesses.
+class SparseToPositionsConverter : public OpConversionPattern<ToPositionsOp> {
 public:
-  using OpAdaptor = typename ToPointersOp::Adaptor;
-  using OpConversionPattern<ToPointersOp>::OpConversionPattern;
+  using OpAdaptor = typename ToPositionsOp::Adaptor;
+  using OpConversionPattern<ToPositionsOp>::OpConversionPattern;
   LogicalResult
-  matchAndRewrite(ToPointersOp op, OpAdaptor adaptor,
+  matchAndRewrite(ToPositionsOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Replace the requested pointer access with corresponding field.
+    // Replace the requested position access with corresponding field.
     // The cast_op is inserted by type converter to intermix 1:N type
     // conversion.
     auto desc = getDescriptorFromTensorTuple(adaptor.getTensor());
-    uint64_t dim = op.getDimension().getZExtValue();
-    rewriter.replaceOp(op, desc.getPtrMemRef(dim));
+    rewriter.replaceOp(op, desc.getPosMemRef(op.getLevel()));
     return success();
   }
 };
 
-/// Sparse codegen rule for index accesses.
-class SparseToIndicesConverter : public OpConversionPattern<ToIndicesOp> {
+/// Sparse codegen rule for accessing the coordinates arrays.
+class SparseToCoordinatesConverter
+    : public OpConversionPattern<ToCoordinatesOp> {
 public:
-  using OpAdaptor = typename ToIndicesOp::Adaptor;
-  using OpConversionPattern<ToIndicesOp>::OpConversionPattern;
+  using OpAdaptor = typename ToCoordinatesOp::Adaptor;
+  using OpConversionPattern<ToCoordinatesOp>::OpConversionPattern;
   LogicalResult
-  matchAndRewrite(ToIndicesOp op, OpAdaptor adaptor,
+  matchAndRewrite(ToCoordinatesOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Replace the requested pointer access with corresponding field.
+    // Replace the requested coordinates access with corresponding field.
     // The cast_op is inserted by type converter to intermix 1:N type
     // conversion.
     Location loc = op.getLoc();
     auto desc = getDescriptorFromTensorTuple(adaptor.getTensor());
-    uint64_t dim = op.getDimension().getZExtValue();
-    Value field = desc.getIdxMemRefOrView(rewriter, loc, dim);
+    Value field = desc.getCrdMemRefOrView(rewriter, loc, op.getLevel());
 
     // Insert a cast to bridge the actual type to the user expected type. If the
     // actual type and the user expected type aren't compatible, the compiler or
@@ -984,16 +1001,16 @@ class SparseToIndicesConverter : public OpConversionPattern<ToIndicesOp> {
   }
 };
 
-/// Sparse codegen rule for accessing the linear indices buffer.
-class SparseToIndicesBufferConverter
-    : public OpConversionPattern<ToIndicesBufferOp> {
+/// Sparse codegen rule for accessing the linear coordinates buffer.
+class SparseToCoordinatesBufferConverter
+    : public OpConversionPattern<ToCoordinatesBufferOp> {
 public:
-  using OpAdaptor = typename ToIndicesBufferOp::Adaptor;
-  using OpConversionPattern<ToIndicesBufferOp>::OpConversionPattern;
+  using OpAdaptor = typename ToCoordinatesBufferOp::Adaptor;
+  using OpConversionPattern<ToCoordinatesBufferOp>::OpConversionPattern;
   LogicalResult
-  matchAndRewrite(ToIndicesBufferOp op, OpAdaptor adaptor,
+  matchAndRewrite(ToCoordinatesBufferOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Replace the requested pointer access with corresponding field.
+    // Replace the requested coordinates access with corresponding field.
     // The cast_op is inserted by type converter to intermix 1:N type
     // conversion.
     auto desc = getDescriptorFromTensorTuple(adaptor.getTensor());
@@ -1011,7 +1028,7 @@ class SparseToValuesConverter : public OpConversionPattern<ToValuesOp> {
   LogicalResult
   matchAndRewrite(ToValuesOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Replace the requested pointer access with corresponding field.
+    // Replace the requested values access with corresponding field.
     // The cast_op is inserted by type converter to intermix 1:N type
     // conversion.
     auto desc = getDescriptorFromTensorTuple(adaptor.getTensor());
@@ -1101,13 +1118,15 @@ class SparseConvertConverter : public OpConversionPattern<ConvertOp> {
   }
 };
 
-class SparseExtractSliceCoverter
+class SparseExtractSliceConverter
     : public OpConversionPattern<tensor::ExtractSliceOp> {
 public:
   using OpConversionPattern::OpConversionPattern;
   LogicalResult
   matchAndRewrite(tensor::ExtractSliceOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
+    Location loc = op.getLoc();
+    MLIRContext *ctx = op.getContext();
     auto srcEnc = getSparseTensorEncoding(op.getSourceType());
     auto dstEnc = getSparseTensorEncoding(op.getResult().getType());
     if (!srcEnc && !dstEnc)
@@ -1118,19 +1137,45 @@ class SparseExtractSliceCoverter
     assert(srcEnc.getDimLevelType() == dstEnc.getDimLevelType());
     assert(srcEnc.getDimOrdering() == dstEnc.getDimOrdering());
     assert(srcEnc.getHigherOrdering() == dstEnc.getHigherOrdering());
-    assert(srcEnc.getPointerBitWidth() == dstEnc.getPointerBitWidth());
-    assert(srcEnc.getIndexBitWidth() == dstEnc.getIndexBitWidth());
-
-    // TODO: support dynamic slices.
-    for (int i = 0, e = op.getSourceType().getRank(); i < e; i++) {
-      assert(op.getStaticStrides()[i] == dstEnc.getStaticDimSliceStride(i));
-      assert(op.getStaticOffsets()[i] == dstEnc.getStaticDimSliceOffset(i));
-      assert(op.getStaticSizes()[i] == dstEnc.getStaticDimSliceSize(i));
+    assert(srcEnc.getPosWidth() == dstEnc.getPosWidth());
+    assert(srcEnc.getCrdWidth() == dstEnc.getCrdWidth());
+
+    SmallVector<Value> fields;
+    auto desc = getMutDescriptorFromTensorTuple(adaptor.getSource(), fields);
+
+    auto newSpec = rewriter.create<StorageSpecifierInitOp>(
+        loc, StorageSpecifierType::get(ctx, dstEnc), desc.getSpecifier());
+    desc.setSpecifier(newSpec);
+
+    // Fills in slice information.
+    for (auto [idx, offset, size, stride] : llvm::enumerate(
+             op.getMixedOffsets(), op.getMixedSizes(), op.getMixedStrides())) {
+      Dimension dim = idx;
+
+      Value offsetV = getValueOrCreateConstantIndexOp(rewriter, loc, offset);
+      Value sizeV = getValueOrCreateConstantIndexOp(rewriter, loc, size);
+      Value strideV = getValueOrCreateConstantIndexOp(rewriter, loc, stride);
+      // TODO: We could probably only set dynamic value here. But it would
+      // requires us to fill the hole when casting a static slice to dynamic
+      // slice.
+      desc.setSpecifierField(rewriter, loc, StorageSpecifierKind::DimOffset,
+                             dim, offsetV);
+
+      // FIXME: we need to distinguish level sizes and dimension size for slices
+      // here. Maybe we should store slice level sizes in a different array
+      // instead of reusing it.
+      assert(srcEnc.hasIdDimOrdering());
+      desc.setSpecifierField(rewriter, loc, StorageSpecifierKind::LvlSize, dim,
+                             sizeV);
+      desc.setSpecifierField(rewriter, loc, StorageSpecifierKind::DimStride,
+                             dim, strideV);
     }
 
-    // TODO: create a new specifer for slices (need to encode slice metadata).
-    // It does not matter now because only constant offset/stride are allowed.
-    rewriter.replaceOp(op, adaptor.getSource());
+    // NOTE: we can not generate tuples directly from descriptor here, as the
+    // descriptor is holding the original type, yet we want the slice type
+    // here (they shared every memref but with an updated specifier).
+    rewriter.replaceOp(op, genTuple(rewriter, loc, op.getResult().getType(),
+                                    desc.getFields()));
     return success();
   }
 };
@@ -1174,11 +1219,12 @@ struct SparsePackOpConverter : public OpConversionPattern<PackOp> {
           case SparseTensorFieldKind::StorageSpec:
             field = SparseTensorSpecifier::getInitValue(rewriter, loc, rtp);
             break;
-          case SparseTensorFieldKind::PtrMemRef: {
-            // TACO-style COO starts with a PtrBuffer
+          case SparseTensorFieldKind::PosMemRef: {
+            // TACO-style COO starts with a PosBuffer
             // By creating a constant value for it, we avoid the complexity of
             // memory management.
-            auto tensorType = RankedTensorType::get({2}, enc.getPointerType());
+            const auto posTp = enc.getPosType();
+            auto tensorType = RankedTensorType::get({2}, posTp);
             auto memrefType = MemRefType::get(tensorType.getShape(),
                                               tensorType.getElementType());
             auto cstPtr = rewriter.create<arith::ConstantOp>(
@@ -1186,35 +1232,34 @@ struct SparsePackOpConverter : public OpConversionPattern<PackOp> {
                 DenseElementsAttr::get(
                     tensorType,
                     ArrayRef<Attribute>{
-                        IntegerAttr::get(enc.getPointerType(), 0),
+                        IntegerAttr::get(posTp, 0),
                         IntegerAttr::get(
-                            enc.getPointerType(),
-                            op.getData().getType().getShape()[0])}));
+                            posTp, op.getValues().getType().getShape()[0])}));
             field = rewriter.create<bufferization::ToMemrefOp>(loc, memrefType,
                                                                cstPtr);
             break;
           }
-          case SparseTensorFieldKind::IdxMemRef: {
-            auto tensorType = op.getIndices().getType();
+          case SparseTensorFieldKind::CrdMemRef: {
+            auto tensorType = op.getCoordinates().getType();
             auto memrefType = MemRefType::get(tensorType.getShape(),
                                               tensorType.getElementType());
-            auto idxMemRef = rewriter.create<bufferization::ToMemrefOp>(
-                op->getLoc(), memrefType, op.getIndices());
+            auto crdMemRef = rewriter.create<bufferization::ToMemrefOp>(
+                op->getLoc(), memrefType, op.getCoordinates());
             ReassociationIndices reassociation;
             for (int i = 0, e = tensorType.getRank(); i < e; i++)
               reassociation.push_back(i);
 
             // Flattened the indices buffer to rank 1.
             field = rewriter.create<memref::CollapseShapeOp>(
-                loc, idxMemRef, ArrayRef<ReassociationIndices>(reassociation));
+                loc, crdMemRef, ArrayRef<ReassociationIndices>(reassociation));
             break;
           }
           case SparseTensorFieldKind::ValMemRef: {
-            auto tensorType = op.getData().getType();
+            auto tensorType = op.getValues().getType();
             auto memrefType = MemRefType::get(tensorType.getShape(),
                                               tensorType.getElementType());
             field = rewriter.create<bufferization::ToMemrefOp>(
-                op->getLoc(), memrefType, op.getData());
+                op->getLoc(), memrefType, op.getValues());
             break;
           }
           }
@@ -1228,15 +1273,18 @@ struct SparsePackOpConverter : public OpConversionPattern<PackOp> {
         });
 
     MutSparseTensorDescriptor desc(rtp, fields);
-    auto noe = linalg::createOrFoldDimOp(rewriter, loc, op.getData(), 0);
-    for (unsigned i = 0, e = rtp.getRank(); i < e; i++) {
-      int dim = rtp.getShape()[i];
-      assert(!ShapedType::isDynamic(dim));
-      desc.setDimSize(rewriter, loc, i, constantIndex(rewriter, loc, dim));
-      if (i == 0)
-        desc.setPtrMemSize(rewriter, loc, i, constantIndex(rewriter, loc, 2));
-
-      desc.setIdxMemSize(rewriter, loc, i, noe);
+    auto noe = linalg::createOrFoldDimOp(rewriter, loc, op.getValues(), 0);
+    // FIXME: should use `SparseTensorType::getLvlRank` in lieu of
+    // `RankedTensorType::getRank`, because the latter introduces dim/lvl
+    // ambiguity.
+    for (Level lvl = 0, lvlRank = rtp.getRank(); lvl < lvlRank; lvl++) {
+      const auto sh = rtp.getShape()[lvl];
+      assert(!ShapedType::isDynamic(sh));
+      desc.setLvlSize(rewriter, loc, lvl, constantIndex(rewriter, loc, sh));
+      if (lvl == 0)
+        desc.setPosMemSize(rewriter, loc, lvl, constantIndex(rewriter, loc, 2));
+
+      desc.setCrdMemSize(rewriter, loc, lvl, noe);
     }
     desc.setValMemSize(rewriter, loc, noe);
 
@@ -1252,39 +1300,43 @@ struct SparseUnpackOpConverter : public OpConversionPattern<UnpackOp> {
                   ConversionPatternRewriter &rewriter) const override {
     auto desc = getDescriptorFromTensorTuple(adaptor.getTensor());
     Location loc = op.getLoc();
-    int64_t rank = op.getTensor().getType().getRank();
+    const auto srcTp = getSparseTensorType(op.getTensor());
+    const Level lvlRank = srcTp.getLvlRank();
 
-    assert(isUniqueCOOType(op.getTensor().getType()) &&
-           desc.getFields().size() == 4);
+    assert(isUniqueCOOType(srcTp) && desc.getFields().size() == 4);
 
-    Value flatBuf = rank == 1 ? desc.getIdxMemRefOrView(rewriter, loc, 0)
-                              : desc.getAOSMemRef();
-    Value dataBuf = desc.getValMemRef();
+    Value flatBuf = lvlRank == 1 ? desc.getCrdMemRefOrView(rewriter, loc, 0)
+                                 : desc.getAOSMemRef();
+    Value valuesBuf = desc.getValMemRef();
 
-    // If frontend requests a static buffer, we reallocate the data/indices
-    // to ensure that we meet their need.
-    TensorType dataTp = op.getData().getType();
-    if (dataTp.hasStaticShape()) {
-      dataBuf = reallocOrSubView(rewriter, loc, dataTp.getShape()[0], dataBuf);
+    // If frontend requests a static buffer, we reallocate the
+    // values/coordinates to ensure that we meet their need.
+    const auto valuesTp = getRankedTensorType(op.getValues());
+    if (valuesTp.hasStaticShape()) {
+      valuesBuf =
+          reallocOrSubView(rewriter, loc, valuesTp.getShape()[0], valuesBuf);
     }
 
-    TensorType indicesTp = op.getIndices().getType();
-    if (indicesTp.hasStaticShape()) {
-      auto len = indicesTp.getShape()[0] * indicesTp.getShape()[1];
+    const auto coordinatesTp = getRankedTensorType(op.getCoordinates());
+    if (coordinatesTp.hasStaticShape()) {
+      auto len = coordinatesTp.getShape()[0] * coordinatesTp.getShape()[1];
       flatBuf = reallocOrSubView(rewriter, loc, len, flatBuf);
     }
 
-    Value idxBuf = rewriter.create<memref::ExpandShapeOp>(
-        loc, MemRefType::get(indicesTp.getShape(), indicesTp.getElementType()),
+    Value coordinatesBuf = rewriter.create<memref::ExpandShapeOp>(
+        loc,
+        MemRefType::get(coordinatesTp.getShape(),
+                        coordinatesTp.getElementType()),
         flatBuf, ArrayRef{ReassociationIndices{0, 1}});
 
     // Converts MemRefs back to Tensors.
-    Value data = rewriter.create<bufferization::ToTensorOp>(loc, dataBuf);
-    Value indices = rewriter.create<bufferization::ToTensorOp>(loc, idxBuf);
-    Value nnz = genCast(rewriter, loc, desc.getValMemSize(rewriter, loc),
-                        op.getNnz().getType());
+    Value values = rewriter.create<bufferization::ToTensorOp>(loc, valuesBuf);
+    Value coordinates =
+        rewriter.create<bufferization::ToTensorOp>(loc, coordinatesBuf);
+    Value nse = genCast(rewriter, loc, desc.getValMemSize(rewriter, loc),
+                        op.getNse().getType());
 
-    rewriter.replaceOp(op, {data, indices, nnz});
+    rewriter.replaceOp(op, {values, coordinates, nse});
     return success();
   }
 };
@@ -1296,52 +1348,54 @@ struct SparseNewOpConverter : public OpConversionPattern<NewOp> {
                   ConversionPatternRewriter &rewriter) const override {
     Location loc = op.getLoc();
     const auto dstTp = getSparseTensorType(op.getResult());
-    const auto encDst = dstTp.getEncoding();
     // Creating COO with NewOp is handled by direct IR codegen. All other cases
     // are handled by rewriting.
-    if (!dstTp.hasEncoding() || getCOOStart(encDst) != 0)
+    if (!dstTp.hasEncoding() || getCOOStart(dstTp.getEncoding()) != 0)
       return failure();
 
     // Implement the NewOp(filename) as follows:
-    //   reader = getSparseTensorReader(filename)
-    //   nse = getSparseTensorNNZ()
-    //   tmp = bufferization.alloc_tensor an ordered COO with
-    //          dst dim ordering, size_hint = nse
-    //   indices = to_indices_buffer(tmp)
-    //   values = to_values(tmp)
-    //   isSorted = getSparseTensorReaderRead(indices, values, dimOrdering)
-    //   if (!isSorted) sort_coo(nse, indices, values)
+    //   %reader = @getSparseTensorReader(%filename)
+    //   %nse = @getSparseTensorNSE(%reader)
+    //   %coo = bufferization.alloc_tensor an ordered COO with
+    //          dst dim ordering, size_hint = %nse
+    //   %coordinates = sparse_tensor.coordinates_buffer(%coo)
+    //   %values = sparse_tensor.values(%coo)
+    //   %isSorted = @sparseTensorReaderReadToBuffers(%coordinates, %values)
+    //   if (! %isSorted) sparse_tensor.sort_coo(%nse, %coordinates, %values)
     //   update storage specifier
-    //   dst = sparse_tensor.ConvertOp tmp
+    //   @delSparseTensorReader(%reader)
 
     // Create a sparse tensor reader.
-    Value fileName = op.getSource();
-    Type opaqueTp = getOpaquePointerType(rewriter);
+    const Value fileName = op.getSource();
+    const Type opaqueTp = getOpaquePointerType(rewriter);
+    // FIXME: use `createCheckedSparseTensorReader` instead, because
+    // `createSparseTensorReader` is unsafe.
     Value reader = createFuncCall(rewriter, loc, "createSparseTensorReader",
                                   {opaqueTp}, {fileName}, EmitCInterface::Off)
                        .getResult(0);
 
-    Type indexTp = rewriter.getIndexType();
+    const Type indexTp = rewriter.getIndexType();
     const Dimension dimRank = dstTp.getDimRank();
+    const Level lvlRank = dstTp.getLvlRank();
 
     // If the result tensor has dynamic dimensions, get the dynamic sizes from
     // the sparse tensor reader.
     SmallVector<Value> dynSizes;
     if (dstTp.hasDynamicDimShape()) {
+      // FIXME: call `getSparseTensorReaderDimSizes` instead, because
+      // `copySparseTensorReaderDimSizes` copies the memref over,
+      // instead of just accessing the reader's memory directly.
       Value dimSizes = genAlloca(rewriter, loc, dimRank, indexTp);
       createFuncCall(rewriter, loc, "copySparseTensorReaderDimSizes", {},
                      {reader, dimSizes}, EmitCInterface::On)
           .getResult(0);
-      ArrayRef<int64_t> dstShape = dstTp.getRankedTensorType().getShape();
-      for (auto &d : llvm::enumerate(dstShape)) {
-        if (d.value() == ShapedType::kDynamic) {
+      for (const auto &d : llvm::enumerate(dstTp.getDimShape()))
+        if (ShapedType::isDynamic(d.value()))
           dynSizes.push_back(rewriter.create<memref::LoadOp>(
               loc, dimSizes, constantIndex(rewriter, loc, d.index())));
-        }
-      }
     }
 
-    Value nse = createFuncCall(rewriter, loc, "getSparseTensorReaderNNZ",
+    Value nse = createFuncCall(rewriter, loc, "getSparseTensorReaderNSE",
                                {indexTp}, {reader}, EmitCInterface::Off)
                     .getResult(0);
     // Construct allocation for each field.
@@ -1350,63 +1404,71 @@ struct SparseNewOpConverter : public OpConversionPattern<NewOp> {
                       fields, nse);
     MutSparseTensorDescriptor desc(dstTp, fields);
 
-    // Read the COO tensor data.
-    Type eltTp = dstTp.getElementType();
-    Type indBufEleTp = getIndexOverheadType(rewriter, encDst);
-    SmallString<32> getReadFuncName{"getSparseTensorReaderRead",
-                                    overheadTypeFunctionSuffix(indBufEleTp),
-                                    primaryTypeFunctionSuffix(eltTp)};
-
-    Value xs = desc.getAOSMemRef();
-    Value ys = desc.getValMemRef();
-    SmallVector<Value> dim2lvlValues(dimRank, Value());
-    if (auto dimOrder = encDst.getDimOrdering()) {
+    // Construct the `dim2lvl` buffer for handing off to the runtime library.
+    // FIXME: This code is (mostly) copied from the SparseTensorConversion.cpp
+    // handling of `NewOp`, and only handles permutations.  Fixing this
+    // requires waiting for wrengr to finish redoing the CL that handles
+    // all dim<->lvl stuff more robustly.
+    SmallVector<Value> dim2lvlValues(dimRank);
+    if (!dstTp.isIdentity()) {
+      const auto dimOrder = dstTp.getDimToLvlMap();
       assert(dimOrder.isPermutation() && "Got non-permutation");
-      for (uint64_t l = 0; l < dimRank; l++) {
-        uint64_t d = dimOrder.getDimPosition(l);
+      for (Level l = 0; l < lvlRank; l++) {
+        const Dimension d = dimOrder.getDimPosition(l);
         dim2lvlValues[d] = constantIndex(rewriter, loc, l);
       }
     } else {
-      for (uint64_t l = 0; l < dimRank; l++)
-        dim2lvlValues[l] = constantIndex(rewriter, loc, l);
+      // The `SparseTensorType` ctor already ensures `dimRank == lvlRank`
+      // when `isIdentity`; so no need to re-assert it here.
+      for (Dimension d = 0; d < dimRank; d++)
+        dim2lvlValues[d] = constantIndex(rewriter, loc, d);
     }
     Value dim2lvl = allocaBuffer(rewriter, loc, dim2lvlValues);
 
-    Value f = constantI1(rewriter, loc, false);
+    // Read the COO tensor data.
+    Value xs = desc.getAOSMemRef();
+    Value ys = desc.getValMemRef();
+
+    const Type boolTp = rewriter.getIntegerType(1);
+    const Type elemTp = dstTp.getElementType();
+    const Type crdTp = dstTp.getCrdType();
+    // FIXME: This function name is weird; should rename to
+    // "sparseTensorReaderReadToBuffers".
+    SmallString<32> readToBuffersFuncName{"getSparseTensorReaderRead",
+                                          overheadTypeFunctionSuffix(crdTp),
+                                          primaryTypeFunctionSuffix(elemTp)};
     Value isSorted =
-        createFuncCall(rewriter, loc, getReadFuncName, {f.getType()},
+        createFuncCall(rewriter, loc, readToBuffersFuncName, {boolTp},
                        {reader, dim2lvl, xs, ys}, EmitCInterface::On)
             .getResult(0);
 
     // If the destination tensor is a sorted COO, we need to sort the COO tensor
     // data if the input elements aren't sorted yet.
-    if (encDst.isOrderedLvl(dimRank - 1)) {
+    if (dstTp.isOrderedLvl(lvlRank - 1)) {
+      Value kFalse = constantI1(rewriter, loc, false);
       Value notSorted = rewriter.create<arith::CmpIOp>(
-          loc, arith::CmpIPredicate::eq, isSorted, f);
+          loc, arith::CmpIPredicate::eq, isSorted, kFalse);
       scf::IfOp ifOp =
           rewriter.create<scf::IfOp>(loc, notSorted, /*else*/ false);
       rewriter.setInsertionPointToStart(&ifOp.getThenRegion().front());
       rewriter.create<SortCooOp>(
-          loc, nse, xs, ValueRange{ys}, rewriter.getIndexAttr(dimRank),
+          loc, nse, xs, ValueRange{ys}, rewriter.getIndexAttr(lvlRank),
           rewriter.getIndexAttr(0), SparseTensorSortKind::HybridQuickSort);
       rewriter.setInsertionPointAfter(ifOp);
     }
 
-    // Set PtrMemRef0[1] = nse.
-    Value c1 = constantIndex(rewriter, loc, 1);
-    Value ptrMemref0 = desc.getPtrMemRef(0);
-    Type ptrEleTy = getMemRefType(ptrMemref0).getElementType();
-    Value ptrNse =
-        ptrEleTy == nse.getType()
-            ? nse
-            : rewriter.create<arith::IndexCastOp>(loc, ptrEleTy, nse);
-    rewriter.create<memref::StoreOp>(loc, ptrNse, ptrMemref0, c1);
+    // Set PosMemRef0[1] = nse.
+    const Value c1 = constantIndex(rewriter, loc, 1);
+    const Value posMemref0 = desc.getPosMemRef(0);
+    const Type posTp = dstTp.getPosType();
+    const Value posNse = genCast(rewriter, loc, nse, posTp);
+    rewriter.create<memref::StoreOp>(loc, posNse, posMemref0, c1);
 
     // Update storage specifier.
-    Value idxSize = rewriter.create<arith::MulIOp>(
-        loc, nse, constantIndex(rewriter, loc, dimRank));
-    desc.setSpecifierField(rewriter, loc, StorageSpecifierKind::IdxMemSize, 0,
-                           idxSize);
+    Value coordinatesSize = rewriter.create<arith::MulIOp>(
+        loc, nse, constantIndex(rewriter, loc, lvlRank));
+    desc.setSpecifierField(rewriter, loc, StorageSpecifierKind::CrdMemSize, 0,
+                           coordinatesSize);
     desc.setSpecifierField(rewriter, loc, StorageSpecifierKind::ValMemSize,
                            std::nullopt, nse);
 
@@ -1434,13 +1496,18 @@ void mlir::populateSparseTensorCodegenPatterns(
   patterns.add<SparsePackOpConverter, SparseUnpackOpConverter,
                SparseReturnConverter, SparseCallConverter, SparseDimOpConverter,
                SparseCastConverter, SparseTensorDeallocConverter,
-               SparseExtractSliceCoverter, SparseTensorLoadConverter,
+               SparseExtractSliceConverter, SparseTensorLoadConverter,
                SparseExpandConverter, SparseCompressConverter,
-               SparseInsertConverter, SparseToPointersConverter,
-               SparseToIndicesConverter, SparseToIndicesBufferConverter,
-               SparseToValuesConverter, SparseConvertConverter,
-               SparseNewOpConverter, SparseNumberOfEntriesConverter>(
-      typeConverter, patterns.getContext());
+               SparseInsertConverter,
+               SparseSliceGetterOpConverter<ToSliceOffsetOp,
+                                            StorageSpecifierKind::DimOffset>,
+               SparseSliceGetterOpConverter<ToSliceStrideOp,
+                                            StorageSpecifierKind::DimStride>,
+               SparseToPositionsConverter, SparseToCoordinatesConverter,
+               SparseToCoordinatesBufferConverter, SparseToValuesConverter,
+               SparseConvertConverter, SparseNewOpConverter,
+               SparseNumberOfEntriesConverter>(typeConverter,
+                                               patterns.getContext());
   patterns.add<SparseTensorAllocConverter>(typeConverter, patterns.getContext(),
                                            enableBufferInitialization);
 }
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp
index 7622554479af5..c3c0a4f88c362 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp
@@ -239,8 +239,8 @@ class NewCallParams final {
   // is there a better way to handle that than this one-off setter method?
   NewCallParams &setTemplateTypes(SparseTensorType stt) {
     const auto enc = stt.getEncoding();
-    params[kParamPtrTp] = constantPointerTypeEncoding(builder, loc, enc);
-    params[kParamIndTp] = constantIndexTypeEncoding(builder, loc, enc);
+    params[kParamPosTp] = constantPosTypeEncoding(builder, loc, enc);
+    params[kParamCrdTp] = constantCrdTypeEncoding(builder, loc, enc);
     params[kParamValTp] =
         constantPrimaryTypeEncoding(builder, loc, stt.getElementType());
     return *this;
@@ -284,8 +284,8 @@ class NewCallParams final {
   static constexpr unsigned kParamLvlTypes = 2;
   static constexpr unsigned kParamLvl2Dim = 3;
   static constexpr unsigned kParamDim2Lvl = 4;
-  static constexpr unsigned kParamPtrTp = 5;
-  static constexpr unsigned kParamIndTp = 6;
+  static constexpr unsigned kParamPosTp = 5;
+  static constexpr unsigned kParamCrdTp = 6;
   static constexpr unsigned kParamValTp = 7;
   static constexpr unsigned kParamAction = 8;
   static constexpr unsigned kParamPtr = 9;
@@ -382,53 +382,38 @@ static void genDelIteratorCall(OpBuilder &builder, Location loc, Type elemTp,
 ///   if val != 0
 ///     t->add(&val, [i1,..,ik], [p1,..,pk]);
 static void genAddEltCall(OpBuilder &builder, Location loc, Type eltType,
-                          Value lvlCOO, Value valPtr, Value dimInd,
+                          Value lvlCOO, Value valPtr, Value dimCoords,
                           Value dim2lvl) {
   SmallString<9> name{"addElt", primaryTypeFunctionSuffix(eltType)};
-  SmallVector<Value, 4> params{lvlCOO, valPtr, dimInd, dim2lvl};
+  SmallVector<Value, 4> params{lvlCOO, valPtr, dimCoords, dim2lvl};
   Type pTp = getOpaquePointerType(builder);
   createFuncCall(builder, loc, name, pTp, params, EmitCInterface::On);
 }
 
 /// Generates a call to `iter->getNext()`.  If there is a next element,
-/// then it is copied into the out-parameters `ind` and `elemPtr`,
+/// then it is copied into the out-parameters `coords` and `elemPtr`,
 /// and the return value is true.  If there isn't a next element, then
 /// the return value is false.
+///
+/// The `coords` argument uses the same coordinate-space as the `iter`
+/// (which can be either dim- or lvl-coords, depending on context).
 static Value genGetNextCall(OpBuilder &builder, Location loc, Value iter,
-                            Value ind, Value elemPtr) {
+                            Value coords, Value elemPtr) {
   Type elemTp = elemPtr.getType().cast<ShapedType>().getElementType();
   SmallString<10> name{"getNext", primaryTypeFunctionSuffix(elemTp)};
-  SmallVector<Value, 3> params{iter, ind, elemPtr};
+  SmallVector<Value, 3> params{iter, coords, elemPtr};
   Type i1 = builder.getI1Type();
   return createFuncCall(builder, loc, name, i1, params, EmitCInterface::On)
       .getResult(0);
 }
 
-/// Converts a pointer to COO (from calls to iter->next()) into a vector of
-/// indices, apply (optional) `offset` on `offsetDim`.
-static SmallVector<Value> loadIndices(OpBuilder &builder, Location loc,
-                                      unsigned rank, Value ind,
-                                      unsigned offsetDim = 0,
-                                      Value offset = Value()) {
-  SmallVector<Value> ivs;
-  ivs.reserve(rank);
-  for (unsigned i = 0; i < rank; i++) {
-    Value idx = constantIndex(builder, loc, i);
-    idx = builder.create<memref::LoadOp>(loc, ind, idx);
-    if (offsetDim == i && offset)
-      idx = builder.create<arith::AddIOp>(loc, idx, offset);
-    ivs.push_back(idx);
-  }
-  return ivs;
-}
-
-/// Inserts a value stored in `elemPtr` into a dense tensor created by
-/// allocDenseTensor().
+/// Loads the value stored in `elemPtr`, and stores it at the coordinates
+/// `cvs` into a dense tensor created by `allocDenseTensor`.
 static void insertScalarIntoDenseTensor(OpBuilder &builder, Location loc,
                                         Value elemPtr, Value tensor,
-                                        ValueRange ivs) {
+                                        ValueRange cvs) {
   Value elemV = builder.create<memref::LoadOp>(loc, elemPtr);
-  builder.create<memref::StoreOp>(loc, elemV, tensor, ivs);
+  builder.create<memref::StoreOp>(loc, elemV, tensor, cvs);
 }
 
 /// Determine if the runtime library supports direct conversion to the
@@ -453,31 +438,17 @@ static bool canUseDirectConversion(ArrayRef<DimLevelType> dimTypes) {
   return true;
 }
 
-/// Helper method to translate indices during a reshaping operation.
+/// Helper method to translate coordinates during a reshaping operation.
 /// TODO: provide as general utility to MLIR at large?
-static void translateIndices(Location loc, ConversionPatternRewriter &rewriter,
-                             ArrayRef<ReassociationIndices> reassociation,
-                             TensorType dstTp, TensorType srcTp, Value dstIdx,
-                             Value srcIdx, ArrayRef<Value> dstShape,
-                             ArrayRef<Value> srcShape) {
-  const Dimension dstRank = dstTp.getRank();
-  const Dimension srcRank = srcTp.getRank();
-
-  SmallVector<Value> srcIndices;
-  srcIndices.reserve(srcRank);
-  for (Dimension d = 0; d < srcRank; d++) {
-    Value idx = rewriter.create<memref::LoadOp>(
-        loc, srcIdx, constantIndex(rewriter, loc, d));
-    srcIndices.push_back(idx);
-  }
-
-  SmallVector<Value> dstIndices;
-  translateIndicesArray(rewriter, loc, reassociation, srcIndices, srcShape,
-                        dstShape, dstIndices);
-
-  for (Dimension d = 0; d < dstRank; d++)
-    rewriter.create<memref::StoreOp>(loc, dstIndices[d], dstIdx,
-                                     constantIndex(rewriter, loc, d));
+static void reshapeCoords(Location loc, OpBuilder &builder,
+                          ArrayRef<ReassociationIndices> reassociation,
+                          ValueRange srcSizes, Value srcCoords,
+                          ValueRange dstSizes, Value dstCoords) {
+  const auto srcCvs = loadAll(builder, loc, srcSizes.size(), srcCoords);
+  SmallVector<Value> dstCvs;
+  reshapeCvs(builder, loc, reassociation, srcSizes, srcCvs, dstSizes, dstCvs);
+  assert(dstCvs.size() == dstSizes.size());
+  storeAll(builder, loc, dstCoords, dstCvs);
 }
 
 /// Generate code for a general sparse to sparse reshaping operation.
@@ -491,7 +462,7 @@ static void translateIndices(Location loc, ConversionPatternRewriter &rewriter,
 ///   iter = src->toCOO();
 ///   coo = newSparseCOO()
 ///   while (elem = iter->getNext()) {
-///     coo->add(reshape(elem.indices), elem.value)
+///     coo->add(reshape(elem.coords), elem.value)
 ///   }
 ///   s = newSparseTensor(coo)
 template <typename ReshapeOp>
@@ -506,7 +477,7 @@ genSparse2SparseReshape(ReshapeOp op, typename ReshapeOp::Adaptor adaptor,
   Type elemTp = srcTp.getElementType();
   assert(elemTp == dstTp.getElementType() &&
          "reshape should not change element type");
-  // Start an iterator over the source tensor (in original index order).
+  // Start an iterator over the source tensor (in coordinate order).
   SmallVector<Value> srcDimSizes =
       getDimSizes(rewriter, loc, srcTp, adaptor.getSrc());
   NewCallParams params(rewriter, loc);
@@ -520,28 +491,31 @@ genSparse2SparseReshape(ReshapeOp op, typename ReshapeOp::Adaptor adaptor,
   else
     genReshapeDstShape(loc, rewriter, dstDimSizes, srcDimSizes,
                        dstTp.getDimShape(), op.getReassociationIndices());
-  Value coo =
+  const Value coo =
       params.genBuffers(dstTp, dstDimSizes).genNewCall(Action::kEmptyCOO);
-  Value dstPerm = params.getDim2LvlMap();
+  const Value dstPerm = params.getDim2LvlMap();
   // Construct a while loop over the iterator.
-  Type iTp = rewriter.getIndexType();
-  Value srcIdx = genAlloca(rewriter, loc, srcTp.getDimRank(), iTp);
-  Value dstIdx = genAlloca(rewriter, loc, dstTp.getDimRank(), iTp);
-  Value elemPtr = genAllocaScalar(rewriter, loc, elemTp);
-  SmallVector<Value> noArgs;
-  SmallVector<Type> noTypes;
+  const Type iTp = rewriter.getIndexType();
+  const Value srcDimCoords = genAlloca(rewriter, loc, srcTp.getDimRank(), iTp);
+  const Value dstDimCoords = genAlloca(rewriter, loc, dstTp.getDimRank(), iTp);
+  const Value elemPtr = genAllocaScalar(rewriter, loc, elemTp);
+  const SmallVector<Value> noArgs;
+  const SmallVector<Type> noTypes;
   auto whileOp = rewriter.create<scf::WhileOp>(loc, noTypes, noArgs);
   Block *before = rewriter.createBlock(&whileOp.getBefore(), {}, noTypes);
   rewriter.setInsertionPointToEnd(before);
-  Value cond = genGetNextCall(rewriter, loc, iter, srcIdx, elemPtr);
+  Value cond = genGetNextCall(rewriter, loc, iter, srcDimCoords, elemPtr);
   rewriter.create<scf::ConditionOp>(loc, cond, before->getArguments());
-  // Translate indices from source to target and insert. Note that we do
+  // Translate coordinates from source to target and insert. Note that we do
   // not need to store the value in elemPtr, as the value is still there.
   Block *after = rewriter.createBlock(&whileOp.getAfter(), {}, noTypes);
   rewriter.setInsertionPointToStart(after);
-  translateIndices(loc, rewriter, op.getReassociationIndices(), dstTp, srcTp,
-                   dstIdx, srcIdx, dstDimSizes, srcDimSizes);
-  genAddEltCall(rewriter, loc, elemTp, coo, elemPtr, dstIdx, dstPerm);
+  // We probably don't need these assertions, but better safe than sorry.
+  assert(srcTp.getDimRank() == srcDimSizes.size());
+  assert(dstTp.getDimRank() == dstDimSizes.size());
+  reshapeCoords(loc, rewriter, op.getReassociationIndices(), srcDimSizes,
+                srcDimCoords, dstDimSizes, dstDimCoords);
+  genAddEltCall(rewriter, loc, elemTp, coo, elemPtr, dstDimCoords, dstPerm);
   rewriter.create<scf::YieldOp>(loc);
   // Final call to construct sparse tensor storage and free temporary resources.
   rewriter.setInsertionPointAfter(whileOp);
@@ -569,7 +543,7 @@ static void genSparseCOOIterationLoop(
   const Dimension dimRank = stt.getDimRank();
   const Type elemTp = stt.getElementType();
 
-  // Start an iterator over the tensor (in original index order).
+  // Start an iterator over the tensor (in coordinate order).
   const auto noPerm = stt.withoutOrdering();
   SmallVector<Value> dimSizes = getDimSizes(rewriter, loc, noPerm, t);
   Value iter = NewCallParams(rewriter, loc)
@@ -577,14 +551,15 @@ static void genSparseCOOIterationLoop(
                    .genNewCall(Action::kToIterator, t);
 
   // Construct a while loop over the iterator.
-  Value srcIdx = genAlloca(rewriter, loc, dimRank, rewriter.getIndexType());
+  const Type iTp = rewriter.getIndexType();
+  Value srcDimCoords = genAlloca(rewriter, loc, dimRank, iTp);
   Value elemPtr = genAllocaScalar(rewriter, loc, elemTp);
-  SmallVector<Value> noArgs;
-  SmallVector<Type> noTypes;
+  const SmallVector<Value> noArgs;
+  const SmallVector<Type> noTypes;
   auto whileOp = rewriter.create<scf::WhileOp>(loc, noTypes, noArgs);
   Block *before = rewriter.createBlock(&whileOp.getBefore(), {}, noTypes);
   rewriter.setInsertionPointToEnd(before);
-  Value cond = genGetNextCall(rewriter, loc, iter, srcIdx, elemPtr);
+  Value cond = genGetNextCall(rewriter, loc, iter, srcDimCoords, elemPtr);
   rewriter.create<scf::ConditionOp>(loc, cond, before->getArguments());
   Block *after = rewriter.createBlock(&whileOp.getAfter(), {}, noTypes);
   rewriter.setInsertionPointToStart(after);
@@ -598,7 +573,7 @@ static void genSparseCOOIterationLoop(
     rewriter.setInsertionPointToStart(&ifOp.getThenRegion().front());
   }
   // Callback here to build loop body.
-  bodyBuilder(rewriter, loc, srcIdx, elemPtr);
+  bodyBuilder(rewriter, loc, srcDimCoords, elemPtr);
 
   // Exit the scope from the IfOp.
   if (hasDenseDim)
@@ -811,8 +786,8 @@ class SparseTensorNewConverter : public OpConversionPattern<NewOp> {
         genLvlTypesBuffer(rewriter, loc, stt),
         lvl2dimBuffer,
         dim2lvlBuffer,
-        constantPointerTypeEncoding(rewriter, loc, stt.getEncoding()),
-        constantIndexTypeEncoding(rewriter, loc, stt.getEncoding()),
+        constantPosTypeEncoding(rewriter, loc, stt.getEncoding()),
+        constantCrdTypeEncoding(rewriter, loc, stt.getEncoding()),
         valTp};
     Value tensor = createFuncCall(rewriter, loc, "newSparseTensorFromReader",
                                   opaqueTp, params, EmitCInterface::On)
@@ -922,8 +897,8 @@ class SparseTensorConvertConverter : public OpConversionPattern<ConvertOp> {
         // the correct sparsity information to either of them.
         const auto mixedEnc = SparseTensorEncodingAttr::get(
             op->getContext(), dstEnc.getDimLevelType(), dstEnc.getDimOrdering(),
-            dstEnc.getHigherOrdering(), srcEnc.getPointerBitWidth(),
-            srcEnc.getIndexBitWidth());
+            dstEnc.getHigherOrdering(), srcEnc.getPosWidth(),
+            srcEnc.getCrdWidth());
         // TODO: This is the only place where `kToCOO` (or `kToIterator`)
         // is called with a non-identity permutation.  Is there any clean
         // way to push the permutation over to the `kFromCOO` side instead?
@@ -942,38 +917,39 @@ class SparseTensorConvertConverter : public OpConversionPattern<ConvertOp> {
       //   dst = new Tensor(0);
       //   iter = new SparseTensorIterator(src);
       //   while (elem = iter->getNext()) {
-      //     dst[elem.indices] = elem.value;
+      //     dst[elem.coords] = elem.value;
       //   }
       //   delete iter;
       //
       // Fabricate a no-permutation encoding for NewCallParams
-      // The pointer/index types must be those of `src`.
+      // The position/coordinate types must be those of `src`.
       // The dimLevelTypes aren't actually used by Action::kToIterator.
       const auto dstEnc = SparseTensorEncodingAttr::get(
           op->getContext(),
           SmallVector<DimLevelType>(dimRank, DimLevelType::Dense), AffineMap(),
-          AffineMap(), srcEnc.getPointerBitWidth(), srcEnc.getIndexBitWidth());
+          AffineMap(), srcEnc.getPosWidth(), srcEnc.getCrdWidth());
       SmallVector<Value> dimSizes = getDimSizes(rewriter, loc, srcTp, src);
       Value iter = NewCallParams(rewriter, loc)
                        .genBuffers(dstTp.withEncoding(dstEnc), dimSizes)
                        .genNewCall(Action::kToIterator, src);
-      Value ind = genAlloca(rewriter, loc, dimRank, rewriter.getIndexType());
+      const Type iTp = rewriter.getIndexType();
+      Value dimCoords = genAlloca(rewriter, loc, dimRank, iTp);
       Value elemPtr = genAllocaScalar(rewriter, loc, elemTp);
       Block *insertionBlock = rewriter.getInsertionBlock();
       // TODO: Dense buffers should be allocated/deallocated via the callback
       // in BufferizationOptions.
       Value dst = allocDenseTensor(rewriter, loc, dstTp, dimSizes);
-      SmallVector<Value> noArgs;
-      SmallVector<Type> noTypes;
+      const SmallVector<Value> noArgs;
+      const SmallVector<Type> noTypes;
       auto whileOp = rewriter.create<scf::WhileOp>(loc, noTypes, noArgs);
       Block *before = rewriter.createBlock(&whileOp.getBefore(), {}, noTypes);
       rewriter.setInsertionPointToEnd(before);
-      Value cond = genGetNextCall(rewriter, loc, iter, ind, elemPtr);
+      Value cond = genGetNextCall(rewriter, loc, iter, dimCoords, elemPtr);
       rewriter.create<scf::ConditionOp>(loc, cond, before->getArguments());
       Block *after = rewriter.createBlock(&whileOp.getAfter(), {}, noTypes);
       rewriter.setInsertionPointToStart(after);
-      SmallVector<Value> ivs = loadIndices(rewriter, loc, dimRank, ind);
-      insertScalarIntoDenseTensor(rewriter, loc, elemPtr, dst, ivs);
+      const auto dcvs = loadAll(rewriter, loc, dimRank, dimCoords);
+      insertScalarIntoDenseTensor(rewriter, loc, elemPtr, dst, dcvs);
       rewriter.create<scf::YieldOp>(loc);
       rewriter.setInsertionPointAfter(whileOp);
       genDelIteratorCall(rewriter, loc, elemTp, iter);
@@ -1004,7 +980,7 @@ class SparseTensorConvertConverter : public OpConversionPattern<ConvertOp> {
     // To fill the COO tensor from a sparse constant in COO format:
     //   for i in range(NNZ)
     //     val = values[i]
-    //     [i1,..,ik] = indices[i]
+    //     [i1,..,ik] = coordinates[i]
     //     t->add(val, [i1,..,ik], [p1,..,pk])
     //
     // Note that the dense tensor traversal code is actually implemented
@@ -1018,18 +994,17 @@ class SparseTensorConvertConverter : public OpConversionPattern<ConvertOp> {
     NewCallParams params(rewriter, loc);
     Value coo =
         params.genBuffers(dstTp, dimSizes).genNewCall(Action::kEmptyCOO);
-    Value ind = genAlloca(rewriter, loc, dimRank, rewriter.getIndexType());
+    const Type iTp = rewriter.getIndexType();
+    Value dimCoords = genAlloca(rewriter, loc, dimRank, iTp);
     Value perm = params.getDim2LvlMap();
     Value elemPtr = genAllocaScalar(rewriter, loc, elemTp);
     genDenseTensorOrSparseConstantIterLoop(
         rewriter, loc, src, dimRank,
-        [&](OpBuilder &builder, Location loc, Value val, ValueRange ivs) {
-          for (Dimension d = 0; d < dimRank; d++) {
-            Value dim = constantIndex(builder, loc, d);
-            builder.create<memref::StoreOp>(loc, ivs[d], ind, dim);
-          }
+        [&](OpBuilder &builder, Location loc, Value val, ValueRange dcvs) {
+          assert(dcvs.size() == static_cast<size_t>(dimRank));
+          storeAll(builder, loc, dimCoords, dcvs);
           builder.create<memref::StoreOp>(loc, val, elemPtr);
-          genAddEltCall(builder, loc, elemTp, coo, elemPtr, ind, perm);
+          genAddEltCall(builder, loc, elemTp, coo, elemPtr, dimCoords, perm);
         });
     // Final call to construct sparse tensor storage.
     Value dst = params.genNewCall(Action::kFromCOO, coo);
@@ -1061,41 +1036,43 @@ class SparseTensorDeallocConverter
   }
 };
 
-/// Sparse conversion rule for pointer accesses.
-class SparseTensorToPointersConverter
-    : public OpConversionPattern<ToPointersOp> {
+/// Sparse conversion rule for position accesses.
+class SparseTensorToPositionsConverter
+    : public OpConversionPattern<ToPositionsOp> {
 public:
   using OpConversionPattern::OpConversionPattern;
   LogicalResult
-  matchAndRewrite(ToPointersOp op, OpAdaptor adaptor,
+  matchAndRewrite(ToPositionsOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    Type resType = op.getType();
-    Type ptrType = resType.cast<ShapedType>().getElementType();
-    SmallString<16> name{"sparsePointers", overheadTypeFunctionSuffix(ptrType)};
-    Value dim =
-        constantIndex(rewriter, op->getLoc(), op.getDimension().getZExtValue());
-    replaceOpWithFuncCall(rewriter, op, name, resType,
-                          {adaptor.getTensor(), dim}, EmitCInterface::On);
+    Type resTp = op.getType();
+    Type posTp = resTp.cast<ShapedType>().getElementType();
+    SmallString<17> name{"sparsePositions", overheadTypeFunctionSuffix(posTp)};
+    Value lvl = constantIndex(rewriter, op->getLoc(), op.getLevel());
+    replaceOpWithFuncCall(rewriter, op, name, resTp, {adaptor.getTensor(), lvl},
+                          EmitCInterface::On);
     return success();
   }
 };
 
-/// Sparse conversion rule for index accesses.
-class SparseTensorToIndicesConverter : public OpConversionPattern<ToIndicesOp> {
+/// Sparse conversion rule for coordinate accesses.
+class SparseTensorToCoordinatesConverter
+    : public OpConversionPattern<ToCoordinatesOp> {
 public:
   using OpConversionPattern::OpConversionPattern;
   LogicalResult
-  matchAndRewrite(ToIndicesOp op, OpAdaptor adaptor,
+  matchAndRewrite(ToCoordinatesOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
+    // TODO: use `SparseTensorType::getCrdType` instead.
     Type resType = op.getType();
-    Type indType = resType.cast<ShapedType>().getElementType();
-    SmallString<15> name{"sparseIndices", overheadTypeFunctionSuffix(indType)};
+    const Type crdTp = resType.cast<ShapedType>().getElementType();
+    SmallString<19> name{"sparseCoordinates",
+                         overheadTypeFunctionSuffix(crdTp)};
     Location loc = op->getLoc();
-    Value dim = constantIndex(rewriter, loc, op.getDimension().getZExtValue());
+    Value lvl = constantIndex(rewriter, loc, op.getLevel());
 
     // The function returns a MemRef without a layout.
-    MemRefType callRetType = get1DMemRefType(indType, false);
-    SmallVector<Value> operands{adaptor.getTensor(), dim};
+    MemRefType callRetType = get1DMemRefType(crdTp, false);
+    SmallVector<Value> operands{adaptor.getTensor(), lvl};
     auto fn = getFunc(op->getParentOfType<ModuleOp>(), name, callRetType,
                       operands, EmitCInterface::On);
     Value callRet =
@@ -1171,21 +1148,19 @@ class SparseTensorInsertConverter : public OpConversionPattern<InsertOp> {
   matchAndRewrite(InsertOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     // Note that the current regime only allows for strict lexicographic
-    // index order. All values are passed by reference through stack
+    // coordinate order. All values are passed by reference through stack
     // allocated memrefs.
     Location loc = op->getLoc();
     const auto stt = getSparseTensorType(op.getTensor());
     const auto elemTp = stt.getElementType();
-    const Dimension dimRank = stt.getDimRank();
-    auto mref = genAlloca(rewriter, loc, dimRank, rewriter.getIndexType());
+    const Level lvlRank = stt.getLvlRank();
+    auto lvlCoords = genAlloca(rewriter, loc, lvlRank, rewriter.getIndexType());
     auto vref = genAllocaScalar(rewriter, loc, elemTp);
-    for (Dimension d = 0; d < dimRank; d++)
-      rewriter.create<memref::StoreOp>(loc, adaptor.getIndices()[d], mref,
-                                       constantIndex(rewriter, loc, d));
+    storeAll(rewriter, loc, lvlCoords, adaptor.getLvlCoords());
     rewriter.create<memref::StoreOp>(loc, adaptor.getValue(), vref);
     SmallString<12> name{"lexInsert", primaryTypeFunctionSuffix(elemTp)};
-    createFuncCall(rewriter, loc, name, {}, {adaptor.getTensor(), mref, vref},
-                   EmitCInterface::On);
+    createFuncCall(rewriter, loc, name, {},
+                   {adaptor.getTensor(), lvlCoords, vref}, EmitCInterface::On);
     rewriter.replaceOp(op, adaptor.getTensor());
     return success();
   }
@@ -1208,12 +1183,12 @@ class SparseTensorExpandConverter : public OpConversionPattern<ExpandOp> {
     // Get the cardinality of valid coordinates for the innermost level.
     Value sz = createOrFoldLvlCall(rewriter, loc, srcTp, adaptor.getTensor(),
                                    srcTp.getLvlRank() - 1);
-    // Allocate temporary buffers for values, filled-switch, and indices.
+    // Allocate temporary buffers for values, filled-switch, and coordinates.
     // We do not use stack buffers for this, since the expanded size may
     // be rather large (as it envelops a single expanded dense dimension).
     Value values = genAlloc(rewriter, loc, sz, eltType);
     Value filled = genAlloc(rewriter, loc, sz, boolType);
-    Value indices = genAlloc(rewriter, loc, sz, idxType);
+    Value lastLvlCoordinates = genAlloc(rewriter, loc, sz, idxType);
     Value zero = constantZero(rewriter, loc, idxType);
     // Reset the values/filled-switch to all-zero/false. Note that this
     // introduces an O(N) operation into the computation, but this reset
@@ -1226,9 +1201,9 @@ class SparseTensorExpandConverter : public OpConversionPattern<ExpandOp> {
     rewriter.create<linalg::FillOp>(
         loc, ValueRange{constantZero(rewriter, loc, boolType)},
         ValueRange{filled});
-    // Replace expansion op with these buffers and initial index.
+    // Replace expansion op with these buffers and initial coordinate.
     assert(op.getNumResults() == 4);
-    rewriter.replaceOp(op, {values, filled, indices, zero});
+    rewriter.replaceOp(op, {values, filled, lastLvlCoordinates, zero});
     return success();
   }
 };
@@ -1252,14 +1227,12 @@ class SparseTensorCompressConverter : public OpConversionPattern<CompressOp> {
     Value tensor = adaptor.getTensor();
     const auto stt = getSparseTensorType(op.getTensor());
     const Type elemTp = stt.getElementType();
-    const Dimension dimRank = stt.getDimRank();
-    auto mref = genAlloca(rewriter, loc, dimRank, rewriter.getIndexType());
-    for (Dimension d = 0; d < dimRank - 1; d++)
-      rewriter.create<memref::StoreOp>(loc, adaptor.getIndices()[d], mref,
-                                       constantIndex(rewriter, loc, d));
+    const Level lvlRank = stt.getLvlRank();
+    auto lvlCoords = genAlloca(rewriter, loc, lvlRank, rewriter.getIndexType());
+    storeAll(rewriter, loc, lvlCoords, adaptor.getLvlCoords());
     SmallString<12> name{"expInsert", primaryTypeFunctionSuffix(elemTp)};
     createFuncCall(rewriter, loc, name, {},
-                   {tensor, mref, values, filled, added, count},
+                   {tensor, lvlCoords, values, filled, added, count},
                    EmitCInterface::On);
     rewriter.replaceOp(op, adaptor.getTensor());
     // Deallocate the buffers on exit of the loop nest.
@@ -1288,7 +1261,7 @@ class SparseTensorConcatConverter : public OpConversionPattern<ConcatenateOp> {
     //      coo->add(adjustForOffset(i,j,k), b[i,j,k])
     //
     //    for elem in sparse_input
-    //      coo->add(adjustForOffset(elem.indices), elem.value)
+    //      coo->add(adjustForOffset(elem.coords), elem.value)
     //    ...
     //    a = newSparseTensor(coo_for_a)
     //    return a
@@ -1301,22 +1274,22 @@ class SparseTensorConcatConverter : public OpConversionPattern<ConcatenateOp> {
     //      a[ adjustForOffset(i,j,k) ] = b[i,j,k]
     //
     //    for elem in sparse_input
-    //      a[ adjustForOffset(elem.indices) ] = elem.value
+    //      a[ adjustForOffset(elem.coords) ] = elem.value
     //    return a
     Location loc = op.getLoc();
     const auto dstTp = getSparseTensorType(op);
     const auto dstEnc = dstTp.getEncoding();
     const Type elemTp = dstTp.getElementType();
-    const Dimension concatDim = op.getDimension().getZExtValue();
+    const Dimension concatDim = op.getDimension();
     const Dimension dimRank = dstTp.getDimRank();
 
     Value dst;     // destination tensor
     Value dstPerm; // destination tensor permutation (if sparse out)
     // A pointer to the value being inserted (if dense => sparse)
     Value elemPtr;
-    // Memory that holds the dim-indices for destination tensor (if sparse out)
-    Value dstInd;
-    // The offset applied to the dimenstion to be concated (starting from 0)
+    // Memory that holds the dim-coords for destination tensor (if sparse out)
+    Value dstDimCoords;
+    // The offset applied to the dimension to be concated (starting from 0)
     Value offset = constantIndex(rewriter, loc, 0);
 
     SmallVector<Value> dimSizes;
@@ -1330,7 +1303,7 @@ class SparseTensorConcatConverter : public OpConversionPattern<ConcatenateOp> {
       // Start a new COO or an initialized annotated all dense sparse tensor.
       dst = params.genBuffers(dstTp, dimSizes)
                 .genNewCall(allDense ? Action::kEmpty : Action::kEmptyCOO);
-      dstInd = genAlloca(rewriter, loc, dimRank, rewriter.getIndexType());
+      dstDimCoords = genAlloca(rewriter, loc, dimRank, rewriter.getIndexType());
       if (allDense) {
         dstTensor = dst;
         // Get the values buffer for the sparse tensor and reshape it to the
@@ -1338,9 +1311,12 @@ class SparseTensorConcatConverter : public OpConversionPattern<ConcatenateOp> {
         dst = genValuesCall(rewriter, loc,
                             MemRefType::get({ShapedType::kDynamic}, elemTp),
                             {dst});
-        // Use the dstInd to store the level sizes.
-        dst =
-            reshapeValuesToLevels(rewriter, loc, dstEnc, dimSizes, dst, dstInd);
+        // Pass the `dstDimCoords` buffer for `reshapeValuesToLevels`
+        // to reuse for storing level-sizes (yes, "level-sizes").
+        // This is safe to do because `dstTp` is a dense-tensor type,
+        // and therefore lvlRank == dimRank.
+        dst = reshapeValuesToLevels(rewriter, loc, dstEnc, dimSizes, dst,
+                                    dstDimCoords);
       } else {
         dstPerm = params.getDim2LvlMap();
         elemPtr = genAllocaScalar(rewriter, loc, elemTp);
@@ -1351,13 +1327,13 @@ class SparseTensorConcatConverter : public OpConversionPattern<ConcatenateOp> {
       dst = allocDenseTensor(rewriter, loc, dstTp, dimSizes);
     }
     const Level lvlRank = dstTp.getLvlRank();
-    const auto dimIvs2LvlIvs = [&](ValueRange dimIvs) -> SmallVector<Value> {
-      SmallVector<Value> lvlIvs;
-      lvlIvs.reserve(lvlRank);
+    const auto dcvs2lcvs = [&](ValueRange dcvs) -> SmallVector<Value> {
+      SmallVector<Value> lcvs;
+      lcvs.reserve(lvlRank);
       for (Level l = 0; l < lvlRank; l++)
         // FIXME: `toOrigDim` is deprecated
-        lvlIvs.push_back(dimIvs[toOrigDim(dstEnc, l)]);
-      return lvlIvs;
+        lcvs.push_back(dcvs[toOrigDim(dstEnc, l)]);
+      return lcvs;
     };
     for (const auto &it : llvm::zip(op.getInputs(), adaptor.getInputs())) {
       Value orignalOp = std::get<0>(it); // Input (with encoding) from Op
@@ -1366,45 +1342,45 @@ class SparseTensorConcatConverter : public OpConversionPattern<ConcatenateOp> {
       if (srcTp.hasEncoding()) {
         genSparseCOOIterationLoop(
             rewriter, loc, adaptedOp, srcTp,
-            [&](OpBuilder &builder, Location loc, Value idx,
+            [&](OpBuilder &builder, Location loc, Value dimCoords,
                 Value elemPtr) -> void {
-              SmallVector<Value> dimIvs =
-                  loadIndices(builder, loc, dimRank, idx, concatDim, offset);
+              const auto dcvs =
+                  loadAll(builder, loc, dimRank, dimCoords, concatDim, offset);
               if (dstTp.hasEncoding() && !allDense) {
                 // Case: sparse => sparse, except for annotated all dense.
-                storeIndices(builder, loc, dimRank, dstInd, dimIvs);
-                genAddEltCall(builder, loc, elemTp, dst, elemPtr, dstInd,
+                storeAll(builder, loc, dstDimCoords, dcvs);
+                genAddEltCall(builder, loc, elemTp, dst, elemPtr, dstDimCoords,
                               dstPerm);
               } else {
                 // Case: sparse => dense, or annotated all dense.
-                const auto lvlIvs = allDense ? dimIvs2LvlIvs(dimIvs) : dimIvs;
-                insertScalarIntoDenseTensor(builder, loc, elemPtr, dst, lvlIvs);
+                const auto lcvs = allDense ? dcvs2lcvs(dcvs) : dcvs;
+                insertScalarIntoDenseTensor(builder, loc, elemPtr, dst, lcvs);
               }
             });
       } else {
         genDenseTensorIterationLoop(
             rewriter, loc, adaptedOp, srcTp,
-            [&](OpBuilder &builder, Location loc, ValueRange dimIvs) -> void {
+            [&](OpBuilder &builder, Location loc, ValueRange dcvs) -> void {
               if (dstTp.hasEncoding() && !allDense) {
                 // Case: dense => sparse, except for annotated all dense.
-                storeIndices(builder, loc, dimRank, dstInd, dimIvs, concatDim,
-                             offset);
-                Value val = genValueForDense(builder, loc, adaptedOp, dimIvs);
+                assert(dcvs.size() == static_cast<size_t>(dimRank));
+                storeAll(builder, loc, dstDimCoords, dcvs, concatDim, offset);
+                Value val = genValueForDense(builder, loc, adaptedOp, dcvs);
                 builder.create<memref::StoreOp>(loc, val, elemPtr);
-                genAddEltCall(builder, loc, elemTp, dst, elemPtr, dstInd,
+                genAddEltCall(builder, loc, elemTp, dst, elemPtr, dstDimCoords,
                               dstPerm);
               } else {
                 // Case: dense => dense, or annotated all dense.
-                Value val = genValueForDense(builder, loc, adaptedOp, dimIvs);
-                // Despite the name, this isn't actually level-ivs until
-                // after the `dimIvs2LvlIvs` call.
-                SmallVector<Value> lvlIvs(dimIvs);
+                Value val = genValueForDense(builder, loc, adaptedOp, dcvs);
+                // Despite the name, this isn't actually level-cvs until
+                // after the `dcvs2lcvs` call.
+                SmallVector<Value> lcvs(dcvs);
                 // Apply offset.
-                lvlIvs[concatDim] = builder.create<arith::AddIOp>(
-                    loc, lvlIvs[concatDim], offset);
+                lcvs[concatDim] =
+                    builder.create<arith::AddIOp>(loc, lcvs[concatDim], offset);
                 if (allDense)
-                  lvlIvs = dimIvs2LvlIvs(lvlIvs);
-                builder.create<memref::StoreOp>(loc, val, dst, lvlIvs);
+                  lcvs = dcvs2lcvs(lcvs);
+                builder.create<memref::StoreOp>(loc, val, dst, lcvs);
               }
             });
       }
@@ -1446,11 +1422,11 @@ class SparseTensorOutConverter : public OpConversionPattern<OutOp> {
     Value coo = NewCallParams(rewriter, loc)
                     .genBuffers(srcTp.withoutOrdering(), dimSizes)
                     .genNewCall(Action::kToCOO, src);
-    // Then output the tensor to external file with indices in the externally
-    // visible lexicographic index order. A sort is required if the source was
-    // not in that order yet (note that the sort can be dropped altogether if
-    // external format does not care about the order at all, but here we assume
-    // it does).
+    // Then output the tensor to external file with coordinates in the
+    // externally visible lexicographic coordinate order.  A sort is
+    // required if the source was not in that order yet (note that the
+    // sort can be dropped altogether if external format does not care
+    // about the order at all, but here we assume it does).
     const Value sort = constantI1(rewriter, loc, !srcTp.isIdentity());
     SmallVector<Value, 3> outParams{coo, adaptor.getOperands()[1], sort};
     const Type elemTp = srcTp.getElementType();
@@ -1482,17 +1458,18 @@ mlir::SparseTensorTypeToPtrConverter::SparseTensorTypeToPtrConverter() {
 void mlir::populateSparseTensorConversionPatterns(
     TypeConverter &typeConverter, RewritePatternSet &patterns,
     const SparseTensorConversionOptions &options) {
-  patterns.add<SparseReturnConverter, SparseTensorToDimSizeConverter,
-               SparseCastConverter, SparseTensorNewConverter,
-               SparseReshapeConverter<tensor::ExpandShapeOp>,
-               SparseReshapeConverter<tensor::CollapseShapeOp>,
-               SparseTensorConcatConverter, SparseTensorAllocConverter,
-               SparseTensorDeallocConverter, SparseTensorToPointersConverter,
-               SparseTensorToIndicesConverter, SparseTensorToValuesConverter,
-               SparseNumberOfEntriesConverter, SparseTensorLoadConverter,
-               SparseTensorInsertConverter, SparseTensorExpandConverter,
-               SparseTensorCompressConverter, SparseTensorOutConverter>(
-      typeConverter, patterns.getContext());
+  patterns
+      .add<SparseReturnConverter, SparseTensorToDimSizeConverter,
+           SparseCastConverter, SparseTensorNewConverter,
+           SparseReshapeConverter<tensor::ExpandShapeOp>,
+           SparseReshapeConverter<tensor::CollapseShapeOp>,
+           SparseTensorConcatConverter, SparseTensorAllocConverter,
+           SparseTensorDeallocConverter, SparseTensorToPositionsConverter,
+           SparseTensorToCoordinatesConverter, SparseTensorToValuesConverter,
+           SparseNumberOfEntriesConverter, SparseTensorLoadConverter,
+           SparseTensorInsertConverter, SparseTensorExpandConverter,
+           SparseTensorCompressConverter, SparseTensorOutConverter>(
+          typeConverter, patterns.getContext());
 
   patterns.add<SparseTensorConvertConverter>(typeConverter,
                                              patterns.getContext(), options);
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorPasses.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorPasses.cpp
index fc4af709d5d8c..2163dcb5fed3a 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorPasses.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorPasses.cpp
@@ -63,12 +63,13 @@ struct SparsificationPass
   SparsificationPass(const SparsificationPass &pass) = default;
   SparsificationPass(const SparsificationOptions &options) {
     parallelization = options.parallelizationStrategy;
+    enableIndexReduction = options.enableIndexReduction;
   }
 
   void runOnOperation() override {
     auto *ctx = &getContext();
     // Translate strategy flags to strategy options.
-    SparsificationOptions options(parallelization);
+    SparsificationOptions options(parallelization, enableIndexReduction);
     // Apply sparsification and cleanup rewriting.
     RewritePatternSet patterns(ctx);
     populateSparsificationPatterns(patterns, options);
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorRewriting.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorRewriting.cpp
index 0663bd927fe5e..b08ca4961f467 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorRewriting.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorRewriting.cpp
@@ -159,16 +159,16 @@ static LogicalResult genForeachOnSparseConstant(ForeachOp op,
   // Foreach on constant.
   foreachInSparseConstant(
       loc, rewriter, attr, op.getOrder().value_or(AffineMap()),
-      [&reduc, &rewriter, op](ArrayRef<Value> coords, Value v) mutable {
+      [&reduc, &rewriter, op](ArrayRef<Value> cvs, Value v) mutable {
         SmallVector<Value> args;
-        args.append(coords.begin(), coords.end());
+        args.append(cvs.begin(), cvs.end());
         args.push_back(v);
         args.append(reduc);
         // Clones the foreach op to get a copy of the loop body.
         auto cloned = cast<ForeachOp>(rewriter.clone(*op.getOperation()));
         assert(args.size() == cloned.getBody()->getNumArguments());
         Operation *yield = cloned.getBody()->getTerminator();
-        rewriter.mergeBlockBefore(cloned.getBody(), op, args);
+        rewriter.inlineBlockBefore(cloned.getBody(), op, args);
         // clean up
         rewriter.eraseOp(cloned);
         reduc = yield->getOperands();
@@ -377,16 +377,16 @@ struct Sparse2SparseReshapeRewriter : public OpRewritePattern<ReshapeOp> {
       ArrayRef<int64_t> dstShape = dstTp.getShape();
       genReshapeDstShape(loc, rewriter, dstSizes, srcSizes, dstShape,
                          op.getReassociationIndices());
-      for (auto &d : llvm::enumerate(dstShape)) {
-        if (d.value() == ShapedType::kDynamic)
-          dstDynSizes.push_back(dstSizes[d.index()]);
+      for (auto [idx, shape] : llvm::enumerate(dstShape)) {
+        if (shape == ShapedType::kDynamic)
+          dstDynSizes.push_back(dstSizes[idx]);
       }
     }
 
     // Implement the sparse2sparse reshape as follows:
     //   %tmp = bufferization.alloc_tensor : unordered COO
     //   foreach srcCoords %srcTensor
-    //     insert translateIndicesArray(srcCoords), %tmp
+    //     insert reshapeCvs(srcCoords), %tmp
     //   %t = sparse_tensor.cast %tmp
     Value nnz = rewriter.create<NumberOfEntriesOp>(loc, srcTensor);
     RankedTensorType cooTp = getUnorderedCOOFromType(dstTp);
@@ -398,18 +398,20 @@ struct Sparse2SparseReshapeRewriter : public OpRewritePattern<ReshapeOp> {
 
     ForeachOp foreachOp = rewriter.create<ForeachOp>(
         loc, srcTensor, cooBuffer,
-        [&](OpBuilder &builder, Location loc, ValueRange args, Value v,
+        [&](OpBuilder &builder, Location loc, ValueRange srcLcvs, Value v,
             ValueRange reduc) {
-          SmallVector<Value> srcIndices;
-          SmallVector<Value> dstIndices;
-          for (Dimension d = 0, dimRank = srcTp.getRank(); d < dimRank; d++) {
+          const Dimension dimRank = srcTp.getRank();
+          SmallVector<Value> srcDcvs;
+          srcDcvs.reserve(dimRank);
+          for (Dimension d = 0; d < dimRank; d++) {
             // FIXME: `toStoredDim` is deprecated
             Level lvl = toStoredDim(encSrc, d);
-            srcIndices.push_back(args[lvl]);
+            srcDcvs.push_back(srcLcvs[lvl]);
           }
-          translateIndicesArray(builder, loc, op.getReassociationIndices(),
-                                srcIndices, srcSizes, dstSizes, dstIndices);
-          auto t = builder.create<InsertOp>(loc, v, reduc.front(), dstIndices);
+          SmallVector<Value> dstDcvs;
+          reshapeCvs(builder, loc, op.getReassociationIndices(), srcSizes,
+                     srcDcvs, dstSizes, dstDcvs);
+          auto t = builder.create<InsertOp>(loc, v, reduc.front(), dstDcvs);
           builder.create<sparse_tensor::YieldOp>(loc, t);
         });
     auto t = rewriter.create<LoadOp>(loc, foreachOp.getResult(0), true);
@@ -468,7 +470,7 @@ struct ConcatenateRewriter : public OpRewritePattern<ConcatenateOp> {
     const Location loc = op.getLoc();
     const auto dstTp = getSparseTensorType(op);
     const Dimension dimRank = dstTp.getDimRank();
-    const Dimension conDim = op.getDimension().getZExtValue();
+    const Dimension conDim = op.getDimension();
     SmallVector<Value> sizes;
     concatSizesFromInputs(rewriter, sizes, loc, dstTp, op.getInputs(), conDim);
 
@@ -523,12 +525,12 @@ struct ConcatenateRewriter : public OpRewritePattern<ConcatenateOp> {
         // Create a view of the values buffer to match the unannotated dense
         // tensor.
         Value valuesBuffer = genToValues(rewriter, loc, dst);
-        Value idxBuffer =
+        Value dimCoords =
             genAlloca(rewriter, loc, dimRank, rewriter.getIndexType(),
                       /*staticShape=*/true);
         annotatedDenseDst = dst;
         dst = reshapeValuesToLevels(rewriter, loc, encDst, sizes, valuesBuffer,
-                                    idxBuffer);
+                                    dimCoords);
       }
     } else {
       // TODO: Dense buffers should be allocated/deallocated via the callback
@@ -546,16 +548,16 @@ struct ConcatenateRewriter : public OpRewritePattern<ConcatenateOp> {
       // output tensor.
       foreachOp = rewriter.create<ForeachOp>(
           loc, input, initArgs,
-          [&](OpBuilder &builder, Location loc, ValueRange args, Value v,
+          [&](OpBuilder &builder, Location loc, ValueRange dcvs, Value v,
               ValueRange reduc) {
-            SmallVector<Value> indices(dstTp.getLvlRank());
+            SmallVector<Value> dstLcvs(dstTp.getLvlRank());
             for (Dimension d = 0; d < dimRank; d++) {
-              Value idx = args[d];
+              Value crd = dcvs[d];
               if (d == conDim)
                 // Transform coordinates for the concatenating dim.
-                idx = builder.create<arith::AddIOp>(loc, idx, offset);
+                crd = builder.create<arith::AddIOp>(loc, crd, offset);
               // FIXME: `toStoredDim` is deprecated
-              indices[toStoredDim(encDst, d)] = idx;
+              dstLcvs[toStoredDim(encDst, d)] = crd;
             }
             if (encDst && !allDense) {
               Value cond = genIsNonzero(rewriter, loc, v);
@@ -563,14 +565,14 @@ struct ConcatenateRewriter : public OpRewritePattern<ConcatenateOp> {
                   loc, TypeRange(reduc.front().getType()), cond, /*else*/ true);
               builder.setInsertionPointToStart(&ifOp.getThenRegion().front());
               Value t =
-                  builder.create<InsertOp>(loc, v, reduc.front(), indices);
+                  builder.create<InsertOp>(loc, v, reduc.front(), dstLcvs);
               rewriter.create<scf::YieldOp>(loc, t);
               rewriter.setInsertionPointToStart(&ifOp.getElseRegion().front());
               rewriter.create<scf::YieldOp>(loc, reduc.front());
               rewriter.setInsertionPointAfter(ifOp);
               rewriter.create<sparse_tensor::YieldOp>(loc, ifOp.getResult(0));
             } else {
-              builder.create<memref::StoreOp>(loc, v, dst, indices);
+              builder.create<memref::StoreOp>(loc, v, dst, dstLcvs);
               builder.create<sparse_tensor::YieldOp>(loc);
             }
           });
@@ -654,7 +656,7 @@ struct ConvertRewriter : public OpRewritePattern<ConvertOp> {
   // To fill the COO tensor from a sparse constant in COO format:
   //   for i in range(NNZ)
   //     val = values[i]
-  //     [i1,..,ik] = indices[i]
+  //     [i1,..,ik] = coordinates[i]
   //     t->add(val, [i1,..,ik], [p1,..,pk])
   LogicalResult dense2SparseRewrite(ConvertOp op,
                                     PatternRewriter &rewriter) const {
@@ -679,31 +681,38 @@ struct ConvertRewriter : public OpRewritePattern<ConvertOp> {
     // COO tensor.
     // TODO: enhance foreachOp to take ordering to remove the need of a
     // temporary COO tensor here.
-    const RankedTensorType bufferTp = dstTp.isIdentity()
+    const RankedTensorType bufferTp = dstTp.isIdentity() || fromSparseConst
                                           ? dstTp.getRankedTensorType()
                                           : getUnorderedCOOFromTypeWithOrdering(
                                                 dstTp, dstTp.getDimToLvlMap());
+    // Only imposes foreach order on dense constant (which will be statically
+    // sorted by the sparse compiler), otherwise the rotated loop sequence
+    // results to bad cache locality.
+    AffineMapAttr foreachOrder = nullptr;
+    if (encDst.getDimOrdering() && fromSparseConst)
+      foreachOrder = AffineMapAttr::get(encDst.getDimOrdering());
+
     auto buffer =
         rewriter.create<AllocTensorOp>(loc, bufferTp, dynSizes).getResult();
     auto foreachOp = rewriter.create<ForeachOp>(
-        loc, src, buffer,
-        [&](OpBuilder &builder, Location loc, ValueRange indices, Value v,
+        loc, src, buffer, foreachOrder,
+        [&](OpBuilder &builder, Location loc, ValueRange dcvs, Value v,
             ValueRange reduc) {
           Value input = reduc.front();
           const Dimension dimRank = dstTp.getDimRank();
-          SmallVector<Value> indicesArray(dimRank);
+          const Level lvlRank = dstTp.getLvlRank();
+          SmallVector<Value> lcvs(lvlRank);
           for (Dimension d = 0; d < dimRank; d++)
             // FIXME: `toStoredDim` is deprecated
-            indicesArray[toStoredDim(encDst, d)] = indices[d];
+            lcvs[toStoredDim(encDst, d)] = dcvs[d];
           if (fromSparseConst) {
-            input = builder.create<InsertOp>(loc, v, input, indicesArray);
+            input = builder.create<InsertOp>(loc, v, input, lcvs);
           } else {
             Value cond = genIsNonzero(builder, loc, v);
             auto ifOp = builder.create<scf::IfOp>(
                 loc, TypeRange(input.getType()), cond, /*else*/ true);
             builder.setInsertionPointToStart(&ifOp.getThenRegion().front());
-            Value insert =
-                builder.create<InsertOp>(loc, v, input, indicesArray);
+            Value insert = builder.create<InsertOp>(loc, v, input, lcvs);
             builder.create<scf::YieldOp>(loc, insert);
             builder.setInsertionPointToStart(&ifOp.getElseRegion().front());
             builder.create<scf::YieldOp>(loc, input);
@@ -728,7 +737,7 @@ struct ConvertRewriter : public OpRewritePattern<ConvertOp> {
   // Handles sparse tensor to dense tensor conversion as follows:
   //   dst = new dense tensor;
   //   foreach elemment in src
-  //     dst[elemment.indices] = element.value
+  //     dst[element.coords] = element.value
   LogicalResult sparse2DenseRewrite(ConvertOp op,
                                     PatternRewriter &rewriter) const {
     Location loc = op->getLoc();
@@ -793,7 +802,6 @@ struct ConvertRewriter : public OpRewritePattern<ConvertOp> {
     // tensor (e.g., src tensor is not ordered or src tensor haves a different
     // dimOrdering).
     if (const SparseTensorType srcTp(srcRTT);
-        !isUniqueCOOType(srcRTT) &&
         !(srcTp.isAllOrdered() && srcTp.hasSameDimToLvlMap(dstTp))) {
       // Construct a COO tensor from the src tensor.
       // TODO: there may be cases for which more efficiently without
@@ -811,16 +819,15 @@ struct ConvertRewriter : public OpRewritePattern<ConvertOp> {
                    .getResult();
       auto foreachOp = rewriter.create<ForeachOp>(
           loc, src, tmpCoo,
-          [&](OpBuilder &builder, Location loc, ValueRange args, Value v,
+          [&](OpBuilder &builder, Location loc, ValueRange dcvs, Value v,
               ValueRange reduc) {
-            SmallVector<Value> dstIndices(dstLvlRank);
+            SmallVector<Value> dstLcvs(dstLvlRank);
             for (Dimension d = 0; d < dimRank; d++) {
               // FIXME: `toStoredDim` is deprecated
               Level l = toStoredDim(encDst, d);
-              dstIndices[l] = args[d];
+              dstLcvs[l] = dcvs[d];
             }
-            auto t =
-                builder.create<InsertOp>(loc, v, reduc.front(), dstIndices);
+            auto t = builder.create<InsertOp>(loc, v, reduc.front(), dstLcvs);
             builder.create<sparse_tensor::YieldOp>(loc, t);
           });
       src = rewriter.create<LoadOp>(loc, foreachOp.getResult(0), true);
@@ -836,18 +843,17 @@ struct ConvertRewriter : public OpRewritePattern<ConvertOp> {
       Value y = genToValues(rewriter, loc, src);
       const auto encSrc = srcTp.getEncoding();
       // Sort the COO tensor so that its elements are ordered via increasing
-      // indices for the storage ordering of the dst tensor. Use SortCoo if the
-      // COO tensor has the same dim ordering as the dst tensor.
+      // coordinates for the storage ordering of the dst tensor.  Use SortCoo
+      // if the COO tensor has the same ordering as the dst tensor.
       if (dimRank > 1 && srcTp.hasSameDimToLvlMap(dstTp)) {
-        MemRefType indTp =
-            get1DMemRefType(getIndexOverheadType(rewriter, encSrc),
-                            /*withLayout=*/false);
-        Value xs = rewriter.create<ToIndicesBufferOp>(loc, indTp, src);
+        MemRefType coordsTp =
+            get1DMemRefType(encSrc.getCrdType(), /*withLayout=*/false);
+        Value xs = rewriter.create<ToCoordinatesBufferOp>(loc, coordsTp, src);
         rewriter.create<SortCooOp>(
             loc, nnz, xs, ValueRange{y}, rewriter.getIndexAttr(dimRank),
             rewriter.getIndexAttr(0), SparseTensorSortKind::HybridQuickSort);
       } else {
-        // Gather the indices-arrays in the dst tensor storage order.
+        // Gather the coordinates-arrays in the dst tensor storage order.
         SmallVector<Value> xs(dstLvlRank);
         const Level srcLvlRank = srcTp.getLvlRank();
         for (Level srcLvl = 0; srcLvl < srcLvlRank; srcLvl++) {
@@ -855,7 +861,8 @@ struct ConvertRewriter : public OpRewritePattern<ConvertOp> {
           Dimension dim = toOrigDim(encSrc, srcLvl);
           // FIXME: `toStoredDim` is deprecated
           Level dstLvl = toStoredDim(encDst, dim);
-          xs[dstLvl] = genToIndices(rewriter, loc, src, srcLvl, /*cooStart=*/0);
+          xs[dstLvl] =
+              genToCoordinates(rewriter, loc, src, srcLvl, /*cooStart=*/0);
         }
         rewriter.create<SortOp>(loc, nnz, xs, ValueRange{y},
                                 SparseTensorSortKind::HybridQuickSort);
@@ -870,17 +877,17 @@ struct ConvertRewriter : public OpRewritePattern<ConvertOp> {
                                            dynDstSizes, Value(),
                                            /*sizeHint=*/nnz, Attribute())
                     .getResult();
-    SmallVector<Value> indices(dstLvlRank);
+    SmallVector<Value> dstLcvs(dstLvlRank);
     auto foreachOp = rewriter.create<ForeachOp>(
         loc, src, dst,
-        [&](OpBuilder &builder, Location loc, ValueRange args, Value v,
+        [&](OpBuilder &builder, Location loc, ValueRange dcvs, Value v,
             ValueRange reduc) {
           for (Dimension d = 0; d < dimRank; d++) {
             // FIXME: `toStoredDim` is deprecated
             Level l = toStoredDim(encDst, d);
-            indices[l] = args[d];
+            dstLcvs[l] = dcvs[d];
           }
-          auto t = builder.create<InsertOp>(loc, v, reduc.front(), indices);
+          auto t = builder.create<InsertOp>(loc, v, reduc.front(), dstLcvs);
           builder.create<sparse_tensor::YieldOp>(loc, t);
         });
 
@@ -913,6 +920,7 @@ struct ForeachRewriter : public OpRewritePattern<ForeachOp> {
     SmallVector<Value> reduc = op.getInitArgs();
     const auto stt = getSparseTensorType(input);
     const Dimension dimRank = stt.getDimRank();
+    const Level lvlRank = stt.getLvlRank();
 
     // Special-case: for each over a sparse constant uses its own rewriting
     // rule.
@@ -930,37 +938,36 @@ struct ForeachRewriter : public OpRewritePattern<ForeachOp> {
         ValueRange{input},
         StringAttr::get(getContext(), ForeachOp::getOperationName()));
     loopEmitter.initializeLoopEmit(rewriter, loc);
-    for (Dimension d = 0; d < dimRank; d++) {
+    for (Level l = 0; l < lvlRank; l++) {
       // TODO: provide utility function for loop sequences that only contains
       // one for loop?
-      Dimension ld =
-          op.getOrder()
-              ? op.getOrder()->getResult(d).cast<AffineDimExpr>().getPosition()
-              : d;
-      loopEmitter.enterNewLoopSeq(rewriter, loc, 0, static_cast<size_t>(ld));
+      // FIXME(wrengr): what is this "ld" supposed to be really?
+      const Level ld = op.getOrder() ? op.getOrder()->getDimPosition(l) : l;
+      loopEmitter.enterNewLoopSeq(rewriter, loc, 0, ld);
       // Note that reduc will be taken care of by loop emitter and get updated
       // in place.
 
-      loopEmitter.enterLoopOverTensorAtDim(rewriter, loc, 0, ld, reduc);
+      loopEmitter.enterLoopOverTensorAtLvl(rewriter, loc, 0, l, reduc);
     }
 
-    SmallVector<Value> coords;
-    coords.reserve(dimRank);
-    loopEmitter.getCoordinateArray(coords);
+    SmallVector<Value> lcvs;
+    lcvs.reserve(lvlRank);
+    loopEmitter.getLoopIVs(lcvs);
 
     if (op.getOrder()) {
-      SmallVector<Value> tmp = coords; // keep a copy
+      // FIXME: There is some dim/lvl confusion here since `dimRank != lvlRank`
+      SmallVector<Value> dcvs = lcvs; // keep a copy
       for (Dimension d = 0; d < dimRank; d++) {
         auto l = op.getOrder()->getDimPosition(d);
-        coords[l] = tmp[d];
+        lcvs[l] = dcvs[d];
       }
     }
     Value vals = loopEmitter.getValBuffer()[0];
-    Value pidx = loopEmitter.getPidxs()[0].back();
-    // Loads the value from sparse tensor using pointer index;
-    // loads the value from dense tensor using coordinate array.
-    Value val = enc ? rewriter.create<memref::LoadOp>(loc, vals, pidx)
-                    : rewriter.create<memref::LoadOp>(loc, vals, coords);
+    Value pos = loopEmitter.getPosits()[0].back();
+    // Loads the value from sparse tensor using position-index;
+    // loads the value from dense tensor using coords.
+    Value val = enc ? rewriter.create<memref::LoadOp>(loc, vals, pos)
+                    : rewriter.create<memref::LoadOp>(loc, vals, lcvs);
 
     // 2. Inline the block in the foreach operator.
     Block *srcBlock = op.getBody();
@@ -969,8 +976,8 @@ struct ForeachRewriter : public OpRewritePattern<ForeachOp> {
     SmallVector<Value> args;
     for (Dimension d = 0; d < dimRank; d++) {
       // FIXME: `toStoredDim` is deprecated
-      Value actual = coords[toStoredDim(enc, d)];
-      args.push_back(actual);
+      Value dimCrd = lcvs[toStoredDim(enc, d)];
+      args.push_back(dimCrd);
     }
     // Remap value.
     args.push_back(val);
@@ -988,7 +995,8 @@ struct ForeachRewriter : public OpRewritePattern<ForeachOp> {
       // This is annoying, since scf.for inserts a implicit yield op when
       // there is no reduction variable upon creation, in this case we need to
       // merge the block *before* the yield op.
-      rewriter.mergeBlockBefore(srcBlock, &*rewriter.getInsertionPoint(), args);
+      rewriter.inlineBlockBefore(srcBlock, &*rewriter.getInsertionPoint(),
+                                 args);
     }
 
     for (Dimension d = 0; d < dimRank; d++) {
@@ -1018,7 +1026,7 @@ struct NewRewriter : public OpRewritePattern<NewOp> {
 
     // Implement the NewOp as follows:
     //   %orderedCoo = sparse_tensor.new %filename
-    //   %t = sparse_tensor.ConvertOp %orderedCoo
+    //   %t = sparse_tensor.convert %orderedCoo
     RankedTensorType cooTp =
         getCOOFromTypeWithOrdering(dstTp, encDst.getDimOrdering(), true);
     Value cooTensor = rewriter.create<NewOp>(loc, cooTp, op.getSource());
@@ -1042,7 +1050,7 @@ struct OutRewriter : public OpRewritePattern<OutOp> {
     Value src = op.getTensor();
     Value nnz = rewriter.create<NumberOfEntriesOp>(loc, src);
 
-    // Allocate a temporary buffer for storing dimension sizes and indices.
+    // Allocate a temporary buffer for storing dimension-sizes/coordinates.
     const auto srcTp = getSparseTensorType(src);
     const Dimension dimRank = srcTp.getDimRank();
     Type indexTp = rewriter.getIndexType();
@@ -1067,7 +1075,7 @@ struct OutRewriter : public OpRewritePattern<OutOp> {
     createFuncCall(rewriter, loc, "outSparseTensorWriterMetaData", {},
                    {writer, rankValue, nnz, dimSizes}, EmitCInterface::On);
 
-    Value indices = dimSizes; // Reuse the dimSizes buffer for indices.
+    Value dimCoords = dimSizes; // Reuse the dimSizes buffer for dimCoords.
     Type eltTp = srcTp.getElementType();
     SmallString<29> outNextFuncName{"outSparseTensorWriterNext",
                                     primaryTypeFunctionSuffix(eltTp)};
@@ -1076,14 +1084,14 @@ struct OutRewriter : public OpRewritePattern<OutOp> {
     // For each element in the source tensor, output the element.
     rewriter.create<ForeachOp>(
         loc, src, std::nullopt,
-        [&](OpBuilder &builder, Location loc, ValueRange args, Value v,
+        [&](OpBuilder &builder, Location loc, ValueRange dcvs, Value v,
             ValueRange reduc) {
           for (Dimension d = 0; d < dimRank; d++) {
-            rewriter.create<memref::StoreOp>(loc, args[d], indices,
+            rewriter.create<memref::StoreOp>(loc, dcvs[d], dimCoords,
                                              constantIndex(builder, loc, d));
           }
           rewriter.create<memref::StoreOp>(loc, v, value);
-          SmallVector<Value> operands{writer, rankValue, indices, value};
+          SmallVector<Value> operands{writer, rankValue, dimCoords, value};
           FlatSymbolRefAttr fn = getFunc(module, outNextFuncName, {}, operands,
                                          EmitCInterface::On);
           builder.create<func::CallOp>(loc, TypeRange(), fn, operands);
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorStorageLayout.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorStorageLayout.cpp
index b3336b5fc6ae2..4c98ae1813758 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorStorageLayout.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorStorageLayout.cpp
@@ -22,11 +22,11 @@ using namespace sparse_tensor;
 // Private helper methods.
 //===----------------------------------------------------------------------===//
 
-static IntegerAttr fromOptionalInt(MLIRContext *ctx,
-                                   std::optional<unsigned> dim) {
-  if (!dim)
-    return nullptr;
-  return IntegerAttr::get(IndexType::get(ctx), dim.value());
+/// Constructs a nullable `LevelAttr` from the `std::optional<Level>`.
+static IntegerAttr optionalLevelAttr(MLIRContext *ctx,
+                                     std::optional<Level> lvl) {
+  return lvl ? IntegerAttr::get(IndexType::get(ctx), lvl.value())
+             : IntegerAttr();
 }
 
 // This is only ever called from `SparseTensorTypeToBufferConverter`,
@@ -82,36 +82,37 @@ Value SparseTensorSpecifier::getInitValue(OpBuilder &builder, Location loc,
 
 Value SparseTensorSpecifier::getSpecifierField(OpBuilder &builder, Location loc,
                                                StorageSpecifierKind kind,
-                                               std::optional<unsigned> dim) {
+                                               std::optional<Level> lvl) {
   return builder.create<GetStorageSpecifierOp>(
-      loc, specifier, kind, fromOptionalInt(specifier.getContext(), dim));
+      loc, specifier, kind, optionalLevelAttr(specifier.getContext(), lvl));
 }
 
 void SparseTensorSpecifier::setSpecifierField(OpBuilder &builder, Location loc,
                                               Value v,
                                               StorageSpecifierKind kind,
-                                              std::optional<unsigned> dim) {
+                                              std::optional<Level> lvl) {
+  // TODO: make `v` have type `TypedValue<IndexType>` instead.
   assert(v.getType().isIndex());
   specifier = builder.create<SetStorageSpecifierOp>(
-      loc, specifier, kind, fromOptionalInt(specifier.getContext(), dim), v);
+      loc, specifier, kind, optionalLevelAttr(specifier.getContext(), lvl), v);
 }
 
 //===----------------------------------------------------------------------===//
 // SparseTensorDescriptor methods.
 //===----------------------------------------------------------------------===//
 
-Value sparse_tensor::SparseTensorDescriptor::getIdxMemRefOrView(
-    OpBuilder &builder, Location loc, Level idxLvl) const {
+Value sparse_tensor::SparseTensorDescriptor::getCrdMemRefOrView(
+    OpBuilder &builder, Location loc, Level lvl) const {
   const Level cooStart = getCOOStart(rType.getEncoding());
-  if (idxLvl < cooStart)
-    return getMemRefField(SparseTensorFieldKind::IdxMemRef, idxLvl);
+  if (lvl < cooStart)
+    return getMemRefField(SparseTensorFieldKind::CrdMemRef, lvl);
 
   Value stride = constantIndex(builder, loc, rType.getLvlRank() - cooStart);
-  Value size = getIdxMemSize(builder, loc, cooStart);
+  Value size = getCrdMemSize(builder, loc, cooStart);
   size = builder.create<arith::DivUIOp>(loc, size, stride);
   return builder.create<memref::SubViewOp>(
-      loc, getMemRefField(SparseTensorFieldKind::IdxMemRef, cooStart),
-      /*offset=*/ValueRange{constantIndex(builder, loc, idxLvl - cooStart)},
+      loc, getMemRefField(SparseTensorFieldKind::CrdMemRef, cooStart),
+      /*offset=*/ValueRange{constantIndex(builder, loc, lvl - cooStart)},
       /*size=*/ValueRange{size},
       /*step=*/ValueRange{stride});
 }
@@ -129,8 +130,8 @@ void sparse_tensor::foreachFieldInSparseTensor(
         callback) {
   assert(enc);
 
-#define RETURN_ON_FALSE(idx, kind, dim, dlt)                                   \
-  if (!(callback(idx, kind, dim, dlt)))                                        \
+#define RETURN_ON_FALSE(fidx, kind, dim, dlt)                                  \
+  if (!(callback(fidx, kind, dim, dlt)))                                       \
     return;
 
   const auto lvlTypes = enc.getDimLevelType();
@@ -145,10 +146,10 @@ void sparse_tensor::foreachFieldInSparseTensor(
     // order.
     const auto dlt = lvlTypes[l];
     if (isCompressedDLT(dlt)) {
-      RETURN_ON_FALSE(fieldIdx++, SparseTensorFieldKind::PtrMemRef, l, dlt);
-      RETURN_ON_FALSE(fieldIdx++, SparseTensorFieldKind::IdxMemRef, l, dlt);
+      RETURN_ON_FALSE(fieldIdx++, SparseTensorFieldKind::PosMemRef, l, dlt);
+      RETURN_ON_FALSE(fieldIdx++, SparseTensorFieldKind::CrdMemRef, l, dlt);
     } else if (isSingletonDLT(dlt)) {
-      RETURN_ON_FALSE(fieldIdx++, SparseTensorFieldKind::IdxMemRef, l, dlt);
+      RETURN_ON_FALSE(fieldIdx++, SparseTensorFieldKind::CrdMemRef, l, dlt);
     } else {
       assert(isDenseDLT(dlt)); // no fields
     }
@@ -171,30 +172,30 @@ void sparse_tensor::foreachFieldAndTypeInSparseTensor(
         callback) {
   assert(stt.hasEncoding());
   // Construct the basic types.
-  Type idxType = stt.getIndexType();
-  Type ptrType = stt.getPointerType();
-  Type eltType = stt.getElementType();
-
-  Type metaDataType = StorageSpecifierType::get(stt.getEncoding());
-  // memref<? x ptr>  pointers
-  Type ptrMemType = MemRefType::get({ShapedType::kDynamic}, ptrType);
-  // memref<? x idx>  indices
-  Type idxMemType = MemRefType::get({ShapedType::kDynamic}, idxType);
+  const Type crdType = stt.getCrdType();
+  const Type posType = stt.getPosType();
+  const Type eltType = stt.getElementType();
+
+  const Type metaDataType = StorageSpecifierType::get(stt.getEncoding());
+  // memref<? x pos>  positions
+  const Type posMemType = MemRefType::get({ShapedType::kDynamic}, posType);
+  // memref<? x crd>  coordinates
+  const Type crdMemType = MemRefType::get({ShapedType::kDynamic}, crdType);
   // memref<? x eltType> values
-  Type valMemType = MemRefType::get({ShapedType::kDynamic}, eltType);
+  const Type valMemType = MemRefType::get({ShapedType::kDynamic}, eltType);
 
   foreachFieldInSparseTensor(
       stt.getEncoding(),
-      [metaDataType, ptrMemType, idxMemType, valMemType,
+      [metaDataType, posMemType, crdMemType, valMemType,
        callback](FieldIndex fieldIdx, SparseTensorFieldKind fieldKind,
                  Level lvl, DimLevelType dlt) -> bool {
         switch (fieldKind) {
         case SparseTensorFieldKind::StorageSpec:
           return callback(metaDataType, fieldIdx, fieldKind, lvl, dlt);
-        case SparseTensorFieldKind::PtrMemRef:
-          return callback(ptrMemType, fieldIdx, fieldKind, lvl, dlt);
-        case SparseTensorFieldKind::IdxMemRef:
-          return callback(idxMemType, fieldIdx, fieldKind, lvl, dlt);
+        case SparseTensorFieldKind::PosMemRef:
+          return callback(posMemType, fieldIdx, fieldKind, lvl, dlt);
+        case SparseTensorFieldKind::CrdMemRef:
+          return callback(crdMemType, fieldIdx, fieldKind, lvl, dlt);
         case SparseTensorFieldKind::ValMemRef:
           return callback(valMemType, fieldIdx, fieldKind, lvl, dlt);
         };
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorStorageLayout.h b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorStorageLayout.h
index c30a15d87baca..788ad28ee4221 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorStorageLayout.h
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorStorageLayout.h
@@ -38,54 +38,68 @@ namespace sparse_tensor {
 //   ;  if dense:
 //        <nothing>
 //   ;  if compresed:
-//        memref<? x ptr>  pointers-l  ; pointers for sparse level l
-//        memref<? x idx>  indices-l   ; indices for sparse level l
+//        memref<? x pos>  positions-l   ; positions for sparse level l
+//        memref<? x crd>  coordinates-l ; coordinates for sparse level l
 //   ;  if singleton:
-//        memref<? x idx>  indices-l   ; indices for singleton level l
+//        memref<? x crd>  coordinates-l ; coordinates for singleton level l
 //
 //   memref<? x eltType> values        ; values
 //
 //   struct sparse_tensor.storage_specifier {
-//     array<rank x int> dimSizes    ; sizes for each dimension
-//     array<n x int> memSizes;      ; sizes for each data memref
+//     array<rank x int> lvlSizes    ; sizes/cardinalities for each level
+//     array<n x int> memSizes;      ; sizes/lengths for each data memref
 //   }
 // };
 //
 // In addition, for a "trailing COO region", defined as a compressed level
-// followed by one ore more singleton levels, the default SOA storage that
+// followed by one or more singleton levels, the default SOA storage that
 // is inherent to the TACO format is optimized into an AOS storage where
-// all indices of a stored element appear consecutively.  In such cases,
-// a special operation (sparse_tensor.indices_buffer) must be used to
-// access the AOS index array. In the code below, the method `getCOOStart`
+// all coordinates of a stored element appear consecutively.  In such cases,
+// a special operation (sparse_tensor.coordinates_buffer) must be used to
+// access the AOS coordinates array. In the code below, the method `getCOOStart`
 // is used to find the start of the "trailing COO region".
 //
+// If the sparse tensor is a slice (produced by `tensor.extract_slice`
+// operation), instead of allocating a new sparse tensor for it, it reuses the
+// same sets of MemRefs but attaching a additional set of slicing-metadata for
+// per-dimension slice offset and stride.
+//
 // Examples.
 //
 // #CSR storage of 2-dim matrix yields
-//   memref<?xindex>                           ; pointers-1
-//   memref<?xindex>                           ; indices-1
+//   memref<?xindex>                           ; positions-1
+//   memref<?xindex>                           ; coordinates-1
 //   memref<?xf64>                             ; values
 //   struct<(array<2 x i64>, array<3 x i64>)>) ; lvl0, lvl1, 3xsizes
 //
 // #COO storage of 2-dim matrix yields
-//   memref<?xindex>,                          ; pointers-0, essentially [0,sz]
-//   memref<?xindex>                           ; AOS index storage
+//   memref<?xindex>,                          ; positions-0, essentially [0,sz]
+//   memref<?xindex>                           ; AOS coordinates storage
 //   memref<?xf64>                             ; values
 //   struct<(array<2 x i64>, array<3 x i64>)>) ; lvl0, lvl1, 3xsizes
 //
+// Slice on #COO storage of 2-dim matrix yields
+//   ;; Inherited from the original sparse tensors
+//   memref<?xindex>,                          ; positions-0, essentially [0,sz]
+//   memref<?xindex>                           ; AOS coordinates storage
+//   memref<?xf64>                             ; values
+//   struct<(array<2 x i64>, array<3 x i64>,   ; lvl0, lvl1, 3xsizes
+//   ;; Extra slicing-metadata
+//           array<2 x i64>, array<2 x i64>)>) ; dim offset, dim stride.
+//
 //===----------------------------------------------------------------------===//
 
 enum class SparseTensorFieldKind : uint32_t {
   StorageSpec = 0,
-  PtrMemRef = 1,
-  IdxMemRef = 2,
+  PosMemRef = 1,
+  CrdMemRef = 2,
   ValMemRef = 3
 };
 
-static_assert(static_cast<uint32_t>(SparseTensorFieldKind::PtrMemRef) ==
-              static_cast<uint32_t>(StorageSpecifierKind::PtrMemSize));
-static_assert(static_cast<uint32_t>(SparseTensorFieldKind::IdxMemRef) ==
-              static_cast<uint32_t>(StorageSpecifierKind::IdxMemSize));
+static_assert(static_cast<uint32_t>(SparseTensorFieldKind::PosMemRef) ==
+              static_cast<uint32_t>(StorageSpecifierKind::PosMemSize));
+static_assert(static_cast<uint32_t>(SparseTensorFieldKind::CrdMemRef) ==
+              static_cast<uint32_t>(StorageSpecifierKind::CrdMemSize));
 static_assert(static_cast<uint32_t>(SparseTensorFieldKind::ValMemRef) ==
               static_cast<uint32_t>(StorageSpecifierKind::ValMemSize));
 
@@ -98,13 +112,13 @@ using FieldIndex = unsigned;
 // `FieldIndex` for their return type, via the same reasoning for why
 // `Dimension`/`Level` are used both for identifiers and ranks.
 
-/// For each field that will be allocated for the given sparse tensor encoding,
-/// calls the callback with the corresponding field index, field kind, dimension
-/// (for sparse tensor level memrefs) and dimlevelType.
-/// The field index always starts with zero and increments by one between two
-/// callback invocations.
-/// Ideally, all other methods should rely on this function to query a sparse
-/// tensor fields instead of relying on ad-hoc index computation.
+/// For each field that will be allocated for the given sparse tensor
+/// encoding, calls the callback with the corresponding field index,
+/// field kind, level, and level-type (the last two are only for level
+/// memrefs).  The field index always starts with zero and increments
+/// by one between each callback invocation.  Ideally, all other methods
+/// should rely on this function to query a sparse tensor fields instead
+/// of relying on ad-hoc index computation.
 void foreachFieldInSparseTensor(
     SparseTensorEncodingAttr,
     llvm::function_ref<bool(
@@ -124,8 +138,8 @@ void foreachFieldAndTypeInSparseTensor(
 // TODO: See note [NUMFIELDS].
 unsigned getNumFieldsFromEncoding(SparseTensorEncodingAttr enc);
 
-/// Gets the total number of data fields (index arrays, pointer arrays, and a
-/// value array) for the given sparse tensor encoding.
+/// Gets the total number of data fields (coordinate arrays, position
+/// arrays, and a value array) for the given sparse tensor encoding.
 // TODO: See note [NUMFIELDS].
 unsigned getNumDataFieldsFromEncoding(SparseTensorEncodingAttr enc);
 
@@ -135,7 +149,7 @@ inline StorageSpecifierKind toSpecifierKind(SparseTensorFieldKind kind) {
 }
 
 inline SparseTensorFieldKind toFieldKind(StorageSpecifierKind kind) {
-  assert(kind != StorageSpecifierKind::DimSize);
+  assert(kind != StorageSpecifierKind::LvlSize);
   return static_cast<SparseTensorFieldKind>(kind);
 }
 
@@ -177,7 +191,7 @@ class StorageLayout {
                          std::optional<Level> lvl) const {
     FieldIndex fieldIdx = -1u;
     unsigned stride = 1;
-    if (kind == SparseTensorFieldKind::IdxMemRef) {
+    if (kind == SparseTensorFieldKind::CrdMemRef) {
       assert(lvl.has_value());
       const Level cooStart = getCOOStart(enc);
       const Level lvlRank = enc.getLvlRank();
@@ -206,40 +220,22 @@ class StorageLayout {
   SparseTensorEncodingAttr enc;
 };
 
-// FIXME: Functions/methods marked with [CLARIFY_DIM_LVL] require
-// clarification on whether their "dim" argument should actually
-// be `Level` or `Dimension`.  In particular, it's unclear whether
-// `StorageSpecifierKind::DimSize` actually means to refer to dimension-sizes
-// vs level-sizes.  If it's the latter (which seems unlikely), then all the
-// noted functions should use the `Level` type alias.  If it's the former,
-// then the functions which specifically use `DimSize` should be changed
-// to use the `Dimension` type alias; however, the functions which take
-// an unknown `StorageSpecifierKind` must be adjusted to ensure that they
-// correctly interpret the "dim" argument since the interpretation depends
-// on the `StorageSpecifierKind` value.  Since wrengr couldn't figure this
-// out from context, Peiming or Bixia should review these functions and
-// update them as appropriate.
-
 class SparseTensorSpecifier {
 public:
   explicit SparseTensorSpecifier(Value specifier)
       : specifier(cast<TypedValue<StorageSpecifierType>>(specifier)) {}
 
-  // Undef value for dimension sizes, all zero value for memory sizes.
+  // Undef value for level-sizes, all zero values for memory-sizes.
   static Value getInitValue(OpBuilder &builder, Location loc,
                             SparseTensorType stt);
 
   /*implicit*/ operator Value() { return specifier; }
 
-  // FIXME: see note [CLARIFY_DIM_LVL].
   Value getSpecifierField(OpBuilder &builder, Location loc,
-                          StorageSpecifierKind kind,
-                          std::optional<unsigned> dim);
+                          StorageSpecifierKind kind, std::optional<Level> lvl);
 
-  // FIXME: see note [CLARIFY_DIM_LVL].
   void setSpecifierField(OpBuilder &builder, Location loc, Value v,
-                         StorageSpecifierKind kind,
-                         std::optional<unsigned> dim);
+                         StorageSpecifierKind kind, std::optional<Level> lvl);
 
 private:
   TypedValue<StorageSpecifierType> specifier;
@@ -280,21 +276,19 @@ class SparseTensorDescriptorImpl {
 
   Value getSpecifier() const { return fields.back(); }
 
-  // FIXME: see note [CLARIFY_DIM_LVL].
   Value getSpecifierField(OpBuilder &builder, Location loc,
                           StorageSpecifierKind kind,
-                          std::optional<unsigned> dim) const {
+                          std::optional<Level> lvl) const {
     SparseTensorSpecifier md(fields.back());
-    return md.getSpecifierField(builder, loc, kind, dim);
+    return md.getSpecifierField(builder, loc, kind, lvl);
   }
 
-  // FIXME: see note [CLARIFY_DIM_LVL].
-  Value getDimSize(OpBuilder &builder, Location loc, unsigned dim) const {
-    return getSpecifierField(builder, loc, StorageSpecifierKind::DimSize, dim);
+  Value getLvlSize(OpBuilder &builder, Location loc, Level lvl) const {
+    return getSpecifierField(builder, loc, StorageSpecifierKind::LvlSize, lvl);
   }
 
-  Value getPtrMemRef(Level lvl) const {
-    return getMemRefField(SparseTensorFieldKind::PtrMemRef, lvl);
+  Value getPosMemRef(Level lvl) const {
+    return getMemRefField(SparseTensorFieldKind::PosMemRef, lvl);
   }
 
   Value getValMemRef() const {
@@ -311,13 +305,13 @@ class SparseTensorDescriptorImpl {
     return getField(fidx);
   }
 
-  Value getPtrMemSize(OpBuilder &builder, Location loc, Level lvl) const {
-    return getSpecifierField(builder, loc, StorageSpecifierKind::PtrMemSize,
+  Value getPosMemSize(OpBuilder &builder, Location loc, Level lvl) const {
+    return getSpecifierField(builder, loc, StorageSpecifierKind::PosMemSize,
                              lvl);
   }
 
-  Value getIdxMemSize(OpBuilder &builder, Location loc, Level lvl) const {
-    return getSpecifierField(builder, loc, StorageSpecifierKind::IdxMemSize,
+  Value getCrdMemSize(OpBuilder &builder, Location loc, Level lvl) const {
+    return getSpecifierField(builder, loc, StorageSpecifierKind::CrdMemSize,
                              lvl);
   }
 
@@ -341,15 +335,15 @@ class SparseTensorDescriptorImpl {
     return fields.drop_back();
   }
 
-  std::pair<FieldIndex, unsigned> getIdxMemRefIndexAndStride(Level lvl) const {
+  std::pair<FieldIndex, unsigned> getCrdMemRefIndexAndStride(Level lvl) const {
     StorageLayout layout(rType.getEncoding());
-    return layout.getFieldIndexAndStride(SparseTensorFieldKind::IdxMemRef, lvl);
+    return layout.getFieldIndexAndStride(SparseTensorFieldKind::CrdMemRef, lvl);
   }
 
   Value getAOSMemRef() const {
     const Level cooStart = getCOOStart(rType.getEncoding());
     assert(cooStart < rType.getLvlRank());
-    return getMemRefField(SparseTensorFieldKind::IdxMemRef, cooStart);
+    return getMemRefField(SparseTensorFieldKind::CrdMemRef, cooStart);
   }
 
   RankedTensorType getRankedTensorType() const { return rType; }
@@ -366,7 +360,7 @@ class SparseTensorDescriptor : public SparseTensorDescriptorImpl<ValueRange> {
   SparseTensorDescriptor(SparseTensorType stt, ValueRange buffers)
       : SparseTensorDescriptorImpl<ValueRange>(stt, buffers) {}
 
-  Value getIdxMemRefOrView(OpBuilder &builder, Location loc, Level lvl) const;
+  Value getCrdMemRefOrView(OpBuilder &builder, Location loc, Level lvl) const;
 };
 
 /// Uses SmallVectorImpl<Value> & for mutable descriptors.
@@ -409,12 +403,13 @@ class MutSparseTensorDescriptor
     fields[fidx] = v;
   }
 
-  // FIXME: see note [CLARIFY_DIM_LVL].
+  void setSpecifier(Value newSpec) { fields.back() = newSpec; }
+
   void setSpecifierField(OpBuilder &builder, Location loc,
-                         StorageSpecifierKind kind, std::optional<unsigned> dim,
+                         StorageSpecifierKind kind, std::optional<Level> lvl,
                          Value v) {
     SparseTensorSpecifier md(fields.back());
-    md.setSpecifierField(builder, loc, v, kind, dim);
+    md.setSpecifierField(builder, loc, v, kind, lvl);
     fields.back() = md;
   }
 
@@ -423,17 +418,16 @@ class MutSparseTensorDescriptor
                       std::nullopt, v);
   }
 
-  void setIdxMemSize(OpBuilder &builder, Location loc, Level lvl, Value v) {
-    setSpecifierField(builder, loc, StorageSpecifierKind::IdxMemSize, lvl, v);
+  void setCrdMemSize(OpBuilder &builder, Location loc, Level lvl, Value v) {
+    setSpecifierField(builder, loc, StorageSpecifierKind::CrdMemSize, lvl, v);
   }
 
-  void setPtrMemSize(OpBuilder &builder, Location loc, Level lvl, Value v) {
-    setSpecifierField(builder, loc, StorageSpecifierKind::PtrMemSize, lvl, v);
+  void setPosMemSize(OpBuilder &builder, Location loc, Level lvl, Value v) {
+    setSpecifierField(builder, loc, StorageSpecifierKind::PosMemSize, lvl, v);
   }
 
-  // FIXME: see note [CLARIFY_DIM_LVL].
-  void setDimSize(OpBuilder &builder, Location loc, unsigned dim, Value v) {
-    setSpecifierField(builder, loc, StorageSpecifierKind::DimSize, dim, v);
+  void setLvlSize(OpBuilder &builder, Location loc, Level lvl, Value v) {
+    setSpecifierField(builder, loc, StorageSpecifierKind::LvlSize, lvl, v);
   }
 };
 
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
index 1772eef57bdcc..e2c6f77958f68 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
@@ -44,13 +44,6 @@ struct VL {
   bool enableSIMDIndex32;
 };
 
-/// Helper to test for given index value.
-static bool isIntValue(Value val, int64_t idx) {
-  if (auto ival = getConstantIntValue(val))
-    return *ival == idx;
-  return false;
-}
-
 /// Helper test for invariant value (defined outside given block).
 static bool isInvariantValue(Value val, Block *block) {
   return val.getDefiningOp() && val.getDefiningOp()->getBlock() != block;
@@ -67,9 +60,9 @@ static VectorType vectorType(VL vl, Type etp) {
   return VectorType::get(vl.vectorLength, etp, numScalableDims);
 }
 
-/// Constructs vector type from pointer.
-static VectorType vectorType(VL vl, Value ptr) {
-  return vectorType(vl, getMemRefType(ptr).getElementType());
+/// Constructs vector type from a memref value.
+static VectorType vectorType(VL vl, Value mem) {
+  return vectorType(vl, getMemRefType(mem).getElementType());
 }
 
 /// Constructs vector iteration mask.
@@ -116,17 +109,17 @@ static Value genVectorInvariantValue(PatternRewriter &rewriter, VL vl,
 /// that the sparse compiler can only generate indirect loads in
 /// the last index, i.e. back().
 static Value genVectorLoad(PatternRewriter &rewriter, Location loc, VL vl,
-                           Value ptr, ArrayRef<Value> idxs, Value vmask) {
-  VectorType vtp = vectorType(vl, ptr);
+                           Value mem, ArrayRef<Value> idxs, Value vmask) {
+  VectorType vtp = vectorType(vl, mem);
   Value pass = constantZero(rewriter, loc, vtp);
   if (idxs.back().getType().isa<VectorType>()) {
     SmallVector<Value> scalarArgs(idxs.begin(), idxs.end());
     Value indexVec = idxs.back();
     scalarArgs.back() = constantIndex(rewriter, loc, 0);
-    return rewriter.create<vector::GatherOp>(loc, vtp, ptr, scalarArgs,
+    return rewriter.create<vector::GatherOp>(loc, vtp, mem, scalarArgs,
                                              indexVec, vmask, pass);
   }
-  return rewriter.create<vector::MaskedLoadOp>(loc, vtp, ptr, idxs, vmask,
+  return rewriter.create<vector::MaskedLoadOp>(loc, vtp, mem, idxs, vmask,
                                                pass);
 }
 
@@ -134,17 +127,17 @@ static Value genVectorLoad(PatternRewriter &rewriter, Location loc, VL vl,
 /// where 'lo' denotes the current index and 'hi = lo + vl - 1'. Note
 /// that the sparse compiler can only generate indirect stores in
 /// the last index, i.e. back().
-static void genVectorStore(PatternRewriter &rewriter, Location loc, Value ptr,
+static void genVectorStore(PatternRewriter &rewriter, Location loc, Value mem,
                            ArrayRef<Value> idxs, Value vmask, Value rhs) {
   if (idxs.back().getType().isa<VectorType>()) {
     SmallVector<Value> scalarArgs(idxs.begin(), idxs.end());
     Value indexVec = idxs.back();
     scalarArgs.back() = constantIndex(rewriter, loc, 0);
-    rewriter.create<vector::ScatterOp>(loc, ptr, scalarArgs, indexVec, vmask,
+    rewriter.create<vector::ScatterOp>(loc, mem, scalarArgs, indexVec, vmask,
                                        rhs);
     return;
   }
-  rewriter.create<vector::MaskedStoreOp>(loc, ptr, idxs, vmask, rhs);
+  rewriter.create<vector::MaskedStoreOp>(loc, mem, idxs, vmask, rhs);
 }
 
 /// Detects a vectorizable reduction operations and returns the
@@ -233,9 +226,9 @@ static Value genVectorReducInit(PatternRewriter &rewriter, Location loc,
 /// See https://llvm.org/docs/GetElementPtr.html for some background on
 /// the complications described below.
 ///
-/// We need to generate a pointer/index load from the sparse storage scheme.
-/// Narrower data types need to be zero extended before casting the value
-/// into the index type used for looping and indexing.
+/// We need to generate a position/coordinate load from the sparse storage
+/// scheme.  Narrower data types need to be zero extended before casting
+/// the value into the `index` type used for looping and indexing.
 ///
 /// For the scalar case, subscripts simply zero extend narrower indices
 /// into 64-bit values before casting to an index type without a performance
@@ -416,8 +409,8 @@ static bool vectorizeExpr(PatternRewriter &rewriter, scf::ForOp forOp, VL vl,
   }
   // Proper load operations. These are either values involved in the
   // actual computation, such as a[i] = b[i] becomes a[lo:hi] = b[lo:hi],
-  // or index values inside the computation that are now fetched from
-  // the sparse storage index arrays, such as a[i] = i becomes
+  // or coordinate values inside the computation that are now fetched from
+  // the sparse storage coordinates arrays, such as a[i] = i becomes
   // a[lo:hi] = ind[lo:hi], where 'lo' denotes the current index
   // and 'hi = lo + vl - 1'.
   if (auto load = dyn_cast<memref::LoadOp>(def)) {
@@ -619,7 +612,7 @@ struct ForOpRewriter : public OpRewritePattern<scf::ForOp> {
     // Check for single block, unit-stride for-loop that is generated by
     // sparse compiler, which means no data dependence analysis is required,
     // and its loop-body is very restricted in form.
-    if (!op.getRegion().hasOneBlock() || !isIntValue(op.getStep(), 1) ||
+    if (!op.getRegion().hasOneBlock() || !isConstantIntValue(op.getStep(), 1) ||
         !op->hasAttr(LoopEmitter::getLoopEmitterLoopAttrName()))
       return failure();
     // Analyze (!codegen) and rewrite (codegen) loop-body.
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/Sparsification.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/Sparsification.cpp
index 575b50577e413..f119ac3ba7ae5 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/Sparsification.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/Sparsification.cpp
@@ -46,7 +46,7 @@ using namespace mlir::sparse_tensor;
 namespace {
 
 /// Iteration graph sorting.
-enum SortMask {
+enum class SortMask : unsigned {
   // The individual mask bits.
   kIncludeDenseOutput = 0x1, // b001
   kIncludeDenseInput = 0x2,  // b010
@@ -57,17 +57,24 @@ enum SortMask {
   kSparseOnly = 0x0,   // b000
 };
 
-/// SortMask tests on individual bits.
-inline static bool includeDenseInput(unsigned mask) {
-  return mask & SortMask::kIncludeDenseInput;
+inline static bool includesAny(SortMask mask1, SortMask mask2) {
+  return static_cast<unsigned>(mask1) & static_cast<unsigned>(mask2);
 }
 
-inline static bool includeDenseOutput(unsigned mask) {
-  return mask & SortMask::kIncludeDenseOutput;
+inline static bool includesDenseInput(SortMask mask) {
+  return includesAny(mask, SortMask::kIncludeDenseInput);
 }
 
-inline static bool includeUndef(unsigned mask) {
-  return mask & SortMask::kIncludeUndef;
+inline static bool includesDenseOutput(SortMask mask) {
+  return includesAny(mask, SortMask::kIncludeDenseOutput);
+}
+
+inline static bool includesDense(SortMask mask) {
+  return includesAny(mask, SortMask::kIncludeDense);
+}
+
+inline static bool includesUndef(SortMask mask) {
+  return includesAny(mask, SortMask::kIncludeUndef);
 }
 
 /// A helper class that visits an affine expression and tries to find an
@@ -92,7 +99,9 @@ class AffineDimFinder : public AffineExprVisitor<AffineDimFinder> {
   AffineDimExpr getDimExpr() const { return pickedDim.cast<AffineDimExpr>(); }
 
 private:
-  /// The picked AffineDimExpr after visit.
+  /// The picked AffineDimExpr after visit.  This must be stored as
+  /// `AffineExpr` rather than `AffineDimExpr`, because the latter
+  /// doesn't have a default ctor.
   AffineExpr pickedDim;
   /// The iterator type that we want.
   utils::IteratorType pickIterType;
@@ -106,20 +115,25 @@ class AffineDimFinder : public AffineExprVisitor<AffineDimFinder> {
 // Sparse compiler analysis methods.
 //===----------------------------------------------------------------------===//
 
+// TODO: the "idx"-vs-"ldx" naming convention is not self-explanatory,
+// and those letters are too easy to confuse visually.  We should switch
+// to a more self-explanatory naming convention like "curLoop"-vs-"prevLoop"
+// (assuming that's the actual meaning behind the "idx"-vs-"ldx" convention).
+
 /// Determines if affine expression is invariant.
-static bool isInvariantAffine(AffineExpr a, ArrayRef<unsigned> loopStack,
-                              unsigned ldx, bool &atLevel) {
+static bool isInvariantAffine(AffineExpr a, ArrayRef<LoopId> loopStack,
+                              LoopId ldx, bool &isAtLoop) {
   switch (a.getKind()) {
   case AffineExprKind::DimId: {
-    unsigned idx = a.cast<AffineDimExpr>().getPosition();
-    if (idx == ldx) {
-      atLevel = true;
-      // Must be invariant if we are at the level.
+    const LoopId i = a.cast<AffineDimExpr>().getPosition();
+    if (i == ldx) {
+      isAtLoop = true;
+      // Must be invariant if we are at the given loop.
       return true;
     }
     bool isInvariant = false;
-    for (unsigned loop : loopStack) {
-      isInvariant = (loop == idx);
+    for (LoopId l : loopStack) {
+      isInvariant = (l == i);
       if (isInvariant)
         break;
     }
@@ -128,8 +142,8 @@ static bool isInvariantAffine(AffineExpr a, ArrayRef<unsigned> loopStack,
   case AffineExprKind::Add:
   case AffineExprKind::Mul: {
     auto binOp = a.cast<AffineBinaryOpExpr>();
-    return isInvariantAffine(binOp.getLHS(), loopStack, ldx, atLevel) &&
-           isInvariantAffine(binOp.getRHS(), loopStack, ldx, atLevel);
+    return isInvariantAffine(binOp.getLHS(), loopStack, ldx, isAtLoop) &&
+           isInvariantAffine(binOp.getRHS(), loopStack, ldx, isAtLoop);
   }
   default: {
     assert(a.isa<AffineConstantExpr>());
@@ -139,34 +153,42 @@ static bool isInvariantAffine(AffineExpr a, ArrayRef<unsigned> loopStack,
 }
 
 /// Determines if affine expression is invariant.
-static bool isInvariantAffine(CodegenEnv &env, AffineExpr a, unsigned ldx,
-                              bool &atLevel) {
-  return isInvariantAffine(a, env.getLoopCurStack(), ldx, atLevel);
+static bool isInvariantAffine(CodegenEnv &env, AffineExpr a, LoopId ldx,
+                              bool &isAtLoop) {
+  return isInvariantAffine(a, env.getCurrentLoopStack(), ldx, isAtLoop);
 }
 
 /// Helper method to construct a permuted dimension ordering
 /// that adheres to the given topological sort.
+//
+// FIXME: does the above actually mean "dimensions", or should it say
+// "level ordering"?  The same dim/lvl confusion applies to all the code
+// and comments in the definition below.
 static AffineMap permute(CodegenEnv &env, AffineMap m) {
   assert(m.getNumDims() + env.merger().getNumFilterLoops() ==
              env.topSortSize() &&
          "size mismatch");
   // Construct the inverse of `m`; to avoid the asymptotic complexity
   // of calling `m.getPermutedPosition` repeatedly.
+  //
+  // The variable `perm` must use `unsigned` rather than `Dimension`/`Level`,
+  // because that's what `AffineMap::getPermutationMap` requires.
+  // TODO: however, `perm` should be renamed to make clear what exactly
+  // it's storing a permutation of.
   SmallVector<unsigned> perm;
-  unsigned numResults = m.getNumResults();
+  const unsigned numResults = m.getNumResults();
   BitVector worklist(numResults, true);
-  unsigned loopDepth = 1;
+  LoopOrd loopDepth = 1;
 
   // Construct the permutation.
   while (worklist.any() && loopDepth <= env.topSortSize()) {
-    unsigned preSize = perm.size();
-    for (auto dim : worklist.set_bits()) {
-      bool atLevel = false;
+    const unsigned preSize = perm.size();
+    for (unsigned dim : worklist.set_bits()) {
+      bool isAtLoop = false;
       if (m.getResult(dim).isa<AffineConstantExpr>() ||
-          (isInvariantAffine(m.getResult(dim),
-                             env.getTopSortSlice(0, loopDepth),
-                             env.topSortAt(loopDepth - 1), atLevel) &&
-           atLevel)) {
+          (isInvariantAffine(m.getResult(dim), env.getLoopStackUpTo(loopDepth),
+                             env.topSortAt(loopDepth - 1), isAtLoop) &&
+           isAtLoop)) {
         // If the matching affine is constant expression or just become
         // invariant. We can visit the dimension now without breaking the
         // topSort constraint.
@@ -178,8 +200,8 @@ static AffineMap permute(CodegenEnv &env, AffineMap m) {
     for (unsigned i = preSize, e = perm.size(); i < e; i++)
       worklist.reset(perm[i]);
 
-    // Tries to entering the next loop level.
-    loopDepth += 1;
+    // Try entering the next loop in the stack.
+    loopDepth++;
   }
 
   assert(perm.size() == numResults);
@@ -192,26 +214,26 @@ static AffineMap permute(CodegenEnv &env, AffineMap m) {
 /// filterIdx stores the current filter loop idx should be used for the next
 /// compound affine sparse level, and it will be incremented by one when
 /// used.
-static bool findAffine(Merger &merger, unsigned tensor, unsigned dim,
-                       AffineExpr a, DimLevelType dlt, unsigned &filterLdx,
+static bool findAffine(Merger &merger, TensorId tid, Level lvl, AffineExpr a,
+                       DimLevelType dlt, LoopId &filterLdx,
                        bool setLvlFormat = true) {
   switch (a.getKind()) {
   case AffineExprKind::DimId: {
-    unsigned idx = a.cast<AffineDimExpr>().getPosition();
-    if (!isUndefDLT(merger.getDimLevelType(tensor, idx)))
+    const LoopId idx = a.cast<AffineDimExpr>().getPosition();
+    if (!isUndefDLT(merger.getDimLevelType(tid, idx)))
       return false; // used more than once
 
     if (setLvlFormat)
-      merger.setDimAndDimLevelType(tensor, idx, dim, dlt);
+      merger.setLevelAndType(tid, idx, lvl, dlt);
     return true;
   }
   case AffineExprKind::Add:
   case AffineExprKind::Mul:
   case AffineExprKind::Constant: {
     if (!isDenseDLT(dlt) && setLvlFormat) {
-      assert(isUndefDLT(merger.getDimLevelType(tensor, filterLdx)));
+      assert(isUndefDLT(merger.getDimLevelType(tid, filterLdx)));
       // Use a filter loop for sparse affine expression.
-      merger.setDimAndDimLevelType(tensor, filterLdx++, dim, dlt);
+      merger.setLevelAndType(tid, filterLdx++, lvl, dlt);
     }
 
     if (auto binOp = a.dyn_cast<AffineBinaryOpExpr>()) {
@@ -219,9 +241,9 @@ static bool findAffine(Merger &merger, unsigned tensor, unsigned dim,
       // either loop index at d0 or d1.
       // We continue the recursion merely to check whether current affine is
       // admissible or not.
-      return findAffine(merger, tensor, dim, binOp.getLHS(), dlt, filterLdx,
+      return findAffine(merger, tid, lvl, binOp.getLHS(), dlt, filterLdx,
                         false) &&
-             findAffine(merger, tensor, dim, binOp.getRHS(), dlt, filterLdx,
+             findAffine(merger, tid, lvl, binOp.getRHS(), dlt, filterLdx,
                         false);
     }
     // Falls through when it is a constant Affine
@@ -232,40 +254,61 @@ static bool findAffine(Merger &merger, unsigned tensor, unsigned dim,
   }
 }
 
-/// Get the total number of compound affine expressions in affineMap that are
-/// attached to the given tensor. For the following inputs:
+/// Get the total number of compound affine expressions in the
+/// `getMatchingIndexingMap` for the given tensor.  For the following inputs:
 ///
-/// affineMap = (d0, d1, d2) => (d0 + d1, d2)
-/// tensor = ["compressed", "compressed"]
+/// map = (d0, d1, d2) => (d0 + d1, d2)
+/// lvlTypes = ["compressed", "compressed"]
 ///
 /// Returns 1 (because the first level is compressed and its corresponding
-/// affineMap is d0 + d1)
-static unsigned getNumCompoundAffineOnSparseDims(AffineMap affineMap,
-                                                 Value tensor) {
+/// indexing-expression is `d0 + d1`)
+static unsigned getNumCompoundAffineOnSparseLvls(AffineMap map, Value tensor) {
+  // The `tensor` is not guaranted to have `RankedTensorType`, therefore
+  // we can't use `getRankedTensorType`/`getSparseTensorType` here.
+  // However, we don't need to handle `StorageSpecifierType`, so we
+  // can use `SparseTensorType` once we guard against non-tensors.
+  const auto rtp = tensor.getType().dyn_cast<RankedTensorType>();
+  if (!rtp)
+    return 0;
+  const SparseTensorType stt(rtp);
+
+  // FIXME: There's some dim/lvl confusion here.  The previous version of
+  // the code asserted that there are `lvlRank`-many expressions, but then
+  // the `exprs[d]` expression assumes there are in fact `dimRank`-many
+  // expressions.  Even though `ArrayRef::operator[]` will check for OOB,
+  // the mismatch between the assertion and the usage belies that this code
+  // cannot support non-permutations.
+  //
+  // Elsewhere in this file the maps returned by
+  // `linalg::GenericOp::getMatchingIndexingMap` are inconsistent about
+  // whether they're expected to have `lvlRank`-many or `dimRank`-many
+  // expressions (cf., `genSubscript` vs `findSparseAnnotations`);
+  // so those are no help in determining which is actually intended.
+  //
+  // For now we work around this problem by asserting the two ranks agree.
+  const Dimension dimRank = stt.getDimRank();
+  const Level lvlRank = stt.getLvlRank();
+  assert(dimRank == lvlRank && "Non-permutations not currently supported");
+  const auto exprs = map.getResults();
+  assert(static_cast<Dimension>(exprs.size()) == dimRank &&
+         "AffineMap does not have dimension-rank many results");
+  (void)dimRank;
   unsigned num = 0;
-  const auto enc = getSparseTensorEncoding(tensor.getType());
-  if (enc) {
-    const ArrayRef<AffineExpr> exps = affineMap.getResults();
-    const Level lvlRank = enc.getLvlRank();
-    assert(static_cast<Level>(exps.size()) == lvlRank);
-    for (Level l = 0; l < lvlRank; l++) {
-      // FIXME: `toOrigDim` is deprecated.
-      const Dimension d = toOrigDim(enc, l);
-      // FIXME: there's some dim/lvl confusion here; since `d` isn't
-      // guaranteed to be in bounds (for non-permutations).
-      if (!exps[d].isa<AffineDimExpr>() && !enc.isDenseLvl(l))
-        num++;
-    }
+  for (Level l = 0; l < lvlRank; l++) {
+    // FIXME: `toOrigDim` is deprecated.
+    const Dimension d = toOrigDim(stt.getEncoding(), l);
+    if (!exprs[d].isa<AffineDimExpr>() && !stt.isDenseLvl(l))
+      num++;
   }
   return num;
 }
 
-/// Get the total number of compound affine expressions attached on a sparse
-/// level in the given GenericOp.
-static unsigned getNumCompoundAffineOnSparseDims(linalg::GenericOp op) {
+/// Get the total number of sparse levels with compound affine
+/// expressions, summed over all operands of the `GenericOp`.
+static unsigned getNumCompoundAffineOnSparseLvls(linalg::GenericOp op) {
   unsigned num = 0;
   for (OpOperand &t : op->getOpOperands())
-    num += getNumCompoundAffineOnSparseDims(op.getMatchingIndexingMap(&t),
+    num += getNumCompoundAffineOnSparseLvls(op.getMatchingIndexingMap(&t),
                                             t.get());
   return num;
 }
@@ -274,7 +317,7 @@ static bool hasCompoundAffineOnSparseOut(linalg::GenericOp op) {
   OpOperand *out = op.getDpsInitOperand(0);
   if (getSparseTensorType(out->get()).isAllDense())
     return false;
-  return getNumCompoundAffineOnSparseDims(op.getMatchingIndexingMap(out),
+  return getNumCompoundAffineOnSparseLvls(op.getMatchingIndexingMap(out),
                                           out->get());
 }
 
@@ -285,7 +328,8 @@ static bool hasCompoundAffineOnSparseOut(linalg::GenericOp op) {
 /// no annotations are found or inadmissible constructs occur.
 static bool findSparseAnnotations(CodegenEnv &env) {
   bool annotated = false;
-  unsigned filterLdx = env.merger().getFilterLoopStartingIdx();
+  // `filterLdx` may be mutated by `findAffine`.
+  LoopId filterLdx = env.merger().getStartingFilterLoopId();
   for (OpOperand &t : env.op()->getOpOperands()) {
     const auto map = env.op().getMatchingIndexingMap(&t);
     const auto enc = getSparseTensorEncoding(t.get().getType());
@@ -295,10 +339,12 @@ static bool findSparseAnnotations(CodegenEnv &env) {
     assert(!enc || lvlRank == enc.getLvlRank());
     assert(static_cast<Level>(env.op().getRank(&t)) == lvlRank);
     for (Level l = 0; l < lvlRank; l++) {
-      const unsigned tensor = t.getOperandNumber();
+      const TensorId tid = t.getOperandNumber();
       // FIXME: `toOrigDim` is deprecated.
+      // FIXME: above we asserted that there are `lvlRank` many results,
+      // but this is assuming there are in fact `dimRank` many results instead.
       const AffineExpr a = map.getResult(toOrigDim(enc, l));
-      if (!findAffine(env.merger(), tensor, l, a, enc.getLvlType(l), filterLdx))
+      if (!findAffine(env.merger(), tid, l, a, enc.getLvlType(l), filterLdx))
         return false; // inadmissible affine expression
     }
   }
@@ -310,14 +356,18 @@ static bool findSparseAnnotations(CodegenEnv &env) {
 /// as we use adj matrix for the graph.
 /// The sorted result will put the first Reduction iterator to the
 /// latest possible index.
-static bool topSortOptimal(CodegenEnv &env, unsigned n,
+/// FIXME(wrengr): correct the above "index"
+///
+/// The `inDegree` is indexed by `LoopId`, and the `adjM` is indexed by
+/// `(LoopId,LoopId)`.
+static bool topSortOptimal(CodegenEnv &env, LoopId n,
                            ArrayRef<utils::IteratorType> iteratorTypes,
                            std::vector<unsigned> &inDegree,
                            std::vector<std::vector<bool>> &adjM) {
-  std::vector<unsigned> redIt;    // reduce iterator with 0 degree
-  std::vector<unsigned> parIt;    // parallel iterator with 0 degree
-  std::vector<unsigned> filterIt; // filter loop with 0 degree
-  for (unsigned i = 0; i < n; i++) {
+  std::vector<LoopId> redIt;    // reduce iterator with 0 degree
+  std::vector<LoopId> parIt;    // parallel iterator with 0 degree
+  std::vector<LoopId> filterIt; // filter loop with 0 degree
+  for (LoopId i = 0; i < n; i++) {
     if (inDegree[i] == 0) {
       if (env.merger().isFilterLoop(i))
         filterIt.push_back(i);
@@ -353,7 +403,7 @@ static bool topSortOptimal(CodegenEnv &env, unsigned n,
     env.topSortPushBack(src);
     it.pop_back();
     // Update in-degree, and push 0-degree node into worklist.
-    for (unsigned dst = 0; dst < n; dst++) {
+    for (LoopId dst = 0; dst < n; dst++) {
       if (adjM[src][dst] && --inDegree[dst] == 0) {
         if (env.merger().isFilterLoop(dst))
           filterIt.push_back(dst);
@@ -374,14 +424,17 @@ static bool topSortOptimal(CodegenEnv &env, unsigned n,
 /// b = (i0 + i1) < fidx => i0 < fidx, i1 < fidx.
 /// The affine expression `b` is empty iff `tidx` have a value, leading to
 /// tidx < a = (i0 + i1) => tidx < i0, tidx < i1.
+///
+/// The `inDegree` is indexed by `LoopId`, and the `adjM` is indexed by
+/// `(LoopId,LoopId)`.
 static void addAffineOrderings(std::vector<std::vector<bool>> &adjM,
                                std::vector<unsigned> &inDegree, AffineExpr a,
-                               AffineExpr b, std::optional<unsigned> fidx,
-                               std::optional<unsigned> tidx) {
+                               AffineExpr b, std::optional<LoopId> fidx,
+                               std::optional<LoopId> tidx) {
   if (!a && !b) {
     // Recursion leaf.
     assert(fidx && tidx);
-    unsigned f = *fidx, t = *tidx;
+    const LoopId f = *fidx, t = *tidx;
     if (!adjM[f][t]) {
       adjM[f][t] = true;
       inDegree[t]++;
@@ -389,10 +442,10 @@ static void addAffineOrderings(std::vector<std::vector<bool>> &adjM,
     return;
   }
   // Picks an affine expression and expand (recurse into) it.
-  auto toExpand = a ? a : b;
+  const auto toExpand = a ? a : b;
   switch (toExpand.getKind()) {
   case AffineExprKind::DimId: {
-    auto idx = toExpand.cast<AffineDimExpr>().getPosition();
+    std::optional<LoopId> idx = toExpand.cast<AffineDimExpr>().getPosition();
     if (toExpand == a)
       addAffineOrderings(adjM, inDegree, AffineExpr(), b, idx, tidx);
     else // toExpand == b
@@ -417,9 +470,9 @@ static void addAffineOrderings(std::vector<std::vector<bool>> &adjM,
 }
 
 static void tryLoosenAffineDenseConstraints(linalg::GenericOp op,
-                                            std::optional<unsigned> &fldx,
+                                            std::optional<LoopId> &fldx,
                                             AffineExpr &fa,
-                                            std::optional<unsigned> &tldx,
+                                            std::optional<LoopId> &tldx,
                                             AffineExpr &ta) {
   // We use a heuristic here to only pick one dim expression from each
   // compound affine expression to establish the order between two dense
@@ -450,17 +503,17 @@ static void tryLoosenAffineDenseConstraints(linalg::GenericOp op,
   }
 }
 
-/// Computes a topologically sorted iteration graph for the linalg
-/// operation. Ensures all tensors are visited in natural index order. This
-/// is essential for sparse storage formats since these only support access
-/// along fixed dimensions. Even for dense storage formats, however, the
-/// natural index order yields innermost unit-stride access with better
-/// spatial locality.
-static bool computeIterationGraph(CodegenEnv &env, unsigned mask,
+/// Computes a topologically sorted iteration graph for the linalg operation.
+/// Ensures all tensors are visited in natural coordinate order.  This is
+/// essential for sparse storage formats since these only support access
+/// along fixed levels.  Even for dense storage formats, however, the natural
+/// coordinate order yields innermost unit-stride access with better spatial
+/// locality.
+static bool computeIterationGraph(CodegenEnv &env, SortMask mask,
                                   OpOperand *skip = nullptr) {
   // Set up an n x n from/to adjacency matrix of the iteration graph
   // for the implicit loop indices i_0 .. i_n-1.
-  const unsigned n = env.merger().getNumLoops();
+  const LoopId n = env.merger().getNumLoops();
   std::vector<std::vector<bool>> adjM(n, std::vector<bool>(n, false));
   std::vector<unsigned> inDegree(n, 0); // in-degree of each node.
   const auto iteratorTypes = env.op().getIteratorTypesArray();
@@ -469,32 +522,30 @@ static bool computeIterationGraph(CodegenEnv &env, unsigned mask,
     // Get map and encoding.
     const auto map = env.op().getMatchingIndexingMap(&t);
     const auto enc = getSparseTensorEncoding(t.get().getType());
-    assert(map.getNumDims() + getNumCompoundAffineOnSparseDims(env.op()) == n);
-
-    bool isDenseInput = !enc && env.op().isDpsInput(&t);
-    bool isDenseOutput = !enc && !isDenseInput;
+    assert(map.getNumDims() + getNumCompoundAffineOnSparseLvls(env.op()) == n);
 
     // Skips dense inputs/outputs when not requested.
-    if ((isDenseInput && !includeDenseInput(mask)) ||
-        (isDenseOutput && !includeDenseOutput(mask)))
+    const bool isDenseInput = !enc && env.op().isDpsInput(&t);
+    const bool isDenseOutput = !enc && !isDenseInput;
+    if ((isDenseInput && !includesDenseInput(mask)) ||
+        (isDenseOutput && !includesDenseOutput(mask)))
       continue;
 
     // Push unrelated loops into sparse iteration space, so these
     // will be skipped more often.
     // TODO: Do we really need this?
-    if (includeUndef(mask)) {
-      unsigned tensor = t.getOperandNumber();
-      for (unsigned i = 0; i < n; i++) {
-        if (isCompressedDLT(env.dlt(tensor, i)) ||
-            isSingletonDLT(env.dlt(tensor, i))) {
-          for (unsigned j = 0; j < n; j++)
+    if (includesUndef(mask)) {
+      const TensorId tensor = t.getOperandNumber();
+      for (LoopId i = 0; i < n; i++) {
+        const auto dltI = env.dlt(tensor, i);
+        if (isCompressedDLT(dltI) || isSingletonDLT(dltI)) {
+          for (LoopId j = 0; j < n; j++)
             if (isUndefDLT(env.dlt(tensor, j))) {
               adjM[i][j] = true;
               inDegree[j]++;
             }
         } else {
-          assert(isDenseDLT(env.dlt(tensor, i)) ||
-                 isUndefDLT(env.dlt(tensor, i)));
+          assert(isDenseDLT(dltI) || isUndefDLT(dltI));
         }
       }
     }
@@ -507,9 +558,11 @@ static bool computeIterationGraph(CodegenEnv &env, unsigned mask,
     assert(!enc || lvlRank == enc.getLvlRank());
     for (Level l = 0; l < lvlRank; l++) {
       // FIXME: `toOrigDim` is deprecated.
+      // FIXME: above we asserted that there are `lvlRank` many results,
+      // but this is assuming there are in fact `dimRank` many results instead.
       AffineExpr ta = map.getResult(toOrigDim(enc, l));
-      std::optional<unsigned> tldx =
-          env.merger().getLoopIdx(t.getOperandNumber(), l);
+      std::optional<LoopId> tldx =
+          env.merger().getLoopId(t.getOperandNumber(), l);
 
       // Filter loops should be constructed after all the dependent loops,
       // i.e., d0 + d1 < filter_loop(d0 + d1)
@@ -531,16 +584,18 @@ static bool computeIterationGraph(CodegenEnv &env, unsigned mask,
 
       if (l > 0) {
         // FIXME: `toOrigDim` is deprecated.
+        // FIXME: above we asserted that there are `lvlRank` many results,
+        // but this is assuming there are in fact `dimRank` many results.
         AffineExpr fa = map.getResult(toOrigDim(enc, l - 1));
-        std::optional<unsigned> fldx =
-            env.merger().getLoopIdx(t.getOperandNumber(), l - 1);
+        std::optional<LoopId> fldx =
+            env.merger().getLoopId(t.getOperandNumber(), l - 1);
 
         // Applying order constraints on every pair of dimExpr between two
         // compound affine expressions can sometime too strict:
         // E.g, for [dense, dense] -> (d0 + d1, d2 + d3).
         // It is totally fine to have loop sequence d0->d2->d1->d3 instead of
         // requiring d0 < d2, d1 < d2, d0 < d3, d1 < d3.
-        if (!(mask & SortMask::kIncludeDense))
+        if (!includesDense(mask))
           tryLoosenAffineDenseConstraints(env.op(), fldx, fa, tldx, ta);
 
         // (d0 + d1) < (d2 + d3), or
@@ -605,38 +660,46 @@ static void genBuffers(CodegenEnv &env, OpBuilder &builder) {
 }
 
 /// Generates index for load/store on sparse tensor.
+// FIXME: It's not entirely clear what "index" means here (i.e., is it
+// a "coordinate", or "Ldx", or what).  So the function should be renamed
+// and/or the documentation expanded in order to clarify.
 static Value genIndex(CodegenEnv &env, OpOperand *t) {
   auto map = env.op().getMatchingIndexingMap(t);
   const auto stt = getSparseTensorType(t->get());
   const Level lvlRank = stt.getLvlRank();
   assert(static_cast<Level>(map.getNumResults()) == lvlRank);
   // FIXME: `toOrigDim` is deprecated.
+  // FIXME: above we asserted that there are `lvlRank` many results,
+  // but this is assuming there are in fact `dimRank` many results instead.
   AffineExpr a = map.getResult(toOrigDim(stt.getEncoding(), lvlRank - 1));
   assert(a.getKind() == AffineExprKind::DimId);
-  unsigned idx = a.cast<AffineDimExpr>().getPosition();
-  return env.getLoopIdxValue(idx);
+  const LoopId idx = a.cast<AffineDimExpr>().getPosition();
+  return env.getLoopVar(idx);
 }
 
 /// Generates subscript for load/store on a dense or sparse tensor.
 static Value genSubscript(CodegenEnv &env, OpBuilder &builder, OpOperand *t,
                           SmallVectorImpl<Value> &args) {
-  linalg::GenericOp op = env.op();
-  unsigned tensor = t->getOperandNumber();
-  auto map = op.getMatchingIndexingMap(t);
+  const Location loc = env.op().getLoc();
+  const TensorId tid = t->getOperandNumber();
+  const auto map = env.op().getMatchingIndexingMap(t);
   const auto stt = getSparseTensorType(t->get());
   if (stt.hasEncoding()) {
-    Value pidx = env.emitter().getPidxs()[tensor].back();
-    assert(pidx);
-    args.push_back(pidx); // position index
+    // For sparse tensors we only push the last-level's position onto `args`.
+    const auto pos = env.emitter().getPosits()[tid].back();
+    assert(pos);
+    args.push_back(pos);
   } else {
+    // For dense tensors we push all level's coordinates onto `args`.
     const Level lvlRank = stt.getLvlRank();
     assert(static_cast<Level>(map.getNumResults()) == lvlRank);
     for (Level l = 0; l < lvlRank; l++) {
-      AffineExpr a = map.getResult(l);
-      args.push_back(env.emitter().genAffine(builder, a, op.getLoc()));
+      const auto lvlExpr = map.getResult(l);
+      const auto lvlCrd = env.emitter().genAffine(builder, loc, lvlExpr);
+      args.push_back(lvlCrd);
     }
   }
-  return env.emitter().getValBuffer()[tensor];
+  return env.emitter().getValBuffer()[tid];
 }
 
 /// Generates insertion code to implement dynamic tensor load.
@@ -644,7 +707,7 @@ static Value genInsertionLoad(CodegenEnv &env, OpBuilder &builder,
                               OpOperand *t) {
   linalg::GenericOp op = env.op();
   Location loc = op.getLoc();
-  // Direct lexicographic index order, tensor loads as zero.
+  // Direct lexicographic coordinate order, tensor loads as zero.
   if (!env.isExpand()) {
     Type tp = getElementTypeOrSelf(t->get().getType());
     return constantZero(builder, loc, tp);
@@ -660,7 +723,7 @@ static Value genInsertionLoadReduce(CodegenEnv &env, OpBuilder &builder,
   linalg::GenericOp op = env.op();
   Location loc = op.getLoc();
   Value identity = env.getCustomRedId();
-  // Direct lexicographic index order, tensor loads as identity.
+  // Direct lexicographic coordinate order, tensor loads as identity.
   if (!env.isExpand())
     return identity;
   // Load from expanded access pattern if filled, identity otherwise.
@@ -677,18 +740,23 @@ static void genInsertionStore(CodegenEnv &env, OpBuilder &builder, OpOperand *t,
                               Value rhs) {
   linalg::GenericOp op = env.op();
   Location loc = op.getLoc();
-  // Direct insertion in lexicographic index order.
+  // Direct insertion in lexicographic coordinate order.
   if (!env.isExpand()) {
-    unsigned rank = op.getRank(t);
-    SmallVector<Value> indices;
-    for (unsigned i = 0; i < rank; i++) {
-      assert(env.emitter().getLoopIV(i));
-      indices.push_back(env.emitter().getLoopIV(i));
+    const LoopOrd numLoops = op.getRank(t);
+    // TODO: rewrite this to use `env.emitter().getLoopIVs(ivs)`
+    // instead.  We just need to either assert that `numLoops ==
+    // env.emitter().getCurrentDepth()`, or else update the `getLoopIVs`
+    // method to take an optional parameter to restrict to a smaller depth.
+    SmallVector<Value> ivs;
+    ivs.reserve(numLoops);
+    for (LoopOrd n = 0; n < numLoops; n++) {
+      const auto iv = env.emitter().getLoopIV(n);
+      assert(iv);
+      ivs.push_back(iv);
     }
     Value chain = env.getInsertionChain();
     if (!env.getValidLexInsert()) {
-      env.updateInsertionChain(
-          builder.create<InsertOp>(loc, rhs, chain, indices));
+      env.updateInsertionChain(builder.create<InsertOp>(loc, rhs, chain, ivs));
     } else {
       // Generates runtime check for a valid lex during reduction,
       // to avoid inserting the identity value for empty reductions.
@@ -702,7 +770,7 @@ static void genInsertionStore(CodegenEnv &env, OpBuilder &builder, OpOperand *t,
           /*else=*/true);
       // True branch.
       builder.setInsertionPointToStart(ifValidLexInsert.thenBlock());
-      Value res = builder.create<InsertOp>(loc, rhs, chain, indices);
+      Value res = builder.create<InsertOp>(loc, rhs, chain, ivs);
       builder.create<scf::YieldOp>(loc, res);
       // False branch.
       builder.setInsertionPointToStart(ifValidLexInsert.elseBlock());
@@ -749,7 +817,7 @@ static void genInsertionStore(CodegenEnv &env, OpBuilder &builder, OpOperand *t,
 }
 
 /// Generates a load on a dense or sparse tensor.
-static Value genTensorLoad(CodegenEnv &env, OpBuilder &builder, unsigned exp) {
+static Value genTensorLoad(CodegenEnv &env, OpBuilder &builder, ExprId exp) {
   // Test if the load was hoisted to a higher loop nest.
   Value val = env.exp(exp).val;
   if (val)
@@ -770,7 +838,7 @@ static Value genTensorLoad(CodegenEnv &env, OpBuilder &builder, unsigned exp) {
 }
 
 /// Generates a store on a dense or sparse tensor.
-static void genTensorStore(CodegenEnv &env, OpBuilder &builder, unsigned exp,
+static void genTensorStore(CodegenEnv &env, OpBuilder &builder, ExprId exp,
                            Value rhs) {
   linalg::GenericOp op = env.op();
   Location loc = op.getLoc();
@@ -818,25 +886,20 @@ static void genTensorStore(CodegenEnv &env, OpBuilder &builder, unsigned exp,
 }
 
 /// Generates an invariant value.
-inline static Value genInvariantValue(CodegenEnv &env, unsigned exp) {
+inline static Value genInvariantValue(CodegenEnv &env, ExprId exp) {
   return env.exp(exp).val;
 }
 
-/// Generates an index value.
-inline static Value genIndexValue(CodegenEnv &env, unsigned idx) {
-  return env.getLoopIdxValue(idx);
-}
-
 /// Semi-ring branches are simply inlined by the sparse compiler. Prior
 /// analysis has verified that all computations are "local" to the inlined
 /// branch or otherwise invariantly defined outside the loop nest, with the
 /// exception of index computations, which need to be relinked to actual
 /// inlined cloned code.
 static Value relinkBranch(CodegenEnv &env, RewriterBase &rewriter, Block *block,
-                          Value e, unsigned ldx) {
+                          Value e, LoopId ldx) {
   if (Operation *def = e.getDefiningOp()) {
     if (auto indexOp = dyn_cast<linalg::IndexOp>(def))
-      return genIndexValue(env, indexOp.getDim());
+      return env.getLoopVar(indexOp.getDim());
     if (def->getBlock() == block) {
       for (unsigned i = 0, n = def->getNumOperands(); i < n; i++) {
         rewriter.updateRootInPlace(def, [&]() {
@@ -850,52 +913,52 @@ static Value relinkBranch(CodegenEnv &env, RewriterBase &rewriter, Block *block,
 }
 
 /// Recursively generates tensor expression.
-static Value genExp(CodegenEnv &env, RewriterBase &rewriter, unsigned exp,
-                    unsigned ldx) {
+static Value genExp(CodegenEnv &env, RewriterBase &rewriter, ExprId e,
+                    LoopId ldx) {
   linalg::GenericOp op = env.op();
   Location loc = op.getLoc();
 
-  if (exp == -1u)
+  if (e == kInvalidId)
     return Value();
-  if (env.exp(exp).kind == Kind::kTensor)
-    return genTensorLoad(env, rewriter, exp);
-  if (env.exp(exp).kind == Kind::kInvariant)
-    return genInvariantValue(env, exp);
-  if (env.exp(exp).kind == Kind::kIndex)
-    return genIndexValue(env, env.exp(exp).index);
-
-  if (env.exp(exp).kind == Kind::kReduce)
-    env.startCustomReduc(exp); // enter custom
-
-  Value v0 = genExp(env, rewriter, env.exp(exp).children.e0, ldx);
-  Value v1 = genExp(env, rewriter, env.exp(exp).children.e1, ldx);
-  Value ee = env.merger().buildExp(rewriter, loc, exp, v0, v1);
-  if (ee && (env.exp(exp).kind == Kind::kUnary ||
-             env.exp(exp).kind == Kind::kBinary ||
-             env.exp(exp).kind == Kind::kBinaryBranch ||
-             env.exp(exp).kind == Kind::kReduce ||
-             env.exp(exp).kind == Kind::kSelect))
+  const TensorExp &exp = env.exp(e);
+  const auto kind = exp.kind;
+  if (kind == Kind::kTensor)
+    return genTensorLoad(env, rewriter, e);
+  if (kind == Kind::kInvariant)
+    return genInvariantValue(env, e);
+  if (kind == Kind::kLoopVar)
+    return env.getLoopVar(exp.loop);
+
+  if (kind == Kind::kReduce)
+    env.startCustomReduc(e); // enter custom
+
+  Value v0 = genExp(env, rewriter, exp.children.e0, ldx);
+  Value v1 = genExp(env, rewriter, exp.children.e1, ldx);
+  Value ee = env.merger().buildExp(rewriter, loc, e, v0, v1);
+  if (ee && (kind == Kind::kUnary || kind == Kind::kBinary ||
+             kind == Kind::kBinaryBranch || kind == Kind::kReduce ||
+             kind == Kind::kSelect))
     ee = relinkBranch(env, rewriter, ee.getParentBlock(), ee, ldx);
 
-  if (env.exp(exp).kind == Kind::kReduce)
+  if (kind == Kind::kReduce)
     env.endCustomReduc(); // exit custom
 
-  if (env.exp(exp).kind == kSelect) {
-    assert(!env.exp(exp).val);
-    env.exp(exp).val = v0; // Preserve value for later use.
+  if (kind == kSelect) {
+    assert(!exp.val);
+    env.exp(e).val = v0; // Preserve value for later use.
   }
 
   return ee;
 }
 
 /// Hoists loop invariant tensor loads for which indices have been exhausted.
-static void genInvariants(CodegenEnv &env, OpBuilder &builder, unsigned exp,
-                          unsigned ldx, bool atStart) {
-  if (exp == -1u)
+static void genInvariants(CodegenEnv &env, OpBuilder &builder, ExprId exp,
+                          LoopId ldx, bool atStart) {
+  if (exp == kInvalidId)
     return;
   if (env.exp(exp).kind == Kind::kTensor) {
     // Inspect tensor indices.
-    bool atLevel = ldx == -1u;
+    bool isAtLoop = ldx == kInvalidId;
     linalg::GenericOp op = env.op();
     OpOperand &t = op->getOpOperand(env.exp(exp).tensor);
     auto map = op.getMatchingIndexingMap(&t);
@@ -904,20 +967,21 @@ static void genInvariants(CodegenEnv &env, OpBuilder &builder, unsigned exp,
     assert(static_cast<Level>(map.getNumResults()) == lvlRank);
     for (Level l = 0; l < lvlRank; l++) {
       // FIXME: `toOrigDim` is deprecated.
+      // FIXME: above we asserted that there are `lvlRank` many results,
+      // but this is assuming there are in fact `dimRank` many results instead.
       AffineExpr a = map.getResult(toOrigDim(stt.getEncoding(), l));
-      std::optional<unsigned> sldx =
-          env.merger().getLoopIdx(t.getOperandNumber(), l);
+      const auto sldx = env.merger().getLoopId(t.getOperandNumber(), l);
       if (sldx && env.merger().isFilterLoop(*sldx)) {
-        if (!env.getLoopIdxValue(*sldx))
+        if (!env.getLoopVar(*sldx))
           // The filter loops has not been constructed.
           return;
         if (*sldx == ldx)
-          atLevel = true;
-      } else if (!isInvariantAffine(env, a, ldx, atLevel))
+          isAtLoop = true;
+      } else if (!isInvariantAffine(env, a, ldx, isAtLoop))
         return; // still in play
     }
-    // All exhausted at this level (atLevel denotes exactly at this level).
-    if (!atLevel)
+    // All exhausted at this level (isAtLoop denotes exactly at this LoopId).
+    if (!isAtLoop)
       return;
     OpOperand *lhs = op.getDpsInitOperand(0);
     if (lhs == &t) {
@@ -937,14 +1001,14 @@ static void genInvariants(CodegenEnv &env, OpBuilder &builder, unsigned exp,
       env.exp(exp).val = atStart ? genTensorLoad(env, builder, exp) : Value();
     }
   } else if (env.exp(exp).kind != Kind::kInvariant &&
-             env.exp(exp).kind != Kind::kIndex) {
+             env.exp(exp).kind != Kind::kLoopVar) {
     // Traverse into the binary operations. Note that we only hoist
     // tensor loads, since subsequent MLIR/LLVM passes know how to
     // deal with all other kinds of derived loop invariants.
     if (env.exp(exp).kind == Kind::kReduce)
       env.startCustomReduc(exp); // enter custom
-    unsigned e0 = env.exp(exp).children.e0;
-    unsigned e1 = env.exp(exp).children.e1;
+    const ExprId e0 = env.exp(exp).children.e0;
+    const ExprId e1 = env.exp(exp).children.e1;
     genInvariants(env, builder, e0, ldx, atStart);
     genInvariants(env, builder, e1, ldx, atStart);
     if (env.exp(exp).kind == Kind::kReduce)
@@ -953,7 +1017,7 @@ static void genInvariants(CodegenEnv &env, OpBuilder &builder, unsigned exp,
 }
 
 /// Generates an expanded access pattern in innermost dimension.
-static void genExpand(CodegenEnv &env, OpBuilder &builder, unsigned at,
+static void genExpand(CodegenEnv &env, OpBuilder &builder, LoopOrd at,
                       bool atStart) {
   linalg::GenericOp op = env.op();
   OpOperand *lhs = op.getDpsInitOperand(0);
@@ -980,7 +1044,7 @@ static void genExpand(CodegenEnv &env, OpBuilder &builder, unsigned at,
                     r.getResult(3));
   } else {
     SmallVector<Value> indices;
-    for (unsigned i = 0; i < at; i++)
+    for (LoopOrd i = 0; i < at; i++)
       indices.push_back(env.emitter().getLoopIV(i));
     Value values = env.getExpandValues();
     Value filled = env.getExpandFilled();
@@ -1022,34 +1086,35 @@ static bool isParallelFor(CodegenEnv &env, bool isOuter, bool isSparse) {
 
 /// Generates a for-loop on a single index.
 static Operation *genFor(CodegenEnv &env, OpBuilder &builder, bool isOuter,
-                         bool isInner, unsigned idx, ArrayRef<size_t> tids,
-                         ArrayRef<size_t> dims) {
+                         bool isInner, LoopId ldx, ArrayRef<TensorId> tids,
+                         ArrayRef<Level> lvls) {
   linalg::GenericOp op = env.op();
   Location loc = op.getLoc();
   auto iteratorTypes = op.getIteratorTypesArray();
-  bool isSparse = llvm::any_of(tids, [idx, &env](size_t tid) {
-    return isCompressedDLT(env.dlt(tid, idx)) ||
-           isSingletonDLT(env.dlt(tid, idx));
+  bool isSparse = llvm::any_of(tids, [ldx, &env](TensorId tid) {
+    const auto dlt = env.dlt(tid, ldx);
+    return isCompressedDLT(dlt) || isSingletonDLT(dlt);
   });
 
   bool isParallel = isParallelFor(env, isOuter, isSparse);
 
   Operation *loop = *env.genLoopBoundary([&](MutableArrayRef<Value> reduc) {
-    if (env.merger().isFilterLoop(idx)) {
-      size_t tid = tids.front(), dim = dims.front();
-      // tids/dims must only have one value because filter loops only
+    if (env.merger().isFilterLoop(ldx)) {
+      const TensorId tid = tids.front();
+      const Level lvl = lvls.front();
+      // tids/lvls must only have one value because filter loops only
       // corresponding to the one and only sparse tensor level.
-      assert(isSparse && tids.size() == 1 && dims.size() == 1);
+      assert(isSparse && tids.size() == 1 && lvls.size() == 1);
       OpOperand *t = &op->getOpOperand(tid);
       auto enc = getSparseTensorEncoding(t->get().getType());
       // Retrieves the affine expression for the filter loop.
       // FIXME: `toOrigDim` is deprecated.
       AffineExpr a =
-          op.getMatchingIndexingMap(t).getResult(toOrigDim(enc, dim));
-      return env.emitter().enterFilterLoopOverTensorAtDim(builder, loc, tid,
-                                                          dim, a, reduc);
+          op.getMatchingIndexingMap(t).getResult(toOrigDim(enc, lvl));
+      return env.emitter().enterFilterLoopOverTensorAtLvl(builder, loc, tid,
+                                                          lvl, a, reduc);
     }
-    return env.emitter().enterLoopOverTensorAtDim(builder, loc, tids, dims,
+    return env.emitter().enterLoopOverTensorAtLvl(builder, loc, tids, lvls,
                                                   reduc, isParallel);
   });
   assert(loop);
@@ -1057,14 +1122,14 @@ static Operation *genFor(CodegenEnv &env, OpBuilder &builder, bool isOuter,
 }
 
 /// Emit a while-loop for co-iteration over multiple indices.
-static Operation *genWhile(CodegenEnv &env, OpBuilder &builder, unsigned idx,
-                           bool needsUniv, ArrayRef<size_t> tids,
-                           ArrayRef<size_t> dims) {
+static Operation *genWhile(CodegenEnv &env, OpBuilder &builder, LoopId idx,
+                           bool needsUniv, ArrayRef<TensorId> tids,
+                           ArrayRef<Level> lvls) {
   Operation *loop = *env.genLoopBoundary([&](MutableArrayRef<Value> reduc) {
     // Construct the while-loop with a parameter for each
     // index.
-    return env.emitter().enterCoIterationOverTensorsAtDims(
-        builder, env.op().getLoc(), tids, dims, needsUniv, reduc);
+    return env.emitter().enterCoIterationOverTensorsAtLvls(
+        builder, env.op().getLoc(), tids, lvls, needsUniv, reduc);
   });
   assert(loop);
   return loop;
@@ -1072,21 +1137,21 @@ static Operation *genWhile(CodegenEnv &env, OpBuilder &builder, unsigned idx,
 
 /// Generates a for-loop or a while-loop, depending on whether it implements
 /// singleton iteration or co-iteration over the given conjunction.
-static Operation *genLoop(CodegenEnv &env, OpBuilder &builder, unsigned at,
-                          bool needsUniv, ArrayRef<size_t> tids,
-                          ArrayRef<size_t> dims, bool isFor) {
-  assert(tids.size() == dims.size());
-  unsigned idx = env.topSortAt(at);
+static Operation *genLoop(CodegenEnv &env, OpBuilder &builder, LoopOrd at,
+                          bool needsUniv, ArrayRef<TensorId> tids,
+                          ArrayRef<Level> lvls, bool isFor) {
+  assert(tids.size() == lvls.size());
+  const LoopId idx = env.topSortAt(at);
   if (isFor) {
     bool isOuter = at == 0;
     bool isInner = at == env.topSortSize() - 1;
-    return genFor(env, builder, isOuter, isInner, idx, tids, dims);
+    return genFor(env, builder, isOuter, isInner, idx, tids, lvls);
   }
-  return genWhile(env, builder, idx, needsUniv, tids, dims);
+  return genWhile(env, builder, idx, needsUniv, tids, lvls);
 }
 
 /// Generates the induction structure for a while-loop.
-static void finalizeWhileOp(CodegenEnv &env, OpBuilder &builder, unsigned idx,
+static void finalizeWhileOp(CodegenEnv &env, OpBuilder &builder, LoopId idx,
                             bool needsUniv, BitVector &induction,
                             scf::WhileOp whileOp) {
   Location loc = env.op().getLoc();
@@ -1126,26 +1191,26 @@ static void finalizeWhileOp(CodegenEnv &env, OpBuilder &builder, unsigned idx,
 }
 
 /// Generates a single if-statement within a while-loop.
-static scf::IfOp genIf(CodegenEnv &env, OpBuilder &builder, unsigned idx,
+static scf::IfOp genIf(CodegenEnv &env, OpBuilder &builder, LoopId ldx,
                        BitVector &conditions) {
   Location loc = env.op().getLoc();
   SmallVector<Type> types;
   Value cond;
-  for (unsigned b = 0, be = conditions.size(); b < be; b++) {
+  for (TensorLoopId b = 0, be = conditions.size(); b < be; b++) {
     if (!conditions[b])
       continue;
-    unsigned tensor = env.merger().tensor(b);
-    assert(idx == env.merger().index(b));
+    const TensorId tid = env.merger().tensor(b);
+    assert(ldx == env.merger().loop(b));
     Value clause;
-    if (isCompressedDLT(env.dlt(b)) || isSingletonDLT(env.dlt(b))) {
-      auto dim = *env.merger().getDimNum(tensor, idx);
-      Value op1 = env.emitter().getCoord()[tensor][dim];
-      Value op2 = env.getLoopIdxValue(idx);
-      clause = builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::eq, op1,
-                                             op2);
+    const auto dlt = env.dlt(b);
+    if (isCompressedDLT(dlt) || isSingletonDLT(dlt)) {
+      const Level lvl = *env.merger().getLvl(tid, ldx);
+      const Value crd = env.emitter().getCoords()[tid][lvl];
+      const Value lvar = env.getLoopVar(ldx);
+      clause = builder.create<arith::CmpIOp>(loc, arith::CmpIPredicate::eq, crd,
+                                             lvar);
     } else {
-      assert(isDenseDLT(env.merger().getDimLevelType(b)) ||
-             isUndefDLT(env.merger().getDimLevelType(b)));
+      assert(isDenseDLT(dlt) || isUndefDLT(dlt));
       clause = constantI1(builder, loc, true);
     }
     cond = cond ? builder.create<arith::AndIOp>(loc, cond, clause) : clause;
@@ -1195,41 +1260,40 @@ static void endIf(CodegenEnv &env, OpBuilder &builder, scf::IfOp ifOp,
 
 /// Starts a loop sequence at given level. Returns true if
 /// the universal loop index must be maintained at this level.
-static bool startLoopSeq(CodegenEnv &env, OpBuilder &builder, unsigned exp,
-                         unsigned at, unsigned idx, unsigned ldx,
-                         unsigned lts) {
-  assert(!env.getLoopIdxValue(idx));
+static bool startLoopSeq(CodegenEnv &env, OpBuilder &builder, ExprId exp,
+                         LoopOrd at, LoopId idx, LoopId ldx, LatSetId lts) {
+  assert(!env.getLoopVar(idx));
   // Emit invariants at this loop sequence level.
   genInvariants(env, builder, exp, ldx, /*atStart=*/true);
   // Emit access pattern expansion for sparse tensor output.
   genExpand(env, builder, at, /*atStart=*/true);
   // Emit further intitialization at this loop sequence level.
-  unsigned l0 = env.set(lts)[0];
+  const LatPointId l0 = env.set(lts)[0];
   bool needsUniv = false;
 
-  SmallVector<size_t> tids;
-  SmallVector<size_t> dims;
-  env.merger().foreachTidDimPairInBits(
-      env.lat(l0).bits, [&](unsigned b, unsigned tid,
-                            std::optional<unsigned> dim, DimLevelType dlt) {
-        assert(env.merger().index(b) == idx);
-        if (isDenseDLT(dlt) || isUndefDLT(dlt)) {
-          needsUniv = true;
-        } else {
-          // sparse/singleton dim levels.
-          tids.push_back(tid);
-          dims.push_back(*dim);
-        }
-      });
+  SmallVector<TensorId> tids;
+  SmallVector<Level> lvls;
+  env.merger().foreachTensorLoopId(l0, [&](TensorLoopId b, TensorId tid,
+                                           std::optional<Level> lvl,
+                                           DimLevelType dlt) {
+    assert(env.merger().loop(b) == idx);
+    if (isDenseDLT(dlt) || isUndefDLT(dlt)) {
+      needsUniv = true;
+    } else {
+      // sparse/singleton levels.
+      tids.push_back(tid);
+      lvls.push_back(*lvl);
+    }
+  });
 
-  env.emitter().enterNewLoopSeq(builder, env.op().getLoc(), tids, dims);
+  env.emitter().enterNewLoopSeq(builder, env.op().getLoc(), tids, lvls);
 
   // Maintain the universal index only if it is actually
   // consumed by a subsequent lattice point.
   if (needsUniv) {
     unsigned lsize = env.set(lts).size();
     for (unsigned i = 1; i < lsize; i++) {
-      unsigned li = env.set(lts)[i];
+      const LatPointId li = env.set(lts)[i];
       if (!env.merger().hasAnySparse(env.lat(li).simple))
         return true;
     }
@@ -1238,23 +1302,25 @@ static bool startLoopSeq(CodegenEnv &env, OpBuilder &builder, unsigned exp,
 }
 
 static void genConstantDenseAddressFromLevel(CodegenEnv &env,
-                                             OpBuilder &builder, unsigned tid,
-                                             Level lvl) {
+                                             OpBuilder &builder, TensorId tid,
+                                             Level startLvl) {
   // TODO: Handle affine expression on output tensor.
   linalg::GenericOp op = env.op();
   assert(tid < op.getNumDpsInputs());
   OpOperand *input = op.getDpsInputOperands()[tid];
-  ArrayRef<AffineExpr> affines = op.getMatchingIndexingMap(input).getResults();
+  const auto lvlExprs = op.getMatchingIndexingMap(input).getResults();
   const auto enc = getSparseTensorEncoding(input->get().getType());
   if (enc) {
+    const Location loc = op.getLoc();
+    const TensorId tid = input->getOperandNumber();
     const Level lvlRank = enc.getLvlRank();
-    assert(affines.size() == static_cast<size_t>(lvlRank));
-    for (Level l = lvl; l < lvlRank; l++) {
+    assert(lvlExprs.size() == static_cast<size_t>(lvlRank));
+    // FIXME: there is dim/lvl confusion here
+    for (Level l = startLvl; l < lvlRank; l++) {
       // FIXME: `toOrigDim` is deprecated.
-      AffineExpr affine = affines[toOrigDim(enc, l)];
-      if (enc.isDenseLvl(l) && affine.isa<AffineConstantExpr>())
-        env.emitter().genDenseAffineAddressAtCurLevel(
-            builder, op.getLoc(), input->getOperandNumber(), l, affine);
+      AffineExpr lvlExpr = lvlExprs[toOrigDim(enc, l)];
+      if (enc.isDenseLvl(l) && lvlExpr.isa<AffineConstantExpr>())
+        env.emitter().genDenseAffineAddress(builder, loc, tid, l, lvlExpr);
       else
         return; // break on first non-dense non-constant level
     }
@@ -1267,132 +1333,133 @@ static void genInitConstantDenseAddress(CodegenEnv &env,
   // starting from the first level as they do not depend on any thing.
   // E.g., [Dense, Dense, Sparse] -> (1, 2, d0), the addresses for the first two
   // levels can be determined before loops.
-  for (unsigned tid = 0, e = env.op().getNumDpsInputs(); tid < e; tid++)
+  for (TensorId tid = 0, e = env.op().getNumDpsInputs(); tid < e; tid++)
     genConstantDenseAddressFromLevel(env, rewriter, tid, 0);
 }
 
 /// Return true if the lattices bit can be iterated by a for loop.
-static bool translateBitsToTidDimPairs(
-    CodegenEnv &env, unsigned li, unsigned idx, SmallVectorImpl<size_t> &tids,
-    SmallVectorImpl<size_t> &dims, SmallVectorImpl<size_t> &affineTids,
-    SmallVectorImpl<size_t> &affineDims, SmallVectorImpl<AffineExpr> &exps) {
-  const BitVector &all = env.lat(li).bits;
+static bool translateBitsToTidLvlPairs(
+    CodegenEnv &env, LatPointId li, LoopId ldx, SmallVectorImpl<TensorId> &tids,
+    SmallVectorImpl<Level> &lvls, SmallVectorImpl<TensorId> &affineTids,
+    SmallVectorImpl<Level> &affineLvls, SmallVectorImpl<AffineExpr> &exps) {
   const BitVector &simple = env.lat(li).simple;
+  const TensorId outTid = env.merger().getOutTensorID();
+  const std::optional<Level> outLvl = env.merger().getLvl(outTid, ldx);
 
   unsigned numloopCond = 0;
-  // Converts bits to array + dim pair
-  env.merger().foreachTidDimPairInBits(
-      all, [&, idx](unsigned b, unsigned tid, std::optional<unsigned> dim,
-                    DimLevelType dlt) {
-        if (simple.test(b)) {
-          if (isUndefDLT(dlt)) {
-            // An undefined dlt in the lattices, we probably mean to iterate
-            // based on the dim of output tensor.
-            // E.g., this could be a synthetic tensor (for invariants and sparse
-            // output tensor).
-            // out[i][j] = invariant; or a broadcast
-            // out[i][j] = in[i] (j is undef for input)
-            tid = env.merger().getOutTensorID();
-            dim = env.merger().getDimNum(tid, idx);
-            // Skips invalid dim (e.g., when this is a zero ranked tensor).
-            if (!dim)
-              return;
-          }
-          tids.push_back(tid);
-          dims.push_back(*dim);
-          numloopCond++;
-        } else if (isDenseDLT(dlt)) {
-          tids.push_back(tid);
-          dims.push_back(*dim);
-        } else {
-          assert(isUndefDLT(dlt));
-          linalg::GenericOp op = env.op();
-          if (tid >= op.getNumDpsInputs())
-            // We only handle affine expression on input tensors (for now).
-            return;
-          OpOperand *operand = &op->getOpOperand(tid);
-          const auto stt = getSparseTensorType(operand->get());
-          // Non-annotated dense tensors requires no special handling.
-          if (!stt.hasEncoding())
-            return;
-
-          ArrayRef<AffineExpr> affines =
-              op.getMatchingIndexingMap(operand).getResults();
-          const Level lvlRank = stt.getLvlRank();
-          assert(affines.size() == static_cast<size_t>(lvlRank));
-          for (Level l = 0; l < lvlRank; l++) {
-            // FIXME: `toOrigDim` is deprecated.
-            AffineExpr exp = affines[toOrigDim(stt.getEncoding(), l)];
-            // Skip simple affine expression and non dense dimensions (which has
-            // it own filter loop).
-            if (exp.isa<AffineDimExpr>() || !stt.isDenseLvl(l))
-              continue;
-
-            // Constant affine expression are handled in genLoop
-            if (!exp.isa<AffineConstantExpr>()) {
-              bool atLevel = false;
-              if (isInvariantAffine(env, exp, idx, atLevel) && atLevel) {
-                // If the compound affine is invariant and we are right at the
-                // level. We need to generate the address according to the
-                // affine expression. This is also the best place we can do it
-                // to avoid putting it inside inner loops.
-                // NOTE: It assumes that the levels of the input tensor are
-                // initialized in order (and it is also currently guaranteed by
-                // computeIterationGraph), another more admissible approach
-                // might be accepting out-of-order access between consecutive
-                // dense levels.
-                affineTids.push_back(tid);
-                affineDims.push_back(l);
-                exps.push_back(exp);
-              }
-            }
+  bool hasNonUnique = false;
+  env.merger().foreachTensorLoopId(li, [&, ldx](TensorLoopId b, TensorId tid,
+                                                std::optional<Level> lvl,
+                                                DimLevelType dlt) {
+    if (simple.test(b)) {
+      if (isUndefDLT(dlt)) {
+        // An undefined dlt in the lattices, we probably mean to
+        // iterate based on the level of output tensor.  E.g., this
+        // could be a synthetic tensor (for invariants and sparse
+        // output tensor).
+        // out[i][j] = invariant; or a broadcast
+        // out[i][j] = in[i] (j is undef for input)
+        tid = outTid;
+        lvl = outLvl;
+        // Skips invalid lvl (e.g., when this is a zero ranked tensor).
+        if (!lvl)
+          return;
+      }
+      hasNonUnique = !isUniqueDLT(dlt) || hasNonUnique;
+      tids.push_back(tid);
+      lvls.push_back(*lvl);
+      numloopCond++;
+    } else if (isDenseDLT(dlt)) {
+      tids.push_back(tid);
+      lvls.push_back(*lvl);
+    } else {
+      assert(isUndefDLT(dlt));
+      linalg::GenericOp op = env.op();
+      if (tid >= op.getNumDpsInputs())
+        // We only handle affine expression on input tensors (for now).
+        return;
+      OpOperand *operand = &op->getOpOperand(tid);
+      const auto stt = getSparseTensorType(operand->get());
+      // Non-annotated dense tensors requires no special handling.
+      if (!stt.hasEncoding())
+        return;
+
+      ArrayRef<AffineExpr> affines =
+          op.getMatchingIndexingMap(operand).getResults();
+      const Level lvlRank = stt.getLvlRank();
+      assert(affines.size() == static_cast<size_t>(lvlRank));
+      for (Level l = 0; l < lvlRank; l++) {
+        // FIXME: `toOrigDim` is deprecated.
+        AffineExpr exp = affines[toOrigDim(stt.getEncoding(), l)];
+        // Skip simple affine expression and non-dense levels (which
+        // have their own filter loop).
+        if (exp.isa<AffineDimExpr>() || !stt.isDenseLvl(l))
+          continue;
+
+        // Constant affine expression are handled in genLoop
+        if (!exp.isa<AffineConstantExpr>()) {
+          bool isAtLoop = false;
+          if (isInvariantAffine(env, exp, ldx, isAtLoop) && isAtLoop) {
+            // If the compound affine is invariant and we are right at the
+            // level. We need to generate the address according to the
+            // affine expression. This is also the best place we can do it
+            // to avoid putting it inside inner loops.
+            // NOTE: It assumes that the levels of the input tensor are
+            // initialized in order (and it is also currently guaranteed by
+            // computeIterationGraph), another more admissible approach
+            // might be accepting out-of-order access between consecutive
+            // dense levels.
+            affineTids.push_back(tid);
+            affineLvls.push_back(l);
+            exps.push_back(exp);
           }
         }
-      });
+      }
+    }
+  });
 
-  if (isDenseDLT(env.dlt(env.merger().getOutTensorID(), idx))) {
+  if (isDenseDLT(env.dlt(outTid, ldx))) {
     // Note that we generate dense indices of the output tensor
     // unconditionally, since they may not appear in the lattice, but may be
     // needed for linearized env.
-    auto dim = *env.merger().getDimNum(env.merger().getOutTensorID(), idx);
-    tids.push_back(env.merger().getOutTensorID());
-    dims.push_back(dim);
+    tids.push_back(outTid);
+    lvls.push_back(*outLvl);
   }
 
   assert(numloopCond > 0);
-  // If we just need to one loop conditions, the loop can be generated by a for
-  // loop.
-  return numloopCond == 1;
+  // If we just need to one loop conditions and the conditions is not imposed on
+  // non-unique level, the loop can be generated by a for loop.
+  return numloopCond == 1 && !hasNonUnique;
 }
 
 /// Starts a single loop in current sequence.
-static Operation *startLoop(CodegenEnv &env, OpBuilder &builder, unsigned at,
-                            unsigned li, bool needsUniv) {
-  // The set of tensors + dims to generate loops on
-  SmallVector<size_t> tids, dims;
+static Operation *startLoop(CodegenEnv &env, OpBuilder &builder, LoopOrd at,
+                            LatPointId li, bool needsUniv) {
+  // The set of tensors + lvls to generate loops on
+  SmallVector<TensorId> tids, affineTids;
+  SmallVector<Level> lvls, affineLvls;
   // The set of dense tensors with non-trivial affine expression that just
   // becomes invariant and the address shall now be generated at the current
   // level.
-  SmallVector<size_t> affineTids, affineDims;
   SmallVector<AffineExpr> affines;
-  bool isFor = translateBitsToTidDimPairs(
-      env, li, env.topSortAt(at), tids, dims, affineTids, affineDims, affines);
+  bool isFor = translateBitsToTidLvlPairs(
+      env, li, env.topSortAt(at), tids, lvls, affineTids, affineLvls, affines);
 
   // Emit the for/while-loop control.
-  Operation *loop = genLoop(env, builder, at, needsUniv, tids, dims, isFor);
-  for (auto [tid, dim, exp] : llvm::zip(affineTids, affineDims, affines)) {
-    env.emitter().genDenseAffineAddressAtCurLevel(builder, env.op().getLoc(),
-                                                  tid, dim, exp);
+  Operation *loop = genLoop(env, builder, at, needsUniv, tids, lvls, isFor);
+  Location loc = env.op().getLoc();
+  for (auto [tid, lvl, exp] : llvm::zip(affineTids, affineLvls, affines)) {
+    env.emitter().genDenseAffineAddress(builder, loc, tid, lvl, exp);
   }
 
-  // Until now, we have entered every <tid, dim> pair in {cond, extra,
-  // affine}Tids/Dims. The addresses of the upcoming levels which are dependent
+  // Until now, we have entered every <tid, lvl> pair in {cond, extra,
+  // affine}Tids/Lvls. The addresses of the upcoming levels which are dependent
   // on constant affines expression may now be determined.
-  auto allTids = llvm::concat<size_t>(tids, affineTids);
-  auto allDims = llvm::concat<size_t>(dims, affineDims);
-  for (auto [tid, dim] : llvm::zip(allTids, allDims)) {
+  auto allTids = llvm::concat<TensorId>(tids, affineTids);
+  auto allLvls = llvm::concat<Level>(lvls, affineLvls);
+  for (auto [tid, lvl] : llvm::zip(allTids, allLvls)) {
     if (tid != env.merger().getOutTensorID())
-      genConstantDenseAddressFromLevel(env, builder, tid, dim + 1);
+      genConstantDenseAddressFromLevel(env, builder, tid, lvl + 1);
   }
 
   return loop;
@@ -1400,7 +1467,7 @@ static Operation *startLoop(CodegenEnv &env, OpBuilder &builder, unsigned at,
 
 /// Ends a single loop in current sequence. Returns new values for needsUniv.
 static bool endLoop(CodegenEnv &env, RewriterBase &rewriter, Operation *loop,
-                    unsigned idx, unsigned li, bool needsUniv) {
+                    LoopId idx, LatPointId li, bool needsUniv) {
   // End a while-loop.
   if (auto whileOp = dyn_cast<scf::WhileOp>(loop)) {
     finalizeWhileOp(env, rewriter, idx, needsUniv, env.lat(li).bits, whileOp);
@@ -1421,9 +1488,9 @@ static bool endLoop(CodegenEnv &env, RewriterBase &rewriter, Operation *loop,
 }
 
 /// Ends a loop sequence at given level.
-static void endLoopSeq(CodegenEnv &env, OpBuilder &builder, unsigned exp,
-                       unsigned at, unsigned idx, unsigned ldx) {
-  assert(env.getLoopIdxValue(idx) == nullptr);
+static void endLoopSeq(CodegenEnv &env, OpBuilder &builder, ExprId exp,
+                       LoopOrd at, LoopId idx, LoopId ldx) {
+  assert(!env.getLoopVar(idx));
   env.emitter().exitCurrentLoopSeq();
   // Unmark bookkeeping of invariants and loop index.
   genInvariants(env, builder, exp, ldx, /*atStart=*/false);
@@ -1434,23 +1501,21 @@ static void endLoopSeq(CodegenEnv &env, OpBuilder &builder, unsigned exp,
 /// Recursively generates code while computing iteration lattices in order
 /// to manage the complexity of implementing co-iteration over unions
 /// and intersections of sparse iterations spaces.
-static void genStmt(CodegenEnv &env, RewriterBase &rewriter, unsigned exp,
-                    unsigned at) {
+static void genStmt(CodegenEnv &env, RewriterBase &rewriter, ExprId exp,
+                    LoopOrd at) {
   // At each leaf, assign remaining tensor (sub)expression to output tensor.
   if (at == env.topSortSize()) {
-    unsigned ldx = env.topSortAt(at - 1);
+    const LoopId ldx = env.topSortAt(at - 1);
     Value rhs = genExp(env, rewriter, exp, ldx);
     genTensorStore(env, rewriter, exp, rhs);
     return;
   }
 
   // Construct iteration lattices for current loop index, with L0 at top.
-  unsigned idx = env.topSortAt(at);
-  unsigned ldx = at == 0 ? -1u : env.topSortAt(at - 1);
-  unsigned lts = env.merger().optimizeSet(env.merger().buildLattices(exp, idx));
-
-  // TODO: sort
-  // TODO: dedup
+  const LoopId idx = env.topSortAt(at);
+  const LoopId ldx = at == 0 ? kInvalidId : env.topSortAt(at - 1);
+  const LatSetId lts =
+      env.merger().optimizeSet(env.merger().buildLattices(exp, idx));
 
   // Start a loop sequence.
   bool needsUniv = startLoopSeq(env, rewriter, exp, at, idx, ldx, lts);
@@ -1459,7 +1524,7 @@ static void genStmt(CodegenEnv &env, RewriterBase &rewriter, unsigned exp,
   unsigned lsize = env.set(lts).size();
   for (unsigned i = 0; i < lsize; i++) {
     // Start a loop.
-    unsigned li = env.set(lts)[i];
+    const LatPointId li = env.set(lts)[i];
     Operation *loop = startLoop(env, rewriter, at, li, needsUniv);
 
     // Visit all lattices points with Li >= Lj to generate the
@@ -1469,8 +1534,8 @@ static void genStmt(CodegenEnv &env, RewriterBase &rewriter, unsigned exp,
     Value insInput = env.getInsertionChain();
     bool isWhile = dyn_cast<scf::WhileOp>(loop) != nullptr;
     for (unsigned j = 0; j < lsize; j++) {
-      unsigned lj = env.set(lts)[j];
-      unsigned ej = env.lat(lj).exp;
+      const LatPointId lj = env.set(lts)[j];
+      const ExprId ej = env.lat(lj).exp;
       if (li == lj || env.merger().latGT(li, lj)) {
         // Recurse into body of each branch.
         if (isWhile) {
@@ -1534,13 +1599,16 @@ struct GenericOpSparsifier : public OpRewritePattern<linalg::GenericOp> {
     if (op.getNumDpsInits() != 1 || hasCompoundAffineOnSparseOut(op))
       return failure();
 
+    if (options.enableIndexReduction)
+      llvm_unreachable("not yet implemented");
+
     // Sets up a code generation environment.
-    unsigned numTensors = op->getNumOperands();
-    unsigned numLoops = op.getNumLoops();
-    unsigned numFilterLoops = getNumCompoundAffineOnSparseDims(op);
+    const unsigned numTensors = op->getNumOperands();
+    const unsigned numLoops = op.getNumLoops();
+    const unsigned numFilterLoops = getNumCompoundAffineOnSparseLvls(op);
     CodegenEnv env(op, options, numTensors, numLoops, numFilterLoops);
 
-    // Detects sparse annotations and translates the per-dimension sparsity
+    // Detects sparse annotations and translates the per-level sparsity
     // information for all tensors to loop indices in the kernel.
     if (!findSparseAnnotations(env))
       return failure();
@@ -1562,11 +1630,11 @@ struct GenericOpSparsifier : public OpRewritePattern<linalg::GenericOp> {
     // computation. Must be ordered from more strict to less strict.
     // Ideally (though might not be guaranteed), the eariler a constraint mask
     // can be satisfied, the faster the generated kernel will be.
-    const auto allMask = {
+    const auto allMasks = {
         SortMask::kIncludeAll,        SortMask::kIncludeDense,
         SortMask::kIncludeDenseInput, SortMask::kIncludeDenseOutput,
         SortMask::kIncludeUndef,      SortMask::kSparseOnly};
-    for (auto mask : allMask) {
+    for (const SortMask mask : allMasks) {
       if (computeIterationGraph(env, mask)) {
         hasCycle = false;
         if (env.isAdmissibleTopoOrder()) {
@@ -1585,7 +1653,7 @@ struct GenericOpSparsifier : public OpRewritePattern<linalg::GenericOp> {
     env.startEmit();
     genBuffers(env, rewriter);
     genInitConstantDenseAddress(env, rewriter);
-    genStmt(env, rewriter, env.getTensorExp(), 0);
+    genStmt(env, rewriter, env.getExprId(), 0);
     genResult(env, rewriter);
     return success();
   }
@@ -1597,7 +1665,7 @@ struct GenericOpSparsifier : public OpRewritePattern<linalg::GenericOp> {
     // sparse input tensor in succession until an acylic
     // iteration graph results.
     for (OpOperand *t : env.op().getDpsInputOperands()) {
-      unsigned tensor = t->getOperandNumber();
+      const TensorId tid = t->getOperandNumber();
       Value tval = t->get();
       auto srcEnc = getSparseTensorEncoding(tval.getType());
       if (!srcEnc || !computeIterationGraph(env, SortMask::kSparseOnly, t))
@@ -1613,13 +1681,13 @@ struct GenericOpSparsifier : public OpRewritePattern<linalg::GenericOp> {
       auto dstEnc = SparseTensorEncodingAttr::get(
           getContext(), srcEnc.getDimLevelType(),
           permute(env, env.op().getMatchingIndexingMap(t)), // new order
-          srcEnc.getHigherOrdering(), srcEnc.getPointerBitWidth(),
-          srcEnc.getIndexBitWidth());
+          srcEnc.getHigherOrdering(), srcEnc.getPosWidth(),
+          srcEnc.getCrdWidth());
       auto dstTp = RankedTensorType::get(srcTp.getShape(),
                                          srcTp.getElementType(), dstEnc);
       auto convert = rewriter.create<ConvertOp>(tval.getLoc(), dstTp, tval);
-      rewriter.updateRootInPlace(
-          env.op(), [&]() { env.op()->setOperand(tensor, convert); });
+      rewriter.updateRootInPlace(env.op(),
+                                 [&]() { env.op()->setOperand(tid, convert); });
       rewriter.setInsertionPointAfter(env.op());
       rewriter.create<bufferization::DeallocTensorOp>(tval.getLoc(), convert);
       return success();
diff --git a/mlir/lib/Dialect/SparseTensor/Utils/Merger.cpp b/mlir/lib/Dialect/SparseTensor/Utils/Merger.cpp
index a0892b25e2c7b..029ce3f3f91ec 100644
--- a/mlir/lib/Dialect/SparseTensor/Utils/Merger.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Utils/Merger.cpp
@@ -30,7 +30,7 @@ static ExpArity getExpArity(Kind k) {
   // Leaf.
   case kTensor:
   case kInvariant:
-  case kIndex:
+  case kLoopVar:
     return ExpArity::kNullary;
   case kAbsF:
   case kAbsC:
@@ -98,20 +98,20 @@ static ExpArity getExpArity(Kind k) {
 // Constructors.
 //===----------------------------------------------------------------------===//
 
-TensorExp::TensorExp(Kind k, unsigned x, unsigned y, Value v, Operation *o)
+TensorExp::TensorExp(Kind k, unsigned x, ExprId y, Value v, Operation *o)
     : kind(k), val(v), op(o) {
   switch (kind) {
   // Leaf.
   case kTensor:
-    assert(x != -1u && y == -1u && !v && !o);
+    assert(x != kInvalidId && y == kInvalidId && !v && !o);
     tensor = x;
     break;
   case kInvariant:
-    assert(x == -1u && y == -1u && v && !o);
+    assert(x == kInvalidId && y == kInvalidId && v && !o);
     break;
-  case kIndex:
-    assert(x != -1u && y == -1u && !v && !o);
-    index = x;
+  case kLoopVar:
+    assert(x != kInvalidId && y == kInvalidId && !v && !o);
+    loop = x;
     break;
   // Unary operations.
   case kAbsF:
@@ -134,7 +134,7 @@ TensorExp::TensorExp(Kind k, unsigned x, unsigned y, Value v, Operation *o)
   case kNegI:
   case kCIm:
   case kCRe:
-    assert(x != -1u && y == -1u && !v && !o);
+    assert(x != kInvalidId && y == kInvalidId && !v && !o);
     children.e0 = x;
     children.e1 = y;
     break;
@@ -149,20 +149,20 @@ TensorExp::TensorExp(Kind k, unsigned x, unsigned y, Value v, Operation *o)
   case kCastIdx:
   case kTruncI:
   case kBitCast:
-    assert(x != -1u && y == -1u && v && !o);
+    assert(x != kInvalidId && y == kInvalidId && v && !o);
     children.e0 = x;
     children.e1 = y;
     break;
   case kBinaryBranch:
   case kSelect:
-    assert(x != -1u && y == -1u && !v && o);
+    assert(x != kInvalidId && y == kInvalidId && !v && o);
     children.e0 = x;
     children.e1 = y;
     break;
   case kUnary:
     // No assertion on y can be made, as the branching paths involve both
-    // a unary (mapSet) and binary (takeDisj) pathway.
-    assert(x != -1u && !v && o);
+    // a unary (`mapSet`) and binary (`disjSet`) pathway.
+    assert(x != kInvalidId && !v && o);
     children.e0 = x;
     children.e1 = y;
     break;
@@ -186,82 +186,89 @@ TensorExp::TensorExp(Kind k, unsigned x, unsigned y, Value v, Operation *o)
   case kShrS:
   case kShrU:
   case kShlI:
-    assert(x != -1u && y != -1u && !v && !o);
+    assert(x != kInvalidId && y != kInvalidId && !v && !o);
     children.e0 = x;
     children.e1 = y;
     break;
   case kBinary:
   case kReduce:
-    assert(x != -1u && y != -1u && !v && o);
+    assert(x != kInvalidId && y != kInvalidId && !v && o);
     children.e0 = x;
     children.e1 = y;
     break;
   }
 }
 
-LatPoint::LatPoint(unsigned n, unsigned e, unsigned b)
-    : bits(n, false), exp(e) {
+LatPoint::LatPoint(const BitVector &bits, ExprId e) : bits(bits), exp(e) {}
+
+LatPoint::LatPoint(unsigned numTensors, unsigned numLoops, TensorId t, LoopId i,
+                   ExprId e)
+    : bits(numLoops * numTensors, false), exp(e) {
+  assert(t < numTensors && i < numLoops);
+  const TensorLoopId b = numTensors * i + t;
   bits.set(b);
 }
 
-LatPoint::LatPoint(const BitVector &b, unsigned e) : bits(b), exp(e) {}
-
-Merger::Merger(unsigned t, unsigned l, unsigned fl)
-    : outTensor(t - 1), syntheticTensor(t), numTensors(t + 1),
-      numNativeLoops(l), numLoops(l + fl), hasSparseOut(false),
-      dimTypes(numTensors,
+Merger::Merger(unsigned numInputOutputTensors, unsigned numNativeLoops,
+               unsigned numFilterLoops)
+    : outTensor(numInputOutputTensors - 1),
+      syntheticTensor(numInputOutputTensors),
+      numTensors(numInputOutputTensors + 1), numNativeLoops(numNativeLoops),
+      numLoops(numNativeLoops + numFilterLoops), hasSparseOut(false),
+      lvlTypes(numTensors,
                std::vector<DimLevelType>(numLoops, DimLevelType::Undef)),
-      loopIdxToDim(numTensors, std::vector<std::optional<unsigned>>(
-                                   numLoops, std::nullopt)),
-      dimToLoopIdx(numTensors, std::vector<std::optional<unsigned>>(
-                                   numLoops, std::nullopt)) {}
+      loopToLvl(numTensors,
+                std::vector<std::optional<Level>>(numLoops, std::nullopt)),
+      lvlToLoop(numTensors,
+                std::vector<std::optional<LoopId>>(numLoops, std::nullopt)) {}
 
 //===----------------------------------------------------------------------===//
 // Lattice methods.
 //===----------------------------------------------------------------------===//
 
-unsigned Merger::addExp(Kind k, unsigned e0, unsigned e1, Value v,
-                        Operation *op) {
-  unsigned e = tensorExps.size();
-  tensorExps.push_back(TensorExp(k, e0, e1, v, op));
+ExprId Merger::addExp(Kind k, unsigned x, ExprId y, Value v, Operation *op) {
+  const ExprId e = tensorExps.size();
+  assert((k != kTensor || x < numTensors) && (k != kLoopVar || x < numLoops));
+  tensorExps.emplace_back(k, x, y, v, op);
   return e;
 }
 
-unsigned Merger::addLat(unsigned t, unsigned i, unsigned e) {
+LatPointId Merger::addLat(TensorId t, LoopId i, ExprId e) {
   assert(t < numTensors && i < numLoops);
-  unsigned p = latPoints.size();
-  latPoints.push_back(LatPoint(numLoops * numTensors, e, numTensors * i + t));
+  const LatPointId p = latPoints.size();
+  latPoints.emplace_back(numTensors, numLoops, t, i, e);
   return p;
 }
 
-unsigned Merger::addSet() {
-  unsigned s = latSets.size();
+LatSetId Merger::addSet() {
+  const LatSetId s = latSets.size();
   latSets.emplace_back();
   return s;
 }
 
-unsigned Merger::conjLatPoint(Kind kind, unsigned p0, unsigned p1,
-                              Operation *op) {
-  unsigned p = latPoints.size();
-  BitVector nb = BitVector(latPoints[p0].bits);
-  nb |= latPoints[p1].bits;
-  unsigned e = addExp(kind, latPoints[p0].exp, latPoints[p1].exp, Value(), op);
-  latPoints.push_back(LatPoint(nb, e));
+LatPointId Merger::conjLat(Kind kind, LatPointId p0, LatPointId p1,
+                           Operation *op) {
+  const LatPointId p = latPoints.size();
+  BitVector bits(latPoints[p0].bits);
+  bits |= latPoints[p1].bits;
+  const ExprId e =
+      addExp(kind, latPoints[p0].exp, latPoints[p1].exp, Value(), op);
+  latPoints.emplace_back(bits, e);
   return p;
 }
 
-unsigned Merger::takeConj(Kind kind, unsigned s0, unsigned s1, Operation *op) {
-  unsigned s = addSet();
-  for (unsigned p0 : latSets[s0])
-    for (unsigned p1 : latSets[s1])
-      latSets[s].push_back(conjLatPoint(kind, p0, p1, op));
+LatSetId Merger::conjSet(Kind kind, LatSetId s0, LatSetId s1, Operation *op) {
+  const LatSetId s = addSet();
+  for (const LatPointId p0 : latSets[s0])
+    for (const LatPointId p1 : latSets[s1])
+      latSets[s].push_back(conjLat(kind, p0, p1, op));
   return s;
 }
 
-unsigned Merger::takeDisj(Kind kind, unsigned s0, unsigned s1, Operation *op) {
-  unsigned s = takeConj(kind, s0, s1, op);
+LatSetId Merger::disjSet(Kind kind, LatSetId s0, LatSetId s1, Operation *op) {
+  const LatSetId s = conjSet(kind, s0, s1, op);
   // Followed by all in s0.
-  for (unsigned p : latSets[s0])
+  for (const LatPointId p : latSets[s0])
     latSets[s].push_back(p);
   // Map binary 0-y to unary -y.
   // TODO: move this if-else logic into buildLattices
@@ -272,56 +279,56 @@ unsigned Merger::takeDisj(Kind kind, unsigned s0, unsigned s1, Operation *op) {
   else if (kind == kSubI)
     s1 = mapSet(kNegI, s1);
   // Followed by all in s1.
-  for (unsigned p : latSets[s1])
+  for (const LatPointId p : latSets[s1])
     latSets[s].push_back(p);
   return s;
 }
 
-unsigned Merger::takeCombi(Kind kind, unsigned s0, unsigned s1, Operation *orig,
-                           bool includeLeft, Kind ltrans, Operation *opleft,
-                           bool includeRight, Kind rtrans, Operation *opright) {
-  unsigned s = takeConj(kind, s0, s1, orig);
+LatSetId Merger::combiSet(Kind kind, LatSetId s0, LatSetId s1, Operation *orig,
+                          bool includeLeft, Kind ltrans, Operation *opleft,
+                          bool includeRight, Kind rtrans, Operation *opright) {
+  const LatSetId s = conjSet(kind, s0, s1, orig);
   // Left Region.
   if (includeLeft) {
     if (opleft)
       s0 = mapSet(ltrans, s0, Value(), opleft);
-    for (unsigned p : latSets[s0])
+    for (const LatPointId p : latSets[s0])
       latSets[s].push_back(p);
   }
   // Right Region.
   if (includeRight) {
     if (opright)
       s1 = mapSet(rtrans, s1, Value(), opright);
-    for (unsigned p : latSets[s1])
+    for (const LatPointId p : latSets[s1])
       latSets[s].push_back(p);
   }
   return s;
 }
 
-unsigned Merger::mapSet(Kind kind, unsigned s0, Value v, Operation *op) {
+LatSetId Merger::mapSet(Kind kind, LatSetId s0, Value v, Operation *op) {
   assert(kAbsF <= kind && kind <= kSelect);
-  unsigned s = addSet();
-  for (unsigned p : latSets[s0]) {
-    unsigned e = addExp(kind, latPoints[p].exp, v, op);
-    latPoints.push_back(LatPoint(latPoints[p].bits, e));
+  const LatSetId s = addSet();
+  for (const LatPointId p : latSets[s0]) {
+    const ExprId e = addExp(kind, latPoints[p].exp, v, op);
+    latPoints.emplace_back(latPoints[p].bits, e);
     latSets[s].push_back(latPoints.size() - 1);
   }
   return s;
 }
 
-unsigned Merger::optimizeSet(unsigned s0) {
-  unsigned s = addSet();
+LatSetId Merger::optimizeSet(LatSetId s0) {
+  const LatSetId s = addSet();
   assert(!latSets[s0].empty());
-  unsigned p0 = latSets[s0][0];
-  for (unsigned p1 : latSets[s0]) {
+  const LatPointId p0 = latSets[s0][0];
+  for (const LatPointId p1 : latSets[s0]) {
     bool add = true;
     if (p0 != p1) {
-      // Is this a straightforward copy?
-      unsigned e = latPoints[p1].exp;
-      if (tensorExps[e].kind == kTensor && tensorExps[e].tensor == outTensor)
+      // Check whether this is a straightforward copy.
+      const ExprId e = latPoints[p1].exp;
+      if (expIsTensor(e, outTensor))
         continue;
-      // Conjunction already covered?
-      for (unsigned p2 : latSets[s]) {
+      // Check whether this conjunction is already covered.
+      for (const LatPointId p2 : latSets[s]) {
         assert(!latGT(p1, p2)); // Lj => Li would be bad
         if (onlyDenseDiff(p2, p1)) {
           add = false;
@@ -333,30 +340,30 @@ unsigned Merger::optimizeSet(unsigned s0) {
     if (add)
       latSets[s].push_back(p1);
   }
-  for (unsigned p : latSets[s])
+  for (const LatPointId p : latSets[s])
     latPoints[p].simple = simplifyCond(s, p);
   return s;
 }
 
-BitVector Merger::simplifyCond(unsigned s0, unsigned p0) {
+BitVector Merger::simplifyCond(LatSetId s0, LatPointId p0) {
   // First determine if this lattice point is a *singleton*, i.e.,
   // the last point in a lattice, no other is less than this one.
   bool isSingleton = true;
-  for (unsigned p1 : latSets[s0]) {
+  for (const LatPointId p1 : latSets[s0]) {
     if (p0 != p1 && latGT(p0, p1)) {
       isSingleton = false;
       break;
     }
   }
 
-  BitVector simple = latPoints[p0].bits;
+  BitVector simple(latPoints[p0].bits);
   bool reset = isSingleton && hasAnySparse(simple);
-  unsigned be = simple.size();
-  unsigned offset = 0; // relative to the end
+  const TensorLoopId be = simple.size();
+  TensorLoopId offset = 0; // relative to the end
   if (!reset)
-    // Starts resetting from a dense dimension, so that the first bit (if kept)
-    // is not undefined dimension type.
-    for (unsigned b = 0; b < be; b++) {
+    // Starts resetting from a dense level, so that the first bit (if kept)
+    // is not undefined level-type.
+    for (TensorLoopId b = 0; b < be; b++) {
       if (simple[b] && isDenseDLT(getDimLevelType(b))) {
         offset = be - b - 1; // relative to the end
         break;
@@ -365,24 +372,26 @@ BitVector Merger::simplifyCond(unsigned s0, unsigned p0) {
 
   // Now apply the two basic rules. We also iterate the bits reversely to always
   // keep the rightmost bit (which could possibly be a synthetic tensor).
-  for (unsigned b = be - 1 - offset, i = 0; i < be;
+  for (TensorLoopId b = be - 1 - offset, i = 0; i < be;
        b = b == 0 ? be - 1 : b - 1, i++) {
-    if (simple[b] && (!isCompressedDLT(getDimLevelType(b)) &&
-                      !isSingletonDLT(getDimLevelType(b)))) {
-      if (reset)
-        simple.reset(b);
-      reset = true;
+    if (simple[b]) {
+      const auto dlt = getDimLevelType(b);
+      if (!isCompressedDLT(dlt) && !isSingletonDLT(dlt)) {
+        if (reset)
+          simple.reset(b);
+        reset = true;
+      }
     }
   }
   return simple;
 }
 
-bool Merger::latGT(unsigned i, unsigned j) const {
+bool Merger::latGT(LatPointId i, LatPointId j) const {
   const BitVector &bitsi = latPoints[i].bits;
   const BitVector &bitsj = latPoints[j].bits;
   assert(bitsi.size() == bitsj.size());
   if (bitsi.count() > bitsj.count()) {
-    for (unsigned b = 0, be = bitsj.size(); b < be; b++)
+    for (TensorLoopId b = 0, be = bitsj.size(); b < be; b++)
       if (bitsj[b] && !bitsi[b])
         return false;
     return true;
@@ -390,13 +399,13 @@ bool Merger::latGT(unsigned i, unsigned j) const {
   return false;
 }
 
-bool Merger::onlyDenseDiff(unsigned i, unsigned j) {
-  BitVector tmp = latPoints[j].bits;
+bool Merger::onlyDenseDiff(LatPointId i, LatPointId j) const {
+  BitVector tmp(latPoints[j].bits);
   tmp ^= latPoints[i].bits;
   return !hasAnySparse(tmp);
 }
 
-bool Merger::expContainsTensor(unsigned e, unsigned t) const {
+bool Merger::expContainsTensor(ExprId e, TensorId t) const {
   if (tensorExps[e].kind == kTensor)
     return tensorExps[e].tensor == t;
 
@@ -404,24 +413,23 @@ bool Merger::expContainsTensor(unsigned e, unsigned t) const {
   case ExpArity::kNullary:
     return false;
   case ExpArity::kUnary: {
-    unsigned op = tensorExps[e].children.e0;
-    if (tensorExps[op].kind == kTensor && tensorExps[op].tensor == t)
+    const ExprId e0 = tensorExps[e].children.e0;
+    if (expIsTensor(e0, t))
       return true;
-    return expContainsTensor(op, t);
+    return expContainsTensor(e0, t);
   }
   case ExpArity::kBinary: {
-    unsigned op1 = tensorExps[e].children.e0;
-    unsigned op2 = tensorExps[e].children.e1;
-    if ((tensorExps[op1].kind == kTensor && tensorExps[op1].tensor == t) ||
-        (tensorExps[op2].kind == kTensor && tensorExps[op2].tensor == t))
+    const ExprId e0 = tensorExps[e].children.e0;
+    const ExprId e1 = tensorExps[e].children.e1;
+    if (expIsTensor(e0, t) || expIsTensor(e1, t))
       return true;
-    return expContainsTensor(op1, t) || expContainsTensor(op2, t);
+    return expContainsTensor(e0, t) || expContainsTensor(e1, t);
   }
   }
   llvm_unreachable("unexpected arity");
 }
 
-bool Merger::hasNegateOnOut(unsigned e) const {
+bool Merger::hasNegateOnOut(ExprId e) const {
   switch (tensorExps[e].kind) {
   case kNegF:
   case kNegC:
@@ -447,13 +455,14 @@ bool Merger::hasNegateOnOut(unsigned e) const {
   llvm_unreachable("unexpected kind");
 }
 
-bool Merger::isSingleCondition(unsigned t, unsigned e) const {
+bool Merger::isSingleCondition(TensorId t, ExprId e) const {
+  assert(t < numTensors && e < tensorExps.size());
   switch (tensorExps[e].kind) {
   // Leaf.
   case kTensor:
     return tensorExps[e].tensor == t;
   case kInvariant:
-  case kIndex:
+  case kLoopVar:
     return false;
   // Unary operations.
   case kAbsF:
@@ -532,10 +541,12 @@ bool Merger::isSingleCondition(unsigned t, unsigned e) const {
 }
 
 bool Merger::hasAnySparse(const BitVector &bits) const {
-  for (unsigned b = 0, be = bits.size(); b < be; b++)
-    if (bits[b] && (isCompressedDLT(getDimLevelType(b)) ||
-                    isSingletonDLT(getDimLevelType(b))))
-      return true;
+  for (TensorLoopId b = 0, be = bits.size(); b < be; b++)
+    if (bits[b]) {
+      const auto dlt = getDimLevelType(b);
+      if (isCompressedDLT(dlt) || isSingletonDLT(dlt))
+        return true;
+    }
   return false;
 }
 
@@ -552,7 +563,7 @@ static const char *kindToOpSymbol(Kind kind) {
     return "tensor";
   case kInvariant:
     return "invariant";
-  case kIndex:
+  case kLoopVar:
     return "index";
   // Unary operations.
   case kAbsF:
@@ -642,7 +653,7 @@ static const char *kindToOpSymbol(Kind kind) {
   llvm_unreachable("unexpected kind for symbol");
 }
 
-void Merger::dumpExp(unsigned e) const {
+void Merger::dumpExp(ExprId e) const {
   switch (tensorExps[e].kind) {
   // Leaf.
   case kTensor:
@@ -655,8 +666,8 @@ void Merger::dumpExp(unsigned e) const {
   case kInvariant:
     llvm::dbgs() << "invariant";
     break;
-  case kIndex:
-    llvm::dbgs() << "index_" << tensorExps[e].index;
+  case kLoopVar:
+    llvm::dbgs() << "loopvar_" << tensorExps[e].loop;
     break;
   // Unary operations.
   case kAbsF:
@@ -726,7 +737,7 @@ void Merger::dumpExp(unsigned e) const {
   }
 }
 
-void Merger::dumpLat(unsigned p) const {
+void Merger::dumpLat(LatPointId p) const {
   llvm::dbgs() << "lat(";
   dumpBits(latPoints[p].bits);
   llvm::dbgs() << " :";
@@ -736,9 +747,9 @@ void Merger::dumpLat(unsigned p) const {
   llvm::dbgs() << " )\n";
 }
 
-void Merger::dumpSet(unsigned s) const {
+void Merger::dumpSet(LatSetId s) const {
   llvm::dbgs() << "{ #" << latSets[s].size() << "\n";
-  for (unsigned p : latSets[s]) {
+  for (const LatPointId p : latSets[s]) {
     llvm::dbgs() << "  ";
     dumpLat(p);
   }
@@ -746,11 +757,11 @@ void Merger::dumpSet(unsigned s) const {
 }
 
 void Merger::dumpBits(const BitVector &bits) const {
-  for (unsigned b = 0, be = bits.size(); b < be; b++) {
+  for (TensorLoopId b = 0, be = bits.size(); b < be; b++) {
     if (bits[b]) {
-      unsigned t = tensor(b);
-      unsigned i = index(b);
-      DimLevelType dlt = dimTypes[t][i];
+      const TensorId t = tensor(b);
+      const LoopId i = loop(b);
+      const auto dlt = lvlTypes[t][i];
       llvm::dbgs() << " i_" << t << "_" << i << "_" << toMLIRString(dlt);
     }
   }
@@ -762,20 +773,20 @@ void Merger::dumpBits(const BitVector &bits) const {
 // Builder methods.
 //===----------------------------------------------------------------------===//
 
-unsigned Merger::buildLattices(unsigned e, unsigned i) {
-  Kind kind = tensorExps[e].kind;
+LatSetId Merger::buildLattices(ExprId e, LoopId i) {
+  const Kind kind = tensorExps[e].kind;
   switch (kind) {
   // Leaf.
   case kTensor:
   case kInvariant:
-  case kIndex: {
-    // Either the index is really used in the tensor expression, or it is
-    // set to the undefined index in that dimension. An invariant expression,
+  case kLoopVar: {
+    // Either the loop-var is really used in the tensor expression, or it is
+    // set to the undefined loop-var in that level. An invariant expression,
     // a proper index value, and a truly dynamic sparse output tensor are set
     // to a synthetic tensor with undefined indices only to ensure the
     // iteration space is not skipped as a result of their contents.
-    unsigned s = addSet();
-    unsigned t = syntheticTensor;
+    const LatSetId s = addSet();
+    TensorId t = syntheticTensor;
     if (kind == kTensor) {
       t = tensorExps[e].tensor;
       if (hasSparseOut && t == outTensor)
@@ -837,7 +848,7 @@ unsigned Merger::buildLattices(unsigned e, unsigned i) {
     //  ----+----------+------------+
     //      | absent() | present(y) |
     {
-      unsigned child0 = buildLattices(tensorExps[e].children.e0, i);
+      const LatSetId child0 = buildLattices(tensorExps[e].children.e0, i);
       UnaryOp unop = cast<UnaryOp>(tensorExps[e].op);
       Region &absentRegion = unop.getAbsentRegion();
 
@@ -849,8 +860,8 @@ unsigned Merger::buildLattices(unsigned e, unsigned i) {
       Block &absentBlock = absentRegion.front();
       YieldOp absentYield = cast<YieldOp>(absentBlock.getTerminator());
       Value absentVal = absentYield.getResult();
-      unsigned rhs = addExp(kInvariant, absentVal);
-      return takeDisj(kind, child0, buildLattices(rhs, i), unop);
+      const ExprId rhs = addExp(kInvariant, absentVal);
+      return disjSet(kind, child0, buildLattices(rhs, i), unop);
     }
   // Binary operations.
   case kMulF:
@@ -866,9 +877,8 @@ unsigned Merger::buildLattices(unsigned e, unsigned i) {
     //   x | 0 |x*y|
     //
     // Note even here, 0*NaN=NaN and 0*Inf=NaN, but that is ignored.
-    return takeConj(kind, // take binary conjunction
-                    buildLattices(tensorExps[e].children.e0, i),
-                    buildLattices(tensorExps[e].children.e1, i));
+    return conjSet(kind, buildLattices(tensorExps[e].children.e0, i),
+                   buildLattices(tensorExps[e].children.e1, i));
   case kDivF:
   case kDivC:
   case kDivS:
@@ -887,9 +897,8 @@ unsigned Merger::buildLattices(unsigned e, unsigned i) {
     //       rules applies (viz. x/c = x*(1/c) as far as lattice
     //       construction is concerned).
     assert(!maybeZero(tensorExps[e].children.e1));
-    return takeConj(kind, // take binary conjunction
-                    buildLattices(tensorExps[e].children.e0, i),
-                    buildLattices(tensorExps[e].children.e1, i));
+    return conjSet(kind, buildLattices(tensorExps[e].children.e0, i),
+                   buildLattices(tensorExps[e].children.e1, i));
   case kAddF:
   case kAddC:
   case kAddI:
@@ -905,9 +914,8 @@ unsigned Merger::buildLattices(unsigned e, unsigned i) {
     //  ---+---+---+    ---+---+---+
     //  !x | 0 | y |    !x | 0 |-y |
     //   x | x |x+y|     x | x |x-y|
-    return takeDisj(kind, // take binary disjunction
-                    buildLattices(tensorExps[e].children.e0, i),
-                    buildLattices(tensorExps[e].children.e1, i));
+    return disjSet(kind, buildLattices(tensorExps[e].children.e0, i),
+                   buildLattices(tensorExps[e].children.e1, i));
   case kShrS:
   case kShrU:
   case kShlI:
@@ -915,9 +923,8 @@ unsigned Merger::buildLattices(unsigned e, unsigned i) {
     // can only occur at the left-hand-side of the operator) can be handled
     // with the conjuction rule.
     assert(isInvariant(tensorExps[e].children.e1));
-    return takeConj(kind, // take binary conjunction
-                    buildLattices(tensorExps[e].children.e0, i),
-                    buildLattices(tensorExps[e].children.e1, i));
+    return conjSet(kind, buildLattices(tensorExps[e].children.e0, i),
+                   buildLattices(tensorExps[e].children.e1, i));
   case kBinary:
     // A custom binary operation.
     //
@@ -926,8 +933,8 @@ unsigned Merger::buildLattices(unsigned e, unsigned i) {
     //    !x  |  empty  |   right(y)   |
     //     x  | left(x) | overlap(x,y) |
     {
-      unsigned child0 = buildLattices(tensorExps[e].children.e0, i);
-      unsigned child1 = buildLattices(tensorExps[e].children.e1, i);
+      const LatSetId child0 = buildLattices(tensorExps[e].children.e0, i);
+      const LatSetId child1 = buildLattices(tensorExps[e].children.e1, i);
       BinaryOp binop = cast<BinaryOp>(tensorExps[e].op);
       Region &leftRegion = binop.getLeftRegion();
       Region &rightRegion = binop.getRightRegion();
@@ -945,20 +952,20 @@ unsigned Merger::buildLattices(unsigned e, unsigned i) {
       }
       bool includeLeft = binop.getLeftIdentity() || !leftRegion.empty();
       bool includeRight = binop.getRightIdentity() || !rightRegion.empty();
-      return takeCombi(kBinary, child0, child1, binop, includeLeft,
-                       kBinaryBranch, leftYield, includeRight, kBinaryBranch,
-                       rightYield);
+      return combiSet(kBinary, child0, child1, binop, includeLeft,
+                      kBinaryBranch, leftYield, includeRight, kBinaryBranch,
+                      rightYield);
     }
   case kReduce:
     // A custom reduce operation.
-    return takeConj(kind, buildLattices(tensorExps[e].children.e0, i),
-                    buildLattices(tensorExps[e].children.e1, i),
-                    tensorExps[e].op);
+    return conjSet(kind, buildLattices(tensorExps[e].children.e0, i),
+                   buildLattices(tensorExps[e].children.e1, i),
+                   tensorExps[e].op);
   }
   llvm_unreachable("unexpected expression kind");
 }
 
-std::optional<unsigned> Merger::buildTensorExpFromLinalg(linalg::GenericOp op) {
+std::optional<ExprId> Merger::buildTensorExpFromLinalg(linalg::GenericOp op) {
   // Build the linalg semantics backward from yield.
   Operation *yield = op.getRegion().front().getTerminator();
   assert(isa<linalg::YieldOp>(yield));
@@ -966,7 +973,7 @@ std::optional<unsigned> Merger::buildTensorExpFromLinalg(linalg::GenericOp op) {
 }
 
 /// Only returns false if we are certain this is a nonzero.
-bool Merger::maybeZero(unsigned e) const {
+bool Merger::maybeZero(ExprId e) const {
   if (tensorExps[e].kind == kInvariant) {
     if (auto c = tensorExps[e].val.getDefiningOp<complex::ConstantOp>()) {
       ArrayAttr arrayAttr = c.getValue();
@@ -981,11 +988,11 @@ bool Merger::maybeZero(unsigned e) const {
   return true;
 }
 
-bool Merger::isInvariant(unsigned e) const {
+bool Merger::isInvariant(ExprId e) const {
   return tensorExps[e].kind == kInvariant;
 }
 
-Type Merger::inferType(unsigned e, Value src) {
+Type Merger::inferType(ExprId e, Value src) const {
   // Obtain the destination type from the cast node.
   Type dtp = tensorExps[e].val.getType();
   // Inspect source type. For vector types, apply the same
@@ -998,7 +1005,7 @@ Type Merger::inferType(unsigned e, Value src) {
 /// Ensures that sparse compiler can generate code for expression.
 static bool isAdmissibleBranchExp(Operation *op, Block *block, Value v) {
   // Arguments are always admissible.
-  if (auto arg = v.dyn_cast<BlockArgument>())
+  if (v.isa<BlockArgument>())
     return true;
   // Accept index anywhere.
   Operation *def = v.getDefiningOp();
@@ -1025,9 +1032,9 @@ static bool isAdmissibleBranch(Operation *op, Region &region) {
   return isAdmissibleBranchExp(op, &region.front(), yield->getOperand(0));
 }
 
-std::optional<unsigned> Merger::buildTensorExp(linalg::GenericOp op, Value v) {
+std::optional<ExprId> Merger::buildTensorExp(linalg::GenericOp op, Value v) {
   if (auto arg = v.dyn_cast<BlockArgument>()) {
-    unsigned argN = arg.getArgNumber();
+    const TensorId argN = arg.getArgNumber();
     // Any argument of the generic op that is not marked as a scalar
     // argument is considered a tensor, indexed by the implicit loop
     // bounds. This includes rank-0 tensor arguments.
@@ -1048,13 +1055,13 @@ std::optional<unsigned> Merger::buildTensorExp(linalg::GenericOp op, Value v) {
   // Construct index operations.
   if (def->getNumOperands() == 0) {
     if (auto indexOp = dyn_cast<linalg::IndexOp>(def))
-      return addExp(kIndex, indexOp.getDim());
+      return addExp(kLoopVar, indexOp.getDim());
   }
   // Construct unary operations if subexpression can be built.
   if (def->getNumOperands() == 1) {
-    auto x = buildTensorExp(op, def->getOperand(0));
+    const auto x = buildTensorExp(op, def->getOperand(0));
     if (x.has_value()) {
-      unsigned e = *x;
+      const ExprId e = *x;
       if (isa<math::AbsFOp>(def))
         return addExp(kAbsF, e);
       if (isa<complex::AbsOp>(def))
@@ -1130,11 +1137,11 @@ std::optional<unsigned> Merger::buildTensorExp(linalg::GenericOp op, Value v) {
   // See buildLattices() for an explanation of rejecting certain
   // division and shift operations.
   if (def->getNumOperands() == 2) {
-    auto x = buildTensorExp(op, def->getOperand(0));
-    auto y = buildTensorExp(op, def->getOperand(1));
+    const auto x = buildTensorExp(op, def->getOperand(0));
+    const auto y = buildTensorExp(op, def->getOperand(1));
     if (x.has_value() && y.has_value()) {
-      unsigned e0 = *x;
-      unsigned e1 = *y;
+      const ExprId e0 = *x;
+      const ExprId e1 = *y;
       if (isa<arith::MulFOp>(def))
         return addExp(kMulF, e0, e1);
       if (isa<complex::MulOp>(def))
@@ -1185,12 +1192,12 @@ std::optional<unsigned> Merger::buildTensorExp(linalg::GenericOp op, Value v) {
   }
   // Construct ternary operations if subexpressions can be built.
   if (def->getNumOperands() == 3) {
-    auto x = buildTensorExp(op, def->getOperand(0));
-    auto y = buildTensorExp(op, def->getOperand(1));
-    auto z = buildTensorExp(op, def->getOperand(2));
+    const auto x = buildTensorExp(op, def->getOperand(0));
+    const auto y = buildTensorExp(op, def->getOperand(1));
+    const auto z = buildTensorExp(op, def->getOperand(2));
     if (x.has_value() && y.has_value() && z.has_value()) {
-      unsigned e0 = *x;
-      unsigned e1 = *y;
+      const ExprId e0 = *x;
+      const ExprId e1 = *y;
       if (auto redop = dyn_cast<sparse_tensor::ReduceOp>(def)) {
         if (isAdmissibleBranch(redop, redop.getRegion()))
           return addExp(kReduce, e0, e1, Value(), def);
@@ -1211,7 +1218,7 @@ static Value insertYieldOp(RewriterBase &rewriter, Location loc, Region &region,
   YieldOp clonedYield = cast<YieldOp>(clonedBlock.getTerminator());
   // Merge cloned block and return yield value.
   Operation *placeholder = rewriter.create<arith::ConstantIndexOp>(loc, 0);
-  rewriter.mergeBlockBefore(&tmpRegion.front(), placeholder, vals);
+  rewriter.inlineBlockBefore(&tmpRegion.front(), placeholder, vals);
   Value val = clonedYield.getResult();
   rewriter.eraseOp(clonedYield);
   rewriter.eraseOp(placeholder);
@@ -1246,13 +1253,13 @@ static Value buildBinaryOverlap(RewriterBase &rewriter, Location loc,
   return insertYieldOp(rewriter, loc, overlapRegion, {v0, v1});
 }
 
-Value Merger::buildExp(RewriterBase &rewriter, Location loc, unsigned e,
-                       Value v0, Value v1) {
+Value Merger::buildExp(RewriterBase &rewriter, Location loc, ExprId e, Value v0,
+                       Value v1) const {
   switch (tensorExps[e].kind) {
   // Leaf.
   case kTensor:
   case kInvariant:
-  case kIndex:
+  case kLoopVar:
     llvm_unreachable("unexpected non-op");
   // Unary operations.
   case kAbsF:
diff --git a/mlir/lib/Dialect/Tensor/IR/TensorInferTypeOpInterfaceImpl.cpp b/mlir/lib/Dialect/Tensor/IR/TensorInferTypeOpInterfaceImpl.cpp
index eb73a2c872956..ea087c1357aec 100644
--- a/mlir/lib/Dialect/Tensor/IR/TensorInferTypeOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Tensor/IR/TensorInferTypeOpInterfaceImpl.cpp
@@ -93,7 +93,7 @@ static OpFoldResult getExpandedOutputDimFromInputShape(
                         .cast<AffineDimExpr>()
                         .getPosition();
   int64_t linearizedStaticDim = 1;
-  for (auto &d :
+  for (auto d :
        llvm::enumerate(dstStaticShape.slice(startPos, endPos - startPos + 1))) {
     if (d.index() + startPos == static_cast<unsigned>(dimIndex))
       continue;
diff --git a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
index 22531338d30c6..f2da1088eb04d 100644
--- a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
+++ b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
@@ -81,11 +81,7 @@ FailureOr<Value> tensor::getOrCreateDestination(OpBuilder &b, Location loc,
   if (!tensorType.hasStaticShape()) {
     // Dynamic shape: Query ReifyRankedShapedTypeOpInterface.
     ReifiedRankedShapedTypeDims reifiedShapes;
-    ReifyRankedShapedTypeOpInterface reifyShapedTypeInterface =
-        dyn_cast<ReifyRankedShapedTypeOpInterface>(opResult.getDefiningOp());
-    if (!reifyShapedTypeInterface)
-      return failure();
-    if (failed(reifyShapedTypeInterface.reifyResultShapes(b, reifiedShapes)))
+    if (failed(reifyResultShapes(b, opResult.getDefiningOp(), reifiedShapes)))
       return failure();
     mixedSizes = reifiedShapes[opResult.getResultNumber()];
   } else {
@@ -1237,7 +1233,7 @@ int64_t ExpandShapeOp::getCorrespondingSourceDim(int64_t resultDim) {
   assert(resultDim >= 0 && resultDim < getResultType().getRank() &&
          "invalid resultDim");
   for (const auto &it : llvm::enumerate(getReassociationIndices()))
-    if (llvm::find(it.value(), resultDim) != it.value().end())
+    if (llvm::is_contained(it.value(), resultDim))
       return it.index();
   llvm_unreachable("could not find reassociation group");
 }
@@ -2205,8 +2201,13 @@ LogicalResult InsertSliceOp::reifyResultShapes(
     OpBuilder &builder, ReifiedRankedShapedTypeDims &reifiedReturnShapes) {
   reifiedReturnShapes.resize(1, SmallVector<OpFoldResult>(getType().getRank()));
   for (auto dim : llvm::seq<int64_t>(0, getType().getRank())) {
-    reifiedReturnShapes[0][dim] =
-        builder.createOrFold<tensor::DimOp>(getLoc(), getDest(), dim);
+    if (getType().isDynamicDim(dim)) {
+      reifiedReturnShapes[0][dim] =
+          builder.createOrFold<tensor::DimOp>(getLoc(), getDest(), dim);
+    } else {
+      reifiedReturnShapes[0][dim] =
+          builder.getIndexAttr(getType().getDimSize(dim));
+    }
   }
   return success();
 }
@@ -2772,7 +2773,7 @@ struct FoldOrthogonalPaddings : public OpRewritePattern<PadOp> {
     // zero-offset and zero-padding tensor::ExtractSliceOp, tensor::PadOp pair
     // exists.
     SmallVector<OpFoldResult> newOffsets(rank, rewriter.getIndexAttr(0));
-    for (auto &en : enumerate(newOffsets)) {
+    for (auto en : enumerate(newOffsets)) {
       OpFoldResult innerOffset = innerSliceOp.getMixedOffsets()[en.index()];
       OpFoldResult outerOffset = outerSliceOp.getMixedOffsets()[en.index()];
       if (!innerDims.test(en.index()) &&
@@ -2795,7 +2796,7 @@ struct FoldOrthogonalPaddings : public OpRewritePattern<PadOp> {
     // tensor::ExtractSliceOp does not match the size of the padded dimension.
     // Otherwise, take the size of the inner tensor::ExtractSliceOp.
     SmallVector<OpFoldResult> newSizes = innerSliceOp.getMixedSizes();
-    for (auto &en : enumerate(newSizes)) {
+    for (auto en : enumerate(newSizes)) {
       if (!outerDims.test(en.index()))
         continue;
       OpFoldResult sliceSize = innerSliceOp.getMixedSizes()[en.index()];
@@ -2812,7 +2813,7 @@ struct FoldOrthogonalPaddings : public OpRewritePattern<PadOp> {
 
     // Combine the high paddings of the two tensor::PadOps.
     SmallVector<OpFoldResult> newHighPad(rank, rewriter.getIndexAttr(0));
-    for (auto &en : enumerate(newHighPad)) {
+    for (auto en : enumerate(newHighPad)) {
       if (innerDims.test(en.index()))
         newHighPad[en.index()] = padOp.getMixedHighPad()[en.index()];
       if (outerDims.test(en.index()))
@@ -3154,9 +3155,15 @@ reifyResultShapesImpl(OpTy op, OpBuilder &builder,
                 "applies to only pack or unpack operations");
   int64_t destRank = op.getDestRank();
   reifiedReturnShapes.resize(1, SmallVector<OpFoldResult>(destRank));
+  ShapedType resultType = op.getResult().getType().template cast<ShapedType>();
   for (auto dim : llvm::seq<int64_t>(0, destRank)) {
-    reifiedReturnShapes[0][dim] =
-        builder.createOrFold<tensor::DimOp>(op.getLoc(), op.getDest(), dim);
+    if (resultType.isDynamicDim(dim)) {
+      reifiedReturnShapes[0][dim] =
+          builder.createOrFold<tensor::DimOp>(op.getLoc(), op.getDest(), dim);
+    } else {
+      reifiedReturnShapes[0][dim] =
+          builder.getIndexAttr(resultType.getDimSize(dim));
+    }
   }
   return success();
 }
@@ -3422,22 +3429,16 @@ SmallVector<int64_t> PackOp::getStaticTiles() {
   return getStaticTilesImpl(*this);
 }
 
-/// Check if we have enough static information to catch undefined behavior when
-/// the tile size does not divide perfectly the dimension of the input tensor.
-static bool
-areNotFullTiles(ArrayRef<int64_t> inputShape,
-                DenseMap<int64_t, OpFoldResult> const &dimAndTileMapping) {
-  int64_t rank = inputShape.size();
-  for (int64_t dim = 0; dim < rank; dim++) {
-    if (ShapedType::isDynamic(inputShape[dim]))
-      continue;
-    auto it = dimAndTileMapping.find(dim);
-    if (it == dimAndTileMapping.end())
+bool PackOp::requirePaddingValue(ArrayRef<int64_t> inputShape,
+                                 ArrayRef<int64_t> innerDimsPos,
+                                 ArrayRef<OpFoldResult> innerTiles) {
+  for (auto [pos, tileSize] : llvm::zip_equal(innerDimsPos, innerTiles)) {
+    if (ShapedType::isDynamic(inputShape[pos]))
       continue;
-    std::optional<int64_t> constantTile = getConstantIntValue(it->second);
+    std::optional<int64_t> constantTile = getConstantIntValue(tileSize);
     if (!constantTile)
       continue;
-    if (inputShape[dim] % (*constantTile) != 0)
+    if (inputShape[pos] % (*constantTile) != 0)
       return true;
   }
   return false;
@@ -3458,9 +3459,9 @@ LogicalResult PackOp::verify() {
            << " but got: " << paddingValue.getType();
   }
 
-  auto dimAndTileMapping = getDimAndTileMapping();
   if (!paddingValue &&
-      areNotFullTiles(getSourceType().getShape(), dimAndTileMapping)) {
+      requirePaddingValue(getSourceType().getShape(), getInnerDimsPos(),
+                          getMixedTiles())) {
     return emitOpError("invalid tile factor provided. Only full tiles are "
                        "supported when padding_value is not set");
   }
diff --git a/mlir/lib/Dialect/Tensor/IR/TensorTilingInterfaceImpl.cpp b/mlir/lib/Dialect/Tensor/IR/TensorTilingInterfaceImpl.cpp
index 457f261ef8837..0faa29ade8047 100644
--- a/mlir/lib/Dialect/Tensor/IR/TensorTilingInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Tensor/IR/TensorTilingInterfaceImpl.cpp
@@ -34,10 +34,7 @@ struct PadOpTiling : public TilingInterface::ExternalModel<PadOpTiling, PadOp> {
 
   SmallVector<Range> getIterationDomain(Operation *op, OpBuilder &b) const {
     ReifiedRankedShapedTypeDims reifiedShapes;
-    ReifyRankedShapedTypeOpInterface reifyShapedTypeInterface =
-        dyn_cast<ReifyRankedShapedTypeOpInterface>(op);
-    (void)reifyShapedTypeInterface.reifyResultShapes(b, reifiedShapes);
-
+    (void)reifyResultShapes(b, op, reifiedShapes);
     Location loc = op->getLoc();
     Value zero = b.create<arith::ConstantIndexOp>(loc, 0);
     Value one = b.create<arith::ConstantIndexOp>(loc, 1);
@@ -49,15 +46,15 @@ struct PadOpTiling : public TilingInterface::ExternalModel<PadOpTiling, PadOp> {
     return loopRanges;
   }
 
-  SmallVector<Operation *>
+  FailureOr<TilingResult>
   getTiledImplementation(Operation *op, OpBuilder &b,
                          ArrayRef<OpFoldResult> offsets,
                          ArrayRef<OpFoldResult> sizes) const {
-    Operation *result =
+    FailureOr<TilingResult> result =
         tensor::bubbleUpPadSlice(b, cast<PadOp>(op), offsets, sizes);
-    if (!result)
-      return {};
-    return {result};
+    if (failed(result))
+      return failure();
+    return result.value();
   }
 
   LogicalResult
@@ -84,7 +81,7 @@ static SmallVector<Range> getPackUnPackIterationDomain(OpTy op,
   Value zero = builder.create<arith::ConstantIndexOp>(loc, 0);
   Value one = builder.create<arith::ConstantIndexOp>(loc, 1);
   ReifiedRankedShapedTypeDims resultShape;
-  (void)op.reifyResultShapes(builder, resultShape);
+  (void)reifyResultShapes(builder, op, resultShape);
   SmallVector<Range> loopBounds(rank);
   for (auto dim : llvm::seq<int64_t>(0, rank)) {
     loopBounds[dim].offset = zero;
@@ -120,7 +117,7 @@ struct PackOpTiling
     return getPackUnPackIterationDomain<PackOp>(cast<PackOp>(op), b);
   }
 
-  SmallVector<Operation *>
+  FailureOr<TilingResult>
   getTiledImplementation(Operation *op, OpBuilder &b,
                          ArrayRef<OpFoldResult> offsets,
                          ArrayRef<OpFoldResult> sizes) const {
@@ -195,7 +192,8 @@ struct PackOpTiling
     Operation *tiledPackOp = b.create<PackOp>(
         loc, TypeRange{extractSlice.getType()}, tiledOperands, op->getAttrs());
 
-    return {tiledPackOp};
+    return TilingResult{{tiledPackOp},
+                        SmallVector<Value>(tiledPackOp->getResults())};
   }
 
   LogicalResult
@@ -216,7 +214,7 @@ struct PackOpTiling
     resultOffsets.append(outputRank - inputRank, zeroAttr);
 
     ReifiedRankedShapedTypeDims outputShape;
-    (void)packOp.reifyResultShapes(b, outputShape);
+    (void)reifyResultShapes(b, packOp, outputShape);
     resultSizes.assign(sizes.begin(), sizes.end());
     for (auto dataTileDim : llvm::seq<unsigned>(inputRank, outputRank))
       resultSizes.push_back(outputShape[0][dataTileDim]);
@@ -356,7 +354,7 @@ struct UnPackOpTiling
   /// (3, 7). In this context, the tiled unpack produces a (3 * n) elements
   /// because there are 3 rows in total. Follow by a tensor.extract_slice op, we
   /// can get the actual result.
-  SmallVector<Operation *>
+  FailureOr<TilingResult>
   getTiledImplementation(Operation *op, OpBuilder &b,
                          ArrayRef<OpFoldResult> offsets,
                          ArrayRef<OpFoldResult> sizes) const {
@@ -415,12 +413,13 @@ struct UnPackOpTiling
         loc, TypeRange{sliceDest.getType()}, tiledOperands, op->getAttrs());
 
     if (isPerfectTilingCase)
-      return {tiledUnpackOp};
+      return TilingResult{{tiledUnpackOp},
+                          SmallVector<Value>(tiledUnpackOp->getResults())};
 
-    Operation *extractSlice =
+    auto extractSlice =
         b.create<ExtractSliceOp>(loc, tiledUnpackOp->getResult(0),
                                  resultOffsetsFromDest, sizes, destStrides);
-    return {tiledUnpackOp, extractSlice};
+    return TilingResult{{tiledUnpackOp}, {extractSlice.getResult()}};
   }
 
   LogicalResult
@@ -434,26 +433,29 @@ struct UnPackOpTiling
     return success();
   }
 
-  FailureOr<Value> generateResultTileValue(Operation *op, OpBuilder &b,
-                                           unsigned resultNumber,
-                                           ArrayRef<OpFoldResult> offsets,
-                                           ArrayRef<OpFoldResult> sizes) const {
-    return getTiledImplementation(op, b, offsets, sizes)
-        .back()
-        ->getResult(resultNumber);
+  FailureOr<TilingResult>
+  generateResultTileValue(Operation *op, OpBuilder &b, unsigned resultNumber,
+                          ArrayRef<OpFoldResult> offsets,
+                          ArrayRef<OpFoldResult> sizes) const {
+    FailureOr<TilingResult> tilingResult =
+        getTiledImplementation(op, b, offsets, sizes);
+    if (failed(tilingResult))
+      return failure();
+    return tilingResult.value();
   }
 };
 
 } // namespace
 
-Operation *tensor::bubbleUpPadSlice(OpBuilder &b, tensor::PadOp padOp,
-                                    ArrayRef<OpFoldResult> offsets,
-                                    ArrayRef<OpFoldResult> sizes,
-                                    bool generateZeroSliceGuard) {
+FailureOr<TilingResult> tensor::bubbleUpPadSlice(OpBuilder &b,
+                                                 tensor::PadOp padOp,
+                                                 ArrayRef<OpFoldResult> offsets,
+                                                 ArrayRef<OpFoldResult> sizes,
+                                                 bool generateZeroSliceGuard) {
   // Only constant padding value supported.
   Value padValue = padOp.getConstantPaddingValue();
   if (!padValue)
-    return nullptr;
+    return failure();
 
   // Helper variables and functions for various arithmetic operations. These
   // are used extensively for computing new offset/length and padding values.
@@ -462,42 +464,29 @@ Operation *tensor::bubbleUpPadSlice(OpBuilder &b, tensor::PadOp padOp,
   bindDims(b.getContext(), dim0, dim1);
   // Add two integers.
   auto addMap = AffineMap::get(2, 0, {dim0 + dim1});
-  auto add = [&](Value v1, Value v2) {
-    return b.createOrFold<AffineApplyOp>(loc, addMap, ValueRange{v1, v2});
+  auto add = [&](OpFoldResult v1, OpFoldResult v2) {
+    return makeComposedFoldedAffineApply(b, loc, addMap, {v1, v2});
   };
   // Subtract two integers.
   auto subMap = AffineMap::get(2, 0, {dim0 - dim1});
-  auto sub = [&](Value v1, Value v2) {
-    return b.createOrFold<AffineApplyOp>(loc, subMap, ValueRange{v1, v2});
+  auto sub = [&](OpFoldResult v1, OpFoldResult v2) {
+    return makeComposedFoldedAffineApply(b, loc, subMap, {v1, v2});
   };
   // Take the minimum of two integers.
   auto idMap = AffineMap::getMultiDimIdentityMap(2, b.getContext());
-  auto min = [&](Value v1, Value v2) {
-    return b.createOrFold<AffineMinOp>(loc, idMap, ValueRange{v1, v2});
+  auto min = [&](OpFoldResult v1, OpFoldResult v2) {
+    return makeComposedFoldedAffineMin(b, loc, idMap, {v1, v2});
   };
   // Take the maximum of two integers.
-  auto max = [&](Value v1, Value v2) {
-    return b.createOrFold<AffineMaxOp>(loc, idMap, ValueRange{v1, v2});
+  auto max = [&](OpFoldResult v1, OpFoldResult v2) {
+    return makeComposedFoldedAffineMax(b, loc, idMap, {v1, v2});
   };
   // Zero index-typed integer.
-  auto zero = b.create<arith::ConstantIndexOp>(loc, 0);
-
-  // Helper function for filling static/dynamic low/high padding indices
-  // vectors of PadOp.
-  auto appendIndex = [&](Value val, SmallVector<Value> &dynIndices,
-                         SmallVector<int64_t> &staticIndices) {
-    if (auto constInt = getConstantIntValue(val)) {
-      staticIndices.push_back(*constInt);
-    } else {
-      staticIndices.push_back(ShapedType::kDynamic);
-      dynIndices.push_back(val);
-    }
-  };
+  OpFoldResult zero = b.getIndexAttr(0);
 
   // Compute new offsets, lengths, low padding, high padding.
   SmallVector<OpFoldResult> newOffsets, newLengths, newStrides;
-  SmallVector<Value> newLows, newHighs;
-  SmallVector<int64_t> staticNewLows, staticNewHighs;
+  SmallVector<OpFoldResult> newLows, newHighs;
   // Set to true if the original data source is not read at all.
   bool hasZeroLen = false;
   // Same as hasZeroLen, but for dynamic dimension sizes. This condition
@@ -506,23 +495,22 @@ Operation *tensor::bubbleUpPadSlice(OpBuilder &b, tensor::PadOp padOp,
 
   int64_t rank = padOp.getSourceType().getRank();
   for (unsigned dim = 0; dim < rank; ++dim) {
-    auto low =
-        getValueOrCreateConstantIndexOp(b, loc, padOp.getMixedLowPad()[dim]);
-    bool hasLowPad = getConstantIntValue(low) != static_cast<int64_t>(0);
-    auto high =
-        getValueOrCreateConstantIndexOp(b, loc, padOp.getMixedHighPad()[dim]);
-    bool hasHighPad = getConstantIntValue(high) != static_cast<int64_t>(0);
-    auto offset = getValueOrCreateConstantIndexOp(b, loc, offsets[dim]);
-    auto length = getValueOrCreateConstantIndexOp(b, loc, sizes[dim]);
-    auto srcSize = b.createOrFold<tensor::DimOp>(loc, padOp.getSource(), dim);
+    auto low = padOp.getMixedLowPad()[dim];
+    bool hasLowPad = !isConstantIntValue(low, 0);
+    auto high = padOp.getMixedHighPad()[dim];
+    bool hasHighPad = !isConstantIntValue(high, 0);
+    auto offset = offsets[dim];
+    auto length = sizes[dim];
+    auto srcSize =
+        tensor::createDimValue(b, loc, padOp.getSource(), dim).value();
 
     // The new amount of low padding is `low - offset`. Except for the case
     // where none of the low padding is read. In that case, the new amount of
     // low padding is zero.
     //
     // Optimization: If low = 0, then newLow = 0.
-    Value newLow = hasLowPad ? max(zero, sub(low, offset)) : zero;
-    appendIndex(newLow, newLows, staticNewLows);
+    OpFoldResult newLow = hasLowPad ? max(zero, sub(low, offset)) : zero;
+    newLows.push_back(newLow);
 
     // Start reading the data from position `offset - low`. Since the original
     // read may have started in the low padding zone, this value could be
@@ -536,9 +524,10 @@ Operation *tensor::bubbleUpPadSlice(OpBuilder &b, tensor::PadOp padOp,
     // no data from the source.)
     //
     // Optimization: If low = 0, then the formula can be simplified.
-    Value newOffset = hasLowPad ? min(max(sub(offset, low), zero), srcSize)
-                                : min(offset, srcSize);
-    newOffsets.push_back(getAsOpFoldResult(newOffset));
+    OpFoldResult newOffset = hasLowPad
+                                 ? min(max(sub(offset, low), zero), srcSize)
+                                 : min(offset, srcSize);
+    newOffsets.push_back(newOffset);
 
     // The original ExtractSliceOp was reading until position `offset +
     // length`. Therefore, the corresponding position within the source tensor
@@ -559,19 +548,21 @@ Operation *tensor::bubbleUpPadSlice(OpBuilder &b, tensor::PadOp padOp,
     // The new ExtractSliceOp length is `endLoc - newOffset`.
     //
     // Optimization: If low = 0, then the formula can be simplified.
-    Value endLoc = hasLowPad
-                       ? min(max(add(sub(offset, low), length), zero), srcSize)
-                       : min(add(offset, length), srcSize);
-    Value newLength = sub(endLoc, newOffset);
-    newLengths.push_back(getAsOpFoldResult(newLength));
+    OpFoldResult endLoc =
+        hasLowPad ? min(max(add(sub(offset, low), length), zero), srcSize)
+                  : min(add(offset, length), srcSize);
+    OpFoldResult newLength = sub(endLoc, newOffset);
+    newLengths.push_back(newLength);
 
     // Check if newLength is zero. In that case, no SubTensorOp should be
     // executed.
-    if (auto newLengthInt = getConstantIntValue(newLength)) {
-      hasZeroLen |= *newLengthInt == 0;
-    } else {
-      Value check = b.create<arith::CmpIOp>(loc, arith::CmpIPredicate::eq,
-                                            newLength, zero);
+    if (isConstantIntValue(newLength, 0)) {
+      hasZeroLen = true;
+    } else if (!hasZeroLen) {
+      Value check = b.create<arith::CmpIOp>(
+          loc, arith::CmpIPredicate::eq,
+          getValueOrCreateConstantIndexOp(b, loc, newLength),
+          getValueOrCreateConstantIndexOp(b, loc, zero));
       dynHasZeroLenCond =
           dynHasZeroLenCond
               ? b.create<arith::OrIOp>(loc, check, dynHasZeroLenCond)
@@ -582,8 +573,9 @@ Operation *tensor::bubbleUpPadSlice(OpBuilder &b, tensor::PadOp padOp,
     // so that the result has the same length as the original ExtractSliceOp.
     // As an optimization, if the original high padding is zero, then the new
     // high padding must also be zero.
-    Value newHigh = hasHighPad ? sub(sub(length, newLength), newLow) : zero;
-    appendIndex(newHigh, newHighs, staticNewHighs);
+    OpFoldResult newHigh =
+        hasHighPad ? sub(sub(length, newLength), newLow) : zero;
+    newHighs.push_back(newHigh);
 
     // Only unit stride supported.
     newStrides.push_back(b.getIndexAttr(1));
@@ -597,7 +589,9 @@ Operation *tensor::bubbleUpPadSlice(OpBuilder &b, tensor::PadOp padOp,
       RankedTensorType::get(shape, padOp.getResultType().getElementType());
 
   // Insert cast to ensure that types match. (May be folded away.)
-  auto castResult = [&](Value val) -> Operation * {
+  auto castResult = [&](Value val) -> Value {
+    if (resultType == val.getType())
+      return val;
     return b.create<tensor::CastOp>(loc, resultType, val);
   };
 
@@ -611,47 +605,54 @@ Operation *tensor::bubbleUpPadSlice(OpBuilder &b, tensor::PadOp padOp,
         [&](OpBuilder &builder, Location gLoc, ValueRange indices) {
           builder.create<tensor::YieldOp>(gLoc, padValue);
         });
-    return castResult(generateOp);
+    return generateOp;
   };
 
   // Emit a SliceOp and a PadOp. Should not be used in cases where
   // the result shape of the new SliceOp has a zero dimension.
   auto createPadOfExtractSlice = [&]() {
     // Create pad(extract_slice(x)).
-    auto newSliceOp = b.create<tensor::ExtractSliceOp>(
+    Value newSliceOp = b.create<tensor::ExtractSliceOp>(
         loc, padOp.getSource(), newOffsets, newLengths, newStrides);
-    auto newPadOp = b.create<PadOp>(loc, newSliceOp, staticNewLows,
-                                    staticNewHighs, newLows, newHighs);
+    auto newPadOp = b.create<PadOp>(loc, Type(), newSliceOp, newLows, newHighs);
 
     // Copy region to new PadOp.
     IRMapping bvm;
     padOp.getRegion().cloneInto(&newPadOp.getRegion(), bvm);
 
     // Cast result and return.
-    return castResult(newPadOp);
+    return newPadOp;
   };
 
   // Rewrite extract_slice(pad(x)) into a GenerateOp it is statically known that
   // the original data source x is not used.
-  if (hasZeroLen)
-    return createGenerateOp();
+  if (hasZeroLen) {
+    Operation *generateOp = createGenerateOp();
+    return TilingResult{{generateOp}, {castResult(generateOp->getResult(0))}};
+  }
 
   // If there are dynamic dimensions: Generate an scf.if check to avoid
   // creating SliceOps with result dimensions of size 0 at runtime.
   if (generateZeroSliceGuard && dynHasZeroLenCond) {
+    Operation *thenOp;
+    Operation *elseOp;
     auto result = b.create<scf::IfOp>(
         loc, dynHasZeroLenCond,
         /*thenBuilder=*/
         [&](OpBuilder &b, Location loc) {
-          b.create<scf::YieldOp>(loc, createGenerateOp()->getResult(0));
+          thenOp = createGenerateOp();
+          b.create<scf::YieldOp>(loc, castResult(thenOp->getResult(0)));
         },
         /*elseBuilder=*/
         [&](OpBuilder &b, Location loc) {
-          b.create<scf::YieldOp>(loc, createPadOfExtractSlice()->getResult(0));
+          elseOp = createPadOfExtractSlice();
+          b.create<scf::YieldOp>(loc, castResult(elseOp->getResult(0)));
         });
-    return result;
+    return TilingResult{{result}, SmallVector<Value>(result->getResults())};
   }
-  return createPadOfExtractSlice();
+
+  Operation *newPadOp = createPadOfExtractSlice();
+  return TilingResult{{newPadOp}, {castResult(newPadOp->getResult(0))}};
 }
 
 void mlir::tensor::registerTilingInterfaceExternalModels(
diff --git a/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp
index fdeb1e490033d..43dedf1c4ce06 100644
--- a/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -762,7 +762,7 @@ struct InsertSliceOpInterface
           return isConstantIntValue(ofr, 0);
         });
     bool sizesMatchDestSizes = llvm::all_of(
-        llvm::enumerate(insertSliceOp.getMixedSizes()), [&](auto &it) {
+        llvm::enumerate(insertSliceOp.getMixedSizes()), [&](const auto &it) {
           return getConstantIntValue(it.value()) ==
                  destType.getDimSize(it.index());
         });
diff --git a/mlir/lib/Dialect/Tensor/Transforms/EmptyOpPatterns.cpp b/mlir/lib/Dialect/Tensor/Transforms/EmptyOpPatterns.cpp
index f9512fd49e695..99679bc9b1378 100644
--- a/mlir/lib/Dialect/Tensor/Transforms/EmptyOpPatterns.cpp
+++ b/mlir/lib/Dialect/Tensor/Transforms/EmptyOpPatterns.cpp
@@ -26,10 +26,7 @@ struct FoldEmptyTensorWithReshapeOp : public OpRewritePattern<ReshapeOp> {
       return failure();
     Location loc = reshapeOp.getLoc();
     ReifiedRankedShapedTypeDims resultShapes;
-    ReifyRankedShapedTypeOpInterface reifyShapedTypeInterface =
-        cast<ReifyRankedShapedTypeOpInterface>(reshapeOp.getOperation());
-    if (failed(reifyShapedTypeInterface.reifyResultShapes(rewriter,
-                                                          resultShapes)) ||
+    if (failed(reifyResultShapes(rewriter, reshapeOp, resultShapes)) ||
         !llvm::hasSingleElement(resultShapes))
       return failure();
     // TODO: Do not drop tensor type encoding.
diff --git a/mlir/lib/Dialect/Tensor/Transforms/ExtractSliceFromReshapeUtils.cpp b/mlir/lib/Dialect/Tensor/Transforms/ExtractSliceFromReshapeUtils.cpp
index 0ef8729347df8..f1ad357098c55 100644
--- a/mlir/lib/Dialect/Tensor/Transforms/ExtractSliceFromReshapeUtils.cpp
+++ b/mlir/lib/Dialect/Tensor/Transforms/ExtractSliceFromReshapeUtils.cpp
@@ -112,9 +112,7 @@ tensor::ExtractSliceFromCollapseHelper::create(OpBuilder &b,
   // Materialize the output shape of the collapse_shape operation. This will
   // create IR describing the output shape in terms of the input shape.
   ReifiedRankedShapedTypeDims reifiedShapes;
-  ReifyRankedShapedTypeOpInterface reifyShapedTypeInterface =
-      dyn_cast<ReifyRankedShapedTypeOpInterface>(op.getOperation());
-  if (failed(reifyShapedTypeInterface.reifyResultShapes(b, reifiedShapes)))
+  if (failed(reifyResultShapes(b, op, reifiedShapes)))
     return failure();
   SmallVector<OpFoldResult> &collapseShapeOutputShape = reifiedShapes[0];
   SmallVector<ReassociationIndices> reassociationIndices =
diff --git a/mlir/lib/Dialect/Tensor/Transforms/SwapExtractSliceWithProducerPatterns.cpp b/mlir/lib/Dialect/Tensor/Transforms/SwapExtractSliceWithProducerPatterns.cpp
index 65176ed7b9e74..40d79c2053817 100644
--- a/mlir/lib/Dialect/Tensor/Transforms/SwapExtractSliceWithProducerPatterns.cpp
+++ b/mlir/lib/Dialect/Tensor/Transforms/SwapExtractSliceWithProducerPatterns.cpp
@@ -20,7 +20,7 @@
 
 using namespace mlir;
 
-FailureOr<Value> tensor::replaceExtractSliceWithTiledProducer(
+FailureOr<TilingResult> tensor::replaceExtractSliceWithTiledProducer(
     OpBuilder &builder, tensor::ExtractSliceOp sliceOp, OpResult producer) {
   auto producerOp = dyn_cast<TilingInterface>(producer.getOwner());
   if (!producerOp)
@@ -32,7 +32,7 @@ FailureOr<Value> tensor::replaceExtractSliceWithTiledProducer(
       }))
     return failure();
 
-  FailureOr<Value> tiledResult = producerOp.generateResultTileValue(
+  FailureOr<TilingResult> tiledResult = producerOp.generateResultTileValue(
       builder, producer.getResultNumber(), sliceOp.getMixedOffsets(),
       sliceOp.getMixedSizes());
   if (failed(tiledResult))
diff --git a/mlir/lib/Dialect/Tensor/Utils/Utils.cpp b/mlir/lib/Dialect/Tensor/Utils/Utils.cpp
index ece3e72af2813..a5847250fa915 100644
--- a/mlir/lib/Dialect/Tensor/Utils/Utils.cpp
+++ b/mlir/lib/Dialect/Tensor/Utils/Utils.cpp
@@ -52,6 +52,20 @@ SmallVector<Value> mlir::tensor::createDynamicDimValues(OpBuilder &b,
   return dynamicDims;
 }
 
+FailureOr<OpFoldResult> mlir::tensor::createDimValue(OpBuilder &b, Location loc,
+                                                     Value rankedTensor,
+                                                     int64_t dim) {
+  auto tensorTy = rankedTensor.getType().dyn_cast<RankedTensorType>();
+  if (!tensorTy)
+    return failure();
+  auto shape = tensorTy.getShape();
+  if (dim >= static_cast<int64_t>(shape.size()))
+    return failure();
+  if (ShapedType::isDynamic(shape[dim]))
+    return OpFoldResult(b.createOrFold<tensor::DimOp>(loc, rankedTensor, dim));
+  return OpFoldResult(b.getIndexAttr(shape[dim]));
+}
+
 SmallVector<OpFoldResult>
 mlir::tensor::createDimValues(OpBuilder &b, Location loc, Value rankedTensor) {
   auto tensorTy = rankedTensor.getType().cast<RankedTensorType>();
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index c3b161ac35032..d7bb6d0bddbf6 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -409,6 +409,16 @@ LogicalResult tosa::RFFT2dOp::inferReturnTypeComponents(
 
   inferredReturnShapes.push_back(ShapedTypeComponents(outputShape));
   inferredReturnShapes.push_back(ShapedTypeComponents(outputShape));
+
+  return success();
+}
+
+LogicalResult tosa::FFT2dOp::inferReturnTypeComponents(
+    MLIRContext *context, ::std::optional<Location> location,
+    ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
+    SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
+  inferredReturnShapes.push_back(ShapedTypeComponents(operands.getShape(0)));
+  inferredReturnShapes.push_back(ShapedTypeComponents(operands.getShape(1)));
   return success();
 }
 
diff --git a/mlir/lib/Dialect/Transform/IR/TransformInterfaces.cpp b/mlir/lib/Dialect/Transform/IR/TransformInterfaces.cpp
index ef5bfd2a85aeb..7afbed33c30a8 100644
--- a/mlir/lib/Dialect/Transform/IR/TransformInterfaces.cpp
+++ b/mlir/lib/Dialect/Transform/IR/TransformInterfaces.cpp
@@ -339,14 +339,7 @@ transform::TransformState::replacePayloadOp(Operation *op,
   }
 
   // Otherwise, replace the pointed-to object of all handles while preserving
-  // their relative order.
-  if (op->getNumResults() != replacement->getNumResults()) {
-    return emitError(op->getLoc())
-           << "cannot replace an op with another op producing a different "
-              "number of results while tracking handles";
-  }
-
-  // Replace the mapped operation if present.
+  // their relative order. First, replace the mapped operation if present.
   for (Value handle : opHandles) {
     Mappings &mappings = getMapping(handle);
     auto it = mappings.direct.find(handle);
@@ -362,9 +355,21 @@ transform::TransformState::replacePayloadOp(Operation *op,
     mappings.reverse[replacement].push_back(handle);
   }
 
-  // Replace the mapped results of the operation.
-  for (auto [origResult, replacementResult, handleList] : llvm::zip(
-           op->getResults(), replacement->getResults(), resultValueHandles)) {
+  // Second, replace the mapped results of the operation.
+  for (auto [origResult, handleList] :
+       llvm::zip(op->getResults(), resultValueHandles)) {
+    // No handles to the value, skip even if there is no replacement.
+    if (handleList.empty())
+      continue;
+
+    unsigned resultNumber = origResult.getResultNumber();
+    if (resultNumber >= replacement->getNumResults()) {
+      return emitError(op->getLoc())
+             << "cannot replace an op with another op producing less results "
+                "while tracking handles";
+    }
+
+    Value replacementResult = replacement->getResult(resultNumber);
     for (Value resultHandle : handleList) {
       Mappings &mappings = getMapping(resultHandle);
       auto it = mappings.values.find(resultHandle);
@@ -964,7 +969,7 @@ void transform::detail::setApplyToOneResults(
       continue;
     assert(transformOp->getNumResults() == partialResults.size() &&
            "expected as many partial results as op as results");
-    for (auto &[i, value] : llvm::enumerate(partialResults))
+    for (auto [i, value] : llvm::enumerate(partialResults))
       transposed[i].push_back(value);
   }
 
diff --git a/mlir/lib/Dialect/Transform/IR/TransformOps.cpp b/mlir/lib/Dialect/Transform/IR/TransformOps.cpp
index 21b3b2dd2045d..cc4382ac1e0a1 100644
--- a/mlir/lib/Dialect/Transform/IR/TransformOps.cpp
+++ b/mlir/lib/Dialect/Transform/IR/TransformOps.cpp
@@ -11,7 +11,6 @@
 #include "mlir/Dialect/Transform/IR/TransformDialect.h"
 #include "mlir/Dialect/Transform/IR/TransformInterfaces.h"
 #include "mlir/Dialect/Transform/IR/TransformTypes.h"
-#include "mlir/Dialect/Transform/IR/TransformUtils.h"
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/Interfaces/ControlFlowInterfaces.h"
 #include "mlir/Rewrite/FrozenRewritePatternSet.h"
@@ -114,7 +113,14 @@ LogicalResult PatternApplicatorExtension::findAllMatches(
   }
 
   PatternApplicator applicator(it->second);
-  transform::TrivialPatternRewriter rewriter(root->getContext());
+  // We want to discourage direct use of PatternRewriter in APIs but In this
+  // very specific case, an IRRewriter is not enough.
+  struct TrivialPatternRewriter : public PatternRewriter {
+  public:
+    explicit TrivialPatternRewriter(MLIRContext *context)
+        : PatternRewriter(context) {}
+  };
+  TrivialPatternRewriter rewriter(root->getContext());
   applicator.applyDefaultCostModel();
   root->walk([&](Operation *op) {
     if (succeeded(applicator.matchAndRewrite(op, rewriter)))
@@ -453,7 +459,7 @@ transform::GetConsumersOfResult::apply(transform::TransformResults &results,
 
 DiagnosedSilenceableFailure
 transform::GetDefiningOp::apply(transform::TransformResults &results,
-                              transform::TransformState &state) {
+                                transform::TransformState &state) {
   SmallVector<Operation *> definingOps;
   for (Value v : state.getPayloadValues(getTarget())) {
     if (v.isa<BlockArgument>()) {
diff --git a/mlir/lib/Dialect/Utils/IndexingUtils.cpp b/mlir/lib/Dialect/Utils/IndexingUtils.cpp
index 4563d74acfde4..eb86e0f782a78 100644
--- a/mlir/lib/Dialect/Utils/IndexingUtils.cpp
+++ b/mlir/lib/Dialect/Utils/IndexingUtils.cpp
@@ -11,27 +11,100 @@
 #include "mlir/IR/AffineExpr.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/MLIRContext.h"
+#include "llvm/ADT/STLExtras.h"
 
 #include <numeric>
 #include <optional>
 
 using namespace mlir;
 
-SmallVector<int64_t> mlir::computeStrides(ArrayRef<int64_t> sizes) {
-  SmallVector<int64_t> strides(sizes.size(), 1);
+template <typename ExprType>
+SmallVector<ExprType> computeSuffixProductImpl(ArrayRef<ExprType> sizes,
+                                               ExprType unit) {
+  if (sizes.empty())
+    return {};
+  SmallVector<ExprType> strides(sizes.size(), unit);
   for (int64_t r = strides.size() - 2; r >= 0; --r)
     strides[r] = strides[r + 1] * sizes[r + 1];
   return strides;
 }
 
-SmallVector<int64_t> mlir::computeElementwiseMul(ArrayRef<int64_t> v1,
-                                                 ArrayRef<int64_t> v2) {
-  SmallVector<int64_t> result;
-  for (auto it : llvm::zip(v1, v2))
+template <typename ExprType>
+SmallVector<ExprType> computeElementwiseMulImpl(ArrayRef<ExprType> v1,
+                                                ArrayRef<ExprType> v2) {
+  // Early exit if both are empty, let zip_equal fail if only 1 is empty.
+  if (v1.empty() && v2.empty())
+    return {};
+  SmallVector<ExprType> result;
+  for (auto it : llvm::zip_equal(v1, v2))
     result.push_back(std::get<0>(it) * std::get<1>(it));
   return result;
 }
 
+template <typename ExprType>
+ExprType linearizeImpl(ArrayRef<ExprType> offsets, ArrayRef<ExprType> basis,
+                       ExprType zero) {
+  assert(offsets.size() == basis.size());
+  ExprType linearIndex = zero;
+  for (unsigned idx = 0, e = basis.size(); idx < e; ++idx)
+    linearIndex = linearIndex + offsets[idx] * basis[idx];
+  return linearIndex;
+}
+
+template <typename ExprType, typename DivOpTy>
+SmallVector<ExprType> delinearizeImpl(ExprType linearIndex,
+                                      ArrayRef<ExprType> strides,
+                                      DivOpTy divOp) {
+  int64_t rank = strides.size();
+  SmallVector<ExprType> offsets(rank);
+  for (int64_t r = 0; r < rank; ++r) {
+    offsets[r] = divOp(linearIndex, strides[r]);
+    linearIndex = linearIndex % strides[r];
+  }
+  return offsets;
+}
+
+//===----------------------------------------------------------------------===//
+// Utils that operate on static integer values.
+//===----------------------------------------------------------------------===//
+
+SmallVector<int64_t> mlir::computeSuffixProduct(ArrayRef<int64_t> sizes) {
+  assert(llvm::all_of(sizes, [](int64_t s) { return s > 0; }) &&
+         "sizes must be nonnegative");
+  int64_t unit = 1;
+  return ::computeSuffixProductImpl(sizes, unit);
+}
+
+SmallVector<int64_t> mlir::computeElementwiseMul(ArrayRef<int64_t> v1,
+                                                 ArrayRef<int64_t> v2) {
+  return computeElementwiseMulImpl(v1, v2);
+}
+
+int64_t mlir::computeMaxLinearIndex(ArrayRef<int64_t> basis) {
+  assert(llvm::all_of(basis, [](int64_t s) { return s > 0; }) &&
+         "basis must be nonnegative");
+  if (basis.empty())
+    return 0;
+  return std::accumulate(basis.begin(), basis.end(), 1,
+                         std::multiplies<int64_t>());
+}
+
+int64_t mlir::linearize(ArrayRef<int64_t> offsets, ArrayRef<int64_t> basis) {
+  assert(llvm::all_of(basis, [](int64_t s) { return s > 0; }) &&
+         "basis must be nonnegative");
+  int64_t zero = 0;
+  return linearizeImpl(offsets, basis, zero);
+}
+
+SmallVector<int64_t> mlir::delinearize(int64_t linearIndex,
+                                       ArrayRef<int64_t> strides) {
+  assert(llvm::all_of(strides, [](int64_t s) { return s > 0; }) &&
+         "strides must be nonnegative");
+  return delinearizeImpl(linearIndex, strides,
+                         [](int64_t e1, int64_t e2) { return e1 / e2; });
+}
+
 std::optional<SmallVector<int64_t>>
 mlir::computeShapeRatio(ArrayRef<int64_t> shape, ArrayRef<int64_t> subShape) {
   if (shape.size() < subShape.size())
@@ -60,35 +133,67 @@ mlir::computeShapeRatio(ArrayRef<int64_t> shape, ArrayRef<int64_t> subShape) {
   return SmallVector<int64_t>{result.rbegin(), result.rend()};
 }
 
-int64_t mlir::linearize(ArrayRef<int64_t> offsets, ArrayRef<int64_t> basis) {
-  assert(offsets.size() == basis.size());
-  int64_t linearIndex = 0;
-  for (unsigned idx = 0, e = basis.size(); idx < e; ++idx)
-    linearIndex += offsets[idx] * basis[idx];
-  return linearIndex;
+//===----------------------------------------------------------------------===//
+// Utils that operate on AffineExpr.
+//===----------------------------------------------------------------------===//
+
+SmallVector<AffineExpr> mlir::computeSuffixProduct(ArrayRef<AffineExpr> sizes) {
+  if (sizes.empty())
+    return {};
+  AffineExpr unit = getAffineConstantExpr(1, sizes.front().getContext());
+  return ::computeSuffixProductImpl(sizes, unit);
 }
 
-llvm::SmallVector<int64_t> mlir::delinearize(ArrayRef<int64_t> sliceStrides,
-                                             int64_t index) {
-  int64_t rank = sliceStrides.size();
-  SmallVector<int64_t> vectorOffsets(rank);
-  for (int64_t r = 0; r < rank; ++r) {
-    assert(sliceStrides[r] > 0);
-    vectorOffsets[r] = index / sliceStrides[r];
-    index %= sliceStrides[r];
-  }
-  return vectorOffsets;
+SmallVector<AffineExpr> mlir::computeElementwiseMul(ArrayRef<AffineExpr> v1,
+                                                    ArrayRef<AffineExpr> v2) {
+  return computeElementwiseMulImpl(v1, v2);
 }
 
-int64_t mlir::computeMaxLinearIndex(ArrayRef<int64_t> basis) {
+AffineExpr mlir::computeMaxLinearIndex(MLIRContext *ctx,
+                                       ArrayRef<AffineExpr> basis) {
   if (basis.empty())
-    return 0;
-  return std::accumulate(basis.begin(), basis.end(), 1,
-                         std::multiplies<int64_t>());
+    return getAffineConstantExpr(0, ctx);
+  return std::accumulate(basis.begin(), basis.end(),
+                         getAffineConstantExpr(1, ctx),
+                         std::multiplies<AffineExpr>());
+}
+
+AffineExpr mlir::linearize(MLIRContext *ctx, ArrayRef<AffineExpr> offsets,
+                           ArrayRef<AffineExpr> basis) {
+  AffineExpr zero = getAffineConstantExpr(0, ctx);
+  return linearizeImpl(offsets, basis, zero);
 }
 
-llvm::SmallVector<int64_t>
+AffineExpr mlir::linearize(MLIRContext *ctx, ArrayRef<AffineExpr> offsets,
+                           ArrayRef<int64_t> basis) {
+  SmallVector<AffineExpr> basisExprs = llvm::to_vector(llvm::map_range(
+      basis, [ctx](int64_t v) { return getAffineConstantExpr(v, ctx); }));
+  return linearize(ctx, offsets, basisExprs);
+}
+
+SmallVector<AffineExpr> mlir::delinearize(AffineExpr linearIndex,
+                                          ArrayRef<AffineExpr> strides) {
+  return delinearizeImpl(
+      linearIndex, strides,
+      [](AffineExpr e1, AffineExpr e2) { return e1.floorDiv(e2); });
+}
+
+SmallVector<AffineExpr> mlir::delinearize(AffineExpr linearIndex,
+                                          ArrayRef<int64_t> strides) {
+  MLIRContext *ctx = linearIndex.getContext();
+  SmallVector<AffineExpr> basisExprs = llvm::to_vector(llvm::map_range(
+      strides, [ctx](int64_t v) { return getAffineConstantExpr(v, ctx); }));
+  return delinearize(linearIndex, ArrayRef<AffineExpr>{basisExprs});
+}
+
+//===----------------------------------------------------------------------===//
+// Permutation utils.
+//===----------------------------------------------------------------------===//
+
+SmallVector<int64_t>
 mlir::invertPermutationVector(ArrayRef<int64_t> permutation) {
+  assert(llvm::all_of(permutation, [](int64_t s) { return s >= 0; }) &&
+         "permutation must be non-negative");
   SmallVector<int64_t> inversion(permutation.size());
   for (const auto &pos : llvm::enumerate(permutation)) {
     inversion[pos.value()] = pos.index();
@@ -97,6 +202,8 @@ mlir::invertPermutationVector(ArrayRef<int64_t> permutation) {
 }
 
 bool mlir::isPermutationVector(ArrayRef<int64_t> interchange) {
+  assert(llvm::all_of(interchange, [](int64_t s) { return s >= 0; }) &&
+         "permutation must be non-negative");
   llvm::SmallDenseSet<int64_t, 4> seenVals;
   for (auto val : interchange) {
     if (seenVals.count(val))
@@ -106,9 +213,9 @@ bool mlir::isPermutationVector(ArrayRef<int64_t> interchange) {
   return seenVals.size() == interchange.size();
 }
 
-llvm::SmallVector<int64_t> mlir::getI64SubArray(ArrayAttr arrayAttr,
-                                                unsigned dropFront,
-                                                unsigned dropBack) {
+SmallVector<int64_t> mlir::getI64SubArray(ArrayAttr arrayAttr,
+                                          unsigned dropFront,
+                                          unsigned dropBack) {
   assert(arrayAttr.size() > dropFront + dropBack && "Out of bounds");
   auto range = arrayAttr.getAsRange<IntegerAttr>();
   SmallVector<int64_t> res;
@@ -118,26 +225,3 @@ llvm::SmallVector<int64_t> mlir::getI64SubArray(ArrayAttr arrayAttr,
     res.push_back((*it).getValue().getSExtValue());
   return res;
 }
-
-mlir::AffineExpr mlir::getLinearAffineExpr(ArrayRef<int64_t> basis,
-                                           mlir::Builder &b) {
-  AffineExpr resultExpr = b.getAffineDimExpr(0);
-  resultExpr = resultExpr * basis[0];
-  for (unsigned i = 1; i < basis.size(); i++)
-    resultExpr = resultExpr + b.getAffineDimExpr(i) * basis[i];
-  return resultExpr;
-}
-
-llvm::SmallVector<mlir::AffineExpr>
-mlir::getDelinearizedAffineExpr(mlir::ArrayRef<int64_t> strides, Builder &b) {
-  AffineExpr resultExpr = b.getAffineDimExpr(0);
-  int64_t rank = strides.size();
-  SmallVector<AffineExpr> vectorOffsets(rank);
-  vectorOffsets[0] = resultExpr.floorDiv(strides[0]);
-  resultExpr = resultExpr % strides[0];
-  for (unsigned i = 1; i < rank; i++) {
-    vectorOffsets[i] = resultExpr.floorDiv(strides[i]);
-    resultExpr = resultExpr % strides[i];
-  }
-  return vectorOffsets;
-}
diff --git a/mlir/lib/Dialect/Utils/StaticValueUtils.cpp b/mlir/lib/Dialect/Utils/StaticValueUtils.cpp
index 5a4b7ea4f9881..e646de95a76c9 100644
--- a/mlir/lib/Dialect/Utils/StaticValueUtils.cpp
+++ b/mlir/lib/Dialect/Utils/StaticValueUtils.cpp
@@ -89,7 +89,7 @@ OpFoldResult getAsOpFoldResult(Value val) {
 /// Given an array of values, try to extract a constant Attribute from each
 /// value. If this fails, return the original value.
 SmallVector<OpFoldResult> getAsOpFoldResult(ValueRange values) {
-  return llvm::to_vector<4>(
+  return llvm::to_vector(
       llvm::map_range(values, [](Value v) { return getAsOpFoldResult(v); }));
 }
 
@@ -102,6 +102,16 @@ SmallVector<OpFoldResult> getAsOpFoldResult(ArrayAttr arrayAttr) {
   return res;
 }
 
+OpFoldResult getAsIndexOpFoldResult(MLIRContext *ctx, int64_t val) {
+  return IntegerAttr::get(IndexType::get(ctx), val);
+}
+
+SmallVector<OpFoldResult> getAsIndexOpFoldResult(MLIRContext *ctx,
+                                                 ArrayRef<int64_t> values) {
+  return llvm::to_vector(llvm::map_range(
+      values, [ctx](int64_t v) { return getAsIndexOpFoldResult(ctx, v); }));
+}
+
 /// If ofr is a constant integer or an IntegerAttr, return the integer.
 std::optional<int64_t> getConstantIntValue(OpFoldResult ofr) {
   // Case 1: Check for Constant integer.
@@ -182,4 +192,40 @@ decomposeMixedValues(Builder &b,
   return {b.getI64ArrayAttr(staticValues), dynamicValues};
 }
 
+/// Helper to sort `values` according to matching `keys`.
+template <typename K, typename V>
+static SmallVector<V>
+getValuesSortedByKeyImpl(ArrayRef<K> keys, ArrayRef<V> values,
+                         llvm::function_ref<bool(K, K)> compare) {
+  if (keys.empty())
+    return SmallVector<V>{values};
+  assert(keys.size() == values.size() && "unexpected mismatching sizes");
+  auto indices = llvm::to_vector(llvm::seq<int64_t>(0, values.size()));
+  std::sort(indices.begin(), indices.end(),
+            [&](int64_t i, int64_t j) { return compare(keys[i], keys[j]); });
+  SmallVector<V> res;
+  res.reserve(values.size());
+  for (int64_t i = 0, e = indices.size(); i < e; ++i)
+    res.push_back(values[indices[i]]);
+  return res;
+}
+
+SmallVector<Value>
+getValuesSortedByKey(ArrayRef<Attribute> keys, ArrayRef<Value> values,
+                     llvm::function_ref<bool(Attribute, Attribute)> compare) {
+  return getValuesSortedByKeyImpl(keys, values, compare);
+}
+
+SmallVector<OpFoldResult>
+getValuesSortedByKey(ArrayRef<Attribute> keys, ArrayRef<OpFoldResult> values,
+                     llvm::function_ref<bool(Attribute, Attribute)> compare) {
+  return getValuesSortedByKeyImpl(keys, values, compare);
+}
+
+SmallVector<int64_t>
+getValuesSortedByKey(ArrayRef<Attribute> keys, ArrayRef<int64_t> values,
+                     llvm::function_ref<bool(Attribute, Attribute)> compare) {
+  return getValuesSortedByKeyImpl(keys, values, compare);
+}
+
 } // namespace mlir
diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index cabe7d7d95563..9796693b4b6cd 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -1558,7 +1558,7 @@ static Value foldExtractFromShapeCast(ExtractOp extractOp) {
         getDimReverse(shapeCastOp.getSourceVectorType(), i + destinationRank);
   }
   std::reverse(newStrides.begin(), newStrides.end());
-  SmallVector<int64_t, 4> newPosition = delinearize(newStrides, position);
+  SmallVector<int64_t, 4> newPosition = delinearize(position, newStrides);
   // OpBuilder is only used as a helper to build an I64ArrayAttr.
   OpBuilder b(extractOp.getContext());
   extractOp->setAttr(ExtractOp::getPositionAttrStrName(),
@@ -5265,13 +5265,38 @@ class FoldTransposeSplat final : public OpRewritePattern<TransposeOp> {
   }
 };
 
+/// Folds transpose(create_mask) into a new transposed create_mask.
+class FoldTransposeCreateMask final : public OpRewritePattern<TransposeOp> {
+public:
+  using OpRewritePattern::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(TransposeOp transposeOp,
+                                PatternRewriter &rewriter) const override {
+    auto createMaskOp =
+        transposeOp.getVector().getDefiningOp<vector::CreateMaskOp>();
+    if (!createMaskOp)
+      return failure();
+
+    // Get the transpose permutation and apply it to the vector.create_mask
+    // operands.
+    auto maskOperands = createMaskOp.getOperands();
+    SmallVector<int64_t> permutation;
+    transposeOp.getTransp(permutation);
+    SmallVector<Value> newOperands(maskOperands.begin(), maskOperands.end());
+    applyPermutationToVector(newOperands, permutation);
+
+    rewriter.replaceOpWithNewOp<vector::CreateMaskOp>(
+        transposeOp, transposeOp.getResultVectorType(), newOperands);
+    return success();
+  }
+};
+
 } // namespace
 
 void vector::TransposeOp::getCanonicalizationPatterns(
     RewritePatternSet &results, MLIRContext *context) {
-  results
-      .add<FoldTransposedScalarBroadcast, TransposeFolder, FoldTransposeSplat>(
-          context);
+  results.add<FoldTransposeCreateMask, FoldTransposedScalarBroadcast,
+              TransposeFolder, FoldTransposeSplat>(context);
 }
 
 void vector::TransposeOp::getTransp(SmallVectorImpl<int64_t> &results) {
diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorDistribute.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorDistribute.cpp
index 22c8ea611b36f..0b36ffedbbdb0 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorDistribute.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorDistribute.cpp
@@ -205,9 +205,10 @@ static WarpExecuteOnLane0Op moveRegionToNewWarpOpAndAppendReturns(
       indices.push_back(yieldValues.size() - 1);
     } else {
       // If the value already exit the region don't create a new output.
-      for (auto &yieldOperand : llvm::enumerate(yieldValues.getArrayRef())) {
-        if (yieldOperand.value() == std::get<0>(newRet)) {
-          indices.push_back(yieldOperand.index());
+      for (auto [idx, yieldOperand] :
+           llvm::enumerate(yieldValues.getArrayRef())) {
+        if (yieldOperand == std::get<0>(newRet)) {
+          indices.push_back(idx);
           break;
         }
       }
diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
index 0844fda09328d..fe59143ebd55f 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
@@ -12,6 +12,8 @@
 
 #include "mlir/Dialect/Vector/Transforms/VectorTransforms.h"
 
+#include <cstdint>
+#include <functional>
 #include <optional>
 #include <type_traits>
 
@@ -21,16 +23,21 @@
 #include "mlir/Dialect/Linalg/IR/Linalg.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/Dialect/Utils/StructuredOpsUtils.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Dialect/Vector/Transforms/VectorRewritePatterns.h"
 #include "mlir/Dialect/Vector/Utils/VectorUtils.h"
 #include "mlir/IR/BuiltinAttributeInterfaces.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/ImplicitLocOpBuilder.h"
+#include "mlir/IR/Location.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/TypeUtilities.h"
 #include "mlir/Interfaces/VectorInterfaces.h"
+#include "mlir/Support/LogicalResult.h"
 
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/MapVector.h"
@@ -454,7 +461,7 @@ class TransposeOpLowering : public OpRewritePattern<vector::TransposeOp> {
 
     for (int64_t linearIdx = 0; linearIdx < numTransposedElements;
          ++linearIdx) {
-      auto extractIdxs = delinearize(prunedInStrides, linearIdx);
+      auto extractIdxs = delinearize(linearIdx, prunedInStrides);
       SmallVector<int64_t> insertIdxs(extractIdxs);
       applyPermutationToVector(insertIdxs, prunedTransp);
       Value extractOp =
@@ -585,8 +592,7 @@ class OuterProductOpLowering : public OpRewritePattern<vector::OuterProductOp> {
         loc, resType, rewriter.getZeroAttr(resType));
     for (int64_t d = 0, e = resType.getDimSize(0); d < e; ++d) {
       auto pos = rewriter.getI64ArrayAttr(d);
-      Value x =
-          rewriter.create<vector::ExtractOp>(loc, op.getLhs(), pos);
+      Value x = rewriter.create<vector::ExtractOp>(loc, op.getLhs(), pos);
       Value a = rewriter.create<vector::BroadcastOp>(loc, rhsType, x);
       Value r = nullptr;
       if (acc)
@@ -3053,6 +3059,230 @@ struct ScanToArithOps : public OpRewritePattern<vector::ScanOp> {
   }
 };
 
+/// Canonicalization of a `vector.contraction %a, %b, %c` with row-major matmul
+/// semantics to a contraction suitable for MMT (matrix matrix multiplication
+/// with the RHS transposed) lowering.
+struct CanonicalizeContractMatmulToMMT final
+    : OpRewritePattern<vector::ContractionOp> {
+  using OpRewritePattern::OpRewritePattern;
+
+  using FilterConstraintType =
+      std::function<LogicalResult(vector::ContractionOp op)>;
+
+  CanonicalizeContractMatmulToMMT(MLIRContext *context, PatternBenefit benefit,
+                                  FilterConstraintType constraint)
+      : OpRewritePattern<vector::ContractionOp>(context, benefit),
+        filter(std::move(constraint)) {}
+
+  LogicalResult matchAndRewrite(vector::ContractionOp op,
+                                PatternRewriter &rewriter) const override {
+    // TODO: Remove native masks from contraction op?
+    if (!op.getMasks().empty())
+      return failure();
+
+    if (failed(filter(op)))
+      return failure();
+
+    Location loc = op.getLoc();
+    Value lhs = op.getLhs();
+    Value rhs = op.getRhs();
+    Value res = op.getAcc();
+
+    // Set up the parallel/reduction structure in right form.
+    using MapList = ArrayRef<ArrayRef<AffineExpr>>;
+    auto infer = [](MapList m) { return AffineMap::inferFromExprList(m); };
+    AffineExpr m;
+    AffineExpr n;
+    AffineExpr k;
+    bindDims(rewriter.getContext(), m, n, k);
+    static constexpr std::array<int64_t, 2> perm = {1, 0};
+    auto iteratorTypes = op.getIteratorTypes().getValue();
+    SmallVector<AffineMap, 4> maps = op.getIndexingMapsArray();
+    if (iteratorTypes.size() != 3 ||
+        !vector::isParallelIterator(iteratorTypes[0]) ||
+        !vector::isParallelIterator(iteratorTypes[1]) ||
+        !vector::isReductionIterator(iteratorTypes[2]))
+      return rewriter.notifyMatchFailure(op, "contraction is not a gemm");
+
+    // The canonical form is "TNT" = A row-major, B col-major, C row-major.
+    const auto canonicalForm = infer({{m, k}, {n, k}, {m, n}});
+    if (maps == canonicalForm)
+      return rewriter.notifyMatchFailure(op, "already in the canonical form");
+
+    // Create a vector transpose making sure to emit zero/sign-extend at the
+    // end.
+    auto createTranspose = [&rewriter, loc](Value mat) -> Value {
+      if (auto sext = mat.getDefiningOp<arith::ExtSIOp>()) {
+        Value trans =
+            rewriter.create<vector::TransposeOp>(loc, sext.getIn(), perm);
+        return rewriter.create<arith::ExtSIOp>(loc, mat.getType(), trans);
+      }
+      if (auto zext = mat.getDefiningOp<arith::ExtUIOp>()) {
+        Value trans =
+            rewriter.create<vector::TransposeOp>(loc, zext.getIn(), perm);
+        return rewriter.create<arith::ExtUIOp>(loc, mat.getType(), trans);
+      }
+      return rewriter.create<vector::TransposeOp>(loc, mat, perm);
+    };
+
+    if (maps == infer({{m, k}, {k, n}, {m, n}})) {
+      rhs = createTranspose(rhs);
+    } else if (maps == infer({{k, m}, {n, k}, {m, n}})) {
+      lhs = createTranspose(lhs);
+    } else if (maps == infer({{k, m}, {k, n}, {m, n}})) {
+      rhs = createTranspose(rhs);
+      lhs = createTranspose(lhs);
+    } else if (maps == infer({{k, m}, {k, n}, {n, m}})) {
+      std::swap(rhs, lhs);
+      rhs = createTranspose(rhs);
+      lhs = createTranspose(lhs);
+    } else if (maps == infer({{k, m}, {n, k}, {n, m}})) {
+      std::swap(rhs, lhs);
+      rhs = createTranspose(rhs);
+    } else if (maps == infer({{m, k}, {k, n}, {n, m}})) {
+      std::swap(lhs, rhs);
+      lhs = createTranspose(lhs);
+    } else if (maps == infer({{m, k}, {n, k}, {n, m}})) {
+      std::swap(lhs, rhs);
+    } else {
+      return rewriter.notifyMatchFailure(op, "unhandled contraction form");
+    }
+    rewriter.replaceOpWithNewOp<vector::ContractionOp>(
+        op, lhs, rhs, res, rewriter.getAffineMapArrayAttr(canonicalForm),
+        op.getIteratorTypes());
+    return success();
+  };
+
+private:
+  FilterConstraintType filter;
+};
+
+/// Flattens 2 or more dimensional `vector.gather` ops by unrolling the
+/// outermost dimension. For example:
+/// ```
+/// %g = vector.gather %base[%c0][%v], %mask, %pass_thru :
+///        ... into vector<2x3xf32>
+///
+/// ==>
+///
+/// %0   = arith.constant dense<0.0> : vector<2x3xf32>
+/// %g0  = vector.gather %base[%c0][%v0], %mask0, %pass_thru0 : ...
+/// %1   = vector.insert %g0, %0 [0] : vector<3xf32> into vector<2x3xf32>
+/// %g1  = vector.gather %base[%c0][%v1], %mask1, %pass_thru1 : ...
+/// %g   = vector.insert %g1, %1 [1] : vector<3xf32> into vector<2x3xf32>
+/// ```
+///
+/// When applied exhaustively, this will produce a sequence of 1-d gather ops.
+struct FlattenGather : OpRewritePattern<vector::GatherOp> {
+  using OpRewritePattern::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::GatherOp op,
+                                PatternRewriter &rewriter) const override {
+    VectorType resultTy = op.getType();
+    if (resultTy.getRank() < 2)
+      return rewriter.notifyMatchFailure(op, "already flat");
+
+    Location loc = op.getLoc();
+    Value indexVec = op.getIndexVec();
+    Value maskVec = op.getMask();
+    Value passThruVec = op.getPassThru();
+
+    Value result = rewriter.create<arith::ConstantOp>(
+        loc, resultTy, rewriter.getZeroAttr(resultTy));
+
+    Type subTy = VectorType::get(resultTy.getShape().drop_front(),
+                                 resultTy.getElementType());
+
+    for (int64_t i = 0, e = resultTy.getShape().front(); i < e; ++i) {
+      int64_t thisIdx[1] = {i};
+
+      Value indexSubVec =
+          rewriter.create<vector::ExtractOp>(loc, indexVec, thisIdx);
+      Value maskSubVec =
+          rewriter.create<vector::ExtractOp>(loc, maskVec, thisIdx);
+      Value passThruSubVec =
+          rewriter.create<vector::ExtractOp>(loc, passThruVec, thisIdx);
+      Value subGather = rewriter.create<vector::GatherOp>(
+          loc, subTy, op.getBase(), op.getIndices(), indexSubVec, maskSubVec,
+          passThruSubVec);
+      result =
+          rewriter.create<vector::InsertOp>(loc, subGather, result, thisIdx);
+    }
+
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
+/// Turns 1-d `vector.gather` into a scalarized sequence of `vector.loads` or
+/// `tensor.extract`s. To avoid out-of-bounds memory accesses, these
+/// loads/extracts are made conditional using `scf.if` ops.
+struct Gather1DToConditionalLoads : OpRewritePattern<vector::GatherOp> {
+  using OpRewritePattern::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::GatherOp op,
+                                PatternRewriter &rewriter) const override {
+    VectorType resultTy = op.getType();
+    if (resultTy.getRank() != 1)
+      return rewriter.notifyMatchFailure(op, "unsupported rank");
+
+    Location loc = op.getLoc();
+    Type elemTy = resultTy.getElementType();
+    // Vector type with a single element. Used to generate `vector.loads`.
+    VectorType elemVecTy = VectorType::get({1}, elemTy);
+
+    Value condMask = op.getMask();
+    Value base = op.getBase();
+    Value indexVec = rewriter.createOrFold<arith::IndexCastOp>(
+        loc, op.getIndexVectorType().clone(rewriter.getIndexType()),
+        op.getIndexVec());
+    auto baseOffsets = llvm::to_vector(op.getIndices());
+    Value lastBaseOffset = baseOffsets.back();
+
+    Value result = op.getPassThru();
+
+    // Emit a conditional access for each vector element.
+    for (int64_t i = 0, e = resultTy.getNumElements(); i < e; ++i) {
+      int64_t thisIdx[1] = {i};
+      Value condition =
+          rewriter.create<vector::ExtractOp>(loc, condMask, thisIdx);
+      Value index = rewriter.create<vector::ExtractOp>(loc, indexVec, thisIdx);
+      baseOffsets.back() =
+          rewriter.createOrFold<arith::AddIOp>(loc, lastBaseOffset, index);
+
+      auto loadBuilder = [&](OpBuilder &b, Location loc) {
+        Value extracted;
+        if (isa<MemRefType>(base.getType())) {
+          // `vector.load` does not support scalar result; emit a vector load
+          // and extract the single result instead.
+          Value load =
+              b.create<vector::LoadOp>(loc, elemVecTy, base, baseOffsets);
+          int64_t zeroIdx[1] = {0};
+          extracted = b.create<vector::ExtractOp>(loc, load, zeroIdx);
+        } else {
+          extracted = b.create<tensor::ExtractOp>(loc, base, baseOffsets);
+        }
+
+        Value newResult =
+            b.create<vector::InsertOp>(loc, extracted, result, thisIdx);
+        b.create<scf::YieldOp>(loc, newResult);
+      };
+      auto passThruBuilder = [result](OpBuilder &b, Location loc) {
+        b.create<scf::YieldOp>(loc, result);
+      };
+
+      result =
+          rewriter
+              .create<scf::IfOp>(loc, condition, /*thenBuilder=*/loadBuilder,
+                                 /*elseBuilder=*/passThruBuilder)
+              .getResult(0);
+    }
+
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
 } // namespace
 
 void mlir::vector::populateVectorMaskMaterializationPatterns(
@@ -3104,6 +3334,14 @@ void mlir::vector::populateVectorContractLoweringPatterns(
       options, patterns.getContext(), benefit);
 }
 
+void mlir::vector::populateVectorContractCanonicalizeMatmulToMMT(
+    RewritePatternSet &patterns,
+    std::function<LogicalResult(vector::ContractionOp)> constraint,
+    PatternBenefit benefit) {
+  patterns.add<CanonicalizeContractMatmulToMMT>(patterns.getContext(), benefit,
+                                                std::move(constraint));
+}
+
 void mlir::vector::populateVectorTransposeLoweringPatterns(
     RewritePatternSet &patterns, VectorTransformsOptions options,
     PatternBenefit benefit) {
@@ -3141,6 +3379,12 @@ void mlir::vector::populateVectorScanLoweringPatterns(
   patterns.add<ScanToArithOps>(patterns.getContext(), benefit);
 }
 
+void mlir::vector::populateVectorGatherLoweringPatterns(
+    RewritePatternSet &patterns, PatternBenefit benefit) {
+  patterns.add<FlattenGather, Gather1DToConditionalLoads>(patterns.getContext(),
+                                                          benefit);
+}
+
 //===----------------------------------------------------------------------===//
 // TableGen'd enum attribute definitions
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
index b2d00255b0d7f..688c0a2d7af0e 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
@@ -31,7 +31,7 @@ using namespace mlir::vector;
 static SmallVector<int64_t> getVectorOffset(ArrayRef<int64_t> ratioStrides,
                                             int64_t index,
                                             ArrayRef<int64_t> targetShape) {
-  return computeElementwiseMul(delinearize(ratioStrides, index), targetShape);
+  return computeElementwiseMul(delinearize(index, ratioStrides), targetShape);
 }
 
 /// A functor that accomplishes the same thing as `getVectorOffset` but
@@ -628,7 +628,7 @@ struct UnrollTransposePattern : public OpRewritePattern<vector::TransposeOp> {
       SmallVector<int64_t> permutedOffsets(elementOffsets.size());
       SmallVector<int64_t> permutedShape(elementOffsets.size());
       // Compute the source offsets and shape.
-      for (auto &indices : llvm::enumerate(permutation)) {
+      for (auto indices : llvm::enumerate(permutation)) {
         permutedOffsets[indices.value()] = elementOffsets[indices.index()];
         permutedShape[indices.value()] = (*targetShape)[indices.index()];
       }
diff --git a/mlir/lib/Dialect/X86Vector/Transforms/AVXTranspose.cpp b/mlir/lib/Dialect/X86Vector/Transforms/AVXTranspose.cpp
index 1558000139f20..daeb128fecb40 100644
--- a/mlir/lib/Dialect/X86Vector/Transforms/AVXTranspose.cpp
+++ b/mlir/lib/Dialect/X86Vector/Transforms/AVXTranspose.cpp
@@ -257,9 +257,9 @@ class TransposeOpLowering : public OpRewritePattern<vector::TransposeOp> {
       return rewriter.notifyMatchFailure(op, "Unsupported vector element type");
 
     SmallVector<int64_t> srcGtOneDims;
-    for (auto &en : llvm::enumerate(srcType.getShape()))
-      if (en.value() > 1)
-        srcGtOneDims.push_back(en.index());
+    for (auto [index, size] : llvm::enumerate(srcType.getShape()))
+      if (size > 1)
+        srcGtOneDims.push_back(index);
 
     if (srcGtOneDims.size() != 2)
       return rewriter.notifyMatchFailure(op, "Unsupported vector type");
diff --git a/mlir/lib/ExecutionEngine/ExecutionEngine.cpp b/mlir/lib/ExecutionEngine/ExecutionEngine.cpp
index b03f0e405c678..b122fdb9b1da1 100644
--- a/mlir/lib/ExecutionEngine/ExecutionEngine.cpp
+++ b/mlir/lib/ExecutionEngine/ExecutionEngine.cpp
@@ -196,17 +196,17 @@ static void packFunctionArguments(Module *module) {
     llvm::Value *argList = interfaceFunc->arg_begin();
     SmallVector<llvm::Value *, 8> args;
     args.reserve(llvm::size(func.args()));
-    for (auto &indexedArg : llvm::enumerate(func.args())) {
+    for (auto [index, arg] : llvm::enumerate(func.args())) {
       llvm::Value *argIndex = llvm::Constant::getIntegerValue(
-          builder.getInt64Ty(), APInt(64, indexedArg.index()));
+          builder.getInt64Ty(), APInt(64, index));
       llvm::Value *argPtrPtr =
           builder.CreateGEP(builder.getInt8PtrTy(), argList, argIndex);
       llvm::Value *argPtr =
           builder.CreateLoad(builder.getInt8PtrTy(), argPtrPtr);
-      llvm::Type *argTy = indexedArg.value().getType();
+      llvm::Type *argTy = arg.getType();
       argPtr = builder.CreateBitCast(argPtr, argTy->getPointerTo());
-      llvm::Value *arg = builder.CreateLoad(argTy, argPtr);
-      args.push_back(arg);
+      llvm::Value *load = builder.CreateLoad(argTy, argPtr);
+      args.push_back(load);
     }
 
     // Call the implementation function with the extracted arguments.
diff --git a/mlir/lib/ExecutionEngine/OptUtils.cpp b/mlir/lib/ExecutionEngine/OptUtils.cpp
index 69216c57c2868..b024a625fbf53 100644
--- a/mlir/lib/ExecutionEngine/OptUtils.cpp
+++ b/mlir/lib/ExecutionEngine/OptUtils.cpp
@@ -84,11 +84,7 @@ mlir::makeOptimizingTransformer(unsigned optLevel, unsigned sizeLevel,
     pb.crossRegisterProxies(lam, fam, cgam, mam);
 
     ModulePassManager mpm;
-    if (*ol == OptimizationLevel::O0)
-      mpm.addPass(pb.buildO0DefaultPipeline(*ol));
-    else
-      mpm.addPass(pb.buildPerModuleDefaultPipeline(*ol));
-
+    mpm.addPass(pb.buildPerModuleDefaultPipeline(*ol));
     mpm.run(*m, mam);
     return Error::success();
   };
diff --git a/mlir/lib/ExecutionEngine/SparseTensor/NNZ.cpp b/mlir/lib/ExecutionEngine/SparseTensor/NNZ.cpp
index 40d7888482f17..c6fd669ad513f 100644
--- a/mlir/lib/ExecutionEngine/SparseTensor/NNZ.cpp
+++ b/mlir/lib/ExecutionEngine/SparseTensor/NNZ.cpp
@@ -53,26 +53,26 @@ SparseTensorNNZ::SparseTensorNNZ(const std::vector<uint64_t> &lvlSizes,
   }
 }
 
-void SparseTensorNNZ::forallIndices(uint64_t stopLvl,
-                                    SparseTensorNNZ::NNZConsumer yield) const {
+void SparseTensorNNZ::forallCoords(uint64_t stopLvl,
+                                   SparseTensorNNZ::NNZConsumer yield) const {
   assert(stopLvl < getLvlRank() && "Level out of bounds");
   assert(isCompressedDLT(lvlTypes[stopLvl]) &&
          "Cannot look up non-compressed levels");
-  forallIndices(yield, stopLvl, 0, 0);
+  forallCoords(yield, stopLvl, 0, 0);
 }
 
-void SparseTensorNNZ::add(const std::vector<uint64_t> &lvlInd) {
+void SparseTensorNNZ::add(const std::vector<uint64_t> &lvlCoords) {
   uint64_t parentPos = 0;
   for (uint64_t l = 0, lvlrank = getLvlRank(); l < lvlrank; ++l) {
     if (isCompressedDLT(lvlTypes[l]))
       nnz[l][parentPos]++;
-    parentPos = parentPos * lvlSizes[l] + lvlInd[l];
+    parentPos = parentPos * lvlSizes[l] + lvlCoords[l];
   }
 }
 
-void SparseTensorNNZ::forallIndices(SparseTensorNNZ::NNZConsumer yield,
-                                    uint64_t stopLvl, uint64_t parentPos,
-                                    uint64_t l) const {
+void SparseTensorNNZ::forallCoords(SparseTensorNNZ::NNZConsumer yield,
+                                   uint64_t stopLvl, uint64_t parentPos,
+                                   uint64_t l) const {
   assert(l <= stopLvl);
   if (l == stopLvl) {
     assert(parentPos < nnz[l].size() && "Cursor is out of range");
@@ -81,6 +81,6 @@ void SparseTensorNNZ::forallIndices(SparseTensorNNZ::NNZConsumer yield,
     const uint64_t sz = lvlSizes[l];
     const uint64_t pstart = parentPos * sz;
     for (uint64_t i = 0; i < sz; ++i)
-      forallIndices(yield, stopLvl, pstart + i, l + 1);
+      forallCoords(yield, stopLvl, pstart + i, l + 1);
   }
 }
diff --git a/mlir/lib/ExecutionEngine/SparseTensor/Storage.cpp b/mlir/lib/ExecutionEngine/SparseTensor/Storage.cpp
index 88e4d39061e9a..2c4f0123ed441 100644
--- a/mlir/lib/ExecutionEngine/SparseTensor/Storage.cpp
+++ b/mlir/lib/ExecutionEngine/SparseTensor/Storage.cpp
@@ -74,19 +74,19 @@ SparseTensorStorageBase::SparseTensorStorageBase( // NOLINT
 MLIR_SPARSETENSOR_FOREVERY_V(IMPL_NEWENUMERATOR)
 #undef IMPL_NEWENUMERATOR
 
-#define IMPL_GETPOINTERS(PNAME, P)                                             \
-  void SparseTensorStorageBase::getPointers(std::vector<P> **, uint64_t) {     \
-    FATAL_PIV("getPointers" #PNAME);                                           \
+#define IMPL_GETPOSITIONS(PNAME, P)                                            \
+  void SparseTensorStorageBase::getPositions(std::vector<P> **, uint64_t) {    \
+    FATAL_PIV("getPositions" #PNAME);                                          \
   }
-MLIR_SPARSETENSOR_FOREVERY_FIXED_O(IMPL_GETPOINTERS)
-#undef IMPL_GETPOINTERS
+MLIR_SPARSETENSOR_FOREVERY_FIXED_O(IMPL_GETPOSITIONS)
+#undef IMPL_GETPOSITIONS
 
-#define IMPL_GETINDICES(INAME, I)                                              \
-  void SparseTensorStorageBase::getIndices(std::vector<I> **, uint64_t) {      \
-    FATAL_PIV("getIndices" #INAME);                                            \
+#define IMPL_GETCOORDINATES(CNAME, C)                                          \
+  void SparseTensorStorageBase::getCoordinates(std::vector<C> **, uint64_t) {  \
+    FATAL_PIV("getCoordinates" #CNAME);                                        \
   }
-MLIR_SPARSETENSOR_FOREVERY_FIXED_O(IMPL_GETINDICES)
-#undef IMPL_GETINDICES
+MLIR_SPARSETENSOR_FOREVERY_FIXED_O(IMPL_GETCOORDINATES)
+#undef IMPL_GETCOORDINATES
 
 #define IMPL_GETVALUES(VNAME, V)                                               \
   void SparseTensorStorageBase::getValues(std::vector<V> **) {                 \
diff --git a/mlir/lib/ExecutionEngine/SparseTensorRuntime.cpp b/mlir/lib/ExecutionEngine/SparseTensorRuntime.cpp
index 31c5332184bfe..eb5f4ac241962 100644
--- a/mlir/lib/ExecutionEngine/SparseTensorRuntime.cpp
+++ b/mlir/lib/ExecutionEngine/SparseTensorRuntime.cpp
@@ -17,7 +17,7 @@
 // The following memory-resident sparse storage schemes are supported:
 //
 // (a) A coordinate scheme for temporarily storing and lexicographically
-//     sorting a sparse tensor by index (SparseTensorCOO).
+//     sorting a sparse tensor by coordinate (SparseTensorCOO).
 //
 // (b) A "one-size-fits-all" sparse tensor storage scheme defined by
 //     per-dimension sparse/dense annnotations together with a dimension
@@ -126,11 +126,11 @@ class SparseTensorIterator final {
 /// `dim2lvl` argument must be a permutation.
 /// Used by `IMPL_CONVERTTOMLIRSPARSETENSOR`.
 //
-// TODO: generalize beyond 64-bit indices.
+// TODO: generalize beyond 64-bit overhead types.
 template <typename V>
 static SparseTensorStorage<uint64_t, uint64_t, V> *
 toMLIRSparseTensor(uint64_t rank, uint64_t nse, const uint64_t *dimSizes,
-                   const V *values, const uint64_t *dimIndices,
+                   const V *values, const uint64_t *dimCoordinates,
                    const uint64_t *dim2lvl, const DimLevelType *lvlTypes) {
 #ifndef NDEBUG
   // Verify that the sparsity values are supported.
@@ -143,22 +143,22 @@ toMLIRSparseTensor(uint64_t rank, uint64_t nse, const uint64_t *dimSizes,
 #endif
   // Verify that `dim2lvl` is a permutation of `[0..(rank-1)]`.
   // NOTE: The construction of `lvlSizes` and `lvl2dim` don't generalize
-  // to arbitrary `dim2lvl` mappings.  Whereas constructing `lvlInd` from
-  // `dimInd` does (though the details would have to be updated, just
+  // to arbitrary `dim2lvl` mappings.  Whereas constructing `lvlCoords` from
+  // `dimCoords` does (though the details would have to be updated, just
   // like for `IMPL_ADDELT`).
-  detail::PermutationRef d2l(rank, dim2lvl);
+  const detail::PermutationRef d2l(rank, dim2lvl);
   // Convert external format to internal COO.
-  auto lvlSizes = d2l.pushforward(rank, dimSizes);
+  const auto lvlSizes = d2l.pushforward(rank, dimSizes);
   auto *lvlCOO = new SparseTensorCOO<V>(lvlSizes, nse);
-  std::vector<uint64_t> lvlInd(rank);
-  const uint64_t *dimInd = dimIndices;
+  std::vector<uint64_t> lvlCoords(rank);
+  const uint64_t *dimCoords = dimCoordinates;
   for (uint64_t i = 0; i < nse; ++i) {
-    d2l.pushforward(rank, dimInd, lvlInd.data());
-    lvlCOO->add(lvlInd, values[i]);
-    dimInd += rank;
+    d2l.pushforward(rank, dimCoords, lvlCoords.data());
+    lvlCOO->add(lvlCoords, values[i]);
+    dimCoords += rank;
   }
   // Return sparse tensor storage format as opaque pointer.
-  auto lvl2dim = d2l.inverse();
+  const auto lvl2dim = d2l.inverse();
   auto *tensor = SparseTensorStorage<uint64_t, uint64_t, V>::newFromCOO(
       rank, dimSizes, rank, lvlTypes, lvl2dim.data(), *lvlCOO);
   delete lvlCOO;
@@ -172,15 +172,15 @@ toMLIRSparseTensor(uint64_t rank, uint64_t nse, const uint64_t *dimSizes,
 // SparseTensorCOO, then to the output.  We may want to reduce the number
 // of copies.
 //
-// TODO: generalize beyond 64-bit indices, no dim ordering, all dimensions
-// compressed
+// TODO: generalize beyond 64-bit overhead types, no dim ordering,
+// all dimensions compressed
 template <typename V>
 static void
 fromMLIRSparseTensor(const SparseTensorStorage<uint64_t, uint64_t, V> *tensor,
                      uint64_t *pRank, uint64_t *pNse, uint64_t **pShape,
-                     V **pValues, uint64_t **pIndices) {
+                     V **pValues, uint64_t **pCoordinates) {
   assert(tensor && "Received nullptr for tensor");
-  uint64_t dimRank = tensor->getDimRank();
+  const uint64_t dimRank = tensor->getDimRank();
   const auto &dimSizes = tensor->getDimSizes();
   std::vector<uint64_t> identityPerm(dimRank);
   std::iota(identityPerm.begin(), identityPerm.end(), 0);
@@ -188,30 +188,31 @@ fromMLIRSparseTensor(const SparseTensorStorage<uint64_t, uint64_t, V> *tensor,
       tensor->toCOO(dimRank, dimSizes.data(), dimRank, identityPerm.data());
 
   const std::vector<Element<V>> &elements = coo->getElements();
-  uint64_t nse = elements.size();
+  const uint64_t nse = elements.size();
 
   const auto &cooSizes = coo->getDimSizes();
   assert(cooSizes.size() == dimRank && "Rank mismatch");
-  uint64_t *shape = new uint64_t[dimRank];
-  std::memcpy((void *)shape, (const void *)cooSizes.data(),
+  uint64_t *dimShape = new uint64_t[dimRank];
+  std::memcpy(static_cast<void *>(dimShape),
+              static_cast<const void *>(cooSizes.data()),
               sizeof(uint64_t) * dimRank);
 
   V *values = new V[nse];
-  uint64_t *indices = new uint64_t[dimRank * nse];
+  uint64_t *coordinates = new uint64_t[dimRank * nse];
 
   for (uint64_t i = 0, base = 0; i < nse; ++i) {
     values[i] = elements[i].value;
     for (uint64_t d = 0; d < dimRank; ++d)
-      indices[base + d] = elements[i].indices[d];
+      coordinates[base + d] = elements[i].coords[d];
     base += dimRank;
   }
 
   delete coo;
   *pRank = dimRank;
   *pNse = nse;
-  *pShape = shape;
+  *pShape = dimShape;
   *pValues = values;
-  *pIndices = indices;
+  *pCoordinates = coordinates;
 }
 
 //===----------------------------------------------------------------------===//
@@ -280,22 +281,22 @@ extern "C" {
 //
 //===----------------------------------------------------------------------===//
 
-#define CASE(p, i, v, P, I, V)                                                 \
-  if (ptrTp == (p) && indTp == (i) && valTp == (v)) {                          \
+#define CASE(p, c, v, P, C, V)                                                 \
+  if (posTp == (p) && crdTp == (c) && valTp == (v)) {                          \
     switch (action) {                                                          \
     case Action::kEmpty:                                                       \
-      return SparseTensorStorage<P, I, V>::newEmpty(                           \
+      return SparseTensorStorage<P, C, V>::newEmpty(                           \
           dimRank, dimSizes, lvlRank, lvlSizes, lvlTypes, lvl2dim);            \
     case Action::kFromCOO: {                                                   \
       assert(ptr && "Received nullptr for SparseTensorCOO object");            \
       auto &coo = *static_cast<SparseTensorCOO<V> *>(ptr);                     \
-      return SparseTensorStorage<P, I, V>::newFromCOO(                         \
+      return SparseTensorStorage<P, C, V>::newFromCOO(                         \
           dimRank, dimSizes, lvlRank, lvlTypes, lvl2dim, coo);                 \
     }                                                                          \
     case Action::kSparseToSparse: {                                            \
       assert(ptr && "Received nullptr for SparseTensorStorage object");        \
       auto &tensor = *static_cast<SparseTensorStorageBase *>(ptr);             \
-      return SparseTensorStorage<P, I, V>::newFromSparseTensor(                \
+      return SparseTensorStorage<P, C, V>::newFromSparseTensor(                \
           dimRank, dimSizes, lvlRank, lvlSizes, lvlTypes, lvl2dim, dimRank,    \
           dim2lvl, tensor);                                                    \
     }                                                                          \
@@ -303,12 +304,12 @@ extern "C" {
       return new SparseTensorCOO<V>(lvlRank, lvlSizes);                        \
     case Action::kToCOO: {                                                     \
       assert(ptr && "Received nullptr for SparseTensorStorage object");        \
-      auto &tensor = *static_cast<SparseTensorStorage<P, I, V> *>(ptr);        \
+      auto &tensor = *static_cast<SparseTensorStorage<P, C, V> *>(ptr);        \
       return tensor.toCOO(lvlRank, lvlSizes, dimRank, dim2lvl);                \
     }                                                                          \
     case Action::kToIterator: {                                                \
       assert(ptr && "Received nullptr for SparseTensorStorage object");        \
-      auto &tensor = *static_cast<SparseTensorStorage<P, I, V> *>(ptr);        \
+      auto &tensor = *static_cast<SparseTensorStorage<P, C, V> *>(ptr);        \
       auto *coo = tensor.toCOO(lvlRank, lvlSizes, dimRank, dim2lvl);           \
       return new SparseTensorIterator<V>(coo);                                 \
     }                                                                          \
@@ -335,8 +336,8 @@ void *_mlir_ciface_newSparseTensor( // NOLINT
     StridedMemRefType<index_type, 1> *lvlSizesRef,
     StridedMemRefType<DimLevelType, 1> *lvlTypesRef,
     StridedMemRefType<index_type, 1> *lvl2dimRef,
-    StridedMemRefType<index_type, 1> *dim2lvlRef, OverheadType ptrTp,
-    OverheadType indTp, PrimaryType valTp, Action action, void *ptr) {
+    StridedMemRefType<index_type, 1> *dim2lvlRef, OverheadType posTp,
+    OverheadType crdTp, PrimaryType valTp, Action action, void *ptr) {
   ASSERT_NO_STRIDE(dimSizesRef);
   ASSERT_NO_STRIDE(lvlSizesRef);
   ASSERT_NO_STRIDE(lvlTypesRef);
@@ -355,10 +356,10 @@ void *_mlir_ciface_newSparseTensor( // NOLINT
 
   // Rewrite kIndex to kU64, to avoid introducing a bunch of new cases.
   // This is safe because of the static_assert above.
-  if (ptrTp == OverheadType::kIndex)
-    ptrTp = OverheadType::kU64;
-  if (indTp == OverheadType::kIndex)
-    indTp = OverheadType::kU64;
+  if (posTp == OverheadType::kIndex)
+    posTp = OverheadType::kU64;
+  if (crdTp == OverheadType::kIndex)
+    crdTp = OverheadType::kU64;
 
   // Double matrices with all combinations of overhead storage.
   CASE(OverheadType::kU64, OverheadType::kU64, PrimaryType::kF64, uint64_t,
@@ -463,8 +464,8 @@ void *_mlir_ciface_newSparseTensor( // NOLINT
   // Unsupported case (add above if needed).
   // TODO: better pretty-printing of enum values!
   MLIR_SPARSETENSOR_FATAL(
-      "unsupported combination of types: <P=%d, I=%d, V=%d>\n",
-      static_cast<int>(ptrTp), static_cast<int>(indTp),
+      "unsupported combination of types: <P=%d, C=%d, V=%d>\n",
+      static_cast<int>(posTp), static_cast<int>(crdTp),
       static_cast<int>(valTp));
 }
 #undef CASE
@@ -484,64 +485,67 @@ MLIR_SPARSETENSOR_FOREVERY_V(IMPL_SPARSEVALUES)
 
 #define IMPL_GETOVERHEAD(NAME, TYPE, LIB)                                      \
   void _mlir_ciface_##NAME(StridedMemRefType<TYPE, 1> *ref, void *tensor,      \
-                           index_type d) {                                     \
+                           index_type lvl) {                                   \
     assert(ref &&tensor);                                                      \
     std::vector<TYPE> *v;                                                      \
-    static_cast<SparseTensorStorageBase *>(tensor)->LIB(&v, d);                \
+    static_cast<SparseTensorStorageBase *>(tensor)->LIB(&v, lvl);              \
     assert(v);                                                                 \
     aliasIntoMemref(v->size(), v->data(), *ref);                               \
   }
-#define IMPL_SPARSEPOINTERS(PNAME, P)                                          \
-  IMPL_GETOVERHEAD(sparsePointers##PNAME, P, getPointers)
-MLIR_SPARSETENSOR_FOREVERY_O(IMPL_SPARSEPOINTERS)
-#undef IMPL_SPARSEPOINTERS
-
-#define IMPL_SPARSEINDICES(INAME, I)                                           \
-  IMPL_GETOVERHEAD(sparseIndices##INAME, I, getIndices)
-MLIR_SPARSETENSOR_FOREVERY_O(IMPL_SPARSEINDICES)
-#undef IMPL_SPARSEINDICES
+#define IMPL_SPARSEPOSITIONS(PNAME, P)                                         \
+  IMPL_GETOVERHEAD(sparsePositions##PNAME, P, getPositions)
+MLIR_SPARSETENSOR_FOREVERY_O(IMPL_SPARSEPOSITIONS)
+#undef IMPL_SPARSEPOSITIONS
+
+#define IMPL_SPARSECOORDINATES(CNAME, C)                                       \
+  IMPL_GETOVERHEAD(sparseCoordinates##CNAME, C, getCoordinates)
+MLIR_SPARSETENSOR_FOREVERY_O(IMPL_SPARSECOORDINATES)
+#undef IMPL_SPARSECOORDINATES
 #undef IMPL_GETOVERHEAD
 
 // TODO: while this API design will work for arbitrary dim2lvl mappings,
-// we should probably move the `dimInd`-to-`lvlInd` computation into codegen
-// (since that could enable optimizations to remove the intermediate memref).
+// we should probably move the `dimCoords`-to-`lvlCoords` computation into
+// codegen (since that could enable optimizations to remove the intermediate
+// memref).
 #define IMPL_ADDELT(VNAME, V)                                                  \
   void *_mlir_ciface_addElt##VNAME(                                            \
       void *lvlCOO, StridedMemRefType<V, 0> *vref,                             \
-      StridedMemRefType<index_type, 1> *dimIndRef,                             \
+      StridedMemRefType<index_type, 1> *dimCoordsRef,                          \
       StridedMemRefType<index_type, 1> *dim2lvlRef) {                          \
     assert(lvlCOO &&vref);                                                     \
-    ASSERT_NO_STRIDE(dimIndRef);                                               \
+    ASSERT_NO_STRIDE(dimCoordsRef);                                            \
     ASSERT_NO_STRIDE(dim2lvlRef);                                              \
-    const uint64_t rank = MEMREF_GET_USIZE(dimIndRef);                         \
+    const uint64_t rank = MEMREF_GET_USIZE(dimCoordsRef);                      \
     ASSERT_USIZE_EQ(dim2lvlRef, rank);                                         \
-    const index_type *dimInd = MEMREF_GET_PAYLOAD(dimIndRef);                  \
+    const index_type *dimCoords = MEMREF_GET_PAYLOAD(dimCoordsRef);            \
     const index_type *dim2lvl = MEMREF_GET_PAYLOAD(dim2lvlRef);                \
-    std::vector<index_type> lvlInd(rank);                                      \
+    std::vector<index_type> lvlCoords(rank);                                   \
     for (uint64_t d = 0; d < rank; ++d)                                        \
-      lvlInd[dim2lvl[d]] = dimInd[d];                                          \
+      lvlCoords[dim2lvl[d]] = dimCoords[d];                                    \
     V *value = MEMREF_GET_PAYLOAD(vref);                                       \
-    static_cast<SparseTensorCOO<V> *>(lvlCOO)->add(lvlInd, *value);            \
+    static_cast<SparseTensorCOO<V> *>(lvlCOO)->add(lvlCoords, *value);         \
     return lvlCOO;                                                             \
   }
 MLIR_SPARSETENSOR_FOREVERY_V(IMPL_ADDELT)
 #undef IMPL_ADDELT
 
+// NOTE: the `cref` argument uses the same coordinate-space as the `iter`
+// (which can be either dim- or lvl-coords, depending on context).
 #define IMPL_GETNEXT(VNAME, V)                                                 \
   bool _mlir_ciface_getNext##VNAME(void *iter,                                 \
-                                   StridedMemRefType<index_type, 1> *iref,     \
+                                   StridedMemRefType<index_type, 1> *cref,     \
                                    StridedMemRefType<V, 0> *vref) {            \
     assert(iter &&vref);                                                       \
-    ASSERT_NO_STRIDE(iref);                                                    \
-    index_type *indx = MEMREF_GET_PAYLOAD(iref);                               \
+    ASSERT_NO_STRIDE(cref);                                                    \
+    index_type *coords = MEMREF_GET_PAYLOAD(cref);                             \
     V *value = MEMREF_GET_PAYLOAD(vref);                                       \
-    const uint64_t isize = MEMREF_GET_USIZE(iref);                             \
+    const uint64_t rank = MEMREF_GET_USIZE(cref);                              \
     const Element<V> *elem =                                                   \
         static_cast<SparseTensorIterator<V> *>(iter)->getNext();               \
     if (elem == nullptr)                                                       \
       return false;                                                            \
-    for (uint64_t r = 0; r < isize; r++)                                       \
-      indx[r] = elem->indices[r];                                              \
+    for (uint64_t d = 0; d < rank; d++)                                        \
+      coords[d] = elem->coords[d];                                             \
     *value = elem->value;                                                      \
     return true;                                                               \
   }
@@ -549,36 +553,37 @@ MLIR_SPARSETENSOR_FOREVERY_V(IMPL_GETNEXT)
 #undef IMPL_GETNEXT
 
 #define IMPL_LEXINSERT(VNAME, V)                                               \
-  void _mlir_ciface_lexInsert##VNAME(void *tensor,                             \
-                                     StridedMemRefType<index_type, 1> *cref,   \
-                                     StridedMemRefType<V, 0> *vref) {          \
-    assert(tensor &&vref);                                                     \
-    ASSERT_NO_STRIDE(cref);                                                    \
-    index_type *cursor = MEMREF_GET_PAYLOAD(cref);                             \
-    assert(cursor);                                                            \
+  void _mlir_ciface_lexInsert##VNAME(                                          \
+      void *t, StridedMemRefType<index_type, 1> *lvlCoordsRef,                 \
+      StridedMemRefType<V, 0> *vref) {                                         \
+    assert(t &&vref);                                                          \
+    auto &tensor = *static_cast<SparseTensorStorageBase *>(t);                 \
+    ASSERT_NO_STRIDE(lvlCoordsRef);                                            \
+    index_type *lvlCoords = MEMREF_GET_PAYLOAD(lvlCoordsRef);                  \
+    assert(lvlCoords);                                                         \
     V *value = MEMREF_GET_PAYLOAD(vref);                                       \
-    static_cast<SparseTensorStorageBase *>(tensor)->lexInsert(cursor, *value); \
+    tensor.lexInsert(lvlCoords, *value);                                       \
   }
 MLIR_SPARSETENSOR_FOREVERY_V(IMPL_LEXINSERT)
 #undef IMPL_LEXINSERT
 
 #define IMPL_EXPINSERT(VNAME, V)                                               \
   void _mlir_ciface_expInsert##VNAME(                                          \
-      void *tensor, StridedMemRefType<index_type, 1> *cref,                    \
+      void *t, StridedMemRefType<index_type, 1> *lvlCoordsRef,                 \
       StridedMemRefType<V, 1> *vref, StridedMemRefType<bool, 1> *fref,         \
       StridedMemRefType<index_type, 1> *aref, index_type count) {              \
-    assert(tensor);                                                            \
-    ASSERT_NO_STRIDE(cref);                                                    \
+    assert(t);                                                                 \
+    auto &tensor = *static_cast<SparseTensorStorageBase *>(t);                 \
+    ASSERT_NO_STRIDE(lvlCoordsRef);                                            \
     ASSERT_NO_STRIDE(vref);                                                    \
     ASSERT_NO_STRIDE(fref);                                                    \
     ASSERT_NO_STRIDE(aref);                                                    \
     ASSERT_USIZE_EQ(vref, MEMREF_GET_USIZE(fref));                             \
-    index_type *cursor = MEMREF_GET_PAYLOAD(cref);                             \
+    index_type *lvlCoords = MEMREF_GET_PAYLOAD(lvlCoordsRef);                  \
     V *values = MEMREF_GET_PAYLOAD(vref);                                      \
     bool *filled = MEMREF_GET_PAYLOAD(fref);                                   \
     index_type *added = MEMREF_GET_PAYLOAD(aref);                              \
-    static_cast<SparseTensorStorageBase *>(tensor)->expInsert(                 \
-        cursor, values, filled, added, count);                                 \
+    tensor.expInsert(lvlCoords, values, filled, added, count);                 \
   }
 MLIR_SPARSETENSOR_FOREVERY_V(IMPL_EXPINSERT)
 #undef IMPL_EXPINSERT
@@ -596,13 +601,13 @@ void *_mlir_ciface_createCheckedSparseTensorReader(
 // FIXME: update `SparseTensorCodegenPass` to use
 // `_mlir_ciface_getSparseTensorReaderDimSizes` instead.
 void _mlir_ciface_copySparseTensorReaderDimSizes(
-    void *p, StridedMemRefType<index_type, 1> *dref) {
+    void *p, StridedMemRefType<index_type, 1> *dimSizesRef) {
   assert(p);
   SparseTensorReader &reader = *static_cast<SparseTensorReader *>(p);
-  ASSERT_NO_STRIDE(dref);
-  const uint64_t dimRank = MEMREF_GET_USIZE(dref);
-  ASSERT_USIZE_EQ(dref, reader.getRank());
-  index_type *dimSizes = MEMREF_GET_PAYLOAD(dref);
+  ASSERT_NO_STRIDE(dimSizesRef);
+  const uint64_t dimRank = MEMREF_GET_USIZE(dimSizesRef);
+  ASSERT_USIZE_EQ(dimSizesRef, reader.getRank());
+  index_type *dimSizes = MEMREF_GET_PAYLOAD(dimSizesRef);
   const index_type *fileSizes = reader.getDimSizes();
   for (uint64_t d = 0; d < dimRank; ++d)
     dimSizes[d] = fileSizes[d];
@@ -618,19 +623,21 @@ void _mlir_ciface_getSparseTensorReaderDimSizes(
 
 #define IMPL_GETNEXT(VNAME, V)                                                 \
   void _mlir_ciface_getSparseTensorReaderNext##VNAME(                          \
-      void *p, StridedMemRefType<index_type, 1> *iref,                         \
+      void *p, StridedMemRefType<index_type, 1> *dimCoordsRef,                 \
       StridedMemRefType<V, 0> *vref) {                                         \
     assert(p &&vref);                                                          \
     auto &reader = *static_cast<SparseTensorReader *>(p);                      \
-    ASSERT_NO_STRIDE(iref);                                                    \
-    const uint64_t rank = MEMREF_GET_USIZE(iref);                              \
-    index_type *indices = MEMREF_GET_PAYLOAD(iref);                            \
+    ASSERT_NO_STRIDE(dimCoordsRef);                                            \
+    const uint64_t dimRank = MEMREF_GET_USIZE(dimCoordsRef);                   \
+    index_type *dimCoords = MEMREF_GET_PAYLOAD(dimCoordsRef);                  \
     V *value = MEMREF_GET_PAYLOAD(vref);                                       \
-    *value = reader.readCOOElement<V>(rank, indices);                          \
+    *value = reader.readElement<V>(dimRank, dimCoords);                        \
   }
 MLIR_SPARSETENSOR_FOREVERY_V(IMPL_GETNEXT)
 #undef IMPL_GETNEXT
 
+// FIXME: This function name is weird; should rename to
+// "sparseTensorReaderReadToBuffers".
 #define IMPL_GETNEXT(VNAME, V, CNAME, C)                                       \
   bool _mlir_ciface_getSparseTensorReaderRead##CNAME##VNAME(                   \
       void *p, StridedMemRefType<index_type, 1> *dim2lvlRef,                   \
@@ -644,7 +651,7 @@ MLIR_SPARSETENSOR_FOREVERY_V(IMPL_GETNEXT)
     const uint64_t vSize = MEMREF_GET_USIZE(vref);                             \
     const uint64_t lvlRank = reader.getRank();                                 \
     assert(vSize *lvlRank <= cSize);                                           \
-    assert(vSize >= reader.getNNZ() && "Not enough space in buffers");         \
+    assert(vSize >= reader.getNSE() && "Not enough space in buffers");         \
     ASSERT_USIZE_EQ(dim2lvlRef, lvlRank);                                      \
     (void)cSize;                                                               \
     (void)vSize;                                                               \
@@ -662,8 +669,8 @@ void *_mlir_ciface_newSparseTensorFromReader(
     void *p, StridedMemRefType<index_type, 1> *lvlSizesRef,
     StridedMemRefType<DimLevelType, 1> *lvlTypesRef,
     StridedMemRefType<index_type, 1> *lvl2dimRef,
-    StridedMemRefType<index_type, 1> *dim2lvlRef, OverheadType ptrTp,
-    OverheadType indTp, PrimaryType valTp) {
+    StridedMemRefType<index_type, 1> *dim2lvlRef, OverheadType posTp,
+    OverheadType crdTp, PrimaryType valTp) {
   assert(p);
   SparseTensorReader &reader = *static_cast<SparseTensorReader *>(p);
   ASSERT_NO_STRIDE(lvlSizesRef);
@@ -683,18 +690,18 @@ void *_mlir_ciface_newSparseTensorFromReader(
   //
   // FIXME(wrengr): Really need to define a separate x-macro for handling
   // all this. (Or ideally some better, entirely-different approach)
-#define CASE(p, i, v, P, I, V)                                                 \
-  if (ptrTp == OverheadType::p && indTp == OverheadType::i &&                  \
+#define CASE(p, c, v, P, C, V)                                                 \
+  if (posTp == OverheadType::p && crdTp == OverheadType::c &&                  \
       valTp == PrimaryType::v)                                                 \
-    return static_cast<void *>(reader.readSparseTensor<P, I, V>(               \
+    return static_cast<void *>(reader.readSparseTensor<P, C, V>(               \
         lvlRank, lvlSizes, lvlTypes, lvl2dim, dim2lvl));
 #define CASE_SECSAME(p, v, P, V) CASE(p, p, v, P, P, V)
   // Rewrite kIndex to kU64, to avoid introducing a bunch of new cases.
   // This is safe because of the static_assert above.
-  if (ptrTp == OverheadType::kIndex)
-    ptrTp = OverheadType::kU64;
-  if (indTp == OverheadType::kIndex)
-    indTp = OverheadType::kU64;
+  if (posTp == OverheadType::kIndex)
+    posTp = OverheadType::kU64;
+  if (crdTp == OverheadType::kIndex)
+    crdTp = OverheadType::kU64;
   // Double matrices with all combinations of overhead storage.
   CASE(kU64, kU64, kF64, uint64_t, uint64_t, double);
   CASE(kU64, kU32, kF64, uint64_t, uint32_t, double);
@@ -762,37 +769,38 @@ void *_mlir_ciface_newSparseTensorFromReader(
   // Unsupported case (add above if needed).
   // TODO: better pretty-printing of enum values!
   MLIR_SPARSETENSOR_FATAL(
-      "unsupported combination of types: <P=%d, I=%d, V=%d>\n",
-      static_cast<int>(ptrTp), static_cast<int>(indTp),
+      "unsupported combination of types: <P=%d, C=%d, V=%d>\n",
+      static_cast<int>(posTp), static_cast<int>(crdTp),
       static_cast<int>(valTp));
 #undef CASE_SECSAME
 #undef CASE
 }
 
 void _mlir_ciface_outSparseTensorWriterMetaData(
-    void *p, index_type rank, index_type nnz,
-    StridedMemRefType<index_type, 1> *dref) {
+    void *p, index_type dimRank, index_type nse,
+    StridedMemRefType<index_type, 1> *dimSizesRef) {
   assert(p);
-  ASSERT_NO_STRIDE(dref);
-  assert(rank != 0);
-  index_type *dimSizes = MEMREF_GET_PAYLOAD(dref);
+  ASSERT_NO_STRIDE(dimSizesRef);
+  assert(dimRank != 0);
+  index_type *dimSizes = MEMREF_GET_PAYLOAD(dimSizesRef);
   SparseTensorWriter &file = *static_cast<SparseTensorWriter *>(p);
-  file << rank << " " << nnz << std::endl;
-  for (index_type r = 0; r < rank - 1; ++r)
-    file << dimSizes[r] << " ";
-  file << dimSizes[rank - 1] << std::endl;
+  file << dimRank << " " << nse << std::endl;
+  for (index_type d = 0; d < dimRank - 1; ++d)
+    file << dimSizes[d] << " ";
+  file << dimSizes[dimRank - 1] << std::endl;
 }
 
 #define IMPL_OUTNEXT(VNAME, V)                                                 \
   void _mlir_ciface_outSparseTensorWriterNext##VNAME(                          \
-      void *p, index_type rank, StridedMemRefType<index_type, 1> *iref,        \
+      void *p, index_type dimRank,                                             \
+      StridedMemRefType<index_type, 1> *dimCoordsRef,                          \
       StridedMemRefType<V, 0> *vref) {                                         \
     assert(p &&vref);                                                          \
-    ASSERT_NO_STRIDE(iref);                                                    \
-    index_type *indices = MEMREF_GET_PAYLOAD(iref);                            \
+    ASSERT_NO_STRIDE(dimCoordsRef);                                            \
+    const index_type *dimCoords = MEMREF_GET_PAYLOAD(dimCoordsRef);            \
     SparseTensorWriter &file = *static_cast<SparseTensorWriter *>(p);          \
-    for (index_type r = 0; r < rank; ++r)                                      \
-      file << (indices[r] + 1) << " ";                                         \
+    for (index_type d = 0; d < dimRank; ++d)                                   \
+      file << (dimCoords[d] + 1) << " ";                                       \
     V *value = MEMREF_GET_PAYLOAD(vref);                                       \
     file << *value << std::endl;                                               \
   }
@@ -872,21 +880,22 @@ void readSparseTensorShape(char *filename, std::vector<uint64_t> *out) {
 // We can't use `static_cast` here because `DimLevelType` is an enum-class.
 #define IMPL_CONVERTTOMLIRSPARSETENSOR(VNAME, V)                               \
   void *convertToMLIRSparseTensor##VNAME(                                      \
-      uint64_t rank, uint64_t nse, uint64_t *shape, V *values,                 \
-      uint64_t *indices, uint64_t *perm, uint8_t *sparse) {                    \
-    return toMLIRSparseTensor<V>(rank, nse, shape, values, indices, perm,      \
-                                 reinterpret_cast<DimLevelType *>(sparse));    \
+      uint64_t rank, uint64_t nse, uint64_t *dimSizes, V *values,              \
+      uint64_t *dimCoordinates, uint64_t *dim2lvl, uint8_t *lvlTypes) {        \
+    return toMLIRSparseTensor<V>(rank, nse, dimSizes, values, dimCoordinates,  \
+                                 dim2lvl,                                      \
+                                 reinterpret_cast<DimLevelType *>(lvlTypes));  \
   }
 MLIR_SPARSETENSOR_FOREVERY_V(IMPL_CONVERTTOMLIRSPARSETENSOR)
 #undef IMPL_CONVERTTOMLIRSPARSETENSOR
 
 #define IMPL_CONVERTFROMMLIRSPARSETENSOR(VNAME, V)                             \
-  void convertFromMLIRSparseTensor##VNAME(void *tensor, uint64_t *pRank,       \
-                                          uint64_t *pNse, uint64_t **pShape,   \
-                                          V **pValues, uint64_t **pIndices) {  \
+  void convertFromMLIRSparseTensor##VNAME(                                     \
+      void *tensor, uint64_t *pRank, uint64_t *pNse, uint64_t **pShape,        \
+      V **pValues, uint64_t **pCoordinates) {                                  \
     fromMLIRSparseTensor<V>(                                                   \
         static_cast<SparseTensorStorage<uint64_t, uint64_t, V> *>(tensor),     \
-        pRank, pNse, pShape, pValues, pIndices);                               \
+        pRank, pNse, pShape, pValues, pCoordinates);                           \
   }
 MLIR_SPARSETENSOR_FOREVERY_V(IMPL_CONVERTFROMMLIRSPARSETENSOR)
 #undef IMPL_CONVERTFROMMLIRSPARSETENSOR
@@ -908,8 +917,8 @@ bool getSparseTensorReaderIsSymmetric(void *p) {
   return static_cast<SparseTensorReader *>(p)->isSymmetric();
 }
 
-index_type getSparseTensorReaderNNZ(void *p) {
-  return static_cast<SparseTensorReader *>(p)->getNNZ();
+index_type getSparseTensorReaderNSE(void *p) {
+  return static_cast<SparseTensorReader *>(p)->getNSE();
 }
 
 index_type getSparseTensorReaderDimSize(void *p, index_type d) {
diff --git a/mlir/lib/IR/AsmPrinter.cpp b/mlir/lib/IR/AsmPrinter.cpp
index 8c5bb3021577d..18c35f0ed8d7e 100644
--- a/mlir/lib/IR/AsmPrinter.cpp
+++ b/mlir/lib/IR/AsmPrinter.cpp
@@ -3274,9 +3274,9 @@ void OperationPrinter::printValueUsers(Value value) {
   // One value might be used as the operand of an operation more than once.
   // Only print the operations results once in that case.
   SmallPtrSet<Operation *, 1> userSet;
-  for (auto &indexedUser : enumerate(value.getUsers())) {
-    if (userSet.insert(indexedUser.value()).second)
-      printUserIDs(indexedUser.value(), indexedUser.index());
+  for (auto [index, user] : enumerate(value.getUsers())) {
+    if (userSet.insert(user).second)
+      printUserIDs(user, index);
   }
 }
 
diff --git a/mlir/lib/IR/MLIRContext.cpp b/mlir/lib/IR/MLIRContext.cpp
index 176a8abe16fc7..29d49edf9efb9 100644
--- a/mlir/lib/IR/MLIRContext.cpp
+++ b/mlir/lib/IR/MLIRContext.cpp
@@ -12,6 +12,7 @@
 #include "AttributeDetail.h"
 #include "IntegerSetDetail.h"
 #include "TypeDetail.h"
+#include "mlir/IR/Action.h"
 #include "mlir/IR/AffineExpr.h"
 #include "mlir/IR/AffineMap.h"
 #include "mlir/IR/Attributes.h"
@@ -23,7 +24,6 @@
 #include "mlir/IR/Location.h"
 #include "mlir/IR/OpImplementation.h"
 #include "mlir/IR/Types.h"
-#include "mlir/Support/DebugAction.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/SmallString.h"
@@ -31,6 +31,7 @@
 #include "llvm/ADT/Twine.h"
 #include "llvm/Support/Allocator.h"
 #include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Compiler.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/Mutex.h"
 #include "llvm/Support/RWMutex.h"
@@ -123,8 +124,10 @@ class MLIRContextImpl {
   // Debugging
   //===--------------------------------------------------------------------===//
 
-  /// An action manager for use within the context.
-  DebugActionManager debugActionManager;
+  /// An action handler for handling actions that are dispatched through this
+  /// context.
+  std::function<void(function_ref<void()>, const tracing::Action &)>
+      actionHandler;
 
   //===--------------------------------------------------------------------===//
   // Diagnostics
@@ -345,13 +348,22 @@ static ArrayRef<T> copyArrayRefInto(llvm::BumpPtrAllocator &allocator,
 }
 
 //===----------------------------------------------------------------------===//
-// Debugging
+// Action Handling
 //===----------------------------------------------------------------------===//
 
-DebugActionManager &MLIRContext::getDebugActionManager() {
-  return getImpl().debugActionManager;
+void MLIRContext::registerActionHandler(HandlerTy handler) {
+  getImpl().actionHandler = std::move(handler);
 }
 
+/// Dispatch the provided action to the handler if any, or just execute it.
+void MLIRContext::executeActionInternal(function_ref<void()> actionFn,
+                                        const tracing::Action &action) {
+  assert(getImpl().actionHandler);
+  getImpl().actionHandler(actionFn, action);
+}
+
+bool MLIRContext::hasActionHandler() { return (bool)getImpl().actionHandler; }
+
 //===----------------------------------------------------------------------===//
 // Diagnostic Handlers
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/IR/PatternMatch.cpp b/mlir/lib/IR/PatternMatch.cpp
index 1fc234e104219..052696d5cb13a 100644
--- a/mlir/lib/IR/PatternMatch.cpp
+++ b/mlir/lib/IR/PatternMatch.cpp
@@ -303,29 +303,6 @@ void RewriterBase::finalizeRootUpdate(Operation *op) {
     rewriteListener->notifyOperationModified(op);
 }
 
-/// Merge the operations of block 'source' into the end of block 'dest'.
-/// 'source's predecessors must be empty or only contain 'dest`.
-/// 'argValues' is used to replace the block arguments of 'source' after
-/// merging.
-void RewriterBase::mergeBlocks(Block *source, Block *dest,
-                               ValueRange argValues) {
-  assert(llvm::all_of(source->getPredecessors(),
-                      [dest](Block *succ) { return succ == dest; }) &&
-         "expected 'source' to have no predecessors or only 'dest'");
-  assert(argValues.size() == source->getNumArguments() &&
-         "incorrect # of argument replacement values");
-
-  // Replace all of the successor arguments with the provided values.
-  for (auto it : llvm::zip(source->getArguments(), argValues))
-    replaceAllUsesWith(std::get<0>(it), std::get<1>(it));
-
-  // Splice the operations of the 'source' block into the 'dest' block and erase
-  // it.
-  dest->getOperations().splice(dest->end(), source->getOperations());
-  source->dropAllUses();
-  source->erase();
-}
-
 /// Find uses of `from` and replace them with `to` if the `functor` returns
 /// true. It also marks every modified uses and notifies the rewriter that an
 /// in-place operation modification is about to happen.
@@ -337,26 +314,48 @@ void RewriterBase::replaceUsesWithIf(Value from, Value to,
   }
 }
 
-// Merge the operations of block 'source' before the operation 'op'. Source
-// block should not have existing predecessors or successors.
-void RewriterBase::mergeBlockBefore(Block *source, Operation *op,
-                                    ValueRange argValues) {
+void RewriterBase::inlineBlockBefore(Block *source, Block *dest,
+                                     Block::iterator before,
+                                     ValueRange argValues) {
+  assert(argValues.size() == source->getNumArguments() &&
+         "incorrect # of argument replacement values");
+
+  // The source block will be deleted, so it should not have any users (i.e.,
+  // there should be no predecessors).
   assert(source->hasNoPredecessors() &&
          "expected 'source' to have no predecessors");
-  assert(source->hasNoSuccessors() &&
-         "expected 'source' to have no successors");
 
-  // Split the block containing 'op' into two, one containing all operations
-  // before 'op' (prologue) and another (epilogue) containing 'op' and all
-  // operations after it.
-  Block *prologue = op->getBlock();
-  Block *epilogue = splitBlock(prologue, op->getIterator());
+  if (dest->end() != before) {
+    // The source block will be inserted in the middle of the dest block, so
+    // the source block should have no successors. Otherwise, the remainder of
+    // the dest block would be unreachable.
+    assert(source->hasNoSuccessors() &&
+           "expected 'source' to have no successors");
+  } else {
+    // The source block will be inserted at the end of the dest block, so the
+    // dest block should have no successors. Otherwise, the inserted operations
+    // will be unreachable.
+    assert(dest->hasNoSuccessors() && "expected 'dest' to have no successors");
+  }
 
-  // Merge the source block at the end of the prologue.
-  mergeBlocks(source, prologue, argValues);
+  // Replace all of the successor arguments with the provided values.
+  for (auto it : llvm::zip(source->getArguments(), argValues))
+    replaceAllUsesWith(std::get<0>(it), std::get<1>(it));
 
-  // Merge the epilogue at the end the prologue.
-  mergeBlocks(epilogue, prologue);
+  // Move operations from the source block to the dest block and erase the
+  // source block.
+  dest->getOperations().splice(before, source->getOperations());
+  source->erase();
+}
+
+void RewriterBase::inlineBlockBefore(Block *source, Operation *op,
+                                     ValueRange argValues) {
+  inlineBlockBefore(source, op->getBlock(), op->getIterator(), argValues);
+}
+
+void RewriterBase::mergeBlocks(Block *source, Block *dest,
+                               ValueRange argValues) {
+  inlineBlockBefore(source, dest, dest->end(), argValues);
 }
 
 /// Split the operations starting at "before" (inclusive) out of the given
diff --git a/mlir/lib/IR/RegionKindInterface.cpp b/mlir/lib/IR/RegionKindInterface.cpp
index 9950b9117d265..cbef3025a5dd6 100644
--- a/mlir/lib/IR/RegionKindInterface.cpp
+++ b/mlir/lib/IR/RegionKindInterface.cpp
@@ -16,3 +16,11 @@
 using namespace mlir;
 
 #include "mlir/IR/RegionKindInterface.cpp.inc"
+
+bool mlir::mayHaveSSADominance(Region &region) {
+  auto regionKindOp =
+      dyn_cast_if_present<RegionKindInterface>(region.getParentOp());
+  if (!regionKindOp)
+    return true;
+  return regionKindOp.hasSSADominance(region.getRegionNumber());
+}
diff --git a/mlir/lib/Interfaces/InferTypeOpInterface.cpp b/mlir/lib/Interfaces/InferTypeOpInterface.cpp
index b76f236c3686f..7d464af78023e 100644
--- a/mlir/lib/Interfaces/InferTypeOpInterface.cpp
+++ b/mlir/lib/Interfaces/InferTypeOpInterface.cpp
@@ -22,6 +22,49 @@ namespace mlir {
 #include "mlir/Interfaces/InferTypeOpInterface.cpp.inc"
 } // namespace mlir
 
+LogicalResult
+mlir::reifyResultShapes(OpBuilder &b, Operation *op,
+                        ReifiedRankedShapedTypeDims &reifiedReturnShapes) {
+  auto reifiableOp = dyn_cast<ReifyRankedShapedTypeOpInterface>(op);
+  if (!reifiableOp)
+    return failure();
+  LogicalResult status = reifiableOp.reifyResultShapes(b, reifiedReturnShapes);
+#ifndef NDEBUG
+  if (failed(status))
+    return failure();
+  // Assert that ReifyRankedShapedTypeOpInterface::reifyResultShapes produced
+  // a correct result.
+  int64_t resultIdx = 0;
+  for (OpResult result : op->getResults()) {
+    auto shapedType = result.getType().dyn_cast<ShapedType>();
+    if (!shapedType)
+      continue;
+    if (!shapedType.hasRank()) {
+      // Nothing to check for unranked shaped values.
+      ++resultIdx;
+      continue;
+    }
+    // Assert one OpFoldResult per dimension.
+    assert(shapedType.getRank() ==
+               static_cast<int64_t>(reifiedReturnShapes[resultIdx].size()) &&
+           "incorrect implementation of ReifyRankedShapedTypeOpInterface");
+    for (int64_t dim = 0; dim < shapedType.getRank(); ++dim) {
+      // reifyResultShapes must return:
+      // * Attribute for static dimensions
+      // * Value for dynamic dimensions
+      assert(shapedType.isDynamicDim(dim) ==
+                 reifiedReturnShapes[resultIdx][dim].is<Value>() &&
+             "incorrect implementation of ReifyRankedShapedTypeOpInterface");
+    }
+    ++resultIdx;
+  }
+  // Assert that every shaped value result was reified.
+  assert(resultIdx == static_cast<int64_t>(reifiedReturnShapes.size()) &&
+         "incorrect implementation of ReifyRankedShapedTypeOpInterface");
+#endif // NDEBUG
+  return status;
+}
+
 bool ShapeAdaptor::hasRank() const {
   if (val.isNull())
     return false;
diff --git a/mlir/lib/Pass/PassStatistics.cpp b/mlir/lib/Pass/PassStatistics.cpp
index a3db6c6dfaa48..779dcfe7b6661 100644
--- a/mlir/lib/Pass/PassStatistics.cpp
+++ b/mlir/lib/Pass/PassStatistics.cpp
@@ -73,8 +73,8 @@ static void printResultsAsList(raw_ostream &os, OpPassManager &pm) {
         for (Pass::Statistic *it : pass->getStatistics())
           passEntry.push_back({it->getName(), it->getDesc(), it->getValue()});
       } else {
-        for (auto &it : llvm::enumerate(pass->getStatistics()))
-          passEntry[it.index()].value += it.value()->getValue();
+        for (auto [idx, statistic] : llvm::enumerate(pass->getStatistics()))
+          passEntry[idx].value += statistic->getValue();
       }
 #endif
       return;
diff --git a/mlir/lib/Support/CMakeLists.txt b/mlir/lib/Support/CMakeLists.txt
index 64dd134047b9a..0dbff287cc86b 100644
--- a/mlir/lib/Support/CMakeLists.txt
+++ b/mlir/lib/Support/CMakeLists.txt
@@ -1,5 +1,4 @@
 set(LLVM_OPTIONAL_SOURCES
-  DebugCounter.cpp
   FileUtilities.cpp
   IndentedOstream.cpp
   InterfaceSupport.cpp
@@ -10,7 +9,6 @@ set(LLVM_OPTIONAL_SOURCES
 )
 
 add_mlir_library(MLIRSupport
-  DebugCounter.cpp
   FileUtilities.cpp
   IndentedOstream.cpp
   InterfaceSupport.cpp
diff --git a/mlir/lib/Support/StorageUniquer.cpp b/mlir/lib/Support/StorageUniquer.cpp
index a33eb53d4d480..f6703185a4582 100644
--- a/mlir/lib/Support/StorageUniquer.cpp
+++ b/mlir/lib/Support/StorageUniquer.cpp
@@ -82,9 +82,6 @@ class ParametricStorageUniquer {
     /// The set containing the allocated storage instances.
     StorageTypeSet instances;
 
-    /// Allocator to use when constructing derived instances.
-    StorageAllocator allocator;
-
 #if LLVM_ENABLE_THREADS != 0
     /// A mutex to keep uniquing thread-safe.
     llvm::sys::SmartRWMutex<true> mutex;
@@ -93,13 +90,12 @@ class ParametricStorageUniquer {
 
   /// Get or create an instance of a param derived type in an thread-unsafe
   /// fashion.
-  BaseStorage *
-  getOrCreateUnsafe(Shard &shard, LookupKey &key,
-                    function_ref<BaseStorage *(StorageAllocator &)> ctorFn) {
+  BaseStorage *getOrCreateUnsafe(Shard &shard, LookupKey &key,
+                                 function_ref<BaseStorage *()> ctorFn) {
     auto existing = shard.instances.insert_as({key.hashValue}, key);
     BaseStorage *&storage = existing.first->storage;
     if (existing.second)
-      storage = ctorFn(shard.allocator);
+      storage = ctorFn();
     return storage;
   }
 
@@ -135,10 +131,9 @@ class ParametricStorageUniquer {
     }
   }
   /// Get or create an instance of a parametric type.
-  BaseStorage *
-  getOrCreate(bool threadingIsEnabled, unsigned hashValue,
-              function_ref<bool(const BaseStorage *)> isEqual,
-              function_ref<BaseStorage *(StorageAllocator &)> ctorFn) {
+  BaseStorage *getOrCreate(bool threadingIsEnabled, unsigned hashValue,
+                           function_ref<bool(const BaseStorage *)> isEqual,
+                           function_ref<BaseStorage *()> ctorFn) {
     Shard &shard = getShard(hashValue);
     ParametricStorageUniquer::LookupKey lookupKey{hashValue, isEqual};
     if (!threadingIsEnabled)
@@ -163,17 +158,20 @@ class ParametricStorageUniquer {
     llvm::sys::SmartScopedWriter<true> typeLock(shard.mutex);
     return localInst = getOrCreateUnsafe(shard, lookupKey, ctorFn);
   }
+
   /// Run a mutation function on the provided storage object in a thread-safe
   /// way.
-  LogicalResult
-  mutate(bool threadingIsEnabled, BaseStorage *storage,
-         function_ref<LogicalResult(StorageAllocator &)> mutationFn) {
-    Shard &shard = getShardFor(storage);
+  LogicalResult mutate(bool threadingIsEnabled, BaseStorage *storage,
+                       function_ref<LogicalResult()> mutationFn) {
     if (!threadingIsEnabled)
-      return mutationFn(shard.allocator);
+      return mutationFn();
 
+    // Get a shard to use for mutating this storage instance. It doesn't need to
+    // be the same shard as the original allocation, but does need to be
+    // deterministic.
+    Shard &shard = getShard(llvm::hash_value(storage));
     llvm::sys::SmartScopedWriter<true> lock(shard.mutex);
-    return mutationFn(shard.allocator);
+    return mutationFn();
   }
 
 private:
@@ -197,18 +195,6 @@ class ParametricStorageUniquer {
     return *shard;
   }
 
-  /// Return the shard that allocated the provided storage object.
-  Shard &getShardFor(BaseStorage *storage) {
-    for (size_t i = 0; i != numShards; ++i) {
-      if (Shard *shard = shards[i].load(std::memory_order_acquire)) {
-        llvm::sys::SmartScopedReader<true> lock(shard->mutex);
-        if (shard->allocator.allocated(storage))
-          return *shard;
-      }
-    }
-    llvm_unreachable("expected storage object to have a valid shard");
-  }
-
   /// A thread local cache for storage objects. This helps to reduce the lock
   /// contention when an object already existing in the cache.
   ThreadLocalCache<StorageTypeSet> localCache;
@@ -281,8 +267,9 @@ struct StorageUniquerImpl {
     assert(parametricUniquers.count(id) &&
            "creating unregistered storage instance");
     ParametricStorageUniquer &storageUniquer = *parametricUniquers[id];
-    return storageUniquer.getOrCreate(threadingIsEnabled, hashValue, isEqual,
-                                      ctorFn);
+    return storageUniquer.getOrCreate(
+        threadingIsEnabled, hashValue, isEqual,
+        [&] { return ctorFn(getThreadSafeAllocator()); });
   }
 
   /// Run a mutation function on the provided storage object in a thread-safe
@@ -293,7 +280,34 @@ struct StorageUniquerImpl {
     assert(parametricUniquers.count(id) &&
            "mutating unregistered storage instance");
     ParametricStorageUniquer &storageUniquer = *parametricUniquers[id];
-    return storageUniquer.mutate(threadingIsEnabled, storage, mutationFn);
+    return storageUniquer.mutate(threadingIsEnabled, storage, [&] {
+      return mutationFn(getThreadSafeAllocator());
+    });
+  }
+
+  /// Return an allocator that can be used to safely allocate instances on the
+  /// current thread.
+  StorageAllocator &getThreadSafeAllocator() {
+#if LLVM_ENABLE_THREADS != 0
+    if (!threadingIsEnabled)
+      return allocator;
+
+    // If the allocator has not been initialized, create a new one.
+    StorageAllocator *&threadAllocator = threadSafeAllocator.get();
+    if (!threadAllocator) {
+      threadAllocator = new StorageAllocator();
+
+      // Record this allocator, given that we don't want it to be destroyed when
+      // the thread dies.
+      llvm::sys::SmartScopedLock<true> lock(threadAllocatorMutex);
+      threadAllocators.push_back(
+          std::unique_ptr<StorageAllocator>(threadAllocator));
+    }
+
+    return *threadAllocator;
+#else
+    return allocator;
+#endif
   }
 
   //===--------------------------------------------------------------------===//
@@ -314,6 +328,22 @@ struct StorageUniquerImpl {
   // Instance Storage
   //===--------------------------------------------------------------------===//
 
+#if LLVM_ENABLE_THREADS != 0
+  /// A thread local set of allocators used for uniquing parametric instances,
+  /// or other data allocated in thread volatile situations.
+  ThreadLocalCache<StorageAllocator *> threadSafeAllocator;
+
+  /// All of the allocators that have been created for thread based allocation.
+  std::vector<std::unique_ptr<StorageAllocator>> threadAllocators;
+
+  /// A mutex used for safely adding a new thread allocator.
+  llvm::sys::SmartMutex<true> threadAllocatorMutex;
+#endif
+
+  /// Main allocator used for uniquing singleton instances, and other state when
+  /// thread safety is guaranteed.
+  StorageAllocator allocator;
+
   /// Map of type ids to the storage uniquer to use for registered objects.
   DenseMap<TypeID, std::unique_ptr<ParametricStorageUniquer>>
       parametricUniquers;
@@ -322,9 +352,6 @@ struct StorageUniquerImpl {
   /// singleton.
   DenseMap<TypeID, BaseStorage *> singletonInstances;
 
-  /// Allocator used for uniquing singleton instances.
-  StorageAllocator singletonAllocator;
-
   /// Flag specifying if multi-threading is enabled within the uniquer.
   bool threadingIsEnabled = true;
 };
@@ -378,7 +405,7 @@ void StorageUniquer::registerSingletonImpl(
     TypeID id, function_ref<BaseStorage *(StorageAllocator &)> ctorFn) {
   assert(!impl->singletonInstances.count(id) &&
          "storage class already registered");
-  impl->singletonInstances.try_emplace(id, ctorFn(impl->singletonAllocator));
+  impl->singletonInstances.try_emplace(id, ctorFn(impl->allocator));
 }
 
 /// Implementation for mutating an instance of a derived storage.
diff --git a/mlir/lib/TableGen/Operator.cpp b/mlir/lib/TableGen/Operator.cpp
index 77e27ccbe018a..291777198eafa 100644
--- a/mlir/lib/TableGen/Operator.cpp
+++ b/mlir/lib/TableGen/Operator.cpp
@@ -408,7 +408,7 @@ void Operator::populateTypeInferenceInfo(
 
   // For all results whose types are buildable, initialize their type inference
   // nodes with an edge to themselves. Mark those nodes are fully-inferred.
-  for (auto &[idx, infer] : llvm::enumerate(inference)) {
+  for (auto [idx, infer] : llvm::enumerate(inference)) {
     if (getResult(idx).constraint.getBuilderCall()) {
       infer.sources.emplace_back(InferredResultType::mapResultIndex(idx),
                                  "$_self");
@@ -693,7 +693,7 @@ void Operator::populateOpStructure() {
     auto verifyTraitValidity = [&](Record *trait) {
       auto *dependentTraits = trait->getValueAsListInit("dependentTraits");
       for (auto *traitInit : *dependentTraits)
-        if (traitSet.find(traitInit) == traitSet.end())
+        if (!traitSet.contains(traitInit))
           PrintFatalError(
               def.getLoc(),
               trait->getValueAsString("trait") + " requires " +
diff --git a/mlir/lib/Target/LLVMIR/ModuleImport.cpp b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
index d24f586b8b055..eecb855275134 100644
--- a/mlir/lib/Target/LLVMIR/ModuleImport.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
@@ -893,6 +893,8 @@ LogicalResult ModuleImport::convertGlobal(llvm::GlobalVariable *globalVar) {
   }
   if (globalVar->hasSection())
     globalOp.setSection(globalVar->getSection());
+  globalOp.setVisibility_(
+      convertVisibilityFromLLVM(globalVar->getVisibility()));
 
   return success();
 }
@@ -1042,7 +1044,7 @@ FailureOr<Value> ModuleImport::convertConstant(llvm::Constant *constant) {
     // resulting in a conflicting `valueMapping` entry.
     llvm::Instruction *inst = constExpr->getAsInstruction();
     auto guard = llvm::make_scope_exit([&]() {
-      assert(noResultOpMapping.find(inst) == noResultOpMapping.end() &&
+      assert(!noResultOpMapping.contains(inst) &&
              "expected constant expression to return a result");
       valueMapping.erase(inst);
       inst->deleteValue();
@@ -1611,6 +1613,8 @@ LogicalResult ModuleImport::processFunction(llvm::Function *func) {
   if (func->hasGC())
     funcOp.setGarbageCollector(StringRef(func->getGC()));
 
+  funcOp.setVisibility_(convertVisibilityFromLLVM(func->getVisibility()));
+
   // Handle Function attributes.
   processFunctionAttributes(func, funcOp);
 
diff --git a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
index 15e7318f55f10..0ce7d4fe12322 100644
--- a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
@@ -520,9 +520,7 @@ void mlir::LLVM::detail::connectPHINodes(Region &region,
     auto phis = llvmBB->phis();
     auto numArguments = bb.getNumArguments();
     assert(numArguments == std::distance(phis.begin(), phis.end()));
-    for (auto &numberedPhiNode : llvm::enumerate(phis)) {
-      auto &phiNode = numberedPhiNode.value();
-      unsigned index = numberedPhiNode.index();
+    for (auto [index, phiNode] : llvm::enumerate(phis)) {
       for (auto *pred : bb.getPredecessors()) {
         // Find the LLVM IR block that contains the converted terminator
         // instruction and use it in the PHI node. Note that this block is not
@@ -706,6 +704,8 @@ LogicalResult ModuleTranslation::convertGlobals() {
     if (alignment.has_value())
       var->setAlignment(llvm::MaybeAlign(alignment.value()));
 
+    var->setVisibility(convertVisibilityToLLVM(op.getVisibility_()));
+
     globalsMapping.try_emplace(op, var);
   }
 
@@ -975,6 +975,9 @@ LogicalResult ModuleTranslation::convertFunctionSignatures() {
     if (failed(forwardPassthroughAttributes(
             function.getLoc(), function.getPassthrough(), llvmFunc)))
       return failure();
+
+    // Convert visibility attribute.
+    llvmFunc->setVisibility(convertVisibilityToLLVM(function.getVisibility_()));
   }
 
   return success();
diff --git a/mlir/lib/Tools/mlir-opt/CMakeLists.txt b/mlir/lib/Tools/mlir-opt/CMakeLists.txt
index dfcff9d034364..64e74e21270ef 100644
--- a/mlir/lib/Tools/mlir-opt/CMakeLists.txt
+++ b/mlir/lib/Tools/mlir-opt/CMakeLists.txt
@@ -6,6 +6,7 @@ add_mlir_library(MLIROptLib
 
   LINK_LIBS PUBLIC
   MLIRBytecodeWriter
+  MLIRDebug
   MLIRPass
   MLIRParser
   MLIRSupport
diff --git a/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp b/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp
index ee64f41299c43..c9b1ae09ccbcd 100644
--- a/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp
+++ b/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp
@@ -13,6 +13,7 @@
 
 #include "mlir/Tools/mlir-opt/MlirOptMain.h"
 #include "mlir/Bytecode/BytecodeWriter.h"
+#include "mlir/Debug/Counter.h"
 #include "mlir/IR/AsmState.h"
 #include "mlir/IR/Attributes.h"
 #include "mlir/IR/BuiltinOps.h"
@@ -23,7 +24,6 @@
 #include "mlir/Parser/Parser.h"
 #include "mlir/Pass/Pass.h"
 #include "mlir/Pass/PassManager.h"
-#include "mlir/Support/DebugCounter.h"
 #include "mlir/Support/FileUtilities.h"
 #include "mlir/Support/Timing.h"
 #include "mlir/Support/ToolUtilities.h"
@@ -31,6 +31,8 @@
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/FileUtilities.h"
 #include "llvm/Support/InitLLVM.h"
+#include "llvm/Support/ManagedStatic.h"
+#include "llvm/Support/Process.h"
 #include "llvm/Support/Regex.h"
 #include "llvm/Support/SourceMgr.h"
 #include "llvm/Support/StringSaver.h"
@@ -40,6 +42,71 @@
 using namespace mlir;
 using namespace llvm;
 
+namespace {
+/// This class is intended to manage the handling of command line options for
+/// creating a *-opt config. This is a singleton.
+struct MlirOptMainConfigCLOptions : public MlirOptMainConfig {
+  MlirOptMainConfigCLOptions() {
+    // These options are static but all uses ExternalStorage to initialize the
+    // members of the parent class. This is unusual but since this class is a
+    // singleton it basically attaches command line option to the singleton
+    // members.
+
+    static cl::opt<bool, /*ExternalStorage=*/true> allowUnregisteredDialects(
+        "allow-unregistered-dialect",
+        cl::desc("Allow operation with no registered dialects"),
+        cl::location(allowUnregisteredDialectsFlag), cl::init(false));
+
+    static cl::opt<bool, /*ExternalStorage=*/true> dumpPassPipeline(
+        "dump-pass-pipeline", cl::desc("Print the pipeline that will be run"),
+        cl::location(dumpPassPipelineFlag), cl::init(false));
+
+    static cl::opt<bool, /*ExternalStorage=*/true> emitBytecode(
+        "emit-bytecode", cl::desc("Emit bytecode when generating output"),
+        cl::location(emitBytecodeFlag), cl::init(false));
+
+    static cl::opt<bool, /*ExternalStorage=*/true> explicitModule(
+        "no-implicit-module",
+        cl::desc("Disable implicit addition of a top-level module op during "
+                 "parsing"),
+        cl::location(useExplicitModuleFlag), cl::init(false));
+
+    static cl::opt<bool, /*ExternalStorage=*/true> showDialects(
+        "show-dialects",
+        cl::desc("Print the list of registered dialects and exit"),
+        cl::location(showDialectsFlag), cl::init(false));
+
+    static cl::opt<bool, /*ExternalStorage=*/true> splitInputFile(
+        "split-input-file",
+        cl::desc("Split the input file into pieces and process each "
+                 "chunk independently"),
+        cl::location(splitInputFileFlag), cl::init(false));
+
+    static cl::opt<bool, /*ExternalStorage=*/true> verifyDiagnostics(
+        "verify-diagnostics",
+        cl::desc("Check that emitted diagnostics match "
+                 "expected-* lines on the corresponding line"),
+        cl::location(verifyDiagnosticsFlag), cl::init(false));
+
+    static cl::opt<bool, /*ExternalStorage=*/true> verifyPasses(
+        "verify-each",
+        cl::desc("Run the verifier after each transformation pass"),
+        cl::location(verifyPassesFlag), cl::init(true));
+
+    static PassPipelineCLParser passPipeline("", "Compiler passes to run", "p");
+    setPassPipelineParser(passPipeline);
+  }
+};
+} // namespace
+
+ManagedStatic<MlirOptMainConfigCLOptions> clOptionsConfig;
+
+void MlirOptMainConfig::registerCLOptions() { *clOptionsConfig; }
+
+MlirOptMainConfig MlirOptMainConfig::createFromCLOptions() {
+  return *clOptionsConfig;
+}
+
 MlirOptMainConfig &MlirOptMainConfig::setPassPipelineParser(
     const PassPipelineCLParser &passPipeline) {
   passPipelineCallback = [&](PassManager &pm) {
@@ -146,7 +213,7 @@ static LogicalResult processBuffer(raw_ostream &os,
   context.allowUnregisteredDialects(config.shouldAllowUnregisteredDialects());
   if (config.shouldVerifyDiagnostics())
     context.printOpOnDiagnostic(false);
-  context.getDebugActionManager().registerActionHandler<DebugCounter>();
+  context.registerActionHandler(tracing::DebugCounter());
 
   // If we are in verify diagnostics mode then we have a lot of work to do,
   // otherwise just perform the actions without worrying about it.
@@ -171,6 +238,12 @@ LogicalResult mlir::MlirOptMain(llvm::raw_ostream &outputStream,
                                 std::unique_ptr<llvm::MemoryBuffer> buffer,
                                 DialectRegistry &registry,
                                 const MlirOptMainConfig &config) {
+  if (config.shouldShowDialects()) {
+    llvm::outs() << "Available Dialects: ";
+    interleave(registry.getDialectNames(), llvm::outs(), ",");
+    llvm::outs() << "\n";
+  }
+
   // The split-input-file mode is a very specific mode that slices the file
   // up into small pieces and checks each independently.
   // We use an explicit threadpool to avoid creating and joining/destroying
@@ -244,54 +317,15 @@ LogicalResult mlir::MlirOptMain(int argc, char **argv, llvm::StringRef toolName,
                                              cl::value_desc("filename"),
                                              cl::init("-"));
 
-  static cl::opt<bool> splitInputFile(
-      "split-input-file",
-      cl::desc("Split the input file into pieces and process each "
-               "chunk independently"),
-      cl::init(false));
-
-  static cl::opt<bool> verifyDiagnostics(
-      "verify-diagnostics",
-      cl::desc("Check that emitted diagnostics match "
-               "expected-* lines on the corresponding line"),
-      cl::init(false));
-
-  static cl::opt<bool> verifyPasses(
-      "verify-each",
-      cl::desc("Run the verifier after each transformation pass"),
-      cl::init(true));
-
-  static cl::opt<bool> allowUnregisteredDialects(
-      "allow-unregistered-dialect",
-      cl::desc("Allow operation with no registered dialects (discouraged: testing only!)"), cl::init(false));
-
-  static cl::opt<bool> showDialects(
-      "show-dialects", cl::desc("Print the list of registered dialects"),
-      cl::init(false));
-
-  static cl::opt<bool> emitBytecode(
-      "emit-bytecode", cl::desc("Emit bytecode when generating output"),
-      cl::init(false));
-
-  static cl::opt<bool> explicitModule{
-      "no-implicit-module",
-      cl::desc(
-          "Disable implicit addition of a top-level module op during parsing"),
-      cl::init(false)};
-
-  static cl::opt<bool> dumpPassPipeline{
-      "dump-pass-pipeline", cl::desc("Print the pipeline that will be run"),
-      cl::init(false)};
-
   InitLLVM y(argc, argv);
 
   // Register any command line options.
+  MlirOptMainConfig::registerCLOptions();
   registerAsmPrinterCLOptions();
   registerMLIRContextCLOptions();
   registerPassManagerCLOptions();
   registerDefaultTimingManagerCLOptions();
-  DebugCounter::registerCLOptions();
-  PassPipelineCLParser passPipeline("", "Compiler passes to run", "p");
+  tracing::DebugCounter::registerCLOptions();
 
   // Build the list of dialects as a header for the --help message.
   std::string helpHeader = (toolName + "\nAvailable Dialects: ").str();
@@ -302,14 +336,16 @@ LogicalResult mlir::MlirOptMain(int argc, char **argv, llvm::StringRef toolName,
   }
   // Parse pass names in main to ensure static initialization completed.
   cl::ParseCommandLineOptions(argc, argv, helpHeader);
+  MlirOptMainConfig config = MlirOptMainConfig::createFromCLOptions();
+  config.preloadDialectsInContext(preloadDialectsInContext);
 
-  if (showDialects) {
-    llvm::outs() << "Available Dialects:\n";
-    interleave(
-        registry.getDialectNames(), llvm::outs(),
-        [](auto name) { llvm::outs() << name; }, "\n");
-    return success();
-  }
+  // When reading from stdin and the input is a tty, it is often a user mistake
+  // and the process "appears to be stuck". Print a message to let the user know
+  // about it!
+  if (inputFilename == "-" &&
+      sys::Process::FileDescriptorIsDisplayed(fileno(stdin)))
+    llvm::errs() << "(processing input from stdin now, hit ctrl-c/ctrl-d to "
+                    "interrupt)\n";
 
   // Set up the input file.
   std::string errorMessage;
@@ -324,18 +360,6 @@ LogicalResult mlir::MlirOptMain(int argc, char **argv, llvm::StringRef toolName,
     llvm::errs() << errorMessage << "\n";
     return failure();
   }
-  // Setup the configuration for the main function.
-  MlirOptMainConfig config;
-  config.setPassPipelineParser(passPipeline)
-      .splitInputFile(splitInputFile)
-      .verifyDiagnostics(verifyDiagnostics)
-      .verifyPasses(verifyPasses)
-      .allowUnregisteredDialects(allowUnregisteredDialects)
-      .preloadDialectsInContext(preloadDialectsInContext)
-      .emitBytecode(emitBytecode)
-      .useExplicitModule(explicitModule)
-      .dumpPassPipeline(dumpPassPipeline);
-
   if (failed(MlirOptMain(output->os(), std::move(file), registry, config)))
     return failure();
 
diff --git a/mlir/lib/Transforms/PrintIR.cpp b/mlir/lib/Transforms/PrintIR.cpp
index 41cb6efd29fca..cc42c7e2f1db0 100644
--- a/mlir/lib/Transforms/PrintIR.cpp
+++ b/mlir/lib/Transforms/PrintIR.cpp
@@ -7,6 +7,8 @@
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/Passes.h"
+#include "llvm/Support/Debug.h"
 
 namespace mlir {
 namespace {
@@ -15,15 +17,21 @@ namespace {
 #include "mlir/Transforms/Passes.h.inc"
 
 struct PrintIRPass : public impl::PrintIRPassBase<PrintIRPass> {
-  PrintIRPass() = default;
-
-  void runOnOperation() override { getOperation()->dump(); }
+  using impl::PrintIRPassBase<PrintIRPass>::PrintIRPassBase;
+
+  void runOnOperation() override {
+    llvm::dbgs() << "// -----// IR Dump";
+    if (!this->label.empty())
+      llvm::dbgs() << " " << this->label;
+    llvm::dbgs() << " //----- //\n";
+    getOperation()->dump();
+  }
 };
 
 } // namespace
 
-std::unique_ptr<Pass> createPrintIRPass() {
-  return std::make_unique<PrintIRPass>();
+std::unique_ptr<Pass> createPrintIRPass(const PrintIRPassOptions &options) {
+  return std::make_unique<PrintIRPass>(options);
 }
 
 } // namespace mlir
\ No newline at end of file
diff --git a/mlir/lib/Transforms/TopologicalSort.cpp b/mlir/lib/Transforms/TopologicalSort.cpp
index 04b52259abade..1219968fb3692 100644
--- a/mlir/lib/Transforms/TopologicalSort.cpp
+++ b/mlir/lib/Transforms/TopologicalSort.cpp
@@ -24,7 +24,7 @@ struct TopologicalSortPass
   void runOnOperation() override {
     // Topologically sort the regions of the operation without SSA dominance.
     getOperation()->walk([](RegionKindInterface op) {
-      for (auto &it : llvm::enumerate(op->getRegions())) {
+      for (auto it : llvm::enumerate(op->getRegions())) {
         if (op.hasSSADominance(it.index()))
           continue;
         for (Block &block : it.value())
diff --git a/mlir/lib/Transforms/Utils/ControlFlowSinkUtils.cpp b/mlir/lib/Transforms/Utils/ControlFlowSinkUtils.cpp
index 4579bfd7a01bb..cfd568f028721 100644
--- a/mlir/lib/Transforms/Utils/ControlFlowSinkUtils.cpp
+++ b/mlir/lib/Transforms/Utils/ControlFlowSinkUtils.cpp
@@ -136,8 +136,8 @@ void mlir::getSinglyExecutedRegionsToSink(RegionBranchOpInterface branch,
                                           SmallVectorImpl<Region *> &regions) {
   // Collect constant operands.
   SmallVector<Attribute> operands(branch->getNumOperands(), Attribute());
-  for (auto &it : llvm::enumerate(branch->getOperands()))
-    (void)matchPattern(it.value(), m_Constant(&operands[it.index()]));
+  for (auto [idx, operand] : llvm::enumerate(branch->getOperands()))
+    (void)matchPattern(operand, m_Constant(&operands[idx]));
 
   // Get the invocation bounds.
   SmallVector<InvocationBounds> bounds;
diff --git a/mlir/lib/Transforms/Utils/DialectConversion.cpp b/mlir/lib/Transforms/Utils/DialectConversion.cpp
index 52ebd54e1bee8..f5e9e71506e38 100644
--- a/mlir/lib/Transforms/Utils/DialectConversion.cpp
+++ b/mlir/lib/Transforms/Utils/DialectConversion.cpp
@@ -12,6 +12,7 @@
 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/FunctionInterfaces.h"
 #include "mlir/IR/IRMapping.h"
+#include "mlir/IR/Iterators.h"
 #include "mlir/Rewrite/PatternApplicator.h"
 #include "llvm/ADT/ScopeExit.h"
 #include "llvm/ADT/SetVector.h"
@@ -27,55 +28,6 @@ using namespace mlir::detail;
 
 #define DEBUG_TYPE "dialect-conversion"
 
-/// Recursively collect all of the operations to convert from within 'region'.
-/// If 'target' is nonnull, operations that are recursively legal have their
-/// regions pre-filtered to avoid considering them for legalization.
-static LogicalResult
-computeConversionSet(iterator_range<Region::iterator> region,
-                     Location regionLoc,
-                     SmallVectorImpl<Operation *> &toConvert,
-                     ConversionTarget *target = nullptr) {
-  if (region.empty())
-    return success();
-
-  // Traverse starting from the entry block.
-  SmallVector<Block *, 16> worklist(1, &*region.begin());
-  DenseSet<Block *> visitedBlocks;
-  visitedBlocks.insert(worklist.front());
-  while (!worklist.empty()) {
-    Block *block = worklist.pop_back_val();
-
-    // Compute the conversion set of each of the nested operations.
-    for (Operation &op : *block) {
-      toConvert.emplace_back(&op);
-
-      // Don't check this operation's children for conversion if the operation
-      // is recursively legal.
-      auto legalityInfo =
-          target ? target->isLegal(&op)
-                 : std::optional<ConversionTarget::LegalOpDetails>();
-      if (legalityInfo && legalityInfo->isRecursivelyLegal)
-        continue;
-      for (auto &region : op.getRegions()) {
-        if (failed(computeConversionSet(region.getBlocks(), region.getLoc(),
-                                        toConvert, target)))
-          return failure();
-      }
-    }
-
-    // Recurse to children that haven't been visited.
-    for (Block *succ : block->getSuccessors())
-      if (visitedBlocks.insert(succ).second)
-        worklist.push_back(succ);
-  }
-
-  // Check that all blocks in the region were visited.
-  if (llvm::any_of(llvm::drop_begin(region, 1),
-                   [&](Block &block) { return !visitedBlocks.count(&block); }))
-    return emitError(regionLoc, "unreachable blocks were not converted");
-  return success();
-}
-
 /// A utility function to log a successful result for the given reason.
 template <typename... Args>
 static void logSuccess(llvm::ScopedPrinter &os, StringRef fmt, Args &&...args) {
@@ -289,7 +241,7 @@ struct OpReplacement {
 enum class BlockActionKind {
   Create,
   Erase,
-  Merge,
+  Inline,
   Move,
   Split,
   TypeConversion
@@ -302,13 +254,14 @@ struct BlockPosition {
   Block *insertAfterBlock;
 };
 
-/// Information needed to undo the merge actions.
-/// - the source block, and
-/// - the Operation that was the last operation in the dest block before the
-///   merge (could be null if the dest block was empty).
-struct MergeInfo {
+/// Information needed to undo inlining actions.
+/// - the source block
+/// - the first inlined operation (could be null if the source block was empty)
+/// - the last inlined operation (could be null if the source block was empty)
+struct InlineInfo {
   Block *sourceBlock;
-  Operation *destBlockLastInst;
+  Operation *firstInlinedInst;
+  Operation *lastInlinedInst;
 };
 
 /// The storage class for an undoable block action (one of BlockActionKind),
@@ -320,9 +273,12 @@ struct BlockAction {
   static BlockAction getErase(Block *block, BlockPosition originalPosition) {
     return {BlockActionKind::Erase, block, {originalPosition}};
   }
-  static BlockAction getMerge(Block *block, Block *sourceBlock) {
-    BlockAction action{BlockActionKind::Merge, block, {}};
-    action.mergeInfo = {sourceBlock, block->empty() ? nullptr : &block->back()};
+  static BlockAction getInline(Block *block, Block *srcBlock,
+                               Block::iterator before) {
+    BlockAction action{BlockActionKind::Inline, block, {}};
+    action.inlineInfo = {srcBlock,
+                         srcBlock->empty() ? nullptr : &srcBlock->front(),
+                         srcBlock->empty() ? nullptr : &srcBlock->back()};
     return action;
   }
   static BlockAction getMove(Block *block, BlockPosition originalPosition) {
@@ -344,16 +300,16 @@ struct BlockAction {
   Block *block;
 
   union {
-    // In use if kind == BlockActionKind::Move or BlockActionKind::Erase, and
+    // In use if kind == BlockActionKind::Inline or BlockActionKind::Erase, and
     // contains a pointer to the region that originally contained the block as
     // well as the position of the block in that region.
     BlockPosition originalPosition;
     // In use if kind == BlockActionKind::Split and contains a pointer to the
     // block that was split into two parts.
     Block *originalBlock;
-    // In use if kind == BlockActionKind::Merge, and contains the information
-    // needed to undo the merge.
-    MergeInfo mergeInfo;
+    // In use if kind == BlockActionKind::Inline, and contains the information
+    // needed to undo the inlining.
+    InlineInfo inlineInfo;
   };
 };
 
@@ -945,17 +901,14 @@ struct ConversionPatternRewriterImpl {
   /// Notifies that a block was split.
   void notifySplitBlock(Block *block, Block *continuation);
 
-  /// Notifies that `block` is being merged with `srcBlock`.
-  void notifyBlocksBeingMerged(Block *block, Block *srcBlock);
+  /// Notifies that a block is being inlined into another block.
+  void notifyBlockBeingInlined(Block *block, Block *srcBlock,
+                               Block::iterator before);
 
   /// Notifies that the blocks of a region are about to be moved.
   void notifyRegionIsBeingInlinedBefore(Region &region, Region &parent,
                                         Region::iterator before);
 
-  /// Notifies that the blocks of a region were cloned into another.
-  void notifyRegionWasClonedBefore(iterator_range<Region::iterator> &blocks,
-                                   Location origRegionLoc);
-
   /// Notifies that a pattern match failed for the given reason.
   LogicalResult
   notifyMatchFailure(Location loc,
@@ -1213,18 +1166,17 @@ void ConversionPatternRewriterImpl::undoBlockActions(
                        action.block);
       break;
     }
-    // Split the block at the position which was originally the end of the
-    // destination block (owned by action), and put the instructions back into
-    // the block used before the merge.
-    case BlockActionKind::Merge: {
-      Block *sourceBlock = action.mergeInfo.sourceBlock;
-      Block::iterator splitPoint =
-          (action.mergeInfo.destBlockLastInst
-               ? ++Block::iterator(action.mergeInfo.destBlockLastInst)
-               : action.block->begin());
-      sourceBlock->getOperations().splice(sourceBlock->begin(),
-                                          action.block->getOperations(),
-                                          splitPoint, action.block->end());
+    // Put the instructions from the destination block (owned by the action)
+    // back into the source block.
+    case BlockActionKind::Inline: {
+      Block *sourceBlock = action.inlineInfo.sourceBlock;
+      if (action.inlineInfo.firstInlinedInst) {
+        assert(action.inlineInfo.lastInlinedInst && "expected operation");
+        sourceBlock->getOperations().splice(
+            sourceBlock->begin(), action.block->getOperations(),
+            Block::iterator(action.inlineInfo.firstInlinedInst),
+            ++Block::iterator(action.inlineInfo.lastInlinedInst));
+      }
       break;
     }
     // Move the block back to its original position.
@@ -1445,9 +1397,9 @@ void ConversionPatternRewriterImpl::notifySplitBlock(Block *block,
   blockActions.push_back(BlockAction::getSplit(continuation, block));
 }
 
-void ConversionPatternRewriterImpl::notifyBlocksBeingMerged(Block *block,
-                                                            Block *srcBlock) {
-  blockActions.push_back(BlockAction::getMerge(block, srcBlock));
+void ConversionPatternRewriterImpl::notifyBlockBeingInlined(
+    Block *block, Block *srcBlock, Block::iterator before) {
+  blockActions.push_back(BlockAction::getInline(block, srcBlock, before));
 }
 
 void ConversionPatternRewriterImpl::notifyRegionIsBeingInlinedBefore(
@@ -1463,20 +1415,6 @@ void ConversionPatternRewriterImpl::notifyRegionIsBeingInlinedBefore(
   blockActions.push_back(BlockAction::getMove(laterBlock, {&region, nullptr}));
 }
 
-void ConversionPatternRewriterImpl::notifyRegionWasClonedBefore(
-    iterator_range<Region::iterator> &blocks, Location origRegionLoc) {
-  for (Block &block : blocks)
-    blockActions.push_back(BlockAction::getCreate(&block));
-
-  // Compute the conversion set for the inlined region.
-  auto result = computeConversionSet(blocks, origRegionLoc, createdOps);
-
-  // This original region has already had its conversion set computed, so there
-  // shouldn't be any new failures.
-  (void)result;
-  assert(succeeded(result) && "expected region to have no unreachable blocks");
-}
-
 LogicalResult ConversionPatternRewriterImpl::notifyMatchFailure(
     Location loc, function_ref<void(Diagnostic &)> reasonCallback) {
   LLVM_DEBUG({
@@ -1603,17 +1541,23 @@ Block *ConversionPatternRewriter::splitBlock(Block *block,
   return continuation;
 }
 
-void ConversionPatternRewriter::mergeBlocks(Block *source, Block *dest,
-                                            ValueRange argValues) {
-  impl->notifyBlocksBeingMerged(dest, source);
-  assert(llvm::all_of(source->getPredecessors(),
-                      [dest](Block *succ) { return succ == dest; }) &&
-         "expected 'source' to have no predecessors or only 'dest'");
+void ConversionPatternRewriter::inlineBlockBefore(Block *source, Block *dest,
+                                                  Block::iterator before,
+                                                  ValueRange argValues) {
   assert(argValues.size() == source->getNumArguments() &&
          "incorrect # of argument replacement values");
+#ifndef NDEBUG
+  auto opIgnored = [&](Operation *op) { return impl->isOpIgnored(op); };
+#endif // NDEBUG
+  // The source block will be deleted, so it should not have any users (i.e.,
+  // there should be no predecessors).
+  assert(llvm::all_of(source->getUsers(), opIgnored) &&
+         "expected 'source' to have no predecessors");
+
+  impl->notifyBlockBeingInlined(dest, source, before);
   for (auto it : llvm::zip(source->getArguments(), argValues))
     replaceUsesOfBlockArgument(std::get<0>(it), std::get<1>(it));
-  dest->getOperations().splice(dest->end(), source->getOperations());
+  dest->getOperations().splice(before, source->getOperations());
   eraseBlock(source);
 }
 
@@ -1630,12 +1574,15 @@ void ConversionPatternRewriter::cloneRegionBefore(Region &region,
                                                   IRMapping &mapping) {
   if (region.empty())
     return;
+
   PatternRewriter::cloneRegionBefore(region, parent, before, mapping);
 
-  // Collect the range of the cloned blocks.
-  auto clonedBeginIt = mapping.lookup(&region.front())->getIterator();
-  auto clonedBlocks = llvm::make_range(clonedBeginIt, before);
-  impl->notifyRegionWasClonedBefore(clonedBlocks, region.getLoc());
+  for (Block &b : ForwardDominanceIterator<>::makeIterable(region)) {
+    Block *cloned = mapping.lookup(&b);
+    impl->notifyCreatedBlock(cloned);
+    cloned->walk<WalkOrder::PreOrder, ForwardDominanceIterator<>>(
+        [&](Operation *op) { notifyOperationInserted(op); });
+  }
 }
 
 void ConversionPatternRewriter::notifyOperationInserted(Operation *op) {
@@ -2444,11 +2391,16 @@ LogicalResult OperationConverter::convertOperations(
   // Compute the set of operations and blocks to convert.
   SmallVector<Operation *> toConvert;
   for (auto *op : ops) {
-    toConvert.emplace_back(op);
-    for (auto &region : op->getRegions())
-      if (failed(computeConversionSet(region.getBlocks(), region.getLoc(),
-                                      toConvert, &target)))
-        return failure();
+    op->walk<WalkOrder::PreOrder, ForwardDominanceIterator<>>(
+        [&](Operation *op) {
+          toConvert.push_back(op);
+          // Don't check this operation's children for conversion if the
+          // operation is recursively legal.
+          auto legalityInfo = target.isLegal(op);
+          if (legalityInfo && legalityInfo->isRecursivelyLegal)
+            return WalkResult::skip();
+          return WalkResult::advance();
+        });
   }
 
   // Convert each operation and discard rewrites on failure.
diff --git a/mlir/python/mlir/_mlir_libs/__init__.py b/mlir/python/mlir/_mlir_libs/__init__.py
index 9ceeef81844c2..7d3d1f6ca873a 100644
--- a/mlir/python/mlir/_mlir_libs/__init__.py
+++ b/mlir/python/mlir/_mlir_libs/__init__.py
@@ -100,8 +100,29 @@ def __init__(self, *args, **kwargs):
       # all dialects. It is being done here in order to preserve existing
       # behavior. See: https://github.com/llvm/llvm-project/issues/56037
       self.load_all_available_dialects()
-
   ir.Context = Context
 
+  class MLIRError(Exception):
+    """
+    An exception with diagnostic information. Has the following fields:
+      message: str
+      error_diagnostics: List[ir.DiagnosticInfo]
+    """
+    def __init__(self, message, error_diagnostics):
+      self.message = message
+      self.error_diagnostics = error_diagnostics
+      super().__init__(message, error_diagnostics)
+
+    def __str__(self):
+      s = self.message
+      if self.error_diagnostics:
+        s += ':'
+      for diag in self.error_diagnostics:
+        s += "\nerror: "  + str(diag.location)[4:-1] + ": " + diag.message.replace('\n', '\n  ')
+        for note in diag.notes:
+          s += "\n note: "  + str(note.location)[4:-1] + ": " + note.message.replace('\n', '\n  ')
+      return s
+  ir.MLIRError = MLIRError
+
 
 _site_initialize()
diff --git a/mlir/test/Bytecode/invalid/invalid-structure.mlir b/mlir/test/Bytecode/invalid/invalid-structure.mlir
index 8b70a3ff25d9a..d98c6c6191b87 100644
--- a/mlir/test/Bytecode/invalid/invalid-structure.mlir
+++ b/mlir/test/Bytecode/invalid/invalid-structure.mlir
@@ -9,7 +9,7 @@
 //===--------------------------------------------------------------------===//
 
 // RUN: not mlir-opt %S/invalid-structure-version.mlirbc 2>&1 | FileCheck %s --check-prefix=VERSION
-// VERSION: bytecode version 127 is newer than the current version 0
+// VERSION: bytecode version 127 is newer than the current version 1
 
 //===--------------------------------------------------------------------===//
 // Producer
diff --git a/mlir/test/Bytecode/versioning/versioned-attr-1.12.mlirbc b/mlir/test/Bytecode/versioning/versioned-attr-1.12.mlirbc
new file mode 100644
index 0000000000000..3645a2f37c4ba
Binary files /dev/null and b/mlir/test/Bytecode/versioning/versioned-attr-1.12.mlirbc differ
diff --git a/mlir/test/Bytecode/versioning/versioned-attr-2.0.mlirbc b/mlir/test/Bytecode/versioning/versioned-attr-2.0.mlirbc
new file mode 100644
index 0000000000000..83ff044d7f907
Binary files /dev/null and b/mlir/test/Bytecode/versioning/versioned-attr-2.0.mlirbc differ
diff --git a/mlir/test/Bytecode/versioning/versioned-op-1.12.mlirbc b/mlir/test/Bytecode/versioning/versioned-op-1.12.mlirbc
new file mode 100644
index 0000000000000..688948257008f
Binary files /dev/null and b/mlir/test/Bytecode/versioning/versioned-op-1.12.mlirbc differ
diff --git a/mlir/test/Bytecode/versioning/versioned-op-2.0.mlirbc b/mlir/test/Bytecode/versioning/versioned-op-2.0.mlirbc
new file mode 100644
index 0000000000000..fdc8e0d28b2f0
Binary files /dev/null and b/mlir/test/Bytecode/versioning/versioned-op-2.0.mlirbc differ
diff --git a/mlir/test/Bytecode/versioning/versioned-op-2.2.mlirbc b/mlir/test/Bytecode/versioning/versioned-op-2.2.mlirbc
new file mode 100644
index 0000000000000..2dc4719ca4975
Binary files /dev/null and b/mlir/test/Bytecode/versioning/versioned-op-2.2.mlirbc differ
diff --git a/mlir/test/Bytecode/versioning/versioned_attr.mlir b/mlir/test/Bytecode/versioning/versioned_attr.mlir
new file mode 100644
index 0000000000000..98756cc66a99c
--- /dev/null
+++ b/mlir/test/Bytecode/versioning/versioned_attr.mlir
@@ -0,0 +1,29 @@
+// This file contains a test case representative of a dialect parsing an
+// attribute with versioned custom encoding.
+
+// Bytecode currently does not support big-endian platforms
+// UNSUPPORTED: target=s390x-{{.*}}
+
+//===--------------------------------------------------------------------===//
+// Test attribute upgrade
+//===--------------------------------------------------------------------===//
+
+// COM: bytecode contains
+// COM: module {
+// COM:   version: 1.12
+// COM:   "test.versionedB"() {attribute = #test.attr_params<24, 42>} : () -> ()
+// COM: }
+// RUN: mlir-opt %S/versioned-attr-1.12.mlirbc 2>&1 | FileCheck %s --check-prefix=CHECK1
+// CHECK1: "test.versionedB"() {attribute = #test.attr_params<42, 24>} : () -> ()
+
+//===--------------------------------------------------------------------===//
+// Test attribute upgrade
+//===--------------------------------------------------------------------===//
+
+// COM: bytecode contains
+// COM: module {
+// COM:   version: 2.0
+// COM:   "test.versionedB"() {attribute = #test.attr_params<42, 24>} : () -> ()
+// COM: }
+// RUN: mlir-opt %S/versioned-attr-2.0.mlirbc 2>&1 | FileCheck %s --check-prefix=CHECK2
+// CHECK2: "test.versionedB"() {attribute = #test.attr_params<42, 24>} : () -> ()
diff --git a/mlir/test/Bytecode/versioning/versioned_op.mlir b/mlir/test/Bytecode/versioning/versioned_op.mlir
new file mode 100644
index 0000000000000..b3141de67823f
--- /dev/null
+++ b/mlir/test/Bytecode/versioning/versioned_op.mlir
@@ -0,0 +1,41 @@
+// This file contains test cases related to the dialect post-parsing upgrade
+// mechanism.
+
+// Bytecode currently does not support big-endian platforms
+// UNSUPPORTED: target=s390x-{{.*}}
+
+//===--------------------------------------------------------------------===//
+// Test generic
+//===--------------------------------------------------------------------===//
+
+// COM: bytecode contains
+// COM: module {
+// COM:   version: 2.0
+// COM:   "test.versionedA"() {dims = 123 : i64, modifier = false} : () -> ()
+// COM: }
+// RUN: mlir-opt %S/versioned-op-2.0.mlirbc 2>&1 | FileCheck %s --check-prefix=CHECK1
+// CHECK1: "test.versionedA"() {dims = 123 : i64, modifier = false} : () -> ()
+
+//===--------------------------------------------------------------------===//
+// Test upgrade
+//===--------------------------------------------------------------------===//
+
+// COM: bytecode contains
+// COM: module {
+// COM:   version: 1.12
+// COM:   "test.versionedA"() {dimensions = 123 : i64} : () -> ()
+// COM: }
+// RUN: mlir-opt %S/versioned-op-1.12.mlirbc 2>&1 | FileCheck %s --check-prefix=CHECK2
+// CHECK2: "test.versionedA"() {dims = 123 : i64, modifier = false} : () -> ()
+
+//===--------------------------------------------------------------------===//
+// Test forbidden downgrade
+//===--------------------------------------------------------------------===//
+
+// COM: bytecode contains
+// COM: module {
+// COM:   version: 2.2
+// COM:   "test.versionedA"() {dims = 123 : i64, modifier = false} : () -> ()
+// COM: }
+// RUN: not mlir-opt %S/versioned-op-2.2.mlirbc 2>&1 | FileCheck %s --check-prefix=ERR_NEW_VERSION
+// ERR_NEW_VERSION: current test dialect version is 2.0, can't parse version: 2.2
diff --git a/mlir/test/CAPI/sparse_tensor.c b/mlir/test/CAPI/sparse_tensor.c
index ebb76ebc34eb2..43e148985bf90 100644
--- a/mlir/test/CAPI/sparse_tensor.c
+++ b/mlir/test/CAPI/sparse_tensor.c
@@ -28,7 +28,7 @@ static int testRoundtripEncoding(MlirContext ctx) {
     "dimLevelType = [ \"dense\", \"compressed\", \"compressed\"], "
     "dimOrdering = affine_map<(d0, d1, d2) -> (d0, d1, d2)>, "
     "higherOrdering = affine_map<(d0, d1)[s0] -> (s0, d0, d1)>, "
-    "pointerBitWidth = 32, indexBitWidth = 64 }>";
+    "posWidth = 32, crdWidth = 64 }>";
   // clang-format on
   MlirAttribute originalAttr =
       mlirAttributeParseGet(ctx, mlirStringRefCreateFromCString(originalAsm));
@@ -46,26 +46,24 @@ static int testRoundtripEncoding(MlirContext ctx) {
   // CHECK: level_type: 4
   // CHECK: level_type: 8
   // CHECK: level_type: 8
-  int numLevelTypes = mlirSparseTensorEncodingGetNumDimLevelTypes(originalAttr);
+  int lvlRank = mlirSparseTensorEncodingGetLvlRank(originalAttr);
   enum MlirSparseTensorDimLevelType *levelTypes =
-      malloc(sizeof(enum MlirSparseTensorDimLevelType) * numLevelTypes);
-  for (int i = 0; i < numLevelTypes; ++i) {
-    levelTypes[i] =
-        mlirSparseTensorEncodingAttrGetDimLevelType(originalAttr, i);
-    fprintf(stderr, "level_type: %d\n", levelTypes[i]);
+      malloc(sizeof(enum MlirSparseTensorDimLevelType) * lvlRank);
+  for (int l = 0; l < lvlRank; ++l) {
+    levelTypes[l] =
+        mlirSparseTensorEncodingAttrGetDimLevelType(originalAttr, l);
+    fprintf(stderr, "level_type: %d\n", levelTypes[l]);
   }
-  // CHECK: pointer: 32
-  int pointerBitWidth =
-      mlirSparseTensorEncodingAttrGetPointerBitWidth(originalAttr);
-  fprintf(stderr, "pointer: %d\n", pointerBitWidth);
-  // CHECK: index: 64
-  int indexBitWidth =
-      mlirSparseTensorEncodingAttrGetIndexBitWidth(originalAttr);
-  fprintf(stderr, "index: %d\n", indexBitWidth);
+  // CHECK: posWidth: 32
+  int posWidth = mlirSparseTensorEncodingAttrGetPosWidth(originalAttr);
+  fprintf(stderr, "posWidth: %d\n", posWidth);
+  // CHECK: crdWidth: 64
+  int crdWidth = mlirSparseTensorEncodingAttrGetCrdWidth(originalAttr);
+  fprintf(stderr, "crdWidth: %d\n", crdWidth);
 
-  MlirAttribute newAttr = mlirSparseTensorEncodingAttrGet(
-      ctx, numLevelTypes, levelTypes, dimOrdering, higherOrdering,
-      pointerBitWidth, indexBitWidth);
+  MlirAttribute newAttr =
+      mlirSparseTensorEncodingAttrGet(ctx, lvlRank, levelTypes, dimOrdering,
+                                      higherOrdering, posWidth, crdWidth);
   mlirAttributeDump(newAttr); // For debugging filecheck output.
   // CHECK: equal: 1
   fprintf(stderr, "equal: %d\n", mlirAttributeEqual(originalAttr, newAttr));
diff --git a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
index 4c32a8648bd1c..d70df982c366a 100644
--- a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
+++ b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
@@ -179,6 +179,13 @@ func.func @int_vector234(%arg0: vector<2xi8>, %arg1: vector<4xi64>) {
   return
 }
 
+// CHECK-LABEL: @index_vector
+func.func @index_vector(%arg0: vector<4xindex>) {
+  // CHECK: spirv.UMod %{{.*}}, %{{.*}}: vector<4xi32>
+  %0 = arith.remui %arg0, %arg0: vector<4xindex>
+  return
+}
+
 // CHECK-LABEL: @vector_srem
 // CHECK-SAME: (%[[LHS:.+]]: vector<3xi16>, %[[RHS:.+]]: vector<3xi16>)
 func.func @vector_srem(%arg0: vector<3xi16>, %arg1: vector<3xi16>) {
@@ -417,6 +424,31 @@ func.func @cmpi(%arg0 : i32, %arg1 : i32) {
   return
 }
 
+// CHECK-LABEL: @indexcmpi
+func.func @indexcmpi(%arg0 : index, %arg1 : index) {
+  // CHECK: spirv.IEqual
+  %0 = arith.cmpi eq, %arg0, %arg1 : index
+  // CHECK: spirv.INotEqual
+  %1 = arith.cmpi ne, %arg0, %arg1 : index
+  // CHECK: spirv.SLessThan
+  %2 = arith.cmpi slt, %arg0, %arg1 : index
+  // CHECK: spirv.SLessThanEqual
+  %3 = arith.cmpi sle, %arg0, %arg1 : index
+  // CHECK: spirv.SGreaterThan
+  %4 = arith.cmpi sgt, %arg0, %arg1 : index
+  // CHECK: spirv.SGreaterThanEqual
+  %5 = arith.cmpi sge, %arg0, %arg1 : index
+  // CHECK: spirv.ULessThan
+  %6 = arith.cmpi ult, %arg0, %arg1 : index
+  // CHECK: spirv.ULessThanEqual
+  %7 = arith.cmpi ule, %arg0, %arg1 : index
+  // CHECK: spirv.UGreaterThan
+  %8 = arith.cmpi ugt, %arg0, %arg1 : index
+  // CHECK: spirv.UGreaterThanEqual
+  %9 = arith.cmpi uge, %arg0, %arg1 : index
+  return
+}
+
 // CHECK-LABEL: @vec1cmpi
 func.func @vec1cmpi(%arg0 : vector<1xi32>, %arg1 : vector<1xi32>) {
   // CHECK: spirv.ULessThan
diff --git a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
index 640cac1abfb37..5e8986a4a9a46 100644
--- a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
@@ -467,7 +467,7 @@ func.func @complex_sign(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK: %[[REAL_IS_ZERO:.*]] = arith.cmpf oeq, %[[REAL]], %[[ZERO]] : f32
-// CHECK: %[[IMAG_IS_ZERO:.*]] = arith.cmpf oeq, %1, %cst : f32
+// CHECK: %[[IMAG_IS_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %[[IS_ZERO:.*]] = arith.andi %[[REAL_IS_ZERO]], %[[IMAG_IS_ZERO]] : i1
 // CHECK: %[[REAL2:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG2:.*]] = complex.im %[[ARG]] : complex<f32>
@@ -682,7 +682,7 @@ func.func @complex_conj(%arg: complex<f32>) -> complex<f32> {
 
 // CHECK-LABEL:   func.func @complex_pow
 func.func @complex_pow(%lhs: complex<f32>,
-                         %rhs: complex<f32>) -> complex<f32> {
+                       %rhs: complex<f32>) -> complex<f32> {
   %pow = complex.pow %lhs, %rhs : complex<f32>
   return %pow : complex<f32>
 }
diff --git a/mlir/test/Conversion/GPUCommon/lower-launch-func-to-gpu-runtime-calls.mlir b/mlir/test/Conversion/GPUCommon/lower-launch-func-to-gpu-runtime-calls.mlir
index 1db1dc8c04b00..0bc1450edc73c 100644
--- a/mlir/test/Conversion/GPUCommon/lower-launch-func-to-gpu-runtime-calls.mlir
+++ b/mlir/test/Conversion/GPUCommon/lower-launch-func-to-gpu-runtime-calls.mlir
@@ -40,9 +40,18 @@ module attributes {gpu.container_module} {
 
   // CHECK: [[STREAM:%.*]] = llvm.call @mgpuStreamCreate
 
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(1 : i32)
+  // CHECK: %[[MEMREF:.*]] = llvm.alloca %[[ONE]] x !llvm.struct[[STRUCT_BODY:<.*>]]
   // CHECK: [[NUM_PARAMS:%.*]] = llvm.mlir.constant(6 : i32) : i32
   // CHECK-NEXT: [[PARAMS:%.*]] = llvm.alloca [[NUM_PARAMS]] x !llvm.ptr
 
+  // CHECK: llvm.getelementptr %[[MEMREF]][0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct[[STRUCT_BODY:<.*>]]
+  // CHECK: llvm.getelementptr %[[MEMREF]][0, 1] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct[[STRUCT_BODY:<.*>]]
+  // CHECK: llvm.getelementptr %[[MEMREF]][0, 2] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct[[STRUCT_BODY:<.*>]]
+  // CHECK: llvm.getelementptr %[[MEMREF]][0, 3] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct[[STRUCT_BODY:<.*>]]
+  // CHECK: llvm.getelementptr %[[MEMREF]][0, 4] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct[[STRUCT_BODY:<.*>]]
+  // CHECK: llvm.getelementptr %[[MEMREF]][0, 5] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct[[STRUCT_BODY:<.*>]]
+
   // CHECK: [[EXTRA_PARAMS:%.*]] = llvm.mlir.null : !llvm.ptr
 
   // CHECK: llvm.call @mgpuLaunchKernel([[FUNC]], [[C8]], [[C8]], [[C8]],
diff --git a/mlir/test/Conversion/MemRefToLLVM/memref-to-llvm.mlir b/mlir/test/Conversion/MemRefToLLVM/memref-to-llvm.mlir
index f6dc44cf4571f..4b4b3836a0075 100644
--- a/mlir/test/Conversion/MemRefToLLVM/memref-to-llvm.mlir
+++ b/mlir/test/Conversion/MemRefToLLVM/memref-to-llvm.mlir
@@ -362,16 +362,47 @@ func.func @generic_atomic_rmw(%I : memref<10xi32>, %i : index) {
     ^bb0(%old_value : i32):
       memref.atomic_yield %old_value : i32
   }
-  // CHECK: [[init:%.*]] = llvm.load %{{.*}} : !llvm.ptr -> i32
-  // CHECK-NEXT: llvm.br ^bb1([[init]] : i32)
-  // CHECK-NEXT: ^bb1([[loaded:%.*]]: i32):
-  // CHECK-NEXT: [[pair:%.*]] = llvm.cmpxchg %{{.*}}, [[loaded]], [[loaded]]
-  // CHECK-SAME:                    acq_rel monotonic : !llvm.ptr, i32
-  // CHECK-NEXT: [[new:%.*]] = llvm.extractvalue [[pair]][0]
-  // CHECK-NEXT: [[ok:%.*]] = llvm.extractvalue [[pair]][1]
-  // CHECK-NEXT: llvm.cond_br [[ok]], ^bb2, ^bb1([[new]] : i32)
   llvm.return
 }
+// CHECK:        %[[INIT:.*]] = llvm.load %{{.*}} : !llvm.ptr -> i32
+// CHECK-NEXT:   llvm.br ^bb1(%[[INIT]] : i32)
+// CHECK-NEXT: ^bb1(%[[LOADED:.*]]: i32):
+// CHECK-NEXT:   %[[PAIR:.*]] = llvm.cmpxchg %{{.*}}, %[[LOADED]], %[[LOADED]]
+// CHECK-SAME:                      acq_rel monotonic : !llvm.ptr, i32
+// CHECK-NEXT:   %[[NEW:.*]] = llvm.extractvalue %[[PAIR]][0]
+// CHECK-NEXT:   %[[OK:.*]] = llvm.extractvalue %[[PAIR]][1]
+// CHECK-NEXT:   llvm.cond_br %[[OK]], ^bb2, ^bb1(%[[NEW]] : i32)
+
+// -----
+
+// CHECK-LABEL: func @generic_atomic_rmw_in_alloca_scope
+func.func @generic_atomic_rmw_in_alloca_scope(){
+  %c1 = arith.constant 1 : index
+  %alloc = memref.alloc() : memref<2x3xi32>
+  memref.alloca_scope  {
+    %0 = memref.generic_atomic_rmw %alloc[%c1, %c1] : memref<2x3xi32> {
+    ^bb0(%arg0: i32):
+      memref.atomic_yield %arg0 : i32
+    }
+  }
+  return
+}
+// CHECK:        %[[STACK_SAVE:.*]] = llvm.intr.stacksave : !llvm.ptr
+// CHECK-NEXT:   llvm.br ^bb1
+// CHECK:      ^bb1:
+// CHECK:        %[[INIT:.*]] = llvm.load %[[BUF:.*]] : !llvm.ptr -> i32
+// CHECK-NEXT:   llvm.br ^bb2(%[[INIT]] : i32)
+// CHECK-NEXT: ^bb2(%[[LOADED:.*]]: i32):
+// CHECK-NEXT:   %[[PAIR:.*]] = llvm.cmpxchg %[[BUF]], %[[LOADED]], %[[LOADED]]
+// CHECK-SAME:     acq_rel monotonic : !llvm.ptr, i32
+// CHECK-NEXT:   %[[NEW:.*]] = llvm.extractvalue %[[PAIR]][0]
+// CHECK-NEXT:   %[[OK:.*]] = llvm.extractvalue %[[PAIR]][1]
+// CHECK-NEXT:   llvm.cond_br %[[OK]], ^bb3, ^bb2(%[[NEW]] : i32)
+// CHECK-NEXT: ^bb3:
+// CHECK-NEXT:   llvm.intr.stackrestore %[[STACK_SAVE]] : !llvm.ptr
+// CHECK-NEXT:   llvm.br ^bb4
+// CHECK-NEXT: ^bb4:
+// CHECK-NEXT:   return
 
 // -----
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm.mlir
index f4125847230de..dfbbd15393b04 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm.mlir
@@ -333,12 +333,11 @@ func.func @mma_sp_sync_f16_16832(%arg0: vector<4x2xf16>,
   // CHECK-NOT llvm.extractvalue
 
   // CHECK: %[[sparseMetadata:.+]] = llvm.bitcast %{{.+}} : vector<2xi16> to i32
-  // CHECK: %[[sparseSelector:.+]] = llvm.mlir.constant(0 : i32) : i32
 
   // CHECK: %[[d:.+]] = llvm.inline_asm has_side_effects asm_dialect = att
-  // CHECK-SAME: "mma.sp.sync.aligned.m16n8k32.row.col.f16.f16.f16.f16 {$0,$1},{$2,$3,$4,$5},{$6,$7,$8,$9},{$10,$11},$12,$13;"
-  // CHECK-SAME: "=r,=r,r,r,r,r,r,r,r,r,r,r,r,r"
-  // CHECK-SAME: %[[sparseMetadata]], %[[sparseSelector]] :
+  // CHECK-SAME: "mma.sp.sync.aligned.m16n8k32.row.col.f16.f16.f16.f16 {$0,$1},{$2,$3,$4,$5},{$6,$7,$8,$9},{$10,$11},$12,0x0;"
+  // CHECK-SAME: "=r,=r,r,r,r,r,r,r,r,r,r,r,r"
+  // CHECK-SAME: %[[sparseMetadata]] :
   // CHECK-SAME: -> !llvm.struct<(vector<2xf16>, vector<2xf16>)>
 
   %d = nvgpu.mma.sp.sync(%arg0, %arg1, %arg2) metadata(%arg3) {mmaShape = [16, 8, 32]} :
@@ -372,12 +371,11 @@ func.func @mma_sp_sync_f16_16816(%arg0: vector<2x2xf16>,
   // CHECK-NOT llvm.extractvalue
 
   // CHECK: %[[sparseMetadata:.+]] = llvm.bitcast %{{.+}} : vector<2xi16> to i32
-  // CHECK: %[[sparseSelector:.+]] = llvm.mlir.constant(0 : i32) : i32
 
   // CHECK: %[[d:.+]] = llvm.inline_asm has_side_effects asm_dialect = att
-  // CHECK-SAME: "mma.sp.sync.aligned.m16n8k16.row.col.f16.f16.f16.f16 {$0,$1},{$2,$3},{$4,$5},{$6,$7},$8,$9;"
-  // CHECK-SAME: "=r,=r,r,r,r,r,r,r,r,r"
-  // CHECK-SAME: %[[sparseMetadata]], %[[sparseSelector]] :
+  // CHECK-SAME: "mma.sp.sync.aligned.m16n8k16.row.col.f16.f16.f16.f16 {$0,$1},{$2,$3},{$4,$5},{$6,$7},$8,0x0;"
+  // CHECK-SAME: "=r,=r,r,r,r,r,r,r,r"
+  // CHECK-SAME: %[[sparseMetadata]] :
   // CHECK-SAME: -> !llvm.struct<(vector<2xf16>, vector<2xf16>)>
 
   %d = nvgpu.mma.sp.sync(%arg0, %arg1, %arg2) metadata(%arg3) {mmaShape = [16, 8, 16]} :
@@ -387,6 +385,29 @@ func.func @mma_sp_sync_f16_16816(%arg0: vector<2x2xf16>,
 
 // -----
 
+// CHECK-LABEL: func @mma_sp_sync_f16_16816_01(
+func.func @mma_sp_sync_f16_16816_01(%arg0: vector<2x2xf16>,
+                                    %arg1: vector<2x2xf16>,
+                                    %arg2: vector<2x2xf16>,
+                                    %arg3: vector<2xi16>) -> vector<2x2xf16> {
+  //
+  // As above, but with sparsity selection 0x01.
+  //
+  // CHECK: %[[sparseMetadata:.+]] = llvm.bitcast %{{.+}} : vector<2xi16> to i32
+  // CHECK: %[[d:.+]] = llvm.inline_asm has_side_effects asm_dialect = att
+  // CHECK-SAME: "mma.sp.sync.aligned.m16n8k16.row.col.f16.f16.f16.f16 {$0,$1},{$2,$3},{$4,$5},{$6,$7},$8,0x1;"
+  // CHECK-SAME: "=r,=r,r,r,r,r,r,r,r"
+  // CHECK-SAME: %[[sparseMetadata]] :
+  // CHECK-SAME: -> !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+
+  %d = nvgpu.mma.sp.sync(%arg0, %arg1, %arg2) metadata(%arg3)
+       {mmaShape = [16, 8, 16], sparsitySelector = 1 : i32} :
+       (vector<2x2xf16>, vector<2x2xf16>, vector<2x2xf16>) -> vector<2x2xf16>
+  return %d : vector<2x2xf16>
+}
+
+// -----
+
 // CHECK-LABEL: func @mma_sp_sync_i8_16864(
 func.func @mma_sp_sync_i8_16864(%arg0: vector<4x4xi8>,
                                 %arg1: vector<4x4xi8>,
@@ -413,12 +434,11 @@ func.func @mma_sp_sync_i8_16864(%arg0: vector<4x4xi8>,
   // CHECK-NOT llvm.extractvalue
 
   // CHECK: %[[sparseMetadata:.+]] = llvm.bitcast %{{.+}} : vector<2xi16> to i32
-  // CHECK: %[[sparseSelector:.+]] = llvm.mlir.constant(0 : i32) : i32
 
   // CHECK: %[[d:.+]] = llvm.inline_asm has_side_effects asm_dialect = att
-  // CHECK-SAME: "mma.sp.sync.aligned.m16n8k64.row.col.satfinite.s32.s8.s8.s32
-  // CHECK-SAME: "=r,=r,=r,=r,r,r,r,r,r,r,r,r,r,r,r,r,r,r"
-  // CHECK-SAME: %[[sparseMetadata]], %[[sparseSelector]] :
+  // CHECK-SAME: "mma.sp.sync.aligned.m16n8k64.row.col.satfinite.s32.s8.s8.s32 {$0,$1,$2,$3},{$4,$5,$6,$7},{$8,$9,$10,$11},{$12,$13,$14,$15},$16,0x0;"
+  // CHECK-SAME: "=r,=r,=r,=r,r,r,r,r,r,r,r,r,r,r,r,r,r"
+  // CHECK-SAME: %[[sparseMetadata]] :
   // CHECK-SAME: -> !llvm.struct<(i32, i32, i32, i32)
 
   %d = nvgpu.mma.sp.sync(%arg0, %arg1, %arg2) metadata(%arg3) {mmaShape = [16, 8, 64]} :
diff --git a/mlir/test/Conversion/OpenMPToLLVM/convert-to-llvmir.mlir b/mlir/test/Conversion/OpenMPToLLVM/convert-to-llvmir.mlir
index 9db9e49859d85..3b683f36a870e 100644
--- a/mlir/test/Conversion/OpenMPToLLVM/convert-to-llvmir.mlir
+++ b/mlir/test/Conversion/OpenMPToLLVM/convert-to-llvmir.mlir
@@ -256,3 +256,66 @@ llvm.func @_QPsb() {
   }
   llvm.return
 }
+
+// -----
+
+// CHECK:  omp.reduction.declare @eqv_reduction : i32 init
+// CHECK:  ^bb0(%{{.*}}: i32):
+// CHECK:    %[[TRUE:.*]] = llvm.mlir.constant(true) : i1
+// CHECK:    %[[TRUE_EXT:.*]] = llvm.zext %[[TRUE]] : i1 to i32
+// CHECK:    omp.yield(%[[TRUE_EXT]] : i32)
+// CHECK:  } combiner {
+// CHECK:  ^bb0(%[[ARG_1:.*]]: i32, %[[ARG_2:.*]]: i32):
+// CHECK:    %[[ZERO:.*]] = llvm.mlir.constant(0 : i64) : i32
+// CHECK:    %[[CMP_1:.*]] = llvm.icmp "ne" %[[ARG_1]], %[[ZERO]] : i32
+// CHECK:    %[[CMP_2:.*]] = llvm.icmp "ne" %[[ARG_2]], %[[ZERO]] : i32
+// CHECK:    %[[COMBINE_VAL:.*]] = llvm.icmp "eq" %[[CMP_1]], %[[CMP_2]] : i1
+// CHECK:    %[[COMBINE_VAL_EXT:.*]] = llvm.zext %[[COMBINE_VAL]] : i1 to i32
+// CHECK:    omp.yield(%[[COMBINE_VAL_EXT]] : i32)
+// CHECK-LABEL:  @_QPsimple_reduction
+// CHECK:    %[[RED_ACCUMULATOR:.*]] = llvm.alloca %{{.*}} x i32 {bindc_name = "x", uniq_name = "_QFsimple_reductionEx"} : (i64) -> !llvm.ptr<i32>
+// CHECK:    omp.parallel
+// CHECK:      omp.wsloop reduction(@eqv_reduction -> %[[RED_ACCUMULATOR]] : !llvm.ptr<i32>) for
+// CHECK:        omp.reduction %{{.*}}, %[[RED_ACCUMULATOR]] : i32, !llvm.ptr<i32>
+// CHECK:        omp.yield
+// CHECK:      omp.terminator
+// CHECK:    llvm.return
+
+omp.reduction.declare @eqv_reduction : i32 init {
+^bb0(%arg0: i32):
+  %0 = llvm.mlir.constant(true) : i1
+  %1 = llvm.zext %0 : i1 to i32
+  omp.yield(%1 : i32)
+} combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %0 = llvm.mlir.constant(0 : i64) : i32
+  %1 = llvm.icmp "ne" %arg0, %0 : i32
+  %2 = llvm.icmp "ne" %arg1, %0 : i32
+  %3 = llvm.icmp "eq" %1, %2 : i1
+  %4 = llvm.zext %3 : i1 to i32
+  omp.yield(%4 : i32)
+}
+llvm.func @_QPsimple_reduction(%arg0: !llvm.ptr<array<100 x i32>> {fir.bindc_name = "y"}) {
+  %0 = llvm.mlir.constant(100 : i32) : i32
+  %1 = llvm.mlir.constant(1 : i32) : i32
+  %2 = llvm.mlir.constant(true) : i1
+  %3 = llvm.mlir.constant(1 : i64) : i64
+  %4 = llvm.alloca %3 x i32 {bindc_name = "x", uniq_name = "_QFsimple_reductionEx"} : (i64) -> !llvm.ptr<i32>
+  %5 = llvm.zext %2 : i1 to i32
+  llvm.store %5, %4 : !llvm.ptr<i32>
+  omp.parallel   {
+    %6 = llvm.alloca %3 x i32 {adapt.valuebyref, in_type = i32, operand_segment_sizes = array<i32: 0, 0>, pinned} : (i64) -> !llvm.ptr<i32>
+    omp.wsloop   reduction(@eqv_reduction -> %4 : !llvm.ptr<i32>) for  (%arg1) : i32 = (%1) to (%0) inclusive step (%1) {
+      llvm.store %arg1, %6 : !llvm.ptr<i32>
+      %7 = llvm.load %6 : !llvm.ptr<i32>
+      %8 = llvm.sext %7 : i32 to i64
+      %9 = llvm.sub %8, %3  : i64
+      %10 = llvm.getelementptr %arg0[0, %9] : (!llvm.ptr<array<100 x i32>>, i64) -> !llvm.ptr<i32>
+      %11 = llvm.load %10 : !llvm.ptr<i32>
+      omp.reduction %11, %4 : i32, !llvm.ptr<i32>
+      omp.yield
+    }
+    omp.terminator
+  }
+  llvm.return
+}
diff --git a/mlir/test/Conversion/SCFToOpenMP/reductions.mlir b/mlir/test/Conversion/SCFToOpenMP/reductions.mlir
index 4cf5f1b0f753c..c5723d96d4f1d 100644
--- a/mlir/test/Conversion/SCFToOpenMP/reductions.mlir
+++ b/mlir/test/Conversion/SCFToOpenMP/reductions.mlir
@@ -81,6 +81,43 @@ func.func @reduction2(%arg0 : index, %arg1 : index, %arg2 : index,
 
 // -----
 
+// Check the generation of declaration for arith.muli.
+// Mostly, the same check as above, except for the types,
+// the name of the op and the init value.
+
+// CHECK: omp.reduction.declare @[[$REDI:.*]] : i32
+
+// CHECK: init
+// CHECK: %[[INIT:.*]] = llvm.mlir.constant(1 : i32)
+// CHECK: omp.yield(%[[INIT]] : i32)
+
+// CHECK: combiner
+// CHECK: ^{{.*}}(%[[ARG0:.*]]: i32, %[[ARG1:.*]]: i32)
+// CHECK: %[[RES:.*]] = arith.muli %[[ARG0]], %[[ARG1]]
+// CHECK: omp.yield(%[[RES]] : i32)
+
+// CHECK-NOT: atomic
+
+// CHECK-LABEL: @reduction_muli
+func.func @reduction_muli(%arg0 : index, %arg1 : index, %arg2 : index,
+                 %arg3 : index, %arg4 : index) {
+  %step = arith.constant 1 : index
+  %one = arith.constant 1 : i32
+  scf.parallel (%i0, %i1) = (%arg0, %arg1) to (%arg2, %arg3)
+                            step (%arg4, %step) init (%one) -> (i32) {
+    // CHECK: omp.reduction
+    %pow2 = arith.constant 2 : i32
+    scf.reduce(%pow2) : i32 {
+    ^bb0(%lhs : i32, %rhs: i32):
+      %res = arith.muli %lhs, %rhs : i32
+      scf.reduce.return %res : i32
+    }
+  }
+  return
+}
+
+// -----
+
 // Only check the declaration here, the rest is same as above.
 // CHECK: omp.reduction.declare @{{.*}} : f32
 
diff --git a/mlir/test/Conversion/SCFToSPIRV/unsupported.mlir b/mlir/test/Conversion/SCFToSPIRV/unsupported.mlir
new file mode 100644
index 0000000000000..6f388f366f744
--- /dev/null
+++ b/mlir/test/Conversion/SCFToSPIRV/unsupported.mlir
@@ -0,0 +1,21 @@
+// RUN: mlir-opt -convert-scf-to-spirv %s -o - | FileCheck %s
+
+// `scf.parallel` conversion is not supported yet.
+// Make sure that we do not accidentally invalidate this functio by removing
+// `scf.yield`.
+// CHECK-LABEL: func.func @func
+// CHECK:         scf.parallel
+// CHECK-NEXT:      spirv.Constant
+// CHECK-NEXT:      memref.store
+// CHECK-NEXT:      scf.yield
+// CHECK:         spirv.Return
+func.func @func(%arg0: i64) {
+  %0 = arith.index_cast %arg0 : i64 to index
+  %alloc = memref.alloc() : memref<16xf32>
+  scf.parallel (%arg1) = (%0) to (%0) step (%0) {
+    %cst = arith.constant 1.000000e+00 : f32
+    memref.store %cst, %alloc[%arg1] : memref<16xf32>
+    scf.yield
+  }
+  return
+}
diff --git a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
index d3a84eb513ab4..133999eff1ec3 100644
--- a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
+++ b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
@@ -96,7 +96,7 @@ func.func @test_abs_dyn(%arg0: tensor<2x?xf32>) -> tensor<2x?xf32> {
 // CHECK-SAME: %[[ARG1:[0-9a-zA-Z_]*]]: tensor<2xf32>
 func.func @test_broadcast(%arg0: tensor<1xf32>, %arg1: tensor<2xf32>) -> tensor<2xf32> {
   // CHECK: [[INIT:%.+]] = tensor.empty() : tensor<2xf32>
-  // CHECK: [[RESHAPE:%.+]] = tensor.collapse_shape %[[ARG0]]
+  // CHECK: [[RESHAPE:%.+]] = "tosa.reshape"(%[[ARG0]])
   // CHECK: [[GENERIC:%.+]] = linalg.generic {indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP1]]], iterator_types = ["parallel"]} ins([[RESHAPE]], %[[ARG1]] : tensor<f32>, tensor<2xf32>) outs([[INIT]] : tensor<2xf32>) {
   // CHECK: ^bb0(%[[ARG2:.*]]: f32, %[[ARG3:.*]]: f32, %[[ARG4:.*]]: f32):
   // CHECK:   [[ELEMENT:%.+]] = arith.addf %[[ARG2]], %[[ARG3]] : f32
@@ -116,7 +116,7 @@ func.func @test_broadcast(%arg0: tensor<1xf32>, %arg1: tensor<2xf32>) -> tensor<
 // CHECK-SAME: %[[ARG1:[0-9a-zA-Z_]*]]: tensor<1xf32>
 func.func @test_broadcast_swapped_args(%arg0: tensor<2xf32>, %arg1: tensor<1xf32>) -> tensor<2xf32> {
   // CHECK: [[INIT:%.+]] = tensor.empty() : tensor<2xf32>
-  // CHECK: [[RESHAPE:%.+]] = tensor.collapse_shape %[[ARG1]]
+  // CHECK: [[RESHAPE:%.+]] = "tosa.reshape"(%[[ARG1]])
   // CHECK: [[GENERIC:%.+]] = linalg.generic {indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP0]]], iterator_types = ["parallel"]} ins(%[[ARG0]], [[RESHAPE]] : tensor<2xf32>, tensor<f32>) outs([[INIT]] : tensor<2xf32>) {
   // CHECK: ^bb0(%[[ARG2:.*]]: f32, %[[ARG3:.*]]: f32, %[[ARG4:.*]]: f32):
   // CHECK:   [[ELEMENT:%.+]] = arith.addf %[[ARG2]], %[[ARG3]] : f32
@@ -137,8 +137,8 @@ func.func @test_broadcast_swapped_args(%arg0: tensor<2xf32>, %arg1: tensor<1xf32
 // CHECK-SAME:  %[[ARG1:[0-9a-zA-Z_]*]]
 func.func @test_multibroadcast(%arg0: tensor<1x3xf32>, %arg1: tensor<2x1xf32>) -> tensor<2x3xf32> {
   // CHECK: [[INIT:%.+]] = tensor.empty() : tensor<2x3xf32>
-  // CHECK: [[RESHAPE1:%.+]] = tensor.collapse_shape %[[ARG0]] {{\[}}[0, 1]]
-  // CHECK: [[RESHAPE2:%.+]] = tensor.collapse_shape %[[ARG1]] {{\[}}[0, 1]]
+  // CHECK: [[RESHAPE1:%.+]] = "tosa.reshape"(%[[ARG0]]) {new_shape = array<i64: 3>}
+  // CHECK: [[RESHAPE2:%.+]] = "tosa.reshape"(%[[ARG1]]) {new_shape = array<i64: 2>}
   // CHECK: [[GENERIC:%.+]] = linalg.generic {indexing_maps = [#[[$MAP1]], #[[$MAP2]], #[[$MAP0]]], iterator_types = ["parallel", "parallel"]} ins([[RESHAPE1]], [[RESHAPE2]] : tensor<3xf32>, tensor<2xf32>) outs([[INIT]] : tensor<2x3xf32>) {
   // CHECK: ^bb0(%[[ARG2:.*]]: f32, %[[ARG3:.*]]: f32, %[[ARG4:.*]]: f32):
   // CHECK:   [[ELEMENT:%.+]] = arith.addf %[[ARG2]], %[[ARG3]] : f32
@@ -536,94 +536,6 @@ func.func @test_negate_quantized(%arg0: tensor<1xi8>) -> () {
   return
 }
 
-// -----
-
-// CHECK-LABEL: @test_reshape_downrank
-// CHECK-SAME: (%[[ARG0:[0-9a-zA-Z_]*]]
-func.func @test_reshape_downrank(%arg0: tensor<2x3xf32>) -> tensor<6xf32> {
-  // CHECK: [[RESHAPE:%.+]] = tensor.collapse_shape %[[ARG0]] {{\[}}[0, 1]]
-  %0 = "tosa.reshape"(%arg0) {new_shape = array<i64: 6>} : (tensor<2x3xf32>) -> tensor<6xf32>
-  // CHECK: return [[RESHAPE]]
-  return %0 : tensor<6xf32>
-}
-
-// -----
-
-// CHECK-LABEL: @test_reshape_downrank_dyn
-// CHECK-SAME: (%[[ARG0:[0-9a-zA-Z_]*]]
-func.func @test_reshape_downrank_dyn(%arg0: tensor<2x?xf32>) -> tensor<?xf32> {
-  // CHECK: [[RESHAPE:%.+]] = tensor.collapse_shape %[[ARG0]] {{\[}}[0, 1]]
-  %0 = "tosa.reshape"(%arg0) {new_shape = array<i64: -1>} : (tensor<2x?xf32>) -> tensor<?xf32>
-  // CHECK: return [[RESHAPE]]
-  return %0 : tensor<?xf32>
-}
-
-// -----
-
-// CHECK-LABEL: @test_reshape_uprank
-// CHECK-SAME: (%[[ARG0:[0-9a-zA-Z_]*]]
-func.func @test_reshape_uprank(%arg0: tensor<6xf32>) -> tensor<2x3xf32> {
-  // CHECK: [[RESHAPE:%.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0, 1]]
-  %0 = "tosa.reshape"(%arg0) {new_shape = array<i64: 2, 3>} : (tensor<6xf32>) -> tensor<2x3xf32>
-  // CHECK: return [[RESHAPE]]
-  return %0 : tensor<2x3xf32>
-}
-
-// -----
-
-// CHECK-LABEL: @test_reshape_uprank_dyn
-// CHECK-SAME: (%[[ARG0:[0-9a-zA-Z_]*]]
-func.func @test_reshape_uprank_dyn(%arg0: tensor<?xf32>) -> tensor<2x?xf32> {
-  // CHECK: [[RESHAPE:%.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0, 1]]
-  %0 = "tosa.reshape"(%arg0) {new_shape = array<i64: 2, -1>} : (tensor<?xf32>) -> tensor<2x?xf32>
-  // CHECK: return [[RESHAPE]]
-  return %0 : tensor<2x?xf32>
-}
-
-// -----
-
-// CHECK-LABEL: @test_reshape_samerank
-//  CHECK-SAME: (%[[ARG0:.*]]: tensor<3x2xf32>)
-func.func @test_reshape_samerank(%arg0: tensor<3x2xf32>) -> tensor<2x3xf32> {
-  // CHECK-NEXT: %[[RESHAPE1:.*]] = tensor.collapse_shape %[[ARG0]] {{\[}}[0, 1]]
-  // CHECK-NEXT: %[[RESHAPE2:.*]] = tensor.expand_shape %[[RESHAPE1]] {{\[}}[0, 1]]
-  %0 = "tosa.reshape"(%arg0) {new_shape = array<i64: 2, 3>} : (tensor<3x2xf32>) -> tensor<2x3xf32>
-  // CHECK-NEXT: return %[[RESHAPE2]]
-  return %0 : tensor<2x3xf32>
-}
-
-// -----
-
-// CHECK-LABEL: @test_reshape_samerank_dyn
-//  CHECK-SAME: (%[[ARG0:.*]]: tensor<?x2xf32>)
-func.func @test_reshape_samerank_dyn(%arg0: tensor<?x2xf32>) -> tensor<2x?xf32> {
-  // CHECK-NEXT: %[[RESHAPE1:.*]] = tensor.collapse_shape %[[ARG0]] {{\[}}[0, 1]]
-  // CHECK-NEXT: %[[RESHAPE2:.*]] = tensor.expand_shape %[[RESHAPE1]] {{\[}}[0, 1]]
-  %0 = "tosa.reshape"(%arg0) {new_shape = array<i64: 2, -1>} : (tensor<?x2xf32>) -> tensor<2x?xf32>
-  // CHECK-NEXT: return %[[RESHAPE2]]
-  return %0 : tensor<2x?xf32>
-}
-
-// -----
-
-// CHECK-LABEL: @test_reshape_downrank_6D
-// CHECK-SAME: (%[[ARG0:[0-9a-zA-Z_]*]]:
-func.func @test_reshape_downrank_6D(%arg0: tensor<1x2x3x5x7x11xf32>) -> tensor<6x5x77xf32> {
-  // CHECK: tensor.collapse_shape %[[ARG0]] {{\[}}[0, 1, 2], [3], [4, 5]]
-  %0 = "tosa.reshape"(%arg0) {new_shape = array<i64: 6, 5, 77>} : (tensor<1x2x3x5x7x11xf32>) -> tensor<6x5x77xf32>
-  return %0 : tensor<6x5x77xf32>
-}
-
-// -----
-
-// CHECK-LABEL: @test_reshape_downrank_6D_dyn
-// CHECK-SAME: (%[[ARG0:[0-9a-zA-Z_]*]]:
-func.func @test_reshape_downrank_6D_dyn(%arg0: tensor<1x2x?x5x7x11xf32>) -> tensor<?x5x77xf32> {
-  // CHECK: tensor.collapse_shape %[[ARG0]] {{\[}}[0, 1, 2, 3, 4, 5]]
-  // CHECK: tensor.expand_shape %{{.*}} {{\[}}[0, 1, 2]]
-  %0 = "tosa.reshape"(%arg0) {new_shape = array<i64: -1, 5, 77>} : (tensor<1x2x?x5x7x11xf32>) -> tensor<?x5x77xf32>
-  return %0 : tensor<?x5x77xf32>
-}
 
 // -----
 
@@ -911,79 +823,6 @@ func.func @reduce_bool(%arg0: tensor<5x4xi1>) -> () {
   return
 }
 
-// -----
-
-// CHECK-LABEL: @concat
-// CHECK-SAME: %[[ARG0:.+]]: tensor<5x1xf32>
-// CHECK-SAME: %[[ARG1:.+]]: tensor<6x1xf32>
-func.func @concat(%arg0: tensor<5x1xf32>, %arg1: tensor<6x1xf32>) -> () {
-  // CHECK: [[AXIS:%.+]] = arith.constant 0
-  // CHECK: [[STRIDE:%.+]]   = arith.constant 1
-  // CHECK: [[OFFSET:%.+]] = arith.constant 0 : index
-  // CHECK: [[IDX0:%.+]] = arith.constant 0 : index
-  // CHECK: [[IDX1:%.+]] = arith.constant 1 : index
-  // CHECK: [[INIT:%.+]] = tensor.empty() : tensor<11x1xf32>
-  // CHECK: [[INSERT0:%.+]] = tensor.insert_slice %[[ARG0]] into [[INIT]][0, 0] [5, 1] [1, 1]
-  // CHECK: [[INSERT1:%.+]] = tensor.insert_slice %[[ARG1]] into [[INSERT0]][5, 0] [6, 1] [1, 1]
-  %0 = "tosa.concat"(%arg0, %arg1) { axis = 0 : i64} : (tensor<5x1xf32>, tensor<6x1xf32>)  -> (tensor<11x1xf32>)
-
-  // CHECK: [[AXIS:%.+]] = arith.constant 1
-  // CHECK: [[STRIDE:%.+]]   = arith.constant 1
-  // CHECK: [[OFFSET:%.+]] = arith.constant 0 : index
-  // CHECK: [[IDX0:%.+]] = arith.constant 0 : index
-  // CHECK: [[IDX1:%.+]] = arith.constant 1 : index
-  // CHECK: [[INIT:%.+]] = tensor.empty() : tensor<5x2xf32>
-  // CHECK: [[INSERT0:%.+]] = tensor.insert_slice %[[ARG0]] into [[INIT]][0, 0] [5, 1] [1, 1]
-  // CHECK: [[INSERT1:%.+]] = tensor.insert_slice %[[ARG0]] into [[INSERT0]][0, 1] [5, 1] [1, 1]
-  %1 = "tosa.concat"(%arg0, %arg0) { axis = 1 : i64} : (tensor<5x1xf32>, tensor<5x1xf32>)  -> (tensor<5x2xf32>)
-  return
-}
-
-// -----
-
-// CHECK-LABEL: @concat_non_axis_dyn
-// CHECK-SAME: (%[[ARG0:[0-9a-zA-Z_]*]]:
-// CHECK-SAME:  %[[ARG1:[0-9a-zA-Z_]*]]
-func.func @concat_non_axis_dyn(%arg0: tensor<5x?xf32>, %arg1: tensor<6x?xf32>) -> () {
-  // CHECK: %[[AXIS:.+]] = arith.constant 0
-  // CHECK: %[[STRIDE:.+]]   = arith.constant 1
-  // CHECK: %[[OFFSET:.+]] = arith.constant 0 : index
-  // CHECK: %[[IDX0:.+]] = arith.constant 0 : index
-  // CHECK: %[[IDX1:.+]] = arith.constant 1 : index
-  // CHECK: %[[SIZE:.+]] = tensor.dim %[[ARG0]], %[[IDX1]]
-  // CHECK: %[[IDX1_2:.+]] = arith.constant 1 : index
-  // CHECK: %[[DYN:.+]] = tensor.dim %[[ARG0]], %[[IDX1_2]]
-  // CHECK: %[[INIT:.+]] = tensor.empty(%[[DYN]]) : tensor<11x?xf32>
-  // CHECK: %[[INSERT0:.+]] = tensor.insert_slice %[[ARG0]] into %[[INIT]][0, 0] [5, %[[SIZE]]] [1, 1]
-  // CHECK: %[[INSERT1:.+]] = tensor.insert_slice %[[ARG1]] into %[[INSERT0]][5, 0] [6, %[[SIZE]]] [1, 1]
-  %0 = "tosa.concat"(%arg0, %arg1) { axis = 0 : i64} : (tensor<5x?xf32>, tensor<6x?xf32>)  -> (tensor<11x?xf32>)
-  return
-}
-
-// -----
-
-// CHECK-LABEL: @concat_axis_dyn
-// CHECK-SAME: (%[[ARG0:[0-9a-zA-Z_]*]]:
-// CHECK-SAME:  %[[ARG1:[0-9a-zA-Z_]*]]:
-func.func @concat_axis_dyn(%arg0: tensor<?x3xf32>, %arg1: tensor<?x3xf32>) -> () {
-  // CHECK: %[[AXIS:.+]] = arith.constant 0
-  // CHECK: %[[STRIDE:.+]]   = arith.constant 1
-  // CHECK: %[[OFFSET:.+]] = arith.constant 0 : index
-  // CHECK: %[[IDX0:.+]] = arith.constant 0 : index
-  // CHECK: %[[SIZE:.+]] = tensor.dim %[[ARG0]], %[[IDX0]]
-  // CHECK: %[[IDX0_2:.+]] = arith.constant 0 : index
-  // CHECK: %[[DYN:.+]] = tensor.dim %[[ARG0]], %[[IDX0_2]]
-  // CHECK: %[[IDX1:.+]] = arith.constant 1 : index
-  // CHECK: %[[INIT:.+]] = tensor.empty(%[[DYN]]) : tensor<?x3xf32>
-  // CHECK: %[[DYN1:.+]] = tensor.dim %[[ARG0]], %[[AXIS]]
-  // CHECK: %[[INSERT0:.+]] = tensor.insert_slice %[[ARG0]] into %[[INIT]][0, 0] [%[[DYN1]], 3] [1, 1]
-  // CHECK: %[[SUM:.+]]  = arith.addi %[[OFFSET]], %[[DYN1]]
-  // CHECK: %[[DYN2:.+]] = tensor.dim %[[ARG1]], %[[AXIS]]
-  // CHECK: %[[INSERT1:.+]] = tensor.insert_slice %[[ARG1]] into %[[INSERT0]][%[[SUM]], 0] [%[[DYN2]], 3] [1, 1]
-  %0 = "tosa.concat"(%arg0, %arg1) { axis = 0 : i64} : (tensor<?x3xf32>, tensor<?x3xf32>)  -> (tensor<?x3xf32>)
-  return
-}
-
 // -----
 // CHECK: #[[$MAP0:.*]] = affine_map<(d0) -> (d0)>
 
@@ -1231,21 +1070,21 @@ func.func @tile(%arg0 : tensor<2x3xi8>) -> () {
   // CHECK: [[GENERIC:%.+]] = linalg.generic {indexing_maps = [#[[$MAP0]], #[[$MAP1]]], iterator_types = ["parallel", "parallel", "parallel", "parallel"]} ins(%[[ARG0]] : tensor<2x3xi8>) outs([[INIT]] : tensor<2x2x1x3xi8>)
   // CHECK: ^bb0(%[[ARG1:[0-9a-zA-Z_]+]]: i8
   // CHECK:   linalg.yield %[[ARG1]] : i8
-  // CHECK: tensor.collapse_shape [[GENERIC]] {{\[}}[0, 1, 2], [3]]
+  // CHECK: "tosa.reshape"([[GENERIC]]) {new_shape = array<i64: 4, 3>}
   %0 = "tosa.tile"(%arg0) {multiples = array<i64: 2, 1>} : (tensor<2x3xi8>)  -> (tensor<4x3xi8>)
 
   // CHECK: [[INIT:%.+]] = tensor.empty()
   // CHECK: [[GENERIC:%.+]] = linalg.generic {indexing_maps = [#[[$MAP0]], #[[$MAP1]]], iterator_types = ["parallel", "parallel", "parallel", "parallel"]} ins(%[[ARG0]] : tensor<2x3xi8>) outs([[INIT]] : tensor<1x2x2x3xi8>)
   // CHECK: ^bb0(%[[ARG1:[0-9a-zA-Z_]+]]: i8
   // CHECK:   linalg.yield %[[ARG1]] : i8
-  // CHECK: tensor.collapse_shape [[GENERIC]] {{\[}}[0, 1], [2, 3]]
+  // CHECK: "tosa.reshape"([[GENERIC]]) {new_shape = array<i64: 2, 6>}
   %1 = "tosa.tile"(%arg0) {multiples = array<i64: 1, 2>} : (tensor<2x3xi8>)  -> (tensor<2x6xi8>)
 
   // CHECK: [[INIT:%.+]] = tensor.empty()
   // CHECK: [[GENERIC:%.+]] = linalg.generic {indexing_maps = [#[[$MAP0]], #[[$MAP1]]], iterator_types = ["parallel", "parallel", "parallel", "parallel"]} ins(%[[ARG0]] : tensor<2x3xi8>) outs([[INIT]] : tensor<5x2x7x3xi8>)
   // CHECK: ^bb0(%[[ARG1:[0-9a-zA-Z_]+]]: i8
   // CHECK:   linalg.yield %[[ARG1]] : i8
-  // CHECK: tensor.collapse_shape [[GENERIC]] {{\[}}[0, 1], [2, 3]]
+  // CHECK: "tosa.reshape"([[GENERIC]]) {new_shape = array<i64: 10, 21>}
   %2 = "tosa.tile"(%arg0) {multiples = array<i64: 5, 7>} : (tensor<2x3xi8>)  -> (tensor<10x21xi8>)
 
   return
@@ -1265,8 +1104,7 @@ func.func @tile_dyn_input(%arg0 : tensor<?x3xi8>) -> () {
   // CHECK: %[[GENERIC:.+]] = linalg.generic {indexing_maps = [#[[$MAP0]], #[[$MAP1]]], iterator_types = ["parallel", "parallel", "parallel", "parallel"]} ins(%[[ARG0]] : tensor<?x3xi8>) outs(%[[INIT]] : tensor<2x?x1x3xi8>)
   // CHECK: ^bb0(%[[ARG1:.+]]: i8,
   // CHECK:   linalg.yield %[[ARG1]] : i8
-  // CHECK: %[[COLLAPSED:.+]] = tensor.collapse_shape %[[GENERIC]] {{\[}}[0, 1, 2, 3]]
-  // CHECK: tensor.expand_shape %[[COLLAPSED]] {{\[}}[0, 1]]
+  // CHECK: "tosa.reshape"(%[[GENERIC]]) {new_shape = array<i64: -9223372036854775808, 3>}
   %0 = "tosa.tile"(%arg0) {multiples = array<i64: 2, 1>} : (tensor<?x3xi8>)  -> (tensor<?x3xi8>)
 
   return
@@ -1286,8 +1124,7 @@ func.func @tile_dyn_multiples(%arg0 : tensor<2x3xi8>) -> () {
   // CHECK: %[[GENERIC:.+]] = linalg.generic {indexing_maps = [#[[$MAP0]], #[[$MAP1]]], iterator_types = ["parallel", "parallel", "parallel", "parallel"]} ins(%[[ARG0]] : tensor<2x3xi8>) outs(%[[INIT]] : tensor<2x2x?x3xi8>)
   // CHECK: ^bb0(%[[ARG1:.+]]: i8,
   // CHECK:   linalg.yield %[[ARG1]] : i8
-  // CHECK: %[[COLLAPSED:.+]] = tensor.collapse_shape %[[GENERIC]] {{\[}}[0, 1, 2, 3]]
-  // CHECK: tensor.expand_shape %[[COLLAPSED]] {{\[}}[0, 1]]
+  // CHECK: "tosa.reshape"(%[[GENERIC]]) {new_shape = array<i64: 2, -9223372036854775808>}
   %0 = "tosa.tile"(%arg0) {multiples = array<i64: 2, -1>} : (tensor<2x3xi8>)  -> (tensor<2x?xi8>)
 
   return
diff --git a/mlir/test/Conversion/TosaToTensor/tosa-to-tensor.mlir b/mlir/test/Conversion/TosaToTensor/tosa-to-tensor.mlir
index b4ed6ca0ceae3..d96c7848ed5b1 100644
--- a/mlir/test/Conversion/TosaToTensor/tosa-to-tensor.mlir
+++ b/mlir/test/Conversion/TosaToTensor/tosa-to-tensor.mlir
@@ -1,6 +1,95 @@
 // RUN: mlir-opt --split-input-file --tosa-to-tensor %s -o -| FileCheck %s
 
-// CHECK-LABEL: @slice
+// CHECK-LABEL: @test_reshape_downrank
+// CHECK-SAME: (%[[ARG0:[0-9a-zA-Z_]*]]
+func.func @test_reshape_downrank(%arg0: tensor<2x3xf32>) -> tensor<6xf32> {
+  // CHECK: [[RESHAPE:%.+]] = tensor.collapse_shape %[[ARG0]] {{\[}}[0, 1]]
+  %0 = "tosa.reshape"(%arg0) {new_shape = array<i64: 6>} : (tensor<2x3xf32>) -> tensor<6xf32>
+  // CHECK: return [[RESHAPE]]
+  return %0 : tensor<6xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @test_reshape_downrank_dyn
+// CHECK-SAME: (%[[ARG0:[0-9a-zA-Z_]*]]
+func.func @test_reshape_downrank_dyn(%arg0: tensor<2x?xf32>) -> tensor<?xf32> {
+  // CHECK: [[RESHAPE:%.+]] = tensor.collapse_shape %[[ARG0]] {{\[}}[0, 1]]
+  %0 = "tosa.reshape"(%arg0) {new_shape = array<i64: -1>} : (tensor<2x?xf32>) -> tensor<?xf32>
+  // CHECK: return [[RESHAPE]]
+  return %0 : tensor<?xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @test_reshape_uprank
+// CHECK-SAME: (%[[ARG0:[0-9a-zA-Z_]*]]
+func.func @test_reshape_uprank(%arg0: tensor<6xf32>) -> tensor<2x3xf32> {
+  // CHECK: [[RESHAPE:%.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0, 1]]
+  %0 = "tosa.reshape"(%arg0) {new_shape = array<i64: 2, 3>} : (tensor<6xf32>) -> tensor<2x3xf32>
+  // CHECK: return [[RESHAPE]]
+  return %0 : tensor<2x3xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @test_reshape_uprank_dyn
+// CHECK-SAME: (%[[ARG0:[0-9a-zA-Z_]*]]
+func.func @test_reshape_uprank_dyn(%arg0: tensor<?xf32>) -> tensor<2x?xf32> {
+  // CHECK: [[RESHAPE:%.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0, 1]]
+  %0 = "tosa.reshape"(%arg0) {new_shape = array<i64: 2, -1>} : (tensor<?xf32>) -> tensor<2x?xf32>
+  // CHECK: return [[RESHAPE]]
+  return %0 : tensor<2x?xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @test_reshape_samerank
+//  CHECK-SAME: (%[[ARG0:.*]]: tensor<3x2xf32>)
+func.func @test_reshape_samerank(%arg0: tensor<3x2xf32>) -> tensor<2x3xf32> {
+  // CHECK-NEXT: %[[RESHAPE1:.*]] = tensor.collapse_shape %[[ARG0]] {{\[}}[0, 1]]
+  // CHECK-NEXT: %[[RESHAPE2:.*]] = tensor.expand_shape %[[RESHAPE1]] {{\[}}[0, 1]]
+  %0 = "tosa.reshape"(%arg0) {new_shape = array<i64: 2, 3>} : (tensor<3x2xf32>) -> tensor<2x3xf32>
+  // CHECK-NEXT: return %[[RESHAPE2]]
+  return %0 : tensor<2x3xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @test_reshape_samerank_dyn
+//  CHECK-SAME: (%[[ARG0:.*]]: tensor<?x2xf32>)
+func.func @test_reshape_samerank_dyn(%arg0: tensor<?x2xf32>) -> tensor<2x?xf32> {
+  // CHECK-NEXT: %[[RESHAPE1:.*]] = tensor.collapse_shape %[[ARG0]] {{\[}}[0, 1]]
+  // CHECK-NEXT: %[[RESHAPE2:.*]] = tensor.expand_shape %[[RESHAPE1]] {{\[}}[0, 1]]
+  %0 = "tosa.reshape"(%arg0) {new_shape = array<i64: 2, -1>} : (tensor<?x2xf32>) -> tensor<2x?xf32>
+  // CHECK-NEXT: return %[[RESHAPE2]]
+  return %0 : tensor<2x?xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @test_reshape_downrank_6D
+// CHECK-SAME: (%[[ARG0:[0-9a-zA-Z_]*]]:
+func.func @test_reshape_downrank_6D(%arg0: tensor<1x2x3x5x7x11xf32>) -> tensor<6x5x77xf32> {
+  // CHECK: tensor.collapse_shape %[[ARG0]] {{\[}}[0, 1, 2], [3], [4, 5]]
+  %0 = "tosa.reshape"(%arg0) {new_shape = array<i64: 6, 5, 77>} : (tensor<1x2x3x5x7x11xf32>) -> tensor<6x5x77xf32>
+  return %0 : tensor<6x5x77xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @test_reshape_downrank_6D_dyn
+// CHECK-SAME: (%[[ARG0:[0-9a-zA-Z_]*]]:
+func.func @test_reshape_downrank_6D_dyn(%arg0: tensor<1x2x?x5x7x11xf32>) -> tensor<?x5x77xf32> {
+  // CHECK: tensor.collapse_shape %[[ARG0]] {{\[}}[0, 1, 2, 3, 4, 5]]
+  // CHECK: tensor.expand_shape %{{.*}} {{\[}}[0, 1, 2]]
+  %0 = "tosa.reshape"(%arg0) {new_shape = array<i64: -1, 5, 77>} : (tensor<1x2x?x5x7x11xf32>) -> tensor<?x5x77xf32>
+  return %0 : tensor<?x5x77xf32>
+}
+
+// -----
+
+// CHECK-LABLE: func @slice
 func.func @slice(%arg0: tensor<6xf32>) ->() {
   // CHECK: [[SLICE:%.+]] = tensor.extract_slice %arg0[2] [1] [1]
   %0 = "tosa.slice"(%arg0) {start = array<i64: 2>, size = array<i64: 1>} : (tensor<6xf32>)  -> (tensor<1xf32>)
@@ -106,3 +195,76 @@ func.func @pad_dyn_padding(%arg0 : tensor<1x2xf32>) -> (tensor<?x9xf32>) {
   %1 = "tosa.pad"(%arg0, %0)  : (tensor<1x2xf32>, tensor<2x2xi32>)  -> (tensor<?x9xf32>)
   return %1 : tensor<?x9xf32>
 }
+
+// -----
+
+// CHECK-LABEL: @concat
+// CHECK-SAME: %[[ARG0:.+]]: tensor<5x1xf32>
+// CHECK-SAME: %[[ARG1:.+]]: tensor<6x1xf32>
+func.func @concat(%arg0: tensor<5x1xf32>, %arg1: tensor<6x1xf32>) -> () {
+  // CHECK: [[AXIS:%.+]] = arith.constant 0
+  // CHECK: [[STRIDE:%.+]]   = arith.constant 1
+  // CHECK: [[OFFSET:%.+]] = arith.constant 0 : index
+  // CHECK: [[IDX0:%.+]] = arith.constant 0 : index
+  // CHECK: [[IDX1:%.+]] = arith.constant 1 : index
+  // CHECK: [[INIT:%.+]] = tensor.empty() : tensor<11x1xf32>
+  // CHECK: [[INSERT0:%.+]] = tensor.insert_slice %[[ARG0]] into [[INIT]][0, 0] [5, 1] [1, 1]
+  // CHECK: [[INSERT1:%.+]] = tensor.insert_slice %[[ARG1]] into [[INSERT0]][5, 0] [6, 1] [1, 1]
+  %0 = "tosa.concat"(%arg0, %arg1) { axis = 0 : i64} : (tensor<5x1xf32>, tensor<6x1xf32>)  -> (tensor<11x1xf32>)
+
+  // CHECK: [[AXIS:%.+]] = arith.constant 1
+  // CHECK: [[STRIDE:%.+]]   = arith.constant 1
+  // CHECK: [[OFFSET:%.+]] = arith.constant 0 : index
+  // CHECK: [[IDX0:%.+]] = arith.constant 0 : index
+  // CHECK: [[IDX1:%.+]] = arith.constant 1 : index
+  // CHECK: [[INIT:%.+]] = tensor.empty() : tensor<5x2xf32>
+  // CHECK: [[INSERT0:%.+]] = tensor.insert_slice %[[ARG0]] into [[INIT]][0, 0] [5, 1] [1, 1]
+  // CHECK: [[INSERT1:%.+]] = tensor.insert_slice %[[ARG0]] into [[INSERT0]][0, 1] [5, 1] [1, 1]
+  %1 = "tosa.concat"(%arg0, %arg0) { axis = 1 : i64} : (tensor<5x1xf32>, tensor<5x1xf32>)  -> (tensor<5x2xf32>)
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @concat_non_axis_dyn
+// CHECK-SAME: (%[[ARG0:[0-9a-zA-Z_]*]]:
+// CHECK-SAME:  %[[ARG1:[0-9a-zA-Z_]*]]
+func.func @concat_non_axis_dyn(%arg0: tensor<5x?xf32>, %arg1: tensor<6x?xf32>) -> () {
+  // CHECK: %[[AXIS:.+]] = arith.constant 0
+  // CHECK: %[[STRIDE:.+]]   = arith.constant 1
+  // CHECK: %[[OFFSET:.+]] = arith.constant 0 : index
+  // CHECK: %[[IDX0:.+]] = arith.constant 0 : index
+  // CHECK: %[[IDX1:.+]] = arith.constant 1 : index
+  // CHECK: %[[SIZE:.+]] = tensor.dim %[[ARG0]], %[[IDX1]]
+  // CHECK: %[[IDX1_2:.+]] = arith.constant 1 : index
+  // CHECK: %[[DYN:.+]] = tensor.dim %[[ARG0]], %[[IDX1_2]]
+  // CHECK: %[[INIT:.+]] = tensor.empty(%[[DYN]]) : tensor<11x?xf32>
+  // CHECK: %[[INSERT0:.+]] = tensor.insert_slice %[[ARG0]] into %[[INIT]][0, 0] [5, %[[SIZE]]] [1, 1]
+  // CHECK: %[[INSERT1:.+]] = tensor.insert_slice %[[ARG1]] into %[[INSERT0]][5, 0] [6, %[[SIZE]]] [1, 1]
+  %0 = "tosa.concat"(%arg0, %arg1) { axis = 0 : i64} : (tensor<5x?xf32>, tensor<6x?xf32>)  -> (tensor<11x?xf32>)
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @concat_axis_dyn
+// CHECK-SAME: (%[[ARG0:[0-9a-zA-Z_]*]]:
+// CHECK-SAME:  %[[ARG1:[0-9a-zA-Z_]*]]:
+func.func @concat_axis_dyn(%arg0: tensor<?x3xf32>, %arg1: tensor<?x3xf32>) -> () {
+  // CHECK: %[[AXIS:.+]] = arith.constant 0
+  // CHECK: %[[STRIDE:.+]]   = arith.constant 1
+  // CHECK: %[[OFFSET:.+]] = arith.constant 0 : index
+  // CHECK: %[[IDX0:.+]] = arith.constant 0 : index
+  // CHECK: %[[SIZE:.+]] = tensor.dim %[[ARG0]], %[[IDX0]]
+  // CHECK: %[[IDX0_2:.+]] = arith.constant 0 : index
+  // CHECK: %[[DYN:.+]] = tensor.dim %[[ARG0]], %[[IDX0_2]]
+  // CHECK: %[[IDX1:.+]] = arith.constant 1 : index
+  // CHECK: %[[INIT:.+]] = tensor.empty(%[[DYN]]) : tensor<?x3xf32>
+  // CHECK: %[[DYN1:.+]] = tensor.dim %[[ARG0]], %[[AXIS]]
+  // CHECK: %[[INSERT0:.+]] = tensor.insert_slice %[[ARG0]] into %[[INIT]][0, 0] [%[[DYN1]], 3] [1, 1]
+  // CHECK: %[[SUM:.+]]  = arith.addi %[[OFFSET]], %[[DYN1]]
+  // CHECK: %[[DYN2:.+]] = tensor.dim %[[ARG1]], %[[AXIS]]
+  // CHECK: %[[INSERT1:.+]] = tensor.insert_slice %[[ARG1]] into %[[INSERT0]][%[[SUM]], 0] [%[[DYN2]], 3] [1, 1]
+  %0 = "tosa.concat"(%arg0, %arg1) { axis = 0 : i64} : (tensor<?x3xf32>, tensor<?x3xf32>)  -> (tensor<?x3xf32>)
+  return
+}
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-to-llvm.mlir b/mlir/test/Conversion/VectorToLLVM/vector-to-llvm.mlir
index 0b9b86ade3323..62d165c8b3998 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-to-llvm.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-to-llvm.mlir
@@ -1374,7 +1374,7 @@ func.func @reduce_fmax_f32(%arg0: vector<16xf32>, %arg1: f32) -> f32 {
 }
 // CHECK-LABEL: @reduce_fmax_f32(
 // CHECK-SAME: %[[A:.*]]: vector<16xf32>, %[[B:.*]]: f32)
-//      CHECK: %[[V:.*]] = "llvm.intr.vector.reduce.fmax"(%[[A]]) : (vector<16xf32>) -> f32
+//      CHECK: %[[V:.*]] = llvm.intr.vector.reduce.fmax(%[[A]]) : (vector<16xf32>) -> f32
 //      CHECK: %[[C0:.*]] = llvm.fcmp "ogt" %[[V]], %[[B]] : f32
 //      CHECK: %[[S0:.*]] = llvm.select %[[C0]], %[[V]], %[[B]] : i1, f32
 //      CHECK: %[[C1:.*]] = llvm.fcmp "uno" %[[V]], %[[B]] : f32
@@ -1390,7 +1390,7 @@ func.func @reduce_fmin_f32(%arg0: vector<16xf32>, %arg1: f32) -> f32 {
 }
 // CHECK-LABEL: @reduce_fmin_f32(
 // CHECK-SAME: %[[A:.*]]: vector<16xf32>, %[[B:.*]]: f32)
-//      CHECK: %[[V:.*]] = "llvm.intr.vector.reduce.fmin"(%[[A]]) : (vector<16xf32>) -> f32
+//      CHECK: %[[V:.*]] = llvm.intr.vector.reduce.fmin(%[[A]]) : (vector<16xf32>) -> f32
 //      CHECK: %[[C0:.*]] = llvm.fcmp "olt" %[[V]], %[[B]] : f32
 //      CHECK: %[[S0:.*]] = llvm.select %[[C0]], %[[V]], %[[B]] : i1, f32
 //      CHECK: %[[C1:.*]] = llvm.fcmp "uno" %[[V]], %[[B]] : f32
diff --git a/mlir/test/Conversion/VectorToSPIRV/vector-reduction-to-spirv-dot-prod.mlir b/mlir/test/Conversion/VectorToSPIRV/vector-reduction-to-spirv-dot-prod.mlir
new file mode 100644
index 0000000000000..bfe6d8608a99d
--- /dev/null
+++ b/mlir/test/Conversion/VectorToSPIRV/vector-reduction-to-spirv-dot-prod.mlir
@@ -0,0 +1,176 @@
+// RUN: mlir-opt --split-input-file --verify-diagnostics \
+// RUN:   --test-vector-reduction-to-spirv-dot-prod %s -o - | FileCheck %s
+
+// Positive tests.
+
+// CHECK-LABEL: func.func @to_sdot
+//  CHECK-SAME:   ([[ARG0:%.+]]: vector<4xi8>, [[ARG1:%.+]]: vector<4xi8>)
+//  CHECK-NEXT:   [[DOT:%.+]] = spirv.SDot [[ARG0]], [[ARG1]] : (vector<4xi8>, vector<4xi8>) -> i32
+//  CHECK-NEXT:   return [[DOT]] : i32
+func.func @to_sdot(%arg0: vector<4xi8>, %arg1: vector<4xi8>) -> i32 {
+  %lhs = arith.extsi %arg0 : vector<4xi8> to vector<4xi32>
+  %rhs = arith.extsi %arg1 : vector<4xi8> to vector<4xi32>
+  %mul = arith.muli %lhs, %rhs : vector<4xi32>
+  %red = vector.reduction <add>, %mul : vector<4xi32> into i32
+  return %red : i32
+}
+
+// CHECK-LABEL: func.func @to_sdot_acc
+//  CHECK-SAME:   ([[ARG0:%.+]]: vector<4xi8>, [[ARG1:%.+]]: vector<4xi8>, [[ACC:%.+]]: i32)
+//  CHECK-NEXT:   [[DOT:%.+]] = spirv.SDotAccSat [[ARG0]], [[ARG1]], [[ACC]] : (vector<4xi8>, vector<4xi8>, i32) -> i32
+//  CHECK-NEXT:   return [[DOT]] : i32
+func.func @to_sdot_acc(%arg0: vector<4xi8>, %arg1: vector<4xi8>, %acc: i32) -> i32 {
+  %lhs = arith.extsi %arg0 : vector<4xi8> to vector<4xi32>
+  %rhs = arith.extsi %arg1 : vector<4xi8> to vector<4xi32>
+  %mul = arith.muli %lhs, %rhs : vector<4xi32>
+  %red = vector.reduction <add>, %mul, %acc : vector<4xi32> into i32
+  return %red : i32
+}
+
+// CHECK-LABEL: func.func @to_sdot_i64
+//  CHECK-SAME:   ([[ARG0:%.+]]: vector<4xi8>, [[ARG1:%.+]]: vector<4xi8>)
+//  CHECK-NEXT:   [[DOT:%.+]] = spirv.SDot [[ARG0]], [[ARG1]] : (vector<4xi8>, vector<4xi8>) -> i64
+//  CHECK-NEXT:   return [[DOT]] : i64
+func.func @to_sdot_i64(%arg0: vector<4xi8>, %arg1: vector<4xi8>) -> i64 {
+  %lhs = arith.extsi %arg0 : vector<4xi8> to vector<4xi64>
+  %rhs = arith.extsi %arg1 : vector<4xi8> to vector<4xi64>
+  %mul = arith.muli %lhs, %rhs : vector<4xi64>
+  %red = vector.reduction <add>, %mul : vector<4xi64> into i64
+  return %red : i64
+}
+
+// CHECK-LABEL: func.func @to_sdot_acc_i64
+//  CHECK-SAME:   ([[ARG0:%.+]]: vector<4xi8>, [[ARG1:%.+]]: vector<4xi8>, [[ACC:%.+]]: i64)
+//  CHECK-NEXT:   [[DOT:%.+]] = spirv.SDotAccSat [[ARG0]], [[ARG1]], [[ACC]] : (vector<4xi8>, vector<4xi8>, i64) -> i64
+//  CHECK-NEXT:   return [[DOT]] : i64
+func.func @to_sdot_acc_i64(%arg0: vector<4xi8>, %arg1: vector<4xi8>, %acc: i64) -> i64 {
+  %lhs = arith.extsi %arg0 : vector<4xi8> to vector<4xi64>
+  %rhs = arith.extsi %arg1 : vector<4xi8> to vector<4xi64>
+  %mul = arith.muli %lhs, %rhs : vector<4xi64>
+  %red = vector.reduction <add>, %mul, %acc : vector<4xi64> into i64
+  return %red : i64
+}
+
+// CHECK-LABEL: func.func @to_udot
+//  CHECK-SAME:   ([[ARG0:%.+]]: vector<4xi8>, [[ARG1:%.+]]: vector<4xi8>)
+//  CHECK-NEXT:   [[DOT:%.+]] = spirv.UDot [[ARG0]], [[ARG1]] : (vector<4xi8>, vector<4xi8>) -> i32
+//  CHECK-NEXT:   return [[DOT]] : i32
+func.func @to_udot(%arg0: vector<4xi8>, %arg1: vector<4xi8>) -> i32 {
+  %lhs = arith.extui %arg0 : vector<4xi8> to vector<4xi32>
+  %rhs = arith.extui %arg1 : vector<4xi8> to vector<4xi32>
+  %mul = arith.muli %lhs, %rhs : vector<4xi32>
+  %red = vector.reduction <add>, %mul : vector<4xi32> into i32
+  return %red : i32
+}
+
+// CHECK-LABEL: func.func @to_udot_acc
+//  CHECK-SAME:   ([[ARG0:%.+]]: vector<4xi8>, [[ARG1:%.+]]: vector<4xi8>, [[ACC:%.+]]: i32)
+//  CHECK-NEXT:   [[DOT:%.+]] = spirv.UDotAccSat [[ARG0]], [[ARG1]], [[ACC]] : (vector<4xi8>, vector<4xi8>, i32) -> i32
+//  CHECK-NEXT:   return [[DOT]] : i32
+func.func @to_udot_acc(%arg0: vector<4xi8>, %arg1: vector<4xi8>, %acc: i32) -> i32 {
+  %lhs = arith.extui %arg0 : vector<4xi8> to vector<4xi32>
+  %rhs = arith.extui %arg1 : vector<4xi8> to vector<4xi32>
+  %mul = arith.muli %lhs, %rhs : vector<4xi32>
+  %red = vector.reduction <add>, %mul, %acc : vector<4xi32> into i32
+  return %red : i32
+}
+
+// CHECK-LABEL: func.func @to_signed_unsigned_dot
+//  CHECK-SAME:   ([[ARG0:%.+]]: vector<4xi8>, [[ARG1:%.+]]: vector<4xi8>)
+//  CHECK-NEXT:   [[DOT:%.+]] = spirv.SUDot [[ARG0]], [[ARG1]] : (vector<4xi8>, vector<4xi8>) -> i32
+//  CHECK-NEXT:   return [[DOT]] : i32
+func.func @to_signed_unsigned_dot(%arg0: vector<4xi8>, %arg1: vector<4xi8>) -> i32 {
+  %lhs = arith.extsi %arg0 : vector<4xi8> to vector<4xi32>
+  %rhs = arith.extui %arg1 : vector<4xi8> to vector<4xi32>
+  %mul = arith.muli %lhs, %rhs : vector<4xi32>
+  %red = vector.reduction <add>, %mul : vector<4xi32> into i32
+  return %red : i32
+}
+
+// CHECK-LABEL: func.func @to_signed_unsigned_dot_acc
+//  CHECK-SAME:   ([[ARG0:%.+]]: vector<4xi8>, [[ARG1:%.+]]: vector<4xi8>, [[ACC:%.+]]: i32)
+//  CHECK-NEXT:   [[DOT:%.+]] = spirv.SUDotAccSat [[ARG0]], [[ARG1]], [[ACC]] : (vector<4xi8>, vector<4xi8>, i32) -> i32
+//  CHECK-NEXT:   return [[DOT]] : i32
+func.func @to_signed_unsigned_dot_acc(%arg0: vector<4xi8>, %arg1: vector<4xi8>, %acc: i32) -> i32 {
+  %lhs = arith.extsi %arg0 : vector<4xi8> to vector<4xi32>
+  %rhs = arith.extui %arg1 : vector<4xi8> to vector<4xi32>
+  %mul = arith.muli %lhs, %rhs : vector<4xi32>
+  %red = vector.reduction <add>, %mul, %acc : vector<4xi32> into i32
+  return %red : i32
+}
+
+// CHECK-LABEL: func.func @to_unsigned_signed_dot
+//  CHECK-SAME:   ([[ARG0:%.+]]: vector<4xi8>, [[ARG1:%.+]]: vector<4xi8>)
+//  CHECK-NEXT:   [[DOT:%.+]] = spirv.SUDot [[ARG1]], [[ARG0]] : (vector<4xi8>, vector<4xi8>) -> i32
+//  CHECK-NEXT:   return [[DOT]] : i32
+func.func @to_unsigned_signed_dot(%arg0: vector<4xi8>, %arg1: vector<4xi8>) -> i32 {
+  %lhs = arith.extui %arg0 : vector<4xi8> to vector<4xi32>
+  %rhs = arith.extsi %arg1 : vector<4xi8> to vector<4xi32>
+  %mul = arith.muli %lhs, %rhs : vector<4xi32>
+  %red = vector.reduction <add>, %mul : vector<4xi32> into i32
+  return %red : i32
+}
+
+// CHECK-LABEL: func.func @to_unsigned_signed_dot_acc
+//  CHECK-SAME:   ([[ARG0:%.+]]: vector<4xi8>, [[ARG1:%.+]]: vector<4xi8>, [[ACC:%.+]]: i32)
+//  CHECK-NEXT:   [[DOT:%.+]] = spirv.SUDotAccSat [[ARG1]], [[ARG0]], [[ACC]] : (vector<4xi8>, vector<4xi8>, i32) -> i32
+//  CHECK-NEXT:   return [[DOT]] : i32
+func.func @to_unsigned_signed_dot_acc(%arg0: vector<4xi8>, %arg1: vector<4xi8>, %acc: i32) -> i32 {
+  %lhs = arith.extui %arg0 : vector<4xi8> to vector<4xi32>
+  %rhs = arith.extsi %arg1 : vector<4xi8> to vector<4xi32>
+  %mul = arith.muli %lhs, %rhs : vector<4xi32>
+  %red = vector.reduction <add>, %mul, %acc : vector<4xi32> into i32
+  return %red : i32
+}
+
+// -----
+// Negative tests.
+
+// CHECK-LABEL: func.func @too_short
+//  CHECK-SAME:   ([[ARG0:%.+]]: vector<3xi8>, [[ARG1:%.+]]: vector<3xi8>)
+//  CHECK:        [[RED:%.+]] = vector.reduction
+//  CHECK-NEXT:   return [[RED]] : i32
+func.func @too_short(%arg0: vector<3xi8>, %arg1: vector<3xi8>) -> i32 {
+  %lhs = arith.extsi %arg0 : vector<3xi8> to vector<3xi32>
+  %rhs = arith.extsi %arg1 : vector<3xi8> to vector<3xi32>
+  %mul = arith.muli %lhs, %rhs : vector<3xi32>
+  %red = vector.reduction <add>, %mul : vector<3xi32> into i32
+  return %red : i32
+}
+
+// CHECK-LABEL: func.func @too_long
+//  CHECK-SAME:   ([[ARG0:%.+]]: vector<6xi8>, [[ARG1:%.+]]: vector<6xi8>)
+//  CHECK:        [[RED:%.+]] = vector.reduction
+//  CHECK-NEXT:   return [[RED]] : i32
+func.func @too_long(%arg0: vector<6xi8>, %arg1: vector<6xi8>) -> i32 {
+  %lhs = arith.extsi %arg0 : vector<6xi8> to vector<6xi32>
+  %rhs = arith.extsi %arg1 : vector<6xi8> to vector<6xi32>
+  %mul = arith.muli %lhs, %rhs : vector<6xi32>
+  %red = vector.reduction <add>, %mul : vector<6xi32> into i32
+  return %red : i32
+}
+
+// CHECK-LABEL: func.func @wrong_reduction_kind
+//  CHECK-SAME:   ([[ARG0:%.+]]: vector<4xi8>, [[ARG1:%.+]]: vector<4xi8>)
+//  CHECK:        [[RED:%.+]] = vector.reduction <mul>
+//  CHECK-NEXT:   return [[RED]] : i32
+func.func @wrong_reduction_kind(%arg0: vector<4xi8>, %arg1: vector<4xi8>) -> i32 {
+  %lhs = arith.extsi %arg0 : vector<4xi8> to vector<4xi32>
+  %rhs = arith.extsi %arg1 : vector<4xi8> to vector<4xi32>
+  %mul = arith.muli %lhs, %rhs : vector<4xi32>
+  %red = vector.reduction <mul>, %mul : vector<4xi32> into i32
+  return %red : i32
+}
+
+// CHECK-LABEL: func.func @wrong_arith_op
+//  CHECK-SAME:   ([[ARG0:%.+]]: vector<4xi8>, [[ARG1:%.+]]: vector<4xi8>)
+//  CHECK:        [[ADD:%.+]] = arith.addi
+//  CHECK:        [[RED:%.+]] = vector.reduction <mul>, [[ADD]]
+//  CHECK-NEXT:   return [[RED]] : i32
+func.func @wrong_arith_op(%arg0: vector<4xi8>, %arg1: vector<4xi8>) -> i32 {
+  %lhs = arith.extsi %arg0 : vector<4xi8> to vector<4xi32>
+  %rhs = arith.extsi %arg1 : vector<4xi8> to vector<4xi32>
+  %add = arith.addi %lhs, %rhs : vector<4xi32>
+  %red = vector.reduction <mul>, %add : vector<4xi32> into i32
+  return %red : i32
+}
diff --git a/mlir/test/Dialect/Affine/decompose-affine-ops.mlir b/mlir/test/Dialect/Affine/decompose-affine-ops.mlir
new file mode 100644
index 0000000000000..6acdc436fe677
--- /dev/null
+++ b/mlir/test/Dialect/Affine/decompose-affine-ops.mlir
@@ -0,0 +1,156 @@
+// RUN: mlir-opt %s -allow-unregistered-dialect -test-decompose-affine-ops -split-input-file -loop-invariant-code-motion -cse | FileCheck %s
+
+// CHECK-DAG: #[[$c42:.*]] = affine_map<() -> (42)>
+// CHECK-DAG: #[[$div32mod4:.*]] = affine_map<()[s0] -> ((s0 floordiv 32) mod 4)>
+// CHECK-DAG: #[[$add:.*]] = affine_map<()[s0, s1] -> (s0 + s1)>
+
+// CHECK-LABEL:  func.func @simple_test_1
+//  CHECK-SAME:  %[[I0:[0-9a-zA-Z]+]]: index,
+//  CHECK-SAME:  %[[I1:[0-9a-zA-Z]+]]: index,
+//  CHECK-SAME:  %[[I2:[0-9a-zA-Z]+]]: index,
+//  CHECK-SAME:  %[[LB:[0-9a-zA-Z]+]]: index,
+//  CHECK-SAME:  %[[UB:[0-9a-zA-Z]+]]: index,
+//  CHECK-SAME:  %[[STEP:[0-9a-zA-Z]+]]: index
+func.func @simple_test_1(%0: index, %1: index, %2: index, %lb: index, %ub: index, %step: index) {
+  // CHECK: %[[c42:.*]] = affine.apply #[[$c42]]()
+  // CHECK: %[[R1:.*]] = affine.apply #[[$div32mod4]]()[%[[I1]]]
+  // CHECK: %[[a:.*]] = affine.apply #[[$add]]()[%[[c42]], %[[R1]]]
+  %a = affine.apply affine_map<(d0) -> ((d0 floordiv 32) mod 4 + 42)>(%1)
+
+  // CHECK:     "some_side_effecting_consumer"(%[[a]]) : (index) -> ()
+  "some_side_effecting_consumer"(%a) : (index) -> ()
+  return
+}
+
+// -----
+
+// CHECK-DAG: #[[$c42:.*]] = affine_map<() -> (42)>
+// CHECK-DAG: #[[$id:.*]] = affine_map<()[s0] -> (s0)>
+// CHECK-DAG: #[[$add:.*]] = affine_map<()[s0, s1] -> (s0 + s1)>
+// CHECK-DAG: #[[$div32div4timesm4:.*]] = affine_map<()[s0] -> (((s0 floordiv 32) floordiv 4) * -4)>
+// CHECK-DAG: #[[$div32:.*]] = affine_map<()[s0] -> (s0 floordiv 32)>
+
+// CHECK-LABEL:  func.func @simple_test_2
+//  CHECK-SAME:  %[[I0:[0-9a-zA-Z]+]]: index,
+//  CHECK-SAME:  %[[I1:[0-9a-zA-Z]+]]: index,
+//  CHECK-SAME:  %[[I2:[0-9a-zA-Z]+]]: index,
+//  CHECK-SAME:  %[[LB:[0-9a-zA-Z]+]]: index,
+//  CHECK-SAME:  %[[UB:[0-9a-zA-Z]+]]: index,
+//  CHECK-SAME:  %[[STEP:[0-9a-zA-Z]+]]: index
+func.func @simple_test_2(%0: index, %1: index, %2: index, %lb: index, %ub: index, %step: index) {
+  // CHECK: %[[c42:.*]] = affine.apply #[[$c42]]()
+  // CHECK: scf.for %[[i:.*]] =
+  scf.for %i = %lb to %ub step %step {
+    // CHECK:   %[[R1:.*]] = affine.apply #[[$id]]()[%[[i]]]
+    // CHECK:   %[[R2:.*]] = affine.apply #[[$add]]()[%[[c42]], %[[R1]]]
+    // CHECK:   scf.for %[[j:.*]] =
+    scf.for %j = %lb to %ub step %step {
+      // CHECK:     %[[R3:.*]] = affine.apply #[[$div32div4timesm4]]()[%[[j]]]
+      // CHECK:     %[[R4:.*]] = affine.apply #[[$add]]()[%[[R2]], %[[R3]]]
+      // CHECK:     %[[R5:.*]] = affine.apply #[[$div32]]()[%[[j]]]
+      // CHECK:      %[[a:.*]] = affine.apply #[[$add]]()[%[[R4]], %[[R5]]]
+      %a = affine.apply affine_map<(d0)[s0] -> ((d0 floordiv 32) mod 4 + s0 + 42)>(%j)[%i]
+
+      // CHECK:     "some_side_effecting_consumer"(%[[a]]) : (index) -> ()
+      "some_side_effecting_consumer"(%a) : (index) -> ()
+    }
+  }
+  return
+}
+
+// -----
+
+// CHECK-DAG: #[[$div4:.*]] = affine_map<()[s0] -> (s0 floordiv 4)>
+// CHECK-DAG: #[[$times32:.*]] = affine_map<()[s0] -> (s0 * 32)>
+// CHECK-DAG: #[[$times16:.*]] = affine_map<()[s0] -> (s0 * 16)>
+// CHECK-DAG: #[[$add:.*]] = affine_map<()[s0, s1] -> (s0 + s1)>
+// CHECK-DAG: #[[$div4timesm32:.*]] = affine_map<()[s0] -> ((s0 floordiv 4) * -32)>
+// CHECK-DAG: #[[$times8:.*]] = affine_map<()[s0] -> (s0 * 8)>
+// CHECK-DAG: #[[$id:.*]] = affine_map<()[s0] -> (s0)>
+// CHECK-DAG: #[[$div32div4timesm4:.*]] = affine_map<()[s0] -> (((s0 floordiv 32) floordiv 4) * -4)>
+// CHECK-DAG: #[[$div32:.*]] = affine_map<()[s0] -> (s0 floordiv 32)>
+
+// CHECK-LABEL:  func.func @larger_test
+//  CHECK-SAME:  %[[I0:[0-9a-zA-Z]+]]: index,
+//  CHECK-SAME:  %[[I1:[0-9a-zA-Z]+]]: index,
+//  CHECK-SAME:  %[[I2:[0-9a-zA-Z]+]]: index,
+//  CHECK-SAME:  %[[LB:[0-9a-zA-Z]+]]: index,
+//  CHECK-SAME:  %[[UB:[0-9a-zA-Z]+]]: index,
+//  CHECK-SAME:  %[[STEP:[0-9a-zA-Z]+]]: index
+func.func @larger_test(%0: index, %1: index, %2: index, %lb: index, %ub: index, %step: index) {
+    %c2 = arith.constant 2 : index
+    %c6 = arith.constant 6 : index
+
+    //      CHECK: %[[R0:.*]] = affine.apply #[[$div4]]()[%[[I0]]]
+    // CHECK-NEXT: %[[R1:.*]] = affine.apply #[[$times16]]()[%[[I1]]]
+    // CHECK-NEXT: %[[R2:.*]] = affine.apply #[[$add]]()[%[[R0]], %[[R1]]]
+    // CHECK-NEXT: %[[R3:.*]] = affine.apply #[[$times32]]()[%[[I2]]]
+
+    // I1 * 16 + I2 * 32 + I0 floordiv 4
+    // CHECK-NEXT: %[[b:.*]] = affine.apply #[[$add]]()[%[[R2]], %[[R3]]]
+
+    // (I0 floordiv 4) * 32
+    // CHECK-NEXT: %[[R5:.*]] = affine.apply #[[$div4timesm32]]()[%[[I0]]]
+    // 8 * I0
+    // CHECK-NEXT: %[[R6:.*]] = affine.apply #[[$times8]]()[%[[I0]]]
+    // 8 * I0 + (I0 floordiv 4) * 32
+    // CHECK-NEXT: %[[c:.*]] = affine.apply #[[$add]]()[%[[R5]], %[[R6]]]
+
+    // CHECK-NEXT: scf.for %[[i:.*]] =
+    scf.for %i = %lb to %ub step %step {
+      // remainder from %a not hoisted above %i.
+      // CHECK-NEXT: %[[R8:.*]] = affine.apply #[[$times32]]()[%[[i]]]
+      // CHECK-NEXT: %[[a:.*]] = affine.apply #[[$add]]()[%[[b]], %[[R8]]]
+
+      // CHECK-NEXT: scf.for %[[j:.*]] =
+      scf.for %j = %lb to %ub step %step {
+        // Gets hoisted partially to i and rest outermost.
+        // The hoisted part is %b.
+        %a = affine.apply affine_map<()[s0, s1, s2, s3] -> (s1 * 16 + s2 * 32 + s3 * 32 + s0 floordiv 4)>()[%0, %1, %2, %i]
+
+        // Gets completely hoisted 
+        %b = affine.apply affine_map<()[s0, s1, s2] -> (s1 * 16 + s2 * 32 + s0 floordiv 4)>()[%0, %1, %2]
+
+        // Gets completely hoisted 
+        %c = affine.apply affine_map<()[s0] -> (s0 * 8 - (s0 floordiv 4) * 32)>()[%0]
+ 
+        // 32 * %j + %c remains here, the rest is hoisted.
+        // CHECK-DAG: %[[R10:.*]] = affine.apply #[[$times32]]()[%[[j]]]
+        // CHECK-DAG: %[[d:.*]] = affine.apply #[[$add]]()[%[[c]], %[[R10]]]
+        %d = affine.apply affine_map<()[s0, s1] -> (s0 * 8 + s1 * 32 - (s0 floordiv 4) * 32)>()[%0, %j]
+
+        // CHECK-DAG: %[[idj:.*]] = affine.apply #[[$id]]()[%[[j]]]
+        // CHECK-NEXT: scf.for %[[k:.*]] =
+        scf.for %k = %lb to %ub step %step {
+          // CHECK-NEXT: %[[idk:.*]] = affine.apply #[[$id]]()[%[[k]]]
+          // CHECK-NEXT: %[[e:.*]] = affine.apply #[[$add]]()[%[[c]], %[[idk]]]
+          %e = affine.apply affine_map<()[s0, s1] -> (s0 + s1 * 8 - (s1 floordiv 4) * 32)>()[%k, %0]
+
+          // CHECK-NEXT: %[[R15:.*]] = affine.apply #[[$div32div4timesm4]]()[%[[k]]]
+          // CHECK-NEXT: %[[R16:.*]] = affine.apply #[[$add]]()[%[[idj]], %[[R15]]]
+          // CHECK-NEXT: %[[R17:.*]] = affine.apply #[[$div32]]()[%[[k]]]
+          // CHECK-NEXT: %[[f:.*]] = affine.apply #[[$add]]()[%[[R16]], %[[R17]]]
+          %f = affine.apply affine_map<(d0)[s0] -> ((d0 floordiv 32) mod 4 + s0)>(%k)[%j]
+
+          // CHECK-NEXT: %[[g:.*]] = affine.apply #[[$add]]()[%[[b]], %[[idk]]]
+          %g = affine.apply affine_map<()[s0, s1, s2, s3] -> (s0 + s2 * 16 + s3 * 32 + s1 floordiv 4)>()[%k, %0, %1, %2]
+          
+          // CHECK-NEXT: "some_side_effecting_consumer"(%[[a]]) : (index) -> ()
+          "some_side_effecting_consumer"(%a) : (index) -> ()
+          // CHECK-NEXT: "some_side_effecting_consumer"(%[[b]]) : (index) -> ()
+          "some_side_effecting_consumer"(%b) : (index) -> ()
+          // CHECK-NEXT: "some_side_effecting_consumer"(%[[c]]) : (index) -> ()
+          "some_side_effecting_consumer"(%c) : (index) -> ()
+          // CHECK-NEXT: "some_side_effecting_consumer"(%[[d]]) : (index) -> ()
+          "some_side_effecting_consumer"(%d) : (index) -> ()
+          // CHECK-NEXT: "some_side_effecting_consumer"(%[[e]]) : (index) -> ()
+          "some_side_effecting_consumer"(%e) : (index) -> ()
+          // CHECK-NEXT: "some_side_effecting_consumer"(%[[f]]) : (index) -> ()
+          "some_side_effecting_consumer"(%f) : (index) -> ()
+          // CHECK-NEXT: "some_side_effecting_consumer"(%[[g]]) : (index) -> ()
+          "some_side_effecting_consumer"(%g) : (index) -> ()
+        }
+    }
+  }   
+  return
+}
diff --git a/mlir/test/Dialect/Affine/scalrep.mlir b/mlir/test/Dialect/Affine/scalrep.mlir
index c5862be6a6e97..64b8534a9816a 100644
--- a/mlir/test/Dialect/Affine/scalrep.mlir
+++ b/mlir/test/Dialect/Affine/scalrep.mlir
@@ -868,3 +868,23 @@ func.func @dead_affine_region_op() {
   // CHECK-NEXT:   affine.load
   return
 }
+
+// We perform no scalar replacement here since we don't depend on dominance
+// info, which would be needed in such cases when ops fall in different blocks
+// of a CFG region.
+
+// CHECK-LABEL: func @cross_block
+func.func @cross_block() {
+  %c10 = arith.constant 10 : index
+  %alloc_83 = memref.alloc() : memref<1x13xf32>
+  %alloc_99 = memref.alloc() : memref<13xi1>
+  %true_110 = arith.constant true
+  affine.store %true_110, %alloc_99[%c10] : memref<13xi1>
+  %true = arith.constant true
+  affine.store %true, %alloc_99[%c10] : memref<13xi1>
+  cf.br ^bb1(%alloc_83 : memref<1x13xf32>)
+^bb1(%35: memref<1x13xf32>):
+  // CHECK: affine.load
+  %69 = affine.load %alloc_99[%c10] : memref<13xi1>
+  return
+}
diff --git a/mlir/test/Dialect/Arith/canonicalize.mlir b/mlir/test/Dialect/Arith/canonicalize.mlir
index eaafa9e93ceaa..396f5ee3dc6ea 100644
--- a/mlir/test/Dialect/Arith/canonicalize.mlir
+++ b/mlir/test/Dialect/Arith/canonicalize.mlir
@@ -735,6 +735,72 @@ func.func @doubleAddSub2(%arg0: index, %arg1 : index) -> index {
   return %add : index
 }
 
+// CHECK-LABEL: @addiMuliToSubiRhsI32
+//  CHECK-SAME:   (%[[ARG0:.+]]: i32, %[[ARG1:.+]]: i32)
+//       CHECK:   %[[SUB:.+]] = arith.subi %[[ARG0]], %[[ARG1]] : i32
+//       CHECK:   return %[[SUB]]
+func.func @addiMuliToSubiRhsI32(%arg0: i32, %arg1: i32) -> i32 {
+  %c-1 = arith.constant -1 : i32
+  %neg = arith.muli %arg1, %c-1 : i32
+  %add = arith.addi %arg0, %neg : i32
+  return %add : i32
+}
+
+// CHECK-LABEL: @addiMuliToSubiRhsIndex
+//  CHECK-SAME:   (%[[ARG0:.+]]: index, %[[ARG1:.+]]: index)
+//       CHECK:   %[[SUB:.+]] = arith.subi %[[ARG0]], %[[ARG1]] : index
+//       CHECK:   return %[[SUB]]
+func.func @addiMuliToSubiRhsIndex(%arg0: index, %arg1: index) -> index {
+  %c-1 = arith.constant -1 : index
+  %neg = arith.muli %arg1, %c-1 : index
+  %add = arith.addi %arg0, %neg : index
+  return %add : index
+}
+
+// CHECK-LABEL: @addiMuliToSubiRhsVector
+//  CHECK-SAME:   (%[[ARG0:.+]]: vector<3xi64>, %[[ARG1:.+]]: vector<3xi64>)
+//       CHECK:   %[[SUB:.+]] = arith.subi %[[ARG0]], %[[ARG1]] : vector<3xi64>
+//       CHECK:   return %[[SUB]]
+func.func @addiMuliToSubiRhsVector(%arg0: vector<3xi64>, %arg1: vector<3xi64>) -> vector<3xi64> {
+  %c-1 = arith.constant dense<-1> : vector<3xi64>
+  %neg = arith.muli %arg1, %c-1 : vector<3xi64>
+  %add = arith.addi %arg0, %neg : vector<3xi64>
+  return %add : vector<3xi64>
+}
+
+// CHECK-LABEL: @addiMuliToSubiLhsI32
+//  CHECK-SAME:   (%[[ARG0:.+]]: i32, %[[ARG1:.+]]: i32)
+//       CHECK:   %[[SUB:.+]] = arith.subi %[[ARG0]], %[[ARG1]] : i32
+//       CHECK:   return %[[SUB]]
+func.func @addiMuliToSubiLhsI32(%arg0: i32, %arg1: i32) -> i32 {
+  %c-1 = arith.constant -1 : i32
+  %neg = arith.muli %arg1, %c-1 : i32
+  %add = arith.addi %neg, %arg0 : i32
+  return %add : i32
+}
+
+// CHECK-LABEL: @addiMuliToSubiLhsIndex
+//  CHECK-SAME:   (%[[ARG0:.+]]: index, %[[ARG1:.+]]: index)
+//       CHECK:   %[[SUB:.+]] = arith.subi %[[ARG0]], %[[ARG1]] : index
+//       CHECK:   return %[[SUB]]
+func.func @addiMuliToSubiLhsIndex(%arg0: index, %arg1: index) -> index {
+  %c-1 = arith.constant -1 : index
+  %neg = arith.muli %arg1, %c-1 : index
+  %add = arith.addi %neg, %arg0 : index
+  return %add : index
+}
+
+// CHECK-LABEL: @addiMuliToSubiLhsVector
+//  CHECK-SAME:   (%[[ARG0:.+]]: vector<3xi64>, %[[ARG1:.+]]: vector<3xi64>)
+//       CHECK:   %[[SUB:.+]] = arith.subi %[[ARG0]], %[[ARG1]] : vector<3xi64>
+//       CHECK:   return %[[SUB]]
+func.func @addiMuliToSubiLhsVector(%arg0: vector<3xi64>, %arg1: vector<3xi64>) -> vector<3xi64> {
+  %c-1 = arith.constant dense<-1> : vector<3xi64>
+  %neg = arith.muli %arg1, %c-1 : vector<3xi64>
+  %add = arith.addi %neg, %arg0 : vector<3xi64>
+  return %add : vector<3xi64>
+}
+
 // CHECK-LABEL: @adduiExtendedZeroRhs
 //  CHECK-NEXT:   %[[false:.+]] = arith.constant false
 //  CHECK-NEXT:   return %arg0, %[[false]]
diff --git a/mlir/test/Dialect/GPU/transform-gpu-failing.mlir b/mlir/test/Dialect/GPU/transform-gpu-failing.mlir
index 84b8a78c461a5..459b800f76d35 100644
--- a/mlir/test/Dialect/GPU/transform-gpu-failing.mlir
+++ b/mlir/test/Dialect/GPU/transform-gpu-failing.mlir
@@ -1,19 +1,19 @@
 // RUN: mlir-opt --test-transform-dialect-interpreter --split-input-file  -canonicalize -cse --verify-diagnostics %s
 
-func.func @map_nested_foreach_to_threads_not_gpu_launch() -> () {
+func.func @map_nested_forall_to_threads_not_gpu_launch() -> () {
   %1 = tensor.empty() : tensor<4xf32>
   return
 }
 transform.sequence failures(propagate) {
 ^bb0(%arg0: !pdl.operation):
   %funcop = transform.structured.match ops{["tensor.empty"]} in %arg0 : (!pdl.operation) -> !pdl.operation
-  // expected-error @below {{Given target is not gpu.launch}}
-  %1 = transform.gpu.map_nested_foreach_to_threads %funcop
+  // expected-error @below {{Given target is not a gpu.launch}}
+  %1 = transform.gpu.map_nested_forall_to_threads %funcop block_dims = [1, 1, 1]
 }
 
 // -----
 
-func.func @map_nested_foreach_to_threads_excessive_threads(%x: memref<2 x 32 x f32>, %y: memref<2 x 32 x f32>, %t: memref<32 x f32>, %alpha : f32, %stream : !gpu.async.token) -> memref<2 x 32 x f32> {
+func.func @map_nested_forall_to_threads_excessive_threads(%x: memref<2 x 32 x f32>, %y: memref<2 x 32 x f32>, %t: memref<32 x f32>, %alpha : f32, %stream : !gpu.async.token) -> memref<2 x 32 x f32> {
   %one = arith.constant 1 : index
   %c900 = arith.constant 900 : index
   %c9 = arith.constant 9 : index
@@ -47,14 +47,14 @@ func.func @map_nested_foreach_to_threads_excessive_threads(%x: memref<2 x 32 x f
 transform.sequence failures(propagate) {
 ^bb1(%arg0: !pdl.operation):
   %funcop = transform.structured.match ops{["gpu.launch"]} in %arg0 : (!pdl.operation) -> !pdl.operation
-  // expected-error @below {{Trying to launch a GPU kernel with gridDim = (1, 1, 1) blockDim = (1200, 9, 1). It is larger than the limits.}}
-  // expected-note @below {{"blockDim" is very large}}
-  transform.gpu.map_nested_foreach_to_threads %funcop { blockDim = [1200, 9, 1] }
+  // expected-error @below {{Trying to launch a GPU kernel with grid_dims = (1, 1, 1) block_dims = (1200, 9, 1). It is larger than the limits.}}
+  // expected-note @below {{"block_dims" is too large}}
+  transform.gpu.map_nested_forall_to_threads %funcop block_dims = [1200, 9, 1]
 }
 
 // -----
 
-func.func @map_nested_foreach_to_threads_fewer_threads(%x: memref<2 x 32 x f32>, %y: memref<2 x 32 x f32>, %t: memref<32 x f32>, %alpha : f32, %stream : !gpu.async.token) -> memref<2 x 32 x f32> {
+func.func @map_nested_forall_to_threads_fewer_threads(%x: memref<2 x 32 x f32>, %y: memref<2 x 32 x f32>, %t: memref<32 x f32>, %alpha : f32, %stream : !gpu.async.token) -> memref<2 x 32 x f32> {
   %one = arith.constant 1 : index
   %c900 = arith.constant 900 : index
   %c9 = arith.constant 9 : index
@@ -89,13 +89,13 @@ func.func @map_nested_foreach_to_threads_fewer_threads(%x: memref<2 x 32 x f32>,
 transform.sequence failures(propagate) {
 ^bb1(%arg0: !pdl.operation):
   %funcop = transform.structured.match ops{["gpu.launch"]} in %arg0 : (!pdl.operation) -> !pdl.operation
-  // expected-error @below {{The requested GPU threads are fewer than the number of loop trip counts. Try to tile scf.forall before mapping or set small blockDim.}}
-  transform.gpu.map_nested_foreach_to_threads %funcop { blockDim = [128, 4, 1] }
+  // expected-error @below {{Trying to map to fewer GPU threads than loop iterations but overprovisioning is not yet supported. Try additional tiling of the before mapping or map to more threads.}}
+  transform.gpu.map_nested_forall_to_threads %funcop block_dims = [128, 4, 1]
 }
 
 // -----
 
-func.func @map_nested_foreach_to_threads_dynamic_trip_count(%x: memref<2 x 32 x f32>, %y: memref<2 x 32 x f32>, %t: memref<32 x f32>, %alpha : f32, %stream : !gpu.async.token, %c9 : index, %c7 : index) -> memref<2 x 32 x f32> {
+func.func @map_nested_forall_to_threads_dynamic_trip_count(%x: memref<2 x 32 x f32>, %y: memref<2 x 32 x f32>, %t: memref<32 x f32>, %alpha : f32, %stream : !gpu.async.token, %c9 : index, %c7 : index) -> memref<2 x 32 x f32> {
   %one = arith.constant 1 : index
   %c900 = arith.constant 900 : index
   %name = gpu.launch async[%stream] blocks(%arg3, %arg4, %arg5) in (%arg9 = %one, %arg10 = %one, %arg11 = %one)
@@ -115,13 +115,13 @@ func.func @map_nested_foreach_to_threads_dynamic_trip_count(%x: memref<2 x 32 x
 transform.sequence failures(propagate) {
 ^bb1(%arg0: !pdl.operation):
   %funcop = transform.structured.match ops{["gpu.launch"]} in %arg0 : (!pdl.operation) -> !pdl.operation
-  // expected-error @below {{unsupported dynamic blockdim size}}
-  transform.gpu.map_nested_foreach_to_threads %funcop { blockDim = [128, 4, 1] }
+  // expected-error @below {{unsupported dynamic sizes}}
+  transform.gpu.map_nested_forall_to_threads %funcop block_dims = [128, 4, 1]
 }
 
 // -----
 
-func.func @map_nested_foreach_to_threads_not_buffer(%x: tensor<32x32xf32>, %y: tensor<32x32xf32>, %z: tensor<32x32xf32>, %stream : !gpu.async.token) {
+func.func @map_nested_forall_to_threads_not_buffer(%x: tensor<32x32xf32>, %y: tensor<32x32xf32>, %z: tensor<32x32xf32>, %stream : !gpu.async.token) {
   %one = arith.constant 1 : index
   %name = gpu.launch async[%stream] blocks(%arg3, %arg4, %arg5) in (%arg9 = %one, %arg10 = %one, %arg11 = %one)
             threads(%arg6, %arg7, %arg8) in (%arg12 = %one, %arg13 = %one, %arg14 = %one)
@@ -135,16 +135,16 @@ func.func @map_nested_foreach_to_threads_not_buffer(%x: tensor<32x32xf32>, %y: t
 transform.sequence failures(propagate) {
 ^bb1(%arg0: !pdl.operation):
   %matmul = transform.structured.match ops{["linalg.matmul"]} in %arg0 : (!pdl.operation) -> !pdl.operation
-  %foreach, %tiled = transform.structured.tile_to_forall_op %matmul num_threads [10, 20, 30] (mapping = [ #gpu.thread<y>, #gpu.thread<x>, #gpu.thread<z> ] )
+  %forall, %tiled = transform.structured.tile_to_forall_op %matmul num_threads [10, 20, 30] (mapping = [ #gpu.thread<y>, #gpu.thread<x>, #gpu.thread<z> ] )
   %funcop = transform.structured.match ops{["gpu.launch"]} in %arg0 : (!pdl.operation) -> !pdl.operation
-  // expected-error @below {{only bufferized scf.forall lowers to gpu.thread_id}}
-  transform.gpu.map_nested_foreach_to_threads %funcop { blockDim = [128, 4, 1] }
+  // expected-error @below {{only bufferized scf.forall can be mapped}}
+  transform.gpu.map_nested_forall_to_threads %funcop block_dims = [128, 4, 1]
 }
 
 // -----
 
 
-func.func @map_foreach_to_blocks_not_gpu_launch() -> () {
+func.func @map_forall_to_blocks_not_gpu_launch() -> () {
   // expected-note @below {{when applied to this payload op}}
   %1 = tensor.empty() : tensor<4xf32>
   return
@@ -153,12 +153,12 @@ transform.sequence failures(propagate) {
 ^bb0(%arg0: !pdl.operation):
   %funcop = transform.structured.match ops{["tensor.empty"]} in %arg0 : (!pdl.operation) -> !pdl.operation
   // expected-error @below {{Given target is not gpu.launch}}
-  %1 = transform.gpu.map_foreach_to_blocks %funcop
+  %1 = transform.gpu.map_forall_to_blocks %funcop
 }
 
 // -----
 
-func.func @map_foreach_to_blocks_not_unique(%x: memref<2 x 32 x f32>, %y: memref<2 x 32 x f32>, %t: memref<32 x f32>, %alpha : f32, %stream : !gpu.async.token) -> memref<2 x 32 x f32> {
+func.func @map_forall_to_blocks_not_unique(%x: memref<2 x 32 x f32>, %y: memref<2 x 32 x f32>, %t: memref<32 x f32>, %alpha : f32, %stream : !gpu.async.token) -> memref<2 x 32 x f32> {
   %one = arith.constant 1 : index
   %c900 = arith.constant 900 : index
   %c9 = arith.constant 9 : index
@@ -190,13 +190,13 @@ transform.sequence failures(propagate) {
 ^bb0(%arg0: !pdl.operation):
   %funcop = transform.structured.match ops{["gpu.launch"]} in %arg0 : (!pdl.operation) -> !pdl.operation
   // expected-error @below {{could not find a unique topLevel scf.forall}}
-  %1 = transform.gpu.map_foreach_to_blocks %funcop
+  %1 = transform.gpu.map_forall_to_blocks %funcop
 }
 
 // -----
 
 // expected-note @below {{when applied to this payload op}}
-func.func @map_foreach_to_blocks_large_loop(%x: memref<2 x 32 x f32>, %y: memref<2 x 32 x f32>, %t: memref<32 x f32>, %alpha : f32, %stream : !gpu.async.token) -> memref<2 x 32 x f32> {
+func.func @map_forall_to_blocks_large_loop(%x: memref<2 x 32 x f32>, %y: memref<2 x 32 x f32>, %t: memref<32 x f32>, %alpha : f32, %stream : !gpu.async.token) -> memref<2 x 32 x f32> {
   %one = arith.constant 1 : index
   %c65537 = arith.constant 65536 : index
   %c9 = arith.constant 9 : index
@@ -223,12 +223,12 @@ transform.sequence failures(propagate) {
 ^bb0(%arg0: !pdl.operation):
   %funcop = transform.structured.match ops{["func.func"]} in %arg0 : (!pdl.operation) -> !pdl.operation
   // expected-error @below {{could not find a unique topLevel scf.forall}}
-  %1 = transform.gpu.map_foreach_to_blocks %funcop { generate_gpu_launch }
+  %1 = transform.gpu.map_forall_to_blocks %funcop { generate_gpu_launch }
 }
 
 // -----
 
-func.func @map_foreach_to_blocks_large_loop(%x: memref<2 x 32 x f32>, %y: memref<2 x 32 x f32>, %t: memref<32 x f32>, %alpha : f32, %stream : !gpu.async.token) -> memref<2 x 32 x f32> {
+func.func @map_forall_to_blocks_large_loop(%x: memref<2 x 32 x f32>, %y: memref<2 x 32 x f32>, %t: memref<32 x f32>, %alpha : f32, %stream : !gpu.async.token) -> memref<2 x 32 x f32> {
   %one = arith.constant 1 : index
   %c65535 = arith.constant 65535 : index
   scf.forall (%i, %j) in (%c65535, %c65535) {
@@ -243,8 +243,8 @@ func.func @map_foreach_to_blocks_large_loop(%x: memref<2 x 32 x f32>, %y: memref
 transform.sequence failures(propagate) {
 ^bb0(%arg0: !pdl.operation):
   %funcop = transform.structured.match ops{["func.func"]} in %arg0 : (!pdl.operation) -> !pdl.operation
-  // expected-error @below {{Trying to launch a GPU kernel with gridDim = (65535, 65535, 1) blockDim = (1, 1, 1). It is larger than the limits.}}
-  %1 = transform.gpu.map_foreach_to_blocks %funcop { generate_gpu_launch }
+  // expected-error @below {{Trying to launch a GPU kernel with grid_dims = (65535, 65535, 1) block_dims = (1, 1, 1). It is larger than the limits.}}
+  %1 = transform.gpu.map_forall_to_blocks %funcop generate_gpu_launch
 }
 
 // -----
@@ -270,8 +270,8 @@ func.func @saxpy2d_singleloop(%x: !type, %y: !type, %stream : !gpu.async.token)
 transform.sequence failures(propagate) {
 ^bb1(%arg0: !pdl.operation):
   %funcop = transform.structured.match ops{["gpu.launch"]} in %arg0 : (!pdl.operation) -> !pdl.operation
-  // expected-error @below {{#gpu.thread<x> is duplicated, cannot map different loops to the same processor}}
-  transform.gpu.map_nested_foreach_to_threads %funcop { blockDim = [32, 32]}
+  // expected-error @below {{duplicated attribute, cannot map different loops to the same processor}}
+  transform.gpu.map_nested_forall_to_threads %funcop block_dims = [32, 32, 1]
 }
 
 // -----
@@ -301,5 +301,5 @@ transform.sequence failures(propagate) {
 ^bb1(%arg0: !pdl.operation):
   %matmul = transform.structured.match ops{["linalg.generic"]} in %arg0 : (!pdl.operation) -> !pdl.operation
   // expected-error @below {{transform.structured.tile_to_forall_op failed to apply}}
-  %foreach, %tiled = transform.structured.tile_to_forall_op %matmul num_threads [10, 20, 30] (mapping = [ #gpu.thread<y>, #gpu.thread<x>, #gpu.thread<z> ] )
+  %forall, %tiled = transform.structured.tile_to_forall_op %matmul num_threads [10, 20, 30] (mapping = [ #gpu.thread<y>, #gpu.thread<x>, #gpu.thread<z> ] )
 }
diff --git a/mlir/test/Dialect/GPU/transform-gpu.mlir b/mlir/test/Dialect/GPU/transform-gpu.mlir
index 9ae28a5e64fc8..fcf56c8024bfa 100644
--- a/mlir/test/Dialect/GPU/transform-gpu.mlir
+++ b/mlir/test/Dialect/GPU/transform-gpu.mlir
@@ -33,7 +33,7 @@ func.func @saxpy2dblock(%x: !type, %y: !type, %t: !type1d, %alpha : f32, %stream
 transform.sequence failures(propagate) {
 ^bb1(%arg0: !pdl.operation):
   %funcop = transform.structured.match ops{["gpu.launch"]} in %arg0 : (!pdl.operation) -> !pdl.operation
-  transform.gpu.map_foreach_to_blocks %funcop { gridDim = [12, 9]}
+  transform.gpu.map_forall_to_blocks %funcop grid_dims = [12, 9, 1]
 }
 
 // -----
@@ -87,7 +87,7 @@ func.func @saxpy2d(%x: !type, %y: !type, %t: !type1d, %alpha : f32, %stream : !g
 transform.sequence failures(propagate) {
 ^bb1(%arg0: !pdl.operation):
   %funcop = transform.structured.match ops{["gpu.launch"]} in %arg0 : (!pdl.operation) -> !pdl.operation
-  transform.gpu.map_nested_foreach_to_threads %funcop { blockDim = [12, 9] }
+  transform.gpu.map_nested_forall_to_threads %funcop block_dims = [12, 9, 1]
 }
 
 // -----
@@ -126,8 +126,8 @@ func.func @saxpy4d(%x: !type4d, %y: !type4d, %alpha : f32) -> !type4d {
 transform.sequence failures(propagate) {
 ^bb1(%arg0: !pdl.operation):
   %funcop = transform.structured.match ops{["func.func"]} in %arg0 : (!pdl.operation) -> !pdl.operation
-  %gpuLaunch = transform.gpu.map_foreach_to_blocks %funcop { generate_gpu_launch }
-  transform.gpu.map_nested_foreach_to_threads %gpuLaunch { blockDim = [32, 4, 1] }
+  %gpuLaunch = transform.gpu.map_forall_to_blocks %funcop { generate_gpu_launch }
+  transform.gpu.map_nested_forall_to_threads %gpuLaunch block_dims = [32, 4, 1]
 }
 
 // -----
@@ -160,7 +160,7 @@ func.func @saxpy2d_no_barrier(%x: !type, %y: !type, %t: !type1d, %alpha : f32, %
 transform.sequence failures(propagate) {
 ^bb1(%arg0: !pdl.operation):
   %funcop = transform.structured.match ops{["gpu.launch"]} in %arg0 : (!pdl.operation) -> !pdl.operation
-  transform.gpu.map_nested_foreach_to_threads %funcop { blockDim = [12, 9, 1], syncAfterDistribute = false }
+  transform.gpu.map_nested_forall_to_threads %funcop block_dims = [12, 9, 1] sync_after_distribute = false
 }
 
 // -----
@@ -192,7 +192,7 @@ func.func @saxpy2d_singleloop(%x: !type, %y: !type, %stream : !gpu.async.token)
 transform.sequence failures(propagate) {
 ^bb1(%arg0: !pdl.operation):
   %funcop = transform.structured.match ops{["gpu.launch"]} in %arg0 : (!pdl.operation) -> !pdl.operation
-  transform.gpu.map_nested_foreach_to_threads %funcop { blockDim = [32]}
+  transform.gpu.map_nested_forall_to_threads %funcop block_dims = [32, 1, 1]
 }
 
 // -----
@@ -228,7 +228,7 @@ func.func @saxpy3d_fold_id_z(%x: !type, %y: !type, %t: !type1d, %alpha : f32, %s
 transform.sequence failures(propagate) {
 ^bb1(%arg0: !pdl.operation):
   %funcop = transform.structured.match ops{["gpu.launch"]} in %arg0 : (!pdl.operation) -> !pdl.operation
-  transform.gpu.map_nested_foreach_to_threads %funcop { blockDim = [12, 9, 1], syncAfterDistribute = false }
+  transform.gpu.map_nested_forall_to_threads %funcop block_dims = [12, 9, 1] sync_after_distribute = false
 }
 
 // -----
@@ -236,29 +236,64 @@ transform.sequence failures(propagate) {
 !type = memref<2 x 32 x f32>
 !type1d = memref<32 x f32>
 
+// CHECK-DAG: #[[$MAPWY:.*]] = affine_map<(d0, d1) -> (((d0 + d1 * 12) floordiv 32) floordiv 4)>
+// CHECK-DAG: #[[$MAPWX:.*]] = affine_map<(d0, d1) -> ((((d0 + d1 * 12) floordiv 32) mod 4) floordiv 2)>
+
+// CHECK-DAG: #[[$MAPLIN:.*]] = affine_map<(d0, d1) -> (d0 + d1 * 12)>
+// CHECK-DAG: #[[$MAPLY:.*]] = affine_map<(d0, d1) -> ((d0 + d1 * 12) floordiv 20)>
+// CHECK-DAG: #[[$MAPLX:.*]] = affine_map<(d0, d1) -> (((d0 + d1 * 12) mod 20) floordiv 10)>
+
 // CHECK-LABEL: func.func @map_multi_level(
 func.func @map_multi_level(%x: !type, %y: !type, %t: !type1d, %alpha : f32, %stream : !gpu.async.token) -> !type {
   %one = arith.constant 1 : index
-  %c12 = arith.constant 12 : index
+  %c10 = arith.constant 10 : index
   %c9 = arith.constant 9 : index
   %c7 = arith.constant 7 : index
-// check that the thread level got distributed but not the warp level.
-//  CHECK-NOT:  {mapping = #gpu.thread
-//      CHECK:  {mapping = [#gpu.warp<x>]}
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+
+  // CHECK-DAG: %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG: %[[C11:.*]] = arith.constant 11 : index
+  // CHECK-DAG: %[[C12:.*]] = arith.constant 12 : index
+  // CHECK-DAG: %[[C20:.*]] = arith.constant 20 : index
+
+  // check that both the thread level and the warp level got distributed.
+  //  CHECK-NOT: #gpu.thread
+  //  CHECK-NOT: #gpu.warp
   %name = gpu.launch async[%stream] blocks(%arg3, %arg4, %arg5) in (%arg9 = %one, %arg10 = %one, %arg11 = %one)
             threads(%arg6, %arg7, %arg8) in (%arg12 = %one, %arg13 = %one, %arg14 = %one)
   {
+    // CHECK-DAG: %[[TIDX:.*]] = gpu.thread_id  x
+    // CHECK-DAG: %[[TIDY:.*]] = gpu.thread_id  y
     scf.forall (%i, %j) in (%c7, %c9) {
-        %4 = memref.load %x[%i, %j] : !type
-        %5 = memref.load %y[%i, %j] : !type
-        %6 = math.fma %alpha, %4, %5 : f32
-        memref.store %6, %y[%i, %j] : !type
-     }  { mapping = [#gpu.thread<y>, #gpu.thread<x>]}
-     scf.forall (%i) in (%c12) {
+      %4 = memref.load %x[%i, %j] : !type
+      %5 = memref.load %y[%i, %j] : !type
+      %6 = math.fma %alpha, %4, %5 : f32
+      memref.store %6, %y[%i, %j] : !type
+    }  { mapping = [#gpu.thread<y>, #gpu.thread<x>]}
+
+    // CHECK-DAG: %[[WIDY:.*]] = affine.apply #[[$MAPWY]](%[[TIDX]], %[[TIDY]])
+    // CHECK-DAG: %[[WIDX:.*]] = affine.apply #[[$MAPWX]](%[[TIDX]], %[[TIDY]])
+    // CHECK-DAG: %[[CMPX:.*]] = arith.cmpi ult, %[[WIDX]], %[[C1]] : index
+    // CHECK-DAG: %[[CMPY:.*]] = arith.cmpi ult, %[[WIDY]], %[[C1]] : index
+    //     CHECK: %[[COND:.*]] = arith.andi %[[CMPY]], %[[CMPX]] : i1
+    //     CHECK: scf.if %[[COND]]
+    scf.forall (%i) in (%c1) {
         %7 = memref.load %t[%i] : !type1d
         %8 = arith.addf %alpha, %7 : f32
         memref.store %8, %t[%i] : !type1d
      }  {mapping = [#gpu.warp<x>] }
+
+    // CHECK-DAG: %[[LIN:.*]] = affine.apply #[[$MAPLIN]](%[[TIDX]], %[[TIDY]])
+    // CHECK-DAG: %[[LIDY:.*]] = affine.apply #[[$MAPLY]](%[[TIDX]], %[[TIDY]])
+    // CHECK-DAG: %[[LIDZ:.*]] = affine.apply #[[$MAPLX]](%[[TIDX]], %[[TIDY]])
+    // CHECK-DAG: %[[COND:.*]] = arith.cmpi ult, %[[LIN]], %[[C20]] : index
+    //     CHECK: scf.if %[[COND]]
+    scf.forall (%i, %j) in (%c10, %c2) {
+        %7 = memref.load %t[%i] : !type1d
+        %8 = arith.addf %alpha, %7 : f32
+        memref.store %8, %t[%j] : !type1d
+     }  {mapping = [#gpu.linear<x>, #gpu.linear<y>] }
     gpu.terminator
   }
   return %y : !type
@@ -267,5 +302,6 @@ func.func @map_multi_level(%x: !type, %y: !type, %t: !type1d, %alpha : f32, %str
 transform.sequence failures(propagate) {
 ^bb1(%arg0: !pdl.operation):
   %funcop = transform.structured.match ops{["gpu.launch"]} in %arg0 : (!pdl.operation) -> !pdl.operation
-  transform.gpu.map_nested_foreach_to_threads %funcop { blockDim = [12, 9] }
+  transform.gpu.map_nested_forall_to_threads %funcop
+    block_dims = [12, 11, 1] warp_dims = [2, 2, 1]
 }
diff --git a/mlir/test/Dialect/LLVMIR/func.mlir b/mlir/test/Dialect/LLVMIR/func.mlir
index 4ad8f445c7844..5cc7d75b627fa 100644
--- a/mlir/test/Dialect/LLVMIR/func.mlir
+++ b/mlir/test/Dialect/LLVMIR/func.mlir
@@ -194,6 +194,16 @@ module {
   llvm.func @memory_attr() attributes {memory = #llvm.memory_effects<other = none, argMem = read, inaccessibleMem = readwrite>} {
     llvm.return
   }
+
+  // CHECK-LABEL: llvm.func hidden @hidden
+  llvm.func hidden @hidden() {
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func protected @protected
+  llvm.func protected @protected() {
+    llvm.return
+  }
 }
 
 // -----
diff --git a/mlir/test/Dialect/LLVMIR/global.mlir b/mlir/test/Dialect/LLVMIR/global.mlir
index 37c55ecfe7a0d..2f0850834a0ef 100644
--- a/mlir/test/Dialect/LLVMIR/global.mlir
+++ b/mlir/test/Dialect/LLVMIR/global.mlir
@@ -90,6 +90,16 @@ llvm.mlir.global private unnamed_addr constant @foo(42 : i64) : i64
 // CHECK: llvm.mlir.global internal constant @sectionvar("teststring") {addr_space = 0 : i32, section = ".mysection"}
 llvm.mlir.global internal constant @sectionvar("teststring")  {section = ".mysection"}: !llvm.array<10 x i8>
 
+// CHECK: llvm.mlir.global internal thread_local constant @thread_local(42 : i32)
+llvm.mlir.global internal thread_local constant @thread_local(42 : i32) : i32
+
+// Visibility types.
+// CHECK: llvm.mlir.global internal hidden constant @hidden(42 : i32)
+llvm.mlir.global internal hidden constant @hidden(42 : i32) : i32
+
+// CHECK: llvm.mlir.global internal protected unnamed_addr @protected(42 : i32)
+llvm.mlir.global internal protected unnamed_addr @protected(42 : i32) : i32
+
 // -----
 
 // expected-error @+1 {{op requires attribute 'sym_name'}}
diff --git a/mlir/test/Dialect/LLVMIR/roundtrip.mlir b/mlir/test/Dialect/LLVMIR/roundtrip.mlir
index f7b340fd66ac8..c9db19b409e25 100644
--- a/mlir/test/Dialect/LLVMIR/roundtrip.mlir
+++ b/mlir/test/Dialect/LLVMIR/roundtrip.mlir
@@ -325,8 +325,8 @@ func.func @mixed_vect(%arg0: vector<8xf32>, %arg1: vector<4xf32>, %arg2: vector<
 func.func @alloca(%size : i64) {
   // CHECK: llvm.alloca %{{.*}} x i32 : (i64) -> !llvm.ptr<i32>
   llvm.alloca %size x i32 {alignment = 0} : (i64) -> (!llvm.ptr<i32>)
-  // CHECK: llvm.alloca %{{.*}} x i32 {alignment = 8 : i64} : (i64) -> !llvm.ptr<i32>
-  llvm.alloca %size x i32 {alignment = 8} : (i64) -> (!llvm.ptr<i32>)
+  // CHECK: llvm.alloca inalloca %{{.*}} x i32 {alignment = 8 : i64} : (i64) -> !llvm.ptr<i32>
+  llvm.alloca inalloca %size x i32 {alignment = 8} : (i64) -> (!llvm.ptr<i32>)
   llvm.return
 }
 
@@ -516,6 +516,11 @@ func.func @fastmathFlags(%arg0: f32, %arg1: f32, %arg2: i32, %arg3: vector<2 x f
   %11 = llvm.intr.sin(%arg0) {fastmathFlags = #llvm.fastmath<fast>} : (f32) -> f32
 // CHECK: {{.*}} = llvm.intr.sin(%arg0) {fastmathFlags = #llvm.fastmath<afn>} : (f32) -> f32
   %12 = llvm.intr.sin(%arg0) {fastmathFlags = #llvm.fastmath<afn>} : (f32) -> f32
+
+// CHECK: {{.*}} = llvm.intr.vector.reduce.fmin(%arg3) {fastmathFlags = #llvm.fastmath<nnan>} : (vector<2xf32>) -> f32
+  %13 = llvm.intr.vector.reduce.fmin(%arg3) {fastmathFlags = #llvm.fastmath<nnan>} : (vector<2xf32>) -> f32
+// CHECK: {{.*}} = llvm.intr.vector.reduce.fmax(%arg3) {fastmathFlags = #llvm.fastmath<nnan>} : (vector<2xf32>) -> f32
+  %14 = llvm.intr.vector.reduce.fmax(%arg3) {fastmathFlags = #llvm.fastmath<nnan>} : (vector<2xf32>) -> f32
   return
 }
 
diff --git a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
index e023e64c7cc8b..82795ec9d4bf8 100644
--- a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
@@ -56,6 +56,23 @@ func.func @depthwise_conv_2d_nhwc_hwc(%input: tensor<1x1x113x96xf32>, %filter: t
   return %0: tensor<1x1x56x96xf32>
 }
 
+// CHECK-LABEL: @conv_2d
+// CHECK-SAME: (%[[ARG0:[0-9a-z]+]]: tensor<1x?xf32>,
+// CHECK-SAME: %[[ARG1:[0-9a-z]+]]: tensor<1x?xf32>,
+// CHECK-SAME: %[[ARG2:[0-9a-z]+]]: tensor<1x?xf32>)
+func.func @conv_2d(%input: tensor<1x?xf32>, %filter: tensor<1x?xf32>, %init: tensor<1x?xf32>) -> tensor<1x?xf32> {
+  // CHECK: %[[SLICE0:.+]] = tensor.extract_slice %[[ARG0]]
+  // CHECK: %[[SLICE1:.+]] = tensor.extract_slice %[[ARG1]]
+  // CHECK: %[[SLICE2:.+]] = tensor.extract_slice %[[ARG2]]
+  // CHECK: %[[SLICERES:.+]] = linalg.conv_1d
+  // CHECK: %[[RES:.+]] = tensor.insert_slice %[[SLICERES]] into %[[ARG2]]
+  %0 = linalg.conv_2d
+     ins (%input, %filter: tensor<1x?xf32>, tensor<1x?xf32>)
+    outs (%init: tensor<1x?xf32>) -> tensor<1x?xf32>
+  // CHECK: return %[[RES]]
+  return %0 : tensor<1x?xf32>
+}
+
 // CHECK-LABEL: @pooling_nhwc_sum
 // CHECK-SAME: %[[ARG0:.+]]: tensor<?x1x?x?xf32>,
 // CHECK-SAME: %[[ARG1:.+]]: tensor<1x?xf32>
diff --git a/mlir/test/Dialect/Linalg/vectorization.mlir b/mlir/test/Dialect/Linalg/vectorization.mlir
index 41e52909fffd9..26e27c108ce81 100644
--- a/mlir/test/Dialect/Linalg/vectorization.mlir
+++ b/mlir/test/Dialect/Linalg/vectorization.mlir
@@ -1783,6 +1783,211 @@ func.func @vectorize_nd_tensor_extract_contiguous_and_gather(%arg0: tensor<6xf32
 // CHECK:           %[[VAL_14:.*]] = vector.transfer_write %[[VAL_13]], %[[VAL_8]]{{\[}}%[[VAL_2]]] {in_bounds = [true]} : vector<5xf32>, tensor<5xf32>
 // CHECK:           return %[[VAL_14]] : tensor<5xf32>
 
+transform.sequence failures(propagate) {
+ ^bb1(%arg1: !pdl.operation):
+   %0 = transform.structured.match ops{["linalg.generic"]} in %arg1 : (!pdl.operation) -> !pdl.operation
+   %1 = get_closest_isolated_parent %0 : (!pdl.operation) -> !pdl.operation
+   %2 = transform.structured.vectorize %1 { vectorize_nd_extract }
+ }
+
+// -----
+
+// The vectorizer converts `affine.apply` so that the subsequent Ops can be vectorised based on the converted ops. Contiguous load.
+func.func @vectorize_nd_tensor_extract_with_affine_apply_contiguous(%6: tensor<80x16xf32>, %arg0: index, %extracted_slice : tensor<1x4xf32>) -> tensor<1x4xf32> {
+  %c79 = arith.constant 79 : index
+  %1 = linalg.generic {
+    indexing_maps = [affine_map<(d0, d1) -> (d0, d1)>],
+    iterator_types = ["parallel", "parallel"]
+  } outs(%extracted_slice : tensor<1x4xf32>) {
+  ^bb0(%out: f32):
+    %2 = linalg.index 1 : index
+    %3 = affine.apply affine_map<(d0, d1) -> (d0 + d1)>(%2, %arg0)
+    %extracted = tensor.extract %6[%c79, %3] : tensor<80x16xf32>
+    linalg.yield %extracted : f32
+  } -> tensor<1x4xf32>
+  return %1 : tensor<1x4xf32>
+}
+
+// CHECK-LABEL:   func.func @vectorize_nd_tensor_extract_with_affine_apply_contiguous(
+// CHECK-SAME:                                                                        %[[VAL_0:.*]]: tensor<80x16xf32>,
+// CHECK-SAME:                                                                        %[[VAL_1:.*]]: index,
+// CHECK-SAME:                                                                        %[[VAL_2:.*]]: tensor<1x4xf32>) -> tensor<1x4xf32> {
+// CHECK:           %[[VAL_3:.*]] = arith.constant dense<[0, 1, 2, 3]> : vector<4xindex>
+// CHECK:           %[[VAL_4:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_5:.*]] = arith.constant 0.000000e+00 : f32
+// CHECK:           %[[VAL_6:.*]] = arith.constant 0 : index
+// CHECK:           %[[VAL_7:.*]] = arith.constant 79 : index
+// CHECK:           %[[VAL_8:.*]] = vector.broadcast %[[VAL_1]] : index to vector<4xindex>
+// CHECK:           %[[VAL_9:.*]] = arith.addi %[[VAL_8]], %[[VAL_3]] : vector<4xindex>
+// CHECK:           %[[VAL_10:.*]] = vector.extractelement %[[VAL_9]]{{\[}}%[[VAL_4]] : i32] : vector<4xindex>
+// CHECK:           %[[VAL_11:.*]] = vector.transfer_read %[[VAL_0]]{{\[}}%[[VAL_7]], %[[VAL_10]]], %[[VAL_5]] {in_bounds = [true, true]} : tensor<80x16xf32>, vector<1x4xf32>
+// CHECK:           %[[VAL_12:.*]] = vector.transfer_write %[[VAL_11]], %[[VAL_2]]{{\[}}%[[VAL_6]], %[[VAL_6]]] {in_bounds = [true, true]} : vector<1x4xf32>, tensor<1x4xf32>
+// CHECK:           return %[[VAL_12]] : tensor<1x4xf32>
+// CHECK:         }
+
+transform.sequence failures(propagate) {
+ ^bb1(%arg1: !pdl.operation):
+   %0 = transform.structured.match ops{["linalg.generic"]} in %arg1 : (!pdl.operation) -> !pdl.operation
+   %1 = get_closest_isolated_parent %0 : (!pdl.operation) -> !pdl.operation
+   %2 = transform.structured.vectorize %1 { vectorize_nd_extract }
+ }
+
+// -----
+
+// The vectorizer converts `affine.apply` so that the subsequent Ops can be vectorised based on the converted ops. Gather load.
+func.func @vectorize_nd_tensor_extract_with_affine_apply_gather(%6: tensor<80x16xf32>, %arg0: index, %extracted_slice : tensor<1x4xf32>) -> tensor<1x4xf32> {
+  %c16 = arith.constant 16 : index
+  %1 = linalg.generic {
+    indexing_maps = [affine_map<(d0, d1) -> (d0, d1)>],
+    iterator_types = ["parallel", "parallel"]
+  } outs(%extracted_slice : tensor<1x4xf32>) {
+  ^bb0(%out: f32):
+    %2 = linalg.index 1 : index
+    %3 = affine.apply affine_map<(d0, d1) -> (d0 + d1)>(%2, %arg0)
+    %extracted = tensor.extract %6[%3, %c16] : tensor<80x16xf32>
+    linalg.yield %extracted : f32
+  } -> tensor<1x4xf32>
+  return %1 : tensor<1x4xf32>
+}
+
+// CHECK-LABEL:   func.func @vectorize_nd_tensor_extract_with_affine_apply_gather(
+// CHECK-SAME:                                                                    %[[VAL_0:.*]]: tensor<80x16xf32>,
+// CHECK-SAME:                                                                    %[[VAL_1:.*]]: index,
+// CHECK-SAME:                                                                    %[[VAL_2:.*]]: tensor<1x4xf32>) -> tensor<1x4xf32> {
+// CHECK:           %[[VAL_3:.*]] = arith.constant dense<[0, 1, 2, 3]> : vector<4xindex>
+// CHECK:           %[[VAL_4:.*]] = arith.constant dense<true> : vector<1x4xi1>
+// CHECK:           %[[VAL_5:.*]] = arith.constant dense<0.000000e+00> : vector<1x4xf32>
+// CHECK:           %[[VAL_6:.*]] = arith.constant 0 : index
+// CHECK:           %[[VAL_7:.*]] = arith.constant dense<16> : vector<1x4xindex>
+// CHECK:           %[[VAL_8:.*]] = vector.broadcast %[[VAL_1]] : index to vector<4xindex>
+// CHECK:           %[[VAL_9:.*]] = arith.addi %[[VAL_8]], %[[VAL_3]] : vector<4xindex>
+// CHECK:           %[[VAL_10:.*]] = vector.broadcast %[[VAL_9]] : vector<4xindex> to vector<1x4xindex>
+// CHECK:           %[[VAL_11:.*]] = arith.muli %[[VAL_10]], %[[VAL_7]] : vector<1x4xindex>
+// CHECK:           %[[VAL_12:.*]] = arith.addi %[[VAL_11]], %[[VAL_7]] : vector<1x4xindex>
+// CHECK:           %[[VAL_13:.*]] = vector.gather %[[VAL_0]]{{\[}}%[[VAL_6]], %[[VAL_6]]] {{\[}}%[[VAL_12]]], %[[VAL_4]], %[[VAL_5]] : tensor<80x16xf32>, vector<1x4xindex>, vector<1x4xi1>, vector<1x4xf32> into vector<1x4xf32>
+// CHECK:           %[[VAL_14:.*]] = vector.transfer_write %[[VAL_13]], %[[VAL_2]]{{\[}}%[[VAL_6]], %[[VAL_6]]] {in_bounds = [true, true]} : vector<1x4xf32>, tensor<1x4xf32>
+// CHECK:           return %[[VAL_14]] : tensor<1x4xf32>
+// CHECK:         }
+
+transform.sequence failures(propagate) {
+ ^bb1(%arg1: !pdl.operation):
+   %0 = transform.structured.match ops{["linalg.generic"]} in %arg1 : (!pdl.operation) -> !pdl.operation
+   %1 = get_closest_isolated_parent %0 : (!pdl.operation) -> !pdl.operation
+   %2 = transform.structured.vectorize %1 { vectorize_nd_extract }
+ }
+
+// -----
+
+// Make sure that non-linear arithmetic operations (e.g. arith.maxsi) are allowed when calculating indices for load operations. Gather load.
+func.func @vectorize_nd_tensor_extract_with_maxsi_gather(%arg0: tensor<80x16xf32>, %extracted_slice : tensor<1x4xf32>) -> tensor<1x4xf32> {
+  %c79 = arith.constant 79 : index
+  %1 = linalg.generic {
+    indexing_maps = [affine_map<(d0, d1) -> (d0, d1)>],
+    iterator_types = ["parallel", "parallel"]
+  } outs(%extracted_slice : tensor<1x4xf32>) {
+  ^bb0(%out: f32):
+    %2 = linalg.index 1 : index
+    %3 = arith.maxsi %2, %c79 : index
+    %extracted = tensor.extract %arg0[%3, %2] : tensor<80x16xf32>
+    linalg.yield %extracted : f32
+  } -> tensor<1x4xf32>
+  return %1 : tensor<1x4xf32>
+}
+
+// CHECK-LABEL:   func.func @vectorize_nd_tensor_extract_with_maxsi_gather(
+// CHECK-SAME:                                                             %[[VAL_0:.*]]: tensor<80x16xf32>,
+// CHECK-SAME:                                                             %[[VAL_1:.*]]: tensor<1x4xf32>) -> tensor<1x4xf32> {
+// CHECK:           %[[VAL_2:.*]] = arith.constant dense<[0, 1, 2, 3]> : vector<4xindex>
+// CHECK:           %[[VAL_3:.*]] = arith.constant dense<1264> : vector<1x4xindex>
+// CHECK:           %[[VAL_4:.*]] = arith.constant dense<true> : vector<1x4xi1>
+// CHECK:           %[[VAL_5:.*]] = arith.constant dense<0.000000e+00> : vector<1x4xf32>
+// CHECK:           %[[VAL_6:.*]] = arith.constant 0 : index
+// CHECK:           %[[VAL_7:.*]] = vector.broadcast %[[VAL_2]] : vector<4xindex> to vector<1x4xindex>
+// CHECK:           %[[VAL_8:.*]] = arith.addi %[[VAL_7]], %[[VAL_3]] : vector<1x4xindex>
+// CHECK:           %[[VAL_9:.*]] = vector.gather %[[VAL_0]]{{\[}}%[[VAL_6]], %[[VAL_6]]] {{\[}}%[[VAL_8]]], %[[VAL_4]], %[[VAL_5]] : tensor<80x16xf32>, vector<1x4xindex>, vector<1x4xi1>, vector<1x4xf32> into vector<1x4xf32>
+// CHECK:           %[[VAL_10:.*]] = vector.transfer_write %[[VAL_9]], %[[VAL_1]]{{\[}}%[[VAL_6]], %[[VAL_6]]] {in_bounds = [true, true]} : vector<1x4xf32>, tensor<1x4xf32>
+// CHECK:           return %[[VAL_10]] : tensor<1x4xf32>
+// CHECK:         }
+
+transform.sequence failures(propagate) {
+ ^bb1(%arg1: !pdl.operation):
+   %0 = transform.structured.match ops{["linalg.generic"]} in %arg1 : (!pdl.operation) -> !pdl.operation
+   %1 = get_closest_isolated_parent %0 : (!pdl.operation) -> !pdl.operation
+   %2 = transform.structured.vectorize %1 { vectorize_nd_extract }
+ }
+
+// -----
+
+// Make sure that non-linear arithmetic operations (e.g. arith.maxsi) are allowed when calculating indices for load operations. Contiguous load.
+func.func @vectorize_nd_tensor_extract_with_maxsi_contiguous(%arg0: tensor<80x16xf32>, %extracted_slice : tensor<1x4xf32>) -> tensor<1x4xf32> {
+  %c16 = arith.constant 16 : index
+  %1 = linalg.generic {
+    indexing_maps = [affine_map<(d0, d1) -> (d0, d1)>],
+    iterator_types = ["parallel", "parallel"]
+  } outs(%extracted_slice : tensor<1x4xf32>) {
+  ^bb0(%out: f32):
+    %2 = linalg.index 0 : index
+    %3 = linalg.index 1 : index
+    %4 = arith.maxsi %2, %c16 : index
+    %extracted = tensor.extract %arg0[%4, %3] : tensor<80x16xf32>
+    linalg.yield %extracted : f32
+  } -> tensor<1x4xf32>
+  return %1 : tensor<1x4xf32>
+}
+
+// CHECK-LABEL:   func.func @vectorize_nd_tensor_extract_with_maxsi_contiguous(
+// CHECK-SAME:                                                                 %[[VAL_0:.*]]: tensor<80x16xf32>,
+// CHECK-SAME:                                                                 %[[VAL_1:.*]]: tensor<1x4xf32>) -> tensor<1x4xf32> {
+// CHECK:           %[[VAL_2:.*]] = arith.constant dense<16> : vector<1x4xindex>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_4:.*]] = arith.constant 0 : index
+// CHECK:           %[[VAL_5:.*]] = arith.constant 0.000000e+00 : f32
+// CHECK:           %[[VAL_6:.*]] = vector.shape_cast %[[VAL_2]] : vector<1x4xindex> to vector<4xindex>
+// CHECK:           %[[VAL_7:.*]] = vector.extractelement %[[VAL_6]]{{\[}}%[[VAL_3]] : i32] : vector<4xindex>
+// CHECK:           %[[VAL_8:.*]] = vector.transfer_read %[[VAL_0]]{{\[}}%[[VAL_7]], %[[VAL_4]]], %[[VAL_5]] {in_bounds = [true, true]} : tensor<80x16xf32>, vector<1x4xf32>
+// CHECK:           %[[VAL_9:.*]] = vector.transfer_write %[[VAL_8]], %[[VAL_1]]{{\[}}%[[VAL_4]], %[[VAL_4]]] {in_bounds = [true, true]} : vector<1x4xf32>, tensor<1x4xf32>
+// CHECK:           return %[[VAL_9]] : tensor<1x4xf32>
+// CHECK:         }
+
+transform.sequence failures(propagate) {
+ ^bb1(%arg1: !pdl.operation):
+   %0 = transform.structured.match ops{["linalg.generic"]} in %arg1 : (!pdl.operation) -> !pdl.operation
+   %1 = get_closest_isolated_parent %0 : (!pdl.operation) -> !pdl.operation
+   %2 = transform.structured.vectorize %1 { vectorize_nd_extract }
+ }
+
+// -----
+
+// The vectorizer assumes it's a gather load whenever using a block argument to calculate an index.
+#map = affine_map<(d0) -> (d0)>
+func.func @vectorize_nd_tensor_extract_block_arg(%arg0: tensor<5x6xf32>, %arg1: tensor<5xindex>) -> tensor<5xf32> {
+ %0 = tensor.empty() : tensor<5xf32>
+ %1 = linalg.generic {indexing_maps = [#map, #map], iterator_types = ["parallel"]} ins(%arg1: tensor<5xindex>) outs(%0 : tensor<5xf32>) {
+ ^bb0(%in: index, %out: f32):
+   %2 = linalg.index 0 : index
+   %extracted_0 = tensor.extract %arg0[%in, %2] : tensor<5x6xf32>
+   linalg.yield %extracted_0 : f32
+ } -> tensor<5xf32>
+ return %1 : tensor<5xf32>
+}
+
+// CHECK-LABEL:   func.func @vectorize_nd_tensor_extract_block_arg(
+// CHECK-SAME:                                                     %[[VAL_0:.*]]: tensor<5x6xf32>,
+// CHECK-SAME:                                                     %[[VAL_1:.*]]: tensor<5xindex>) -> tensor<5xf32> {
+// CHECK:           %[[VAL_2:.*]] = arith.constant 0 : index
+// CHECK:           %[[VAL_3:.*]] = arith.constant dense<[0, 1, 2, 3, 4]> : vector<5xindex>
+// CHECK:           %[[VAL_4:.*]] = arith.constant dense<true> : vector<5xi1>
+// CHECK:           %[[VAL_5:.*]] = arith.constant dense<0.000000e+00> : vector<5xf32>
+// CHECK:           %[[VAL_6:.*]] = arith.constant dense<6> : vector<5xindex>
+// CHECK:           %[[VAL_7:.*]] = tensor.empty() : tensor<5xf32>
+// CHECK:           %[[VAL_8:.*]] = vector.transfer_read %[[VAL_1]]{{\[}}%[[VAL_2]]], %[[VAL_2]] {in_bounds = [true]} : tensor<5xindex>, vector<5xindex>
+// CHECK:           %[[VAL_9:.*]] = arith.muli %[[VAL_8]], %[[VAL_6]] : vector<5xindex>
+// CHECK:           %[[VAL_10:.*]] = arith.addi %[[VAL_9]], %[[VAL_3]] : vector<5xindex>
+// CHECK:           %[[VAL_11:.*]] = vector.gather %[[VAL_0]]{{\[}}%[[VAL_2]], %[[VAL_2]]] {{\[}}%[[VAL_10]]], %[[VAL_4]], %[[VAL_5]] : tensor<5x6xf32>, vector<5xindex>, vector<5xi1>, vector<5xf32> into vector<5xf32>
+// CHECK:           %[[VAL_12:.*]] = vector.transfer_write %[[VAL_11]], %[[VAL_7]]{{\[}}%[[VAL_2]]] {in_bounds = [true]} : vector<5xf32>, tensor<5xf32>
+// CHECK:           return %[[VAL_12]] : tensor<5xf32>
+// CHECK:         }
+
 transform.sequence failures(propagate) {
  ^bb1(%arg1: !pdl.operation):
    %0 = transform.structured.match ops{["linalg.generic"]} in %arg1 : (!pdl.operation) -> !pdl.operation
diff --git a/mlir/test/Dialect/Linalg/vectorize-convolution.mlir b/mlir/test/Dialect/Linalg/vectorize-convolution.mlir
index 91d822d804c05..db235fad7d2ee 100644
--- a/mlir/test/Dialect/Linalg/vectorize-convolution.mlir
+++ b/mlir/test/Dialect/Linalg/vectorize-convolution.mlir
@@ -461,6 +461,59 @@ func.func @conv1d_ncw_4x8x2_memref(%input: memref<4x3x6xf32>, %filter: memref<8x
 //      CHECK:   vector.transfer_write %[[RES]], %[[OUTPUT]][%[[C0]], %[[C0]], %[[C0]]]
 
 
+// -----
+
+func.func @conv1d_8_tensor(%input: tensor<11xf32>, %filter: tensor<4xf32>, %output: tensor<8xf32>) -> tensor<8xf32> {
+  %0 = linalg.conv_1d ins(%input, %filter : tensor<11xf32>, tensor<4xf32>)
+                     outs(%output : tensor<8xf32>) -> tensor<8xf32>
+  return %0 : tensor<8xf32>
+}
+
+//      CHECK: func @conv1d_8_tensor
+// CHECK-SAME: (%[[INPUT:.+]]: tensor<11xf32>, %[[FILTER:.+]]: tensor<4xf32>, %[[OUTPUT:.+]]: tensor<8xf32>)
+
+//  CHECK-DAG:   %[[C0:.+]] = arith.constant 0 : index
+//  CHECK-DAG:   %[[F0:.+]] = arith.constant 0.000000e+00 : f32
+
+/// Read the whole data in one shot.
+//  CHECK-DAG:   %[[V_INPUT_R:.+]] = vector.transfer_read %[[INPUT]][%[[C0]]], %[[F0]]
+//  CHECK-DAG:  %[[V_FILTER_R:.+]] = vector.transfer_read %[[FILTER]][%[[C0]]], %[[F0]]
+//  CHECK-DAG:  %[[V_OUTPUT_R:.+]] = vector.transfer_read %[[OUTPUT]][%[[C0]]], %[[F0]]
+
+//      CHECK:   %[[V_INPUT_0:.+]] = vector.extract_strided_slice %[[V_INPUT_R]]
+// CHECK-SAME:     {offsets = [0], sizes = [8], strides = [1]} : vector<11xf32> to vector<8xf32>
+//      CHECK:   %[[V_INPUT_1:.+]] = vector.extract_strided_slice %[[V_INPUT_R]]
+// CHECK-SAME:     {offsets = [1], sizes = [8], strides = [1]} : vector<11xf32> to vector<8xf32>
+//      CHECK:   %[[V_INPUT_2:.+]] = vector.extract_strided_slice %[[V_INPUT_R]]
+// CHECK-SAME:     {offsets = [2], sizes = [8], strides = [1]} : vector<11xf32> to vector<8xf32>
+//      CHECK:   %[[V_INPUT_3:.+]] = vector.extract_strided_slice %[[V_INPUT_R]]
+// CHECK-SAME:     {offsets = [3], sizes = [8], strides = [1]} : vector<11xf32> to vector<8xf32>
+
+//      CHECK:  %[[V_FILTER_0:.+]] = vector.extract %[[V_FILTER_R]][0] : vector<4xf32>
+//      CHECK:  %[[V_FILTER_1:.+]] = vector.extract %[[V_FILTER_R]][1] : vector<4xf32>
+//      CHECK:  %[[V_FILTER_2:.+]] = vector.extract %[[V_FILTER_R]][2] : vector<4xf32>
+//      CHECK:  %[[V_FILTER_3:.+]] = vector.extract %[[V_FILTER_R]][3] : vector<4xf32>
+
+/// w == 0, kw == 0
+//      CHECK:   %[[RES_0:.+]] = vector.outerproduct
+// CHECK-SAME:     %[[V_INPUT_0]], %[[V_FILTER_0]], %[[V_OUTPUT_R]] {kind = #vector.kind<add>}
+// CHECK-SAME:     : vector<8xf32>, f32
+/// w == 1, kw == 1
+//      CHECK:   %[[RES_1:.+]] = vector.outerproduct
+// CHECK-SAME:     %[[V_INPUT_1]], %[[V_FILTER_1]], %[[RES_0]] {kind = #vector.kind<add>}
+// CHECK-SAME:     : vector<8xf32>, f32
+/// w == 2, kw == 2
+//      CHECK:   %[[RES_2:.+]] = vector.outerproduct
+// CHECK-SAME:     %[[V_INPUT_2]], %[[V_FILTER_2]], %[[RES_1]] {kind = #vector.kind<add>}
+// CHECK-SAME:     : vector<8xf32>, f32
+/// w == 3, kw == 3
+//      CHECK:   %[[RES_3:.+]] = vector.outerproduct
+// CHECK-SAME:     %[[V_INPUT_3]], %[[V_FILTER_3]], %[[RES_2]] {kind = #vector.kind<add>}
+// CHECK-SAME:     : vector<8xf32>, f32
+
+// Write the result back in one shot.
+//      CHECK:   vector.transfer_write %[[RES_3]], %[[OUTPUT]][%[[C0]]]
+
 // -----
 
 func.func @depthwise_conv1d_nwc_wc_3x5x4xf32_memref(%input: memref<3x5x4xf32>, %filter: memref<2x4xf32>, %output: memref<3x2x4xf32>) {
diff --git a/mlir/test/Dialect/Math/polynomial-approximation.mlir b/mlir/test/Dialect/Math/polynomial-approximation.mlir
index 33ac11b29b9c6..4b490e4ea990c 100644
--- a/mlir/test/Dialect/Math/polynomial-approximation.mlir
+++ b/mlir/test/Dialect/Math/polynomial-approximation.mlir
@@ -593,3 +593,53 @@ func.func @atan2_scalar(%arg0: f16, %arg1: f16) -> f16 {
   %0 = math.atan2 %arg0, %arg1 : f16
   return %0 : f16
 }
+
+// CHECK-LABEL: @cbrt_vector
+// CHECK-SAME: %[[ARG0:.+]]: vector<4xf32>
+
+// CHECK: %[[TWO_INT:.+]] = arith.constant dense<2>
+// CHECK: %[[FOUR_INT:.+]] = arith.constant dense<4>
+// CHECK: %[[EIGHT_INT:.+]] = arith.constant dense<8>
+// CHECK: %[[MAGIC:.+]] = arith.constant dense<709965728>
+// CHECK: %[[THIRD_FP:.+]] = arith.constant dense<0.333333343> : vector<4xf32>
+// CHECK: %[[TWO_FP:.+]] = arith.constant dense<2.000000e+00> : vector<4xf32>
+// CHECK: %[[ZERO_FP:.+]] = arith.constant dense<0.000000e+00> : vector<4xf32>
+
+// CHECK: %[[ABS:.+]] = math.absf %[[ARG0]] : vector<4xf32>
+
+// Perform the initial approximation:
+// CHECK: %[[CAST:.+]] = arith.bitcast %[[ABS]] : vector<4xf32> to vector<4xi32>
+// CHECK: %[[SH_TWO:.+]] = arith.shrsi %[[CAST]], %[[TWO_INT]]
+// CHECK: %[[SH_FOUR:.+]] = arith.shrsi %[[CAST]], %[[FOUR_INT]]
+// CHECK: %[[APPROX0:.+]] = arith.addi %[[SH_TWO]], %[[SH_FOUR]]
+// CHECK: %[[SH_FOUR:.+]] = arith.shrsi %[[APPROX0]], %[[FOUR_INT]]
+// CHECK: %[[APPROX1:.+]] = arith.addi %[[APPROX0]], %[[SH_FOUR]]
+// CHECK: %[[SH_EIGHT:.+]] = arith.shrsi %[[APPROX1]], %[[EIGHT_INT]]
+// CHECK: %[[APPROX2:.+]] = arith.addi %[[APPROX1]], %[[SH_EIGHT]]
+// CHECK: %[[FIX:.+]] = arith.addi %[[APPROX2]], %[[MAGIC]]
+// CHECK: %[[BCAST:.+]] = arith.bitcast %[[FIX]]
+
+// First Newton Step:
+// CHECK: %[[SQR:.+]] = arith.mulf %[[BCAST]], %[[BCAST]]
+// CHECK: %[[DOUBLE:.+]] = arith.mulf %[[BCAST]], %[[TWO_FP]]
+// CHECK: %[[DIV:.+]] = arith.divf %[[ABS]], %[[SQR]]
+// CHECK: %[[ADD:.+]] = arith.addf %[[DOUBLE]], %[[DIV]]
+// CHECK: %[[APPROX3:.+]] = arith.mulf %[[ADD]], %[[THIRD_FP]]
+
+// Second Newton Step:
+// CHECK: %[[SQR:.+]] = arith.mulf %[[APPROX3]], %[[APPROX3]]
+// CHECK: %[[DOUBLE:.+]] = arith.mulf %[[APPROX3]], %[[TWO_FP]]
+// CHECK: %[[DIV:.+]] = arith.divf %[[ABS]], %[[SQR]]
+// CHECK: %[[ADD:.+]] = arith.addf %[[DOUBLE]], %[[DIV]]
+// CHECK: %[[APPROX4:.+]] = arith.mulf %[[ADD]], %[[THIRD_FP]]
+
+// Check for zero special case and copy the sign:
+// CHECK: %[[CMP:.+]] = arith.cmpf oeq, %[[ABS]], %[[ZERO_FP]]
+// CHECK: %[[SEL:.+]] = arith.select %[[CMP]], %[[ZERO_FP]], %[[APPROX4]]
+// CHECK: %[[SIGN:.+]] = math.copysign %[[SEL]], %[[ARG0]]
+// CHECK: return %[[SIGN]]
+
+func.func @cbrt_vector(%arg0: vector<4xf32>) -> vector<4xf32> {
+  %0 = "math.cbrt"(%arg0) : (vector<4xf32>) -> vector<4xf32>
+  func.return %0 : vector<4xf32>
+}
\ No newline at end of file
diff --git a/mlir/test/Dialect/MemRef/fold-memref-alias-ops.mlir b/mlir/test/Dialect/MemRef/fold-memref-alias-ops.mlir
index f0d5008f991aa..e0e8bd48d6f86 100644
--- a/mlir/test/Dialect/MemRef/fold-memref-alias-ops.mlir
+++ b/mlir/test/Dialect/MemRef/fold-memref-alias-ops.mlir
@@ -524,3 +524,54 @@ func.func @fold_store_keep_nontemporal(%arg0 : memref<12x32xf32>, %arg1 : index,
   memref.store %arg5, %0[%arg3, %arg4] {nontemporal=true}: memref<4x4xf32, strided<[64, 3], offset: ?>>
   return
 }
+
+// -----
+
+func.func @fold_gpu_subgroup_mma_load_matrix_1d(%src: memref<?xvector<4xf32>>, %offset: index, %i: index) -> !gpu.mma_matrix<16x16xf16, "COp"> {
+  %subview = memref.subview %src[%offset] [81920] [1] : memref<?xvector<4xf32>> to memref<81920xvector<4xf32>, strided<[1], offset: ?>>
+  %matrix = gpu.subgroup_mma_load_matrix %subview[%i] {leadDimension = 160 : index} : memref<81920xvector<4xf32>, strided<[1], offset: ?>> -> !gpu.mma_matrix<16x16xf16, "COp">
+  return %matrix: !gpu.mma_matrix<16x16xf16, "COp">
+}
+
+//  CHECK-DAG: #[[MAP:.+]] = affine_map<(d0)[s0] -> (d0 + s0)>
+//      CHECK: func.func @fold_gpu_subgroup_mma_load_matrix_1d
+// CHECK-SAME: (%[[SRC:.+]]: memref<?xvector<4xf32>>, %[[OFFSET:.+]]: index, %[[I:.+]]: index)
+//      CHECK:   %[[APPLY:.+]] = affine.apply #[[MAP]](%[[I]])[%[[OFFSET]]]
+//      CHECK:   %[[LOAD:.+]] = gpu.subgroup_mma_load_matrix %[[SRC]][%[[APPLY]]] {leadDimension = 160 : index} : memref<?xvector<4xf32>> -> !gpu.mma_matrix<16x16xf16, "COp">
+//      CHECK:   return %[[LOAD]]
+
+// -----
+
+func.func @fold_gpu_subgroup_mma_store_matrix_1d(%dst: memref<?xvector<4xf32>>, %offset: index, %i: index, %matrix: !gpu.mma_matrix<16x16xf16, "COp">) {
+  %subview = memref.subview %dst[%offset] [81920] [1] : memref<?xvector<4xf32>> to memref<81920xvector<4xf32>, strided<[1], offset: ?>>
+  gpu.subgroup_mma_store_matrix %matrix, %subview[%i] {leadDimension = 160 : index} : !gpu.mma_matrix<16x16xf16, "COp">, memref<81920xvector<4xf32>, strided<[1], offset: ?>>
+  return
+}
+
+//  CHECK-DAG: #[[MAP:.+]] = affine_map<(d0)[s0] -> (d0 + s0)>
+//      CHECK: func.func @fold_gpu_subgroup_mma_store_matrix_1d
+// CHECK-SAME: (%[[DST:.+]]: memref<?xvector<4xf32>>, %[[OFFSET:.+]]: index, %[[I0:.+]]: index, %[[VAL:.+]]: !gpu.mma_matrix<16x16xf16, "COp">)
+//      CHECK:   %[[APPLY:.+]] = affine.apply #[[MAP]](%[[I0]])[%[[OFFSET]]]
+//      CHECK:   gpu.subgroup_mma_store_matrix %[[VAL]], %[[DST]][%[[APPLY]]] {leadDimension = 160 : index} : !gpu.mma_matrix<16x16xf16, "COp">, memref<?xvector<4xf32>>
+
+// -----
+
+// CHECK-LABEL: func.func @fold_gpu_subgroup_mma_load_matrix_2d
+//  CHECK-SAME: %[[SRC:.+]]: memref<128x128xf32>
+func.func @fold_gpu_subgroup_mma_load_matrix_2d(%arg0 : memref<128x128xf32>, %arg1 : index, %arg2 : index, %arg3 : index, %arg4 : index) -> !gpu.mma_matrix<16x16xf16, "COp"> {
+  %subview = memref.subview %arg0[%arg1, %arg2][64, 32][2, 1] : memref<128x128xf32> to memref<64x32xf32, strided<[64, 1], offset: ?>>
+  // CHECK: gpu.subgroup_mma_load_matrix %[[SRC]][{{.+}}] {leadDimension = 32 : index} : memref<128x128xf32> -> !gpu.mma_matrix<16x16xf16, "COp">
+  %matrix = gpu.subgroup_mma_load_matrix %subview[%arg3, %arg4] {leadDimension = 32 : index} : memref<64x32xf32, strided<[64, 1], offset: ?>> -> !gpu.mma_matrix<16x16xf16, "COp">
+  return %matrix : !gpu.mma_matrix<16x16xf16, "COp">
+}
+
+// -----
+
+// CHECK-LABEL: func.func @fold_gpu_subgroup_mma_load_matrix_2d
+//  CHECK-SAME: %[[DST:.+]]: memref<128x128xf32>
+func.func @fold_gpu_subgroup_mma_load_matrix_2d(%arg0 : memref<128x128xf32>, %arg1 : index, %arg2 : index, %arg3 : index, %arg4 : index, %matrix: !gpu.mma_matrix<16x16xf16, "COp">) {
+  %subview = memref.subview %arg0[%arg1, %arg2][64, 32][2, 1] : memref<128x128xf32> to memref<64x32xf32, strided<[64, 1], offset: ?>>
+  // CHECK: gpu.subgroup_mma_store_matrix %{{.+}}, %[[DST]][{{.+}}] {leadDimension = 32 : index} : !gpu.mma_matrix<16x16xf16, "COp">, memref<128x128xf32>
+  gpu.subgroup_mma_store_matrix %matrix, %subview[%arg3, %arg4] {leadDimension = 32 : index} :  !gpu.mma_matrix<16x16xf16, "COp">, memref<64x32xf32, strided<[64, 1], offset: ?>>
+  return
+}
diff --git a/mlir/test/Dialect/NVGPU/invalid.mlir b/mlir/test/Dialect/NVGPU/invalid.mlir
index 57f91b334359e..02ee7885ed927 100644
--- a/mlir/test/Dialect/NVGPU/invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/invalid.mlir
@@ -130,7 +130,6 @@ func.func @async_cp_memory_space(%dst : memref<16xf32>, %src : memref<16xf32>, %
   nvgpu.device_async_copy %src[%i], %dst[%i], 16 : memref<16xf32> to memref<16xf32>
   return
 }
-
 // -----
 
 func.func @async_cp_memref_type(%dst : memref<16xi32, 3>, %src : memref<16xf32>, %i : index) -> () {
@@ -138,7 +137,6 @@ func.func @async_cp_memref_type(%dst : memref<16xi32, 3>, %src : memref<16xf32>,
   nvgpu.device_async_copy %src[%i], %dst[%i], 16 : memref<16xf32> to memref<16xi32, 3>
   return
 }
-
 // -----
 
 func.func @async_cp_num_src_indices(%dst : memref<16xf32, 3>, %src : memref<16x16xf32>, %i : index) -> () {
@@ -146,7 +144,6 @@ func.func @async_cp_num_src_indices(%dst : memref<16xf32, 3>, %src : memref<16x1
   nvgpu.device_async_copy %src[%i], %dst[%i], 16 : memref<16x16xf32> to memref<16xf32, 3>
   return
 }
-
 // -----
 
 func.func @async_cp_num_dst_indices(%dst : memref<16x16xf32, 3>, %src : memref<16xf32>, %i : index) -> () {
@@ -154,7 +151,6 @@ func.func @async_cp_num_dst_indices(%dst : memref<16x16xf32, 3>, %src : memref<1
   nvgpu.device_async_copy %src[%i], %dst[%i], 16 : memref<16xf32> to memref<16x16xf32, 3>
   return
 }
-
 // -----
 
 func.func @async_cp_num_src_stride(
@@ -166,7 +162,6 @@ func.func @async_cp_num_src_stride(
     memref<200x100xf32, affine_map<(d0, d1) -> (200*d0 + 2*d1)>> to memref<200x100xf32, 3>
   return
 }
-
 // -----
 
 func.func @async_cp_num_dst_stride(
@@ -178,3 +173,15 @@ func.func @async_cp_num_dst_stride(
     memref<200x100xf32> to memref<200x100xf32, affine_map<(d0, d1) -> (200*d0 + 2*d1)>, 3>
   return
 }
+// -----
+
+// 42 is never the answer!
+func.func @mma_sp_sync_f16_16816(%arg0: vector<2x2xf16>,
+                                 %arg1: vector<2x2xf16>,
+                                 %arg2: vector<2x2xf16>,
+                                 %arg3: vector<2xi16>) -> vector<2x2xf16> {
+  // expected-error @+1 {{'nvgpu.mma.sp.sync' op sparsity selector should be 0 or 1}}
+  %d = nvgpu.mma.sp.sync(%arg0, %arg1, %arg2) metadata(%arg3) {mmaShape = [16, 8, 16], sparsitySelector = 42 : i32} :
+       (vector<2x2xf16>, vector<2x2xf16>, vector<2x2xf16>) -> vector<2x2xf16>
+  return %d : vector<2x2xf16>
+}
diff --git a/mlir/test/Dialect/SCF/loop-pipelining.mlir b/mlir/test/Dialect/SCF/loop-pipelining.mlir
index 68b513362a250..0309287e409c1 100644
--- a/mlir/test/Dialect/SCF/loop-pipelining.mlir
+++ b/mlir/test/Dialect/SCF/loop-pipelining.mlir
@@ -627,3 +627,47 @@ func.func @pipeline_op_with_region(%A: memref<?xf32>, %B: memref<?xf32>, %result
   }  { __test_pipelining_loop__ }
   return
 }
+
+// -----
+
+// CHECK-LABEL: @backedge_mix_order
+//  CHECK-SAME:   (%[[A:.*]]: memref<?xf32>) -> f32 {
+//   CHECK-DAG:   %[[C0:.*]] = arith.constant 0 : index
+//   CHECK-DAG:   %[[C1:.*]] = arith.constant 1 : index
+//   CHECK-DAG:   %[[C3:.*]] = arith.constant 3 : index
+//   CHECK-DAG:   %[[CSTF:.*]] = arith.constant 2.000000e+00 : f32
+// Prologue:
+//       CHECK:   %[[L0:.*]] = memref.load %[[A]][%[[C0]]] : memref<?xf32>
+//  CHECK-NEXT:   %[[L1:.*]] = memref.load %[[A]][%[[C1]]] : memref<?xf32>
+// Kernel:
+//  CHECK-NEXT:   %[[R:.*]]:3 = scf.for %[[IV:.*]] = %[[C0]] to %[[C3]]
+//  CHECK-SAME:     step %[[C1]] iter_args(%[[C:.*]] = %[[CSTF]],
+//  CHECK-SAME:     %[[ARG1:.*]] = %[[L0]], %[[ARG2:.*]] = %[[L1]]) -> (f32, f32, f32) {
+//  CHECK-NEXT:     %[[IV2:.*]] = arith.addi %[[IV]], %[[C1]] : index
+//  CHECK-NEXT:     %[[L2:.*]] = memref.load %[[A]][%[[IV2]]] : memref<?xf32>
+//  CHECK-NEXT:     %[[MUL0:.*]] = arith.mulf %[[C]], %[[ARG1]] : f32
+//  CHECK-NEXT:     %[[IV3:.*]] = arith.addi %[[IV]], %[[C1]] : index
+//  CHECK-NEXT:     %[[IV4:.*]] = arith.addi %[[IV3]], %[[C1]] : index
+//  CHECK-NEXT:     %[[L3:.*]] = memref.load %[[A]][%[[IV4]]] : memref<?xf32>
+//  CHECK-NEXT:     %[[MUL1:.*]] = arith.mulf %[[ARG2]], %[[MUL0]] : f32
+//  CHECK-NEXT:     scf.yield %[[MUL1]], %[[L2]], %[[L3]] : f32, f32, f32
+//  CHECK-NEXT:   }
+// Epilogue:
+//  CHECK-NEXT:   %[[MUL1:.*]] = arith.mulf %[[R]]#0, %[[R]]#1 : f32
+//  CHECK-NEXT:   %[[MUL2:.*]] = arith.mulf %[[R]]#2, %[[MUL1]] : f32
+//  CHECK-NEXT:   return %[[MUL2]] : f32
+func.func @backedge_mix_order(%A: memref<?xf32>) -> f32 {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %cf = arith.constant 2.0 : f32
+  %r = scf.for %i0 = %c0 to %c4 step %c1 iter_args(%arg0 = %cf) -> (f32) {
+    %A_elem = memref.load %A[%i0] { __test_pipelining_stage__ = 0, __test_pipelining_op_order__ = 0 } : memref<?xf32>
+    %A2_elem = arith.mulf %arg0, %A_elem { __test_pipelining_stage__ = 1, __test_pipelining_op_order__ = 1 } : f32
+    %i1 = arith.addi %i0, %c1 { __test_pipelining_stage__ = 0, __test_pipelining_op_order__ = 2 } : index
+    %A1_elem = memref.load %A[%i1] { __test_pipelining_stage__ = 0, __test_pipelining_op_order__ = 3 } : memref<?xf32>
+    %A3_elem = arith.mulf %A1_elem, %A2_elem { __test_pipelining_stage__ = 1, __test_pipelining_op_order__ = 4 } : f32
+    scf.yield %A3_elem : f32
+  }  { __test_pipelining_loop__ }
+  return %r : f32
+}
\ No newline at end of file
diff --git a/mlir/test/Dialect/SCF/ops.mlir b/mlir/test/Dialect/SCF/ops.mlir
index 174ae539fe56c..46d175d6870ce 100644
--- a/mlir/test/Dialect/SCF/ops.mlir
+++ b/mlir/test/Dialect/SCF/ops.mlir
@@ -37,6 +37,18 @@ func.func @std_for_i32(%arg0 : i32, %arg1 : i32, %arg2 : i32) {
 //  CHECK-NEXT:   scf.for %{{.*}} = %{{.*}} to %{{.*}} step %{{.*}} : i32 {
 //  CHECK-NEXT:     scf.for %{{.*}} = %{{.*}} to %{{.*}} step %{{.*}} : i32 {
 
+func.func @scf_for_i64_iter(%arg1: i64, %arg2: i64) {
+  %c1_i64 = arith.constant 1 : i64
+  %c0_i64 = arith.constant 0 : i64
+  %0 = scf.for %arg3 = %arg1 to %arg2 step %c1_i64 iter_args(%arg4 = %c0_i64) -> (i64) : i64 {
+    %1 = arith.addi %arg4, %arg3 : i64
+    scf.yield %1 : i64
+  }
+  return
+}
+// CHECK-LABEL: scf_for_i64_iter
+// CHECK: scf.for %{{.*}} = %{{.*}} to %{{.*}} step %{{.*}} iter_args(%{{.*}} = %{{.*}} -> (i64) : i64 {
+
 func.func @std_if(%arg0: i1, %arg1: f32) {
   scf.if %arg0 {
     %0 = arith.addf %arg1, %arg1 : f32
diff --git a/mlir/test/Dialect/SPIRV/Transforms/unify-aliased-resource.mlir b/mlir/test/Dialect/SPIRV/Transforms/unify-aliased-resource.mlir
index 1d532f3f13812..8801fdb0e1653 100644
--- a/mlir/test/Dialect/SPIRV/Transforms/unify-aliased-resource.mlir
+++ b/mlir/test/Dialect/SPIRV/Transforms/unify-aliased-resource.mlir
@@ -32,6 +32,33 @@ spirv.module Logical GLSL450 {
 
 // -----
 
+spirv.module Logical GLSL450 {
+  spirv.GlobalVariable @var01s bind(0, 1) {aliased} : !spirv.ptr<!spirv.struct<(!spirv.rtarray<f32, stride=4> [0])>, StorageBuffer>
+  spirv.GlobalVariable @var01v bind(0, 1) {aliased} : !spirv.ptr<!spirv.struct<(!spirv.rtarray<vector<4xf32>, stride=16> [0])>, StorageBuffer>
+
+  spirv.func @load_store_scalar_64bit(%index: i64) -> f32 "None" {
+    %c0 = spirv.Constant 0 : i64
+    %addr = spirv.mlir.addressof @var01s : !spirv.ptr<!spirv.struct<(!spirv.rtarray<f32, stride=4> [0])>, StorageBuffer>
+    %ac = spirv.AccessChain %addr[%c0, %index] : !spirv.ptr<!spirv.struct<(!spirv.rtarray<f32, stride=4> [0])>, StorageBuffer>, i64, i64
+    %value = spirv.Load "StorageBuffer" %ac : f32
+    spirv.Store "StorageBuffer" %ac, %value : f32
+    spirv.ReturnValue %value : f32
+  }
+}
+
+// CHECK-LABEL: spirv.module
+
+// CHECK-NOT: @var01s
+//     CHECK: spirv.GlobalVariable @var01v bind(0, 1) : !spirv.ptr<!spirv.struct<(!spirv.rtarray<vector<4xf32>, stride=16> [0])>, StorageBuffer>
+// CHECK-NOT: @var01s
+
+//     CHECK: spirv.func @load_store_scalar_64bit(%[[INDEX:.+]]: i64)
+// CHECK-DAG:   %[[C4:.+]] = spirv.Constant 4 : i64
+//     CHECK:   spirv.SDiv %[[INDEX]], %[[C4]] : i64
+//     CHECK:   spirv.SMod %[[INDEX]], %[[C4]] : i64
+
+// -----
+
 spirv.module Logical GLSL450 {
   spirv.GlobalVariable @var01s bind(0, 1) {aliased} : !spirv.ptr<!spirv.struct<(!spirv.rtarray<f32, stride=4> [0])>, StorageBuffer>
   spirv.GlobalVariable @var01v bind(0, 1) {aliased} : !spirv.ptr<!spirv.struct<(!spirv.rtarray<vector<4xf32>, stride=16> [0])>, StorageBuffer>
diff --git a/mlir/test/Dialect/SparseTensor/buffer_rewriting.mlir b/mlir/test/Dialect/SparseTensor/buffer_rewriting.mlir
index dbe0c972e6614..b0c1d5469cfd2 100644
--- a/mlir/test/Dialect/SparseTensor/buffer_rewriting.mlir
+++ b/mlir/test/Dialect/SparseTensor/buffer_rewriting.mlir
@@ -75,54 +75,302 @@ func.func @sparse_push_back_inbound(%arg0: index, %arg1: memref<?xf64>, %arg2: f
 
 // -----
 
-// CHECK-LABEL:   func.func private @_sparse_less_than_1_i8(
-// CHECK-SAME:                                              %[[I:arg0]]: index,
-// CHECK-SAME:                                              %[[J:.*]]: index,
-// CHECK-SAME:                                              %[[X0:.*]]: memref<?xi8>) -> i1 {
-// CHECK:           %[[VI:.*]] = memref.load %[[X0]]{{\[}}%[[I]]]
-// CHECK:           %[[VJ:.*]] = memref.load %[[X0]]{{\[}}%[[J]]]
-// CHECK:           %[[C:.*]] = arith.cmpi ult, %[[VI]], %[[VJ]]
-// CHECK:           return %[[C]]
-// CHECK:         }
-
-// CHECK-LABEL:   func.func private @_sparse_compare_eq_1_i8(
-// CHECK-SAME:                                               %[[I:arg0]]: index,
-// CHECK-SAME:                                               %[[J:.*]]: index,
-// CHECK-SAME:                                               %[[X0:.*]]: memref<?xi8>) -> i1 {
-// CHECK:           %[[VI:.*]] = memref.load %[[X0]]{{\[}}%[[I]]]
-// CHECK:           %[[VJ:.*]] = memref.load %[[X0]]{{\[}}%[[J]]]
-// CHECK:           %[[C:.*]] = arith.cmpi eq, %[[VI]], %[[VJ]]
-// CHECK:           return %[[C]]
-// CHECK:         }
-
 // CHECK-LABEL:   func.func private @_sparse_partition_1_i8_f32_index(
-// CHECK-SAME:                                                        %[[L:arg0]]: index,
-// CHECK-SAME:                                                        %[[H:.*]]: index,
-// CHECK-SAME:                                                        %[[X0:.*]]: memref<?xi8>,
-// CHECK-SAME:                                                        %[[Y0:.*]]: memref<?xf32>,
-// CHECK-SAME:                                                        %[[Y1:.*]]: memref<?xindex>) -> index {
-// CHECK:           %[[C1:.*]] = arith.constant 1
-// CHECK:           %[[VAL_6:.*]] = arith.constant -
-// CHECK:           %[[SUM:.*]] = arith.addi %[[L]], %[[H]]
-// CHECK:           %[[P:.*]] = arith.shrui %[[SUM]], %[[C1]]
-// CHECK:           %[[J:.*]] = arith.subi %[[H]], %[[C1]]
-// CHECK:           %[[W:.*]]:3 = scf.while (%[[Ib:.*]] = %[[L]], %[[Jb:.*]] = %[[J]], %[[pb:.*]] = %[[P]]) : (index, index, index) -> (index, index, index) {
-// CHECK:             %[[Cn:.*]] = arith.cmpi ult, %[[Ib]], %[[Jb]]
-// CHECK:             scf.condition(%[[Cn]]) %[[Ib]], %[[Jb]], %[[pb]]
+// CHECK-SAME:                                                        %[[VAL_0:.*0]]: index,
+// CHECK-SAME:                                                        %[[VAL_1:.*1]]: index,
+// CHECK-SAME:                                                        %[[VAL_2:.*2]]: memref<?xi8>,
+// CHECK-SAME:                                                        %[[VAL_3:.*3]]: memref<?xf32>,
+// CHECK-SAME:                                                        %[[VAL_4:.*4]]: memref<?xindex>) -> index {
+// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1
+// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 1000
+// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant -1
+// CHECK:           %[[VAL_8:.*]] = arith.addi %[[VAL_0]], %[[VAL_1]]
+// CHECK:           %[[VAL_9:.*]] = arith.shrui %[[VAL_8]], %[[VAL_5]]
+// CHECK:           %[[VAL_10:.*]] = arith.subi %[[VAL_1]], %[[VAL_5]]
+// CHECK:           %[[VAL_11:.*]] = arith.subi %[[VAL_1]], %[[VAL_0]]
+// CHECK:           %[[VAL_12:.*]] = arith.cmpi ult, %[[VAL_11]], %[[VAL_6]]
+// CHECK:           scf.if %[[VAL_12]] {
+// CHECK:             %[[VAL_13:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:             %[[VAL_14:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_0]]]
+// CHECK:             %[[VAL_15:.*]] = arith.cmpi ult, %[[VAL_13]], %[[VAL_14]]
+// CHECK:             scf.if %[[VAL_15]] {
+// CHECK:               %[[VAL_16:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_17:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_0]]]
+// CHECK:               memref.store %[[VAL_17]], %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:               memref.store %[[VAL_16]], %[[VAL_2]]{{\[}}%[[VAL_0]]]
+// CHECK:               %[[VAL_18:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_19:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_0]]]
+// CHECK:               memref.store %[[VAL_19]], %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:               memref.store %[[VAL_18]], %[[VAL_3]]{{\[}}%[[VAL_0]]]
+// CHECK:               %[[VAL_20:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_21:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_0]]]
+// CHECK:               memref.store %[[VAL_21]], %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:               memref.store %[[VAL_20]], %[[VAL_4]]{{\[}}%[[VAL_0]]]
+// CHECK:             }
+// CHECK:             %[[VAL_22:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_10]]]
+// CHECK:             %[[VAL_23:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:             %[[VAL_24:.*]] = arith.cmpi ult, %[[VAL_22]], %[[VAL_23]]
+// CHECK:             scf.if %[[VAL_24]] {
+// CHECK:               %[[VAL_25:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_10]]]
+// CHECK:               %[[VAL_26:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:               memref.store %[[VAL_26]], %[[VAL_2]]{{\[}}%[[VAL_10]]]
+// CHECK:               memref.store %[[VAL_25]], %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_27:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_10]]]
+// CHECK:               %[[VAL_28:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:               memref.store %[[VAL_28]], %[[VAL_3]]{{\[}}%[[VAL_10]]]
+// CHECK:               memref.store %[[VAL_27]], %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_29:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_10]]]
+// CHECK:               %[[VAL_30:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:               memref.store %[[VAL_30]], %[[VAL_4]]{{\[}}%[[VAL_10]]]
+// CHECK:               memref.store %[[VAL_29]], %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_31:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_32:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_0]]]
+// CHECK:               %[[VAL_33:.*]] = arith.cmpi ult, %[[VAL_31]], %[[VAL_32]]
+// CHECK:               scf.if %[[VAL_33]] {
+// CHECK:                 %[[VAL_34:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                 %[[VAL_35:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_0]]]
+// CHECK:                 memref.store %[[VAL_35]], %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                 memref.store %[[VAL_34]], %[[VAL_2]]{{\[}}%[[VAL_0]]]
+// CHECK:                 %[[VAL_36:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:                 %[[VAL_37:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_0]]]
+// CHECK:                 memref.store %[[VAL_37]], %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:                 memref.store %[[VAL_36]], %[[VAL_3]]{{\[}}%[[VAL_0]]]
+// CHECK:                 %[[VAL_38:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:                 %[[VAL_39:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_0]]]
+// CHECK:                 memref.store %[[VAL_39]], %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:                 memref.store %[[VAL_38]], %[[VAL_4]]{{\[}}%[[VAL_0]]]
+// CHECK:               }
+// CHECK:             }
+// CHECK:           } else {
+// CHECK:             %[[VAL_40:.*]] = arith.addi %[[VAL_9]], %[[VAL_1]]
+// CHECK:             %[[VAL_41:.*]] = arith.shrui %[[VAL_40]], %[[VAL_5]]
+// CHECK:             %[[VAL_42:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:             %[[VAL_43:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_0]]]
+// CHECK:             %[[VAL_44:.*]] = arith.cmpi ult, %[[VAL_42]], %[[VAL_43]]
+// CHECK:             scf.if %[[VAL_44]] {
+// CHECK:               %[[VAL_45:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_46:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_0]]]
+// CHECK:               memref.store %[[VAL_46]], %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:               memref.store %[[VAL_45]], %[[VAL_2]]{{\[}}%[[VAL_0]]]
+// CHECK:               %[[VAL_47:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_48:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_0]]]
+// CHECK:               memref.store %[[VAL_48]], %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:               memref.store %[[VAL_47]], %[[VAL_3]]{{\[}}%[[VAL_0]]]
+// CHECK:               %[[VAL_49:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_50:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_0]]]
+// CHECK:               memref.store %[[VAL_50]], %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:               memref.store %[[VAL_49]], %[[VAL_4]]{{\[}}%[[VAL_0]]]
+// CHECK:             }
+// CHECK:             %[[VAL_51:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:             %[[VAL_52:.*]] = arith.cmpi ult, %[[VAL_51]], %[[VAL_51]]
+// CHECK:             scf.if %[[VAL_52]] {
+// CHECK:               %[[VAL_53:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:               memref.store %[[VAL_53]], %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:               memref.store %[[VAL_53]], %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_54:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:               memref.store %[[VAL_54]], %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:               memref.store %[[VAL_54]], %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_55:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:               memref.store %[[VAL_55]], %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:               memref.store %[[VAL_55]], %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_56:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_57:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_0]]]
+// CHECK:               %[[VAL_58:.*]] = arith.cmpi ult, %[[VAL_56]], %[[VAL_57]]
+// CHECK:               scf.if %[[VAL_58]] {
+// CHECK:                 %[[VAL_59:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                 %[[VAL_60:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_0]]]
+// CHECK:                 memref.store %[[VAL_60]], %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                 memref.store %[[VAL_59]], %[[VAL_2]]{{\[}}%[[VAL_0]]]
+// CHECK:                 %[[VAL_61:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:                 %[[VAL_62:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_0]]]
+// CHECK:                 memref.store %[[VAL_62]], %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:                 memref.store %[[VAL_61]], %[[VAL_3]]{{\[}}%[[VAL_0]]]
+// CHECK:                 %[[VAL_63:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:                 %[[VAL_64:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_0]]]
+// CHECK:                 memref.store %[[VAL_64]], %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:                 memref.store %[[VAL_63]], %[[VAL_4]]{{\[}}%[[VAL_0]]]
+// CHECK:               }
+// CHECK:             }
+// CHECK:             %[[VAL_65:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_41]]]
+// CHECK:             %[[VAL_66:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:             %[[VAL_67:.*]] = arith.cmpi ult, %[[VAL_65]], %[[VAL_66]]
+// CHECK:             scf.if %[[VAL_67]] {
+// CHECK:               %[[VAL_68:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_41]]]
+// CHECK:               %[[VAL_69:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:               memref.store %[[VAL_69]], %[[VAL_2]]{{\[}}%[[VAL_41]]]
+// CHECK:               memref.store %[[VAL_68]], %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_70:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_41]]]
+// CHECK:               %[[VAL_71:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:               memref.store %[[VAL_71]], %[[VAL_3]]{{\[}}%[[VAL_41]]]
+// CHECK:               memref.store %[[VAL_70]], %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_72:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_41]]]
+// CHECK:               %[[VAL_73:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:               memref.store %[[VAL_73]], %[[VAL_4]]{{\[}}%[[VAL_41]]]
+// CHECK:               memref.store %[[VAL_72]], %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_74:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_75:.*]] = arith.cmpi ult, %[[VAL_74]], %[[VAL_74]]
+// CHECK:               scf.if %[[VAL_75]] {
+// CHECK:                 %[[VAL_76:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                 memref.store %[[VAL_76]], %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                 memref.store %[[VAL_76]], %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                 %[[VAL_77:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:                 memref.store %[[VAL_77]], %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:                 memref.store %[[VAL_77]], %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:                 %[[VAL_78:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:                 memref.store %[[VAL_78]], %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:                 memref.store %[[VAL_78]], %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:                 %[[VAL_79:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                 %[[VAL_80:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_0]]]
+// CHECK:                 %[[VAL_81:.*]] = arith.cmpi ult, %[[VAL_79]], %[[VAL_80]]
+// CHECK:                 scf.if %[[VAL_81]] {
+// CHECK:                   %[[VAL_82:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                   %[[VAL_83:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_0]]]
+// CHECK:                   memref.store %[[VAL_83]], %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                   memref.store %[[VAL_82]], %[[VAL_2]]{{\[}}%[[VAL_0]]]
+// CHECK:                   %[[VAL_84:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:                   %[[VAL_85:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_0]]]
+// CHECK:                   memref.store %[[VAL_85]], %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:                   memref.store %[[VAL_84]], %[[VAL_3]]{{\[}}%[[VAL_0]]]
+// CHECK:                   %[[VAL_86:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:                   %[[VAL_87:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_0]]]
+// CHECK:                   memref.store %[[VAL_87]], %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:                   memref.store %[[VAL_86]], %[[VAL_4]]{{\[}}%[[VAL_0]]]
+// CHECK:                 }
+// CHECK:               }
+// CHECK:             }
+// CHECK:             %[[VAL_88:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_10]]]
+// CHECK:             %[[VAL_89:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_41]]]
+// CHECK:             %[[VAL_90:.*]] = arith.cmpi ult, %[[VAL_88]], %[[VAL_89]]
+// CHECK:             scf.if %[[VAL_90]] {
+// CHECK:               %[[VAL_91:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_10]]]
+// CHECK:               %[[VAL_92:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_41]]]
+// CHECK:               memref.store %[[VAL_92]], %[[VAL_2]]{{\[}}%[[VAL_10]]]
+// CHECK:               memref.store %[[VAL_91]], %[[VAL_2]]{{\[}}%[[VAL_41]]]
+// CHECK:               %[[VAL_93:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_10]]]
+// CHECK:               %[[VAL_94:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_41]]]
+// CHECK:               memref.store %[[VAL_94]], %[[VAL_3]]{{\[}}%[[VAL_10]]]
+// CHECK:               memref.store %[[VAL_93]], %[[VAL_3]]{{\[}}%[[VAL_41]]]
+// CHECK:               %[[VAL_95:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_10]]]
+// CHECK:               %[[VAL_96:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_41]]]
+// CHECK:               memref.store %[[VAL_96]], %[[VAL_4]]{{\[}}%[[VAL_10]]]
+// CHECK:               memref.store %[[VAL_95]], %[[VAL_4]]{{\[}}%[[VAL_41]]]
+// CHECK:               %[[VAL_97:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_41]]]
+// CHECK:               %[[VAL_98:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:               %[[VAL_99:.*]] = arith.cmpi ult, %[[VAL_97]], %[[VAL_98]]
+// CHECK:               scf.if %[[VAL_99]] {
+// CHECK:                 %[[VAL_100:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_41]]]
+// CHECK:                 %[[VAL_101:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                 memref.store %[[VAL_101]], %[[VAL_2]]{{\[}}%[[VAL_41]]]
+// CHECK:                 memref.store %[[VAL_100]], %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                 %[[VAL_102:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_41]]]
+// CHECK:                 %[[VAL_103:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:                 memref.store %[[VAL_103]], %[[VAL_3]]{{\[}}%[[VAL_41]]]
+// CHECK:                 memref.store %[[VAL_102]], %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:                 %[[VAL_104:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_41]]]
+// CHECK:                 %[[VAL_105:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:                 memref.store %[[VAL_105]], %[[VAL_4]]{{\[}}%[[VAL_41]]]
+// CHECK:                 memref.store %[[VAL_104]], %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:                 %[[VAL_106:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                 %[[VAL_107:.*]] = arith.cmpi ult, %[[VAL_106]], %[[VAL_106]]
+// CHECK:                 scf.if %[[VAL_107]] {
+// CHECK:                   %[[VAL_108:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                   memref.store %[[VAL_108]], %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                   memref.store %[[VAL_108]], %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                   %[[VAL_109:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:                   memref.store %[[VAL_109]], %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:                   memref.store %[[VAL_109]], %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:                   %[[VAL_110:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:                   memref.store %[[VAL_110]], %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:                   memref.store %[[VAL_110]], %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:                   %[[VAL_111:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                   %[[VAL_112:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_0]]]
+// CHECK:                   %[[VAL_113:.*]] = arith.cmpi ult, %[[VAL_111]], %[[VAL_112]]
+// CHECK:                   scf.if %[[VAL_113]] {
+// CHECK:                     %[[VAL_114:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                     %[[VAL_115:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_0]]]
+// CHECK:                     memref.store %[[VAL_115]], %[[VAL_2]]{{\[}}%[[VAL_9]]]
+// CHECK:                     memref.store %[[VAL_114]], %[[VAL_2]]{{\[}}%[[VAL_0]]]
+// CHECK:                     %[[VAL_116:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:                     %[[VAL_117:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_0]]]
+// CHECK:                     memref.store %[[VAL_117]], %[[VAL_3]]{{\[}}%[[VAL_9]]]
+// CHECK:                     memref.store %[[VAL_116]], %[[VAL_3]]{{\[}}%[[VAL_0]]]
+// CHECK:                     %[[VAL_118:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:                     %[[VAL_119:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_0]]]
+// CHECK:                     memref.store %[[VAL_119]], %[[VAL_4]]{{\[}}%[[VAL_9]]]
+// CHECK:                     memref.store %[[VAL_118]], %[[VAL_4]]{{\[}}%[[VAL_0]]]
+// CHECK:                   }
+// CHECK:                 }
+// CHECK:               }
+// CHECK:             }
+// CHECK:           }
+// CHECK:           %[[VAL_120:.*]]:3 = scf.while (%[[VAL_121:.*]] = %[[VAL_0]], %[[VAL_122:.*]] = %[[VAL_10]], %[[VAL_123:.*]] = %[[VAL_9]])
+// CHECK:             %[[VAL_124:.*]] = arith.cmpi ult, %[[VAL_121]], %[[VAL_122]]
+// CHECK:             scf.condition(%[[VAL_124]]) %[[VAL_121]], %[[VAL_122]], %[[VAL_123]]
 // CHECK:           } do {
-// CHECK:           ^bb0(%[[Ia:.*]]: index, %[[Ja:.*]]: index, %[[Pa:.*]]: index):
-// CHECK:             %[[I2:.*]] = scf.while
-// CHECK:             %[[Ieq:.*]] = func.call @_sparse_compare_eq_1_i8(%[[I2:.*]], %[[Pa]], %[[X0]])
-// CHECK:             %[[J2:.*]] = scf.while
-// CHECK:             %[[Jeq:.*]] = func.call @_sparse_compare_eq_1_i8(%[[J2:.*]], %[[Pa]], %[[X0]])
-// CHECK:             %[[Cn2:.*]] = arith.cmpi ult, %[[I2]], %[[J2]]
-// CHECK:             %[[If:.*]]:3 = scf.if %[[Cn2]] -> (index, index, index) {
+// CHECK:           ^bb0(%[[VAL_125:.*]]: index, %[[VAL_126:.*]]: index, %[[VAL_127:.*]]: index)
+// CHECK:             %[[VAL_128:.*]] = scf.while (%[[VAL_129:.*]] = %[[VAL_125]])
+// CHECK:               %[[VAL_130:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_129]]]
+// CHECK:               %[[VAL_131:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_127]]]
+// CHECK:               %[[VAL_132:.*]] = arith.cmpi ult, %[[VAL_130]], %[[VAL_131]]
+// CHECK:               scf.condition(%[[VAL_132]]) %[[VAL_129]]
+// CHECK:             } do {
+// CHECK:             ^bb0(%[[VAL_133:.*]]: index):
+// CHECK:               %[[VAL_134:.*]] = arith.addi %[[VAL_133]], %[[VAL_5]]
+// CHECK:               scf.yield %[[VAL_134]]
+// CHECK:             }
+// CHECK:             %[[VAL_135:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_136:.*]]]
+// CHECK:             %[[VAL_137:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_127]]]
+// CHECK:             %[[VAL_138:.*]] = arith.cmpi eq, %[[VAL_135]], %[[VAL_137]]
+// CHECK:             %[[VAL_139:.*]] = scf.while (%[[VAL_140:.*]] = %[[VAL_126]])
+// CHECK:               %[[VAL_141:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_127]]]
+// CHECK:               %[[VAL_142:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_140]]]
+// CHECK:               %[[VAL_143:.*]] = arith.cmpi ult, %[[VAL_141]], %[[VAL_142]]
+// CHECK:               scf.condition(%[[VAL_143]]) %[[VAL_140]]
+// CHECK:             } do {
+// CHECK:             ^bb0(%[[VAL_144:.*]]: index):
+// CHECK:               %[[VAL_145:.*]] = arith.addi %[[VAL_144]], %[[VAL_7]]
+// CHECK:               scf.yield %[[VAL_145]]
+// CHECK:             }
+// CHECK:             %[[VAL_146:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_147:.*]]]
+// CHECK:             %[[VAL_148:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_127]]]
+// CHECK:             %[[VAL_149:.*]] = arith.cmpi eq, %[[VAL_146]], %[[VAL_148]]
+// CHECK:             %[[VAL_150:.*]] = arith.cmpi ult, %[[VAL_136]], %[[VAL_147]]
+// CHECK:             %[[VAL_151:.*]]:3 = scf.if %[[VAL_150]]
+// CHECK:               %[[VAL_152:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_136]]]
+// CHECK:               %[[VAL_153:.*]] = memref.load %[[VAL_2]]{{\[}}%[[VAL_147]]]
+// CHECK:               memref.store %[[VAL_153]], %[[VAL_2]]{{\[}}%[[VAL_136]]]
+// CHECK:               memref.store %[[VAL_152]], %[[VAL_2]]{{\[}}%[[VAL_147]]]
+// CHECK:               %[[VAL_154:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_136]]]
+// CHECK:               %[[VAL_155:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_147]]]
+// CHECK:               memref.store %[[VAL_155]], %[[VAL_3]]{{\[}}%[[VAL_136]]]
+// CHECK:               memref.store %[[VAL_154]], %[[VAL_3]]{{\[}}%[[VAL_147]]]
+// CHECK:               %[[VAL_156:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_136]]]
+// CHECK:               %[[VAL_157:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_147]]]
+// CHECK:               memref.store %[[VAL_157]], %[[VAL_4]]{{\[}}%[[VAL_136]]]
+// CHECK:               memref.store %[[VAL_156]], %[[VAL_4]]{{\[}}%[[VAL_147]]]
+// CHECK:               %[[VAL_158:.*]] = arith.cmpi eq, %[[VAL_136]], %[[VAL_127]]
+// CHECK:               %[[VAL_159:.*]] = scf.if %[[VAL_158]]
+// CHECK:                 scf.yield %[[VAL_147]]
+// CHECK:               } else {
+// CHECK:                 %[[VAL_160:.*]] = arith.cmpi eq, %[[VAL_147]], %[[VAL_127]]
+// CHECK:                 %[[VAL_161:.*]] = arith.select %[[VAL_160]], %[[VAL_136]], %[[VAL_127]]
+// CHECK:                 scf.yield %[[VAL_161]]
+// CHECK:               }
+// CHECK:               %[[VAL_162:.*]] = arith.andi %[[VAL_138]], %[[VAL_149]] : i1
+// CHECK:               %[[VAL_163:.*]]:2 = scf.if %[[VAL_162]]
+// CHECK:                 %[[VAL_164:.*]] = arith.addi %[[VAL_136]], %[[VAL_5]]
+// CHECK:                 %[[VAL_165:.*]] = arith.subi %[[VAL_147]], %[[VAL_5]]
+// CHECK:                 scf.yield %[[VAL_164]], %[[VAL_165]]
+// CHECK:               } else {
+// CHECK:                 scf.yield %[[VAL_136]], %[[VAL_147]]
+// CHECK:               }
+// CHECK:               scf.yield %[[VAL_166:.*]]#0, %[[VAL_166]]#1, %[[VAL_167:.*]]
 // CHECK:             } else {
-// CHECK:               scf.yield %[[I2]], %[[J2]], %[[Pa]]
+// CHECK:               scf.yield %[[VAL_136]], %[[VAL_147]], %[[VAL_127]]
 // CHECK:             }
-// CHECK:             scf.yield %[[If:.*]]#0, %[[If]]#1, %[[If]]#2
+// CHECK:             scf.yield %[[VAL_168:.*]]#0, %[[VAL_168]]#1, %[[VAL_168]]#2
 // CHECK:           }
-// CHECK:           return %[[W:.*]]#2
+// CHECK:           return %[[VAL_169:.*]]#2
 // CHECK:         }
 
 // CHECK-LABEL:   func.func private @_sparse_qsort_1_i8_f32_index(
@@ -132,13 +380,24 @@ func.func @sparse_push_back_inbound(%arg0: index, %arg1: memref<?xf64>, %arg2: f
 // CHECK-SAME:                                                   %[[Y0:.*]]: memref<?xf32>,
 // CHECK-SAME:                                                   %[[Y1:.*]]: memref<?xindex>) {
 // CHECK:           %[[C1:.*]] = arith.constant 1
-// CHECK:           %[[Lb:.*]] = arith.addi %[[L]], %[[C1]]
-// CHECK:           %[[COND:.*]] = arith.cmpi ult, %[[Lb]], %[[H]]
-// CHECK:           scf.if %[[COND]] {
-// CHECK:             %[[P:.*]] = func.call @_sparse_partition_1_i8_f32_index(%[[L]], %[[H]], %[[X0]], %[[Y0]], %[[Y1]])
-// CHECK:             func.call @_sparse_qsort_1_i8_f32_index(%[[L]], %[[P]], %[[X0]], %[[Y0]], %[[Y1]])
-// CHECK:             %[[P2:.*]] = arith.addi %[[P]], %[[C1]] : index
-// CHECK:             func.call @_sparse_qsort_1_i8_f32_index(%[[P2]], %[[H]], %[[X0]], %[[Y0]], %[[Y1]])
+// CHECK:           scf.while (%[[L2:.*]] = %[[L]], %[[H2:.*]] = %[[H]])
+// CHECK:             %[[Lb:.*]] = arith.addi %[[L2]], %[[C1]]
+// CHECK:             %[[COND:.*]] = arith.cmpi ult, %[[Lb]], %[[H2]]
+// CHECK:             scf.condition(%[[COND]]) %[[L2]], %[[H2]]
+// CHECK:           } do {
+// CHECK:           ^bb0(%[[L3:.*]]: index, %[[H3:.*]]: index)
+// CHECK:             %[[P:.*]] = func.call @_sparse_partition_1_i8_f32_index(%[[L3]], %[[H3]], %[[X0]], %[[Y0]], %[[Y1]])
+// CHECK:             %[[PP1:.*]] = arith.addi %[[P]], %[[C1]] : index
+// CHECK:             %[[LenL:.*]] = arith.subi %[[P]], %[[L3]]
+// CHECK:             %[[LenH:.*]] = arith.subi %[[H3]], %[[P]]
+// CHECK:             %[[Cmp:.*]] = arith.cmpi ule, %[[LenL]], %[[LenH]]
+// CHECK:             %[[L4:.*]] = arith.select %[[Cmp]], %[[PP1]], %[[L3]]
+// CHECK:             %[[H4:.*]] = arith.select %[[Cmp]], %[[H3]], %[[P]]
+// CHECK:             scf.if %[[Cmp]]
+// CHECK:               func.call @_sparse_qsort_1_i8_f32_index(%[[L3]], %[[P]], %[[X0]], %[[Y0]], %[[Y1]])
+// CHECK:             else
+// CHECK:               func.call @_sparse_qsort_1_i8_f32_index(%[[PP1]], %[[H3]], %[[X0]], %[[Y0]], %[[Y1]])
+// CHECK:             scf.yield %[[L4]], %[[H4]]
 // CHECK:           }
 // CHECK:           return
 // CHECK:         }
@@ -165,8 +424,6 @@ func.func @sparse_sort_1d2v_quick(%arg0: index, %arg1: memref<10xi8>, %arg2: mem
 // Only check the generated supporting function now. We have integration test
 // to verify correctness of the generated code.
 //
-// CHECK-DAG:     func.func private @_sparse_less_than_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>) -> i1 {
-// CHECK-DAG:     func.func private @_sparse_compare_eq_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>) -> i1 {
 // CHECK-DAG:     func.func private @_sparse_partition_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>) -> index {
 // CHECK-DAG:     func.func private @_sparse_qsort_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>) {
 // CHECK-LABEL:   func.func @sparse_sort_3d_quick
@@ -180,14 +437,12 @@ func.func @sparse_sort_3d_quick(%arg0: index, %arg1: memref<10xindex>, %arg2: me
 // Only check the generated supporting function now. We have integration test
 // to verify correctness of the generated code.
 //
-// CHECK-DAG:     func.func private @_sparse_less_than_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>) -> i1 {
 // CHECK-DAG:     func.func private @_sparse_binary_search_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>) -> index {
 // CHECK-DAG:     func.func private @_sparse_sort_stable_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>) {
 // CHECK-DAG:     func.func private @_sparse_shift_down_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>, %arg5: index) {
 // CHECK-DAG:     func.func private @_sparse_heap_sort_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>) {
-// CHECK-DAG:     func.func private @_sparse_compare_eq_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>) -> i1 {
 // CHECK-DAG:     func.func private @_sparse_partition_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>) -> index {
-// CHECK-DAG:     func.func private @_sparse_hybrid_qsort_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>, %arg5: memref<i64>) {
+// CHECK-DAG:     func.func private @_sparse_hybrid_qsort_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>, %arg5: i64) {
 // CHECK-LABEL:   func.func @sparse_sort_3d_hybrid
 func.func @sparse_sort_3d_hybrid(%arg0: index, %arg1: memref<10xindex>, %arg2: memref<?xindex>, %arg3: memref<10xindex>) -> (memref<10xindex>, memref<?xindex>, memref<10xindex>) {
   sparse_tensor.sort hybrid_quick_sort %arg0, %arg1, %arg2, %arg3 : memref<10xindex>, memref<?xindex>, memref<10xindex>
@@ -199,7 +454,6 @@ func.func @sparse_sort_3d_hybrid(%arg0: index, %arg1: memref<10xindex>, %arg2: m
 // Only check the generated supporting functions. We have integration test to
 // verify correctness of the generated code.
 //
-// CHECK-DAG:     func.func private @_sparse_less_than_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>) -> i1 {
 // CHECK-DAG:     func.func private @_sparse_binary_search_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>) -> index {
 // CHECK-DAG:     func.func private @_sparse_sort_stable_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>) {
 // CHECK-LABEL:   func.func @sparse_sort_3d_stable
@@ -213,7 +467,6 @@ func.func @sparse_sort_3d_stable(%arg0: index, %arg1: memref<10xindex>, %arg2: m
 // Only check the generated supporting functions. We have integration test to
 // verify correctness of the generated code.
 //
-// CHECK-DAG:     func.func private @_sparse_less_than_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>) -> i1 {
 // CHECK-DAG:     func.func private @_sparse_shift_down_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>, %arg5: index) {
 // CHECK-DAG:     func.func private @_sparse_heap_sort_3_index(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xindex>, %arg4: memref<?xindex>) {
 // CHECK-LABEL:   func.func @sparse_sort_3d_heap
@@ -227,8 +480,6 @@ func.func @sparse_sort_3d_heap(%arg0: index, %arg1: memref<10xindex>, %arg2: mem
 // Only check the generated supporting functions. We have integration test to
 // verify correctness of the generated code.
 //
-// CHECK-DAG:     func.func private @_sparse_less_than_2_index_coo_1(%arg0: index, %arg1: index, %arg2: memref<?xindex>) -> i1 {
-// CHECK-DAG:     func.func private @_sparse_compare_eq_2_index_coo_1(%arg0: index, %arg1: index, %arg2: memref<?xindex>) -> i1 {
 // CHECK-DAG:     func.func private @_sparse_partition_2_index_coo_1_f32_i32(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xf32>, %arg4: memref<?xi32>) -> index {
 // CHECK-DAG:     func.func private @_sparse_qsort_2_index_coo_1_f32_i32(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xf32>, %arg4: memref<?xi32>) {
 // CHECK-LABEL:   func.func @sparse_sort_coo_quick
@@ -242,14 +493,12 @@ func.func @sparse_sort_coo_quick(%arg0: index, %arg1: memref<100xindex>, %arg2:
 // Only check the generated supporting functions. We have integration test to
 // verify correctness of the generated code.
 //
-// CHECK-DAG:     func.func private @_sparse_less_than_2_index_coo_1(%arg0: index, %arg1: index, %arg2: memref<?xindex>) -> i1 {
 // CHECK-DAG:     func.func private @_sparse_binary_search_2_index_coo_1_f32_i32(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xf32>, %arg4: memref<?xi32>) -> index {
 // CHECK-DAG:     func.func private @_sparse_sort_stable_2_index_coo_1_f32_i32(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xf32>, %arg4: memref<?xi32>) {
 // CHECK-DAG:     func.func private @_sparse_shift_down_2_index_coo_1_f32_i32(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xf32>, %arg4: memref<?xi32>, %arg5: index) {
 // CHECK-DAG:     func.func private @_sparse_heap_sort_2_index_coo_1_f32_i32(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xf32>, %arg4: memref<?xi32>) {
-// CHECK-DAG:     func.func private @_sparse_compare_eq_2_index_coo_1(%arg0: index, %arg1: index, %arg2: memref<?xindex>) -> i1 {
 // CHECK-DAG:     func.func private @_sparse_partition_2_index_coo_1_f32_i32(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xf32>, %arg4: memref<?xi32>) -> index {
-// CHECK-DAG:     func.func private @_sparse_hybrid_qsort_2_index_coo_1_f32_i32(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xf32>, %arg4: memref<?xi32>, %arg5: memref<i64>) {
+// CHECK-DAG:     func.func private @_sparse_hybrid_qsort_2_index_coo_1_f32_i32(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xf32>, %arg4: memref<?xi32>, %arg5: i64) {
 // CHECK-LABEL:   func.func @sparse_sort_coo_hybrid
 func.func @sparse_sort_coo_hybrid(%arg0: index, %arg1: memref<100xindex>, %arg2: memref<?xf32>, %arg3: memref<10xi32>) -> (memref<100xindex>, memref<?xf32>, memref<10xi32>) {
   sparse_tensor.sort_coo hybrid_quick_sort %arg0, %arg1 jointly %arg2, %arg3 {nx = 2 : index, ny = 1: index} : memref<100xindex> jointly memref<?xf32>, memref<10xi32>
@@ -261,7 +510,6 @@ func.func @sparse_sort_coo_hybrid(%arg0: index, %arg1: memref<100xindex>, %arg2:
 // Only check the generated supporting functions. We have integration test to
 // verify correctness of the generated code.
 //
-// CHECK-DAG:     func.func private @_sparse_less_than_2_index_coo_1(%arg0: index, %arg1: index, %arg2: memref<?xindex>) -> i1 {
 // CHECK-DAG:     func.func private @_sparse_binary_search_2_index_coo_1_f32_i32(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xf32>, %arg4: memref<?xi32>) -> index {
 // CHECK-DAG:     func.func private @_sparse_sort_stable_2_index_coo_1_f32_i32(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xf32>, %arg4: memref<?xi32>) {
 // CHECK-LABEL:   func.func @sparse_sort_coo_stable
@@ -275,7 +523,6 @@ func.func @sparse_sort_coo_stable(%arg0: index, %arg1: memref<100xindex>, %arg2:
 // Only check the generated supporting functions. We have integration test to
 // verify correctness of the generated code.
 //
-// CHECK-DAG:     func.func private @_sparse_less_than_2_index_coo_1(%arg0: index, %arg1: index, %arg2: memref<?xindex>) -> i1 {
 // CHECK-DAG:     func.func private @_sparse_shift_down_2_index_coo_1_f32_i32(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xf32>, %arg4: memref<?xi32>, %arg5: index) {
 // CHECK-DAG:     func.func private @_sparse_heap_sort_2_index_coo_1_f32_i32(%arg0: index, %arg1: index, %arg2: memref<?xindex>, %arg3: memref<?xf32>, %arg4: memref<?xi32>) {
 // CHECK-LABEL:   func.func @sparse_sort_coo_heap
diff --git a/mlir/test/Dialect/SparseTensor/codegen.mlir b/mlir/test/Dialect/SparseTensor/codegen.mlir
index 827637bb4bc3b..87db4743026db 100644
--- a/mlir/test/Dialect/SparseTensor/codegen.mlir
+++ b/mlir/test/Dialect/SparseTensor/codegen.mlir
@@ -4,26 +4,26 @@
 
 #SparseVector = #sparse_tensor.encoding<{
   dimLevelType = [ "compressed" ],
-  indexBitWidth = 64,
-  pointerBitWidth = 32
+  crdWidth = 64,
+  posWidth = 32
 }>
 
 #Dense2D = #sparse_tensor.encoding<{
   dimLevelType = [ "dense", "dense" ],
-  indexBitWidth = 64,
-  pointerBitWidth = 32
+  crdWidth = 64,
+  posWidth = 32
 }>
 
 #Row = #sparse_tensor.encoding<{
   dimLevelType = [ "compressed", "dense" ],
-  indexBitWidth = 64,
-  pointerBitWidth = 32
+  crdWidth = 64,
+  posWidth = 32
 }>
 
 #CSR = #sparse_tensor.encoding<{
   dimLevelType = [ "dense", "compressed" ],
-  indexBitWidth = 64,
-  pointerBitWidth = 32
+  crdWidth = 64,
+  posWidth = 32
 }>
 
 #UCSR = #sparse_tensor.encoding<{
@@ -37,8 +37,8 @@
 
 #DCSR = #sparse_tensor.encoding<{
   dimLevelType = [ "compressed", "compressed" ],
-  indexBitWidth = 64,
-  pointerBitWidth = 32
+  crdWidth = 64,
+  posWidth = 32
 }>
 
 #Dense3D = #sparse_tensor.encoding<{
@@ -194,7 +194,7 @@ func.func @sparse_dense_3d(%arg0: tensor<10x20x30xf64, #Dense3D>) -> index {
 // CHECK-LABEL: func @sparse_dense_3d_dyn(
 //  CHECK-SAME: %[[A0:.*]]: memref<?xf64>,
 //  CHECK-SAME: %[[A1:.*]]: !sparse_tensor.storage_specifier
-//       CHECK: %[[A2:.*]] = sparse_tensor.storage_specifier.get %[[A1]] dim_sz at 2
+//       CHECK: %[[A2:.*]] = sparse_tensor.storage_specifier.get %[[A1]] lvl_sz at 2
 //       CHECK: return %[[A2]] : index
 func.func @sparse_dense_3d_dyn(%arg0: tensor<?x?x?xf64, #Dense3D>) -> index {
   %c = arith.constant 1 : index
@@ -202,7 +202,7 @@ func.func @sparse_dense_3d_dyn(%arg0: tensor<?x?x?xf64, #Dense3D>) -> index {
   return %0 : index
 }
 
-// CHECK-LABEL: func @sparse_pointers_dcsr(
+// CHECK-LABEL: func @sparse_positions_dcsr(
 //  CHECK-SAME: %[[A0:.*0]]: memref<?xi32>,
 //  CHECK-SAME: %[[A1:.*1]]: memref<?xi64>,
 //  CHECK-SAME: %[[A2:.*2]]: memref<?xi32>,
@@ -210,8 +210,8 @@ func.func @sparse_dense_3d_dyn(%arg0: tensor<?x?x?xf64, #Dense3D>) -> index {
 //  CHECK-SAME: %[[A4:.*4]]: memref<?xf64>,
 //  CHECK-SAME: %[[A5:.*5]]: !sparse_tensor.storage_specifier
 //       CHECK: return %[[A2]] : memref<?xi32>
-func.func @sparse_pointers_dcsr(%arg0: tensor<?x?xf64, #DCSR>) -> memref<?xi32> {
-  %0 = sparse_tensor.pointers %arg0 { dimension = 1 : index } : tensor<?x?xf64, #DCSR> to memref<?xi32>
+func.func @sparse_positions_dcsr(%arg0: tensor<?x?xf64, #DCSR>) -> memref<?xi32> {
+  %0 = sparse_tensor.positions %arg0 { level = 1 : index } : tensor<?x?xf64, #DCSR> to memref<?xi32>
   return %0 : memref<?xi32>
 }
 
@@ -224,7 +224,7 @@ func.func @sparse_pointers_dcsr(%arg0: tensor<?x?xf64, #DCSR>) -> memref<?xi32>
 //  CHECK-SAME: %[[A5:.*5]]: !sparse_tensor.storage_specifier
 //       CHECK: return %[[A3]] : memref<?xi64>
 func.func @sparse_indices_dcsr(%arg0: tensor<?x?xf64, #DCSR>) -> memref<?xi64> {
-  %0 = sparse_tensor.indices %arg0 { dimension = 1 : index } : tensor<?x?xf64, #DCSR> to memref<?xi64>
+  %0 = sparse_tensor.coordinates %arg0 { level = 1 : index } : tensor<?x?xf64, #DCSR> to memref<?xi64>
   return %0 : memref<?xi64>
 }
 
@@ -263,13 +263,13 @@ func.func @sparse_values_coo(%arg0: tensor<?x?x?xf64, #ccoo>) -> memref<?xf64> {
 //  CHECK-SAME: %[[A4:.*4]]: memref<?xf64>,
 //  CHECK-SAME: %[[A5:.*5]]: !sparse_tensor.storage_specifier
 //       CHECK: %[[C2:.*]] = arith.constant 2 : index
-//       CHECK: %[[S0:.*]] = sparse_tensor.storage_specifier.get %[[A5]]  idx_mem_sz at 1
+//       CHECK: %[[S0:.*]] = sparse_tensor.storage_specifier.get %[[A5]]  crd_mem_sz at 1
 //       CHECK: %[[S2:.*]] = arith.divui %[[S0]], %[[C2]] : index
 //       CHECK: %[[R1:.*]] = memref.subview %[[A3]][0] {{\[}}%[[S2]]] [2] : memref<?xindex> to memref<?xindex, strided<[2]>>
 //       CHECK: %[[R2:.*]] = memref.cast %[[R1]] : memref<?xindex, strided<[2]>> to memref<?xindex, strided<[?], offset: ?>>
 //       CHECK: return %[[R2]] : memref<?xindex, strided<[?], offset: ?>>
 func.func @sparse_indices_coo(%arg0: tensor<?x?x?xf64, #ccoo>) -> memref<?xindex, strided<[?], offset: ?>> {
-  %0 = sparse_tensor.indices  %arg0 { dimension = 1 : index } : tensor<?x?x?xf64, #ccoo> to memref<?xindex, strided<[?], offset: ?>>
+  %0 = sparse_tensor.coordinates  %arg0 { level = 1 : index } : tensor<?x?x?xf64, #ccoo> to memref<?xindex, strided<[?], offset: ?>>
   return %0 : memref<?xindex, strided<[?], offset: ?>>
 }
 
@@ -282,7 +282,7 @@ func.func @sparse_indices_coo(%arg0: tensor<?x?x?xf64, #ccoo>) -> memref<?xindex
 //  CHECK-SAME: %[[A5:.*5]]: !sparse_tensor.storage_specifier
 //       CHECK: return %[[A3]] : memref<?xindex>
 func.func @sparse_indices_buffer_coo(%arg0: tensor<?x?x?xf64, #ccoo>) -> memref<?xindex> {
-  %0 = sparse_tensor.indices_buffer  %arg0 : tensor<?x?x?xf64, #ccoo> to memref<?xindex>
+  %0 = sparse_tensor.coordinates_buffer  %arg0 : tensor<?x?x?xf64, #ccoo> to memref<?xindex>
   return %0 : memref<?xindex>
 }
 
@@ -323,13 +323,13 @@ func.func @sparse_dealloc_csr(%arg0: tensor<?x?xf64, #CSR>) {
 //       CHECK:     %[[A7:.*]] = memref.alloc() : memref<16xf64>
 //       CHECK:     %[[A8:.*]] = memref.cast %[[A7]] : memref<16xf64> to memref<?xf64>
 //       CHECK:     %[[A9:.*]] = sparse_tensor.storage_specifier.init : !sparse_tensor.storage_specifier
-//       CHECK:     %[[A11:.*]] = sparse_tensor.storage_specifier.set %[[A9]]  dim_sz at 0 with %[[A0]] : !sparse_tensor.storage_specifier
-//       CHECK:     %[[A12:.*]] = sparse_tensor.storage_specifier.set %[[A11]]  dim_sz at 1 with %[[A1]] : !sparse_tensor.storage_specifier
-//       CHECK:     %[[A14:.*]] = sparse_tensor.storage_specifier.get %[[A12]]  ptr_mem_sz at 1 : !sparse_tensor.storage_specifier
+//       CHECK:     %[[A11:.*]] = sparse_tensor.storage_specifier.set %[[A9]]  lvl_sz at 0 with %[[A0]] : !sparse_tensor.storage_specifier
+//       CHECK:     %[[A12:.*]] = sparse_tensor.storage_specifier.set %[[A11]]  lvl_sz at 1 with %[[A1]] : !sparse_tensor.storage_specifier
+//       CHECK:     %[[A14:.*]] = sparse_tensor.storage_specifier.get %[[A12]]  pos_mem_sz at 1 : !sparse_tensor.storage_specifier
 //       CHECK:     %[[A15:.*]], %[[A17:.*]] = sparse_tensor.push_back %[[A14]], %[[A4]], %[[A2]] : index, memref<?xindex>, index
-//       CHECK:     %[[A18:.*]] = sparse_tensor.storage_specifier.set %[[A12]]  ptr_mem_sz at 1 with %[[A17]] : !sparse_tensor.storage_specifier
+//       CHECK:     %[[A18:.*]] = sparse_tensor.storage_specifier.set %[[A12]]  pos_mem_sz at 1 with %[[A17]] : !sparse_tensor.storage_specifier
 //       CHECK:     %[[A23:.*]], %[[A25:.*]] = sparse_tensor.push_back %[[A17]], %[[A15]], %[[A2]], %[[A0]] : index, memref<?xindex>, index, index
-//       CHECK:     %[[A26:.*]] = sparse_tensor.storage_specifier.set %[[A18]]  ptr_mem_sz at 1 with %[[A25]] : !sparse_tensor.storage_specifier
+//       CHECK:     %[[A26:.*]] = sparse_tensor.storage_specifier.set %[[A18]]  pos_mem_sz at 1 with %[[A25]] : !sparse_tensor.storage_specifier
 //       CHECK:     return %[[A23]], %[[A6]], %[[A8]], %[[A26]] : memref<?xindex>, memref<?xindex>, memref<?xf64>, !sparse_tensor.storage_specifier
 func.func @sparse_alloc_csc(%arg0: index) -> tensor<10x?xf64, #CSC> {
   %0 = bufferization.alloc_tensor(%arg0) : tensor<10x?xf64, #CSC>
@@ -346,9 +346,9 @@ func.func @sparse_alloc_csc(%arg0: index) -> tensor<10x?xf64, #CSC> {
 //       CHECK:     %[[A5:.*]] = memref.alloc() : memref<6000xf64>
 //       CHECK:     %[[A6:.*]] = memref.cast %[[A5]] : memref<6000xf64> to memref<?xf64>
 //       CHECK:     %[[A7:.*]] = sparse_tensor.storage_specifier.init : !sparse_tensor.storage_specifier
-//       CHECK:     %[[A8:.*]] = sparse_tensor.storage_specifier.set %[[A7]]  dim_sz at 0 with %[[A3]] : !sparse_tensor.storage_specifier
-//       CHECK:     %[[A9:.*]] = sparse_tensor.storage_specifier.set %[[A8]]  dim_sz at 1 with %[[A2]] : !sparse_tensor.storage_specifier
-//       CHECK:     %[[A10:.*]] = sparse_tensor.storage_specifier.set %[[A9]]  dim_sz at 2 with %[[A1]] : !sparse_tensor.storage_specifier
+//       CHECK:     %[[A8:.*]] = sparse_tensor.storage_specifier.set %[[A7]]  lvl_sz at 0 with %[[A3]] : !sparse_tensor.storage_specifier
+//       CHECK:     %[[A9:.*]] = sparse_tensor.storage_specifier.set %[[A8]]  lvl_sz at 1 with %[[A2]] : !sparse_tensor.storage_specifier
+//       CHECK:     %[[A10:.*]] = sparse_tensor.storage_specifier.set %[[A9]]  lvl_sz at 2 with %[[A1]] : !sparse_tensor.storage_specifier
 //       CHECK:     %[[A12:.*]] = sparse_tensor.storage_specifier.get %[[A10]]  val_mem_sz : !sparse_tensor.storage_specifier
 //       CHECK:     %[[A15:.*]], %[[A14:.*]] = sparse_tensor.push_back %[[A12]], %[[A6]], %[[A4]], %[[A0]] : index, memref<?xf64>, f64, index
 //       CHECK:     %[[A16:.*]] = sparse_tensor.storage_specifier.set %[[A10]]  val_mem_sz with %[[A14]] : !sparse_tensor.storage_specifier
@@ -499,7 +499,7 @@ func.func @sparse_compression_1d(%tensor: tensor<100xf64, #SV>,
 //       CHECK:     memref.dealloc %[[A4]] : memref<?xf64>
 //       CHECK:     memref.dealloc %[[A5]] : memref<?xi1>
 //       CHECK:     memref.dealloc %[[A6]] : memref<?xindex>
-//       CHECK:     %[[A25:.*]] = sparse_tensor.storage_specifier.get %[[A24:.*]]#3  ptr_mem_sz at 1 : !sparse_tensor.storage_specifier
+//       CHECK:     %[[A25:.*]] = sparse_tensor.storage_specifier.get %[[A24:.*]]#3  pos_mem_sz at 1 : !sparse_tensor.storage_specifier
 //       CHECK:     %[[A26:.*]] = memref.load %[[A24]]#0{{\[}}%[[A13]]] : memref<?xi32>
 //       CHECK:     %[[A27:.*]] = scf.for %[[A28:.*]] = %[[A12]] to %[[A25]] step %[[A12]] iter_args(%[[A29:.*]] = %[[A26]]) -> (i32) {
 //       CHECK:       %[[A30:.*]] = memref.load %[[A24]]#0{{\[}}%[[A28]]] : memref<?xi32>
@@ -557,7 +557,7 @@ func.func @sparse_compression(%tensor: tensor<8x8xf64, #CSR>,
 //       CHECK:     memref.dealloc %[[A4]] : memref<?xf64>
 //       CHECK:     memref.dealloc %[[A5]] : memref<?xi1>
 //       CHECK:     memref.dealloc %[[A6]] : memref<?xindex>
-//       CHECK:     %[[A24:.*]] = sparse_tensor.storage_specifier.get %[[A23:.*]]#3  ptr_mem_sz at 1 : !sparse_tensor.storage_specifier
+//       CHECK:     %[[A24:.*]] = sparse_tensor.storage_specifier.get %[[A23:.*]]#3  pos_mem_sz at 1 : !sparse_tensor.storage_specifier
 //       CHECK:     %[[A25:.*]] = memref.load %[[A23]]#0{{\[}}%[[A11]]] : memref<?xindex>
 //       CHECK:     %[[A26:.*]] = scf.for %[[A27:.*]] = %[[A12]] to %[[A24]] step %[[A12]] iter_args(%[[A28:.*]] = %[[A25]]) -> (index) {
 //       CHECK:       %[[A29:.*]] = memref.load %[[A23]]#0{{\[}}%[[A27]]] : memref<?xindex>
@@ -675,20 +675,20 @@ func.func @sparse_nop_convert(%arg0: tensor<32xf32, #SparseVector>) -> tensor<?x
 //       CHECK: call @copySparseTensorReaderDimSizes(%[[A5]], %[[A7]]) : (!llvm.ptr<i8>, memref<?xindex>) -> ()
 //       CHECK: %[[A8:.*]] = memref.load %[[A6]]{{\[}}%[[A3]]] : memref<2xindex>
 //       CHECK: %[[A9:.*]] = memref.load %[[A6]]{{\[}}%[[A2]]] : memref<2xindex>
-//       CHECK: %[[A10:.*]] = call @getSparseTensorReaderNNZ(%[[A5]])
+//       CHECK: %[[A10:.*]] = call @getSparseTensorReaderNSE(%[[A5]])
 //       CHECK: %[[A11:.*]] = arith.muli %[[A10]], %[[A4]] : index
 //       CHECK: %[[A12:.*]] = memref.alloc() : memref<2xindex>
 //       CHECK: %[[A13:.*]] = memref.cast %[[A12]] : memref<2xindex> to memref<?xindex>
 //       CHECK: %[[A14:.*]] = memref.alloc(%[[A11]]) : memref<?xindex>
 //       CHECK: %[[A15:.*]] = memref.alloc(%[[A10]]) : memref<?xf32>
 //       CHECK: %[[A16:.*]] = sparse_tensor.storage_specifier.init : !sparse_tensor.storage_specifier<#sparse_tensor.encoding<{ dimLevelType = [ "compressed", "singleton" ] }>>
-//       CHECK: %[[A18:.*]] = sparse_tensor.storage_specifier.set %[[A16]]  dim_sz at 0 with %[[A8]]
-//       CHECK: %[[A19:.*]] = sparse_tensor.storage_specifier.get %[[A18]]  ptr_mem_sz at 0
+//       CHECK: %[[A18:.*]] = sparse_tensor.storage_specifier.set %[[A16]]  lvl_sz at 0 with %[[A8]]
+//       CHECK: %[[A19:.*]] = sparse_tensor.storage_specifier.get %[[A18]]  pos_mem_sz at 0
 //       CHECK: %[[A21:.*]], %[[A22:.*]] = sparse_tensor.push_back %[[A19]], %[[A13]], %[[A3]]
-//       CHECK: %[[A24:.*]] = sparse_tensor.storage_specifier.set %[[A18]]  ptr_mem_sz at 0 with %[[A22]]
-//       CHECK: %[[A26:.*]] = sparse_tensor.storage_specifier.set %[[A24]]  dim_sz at 1 with %[[A9]]
+//       CHECK: %[[A24:.*]] = sparse_tensor.storage_specifier.set %[[A18]]  pos_mem_sz at 0 with %[[A22]]
+//       CHECK: %[[A26:.*]] = sparse_tensor.storage_specifier.set %[[A24]]  lvl_sz at 1 with %[[A9]]
 //       CHECK: %[[A27:.*]], %[[A28:.*]] = sparse_tensor.push_back %[[A22]], %[[A21]], %[[A3]], %[[A2]]
-//       CHECK: %[[A30:.*]] = sparse_tensor.storage_specifier.set %[[A26]]  ptr_mem_sz at 0 with %[[A28]]
+//       CHECK: %[[A30:.*]] = sparse_tensor.storage_specifier.set %[[A26]]  pos_mem_sz at 0 with %[[A28]]
 //       CHECK: %[[A31:.*]] = memref.alloca() : memref<2xindex>
 //       CHECK: %[[A32:.*]] = memref.cast %[[A31]] : memref<2xindex> to memref<?xindex>
 //       CHECK: memref.store %[[A3]], %[[A31]]{{\[}}%[[A3]]] : memref<2xindex>
@@ -699,7 +699,7 @@ func.func @sparse_nop_convert(%arg0: tensor<32xf32, #SparseVector>) -> tensor<?x
 //       CHECK:   sparse_tensor.sort_coo  hybrid_quick_sort %[[A10]], %[[A14]] jointly %[[A15]] {nx = 2 : index, ny = 0 : index} : memref<?xindex> jointly memref<?xf32>
 //       CHECK: }
 //       CHECK: memref.store %[[A10]], %[[A27]]{{\[}}%[[A2]]] : memref<?xindex>
-//       CHECK: %[[A36:.*]] = sparse_tensor.storage_specifier.set %[[A30]]  idx_mem_sz at 0 with %[[A11]]
+//       CHECK: %[[A36:.*]] = sparse_tensor.storage_specifier.set %[[A30]]  crd_mem_sz at 0 with %[[A11]]
 //       CHECK: %[[A38:.*]] = sparse_tensor.storage_specifier.set %[[A36]]  val_mem_sz with %[[A10]]
 //       CHECK: call @delSparseTensorReader(%[[A5]]) : (!llvm.ptr<i8>) -> ()
 //       CHECK: return %[[A27]], %[[A14]], %[[A15]], %[[A38]]
@@ -719,31 +719,31 @@ func.func @sparse_new_coo(%arg0: !llvm.ptr<i8>) -> tensor<?x?xf32, #Coo> {
 //       CHECK: call @copySparseTensorReaderDimSizes(%[[A4]], %[[A6]])
 //       CHECK: %[[A7:.*]] = memref.load %[[A5]]{{\[}}%[[A2]]] : memref<2xindex>
 //       CHECK: %[[A8:.*]] = memref.load %[[A5]]{{\[}}%[[A1]]] : memref<2xindex>
-//       CHECK: %[[A9:.*]] = call @getSparseTensorReaderNNZ(%[[A4]])
+//       CHECK: %[[A9:.*]] = call @getSparseTensorReaderNSE(%[[A4]])
 //       CHECK: %[[A10:.*]] = arith.muli %[[A9]], %[[A3]] : index
 //       CHECK: %[[A11:.*]] = memref.alloc() : memref<2xindex>
 //       CHECK: %[[A12:.*]] = memref.cast %[[A11]] : memref<2xindex> to memref<?xindex>
 //       CHECK: %[[A13:.*]] = memref.alloc(%[[A10]]) : memref<?xindex>
 //       CHECK: %[[A14:.*]] = memref.alloc(%[[A9]]) : memref<?xf32>
 //       CHECK: %[[A15:.*]] = sparse_tensor.storage_specifier.init : !sparse_tensor.storage_specifier<#sparse_tensor.encoding<{ dimLevelType = [ "compressed", "singleton" ] }>>
-//       CHECK: %[[A17:.*]] = sparse_tensor.storage_specifier.set %[[A15]]  dim_sz at 0 with %[[A8]]
-//       CHECK: %[[A18:.*]] = sparse_tensor.storage_specifier.get %[[A17]]  ptr_mem_sz at 0
+//       CHECK: %[[A17:.*]] = sparse_tensor.storage_specifier.set %[[A15]]  lvl_sz at 0 with %[[A8]]
+//       CHECK: %[[A18:.*]] = sparse_tensor.storage_specifier.get %[[A17]]  pos_mem_sz at 0
 //       CHECK: %[[A20:.*]], %[[A21:.*]] = sparse_tensor.push_back %[[A18]], %[[A12]], %[[A2]]
-//       CHECK: %[[A23:.*]] = sparse_tensor.storage_specifier.set %[[A17]]  ptr_mem_sz at 0 with %[[A21]]
-//       CHECK: %[[A25:.*]] = sparse_tensor.storage_specifier.set %[[A23]]  dim_sz at 1 with %[[A7]]
+//       CHECK: %[[A23:.*]] = sparse_tensor.storage_specifier.set %[[A17]]  pos_mem_sz at 0 with %[[A21]]
+//       CHECK: %[[A25:.*]] = sparse_tensor.storage_specifier.set %[[A23]]  lvl_sz at 1 with %[[A7]]
 //       CHECK: %[[A26:.*]], %[[A27:.*]] = sparse_tensor.push_back %[[A21]], %[[A20]], %[[A2]], %[[A1]]
-//       CHECK: %[[A29:.*]] = sparse_tensor.storage_specifier.set %[[A25]]  ptr_mem_sz at 0 with %[[A27]]
+//       CHECK: %[[A29:.*]] = sparse_tensor.storage_specifier.set %[[A25]]  pos_mem_sz at 0 with %[[A27]]
 //       CHECK: %[[A30:.*]] = memref.alloca() : memref<2xindex>
 //       CHECK: %[[A31:.*]] = memref.cast %[[A30]] : memref<2xindex> to memref<?xindex>
 //       CHECK: memref.store %[[A1]], %[[A30]]{{\[}}%[[A2]]] : memref<2xindex>
 //       CHECK: memref.store %[[A2]], %[[A30]]{{\[}}%[[A1]]] : memref<2xindex>
 //       CHECK: %[[A32:.*]] = call @getSparseTensorReaderRead0F32(%[[A4]], %[[A31]], %[[A13]], %[[A14]])
 //       CHECK: memref.store %[[A9]], %[[A26]]{{\[}}%[[A1]]] : memref<?xindex>
-//       CHECK: %[[A34:.*]] = sparse_tensor.storage_specifier.set %[[A29]]  idx_mem_sz at 0 with %[[A10]]
+//       CHECK: %[[A34:.*]] = sparse_tensor.storage_specifier.set %[[A29]]  crd_mem_sz at 0 with %[[A10]]
 //       CHECK: %[[A36:.*]] = sparse_tensor.storage_specifier.set %[[A34]]  val_mem_sz with %[[A9]]
 //       CHECK: call @delSparseTensorReader(%[[A4]]) : (!llvm.ptr<i8>) -> ()
 //       CHECK: return %[[A26]], %[[A13]], %[[A14]], %[[A36]]
 func.func @sparse_new_coo_permute_no(%arg0: !llvm.ptr<i8>) -> tensor<?x?xf32, #CooPNo> {
   %0 = sparse_tensor.new %arg0 : !llvm.ptr<i8> to tensor<?x?xf32, #CooPNo>
   return %0 : tensor<?x?xf32, #CooPNo>
-}
\ No newline at end of file
+}
diff --git a/mlir/test/Dialect/SparseTensor/codegen_buffer_initialization.mlir b/mlir/test/Dialect/SparseTensor/codegen_buffer_initialization.mlir
index 7a0d668082d52..688cde550f2e7 100644
--- a/mlir/test/Dialect/SparseTensor/codegen_buffer_initialization.mlir
+++ b/mlir/test/Dialect/SparseTensor/codegen_buffer_initialization.mlir
@@ -17,12 +17,12 @@
 //       CHECK:     %[[VAL_9:.*]] = memref.cast %[[VAL_8]] : memref<16xf64> to memref<?xf64>
 //       CHECK:     linalg.fill ins(%[[VAL_2]] : f64) outs(%[[VAL_8]] : memref<16xf64>)
 //       CHECK:     %[[VAL_10:.*]] = sparse_tensor.storage_specifier.init : !sparse_tensor.storage_specifier
-//       CHECK:     %[[VAL_12:.*]] = sparse_tensor.storage_specifier.set %[[VAL_10]]  dim_sz at 0 with %[[VAL_0]] : !sparse_tensor.storage_specifier
-//       CHECK:     %[[VAL_14:.*]] = sparse_tensor.storage_specifier.get %[[VAL_12]]  ptr_mem_sz at 0 : !sparse_tensor.storage_specifier
+//       CHECK:     %[[VAL_12:.*]] = sparse_tensor.storage_specifier.set %[[VAL_10]]  lvl_sz at 0 with %[[VAL_0]] : !sparse_tensor.storage_specifier
+//       CHECK:     %[[VAL_14:.*]] = sparse_tensor.storage_specifier.get %[[VAL_12]]  pos_mem_sz at 0 : !sparse_tensor.storage_specifier
 //       CHECK:     %[[VAL_15:.*]], %[[VAL_17:.*]] = sparse_tensor.push_back %[[VAL_14]], %[[VAL_5]], %[[VAL_3]] : index, memref<?xindex>, index
-//       CHECK:     %[[VAL_18:.*]] = sparse_tensor.storage_specifier.set %[[VAL_12]]  ptr_mem_sz at 0 with %[[VAL_17]] : !sparse_tensor.storage_specifier
+//       CHECK:     %[[VAL_18:.*]] = sparse_tensor.storage_specifier.set %[[VAL_12]]  pos_mem_sz at 0 with %[[VAL_17]] : !sparse_tensor.storage_specifier
 //       CHECK:     %[[VAL_19:.*]], %[[VAL_21:.*]] = sparse_tensor.push_back %[[VAL_17]], %[[VAL_15]], %[[VAL_3]], %[[VAL_1]] : index, memref<?xindex>, index, index
-//       CHECK:     %[[VAL_22:.*]] = sparse_tensor.storage_specifier.set %[[VAL_18]]  ptr_mem_sz at 0 with %[[VAL_21]] : !sparse_tensor.storage_specifier
+//       CHECK:     %[[VAL_22:.*]] = sparse_tensor.storage_specifier.set %[[VAL_18]]  pos_mem_sz at 0 with %[[VAL_21]] : !sparse_tensor.storage_specifier
 //       CHECK:     return %[[VAL_19]], %[[VAL_7]], %[[VAL_9]], %[[VAL_22]] : memref<?xindex>, memref<?xindex>, memref<?xf64>, !sparse_tensor.storage_specifier
 func.func @sparse_alloc_sparse_vector(%arg0: index) -> tensor<?xf64, #SV> {
   %0 = bufferization.alloc_tensor(%arg0) : tensor<?xf64, #SV>
diff --git a/mlir/test/Dialect/SparseTensor/conversion.mlir b/mlir/test/Dialect/SparseTensor/conversion.mlir
index dc4efae50c01b..5fdaf1bb7dbd1 100644
--- a/mlir/test/Dialect/SparseTensor/conversion.mlir
+++ b/mlir/test/Dialect/SparseTensor/conversion.mlir
@@ -6,14 +6,14 @@
 
 #SparseVector64 = #sparse_tensor.encoding<{
   dimLevelType = ["compressed"],
-  pointerBitWidth = 64,
-  indexBitWidth = 64
+  posWidth = 64,
+  crdWidth = 64
 }>
 
 #SparseVector32 = #sparse_tensor.encoding<{
   dimLevelType = ["compressed"],
-  pointerBitWidth = 32,
-  indexBitWidth = 32
+  posWidth = 32,
+  crdWidth = 32
 }>
 
 #CSR = #sparse_tensor.encoding<{
@@ -174,63 +174,63 @@ func.func @sparse_nop_cast(%arg0: tensor<64xf32, #SparseVector>) -> tensor<?xf32
   return %0 : tensor<?xf32, #SparseVector>
 }
 
-// CHECK-LABEL: func @sparse_pointers(
+// CHECK-LABEL: func @sparse_positions(
 //  CHECK-SAME: %[[A:.*]]: !llvm.ptr<i8>)
 //       CHECK: %[[C:.*]] = arith.constant 0 : index
-//       CHECK: %[[T:.*]] = call @sparsePointers0(%[[A]], %[[C]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+//       CHECK: %[[T:.*]] = call @sparsePositions0(%[[A]], %[[C]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
 //       CHECK: return %[[T]] : memref<?xindex>
-func.func @sparse_pointers(%arg0: tensor<128xf64, #SparseVector>) -> memref<?xindex> {
-  %0 = sparse_tensor.pointers %arg0 { dimension = 0 : index } : tensor<128xf64, #SparseVector> to memref<?xindex>
+func.func @sparse_positions(%arg0: tensor<128xf64, #SparseVector>) -> memref<?xindex> {
+  %0 = sparse_tensor.positions %arg0 { level = 0 : index } : tensor<128xf64, #SparseVector> to memref<?xindex>
   return %0 : memref<?xindex>
 }
 
-// CHECK-LABEL: func @sparse_pointers64(
+// CHECK-LABEL: func @sparse_positions64(
 //  CHECK-SAME: %[[A:.*]]: !llvm.ptr<i8>)
 //       CHECK: %[[C:.*]] = arith.constant 0 : index
-//       CHECK: %[[T:.*]] = call @sparsePointers64(%[[A]], %[[C]]) : (!llvm.ptr<i8>, index) -> memref<?xi64>
+//       CHECK: %[[T:.*]] = call @sparsePositions64(%[[A]], %[[C]]) : (!llvm.ptr<i8>, index) -> memref<?xi64>
 //       CHECK: return %[[T]] : memref<?xi64>
-func.func @sparse_pointers64(%arg0: tensor<128xf64, #SparseVector64>) -> memref<?xi64> {
-  %0 = sparse_tensor.pointers %arg0 { dimension = 0 : index } : tensor<128xf64, #SparseVector64> to memref<?xi64>
+func.func @sparse_positions64(%arg0: tensor<128xf64, #SparseVector64>) -> memref<?xi64> {
+  %0 = sparse_tensor.positions %arg0 { level = 0 : index } : tensor<128xf64, #SparseVector64> to memref<?xi64>
   return %0 : memref<?xi64>
 }
 
-// CHECK-LABEL: func @sparse_pointers32(
+// CHECK-LABEL: func @sparse_positions32(
 //  CHECK-SAME: %[[A:.*]]: !llvm.ptr<i8>)
 //       CHECK: %[[C:.*]] = arith.constant 0 : index
-//       CHECK: %[[T:.*]] = call @sparsePointers32(%[[A]], %[[C]]) : (!llvm.ptr<i8>, index) -> memref<?xi32>
+//       CHECK: %[[T:.*]] = call @sparsePositions32(%[[A]], %[[C]]) : (!llvm.ptr<i8>, index) -> memref<?xi32>
 //       CHECK: return %[[T]] : memref<?xi32>
-func.func @sparse_pointers32(%arg0: tensor<128xf64, #SparseVector32>) -> memref<?xi32> {
-  %0 = sparse_tensor.pointers %arg0 { dimension = 0 : index } : tensor<128xf64, #SparseVector32> to memref<?xi32>
+func.func @sparse_positions32(%arg0: tensor<128xf64, #SparseVector32>) -> memref<?xi32> {
+  %0 = sparse_tensor.positions %arg0 { level = 0 : index } : tensor<128xf64, #SparseVector32> to memref<?xi32>
   return %0 : memref<?xi32>
 }
 
 // CHECK-LABEL: func @sparse_indices(
 //  CHECK-SAME: %[[A:.*]]: !llvm.ptr<i8>)
 //       CHECK: %[[C:.*]] = arith.constant 0 : index
-//       CHECK: %[[T:.*]] = call @sparseIndices0(%[[A]], %[[C]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+//       CHECK: %[[T:.*]] = call @sparseCoordinates0(%[[A]], %[[C]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
 //       CHECK: return %[[T]] : memref<?xindex>
 func.func @sparse_indices(%arg0: tensor<128xf64, #SparseVector>) -> memref<?xindex> {
-  %0 = sparse_tensor.indices %arg0 { dimension = 0 : index } : tensor<128xf64, #SparseVector> to memref<?xindex>
+  %0 = sparse_tensor.coordinates %arg0 { level = 0 : index } : tensor<128xf64, #SparseVector> to memref<?xindex>
   return %0 : memref<?xindex>
 }
 
 // CHECK-LABEL: func @sparse_indices64(
 //  CHECK-SAME: %[[A:.*]]: !llvm.ptr<i8>)
 //       CHECK: %[[C:.*]] = arith.constant 0 : index
-//       CHECK: %[[T:.*]] = call @sparseIndices64(%[[A]], %[[C]]) : (!llvm.ptr<i8>, index) -> memref<?xi64>
+//       CHECK: %[[T:.*]] = call @sparseCoordinates64(%[[A]], %[[C]]) : (!llvm.ptr<i8>, index) -> memref<?xi64>
 //       CHECK: return %[[T]] : memref<?xi64>
 func.func @sparse_indices64(%arg0: tensor<128xf64, #SparseVector64>) -> memref<?xi64> {
-  %0 = sparse_tensor.indices %arg0 { dimension = 0 : index } : tensor<128xf64, #SparseVector64> to memref<?xi64>
+  %0 = sparse_tensor.coordinates %arg0 { level = 0 : index } : tensor<128xf64, #SparseVector64> to memref<?xi64>
   return %0 : memref<?xi64>
 }
 
 // CHECK-LABEL: func @sparse_indices32(
 //  CHECK-SAME: %[[A:.*]]: !llvm.ptr<i8>)
 //       CHECK: %[[C:.*]] = arith.constant 0 : index
-//       CHECK: %[[T:.*]] = call @sparseIndices32(%[[A]], %[[C]]) : (!llvm.ptr<i8>, index) -> memref<?xi32>
+//       CHECK: %[[T:.*]] = call @sparseCoordinates32(%[[A]], %[[C]]) : (!llvm.ptr<i8>, index) -> memref<?xi32>
 //       CHECK: return %[[T]] : memref<?xi32>
 func.func @sparse_indices32(%arg0: tensor<128xf64, #SparseVector32>) -> memref<?xi32> {
-  %0 = sparse_tensor.indices %arg0 { dimension = 0 : index } : tensor<128xf64, #SparseVector32> to memref<?xi32>
+  %0 = sparse_tensor.coordinates %arg0 { level = 0 : index } : tensor<128xf64, #SparseVector32> to memref<?xi32>
   return %0 : memref<?xi32>
 }
 
diff --git a/mlir/test/Dialect/SparseTensor/convert_dense2sparse.mlir b/mlir/test/Dialect/SparseTensor/convert_dense2sparse.mlir
index 2646b2db71148..92f63767a436f 100644
--- a/mlir/test/Dialect/SparseTensor/convert_dense2sparse.mlir
+++ b/mlir/test/Dialect/SparseTensor/convert_dense2sparse.mlir
@@ -183,30 +183,17 @@ func.func @sparse_constant() -> tensor<8x7xf32, #CSR>{
   return %1 : tensor<8x7xf32, #CSR>
 }
 
-// CHECK-RWT-LABEL:   func.func @sparse_constant_csc() -> tensor<8x7xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)> }>> {
-//       CHECK-RWT:     %[[F0:.*]] = arith.constant sparse<{{\[\[}}0, 0], [1, 6]], [1.000000e+00, 5.000000e+00]> : tensor<8x7xf32>
-//       CHECK-RWT:     %[[T0:.*]] = bufferization.alloc_tensor()
-//       CHECK-RWT:     %[[T1:.*]] = sparse_tensor.foreach in %[[F0]] init(%[[T0]])
-//       CHECK-RWT:     ^bb0(%[[L0I0:.*]]: index, %[[L0I1:.*]]: index, %[[L0V:.*]]: f32, %[[L0T:.*]]: tensor
-//       CHECK-RWT:       %[[L0T2:.*]] = sparse_tensor.insert %[[L0V]] into %[[L0T]]{{\[}}%[[L0I1]], %[[L0I0]]]
-//       CHECK-RWT:       sparse_tensor.yield %[[L0T2]]
-//       CHECK-RWT:     }
-//       CHECK-RWT:     %[[COO:.*]] = sparse_tensor.load %[[T1]] hasInserts
-//       CHECK-RWT:     %[[NNZ:.*]] = sparse_tensor.number_of_entries %[[COO]]
-//       CHECK-RWT:     %[[V:.*]] = sparse_tensor.values %[[COO]]
-//       CHECK-RWT:     %[[I:.*]] = sparse_tensor.indices_buffer %[[COO]]
-//       CHECK-RWT:     sparse_tensor.sort_coo hybrid_quick_sort %[[NNZ]], %[[I]] jointly %[[V]] {nx = 2 : index, ny = 0 : index}
-//       CHECK-RWT:     %[[T3:.*]] = bufferization.alloc_tensor()
-//       CHECK-RWT:     %[[T4:.*]] = sparse_tensor.foreach in %[[COO]] init(%[[T3]])
-//       CHECK-RWT:     ^bb0(%[[L1I0:.*]]: index, %[[L1I1:.*]]: index, %[[L1V:.*]]: f32, %[[L1T:.*]]: tensor
-//       CHECK-RWT:       %[[L1T2:.*]] = sparse_tensor.insert %[[L1V]] into %[[L1T]]{{\[}}%[[L1I1]], %[[L1I0]]]
-//       CHECK-RWT:       sparse_tensor.yield %[[L1T2]]
-//       CHECK-RWT:     }
-//       CHECK-RWT:     %[[T5:.*]] = sparse_tensor.load %[[T4]] hasInserts
-//       CHECK-RWT:     %[[T6:.*]] = sparse_tensor.convert %[[T5]]
-//       CHECK-RWT:     bufferization.dealloc_tensor %[[COO]]
-//       CHECK-RWT:     return %[[T6]]
-//       CHECK-RWT:   }
+// CHECK-RWT-LABEL:   func.func @sparse_constant_csc() -> tensor<8x7xf32,
+//       CHECK-RWT:    %[[VAL_0:.*]] = arith.constant sparse<{{\[\[}}0, 0], [1, 6]], [1.000000e+00, 5.000000e+00]> : tensor<8x7xf32>
+//       CHECK-RWT:    %[[VAL_1:.*]] = bufferization.alloc_tensor() :
+//       CHECK-RWT:    %[[VAL_2:.*]] = sparse_tensor.foreach in %[[VAL_0]] init(%[[VAL_1]]) {order = #map} : tensor<8x7xf32>,
+//       CHECK-RWT:    ^bb0(%[[VAL_3:.*]]: index, %[[VAL_4:.*]]: index, %[[VAL_5:.*]]: f32, %[[VAL_6:.*]]: tensor
+//       CHECK-RWT:      %[[VAL_7:.*]] = sparse_tensor.insert %[[VAL_5]] into %[[VAL_6]]{{\[}}%[[VAL_4]], %[[VAL_3]]] :
+//       CHECK-RWT:      sparse_tensor.yield %[[VAL_7]] :
+//       CHECK-RWT:    }
+//       CHECK-RWT:    %[[VAL_8:.*]] = sparse_tensor.load %[[VAL_9:.*]] hasInserts :
+//       CHECK-RWT:    return %[[VAL_8]] :
+//       CHECK-RWT:  }
 func.func @sparse_constant_csc() -> tensor<8x7xf32, #CSC>{
   // Initialize a tensor.
   %0 = arith.constant sparse<[[0, 0], [1, 6]], [1.0, 5.0]> : tensor<8x7xf32>
diff --git a/mlir/test/Dialect/SparseTensor/convert_sparse2sparse.mlir b/mlir/test/Dialect/SparseTensor/convert_sparse2sparse.mlir
index e618ee81dd3bc..5c8ebe325d6d2 100644
--- a/mlir/test/Dialect/SparseTensor/convert_sparse2sparse.mlir
+++ b/mlir/test/Dialect/SparseTensor/convert_sparse2sparse.mlir
@@ -11,14 +11,14 @@
 
 #SparseVector64 = #sparse_tensor.encoding<{
   dimLevelType = ["compressed"],
-  pointerBitWidth = 64,
-  indexBitWidth = 64
+  posWidth = 64,
+  crdWidth = 64
 }>
 
 #SparseVector32 = #sparse_tensor.encoding<{
   dimLevelType = ["compressed"],
-  pointerBitWidth = 32,
-  indexBitWidth = 32
+  posWidth = 32,
+  crdWidth = 32
 }>
 
 #SparseVector = #sparse_tensor.encoding<{
@@ -107,14 +107,14 @@ func.func @sparse_convert(%arg0: tensor<?xf32, #SparseVector64>) -> tensor<?xf32
 
 #SparseSingleton64 = #sparse_tensor.encoding<{
   dimLevelType = ["singleton"],
-  pointerBitWidth = 64,
-  indexBitWidth = 64
+  posWidth = 64,
+  crdWidth = 64
 }>
 
 #SparseSingleton32 = #sparse_tensor.encoding<{
   dimLevelType = ["singleton"],
-  pointerBitWidth = 32,
-  indexBitWidth = 32
+  posWidth = 32,
+  crdWidth = 32
 }>
 
 // CHECK-COO-LABEL: func @sparse_convert_singleton(
@@ -153,22 +153,34 @@ func.func @sparse_convert_singleton(%arg0: tensor<?xf32, #SparseSingleton64>) ->
 }
 
 // CHECK-RWT-LABEL: func.func @sparse_convert_permuted(
-//  CHECK-RWT-SAME: %[[COO:.*]]:
-//   CHECK-RWT-DAG: %[[C0:.*]] = arith.constant 0 : index
-//   CHECK-RWT-DAG: %[[C1:.*]] = arith.constant 1 : index
-//   CHECK-RWT-DAG: %[[C2:.*]] = arith.constant 2 : index
-//       CHECK-RWT: %[[D0:.*]] = tensor.dim %[[COO]], %[[C0]]
-//       CHECK-RWT: %[[D1:.*]] = tensor.dim %[[COO]], %[[C1]]
-//       CHECK-RWT: %[[D2:.*]] = tensor.dim %[[COO]], %[[C2]]
-//       CHECK-RWT: %[[T1:.*]] = bufferization.alloc_tensor(%[[D0]], %[[D1]], %[[D2]])
-//       CHECK-RWT: %[[T2:.*]] = sparse_tensor.foreach in %[[COO]] init(%[[T1]])
-//       CHECK-RWT: ^bb0(%[[LI0:.*]]: index, %[[LI1:.*]]: index, %[[LI2:.*]]: index, %[[LV:.*]]: f32, %[[LT1:.*]]: tensor<?x?x?xf32,
-//       CHECK-RWT:   %[[LT2:.*]] = sparse_tensor.insert %[[LV]] into %[[LT1]]{{\[}}%[[LI2]], %[[LI0]], %[[LI1]]]
-//       CHECK-RWT:   sparse_tensor.yield %[[LT2]]
+//  CHECK-RWT-SAME: %[[VAL_0:.*]]: tensor<?x?x?xf32, #{{.*}}>>) -> tensor<?x?x?xf32, #{{.*}}>> {
+//   CHECK-RWT-DAG: %[[VAL_1:.*]] = arith.constant 0 : index
+//   CHECK-RWT-DAG: %[[VAL_2:.*]] = arith.constant 1 : index
+//   CHECK-RWT-DAG: %[[VAL_3:.*]] = arith.constant 2 : index
+//   CHECK-RWT-DAG: %[[VAL_4:.*]] = tensor.dim %[[VAL_0]], %[[VAL_1]]
+//   CHECK-RWT-DAG: %[[VAL_5:.*]] = tensor.dim %[[VAL_0]], %[[VAL_2]]
+//   CHECK-RWT-DAG: %[[VAL_6:.*]] = tensor.dim %[[VAL_0]], %[[VAL_3]]
+//   CHECK-RWT-DAG: %[[VAL_7:.*]] = sparse_tensor.number_of_entries %[[VAL_0]]
+//       CHECK-RWT: %[[VAL_8:.*]] = bufferization.alloc_tensor(%[[VAL_4]], %[[VAL_5]], %[[VAL_6]]) size_hint=%[[VAL_7]]
+//       CHECK-RWT: %[[VAL_9:.*]] = sparse_tensor.foreach in %[[VAL_0]] init(%[[VAL_8]])
+//       CHECK-RWT: ^bb0(%[[VAL_10:.*]]: index, %[[VAL_11:.*]]: index, %[[VAL_12:.*]]: index, %[[VAL_13:.*]]: f32, %[[VAL_14:.*]]: tensor<?x?x?xf32, #{{.*}}>>):
+//       CHECK-RWT:   %[[VAL_15:.*]] = sparse_tensor.insert %[[VAL_13]] into %[[VAL_14]]{{\[}}%[[VAL_12]], %[[VAL_10]], %[[VAL_11]]]
+//       CHECK-RWT:   sparse_tensor.yield %[[VAL_15]] : tensor<?x?x?xf32, #{{.*}}>>
 //       CHECK-RWT: }
-//       CHECK-RWT: %[[T3:.*]] = sparse_tensor.load %[[T2:.*]] hasInserts
-//       CHECK-RWT: %[[T4:.*]] = sparse_tensor.convert %[[T3]]
-//       CHECK-RWT: return %[[T4]]
+//       CHECK-RWT: %[[VAL_16:.*]] = sparse_tensor.load %[[VAL_17:.*]] hasInserts : tensor<?x?x?xf32, #{{.*}}>>
+//       CHECK-RWT: %[[VAL_18:.*]] = sparse_tensor.values %[[VAL_16]] : tensor<?x?x?xf32, #{{.*}}>> to memref<?xf32>
+//       CHECK-RWT: %[[VAL_19:.*]] = sparse_tensor.coordinates_buffer %[[VAL_16]] : tensor<?x?x?xf32, #{{.*}}>> to memref<?xindex>
+//       CHECK-RWT: sparse_tensor.sort_coo  hybrid_quick_sort %[[VAL_7]], %[[VAL_19]] jointly %[[VAL_18]] {nx = 3 : index, ny = 0 : index}
+//       CHECK-RWT: %[[VAL_20:.*]] = bufferization.alloc_tensor(%[[VAL_4]], %[[VAL_5]], %[[VAL_6]]) size_hint=%[[VAL_7]]
+//       CHECK-RWT: %[[VAL_21:.*]] = sparse_tensor.foreach in %[[VAL_16]] init(%[[VAL_20]])
+//       CHECK-RWT: ^bb0(%[[VAL_22:.*]]: index, %[[VAL_23:.*]]: index, %[[VAL_24:.*]]: index, %[[VAL_25:.*]]: f32, %[[VAL_26:.*]]: tensor<?x?x?xf32, #{{.*}}>>):
+//       CHECK-RWT:   %[[VAL_27:.*]] = sparse_tensor.insert %[[VAL_25]] into %[[VAL_26]]{{\[}}%[[VAL_24]], %[[VAL_22]], %[[VAL_23]]]
+//       CHECK-RWT:   sparse_tensor.yield %[[VAL_27]]
+//       CHECK-RWT: }
+//       CHECK-RWT: bufferization.dealloc_tensor %[[VAL_16]]
+//       CHECK-RWT: %[[VAL_28:.*]] = sparse_tensor.load %[[VAL_29:.*]] hasInserts
+//       CHECK-RWT: %[[VAL_30:.*]] = sparse_tensor.convert %[[VAL_28]]
+//       CHECK-RWT: return %[[VAL_30]]
 func.func @sparse_convert_permuted(%arg0: tensor<?x?x?xf32, #SortedCOO3D>) -> tensor<?x?x?xf32, #TsssPermuted> {
   %0 = sparse_tensor.convert %arg0 : tensor<?x?x?xf32, #SortedCOO3D> to tensor<?x?x?xf32, #TsssPermuted>
   return %0 : tensor<?x?x?xf32, #TsssPermuted>
diff --git a/mlir/test/Dialect/SparseTensor/convert_sparse2sparse_element.mlir b/mlir/test/Dialect/SparseTensor/convert_sparse2sparse_element.mlir
index e45d7942f303d..9795ef23d711e 100644
--- a/mlir/test/Dialect/SparseTensor/convert_sparse2sparse_element.mlir
+++ b/mlir/test/Dialect/SparseTensor/convert_sparse2sparse_element.mlir
@@ -2,14 +2,14 @@
 
 #SparseVector64 = #sparse_tensor.encoding<{
   dimLevelType = ["compressed"],
-  pointerBitWidth = 64,
-  indexBitWidth = 64
+  posWidth = 64,
+  crdWidth = 64
 }>
 
 #SparseVector32 = #sparse_tensor.encoding<{
   dimLevelType = ["compressed"],
-  pointerBitWidth = 32,
-  indexBitWidth = 32
+  posWidth = 32,
+  crdWidth = 32
 }>
 
 
diff --git a/mlir/test/Dialect/SparseTensor/fold.mlir b/mlir/test/Dialect/SparseTensor/fold.mlir
index 3e559109189fe..229d27ed82775 100644
--- a/mlir/test/Dialect/SparseTensor/fold.mlir
+++ b/mlir/test/Dialect/SparseTensor/fold.mlir
@@ -22,13 +22,13 @@ func.func @sparse_dce_convert(%arg0: tensor<64xf32>) {
 
 // CHECK-LABEL: func @sparse_dce_getters(
 //  CHECK-SAME: %[[A:.*]]: tensor<64xf32, #sparse_tensor.encoding<{{{.*}}}>>)
-//   CHECK-NOT: sparse_tensor.pointers
-//   CHECK-NOT: sparse_tensor.indices
+//   CHECK-NOT: sparse_tensor.positions
+//   CHECK-NOT: sparse_tensor.coordinates
 //   CHECK-NOT: sparse_tensor.values
 //       CHECK: return
 func.func @sparse_dce_getters(%arg0: tensor<64xf32, #SparseVector>) {
-  %0 = sparse_tensor.pointers %arg0 { dimension = 0 : index } : tensor<64xf32, #SparseVector> to memref<?xindex>
-  %1 = sparse_tensor.indices %arg0 { dimension = 0 : index } : tensor<64xf32, #SparseVector> to memref<?xindex>
+  %0 = sparse_tensor.positions %arg0 { level = 0 : index } : tensor<64xf32, #SparseVector> to memref<?xindex>
+  %1 = sparse_tensor.coordinates %arg0 { level = 0 : index } : tensor<64xf32, #SparseVector> to memref<?xindex>
   %2 = sparse_tensor.values %arg0 : tensor<64xf32, #SparseVector> to memref<?xf32>
   return
 }
@@ -54,11 +54,11 @@ func.func @sparse_concat_dce(%arg0: tensor<2xf64, #SparseVector>,
 //   CHECK-NOT:  sparse_tensor.storage_specifier.get
 //       CHECK:  return %[[A1]]
 func.func @sparse_get_specifier_dce_fold(%arg0: !sparse_tensor.storage_specifier<#SparseVector>, %arg1: index, %arg2: index) -> index {
-  %0 = sparse_tensor.storage_specifier.set %arg0 dim_sz at 0 with %arg1
+  %0 = sparse_tensor.storage_specifier.set %arg0 lvl_sz at 0 with %arg1
        : !sparse_tensor.storage_specifier<#SparseVector>
-  %1 = sparse_tensor.storage_specifier.set %0 ptr_mem_sz at 0 with %arg2
+  %1 = sparse_tensor.storage_specifier.set %0 pos_mem_sz at 0 with %arg2
        : !sparse_tensor.storage_specifier<#SparseVector>
-  %2 = sparse_tensor.storage_specifier.get %1 dim_sz at 0
+  %2 = sparse_tensor.storage_specifier.get %1 lvl_sz at 0
        : !sparse_tensor.storage_specifier<#SparseVector>
   return %2 : index
 }
diff --git a/mlir/test/Dialect/SparseTensor/invalid.mlir b/mlir/test/Dialect/SparseTensor/invalid.mlir
index 8d0a0e7d69869..6c0f13ae1b564 100644
--- a/mlir/test/Dialect/SparseTensor/invalid.mlir
+++ b/mlir/test/Dialect/SparseTensor/invalid.mlir
@@ -8,135 +8,135 @@ func.func @invalid_new_dense(%arg0: !llvm.ptr<i8>) -> tensor<32xf32> {
 
 // -----
 
-#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], indexBitWidth=32}>
+#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], crdWidth=32}>
 
-func.func @non_static_pack_ret(%data: tensor<6xf64>, %index: tensor<6x1xi32>)
+func.func @non_static_pack_ret(%values: tensor<6xf64>, %coordinates: tensor<6x1xi32>)
                             -> tensor<?xf64, #SparseVector> {
-  // expected-error@+1 {{all input types must be statically shaped}}
-  %0 = sparse_tensor.pack %data, %index : tensor<6xf64>, tensor<6x1xi32>
-                                       to tensor<?xf64, #SparseVector>
+  // expected-error@+1 {{the sparse-tensor must have static shape}}
+  %0 = sparse_tensor.pack %values, %coordinates
+     : tensor<6xf64>, tensor<6x1xi32> to tensor<?xf64, #SparseVector>
   return %0 : tensor<?xf64, #SparseVector>
 }
 
 // -----
 
-#DenseVector = #sparse_tensor.encoding<{dimLevelType = ["dense"], indexBitWidth=32}>
+#DenseVector = #sparse_tensor.encoding<{dimLevelType = ["dense"], crdWidth=32}>
 
-func.func @invalid_pack_dense(%data: tensor<6xf64>, %index: tensor<6x1xi32>)
+func.func @invalid_pack_dense(%values: tensor<6xf64>, %coordinates: tensor<6x1xi32>)
                             -> tensor<100xf64, #DenseVector> {
-  // expected-error@+1 {{must operate on a COO tensor}}
-  %0 = sparse_tensor.pack %data, %index : tensor<6xf64>, tensor<6x1xi32>
-                                       to tensor<100xf64, #DenseVector>
+  // expected-error@+1 {{the sparse-tensor must have a COO type}}
+  %0 = sparse_tensor.pack %values, %coordinates
+     : tensor<6xf64>, tensor<6x1xi32> to tensor<100xf64, #DenseVector>
   return %0 : tensor<100xf64, #DenseVector>
 }
 
 // -----
 
-#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], indexBitWidth=32}>
+#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], crdWidth=32}>
 
-func.func @invalid_pack_data(%data: tensor<6x1xf64>, %index: tensor<6x1xi32>)
+func.func @invalid_pack_data(%values: tensor<6x1xf64>, %coordinates: tensor<6x1xi32>)
                             -> tensor<100xf64, #SparseVector> {
   // expected-error@+1 {{'sparse_tensor.pack' op operand #0 must be 1D tensor of any type values}}
-  %0 = sparse_tensor.pack %data, %index : tensor<6x1xf64>, tensor<6x1xi32>
-                                       to tensor<100xf64, #SparseVector>
+  %0 = sparse_tensor.pack %values, %coordinates
+     : tensor<6x1xf64>, tensor<6x1xi32> to tensor<100xf64, #SparseVector>
   return %0 : tensor<100xf64, #SparseVector>
 }
 
 // -----
 
-#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], indexBitWidth=32}>
+#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], crdWidth=32}>
 
-func.func @invalid_pack_type(%data: tensor<6xf64>, %index: tensor<6x1xi32>)
+func.func @invalid_pack_type(%values: tensor<6xf64>, %coordinates: tensor<6x1xi32>)
                             -> tensor<100xf32, #SparseVector> {
-  // expected-error@+1 {{unmatched type between input and output}}
-  %0 = sparse_tensor.pack %data, %index : tensor<6xf64>, tensor<6x1xi32>
-                                       to tensor<100xf32, #SparseVector>
+  // expected-error@+1 {{input/output element-types don't match}}
+  %0 = sparse_tensor.pack %values, %coordinates
+     : tensor<6xf64>, tensor<6x1xi32> to tensor<100xf32, #SparseVector>
   return %0 : tensor<100xf32, #SparseVector>
 }
 
 // -----
 
-#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], indexBitWidth=32}>
+#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], crdWidth=32}>
 
-func.func @invalid_pack_type(%data: tensor<5xf64>, %index: tensor<6x1xi32>)
+func.func @invalid_pack_type(%values: tensor<5xf64>, %coordinates: tensor<6x1xi32>)
                             -> tensor<100xf64, #SparseVector> {
-  // expected-error@+1 {{unmatched number of elements in data and indices}}
-  %0 = sparse_tensor.pack %data, %index : tensor<5xf64>, tensor<6x1xi32>
-                                       to tensor<100xf64, #SparseVector>
+  // expected-error@+1 {{values/coordinates number-of-elements don't match}}
+  %0 = sparse_tensor.pack %values, %coordinates
+     : tensor<5xf64>, tensor<6x1xi32> to tensor<100xf64, #SparseVector>
   return %0 : tensor<100xf64, #SparseVector>
 }
 
 // -----
 
-#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], indexBitWidth=32}>
+#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], crdWidth=32}>
 
-func.func @invalid_pack_type(%data: tensor<6xf64>, %index: tensor<6x2xi32>)
+func.func @invalid_pack_type(%values: tensor<6xf64>, %coordinates: tensor<6x2xi32>)
                             -> tensor<100xf64, #SparseVector> {
-  // expected-error@+1 {{unmatched rank between input and output}}
-  %0 = sparse_tensor.pack %data, %index : tensor<6xf64>, tensor<6x2xi32>
-                                       to tensor<100xf64, #SparseVector>
+  // expected-error@+1 {{input/output level-ranks don't match}}
+  %0 = sparse_tensor.pack %values, %coordinates
+     : tensor<6xf64>, tensor<6x2xi32> to tensor<100xf64, #SparseVector>
   return %0 : tensor<100xf64, #SparseVector>
 }
 
 // -----
 
-#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], indexBitWidth=32}>
+#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], crdWidth=32}>
 
 func.func @invalid_unpack_type(%sp: tensor<100xf32, #SparseVector>)
                             -> (tensor<6xf64>, tensor<6x1xi32>, i32) {
-  // expected-error@+1 {{unmatched type between input and output}}
-  %d, %i, %n = sparse_tensor.unpack %sp :  tensor<100xf32, #SparseVector>
-                                        to tensor<6xf64>, tensor<6x1xi32>, i32
-  return %d, %i, %n : tensor<6xf64>, tensor<6x1xi32>, i32
+  // expected-error@+1 {{input/output element-types don't match}}
+  %values, %coordinates, %nse = sparse_tensor.unpack %sp
+     : tensor<100xf32, #SparseVector> to tensor<6xf64>, tensor<6x1xi32>, i32
+  return %values, %coordinates, %nse : tensor<6xf64>, tensor<6x1xi32>, i32
 }
 
 // -----
 
-#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], indexBitWidth=32}>
+#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], crdWidth=32}>
 
 func.func @invalid_unpack_type(%sp: tensor<100xf32, #SparseVector>)
                             -> (tensor<5xf32>, tensor<6x1xi32>, i32) {
-  // expected-error@+1 {{unmatched number of elements in data and indices}}
-  %d, %i, %n = sparse_tensor.unpack %sp :  tensor<100xf32, #SparseVector>
-                                        to tensor<5xf32>, tensor<6x1xi32>, i32
-  return %d, %i, %n : tensor<5xf32>, tensor<6x1xi32>, i32
+  // expected-error@+1 {{values/coordinates number-of-elements don't match}}
+  %values, %coordinates, %nse = sparse_tensor.unpack %sp
+     : tensor<100xf32, #SparseVector> to tensor<5xf32>, tensor<6x1xi32>, i32
+  return %values, %coordinates, %nse : tensor<5xf32>, tensor<6x1xi32>, i32
 }
 
 // -----
 
-#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], indexBitWidth=32}>
+#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], crdWidth=32}>
 
 func.func @invalid_unpack_type(%sp: tensor<100xf32, #SparseVector>)
                             -> (tensor<6xf32>, tensor<6x2xi32>, i32) {
-  // expected-error@+1 {{unmatched rank between input and output}}
-  %d, %i, %n = sparse_tensor.unpack %sp :  tensor<100xf32, #SparseVector>
-                                        to tensor<6xf32>, tensor<6x2xi32>, i32
-  return %d, %i, %n : tensor<6xf32>, tensor<6x2xi32>, i32
+  // expected-error@+1 {{input/output level-ranks don't match}}
+  %values, %coordinates, %nse = sparse_tensor.unpack %sp
+     : tensor<100xf32, #SparseVector> to tensor<6xf32>, tensor<6x2xi32>, i32
+  return %values, %coordinates, %nse : tensor<6xf32>, tensor<6x2xi32>, i32
 }
 
 // -----
 
-func.func @invalid_pointers_dense(%arg0: tensor<128xf64>) -> memref<?xindex> {
-  // expected-error@+1 {{'sparse_tensor.pointers' op operand #0 must be sparse tensor of any type values, but got 'tensor<128xf64>'}}
-  %0 = sparse_tensor.pointers %arg0 { dimension = 0 : index } : tensor<128xf64> to memref<?xindex>
+func.func @invalid_positions_dense(%arg0: tensor<128xf64>) -> memref<?xindex> {
+  // expected-error@+1 {{'sparse_tensor.positions' op operand #0 must be sparse tensor of any type values, but got 'tensor<128xf64>'}}
+  %0 = sparse_tensor.positions %arg0 { level = 0 : index } : tensor<128xf64> to memref<?xindex>
   return %0 : memref<?xindex>
 }
 
 // -----
 
-func.func @invalid_pointers_unranked(%arg0: tensor<*xf64>) -> memref<?xindex> {
-  // expected-error@+1 {{'sparse_tensor.pointers' op operand #0 must be sparse tensor of any type values, but got 'tensor<*xf64>'}}
-  %0 = sparse_tensor.pointers %arg0 { dimension = 0 : index } : tensor<*xf64> to memref<?xindex>
+func.func @invalid_positions_unranked(%arg0: tensor<*xf64>) -> memref<?xindex> {
+  // expected-error@+1 {{'sparse_tensor.positions' op operand #0 must be sparse tensor of any type values, but got 'tensor<*xf64>'}}
+  %0 = sparse_tensor.positions %arg0 { level = 0 : index } : tensor<*xf64> to memref<?xindex>
   return %0 : memref<?xindex>
 }
 
 // -----
 
-#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], pointerBitWidth=32}>
+#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], posWidth=32}>
 
-func.func @mismatch_pointers_types(%arg0: tensor<128xf64, #SparseVector>) -> memref<?xindex> {
-  // expected-error@+1 {{unexpected type for pointers}}
-  %0 = sparse_tensor.pointers %arg0 { dimension = 0 : index } : tensor<128xf64, #SparseVector> to memref<?xindex>
+func.func @mismatch_positions_types(%arg0: tensor<128xf64, #SparseVector>) -> memref<?xindex> {
+  // expected-error@+1 {{unexpected type for positions}}
+  %0 = sparse_tensor.positions %arg0 { level = 0 : index } : tensor<128xf64, #SparseVector> to memref<?xindex>
   return %0 : memref<?xindex>
 }
 
@@ -144,25 +144,25 @@ func.func @mismatch_pointers_types(%arg0: tensor<128xf64, #SparseVector>) -> mem
 
 #SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"]}>
 
-func.func @pointers_oob(%arg0: tensor<128xf64, #SparseVector>) -> memref<?xindex> {
-  // expected-error@+1 {{requested pointers dimension out of bounds}}
-  %0 = sparse_tensor.pointers %arg0 { dimension = 1 : index } : tensor<128xf64, #SparseVector> to memref<?xindex>
+func.func @positions_oob(%arg0: tensor<128xf64, #SparseVector>) -> memref<?xindex> {
+  // expected-error@+1 {{requested level is out of bounds}}
+  %0 = sparse_tensor.positions %arg0 { level = 1 : index } : tensor<128xf64, #SparseVector> to memref<?xindex>
   return %0 : memref<?xindex>
 }
 
 // -----
 
 func.func @invalid_indices_dense(%arg0: tensor<10x10xi32>) -> memref<?xindex> {
-  // expected-error@+1 {{'sparse_tensor.indices' op operand #0 must be sparse tensor of any type values, but got 'tensor<10x10xi32>'}}
-  %0 = sparse_tensor.indices %arg0 { dimension = 1 : index } : tensor<10x10xi32> to memref<?xindex>
+  // expected-error@+1 {{'sparse_tensor.coordinates' op operand #0 must be sparse tensor of any type values, but got 'tensor<10x10xi32>'}}
+  %0 = sparse_tensor.coordinates %arg0 { level = 1 : index } : tensor<10x10xi32> to memref<?xindex>
   return %0 : memref<?xindex>
 }
 
 // -----
 
 func.func @invalid_indices_unranked(%arg0: tensor<*xf64>) -> memref<?xindex> {
-  // expected-error@+1 {{'sparse_tensor.indices' op operand #0 must be sparse tensor of any type values, but got 'tensor<*xf64>'}}
-  %0 = sparse_tensor.indices %arg0 { dimension = 0 : index } : tensor<*xf64> to memref<?xindex>
+  // expected-error@+1 {{'sparse_tensor.coordinates' op operand #0 must be sparse tensor of any type values, but got 'tensor<*xf64>'}}
+  %0 = sparse_tensor.coordinates %arg0 { level = 0 : index } : tensor<*xf64> to memref<?xindex>
   return %0 : memref<?xindex>
 }
 
@@ -171,8 +171,8 @@ func.func @invalid_indices_unranked(%arg0: tensor<*xf64>) -> memref<?xindex> {
 #SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"]}>
 
 func.func @mismatch_indices_types(%arg0: tensor<?xf64, #SparseVector>) -> memref<?xi32> {
-  // expected-error@+1 {{unexpected type for indices}}
-  %0 = sparse_tensor.indices %arg0 { dimension = 0 : index } : tensor<?xf64, #SparseVector> to memref<?xi32>
+  // expected-error@+1 {{unexpected type for coordinates}}
+  %0 = sparse_tensor.coordinates %arg0 { level = 0 : index } : tensor<?xf64, #SparseVector> to memref<?xi32>
   return %0 : memref<?xi32>
 }
 
@@ -181,8 +181,8 @@ func.func @mismatch_indices_types(%arg0: tensor<?xf64, #SparseVector>) -> memref
 #SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"]}>
 
 func.func @indices_oob(%arg0: tensor<128xf64, #SparseVector>) -> memref<?xindex> {
-  // expected-error@+1 {{requested indices dimension out of bounds}}
-  %0 = sparse_tensor.indices %arg0 { dimension = 1 : index } : tensor<128xf64, #SparseVector> to memref<?xindex>
+  // expected-error@+1 {{requested level is out of bounds}}
+  %0 = sparse_tensor.coordinates %arg0 { level = 1 : index } : tensor<128xf64, #SparseVector> to memref<?xindex>
   return %0 : memref<?xindex>
 }
 
@@ -200,7 +200,7 @@ func.func @invalid_values_dense(%arg0: tensor<1024xf32>) -> memref<?xf32> {
 
 func.func @indices_buffer_noncoo(%arg0: tensor<128xf64, #SparseVector>) -> memref<?xindex> {
   // expected-error@+1 {{expected sparse tensor with a COO region}}
-  %0 = sparse_tensor.indices_buffer %arg0 : tensor<128xf64, #SparseVector> to memref<?xindex>
+  %0 = sparse_tensor.coordinates_buffer %arg0 : tensor<128xf64, #SparseVector> to memref<?xindex>
   return %0 : memref<?xindex>
 }
 
@@ -208,7 +208,7 @@ func.func @indices_buffer_noncoo(%arg0: tensor<128xf64, #SparseVector>) -> memre
 
 func.func @indices_buffer_dense(%arg0: tensor<1024xf32>) -> memref<?xindex> {
   // expected-error@+1 {{must be sparse tensor of any type values}}
-  %0 = sparse_tensor.indices_buffer %arg0 : tensor<1024xf32> to memref<?xindex>
+  %0 = sparse_tensor.coordinates_buffer %arg0 : tensor<1024xf32> to memref<?xindex>
   return %0 : memref<?xindex>
 }
 
@@ -263,9 +263,20 @@ func.func @sparse_get_md(%arg0: !sparse_tensor.storage_specifier<#SparseVector>)
 
 #SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"]}>
 
+func.func @sparse_get_md(%arg0: !sparse_tensor.storage_specifier<#SparseVector>) -> i64 {
+  // expected-error@+1 {{requested slice data on non-slice tensor}}
+  %0 = sparse_tensor.storage_specifier.get %arg0 dim_offset at 0
+       : !sparse_tensor.storage_specifier<#SparseVector>
+  return %0 : index
+}
+
+// -----
+
+#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"]}>
+
 func.func @sparse_get_md(%arg0: !sparse_tensor.storage_specifier<#SparseVector>) -> index {
   // expected-error@+1 {{missing level argument}}
-  %0 = sparse_tensor.storage_specifier.get %arg0 idx_mem_sz
+  %0 = sparse_tensor.storage_specifier.get %arg0 crd_mem_sz
        : !sparse_tensor.storage_specifier<#SparseVector>
   return %0 : index
 }
@@ -275,8 +286,8 @@ func.func @sparse_get_md(%arg0: !sparse_tensor.storage_specifier<#SparseVector>)
 #SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"]}>
 
 func.func @sparse_get_md(%arg0: !sparse_tensor.storage_specifier<#SparseVector>) -> index {
-  // expected-error@+1 {{requested level out of bound}}
-  %0 = sparse_tensor.storage_specifier.get %arg0 dim_sz at 1
+  // expected-error@+1 {{requested level is out of bounds}}
+  %0 = sparse_tensor.storage_specifier.get %arg0 lvl_sz at 1
        : !sparse_tensor.storage_specifier<#SparseVector>
   return %0 : index
 }
@@ -286,8 +297,8 @@ func.func @sparse_get_md(%arg0: !sparse_tensor.storage_specifier<#SparseVector>)
 #COO = #sparse_tensor.encoding<{dimLevelType = ["compressed-nu", "singleton"]}>
 
 func.func @sparse_get_md(%arg0: !sparse_tensor.storage_specifier<#COO>) -> index {
-  // expected-error@+1 {{requested pointer memory size on a singleton level}}
-  %0 = sparse_tensor.storage_specifier.get %arg0 ptr_mem_sz at 1
+  // expected-error@+1 {{requested position memory size on a singleton level}}
+  %0 = sparse_tensor.storage_specifier.get %arg0 pos_mem_sz at 1
        : !sparse_tensor.storage_specifier<#COO>
   return %0 : index
 }
@@ -313,7 +324,7 @@ func.func @sparse_unannotated_insert(%arg0: tensor<128xf64>, %arg1: index, %arg2
 #CSR = #sparse_tensor.encoding<{dimLevelType = ["dense", "compressed"]}>
 
 func.func @sparse_wrong_arity_insert(%arg0: tensor<128x64xf64, #CSR>, %arg1: index, %arg2: f64) {
-  // expected-error@+1 {{'sparse_tensor.insert' op incorrect number of indices}}
+  // expected-error@+1 {{'sparse_tensor.insert' op incorrect number of coordinates}}
   sparse_tensor.insert %arg2 into %arg0[%arg1] : tensor<128x64xf64, #CSR>
   return
 }
@@ -368,7 +379,7 @@ func.func @sparse_wrong_arity_compression(%arg0: memref<?xf64>,
                                           %arg3: index,
                                           %arg4: tensor<8x8xf64, #CSR>,
                                           %arg5: index) {
-  // expected-error@+1 {{'sparse_tensor.compress' op incorrect number of indices}}
+  // expected-error@+1 {{'sparse_tensor.compress' op incorrect number of coordinates}}
   sparse_tensor.compress %arg0, %arg1, %arg2, %arg3 into %arg4[%arg5,%arg5]
     : memref<?xf64>, memref<?xi1>, memref<?xindex>, tensor<8x8xf64, #CSR>
   return
diff --git a/mlir/test/Dialect/SparseTensor/invalid_encoding.mlir b/mlir/test/Dialect/SparseTensor/invalid_encoding.mlir
index fa6466094b972..42518bf49f84d 100644
--- a/mlir/test/Dialect/SparseTensor/invalid_encoding.mlir
+++ b/mlir/test/Dialect/SparseTensor/invalid_encoding.mlir
@@ -42,22 +42,22 @@ func.func private @tensor_no_permutation(%arg0: tensor<16x32xf32, #a>) -> ()
 
 // -----
 
-#a = #sparse_tensor.encoding<{pointerBitWidth = "x"}> // expected-error {{expected an integral pointer bitwidth}}
+#a = #sparse_tensor.encoding<{posWidth = "x"}> // expected-error {{expected an integral position bitwidth}}
 func.func private @tensor_no_int_ptr(%arg0: tensor<16x32xf32, #a>) -> ()
 
 // -----
 
-#a = #sparse_tensor.encoding<{pointerBitWidth = 42}> // expected-error {{unexpected pointer bitwidth: 42}}
+#a = #sparse_tensor.encoding<{posWidth = 42}> // expected-error {{unexpected position bitwidth: 42}}
 func.func private @tensor_invalid_int_ptr(%arg0: tensor<16x32xf32, #a>) -> ()
 
 // -----
 
-#a = #sparse_tensor.encoding<{indexBitWidth = "not really"}> // expected-error {{expected an integral index bitwidth}}
+#a = #sparse_tensor.encoding<{crdWidth = "not really"}> // expected-error {{expected an integral index bitwidth}}
 func.func private @tensor_no_int_index(%arg0: tensor<16x32xf32, #a>) -> ()
 
 // -----
 
-#a = #sparse_tensor.encoding<{indexBitWidth = 128}> // expected-error {{unexpected index bitwidth: 128}}
+#a = #sparse_tensor.encoding<{crdWidth = 128}> // expected-error {{unexpected coordinate bitwidth: 128}}
 func.func private @tensor_invalid_int_index(%arg0: tensor<16x32xf32, #a>) -> ()
 
 // -----
diff --git a/mlir/test/Dialect/SparseTensor/roundtrip.mlir b/mlir/test/Dialect/SparseTensor/roundtrip.mlir
index 608b6c80e24f3..ff622a4bb408f 100644
--- a/mlir/test/Dialect/SparseTensor/roundtrip.mlir
+++ b/mlir/test/Dialect/SparseTensor/roundtrip.mlir
@@ -13,7 +13,7 @@ func.func @sparse_new(%arg0: !llvm.ptr<i8>) -> tensor<128xf64, #SparseVector> {
 
 // -----
 
-#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], indexBitWidth=32}>
+#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], crdWidth=32}>
 
 // CHECK-LABEL: func @sparse_pack(
 // CHECK-SAME: %[[D:.*]]: tensor<6xf64>,
@@ -29,7 +29,7 @@ func.func @sparse_pack(%data: tensor<6xf64>, %index: tensor<6x1xi32>)
 
 // -----
 
-#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], indexBitWidth=32}>
+#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"], crdWidth=32}>
 
 // CHECK-LABEL: func @sparse_unpack(
 //  CHECK-SAME: %[[T:.*]]: tensor<100xf64, #
@@ -85,12 +85,12 @@ func.func @sparse_convert_3d_from_sparse(%arg0: tensor<8x8x8xf64, #SparseTensor>
 
 #SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"]}>
 
-// CHECK-LABEL: func @sparse_pointers(
+// CHECK-LABEL: func @sparse_positions(
 //  CHECK-SAME: %[[A:.*]]: tensor<128xf64, #{{.*}}>)
-//       CHECK: %[[T:.*]] = sparse_tensor.pointers %[[A]] {dimension = 0 : index} : tensor<128xf64, #{{.*}}> to memref<?xindex>
+//       CHECK: %[[T:.*]] = sparse_tensor.positions %[[A]] {level = 0 : index} : tensor<128xf64, #{{.*}}> to memref<?xindex>
 //       CHECK: return %[[T]] : memref<?xindex>
-func.func @sparse_pointers(%arg0: tensor<128xf64, #SparseVector>) -> memref<?xindex> {
-  %0 = sparse_tensor.pointers %arg0 {dimension = 0 : index} : tensor<128xf64, #SparseVector> to memref<?xindex>
+func.func @sparse_positions(%arg0: tensor<128xf64, #SparseVector>) -> memref<?xindex> {
+  %0 = sparse_tensor.positions %arg0 {level = 0 : index} : tensor<128xf64, #SparseVector> to memref<?xindex>
   return %0 : memref<?xindex>
 }
 
@@ -100,10 +100,10 @@ func.func @sparse_pointers(%arg0: tensor<128xf64, #SparseVector>) -> memref<?xin
 
 // CHECK-LABEL: func @sparse_indices_buffer(
 //  CHECK-SAME: %[[A:.*]]: tensor<?x?xf64, #{{.*}}>)
-//       CHECK: %[[T:.*]] = sparse_tensor.indices_buffer %[[A]] : tensor<?x?xf64, #{{.*}}> to memref<?xindex>
+//       CHECK: %[[T:.*]] = sparse_tensor.coordinates_buffer %[[A]] : tensor<?x?xf64, #{{.*}}> to memref<?xindex>
 //       CHECK: return %[[T]] : memref<?xindex>
 func.func @sparse_indices_buffer(%arg0: tensor<?x?xf64, #COO>) -> memref<?xindex> {
-  %0 = sparse_tensor.indices_buffer %arg0 : tensor<?x?xf64, #COO> to memref<?xindex>
+  %0 = sparse_tensor.coordinates_buffer %arg0 : tensor<?x?xf64, #COO> to memref<?xindex>
   return %0 : memref<?xindex>
 }
 
@@ -113,10 +113,10 @@ func.func @sparse_indices_buffer(%arg0: tensor<?x?xf64, #COO>) -> memref<?xindex
 
 // CHECK-LABEL: func @sparse_indices(
 //  CHECK-SAME: %[[A:.*]]: tensor<128xf64, #{{.*}}>)
-//       CHECK: %[[T:.*]] = sparse_tensor.indices %[[A]] {dimension = 0 : index} : tensor<128xf64, #{{.*}}> to memref<?xindex>
+//       CHECK: %[[T:.*]] = sparse_tensor.coordinates %[[A]] {level = 0 : index} : tensor<128xf64, #{{.*}}> to memref<?xindex>
 //       CHECK: return %[[T]] : memref<?xindex>
 func.func @sparse_indices(%arg0: tensor<128xf64, #SparseVector>) -> memref<?xindex> {
-  %0 = sparse_tensor.indices %arg0 {dimension = 0 : index} : tensor<128xf64, #SparseVector> to memref<?xindex>
+  %0 = sparse_tensor.coordinates %arg0 {level = 0 : index} : tensor<128xf64, #SparseVector> to memref<?xindex>
   return %0 : memref<?xindex>
 }
 
@@ -179,18 +179,72 @@ func.func @sparse_metadata_init() -> !sparse_tensor.storage_specifier<#SparseVec
 
 // -----
 
+#SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"]}>
+#SparseVector_Slice = #sparse_tensor.encoding<{
+  dimLevelType = ["compressed"],
+  slice = [ (?, ?, ?) ]
+}>
+
+// CHECK-LABEL: func @sparse_metadata_init(
+//  CHECK-SAME: %[[A:.*]]: !sparse_tensor.storage_specifier<#{{.*}}>
+//       CHECK: %[[T:.*]] = sparse_tensor.storage_specifier.init with %[[A]] :
+//       CHECK: return %[[T]] : !sparse_tensor.storage_specifier<#{{.*}}>
+func.func @sparse_metadata_init(%src : !sparse_tensor.storage_specifier<#SparseVector>)
+                                    -> !sparse_tensor.storage_specifier<#SparseVector_Slice> {
+  %0 = sparse_tensor.storage_specifier.init with %src : from !sparse_tensor.storage_specifier<#SparseVector>
+                                                          to !sparse_tensor.storage_specifier<#SparseVector_Slice>
+  return %0 : !sparse_tensor.storage_specifier<#SparseVector_Slice>
+}
+
+// -----
+
 #SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"]}>
 
 // CHECK-LABEL: func @sparse_get_md(
 //  CHECK-SAME: %[[A:.*]]: !sparse_tensor.storage_specifier<#{{.*}}>
-//       CHECK: %[[T:.*]] = sparse_tensor.storage_specifier.get %[[A]] dim_sz at 0
+//       CHECK: %[[T:.*]] = sparse_tensor.storage_specifier.get %[[A]] lvl_sz at 0
 //       CHECK: return %[[T]] : index
 func.func @sparse_get_md(%arg0: !sparse_tensor.storage_specifier<#SparseVector>) -> index {
-  %0 = sparse_tensor.storage_specifier.get %arg0 dim_sz at 0
+  %0 = sparse_tensor.storage_specifier.get %arg0 lvl_sz at 0
        : !sparse_tensor.storage_specifier<#SparseVector>
   return %0 : index
 }
 
+// -----
+
+#SparseVector_Slice = #sparse_tensor.encoding<{
+  dimLevelType = ["compressed"],
+  slice = [ (?, ?, ?) ]
+}>
+
+// CHECK-LABEL: func @sparse_get_md(
+//  CHECK-SAME: %[[A:.*]]: !sparse_tensor.storage_specifier<#{{.*}}>
+//       CHECK: %[[T:.*]] = sparse_tensor.storage_specifier.get %[[A]] dim_offset at 0
+//       CHECK: return %[[T]] : index
+func.func @sparse_get_md(%arg0: !sparse_tensor.storage_specifier<#SparseVector_Slice>) -> index {
+  %0 = sparse_tensor.storage_specifier.get %arg0 dim_offset at 0
+       : !sparse_tensor.storage_specifier<#SparseVector_Slice>
+  return %0 : index
+}
+
+// -----
+
+#SparseVector = #sparse_tensor.encoding<{
+  dimLevelType = ["compressed"],
+  slice = [ (?, ?, ?) ]
+}>
+
+// CHECK-LABEL: func @sparse_get_md(
+//  CHECK-SAME: %[[A:.*]]: !sparse_tensor.storage_specifier<#{{.*}}>
+//       CHECK: %[[T:.*]] = sparse_tensor.storage_specifier.get %[[A]] dim_stride at 0
+//       CHECK: return %[[T]] : index
+func.func @sparse_get_md(%arg0: !sparse_tensor.storage_specifier<#SparseVector>) -> index {
+  %0 = sparse_tensor.storage_specifier.get %arg0 dim_stride at 0
+       : !sparse_tensor.storage_specifier<#SparseVector>
+  return %0 : index
+}
+
+
 // -----
 
 #SparseVector = #sparse_tensor.encoding<{dimLevelType = ["compressed"]}>
@@ -198,11 +252,11 @@ func.func @sparse_get_md(%arg0: !sparse_tensor.storage_specifier<#SparseVector>)
 // CHECK-LABEL: func @sparse_set_md(
 //  CHECK-SAME: %[[A:.*]]: !sparse_tensor.storage_specifier<#{{.*}}>,
 //  CHECK-SAME: %[[I:.*]]: index)
-//       CHECK: %[[T:.*]] = sparse_tensor.storage_specifier.set %[[A]] dim_sz at 0 with %[[I]]
+//       CHECK: %[[T:.*]] = sparse_tensor.storage_specifier.set %[[A]] lvl_sz at 0 with %[[I]]
 //       CHECK: return %[[T]] : !sparse_tensor.storage_specifier<#{{.*}}>
 func.func @sparse_set_md(%arg0: !sparse_tensor.storage_specifier<#SparseVector>, %arg1: index)
           -> !sparse_tensor.storage_specifier<#SparseVector> {
-  %0 = sparse_tensor.storage_specifier.set %arg0 dim_sz at 0 with %arg1
+  %0 = sparse_tensor.storage_specifier.set %arg0 lvl_sz at 0 with %arg1
        : !sparse_tensor.storage_specifier<#SparseVector>
   return %0 : !sparse_tensor.storage_specifier<#SparseVector>
 }
diff --git a/mlir/test/Dialect/SparseTensor/roundtrip_encoding.mlir b/mlir/test/Dialect/SparseTensor/roundtrip_encoding.mlir
index b7bc7c619aa60..d2d6fab60f124 100644
--- a/mlir/test/Dialect/SparseTensor/roundtrip_encoding.mlir
+++ b/mlir/test/Dialect/SparseTensor/roundtrip_encoding.mlir
@@ -9,12 +9,12 @@ func.func private @sparse_1d_tensor(tensor<32xf64, #sparse_tensor.encoding<{ dim
 #CSR = #sparse_tensor.encoding<{
   dimLevelType = [ "dense", "compressed" ],
   dimOrdering = affine_map<(i,j) -> (i,j)>,
-  pointerBitWidth = 64,
-  indexBitWidth = 64
+  posWidth = 64,
+  crdWidth = 64
 }>
 
 // CHECK-LABEL: func private @sparse_csr(
-// CHECK-SAME: tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ], pointerBitWidth = 64, indexBitWidth = 64 }>>)
+// CHECK-SAME: tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ], posWidth = 64, crdWidth = 64 }>>)
 func.func private @sparse_csr(tensor<?x?xf32, #CSR>)
 
 // -----
@@ -22,8 +22,8 @@ func.func private @sparse_csr(tensor<?x?xf32, #CSR>)
 #CSC = #sparse_tensor.encoding<{
   dimLevelType = [ "dense", "compressed" ],
   dimOrdering = affine_map<(i,j) -> (j,i)>,
-  pointerBitWidth = 0,
-  indexBitWidth = 0
+  posWidth = 0,
+  crdWidth = 0
 }>
 
 // CHECK-LABEL: func private @sparse_csc(
@@ -35,12 +35,12 @@ func.func private @sparse_csc(tensor<?x?xf32, #CSC>)
 #DCSC = #sparse_tensor.encoding<{
   dimLevelType = [ "compressed", "compressed" ],
   dimOrdering = affine_map<(i,j) -> (j,i)>,
-  pointerBitWidth = 0,
-  indexBitWidth = 64
+  posWidth = 0,
+  crdWidth = 64
 }>
 
 // CHECK-LABEL: func private @sparse_dcsc(
-// CHECK-SAME: tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)>, indexBitWidth = 64 }>>)
+// CHECK-SAME: tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)>, crdWidth = 64 }>>)
 func.func private @sparse_dcsc(tensor<?x?xf32, #DCSC>)
 
 // -----
diff --git a/mlir/test/Dialect/SparseTensor/sorted_coo.mlir b/mlir/test/Dialect/SparseTensor/sorted_coo.mlir
index c30525f224a68..7c6884f558822 100644
--- a/mlir/test/Dialect/SparseTensor/sorted_coo.mlir
+++ b/mlir/test/Dialect/SparseTensor/sorted_coo.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt %s -sparsification | FileCheck %s
+// RUN: mlir-opt %s -sparsification --canonicalize | FileCheck %s
 
 #SortedCOO = #sparse_tensor.encoding<{
   dimLevelType = [ "compressed-nu", "singleton" ]
@@ -36,23 +36,48 @@
 // Kernels that operate on SortedCOO format.
 //
 
-// CHECK-LABEL: func.func @sparse_scale(
-// CHECK-SAME:    %[[VAL_0:.*]]: tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>>) -> tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> {
-// CHECK-DAG:     %[[VAL_1:.*]] = arith.constant 0 : index
-// CHECK-DAG:     %[[VAL_2:.*]] = arith.constant 1 : index
-// CHECK-DAG:     %[[VAL_3:.*]] = arith.constant 2.000000e+00 : f32
-// CHECK-DAG:     %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex>
-// CHECK-DAG:     %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xf32>
-// CHECK:         %[[VAL_6:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_1]]] : memref<?xindex>
-// CHECK:         %[[VAL_7:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_2]]] : memref<?xindex>
-// CHECK:         scf.for %[[VAL_8:.*]] = %[[VAL_6]] to %[[VAL_7]] step %[[VAL_2]] {
-// CHECK:           %[[VAL_9:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_8]]] : memref<?xf32>
-// CHECK:           %[[VAL_10:.*]] = arith.mulf %[[VAL_9]], %[[VAL_3]] : f32
-// CHECK:           memref.store %[[VAL_10]], %[[VAL_5]]{{\[}}%[[VAL_8]]] : memref<?xf32>
+// CHECK-LABEL:   func.func @sparse_scale(
+// CHECK-SAME:      %[[VAL_0:.*]]: tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>>) -> tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> {
+// CHECK-DAG:       %[[VAL_1:.*]] = arith.constant false
+// CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 0 : index
+// CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 1 : index
+// CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 2.000000e+00 : f32
+// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex, strided<[?], offset: ?>>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xf32>
+// CHECK-DAG:       %[[VAL_8:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_2]]] : memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_3]]] : memref<?xindex>
+// CHECK:           %[[VAL_10:.*]] = scf.while (%[[VAL_11:.*]] = %[[VAL_8]]) : (index) -> index {
+// CHECK:             %[[VAL_12:.*]] = arith.cmpi ult, %[[VAL_11]], %[[VAL_9]] : index
+// CHECK:             scf.condition(%[[VAL_12]]) %[[VAL_11]] : index
+// CHECK:           } do {
+// CHECK:           ^bb0(%[[VAL_13:.*]]: index):
+// CHECK:             %[[VAL_14:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_13]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:             %[[VAL_15:.*]] = scf.while (%[[VAL_16:.*]] = %[[VAL_13]]) : (index) -> index {
+// CHECK:               %[[VAL_17:.*]] = arith.cmpi ult, %[[VAL_16]], %[[VAL_9]] : index
+// CHECK:               %[[VAL_18:.*]] = scf.if %[[VAL_17]] -> (i1) {
+// CHECK:                 %[[VAL_19:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_16]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:                 %[[VAL_20:.*]] = arith.cmpi eq, %[[VAL_19]], %[[VAL_14]] : index
+// CHECK:                 scf.yield %[[VAL_20]] : i1
+// CHECK:               } else {
+// CHECK:                 scf.yield %[[VAL_1]] : i1
+// CHECK:               }
+// CHECK:               scf.condition(%[[VAL_21:.*]]) %[[VAL_16]] : index
+// CHECK:             } do {
+// CHECK:             ^bb0(%[[VAL_22:.*]]: index):
+// CHECK:               %[[VAL_23:.*]] = arith.addi %[[VAL_22]], %[[VAL_3]] : index
+// CHECK:               scf.yield %[[VAL_23]] : index
+// CHECK:             }
+// CHECK:             scf.for %[[VAL_24:.*]] = %[[VAL_13]] to %[[VAL_25:.*]] step %[[VAL_3]] {
+// CHECK:               %[[VAL_26:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_24]]] : memref<?xf32>
+// CHECK:               %[[VAL_27:.*]] = arith.mulf %[[VAL_26]], %[[VAL_4]] : f32
+// CHECK:               memref.store %[[VAL_27]], %[[VAL_7]]{{\[}}%[[VAL_24]]] : memref<?xf32>
+// CHECK:             } {"Emitted from" = "linalg.generic"}
+// CHECK:             scf.yield %[[VAL_28:.*]] : index
+// CHECK:           } attributes {"Emitted from" = "linalg.generic"}
+// CHECK:           %[[VAL_29:.*]] = sparse_tensor.load %[[VAL_0]] : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>>
+// CHECK:           return %[[VAL_29]] : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>>
 // CHECK:         }
-// CHECK:         %[[VAL_11:.*]] = sparse_tensor.load %[[VAL_0]] : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>>
-// CHECK:         return %[[VAL_11]] : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>>
-// CHECK:       }
 func.func @sparse_scale(%argx: tensor<?x?xf32, #SortedCOO>) -> tensor<?x?xf32, #SortedCOO> {
   %c = arith.constant 2.0 : f32
   %0 = linalg.generic #trait_scale
@@ -64,33 +89,57 @@ func.func @sparse_scale(%argx: tensor<?x?xf32, #SortedCOO>) -> tensor<?x?xf32, #
   return %0 : tensor<?x?xf32, #SortedCOO>
 }
 
-// CHECK-LABEL: func.func @matvec(
-// CHECK-SAME:    %[[VAL_0:.*]]: tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>>,
-// CHECK-SAME:    %[[VAL_1:.*]]: tensor<64xf64>,
-// CHECK-SAME:    %[[VAL_2:.*]]: tensor<32xf64>) -> tensor<32xf64> {
-// CHECK-DAG:     %[[VAL_3:.*]] = arith.constant 0 : index
-// CHECK-DAG:     %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK-DAG:     %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex>
-// CHECK-DAG:     %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex, strided<[?], offset: ?>>
-// CHECK-DAG:     %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex, strided<[?], offset: ?>>
-// CHECK-DAG:     %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xf64>
-// CHECK-DAG:     %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_1]] : memref<64xf64>
-// CHECK-DAG:     %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xf64>
-// CHECK-DAG:     %[[VAL_11:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_3]]] : memref<?xindex>
-// CHECK-DAG:     %[[VAL_12:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_4]]] : memref<?xindex>
-// CHECK:         scf.for %[[VAL_13:.*]] = %[[VAL_11]] to %[[VAL_12]] step %[[VAL_4]] {
-// CHECK:           %[[VAL_14:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_13]]] : memref<?xindex, strided<[?], offset: ?>>
-// CHECK:           %[[VAL_15:.*]] = memref.load %[[VAL_10]]{{\[}}%[[VAL_14]]] : memref<32xf64>
-// CHECK:           %[[VAL_16:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_13]]] : memref<?xindex, strided<[?], offset: ?>>
-// CHECK:           %[[VAL_17:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_13]]] : memref<?xf64>
-// CHECK:           %[[VAL_18:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_16]]] : memref<64xf64>
-// CHECK:           %[[VAL_19:.*]] = arith.mulf %[[VAL_17]], %[[VAL_18]] : f64
-// CHECK:           %[[VAL_20:.*]] = arith.addf %[[VAL_15]], %[[VAL_19]] : f64
-// CHECK:           memref.store %[[VAL_20]], %[[VAL_10]]{{\[}}%[[VAL_14]]] : memref<32xf64>
+// CHECK-LABEL:   func.func @matvec(
+// CHECK-SAME:      %[[VAL_0:.*]]: tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>>,
+// CHECK-SAME:      %[[VAL_1:.*]]: tensor<64xf64>,
+// CHECK-SAME:      %[[VAL_2:.*]]: tensor<32xf64>) -> tensor<32xf64> {
+// CHECK-DAG:       %[[VAL_3:.*]] = arith.constant false
+// CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
+// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex, strided<[?], offset: ?>>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex, strided<[?], offset: ?>>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xf64>
+// CHECK:           %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xf64>
+// CHECK:           %[[VAL_11:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_4]]] : memref<?xindex>
+// CHECK:           %[[VAL_12:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_5]]] : memref<?xindex>
+// CHECK:           %[[VAL_13:.*]] = scf.while (%[[VAL_14:.*]] = %[[VAL_11]]) : (index) -> index {
+// CHECK:             %[[VAL_15:.*]] = arith.cmpi ult, %[[VAL_14]], %[[VAL_12]] : index
+// CHECK:             scf.condition(%[[VAL_15]]) %[[VAL_14]] : index
+// CHECK:           } do {
+// CHECK:           ^bb0(%[[VAL_16:.*]]: index):
+// CHECK:             %[[VAL_17:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_16]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:             %[[VAL_18:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_16]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:             %[[VAL_19:.*]] = scf.while (%[[VAL_20:.*]] = %[[VAL_16]]) : (index) -> index {
+// CHECK:               %[[VAL_21:.*]] = arith.cmpi ult, %[[VAL_20]], %[[VAL_12]] : index
+// CHECK:               %[[VAL_22:.*]] = scf.if %[[VAL_21]] -> (i1) {
+// CHECK:                 %[[VAL_23:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_20]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:                 %[[VAL_24:.*]] = arith.cmpi eq, %[[VAL_23]], %[[VAL_18]] : index
+// CHECK:                 scf.yield %[[VAL_24]] : i1
+// CHECK:               } else {
+// CHECK:                 scf.yield %[[VAL_3]] : i1
+// CHECK:               }
+// CHECK:               scf.condition(%[[VAL_25:.*]]) %[[VAL_20]] : index
+// CHECK:             } do {
+// CHECK:             ^bb0(%[[VAL_26:.*]]: index):
+// CHECK:               %[[VAL_27:.*]] = arith.addi %[[VAL_26]], %[[VAL_5]] : index
+// CHECK:               scf.yield %[[VAL_27]] : index
+// CHECK:             }
+// CHECK:             %[[VAL_28:.*]] = tensor.extract %[[VAL_2]]{{\[}}%[[VAL_17]]] : tensor<32xf64>
+// CHECK:             %[[VAL_29:.*]] = scf.for %[[VAL_30:.*]] = %[[VAL_16]] to %[[VAL_31:.*]] step %[[VAL_5]] iter_args(%[[VAL_32:.*]] = %[[VAL_28]]) -> (f64) {
+// CHECK:               %[[VAL_33:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_30]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:               %[[VAL_34:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_30]]] : memref<?xf64>
+// CHECK:               %[[VAL_35:.*]] = tensor.extract %[[VAL_1]]{{\[}}%[[VAL_33]]] : tensor<64xf64>
+// CHECK:               %[[VAL_36:.*]] = arith.mulf %[[VAL_34]], %[[VAL_35]] : f64
+// CHECK:               %[[VAL_37:.*]] = arith.addf %[[VAL_32]], %[[VAL_36]] : f64
+// CHECK:               scf.yield %[[VAL_37]] : f64
+// CHECK:             } {"Emitted from" = "linalg.generic"}
+// CHECK:             memref.store %[[VAL_38:.*]], %[[VAL_10]]{{\[}}%[[VAL_17]]] : memref<32xf64>
+// CHECK:             scf.yield %[[VAL_39:.*]] : index
+// CHECK:           } attributes {"Emitted from" = "linalg.generic"}
+// CHECK:           %[[VAL_40:.*]] = bufferization.to_tensor %[[VAL_10]] : memref<32xf64>
+// CHECK:           return %[[VAL_40]] : tensor<32xf64>
 // CHECK:         }
-// CHECK:         %[[VAL_21:.*]] = bufferization.to_tensor %[[VAL_10]] : memref<32xf64>
-// CHECK:         return %[[VAL_21]] : tensor<32xf64>
-// CHECK:       }
 func.func @matvec(%arga: tensor<32x64xf64, #SortedCOO>,
                   %argb: tensor<64xf64>,
                   %argx: tensor<32xf64>) -> tensor<32xf64> {
@@ -105,86 +154,113 @@ func.func @matvec(%arga: tensor<32x64xf64, #SortedCOO>,
   return %0 : tensor<32xf64>
 }
 
-// CHECK-LABEL: func.func @mateltmul(
-// CHECK-SAME:    %[[VAL_0:.*0]]: tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>>,
-// CHECK-SAME:    %[[VAL_1:.*1]]: tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>>,
-// CHECK-SAME:    %[[VAL_2:.*2]]: tensor<32x64xf64>) -> tensor<32x64xf64> {
-// CHECK-DAG:     %[[VAL_3:.*]] = arith.constant 0.000000e+00 : f64
-// CHECK-DAG:     %[[VAL_4:.*]] = arith.constant 0 : index
-// CHECK-DAG:     %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-DAG:     %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex>
-// CHECK-DAG:     %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex, strided<[?], offset: ?>>
-// CHECK-DAG:     %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex, strided<[?], offset: ?>>
-// CHECK-DAG:     %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xf64>
-// CHECK-DAG:     %[[VAL_10:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex>
-// CHECK-DAG:     %[[VAL_11:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex, strided<[?], offset: ?>>
-// CHECK-DAG:     %[[VAL_12:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex, strided<[?], offset: ?>>
-// CHECK-DAG:     %[[VAL_13:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xf64>
-// CHECK-DAG:     %[[VAL_14:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x64xf64>
-// CHECK-DAG:     linalg.fill ins(%[[VAL_3]] : f64) outs(%[[VAL_14]] : memref<32x64xf64>)
-// CHECK-DAG:     %[[VAL_15:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_4]]] : memref<?xindex>
-// CHECK-DAG:     %[[VAL_16:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_5]]] : memref<?xindex>
-// CHECK-DAG:     %[[VAL_17:.*]] = memref.load %[[VAL_10]]{{\[}}%[[VAL_4]]] : memref<?xindex>
-// CHECK-DAG:     %[[VAL_18:.*]] = memref.load %[[VAL_10]]{{\[}}%[[VAL_5]]] : memref<?xindex>
-// CHECK:         %[[VAL_19:.*]]:2 = scf.while (%[[VAL_20:.*]] = %[[VAL_15]], %[[VAL_21:.*]] = %[[VAL_17]]) : (index, index) -> (index, index) {
-// CHECK:           %[[VAL_22:.*]] = arith.cmpi ult, %[[VAL_20]], %[[VAL_16]] : index
-// CHECK:           %[[VAL_23:.*]] = arith.cmpi ult, %[[VAL_21]], %[[VAL_18]] : index
-// CHECK:           %[[VAL_24:.*]] = arith.andi %[[VAL_22]], %[[VAL_23]] : i1
-// CHECK:           scf.condition(%[[VAL_24]]) %[[VAL_20]], %[[VAL_21]] : index, index
-// CHECK:         } do {
-// CHECK:         ^bb0(%[[VAL_25:.*]]: index, %[[VAL_26:.*]]: index):
-// CHECK:           %[[VAL_27:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_25]]] : memref<?xindex, strided<[?], offset: ?>>
-// CHECK:           %[[VAL_28:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_26]]] : memref<?xindex, strided<[?], offset: ?>>
-// CHECK:           %[[VAL_29:.*]] = arith.cmpi ult, %[[VAL_28]], %[[VAL_27]] : index
-// CHECK:           %[[VAL_30:.*]] = arith.select %[[VAL_29]], %[[VAL_28]], %[[VAL_27]] : index
-// CHECK:           %[[VAL_31:.*]] = arith.cmpi eq, %[[VAL_27]], %[[VAL_30]] : index
-// CHECK:           %[[VAL_32:.*]] = arith.cmpi eq, %[[VAL_28]], %[[VAL_30]] : index
-// CHECK:           %[[VAL_33:.*]] = arith.andi %[[VAL_31]], %[[VAL_32]] : i1
-// CHECK:           scf.if %[[VAL_33]] {
-// CHECK:             %[[VAL_34:.*]] = arith.addi %[[VAL_25]], %[[VAL_5]] : index
-// CHECK:             %[[VAL_35:.*]] = arith.addi %[[VAL_26]], %[[VAL_5]] : index
-// CHECK:             %[[VAL_36:.*]]:2 = scf.while (%[[VAL_37:.*]] = %[[VAL_25]], %[[VAL_38:.*]] = %[[VAL_26]]) : (index, index) -> (index, index) {
-// CHECK:               %[[VAL_39:.*]] = arith.cmpi ult, %[[VAL_37]], %[[VAL_34]] : index
-// CHECK:               %[[VAL_40:.*]] = arith.cmpi ult, %[[VAL_38]], %[[VAL_35]] : index
-// CHECK:               %[[VAL_41:.*]] = arith.andi %[[VAL_39]], %[[VAL_40]] : i1
-// CHECK:               scf.condition(%[[VAL_41]]) %[[VAL_37]], %[[VAL_38]] : index, index
+// CHECK-LABEL:   func.func @mateltmul(
+// CHECK-SAME:      %[[VAL_0:.*0]]: tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>>,
+// CHECK-SAME:      %[[VAL_1:.*1]]: tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>>,
+// CHECK-SAME:      %[[VAL_2:.*2]]: tensor<32x64xf64>) -> tensor<32x64xf64> {
+// CHECK-DAG:       %[[VAL_3:.*]] = arith.constant false
+// CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0.000000e+00 : f64
+// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0 : index
+// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 1 : index
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex, strided<[?], offset: ?>>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex, strided<[?], offset: ?>>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xf64>
+// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex, strided<[?], offset: ?>>
+// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 1 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xindex, strided<[?], offset: ?>>
+// CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ] }>> to memref<?xf64>
+// CHECK:           %[[VAL_15:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x64xf64>
+// CHECK:           linalg.fill ins(%[[VAL_4]] : f64) outs(%[[VAL_15]] : memref<32x64xf64>)
+// CHECK:           %[[VAL_16:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_5]]] : memref<?xindex>
+// CHECK:           %[[VAL_17:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_6]]] : memref<?xindex>
+// CHECK:           %[[VAL_18:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_5]]] : memref<?xindex>
+// CHECK:           %[[VAL_19:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_6]]] : memref<?xindex>
+// CHECK:           %[[VAL_20:.*]]:2 = scf.while (%[[VAL_21:.*]] = %[[VAL_16]], %[[VAL_22:.*]] = %[[VAL_18]]) : (index, index) -> (index, index) {
+// CHECK:             %[[VAL_23:.*]] = arith.cmpi ult, %[[VAL_21]], %[[VAL_17]] : index
+// CHECK:             %[[VAL_24:.*]] = arith.cmpi ult, %[[VAL_22]], %[[VAL_19]] : index
+// CHECK:             %[[VAL_25:.*]] = arith.andi %[[VAL_23]], %[[VAL_24]] : i1
+// CHECK:             scf.condition(%[[VAL_25]]) %[[VAL_21]], %[[VAL_22]] : index, index
+// CHECK:           } do {
+// CHECK:           ^bb0(%[[VAL_26:.*]]: index, %[[VAL_27:.*]]: index):
+// CHECK:             %[[VAL_28:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_26]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:             %[[VAL_29:.*]] = memref.load %[[VAL_12]]{{\[}}%[[VAL_27]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:             %[[VAL_30:.*]] = arith.cmpi ult, %[[VAL_29]], %[[VAL_28]] : index
+// CHECK:             %[[VAL_31:.*]] = arith.select %[[VAL_30]], %[[VAL_29]], %[[VAL_28]] : index
+// CHECK:             %[[VAL_32:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_26]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:             %[[VAL_33:.*]] = scf.while (%[[VAL_34:.*]] = %[[VAL_26]]) : (index) -> index {
+// CHECK:               %[[VAL_35:.*]] = arith.cmpi ult, %[[VAL_34]], %[[VAL_17]] : index
+// CHECK:               %[[VAL_36:.*]] = scf.if %[[VAL_35]] -> (i1) {
+// CHECK:                 %[[VAL_37:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_34]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:                 %[[VAL_38:.*]] = arith.cmpi eq, %[[VAL_37]], %[[VAL_32]] : index
+// CHECK:                 scf.yield %[[VAL_38]] : i1
+// CHECK:               } else {
+// CHECK:                 scf.yield %[[VAL_3]] : i1
+// CHECK:               }
+// CHECK:               scf.condition(%[[VAL_39:.*]]) %[[VAL_34]] : index
 // CHECK:             } do {
-// CHECK:             ^bb0(%[[VAL_42:.*]]: index, %[[VAL_43:.*]]: index):
-// CHECK:               %[[VAL_44:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_42]]] : memref<?xindex, strided<[?], offset: ?>>
-// CHECK:               %[[VAL_45:.*]] = memref.load %[[VAL_12]]{{\[}}%[[VAL_43]]] : memref<?xindex, strided<[?], offset: ?>>
-// CHECK:               %[[VAL_46:.*]] = arith.cmpi ult, %[[VAL_45]], %[[VAL_44]] : index
-// CHECK:               %[[VAL_47:.*]] = arith.select %[[VAL_46]], %[[VAL_45]], %[[VAL_44]] : index
-// CHECK:               %[[VAL_48:.*]] = arith.cmpi eq, %[[VAL_44]], %[[VAL_47]] : index
-// CHECK:               %[[VAL_49:.*]] = arith.cmpi eq, %[[VAL_45]], %[[VAL_47]] : index
-// CHECK:               %[[VAL_50:.*]] = arith.andi %[[VAL_48]], %[[VAL_49]] : i1
-// CHECK:               scf.if %[[VAL_50]] {
-// CHECK:                 %[[VAL_51:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_42]]] : memref<?xf64>
-// CHECK:                 %[[VAL_52:.*]] = memref.load %[[VAL_13]]{{\[}}%[[VAL_43]]] : memref<?xf64>
-// CHECK:                 %[[VAL_53:.*]] = arith.mulf %[[VAL_51]], %[[VAL_52]] : f64
-// CHECK:                 memref.store %[[VAL_53]], %[[VAL_14]]{{\[}}%[[VAL_30]], %[[VAL_47]]] : memref<32x64xf64>
+// CHECK:             ^bb0(%[[VAL_40:.*]]: index):
+// CHECK:               %[[VAL_41:.*]] = arith.addi %[[VAL_40]], %[[VAL_6]] : index
+// CHECK:               scf.yield %[[VAL_41]] : index
+// CHECK:             }
+// CHECK:             %[[VAL_42:.*]] = memref.load %[[VAL_12]]{{\[}}%[[VAL_27]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:             %[[VAL_43:.*]] = scf.while (%[[VAL_44:.*]] = %[[VAL_27]]) : (index) -> index {
+// CHECK:               %[[VAL_45:.*]] = arith.cmpi ult, %[[VAL_44]], %[[VAL_19]] : index
+// CHECK:               %[[VAL_46:.*]] = scf.if %[[VAL_45]] -> (i1) {
+// CHECK:                 %[[VAL_47:.*]] = memref.load %[[VAL_12]]{{\[}}%[[VAL_44]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:                 %[[VAL_48:.*]] = arith.cmpi eq, %[[VAL_47]], %[[VAL_42]] : index
+// CHECK:                 scf.yield %[[VAL_48]] : i1
 // CHECK:               } else {
+// CHECK:                 scf.yield %[[VAL_3]] : i1
 // CHECK:               }
-// CHECK:               %[[VAL_54:.*]] = arith.cmpi eq, %[[VAL_44]], %[[VAL_47]] : index
-// CHECK:               %[[VAL_55:.*]] = arith.addi %[[VAL_42]], %[[VAL_5]] : index
-// CHECK:               %[[VAL_56:.*]] = arith.select %[[VAL_54]], %[[VAL_55]], %[[VAL_42]] : index
-// CHECK:               %[[VAL_57:.*]] = arith.cmpi eq, %[[VAL_45]], %[[VAL_47]] : index
-// CHECK:               %[[VAL_58:.*]] = arith.addi %[[VAL_43]], %[[VAL_5]] : index
-// CHECK:               %[[VAL_59:.*]] = arith.select %[[VAL_57]], %[[VAL_58]], %[[VAL_43]] : index
-// CHECK:               scf.yield %[[VAL_56]], %[[VAL_59]] : index, index
+// CHECK:               scf.condition(%[[VAL_49:.*]]) %[[VAL_44]] : index
+// CHECK:             } do {
+// CHECK:             ^bb0(%[[VAL_50:.*]]: index):
+// CHECK:               %[[VAL_51:.*]] = arith.addi %[[VAL_50]], %[[VAL_6]] : index
+// CHECK:               scf.yield %[[VAL_51]] : index
+// CHECK:             }
+// CHECK:             %[[VAL_52:.*]] = arith.cmpi eq, %[[VAL_28]], %[[VAL_31]] : index
+// CHECK:             %[[VAL_53:.*]] = arith.cmpi eq, %[[VAL_29]], %[[VAL_31]] : index
+// CHECK:             %[[VAL_54:.*]] = arith.andi %[[VAL_52]], %[[VAL_53]] : i1
+// CHECK:             scf.if %[[VAL_54]] {
+// CHECK:               %[[VAL_55:.*]]:2 = scf.while (%[[VAL_56:.*]] = %[[VAL_26]], %[[VAL_57:.*]] = %[[VAL_27]]) : (index, index) -> (index, index) {
+// CHECK:                 %[[VAL_58:.*]] = arith.cmpi ult, %[[VAL_56]], %[[VAL_59:.*]] : index
+// CHECK:                 %[[VAL_60:.*]] = arith.cmpi ult, %[[VAL_57]], %[[VAL_61:.*]] : index
+// CHECK:                 %[[VAL_62:.*]] = arith.andi %[[VAL_58]], %[[VAL_60]] : i1
+// CHECK:                 scf.condition(%[[VAL_62]]) %[[VAL_56]], %[[VAL_57]] : index, index
+// CHECK:               } do {
+// CHECK:               ^bb0(%[[VAL_63:.*]]: index, %[[VAL_64:.*]]: index):
+// CHECK:                 %[[VAL_65:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_63]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:                 %[[VAL_66:.*]] = memref.load %[[VAL_13]]{{\[}}%[[VAL_64]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:                 %[[VAL_67:.*]] = arith.cmpi ult, %[[VAL_66]], %[[VAL_65]] : index
+// CHECK:                 %[[VAL_68:.*]] = arith.select %[[VAL_67]], %[[VAL_66]], %[[VAL_65]] : index
+// CHECK:                 %[[VAL_69:.*]] = arith.cmpi eq, %[[VAL_65]], %[[VAL_68]] : index
+// CHECK:                 %[[VAL_70:.*]] = arith.cmpi eq, %[[VAL_66]], %[[VAL_68]] : index
+// CHECK:                 %[[VAL_71:.*]] = arith.andi %[[VAL_69]], %[[VAL_70]] : i1
+// CHECK:                 scf.if %[[VAL_71]] {
+// CHECK:                   %[[VAL_72:.*]] = memref.load %[[VAL_10]]{{\[}}%[[VAL_63]]] : memref<?xf64>
+// CHECK:                   %[[VAL_73:.*]] = memref.load %[[VAL_14]]{{\[}}%[[VAL_64]]] : memref<?xf64>
+// CHECK:                   %[[VAL_74:.*]] = arith.mulf %[[VAL_72]], %[[VAL_73]] : f64
+// CHECK:                   memref.store %[[VAL_74]], %[[VAL_15]]{{\[}}%[[VAL_31]], %[[VAL_68]]] : memref<32x64xf64>
+// CHECK:                 }
+// CHECK:                 %[[VAL_75:.*]] = arith.cmpi eq, %[[VAL_65]], %[[VAL_68]] : index
+// CHECK:                 %[[VAL_76:.*]] = arith.addi %[[VAL_63]], %[[VAL_6]] : index
+// CHECK:                 %[[VAL_77:.*]] = arith.select %[[VAL_75]], %[[VAL_76]], %[[VAL_63]] : index
+// CHECK:                 %[[VAL_78:.*]] = arith.cmpi eq, %[[VAL_66]], %[[VAL_68]] : index
+// CHECK:                 %[[VAL_79:.*]] = arith.addi %[[VAL_64]], %[[VAL_6]] : index
+// CHECK:                 %[[VAL_80:.*]] = arith.select %[[VAL_78]], %[[VAL_79]], %[[VAL_64]] : index
+// CHECK:                 scf.yield %[[VAL_77]], %[[VAL_80]] : index, index
+// CHECK:               } attributes {"Emitted from" = "linalg.generic"}
 // CHECK:             }
-// CHECK:           } else {
-// CHECK:           }
-// CHECK:           %[[VAL_60:.*]] = arith.cmpi eq, %[[VAL_27]], %[[VAL_30]] : index
-// CHECK:           %[[VAL_61:.*]] = arith.addi %[[VAL_25]], %[[VAL_5]] : index
-// CHECK:           %[[VAL_62:.*]] = arith.select %[[VAL_60]], %[[VAL_61]], %[[VAL_25]] : index
-// CHECK:           %[[VAL_63:.*]] = arith.cmpi eq, %[[VAL_28]], %[[VAL_30]] : index
-// CHECK:           %[[VAL_64:.*]] = arith.addi %[[VAL_26]], %[[VAL_5]] : index
-// CHECK:           %[[VAL_65:.*]] = arith.select %[[VAL_63]], %[[VAL_64]], %[[VAL_26]] : index
-// CHECK:           scf.yield %[[VAL_62]], %[[VAL_65]] : index, index
+// CHECK:             %[[VAL_81:.*]] = arith.cmpi eq, %[[VAL_28]], %[[VAL_31]] : index
+// CHECK:             %[[VAL_82:.*]] = arith.select %[[VAL_81]], %[[VAL_83:.*]], %[[VAL_26]] : index
+// CHECK:             %[[VAL_84:.*]] = arith.cmpi eq, %[[VAL_29]], %[[VAL_31]] : index
+// CHECK:             %[[VAL_85:.*]] = arith.select %[[VAL_84]], %[[VAL_86:.*]], %[[VAL_27]] : index
+// CHECK:             scf.yield %[[VAL_82]], %[[VAL_85]] : index, index
+// CHECK:           } attributes {"Emitted from" = "linalg.generic"}
+// CHECK:           %[[VAL_87:.*]] = bufferization.to_tensor %[[VAL_15]] : memref<32x64xf64>
+// CHECK:           return %[[VAL_87]] : tensor<32x64xf64>
 // CHECK:         }
-// CHECK:         %[[VAL_66:.*]] = bufferization.to_tensor %[[VAL_14]] : memref<32x64xf64>
-// CHECK:         return %[[VAL_66]] : tensor<32x64xf64>
-// CHECK:       }
 func.func @mateltmul(%argx: tensor<32x64xf64, #SortedCOO>,
                      %argy: tensor<32x64xf64, #SortedCOO>,
                      %argz: tensor<32x64xf64>) -> tensor<32x64xf64> {
diff --git a/mlir/test/Dialect/SparseTensor/sparse_1d.mlir b/mlir/test/Dialect/SparseTensor/sparse_1d.mlir
index 0d5d554f9f326..5183e8da5d369 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_1d.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_1d.mlir
@@ -110,8 +110,8 @@ func.func @mul_d(%arga: tensor<32xf32, #DV>, %argb: f32, %argx: tensor<32xf32>)
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant true
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_12:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_4]]] : memref<?xindex>
 // CHECK-DAG:       %[[VAL_13:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_6]]] : memref<?xindex>
@@ -162,8 +162,8 @@ func.func @add_s(%arga: tensor<32xf32, #SV>, %argb: f32, %argx: tensor<32xf32>)
 // CHECK-SAME:      %[[VAL_1:.*]]: tensor<32xf32>) -> tensor<32xf32> {
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_1]]
 // CHECK-DAG:       %[[VAL_9:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_2]]] : memref<?xindex>
@@ -202,8 +202,8 @@ func.func @repeated_add_s(%arga: tensor<32xf32, #SV>, %argx: tensor<32xf32>) ->
 // CHECK-SAME:      %[[VAL_2:.*]]: tensor<32xf32>) -> tensor<32xf32> {
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_2]]
 // CHECK-DAG:       linalg.fill ins(%{{.*}} : f32) outs(%[[VAL_9]] : memref<32xf32>)
@@ -310,8 +310,8 @@ func.func @mul_dd(%arga: tensor<32xf32, #DV>, %argb: tensor<32xf32>, %argx: tens
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant true
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_7:.*]] = bufferization.to_memref %[[VAL_0]] : memref<32xf32>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_2]]
 // CHECK-DAG:       linalg.fill ins(%{{.*}} : f32) outs(%[[VAL_12]] : memref<32xf32>)
@@ -367,8 +367,8 @@ func.func @add_ds(%arga: tensor<32xf32>, %argb: tensor<32xf32, #SV>, %argx: tens
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = bufferization.to_memref %[[VAL_0]] : memref<32xf32>
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_2]]
 // CHECK-DAG:       linalg.fill ins(%{{.*}} : f32) outs(%[[VAL_10]] : memref<32xf32>)
@@ -403,8 +403,8 @@ func.func @mul_ds(%arga: tensor<32xf32>, %argb: tensor<32xf32, #SV>, %argx: tens
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant true
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xf32>
 // CHECK-DAG:       %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_2]]
@@ -460,8 +460,8 @@ func.func @add_sd(%arga: tensor<32xf32, #SV>, %argb: tensor<32xf32>, %argx: tens
 // CHECK-SAME:      %[[VAL_2:.*]]: tensor<32xf32>) -> tensor<32xf32> {
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xf32>
 // CHECK-DAG:       %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_2]]
@@ -495,11 +495,11 @@ func.func @mul_sd(%arga: tensor<32xf32, #SV>, %argb: tensor<32xf32>, %argx: tens
 // CHECK-SAME:      %[[VAL_2:.*2]]: tensor<32xf32>) -> tensor<32xf32> {
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_2]]
 // CHECK-DAG:       linalg.fill ins(%{{.*}} : f32) outs(%[[VAL_12]] : memref<32xf32>)
@@ -578,11 +578,11 @@ func.func @add_ss(%arga: tensor<32xf32, #SV>, %argb: tensor<32xf32, #SV>, %argx:
 // CHECK-SAME:      %[[VAL_2:.*2]]: tensor<32xf32>) -> tensor<32xf32> {
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_2]]
 // CHECK-DAG:       linalg.fill ins(%{{.*}} : f32) outs(%[[VAL_12]] : memref<32xf32>)
@@ -640,11 +640,11 @@ func.func @mul_ss(%arga: tensor<32xf32, #SV>, %argb: tensor<32xf32, #SV>, %argx:
 // CHECK-SAME:      %[[VAL_3:.*3]]: tensor<16xf32>) -> tensor<16xf32> {
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_3]]
 // CHECK-DAG:       linalg.fill ins(%{{.*}} : f32) outs(%[[VAL_13]] : memref<16xf32>)
@@ -733,11 +733,11 @@ func.func @two_way_inv(%arga: tensor<16xf32, #SV>, %argb: tensor<16xf32, #SV>, %
 // CHECK-SAME:      %[[VAL_3:.*3]]: tensor<16xf32>) -> tensor<16xf32> {
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_3]]
 // CHECK-DAG:       linalg.fill ins(%{{.*}} : f32) outs(%[[VAL_13]] : memref<16xf32>)
@@ -832,7 +832,7 @@ func.func @two_way_inv_alt(%arga: tensor<16xf32, #SV>,
 // CHECK-SAME:      %[[VAL_1:.*]]: tensor<f32>) -> tensor<f32> {
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_6:.*]] = bufferization.to_memref %[[VAL_1]] : memref<f32>
 // CHECK-DAG:       %[[VAL_8:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_2]]] : memref<?xindex>
@@ -874,11 +874,11 @@ func.func @sum_reduction(%arga: tensor<?xf32, #SV>, %argx: tensor<f32>) -> tenso
 // CHECK-SAME:      %[[VAL_2:.*2]]: tensor<f32>) -> tensor<f32> {
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_2]] : memref<f32>
 // CHECK-DAG:           %[[VAL_13:.*]] = memref.load %[[VAL_11]][] : memref<f32>
@@ -981,12 +981,12 @@ func.func @sum_reduction_ss(%arga: tensor<16xf32, #SV>,
 // CHECK-SAME:      %[[VAL_3:.*3]]: tensor<f32>) -> tensor<f32> {
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_1]] : memref<f32>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.pointers %[[VAL_2]] {dimension = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.indices %[[VAL_2]] {dimension = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.positions %[[VAL_2]] {level = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.coordinates %[[VAL_2]] {level = 0 : index} : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.values %[[VAL_2]] : tensor<16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_3]] : memref<f32>
 // CHECK-DAG:       %[[VAL_15:.*]] = memref.load %[[VAL_13]][] : memref<f32>
@@ -1099,12 +1099,12 @@ func.func @sum_reduction_inv(%arga: tensor<16xf32, #SV>,
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant true
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_0]] : memref<?xf64>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf64>
 // CHECK-DAG:       %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_2]] : memref<?xf64>
-// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.pointers %[[VAL_3]] {dimension = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.indices %[[VAL_3]] {dimension = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.positions %[[VAL_3]] {level = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.coordinates %[[VAL_3]] {level = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_15:.*]] = sparse_tensor.values %[[VAL_3]] : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf64>
 // CHECK-DAG:       %[[VAL_16:.*]] = tensor.dim %[[VAL_0]], %[[VAL_5]] : tensor<?xf64>
 // CHECK-DAG:       %[[VAL_18:.*]] = bufferization.to_memref %[[VAL_4]]
@@ -1274,14 +1274,14 @@ func.func @four_tensors_op(%arga: tensor<?xf64>,
 // CHECK-SAME:      %[[VAL_3:.*3]]: tensor<f64>) -> tensor<f64> {
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf64>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf64>
-// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.pointers %[[VAL_2]] {dimension = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.indices %[[VAL_2]] {dimension = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.positions %[[VAL_2]] {level = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.coordinates %[[VAL_2]] {level = 0 : index} : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.values %[[VAL_2]] : tensor<?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf64>
 // CHECK-DAG:       %[[VAL_15:.*]] = bufferization.to_memref %[[VAL_3]] : memref<f64>
 // CHECK-DAG:       %[[VAL_17:.*]] = memref.load %[[VAL_15]][] : memref<f64>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_2d.mlir b/mlir/test/Dialect/SparseTensor/sparse_2d.mlir
index 97ee3738ec827..9b2717bf8d29c 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_2d.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_2d.mlir
@@ -97,8 +97,8 @@ func.func @mul_dd(%arga: tensor<32x16xf32, #Tdd>, %argb: tensor<32x16xf32>, %arg
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant true
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16xf32>
 // CHECK-DAG:       %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16xf32>
@@ -158,8 +158,8 @@ func.func @add_ds(%arga: tensor<32x16xf32, #Tds>, %argb: tensor<32x16xf32>, %arg
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 32 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16xf32>
 // CHECK-DAG:       %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16xf32>
@@ -199,8 +199,8 @@ func.func @mul_ds(%arga: tensor<32x16xf32, #Tds>, %argb: tensor<32x16xf32>, %arg
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant true
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16xf32>
 // CHECK-DAG:       %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16xf32>
@@ -265,8 +265,8 @@ func.func @add_sd(%arga: tensor<32x16xf32, #Tsd>, %argb: tensor<32x16xf32>, %arg
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 16 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16xf32>
 // CHECK-DAG:       %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16xf32>
@@ -307,10 +307,10 @@ func.func @mul_sd(%arga: tensor<32x16xf32, #Tsd>, %argb: tensor<32x16xf32>, %arg
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant true
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16xf32>
 // CHECK-DAG:       %[[VAL_15:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16xf32>
@@ -398,10 +398,10 @@ func.func @add_ss(%arga: tensor<32x16xf32, #Tss>, %argb: tensor<32x16xf32>, %arg
 // CHECK-SAME:      %[[VAL_2:.*]]: tensor<32x16xf32>) -> tensor<32x16xf32> {
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16xf32>
 // CHECK-DAG:       %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16xf32>
@@ -441,15 +441,15 @@ func.func @mul_ss(%arga: tensor<32x16xf32, #Tss>, %argb: tensor<32x16xf32>, %arg
 // CHECK-SAME:      %[[VAL_2:.*2]]: tensor<32x16xf32>) -> tensor<32x16xf32> {
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf32>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_16:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16xf32>
 // CHECK:           linalg.fill ins(%{{.*}} : f32) outs(%[[VAL_16]] : memref<32x16xf32>)
@@ -605,15 +605,15 @@ func.func @add_ss_ss(%arga: tensor<32x16xf32, #Tss>, %argb: tensor<32x16xf32, #T
 // CHECK-SAME:      %[[VAL_2:.*2]]: tensor<32x16xf32>) -> tensor<32x16xf32> {
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf32>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_16:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16xf32>
 // CHECK:           linalg.fill ins(%{{.*}} : f32) outs(%[[VAL_16]] : memref<32x16xf32>)
@@ -704,11 +704,11 @@ func.func @mul_ss_ss(%arga: tensor<32x16xf32, #Tss>, %argb: tensor<32x16xf32, #T
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant true
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xf32>
-// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_15:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16xf32>
 // CHECK:           linalg.fill ins(%{{.*}} : f32) outs(%[[VAL_15]] : memref<32x16xf32>)
@@ -808,11 +808,11 @@ func.func @add_sd_ds(%arga: tensor<32x16xf32, #Tsd>, %argb: tensor<32x16xf32, #T
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 16 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense" ] }>> to memref<?xf32>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16xf32>
 // CHECK:           linalg.fill ins(%{{.*}} : f32) outs(%[[VAL_13]] : memref<32x16xf32>)
@@ -864,8 +864,8 @@ func.func @mul_sd_ds(%arga: tensor<32x16xf32, #Tsd>, %argb: tensor<32x16xf32, #T
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 16 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<16x32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<16x32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<16x32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<16x32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<16x32xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xf32>
 // CHECK-DAG:       %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_2]] : memref<16xf32>
@@ -914,7 +914,7 @@ func.func @matvec(%argA: tensor<16x32xf32, #Tds>, %argb: tensor<32xf32>, %argx:
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 10 : index
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
-// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_1]] : memref<f32>
 // CHECK:           %[[VAL_9:.*]] = memref.load %[[VAL_8]][] : memref<f32>
@@ -959,8 +959,8 @@ func.func @sum_reduction(%arga: tensor<10x20xf32, #Tds>, %argx: tensor<f32>) ->
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 2.000000e+00 : f64
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<?x?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<?x?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<?x?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<?x?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<?x?xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xf64>
 // CHECK-DAG:       %[[VAL_8:.*]] = tensor.dim %[[VAL_0]], %[[VAL_3]] : tensor<?x?xf64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK-DAG:       %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_1]] : memref<?x?xf64>
@@ -1009,10 +1009,10 @@ func.func @scale(%arga: tensor<?x?xf64, #Tds>, %argx: tensor<?x?xf64>) -> tensor
 // CHECK-SAME:      %[[VAL_3:.*3]]: tensor<?x?xf32>) -> tensor<?x?xf32> {
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0 : index
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_11:.*]] = tensor.dim %[[VAL_1]], %[[VAL_4]] : tensor<?x?xf32>
 // CHECK-DAG:       %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_1]] : memref<?x?xf32>
@@ -1081,16 +1081,16 @@ func.func @sampled_dense_dense(%args: tensor<?x?xf32, #Tss>,
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant true
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf32>
-// CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_15:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_15:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK:           %[[VAL_16:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xf32>
-// CHECK-DAG:       %[[VAL_17:.*]] = sparse_tensor.pointers %[[VAL_2]] {dimension = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_18:.*]] = sparse_tensor.indices %[[VAL_2]] {dimension = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_17:.*]] = sparse_tensor.positions %[[VAL_2]] {level = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_18:.*]] = sparse_tensor.coordinates %[[VAL_2]] {level = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_19:.*]] = sparse_tensor.values %[[VAL_2]] : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_20:.*]] = bufferization.to_memref %[[VAL_3]] : memref<?xf32>
 // CHECK-DAG:       %[[VAL_21:.*]] = bufferization.to_memref %[[VAL_4]] : memref<f32>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_3d.mlir b/mlir/test/Dialect/SparseTensor/sparse_3d.mlir
index d1b07ad44a2aa..a49958c650969 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_3d.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_3d.mlir
@@ -117,8 +117,8 @@ func.func @mul_ddd(%arga: tensor<32x16x8xf32, #Tddd>, %argb: tensor<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant true
 // CHECK-DAG:       %[[VAL_9:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_15:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16x8xf32>
@@ -184,8 +184,8 @@ func.func @add_dds(%arga: tensor<32x16x8xf32, #Tdds>, %argb: tensor<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 16 : index
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16x8xf32>
@@ -231,8 +231,8 @@ func.func @mul_dds(%arga: tensor<32x16x8xf32, #Tdds>, %argb: tensor<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant true
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "dense" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "dense" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "dense" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_14:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16x8xf32>
@@ -302,8 +302,8 @@ func.func @add_dsd(%arga: tensor<32x16x8xf32, #Tdsd>, %argb: tensor<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 8 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "dense" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "dense" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "dense" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16x8xf32>
@@ -349,10 +349,10 @@ func.func @mul_dsd(%arga: tensor<32x16x8xf32, #Tdsd>, %argb: tensor<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant true
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_9:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_15:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_17:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16x8xf32>
@@ -445,10 +445,10 @@ func.func @add_dss(%arga: tensor<32x16x8xf32, #Tdss>, %argb: tensor<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 32 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_14:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16x8xf32>
@@ -496,8 +496,8 @@ func.func @mul_dss(%arga: tensor<32x16x8xf32, #Tdss>, %argb: tensor<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant true
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "dense" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "dense" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "dense" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_14:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16x8xf32>
@@ -572,8 +572,8 @@ func.func @add_sdd(%arga: tensor<32x16x8xf32, #Tsdd>, %argb: tensor<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 8 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "dense" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "dense" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "dense" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16x8xf32>
@@ -620,10 +620,10 @@ func.func @mul_sdd(%arga: tensor<32x16x8xf32, #Tsdd>, %argb: tensor<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant true
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_9:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_15:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_17:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16x8xf32>
@@ -721,10 +721,10 @@ func.func @add_sds(%arga: tensor<32x16x8xf32, #Tsds>, %argb: tensor<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 16 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "dense", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_14:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16x8xf32>
@@ -773,10 +773,10 @@ func.func @mul_sds(%arga: tensor<32x16x8xf32, #Tsds>, %argb: tensor<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant true
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "dense" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "dense" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "dense" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "dense" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "dense" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_14:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_16:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16x8xf32>
@@ -878,10 +878,10 @@ func.func @add_ssd(%arga: tensor<32x16x8xf32, #Tssd>, %argb: tensor<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 8 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "dense" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "dense" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "dense" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "dense" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "dense" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16x8xf32>
@@ -930,12 +930,12 @@ func.func @mul_ssd(%arga: tensor<32x16x8xf32, #Tssd>, %argb: tensor<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant true
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_9:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_15:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_15:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_16:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_17:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_19:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16x8xf32>
@@ -1060,12 +1060,12 @@ func.func @add_sss(%arga: tensor<32x16x8xf32, #Tsss>, %argb: tensor<32x16x8xf32>
 // CHECK-DAG:       %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 2 : index} : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16x8xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32x16x8xf32>
 // CHECK-DAG:       %[[VAL_15:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16x8xf32>
@@ -1123,8 +1123,8 @@ func.func @mul_sss(%arga: tensor<32x16x8xf32, #Tsss>, %argb: tensor<32x16x8xf32>
 // CHECK-SAME:      %[[VAL_3:.*3]]: tensor<?x?xf32>) -> tensor<?x?xf32> {
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 2 : index} : tensor<?x?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 2 : index} : tensor<?x?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 2 : index} : tensor<?x?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 2 : index} : tensor<?x?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?x?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "compressed" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_10:.*]] = tensor.dim %[[VAL_1]], %[[VAL_6]] : tensor<?x?x?xf32, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK-DAG:       %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_2]] : memref<?x?xf32>
@@ -1187,9 +1187,9 @@ func.func @kernel_3d(%arga: tensor<?x?xf32>,
 // CHECK-SAME:      %[[VAL_1:.*]]: tensor<f32>) -> tensor<f32> {
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<10x20x30xf32, #sparse_tensor.encoding<{{{.*}}>>
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<10x20x30xf32, #sparse_tensor.encoding<{{{.*}}>>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 2 : index} : tensor<10x20x30xf32, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<10x20x30xf32, #sparse_tensor.encoding<{{{.*}}>>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<10x20x30xf32, #sparse_tensor.encoding<{{{.*}}>>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 2 : index} : tensor<10x20x30xf32, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<10x20x30xf32, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK-DAG:       %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_1]] : memref<f32>
 // CHECK:           %[[VAL_11:.*]] = memref.load %[[VAL_10]][] : memref<f32>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_affine.mlir b/mlir/test/Dialect/SparseTensor/sparse_affine.mlir
index 3d2f70611eb6e..97293348774ca 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_affine.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_affine.mlir
@@ -23,8 +23,8 @@
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 3 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{{{.*}}}>>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xf32, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK-DAG:       %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_1]] : memref<4xf32>
 // CHECK-DAG:       %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xf32>
@@ -64,10 +64,10 @@ func.func @mul_inv_dense1d(%arga: tensor<32xf32, #SpVec>,
 // CHECK:           %[[VAL_4:.*]] = arith.constant 3 : index
 // CHECK:           %[[VAL_5:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK:           %[[VAL_6:.*]] = bufferization.alloc_tensor() : tensor<32xf32, #sparse_tensor.encoding<{{{.*}}}>>
-// CHECK:           %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xf32, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:           %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xf32, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xf32>
-// CHECK:           %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<4xf32, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:           %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<4xf32, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<4xf32, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<4xf32, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:           %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<4xf32, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xf32>
 // CHECK:           %[[VAL_12:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_2]]] : memref<?xindex>
 // CHECK:           %[[VAL_13:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_3]]] : memref<?xindex>
@@ -162,8 +162,8 @@ func.func @mul_inv_enc_dense1d(%arga: tensor<32xf32, #EncDenseVec>,
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 2 : index
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xi32, #sparse_tensor.encoding<{{{.*}}}>>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xi32, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xi32, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xi32, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xi32, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK-DAG:       %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_1]] : memref<34xi32>
 // CHECK-DAG:       %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xi32>
@@ -201,11 +201,11 @@ func.func @and_affine_dense1d(%arga: tensor<32xi32, #SpVec>,
 // CHECK:           %[[VAL_3:.*]] = arith.constant 1 : index
 // CHECK:           %[[VAL_4:.*]] = arith.constant 2 : index
 // CHECK:           %[[VAL_5:.*]] = bufferization.alloc_tensor() : tensor<32xi32, #sparse_tensor.encoding<{{{.*}}}>>
-// CHECK:           %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xi32, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:           %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xi32, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xi32, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xi32, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:           %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xi32, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xi32>
-// CHECK:           %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<34xi32, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:           %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<34xi32, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<34xi32, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<34xi32, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:           %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<34xi32, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xi32>
 // CHECK:           %[[VAL_12:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_2]]] : memref<?xindex>
 // CHECK:           %[[VAL_13:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_3]]] : memref<?xindex>
@@ -264,8 +264,8 @@ func.func @and_affine_sparse1d(%arga: tensor<32xi32, #SpVec>,
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 2 : index
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 3 : index
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf64, #sparse_tensor.encoding<{{{.*}}}>>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16xf64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x16xf64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16xf64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK-DAG:       %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_1]] : memref<34x19xf64>
 // CHECK-DAG:       %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16xf64>
@@ -315,11 +315,11 @@ func.func @mul_affine_dense2d(%arga: tensor<32x16xf64, #CSR>,
 // CHECK-DAG:       %[[VAL_TRUE:.*]] = arith.constant true
 // CHECK-DAG:       %[[VAL_FALSE:.*]] = arith.constant false
 // CHECK:           %[[VAL_8:.*]] = bufferization.alloc_tensor() : tensor<32x16xf64, #sparse_tensor.encoding<{{{.*}}}>>
-// CHECK:           %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:           %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x16xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:           %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xf64>
-// CHECK:           %[[VAL_12:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index} : tensor<34x19xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:           %[[VAL_13:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index} : tensor<34x19xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_12:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 1 : index} : tensor<34x19xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_13:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 1 : index} : tensor<34x19xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:           %[[VAL_14:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<34x19xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xf64>
 // CHECK:           %[[VAL_15:.*]] = scf.for %[[VAL_16:.*]] = %[[VAL_3]] to %[[VAL_2]] step %[[VAL_4]] iter_args(%[[VAL_17:.*]] = %[[VAL_8]]) -> (tensor<32x16xf64, #sparse_tensor.encoding<{{{.*}}}>>) {
 // CHECK:             %[[VAL_18:.*]] = arith.addi %[[VAL_16]], %[[VAL_5]] : index
@@ -391,11 +391,11 @@ func.func @mul_affine_sparse2d(%arga: tensor<32x16xf64, #CSR>,
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 2 : index
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 3 : index
-// CHECK:           %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<34x16xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:           %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<34x16xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<34x16xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<34x16xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:           %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<34x16xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xf64>
-// CHECK:           %[[VAL_11:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<32x19xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:           %[[VAL_12:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<32x19xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_11:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<32x19xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_12:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<32x19xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:           %[[VAL_13:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<32x19xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xf64>
 // CHECK:           %[[VAL_14:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16xf64>
 // CHECK:           %[[VAL_15:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_4]]] : memref<?xindex>
@@ -455,11 +455,11 @@ func.func @mul_affine_dense_dim_2d(%arga: tensor<34x16xf64, #CSR>,
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 3 : index
-// CHECK:           %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<34x16xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:           %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<34x16xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<34x16xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<34x16xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:           %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<34x16xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xf64>
-// CHECK:           %[[VAL_11:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<32x19xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:           %[[VAL_12:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<32x19xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_11:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<32x19xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_12:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<32x19xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:           %[[VAL_13:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<32x19xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xf64>
 // CHECK:           %[[VAL_14:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32x16xf64>
 // CHECK:           %[[VAL_15:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_5]]] : memref<?xindex>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_broadcast.mlir b/mlir/test/Dialect/SparseTensor/sparse_broadcast.mlir
index fd30b3fa5240d..a2b1926cdec4e 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_broadcast.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_broadcast.mlir
@@ -17,10 +17,10 @@
 //   CHECK-DAG:  %[[TMP_c0:.*]] = arith.constant 0 : index
 //   CHECK-DAG:  %[[TMP_c1:.*]] = arith.constant 1 : index
 //       CHECK:  %[[TMP_0:.*]] = bufferization.alloc_tensor()
-//       CHECK:  %[[TMP_1:.*]] = sparse_tensor.pointers %[[TMP_arg0]] {dimension = 0 : index}
-//       CHECK:  %[[TMP_2:.*]] = sparse_tensor.indices %[[TMP_arg0]] {dimension = 0 : index}
-//       CHECK:  %[[TMP_3:.*]] = sparse_tensor.pointers %[[TMP_arg0]] {dimension = 1 : index}
-//       CHECK:  %[[TMP_4:.*]] = sparse_tensor.indices %[[TMP_arg0]] {dimension = 1 : index}
+//       CHECK:  %[[TMP_1:.*]] = sparse_tensor.positions %[[TMP_arg0]] {level = 0 : index}
+//       CHECK:  %[[TMP_2:.*]] = sparse_tensor.coordinates %[[TMP_arg0]] {level = 0 : index}
+//       CHECK:  %[[TMP_3:.*]] = sparse_tensor.positions %[[TMP_arg0]] {level = 1 : index}
+//       CHECK:  %[[TMP_4:.*]] = sparse_tensor.coordinates %[[TMP_arg0]] {level = 1 : index}
 //       CHECK:  %[[TMP_5:.*]] = sparse_tensor.values %[[TMP_arg0]]
 //       CHECK:  %[[TMP_6:.*]] = memref.load %[[TMP_1]][%[[TMP_c0]]] : memref<?xindex>
 //       CHECK:  %[[TMP_7:.*]] = memref.load %[[TMP_1]][%[[TMP_c1]]] : memref<?xindex>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_concat_codegen.mlir b/mlir/test/Dialect/SparseTensor/sparse_concat_codegen.mlir
index 560189486bb79..434807a5b5ad2 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_concat_codegen.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_concat_codegen.mlir
@@ -16,10 +16,10 @@
 //       CHECK:  %[[TMP_c5:.*]] = arith.constant 5 : index
 //       CHECK:  %[[TMP_c2:.*]] = arith.constant 2 : index
 //       CHECK:  %[[TMP_0:.*]] = bufferization.alloc_tensor() : tensor<9x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_1:.*]] = sparse_tensor.pointers %[[TMP_arg0]] {dimension = 0 : index} : tensor<2x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_2:.*]] = sparse_tensor.indices %[[TMP_arg0]] {dimension = 0 : index} : tensor<2x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_3:.*]] = sparse_tensor.pointers %[[TMP_arg0]] {dimension = 1 : index} : tensor<2x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_4:.*]] = sparse_tensor.indices %[[TMP_arg0]] {dimension = 1 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_1:.*]] = sparse_tensor.positions %[[TMP_arg0]] {level = 0 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_2:.*]] = sparse_tensor.coordinates %[[TMP_arg0]] {level = 0 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_3:.*]] = sparse_tensor.positions %[[TMP_arg0]] {level = 1 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_4:.*]] = sparse_tensor.coordinates %[[TMP_arg0]] {level = 1 : index} : tensor<2x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_5:.*]] = sparse_tensor.values %[[TMP_arg0]] : tensor<2x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_6:.*]] = memref.load %[[TMP_1]][%[[TMP_c0]]] : memref<?xindex>
 //       CHECK:  %[[TMP_7:.*]] = memref.load %[[TMP_1]][%[[TMP_c1]]] : memref<?xindex>
@@ -36,10 +36,10 @@
 //       CHECK:    }
 //       CHECK:    scf.yield %[[RET_4]]
 //       CHECK:  }
-//       CHECK:  %[[TMP_8:.*]] = sparse_tensor.pointers %[[TMP_arg1]] {dimension = 0 : index} : tensor<3x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_9:.*]] = sparse_tensor.indices %[[TMP_arg1]] {dimension = 0 : index} : tensor<3x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_10:.*]] = sparse_tensor.pointers %[[TMP_arg1]] {dimension = 1 : index} : tensor<3x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_11:.*]] = sparse_tensor.indices %[[TMP_arg1]] {dimension = 1 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_8:.*]] = sparse_tensor.positions %[[TMP_arg1]] {level = 0 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_9:.*]] = sparse_tensor.coordinates %[[TMP_arg1]] {level = 0 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_10:.*]] = sparse_tensor.positions %[[TMP_arg1]] {level = 1 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_11:.*]] = sparse_tensor.coordinates %[[TMP_arg1]] {level = 1 : index} : tensor<3x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_12:.*]] = sparse_tensor.values %[[TMP_arg1]] : tensor<3x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_13:.*]] = memref.load %[[TMP_8]][%[[TMP_c0]]] : memref<?xindex>
 //       CHECK:  %[[TMP_14:.*]] = memref.load %[[TMP_8]][%[[TMP_c1]]] : memref<?xindex>
@@ -57,10 +57,10 @@
 //       CHECK:    }
 //       CHECK:    scf.yield %[[RET_5]]
 //       CHECK:  }
-//       CHECK:  %[[TMP_15:.*]] = sparse_tensor.pointers %[[TMP_arg2]] {dimension = 0 : index} : tensor<4x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_16:.*]] = sparse_tensor.indices %[[TMP_arg2]] {dimension = 0 : index} : tensor<4x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_17:.*]] = sparse_tensor.pointers %[[TMP_arg2]] {dimension = 1 : index} : tensor<4x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_18:.*]] = sparse_tensor.indices %[[TMP_arg2]] {dimension = 1 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_15:.*]] = sparse_tensor.positions %[[TMP_arg2]] {level = 0 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_16:.*]] = sparse_tensor.coordinates %[[TMP_arg2]] {level = 0 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_17:.*]] = sparse_tensor.positions %[[TMP_arg2]] {level = 1 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_18:.*]] = sparse_tensor.coordinates %[[TMP_arg2]] {level = 1 : index} : tensor<4x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_19:.*]] = sparse_tensor.values %[[TMP_arg2]] : tensor<4x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_20:.*]] = memref.load %[[TMP_15]][%[[TMP_c0]]] : memref<?xindex>
 //       CHECK:  %[[TMP_21:.*]] = memref.load %[[TMP_15]][%[[TMP_c1]]] : memref<?xindex>
@@ -102,10 +102,10 @@ func.func @concat_sparse_sparse(%arg0: tensor<2x4xf64, #DCSR>,
 //   CHECK-DAG:  %[[TMP_c9:.*]] = arith.constant 9 : index
 //   CHECK-DAG:  %[[TMP_c4:.*]] = arith.constant 4 : index
 //       CHECK:  %[[TMP_0:.*]] = bufferization.alloc_tensor(%[[TMP_c9]], %[[TMP_c4]]) : tensor<?x?xf64, #sparse_tensor
-//       CHECK:  %[[TMP_1:.*]] = sparse_tensor.pointers %[[TMP_arg0]] {dimension = 0 : index} : tensor<2x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_2:.*]] = sparse_tensor.indices %[[TMP_arg0]] {dimension = 0 : index} : tensor<2x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_3:.*]] = sparse_tensor.pointers %[[TMP_arg0]] {dimension = 1 : index} : tensor<2x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_4:.*]] = sparse_tensor.indices %[[TMP_arg0]] {dimension = 1 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_1:.*]] = sparse_tensor.positions %[[TMP_arg0]] {level = 0 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_2:.*]] = sparse_tensor.coordinates %[[TMP_arg0]] {level = 0 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_3:.*]] = sparse_tensor.positions %[[TMP_arg0]] {level = 1 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_4:.*]] = sparse_tensor.coordinates %[[TMP_arg0]] {level = 1 : index} : tensor<2x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_5:.*]] = sparse_tensor.values %[[TMP_arg0]] : tensor<2x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_6:.*]] = memref.load %[[TMP_1]][%[[TMP_c0]]] : memref<?xindex>
 //       CHECK:  %[[TMP_7:.*]] = memref.load %[[TMP_1]][%[[TMP_c1]]] : memref<?xindex>
@@ -122,10 +122,10 @@ func.func @concat_sparse_sparse(%arg0: tensor<2x4xf64, #DCSR>,
 //       CHECK:    }
 //       CHECK:    scf.yield %[[RET_4]]
 //       CHECK:  }
-//       CHECK:  %[[TMP_8:.*]] = sparse_tensor.pointers %[[TMP_arg1]] {dimension = 0 : index} : tensor<3x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_9:.*]] = sparse_tensor.indices %[[TMP_arg1]] {dimension = 0 : index} : tensor<3x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_10:.*]] = sparse_tensor.pointers %[[TMP_arg1]] {dimension = 1 : index} : tensor<3x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_11:.*]] = sparse_tensor.indices %[[TMP_arg1]] {dimension = 1 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_8:.*]] = sparse_tensor.positions %[[TMP_arg1]] {level = 0 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_9:.*]] = sparse_tensor.coordinates %[[TMP_arg1]] {level = 0 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_10:.*]] = sparse_tensor.positions %[[TMP_arg1]] {level = 1 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_11:.*]] = sparse_tensor.coordinates %[[TMP_arg1]] {level = 1 : index} : tensor<3x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_12:.*]] = sparse_tensor.values %[[TMP_arg1]] : tensor<3x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_13:.*]] = memref.load %[[TMP_8]][%[[TMP_c0]]] : memref<?xindex>
 //       CHECK:  %[[TMP_14:.*]] = memref.load %[[TMP_8]][%[[TMP_c1]]] : memref<?xindex>
@@ -143,10 +143,10 @@ func.func @concat_sparse_sparse(%arg0: tensor<2x4xf64, #DCSR>,
 //       CHECK:    }
 //       CHECK:    scf.yield %[[RET_5]]
 //       CHECK:  }
-//       CHECK:  %[[TMP_15:.*]] = sparse_tensor.pointers %[[TMP_arg2]] {dimension = 0 : index} : tensor<4x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_16:.*]] = sparse_tensor.indices %[[TMP_arg2]] {dimension = 0 : index} : tensor<4x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_17:.*]] = sparse_tensor.pointers %[[TMP_arg2]] {dimension = 1 : index} : tensor<4x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_18:.*]] = sparse_tensor.indices %[[TMP_arg2]] {dimension = 1 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_15:.*]] = sparse_tensor.positions %[[TMP_arg2]] {level = 0 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_16:.*]] = sparse_tensor.coordinates %[[TMP_arg2]] {level = 0 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_17:.*]] = sparse_tensor.positions %[[TMP_arg2]] {level = 1 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_18:.*]] = sparse_tensor.coordinates %[[TMP_arg2]] {level = 1 : index} : tensor<4x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_19:.*]] = sparse_tensor.values %[[TMP_arg2]] : tensor<4x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_20:.*]] = memref.load %[[TMP_15]][%[[TMP_c0]]] : memref<?xindex>
 //       CHECK:  %[[TMP_21:.*]] = memref.load %[[TMP_15]][%[[TMP_c1]]] : memref<?xindex>
@@ -190,10 +190,10 @@ func.func @concat_sparse_sparse_dynamic(%arg0: tensor<2x4xf64, #DCSR>,
 //   CHECK-DAG:  %[[TMP_d0:.*]] = arith.constant 0.000000e+00 : f64
 //       CHECK:  %[[A:.*]] = memref.alloc(%[[TMP_c9]], %[[TMP_c4]]) : memref<?x?xf64>
 //       CHECK:  linalg.fill ins(%[[TMP_d0]] : f64) outs(%[[A]] : memref<?x?xf64>)
-//       CHECK:  %[[TMP_1:.*]] = sparse_tensor.pointers %[[TMP_arg0]] {dimension = 0 : index} : tensor<2x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_2:.*]] = sparse_tensor.indices %[[TMP_arg0]] {dimension = 0 : index} : tensor<2x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_3:.*]] = sparse_tensor.pointers %[[TMP_arg0]] {dimension = 1 : index} : tensor<2x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_4:.*]] = sparse_tensor.indices %[[TMP_arg0]] {dimension = 1 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_1:.*]] = sparse_tensor.positions %[[TMP_arg0]] {level = 0 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_2:.*]] = sparse_tensor.coordinates %[[TMP_arg0]] {level = 0 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_3:.*]] = sparse_tensor.positions %[[TMP_arg0]] {level = 1 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_4:.*]] = sparse_tensor.coordinates %[[TMP_arg0]] {level = 1 : index} : tensor<2x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_5:.*]] = sparse_tensor.values %[[TMP_arg0]] : tensor<2x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_6:.*]] = memref.load %[[TMP_1]][%[[TMP_c0]]] : memref<?xindex>
 //       CHECK:  %[[TMP_7:.*]] = memref.load %[[TMP_1]][%[[TMP_c1]]] : memref<?xindex>
@@ -208,10 +208,10 @@ func.func @concat_sparse_sparse_dynamic(%arg0: tensor<2x4xf64, #DCSR>,
 //       CHECK:      memref.store %[[TMP_28]], %[[A]]{{\[}}%[[TMP_23]], %[[TMP_27]]] : memref<?x?xf64>
 //       CHECK:    }
 //       CHECK:  }
-//       CHECK:  %[[TMP_8:.*]] = sparse_tensor.pointers %[[TMP_arg1]] {dimension = 0 : index} : tensor<3x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_9:.*]] = sparse_tensor.indices %[[TMP_arg1]] {dimension = 0 : index} : tensor<3x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_10:.*]] = sparse_tensor.pointers %[[TMP_arg1]] {dimension = 1 : index} : tensor<3x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_11:.*]] = sparse_tensor.indices %[[TMP_arg1]] {dimension = 1 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_8:.*]] = sparse_tensor.positions %[[TMP_arg1]] {level = 0 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_9:.*]] = sparse_tensor.coordinates %[[TMP_arg1]] {level = 0 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_10:.*]] = sparse_tensor.positions %[[TMP_arg1]] {level = 1 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_11:.*]] = sparse_tensor.coordinates %[[TMP_arg1]] {level = 1 : index} : tensor<3x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_12:.*]] = sparse_tensor.values %[[TMP_arg1]] : tensor<3x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_13:.*]] = memref.load %[[TMP_8]][%[[TMP_c0]]] : memref<?xindex>
 //       CHECK:  %[[TMP_14:.*]] = memref.load %[[TMP_8]][%[[TMP_c1]]] : memref<?xindex>
@@ -227,10 +227,10 @@ func.func @concat_sparse_sparse_dynamic(%arg0: tensor<2x4xf64, #DCSR>,
 //       CHECK:      memref.store %[[TMP_28]], %[[A]]{{\[}}%[[TMP_29]], %[[TMP_27]]] : memref<?x?xf64>
 //       CHECK:    }
 //       CHECK:  }
-//       CHECK:  %[[TMP_15:.*]] = sparse_tensor.pointers %[[TMP_arg2]] {dimension = 0 : index} : tensor<4x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_16:.*]] = sparse_tensor.indices %[[TMP_arg2]] {dimension = 0 : index} : tensor<4x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_17:.*]] = sparse_tensor.pointers %[[TMP_arg2]] {dimension = 1 : index} : tensor<4x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_18:.*]] = sparse_tensor.indices %[[TMP_arg2]] {dimension = 1 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_15:.*]] = sparse_tensor.positions %[[TMP_arg2]] {level = 0 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_16:.*]] = sparse_tensor.coordinates %[[TMP_arg2]] {level = 0 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_17:.*]] = sparse_tensor.positions %[[TMP_arg2]] {level = 1 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_18:.*]] = sparse_tensor.coordinates %[[TMP_arg2]] {level = 1 : index} : tensor<4x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_19:.*]] = sparse_tensor.values %[[TMP_arg2]] : tensor<4x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_20:.*]] = memref.load %[[TMP_15]][%[[TMP_c0]]] : memref<?xindex>
 //       CHECK:  %[[TMP_21:.*]] = memref.load %[[TMP_15]][%[[TMP_c1]]] : memref<?xindex>
@@ -275,10 +275,10 @@ func.func @concat_sparse_sparse_dense(%arg0: tensor<2x4xf64, #DCSR>,
 //       CHECK:  memref.store %[[TMP_c9]], %[[DIM_0]][%[[TMP_c0]]] : memref<2xindex>
 //       CHECK:  memref.store %[[TMP_c4]], %[[DIM_0]][%[[TMP_c1]]] : memref<2xindex>
 //       CHECK:  %[[VAL_1:.*]] = memref.reshape %[[VAL_0]](%[[DIM_0]]) : (memref<?xf64>, memref<2xindex>) -> memref<?x?xf64>
-//       CHECK:  %[[TMP_1:.*]] = sparse_tensor.pointers %[[TMP_arg0]] {dimension = 0 : index} : tensor<2x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_2:.*]] = sparse_tensor.indices %[[TMP_arg0]] {dimension = 0 : index} : tensor<2x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_3:.*]] = sparse_tensor.pointers %[[TMP_arg0]] {dimension = 1 : index} : tensor<2x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_4:.*]] = sparse_tensor.indices %[[TMP_arg0]] {dimension = 1 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_1:.*]] = sparse_tensor.positions %[[TMP_arg0]] {level = 0 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_2:.*]] = sparse_tensor.coordinates %[[TMP_arg0]] {level = 0 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_3:.*]] = sparse_tensor.positions %[[TMP_arg0]] {level = 1 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_4:.*]] = sparse_tensor.coordinates %[[TMP_arg0]] {level = 1 : index} : tensor<2x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_5:.*]] = sparse_tensor.values %[[TMP_arg0]] : tensor<2x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_6:.*]] = memref.load %[[TMP_1]][%[[TMP_c0]]] : memref<?xindex>
 //       CHECK:  %[[TMP_7:.*]] = memref.load %[[TMP_1]][%[[TMP_c1]]] : memref<?xindex>
@@ -293,10 +293,10 @@ func.func @concat_sparse_sparse_dense(%arg0: tensor<2x4xf64, #DCSR>,
 //       CHECK:      memref.store %[[TMP_28]], %[[VAL_1]][%[[TMP_23]], %[[TMP_27]]] : memref<?x?xf64>
 //       CHECK:    }
 //       CHECK:  }
-//       CHECK:  %[[TMP_8:.*]] = sparse_tensor.pointers %[[TMP_arg1]] {dimension = 0 : index} : tensor<3x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_9:.*]] = sparse_tensor.indices %[[TMP_arg1]] {dimension = 0 : index} : tensor<3x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_10:.*]] = sparse_tensor.pointers %[[TMP_arg1]] {dimension = 1 : index} : tensor<3x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_11:.*]] = sparse_tensor.indices %[[TMP_arg1]] {dimension = 1 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_8:.*]] = sparse_tensor.positions %[[TMP_arg1]] {level = 0 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_9:.*]] = sparse_tensor.coordinates %[[TMP_arg1]] {level = 0 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_10:.*]] = sparse_tensor.positions %[[TMP_arg1]] {level = 1 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_11:.*]] = sparse_tensor.coordinates %[[TMP_arg1]] {level = 1 : index} : tensor<3x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_12:.*]] = sparse_tensor.values %[[TMP_arg1]] : tensor<3x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_13:.*]] = memref.load %[[TMP_8]][%[[TMP_c0]]] : memref<?xindex>
 //       CHECK:  %[[TMP_14:.*]] = memref.load %[[TMP_8]][%[[TMP_c1]]] : memref<?xindex>
@@ -312,10 +312,10 @@ func.func @concat_sparse_sparse_dense(%arg0: tensor<2x4xf64, #DCSR>,
 //       CHECK:      memref.store %[[TMP_28]], %[[VAL_1]][%[[TMP_29]], %[[TMP_27]]] : memref<?x?xf64>
 //       CHECK:    }
 //       CHECK:  }
-//       CHECK:  %[[TMP_15:.*]] = sparse_tensor.pointers %[[TMP_arg2]] {dimension = 0 : index} : tensor<4x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_16:.*]] = sparse_tensor.indices %[[TMP_arg2]] {dimension = 0 : index} : tensor<4x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_17:.*]] = sparse_tensor.pointers %[[TMP_arg2]] {dimension = 1 : index} : tensor<4x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_18:.*]] = sparse_tensor.indices %[[TMP_arg2]] {dimension = 1 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_15:.*]] = sparse_tensor.positions %[[TMP_arg2]] {level = 0 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_16:.*]] = sparse_tensor.coordinates %[[TMP_arg2]] {level = 0 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_17:.*]] = sparse_tensor.positions %[[TMP_arg2]] {level = 1 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_18:.*]] = sparse_tensor.coordinates %[[TMP_arg2]] {level = 1 : index} : tensor<4x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_19:.*]] = sparse_tensor.values %[[TMP_arg2]] : tensor<4x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_20:.*]] = memref.load %[[TMP_15]][%[[TMP_c0]]] : memref<?xindex>
 //       CHECK:  %[[TMP_21:.*]] = memref.load %[[TMP_15]][%[[TMP_c1]]] : memref<?xindex>
@@ -360,10 +360,10 @@ func.func @concat_sparse_sparse_annotated_dense(%arg0: tensor<2x4xf64, #DCSR>,
 //       CHECK:  memref.store %[[TMP_c4]], %[[DIM_0]][%[[TMP_c0]]] : memref<2xindex>
 //       CHECK:  memref.store %[[TMP_c9]], %[[DIM_0]][%[[TMP_c1]]] : memref<2xindex>
 //       CHECK:  %[[VAL_1:.*]] = memref.reshape %[[VAL_0]](%[[DIM_0]]) : (memref<?xf64>, memref<2xindex>) -> memref<?x?xf64>
-//       CHECK:  %[[TMP_1:.*]] = sparse_tensor.pointers %[[TMP_arg0]] {dimension = 0 : index} : tensor<2x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_2:.*]] = sparse_tensor.indices %[[TMP_arg0]] {dimension = 0 : index} : tensor<2x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_3:.*]] = sparse_tensor.pointers %[[TMP_arg0]] {dimension = 1 : index} : tensor<2x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_4:.*]] = sparse_tensor.indices %[[TMP_arg0]] {dimension = 1 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_1:.*]] = sparse_tensor.positions %[[TMP_arg0]] {level = 0 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_2:.*]] = sparse_tensor.coordinates %[[TMP_arg0]] {level = 0 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_3:.*]] = sparse_tensor.positions %[[TMP_arg0]] {level = 1 : index} : tensor<2x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_4:.*]] = sparse_tensor.coordinates %[[TMP_arg0]] {level = 1 : index} : tensor<2x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_5:.*]] = sparse_tensor.values %[[TMP_arg0]] : tensor<2x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_6:.*]] = memref.load %[[TMP_1]][%[[TMP_c0]]] : memref<?xindex>
 //       CHECK:  %[[TMP_7:.*]] = memref.load %[[TMP_1]][%[[TMP_c1]]] : memref<?xindex>
@@ -378,10 +378,10 @@ func.func @concat_sparse_sparse_annotated_dense(%arg0: tensor<2x4xf64, #DCSR>,
 //       CHECK:      memref.store %[[TMP_28]], %[[VAL_1]][%[[TMP_27]], %[[TMP_23]]] : memref<?x?xf64>
 //       CHECK:    }
 //       CHECK:  }
-//       CHECK:  %[[TMP_8:.*]] = sparse_tensor.pointers %[[TMP_arg1]] {dimension = 0 : index} : tensor<3x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_9:.*]] = sparse_tensor.indices %[[TMP_arg1]] {dimension = 0 : index} : tensor<3x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_10:.*]] = sparse_tensor.pointers %[[TMP_arg1]] {dimension = 1 : index} : tensor<3x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_11:.*]] = sparse_tensor.indices %[[TMP_arg1]] {dimension = 1 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_8:.*]] = sparse_tensor.positions %[[TMP_arg1]] {level = 0 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_9:.*]] = sparse_tensor.coordinates %[[TMP_arg1]] {level = 0 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_10:.*]] = sparse_tensor.positions %[[TMP_arg1]] {level = 1 : index} : tensor<3x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_11:.*]] = sparse_tensor.coordinates %[[TMP_arg1]] {level = 1 : index} : tensor<3x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_12:.*]] = sparse_tensor.values %[[TMP_arg1]] : tensor<3x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_13:.*]] = memref.load %[[TMP_8]][%[[TMP_c0]]] : memref<?xindex>
 //       CHECK:  %[[TMP_14:.*]] = memref.load %[[TMP_8]][%[[TMP_c1]]] : memref<?xindex>
@@ -397,10 +397,10 @@ func.func @concat_sparse_sparse_annotated_dense(%arg0: tensor<2x4xf64, #DCSR>,
 //       CHECK:      memref.store %[[TMP_28]], %[[VAL_1]][%[[TMP_27]], %[[TMP_29]]] : memref<?x?xf64>
 //       CHECK:    }
 //       CHECK:  }
-//       CHECK:  %[[TMP_15:.*]] = sparse_tensor.pointers %[[TMP_arg2]] {dimension = 0 : index} : tensor<4x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_16:.*]] = sparse_tensor.indices %[[TMP_arg2]] {dimension = 0 : index} : tensor<4x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_17:.*]] = sparse_tensor.pointers %[[TMP_arg2]] {dimension = 1 : index} : tensor<4x4xf64, #sparse_tensor
-//       CHECK:  %[[TMP_18:.*]] = sparse_tensor.indices %[[TMP_arg2]] {dimension = 1 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_15:.*]] = sparse_tensor.positions %[[TMP_arg2]] {level = 0 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_16:.*]] = sparse_tensor.coordinates %[[TMP_arg2]] {level = 0 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_17:.*]] = sparse_tensor.positions %[[TMP_arg2]] {level = 1 : index} : tensor<4x4xf64, #sparse_tensor
+//       CHECK:  %[[TMP_18:.*]] = sparse_tensor.coordinates %[[TMP_arg2]] {level = 1 : index} : tensor<4x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_19:.*]] = sparse_tensor.values %[[TMP_arg2]] : tensor<4x4xf64, #sparse_tensor
 //       CHECK:  %[[TMP_20:.*]] = memref.load %[[TMP_15]][%[[TMP_c0]]] : memref<?xindex>
 //       CHECK:  %[[TMP_21:.*]] = memref.load %[[TMP_15]][%[[TMP_c1]]] : memref<?xindex>
@@ -427,4 +427,4 @@ func.func @concat_sparse_sparse_annotated_dense_permute(%arg0: tensor<2x4xf64, #
            tensor<3x4xf64, #DCSR>,
            tensor<4x4xf64, #DCSR> to tensor<?x?xf64, #DENSE_P>
     return %0 : tensor<?x?xf64, #DENSE_P>
-}
\ No newline at end of file
+}
diff --git a/mlir/test/Dialect/SparseTensor/sparse_extract_slice.mlir b/mlir/test/Dialect/SparseTensor/sparse_extract_slice.mlir
new file mode 100644
index 0000000000000..745b0a8f376d5
--- /dev/null
+++ b/mlir/test/Dialect/SparseTensor/sparse_extract_slice.mlir
@@ -0,0 +1,33 @@
+// RUN: mlir-opt %s --sparse-tensor-codegen --cse |  FileCheck %s
+
+#CSR = #sparse_tensor.encoding<{
+  dimLevelType = [ "dense", "compressed" ]
+}>
+
+#CSR_SLICE = #sparse_tensor.encoding<{
+  dimLevelType = [ "dense", "compressed" ],
+  slice = [ (0, 4, 1), (0, 8, 1) ]
+}>
+
+// CHECK-LABEL:   func.func @sparse_slice(
+// CHECK-SAME:                            %[[VAL_0:.*0]]: memref<?xindex>,
+// CHECK-SAME:                            %[[VAL_1:.*1]]: memref<?xindex>,
+// CHECK-SAME:                            %[[VAL_2:.*2]]: memref<?xf64>,
+// CHECK-SAME:                            %[[VAL_3:.*3]]: !sparse_tensor.storage_specifier<#sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>>)
+// CHECK:           %[[VAL_4:.*]] = sparse_tensor.storage_specifier.init with %[[VAL_3]]
+// CHECK:           %[[VAL_5:.*]] = arith.constant 0 : index
+// CHECK:           %[[VAL_6:.*]] = arith.constant 4 : index
+// CHECK:           %[[VAL_7:.*]] = arith.constant 1 : index
+// CHECK:           %[[VAL_8:.*]] = sparse_tensor.storage_specifier.set %[[VAL_4]]  dim_offset at 0 with %[[VAL_5]]
+// CHECK:           %[[VAL_9:.*]] = sparse_tensor.storage_specifier.set %[[VAL_8]]  lvl_sz at 0 with %[[VAL_6]]
+// CHECK:           %[[VAL_10:.*]] = sparse_tensor.storage_specifier.set %[[VAL_9]]  dim_stride at 0 with %[[VAL_7]]
+// CHECK:           %[[VAL_11:.*]] = arith.constant 8 : index
+// CHECK:           %[[VAL_12:.*]] = sparse_tensor.storage_specifier.set %[[VAL_10]]  dim_offset at 1 with %[[VAL_5]]
+// CHECK:           %[[VAL_13:.*]] = sparse_tensor.storage_specifier.set %[[VAL_12]]  lvl_sz at 1 with %[[VAL_11]]
+// CHECK:           %[[VAL_14:.*]] = sparse_tensor.storage_specifier.set %[[VAL_13]]  dim_stride at 1 with %[[VAL_7]]
+// CHECK:           return %[[VAL_0]], %[[VAL_1]], %[[VAL_2]], %[[VAL_14]]
+func.func @sparse_slice(%t1 : tensor<8x8xf64, #CSR>) -> tensor<4x8xf64, #CSR_SLICE> {
+  %a1 = tensor.extract_slice %t1[0, 0][4, 8][1, 1] : tensor<8x8xf64, #CSR> to
+                                                     tensor<4x8xf64, #CSR_SLICE>
+  return %a1 : tensor<4x8xf64, #CSR_SLICE>
+}
diff --git a/mlir/test/Dialect/SparseTensor/sparse_fill_zero.mlir b/mlir/test/Dialect/SparseTensor/sparse_fill_zero.mlir
index 7571e3dbb9087..06b3067f407e7 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_fill_zero.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_fill_zero.mlir
@@ -41,15 +41,15 @@
 // CHECK:           %[[VAL_25:.*]] = memref.cast %[[VAL_24]] : memref<300xindex> to memref<?xindex>
 // CHECK:           linalg.fill ins(%[[F0]] : f64) outs(%[[VAL_20]] : memref<300xf64>)
 // CHECK:           linalg.fill ins(%[[False]] : i1) outs(%[[VAL_22]] : memref<300xi1>)
-// CHECK:           %[[VAL_26:.*]] = call @sparsePointers0(%[[Arg0]], %[[I0]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
-// CHECK:           %[[VAL_27:.*]] = call @sparseIndices0(%[[Arg0]], %[[I0]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
-// CHECK:           %[[VAL_28:.*]] = call @sparsePointers0(%[[Arg0]], %[[I1]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
-// CHECK:           %[[VAL_29:.*]] = call @sparseIndices0(%[[Arg0]], %[[I1]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK:           %[[VAL_26:.*]] = call @sparsePositions0(%[[Arg0]], %[[I0]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK:           %[[VAL_27:.*]] = call @sparseCoordinates0(%[[Arg0]], %[[I0]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK:           %[[VAL_28:.*]] = call @sparsePositions0(%[[Arg0]], %[[I1]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK:           %[[VAL_29:.*]] = call @sparseCoordinates0(%[[Arg0]], %[[I1]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
 // CHECK:           %[[VAL_30:.*]] = call @sparseValuesF64(%[[Arg0]]) : (!llvm.ptr<i8>) -> memref<?xf64>
-// CHECK:           %[[VAL_31:.*]] = call @sparsePointers0(%[[Arg1]], %[[I0]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
-// CHECK:           %[[VAL_32:.*]] = call @sparseIndices0(%[[Arg1]], %[[I0]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
-// CHECK:           %[[VAL_33:.*]] = call @sparsePointers0(%[[Arg1]], %[[I1]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
-// CHECK:           %[[VAL_34:.*]] = call @sparseIndices0(%[[Arg1]], %[[I1]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK:           %[[VAL_31:.*]] = call @sparsePositions0(%[[Arg1]], %[[I0]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK:           %[[VAL_32:.*]] = call @sparseCoordinates0(%[[Arg1]], %[[I0]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK:           %[[VAL_33:.*]] = call @sparsePositions0(%[[Arg1]], %[[I1]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK:           %[[VAL_34:.*]] = call @sparseCoordinates0(%[[Arg1]], %[[I1]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
 // CHECK:           %[[VAL_35:.*]] = call @sparseValuesF64(%[[Arg1]]) : (!llvm.ptr<i8>) -> memref<?xf64>
 // CHECK:           %[[VAL_36:.*]] = memref.load %[[VAL_26]]{{\[}}%[[I0]]] : memref<?xindex>
 // CHECK:           %[[VAL_37:.*]] = memref.load %[[VAL_26]]{{\[}}%[[I1]]] : memref<?xindex>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_foreach.mlir b/mlir/test/Dialect/SparseTensor/sparse_foreach.mlir
new file mode 100644
index 0000000000000..8d72e9455c187
--- /dev/null
+++ b/mlir/test/Dialect/SparseTensor/sparse_foreach.mlir
@@ -0,0 +1,141 @@
+// RUN: mlir-opt %s --post-sparsification-rewrite="enable-runtime-library=false enable-foreach=true" --canonicalize | FileCheck %s
+
+// CHECK-LABEL: func.func @sparse_foreach_constant
+// CHECK-DAG:   %[[C1:.*]] = arith.constant 1 : index
+// CHECK-DAG:   %[[V1:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK-DAG:   %[[C2:.*]] = arith.constant 2 : index
+// CHECK-DAG:   %[[V3:.*]] = arith.constant 1.000000e+00 : f32
+// CHECK-DAG:   %[[V4:.*]] = arith.constant 6.000000e+00 : f32
+//               (1, 1) -> (2, 1) -> (2, 2)
+// CHECK-NEXT:  "test.use"(%[[C1]], %[[C1]], %[[V1]])
+// CHECK-NEXT:  "test.use"(%[[C2]], %[[C1]], %[[V3]])
+// CHECK-NEXT:  "test.use"(%[[C1]], %[[C2]], %[[V4]])
+//               (1, 1) -> (1, 2) -> (2, 1)
+// CHECK-NEXT:  "test.use"(%[[C1]], %[[C1]], %[[V1]])
+// CHECK-NEXT:  "test.use"(%[[C1]], %[[C2]], %[[V4]])
+// CHECK-NEXT:  "test.use"(%[[C2]], %[[C1]], %[[V3]])
+func.func @sparse_foreach_constant() -> () {
+  %cst = arith.constant sparse<[[2, 1], [1, 1], [1, 2]], [1.0, 5.0, 6.0]> : tensor<8x7xf32>
+  // Make use the sparse constant are properly sorted based on the requested order.
+  sparse_tensor.foreach in %cst { order = affine_map<(d0, d1) -> (d1, d0)> } : tensor<8x7xf32> do {
+  ^bb0(%arg0: index, %arg1: index, %arg2: f32):
+    "test.use" (%arg0, %arg1, %arg2): (index,index,f32)->()
+  }
+  sparse_tensor.foreach in %cst : tensor<8x7xf32> do {
+  ^bb0(%arg0: index, %arg1: index, %arg2: f32):
+    "test.use" (%arg0, %arg1, %arg2): (index,index,f32)->()
+  }
+  return
+}
+
+#CSR_SLICE = #sparse_tensor.encoding<{
+  dimLevelType = [ "compressed", "compressed" ],
+  slice = [ (0, 4, 1), (2, 4, 1) ]
+}>
+
+#CSR_SLICE_DYN = #sparse_tensor.encoding<{
+  dimLevelType = [ "compressed", "compressed" ],
+  slice = [ (?, ?, ?), (?, ?, ?) ]
+}>
+
+
+// CHECK-LABEL:   func.func @foreach_print_slice_dyn(
+// CHECK-SAME:                                       %[[VAL_0:.*]]: tensor<?x?xf64,
+// CHECK-DAG:       %[[VAL_1:.*]] = arith.constant 0 : index
+// CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 1 : index
+// CHECK-DAG:       %[[VAL_3:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<?x?xf64,
+// CHECK-DAG:       %[[VAL_4:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<?x?xf64,
+// CHECK-DAG:       %[[VAL_5:.*]] = tensor.dim %[[VAL_0]], %[[VAL_1]] : tensor<?x?xf64,
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.slice.offset %[[VAL_0]] at 0 : tensor<?x?xf64,
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.slice.stride %[[VAL_0]] at 0 : tensor<?x?xf64,
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<?x?xf64,
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<?x?xf64,
+// CHECK-DAG:       %[[VAL_10:.*]] = tensor.dim %[[VAL_0]], %[[VAL_2]] : tensor<?x?xf64,
+// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.slice.offset %[[VAL_0]] at 1 : tensor<?x?xf64,
+// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.slice.stride %[[VAL_0]] at 1 : tensor<?x?xf64,
+// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<?x?xf64,
+// CHECK:           %[[VAL_14:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_1]]] : memref<?xindex>
+// CHECK:           %[[VAL_15:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_2]]] : memref<?xindex>
+// CHECK:           scf.for %[[VAL_16:.*]] = %[[VAL_14]] to %[[VAL_15]] step %[[VAL_2]] {
+// CHECK:             %[[VAL_17:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_16]]] : memref<?xindex>
+// CHECK:             %[[VAL_18:.*]] = arith.subi %[[VAL_17]], %[[VAL_6]] : index
+// CHECK:             %[[VAL_19:.*]] = arith.remui %[[VAL_18]], %[[VAL_7]] : index
+// CHECK:             %[[VAL_20:.*]] = arith.divui %[[VAL_18]], %[[VAL_7]] : index
+// CHECK:             %[[VAL_21:.*]] = arith.cmpi uge, %[[VAL_17]], %[[VAL_6]] : index
+// CHECK:             %[[VAL_22:.*]] = arith.cmpi ult, %[[VAL_20]], %[[VAL_5]] : index
+// CHECK:             %[[VAL_23:.*]] = arith.cmpi eq, %[[VAL_19]], %[[VAL_1]] : index
+// CHECK:             %[[VAL_24:.*]] = arith.andi %[[VAL_21]], %[[VAL_22]] : i1
+// CHECK:             %[[VAL_25:.*]] = arith.andi %[[VAL_24]], %[[VAL_23]] : i1
+// CHECK:             scf.if %[[VAL_25]] {
+// CHECK:               %[[VAL_26:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_16]]] : memref<?xindex>
+// CHECK:               %[[VAL_27:.*]] = arith.addi %[[VAL_16]], %[[VAL_2]] : index
+// CHECK:               %[[VAL_28:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_27]]] : memref<?xindex>
+// CHECK:               scf.for %[[VAL_29:.*]] = %[[VAL_26]] to %[[VAL_28]] step %[[VAL_2]] {
+// CHECK:                 %[[VAL_30:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_29]]] : memref<?xindex>
+// CHECK:                 %[[VAL_31:.*]] = arith.subi %[[VAL_30]], %[[VAL_11]] : index
+// CHECK:                 %[[VAL_32:.*]] = arith.remui %[[VAL_31]], %[[VAL_12]] : index
+// CHECK:                 %[[VAL_33:.*]] = arith.divui %[[VAL_31]], %[[VAL_12]] : index
+// CHECK:                 %[[VAL_34:.*]] = arith.cmpi uge, %[[VAL_30]], %[[VAL_11]] : index
+// CHECK:                 %[[VAL_35:.*]] = arith.cmpi ult, %[[VAL_33]], %[[VAL_10]] : index
+// CHECK:                 %[[VAL_36:.*]] = arith.cmpi eq, %[[VAL_32]], %[[VAL_1]] : index
+// CHECK:                 %[[VAL_37:.*]] = arith.andi %[[VAL_34]], %[[VAL_35]] : i1
+// CHECK:                 %[[VAL_38:.*]] = arith.andi %[[VAL_37]], %[[VAL_36]] : i1
+// CHECK:                 scf.if %[[VAL_38]] {
+// CHECK:                   %[[VAL_39:.*]] = memref.load %[[VAL_13]]{{\[}}%[[VAL_29]]] : memref<?xf64>
+// CHECK:                   "test.use"(%[[VAL_39]]) : (f64) -> ()
+// CHECK:                 }
+// CHECK:               }
+// CHECK:             }
+// CHECK:           }
+// CHECK:           return
+//
+func.func @foreach_print_slice_dyn(%A: tensor<?x?xf64, #CSR_SLICE_DYN>) {
+  sparse_tensor.foreach in %A : tensor<?x?xf64, #CSR_SLICE_DYN> do {
+  ^bb0(%1: index, %2: index, %v: f64) :
+    "test.use" (%v) : (f64) -> ()
+  }
+  return
+}
+
+// CHECK-LABEL:   func.func @foreach_print_slice(
+// CHECK-SAME:                                   %[[VAL_0:.*]]: tensor<4x4xf64,
+// CHECK-DAG:       %[[VAL_1:.*]] = arith.constant 4 : index
+// CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 2 : index
+// CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 0 : index
+// CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1 : index
+// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<4x4xf64,
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<4x4xf64,
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<4x4xf64,
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<4x4xf64,
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<4x4xf64,
+// CHECK-DAG:       %[[VAL_10:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_3]]] : memref<?xindex>
+// CHECK:           %[[VAL_11:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_4]]] : memref<?xindex>
+// CHECK:           scf.for %[[VAL_12:.*]] = %[[VAL_10]] to %[[VAL_11]] step %[[VAL_4]] {
+// CHECK:             %[[VAL_13:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_12]]] : memref<?xindex>
+// CHECK:             %[[VAL_14:.*]] = arith.cmpi ult, %[[VAL_13]], %[[VAL_1]] : index
+// CHECK:             scf.if %[[VAL_14]] {
+// CHECK:               %[[VAL_15:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_12]]] : memref<?xindex>
+// CHECK:               %[[VAL_16:.*]] = arith.addi %[[VAL_12]], %[[VAL_4]] : index
+// CHECK:               %[[VAL_17:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_16]]] : memref<?xindex>
+// CHECK:               scf.for %[[VAL_18:.*]] = %[[VAL_15]] to %[[VAL_17]] step %[[VAL_4]] {
+// CHECK:                 %[[VAL_19:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_18]]] : memref<?xindex>
+// CHECK:                 %[[VAL_20:.*]] = arith.subi %[[VAL_19]], %[[VAL_2]] : index
+// CHECK:                 %[[VAL_21:.*]] = arith.cmpi uge, %[[VAL_19]], %[[VAL_2]] : index
+// CHECK:                 %[[VAL_22:.*]] = arith.cmpi ult, %[[VAL_20]], %[[VAL_1]] : index
+// CHECK:                 %[[VAL_23:.*]] = arith.andi %[[VAL_21]], %[[VAL_22]] : i1
+// CHECK:                 scf.if %[[VAL_23]] {
+// CHECK:                   %[[VAL_24:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_18]]] : memref<?xf64>
+// CHECK:                   "test.use"(%[[VAL_24]]) : (f64) -> ()
+// CHECK:                 }
+// CHECK:               }
+// CHECK:             }
+// CHECK:           }
+// CHECK:           return
+//
+func.func @foreach_print_slice(%A: tensor<4x4xf64, #CSR_SLICE>) {
+  sparse_tensor.foreach in %A : tensor<4x4xf64, #CSR_SLICE> do {
+  ^bb0(%1: index, %2: index, %v: f64) :
+    "test.use" (%v) : (f64) -> ()
+  }
+  return
+}
\ No newline at end of file
diff --git a/mlir/test/Dialect/SparseTensor/sparse_fp_ops.mlir b/mlir/test/Dialect/SparseTensor/sparse_fp_ops.mlir
index 975f160ad73fc..0d612c4bc3113 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_fp_ops.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_fp_ops.mlir
@@ -35,8 +35,8 @@
 // CHECK-SAME:    %[[VAL_1:.*]]: tensor<32xf64>) -> tensor<32xf64> {
 // CHECK-DAG:     %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-DAG:     %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK:         %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:         %[[VAL_5:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:         %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:         %[[VAL_5:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:         %[[VAL_6:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xf64>
 // CHECK:         %[[VAL_7:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xf64>
 // CHECK:         %[[VAL_8:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_2]]] : memref<?xindex>
@@ -67,8 +67,8 @@ func.func @abs(%arga: tensor<32xf64, #SV>,
 // CHECK-SAME:    %[[VAL_1:.*]]: tensor<32xf64>) -> tensor<32xf64> {
 // CHECK-DAG:     %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-DAG:     %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK:         %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:         %[[VAL_5:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:         %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:         %[[VAL_5:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:         %[[VAL_6:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xf64>
 // CHECK:         %[[VAL_7:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xf64>
 // CHECK:         %[[VAL_8:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_2]]] : memref<?xindex>
@@ -99,8 +99,8 @@ func.func @ceil(%arga: tensor<32xf64, #SV>,
 // CHECK-SAME:    %[[VAL_1:.*]]: tensor<32xf64>) -> tensor<32xf64> {
 // CHECK-DAG:     %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-DAG:     %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK:         %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:         %[[VAL_5:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:         %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:         %[[VAL_5:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:         %[[VAL_6:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xf64>
 // CHECK:         %[[VAL_7:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xf64>
 // CHECK:         %[[VAL_8:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_2]]] : memref<?xindex>
@@ -131,8 +131,8 @@ func.func @floor(%arga: tensor<32xf64, #SV>,
 // CHECK-SAME:    %[[VAL_1:.*]]: tensor<32xf64>) -> tensor<32xf64> {
 // CHECK-DAG:     %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-DAG:     %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK:         %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:         %[[VAL_5:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:         %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:         %[[VAL_5:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:         %[[VAL_6:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xf64>
 // CHECK:         %[[VAL_7:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xf64>
 // CHECK:         %[[VAL_8:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_2]]] : memref<?xindex>
@@ -166,8 +166,8 @@ func.func @neg(%arga: tensor<32xf64, #SV>,
 // CHECK-DAG:     %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:     %[[VAL_5:.*]] = arith.constant true
 // CHECK-DAG:     %[[VAL_6:.*]] = arith.constant 1 : index
-// CHECK:         %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>>
-// CHECK:         %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK:         %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK:         %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK:         %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK:         %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xf64>
 // CHECK:         %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xf64>
@@ -226,8 +226,8 @@ func.func @add(%arga: tensor<32xf64, #SV>,
 // CHECK-DAG:     %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:     %[[VAL_5:.*]] = arith.constant true
 // CHECK-DAG:     %[[VAL_6:.*]] = arith.constant 1 : index
-// CHECK:         %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:         %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:         %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:         %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:         %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xf64>
 // CHECK:         %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xf64>
 // CHECK:         %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xf64>
@@ -286,8 +286,8 @@ func.func @sub(%arga: tensor<32xf64, #SV>,
 // CHECK-SAME:    %[[VAL_2:.*]]: tensor<32xf64>) -> tensor<32xf64> {
 // CHECK-DAG:     %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:     %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK:         %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>>
-// CHECK:         %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK:         %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK:         %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK:         %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK:         %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xf64>
 // CHECK:         %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xf64>
@@ -322,8 +322,8 @@ func.func @mul(%arga: tensor<32xf64, #SV>,
 // CHECK-DAG:     %[[VAL_2:.*]] = arith.constant 2.000000e+00 : f64
 // CHECK-DAG:     %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:     %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK:         %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>>
-// CHECK:         %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK:         %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK:         %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK:         %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xf64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK:         %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xf64>
 // CHECK:         %[[VAL_9:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_3]]] : memref<?xindex>
@@ -355,8 +355,8 @@ func.func @divbyc(%arga: tensor<32xf64, #SV>,
 // CHECK-DAG:       %[[VAL_1:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 1 : index
 // CHECK:           %[[VAL_3:.*]] = bufferization.alloc_tensor() : tensor<32xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>>
-// CHECK:           %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_5:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_5:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK:           %[[VAL_6:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf64>
 // CHECK:           %[[VAL_7:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_1]]] : memref<?xindex>
 // CHECK:           %[[VAL_8:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_2]]] : memref<?xindex>
@@ -403,8 +403,8 @@ func.func @zero_preserving_math(%arga: tensor<32xf64, #SV>) -> tensor<32xf64, #S
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 1 : index
 // CHECK:           %[[VAL_3:.*]] = complex.constant [0.000000e+00, 1.000000e+00] : complex<f64>
 // CHECK:           %[[VAL_4:.*]] = bufferization.alloc_tensor() : tensor<32xcomplex<f64>, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>>
-// CHECK:           %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xcomplex<f64>, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xcomplex<f64>, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xcomplex<f64>, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xcomplex<f64>, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK:           %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xcomplex<f64>, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xcomplex<f64>>
 // CHECK:           %[[VAL_8:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_1]]] : memref<?xindex>
 // CHECK:           %[[VAL_9:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_2]]] : memref<?xindex>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_index.mlir b/mlir/test/Dialect/SparseTensor/sparse_index.mlir
index 5cfa3ac9b6a42..7ffd9c3ac14fe 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_index.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_index.mlir
@@ -76,10 +76,10 @@ func.func @dense_index(%arga: tensor<?x?xi64, #DenseMatrix>)
 // CHECK-DAG:       %[[VAL_3:.*]] = tensor.dim %[[VAL_0]], %[[VAL_1]] : tensor<?x?xi64, #sparse_tensor.encoding
 // CHECK-DAG:       %[[VAL_4:.*]] = tensor.dim %[[VAL_0]], %[[VAL_1]] : tensor<?x?xi64, #sparse_tensor.encoding
 // CHECK-DAG:       %[[VAL_5:.*]] = bufferization.alloc_tensor(%[[VAL_3]], %[[VAL_4]]) : tensor<?x?xi64, #sparse_tensor.encoding
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<?x?xi64, #sparse_tensor.encoding
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<?x?xi64, #sparse_tensor.encoding
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<?x?xi64, #sparse_tensor.encoding
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<?x?xi64, #sparse_tensor.encoding
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<?x?xi64, #sparse_tensor.encoding
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<?x?xi64, #sparse_tensor.encoding
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<?x?xi64, #sparse_tensor.encoding
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<?x?xi64, #sparse_tensor.encoding
 // CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<?x?xi64, #sparse_tensor.encoding
 // CHECK:           %[[VAL_11:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_1]]] : memref<?xindex>
 // CHECK:           %[[VAL_12:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_2]]] : memref<?xindex>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_int_ops.mlir b/mlir/test/Dialect/SparseTensor/sparse_int_ops.mlir
index 2054d97c6b2d1..742b8051f3a54 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_int_ops.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_int_ops.mlir
@@ -30,8 +30,8 @@
 // CHECK-DAG:           %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:           %[[VAL_5:.*]] = arith.constant true
 // CHECK-DAG:           %[[VAL_6:.*]] = arith.constant 1 : index
-// CHECK:           %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
-// CHECK:           %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK:           %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK:           %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK:           %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK:           %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xi64>
 // CHECK:           %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xi64>
@@ -91,8 +91,8 @@ func.func @add(%arga: tensor<32xi64, #SV>,
 // CHECK-DAG:           %[[VAL_5:.*]] = arith.constant true
 // CHECK-DAG:           %[[VAL_6:.*]] = arith.constant 1 : index
 // CHECK-DAG:           %[[VAL_7:.*]] = arith.constant 0 : i64
-// CHECK:           %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
-// CHECK:           %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK:           %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK:           %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK:           %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK:           %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xi64>
 // CHECK:           %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xi64>
@@ -151,8 +151,8 @@ func.func @sub(%arga: tensor<32xi64, #SV>,
 // CHECK-SAME:              %[[VAL_2:.*]]: tensor<32xi64>) -> tensor<32xi64> {
 // CHECK-DAG:           %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:           %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK:           %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
-// CHECK:           %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK:           %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK:           %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK:           %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK:           %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xi64>
 // CHECK:           %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xi64>
@@ -187,8 +187,8 @@ func.func @mul(%arga: tensor<32xi64, #SV>,
 // CHECK-DAG:           %[[VAL_2:.*]] = arith.constant 2 : i64
 // CHECK-DAG:           %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:           %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK:           %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
-// CHECK:           %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK:           %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK:           %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK:           %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK:           %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xi64>
 // CHECK:           %[[VAL_9:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_3]]] : memref<?xindex>
@@ -221,8 +221,8 @@ func.func @divsbyc(%arga: tensor<32xi64, #SV>,
 // CHECK-DAG:           %[[VAL_2:.*]] = arith.constant 2 : i64
 // CHECK-DAG:           %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:           %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK:           %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{.*}}}>>
-// CHECK:           %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK:           %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{.*}}}>>
+// CHECK:           %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK:           %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK:           %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xi64>
 // CHECK:           %[[VAL_9:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_3]]] : memref<?xindex>
@@ -255,8 +255,8 @@ func.func @divubyc(%arga: tensor<32xi64, #SV>,
 // CHECK-SAME:              %[[VAL_2:.*]]: tensor<32xi64>) -> tensor<32xi64> {
 // CHECK-DAG:           %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:           %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK:           %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:           %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:           %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xi64>
 // CHECK:           %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xi64>
 // CHECK:           %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xi64>
@@ -293,8 +293,8 @@ func.func @and(%arga: tensor<32xi64, #SV>,
 // CHECK-DAG:           %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:           %[[VAL_5:.*]] = arith.constant true
 // CHECK-DAG:           %[[VAL_6:.*]] = arith.constant 1 : index
-// CHECK:           %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:           %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:           %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xi64>
 // CHECK:           %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xi64>
 // CHECK:           %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xi64>
@@ -353,8 +353,8 @@ func.func @or(%arga: tensor<32xi64, #SV>,
 // CHECK-DAG:           %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:           %[[VAL_5:.*]] = arith.constant true
 // CHECK-DAG:           %[[VAL_6:.*]] = arith.constant 1 : index
-// CHECK:           %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:           %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:           %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xi64>
 // CHECK:           %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xi64>
 // CHECK:           %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xi64>
@@ -411,8 +411,8 @@ func.func @xor(%arga: tensor<32xi64, #SV>,
 // CHECK-DAG:           %[[VAL_2:.*]] = arith.constant 2 : i64
 // CHECK-DAG:           %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:           %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK:           %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:           %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:           %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xi64>
 // CHECK:           %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xi64>
 // CHECK:           %[[VAL_9:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_3]]] : memref<?xindex>
@@ -445,8 +445,8 @@ func.func @ashrbyc(%arga: tensor<32xi64, #SV>,
 // CHECK-DAG:           %[[VAL_2:.*]] = arith.constant 2 : i64
 // CHECK-DAG:           %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:           %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK:           %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:           %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:           %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xi64>
 // CHECK:           %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xi64>
 // CHECK:           %[[VAL_9:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_3]]] : memref<?xindex>
@@ -479,8 +479,8 @@ func.func @lsrbyc(%arga: tensor<32xi64, #SV>,
 // CHECK-DAG:           %[[VAL_2:.*]] = arith.constant 2 : i64
 // CHECK-DAG:           %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:           %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK:           %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
-// CHECK:           %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
+// CHECK:           %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xindex>
 // CHECK:           %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32xi64, #sparse_tensor.encoding<{{{.*}}}>> to memref<?xi64>
 // CHECK:           %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_1]] : memref<32xi64>
 // CHECK:           %[[VAL_9:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_3]]] : memref<?xindex>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_kernels.mlir b/mlir/test/Dialect/SparseTensor/sparse_kernels.mlir
index 064c879cb16af..55288c3282f9c 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_kernels.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_kernels.mlir
@@ -13,10 +13,10 @@
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 30 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf32>
 // CHECK:           %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_1]] : memref<20x30xf32>
 // CHECK:           %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_2]] : memref<10x30xf32>
@@ -59,10 +59,10 @@ func.func @matmul1(%a: tensor<10x20xf32, #DCSR>,
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_6:.*]] = bufferization.to_memref %[[VAL_0]] : memref<10x20xf32>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index}
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index}
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index}
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index}
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index}
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index}
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 1 : index}
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 1 : index}
 // CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_1]]
 // CHECK:           %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_2]] : memref<10x30xf32>
 // CHECK:           scf.for %[[VAL_13:.*]] = %[[VAL_4]] to %[[VAL_3]] step %[[VAL_5]] {
@@ -109,15 +109,15 @@ func.func @matmul_sparse_rhs(%a: tensor<10x20xf32>,
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant false
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant true
 // CHECK-DAG:       %[[VAL_6:.*]] = bufferization.alloc_tensor() : tensor<4x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<4x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<4x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<4x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<4x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<4x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<4x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<4x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<4x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<4x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf64>
-// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<8x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<8x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index} : tensor<8x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_15:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index} : tensor<8x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<8x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<8x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 1 : index} : tensor<8x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_15:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 1 : index} : tensor<8x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_16:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<8x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf64>
 // CHECK:           %[[VAL_17:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_2]]] : memref<?xindex>
 // CHECK:           %[[VAL_18:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_3]]] : memref<?xindex>
@@ -203,10 +203,10 @@ func.func @matmul2(%A: tensor<4x8xf64, #DCSR>,
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_6:.*]] = bufferization.to_memref %[[VAL_0]] : memref<8x8xi32>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<3x3xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<3x3xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index} : tensor<3x3xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index} : tensor<3x3xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<3x3xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<3x3xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 1 : index} : tensor<3x3xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 1 : index} : tensor<3x3xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<3x3xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xi32>
 // CHECK:           %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_2]] : memref<6x6xi32>
 // CHECK:           scf.for %[[VAL_13:.*]] = %[[VAL_4]] to %[[VAL_3]] step %[[VAL_5]] {
@@ -254,10 +254,10 @@ func.func @conv2d(%input:  tensor<8x8xi32>,
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 2 : i64
 // CHECK-DAG:       %[[VAL_7:.*]] = bufferization.to_memref %[[VAL_0]] : memref<5x3xi8>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<3x6xi8, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<3x6xi8, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index} : tensor<3x6xi8, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index} : tensor<3x6xi8, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<3x6xi8, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<3x6xi8, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 1 : index} : tensor<3x6xi8, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 1 : index} : tensor<3x6xi8, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<3x6xi8, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xi8>
 // CHECK:           %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_2]] : memref<5x6xi64>
 // CHECK:           scf.for %[[VAL_14:.*]] = %[[VAL_4]] to %[[VAL_3]] step %[[VAL_5]] {
@@ -302,11 +302,11 @@ func.func @quantized_matmul(%input1: tensor<5x3xi8>,
 // CHECK-SAME:      %[[VAL_2:.*2]]: tensor<f32>) -> tensor<f32> {
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK:           %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_2]] : memref<f32>
 // CHECK:           %[[VAL_12:.*]] = memref.load %[[VAL_11]][] : memref<f32>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_lower.mlir b/mlir/test/Dialect/SparseTensor/sparse_lower.mlir
index 840da19f815a8..c904a1e05c73e 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_lower.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_lower.mlir
@@ -27,8 +27,8 @@
 // CHECK-HIR-DAG:       %[[VAL_3:.*]] = arith.constant 32 : index
 // CHECK-HIR-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-HIR-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-HIR-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{{{.*}}}>>
-// CHECK-HIR-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK-HIR-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK-HIR-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK-HIR-DAG:       %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x64xf64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK-HIR-DAG:       %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_1]] : memref<64xf64>
 // CHECK-HIR-DAG:       %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xf64>
@@ -58,8 +58,8 @@
 // CHECK-MIR-DAG:       %[[VAL_3:.*]] = arith.constant 32 : index
 // CHECK-MIR-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-MIR-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-MIR-DAG:       %[[VAL_6:.*]] = call @sparsePointers0(%[[VAL_0]], %[[VAL_5]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
-// CHECK-MIR-DAG:       %[[VAL_7:.*]] = call @sparseIndices0(%[[VAL_0]], %[[VAL_5]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK-MIR-DAG:       %[[VAL_6:.*]] = call @sparsePositions0(%[[VAL_0]], %[[VAL_5]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK-MIR-DAG:       %[[VAL_7:.*]] = call @sparseCoordinates0(%[[VAL_0]], %[[VAL_5]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
 // CHECK-MIR-DAG:       %[[VAL_8:.*]] = call @sparseValuesF64(%[[VAL_0]]) : (!llvm.ptr<i8>) -> memref<?xf64>
 // CHECK-MIR-DAG:       %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_1]] : memref<64xf64>
 // CHECK-MIR-DAG:       %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xf64>
@@ -89,8 +89,8 @@
 // CHECK-LIR-DAG:       %[[VAL_3:.*]] = arith.constant 32 : index
 // CHECK-LIR-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-LIR-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-LIR-DAG:       %[[VAL_6:.*]] = call @sparsePointers0(%[[VAL_0]], %[[VAL_5]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
-// CHECK-LIR-DAG:       %[[VAL_7:.*]] = call @sparseIndices0(%[[VAL_0]], %[[VAL_5]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK-LIR-DAG:       %[[VAL_6:.*]] = call @sparsePositions0(%[[VAL_0]], %[[VAL_5]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK-LIR-DAG:       %[[VAL_7:.*]] = call @sparseCoordinates0(%[[VAL_0]], %[[VAL_5]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
 // CHECK-LIR-DAG:       %[[VAL_8:.*]] = call @sparseValuesF64(%[[VAL_0]]) : (!llvm.ptr<i8>) -> memref<?xf64>
 // CHECK-LIR:           scf.for %[[VAL_12:.*]] = %[[VAL_4]] to %[[VAL_3]] step %[[VAL_5]] {
 // CHECK-LIR-DAG:         %[[VAL_13:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_12]]] : memref<?xindex>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_lower_col.mlir b/mlir/test/Dialect/SparseTensor/sparse_lower_col.mlir
index d2aa35acec264..d97ca0e7fdaa4 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_lower_col.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_lower_col.mlir
@@ -30,8 +30,8 @@
 // CHECK-HIR-DAG:       %[[VAL_3:.*]] = arith.constant 64 : index
 // CHECK-HIR-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-HIR-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-HIR-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)> }>> to memref<?xindex>
-// CHECK-HIR-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)> }>> to memref<?xindex>
+// CHECK-HIR-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)> }>> to memref<?xindex>
+// CHECK-HIR-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)> }>> to memref<?xindex>
 // CHECK-HIR-DAG:       %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x64xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)> }>> to memref<?xf64>
 // CHECK-HIR-DAG:       %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_1]] : memref<64xf64>
 // CHECK-HIR-DAG:       %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xf64>
@@ -60,8 +60,8 @@
 // CHECK-MIR-DAG:       %[[VAL_3:.*]] = arith.constant 64 : index
 // CHECK-MIR-DAG:       %[[VAL_5:.*]] = arith.constant 0 : index
 // CHECK-MIR-DAG:       %[[VAL_6:.*]] = arith.constant 1 : index
-// CHECK-MIR-DAG:       %[[VAL_7:.*]] = call @sparsePointers0(%[[VAL_0]], %[[VAL_6]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
-// CHECK-MIR-DAG:       %[[VAL_8:.*]] = call @sparseIndices0(%[[VAL_0]], %[[VAL_6]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK-MIR-DAG:       %[[VAL_7:.*]] = call @sparsePositions0(%[[VAL_0]], %[[VAL_6]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK-MIR-DAG:       %[[VAL_8:.*]] = call @sparseCoordinates0(%[[VAL_0]], %[[VAL_6]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
 // CHECK-MIR-DAG:       %[[VAL_9:.*]] = call @sparseValuesF64(%[[VAL_0]]) : (!llvm.ptr<i8>) -> memref<?xf64>
 // CHECK-MIR-DAG:       %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_1]] : memref<64xf64>
 // CHECK-MIR-DAG:       %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xf64>
@@ -90,8 +90,8 @@
 // CHECK-LIR-DAG:       %[[VAL_3:.*]] = arith.constant 64 : index
 // CHECK-LIR-DAG:       %[[VAL_5:.*]] = arith.constant 0 : index
 // CHECK-LIR-DAG:       %[[VAL_6:.*]] = arith.constant 1 : index
-// CHECK-LIR-DAG:       %[[VAL_7:.*]] = call @sparsePointers0(%[[VAL_0]], %[[VAL_6]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
-// CHECK-LIR-DAG:       %[[VAL_8:.*]] = call @sparseIndices0(%[[VAL_0]], %[[VAL_6]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK-LIR-DAG:       %[[VAL_7:.*]] = call @sparsePositions0(%[[VAL_0]], %[[VAL_6]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK-LIR-DAG:       %[[VAL_8:.*]] = call @sparseCoordinates0(%[[VAL_0]], %[[VAL_6]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
 // CHECK-LIR-DAG:       %[[VAL_9:.*]] = call @sparseValuesF64(%[[VAL_0]]) : (!llvm.ptr<i8>) -> memref<?xf64>
 // CHECK-LIR:           scf.for %[[VAL_13:.*]] = %[[VAL_5]] to %[[VAL_3]] step %[[VAL_6]] {
 // CHECK-LIR:             %[[VAL_14:.*]] = memref.load %[[VAL_1]]{{\[}}%[[VAL_13]]] : memref<64xf64>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_lower_inplace.mlir b/mlir/test/Dialect/SparseTensor/sparse_lower_inplace.mlir
index 3f6e2ac8e6144..2b3217ac35485 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_lower_inplace.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_lower_inplace.mlir
@@ -27,8 +27,8 @@
 // CHECK-HIR-DAG:       %[[VAL_3:.*]] = arith.constant 32 : index
 // CHECK-HIR-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-HIR-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-HIR:           %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{{{.*}}}>>
-// CHECK-HIR:           %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK-HIR:           %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{{{.*}}}>>
+// CHECK-HIR:           %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x64xf64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK-HIR:           %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x64xf64, #sparse_tensor.encoding<{{{.*}}}>>
 // CHECK-HIR:           %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_1]] : memref<64xf64>
 // CHECK-HIR:           %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xf64>
@@ -58,8 +58,8 @@
 // CHECK-MIR-DAG:       %[[VAL_3:.*]] = arith.constant 32 : index
 // CHECK-MIR-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-MIR-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-MIR:           %[[VAL_6:.*]] = call @sparsePointers0(%[[VAL_0]], %[[VAL_5]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
-// CHECK-MIR:           %[[VAL_7:.*]] = call @sparseIndices0(%[[VAL_0]], %[[VAL_5]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK-MIR:           %[[VAL_6:.*]] = call @sparsePositions0(%[[VAL_0]], %[[VAL_5]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK-MIR:           %[[VAL_7:.*]] = call @sparseCoordinates0(%[[VAL_0]], %[[VAL_5]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
 // CHECK-MIR:           %[[VAL_8:.*]] = call @sparseValuesF64(%[[VAL_0]]) : (!llvm.ptr<i8>) -> memref<?xf64>
 // CHECK-MIR:           %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_1]] : memref<64xf64>
 // CHECK-MIR:           %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_2]] : memref<32xf64>
@@ -89,8 +89,8 @@
 // CHECK-LIR-DAG:       %[[VAL_3:.*]] = arith.constant 32 : index
 // CHECK-LIR-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-LIR-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-LIR:           %[[VAL_6:.*]] = call @sparsePointers0(%[[VAL_0]], %[[VAL_5]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
-// CHECK-LIR:           %[[VAL_7:.*]] = call @sparseIndices0(%[[VAL_0]], %[[VAL_5]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK-LIR:           %[[VAL_6:.*]] = call @sparsePositions0(%[[VAL_0]], %[[VAL_5]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
+// CHECK-LIR:           %[[VAL_7:.*]] = call @sparseCoordinates0(%[[VAL_0]], %[[VAL_5]]) : (!llvm.ptr<i8>, index) -> memref<?xindex>
 // CHECK-LIR:           %[[VAL_8:.*]] = call @sparseValuesF64(%[[VAL_0]]) : (!llvm.ptr<i8>) -> memref<?xf64>
 // CHECK-LIR:           scf.for %[[VAL_9:.*]] = %[[VAL_4]] to %[[VAL_3]] step %[[VAL_5]] {
 // CHECK-LIR-DAG:         %[[VAL_10:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_9]]] : memref<?xindex>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_matmul_codegen.mlir b/mlir/test/Dialect/SparseTensor/sparse_matmul_codegen.mlir
index aaa7b581675d8..cdb364c093f8f 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_matmul_codegen.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_matmul_codegen.mlir
@@ -25,7 +25,7 @@
 // CHECK:           %[[VAL_9:.*]] = arith.addi %[[VAL_4]], %[[VAL_8]] : index
 // CHECK:           %[[VAL_10:.*]] = memref.load %[[VAL_0]]{{\[}}%[[VAL_4]]] : memref<?xindex>
 // CHECK:           %[[VAL_11:.*]] = memref.load %[[VAL_0]]{{\[}}%[[VAL_9]]] : memref<?xindex>
-// CHECK:           %[[VAL_13:.*]] = sparse_tensor.storage_specifier.get %[[VAL_3]]  idx_mem_sz at 1 : !sparse_tensor.storage_specifier
+// CHECK:           %[[VAL_13:.*]] = sparse_tensor.storage_specifier.get %[[VAL_3]]  crd_mem_sz at 1 : !sparse_tensor.storage_specifier
 // CHECK:           %[[VAL_14:.*]] = arith.subi %[[VAL_11]], %[[VAL_8]] : index
 // CHECK:           %[[VAL_15:.*]] = arith.cmpi ult, %[[VAL_10]], %[[VAL_11]] : index
 // CHECK:           %[[VAL_16:.*]] = scf.if %[[VAL_15]] -> (i1) {
@@ -42,7 +42,7 @@
 // CHECK:             %[[VAL_21:.*]] = arith.addi %[[VAL_13]], %[[VAL_8]] : index
 // CHECK:             memref.store %[[VAL_21]], %[[VAL_0]]{{\[}}%[[VAL_9]]] : memref<?xindex>
 // CHECK:             %[[VAL_22:.*]], %[[VAL_24:.*]] = sparse_tensor.push_back %[[VAL_13]], %[[VAL_1]], %[[VAL_5]] : index, memref<?xindex>, index
-// CHECK:             %[[VAL_25:.*]] = sparse_tensor.storage_specifier.set %[[VAL_3]]  idx_mem_sz at 1 with %[[VAL_24]] : !sparse_tensor.storage_specifier
+// CHECK:             %[[VAL_25:.*]] = sparse_tensor.storage_specifier.set %[[VAL_3]]  crd_mem_sz at 1 with %[[VAL_24]] : !sparse_tensor.storage_specifier
 // CHECK:             scf.yield %[[VAL_22]], %[[VAL_25]] : memref<?xindex>, !sparse_tensor.storage_specifier
 // CHECK:           }
 // CHECK:           %[[VAL_28:.*]] = sparse_tensor.storage_specifier.get %[[VAL_27:.*]]#1  val_mem_sz : !sparse_tensor.storage_specifier
@@ -73,13 +73,13 @@
 // CHECK:           %[[VAL_18:.*]] = memref.alloc() : memref<16xf64>
 // CHECK:           %[[VAL_19:.*]] = memref.cast %[[VAL_18]] : memref<16xf64> to memref<?xf64>
 // CHECK:           %[[VAL_20:.*]] = sparse_tensor.storage_specifier.init : !sparse_tensor.storage_specifier
-// CHECK:           %[[VAL_21:.*]] = sparse_tensor.storage_specifier.set %[[VAL_20]]  dim_sz at 0 with %[[VAL_8]] : !sparse_tensor.storage_specifier
-// CHECK:           %[[VAL_22:.*]] = sparse_tensor.storage_specifier.set %[[VAL_21]]  dim_sz at 1 with %[[VAL_8]] : !sparse_tensor.storage_specifier
-// CHECK:           %[[VAL_23:.*]] = sparse_tensor.storage_specifier.get %[[VAL_22]]  ptr_mem_sz at 1 : !sparse_tensor.storage_specifier
+// CHECK:           %[[VAL_21:.*]] = sparse_tensor.storage_specifier.set %[[VAL_20]]  lvl_sz at 0 with %[[VAL_8]] : !sparse_tensor.storage_specifier
+// CHECK:           %[[VAL_22:.*]] = sparse_tensor.storage_specifier.set %[[VAL_21]]  lvl_sz at 1 with %[[VAL_8]] : !sparse_tensor.storage_specifier
+// CHECK:           %[[VAL_23:.*]] = sparse_tensor.storage_specifier.get %[[VAL_22]]  pos_mem_sz at 1 : !sparse_tensor.storage_specifier
 // CHECK:           %[[VAL_24:.*]], %[[VAL_25:.*]] = sparse_tensor.push_back %[[VAL_23]], %[[VAL_15]], %[[VAL_10]] : index, memref<?xindex>, index
-// CHECK:           %[[VAL_26:.*]] = sparse_tensor.storage_specifier.set %[[VAL_22]]  ptr_mem_sz at 1 with %[[VAL_25]] : !sparse_tensor.storage_specifier
+// CHECK:           %[[VAL_26:.*]] = sparse_tensor.storage_specifier.set %[[VAL_22]]  pos_mem_sz at 1 with %[[VAL_25]] : !sparse_tensor.storage_specifier
 // CHECK:           %[[VAL_27:.*]], %[[VAL_28:.*]] = sparse_tensor.push_back %[[VAL_25]], %[[VAL_24]], %[[VAL_10]], %[[VAL_8]] : index, memref<?xindex>, index, index
-// CHECK:           %[[VAL_29:.*]] = sparse_tensor.storage_specifier.set %[[VAL_26]]  ptr_mem_sz at 1 with %[[VAL_28]] : !sparse_tensor.storage_specifier
+// CHECK:           %[[VAL_29:.*]] = sparse_tensor.storage_specifier.set %[[VAL_26]]  pos_mem_sz at 1 with %[[VAL_28]] : !sparse_tensor.storage_specifier
 // CHECK:           %[[VAL_30:.*]] = memref.alloc() : memref<4xf64>
 // CHECK:           %[[VAL_31:.*]] = m
 // CHECK:           %[[VAL_32:.*]] = memref.alloc() : memref<4xindex>
@@ -131,7 +131,7 @@
 // CHECK:           memref.dealloc %[[VAL_30]] : memref<4xf64>
 // CHECK:           memref.dealloc %[[VAL_31]] : memref<4xi1>
 // CHECK:           memref.dealloc %[[VAL_32]] : memref<4xindex>
-// CHECK:           %[[VAL_76:.*]] = sparse_tensor.storage_specifier.get %[[VAL_77:.*]]#3  ptr_mem_sz at 1 : !sparse_tensor.storage_specifier
+// CHECK:           %[[VAL_76:.*]] = sparse_tensor.storage_specifier.get %[[VAL_77:.*]]#3  pos_mem_sz at 1 : !sparse_tensor.storage_specifier
 // CHECK:           %[[VAL_78:.*]] = memref.load %[[VAL_77]]#0{{\[}}%[[VAL_10]]] : memref<?xindex>
 // CHECK:           %[[VAL_79:.*]] = scf.for %[[VAL_80:.*]] = %[[VAL_11]] to %[[VAL_76]] step %[[VAL_11]] iter_args(%[[VAL_81:.*]] = %[[VAL_78]]) -> (index) {
 // CHECK:             %[[VAL_82:.*]] = memref.load %[[VAL_77]]#0{{\[}}%[[VAL_80]]] : memref<?xindex>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_nd.mlir b/mlir/test/Dialect/SparseTensor/sparse_nd.mlir
index 09ba1e6eb460a..c99a34b5ce38a 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_nd.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_nd.mlir
@@ -34,10 +34,10 @@
 // CHECK-DAG:       %[[VAL_11:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_12:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_0]] : memref<10x20x30x40x50x60x70x80xf32>
-// CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 3 : index} : tensor<80x70x60x50x40x30x20x10xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "dense", "compressed", "compressed", "dense", "dense", "dense" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_15:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 3 : index} : tensor<80x70x60x50x40x30x20x10xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "dense", "compressed", "compressed", "dense", "dense", "dense" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_16:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 4 : index} : tensor<80x70x60x50x40x30x20x10xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "dense", "compressed", "compressed", "dense", "dense", "dense" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_17:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 4 : index} : tensor<80x70x60x50x40x30x20x10xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "dense", "compressed", "compressed", "dense", "dense", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 3 : index} : tensor<80x70x60x50x40x30x20x10xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "dense", "compressed", "compressed", "dense", "dense", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_15:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 3 : index} : tensor<80x70x60x50x40x30x20x10xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "dense", "compressed", "compressed", "dense", "dense", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_16:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 4 : index} : tensor<80x70x60x50x40x30x20x10xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "dense", "compressed", "compressed", "dense", "dense", "dense" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_17:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 4 : index} : tensor<80x70x60x50x40x30x20x10xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "dense", "compressed", "compressed", "dense", "dense", "dense" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_18:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<80x70x60x50x40x30x20x10xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "dense", "dense", "compressed", "compressed", "dense", "dense", "dense" ] }>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_20:.*]] = bufferization.to_memref %[[VAL_2]] : memref<10x20x30x40x50x60x70x80xf32>
 // CHECK:           linalg.fill ins(%[[ZERO]] : f32) outs(%[[VAL_20]] : memref<10x20x30x40x50x60x70x80xf32>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_out.mlir b/mlir/test/Dialect/SparseTensor/sparse_out.mlir
index e6d6c2364fc16..e57e751f6490b 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_out.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_out.mlir
@@ -27,8 +27,8 @@
 // CHECK-DAG:       %[[VAL_1:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 2.000000e+00 : f32
-// CHECK-DAG:       %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf32>
 // CHECK:           %[[VAL_7:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_1]]] : memref<?xindex>
 // CHECK:           %[[VAL_8:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_2]]] : memref<?xindex>
@@ -60,8 +60,8 @@ func.func @sparse_simply_dynamic1(%argx: tensor<32x16xf32, #DCSR>) -> tensor<32x
 // CHECK-SAME:      %[[VAL_0:.*]]: tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>) -> tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> {
 // CHECK-DAG:       %[[VAL_1:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_3:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_3:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf32>
 // CHECK:           %[[VAL_6:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_1]]] : memref<?xindex>
 // CHECK:           %[[VAL_7:.*]] = memref.load %[[VAL_3]]{{\[}}%[[VAL_2]]] : memref<?xindex>
@@ -105,8 +105,8 @@ func.func @sparse_simply_dynamic2(%argx: tensor<32x16xf32, #DCSR>) -> tensor<32x
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 2.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_5:.*]] = bufferization.alloc_tensor() : tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xf32>
 // CHECK:           %[[VAL_9:.*]] = scf.for %[[VAL_10:.*]] = %[[VAL_2]] to %[[VAL_1]] step %[[VAL_3]] iter_args(%[[VAL_11:.*]] = %[[VAL_5]]) -> (tensor<10x20xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>) {
 // CHECK:             %[[VAL_12:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_10]]] : memref<?xindex>
@@ -158,19 +158,19 @@ func.func @sparse_truly_dynamic(%arga: tensor<10x20xf32, #CSR>) -> tensor<10x20x
 // CHECK:           %[[VAL_5:.*]] = tensor.dim %[[VAL_0]], %[[VAL_2]] : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>>
 // CHECK:           %[[VAL_6:.*]] = tensor.dim %[[VAL_0]], %[[VAL_3]] : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>>
 // CHECK:           %[[VAL_7:.*]] = bufferization.alloc_tensor(%[[VAL_5]], %[[VAL_6]]) : tensor<?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>
-// CHECK:           %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_10:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_11:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_12:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 2 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_13:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 2 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_10:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_11:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_12:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 2 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_13:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 2 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK:           %[[VAL_14:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xi32>
-// CHECK:           %[[VAL_15:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_16:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_17:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_18:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_19:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 2 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_20:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 2 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_15:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_16:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_17:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 1 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_18:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 1 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_19:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 2 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_20:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 2 : index} : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK:           %[[VAL_21:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?x?x?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed", "compressed" ] }>> to memref<?xi32>
 // CHECK:           %[[VAL_22:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_2]]] : memref<?xindex>
 // CHECK:           %[[VAL_23:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_3]]] : memref<?xindex>
@@ -321,15 +321,15 @@ func.func @sumred(%arga: tensor<?x?x?xi32, #SparseTensor>,
 // CHECK:           %[[VAL_6:.*]] = tensor.dim %[[VAL_0]], %[[VAL_2]] : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>
 // CHECK:           %[[VAL_7:.*]] = tensor.dim %[[VAL_1]], %[[VAL_3]] : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>
 // CHECK:           %[[VAL_8:.*]] = bufferization.alloc_tensor(%[[VAL_6]], %[[VAL_7]]) : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>
-// CHECK:           %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_11:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_12:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_11:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_12:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK:           %[[VAL_13:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf32>
-// CHECK:           %[[VAL_14:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_15:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_16:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_17:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_14:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_15:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_16:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_17:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK:           %[[VAL_18:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf32>
 // CHECK:           %[[VAL_19:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_2]]] : memref<?xindex>
 // CHECK:           %[[VAL_20:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_3]]] : memref<?xindex>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_outbuf.mlir b/mlir/test/Dialect/SparseTensor/sparse_outbuf.mlir
index 3923c5e542a56..33aedfe122dba 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_outbuf.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_outbuf.mlir
@@ -16,8 +16,8 @@
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK:           %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<10xi32, #{{.*}}> to memref<?xindex>
-// CHECK:           %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<10xi32, #{{.*}}> to memref<?xindex>
+// CHECK:           %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<10xi32, #{{.*}}> to memref<?xindex>
+// CHECK:           %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<10xi32, #{{.*}}> to memref<?xindex>
 // CHECK:           %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<10xi32, #{{.*}}> to memref<?xi32>
 // CHECK:           %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_1]] : memref<10xf32>
 // CHECK:           linalg.fill ins(%[[VAL_3]] : f32) outs(%[[VAL_8]] : memref<10xf32>)
@@ -50,8 +50,8 @@ func.func @allout_inplace(%arga: tensor<10xi32, #SV>,
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 1 : index
 // CHECK:           %[[VAL_4:.*]] = bufferization.alloc_tensor() : tensor<10xf32>
-// CHECK:           %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<10xi32, #{{.*}}> to memref<?xindex>
-// CHECK:           %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<10xi32, #{{.*}}> to memref<?xindex>
+// CHECK:           %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<10xi32, #{{.*}}> to memref<?xindex>
+// CHECK:           %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<10xi32, #{{.*}}> to memref<?xindex>
 // CHECK:           %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<10xi32, #{{.*}}> to memref<?xi32>
 // CHECK:           %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_4]] : memref<10xf32>
 // CHECK:           linalg.fill ins(%[[VAL_2]] : f32) outs(%[[VAL_8]] : memref<10xf32>)
@@ -83,8 +83,8 @@ func.func @allout_materialize(%arga: tensor<10xi32, #SV>) -> tensor<10xf32> {
 // CHECK-SAME:      %[[VAL_1:.*]]: tensor<10xf32>) -> tensor<10xf32> {
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK:           %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<10xf32, #{{.*}}> to memref<?xindex>
-// CHECK:           %[[VAL_5:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<10xf32, #{{.*}}> to memref<?xindex>
+// CHECK:           %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<10xf32, #{{.*}}> to memref<?xindex>
+// CHECK:           %[[VAL_5:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<10xf32, #{{.*}}> to memref<?xindex>
 // CHECK:           %[[VAL_6:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<10xf32, #{{.*}}> to memref<?xf32>
 // CHECK:           %[[VAL_7:.*]] = bufferization.to_memref %[[VAL_1]] : memref<10xf32>
 // CHECK:           %[[VAL_8:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_2]]] : memref<?xindex>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_pack.mlir b/mlir/test/Dialect/SparseTensor/sparse_pack.mlir
index cdfd856b19d93..99befbeb2f1a5 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_pack.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_pack.mlir
@@ -2,7 +2,7 @@
 
 #COO = #sparse_tensor.encoding<{
   dimLevelType = ["compressed-nu", "singleton"],
-  indexBitWidth=32
+  crdWidth=32
 }>
 
 // CHECK-LABEL:   func.func @sparse_pack(
@@ -19,19 +19,19 @@
 // CHECK:           %[[VAL_10:.*]] = sparse_tensor.storage_specifier.init :
 // CHECK:           %[[VAL_11:.*]] = arith.constant 6 : index
 // CHECK:           %[[VAL_12:.*]] = arith.constant 100 : index
-// CHECK:           %[[VAL_14:.*]] = sparse_tensor.storage_specifier.set %[[VAL_10]]  dim_sz at 0 with %[[VAL_12]]
+// CHECK:           %[[VAL_14:.*]] = sparse_tensor.storage_specifier.set %[[VAL_10]]  lvl_sz at 0 with %[[VAL_12]]
 // CHECK:           %[[VAL_15:.*]] = arith.constant 2 : index
-// CHECK:           %[[VAL_17:.*]] = sparse_tensor.storage_specifier.set %[[VAL_14]]  ptr_mem_sz at 0 with %[[VAL_15]]
-// CHECK:           %[[VAL_19:.*]] = sparse_tensor.storage_specifier.set %[[VAL_17]]  idx_mem_sz at 0 with %[[VAL_11]]
-// CHECK:           %[[VAL_20:.*]] = sparse_tensor.storage_specifier.set %[[VAL_19]]  dim_sz at 1 with %[[VAL_12]]
-// CHECK:           %[[VAL_21:.*]] = sparse_tensor.storage_specifier.set %[[VAL_20]]  idx_mem_sz at 1 with %[[VAL_11]]
+// CHECK:           %[[VAL_17:.*]] = sparse_tensor.storage_specifier.set %[[VAL_14]]  pos_mem_sz at 0 with %[[VAL_15]]
+// CHECK:           %[[VAL_19:.*]] = sparse_tensor.storage_specifier.set %[[VAL_17]]  crd_mem_sz at 0 with %[[VAL_11]]
+// CHECK:           %[[VAL_20:.*]] = sparse_tensor.storage_specifier.set %[[VAL_19]]  lvl_sz at 1 with %[[VAL_12]]
+// CHECK:           %[[VAL_21:.*]] = sparse_tensor.storage_specifier.set %[[VAL_20]]  crd_mem_sz at 1 with %[[VAL_11]]
 // CHECK:           %[[VAL_22:.*]] = sparse_tensor.storage_specifier.set %[[VAL_21]]  val_mem_sz with %[[VAL_11]]
 // CHECK:           return %[[VAL_4]], %[[VAL_7]], %[[VAL_9]], %[[VAL_22]] : memref<?xindex>, memref<?xi32>, memref<?xf64>,
 // CHECK:         }
-func.func @sparse_pack(%data: tensor<6xf64>, %index: tensor<6x2xi32>)
+func.func @sparse_pack(%values: tensor<6xf64>, %coordinates: tensor<6x2xi32>)
                     -> tensor<100x100xf64, #COO> {
-  %0 = sparse_tensor.pack %data, %index : tensor<6xf64>, tensor<6x2xi32>
-                                       to tensor<100x100xf64, #COO>
+  %0 = sparse_tensor.pack %values, %coordinates
+     : tensor<6xf64>, tensor<6x2xi32> to tensor<100x100xf64, #COO>
   return %0 : tensor<100x100xf64, #COO>
 }
 
@@ -43,7 +43,7 @@ func.func @sparse_pack(%data: tensor<6xf64>, %index: tensor<6x2xi32>)
 // CHECK:           %[[VAL_4:.*]] = arith.constant 6 : index
 // CHECK:           %[[VAL_5:.*]] = arith.constant 0 : index
 // CHECK:           %[[VAL_6:.*]] = memref.dim %[[VAL_2]], %[[VAL_5]] : memref<?xf64>
-// CHECK:           %[[VAL_7:.*]] = arith.cmpi ult, %[[VAL_4]], %[[VAL_6]] : index
+// CHECK:           %[[VAL_7:.*]] = arith.cmpi ugt, %[[VAL_4]], %[[VAL_6]] : index
 // CHECK:           %[[VAL_8:.*]] = scf.if %[[VAL_7]] -> (memref<6xf64>) {
 // CHECK:             %[[VAL_9:.*]] = memref.realloc %[[VAL_2]] : memref<?xf64> to memref<6xf64>
 // CHECK:             scf.yield %[[VAL_9]] : memref<6xf64>
@@ -53,7 +53,7 @@ func.func @sparse_pack(%data: tensor<6xf64>, %index: tensor<6x2xi32>)
 // CHECK:           }
 // CHECK:           %[[VAL_11:.*]] = arith.constant 12 : index
 // CHECK:           %[[VAL_12:.*]] = memref.dim %[[VAL_1]], %[[VAL_5]] : memref<?xi32>
-// CHECK:           %[[VAL_13:.*]] = arith.cmpi ult, %[[VAL_11]], %[[VAL_12]] : index
+// CHECK:           %[[VAL_13:.*]] = arith.cmpi ugt, %[[VAL_11]], %[[VAL_12]] : index
 // CHECK:           %[[VAL_14:.*]] = scf.if %[[VAL_13]] -> (memref<12xi32>) {
 // CHECK:             %[[VAL_15:.*]] = memref.realloc %[[VAL_1]] : memref<?xi32> to memref<12xi32>
 // CHECK:             scf.yield %[[VAL_15]] : memref<12xi32>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_parallel_reduce.mlir b/mlir/test/Dialect/SparseTensor/sparse_parallel_reduce.mlir
index 8ba66d2c92ae1..572710644c2d4 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_parallel_reduce.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_parallel_reduce.mlir
@@ -21,8 +21,8 @@
 //   CHECK-DAG:  %[[TMP_c16:.*]] = arith.constant 16 : index
 //   CHECK-DAG:  %[[TMP_c0:.*]] = arith.constant 0 : index
 //   CHECK-DAG:  %[[TMP_c1:.*]] = arith.constant 1 : index
-//       CHECK:  %[[TMP_0:.*]] = sparse_tensor.pointers %[[TMP_arg0]] {dimension = 1 : index}
-//       CHECK:  %[[TMP_1:.*]] = sparse_tensor.indices %[[TMP_arg0]] {dimension = 1 : index}
+//       CHECK:  %[[TMP_0:.*]] = sparse_tensor.positions %[[TMP_arg0]] {level = 1 : index}
+//       CHECK:  %[[TMP_1:.*]] = sparse_tensor.coordinates %[[TMP_arg0]] {level = 1 : index}
 //       CHECK:  %[[TMP_2:.*]] = sparse_tensor.values %[[TMP_arg0]]
 //       CHECK:  %[[TMP_3:.*]] = bufferization.to_memref %[[TMP_arg1]] : memref<32xf32>
 //       CHECK:  %[[TMP_4:.*]] = bufferization.to_memref %[[TMP_arg2]] : memref<16xf32>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_reshape.mlir b/mlir/test/Dialect/SparseTensor/sparse_reshape.mlir
index a6790458b0902..6eb754db9ce3d 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_reshape.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_reshape.mlir
@@ -47,8 +47,8 @@
 // CHECK-RWT-DAG:     %[[C0:.*]] = arith.constant 0 : index
 // CHECK-RWT-DAG:     %[[C1:.*]] = arith.constant 1 : index
 // CHECK-RWT:         %[[B:.*]] = bufferization.alloc_tensor()
-// CHECK-RWT:         %[[P0:.*]] = sparse_tensor.pointers %[[S]] {dimension = 0 : index}
-// CHECK-RWT:         %[[I0:.*]] = sparse_tensor.indices %[[S]] {dimension = 0 : index}
+// CHECK-RWT:         %[[P0:.*]] = sparse_tensor.positions %[[S]] {level = 0 : index}
+// CHECK-RWT:         %[[I0:.*]] = sparse_tensor.coordinates %[[S]] {level = 0 : index}
 // CHECK-RWT:         %[[V:.*]] = sparse_tensor.values %[[S]]
 // CHECK-RWT:         %[[S0:.*]] = memref.load %[[P0]]{{\[}}%[[C0]]] : memref<?xindex>
 // CHECK-RWT:         %[[E0:.*]] = memref.load %[[P0]]{{\[}}%[[C1]]] : memref<?xindex>
@@ -111,10 +111,10 @@ func.func @sparse_expand(%arg0: tensor<100xf64, #SparseVector>) -> tensor<10x10x
 // CHECK-RWT-DAG:     %[[C0:.*]] = arith.constant 0 : index
 // CHECK-RWT-DAG:     %[[C1:.*]] = arith.constant 1 : index
 // CHECK-RWT:         %[[B:.*]] = bufferization.alloc_tensor()
-// CHECK-RWT:         %[[P0:.*]] = sparse_tensor.pointers %[[S]] {dimension = 0 : index}
-// CHECK-RWT:         %[[I0:.*]] = sparse_tensor.indices %[[S]] {dimension = 0 : index}
-// CHECK-RWT:         %[[P1:.*]] = sparse_tensor.pointers %[[S]] {dimension = 1 : index}
-// CHECK-RWT:         %[[I1:.*]] = sparse_tensor.indices %[[S]] {dimension = 1 : index}
+// CHECK-RWT:         %[[P0:.*]] = sparse_tensor.positions %[[S]] {level = 0 : index}
+// CHECK-RWT:         %[[I0:.*]] = sparse_tensor.coordinates %[[S]] {level = 0 : index}
+// CHECK-RWT:         %[[P1:.*]] = sparse_tensor.positions %[[S]] {level = 1 : index}
+// CHECK-RWT:         %[[I1:.*]] = sparse_tensor.coordinates %[[S]] {level = 1 : index}
 // CHECK-RWT:         %[[V:.*]] = sparse_tensor.values %[[S]]
 // CHECK-RWT:         %[[S0:.*]] = memref.load %[[P0]]{{\[}}%[[C0]]] : memref<?xindex>
 // CHECK-RWT:         %[[E0:.*]] = memref.load %[[P0]]{{\[}}%[[C1]]] : memref<?xindex>
@@ -191,8 +191,8 @@ func.func @sparse_collapse(%arg0: tensor<10x10xf64, #SparseMatrix>) -> tensor<10
 // CHECK-RWT:         %[[SD:.*]] = tensor.dim %[[S]], %[[C0]]
 // CHECK-RWT:         %[[DD0:.*]] = arith.divui %[[SD]], %[[C10]] : index
 // CHECK-RWT:         %[[B:.*]] = bufferization.alloc_tensor(%[[DD0]])
-// CHECK-RWT:         %[[P0:.*]] = sparse_tensor.pointers %[[S]] {dimension = 0 : index}
-// CHECK-RWT:         %[[I0:.*]] = sparse_tensor.indices %[[S]] {dimension = 0 : index}
+// CHECK-RWT:         %[[P0:.*]] = sparse_tensor.positions %[[S]] {level = 0 : index}
+// CHECK-RWT:         %[[I0:.*]] = sparse_tensor.coordinates %[[S]] {level = 0 : index}
 // CHECK-RWT:         %[[V:.*]] = sparse_tensor.values %[[S]]
 // CHECK-RWT:         %[[S0:.*]] = memref.load %[[P0]]{{\[}}%[[C0]]] : memref<?xindex>
 // CHECK-RWT:         %[[E0:.*]] = memref.load %[[P0]]{{\[}}%[[C1]]] : memref<?xindex>
@@ -265,10 +265,10 @@ func.func @dynamic_sparse_expand(%arg0: tensor<?xf64, #SparseVector>) -> tensor<
 // CHECK-RWT:         %[[SD1:.*]] = tensor.dim %[[S]], %[[C1]]
 // CHECK-RWT:         %[[DD0:.*]] = arith.muli %[[SD1]], %[[C10]] : index
 // CHECK-RWT:         %[[B:.*]] = bufferization.alloc_tensor(%[[DD0]])
-// CHECK-RWT:         %[[P0:.*]] = sparse_tensor.pointers %[[S]] {dimension = 0 : index}
-// CHECK-RWT:         %[[I0:.*]] = sparse_tensor.indices %[[S]] {dimension = 0 : index}
-// CHECK-RWT:         %[[P1:.*]] = sparse_tensor.pointers %[[S]] {dimension = 1 : index}
-// CHECK-RWT:         %[[I1:.*]] = sparse_tensor.indices %[[S]] {dimension = 1 : index}
+// CHECK-RWT:         %[[P0:.*]] = sparse_tensor.positions %[[S]] {level = 0 : index}
+// CHECK-RWT:         %[[I0:.*]] = sparse_tensor.coordinates %[[S]] {level = 0 : index}
+// CHECK-RWT:         %[[P1:.*]] = sparse_tensor.positions %[[S]] {level = 1 : index}
+// CHECK-RWT:         %[[I1:.*]] = sparse_tensor.coordinates %[[S]] {level = 1 : index}
 // CHECK-RWT:         %[[V:.*]] = sparse_tensor.values %[[S]]
 // CHECK-RWT:         %[[S0:.*]] = memref.load %[[P0]]{{\[}}%[[C0]]] : memref<?xindex>
 // CHECK-RWT:         %[[E0:.*]] = memref.load %[[P0]]{{\[}}%[[C1]]] : memref<?xindex>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_reshape_dot.mlir b/mlir/test/Dialect/SparseTensor/sparse_reshape_dot.mlir
index e52aca5533574..a8087a56906b0 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_reshape_dot.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_reshape_dot.mlir
@@ -1,53 +1,79 @@
 // RUN: mlir-opt %s --linalg-generalize-named-ops --sparsification --cse --canonicalize | FileCheck %s
 
-#COO_2D = #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ], pointerBitWidth = 32, indexBitWidth = 32 }>
-#COO_3D = #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton-nu", "singleton" ], pointerBitWidth = 32, indexBitWidth = 32 }>
+#COO_2D = #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ], posWidth = 32, crdWidth = 32 }>
+#COO_3D = #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton-nu", "singleton" ], posWidth = 32, crdWidth = 32 }>
+
 
 // CHECK-LABEL:   func.func @sparse_reshape_fused(
 // CHECK-SAME:      %[[VAL_0:.*]]: tensor<5x6xf32>,
-// CHECK-SAME:      %[[VAL_1:.*]]: tensor<6x2x3xf32,
-// CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 5 : index
-// CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 3 : index
-// CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
-// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_6:.*]] = tensor.empty() : tensor<5x6xf32>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index}
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index}
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index}
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 2 : index}
-// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_1]]
-// CHECK-DAG:       %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_6]] : memref<5x6xf32>
-// CHECK:           scf.for %[[VAL_13:.*]] = %[[VAL_4]] to %[[VAL_2]] step %[[VAL_5]] {
-// CHECK:             %[[VAL_14:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_4]]] : memref<?xi32>
-// CHECK:             %[[VAL_15:.*]] = arith.extui %[[VAL_14]] : i32 to i64
-// CHECK:             %[[VAL_16:.*]] = arith.index_cast %[[VAL_15]] : i64 to index
-// CHECK:             %[[VAL_17:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_5]]] : memref<?xi32>
-// CHECK:             %[[VAL_18:.*]] = arith.extui %[[VAL_17]] : i32 to i64
-// CHECK:             %[[VAL_19:.*]] = arith.index_cast %[[VAL_18]] : i64 to index
-// CHECK:             scf.for %[[VAL_20:.*]] = %[[VAL_16]] to %[[VAL_19]] step %[[VAL_5]] {
-// CHECK:               %[[VAL_21:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_20]]] : memref<?xi32, strided<[?], offset: ?>>
-// CHECK:               %[[VAL_22:.*]] = arith.extui %[[VAL_21]] : i32 to i64
-// CHECK:               %[[VAL_23:.*]] = arith.index_cast %[[VAL_22]] : i64 to index
-// CHECK:               %[[VAL_24:.*]] = tensor.extract %[[VAL_0]]{{\[}}%[[VAL_13]], %[[VAL_23]]] : tensor<5x6xf32>
-// CHECK:               %[[VAL_25:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_20]]] : memref<?xi32, strided<[?], offset: ?>>
+// CHECK-SAME:      %[[VAL_1:.*]]: tensor<6x2x3xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton-nu", "singleton" ], posWidth = 32, crdWidth = 32 }>>) -> tensor<?x?x?xf32> {
+// CHECK-DAG:       %[[VAL_2:.*]] = arith.constant false
+// CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 5 : index
+// CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 3 : index
+// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0 : index
+// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 1 : index
+// CHECK-DAG:       %[[VAL_7:.*]] = tensor.empty() : tensor<5x6xf32>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index}
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 0 : index}
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 1 : index}
+// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 2 : index}
+// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.values %[[VAL_1]]
+// CHECK-DAG:       %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_7]] : memref<5x6xf32>
+// CHECK:           scf.for %[[VAL_14:.*]] = %[[VAL_5]] to %[[VAL_3]] step %[[VAL_6]] {
+// CHECK:             %[[VAL_15:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_5]]] : memref<?xi32>
+// CHECK:             %[[VAL_16:.*]] = arith.extui %[[VAL_15]] : i32 to i64
+// CHECK:             %[[VAL_17:.*]] = arith.index_cast %[[VAL_16]] : i64 to index
+// CHECK:             %[[VAL_18:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_6]]] : memref<?xi32>
+// CHECK:             %[[VAL_19:.*]] = arith.extui %[[VAL_18]] : i32 to i64
+// CHECK:             %[[VAL_20:.*]] = arith.index_cast %[[VAL_19]] : i64 to index
+// CHECK:             %[[VAL_21:.*]] = scf.while (%[[VAL_22:.*]] = %[[VAL_17]]) : (index) -> index {
+// CHECK:               %[[VAL_23:.*]] = arith.cmpi ult, %[[VAL_22]], %[[VAL_20]] : index
+// CHECK:               scf.condition(%[[VAL_23]]) %[[VAL_22]] : index
+// CHECK:             } do {
+// CHECK:             ^bb0(%[[VAL_24:.*]]: index):
+// CHECK:               %[[VAL_25:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_24]]] : memref<?xi32, strided<[?], offset: ?>>
 // CHECK:               %[[VAL_26:.*]] = arith.extui %[[VAL_25]] : i32 to i64
 // CHECK:               %[[VAL_27:.*]] = arith.index_cast %[[VAL_26]] : i64 to index
-// CHECK:               %[[VAL_28:.*]] = arith.muli %[[VAL_27]], %[[VAL_3]] : index
-// CHECK:               %[[VAL_29:.*]] = memref.load %[[VAL_10]]{{\[}}%[[VAL_20]]] : memref<?xi32, strided<[?], offset: ?>>
-// CHECK:               %[[VAL_30:.*]] = arith.extui %[[VAL_29]] : i32 to i64
-// CHECK:               %[[VAL_31:.*]] = arith.index_cast %[[VAL_30]] : i64 to index
-// CHECK:               %[[VAL_32:.*]] = arith.addi %[[VAL_28]], %[[VAL_31]] : index
-// CHECK:               %[[VAL_33:.*]] = tensor.extract %[[VAL_6]]{{\[}}%[[VAL_13]], %[[VAL_32]]] : tensor<5x6xf32>
-// CHECK:               %[[VAL_34:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_20]]] : memref<?xf32>
-// CHECK:               %[[VAL_35:.*]] = arith.mulf %[[VAL_24]], %[[VAL_34]] : f32
-// CHECK:               %[[VAL_36:.*]] = arith.addf %[[VAL_33]], %[[VAL_35]] : f32
-// CHECK:               memref.store %[[VAL_36]], %[[VAL_12]]{{\[}}%[[VAL_13]], %[[VAL_32]]] : memref<5x6xf32>
-// CHECK:             } {"Emitted from" = "linalg.generic"}
+// CHECK:               %[[VAL_28:.*]] = scf.while (%[[VAL_29:.*]] = %[[VAL_24]]) : (index) -> index {
+// CHECK:                 %[[VAL_30:.*]] = arith.cmpi ult, %[[VAL_29]], %[[VAL_20]] : index
+// CHECK:                 %[[VAL_31:.*]] = scf.if %[[VAL_30]] -> (i1) {
+// CHECK:                   %[[VAL_32:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_29]]] : memref<?xi32, strided<[?], offset: ?>>
+// CHECK:                   %[[VAL_33:.*]] = arith.extui %[[VAL_32]] : i32 to i64
+// CHECK:                   %[[VAL_34:.*]] = arith.index_cast %[[VAL_33]] : i64 to index
+// CHECK:                   %[[VAL_35:.*]] = arith.cmpi eq, %[[VAL_34]], %[[VAL_27]] : index
+// CHECK:                   scf.yield %[[VAL_35]] : i1
+// CHECK:                 } else {
+// CHECK:                   scf.yield %[[VAL_2]] : i1
+// CHECK:                 }
+// CHECK:                 scf.condition(%[[VAL_36:.*]]) %[[VAL_29]] : index
+// CHECK:               } do {
+// CHECK:               ^bb0(%[[VAL_37:.*]]: index):
+// CHECK:                 %[[VAL_38:.*]] = arith.addi %[[VAL_37]], %[[VAL_6]] : index
+// CHECK:                 scf.yield %[[VAL_38]] : index
+// CHECK:               }
+// CHECK:               %[[VAL_39:.*]] = tensor.extract %[[VAL_0]]{{\[}}%[[VAL_14]], %[[VAL_27]]] : tensor<5x6xf32>
+// CHECK:               scf.for %[[VAL_40:.*]] = %[[VAL_24]] to %[[VAL_41:.*]] step %[[VAL_6]] {
+// CHECK:                 %[[VAL_42:.*]] = memref.load %[[VAL_10]]{{\[}}%[[VAL_40]]] : memref<?xi32, strided<[?], offset: ?>>
+// CHECK:                 %[[VAL_43:.*]] = arith.extui %[[VAL_42]] : i32 to i64
+// CHECK:                 %[[VAL_44:.*]] = arith.index_cast %[[VAL_43]] : i64 to index
+// CHECK:                 %[[VAL_45:.*]] = arith.muli %[[VAL_44]], %[[VAL_4]] : index
+// CHECK:                 %[[VAL_46:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_40]]] : memref<?xi32, strided<[?], offset: ?>>
+// CHECK:                 %[[VAL_47:.*]] = arith.extui %[[VAL_46]] : i32 to i64
+// CHECK:                 %[[VAL_48:.*]] = arith.index_cast %[[VAL_47]] : i64 to index
+// CHECK:                 %[[VAL_49:.*]] = arith.addi %[[VAL_45]], %[[VAL_48]] : index
+// CHECK:                 %[[VAL_50:.*]] = tensor.extract %[[VAL_7]]{{\[}}%[[VAL_14]], %[[VAL_49]]] : tensor<5x6xf32>
+// CHECK:                 %[[VAL_51:.*]] = memref.load %[[VAL_12]]{{\[}}%[[VAL_40]]] : memref<?xf32>
+// CHECK:                 %[[VAL_52:.*]] = arith.mulf %[[VAL_39]], %[[VAL_51]] : f32
+// CHECK:                 %[[VAL_53:.*]] = arith.addf %[[VAL_50]], %[[VAL_52]] : f32
+// CHECK:                 memref.store %[[VAL_53]], %[[VAL_13]]{{\[}}%[[VAL_14]], %[[VAL_49]]] : memref<5x6xf32>
+// CHECK:               } {"Emitted from" = "linalg.generic"}
+// CHECK:               scf.yield %[[VAL_54:.*]] : index
+// CHECK:             } attributes {"Emitted from" = "linalg.generic"}
 // CHECK:           } {"Emitted from" = "linalg.generic"}
-// CHECK:           %[[VAL_37:.*]] = bufferization.to_tensor %[[VAL_12]] : memref<5x6xf32>
-// CHECK:           %[[VAL_38:.*]] = tensor.expand_shape %[[VAL_37]] {{\[\[}}0], [1, 2]] : tensor<5x6xf32> into tensor<5x2x3xf32>
-// CHECK:           %[[VAL_39:.*]] = tensor.cast %[[VAL_38]] : tensor<5x2x3xf32> to tensor<?x?x?xf32>
-// CHECK:           return %[[VAL_39]] : tensor<?x?x?xf32>
+// CHECK:           %[[VAL_55:.*]] = bufferization.to_tensor %[[VAL_13]] : memref<5x6xf32>
+// CHECK:           %[[VAL_56:.*]] = tensor.expand_shape %[[VAL_55]] {{\[\[}}0], [1, 2]] : tensor<5x6xf32> into tensor<5x2x3xf32>
+// CHECK:           %[[VAL_57:.*]] = tensor.cast %[[VAL_56]] : tensor<5x2x3xf32> to tensor<?x?x?xf32>
+// CHECK:           return %[[VAL_57]] : tensor<?x?x?xf32>
 // CHECK:         }
 func.func @sparse_reshape_fused(%arg0: tensor<5x6xf32>, %arg1: tensor<6x2x3xf32, #COO_3D>) -> tensor<?x?x?xf32> {
   %collapsed = tensor.collapse_shape %arg1 [[0], [1, 2]] : tensor<6x2x3xf32, #COO_3D> into tensor<6x6xf32, #COO_2D>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_scalars.mlir b/mlir/test/Dialect/SparseTensor/sparse_scalars.mlir
index de8d36861a779..63fd33fcd792c 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_scalars.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_scalars.mlir
@@ -28,10 +28,10 @@
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.addf %[[VAL_2]], %[[VAL_3]] : f32
-// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_10:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<32x16xf32, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<32x16xf32, #sparse_tensor.encoding<{{.*}}>> to memref<?xf32>
 // CHECK-DAG:       %[[VAL_14:.*]] = bufferization.to_memref %[[VAL_1]] : memref<f32>
 // CHECK-DAG:       %[[VAL_15:.*]] = bufferization.to_memref %[[VAL_4]] : memref<32x16xf32>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_sddmm.mlir b/mlir/test/Dialect/SparseTensor/sparse_sddmm.mlir
index 50d424fcd73c7..cfab72c3ebfb1 100755
--- a/mlir/test/Dialect/SparseTensor/sparse_sddmm.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_sddmm.mlir
@@ -68,10 +68,10 @@ func.func @fold_yield_direct_zero() -> tensor<32xf64> {
 // CHECK-DAG:       %[[VAL_8:.*]] = bufferization.alloc_tensor() copy(%[[VAL_6]]) {bufferization.escape = [false], memory_space = 0 : i64} : tensor<8x8xf64>
 // CHECK-DAG:       %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_1]] : memref<8x8xf64>
 // CHECK-DAG:       %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_2]] : memref<8x8xf64>
-// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_11:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_12:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_15:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<8x8xf64, #sparse_tensor.encoding<{{.*}}>> to memref<?xf64>
 // CHECK:           %[[VAL_16:.*]] = bufferization.to_memref %[[VAL_8]] : memref<8x8xf64>
 // CHECK:           %[[VAL_17:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_4]]] : memref<?xindex>
@@ -136,10 +136,10 @@ func.func @sampled_dd_unfused(%args: tensor<8x8xf64, #SM>,
 // CHECK-DAG:       %[[VAL_10:.*]] = bufferization.alloc_tensor() {bufferization.escape = [false]} : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>
 // CHECK-DAG:       %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_1]] : memref<8x8xf64>
 // CHECK-DAG:       %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_2]] : memref<8x8xf64>
-// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_15:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_16:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_13:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_14:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_15:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_16:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_17:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf64>
 // CHECK:           %[[VAL_18:.*]] = memref.load %[[VAL_13]]{{\[}}%[[VAL_4]]] : memref<?xindex>
 // CHECK:           %[[VAL_19:.*]] = memref.load %[[VAL_13]]{{\[}}%[[VAL_5]]] : memref<?xindex>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_sddmm_org.mlir b/mlir/test/Dialect/SparseTensor/sparse_sddmm_org.mlir
index 9e56f1cf2d7dd..aa6d202bb734d 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_sddmm_org.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_sddmm_org.mlir
@@ -32,10 +32,10 @@
 // CHECK:           %[[VAL_8:.*]] = bufferization.alloc_tensor() : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>
 // CHECK:           %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_1]] : memref<8x8xf64>
 // CHECK:           %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_2]] : memref<8x8xf64>
-// CHECK:           %[[VAL_11:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_12:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_13:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_14:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_11:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_12:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_13:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 1 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_14:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 1 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK:           %[[VAL_15:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf64>
 // CHECK:           %[[VAL_16:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_4]]] : memref<?xindex>
 // CHECK:           %[[VAL_17:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_5]]] : memref<?xindex>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_storage.mlir b/mlir/test/Dialect/SparseTensor/sparse_storage.mlir
index 66ab1ce349728..a8714e41a87da 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_storage.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_storage.mlir
@@ -2,14 +2,14 @@
 
 #SparseVector64 = #sparse_tensor.encoding<{
   dimLevelType = [ "compressed" ],
-  pointerBitWidth = 64,
-  indexBitWidth = 64
+  posWidth = 64,
+  crdWidth = 64
 }>
 
 #SparseVector32 = #sparse_tensor.encoding<{
   dimLevelType = [ "compressed" ],
-  pointerBitWidth = 32,
-  indexBitWidth = 32
+  posWidth = 32,
+  crdWidth = 32
 }>
 
 #trait_mul = {
diff --git a/mlir/test/Dialect/SparseTensor/sparse_transpose.mlir b/mlir/test/Dialect/SparseTensor/sparse_transpose.mlir
index e909e097aa1c8..fdcfd736b9555 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_transpose.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_transpose.mlir
@@ -21,10 +21,10 @@
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_3:.*]] = bufferization.alloc_tensor() : tensor<4x3xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>
 // CHECK-DAG:       %[[VAL_4:.*]] = sparse_tensor.convert %[[VAL_0]] : tensor<3x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to tensor<3x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)> }>>
-// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_4]] {dimension = 0 : index} : tensor<3x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)> }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_4]] {dimension = 0 : index} : tensor<3x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)> }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_4]] {dimension = 1 : index} : tensor<3x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)> }>> to memref<?xindex>
-// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_4]] {dimension = 1 : index} : tensor<3x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)> }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_4]] {level = 0 : index} : tensor<3x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)> }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_6:.*]] = sparse_tensor.coordinates %[[VAL_4]] {level = 0 : index} : tensor<3x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)> }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_4]] {level = 1 : index} : tensor<3x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)> }>> to memref<?xindex>
+// CHECK-DAG:       %[[VAL_8:.*]] = sparse_tensor.coordinates %[[VAL_4]] {level = 1 : index} : tensor<3x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)> }>> to memref<?xindex>
 // CHECK-DAG:       %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_4]] : tensor<3x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)> }>> to memref<?xf64>
 // CHECK:           %[[VAL_10:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_1]]] : memref<?xindex>
 // CHECK:           %[[VAL_11:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_2]]] : memref<?xindex>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_vector.mlir b/mlir/test/Dialect/SparseTensor/sparse_vector.mlir
index a5c0239dc7edd..a14bec9d05305 100755
--- a/mlir/test/Dialect/SparseTensor/sparse_vector.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_vector.mlir
@@ -87,8 +87,8 @@ func.func @scale_d(%arga: tensor<1024xf32, #DenseVector>, %b: f32, %argx: tensor
 
 #SparseVector = #sparse_tensor.encoding<{
   dimLevelType = [ "compressed" ],
-  pointerBitWidth = 32,
-  indexBitWidth = 32
+  posWidth = 32,
+  crdWidth = 32
 }>
 
 #trait_mul_s = {
@@ -310,8 +310,8 @@ func.func @reduction_d(%arga: tensor<1024xf32, #DenseVector>,
 
 #SparseMatrix = #sparse_tensor.encoding<{
   dimLevelType = [ "dense", "compressed" ],
-  pointerBitWidth = 32,
-  indexBitWidth = 32
+  posWidth = 32,
+  crdWidth = 32
 }>
 
 #trait_mul_ds = {
diff --git a/mlir/test/Dialect/SparseTensor/sparse_vector_chain.mlir b/mlir/test/Dialect/SparseTensor/sparse_vector_chain.mlir
index 824fd40448d20..37545618bd58e 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_vector_chain.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_vector_chain.mlir
@@ -25,11 +25,11 @@
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 64 : index
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 1 : index
-// CHECK:           %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index} : tensor<64x32xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index} : tensor<64x32xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 1 : index} : tensor<64x32xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_1]] {level = 1 : index} : tensor<64x32xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK:           %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<64x32xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xf64>
-// CHECK:           %[[VAL_11:.*]] = sparse_tensor.pointers %[[VAL_2]] {dimension = 1 : index} : tensor<64x32xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_12:.*]] = sparse_tensor.indices %[[VAL_2]] {dimension = 1 : index} : tensor<64x32xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_11:.*]] = sparse_tensor.positions %[[VAL_2]] {level = 1 : index} : tensor<64x32xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_12:.*]] = sparse_tensor.coordinates %[[VAL_2]] {level = 1 : index} : tensor<64x32xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK:           %[[VAL_13:.*]] = sparse_tensor.values %[[VAL_2]] : tensor<64x32xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xf64>
 // CHECK:           %[[VAL_14:.*]] = bufferization.to_memref %[[VAL_0]] : memref<f64>
 // CHECK:           %[[VAL_15:.*]] = memref.load %[[VAL_14]][] : memref<f64>
diff --git a/mlir/test/Dialect/SparseTensor/sparse_vector_index.mlir b/mlir/test/Dialect/SparseTensor/sparse_vector_index.mlir
index 7bb8254e04856..637a1bb350729 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_vector_index.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_vector_index.mlir
@@ -25,8 +25,8 @@
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_7:.*]] = tensor.empty() : tensor<8xi64>
-// CHECK:           %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<8xi64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<8xi64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<8xi64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<8xi64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK:           %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<8xi64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xi64>
 // CHECK:           %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_7]] : memref<8xi64>
 // CHECK:           linalg.fill ins(%[[VAL_4]] : i64) outs(%[[VAL_11]] : memref<8xi64>)
@@ -67,8 +67,8 @@ func.func @sparse_index_1d_conj(%arga: tensor<8xi64, #SparseVector>) -> tensor<8
 // CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
 // CHECK-DAG:       %[[VAL_6:.*]] = arith.constant true
 // CHECK-DAG:       %[[VAL_7:.*]] = tensor.empty() : tensor<8xi64>
-// CHECK:           %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<8xi64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
-// CHECK:           %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<8xi64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_0]] {level = 0 : index} : tensor<8xi64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK:           %[[VAL_9:.*]] = sparse_tensor.coordinates %[[VAL_0]] {level = 0 : index} : tensor<8xi64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK:           %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<8xi64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xi64>
 // CHECK:           %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_7]] : memref<8xi64>
 // CHECK:           linalg.fill ins(%[[VAL_3]] : i64) outs(%[[VAL_11]] : memref<8xi64>)
diff --git a/mlir/test/Dialect/SparseTensor/sparse_vector_peeled.mlir b/mlir/test/Dialect/SparseTensor/sparse_vector_peeled.mlir
index d379965ef01df..65df2c11fee6c 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_vector_peeled.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_vector_peeled.mlir
@@ -3,8 +3,8 @@
 
 #SparseVector = #sparse_tensor.encoding<{
   dimLevelType = [ "compressed" ],
-  pointerBitWidth = 32,
-  indexBitWidth = 32
+  posWidth = 32,
+  crdWidth = 32
 }>
 
 #trait_mul_s = {
diff --git a/mlir/test/Dialect/SparseTensor/specifier_to_llvm.mlir b/mlir/test/Dialect/SparseTensor/specifier_to_llvm.mlir
index 36dce9e417b5f..9fb512c98112b 100644
--- a/mlir/test/Dialect/SparseTensor/specifier_to_llvm.mlir
+++ b/mlir/test/Dialect/SparseTensor/specifier_to_llvm.mlir
@@ -21,7 +21,7 @@ func.func @sparse_metadata_init() -> !sparse_tensor.storage_specifier<#CSR> {
 // CHECK:           %[[CAST:.*]] = arith.index_cast %[[VAL_1]] : i64 to index
 // CHECK:           return %[[CAST]] : index
 func.func @sparse_get_md(%arg0: !sparse_tensor.storage_specifier<#CSR>) -> index {
-  %0 = sparse_tensor.storage_specifier.get %arg0 dim_sz at 0
+  %0 = sparse_tensor.storage_specifier.get %arg0 lvl_sz at 0
        : !sparse_tensor.storage_specifier<#CSR>
   return %0 : index
 }
@@ -34,7 +34,7 @@ func.func @sparse_get_md(%arg0: !sparse_tensor.storage_specifier<#CSR>) -> index
 // CHECK:           return %[[VAL_2]] : !llvm.struct<(array<2 x i64>, array<3 x i64>)>
 func.func @sparse_set_md(%arg0: !sparse_tensor.storage_specifier<#CSR>, %arg1: index)
           -> !sparse_tensor.storage_specifier<#CSR> {
-  %0 = sparse_tensor.storage_specifier.set %arg0 dim_sz at 0 with %arg1
+  %0 = sparse_tensor.storage_specifier.set %arg0 lvl_sz at 0 with %arg1
        : !sparse_tensor.storage_specifier<#CSR>
   return %0 : !sparse_tensor.storage_specifier<#CSR>
 }
diff --git a/mlir/test/Dialect/SparseTensor/vectorize_reduction.mlir b/mlir/test/Dialect/SparseTensor/vectorize_reduction.mlir
index 57df898e8209d..bd68bff43ac55 100644
--- a/mlir/test/Dialect/SparseTensor/vectorize_reduction.mlir
+++ b/mlir/test/Dialect/SparseTensor/vectorize_reduction.mlir
@@ -35,7 +35,7 @@
 // CHECK-ON-DAG:       %[[VAL_5:.*]] = arith.constant 0 : index
 // CHECK-ON-DAG:       %[[VAL_6:.*]] = arith.constant 1 : index
 // CHECK-ON-DAG:       %[[VAL_7:.*]] = tensor.dim %[[VAL_1]], %[[VAL_5]] : tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>>
-// CHECK-ON:           %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index} : tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-ON:           %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 1 : index} : tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK-ON:           %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xf64>
 // CHECK-ON:           %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_0]] : memref<f64>
 // CHECK-ON:           %[[VAL_11:.*]] = memref.load %[[VAL_10]][] : memref<f64>
@@ -66,7 +66,7 @@
 // CHECK-OFF-DAG:       %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-OFF-DAG:       %[[VAL_3:.*]] = arith.constant 1 : index
 // CHECK-OFF:           %[[VAL_4:.*]] = tensor.dim %[[VAL_1]], %[[VAL_2]] : tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>>
-// CHECK-OFF:           %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index} : tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
+// CHECK-OFF:           %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 1 : index} : tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xindex>
 // CHECK-OFF:           %[[VAL_6:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ] }>> to memref<?xf64>
 // CHECK-OFF:           %[[VAL_7:.*]] = bufferization.to_memref %[[VAL_0]] : memref<f64>
 // CHECK-OFF:           %[[VAL_8:.*]] = memref.load %[[VAL_7]][] : memref<f64>
@@ -121,8 +121,8 @@ func.func @sparse_product_reduction_dense_sparse(%argx: tensor<f64>,
 // CHECK-ON-DAG:       %[[VAL_4:.*]] = arith.constant dense<0.000000e+00> : vector<8xf64>
 // CHECK-ON-DAG:       %[[VAL_5:.*]] = arith.constant 0 : index
 // CHECK-ON-DAG:       %[[VAL_6:.*]] = arith.constant 1 : index
-// CHECK-ON:           %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-ON:           %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index} : tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-ON:           %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-ON:           %[[VAL_8:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 1 : index} : tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-ON:           %[[VAL_9:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf64>
 // CHECK-ON:           %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_0]] : memref<f64>
 // CHECK-ON:           %[[VAL_11:.*]] = memref.load %[[VAL_10]][] : memref<f64>
@@ -154,8 +154,8 @@ func.func @sparse_product_reduction_dense_sparse(%argx: tensor<f64>,
 // CHECK-OFF-SAME:                                                     %[[VAL_1:.*]]: tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>) -> tensor<f64> {
 // CHECK-OFF-DAG:       %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-OFF-DAG:       %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK-OFF:           %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
-// CHECK-OFF:           %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index} : tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-OFF:           %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
+// CHECK-OFF:           %[[VAL_5:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 1 : index} : tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex>
 // CHECK-OFF:           %[[VAL_6:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?x128xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf64>
 // CHECK-OFF:           %[[VAL_7:.*]] = bufferization.to_memref %[[VAL_0]] : memref<f64>
 // CHECK-OFF:           %[[VAL_8:.*]] = memref.load %[[VAL_7]][] : memref<f64>
@@ -216,7 +216,7 @@ func.func @sparse_product_reduction_sparse_sparse(%argx: tensor<f64>,
 // CHECK-ON-DAG:       %[[VAL_3:.*]] = arith.constant dense<0> : vector<8xi13>
 // CHECK-ON-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-ON-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-ON:           %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xi13, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-ON:           %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi13, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-ON:           %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi13, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xi13>
 // CHECK-ON:           %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_0]] : memref<i13>
 // CHECK-ON:           %[[VAL_9:.*]] = memref.load %[[VAL_8]][] : memref<i13>
@@ -242,7 +242,7 @@ func.func @sparse_product_reduction_sparse_sparse(%argx: tensor<f64>,
 // CHECK-OFF-SAME:                                    %[[VAL_1:.*]]: tensor<?xi13, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>>) -> tensor<i13> {
 // CHECK-OFF-DAG:       %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-OFF-DAG:       %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK-OFF:           %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xi13, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-OFF:           %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi13, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-OFF:           %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi13, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xi13>
 // CHECK-OFF:           %[[VAL_6:.*]] = bufferization.to_memref %[[VAL_0]] : memref<i13>
 // CHECK-OFF:           %[[VAL_7:.*]] = memref.load %[[VAL_6]][] : memref<i13>
@@ -294,7 +294,7 @@ func.func @sparse_reduction_ori(%argx: tensor<i13>,
 // CHECK-ON-DAG:       %[[VAL_3:.*]] = arith.constant dense<0> : vector<8xi13>
 // CHECK-ON-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-ON-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-ON:           %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xi13, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-ON:           %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi13, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-ON:           %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi13, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xi13>
 // CHECK-ON:           %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_0]] : memref<i13>
 // CHECK-ON:           %[[VAL_9:.*]] = memref.load %[[VAL_8]][] : memref<i13>
@@ -320,7 +320,7 @@ func.func @sparse_reduction_ori(%argx: tensor<i13>,
 // CHECK-OFF-SAME:                                    %[[VAL_1:.*]]: tensor<?xi13, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>>) -> tensor<i13> {
 // CHECK-OFF-DAG:       %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-OFF-DAG:       %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK-OFF:           %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xi13, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-OFF:           %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi13, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-OFF:           %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi13, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xi13>
 // CHECK-OFF:           %[[VAL_6:.*]] = bufferization.to_memref %[[VAL_0]] : memref<i13>
 // CHECK-OFF:           %[[VAL_7:.*]] = memref.load %[[VAL_6]][] : memref<i13>
@@ -369,7 +369,7 @@ func.func @sparse_reduction_ori_accumulator_on_rhs(%argx: tensor<i13>,
 // CHECK-ON-DAG:       %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-ON-DAG:       %[[VAL_4:.*]] = arith.constant dense<0> : vector<8xi32>
 // CHECK-ON-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-ON:           %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-ON:           %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-ON:           %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xi32>
 // CHECK-ON:           %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_0]] : memref<i32>
 // CHECK-ON:           %[[VAL_9:.*]] = memref.load %[[VAL_8]][] : memref<i32>
@@ -395,7 +395,7 @@ func.func @sparse_reduction_ori_accumulator_on_rhs(%argx: tensor<i13>,
 // CHECK-OFF-SAME:                                     %[[VAL_1:.*]]: tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>>) -> tensor<i32> {
 // CHECK-OFF-DAG:       %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-OFF-DAG:       %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK-OFF:           %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-OFF:           %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-OFF:           %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xi32>
 // CHECK-OFF:           %[[VAL_6:.*]] = bufferization.to_memref %[[VAL_0]] : memref<i32>
 // CHECK-OFF:           %[[VAL_7:.*]] = memref.load %[[VAL_6]][] : memref<i32>
@@ -446,7 +446,7 @@ func.func @sparse_reduction_subi(%argx: tensor<i32>,
 // CHECK-ON-DAG:  %[[VAL_3:.*]] = arith.constant dense<0> : vector<8xi32>
 // CHECK-ON-DAG:  %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-ON-DAG:  %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-ON:  %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-ON:  %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-ON:  %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xi32>
 // CHECK-ON:  %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_0]] : memref<i32>
 // CHECK-ON:  %[[VAL_9:.*]] = memref.load %[[VAL_8]][] : memref<i32>
@@ -472,7 +472,7 @@ func.func @sparse_reduction_subi(%argx: tensor<i32>,
 // CHECK-OFF-SAME:  %[[VAL_1:.*]]: tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>>) -> tensor<i32> {
 // CHECK-OFF-DAG:   %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-OFF-DAG:   %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK-OFF:   %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-OFF:   %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-OFF:   %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xi32>
 // CHECK-OFF:   %[[VAL_6:.*]] = bufferization.to_memref %[[VAL_0]] : memref<i32>
 // CHECK-OFF:   %[[VAL_7:.*]] = memref.load %[[VAL_6]][] : memref<i32>
@@ -520,7 +520,7 @@ func.func @sparse_reduction_xor(%argx: tensor<i32>,
 // CHECK-ON-DAG:   %[[VAL_3:.*]] = arith.constant dense<0> : vector<8xi32>
 // CHECK-ON-DAG:   %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-ON-DAG:   %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-ON:   %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-ON:   %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-ON:   %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xi32>
 // CHECK-ON:   %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_0]] : memref<i32>
 // CHECK-ON:   %[[VAL_9:.*]] = memref.load %[[VAL_8]][] : memref<i32>
@@ -546,7 +546,7 @@ func.func @sparse_reduction_xor(%argx: tensor<i32>,
 // CHECK-OFF-SAME:   %[[VAL_1:.*]]: tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>>) -> tensor<i32> {
 // CHECK-OFF-DAG:   %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-OFF-DAG:   %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK-OFF:   %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-OFF:   %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-OFF:   %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xi32>
 // CHECK-OFF:   %[[VAL_6:.*]] = bufferization.to_memref %[[VAL_0]] : memref<i32>
 // CHECK-OFF:   %[[VAL_7:.*]] = memref.load %[[VAL_6]][] : memref<i32>
@@ -595,7 +595,7 @@ func.func @sparse_reduction_and(%argx: tensor<i32>,
 // CHECK-ON-DAG:   %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-ON-DAG:   %[[VAL_5:.*]] = arith.constant dense<0> : vector<8xi32>
 // CHECK-ON-DAG:   %[[VAL_6:.*]] = arith.constant 1 : index
-// CHECK-ON:   %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-ON:   %[[VAL_7:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-ON:   %[[VAL_8:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xi32>
 // CHECK-ON:   %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_0]] : memref<i32>
 // CHECK-ON:   %[[VAL_10:.*]] = memref.load %[[VAL_9]][] : memref<i32>
@@ -621,7 +621,7 @@ func.func @sparse_reduction_and(%argx: tensor<i32>,
 // CHECK-OFF-SAME:   %[[VAL_1:.*]]: tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>>) -> tensor<i32> {
 // CHECK-OFF-DAG:   %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-OFF-DAG:   %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK-OFF:   %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-OFF:   %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-OFF:   %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xi32>
 // CHECK-OFF:   %[[VAL_6:.*]] = bufferization.to_memref %[[VAL_0]] : memref<i32>
 // CHECK-OFF:   %[[VAL_7:.*]] = memref.load %[[VAL_6]][] : memref<i32>
@@ -669,7 +669,7 @@ func.func @sparse_reduction_muli(%argx: tensor<i32>,
 // CHECK-ON-DAG:   %[[VAL_3:.*]] = arith.constant dense<0> : vector<8xi32>
 // CHECK-ON-DAG:   %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-ON-DAG:   %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-ON:   %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-ON:   %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-ON:   %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xi32>
 // CHECK-ON:   %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_0]] : memref<i32>
 // CHECK-ON:   %[[VAL_9:.*]] = memref.load %[[VAL_8]][] : memref<i32>
@@ -695,7 +695,7 @@ func.func @sparse_reduction_muli(%argx: tensor<i32>,
 // CHECK-OFF-SAME:   %[[VAL_1:.*]]: tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>>) -> tensor<i32> {
 // CHECK-OFF-DAG:   %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-OFF-DAG:   %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK-OFF:   %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-OFF:   %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-OFF:   %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xi32>
 // CHECK-OFF:   %[[VAL_6:.*]] = bufferization.to_memref %[[VAL_0]] : memref<i32>
 // CHECK-OFF:   %[[VAL_7:.*]] = memref.load %[[VAL_6]][] : memref<i32>
@@ -743,7 +743,7 @@ func.func @sparse_reduction_addi(%argx: tensor<i32>,
 // CHECK-ON-DAG:   %[[VAL_3:.*]] = arith.constant dense<0.000000e+00> : vector<8xf32>
 // CHECK-ON-DAG:   %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-ON-DAG:   %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-ON:   %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-ON:   %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-ON:   %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-ON:   %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_0]] : memref<f32>
 // CHECK-ON:   %[[VAL_9:.*]] = memref.load %[[VAL_8]][] : memref<f32>
@@ -769,7 +769,7 @@ func.func @sparse_reduction_addi(%argx: tensor<i32>,
 // CHECK-OFF-SAME:   %[[VAL_1:.*]]: tensor<?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>>) -> tensor<f32> {
 // CHECK-OFF-DAG:   %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-OFF-DAG:   %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK-OFF:   %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-OFF:   %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-OFF:   %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-OFF:   %[[VAL_6:.*]] = bufferization.to_memref %[[VAL_0]] : memref<f32>
 // CHECK-OFF:   %[[VAL_7:.*]] = memref.load %[[VAL_6]][] : memref<f32>
@@ -817,7 +817,7 @@ func.func @sparse_reduction_subf(%argx: tensor<f32>,
 // CHECK-ON-DAG:   %[[VAL_3:.*]] = arith.constant dense<0.000000e+00> : vector<8xf32>
 // CHECK-ON-DAG:   %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-ON-DAG:   %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-ON:   %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-ON:   %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-ON:   %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-ON:   %[[VAL_8:.*]] = bufferization.to_memref %[[VAL_0]] : memref<f32>
 // CHECK-ON:   %[[VAL_9:.*]] = memref.load %[[VAL_8]][] : memref<f32>
@@ -843,7 +843,7 @@ func.func @sparse_reduction_subf(%argx: tensor<f32>,
 // CHECK-OFF-SAME:    %[[VAL_1:.*]]: tensor<?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>>) -> tensor<f32> {
 // CHECK-OFF-DAG:   %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-OFF-DAG:   %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK-OFF:   %[[VAL_4:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
+// CHECK-OFF:   %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
 // CHECK-OFF:   %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
 // CHECK-OFF:   %[[VAL_6:.*]] = bufferization.to_memref %[[VAL_0]] : memref<f32>
 // CHECK-OFF:   %[[VAL_7:.*]] = memref.load %[[VAL_6]][] : memref<f32>
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index fa8257746e875..68eca320aa46b 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -51,6 +51,13 @@ func.func @test_depthwise_conv2d(%arg0: tensor<1x4x4x4xf32>, %arg1: tensor<1x1x4
   return %2 : tensor<1x4x4x8xf32>
 }
 
+// -----
+// CHECK-LABEL: fft2d
+func.func @test_fft2d(%arg0: tensor<1x4x8xf32>, %arg1: tensor<1x4x8xf32>) -> (tensor<1x4x8xf32>, tensor<1x4x8xf32>) {
+  %0, %1 = "tosa.fft2d"(%arg0, %arg1) {inverse = false} : (tensor<1x4x8xf32>, tensor<1x4x8xf32>) -> (tensor<1x4x8xf32>, tensor<1x4x8xf32>)
+  return %0, %1 : tensor<1x4x8xf32>, tensor<1x4x8xf32>
+}
+
 // -----
 // CHECK-LABEL: fully_connected
 func.func @test_fully_connected(%arg0: tensor<14x19xf32>, %arg1: tensor<19x28xf32>, %arg2: tensor<28xf32>) -> tensor<14x28xf32> {
diff --git a/mlir/test/Dialect/Tosa/tosa-infer-shapes.mlir b/mlir/test/Dialect/Tosa/tosa-infer-shapes.mlir
index c955d57119733..94eea3b36eae2 100644
--- a/mlir/test/Dialect/Tosa/tosa-infer-shapes.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-infer-shapes.mlir
@@ -1216,3 +1216,21 @@ func.func @test_dynamic_width_rfft2d(%arg0 : tensor<5x2x?xf32>) -> () {
   %output_real, %output_imag = "tosa.rfft2d"(%arg0) {} : (tensor<5x2x?xf32>) -> (tensor<?x?x?xf32>, tensor<?x?x?xf32>)
   return
 }
+
+// -----
+
+// CHECK-LABEL: @test_static_fft2d
+func.func @test_static_fft2d(%arg0: tensor<1x4x8xf32>, %arg1: tensor<1x4x8xf32>) -> (tensor<1x4x8xf32>, tensor<1x4x8xf32>) {
+  // CHECK: -> (tensor<1x4x8xf32>, tensor<1x4x8xf32>)
+  %output_real, %output_imag = "tosa.fft2d"(%arg0, %arg1) {inverse = false} : (tensor<1x4x8xf32>, tensor<1x4x8xf32>) -> (tensor<1x4x8xf32>, tensor<1x4x8xf32>)
+  return %output_real, %output_imag : tensor<1x4x8xf32>, tensor<1x4x8xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @test_dynamic_batch_fft2d
+func.func @test_dynamic_batch_fft2d(%arg0: tensor<?x4x8xf32>, %arg1: tensor<?x4x8xf32>) -> (tensor<?x4x8xf32>, tensor<?x4x8xf32>) {
+  // CHECK: -> (tensor<?x4x8xf32>, tensor<?x4x8xf32>)
+  %output_real, %output_imag = "tosa.fft2d"(%arg0, %arg1) {inverse = false} : (tensor<?x4x8xf32>, tensor<?x4x8xf32>) -> (tensor<?x4x8xf32>, tensor<?x4x8xf32>)
+  return %output_real, %output_imag : tensor<?x4x8xf32>, tensor<?x4x8xf32>
+}
diff --git a/mlir/test/Dialect/Transform/transform-state-extension.mlir b/mlir/test/Dialect/Transform/transform-state-extension.mlir
index 054ee077496c5..7885e08291716 100644
--- a/mlir/test/Dialect/Transform/transform-state-extension.mlir
+++ b/mlir/test/Dialect/Transform/transform-state-extension.mlir
@@ -47,15 +47,36 @@ module {
 
 // -----
 
-// expected-error @below {{cannot replace an op with another op producing a different number of results while tracking handles}}
-module {
-  transform.sequence failures(propagate) {
-  ^bb0(%arg0: !pdl.operation):
-    test_add_test_extension "A"
-    %dummy = test_remap_operand_to_self %arg0 : !transform.any_op
-  }
+transform.sequence failures(propagate) {
+^bb0(%arg0: !pdl.operation):
+  test_add_test_extension "A"
+   // This is okay because we are replacing the top-level module opeation
+   // (0 results) with this operation that has _more_ (1) results.
+  %dummy = test_remap_operand_to_self %arg0 : !pdl.operation
+}
+
+// -----
+
+transform.sequence failures(propagate) {
+^bb0(%arg0: !pdl.operation):
+  test_add_test_extension "A"
+  %dummy = test_remap_operand_to_self %arg0 : !pdl.operation
+  // This is still okay. Even though we are replacing the previous
+  // operation with (1 result) with this operation that has less (0) results,
+  // there is no handle to the result, hence no issue with value handle update.
+  test_remap_operand_to_self %dummy
 }
 
+// -----
+
+transform.sequence failures(propagate) {
+^bb0(%arg0: !pdl.operation):
+  test_add_test_extension "A"
+  // expected-error @below {{cannot replace an op with another op producing less results while tracking handles}}
+  %dummy = test_remap_operand_to_self %arg0 : !pdl.operation
+  %valuehandle = transform.get_result %dummy[0] : (!pdl.operation) -> !transform.any_value
+  test_remap_operand_to_self %dummy
+}
 
 // -----
 
diff --git a/mlir/test/Dialect/Vector/canonicalize.mlir b/mlir/test/Dialect/Vector/canonicalize.mlir
index 053e3620cab2e..f82540c28f829 100644
--- a/mlir/test/Dialect/Vector/canonicalize.mlir
+++ b/mlir/test/Dialect/Vector/canonicalize.mlir
@@ -54,6 +54,19 @@ func.func @create_vector_mask_to_constant_mask_truncation_zero() -> (vector<4x3x
 
 // -----
 
+// CHECK-LABEL: create_mask_transpose_to_transposed_create_mask
+//  CHECK-SAME: %[[DIM0:.*]]: index, %[[DIM1:.*]]: index, %[[DIM2:.*]]: index
+func.func @create_mask_transpose_to_transposed_create_mask(
+  %dim0: index, %dim1: index, %dim2: index) -> (vector<2x3x4xi1>, vector<4x2x3xi1>) {
+  // CHECK: vector.create_mask %[[DIM0]], %[[DIM1]], %[[DIM2]] : vector<2x3x4xi1>
+  // CHECK: vector.create_mask %[[DIM2]], %[[DIM0]], %[[DIM1]] : vector<4x2x3xi1>
+  %0 = vector.create_mask %dim0, %dim1, %dim2 : vector<2x3x4xi1>
+  %1 = vector.transpose %0, [2, 0, 1] : vector<2x3x4xi1> to vector<4x2x3xi1>
+  return %0, %1 : vector<2x3x4xi1>, vector<4x2x3xi1>
+}
+
+// -----
+
 func.func @extract_strided_slice_of_constant_mask() -> (vector<2x2xi1>) {
   %0 = vector.constant_mask [2, 2] : vector<4x3xi1>
   %1 = vector.extract_strided_slice %0
diff --git a/mlir/test/Dialect/Vector/vector-contract-matmul-transforms.mlir b/mlir/test/Dialect/Vector/vector-contract-matmul-transforms.mlir
new file mode 100644
index 0000000000000..d0be1230b46b3
--- /dev/null
+++ b/mlir/test/Dialect/Vector/vector-contract-matmul-transforms.mlir
@@ -0,0 +1,198 @@
+// RUN: mlir-opt %s -test-vector-contraction-prepare-for-mmt-lowering | FileCheck %s
+
+// CHECK-LABEL: func.func @not_matmul
+// CHECK-SAME:    ([[ARG0:%.+]]: vector<4xf32>, [[ARG1:%.+]]: vector<4xf32>, [[ARG2:%.+]]: f32)
+// CHECK-NEXT:    vector.contract
+// CHECK-NEXT:    return
+func.func @not_matmul(%arg0: vector<4xf32>, %arg1: vector<4xf32>, %arg2: f32) -> f32 {
+  %0 = vector.contract {indexing_maps = [affine_map<(d0) -> (d0)>,
+                                         affine_map<(d0) -> (d0)>,
+                                         affine_map<(d0) -> ()>],
+                        iterator_types = ["reduction"],
+                        kind = #vector.kind<add>} %arg0, %arg1, %arg2 :
+         vector<4xf32>, vector<4xf32> into f32
+  return %0 : f32
+}
+
+// This contraction is already in the canonical form.
+// CHECK-LABEL: func.func @matmul_mk_nk_mn_4x4xi32
+// CHECK-SAME:    ([[ARG0:%.+]]: vector<4x4xi32>, [[ARG1:%.+]]: vector<4x4xi32>, [[ARG2:%.+]]: vector<4x4xi32>)
+// CHECK-NEXT:    [[RES:%.+]]   = vector.contract {{.+}} [[ARG0]], [[ARG1]], [[ARG2]]
+// CHECK-NEXT:    return [[RES]]
+func.func @matmul_mk_nk_mn_4x4xi32(%arg0: vector<4x4xi32>, %arg1: vector<4x4xi32>, %arg2: vector<4x4xi32>) -> vector<4x4xi32> {
+  %res = vector.contract {indexing_maps = [affine_map<(d0, d1, d2) -> (d0, d2)>,
+                                           affine_map<(d0, d1, d2) -> (d1, d2)>,
+                                           affine_map<(d0, d1, d2) -> (d0, d1)>],
+                          iterator_types = ["parallel", "parallel", "reduction"],
+                          kind = #vector.kind<add>} %arg0, %arg1, %arg2 : vector<4x4xi32>, vector<4x4xi32> into vector<4x4xi32>
+  return %res : vector<4x4xi32>
+}
+
+// CHECK-LABEL: func.func @matmul_mk_kn_mn_4x4xi32
+// CHECK-SAME:    ([[ARG0:%.+]]: vector<4x4xi32>, [[ARG1:%.+]]: vector<4x4xi32>, [[ARG2:%.+]]: vector<4x4xi32>)
+// CHECK-NEXT:    [[TRANS:%.+]] = vector.transpose [[ARG1]], [1, 0] : vector<4x4xi32> to vector<4x4xi32>
+// CHECK-NEXT:    [[RES:%.+]]   = vector.contract {{.+}} [[ARG0]], [[TRANS]], [[ARG2]]
+// CHECK-NEXT:    return [[RES]]
+func.func @matmul_mk_kn_mn_4x4xi32(%arg0: vector<4x4xi32>, %arg1: vector<4x4xi32>, %arg2: vector<4x4xi32>) -> vector<4x4xi32> {
+  %res = vector.contract {indexing_maps = [affine_map<(d0, d1, d2) -> (d0, d2)>,
+                                           affine_map<(d0, d1, d2) -> (d2, d1)>,
+                                           affine_map<(d0, d1, d2) -> (d0, d1)>],
+                          iterator_types = ["parallel", "parallel", "reduction"],
+                          kind = #vector.kind<add>} %arg0, %arg1, %arg2 : vector<4x4xi32>, vector<4x4xi32> into vector<4x4xi32>
+  return %res : vector<4x4xi32>
+}
+
+// CHECK-LABEL: func.func @matmul_mk_kn_mn_4x4xi8_extsi_i32
+// CHECK-SAME:    ([[ARG0:%.+]]: vector<4x4xi8>, [[ARG1:%.+]]: vector<4x4xi8>, [[ARG2:%.+]]: vector<4x4xi32>)
+// CHECK-NEXT:    [[LHS:%.+]]   = arith.extsi [[ARG0]] : vector<4x4xi8> to vector<4x4xi32>
+// CHECK-NEXT:    [[TRANS:%.+]] = vector.transpose [[ARG1]], [1, 0] : vector<4x4xi8> to vector<4x4xi8>
+// CHECK-NEXT:    [[RHS:%.+]]   = arith.extsi [[TRANS]] : vector<4x4xi8> to vector<4x4xi32>
+// CHECK-NEXT:    [[RES:%.+]]   = vector.contract {{.+}} [[LHS]], [[RHS]], [[ARG2]]
+// CHECK-NEXT:    return [[RES]]
+func.func @matmul_mk_kn_mn_4x4xi8_extsi_i32(%arg0: vector<4x4xi8>, %arg1: vector<4x4xi8>, %arg2: vector<4x4xi32>) -> vector<4x4xi32> {
+  %lhs = arith.extsi %arg0: vector<4x4xi8> to vector<4x4xi32>
+  %rhs = arith.extsi %arg1: vector<4x4xi8> to vector<4x4xi32>
+  %res = vector.contract {indexing_maps = [affine_map<(d0, d1, d2) -> (d0, d2)>,
+                                           affine_map<(d0, d1, d2) -> (d2, d1)>,
+                                           affine_map<(d0, d1, d2) -> (d0, d1)>],
+                          iterator_types = ["parallel", "parallel", "reduction"],
+                          kind = #vector.kind<add>} %lhs, %rhs, %arg2 : vector<4x4xi32>, vector<4x4xi32> into vector<4x4xi32>
+  return %res : vector<4x4xi32>
+}
+
+// Check that non-square shapes are also handled.
+// CHECK-LABEL: func.func @matmul_mk_kn_mn_4x16xi32
+// CHECK-SAME:    ([[ARG0:%.+]]: vector<4x16xi32>, [[ARG1:%.+]]: vector<16x4xi32>, [[ARG2:%.+]]: vector<4x4xi32>)
+// CHECK-NEXT:    [[TRANS:%.+]] = vector.transpose [[ARG1]], [1, 0] : vector<16x4xi32> to vector<4x16xi32>
+// CHECK-NEXT:    [[RES:%.+]]   = vector.contract {{.+}} [[ARG0]], [[TRANS]], [[ARG2]]
+// CHECK-NEXT:    return [[RES]]
+func.func @matmul_mk_kn_mn_4x16xi32(%arg0: vector<4x16xi32>, %arg1: vector<16x4xi32>, %arg2: vector<4x4xi32>) -> vector<4x4xi32> {
+  %res = vector.contract {indexing_maps = [affine_map<(d0, d1, d2) -> (d0, d2)>,
+                                           affine_map<(d0, d1, d2) -> (d2, d1)>,
+                                           affine_map<(d0, d1, d2) -> (d0, d1)>],
+                          iterator_types = ["parallel", "parallel", "reduction"],
+                          kind = #vector.kind<add>} %arg0, %arg1, %arg2 : vector<4x16xi32>, vector<16x4xi32> into vector<4x4xi32>
+  return %res : vector<4x4xi32>
+}
+
+// CHECK-LABEL: func.func @matmul_mk_kn_mn_4x4xi8_extui_i32
+// CHECK-SAME:    ([[ARG0:%.+]]: vector<4x4xi8>, [[ARG1:%.+]]: vector<4x4xi8>, [[ARG2:%.+]]: vector<4x4xi32>)
+// CHECK-NEXT:    [[LHS:%.+]]   = arith.extui [[ARG0]] : vector<4x4xi8> to vector<4x4xi32>
+// CHECK-NEXT:    [[TRANS:%.+]] = vector.transpose [[ARG1]], [1, 0] : vector<4x4xi8> to vector<4x4xi8>
+// CHECK-NEXT:    [[RHS:%.+]]   = arith.extui [[TRANS]] : vector<4x4xi8> to vector<4x4xi32>
+// CHECK-NEXT:    [[RES:%.+]]   = vector.contract {{.+}} [[LHS]], [[RHS]], [[ARG2]]
+// CHECK-NEXT:    return [[RES]]
+func.func @matmul_mk_kn_mn_4x4xi8_extui_i32(%arg0: vector<4x4xi8>, %arg1: vector<4x4xi8>, %arg2: vector<4x4xi32>) -> vector<4x4xi32> {
+  %lhs = arith.extui %arg0: vector<4x4xi8> to vector<4x4xi32>
+  %rhs = arith.extui %arg1: vector<4x4xi8> to vector<4x4xi32>
+  %res = vector.contract {indexing_maps = [affine_map<(d0, d1, d2) -> (d0, d2)>,
+                                           affine_map<(d0, d1, d2) -> (d2, d1)>,
+                                           affine_map<(d0, d1, d2) -> (d0, d1)>],
+                          iterator_types = ["parallel", "parallel", "reduction"],
+                          kind = #vector.kind<add>} %lhs, %rhs, %arg2 : vector<4x4xi32>, vector<4x4xi32> into vector<4x4xi32>
+  return %res : vector<4x4xi32>
+}
+
+// CHECK-LABEL: func.func @matmul_km_nk_mn_4x4xi32
+// CHECK-SAME:    ([[ARG0:%.+]]: vector<4x4xi32>, [[ARG1:%.+]]: vector<4x4xi32>, [[ARG2:%.+]]: vector<4x4xi32>)
+// CHECK-NEXT:    [[TRANS:%.+]] = vector.transpose [[ARG0]], [1, 0] : vector<4x4xi32> to vector<4x4xi32>
+// CHECK-NEXT:    [[RES:%.+]]   = vector.contract {{.+}} [[TRANS]], [[ARG1]], [[ARG2]]
+// CHECK-NEXT:    return [[RES]]
+func.func @matmul_km_nk_mn_4x4xi32(%arg0: vector<4x4xi32>, %arg1: vector<4x4xi32>, %arg2: vector<4x4xi32>) -> vector<4x4xi32> {
+  %res = vector.contract {indexing_maps = [affine_map<(d0, d1, d2) -> (d2, d0)>,
+                                           affine_map<(d0, d1, d2) -> (d1, d2)>,
+                                           affine_map<(d0, d1, d2) -> (d0, d1)>],
+                          iterator_types = ["parallel", "parallel", "reduction"],
+                          kind = #vector.kind<add>} %arg0, %arg1, %arg2 : vector<4x4xi32>, vector<4x4xi32> into vector<4x4xi32>
+  return %res : vector<4x4xi32>
+}
+
+// CHECK-LABEL: func.func @matmul_km_kn_mn_4x4xi32
+// CHECK-SAME:    ([[ARG0:%.+]]: vector<4x4xi32>, [[ARG1:%.+]]: vector<4x4xi32>, [[ARG2:%.+]]: vector<4x4xi32>)
+// CHECK-DAG:     [[LHS:%.+]] = vector.transpose [[ARG0]], [1, 0] : vector<4x4xi32> to vector<4x4xi32>
+// CHECK-DAG:     [[RHS:%.+]] = vector.transpose [[ARG1]], [1, 0] : vector<4x4xi32> to vector<4x4xi32>
+// CHECK-NEXT:    [[RES:%.+]] = vector.contract {{.+}} [[LHS]], [[RHS]], [[ARG2]]
+// CHECK-NEXT:    return [[RES]]
+func.func @matmul_km_kn_mn_4x4xi32(%arg0: vector<4x4xi32>, %arg1: vector<4x4xi32>, %arg2: vector<4x4xi32>) -> vector<4x4xi32> {
+  %res = vector.contract {indexing_maps = [affine_map<(d0, d1, d2) -> (d2, d0)>,
+                                           affine_map<(d0, d1, d2) -> (d2, d1)>,
+                                           affine_map<(d0, d1, d2) -> (d0, d1)>],
+                          iterator_types = ["parallel", "parallel", "reduction"],
+                          kind = #vector.kind<add>} %arg0, %arg1, %arg2 : vector<4x4xi32>, vector<4x4xi32> into vector<4x4xi32>
+  return %res : vector<4x4xi32>
+}
+
+// CHECK-LABEL: func.func @matmul_km_kn_mn_4x4xi8_mixed_ext_i32
+// CHECK-SAME:    ([[ARG0:%.+]]: vector<4x4xi8>, [[ARG1:%.+]]: vector<4x4xi8>, [[ARG2:%.+]]: vector<4x4xi32>)
+// CHECK-DAG:     [[LHST:%.+]] = vector.transpose [[ARG0]], [1, 0] : vector<4x4xi8> to vector<4x4xi8>
+// CHECK-DAG:     [[LHS:%.+]]  = arith.extsi [[LHST]] : vector<4x4xi8> to vector<4x4xi32>
+// CHECK-DAG:     [[RHST:%.+]] = vector.transpose [[ARG1]], [1, 0] : vector<4x4xi8> to vector<4x4xi8>
+// CHECK-DAG:     [[RHS:%.+]]  = arith.extui [[RHST]] : vector<4x4xi8> to vector<4x4xi32>
+// CHECK-NEXT:    [[RES:%.+]]  = vector.contract {{.+}} [[LHS]], [[RHS]], [[ARG2]]
+// CHECK-NEXT:    return [[RES]]
+func.func @matmul_km_kn_mn_4x4xi8_mixed_ext_i32(%arg0: vector<4x4xi8>, %arg1: vector<4x4xi8>, %arg2: vector<4x4xi32>) -> vector<4x4xi32> {
+  %lhs = arith.extsi %arg0 : vector<4x4xi8> to vector<4x4xi32>
+  %rhs = arith.extui %arg1 : vector<4x4xi8> to vector<4x4xi32>
+  %res = vector.contract {indexing_maps = [affine_map<(d0, d1, d2) -> (d2, d0)>,
+                                           affine_map<(d0, d1, d2) -> (d2, d1)>,
+                                           affine_map<(d0, d1, d2) -> (d0, d1)>],
+                          iterator_types = ["parallel", "parallel", "reduction"],
+                          kind = #vector.kind<add>} %lhs, %rhs, %arg2 : vector<4x4xi32>, vector<4x4xi32> into vector<4x4xi32>
+  return %res : vector<4x4xi32>
+}
+
+// CHECK-LABEL: func.func @matmul_mk_nk_nm_4x4xi32
+// CHECK-SAME:    ([[ARG0:%.+]]: vector<4x4xi32>, [[ARG1:%.+]]: vector<4x4xi32>, [[ARG2:%.+]]: vector<4x4xi32>)
+// CHECK-NEXT:    [[RES:%.+]]   = vector.contract {{.+}} [[ARG1]], [[ARG0]], [[ARG2]]
+// CHECK-NEXT:    return [[RES]]
+func.func @matmul_mk_nk_nm_4x4xi32(%arg0: vector<4x4xi32>, %arg1: vector<4x4xi32>, %arg2: vector<4x4xi32>) -> vector<4x4xi32> {
+  %res = vector.contract {indexing_maps = [affine_map<(d0, d1, d2) -> (d0, d2)>,
+                                           affine_map<(d0, d1, d2) -> (d1, d2)>,
+                                           affine_map<(d0, d1, d2) -> (d1, d0)>],
+                          iterator_types = ["parallel", "parallel", "reduction"],
+                          kind = #vector.kind<add>} %arg0, %arg1, %arg2 : vector<4x4xi32>, vector<4x4xi32> into vector<4x4xi32>
+  return %res : vector<4x4xi32>
+}
+
+// CHECK-LABEL: func.func @matmul_km_kn_nm_4x4xi32
+// CHECK-SAME:    ([[ARG0:%.+]]: vector<4x4xi32>, [[ARG1:%.+]]: vector<4x4xi32>, [[ARG2:%.+]]: vector<4x4xi32>)
+// CHECK-DAG:     [[LHS:%.+]] = vector.transpose [[ARG0]], [1, 0] : vector<4x4xi32> to vector<4x4xi32>
+// CHECK-DAG:     [[RHS:%.+]] = vector.transpose [[ARG1]], [1, 0] : vector<4x4xi32> to vector<4x4xi32>
+// CHECK-NEXT:    [[RES:%.+]] = vector.contract {{.+}} [[RHS]], [[LHS]], [[ARG2]]
+// CHECK-NEXT:    return [[RES]]
+func.func @matmul_km_kn_nm_4x4xi32(%arg0: vector<4x4xi32>, %arg1: vector<4x4xi32>, %arg2: vector<4x4xi32>) -> vector<4x4xi32> {
+  %res = vector.contract {indexing_maps = [affine_map<(d0, d1, d2) -> (d2, d0)>,
+                                           affine_map<(d0, d1, d2) -> (d2, d1)>,
+                                           affine_map<(d0, d1, d2) -> (d1, d0)>],
+                          iterator_types = ["parallel", "parallel", "reduction"],
+                          kind = #vector.kind<add>} %arg0, %arg1, %arg2 : vector<4x4xi32>, vector<4x4xi32> into vector<4x4xi32>
+  return %res : vector<4x4xi32>
+}
+
+// CHECK-LABEL: func.func @matmul_mk_kn_nm_4x4xi32
+// CHECK-SAME:    ([[ARG0:%.+]]: vector<4x4xi32>, [[ARG1:%.+]]: vector<4x4xi32>, [[ARG2:%.+]]: vector<4x4xi32>)
+// CHECK-DAG:     [[RHS:%.+]] = vector.transpose [[ARG1]], [1, 0] : vector<4x4xi32> to vector<4x4xi32>
+// CHECK-NEXT:    [[RES:%.+]] = vector.contract {{.+}} [[RHS]], [[ARG0]], [[ARG2]]
+// CHECK-NEXT:    return [[RES]]
+func.func @matmul_mk_kn_nm_4x4xi32(%arg0: vector<4x4xi32>, %arg1: vector<4x4xi32>, %arg2: vector<4x4xi32>) -> vector<4x4xi32> {
+  %res = vector.contract {indexing_maps = [affine_map<(d0, d1, d2) -> (d0, d2)>,
+                                           affine_map<(d0, d1, d2) -> (d2, d1)>,
+                                           affine_map<(d0, d1, d2) -> (d1, d0)>],
+                          iterator_types = ["parallel", "parallel", "reduction"],
+                          kind = #vector.kind<add>} %arg0, %arg1, %arg2 : vector<4x4xi32>, vector<4x4xi32> into vector<4x4xi32>
+  return %res : vector<4x4xi32>
+}
+
+// CHECK-LABEL: func.func @matmul_km_nk_nm_4x4xi32
+// CHECK-SAME:    ([[ARG0:%.+]]: vector<4x4xi32>, [[ARG1:%.+]]: vector<4x4xi32>, [[ARG2:%.+]]: vector<4x4xi32>)
+// CHECK-DAG:     [[LHS:%.+]] = vector.transpose [[ARG0]], [1, 0] : vector<4x4xi32> to vector<4x4xi32>
+// CHECK-NEXT:    [[RES:%.+]] = vector.contract {{.+}} [[ARG1]], [[LHS]], [[ARG2]]
+// CHECK-NEXT:    return [[RES]]
+func.func @matmul_km_nk_nm_4x4xi32(%arg0: vector<4x4xi32>, %arg1: vector<4x4xi32>, %arg2: vector<4x4xi32>) -> vector<4x4xi32> {
+  %res = vector.contract {indexing_maps = [affine_map<(d0, d1, d2) -> (d2, d0)>,
+                                           affine_map<(d0, d1, d2) -> (d1, d2)>,
+                                           affine_map<(d0, d1, d2) -> (d1, d0)>],
+                          iterator_types = ["parallel", "parallel", "reduction"],
+                          kind = #vector.kind<add>} %arg0, %arg1, %arg2 : vector<4x4xi32>, vector<4x4xi32> into vector<4x4xi32>
+  return %res : vector<4x4xi32>
+}
diff --git a/mlir/test/Dialect/Vector/vector-gather-lowering.mlir b/mlir/test/Dialect/Vector/vector-gather-lowering.mlir
new file mode 100644
index 0000000000000..c98a71df26c2f
--- /dev/null
+++ b/mlir/test/Dialect/Vector/vector-gather-lowering.mlir
@@ -0,0 +1,153 @@
+// RUN: mlir-opt %s --test-vector-gather-lowering | FileCheck %s
+// RUN: mlir-opt %s --test-vector-gather-lowering --canonicalize | FileCheck %s --check-prefix=CANON
+
+// CHECK-LABEL: @gather_memref_1d
+// CHECK-SAME:    ([[BASE:%.+]]: memref<?xf32>, [[IDXVEC:%.+]]: vector<2xindex>, [[MASK:%.+]]: vector<2xi1>, [[PASS:%.+]]: vector<2xf32>)
+// CHECK-DAG:     [[M0:%.+]]    = vector.extract [[MASK]][0] : vector<2xi1>
+// CHECK-DAG:     %[[IDX0:.+]]  = vector.extract [[IDXVEC]][0] : vector<2xindex>
+// CHECK-NEXT:    [[RES0:%.+]]  = scf.if [[M0]] -> (vector<2xf32>)
+// CHECK-NEXT:      [[LD0:%.+]]   = vector.load [[BASE]][%[[IDX0]]] : memref<?xf32>, vector<1xf32>
+// CHECK-NEXT:      [[ELEM0:%.+]] = vector.extract [[LD0]][0] : vector<1xf32>
+// CHECK-NEXT:      [[INS0:%.+]]  = vector.insert [[ELEM0]], [[PASS]] [0] : f32 into vector<2xf32>
+// CHECK-NEXT:      scf.yield [[INS0]] : vector<2xf32>
+// CHECK-NEXT:    else
+// CHECK-NEXT:      scf.yield [[PASS]] : vector<2xf32>
+// CHECK-DAG:     [[M1:%.+]]    = vector.extract [[MASK]][1] : vector<2xi1>
+// CHECK-DAG:     %[[IDX1:.+]]  = vector.extract [[IDXVEC]][1] : vector<2xindex>
+// CHECK-NEXT:    [[RES1:%.+]]  = scf.if [[M1]] -> (vector<2xf32>)
+// CHECK-NEXT:      [[LD1:%.+]]   = vector.load [[BASE]][%[[IDX1]]] : memref<?xf32>, vector<1xf32>
+// CHECK-NEXT:      [[ELEM1:%.+]] = vector.extract [[LD1]][0] : vector<1xf32>
+// CHECK-NEXT:      [[INS1:%.+]]  = vector.insert [[ELEM1]], [[RES0]] [1] : f32 into vector<2xf32>
+// CHECK-NEXT:      scf.yield [[INS1]] : vector<2xf32>
+// CHECK-NEXT:    else
+// CHECK-NEXT:      scf.yield [[RES0]] : vector<2xf32>
+// CHECK:         return [[RES1]] : vector<2xf32>
+func.func @gather_memref_1d(%base: memref<?xf32>, %v: vector<2xindex>, %mask: vector<2xi1>, %pass_thru: vector<2xf32>) -> vector<2xf32> {
+  %c0 = arith.constant 0 : index
+  %0 = vector.gather %base[%c0][%v], %mask, %pass_thru : memref<?xf32>, vector<2xindex>, vector<2xi1>, vector<2xf32> into vector<2xf32>
+  return %0 : vector<2xf32>
+}
+
+// CHECK-LABEL: @gather_memref_1d_i32_index
+// CHECK-SAME:    ([[BASE:%.+]]: memref<?xf32>, [[IDXVEC:%.+]]: vector<2xi32>, [[MASK:%.+]]: vector<2xi1>, [[PASS:%.+]]: vector<2xf32>)
+// CHECK-DAG:     [[C42:%.+]]   = arith.constant 42 : index
+// CHECK-DAG:     [[IDXS:%.+]]  = arith.index_cast [[IDXVEC]] : vector<2xi32> to vector<2xindex>
+// CHECK-DAG:     [[IDX0:%.+]]  = vector.extract [[IDXS]][0] : vector<2xindex>
+// CHECK-NEXT:    %[[OFF0:.+]]  = arith.addi [[IDX0]], [[C42]] : index
+// CHECK-NEXT:    [[RES0:%.+]]  = scf.if
+// CHECK-NEXT:      [[LD0:%.+]]   = vector.load [[BASE]][%[[OFF0]]] : memref<?xf32>, vector<1xf32>
+// CHECK:         else
+// CHECK:         [[IDX1:%.+]]  = vector.extract [[IDXS]][1] : vector<2xindex>
+// CHECK:         %[[OFF1:.+]]  = arith.addi [[IDX1]], [[C42]] : index
+// CHECK:         [[RES1:%.+]]  = scf.if
+// CHECK-NEXT:      [[LD1:%.+]]   = vector.load [[BASE]][%[[OFF1]]] : memref<?xf32>, vector<1xf32>
+// CHECK:         else
+// CHECK:         return [[RES1]] : vector<2xf32>
+func.func @gather_memref_1d_i32_index(%base: memref<?xf32>, %v: vector<2xi32>, %mask: vector<2xi1>, %pass_thru: vector<2xf32>) -> vector<2xf32> {
+  %c0 = arith.constant 42 : index
+  %0 = vector.gather %base[%c0][%v], %mask, %pass_thru : memref<?xf32>, vector<2xi32>, vector<2xi1>, vector<2xf32> into vector<2xf32>
+  return %0 : vector<2xf32>
+}
+
+// CHECK-LABEL: @gather_memref_2d
+// CHECK-SAME:    ([[BASE:%.+]]: memref<?x?xf32>, [[IDXVEC:%.+]]: vector<2x3xindex>, [[MASK:%.+]]: vector<2x3xi1>, [[PASS:%.+]]: vector<2x3xf32>)
+// CHECK-DAG:     %[[C0:.+]]    = arith.constant 0 : index
+// CHECK-DAG:     %[[C1:.+]]    = arith.constant 1 : index
+// CHECK-DAG:     [[PTV0:%.+]]  = vector.extract [[PASS]][0] : vector<2x3xf32>
+// CHECK-DAG:     [[M0:%.+]]    = vector.extract [[MASK]][0, 0] : vector<2x3xi1>
+// CHECK-DAG:     [[IDX0:%.+]]  = vector.extract [[IDXVEC]][0, 0] : vector<2x3xindex>
+// CHECK-NEXT:    %[[OFF0:.+]]  = arith.addi [[IDX0]], %[[C1]] : index
+// CHECK-NEXT:    [[RES0:%.+]]  = scf.if [[M0]] -> (vector<3xf32>)
+// CHECK-NEXT:      [[LD0:%.+]]   = vector.load [[BASE]][%[[C0]], %[[OFF0]]] : memref<?x?xf32>, vector<1xf32>
+// CHECK-NEXT:      [[ELEM0:%.+]] = vector.extract [[LD0]][0] : vector<1xf32>
+// CHECK-NEXT:      [[INS0:%.+]]  = vector.insert [[ELEM0]], [[PTV0]] [0] : f32 into vector<3xf32>
+// CHECK-NEXT:      scf.yield [[INS0]] : vector<3xf32>
+// CHECK-NEXT:    else
+// CHECK-NEXT:      scf.yield [[PTV0]] : vector<3xf32>
+// CHECK-COUNT-5: scf.if
+// CHECK:         [[FINAL:%.+]] = vector.insert %{{.+}}, %{{.+}} [1] : vector<3xf32> into vector<2x3xf32>
+// CHECK-NEXT:    return [[FINAL]] : vector<2x3xf32>
+ func.func @gather_memref_2d(%base: memref<?x?xf32>, %v: vector<2x3xindex>, %mask: vector<2x3xi1>, %pass_thru: vector<2x3xf32>) -> vector<2x3xf32> {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+  %0 = vector.gather %base[%c0, %c1][%v], %mask, %pass_thru : memref<?x?xf32>, vector<2x3xindex>, vector<2x3xi1>, vector<2x3xf32> into vector<2x3xf32>
+  return %0 : vector<2x3xf32>
+ }
+
+// CHECK-LABEL: @gather_tensor_1d
+// CHECK-SAME:    ([[BASE:%.+]]: tensor<?xf32>, [[IDXVEC:%.+]]: vector<2xindex>, [[MASK:%.+]]: vector<2xi1>, [[PASS:%.+]]: vector<2xf32>)
+// CHECK-DAG:     [[M0:%.+]]    = vector.extract [[MASK]][0] : vector<2xi1>
+// CHECK-DAG:     %[[IDX0:.+]]  = vector.extract [[IDXVEC]][0] : vector<2xindex>
+// CHECK-NEXT:    [[RES0:%.+]]  = scf.if [[M0]] -> (vector<2xf32>)
+// CHECK-NEXT:      [[ELEM0:%.+]] = tensor.extract [[BASE]][%[[IDX0]]] : tensor<?xf32>
+// CHECK-NEXT:      [[INS0:%.+]]  = vector.insert [[ELEM0]], [[PASS]] [0] : f32 into vector<2xf32>
+// CHECK-NEXT:      scf.yield [[INS0]] : vector<2xf32>
+// CHECK-NEXT:    else
+// CHECK-NEXT:      scf.yield [[PASS]] : vector<2xf32>
+// CHECK-DAG:     [[M1:%.+]]    = vector.extract [[MASK]][1] : vector<2xi1>
+// CHECK-DAG:     %[[IDX1:.+]]  = vector.extract [[IDXVEC]][1] : vector<2xindex>
+// CHECK-NEXT:    [[RES1:%.+]]  = scf.if [[M1]] -> (vector<2xf32>)
+// CHECK-NEXT:      [[ELEM1:%.+]] = tensor.extract [[BASE]][%[[IDX1]]] : tensor<?xf32>
+// CHECK-NEXT:      [[INS1:%.+]]  = vector.insert [[ELEM1]], [[RES0]] [1] : f32 into vector<2xf32>
+// CHECK-NEXT:      scf.yield [[INS1]] : vector<2xf32>
+// CHECK-NEXT:    else
+// CHECK-NEXT:      scf.yield [[RES0]] : vector<2xf32>
+// CHECK:         return [[RES1]] : vector<2xf32>
+func.func @gather_tensor_1d(%base: tensor<?xf32>, %v: vector<2xindex>, %mask: vector<2xi1>, %pass_thru: vector<2xf32>) -> vector<2xf32> {
+  %c0 = arith.constant 0 : index
+  %0 = vector.gather %base[%c0][%v], %mask, %pass_thru : tensor<?xf32>, vector<2xindex>, vector<2xi1>, vector<2xf32> into vector<2xf32>
+  return %0 : vector<2xf32>
+}
+
+// CHECK-LABEL: @gather_tensor_2d
+// CHECK:  scf.if
+// CHECK:    tensor.extract
+// CHECK:  else
+// CHECK:  scf.if
+// CHECK:    tensor.extract
+// CHECK:  else
+// CHECK:  scf.if
+// CHECK:    tensor.extract
+// CHECK:  else
+// CHECK:  scf.if
+// CHECK:    tensor.extract
+// CHECK:  else
+// CHECK:  scf.if
+// CHECK:    tensor.extract
+// CHECK:  else
+// CHECK:  scf.if
+// CHECK:    tensor.extract
+// CHECK:  else
+// CHECK:       [[FINAL:%.+]] = vector.insert %{{.+}}, %{{.+}} [1] : vector<3xf32> into vector<2x3xf32>
+// CHECK-NEXT:  return [[FINAL]] : vector<2x3xf32>
+ func.func @gather_tensor_2d(%base: tensor<?x?xf32>, %v: vector<2x3xindex>, %mask: vector<2x3xi1>, %pass_thru: vector<2x3xf32>) -> vector<2x3xf32> {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+  %0 = vector.gather %base[%c0, %c1][%v], %mask, %pass_thru : tensor<?x?xf32>, vector<2x3xindex>, vector<2x3xi1>, vector<2x3xf32> into vector<2x3xf32>
+  return %0 : vector<2x3xf32>
+ }
+
+// Check that all-set and no-set maskes get optimized out after canonicalization.
+
+// CANON-LABEL: @gather_tensor_1d_all_set
+// CANON-NOT:     scf.if
+// CANON:         tensor.extract
+// CANON:         tensor.extract
+// CANON:         [[FINAL:%.+]] = vector.insert %{{.+}}, %{{.+}} [1] : f32 into vector<2xf32>
+// CANON-NEXT:    return [[FINAL]] : vector<2xf32>
+func.func @gather_tensor_1d_all_set(%base: tensor<?xf32>, %v: vector<2xindex>, %pass_thru: vector<2xf32>) -> vector<2xf32> {
+  %mask = arith.constant dense <true> : vector<2xi1>
+  %c0 = arith.constant 0 : index
+  %0 = vector.gather %base[%c0][%v], %mask, %pass_thru : tensor<?xf32>, vector<2xindex>, vector<2xi1>, vector<2xf32> into vector<2xf32>
+  return %0 : vector<2xf32>
+}
+
+// CANON-LABEL: @gather_tensor_1d_none_set
+// CANON-SAME:    ([[BASE:%.+]]: tensor<?xf32>, [[IDXVEC:%.+]]: vector<2xindex>, [[PASS:%.+]]: vector<2xf32>)
+// CANON-NEXT:    return [[PASS]] : vector<2xf32>
+func.func @gather_tensor_1d_none_set(%base: tensor<?xf32>, %v: vector<2xindex>, %pass_thru: vector<2xf32>) -> vector<2xf32> {
+  %mask = arith.constant dense <false> : vector<2xi1>
+  %c0 = arith.constant 0 : index
+  %0 = vector.gather %base[%c0][%v], %mask, %pass_thru : tensor<?xf32>, vector<2xindex>, vector<2xi1>, vector<2xf32> into vector<2xf32>
+  return %0 : vector<2xf32>
+}
diff --git a/mlir/test/IR/pretty_printed_region_op.mlir b/mlir/test/IR/pretty_printed_region_op.mlir
index f009231735a57..0a81f9d53a309 100644
--- a/mlir/test/IR/pretty_printed_region_op.mlir
+++ b/mlir/test/IR/pretty_printed_region_op.mlir
@@ -5,14 +5,14 @@
 // -----
 
 func.func @pretty_printed_region_op(%arg0 : f32, %arg1 : f32) -> (f32) {
-// CHECK-CUSTOM:  test.pretty_printed_region %arg1, %arg0 start special.op end : (f32, f32) -> f32
+// CHECK-CUSTOM:  test.pretty_printed_region %arg1, %arg0 start test.special.op end : (f32, f32) -> f32
 // CHECK-GENERIC: "test.pretty_printed_region"(%arg1, %arg0)
 // CHECK-GENERIC:   ^bb0(%arg[[x:[0-9]+]]: f32, %arg[[y:[0-9]+]]: f32
-// CHECK-GENERIC:     %[[RES:.*]] = "special.op"(%arg[[x]], %arg[[y]]) : (f32, f32) -> f32
+// CHECK-GENERIC:     %[[RES:.*]] = "test.special.op"(%arg[[x]], %arg[[y]]) : (f32, f32) -> f32
 // CHECK-GENERIC:     "test.return"(%[[RES]]) : (f32) -> ()
 // CHECK-GENERIC:  : (f32, f32) -> f32
 
-  %res = test.pretty_printed_region %arg1, %arg0 start special.op end : (f32, f32) -> (f32) loc("some_NameLoc")
+  %res = test.pretty_printed_region %arg1, %arg0 start test.special.op end : (f32, f32) -> (f32) loc("some_NameLoc")
   return %res : f32
 }
 
@@ -22,13 +22,13 @@ func.func @pretty_printed_region_op(%arg0 : f32, %arg1 : f32) -> (f32) {
 // CHECK-CUSTOM:  test.pretty_printed_region %arg1, %arg0
 // CHECK-GENERIC: "test.pretty_printed_region"(%arg1, %arg0)
 // CHECK:          ^bb0(%arg[[x:[0-9]+]]: f32, %arg[[y:[0-9]+]]: f32):
-// CHECK:            %[[RES:.*]] = "non.special.op"(%arg[[x]], %arg[[y]]) : (f32, f32) -> f32
+// CHECK:            %[[RES:.*]] = "test.non.special.op"(%arg[[x]], %arg[[y]]) : (f32, f32) -> f32
 // CHECK:            "test.return"(%[[RES]]) : (f32) -> ()
 // CHECK:          : (f32, f32) -> f32
 
   %0 = test.pretty_printed_region %arg1, %arg0 ({
     ^bb0(%arg2: f32, %arg3: f32):
-      %1 = "non.special.op"(%arg2, %arg3) : (f32, f32) -> f32
+      %1 = "test.non.special.op"(%arg2, %arg3) : (f32, f32) -> f32
       "test.return"(%1) : (f32) -> ()
     }) : (f32, f32) -> f32
     return %0 : f32
@@ -39,11 +39,11 @@ func.func @pretty_printed_region_op(%arg0 : f32, %arg1 : f32) -> (f32) {
 func.func @pretty_printed_region_op_deferred_loc(%arg0 : f32, %arg1 : f32) -> (f32) {
 // CHECK-LOCATION: "test.pretty_printed_region"(%arg1, %arg0)
 // CHECK-LOCATION:   ^bb0(%arg[[x:[0-9]+]]: f32 loc("foo"), %arg[[y:[0-9]+]]: f32 loc("foo")
-// CHECK-LOCATION:     %[[RES:.*]] = "special.op"(%arg[[x]], %arg[[y]]) : (f32, f32) -> f32
+// CHECK-LOCATION:     %[[RES:.*]] = "test.special.op"(%arg[[x]], %arg[[y]]) : (f32, f32) -> f32
 // CHECK-LOCATION:     "test.return"(%[[RES]]) : (f32) -> ()
 // CHECK-LOCATION:  : (f32, f32) -> f32
 
-  %res = test.pretty_printed_region %arg1, %arg0 start special.op end : (f32, f32) -> (f32) loc("foo")
+  %res = test.pretty_printed_region %arg1, %arg0 start test.special.op end : (f32, f32) -> (f32) loc("foo")
   return %res : f32
 }
 
diff --git a/mlir/test/IR/visitors.mlir b/mlir/test/IR/visitors.mlir
index 0c9fabeee5377..ddbc334fa4eed 100644
--- a/mlir/test/IR/visitors.mlir
+++ b/mlir/test/IR/visitors.mlir
@@ -250,3 +250,85 @@ func.func @unstructured_cfg() {
 // CHECK:       Erasing block ^bb0 from region 0 from operation 'regionOp0'
 // CHECK:       Erasing block ^bb0 from region 0 from operation 'func.func'
 // CHECK:       Erasing block ^bb0 from region 0 from operation 'builtin.module'
+
+// -----
+
+func.func @unordered_cfg_with_loop() {
+  "regionOp0"() ({
+    ^bb0:
+      %c = "op0"() : () -> (i1)
+      cf.cond_br %c, ^bb2, ^bb3
+    ^bb1:
+      "op1"(%val) : (i32) -> ()
+      cf.br ^bb5
+    ^bb2:
+      %val = "op2"() : () -> (i32)
+      cf.br ^bb1
+    ^bb3:
+      "op3"() : () -> ()
+      cf.br ^bb2
+    ^bb4:
+      "op4"() : () -> ()
+      cf.br ^bb2
+    ^bb5:
+      "op5"() : () -> ()
+      cf.br ^bb7
+    ^bb6:
+      "op6"() : () -> ()
+      cf.br ^bb6
+    ^bb7:
+      "op7"() : () -> ()
+  }) : () -> ()
+  return
+}
+
+//      4
+//      |
+//      v
+// 0 -> 2 --> 1 --> 5 --> 7
+// |    ^
+// |    |      6 --
+// |   /       ^   \
+// |  /         \  /
+// v /           --
+//  3
+
+// CHECK-LABEL: Op forward dominance post-order visits
+// CHECK:       Visiting op 'op0'
+// CHECK:       Visiting op 'cf.cond_br'
+// CHECK:       Visiting op 'op2'
+// CHECK:       Visiting op 'cf.br'
+// CHECK:       Visiting op 'op1'
+// CHECK:       Visiting op 'cf.br'
+// CHECK:       Visiting op 'op5'
+// CHECK:       Visiting op 'cf.br'
+// CHECK:       Visiting op 'op7'
+// CHECK:       Visiting op 'op3'
+// CHECK:       Visiting op 'cf.br'
+// CHECK-NOT:   Visiting op 'op6'
+// CHECK:       Visiting op 'regionOp0'
+// CHECK:       Visiting op 'func.return'
+// CHECK:       Visiting op 'func.func'
+
+// CHECK-LABEL: Block forward dominance post-order visits
+// CHECK:       Visiting block ^bb0 from region 0 from operation 'regionOp0'
+// CHECK:       Visiting block ^bb2 from region 0 from operation 'regionOp0'
+// CHECK:       Visiting block ^bb1 from region 0 from operation 'regionOp0'
+// CHECK:       Visiting block ^bb5 from region 0 from operation 'regionOp0'
+// CHECK:       Visiting block ^bb7 from region 0 from operation 'regionOp0'
+// CHECK:       Visiting block ^bb3 from region 0 from operation 'regionOp0'
+// CHECK:       Visiting block ^bb0 from region 0 from operation 'func.func'
+
+// CHECK-LABEL: Region forward dominance post-order visits
+// CHECK:       Visiting region 0 from operation 'regionOp0'
+// CHECK:       Visiting region 0 from operation 'func.func'
+
+// CHECK-LABEL: Block pre-order erasures (skip)
+// CHECK:       Erasing block ^bb0 from region 0 from operation 'regionOp0'
+// CHECK:       Cannot erase block ^bb0 from region 0 from operation 'regionOp0', still has uses
+// CHECK:       Cannot erase block ^bb1 from region 0 from operation 'regionOp0', still has uses
+// CHECK:       Erasing block ^bb2 from region 0 from operation 'regionOp0'
+// CHECK:       Erasing block ^bb2 from region 0 from operation 'regionOp0'
+// CHECK:       Cannot erase block ^bb2 from region 0 from operation 'regionOp0', still has uses
+// CHECK:       Cannot erase block ^bb3 from region 0 from operation 'regionOp0', still has uses
+// CHECK:       Cannot erase block ^bb4 from region 0 from operation 'regionOp0', still has uses
diff --git a/mlir/test/Integration/Dialect/LLVMIR/CPU/test-vector-reductions-fp.mlir b/mlir/test/Integration/Dialect/LLVMIR/CPU/test-vector-reductions-fp.mlir
index 1febc6c483ae4..f2e59e1d54824 100644
--- a/mlir/test/Integration/Dialect/LLVMIR/CPU/test-vector-reductions-fp.mlir
+++ b/mlir/test/Integration/Dialect/LLVMIR/CPU/test-vector-reductions-fp.mlir
@@ -23,13 +23,13 @@ module {
     %12 = llvm.mlir.constant(3 : i64) : i64
     %v = llvm.insertelement %3, %11[%12 : i64] : vector<4xf32>
 
-    %max = "llvm.intr.vector.reduce.fmax"(%v)
+    %max = llvm.intr.vector.reduce.fmax(%v)
         : (vector<4xf32>) -> f32
     llvm.call @printF32(%max) : (f32) -> ()
     llvm.call @printNewline() : () -> ()
     // CHECK: 4
 
-    %min = "llvm.intr.vector.reduce.fmin"(%v)
+    %min = llvm.intr.vector.reduce.fmin(%v)
         : (vector<4xf32>) -> f32
     llvm.call @printF32(%min) : (f32) -> ()
     llvm.call @printNewline() : () -> ()
diff --git a/mlir/test/Integration/Dialect/SparseTensor/CPU/reshape_dot.mlir b/mlir/test/Integration/Dialect/SparseTensor/CPU/reshape_dot.mlir
index d163e3e980ac8..7ae4d59c3a90c 100644
--- a/mlir/test/Integration/Dialect/SparseTensor/CPU/reshape_dot.mlir
+++ b/mlir/test/Integration/Dialect/SparseTensor/CPU/reshape_dot.mlir
@@ -16,8 +16,8 @@
 // REDEFINE: %{option} = "enable-runtime-library=false vl=2 reassociate-fp-reductions=true enable-index-optimizations=true"
 // RUN: %{compile} | %{run}
 
-#COO_2D = #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ], pointerBitWidth = 32, indexBitWidth = 32 }>
-#COO_3D = #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton-nu", "singleton" ], pointerBitWidth = 32, indexBitWidth = 32 }>
+#COO_2D = #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton" ], posWidth = 32, crdWidth = 32 }>
+#COO_3D = #sparse_tensor.encoding<{ dimLevelType = [ "compressed-nu", "singleton-nu", "singleton" ], posWidth = 32, crdWidth = 32 }>
 
 module {
   func.func private @printMemref3dF32(%ptr : tensor<?x?x?xf32>) attributes { llvm.emit_c_interface }
@@ -56,6 +56,18 @@ module {
     return %ret1 :  tensor<?x?x?xf32>
   }
 
+  func.func @test_sparse_all_2(%arg0: tensor<5x6xf32, #COO_2D>, %arg1: tensor<2x3x6xf32, #COO_3D>) -> tensor<?x?x?xf32> {
+    // collapse the first two level this time, as this is the level requires coiterations.
+    %collapsed = tensor.collapse_shape %arg1 [[0, 1], [2]] : tensor<2x3x6xf32, #COO_3D> into tensor<6x6xf32, #COO_2D>
+    %0 = tensor.empty() : tensor<5x6xf32>
+    %cst = arith.constant 0.000000e+00 : f32
+    %1 = linalg.fill ins(%cst : f32) outs(%0 : tensor<5x6xf32>) -> tensor<5x6xf32>
+    %2 = linalg.matmul ins(%arg0, %collapsed : tensor<5x6xf32, #COO_2D>, tensor<6x6xf32, #COO_2D>) outs(%1 : tensor<5x6xf32>) -> tensor<5x6xf32>
+    %expanded = tensor.expand_shape %2 [[0], [1, 2]] : tensor<5x6xf32> into tensor<5x2x3xf32>
+    %ret1 = tensor.cast %expanded : tensor<5x2x3xf32> to tensor<?x?x?xf32>
+    return %ret1 : tensor<?x?x?xf32>
+  }
+
 
   func.func @entry() {
     // Setup two sparse vectors.
@@ -68,9 +80,12 @@ module {
       [ [0, 0, 0], [1, 1, 1], [2, 1, 1] ],
         [     6.0,       7.0,      8.0]
     > : tensor<6x2x3xf32>
+    %shape = arith.constant dense<[2, 3, 6]> : tensor<3xi32>
 
+    %d3 = tensor.reshape %d2(%shape): (tensor<6x2x3xf32>, tensor<3xi32>) -> tensor<2x3x6xf32>
     %s1 = sparse_tensor.convert %d1 : tensor<5x6xf32> to tensor<5x6xf32, #COO_2D>
     %s2 = sparse_tensor.convert %d2 : tensor<6x2x3xf32> to tensor<6x2x3xf32, #COO_3D>
+    %s3 = sparse_tensor.convert %d3 : tensor<2x3x6xf32> to tensor<2x3x6xf32, #COO_3D>
 
     //      CHECK: Memref base@ = {{.*}} rank = 3 offset = 0 sizes = [5, 2, 3] strides = [6, 3, 1] data =
     // CHECK-NEXT:[
@@ -134,11 +149,34 @@ module {
     %so2 = call @test_sparse_all(%s1, %s2): (tensor<5x6xf32, #COO_2D>, tensor<6x2x3xf32, #COO_3D>) -> tensor<?x?x?xf32>
     call @printMemref3dF32(%so2) : (tensor<?x?x?xf32>) -> ()
 
+    // Same results.
+    // CHECK-NEXT: Memref base@ = {{.*}} rank = 3 offset = 0 sizes = [5, 2, 3] strides = [6, 3, 1] data =
+    // CHECK-NEXT:[
+    // CHECK-SAME: [
+    // CHECK-SAME:  [6,    0,    0],
+    // CHECK-NEXT:  [0,    0,    0]],
+    // CHECK-NEXT: [
+    // CHECK-SAME:  [0,    0,    0],
+    // CHECK-NEXT:  [0,    14,    0]],
+    // CHECK-NEXT: [
+    // CHECK-SAME:  [0,    0,    0],
+    // CHECK-NEXT:  [0,    24,    0]],
+    // CHECK-NEXT: [
+    // CHECK-SAME:  [0,    0,    0],
+    // CHECK-NEXT:  [0,    0,    0]],
+    // CHECK-NEXT: [
+    // CHECK-SAME:  [0,    0,    0],
+    // CHECK-NEXT:  [0,    0,    0]]]
+    %so3 = call @test_sparse_all_2(%s1, %s3): (tensor<5x6xf32, #COO_2D>, tensor<2x3x6xf32, #COO_3D>) -> tensor<?x?x?xf32>
+    call @printMemref3dF32(%so2) : (tensor<?x?x?xf32>) -> ()
+
     bufferization.dealloc_tensor %s1 : tensor<5x6xf32, #COO_2D>
     bufferization.dealloc_tensor %s2 : tensor<6x2x3xf32, #COO_3D>
+    bufferization.dealloc_tensor %s3 : tensor<2x3x6xf32, #COO_3D>
     bufferization.dealloc_tensor %do1 : tensor<?x?x?xf32>
     bufferization.dealloc_tensor %so1 : tensor<?x?x?xf32>
     bufferization.dealloc_tensor %so2 : tensor<?x?x?xf32>
+    bufferization.dealloc_tensor %so3 : tensor<?x?x?xf32>
     return
   }
 }
diff --git a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_constant_to_sparse_tensor.mlir b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_constant_to_sparse_tensor.mlir
index 442e9b30dfd92..4bf7735a8a989 100644
--- a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_constant_to_sparse_tensor.mlir
+++ b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_constant_to_sparse_tensor.mlir
@@ -48,12 +48,12 @@ module {
     %ts = sparse_tensor.convert %ti : tensor<10x8xf64> to tensor<10x8xf64, #Tensor1>
 
     // CHECK: ( 0, 1, 4, 5, 6, 9 )
-    %i0 = sparse_tensor.indices %ts { dimension = 0 : index } : tensor<10x8xf64, #Tensor1> to memref<?xindex>
+    %i0 = sparse_tensor.coordinates %ts { level = 0 : index } : tensor<10x8xf64, #Tensor1> to memref<?xindex>
     %i0r = vector.transfer_read %i0[%c0], %c0: memref<?xindex>, vector<6xindex>
     vector.print %i0r : vector<6xindex>
 
     // CHECK: ( 0, 7, 2, 2, 3, 4, 6, 7 )
-    %i1 = sparse_tensor.indices %ts { dimension = 1 : index } : tensor<10x8xf64, #Tensor1> to memref<?xindex>
+    %i1 = sparse_tensor.coordinates %ts { level = 1 : index } : tensor<10x8xf64, #Tensor1> to memref<?xindex>
     %i1r = vector.transfer_read %i1[%c0], %c0: memref<?xindex>, vector<8xindex>
     vector.print %i1r : vector<8xindex>
 
diff --git a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_conversion.mlir b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_conversion.mlir
index 13caf9b4ff5ff..6053bb17307fc 100644
--- a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_conversion.mlir
+++ b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_conversion.mlir
@@ -180,7 +180,7 @@ module {
     call @dumpf64(%iv) : (memref<?xf64>) -> ()
 
     //
-    // Check indices.
+    // Check coordinates.
     //
     // CHECK-NEXT: ( 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
     // CHECK-NEXT: ( 0, 1, 2, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
@@ -219,45 +219,45 @@ module {
     // CHECK-NEXT: ( 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
     // CHECK-NEXT: ( 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0 )
     //
-    %v10 = sparse_tensor.indices %1 { dimension = 0 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
-    %v11 = sparse_tensor.indices %1 { dimension = 1 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
-    %v12 = sparse_tensor.indices %1 { dimension = 2 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
-    %v20 = sparse_tensor.indices %2 { dimension = 0 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
-    %v21 = sparse_tensor.indices %2 { dimension = 1 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
-    %v22 = sparse_tensor.indices %2 { dimension = 2 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
-    %v30 = sparse_tensor.indices %3 { dimension = 0 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
-    %v31 = sparse_tensor.indices %3 { dimension = 1 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
-    %v32 = sparse_tensor.indices %3 { dimension = 2 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
+    %v10 = sparse_tensor.coordinates %1 { level = 0 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
+    %v11 = sparse_tensor.coordinates %1 { level = 1 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
+    %v12 = sparse_tensor.coordinates %1 { level = 2 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
+    %v20 = sparse_tensor.coordinates %2 { level = 0 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
+    %v21 = sparse_tensor.coordinates %2 { level = 1 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
+    %v22 = sparse_tensor.coordinates %2 { level = 2 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
+    %v30 = sparse_tensor.coordinates %3 { level = 0 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
+    %v31 = sparse_tensor.coordinates %3 { level = 1 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
+    %v32 = sparse_tensor.coordinates %3 { level = 2 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
 
-    %a10 = sparse_tensor.indices %a { dimension = 0 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
-    %a11 = sparse_tensor.indices %a { dimension = 1 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
-    %a12 = sparse_tensor.indices %a { dimension = 2 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
-    %b10 = sparse_tensor.indices %b { dimension = 0 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
-    %b11 = sparse_tensor.indices %b { dimension = 1 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
-    %b12 = sparse_tensor.indices %b { dimension = 2 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
-    %c10 = sparse_tensor.indices %c { dimension = 0 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
-    %c11 = sparse_tensor.indices %c { dimension = 1 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
-    %c12 = sparse_tensor.indices %c { dimension = 2 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
+    %a10 = sparse_tensor.coordinates %a { level = 0 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
+    %a11 = sparse_tensor.coordinates %a { level = 1 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
+    %a12 = sparse_tensor.coordinates %a { level = 2 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
+    %b10 = sparse_tensor.coordinates %b { level = 0 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
+    %b11 = sparse_tensor.coordinates %b { level = 1 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
+    %b12 = sparse_tensor.coordinates %b { level = 2 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
+    %c10 = sparse_tensor.coordinates %c { level = 0 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
+    %c11 = sparse_tensor.coordinates %c { level = 1 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
+    %c12 = sparse_tensor.coordinates %c { level = 2 : index } : tensor<2x3x4xf64, #Tensor1> to memref<?xindex>
 
-    %d20 = sparse_tensor.indices %d { dimension = 0 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
-    %d21 = sparse_tensor.indices %d { dimension = 1 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
-    %d22 = sparse_tensor.indices %d { dimension = 2 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
-    %e20 = sparse_tensor.indices %e { dimension = 0 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
-    %e21 = sparse_tensor.indices %e { dimension = 1 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
-    %e22 = sparse_tensor.indices %e { dimension = 2 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
-    %f20 = sparse_tensor.indices %f { dimension = 0 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
-    %f21 = sparse_tensor.indices %f { dimension = 1 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
-    %f22 = sparse_tensor.indices %f { dimension = 2 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
+    %d20 = sparse_tensor.coordinates %d { level = 0 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
+    %d21 = sparse_tensor.coordinates %d { level = 1 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
+    %d22 = sparse_tensor.coordinates %d { level = 2 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
+    %e20 = sparse_tensor.coordinates %e { level = 0 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
+    %e21 = sparse_tensor.coordinates %e { level = 1 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
+    %e22 = sparse_tensor.coordinates %e { level = 2 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
+    %f20 = sparse_tensor.coordinates %f { level = 0 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
+    %f21 = sparse_tensor.coordinates %f { level = 1 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
+    %f22 = sparse_tensor.coordinates %f { level = 2 : index } : tensor<2x3x4xf64, #Tensor2> to memref<?xindex>
 
-    %g30 = sparse_tensor.indices %g { dimension = 0 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
-    %g31 = sparse_tensor.indices %g { dimension = 1 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
-    %g32 = sparse_tensor.indices %g { dimension = 2 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
-    %h30 = sparse_tensor.indices %h { dimension = 0 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
-    %h31 = sparse_tensor.indices %h { dimension = 1 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
-    %h32 = sparse_tensor.indices %h { dimension = 2 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
-    %i30 = sparse_tensor.indices %i { dimension = 0 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
-    %i31 = sparse_tensor.indices %i { dimension = 1 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
-    %i32 = sparse_tensor.indices %i { dimension = 2 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
+    %g30 = sparse_tensor.coordinates %g { level = 0 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
+    %g31 = sparse_tensor.coordinates %g { level = 1 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
+    %g32 = sparse_tensor.coordinates %g { level = 2 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
+    %h30 = sparse_tensor.coordinates %h { level = 0 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
+    %h31 = sparse_tensor.coordinates %h { level = 1 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
+    %h32 = sparse_tensor.coordinates %h { level = 2 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
+    %i30 = sparse_tensor.coordinates %i { level = 0 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
+    %i31 = sparse_tensor.coordinates %i { level = 1 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
+    %i32 = sparse_tensor.coordinates %i { level = 2 : index } : tensor<2x3x4xf64, #Tensor3> to memref<?xindex>
 
     call @dumpidx(%v10) : (memref<?xindex>) -> ()
     call @dumpidx(%v11) : (memref<?xindex>) -> ()
diff --git a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_conversion_ptr.mlir b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_conversion_ptr.mlir
index e05fcd40e472f..ef0ba27a36b8e 100644
--- a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_conversion_ptr.mlir
+++ b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_conversion_ptr.mlir
@@ -28,27 +28,27 @@
 
 #DCSR  = #sparse_tensor.encoding<{
   dimLevelType = [ "compressed", "compressed" ],
-  pointerBitWidth = 8,
-  indexBitWidth = 8
+  posWidth = 8,
+  crdWidth = 8
 }>
 
 #DCSC  = #sparse_tensor.encoding<{
   dimLevelType = [ "compressed", "compressed" ],
   dimOrdering = affine_map<(i,j) -> (j,i)>,
-  pointerBitWidth = 64,
-  indexBitWidth = 64
+  posWidth = 64,
+  crdWidth = 64
 }>
 
 #CSC  = #sparse_tensor.encoding<{
   dimLevelType = [ "dense", "compressed" ],
   dimOrdering = affine_map<(i,j) -> (j,i)>,
-  pointerBitWidth = 16,
-  indexBitWidth = 32
+  posWidth = 16,
+  crdWidth = 32
 }>
 
 //
 // Integration test that tests conversions between sparse tensors,
-// where the pointer and index sizes in the overhead storage change
+// where the position and index sizes in the overhead storage change
 // in addition to layout.
 //
 module {
@@ -136,12 +136,12 @@ module {
     // CHECK-NEXT: ( 0, 1, 63, 0, 1, 0, 63, 0 )
     // CHECK-NEXT: ( 0, 1, 63, 0, 1, 0, 63, 0 )
     //
-    %i1 = sparse_tensor.indices %1 { dimension = 1 : index } : tensor<32x64xf64, #DCSR> to memref<?xi8>
-    %i2 = sparse_tensor.indices %2 { dimension = 1 : index } : tensor<32x64xf64, #DCSC> to memref<?xi64>
-    %i3 = sparse_tensor.indices %3 { dimension = 1 : index } : tensor<32x64xf64, #CSC>  to memref<?xi32>
-    %i4 = sparse_tensor.indices %4 { dimension = 1 : index } : tensor<32x64xf64, #DCSC> to memref<?xi64>
-    %i5 = sparse_tensor.indices %5 { dimension = 1 : index } : tensor<32x64xf64, #DCSR> to memref<?xi8>
-    %i6 = sparse_tensor.indices %6 { dimension = 1 : index } : tensor<32x64xf64, #DCSR> to memref<?xi8>
+    %i1 = sparse_tensor.coordinates %1 { level = 1 : index } : tensor<32x64xf64, #DCSR> to memref<?xi8>
+    %i2 = sparse_tensor.coordinates %2 { level = 1 : index } : tensor<32x64xf64, #DCSC> to memref<?xi64>
+    %i3 = sparse_tensor.coordinates %3 { level = 1 : index } : tensor<32x64xf64, #CSC>  to memref<?xi32>
+    %i4 = sparse_tensor.coordinates %4 { level = 1 : index } : tensor<32x64xf64, #DCSC> to memref<?xi64>
+    %i5 = sparse_tensor.coordinates %5 { level = 1 : index } : tensor<32x64xf64, #DCSR> to memref<?xi8>
+    %i6 = sparse_tensor.coordinates %6 { level = 1 : index } : tensor<32x64xf64, #DCSR> to memref<?xi8>
     call @dumpi08(%i1) : (memref<?xi8>)  -> ()
     call @dumpi64(%i2) : (memref<?xi64>) -> ()
     call @dumpi32(%i3) : (memref<?xi32>) -> ()
diff --git a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_coo_test.mlir b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_coo_test.mlir
new file mode 100644
index 0000000000000..8681d33ce0f80
--- /dev/null
+++ b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_coo_test.mlir
@@ -0,0 +1,184 @@
+// DEFINE: %{option} = enable-runtime-library=true
+// DEFINE: %{compile} = mlir-opt %s --sparse-compiler=%{option}
+// DEFINE: %{run} = mlir-cpu-runner \
+// DEFINE:  -e entry -entry-point-result=void  \
+// DEFINE:  -shared-libs=%mlir_c_runner_utils | \
+// DEFINE: FileCheck %s
+//
+// RUN: %{compile} | %{run}
+//
+// Do the same run, but now with direct IR generation.
+// REDEFINE: %{option} = enable-runtime-library=false
+// RUN: %{compile} | %{run}
+//
+// Do the same run, but now with direct IR generation and vectorization.
+// REDEFINE: %{option} = "enable-runtime-library=false vl=2 reassociate-fp-reductions=true enable-index-optimizations=true"
+// RUN: %{compile} | %{run}
+//
+// Do the same run, but now with direct IR generation and, if available, VLA
+// vectorization.
+// REDEFINE: %{option} = "enable-runtime-library=false vl=4 enable-arm-sve=%ENABLE_VLA"
+// REDEFINE: %{run} = %lli \
+// REDEFINE:   --entry-function=entry_lli \
+// REDEFINE:   --extra-module=%S/Inputs/main_for_lli.ll \
+// REDEFINE:   %VLA_ARCH_ATTR_OPTIONS \
+// REDEFINE:   --dlopen=%mlir_native_utils_lib_dir/libmlir_c_runner_utils%shlibext | \
+// REDEFINE: FileCheck %s
+// RUN: %{compile} | mlir-translate -mlir-to-llvmir | %{run}
+
+#SortedCOO = #sparse_tensor.encoding<{
+  dimLevelType = [ "compressed-nu", "singleton" ]
+}>
+
+#CSR = #sparse_tensor.encoding<{
+  dimLevelType = [ "dense", "compressed" ]
+}>
+
+#trait = {
+  indexing_maps = [
+    affine_map<(i,j) -> (i,j)>,  // A
+    affine_map<(i,j) -> (i,j)>,  // B
+    affine_map<(i,j) -> (i,j)>   // X (out)
+  ],
+  iterator_types = ["parallel", "parallel"],
+  doc = "X(i,j) = A(i,j) + B(i,j)"
+}
+
+module {
+  func.func @add_coo_csr(%arga: tensor<8x8xf32, #CSR>,
+                         %argb: tensor<8x8xf32, #SortedCOO>)
+		         -> tensor<8x8xf32> {
+    %empty = tensor.empty() : tensor<8x8xf32>
+    %zero = arith.constant 0.000000e+00 : f32
+    %init = linalg.fill
+        ins(%zero : f32)
+        outs(%empty : tensor<8x8xf32>) -> tensor<8x8xf32>
+    %0 = linalg.generic #trait
+      ins(%arga, %argb: tensor<8x8xf32, #CSR>,
+                        tensor<8x8xf32, #SortedCOO>)
+      outs(%init: tensor<8x8xf32>) {
+        ^bb(%a: f32, %b: f32, %x: f32):
+          %0 = arith.addf %a, %b : f32
+          linalg.yield %0 : f32
+        } -> tensor<8x8xf32>
+    return %0 : tensor<8x8xf32>
+  }
+
+  func.func @add_coo_coo(%arga: tensor<8x8xf32, #SortedCOO>,
+                         %argb: tensor<8x8xf32, #SortedCOO>)
+		         -> tensor<8x8xf32> {
+    %empty = tensor.empty() : tensor<8x8xf32>
+    %zero = arith.constant 0.000000e+00 : f32
+    %init = linalg.fill
+        ins(%zero : f32)
+        outs(%empty : tensor<8x8xf32>) -> tensor<8x8xf32>
+    %0 = linalg.generic #trait
+      ins(%arga, %argb: tensor<8x8xf32, #SortedCOO>,
+                        tensor<8x8xf32, #SortedCOO>)
+      outs(%init: tensor<8x8xf32>) {
+        ^bb(%a: f32, %b: f32, %x: f32):
+          %0 = arith.addf %a, %b : f32
+          linalg.yield %0 : f32
+        } -> tensor<8x8xf32>
+    return %0 : tensor<8x8xf32>
+  }
+
+  func.func @add_coo_dense(%arga: tensor<8x8xf32>,
+                           %argb: tensor<8x8xf32, #SortedCOO>)
+  	    	         -> tensor<8x8xf32> {
+    %empty = tensor.empty() : tensor<8x8xf32>
+    %zero = arith.constant 0.000000e+00 : f32
+    %init = linalg.fill
+        ins(%zero : f32)
+        outs(%empty : tensor<8x8xf32>) -> tensor<8x8xf32>
+    %0 = linalg.generic #trait
+      ins(%arga, %argb: tensor<8x8xf32>,
+                        tensor<8x8xf32, #SortedCOO>)
+      outs(%init: tensor<8x8xf32>) {
+        ^bb(%a: f32, %b: f32, %x: f32):
+          %0 = arith.addf %a, %b : f32
+          linalg.yield %0 : f32
+        } -> tensor<8x8xf32>
+    return %0 : tensor<8x8xf32>
+  }
+
+  func.func @entry() {
+    %c0  = arith.constant 0 : index
+    %c1  = arith.constant 1 : index
+    %c8  = arith.constant 8 : index
+
+    %A = arith.constant dense<
+        [ [ 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0 ],
+          [ 1.1, 2.1, 3.1, 4.1, 5.1, 6.1, 7.1, 8.1 ],
+          [ 2.2, 2.2, 3.2, 4.2, 5.2, 6.2, 7.2, 8.2 ],
+          [ 3.3, 2.3, 3.3, 4.3, 5.3, 6.3, 7.3, 8.3 ],
+          [ 4.4, 2.4, 3.4, 4.4, 5.4, 6.4, 7.4, 8.4 ],
+          [ 5.5, 2.5, 3.5, 4.5, 5.5, 6.5, 7.5, 8.5 ],
+          [ 6.6, 2.6, 3.6, 4.6, 5.6, 6.6, 7.6, 8.6 ],
+          [ 7.7, 2.7, 3.7, 4.7, 5.7, 6.7, 7.7, 8.7 ] ]
+    > : tensor<8x8xf32>
+    %B = arith.constant dense<
+        [ [ 7.8, 2.8, 3.8, 0.8, 3.8, 0.1, 7.8, 8.8 ],
+          [ 3.3, 2.3, 1.3, 4.3, 3.3, 6.3, 9.3, 8.3 ],
+          [ 6.6, 2.6, 3.6, 4.6, 3.6, 6.6, 7.6, 7.6 ],
+          [ 1.0, 3.0, 3.0, 4.0, 3.0, 6.0, 7.0, 8.0 ],
+          [ 0.1, 2.1, 3.1, 4.1, 3.1, 6.1, 7.1, 8.1 ],
+          [ 4.4, 2.4, 3.4, 4.4, 3.4, 6.4, 8.4, 8.4 ],
+          [ 5.5, 3.5, 1.5, 4.5, 3.5, 6.5, 7.5, 8.5 ],
+          [ 7.7, 2.7, 3.7, 0.7, 5.7, 3.7, 3.7, 0.7 ] ]
+    > : tensor<8x8xf32>
+
+    // Stress test with a "sparse" version of A and B.
+    %CSR_A = sparse_tensor.convert %A
+      : tensor<8x8xf32> to tensor<8x8xf32, #CSR>
+    %COO_A = sparse_tensor.convert %A
+      : tensor<8x8xf32> to tensor<8x8xf32, #SortedCOO>
+    %COO_B = sparse_tensor.convert %B
+      : tensor<8x8xf32> to tensor<8x8xf32, #SortedCOO>
+
+    %C1 = call @add_coo_dense(%A, %COO_B) : (tensor<8x8xf32>,
+                                             tensor<8x8xf32, #SortedCOO>)
+                                          -> tensor<8x8xf32>
+    %C2 = call @add_coo_csr(%CSR_A, %COO_B) : (tensor<8x8xf32, #CSR>,
+                                               tensor<8x8xf32, #SortedCOO>)
+                                            -> tensor<8x8xf32>
+    %C3 = call @add_coo_coo(%COO_A, %COO_B) : (tensor<8x8xf32, #SortedCOO>,
+                                               tensor<8x8xf32, #SortedCOO>)
+                                            -> tensor<8x8xf32>
+    //
+    // Verify computed matrix C.
+    //
+    // CHECK-COUNT-3:      ( 8.8, 4.8, 6.8, 4.8, 8.8, 6.1, 14.8, 16.8 )
+    // CHECK-NEXT-COUNT-3: ( 4.4, 4.4, 4.4, 8.4, 8.4, 12.4, 16.4, 16.4 )
+    // CHECK-NEXT-COUNT-3: ( 8.8, 4.8, 6.8, 8.8, 8.8, 12.8, 14.8, 15.8 )
+    // CHECK-NEXT-COUNT-3: ( 4.3, 5.3, 6.3, 8.3, 8.3, 12.3, 14.3, 16.3 )
+    // CHECK-NEXT-COUNT-3: ( 4.5, 4.5, 6.5, 8.5, 8.5, 12.5, 14.5, 16.5 )
+    // CHECK-NEXT-COUNT-3: ( 9.9, 4.9, 6.9, 8.9, 8.9, 12.9, 15.9, 16.9 )
+    // CHECK-NEXT-COUNT-3: ( 12.1, 6.1, 5.1, 9.1, 9.1, 13.1, 15.1, 17.1 )
+    // CHECK-NEXT-COUNT-3: ( 15.4, 5.4, 7.4, 5.4, 11.4, 10.4, 11.4, 9.4 )
+    //
+    %f0  = arith.constant 0.0 : f32
+    scf.for %i = %c0 to %c8 step %c1 {
+      %v1 = vector.transfer_read %C1[%i, %c0], %f0
+        : tensor<8x8xf32>, vector<8xf32>
+      %v2 = vector.transfer_read %C2[%i, %c0], %f0
+        : tensor<8x8xf32>, vector<8xf32>
+      %v3 = vector.transfer_read %C3[%i, %c0], %f0
+        : tensor<8x8xf32>, vector<8xf32>
+      vector.print %v1 : vector<8xf32>
+      vector.print %v2 : vector<8xf32>
+      vector.print %v3 : vector<8xf32>
+    }
+
+    // Release resources.
+    bufferization.dealloc_tensor %C1 : tensor<8x8xf32>
+    bufferization.dealloc_tensor %C2 : tensor<8x8xf32>
+    bufferization.dealloc_tensor %C3 : tensor<8x8xf32>
+    bufferization.dealloc_tensor %CSR_A : tensor<8x8xf32, #CSR>
+    bufferization.dealloc_tensor %COO_A : tensor<8x8xf32, #SortedCOO>
+    bufferization.dealloc_tensor %COO_B : tensor<8x8xf32, #SortedCOO>
+
+
+    return
+  }
+}
\ No newline at end of file
diff --git a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_file_io.mlir b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_file_io.mlir
index b6fea69d6d6b1..a868280abaede 100644
--- a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_file_io.mlir
+++ b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_file_io.mlir
@@ -39,7 +39,7 @@ module {
   func.func private @createSparseTensorReader(!Filename) -> (!TensorReader)
   func.func private @delSparseTensorReader(!TensorReader) -> ()
   func.func private @getSparseTensorReaderRank(!TensorReader) -> (index)
-  func.func private @getSparseTensorReaderNNZ(!TensorReader) -> (index)
+  func.func private @getSparseTensorReaderNSE(!TensorReader) -> (index)
   func.func private @getSparseTensorReaderIsSymmetric(!TensorReader) -> (i1)
   func.func private @copySparseTensorReaderDimSizes(!TensorReader,
     memref<?xindex>) -> () attributes { llvm.emit_c_interface }
@@ -87,12 +87,12 @@ module {
     %c2 = arith.constant 2 : index
 
     %rank = call @getSparseTensorReaderRank(%tensor) : (!TensorReader) -> index
-    %nnz = call @getSparseTensorReaderNNZ(%tensor) : (!TensorReader) -> index
+    %nse = call @getSparseTensorReaderNSE(%tensor) : (!TensorReader) -> index
 
     // Assume rank == 2.
-    %isize = arith.muli %c2, %nnz : index
+    %isize = arith.muli %c2, %nse : index
     %xs = memref.alloc(%isize) : memref<?xindex>
-    %vs = memref.alloc(%nnz) : memref<?xf32>
+    %vs = memref.alloc(%nse) : memref<?xf32>
     %dim2lvl = memref.alloca(%c2) : memref<?xindex>
     memref.store %c0, %dim2lvl[%c0] : memref<?xindex>
     memref.store %c1, %dim2lvl[%c1] : memref<?xindex>
@@ -110,8 +110,8 @@ module {
       : (!Filename) -> (!TensorReader)
     %rank = call @getSparseTensorReaderRank(%tensor) : (!TensorReader) -> index
     vector.print %rank : index
-    %nnz = call @getSparseTensorReaderNNZ(%tensor) : (!TensorReader) -> index
-    vector.print %nnz : index
+    %nse = call @getSparseTensorReaderNSE(%tensor) : (!TensorReader) -> index
+    vector.print %nse : index
     %symmetric = call @getSparseTensorReaderIsSymmetric(%tensor)
       : (!TensorReader) -> i1
     vector.print %symmetric : i1
@@ -122,9 +122,9 @@ module {
 
     %xs, %vs, %isSorted = call @readTensorFile(%tensor)
       : (!TensorReader) -> (memref<?xindex>, memref<?xf32>, i1)
-    %x0s = memref.subview %xs[%c0][%nnz][%c2]
+    %x0s = memref.subview %xs[%c0][%nse][%c2]
       : memref<?xindex> to memref<?xindex, strided<[2], offset: ?>>
-    %x1s = memref.subview %xs[%c1][%nnz][%c2]
+    %x1s = memref.subview %xs[%c1][%nse][%c2]
       : memref<?xindex> to memref<?xindex, strided<[2], offset: ?>>
     vector.print %isSorted : i1
     call @dumpi2(%x0s) : (memref<?xindex, strided<[2], offset: ?>>) -> ()
@@ -152,18 +152,18 @@ module {
       : (!Filename) -> (!TensorWriter)
 
     %rank = call @getSparseTensorReaderRank(%tensor0) : (!TensorReader) -> index
-    %nnz = call @getSparseTensorReaderNNZ(%tensor0) : (!TensorReader) -> index
+    %nse = call @getSparseTensorReaderNSE(%tensor0) : (!TensorReader) -> index
     %dimSizes = memref.alloc(%rank) : memref<?xindex>
     func.call @copySparseTensorReaderDimSizes(%tensor0, %dimSizes)
       : (!TensorReader, memref<?xindex>) -> ()
-    call @outSparseTensorWriterMetaData(%tensor1, %rank, %nnz, %dimSizes)
+    call @outSparseTensorWriterMetaData(%tensor1, %rank, %nse, %dimSizes)
       : (!TensorWriter, index, index, memref<?xindex>) -> ()
 
     //TODO: handle isSymmetric.
     // Assume rank == 2.
     %indices = memref.alloc(%rank) : memref<?xindex>
     %value = memref.alloca() : memref<f32>
-    scf.for %i = %c0 to %nnz step %c1 {
+    scf.for %i = %c0 to %nse step %c1 {
       func.call @getSparseTensorReaderNextF32(%tensor0, %indices, %value)
         : (!TensorReader, memref<?xindex>, memref<f32>) -> ()
       func.call @outSparseTensorWriterNextF32(%tensor1, %rank, %indices, %value)
@@ -223,4 +223,4 @@ module {
 
     return
   }
-}
\ No newline at end of file
+}
diff --git a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_foreach_slices.mlir b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_foreach_slices.mlir
index 548818ccb02db..d51374b1fe3f1 100644
--- a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_foreach_slices.mlir
+++ b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_foreach_slices.mlir
@@ -2,13 +2,14 @@
 // DEFINE: %{command} = mlir-opt %s --sparse-compiler=%{option} | \
 // DEFINE: mlir-cpu-runner \
 // DEFINE:  -e entry -entry-point-result=void  \
-// DEFINE:  -shared-libs=%mlir_c_runner_utils | \
+// DEFINE:  -shared-libs=%mlir_lib_dir/libmlir_c_runner_utils%shlibext | \
 // DEFINE: FileCheck %s
 //
 // RUN: %{command}
 //
 
 // TODO: support slices on lib path
+
 #CSR = #sparse_tensor.encoding<{
   dimLevelType = [ "dense", "compressed" ]
 }>
@@ -18,6 +19,27 @@
   slice = [ (1, 4, 1), (1, 4, 2) ]
 }>
 
+#CSR_SLICE_DYN = #sparse_tensor.encoding<{
+  dimLevelType = [ "dense", "compressed" ],
+  slice = [ (?, ?, ?), (?, ?, ?) ]
+}>
+
+#COO = #sparse_tensor.encoding<{
+  dimLevelType = [ "compressed-nu", "singleton" ]
+}>
+
+#COO_SLICE = #sparse_tensor.encoding<{
+  dimLevelType = [ "compressed-nu", "singleton" ],
+  slice = [ (1, 4, 1), (1, 4, 2) ]
+}>
+
+#COO_SLICE_DYN = #sparse_tensor.encoding<{
+  dimLevelType = [ "compressed-nu", "singleton" ],
+  slice = [ (?, ?, ?), (?, ?, ?) ]
+}>
+
+
+
 module {
   func.func @foreach_print_non_slice(%A: tensor<4x4xf64, #CSR>) {
     sparse_tensor.foreach in %A : tensor<4x4xf64, #CSR> do {
@@ -39,8 +61,42 @@ module {
     return
   }
 
+  func.func @foreach_print_slice_dyn(%A: tensor<?x?xf64, #CSR_SLICE_DYN>) {
+    sparse_tensor.foreach in %A : tensor<?x?xf64, #CSR_SLICE_DYN> do {
+    ^bb0(%1: index, %2: index, %v: f64) :
+      vector.print %1: index
+      vector.print %2: index
+      vector.print %v: f64
+    }
+    return
+  }
+
+  func.func @foreach_print_slice_coo(%A: tensor<4x4xf64, #COO_SLICE>) {
+    sparse_tensor.foreach in %A : tensor<4x4xf64, #COO_SLICE> do {
+    ^bb0(%1: index, %2: index, %v: f64) :
+      vector.print %1: index
+      vector.print %2: index
+      vector.print %v: f64
+    }
+    return
+  }
+
+  func.func @foreach_print_slice_coo_dyn(%A: tensor<?x?xf64, #COO_SLICE_DYN>) {
+    sparse_tensor.foreach in %A : tensor<?x?xf64, #COO_SLICE_DYN> do {
+    ^bb0(%1: index, %2: index, %v: f64) :
+      vector.print %1: index
+      vector.print %2: index
+      vector.print %v: f64
+    }
+    return
+  }
+
   func.func @entry() {
     %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : index
+    %c2 = arith.constant 2 : index
+    %c4 = arith.constant 4 : index
+
     %sa = arith.constant dense<[
         [ 0.0, 2.1, 0.0, 0.0, 0.0, 6.1, 0.0, 0.0 ],
         [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ],
@@ -52,9 +108,14 @@ module {
         [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0 ]
     ]> : tensor<8x8xf64>
 
+
     %tmp = sparse_tensor.convert %sa : tensor<8x8xf64> to tensor<8x8xf64, #CSR>
     %a = tensor.extract_slice %tmp[1, 1][4, 4][1, 2] : tensor<8x8xf64, #CSR> to
                                                        tensor<4x4xf64, #CSR_SLICE>
+
+    %tmp_coo = sparse_tensor.convert %sa : tensor<8x8xf64> to tensor<8x8xf64, #COO>
+    %a_coo = tensor.extract_slice %tmp_coo[1, 1][4, 4][1, 2] : tensor<8x8xf64, #COO> to
+                                                               tensor<4x4xf64, #COO_SLICE>
     // Foreach on sparse tensor slices directly
     //
     // CHECK: 1
@@ -68,11 +129,23 @@ module {
     // CHECK-NEXT: 2.1
     //
     call @foreach_print_slice(%a) : (tensor<4x4xf64, #CSR_SLICE>) -> ()
+    // Same results for COO
+    // CHECK-NEXT: 1
+    // CHECK-NEXT: 0
+    // CHECK-NEXT: 2.3
+    // CHECK-NEXT: 2
+    // CHECK-NEXT: 3
+    // CHECK-NEXT: 1
+    // CHECK-NEXT: 3
+    // CHECK-NEXT: 2
+    // CHECK-NEXT: 2.1
+    //
+    call @foreach_print_slice_coo(%a_coo) : (tensor<4x4xf64, #COO_SLICE>) -> ()
 
     %dense = tensor.extract_slice %sa[1, 1][4, 4][1, 2] : tensor<8x8xf64> to
                                                           tensor<4x4xf64>
     %b = sparse_tensor.convert %dense : tensor<4x4xf64> to tensor<4x4xf64, #CSR>
-    // Foreach on sparse tensor instead of slice should yield the same result.
+    // Foreach on sparse tensor instead of slice they should yield the same result.
     //
     // CHECK-NEXT: 1
     // CHECK-NEXT: 0
@@ -86,8 +159,45 @@ module {
     //
     call @foreach_print_non_slice(%b) : (tensor<4x4xf64, #CSR>) -> ()
 
-    bufferization.dealloc_tensor %b : tensor<4x4xf64, #CSR>
+    // The same slice, but with dynamic encoding.
+    // TODO: Investigates why reusing the same %tmp above would cause bufferization
+    // errors.
+    %tmp1 = sparse_tensor.convert %sa : tensor<8x8xf64> to tensor<8x8xf64, #CSR>
+    %a_dyn = tensor.extract_slice %tmp1[%c1, %c1][%c4, %c4][%c1, %c2] : tensor<8x8xf64, #CSR> to
+                                                                        tensor<?x?xf64, #CSR_SLICE_DYN>
+
+    %tmp1_coo = sparse_tensor.convert %sa : tensor<8x8xf64> to tensor<8x8xf64, #COO>
+    %a_dyn_coo = tensor.extract_slice %tmp1_coo[%c1, %c1][%c4, %c4][%c1, %c2] : tensor<8x8xf64, #COO> to
+                                                                                tensor<?x?xf64, #COO_SLICE_DYN>
+    //
+    // CHECK-NEXT: 1
+    // CHECK-NEXT: 0
+    // CHECK-NEXT: 2.3
+    // CHECK-NEXT: 2
+    // CHECK-NEXT: 3
+    // CHECK-NEXT: 1
+    // CHECK-NEXT: 3
+    // CHECK-NEXT: 2
+    // CHECK-NEXT: 2.1
+    //
+    call @foreach_print_slice_dyn(%a_dyn) : (tensor<?x?xf64, #CSR_SLICE_DYN>) -> ()
+    // CHECK-NEXT: 1
+    // CHECK-NEXT: 0
+    // CHECK-NEXT: 2.3
+    // CHECK-NEXT: 2
+    // CHECK-NEXT: 3
+    // CHECK-NEXT: 1
+    // CHECK-NEXT: 3
+    // CHECK-NEXT: 2
+    // CHECK-NEXT: 2.1
+    //
+    call @foreach_print_slice_coo_dyn(%a_dyn_coo) : (tensor<?x?xf64, #COO_SLICE_DYN>) -> ()
+
     bufferization.dealloc_tensor %tmp : tensor<8x8xf64, #CSR>
+    bufferization.dealloc_tensor %tmp1 : tensor<8x8xf64, #CSR>
+    bufferization.dealloc_tensor %tmp_coo : tensor<8x8xf64, #COO>
+    bufferization.dealloc_tensor %tmp1_coo : tensor<8x8xf64, #COO>
+    bufferization.dealloc_tensor %b : tensor<4x4xf64, #CSR>
     return
   }
 }
diff --git a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_insert_1d.mlir b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_insert_1d.mlir
index 26170e86d2297..c5a66fa73b8e3 100644
--- a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_insert_1d.mlir
+++ b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_insert_1d.mlir
@@ -36,13 +36,13 @@
 
 module {
 
-  // Dumps pointers, indices, values for verification.
+  // Dumps positions, indices, values for verification.
   func.func @dump(%argx: tensor<1024xf32, #SparseVector>) {
     %c0 = arith.constant 0 : index
     %f0 = arith.constant 0.0 : f32
-    %p = sparse_tensor.pointers %argx { dimension = 0 : index }
+    %p = sparse_tensor.positions %argx { level = 0 : index }
        : tensor<1024xf32, #SparseVector> to memref<?xindex>
-    %i = sparse_tensor.indices %argx { dimension = 0 : index }
+    %i = sparse_tensor.coordinates %argx { level = 0 : index }
        : tensor<1024xf32, #SparseVector> to memref<?xindex>
     %v = sparse_tensor.values %argx
        : tensor<1024xf32, #SparseVector> to memref<?xf32>
diff --git a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_insert_2d.mlir b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_insert_2d.mlir
index 582c0741f7d5a..5ae72e028c0c1 100644
--- a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_insert_2d.mlir
+++ b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_insert_2d.mlir
@@ -57,9 +57,9 @@ module {
     %c0 = arith.constant 0 : index
     %cu = arith.constant -1 : index
     %fu = arith.constant 99.0 : f64
-    %p0 = sparse_tensor.pointers %arg0 { dimension = 0 : index } : tensor<4x3xf64, #SortedCOO> to memref<?xindex>
-    %i0 = sparse_tensor.indices  %arg0 { dimension = 0 : index } : tensor<4x3xf64, #SortedCOO> to memref<?xindex, strided<[?], offset: ?>>
-    %i1 = sparse_tensor.indices  %arg0 { dimension = 1 : index } : tensor<4x3xf64, #SortedCOO> to memref<?xindex, strided<[?], offset: ?>>
+    %p0 = sparse_tensor.positions %arg0 { level = 0 : index } : tensor<4x3xf64, #SortedCOO> to memref<?xindex>
+    %i0 = sparse_tensor.coordinates  %arg0 { level = 0 : index } : tensor<4x3xf64, #SortedCOO> to memref<?xindex, strided<[?], offset: ?>>
+    %i1 = sparse_tensor.coordinates  %arg0 { level = 1 : index } : tensor<4x3xf64, #SortedCOO> to memref<?xindex, strided<[?], offset: ?>>
     %v = sparse_tensor.values %arg0 : tensor<4x3xf64, #SortedCOO> to memref<?xf64>
     %vp0 = vector.transfer_read %p0[%c0], %cu: memref<?xindex>, vector<2xindex>
     vector.print %vp0 : vector<2xindex>
@@ -76,8 +76,8 @@ module {
     %c0 = arith.constant 0 : index
     %cu = arith.constant -1 : index
     %fu = arith.constant 99.0 : f64
-    %p1 = sparse_tensor.pointers %arg0 { dimension = 1 : index } : tensor<4x3xf64, #CSR> to memref<?xindex>
-    %i1 = sparse_tensor.indices  %arg0 { dimension = 1 : index } : tensor<4x3xf64, #CSR> to memref<?xindex>
+    %p1 = sparse_tensor.positions %arg0 { level = 1 : index } : tensor<4x3xf64, #CSR> to memref<?xindex>
+    %i1 = sparse_tensor.coordinates  %arg0 { level = 1 : index } : tensor<4x3xf64, #CSR> to memref<?xindex>
     %v = sparse_tensor.values %arg0 : tensor<4x3xf64, #CSR> to memref<?xf64>
     %vp1 = vector.transfer_read %p1[%c0], %cu: memref<?xindex>, vector<5xindex>
     vector.print %vp1 : vector<5xindex>
@@ -92,10 +92,10 @@ module {
     %c0 = arith.constant 0 : index
     %cu = arith.constant -1 : index
     %fu = arith.constant 99.0 : f64
-    %p0 = sparse_tensor.pointers %arg0 { dimension = 0 : index } : tensor<4x3xf64, #DCSR> to memref<?xindex>
-    %i0 = sparse_tensor.indices  %arg0 { dimension = 0 : index } : tensor<4x3xf64, #DCSR> to memref<?xindex>
-    %p1 = sparse_tensor.pointers %arg0 { dimension = 1 : index } : tensor<4x3xf64, #DCSR> to memref<?xindex>
-    %i1 = sparse_tensor.indices  %arg0 { dimension = 1 : index } : tensor<4x3xf64, #DCSR> to memref<?xindex>
+    %p0 = sparse_tensor.positions %arg0 { level = 0 : index } : tensor<4x3xf64, #DCSR> to memref<?xindex>
+    %i0 = sparse_tensor.coordinates  %arg0 { level = 0 : index } : tensor<4x3xf64, #DCSR> to memref<?xindex>
+    %p1 = sparse_tensor.positions %arg0 { level = 1 : index } : tensor<4x3xf64, #DCSR> to memref<?xindex>
+    %i1 = sparse_tensor.coordinates  %arg0 { level = 1 : index } : tensor<4x3xf64, #DCSR> to memref<?xindex>
     %v = sparse_tensor.values %arg0 : tensor<4x3xf64, #DCSR> to memref<?xf64>
     %vp0 = vector.transfer_read %p0[%c0], %cu: memref<?xindex>, vector<2xindex>
     vector.print %vp0 : vector<2xindex>
@@ -114,8 +114,8 @@ module {
     %c0 = arith.constant 0 : index
     %cu = arith.constant -1 : index
     %fu = arith.constant 99.0 : f64
-    %p0 = sparse_tensor.pointers %arg0 { dimension = 0 : index } : tensor<4x3xf64, #Row> to memref<?xindex>
-    %i0 = sparse_tensor.indices  %arg0 { dimension = 0 : index } : tensor<4x3xf64, #Row> to memref<?xindex>
+    %p0 = sparse_tensor.positions %arg0 { level = 0 : index } : tensor<4x3xf64, #Row> to memref<?xindex>
+    %i0 = sparse_tensor.coordinates  %arg0 { level = 0 : index } : tensor<4x3xf64, #Row> to memref<?xindex>
     %v = sparse_tensor.values %arg0 : tensor<4x3xf64, #Row> to memref<?xf64>
     %vp0 = vector.transfer_read %p0[%c0], %cu: memref<?xindex>, vector<2xindex>
     vector.print %vp0 : vector<2xindex>
diff --git a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_insert_3d.mlir b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_insert_3d.mlir
index 34bf6e514f682..0d2aa4990655e 100644
--- a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_insert_3d.mlir
+++ b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_insert_3d.mlir
@@ -44,10 +44,10 @@ module {
   func.func @dump(%arg0: tensor<5x4x3xf64, #TensorCSR>) {
     %c0 = arith.constant 0 : index
     %fu = arith.constant 99.0 : f64
-    %p0 = sparse_tensor.pointers %arg0 { dimension = 0 : index } : tensor<5x4x3xf64, #TensorCSR> to memref<?xindex>
-    %i0 = sparse_tensor.indices  %arg0 { dimension = 0 : index } : tensor<5x4x3xf64, #TensorCSR> to memref<?xindex>
-    %p2 = sparse_tensor.pointers %arg0 { dimension = 2 : index } : tensor<5x4x3xf64, #TensorCSR> to memref<?xindex>
-    %i2 = sparse_tensor.indices  %arg0 { dimension = 2 : index } : tensor<5x4x3xf64, #TensorCSR> to memref<?xindex>
+    %p0 = sparse_tensor.positions %arg0 { level = 0 : index } : tensor<5x4x3xf64, #TensorCSR> to memref<?xindex>
+    %i0 = sparse_tensor.coordinates  %arg0 { level = 0 : index } : tensor<5x4x3xf64, #TensorCSR> to memref<?xindex>
+    %p2 = sparse_tensor.positions %arg0 { level = 2 : index } : tensor<5x4x3xf64, #TensorCSR> to memref<?xindex>
+    %i2 = sparse_tensor.coordinates  %arg0 { level = 2 : index } : tensor<5x4x3xf64, #TensorCSR> to memref<?xindex>
     %v = sparse_tensor.values %arg0 : tensor<5x4x3xf64, #TensorCSR> to memref<?xf64>
     %vp0 = vector.transfer_read %p0[%c0], %c0: memref<?xindex>, vector<2xindex>
     vector.print %vp0 : vector<2xindex>
@@ -65,10 +65,10 @@ module {
   func.func @dump_row(%arg0: tensor<5x4x3xf64, #TensorRow>) {
     %c0 = arith.constant 0 : index
     %fu = arith.constant 99.0 : f64
-    %p0 = sparse_tensor.pointers %arg0 { dimension = 0 : index } : tensor<5x4x3xf64, #TensorRow> to memref<?xindex>
-    %i0 = sparse_tensor.indices  %arg0 { dimension = 0 : index } : tensor<5x4x3xf64, #TensorRow> to memref<?xindex>
-    %p1 = sparse_tensor.pointers %arg0 { dimension = 1 : index } : tensor<5x4x3xf64, #TensorRow> to memref<?xindex>
-    %i1 = sparse_tensor.indices  %arg0 { dimension = 1 : index } : tensor<5x4x3xf64, #TensorRow> to memref<?xindex>
+    %p0 = sparse_tensor.positions %arg0 { level = 0 : index } : tensor<5x4x3xf64, #TensorRow> to memref<?xindex>
+    %i0 = sparse_tensor.coordinates  %arg0 { level = 0 : index } : tensor<5x4x3xf64, #TensorRow> to memref<?xindex>
+    %p1 = sparse_tensor.positions %arg0 { level = 1 : index } : tensor<5x4x3xf64, #TensorRow> to memref<?xindex>
+    %i1 = sparse_tensor.coordinates  %arg0 { level = 1 : index } : tensor<5x4x3xf64, #TensorRow> to memref<?xindex>
     %v = sparse_tensor.values %arg0 : tensor<5x4x3xf64, #TensorRow> to memref<?xf64>
     %vp0 = vector.transfer_read %p0[%c0], %c0: memref<?xindex>, vector<2xindex>
     vector.print %vp0 : vector<2xindex>
@@ -86,11 +86,11 @@ module {
 func.func @dump_ccoo(%arg0: tensor<5x4x3xf64, #CCoo>) {
     %c0 = arith.constant 0 : index
     %fu = arith.constant 99.0 : f64
-    %p0 = sparse_tensor.pointers %arg0 { dimension = 0 : index } : tensor<5x4x3xf64, #CCoo> to memref<?xindex>
-    %i0 = sparse_tensor.indices  %arg0 { dimension = 0 : index } : tensor<5x4x3xf64, #CCoo> to memref<?xindex>
-    %p1 = sparse_tensor.pointers %arg0 { dimension = 1 : index } : tensor<5x4x3xf64, #CCoo> to memref<?xindex>
-    %i1 = sparse_tensor.indices  %arg0 { dimension = 1 : index } : tensor<5x4x3xf64, #CCoo> to memref<?xindex>
-    %i2 = sparse_tensor.indices  %arg0 { dimension = 2 : index } : tensor<5x4x3xf64, #CCoo> to memref<?xindex>
+    %p0 = sparse_tensor.positions %arg0 { level = 0 : index } : tensor<5x4x3xf64, #CCoo> to memref<?xindex>
+    %i0 = sparse_tensor.coordinates  %arg0 { level = 0 : index } : tensor<5x4x3xf64, #CCoo> to memref<?xindex>
+    %p1 = sparse_tensor.positions %arg0 { level = 1 : index } : tensor<5x4x3xf64, #CCoo> to memref<?xindex>
+    %i1 = sparse_tensor.coordinates  %arg0 { level = 1 : index } : tensor<5x4x3xf64, #CCoo> to memref<?xindex>
+    %i2 = sparse_tensor.coordinates  %arg0 { level = 2 : index } : tensor<5x4x3xf64, #CCoo> to memref<?xindex>
     %v = sparse_tensor.values %arg0 : tensor<5x4x3xf64, #CCoo> to memref<?xf64>
     %vp0 = vector.transfer_read %p0[%c0], %c0: memref<?xindex>, vector<2xindex>
     vector.print %vp0 : vector<2xindex>
@@ -110,9 +110,9 @@ func.func @dump_ccoo(%arg0: tensor<5x4x3xf64, #CCoo>) {
 func.func @dump_dcoo(%arg0: tensor<5x4x3xf64, #DCoo>) {
     %c0 = arith.constant 0 : index
     %fu = arith.constant 99.0 : f64
-    %p1 = sparse_tensor.pointers %arg0 { dimension = 1 : index } : tensor<5x4x3xf64, #DCoo> to memref<?xindex>
-    %i1 = sparse_tensor.indices  %arg0 { dimension = 1 : index } : tensor<5x4x3xf64, #DCoo> to memref<?xindex>
-    %i2 = sparse_tensor.indices  %arg0 { dimension = 2 : index } : tensor<5x4x3xf64, #DCoo> to memref<?xindex>
+    %p1 = sparse_tensor.positions %arg0 { level = 1 : index } : tensor<5x4x3xf64, #DCoo> to memref<?xindex>
+    %i1 = sparse_tensor.coordinates  %arg0 { level = 1 : index } : tensor<5x4x3xf64, #DCoo> to memref<?xindex>
+    %i2 = sparse_tensor.coordinates  %arg0 { level = 2 : index } : tensor<5x4x3xf64, #DCoo> to memref<?xindex>
     %v = sparse_tensor.values %arg0 : tensor<5x4x3xf64, #DCoo> to memref<?xf64>
     %vp1 = vector.transfer_read %p1[%c0], %c0: memref<?xindex>, vector<6xindex>
     vector.print %vp1 : vector<6xindex>
diff --git a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_matmul_slice.mlir b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_matmul_slice.mlir
index 8f77b3d6a16bb..1187cfe9655a8 100644
--- a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_matmul_slice.mlir
+++ b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_matmul_slice.mlir
@@ -28,6 +28,10 @@
   slice = [ (0, 4, 1), (0, 8, 1) ]
 }>
 
+#COO = #sparse_tensor.encoding<{
+  dimLevelType = [ "compressed-nu", "singleton" ]
+}>
+
 #CSR_SLICE_1 = #sparse_tensor.encoding<{
   dimLevelType = [ "dense", "compressed" ],
   slice = [ (0, 4, 2), (0, 4, 1) ]
@@ -38,10 +42,41 @@
   slice = [ (0, 4, 2), (1, 4, 1) ]
 }>
 
+#COO_SLICE_1 = #sparse_tensor.encoding<{
+  dimLevelType = [ "compressed-nu", "singleton" ],
+  slice = [ (0, 4, 2), (0, 4, 1) ]
+}>
+
+#COO_SLICE_2 = #sparse_tensor.encoding<{
+  dimLevelType = [ "compressed-nu", "singleton" ],
+  slice = [ (0, 4, 2), (1, 4, 1) ]
+}>
+
+#CSR_SLICE_dyn = #sparse_tensor.encoding<{
+  dimLevelType = [ "dense", "compressed" ],
+  slice = [ (?, 4, ?), (?, 4, ?) ]
+}>
+
+#DCSR_SLICE_dyn = #sparse_tensor.encoding<{
+  dimLevelType = [ "compressed", "compressed" ],
+  slice = [ (?, 4, ?), (?, 4, ?) ]
+}>
+
 module {
   func.func private @printMemrefF64(%ptr : tensor<*xf64>)
   func.func private @printMemref1dF64(%ptr : memref<?xf64>) attributes { llvm.emit_c_interface }
 
+  //
+  // Computes C = A x B with all matrices dynamic sparse slice (SpMSpM) in CSR and DCSR
+  //
+  func.func @matmul_dyn(%A: tensor<4x4xf64, #CSR_SLICE_dyn>,
+                        %B: tensor<4x4xf64, #DCSR_SLICE_dyn>) -> tensor<4x4xf64, #CSR> {
+    %C = bufferization.alloc_tensor() : tensor<4x4xf64, #CSR>
+    %D = linalg.matmul
+      ins(%A, %B: tensor<4x4xf64, #CSR_SLICE_dyn>, tensor<4x4xf64, #DCSR_SLICE_dyn>)
+         outs(%C: tensor<4x4xf64, #CSR>) -> tensor<4x4xf64, #CSR>
+    return %D: tensor<4x4xf64, #CSR>
+  }
 
   //
   // Computes C = A x B with one matrix CSR sparse slices and the other DSCR sparse slice.
@@ -79,11 +114,24 @@ module {
     return %D: tensor<4x4xf64, #DCSR>
   }
 
+  //
+  // Computes C = A x B with two COO slices.
+  //
+  func.func @matmul5(%A: tensor<4x4xf64, #COO_SLICE_1>,
+                     %B: tensor<4x4xf64, #COO_SLICE_2>) -> tensor<4x4xf64, #COO> {
+    %C = bufferization.alloc_tensor() : tensor<4x4xf64, #COO>
+    %D = linalg.matmul
+      ins(%A, %B: tensor<4x4xf64, #COO_SLICE_1>, tensor<4x4xf64, #COO_SLICE_2>)
+         outs(%C: tensor<4x4xf64, #COO>) -> tensor<4x4xf64, #COO>
+    return %D: tensor<4x4xf64, #COO>
+  }
   //
   // Main driver.
   //
   func.func @entry() {
-    %c0 = arith.constant 0 : index
+    %c_0 = arith.constant 0 : index
+    %c_1 = arith.constant 1 : index
+    %c_2 = arith.constant 2 : index
     %f0 = arith.constant 0.0 : f64
 
     %sa = arith.constant dense<[
@@ -158,11 +206,43 @@ module {
     %4 = call @matmul1(%s2, %s1)
        : (tensor<4x4xf64, #CSR_SLICE_1>,
           tensor<4x4xf64, #DCSR_SLICE_1>) -> tensor<4x4xf64, #CSR>
-
     %c4 = sparse_tensor.convert %4 : tensor<4x4xf64, #CSR> to tensor<4x4xf64>
     %c4u = tensor.cast %c4 : tensor<4x4xf64> to tensor<*xf64>
     call @printMemrefF64(%c4u) : (tensor<*xf64>) -> ()
 
+    // slice coo x slice coo
+    //
+    // CHECK:      [2.3,   0,   0,   0],
+    // CHECK-NEXT: [6.9,   0,   0,   0],
+    // CHECK-NEXT: [0,   0,   0,   0],
+    // CHECK-NEXT: [12.6,   0,   0,   0]]
+    //
+    %t1_coo = sparse_tensor.convert %sa : tensor<8x8xf64> to tensor<8x8xf64, #COO>
+    %b1_coo = sparse_tensor.convert %sb : tensor<8x4xf64> to tensor<8x4xf64, #COO>
+    %s2_coo = tensor.extract_slice %b1_coo[0, 0][4, 4][2, 1] : tensor<8x4xf64, #COO> to tensor<4x4xf64, #COO_SLICE_1>
+    %s1_coo = tensor.extract_slice %t1_coo[0, 1][4, 4][2, 1] : tensor<8x8xf64, #COO> to tensor<4x4xf64, #COO_SLICE_2>
+    %o_coo = call @matmul5(%s2_coo, %s1_coo) : (tensor<4x4xf64, #COO_SLICE_1>, tensor<4x4xf64, #COO_SLICE_2>) -> tensor<4x4xf64, #COO>
+
+    %c4_coo = sparse_tensor.convert %o_coo : tensor<4x4xf64, #COO> to tensor<4x4xf64>
+    %c4u_coo = tensor.cast %c4_coo : tensor<4x4xf64> to tensor<*xf64>
+    call @printMemrefF64(%c4u_coo) : (tensor<*xf64>) -> ()
+
+    // slice x slice (same as above, but with dynamic stride information)
+    //
+    // CHECK:      [2.3,   0,   0,   0],
+    // CHECK-NEXT: [6.9,   0,   0,   0],
+    // CHECK-NEXT: [0,   0,   0,   0],
+    // CHECK-NEXT: [12.6,   0,   0,   0]]
+    //
+    %s1_dyn = tensor.extract_slice %tmp[%c_0, %c_1][4, 4][%c_2, %c_1] : tensor<8x8xf64, #DCSR> to tensor<4x4xf64, #DCSR_SLICE_dyn>
+    %s2_dyn = tensor.extract_slice %b1[%c_0, %c_0][4, 4][%c_2, %c_1] : tensor<8x4xf64, #CSR> to tensor<4x4xf64, #CSR_SLICE_dyn>
+    %dyn_4 = call @matmul_dyn(%s2_dyn, %s1_dyn)
+       : (tensor<4x4xf64, #CSR_SLICE_dyn>,
+          tensor<4x4xf64, #DCSR_SLICE_dyn>) -> tensor<4x4xf64, #CSR>
+    %c4_dyn = sparse_tensor.convert %dyn_4 : tensor<4x4xf64, #CSR> to tensor<4x4xf64>
+    %c4u_dyn = tensor.cast %c4_dyn : tensor<4x4xf64> to tensor<*xf64>
+    call @printMemrefF64(%c4u_dyn) : (tensor<*xf64>) -> ()
+
     // sparse slices should generate the same result as dense slices
     //
     // CHECK:      [2.3,   0,   0,   0],
@@ -179,14 +259,18 @@ module {
     %du = tensor.cast %r : tensor<4x4xf64> to tensor<*xf64>
     call @printMemrefF64(%du) : (tensor<*xf64>) -> ()
 
-    // Releases resources.
+    // Releases resources (we do not need to deallocate slices).
     bufferization.dealloc_tensor %b1 : tensor<8x4xf64, #CSR>
     bufferization.dealloc_tensor %t1 : tensor<8x8xf64, #CSR>
+    bufferization.dealloc_tensor %b1_coo : tensor<8x4xf64, #COO>
+    bufferization.dealloc_tensor %t1_coo : tensor<8x8xf64, #COO>
+    bufferization.dealloc_tensor %o_coo : tensor<4x4xf64, #COO>
     bufferization.dealloc_tensor %b  : tensor<8x4xf64, #DCSR>
     bufferization.dealloc_tensor %tmp: tensor<8x8xf64, #DCSR>
     bufferization.dealloc_tensor %4  : tensor<4x4xf64, #CSR>
     bufferization.dealloc_tensor %3  : tensor<4x4xf64, #CSR>
     bufferization.dealloc_tensor %2  : tensor<4x4xf64, #DCSR>
+    bufferization.dealloc_tensor %dyn_4 : tensor<4x4xf64, #CSR>
 
     return
   }
diff --git a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_matvec.mlir b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_matvec.mlir
index 0acd456d3dd47..23bc5acbe7b01 100644
--- a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_matvec.mlir
+++ b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_matvec.mlir
@@ -39,8 +39,8 @@
 
 #SparseMatrix = #sparse_tensor.encoding<{
   dimLevelType = [ "dense", "compressed" ],
-  pointerBitWidth = 8,
-  indexBitWidth = 8
+  posWidth = 8,
+  crdWidth = 8
 }>
 
 #matvec = {
diff --git a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_pack.mlir b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_pack.mlir
index 46b6512723db0..ef27050eab32c 100644
--- a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_pack.mlir
+++ b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_pack.mlir
@@ -27,8 +27,8 @@
 
 #SortedCOOI32 = #sparse_tensor.encoding<{
   dimLevelType = [ "compressed-nu", "singleton" ],
-  pointerBitWidth = 32,
-  indexBitWidth = 32
+  posWidth = 32,
+  crdWidth = 32
 }>
 
 module {
diff --git a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_re_im.mlir b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_re_im.mlir
index 644e970cf6ab4..34e76e0281843 100644
--- a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_re_im.mlir
+++ b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_re_im.mlir
@@ -76,8 +76,8 @@ module {
     %values = sparse_tensor.values %arg0 : tensor<?xf32, #SparseVector> to memref<?xf32>
     %0 = vector.transfer_read %values[%c0], %d0: memref<?xf32>, vector<3xf32>
     vector.print %0 : vector<3xf32>
-    %indices = sparse_tensor.indices %arg0 { dimension = 0 : index } : tensor<?xf32, #SparseVector> to memref<?xindex>
-    %1 = vector.transfer_read %indices[%c0], %c0: memref<?xindex>, vector<3xindex>
+    %coordinates = sparse_tensor.coordinates %arg0 { level = 0 : index } : tensor<?xf32, #SparseVector> to memref<?xindex>
+    %1 = vector.transfer_read %coordinates[%c0], %c0: memref<?xindex>, vector<3xindex>
     vector.print %1 : vector<3xindex>
     return
   }
diff --git a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sampled_matmul.mlir b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sampled_matmul.mlir
index 0a2a415378c55..05fc7f65aad3e 100644
--- a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sampled_matmul.mlir
+++ b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sampled_matmul.mlir
@@ -32,8 +32,8 @@
 
 #SparseMatrix = #sparse_tensor.encoding<{
   dimLevelType = [ "compressed", "compressed" ],
-  pointerBitWidth = 32,
-  indexBitWidth = 32
+  posWidth = 32,
+  crdWidth = 32
 }>
 
 #trait_sampled_dense_dense = {
diff --git a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sorted_coo.mlir b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sorted_coo.mlir
index 77110b23bb641..3ebd9dff9111e 100644
--- a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sorted_coo.mlir
+++ b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sorted_coo.mlir
@@ -125,11 +125,11 @@ module {
     // CHECK-NEXT: ( 0, 126, 127, 254, 1, 253, 2, 0, 1, 3, 98, 126, 127, 128, 249, 253, 255, 0, 0, 0 )
     // CHECK-NEXT: ( -1, 2, -3, 4, -5, 6, -7, 8, -9, 10, -11, 12, -13, 14, -15, 16, -17, 0, 0, 0 )
     //
-    %p0 = sparse_tensor.pointers %0 { dimension = 0 : index }
+    %p0 = sparse_tensor.positions %0 { level = 0 : index }
       : tensor<?x?xf64, #SortedCOO> to memref<?xindex>
-    %i00 = sparse_tensor.indices %0 { dimension = 0 : index }
+    %i00 = sparse_tensor.coordinates %0 { level = 0 : index }
       : tensor<?x?xf64, #SortedCOO> to memref<?xindex, strided<[?], offset: ?>>
-    %i01 = sparse_tensor.indices %0 { dimension = 1 : index }
+    %i01 = sparse_tensor.coordinates %0 { level = 1 : index }
       : tensor<?x?xf64, #SortedCOO> to memref<?xindex, strided<[?], offset: ?>>
     %v0 = sparse_tensor.values %0
       : tensor<?x?xf64, #SortedCOO> to memref<?xf64>
@@ -144,11 +144,11 @@ module {
     // CHECK-NEXT: ( 0, 3, 1, 3, 2, 3, 3, 0, 3, 0, 3, 3, 3, 1, 3, 0, 3, 0, 0, 0 )
     // CHECK-NEXT: ( -1, 8, -5, -9, -7, 10, -11, 2, 12, -3, -13, 14, -15, 6, 16, 4, -17, 0, 0, 0 )
     //
-    %p1 = sparse_tensor.pointers %1 { dimension = 0 : index }
+    %p1 = sparse_tensor.positions %1 { level = 0 : index }
       : tensor<?x?xf64, #SortedCOOPermuted> to memref<?xindex>
-    %i10 = sparse_tensor.indices %1 { dimension = 0 : index }
+    %i10 = sparse_tensor.coordinates %1 { level = 0 : index }
       : tensor<?x?xf64, #SortedCOOPermuted> to memref<?xindex, strided<[?], offset: ?>>
-    %i11 = sparse_tensor.indices %1 { dimension = 1 : index }
+    %i11 = sparse_tensor.coordinates %1 { level = 1 : index }
       : tensor<?x?xf64, #SortedCOOPermuted> to memref<?xindex, strided<[?], offset: ?>>
     %v1 = sparse_tensor.values %1
       : tensor<?x?xf64, #SortedCOOPermuted> to memref<?xf64>
@@ -164,13 +164,13 @@ module {
     // CHECK-NEXT: ( 0, 0, 1, 1, 2, 2, 2, 2, 0, 0, 0, 1, 1, 1, 1, 2, 2, 0, 0, 0 )
     // CHECK-NEXT: ( 3, 63, 11, 100, 66, 61, 13, 43, 77, 10, 46, 61, 53, 3, 75, 22, 18, 0, 0, 0 )
     //
-    %p2 = sparse_tensor.pointers %2 { dimension = 0 : index }
+    %p2 = sparse_tensor.positions %2 { level = 0 : index }
       : tensor<?x?x?xf64, #SortedCOO3D> to memref<?xindex>
-    %i20 = sparse_tensor.indices %2 { dimension = 0 : index }
+    %i20 = sparse_tensor.coordinates %2 { level = 0 : index }
       : tensor<?x?x?xf64, #SortedCOO3D> to memref<?xindex, strided<[?], offset: ?>>
-    %i21 = sparse_tensor.indices %2 { dimension = 1 : index }
+    %i21 = sparse_tensor.coordinates %2 { level = 1 : index }
       : tensor<?x?x?xf64, #SortedCOO3D> to memref<?xindex, strided<[?], offset: ?>>
-    %i22 = sparse_tensor.indices %2 { dimension = 2 : index }
+    %i22 = sparse_tensor.coordinates %2 { level = 2 : index }
       : tensor<?x?x?xf64, #SortedCOO3D> to memref<?xindex, strided<[?], offset: ?>>
     %v2 = sparse_tensor.values %2
       : tensor<?x?x?xf64, #SortedCOO3D> to memref<?xf64>
@@ -187,13 +187,13 @@ module {
     // CHECK-NEXT: ( 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0 )
     // CHECK-NEXT: ( 66, 77, 61, 11, 61, 53, 22, 3, 100, 13, 10, 3, 18, 63, 43, 46, 75, 0, 0, 0 )
     //
-    %p3 = sparse_tensor.pointers %3 { dimension = 0 : index }
+    %p3 = sparse_tensor.positions %3 { level = 0 : index }
       : tensor<?x?x?xf64, #SortedCOO3DPermuted> to memref<?xindex>
-    %i30 = sparse_tensor.indices %3 { dimension = 0 : index }
+    %i30 = sparse_tensor.coordinates %3 { level = 0 : index }
       : tensor<?x?x?xf64, #SortedCOO3DPermuted> to memref<?xindex, strided<[?], offset: ?>>
-    %i31 = sparse_tensor.indices %3 { dimension = 1 : index }
+    %i31 = sparse_tensor.coordinates %3 { level = 1 : index }
       : tensor<?x?x?xf64, #SortedCOO3DPermuted> to memref<?xindex, strided<[?], offset: ?>>
-    %i32 = sparse_tensor.indices %3 { dimension = 2 : index }
+    %i32 = sparse_tensor.coordinates %3 { level = 2 : index }
       : tensor<?x?x?xf64, #SortedCOO3DPermuted> to memref<?xindex, strided<[?], offset: ?>>
     %v3 = sparse_tensor.values %3
       : tensor<?x?x?xf64, #SortedCOO3DPermuted> to memref<?xf64>
@@ -209,11 +209,11 @@ module {
     // CHECK-NEXT: ( 0, 3, 0, 3, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
    // CHECK-NEXT: ( 6, 5, 4, 3, 2, 11, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
     //
-    %p4 = sparse_tensor.pointers %4 { dimension = 0 : index }
+    %p4 = sparse_tensor.positions %4 { level = 0 : index }
       : tensor<?x?xf64, #SortedCOO> to memref<?xindex>
-    %i40 = sparse_tensor.indices %4 { dimension = 0 : index }
+    %i40 = sparse_tensor.coordinates %4 { level = 0 : index }
       : tensor<?x?xf64, #SortedCOO> to memref<?xindex, strided<[?], offset: ?>>
-    %i41 = sparse_tensor.indices %4 { dimension = 1 : index }
+    %i41 = sparse_tensor.coordinates %4 { level = 1 : index }
       : tensor<?x?xf64, #SortedCOO> to memref<?xindex, strided<[?], offset: ?>>
     %v4 = sparse_tensor.values %4
       : tensor<?x?xf64, #SortedCOO> to memref<?xf64>
@@ -232,11 +232,11 @@ module {
     // CHECK-NEXT: ( 0, 3, 0, 3, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
     // CHECK-NEXT: ( 12, 10, 8, 6, 4, 22, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
     //
-    %p5 = sparse_tensor.pointers %5 { dimension = 0 : index }
+    %p5 = sparse_tensor.positions %5 { level = 0 : index }
       : tensor<?x?xf64, #SortedCOO> to memref<?xindex>
-    %i50 = sparse_tensor.indices %5 { dimension = 0 : index }
+    %i50 = sparse_tensor.coordinates %5 { level = 0 : index }
       : tensor<?x?xf64, #SortedCOO> to memref<?xindex, strided<[?], offset: ?>>
-    %i51 = sparse_tensor.indices %5 { dimension = 1 : index }
+    %i51 = sparse_tensor.coordinates %5 { level = 1 : index }
       : tensor<?x?xf64, #SortedCOO> to memref<?xindex, strided<[?], offset: ?>>
     %v5 = sparse_tensor.values %5
       : tensor<?x?xf64, #SortedCOO> to memref<?xf64>
diff --git a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_storage.mlir b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_storage.mlir
index 7b1db663f3a65..9454c06067659 100644
--- a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_storage.mlir
+++ b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_storage.mlir
@@ -119,7 +119,7 @@ module {
     //
     // Inspect storage scheme of CSR.
     //
-    // pointers(1)
+    // positions(1)
     // indices(1)
     // values
     //
@@ -127,10 +127,10 @@ module {
     // CHECK: ( 0, 2, 7, 2, 3, 4, 1, 2, 7, 2, 6, 7, 1, 2, 6, 7, 6 )
     // CHECK: ( 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17 )
     //
-    %7 = sparse_tensor.pointers %1 { dimension = 1 : index } : tensor<10x8xf64, #CSR> to memref<?xindex>
+    %7 = sparse_tensor.positions %1 { level = 1 : index } : tensor<10x8xf64, #CSR> to memref<?xindex>
     %8 = vector.transfer_read %7[%c0], %c0: memref<?xindex>, vector<11xindex>
     vector.print %8 : vector<11xindex>
-    %9 = sparse_tensor.indices %1 { dimension = 1 : index } : tensor<10x8xf64, #CSR> to memref<?xindex>
+    %9 = sparse_tensor.coordinates %1 { level = 1 : index } : tensor<10x8xf64, #CSR> to memref<?xindex>
     %10 = vector.transfer_read %9[%c0], %c0: memref<?xindex>, vector<17xindex>
     vector.print %10 : vector<17xindex>
     %11 = sparse_tensor.values %1 : tensor<10x8xf64, #CSR> to memref<?xf64>
@@ -140,9 +140,9 @@ module {
     //
     // Inspect storage scheme of DCSR.
     //
-    // pointers(0)
+    // positions(0)
     // indices(0)
-    // pointers(1)
+    // positions(1)
     // indices(1)
     // values
     //
@@ -152,16 +152,16 @@ module {
     // CHECK: ( 0, 2, 7, 2, 3, 4, 1, 2, 7, 2, 6, 7, 1, 2, 6, 7, 6 )
     // CHECK: ( 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17 )
     //
-    %13 = sparse_tensor.pointers %2 { dimension = 0 : index } : tensor<10x8xf64, #DCSR> to memref<?xindex>
+    %13 = sparse_tensor.positions %2 { level = 0 : index } : tensor<10x8xf64, #DCSR> to memref<?xindex>
     %14 = vector.transfer_read %13[%c0], %c0: memref<?xindex>, vector<2xindex>
     vector.print %14 : vector<2xindex>
-    %15 = sparse_tensor.indices %2 { dimension = 0 : index } : tensor<10x8xf64, #DCSR> to memref<?xindex>
+    %15 = sparse_tensor.coordinates %2 { level = 0 : index } : tensor<10x8xf64, #DCSR> to memref<?xindex>
     %16 = vector.transfer_read %15[%c0], %c0: memref<?xindex>, vector<8xindex>
     vector.print %16 : vector<8xindex>
-    %17 = sparse_tensor.pointers %2 { dimension = 1 : index } : tensor<10x8xf64, #DCSR> to memref<?xindex>
+    %17 = sparse_tensor.positions %2 { level = 1 : index } : tensor<10x8xf64, #DCSR> to memref<?xindex>
     %18 = vector.transfer_read %17[%c0], %c0: memref<?xindex>, vector<9xindex>
     vector.print %18 : vector<9xindex>
-    %19 = sparse_tensor.indices %2 { dimension = 1 : index } : tensor<10x8xf64, #DCSR> to memref<?xindex>
+    %19 = sparse_tensor.coordinates %2 { level = 1 : index } : tensor<10x8xf64, #DCSR> to memref<?xindex>
     %20 = vector.transfer_read %19[%c0], %c0: memref<?xindex>, vector<17xindex>
     vector.print %20 : vector<17xindex>
     %21 = sparse_tensor.values %2 : tensor<10x8xf64, #DCSR> to memref<?xf64>
@@ -171,7 +171,7 @@ module {
     //
     // Inspect storage scheme of CSC.
     //
-    // pointers(1)
+    // positions(1)
     // indices(1)
     // values
     //
@@ -179,10 +179,10 @@ module {
     // CHECK: ( 0, 5, 7, 0, 2, 5, 6, 7, 3, 4, 6, 7, 9, 0, 5, 6, 7 )
     // CHECK: ( 1, 7, 13, 2, 4, 8, 10, 14, 5, 6, 11, 15, 17, 3, 9, 12, 16 )
     //
-    %23 = sparse_tensor.pointers %3 { dimension = 1 : index } : tensor<10x8xf64, #CSC> to memref<?xindex>
+    %23 = sparse_tensor.positions %3 { level = 1 : index } : tensor<10x8xf64, #CSC> to memref<?xindex>
     %24 = vector.transfer_read %23[%c0], %c0: memref<?xindex>, vector<9xindex>
     vector.print %24 : vector<9xindex>
-    %25 = sparse_tensor.indices %3 { dimension = 1 : index } : tensor<10x8xf64, #CSC> to memref<?xindex>
+    %25 = sparse_tensor.coordinates %3 { level = 1 : index } : tensor<10x8xf64, #CSC> to memref<?xindex>
     %26 = vector.transfer_read %25[%c0], %c0: memref<?xindex>, vector<17xindex>
     vector.print %26 : vector<17xindex>
     %27 = sparse_tensor.values %3 : tensor<10x8xf64, #CSC> to memref<?xf64>
@@ -192,9 +192,9 @@ module {
     //
     // Inspect storage scheme of DCSC.
     //
-    // pointers(0)
+    // positions(0)
     // indices(0)
-    // pointers(1)
+    // positions(1)
     // indices(1)
     // values
     //
@@ -204,16 +204,16 @@ module {
     // CHECK: ( 0, 5, 7, 0, 2, 5, 6, 7, 3, 4, 6, 7, 9, 0, 5, 6, 7 )
     // CHECK: ( 1, 7, 13, 2, 4, 8, 10, 14, 5, 6, 11, 15, 17, 3, 9, 12, 16 )
     //
-    %29 = sparse_tensor.pointers %4 { dimension = 0 : index } : tensor<10x8xf64, #DCSC> to memref<?xindex>
+    %29 = sparse_tensor.positions %4 { level = 0 : index } : tensor<10x8xf64, #DCSC> to memref<?xindex>
     %30 = vector.transfer_read %29[%c0], %c0: memref<?xindex>, vector<2xindex>
     vector.print %30 : vector<2xindex>
-    %31 = sparse_tensor.indices %4 { dimension = 0 : index } : tensor<10x8xf64, #DCSC> to memref<?xindex>
+    %31 = sparse_tensor.coordinates %4 { level = 0 : index } : tensor<10x8xf64, #DCSC> to memref<?xindex>
     %32 = vector.transfer_read %31[%c0], %c0: memref<?xindex>, vector<7xindex>
     vector.print %32 : vector<7xindex>
-    %33 = sparse_tensor.pointers %4 { dimension = 1 : index } : tensor<10x8xf64, #DCSC> to memref<?xindex>
+    %33 = sparse_tensor.positions %4 { level = 1 : index } : tensor<10x8xf64, #DCSC> to memref<?xindex>
     %34 = vector.transfer_read %33[%c0], %c0: memref<?xindex>, vector<8xindex>
     vector.print %34 : vector<8xindex>
-    %35 = sparse_tensor.indices %4 { dimension = 1 : index } : tensor<10x8xf64, #DCSC> to memref<?xindex>
+    %35 = sparse_tensor.coordinates %4 { level = 1 : index } : tensor<10x8xf64, #DCSC> to memref<?xindex>
     %36 = vector.transfer_read %35[%c0], %c0: memref<?xindex>, vector<17xindex>
     vector.print %36 : vector<17xindex>
     %37 = sparse_tensor.values %4 : tensor<10x8xf64, #DCSC> to memref<?xf64>
@@ -223,7 +223,7 @@ module {
     //
     // Inspect storage scheme of BlockRow.
     //
-    // pointers(0)
+    // positions(0)
     // indices(0)
     // values
     //
@@ -234,10 +234,10 @@ module {
     // CHECK-SAME: 0, 7, 8, 0, 0, 0, 0, 9, 0, 0, 10, 0, 0, 0, 11, 12,
     // CHECK-SAME: 0, 13, 14, 0, 0, 0, 15, 16, 0, 0, 0, 0, 0, 0, 17, 0 )
     //
-    %39 = sparse_tensor.pointers %x { dimension = 0 : index } : tensor<10x8xf64, #BlockRow> to memref<?xindex>
+    %39 = sparse_tensor.positions %x { level = 0 : index } : tensor<10x8xf64, #BlockRow> to memref<?xindex>
     %40 = vector.transfer_read %39[%c0], %c0: memref<?xindex>, vector<2xindex>
     vector.print %40 : vector<2xindex>
-    %41 = sparse_tensor.indices %x { dimension = 0 : index } : tensor<10x8xf64, #BlockRow> to memref<?xindex>
+    %41 = sparse_tensor.coordinates %x { level = 0 : index } : tensor<10x8xf64, #BlockRow> to memref<?xindex>
     %42 = vector.transfer_read %41[%c0], %c0: memref<?xindex>, vector<8xindex>
     vector.print %42 : vector<8xindex>
     %43 = sparse_tensor.values %x : tensor<10x8xf64, #BlockRow> to memref<?xf64>
@@ -247,7 +247,7 @@ module {
     //
     // Inspect storage scheme of BlockCol.
     //
-    // pointers(0)
+    // positions(0)
     // indices(0)
     // values
     //
@@ -257,10 +257,10 @@ module {
     // CHECK-SAME: 0, 8, 10, 14, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0,
     // CHECK-SAME: 0, 0, 0, 0, 0, 0, 0, 0, 0, 11, 15, 0, 17, 3, 0, 0, 0, 0, 9, 12, 16, 0, 0 )
     //
-    %45 = sparse_tensor.pointers %y { dimension = 0 : index } : tensor<10x8xf64, #BlockCol> to memref<?xindex>
+    %45 = sparse_tensor.positions %y { level = 0 : index } : tensor<10x8xf64, #BlockCol> to memref<?xindex>
     %46 = vector.transfer_read %45[%c0], %c0: memref<?xindex>, vector<2xindex>
     vector.print %46 : vector<2xindex>
-    %47 = sparse_tensor.indices %y { dimension = 0 : index } : tensor<10x8xf64, #BlockCol> to memref<?xindex>
+    %47 = sparse_tensor.coordinates %y { level = 0 : index } : tensor<10x8xf64, #BlockCol> to memref<?xindex>
     %48 = vector.transfer_read %47[%c0], %c0: memref<?xindex>, vector<7xindex>
     vector.print %48 : vector<7xindex>
     %49 = sparse_tensor.values %y : tensor<10x8xf64, #BlockCol> to memref<?xf64>
diff --git a/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_transpose_coo.mlir b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_transpose_coo.mlir
new file mode 100644
index 0000000000000..27964f47913c2
--- /dev/null
+++ b/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_transpose_coo.mlir
@@ -0,0 +1,97 @@
+// DEFINE: %{option} = enable-runtime-library=true
+// DEFINE: %{compile} = mlir-opt %s --sparse-compiler=%{option}
+// DEFINE: %{run} = mlir-cpu-runner \
+// DEFINE:  -e entry -entry-point-result=void  \
+// DEFINE:  -shared-libs=%mlir_c_runner_utils | \
+// DEFINE: FileCheck %s
+//
+// FIXME: lib path does not support all of COO yet
+// R_U_N: %{compile} | %{run}
+//
+// Do the same run, but now with direct IR generation.
+// REDEFINE: %{option} = enable-runtime-library=false
+// RUN: %{compile} | %{run}
+//
+// Do the same run, but now with direct IR generation and vectorization.
+// REDEFINE: %{option} = "enable-runtime-library=false vl=2 reassociate-fp-reductions=true enable-index-optimizations=true"
+// RUN: %{compile} | %{run}
+//
+// Do the same run, but now with direct IR generation and, if available, VLA
+// vectorization.
+// REDEFINE: %{option} = "enable-runtime-library=false vl=4 enable-arm-sve=%ENABLE_VLA"
+// REDEFINE: %{run} = %lli \
+// REDEFINE:   --entry-function=entry_lli \
+// REDEFINE:   --extra-module=%S/Inputs/main_for_lli.ll \
+// REDEFINE:   %VLA_ARCH_ATTR_OPTIONS \
+// REDEFINE:   --dlopen=%mlir_native_utils_lib_dir/libmlir_c_runner_utils%shlibext | \
+// REDEFINE: FileCheck %s
+// RUN: %{compile} | mlir-translate -mlir-to-llvmir | %{run}
+
+#SortedCOO = #sparse_tensor.encoding<{
+  dimLevelType = [ "compressed-nu", "singleton" ]
+}>
+
+module {
+
+  // A linalg representation of some higher "transpose" op.
+  func.func @transpose_coo(%arga: tensor<10x5xf32, #SortedCOO>)
+                               -> tensor<5x10xf32, #SortedCOO> {
+    %0 = bufferization.alloc_tensor() : tensor<5x10xf32, #SortedCOO>
+    %1 = linalg.generic {
+      indexing_maps = [affine_map<(d0, d1) -> (d1, d0)>,
+                       affine_map<(d0, d1) -> (d0, d1)>],
+      iterator_types = ["parallel", "parallel"]}
+      ins(%arga : tensor<10x5xf32, #SortedCOO>)
+      outs(%0 : tensor<5x10xf32, #SortedCOO>) {
+        ^bb0(%in: f32, %out: f32):
+           linalg.yield %in : f32
+      } -> tensor<5x10xf32, #SortedCOO>
+    return %1 : tensor<5x10xf32, #SortedCOO>
+  }
+
+  func.func @entry() {
+    %f0  = arith.constant 0.0 : f32
+    %c0  = arith.constant 0   : index
+    %c1  = arith.constant 1   : index
+
+    %A = arith.constant dense<
+        [ [ 10.0, 20.0, 30.0, 40.0, 50.0 ],
+          [ 11.0, 21.0, 31.0, 41.0, 51.0 ],
+          [ 12.0, 22.0, 32.0, 42.0, 52.0 ],
+          [ 13.0, 23.0, 33.0, 43.0, 53.0 ],
+          [ 14.0, 24.0, 34.0, 44.0, 54.0 ],
+          [ 15.0, 25.0, 35.0, 45.0, 55.0 ],
+          [ 16.0, 26.0, 36.0, 46.0, 56.0 ],
+          [ 17.0, 27.0, 37.0, 47.0, 57.0 ],
+          [ 18.0, 28.0, 38.0, 48.0, 58.0 ],
+          [ 19.0, 29.0, 39.0, 49.0, 59.0 ] ]
+    > : tensor<10x5xf32>
+
+    // Stress test with a "sparse" version of A.
+    %SA = sparse_tensor.convert %A
+      : tensor<10x5xf32> to tensor<10x5xf32, #SortedCOO>
+    %SAT = call @transpose_coo(%SA) : (tensor<10x5xf32, #SortedCOO>)
+                                    -> tensor<5x10xf32, #SortedCOO>
+
+    //
+    // Verify original and transposed sorted COO.
+    //
+    // CHECK:      ( 10, 20, 30, 40, 50, 11, 21, 31, 41, 51, 12, 22, 32, 42, 52, 13, 23, 33, 43, 53, 14, 24, 34, 44, 54, 15, 25, 35, 45, 55, 16, 26, 36, 46, 56, 17, 27, 37, 47, 57, 18, 28, 38, 48, 58, 19, 29, 39, 49, 59 )
+    // CHECK-NEXT: ( 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59 )
+    //
+    %va  = sparse_tensor.values %SA
+      : tensor<10x5xf32, #SortedCOO> to memref<?xf32>
+    %vat = sparse_tensor.values %SAT
+      : tensor<5x10xf32, #SortedCOO> to memref<?xf32>
+    %v1 = vector.transfer_read %va[%c0],  %f0 : memref<?xf32>, vector<50xf32>
+    %v2 = vector.transfer_read %vat[%c0], %f0 : memref<?xf32>, vector<50xf32>
+    vector.print %v1 : vector<50xf32>
+    vector.print %v2 : vector<50xf32>
+
+    // Release resources.
+    bufferization.dealloc_tensor %SA  : tensor<10x5xf32, #SortedCOO>
+    bufferization.dealloc_tensor %SAT : tensor<5x10xf32, #SortedCOO>
+
+    return
+  }
+}
diff --git a/mlir/test/Integration/Dialect/SparseTensor/taco/tools/mlir_pytaco.py b/mlir/test/Integration/Dialect/SparseTensor/taco/tools/mlir_pytaco.py
index f0f6ecad83f03..44d28b08d8b30 100644
--- a/mlir/test/Integration/Dialect/SparseTensor/taco/tools/mlir_pytaco.py
+++ b/mlir/test/Integration/Dialect/SparseTensor/taco/tools/mlir_pytaco.py
@@ -48,9 +48,9 @@
 _TACO_INDEX_PREFIX = "i"
 _TACO_TENSOR_PREFIX = "A"
 
-# Bitwidths for pointers and indices.
-_POINTER_BIT_WIDTH = 0
-_INDEX_BIT_WIDTH = 0
+# Bitwidths for positions and coordinates.
+_POS_WIDTH = 0
+_CRD_WIDTH = 0
 # The entry point to the JIT compiled program.
 _ENTRY_NAME = "main"
 
@@ -366,8 +366,7 @@ def mlir_tensor_attr(self) -> Optional[sparse_tensor.EncodingAttr]:
     mlir_storage_format = [f.value for f in self.format_pack.formats]
     return sparse_tensor.EncodingAttr.get(mlir_storage_format,
                                           ir.AffineMap.get_permutation(order),
-                                          None, _POINTER_BIT_WIDTH,
-                                          _INDEX_BIT_WIDTH)
+                                          None, _POS_WIDTH, _CRD_WIDTH)
 
 
 def _make_format(formats: List[ModeFormat],
diff --git a/mlir/test/Pass/crash-recovery-dynamic-failure.mlir b/mlir/test/Pass/crash-recovery-dynamic-failure.mlir
index 69e087d5aa83a..cea4f6a79963b 100644
--- a/mlir/test/Pass/crash-recovery-dynamic-failure.mlir
+++ b/mlir/test/Pass/crash-recovery-dynamic-failure.mlir
@@ -1,10 +1,10 @@
+// The crash recovery mechanism will leak memory allocated in the crashing thread.
+// RUN: export LSAN_OPTIONS=detect_leaks=0
+
 // Check that local reproducers will also traverse dynamic pass pipelines.
 // RUN: mlir-opt %s -pass-pipeline='builtin.module(test-module-pass,test-dynamic-pipeline{op-name=inner_mod1 run-on-nested-operations=1 dynamic-pipeline=test-pass-failure})' -mlir-pass-pipeline-crash-reproducer=%t -verify-diagnostics -mlir-pass-pipeline-local-reproducer --mlir-disable-threading
 // RUN: cat %t | FileCheck -check-prefix=REPRO_LOCAL_DYNAMIC_FAILURE %s
 
-// The crash recovery mechanism will leak memory allocated in the crashing thread.
-// UNSUPPORTED: asan
-
 module @inner_mod1 {
   // expected-error@below {{Failures have been detected while processing an MLIR pass pipeline}}
   // expected-note@below {{Pipeline failed while executing}}
diff --git a/mlir/test/Pass/crash-recovery.mlir b/mlir/test/Pass/crash-recovery.mlir
index e636064d26e34..e5843d6207c67 100644
--- a/mlir/test/Pass/crash-recovery.mlir
+++ b/mlir/test/Pass/crash-recovery.mlir
@@ -1,3 +1,5 @@
+// The crash recovery mechanism will leak memory allocated in the crashing thread.
+// RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: mlir-opt %s -pass-pipeline='builtin.module(builtin.module(test-module-pass, test-pass-crash))' -mlir-pass-pipeline-crash-reproducer=%t -verify-diagnostics
 // RUN: cat %t | FileCheck -check-prefix=REPRO %s
 // RUN: mlir-opt %s -pass-pipeline='builtin.module(builtin.module(test-module-pass, test-pass-crash))' -mlir-pass-pipeline-crash-reproducer=%t -verify-diagnostics -mlir-pass-pipeline-local-reproducer -mlir-disable-threading
diff --git a/mlir/test/Target/LLVMIR/Import/fastmath.ll b/mlir/test/Target/LLVMIR/Import/fastmath.ll
index c54677505bd79..9b30c3218b810 100644
--- a/mlir/test/Target/LLVMIR/Import/fastmath.ll
+++ b/mlir/test/Target/LLVMIR/Import/fastmath.ll
@@ -1,7 +1,7 @@
 ; RUN: mlir-translate -import-llvm -split-input-file %s | FileCheck %s
 
 ; CHECK-LABEL: @fastmath_inst
-define void @fastmath_inst(float %arg1, float %arg2) {
+define void @fastmath_inst(float %arg1, float %arg2, i1 %arg3) {
   ; CHECK: llvm.fadd %{{.*}}, %{{.*}}  {fastmathFlags = #llvm.fastmath<nnan, ninf>} : f32
   %1 = fadd nnan ninf float %arg1, %arg2
   ; CHECK: llvm.fsub %{{.*}}, %{{.*}}  {fastmathFlags = #llvm.fastmath<nsz>} : f32
@@ -12,6 +12,8 @@ define void @fastmath_inst(float %arg1, float %arg2) {
   %4 = fdiv afn reassoc float %arg1, %arg2
   ; CHECK: llvm.fneg %{{.*}}  {fastmathFlags = #llvm.fastmath<fast>} : f32
   %5 = fneg fast float %arg1
+  ; CHECK: llvm.select %{{.*}}, %{{.*}}, %{{.*}} {fastmathFlags = #llvm.fastmath<contract>} : i1, f32
+  %6 = select contract i1 %arg3, float %arg1, float %arg2
   ret void
 }
 
@@ -41,9 +43,11 @@ declare float @llvm.exp.f32(float)
 declare float @llvm.powi.f32.i32(float, i32)
 declare float @llvm.pow.f32(float, float)
 declare float @llvm.fmuladd.f32(float, float, float)
+declare float @llvm.vector.reduce.fmin.v2f32(<2 x float>)
+declare float @llvm.vector.reduce.fmax.v2f32(<2 x float>)
 
 ; CHECK-LABEL: @fastmath_intr
-define void @fastmath_intr(float %arg1, i32 %arg2) {
+define void @fastmath_intr(float %arg1, i32 %arg2, <2 x float> %arg3) {
   ; CHECK: llvm.intr.exp(%{{.*}}) {fastmathFlags = #llvm.fastmath<nnan, ninf>} : (f32) -> f32
   %1 = call nnan ninf float @llvm.exp.f32(float %arg1)
   ; CHECK: llvm.intr.powi(%{{.*}}, %{{.*}}) {fastmathFlags = #llvm.fastmath<fast>} : (f32, i32) -> f32
@@ -52,5 +56,10 @@ define void @fastmath_intr(float %arg1, i32 %arg2) {
   %3 = call fast float @llvm.pow.f32(float %arg1, float %arg1)
   ; CHECK: llvm.intr.fmuladd(%{{.*}}, %{{.*}}, %{{.*}}) {fastmathFlags = #llvm.fastmath<fast>} : (f32, f32, f32) -> f32
   %4 = call fast float @llvm.fmuladd.f32(float %arg1, float %arg1, float %arg1)
+  ; CHECK: %{{.*}} = llvm.intr.vector.reduce.fmin({{.*}}) {fastmathFlags = #llvm.fastmath<nnan>} : (vector<2xf32>) -> f32
+  %5 = call nnan float @llvm.vector.reduce.fmin.v2f32(<2 x float> %arg3)
+  ; CHECK: %{{.*}} = llvm.intr.vector.reduce.fmax({{.*}}) {fastmathFlags = #llvm.fastmath<nnan>} : (vector<2xf32>) -> f32
+  %6 = call nnan float @llvm.vector.reduce.fmax.v2f32(<2 x float> %arg3)
+
   ret void
 }
diff --git a/mlir/test/Target/LLVMIR/Import/function-attributes.ll b/mlir/test/Target/LLVMIR/Import/function-attributes.ll
index e70dc7622c76d..e2696ad951b74 100644
--- a/mlir/test/Target/LLVMIR/Import/function-attributes.ll
+++ b/mlir/test/Target/LLVMIR/Import/function-attributes.ll
@@ -179,3 +179,17 @@ define void @passthrough_combined() alignstack(16) noinline "probe-stack" "alloc
 define void @passthrough_string_only() "no-enum-attr" {
   ret void
 }
+
+// -----
+
+; CHECK-LABEL: llvm.func hidden @hidden()
+define hidden void @hidden() {
+  ret void
+}
+
+// -----
+
+; CHECK-LABEL: llvm.func protected @protected()
+define protected void @protected() {
+  ret void
+}
diff --git a/mlir/test/Target/LLVMIR/Import/global-variables.ll b/mlir/test/Target/LLVMIR/Import/global-variables.ll
index 3b8be65eacd47..81aa4a0f152fd 100644
--- a/mlir/test/Target/LLVMIR/Import/global-variables.ll
+++ b/mlir/test/Target/LLVMIR/Import/global-variables.ll
@@ -196,3 +196,13 @@ define void @foo() {
 define void @bar() {
   ret void
 }
+
+; // -----
+
+; Visibility attribute.
+
+; CHECK: llvm.mlir.global external hidden constant @hidden("string")
+@hidden = hidden constant [6 x i8] c"string"
+
+; CHECK: llvm.mlir.global external protected constant @protected(42 : i64)
+@protected = protected constant i64 42
diff --git a/mlir/test/Target/LLVMIR/Import/instructions.ll b/mlir/test/Target/LLVMIR/Import/instructions.ll
index 47076a7abb018..cbdb0ebe295ef 100644
--- a/mlir/test/Target/LLVMIR/Import/instructions.ll
+++ b/mlir/test/Target/LLVMIR/Import/instructions.ll
@@ -341,9 +341,11 @@ define ptr @alloca(i64 %size) {
   ; CHECK:  llvm.alloca %[[C1]] x f64 {alignment = 8 : i64} : (i32) -> !llvm.ptr
   ; CHECK:  llvm.alloca %[[SIZE]] x i32 {alignment = 8 : i64} : (i64) -> !llvm.ptr
   ; CHECK:  llvm.alloca %[[SIZE]] x i32 {alignment = 4 : i64} : (i64) -> !llvm.ptr<3>
+  ; CHECK:  llvm.alloca inalloca %[[SIZE]] x i32 {alignment = 4 : i64} : (i64) -> !llvm.ptr
   %1 = alloca double
   %2 = alloca i32, i64 %size, align 8
   %3 = alloca i32, i64 %size, addrspace(3)
+  %4 = alloca inalloca i32, i64 %size
   ret ptr %1
 }
 
diff --git a/mlir/test/Target/LLVMIR/Import/intrinsic.ll b/mlir/test/Target/LLVMIR/Import/intrinsic.ll
index 987fc0c63e006..7caca3cd55aed 100644
--- a/mlir/test/Target/LLVMIR/Import/intrinsic.ll
+++ b/mlir/test/Target/LLVMIR/Import/intrinsic.ll
@@ -231,9 +231,9 @@ define void @vector_reductions(float %0, <8 x float> %1, <8 x i32> %2) {
   %4 = call i32 @llvm.vector.reduce.add.v8i32(<8 x i32> %2)
   ; CHECK: "llvm.intr.vector.reduce.and"(%{{.*}}) : (vector<8xi32>) -> i32
   %5 = call i32 @llvm.vector.reduce.and.v8i32(<8 x i32> %2)
-  ; CHECK: "llvm.intr.vector.reduce.fmax"(%{{.*}}) : (vector<8xf32>) -> f32
+  ; CHECK: llvm.intr.vector.reduce.fmax(%{{.*}}) : (vector<8xf32>) -> f32
   %6 = call float @llvm.vector.reduce.fmax.v8f32(<8 x float> %1)
-  ; CHECK: "llvm.intr.vector.reduce.fmin"(%{{.*}}) : (vector<8xf32>) -> f32
+  ; CHECK: llvm.intr.vector.reduce.fmin(%{{.*}}) : (vector<8xf32>) -> f32
   %7 = call float @llvm.vector.reduce.fmin.v8f32(<8 x float> %1)
   ; CHECK: "llvm.intr.vector.reduce.mul"(%{{.*}}) : (vector<8xi32>) -> i32
   %8 = call i32 @llvm.vector.reduce.mul.v8i32(<8 x i32> %2)
diff --git a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
index 681ba0c2e5485..c01729a401d99 100644
--- a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
@@ -251,9 +251,9 @@ llvm.func @vector_reductions(%arg0: f32, %arg1: vector<8xf32>, %arg2: vector<8xi
   // CHECK: call i32 @llvm.vector.reduce.and.v8i32
   "llvm.intr.vector.reduce.and"(%arg2) : (vector<8xi32>) -> i32
   // CHECK: call float @llvm.vector.reduce.fmax.v8f32
-  "llvm.intr.vector.reduce.fmax"(%arg1) : (vector<8xf32>) -> f32
+  llvm.intr.vector.reduce.fmax(%arg1) : (vector<8xf32>) -> f32
   // CHECK: call float @llvm.vector.reduce.fmin.v8f32
-  "llvm.intr.vector.reduce.fmin"(%arg1) : (vector<8xf32>) -> f32
+  llvm.intr.vector.reduce.fmin(%arg1) : (vector<8xf32>) -> f32
   // CHECK: call i32 @llvm.vector.reduce.mul.v8i32
   "llvm.intr.vector.reduce.mul"(%arg2) : (vector<8xi32>) -> i32
   // CHECK: call i32 @llvm.vector.reduce.or.v8i32
@@ -764,15 +764,15 @@ llvm.func @lifetime(%p: !llvm.ptr) {
 // CHECK-DAG: declare void @llvm.memcpy.p0.p0.i32(ptr noalias nocapture writeonly, ptr noalias nocapture readonly, i32, i1 immarg)
 // CHECK-DAG: declare void @llvm.memcpy.inline.p0.p0.i64(ptr noalias nocapture writeonly, ptr noalias nocapture readonly, i64 immarg, i1 immarg)
 // CHECK-DAG: declare { i32, i1 } @llvm.sadd.with.overflow.i32(i32, i32)
-// CHECK-DAG: declare { <8 x i32>, <8 x i1> } @llvm.sadd.with.overflow.v8i32(<8 x i32>, <8 x i32>) #0
+// CHECK-DAG: declare { <8 x i32>, <8 x i1> } @llvm.sadd.with.overflow.v8i32(<8 x i32>, <8 x i32>)
 // CHECK-DAG: declare { i32, i1 } @llvm.uadd.with.overflow.i32(i32, i32)
-// CHECK-DAG: declare { <8 x i32>, <8 x i1> } @llvm.uadd.with.overflow.v8i32(<8 x i32>, <8 x i32>) #0
+// CHECK-DAG: declare { <8 x i32>, <8 x i1> } @llvm.uadd.with.overflow.v8i32(<8 x i32>, <8 x i32>)
 // CHECK-DAG: declare { i32, i1 } @llvm.ssub.with.overflow.i32(i32, i32)
-// CHECK-DAG: declare { <8 x i32>, <8 x i1> } @llvm.ssub.with.overflow.v8i32(<8 x i32>, <8 x i32>) #0
+// CHECK-DAG: declare { <8 x i32>, <8 x i1> } @llvm.ssub.with.overflow.v8i32(<8 x i32>, <8 x i32>)
 // CHECK-DAG: declare { i32, i1 } @llvm.usub.with.overflow.i32(i32, i32)
-// CHECK-DAG: declare { <8 x i32>, <8 x i1> } @llvm.usub.with.overflow.v8i32(<8 x i32>, <8 x i32>) #0
+// CHECK-DAG: declare { <8 x i32>, <8 x i1> } @llvm.usub.with.overflow.v8i32(<8 x i32>, <8 x i32>)
 // CHECK-DAG: declare { i32, i1 } @llvm.umul.with.overflow.i32(i32, i32)
-// CHECK-DAG: declare { <8 x i32>, <8 x i1> } @llvm.umul.with.overflow.v8i32(<8 x i32>, <8 x i32>) #0
+// CHECK-DAG: declare { <8 x i32>, <8 x i1> } @llvm.umul.with.overflow.v8i32(<8 x i32>, <8 x i32>)
 // CHECK-DAG: declare token @llvm.coro.id(i32, ptr readnone, ptr nocapture readonly, ptr)
 // CHECK-DAG: declare ptr @llvm.coro.begin(token, ptr writeonly)
 // CHECK-DAG: declare i64 @llvm.coro.size.i64()
@@ -782,57 +782,57 @@ llvm.func @lifetime(%p: !llvm.ptr) {
 // CHECK-DAG: declare i1 @llvm.coro.end(ptr, i1)
 // CHECK-DAG: declare ptr @llvm.coro.free(token, ptr nocapture readonly)
 // CHECK-DAG: declare void @llvm.coro.resume(ptr)
-// CHECK-DAG: declare <8 x i32> @llvm.vp.add.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x i32> @llvm.vp.sub.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x i32> @llvm.vp.mul.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x i32> @llvm.vp.sdiv.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x i32> @llvm.vp.udiv.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x i32> @llvm.vp.srem.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x i32> @llvm.vp.urem.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x i32> @llvm.vp.ashr.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x i32> @llvm.vp.lshr.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x i32> @llvm.vp.shl.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x i32> @llvm.vp.or.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x i32> @llvm.vp.and.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x i32> @llvm.vp.xor.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x float> @llvm.vp.fadd.v8f32(<8 x float>, <8 x float>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x float> @llvm.vp.fsub.v8f32(<8 x float>, <8 x float>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x float> @llvm.vp.fmul.v8f32(<8 x float>, <8 x float>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x float> @llvm.vp.fdiv.v8f32(<8 x float>, <8 x float>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x float> @llvm.vp.frem.v8f32(<8 x float>, <8 x float>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x float> @llvm.vp.fneg.v8f32(<8 x float>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x float> @llvm.vp.fma.v8f32(<8 x float>, <8 x float>, <8 x float>, <8 x i1>, i32) #2
-// CHECK-DAG: declare i32 @llvm.vp.reduce.add.v8i32(i32, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare i32 @llvm.vp.reduce.mul.v8i32(i32, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare i32 @llvm.vp.reduce.and.v8i32(i32, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare i32 @llvm.vp.reduce.or.v8i32(i32, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare i32 @llvm.vp.reduce.xor.v8i32(i32, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare i32 @llvm.vp.reduce.smax.v8i32(i32, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare i32 @llvm.vp.reduce.smin.v8i32(i32, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare i32 @llvm.vp.reduce.umax.v8i32(i32, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare i32 @llvm.vp.reduce.umin.v8i32(i32, <8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare float @llvm.vp.reduce.fadd.v8f32(float, <8 x float>, <8 x i1>, i32) #2
-// CHECK-DAG: declare float @llvm.vp.reduce.fmul.v8f32(float, <8 x float>, <8 x i1>, i32) #2
-// CHECK-DAG: declare float @llvm.vp.reduce.fmax.v8f32(float, <8 x float>, <8 x i1>, i32) #2
-// CHECK-DAG: declare float @llvm.vp.reduce.fmin.v8f32(float, <8 x float>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x i32> @llvm.vp.select.v8i32(<8 x i1>, <8 x i32>, <8 x i32>, i32) #2
-// CHECK-DAG: declare <8 x i32> @llvm.vp.merge.v8i32(<8 x i1>, <8 x i32>, <8 x i32>, i32) #2
-// CHECK-DAG: declare void @llvm.experimental.vp.strided.store.v8i32.p0.i32(<8 x i32>, ptr nocapture, i32, <8 x i1>, i32) #4
-// CHECK-DAG: declare <8 x i32> @llvm.experimental.vp.strided.load.v8i32.p0.i32(ptr nocapture, i32, <8 x i1>, i32) #3
-// CHECK-DAG: declare <8 x i32> @llvm.vp.trunc.v8i32.v8i64(<8 x i64>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x i64> @llvm.vp.zext.v8i64.v8i32(<8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x i64> @llvm.vp.sext.v8i64.v8i32(<8 x i32>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x float> @llvm.vp.fptrunc.v8f32.v8f64(<8 x double>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x double> @llvm.vp.fpext.v8f64.v8f32(<8 x float>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x i64> @llvm.vp.fptoui.v8i64.v8f64(<8 x double>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x i64> @llvm.vp.fptosi.v8i64.v8f64(<8 x double>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x i64> @llvm.vp.ptrtoint.v8i64.v8p0(<8 x ptr>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <8 x ptr> @llvm.vp.inttoptr.v8p0.v8i64(<8 x i64>, <8 x i1>, i32) #2
-// CHECK-DAG: declare <vscale x 4 x i32> @llvm.vector.insert.nxv4i32.v8i32(<vscale x 4 x i32>, <8 x i32>, i64 immarg) #0
-// CHECK-DAG: declare <vscale x 4 x i32> @llvm.vector.insert.nxv4i32.v4i32(<vscale x 4 x i32>, <4 x i32>, i64 immarg) #0
-// CHECK-DAG: declare <8 x i32> @llvm.vector.insert.v8i32.v4i32(<8 x i32>, <4 x i32>, i64 immarg) #0
-// CHECK-DAG: declare <8 x i32> @llvm.vector.extract.v8i32.nxv4i32(<vscale x 4 x i32>, i64 immarg) #0
-// CHECK-DAG: declare <4 x i32> @llvm.vector.extract.v4i32.nxv4i32(<vscale x 4 x i32>, i64 immarg) #0
-// CHECK-DAG: declare <2 x i32> @llvm.vector.extract.v2i32.v8i32(<8 x i32>, i64 immarg) #0
+// CHECK-DAG: declare <8 x i32> @llvm.vp.add.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i32> @llvm.vp.sub.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i32> @llvm.vp.mul.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i32> @llvm.vp.sdiv.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i32> @llvm.vp.udiv.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i32> @llvm.vp.srem.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i32> @llvm.vp.urem.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i32> @llvm.vp.ashr.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i32> @llvm.vp.lshr.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i32> @llvm.vp.shl.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i32> @llvm.vp.or.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i32> @llvm.vp.and.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i32> @llvm.vp.xor.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x float> @llvm.vp.fadd.v8f32(<8 x float>, <8 x float>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x float> @llvm.vp.fsub.v8f32(<8 x float>, <8 x float>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x float> @llvm.vp.fmul.v8f32(<8 x float>, <8 x float>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x float> @llvm.vp.fdiv.v8f32(<8 x float>, <8 x float>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x float> @llvm.vp.frem.v8f32(<8 x float>, <8 x float>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x float> @llvm.vp.fneg.v8f32(<8 x float>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x float> @llvm.vp.fma.v8f32(<8 x float>, <8 x float>, <8 x float>, <8 x i1>, i32)
+// CHECK-DAG: declare i32 @llvm.vp.reduce.add.v8i32(i32, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare i32 @llvm.vp.reduce.mul.v8i32(i32, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare i32 @llvm.vp.reduce.and.v8i32(i32, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare i32 @llvm.vp.reduce.or.v8i32(i32, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare i32 @llvm.vp.reduce.xor.v8i32(i32, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare i32 @llvm.vp.reduce.smax.v8i32(i32, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare i32 @llvm.vp.reduce.smin.v8i32(i32, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare i32 @llvm.vp.reduce.umax.v8i32(i32, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare i32 @llvm.vp.reduce.umin.v8i32(i32, <8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare float @llvm.vp.reduce.fadd.v8f32(float, <8 x float>, <8 x i1>, i32)
+// CHECK-DAG: declare float @llvm.vp.reduce.fmul.v8f32(float, <8 x float>, <8 x i1>, i32)
+// CHECK-DAG: declare float @llvm.vp.reduce.fmax.v8f32(float, <8 x float>, <8 x i1>, i32)
+// CHECK-DAG: declare float @llvm.vp.reduce.fmin.v8f32(float, <8 x float>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i32> @llvm.vp.select.v8i32(<8 x i1>, <8 x i32>, <8 x i32>, i32)
+// CHECK-DAG: declare <8 x i32> @llvm.vp.merge.v8i32(<8 x i1>, <8 x i32>, <8 x i32>, i32)
+// CHECK-DAG: declare void @llvm.experimental.vp.strided.store.v8i32.p0.i32(<8 x i32>, ptr nocapture, i32, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i32> @llvm.experimental.vp.strided.load.v8i32.p0.i32(ptr nocapture, i32, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i32> @llvm.vp.trunc.v8i32.v8i64(<8 x i64>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i64> @llvm.vp.zext.v8i64.v8i32(<8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i64> @llvm.vp.sext.v8i64.v8i32(<8 x i32>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x float> @llvm.vp.fptrunc.v8f32.v8f64(<8 x double>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x double> @llvm.vp.fpext.v8f64.v8f32(<8 x float>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i64> @llvm.vp.fptoui.v8i64.v8f64(<8 x double>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i64> @llvm.vp.fptosi.v8i64.v8f64(<8 x double>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x i64> @llvm.vp.ptrtoint.v8i64.v8p0(<8 x ptr>, <8 x i1>, i32)
+// CHECK-DAG: declare <8 x ptr> @llvm.vp.inttoptr.v8p0.v8i64(<8 x i64>, <8 x i1>, i32)
+// CHECK-DAG: declare <vscale x 4 x i32> @llvm.vector.insert.nxv4i32.v8i32(<vscale x 4 x i32>, <8 x i32>, i64 immarg)
+// CHECK-DAG: declare <vscale x 4 x i32> @llvm.vector.insert.nxv4i32.v4i32(<vscale x 4 x i32>, <4 x i32>, i64 immarg)
+// CHECK-DAG: declare <8 x i32> @llvm.vector.insert.v8i32.v4i32(<8 x i32>, <4 x i32>, i64 immarg)
+// CHECK-DAG: declare <8 x i32> @llvm.vector.extract.v8i32.nxv4i32(<vscale x 4 x i32>, i64 immarg)
+// CHECK-DAG: declare <4 x i32> @llvm.vector.extract.v4i32.nxv4i32(<vscale x 4 x i32>, i64 immarg)
+// CHECK-DAG: declare <2 x i32> @llvm.vector.extract.v2i32.v8i32(<8 x i32>, i64 immarg)
 // CHECK-DAG: declare void @llvm.lifetime.start.p0(i64 immarg, ptr nocapture)
 // CHECK-DAG: declare void @llvm.lifetime.end.p0(i64 immarg, ptr nocapture)
diff --git a/mlir/test/Target/LLVMIR/llvmir-invalid.mlir b/mlir/test/Target/LLVMIR/llvmir-invalid.mlir
index 19e0b1501e090..a373bd62f2247 100644
--- a/mlir/test/Target/LLVMIR/llvmir-invalid.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-invalid.mlir
@@ -172,7 +172,7 @@ llvm.func @vec_reduce_add_intr_wrong_type(%arg0 : vector<4xi32>) -> f32 {
 
 llvm.func @vec_reduce_fmax_intr_wrong_type(%arg0 : vector<4xi32>) -> i32 {
   // expected-error @below{{op operand #0 must be LLVM dialect-compatible vector of floating-point}}
-  %0 = "llvm.intr.vector.reduce.fmax"(%arg0) : (vector<4xi32>) -> i32
+  %0 = llvm.intr.vector.reduce.fmax(%arg0) : (vector<4xi32>) -> i32
   llvm.return %0 : i32
 }
 
diff --git a/mlir/test/Target/LLVMIR/llvmir.mlir b/mlir/test/Target/LLVMIR/llvmir.mlir
index 01ca6d2bf2ab9..ce65ff995709c 100644
--- a/mlir/test/Target/LLVMIR/llvmir.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir.mlir
@@ -110,6 +110,17 @@ llvm.mlir.global weak_odr @weak_odr(42 : i32) : i32
 // CHECK: @external = external global i32
 llvm.mlir.global external @external() : i32
 
+
+//
+// Visibility attribute.
+//
+
+// CHECK: @hidden = hidden constant [6 x i8] c"string"
+llvm.mlir.global external hidden constant @hidden("string")
+
+// CHECK: @protected = protected constant i64 42
+llvm.mlir.global external protected constant @protected(42 : i64) : i64
+
 //
 // UnnamedAddr attribute.
 //
@@ -484,6 +495,20 @@ llvm.func internal @func_internal() {
   llvm.return
 }
 
+//
+// Visibility attribute.
+//
+
+// CHECK-LABEL: define hidden void @hidden_func()
+llvm.func hidden @hidden_func() {
+  llvm.return
+}
+
+// CHECK-LABEL: define protected void @protected_func()
+llvm.func protected @protected_func() {
+  llvm.return
+}
+
 //
 // dso_local attribute.
 //
@@ -944,6 +969,11 @@ llvm.func @vector_splat_nonzero_scalable() -> vector<[4]xf32> {
   llvm.return %0 : vector<[4]xf32>
 }
 
+// CHECK-LABEL: @f8_ptrs(ptr {{%.*}}, ptr {{%.*}})
+llvm.func @f8_ptrs(%arg0: !llvm.ptr<f8E5M2>, %arg1: !llvm.ptr<f8E4M3FN>) {
+  llvm.return
+}
+
 // CHECK-LABEL: @ops
 llvm.func @ops(%arg0: f32, %arg1: f32, %arg2: i32, %arg3: i32) -> !llvm.struct<(f32, i32)> {
 // CHECK-NEXT: fsub float %0, %1
@@ -1353,6 +1383,8 @@ llvm.func @alloca(%size : i64) {
   llvm.alloca %size x i32 {alignment = 8} : (i64) -> (!llvm.ptr<i32>)
   // CHECK-NEXT: alloca {{.*}} addrspace(3)
   llvm.alloca %size x i32 {alignment = 0} : (i64) -> (!llvm.ptr<i32, 3>)
+  // CHECK-NEXT: alloca inalloca {{.*}} align 4
+  llvm.alloca inalloca %size x i32 : (i64) -> !llvm.ptr
   llvm.return
 }
 
@@ -1864,7 +1896,7 @@ llvm.func @useInlineAsm(%arg0: i32) {
 llvm.func @fastmathFlagsFunc(f32) -> f32
 
 // CHECK-LABEL: @fastmathFlags
-llvm.func @fastmathFlags(%arg0: f32) {
+llvm.func @fastmathFlags(%arg0: f32, %arg1 : vector<2xf32>) {
 // CHECK: {{.*}} = fadd nnan ninf float {{.*}}, {{.*}}
 // CHECK: {{.*}} = fsub nnan ninf float {{.*}}, {{.*}}
 // CHECK: {{.*}} = fmul nnan ninf float {{.*}}, {{.*}}
@@ -1911,6 +1943,16 @@ llvm.func @fastmathFlags(%arg0: f32) {
   %19 = "llvm.intr.powi"(%arg0, %exp) {fastmathFlags = #llvm.fastmath<fast>} : (f32, i32) -> f32
 // CHECK: call afn float @llvm.powi.f32.i32(float {{.*}}, i32 {{.*}})
   %20 = "llvm.intr.powi"(%arg0, %exp) {fastmathFlags = #llvm.fastmath<afn>} : (f32, i32) -> f32
+
+// CHECK: call nnan float @llvm.vector.reduce.fmax.v2f32(<2 x float> {{.*}})
+// CHECK: call nnan float @llvm.vector.reduce.fmin.v2f32(<2 x float> {{.*}})
+  %21 = llvm.intr.vector.reduce.fmax(%arg1) {fastmathFlags = #llvm.fastmath<nnan>} : (vector<2xf32>) -> f32
+  %22 = llvm.intr.vector.reduce.fmin(%arg1) {fastmathFlags = #llvm.fastmath<nnan>} : (vector<2xf32>) -> f32
+
+
+  %23 = llvm.mlir.constant(true) : i1
+// CHECK: select contract i1
+  %24 = llvm.select %23, %arg0, %20 {fastmathFlags = #llvm.fastmath<contract>} : i1, f32
   llvm.return
 }
 
diff --git a/mlir/test/Transforms/inlining-dump-default-pipeline.mlir b/mlir/test/Transforms/inlining-dump-default-pipeline.mlir
new file mode 100644
index 0000000000000..e2c31867a8e04
--- /dev/null
+++ b/mlir/test/Transforms/inlining-dump-default-pipeline.mlir
@@ -0,0 +1,2 @@
+// RUN: mlir-opt %s -pass-pipeline="builtin.module(inline)" -dump-pass-pipeline 2>&1 | FileCheck %s
+// CHECK: builtin.module(inline{default-pipeline=canonicalize max-iterations=4 })
diff --git a/mlir/test/Transforms/memref-bound-check.mlir b/mlir/test/Transforms/memref-bound-check.mlir
index 5c707e6900a48..fce6bdbca4aa1 100644
--- a/mlir/test/Transforms/memref-bound-check.mlir
+++ b/mlir/test/Transforms/memref-bound-check.mlir
@@ -293,3 +293,14 @@ func.func @zero_d_memref() {
   }
   return
 }
+
+// CHECK-LABEL: func @affine_parallel
+func.func @affine_parallel(%M: memref<2048x2048xf64>) {
+  affine.parallel (%i) = (0) to (3000) {
+    affine.for %j = 0 to 2048 {
+      affine.load %M[%i, %j] : memref<2048x2048xf64>
+      // expected-error@above {{'affine.load' op memref out of upper bound access along dimension #1}}
+    }
+  }
+  return
+}
diff --git a/mlir/test/lib/Analysis/DataFlow/TestDenseDataFlowAnalysis.cpp b/mlir/test/lib/Analysis/DataFlow/TestDenseDataFlowAnalysis.cpp
index f88544032c21a..e9769c10e63fe 100644
--- a/mlir/test/lib/Analysis/DataFlow/TestDenseDataFlowAnalysis.cpp
+++ b/mlir/test/lib/Analysis/DataFlow/TestDenseDataFlowAnalysis.cpp
@@ -254,9 +254,9 @@ struct TestLastModifiedPass
       const LastModification *lastMods =
           solver.lookupState<LastModification>(op);
       assert(lastMods && "expected a dense lattice");
-      for (auto &it : llvm::enumerate(op->getOperands())) {
-        os << " operand #" << it.index() << "\n";
-        Value value = getMostUnderlyingValue(it.value(), [&](Value value) {
+      for (auto [index, operand] : llvm::enumerate(op->getOperands())) {
+        os << " operand #" << index << "\n";
+        Value value = getMostUnderlyingValue(operand, [&](Value value) {
           return solver.lookupState<UnderlyingValueLattice>(value);
         });
         assert(value && "expected an underlying value");
diff --git a/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp b/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp
index 7c384e1a48ccd..7ddcfdde06c0e 100644
--- a/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp
+++ b/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp
@@ -81,10 +81,9 @@ static void checkDependences(ArrayRef<Operation *> loadsAndStores) {
       unsigned numCommonLoops =
           getNumCommonSurroundingLoops(*srcOpInst, *dstOpInst);
       for (unsigned d = 1; d <= numCommonLoops + 1; ++d) {
-        FlatAffineValueConstraints dependenceConstraints;
         SmallVector<DependenceComponent, 2> dependenceComponents;
         DependenceResult result = checkMemrefAccessDependence(
-            srcAccess, dstAccess, d, &dependenceConstraints,
+            srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &dependenceComponents);
         if (result.value == DependenceResult::Failure) {
           srcOpInst->emitError("dependence check failed");
diff --git a/mlir/test/lib/Conversion/CMakeLists.txt b/mlir/test/lib/Conversion/CMakeLists.txt
index bc6103dedd490..14f0e0dbe1802 100644
--- a/mlir/test/lib/Conversion/CMakeLists.txt
+++ b/mlir/test/lib/Conversion/CMakeLists.txt
@@ -1 +1,2 @@
 add_subdirectory(FuncToLLVM)
+add_subdirectory(VectorToSPIRV)
diff --git a/mlir/test/lib/Conversion/VectorToSPIRV/CMakeLists.txt b/mlir/test/lib/Conversion/VectorToSPIRV/CMakeLists.txt
new file mode 100644
index 0000000000000..09ed283ac97bc
--- /dev/null
+++ b/mlir/test/lib/Conversion/VectorToSPIRV/CMakeLists.txt
@@ -0,0 +1,15 @@
+# Exclude tests from libMLIR.so
+add_mlir_library(MLIRTestVectorToSPIRV
+  TestVectorReductionToSPIRVDotProd.cpp
+
+  EXCLUDE_FROM_LIBMLIR
+
+  LINK_LIBS PUBLIC
+  MLIRVectorToSPIRV
+  MLIRArithDialect
+  MLIRFuncDialect
+  MLIRSPIRVDialect
+  MLIRVectorDialect
+  MLIRPass
+  MLIRTransforms
+  )
diff --git a/mlir/test/lib/Conversion/VectorToSPIRV/TestVectorReductionToSPIRVDotProd.cpp b/mlir/test/lib/Conversion/VectorToSPIRV/TestVectorReductionToSPIRVDotProd.cpp
new file mode 100644
index 0000000000000..1864d2f7f5036
--- /dev/null
+++ b/mlir/test/lib/Conversion/VectorToSPIRV/TestVectorReductionToSPIRVDotProd.cpp
@@ -0,0 +1,55 @@
+//===- TestVectorReductionToSPIRVDotProd.cpp - Test reduction to dot prod -===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Conversion/VectorToSPIRV/VectorToSPIRV.h"
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/SPIRV/IR/SPIRVDialect.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Pass/Pass.h"
+#include "mlir/Pass/PassManager.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+
+namespace mlir {
+namespace {
+
+struct TestVectorReductionToSPIRVDotProd
+    : PassWrapper<TestVectorReductionToSPIRVDotProd,
+                  OperationPass<func::FuncOp>> {
+  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(
+      TestVectorReductionToSPIRVDotProd)
+
+  StringRef getArgument() const final {
+    return "test-vector-reduction-to-spirv-dot-prod";
+  }
+
+  StringRef getDescription() const final {
+    return "Test lowering patterns that converts vector.reduction to SPIR-V "
+           "integer dot product ops";
+  }
+
+  void getDependentDialects(DialectRegistry &registry) const override {
+    registry.insert<arith::ArithDialect, func::FuncDialect, spirv::SPIRVDialect,
+                    vector::VectorDialect>();
+  }
+
+  void runOnOperation() override {
+    RewritePatternSet patterns(&getContext());
+    populateVectorReductionToSPIRVDotProductPatterns(patterns);
+    (void)applyPatternsAndFoldGreedily(getOperation(), std::move(patterns));
+  }
+};
+
+} // namespace
+
+namespace test {
+void registerTestVectorReductionToSPIRVDotProd() {
+  PassRegistration<TestVectorReductionToSPIRVDotProd>();
+}
+} // namespace test
+} // namespace mlir
diff --git a/mlir/test/lib/Dialect/Affine/CMakeLists.txt b/mlir/test/lib/Dialect/Affine/CMakeLists.txt
index 4044573b7c6e9..8fe9113f69848 100644
--- a/mlir/test/lib/Dialect/Affine/CMakeLists.txt
+++ b/mlir/test/lib/Dialect/Affine/CMakeLists.txt
@@ -3,6 +3,7 @@ add_mlir_library(MLIRAffineTransformsTestPasses
   TestAffineDataCopy.cpp
   TestAffineLoopUnswitching.cpp
   TestAffineLoopParametricTiling.cpp
+  TestDecomposeAffineOps.cpp
   TestLoopFusion.cpp
   TestLoopMapping.cpp
   TestLoopPermutation.cpp
diff --git a/mlir/test/lib/Dialect/Affine/TestAffineDataCopy.cpp b/mlir/test/lib/Dialect/Affine/TestAffineDataCopy.cpp
index 62671dde34b0e..c602095bbfeed 100644
--- a/mlir/test/lib/Dialect/Affine/TestAffineDataCopy.cpp
+++ b/mlir/test/lib/Dialect/Affine/TestAffineDataCopy.cpp
@@ -60,7 +60,8 @@ void TestAffineDataCopy::runOnOperation() {
   // Gather all AffineForOps by loop depth.
   std::vector<SmallVector<AffineForOp, 2>> depthToLoops;
   gatherLoops(getOperation(), depthToLoops);
-  assert(!depthToLoops.empty() && "Loop nest not found");
+  if (depthToLoops.empty())
+    return;
 
   // Only support tests with a single loop nest and a single innermost loop
   // for now.
diff --git a/mlir/test/lib/Dialect/Affine/TestDecomposeAffineOps.cpp b/mlir/test/lib/Dialect/Affine/TestDecomposeAffineOps.cpp
new file mode 100644
index 0000000000000..bdb494d83a474
--- /dev/null
+++ b/mlir/test/lib/Dialect/Affine/TestDecomposeAffineOps.cpp
@@ -0,0 +1,57 @@
+//===- TestDecomposeAffineOps.cpp - Test affine ops decomposition utility -===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements a pass to test affine data copy utility functions and
+// options.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Affine/IR/AffineOps.h"
+#include "mlir/Dialect/Affine/Transforms/Transforms.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/Passes.h"
+
+#define PASS_NAME "test-decompose-affine-ops"
+
+using namespace mlir;
+
+namespace {
+
+struct TestDecomposeAffineOps
+    : public PassWrapper<TestDecomposeAffineOps, OperationPass<func::FuncOp>> {
+  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(TestDecomposeAffineOps)
+
+  StringRef getArgument() const final { return PASS_NAME; }
+  StringRef getDescription() const final {
+    return "Tests affine ops decomposition utility functions.";
+  }
+  TestDecomposeAffineOps() = default;
+  TestDecomposeAffineOps(const TestDecomposeAffineOps &pass)
+      : PassWrapper(pass){};
+
+  void runOnOperation() override;
+};
+
+} // namespace
+
+void TestDecomposeAffineOps::runOnOperation() {
+  IRRewriter rewriter(&getContext());
+  this->getOperation().walk([&](AffineApplyOp op) {
+    rewriter.setInsertionPoint(op);
+    reorderOperandsByHoistability(rewriter, op);
+    (void)decompose(rewriter, op);
+  });
+}
+
+namespace mlir {
+void registerTestDecomposeAffineOpPass() {
+  PassRegistration<TestDecomposeAffineOps>();
+}
+} // namespace mlir
diff --git a/mlir/test/lib/Dialect/SCF/TestSCFUtils.cpp b/mlir/test/lib/Dialect/SCF/TestSCFUtils.cpp
index 3fb82acc78228..f35e774c9704f 100644
--- a/mlir/test/lib/Dialect/SCF/TestSCFUtils.cpp
+++ b/mlir/test/lib/Dialect/SCF/TestSCFUtils.cpp
@@ -150,8 +150,8 @@ struct TestSCFPipeliningPass
 
   /// Helper to generate "predicated" version of `op`. For simplicity we just
   /// wrap the operation in a scf.ifOp operation.
-  static Operation *predicateOp(Operation *op, Value pred,
-                                PatternRewriter &rewriter) {
+  static Operation *predicateOp(RewriterBase &rewriter, Operation *op,
+                                Value pred) {
     Location loc = op->getLoc();
     auto ifOp =
         rewriter.create<scf::IfOp>(loc, op->getResultTypes(), pred, true);
diff --git a/mlir/test/lib/Dialect/Tensor/TestTensorTransforms.cpp b/mlir/test/lib/Dialect/Tensor/TestTensorTransforms.cpp
index be375e47dcf5c..38988923fef67 100644
--- a/mlir/test/lib/Dialect/Tensor/TestTensorTransforms.cpp
+++ b/mlir/test/lib/Dialect/Tensor/TestTensorTransforms.cpp
@@ -188,7 +188,7 @@ struct RewriteExtractSliceFromCollapseShapeBase
 
     // Materialize the output shape values of the slice operation.
     ReifiedRankedShapedTypeDims reifiedShapes;
-    if (failed(op.reifyResultShapes(rewriter, reifiedShapes)))
+    if (failed(reifyResultShapes(rewriter, op, reifiedShapes)))
       return rewriter.notifyMatchFailure(op, "failed to reify result shapes");
 
     // Create the destination tensor using the above values.
diff --git a/mlir/test/lib/Dialect/Test/TestDialect.cpp b/mlir/test/lib/Dialect/Test/TestDialect.cpp
index 5bafedee2d75d..97c77b0eb4898 100644
--- a/mlir/test/lib/Dialect/Test/TestDialect.cpp
+++ b/mlir/test/lib/Dialect/Test/TestDialect.cpp
@@ -10,15 +10,14 @@
 #include "TestAttributes.h"
 #include "TestInterfaces.h"
 #include "TestTypes.h"
+#include "mlir/Bytecode/BytecodeImplementation.h"
 #include "mlir/Dialect/Arith/IR/Arith.h"
-#include "mlir/Dialect/DLTI/DLTI.h"
 #include "mlir/Dialect/Func/IR/FuncOps.h"
 #include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/IR/AsmState.h"
 #include "mlir/IR/BuiltinAttributes.h"
 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/Diagnostics.h"
-#include "mlir/IR/DialectImplementation.h"
 #include "mlir/IR/ExtensibleDialect.h"
 #include "mlir/IR/MLIRContext.h"
 #include "mlir/IR/OperationSupport.h"
@@ -32,9 +31,9 @@
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringSwitch.h"
-#include <optional>
 
 #include <numeric>
+#include <optional>
 
 // Include this before the using namespace lines below to
 // test that we don't have namespace dependencies.
@@ -47,6 +46,15 @@ void test::registerTestDialect(DialectRegistry &registry) {
   registry.insert<TestDialect>();
 }
 
+//===----------------------------------------------------------------------===//
+// TestDialect version utilities
+//===----------------------------------------------------------------------===//
+
+struct TestDialectVersion : public DialectVersion {
+  uint32_t major = 2;
+  uint32_t minor = 0;
+};
+
 //===----------------------------------------------------------------------===//
 // TestDialect Interfaces
 //===----------------------------------------------------------------------===//
@@ -70,6 +78,107 @@ struct TestResourceBlobManagerInterface
       TestDialectResourceBlobHandle>::ResourceBlobManagerDialectInterfaceBase;
 };
 
+namespace {
+enum test_encoding { k_attr_params = 0 };
+}
+
+// Test support for interacting with the Bytecode reader/writer.
+struct TestBytecodeDialectInterface : public BytecodeDialectInterface {
+  using BytecodeDialectInterface::BytecodeDialectInterface;
+  TestBytecodeDialectInterface(Dialect *dialect)
+      : BytecodeDialectInterface(dialect) {}
+
+  LogicalResult writeAttribute(Attribute attr,
+                               DialectBytecodeWriter &writer) const final {
+    if (auto concreteAttr = llvm::dyn_cast<TestAttrParamsAttr>(attr)) {
+      writer.writeVarInt(test_encoding::k_attr_params);
+      writer.writeVarInt(concreteAttr.getV0());
+      writer.writeVarInt(concreteAttr.getV1());
+      return success();
+    }
+    writer.writeAttribute(attr);
+    return success();
+  }
+
+  Attribute readAttribute(DialectBytecodeReader &reader,
+                          const DialectVersion &version_) const final {
+    const auto &version = static_cast<const TestDialectVersion &>(version_);
+    if (version.major < 2)
+      return readAttrOldEncoding(reader);
+    if (version.major == 2 && version.minor == 0)
+      return readAttrNewEncoding(reader);
+    // Forbid reading future versions by returning nullptr.
+    return Attribute();
+  }
+
+  // Emit a specific version of the dialect.
+  void writeVersion(DialectBytecodeWriter &writer) const final {
+    auto version = TestDialectVersion();
+    writer.writeVarInt(version.major); // major
+    writer.writeVarInt(version.minor); // minor
+  }
+
+  std::unique_ptr<DialectVersion>
+  readVersion(DialectBytecodeReader &reader) const final {
+    uint64_t major, minor;
+    if (failed(reader.readVarInt(major)) || failed(reader.readVarInt(minor)))
+      return nullptr;
+    auto version = std::make_unique<TestDialectVersion>();
+    version->major = major;
+    version->minor = minor;
+    return version;
+  }
+
+  LogicalResult upgradeFromVersion(Operation *topLevelOp,
+                                   const DialectVersion &version_) const final {
+    const auto &version = static_cast<const TestDialectVersion &>(version_);
+    if ((version.major == 2) && (version.minor == 0))
+      return success();
+    if (version.major > 2 || (version.major == 2 && version.minor > 0)) {
+      return topLevelOp->emitError()
+             << "current test dialect version is 2.0, can't parse version: "
+             << version.major << "." << version.minor;
+    }
+    // Prior version 2.0, the old op supported only a single attribute called
+    // "dimensions". We can perform the upgrade.
+    topLevelOp->walk([](TestVersionedOpA op) {
+      if (auto dims = op->getAttr("dimensions")) {
+        op->removeAttr("dimensions");
+        op->setAttr("dims", dims);
+      }
+      op->setAttr("modifier", BoolAttr::get(op->getContext(), false));
+    });
+    return success();
+  }
+
+private:
+  Attribute readAttrNewEncoding(DialectBytecodeReader &reader) const {
+    uint64_t encoding;
+    if (failed(reader.readVarInt(encoding)) ||
+        encoding != test_encoding::k_attr_params)
+      return Attribute();
+    // The new encoding has v0 first, v1 second.
+    uint64_t v0, v1;
+    if (failed(reader.readVarInt(v0)) || failed(reader.readVarInt(v1)))
+      return Attribute();
+    return TestAttrParamsAttr::get(getContext(), static_cast<int>(v0),
+                                   static_cast<int>(v1));
+  }
+
+  Attribute readAttrOldEncoding(DialectBytecodeReader &reader) const {
+    uint64_t encoding;
+    if (failed(reader.readVarInt(encoding)) ||
+        encoding != test_encoding::k_attr_params)
+      return Attribute();
+    // The old encoding has v1 first, v0 second.
+    uint64_t v0, v1;
+    if (failed(reader.readVarInt(v1)) || failed(reader.readVarInt(v0)))
+      return Attribute();
+    return TestAttrParamsAttr::get(getContext(), static_cast<int>(v0),
+                                   static_cast<int>(v1));
+  }
+};
+
 // Test support for interacting with the AsmPrinter.
 struct TestOpAsmInterface : public OpAsmDialectInterface {
   using OpAsmDialectInterface::OpAsmDialectInterface;
@@ -367,7 +476,7 @@ void TestDialect::initialize() {
   addInterface<TestOpAsmInterface>(blobInterface);
 
   addInterfaces<TestDialectFoldInterface, TestInlinerInterface,
-                TestReductionPatternInterface>();
+                TestReductionPatternInterface, TestBytecodeDialectInterface>();
   allowUnknownOperations();
 
   // Instantiate our fallback op interface that we'll use on specific
@@ -1009,10 +1118,10 @@ void PrettyPrintedRegionOp::print(OpAsmPrinter &p) {
   // Assuming that region has a single non-terminator inner-op, if the inner-op
   // meets some criteria (which in this case is a simple one  based on the name
   // of inner-op), then we can print the entire region in a succinct way.
-  // Here we assume that the prototype of "special.op" can be trivially derived
+  // Here we assume that the prototype of "test.special.op" can be trivially derived
   // while parsing it back.
-  if (innerOp.getName().getStringRef().equals("special.op")) {
-    p << " start special.op end";
+  if (innerOp.getName().getStringRef().equals("test.special.op")) {
+    p << " start test.special.op end";
   } else {
     p << " (";
     p.printRegion(getRegion());
@@ -1103,9 +1212,7 @@ OpFoldResult TestOpWithRegionFold::fold(FoldAdaptor adaptor) {
   return getOperand();
 }
 
-OpFoldResult TestOpConstant::fold(FoldAdaptor adaptor) {
-  return getValue();
-}
+OpFoldResult TestOpConstant::fold(FoldAdaptor adaptor) { return getValue(); }
 
 LogicalResult TestOpWithVariadicResultsAndFolder::fold(
     FoldAdaptor adaptor, SmallVectorImpl<OpFoldResult> &results) {
@@ -1241,11 +1348,16 @@ LogicalResult OpWithResultShapePerDimInterfaceOp::reifyResultShapes(
   Location loc = getLoc();
   shapes.reserve(getNumOperands());
   for (Value operand : llvm::reverse(getOperands())) {
+    auto tensorType = operand.getType().cast<RankedTensorType>();
     auto currShape = llvm::to_vector<4>(llvm::map_range(
-        llvm::seq<int64_t>(
-            0, operand.getType().cast<RankedTensorType>().getRank()),
+        llvm::seq<int64_t>(0, tensorType.getRank()),
         [&](int64_t dim) -> OpFoldResult {
-          return builder.createOrFold<tensor::DimOp>(loc, operand, dim);
+          return tensorType.isDynamicDim(dim)
+                     ? static_cast<OpFoldResult>(
+                           builder.createOrFold<tensor::DimOp>(loc, operand,
+                                                               dim))
+                     : static_cast<OpFoldResult>(
+                           builder.getIndexAttr(tensorType.getDimSize(dim)));
         }));
     shapes.emplace_back(std::move(currShape));
   }
diff --git a/mlir/test/lib/Dialect/Test/TestOps.td b/mlir/test/lib/Dialect/Test/TestOps.td
index b4447eb8cc142..3f642b8a87ea2 100644
--- a/mlir/test/lib/Dialect/Test/TestOps.td
+++ b/mlir/test/lib/Dialect/Test/TestOps.td
@@ -3149,4 +3149,43 @@ def TestCSEOfSingleBlockOp : TEST_Op<"cse_of_single_block_op",
   }];
 }
 
+//===----------------------------------------------------------------------===//
+// Test Ops to upgrade base on the dialect versions
+//===----------------------------------------------------------------------===//
+
+def TestVersionedOpA : TEST_Op<"versionedA"> {
+  // A previous version of the dialect (let's say 1.*) supported an attribute
+  // named "dimensions":
+  // let arguments = (ins
+  //   AnyI64Attr:$dimensions
+  // );
+
+  // In the current version (2.0) "dimensions" was renamed to "dims", and a new
+  // boolean attribute "modifier" was added. The previous version of the op
+  // corresponds to "modifier=false". We support loading old IR through
+  // upgrading, see `upgradeFromVersion()` in `TestBytecodeDialectInterface`.
+  let arguments = (ins
+   AnyI64Attr:$dims,
+   BoolAttr:$modifier
+  );
+}
+
+def TestVersionedOpB : TEST_Op<"versionedB"> {
+  // A previous version of the dialect (let's say 1.*) we encoded TestAttrParams
+  // with a custom encoding:
+  //
+  //    #test.attr_params<X, Y> -> { varInt: Y, varInt: X }
+  //
+  // In the current version (2.0) the encoding changed and the two parameters of
+  // the attribute are swapped:
+  //
+  //    #test.attr_params<X, Y> -> { varInt: X, varInt: Y }
+  //
+  // We support loading old IR through a custom readAttribute method, see
+  // `readAttribute()` in `TestBytecodeDialectInterface`
+  let arguments = (ins
+    TestAttrParams:$attribute
+  );
+}
+
 #endif // TEST_OPS
diff --git a/mlir/test/lib/Dialect/Test/TestPatterns.cpp b/mlir/test/lib/Dialect/Test/TestPatterns.cpp
index 66c369d845bdf..20d3d486c921f 100644
--- a/mlir/test/lib/Dialect/Test/TestPatterns.cpp
+++ b/mlir/test/lib/Dialect/Test/TestPatterns.cpp
@@ -86,14 +86,13 @@ struct FoldingPattern : public RewritePattern {
 
   LogicalResult matchAndRewrite(Operation *op,
                                 PatternRewriter &rewriter) const override {
-    // Exercise OperationFolder API for a single-result operation that is folded
-    // upon construction. The operation being created through the folder has an
-    // in-place folder, and it should be still present in the output.
-    // Furthermore, the folder should not crash when attempting to recover the
-    // (unchanged) operation result.
-    OperationFolder folder(op->getContext());
-    Value result = folder.create<TestOpInPlaceFold>(
-        rewriter, op->getLoc(), rewriter.getIntegerType(32), op->getOperand(0));
+    // Exercise createOrFold API for a single-result operation that is folded
+    // upon construction. The operation being created has an in-place folder,
+    // and it should be still present in the output. Furthermore, the folder
+    // should not crash when attempting to recover the (unchanged) operation
+    // result.
+    Value result = rewriter.createOrFold<TestOpInPlaceFold>(
+        op->getLoc(), rewriter.getIntegerType(32), op->getOperand(0));
     assert(result);
     rewriter.replaceOp(op, result);
     return success();
@@ -1592,7 +1591,7 @@ struct TestMergeSingleBlockOps
     Block &innerBlock = op.getRegion().front();
     TerminatorOp innerTerminator =
         cast<TerminatorOp>(innerBlock.getTerminator());
-    rewriter.mergeBlockBefore(&innerBlock, op);
+    rewriter.inlineBlockBefore(&innerBlock, op);
     rewriter.eraseOp(innerTerminator);
     rewriter.eraseOp(op);
     rewriter.updateRootInPlace(op, [] {});
diff --git a/mlir/test/lib/Dialect/Transform/TestTransformDialectExtension.cpp b/mlir/test/lib/Dialect/Transform/TestTransformDialectExtension.cpp
index e32cb3ec891ab..79b5256ef1e23 100644
--- a/mlir/test/lib/Dialect/Transform/TestTransformDialectExtension.cpp
+++ b/mlir/test/lib/Dialect/Transform/TestTransformDialectExtension.cpp
@@ -297,6 +297,8 @@ DiagnosedSilenceableFailure mlir::test::TestRemapOperandPayloadToSelfOp::apply(
   if (failed(extension->updateMapping(state.getPayloadOps(getOperand()).front(),
                                       getOperation())))
     return DiagnosedSilenceableFailure::definiteFailure();
+  if (getNumResults() > 0)
+    results.set(getResult(0).cast<OpResult>(), getOperation());
   return DiagnosedSilenceableFailure::success();
 }
 
diff --git a/mlir/test/lib/Dialect/Vector/CMakeLists.txt b/mlir/test/lib/Dialect/Vector/CMakeLists.txt
index 415aca5626da5..aa8087e8b2273 100644
--- a/mlir/test/lib/Dialect/Vector/CMakeLists.txt
+++ b/mlir/test/lib/Dialect/Vector/CMakeLists.txt
@@ -7,6 +7,7 @@ add_mlir_library(MLIRVectorTestPasses
   LINK_LIBS PUBLIC
   MLIRAffineDialect
   MLIRAnalysis
+  MLIRArithDialect
   MLIRFuncDialect
   MLIRGPUOps
   MLIRLinalgDialect
@@ -15,6 +16,7 @@ add_mlir_library(MLIRVectorTestPasses
   MLIRMemRefDialect
   MLIRPass
   MLIRSCFDialect
+  MLIRTensorDialect
   MLIRTransformUtils
   MLIRVectorDialect
   MLIRVectorToSCF
diff --git a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
index 68edea5efca09..5a21bff0b39c3 100644
--- a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
+++ b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
@@ -11,6 +11,7 @@
 
 #include "mlir/Analysis/SliceAnalysis.h"
 #include "mlir/Dialect/Affine/IR/AffineOps.h"
+#include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/Func/IR/FuncOps.h"
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
@@ -19,6 +20,7 @@
 #include "mlir/Dialect/Linalg/Transforms/Transforms.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/Vector/Transforms/VectorDistribution.h"
 #include "mlir/Dialect/Vector/Transforms/VectorRewritePatterns.h"
@@ -199,6 +201,33 @@ struct TestVectorContractionLowering
   }
 };
 
+struct TestVectorContractionPrepareForMMTLowering
+    : public PassWrapper<TestVectorContractionPrepareForMMTLowering,
+                         OperationPass<func::FuncOp>> {
+  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(
+      TestVectorContractionPrepareForMMTLowering)
+
+  StringRef getArgument() const final {
+    return "test-vector-contraction-prepare-for-mmt-lowering";
+  }
+  StringRef getDescription() const final {
+    return "Test vector.contraction matmul canonicalization for MMT lowering.";
+  }
+  TestVectorContractionPrepareForMMTLowering() = default;
+
+  void getDependentDialects(DialectRegistry &registry) const override {
+    registry
+        .insert<AffineDialect, arith::ArithDialect, vector::VectorDialect>();
+  }
+
+  void runOnOperation() override {
+    MLIRContext *ctx = &getContext();
+    RewritePatternSet patterns(ctx);
+    vector::populateVectorContractCanonicalizeMatmulToMMT(patterns);
+    (void)applyPatternsAndFoldGreedily(getOperation(), std::move(patterns));
+  }
+};
+
 struct TestVectorTransposeLowering
     : public PassWrapper<TestVectorTransposeLowering,
                          OperationPass<func::FuncOp>> {
@@ -883,6 +912,29 @@ struct TestCreateVectorBroadcast
   }
 };
 
+struct TestVectorGatherLowering
+    : public PassWrapper<TestVectorGatherLowering,
+                         OperationPass<func::FuncOp>> {
+  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(TestVectorGatherLowering)
+
+  StringRef getArgument() const final { return "test-vector-gather-lowering"; }
+  StringRef getDescription() const final {
+    return "Test patterns that lower the gather op in the vector conditional "
+           "loads";
+  }
+  void getDependentDialects(DialectRegistry &registry) const override {
+    registry.insert<arith::ArithDialect, func::FuncDialect,
+                    memref::MemRefDialect, scf::SCFDialect,
+                    tensor::TensorDialect, vector::VectorDialect>();
+  }
+
+  void runOnOperation() override {
+    RewritePatternSet patterns(&getContext());
+    populateVectorGatherLoweringPatterns(patterns);
+    (void)applyPatternsAndFoldGreedily(getOperation(), std::move(patterns));
+  }
+};
+
 } // namespace
 
 namespace mlir {
@@ -892,6 +944,8 @@ void registerTestVectorLowerings() {
 
   PassRegistration<TestVectorContractionLowering>();
 
+  PassRegistration<TestVectorContractionPrepareForMMTLowering>();
+
   PassRegistration<TestVectorTransposeLowering>();
 
   PassRegistration<TestVectorUnrollingPatterns>();
@@ -923,6 +977,8 @@ void registerTestVectorLowerings() {
   PassRegistration<TestVectorExtractStridedSliceLowering>();
 
   PassRegistration<TestCreateVectorBroadcast>();
+
+  PassRegistration<TestVectorGatherLowering>();
 }
 } // namespace test
 } // namespace mlir
diff --git a/mlir/test/lib/IR/TestVisitors.cpp b/mlir/test/lib/IR/TestVisitors.cpp
index 3211d10bab9fa..6ed4abc71b7db 100644
--- a/mlir/test/lib/IR/TestVisitors.cpp
+++ b/mlir/test/lib/IR/TestVisitors.cpp
@@ -7,6 +7,8 @@
 //===----------------------------------------------------------------------===//
 
 #include "mlir/IR/BuiltinOps.h"
+#include "mlir/IR/FunctionInterfaces.h"
+#include "mlir/IR/Iterators.h"
 #include "mlir/Pass/Pass.h"
 
 using namespace mlir;
@@ -74,6 +76,23 @@ static void testPureCallbacks(Operation *op) {
   llvm::outs() << "Region reverse post-order visits"
                << "\n";
   op->walk<WalkOrder::PostOrder, ReverseIterator>(regionPure);
+
+  // This test case tests "NoGraphRegions = true", so start the walk with
+  // functions.
+  op->walk([&](FunctionOpInterface funcOp) {
+    llvm::outs() << "Op forward dominance post-order visits"
+                 << "\n";
+    funcOp->walk<WalkOrder::PostOrder,
+                 ForwardDominanceIterator</*NoGraphRegions=*/true>>(opPure);
+    llvm::outs() << "Block forward dominance post-order visits"
+                 << "\n";
+    funcOp->walk<WalkOrder::PostOrder,
+                 ForwardDominanceIterator</*NoGraphRegions=*/true>>(blockPure);
+    llvm::outs() << "Region forward dominance post-order visits"
+                 << "\n";
+    funcOp->walk<WalkOrder::PostOrder,
+                 ForwardDominanceIterator</*NoGraphRegions=*/true>>(regionPure);
+  });
 }
 
 /// Tests erasure callbacks that skip the walk.
@@ -98,11 +117,18 @@ static void testSkipErasureCallbacks(Operation *op) {
     if (isa<ModuleOp>(parentOp) || isa<ModuleOp>(parentOp->getParentOp()))
       return WalkResult::advance();
 
-    llvm::outs() << "Erasing ";
-    printBlock(block);
-    llvm::outs() << "\n";
-    block->erase();
-    return WalkResult::skip();
+    if (block->use_empty()) {
+      llvm::outs() << "Erasing ";
+      printBlock(block);
+      llvm::outs() << "\n";
+      block->erase();
+      return WalkResult::skip();
+    } else {
+      llvm::outs() << "Cannot erase ";
+      printBlock(block);
+      llvm::outs() << ", still has uses\n";
+      return WalkResult::advance();
+    }
   };
 
   llvm::outs() << "Op pre-order erasures (skip)"
@@ -141,10 +167,16 @@ static void testNoSkipErasureCallbacks(Operation *op) {
     op->erase();
   };
   auto noSkipBlockErasure = [](Block *block) {
-    llvm::outs() << "Erasing ";
-    printBlock(block);
-    llvm::outs() << "\n";
-    block->erase();
+    if (block->use_empty()) {
+      llvm::outs() << "Erasing ";
+      printBlock(block);
+      llvm::outs() << "\n";
+      block->erase();
+    } else {
+      llvm::outs() << "Cannot erase ";
+      printBlock(block);
+      llvm::outs() << ", still has uses\n";
+    }
   };
 
   llvm::outs() << "Op post-order erasures (no skip)"
diff --git a/mlir/test/mlir-cpu-runner/async-group.mlir b/mlir/test/mlir-cpu-runner/async-group.mlir
index 6407c324365ae..9735a5d330ee5 100644
--- a/mlir/test/mlir-cpu-runner/async-group.mlir
+++ b/mlir/test/mlir-cpu-runner/async-group.mlir
@@ -1,3 +1,10 @@
+// This is crashing in CI "most of the time" on a AMD Rome CPU VM on GCP with:
+//    Tracer caught signal 11: addr=0x7a800028 pc=0x2e81ba sp=0x7efd2a7ffd50
+//    LeakSanitizer has encountered a fatal error.
+// This is hard to reproduce locally unfortunately. Disable it with ASAN/LSAN
+// to keep the bot green for now.
+// RUN: export LSAN_OPTIONS=detect_leaks=0
+
 // RUN:   mlir-opt %s -pass-pipeline="builtin.module(async-to-async-runtime,func.func(async-runtime-ref-counting,async-runtime-ref-counting-opt),convert-async-to-llvm,func.func(convert-arith-to-llvm),convert-func-to-llvm,reconcile-unrealized-casts)" \
 // RUN: | mlir-cpu-runner                                                      \
 // RUN:     -e main -entry-point-result=void -O0                               \
@@ -6,13 +13,6 @@
 // RUN:     -shared-libs=%mlir_async_runtime   \
 // RUN: | FileCheck %s
 
-// This is crashing in CI "most of the time" on a AMD Rome CPU VM on GCP with:
-//    Tracer caught signal 11: addr=0x7a800028 pc=0x2e81ba sp=0x7efd2a7ffd50
-//    LeakSanitizer has encountered a fatal error.
-// This is hard to reproduce locally unfortunately. Disable it with ASAN/LSAN
-// to keep the bot green for now.
-// UNSUPPORTED: asan
-
 // FIXME: https://github.com/llvm/llvm-project/issues/57231
 // UNSUPPORTED: hwasan
 // FIXME: Windows does not have aligned_alloc
diff --git a/mlir/test/mlir-cpu-runner/bare-ptr-call-conv.mlir b/mlir/test/mlir-cpu-runner/bare-ptr-call-conv.mlir
index b934480c923bf..8bbaf3fbcd5f3 100644
--- a/mlir/test/mlir-cpu-runner/bare-ptr-call-conv.mlir
+++ b/mlir/test/mlir-cpu-runner/bare-ptr-call-conv.mlir
@@ -26,8 +26,6 @@ func.func @simple_add1_add2_test(%arg0: memref<2xf32>, %arg1: memref<2xf32>) {
 }
 
 // External declarations.
-llvm.func @malloc(i64) -> !llvm.ptr<i8>
-llvm.func @free(!llvm.ptr<i8>)
 func.func private @printF32(%arg0: f32)
 func.func private @printComma()
 func.func private @printNewline()
diff --git a/mlir/test/mlir-cpu-runner/math-polynomial-approx.mlir b/mlir/test/mlir-cpu-runner/math-polynomial-approx.mlir
index dbd816639ede3..665d3280c0c46 100644
--- a/mlir/test/mlir-cpu-runner/math-polynomial-approx.mlir
+++ b/mlir/test/mlir-cpu-runner/math-polynomial-approx.mlir
@@ -568,6 +568,48 @@ func.func @atan2() {
 }
 
 
+// -------------------------------------------------------------------------- //
+// Cbrt.
+// -------------------------------------------------------------------------- //
+
+func.func @cbrt_f32(%a : f32) {
+  %r = math.cbrt %a : f32
+  vector.print %r : f32
+  return
+}
+
+func.func @cbrt() {
+  // CHECK: 1
+  %a = arith.constant 1.0 : f32
+  call @cbrt_f32(%a) : (f32) -> ()
+
+  // CHECK: -1
+  %b = arith.constant -1.0 : f32
+  call @cbrt_f32(%b) : (f32) -> ()
+
+  // CHECK: 0
+  %c = arith.constant 0.0 : f32
+  call @cbrt_f32(%c) : (f32) -> ()
+
+  // CHECK: -0
+  %d = arith.constant -0.0 : f32
+  call @cbrt_f32(%d) : (f32) -> ()
+
+  // CHECK: 10
+  %e = arith.constant 1000.0 : f32
+  call @cbrt_f32(%e) : (f32) -> ()
+
+  // CHECK: -10
+  %f = arith.constant -1000.0 : f32
+  call @cbrt_f32(%f) : (f32) -> ()
+
+  // CHECK: 2.57128
+  %g = arith.constant 17.0 : f32
+  call @cbrt_f32(%g) : (f32) -> ()
+
+  return
+}
+
 func.func @main() {
   call @tanh(): () -> ()
   call @log(): () -> ()
@@ -580,5 +622,8 @@ func.func @main() {
   call @cos(): () -> ()
   call @atan() : () -> ()
   call @atan2() : () -> ()
+  call @cbrt() : () -> ()
   return
 }
+
+
diff --git a/mlir/test/mlir-opt/commandline.mlir b/mlir/test/mlir-opt/commandline.mlir
index bdcdf9de29ff8..70766498b5d84 100644
--- a/mlir/test/mlir-opt/commandline.mlir
+++ b/mlir/test/mlir-opt/commandline.mlir
@@ -1,45 +1,45 @@
-// RUN: mlir-opt --show-dialects | FileCheck %s
+// RUN: echo "" | mlir-opt --show-dialects | FileCheck %s
 // CHECK: Available Dialects:
-// CHECK-NEXT: acc
-// CHECK-NEXT: affine
-// CHECK-NEXT: amdgpu
-// CHECK-NEXT: amx
-// CHECK-NEXT: arith
-// CHECK-NEXT: arm_neon
-// CHECK-NEXT: arm_sve
-// CHECK-NEXT: async
-// CHECK-NEXT: bufferization
-// CHECK-NEXT: builtin
-// CHECK-NEXT: cf
-// CHECK-NEXT: complex
-// CHECK-NEXT: dlti
-// CHECK-NEXT: emitc
-// CHECK-NEXT: func
-// CHECK-NEXT: gpu
-// CHECK-NEXT: index
-// CHECK-NEXT: linalg
-// CHECK-NEXT: llvm
-// CHECK-NEXT: math
-// CHECK-NEXT: memref
-// CHECK-NEXT: ml_program
-// CHECK-NEXT: nvgpu
-// CHECK-NEXT: nvvm
-// CHECK-NEXT: omp
-// CHECK-NEXT: pdl
-// CHECK-NEXT: pdl_interp
-// CHECK-NEXT: quant
-// CHECK-NEXT: rocdl
-// CHECK-NEXT: scf
-// CHECK-NEXT: shape
-// CHECK-NEXT: sparse_tensor
-// CHECK-NEXT: spirv
-// CHECK-NEXT: tensor
-// CHECK-NEXT: test
-// CHECK-NEXT: test_dyn
-// CHECK-NEXT: tosa
-// CHECK-NEXT: transform
-// CHECK-NEXT: vector
-// CHECK-NEXT: x86vector
+// CHECK-SAME: acc
+// CHECK-SAME: affine
+// CHECK-SAME: amdgpu
+// CHECK-SAME: amx
+// CHECK-SAME: arith
+// CHECK-SAME: arm_neon
+// CHECK-SAME: arm_sve
+// CHECK-SAME: async
+// CHECK-SAME: bufferization
+// CHECK-SAME: builtin
+// CHECK-SAME: cf
+// CHECK-SAME: complex
+// CHECK-SAME: dlti
+// CHECK-SAME: emitc
+// CHECK-SAME: func
+// CHECK-SAME: gpu
+// CHECK-SAME: index
+// CHECK-SAME: linalg
+// CHECK-SAME: llvm
+// CHECK-SAME: math
+// CHECK-SAME: memref
+// CHECK-SAME: ml_program
+// CHECK-SAME: nvgpu
+// CHECK-SAME: nvvm
+// CHECK-SAME: omp
+// CHECK-SAME: pdl
+// CHECK-SAME: pdl_interp
+// CHECK-SAME: quant
+// CHECK-SAME: rocdl
+// CHECK-SAME: scf
+// CHECK-SAME: shape
+// CHECK-SAME: sparse_tensor
+// CHECK-SAME: spirv
+// CHECK-SAME: tensor
+// CHECK-SAME: test
+// CHECK-SAME: test_dyn
+// CHECK-SAME: tosa
+// CHECK-SAME: transform
+// CHECK-SAME: vector
+// CHECK-SAME: x86vector
 
 // RUN: mlir-opt --help-hidden | FileCheck %s -check-prefix=CHECK-HELP
 // CHECK-HELP: -p - Alias for --pass-pipeline
diff --git a/mlir/test/python/dialects/builtin.py b/mlir/test/python/dialects/builtin.py
index 94e29892ba7b1..eab24b5c796b0 100644
--- a/mlir/test/python/dialects/builtin.py
+++ b/mlir/test/python/dialects/builtin.py
@@ -3,6 +3,7 @@
 from mlir.ir import *
 import mlir.dialects.builtin as builtin
 import mlir.dialects.func as func
+import numpy as np
 
 
 def run(f):
@@ -221,3 +222,17 @@ def testFuncArgumentAccess():
   # CHECK: %{{.*}}: f32 {custom_dialect.foo = "qux"},
   # CHECK: %{{.*}}: f32)
   print(module)
+
+
+# CHECK-LABEL: testDenseElementsAttr
+@run
+def testDenseElementsAttr():
+  with Context(), Location.unknown():
+    values = np.arange(4, dtype=np.int32)
+    i32 = IntegerType.get_signless(32)
+    print(DenseElementsAttr.get(values, type=i32))
+    # CHECK{LITERAL}: dense<[0, 1, 2, 3]> : tensor<4xi32>
+    print(DenseElementsAttr.get(values, type=i32, shape=(2, 2)))
+    # CHECK{LITERAL}: dense<[[0, 1], [2, 3]]> : tensor<2x2xi32>
+    print(DenseElementsAttr.get(values, type=VectorType.get((2, 2), i32)))
+    # CHECK{LITERAL}: dense<[[0, 1], [2, 3]]> : vector<2x2xi32>
diff --git a/mlir/test/python/dialects/sparse_tensor/dialect.py b/mlir/test/python/dialects/sparse_tensor/dialect.py
index e8e753df61878..988d1a29f938f 100644
--- a/mlir/test/python/dialects/sparse_tensor/dialect.py
+++ b/mlir/test/python/dialects/sparse_tensor/dialect.py
@@ -15,10 +15,10 @@ def testEncodingAttr1D():
   with Context() as ctx:
     parsed = Attribute.parse('#sparse_tensor.encoding<{'
                              '  dimLevelType = [ "compressed" ],'
-                             '  pointerBitWidth = 16,'
-                             '  indexBitWidth = 32'
+                             '  posWidth = 16,'
+                             '  crdWidth = 32'
                              '}>')
-    # CHECK: #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ], pointerBitWidth = 16, indexBitWidth = 32 }>
+    # CHECK: #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ], posWidth = 16, crdWidth = 32 }>
     print(parsed)
 
     casted = st.EncodingAttr(parsed)
@@ -29,10 +29,10 @@ def testEncodingAttr1D():
     print(f"dim_level_types: {casted.dim_level_types}")
     # CHECK: dim_ordering: None
     print(f"dim_ordering: {casted.dim_ordering}")
-    # CHECK: pointer_bit_width: 16
-    print(f"pointer_bit_width: {casted.pointer_bit_width}")
-    # CHECK: index_bit_width: 32
-    print(f"index_bit_width: {casted.index_bit_width}")
+    # CHECK: pos_width: 16
+    print(f"pos_width: {casted.pos_width}")
+    # CHECK: crd_width: 32
+    print(f"crd_width: {casted.crd_width}")
 
     created = st.EncodingAttr.get(casted.dim_level_types, None, None, 0, 0)
     # CHECK: #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>
@@ -43,8 +43,8 @@ def testEncodingAttr1D():
     # Verify that the factory creates an instance of the proper type.
     # CHECK: is_proper_instance: True
     print(f"is_proper_instance: {isinstance(created, st.EncodingAttr)}")
-    # CHECK: created_pointer_bit_width: 0
-    print(f"created_pointer_bit_width: {created.pointer_bit_width}")
+    # CHECK: created_pos_width: 0
+    print(f"created_pos_width: {created.pos_width}")
 
 
 # CHECK-LABEL: TEST: testEncodingAttr2D
@@ -54,10 +54,10 @@ def testEncodingAttr2D():
     parsed = Attribute.parse('#sparse_tensor.encoding<{'
                              '  dimLevelType = [ "dense", "compressed" ],'
                              '  dimOrdering = affine_map<(d0, d1) -> (d1, d0)>,'
-                             '  pointerBitWidth = 8,'
-                             '  indexBitWidth = 32'
+                             '  posWidth = 8,'
+                             '  crdWidth = 32'
                              '}>')
-    # CHECK: #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)>, pointerBitWidth = 8, indexBitWidth = 32 }>
+    # CHECK: #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)>, posWidth = 8, crdWidth = 32 }>
     print(parsed)
 
     casted = st.EncodingAttr(parsed)
@@ -68,14 +68,14 @@ def testEncodingAttr2D():
     print(f"dim_level_types: {casted.dim_level_types}")
     # CHECK: dim_ordering: (d0, d1) -> (d1, d0)
     print(f"dim_ordering: {casted.dim_ordering}")
-    # CHECK: pointer_bit_width: 8
-    print(f"pointer_bit_width: {casted.pointer_bit_width}")
-    # CHECK: index_bit_width: 32
-    print(f"index_bit_width: {casted.index_bit_width}")
+    # CHECK: pos_width: 8
+    print(f"pos_width: {casted.pos_width}")
+    # CHECK: crd_width: 32
+    print(f"crd_width: {casted.crd_width}")
 
     created = st.EncodingAttr.get(casted.dim_level_types, casted.dim_ordering,
                                   casted.higher_ordering, 8, 32)
-    # CHECK: #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)>, pointerBitWidth = 8, indexBitWidth = 32 }>
+    # CHECK: #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed" ], dimOrdering = affine_map<(d0, d1) -> (d1, d0)>, posWidth = 8, crdWidth = 32 }>
     print(created)
     # CHECK: created_equal: True
     print(f"created_equal: {created == casted}")
@@ -88,12 +88,12 @@ def testEncodingAttrOnTensorType():
     encoding = st.EncodingAttr(
         Attribute.parse('#sparse_tensor.encoding<{'
                         '  dimLevelType = [ "compressed" ], '
-                        '  pointerBitWidth = 64,'
-                        '  indexBitWidth = 32'
+                        '  posWidth = 64,'
+                        '  crdWidth = 32'
                         '}>'))
     tt = RankedTensorType.get((1024,), F32Type.get(), encoding=encoding)
-    # CHECK: tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ], pointerBitWidth = 64, indexBitWidth = 32 }>>
+    # CHECK: tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ], posWidth = 64, crdWidth = 32 }>>
     print(tt)
-    # CHECK: #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ], pointerBitWidth = 64, indexBitWidth = 32 }>
+    # CHECK: #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ], posWidth = 64, crdWidth = 32 }>
     print(tt.encoding)
     assert tt.encoding == encoding
diff --git a/mlir/test/python/ir/attributes.py b/mlir/test/python/ir/attributes.py
index 684d52c3ae28f..1e1589d6d5f4c 100644
--- a/mlir/test/python/ir/attributes.py
+++ b/mlir/test/python/ir/attributes.py
@@ -28,16 +28,17 @@ def testParsePrint():
 
 
 # CHECK-LABEL: TEST: testParseError
-# TODO: Hook the diagnostic manager to capture a more meaningful error
-# message.
 @run
 def testParseError():
   with Context():
     try:
       t = Attribute.parse("BAD_ATTR_DOES_NOT_EXIST")
-    except ValueError as e:
-      # CHECK: Unable to parse attribute: 'BAD_ATTR_DOES_NOT_EXIST'
-      print("testParseError:", e)
+    except MLIRError as e:
+      # CHECK: testParseError: <
+      # CHECK:   Unable to parse attribute:
+      # CHECK:   error: "BAD_ATTR_DOES_NOT_EXIST":1:1: expected attribute value
+      # CHECK: >
+      print(f"testParseError: <{e}>")
     else:
       print("Exception not produced")
 
@@ -180,8 +181,9 @@ def testFloatAttr():
     try:
       fattr_invalid = FloatAttr.get(
           IntegerType.get_signless(32), 42)
-    except ValueError as e:
-      # CHECK: invalid 'Type(i32)' and expected floating point type.
+    except MLIRError as e:
+      # CHECK: Invalid attribute:
+      # CHECK: error: unknown: expected floating point type
       print(e)
     else:
       print("Exception not produced")
diff --git a/mlir/test/python/ir/builtin_types.py b/mlir/test/python/ir/builtin_types.py
index 7af81859e3e7a..594cc6620e396 100644
--- a/mlir/test/python/ir/builtin_types.py
+++ b/mlir/test/python/ir/builtin_types.py
@@ -26,16 +26,17 @@ def testParsePrint():
 
 
 # CHECK-LABEL: TEST: testParseError
-# TODO: Hook the diagnostic manager to capture a more meaningful error
-# message.
 @run
 def testParseError():
   ctx = Context()
   try:
     t = Type.parse("BAD_TYPE_DOES_NOT_EXIST", ctx)
-  except ValueError as e:
-    # CHECK: Unable to parse type: 'BAD_TYPE_DOES_NOT_EXIST'
-    print("testParseError:", e)
+  except MLIRError as e:
+    # CHECK: testParseError: <
+    # CHECK:   Unable to parse type:
+    # CHECK:   error: "BAD_TYPE_DOES_NOT_EXIST":1:1: expected non-function type
+    # CHECK: >
+    print(f"testParseError: <{e}>")
   else:
     print("Exception not produced")
 
@@ -292,8 +293,9 @@ def testVectorType():
     none = NoneType.get()
     try:
       vector_invalid = VectorType.get(shape, none)
-    except ValueError as e:
-      # CHECK: invalid 'Type(none)' and expected floating point or integer type.
+    except MLIRError as e:
+      # CHECK: Invalid type:
+      # CHECK: error: unknown: vector elements must be int/index/float type but got 'none'
       print(e)
     else:
       print("Exception not produced")
@@ -313,9 +315,9 @@ def testRankedTensorType():
     none = NoneType.get()
     try:
       tensor_invalid = RankedTensorType.get(shape, none)
-    except ValueError as e:
-      # CHECK: invalid 'Type(none)' and expected floating point, integer, vector
-      # CHECK: or complex type.
+    except MLIRError as e:
+      # CHECK: Invalid type:
+      # CHECK: error: unknown: invalid tensor element type: 'none'
       print(e)
     else:
       print("Exception not produced")
@@ -361,9 +363,9 @@ def testUnrankedTensorType():
     none = NoneType.get()
     try:
       tensor_invalid = UnrankedTensorType.get(none)
-    except ValueError as e:
-      # CHECK: invalid 'Type(none)' and expected floating point, integer, vector
-      # CHECK: or complex type.
+    except MLIRError as e:
+      # CHECK: Invalid type:
+      # CHECK: error: unknown: invalid tensor element type: 'none'
       print(e)
     else:
       print("Exception not produced")
@@ -400,9 +402,9 @@ def testMemRefType():
     none = NoneType.get()
     try:
       memref_invalid = MemRefType.get(shape, none)
-    except ValueError as e:
-      # CHECK: invalid 'Type(none)' and expected floating point, integer, vector
-      # CHECK: or complex type.
+    except MLIRError as e:
+      # CHECK: Invalid type:
+      # CHECK: error: unknown: invalid memref element type
       print(e)
     else:
       print("Exception not produced")
@@ -444,9 +446,9 @@ def testUnrankedMemRefType():
     none = NoneType.get()
     try:
       memref_invalid = UnrankedMemRefType.get(none, Attribute.parse("2"))
-    except ValueError as e:
-      # CHECK: invalid 'Type(none)' and expected floating point, integer, vector
-      # CHECK: or complex type.
+    except MLIRError as e:
+      # CHECK: Invalid type:
+      # CHECK: error: unknown: invalid memref element type
       print(e)
     else:
       print("Exception not produced")
diff --git a/mlir/test/python/ir/diagnostic_handler.py b/mlir/test/python/ir/diagnostic_handler.py
index d973db24c366b..cc07f6eaf56ed 100644
--- a/mlir/test/python/ir/diagnostic_handler.py
+++ b/mlir/test/python/ir/diagnostic_handler.py
@@ -89,6 +89,7 @@ def callback(d):
 @run
 def testDiagnosticNonEmptyNotes():
   ctx = Context()
+  ctx.emit_error_diagnostics = True
   def callback(d):
     # CHECK: DIAGNOSTIC:
     # CHECK:   message='arith.addi' op requires one result
@@ -99,7 +100,10 @@ def callback(d):
     return True
   handler = ctx.attach_diagnostic_handler(callback)
   loc = Location.unknown(ctx)
-  Operation.create('arith.addi', loc=loc).verify()
+  try:
+    Operation.create('arith.addi', loc=loc).verify()
+  except MLIRError:
+    pass
   assert not handler.had_error
 
 # CHECK-LABEL: TEST: testDiagnosticCallbackException
diff --git a/mlir/test/python/ir/exception.py b/mlir/test/python/ir/exception.py
new file mode 100644
index 0000000000000..6cb2375a13247
--- /dev/null
+++ b/mlir/test/python/ir/exception.py
@@ -0,0 +1,77 @@
+# RUN: %PYTHON %s | FileCheck %s
+
+import gc
+from mlir.ir import *
+
+def run(f):
+  print("\nTEST:", f.__name__)
+  f()
+  gc.collect()
+  assert Context._get_live_count() == 0
+  return f
+
+
+# CHECK-LABEL: TEST: test_exception
+@run
+def test_exception():
+  ctx =  Context()
+  ctx.allow_unregistered_dialects = True
+  try:
+    Operation.parse("""
+      func.func @foo() {
+          "test.use"(%0) : (i64) -> ()  loc("use")
+          %0 = "test.def"() : () -> i64 loc("def")
+          return
+      }
+    """, context=ctx)
+  except MLIRError as e:
+    # CHECK: Exception: <
+    # CHECK:   Unable to parse operation assembly:
+    # CHECK:   error: "use": operand #0 does not dominate this use
+    # CHECK:    note: "use": see current operation: "test.use"(%0) : (i64) -> ()
+    # CHECK:    note: "def": operand defined here (op in the same block)
+    # CHECK: >
+    print(f"Exception: <{e}>")
+
+    # CHECK: message: Unable to parse operation assembly
+    print(f"message: {e.message}")
+
+    # CHECK: error_diagnostics[0]:           loc("use") operand #0 does not dominate this use
+    # CHECK: error_diagnostics[0].notes[0]:  loc("use") see current operation: "test.use"(%0) : (i64) -> ()
+    # CHECK: error_diagnostics[0].notes[1]:  loc("def") operand defined here (op in the same block)
+    print("error_diagnostics[0]:          ", e.error_diagnostics[0].location, e.error_diagnostics[0].message)
+    print("error_diagnostics[0].notes[0]: ", e.error_diagnostics[0].notes[0].location, e.error_diagnostics[0].notes[0].message)
+    print("error_diagnostics[0].notes[1]: ", e.error_diagnostics[0].notes[1].location, e.error_diagnostics[0].notes[1].message)
+
+
+# CHECK-LABEL: test_emit_error_diagnostics
+@run
+def test_emit_error_diagnostics():
+  ctx = Context()
+  loc = Location.unknown(ctx)
+  handler_diags = []
+  def handler(d):
+    handler_diags.append(str(d))
+    return True
+  ctx.attach_diagnostic_handler(handler)
+
+  try:
+    Attribute.parse("not an attr", ctx)
+  except MLIRError as e:
+    # CHECK: emit_error_diagnostics=False:
+    # CHECK: e.error_diagnostics: ['expected attribute value']
+    # CHECK: handler_diags: []
+    print(f"emit_error_diagnostics=False:")
+    print(f"e.error_diagnostics: {[str(diag) for diag in e.error_diagnostics]}")
+    print(f"handler_diags: {handler_diags}")
+
+  ctx.emit_error_diagnostics = True
+  try:
+    Attribute.parse("not an attr", ctx)
+  except MLIRError as e:
+    # CHECK: emit_error_diagnostics=True:
+    # CHECK: e.error_diagnostics: []
+    # CHECK: handler_diags: ['expected attribute value']
+    print(f"emit_error_diagnostics=True:")
+    print(f"e.error_diagnostics: {[str(diag) for diag in e.error_diagnostics]}")
+    print(f"handler_diags: {handler_diags}")
diff --git a/mlir/test/python/ir/module.py b/mlir/test/python/ir/module.py
index f0b62435f514b..2d00923683339 100644
--- a/mlir/test/python/ir/module.py
+++ b/mlir/test/python/ir/module.py
@@ -28,14 +28,17 @@ def testParseSuccess():
 
 # Verify parse error.
 # CHECK-LABEL: TEST: testParseError
-# CHECK: testParseError: Unable to parse module assembly (see diagnostics)
+# CHECK: testParseError: <
+# CHECK:   Unable to parse module assembly:
+# CHECK:   error: "-":1:1: expected operation name in quotes
+# CHECK: >
 @run
 def testParseError():
   ctx = Context()
   try:
     module = Module.parse(r"""}SYNTAX ERROR{""", ctx)
-  except ValueError as e:
-    print("testParseError:", e)
+  except MLIRError as e:
+    print(f"testParseError: <{e}>")
   else:
     print("Exception not produced")
 
diff --git a/mlir/test/python/ir/operation.py b/mlir/test/python/ir/operation.py
index be7467d12ff13..941420e8d1ff1 100644
--- a/mlir/test/python/ir/operation.py
+++ b/mlir/test/python/ir/operation.py
@@ -685,8 +685,19 @@ def testInvalidOperationStrSoftFails():
     # CHECK: "builtin.module"() ({
     # CHECK: }) : () -> ()
     print(invalid_op)
-    # CHECK: .verify = False
-    print(f".verify = {invalid_op.operation.verify()}")
+    try:
+      invalid_op.verify()
+    except MLIRError as e:
+      # CHECK: Exception: <
+      # CHECK:   Verification failed:
+      # CHECK:   error: unknown: 'builtin.module' op requires one region
+      # CHECK:    note: unknown: see current operation:
+      # CHECK:     "builtin.module"() ({
+      # CHECK:     ^bb0:
+      # CHECK:     }, {
+      # CHECK:     }) : () -> ()
+      # CHECK: >
+      print(f"Exception: <{e}>")
 
 
 # CHECK-LABEL: TEST: testInvalidModuleStrSoftFails
@@ -920,7 +931,7 @@ def testOperationParse():
     assert isinstance(m, ModuleOp)
     try:
       ModuleOp.parse('"test.foo"() : () -> ()')
-    except ValueError as e:
+    except MLIRError as e:
       # CHECK: error: Expected a 'builtin.module' op, got: 'test.foo'
       print(f"error: {e}")
     else:
diff --git a/mlir/test/python/pass_manager.py b/mlir/test/python/pass_manager.py
index b3acd359a207d..8b276537dddcc 100644
--- a/mlir/test/python/pass_manager.py
+++ b/mlir/test/python/pass_manager.py
@@ -118,7 +118,7 @@ def testInvalidNesting():
 
 
 # Verify that a pass manager can execute on IR
-# CHECK-LABEL: TEST: testRun
+# CHECK-LABEL: TEST: testRunPipeline
 def testRunPipeline():
   with Context():
     pm = PassManager.parse("any(print-op-stats{json=false})")
@@ -128,3 +128,20 @@ def testRunPipeline():
 # CHECK: func.func      , 1
 # CHECK: func.return        , 1
 run(testRunPipeline)
+
+# CHECK-LABEL: TEST: testRunPipelineError
+@run
+def testRunPipelineError():
+  with Context() as ctx:
+    ctx.allow_unregistered_dialects = True
+    op = Operation.parse('"test.op"() : () -> ()')
+    pm = PassManager.parse("any(cse)")
+    try:
+      pm.run(op)
+    except MLIRError as e:
+      # CHECK: Exception: <
+      # CHECK:   Failure while executing pass pipeline:
+      # CHECK:   error: "-":1:1: 'test.op' op trying to schedule a pass on an unregistered operation
+      # CHECK:    note: "-":1:1: see current operation: "test.op"() : () -> ()
+      # CHECK: >
+      print(f"Exception: <{e}>")
diff --git a/mlir/tools/mlir-linalg-ods-gen/mlir-linalg-ods-yaml-gen.cpp b/mlir/tools/mlir-linalg-ods-gen/mlir-linalg-ods-yaml-gen.cpp
index a51bdb50da9ed..52ea148179357 100644
--- a/mlir/tools/mlir-linalg-ods-gen/mlir-linalg-ods-yaml-gen.cpp
+++ b/mlir/tools/mlir-linalg-ods-gen/mlir-linalg-ods-yaml-gen.cpp
@@ -869,14 +869,14 @@ exprs.push_back(getAffineConstantExpr(cst{1}, context));
           if (arg.kind != LinalgOperandDefKind::IndexAttr)
             continue;
           assert(arg.indexAttrMap);
-          for (auto &en :
+          for (auto [idx, result] :
                llvm::enumerate(arg.indexAttrMap->affineMap().getResults())) {
-            if (auto symbol = en.value().dyn_cast<AffineSymbolExpr>()) {
+            if (auto symbol = result.dyn_cast<AffineSymbolExpr>()) {
               std::string argName = arg.name;
               argName[0] = toupper(argName[0]);
               symbolBindings[symbol.getPosition()] =
                   llvm::formatv(structuredOpAccessAttrFormat, argName,
-                                symbol.getPosition(), en.index());
+                                symbol.getPosition(), idx);
             }
           }
         }
diff --git a/mlir/tools/mlir-opt/CMakeLists.txt b/mlir/tools/mlir-opt/CMakeLists.txt
index fb7b4e77c5ec8..f84fbe631cf16 100644
--- a/mlir/tools/mlir-opt/CMakeLists.txt
+++ b/mlir/tools/mlir-opt/CMakeLists.txt
@@ -41,6 +41,7 @@ if(MLIR_INCLUDE_TESTS)
     MLIRTestTransforms
     MLIRTilingInterfaceTestPasses
     MLIRVectorTestPasses
+    MLIRTestVectorToSPIRV
     MLIRLLVMTestPasses
     )
 endif()
diff --git a/mlir/tools/mlir-opt/mlir-opt.cpp b/mlir/tools/mlir-opt/mlir-opt.cpp
index b56c883da587b..e7ca06b4bef9d 100644
--- a/mlir/tools/mlir-opt/mlir-opt.cpp
+++ b/mlir/tools/mlir-opt/mlir-opt.cpp
@@ -40,6 +40,7 @@ void registerSliceAnalysisTestPass();
 void registerSymbolTestPasses();
 void registerRegionTestPasses();
 void registerTestAffineDataCopyPass();
+void registerTestDecomposeAffineOpPass();
 void registerTestAffineLoopUnswitchingPass();
 void registerTestAllReduceLoweringPass();
 void registerTestFunc();
@@ -124,6 +125,7 @@ void registerTestTransformDialectEraseSchedulePass();
 void registerTestTransformDialectInterpreterPass();
 void registerTestWrittenToPass();
 void registerTestVectorLowerings();
+void registerTestVectorReductionToSPIRVDotProd();
 void registerTestNvgpuLowerings();
 } // namespace test
 } // namespace mlir
@@ -147,6 +149,7 @@ void registerTestPasses() {
   registerSymbolTestPasses();
   registerRegionTestPasses();
   registerTestAffineDataCopyPass();
+  registerTestDecomposeAffineOpPass();
   registerTestAffineLoopUnswitchingPass();
   registerTestAllReduceLoweringPass();
   registerTestFunc();
@@ -231,6 +234,7 @@ void registerTestPasses() {
   mlir::test::registerTestTransformDialectEraseSchedulePass();
   mlir::test::registerTestTransformDialectInterpreterPass();
   mlir::test::registerTestVectorLowerings();
+  mlir::test::registerTestVectorReductionToSPIRVDotProd();
   mlir::test::registerTestNvgpuLowerings();
   mlir::test::registerTestWrittenToPass();
 }
diff --git a/mlir/tools/mlir-tblgen/AttrOrTypeFormatGen.cpp b/mlir/tools/mlir-tblgen/AttrOrTypeFormatGen.cpp
index 44b78faf9b4b8..4e248e2de57f1 100644
--- a/mlir/tools/mlir-tblgen/AttrOrTypeFormatGen.cpp
+++ b/mlir/tools/mlir-tblgen/AttrOrTypeFormatGen.cpp
@@ -951,16 +951,16 @@ class DefFormatParser : public FormatParser {
 LogicalResult DefFormatParser::verify(SMLoc loc,
                                       ArrayRef<FormatElement *> elements) {
   // Check that all parameters are referenced in the format.
-  for (auto &it : llvm::enumerate(def.getParameters())) {
-    if (it.value().isOptional())
+  for (auto [index, param] : llvm::enumerate(def.getParameters())) {
+    if (param.isOptional())
       continue;
-    if (!seenParams.test(it.index())) {
-      if (isa<AttributeSelfTypeParameter>(it.value()))
+    if (!seenParams.test(index)) {
+      if (isa<AttributeSelfTypeParameter>(param))
         continue;
       return emitError(loc, "format is missing reference to parameter: " +
-                                it.value().getName());
+                                param.getName());
     }
-    if (isa<AttributeSelfTypeParameter>(it.value())) {
+    if (isa<AttributeSelfTypeParameter>(param)) {
       return emitError(loc,
                        "unexpected self type parameter in assembly format");
     }
diff --git a/mlir/tools/mlir-tblgen/EnumsGen.cpp b/mlir/tools/mlir-tblgen/EnumsGen.cpp
index 10725f7177569..6d325d4461a3e 100644
--- a/mlir/tools/mlir-tblgen/EnumsGen.cpp
+++ b/mlir/tools/mlir-tblgen/EnumsGen.cpp
@@ -123,7 +123,7 @@ inline ::llvm::raw_ostream &operator<<(::llvm::raw_ostream &p, {0} value) {{
   // case value to determine when to print in the string form.
   if (nonKeywordCases.any()) {
     os << "  switch (value) {\n";
-    for (auto &it : llvm::enumerate(cases)) {
+    for (auto it : llvm::enumerate(cases)) {
       if (nonKeywordCases.test(it.index()))
         continue;
       StringRef symbol = it.value().getSymbol();
diff --git a/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp b/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp
index 5a7e3a01b6780..e3e7ae087085a 100644
--- a/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp
+++ b/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp
@@ -934,15 +934,16 @@ void OpEmitter::genAttrNameGetters() {
   // Generate the <attr>AttrName methods, that expose the attribute names to
   // users.
   const char *attrNameMethodBody = "  return getAttributeNameForIndex({0});";
-  for (auto &attrIt : llvm::enumerate(llvm::make_first_range(attributes))) {
-    std::string name = op.getGetterName(attrIt.value());
+  for (auto [index, attr] :
+       llvm::enumerate(llvm::make_first_range(attributes))) {
+    std::string name = op.getGetterName(attr);
     std::string methodName = name + "AttrName";
 
     // Generate the non-static variant.
     {
       auto *method = opClass.addInlineMethod("::mlir::StringAttr", methodName);
       ERROR_IF_PRUNED(method, methodName, op);
-      method->body() << llvm::formatv(attrNameMethodBody, attrIt.index());
+      method->body() << llvm::formatv(attrNameMethodBody, index);
     }
 
     // Generate the static variant.
@@ -952,7 +953,7 @@ void OpEmitter::genAttrNameGetters() {
           MethodParameter("::mlir::OperationName", "name"));
       ERROR_IF_PRUNED(method, methodName, op);
       method->body() << llvm::formatv(attrNameMethodBody,
-                                      "name, " + Twine(attrIt.index()));
+                                      "name, " + Twine(index));
     }
   }
 }
@@ -2801,7 +2802,7 @@ void OpEmitter::genSuccessorVerifier(MethodBody &body) {
 
   body << "  {\n    unsigned index = 0; (void)index;\n";
 
-  for (auto &it : llvm::enumerate(successors)) {
+  for (auto it : llvm::enumerate(successors)) {
     const auto &successor = it.value();
     if (canSkip(successor))
       continue;
diff --git a/mlir/tools/mlir-tblgen/OpFormatGen.cpp b/mlir/tools/mlir-tblgen/OpFormatGen.cpp
index c88738f6aff98..ed84bcc049a65 100644
--- a/mlir/tools/mlir-tblgen/OpFormatGen.cpp
+++ b/mlir/tools/mlir-tblgen/OpFormatGen.cpp
@@ -1827,7 +1827,7 @@ static void genEnumAttrPrinter(const NamedAttribute *var, const Operator &op,
   // Get a string containing all of the cases that can't be represented with a
   // keyword.
   BitVector nonKeywordCases(cases.size());
-  for (auto &it : llvm::enumerate(cases)) {
+  for (auto it : llvm::enumerate(cases)) {
     if (!canFormatStringAsKeyword(it.value().getStr()))
       nonKeywordCases.set(it.index());
   }
@@ -1836,7 +1836,7 @@ static void genEnumAttrPrinter(const NamedAttribute *var, const Operator &op,
   // overlap with other cases. For simplicity sake, only allow cases with a
   // single bit value.
   if (enumAttr.isBitEnum()) {
-    for (auto &it : llvm::enumerate(cases)) {
+    for (auto it : llvm::enumerate(cases)) {
       int64_t value = it.value().getValue();
       if (value < 0 || !llvm::isPowerOf2_64(value))
         nonKeywordCases.set(it.index());
@@ -1849,7 +1849,7 @@ static void genEnumAttrPrinter(const NamedAttribute *var, const Operator &op,
     body << "    switch (caseValue) {\n";
     StringRef cppNamespace = enumAttr.getCppNamespace();
     StringRef enumName = enumAttr.getEnumClassName();
-    for (auto &it : llvm::enumerate(cases)) {
+    for (auto it : llvm::enumerate(cases)) {
       if (nonKeywordCases.test(it.index()))
         continue;
       StringRef symbol = it.value().getSymbol();
diff --git a/mlir/tools/mlir-tblgen/RewriterGen.cpp b/mlir/tools/mlir-tblgen/RewriterGen.cpp
index a13429ea8bfb3..abbe8400b5fe0 100644
--- a/mlir/tools/mlir-tblgen/RewriterGen.cpp
+++ b/mlir/tools/mlir-tblgen/RewriterGen.cpp
@@ -1703,7 +1703,7 @@ void StaticMatcherHelper::populateStaticMatchers(raw_ostream &os) {
 
     std::string funcName =
         formatv("static_dag_matcher_{0}", staticMatcherCounter++);
-    assert(matcherNames.find(node) == matcherNames.end());
+    assert(!matcherNames.contains(node));
     PatternEmitter(dagInfo.second, &opMap, os, *this)
         .emitStaticMatcher(node, funcName);
     matcherNames[node] = funcName;
diff --git a/mlir/unittests/CMakeLists.txt b/mlir/unittests/CMakeLists.txt
index 78a0f99c5e3c9..137fc97a3fc02 100644
--- a/mlir/unittests/CMakeLists.txt
+++ b/mlir/unittests/CMakeLists.txt
@@ -6,6 +6,7 @@ endfunction()
 
 add_subdirectory(Analysis)
 add_subdirectory(Conversion)
+add_subdirectory(Debug)
 add_subdirectory(Dialect)
 add_subdirectory(Interfaces)
 add_subdirectory(IR)
diff --git a/mlir/unittests/Debug/CMakeLists.txt b/mlir/unittests/Debug/CMakeLists.txt
new file mode 100644
index 0000000000000..5ea18d2751de0
--- /dev/null
+++ b/mlir/unittests/Debug/CMakeLists.txt
@@ -0,0 +1,7 @@
+add_mlir_unittest(MLIRDebugTests
+  DebugCounterTest.cpp
+  ExecutionContextTest.cpp
+)
+
+target_link_libraries(MLIRDebugTests
+  PRIVATE MLIRDebug)
diff --git a/mlir/unittests/Debug/DebugCounterTest.cpp b/mlir/unittests/Debug/DebugCounterTest.cpp
new file mode 100644
index 0000000000000..eeeb538a94ba3
--- /dev/null
+++ b/mlir/unittests/Debug/DebugCounterTest.cpp
@@ -0,0 +1,49 @@
+//===- DebugCounterTest.cpp - Debug Counter Tests -------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Debug/Counter.h"
+#include "mlir/Support/TypeID.h"
+#include "gmock/gmock.h"
+
+using namespace mlir;
+using namespace mlir::tracing;
+
+namespace {
+
+struct CounterAction : public ActionImpl<CounterAction> {
+  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(CounterAction)
+  static constexpr StringLiteral tag = "counter-action";
+};
+
+TEST(DebugCounterTest, CounterTest) {
+  DebugCounter counter;
+  counter.addCounter(CounterAction::tag, /*countToSkip=*/1,
+                     /*countToStopAfter=*/3);
+
+  int count = 0;
+  auto noOp = [&]() {
+    ++count;
+    return;
+  };
+
+  // The first execution is skipped.
+  counter(noOp, CounterAction{});
+  EXPECT_EQ(count, 0);
+
+  // The counter stops after 3 successful executions.
+  counter(noOp, CounterAction{});
+  EXPECT_EQ(count, 1);
+  counter(noOp, CounterAction{});
+  EXPECT_EQ(count, 2);
+  counter(noOp, CounterAction{});
+  EXPECT_EQ(count, 3);
+  counter(noOp, CounterAction{});
+  EXPECT_EQ(count, 3);
+}
+
+} // namespace
diff --git a/mlir/unittests/Debug/ExecutionContextTest.cpp b/mlir/unittests/Debug/ExecutionContextTest.cpp
new file mode 100644
index 0000000000000..d757d5451afec
--- /dev/null
+++ b/mlir/unittests/Debug/ExecutionContextTest.cpp
@@ -0,0 +1,352 @@
+//===- ExecutionContextTest.cpp - Debug Execution Context first impl ------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Debug/ExecutionContext.h"
+#include "mlir/Debug/BreakpointManagers/TagBreakpointManager.h"
+#include "llvm/ADT/MapVector.h"
+#include "gmock/gmock.h"
+
+using namespace mlir;
+using namespace mlir::tracing;
+
+namespace {
+struct DebuggerAction : public ActionImpl<DebuggerAction> {
+  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(DebuggerAction)
+  static constexpr StringLiteral tag = "debugger-action";
+};
+struct OtherAction : public ActionImpl<OtherAction> {
+  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(OtherAction)
+  static constexpr StringLiteral tag = "other-action";
+};
+struct ThirdAction : public ActionImpl<ThirdAction> {
+  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(ThirdAction)
+  static constexpr StringLiteral tag = "third-action";
+};
+
+// Simple action that does nothing.
+void noOp() { return; }
+
+/// This test executes a stack of nested action and check that the backtrace is
+/// as expect.
+TEST(ExecutionContext, ActionActiveStackTest) {
+
+  // We'll break three time, once on each action, the backtraces should match
+  // each of the entries here.
+  std::vector<std::vector<StringRef>> expectedStacks = {
+      {DebuggerAction::tag},
+      {OtherAction::tag, DebuggerAction::tag},
+      {ThirdAction::tag, OtherAction::tag, DebuggerAction::tag}};
+
+  auto checkStacks = [&](const ActionActiveStack *backtrace,
+                         const std::vector<StringRef> &currentStack) {
+    ASSERT_EQ((int)currentStack.size(), backtrace->getDepth() + 1);
+    for (StringRef stackEntry : currentStack) {
+      ASSERT_NE(backtrace, nullptr);
+      ASSERT_EQ(stackEntry, backtrace->getAction().getTag());
+      backtrace = backtrace->getParent();
+    }
+  };
+
+  std::vector<ExecutionContext::Control> controlSequence = {
+      ExecutionContext::Step, ExecutionContext::Step, ExecutionContext::Apply};
+  int idx = 0;
+  StringRef current;
+  int currentDepth = -1;
+  auto onBreakpoint = [&](const ActionActiveStack *backtrace) {
+    current = backtrace->getAction().getTag();
+    currentDepth = backtrace->getDepth();
+    checkStacks(backtrace, expectedStacks[idx]);
+    return controlSequence[idx++];
+  };
+
+  TagBreakpointManager simpleManager;
+  ExecutionContext executionCtx(onBreakpoint);
+  executionCtx.addBreakpointManager(&simpleManager);
+  std::vector<TagBreakpoint *> breakpoints;
+  breakpoints.push_back(simpleManager.addBreakpoint(DebuggerAction::tag));
+  breakpoints.push_back(simpleManager.addBreakpoint(OtherAction::tag));
+  breakpoints.push_back(simpleManager.addBreakpoint(ThirdAction::tag));
+
+  auto third = [&]() {
+    EXPECT_EQ(current, ThirdAction::tag);
+    EXPECT_EQ(currentDepth, 2);
+  };
+  auto nested = [&]() {
+    EXPECT_EQ(current, OtherAction::tag);
+    EXPECT_EQ(currentDepth, 1);
+    executionCtx(third, ThirdAction{});
+  };
+  auto original = [&]() {
+    EXPECT_EQ(current, DebuggerAction::tag);
+    EXPECT_EQ(currentDepth, 0);
+    executionCtx(nested, OtherAction{});
+    return;
+  };
+
+  executionCtx(original, DebuggerAction{});
+}
+
+TEST(ExecutionContext, DebuggerTest) {
+  // Check matching and non matching breakpoints, with various enable/disable
+  // schemes.
+  int match = 0;
+  auto onBreakpoint = [&match](const ActionActiveStack *backtrace) {
+    match++;
+    return ExecutionContext::Skip;
+  };
+  TagBreakpointManager simpleManager;
+  ExecutionContext executionCtx(onBreakpoint);
+  executionCtx.addBreakpointManager(&simpleManager);
+
+  executionCtx(noOp, DebuggerAction{});
+  EXPECT_EQ(match, 0);
+
+  Breakpoint *dbgBreakpoint = simpleManager.addBreakpoint(DebuggerAction::tag);
+  executionCtx(noOp, DebuggerAction{});
+  EXPECT_EQ(match, 1);
+
+  dbgBreakpoint->disable();
+  executionCtx(noOp, DebuggerAction{});
+  EXPECT_EQ(match, 1);
+
+  dbgBreakpoint->enable();
+  executionCtx(noOp, DebuggerAction{});
+  EXPECT_EQ(match, 2);
+
+  executionCtx(noOp, OtherAction{});
+  EXPECT_EQ(match, 2);
+}
+
+TEST(ExecutionContext, ApplyTest) {
+  // Test the "apply" control.
+  std::vector<StringRef> tagSequence = {DebuggerAction::tag};
+  std::vector<ExecutionContext::Control> controlSequence = {
+      ExecutionContext::Apply};
+  int idx = 0, counter = 0;
+  auto onBreakpoint = [&](const ActionActiveStack *backtrace) {
+    ++counter;
+    EXPECT_EQ(tagSequence[idx], backtrace->getAction().getTag());
+    return controlSequence[idx++];
+  };
+  auto callback = [&]() { EXPECT_EQ(counter, 1); };
+  TagBreakpointManager simpleManager;
+  ExecutionContext executionCtx(onBreakpoint);
+  executionCtx.addBreakpointManager(&simpleManager);
+  simpleManager.addBreakpoint(DebuggerAction::tag);
+
+  executionCtx(callback, DebuggerAction{});
+  EXPECT_EQ(counter, 1);
+}
+
+TEST(ExecutionContext, SkipTest) {
+  // Test the "skip" control.
+  std::vector<StringRef> tagSequence = {DebuggerAction::tag,
+                                        DebuggerAction::tag};
+  std::vector<ExecutionContext::Control> controlSequence = {
+      ExecutionContext::Apply, ExecutionContext::Skip};
+  int idx = 0, counter = 0, executionCounter = 0;
+  auto onBreakpoint = [&](const ActionActiveStack *backtrace) {
+    ++counter;
+    EXPECT_EQ(tagSequence[idx], backtrace->getAction().getTag());
+    return controlSequence[idx++];
+  };
+  auto callback = [&]() { ++executionCounter; };
+  TagBreakpointManager simpleManager;
+  ExecutionContext executionCtx(onBreakpoint);
+  executionCtx.addBreakpointManager(&simpleManager);
+  simpleManager.addBreakpoint(DebuggerAction::tag);
+
+  executionCtx(callback, DebuggerAction{});
+  executionCtx(callback, DebuggerAction{});
+  EXPECT_EQ(counter, 2);
+  EXPECT_EQ(executionCounter, 1);
+}
+
+TEST(ExecutionContext, StepApplyTest) {
+  // Test the "step" control with a nested action.
+  std::vector<StringRef> tagSequence = {DebuggerAction::tag, OtherAction::tag};
+  std::vector<ExecutionContext::Control> controlSequence = {
+      ExecutionContext::Step, ExecutionContext::Apply};
+  int idx = 0, counter = 0;
+  auto onBreakpoint = [&](const ActionActiveStack *backtrace) {
+    ++counter;
+    EXPECT_EQ(tagSequence[idx], backtrace->getAction().getTag());
+    return controlSequence[idx++];
+  };
+  TagBreakpointManager simpleManager;
+  ExecutionContext executionCtx(onBreakpoint);
+  executionCtx.addBreakpointManager(&simpleManager);
+  simpleManager.addBreakpoint(DebuggerAction::tag);
+  auto nested = [&]() { EXPECT_EQ(counter, 2); };
+  auto original = [&]() {
+    EXPECT_EQ(counter, 1);
+    executionCtx(nested, OtherAction{});
+  };
+
+  executionCtx(original, DebuggerAction{});
+  EXPECT_EQ(counter, 2);
+}
+
+TEST(ExecutionContext, StepNothingInsideTest) {
+  // Test the "step" control without a nested action.
+  std::vector<StringRef> tagSequence = {DebuggerAction::tag,
+                                        DebuggerAction::tag};
+  std::vector<ExecutionContext::Control> controlSequence = {
+      ExecutionContext::Step, ExecutionContext::Step};
+  int idx = 0, counter = 0;
+  auto onBreakpoint = [&](const ActionActiveStack *backtrace) {
+    ++counter;
+    EXPECT_EQ(tagSequence[idx], backtrace->getAction().getTag());
+    return controlSequence[idx++];
+  };
+  auto callback = [&]() { EXPECT_EQ(counter, 1); };
+  TagBreakpointManager simpleManager;
+  ExecutionContext executionCtx(onBreakpoint);
+  executionCtx.addBreakpointManager(&simpleManager);
+  simpleManager.addBreakpoint(DebuggerAction::tag);
+
+  executionCtx(callback, DebuggerAction{});
+  EXPECT_EQ(counter, 2);
+}
+
+TEST(ExecutionContext, NextTest) {
+  // Test the "next" control.
+  std::vector<StringRef> tagSequence = {DebuggerAction::tag,
+                                        DebuggerAction::tag};
+  std::vector<ExecutionContext::Control> controlSequence = {
+      ExecutionContext::Next, ExecutionContext::Next};
+  int idx = 0, counter = 0;
+  auto onBreakpoint = [&](const ActionActiveStack *backtrace) {
+    ++counter;
+    EXPECT_EQ(tagSequence[idx], backtrace->getAction().getTag());
+    return controlSequence[idx++];
+  };
+  auto callback = [&]() { EXPECT_EQ(counter, 1); };
+  TagBreakpointManager simpleManager;
+  ExecutionContext executionCtx(onBreakpoint);
+  executionCtx.addBreakpointManager(&simpleManager);
+  simpleManager.addBreakpoint(DebuggerAction::tag);
+
+  executionCtx(callback, DebuggerAction{});
+  EXPECT_EQ(counter, 2);
+}
+
+TEST(ExecutionContext, FinishTest) {
+  // Test the "finish" control.
+  std::vector<StringRef> tagSequence = {DebuggerAction::tag, OtherAction::tag,
+                                        DebuggerAction::tag};
+  std::vector<ExecutionContext::Control> controlSequence = {
+      ExecutionContext::Step, ExecutionContext::Finish,
+      ExecutionContext::Apply};
+  int idx = 0, counter = 0;
+  auto onBreakpoint = [&](const ActionActiveStack *backtrace) {
+    ++counter;
+    EXPECT_EQ(tagSequence[idx], backtrace->getAction().getTag());
+    return controlSequence[idx++];
+  };
+  TagBreakpointManager simpleManager;
+  ExecutionContext executionCtx(onBreakpoint);
+  executionCtx.addBreakpointManager(&simpleManager);
+  simpleManager.addBreakpoint(DebuggerAction::tag);
+  auto nested = [&]() { EXPECT_EQ(counter, 2); };
+  auto original = [&]() {
+    EXPECT_EQ(counter, 1);
+    executionCtx(nested, OtherAction{});
+    EXPECT_EQ(counter, 2);
+  };
+
+  executionCtx(original, DebuggerAction{});
+  EXPECT_EQ(counter, 3);
+}
+
+TEST(ExecutionContext, FinishBreakpointInNestedTest) {
+  // Test the "finish" control with a breakpoint in the nested action.
+  std::vector<StringRef> tagSequence = {OtherAction::tag, DebuggerAction::tag};
+  std::vector<ExecutionContext::Control> controlSequence = {
+      ExecutionContext::Finish, ExecutionContext::Apply};
+  int idx = 0, counter = 0;
+  auto onBreakpoint = [&](const ActionActiveStack *backtrace) {
+    ++counter;
+    EXPECT_EQ(tagSequence[idx], backtrace->getAction().getTag());
+    return controlSequence[idx++];
+  };
+  TagBreakpointManager simpleManager;
+  ExecutionContext executionCtx(onBreakpoint);
+  executionCtx.addBreakpointManager(&simpleManager);
+  simpleManager.addBreakpoint(OtherAction::tag);
+
+  auto nested = [&]() { EXPECT_EQ(counter, 1); };
+  auto original = [&]() {
+    EXPECT_EQ(counter, 0);
+    executionCtx(nested, OtherAction{});
+    EXPECT_EQ(counter, 1);
+  };
+
+  executionCtx(original, DebuggerAction{});
+  EXPECT_EQ(counter, 2);
+}
+
+TEST(ExecutionContext, FinishNothingBackTest) {
+  // Test the "finish" control without a nested action.
+  std::vector<StringRef> tagSequence = {DebuggerAction::tag};
+  std::vector<ExecutionContext::Control> controlSequence = {
+      ExecutionContext::Finish};
+  int idx = 0, counter = 0;
+  auto onBreakpoint = [&](const ActionActiveStack *backtrace) {
+    ++counter;
+    EXPECT_EQ(tagSequence[idx], backtrace->getAction().getTag());
+    return controlSequence[idx++];
+  };
+  auto callback = [&]() { EXPECT_EQ(counter, 1); };
+  TagBreakpointManager simpleManager;
+  ExecutionContext executionCtx(onBreakpoint);
+  executionCtx.addBreakpointManager(&simpleManager);
+  simpleManager.addBreakpoint(DebuggerAction::tag);
+
+  executionCtx(callback, DebuggerAction{});
+  EXPECT_EQ(counter, 1);
+}
+
+TEST(ExecutionContext, EnableDisableBreakpointOnCallback) {
+  // Test enabling and disabling breakpoints while executing the action.
+  std::vector<StringRef> tagSequence = {DebuggerAction::tag, ThirdAction::tag,
+                                        OtherAction::tag, DebuggerAction::tag};
+  std::vector<ExecutionContext::Control> controlSequence = {
+      ExecutionContext::Apply, ExecutionContext::Finish,
+      ExecutionContext::Finish, ExecutionContext::Apply};
+  int idx = 0, counter = 0;
+  auto onBreakpoint = [&](const ActionActiveStack *backtrace) {
+    ++counter;
+    EXPECT_EQ(tagSequence[idx], backtrace->getAction().getTag());
+    return controlSequence[idx++];
+  };
+
+  TagBreakpointManager simpleManager;
+  ExecutionContext executionCtx(onBreakpoint);
+  executionCtx.addBreakpointManager(&simpleManager);
+  simpleManager.addBreakpoint(DebuggerAction::tag);
+  Breakpoint *toBeDisabled = simpleManager.addBreakpoint(OtherAction::tag);
+
+  auto third = [&]() { EXPECT_EQ(counter, 2); };
+  auto nested = [&]() {
+    EXPECT_EQ(counter, 1);
+    executionCtx(third, ThirdAction{});
+    EXPECT_EQ(counter, 2);
+  };
+  auto original = [&]() {
+    EXPECT_EQ(counter, 1);
+    toBeDisabled->disable();
+    simpleManager.addBreakpoint(ThirdAction::tag);
+    executionCtx(nested, OtherAction{});
+    EXPECT_EQ(counter, 3);
+  };
+
+  executionCtx(original, DebuggerAction{});
+  EXPECT_EQ(counter, 4);
+}
+} // namespace
diff --git a/mlir/unittests/Dialect/SparseTensor/MergerTest.cpp b/mlir/unittests/Dialect/SparseTensor/MergerTest.cpp
index 929900142d272..10d350f7c6b97 100644
--- a/mlir/unittests/Dialect/SparseTensor/MergerTest.cpp
+++ b/mlir/unittests/Dialect/SparseTensor/MergerTest.cpp
@@ -225,7 +225,7 @@ class MergerTestBase : public ::testing::Test {
     case kTensor:
       return tensorExp.tensor == pattern->tensorNum;
     case kInvariant:
-    case kIndex:
+    case kLoopVar:
       llvm_unreachable("invariant not handled yet");
     // Unary operations.
     case kAbsF:
@@ -313,15 +313,15 @@ class MergerTest3T1L : public MergerTestBase {
 
     // Tensor 0: sparse input vector.
     merger.addExp(Kind::kTensor, t0, -1u);
-    merger.setDimAndDimLevelType(t0, l0, 0, DimLevelType::Compressed);
+    merger.setLevelAndType(t0, l0, 0, DimLevelType::Compressed);
 
     // Tensor 1: sparse input vector.
     merger.addExp(Kind::kTensor, t1, -1u);
-    merger.setDimAndDimLevelType(t1, l0, 0, DimLevelType::Compressed);
+    merger.setLevelAndType(t1, l0, 0, DimLevelType::Compressed);
 
     // Tensor 2: dense output vector.
     merger.addExp(Kind::kTensor, t2, -1u);
-    merger.setDimAndDimLevelType(t2, l0, 0, DimLevelType::Dense);
+    merger.setLevelAndType(t2, l0, 0, DimLevelType::Dense);
   }
 };
 
@@ -338,19 +338,19 @@ class MergerTest4T1L : public MergerTestBase {
 
     // Tensor 0: sparse input vector.
     merger.addExp(Kind::kTensor, t0, -1u);
-    merger.setDimAndDimLevelType(t0, l0, 0, DimLevelType::Compressed);
+    merger.setLevelAndType(t0, l0, 0, DimLevelType::Compressed);
 
     // Tensor 1: sparse input vector.
     merger.addExp(Kind::kTensor, t1, -1u);
-    merger.setDimAndDimLevelType(t1, l0, 0, DimLevelType::Compressed);
+    merger.setLevelAndType(t1, l0, 0, DimLevelType::Compressed);
 
     // Tensor 2: sparse input vector
     merger.addExp(Kind::kTensor, t2, -1u);
-    merger.setDimAndDimLevelType(t2, l0, 0, DimLevelType::Compressed);
+    merger.setLevelAndType(t2, l0, 0, DimLevelType::Compressed);
 
     // Tensor 3: dense output vector
     merger.addExp(Kind::kTensor, t3, -1u);
-    merger.setDimAndDimLevelType(t3, l0, 0, DimLevelType::Dense);
+    merger.setLevelAndType(t3, l0, 0, DimLevelType::Dense);
   }
 };
 
@@ -371,15 +371,15 @@ class MergerTest3T1LD : public MergerTestBase {
 
     // Tensor 0: sparse input vector.
     merger.addExp(Kind::kTensor, t0, -1u);
-    merger.setDimAndDimLevelType(t0, l0, 0, DimLevelType::Compressed);
+    merger.setLevelAndType(t0, l0, 0, DimLevelType::Compressed);
 
     // Tensor 1: dense input vector.
     merger.addExp(Kind::kTensor, t1, -1u);
-    merger.setDimAndDimLevelType(t1, l0, 0, DimLevelType::Dense);
+    merger.setLevelAndType(t1, l0, 0, DimLevelType::Dense);
 
     // Tensor 2: dense output vector.
     merger.addExp(Kind::kTensor, t2, -1u);
-    merger.setDimAndDimLevelType(t2, l0, 0, DimLevelType::Dense);
+    merger.setLevelAndType(t2, l0, 0, DimLevelType::Dense);
   }
 };
 
@@ -400,19 +400,19 @@ class MergerTest4T1LU : public MergerTestBase {
 
     // Tensor 0: undef input vector.
     merger.addExp(Kind::kTensor, t0, -1u);
-    merger.setDimAndDimLevelType(t0, l0, 0, DimLevelType::Undef);
+    merger.setLevelAndType(t0, l0, 0, DimLevelType::Undef);
 
     // Tensor 1: dense input vector.
     merger.addExp(Kind::kTensor, t1, -1u);
-    merger.setDimAndDimLevelType(t1, l0, 0, DimLevelType::Dense);
+    merger.setLevelAndType(t1, l0, 0, DimLevelType::Dense);
 
     // Tensor 2: undef input vector.
     merger.addExp(Kind::kTensor, t2, -1u);
-    merger.setDimAndDimLevelType(t2, l0, 0, DimLevelType::Undef);
+    merger.setLevelAndType(t2, l0, 0, DimLevelType::Undef);
 
     // Tensor 3: dense output vector.
     merger.addExp(Kind::kTensor, t3, -1u);
-    merger.setDimAndDimLevelType(t3, l0, 0, DimLevelType::Dense);
+    merger.setLevelAndType(t3, l0, 0, DimLevelType::Dense);
   }
 };
 
@@ -436,15 +436,15 @@ class MergerTest3T1LSo : public MergerTestBase {
 
     // Tensor 0: undef input vector.
     merger.addExp(Kind::kTensor, t0, -1u);
-    merger.setDimAndDimLevelType(t0, l0, 0, DimLevelType::Undef);
+    merger.setLevelAndType(t0, l0, 0, DimLevelType::Undef);
 
     // Tensor 1: undef input vector.
     merger.addExp(Kind::kTensor, t1, -1u);
-    merger.setDimAndDimLevelType(t1, l0, 0, DimLevelType::Undef);
+    merger.setLevelAndType(t1, l0, 0, DimLevelType::Undef);
 
     // Tensor 2: sparse output vector.
     merger.addExp(Kind::kTensor, t2, -1u);
-    merger.setDimAndDimLevelType(t2, l0, 0, DimLevelType::Compressed);
+    merger.setLevelAndType(t2, l0, 0, DimLevelType::Compressed);
   }
 };
 
diff --git a/mlir/unittests/Support/CMakeLists.txt b/mlir/unittests/Support/CMakeLists.txt
index fd1e66205c071..09cc2c5bf1d99 100644
--- a/mlir/unittests/Support/CMakeLists.txt
+++ b/mlir/unittests/Support/CMakeLists.txt
@@ -1,6 +1,4 @@
 add_mlir_unittest(MLIRSupportTests
-  DebugActionTest.cpp
-  DebugCounterTest.cpp
   IndentedOstreamTest.cpp
   MathExtrasTest.cpp
   StorageUniquerTest.cpp
diff --git a/mlir/unittests/Support/DebugActionTest.cpp b/mlir/unittests/Support/DebugActionTest.cpp
deleted file mode 100644
index be2ca2e1b1b1a..0000000000000
--- a/mlir/unittests/Support/DebugActionTest.cpp
+++ /dev/null
@@ -1,114 +0,0 @@
-//===- DebugActionTest.cpp - Debug Action Tests ---------------------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#include "mlir/Support/DebugAction.h"
-#include "mlir/Support/TypeID.h"
-#include "gmock/gmock.h"
-
-// DebugActionManager is only enabled in DEBUG mode.
-#if LLVM_ENABLE_ABI_BREAKING_CHECKS
-
-using namespace mlir;
-
-namespace {
-struct SimpleAction : DebugAction<SimpleAction> {
-  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(SimpleAction)
-  static StringRef getTag() { return "simple-action"; }
-  static StringRef getDescription() { return "simple-action-description"; }
-};
-struct OtherSimpleAction : DebugAction<OtherSimpleAction> {
-  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(OtherSimpleAction)
-  static StringRef getTag() { return "other-simple-action"; }
-  static StringRef getDescription() {
-    return "other-simple-action-description";
-  }
-};
-struct ParametricAction : DebugAction<ParametricAction, bool> {
-  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(ParametricAction)
-  static StringRef getTag() { return "param-action"; }
-  static StringRef getDescription() { return "param-action-description"; }
-};
-
-TEST(DebugActionTest, GenericHandler) {
-  DebugActionManager manager;
-
-  // A generic handler that always executes the simple action, but not the
-  // parametric action.
-  struct GenericHandler : DebugActionManager::GenericHandler {
-    FailureOr<bool> shouldExecute(StringRef tag, StringRef desc) final {
-      if (tag == SimpleAction::getTag()) {
-        EXPECT_EQ(desc, SimpleAction::getDescription());
-        return true;
-      }
-
-      EXPECT_EQ(tag, ParametricAction::getTag());
-      EXPECT_EQ(desc, ParametricAction::getDescription());
-      return false;
-    }
-  };
-  manager.registerActionHandler<GenericHandler>();
-
-  EXPECT_TRUE(manager.shouldExecute<SimpleAction>());
-  EXPECT_FALSE(manager.shouldExecute<ParametricAction>(true));
-}
-
-TEST(DebugActionTest, ActionSpecificHandler) {
-  DebugActionManager manager;
-
-  // Handler that simply uses the input as the decider.
-  struct ActionSpecificHandler : ParametricAction::Handler {
-    FailureOr<bool> shouldExecute(bool shouldExecuteParam) final {
-      return shouldExecuteParam;
-    }
-  };
-  manager.registerActionHandler<ActionSpecificHandler>();
-
-  EXPECT_TRUE(manager.shouldExecute<ParametricAction>(true));
-  EXPECT_FALSE(manager.shouldExecute<ParametricAction>(false));
-
-  // There is no handler for the simple action, so it is always executed.
-  EXPECT_TRUE(manager.shouldExecute<SimpleAction>());
-}
-
-TEST(DebugActionTest, DebugCounterHandler) {
-  DebugActionManager manager;
-
-  // Handler that uses the number of action executions as the decider.
-  struct DebugCounterHandler : SimpleAction::Handler {
-    FailureOr<bool> shouldExecute() final { return numExecutions++ < 3; }
-    unsigned numExecutions = 0;
-  };
-  manager.registerActionHandler<DebugCounterHandler>();
-
-  // Check that the action is executed 3 times, but no more after.
-  EXPECT_TRUE(manager.shouldExecute<SimpleAction>());
-  EXPECT_TRUE(manager.shouldExecute<SimpleAction>());
-  EXPECT_TRUE(manager.shouldExecute<SimpleAction>());
-  EXPECT_FALSE(manager.shouldExecute<SimpleAction>());
-  EXPECT_FALSE(manager.shouldExecute<SimpleAction>());
-}
-
-TEST(DebugActionTest, NonOverlappingActionSpecificHandlers) {
-  DebugActionManager manager;
-
-  // One handler returns true and another returns false
-  struct SimpleActionHandler : SimpleAction::Handler {
-    FailureOr<bool> shouldExecute() final { return true; }
-  };
-  struct OtherSimpleActionHandler : OtherSimpleAction::Handler {
-    FailureOr<bool> shouldExecute() final { return false; }
-  };
-  manager.registerActionHandler<SimpleActionHandler>();
-  manager.registerActionHandler<OtherSimpleActionHandler>();
-  EXPECT_TRUE(manager.shouldExecute<SimpleAction>());
-  EXPECT_FALSE(manager.shouldExecute<OtherSimpleAction>());
-}
-
-} // namespace
-
-#endif
diff --git a/mlir/unittests/Support/DebugCounterTest.cpp b/mlir/unittests/Support/DebugCounterTest.cpp
deleted file mode 100644
index bf8d0279938c3..0000000000000
--- a/mlir/unittests/Support/DebugCounterTest.cpp
+++ /dev/null
@@ -1,46 +0,0 @@
-//===- DebugCounterTest.cpp - Debug Counter Tests -------------------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#include "mlir/Support/DebugCounter.h"
-#include "mlir/Support/TypeID.h"
-#include "gmock/gmock.h"
-
-using namespace mlir;
-
-// DebugActionManager is only enabled in DEBUG mode.
-#if LLVM_ENABLE_ABI_BREAKING_CHECKS
-
-namespace {
-
-struct CounterAction : public DebugAction<CounterAction> {
-  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(CounterAction)
-  static StringRef getTag() { return "counter-action"; }
-  static StringRef getDescription() { return "Test action for debug counters"; }
-};
-
-TEST(DebugCounterTest, CounterTest) {
-  std::unique_ptr<DebugCounter> counter = std::make_unique<DebugCounter>();
-  counter->addCounter(CounterAction::getTag(), /*countToSkip=*/1,
-                      /*countToStopAfter=*/3);
-
-  DebugActionManager manager;
-  manager.registerActionHandler(std::move(counter));
-
-  // The first execution is skipped.
-  EXPECT_FALSE(manager.shouldExecute<CounterAction>());
-
-  // The counter stops after 3 successful executions.
-  EXPECT_TRUE(manager.shouldExecute<CounterAction>());
-  EXPECT_TRUE(manager.shouldExecute<CounterAction>());
-  EXPECT_TRUE(manager.shouldExecute<CounterAction>());
-  EXPECT_FALSE(manager.shouldExecute<CounterAction>());
-}
-
-} // namespace
-
-#endif
diff --git a/openmp/docs/design/Runtimes.rst b/openmp/docs/design/Runtimes.rst
index 02424114635c8..63fa3cd53d08f 100644
--- a/openmp/docs/design/Runtimes.rst
+++ b/openmp/docs/design/Runtimes.rst
@@ -1123,8 +1123,119 @@ transformed and loaded back into the JIT pipeline via
 LLVM/OpenMP Target Host Runtime Plugins (``libomptarget.rtl.XXXX``)
 -------------------------------------------------------------------
 
-.. _device_runtime:
+The LLVM/OpenMP target host runtime plugins were recently re-implemented,
+temporarily renamed as the NextGen plugins, and set as the default and only
+plugins' implementation. Currently, these plugins have support for the NVIDIA
+and AMDGPU devices as well as the GenericELF64bit host-simulated device.
+
+The source code of the common infrastructure and the vendor-specific plugins is
+in the ``openmp/libomptarget/nextgen-plugins`` directory in the LLVM project
+repository. The plugin infrastructure aims at unifying the plugin code and logic
+into a generic interface using object-oriented C++. There is a plugin interface
+composed by multiple generic C++ classes which implement the common logic that
+every vendor-specific plugin should provide. In turn, the specific plugins
+inherit from those generic classes and implement the required functions that
+depend on the specific vendor API. As an example, some generic classes that the
+plugin interface define are for representing a device, a device image, an
+efficient resource manager, etc.
+
+With this common plugin infrastructure, several tasks have been simplified:
+adding a new vendor-specific plugin, adding generic features or optimizations
+to all plugins, debugging plugins, etc.
 
+Environment Variables
+^^^^^^^^^^^^^^^^^^^^^
+
+There are several environment variables to change the behavior of the plugins:
+
+* ``LIBOMPTARGET_SHARED_MEMORY_SIZE``
+* ``LIBOMPTARGET_STACK_SIZE``
+* ``LIBOMPTARGET_HEAP_SIZE``
+* ``LIBOMPTARGET_NUM_INITIAL_STREAMS``
+* ``LIBOMPTARGET_NUM_INITIAL_EVENTS``
+* ``LIBOMPTARGET_LOCK_MAPPED_HOST_BUFFERS``
+* ``LIBOMPTARGET_AMDGPU_NUM_HSA_QUEUES``
+* ``LIBOMPTARGET_AMDGPU_HSA_QUEUE_SIZE``
+* ``LIBOMPTARGET_AMDGPU_TEAMS_PER_CU``
+* ``LIBOMPTARGET_AMDGPU_MAX_ASYNC_COPY_BYTES``
+* ``LIBOMPTARGET_AMDGPU_NUM_INITIAL_HSA_SIGNALS``
+
+The environment variables ``LIBOMPTARGET_SHARED_MEMORY_SIZE``,
+``LIBOMPTARGET_STACK_SIZE`` and ``LIBOMPTARGET_HEAP_SIZE`` are described in
+:ref:`libopenmptarget_environment_vars`.
+
+LIBOMPTARGET_NUM_INITIAL_STREAMS
+""""""""""""""""""""""""""""""""
+
+This environment variable sets the number of pre-created streams in the plugin
+(if supported) at initialization. More streams will be created dynamically
+throughout the execution if needed. A stream is a queue of asynchronous
+operations (e.g., kernel launches and memory copies) that are executed
+sequentially. Parallelism is achieved by featuring multiple streams. The
+``libomptarget`` leverages streams to exploit parallelism between plugin
+operations. The default value is ``32``.
+
+LIBOMPTARGET_NUM_INITIAL_EVENTS
+"""""""""""""""""""""""""""""""
+
+This environment variable sets the number of pre-created events in the
+plugin (if supported) at initialization. More events will be created
+dynamically throughout the execution if needed. An event is used to synchronize
+a stream with another efficiently. The default value is ``32``.
+
+LIBOMPTARGET_LOCK_MAPPED_HOST_BUFFERS
+"""""""""""""""""""""""""""""""""""""
+
+This environment variable indicates whether the host buffers mapped by the user
+should be automatically locked/pinned by the plugin. Pinned host buffers allow
+true asynchronous copies between the host and devices. Enabling this feature can
+increase the performance of applications that are intensive in host-device
+memory transfers. The default value is ``false``.
+
+LIBOMPTARGET_AMDGPU_NUM_HSA_QUEUES
+""""""""""""""""""""""""""""""""""
+
+This environment variable controls the number of HSA queues per device in the
+AMDGPU plugin. An HSA queue is a runtime-allocated resource that contains an
+AQL (Architected Queuing Language) packet buffer and is associated with an AQL
+packet processor. HSA queues are used for inserting kernel packets to launching
+kernel executions. A high number of HSA queues may degrade the performance. The
+default value is ``4``.
+
+LIBOMPTARGET_AMDGPU_HSA_QUEUE_SIZE
+""""""""""""""""""""""""""""""""""
+
+This environment variable controls the size of each HSA queue in the AMDGPU
+plugin. The size is the number of AQL packets an HSA queue is expected to hold.
+It is also the number of AQL packets that can be pushed into each queue without
+waiting the driver to process them. The default value is ``512``.
+
+LIBOMPTARGET_AMDGPU_TEAMS_PER_CU
+""""""""""""""""""""""""""""""""
+
+This environment variable controls the default number of teams relative to the
+number of compute units (CUs) of the AMDGPU device. The default number of teams
+is ``#default_teams = #teams_per_CU * #CUs``. The default value of teams per CU
+is ``4``.
+
+LIBOMPTARGET_AMDGPU_MAX_ASYNC_COPY_BYTES
+""""""""""""""""""""""""""""""""""""""""
+
+This environment variable specifies the maximum size in bytes where the memory
+copies are asynchronous operations in the AMDGPU plugin. Up to this transfer
+size, the memory copies are asychronous operations pushed to the corresponding
+stream. For larger transfers, they are synchronous transfers. Memory copies
+involving already locked/pinned host buffers are always asychronous. The default
+value is ``1*1024*1024`` bytes (1 MB).
+
+LIBOMPTARGET_AMDGPU_NUM_INITIAL_HSA_SIGNALS
+"""""""""""""""""""""""""""""""""""""""""""
+
+This environment variable controls the initial number of HSA signals per device
+in the AMDGPU plugin. There is one resource manager of signals per device
+managing several pre-created signals. These signals are mainly used by AMDGPU
+streams. More HSA signals will be created dynamically throughout the execution
+if needed. The default value is ``64``.
 
 .. _remote_offloading_plugin:
 
diff --git a/openmp/libomptarget/DeviceRTL/CMakeLists.txt b/openmp/libomptarget/DeviceRTL/CMakeLists.txt
index 6844e885fa6d0..53da2661011e5 100644
--- a/openmp/libomptarget/DeviceRTL/CMakeLists.txt
+++ b/openmp/libomptarget/DeviceRTL/CMakeLists.txt
@@ -56,33 +56,29 @@ set(devicertl_base_directory ${CMAKE_CURRENT_SOURCE_DIR})
 set(include_directory ${devicertl_base_directory}/include)
 set(source_directory ${devicertl_base_directory}/src)
 
-set(all_capabilities 35 37 50 52 53 60 61 62 70 72 75 80 86 89 90)
-
-set(LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES ${all_capabilities} CACHE STRING
-  "List of CUDA Compute Capabilities to be used to compile the NVPTX DeviceRTL.")
-string(TOLOWER ${LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES} LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES)
-
-if (LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES STREQUAL "all")
-  set(nvptx_sm_list ${all_capabilities})
-elseif(LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES STREQUAL "auto")
-  if (NOT LIBOMPTARGET_DEP_CUDA_FOUND)
-    libomptarget_error_say("[NVPTX] Cannot auto detect compute capability as CUDA not found.")
-  endif()
-  set(nvptx_sm_list ${LIBOMPTARGET_DEP_CUDA_ARCH})
-else()
-  string(REPLACE "," ";" nvptx_sm_list "${LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES}")
-endif()
-
-# Check all SM values
-foreach(sm ${nvptx_sm_list})
-  if (NOT ${sm} IN_LIST all_capabilities)
-    libomptarget_warning_say("[NVPTX] Compute capability ${sm} is not supported. Make sure clang can work with it.")
+set(all_amdgpu_architectures "gfx700;gfx701;gfx801;gfx803;gfx900;gfx902;gfx906"
+                             "gfx908;gfx90a;gfx90c;gfx940;gfx1010;gfx1030"
+                             "gfx1031;gfx1032;gfx1033;gfx1034;gfx1035;gfx1036"
+                             "gfx1100;gfx1101;gfx1102;gfx1103")
+set(all_nvptx_architectures "sm_35;sm_37;sm_50;sm_52;sm_53;sm_60;sm_61;sm_62"
+                            "sm_70;sm_72;sm_75;sm_80;sm_86;sm_89;sm_90")
+set(all_gpu_architectures
+    "${all_amdgpu_architectures};${all_nvptx_architectures}")
+
+set(LIBOMPTARGET_DEVICE_ARCHITECTURES "all" CACHE STRING
+    "List of device architectures to be used to compile the OpenMP DeviceRTL.")
+
+if(LIBOMPTARGET_DEVICE_ARCHITECTURES STREQUAL "all")
+  set(LIBOMPTARGET_DEVICE_ARCHITECTURES ${all_gpu_architectures})
+elseif(LIBOMPTARGET_DEVICE_ARCHITECTURES STREQUAL "auto")
+  if(NOT LIBOMPTARGET_NVPTX_ARCH AND NOT LIBOMPTARGET_AMDGPU_ARCH)
+    libomptarget_error_say(
+      "Could not find 'amdgpu-arch' and 'nvptx-arch' tools required for 'auto'")
+  elseif(NOT LIBOMPTARGET_FOUND_NVIDIA_GPU AND NOT LIBOMPTARGET_FOUND_AMDGPU_GPU)
+    libomptarget_error_say("No AMD or NVIDIA GPU found on the system when using 'auto'")
   endif()
-endforeach()
-
-set(amdgpu_mcpus gfx700 gfx701 gfx801 gfx803 gfx900 gfx902 gfx906 gfx908 gfx90a gfx90c gfx940 gfx1010 gfx1030 gfx1031 gfx1032 gfx1033 gfx1034 gfx1035 gfx1036 gfx1100 gfx1101 gfx1102 gfx1103)
-if (DEFINED LIBOMPTARGET_AMDGCN_GFXLIST)
-  set(amdgpu_mcpus ${LIBOMPTARGET_AMDGCN_GFXLIST})
+  set(LIBOMPTARGET_DEVICE_ARCHITECTURES
+      "${LIBOMPTARGET_NVPTX_DETECTED_ARCH_LIST};${LIBOMPTARGET_AMDGPU_DETECTED_ARCH_LIST}")
 endif()
 
 set(include_files
@@ -272,15 +268,17 @@ function(compileDeviceRTLLibrary target_cpu target_name target_triple)
   set_property(TARGET omptarget.devicertl.all_objs APPEND PROPERTY IMPORTED_OBJECTS ${output_name})
 endfunction()
 
-# Generate a Bitcode library for all the compute capabilities the user requested
+# Generate a Bitcode library for all the gpu architectures the user requested.
 add_custom_target(omptarget.devicertl.nvptx)
-foreach(sm ${nvptx_sm_list})
-  compileDeviceRTLLibrary(sm_${sm} nvptx nvptx64-nvidia-cuda -fopenmp-targets=nvptx64-nvidia-cuda --cuda-feature=+ptx61)
-endforeach()
-
 add_custom_target(omptarget.devicertl.amdgpu)
-foreach(mcpu ${amdgpu_mcpus})
-  compileDeviceRTLLibrary(${mcpu} amdgpu amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa)
+foreach(gpu_arch ${LIBOMPTARGET_DEVICE_ARCHITECTURES})
+  if("${gpu_arch}" IN_LIST all_amdgpu_architectures)
+    compileDeviceRTLLibrary(${gpu_arch} amdgpu amdgcn-amd-amdhsa)
+  elseif("${gpu_arch}" IN_LIST all_nvptx_architectures)
+    compileDeviceRTLLibrary(${gpu_arch} nvptx nvptx64-nvidia-cuda --cuda-feature=+ptx61)
+  else()
+    libomptarget_error_say("Unknown GPU architecture '${gpu_arch}'")
+  endif()
 endforeach()
 
 # Archive all the object files generated above into a static library
diff --git a/openmp/libomptarget/cmake/Modules/LibomptargetGetDependencies.cmake b/openmp/libomptarget/cmake/Modules/LibomptargetGetDependencies.cmake
index fb4935a61c565..1db8423ec06b4 100644
--- a/openmp/libomptarget/cmake/Modules/LibomptargetGetDependencies.cmake
+++ b/openmp/libomptarget/cmake/Modules/LibomptargetGetDependencies.cmake
@@ -115,12 +115,11 @@ if(LIBOMPTARGET_NVPTX_ARCH)
   execute_process(COMMAND ${LIBOMPTARGET_NVPTX_ARCH}
                   OUTPUT_VARIABLE LIBOMPTARGET_NVPTX_ARCH_OUTPUT
                   OUTPUT_STRIP_TRAILING_WHITESPACE)
-  string(FIND "${LIBOMPTARGET_NVPTX_ARCH_OUTPUT}" "\n" first_arch_string)
-  string(SUBSTRING "${LIBOMPTARGET_NVPTX_ARCH_OUTPUT}" 0 ${first_arch_string}
-         arch_string)
-  if(arch_string)
+  string(REPLACE "\n" ";" nvptx_arch_list "${LIBOMPTARGET_NVPTX_ARCH_OUTPUT}")
+  if(nvptx_arch_list)
     set(LIBOMPTARGET_FOUND_NVIDIA_GPU TRUE)
-    set(LIBOMPTARGET_DEP_CUDA_ARCH "${arch_string}")
+    set(LIBOMPTARGET_NVPTX_DETECTED_ARCH_LIST "${nvptx_arch_list}")
+    list(GET nvptx_arch_list 0 LIBOMPTARGET_DEP_CUDA_ARCH)
   endif()
 endif()
 
@@ -134,12 +133,10 @@ if(LIBOMPTARGET_AMDGPU_ARCH)
   execute_process(COMMAND ${LIBOMPTARGET_AMDGPU_ARCH}
                   OUTPUT_VARIABLE LIBOMPTARGET_AMDGPU_ARCH_OUTPUT
                   OUTPUT_STRIP_TRAILING_WHITESPACE)
-  string(FIND "${LIBOMPTARGET_AMDGPU_ARCH_OUTPUT}" "\n" first_arch_string)
-  string(SUBSTRING "${LIBOMPTARGET_AMDGPU_ARCH_OUTPUT}" 0 ${first_arch_string}
-         arch_string)
-  if(arch_string)
+  string(REPLACE "\n" ";" amdgpu_arch_list "${LIBOMPTARGET_AMDGPU_ARCH_OUTPUT}")
+  if(amdgpu_arch_list)
     set(LIBOMPTARGET_FOUND_AMDGPU_GPU TRUE)
-    set(LIBOMPTARGET_DEP_AMDGPU_ARCH "${arch_string}")
+    set(LIBOMPTARGET_AMDGPU_DETECTED_ARCH_LIST "${amdgpu_arch_list}")
   endif()
 endif()
 
diff --git a/openmp/libomptarget/plugins-nextgen/amdgpu/src/rtl.cpp b/openmp/libomptarget/plugins-nextgen/amdgpu/src/rtl.cpp
index 99b45ad386d8a..e03825651286d 100644
--- a/openmp/libomptarget/plugins-nextgen/amdgpu/src/rtl.cpp
+++ b/openmp/libomptarget/plugins-nextgen/amdgpu/src/rtl.cpp
@@ -2614,33 +2614,39 @@ Error AMDGPUKernelTy::printLaunchInfoDetails(GenericDeviceTy &GenericDevice,
     return Plugin::success();
 
   // General Info
-  auto ConstWGSize = getDefaultNumThreads(GenericDevice);
   auto NumGroups = NumBlocks;
-  auto ThreadsPerGroup = getDefaultNumThreads(GenericDevice);
-  auto NumTeams = KernelArgs.NumTeams[0];       // Only first dimension
-  auto ThreadLimit = KernelArgs.ThreadLimit[0]; // Only first dimension
+  auto ThreadsPerGroup = NumThreads;
 
   // Kernel Arguments Info
   auto ArgNum = KernelArgs.NumArgs;
   auto LoopTripCount = KernelArgs.Tripcount;
 
-  // Details for AMDGPU kernels
+  // Details for AMDGPU kernels (read from image)
+  // https://www.llvm.org/docs/AMDGPUUsage.html#code-object-v4-metadata
   auto GroupSegmentSize = (*KernelInfo).GroupSegmentList;
   auto SGPRCount = (*KernelInfo).SGPRCount;
   auto VGPRCount = (*KernelInfo).VGPRCount;
   auto SGPRSpillCount = (*KernelInfo).SGPRSpillCount;
   auto VGPRSpillCount = (*KernelInfo).VGPRSpillCount;
-
-  // TODO set correctly once host services available
-  auto HostCallRequired = false;
+  auto MaxFlatWorkgroupSize = (*KernelInfo).MaxFlatWorkgroupSize;
+
+  // Prints additional launch info that contains the following.
+  // Num Args: The number of kernel arguments
+  // Teams x Thrds: The number of teams and the number of threads actually
+  // running.
+  // MaxFlatWorkgroupSize: Maximum flat work-group size supported by the
+  // kernel in work-items
+  // LDS Usage: Amount of bytes used in LDS storage
+  // S/VGPR Count: the number of S/V GPRs occupied by the kernel
+  // S/VGPR Spill Count: how many S/VGPRs are spilled by the kernel
+  // Tripcount: loop tripcount for the kernel
   INFO(OMP_INFOTYPE_PLUGIN_KERNEL, GenericDevice.getDeviceId(),
-       "SGN:%s ConstWGSize:%d args:%d teamsXthrds:(%4luX%4d) "
-       "reqd:(%4dX%4d) lds_usage:%uB sgpr_count:%u vgpr_count:%u "
-       "sgpr_spill_count:%u vgpr_spill_count:%u tripcount:%lu rpc:%d n:%s\n",
-       getExecutionModeName(), ConstWGSize, ArgNum, NumGroups, ThreadsPerGroup,
-       NumTeams, ThreadLimit, GroupSegmentSize, SGPRCount, VGPRCount,
-       SGPRSpillCount, VGPRSpillCount, LoopTripCount, HostCallRequired,
-       getName());
+       "#Args: %d Teams x Thrds: %4lux%4u (MaxFlatWorkGroupSize: %u) LDS "
+       "Usage: %uB #SGPRs/VGPRs: %u/%u #SGPR/VGPR Spills: %u/%u Tripcount: "
+       "%lu\n",
+       ArgNum, NumGroups, ThreadsPerGroup, MaxFlatWorkgroupSize,
+       GroupSegmentSize, SGPRCount, VGPRCount, SGPRSpillCount, VGPRSpillCount,
+       LoopTripCount);
 
   return Plugin::success();
 }
diff --git a/openmp/libomptarget/plugins-nextgen/amdgpu/utils/UtilitiesRTL.h b/openmp/libomptarget/plugins-nextgen/amdgpu/utils/UtilitiesRTL.h
index 1c549189b5c4d..cdf1d10980e9a 100644
--- a/openmp/libomptarget/plugins-nextgen/amdgpu/utils/UtilitiesRTL.h
+++ b/openmp/libomptarget/plugins-nextgen/amdgpu/utils/UtilitiesRTL.h
@@ -146,6 +146,10 @@ struct KernelMetaDataTy {
   uint32_t KernelSegmentSize;
   uint32_t ExplicitArgumentCount;
   uint32_t ImplicitArgumentCount;
+  uint32_t RequestedWorkgroupSize[3];
+  uint32_t WorkgroupSizeHint[3];
+  uint32_t WavefronSize;
+  uint32_t MaxFlatWorkgroupSize;
 };
 namespace {
 
@@ -194,6 +198,19 @@ class KernelInfoReader {
       return DK.getString() == SK;
     };
 
+    const auto getSequenceOfThreeInts = [](msgpack::DocNode &DN,
+                                           uint32_t *Vals) {
+      assert(DN.isArray() && "MsgPack DocNode is an array node");
+      auto DNA = DN.getArray();
+      assert(DNA.size() == 3 && "ArrayNode has at most three elements");
+
+      int i = 0;
+      for (auto DNABegin = DNA.begin(), DNAEnd = DNA.end(); DNABegin != DNAEnd;
+           ++DNABegin) {
+        Vals[i++] = DNABegin->getUInt();
+      }
+    };
+
     if (isKey(V.first, ".name")) {
       KernelName = V.second.toString();
     } else if (isKey(V.first, ".sgpr_count")) {
@@ -208,6 +225,14 @@ class KernelInfoReader {
       KernelData.PrivateSegmentSize = V.second.getUInt();
     } else if (isKey(V.first, ".group_segement_fixed_size")) {
       KernelData.GroupSegmentList = V.second.getUInt();
+    } else if (isKey(V.first, ".reqd_workgroup_size")) {
+      getSequenceOfThreeInts(V.second, KernelData.RequestedWorkgroupSize);
+    } else if (isKey(V.first, ".workgroup_size_hint")) {
+      getSequenceOfThreeInts(V.second, KernelData.WorkgroupSizeHint);
+    } else if (isKey(V.first, ".wavefront_size")) {
+      KernelData.WavefronSize = V.second.getUInt();
+    } else if (isKey(V.first, ".max_flat_workgroup_size")) {
+      KernelData.MaxFlatWorkgroupSize = V.second.getUInt();
     }
 
     return Error::success();
@@ -295,6 +320,7 @@ Error readAMDGPUMetaDataFromImage(MemoryBufferRef MemBuffer,
 
   return Error::success();
 }
+
 } // namespace utils
 } // namespace plugin
 } // namespace target
diff --git a/openmp/libomptarget/plugins-nextgen/common/PluginInterface/JIT.cpp b/openmp/libomptarget/plugins-nextgen/common/PluginInterface/JIT.cpp
index 3983f7044dcf7..bdccefc616712 100644
--- a/openmp/libomptarget/plugins-nextgen/common/PluginInterface/JIT.cpp
+++ b/openmp/libomptarget/plugins-nextgen/common/PluginInterface/JIT.cpp
@@ -214,11 +214,7 @@ void JITEngine::opt(TargetMachine *TM, TargetLibraryInfoImpl *TLII, Module &M,
   PB.registerLoopAnalyses(LAM);
   PB.crossRegisterProxies(LAM, FAM, CGAM, MAM);
 
-  if (OptLevel)
-    MPM.addPass(PB.buildPerModuleDefaultPipeline(getOptLevel(OptLevel)));
-  else
-    MPM.addPass(PB.buildO0DefaultPipeline(getOptLevel(OptLevel)));
-
+  MPM.addPass(PB.buildPerModuleDefaultPipeline(getOptLevel(OptLevel)));
   MPM.run(M, MAM);
 }
 
diff --git a/openmp/libomptarget/src/omptarget.cpp b/openmp/libomptarget/src/omptarget.cpp
index 9d800d75f0f49..392a9f79bd760 100644
--- a/openmp/libomptarget/src/omptarget.cpp
+++ b/openmp/libomptarget/src/omptarget.cpp
@@ -274,6 +274,17 @@ void handleTargetOutcome(bool Success, ident_t *Loc) {
         FAILURE_MESSAGE("Consult https://openmp.llvm.org/design/Runtimes.html "
                         "for debugging options.\n");
 
+      if (PM->RTLs.UsedRTLs.empty()) {
+        llvm::SmallVector<llvm::StringRef> Archs;
+        llvm::transform(PM->Images, std::back_inserter(Archs),
+                        [](const auto &x) {
+                          return !x.second.Arch ? "empty" : x.second.Arch;
+                        });
+        FAILURE_MESSAGE(
+            "No images found compatible with the installed hardware. ");
+        fprintf(stderr, "Found (%s)\n", llvm::join(Archs, ",").c_str());
+      }
+
       SourceInfo Info(Loc);
       if (Info.isAvailible())
         fprintf(stderr, "%s:%d:%d: ", Info.getFilename(), Info.getLine(),
diff --git a/openmp/libomptarget/test/lit.cfg b/openmp/libomptarget/test/lit.cfg
index a10e1e8045884..33138e910d5fc 100644
--- a/openmp/libomptarget/test/lit.cfg
+++ b/openmp/libomptarget/test/lit.cfg
@@ -80,6 +80,9 @@ for feature in config.test_compiler_features:
 if config.libomptarget_debug:
   config.available_features.add('libomptarget-debug')
 
+if config.has_libomptarget_ompt:
+  config.available_features.add('ompt')
+
 config.available_features.add(config.libomptarget_current_target)
 
 # Determine whether the test system supports unified memory.
@@ -100,8 +103,6 @@ if supports_unified_shared_memory:
    config.available_features.add('unified_shared_memory')
 
 # Setup environment to find dynamic library at runtime
-# Disable the implicit path to increase certainty over which library is picked up
-config.test_flags += " -fno-openmp-implicit-rpath"
 if config.operating_system == 'Windows':
     append_dynamic_library_path('PATH', config.library_dir, ";")
     append_dynamic_library_path('PATH', config.omp_host_rtl_directory, ";")
diff --git a/openmp/libomptarget/test/lit.site.cfg.in b/openmp/libomptarget/test/lit.site.cfg.in
index d56307254f794..e388a801e6c9c 100644
--- a/openmp/libomptarget/test/lit.site.cfg.in
+++ b/openmp/libomptarget/test/lit.site.cfg.in
@@ -19,6 +19,7 @@ config.libomptarget_current_target = "@CURRENT_TARGET@"
 config.libomptarget_filecheck = "@OPENMP_FILECHECK_EXECUTABLE@"
 config.libomptarget_not = "@OPENMP_NOT_EXECUTABLE@"
 config.libomptarget_debug = @LIBOMPTARGET_DEBUG@
+config.has_libomptarget_ompt = @LIBOMPTARGET_OMPT_SUPPORT@
 
 # Let the main config do the real work.
 lit_config.load_config(config, "@CMAKE_CURRENT_SOURCE_DIR@/lit.cfg")
diff --git a/openmp/libomptarget/test/mapping/target_derefence_array_pointrs.cpp b/openmp/libomptarget/test/mapping/target_derefence_array_pointrs.cpp
new file mode 100644
index 0000000000000..7d4bc9ea22478
--- /dev/null
+++ b/openmp/libomptarget/test/mapping/target_derefence_array_pointrs.cpp
@@ -0,0 +1,107 @@
+// RUN: %libomptarget-compilexx-generic -fopenmp-version=51
+// RUN: %libomptarget-run-generic 2>&1 \
+// RUN: | %fcheck-generic
+
+// UNSUPPORTED: amdgcn-amd-amdhsa
+
+#include <stdio.h>
+#include <stdlib.h>
+
+void foo(int **t1d) {
+  int ***t2d = &t1d;
+  int ****t3d = &t2d;
+  *t1d = (int *)malloc(3 * sizeof(int));
+  int j, a = 0, b = 0;
+
+  for (j = 0; j < 3; j++)
+    (*t1d)[j] = 0;
+#pragma omp target map(tofrom : (*t1d)[0 : 3])
+  { (*t1d)[1] = 1; }
+  // CHECK: 1
+  printf("%d\n", (*t1d)[1]);
+#pragma omp target map(tofrom : (**t2d)[0 : 3])
+  { (**t2d)[1] = 2; }
+  // CHECK: 2
+  printf("%d\n", (**t2d)[1]);
+#pragma omp target map(tofrom : (***t3d)[0 : 3])
+  { (***t3d)[1] = 3; }
+  // CHECK: 3
+  printf("%d\n", (***t3d)[1]);
+#pragma omp target map(tofrom : (**t1d))
+  { (*t1d)[0] = 4; }
+  // CHECK: 4
+  printf("%d\n", (*t1d)[0]);
+#pragma omp target map(tofrom : (*(*(t1d + a) + b)))
+  { *(*(t1d + a) + b) = 5; }
+  // CHECK: 5
+  printf("%d\n", *(*(t1d + a) + b));
+}
+
+typedef int(T)[3];
+void bar() {
+  T **a;
+  int b[2][3];
+  int(*p)[3] = b;
+  a = &p;
+  for (int i = 0; i < 3; i++) {
+    (**a)[1] = i;
+  }
+#pragma omp target map((**a)[ : 3])
+  {
+    (**a)[1] = 6;
+    // CHECK: 6
+    printf("%d\n", (**a)[1]);
+  }
+}
+
+struct SSA {
+  int i;
+  SSA *sa;
+  SSA() {
+    i = 1;
+    sa = this;
+  }
+};
+
+void zoo(int **f, SSA *sa) {
+  int *t = *f;
+  f = (int **)malloc(sa->i * 4 * sizeof(int));
+  t = (int *)malloc(sa->i * sizeof(int));
+  *(f + sa->i + 1) = t;
+  *(sa->sa->i + *(f + sa->i + 1)) = 4;
+  printf("%d\n", *(sa->sa->i + *(1 + sa->i + f)));
+#pragma omp target map(sa, *(sa->sa->i + *(1 + sa->i + f)))
+  { *(sa->sa->i + *(1 + sa->i + f)) = 7; }
+  // CHECK: 7
+  printf("%d\n", *(sa->sa->i + *(1 + sa->i + f)));
+}
+
+void xoo() {
+  int *x = 0;
+  SSA *sa = new SSA();
+  zoo(&x, sa);
+}
+
+void yoo(int **x) {
+  *x = (int *)malloc(2 * sizeof(int));
+#pragma omp target map(**x)
+  {
+    **x = 8;
+    // CHECK: 8
+    printf("%d\n", **x);
+  }
+#pragma omp target map(*(*x + 1))
+  {
+    *(*x + 1) = 9;
+    // CHECK: 9
+    printf("%d\n", *(*x + 1));
+  }
+}
+
+int main() {
+  int *data = 0;
+  foo(&data);
+  bar();
+  xoo();
+  yoo(&data);
+}
diff --git a/openmp/libomptarget/test/mapping/target_map_for_member_data.cpp b/openmp/libomptarget/test/mapping/target_map_for_member_data.cpp
index 310798c259114..fafff6b18a2bc 100644
--- a/openmp/libomptarget/test/mapping/target_map_for_member_data.cpp
+++ b/openmp/libomptarget/test/mapping/target_map_for_member_data.cpp
@@ -68,11 +68,13 @@ class C : public BASE {
     auto Asize = 4;
     auto Csize = 4;
 
-#pragma omp target data map(to : d.A) map(from : d.C)
+#pragma omp target data map(from : d.C)
     {
 #pragma omp target teams firstprivate(Csize)
       d.C = 1;
     }
+#pragma omp target map(from : d.A)
+    d.A = 3;
   }
 };
 
@@ -91,4 +93,6 @@ int main(int argc, char *argv[]) {
   z.bar(d);
   // CHECK 1
   printf("%d\n", d.C);
+  // CHECK 3
+  printf("%d\n", d.A);
 }
diff --git a/openmp/libomptarget/test/offloading/info.c b/openmp/libomptarget/test/offloading/info.c
index f24727400bcdb..c2f8450064106 100644
--- a/openmp/libomptarget/test/offloading/info.c
+++ b/openmp/libomptarget/test/offloading/info.c
@@ -39,7 +39,7 @@ int main() {
 // INFO: info: Entering OpenMP kernel at info.c:{{[0-9]+}}:{{[0-9]+}} with 1 arguments:
 // INFO: info: firstprivate(val)[4]
 // INFO: info: Launching kernel __omp_offloading_{{.*}}main{{.*}} with {{[0-9]+}} blocks and {{[0-9]+}} threads in Generic mode
-// AMDGPU: AMDGPU device {{[0-9]}} info: SGN:Generic ConstWGSize:{{[0-9]+}} args:{{[0-9]}} teamsXthrds:({{   [0-9]+}}X {{[0-9]+}}) reqd:(   {{[0-9]+}}X   {{[0-9]+}}) lds_usage:{{[0-9]+}}B sgpr_count:{{[0-9]+}} vgpr_count:{{[0-9]+}} sgpr_spill_count:{{[0-9]+}} vgpr_spill_count:{{[0-9]+}} tripcount:{{[0-9]+}} rpc:0 n:__omp_offloading_{{.*}}main{{.*}}
+// AMDGPU: AMDGPU device {{[0-9]}} info: #Args: {{[0-9]}} Teams x Thrds: {{[0-9]+}}x {{[0-9]+}} (MaxFlatWorkGroupSize: {{[0-9]+}}) LDS Usage: {{[0-9]+}}B #SGPRs/VGPRs: {{[0-9]+}}/{{[0-9]+}} #SGPR/VGPR Spills: {{[0-9]+}}/{{[0-9]+}} Tripcount: {{[0-9]+}}
 // INFO: info: OpenMP Host-Device pointer mappings after block at info.c:{{[0-9]+}}:{{[0-9]+}}:
 // INFO: info: Host Ptr           Target Ptr         Size (B) DynRefCount HoldRefCount Declaration
 // INFO: info: {{.*}}             {{.*}}             256      1           0            C[0:64] at info.c:{{[0-9]+}}:{{[0-9]+}}
diff --git a/openmp/libomptarget/test/ompt/callbacks.h b/openmp/libomptarget/test/ompt/callbacks.h
new file mode 100644
index 0000000000000..083006f756def
--- /dev/null
+++ b/openmp/libomptarget/test/ompt/callbacks.h
@@ -0,0 +1,129 @@
+#include <assert.h>
+#include <stdlib.h>
+
+// Tool related code below
+#include <omp-tools.h>
+
+// For EMI callbacks
+ompt_id_t next_op_id = 0x8000000000000001;
+
+// OMPT callbacks
+
+// Synchronous callbacks
+static void on_ompt_callback_device_initialize(int device_num, const char *type,
+                                               ompt_device_t *device,
+                                               ompt_function_lookup_t lookup,
+                                               const char *documentation) {
+  printf("Callback Init: device_num=%d type=%s device=%p lookup=%p doc=%p\n",
+         device_num, type, device, lookup, documentation);
+}
+
+static void on_ompt_callback_device_finalize(int device_num) {
+  printf("Callback Fini: device_num=%d\n", device_num);
+}
+
+static void on_ompt_callback_device_load(int device_num, const char *filename,
+                                         int64_t offset_in_file,
+                                         void *vma_in_file, size_t bytes,
+                                         void *host_addr, void *device_addr,
+                                         uint64_t module_id) {
+  printf("Callback Load: device_num:%d module_id:%lu filename:%s host_adddr:%p "
+         "device_addr:%p bytes:%lu\n",
+         device_num, module_id, filename, host_addr, device_addr, bytes);
+}
+
+static void on_ompt_callback_target_data_op(
+    ompt_id_t target_id, ompt_id_t host_op_id, ompt_target_data_op_t optype,
+    void *src_addr, int src_device_num, void *dest_addr, int dest_device_num,
+    size_t bytes, const void *codeptr_ra) {
+  assert(codeptr_ra != 0 && "Unexpected null codeptr");
+  // Both src and dest must not be null
+  assert((src_addr != 0 || dest_addr != 0) && "Both src and dest addr null");
+  printf("  Callback DataOp: target_id=%lu host_op_id=%lu optype=%d src=%p "
+         "src_device_num=%d "
+         "dest=%p dest_device_num=%d bytes=%lu code=%p\n",
+         target_id, host_op_id, optype, src_addr, src_device_num, dest_addr,
+         dest_device_num, bytes, codeptr_ra);
+}
+
+static void on_ompt_callback_target(ompt_target_t kind,
+                                    ompt_scope_endpoint_t endpoint,
+                                    int device_num, ompt_data_t *task_data,
+                                    ompt_id_t target_id,
+                                    const void *codeptr_ra) {
+  assert(codeptr_ra != 0 && "Unexpected null codeptr");
+  printf("Callback Target: target_id=%lu kind=%d endpoint=%d device_num=%d "
+         "code=%p\n",
+         target_id, kind, endpoint, device_num, codeptr_ra);
+}
+
+static void on_ompt_callback_target_submit(ompt_id_t target_id,
+                                           ompt_id_t host_op_id,
+                                           unsigned int requested_num_teams) {
+  printf("  Callback Submit: target_id=%lu host_op_id=%lu req_num_teams=%d\n",
+         target_id, host_op_id, requested_num_teams);
+}
+
+static void on_ompt_callback_target_map(ompt_id_t target_id,
+                                        unsigned int nitems, void **host_addr,
+                                        void **device_addr, size_t *bytes,
+                                        unsigned int *mapping_flags,
+                                        const void *codeptr_ra) {
+  printf("Target map callback is unimplemented\n");
+  abort();
+}
+
+static void on_ompt_callback_target_data_op_emi(
+    ompt_scope_endpoint_t endpoint, ompt_data_t *target_task_data,
+    ompt_data_t *target_data, ompt_id_t *host_op_id,
+    ompt_target_data_op_t optype, void *src_addr, int src_device_num,
+    void *dest_addr, int dest_device_num, size_t bytes,
+    const void *codeptr_ra) {
+  assert(codeptr_ra != 0 && "Unexpected null codeptr");
+  // Both src and dest must not be null
+  assert((src_addr != 0 || dest_addr != 0) && "Both src and dest addr null");
+  if (endpoint == ompt_scope_begin)
+    *host_op_id = next_op_id++;
+  printf("  Callback DataOp EMI: endpoint=%d optype=%d target_task_data=%p "
+         "(0x%lx) target_data=%p (0x%lx) host_op_id=%p (0x%lx) src=%p "
+         "src_device_num=%d "
+         "dest=%p dest_device_num=%d bytes=%lu code=%p\n",
+         endpoint, optype, target_task_data, target_task_data->value,
+         target_data, target_data->value, host_op_id, *host_op_id, src_addr,
+         src_device_num, dest_addr, dest_device_num, bytes, codeptr_ra);
+}
+
+static void on_ompt_callback_target_emi(ompt_target_t kind,
+                                        ompt_scope_endpoint_t endpoint,
+                                        int device_num, ompt_data_t *task_data,
+                                        ompt_data_t *target_task_data,
+                                        ompt_data_t *target_data,
+                                        const void *codeptr_ra) {
+  assert(codeptr_ra != 0 && "Unexpected null codeptr");
+  if (endpoint == ompt_scope_begin)
+    target_data->value = next_op_id++;
+  printf("Callback Target EMI: kind=%d endpoint=%d device_num=%d task_data=%p "
+         "(0x%lx) target_task_data=%p (0x%lx) target_data=%p (0x%lx) code=%p\n",
+         kind, endpoint, device_num, task_data, task_data->value,
+         target_task_data, target_task_data->value, target_data,
+         target_data->value, codeptr_ra);
+}
+
+static void on_ompt_callback_target_submit_emi(
+    ompt_scope_endpoint_t endpoint, ompt_data_t *target_data,
+    ompt_id_t *host_op_id, unsigned int requested_num_teams) {
+  printf("  Callback Submit EMI: endpoint=%d  req_num_teams=%d target_data=%p "
+         "(0x%lx) host_op_id=%p (0x%lx)\n",
+         endpoint, requested_num_teams, target_data, target_data->value,
+         host_op_id, *host_op_id);
+}
+
+static void on_ompt_callback_target_map_emi(ompt_data_t *target_data,
+                                            unsigned int nitems,
+                                            void **host_addr,
+                                            void **device_addr, size_t *bytes,
+                                            unsigned int *mapping_flags,
+                                            const void *codeptr_ra) {
+  printf("Target map emi callback is unimplemented\n");
+  abort();
+}
diff --git a/openmp/libomptarget/test/ompt/register_both.h b/openmp/libomptarget/test/ompt/register_both.h
new file mode 100644
index 0000000000000..afdf094bb4eff
--- /dev/null
+++ b/openmp/libomptarget/test/ompt/register_both.h
@@ -0,0 +1,49 @@
+#include <omp-tools.h>
+
+// From openmp/runtime/test/ompt/callback.h
+#define register_ompt_callback_t(name, type)                                   \
+  do {                                                                         \
+    type f_##name = &on_##name;                                                \
+    if (ompt_set_callback(name, (ompt_callback_t)f_##name) == ompt_set_never)  \
+      printf("0: Could not register callback '" #name "'\n");                  \
+  } while (0)
+
+#define register_ompt_callback(name) register_ompt_callback_t(name, name##_t)
+
+// OMPT entry point handles
+static ompt_set_callback_t ompt_set_callback = 0;
+
+// Init functions
+int ompt_initialize(ompt_function_lookup_t lookup, int initial_device_num,
+                    ompt_data_t *tool_data) {
+  ompt_set_callback = (ompt_set_callback_t)lookup("ompt_set_callback");
+
+  if (!ompt_set_callback)
+    return 0; // failed
+
+  register_ompt_callback(ompt_callback_device_initialize);
+  register_ompt_callback(ompt_callback_device_finalize);
+  register_ompt_callback(ompt_callback_device_load);
+  register_ompt_callback(ompt_callback_target_data_op_emi);
+  register_ompt_callback(ompt_callback_target_data_op);
+  register_ompt_callback(ompt_callback_target);
+  register_ompt_callback(ompt_callback_target_emi);
+  register_ompt_callback(ompt_callback_target_submit);
+
+  return 1; // success
+}
+
+void ompt_finalize(ompt_data_t *tool_data) {}
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+ompt_start_tool_result_t *ompt_start_tool(unsigned int omp_version,
+                                          const char *runtime_version) {
+  static ompt_start_tool_result_t ompt_start_tool_result = {&ompt_initialize,
+                                                            &ompt_finalize, 0};
+  return &ompt_start_tool_result;
+}
+#ifdef __cplusplus
+}
+#endif
diff --git a/openmp/libomptarget/test/ompt/register_emi.h b/openmp/libomptarget/test/ompt/register_emi.h
new file mode 100644
index 0000000000000..2c70c16424bcc
--- /dev/null
+++ b/openmp/libomptarget/test/ompt/register_emi.h
@@ -0,0 +1,47 @@
+#include <omp-tools.h>
+
+// From openmp/runtime/test/ompt/callback.h
+#define register_ompt_callback_t(name, type)                                   \
+  do {                                                                         \
+    type f_##name = &on_##name;                                                \
+    if (ompt_set_callback(name, (ompt_callback_t)f_##name) == ompt_set_never)  \
+      printf("0: Could not register callback '" #name "'\n");                  \
+  } while (0)
+
+#define register_ompt_callback(name) register_ompt_callback_t(name, name##_t)
+
+// OMPT entry point handles
+static ompt_set_callback_t ompt_set_callback = 0;
+
+// Init functions
+int ompt_initialize(ompt_function_lookup_t lookup, int initial_device_num,
+                    ompt_data_t *tool_data) {
+  ompt_set_callback = (ompt_set_callback_t)lookup("ompt_set_callback");
+
+  if (!ompt_set_callback)
+    return 0; // failed
+
+  register_ompt_callback(ompt_callback_device_initialize);
+  register_ompt_callback(ompt_callback_device_finalize);
+  register_ompt_callback(ompt_callback_device_load);
+  register_ompt_callback(ompt_callback_target_data_op_emi);
+  register_ompt_callback(ompt_callback_target_emi);
+  register_ompt_callback(ompt_callback_target_submit_emi);
+
+  return 1; // success
+}
+
+void ompt_finalize(ompt_data_t *tool_data) {}
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+ompt_start_tool_result_t *ompt_start_tool(unsigned int omp_version,
+                                          const char *runtime_version) {
+  static ompt_start_tool_result_t ompt_start_tool_result = {&ompt_initialize,
+                                                            &ompt_finalize, 0};
+  return &ompt_start_tool_result;
+}
+#ifdef __cplusplus
+}
+#endif
diff --git a/openmp/libomptarget/test/ompt/register_emi_map.h b/openmp/libomptarget/test/ompt/register_emi_map.h
new file mode 100644
index 0000000000000..ccf65914cff18
--- /dev/null
+++ b/openmp/libomptarget/test/ompt/register_emi_map.h
@@ -0,0 +1,48 @@
+#include <omp-tools.h>
+
+// From openmp/runtime/test/ompt/callback.h
+#define register_ompt_callback_t(name, type)                                   \
+  do {                                                                         \
+    type f_##name = &on_##name;                                                \
+    if (ompt_set_callback(name, (ompt_callback_t)f_##name) == ompt_set_never)  \
+      printf("0: Could not register callback '" #name "'\n");                  \
+  } while (0)
+
+#define register_ompt_callback(name) register_ompt_callback_t(name, name##_t)
+
+// OMPT entry point handles
+static ompt_set_callback_t ompt_set_callback = 0;
+
+// Init functions
+int ompt_initialize(ompt_function_lookup_t lookup, int initial_device_num,
+                    ompt_data_t *tool_data) {
+  ompt_set_callback = (ompt_set_callback_t)lookup("ompt_set_callback");
+
+  if (!ompt_set_callback)
+    return 0; // failed
+
+  register_ompt_callback(ompt_callback_device_initialize);
+  register_ompt_callback(ompt_callback_device_finalize);
+  register_ompt_callback(ompt_callback_device_load);
+  register_ompt_callback(ompt_callback_target_data_op_emi);
+  register_ompt_callback(ompt_callback_target_emi);
+  register_ompt_callback(ompt_callback_target_submit_emi);
+  register_ompt_callback(ompt_callback_target_map_emi);
+
+  return 1; // success
+}
+
+void ompt_finalize(ompt_data_t *tool_data) {}
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+ompt_start_tool_result_t *ompt_start_tool(unsigned int omp_version,
+                                          const char *runtime_version) {
+  static ompt_start_tool_result_t ompt_start_tool_result = {&ompt_initialize,
+                                                            &ompt_finalize, 0};
+  return &ompt_start_tool_result;
+}
+#ifdef __cplusplus
+}
+#endif
diff --git a/openmp/libomptarget/test/ompt/register_no_device_init.h b/openmp/libomptarget/test/ompt/register_no_device_init.h
new file mode 100644
index 0000000000000..874e3d399fad8
--- /dev/null
+++ b/openmp/libomptarget/test/ompt/register_no_device_init.h
@@ -0,0 +1,47 @@
+#include <omp-tools.h>
+
+// From openmp/runtime/test/ompt/callback.h
+#define register_ompt_callback_t(name, type)                                   \
+  do {                                                                         \
+    type f_##name = &on_##name;                                                \
+    if (ompt_set_callback(name, (ompt_callback_t)f_##name) == ompt_set_never)  \
+      printf("0: Could not register callback '" #name "'\n");                  \
+  } while (0)
+
+#define register_ompt_callback(name) register_ompt_callback_t(name, name##_t)
+
+// OMPT entry point handles
+static ompt_set_callback_t ompt_set_callback = 0;
+
+// Init functions
+int ompt_initialize(ompt_function_lookup_t lookup, int initial_device_num,
+                    ompt_data_t *tool_data) {
+  ompt_set_callback = (ompt_set_callback_t)lookup("ompt_set_callback");
+
+  if (!ompt_set_callback)
+    return 0; // failed
+
+  // If no device init callback is registered, the other callbacks won't be
+  // activated.
+  register_ompt_callback(ompt_callback_device_load);
+  register_ompt_callback(ompt_callback_target_data_op);
+  register_ompt_callback(ompt_callback_target);
+  register_ompt_callback(ompt_callback_target_submit);
+
+  return 1; // success
+}
+
+void ompt_finalize(ompt_data_t *tool_data) {}
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+ompt_start_tool_result_t *ompt_start_tool(unsigned int omp_version,
+                                          const char *runtime_version) {
+  static ompt_start_tool_result_t ompt_start_tool_result = {&ompt_initialize,
+                                                            &ompt_finalize, 0};
+  return &ompt_start_tool_result;
+}
+#ifdef __cplusplus
+}
+#endif
diff --git a/openmp/libomptarget/test/ompt/register_non_emi.h b/openmp/libomptarget/test/ompt/register_non_emi.h
new file mode 100644
index 0000000000000..66c8aaaafd193
--- /dev/null
+++ b/openmp/libomptarget/test/ompt/register_non_emi.h
@@ -0,0 +1,47 @@
+#include <omp-tools.h>
+
+// From openmp/runtime/test/ompt/callback.h
+#define register_ompt_callback_t(name, type)                                   \
+  do {                                                                         \
+    type f_##name = &on_##name;                                                \
+    if (ompt_set_callback(name, (ompt_callback_t)f_##name) == ompt_set_never)  \
+      printf("0: Could not register callback '" #name "'\n");                  \
+  } while (0)
+
+#define register_ompt_callback(name) register_ompt_callback_t(name, name##_t)
+
+// OMPT entry point handles
+static ompt_set_callback_t ompt_set_callback = 0;
+
+// Init functions
+int ompt_initialize(ompt_function_lookup_t lookup, int initial_device_num,
+                    ompt_data_t *tool_data) {
+  ompt_set_callback = (ompt_set_callback_t)lookup("ompt_set_callback");
+
+  if (!ompt_set_callback)
+    return 0; // failed
+
+  register_ompt_callback(ompt_callback_device_initialize);
+  register_ompt_callback(ompt_callback_device_finalize);
+  register_ompt_callback(ompt_callback_device_load);
+  register_ompt_callback(ompt_callback_target_data_op);
+  register_ompt_callback(ompt_callback_target);
+  register_ompt_callback(ompt_callback_target_submit);
+
+  return 1; // success
+}
+
+void ompt_finalize(ompt_data_t *tool_data) {}
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+ompt_start_tool_result_t *ompt_start_tool(unsigned int omp_version,
+                                          const char *runtime_version) {
+  static ompt_start_tool_result_t ompt_start_tool_result = {&ompt_initialize,
+                                                            &ompt_finalize, 0};
+  return &ompt_start_tool_result;
+}
+#ifdef __cplusplus
+}
+#endif
diff --git a/openmp/libomptarget/test/ompt/register_non_emi_map.h b/openmp/libomptarget/test/ompt/register_non_emi_map.h
new file mode 100644
index 0000000000000..b5c663410d55a
--- /dev/null
+++ b/openmp/libomptarget/test/ompt/register_non_emi_map.h
@@ -0,0 +1,48 @@
+#include <omp-tools.h>
+
+// From openmp/runtime/test/ompt/callback.h
+#define register_ompt_callback_t(name, type)                                   \
+  do {                                                                         \
+    type f_##name = &on_##name;                                                \
+    if (ompt_set_callback(name, (ompt_callback_t)f_##name) == ompt_set_never)  \
+      printf("0: Could not register callback '" #name "'\n");                  \
+  } while (0)
+
+#define register_ompt_callback(name) register_ompt_callback_t(name, name##_t)
+
+// OMPT entry point handles
+static ompt_set_callback_t ompt_set_callback = 0;
+
+// Init functions
+int ompt_initialize(ompt_function_lookup_t lookup, int initial_device_num,
+                    ompt_data_t *tool_data) {
+  ompt_set_callback = (ompt_set_callback_t)lookup("ompt_set_callback");
+
+  if (!ompt_set_callback)
+    return 0; // failed
+
+  register_ompt_callback(ompt_callback_device_initialize);
+  register_ompt_callback(ompt_callback_device_finalize);
+  register_ompt_callback(ompt_callback_device_load);
+  register_ompt_callback(ompt_callback_target_data_op);
+  register_ompt_callback(ompt_callback_target);
+  register_ompt_callback(ompt_callback_target_submit);
+  register_ompt_callback(ompt_callback_target_map);
+
+  return 1; // success
+}
+
+void ompt_finalize(ompt_data_t *tool_data) {}
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+ompt_start_tool_result_t *ompt_start_tool(unsigned int omp_version,
+                                          const char *runtime_version) {
+  static ompt_start_tool_result_t ompt_start_tool_result = {&ompt_initialize,
+                                                            &ompt_finalize, 0};
+  return &ompt_start_tool_result;
+}
+#ifdef __cplusplus
+}
+#endif
diff --git a/openmp/libomptarget/test/ompt/register_wrong_return.h b/openmp/libomptarget/test/ompt/register_wrong_return.h
new file mode 100644
index 0000000000000..7de3feef7dc92
--- /dev/null
+++ b/openmp/libomptarget/test/ompt/register_wrong_return.h
@@ -0,0 +1,47 @@
+#include <omp-tools.h>
+
+// From openmp/runtime/test/ompt/callback.h
+#define register_ompt_callback_t(name, type)                                   \
+  do {                                                                         \
+    type f_##name = &on_##name;                                                \
+    if (ompt_set_callback(name, (ompt_callback_t)f_##name) == ompt_set_never)  \
+      printf("0: Could not register callback '" #name "'\n");                  \
+  } while (0)
+
+#define register_ompt_callback(name) register_ompt_callback_t(name, name##_t)
+
+// OMPT entry point handles
+static ompt_set_callback_t ompt_set_callback = 0;
+
+// Init functions
+int ompt_initialize(ompt_function_lookup_t lookup, int initial_device_num,
+                    ompt_data_t *tool_data) {
+  ompt_set_callback = (ompt_set_callback_t)lookup("ompt_set_callback");
+
+  if (!ompt_set_callback)
+    return 1; // failed but wrongly returning 1
+
+  register_ompt_callback(ompt_callback_device_initialize);
+  register_ompt_callback(ompt_callback_device_finalize);
+  register_ompt_callback(ompt_callback_device_load);
+  register_ompt_callback(ompt_callback_target_data_op);
+  register_ompt_callback(ompt_callback_target);
+  register_ompt_callback(ompt_callback_target_submit);
+
+  return 0; // success but should return 1 according to the spec
+}
+
+void ompt_finalize(ompt_data_t *tool_data) {}
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+ompt_start_tool_result_t *ompt_start_tool(unsigned int omp_version,
+                                          const char *runtime_version) {
+  static ompt_start_tool_result_t ompt_start_tool_result = {&ompt_initialize,
+                                                            &ompt_finalize, 0};
+  return &ompt_start_tool_result;
+}
+#ifdef __cplusplus
+}
+#endif
diff --git a/openmp/libomptarget/test/ompt/veccopy.c b/openmp/libomptarget/test/ompt/veccopy.c
new file mode 100644
index 0000000000000..cb8db1b1e15ee
--- /dev/null
+++ b/openmp/libomptarget/test/ompt/veccopy.c
@@ -0,0 +1,66 @@
+// RUN: %libomptarget-compile-run-and-check-generic
+// REQUIRES: ompt
+// UNSUPPORTED: nvptx64-nvidia-cuda
+// UNSUPPORTED: nvptx64-nvidia-cuda-oldDriver
+// UNSUPPORTED: nvptx64-nvidia-cuda-LTO
+// UNSUPPORTED: x86_64-pc-linux-gnu
+// UNSUPPORTED: x86_64-pc-linux-gnu-oldDriver
+// UNSUPPORTED: x86_64-pc-linux-gnu-LTO
+
+/*
+ * Example OpenMP program that registers non-EMI callbacks
+ */
+
+#include <omp.h>
+#include <stdio.h>
+
+#include "callbacks.h"
+#include "register_non_emi.h"
+
+int main() {
+  int N = 100000;
+
+  int a[N];
+  int b[N];
+
+  int i;
+
+  for (i = 0; i < N; i++)
+    a[i] = 0;
+
+  for (i = 0; i < N; i++)
+    b[i] = i;
+
+#pragma omp target parallel for
+  {
+    for (int j = 0; j < N; j++)
+      a[j] = b[j];
+  }
+
+#pragma omp target teams distribute parallel for
+  {
+    for (int j = 0; j < N; j++)
+      a[j] = b[j];
+  }
+
+  int rc = 0;
+  for (i = 0; i < N; i++)
+    if (a[i] != b[i]) {
+      rc++;
+      printf("Wrong value: a[%d]=%d\n", i, a[i]);
+    }
+
+  if (!rc)
+    printf("Success\n");
+
+  return rc;
+}
+
+/// CHECK: Could not register callback 'ompt_callback_device_initialize'
+/// CHECK: Could not register callback 'ompt_callback_device_finalize'
+/// CHECK: Could not register callback 'ompt_callback_device_load'
+/// CHECK: Could not register callback 'ompt_callback_target_data_op'
+/// CHECK: Could not register callback 'ompt_callback_target'
+/// CHECK: Could not register callback 'ompt_callback_target_submit'
+
+/// CHECK: Success
diff --git a/openmp/libomptarget/test/ompt/veccopy_disallow_both.c b/openmp/libomptarget/test/ompt/veccopy_disallow_both.c
new file mode 100644
index 0000000000000..85f95efb0364a
--- /dev/null
+++ b/openmp/libomptarget/test/ompt/veccopy_disallow_both.c
@@ -0,0 +1,69 @@
+// RUN: %libomptarget-compile-run-and-check-generic
+// REQUIRES: ompt
+// UNSUPPORTED: nvptx64-nvidia-cuda
+// UNSUPPORTED: nvptx64-nvidia-cuda-oldDriver
+// UNSUPPORTED: nvptx64-nvidia-cuda-LTO
+// UNSUPPORTED: x86_64-pc-linux-gnu
+// UNSUPPORTED: x86_64-pc-linux-gnu-oldDriver
+// UNSUPPORTED: x86_64-pc-linux-gnu-LTO
+
+/*
+ * Example OpenMP program that shows that both EMI and non-EMI
+ * callbacks cannot be registered for the same type. In the
+ * current implementation, the EMI callback overrides the non-EMI
+ * callback.
+ */
+
+#include <omp.h>
+#include <stdio.h>
+
+#include "callbacks.h"
+#include "register_both.h"
+
+int main() {
+  int N = 100000;
+
+  int a[N];
+  int b[N];
+
+  int i;
+
+  for (i = 0; i < N; i++)
+    a[i] = 0;
+
+  for (i = 0; i < N; i++)
+    b[i] = i;
+
+#pragma omp target parallel for
+  {
+    for (int j = 0; j < N; j++)
+      a[j] = b[j];
+  }
+
+#pragma omp target teams distribute parallel for
+  {
+    for (int j = 0; j < N; j++)
+      a[j] = b[j];
+  }
+
+  int rc = 0;
+  for (i = 0; i < N; i++)
+    if (a[i] != b[i]) {
+      rc++;
+      printf("Wrong value: a[%d]=%d\n", i, a[i]);
+    }
+
+  if (!rc)
+    printf("Success\n");
+
+  return rc;
+}
+
+/// CHECK: Could not register callback 'ompt_callback_device_initialize'
+/// CHECK: Could not register callback 'ompt_callback_device_finalize'
+/// CHECK: Could not register callback 'ompt_callback_device_load'
+/// CHECK: Could not register callback 'ompt_callback_target_data_op'
+/// CHECK: Could not register callback 'ompt_callback_target'
+/// CHECK: Could not register callback 'ompt_callback_target_submit'
+
+/// CHECK: Success
diff --git a/openmp/libomptarget/test/ompt/veccopy_emi.c b/openmp/libomptarget/test/ompt/veccopy_emi.c
new file mode 100644
index 0000000000000..5e041a2fbf639
--- /dev/null
+++ b/openmp/libomptarget/test/ompt/veccopy_emi.c
@@ -0,0 +1,67 @@
+// RUN: %libomptarget-compile-run-and-check-generic
+// REQUIRES: ompt
+// UNSUPPORTED: nvptx64-nvidia-cuda
+// UNSUPPORTED: nvptx64-nvidia-cuda-oldDriver
+// UNSUPPORTED: nvptx64-nvidia-cuda-LTO
+// UNSUPPORTED: x86_64-pc-linux-gnu
+// UNSUPPORTED: x86_64-pc-linux-gnu-oldDriver
+// UNSUPPORTED: x86_64-pc-linux-gnu-LTO
+
+/*
+ * Example OpenMP program that registers EMI callbacks
+ */
+
+#include <assert.h>
+#include <omp.h>
+#include <stdio.h>
+
+#include "callbacks.h"
+#include "register_emi.h"
+
+int main() {
+  int N = 100000;
+
+  int a[N];
+  int b[N];
+
+  int i;
+
+  for (i = 0; i < N; i++)
+    a[i] = 0;
+
+  for (i = 0; i < N; i++)
+    b[i] = i;
+
+#pragma omp target parallel for
+  {
+    for (int j = 0; j < N; j++)
+      a[j] = b[j];
+  }
+
+#pragma omp target teams distribute parallel for
+  {
+    for (int j = 0; j < N; j++)
+      a[j] = b[j];
+  }
+
+  int rc = 0;
+  for (i = 0; i < N; i++)
+    if (a[i] != b[i]) {
+      rc++;
+      printf("Wrong value: a[%d]=%d\n", i, a[i]);
+    }
+
+  if (!rc)
+    printf("Success\n");
+
+  return rc;
+}
+
+/// CHECK: Could not register callback 'ompt_callback_device_initialize'
+/// CHECK: Could not register callback 'ompt_callback_device_finalize'
+/// CHECK: Could not register callback 'ompt_callback_device_load'
+/// CHECK: Could not register callback 'ompt_callback_target_data_op_emi'
+/// CHECK: Could not register callback 'ompt_callback_target_emi'
+/// CHECK: Could not register callback 'ompt_callback_target_submit_emi'
+
+/// CHECK: Success
diff --git a/openmp/libomptarget/test/ompt/veccopy_emi_map.c b/openmp/libomptarget/test/ompt/veccopy_emi_map.c
new file mode 100644
index 0000000000000..e237a4d2eaa9a
--- /dev/null
+++ b/openmp/libomptarget/test/ompt/veccopy_emi_map.c
@@ -0,0 +1,68 @@
+// RUN: %libomptarget-compile-run-and-check-generic
+// REQUIRES: ompt
+// UNSUPPORTED: nvptx64-nvidia-cuda
+// UNSUPPORTED: nvptx64-nvidia-cuda-oldDriver
+// UNSUPPORTED: nvptx64-nvidia-cuda-LTO
+// UNSUPPORTED: x86_64-pc-linux-gnu
+// UNSUPPORTED: x86_64-pc-linux-gnu-oldDriver
+// UNSUPPORTED: x86_64-pc-linux-gnu-LTO
+
+/*
+ * Example OpenMP program that shows that map-EMI callbacks are not supported.
+ */
+
+#include <assert.h>
+#include <omp.h>
+#include <stdio.h>
+
+#include "callbacks.h"
+#include "register_emi_map.h"
+
+int main() {
+  int N = 100000;
+
+  int a[N];
+  int b[N];
+
+  int i;
+
+  for (i = 0; i < N; i++)
+    a[i] = 0;
+
+  for (i = 0; i < N; i++)
+    b[i] = i;
+
+#pragma omp target parallel for
+  {
+    for (int j = 0; j < N; j++)
+      a[j] = b[j];
+  }
+
+#pragma omp target teams distribute parallel for
+  {
+    for (int j = 0; j < N; j++)
+      a[j] = b[j];
+  }
+
+  int rc = 0;
+  for (i = 0; i < N; i++)
+    if (a[i] != b[i]) {
+      rc++;
+      printf("Wrong value: a[%d]=%d\n", i, a[i]);
+    }
+
+  if (!rc)
+    printf("Success\n");
+
+  return rc;
+}
+
+/// CHECK: Could not register callback 'ompt_callback_device_initialize'
+/// CHECK: Could not register callback 'ompt_callback_device_finalize'
+/// CHECK: Could not register callback 'ompt_callback_device_load'
+/// CHECK: Could not register callback 'ompt_callback_target_data_op_emi'
+/// CHECK: Could not register callback 'ompt_callback_target_emi'
+/// CHECK: Could not register callback 'ompt_callback_target_submit_emi'
+/// CHECK: Could not register callback 'ompt_callback_target_map_emi'
+
+/// CHECK: Success
diff --git a/openmp/libomptarget/test/ompt/veccopy_map.c b/openmp/libomptarget/test/ompt/veccopy_map.c
new file mode 100644
index 0000000000000..977562be400bc
--- /dev/null
+++ b/openmp/libomptarget/test/ompt/veccopy_map.c
@@ -0,0 +1,66 @@
+// RUN: %libomptarget-compile-run-and-check-generic
+// REQUIRES: ompt
+// UNSUPPORTED: nvptx64-nvidia-cuda
+// UNSUPPORTED: nvptx64-nvidia-cuda-oldDriver
+// UNSUPPORTED: nvptx64-nvidia-cuda-LTO
+// UNSUPPORTED: x86_64-pc-linux-gnu
+// UNSUPPORTED: x86_64-pc-linux-gnu-oldDriver
+// UNSUPPORTED: x86_64-pc-linux-gnu-LTO
+
+/*
+ * Example OpenMP program that shows that map callbacks are not supported.
+ */
+
+#include <omp.h>
+#include <stdio.h>
+
+#include "callbacks.h"
+#include "register_non_emi_map.h"
+
+int main() {
+  int N = 100000;
+
+  int a[N];
+  int b[N];
+
+  int i;
+
+  for (i = 0; i < N; i++)
+    a[i] = 0;
+
+  for (i = 0; i < N; i++)
+    b[i] = i;
+
+#pragma omp target parallel for
+  {
+    for (int j = 0; j < N; j++)
+      a[j] = b[j];
+  }
+
+#pragma omp target teams distribute parallel for
+  {
+    for (int j = 0; j < N; j++)
+      a[j] = b[j];
+  }
+
+  int rc = 0;
+  for (i = 0; i < N; i++)
+    if (a[i] != b[i]) {
+      rc++;
+      printf("Wrong value: a[%d]=%d\n", i, a[i]);
+    }
+
+  if (!rc)
+    printf("Success\n");
+
+  return rc;
+}
+
+/// CHECK: Could not register callback 'ompt_callback_device_initialize'
+/// CHECK: Could not register callback 'ompt_callback_device_finalize'
+/// CHECK: Could not register callback 'ompt_callback_device_load'
+/// CHECK: Could not register callback 'ompt_callback_target_data_op'
+/// CHECK: Could not register callback 'ompt_callback_target'
+/// CHECK: Could not register callback 'ompt_callback_target_submit'
+
+/// CHECK: Success
diff --git a/openmp/libomptarget/test/ompt/veccopy_no_device_init.c b/openmp/libomptarget/test/ompt/veccopy_no_device_init.c
new file mode 100644
index 0000000000000..165ba119a9c84
--- /dev/null
+++ b/openmp/libomptarget/test/ompt/veccopy_no_device_init.c
@@ -0,0 +1,65 @@
+// RUN: %libomptarget-compile-run-and-check-generic
+// REQUIRES: ompt
+// UNSUPPORTED: nvptx64-nvidia-cuda
+// UNSUPPORTED: nvptx64-nvidia-cuda-oldDriver
+// UNSUPPORTED: nvptx64-nvidia-cuda-LTO
+// UNSUPPORTED: x86_64-pc-linux-gnu
+// UNSUPPORTED: x86_64-pc-linux-gnu-oldDriver
+// UNSUPPORTED: x86_64-pc-linux-gnu-LTO
+
+/*
+ * Example OpenMP program that shows that if no device init callback
+ * is registered, the other callbacks won't be activated.
+ */
+
+#include <omp.h>
+#include <stdio.h>
+
+#include "callbacks.h"
+#include "register_no_device_init.h"
+
+int main() {
+  int N = 100000;
+
+  int a[N];
+  int b[N];
+
+  int i;
+
+  for (i = 0; i < N; i++)
+    a[i] = 0;
+
+  for (i = 0; i < N; i++)
+    b[i] = i;
+
+#pragma omp target parallel for
+  {
+    for (int j = 0; j < N; j++)
+      a[j] = b[j];
+  }
+
+#pragma omp target teams distribute parallel for
+  {
+    for (int j = 0; j < N; j++)
+      a[j] = b[j];
+  }
+
+  int rc = 0;
+  for (i = 0; i < N; i++)
+    if (a[i] != b[i]) {
+      rc++;
+      printf("Wrong value: a[%d]=%d\n", i, a[i]);
+    }
+
+  if (!rc)
+    printf("Success\n");
+
+  return rc;
+}
+
+/// CHECK: Could not register callback 'ompt_callback_device_load'
+/// CHECK: Could not register callback 'ompt_callback_target_data_op'
+/// CHECK: Could not register callback 'ompt_callback_target'
+/// CHECK: Could not register callback 'ompt_callback_target_submit'
+
+/// CHECK: Success
diff --git a/openmp/libomptarget/test/ompt/veccopy_wrong_return.c b/openmp/libomptarget/test/ompt/veccopy_wrong_return.c
new file mode 100644
index 0000000000000..c84053957d200
--- /dev/null
+++ b/openmp/libomptarget/test/ompt/veccopy_wrong_return.c
@@ -0,0 +1,67 @@
+// RUN: %libomptarget-compile-run-and-check-generic
+// REQUIRES: ompt
+// UNSUPPORTED: nvptx64-nvidia-cuda
+// UNSUPPORTED: nvptx64-nvidia-cuda-oldDriver
+// UNSUPPORTED: nvptx64-nvidia-cuda-LTO
+// UNSUPPORTED: x86_64-pc-linux-gnu
+// UNSUPPORTED: x86_64-pc-linux-gnu-oldDriver
+// UNSUPPORTED: x86_64-pc-linux-gnu-LTO
+
+/*
+ * Example OpenMP program that shows that if the initialize function
+ * returns the wrong status code, the callbacks won't be activated.
+ */
+
+#include <omp.h>
+#include <stdio.h>
+
+#include "callbacks.h"
+#include "register_wrong_return.h"
+
+int main() {
+  int N = 100000;
+
+  int a[N];
+  int b[N];
+
+  int i;
+
+  for (i = 0; i < N; i++)
+    a[i] = 0;
+
+  for (i = 0; i < N; i++)
+    b[i] = i;
+
+#pragma omp target parallel for
+  {
+    for (int j = 0; j < N; j++)
+      a[j] = b[j];
+  }
+
+#pragma omp target teams distribute parallel for
+  {
+    for (int j = 0; j < N; j++)
+      a[j] = b[j];
+  }
+
+  int rc = 0;
+  for (i = 0; i < N; i++)
+    if (a[i] != b[i]) {
+      rc++;
+      printf("Wrong value: a[%d]=%d\n", i, a[i]);
+    }
+
+  if (!rc)
+    printf("Success\n");
+
+  return rc;
+}
+
+/// CHECK: Could not register callback 'ompt_callback_device_initialize'
+/// CHECK: Could not register callback 'ompt_callback_device_finalize'
+/// CHECK: Could not register callback 'ompt_callback_device_load'
+/// CHECK: Could not register callback 'ompt_callback_target_data_op'
+/// CHECK: Could not register callback 'ompt_callback_target'
+/// CHECK: Could not register callback 'ompt_callback_target_submit'
+
+/// CHECK: Success
diff --git a/openmp/runtime/src/dllexports b/openmp/runtime/src/dllexports
index 991f173b66ab2..1926683645045 100644
--- a/openmp/runtime/src/dllexports
+++ b/openmp/runtime/src/dllexports
@@ -186,9 +186,6 @@
         __kmp_omp_debug_struct_info             DATA
     %endif
 
-    # Symbols for MS mutual detection:
-    _You_must_link_with_exactly_one_OpenMP_library    DATA
-    _You_must_link_with_Intel_OpenMP_library          DATA
     __kmp_wait_64
     __kmp_release_64
 
diff --git a/openmp/runtime/src/kmp.h b/openmp/runtime/src/kmp.h
index 08ce0a4af889b..43e723fa7387b 100644
--- a/openmp/runtime/src/kmp.h
+++ b/openmp/runtime/src/kmp.h
@@ -4184,13 +4184,6 @@ KMP_EXPORT void *__kmpc_threadprivate_cached(ident_t *loc, kmp_int32 global_tid,
                                              void *data, size_t size,
                                              void ***cache);
 
-// Symbols for MS mutual detection.
-extern int _You_must_link_with_exactly_one_OpenMP_library;
-extern int _You_must_link_with_Intel_OpenMP_library;
-#if KMP_OS_WINDOWS && (KMP_VERSION_MAJOR > 4)
-extern int _You_must_link_with_Microsoft_OpenMP_library;
-#endif
-
 // The routines below are not exported.
 // Consider making them 'static' in corresponding source files.
 void kmp_threadprivate_insert_private_data(int gtid, void *pc_addr,
diff --git a/openmp/runtime/src/kmp_global.cpp b/openmp/runtime/src/kmp_global.cpp
index b1eca773db1eb..0163846b44546 100644
--- a/openmp/runtime/src/kmp_global.cpp
+++ b/openmp/runtime/src/kmp_global.cpp
@@ -547,13 +547,6 @@ int get_suspend_count_(void) {
 void set_suspend_count_(int *value) { __kmp_suspend_count = *value; }
 #endif
 
-// Symbols for MS mutual detection.
-int _You_must_link_with_exactly_one_OpenMP_library = 1;
-int _You_must_link_with_Intel_OpenMP_library = 1;
-#if KMP_OS_WINDOWS && (KMP_VERSION_MAJOR > 4)
-int _You_must_link_with_Microsoft_OpenMP_library = 1;
-#endif
-
 kmp_target_offload_kind_t __kmp_target_offload = tgt_default;
 
 // OMP Pause Resources
diff --git a/openmp/runtime/src/kmp_import.cpp b/openmp/runtime/src/kmp_import.cpp
deleted file mode 100644
index 39d841d9d4688..0000000000000
--- a/openmp/runtime/src/kmp_import.cpp
+++ /dev/null
@@ -1,33 +0,0 @@
-/*
- * kmp_import.cpp
- */
-
-//===----------------------------------------------------------------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-/* Object generated from this source file is linked to Windows* OS DLL import
-   library (libompmd.lib) only! It is not a part of regular static or dynamic
-   OpenMP RTL. Any code that just needs to go in the libompmd.lib (but not in
-   libompmt.lib and libompmd.dll) should be placed in this file. */
-
-#ifdef __cplusplus
-extern "C" {
-#endif
-
-/*These symbols are required for mutual exclusion with Microsoft OpenMP RTL
-  (and compatibility with MS Compiler). */
-
-int _You_must_link_with_exactly_one_OpenMP_library = 1;
-int _You_must_link_with_Intel_OpenMP_library = 1;
-int _You_must_link_with_Microsoft_OpenMP_library = 1;
-
-#ifdef __cplusplus
-}
-#endif
-
-// end of file //
diff --git a/openmp/runtime/test/worksharing/for/omp_for_non_rectangular.c b/openmp/runtime/test/worksharing/for/omp_for_non_rectangular.c
new file mode 100644
index 0000000000000..0b9451ac11696
--- /dev/null
+++ b/openmp/runtime/test/worksharing/for/omp_for_non_rectangular.c
@@ -0,0 +1,37 @@
+// RUN: %libomp-compile-and-run
+
+#define N 10
+int arr[N][N][N];
+
+int collapsed(int mp) {
+  int Cnt = 0;
+#pragma omp for collapse(3)
+  for (int j = 0; j < mp; ++j) {
+    for (int i = j; i < mp; ++i) {
+      for (int i0 = 0; i0 < N; ++i0) {
+        arr[j][i][i0] = 1;
+        ++Cnt;
+      }
+    }
+  }
+  return Cnt;
+}
+
+int test(int mp) {
+  int Failed = 0;
+  for (int j = 0; j < mp; ++j) {
+    for (int i = 0; i < mp; ++i) {
+      for (int i0 = 0; i0 < N; ++i0) {
+        if ((i < j && arr[j][i][i0] == 0) || (i >= j && arr[j][i][i0] == 1))
+          continue;
+        ++Failed;
+      }
+    }
+  }
+  return Failed;
+}
+
+int main() {
+  int Res = collapsed(N);
+  return Res == N * (N + 1) && test(N) == 0;
+}
diff --git a/polly/CMakeLists.txt b/polly/CMakeLists.txt
index b81ce165b62fd..536f11f968292 100644
--- a/polly/CMakeLists.txt
+++ b/polly/CMakeLists.txt
@@ -85,31 +85,6 @@ endif ()
 
 SET(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
 
-option(POLLY_ENABLE_GPGPU_CODEGEN "Enable GPGPU code generation feature" OFF)
-set(GPU_CODEGEN FALSE)
-if (POLLY_ENABLE_GPGPU_CODEGEN)
-  # Do not require CUDA/OpenCL, as GPU code generation test cases can be run
-  # without a CUDA/OpenCL library.
-  if ("NVPTX" IN_LIST LLVM_TARGETS_TO_BUILD)
-    FIND_PACKAGE(CUDA)
-    FIND_PACKAGE(OpenCL)
-    set(GPU_CODEGEN TRUE)
-  else()
-    message(WARNING "The LLVM NVPTX target is required for GPU code generation")
-  endif()
-endif(POLLY_ENABLE_GPGPU_CODEGEN)
-
-
-# Support GPGPU code generation if the library is available.
-if (CUDA_FOUND)
-  add_definitions(-DHAS_LIBCUDART)
-  INCLUDE_DIRECTORIES( ${CUDA_INCLUDE_DIRS} )
-endif(CUDA_FOUND)
-if (OpenCL_FOUND)
-  add_definitions(-DHAS_LIBOPENCL)
-  INCLUDE_DIRECTORIES( ${OpenCL_INCLUDE_DIR} )
-endif(OpenCL_FOUND)
-
 option(POLLY_BUNDLED_ISL "Use the bundled version of libisl included in Polly" ON)
 if (NOT POLLY_BUNDLED_ISL)
   find_package(ISL MODULE REQUIRED)
@@ -155,7 +130,6 @@ add_subdirectory(test)
 if (POLLY_GTEST_AVAIL)
   add_subdirectory(unittests)
 endif ()
-add_subdirectory(tools)
 add_subdirectory(cmake)
 # TODO: docs.
 
diff --git a/polly/cmake/CMakeLists.txt b/polly/cmake/CMakeLists.txt
index b6181fbe4cc55..e74cf5f441338 100644
--- a/polly/cmake/CMakeLists.txt
+++ b/polly/cmake/CMakeLists.txt
@@ -27,9 +27,6 @@ if (NOT WIN32 AND LLVM_ENABLE_PIC)
   # LLVMPolly is a dummy target on Win or if PIC code is disabled.
   list(APPEND POLLY_CONFIG_EXPORTED_TARGETS LLVMPolly)
 endif()
-if (POLLY_ENABLE_GPGPU_CODEGEN)
-  list(APPEND POLLY_CONFIG_EXPORTED_TARGETS PollyPPCG)
-endif()
 
 # Get the target type for every exported target
 foreach(tgt IN LISTS POLLY_CONFIG_EXPORTED_TARGETS)
diff --git a/polly/cmake/PollyConfig.cmake.in b/polly/cmake/PollyConfig.cmake.in
index 4513ec4231821..18c8090bafeae 100644
--- a/polly/cmake/PollyConfig.cmake.in
+++ b/polly/cmake/PollyConfig.cmake.in
@@ -8,7 +8,6 @@ find_package(LLVM ${LLVM_VERSION} EXACT REQUIRED CONFIG
 
 set(Polly_CMAKE_DIR ${CMAKE_CURRENT_LIST_DIR})
 set(Polly_BUNDLED_ISL @POLLY_BUNDLED_ISL@)
-set(Polly_ENABLE_GPGPU_CODEGEN @POLLY_ENABLE_GPGPU_CODEGEN@)
 
 set(Polly_DEFINITIONS ${LLVM_DEFINITIONS})
 set(Polly_INCLUDE_DIRS @POLLY_CONFIG_INCLUDE_DIRS@ ${LLVM_INCLUDE_DIRS})
@@ -19,17 +18,9 @@ set(Polly_LIBRARIES ${LLVM_LIBRARIES} ${Polly_EXPORTED_TARGETS})
 # Imported Targets:
 @ISL_CONFIG_CODE@
 
-if (Polly_ENABLE_GPGPU_CODEGEN AND NOT TARGET PollyPPCG)
-  add_library(PollyPPCG @POLLY_CONFIG_TARGET_PollyPPCG_TYPE@ IMPORTED)
-  set_property(TARGET PollyPPCG PROPERTY INTERFACE_LINK_LIBRARIES @ISL_TARGET@)
-endif()
-
 if (NOT TARGET Polly)
   add_library(Polly @POLLY_CONFIG_TARGET_Polly_TYPE@ IMPORTED)
   set_property(TARGET Polly PROPERTY INTERFACE_LINK_LIBRARIES @ISL_TARGET@)
-  if (Polly_ENABLE_GPGPU_CODEGEN)
-    set_property(TARGET Polly APPEND PROPERTY INTERFACE_LINK_LIBRARIES PollyPPCG)
-  endif()
 endif()
 
 if (NOT TARGET LLVMPolly)
diff --git a/polly/docs/ReleaseNotes.rst b/polly/docs/ReleaseNotes.rst
index bf6f7b9277d3f..691b6e6f45b77 100644
--- a/polly/docs/ReleaseNotes.rst
+++ b/polly/docs/ReleaseNotes.rst
@@ -10,3 +10,16 @@ In Polly |version| the following important changes have been incorporated.
     These release notes are for the next release of Polly and describe
     the new features that have recently been committed to our development
     branch.
+
+
+- Support for -polly-vectorizer=polly has been removed. Polly's internal
+  vectorizer is not well maintained and is known to not work in some cases
+  such as region ScopStmts. Unlike LLVM's LoopVectorize pass it also does
+  not have a target-dependent cost heuristics, and we recommend using
+  LoopVectorize instead of -polly-vectorizer=polly.
+
+  In the future we hope that Polly can collaborate better with LoopVectorize,
+  like Polly marking a loop is safe to vectorize with a specific simd width,
+  instead of replicating its functionality.
+
+- Polly-ACC has been removed.
diff --git a/polly/include/polly/CodeGen/BlockGenerators.h b/polly/include/polly/CodeGen/BlockGenerators.h
index a30eb8b4bb931..13c27328d8c7e 100644
--- a/polly/include/polly/CodeGen/BlockGenerators.h
+++ b/polly/include/polly/CodeGen/BlockGenerators.h
@@ -622,186 +622,6 @@ class BlockGenerator {
   void invalidateScalarEvolution(Scop &S);
 };
 
-/// Generate a new vector basic block for a polyhedral statement.
-///
-/// The only public function exposed is generate().
-class VectorBlockGenerator final : BlockGenerator {
-public:
-  /// Generate a new vector basic block for a ScoPStmt.
-  ///
-  /// This code generation is similar to the normal, scalar code generation,
-  /// except that each instruction is code generated for several vector lanes
-  /// at a time. If possible instructions are issued as actual vector
-  /// instructions, but e.g. for address calculation instructions we currently
-  /// generate scalar instructions for each vector lane.
-  ///
-  /// @param BlockGen    A block generator object used as parent.
-  /// @param Stmt        The statement to code generate.
-  /// @param VLTS        A mapping from loops virtual canonical induction
-  ///                    variable to their new values
-  ///                    (for values recalculated in the new ScoP, but not
-  ///                     within this basic block), one for each lane.
-  /// @param Schedule    A map from the statement to a schedule where the
-  ///                    innermost dimension is the dimension of the innermost
-  ///                    loop containing the statement.
-  /// @param NewAccesses A map from memory access ids to new ast expressions,
-  ///                    which may contain new access expressions for certain
-  ///                    memory accesses.
-  static void generate(BlockGenerator &BlockGen, ScopStmt &Stmt,
-                       std::vector<LoopToScevMapT> &VLTS,
-                       __isl_keep isl_map *Schedule,
-                       __isl_keep isl_id_to_ast_expr *NewAccesses) {
-    VectorBlockGenerator Generator(BlockGen, VLTS, Schedule);
-    Generator.copyStmt(Stmt, NewAccesses);
-  }
-
-private:
-  // This is a vector of loop->scev maps.  The first map is used for the first
-  // vector lane, ...
-  // Each map, contains information about Instructions in the old ScoP, which
-  // are recalculated in the new SCoP. When copying the basic block, we replace
-  // all references to the old instructions with their recalculated values.
-  //
-  // For example, when the code generator produces this AST:
-  //
-  //   for (int c1 = 0; c1 <= 1023; c1 += 1)
-  //     for (int c2 = 0; c2 <= 1023; c2 += VF)
-  //       for (int lane = 0; lane <= VF; lane += 1)
-  //         Stmt(c2 + lane + 3, c1);
-  //
-  // VLTS[lane] contains a map:
-  //   "outer loop in the old loop nest" -> SCEV("c2 + lane + 3"),
-  //   "inner loop in the old loop nest" -> SCEV("c1").
-  std::vector<LoopToScevMapT> &VLTS;
-
-  // A map from the statement to a schedule where the innermost dimension is the
-  // dimension of the innermost loop containing the statement.
-  isl_map *Schedule;
-
-  VectorBlockGenerator(BlockGenerator &BlockGen,
-                       std::vector<LoopToScevMapT> &VLTS,
-                       __isl_keep isl_map *Schedule);
-
-  int getVectorWidth();
-
-  Value *getVectorValue(ScopStmt &Stmt, Value *Old, ValueMapT &VectorMap,
-                        VectorValueMapT &ScalarMaps, Loop *L);
-
-  /// Load a vector from a set of adjacent scalars
-  ///
-  /// In case a set of scalars is known to be next to each other in memory,
-  /// create a vector load that loads those scalars
-  ///
-  /// %vector_ptr= bitcast double* %p to <4 x double>*
-  /// %vec_full = load <4 x double>* %vector_ptr
-  ///
-  /// @param Stmt           The statement to code generate.
-  /// @param NegativeStride This is used to indicate a -1 stride. In such
-  ///                       a case we load the end of a base address and
-  ///                       shuffle the accesses in reverse order into the
-  ///                       vector. By default we would do only positive
-  ///                       strides.
-  ///
-  /// @param NewAccesses    A map from memory access ids to new ast
-  ///                       expressions, which may contain new access
-  ///                       expressions for certain memory accesses.
-  Value *generateStrideOneLoad(ScopStmt &Stmt, LoadInst *Load,
-                               VectorValueMapT &ScalarMaps,
-                               __isl_keep isl_id_to_ast_expr *NewAccesses,
-                               bool NegativeStride);
-
-  /// Load a vector initialized from a single scalar in memory
-  ///
-  /// In case all elements of a vector are initialized to the same
-  /// scalar value, this value is loaded and shuffled into all elements
-  /// of the vector.
-  ///
-  /// %splat_one = load <1 x double>* %p
-  /// %splat = shufflevector <1 x double> %splat_one, <1 x
-  ///       double> %splat_one, <4 x i32> zeroinitializer
-  ///
-  /// @param NewAccesses A map from memory access ids to new ast expressions,
-  ///                    which may contain new access expressions for certain
-  ///                    memory accesses.
-  Value *generateStrideZeroLoad(ScopStmt &Stmt, LoadInst *Load,
-                                ValueMapT &BBMap,
-                                __isl_keep isl_id_to_ast_expr *NewAccesses);
-
-  /// Load a vector from scalars distributed in memory
-  ///
-  /// In case some scalars a distributed randomly in memory. Create a vector
-  /// by loading each scalar and by inserting one after the other into the
-  /// vector.
-  ///
-  /// %scalar_1= load double* %p_1
-  /// %vec_1 = insertelement <2 x double> undef, double %scalar_1, i32 0
-  /// %scalar 2 = load double* %p_2
-  /// %vec_2 = insertelement <2 x double> %vec_1, double %scalar_1, i32 1
-  ///
-  /// @param NewAccesses A map from memory access ids to new ast expressions,
-  ///                    which may contain new access expressions for certain
-  ///                    memory accesses.
-  Value *generateUnknownStrideLoad(ScopStmt &Stmt, LoadInst *Load,
-                                   VectorValueMapT &ScalarMaps,
-                                   __isl_keep isl_id_to_ast_expr *NewAccesses);
-
-  /// @param NewAccesses A map from memory access ids to new ast expressions,
-  ///                    which may contain new access expressions for certain
-  ///                    memory accesses.
-  void generateLoad(ScopStmt &Stmt, LoadInst *Load, ValueMapT &VectorMap,
-                    VectorValueMapT &ScalarMaps,
-                    __isl_keep isl_id_to_ast_expr *NewAccesses);
-
-  void copyUnaryInst(ScopStmt &Stmt, UnaryInstruction *Inst,
-                     ValueMapT &VectorMap, VectorValueMapT &ScalarMaps);
-
-  void copyBinaryInst(ScopStmt &Stmt, BinaryOperator *Inst,
-                      ValueMapT &VectorMap, VectorValueMapT &ScalarMaps);
-
-  /// @param NewAccesses A map from memory access ids to new ast expressions,
-  ///                    which may contain new access expressions for certain
-  ///                    memory accesses.
-  void copyStore(ScopStmt &Stmt, StoreInst *Store, ValueMapT &VectorMap,
-                 VectorValueMapT &ScalarMaps,
-                 __isl_keep isl_id_to_ast_expr *NewAccesses);
-
-  /// @param NewAccesses A map from memory access ids to new ast expressions,
-  ///                    which may contain new access expressions for certain
-  ///                    memory accesses.
-  void copyInstScalarized(ScopStmt &Stmt, Instruction *Inst,
-                          ValueMapT &VectorMap, VectorValueMapT &ScalarMaps,
-                          __isl_keep isl_id_to_ast_expr *NewAccesses);
-
-  bool extractScalarValues(const Instruction *Inst, ValueMapT &VectorMap,
-                           VectorValueMapT &ScalarMaps);
-
-  bool hasVectorOperands(const Instruction *Inst, ValueMapT &VectorMap);
-
-  /// Generate vector loads for scalars.
-  ///
-  /// @param Stmt           The scop statement for which to generate the loads.
-  /// @param VectorBlockMap A map that will be updated to relate the original
-  ///                       values with the newly generated vector loads.
-  void generateScalarVectorLoads(ScopStmt &Stmt, ValueMapT &VectorBlockMap);
-
-  /// Verify absence of scalar stores.
-  ///
-  /// @param Stmt The scop statement to check for scalar stores.
-  void verifyNoScalarStores(ScopStmt &Stmt);
-
-  /// @param NewAccesses A map from memory access ids to new ast expressions,
-  ///                    which may contain new access expressions for certain
-  ///                    memory accesses.
-  void copyInstruction(ScopStmt &Stmt, Instruction *Inst, ValueMapT &VectorMap,
-                       VectorValueMapT &ScalarMaps,
-                       __isl_keep isl_id_to_ast_expr *NewAccesses);
-
-  /// @param NewAccesses A map from memory access ids to new ast expressions,
-  ///                    which may contain new access expressions for certain
-  ///                    memory accesses.
-  void copyStmt(ScopStmt &Stmt, __isl_keep isl_id_to_ast_expr *NewAccesses);
-};
-
 /// Generator for new versions of polyhedral region statements.
 class RegionGenerator final : BlockGenerator {
 public:
diff --git a/polly/include/polly/CodeGen/CodeGeneration.h b/polly/include/polly/CodeGen/CodeGeneration.h
index 77668cbbda754..57aec1d70cc72 100644
--- a/polly/include/polly/CodeGen/CodeGeneration.h
+++ b/polly/include/polly/CodeGen/CodeGeneration.h
@@ -18,7 +18,6 @@ namespace polly {
 enum VectorizerChoice {
   VECTORIZER_NONE,
   VECTORIZER_STRIPMINE,
-  VECTORIZER_POLLY,
 };
 extern VectorizerChoice PollyVectorizerChoice;
 
diff --git a/polly/include/polly/CodeGen/IslNodeBuilder.h b/polly/include/polly/CodeGen/IslNodeBuilder.h
index 2dc7f019e84cf..05f53d79d74a4 100644
--- a/polly/include/polly/CodeGen/IslNodeBuilder.h
+++ b/polly/include/polly/CodeGen/IslNodeBuilder.h
@@ -310,7 +310,6 @@ class IslNodeBuilder {
   /// @returns False, iff a problem occurred and the load was not preloaded.
   bool preloadInvariantEquivClass(InvariantEquivClassTy &IAClass);
 
-  void createForVector(__isl_take isl_ast_node *For, int VectorWidth);
   void createForSequential(isl::ast_node_for For, bool MarkParallel);
 
   /// Create LLVM-IR that executes a for node thread parallel.
@@ -375,10 +374,6 @@ class IslNodeBuilder {
                                  std::vector<Value *> &IVS,
                                  __isl_take isl_id *IteratorID);
   virtual void createIf(__isl_take isl_ast_node *If);
-  void createUserVector(__isl_take isl_ast_node *User,
-                        std::vector<Value *> &IVS,
-                        __isl_take isl_id *IteratorID,
-                        __isl_take isl_union_map *Schedule);
   virtual void createUser(__isl_take isl_ast_node *User);
   virtual void createBlock(__isl_take isl_ast_node *Block);
 
diff --git a/polly/include/polly/CodeGen/PPCGCodeGeneration.h b/polly/include/polly/CodeGen/PPCGCodeGeneration.h
deleted file mode 100644
index 9a6c596e5f494..0000000000000
--- a/polly/include/polly/CodeGen/PPCGCodeGeneration.h
+++ /dev/null
@@ -1,33 +0,0 @@
-//===--- polly/PPCGCodeGeneration.h - Polly Accelerator Code Generation. --===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// Take a scop created by ScopInfo and map it to GPU code using the ppcg
-// GPU mapping strategy.
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef POLLY_PPCGCODEGENERATION_H
-#define POLLY_PPCGCODEGENERATION_H
-
-/// The GPU Architecture to target.
-enum GPUArch { NVPTX64, SPIR32, SPIR64 };
-
-/// The GPU Runtime implementation to use.
-enum GPURuntime { CUDA, OpenCL };
-
-namespace polly {
-extern bool PollyManagedMemory;
-
-/// Use for pass instantiation defaults.
-/// @{
-extern GPURuntime GPURuntimeChoice;
-extern GPUArch GPUArchChoice;
-/// @}
-} // namespace polly
-
-#endif // POLLY_PPCGCODEGENERATION_H
diff --git a/polly/include/polly/CodeGen/RuntimeDebugBuilder.h b/polly/include/polly/CodeGen/RuntimeDebugBuilder.h
index c40b53c1b54d8..46193f9ccaf2c 100644
--- a/polly/include/polly/CodeGen/RuntimeDebugBuilder.h
+++ b/polly/include/polly/CodeGen/RuntimeDebugBuilder.h
@@ -30,24 +30,20 @@ namespace polly {
 struct RuntimeDebugBuilder {
 
   /// Generate a constant string into the builder's llvm::Module which can be
-  /// passed to createGPUPrinter() or createGPUPrinter().
+  /// passed to createCPUPrinter().
   ///
   /// @param Builder The builder used to emit the printer calls.
   /// @param Str     The string to be printed.
 
   /// @return        A global containing @p Str.
   static llvm::Value *getPrintableString(PollyIRBuilder &Builder,
-                                         llvm::StringRef Str) {
-    // TODO: Get rid of magic number 4. It it NVPTX's constant address space and
-    // works on X86 (CPU) only because its backend ignores the address space.
-    return Builder.CreateGlobalStringPtr(Str, "", 4);
-  }
+                                         llvm::StringRef Str);
 
   /// Return whether an llvm::Value of the type @p Ty is printable for
   /// debugging.
   ///
-  /// That is, whether such a value can be passed to createGPUPrinter() or
-  /// createGPUPrinter() to be dumped as runtime.  If false is returned, those
+  /// That is, whether such a value can be passed to createGPUPrinter()
+  /// to be dumped as runtime.  If false is returned, those
   /// functions will fail.
   static bool isPrintable(llvm::Type *Ty);
 
@@ -64,62 +60,41 @@ struct RuntimeDebugBuilder {
   template <typename... Args>
   static void createCPUPrinter(PollyIRBuilder &Builder, Args... args) {
     std::vector<llvm::Value *> Vector;
-    createPrinter(Builder, /* CPU */ false, Vector, args...);
-  }
-
-  /// Print a set of LLVM-IR Values or StringRefs on an NVIDIA GPU.
-  ///
-  ///  This function emits a call to vprintf that will print the given
-  ///  arguments from within a kernel thread. It is useful for debugging
-  ///  CUDA program kernels. All arguments given in this list will be
-  ///  automatically concatenated and the resulting string will be printed
-  ///  atomically. We also support ArrayRef arguments, which can be used to
-  ///  provide for example a list of thread-id values.
-  ///
-  ///  @param Builder The builder used to emit the printer calls.
-  ///  @param Args    The list of values to print.
-  template <typename... Args>
-  static void createGPUPrinter(PollyIRBuilder &Builder, Args... args) {
-    std::vector<llvm::Value *> Vector;
-    createPrinter(Builder, /* GPU */ true, Vector, args...);
+    createPrinter(Builder, Vector, args...);
   }
 
 private:
   /// Handle Values.
   template <typename... Args>
-  static void createPrinter(PollyIRBuilder &Builder, bool UseGPU,
+  static void createPrinter(PollyIRBuilder &Builder,
                             std::vector<llvm::Value *> &Values,
                             llvm::Value *Value, Args... args) {
     Values.push_back(Value);
-    createPrinter(Builder, UseGPU, Values, args...);
+    createPrinter(Builder, Values, args...);
   }
 
   /// Handle StringRefs.
   template <typename... Args>
-  static void createPrinter(PollyIRBuilder &Builder, bool UseGPU,
+  static void createPrinter(PollyIRBuilder &Builder,
                             std::vector<llvm::Value *> &Values,
                             llvm::StringRef String, Args... args) {
     Values.push_back(getPrintableString(Builder, String));
-    createPrinter(Builder, UseGPU, Values, args...);
+    createPrinter(Builder, Values, args...);
   }
 
   /// Handle ArrayRefs.
   template <typename... Args>
-  static void createPrinter(PollyIRBuilder &Builder, bool UseGPU,
+  static void createPrinter(PollyIRBuilder &Builder,
                             std::vector<llvm::Value *> &Values,
                             llvm::ArrayRef<llvm::Value *> Array, Args... args) {
     Values.insert(Values.end(), Array.begin(), Array.end());
-    createPrinter(Builder, UseGPU, Values, args...);
+    createPrinter(Builder, Values, args...);
   }
 
   /// Print a list of Values.
-  static void createPrinter(PollyIRBuilder &Builder, bool UseGPU,
+  static void createPrinter(PollyIRBuilder &Builder,
                             llvm::ArrayRef<llvm::Value *> Values);
 
-  /// Print a list of Values on a GPU.
-  static void createGPUPrinterT(PollyIRBuilder &Builder,
-                                llvm::ArrayRef<llvm::Value *> Values);
-
   /// Print a list of Values on a CPU.
   static void createCPUPrinterT(PollyIRBuilder &Builder,
                                 llvm::ArrayRef<llvm::Value *> Values);
@@ -145,22 +120,6 @@ struct RuntimeDebugBuilder {
   ///
   /// @parma Builder The builder used to insert the code.
   static void createFlush(PollyIRBuilder &Builder);
-
-  /// Get (and possibly insert) a NVIDIA address space cast call.
-  static llvm::Function *getAddressSpaceCast(PollyIRBuilder &Builder,
-                                             unsigned Src, unsigned Dst,
-                                             unsigned SrcBits = 8,
-                                             unsigned DstBits = 8);
-
-  /// Get identifiers that describe the currently executed GPU thread.
-  ///
-  /// The result will be a vector that if passed to the GPU printer will result
-  /// into a string (initialized to values corresponding to the printing
-  /// thread):
-  ///
-  ///   "> block-id: bidx bid1y bidz | thread-id: tidx tidy tidz "
-  static std::vector<llvm::Value *>
-  getGPUThreadIdentifiers(PollyIRBuilder &Builder);
 };
 } // namespace polly
 
diff --git a/polly/include/polly/Config/config.h.cmake b/polly/include/polly/Config/config.h.cmake
index aaa101f5b8d70..74f509623bfef 100644
--- a/polly/include/polly/Config/config.h.cmake
+++ b/polly/include/polly/Config/config.h.cmake
@@ -12,7 +12,4 @@
 #ifndef POLLY_CONFIG_H
 #define POLLY_CONFIG_H
 
-#cmakedefine CUDA_FOUND
-#cmakedefine GPU_CODEGEN
-
 #endif
diff --git a/polly/include/polly/LinkAllPasses.h b/polly/include/polly/LinkAllPasses.h
index 71a74e6845761..a2f8f33299918 100644
--- a/polly/include/polly/LinkAllPasses.h
+++ b/polly/include/polly/LinkAllPasses.h
@@ -14,7 +14,6 @@
 #ifndef POLLY_LINKALLPASSES_H
 #define POLLY_LINKALLPASSES_H
 
-#include "polly/CodeGen/PPCGCodeGeneration.h"
 #include "polly/Config/config.h"
 #include "polly/Support/DumpFunctionPass.h"
 #include "polly/Support/DumpModulePass.h"
@@ -54,14 +53,6 @@ llvm::Pass *createScopInfoPrinterLegacyFunctionPass(llvm::raw_ostream &OS);
 llvm::Pass *createIslAstInfoWrapperPassPass();
 llvm::Pass *createIslAstInfoPrinterLegacyPass(llvm::raw_ostream &OS);
 llvm::Pass *createCodeGenerationPass();
-#ifdef GPU_CODEGEN
-llvm::Pass *createPPCGCodeGenerationPass(GPUArch Arch = GPUArch::NVPTX64,
-                                         GPURuntime Runtime = GPURuntime::CUDA);
-
-llvm::Pass *
-createManagedMemoryRewritePassPass(GPUArch Arch = GPUArch::NVPTX64,
-                                   GPURuntime Runtime = GPURuntime::CUDA);
-#endif
 llvm::Pass *createIslScheduleOptimizerWrapperPass();
 llvm::Pass *createIslScheduleOptimizerPrinterLegacyPass(llvm::raw_ostream &OS);
 llvm::Pass *createFlattenSchedulePass();
@@ -113,10 +104,6 @@ struct PollyForcePassLinking {
     polly::createIslAstInfoWrapperPassPass();
     polly::createIslAstInfoPrinterLegacyPass(llvm::outs());
     polly::createCodeGenerationPass();
-#ifdef GPU_CODEGEN
-    polly::createPPCGCodeGenerationPass();
-    polly::createManagedMemoryRewritePassPass();
-#endif
     polly::createIslScheduleOptimizerWrapperPass();
     polly::createIslScheduleOptimizerPrinterLegacyPass(llvm::outs());
     polly::createMaximalStaticExpansionPass();
@@ -156,10 +143,6 @@ void initializeDependenceInfoPrinterLegacyFunctionPassPass(
 void initializeIslAstInfoWrapperPassPass(llvm::PassRegistry &);
 void initializeIslAstInfoPrinterLegacyPassPass(llvm::PassRegistry &);
 void initializeCodeGenerationPass(llvm::PassRegistry &);
-#ifdef GPU_CODEGEN
-void initializePPCGCodeGenerationPass(llvm::PassRegistry &);
-void initializeManagedMemoryRewritePassPass(llvm::PassRegistry &);
-#endif
 void initializeIslScheduleOptimizerWrapperPassPass(llvm::PassRegistry &);
 void initializeIslScheduleOptimizerPrinterLegacyPassPass(llvm::PassRegistry &);
 void initializeMaximalStaticExpanderWrapperPassPass(llvm::PassRegistry &);
diff --git a/polly/include/polly/ScopInfo.h b/polly/include/polly/ScopInfo.h
index e62ac9372a5c7..2e6853d5446fb 100644
--- a/polly/include/polly/ScopInfo.h
+++ b/polly/include/polly/ScopInfo.h
@@ -1684,9 +1684,6 @@ class Scop final {
   /// Number of copy statements.
   unsigned CopyStmtsNum = 0;
 
-  /// Flag to indicate if the Scop is to be skipped.
-  bool SkipScop = false;
-
   using StmtSet = std::list<ScopStmt>;
 
   /// The statements in this Scop.
@@ -2144,12 +2141,6 @@ class Scop final {
   /// Check if the SCoP has been optimized by the scheduler.
   bool isOptimized() const { return IsOptimized; }
 
-  /// Mark the SCoP to be skipped by ScopPass passes.
-  void markAsToBeSkipped() { SkipScop = true; }
-
-  /// Check if the SCoP is to be skipped by ScopPass passes.
-  bool isToBeSkipped() const { return SkipScop; }
-
   /// Return the ID of the Scop
   int getID() const { return ID; }
 
diff --git a/polly/include/polly/Support/LinkGPURuntime.h b/polly/include/polly/Support/LinkGPURuntime.h
deleted file mode 100644
index c632d455a6375..0000000000000
--- a/polly/include/polly/Support/LinkGPURuntime.h
+++ /dev/null
@@ -1,42 +0,0 @@
-//===- Support/LinkGPURuntime.h -- Headerfile to help force-link GPURuntime  =//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// This header helps pull in libGPURuntime.so
-//
-//===----------------------------------------------------------------------===//
-#ifndef POLLY_LINK_GPURUNTIME
-#define POLLY_LINK_GPURUNTIME
-
-extern "C" {
-#include "GPURuntime/GPUJIT.h"
-}
-
-namespace polly {
-struct ForceGPURuntimeLinking {
-  ForceGPURuntimeLinking() {
-    if (std::getenv("bar") != (char *)-1)
-      return;
-    // We must reference GPURuntime in such a way that compilers will not
-    // delete it all as dead code, even with whole program optimization,
-    // yet is effectively a NO-OP. As the compiler isn't smart enough
-    // to know that getenv() never returns -1, this will do the job.
-    polly_initContextCL();
-    polly_initContextCUDA();
-    polly_getKernel(nullptr, nullptr);
-    polly_freeKernel(nullptr);
-    polly_copyFromHostToDevice(nullptr, nullptr, 0);
-    polly_copyFromDeviceToHost(nullptr, nullptr, 0);
-    polly_synchronizeDevice();
-    polly_launchKernel(nullptr, 0, 0, 0, 0, 0, nullptr);
-    polly_freeDeviceMemory(nullptr);
-    polly_freeContext(nullptr);
-    polly_synchronizeDevice();
-  }
-} structure;
-} // namespace polly
-#endif
diff --git a/polly/lib/CMakeLists.txt b/polly/lib/CMakeLists.txt
index 95e653f65da82..e17333b76dca5 100644
--- a/polly/lib/CMakeLists.txt
+++ b/polly/lib/CMakeLists.txt
@@ -6,13 +6,6 @@ set(ISL_CODEGEN_FILES
     CodeGen/IslNodeBuilder.cpp
     CodeGen/CodeGeneration.cpp)
 
-if (GPU_CODEGEN)
-  set (GPGPU_CODEGEN_FILES
-       CodeGen/PPCGCodeGeneration.cpp
-       CodeGen/ManagedMemoryRewrite.cpp
-       )
-endif (GPU_CODEGEN)
-
 # Compile ISL into a separate library.
 add_subdirectory(External)
 
@@ -44,12 +37,6 @@ set(POLLY_COMPONENTS
     Vectorize
 )
 
-# Polly-ACC requires the NVPTX backend to work. Ask LLVM about its libraries.
-if (GPU_CODEGEN)
-  # This call emits an error if they NVPTX backend is not enable.
-  list(APPEND POLLY_COMPONENTS NVPTX)
-endif ()
-
 # Use an object-library to add the same files to multiple libs without requiring
 # the sources them to be recompiled for each of them.
 add_llvm_pass_plugin(Polly
@@ -73,7 +60,6 @@ add_llvm_pass_plugin(Polly
   CodeGen/Utils.cpp
   CodeGen/RuntimeDebugBuilder.cpp
   CodeGen/PerfMonitor.cpp
-  ${GPGPU_CODEGEN_FILES}
   Exchange/JSONExporter.cpp
   Support/GICHelper.cpp
   Support/SCEVAffinator.cpp
@@ -127,16 +113,6 @@ target_link_libraries(Polly PUBLIC
   ${ISL_TARGET}
 )
 
-# Additional dependencies for Polly-ACC.
-if (GPU_CODEGEN)
-  target_link_libraries(Polly PUBLIC PollyPPCG)
-endif ()
-
-if (NOT LLVM_LINK_LLVM_DYLIB AND NOT LLVM_POLLY_LINK_INTO_TOOLS)
-    # Polly-ACC requires the NVPTX target to be present in the executable it is linked to
-    set_property(TARGET bugpoint APPEND PROPERTY LINK_LIBRARIES LLVMTarget)
-endif ()
-
 # Create a loadable module Polly.so that can be loaded using
 # LLVM's/clang's "-load" option.
 if (WIN32 OR NOT LLVM_ENABLE_PIC)
@@ -154,14 +130,7 @@ else ()
   # to be already available in the address space the module is loaded into.
   # Adding them once more would have the effect that both copies try to register
   # the same command line options, to which LLVM reacts with an error.
-  # If Polly-ACC is enabled, the NVPTX target is also expected to reside in the
-  # hosts. This is not the case for bugpoint. Use LLVM_POLLY_LINK_INTO_TOOLS=ON
-  # instead which will automatically resolve the additional dependencies by
-  # Polly.
   target_link_libraries(LLVMPolly PUBLIC ${ISL_TARGET})
-  if (GPU_CODEGEN)
-    target_link_libraries(LLVMPolly PUBLIC PollyPPCG)
-  endif ()
 
   set_target_properties(LLVMPolly
     PROPERTIES
diff --git a/polly/lib/CodeGen/BlockGenerators.cpp b/polly/lib/CodeGen/BlockGenerators.cpp
index 60d6b763dfb5e..1c18f2dbbbdbb 100644
--- a/polly/lib/CodeGen/BlockGenerators.cpp
+++ b/polly/lib/CodeGen/BlockGenerators.cpp
@@ -238,14 +238,8 @@ void BlockGenerator::copyInstScalar(ScopStmt &Stmt, Instruction *Inst,
   Builder.Insert(NewInst);
   BBMap[Inst] = NewInst;
 
-  // When copying the instruction onto the Module meant for the GPU,
-  // debug metadata attached to an instruction causes all related
-  // metadata to be pulled into the Module. This includes the DICompileUnit,
-  // which will not be listed in llvm.dbg.cu of the Module since the Module
-  // doesn't contain one. This fails the verification of the Module and the
-  // subsequent generation of the ASM string.
-  if (NewInst->getModule() != Inst->getModule())
-    NewInst->setDebugLoc(llvm::DebugLoc());
+  assert(NewInst->getModule() == Inst->getModule() &&
+         "Expecting instructions to be in the same module");
 
   if (!NewInst->getType()->isVoidTy())
     NewInst->setName("p_" + Inst->getName());
@@ -1009,391 +1003,6 @@ void BlockGenerator::finalizeSCoP(Scop &S) {
   invalidateScalarEvolution(S);
 }
 
-VectorBlockGenerator::VectorBlockGenerator(BlockGenerator &BlockGen,
-                                           std::vector<LoopToScevMapT> &VLTS,
-                                           isl_map *Schedule)
-    : BlockGenerator(BlockGen), VLTS(VLTS), Schedule(Schedule) {
-  assert(Schedule && "No statement domain provided");
-}
-
-Value *VectorBlockGenerator::getVectorValue(ScopStmt &Stmt, Value *Old,
-                                            ValueMapT &VectorMap,
-                                            VectorValueMapT &ScalarMaps,
-                                            Loop *L) {
-  if (Value *NewValue = VectorMap.lookup(Old))
-    return NewValue;
-
-  int Width = getVectorWidth();
-
-  Value *Vector = UndefValue::get(FixedVectorType::get(Old->getType(), Width));
-
-  for (int Lane = 0; Lane < Width; Lane++)
-    Vector = Builder.CreateInsertElement(
-        Vector, getNewValue(Stmt, Old, ScalarMaps[Lane], VLTS[Lane], L),
-        Builder.getInt32(Lane));
-
-  VectorMap[Old] = Vector;
-
-  return Vector;
-}
-
-Value *VectorBlockGenerator::generateStrideOneLoad(
-    ScopStmt &Stmt, LoadInst *Load, VectorValueMapT &ScalarMaps,
-    __isl_keep isl_id_to_ast_expr *NewAccesses, bool NegativeStride = false) {
-  unsigned VectorWidth = getVectorWidth();
-  Type *VectorType = FixedVectorType::get(Load->getType(), VectorWidth);
-  Type *VectorPtrType =
-      PointerType::get(VectorType, Load->getPointerAddressSpace());
-  unsigned Offset = NegativeStride ? VectorWidth - 1 : 0;
-
-  Value *NewPointer = generateLocationAccessed(Stmt, Load, ScalarMaps[Offset],
-                                               VLTS[Offset], NewAccesses);
-  Value *VectorPtr =
-      Builder.CreateBitCast(NewPointer, VectorPtrType, "vector_ptr");
-  LoadInst *VecLoad = Builder.CreateLoad(VectorType, VectorPtr,
-                                         Load->getName() + "_p_vec_full");
-  if (!Aligned)
-    VecLoad->setAlignment(Align(8));
-
-  if (NegativeStride) {
-    SmallVector<Constant *, 16> Indices;
-    for (int i = VectorWidth - 1; i >= 0; i--)
-      Indices.push_back(ConstantInt::get(Builder.getInt32Ty(), i));
-    Constant *SV = llvm::ConstantVector::get(Indices);
-    Value *RevVecLoad = Builder.CreateShuffleVector(
-        VecLoad, VecLoad, SV, Load->getName() + "_reverse");
-    return RevVecLoad;
-  }
-
-  return VecLoad;
-}
-
-Value *VectorBlockGenerator::generateStrideZeroLoad(
-    ScopStmt &Stmt, LoadInst *Load, ValueMapT &BBMap,
-    __isl_keep isl_id_to_ast_expr *NewAccesses) {
-  Type *VectorType = FixedVectorType::get(Load->getType(), 1);
-  Type *VectorPtrType =
-      PointerType::get(VectorType, Load->getPointerAddressSpace());
-  Value *NewPointer =
-      generateLocationAccessed(Stmt, Load, BBMap, VLTS[0], NewAccesses);
-  Value *VectorPtr = Builder.CreateBitCast(NewPointer, VectorPtrType,
-                                           Load->getName() + "_p_vec_p");
-  LoadInst *ScalarLoad = Builder.CreateLoad(VectorType, VectorPtr,
-                                            Load->getName() + "_p_splat_one");
-
-  if (!Aligned)
-    ScalarLoad->setAlignment(Align(8));
-
-  Constant *SplatVector = Constant::getNullValue(
-      FixedVectorType::get(Builder.getInt32Ty(), getVectorWidth()));
-
-  Value *VectorLoad = Builder.CreateShuffleVector(
-      ScalarLoad, ScalarLoad, SplatVector, Load->getName() + "_p_splat");
-  return VectorLoad;
-}
-
-Value *VectorBlockGenerator::generateUnknownStrideLoad(
-    ScopStmt &Stmt, LoadInst *Load, VectorValueMapT &ScalarMaps,
-    __isl_keep isl_id_to_ast_expr *NewAccesses) {
-  int VectorWidth = getVectorWidth();
-  Type *ElemTy = Load->getType();
-  auto *FVTy = FixedVectorType::get(ElemTy, VectorWidth);
-
-  Value *Vector = UndefValue::get(FVTy);
-
-  for (int i = 0; i < VectorWidth; i++) {
-    Value *NewPointer = generateLocationAccessed(Stmt, Load, ScalarMaps[i],
-                                                 VLTS[i], NewAccesses);
-    Value *ScalarLoad =
-        Builder.CreateLoad(ElemTy, NewPointer, Load->getName() + "_p_scalar_");
-    Vector = Builder.CreateInsertElement(
-        Vector, ScalarLoad, Builder.getInt32(i), Load->getName() + "_p_vec_");
-  }
-
-  return Vector;
-}
-
-void VectorBlockGenerator::generateLoad(
-    ScopStmt &Stmt, LoadInst *Load, ValueMapT &VectorMap,
-    VectorValueMapT &ScalarMaps, __isl_keep isl_id_to_ast_expr *NewAccesses) {
-  if (Value *PreloadLoad = GlobalMap.lookup(Load)) {
-    VectorMap[Load] = Builder.CreateVectorSplat(getVectorWidth(), PreloadLoad,
-                                                Load->getName() + "_p");
-    return;
-  }
-
-  if (!VectorType::isValidElementType(Load->getType())) {
-    for (int i = 0; i < getVectorWidth(); i++)
-      ScalarMaps[i][Load] =
-          generateArrayLoad(Stmt, Load, ScalarMaps[i], VLTS[i], NewAccesses);
-    return;
-  }
-
-  const MemoryAccess &Access = Stmt.getArrayAccessFor(Load);
-
-  // Make sure we have scalar values available to access the pointer to
-  // the data location.
-  extractScalarValues(Load, VectorMap, ScalarMaps);
-
-  Value *NewLoad;
-  if (Access.isStrideZero(isl::manage_copy(Schedule)))
-    NewLoad = generateStrideZeroLoad(Stmt, Load, ScalarMaps[0], NewAccesses);
-  else if (Access.isStrideOne(isl::manage_copy(Schedule)))
-    NewLoad = generateStrideOneLoad(Stmt, Load, ScalarMaps, NewAccesses);
-  else if (Access.isStrideX(isl::manage_copy(Schedule), -1))
-    NewLoad = generateStrideOneLoad(Stmt, Load, ScalarMaps, NewAccesses, true);
-  else
-    NewLoad = generateUnknownStrideLoad(Stmt, Load, ScalarMaps, NewAccesses);
-
-  VectorMap[Load] = NewLoad;
-}
-
-void VectorBlockGenerator::copyUnaryInst(ScopStmt &Stmt, UnaryInstruction *Inst,
-                                         ValueMapT &VectorMap,
-                                         VectorValueMapT &ScalarMaps) {
-  int VectorWidth = getVectorWidth();
-  Value *NewOperand = getVectorValue(Stmt, Inst->getOperand(0), VectorMap,
-                                     ScalarMaps, getLoopForStmt(Stmt));
-
-  assert(isa<CastInst>(Inst) && "Can not generate vector code for instruction");
-
-  const CastInst *Cast = dyn_cast<CastInst>(Inst);
-  auto *DestType = FixedVectorType::get(Inst->getType(), VectorWidth);
-  VectorMap[Inst] = Builder.CreateCast(Cast->getOpcode(), NewOperand, DestType);
-}
-
-void VectorBlockGenerator::copyBinaryInst(ScopStmt &Stmt, BinaryOperator *Inst,
-                                          ValueMapT &VectorMap,
-                                          VectorValueMapT &ScalarMaps) {
-  Loop *L = getLoopForStmt(Stmt);
-  Value *OpZero = Inst->getOperand(0);
-  Value *OpOne = Inst->getOperand(1);
-
-  Value *NewOpZero, *NewOpOne;
-  NewOpZero = getVectorValue(Stmt, OpZero, VectorMap, ScalarMaps, L);
-  NewOpOne = getVectorValue(Stmt, OpOne, VectorMap, ScalarMaps, L);
-
-  Value *NewInst = Builder.CreateBinOp(Inst->getOpcode(), NewOpZero, NewOpOne,
-                                       Inst->getName() + "p_vec");
-  VectorMap[Inst] = NewInst;
-}
-
-void VectorBlockGenerator::copyStore(
-    ScopStmt &Stmt, StoreInst *Store, ValueMapT &VectorMap,
-    VectorValueMapT &ScalarMaps, __isl_keep isl_id_to_ast_expr *NewAccesses) {
-  const MemoryAccess &Access = Stmt.getArrayAccessFor(Store);
-
-  Value *Vector = getVectorValue(Stmt, Store->getValueOperand(), VectorMap,
-                                 ScalarMaps, getLoopForStmt(Stmt));
-
-  // Make sure we have scalar values available to access the pointer to
-  // the data location.
-  extractScalarValues(Store, VectorMap, ScalarMaps);
-
-  if (Access.isStrideOne(isl::manage_copy(Schedule))) {
-    Type *VectorType = FixedVectorType::get(Store->getValueOperand()->getType(),
-                                            getVectorWidth());
-    Type *VectorPtrType =
-        PointerType::get(VectorType, Store->getPointerAddressSpace());
-    Value *NewPointer = generateLocationAccessed(Stmt, Store, ScalarMaps[0],
-                                                 VLTS[0], NewAccesses);
-
-    Value *VectorPtr =
-        Builder.CreateBitCast(NewPointer, VectorPtrType, "vector_ptr");
-    StoreInst *Store = Builder.CreateStore(Vector, VectorPtr);
-
-    if (!Aligned)
-      Store->setAlignment(Align(8));
-  } else {
-    for (unsigned i = 0; i < ScalarMaps.size(); i++) {
-      Value *Scalar = Builder.CreateExtractElement(Vector, Builder.getInt32(i));
-      Value *NewPointer = generateLocationAccessed(Stmt, Store, ScalarMaps[i],
-                                                   VLTS[i], NewAccesses);
-      Builder.CreateStore(Scalar, NewPointer);
-    }
-  }
-}
-
-bool VectorBlockGenerator::hasVectorOperands(const Instruction *Inst,
-                                             ValueMapT &VectorMap) {
-  for (Value *Operand : Inst->operands())
-    if (VectorMap.count(Operand))
-      return true;
-  return false;
-}
-
-bool VectorBlockGenerator::extractScalarValues(const Instruction *Inst,
-                                               ValueMapT &VectorMap,
-                                               VectorValueMapT &ScalarMaps) {
-  bool HasVectorOperand = false;
-  int VectorWidth = getVectorWidth();
-
-  for (Value *Operand : Inst->operands()) {
-    ValueMapT::iterator VecOp = VectorMap.find(Operand);
-
-    if (VecOp == VectorMap.end())
-      continue;
-
-    HasVectorOperand = true;
-    Value *NewVector = VecOp->second;
-
-    for (int i = 0; i < VectorWidth; ++i) {
-      ValueMapT &SM = ScalarMaps[i];
-
-      // If there is one scalar extracted, all scalar elements should have
-      // already been extracted by the code here. So no need to check for the
-      // existence of all of them.
-      if (SM.count(Operand))
-        break;
-
-      SM[Operand] =
-          Builder.CreateExtractElement(NewVector, Builder.getInt32(i));
-    }
-  }
-
-  return HasVectorOperand;
-}
-
-void VectorBlockGenerator::copyInstScalarized(
-    ScopStmt &Stmt, Instruction *Inst, ValueMapT &VectorMap,
-    VectorValueMapT &ScalarMaps, __isl_keep isl_id_to_ast_expr *NewAccesses) {
-  bool HasVectorOperand;
-  int VectorWidth = getVectorWidth();
-
-  HasVectorOperand = extractScalarValues(Inst, VectorMap, ScalarMaps);
-
-  for (int VectorLane = 0; VectorLane < getVectorWidth(); VectorLane++)
-    BlockGenerator::copyInstruction(Stmt, Inst, ScalarMaps[VectorLane],
-                                    VLTS[VectorLane], NewAccesses);
-
-  if (!VectorType::isValidElementType(Inst->getType()) || !HasVectorOperand)
-    return;
-
-  // Make the result available as vector value.
-  auto *FVTy = FixedVectorType::get(Inst->getType(), VectorWidth);
-  Value *Vector = UndefValue::get(FVTy);
-
-  for (int i = 0; i < VectorWidth; i++)
-    Vector = Builder.CreateInsertElement(Vector, ScalarMaps[i][Inst],
-                                         Builder.getInt32(i));
-
-  VectorMap[Inst] = Vector;
-}
-
-int VectorBlockGenerator::getVectorWidth() { return VLTS.size(); }
-
-void VectorBlockGenerator::copyInstruction(
-    ScopStmt &Stmt, Instruction *Inst, ValueMapT &VectorMap,
-    VectorValueMapT &ScalarMaps, __isl_keep isl_id_to_ast_expr *NewAccesses) {
-  // Terminator instructions control the control flow. They are explicitly
-  // expressed in the clast and do not need to be copied.
-  if (Inst->isTerminator())
-    return;
-
-  if (canSyntheziseInStmt(Stmt, Inst))
-    return;
-
-  if (auto *Load = dyn_cast<LoadInst>(Inst)) {
-    generateLoad(Stmt, Load, VectorMap, ScalarMaps, NewAccesses);
-    return;
-  }
-
-  if (hasVectorOperands(Inst, VectorMap)) {
-    if (auto *Store = dyn_cast<StoreInst>(Inst)) {
-      // Identified as redundant by -polly-simplify.
-      if (!Stmt.getArrayAccessOrNULLFor(Store))
-        return;
-
-      copyStore(Stmt, Store, VectorMap, ScalarMaps, NewAccesses);
-      return;
-    }
-
-    if (auto *Unary = dyn_cast<UnaryInstruction>(Inst)) {
-      copyUnaryInst(Stmt, Unary, VectorMap, ScalarMaps);
-      return;
-    }
-
-    if (auto *Binary = dyn_cast<BinaryOperator>(Inst)) {
-      copyBinaryInst(Stmt, Binary, VectorMap, ScalarMaps);
-      return;
-    }
-
-    // Fallthrough: We generate scalar instructions, if we don't know how to
-    // generate vector code.
-  }
-
-  copyInstScalarized(Stmt, Inst, VectorMap, ScalarMaps, NewAccesses);
-}
-
-void VectorBlockGenerator::generateScalarVectorLoads(
-    ScopStmt &Stmt, ValueMapT &VectorBlockMap) {
-  for (MemoryAccess *MA : Stmt) {
-    if (MA->isArrayKind() || MA->isWrite())
-      continue;
-
-    auto *Address = getOrCreateAlloca(*MA);
-    Type *VectorType = FixedVectorType::get(MA->getElementType(), 1);
-    Type *VectorPtrType = PointerType::get(
-        VectorType, Address->getType()->getPointerAddressSpace());
-    Value *VectorPtr = Builder.CreateBitCast(Address, VectorPtrType,
-                                             Address->getName() + "_p_vec_p");
-    auto *Val = Builder.CreateLoad(VectorType, VectorPtr,
-                                   Address->getName() + ".reload");
-    Constant *SplatVector = Constant::getNullValue(
-        FixedVectorType::get(Builder.getInt32Ty(), getVectorWidth()));
-
-    Value *VectorVal = Builder.CreateShuffleVector(
-        Val, Val, SplatVector, Address->getName() + "_p_splat");
-    VectorBlockMap[MA->getAccessValue()] = VectorVal;
-  }
-}
-
-void VectorBlockGenerator::verifyNoScalarStores(ScopStmt &Stmt) {
-  for (MemoryAccess *MA : Stmt) {
-    if (MA->isArrayKind() || MA->isRead())
-      continue;
-
-    llvm_unreachable("Scalar stores not expected in vector loop");
-  }
-}
-
-void VectorBlockGenerator::copyStmt(
-    ScopStmt &Stmt, __isl_keep isl_id_to_ast_expr *NewAccesses) {
-  assert(Stmt.isBlockStmt() &&
-         "TODO: Only block statements can be copied by the vector block "
-         "generator");
-
-  BasicBlock *BB = Stmt.getBasicBlock();
-  BasicBlock *CopyBB = SplitBlock(Builder.GetInsertBlock(),
-                                  &*Builder.GetInsertPoint(), &DT, &LI);
-  CopyBB->setName("polly.stmt." + BB->getName());
-  Builder.SetInsertPoint(&CopyBB->front());
-
-  // Create two maps that store the mapping from the original instructions of
-  // the old basic block to their copies in the new basic block. Those maps
-  // are basic block local.
-  //
-  // As vector code generation is supported there is one map for scalar values
-  // and one for vector values.
-  //
-  // In case we just do scalar code generation, the vectorMap is not used and
-  // the scalarMap has just one dimension, which contains the mapping.
-  //
-  // In case vector code generation is done, an instruction may either appear
-  // in the vector map once (as it is calculating >vectorwidth< values at a
-  // time. Or (if the values are calculated using scalar operations), it
-  // appears once in every dimension of the scalarMap.
-  VectorValueMapT ScalarBlockMap(getVectorWidth());
-  ValueMapT VectorBlockMap;
-
-  generateScalarVectorLoads(Stmt, VectorBlockMap);
-
-  for (Instruction *Inst : Stmt.getInstructions())
-    copyInstruction(Stmt, Inst, VectorBlockMap, ScalarBlockMap, NewAccesses);
-
-  verifyNoScalarStores(Stmt);
-}
-
 BasicBlock *RegionGenerator::repairDominance(BasicBlock *BB,
                                              BasicBlock *BBCopy) {
 
diff --git a/polly/lib/CodeGen/CodeGeneration.cpp b/polly/lib/CodeGen/CodeGeneration.cpp
index f290be6712283..3792d995fd990 100644
--- a/polly/lib/CodeGen/CodeGeneration.cpp
+++ b/polly/lib/CodeGen/CodeGeneration.cpp
@@ -323,10 +323,6 @@ class CodeGeneration final : public ScopPass {
 
   /// Generate LLVM-IR for the SCoP @p S.
   bool runOnScop(Scop &S) override {
-    // Skip SCoPs in case they're already code-generated by PPCGCodeGeneration.
-    if (S.isToBeSkipped())
-      return false;
-
     AI = &getAnalysis<IslAstInfoWrapperPass>().getAI();
     LI = &getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
     DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
diff --git a/polly/lib/CodeGen/IslAst.cpp b/polly/lib/CodeGen/IslAst.cpp
index 5041db931a19c..fe2b5d9104d7d 100644
--- a/polly/lib/CodeGen/IslAst.cpp
+++ b/polly/lib/CodeGen/IslAst.cpp
@@ -638,10 +638,6 @@ isl::ast_build IslAstInfo::getBuild(const isl::ast_node &Node) {
 static std::unique_ptr<IslAstInfo> runIslAst(
     Scop &Scop,
     function_ref<const Dependences &(Dependences::AnalysisLevel)> GetDeps) {
-  // Skip SCoPs in case they're already handled by PPCGCodeGeneration.
-  if (Scop.isToBeSkipped())
-    return {};
-
   ScopsProcessed++;
 
   const Dependences &D = GetDeps(Dependences::AL_Statement);
diff --git a/polly/lib/CodeGen/IslExprBuilder.cpp b/polly/lib/CodeGen/IslExprBuilder.cpp
index 12b724c8f7bf4..f40511e0273a2 100644
--- a/polly/lib/CodeGen/IslExprBuilder.cpp
+++ b/polly/lib/CodeGen/IslExprBuilder.cpp
@@ -271,13 +271,6 @@ IslExprBuilder::createAccessAddress(__isl_take isl_ast_expr *Expr) {
   assert(Base->getType()->isPointerTy() && "Access base should be a pointer");
   StringRef BaseName = Base->getName();
 
-  auto PointerTy = PointerType::get(SAI->getElementType(),
-                                    Base->getType()->getPointerAddressSpace());
-  if (Base->getType() != PointerTy) {
-    Base =
-        Builder.CreateBitCast(Base, PointerTy, "polly.access.cast." + BaseName);
-  }
-
   if (isl_ast_expr_get_op_n_arg(Expr) == 1) {
     isl_ast_expr_free(Expr);
     if (PollyDebugPrinting)
diff --git a/polly/lib/CodeGen/IslNodeBuilder.cpp b/polly/lib/CodeGen/IslNodeBuilder.cpp
index 843fa0e86a954..c91dd76657570 100644
--- a/polly/lib/CodeGen/IslNodeBuilder.cpp
+++ b/polly/lib/CodeGen/IslNodeBuilder.cpp
@@ -77,7 +77,6 @@ STATISTIC(VersionedScops, "Number of SCoPs that required versioning.");
 
 STATISTIC(SequentialLoops, "Number of generated sequential for-loops");
 STATISTIC(ParallelLoops, "Number of generated parallel for-loops");
-STATISTIC(VectorLoops, "Number of generated vector for-loops");
 STATISTIC(IfConditions, "Number of generated if-conditions");
 
 /// OpenMP backend options
@@ -390,30 +389,6 @@ Value *IslNodeBuilder::getLatestValue(Value *Original) const {
   return It->second;
 }
 
-void IslNodeBuilder::createUserVector(__isl_take isl_ast_node *User,
-                                      std::vector<Value *> &IVS,
-                                      __isl_take isl_id *IteratorID,
-                                      __isl_take isl_union_map *Schedule) {
-  isl_ast_expr *Expr = isl_ast_node_user_get_expr(User);
-  isl_ast_expr *StmtExpr = isl_ast_expr_get_op_arg(Expr, 0);
-  isl_id *Id = isl_ast_expr_get_id(StmtExpr);
-  isl_ast_expr_free(StmtExpr);
-  ScopStmt *Stmt = (ScopStmt *)isl_id_get_user(Id);
-  std::vector<LoopToScevMapT> VLTS(IVS.size());
-
-  isl_union_set *Domain = isl_union_set_from_set(Stmt->getDomain().release());
-  Schedule = isl_union_map_intersect_domain(Schedule, Domain);
-  isl_map *S = isl_map_from_union_map(Schedule);
-
-  auto *NewAccesses = createNewAccesses(Stmt, User);
-  createSubstitutionsVector(Expr, Stmt, VLTS, IVS, IteratorID);
-  VectorBlockGenerator::generate(BlockGen, *Stmt, VLTS, S, NewAccesses);
-  isl_id_to_ast_expr_free(NewAccesses);
-  isl_map_free(S);
-  isl_id_free(Id);
-  isl_ast_node_free(User);
-}
-
 void IslNodeBuilder::createMark(__isl_take isl_ast_node *Node) {
   auto *Id = isl_ast_node_mark_get_id(Node);
   auto Child = isl_ast_node_mark_get_node(Node);
@@ -422,13 +397,7 @@ void IslNodeBuilder::createMark(__isl_take isl_ast_node *Node) {
   // it will be optimized away and we should skip it.
   if (strcmp(isl_id_get_name(Id), "SIMD") == 0 &&
       isl_ast_node_get_type(Child) == isl_ast_node_for) {
-    bool Vector = PollyVectorizerChoice == VECTORIZER_POLLY;
-    int VectorWidth =
-        getNumberOfIterations(isl::manage_copy(Child).as<isl::ast_node_for>());
-    if (Vector && 1 < VectorWidth && VectorWidth <= 16)
-      createForVector(Child, VectorWidth);
-    else
-      createForSequential(isl::manage(Child).as<isl::ast_node_for>(), true);
+    createForSequential(isl::manage(Child).as<isl::ast_node_for>(), true);
     isl_id_free(Id);
     return;
   }
@@ -456,67 +425,6 @@ void IslNodeBuilder::createMark(__isl_take isl_ast_node *Node) {
   isl_id_free(Id);
 }
 
-void IslNodeBuilder::createForVector(__isl_take isl_ast_node *For,
-                                     int VectorWidth) {
-  isl_ast_node *Body = isl_ast_node_for_get_body(For);
-  isl_ast_expr *Init = isl_ast_node_for_get_init(For);
-  isl_ast_expr *Inc = isl_ast_node_for_get_inc(For);
-  isl_ast_expr *Iterator = isl_ast_node_for_get_iterator(For);
-  isl_id *IteratorID = isl_ast_expr_get_id(Iterator);
-
-  Value *ValueLB = ExprBuilder.create(Init);
-  Value *ValueInc = ExprBuilder.create(Inc);
-
-  Type *MaxType = ExprBuilder.getType(Iterator);
-  MaxType = ExprBuilder.getWidestType(MaxType, ValueLB->getType());
-  MaxType = ExprBuilder.getWidestType(MaxType, ValueInc->getType());
-
-  if (MaxType != ValueLB->getType())
-    ValueLB = Builder.CreateSExt(ValueLB, MaxType);
-  if (MaxType != ValueInc->getType())
-    ValueInc = Builder.CreateSExt(ValueInc, MaxType);
-
-  std::vector<Value *> IVS(VectorWidth);
-  IVS[0] = ValueLB;
-
-  for (int i = 1; i < VectorWidth; i++)
-    IVS[i] = Builder.CreateAdd(IVS[i - 1], ValueInc, "p_vector_iv");
-
-  isl::union_map Schedule = getScheduleForAstNode(isl::manage_copy(For));
-  assert(!Schedule.is_null() &&
-         "For statement annotation does not contain its schedule");
-
-  IDToValue[IteratorID] = ValueLB;
-
-  switch (isl_ast_node_get_type(Body)) {
-  case isl_ast_node_user:
-    createUserVector(Body, IVS, isl_id_copy(IteratorID), Schedule.copy());
-    break;
-  case isl_ast_node_block: {
-    isl_ast_node_list *List = isl_ast_node_block_get_children(Body);
-
-    for (int i = 0; i < isl_ast_node_list_n_ast_node(List); ++i)
-      createUserVector(isl_ast_node_list_get_ast_node(List, i), IVS,
-                       isl_id_copy(IteratorID), Schedule.copy());
-
-    isl_ast_node_free(Body);
-    isl_ast_node_list_free(List);
-    break;
-  }
-  default:
-    isl_ast_node_dump(Body);
-    llvm_unreachable("Unhandled isl_ast_node in vectorizer");
-  }
-
-  IDToValue.erase(IDToValue.find(IteratorID));
-  isl_id_free(IteratorID);
-
-  isl_ast_node_free(For);
-  isl_ast_expr_free(Iterator);
-
-  VectorLoops++;
-}
-
 /// Restore the initial ordering of dimensions of the band node
 ///
 /// In case the band node represents all the dimensions of the iteration
@@ -761,46 +669,7 @@ void IslNodeBuilder::createForParallel(__isl_take isl_ast_node *For) {
   ParallelLoops++;
 }
 
-/// Return whether any of @p Node's statements contain partial accesses.
-///
-/// Partial accesses are not supported by Polly's vector code generator.
-static bool hasPartialAccesses(__isl_take isl_ast_node *Node) {
-  return isl_ast_node_foreach_descendant_top_down(
-             Node,
-             [](isl_ast_node *Node, void *User) -> isl_bool {
-               if (isl_ast_node_get_type(Node) != isl_ast_node_user)
-                 return isl_bool_true;
-
-               isl::ast_expr Expr =
-                   isl::manage(isl_ast_node_user_get_expr(Node));
-               isl::ast_expr StmtExpr = Expr.get_op_arg(0);
-               isl::id Id = StmtExpr.get_id();
-
-               ScopStmt *Stmt =
-                   static_cast<ScopStmt *>(isl_id_get_user(Id.get()));
-               isl::set StmtDom = Stmt->getDomain();
-               for (auto *MA : *Stmt) {
-                 if (MA->isLatestPartialAccess())
-                   return isl_bool_error;
-               }
-               return isl_bool_true;
-             },
-             nullptr) == isl_stat_error;
-}
-
 void IslNodeBuilder::createFor(__isl_take isl_ast_node *For) {
-  bool Vector = PollyVectorizerChoice == VECTORIZER_POLLY;
-
-  if (Vector && IslAstInfo::isInnermostParallel(isl::manage_copy(For)) &&
-      !IslAstInfo::isReductionParallel(isl::manage_copy(For))) {
-    int VectorWidth =
-        getNumberOfIterations(isl::manage_copy(For).as<isl::ast_node_for>());
-    if (1 < VectorWidth && VectorWidth <= 16 && !hasPartialAccesses(For)) {
-      createForVector(For, VectorWidth);
-      return;
-    }
-  }
-
   if (IslAstInfo::isExecutedInParallel(isl::manage_copy(For))) {
     createForParallel(For);
     return;
@@ -981,7 +850,7 @@ void IslNodeBuilder::generateCopyStmt(
 }
 
 Value *IslNodeBuilder::materializeNonScopLoopInductionVariable(const Loop *L) {
-  assert(OutsideLoopIterations.find(L) == OutsideLoopIterations.end() &&
+  assert(!OutsideLoopIterations.contains(L) &&
          "trying to materialize loop induction variable twice");
   const SCEV *OuterLIV = SE.getAddRecExpr(SE.getUnknown(Builder.getInt64(0)),
                                           SE.getUnknown(Builder.getInt64(1)), L,
diff --git a/polly/lib/CodeGen/LoopGenerators.cpp b/polly/lib/CodeGen/LoopGenerators.cpp
index ebd4b22476c90..5c99515c54440 100644
--- a/polly/lib/CodeGen/LoopGenerators.cpp
+++ b/polly/lib/CodeGen/LoopGenerators.cpp
@@ -186,15 +186,12 @@ Value *ParallelLoopGenerator::createParallelLoop(
   *LoopBody = Builder.GetInsertPoint();
   Builder.SetInsertPoint(&*BeforeLoop);
 
-  Value *SubFnParam = Builder.CreateBitCast(Struct, Builder.getInt8PtrTy(),
-                                            "polly.par.userContext");
-
   // Add one as the upper bound provided by OpenMP is a < comparison
   // whereas the codegenForSequential function creates a <= comparison.
   UB = Builder.CreateAdd(UB, ConstantInt::get(LongType, 1));
 
   // Execute the prepared subfunction in parallel.
-  deployParallelExecution(SubFn, SubFnParam, LB, UB, Stride);
+  deployParallelExecution(SubFn, Struct, LB, UB, Stride);
 
   return IV;
 }
diff --git a/polly/lib/CodeGen/LoopGeneratorsGOMP.cpp b/polly/lib/CodeGen/LoopGeneratorsGOMP.cpp
index 1013f12e2a795..254c5e973f664 100644
--- a/polly/lib/CodeGen/LoopGeneratorsGOMP.cpp
+++ b/polly/lib/CodeGen/LoopGeneratorsGOMP.cpp
@@ -128,8 +128,7 @@ ParallelLoopGeneratorGOMP::createSubFn(Value *Stride, AllocaInst *StructData,
   Builder.SetInsertPoint(HeaderBB);
   Value *LBPtr = Builder.CreateAlloca(LongType, nullptr, "polly.par.LBPtr");
   Value *UBPtr = Builder.CreateAlloca(LongType, nullptr, "polly.par.UBPtr");
-  Value *UserContext = Builder.CreateBitCast(
-      &*SubFn->arg_begin(), StructData->getType(), "polly.par.userContext");
+  Value *UserContext = &*SubFn->arg_begin();
 
   extractValuesFromStruct(Data, StructData->getAllocatedType(), UserContext,
                           Map);
diff --git a/polly/lib/CodeGen/LoopGeneratorsKMP.cpp b/polly/lib/CodeGen/LoopGeneratorsKMP.cpp
index 5244e89e31072..5bcfa664c11b6 100644
--- a/polly/lib/CodeGen/LoopGeneratorsKMP.cpp
+++ b/polly/lib/CodeGen/LoopGeneratorsKMP.cpp
@@ -175,11 +175,7 @@ ParallelLoopGeneratorKMP::createSubFn(Value *SequentialLoopStride,
   std::advance(AI, 1);
   Value *Shared = &*AI;
 
-  Value *UserContext = Builder.CreateBitCast(Shared, StructData->getType(),
-                                             "polly.par.userContext");
-
-  extractValuesFromStruct(Data, StructData->getAllocatedType(), UserContext,
-                          Map);
+  extractValuesFromStruct(Data, StructData->getAllocatedType(), Shared, Map);
 
   const auto Alignment = llvm::Align(is64BitArch() ? 8 : 4);
   Value *ID = Builder.CreateAlignedLoad(Builder.getInt32Ty(), IDPtr, Alignment,
diff --git a/polly/lib/CodeGen/ManagedMemoryRewrite.cpp b/polly/lib/CodeGen/ManagedMemoryRewrite.cpp
deleted file mode 100644
index 9b8e773d40895..0000000000000
--- a/polly/lib/CodeGen/ManagedMemoryRewrite.cpp
+++ /dev/null
@@ -1,427 +0,0 @@
-//===---- ManagedMemoryRewrite.cpp - Rewrite global & malloc'd memory -----===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// Take a module and rewrite:
-// 1. `malloc` -> `polly_mallocManaged`
-// 2. `free` -> `polly_freeManaged`
-// 3. global arrays with initializers -> global arrays that are initialized
-//                                       with a constructor call to
-//                                       `polly_mallocManaged`.
-//
-//===----------------------------------------------------------------------===//
-
-#include "polly/CodeGen/IRBuilder.h"
-#include "polly/CodeGen/PPCGCodeGeneration.h"
-#include "polly/DependenceInfo.h"
-#include "polly/LinkAllPasses.h"
-#include "polly/Options.h"
-#include "polly/ScopDetection.h"
-#include "llvm/ADT/SmallSet.h"
-#include "llvm/Analysis/CaptureTracking.h"
-#include "llvm/InitializePasses.h"
-#include "llvm/Transforms/Utils/ModuleUtils.h"
-
-using namespace llvm;
-using namespace polly;
-
-static cl::opt<bool> RewriteAllocas(
-    "polly-acc-rewrite-allocas",
-    cl::desc(
-        "Ask the managed memory rewriter to also rewrite alloca instructions"),
-    cl::Hidden, cl::cat(PollyCategory));
-
-static cl::opt<bool> IgnoreLinkageForGlobals(
-    "polly-acc-rewrite-ignore-linkage-for-globals",
-    cl::desc(
-        "By default, we only rewrite globals with internal linkage. This flag "
-        "enables rewriting of globals regardless of linkage"),
-    cl::Hidden, cl::cat(PollyCategory));
-
-#define DEBUG_TYPE "polly-acc-rewrite-managed-memory"
-namespace {
-
-static llvm::Function *getOrCreatePollyMallocManaged(Module &M) {
-  const char *Name = "polly_mallocManaged";
-  Function *F = M.getFunction(Name);
-
-  // If F is not available, declare it.
-  if (!F) {
-    GlobalValue::LinkageTypes Linkage = Function::ExternalLinkage;
-    PollyIRBuilder Builder(M.getContext());
-    // TODO: How do I get `size_t`? I assume from DataLayout?
-    FunctionType *Ty = FunctionType::get(Builder.getInt8PtrTy(),
-                                         {Builder.getInt64Ty()}, false);
-    F = Function::Create(Ty, Linkage, Name, &M);
-  }
-
-  return F;
-}
-
-static llvm::Function *getOrCreatePollyFreeManaged(Module &M) {
-  const char *Name = "polly_freeManaged";
-  Function *F = M.getFunction(Name);
-
-  // If F is not available, declare it.
-  if (!F) {
-    GlobalValue::LinkageTypes Linkage = Function::ExternalLinkage;
-    PollyIRBuilder Builder(M.getContext());
-    // TODO: How do I get `size_t`? I assume from DataLayout?
-    FunctionType *Ty =
-        FunctionType::get(Builder.getVoidTy(), {Builder.getInt8PtrTy()}, false);
-    F = Function::Create(Ty, Linkage, Name, &M);
-  }
-
-  return F;
-}
-
-// Expand a constant expression `Cur`, which is used at instruction `Parent`
-// at index `index`.
-// Since a constant expression can expand to multiple instructions, store all
-// the expands into a set called `Expands`.
-// Note that this goes inorder on the constant expression tree.
-// A * ((B * D) + C)
-// will be processed with first A, then B * D, then B, then D, and then C.
-// Though ConstantExprs are not treated as "trees" but as DAGs, since you can
-// have something like this:
-//    *
-//   /  \
-//   \  /
-//    (D)
-//
-// For the purposes of this expansion, we expand the two occurences of D
-// separately. Therefore, we expand the DAG into the tree:
-//  *
-// / \
-// D  D
-// TODO: We don't _have_to do this, but this is the simplest solution.
-// We can write a solution that keeps track of which constants have been
-// already expanded.
-static void expandConstantExpr(ConstantExpr *Cur, PollyIRBuilder &Builder,
-                               Instruction *Parent, int index,
-                               SmallPtrSet<Instruction *, 4> &Expands) {
-  assert(Cur && "invalid constant expression passed");
-  Instruction *I = Cur->getAsInstruction();
-  assert(I && "unable to convert ConstantExpr to Instruction");
-
-  LLVM_DEBUG(dbgs() << "Expanding ConstantExpression: (" << *Cur
-                    << ") in Instruction: (" << *I << ")\n";);
-
-  // Invalidate `Cur` so that no one after this point uses `Cur`. Rather,
-  // they should mutate `I`.
-  Cur = nullptr;
-
-  Expands.insert(I);
-  Parent->setOperand(index, I);
-
-  // The things that `Parent` uses (its operands) should be created
-  // before `Parent`.
-  Builder.SetInsertPoint(Parent);
-  Builder.Insert(I);
-
-  for (unsigned i = 0; i < I->getNumOperands(); i++) {
-    Value *Op = I->getOperand(i);
-    assert(isa<Constant>(Op) && "constant must have a constant operand");
-
-    if (ConstantExpr *CExprOp = dyn_cast<ConstantExpr>(Op))
-      expandConstantExpr(CExprOp, Builder, I, i, Expands);
-  }
-}
-
-// Edit all uses of `OldVal` to NewVal` in `Inst`. This will rewrite
-// `ConstantExpr`s that are used in the `Inst`.
-// Note that `replaceAllUsesWith` is insufficient for this purpose because it
-// does not rewrite values in `ConstantExpr`s.
-static void rewriteOldValToNew(Instruction *Inst, Value *OldVal, Value *NewVal,
-                               PollyIRBuilder &Builder) {
-
-  // This contains a set of instructions in which OldVal must be replaced.
-  // We start with `Inst`, and we fill it up with the expanded `ConstantExpr`s
-  // from `Inst`s arguments.
-  // We need to go through this process because `replaceAllUsesWith` does not
-  // actually edit `ConstantExpr`s.
-  SmallPtrSet<Instruction *, 4> InstsToVisit = {Inst};
-
-  // Expand all `ConstantExpr`s and place it in `InstsToVisit`.
-  for (unsigned i = 0; i < Inst->getNumOperands(); i++) {
-    Value *Operand = Inst->getOperand(i);
-    if (ConstantExpr *ValueConstExpr = dyn_cast<ConstantExpr>(Operand))
-      expandConstantExpr(ValueConstExpr, Builder, Inst, i, InstsToVisit);
-  }
-
-  // Now visit each instruction and use `replaceUsesOfWith`. We know that
-  // will work because `I` cannot have any `ConstantExpr` within it.
-  for (Instruction *I : InstsToVisit)
-    I->replaceUsesOfWith(OldVal, NewVal);
-}
-
-// Given a value `Current`, return all Instructions that may contain `Current`
-// in an expression.
-// We need this auxiliary function, because if we have a
-// `Constant` that is a user of `V`, we need to recurse into the
-// `Constant`s uses to gather the root instruction.
-static void getInstructionUsersOfValue(Value *V,
-                                       SmallVector<Instruction *, 4> &Owners) {
-  if (auto *I = dyn_cast<Instruction>(V)) {
-    Owners.push_back(I);
-  } else {
-    // Anything that is a `User` must be a constant or an instruction.
-    auto *C = cast<Constant>(V);
-    for (Use &CUse : C->uses())
-      getInstructionUsersOfValue(CUse.getUser(), Owners);
-  }
-}
-
-static void
-replaceGlobalArray(Module &M, const DataLayout &DL, GlobalVariable &Array,
-                   SmallPtrSet<GlobalVariable *, 4> &ReplacedGlobals) {
-  // We only want arrays.
-  ArrayType *ArrayTy = dyn_cast<ArrayType>(Array.getValueType());
-  if (!ArrayTy)
-    return;
-  Type *ElemTy = ArrayTy->getElementType();
-  PointerType *ElemPtrTy = ElemTy->getPointerTo();
-
-  // We only wish to replace arrays that are visible in the module they
-  // inhabit. Otherwise, our type edit from [T] to T* would be illegal across
-  // modules.
-  const bool OnlyVisibleInsideModule = Array.hasPrivateLinkage() ||
-                                       Array.hasInternalLinkage() ||
-                                       IgnoreLinkageForGlobals;
-  if (!OnlyVisibleInsideModule) {
-    LLVM_DEBUG(
-        dbgs() << "Not rewriting (" << Array
-               << ") to managed memory "
-                  "because it could be visible externally. To force rewrite, "
-                  "use -polly-acc-rewrite-ignore-linkage-for-globals.\n");
-    return;
-  }
-
-  if (!Array.hasInitializer() ||
-      !isa<ConstantAggregateZero>(Array.getInitializer())) {
-    LLVM_DEBUG(dbgs() << "Not rewriting (" << Array
-                      << ") to managed memory "
-                         "because it has an initializer which is "
-                         "not a zeroinitializer.\n");
-    return;
-  }
-
-  // At this point, we have committed to replacing this array.
-  ReplacedGlobals.insert(&Array);
-
-  std::string NewName = Array.getName().str();
-  NewName += ".toptr";
-  GlobalVariable *ReplacementToArr =
-      cast<GlobalVariable>(M.getOrInsertGlobal(NewName, ElemPtrTy));
-  ReplacementToArr->setInitializer(ConstantPointerNull::get(ElemPtrTy));
-
-  Function *PollyMallocManaged = getOrCreatePollyMallocManaged(M);
-  std::string FnName = Array.getName().str();
-  FnName += ".constructor";
-  PollyIRBuilder Builder(M.getContext());
-  FunctionType *Ty = FunctionType::get(Builder.getVoidTy(), false);
-  const GlobalValue::LinkageTypes Linkage = Function::ExternalLinkage;
-  Function *F = Function::Create(Ty, Linkage, FnName, &M);
-  BasicBlock *Start = BasicBlock::Create(M.getContext(), "entry", F);
-  Builder.SetInsertPoint(Start);
-
-  const uint64_t ArraySizeInt = DL.getTypeAllocSize(ArrayTy);
-  Value *ArraySize = Builder.getInt64(ArraySizeInt);
-  ArraySize->setName("array.size");
-
-  Value *AllocatedMemRaw =
-      Builder.CreateCall(PollyMallocManaged, {ArraySize}, "mem.raw");
-  Value *AllocatedMemTyped =
-      Builder.CreatePointerCast(AllocatedMemRaw, ElemPtrTy, "mem.typed");
-  Builder.CreateStore(AllocatedMemTyped, ReplacementToArr);
-  Builder.CreateRetVoid();
-
-  const int Priority = 0;
-  appendToGlobalCtors(M, F, Priority, ReplacementToArr);
-
-  SmallVector<Instruction *, 4> ArrayUserInstructions;
-  // Get all instructions that use array. We need to do this weird thing
-  // because `Constant`s that contain this array neeed to be expanded into
-  // instructions so that we can replace their parameters. `Constant`s cannot
-  // be edited easily, so we choose to convert all `Constant`s to
-  // `Instruction`s and handle all of the uses of `Array` uniformly.
-  for (Use &ArrayUse : Array.uses())
-    getInstructionUsersOfValue(ArrayUse.getUser(), ArrayUserInstructions);
-
-  for (Instruction *UserOfArrayInst : ArrayUserInstructions) {
-
-    Builder.SetInsertPoint(UserOfArrayInst);
-    // <ty>** -> <ty>*
-    Value *ArrPtrLoaded =
-        Builder.CreateLoad(ElemPtrTy, ReplacementToArr, "arrptr.load");
-    // <ty>* -> [ty]*
-    Value *ArrPtrLoadedBitcasted = Builder.CreateBitCast(
-        ArrPtrLoaded, ArrayTy->getPointerTo(), "arrptr.bitcast");
-    rewriteOldValToNew(UserOfArrayInst, &Array, ArrPtrLoadedBitcasted, Builder);
-  }
-}
-
-// We return all `allocas` that may need to be converted to a call to
-// cudaMallocManaged.
-static void getAllocasToBeManaged(Function &F,
-                                  SmallSet<AllocaInst *, 4> &Allocas) {
-  for (BasicBlock &BB : F) {
-    for (Instruction &I : BB) {
-      auto *Alloca = dyn_cast<AllocaInst>(&I);
-      if (!Alloca)
-        continue;
-      LLVM_DEBUG(dbgs() << "Checking if (" << *Alloca << ") may be captured: ");
-
-      if (PointerMayBeCaptured(Alloca, /* ReturnCaptures */ false,
-                               /* StoreCaptures */ true)) {
-        Allocas.insert(Alloca);
-        LLVM_DEBUG(dbgs() << "YES (captured).\n");
-      } else {
-        LLVM_DEBUG(dbgs() << "NO (not captured).\n");
-      }
-    }
-  }
-}
-
-static void rewriteAllocaAsManagedMemory(AllocaInst *Alloca,
-                                         const DataLayout &DL) {
-  LLVM_DEBUG(dbgs() << "rewriting: (" << *Alloca << ") to managed mem.\n");
-  Module *M = Alloca->getModule();
-  assert(M && "Alloca does not have a module");
-
-  PollyIRBuilder Builder(M->getContext());
-  Builder.SetInsertPoint(Alloca);
-
-  Function *MallocManagedFn =
-      getOrCreatePollyMallocManaged(*Alloca->getModule());
-  const uint64_t Size = DL.getTypeAllocSize(Alloca->getAllocatedType());
-  Value *SizeVal = Builder.getInt64(Size);
-  Value *RawManagedMem = Builder.CreateCall(MallocManagedFn, {SizeVal});
-  Value *Bitcasted = Builder.CreateBitCast(RawManagedMem, Alloca->getType());
-
-  Function *F = Alloca->getFunction();
-  assert(F && "Alloca has invalid function");
-
-  Bitcasted->takeName(Alloca);
-  Alloca->replaceAllUsesWith(Bitcasted);
-  Alloca->eraseFromParent();
-
-  for (BasicBlock &BB : *F) {
-    ReturnInst *Return = dyn_cast<ReturnInst>(BB.getTerminator());
-    if (!Return)
-      continue;
-    Builder.SetInsertPoint(Return);
-
-    Function *FreeManagedFn = getOrCreatePollyFreeManaged(*M);
-    Builder.CreateCall(FreeManagedFn, {RawManagedMem});
-  }
-}
-
-// Replace all uses of `Old` with `New`, even inside `ConstantExpr`.
-//
-// `replaceAllUsesWith` does replace values in `ConstantExpr`. This function
-// actually does replace it in `ConstantExpr`. The caveat is that if there is
-// a use that is *outside* a function (say, at global declarations), we fail.
-// So, this is meant to be used on values which we know will only be used
-// within functions.
-//
-// This process works by looking through the uses of `Old`. If it finds a
-// `ConstantExpr`, it recursively looks for the owning instruction.
-// Then, it expands all the `ConstantExpr` to instructions and replaces
-// `Old` with `New` in the expanded instructions.
-static void replaceAllUsesAndConstantUses(Value *Old, Value *New,
-                                          PollyIRBuilder &Builder) {
-  SmallVector<Instruction *, 4> UserInstructions;
-  // Get all instructions that use array. We need to do this weird thing
-  // because `Constant`s that contain this array neeed to be expanded into
-  // instructions so that we can replace their parameters. `Constant`s cannot
-  // be edited easily, so we choose to convert all `Constant`s to
-  // `Instruction`s and handle all of the uses of `Array` uniformly.
-  for (Use &ArrayUse : Old->uses())
-    getInstructionUsersOfValue(ArrayUse.getUser(), UserInstructions);
-
-  for (Instruction *I : UserInstructions)
-    rewriteOldValToNew(I, Old, New, Builder);
-}
-
-class ManagedMemoryRewritePass final : public ModulePass {
-public:
-  static char ID;
-  GPUArch Architecture;
-  GPURuntime Runtime;
-
-  ManagedMemoryRewritePass() : ModulePass(ID) {}
-  bool runOnModule(Module &M) override {
-    const DataLayout &DL = M.getDataLayout();
-
-    Function *Malloc = M.getFunction("malloc");
-
-    if (Malloc) {
-      PollyIRBuilder Builder(M.getContext());
-      Function *PollyMallocManaged = getOrCreatePollyMallocManaged(M);
-      assert(PollyMallocManaged && "unable to create polly_mallocManaged");
-
-      replaceAllUsesAndConstantUses(Malloc, PollyMallocManaged, Builder);
-      Malloc->eraseFromParent();
-    }
-
-    Function *Free = M.getFunction("free");
-
-    if (Free) {
-      PollyIRBuilder Builder(M.getContext());
-      Function *PollyFreeManaged = getOrCreatePollyFreeManaged(M);
-      assert(PollyFreeManaged && "unable to create polly_freeManaged");
-
-      replaceAllUsesAndConstantUses(Free, PollyFreeManaged, Builder);
-      Free->eraseFromParent();
-    }
-
-    SmallPtrSet<GlobalVariable *, 4> GlobalsToErase;
-    for (GlobalVariable &Global : M.globals())
-      replaceGlobalArray(M, DL, Global, GlobalsToErase);
-    for (GlobalVariable *G : GlobalsToErase)
-      G->eraseFromParent();
-
-    // Rewrite allocas to cudaMallocs if we are asked to do so.
-    if (RewriteAllocas) {
-      SmallSet<AllocaInst *, 4> AllocasToBeManaged;
-      for (Function &F : M.functions())
-        getAllocasToBeManaged(F, AllocasToBeManaged);
-
-      for (AllocaInst *Alloca : AllocasToBeManaged)
-        rewriteAllocaAsManagedMemory(Alloca, DL);
-    }
-
-    return true;
-  }
-};
-} // namespace
-char ManagedMemoryRewritePass::ID = 42;
-
-Pass *polly::createManagedMemoryRewritePassPass(GPUArch Arch,
-                                                GPURuntime Runtime) {
-  ManagedMemoryRewritePass *pass = new ManagedMemoryRewritePass();
-  pass->Runtime = Runtime;
-  pass->Architecture = Arch;
-  return pass;
-}
-
-INITIALIZE_PASS_BEGIN(
-    ManagedMemoryRewritePass, "polly-acc-rewrite-managed-memory",
-    "Polly - Rewrite all allocations in heap & data section to managed memory",
-    false, false)
-INITIALIZE_PASS_DEPENDENCY(PPCGCodeGeneration);
-INITIALIZE_PASS_DEPENDENCY(DependenceInfo);
-INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass);
-INITIALIZE_PASS_DEPENDENCY(LoopInfoWrapperPass);
-INITIALIZE_PASS_DEPENDENCY(RegionInfoPass);
-INITIALIZE_PASS_DEPENDENCY(ScalarEvolutionWrapperPass);
-INITIALIZE_PASS_DEPENDENCY(ScopDetectionWrapperPass);
-INITIALIZE_PASS_END(
-    ManagedMemoryRewritePass, "polly-acc-rewrite-managed-memory",
-    "Polly - Rewrite all allocations in heap & data section to managed memory",
-    false, false)
diff --git a/polly/lib/CodeGen/PPCGCodeGeneration.cpp b/polly/lib/CodeGen/PPCGCodeGeneration.cpp
deleted file mode 100644
index 41b90bc976c2f..0000000000000
--- a/polly/lib/CodeGen/PPCGCodeGeneration.cpp
+++ /dev/null
@@ -1,3657 +0,0 @@
-//===------ PPCGCodeGeneration.cpp - Polly Accelerator Code Generation. ---===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// Take a scop created by ScopInfo and map it to GPU code using the ppcg
-// GPU mapping strategy.
-//
-//===----------------------------------------------------------------------===//
-
-#include "polly/CodeGen/PPCGCodeGeneration.h"
-#include "polly/CodeGen/CodeGeneration.h"
-#include "polly/CodeGen/IslAst.h"
-#include "polly/CodeGen/IslNodeBuilder.h"
-#include "polly/CodeGen/PerfMonitor.h"
-#include "polly/CodeGen/Utils.h"
-#include "polly/DependenceInfo.h"
-#include "polly/LinkAllPasses.h"
-#include "polly/Options.h"
-#include "polly/ScopDetection.h"
-#include "polly/ScopInfo.h"
-#include "polly/Support/ISLTools.h"
-#include "polly/Support/SCEVValidator.h"
-#include "llvm/ADT/PostOrderIterator.h"
-#include "llvm/Analysis/TargetTransformInfo.h"
-#include "llvm/IR/IntrinsicsNVPTX.h"
-#include "llvm/IR/LegacyPassManager.h"
-#include "llvm/IR/Verifier.h"
-#include "llvm/IRReader/IRReader.h"
-#include "llvm/InitializePasses.h"
-#include "llvm/Linker/Linker.h"
-#include "llvm/MC/TargetRegistry.h"
-#include "llvm/Support/SourceMgr.h"
-#include "llvm/Target/TargetMachine.h"
-#include "llvm/Transforms/IPO/PassManagerBuilder.h"
-#include "llvm/Transforms/Utils/BasicBlockUtils.h"
-#include "isl/union_map.h"
-#include <algorithm>
-
-extern "C" {
-#include "ppcg/cuda.h"
-#include "ppcg/gpu.h"
-#include "ppcg/ppcg.h"
-}
-
-#include "llvm/Support/Debug.h"
-
-using namespace polly;
-using namespace llvm;
-
-#define DEBUG_TYPE "polly-codegen-ppcg"
-
-static cl::opt<bool> DumpSchedule("polly-acc-dump-schedule",
-                                  cl::desc("Dump the computed GPU Schedule"),
-                                  cl::Hidden, cl::cat(PollyCategory));
-
-static cl::opt<bool>
-    DumpCode("polly-acc-dump-code",
-             cl::desc("Dump C code describing the GPU mapping"), cl::Hidden,
-             cl::cat(PollyCategory));
-
-static cl::opt<bool> DumpKernelIR("polly-acc-dump-kernel-ir",
-                                  cl::desc("Dump the kernel LLVM-IR"),
-                                  cl::Hidden, cl::cat(PollyCategory));
-
-static cl::opt<bool> DumpKernelASM("polly-acc-dump-kernel-asm",
-                                   cl::desc("Dump the kernel assembly code"),
-                                   cl::Hidden, cl::cat(PollyCategory));
-
-static cl::opt<bool> FastMath("polly-acc-fastmath",
-                              cl::desc("Allow unsafe math optimizations"),
-                              cl::Hidden, cl::cat(PollyCategory));
-static cl::opt<bool> SharedMemory("polly-acc-use-shared",
-                                  cl::desc("Use shared memory"), cl::Hidden,
-                                  cl::cat(PollyCategory));
-static cl::opt<bool> PrivateMemory("polly-acc-use-private",
-                                   cl::desc("Use private memory"), cl::Hidden,
-                                   cl::cat(PollyCategory));
-
-bool polly::PollyManagedMemory;
-static cl::opt<bool, true>
-    XManagedMemory("polly-acc-codegen-managed-memory",
-                   cl::desc("Generate Host kernel code assuming"
-                            " that all memory has been"
-                            " declared as managed memory"),
-                   cl::location(PollyManagedMemory), cl::Hidden,
-                   cl::init(false), cl::cat(PollyCategory));
-
-static cl::opt<bool>
-    FailOnVerifyModuleFailure("polly-acc-fail-on-verify-module-failure",
-                              cl::desc("Fail and generate a backtrace if"
-                                       " verifyModule fails on the GPU "
-                                       " kernel module."),
-                              cl::Hidden, cl::cat(PollyCategory));
-
-static cl::opt<std::string> CUDALibDevice(
-    "polly-acc-libdevice", cl::desc("Path to CUDA libdevice"), cl::Hidden,
-    cl::init("/usr/local/cuda/nvvm/libdevice/libdevice.compute_20.10.ll"),
-    cl::cat(PollyCategory));
-
-static cl::opt<std::string>
-    CudaVersion("polly-acc-cuda-version",
-                cl::desc("The CUDA version to compile for"), cl::Hidden,
-                cl::init("sm_30"), cl::cat(PollyCategory));
-
-static cl::opt<int>
-    MinCompute("polly-acc-mincompute",
-               cl::desc("Minimal number of compute statements to run on GPU."),
-               cl::Hidden, cl::init(10 * 512 * 512));
-
-GPURuntime polly::GPURuntimeChoice;
-static cl::opt<GPURuntime, true>
-    XGPURuntimeChoice("polly-gpu-runtime",
-                      cl::desc("The GPU Runtime API to target"),
-                      cl::values(clEnumValN(GPURuntime::CUDA, "libcudart",
-                                            "use the CUDA Runtime API"),
-                                 clEnumValN(GPURuntime::OpenCL, "libopencl",
-                                            "use the OpenCL Runtime API")),
-                      cl::location(polly::GPURuntimeChoice),
-                      cl::init(GPURuntime::CUDA), cl::cat(PollyCategory));
-
-GPUArch polly::GPUArchChoice;
-static cl::opt<GPUArch, true>
-    XGPUArchChoice("polly-gpu-arch", cl::desc("The GPU Architecture to target"),
-                   cl::values(clEnumValN(GPUArch::NVPTX64, "nvptx64",
-                                         "target NVIDIA 64-bit architecture"),
-                              clEnumValN(GPUArch::SPIR32, "spir32",
-                                         "target SPIR 32-bit architecture"),
-                              clEnumValN(GPUArch::SPIR64, "spir64",
-                                         "target SPIR 64-bit architecture")),
-                   cl::location(polly::GPUArchChoice),
-                   cl::init(GPUArch::NVPTX64), cl::cat(PollyCategory));
-
-extern bool polly::PerfMonitoring;
-
-/// Return  a unique name for a Scop, which is the scop region with the
-/// function name.
-std::string getUniqueScopName(const Scop *S) {
-  return "Scop Region: " + S->getNameStr() +
-         " | Function: " + std::string(S->getFunction().getName());
-}
-
-/// Used to store information PPCG wants for kills. This information is
-/// used by live range reordering.
-///
-/// @see computeLiveRangeReordering
-/// @see GPUNodeBuilder::createPPCGScop
-/// @see GPUNodeBuilder::createPPCGProg
-struct MustKillsInfo {
-  /// Collection of all kill statements that will be sequenced at the end of
-  /// PPCGScop->schedule.
-  ///
-  /// The nodes in `KillsSchedule` will be merged using `isl_schedule_set`
-  /// which merges schedules in *arbitrary* order.
-  /// (we don't care about the order of the kills anyway).
-  isl::schedule KillsSchedule;
-  /// Map from kill statement instances to scalars that need to be
-  /// killed.
-  ///
-  /// We currently derive kill information for:
-  ///  1. phi nodes. PHI nodes are not alive outside the scop and can
-  ///     consequently all be killed.
-  ///  2. Scalar arrays that are not used outside the Scop. This is
-  ///     checked by `isScalarUsesContainedInScop`.
-  /// [params] -> { [Stmt_phantom[] -> ref_phantom[]] -> scalar_to_kill[] }
-  isl::union_map TaggedMustKills;
-
-  /// Tagged must kills stripped of the tags.
-  /// [params] -> { Stmt_phantom[]  -> scalar_to_kill[] }
-  isl::union_map MustKills;
-
-  MustKillsInfo() : KillsSchedule() {}
-};
-
-/// Check if SAI's uses are entirely contained within Scop S.
-/// If a scalar is used only with a Scop, we are free to kill it, as no data
-/// can flow in/out of the value any more.
-/// @see computeMustKillsInfo
-static bool isScalarUsesContainedInScop(const Scop &S,
-                                        const ScopArrayInfo *SAI) {
-  assert(SAI->isValueKind() && "this function only deals with scalars."
-                               " Dealing with arrays required alias analysis");
-
-  const Region &R = S.getRegion();
-  for (User *U : SAI->getBasePtr()->users()) {
-    Instruction *I = dyn_cast<Instruction>(U);
-    assert(I && "invalid user of scop array info");
-    if (!R.contains(I))
-      return false;
-  }
-  return true;
-}
-
-/// Compute must-kills needed to enable live range reordering with PPCG.
-///
-/// @params S The Scop to compute live range reordering information
-/// @returns live range reordering information that can be used to setup
-/// PPCG.
-static MustKillsInfo computeMustKillsInfo(const Scop &S) {
-  const isl::space ParamSpace = S.getParamSpace();
-  MustKillsInfo Info;
-
-  // 1. Collect all ScopArrayInfo that satisfy *any* of the criteria:
-  //      1.1 phi nodes in scop.
-  //      1.2 scalars that are only used within the scop
-  SmallVector<isl::id, 4> KillMemIds;
-  for (ScopArrayInfo *SAI : S.arrays()) {
-    if (SAI->isPHIKind() ||
-        (SAI->isValueKind() && isScalarUsesContainedInScop(S, SAI)))
-      KillMemIds.push_back(isl::manage(SAI->getBasePtrId().release()));
-  }
-
-  Info.TaggedMustKills = isl::union_map::empty(ParamSpace.ctx());
-  Info.MustKills = isl::union_map::empty(ParamSpace.ctx());
-
-  // Initialising KillsSchedule to `isl_set_empty` creates an empty node in the
-  // schedule:
-  //     - filter: "[control] -> { }"
-  // So, we choose to not create this to keep the output a little nicer,
-  // at the cost of some code complexity.
-  Info.KillsSchedule = {};
-
-  for (isl::id &ToKillId : KillMemIds) {
-    isl::id KillStmtId = isl::id::alloc(
-        S.getIslCtx(),
-        std::string("SKill_phantom_").append(ToKillId.get_name()), nullptr);
-
-    // NOTE: construction of tagged_must_kill:
-    // 2. We need to construct a map:
-    //     [param] -> { [Stmt_phantom[] -> ref_phantom[]] -> scalar_to_kill[] }
-    // To construct this, we use `isl_map_domain_product` on 2 maps`:
-    // 2a. StmtToScalar:
-    //         [param] -> { Stmt_phantom[] -> scalar_to_kill[] }
-    // 2b. PhantomRefToScalar:
-    //         [param] -> { ref_phantom[] -> scalar_to_kill[] }
-    //
-    // Combining these with `isl_map_domain_product` gives us
-    // TaggedMustKill:
-    //     [param] -> { [Stmt[] -> phantom_ref[]] -> scalar_to_kill[] }
-
-    // 2a. [param] -> { Stmt[] -> scalar_to_kill[] }
-    isl::map StmtToScalar = isl::map::universe(ParamSpace);
-    StmtToScalar = StmtToScalar.set_tuple_id(isl::dim::in, isl::id(KillStmtId));
-    StmtToScalar = StmtToScalar.set_tuple_id(isl::dim::out, isl::id(ToKillId));
-
-    isl::id PhantomRefId = isl::id::alloc(
-        S.getIslCtx(), std::string("ref_phantom") + ToKillId.get_name(),
-        nullptr);
-
-    // 2b. [param] -> { phantom_ref[] -> scalar_to_kill[] }
-    isl::map PhantomRefToScalar = isl::map::universe(ParamSpace);
-    PhantomRefToScalar =
-        PhantomRefToScalar.set_tuple_id(isl::dim::in, PhantomRefId);
-    PhantomRefToScalar =
-        PhantomRefToScalar.set_tuple_id(isl::dim::out, ToKillId);
-
-    // 2. [param] -> { [Stmt[] -> phantom_ref[]] -> scalar_to_kill[] }
-    isl::map TaggedMustKill = StmtToScalar.domain_product(PhantomRefToScalar);
-    Info.TaggedMustKills = Info.TaggedMustKills.unite(TaggedMustKill);
-
-    // 2. [param] -> { Stmt[] -> scalar_to_kill[] }
-    Info.MustKills = Info.TaggedMustKills.domain_factor_domain();
-
-    // 3. Create the kill schedule of the form:
-    //     "[param] -> { Stmt_phantom[] }"
-    // Then add this to Info.KillsSchedule.
-    isl::space KillStmtSpace = ParamSpace;
-    KillStmtSpace = KillStmtSpace.set_tuple_id(isl::dim::set, KillStmtId);
-    isl::union_set KillStmtDomain = isl::set::universe(KillStmtSpace);
-
-    isl::schedule KillSchedule = isl::schedule::from_domain(KillStmtDomain);
-    if (!Info.KillsSchedule.is_null())
-      Info.KillsSchedule = isl::manage(
-          isl_schedule_set(Info.KillsSchedule.release(), KillSchedule.copy()));
-    else
-      Info.KillsSchedule = KillSchedule;
-  }
-
-  return Info;
-}
-
-/// Create the ast expressions for a ScopStmt.
-///
-/// This function is a callback for to generate the ast expressions for each
-/// of the scheduled ScopStmts.
-static __isl_give isl_id_to_ast_expr *pollyBuildAstExprForStmt(
-    void *StmtT, __isl_take isl_ast_build *Build_C,
-    isl_multi_pw_aff *(*FunctionIndex)(__isl_take isl_multi_pw_aff *MPA,
-                                       isl_id *Id, void *User),
-    void *UserIndex,
-    isl_ast_expr *(*FunctionExpr)(isl_ast_expr *Expr, isl_id *Id, void *User),
-    void *UserExpr) {
-
-  ScopStmt *Stmt = (ScopStmt *)StmtT;
-
-  if (!Stmt || !Build_C)
-    return NULL;
-
-  isl::ast_build Build = isl::manage_copy(Build_C);
-  isl::ctx Ctx = Build.ctx();
-  isl::id_to_ast_expr RefToExpr = isl::id_to_ast_expr::alloc(Ctx, 0);
-
-  Stmt->setAstBuild(Build);
-
-  for (MemoryAccess *Acc : *Stmt) {
-    isl::map AddrFunc = Acc->getAddressFunction();
-    AddrFunc = AddrFunc.intersect_domain(Stmt->getDomain());
-
-    isl::id RefId = Acc->getId();
-    isl::pw_multi_aff PMA = isl::pw_multi_aff::from_map(AddrFunc);
-
-    isl::multi_pw_aff MPA = isl::multi_pw_aff(PMA);
-    MPA = MPA.coalesce();
-    MPA = isl::manage(FunctionIndex(MPA.release(), RefId.get(), UserIndex));
-
-    isl::ast_expr Access = Build.access_from(MPA);
-    Access = isl::manage(FunctionExpr(Access.release(), RefId.get(), UserExpr));
-    RefToExpr = RefToExpr.set(RefId, Access);
-  }
-
-  return RefToExpr.release();
-}
-
-/// Given a LLVM Type, compute its size in bytes,
-static int computeSizeInBytes(const Type *T) {
-  int bytes = T->getPrimitiveSizeInBits() / 8;
-  if (bytes == 0)
-    bytes = T->getScalarSizeInBits() / 8;
-  return bytes;
-}
-
-/// Generate code for a GPU specific isl AST.
-///
-/// The GPUNodeBuilder augments the general existing IslNodeBuilder, which
-/// generates code for general-purpose AST nodes, with special functionality
-/// for generating GPU specific user nodes.
-///
-/// @see GPUNodeBuilder::createUser
-class GPUNodeBuilder final : public IslNodeBuilder {
-public:
-  GPUNodeBuilder(PollyIRBuilder &Builder, ScopAnnotator &Annotator,
-                 const DataLayout &DL, LoopInfo &LI, ScalarEvolution &SE,
-                 DominatorTree &DT, Scop &S, BasicBlock *StartBlock,
-                 gpu_prog *Prog, GPURuntime Runtime, GPUArch Arch)
-      : IslNodeBuilder(Builder, Annotator, DL, LI, SE, DT, S, StartBlock),
-        Prog(Prog), Runtime(Runtime), Arch(Arch) {
-    getExprBuilder().setIDToSAI(&IDToSAI);
-  }
-
-  /// Create after-run-time-check initialization code.
-  void initializeAfterRTH();
-
-  /// Finalize the generated scop.
-  void finalize() override;
-
-  /// Track if the full build process was successful.
-  ///
-  /// This value is set to false, if throughout the build process an error
-  /// occurred which prevents us from generating valid GPU code.
-  bool BuildSuccessful = true;
-
-  /// The maximal number of loops surrounding a sequential kernel.
-  unsigned DeepestSequential = 0;
-
-  /// The maximal number of loops surrounding a parallel kernel.
-  unsigned DeepestParallel = 0;
-
-  /// Return the name to set for the ptx_kernel.
-  std::string getKernelFuncName(int Kernel_id);
-
-private:
-  /// A vector of array base pointers for which a new ScopArrayInfo was created.
-  ///
-  /// This vector is used to delete the ScopArrayInfo when it is not needed any
-  /// more.
-  std::vector<Value *> LocalArrays;
-
-  /// A map from ScopArrays to their corresponding device allocations.
-  std::map<ScopArrayInfo *, Value *> DeviceAllocations;
-
-  /// The current GPU context.
-  Value *GPUContext;
-
-  /// The set of isl_ids allocated in the kernel
-  std::vector<isl_id *> KernelIds;
-
-  /// A module containing GPU code.
-  ///
-  /// This pointer is only set in case we are currently generating GPU code.
-  std::unique_ptr<Module> GPUModule;
-
-  /// The GPU program we generate code for.
-  gpu_prog *Prog;
-
-  /// The GPU Runtime implementation to use (OpenCL or CUDA).
-  GPURuntime Runtime;
-
-  /// The GPU Architecture to target.
-  GPUArch Arch;
-
-  /// Class to free isl_ids.
-  class IslIdDeleter final {
-  public:
-    void operator()(__isl_take isl_id *Id) { isl_id_free(Id); };
-  };
-
-  /// A set containing all isl_ids allocated in a GPU kernel.
-  ///
-  /// By releasing this set all isl_ids will be freed.
-  std::set<std::unique_ptr<isl_id, IslIdDeleter>> KernelIDs;
-
-  IslExprBuilder::IDToScopArrayInfoTy IDToSAI;
-
-  /// Create code for user-defined AST nodes.
-  ///
-  /// These AST nodes can be of type:
-  ///
-  ///   - ScopStmt:      A computational statement (TODO)
-  ///   - Kernel:        A GPU kernel call (TODO)
-  ///   - Data-Transfer: A GPU <-> CPU data-transfer
-  ///   - In-kernel synchronization
-  ///   - In-kernel memory copy statement
-  ///
-  /// @param UserStmt The ast node to generate code for.
-  void createUser(__isl_take isl_ast_node *UserStmt) override;
-
-  void createFor(__isl_take isl_ast_node *Node) override;
-
-  enum DataDirection { HOST_TO_DEVICE, DEVICE_TO_HOST };
-
-  /// Create code for a data transfer statement
-  ///
-  /// @param TransferStmt The data transfer statement.
-  /// @param Direction The direction in which to transfer data.
-  void createDataTransfer(__isl_take isl_ast_node *TransferStmt,
-                          enum DataDirection Direction);
-
-  /// Find llvm::Values referenced in GPU kernel.
-  ///
-  /// @param Kernel The kernel to scan for llvm::Values
-  ///
-  /// @returns A tuple, whose:
-  ///          - First element contains the set of values referenced by the
-  ///            kernel
-  ///          - Second element contains the set of functions referenced by the
-  ///             kernel. All functions in the set satisfy
-  ///             `isValidFunctionInKernel`.
-  ///          - Third element contains loops that have induction variables
-  ///            which are used in the kernel, *and* these loops are *neither*
-  ///            in the scop, nor do they immediately surroung the Scop.
-  ///            See [Code generation of induction variables of loops outside
-  ///            Scops]
-  std::tuple<SetVector<Value *>, SetVector<Function *>, SetVector<const Loop *>,
-             isl::space>
-  getReferencesInKernel(ppcg_kernel *Kernel);
-
-  /// Compute the sizes of the execution grid for a given kernel.
-  ///
-  /// @param Kernel The kernel to compute grid sizes for.
-  ///
-  /// @returns A tuple with grid sizes for X and Y dimension
-  std::tuple<Value *, Value *> getGridSizes(ppcg_kernel *Kernel);
-
-  /// Get the managed array pointer for sending host pointers to the device.
-  /// \note
-  /// This is to be used only with managed memory
-  Value *getManagedDeviceArray(gpu_array_info *Array, ScopArrayInfo *ArrayInfo);
-
-  /// Compute the sizes of the thread blocks for a given kernel.
-  ///
-  /// @param Kernel The kernel to compute thread block sizes for.
-  ///
-  /// @returns A tuple with thread block sizes for X, Y, and Z dimensions.
-  std::tuple<Value *, Value *, Value *> getBlockSizes(ppcg_kernel *Kernel);
-
-  /// Store a specific kernel launch parameter in the array of kernel launch
-  /// parameters.
-  ///
-  /// @param ArrayTy    Array type of \p Parameters.
-  /// @param Parameters The list of parameters in which to store.
-  /// @param Param      The kernel launch parameter to store.
-  /// @param Index      The index in the parameter list, at which to store the
-  ///                   parameter.
-  void insertStoreParameter(Type *ArrayTy, Instruction *Parameters,
-                            Instruction *Param, int Index);
-
-  /// Create kernel launch parameters.
-  ///
-  /// @param Kernel        The kernel to create parameters for.
-  /// @param F             The kernel function that has been created.
-  /// @param SubtreeValues The set of llvm::Values referenced by this kernel.
-  ///
-  /// @returns A stack allocated array with pointers to the parameter
-  ///          values that are passed to the kernel.
-  Value *createLaunchParameters(ppcg_kernel *Kernel, Function *F,
-                                SetVector<Value *> SubtreeValues);
-
-  /// Create declarations for kernel variable.
-  ///
-  /// This includes shared memory declarations.
-  ///
-  /// @param Kernel        The kernel definition to create variables for.
-  /// @param FN            The function into which to generate the variables.
-  void createKernelVariables(ppcg_kernel *Kernel, Function *FN);
-
-  /// Add CUDA annotations to module.
-  ///
-  /// Add a set of CUDA annotations that declares the maximal block dimensions
-  /// that will be used to execute the CUDA kernel. This allows the NVIDIA
-  /// PTX compiler to bound the number of allocated registers to ensure the
-  /// resulting kernel is known to run with up to as many block dimensions
-  /// as specified here.
-  ///
-  /// @param M         The module to add the annotations to.
-  /// @param BlockDimX The size of block dimension X.
-  /// @param BlockDimY The size of block dimension Y.
-  /// @param BlockDimZ The size of block dimension Z.
-  void addCUDAAnnotations(Module *M, Value *BlockDimX, Value *BlockDimY,
-                          Value *BlockDimZ);
-
-  /// Create GPU kernel.
-  ///
-  /// Code generate the kernel described by @p KernelStmt.
-  ///
-  /// @param KernelStmt The ast node to generate kernel code for.
-  void createKernel(__isl_take isl_ast_node *KernelStmt);
-
-  /// Generate code that computes the size of an array.
-  ///
-  /// @param Array The array for which to compute a size.
-  Value *getArraySize(gpu_array_info *Array);
-
-  /// Generate code to compute the minimal offset at which an array is accessed.
-  ///
-  /// The offset of an array is the minimal array location accessed in a scop.
-  ///
-  /// Example:
-  ///
-  ///   for (long i = 0; i < 100; i++)
-  ///     A[i + 42] += ...
-  ///
-  ///   getArrayOffset(A) results in 42.
-  ///
-  /// @param Array The array for which to compute the offset.
-  /// @returns An llvm::Value that contains the offset of the array.
-  Value *getArrayOffset(gpu_array_info *Array);
-
-  /// Prepare the kernel arguments for kernel code generation
-  ///
-  /// @param Kernel The kernel to generate code for.
-  /// @param FN     The function created for the kernel.
-  void prepareKernelArguments(ppcg_kernel *Kernel, Function *FN);
-
-  /// Create kernel function.
-  ///
-  /// Create a kernel function located in a newly created module that can serve
-  /// as target for device code generation. Set the Builder to point to the
-  /// start block of this newly created function.
-  ///
-  /// @param Kernel The kernel to generate code for.
-  /// @param SubtreeValues The set of llvm::Values referenced by this kernel.
-  /// @param SubtreeFunctions The set of llvm::Functions referenced by this
-  ///                         kernel.
-  void createKernelFunction(ppcg_kernel *Kernel,
-                            SetVector<Value *> &SubtreeValues,
-                            SetVector<Function *> &SubtreeFunctions);
-
-  /// Create the declaration of a kernel function.
-  ///
-  /// The kernel function takes as arguments:
-  ///
-  ///   - One i8 pointer for each external array reference used in the kernel.
-  ///   - Host iterators
-  ///   - Parameters
-  ///   - Other LLVM Value references (TODO)
-  ///
-  /// @param Kernel The kernel to generate the function declaration for.
-  /// @param SubtreeValues The set of llvm::Values referenced by this kernel.
-  ///
-  /// @returns The newly declared function.
-  Function *createKernelFunctionDecl(ppcg_kernel *Kernel,
-                                     SetVector<Value *> &SubtreeValues);
-
-  /// Insert intrinsic functions to obtain thread and block ids.
-  ///
-  /// @param The kernel to generate the intrinsic functions for.
-  void insertKernelIntrinsics(ppcg_kernel *Kernel);
-
-  /// Insert function calls to retrieve the SPIR group/local ids.
-  ///
-  /// @param Kernel The kernel to generate the function calls for.
-  /// @param SizeTypeIs64Bit Whether size_t of the openCl device is 64bit.
-  void insertKernelCallsSPIR(ppcg_kernel *Kernel, bool SizeTypeIs64bit);
-
-  /// Setup the creation of functions referenced by the GPU kernel.
-  ///
-  /// 1. Create new function declarations in GPUModule which are the same as
-  /// SubtreeFunctions.
-  ///
-  /// 2. Populate IslNodeBuilder::ValueMap with mappings from
-  /// old functions (that come from the original module) to new functions
-  /// (that are created within GPUModule). That way, we generate references
-  /// to the correct function (in GPUModule) in BlockGenerator.
-  ///
-  /// @see IslNodeBuilder::ValueMap
-  /// @see BlockGenerator::GlobalMap
-  /// @see BlockGenerator::getNewValue
-  /// @see GPUNodeBuilder::getReferencesInKernel.
-  ///
-  /// @param SubtreeFunctions The set of llvm::Functions referenced by
-  ///                         this kernel.
-  void setupKernelSubtreeFunctions(SetVector<Function *> SubtreeFunctions);
-
-  /// Create a global-to-shared or shared-to-global copy statement.
-  ///
-  /// @param CopyStmt The copy statement to generate code for
-  void createKernelCopy(ppcg_kernel_stmt *CopyStmt);
-
-  /// Create code for a ScopStmt called in @p Expr.
-  ///
-  /// @param Expr The expression containing the call.
-  /// @param KernelStmt The kernel statement referenced in the call.
-  void createScopStmt(isl_ast_expr *Expr, ppcg_kernel_stmt *KernelStmt);
-
-  /// Create an in-kernel synchronization call.
-  void createKernelSync();
-
-  /// Create a PTX assembly string for the current GPU kernel.
-  ///
-  /// @returns A string containing the corresponding PTX assembly code.
-  std::string createKernelASM();
-
-  /// Remove references from the dominator tree to the kernel function @p F.
-  ///
-  /// @param F The function to remove references to.
-  void clearDominators(Function *F);
-
-  /// Remove references from scalar evolution to the kernel function @p F.
-  ///
-  /// @param F The function to remove references to.
-  void clearScalarEvolution(Function *F);
-
-  /// Remove references from loop info to the kernel function @p F.
-  ///
-  /// @param F The function to remove references to.
-  void clearLoops(Function *F);
-
-  /// Check if the scop requires to be linked with CUDA's libdevice.
-  bool requiresCUDALibDevice();
-
-  /// Link with the NVIDIA libdevice library (if needed and available).
-  void addCUDALibDevice();
-
-  /// Finalize the generation of the kernel function.
-  ///
-  /// Free the LLVM-IR module corresponding to the kernel and -- if requested --
-  /// dump its IR to stderr.
-  ///
-  /// @returns The Assembly string of the kernel.
-  std::string finalizeKernelFunction();
-
-  /// Finalize the generation of the kernel arguments.
-  ///
-  /// This function ensures that not-read-only scalars used in a kernel are
-  /// stored back to the global memory location they are backed with before
-  /// the kernel terminates.
-  ///
-  /// @params Kernel The kernel to finalize kernel arguments for.
-  void finalizeKernelArguments(ppcg_kernel *Kernel);
-
-  /// Create code that allocates memory to store arrays on device.
-  void allocateDeviceArrays();
-
-  /// Create code to prepare the managed device pointers.
-  void prepareManagedDeviceArrays();
-
-  /// Free all allocated device arrays.
-  void freeDeviceArrays();
-
-  /// Create a call to initialize the GPU context.
-  ///
-  /// @returns A pointer to the newly initialized context.
-  Value *createCallInitContext();
-
-  /// Create a call to get the device pointer for a kernel allocation.
-  ///
-  /// @param Allocation The Polly GPU allocation
-  ///
-  /// @returns The device parameter corresponding to this allocation.
-  Value *createCallGetDevicePtr(Value *Allocation);
-
-  /// Create a call to free the GPU context.
-  ///
-  /// @param Context A pointer to an initialized GPU context.
-  void createCallFreeContext(Value *Context);
-
-  /// Create a call to allocate memory on the device.
-  ///
-  /// @param Size The size of memory to allocate
-  ///
-  /// @returns A pointer that identifies this allocation.
-  Value *createCallAllocateMemoryForDevice(Value *Size);
-
-  /// Create a call to free a device array.
-  ///
-  /// @param Array The device array to free.
-  void createCallFreeDeviceMemory(Value *Array);
-
-  /// Create a call to copy data from host to device.
-  ///
-  /// @param HostPtr A pointer to the host data that should be copied.
-  /// @param DevicePtr A device pointer specifying the location to copy to.
-  void createCallCopyFromHostToDevice(Value *HostPtr, Value *DevicePtr,
-                                      Value *Size);
-
-  /// Create a call to copy data from device to host.
-  ///
-  /// @param DevicePtr A pointer to the device data that should be copied.
-  /// @param HostPtr A host pointer specifying the location to copy to.
-  void createCallCopyFromDeviceToHost(Value *DevicePtr, Value *HostPtr,
-                                      Value *Size);
-
-  /// Create a call to synchronize Host & Device.
-  /// \note
-  /// This is to be used only with managed memory.
-  void createCallSynchronizeDevice();
-
-  /// Create a call to get a kernel from an assembly string.
-  ///
-  /// @param Buffer The string describing the kernel.
-  /// @param Entry  The name of the kernel function to call.
-  ///
-  /// @returns A pointer to a kernel object
-  Value *createCallGetKernel(Value *Buffer, Value *Entry);
-
-  /// Create a call to free a GPU kernel.
-  ///
-  /// @param GPUKernel THe kernel to free.
-  void createCallFreeKernel(Value *GPUKernel);
-
-  /// Create a call to launch a GPU kernel.
-  ///
-  /// @param GPUKernel  The kernel to launch.
-  /// @param GridDimX   The size of the first grid dimension.
-  /// @param GridDimY   The size of the second grid dimension.
-  /// @param GridBlockX The size of the first block dimension.
-  /// @param GridBlockY The size of the second block dimension.
-  /// @param GridBlockZ The size of the third block dimension.
-  /// @param Parameters A pointer to an array that contains itself pointers to
-  ///                   the parameter values passed for each kernel argument.
-  void createCallLaunchKernel(Value *GPUKernel, Value *GridDimX,
-                              Value *GridDimY, Value *BlockDimX,
-                              Value *BlockDimY, Value *BlockDimZ,
-                              Value *Parameters);
-};
-
-std::string GPUNodeBuilder::getKernelFuncName(int Kernel_id) {
-  return "FUNC_" + S.getFunction().getName().str() + "_SCOP_" +
-         std::to_string(S.getID()) + "_KERNEL_" + std::to_string(Kernel_id);
-}
-
-void GPUNodeBuilder::initializeAfterRTH() {
-  BasicBlock *NewBB = SplitBlock(Builder.GetInsertBlock(),
-                                 &*Builder.GetInsertPoint(), &DT, &LI);
-  NewBB->setName("polly.acc.initialize");
-  Builder.SetInsertPoint(&NewBB->front());
-
-  GPUContext = createCallInitContext();
-
-  if (!PollyManagedMemory)
-    allocateDeviceArrays();
-  else
-    prepareManagedDeviceArrays();
-}
-
-void GPUNodeBuilder::finalize() {
-  if (!PollyManagedMemory)
-    freeDeviceArrays();
-
-  createCallFreeContext(GPUContext);
-  IslNodeBuilder::finalize();
-}
-
-void GPUNodeBuilder::allocateDeviceArrays() {
-  assert(!PollyManagedMemory &&
-         "Managed memory will directly send host pointers "
-         "to the kernel. There is no need for device arrays");
-  isl_ast_build *Build = isl_ast_build_from_context(S.getContext().release());
-
-  for (int i = 0; i < Prog->n_array; ++i) {
-    gpu_array_info *Array = &Prog->array[i];
-    auto *ScopArray = (ScopArrayInfo *)Array->user;
-    std::string DevArrayName("p_dev_array_");
-    DevArrayName.append(Array->name);
-
-    Value *ArraySize = getArraySize(Array);
-    Value *Offset = getArrayOffset(Array);
-    if (Offset)
-      ArraySize = Builder.CreateSub(
-          ArraySize,
-          Builder.CreateMul(Offset,
-                            Builder.getInt64(ScopArray->getElemSizeInBytes())));
-    const SCEV *SizeSCEV = SE.getSCEV(ArraySize);
-    // It makes no sense to have an array of size 0. The CUDA API will
-    // throw an error anyway if we invoke `cuMallocManaged` with size `0`. We
-    // choose to be defensive and catch this at the compile phase. It is
-    // most likely that we are doing something wrong with size computation.
-    if (SizeSCEV->isZero()) {
-      errs() << getUniqueScopName(&S)
-             << " has computed array size 0: " << *ArraySize
-             << " | for array: " << *(ScopArray->getBasePtr())
-             << ". This is illegal, exiting.\n";
-      report_fatal_error("array size was computed to be 0");
-    }
-
-    Value *DevArray = createCallAllocateMemoryForDevice(ArraySize);
-    DevArray->setName(DevArrayName);
-    DeviceAllocations[ScopArray] = DevArray;
-  }
-
-  isl_ast_build_free(Build);
-}
-
-void GPUNodeBuilder::prepareManagedDeviceArrays() {
-  assert(PollyManagedMemory &&
-         "Device array most only be prepared in managed-memory mode");
-  for (int i = 0; i < Prog->n_array; ++i) {
-    gpu_array_info *Array = &Prog->array[i];
-    ScopArrayInfo *ScopArray = (ScopArrayInfo *)Array->user;
-    Value *HostPtr;
-
-    if (gpu_array_is_scalar(Array))
-      HostPtr = BlockGen.getOrCreateAlloca(ScopArray);
-    else
-      HostPtr = ScopArray->getBasePtr();
-    HostPtr = getLatestValue(HostPtr);
-
-    Value *Offset = getArrayOffset(Array);
-    if (Offset) {
-      HostPtr = Builder.CreatePointerCast(
-          HostPtr, ScopArray->getElementType()->getPointerTo());
-      HostPtr = Builder.CreateGEP(ScopArray->getElementType(), HostPtr, Offset);
-    }
-
-    HostPtr = Builder.CreatePointerCast(HostPtr, Builder.getInt8PtrTy());
-    DeviceAllocations[ScopArray] = HostPtr;
-  }
-}
-
-void GPUNodeBuilder::addCUDAAnnotations(Module *M, Value *BlockDimX,
-                                        Value *BlockDimY, Value *BlockDimZ) {
-  auto AnnotationNode = M->getOrInsertNamedMetadata("nvvm.annotations");
-
-  for (auto &F : *M) {
-    if (F.getCallingConv() != CallingConv::PTX_Kernel)
-      continue;
-
-    Value *V[] = {BlockDimX, BlockDimY, BlockDimZ};
-
-    Metadata *Elements[] = {
-        ValueAsMetadata::get(&F),   MDString::get(M->getContext(), "maxntidx"),
-        ValueAsMetadata::get(V[0]), MDString::get(M->getContext(), "maxntidy"),
-        ValueAsMetadata::get(V[1]), MDString::get(M->getContext(), "maxntidz"),
-        ValueAsMetadata::get(V[2]),
-    };
-    MDNode *Node = MDNode::get(M->getContext(), Elements);
-    AnnotationNode->addOperand(Node);
-  }
-}
-
-void GPUNodeBuilder::freeDeviceArrays() {
-  assert(!PollyManagedMemory && "Managed memory does not use device arrays");
-  for (auto &Array : DeviceAllocations)
-    createCallFreeDeviceMemory(Array.second);
-}
-
-Value *GPUNodeBuilder::createCallGetKernel(Value *Buffer, Value *Entry) {
-  const char *Name = "polly_getKernel";
-  Module *M = Builder.GetInsertBlock()->getParent()->getParent();
-  Function *F = M->getFunction(Name);
-
-  // If F is not available, declare it.
-  if (!F) {
-    GlobalValue::LinkageTypes Linkage = Function::ExternalLinkage;
-    std::vector<Type *> Args;
-    Args.push_back(Builder.getInt8PtrTy());
-    Args.push_back(Builder.getInt8PtrTy());
-    FunctionType *Ty = FunctionType::get(Builder.getInt8PtrTy(), Args, false);
-    F = Function::Create(Ty, Linkage, Name, M);
-  }
-
-  return Builder.CreateCall(F, {Buffer, Entry});
-}
-
-Value *GPUNodeBuilder::createCallGetDevicePtr(Value *Allocation) {
-  const char *Name = "polly_getDevicePtr";
-  Module *M = Builder.GetInsertBlock()->getParent()->getParent();
-  Function *F = M->getFunction(Name);
-
-  // If F is not available, declare it.
-  if (!F) {
-    GlobalValue::LinkageTypes Linkage = Function::ExternalLinkage;
-    std::vector<Type *> Args;
-    Args.push_back(Builder.getInt8PtrTy());
-    FunctionType *Ty = FunctionType::get(Builder.getInt8PtrTy(), Args, false);
-    F = Function::Create(Ty, Linkage, Name, M);
-  }
-
-  return Builder.CreateCall(F, {Allocation});
-}
-
-void GPUNodeBuilder::createCallLaunchKernel(Value *GPUKernel, Value *GridDimX,
-                                            Value *GridDimY, Value *BlockDimX,
-                                            Value *BlockDimY, Value *BlockDimZ,
-                                            Value *Parameters) {
-  const char *Name = "polly_launchKernel";
-  Module *M = Builder.GetInsertBlock()->getParent()->getParent();
-  Function *F = M->getFunction(Name);
-
-  // If F is not available, declare it.
-  if (!F) {
-    GlobalValue::LinkageTypes Linkage = Function::ExternalLinkage;
-    std::vector<Type *> Args;
-    Args.push_back(Builder.getInt8PtrTy());
-    Args.push_back(Builder.getInt32Ty());
-    Args.push_back(Builder.getInt32Ty());
-    Args.push_back(Builder.getInt32Ty());
-    Args.push_back(Builder.getInt32Ty());
-    Args.push_back(Builder.getInt32Ty());
-    Args.push_back(Builder.getInt8PtrTy());
-    FunctionType *Ty = FunctionType::get(Builder.getVoidTy(), Args, false);
-    F = Function::Create(Ty, Linkage, Name, M);
-  }
-
-  Builder.CreateCall(F, {GPUKernel, GridDimX, GridDimY, BlockDimX, BlockDimY,
-                         BlockDimZ, Parameters});
-}
-
-void GPUNodeBuilder::createCallFreeKernel(Value *GPUKernel) {
-  const char *Name = "polly_freeKernel";
-  Module *M = Builder.GetInsertBlock()->getParent()->getParent();
-  Function *F = M->getFunction(Name);
-
-  // If F is not available, declare it.
-  if (!F) {
-    GlobalValue::LinkageTypes Linkage = Function::ExternalLinkage;
-    std::vector<Type *> Args;
-    Args.push_back(Builder.getInt8PtrTy());
-    FunctionType *Ty = FunctionType::get(Builder.getVoidTy(), Args, false);
-    F = Function::Create(Ty, Linkage, Name, M);
-  }
-
-  Builder.CreateCall(F, {GPUKernel});
-}
-
-void GPUNodeBuilder::createCallFreeDeviceMemory(Value *Array) {
-  assert(!PollyManagedMemory &&
-         "Managed memory does not allocate or free memory "
-         "for device");
-  const char *Name = "polly_freeDeviceMemory";
-  Module *M = Builder.GetInsertBlock()->getParent()->getParent();
-  Function *F = M->getFunction(Name);
-
-  // If F is not available, declare it.
-  if (!F) {
-    GlobalValue::LinkageTypes Linkage = Function::ExternalLinkage;
-    std::vector<Type *> Args;
-    Args.push_back(Builder.getInt8PtrTy());
-    FunctionType *Ty = FunctionType::get(Builder.getVoidTy(), Args, false);
-    F = Function::Create(Ty, Linkage, Name, M);
-  }
-
-  Builder.CreateCall(F, {Array});
-}
-
-Value *GPUNodeBuilder::createCallAllocateMemoryForDevice(Value *Size) {
-  assert(!PollyManagedMemory &&
-         "Managed memory does not allocate or free memory "
-         "for device");
-  const char *Name = "polly_allocateMemoryForDevice";
-  Module *M = Builder.GetInsertBlock()->getParent()->getParent();
-  Function *F = M->getFunction(Name);
-
-  // If F is not available, declare it.
-  if (!F) {
-    GlobalValue::LinkageTypes Linkage = Function::ExternalLinkage;
-    std::vector<Type *> Args;
-    Args.push_back(Builder.getInt64Ty());
-    FunctionType *Ty = FunctionType::get(Builder.getInt8PtrTy(), Args, false);
-    F = Function::Create(Ty, Linkage, Name, M);
-  }
-
-  return Builder.CreateCall(F, {Size});
-}
-
-void GPUNodeBuilder::createCallCopyFromHostToDevice(Value *HostData,
-                                                    Value *DeviceData,
-                                                    Value *Size) {
-  assert(!PollyManagedMemory &&
-         "Managed memory does not transfer memory between "
-         "device and host");
-  const char *Name = "polly_copyFromHostToDevice";
-  Module *M = Builder.GetInsertBlock()->getParent()->getParent();
-  Function *F = M->getFunction(Name);
-
-  // If F is not available, declare it.
-  if (!F) {
-    GlobalValue::LinkageTypes Linkage = Function::ExternalLinkage;
-    std::vector<Type *> Args;
-    Args.push_back(Builder.getInt8PtrTy());
-    Args.push_back(Builder.getInt8PtrTy());
-    Args.push_back(Builder.getInt64Ty());
-    FunctionType *Ty = FunctionType::get(Builder.getVoidTy(), Args, false);
-    F = Function::Create(Ty, Linkage, Name, M);
-  }
-
-  Builder.CreateCall(F, {HostData, DeviceData, Size});
-}
-
-void GPUNodeBuilder::createCallCopyFromDeviceToHost(Value *DeviceData,
-                                                    Value *HostData,
-                                                    Value *Size) {
-  assert(!PollyManagedMemory &&
-         "Managed memory does not transfer memory between "
-         "device and host");
-  const char *Name = "polly_copyFromDeviceToHost";
-  Module *M = Builder.GetInsertBlock()->getParent()->getParent();
-  Function *F = M->getFunction(Name);
-
-  // If F is not available, declare it.
-  if (!F) {
-    GlobalValue::LinkageTypes Linkage = Function::ExternalLinkage;
-    std::vector<Type *> Args;
-    Args.push_back(Builder.getInt8PtrTy());
-    Args.push_back(Builder.getInt8PtrTy());
-    Args.push_back(Builder.getInt64Ty());
-    FunctionType *Ty = FunctionType::get(Builder.getVoidTy(), Args, false);
-    F = Function::Create(Ty, Linkage, Name, M);
-  }
-
-  Builder.CreateCall(F, {DeviceData, HostData, Size});
-}
-
-void GPUNodeBuilder::createCallSynchronizeDevice() {
-  assert(PollyManagedMemory && "explicit synchronization is only necessary for "
-                               "managed memory");
-  const char *Name = "polly_synchronizeDevice";
-  Module *M = Builder.GetInsertBlock()->getParent()->getParent();
-  Function *F = M->getFunction(Name);
-
-  // If F is not available, declare it.
-  if (!F) {
-    GlobalValue::LinkageTypes Linkage = Function::ExternalLinkage;
-    FunctionType *Ty = FunctionType::get(Builder.getVoidTy(), false);
-    F = Function::Create(Ty, Linkage, Name, M);
-  }
-
-  Builder.CreateCall(F);
-}
-
-Value *GPUNodeBuilder::createCallInitContext() {
-  const char *Name;
-
-  switch (Runtime) {
-  case GPURuntime::CUDA:
-    Name = "polly_initContextCUDA";
-    break;
-  case GPURuntime::OpenCL:
-    Name = "polly_initContextCL";
-    break;
-  }
-
-  Module *M = Builder.GetInsertBlock()->getParent()->getParent();
-  Function *F = M->getFunction(Name);
-
-  // If F is not available, declare it.
-  if (!F) {
-    GlobalValue::LinkageTypes Linkage = Function::ExternalLinkage;
-    std::vector<Type *> Args;
-    FunctionType *Ty = FunctionType::get(Builder.getInt8PtrTy(), Args, false);
-    F = Function::Create(Ty, Linkage, Name, M);
-  }
-
-  return Builder.CreateCall(F, {});
-}
-
-void GPUNodeBuilder::createCallFreeContext(Value *Context) {
-  const char *Name = "polly_freeContext";
-  Module *M = Builder.GetInsertBlock()->getParent()->getParent();
-  Function *F = M->getFunction(Name);
-
-  // If F is not available, declare it.
-  if (!F) {
-    GlobalValue::LinkageTypes Linkage = Function::ExternalLinkage;
-    std::vector<Type *> Args;
-    Args.push_back(Builder.getInt8PtrTy());
-    FunctionType *Ty = FunctionType::get(Builder.getVoidTy(), Args, false);
-    F = Function::Create(Ty, Linkage, Name, M);
-  }
-
-  Builder.CreateCall(F, {Context});
-}
-
-/// Check if one string is a prefix of another.
-///
-/// @param String The string in which to look for the prefix.
-/// @param Prefix The prefix to look for.
-static bool isPrefix(std::string String, std::string Prefix) {
-  return String.find(Prefix) == 0;
-}
-
-Value *GPUNodeBuilder::getArraySize(gpu_array_info *Array) {
-  isl::ast_build Build = isl::ast_build::from_context(S.getContext());
-  Value *ArraySize = ConstantInt::get(Builder.getInt64Ty(), Array->size);
-
-  if (!gpu_array_is_scalar(Array)) {
-    isl::multi_pw_aff ArrayBound = isl::manage_copy(Array->bound);
-
-    isl::pw_aff OffsetDimZero = ArrayBound.at(0);
-    isl::ast_expr Res = Build.expr_from(OffsetDimZero);
-
-    for (unsigned int i = 1; i < Array->n_index; i++) {
-      isl::pw_aff Bound_I = ArrayBound.at(i);
-      isl::ast_expr Expr = Build.expr_from(Bound_I);
-      Res = Res.mul(Expr);
-    }
-
-    Value *NumElements = ExprBuilder.create(Res.release());
-    if (NumElements->getType() != ArraySize->getType())
-      NumElements = Builder.CreateSExt(NumElements, ArraySize->getType());
-    ArraySize = Builder.CreateMul(ArraySize, NumElements);
-  }
-  return ArraySize;
-}
-
-Value *GPUNodeBuilder::getArrayOffset(gpu_array_info *Array) {
-  if (gpu_array_is_scalar(Array))
-    return nullptr;
-
-  isl::ast_build Build = isl::ast_build::from_context(S.getContext());
-
-  isl::set Min = isl::manage_copy(Array->extent).lexmin();
-
-  isl::set ZeroSet = isl::set::universe(Min.get_space());
-
-  for (unsigned i : rangeIslSize(0, Min.tuple_dim()))
-    ZeroSet = ZeroSet.fix_si(isl::dim::set, i, 0);
-
-  if (Min.is_subset(ZeroSet)) {
-    return nullptr;
-  }
-
-  isl::ast_expr Result = isl::ast_expr::from_val(isl::val(Min.ctx(), 0));
-
-  for (unsigned i : rangeIslSize(0, Min.tuple_dim())) {
-    if (i > 0) {
-      isl::pw_aff Bound_I =
-          isl::manage(isl_multi_pw_aff_get_pw_aff(Array->bound, i - 1));
-      isl::ast_expr BExpr = Build.expr_from(Bound_I);
-      Result = Result.mul(BExpr);
-    }
-    isl::pw_aff DimMin = Min.dim_min(i);
-    isl::ast_expr MExpr = Build.expr_from(DimMin);
-    Result = Result.add(MExpr);
-  }
-
-  return ExprBuilder.create(Result.release());
-}
-
-Value *GPUNodeBuilder::getManagedDeviceArray(gpu_array_info *Array,
-                                             ScopArrayInfo *ArrayInfo) {
-  assert(PollyManagedMemory && "Only used when you wish to get a host "
-                               "pointer for sending data to the kernel, "
-                               "with managed memory");
-  std::map<ScopArrayInfo *, Value *>::iterator it;
-  it = DeviceAllocations.find(ArrayInfo);
-  assert(it != DeviceAllocations.end() &&
-         "Device array expected to be available");
-  return it->second;
-}
-
-void GPUNodeBuilder::createDataTransfer(__isl_take isl_ast_node *TransferStmt,
-                                        enum DataDirection Direction) {
-  assert(!PollyManagedMemory && "Managed memory needs no data transfers");
-  isl_ast_expr *Expr = isl_ast_node_user_get_expr(TransferStmt);
-  isl_ast_expr *Arg = isl_ast_expr_get_op_arg(Expr, 0);
-  isl_id *Id = isl_ast_expr_get_id(Arg);
-  auto Array = (gpu_array_info *)isl_id_get_user(Id);
-  auto ScopArray = (ScopArrayInfo *)(Array->user);
-
-  Value *Size = getArraySize(Array);
-  Value *Offset = getArrayOffset(Array);
-  Value *DevPtr = DeviceAllocations[ScopArray];
-
-  Value *HostPtr;
-
-  if (gpu_array_is_scalar(Array))
-    HostPtr = BlockGen.getOrCreateAlloca(ScopArray);
-  else
-    HostPtr = ScopArray->getBasePtr();
-  HostPtr = getLatestValue(HostPtr);
-
-  if (Offset) {
-    HostPtr = Builder.CreatePointerCast(
-        HostPtr, ScopArray->getElementType()->getPointerTo());
-    HostPtr = Builder.CreateGEP(ScopArray->getElementType(), HostPtr, Offset);
-  }
-
-  HostPtr = Builder.CreatePointerCast(HostPtr, Builder.getInt8PtrTy());
-
-  if (Offset) {
-    Size = Builder.CreateSub(
-        Size, Builder.CreateMul(
-                  Offset, Builder.getInt64(ScopArray->getElemSizeInBytes())));
-  }
-
-  if (Direction == HOST_TO_DEVICE)
-    createCallCopyFromHostToDevice(HostPtr, DevPtr, Size);
-  else
-    createCallCopyFromDeviceToHost(DevPtr, HostPtr, Size);
-
-  isl_id_free(Id);
-  isl_ast_expr_free(Arg);
-  isl_ast_expr_free(Expr);
-  isl_ast_node_free(TransferStmt);
-}
-
-void GPUNodeBuilder::createUser(__isl_take isl_ast_node *UserStmt) {
-  isl_ast_expr *Expr = isl_ast_node_user_get_expr(UserStmt);
-  isl_ast_expr *StmtExpr = isl_ast_expr_get_op_arg(Expr, 0);
-  isl_id *Id = isl_ast_expr_get_id(StmtExpr);
-  isl_id_free(Id);
-  isl_ast_expr_free(StmtExpr);
-
-  const char *Str = isl_id_get_name(Id);
-  if (!strcmp(Str, "kernel")) {
-    createKernel(UserStmt);
-    if (PollyManagedMemory)
-      createCallSynchronizeDevice();
-    isl_ast_expr_free(Expr);
-    return;
-  }
-  if (!strcmp(Str, "init_device")) {
-    initializeAfterRTH();
-    isl_ast_node_free(UserStmt);
-    isl_ast_expr_free(Expr);
-    return;
-  }
-  if (!strcmp(Str, "clear_device")) {
-    finalize();
-    isl_ast_node_free(UserStmt);
-    isl_ast_expr_free(Expr);
-    return;
-  }
-  if (isPrefix(Str, "to_device")) {
-    if (!PollyManagedMemory)
-      createDataTransfer(UserStmt, HOST_TO_DEVICE);
-    else
-      isl_ast_node_free(UserStmt);
-
-    isl_ast_expr_free(Expr);
-    return;
-  }
-
-  if (isPrefix(Str, "from_device")) {
-    if (!PollyManagedMemory) {
-      createDataTransfer(UserStmt, DEVICE_TO_HOST);
-    } else {
-      isl_ast_node_free(UserStmt);
-    }
-    isl_ast_expr_free(Expr);
-    return;
-  }
-
-  isl_id *Anno = isl_ast_node_get_annotation(UserStmt);
-  struct ppcg_kernel_stmt *KernelStmt =
-      (struct ppcg_kernel_stmt *)isl_id_get_user(Anno);
-  isl_id_free(Anno);
-
-  switch (KernelStmt->type) {
-  case ppcg_kernel_domain:
-    createScopStmt(Expr, KernelStmt);
-    isl_ast_node_free(UserStmt);
-    return;
-  case ppcg_kernel_copy:
-    createKernelCopy(KernelStmt);
-    isl_ast_expr_free(Expr);
-    isl_ast_node_free(UserStmt);
-    return;
-  case ppcg_kernel_sync:
-    createKernelSync();
-    isl_ast_expr_free(Expr);
-    isl_ast_node_free(UserStmt);
-    return;
-  }
-
-  isl_ast_expr_free(Expr);
-  isl_ast_node_free(UserStmt);
-}
-
-void GPUNodeBuilder::createFor(__isl_take isl_ast_node *Node) {
-  createForSequential(isl::manage(Node).as<isl::ast_node_for>(), false);
-}
-
-void GPUNodeBuilder::createKernelCopy(ppcg_kernel_stmt *KernelStmt) {
-  isl_ast_expr *LocalIndex = isl_ast_expr_copy(KernelStmt->u.c.local_index);
-  auto LocalAddr = ExprBuilder.createAccessAddress(LocalIndex);
-  isl_ast_expr *Index = isl_ast_expr_copy(KernelStmt->u.c.index);
-  auto GlobalAddr = ExprBuilder.createAccessAddress(Index);
-
-  if (KernelStmt->u.c.read) {
-    LoadInst *Load =
-        Builder.CreateLoad(GlobalAddr.second, GlobalAddr.first, "shared.read");
-    Builder.CreateStore(Load, LocalAddr.first);
-  } else {
-    LoadInst *Load =
-        Builder.CreateLoad(LocalAddr.second, LocalAddr.first, "shared.write");
-    Builder.CreateStore(Load, GlobalAddr.first);
-  }
-}
-
-void GPUNodeBuilder::createScopStmt(isl_ast_expr *Expr,
-                                    ppcg_kernel_stmt *KernelStmt) {
-  auto Stmt = (ScopStmt *)KernelStmt->u.d.stmt->stmt;
-  isl_id_to_ast_expr *Indexes = KernelStmt->u.d.ref2expr;
-
-  LoopToScevMapT LTS;
-  LTS.insert(OutsideLoopIterations.begin(), OutsideLoopIterations.end());
-
-  createSubstitutions(Expr, Stmt, LTS);
-
-  if (Stmt->isBlockStmt())
-    BlockGen.copyStmt(*Stmt, LTS, Indexes);
-  else
-    RegionGen.copyStmt(*Stmt, LTS, Indexes);
-}
-
-void GPUNodeBuilder::createKernelSync() {
-  Module *M = Builder.GetInsertBlock()->getParent()->getParent();
-  const char *SpirName = "__gen_ocl_barrier_global";
-
-  Function *Sync;
-
-  switch (Arch) {
-  case GPUArch::SPIR64:
-  case GPUArch::SPIR32:
-    Sync = M->getFunction(SpirName);
-
-    // If Sync is not available, declare it.
-    if (!Sync) {
-      GlobalValue::LinkageTypes Linkage = Function::ExternalLinkage;
-      std::vector<Type *> Args;
-      FunctionType *Ty = FunctionType::get(Builder.getVoidTy(), Args, false);
-      Sync = Function::Create(Ty, Linkage, SpirName, M);
-      Sync->setCallingConv(CallingConv::SPIR_FUNC);
-    }
-    break;
-  case GPUArch::NVPTX64:
-    Sync = Intrinsic::getDeclaration(M, Intrinsic::nvvm_barrier0);
-    break;
-  }
-
-  Builder.CreateCall(Sync, {});
-}
-
-/// Collect llvm::Values referenced from @p Node
-///
-/// This function only applies to isl_ast_nodes that are user_nodes referring
-/// to a ScopStmt. All other node types are ignore.
-///
-/// @param Node The node to collect references for.
-/// @param User A user pointer used as storage for the data that is collected.
-///
-/// @returns isl_bool_true if data could be collected successfully.
-isl_bool collectReferencesInGPUStmt(__isl_keep isl_ast_node *Node, void *User) {
-  if (isl_ast_node_get_type(Node) != isl_ast_node_user)
-    return isl_bool_true;
-
-  isl_ast_expr *Expr = isl_ast_node_user_get_expr(Node);
-  isl_ast_expr *StmtExpr = isl_ast_expr_get_op_arg(Expr, 0);
-  isl_id *Id = isl_ast_expr_get_id(StmtExpr);
-  const char *Str = isl_id_get_name(Id);
-  isl_id_free(Id);
-  isl_ast_expr_free(StmtExpr);
-  isl_ast_expr_free(Expr);
-
-  if (!isPrefix(Str, "Stmt"))
-    return isl_bool_true;
-
-  Id = isl_ast_node_get_annotation(Node);
-  auto *KernelStmt = (ppcg_kernel_stmt *)isl_id_get_user(Id);
-  auto Stmt = (ScopStmt *)KernelStmt->u.d.stmt->stmt;
-  isl_id_free(Id);
-
-  addReferencesFromStmt(Stmt, User, false /* CreateScalarRefs */);
-
-  return isl_bool_true;
-}
-
-/// A list of functions that are available in NVIDIA's libdevice.
-const std::set<std::string> CUDALibDeviceFunctions = {
-    "exp",      "expf",      "expl",      "cos", "cosf", "sqrt", "sqrtf",
-    "copysign", "copysignf", "copysignl", "log", "logf", "powi", "powif"};
-
-// A map from intrinsics to their corresponding libdevice functions.
-const std::map<std::string, std::string> IntrinsicToLibdeviceFunc = {
-    {"llvm.exp.f64", "exp"},
-    {"llvm.exp.f32", "expf"},
-    {"llvm.powi.f64.i32", "powi"},
-    {"llvm.powi.f32.i32", "powif"}};
-
-/// Return the corresponding CUDA libdevice function name @p Name.
-/// Note that this function will try to convert instrinsics in the list
-/// IntrinsicToLibdeviceFunc into libdevice functions.
-/// This is because some intrinsics such as `exp`
-/// are not supported by the NVPTX backend.
-/// If this restriction of the backend is lifted, we should refactor our code
-/// so that we use intrinsics whenever possible.
-///
-/// Return "" if we are not compiling for CUDA.
-std::string getCUDALibDeviceFuntion(StringRef NameRef) {
-  std::string Name = NameRef.str();
-  auto It = IntrinsicToLibdeviceFunc.find(Name);
-  if (It != IntrinsicToLibdeviceFunc.end())
-    return getCUDALibDeviceFuntion(It->second);
-
-  if (CUDALibDeviceFunctions.count(Name))
-    return ("__nv_" + Name);
-
-  return "";
-}
-
-/// Check if F is a function that we can code-generate in a GPU kernel.
-static bool isValidFunctionInKernel(llvm::Function *F, bool AllowLibDevice) {
-  assert(F && "F is an invalid pointer");
-  // We string compare against the name of the function to allow
-  // all variants of the intrinsic "llvm.sqrt.*", "llvm.fabs", and
-  // "llvm.copysign".
-  const StringRef Name = F->getName();
-
-  if (AllowLibDevice && getCUDALibDeviceFuntion(Name).length() > 0)
-    return true;
-
-  return F->isIntrinsic() &&
-         (Name.startswith("llvm.sqrt") || Name.startswith("llvm.fabs") ||
-          Name.startswith("llvm.copysign"));
-}
-
-/// Do not take `Function` as a subtree value.
-///
-/// We try to take the reference of all subtree values and pass them along
-/// to the kernel from the host. Taking an address of any function and
-/// trying to pass along is nonsensical. Only allow `Value`s that are not
-/// `Function`s.
-static bool isValidSubtreeValue(llvm::Value *V) { return !isa<Function>(V); }
-
-/// Return `Function`s from `RawSubtreeValues`.
-static SetVector<Function *>
-getFunctionsFromRawSubtreeValues(SetVector<Value *> RawSubtreeValues,
-                                 bool AllowCUDALibDevice) {
-  SetVector<Function *> SubtreeFunctions;
-  for (Value *It : RawSubtreeValues) {
-    Function *F = dyn_cast<Function>(It);
-    if (F) {
-      assert(isValidFunctionInKernel(F, AllowCUDALibDevice) &&
-             "Code should have bailed out by "
-             "this point if an invalid function "
-             "were present in a kernel.");
-      SubtreeFunctions.insert(F);
-    }
-  }
-  return SubtreeFunctions;
-}
-
-std::tuple<SetVector<Value *>, SetVector<Function *>, SetVector<const Loop *>,
-           isl::space>
-GPUNodeBuilder::getReferencesInKernel(ppcg_kernel *Kernel) {
-  SetVector<Value *> SubtreeValues;
-  SetVector<const SCEV *> SCEVs;
-  SetVector<const Loop *> Loops;
-  isl::space ParamSpace = isl::space(S.getIslCtx(), 0, 0).params();
-  SubtreeReferences References = {
-      LI,         SE, S, ValueMap, SubtreeValues, SCEVs, getBlockGenerator(),
-      &ParamSpace};
-
-  for (const auto &I : IDToValue)
-    SubtreeValues.insert(I.second);
-
-  // NOTE: this is populated in IslNodeBuilder::addParameters
-  // See [Code generation of induction variables of loops outside Scops].
-  for (const auto &I : OutsideLoopIterations)
-    SubtreeValues.insert(cast<SCEVUnknown>(I.second)->getValue());
-
-  isl_ast_node_foreach_descendant_top_down(
-      Kernel->tree, collectReferencesInGPUStmt, &References);
-
-  for (const SCEV *Expr : SCEVs) {
-    findValues(Expr, SE, SubtreeValues);
-    findLoops(Expr, Loops);
-  }
-
-  Loops.remove_if([this](const Loop *L) {
-    return S.contains(L) || L->contains(S.getEntry());
-  });
-
-  for (auto &SAI : S.arrays())
-    SubtreeValues.remove(SAI->getBasePtr());
-
-  isl_space *Space = S.getParamSpace().release();
-  for (long i = 0, n = isl_space_dim(Space, isl_dim_param); i < n; i++) {
-    isl_id *Id = isl_space_get_dim_id(Space, isl_dim_param, i);
-    assert(IDToValue.count(Id));
-    Value *Val = IDToValue[Id];
-    SubtreeValues.remove(Val);
-    isl_id_free(Id);
-  }
-  isl_space_free(Space);
-
-  for (long i = 0, n = isl_space_dim(Kernel->space, isl_dim_set); i < n; i++) {
-    isl_id *Id = isl_space_get_dim_id(Kernel->space, isl_dim_set, i);
-    assert(IDToValue.count(Id));
-    Value *Val = IDToValue[Id];
-    SubtreeValues.remove(Val);
-    isl_id_free(Id);
-  }
-
-  // Note: { ValidSubtreeValues, ValidSubtreeFunctions } partitions
-  // SubtreeValues. This is important, because we should not lose any
-  // SubtreeValues in the process of constructing the
-  // "ValidSubtree{Values, Functions} sets. Nor should the set
-  // ValidSubtree{Values, Functions} have any common element.
-  auto ValidSubtreeValuesIt =
-      make_filter_range(SubtreeValues, isValidSubtreeValue);
-  SetVector<Value *> ValidSubtreeValues(ValidSubtreeValuesIt.begin(),
-                                        ValidSubtreeValuesIt.end());
-
-  bool AllowCUDALibDevice = Arch == GPUArch::NVPTX64;
-
-  SetVector<Function *> ValidSubtreeFunctions(
-      getFunctionsFromRawSubtreeValues(SubtreeValues, AllowCUDALibDevice));
-
-  // @see IslNodeBuilder::getReferencesInSubtree
-  SetVector<Value *> ReplacedValues;
-  for (Value *V : ValidSubtreeValues) {
-    auto It = ValueMap.find(V);
-    if (It == ValueMap.end())
-      ReplacedValues.insert(V);
-    else
-      ReplacedValues.insert(It->second);
-  }
-  return std::make_tuple(ReplacedValues, ValidSubtreeFunctions, Loops,
-                         ParamSpace);
-}
-
-void GPUNodeBuilder::clearDominators(Function *F) {
-  DomTreeNode *N = DT.getNode(&F->getEntryBlock());
-  std::vector<BasicBlock *> Nodes;
-  for (po_iterator<DomTreeNode *> I = po_begin(N), E = po_end(N); I != E; ++I)
-    Nodes.push_back(I->getBlock());
-
-  for (BasicBlock *BB : Nodes)
-    DT.eraseNode(BB);
-}
-
-void GPUNodeBuilder::clearScalarEvolution(Function *F) {
-  for (BasicBlock &BB : *F) {
-    Loop *L = LI.getLoopFor(&BB);
-    if (L)
-      SE.forgetLoop(L);
-  }
-}
-
-void GPUNodeBuilder::clearLoops(Function *F) {
-  SmallSet<Loop *, 1> WorkList;
-  for (BasicBlock &BB : *F) {
-    Loop *L = LI.getLoopFor(&BB);
-    if (L)
-      WorkList.insert(L);
-  }
-  for (auto *L : WorkList)
-    LI.erase(L);
-}
-
-std::tuple<Value *, Value *> GPUNodeBuilder::getGridSizes(ppcg_kernel *Kernel) {
-  std::vector<Value *> Sizes;
-  isl::ast_build Context = isl::ast_build::from_context(S.getContext());
-
-  isl::multi_pw_aff GridSizePwAffs = isl::manage_copy(Kernel->grid_size);
-  for (long i = 0; i < Kernel->n_grid; i++) {
-    isl::pw_aff Size = GridSizePwAffs.at(i);
-    isl::ast_expr GridSize = Context.expr_from(Size);
-    Value *Res = ExprBuilder.create(GridSize.release());
-    Res = Builder.CreateTrunc(Res, Builder.getInt32Ty());
-    Sizes.push_back(Res);
-  }
-
-  for (long i = Kernel->n_grid; i < 3; i++)
-    Sizes.push_back(ConstantInt::get(Builder.getInt32Ty(), 1));
-
-  return std::make_tuple(Sizes[0], Sizes[1]);
-}
-
-std::tuple<Value *, Value *, Value *>
-GPUNodeBuilder::getBlockSizes(ppcg_kernel *Kernel) {
-  std::vector<Value *> Sizes;
-
-  for (long i = 0; i < Kernel->n_block; i++) {
-    Value *Res = ConstantInt::get(Builder.getInt32Ty(), Kernel->block_dim[i]);
-    Sizes.push_back(Res);
-  }
-
-  for (long i = Kernel->n_block; i < 3; i++)
-    Sizes.push_back(ConstantInt::get(Builder.getInt32Ty(), 1));
-
-  return std::make_tuple(Sizes[0], Sizes[1], Sizes[2]);
-}
-
-void GPUNodeBuilder::insertStoreParameter(Type *ArrayTy,
-                                          Instruction *Parameters,
-                                          Instruction *Param, int Index) {
-  Value *Slot = Builder.CreateGEP(
-      ArrayTy, Parameters, {Builder.getInt64(0), Builder.getInt64(Index)});
-  Value *ParamTyped = Builder.CreatePointerCast(Param, Builder.getInt8PtrTy());
-  Builder.CreateStore(ParamTyped, Slot);
-}
-
-Value *
-GPUNodeBuilder::createLaunchParameters(ppcg_kernel *Kernel, Function *F,
-                                       SetVector<Value *> SubtreeValues) {
-  const int NumArgs = F->arg_size();
-  std::vector<int> ArgSizes(NumArgs);
-
-  // If we are using the OpenCL Runtime, we need to add the kernel argument
-  // sizes to the end of the launch-parameter list, so OpenCL can determine
-  // how big the respective kernel arguments are.
-  // Here we need to reserve adequate space for that.
-  Type *ArrayTy;
-  if (Runtime == GPURuntime::OpenCL)
-    ArrayTy = ArrayType::get(Builder.getInt8PtrTy(), 2 * NumArgs);
-  else
-    ArrayTy = ArrayType::get(Builder.getInt8PtrTy(), NumArgs);
-
-  BasicBlock *EntryBlock =
-      &Builder.GetInsertBlock()->getParent()->getEntryBlock();
-  auto AddressSpace = F->getParent()->getDataLayout().getAllocaAddrSpace();
-  std::string Launch = "polly_launch_" + std::to_string(Kernel->id);
-  Instruction *Parameters = new AllocaInst(
-      ArrayTy, AddressSpace, Launch + "_params", EntryBlock->getTerminator());
-
-  int Index = 0;
-  for (long i = 0; i < Prog->n_array; i++) {
-    if (!ppcg_kernel_requires_array_argument(Kernel, i))
-      continue;
-
-    isl_id *Id = isl_space_get_tuple_id(Prog->array[i].space, isl_dim_set);
-    const ScopArrayInfo *SAI = ScopArrayInfo::getFromId(isl::manage(Id));
-
-    if (Runtime == GPURuntime::OpenCL)
-      ArgSizes[Index] = SAI->getElemSizeInBytes();
-
-    Value *DevArray = nullptr;
-    if (PollyManagedMemory) {
-      DevArray = getManagedDeviceArray(&Prog->array[i],
-                                       const_cast<ScopArrayInfo *>(SAI));
-    } else {
-      DevArray = DeviceAllocations[const_cast<ScopArrayInfo *>(SAI)];
-      DevArray = createCallGetDevicePtr(DevArray);
-    }
-    assert(DevArray != nullptr && "Array to be offloaded to device not "
-                                  "initialized");
-    Value *Offset = getArrayOffset(&Prog->array[i]);
-
-    if (Offset) {
-      DevArray = Builder.CreatePointerCast(
-          DevArray, SAI->getElementType()->getPointerTo());
-      DevArray = Builder.CreateGEP(SAI->getElementType(), DevArray,
-                                   Builder.CreateNeg(Offset));
-      DevArray = Builder.CreatePointerCast(DevArray, Builder.getInt8PtrTy());
-    }
-    Value *Slot = Builder.CreateGEP(
-        ArrayTy, Parameters, {Builder.getInt64(0), Builder.getInt64(Index)});
-
-    if (gpu_array_is_read_only_scalar(&Prog->array[i])) {
-      Value *ValPtr = nullptr;
-      if (PollyManagedMemory)
-        ValPtr = DevArray;
-      else
-        ValPtr = BlockGen.getOrCreateAlloca(SAI);
-
-      assert(ValPtr != nullptr && "ValPtr that should point to a valid object"
-                                  " to be stored into Parameters");
-      Value *ValPtrCast =
-          Builder.CreatePointerCast(ValPtr, Builder.getInt8PtrTy());
-      Builder.CreateStore(ValPtrCast, Slot);
-    } else {
-      Instruction *Param =
-          new AllocaInst(Builder.getInt8PtrTy(), AddressSpace,
-                         Launch + "_param_" + std::to_string(Index),
-                         EntryBlock->getTerminator());
-      Builder.CreateStore(DevArray, Param);
-      Value *ParamTyped =
-          Builder.CreatePointerCast(Param, Builder.getInt8PtrTy());
-      Builder.CreateStore(ParamTyped, Slot);
-    }
-    Index++;
-  }
-
-  int NumHostIters = isl_space_dim(Kernel->space, isl_dim_set);
-
-  for (long i = 0; i < NumHostIters; i++) {
-    isl_id *Id = isl_space_get_dim_id(Kernel->space, isl_dim_set, i);
-    Value *Val = IDToValue[Id];
-    isl_id_free(Id);
-
-    if (Runtime == GPURuntime::OpenCL)
-      ArgSizes[Index] = computeSizeInBytes(Val->getType());
-
-    Instruction *Param =
-        new AllocaInst(Val->getType(), AddressSpace,
-                       Launch + "_param_" + std::to_string(Index),
-                       EntryBlock->getTerminator());
-    Builder.CreateStore(Val, Param);
-    insertStoreParameter(ArrayTy, Parameters, Param, Index);
-    Index++;
-  }
-
-  int NumVars = isl_space_dim(Kernel->space, isl_dim_param);
-
-  for (long i = 0; i < NumVars; i++) {
-    isl_id *Id = isl_space_get_dim_id(Kernel->space, isl_dim_param, i);
-    Value *Val = IDToValue[Id];
-    if (ValueMap.count(Val))
-      Val = ValueMap[Val];
-    isl_id_free(Id);
-
-    if (Runtime == GPURuntime::OpenCL)
-      ArgSizes[Index] = computeSizeInBytes(Val->getType());
-
-    Instruction *Param =
-        new AllocaInst(Val->getType(), AddressSpace,
-                       Launch + "_param_" + std::to_string(Index),
-                       EntryBlock->getTerminator());
-    Builder.CreateStore(Val, Param);
-    insertStoreParameter(ArrayTy, Parameters, Param, Index);
-    Index++;
-  }
-
-  for (auto Val : SubtreeValues) {
-    if (Runtime == GPURuntime::OpenCL)
-      ArgSizes[Index] = computeSizeInBytes(Val->getType());
-
-    Instruction *Param =
-        new AllocaInst(Val->getType(), AddressSpace,
-                       Launch + "_param_" + std::to_string(Index),
-                       EntryBlock->getTerminator());
-    Builder.CreateStore(Val, Param);
-    insertStoreParameter(ArrayTy, Parameters, Param, Index);
-    Index++;
-  }
-
-  if (Runtime == GPURuntime::OpenCL) {
-    for (int i = 0; i < NumArgs; i++) {
-      Value *Val = ConstantInt::get(Builder.getInt32Ty(), ArgSizes[i]);
-      Instruction *Param =
-          new AllocaInst(Builder.getInt32Ty(), AddressSpace,
-                         Launch + "_param_size_" + std::to_string(i),
-                         EntryBlock->getTerminator());
-      Builder.CreateStore(Val, Param);
-      insertStoreParameter(ArrayTy, Parameters, Param, Index);
-      Index++;
-    }
-  }
-
-  auto Location = EntryBlock->getTerminator();
-  return new BitCastInst(Parameters, Builder.getInt8PtrTy(),
-                         Launch + "_params_i8ptr", Location);
-}
-
-void GPUNodeBuilder::setupKernelSubtreeFunctions(
-    SetVector<Function *> SubtreeFunctions) {
-  for (auto Fn : SubtreeFunctions) {
-    const std::string ClonedFnName = Fn->getName().str();
-    Function *Clone = GPUModule->getFunction(ClonedFnName);
-    if (!Clone)
-      Clone =
-          Function::Create(Fn->getFunctionType(), GlobalValue::ExternalLinkage,
-                           ClonedFnName, GPUModule.get());
-    assert(Clone && "Expected cloned function to be initialized.");
-    assert(ValueMap.find(Fn) == ValueMap.end() &&
-           "Fn already present in ValueMap");
-    ValueMap[Fn] = Clone;
-  }
-}
-void GPUNodeBuilder::createKernel(__isl_take isl_ast_node *KernelStmt) {
-  isl_id *Id = isl_ast_node_get_annotation(KernelStmt);
-  ppcg_kernel *Kernel = (ppcg_kernel *)isl_id_get_user(Id);
-  isl_id_free(Id);
-  isl_ast_node_free(KernelStmt);
-
-  if (Kernel->n_grid > 1)
-    DeepestParallel = std::max(
-        DeepestParallel, (unsigned)isl_space_dim(Kernel->space, isl_dim_set));
-  else
-    DeepestSequential = std::max(
-        DeepestSequential, (unsigned)isl_space_dim(Kernel->space, isl_dim_set));
-
-  Value *BlockDimX, *BlockDimY, *BlockDimZ;
-  std::tie(BlockDimX, BlockDimY, BlockDimZ) = getBlockSizes(Kernel);
-
-  SetVector<Value *> SubtreeValues;
-  SetVector<Function *> SubtreeFunctions;
-  SetVector<const Loop *> Loops;
-  isl::space ParamSpace;
-  std::tie(SubtreeValues, SubtreeFunctions, Loops, ParamSpace) =
-      getReferencesInKernel(Kernel);
-
-  // Add parameters that appear only in the access function to the kernel
-  // space. This is important to make sure that all isl_ids are passed as
-  // parameters to the kernel, even though we may not have all parameters
-  // in the context to improve compile time.
-  Kernel->space = isl_space_align_params(Kernel->space, ParamSpace.release());
-
-  assert(Kernel->tree && "Device AST of kernel node is empty");
-
-  Instruction &HostInsertPoint = *Builder.GetInsertPoint();
-  IslExprBuilder::IDToValueTy HostIDs = IDToValue;
-  ValueMapT HostValueMap = ValueMap;
-  BlockGenerator::AllocaMapTy HostScalarMap = ScalarMap;
-  ScalarMap.clear();
-  BlockGenerator::EscapeUsersAllocaMapTy HostEscapeMap = EscapeMap;
-  EscapeMap.clear();
-
-  // Create for all loops we depend on values that contain the current loop
-  // iteration. These values are necessary to generate code for SCEVs that
-  // depend on such loops. As a result we need to pass them to the subfunction.
-  for (const Loop *L : Loops) {
-    const SCEV *OuterLIV = SE.getAddRecExpr(SE.getUnknown(Builder.getInt64(0)),
-                                            SE.getUnknown(Builder.getInt64(1)),
-                                            L, SCEV::FlagAnyWrap);
-    Value *V = generateSCEV(OuterLIV);
-    OutsideLoopIterations[L] = SE.getUnknown(V);
-    SubtreeValues.insert(V);
-  }
-
-  createKernelFunction(Kernel, SubtreeValues, SubtreeFunctions);
-  setupKernelSubtreeFunctions(SubtreeFunctions);
-
-  create(isl_ast_node_copy(Kernel->tree));
-
-  finalizeKernelArguments(Kernel);
-  Function *F = Builder.GetInsertBlock()->getParent();
-  if (Arch == GPUArch::NVPTX64)
-    addCUDAAnnotations(F->getParent(), BlockDimX, BlockDimY, BlockDimZ);
-  clearDominators(F);
-  clearScalarEvolution(F);
-  clearLoops(F);
-
-  IDToValue = HostIDs;
-
-  ValueMap = std::move(HostValueMap);
-  ScalarMap = std::move(HostScalarMap);
-  EscapeMap = std::move(HostEscapeMap);
-  IDToSAI.clear();
-  Annotator.resetAlternativeAliasBases();
-  for (auto &BasePtr : LocalArrays)
-    S.invalidateScopArrayInfo(BasePtr, MemoryKind::Array);
-  LocalArrays.clear();
-
-  std::string ASMString = finalizeKernelFunction();
-  Builder.SetInsertPoint(&HostInsertPoint);
-  Value *Parameters = createLaunchParameters(Kernel, F, SubtreeValues);
-
-  std::string Name = getKernelFuncName(Kernel->id);
-  Value *KernelString = Builder.CreateGlobalStringPtr(ASMString, Name);
-  Value *NameString = Builder.CreateGlobalStringPtr(Name, Name + "_name");
-  Value *GPUKernel = createCallGetKernel(KernelString, NameString);
-
-  Value *GridDimX, *GridDimY;
-  std::tie(GridDimX, GridDimY) = getGridSizes(Kernel);
-
-  createCallLaunchKernel(GPUKernel, GridDimX, GridDimY, BlockDimX, BlockDimY,
-                         BlockDimZ, Parameters);
-  createCallFreeKernel(GPUKernel);
-
-  for (auto Id : KernelIds)
-    isl_id_free(Id);
-
-  KernelIds.clear();
-}
-
-/// Compute the DataLayout string for the NVPTX backend.
-///
-/// @param is64Bit Are we looking for a 64 bit architecture?
-static std::string computeNVPTXDataLayout(bool is64Bit) {
-  std::string Ret = "";
-
-  if (!is64Bit) {
-    Ret += "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:"
-           "64-i128:128:128-f32:32:32-f64:64:64-v16:16:16-v32:32:32-v64:64:"
-           "64-v128:128:128-n16:32:64";
-  } else {
-    Ret += "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:"
-           "64-i128:128:128-f32:32:32-f64:64:64-v16:16:16-v32:32:32-v64:64:"
-           "64-v128:128:128-n16:32:64";
-  }
-
-  return Ret;
-}
-
-/// Compute the DataLayout string for a SPIR kernel.
-///
-/// @param is64Bit Are we looking for a 64 bit architecture?
-static std::string computeSPIRDataLayout(bool is64Bit) {
-  std::string Ret = "";
-
-  if (!is64Bit) {
-    Ret += "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:"
-           "64-i128:128:128-f32:32:32-f64:64:64-v16:16:16-v24:32:32-v32:32:"
-           "32-v48:64:64-v64:64:64-v96:128:128-v128:128:128-v192:"
-           "256:256-v256:256:256-v512:512:512-v1024:1024:1024";
-  } else {
-    Ret += "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:"
-           "64-i128:128:128-f32:32:32-f64:64:64-v16:16:16-v24:32:32-v32:32:"
-           "32-v48:64:64-v64:64:64-v96:128:128-v128:128:128-v192:"
-           "256:256-v256:256:256-v512:512:512-v1024:1024:1024";
-  }
-
-  return Ret;
-}
-
-Function *
-GPUNodeBuilder::createKernelFunctionDecl(ppcg_kernel *Kernel,
-                                         SetVector<Value *> &SubtreeValues) {
-  std::vector<Type *> Args;
-  std::string Identifier = getKernelFuncName(Kernel->id);
-
-  std::vector<Metadata *> MemoryType;
-
-  for (long i = 0; i < Prog->n_array; i++) {
-    if (!ppcg_kernel_requires_array_argument(Kernel, i))
-      continue;
-
-    if (gpu_array_is_read_only_scalar(&Prog->array[i])) {
-      isl_id *Id = isl_space_get_tuple_id(Prog->array[i].space, isl_dim_set);
-      const ScopArrayInfo *SAI = ScopArrayInfo::getFromId(isl::manage(Id));
-      Args.push_back(SAI->getElementType());
-      MemoryType.push_back(
-          ConstantAsMetadata::get(ConstantInt::get(Builder.getInt32Ty(), 0)));
-    } else {
-      static const int UseGlobalMemory = 1;
-      Args.push_back(Builder.getInt8PtrTy(UseGlobalMemory));
-      MemoryType.push_back(
-          ConstantAsMetadata::get(ConstantInt::get(Builder.getInt32Ty(), 1)));
-    }
-  }
-
-  int NumHostIters = isl_space_dim(Kernel->space, isl_dim_set);
-
-  for (long i = 0; i < NumHostIters; i++) {
-    Args.push_back(Builder.getInt64Ty());
-    MemoryType.push_back(
-        ConstantAsMetadata::get(ConstantInt::get(Builder.getInt32Ty(), 0)));
-  }
-
-  int NumVars = isl_space_dim(Kernel->space, isl_dim_param);
-
-  for (long i = 0; i < NumVars; i++) {
-    isl_id *Id = isl_space_get_dim_id(Kernel->space, isl_dim_param, i);
-    Value *Val = IDToValue[Id];
-    isl_id_free(Id);
-    Args.push_back(Val->getType());
-    MemoryType.push_back(
-        ConstantAsMetadata::get(ConstantInt::get(Builder.getInt32Ty(), 0)));
-  }
-
-  for (auto *V : SubtreeValues) {
-    Args.push_back(V->getType());
-    MemoryType.push_back(
-        ConstantAsMetadata::get(ConstantInt::get(Builder.getInt32Ty(), 0)));
-  }
-
-  auto *FT = FunctionType::get(Builder.getVoidTy(), Args, false);
-  auto *FN = Function::Create(FT, Function::ExternalLinkage, Identifier,
-                              GPUModule.get());
-
-  std::vector<Metadata *> EmptyStrings;
-
-  for (unsigned int i = 0; i < MemoryType.size(); i++) {
-    EmptyStrings.push_back(MDString::get(FN->getContext(), ""));
-  }
-
-  if (Arch == GPUArch::SPIR32 || Arch == GPUArch::SPIR64) {
-    FN->setMetadata("kernel_arg_addr_space",
-                    MDNode::get(FN->getContext(), MemoryType));
-    FN->setMetadata("kernel_arg_name",
-                    MDNode::get(FN->getContext(), EmptyStrings));
-    FN->setMetadata("kernel_arg_access_qual",
-                    MDNode::get(FN->getContext(), EmptyStrings));
-    FN->setMetadata("kernel_arg_type",
-                    MDNode::get(FN->getContext(), EmptyStrings));
-    FN->setMetadata("kernel_arg_type_qual",
-                    MDNode::get(FN->getContext(), EmptyStrings));
-    FN->setMetadata("kernel_arg_base_type",
-                    MDNode::get(FN->getContext(), EmptyStrings));
-  }
-
-  switch (Arch) {
-  case GPUArch::NVPTX64:
-    FN->setCallingConv(CallingConv::PTX_Kernel);
-    break;
-  case GPUArch::SPIR32:
-  case GPUArch::SPIR64:
-    FN->setCallingConv(CallingConv::SPIR_KERNEL);
-    break;
-  }
-
-  auto Arg = FN->arg_begin();
-  for (long i = 0; i < Kernel->n_array; i++) {
-    if (!ppcg_kernel_requires_array_argument(Kernel, i))
-      continue;
-
-    Arg->setName(Kernel->array[i].array->name);
-
-    isl_id *Id = isl_space_get_tuple_id(Prog->array[i].space, isl_dim_set);
-    const ScopArrayInfo *SAI = ScopArrayInfo::getFromId(isl::manage_copy(Id));
-    Type *EleTy = SAI->getElementType();
-    Value *Val = &*Arg;
-    SmallVector<const SCEV *, 4> Sizes;
-    isl_ast_build *Build =
-        isl_ast_build_from_context(isl_set_copy(Prog->context));
-    Sizes.push_back(nullptr);
-    for (long j = 1, n = Kernel->array[i].array->n_index; j < n; j++) {
-      isl_ast_expr *DimSize = isl_ast_build_expr_from_pw_aff(
-          Build, isl_multi_pw_aff_get_pw_aff(Kernel->array[i].array->bound, j));
-      auto V = ExprBuilder.create(DimSize);
-      Sizes.push_back(SE.getSCEV(V));
-    }
-    const ScopArrayInfo *SAIRep =
-        S.getOrCreateScopArrayInfo(Val, EleTy, Sizes, MemoryKind::Array);
-    LocalArrays.push_back(Val);
-
-    isl_ast_build_free(Build);
-    KernelIds.push_back(Id);
-    IDToSAI[Id] = SAIRep;
-    Arg++;
-  }
-
-  for (long i = 0; i < NumHostIters; i++) {
-    isl_id *Id = isl_space_get_dim_id(Kernel->space, isl_dim_set, i);
-    Arg->setName(isl_id_get_name(Id));
-    IDToValue[Id] = &*Arg;
-    KernelIDs.insert(std::unique_ptr<isl_id, IslIdDeleter>(Id));
-    Arg++;
-  }
-
-  for (long i = 0; i < NumVars; i++) {
-    isl_id *Id = isl_space_get_dim_id(Kernel->space, isl_dim_param, i);
-    Arg->setName(isl_id_get_name(Id));
-    Value *Val = IDToValue[Id];
-    ValueMap[Val] = &*Arg;
-    IDToValue[Id] = &*Arg;
-    KernelIDs.insert(std::unique_ptr<isl_id, IslIdDeleter>(Id));
-    Arg++;
-  }
-
-  for (auto *V : SubtreeValues) {
-    Arg->setName(V->getName());
-    ValueMap[V] = &*Arg;
-    Arg++;
-  }
-
-  return FN;
-}
-
-void GPUNodeBuilder::insertKernelIntrinsics(ppcg_kernel *Kernel) {
-  Intrinsic::ID IntrinsicsBID[2];
-  Intrinsic::ID IntrinsicsTID[3];
-
-  switch (Arch) {
-  case GPUArch::SPIR64:
-  case GPUArch::SPIR32:
-    llvm_unreachable("Cannot generate NVVM intrinsics for SPIR");
-  case GPUArch::NVPTX64:
-    IntrinsicsBID[0] = Intrinsic::nvvm_read_ptx_sreg_ctaid_x;
-    IntrinsicsBID[1] = Intrinsic::nvvm_read_ptx_sreg_ctaid_y;
-
-    IntrinsicsTID[0] = Intrinsic::nvvm_read_ptx_sreg_tid_x;
-    IntrinsicsTID[1] = Intrinsic::nvvm_read_ptx_sreg_tid_y;
-    IntrinsicsTID[2] = Intrinsic::nvvm_read_ptx_sreg_tid_z;
-    break;
-  }
-
-  auto addId = [this](__isl_take isl_id *Id, Intrinsic::ID Intr) mutable {
-    std::string Name = isl_id_get_name(Id);
-    Module *M = Builder.GetInsertBlock()->getParent()->getParent();
-    Function *IntrinsicFn = Intrinsic::getDeclaration(M, Intr);
-    Value *Val = Builder.CreateCall(IntrinsicFn, {});
-    Val = Builder.CreateIntCast(Val, Builder.getInt64Ty(), false, Name);
-    IDToValue[Id] = Val;
-    KernelIDs.insert(std::unique_ptr<isl_id, IslIdDeleter>(Id));
-  };
-
-  for (int i = 0; i < Kernel->n_grid; ++i) {
-    isl_id *Id = isl_id_list_get_id(Kernel->block_ids, i);
-    addId(Id, IntrinsicsBID[i]);
-  }
-
-  for (int i = 0; i < Kernel->n_block; ++i) {
-    isl_id *Id = isl_id_list_get_id(Kernel->thread_ids, i);
-    addId(Id, IntrinsicsTID[i]);
-  }
-}
-
-void GPUNodeBuilder::insertKernelCallsSPIR(ppcg_kernel *Kernel,
-                                           bool SizeTypeIs64bit) {
-  const char *GroupName[3] = {"__gen_ocl_get_group_id0",
-                              "__gen_ocl_get_group_id1",
-                              "__gen_ocl_get_group_id2"};
-
-  const char *LocalName[3] = {"__gen_ocl_get_local_id0",
-                              "__gen_ocl_get_local_id1",
-                              "__gen_ocl_get_local_id2"};
-  IntegerType *SizeT =
-      SizeTypeIs64bit ? Builder.getInt64Ty() : Builder.getInt32Ty();
-
-  auto createFunc = [this](const char *Name, __isl_take isl_id *Id,
-                           IntegerType *SizeT) mutable {
-    Module *M = Builder.GetInsertBlock()->getParent()->getParent();
-    Function *FN = M->getFunction(Name);
-
-    // If FN is not available, declare it.
-    if (!FN) {
-      GlobalValue::LinkageTypes Linkage = Function::ExternalLinkage;
-      std::vector<Type *> Args;
-      FunctionType *Ty = FunctionType::get(SizeT, Args, false);
-      FN = Function::Create(Ty, Linkage, Name, M);
-      FN->setCallingConv(CallingConv::SPIR_FUNC);
-    }
-
-    Value *Val = Builder.CreateCall(FN, {});
-    if (SizeT == Builder.getInt32Ty())
-      Val = Builder.CreateIntCast(Val, Builder.getInt64Ty(), false, Name);
-    IDToValue[Id] = Val;
-    KernelIDs.insert(std::unique_ptr<isl_id, IslIdDeleter>(Id));
-  };
-
-  for (int i = 0; i < Kernel->n_grid; ++i)
-    createFunc(GroupName[i], isl_id_list_get_id(Kernel->block_ids, i), SizeT);
-
-  for (int i = 0; i < Kernel->n_block; ++i)
-    createFunc(LocalName[i], isl_id_list_get_id(Kernel->thread_ids, i), SizeT);
-}
-
-void GPUNodeBuilder::prepareKernelArguments(ppcg_kernel *Kernel, Function *FN) {
-  auto Arg = FN->arg_begin();
-  for (long i = 0; i < Kernel->n_array; i++) {
-    if (!ppcg_kernel_requires_array_argument(Kernel, i))
-      continue;
-
-    isl_id *Id = isl_space_get_tuple_id(Prog->array[i].space, isl_dim_set);
-    const ScopArrayInfo *SAI = ScopArrayInfo::getFromId(isl::manage_copy(Id));
-    isl_id_free(Id);
-
-    if (SAI->getNumberOfDimensions() > 0) {
-      Arg++;
-      continue;
-    }
-
-    Value *Val = &*Arg;
-
-    if (!gpu_array_is_read_only_scalar(&Prog->array[i])) {
-      Type *TypePtr = SAI->getElementType()->getPointerTo();
-      Value *TypedArgPtr = Builder.CreatePointerCast(Val, TypePtr);
-      Val = Builder.CreateLoad(SAI->getElementType(), TypedArgPtr);
-    }
-
-    Value *Alloca = BlockGen.getOrCreateAlloca(SAI);
-    Builder.CreateStore(Val, Alloca);
-
-    Arg++;
-  }
-}
-
-void GPUNodeBuilder::finalizeKernelArguments(ppcg_kernel *Kernel) {
-  auto *FN = Builder.GetInsertBlock()->getParent();
-  auto Arg = FN->arg_begin();
-
-  bool StoredScalar = false;
-  for (long i = 0; i < Kernel->n_array; i++) {
-    if (!ppcg_kernel_requires_array_argument(Kernel, i))
-      continue;
-
-    isl_id *Id = isl_space_get_tuple_id(Prog->array[i].space, isl_dim_set);
-    const ScopArrayInfo *SAI = ScopArrayInfo::getFromId(isl::manage_copy(Id));
-    isl_id_free(Id);
-
-    if (SAI->getNumberOfDimensions() > 0) {
-      Arg++;
-      continue;
-    }
-
-    if (gpu_array_is_read_only_scalar(&Prog->array[i])) {
-      Arg++;
-      continue;
-    }
-
-    Value *Alloca = BlockGen.getOrCreateAlloca(SAI);
-    Value *ArgPtr = &*Arg;
-    Type *TypePtr = SAI->getElementType()->getPointerTo();
-    Value *TypedArgPtr = Builder.CreatePointerCast(ArgPtr, TypePtr);
-    Value *Val = Builder.CreateLoad(SAI->getElementType(), Alloca);
-    Builder.CreateStore(Val, TypedArgPtr);
-    StoredScalar = true;
-
-    Arg++;
-  }
-
-  if (StoredScalar) {
-    /// In case more than one thread contains scalar stores, the generated
-    /// code might be incorrect, if we only store at the end of the kernel.
-    /// To support this case we need to store these scalars back at each
-    /// memory store or at least before each kernel barrier.
-    if (Kernel->n_block != 0 || Kernel->n_grid != 0) {
-      BuildSuccessful = 0;
-      LLVM_DEBUG(
-          dbgs() << getUniqueScopName(&S)
-                 << " has a store to a scalar value that"
-                    " would be undefined to run in parallel. Bailing out.\n";);
-    }
-  }
-}
-
-void GPUNodeBuilder::createKernelVariables(ppcg_kernel *Kernel, Function *FN) {
-  Module *M = Builder.GetInsertBlock()->getParent()->getParent();
-
-  for (int i = 0; i < Kernel->n_var; ++i) {
-    struct ppcg_kernel_var &Var = Kernel->var[i];
-    isl_id *Id = isl_space_get_tuple_id(Var.array->space, isl_dim_set);
-    Type *EleTy = ScopArrayInfo::getFromId(isl::manage(Id))->getElementType();
-
-    Type *ArrayTy = EleTy;
-    SmallVector<const SCEV *, 4> Sizes;
-
-    Sizes.push_back(nullptr);
-    for (unsigned int j = 1; j < Var.array->n_index; ++j) {
-      isl_val *Val = isl_vec_get_element_val(Var.size, j);
-      long Bound = isl_val_get_num_si(Val);
-      isl_val_free(Val);
-      Sizes.push_back(S.getSE()->getConstant(Builder.getInt64Ty(), Bound));
-    }
-
-    for (int j = Var.array->n_index - 1; j >= 0; --j) {
-      isl_val *Val = isl_vec_get_element_val(Var.size, j);
-      long Bound = isl_val_get_num_si(Val);
-      isl_val_free(Val);
-      ArrayTy = ArrayType::get(ArrayTy, Bound);
-    }
-
-    const ScopArrayInfo *SAI;
-    Value *Allocation;
-    if (Var.type == ppcg_access_shared) {
-      auto GlobalVar = new GlobalVariable(
-          *M, ArrayTy, false, GlobalValue::InternalLinkage, 0, Var.name,
-          nullptr, GlobalValue::ThreadLocalMode::NotThreadLocal, 3);
-      GlobalVar->setAlignment(llvm::Align(EleTy->getPrimitiveSizeInBits() / 8));
-      GlobalVar->setInitializer(Constant::getNullValue(ArrayTy));
-
-      Allocation = GlobalVar;
-    } else if (Var.type == ppcg_access_private) {
-      Allocation = Builder.CreateAlloca(ArrayTy, 0, "private_array");
-    } else {
-      llvm_unreachable("unknown variable type");
-    }
-    SAI =
-        S.getOrCreateScopArrayInfo(Allocation, EleTy, Sizes, MemoryKind::Array);
-    Id = isl_id_alloc(S.getIslCtx().get(), Var.name, nullptr);
-    IDToValue[Id] = Allocation;
-    LocalArrays.push_back(Allocation);
-    KernelIds.push_back(Id);
-    IDToSAI[Id] = SAI;
-  }
-}
-
-void GPUNodeBuilder::createKernelFunction(
-    ppcg_kernel *Kernel, SetVector<Value *> &SubtreeValues,
-    SetVector<Function *> &SubtreeFunctions) {
-  std::string Identifier = getKernelFuncName(Kernel->id);
-  GPUModule.reset(new Module(Identifier, Builder.getContext()));
-
-  switch (Arch) {
-  case GPUArch::NVPTX64:
-    if (Runtime == GPURuntime::CUDA)
-      GPUModule->setTargetTriple(Triple::normalize("nvptx64-nvidia-cuda"));
-    else if (Runtime == GPURuntime::OpenCL)
-      GPUModule->setTargetTriple(Triple::normalize("nvptx64-nvidia-nvcl"));
-    GPUModule->setDataLayout(computeNVPTXDataLayout(true /* is64Bit */));
-    break;
-  case GPUArch::SPIR32:
-    GPUModule->setTargetTriple(Triple::normalize("spir-unknown-unknown"));
-    GPUModule->setDataLayout(computeSPIRDataLayout(false /* is64Bit */));
-    break;
-  case GPUArch::SPIR64:
-    GPUModule->setTargetTriple(Triple::normalize("spir64-unknown-unknown"));
-    GPUModule->setDataLayout(computeSPIRDataLayout(true /* is64Bit */));
-    break;
-  }
-
-  Function *FN = createKernelFunctionDecl(Kernel, SubtreeValues);
-
-  BasicBlock *PrevBlock = Builder.GetInsertBlock();
-  auto EntryBlock = BasicBlock::Create(Builder.getContext(), "entry", FN);
-
-  DT.addNewBlock(EntryBlock, PrevBlock);
-
-  Builder.SetInsertPoint(EntryBlock);
-  Builder.CreateRetVoid();
-  Builder.SetInsertPoint(EntryBlock, EntryBlock->begin());
-
-  ScopDetection::markFunctionAsInvalid(FN);
-
-  prepareKernelArguments(Kernel, FN);
-  createKernelVariables(Kernel, FN);
-
-  switch (Arch) {
-  case GPUArch::NVPTX64:
-    insertKernelIntrinsics(Kernel);
-    break;
-  case GPUArch::SPIR32:
-    insertKernelCallsSPIR(Kernel, false);
-    break;
-  case GPUArch::SPIR64:
-    insertKernelCallsSPIR(Kernel, true);
-    break;
-  }
-}
-
-std::string GPUNodeBuilder::createKernelASM() {
-  llvm::Triple GPUTriple;
-
-  switch (Arch) {
-  case GPUArch::NVPTX64:
-    switch (Runtime) {
-    case GPURuntime::CUDA:
-      GPUTriple = llvm::Triple(Triple::normalize("nvptx64-nvidia-cuda"));
-      break;
-    case GPURuntime::OpenCL:
-      GPUTriple = llvm::Triple(Triple::normalize("nvptx64-nvidia-nvcl"));
-      break;
-    }
-    break;
-  case GPUArch::SPIR64:
-  case GPUArch::SPIR32:
-    std::string SPIRAssembly;
-    raw_string_ostream IROstream(SPIRAssembly);
-    IROstream << *GPUModule;
-    IROstream.flush();
-    return SPIRAssembly;
-  }
-
-  std::string ErrMsg;
-  auto GPUTarget = TargetRegistry::lookupTarget(GPUTriple.getTriple(), ErrMsg);
-
-  if (!GPUTarget) {
-    errs() << ErrMsg << "\n";
-    return "";
-  }
-
-  TargetOptions Options;
-  Options.UnsafeFPMath = FastMath;
-
-  std::string subtarget;
-
-  switch (Arch) {
-  case GPUArch::NVPTX64:
-    subtarget = CudaVersion;
-    break;
-  case GPUArch::SPIR32:
-  case GPUArch::SPIR64:
-    llvm_unreachable("No subtarget for SPIR architecture");
-  }
-
-  std::unique_ptr<TargetMachine> TargetM(GPUTarget->createTargetMachine(
-      GPUTriple.getTriple(), subtarget, "", Options, std::nullopt));
-
-  SmallString<0> ASMString;
-  raw_svector_ostream ASMStream(ASMString);
-  llvm::legacy::PassManager PM;
-
-  PM.add(createTargetTransformInfoWrapperPass(TargetM->getTargetIRAnalysis()));
-
-  if (TargetM->addPassesToEmitFile(PM, ASMStream, nullptr, CGFT_AssemblyFile,
-                                   true /* verify */)) {
-    errs() << "The target does not support generation of this file type!\n";
-    return "";
-  }
-
-  PM.run(*GPUModule);
-
-  return ASMStream.str().str();
-}
-
-bool GPUNodeBuilder::requiresCUDALibDevice() {
-  bool RequiresLibDevice = false;
-  for (Function &F : GPUModule->functions()) {
-    if (!F.isDeclaration())
-      continue;
-
-    const std::string CUDALibDeviceFunc = getCUDALibDeviceFuntion(F.getName());
-    if (CUDALibDeviceFunc.length() != 0) {
-      // We need to handle the case where a module looks like this:
-      // @expf(..)
-      // @llvm.exp.f64(..)
-      // Both of these functions would be renamed to `__nv_expf`.
-      //
-      // So, we must first check for the existence of the libdevice function.
-      // If this exists, we replace our current function with it.
-      //
-      // If it does not exist, we rename the current function to the
-      // libdevice functiono name.
-      if (Function *Replacement = F.getParent()->getFunction(CUDALibDeviceFunc))
-        F.replaceAllUsesWith(Replacement);
-      else
-        F.setName(CUDALibDeviceFunc);
-      RequiresLibDevice = true;
-    }
-  }
-
-  return RequiresLibDevice;
-}
-
-void GPUNodeBuilder::addCUDALibDevice() {
-  if (Arch != GPUArch::NVPTX64)
-    return;
-
-  if (requiresCUDALibDevice()) {
-    SMDiagnostic Error;
-
-    errs() << CUDALibDevice << "\n";
-    auto LibDeviceModule =
-        parseIRFile(CUDALibDevice, Error, GPUModule->getContext());
-
-    if (!LibDeviceModule) {
-      BuildSuccessful = false;
-      report_fatal_error("Could not find or load libdevice. Skipping GPU "
-                         "kernel generation. Please set -polly-acc-libdevice "
-                         "accordingly.\n");
-      return;
-    }
-
-    Linker L(*GPUModule);
-
-    // Set an nvptx64 target triple to avoid linker warnings. The original
-    // triple of the libdevice files are nvptx-unknown-unknown.
-    LibDeviceModule->setTargetTriple(Triple::normalize("nvptx64-nvidia-cuda"));
-    L.linkInModule(std::move(LibDeviceModule), Linker::LinkOnlyNeeded);
-  }
-}
-
-std::string GPUNodeBuilder::finalizeKernelFunction() {
-
-  if (verifyModule(*GPUModule)) {
-    LLVM_DEBUG(dbgs() << "verifyModule failed on module:\n";
-               GPUModule->print(dbgs(), nullptr); dbgs() << "\n";);
-    LLVM_DEBUG(dbgs() << "verifyModule Error:\n";
-               verifyModule(*GPUModule, &dbgs()););
-
-    if (FailOnVerifyModuleFailure)
-      llvm_unreachable("VerifyModule failed.");
-
-    BuildSuccessful = false;
-    return "";
-  }
-
-  addCUDALibDevice();
-
-  if (DumpKernelIR)
-    outs() << *GPUModule << "\n";
-
-  if (Arch != GPUArch::SPIR32 && Arch != GPUArch::SPIR64) {
-    // Optimize module.
-    llvm::legacy::PassManager OptPasses;
-    PassManagerBuilder PassBuilder;
-    PassBuilder.OptLevel = 3;
-    PassBuilder.SizeLevel = 0;
-    PassBuilder.populateModulePassManager(OptPasses);
-    OptPasses.run(*GPUModule);
-  }
-
-  std::string Assembly = createKernelASM();
-
-  if (DumpKernelASM)
-    outs() << Assembly << "\n";
-
-  GPUModule.release();
-  KernelIDs.clear();
-
-  return Assembly;
-}
-/// Construct an `isl_pw_aff_list` from a vector of `isl_pw_aff`
-/// @param PwAffs The list of piecewise affine functions to create an
-///               `isl_pw_aff_list` from. We expect an rvalue ref because
-///               all the isl_pw_aff are used up by this function.
-///
-/// @returns  The `isl_pw_aff_list`.
-__isl_give isl_pw_aff_list *
-createPwAffList(isl_ctx *Context,
-                const std::vector<__isl_take isl_pw_aff *> &&PwAffs) {
-  isl_pw_aff_list *List = isl_pw_aff_list_alloc(Context, PwAffs.size());
-
-  for (unsigned i = 0; i < PwAffs.size(); i++) {
-    List = isl_pw_aff_list_insert(List, i, PwAffs[i]);
-  }
-  return List;
-}
-
-/// Align all the `PwAffs` such that they have the same parameter dimensions.
-///
-/// We loop over all `pw_aff` and align all of their spaces together to
-/// create a common space for all the `pw_aff`. This common space is the
-/// `AlignSpace`. We then align all the `pw_aff` to this space. We start
-/// with the given `SeedSpace`.
-/// @param PwAffs    The list of piecewise affine functions we want to align.
-///                  This is an rvalue reference because the entire vector is
-///                  used up by the end of the operation.
-/// @param SeedSpace The space to start the alignment process with.
-/// @returns         A std::pair, whose first element is the aligned space,
-///                  whose second element is the vector of aligned piecewise
-///                  affines.
-static std::pair<__isl_give isl_space *, std::vector<__isl_give isl_pw_aff *>>
-alignPwAffs(const std::vector<__isl_take isl_pw_aff *> &&PwAffs,
-            __isl_take isl_space *SeedSpace) {
-  assert(SeedSpace && "Invalid seed space given.");
-
-  isl_space *AlignSpace = SeedSpace;
-  for (isl_pw_aff *PwAff : PwAffs) {
-    isl_space *PwAffSpace = isl_pw_aff_get_domain_space(PwAff);
-    AlignSpace = isl_space_align_params(AlignSpace, PwAffSpace);
-  }
-  std::vector<isl_pw_aff *> AdjustedPwAffs;
-
-  for (unsigned i = 0; i < PwAffs.size(); i++) {
-    isl_pw_aff *Adjusted = PwAffs[i];
-    assert(Adjusted && "Invalid pw_aff given.");
-    Adjusted = isl_pw_aff_align_params(Adjusted, isl_space_copy(AlignSpace));
-    AdjustedPwAffs.push_back(Adjusted);
-  }
-  return std::make_pair(AlignSpace, AdjustedPwAffs);
-}
-
-namespace {
-class PPCGCodeGeneration final : public ScopPass {
-public:
-  static char ID;
-
-  GPURuntime Runtime = GPURuntime::CUDA;
-
-  GPUArch Architecture = GPUArch::NVPTX64;
-
-  /// The scop that is currently processed.
-  Scop *S;
-
-  LoopInfo *LI;
-  DominatorTree *DT;
-  ScalarEvolution *SE;
-  const DataLayout *DL;
-  RegionInfo *RI;
-
-  PPCGCodeGeneration() : ScopPass(ID) {
-    // Apply defaults.
-    Runtime = GPURuntimeChoice;
-    Architecture = GPUArchChoice;
-  }
-
-  /// Construct compilation options for PPCG.
-  ///
-  /// @returns The compilation options.
-  ppcg_options *createPPCGOptions() {
-    auto DebugOptions =
-        (ppcg_debug_options *)malloc(sizeof(ppcg_debug_options));
-    auto Options = (ppcg_options *)malloc(sizeof(ppcg_options));
-
-    DebugOptions->dump_schedule_constraints = false;
-    DebugOptions->dump_schedule = false;
-    DebugOptions->dump_final_schedule = false;
-    DebugOptions->dump_sizes = false;
-    DebugOptions->verbose = false;
-
-    Options->debug = DebugOptions;
-
-    Options->group_chains = false;
-    Options->reschedule = true;
-    Options->scale_tile_loops = false;
-    Options->wrap = false;
-
-    Options->non_negative_parameters = false;
-    Options->ctx = nullptr;
-    Options->sizes = nullptr;
-
-    Options->tile = true;
-    Options->tile_size = 32;
-
-    Options->isolate_full_tiles = false;
-
-    Options->use_private_memory = PrivateMemory;
-    Options->use_shared_memory = SharedMemory;
-    Options->max_shared_memory = 48 * 1024;
-
-    Options->target = PPCG_TARGET_CUDA;
-    Options->openmp = false;
-    Options->linearize_device_arrays = true;
-    Options->allow_gnu_extensions = false;
-
-    Options->unroll_copy_shared = false;
-    Options->unroll_gpu_tile = false;
-    Options->live_range_reordering = true;
-
-    Options->live_range_reordering = true;
-    Options->hybrid = false;
-    Options->opencl_compiler_options = nullptr;
-    Options->opencl_use_gpu = false;
-    Options->opencl_n_include_file = 0;
-    Options->opencl_include_files = nullptr;
-    Options->opencl_print_kernel_types = false;
-    Options->opencl_embed_kernel_code = false;
-
-    Options->save_schedule_file = nullptr;
-    Options->load_schedule_file = nullptr;
-
-    return Options;
-  }
-
-  /// Get a tagged access relation containing all accesses of type @p AccessTy.
-  ///
-  /// Instead of a normal access of the form:
-  ///
-  ///   Stmt[i,j,k] -> Array[f_0(i,j,k), f_1(i,j,k)]
-  ///
-  /// a tagged access has the form
-  ///
-  ///   [Stmt[i,j,k] -> id[]] -> Array[f_0(i,j,k), f_1(i,j,k)]
-  ///
-  /// where 'id' is an additional space that references the memory access that
-  /// triggered the access.
-  ///
-  /// @param AccessTy The type of the memory accesses to collect.
-  ///
-  /// @return The relation describing all tagged memory accesses.
-  isl_union_map *getTaggedAccesses(enum MemoryAccess::AccessType AccessTy) {
-    isl_union_map *Accesses = isl_union_map_empty(S->getParamSpace().release());
-
-    for (auto &Stmt : *S)
-      for (auto &Acc : Stmt)
-        if (Acc->getType() == AccessTy) {
-          isl_map *Relation = Acc->getAccessRelation().release();
-          Relation =
-              isl_map_intersect_domain(Relation, Stmt.getDomain().release());
-
-          isl_space *Space = isl_map_get_space(Relation);
-          Space = isl_space_range(Space);
-          Space = isl_space_from_range(Space);
-          Space =
-              isl_space_set_tuple_id(Space, isl_dim_in, Acc->getId().release());
-          isl_map *Universe = isl_map_universe(Space);
-          Relation = isl_map_domain_product(Relation, Universe);
-          Accesses = isl_union_map_add_map(Accesses, Relation);
-        }
-
-    return Accesses;
-  }
-
-  /// Get the set of all read accesses, tagged with the access id.
-  ///
-  /// @see getTaggedAccesses
-  isl_union_map *getTaggedReads() {
-    return getTaggedAccesses(MemoryAccess::READ);
-  }
-
-  /// Get the set of all may (and must) accesses, tagged with the access id.
-  ///
-  /// @see getTaggedAccesses
-  isl_union_map *getTaggedMayWrites() {
-    return isl_union_map_union(getTaggedAccesses(MemoryAccess::MAY_WRITE),
-                               getTaggedAccesses(MemoryAccess::MUST_WRITE));
-  }
-
-  /// Get the set of all must accesses, tagged with the access id.
-  ///
-  /// @see getTaggedAccesses
-  isl_union_map *getTaggedMustWrites() {
-    return getTaggedAccesses(MemoryAccess::MUST_WRITE);
-  }
-
-  /// Collect parameter and array names as isl_ids.
-  ///
-  /// To reason about the different parameters and arrays used, ppcg requires
-  /// a list of all isl_ids in use. As PPCG traditionally performs
-  /// source-to-source compilation each of these isl_ids is mapped to the
-  /// expression that represents it. As we do not have a corresponding
-  /// expression in Polly, we just map each id to a 'zero' expression to match
-  /// the data format that ppcg expects.
-  ///
-  /// @returns Retun a map from collected ids to 'zero' ast expressions.
-  __isl_give isl_id_to_ast_expr *getNames() {
-    auto *Names = isl_id_to_ast_expr_alloc(
-        S->getIslCtx().get(),
-        S->getNumParams() + std::distance(S->array_begin(), S->array_end()));
-    auto *Zero = isl_ast_expr_from_val(isl_val_zero(S->getIslCtx().get()));
-
-    for (const SCEV *P : S->parameters()) {
-      isl_id *Id = S->getIdForParam(P).release();
-      Names = isl_id_to_ast_expr_set(Names, Id, isl_ast_expr_copy(Zero));
-    }
-
-    for (auto &Array : S->arrays()) {
-      auto Id = Array->getBasePtrId().release();
-      Names = isl_id_to_ast_expr_set(Names, Id, isl_ast_expr_copy(Zero));
-    }
-
-    isl_ast_expr_free(Zero);
-
-    return Names;
-  }
-
-  /// Create a new PPCG scop from the current scop.
-  ///
-  /// The PPCG scop is initialized with data from the current polly::Scop. From
-  /// this initial data, the data-dependences in the PPCG scop are initialized.
-  /// We do not use Polly's dependence analysis for now, to ensure we match
-  /// the PPCG default behaviour more closely.
-  ///
-  /// @returns A new ppcg scop.
-  ppcg_scop *createPPCGScop() {
-    MustKillsInfo KillsInfo = computeMustKillsInfo(*S);
-
-    auto PPCGScop = (ppcg_scop *)malloc(sizeof(ppcg_scop));
-
-    PPCGScop->options = createPPCGOptions();
-    // enable live range reordering
-    PPCGScop->options->live_range_reordering = 1;
-
-    PPCGScop->start = 0;
-    PPCGScop->end = 0;
-
-    PPCGScop->context = S->getContext().release();
-    PPCGScop->domain = S->getDomains().release();
-    // TODO: investigate this further. PPCG calls collect_call_domains.
-    PPCGScop->call = isl_union_set_from_set(S->getContext().release());
-    PPCGScop->tagged_reads = getTaggedReads();
-    PPCGScop->reads = S->getReads().release();
-    PPCGScop->live_in = nullptr;
-    PPCGScop->tagged_may_writes = getTaggedMayWrites();
-    PPCGScop->may_writes = S->getWrites().release();
-    PPCGScop->tagged_must_writes = getTaggedMustWrites();
-    PPCGScop->must_writes = S->getMustWrites().release();
-    PPCGScop->live_out = nullptr;
-    PPCGScop->tagged_must_kills = KillsInfo.TaggedMustKills.release();
-    PPCGScop->must_kills = KillsInfo.MustKills.release();
-
-    PPCGScop->tagger = nullptr;
-    PPCGScop->independence =
-        isl_union_map_empty(isl_set_get_space(PPCGScop->context));
-    PPCGScop->dep_flow = nullptr;
-    PPCGScop->tagged_dep_flow = nullptr;
-    PPCGScop->dep_false = nullptr;
-    PPCGScop->dep_forced = nullptr;
-    PPCGScop->dep_order = nullptr;
-    PPCGScop->tagged_dep_order = nullptr;
-
-    PPCGScop->schedule = S->getScheduleTree().release();
-    // If we have something non-trivial to kill, add it to the schedule
-    if (KillsInfo.KillsSchedule.get())
-      PPCGScop->schedule = isl_schedule_sequence(
-          PPCGScop->schedule, KillsInfo.KillsSchedule.release());
-
-    PPCGScop->names = getNames();
-    PPCGScop->pet = nullptr;
-
-    compute_tagger(PPCGScop);
-    compute_dependences(PPCGScop);
-    eliminate_dead_code(PPCGScop);
-
-    return PPCGScop;
-  }
-
-  /// Collect the array accesses in a statement.
-  ///
-  /// @param Stmt The statement for which to collect the accesses.
-  ///
-  /// @returns A list of array accesses.
-  gpu_stmt_access *getStmtAccesses(ScopStmt &Stmt) {
-    gpu_stmt_access *Accesses = nullptr;
-
-    for (MemoryAccess *Acc : Stmt) {
-      auto Access =
-          isl_alloc_type(S->getIslCtx().get(), struct gpu_stmt_access);
-      Access->read = Acc->isRead();
-      Access->write = Acc->isWrite();
-      Access->access = Acc->getAccessRelation().release();
-      isl_space *Space = isl_map_get_space(Access->access);
-      Space = isl_space_range(Space);
-      Space = isl_space_from_range(Space);
-      Space = isl_space_set_tuple_id(Space, isl_dim_in, Acc->getId().release());
-      isl_map *Universe = isl_map_universe(Space);
-      Access->tagged_access =
-          isl_map_domain_product(Acc->getAccessRelation().release(), Universe);
-      Access->exact_write = !Acc->isMayWrite();
-      Access->ref_id = Acc->getId().release();
-      Access->next = Accesses;
-      Access->n_index = Acc->getScopArrayInfo()->getNumberOfDimensions();
-      // TODO: Also mark one-element accesses to arrays as fixed-element.
-      Access->fixed_element =
-          Acc->isLatestScalarKind() ? isl_bool_true : isl_bool_false;
-      Accesses = Access;
-    }
-
-    return Accesses;
-  }
-
-  /// Collect the list of GPU statements.
-  ///
-  /// Each statement has an id, a pointer to the underlying data structure,
-  /// as well as a list with all memory accesses.
-  ///
-  /// TODO: Initialize the list of memory accesses.
-  ///
-  /// @returns A linked-list of statements.
-  gpu_stmt *getStatements() {
-    gpu_stmt *Stmts = isl_calloc_array(S->getIslCtx().get(), struct gpu_stmt,
-                                       std::distance(S->begin(), S->end()));
-
-    int i = 0;
-    for (auto &Stmt : *S) {
-      gpu_stmt *GPUStmt = &Stmts[i];
-
-      GPUStmt->id = Stmt.getDomainId().release();
-
-      // We use the pet stmt pointer to keep track of the Polly statements.
-      GPUStmt->stmt = (pet_stmt *)&Stmt;
-      GPUStmt->accesses = getStmtAccesses(Stmt);
-      i++;
-    }
-
-    return Stmts;
-  }
-
-  /// Derive the extent of an array.
-  ///
-  /// The extent of an array is the set of elements that are within the
-  /// accessed array. For the inner dimensions, the extent constraints are
-  /// 0 and the size of the corresponding array dimension. For the first
-  /// (outermost) dimension, the extent constraints are the minimal and maximal
-  /// subscript value for the first dimension.
-  ///
-  /// @param Array The array to derive the extent for.
-  ///
-  /// @returns An isl_set describing the extent of the array.
-  isl::set getExtent(ScopArrayInfo *Array) {
-    unsigned NumDims = Array->getNumberOfDimensions();
-
-    if (Array->getNumberOfDimensions() == 0)
-      return isl::set::universe(Array->getSpace());
-
-    isl::union_map Accesses = S->getAccesses(Array);
-    isl::union_set AccessUSet = Accesses.range();
-    AccessUSet = AccessUSet.coalesce();
-    AccessUSet = AccessUSet.detect_equalities();
-    AccessUSet = AccessUSet.coalesce();
-
-    if (AccessUSet.is_empty())
-      return isl::set::empty(Array->getSpace());
-
-    isl::set AccessSet = AccessUSet.extract_set(Array->getSpace());
-
-    isl::local_space LS = isl::local_space(Array->getSpace());
-
-    isl::pw_aff Val = isl::aff::var_on_domain(LS, isl::dim::set, 0);
-    isl::pw_aff OuterMin = AccessSet.dim_min(0);
-    isl::pw_aff OuterMax = AccessSet.dim_max(0);
-    OuterMin = OuterMin.add_dims(isl::dim::in,
-                                 unsignedFromIslSize(Val.dim(isl::dim::in)));
-    OuterMax = OuterMax.add_dims(isl::dim::in,
-                                 unsignedFromIslSize(Val.dim(isl::dim::in)));
-    OuterMin = OuterMin.set_tuple_id(isl::dim::in, Array->getBasePtrId());
-    OuterMax = OuterMax.set_tuple_id(isl::dim::in, Array->getBasePtrId());
-
-    isl::set Extent = isl::set::universe(Array->getSpace());
-
-    Extent = Extent.intersect(OuterMin.le_set(Val));
-    Extent = Extent.intersect(OuterMax.ge_set(Val));
-
-    for (unsigned i = 1; i < NumDims; ++i)
-      Extent = Extent.lower_bound_si(isl::dim::set, i, 0);
-
-    for (unsigned i = 0; i < NumDims; ++i) {
-      isl::pw_aff PwAff = Array->getDimensionSizePw(i);
-
-      // isl_pw_aff can be NULL for zero dimension. Only in the case of a
-      // Fortran array will we have a legitimate dimension.
-      if (PwAff.is_null()) {
-        assert(i == 0 && "invalid dimension isl_pw_aff for nonzero dimension");
-        continue;
-      }
-
-      isl::pw_aff Val = isl::aff::var_on_domain(
-          isl::local_space(Array->getSpace()), isl::dim::set, i);
-      PwAff = PwAff.add_dims(isl::dim::in,
-                             unsignedFromIslSize(Val.dim(isl::dim::in)));
-      PwAff = PwAff.set_tuple_id(isl::dim::in, Val.get_tuple_id(isl::dim::in));
-      isl::set Set = PwAff.gt_set(Val);
-      Extent = Set.intersect(Extent);
-    }
-
-    return Extent;
-  }
-
-  /// Derive the bounds of an array.
-  ///
-  /// For the first dimension we derive the bound of the array from the extent
-  /// of this dimension. For inner dimensions we obtain their size directly from
-  /// ScopArrayInfo.
-  ///
-  /// @param PPCGArray The array to compute bounds for.
-  /// @param Array The polly array from which to take the information.
-  void setArrayBounds(gpu_array_info &PPCGArray, ScopArrayInfo *Array) {
-    std::vector<isl_pw_aff *> Bounds;
-
-    if (PPCGArray.n_index > 0) {
-      if (isl_set_is_empty(PPCGArray.extent)) {
-        isl_set *Dom = isl_set_copy(PPCGArray.extent);
-        isl_local_space *LS = isl_local_space_from_space(
-            isl_space_params(isl_set_get_space(Dom)));
-        isl_set_free(Dom);
-        isl_pw_aff *Zero = isl_pw_aff_from_aff(isl_aff_zero_on_domain(LS));
-        Bounds.push_back(Zero);
-      } else {
-        isl_set *Dom = isl_set_copy(PPCGArray.extent);
-        Dom = isl_set_project_out(Dom, isl_dim_set, 1, PPCGArray.n_index - 1);
-        isl_pw_aff *Bound = isl_set_dim_max(isl_set_copy(Dom), 0);
-        isl_set_free(Dom);
-        Dom = isl_pw_aff_domain(isl_pw_aff_copy(Bound));
-        isl_local_space *LS =
-            isl_local_space_from_space(isl_set_get_space(Dom));
-        isl_aff *One = isl_aff_zero_on_domain(LS);
-        One = isl_aff_add_constant_si(One, 1);
-        Bound = isl_pw_aff_add(Bound, isl_pw_aff_alloc(Dom, One));
-        Bound = isl_pw_aff_gist(Bound, S->getContext().release());
-        Bounds.push_back(Bound);
-      }
-    }
-
-    for (unsigned i = 1; i < PPCGArray.n_index; ++i) {
-      isl_pw_aff *Bound = Array->getDimensionSizePw(i).release();
-      auto LS = isl_pw_aff_get_domain_space(Bound);
-      auto Aff = isl_multi_aff_zero(LS);
-
-      // We need types to work out, which is why we perform this weird dance
-      // with `Aff` and `Bound`. Consider this example:
-
-      // LS: [p] -> { [] }
-      // Zero: [p] -> { [] } | Implicitly, is [p] -> { ~ -> [] }.
-      // This `~` is used to denote a "null space" (which is different from
-      // a *zero dimensional* space), which is something that ISL does not
-      // show you when pretty printing.
-
-      // Bound: [p] -> { [] -> [(10p)] } | Here, the [] is a *zero dimensional*
-      // space, not a "null space" which does not exist at all.
-
-      // When we pullback (precompose) `Bound` with `Zero`, we get:
-      // Bound . Zero =
-      //     ([p] -> { [] -> [(10p)] }) . ([p] -> {~ -> [] }) =
-      //     [p] -> { ~ -> [(10p)] } =
-      //     [p] -> [(10p)] (as ISL pretty prints it)
-      // Bound Pullback: [p] -> { [(10p)] }
-
-      // We want this kind of an expression for Bound, without a
-      // zero dimensional input, but with a "null space" input for the types
-      // to work out later on, as far as I (Siddharth Bhat) understand.
-      // I was unable to find a reference to this in the ISL manual.
-      // References: Tobias Grosser.
-
-      Bound = isl_pw_aff_pullback_multi_aff(Bound, Aff);
-      Bounds.push_back(Bound);
-    }
-
-    /// To construct a `isl_multi_pw_aff`, we need all the indivisual `pw_aff`
-    /// to have the same parameter dimensions. So, we need to align them to an
-    /// appropriate space.
-    /// Scop::Context is _not_ an appropriate space, because when we have
-    /// `-polly-ignore-parameter-bounds` enabled, the Scop::Context does not
-    /// contain all parameter dimensions.
-    /// So, use the helper `alignPwAffs` to align all the `isl_pw_aff` together.
-    isl_space *SeedAlignSpace = S->getParamSpace().release();
-    SeedAlignSpace = isl_space_add_dims(SeedAlignSpace, isl_dim_set, 1);
-
-    isl_space *AlignSpace = nullptr;
-    std::vector<isl_pw_aff *> AlignedBounds;
-    std::tie(AlignSpace, AlignedBounds) =
-        alignPwAffs(std::move(Bounds), SeedAlignSpace);
-
-    assert(AlignSpace && "alignPwAffs did not initialise AlignSpace");
-
-    isl_pw_aff_list *BoundsList =
-        createPwAffList(S->getIslCtx().get(), std::move(AlignedBounds));
-
-    isl_space *BoundsSpace = isl_set_get_space(PPCGArray.extent);
-    BoundsSpace = isl_space_align_params(BoundsSpace, AlignSpace);
-
-    assert(BoundsSpace && "Unable to access space of array.");
-    assert(BoundsList && "Unable to access list of bounds.");
-
-    PPCGArray.bound =
-        isl_multi_pw_aff_from_pw_aff_list(BoundsSpace, BoundsList);
-    assert(PPCGArray.bound && "PPCGArray.bound was not constructed correctly.");
-  }
-
-  /// Create the arrays for @p PPCGProg.
-  ///
-  /// @param PPCGProg The program to compute the arrays for.
-  void createArrays(gpu_prog *PPCGProg,
-                    const SmallVector<ScopArrayInfo *, 4> &ValidSAIs) {
-    int i = 0;
-    for (auto &Array : ValidSAIs) {
-      std::string TypeName;
-      raw_string_ostream OS(TypeName);
-
-      OS << *Array->getElementType();
-      TypeName = OS.str();
-
-      gpu_array_info &PPCGArray = PPCGProg->array[i];
-
-      PPCGArray.space = Array->getSpace().release();
-      PPCGArray.type = strdup(TypeName.c_str());
-      PPCGArray.size = DL->getTypeAllocSize(Array->getElementType());
-      PPCGArray.name = strdup(Array->getName().c_str());
-      PPCGArray.extent = nullptr;
-      PPCGArray.n_index = Array->getNumberOfDimensions();
-      PPCGArray.extent = getExtent(Array).release();
-      PPCGArray.n_ref = 0;
-      PPCGArray.refs = nullptr;
-      PPCGArray.accessed = true;
-      PPCGArray.read_only_scalar =
-          Array->isReadOnly() && Array->getNumberOfDimensions() == 0;
-      PPCGArray.has_compound_element = false;
-      PPCGArray.local = false;
-      PPCGArray.declare_local = false;
-      PPCGArray.global = false;
-      PPCGArray.linearize = false;
-      PPCGArray.dep_order = nullptr;
-      PPCGArray.user = Array;
-
-      PPCGArray.bound = nullptr;
-      setArrayBounds(PPCGArray, Array);
-      i++;
-
-      collect_references(PPCGProg, &PPCGArray);
-      PPCGArray.only_fixed_element = only_fixed_element_accessed(&PPCGArray);
-    }
-  }
-
-  /// Create an identity map between the arrays in the scop.
-  ///
-  /// @returns An identity map between the arrays in the scop.
-  isl_union_map *getArrayIdentity() {
-    isl_union_map *Maps = isl_union_map_empty(S->getParamSpace().release());
-
-    for (auto &Array : S->arrays()) {
-      isl_space *Space = Array->getSpace().release();
-      Space = isl_space_map_from_set(Space);
-      isl_map *Identity = isl_map_identity(Space);
-      Maps = isl_union_map_add_map(Maps, Identity);
-    }
-
-    return Maps;
-  }
-
-  /// Create a default-initialized PPCG GPU program.
-  ///
-  /// @returns A new gpu program description.
-  gpu_prog *createPPCGProg(ppcg_scop *PPCGScop) {
-
-    if (!PPCGScop)
-      return nullptr;
-
-    auto PPCGProg = isl_calloc_type(S->getIslCtx().get(), struct gpu_prog);
-
-    PPCGProg->ctx = S->getIslCtx().get();
-    PPCGProg->scop = PPCGScop;
-    PPCGProg->context = isl_set_copy(PPCGScop->context);
-    PPCGProg->read = isl_union_map_copy(PPCGScop->reads);
-    PPCGProg->may_write = isl_union_map_copy(PPCGScop->may_writes);
-    PPCGProg->must_write = isl_union_map_copy(PPCGScop->must_writes);
-    PPCGProg->tagged_must_kill =
-        isl_union_map_copy(PPCGScop->tagged_must_kills);
-    PPCGProg->to_inner = getArrayIdentity();
-    PPCGProg->to_outer = getArrayIdentity();
-    // TODO: verify that this assignment is correct.
-    PPCGProg->any_to_outer = nullptr;
-    PPCGProg->n_stmts = std::distance(S->begin(), S->end());
-    PPCGProg->stmts = getStatements();
-
-    // Only consider arrays that have a non-empty extent.
-    // Otherwise, this will cause us to consider the following kinds of
-    // empty arrays:
-    //     1. Invariant loads that are represented by SAI objects.
-    //     2. Arrays with statically known zero size.
-    auto ValidSAIsRange =
-        make_filter_range(S->arrays(), [this](ScopArrayInfo *SAI) -> bool {
-          return !getExtent(SAI).is_empty();
-        });
-    SmallVector<ScopArrayInfo *, 4> ValidSAIs(ValidSAIsRange.begin(),
-                                              ValidSAIsRange.end());
-
-    PPCGProg->n_array =
-        ValidSAIs.size(); // std::distance(S->array_begin(), S->array_end());
-    PPCGProg->array = isl_calloc_array(
-        S->getIslCtx().get(), struct gpu_array_info, PPCGProg->n_array);
-
-    createArrays(PPCGProg, ValidSAIs);
-
-    PPCGProg->array_order = nullptr;
-    collect_order_dependences(PPCGProg);
-
-    PPCGProg->may_persist = compute_may_persist(PPCGProg);
-    return PPCGProg;
-  }
-
-  struct PrintGPUUserData {
-    struct cuda_info *CudaInfo;
-    struct gpu_prog *PPCGProg;
-    std::vector<ppcg_kernel *> Kernels;
-  };
-
-  /// Print a user statement node in the host code.
-  ///
-  /// We use ppcg's printing facilities to print the actual statement and
-  /// additionally build up a list of all kernels that are encountered in the
-  /// host ast.
-  ///
-  /// @param P The printer to print to
-  /// @param Options The printing options to use
-  /// @param Node The node to print
-  /// @param User A user pointer to carry additional data. This pointer is
-  ///             expected to be of type PrintGPUUserData.
-  ///
-  /// @returns A printer to which the output has been printed.
-  static __isl_give isl_printer *
-  printHostUser(__isl_take isl_printer *P,
-                __isl_take isl_ast_print_options *Options,
-                __isl_take isl_ast_node *Node, void *User) {
-    auto Data = (struct PrintGPUUserData *)User;
-    auto Id = isl_ast_node_get_annotation(Node);
-
-    if (Id) {
-      bool IsUser = !strcmp(isl_id_get_name(Id), "user");
-
-      // If this is a user statement, format it ourselves as ppcg would
-      // otherwise try to call pet functionality that is not available in
-      // Polly.
-      if (IsUser) {
-        P = isl_printer_start_line(P);
-        P = isl_printer_print_ast_node(P, Node);
-        P = isl_printer_end_line(P);
-        isl_id_free(Id);
-        isl_ast_print_options_free(Options);
-        return P;
-      }
-
-      auto Kernel = (struct ppcg_kernel *)isl_id_get_user(Id);
-      isl_id_free(Id);
-      Data->Kernels.push_back(Kernel);
-    }
-
-    return print_host_user(P, Options, Node, User);
-  }
-
-  /// Print C code corresponding to the control flow in @p Kernel.
-  ///
-  /// @param Kernel The kernel to print
-  void printKernel(ppcg_kernel *Kernel) {
-    auto *P = isl_printer_to_str(S->getIslCtx().get());
-    P = isl_printer_set_output_format(P, ISL_FORMAT_C);
-    auto *Options = isl_ast_print_options_alloc(S->getIslCtx().get());
-    P = isl_ast_node_print(Kernel->tree, P, Options);
-    char *String = isl_printer_get_str(P);
-    outs() << String << "\n";
-    free(String);
-    isl_printer_free(P);
-  }
-
-  /// Print C code corresponding to the GPU code described by @p Tree.
-  ///
-  /// @param Tree An AST describing GPU code
-  /// @param PPCGProg The PPCG program from which @Tree has been constructed.
-  void printGPUTree(isl_ast_node *Tree, gpu_prog *PPCGProg) {
-    auto *P = isl_printer_to_str(S->getIslCtx().get());
-    P = isl_printer_set_output_format(P, ISL_FORMAT_C);
-
-    PrintGPUUserData Data;
-    Data.PPCGProg = PPCGProg;
-
-    auto *Options = isl_ast_print_options_alloc(S->getIslCtx().get());
-    Options =
-        isl_ast_print_options_set_print_user(Options, printHostUser, &Data);
-    P = isl_ast_node_print(Tree, P, Options);
-    char *String = isl_printer_get_str(P);
-    outs() << "# host\n";
-    outs() << String << "\n";
-    free(String);
-    isl_printer_free(P);
-
-    for (auto Kernel : Data.Kernels) {
-      outs() << "# kernel" << Kernel->id << "\n";
-      printKernel(Kernel);
-    }
-  }
-
-  // Generate a GPU program using PPCG.
-  //
-  // GPU mapping consists of multiple steps:
-  //
-  //  1) Compute new schedule for the program.
-  //  2) Map schedule to GPU (TODO)
-  //  3) Generate code for new schedule (TODO)
-  //
-  // We do not use here the Polly ScheduleOptimizer, as the schedule optimizer
-  // is mostly CPU specific. Instead, we use PPCG's GPU code generation
-  // strategy directly from this pass.
-  gpu_gen *generateGPU(ppcg_scop *PPCGScop, gpu_prog *PPCGProg) {
-
-    auto PPCGGen = isl_calloc_type(S->getIslCtx().get(), struct gpu_gen);
-
-    PPCGGen->ctx = S->getIslCtx().get();
-    PPCGGen->options = PPCGScop->options;
-    PPCGGen->print = nullptr;
-    PPCGGen->print_user = nullptr;
-    PPCGGen->build_ast_expr = &pollyBuildAstExprForStmt;
-    PPCGGen->prog = PPCGProg;
-    PPCGGen->tree = nullptr;
-    PPCGGen->types.n = 0;
-    PPCGGen->types.name = nullptr;
-    PPCGGen->sizes = nullptr;
-    PPCGGen->used_sizes = nullptr;
-    PPCGGen->kernel_id = 0;
-
-    // Set scheduling strategy to same strategy PPCG is using.
-    isl_options_set_schedule_serialize_sccs(PPCGGen->ctx, false);
-    isl_options_set_schedule_outer_coincidence(PPCGGen->ctx, true);
-    isl_options_set_schedule_maximize_band_depth(PPCGGen->ctx, true);
-    isl_options_set_schedule_whole_component(PPCGGen->ctx, false);
-
-    isl_schedule *Schedule = get_schedule(PPCGGen);
-
-    int has_permutable = has_any_permutable_node(Schedule);
-
-    Schedule =
-        isl_schedule_align_params(Schedule, S->getFullParamSpace().release());
-
-    if (!has_permutable || has_permutable < 0) {
-      Schedule = isl_schedule_free(Schedule);
-      LLVM_DEBUG(dbgs() << getUniqueScopName(S)
-                        << " does not have permutable bands. Bailing out\n";);
-    } else {
-      const bool CreateTransferToFromDevice = !PollyManagedMemory;
-      Schedule = map_to_device(PPCGGen, Schedule, CreateTransferToFromDevice);
-      PPCGGen->tree = generate_code(PPCGGen, isl_schedule_copy(Schedule));
-    }
-
-    if (DumpSchedule) {
-      isl_printer *P = isl_printer_to_str(S->getIslCtx().get());
-      P = isl_printer_set_yaml_style(P, ISL_YAML_STYLE_BLOCK);
-      P = isl_printer_print_str(P, "Schedule\n");
-      P = isl_printer_print_str(P, "========\n");
-      if (Schedule)
-        P = isl_printer_print_schedule(P, Schedule);
-      else
-        P = isl_printer_print_str(P, "No schedule found\n");
-
-      outs() << isl_printer_get_str(P) << "\n";
-      isl_printer_free(P);
-    }
-
-    if (DumpCode) {
-      outs() << "Code\n";
-      outs() << "====\n";
-      if (PPCGGen->tree)
-        printGPUTree(PPCGGen->tree, PPCGProg);
-      else
-        outs() << "No code generated\n";
-    }
-
-    isl_schedule_free(Schedule);
-
-    return PPCGGen;
-  }
-
-  /// Free gpu_gen structure.
-  ///
-  /// @param PPCGGen The ppcg_gen object to free.
-  void freePPCGGen(gpu_gen *PPCGGen) {
-    isl_ast_node_free(PPCGGen->tree);
-    isl_union_map_free(PPCGGen->sizes);
-    isl_union_map_free(PPCGGen->used_sizes);
-    free(PPCGGen);
-  }
-
-  /// Free the options in the ppcg scop structure.
-  ///
-  /// ppcg is not freeing these options for us. To avoid leaks we do this
-  /// ourselves.
-  ///
-  /// @param PPCGScop The scop referencing the options to free.
-  void freeOptions(ppcg_scop *PPCGScop) {
-    free(PPCGScop->options->debug);
-    PPCGScop->options->debug = nullptr;
-    free(PPCGScop->options);
-    PPCGScop->options = nullptr;
-  }
-
-  /// Approximate the number of points in the set.
-  ///
-  /// This function returns an ast expression that overapproximates the number
-  /// of points in an isl set through the rectangular hull surrounding this set.
-  ///
-  /// @param Set   The set to count.
-  /// @param Build The isl ast build object to use for creating the ast
-  ///              expression.
-  ///
-  /// @returns An approximation of the number of points in the set.
-  __isl_give isl_ast_expr *approxPointsInSet(__isl_take isl_set *Set,
-                                             __isl_keep isl_ast_build *Build) {
-
-    isl_val *One = isl_val_int_from_si(isl_set_get_ctx(Set), 1);
-    auto *Expr = isl_ast_expr_from_val(isl_val_copy(One));
-
-    isl_space *Space = isl_set_get_space(Set);
-    Space = isl_space_params(Space);
-    auto *Univ = isl_set_universe(Space);
-    isl_pw_aff *OneAff = isl_pw_aff_val_on_domain(Univ, One);
-
-    for (long i = 0, n = isl_set_dim(Set, isl_dim_set); i < n; i++) {
-      isl_pw_aff *Max = isl_set_dim_max(isl_set_copy(Set), i);
-      isl_pw_aff *Min = isl_set_dim_min(isl_set_copy(Set), i);
-      isl_pw_aff *DimSize = isl_pw_aff_sub(Max, Min);
-      DimSize = isl_pw_aff_add(DimSize, isl_pw_aff_copy(OneAff));
-      auto DimSizeExpr = isl_ast_build_expr_from_pw_aff(Build, DimSize);
-      Expr = isl_ast_expr_mul(Expr, DimSizeExpr);
-    }
-
-    isl_set_free(Set);
-    isl_pw_aff_free(OneAff);
-
-    return Expr;
-  }
-
-  /// Approximate a number of dynamic instructions executed by a given
-  /// statement.
-  ///
-  /// @param Stmt  The statement for which to compute the number of dynamic
-  ///              instructions.
-  /// @param Build The isl ast build object to use for creating the ast
-  ///              expression.
-  /// @returns An approximation of the number of dynamic instructions executed
-  ///          by @p Stmt.
-  __isl_give isl_ast_expr *approxDynamicInst(ScopStmt &Stmt,
-                                             __isl_keep isl_ast_build *Build) {
-    auto Iterations = approxPointsInSet(Stmt.getDomain().release(), Build);
-
-    long InstCount = 0;
-
-    if (Stmt.isBlockStmt()) {
-      auto *BB = Stmt.getBasicBlock();
-      InstCount = std::distance(BB->begin(), BB->end());
-    } else {
-      auto *R = Stmt.getRegion();
-
-      for (auto *BB : R->blocks()) {
-        InstCount += std::distance(BB->begin(), BB->end());
-      }
-    }
-
-    isl_val *InstVal = isl_val_int_from_si(S->getIslCtx().get(), InstCount);
-    auto *InstExpr = isl_ast_expr_from_val(InstVal);
-    return isl_ast_expr_mul(InstExpr, Iterations);
-  }
-
-  /// Approximate dynamic instructions executed in scop.
-  ///
-  /// @param S     The scop for which to approximate dynamic instructions.
-  /// @param Build The isl ast build object to use for creating the ast
-  ///              expression.
-  /// @returns An approximation of the number of dynamic instructions executed
-  ///          in @p S.
-  __isl_give isl_ast_expr *
-  getNumberOfIterations(Scop &S, __isl_keep isl_ast_build *Build) {
-    isl_ast_expr *Instructions;
-
-    isl_val *Zero = isl_val_int_from_si(S.getIslCtx().get(), 0);
-    Instructions = isl_ast_expr_from_val(Zero);
-
-    for (ScopStmt &Stmt : S) {
-      isl_ast_expr *StmtInstructions = approxDynamicInst(Stmt, Build);
-      Instructions = isl_ast_expr_add(Instructions, StmtInstructions);
-    }
-    return Instructions;
-  }
-
-  /// Create a check that ensures sufficient compute in scop.
-  ///
-  /// @param S     The scop for which to ensure sufficient compute.
-  /// @param Build The isl ast build object to use for creating the ast
-  ///              expression.
-  /// @returns An expression that evaluates to TRUE in case of sufficient
-  ///          compute and to FALSE, otherwise.
-  __isl_give isl_ast_expr *
-  createSufficientComputeCheck(Scop &S, __isl_keep isl_ast_build *Build) {
-    auto Iterations = getNumberOfIterations(S, Build);
-    auto *MinComputeVal = isl_val_int_from_si(S.getIslCtx().get(), MinCompute);
-    auto *MinComputeExpr = isl_ast_expr_from_val(MinComputeVal);
-    return isl_ast_expr_ge(Iterations, MinComputeExpr);
-  }
-
-  /// Check if the basic block contains a function we cannot codegen for GPU
-  /// kernels.
-  ///
-  /// If this basic block does something with a `Function` other than calling
-  /// a function that we support in a kernel, return true.
-  bool containsInvalidKernelFunctionInBlock(const BasicBlock *BB,
-                                            bool AllowCUDALibDevice) {
-    for (const Instruction &Inst : *BB) {
-      const CallInst *Call = dyn_cast<CallInst>(&Inst);
-      if (Call && isValidFunctionInKernel(Call->getCalledFunction(),
-                                          AllowCUDALibDevice))
-        continue;
-
-      for (Value *Op : Inst.operands())
-        // Look for functions among operands of Inst.
-        if (isa<Function>(Op->stripPointerCasts())) {
-          LLVM_DEBUG(dbgs()
-                     << Inst << " has illegal use of function in kernel.\n");
-          return true;
-        }
-    }
-    return false;
-  }
-
-  /// Return whether the Scop S uses functions in a way that we do not support.
-  bool containsInvalidKernelFunction(const Scop &S, bool AllowCUDALibDevice) {
-    for (auto &Stmt : S) {
-      if (Stmt.isBlockStmt()) {
-        if (containsInvalidKernelFunctionInBlock(Stmt.getBasicBlock(),
-                                                 AllowCUDALibDevice))
-          return true;
-      } else {
-        assert(Stmt.isRegionStmt() &&
-               "Stmt was neither block nor region statement");
-        for (const BasicBlock *BB : Stmt.getRegion()->blocks())
-          if (containsInvalidKernelFunctionInBlock(BB, AllowCUDALibDevice))
-            return true;
-      }
-    }
-    return false;
-  }
-
-  /// Generate code for a given GPU AST described by @p Root.
-  ///
-  /// @param Root An isl_ast_node pointing to the root of the GPU AST.
-  /// @param Prog The GPU Program to generate code for.
-  void generateCode(__isl_take isl_ast_node *Root, gpu_prog *Prog) {
-    ScopAnnotator Annotator;
-    Annotator.buildAliasScopes(*S);
-
-    Region *R = &S->getRegion();
-
-    simplifyRegion(R, DT, LI, RI);
-
-    BasicBlock *EnteringBB = R->getEnteringBlock();
-
-    PollyIRBuilder Builder(EnteringBB->getContext(), ConstantFolder(),
-                           IRInserter(Annotator));
-    Builder.SetInsertPoint(EnteringBB->getTerminator());
-
-    // Only build the run-time condition and parameters _after_ having
-    // introduced the conditional branch. This is important as the conditional
-    // branch will guard the original scop from new induction variables that
-    // the SCEVExpander may introduce while code generating the parameters and
-    // which may introduce scalar dependences that prevent us from correctly
-    // code generating this scop.
-    BBPair StartExitBlocks;
-    BranchInst *CondBr = nullptr;
-    std::tie(StartExitBlocks, CondBr) =
-        executeScopConditionally(*S, Builder.getTrue(), *DT, *RI, *LI);
-    BasicBlock *StartBlock = std::get<0>(StartExitBlocks);
-
-    assert(CondBr && "CondBr not initialized by executeScopConditionally");
-
-    GPUNodeBuilder NodeBuilder(Builder, Annotator, *DL, *LI, *SE, *DT, *S,
-                               StartBlock, Prog, Runtime, Architecture);
-
-    // TODO: Handle LICM
-    auto SplitBlock = StartBlock->getSinglePredecessor();
-    Builder.SetInsertPoint(SplitBlock->getTerminator());
-
-    isl_ast_build *Build = isl_ast_build_alloc(S->getIslCtx().get());
-    isl::ast_expr Condition =
-        IslAst::buildRunCondition(*S, isl::manage_copy(Build));
-    isl_ast_expr *SufficientCompute = createSufficientComputeCheck(*S, Build);
-    Condition =
-        isl::manage(isl_ast_expr_and(Condition.release(), SufficientCompute));
-    isl_ast_build_free(Build);
-
-    // preload invariant loads. Note: This should happen before the RTC
-    // because the RTC may depend on values that are invariant load hoisted.
-    if (!NodeBuilder.preloadInvariantLoads()) {
-      // Patch the introduced branch condition to ensure that we always execute
-      // the original SCoP.
-      auto *FalseI1 = Builder.getFalse();
-      auto *SplitBBTerm = Builder.GetInsertBlock()->getTerminator();
-      SplitBBTerm->setOperand(0, FalseI1);
-
-      LLVM_DEBUG(dbgs() << "preloading invariant loads failed in function: " +
-                               S->getFunction().getName() +
-                               " | Scop Region: " + S->getNameStr());
-      // adjust the dominator tree accordingly.
-      auto *ExitingBlock = StartBlock->getUniqueSuccessor();
-      assert(ExitingBlock);
-      auto *MergeBlock = ExitingBlock->getUniqueSuccessor();
-      assert(MergeBlock);
-      polly::markBlockUnreachable(*StartBlock, Builder);
-      polly::markBlockUnreachable(*ExitingBlock, Builder);
-      auto *ExitingBB = S->getExitingBlock();
-      assert(ExitingBB);
-
-      DT->changeImmediateDominator(MergeBlock, ExitingBB);
-      DT->eraseNode(ExitingBlock);
-      isl_ast_node_free(Root);
-    } else {
-
-      if (polly::PerfMonitoring) {
-        PerfMonitor P(*S, EnteringBB->getParent()->getParent());
-        P.initialize();
-        P.insertRegionStart(SplitBlock->getTerminator());
-
-        // TODO: actually think if this is the correct exiting block to place
-        // the `end` performance marker. Invariant load hoisting changes
-        // the CFG in a way that I do not precisely understand, so I
-        // (Siddharth<siddu.druid@gmail.com>) should come back to this and
-        // think about which exiting block to use.
-        auto *ExitingBlock = StartBlock->getUniqueSuccessor();
-        assert(ExitingBlock);
-        BasicBlock *MergeBlock = ExitingBlock->getUniqueSuccessor();
-        P.insertRegionEnd(MergeBlock->getTerminator());
-      }
-
-      NodeBuilder.addParameters(S->getContext().release());
-      Value *RTC = NodeBuilder.createRTC(Condition.release());
-      Builder.GetInsertBlock()->getTerminator()->setOperand(0, RTC);
-
-      Builder.SetInsertPoint(&*StartBlock->begin());
-
-      NodeBuilder.create(Root);
-    }
-
-    /// In case a sequential kernel has more surrounding loops as any parallel
-    /// kernel, the SCoP is probably mostly sequential. Hence, there is no
-    /// point in running it on a GPU.
-    if (NodeBuilder.DeepestSequential > NodeBuilder.DeepestParallel)
-      CondBr->setOperand(0, Builder.getFalse());
-
-    if (!NodeBuilder.BuildSuccessful)
-      CondBr->setOperand(0, Builder.getFalse());
-  }
-
-  bool runOnScop(Scop &CurrentScop) override {
-    S = &CurrentScop;
-    LI = &getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
-    DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
-    SE = &getAnalysis<ScalarEvolutionWrapperPass>().getSE();
-    DL = &S->getRegion().getEntry()->getModule()->getDataLayout();
-    RI = &getAnalysis<RegionInfoPass>().getRegionInfo();
-
-    LLVM_DEBUG(dbgs() << "PPCGCodeGen running on : " << getUniqueScopName(S)
-                      << " | loop depth: " << S->getMaxLoopDepth() << "\n");
-
-    // We currently do not support functions other than intrinsics inside
-    // kernels, as code generation will need to offload function calls to the
-    // kernel. This may lead to a kernel trying to call a function on the host.
-    // This also allows us to prevent codegen from trying to take the
-    // address of an intrinsic function to send to the kernel.
-    if (containsInvalidKernelFunction(CurrentScop,
-                                      Architecture == GPUArch::NVPTX64)) {
-      LLVM_DEBUG(
-          dbgs() << getUniqueScopName(S)
-                 << " contains function which cannot be materialised in a GPU "
-                    "kernel. Bailing out.\n";);
-      return false;
-    }
-
-    auto PPCGScop = createPPCGScop();
-    auto PPCGProg = createPPCGProg(PPCGScop);
-    auto PPCGGen = generateGPU(PPCGScop, PPCGProg);
-
-    if (PPCGGen->tree) {
-      generateCode(isl_ast_node_copy(PPCGGen->tree), PPCGProg);
-      CurrentScop.markAsToBeSkipped();
-    } else {
-      LLVM_DEBUG(dbgs() << getUniqueScopName(S)
-                        << " has empty PPCGGen->tree. Bailing out.\n");
-    }
-
-    freeOptions(PPCGScop);
-    freePPCGGen(PPCGGen);
-    gpu_prog_free(PPCGProg);
-    ppcg_scop_free(PPCGScop);
-
-    return true;
-  }
-
-  void printScop(raw_ostream &, Scop &) const override {}
-
-  void getAnalysisUsage(AnalysisUsage &AU) const override {
-    ScopPass::getAnalysisUsage(AU);
-
-    AU.addRequired<DominatorTreeWrapperPass>();
-    AU.addRequired<RegionInfoPass>();
-    AU.addRequired<ScalarEvolutionWrapperPass>();
-    AU.addRequired<ScopDetectionWrapperPass>();
-    AU.addRequired<ScopInfoRegionPass>();
-    AU.addRequired<LoopInfoWrapperPass>();
-
-    // FIXME: We do not yet add regions for the newly generated code to the
-    //        region tree.
-  }
-};
-} // namespace
-
-char PPCGCodeGeneration::ID = 1;
-
-Pass *polly::createPPCGCodeGenerationPass(GPUArch Arch, GPURuntime Runtime) {
-  PPCGCodeGeneration *generator = new PPCGCodeGeneration();
-  generator->Runtime = Runtime;
-  generator->Architecture = Arch;
-  return generator;
-}
-
-INITIALIZE_PASS_BEGIN(PPCGCodeGeneration, "polly-codegen-ppcg",
-                      "Polly - Apply PPCG translation to SCOP", false, false)
-INITIALIZE_PASS_DEPENDENCY(DependenceInfo);
-INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass);
-INITIALIZE_PASS_DEPENDENCY(LoopInfoWrapperPass);
-INITIALIZE_PASS_DEPENDENCY(RegionInfoPass);
-INITIALIZE_PASS_DEPENDENCY(ScalarEvolutionWrapperPass);
-INITIALIZE_PASS_DEPENDENCY(ScopDetectionWrapperPass);
-INITIALIZE_PASS_END(PPCGCodeGeneration, "polly-codegen-ppcg",
-                    "Polly - Apply PPCG translation to SCOP", false, false)
diff --git a/polly/lib/CodeGen/RuntimeDebugBuilder.cpp b/polly/lib/CodeGen/RuntimeDebugBuilder.cpp
index 680a7ff05982a..fe30924c02862 100644
--- a/polly/lib/CodeGen/RuntimeDebugBuilder.cpp
+++ b/polly/lib/CodeGen/RuntimeDebugBuilder.cpp
@@ -9,7 +9,6 @@
 //===----------------------------------------------------------------------===//
 
 #include "polly/CodeGen/RuntimeDebugBuilder.h"
-#include "llvm/IR/IntrinsicsNVPTX.h"
 #include "llvm/IR/Module.h"
 #include <string>
 #include <vector>
@@ -17,6 +16,16 @@
 using namespace llvm;
 using namespace polly;
 
+llvm::Value *RuntimeDebugBuilder::getPrintableString(PollyIRBuilder &Builder,
+                                                     llvm::StringRef Str) {
+  // FIXME: addressspace(4) is a marker for a string (for the %s conversion
+  // specifier) but should be using the default address space. This only works
+  // because CPU backends typically ignore the address space. For constant
+  // strings as returned by getPrintableString, the format string should instead
+  // directly spell out the string.
+  return Builder.CreateGlobalStringPtr(Str, "", 4);
+}
+
 Function *RuntimeDebugBuilder::getVPrintF(PollyIRBuilder &Builder) {
   Module *M = Builder.GetInsertBlock()->getParent()->getParent();
   const char *Name = "vprintf";
@@ -33,72 +42,9 @@ Function *RuntimeDebugBuilder::getVPrintF(PollyIRBuilder &Builder) {
   return F;
 }
 
-Function *RuntimeDebugBuilder::getAddressSpaceCast(PollyIRBuilder &Builder,
-                                                   unsigned Src, unsigned Dst,
-                                                   unsigned SrcBits,
-                                                   unsigned DstBits) {
-  Module *M = Builder.GetInsertBlock()->getParent()->getParent();
-  auto Name = std::string("llvm.nvvm.ptr.constant.to.gen.p") +
-              std::to_string(Dst) + "i" + std::to_string(DstBits) + ".p" +
-              std::to_string(Src) + "i" + std::to_string(SrcBits);
-  Function *F = M->getFunction(Name);
-
-  if (!F) {
-    GlobalValue::LinkageTypes Linkage = Function::ExternalLinkage;
-    FunctionType *Ty = FunctionType::get(
-        PointerType::get(Builder.getIntNTy(DstBits), Dst),
-        PointerType::get(Builder.getIntNTy(SrcBits), Src), false);
-    F = Function::Create(Ty, Linkage, Name, M);
-  }
-
-  return F;
-}
-
-std::vector<Value *>
-RuntimeDebugBuilder::getGPUThreadIdentifiers(PollyIRBuilder &Builder) {
-  std::vector<Value *> Identifiers;
-
-  auto M = Builder.GetInsertBlock()->getParent()->getParent();
-
-  std::vector<Function *> BlockIDs = {
-      Intrinsic::getDeclaration(M, Intrinsic::nvvm_read_ptx_sreg_ctaid_x),
-      Intrinsic::getDeclaration(M, Intrinsic::nvvm_read_ptx_sreg_ctaid_y),
-      Intrinsic::getDeclaration(M, Intrinsic::nvvm_read_ptx_sreg_ctaid_z),
-  };
-
-  Identifiers.push_back(Builder.CreateGlobalStringPtr("> block-id: ", "", 4));
-  for (auto GetID : BlockIDs) {
-    Value *Id = Builder.CreateCall(GetID, {});
-    Id = Builder.CreateIntCast(Id, Builder.getInt64Ty(), false);
-    Identifiers.push_back(Id);
-    Identifiers.push_back(Builder.CreateGlobalStringPtr(" ", "", 4));
-  }
-
-  Identifiers.push_back(Builder.CreateGlobalStringPtr("| ", "", 4));
-
-  std::vector<Function *> ThreadIDs = {
-      Intrinsic::getDeclaration(M, Intrinsic::nvvm_read_ptx_sreg_tid_x),
-      Intrinsic::getDeclaration(M, Intrinsic::nvvm_read_ptx_sreg_tid_y),
-      Intrinsic::getDeclaration(M, Intrinsic::nvvm_read_ptx_sreg_tid_z),
-  };
-
-  Identifiers.push_back(Builder.CreateGlobalStringPtr("thread-id: ", "", 4));
-  for (auto GetId : ThreadIDs) {
-    Value *Id = Builder.CreateCall(GetId, {});
-    Id = Builder.CreateIntCast(Id, Builder.getInt64Ty(), false);
-    Identifiers.push_back(Id);
-    Identifiers.push_back(Builder.CreateGlobalStringPtr(" ", "", 4));
-  }
-
-  return Identifiers;
-}
-
-void RuntimeDebugBuilder::createPrinter(PollyIRBuilder &Builder, bool IsGPU,
+void RuntimeDebugBuilder::createPrinter(PollyIRBuilder &Builder,
                                         ArrayRef<Value *> Values) {
-  if (IsGPU)
-    createGPUPrinterT(Builder, Values);
-  else
-    createCPUPrinterT(Builder, Values);
+  createCPUPrinterT(Builder, Values);
 }
 
 bool RuntimeDebugBuilder::isPrintable(Type *Ty) {
@@ -169,78 +115,6 @@ void RuntimeDebugBuilder::createCPUPrinterT(PollyIRBuilder &Builder,
   createFlush(Builder);
 }
 
-void RuntimeDebugBuilder::createGPUPrinterT(PollyIRBuilder &Builder,
-                                            ArrayRef<Value *> Values) {
-  std::string str;
-
-  auto *Zero = Builder.getInt64(0);
-
-  auto ToPrint = getGPUThreadIdentifiers(Builder);
-
-  ToPrint.push_back(Builder.CreateGlobalStringPtr("\n  ", "", 4));
-  ToPrint.insert(ToPrint.end(), Values.begin(), Values.end());
-
-  const DataLayout &DL = Builder.GetInsertBlock()->getModule()->getDataLayout();
-
-  // Allocate print buffer (assuming 2*32 bit per element)
-  auto T = ArrayType::get(Builder.getInt32Ty(), ToPrint.size() * 2);
-  Value *Data = new AllocaInst(
-      T, DL.getAllocaAddrSpace(), "polly.vprint.buffer",
-      &Builder.GetInsertBlock()->getParent()->getEntryBlock().front());
-  auto *DataPtr = Builder.CreateGEP(T, Data, {Zero, Zero});
-
-  int Offset = 0;
-  for (auto Val : ToPrint) {
-    auto Ptr = Builder.CreateGEP(Builder.getInt32Ty(), DataPtr,
-                                 Builder.getInt64(Offset));
-    Type *Ty = Val->getType();
-
-    if (Ty->isFloatingPointTy()) {
-      if (!Ty->isDoubleTy())
-        Val = Builder.CreateFPExt(Val, Builder.getDoubleTy());
-    } else if (Ty->isIntegerTy()) {
-      if (Ty->getIntegerBitWidth() < 64) {
-        Val = Builder.CreateSExt(Val, Builder.getInt64Ty());
-      } else {
-        assert(Ty->getIntegerBitWidth() == 64 &&
-               "Integer types larger 64 bit not supported");
-        // fallthrough
-      }
-    } else if (isa<PointerType>(Ty)) {
-      if (Ty == Builder.getInt8PtrTy(4)) {
-        // Pointers in constant address space are printed as strings
-        Val = Builder.CreateGEP(Builder.getInt8Ty(), Val, Builder.getInt64(0));
-        auto F = RuntimeDebugBuilder::getAddressSpaceCast(Builder, 4, 0);
-        Val = Builder.CreateCall(F, Val);
-      } else {
-        Val = Builder.CreatePtrToInt(Val, Builder.getInt64Ty());
-      }
-    } else {
-      llvm_unreachable("Unknown type");
-    }
-
-    Ty = Val->getType();
-    Ptr = Builder.CreatePointerBitCastOrAddrSpaceCast(Ptr, Ty->getPointerTo(5));
-    Builder.CreateAlignedStore(Val, Ptr, Align(4));
-
-    if (Ty->isFloatingPointTy())
-      str += "%f";
-    else if (Ty->isIntegerTy())
-      str += "%ld";
-    else
-      str += "%s";
-
-    Offset += 2;
-  }
-
-  Value *Format = Builder.CreateGlobalStringPtr(str, "polly.vprintf.buffer", 4);
-  Format = Builder.CreateCall(getAddressSpaceCast(Builder, 4, 0), Format);
-
-  Data = Builder.CreateBitCast(Data, Builder.getInt8PtrTy());
-
-  Builder.CreateCall(getVPrintF(Builder), {Format, Data});
-}
-
 Function *RuntimeDebugBuilder::getPrintF(PollyIRBuilder &Builder) {
   Module *M = Builder.GetInsertBlock()->getParent()->getParent();
   const char *Name = "printf";
diff --git a/polly/lib/Exchange/JSONExporter.cpp b/polly/lib/Exchange/JSONExporter.cpp
index 6eafa5b10f0c2..74d4e6c7993fa 100644
--- a/polly/lib/Exchange/JSONExporter.cpp
+++ b/polly/lib/Exchange/JSONExporter.cpp
@@ -326,7 +326,7 @@ static bool importSchedule(Scop &S, const json::Object &JScop,
 
   auto ScheduleMap = isl::union_map::empty(S.getIslCtx());
   for (ScopStmt &Stmt : S) {
-    if (NewSchedule.find(&Stmt) != NewSchedule.end())
+    if (NewSchedule.contains(&Stmt))
       ScheduleMap = ScheduleMap.unite(NewSchedule[&Stmt]);
     else
       ScheduleMap = ScheduleMap.unite(Stmt.getSchedule());
diff --git a/polly/lib/External/CMakeLists.txt b/polly/lib/External/CMakeLists.txt
index 8a360e453ce1f..458925f9b6e4d 100644
--- a/polly/lib/External/CMakeLists.txt
+++ b/polly/lib/External/CMakeLists.txt
@@ -314,91 +314,3 @@ if (POLLY_BUNDLED_ISL)
   target_compile_options(PollyISL PRIVATE ${DISABLE_WARNING_FLAGS})
   target_compile_options(polly-isl-test PRIVATE ${DISABLE_WARNING_FLAGS})
 endif (POLLY_BUNDLED_ISL)
-
-
-# External: Polyhedral Parallel Code Generator
-if (GPU_CODEGEN)
-  set(PET_SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/pet")
-  set(PPCG_SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/ppcg")
-  set(PPCG_BINARY_DIR "${CMAKE_CURRENT_BINARY_DIR}/ppcg")
-
-  # Determine version of ppcg
-  if (EXISTS "${PPCG_SOURCE_DIR}/GIT_HEAD_ID")
-    # The source comes from a 'make dist' archive
-    file(READ "${PPCG_SOURCE_DIR}/GIT_HEAD_ID" PPCG_GIT_HEAD_ID)
-    string(STRIP "${PPCG_GIT_HEAD_ID}" PPCG_GIT_HEAD_ID)
-  elseif (EXISTS "${PPCG_SOURCE_DIR}/gitversion.h")
-    # The source directory is preconfigured
-    file(READ "${PPCG_SOURCE_DIR}/gitversion.h" GITVERSION_H)
-    string(REGEX REPLACE ".*\\\"([^\\\"]*)\\\".*" "\\1" PPCG_GIT_HEAD_ID "${GITVERSION_H}")
-  elseif ()
-    # Unknown revision
-    # TODO: We could look for a .git and get the revision from HEAD
-    set(PPCG_GIT_HEAD_ID "UNKNOWN")
-  endif ()
-
-  message(STATUS "PPCG version: ${PPCG_GIT_HEAD_ID}")
-
-  set (PPCG_FILES
-       ppcg/cuda.c
-       ppcg/cuda_common.c
-       ppcg/external.c
-       ppcg/gpu_array_tile.c
-       ppcg/gpu.c
-       ppcg/gpu_array_tile.c
-       ppcg/gpu_group.c
-       ppcg/gpu_hybrid.c
-       ppcg/gpu_print.c
-       ppcg/gpu_tree.c
-       ppcg/grouping.c
-       ppcg/hybrid.c
-       ppcg/ppcg.c
-       ppcg/ppcg_options.c
-       ppcg/print.c
-       ppcg/schedule.c
-       ppcg/util.c
-       )
-
-  include_directories(BEFORE
-    ${PPCG_BINARY_DIR}
-    ${PPCG_SOURCE_DIR}/imath
-    ${PPCG_SOURCE_DIR}/include
-    ${PET_SOURCE_DIR}/include
-  )
-
-  add_polly_library(PollyPPCG
-    ${PPCG_FILES}
-  )
-
-  target_link_libraries(PollyPPCG PUBLIC ${ISL_TARGET})
-
-  # Disable warnings for upstream projects.
-  if (MSVC)
-    set(DISABLE_WARNING_FLAGS
-      -wd4018 # 'expression' : signed/unsigned mismatch
-      -wd4090 # 'operation' : different 'modifier' qualifiers
-      -wd4200 # nonstandard extension used: zero-sized array in struct/union
-      -wd4201 # nonstandard extension used: nameless struct/union
-      -wd4334 # 'operator': result of 32-bit shift implicitly converted to 64 bits (was 64-bit shift intended?)
-      -wd4221 # nonstandard extension used : 'identifier' : cannot be initialized using address of automatic variable
-    )
-    if (POLLY_BUNDLED_ISL)
-      target_compile_options(PollyISL PRIVATE ${DISABLE_WARNING_FLAGS})
-      target_compile_options(polly-isl-test PRIVATE ${DISABLE_WARNING_FLAGS})
-    endif (POLLY_BUNDLED_ISL)
-    target_compile_options(PollyPPCG PRIVATE ${DISABLE_WARNING_FLAGS})
-  else ()
-    if (POLLY_BUNDLED_ISL)
-      set_target_properties(PollyISL polly-isl-test PROPERTIES COMPILE_FLAGS "-w")
-    endif (POLLY_BUNDLED_ISL)
-    set_target_properties(PollyPPCG PROPERTIES COMPILE_FLAGS "-w")
-  endif ()
-
-  if(MSVC)
-    # In the Windows API (with some exceptions), the maximum length for a path is
-    # MAX_PATH, which is defined as 260 characters.
-    target_compile_definitions(PollyPPCG PRIVATE "-DPATH_MAX=260")
-  endif ()
-
-  target_compile_options(PollyPPCG PRIVATE ${DISABLE_WARNING_FLAGS})
-endif ()
diff --git a/polly/lib/External/pet/include/pet.h b/polly/lib/External/pet/include/pet.h
deleted file mode 100644
index 28a28b8bf0a8c..0000000000000
--- a/polly/lib/External/pet/include/pet.h
+++ /dev/null
@@ -1,622 +0,0 @@
-#ifndef PET_H
-#define PET_H
-
-#include <isl/aff.h>
-#include <isl/arg.h>
-#include <isl/ast_build.h>
-#include <isl/set.h>
-#include <isl/map.h>
-#include <isl/union_map.h>
-#include <isl/printer.h>
-#include <isl/id_to_ast_expr.h>
-#include <isl/id_to_pw_aff.h>
-#include <isl/schedule.h>
-
-#if defined(__cplusplus)
-extern "C" {
-#endif
-
-struct pet_options;
-ISL_ARG_DECL(pet_options, struct pet_options, pet_options_args)
-
-/* Create an isl_ctx that references the pet options. */
-isl_ctx *isl_ctx_alloc_with_pet_options();
-
-/* If autodetect is set, any valid scop is extracted.
- * Otherwise, the scop needs to be delimited by pragmas.
- */
-int pet_options_set_autodetect(isl_ctx *ctx, int val);
-int pet_options_get_autodetect(isl_ctx *ctx);
-
-int pet_options_set_detect_conditional_assignment(isl_ctx *ctx, int val);
-int pet_options_get_detect_conditional_assignment(isl_ctx *ctx);
-
-/* If encapsulate-dynamic-control is set, then any dynamic control
- * in the input program will be encapsulated in macro statements.
- * This means in particular that no statements with arguments
- * will be created.
- */
-int pet_options_set_encapsulate_dynamic_control(isl_ctx *ctx, int val);
-int pet_options_get_encapsulate_dynamic_control(isl_ctx *ctx);
-
-#define	PET_OVERFLOW_AVOID	0
-#define	PET_OVERFLOW_IGNORE	1
-int pet_options_set_signed_overflow(isl_ctx *ctx, int val);
-int pet_options_get_signed_overflow(isl_ctx *ctx);
-
-struct pet_loc;
-typedef struct pet_loc pet_loc;
-
-/* Return an additional reference to "loc". */
-__isl_give pet_loc *pet_loc_copy(__isl_keep pet_loc *loc);
-/* Free a reference to "loc". */
-pet_loc *pet_loc_free(__isl_take pet_loc *loc);
-
-/* Return the offset in the input file of the start of "loc". */
-unsigned pet_loc_get_start(__isl_keep pet_loc *loc);
-/* Return the offset in the input file of the character after "loc". */
-unsigned pet_loc_get_end(__isl_keep pet_loc *loc);
-/* Return the line number of a line within the "loc" region. */
-int pet_loc_get_line(__isl_keep pet_loc *loc);
-/* Return the indentation of the "loc" region. */
-__isl_keep const char *pet_loc_get_indent(__isl_keep pet_loc *loc);
-
-enum pet_expr_type {
-	pet_expr_error = -1,
-	pet_expr_access,
-	pet_expr_call,
-	pet_expr_cast,
-	pet_expr_int,
-	pet_expr_double,
-	pet_expr_op
-};
-
-enum pet_op_type {
-	/* only compound assignments operators before assignment */
-	pet_op_add_assign,
-	pet_op_sub_assign,
-	pet_op_mul_assign,
-	pet_op_div_assign,
-	pet_op_and_assign,
-	pet_op_xor_assign,
-	pet_op_or_assign,
-	pet_op_assign,
-	pet_op_add,
-	pet_op_sub,
-	pet_op_mul,
-	pet_op_div,
-	pet_op_mod,
-	pet_op_shl,
-	pet_op_shr,
-	pet_op_eq,
-	pet_op_ne,
-	pet_op_le,
-	pet_op_ge,
-	pet_op_lt,
-	pet_op_gt,
-	pet_op_minus,
-	pet_op_post_inc,
-	pet_op_post_dec,
-	pet_op_pre_inc,
-	pet_op_pre_dec,
-	pet_op_address_of,
-	pet_op_assume,
-	pet_op_kill,
-	pet_op_and,
-	pet_op_xor,
-	pet_op_or,
-	pet_op_not,
-	pet_op_land,
-	pet_op_lor,
-	pet_op_lnot,
-	pet_op_cond,
-	pet_op_last
-};
-
-/* Index into the pet_expr->args array when pet_expr->type == pet_expr_unary
- */
-enum pet_un_arg_type {
-	pet_un_arg
-};
-
-/* Indices into the pet_expr->args array when
- * pet_expr->type == pet_expr_binary
- */
-enum pet_bin_arg_type {
-	pet_bin_lhs,
-	pet_bin_rhs
-};
-
-/* Indices into the pet_expr->args array when
- * pet_expr->type == pet_expr_ternary
- */
-enum pet_ter_arg_type {
-	pet_ter_cond,
-	pet_ter_true,
-	pet_ter_false
-};
-
-struct pet_expr;
-typedef struct pet_expr pet_expr;
-
-/* Return an additional reference to "expr". */
-__isl_give pet_expr *pet_expr_copy(__isl_keep pet_expr *expr);
-/* Free a reference to "expr". */
-__isl_null pet_expr *pet_expr_free(__isl_take pet_expr *expr);
-
-/* Return the isl_ctx in which "expr" was created. */
-isl_ctx *pet_expr_get_ctx(__isl_keep pet_expr *expr);
-
-/* Return the type of "expr". */
-enum pet_expr_type pet_expr_get_type(__isl_keep pet_expr *expr);
-/* Return the number of arguments of "expr". */
-int pet_expr_get_n_arg(__isl_keep pet_expr *expr);
-/* Set the number of arguments of "expr" to "n". */
-__isl_give pet_expr *pet_expr_set_n_arg(__isl_take pet_expr *expr, int n);
-/* Return the argument of "expr" at position "pos". */
-__isl_give pet_expr *pet_expr_get_arg(__isl_keep pet_expr *expr, int pos);
-/* Replace the argument of "expr" at position "pos" by "arg". */
-__isl_give pet_expr *pet_expr_set_arg(__isl_take pet_expr *expr, int pos,
-	__isl_take pet_expr *arg);
-
-/* Return the operation type of operation expression "expr". */
-enum pet_op_type pet_expr_op_get_type(__isl_keep pet_expr *expr);
-/* Replace the operation type of operation expression "expr" by "type". */
-__isl_give pet_expr *pet_expr_op_set_type(__isl_take pet_expr *expr,
-	enum pet_op_type type);
-
-/* Construct a (read) access pet_expr from an index expression. */
-__isl_give pet_expr *pet_expr_from_index(__isl_take isl_multi_pw_aff *index);
-
-/* Does "expr" represent an affine expression? */
-isl_bool pet_expr_is_affine(__isl_keep pet_expr *expr);
-/* Does the access expression "expr" read the accessed elements? */
-isl_bool pet_expr_access_is_read(__isl_keep pet_expr *expr);
-/* Does the access expression "expr" write to the accessed elements? */
-isl_bool pet_expr_access_is_write(__isl_keep pet_expr *expr);
-/* Does the access expression "expr" kill the accessed elements? */
-isl_bool pet_expr_access_is_kill(__isl_keep pet_expr *expr);
-/* Mark "expr" as a read depending on "read". */
-__isl_give pet_expr *pet_expr_access_set_read(__isl_take pet_expr *expr,
-	int read);
-/* Mark "expr" as a write depending on "write". */
-__isl_give pet_expr *pet_expr_access_set_write(__isl_take pet_expr *expr,
-	int write);
-/* Mark "expr" as a kill depending on "kill". */
-__isl_give pet_expr *pet_expr_access_set_kill(__isl_take pet_expr *expr,
-	int kill);
-/* Return the reference identifier of access expression "expr". */
-__isl_give isl_id *pet_expr_access_get_ref_id(__isl_keep pet_expr *expr);
-/* Replace the reference identifier of access expression "expr" by "ref_id". */
-__isl_give pet_expr *pet_expr_access_set_ref_id(__isl_take pet_expr *expr,
-	__isl_take isl_id *ref_id);
-/* Return the identifier of the outer array accessed by "expr". */
-__isl_give isl_id *pet_expr_access_get_id(__isl_keep pet_expr *expr);
-/* Return the index expression of access expression "expr". */
-__isl_give isl_multi_pw_aff *pet_expr_access_get_index(
-	__isl_keep pet_expr *expr);
-
-/* Return the potential read access relation of access expression "expr". */
-__isl_give isl_union_map *pet_expr_access_get_may_read(
-	__isl_keep pet_expr *expr);
-/* Return the potential write access relation of access expression "expr". */
-__isl_give isl_union_map *pet_expr_access_get_may_write(
-	__isl_keep pet_expr *expr);
-/* Return the definite write access relation of access expression "expr". */
-__isl_give isl_union_map *pet_expr_access_get_must_write(
-	__isl_keep pet_expr *expr);
-/* Return the argument dependent potential read access relation of "expr". */
-__isl_give isl_union_map *pet_expr_access_get_dependent_may_read(
-	__isl_keep pet_expr *expr);
-/* Return the argument dependent potential write access relation of "expr". */
-__isl_give isl_union_map *pet_expr_access_get_dependent_may_write(
-	__isl_keep pet_expr *expr);
-/* Return the argument dependent definite write access relation of "expr". */
-__isl_give isl_union_map *pet_expr_access_get_dependent_must_write(
-	__isl_keep pet_expr *expr);
-/* Return the tagged potential read access relation of access "expr". */
-__isl_give isl_union_map *pet_expr_access_get_tagged_may_read(
-	__isl_keep pet_expr *expr);
-/* Return the tagged potential write access relation of access "expr". */
-__isl_give isl_union_map *pet_expr_access_get_tagged_may_write(
-	__isl_keep pet_expr *expr);
-
-/* Return the name of the function called by "expr". */
-__isl_keep const char *pet_expr_call_get_name(__isl_keep pet_expr *expr);
-/* Replace the name of the function called by "expr" by "name". */
-__isl_give pet_expr *pet_expr_call_set_name(__isl_take pet_expr *expr,
-	__isl_keep const char *name);
-
-/* Create a pet_expr representing a cast of "arg" to "type_name". */
-__isl_give pet_expr *pet_expr_new_cast(const char *type_name,
-	__isl_take pet_expr *arg);
-/* Replace the type of the cast performed by "expr" by "name". */
-__isl_give pet_expr *pet_expr_cast_set_type_name(__isl_take pet_expr *expr,
-	__isl_keep const char *name);
-
-/* Return the value of the integer represented by "expr". */
-__isl_give isl_val *pet_expr_int_get_val(__isl_keep pet_expr *expr);
-/* Replace the value of the integer represented by "expr" by "v". */
-__isl_give pet_expr *pet_expr_int_set_val(__isl_take pet_expr *expr,
-	__isl_take isl_val *v);
-
-/* Return a string representation of the double expression "expr". */
-__isl_give char *pet_expr_double_get_str(__isl_keep pet_expr *expr);
-/* Replace value and string representation of the double expression "expr" */
-__isl_give pet_expr *pet_expr_double_set(__isl_take pet_expr *expr,
-	double d, __isl_keep const char *s);
-
-/* Call "fn" on each of the subexpressions of "expr" of type pet_expr_access. */
-int pet_expr_foreach_access_expr(__isl_keep pet_expr *expr,
-	int (*fn)(__isl_keep pet_expr *expr, void *user), void *user);
-/* Call "fn" on each of the subexpressions of "expr" of type pet_expr_call. */
-int pet_expr_foreach_call_expr(__isl_keep pet_expr *expr,
-	int (*fn)(__isl_keep pet_expr *expr, void *user), void *user);
-
-struct pet_context;
-typedef struct pet_context pet_context;
-
-/* Create a context with the given domain. */
-__isl_give pet_context *pet_context_alloc(__isl_take isl_set *domain);
-/* Return an additional reference to "pc". */
-__isl_give pet_context *pet_context_copy(__isl_keep pet_context *pc);
-/* Free a reference to "pc". */
-__isl_null pet_context *pet_context_free(__isl_take pet_context *pc);
-
-/* Return the isl_ctx in which "pc" was created. */
-isl_ctx *pet_context_get_ctx(__isl_keep pet_context *pc);
-
-/* Extract an affine expression defined over the domain of "pc" from "expr"
- * or return NaN.
- */
-__isl_give isl_pw_aff *pet_expr_extract_affine(__isl_keep pet_expr *expr,
-	__isl_keep pet_context *pc);
-
-void pet_expr_dump(__isl_keep pet_expr *expr);
-
-enum pet_tree_type {
-	pet_tree_error = -1,
-	pet_tree_expr,
-	pet_tree_block,
-	pet_tree_break,
-	pet_tree_continue,
-	pet_tree_decl,		/* A declaration without initialization */
-	pet_tree_decl_init,	/* A declaration with initialization */
-	pet_tree_if,		/* An if without an else branch */
-	pet_tree_if_else,	/* An if with an else branch */
-	pet_tree_for,
-	pet_tree_infinite_loop,
-	pet_tree_while,
-	pet_tree_return,
-};
-
-struct pet_tree;
-typedef struct pet_tree pet_tree;
-
-/* Return the isl_ctx in which "tree" was created. */
-isl_ctx *pet_tree_get_ctx(__isl_keep pet_tree *tree);
-
-/* Return an additional reference to "tree". */
-__isl_give pet_tree *pet_tree_copy(__isl_keep pet_tree *tree);
-/* Free a reference to "tree". */
-__isl_null pet_tree *pet_tree_free(__isl_take pet_tree *tree);
-
-/* Return the location of "tree". */
-__isl_give pet_loc *pet_tree_get_loc(__isl_keep pet_tree *tree);
-
-/* Return the type of "tree". */
-enum pet_tree_type pet_tree_get_type(__isl_keep pet_tree *tree);
-
-/* Return the expression of the expression tree "tree". */
-__isl_give pet_expr *pet_tree_expr_get_expr(__isl_keep pet_tree *tree);
-
-/* Return the expression returned by the return tree "tree". */
-__isl_give pet_expr *pet_tree_return_get_expr(__isl_keep pet_tree *tree);
-
-/* Return the number of children of the block tree "tree". */
-int pet_tree_block_n_child(__isl_keep pet_tree *tree);
-/* Return child "pos" of the block tree "tree". */
-__isl_give pet_tree *pet_tree_block_get_child(__isl_keep pet_tree *tree,
-	int pos);
-
-/* Is "tree" a declaration (with or without initialization)? */
-int pet_tree_is_decl(__isl_keep pet_tree *tree);
-/* Return the variable declared by the declaration tree "tree". */
-__isl_give pet_expr *pet_tree_decl_get_var(__isl_keep pet_tree *tree);
-/* Return the initial value of the pet_tree_decl_init tree "tree". */
-__isl_give pet_expr *pet_tree_decl_get_init(__isl_keep pet_tree *tree);
-
-/* Return the condition of the if tree "tree". */
-__isl_give pet_expr *pet_tree_if_get_cond(__isl_keep pet_tree *tree);
-/* Return the then branch of the if tree "tree". */
-__isl_give pet_tree *pet_tree_if_get_then(__isl_keep pet_tree *tree);
-/* Return the else branch of the if tree with else branch "tree". */
-__isl_give pet_tree *pet_tree_if_get_else(__isl_keep pet_tree *tree);
-
-/* Is "tree" a for loop, a while loop or an infinite loop? */
-int pet_tree_is_loop(__isl_keep pet_tree *tree);
-/* Return the induction variable of the for loop "tree" */
-__isl_give pet_expr *pet_tree_loop_get_var(__isl_keep pet_tree *tree);
-/* Return the initial value of the induction variable of the for loop "tree" */
-__isl_give pet_expr *pet_tree_loop_get_init(__isl_keep pet_tree *tree);
-/* Return the condition of the loop tree "tree" */
-__isl_give pet_expr *pet_tree_loop_get_cond(__isl_keep pet_tree *tree);
-/* Return the induction variable of the for loop "tree" */
-__isl_give pet_expr *pet_tree_loop_get_inc(__isl_keep pet_tree *tree);
-/* Return the body of the loop tree "tree" */
-__isl_give pet_tree *pet_tree_loop_get_body(__isl_keep pet_tree *tree);
-
-/* Call "fn" on each top-level expression in the nodes of "tree" */
-int pet_tree_foreach_expr(__isl_keep pet_tree *tree,
-	int (*fn)(__isl_keep pet_expr *expr, void *user), void *user);
-/* Call "fn" on each access subexpression in the nodes of "tree" */
-int pet_tree_foreach_access_expr(__isl_keep pet_tree *tree,
-	int (*fn)(__isl_keep pet_expr *expr, void *user), void *user);
-/* Modify all call subexpressions in the nodes of "tree" through "fn". */
-__isl_give pet_tree *pet_tree_map_call_expr(__isl_take pet_tree *tree,
-	__isl_give pet_expr *(*fn)(__isl_take pet_expr *expr, void *user),
-	void *user);
-
-void pet_tree_dump(__isl_keep pet_tree *tree);
-
-/* "loc" represents the region of the source code that is represented
- * by this statement.
- *
- * If the statement has arguments, i.e., n_arg != 0, then
- * "domain" is a wrapped map, mapping the iteration domain
- * to the values of the arguments for which this statement
- * is executed.
- * Otherwise, it is simply the iteration domain.
- *
- * If one of the arguments is an access expression that accesses
- * more than one element for a given iteration, then the constraints
- * on the value of this argument (encoded in "domain") should be satisfied
- * for all of those accessed elements.
- */
-struct pet_stmt {
-	pet_loc *loc;
-	isl_set *domain;
-	pet_tree *body;
-
-	unsigned n_arg;
-	pet_expr **args;
-};
-
-/* Return the iteration space of "stmt". */
-__isl_give isl_space *pet_stmt_get_space(struct pet_stmt *stmt);
-
-/* Is "stmt" an assignment statement? */
-int pet_stmt_is_assign(struct pet_stmt *stmt);
-/* Is "stmt" a kill statement? */
-int pet_stmt_is_kill(struct pet_stmt *stmt);
-
-/* pet_stmt_build_ast_exprs is currently limited to only handle
- * some forms of data dependent accesses.
- * If pet_stmt_can_build_ast_exprs returns 1, then pet_stmt_build_ast_exprs
- * can safely be called on "stmt".
- */
-int pet_stmt_can_build_ast_exprs(struct pet_stmt *stmt);
-/* Construct an associative array from reference identifiers of
- * access expressions in "stmt" to the corresponding isl_ast_expr.
- * Each index expression is first transformed through "fn_index"
- * (if not NULL).  Then an AST expression is generated using "build".
- * Finally, the AST expression is transformed using "fn_expr"
- * (if not NULL).
- */
-__isl_give isl_id_to_ast_expr *pet_stmt_build_ast_exprs(struct pet_stmt *stmt,
-	__isl_keep isl_ast_build *build,
-	__isl_give isl_multi_pw_aff *(*fn_index)(
-		__isl_take isl_multi_pw_aff *mpa, __isl_keep isl_id *id,
-		void *user), void *user_index,
-	__isl_give isl_ast_expr *(*fn_expr)(__isl_take isl_ast_expr *expr,
-		__isl_keep isl_id *id, void *user), void *user_expr);
-
-/* Print "stmt" to "p".
- *
- * The access expressions in "stmt" are replaced by the isl_ast_expr
- * associated to its reference identifier in "ref2expr".
- */
-__isl_give isl_printer *pet_stmt_print_body(struct pet_stmt *stmt,
-	__isl_take isl_printer *p, __isl_keep isl_id_to_ast_expr *ref2expr);
-
-/* This structure represents a defined type.
- * "name" is the name of the type, while "definition" is a string
- * representation of its definition.
- */
-struct pet_type {
-	char *name;
-	char *definition;
-};
-
-/* context holds constraints on the parameter that ensure that
- * this array has a valid (i.e., non-negative) size
- *
- * extent holds constraints on the indices
- *
- * value_bounds holds constraints on the elements of the array
- * and may be NULL if no such constraints were specified by the user
- *
- * element_size is the size in bytes of each array element
- * element_type is the type of the array elements.
- * element_is_record is set if this type is a record type.
- *
- * live_out is set if the array appears in a live-out pragma
- *
- * if uniquely_defined is set then the array is written by a single access
- * such that any element that is ever read
- * is known to be assigned exactly once before the read
- *
- * declared is set if the array was declared somewhere inside the scop.
- * exposed is set if the declared array is visible outside the scop.
- * outer is set if the type of the array elements is a record and
- * the fields of this record are represented by separate pet_array structures.
- */
-struct pet_array {
-	isl_set *context;
-	isl_set *extent;
-	isl_set *value_bounds;
-	char *element_type;
-	int element_is_record;
-	int element_size;
-	int live_out;
-	int uniquely_defined;
-	int declared;
-	int exposed;
-	int outer;
-};
-
-/* This structure represents an implication on a boolean filter.
- * In particular, if the filter value of an element in the domain
- * of "extension" is equal to "satisfied", then the filter values
- * of the corresponding images in "extension" are also equal
- * to "satisfied".
- */
-struct pet_implication {
-	int satisfied;
-	isl_map *extension;
-};
-
-/* This structure represents an independence implied by a for loop
- * that is marked as independent in the source code.
- * "filter" contains pairs of statement instances that are guaranteed
- * not to be dependent on each other based on the independent for loop,
- * assuming that no dependences carried by this loop are implied
- * by the variables in "local".
- * "local" contains the variables that are local to the loop that was
- * marked independent.
- */
-struct pet_independence {
-	isl_union_map *filter;
-	isl_union_set *local;
-};
-
-/* "loc" represents the region of the source code that is represented
- * by this scop.
- * If the scop was detected based on scop and endscop pragmas, then
- * the lines containing these pragmas are included in this region.
- * In the final result, the context describes the set of parameter values
- * for which the scop can be executed.
- * During the construction of the pet_scop, the context lives in a set space
- * where each dimension refers to an outer loop.
- * context_value describes assignments to the parameters (if any)
- * outside of the scop.
- *
- * "schedule" is the schedule of the statements in the scop.
- *
- * The n_type types define types that may be referenced from by the arrays.
- *
- * The n_implication implications describe implications on boolean filters.
- *
- * The n_independence independences describe independences implied
- * by for loops that are marked independent in the source code.
- */
-struct pet_scop {
-	pet_loc *loc;
-
-	isl_set *context;
-	isl_set *context_value;
-	isl_schedule *schedule;
-
-	int n_type;
-	struct pet_type **types;
-
-	int n_array;
-	struct pet_array **arrays;
-
-	int n_stmt;
-	struct pet_stmt **stmts;
-
-	int n_implication;
-	struct pet_implication **implications;
-
-	int n_independence;
-	struct pet_independence **independences;
-};
-typedef struct pet_scop pet_scop;
-
-/* Return a textual representation of the operator. */
-const char *pet_op_str(enum pet_op_type op);
-int pet_op_is_inc_dec(enum pet_op_type op);
-
-/* Extract a pet_scop from a C source file.
- * If function is not NULL, then the pet_scop is extracted from
- * a function with that name.
- */
-__isl_give pet_scop *pet_scop_extract_from_C_source(isl_ctx *ctx,
-	const char *filename, const char *function);
-
-/* Transform the C source file "input" by rewriting each scop
- * When autodetecting scops, at most one scop per function is rewritten.
- * The transformed C code is written to "output".
- */
-int pet_transform_C_source(isl_ctx *ctx, const char *input, FILE *output,
-	__isl_give isl_printer *(*transform)(__isl_take isl_printer *p,
-		__isl_take pet_scop *scop, void *user), void *user);
-/* Given a scop and a printer passed to a pet_transform_C_source callback,
- * print the original corresponding code to the printer.
- */
-__isl_give isl_printer *pet_scop_print_original(__isl_keep pet_scop *scop,
-	__isl_take isl_printer *p);
-
-/* Update all isl_sets and isl_maps such that they all have the same
- * parameters in the same order.
- */
-__isl_give pet_scop *pet_scop_align_params(__isl_take pet_scop *scop);
-
-/* Does "scop" contain any data dependent accesses? */
-int pet_scop_has_data_dependent_accesses(__isl_keep pet_scop *scop);
-/* Does "scop" contain any data dependent conditions? */
-int pet_scop_has_data_dependent_conditions(__isl_keep pet_scop *scop);
-/* pet_stmt_build_ast_exprs is currently limited to only handle
- * some forms of data dependent accesses.
- * If pet_scop_can_build_ast_exprs returns 1, then pet_stmt_build_ast_exprs
- * can safely be called on all statements in the scop.
- */
-int pet_scop_can_build_ast_exprs(__isl_keep pet_scop *scop);
-
-void pet_scop_dump(__isl_keep pet_scop *scop);
-__isl_null pet_scop *pet_scop_free(__isl_take pet_scop *scop);
-
-/* Return the context of "scop". */
-__isl_give isl_set *pet_scop_get_context(__isl_keep pet_scop *scop);
-/* Return the schedule of "scop". */
-__isl_give isl_schedule *pet_scop_get_schedule(__isl_keep pet_scop *scop);
-/* Return the set of all statement instances. */
-__isl_give isl_union_set *pet_scop_get_instance_set(__isl_keep pet_scop *scop);
-/* Return the potential read access relation. */
-__isl_give isl_union_map *pet_scop_get_may_reads(__isl_keep pet_scop *scop);
-/* Return the tagged potential read access relation. */
-__isl_give isl_union_map *pet_scop_get_tagged_may_reads(
-	__isl_keep pet_scop *scop);
-/* Return the potential write access relation. */
-__isl_give isl_union_map *pet_scop_get_may_writes(__isl_keep pet_scop *scop);
-/* Return the definite write access relation. */
-__isl_give isl_union_map *pet_scop_get_must_writes(__isl_keep pet_scop *scop);
-/* Return the tagged potential write access relation. */
-__isl_give isl_union_map *pet_scop_get_tagged_may_writes(
-	__isl_keep pet_scop *scop);
-/* Return the tagged definite write access relation. */
-__isl_give isl_union_map *pet_scop_get_tagged_must_writes(
-	__isl_keep pet_scop *scop);
-/* Return the definite kill access relation. */
-__isl_give isl_union_map *pet_scop_get_must_kills(__isl_keep pet_scop *scop);
-/* Return the tagged definite kill access relation. */
-__isl_give isl_union_map *pet_scop_get_tagged_must_kills(
-	__isl_keep pet_scop *scop);
-
-/* Compute a mapping from all outermost arrays (of structs) in scop
- * to their innermost members.
- */
-__isl_give isl_union_map *pet_scop_compute_outer_to_inner(
-	__isl_keep pet_scop *scop);
-/* Compute a mapping from all outermost arrays (of structs) in scop
- * to their members, including the outermost arrays themselves.
- */
-__isl_give isl_union_map *pet_scop_compute_outer_to_any(
-	__isl_keep pet_scop *scop);
-
-#if defined(__cplusplus)
-}
-#endif
-
-#endif
diff --git a/polly/lib/External/ppcg/ChangeLog b/polly/lib/External/ppcg/ChangeLog
deleted file mode 100644
index e902701903374..0000000000000
--- a/polly/lib/External/ppcg/ChangeLog
+++ /dev/null
@@ -1,29 +0,0 @@
-version: 0.07
-date: Tue Feb  7 17:23:22 CET 2017
-changes:
-	- support hybrid tiling
----
-version: 0.06
-date: Fri May  6 12:08:50 CEST 2016
-changes:
-	- use PPCG specific macro names in generated code
-	- complete transition to schedule trees
-	- maximize coincidence by default
-	- map arrays with constant index expressions to private memory
-	- optionally group chains of statements
----
-version: 0.05
-date: Fri Jan 15 09:30:23 CET 2016
-changes:
-	- fix live-out computation
-	- optionally compute schedule for C target
-	- optionally perform tiling for C target
-	- create single kernel for non-permutable subtree
----
-version: 0.04
-date: Wed Jun 17 10:52:58 CEST 2015
-changes:
-	- use schedule trees
-	- fix live-range reordering
-	- improve generation of synchronization
-	- exploit independences during dependence analysis
diff --git a/polly/lib/External/ppcg/GIT_HEAD_ID b/polly/lib/External/ppcg/GIT_HEAD_ID
deleted file mode 100644
index 642f492dd4d2a..0000000000000
--- a/polly/lib/External/ppcg/GIT_HEAD_ID
+++ /dev/null
@@ -1 +0,0 @@
-ppcg-0.07
diff --git a/polly/lib/External/ppcg/README b/polly/lib/External/ppcg/README
deleted file mode 100644
index 450786ab44311..0000000000000
--- a/polly/lib/External/ppcg/README
+++ /dev/null
@@ -1,246 +0,0 @@
-Requirements:
-
-- automake, autoconf, libtool
-	(not needed when compiling a release)
-- pkg-config (http://www.freedesktop.org/wiki/Software/pkg-config)
-	(not needed when compiling a release using the included isl and pet)
-- gmp (http://gmplib.org/)
-- libyaml (http://pyyaml.org/wiki/LibYAML)
-	(only needed if you want to compile the pet executable)
-- LLVM/clang libraries, 2.9 or higher (http://clang.llvm.org/get_started.html)
-	Unless you have some other reasons for wanting to use the svn version,
-	it is best to install the latest release (3.9).
-	For more details, see pet/README.
-
-If you are installing on Ubuntu, then you can install the following packages:
-
-automake autoconf libtool pkg-config libgmp3-dev libyaml-dev libclang-dev llvm
-
-Note that you need at least version 3.2 of libclang-dev (ubuntu raring).
-Older versions of this package did not include the required libraries.
-If you are using an older version of ubuntu, then you need to compile and
-install LLVM/clang from source.
-
-
-Preparing:
-
-Grab the latest release and extract it or get the source from
-the git repository as follows.  This process requires autoconf,
-automake, libtool and pkg-config.
-
-	git clone git://repo.or.cz/ppcg.git
-	cd ppcg
-	./get_submodules.sh
-	./autogen.sh
-
-
-Compilation:
-
-	./configure
-	make
-	make check
-
-If you have installed any of the required libraries in a non-standard
-location, then you may need to use the --with-gmp-prefix,
---with-libyaml-prefix and/or --with-clang-prefix options
-when calling "./configure".
-
-
-Using PPCG to generate CUDA or OpenCL code
-
-To convert a fragment of a C program to CUDA, insert a line containing
-
-	#pragma scop
-
-before the fragment and add a line containing
-
-	#pragma endscop
-
-after the fragment.  To generate CUDA code run
-	
-	ppcg --target=cuda file.c
-
-where file.c is the file containing the fragment.  The generated
-code is stored in file_host.cu and file_kernel.cu.
-
-To generate OpenCL code run
-
-	ppcg --target=opencl file.c
-
-where file.c is the file containing the fragment.  The generated code
-is stored in file_host.c and file_kernel.cl.
-
-
-Specifying tile, grid and block sizes
-
-The iterations space tile size, grid size and block size can
-be specified using the --sizes option.  The argument is a union map
-in isl notation mapping kernels identified by their sequence number
-in a "kernel" space to singleton sets in the "tile", "grid" and "block"
-spaces.  The sizes are specified outermost to innermost.
-
-The dimension of the "tile" space indicates the (maximal) number of loop
-dimensions to tile.  The elements of the single integer tuple
-specify the tile sizes in each dimension.
-In case of hybrid tiling, the first element is half the size of
-the tile in the time (sequential) dimension.  The second element
-specifies the number of elements in the base of the hexagon.
-The remaining elements specify the tile sizes in the remaining space
-dimensions.
-
-The dimension of the "grid" space indicates the (maximal) number of block
-dimensions in the grid.  The elements of the single integer tuple
-specify the number of blocks in each dimension.
-
-The dimension of the "block" space indicates the (maximal) number of thread
-dimensions in the grid.  The elements of the single integer tuple
-specify the number of threads in each dimension.
-
-For example,
-
-    { kernel[0] -> tile[64,64]; kernel[i] -> block[16] : i != 4 }
-
-specifies that in kernel 0, two loops should be tiled with a tile
-size of 64 in both dimensions and that all kernels except kernel 4
-should be run using a block of 16 threads.
-
-Since PPCG performs some scheduling, it can be difficult to predict
-what exactly will end up in a kernel.  If you want to specify
-tile, grid or block sizes, you may want to run PPCG first with the defaults,
-examine the kernels and then run PPCG again with the desired sizes.
-Instead of examining the kernels, you can also specify the option
---dump-sizes on the first run to obtain the effectively used default sizes.
-
-
-Compiling the generated CUDA code with nvcc
-
-To get optimal performance from nvcc, it is important to choose --arch
-according to your target GPU.  Specifically, use the flag "--arch sm_20"
-for fermi, "--arch sm_30" for GK10x Kepler and "--arch sm_35" for
-GK110 Kepler.  We discourage the use of older cards as we have seen
-correctness issues with compilation for older architectures.
-Note that in the absence of any --arch flag, nvcc defaults to
-"--arch sm_13". This will not only be slower, but can also cause
-correctness issues.
-If you want to obtain results that are identical to those obtained
-by the original code, then you may need to disable some optimizations
-by passing the "--fmad=false" option.
-
-
-Compiling the generated OpenCL code with gcc
-
-To compile the host code you need to link against the file
-ocl_utilities.c which contains utility functions used by the generated
-OpenCL host code.  To compile the host code with gcc, run
-
-  gcc -std=c99 file_host.c ocl_utilities.c -lOpenCL
-
-Note that we have experienced the generated OpenCL code freezing
-on some inputs (e.g., the PolyBench symm benchmark) when using
-at least some version of the Nvidia OpenCL library, while the
-corresponding CUDA code runs fine.
-We have experienced no such freezes when using AMD, ARM or Intel
-OpenCL libraries.
-
-By default, the compiled executable will need the _kernel.cl file at
-run time.  Alternatively, the option --opencl-embed-kernel-code may be
-given to place the kernel code in a string literal.  The kernel code is
-then compiled into the host binary, such that the _kernel.cl file is no
-longer needed at run time.  Any kernel include files, in particular
-those supplied using --opencl-include-file, will still be required at
-run time.
-
-
-Function calls
-
-Function calls inside the analyzed fragment are reproduced
-in the CUDA or OpenCL code, but for now it is left to the user
-to make sure that the functions that are being called are
-available from the generated kernels.
-
-In the case of OpenCL code, the --opencl-include-file option
-may be used to specify one or more files to be #include'd
-from the generated code.  These files may then contain
-the definitions of the functions being called from the
-program fragment.  If the pathnames of the included files
-are relative to the current directory, then you may need
-to additionally specify the --opencl-compiler-options=-I.
-to make sure that the files can be found by the OpenCL compiler.
-The included files may contain definitions of types used by the
-generated kernels.  By default, PPCG generates definitions for
-types as needed, but these definitions may collide with those in
-the included files, as PPCG does not consider the contents of the
-included files.  The --no-opencl-print-kernel-types will prevent
-PPCG from generating type definitions.
-
-
-GNU extensions
-
-By default, PPCG may print out macro definitions that involve
-GNU extensions such as __typeof__ and statement expressions.
-Some compilers may not support these extensions.
-In particular, OpenCL 1.2 beignet 1.1.1 (git-6de6918)
-has been reported not to support __typeof__.
-The use of these extensions can be turned off with the
---no-allow-gnu-extensions option.
-
-
-Processing PolyBench
-
-When processing a PolyBench/C 3.2 benchmark, you should always specify
--DPOLYBENCH_USE_C99_PROTO on the ppcg command line.  Otherwise, the source
-files are inconsistent, having fixed size arrays but parametrically
-bounded loops iterating over them.
-However, you should not specify this define when compiling
-the PPCG generated code using nvcc since CUDA does not support VLAs.
-
-
-CUDA and function overloading
-
-While CUDA supports function overloading based on the arguments types,
-no such function overloading exists in the input language C.  Since PPCG
-simply prints out the same function name as in the original code, this
-may result in a different function being called based on the types
-of the arguments.  For example, if the original code contains a call
-to the function sqrt() with a float argument, then the argument will
-be promoted to a double and the sqrt() function will be called.
-In the transformed (CUDA) code, however, overloading will cause the
-function sqrtf() to be called.  Until this issue has been resolved in PPCG,
-we recommend that users either explicitly call the function sqrtf() or
-explicitly cast the argument to double in the input code.
-
-
-Contact
-
-For bug reports, feature requests and questions,
-contact http://groups.google.com/group/isl-development
-
-Whenever you report a bug, please mention the exact version of PPCG
-that you are using (output of "./ppcg --version").  If you are unable
-to compile PPCG, then report the git version (output of "git describe")
-or the version number included in the name of the tarball.
-
-
-Citing PPCG
-
-If you use PPCG for your research, you are invited to cite
-the following paper.
-
-@article{Verdoolaege2013PPCG,
-    author = {Verdoolaege, Sven and Juega, Juan Carlos and Cohen, Albert and
-		G\'{o}mez, Jos{\'e} Ignacio and Tenllado, Christian and
-		Catthoor, Francky},
-    title = {Polyhedral parallel code generation for CUDA},
-    journal = {ACM Trans. Archit. Code Optim.},
-    issue_date = {January 2013},
-    volume = {9},
-    number = {4},
-    month = jan,
-    year = {2013},
-    issn = {1544-3566},
-    pages = {54:1--54:23},
-    doi = {10.1145/2400682.2400713},
-    acmid = {2400713},
-    publisher = {ACM},
-    address = {New York, NY, USA},
-}
diff --git a/polly/lib/External/ppcg/cpu.c b/polly/lib/External/ppcg/cpu.c
deleted file mode 100644
index 7ca6f1a57e83f..0000000000000
--- a/polly/lib/External/ppcg/cpu.c
+++ /dev/null
@@ -1,802 +0,0 @@
-/*
- * Copyright 2012 INRIA Paris-Rocquencourt
- * Copyright 2012 Ecole Normale Superieure
- *
- * Use of this software is governed by the MIT license
- *
- * Written by Tobias Grosser, INRIA Paris-Rocquencourt,
- * Domaine de Voluceau, Rocquenqourt, B.P. 105,
- * 78153 Le Chesnay Cedex France
- * and Sven Verdoolaege,
- * Ecole Normale Superieure, 45 rue d'Ulm, 75230 Paris, France
- */
-
-#include <limits.h>
-#include <stdio.h>
-#include <string.h>
-
-#include <isl/aff.h>
-#include <isl/ctx.h>
-#include <isl/flow.h>
-#include <isl/map.h>
-#include <isl/ast_build.h>
-#include <isl/schedule.h>
-#include <isl/schedule_node.h>
-#include <pet.h>
-
-#include "ppcg.h"
-#include "ppcg_options.h"
-#include "cpu.h"
-#include "print.h"
-#include "schedule.h"
-#include "util.h"
-
-/* Representation of a statement inside a generated AST.
- *
- * "stmt" refers to the original statement.
- * "ref2expr" maps the reference identifier of each access in
- * the statement to an AST expression that should be printed
- * at the place of the access.
- */
-struct ppcg_stmt {
-	struct pet_stmt *stmt;
-
-	isl_id_to_ast_expr *ref2expr;
-};
-
-static void ppcg_stmt_free(void *user)
-{
-	struct ppcg_stmt *stmt = user;
-
-	if (!stmt)
-		return;
-
-	isl_id_to_ast_expr_free(stmt->ref2expr);
-
-	free(stmt);
-}
-
-/* Derive the output file name from the input file name.
- * 'input' is the entire path of the input file. The output
- * is the file name plus the additional extension.
- *
- * We will basically replace everything after the last point
- * with '.ppcg.c'. This means file.c becomes file.ppcg.c
- */
-static FILE *get_output_file(const char *input, const char *output)
-{
-	char name[PATH_MAX];
-	const char *ext;
-	const char ppcg_marker[] = ".ppcg";
-	int len;
-	FILE *file;
-
-	len = ppcg_extract_base_name(name, input);
-
-	strcpy(name + len, ppcg_marker);
-	ext = strrchr(input, '.');
-	strcpy(name + len + sizeof(ppcg_marker) - 1, ext ? ext : ".c");
-
-	if (!output)
-		output = name;
-
-	file = fopen(output, "w");
-	if (!file) {
-		fprintf(stderr, "Unable to open '%s' for writing\n", output);
-		return NULL;
-	}
-
-	return file;
-}
-
-/* Data used to annotate for nodes in the ast.
- */
-struct ast_node_userinfo {
-	/* The for node is an openmp parallel for node. */
-	int is_openmp;
-};
-
-/* Information used while building the ast.
- */
-struct ast_build_userinfo {
-	/* The current ppcg scop. */
-	struct ppcg_scop *scop;
-
-	/* Are we currently in a parallel for loop? */
-	int in_parallel_for;
-};
-
-/* Check if the current scheduling dimension is parallel.
- *
- * We check for parallelism by verifying that the loop does not carry any
- * dependences.
- * If the live_range_reordering option is set, then this currently
- * includes the order dependences.  In principle, non-zero order dependences
- * could be allowed, but this would require privatization and/or expansion.
- *
- * Parallelism test: if the distance is zero in all outer dimensions, then it
- * has to be zero in the current dimension as well.
- * Implementation: first, translate dependences into time space, then force
- * outer dimensions to be equal.  If the distance is zero in the current
- * dimension, then the loop is parallel.
- * The distance is zero in the current dimension if it is a subset of a map
- * with equal values for the current dimension.
- */
-static int ast_schedule_dim_is_parallel(__isl_keep isl_ast_build *build,
-	struct ppcg_scop *scop)
-{
-	isl_union_map *schedule, *deps;
-	isl_map *schedule_deps, *test;
-	isl_space *schedule_space;
-	unsigned i, dimension, is_parallel;
-
-	schedule = isl_ast_build_get_schedule(build);
-	schedule_space = isl_ast_build_get_schedule_space(build);
-
-	dimension = isl_space_dim(schedule_space, isl_dim_out) - 1;
-
-	deps = isl_union_map_copy(scop->dep_flow);
-	deps = isl_union_map_union(deps, isl_union_map_copy(scop->dep_false));
-	if (scop->options->live_range_reordering) {
-		isl_union_map *order = isl_union_map_copy(scop->dep_order);
-		deps = isl_union_map_union(deps, order);
-	}
-	deps = isl_union_map_apply_range(deps, isl_union_map_copy(schedule));
-	deps = isl_union_map_apply_domain(deps, schedule);
-
-	if (isl_union_map_is_empty(deps)) {
-		isl_union_map_free(deps);
-		isl_space_free(schedule_space);
-		return 1;
-	}
-
-	schedule_deps = isl_map_from_union_map(deps);
-
-	for (i = 0; i < dimension; i++)
-		schedule_deps = isl_map_equate(schedule_deps, isl_dim_out, i,
-					       isl_dim_in, i);
-
-	test = isl_map_universe(isl_map_get_space(schedule_deps));
-	test = isl_map_equate(test, isl_dim_out, dimension, isl_dim_in,
-			      dimension);
-	is_parallel = isl_map_is_subset(schedule_deps, test);
-
-	isl_space_free(schedule_space);
-	isl_map_free(test);
-	isl_map_free(schedule_deps);
-
-	return is_parallel;
-}
-
-/* Mark a for node openmp parallel, if it is the outermost parallel for node.
- */
-static void mark_openmp_parallel(__isl_keep isl_ast_build *build,
-	struct ast_build_userinfo *build_info,
-	struct ast_node_userinfo *node_info)
-{
-	if (build_info->in_parallel_for)
-		return;
-
-	if (ast_schedule_dim_is_parallel(build, build_info->scop)) {
-		build_info->in_parallel_for = 1;
-		node_info->is_openmp = 1;
-	}
-}
-
-/* Allocate an ast_node_info structure and initialize it with default values.
- */
-static struct ast_node_userinfo *allocate_ast_node_userinfo()
-{
-	struct ast_node_userinfo *node_info;
-	node_info = (struct ast_node_userinfo *)
-		malloc(sizeof(struct ast_node_userinfo));
-	node_info->is_openmp = 0;
-	return node_info;
-}
-
-/* Free an ast_node_info structure.
- */
-static void free_ast_node_userinfo(void *ptr)
-{
-	struct ast_node_userinfo *info;
-	info = (struct ast_node_userinfo *) ptr;
-	free(info);
-}
-
-/* This method is executed before the construction of a for node. It creates
- * an isl_id that is used to annotate the subsequently generated ast for nodes.
- *
- * In this function we also run the following analyses:
- *
- * 	- Detection of openmp parallel loops
- */
-static __isl_give isl_id *ast_build_before_for(
-	__isl_keep isl_ast_build *build, void *user)
-{
-	isl_id *id;
-	struct ast_build_userinfo *build_info;
-	struct ast_node_userinfo *node_info;
-
-	build_info = (struct ast_build_userinfo *) user;
-	node_info = allocate_ast_node_userinfo();
-	id = isl_id_alloc(isl_ast_build_get_ctx(build), "", node_info);
-	id = isl_id_set_free_user(id, free_ast_node_userinfo);
-
-	mark_openmp_parallel(build, build_info, node_info);
-
-	return id;
-}
-
-/* This method is executed after the construction of a for node.
- *
- * It performs the following actions:
- *
- * 	- Reset the 'in_parallel_for' flag, as soon as we leave a for node,
- * 	  that is marked as openmp parallel.
- *
- */
-static __isl_give isl_ast_node *ast_build_after_for(
-	__isl_take isl_ast_node *node, __isl_keep isl_ast_build *build,
-	void *user)
-{
-	isl_id *id;
-	struct ast_build_userinfo *build_info;
-	struct ast_node_userinfo *info;
-
-	id = isl_ast_node_get_annotation(node);
-	info = isl_id_get_user(id);
-
-	if (info && info->is_openmp) {
-		build_info = (struct ast_build_userinfo *) user;
-		build_info->in_parallel_for = 0;
-	}
-
-	isl_id_free(id);
-
-	return node;
-}
-
-/* Find the element in scop->stmts that has the given "id".
- */
-static struct pet_stmt *find_stmt(struct ppcg_scop *scop, __isl_keep isl_id *id)
-{
-	int i;
-
-	for (i = 0; i < scop->pet->n_stmt; ++i) {
-		struct pet_stmt *stmt = scop->pet->stmts[i];
-		isl_id *id_i;
-
-		id_i = isl_set_get_tuple_id(stmt->domain);
-		isl_id_free(id_i);
-
-		if (id_i == id)
-			return stmt;
-	}
-
-	isl_die(isl_id_get_ctx(id), isl_error_internal,
-		"statement not found", return NULL);
-}
-
-/* Print a user statement in the generated AST.
- * The ppcg_stmt has been attached to the node in at_each_domain.
- */
-static __isl_give isl_printer *print_user(__isl_take isl_printer *p,
-	__isl_take isl_ast_print_options *print_options,
-	__isl_keep isl_ast_node *node, void *user)
-{
-	struct ppcg_stmt *stmt;
-	isl_id *id;
-
-	id = isl_ast_node_get_annotation(node);
-	stmt = isl_id_get_user(id);
-	isl_id_free(id);
-
-	p = pet_stmt_print_body(stmt->stmt, p, stmt->ref2expr);
-
-	isl_ast_print_options_free(print_options);
-
-	return p;
-}
-
-
-/* Print a for loop node as an openmp parallel loop.
- *
- * To print an openmp parallel loop we print a normal for loop, but add
- * "#pragma openmp parallel for" in front.
- *
- * Variables that are declared within the body of this for loop are
- * automatically openmp 'private'. Iterators declared outside of the
- * for loop are automatically openmp 'shared'. As ppcg declares all iterators
- * at the position where they are assigned, there is no need to explicitly mark
- * variables. Their automatically assigned type is already correct.
- *
- * This function only generates valid OpenMP code, if the ast was generated
- * with the 'atomic-bounds' option enabled.
- *
- */
-static __isl_give isl_printer *print_for_with_openmp(
-	__isl_keep isl_ast_node *node, __isl_take isl_printer *p,
-	__isl_take isl_ast_print_options *print_options)
-{
-	p = isl_printer_start_line(p);
-	p = isl_printer_print_str(p, "#pragma omp parallel for");
-	p = isl_printer_end_line(p);
-
-	p = isl_ast_node_for_print(node, p, print_options);
-
-	return p;
-}
-
-/* Print a for node.
- *
- * Depending on how the node is annotated, we either print a normal
- * for node or an openmp parallel for node.
- */
-static __isl_give isl_printer *print_for(__isl_take isl_printer *p,
-	__isl_take isl_ast_print_options *print_options,
-	__isl_keep isl_ast_node *node, void *user)
-{
-	isl_id *id;
-	int openmp;
-
-	openmp = 0;
-	id = isl_ast_node_get_annotation(node);
-
-	if (id) {
-		struct ast_node_userinfo *info;
-
-		info = (struct ast_node_userinfo *) isl_id_get_user(id);
-		if (info && info->is_openmp)
-			openmp = 1;
-	}
-
-	if (openmp)
-		p = print_for_with_openmp(node, p, print_options);
-	else
-		p = isl_ast_node_for_print(node, p, print_options);
-
-	isl_id_free(id);
-
-	return p;
-}
-
-/* Index transformation callback for pet_stmt_build_ast_exprs.
- *
- * "index" expresses the array indices in terms of statement iterators
- * "iterator_map" expresses the statement iterators in terms of
- * AST loop iterators.
- *
- * The result expresses the array indices in terms of
- * AST loop iterators.
- */
-static __isl_give isl_multi_pw_aff *pullback_index(
-	__isl_take isl_multi_pw_aff *index, __isl_keep isl_id *id, void *user)
-{
-	isl_pw_multi_aff *iterator_map = user;
-
-	iterator_map = isl_pw_multi_aff_copy(iterator_map);
-	return isl_multi_pw_aff_pullback_pw_multi_aff(index, iterator_map);
-}
-
-/* Transform the accesses in the statement associated to the domain
- * called by "node" to refer to the AST loop iterators, construct
- * corresponding AST expressions using "build",
- * collect them in a ppcg_stmt and annotate the node with the ppcg_stmt.
- */
-static __isl_give isl_ast_node *at_each_domain(__isl_take isl_ast_node *node,
-	__isl_keep isl_ast_build *build, void *user)
-{
-	struct ppcg_scop *scop = user;
-	isl_ast_expr *expr, *arg;
-	isl_ctx *ctx;
-	isl_id *id;
-	isl_map *map;
-	isl_pw_multi_aff *iterator_map;
-	struct ppcg_stmt *stmt;
-
-	ctx = isl_ast_node_get_ctx(node);
-	stmt = isl_calloc_type(ctx, struct ppcg_stmt);
-	if (!stmt)
-		goto error;
-
-	expr = isl_ast_node_user_get_expr(node);
-	arg = isl_ast_expr_get_op_arg(expr, 0);
-	isl_ast_expr_free(expr);
-	id = isl_ast_expr_get_id(arg);
-	isl_ast_expr_free(arg);
-	stmt->stmt = find_stmt(scop, id);
-	isl_id_free(id);
-	if (!stmt->stmt)
-		goto error;
-
-	map = isl_map_from_union_map(isl_ast_build_get_schedule(build));
-	map = isl_map_reverse(map);
-	iterator_map = isl_pw_multi_aff_from_map(map);
-	stmt->ref2expr = pet_stmt_build_ast_exprs(stmt->stmt, build,
-				    &pullback_index, iterator_map, NULL, NULL);
-	isl_pw_multi_aff_free(iterator_map);
-
-	id = isl_id_alloc(isl_ast_node_get_ctx(node), NULL, stmt);
-	id = isl_id_set_free_user(id, &ppcg_stmt_free);
-	return isl_ast_node_set_annotation(node, id);
-error:
-	ppcg_stmt_free(stmt);
-	return isl_ast_node_free(node);
-}
-
-/* Set *depth (initialized to 0 by the caller) to the maximum
- * of the schedule depths of the leaf nodes for which this function is called.
- */
-static isl_bool update_depth(__isl_keep isl_schedule_node *node, void *user)
-{
-	int *depth = user;
-	int node_depth;
-
-	if (isl_schedule_node_get_type(node) != isl_schedule_node_leaf)
-		return isl_bool_true;
-	node_depth = isl_schedule_node_get_schedule_depth(node);
-	if (node_depth > *depth)
-		*depth = node_depth;
-
-	return isl_bool_false;
-}
-
-/* This function is called for each node in a CPU AST.
- * In case of a user node, print the macro definitions required
- * for printing the AST expressions in the annotation, if any.
- * For other nodes, return true such that descendants are also
- * visited.
- *
- * In particular, print the macro definitions needed for the substitutions
- * of the original user statements.
- */
-static isl_bool at_node(__isl_keep isl_ast_node *node, void *user)
-{
-	struct ppcg_stmt *stmt;
-	isl_id *id;
-	isl_printer **p = user;
-
-	if (isl_ast_node_get_type(node) != isl_ast_node_user)
-		return isl_bool_true;
-
-	id = isl_ast_node_get_annotation(node);
-	stmt = isl_id_get_user(id);
-	isl_id_free(id);
-
-	if (!stmt)
-		return isl_bool_error;
-
-	*p = ppcg_print_body_macros(*p, stmt->ref2expr);
-	if (!*p)
-		return isl_bool_error;
-
-	return isl_bool_false;
-}
-
-/* Print the required macros for the CPU AST "node" to "p",
- * including those needed for the user statements inside the AST.
- */
-static __isl_give isl_printer *cpu_print_macros(__isl_take isl_printer *p,
-	__isl_keep isl_ast_node *node)
-{
-	if (isl_ast_node_foreach_descendant_top_down(node, &at_node, &p) < 0)
-		return isl_printer_free(p);
-	p = ppcg_print_macros(p, node);
-	return p;
-}
-
-/* Code generate the scop 'scop' using "schedule"
- * and print the corresponding C code to 'p'.
- */
-static __isl_give isl_printer *print_scop(struct ppcg_scop *scop,
-	__isl_take isl_schedule *schedule, __isl_take isl_printer *p,
-	struct ppcg_options *options)
-{
-	isl_ctx *ctx = isl_printer_get_ctx(p);
-	isl_ast_build *build;
-	isl_ast_print_options *print_options;
-	isl_ast_node *tree;
-	isl_id_list *iterators;
-	struct ast_build_userinfo build_info;
-	int depth;
-
-	depth = 0;
-	if (isl_schedule_foreach_schedule_node_top_down(schedule, &update_depth,
-						&depth) < 0)
-		goto error;
-
-	build = isl_ast_build_alloc(ctx);
-	iterators = ppcg_scop_generate_names(scop, depth, "c");
-	build = isl_ast_build_set_iterators(build, iterators);
-	build = isl_ast_build_set_at_each_domain(build, &at_each_domain, scop);
-
-	if (options->openmp) {
-		build_info.scop = scop;
-		build_info.in_parallel_for = 0;
-
-		build = isl_ast_build_set_before_each_for(build,
-							&ast_build_before_for,
-							&build_info);
-		build = isl_ast_build_set_after_each_for(build,
-							&ast_build_after_for,
-							&build_info);
-	}
-
-	tree = isl_ast_build_node_from_schedule(build, schedule);
-	isl_ast_build_free(build);
-
-	print_options = isl_ast_print_options_alloc(ctx);
-	print_options = isl_ast_print_options_set_print_user(print_options,
-							&print_user, NULL);
-
-	print_options = isl_ast_print_options_set_print_for(print_options,
-							&print_for, NULL);
-
-	p = cpu_print_macros(p, tree);
-	p = isl_ast_node_print(tree, p, print_options);
-
-	isl_ast_node_free(tree);
-
-	return p;
-error:
-	isl_schedule_free(schedule);
-	isl_printer_free(p);
-	return NULL;
-}
-
-/* Tile the band node "node" with tile sizes "sizes" and
- * mark all members of the resulting tile node as "atomic".
- */
-static __isl_give isl_schedule_node *tile(__isl_take isl_schedule_node *node,
-	__isl_take isl_multi_val *sizes)
-{
-	node = isl_schedule_node_band_tile(node, sizes);
-	node = ppcg_set_schedule_node_type(node, isl_ast_loop_atomic);
-
-	return node;
-}
-
-/* Tile "node", if it is a band node with at least 2 members.
- * The tile sizes are set from the "tile_size" option.
- */
-static __isl_give isl_schedule_node *tile_band(
-	__isl_take isl_schedule_node *node, void *user)
-{
-	struct ppcg_scop *scop = user;
-	int n;
-	isl_space *space;
-	isl_multi_val *sizes;
-
-	if (isl_schedule_node_get_type(node) != isl_schedule_node_band)
-		return node;
-
-	n = isl_schedule_node_band_n_member(node);
-	if (n <= 1)
-		return node;
-
-	space = isl_schedule_node_band_get_space(node);
-	sizes = ppcg_multi_val_from_int(space, scop->options->tile_size);
-
-	return tile(node, sizes);
-}
-
-/* Construct schedule constraints from the dependences in ps
- * for the purpose of computing a schedule for a CPU.
- *
- * The proximity constraints are set to the flow dependences.
- *
- * If live-range reordering is allowed then the conditional validity
- * constraints are set to the order dependences with the flow dependences
- * as condition.  That is, a live-range (flow dependence) will be either
- * local to an iteration of a band or all adjacent order dependences
- * will be respected by the band.
- * The validity constraints are set to the union of the flow dependences
- * and the forced dependences, while the coincidence constraints
- * are set to the union of the flow dependences, the forced dependences and
- * the order dependences.
- *
- * If live-range reordering is not allowed, then both the validity
- * and the coincidence constraints are set to the union of the flow
- * dependences and the false dependences.
- *
- * Note that the coincidence constraints are only set when the "openmp"
- * options is set.  Even though the way openmp pragmas are introduced
- * does not rely on the coincident property of the schedule band members,
- * the coincidence constraints do affect the way the schedule is constructed,
- * such that more schedule dimensions should be detected as parallel
- * by ast_schedule_dim_is_parallel.
- * Since the order dependences are also taken into account by
- * ast_schedule_dim_is_parallel, they are also added to
- * the coincidence constraints.  If the openmp handling learns
- * how to privatize some memory, then the corresponding order
- * dependences can be removed from the coincidence constraints.
- */
-static __isl_give isl_schedule_constraints *construct_cpu_schedule_constraints(
-	struct ppcg_scop *ps)
-{
-	isl_schedule_constraints *sc;
-	isl_union_map *validity, *coincidence;
-
-	sc = isl_schedule_constraints_on_domain(isl_union_set_copy(ps->domain));
-	if (ps->options->live_range_reordering) {
-		sc = isl_schedule_constraints_set_conditional_validity(sc,
-				isl_union_map_copy(ps->tagged_dep_flow),
-				isl_union_map_copy(ps->tagged_dep_order));
-		validity = isl_union_map_copy(ps->dep_flow);
-		validity = isl_union_map_union(validity,
-				isl_union_map_copy(ps->dep_forced));
-		if (ps->options->openmp) {
-			coincidence = isl_union_map_copy(validity);
-			coincidence = isl_union_map_union(coincidence,
-					isl_union_map_copy(ps->dep_order));
-		}
-	} else {
-		validity = isl_union_map_copy(ps->dep_flow);
-		validity = isl_union_map_union(validity,
-				isl_union_map_copy(ps->dep_false));
-		if (ps->options->openmp)
-			coincidence = isl_union_map_copy(validity);
-	}
-	if (ps->options->openmp)
-		sc = isl_schedule_constraints_set_coincidence(sc, coincidence);
-	sc = isl_schedule_constraints_set_validity(sc, validity);
-	sc = isl_schedule_constraints_set_proximity(sc,
-					isl_union_map_copy(ps->dep_flow));
-
-	return sc;
-}
-
-/* Compute a schedule for the scop "ps".
- *
- * First derive the appropriate schedule constraints from the dependences
- * in "ps" and then compute a schedule from those schedule constraints,
- * possibly grouping statement instances based on the input schedule.
- */
-static __isl_give isl_schedule *compute_cpu_schedule(struct ppcg_scop *ps)
-{
-	isl_schedule_constraints *sc;
-	isl_schedule *schedule;
-
-	if (!ps)
-		return NULL;
-
-	sc = construct_cpu_schedule_constraints(ps);
-
-	if (ps->options->debug->dump_schedule_constraints)
-		isl_schedule_constraints_dump(sc);
-	schedule = ppcg_compute_schedule(sc, ps->schedule, ps->options);
-
-	return schedule;
-}
-
-/* Compute a new schedule to the scop "ps" if the reschedule option is set.
- * Otherwise, return a copy of the original schedule.
- */
-static __isl_give isl_schedule *optionally_compute_schedule(void *user)
-{
-	struct ppcg_scop *ps = user;
-
-	if (!ps)
-		return NULL;
-	if (!ps->options->reschedule)
-		return isl_schedule_copy(ps->schedule);
-	return compute_cpu_schedule(ps);
-}
-
-/* Compute a schedule based on the dependences in "ps" and
- * tile it if requested by the user.
- */
-static __isl_give isl_schedule *get_schedule(struct ppcg_scop *ps,
-	struct ppcg_options *options)
-{
-	isl_ctx *ctx;
-	isl_schedule *schedule;
-
-	if (!ps)
-		return NULL;
-
-	ctx = isl_union_set_get_ctx(ps->domain);
-	schedule = ppcg_get_schedule(ctx, options,
-				    &optionally_compute_schedule, ps);
-	if (ps->options->tile)
-		schedule = isl_schedule_map_schedule_node_bottom_up(schedule,
-							&tile_band, ps);
-
-	return schedule;
-}
-
-/* Generate CPU code for the scop "ps" using "schedule" and
- * print the corresponding C code to "p", including variable declarations.
- */
-static __isl_give isl_printer *print_cpu_with_schedule(
-	__isl_take isl_printer *p, struct ppcg_scop *ps,
-	__isl_take isl_schedule *schedule, struct ppcg_options *options)
-{
-	int hidden;
-	isl_set *context;
-
-	p = isl_printer_start_line(p);
-	p = isl_printer_print_str(p, "/* ppcg generated CPU code */");
-	p = isl_printer_end_line(p);
-
-	p = isl_printer_start_line(p);
-	p = isl_printer_end_line(p);
-
-	p = ppcg_set_macro_names(p);
-	p = ppcg_print_exposed_declarations(p, ps);
-	hidden = ppcg_scop_any_hidden_declarations(ps);
-	if (hidden) {
-		p = ppcg_start_block(p);
-		p = ppcg_print_hidden_declarations(p, ps);
-	}
-
-	context = isl_set_copy(ps->context);
-	context = isl_set_from_params(context);
-	schedule = isl_schedule_insert_context(schedule, context);
-	if (options->debug->dump_final_schedule)
-		isl_schedule_dump(schedule);
-	p = print_scop(ps, schedule, p, options);
-	if (hidden)
-		p = ppcg_end_block(p);
-
-	return p;
-}
-
-/* Generate CPU code for the scop "ps" and print the corresponding C code
- * to "p", including variable declarations.
- */
-__isl_give isl_printer *print_cpu(__isl_take isl_printer *p,
-	struct ppcg_scop *ps, struct ppcg_options *options)
-{
-	isl_schedule *schedule;
-
-	schedule = isl_schedule_copy(ps->schedule);
-	return print_cpu_with_schedule(p, ps, schedule, options);
-}
-
-/* Generate CPU code for "scop" and print it to "p".
- *
- * First obtain a schedule for "scop" and then print code for "scop"
- * using that schedule.
- */
-static __isl_give isl_printer *generate(__isl_take isl_printer *p,
-	struct ppcg_scop *scop, struct ppcg_options *options)
-{
-	isl_schedule *schedule;
-
-	schedule = get_schedule(scop, options);
-
-	return print_cpu_with_schedule(p, scop, schedule, options);
-}
-
-/* Wrapper around generate for use as a ppcg_transform callback.
- */
-static __isl_give isl_printer *print_cpu_wrap(__isl_take isl_printer *p,
-	struct ppcg_scop *scop, void *user)
-{
-	struct ppcg_options *options = user;
-
-	return generate(p, scop, options);
-}
-
-/* Transform the code in the file called "input" by replacing
- * all scops by corresponding CPU code and write the results to a file
- * called "output".
- */
-int generate_cpu(isl_ctx *ctx, struct ppcg_options *options,
-	const char *input, const char *output)
-{
-	FILE *output_file;
-	int r;
-
-	output_file = get_output_file(input, output);
-	if (!output_file)
-		return -1;
-
-	r = ppcg_transform(ctx, input, output_file, options,
-					&print_cpu_wrap, options);
-
-	fclose(output_file);
-
-	return r;
-}
diff --git a/polly/lib/External/ppcg/cpu.h b/polly/lib/External/ppcg/cpu.h
deleted file mode 100644
index 6caf993aad5c2..0000000000000
--- a/polly/lib/External/ppcg/cpu.h
+++ /dev/null
@@ -1,15 +0,0 @@
-#ifndef _CPU_H
-#define _CPU_H
-
-#include <isl/ctx.h>
-
-#include "ppcg.h"
-
-struct ppcg_options;
-
-__isl_give isl_printer *print_cpu(__isl_take isl_printer *p,
-	struct ppcg_scop *ps, struct ppcg_options *options);
-int generate_cpu(isl_ctx *ctx, struct ppcg_options *options,
-	const char *input, const char *output);
-
-#endif
diff --git a/polly/lib/External/ppcg/cuda.c b/polly/lib/External/ppcg/cuda.c
deleted file mode 100644
index 380d6526bb1fd..0000000000000
--- a/polly/lib/External/ppcg/cuda.c
+++ /dev/null
@@ -1,730 +0,0 @@
-/*
- * Copyright 2012      Ecole Normale Superieure
- *
- * Use of this software is governed by the MIT license
- *
- * Written by Sven Verdoolaege,
- * Ecole Normale Superieure, 45 rue d’Ulm, 75230 Paris, France
- */
-
-#include <isl/aff.h>
-#include <isl/ast.h>
-
-#include "cuda_common.h"
-#include "cuda.h"
-#include "gpu.h"
-#include "gpu_print.h"
-#include "print.h"
-#include "util.h"
-
-static __isl_give isl_printer *print_cuda_macros(__isl_take isl_printer *p)
-{
-	const char *macros =
-		"#define cudaCheckReturn(ret) \\\n"
-		"  do { \\\n"
-		"    cudaError_t cudaCheckReturn_e = (ret); \\\n"
-		"    if (cudaCheckReturn_e != cudaSuccess) { \\\n"
-		"      fprintf(stderr, \"CUDA error: %s\\n\", "
-		"cudaGetErrorString(cudaCheckReturn_e)); \\\n"
-		"      fflush(stderr); \\\n"
-		"    } \\\n"
-		"    assert(cudaCheckReturn_e == cudaSuccess); \\\n"
-		"  } while(0)\n"
-		"#define cudaCheckKernel() \\\n"
-		"  do { \\\n"
-		"    cudaCheckReturn(cudaGetLastError()); \\\n"
-		"  } while(0)\n\n";
-
-	p = isl_printer_print_str(p, macros);
-	return p;
-}
-
-/* Print a declaration for the device array corresponding to "array" on "p".
- */
-static __isl_give isl_printer *declare_device_array(__isl_take isl_printer *p,
-	struct gpu_array_info *array)
-{
-	int i;
-
-	p = isl_printer_start_line(p);
-	p = isl_printer_print_str(p, array->type);
-	p = isl_printer_print_str(p, " ");
-	if (!array->linearize && array->n_index > 1)
-		p = isl_printer_print_str(p, "(");
-	p = isl_printer_print_str(p, "*dev_");
-	p = isl_printer_print_str(p, array->name);
-	if (!array->linearize && array->n_index > 1) {
-		p = isl_printer_print_str(p, ")");
-		for (i = 1; i < array->n_index; i++) {
-			isl_ast_expr *bound;
-			bound = isl_ast_expr_get_op_arg(array->bound_expr,
-							1 + i);
-			p = isl_printer_print_str(p, "[");
-			p = isl_printer_print_ast_expr(p, bound);
-			p = isl_printer_print_str(p, "]");
-			isl_ast_expr_free(bound);
-		}
-	}
-	p = isl_printer_print_str(p, ";");
-	p = isl_printer_end_line(p);
-
-	return p;
-}
-
-static __isl_give isl_printer *declare_device_arrays(__isl_take isl_printer *p,
-	struct gpu_prog *prog)
-{
-	int i;
-
-	for (i = 0; i < prog->n_array; ++i) {
-		if (!gpu_array_requires_device_allocation(&prog->array[i]))
-			continue;
-
-		p = declare_device_array(p, &prog->array[i]);
-	}
-	p = isl_printer_start_line(p);
-	p = isl_printer_end_line(p);
-	return p;
-}
-
-static __isl_give isl_printer *allocate_device_arrays(
-	__isl_take isl_printer *p, struct gpu_prog *prog)
-{
-	int i;
-
-	for (i = 0; i < prog->n_array; ++i) {
-		struct gpu_array_info *array = &prog->array[i];
-
-		if (!gpu_array_requires_device_allocation(&prog->array[i]))
-			continue;
-		p = ppcg_ast_expr_print_macros(array->bound_expr, p);
-		p = isl_printer_start_line(p);
-		p = isl_printer_print_str(p,
-			"cudaCheckReturn(cudaMalloc((void **) &dev_");
-		p = isl_printer_print_str(p, prog->array[i].name);
-		p = isl_printer_print_str(p, ", ");
-		p = gpu_array_info_print_size(p, &prog->array[i]);
-		p = isl_printer_print_str(p, "));");
-		p = isl_printer_end_line(p);
-	}
-	p = isl_printer_start_line(p);
-	p = isl_printer_end_line(p);
-	return p;
-}
-
-static __isl_give isl_printer *free_device_arrays(__isl_take isl_printer *p,
-	struct gpu_prog *prog)
-{
-	int i;
-
-	for (i = 0; i < prog->n_array; ++i) {
-		if (!gpu_array_requires_device_allocation(&prog->array[i]))
-			continue;
-		p = isl_printer_start_line(p);
-		p = isl_printer_print_str(p, "cudaCheckReturn(cudaFree(dev_");
-		p = isl_printer_print_str(p, prog->array[i].name);
-		p = isl_printer_print_str(p, "));");
-		p = isl_printer_end_line(p);
-	}
-
-	return p;
-}
-
-/* Print code to "p" for copying "array" from the host to the device
- * in its entirety.  The bounds on the extent of "array" have
- * been precomputed in extract_array_info and are used in
- * gpu_array_info_print_size.
- */
-static __isl_give isl_printer *copy_array_to_device(__isl_take isl_printer *p,
-	struct gpu_array_info *array)
-{
-	p = isl_printer_start_line(p);
-	p = isl_printer_print_str(p, "cudaCheckReturn(cudaMemcpy(dev_");
-	p = isl_printer_print_str(p, array->name);
-	p = isl_printer_print_str(p, ", ");
-
-	if (gpu_array_is_scalar(array))
-		p = isl_printer_print_str(p, "&");
-	p = isl_printer_print_str(p, array->name);
-	p = isl_printer_print_str(p, ", ");
-
-	p = gpu_array_info_print_size(p, array);
-	p = isl_printer_print_str(p, ", cudaMemcpyHostToDevice));");
-	p = isl_printer_end_line(p);
-
-	return p;
-}
-
-/* Print code to "p" for copying "array" back from the device to the host
- * in its entirety.  The bounds on the extent of "array" have
- * been precomputed in extract_array_info and are used in
- * gpu_array_info_print_size.
- */
-static __isl_give isl_printer *copy_array_from_device(
-	__isl_take isl_printer *p, struct gpu_array_info *array)
-{
-	p = isl_printer_start_line(p);
-	p = isl_printer_print_str(p, "cudaCheckReturn(cudaMemcpy(");
-	if (gpu_array_is_scalar(array))
-		p = isl_printer_print_str(p, "&");
-	p = isl_printer_print_str(p, array->name);
-	p = isl_printer_print_str(p, ", dev_");
-	p = isl_printer_print_str(p, array->name);
-	p = isl_printer_print_str(p, ", ");
-	p = gpu_array_info_print_size(p, array);
-	p = isl_printer_print_str(p, ", cudaMemcpyDeviceToHost));");
-	p = isl_printer_end_line(p);
-
-	return p;
-}
-
-static __isl_give isl_printer* print_reverse_list(__isl_take isl_printer *p, int len, int *list)
-{
-	int i;
-
-	if (len == 0)
-		return p;
-
-	p = isl_printer_print_str(p, "(");
-	for (i = 0; i < len; ++i) {
-		if (i)
-			p = isl_printer_print_str(p, ", ");
-		p = isl_printer_print_int(p, list[len - 1 - i]);
-	}
-	return isl_printer_print_str(p, ")");
-}
-
-/* Print the effective grid size as a list of the sizes in each
- * dimension, from innermost to outermost.
- */
-static __isl_give isl_printer *print_grid_size(__isl_take isl_printer *p,
-	struct ppcg_kernel *kernel)
-{
-	int i;
-	int dim;
-
-	dim = isl_multi_pw_aff_dim(kernel->grid_size, isl_dim_set);
-	if (dim == 0)
-		return p;
-
-	p = isl_printer_print_str(p, "(");
-	for (i = dim - 1; i >= 0; --i) {
-		isl_ast_expr *bound;
-
-		bound = isl_ast_expr_get_op_arg(kernel->grid_size_expr, 1 + i);
-		p = isl_printer_print_ast_expr(p, bound);
-		isl_ast_expr_free(bound);
-
-		if (i > 0)
-			p = isl_printer_print_str(p, ", ");
-	}
-
-	p = isl_printer_print_str(p, ")");
-
-	return p;
-}
-
-/* Print the grid definition.
- */
-static __isl_give isl_printer *print_grid(__isl_take isl_printer *p,
-	struct ppcg_kernel *kernel)
-{
-	p = isl_printer_start_line(p);
-	p = isl_printer_print_str(p, "dim3 k");
-	p = isl_printer_print_int(p, kernel->id);
-	p = isl_printer_print_str(p, "_dimGrid");
-	p = print_grid_size(p, kernel);
-	p = isl_printer_print_str(p, ";");
-	p = isl_printer_end_line(p);
-
-	return p;
-}
-
-/* Print the arguments to a kernel declaration or call.  If "types" is set,
- * then print a declaration (including the types of the arguments).
- *
- * The arguments are printed in the following order
- * - the arrays accessed by the kernel
- * - the parameters
- * - the host loop iterators
- */
-static __isl_give isl_printer *print_kernel_arguments(__isl_take isl_printer *p,
-	struct gpu_prog *prog, struct ppcg_kernel *kernel, int types)
-{
-	int i, n;
-	int first = 1;
-	unsigned nparam;
-	isl_space *space;
-	const char *type;
-
-	for (i = 0; i < prog->n_array; ++i) {
-		int required;
-
-		required = ppcg_kernel_requires_array_argument(kernel, i);
-		if (required < 0)
-			return isl_printer_free(p);
-		if (!required)
-			continue;
-
-		if (!first)
-			p = isl_printer_print_str(p, ", ");
-
-		if (types)
-			p = gpu_array_info_print_declaration_argument(p,
-				&prog->array[i], NULL);
-		else
-			p = gpu_array_info_print_call_argument(p,
-				&prog->array[i]);
-
-		first = 0;
-	}
-
-	space = isl_union_set_get_space(kernel->arrays);
-	nparam = isl_space_dim(space, isl_dim_param);
-	for (i = 0; i < nparam; ++i) {
-		const char *name;
-
-		name = isl_space_get_dim_name(space, isl_dim_param, i);
-
-		if (!first)
-			p = isl_printer_print_str(p, ", ");
-		if (types)
-			p = isl_printer_print_str(p, "int ");
-		p = isl_printer_print_str(p, name);
-
-		first = 0;
-	}
-	isl_space_free(space);
-
-	n = isl_space_dim(kernel->space, isl_dim_set);
-	type = isl_options_get_ast_iterator_type(prog->ctx);
-	for (i = 0; i < n; ++i) {
-		const char *name;
-
-		if (!first)
-			p = isl_printer_print_str(p, ", ");
-		name = isl_space_get_dim_name(kernel->space, isl_dim_set, i);
-		if (types) {
-			p = isl_printer_print_str(p, type);
-			p = isl_printer_print_str(p, " ");
-		}
-		p = isl_printer_print_str(p, name);
-
-		first = 0;
-	}
-
-	return p;
-}
-
-/* Print the header of the given kernel.
- */
-static __isl_give isl_printer *print_kernel_header(__isl_take isl_printer *p,
-	struct gpu_prog *prog, struct ppcg_kernel *kernel)
-{
-	p = isl_printer_start_line(p);
-	p = isl_printer_print_str(p, "__global__ void kernel");
-	p = isl_printer_print_int(p, kernel->id);
-	p = isl_printer_print_str(p, "(");
-	p = print_kernel_arguments(p, prog, kernel, 1);
-	p = isl_printer_print_str(p, ")");
-
-	return p;
-}
-
-/* Print the header of the given kernel to both gen->cuda.kernel_h
- * and gen->cuda.kernel_c.
- */
-static void print_kernel_headers(struct gpu_prog *prog,
-	struct ppcg_kernel *kernel, struct cuda_info *cuda)
-{
-	isl_printer *p;
-
-	p = isl_printer_to_file(prog->ctx, cuda->kernel_h);
-	p = isl_printer_set_output_format(p, ISL_FORMAT_C);
-	p = print_kernel_header(p, prog, kernel);
-	p = isl_printer_print_str(p, ";");
-	p = isl_printer_end_line(p);
-	isl_printer_free(p);
-
-	p = isl_printer_to_file(prog->ctx, cuda->kernel_c);
-	p = isl_printer_set_output_format(p, ISL_FORMAT_C);
-	p = print_kernel_header(p, prog, kernel);
-	p = isl_printer_end_line(p);
-	isl_printer_free(p);
-}
-
-static void print_indent(FILE *dst, int indent)
-{
-	fprintf(dst, "%*s", indent, "");
-}
-
-/* Print a list of iterators of type "type" with names "ids" to "out".
- * Each iterator is assigned one of the cuda identifiers in cuda_dims.
- * In particular, the last iterator is assigned the x identifier
- * (the first in the list of cuda identifiers).
- */
-static void print_iterators(FILE *out, const char *type,
-	__isl_keep isl_id_list *ids, const char *cuda_dims[])
-{
-	int i, n;
-
-	n = isl_id_list_n_id(ids);
-	if (n <= 0)
-		return;
-	print_indent(out, 4);
-	fprintf(out, "%s ", type);
-	for (i = 0; i < n; ++i) {
-		isl_id *id;
-
-		if (i)
-			fprintf(out, ", ");
-		id = isl_id_list_get_id(ids, i);
-		fprintf(out, "%s = %s", isl_id_get_name(id),
-			cuda_dims[n - 1 - i]);
-		isl_id_free(id);
-	}
-	fprintf(out, ";\n");
-}
-
-static void print_kernel_iterators(FILE *out, struct ppcg_kernel *kernel)
-{
-	isl_ctx *ctx = isl_ast_node_get_ctx(kernel->tree);
-	const char *type;
-	const char *block_dims[] = { "blockIdx.x", "blockIdx.y" };
-	const char *thread_dims[] = { "threadIdx.x", "threadIdx.y",
-					"threadIdx.z" };
-
-	type = isl_options_get_ast_iterator_type(ctx);
-
-	print_iterators(out, type, kernel->block_ids, block_dims);
-	print_iterators(out, type, kernel->thread_ids, thread_dims);
-}
-
-static __isl_give isl_printer *print_kernel_var(__isl_take isl_printer *p,
-	struct ppcg_kernel_var *var)
-{
-	int j;
-
-	p = isl_printer_start_line(p);
-	if (var->type == ppcg_access_shared)
-		p = isl_printer_print_str(p, "__shared__ ");
-	p = isl_printer_print_str(p, var->array->type);
-	p = isl_printer_print_str(p, " ");
-	p = isl_printer_print_str(p,  var->name);
-	for (j = 0; j < var->array->n_index; ++j) {
-		isl_val *v;
-
-		p = isl_printer_print_str(p, "[");
-		v = isl_vec_get_element_val(var->size, j);
-		p = isl_printer_print_val(p, v);
-		isl_val_free(v);
-		p = isl_printer_print_str(p, "]");
-	}
-	p = isl_printer_print_str(p, ";");
-	p = isl_printer_end_line(p);
-
-	return p;
-}
-
-static __isl_give isl_printer *print_kernel_vars(__isl_take isl_printer *p,
-	struct ppcg_kernel *kernel)
-{
-	int i;
-
-	for (i = 0; i < kernel->n_var; ++i)
-		p = print_kernel_var(p, &kernel->var[i]);
-
-	return p;
-}
-
-/* Print a sync statement.
- */
-static __isl_give isl_printer *print_sync(__isl_take isl_printer *p,
-	struct ppcg_kernel_stmt *stmt)
-{
-	p = isl_printer_start_line(p);
-	p = isl_printer_print_str(p, "__syncthreads();");
-	p = isl_printer_end_line(p);
-
-	return p;
-}
-
-/* This function is called for each user statement in the AST,
- * i.e., for each kernel body statement, copy statement or sync statement.
- */
-static __isl_give isl_printer *print_kernel_stmt(__isl_take isl_printer *p,
-	__isl_take isl_ast_print_options *print_options,
-	__isl_keep isl_ast_node *node, void *user)
-{
-	isl_id *id;
-	struct ppcg_kernel_stmt *stmt;
-
-	id = isl_ast_node_get_annotation(node);
-	stmt = isl_id_get_user(id);
-	isl_id_free(id);
-
-	isl_ast_print_options_free(print_options);
-
-	switch (stmt->type) {
-	case ppcg_kernel_copy:
-		return ppcg_kernel_print_copy(p, stmt);
-	case ppcg_kernel_sync:
-		return print_sync(p, stmt);
-	case ppcg_kernel_domain:
-		return ppcg_kernel_print_domain(p, stmt);
-	}
-
-	return p;
-}
-
-static void print_kernel(struct gpu_prog *prog, struct ppcg_kernel *kernel,
-	struct cuda_info *cuda)
-{
-	isl_ctx *ctx = isl_ast_node_get_ctx(kernel->tree);
-	isl_ast_print_options *print_options;
-	isl_printer *p;
-
-	print_kernel_headers(prog, kernel, cuda);
-	fprintf(cuda->kernel_c, "{\n");
-	print_kernel_iterators(cuda->kernel_c, kernel);
-
-	p = isl_printer_to_file(ctx, cuda->kernel_c);
-	p = isl_printer_set_output_format(p, ISL_FORMAT_C);
-	p = isl_printer_indent(p, 4);
-
-	p = print_kernel_vars(p, kernel);
-	p = isl_printer_end_line(p);
-	p = ppcg_set_macro_names(p);
-	p = gpu_print_macros(p, kernel->tree);
-
-	print_options = isl_ast_print_options_alloc(ctx);
-	print_options = isl_ast_print_options_set_print_user(print_options,
-							&print_kernel_stmt, NULL);
-	p = isl_ast_node_print(kernel->tree, p, print_options);
-	isl_printer_free(p);
-
-	fprintf(cuda->kernel_c, "}\n");
-}
-
-/* Print code for initializing the device for execution of the transformed
- * code.  This includes declaring locally defined variables as well as
- * declaring and allocating the required copies of arrays on the device.
- */
-static __isl_give isl_printer *init_device(__isl_take isl_printer *p,
-	struct gpu_prog *prog)
-{
-	p = print_cuda_macros(p);
-
-	p = gpu_print_local_declarations(p, prog);
-	p = declare_device_arrays(p, prog);
-	p = allocate_device_arrays(p, prog);
-
-	return p;
-}
-
-/* Print code for clearing the device after execution of the transformed code.
- * In particular, free the memory that was allocated on the device.
- */
-static __isl_give isl_printer *clear_device(__isl_take isl_printer *p,
-	struct gpu_prog *prog)
-{
-	p = free_device_arrays(p, prog);
-
-	return p;
-}
-
-/* Print a statement for copying an array to or from the device,
- * or for initializing or clearing the device.
- * The statement identifier of a copying node is called
- * "to_device_<array name>" or "from_device_<array name>" and
- * its user pointer points to the gpu_array_info of the array
- * that needs to be copied.
- * The node for initializing the device is called "init_device".
- * The node for clearing the device is called "clear_device".
- *
- * Extract the array (if any) from the identifier and call
- * init_device, clear_device, copy_array_to_device or copy_array_from_device.
- */
-static __isl_give isl_printer *print_device_node(__isl_take isl_printer *p,
-	__isl_keep isl_ast_node *node, struct gpu_prog *prog)
-{
-	isl_ast_expr *expr, *arg;
-	isl_id *id;
-	const char *name;
-	struct gpu_array_info *array;
-
-	expr = isl_ast_node_user_get_expr(node);
-	arg = isl_ast_expr_get_op_arg(expr, 0);
-	id = isl_ast_expr_get_id(arg);
-	name = isl_id_get_name(id);
-	array = isl_id_get_user(id);
-	isl_id_free(id);
-	isl_ast_expr_free(arg);
-	isl_ast_expr_free(expr);
-
-	if (!name)
-		return isl_printer_free(p);
-	if (!strcmp(name, "init_device"))
-		return init_device(p, prog);
-	if (!strcmp(name, "clear_device"))
-		return clear_device(p, prog);
-	if (!array)
-		return isl_printer_free(p);
-
-	if (!prefixcmp(name, "to_device"))
-		return copy_array_to_device(p, array);
-	else
-		return copy_array_from_device(p, array);
-}
-
-struct print_host_user_data {
-	struct cuda_info *cuda;
-	struct gpu_prog *prog;
-};
-
-/* Print the user statement of the host code to "p".
- *
- * The host code may contain original user statements, kernel launches,
- * statements that copy data to/from the device and statements
- * the initialize or clear the device.
- * The original user statements and the kernel launches have
- * an associated annotation, while the other statements do not.
- * The latter are handled by print_device_node.
- * The annotation on the user statements is called "user".
- *
- * In case of a kernel launch, print a block of statements that
- * defines the grid and the block and then launches the kernel.
- */
-__isl_give isl_printer *print_host_user(__isl_take isl_printer *p,
-	__isl_take isl_ast_print_options *print_options,
-	__isl_keep isl_ast_node *node, void *user)
-{
-	isl_id *id;
-	int is_user;
-	struct ppcg_kernel *kernel;
-	struct ppcg_kernel_stmt *stmt;
-	struct print_host_user_data *data;
-
-	isl_ast_print_options_free(print_options);
-
-	data = (struct print_host_user_data *) user;
-
-	id = isl_ast_node_get_annotation(node);
-	if (!id)
-		return print_device_node(p, node, data->prog);
-
-	is_user = !strcmp(isl_id_get_name(id), "user");
-	kernel = is_user ? NULL : isl_id_get_user(id);
-	stmt = is_user ? isl_id_get_user(id) : NULL;
-	isl_id_free(id);
-
-	if (is_user)
-		return ppcg_kernel_print_domain(p, stmt);
-
-	p = ppcg_start_block(p);
-
-	p = isl_printer_start_line(p);
-	p = isl_printer_print_str(p, "dim3 k");
-	p = isl_printer_print_int(p, kernel->id);
-	p = isl_printer_print_str(p, "_dimBlock");
-	p = print_reverse_list(p, kernel->n_block, kernel->block_dim);
-	p = isl_printer_print_str(p, ";");
-	p = isl_printer_end_line(p);
-
-	p = print_grid(p, kernel);
-
-	p = isl_printer_start_line(p);
-	p = isl_printer_print_str(p, "kernel");
-	p = isl_printer_print_int(p, kernel->id);
-	p = isl_printer_print_str(p, " <<<k");
-	p = isl_printer_print_int(p, kernel->id);
-	p = isl_printer_print_str(p, "_dimGrid, k");
-	p = isl_printer_print_int(p, kernel->id);
-	p = isl_printer_print_str(p, "_dimBlock>>> (");
-	p = print_kernel_arguments(p, data->prog, kernel, 0);
-	p = isl_printer_print_str(p, ");");
-	p = isl_printer_end_line(p);
-
-	p = isl_printer_start_line(p);
-	p = isl_printer_print_str(p, "cudaCheckKernel();");
-	p = isl_printer_end_line(p);
-
-	p = ppcg_end_block(p);
-
-	p = isl_printer_start_line(p);
-	p = isl_printer_end_line(p);
-
-#if 0
-	print_kernel(data->prog, kernel, data->cuda);
-#endif
-
-	return p;
-}
-
-static __isl_give isl_printer *print_host_code(__isl_take isl_printer *p,
-	struct gpu_prog *prog, __isl_keep isl_ast_node *tree,
-	struct cuda_info *cuda)
-{
-	isl_ast_print_options *print_options;
-	isl_ctx *ctx = isl_ast_node_get_ctx(tree);
-	struct print_host_user_data data = { cuda, prog };
-
-	print_options = isl_ast_print_options_alloc(ctx);
-	print_options = isl_ast_print_options_set_print_user(print_options,
-						&print_host_user, &data);
-
-	p = gpu_print_macros(p, tree);
-	p = isl_ast_node_print(tree, p, print_options);
-
-	return p;
-}
-
-/* Given a gpu_prog "prog" and the corresponding transformed AST
- * "tree", print the entire CUDA code to "p".
- * "types" collects the types for which a definition has already
- * been printed.
- */
-static __isl_give isl_printer *print_cuda(__isl_take isl_printer *p,
-	struct gpu_prog *prog, __isl_keep isl_ast_node *tree,
-	struct gpu_types *types, void *user)
-{
-	struct cuda_info *cuda = user;
-	isl_printer *kernel;
-
-	kernel = isl_printer_to_file(isl_printer_get_ctx(p), cuda->kernel_c);
-	kernel = isl_printer_set_output_format(kernel, ISL_FORMAT_C);
-	kernel = gpu_print_types(kernel, types, prog);
-	isl_printer_free(kernel);
-
-	if (!kernel)
-		return isl_printer_free(p);
-
-	p = print_host_code(p, prog, tree, cuda);
-
-	return p;
-}
-
-/* Transform the code in the file called "input" by replacing
- * all scops by corresponding CUDA code.
- * The names of the output files are derived from "input".
- *
- * We let generate_gpu do all the hard work and then let it call
- * us back for printing the AST in print_cuda.
- *
- * To prepare for this printing, we first open the output files
- * and we close them after generate_gpu has finished.
- */
-int generate_cuda(isl_ctx *ctx, struct ppcg_options *options,
-	const char *input)
-{
-	struct cuda_info cuda;
-	int r;
-
-	cuda_open_files(&cuda, input);
-
-	r = generate_gpu(ctx, input, cuda.host_c, options, &print_cuda, &cuda);
-
-	cuda_close_files(&cuda);
-
-	return r;
-}
diff --git a/polly/lib/External/ppcg/cuda.h b/polly/lib/External/ppcg/cuda.h
deleted file mode 100644
index 450fc697ca04a..0000000000000
--- a/polly/lib/External/ppcg/cuda.h
+++ /dev/null
@@ -1,13 +0,0 @@
-#ifndef _CUDA_H
-#define _CUDA_H
-
-#include "ppcg_options.h"
-#include "ppcg.h"
-
-int generate_cuda(isl_ctx *ctx, struct ppcg_options *options,
-	const char *input);
-
-__isl_give isl_printer *print_host_user(__isl_take isl_printer *p,
-	__isl_take isl_ast_print_options *print_options,
-	__isl_keep isl_ast_node *node, void *user);
-#endif
diff --git a/polly/lib/External/ppcg/cuda_common.c b/polly/lib/External/ppcg/cuda_common.c
deleted file mode 100644
index 497353f4319f4..0000000000000
--- a/polly/lib/External/ppcg/cuda_common.c
+++ /dev/null
@@ -1,50 +0,0 @@
-/*
- * Copyright 2010      INRIA Saclay
- *
- * Use of this software is governed by the MIT license
- *
- * Written by Sven Verdoolaege, INRIA Saclay - Ile-de-France,
- * Parc Club Orsay Universite, ZAC des vignes, 4 rue Jacques Monod,
- * 91893 Orsay, France
- */
-
-#include <ctype.h>
-#include <limits.h>
-#include <string.h>
-
-#include "cuda_common.h"
-#include "ppcg.h"
-
-/* Open the host .cu file and the kernel .hu and .cu files for writing.
- * Add the necessary includes.
- */
-void cuda_open_files(struct cuda_info *info, const char *input)
-{
-    char name[PATH_MAX];
-    int len;
-
-    len = ppcg_extract_base_name(name, input);
-
-    strcpy(name + len, "_host.cu");
-    info->host_c = fopen(name, "w");
-
-    strcpy(name + len, "_kernel.cu");
-    info->kernel_c = fopen(name, "w");
-
-    strcpy(name + len, "_kernel.hu");
-    info->kernel_h = fopen(name, "w");
-    fprintf(info->host_c, "#include <assert.h>\n");
-    fprintf(info->host_c, "#include <stdio.h>\n");
-    fprintf(info->host_c, "#include \"%s\"\n", name);
-    fprintf(info->kernel_c, "#include \"%s\"\n", name);
-    fprintf(info->kernel_h, "#include \"cuda.h\"\n\n");
-}
-
-/* Close all output files.
- */
-void cuda_close_files(struct cuda_info *info)
-{
-    fclose(info->kernel_c);
-    fclose(info->kernel_h);
-    fclose(info->host_c);
-}
diff --git a/polly/lib/External/ppcg/cuda_common.h b/polly/lib/External/ppcg/cuda_common.h
deleted file mode 100644
index 2a7db952cdf79..0000000000000
--- a/polly/lib/External/ppcg/cuda_common.h
+++ /dev/null
@@ -1,15 +0,0 @@
-#ifndef _CUDA_COMMON_H_
-#define _CUDA_COMMON_H_
-
-#include <stdio.h>
-
-struct cuda_info {
-	FILE *host_c;
-	FILE *kernel_c;
-	FILE *kernel_h;
-};
-
-void cuda_open_files(struct cuda_info *info, const char *input);
-void cuda_close_files(struct cuda_info *info);
-
-#endif
diff --git a/polly/lib/External/ppcg/external.c b/polly/lib/External/ppcg/external.c
deleted file mode 100644
index c5ef6320e64fb..0000000000000
--- a/polly/lib/External/ppcg/external.c
+++ /dev/null
@@ -1,192 +0,0 @@
-#include <assert.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <pet.h>
-#include "cpu.h"
-#include "opencl.h"
-
-
-#define die() { \
-  fprintf(stderr, "Dummy function %s called\n", __FUNCTION__); \
-  abort(); \
-}
-
-__isl_give isl_union_map *pet_scop_compute_outer_to_any(
-  __isl_keep pet_scop *scop) {
-  die();
-}
-__isl_give isl_union_map *pet_scop_compute_outer_to_inner(
-  __isl_keep pet_scop *scop) {
-  die();
-}
-enum pet_tree_type pet_tree_get_type(__isl_keep pet_tree *tree) {
-  die();
-}
-int pet_tree_foreach_access_expr(__isl_keep pet_tree *tree,
-  int (*fn)(__isl_keep pet_expr *expr, void *user), void *user) {
-  die();
-}
-isl_ctx *pet_expr_get_ctx(__isl_keep pet_expr *expr) {
-  die();
-}
-isl_bool pet_expr_access_is_read(__isl_keep pet_expr *expr) {
-  die();
-}
-isl_bool pet_expr_access_is_write(__isl_keep pet_expr *expr) {
-  die();
-}
-__isl_give isl_union_map *pet_expr_access_get_tagged_may_read(
-  __isl_keep pet_expr *expr) {
-  die();
-}
-__isl_give isl_union_map *pet_expr_access_get_tagged_may_write(
-  __isl_keep pet_expr *expr) {
-  die();
-}
-__isl_give isl_union_map *pet_expr_access_get_must_write(
-  __isl_keep pet_expr *expr) {
-  die();
-}
-__isl_give isl_multi_pw_aff *pet_expr_access_get_index(
-  __isl_keep pet_expr *expr) {
-  die();
-}
-__isl_give isl_id *pet_expr_access_get_ref_id(__isl_keep pet_expr *expr) {
-  die();
-}
-__isl_give isl_printer *print_cpu(__isl_take isl_printer *p,
-  struct ppcg_scop *ps, struct ppcg_options *options) {
-  die();
-}
-
-__isl_give isl_printer *pet_stmt_print_body(struct pet_stmt *stmt,
-  __isl_take isl_printer *p, __isl_keep isl_id_to_ast_expr *ref2expr) {
-  die();
-}
-unsigned pet_loc_get_start(__isl_keep pet_loc *loc) {
-  die();
-}
-unsigned pet_loc_get_end(__isl_keep pet_loc *loc) {
-  die();
-}
-int pet_transform_C_source(isl_ctx *ctx, const char *input, FILE *output,
-  __isl_give isl_printer *(*transform)(__isl_take isl_printer *p,
-    __isl_take pet_scop *scop, void *user), void *user) {
-  die();
-}
-__isl_give isl_printer *pet_scop_print_original(__isl_keep pet_scop *scop,
-  __isl_take isl_printer *p) {
-  die();
-}
-__isl_null pet_scop *pet_scop_free(__isl_take pet_scop *scop) {
-  die();
-}
-__isl_give pet_scop *pet_scop_align_params(__isl_take pet_scop *scop) {
-  die();
-}
-int pet_scop_can_build_ast_exprs(__isl_keep pet_scop *scop) {
-  die();
-}
-int pet_scop_has_data_dependent_conditions(__isl_keep pet_scop *scop) {
-  die();
-}
-int pet_tree_foreach_expr(__isl_keep pet_tree *tree,
-  int (*fn)(__isl_keep pet_expr *expr, void *user), void *user) {
-  die();
-}
-int pet_expr_foreach_call_expr(__isl_keep pet_expr *expr,
-  int (*fn)(__isl_keep pet_expr *expr, void *user), void *user) {
-  die();
-}
-int pet_stmt_is_kill(struct pet_stmt *stmt) {
-  die();
-}
-struct isl_args pet_options_args;
-const char *ppcg_version(void) {
-  die();
-}
-int pet_options_set_encapsulate_dynamic_control(isl_ctx *ctx, int val) {
-  die();
-}
-int generate_opencl(isl_ctx *ctx, struct ppcg_options *options,
-  const char *input, const char *output) {
-  die();
-}
-int generate_cpu(isl_ctx *ctx, struct ppcg_options *options,
-  const char *input, const char *output) {
-  die();
-}
-__isl_give isl_id_to_ast_expr *pet_stmt_build_ast_exprs(struct pet_stmt *stmt,
-  __isl_keep isl_ast_build *build,
-  __isl_give isl_multi_pw_aff *(*fn_index)(
-    __isl_take isl_multi_pw_aff *mpa, __isl_keep isl_id *id,
-    void *user), void *user_index,
-  __isl_give isl_ast_expr *(*fn_expr)(__isl_take isl_ast_expr *expr,
-    __isl_keep isl_id *id, void *user), void *user_expr) {
-  die();
-}
-__isl_give isl_union_map *pet_scop_get_tagged_may_reads(
-  __isl_keep pet_scop *scop) {
-  die();
-}
-__isl_give isl_union_map *pet_scop_get_may_reads(__isl_keep pet_scop *scop) {
-  die();
-}
-__isl_give isl_union_map *pet_scop_get_may_writes(__isl_keep pet_scop *scop) {
-  die();
-}
-__isl_give isl_union_map *pet_scop_get_must_writes(__isl_keep pet_scop *scop) {
-  die();
-}
-__isl_give isl_union_map *pet_scop_get_tagged_may_writes(
-  __isl_keep pet_scop *scop) {
-  die();
-}
-__isl_give isl_union_map *pet_scop_get_tagged_must_writes(
-  __isl_keep pet_scop *scop) {
-  die();
-}
-__isl_give isl_union_map *pet_scop_get_must_kills(__isl_keep pet_scop *scop) {
-  die();
-}
-__isl_give isl_union_map *pet_scop_get_tagged_must_kills(
-  __isl_keep pet_scop *scop) {
-  die();
-}
-__isl_keep const char *pet_expr_call_get_name(__isl_keep pet_expr *expr) {
-  die();
-}
-__isl_give pet_expr *pet_expr_call_set_name(__isl_take pet_expr *expr,
-  __isl_keep const char *name) {
-  die();
-}
-__isl_give pet_expr *pet_expr_get_arg(__isl_keep pet_expr *expr, int pos) {
-  die();
-}
-__isl_give pet_expr *pet_expr_new_cast(const char *type_name,
-  __isl_take pet_expr *arg) {
-  die();
-}
-__isl_give pet_expr *pet_expr_set_arg(__isl_take pet_expr *expr, int pos,
-  __isl_take pet_expr *arg) {
-  die();
-}
-__isl_give pet_tree *pet_tree_copy(__isl_keep pet_tree *tree) {
-  die();
-}
-__isl_null pet_tree *pet_tree_free(__isl_take pet_tree *tree) {
-  die();
-}
-__isl_give pet_tree *pet_tree_map_call_expr(__isl_take pet_tree *tree,
-  __isl_give pet_expr *(*fn)(__isl_take pet_expr *expr, void *user),
-  void *user) {
-  die();
-}
-__isl_give isl_union_map *pet_expr_access_get_may_read(
-  __isl_keep pet_expr *expr) {
-  die();
-}
-__isl_give isl_union_map *pet_expr_access_get_may_write(
-  __isl_keep pet_expr *expr) {
-  die();
-}
diff --git a/polly/lib/External/ppcg/gpu.c b/polly/lib/External/ppcg/gpu.c
deleted file mode 100644
index cfd998f1c315b..0000000000000
--- a/polly/lib/External/ppcg/gpu.c
+++ /dev/null
@@ -1,5849 +0,0 @@
-/*
- * Copyright 2010-2011 INRIA Saclay
- * Copyright 2012-2013 Ecole Normale Superieure
- * Copyright 2015-2016 Sven Verdoolaege
- *
- * Use of this software is governed by the MIT license
- *
- * Written by Sven Verdoolaege, INRIA Saclay - Ile-de-France,
- * Parc Club Orsay Universite, ZAC des vignes, 4 rue Jacques Monod,
- * 91893 Orsay, France
- * and Ecole Normale Superieure, 45 rue d’Ulm, 75230 Paris, France
- */
-
-#include <assert.h>
-#include <stdlib.h>
-#include <string.h>
-
-#include <isl/polynomial.h>
-#include <isl/union_set.h>
-#include <isl/aff.h>
-#include <isl/ilp.h>
-#include <isl/flow.h>
-#include <isl/schedule.h>
-#include <isl/schedule_node.h>
-#include <isl/options.h>
-#include <isl/ast_build.h>
-
-#include "cpu.h"
-#include "gpu.h"
-#include "gpu_array_tile.h"
-#include "gpu_group.h"
-#include "gpu_hybrid.h"
-#include "gpu_tree.h"
-#include "hybrid.h"
-#include "schedule.h"
-#include "ppcg_options.h"
-#include "print.h"
-#include "util.h"
-
-struct gpu_array_info;
-
-/* Return the name of the outer array (of structs) accessed by "access".
- */
-static const char *get_outer_array_name(__isl_keep isl_map *access)
-{
-	isl_space *space;
-	const char *name;
-
-	space = isl_space_range(isl_map_get_space(access));
-	while (space && isl_space_is_wrapping(space))
-		space = isl_space_domain(isl_space_unwrap(space));
-	name = isl_space_get_tuple_name(space, isl_dim_set);
-	isl_space_free(space);
-
-	return name;
-}
-
-/* Collect all references to the given array and store pointers to them
- * in array->refs.
- */
-void collect_references(struct gpu_prog *prog,
-	struct gpu_array_info *array)
-{
-	int i;
-	int n;
-
-	n = 0;
-	for (i = 0; i < prog->n_stmts; ++i) {
-		struct gpu_stmt *stmt = &prog->stmts[i];
-		struct gpu_stmt_access *access;
-
-		for (access = stmt->accesses; access; access = access->next) {
-			const char *name;
-			name = get_outer_array_name(access->access);
-			if (name && !strcmp(array->name, name))
-				n++;
-		}
-	}
-
-	array->n_ref = n;
-	array->refs = isl_alloc_array(prog->ctx, struct gpu_stmt_access *, n);
-	assert(array->refs);
-
-	n = 0;
-	for (i = 0; i < prog->n_stmts; ++i) {
-		struct gpu_stmt *stmt = &prog->stmts[i];
-		struct gpu_stmt_access *access;
-
-		for (access = stmt->accesses; access; access = access->next) {
-			const char *name;
-			name = get_outer_array_name(access->access);
-			if (!name || strcmp(array->name, name))
-				continue;
-
-			array->refs[n++] = access;
-		}
-	}
-}
-
-/* Compute and return the extent of "array", taking into account the set of
- * accessed elements.
- *
- * In particular, the extent in the outer dimension is taken
- * from "accessed", while the extents in the remaining dimensions
- * are taken from array->extent.
- *
- * The extent in the outer dimension cannot be taken from array->extent
- * because that may be unbounded.  Furthermore, even if it is bounded,
- * it may be larger than the piece of the array that is being accessed.
- */
-static __isl_give isl_set *compute_extent(struct pet_array *array,
-	__isl_keep isl_set *accessed)
-{
-	int n_index;
-	isl_id *id;
-	isl_set *outer;
-	isl_set *extent;
-
-	extent = isl_set_copy(array->extent);
-
-	n_index = isl_set_dim(accessed, isl_dim_set);
-	if (n_index == 0)
-		return extent;
-
-	extent = isl_set_project_out(extent, isl_dim_set, 0, 1);
-	outer = isl_set_copy(accessed);
-	outer = isl_set_project_out(outer, isl_dim_set, 1, n_index - 1);
-	extent = isl_set_flat_product(outer, extent);
-	id = isl_set_get_tuple_id(accessed);
-	extent = isl_set_set_tuple_id(extent, id);
-
-	return extent;
-}
-
-/* Is the array "array" being extracted a read-only scalar?
- *
- * That is, is "array" a scalar that is never possibly written to.
- * An array containing structures is never considered to be a scalar.
- */
-static int is_read_only_scalar(struct gpu_array_info *array,
-	struct gpu_prog *prog)
-{
-	isl_set *space;
-	isl_union_map *write;
-	int empty;
-
-	if (array->has_compound_element)
-		return 0;
-	if (array->n_index != 0)
-		return 0;
-
-	write = isl_union_map_copy(prog->may_write);
-	space = isl_set_universe(isl_space_copy(array->space));
-	write = isl_union_map_intersect_range(write,
-						isl_union_set_from_set(space));
-	empty = isl_union_map_is_empty(write);
-	isl_union_map_free(write);
-
-	return empty;
-}
-
-/* Is "array" only accessed as individual, fixed elements?
- * That is, does each access to "array" access a single, fixed element?
- */
-isl_bool only_fixed_element_accessed(struct gpu_array_info *array)
-{
-	int i;
-
-	for (i = 0; i < array->n_ref; ++i)
-		if (!array->refs[i]->fixed_element)
-			return isl_bool_false;
-
-	return isl_bool_true;
-}
-
-/* Compute bounds on the host array "pa" based on the corresponding
- * accessed elements in "arrays"
- * and collect all references to the array.
- * Store the results in "info".
- *
- * If the array is zero-dimensional and does not contain structures,
- * i.e., if the array is a scalar, we check whether it is read-only.
- * We also check whether the array is accessed at all.
- */
-static int extract_array_info(struct gpu_prog *prog,
-	struct gpu_array_info *info, struct pet_array *pa,
-	__isl_keep isl_union_set *arrays)
-{
-	int empty;
-	const char *name;
-	int n_index;
-	isl_multi_pw_aff *bounds;
-	isl_set *accessed, *extent;
-
-	n_index = isl_set_dim(pa->extent, isl_dim_set);
-	name = isl_set_get_tuple_name(pa->extent);
-
-	info->space = isl_set_get_space(pa->extent);
-	info->name = strdup(name);
-	info->n_index = n_index;
-	info->linearize = prog->scop->options->linearize_device_arrays;
-
-	info->type = strdup(pa->element_type);
-	info->size = pa->element_size;
-	info->local = pa->declared && !pa->exposed;
-	info->has_compound_element = pa->element_is_record;
-	info->read_only_scalar = is_read_only_scalar(info, prog);
-
-	info->declared_extent = isl_set_copy(pa->extent);
-	accessed = isl_union_set_extract_set(arrays,
-					    isl_space_copy(info->space));
-	empty = isl_set_is_empty(accessed);
-	extent = compute_extent(pa, accessed);
-	isl_set_free(accessed);
-	info->extent = extent;
-	if (empty < 0)
-		return -1;
-	info->accessed = !empty;
-	bounds = ppcg_size_from_extent(isl_set_copy(extent));
-	bounds = isl_multi_pw_aff_gist(bounds, isl_set_copy(prog->context));
-	if (!bounds)
-		return -1;
-	if (!isl_multi_pw_aff_is_cst(bounds))
-		info->linearize = 1;
-	info->bound = bounds;
-
-	collect_references(prog, info);
-	info->only_fixed_element = only_fixed_element_accessed(info);
-
-	return 0;
-}
-
-/* Remove independence from the order constraints "order" on array "array".
- * Since the pairs of iterations in the filter relation of an independence
- * are guaranteed to be completely independent by the user, there is
- * no need to ensure that live ranges are ordered along those pairs.
- * We make an exception for local variables, though, as the independence
- * guarantee does not apply to those.
- *
- * The order constraints are used in two places.
- * Those on scalars are used in check_scalar_live_ranges to check if
- * we need to force the scalar to be private.  Any non-local scalar
- * should not be forced scalar if it only appears in independent loops.
- * Those on non-scalars are added to the coincidence constraints
- * in compute_schedule because we do not support any array expansion.
- * Accesses to non-local arrays should not prevent a loop from being
- * considered coincident so we should indeed remove those constraints
- * from the order constraints.
- */
-static __isl_give isl_union_map *remove_independences(struct gpu_prog *prog,
-	struct gpu_array_info *array, __isl_take isl_union_map *order)
-{
-	// We do not have independence information in Polly. Hence, make this
-	// function a no-op.
-	return order;
-	int i;
-
-	for (i = 0; i < prog->scop->pet->n_independence; ++i) {
-		struct pet_independence *pi = prog->scop->pet->independences[i];
-		if (isl_union_set_contains(pi->local, array->space))
-			continue;
-
-		order = isl_union_map_subtract(order,
-						isl_union_map_copy(pi->filter));
-	}
-
-	return order;
-}
-
-/* For each array in "prog", store the (untagged) order dependences
- * derived from the array in array->dep_order.
- * In particular, consider all references that access the given array
- * and take the order dependences that have one of these references
- * as source.  (Since an order dependence relates two references to
- * the same array, the target of these order dependences will also
- * be one of these references.)
- * Additionally, store the union of these array->dep_order relations
- * for all arrays that cannot be mapped to private memory in prog->array_order.
- */
-void collect_order_dependences(struct gpu_prog *prog)
-{
-	int i;
-	isl_space *space;
-	isl_union_map *accesses;
-
-	space = isl_union_map_get_space(prog->read);
-	prog->array_order = isl_union_map_empty(space);
-
-	accesses = isl_union_map_copy(prog->scop->tagged_reads);
-	accesses = isl_union_map_union(accesses,
-			    isl_union_map_copy(prog->scop->tagged_may_writes));
-	accesses = isl_union_map_universe(accesses);
-	accesses = isl_union_map_apply_range(accesses,
-					    isl_union_map_copy(prog->to_outer));
-
-	for (i = 0; i < prog->n_array; ++i) {
-		struct gpu_array_info *array = &prog->array[i];
-		isl_set *set;
-		isl_union_set *uset;
-		isl_union_map *order;
-
-		set = isl_set_universe(isl_space_copy(array->space));
-		uset = isl_union_set_from_set(set);
-		uset = isl_union_map_domain(
-		    isl_union_map_intersect_range(isl_union_map_copy(accesses),
-						    uset));
-		order = isl_union_map_copy(prog->scop->tagged_dep_order);
-		order = isl_union_map_intersect_domain(order, uset);
-		order = isl_union_map_zip(order);
-		order = isl_union_set_unwrap(isl_union_map_domain(order));
-		order = remove_independences(prog, array, order);
-		array->dep_order = order;
-
-		if (gpu_array_can_be_private(array))
-			continue;
-
-		prog->array_order = isl_union_map_union(prog->array_order,
-					isl_union_map_copy(array->dep_order));
-	}
-
-	isl_union_map_free(accesses);
-}
-
-/* Construct a gpu_array_info for each array referenced by prog->scop and
- * collect them in prog->array.
- *
- * The sizes are based on the extents and the set of possibly accessed
- * elements by "prog".
- * If there are any member accesses involved, then they are first mapped
- * to the outer arrays of structs.
- * Only extract gpu_array_info entries for these outer arrays.
- *
- * If we are allowing live range reordering, then also set
- * the dep_order field.  Otherwise leave it NULL.
- */
-static int collect_array_info(struct gpu_prog *prog)
-{
-	int i;
-	int r = 0;
-	isl_union_set *arrays;
-
-	arrays = isl_union_map_range(isl_union_map_copy(prog->read));
-	arrays = isl_union_set_union(arrays,
-		    isl_union_map_range(isl_union_map_copy(prog->may_write)));
-
-	arrays = isl_union_set_apply(arrays,
-					isl_union_map_copy(prog->to_outer));
-
-	arrays = isl_union_set_coalesce(arrays);
-
-	prog->n_array = prog->scop->pet->n_array;
-	prog->array = isl_calloc_array(prog->ctx,
-				     struct gpu_array_info, prog->n_array);
-	assert(prog->array);
-	prog->n_array = 0;
-	for (i = 0; i < prog->scop->pet->n_array; ++i) {
-		isl_bool field;
-
-		field = isl_set_is_wrapping(prog->scop->pet->arrays[i]->extent);
-		if (field < 0)
-			break;
-		if (field)
-			continue;
-		if (extract_array_info(prog, &prog->array[prog->n_array++],
-					prog->scop->pet->arrays[i], arrays) < 0)
-			r = -1;
-	}
-	if (i < prog->scop->pet->n_array)
-		r = -1;
-
-	isl_union_set_free(arrays);
-
-	if (prog->scop->options->live_range_reordering)
-		collect_order_dependences(prog);
-
-	return r;
-}
-
-static void free_array_info(struct gpu_prog *prog)
-{
-	int i;
-
-	for (i = 0; i < prog->n_array; ++i) {
-		free(prog->array[i].type);
-		free(prog->array[i].name);
-		isl_multi_pw_aff_free(prog->array[i].bound);
-		isl_ast_expr_free(prog->array[i].bound_expr);
-		isl_space_free(prog->array[i].space);
-		isl_set_free(prog->array[i].declared_extent);
-		isl_set_free(prog->array[i].extent);
-		isl_ast_expr_free(prog->array[i].declared_size);
-		free(prog->array[i].refs);
-		isl_union_map_free(prog->array[i].dep_order);
-	}
-	free(prog->array);
-}
-
-/* Check if a gpu array is a scalar.  A scalar is a value that is not stored
- * as an array or through a pointer reference, but as a single data element.
- * At the moment, scalars are represented as zero-dimensional arrays.
- * Note that the single data element may be an entire structure.
- */
-int gpu_array_is_scalar(struct gpu_array_info *array)
-{
-	return array->n_index == 0;
-}
-
-/* Can "array" be mapped to private memory?
- * That is, is it only accessed as individual elements with
- * constant index expressions?
- */
-isl_bool gpu_array_can_be_private(struct gpu_array_info *array)
-{
-	if (!array)
-		return isl_bool_error;
-	return array->only_fixed_element;
-}
-
-/* Is "array" a read-only scalar?
- */
-int gpu_array_is_read_only_scalar(struct gpu_array_info *array)
-{
-	return array->read_only_scalar;
-}
-
-/* Does "array" need to be allocated on the device?
- * If it is a read-only scalar, then it will be passed as an argument
- * to the kernel and therefore does not require any allocation.
- * If this device memory is not accessed at all, then it does not
- * need to be allocated either.
- */
-int gpu_array_requires_device_allocation(struct gpu_array_info *array)
-{
-	if (gpu_array_is_read_only_scalar(array))
-		return 0;
-	if (!array->global)
-		return 0;
-	return 1;
-}
-
-/* Return the set of parameter values for which the array has a positive
- * size in all dimensions.
- * If the sizes are only valid for some parameter values, then those
- * constraints are also taken into account.
- */
-__isl_give isl_set *gpu_array_positive_size_guard(struct gpu_array_info *array)
-{
-	int i;
-	isl_space *space;
-	isl_set *guard;
-
-	if (!array)
-		return NULL;
-
-	space = isl_space_params(isl_space_copy(array->space));
-	guard = isl_set_universe(space);
-
-	for (i = 0; i < array->n_index; ++i) {
-		isl_pw_aff *bound;
-		isl_set *guard_i, *zero;
-
-		bound = isl_multi_pw_aff_get_pw_aff(array->bound, i);
-		guard_i = isl_pw_aff_nonneg_set(isl_pw_aff_copy(bound));
-		zero = isl_pw_aff_zero_set(bound);
-		guard_i = isl_set_subtract(guard_i, zero);
-		guard = isl_set_intersect(guard, guard_i);
-	}
-
-	return guard;
-}
-
-/* Internal data structure for extract_size_of_type.
- * "type" specifies the name of the space that we want to extract.
- * "res" is used to store the subset of that space.
- */
-struct ppcg_extract_size_data {
-	const char *type;
-	isl_set *res;
-};
-
-/* This function is called for each set in a union_set.
- * If the name of the set matches data->type, we store the
- * set in data->res.
- */
-static isl_stat extract_size_of_type(__isl_take isl_set *size, void *user)
-{
-	struct ppcg_extract_size_data *data = user;
-	const char *name;
-
-	name = isl_set_get_tuple_name(size);
-	if (name && !strcmp(name, data->type)) {
-		data->res = size;
-		return isl_stat_error;
-	}
-
-	isl_set_free(size);
-	return isl_stat_ok;
-}
-
-/* Given a union map { kernel[i] -> *[...] },
- * return the range in the space called "type" for the kernel with
- * sequence number "id".
- */
-static __isl_give isl_set *extract_sizes(__isl_keep isl_union_map *sizes,
-	const char *type, int id)
-{
-	isl_space *space;
-	isl_set *dom;
-	isl_union_set *local_sizes;
-	struct ppcg_extract_size_data data = { type, NULL };
-
-	if (!sizes)
-		return NULL;
-
-	space = isl_union_map_get_space(sizes);
-	space = isl_space_set_from_params(space);
-	space = isl_space_add_dims(space, isl_dim_set, 1);
-	space = isl_space_set_tuple_name(space, isl_dim_set, "kernel");
-	dom = isl_set_universe(space);
-	dom = isl_set_fix_si(dom, isl_dim_set, 0, id);
-
-	local_sizes = isl_union_set_apply(isl_union_set_from_set(dom),
-					isl_union_map_copy(sizes));
-	isl_union_set_foreach_set(local_sizes, &extract_size_of_type, &data);
-	isl_union_set_free(local_sizes);
-	return data.res;
-}
-
-/* Given a singleton set, extract the first (at most *len) elements
- * of the single integer tuple into *sizes and update *len if needed.
- */
-static void read_sizes_from_set(__isl_take isl_set *set, int *sizes, int *len)
-{
-	int i;
-	int dim;
-
-	if (!set)
-		return;
-
-	dim = isl_set_dim(set, isl_dim_set);
-	if (dim < *len)
-		*len = dim;
-
-	for (i = 0; i < *len; ++i) {
-		isl_val *v;
-
-		v = isl_set_plain_get_val_if_fixed(set, isl_dim_set, i);
-		assert(v);
-
-		sizes[i] = isl_val_get_num_si(v);
-		isl_val_free(v);
-	}
-
-	isl_set_free(set);
-}
-
-/* Add the map { kernel[id] -> type[sizes] } to gen->used_sizes,
- * if the option debug->dump_sizes is set.
- */
-static void set_used_sizes(struct gpu_gen *gen, const char *type, int id,
-	int *sizes, int len)
-{
-	int i;
-	isl_space *space;
-	isl_map *map;
-
-	if (!gen->options->debug->dump_sizes)
-		return;
-
-	space = isl_union_map_get_space(gen->used_sizes);
-	space = isl_space_set_from_params(space);
-	space = isl_space_add_dims(space, isl_dim_set, 1);
-	space = isl_space_set_tuple_name(space, isl_dim_set, "kernel");
-	space = isl_space_from_domain(space);
-	space = isl_space_add_dims(space, isl_dim_out, len);
-	space = isl_space_set_tuple_name(space, isl_dim_out, type);
-
-	map = isl_map_universe(space);
-	map = isl_map_fix_si(map, isl_dim_in, 0, id);
-	for (i = 0; i < len; ++i)
-		map = isl_map_fix_si(map, isl_dim_out, i, sizes[i]);
-
-	gen->used_sizes = isl_union_map_add_map(gen->used_sizes, map);
-}
-
-/* Extract user specified "tile" sizes from the "sizes" command line option,
- * defaulting to option->tile_size in each dimension.
- * *tile_len contains the maximum number of tile sizes needed.
- * Update *tile_len to the number of specified tile sizes, if any, and
- * return a pointer to the tile sizes (or NULL on error).
- * Add the effectively used sizes to gen->used_sizes.
- */
-static int *read_tile_sizes(struct gpu_gen *gen, int *tile_len)
-{
-	int n;
-	int *tile_size;
-	isl_set *size;
-
-	tile_size = isl_alloc_array(gen->ctx, int, *tile_len);
-	if (!tile_size)
-		return NULL;
-	for (n = 0; n < *tile_len; ++n)
-		tile_size[n] = gen->options->tile_size;
-
-	size = extract_sizes(gen->sizes, "tile", gen->kernel_id);
-	read_sizes_from_set(size, tile_size, tile_len);
-	set_used_sizes(gen, "tile", gen->kernel_id, tile_size, *tile_len);
-
-	return tile_size;
-}
-
-/* Extract user specified "block" sizes from the "sizes" command line option,
- * after filling in some potentially useful defaults.
- */
-static void read_block_sizes(struct ppcg_kernel *kernel,
-	__isl_keep isl_union_map *sizes)
-{
-	isl_set *size;
-
-	if (kernel->n_block > 3)
-		kernel->n_block = 3;
-	switch (kernel->n_block) {
-	case 1:
-		kernel->block_dim[0] = 512;
-		break;
-	case 2:
-		kernel->block_dim[0] = 32;
-		kernel->block_dim[1] = 16;
-		break;
-	default:
-		kernel->block_dim[0] = 32;
-		kernel->block_dim[1] = 4;
-		kernel->block_dim[2] = 4;
-		break;
-	}
-
-	size = extract_sizes(sizes, "block", kernel->id);
-	read_sizes_from_set(size, kernel->block_dim, &kernel->n_block);
-}
-
-/* Extract user specified "grid" sizes from the "sizes" command line option,
- * after filling in some potentially useful defaults.
- */
-static void read_grid_sizes(struct ppcg_kernel *kernel,
-	__isl_keep isl_union_map *sizes)
-{
-	isl_set *size;
-
-	if (kernel->n_grid > 2)
-		kernel->n_grid = 2;
-	switch (kernel->n_grid) {
-	case 1:
-		kernel->grid_dim[0] = 32768;
-		break;
-	default:
-		kernel->grid_dim[0] = 256;
-		kernel->grid_dim[1] = 256;
-		break;
-	}
-
-	size = extract_sizes(sizes, "grid", kernel->id);
-	read_sizes_from_set(size, kernel->grid_dim, &kernel->n_grid);
-}
-
-/* Extract user specified grid and block sizes from the gen->sizes
- * command line option after filling in some potentially useful defaults.
- * Store the extracted sizes in "kernel".
- * Add the effectively used sizes to gen->used_sizes.
- */
-static void read_grid_and_block_sizes(struct ppcg_kernel *kernel,
-	struct gpu_gen *gen)
-{
-	read_block_sizes(kernel, gen->sizes);
-	read_grid_sizes(kernel, gen->sizes);
-	set_used_sizes(gen, "block", kernel->id,
-					    kernel->block_dim, kernel->n_block);
-	set_used_sizes(gen, "grid", kernel->id,
-					    kernel->grid_dim, kernel->n_grid);
-}
-
-static void *free_stmts(struct gpu_stmt *stmts, int n)
-{
-	int i;
-
-	if (!stmts)
-		return NULL;
-
-	for (i = 0; i < n; ++i) {
-		struct gpu_stmt_access *access, *next;
-
-		for (access = stmts[i].accesses; access; access = next) {
-			next = access->next;
-			isl_id_free(access->ref_id);
-			isl_map_free(access->access);
-			isl_map_free(access->tagged_access);
-			free(access);
-		}
-
-		isl_id_free(stmts[i].id);
-	}
-	free(stmts);
-
-	return NULL;
-}
-
-/* Add parameters p[i] with identifiers "ids" to "set",
- * with bounds to 0 <= p[i] < size[i].
- */
-__isl_give isl_set *add_bounded_parameters(__isl_take isl_set *set,
-	int *size, __isl_keep isl_id_list *ids)
-{
-	int i, len;
-	unsigned nparam;
-
-	len = isl_id_list_n_id(ids);
-	nparam = isl_set_dim(set, isl_dim_param);
-	set = isl_set_add_dims(set, isl_dim_param, len);
-
-	for (i = 0; i < len; ++i) {
-		isl_id *id;
-
-		id = isl_id_list_get_id(ids, i);
-		set = isl_set_set_dim_id(set, isl_dim_param, nparam + i, id);
-		set = isl_set_lower_bound_si(set, isl_dim_param, nparam + i, 0);
-		set = isl_set_upper_bound_si(set, isl_dim_param,
-					    nparam + i, size[i] - 1);
-	}
-
-	return set;
-}
-
-/* Add "len" parameters p[i] with identifiers "ids" and intersect "set"
- * with
- *
- *	{ : 0 <= p[i] < size[i] }
- *
- * or an overapproximation.
- */
-static __isl_give isl_set *add_bounded_parameters_dynamic(
-	__isl_take isl_set *set, __isl_keep isl_multi_pw_aff *size,
-	__isl_keep isl_id_list *ids)
-{
-	int i, len;
-	unsigned nparam;
-	isl_space *space;
-	isl_local_space *ls;
-
-	len = isl_multi_pw_aff_dim(size, isl_dim_out);
-	nparam = isl_set_dim(set, isl_dim_param);
-	set = isl_set_add_dims(set, isl_dim_param, len);
-
-	for (i = 0; i < len; ++i) {
-		isl_id *id;
-
-		id = isl_id_list_get_id(ids, i);
-		set = isl_set_set_dim_id(set, isl_dim_param, nparam + i, id);
-	}
-
-	space = isl_space_params(isl_set_get_space(set));
-	ls = isl_local_space_from_space(space);
-	for (i = 0; i < len; ++i) {
-		isl_pw_aff *param, *size_i, *zero;
-		isl_set *bound;
-
-		param = isl_pw_aff_var_on_domain(isl_local_space_copy(ls),
-						isl_dim_param, nparam + i);
-
-		size_i = isl_multi_pw_aff_get_pw_aff(size, i);
-		bound = isl_pw_aff_lt_set(isl_pw_aff_copy(param), size_i);
-		bound = isl_set_from_basic_set(isl_set_simple_hull(bound));
-		set = isl_set_intersect_params(set, bound);
-
-		zero = isl_pw_aff_zero_on_domain(isl_local_space_copy(ls));
-		bound = isl_pw_aff_ge_set(param, zero);
-		set = isl_set_intersect_params(set, bound);
-	}
-	isl_local_space_free(ls);
-
-	return set;
-}
-
-/* Return the union of all tagged access relations in the group.
- */
-static __isl_give isl_union_map *group_tagged_access_relation(
-	struct gpu_array_ref_group *group)
-{
-	int i;
-	isl_union_map *access;
-
-	access = isl_union_map_empty(isl_map_get_space(group->access));
-	for (i = 0; i < group->n_ref; ++i) {
-		isl_map *map_i;
-
-		map_i = isl_map_copy(group->refs[i]->tagged_access);
-		access = isl_union_map_union(access,
-					    isl_union_map_from_map(map_i));
-	}
-
-	return access;
-}
-
-/* Return the extent of "array", recomputed from the bounds.
- * The recomputed extent may be simpler than the original extent.
- */
-static __isl_give isl_set *array_extent(struct gpu_array_info *array)
-{
-	int i;
-	isl_id *id;
-	isl_space *space;
-	isl_local_space *ls;
-	isl_set *extent;
-
-	id = isl_set_get_tuple_id(array->extent);
-	space = isl_set_get_space(array->extent);
-	extent = isl_set_universe(isl_space_copy(space));
-	ls = isl_local_space_from_space(space);
-	for (i = 0; i < array->n_index; ++i) {
-		isl_pw_aff *bound;
-		isl_aff *aff;
-		isl_pw_aff *index;
-		isl_set *lt;
-
-		extent = isl_set_lower_bound_si(extent, isl_dim_set, i, 0);
-
-		aff = isl_aff_var_on_domain(isl_local_space_copy(ls),
-						isl_dim_set, i);
-		index = isl_pw_aff_from_aff(aff);
-		bound = isl_multi_pw_aff_get_pw_aff(array->bound, i);
-		bound = isl_pw_aff_from_range(bound);
-		bound = isl_pw_aff_add_dims(bound, isl_dim_in, array->n_index);
-		bound = isl_pw_aff_set_tuple_id(bound, isl_dim_in,
-						isl_id_copy(id));
-		lt = isl_pw_aff_lt_set(index, bound);
-		extent = isl_set_intersect(extent, lt);
-	}
-	isl_local_space_free(ls);
-	isl_id_free(id);
-
-	return extent;
-}
-
-/* Return a map from the first group->shared_tile->depth dimensions
- * of the computed schedule to the array tile in
- * global memory that corresponds to the shared memory copy.
- *
- * In particular, return a map
- *
- *	{ D[i] -> A[a] }
- *
- * with constraints
- *
- *	tile_offset(i) <= a <= tile_offset(i) + tile_size - 1		(1)
- *
- * and
- *
- *	0 <= a <= array_size - 1					(2)
- *
- * Note that if some stride has been detected (i.e., when
- * group->shared_tile->bound[i].shift is set), then a in (1) refers
- * to the shifted and scaled down version.
- *
- * Constraints (1) are obtained by mapping the size constraints on the
- * shared/private memory tile back to the access relation.
- * Constraints (2) are obtained from the (recomputed) extent.
- */
-static __isl_give isl_map *group_tile(struct gpu_array_ref_group *group)
-{
-	int i;
-	int n_index = group->array->n_index;
-	isl_map *tile;
-	isl_space *space;
-	isl_set *local;
-	isl_set *extent;
-
-	space = isl_multi_aff_get_space(group->shared_tile->tiling);
-	space = isl_space_range(space);
-	local = isl_set_universe(space);
-	for (i = 0; i < n_index; ++i) {
-		isl_val *bound;
-
-		local = isl_set_lower_bound_si(local, isl_dim_set, i, 0);
-		bound = isl_val_copy(group->shared_tile->bound[i].size);
-		bound = isl_val_sub_ui(bound, 1);
-		local = isl_set_upper_bound_val(local, isl_dim_set, i, bound);
-	}
-	local = isl_set_preimage_multi_aff(local,
-				isl_multi_aff_copy(group->shared_tile->tiling));
-	tile = isl_set_unwrap(local);
-	extent = array_extent(group->array);
-	tile = isl_map_intersect_range(tile, extent);
-
-	return tile;
-}
-
-/* Given a mapping "iterator_map" from the AST schedule to a domain,
- * return the corresponding mapping from the AST schedule to
- * to the outer kernel->copy_schedule_dim dimensions of
- * the schedule computed by PPCG for this kernel.
- *
- * Note that kernel->copy_schedule_dim is at least as large as
- * the largest depth of any array reference group associated to the kernel.
- * This is needed as the returned schedule is used to extract a mapping
- * to the outer tile->depth dimensions in transform_index.
- */
-static __isl_give isl_pw_multi_aff *compute_sched_to_copy(
-	struct ppcg_kernel *kernel, __isl_take isl_pw_multi_aff *iterator_map)
-{
-	isl_union_pw_multi_aff *upma;
-	isl_pw_multi_aff *pma;
-	isl_space *space;
-
-	space = isl_space_range(isl_pw_multi_aff_get_space(iterator_map));
-	space = isl_space_from_domain(space);
-	space = isl_space_add_dims(space, isl_dim_out,
-					kernel->copy_schedule_dim);
-
-	upma = isl_union_pw_multi_aff_copy(kernel->copy_schedule);
-	pma = isl_union_pw_multi_aff_extract_pw_multi_aff(upma, space);
-	isl_union_pw_multi_aff_free(upma);
-
-	return isl_pw_multi_aff_pullback_pw_multi_aff(pma, iterator_map);
-}
-
-/* If max_shared_memory is not set to infinity (-1), then make
- * sure that the total amount of shared memory required by the
- * array reference groups mapped to shared memory by "kernel"
- * is no larger than this maximum.
- *
- * We apply a greedy approach and discard (keep in global memory)
- * those groups that would result in a total memory size that
- * is larger than the maximum.
- *
- * This function should be called after any function that may
- * affect the decision on whether to place a reference group
- * in private, shared or global memory.
- */
-static void check_shared_memory_bound(struct ppcg_kernel *kernel)
-{
-	int i, j;
-	isl_val *left, *size;
-
-	if (kernel->options->max_shared_memory < 0)
-		return;
-
-	left = isl_val_int_from_si(kernel->ctx,
-				    kernel->options->max_shared_memory);
-
-	for (i = 0; i < kernel->n_array; ++i) {
-		struct gpu_local_array_info *local = &kernel->array[i];
-
-		for (j = 0; j < local->n_group; ++j) {
-			struct gpu_array_ref_group *group;
-			enum ppcg_group_access_type type;
-
-			group = local->groups[j];
-			type = gpu_array_ref_group_type(group);
-			if (type != ppcg_access_shared)
-				continue;
-
-			size = gpu_array_tile_size(group->shared_tile);
-			size = isl_val_mul_ui(size, local->array->size);
-
-			if (isl_val_le(size, left)) {
-				left = isl_val_sub(left, size);
-				continue;
-			}
-			isl_val_free(size);
-
-			group->shared_tile =
-					gpu_array_tile_free(group->shared_tile);
-		}
-	}
-
-	isl_val_free(left);
-}
-
-/* Mark all arrays of "kernel" that have an array reference group
- * that is not mapped to private or shared memory as
- * accessing the corresponding global device memory.
- */
-static void mark_global_arrays(struct ppcg_kernel *kernel)
-{
-	int i, j;
-
-	for (i = 0; i < kernel->n_array; ++i) {
-		struct gpu_local_array_info *local = &kernel->array[i];
-
-		if (local->global)
-			continue;
-		for (j = 0; j < local->n_group; ++j) {
-			if (gpu_array_ref_group_tile(local->groups[j]))
-				continue;
-
-			local->global = 1;
-			local->array->global = 1;
-			break;
-		}
-	}
-}
-
-/* Compute a tiling for all the array reference groups in "kernel".
- */
-static void compute_group_tilings(struct ppcg_kernel *kernel)
-{
-	int i, j;
-
-	for (i = 0; i < kernel->n_array; ++i) {
-		struct gpu_local_array_info *array = &kernel->array[i];
-
-		for (j = 0; j < array->n_group; ++j)
-			gpu_array_ref_group_compute_tiling(array->groups[j]);
-	}
-}
-
-/* Compute the effective grid size as a list of the sizes in each dimension.
- *
- * The grid size specified by the user or set by default
- * in read_grid_sizes() and applied by the block filter,
- * may be too large for the given code in the sense that
- * it may contain blocks that don't need to execute anything.
- * We therefore don't return this grid size, but instead the
- * smallest grid size that ensures that all blocks that actually
- * execute code are included in the grid.
- *
- * We first extract a description of the grid, i.e., the possible values
- * of the block ids, from the domain elements in "domain" and
- * kernel->block_filter.
- * The block ids are parameters in kernel->block_filter.
- * We simply need to change them into set dimensions.
- *
- * Then, for each block dimension, we compute the maximal value of the block id
- * and add one.
- */
-static __isl_give isl_multi_pw_aff *extract_grid_size(
-	struct ppcg_kernel *kernel, __isl_take isl_union_set *domain)
-{
-	int i;
-	isl_set *grid;
-	isl_set *context;
-	isl_multi_pw_aff *size;
-
-	domain = isl_union_set_intersect(domain,
-				    isl_union_set_copy(kernel->block_filter));
-	grid = isl_union_set_params(domain);
-	grid = isl_set_from_params(grid);
-	grid = isl_set_add_dims(grid, isl_dim_set, kernel->n_grid);
-	for (i = 0; i < kernel->n_grid; ++i) {
-		int pos;
-		isl_id *id;
-
-		id = isl_id_list_get_id(kernel->block_ids, i);
-		pos = isl_set_find_dim_by_id(grid, isl_dim_param, id);
-		isl_id_free(id);
-		assert(pos >= 0);
-		grid = isl_set_equate(grid, isl_dim_param, pos, isl_dim_set, i);
-		grid = isl_set_project_out(grid, isl_dim_param, pos, 1);
-	}
-
-	grid = isl_set_coalesce(grid);
-	size = ppcg_size_from_extent(grid);
-	context = isl_set_params(isl_set_copy(kernel->context));
-	return isl_multi_pw_aff_gist(size, context);
-}
-
-/* Compute the size of a fixed bounding box around the origin and "set",
- * where "set" is assumed to contain only non-negative elements,
- * and store the results in "size".
- * In particular, compute the maximal value of "set" in each direction
- * and add one.
- */
-static void extract_fixed_size(__isl_take isl_set *set, int *size)
-{
-	int i, n;
-	isl_local_space *ls;
-	isl_aff *obj;
-
-	n = isl_set_dim(set, isl_dim_set);
-	ls = isl_local_space_from_space(isl_set_get_space(set));
-	obj = isl_aff_zero_on_domain(ls);
-	for (i = 0; i < n; ++i) {
-		isl_val *max;
-
-		obj = isl_aff_set_coefficient_si(obj, isl_dim_in, i, 1);
-		max = isl_set_max_val(set, obj);
-		size[i] = isl_val_get_num_si(max) + 1;
-		isl_val_free(max);
-		obj = isl_aff_set_coefficient_si(obj, isl_dim_in, i, 0);
-	}
-	isl_aff_free(obj);
-	isl_set_free(set);
-}
-
-/* Compute the effective block size as a list of the sizes in each dimension
- * and store the sizes in kernel->block_dim.
- *
- * The block size specified by the user or set by default
- * in read_block_sizes() and applied by the thread filter,
- * may be too large for the given code in the sense that
- * it may contain threads that don't need to execute anything.
- * We therefore update this block size in kernel->block_dim
- * to the smallest block size that ensures that all threads
- * that actually execute code are included in the block.
- *
- * The set of possible values of the thread ids is obtained from
- * the domain elements "domain" and kernel->thread_filter.
- * The current implementation eliminates all parameters, ensuring
- * that the size is a fixed constant in each dimension.
- * In principle we could also compute parametric sizes.
- * We would have to make sure to project out all b%d and t%d parameters,
- * however.
- */
-static isl_stat extract_block_size(struct ppcg_kernel *kernel,
-	__isl_take isl_union_set *domain)
-{
-	int i;
-	int nparam;
-	isl_set *block;
-
-	domain = isl_union_set_intersect(domain,
-				    isl_union_set_copy(kernel->thread_filter));
-	block = isl_union_set_params(domain);
-	block = isl_set_from_params(block);
-	block = isl_set_add_dims(block, isl_dim_set, kernel->n_block);
-	for (i = 0; i < kernel->n_block; ++i) {
-		int pos;
-		isl_id *id;
-
-		if (!block)
-			return isl_stat_error;
-
-		id = isl_id_list_get_id(kernel->thread_ids, i);
-		pos = isl_set_find_dim_by_id(block, isl_dim_param, id);
-		isl_id_free(id);
-		if (pos < 0)
-			isl_die(isl_set_get_ctx(block), isl_error_internal,
-				"missing constraints on thread identifier",
-				block = isl_set_free(block));
-		block = isl_set_equate(block, isl_dim_param, pos,
-					isl_dim_set, i);
-	}
-	nparam = isl_set_dim(block, isl_dim_param);
-	block = isl_set_project_out(block, isl_dim_param, 0, nparam);
-
-	if (!block)
-		return isl_stat_error;
-
-	extract_fixed_size(block, kernel->block_dim);
-
-	return isl_stat_ok;
-}
-
-struct ppcg_kernel *ppcg_kernel_free(struct ppcg_kernel *kernel)
-{
-	int i, j;
-
-	if (!kernel)
-		return NULL;
-
-	isl_id_list_free(kernel->block_ids);
-	isl_id_list_free(kernel->thread_ids);
-	isl_multi_pw_aff_free(kernel->grid_size);
-	isl_ast_expr_free(kernel->grid_size_expr);
-	isl_set_free(kernel->context);
-	isl_union_set_free(kernel->core);
-	isl_union_set_free(kernel->arrays);
-	isl_union_pw_multi_aff_free(kernel->contraction);
-	isl_union_set_free(kernel->expanded_domain);
-	isl_space_free(kernel->space);
-	isl_ast_node_free(kernel->tree);
-	isl_union_set_free(kernel->block_filter);
-	isl_union_set_free(kernel->thread_filter);
-	isl_union_pw_multi_aff_free(kernel->copy_schedule);
-	isl_union_set_free(kernel->sync_writes);
-
-	for (i = 0; i < kernel->n_array; ++i) {
-		struct gpu_local_array_info *array = &kernel->array[i];
-
-		for (j = 0; j < array->n_group; ++j)
-			gpu_array_ref_group_free(array->groups[j]);
-		free(array->groups);
-
-		isl_multi_pw_aff_free(array->bound);
-		isl_ast_expr_free(array->bound_expr);
-	}
-	free(kernel->array);
-
-	for (i = 0; i < kernel->n_var; ++i) {
-		free(kernel->var[i].name);
-		isl_vec_free(kernel->var[i].size);
-	}
-	free(kernel->var);
-
-	free(kernel);
-
-	return NULL;
-}
-
-/* Wrapper around ppcg_kernel_free for use as a isl_id_set_free_user callback.
- */
-static void ppcg_kernel_free_wrap(void *user)
-{
-	struct ppcg_kernel *kernel = user;
-
-	ppcg_kernel_free(kernel);
-}
-
-static void create_kernel_var(isl_ctx *ctx, struct gpu_array_ref_group *group,
-	struct ppcg_kernel_var *var)
-{
-	int j;
-	struct gpu_array_tile *tile;
-	isl_printer *p;
-
-	var->array = group->array;
-
-	var->type = gpu_array_ref_group_type(group);
-	tile = gpu_array_ref_group_tile(group);
-
-	p = isl_printer_to_str(ctx);
-	p = gpu_array_ref_group_print_name(group, p);
-	var->name = isl_printer_get_str(p);
-	isl_printer_free(p);
-
-	var->size = isl_vec_alloc(ctx, group->array->n_index);
-
-	for (j = 0; j < group->array->n_index; ++j)
-		var->size = isl_vec_set_element_val(var->size, j,
-					    isl_val_copy(tile->bound[j].size));
-}
-
-static int create_kernel_vars(struct ppcg_kernel *kernel)
-{
-	int i, j, n;
-
-	n = 0;
-	for (i = 0; i < kernel->n_array; ++i) {
-		struct gpu_local_array_info *array = &kernel->array[i];
-
-		for (j = 0; j < array->n_group; ++j) {
-			struct gpu_array_ref_group *group = array->groups[j];
-			enum ppcg_group_access_type type;
-
-			type = gpu_array_ref_group_type(group);
-			if (type != ppcg_access_global)
-				++n;
-		}
-	}
-
-	kernel->n_var = n;
-	kernel->var = isl_calloc_array(kernel->ctx, struct ppcg_kernel_var, n);
-	if (!kernel->var)
-		return -1;
-
-	n = 0;
-	for (i = 0; i < kernel->n_array; ++i) {
-		struct gpu_local_array_info *array = &kernel->array[i];
-
-		for (j = 0; j < array->n_group; ++j) {
-			struct gpu_array_ref_group *group = array->groups[j];
-			enum ppcg_group_access_type type;
-
-			type = gpu_array_ref_group_type(group);
-			if (type == ppcg_access_global)
-				continue;
-			create_kernel_var(kernel->ctx, group, &kernel->var[n]);
-			++n;
-		}
-	}
-
-	return 0;
-}
-
-/* Replace "pa" by the zero function defined over the universe domain
- * in the space of "pa".
- */
-static __isl_give isl_pw_aff *set_universally_zero(__isl_take isl_pw_aff *pa)
-{
-	isl_space *space;
-	isl_aff *zero;
-
-	space = isl_space_domain(isl_pw_aff_get_space(pa));
-	isl_pw_aff_free(pa);
-	zero = isl_aff_zero_on_domain(isl_local_space_from_space(space));
-
-	return isl_pw_aff_from_aff(zero);
-}
-
-/* The sizes of the arrays on the host that have been computed by
- * extract_array_info may depend on the parameters.  Use the extra
- * constraints on the parameters that are valid at "host_domain"
- * to simplify these expressions and store the results in kernel->array.
- *
- * We only need these localized bounds for arrays that are accessed
- * by the current kernel.  If we have found at least one reference group
- * then the array is accessed by the kernel.
- *
- * The resulting sizes may be functions that are nowhere defined
- * in case the access function cannot possibly access anything inside
- * the kernel for some reason.  If so, they are replaced by the zero
- * function.  Since the access function cannot actually access anything,
- * there is no harm in printing the array sizes as zero.
- */
-static void localize_bounds(struct ppcg_kernel *kernel,
-	__isl_keep isl_set *host_domain)
-{
-	int i, j;
-	isl_set *context;
-
-	context = isl_set_copy(host_domain);
-	context = isl_set_params(context);
-
-	for (i = 0; i < kernel->n_array; ++i) {
-		struct gpu_local_array_info *local = &kernel->array[i];
-		isl_multi_pw_aff *bound;
-		int n_index;
-
-		if (local->n_group == 0)
-			continue;
-
-		n_index = local->array->n_index;
-		bound = isl_multi_pw_aff_copy(local->array->bound);
-
-		for (j = 0; j < n_index; ++j) {
-			isl_pw_aff *pwaff;
-			int empty;
-
-			pwaff = isl_multi_pw_aff_get_pw_aff(bound, j);
-			pwaff = isl_pw_aff_gist(pwaff, isl_set_copy(context));
-			empty = isl_pw_aff_is_empty(pwaff);
-			if (empty < 0)
-				pwaff = isl_pw_aff_free(pwaff);
-			else if (empty)
-				pwaff = set_universally_zero(pwaff);
-			bound = isl_multi_pw_aff_set_pw_aff(bound, j, pwaff);
-		}
-
-		local->n_index = n_index;
-		local->bound = bound;
-	}
-	isl_set_free(context);
-}
-
-/* Create the array of gpu_local_array_info structures "array"
- * inside "kernel".  The number of elements in this array is
- * the same as the number of arrays in "prog".
- * Initialize the "array" field of each local array to point
- * to the corresponding array in "prog".
- */
-static struct ppcg_kernel *ppcg_kernel_create_local_arrays(
-	struct ppcg_kernel *kernel, struct gpu_prog *prog)
-{
-	int i;
-	isl_ctx *ctx;
-
-	ctx = isl_set_get_ctx(prog->context);
-	kernel->array = isl_calloc_array(ctx,
-			    struct gpu_local_array_info, prog->n_array);
-	if (!kernel->array)
-		return ppcg_kernel_free(kernel);
-	kernel->n_array = prog->n_array;
-
-	for (i = 0; i < prog->n_array; ++i)
-		kernel->array[i].array = &prog->array[i];
-
-	return kernel;
-}
-
-/* Does "kernel" need to be passed an argument corresponding to array "i"?
- *
- * The argument is only needed if the kernel accesses this device memory.
- */
-int ppcg_kernel_requires_array_argument(struct ppcg_kernel *kernel, int i)
-{
-	return kernel->array[i].global;
-}
-
-/* Find the element in gen->stmt that has the given "id".
- * Return NULL if no such gpu_stmt can be found.
- */
-static struct gpu_stmt *find_stmt(struct gpu_prog *prog, __isl_keep isl_id *id)
-{
-	int i;
-
-	for (i = 0; i < prog->n_stmts; ++i) {
-		if (id == prog->stmts[i].id)
-			break;
-	}
-
-	return i < prog->n_stmts ? &prog->stmts[i] : NULL;
-}
-
-void ppcg_kernel_stmt_free(void *user)
-{
-	struct ppcg_kernel_stmt *stmt = user;
-
-	if (!stmt)
-		return;
-
-	switch (stmt->type) {
-	case ppcg_kernel_copy:
-		isl_ast_expr_free(stmt->u.c.index);
-		isl_ast_expr_free(stmt->u.c.local_index);
-		break;
-	case ppcg_kernel_domain:
-		isl_id_to_ast_expr_free(stmt->u.d.ref2expr);
-		break;
-	case ppcg_kernel_sync:
-		break;
-	}
-
-	free(stmt);
-}
-
-/* Return the gpu_stmt_access in the list "accesses" that corresponds
- * to "ref_id".
- */
-static struct gpu_stmt_access *find_access(struct gpu_stmt_access *accesses,
-	__isl_keep isl_id *ref_id)
-{
-	struct gpu_stmt_access *access;
-
-	for (access = accesses; access; access = access->next)
-		if (access->ref_id == ref_id)
-			return access;
-
-	return NULL;
-}
-
-/* Return the index of the array called "name" in the list of arrays.
- */
-static int find_array_index(struct ppcg_kernel *kernel, const char *name)
-{
-	int i;
-
-	for (i = 0; i < kernel->n_array; ++i)
-		if (!strcmp(name, kernel->array[i].array->name))
-			return i;
-
-	return -1;
-}
-
-/* Internal data structure for the index and AST expression transformation
- * callbacks for pet_stmt_build_ast_exprs.
- *
- * "kernel" is the kernel for which are computing AST expressions and
- * may be NULL if we are not inside a kernel.
- * "accesses" is the list of gpu_stmt_access in the statement.
- * "iterator_map" expresses the statement iterators in terms of
- * the AST loop iterators.
- * "sched2copy" expresses the outer copy_schedule_dim dimensions of
- * the kernel schedule in terms of the AST loop iterators and
- * may be NULL if we are not inside a kernel.
- *
- * The following fields are set in transform_index and used in transform_expr.
- * "array" is the array that is being accessed.
- * "global" is set if the global array is accessed (rather than
- * shared/private memory).
- * "local_array" refers to information on the array specialized
- * to the current kernel.
- */
-struct ppcg_transform_data {
-    struct ppcg_options *options;
-    struct ppcg_kernel *kernel;
-	struct gpu_stmt_access *accesses;
-	isl_pw_multi_aff *iterator_map;
-	isl_pw_multi_aff *sched2copy;
-
-	struct gpu_array_info *array;
-	int global;
-	struct gpu_local_array_info *local_array;
-};
-
-/* Return a pointer to the gpu_array_ref_group in "local"
- * that contains the reference "access".
- * Return NULL if no such group can be found.
- */
-static struct gpu_array_ref_group *find_ref_group(
-	struct gpu_local_array_info *local, struct gpu_stmt_access *access)
-{
-	int i, j;
-
-	for (i = 0; i < local->n_group; ++i) {
-		struct gpu_array_ref_group *group = local->groups[i];
-
-		for (j = 0; j < group->n_ref; ++j)
-			if (group->refs[j] == access)
-				return group;
-	}
-
-	return NULL;
-}
-
-/* Given an index expression "index" of the form
- *
- *	L -> F(A),
- *
- * with F(A) either A or some subfield of A and L the AST loop iterators,
- * and a tiling "tiling" of the form
- *
- *	[L -> A] -> T
- *
- * apply the tiling to the outer array in the index expression to obtain
- *
- *	L -> T(A)
- *
- * If F(A) is some subfield of A, then separate the member access
- * into the base index expression and the field index expression,
- * apply the tiling to the base index expression and combine the result
- * with the field index expression.
- *
- * If F(A) is A, then modify index to keep track of the iterators
- *
- *	L -> [L -> A]
- *
- * and combine the result with the tiling to obtain a tiled index expression
- * in terms of the AST loop iterators
- *
- *	L -> T
- */
-static __isl_give isl_multi_pw_aff *tile_outer(
-	__isl_take isl_multi_pw_aff *index, __isl_take isl_multi_pw_aff *tiling)
-{
-	isl_bool is_wrapping;
-	isl_space *space;
-	isl_multi_pw_aff *mpa;
-
-	is_wrapping = isl_multi_pw_aff_range_is_wrapping(index);
-	if (is_wrapping < 0)
-		goto error;
-	if (is_wrapping) {
-		isl_multi_pw_aff *field;
-
-		field = isl_multi_pw_aff_copy(index);
-		field = isl_multi_pw_aff_range_factor_range(field);
-		index = isl_multi_pw_aff_range_factor_domain(index);
-		index = tile_outer(index, tiling);
-		return isl_multi_pw_aff_range_product(index, field);
-	}
-
-	space = isl_space_domain(isl_multi_pw_aff_get_space(index));
-	space = isl_space_map_from_set(space);
-	mpa = isl_multi_pw_aff_identity(space);
-	index = isl_multi_pw_aff_range_product(mpa, index);
-	index = isl_multi_pw_aff_pullback_multi_pw_aff(tiling, index);
-
-	return index;
-error:
-	isl_multi_pw_aff_free(index);
-	isl_multi_pw_aff_free(tiling);
-	return NULL;
-}
-
-/* Index transformation callback for pet_stmt_build_ast_exprs.
- *
- * "index" expresses the array indices in terms of statement iterators
- *
- * We first reformulate "index" in terms of the AST loop iterators.
- * Then we check if we are accessing the global array or
- * a shared/private copy.  In particular, if we are not inside a kernel
- * then we must be accessing a global array.
- * In the former case, we simply return
- * the updated index.  If "index" is an affine expression rather
- * than an array access, then we also return the updated index here.
- *
- * If no reference groups have been computed for the array,
- * then we can only be accessing the global array.
- *
- * Otherwise, we apply the tiling to the index.
- * This tiling is of the form
- *
- *	[D -> A] -> T
- *
- * where D corresponds to the outer tile->depth dimensions of
- * the kernel schedule.
- * The index is of the form
- *
- *	L -> A
- *
- * We update the tiling to refer to the AST loop iterators
- *
- *	[L -> A] -> T
- *
- * and combine it with the index to obtain a tiled index expression in terms
- * of the AST loop iterators
- *
- *	L -> T
- *
- * Note that while the tiling applies directly to an outer array.
- * the index may refer to some subfield of this outer array.
- * In such cases, the result will refer to the same subfield of the tile.
- * That is, an index expression of the form  L -> F(A) will be transformed
- * into an index expression of the form L -> F(T).
- */
-static __isl_give isl_multi_pw_aff *transform_index(
-	__isl_take isl_multi_pw_aff *index, __isl_keep isl_id *ref_id,
-	void *user)
-{
-	struct ppcg_transform_data *data = user;
-	struct gpu_stmt_access *access;
-	struct gpu_array_ref_group *group;
-	struct gpu_array_tile *tile;
-	isl_pw_multi_aff *iterator_map;
-	int i;
-	int dim;
-	const char *name;
-	isl_space *space;
-	isl_multi_pw_aff *tiling;
-	isl_pw_multi_aff *pma;
-	isl_pw_multi_aff *sched2depth;
-
-	data->array = NULL;
-
-	iterator_map = isl_pw_multi_aff_copy(data->iterator_map);
-	index = isl_multi_pw_aff_pullback_pw_multi_aff(index, iterator_map);
-
-	if (!data->kernel)
-		return index;
-
-	access = find_access(data->accesses, ref_id);
-	if (!access)
-		return index;
-	if (!isl_map_has_tuple_name(access->access, isl_dim_out))
-		return index;
-
-	name = get_outer_array_name(access->access);
-	i = find_array_index(data->kernel, name);
-	if (i < 0)
-		isl_die(isl_multi_pw_aff_get_ctx(index), isl_error_internal,
-			"cannot find array",
-			return isl_multi_pw_aff_free(index));
-	data->local_array = &data->kernel->array[i];
-	data->array = data->local_array->array;
-
-	group = find_ref_group(data->local_array, access);
-	if (!group) {
-		data->global = 1;
-		return index;
-	}
-
-	tile = gpu_array_ref_group_tile(group);
-	data->global = !tile;
-	if (!tile)
-		return index;
-
-	space = isl_space_domain(isl_multi_aff_get_space(tile->tiling));
-	space = isl_space_range(isl_space_unwrap(space));
-	space = isl_space_map_from_set(space);
-	pma = isl_pw_multi_aff_identity(space);
-	sched2depth = isl_pw_multi_aff_copy(data->sched2copy);
-	dim = isl_pw_multi_aff_dim(sched2depth, isl_dim_out);
-	sched2depth = isl_pw_multi_aff_drop_dims(sched2depth, isl_dim_out,
-					    tile->depth, dim - tile->depth);
-	pma = isl_pw_multi_aff_product(sched2depth, pma);
-	tiling = isl_multi_pw_aff_from_multi_aff(
-				    isl_multi_aff_copy(tile->tiling));
-	tiling = isl_multi_pw_aff_pullback_pw_multi_aff(tiling, pma);
-
-	index = tile_outer(index, tiling);
-
-	return index;
-}
-
-/* Dereference "expr" by adding an index [0].
- * The original "expr" is assumed not to have any indices.
- *
- * If "expr" is a member access, then the dereferencing needs
- * to be applied to the structure argument of this member access.
- */
-static __isl_give isl_ast_expr *dereference(__isl_take isl_ast_expr *expr)
-{
-	isl_ctx *ctx;
-	isl_ast_expr *arg0, *res;
-	isl_ast_expr_list *list;
-
-	arg0 = isl_ast_expr_get_op_arg(expr, 0);
-	if (!arg0)
-		return isl_ast_expr_free(expr);
-	if (isl_ast_expr_get_type(arg0) == isl_ast_expr_op &&
-	    isl_ast_expr_get_op_type(arg0) == isl_ast_op_member) {
-		isl_ast_expr *arg;
-
-		arg = isl_ast_expr_get_op_arg(arg0, 0);
-		arg = dereference(arg);
-		arg0 = isl_ast_expr_set_op_arg(arg0, 0, arg);
-		expr = isl_ast_expr_set_op_arg(expr, 0, arg0);
-
-		return expr;
-	}
-	isl_ast_expr_free(arg0);
-
-	ctx = isl_ast_expr_get_ctx(expr);
-	res = isl_ast_expr_from_val(isl_val_zero(ctx));
-	list = isl_ast_expr_list_from_ast_expr(res);
-	res = isl_ast_expr_get_op_arg(expr, 0);
-	res = isl_ast_expr_access(res, list);
-	isl_ast_expr_free(expr);
-
-	return res;
-}
-
-/* Linearize the index expression "expr" based on the array bounds
- * of "array".
- *
- * That is, transform expression
- *
- *	A[i_0][i_1]...[i_n]
- *
- * to
- *
- *	A[(..((i_0 * b_1 + i_1) ... ) * b_n + i_n]
- *
- * where b_0, b_1, ..., b_n are the bounds on the array.
- *
- * If the base of "expr" is a member access, then the linearization needs
- * to be applied to the structure argument of this member access.
- *
- * In the base case, if "expr" has no arguments (other than the name of
- * the array), then we are passing an entire array to a function.
- * In this case, there is nothing to linearize.
- * Note that at this point an expression with no arguments can
- * only be an entire array because the scalar case and
- * the case of single struct are handled by the caller.
- *
- * If the number of specified index expressions in "expr"
- * is smaller than the dimension of the accessed array,
- * then the missing i_j also do not appear in the linearized expression.
- * Furthermore, since such an expression does not refer to a single
- * element while the default linearized expression would refer to
- * a single element, we return the expression
- *
- *	A + (..((i_0 * b_1 + i_1) ... ) * b_l + i_l)
- *
- * instead.  Note that because of the special case handling above,
- * we can assume here that there is at least one index expression.
- */
-__isl_give isl_ast_expr *gpu_local_array_info_linearize_index(
-	struct gpu_local_array_info *array, __isl_take isl_ast_expr *expr)
-{
-	int i, n;
-	isl_ast_expr *arg0;
-	isl_ast_expr *res;
-	isl_ast_expr_list *list;
-
-	arg0 = isl_ast_expr_get_op_arg(expr, 0);
-	if (isl_ast_expr_get_type(arg0) == isl_ast_expr_op &&
-	    isl_ast_expr_get_op_type(arg0) == isl_ast_op_member) {
-		isl_ast_expr *arg;
-
-		arg = isl_ast_expr_get_op_arg(arg0, 0);
-		arg = gpu_local_array_info_linearize_index(array, arg);
-		arg0 = isl_ast_expr_set_op_arg(arg0, 0, arg);
-		expr = isl_ast_expr_set_op_arg(expr, 0, arg0);
-
-		return expr;
-	}
-	isl_ast_expr_free(arg0);
-
-	if (isl_ast_expr_get_op_n_arg(expr) == 1)
-		return expr;
-
-	n = isl_ast_expr_get_op_n_arg(expr);
-	res = isl_ast_expr_get_op_arg(expr, 1);
-	for (i = 1; i < array->n_index; ++i) {
-		isl_ast_expr *expr_i;
-
-		expr_i = isl_ast_expr_get_op_arg(array->bound_expr, 1 + i);
-		res = isl_ast_expr_mul(res, expr_i);
-
-		if (i + 1 >= n)
-			continue;
-		expr_i = isl_ast_expr_get_op_arg(expr, i + 1);
-		res = isl_ast_expr_add(res, expr_i);
-	}
-
-	if (1 + array->n_index > n) {
-		res = isl_ast_expr_add(isl_ast_expr_get_op_arg(expr, 0), res);
-	} else {
-		list = isl_ast_expr_list_from_ast_expr(res);
-		res = isl_ast_expr_get_op_arg(expr, 0);
-		res = isl_ast_expr_access(res, list);
-	}
-
-	isl_ast_expr_free(expr);
-
-	return res;
-}
-
-/* AST expression transformation callback for pet_stmt_build_ast_exprs.
- *
- * If the AST expression refers to an array that is not accessed
- * at all, then this means the value of the expression is not used,
- * so we might as well print zero (NULL pointer) instead.
- *
- * If the AST expression refers to a global scalar that is not
- * a read-only scalar, then its address was passed to the kernel and
- * we need to dereference it.
- *
- * If the AST expression refers to an access to a global array,
- * then we linearize the access exploiting the bounds in data->local_array.
- */
-static __isl_give isl_ast_expr *transform_expr(__isl_take isl_ast_expr *expr,
-	__isl_keep isl_id *id, void *user)
-{
-	struct ppcg_transform_data *data = user;
-
-	if (!data->array)
-		return expr;
-	if (!data->array->accessed) {
-		isl_ctx *ctx;
-
-		ctx = isl_ast_expr_get_ctx(expr);
-		isl_ast_expr_free(expr);
-		return isl_ast_expr_from_val(isl_val_zero(ctx));
-	}
-	if (gpu_array_is_read_only_scalar(data->array))
-		return expr;
-	if (!data->global)
-		return expr;
-	if (data->array->n_index == 0)
-		return dereference(expr);
-	if (!data->array->linearize)
-		return expr;
-
-	return gpu_local_array_info_linearize_index(data->local_array, expr);
-}
-
-/* This function is called for each instance of a user statement
- * in the kernel "kernel", identified by "gpu_stmt".
- * "kernel" may be NULL if we are not inside a kernel.
- *
- * We attach a struct ppcg_kernel_stmt to the "node", containing
- * a computed AST expression for each access, through an annotation
- * with name "user".
- * These AST expressions are computed from iterator_map,
- * which expresses the domain
- * elements in terms of the generated loops, and sched2copy,
- * which expresses the outer copy_schedule_dim dimensions of
- * the kernel schedule computed by PPCG in terms of the generated loops.
- */
-static __isl_give isl_ast_node *create_domain_leaf(
-	struct ppcg_kernel *kernel, __isl_take isl_ast_node *node,
-	__isl_keep isl_ast_build *build, struct gpu_stmt *gpu_stmt,
-    struct gpu_gen *gen)
-{
-	struct ppcg_transform_data data;
-	struct ppcg_kernel_stmt *stmt;
-	isl_ctx *ctx;
-	isl_id *id;
-	isl_pw_multi_aff *sched2copy;
-	isl_map *map;
-	isl_pw_multi_aff *iterator_map;
-	isl_union_map *schedule;
-
-	if (!node)
-		return NULL;
-	ctx = isl_ast_node_get_ctx(node);
-
-	stmt = isl_calloc_type(ctx, struct ppcg_kernel_stmt);
-	if (!stmt)
-		return isl_ast_node_free(node);
-
-	schedule = isl_ast_build_get_schedule(build);
-	map = isl_map_reverse(isl_map_from_union_map(schedule));
-	iterator_map = isl_pw_multi_aff_from_map(map);
-	if (kernel)
-		sched2copy = compute_sched_to_copy(kernel,
-					isl_pw_multi_aff_copy(iterator_map));
-	else
-		sched2copy = NULL;
-
-	stmt->type = ppcg_kernel_domain;
-	stmt->u.d.stmt = gpu_stmt;
-
-	data.kernel = kernel;
-	data.accesses = stmt->u.d.stmt->accesses;
-	data.iterator_map = iterator_map;
-	data.sched2copy = sched2copy;
-	stmt->u.d.ref2expr = gen->build_ast_expr(stmt->u.d.stmt->stmt,
-					    build, &transform_index, &data,
-					    &transform_expr, &data);
-
-	isl_pw_multi_aff_free(iterator_map);
-	isl_pw_multi_aff_free(sched2copy);
-
-	id = isl_id_alloc(ctx, "user", stmt);
-	id = isl_id_set_free_user(id, &ppcg_kernel_stmt_free);
-	return isl_ast_node_set_annotation(node, id);
-}
-
-/* This function is called for each statement node in the AST
- * for copying to or from shared/private memory.
- * Attach a pointer to a ppcg_kernel_stmt representing the copy
- * statement to the node.
- * The statement name is "read" or "write", depending on whether we are
- * reading from global memory or writing to global memory.
- *
- * The schedule is of the form
- *
- *	type[D -> A] -> L
- *
- * where D corresponds to the outer tile->depth dimensions of
- * the kernel schedule, A to the global array and L to the outer
- * generated AST schedule.
- * We compute the inverse and strip off the type, resulting in
- *
- *	L -> [D -> A]
- *
- * We combine this mapping with on the one hand the projection
- *
- *	[D -> A] -> A
- *
- * and on the other hand the group tiling
- *
- *	[D -> A] -> T
- *
- * resulting in
- *
- *	L -> A		and 	L -> T
- *
- * and store the corresponding expressions in stmt->index and stmt->local_index,
- * where stmt points to the ppcg_kernel_stmt that is attached to the node.
- * stmt->index is linearized if the global memory array is linearized.
- */
-static __isl_give isl_ast_node *create_access_leaf(struct ppcg_kernel *kernel,
-	struct gpu_array_ref_group *group, __isl_take isl_ast_node *node,
-	__isl_keep isl_ast_build *build)
-{
-	struct ppcg_kernel_stmt *stmt;
-	struct gpu_array_tile *tile;
-	isl_id *id;
-	isl_ast_expr *expr;
-	isl_space *space;
-	isl_map *access;
-	isl_pw_multi_aff *pma, *pma2;
-	const char *type;
-
-	stmt = isl_calloc_type(kernel->ctx, struct ppcg_kernel_stmt);
-	if (!stmt)
-		return isl_ast_node_free(node);
-
-	access = isl_map_from_union_map(isl_ast_build_get_schedule(build));
-	type = isl_map_get_tuple_name(access, isl_dim_in);
-	stmt->u.c.read = !strcmp(type, "read");
-	access = isl_map_reverse(access);
-	pma = isl_pw_multi_aff_from_map(access);
-	pma = isl_pw_multi_aff_reset_tuple_id(pma, isl_dim_out);
-
-	space = isl_space_range(isl_pw_multi_aff_get_space(pma));
-	space = isl_space_unwrap(space);
-	pma2 = isl_pw_multi_aff_range_map(space);
-	pma2 = isl_pw_multi_aff_pullback_pw_multi_aff(pma2,
-						    isl_pw_multi_aff_copy(pma));
-	expr = isl_ast_build_access_from_pw_multi_aff(build, pma2);
-	if (group->array->linearize)
-		expr = gpu_local_array_info_linearize_index(group->local_array,
-							    expr);
-	stmt->u.c.index = expr;
-
-	tile = gpu_array_ref_group_tile(group);
-	pma2 = isl_pw_multi_aff_from_multi_aff(
-					    isl_multi_aff_copy(tile->tiling));
-	pma2 = isl_pw_multi_aff_pullback_pw_multi_aff(pma2, pma);
-	expr = isl_ast_build_access_from_pw_multi_aff(build, pma2);
-	stmt->u.c.local_index = expr;
-
-	stmt->u.c.array = group->array;
-	stmt->u.c.local_array = group->local_array;
-	stmt->type = ppcg_kernel_copy;
-
-	id = isl_id_alloc(kernel->ctx, "copy", stmt);
-	id = isl_id_set_free_user(id, &ppcg_kernel_stmt_free);
-	return isl_ast_node_set_annotation(node, id);
-}
-
-/* Create a synchronization ppcg_kernel_stmt and
- * attach it to the node "node" representing the synchronization.
- */
-static __isl_give isl_ast_node *create_sync_leaf(
-	struct ppcg_kernel *kernel, __isl_take isl_ast_node *node,
-	__isl_keep isl_ast_build *build)
-{
-	struct ppcg_kernel_stmt *stmt;
-	isl_id *id;
-
-	stmt = isl_calloc_type(kernel->ctx, struct ppcg_kernel_stmt);
-	if (!stmt)
-		return isl_ast_node_free(node);
-
-	stmt->type = ppcg_kernel_sync;
-	id = isl_id_alloc(kernel->ctx, "sync", stmt);
-	id = isl_id_set_free_user(id, &ppcg_kernel_stmt_free);
-	return isl_ast_node_set_annotation(node, id);
-}
-
-/* Build AST expressions for the device array sizes of all arrays in "prog"
- * that require allocation on the device using "build", as well as
- * for the original array sizes of all arrays that need to be declared
- * on the host.
- * "node" is freed in case of error.
- */
-static __isl_give isl_ast_node *build_array_bounds(
-	__isl_take isl_ast_node *node, struct gpu_prog *prog,
-	__isl_keep isl_ast_build *build)
-{
-	int i;
-
-	for (i = 0; i < prog->n_array; ++i) {
-		struct gpu_array_info *array = &prog->array[i];
-		isl_multi_pw_aff *size;
-		isl_ast_expr *expr;
-
-		if (!gpu_array_requires_device_allocation(array))
-			continue;
-
-		size = isl_multi_pw_aff_copy(array->bound);
-		expr = ppcg_build_size_expr(size, build);
-		array->bound_expr = expr;
-		if (!expr)
-			return isl_ast_node_free(node);
-	}
-
-	for (i = 0; i < prog->n_array; ++i) {
-		struct gpu_array_info *array = &prog->array[i];
-		isl_set *extent;
-		isl_multi_pw_aff *size;
-		isl_ast_expr *expr;
-
-		if (!array->declare_local)
-			continue;
-		extent = isl_set_copy(array->declared_extent);
-		size = ppcg_size_from_extent(extent);
-		expr = ppcg_build_size_expr(size, build);
-		array->declared_size = expr;
-		if (!expr)
-			return isl_ast_node_free(node);
-	}
-
-	return node;
-}
-
-/* Internal data structure for at_domain.
- *
- * "prog" represents the entire scop.
- * "kernel" points to the kernel to which the current schedule node
- * belongs.  It is set by before_mark and reset by after_mark.
- * It may be NULL if we are outside any kernel.
- */
-struct ppcg_at_domain_data {
-    struct gpu_prog *prog;
-    struct gpu_gen *gen;
-    struct ppcg_kernel *kernel;
-};
-
-/* This function is called for each instance of a user statement
- * in the kernel.  This may be one of the original user statements
- * or a statement introduced by PPCG.
- *
- * We first check if the statement id corresponds to a gpu statement,
- * which indicates the statement is an original user statement. Any statement
- * that is not an original user statement has been introduced by PPCG and
- * requires special handling.
- *
- * If the user statement is one of the original user statements, then we call
- * create_domain_leaf.  If it is "init_device", then we call
- * build_array_bounds.  Otherwise, we check if it is a copy or synchronization
- * statement and call the appropriate functions.  Statements that copy an array
- * to/from the device do not need any further treatment.
- * Neither does "clear_device".
- */
-static __isl_give isl_ast_node *at_domain(__isl_take isl_ast_node *node,
-	__isl_keep isl_ast_build *build, void *user)
-{
-	struct ppcg_at_domain_data *data = user;
-	struct gpu_stmt *gpu_stmt;
-	isl_ast_expr *expr, *arg;
-	isl_id *id;
-	int is_sync;
-	const char *name;
-	void *p;
-
-	expr = isl_ast_node_user_get_expr(node);
-	arg = isl_ast_expr_get_op_arg(expr, 0);
-	id = isl_ast_expr_get_id(arg);
-	name = isl_id_get_name(id);
-	p = isl_id_get_user(id);
-	isl_ast_expr_free(expr);
-	isl_ast_expr_free(arg);
-
-	gpu_stmt = find_stmt(data->prog, id);
-	is_sync = gpu_tree_id_is_sync(id, data->kernel);
-	isl_id_free(id);
-
-	if (gpu_stmt)
-		return create_domain_leaf(data->kernel, node, build, gpu_stmt,
-                                  data->gen);
-
-	if (!prefixcmp(name, "to_device_") || !prefixcmp(name, "from_device_"))
-		return node;
-	if (!strcmp(name, "init_device"))
-		return build_array_bounds(node, data->prog, build);
-	if (!strcmp(name, "clear_device"))
-		return node;
-	if (is_sync < 0)
-		return isl_ast_node_free(node);
-	if (!strcmp(name, "read") || !strcmp(name, "write")) {
-		struct gpu_array_ref_group *group = p;
-		return create_access_leaf(data->kernel, group, node, build);
-	}
-	if (!is_sync)
-		isl_die(data->prog->ctx, isl_error_internal,
-			"unknown statement type",
-			return isl_ast_node_free(node));
-	return create_sync_leaf(data->kernel, node, build);
-}
-
-/* Given a set of wrapped references "ref", return the corresponding
- * access relations based on the tagged access relations "tagged".
- *
- * The elements of "ref" are of the form
- *
- *	[D -> R]
- *
- * with D an iteration domains and R a reference.
- * The elements of "tagged" are of the form
- *
- *	[D -> R] -> A
- *
- * with A an array.
- *
- * Extend "tagged" to include the iteration domain in the range, i.e.,
- *
- *	[D -> R] -> [D -> A]
- *
- * apply the result to "ref" and then unwrap the resulting set
- * to obtain relations of the form
- *
- *	D -> A
- */
-static __isl_give isl_union_map *wrapped_reference_to_access(
-	__isl_take isl_union_set *ref, __isl_take isl_union_map *tagged)
-{
-	isl_union_map *tag2access;
-
-	tag2access = isl_union_map_copy(tagged);
-	tag2access = isl_union_map_universe(tag2access);
-	tag2access = isl_union_set_unwrap(isl_union_map_domain(tag2access));
-	tag2access = isl_union_map_domain_map(tag2access);
-	tag2access = isl_union_map_range_product(tag2access, tagged);
-
-	ref = isl_union_set_coalesce(ref);
-	ref = isl_union_set_apply(ref, tag2access);
-
-	return isl_union_set_unwrap(ref);
-}
-
-/* Given an access relation "access" from one or more array reference groups,
- * remove those reads if ("read" is 1) or writes (if "read" is 0)
- * that are only needed to communicate data within
- * the same iteration of "sched".
- * The domain of "sched" corresponds to the original statement instances,
- * i.e., those that appear in the domains of the access relations.
- * "tagged" contains all tagged access relations to all
- * the array reference groups accessed by "access" from statement
- * instances scheduled by "sched".
- *
- * If the access is a read then it is either an element of
- *
- *	live_in union (range flow)
- *
- * where live_in and flow may be overapproximations, or
- * it reads an uninitialized value (that is not live-in because
- * there is an intermediate kill) or it reads a value that was
- * written within the same (compound) statement instance.
- * If the access is a write then it is either an element of
- *
- *	live_out union (domain flow)
- *
- * or it writes a value that is never read (and is not live-out
- * because of an intermediate kill) or only
- * within the same (compound) statement instance.
- * In both cases, the access relation is also a subset of
- * the group access relation.
- *
- * The cases where an uninitialized value is read or a value is written
- * that is never read or where the dataflow occurs within a statement
- * instance are also considered local and may also be removed.
- *
- * Essentially, we compute the intersection of "access" with either
- *
- *	live_in union (range non-local-flow)
- *
- * or
- *
- *	live_out union (domain non-local-flow)
- *
- * We first construct a relation "local"
- *
- *	[[D -> R] -> [D' -> R']]
- *
- * of pairs of domain iterations accessing the reference group
- * and references in the group that are coscheduled by "sched".
- *
- * If this relation does not intersect the dataflow dependences,
- * then there is nothing we can possibly remove, unless the dataflow
- * dependences themselves only relate a subset of the accesses.
- * In particular, the accesses may not be involved in any dataflow
- * dependences, either because they are uninitialized reads/dead writes
- * or because the dataflow occurs inside a statement instance.
- *
- * Since the computation below may break up the access relation
- * into smaller pieces, we only perform the intersection with
- * the non-local dependent accesses if the local pairs
- * intersect the dataflow dependences.  Otherwise, we intersect
- * with the universe of the non-local dependent accesses.
- * This should at least remove accesses from statements that
- * do not participate in any dependences.
- *
- * In particular, we remove the "local" dataflow dependences from
- * the set of all dataflow dependences, or at least those
- * that may contribute to a domain/range that intersects
- * the domain of "access".
- * Note that if the potential dataflow dependences are an overapproximation
- * of the actual dataflow dependences, then the result remains an
- * overapproximation of the non-local dataflow dependences.
- * Copying to/from global memory is only needed for the references
- * in the domain/range of the result or for accesses that are live out/in
- * for the entire scop.
- *
- * We therefore map the domain/range of the "external" relation
- * to the corresponding access relation and take the union with
- * the live out/in relation.
- */
-static __isl_give isl_union_map *remove_local_accesses(
-	struct gpu_prog *prog, __isl_take isl_union_map *tagged,
-	__isl_take isl_union_map *access, __isl_take isl_union_map *sched,
-	int read)
-{
-	int empty;
-	isl_union_pw_multi_aff *tagger;
-	isl_union_set *domain, *access_domain;
-	isl_union_map *local, *external, *universe;
-	isl_union_set *tag_set;
-
-	if (isl_union_map_is_empty(access)) {
-		isl_union_map_free(sched);
-		isl_union_map_free(tagged);
-		return access;
-	}
-
-	tagger = isl_union_pw_multi_aff_copy(prog->scop->tagger);
-	domain = isl_union_map_domain(isl_union_map_copy(tagged));
-	tagger = isl_union_pw_multi_aff_intersect_domain(tagger,
-					isl_union_set_copy(domain));
-	sched = isl_union_map_preimage_domain_union_pw_multi_aff(sched, tagger);
-
-	local = isl_union_map_apply_range(sched,
-			    isl_union_map_reverse(isl_union_map_copy(sched)));
-	local = isl_union_map_intersect(local,
-			isl_union_map_copy(prog->scop->tagged_dep_flow));
-
-	empty = isl_union_map_is_empty(local);
-
-	external = isl_union_map_copy(prog->scop->tagged_dep_flow);
-	universe = isl_union_map_universe(isl_union_map_copy(access));
-	access_domain = isl_union_map_domain(universe);
-	domain = isl_union_set_universe(domain);
-	universe = isl_union_set_unwrap(domain);
-	universe = isl_union_map_intersect_domain(universe, access_domain);
-	domain = isl_union_map_wrap(universe);
-	if (read)
-		external = isl_union_map_intersect_range(external, domain);
-	else
-		external = isl_union_map_intersect_domain(external, domain);
-	external = isl_union_map_intersect_params(external,
-				isl_set_copy(prog->scop->context));
-	external = isl_union_map_subtract(external, local);
-
-	if (read) {
-		tag_set = isl_union_map_range(external);
-		external = wrapped_reference_to_access(tag_set, tagged);
-		external = isl_union_map_union(external,
-				isl_union_map_copy(prog->scop->live_in));
-	} else {
-		tag_set = isl_union_map_domain(external);
-		external = wrapped_reference_to_access(tag_set, tagged);
-		external = isl_union_map_union(external,
-				isl_union_map_copy(prog->scop->live_out));
-	}
-
-	if (empty < 0)
-		external = isl_union_map_free(external);
-	else if (empty)
-		external = isl_union_map_universe(external);
-
-	access = isl_union_map_intersect(access, external);
-
-	return access;
-}
-
-/* Given an access relation "access" from "group", remove those reads
- * if ("read" is 1) or writes (if "read" is 0) that are only needed to
- * communicate data within the same iteration of the schedule "prefix"
- * at the position where the copying of the group is inserted.
- * That is, the output dimension of "prefix"
- * is equal to tile->depth.
- * The domain of "prefix" corresponds to the original statement instances,
- * i.e., those that appear in the domains of the access relations.
- *
- * Extract the tagged access relation of "group" and
- * then call remove_local_accesses.
- */
-static __isl_give isl_union_map *remove_local_accesses_group(
-	struct ppcg_kernel *kernel, struct gpu_array_ref_group *group,
-	__isl_take isl_union_map *access, __isl_keep isl_union_map *prefix,
-	int read)
-{
-	isl_union_map *sched, *tagged;
-
-	if (isl_union_map_is_empty(access))
-		return access;
-
-	tagged = group_tagged_access_relation(group);
-	sched = isl_union_map_copy(prefix);
-
-	return remove_local_accesses(kernel->prog, tagged, access, sched, read);
-}
-
-/* Build an access AST expression for the effective grid size using "build".
- * Store the result in kernel->grid_size_expr.
- */
-static isl_stat build_grid_size(struct ppcg_kernel *kernel,
-	__isl_keep isl_ast_build *build)
-{
-	isl_multi_pw_aff *size;
-
-	size = isl_multi_pw_aff_copy(kernel->grid_size);
-	size = isl_multi_pw_aff_set_tuple_name(size, isl_dim_out, "grid");
-	kernel->grid_size_expr = ppcg_build_size_expr(size, build);
-
-	if (!kernel->grid_size_expr)
-		return isl_stat_error;
-	return isl_stat_ok;
-}
-
-/* Build access AST expressions for the localized array sizes using "build".
- * Store the result in local->bound_expr.
- * Only do this for arrays for which localized bounds have been computed.
- */
-static isl_stat build_local_array_sizes(struct ppcg_kernel *kernel,
-	__isl_keep isl_ast_build *build)
-{
-	int i;
-
-	for (i = 0; i < kernel->n_array; ++i) {
-		struct gpu_local_array_info *local = &kernel->array[i];
-		isl_multi_pw_aff *size;
-
-		if (local->n_group == 0)
-			continue;
-		size = isl_multi_pw_aff_copy(local->bound);
-		local->bound_expr = ppcg_build_size_expr(size, build);
-		if (!local->bound_expr)
-			return isl_stat_error;
-	}
-
-	return isl_stat_ok;
-}
-
-/* Build access AST expressions for the effective grid size and
- * the localized array sizes using "build".
- */
-static isl_stat build_grid_and_local_array_sizes(struct ppcg_kernel *kernel,
-	__isl_keep isl_ast_build *build)
-{
-	if (build_grid_size(kernel, build) < 0)
-		return isl_stat_error;
-	if (build_local_array_sizes(kernel, build) < 0)
-		return isl_stat_error;
-	return isl_stat_ok;
-}
-
-/* This function is called before the AST generator starts traversing
- * the schedule subtree of a node with mark "mark".
- *
- * If the mark is called "kernel", store the kernel pointer in data->kernel
- * for use in at_domain and build AST expressions for the grid size and
- * the localized array sizes.
- */
-static isl_stat before_mark(__isl_keep isl_id *mark,
-	__isl_keep isl_ast_build *build, void *user)
-{
-	struct ppcg_at_domain_data *data = user;
-
-	if (!mark)
-		return isl_stat_error;
-	if (!strcmp(isl_id_get_name(mark), "kernel")) {
-		data->kernel = isl_id_get_user(mark);
-		if (build_grid_and_local_array_sizes(data->kernel, build) < 0)
-			return isl_stat_error;
-	}
-	return isl_stat_ok;
-}
-
-/* This function is called after the AST generator has finished traversing
- * the schedule subtree of a mark node.  "node" points to the corresponding
- * mark AST node.
- *
- * If the mark is called "kernel", then replace "node" by a user node
- * that "calls" the kernel, representing the launch of the kernel.
- * The original "node" is stored inside the kernel object so that
- * it can be used to print the device code.
- * Note that this assumes that a kernel is only launched once.
- * Also clear data->kernel.
- */
-static __isl_give isl_ast_node *after_mark(__isl_take isl_ast_node *node,
-        __isl_keep isl_ast_build *build, void *user)
-{
-	isl_ctx *ctx;
-	isl_id *id;
-	isl_ast_expr *expr;
-	isl_ast_expr_list *list;
-	struct ppcg_kernel *kernel;
-	struct ppcg_at_domain_data *data = user;
-
-	ctx = isl_ast_node_get_ctx(node);
-	id = isl_ast_node_mark_get_id(node);
-	if (!id)
-		return isl_ast_node_free(node);
-	if (strcmp(isl_id_get_name(id), "kernel") || !data->kernel) {
-		isl_id_free(id);
-		return node;
-	}
-	kernel = data->kernel;
-	data->kernel = NULL;
-	kernel->space = isl_ast_build_get_schedule_space(build);
-	kernel->tree = isl_ast_node_mark_get_node(node);
-	isl_ast_node_free(node);
-
-	expr = isl_ast_expr_from_id(isl_id_copy(id));
-	list = isl_ast_expr_list_alloc(ctx, 0);
-	expr = isl_ast_expr_call(expr, list);
-	node = isl_ast_node_alloc_user(expr);
-	node = isl_ast_node_set_annotation(node, id);
-
-	return node;
-}
-
-static isl_bool update_depth(__isl_keep isl_schedule_node *node, void *user)
-{
-	int *depth = user;
-	int node_depth;
-
-	if (isl_schedule_node_get_type(node) != isl_schedule_node_leaf)
-		return isl_bool_true;
-	node_depth = isl_schedule_node_get_schedule_depth(node);
-	if (node_depth > *depth)
-		*depth = node_depth;
-
-	return isl_bool_false;
-}
-
-/* Use isl to generate code for both the host and the device
- * from "schedule".
- * The device code is marked by "kernel" mark nodes in the schedule tree,
- * containing a pointer to a ppcg_kernel object.
- * The returned AST only contains the AST for the host code.
- * The ASTs for the device code are embedded in ppcg_kernel objects
- * attached to the leaf nodes that call "kernel".
- */
-__isl_give isl_ast_node *generate_code(struct gpu_gen *gen,
-	__isl_take isl_schedule *schedule)
-{
-	struct ppcg_at_domain_data data;
-	isl_ast_build *build;
-	isl_ast_node *tree;
-	isl_id_list *iterators;
-	int depth;
-
-    data.prog = gen->prog;
-    data.gen = gen;
-	data.kernel = NULL;
-
-	depth = 0;
-	if (isl_schedule_foreach_schedule_node_top_down(schedule, &update_depth,
-						&depth) < 0)
-		return NULL;
-	build = isl_ast_build_alloc(gen->prog->ctx);
-	iterators = ppcg_scop_generate_names(gen->prog->scop, depth, "c");
-	build = isl_ast_build_set_iterators(build, iterators);
-	build = isl_ast_build_set_at_each_domain(build, &at_domain, &data);
-	build = isl_ast_build_set_before_each_mark(build, &before_mark, &data);
-	build = isl_ast_build_set_after_each_mark(build, &after_mark, &data);
-	if (gen->prog->scop->options->debug->dump_final_schedule)
-		isl_schedule_dump(schedule);
-	tree = isl_ast_build_node_from_schedule(build, schedule);
-	isl_ast_build_free(build);
-
-	return tree;
-}
-
-__isl_give isl_union_map *extract_sizes_from_str(isl_ctx *ctx, const char *str)
-{
-	if (!str)
-		return NULL;
-	return isl_union_map_read_from_str(ctx, str);
-}
-
-/* Can "node" be tiled and then mapped to block and thread identifiers?
- * That is, is it permutable with at least one coincident dimension?
- */
-static int is_permutable(__isl_keep isl_schedule_node *node)
-{
-	if (!node)
-		return -1;
-
-	if (isl_schedule_node_get_type(node) != isl_schedule_node_band)
-		return 0;
-	if (!isl_schedule_node_band_get_permutable(node))
-		return 0;
-	if (isl_schedule_node_band_n_member(node) < 1)
-		return 0;
-	if (!isl_schedule_node_band_member_get_coincident(node, 0))
-		return 0;
-
-	return 1;
-}
-
-/* A isl_schedule_foreach_schedule_node_top_down callback
- * for setting *any_permutable and aborting the search
- * if "node" is a permutable band with coincident dimensions.
- * Otherwise, continue searching.
- */
-static isl_bool set_permutable(__isl_keep isl_schedule_node *node, void *user)
-{
-	int *any_permutable = user;
-	int permutable;
-
-	permutable = is_permutable(node);
-	if (permutable < 0)
-		return isl_bool_error;
-	if (!permutable)
-		return isl_bool_true;
-
-	*any_permutable = 1;
-
-	return isl_bool_error;
-}
-
-/* Does the subtree rooted at "node" have any suitably permutable band nodes?
- * That is, does it have any nodes that are permutable and that
- * have a least one coincident dimension?
- */
-static int subtree_has_permutable_bands(__isl_keep isl_schedule_node *node)
-{
-	int any_parallelism = 0;
-
-	if (isl_schedule_node_foreach_descendant_top_down(node, &set_permutable,
-						&any_parallelism) < 0 &&
-	    !any_parallelism)
-		return -1;
-
-	return any_parallelism;
-}
-
-/* Does "schedule" contain any permutable band with at least one coincident
- * member?
- */
-int has_any_permutable_node(__isl_keep isl_schedule *schedule)
-{
-	isl_schedule_node *root;
-	int any_permutable;
-
-	root = isl_schedule_get_root(schedule);
-	any_permutable = subtree_has_permutable_bands(root);
-	isl_schedule_node_free(root);
-
-	return any_permutable;
-}
-
-/* Is "node" a candidate for mapping to block and thread identifiers?
- * In particular, is it permutable with at least one coincident dimension?
- * Alternatively, does the subtree rooted at "node" not contain
- * any such permutable node?  Filter nodes are skipped in this case,
- * because a band node will be inserted in front of the returned
- * node and this is not possible for filter nodes that are children
- * of set or sequence nodes.
- */
-static int is_candidate(__isl_keep isl_schedule_node *node)
-{
-	int permutable;
-
-	if (isl_schedule_node_get_type(node) == isl_schedule_node_leaf)
-		return 1;
-	permutable = is_permutable(node);
-	if (permutable < 0 || permutable)
-		return permutable;
-	if (isl_schedule_node_get_type(node) == isl_schedule_node_filter)
-		return 0;
-	permutable = subtree_has_permutable_bands(node);
-	if (permutable < 0)
-		return -1;
-	return !permutable;
-}
-
-/* Is "node" the outermost node in its branch that can be tiled
- * and then mapped to block and thread identifiers?
- * If there are no such nodes in the subtree at "node" and
- * if "node" is not a filter node, then it is accepted too.
- */
-static int is_outer_tilable(__isl_keep isl_schedule_node *node)
-{
-	int tilable;
-	isl_schedule_node *ancestor;
-
-	tilable = is_candidate(node);
-	if (tilable < 0)
-		return -1;
-	if (!tilable)
-		return 0;
-
-	tilable = 0;
-	ancestor = isl_schedule_node_copy(node);
-	while (isl_schedule_node_has_parent(ancestor)) {
-		ancestor = isl_schedule_node_parent(ancestor);
-
-		tilable = is_candidate(ancestor);
-		if (tilable < 0 || tilable)
-			break;
-	}
-
-	isl_schedule_node_free(ancestor);
-	return tilable < 0 ? -1 : !tilable;
-}
-
-/* Collect the references to all writes in "group".
- * Each reference is represented by a universe set in a space
- *
- *	[S[i,j] -> R[]]
- *
- * with S[i,j] the statement instance space and R[] the array reference.
- */
-static __isl_give isl_union_set *group_tagged_writes(
-	struct gpu_array_ref_group *group)
-{
-	int i;
-	isl_space *space;
-	isl_union_set *writes;
-
-	space = isl_map_get_space(group->access);
-	writes = isl_union_set_empty(space);
-	for (i = 0; i < group->n_ref; ++i) {
-		isl_space *space;
-		isl_set *writes_i;
-
-		if (!group->refs[i]->write)
-			continue;
-
-		space = isl_map_get_space(group->refs[i]->tagged_access);
-		space = isl_space_domain(space);
-		writes_i = isl_set_universe(space);
-		writes = isl_union_set_add_set(writes, writes_i);
-	}
-
-	return writes;
-}
-
-/* Is there any write access in "group" that requires synchronization
- * on a write to global memory?
- * We currently take into account all writes that would require
- * synchronization at the thread level depth, but if the copying
- * for this group is performed at an outer level, then we do not
- * actually need to take into account dependences at intermediate levels.
- */
-static int any_sync_writes_in_group(struct ppcg_kernel *kernel,
-	struct gpu_array_ref_group *group)
-{
-	isl_union_set *writes;
-	int empty, disjoint;
-
-	empty = isl_union_set_is_empty(kernel->sync_writes);
-	if (empty < 0)
-		return -1;
-	if (empty)
-		return 0;
-
-	writes = group_tagged_writes(group);
-	disjoint = isl_union_set_is_disjoint(kernel->sync_writes, writes);
-	isl_union_set_free(writes);
-
-	return disjoint < 0 ? -1 : !disjoint;
-}
-
-/* Collect the references to all writes in "kernel" that write directly
- * to global or shared memory, i.e., that are not mapped to private memory.
- * Each reference is represented by a universe set in a space
- *
- *	[S[i,j] -> R[]]
- *
- * with S[i,j] the statement instance space and R[] the array reference.
- */
-static __isl_give isl_union_set *collect_non_private_tagged_writes(
-	struct ppcg_kernel *kernel)
-{
-	isl_union_set *writes;
-	int i, j;
-
-	writes = isl_union_set_empty(isl_union_set_get_space(kernel->arrays));
-
-	for (i = 0; i < kernel->n_array; ++i) {
-		struct gpu_local_array_info *array = &kernel->array[i];
-
-		for (j = 0; j < array->n_group; ++j) {
-			struct gpu_array_ref_group *group = array->groups[j];
-			enum ppcg_group_access_type type;
-			isl_union_set *writes_ij;
-
-			if (!group->write)
-				continue;
-			type = gpu_array_ref_group_type(group);
-			if (type == ppcg_access_private)
-				continue;
-			writes_ij = group_tagged_writes(group);
-			writes = isl_union_set_union(writes, writes_ij);
-		}
-	}
-
-	return writes;
-}
-
-/* Are there any direct writes to global memory that require
- * synchronization?
- */
-static int any_global_or_shared_sync_writes(struct ppcg_kernel *kernel)
-{
-	isl_union_set *writes;
-	int empty, disjoint;
-
-	empty = isl_union_set_is_empty(kernel->sync_writes);
-	if (empty < 0)
-		return -1;
-	if (empty)
-		return 0;
-
-	writes = collect_non_private_tagged_writes(kernel);
-	disjoint = isl_union_set_is_disjoint(kernel->sync_writes, writes);
-	isl_union_set_free(writes);
-
-	return disjoint < 0 ? -1 : !disjoint;
-}
-
-/* Construct an isl_multi_val for use as tile sizes for tiling "node"
- * from the elements in "tile_size".
- */
-static __isl_give isl_multi_val *construct_band_tiles_sizes(
-	__isl_keep isl_schedule_node *node, int *tile_size)
-{
-	isl_space *space;
-
-	if (!node)
-		return NULL;
-
-	space = isl_schedule_node_band_get_space(node);
-	return ppcg_multi_val_from_int_list(space, tile_size);
-}
-
-/* Replace the partial schedule S of the band node "node" by
- *
- *	floor(S/f)
- *
- * or
- *
- *	f * floor(S/f)
- *
- * if scale_tile_loops is set, with f the integers in "factor".
- * The list that "factor" points to is assumed to contain at least
- * as many elements as the number of members in the band.
- */
-static __isl_give isl_schedule_node *snap_band_to_sizes(
-	__isl_take isl_schedule_node *node, int *factor,
-	struct ppcg_options *options)
-{
-	isl_multi_val *mv;
-
-	mv = construct_band_tiles_sizes(node, factor);
-	node = isl_schedule_node_band_scale_down(node, isl_multi_val_copy(mv));
-	if (options->scale_tile_loops)
-		node = isl_schedule_node_band_scale(node,
-							isl_multi_val_copy(mv));
-	isl_multi_val_free(mv);
-
-	return node;
-}
-
-/* Tile "band" with tile size specified by "sizes".
- *
- * Since the tile loops will be mapped to block ids, we forcibly
- * turn off tile loop scaling.  We may want to enable tile loop scaling
- * at some later point, but then we would have to support the detection
- * of strides during the mapping to block ids.
- * Similarly, since the point loops will be mapped to thread ids,
- * we forcibly shift the point loops so that they start at zero.
- */
-static __isl_give isl_schedule_node *tile_band(
-	__isl_take isl_schedule_node *node, __isl_take isl_multi_val *sizes)
-{
-	isl_ctx *ctx = isl_schedule_node_get_ctx(node);
-	int scale_tile;
-	int shift_point;
-
-	scale_tile = isl_options_get_tile_scale_tile_loops(ctx);
-	isl_options_set_tile_scale_tile_loops(ctx, 0);
-	shift_point = isl_options_get_tile_shift_point_loops(ctx);
-	isl_options_set_tile_shift_point_loops(ctx, 1);
-
-	node = isl_schedule_node_band_tile(node, sizes);
-
-	isl_options_set_tile_scale_tile_loops(ctx, scale_tile);
-	isl_options_set_tile_shift_point_loops(ctx, shift_point);
-
-	return node;
-}
-
-/* Extract the set of parameter values and outer schedule dimensions
- * for which any statement instance
- * in the kernel inserted at "node" needs to be executed.
- * Intersect the set of parameter values derived from the host schedule
- * relation with the context of "prog".
- */
-static __isl_give isl_set *extract_context(__isl_keep isl_schedule_node *node,
-	struct gpu_prog *prog)
-{
-	isl_union_map *schedule;
-	isl_union_set *schedule_domain;
-	isl_set *context;
-	int empty;
-
-	schedule = isl_schedule_node_get_prefix_schedule_relation(node);
-	schedule_domain = isl_union_map_range(schedule);
-	empty = isl_union_set_is_empty(schedule_domain);
-	if (empty < 0) {
-		isl_union_set_free(schedule_domain);
-		return NULL;
-	}
-	if (empty) {
-		int depth;
-		isl_space *space;
-
-		space = isl_union_set_get_space(schedule_domain);
-		isl_union_set_free(schedule_domain);
-		space = isl_space_set_from_params(space);
-		depth = isl_schedule_node_get_schedule_depth(node);
-		space = isl_space_add_dims(space, isl_dim_set, depth);
-		context = isl_set_empty(space);
-	} else {
-		context = isl_set_from_union_set(schedule_domain);
-	}
-	context = isl_set_intersect_params(context,
-					    isl_set_copy(prog->context));
-
-	return context;
-}
-
-/* Return the set of outer array elements accessed by
- * by the statement instances in "domain" in "prog".
- * The instances in "domain" are those that appear
- * in the domains of the access relations in "prog".
- */
-static __isl_give isl_union_set *accessed_by_domain(
-	__isl_take isl_union_set *domain, struct gpu_prog *prog)
-{
-	isl_union_map *access;
-	isl_union_set *arrays;
-
-	access = isl_union_map_union(isl_union_map_copy(prog->read),
-				     isl_union_map_copy(prog->may_write));
-	access = isl_union_map_intersect_domain(access, domain);
-	arrays = isl_union_map_range(access);
-	arrays = isl_union_set_apply(arrays,
-				isl_union_map_copy(prog->to_outer));
-
-	return arrays;
-}
-
-/* Return the number of outer band members of the band node "node"
- * that are marked coincident.
- */
-static int n_outer_coincidence(__isl_keep isl_schedule_node *node)
-{
-	int i, n;
-
-	n = isl_schedule_node_band_n_member(node);
-
-	for (i = 0; i < n; ++i)
-		if (!isl_schedule_node_band_member_get_coincident(node, i))
-			break;
-
-	return i;
-}
-
-/* If the band node "node" has more than "n" members, then split off
- * the first "n" of them.
- */
-static __isl_give isl_schedule_node *split_band(
-	__isl_take isl_schedule_node *node, int n)
-{
-	int dim;
-
-	dim = isl_schedule_node_band_n_member(node);
-	if (n < dim)
-		node = isl_schedule_node_band_split(node, n);
-
-	return node;
-}
-
-/* Scale a band node that may have been split by split_band.
- * "sizes" are the scaling factors for the original node.
- * "node" either points to the original band node, or the outer
- * of the two pieces after splitting.
- *
- * If the number of elements in "node" is smaller than the number of
- * elements in "sizes", then some splitting has occurred and we split
- * "sizes" in the same way.
- */
-static __isl_give isl_schedule_node *scale_band(
-	__isl_take isl_schedule_node *node, __isl_take isl_multi_val *sizes)
-{
-	int n, dim;
-
-	n = isl_multi_val_dim(sizes, isl_dim_set);
-	dim = isl_schedule_node_band_n_member(node);
-	if (n > dim) {
-		isl_multi_val *sizes2;
-
-		sizes2 = isl_multi_val_copy(sizes);
-		sizes = isl_multi_val_drop_dims(sizes,
-						isl_dim_set, dim, n - dim);
-		sizes2 = isl_multi_val_drop_dims(sizes2, isl_dim_set, 0, dim);
-		node = isl_schedule_node_child(node, 0);
-		node = isl_schedule_node_band_scale(node, sizes2);
-		node = isl_schedule_node_parent(node);
-	}
-
-	return isl_schedule_node_band_scale(node, sizes);
-}
-
-/* Return an isl_multi_aff, with as elements the parameters in "space"
- * that have the names specified by the elements in "names".
- * If (some of) these parameters do not already appear in "space",
- * then they are added first.
- */
-static __isl_give isl_multi_aff *parameter_vector(__isl_take isl_space *space,
-	__isl_keep isl_id_list *names)
-{
-	int i, n;
-	isl_local_space *ls;
-	isl_multi_aff *ma;
-
-	if (!names)
-		space = isl_space_free(space);
-
-	n = isl_id_list_n_id(names);
-	for (i = 0; i < n; ++i) {
-		int pos;
-		isl_id *id;
-
-		id = isl_id_list_get_id(names, i);
-		pos = isl_space_find_dim_by_id(space, isl_dim_param, id);
-		if (pos >= 0) {
-			isl_id_free(id);
-			continue;
-		}
-		pos = isl_space_dim(space, isl_dim_param);
-		space = isl_space_add_dims(space, isl_dim_param, 1);
-		space = isl_space_set_dim_id(space, isl_dim_param, pos, id);
-	}
-	ma = isl_multi_aff_zero(isl_space_copy(space));
-	ls = isl_local_space_from_space(isl_space_domain(space));
-	for (i = 0; i < n; ++i) {
-		int pos;
-		isl_id *id;
-		isl_aff *aff;
-
-		id = isl_id_list_get_id(names, i);
-		pos = isl_space_find_dim_by_id(space, isl_dim_param, id);
-		isl_id_free(id);
-		aff = isl_aff_var_on_domain(isl_local_space_copy(ls),
-					    isl_dim_param, pos);
-		ma = isl_multi_aff_set_aff(ma, i, aff);
-	}
-	isl_local_space_free(ls);
-
-	return ma;
-}
-
-/* Return constraints on the domain elements that equate a sequence of
- * parameters called "names", to the partial schedule
- * of "node" modulo the integers in "size".
- * The number of elements in the array "size" should be equal
- * to the number of elements in "names".
- * The number of members of the band node "node" should be smaller
- * than or equal to this number.  If it is smaller, then the first
- * elements of "names" are equated to zero.
- */
-static __isl_give isl_union_set *set_schedule_modulo(
-	__isl_keep isl_schedule_node *node, __isl_keep isl_id_list *names,
-	int *size)
-{
-	int n, n_zero;
-	isl_space *space;
-	isl_multi_aff *ma;
-	isl_multi_union_pw_aff *mupa, *mupa2;
-	isl_multi_val *mv;
-	isl_union_set *domain;
-
-	if (!node)
-		return NULL;
-	n = isl_id_list_n_id(names);
-	if (n == 0)
-		return isl_schedule_node_get_universe_domain(node);
-	n_zero = n - isl_schedule_node_band_n_member(node);
-
-	mupa = isl_schedule_node_band_get_partial_schedule(node);
-	mv = construct_band_tiles_sizes(node, size + n_zero);
-	mupa = isl_multi_union_pw_aff_mod_multi_val(mupa, mv);
-
-	space = isl_multi_union_pw_aff_get_space(mupa);
-	space = isl_space_params(space);
-	space = isl_space_set_from_params(space);
-	space = isl_space_add_dims(space, isl_dim_set, n_zero);
-	ma = isl_multi_aff_zero(space);
-
-	domain = isl_schedule_node_get_universe_domain(node);
-	mupa2 = isl_multi_union_pw_aff_multi_aff_on_domain(
-						isl_union_set_copy(domain), ma);
-	mupa = isl_multi_union_pw_aff_range_product(mupa2, mupa);
-
-	space = isl_multi_union_pw_aff_get_space(mupa);
-	ma = parameter_vector(space, names);
-
-	mupa2 = isl_multi_union_pw_aff_multi_aff_on_domain(domain, ma);
-	mupa = isl_multi_union_pw_aff_sub(mupa, mupa2);
-
-	return isl_multi_union_pw_aff_zero_union_set(mupa);
-}
-
-/* Insert a context node at "node" introducing the block and thread
- * identifiers along with their bounds, which are stored in kernel->grid_size
- * and kernel->block_dim.
- * Note that the bounds on the block identifiers may implicitly impose
- * constraints on the parameters.  A guard needs to be inserted
- * in the schedule tree to ensure that those bounds hold at "node".
- * This guard is inserted in insert_guard.
- */
-static __isl_give isl_schedule_node *insert_context(struct ppcg_kernel *kernel,
-	__isl_take isl_schedule_node *node)
-{
-	isl_set *context;
-
-	context = isl_set_universe(isl_set_get_space(kernel->context));
-
-	context = add_bounded_parameters_dynamic(context,
-					kernel->grid_size, kernel->block_ids);
-	context = add_bounded_parameters(context,
-					kernel->block_dim, kernel->thread_ids);
-
-	node = isl_schedule_node_insert_context(node, context);
-
-	return node;
-}
-
-/* Insert a guard that eliminates kernel launches where the kernel
- * obviously does not have any work to do.
- *
- * In particular, eliminate kernel launches where there are obviously
- * zero blocks.
- * Use the same block size constraints that are used to create the context
- * to ensure that all constraints implicit in the constructed context
- * are imposed by the guard.
- *
- * Additionally, add other constraints that are valid
- * for each executed instance ("context"), as long as this does not result
- * in a disjunction.
- */
-static __isl_give isl_schedule_node *insert_guard(
-	__isl_take isl_schedule_node *node, __isl_keep isl_set *context,
-	__isl_keep isl_multi_pw_aff *size, struct ppcg_scop *scop)
-{
-	unsigned nparam, n;
-	isl_set *guard;
-	isl_id_list *ids;
-
-	guard = isl_set_copy(context);
-	guard = isl_set_compute_divs(guard);
-	guard = isl_set_from_basic_set(isl_set_simple_hull(guard));
-
-	nparam = isl_set_dim(guard, isl_dim_param);
-	n = isl_multi_pw_aff_dim(size, isl_dim_out);
-	ids = ppcg_scop_generate_names(scop, n, "__ppcg_tmp");
-	guard = add_bounded_parameters_dynamic(guard, size, ids);
-	isl_id_list_free(ids);
-	guard = isl_set_project_out(guard, isl_dim_param, nparam, n);
-
-	node = isl_schedule_node_insert_guard(node, guard);
-
-	return node;
-}
-
-/* Does any array reference group mapping require the band that is mapped
- * to threads to be unrolled?
- */
-static int kernel_requires_unroll(struct ppcg_kernel *kernel)
-{
-	int i, j;
-
-	for (i = 0; i < kernel->n_array; ++i) {
-		struct gpu_local_array_info *array = &kernel->array[i];
-
-		for (j = 0; j < array->n_group; ++j) {
-			struct gpu_array_ref_group *group = array->groups[j];
-			if (gpu_array_ref_group_requires_unroll(group))
-				return 1;
-		}
-	}
-
-	return 0;
-}
-
-/* Mark the given band node "node" for unrolling by the AST generator and
- * then sink it to the leaves of the schedule tree.
- * All dimensions of "node" are assumed to be coincident, such that this
- * sinking is a valid operation.
- */
-static __isl_give isl_schedule_node *unroll(__isl_take isl_schedule_node *node)
-{
-	node = ppcg_set_schedule_node_type(node, isl_ast_loop_unroll);
-
-	node = isl_schedule_node_band_sink(node);
-
-	return node;
-}
-
-/* Insert a synchronization node in the schedule tree of "node"
- * after the core computation of "kernel" at the level of the band
- * that is mapped to threads, except if that level is equal to
- * that of the band that is mapped to blocks or if there are no writes
- * to global or shared memory in the core computation that require
- * synchronization.
- * If there are any writes to shared memory and the shared memory
- * copying is performed at the same level, then synchronization
- * is needed between the core and the copying anyway, so we might
- * as well add it here.  If the copying is performed at a higher
- * level, then different iterations of intermediate schedule dimensions
- * may have a different mapping from between shared memory elements and
- * threads, such that synchronization is required after the core.
- * "node" is assumed to point to the kernel node.
- *
- * If the shared and the thread mark point to the same node, then make
- * sure the synchronization is inserted outside of the shared mark.
- */
-static __isl_give isl_schedule_node *add_sync(struct ppcg_kernel *kernel,
-	__isl_take isl_schedule_node *node)
-{
-	int depth;
-	int need_sync;
-
-	need_sync = any_global_or_shared_sync_writes(kernel);
-	if (need_sync < 0)
-		return isl_schedule_node_free(node);
-	if (!need_sync)
-		return node;
-
-	node = gpu_tree_move_down_to_thread(node, kernel->core);
-	depth = isl_schedule_node_get_schedule_depth(node);
-	node = gpu_tree_move_up_to_kernel(node);
-	if (depth == isl_schedule_node_get_schedule_depth(node))
-		return node;
-
-	node = gpu_tree_move_down_to_depth(node, depth, kernel->core);
-	node = gpu_tree_ensure_following_sync(node, kernel);
-
-	node = gpu_tree_move_up_to_kernel(node);
-
-	return node;
-}
-
-/* Return a read ("read" is 1) or write access relation for "group"
- * with those accesses removed that are only needed to communicate data
- * within the subtree of the schedule rooted at "node".
- * Furthermore, include the prefix schedule at "node".
- * That is, return a relation of the form
- *
- *	S -> [D -> A]
- *
- * with D the outer schedule dimensions at "node".
- */
-static __isl_give isl_union_map *anchored_non_local_accesses(
-	struct ppcg_kernel *kernel, struct gpu_array_ref_group *group,
-	__isl_take isl_schedule_node *node, int read)
-{
-	isl_union_map *access;
-	isl_union_map *prefix;
-
-	prefix = isl_schedule_node_get_prefix_schedule_relation(node);
-	prefix = isl_union_map_preimage_domain_union_pw_multi_aff(prefix,
-			    isl_union_pw_multi_aff_copy(kernel->contraction));
-	access = gpu_array_ref_group_access_relation(group, read, !read);
-	access = remove_local_accesses_group(kernel, group, access, prefix,
-						read);
-	access = isl_union_map_range_product(prefix, access);
-
-	return access;
-}
-
-/* Given an array reference group "group", create a mapping
- *
- *	read[D -> A] -> [D -> A]
- *
- * if "read" is set or
- *
- *	write[D -> A] -> [D -> A]
- *
- * if "read" is not set.
- * D corresponds to the outer tile->depth dimensions of
- * the kernel schedule.
- */
-static __isl_give isl_multi_aff *create_from_access(isl_ctx *ctx,
-	struct gpu_array_ref_group *group, int read)
-{
-	struct gpu_array_tile *tile;
-	isl_space *space;
-	isl_id *id;
-
-	tile = gpu_array_ref_group_tile(group);
-	space = isl_space_copy(group->array->space);
-	space = isl_space_from_range(space);
-	space = isl_space_add_dims(space, isl_dim_in, tile->depth);
-	space = isl_space_wrap(space);
-	space = isl_space_map_from_set(space);
-
-	id = isl_id_alloc(ctx, read ? "read" : "write", group);
-	space = isl_space_set_tuple_id(space, isl_dim_in, id);
-
-	return isl_multi_aff_identity(space);
-}
-
-/* If any writes in "group" require synchronization, then make sure
- * that there is a synchronization node for "kernel" after the node
- * following "node" in a sequence.
- *
- * If "shared" is set and no synchronization is needed for
- * the writes to global memory, then add synchronization before
- * the kernel to protect shared memory from being overwritten
- * by the next iteration of the core computation.
- * No additional synchronization is needed to protect against
- * the next copy into shared memory because each element of
- * the shared memory tile is always copied by the same thread.
- */
-static __isl_give isl_schedule_node *add_group_write_sync(
-	__isl_take isl_schedule_node *node, struct ppcg_kernel *kernel,
-	struct gpu_array_ref_group *group, int shared)
-{
-	int need_sync;
-
-	need_sync = any_sync_writes_in_group(kernel, group);
-	if (need_sync < 0)
-		return isl_schedule_node_free(node);
-	if (need_sync) {
-		node = isl_schedule_node_parent(node);
-		node = isl_schedule_node_next_sibling(node);
-		node = isl_schedule_node_child(node, 0);
-		node = gpu_tree_ensure_following_sync(node, kernel);
-	} else if (shared) {
-		struct gpu_array_tile *tile;
-
-		tile = gpu_array_ref_group_tile(group);
-		node = isl_schedule_node_parent(node);
-		node = isl_schedule_node_parent(node);
-		node = gpu_tree_move_down_to_depth(node, tile->depth,
-							kernel->core);
-		node = gpu_tree_move_left_to_sync(node, kernel);
-	}
-
-	return node;
-}
-
-/* Add copy statements to the schedule tree of "node"
- * for reading from global memory to private memory (if "read" is set) or
- * for writing back from private memory to global memory
- * (if "read" is not set) for the array reference group "group" that
- * is mapped to private memory.
- * On input, "node" points to the kernel node, and it is moved
- * back there on output.
- *
- * The copies are performed in the order of the array elements.
- * The copy statement instances include a reference to the outer
- * tile->depth dimensions of the kernel schedule for ease of
- * combining them with the group tiling.
- *
- * That is, the extra schedule is of the form
- *
- *	type[D -> A] -> A
- *
- * where D corresponds to the outer tile->depth dimensions of
- * the kernel schedule and A to the global array.
- * This schedule is unrolled because registers are not addressable.
- *
- * The copying is inserted in the schedule tree through an extension
- * of the form
- *
- *	D -> type[D -> A]
- *
- * where the extra domain elements type[D -> A] are those accessed
- * by the group.
- * A filter is inserted on type[D -> A] to ensure that the element
- * is read/written by the same thread that needs the element.
- * This filter is obtained by applying
- *
- *	S -> type[D -> A]
- *
- * to the thread filter for the core statements.
- *
- * The extension is inserted before the core computation in case of a read
- * and after the core computation in case of a write.
- * In the latter case, we also make sure that there is a synchronization
- * node after the write to global memory, unless this write is performed
- * at the outer level of the kernel.
- * In principle, this synchronization could be inserted higher
- * in the schedule tree depending on where the corresponding reads
- * from global memory are performed.
- */
-static __isl_give isl_schedule_node *add_copies_group_private(
-	struct ppcg_kernel *kernel, struct gpu_array_ref_group *group,
-	__isl_take isl_schedule_node *node, int read)
-{
-	struct gpu_array_tile *tile;
-	isl_union_map *access;
-	isl_union_set *domain;
-	isl_space *space;
-	isl_multi_aff *from_access;
-	isl_multi_pw_aff *mpa;
-	isl_multi_union_pw_aff *mupa;
-	isl_union_pw_multi_aff *contraction;
-	isl_schedule_node *graft;
-	isl_union_set *filter;
-	int kernel_depth;
-	int empty;
-
-	kernel_depth = isl_schedule_node_get_schedule_depth(node);
-	tile = gpu_array_ref_group_tile(group);
-	node = gpu_tree_move_down_to_depth(node, tile->depth, kernel->core);
-
-	access = anchored_non_local_accesses(kernel, group, node, read);
-	empty = isl_union_map_is_empty(access);
-	if (empty < 0 || empty) {
-		isl_union_map_free(access);
-		if (empty < 0)
-			return isl_schedule_node_free(node);
-		return gpu_tree_move_up_to_kernel(node);
-	}
-
-	group->array->global = 1;
-	group->local_array->global = 1;
-
-	from_access = create_from_access(kernel->ctx, group, read);
-	space = isl_space_domain(isl_multi_aff_get_space(from_access));
-	access = isl_union_map_preimage_range_multi_aff(access, from_access);
-
-	filter = isl_union_set_copy(kernel->thread_filter);
-	contraction = isl_union_pw_multi_aff_copy(kernel->contraction);
-	filter = isl_union_set_preimage_union_pw_multi_aff(filter, contraction);
-	filter = isl_union_set_apply(filter, isl_union_map_copy(access));
-	filter = isl_union_set_detect_equalities(filter);
-	filter = isl_union_set_coalesce(filter);
-
-	domain = isl_union_map_range(access);
-	access = isl_union_set_wrapped_domain_map(domain);
-	access = isl_union_map_reverse(access);
-	access = isl_union_map_coalesce(access);
-	graft = isl_schedule_node_from_extension(access);
-
-	space = isl_space_map_from_set(space);
-	mpa = isl_multi_pw_aff_identity(space);
-	mpa = isl_multi_pw_aff_range_factor_range(mpa);
-	mupa = isl_multi_union_pw_aff_from_multi_pw_aff(mpa);
-
-	graft = isl_schedule_node_child(graft, 0);
-	graft = isl_schedule_node_insert_partial_schedule(graft, mupa);
-	graft = unroll(graft);
-
-	graft = isl_schedule_node_insert_filter(graft, filter);
-
-	graft = isl_schedule_node_parent(graft);
-
-	if (read)
-		node = isl_schedule_node_graft_before(node, graft);
-	else {
-		node = isl_schedule_node_graft_after(node, graft);
-		if (kernel_depth < tile->depth)
-			node = add_group_write_sync(node, kernel, group, 0);
-	}
-
-	node = gpu_tree_move_up_to_kernel(node);
-
-	return node;
-}
-
-/* Add copy statements to the schedule tree of "node"
- * for reading from global memory to shared memory (if "read" is set) or
- * for writing back from shared memory to global memory
- * (if "read" is not set) for the array reference group "group" that
- * is mapped to shared memory.
- * On input, "node" points to the kernel node, and it is moved
- * back there on output.
- *
- * The copies are performed in the order of the corresponding shared
- * memory tile.
- * The copy statement instances include a reference to the outer
- * tile->depth dimensions of the kernel schedule for ease of
- * combining them with the group tiling.
- *
- * If we are performing a read from global memory to shared memory and
- * if the array involved is not a scalar, then we copy
- * the entire tile to shared memory.  This may result in some extra
- * elements getting copied, but it should lead to simpler code
- * (which means that fewer registers may be needed) and less divergence.
- *
- * Otherwise, we only copy the elements that will be read or have been written
- * in the kernel.
- *
- * That is, the extra schedule is of the form
- *
- *	type[D -> A] -> T
- *
- * where D corresponds to the outer tile->depth dimensions of
- * the kernel schedule, A to the global array and T is the corresponding
- * shared memory tile.
- *
- * The copying is inserted in the schedule tree through an extension
- * of the form
- *
- *	D -> type[D -> A]
- *
- * where the extra domain elements type[D -> A] are those accessed
- * by the group.  In the case of read from a non-scalar, this set
- * is replaced by the entire shared memory tile.
- *
- * If the "unroll_copy_shared" option is set, then the AST generator
- * is instructed to unroll the copying code.
- *
- * A filter is inserted on type[D -> A] to map the copy instances
- * to the threads.  In particular, the thread identifiers are
- * equated to the position inside the shared memory tile (T)
- * modulo the block size.
- * We try to align the innermost tile dimension with the innermost
- * thread identifier (x) as a heuristic to improve coalescing.
- * In particular, if the dimension of the tile is greater than
- * the dimension of the block, then the schedule mapping to the tile
- * is broken up into two pieces and the filter is applied to the inner part.
- * If, on the other hand, the dimension of the tile is smaller than
- * the dimension of the block, then the initial thread identifiers
- * are equated to zero and the remaining thread identifiers are
- * matched to the memory tile.
- *
- * The extension is inserted before the core computation in case of a read
- * and after the core computation in case of a write.
- * In the case of a read, we first need to make sure there is some
- * synchronization before the core computation such that we can put the read
- * from global memory to shared memory before that synchronization.
- * This ensures that all threads have finished copying into shared memory
- * before the shared memory is used.
- * We also need to make sure that there is a synchronization node after
- * the core computation to ensure that the next load into shared memory
- * only happens after all data has been used.  There is no need for
- * this synchronization if we are at the outer level since then there
- * won't be a next load.
- * In the case of a write, we need to make sure there is some synchronization
- * after the core computation such taht we can put the write from shared
- * memory to global memory after that synchronization.
- * Unless we are at the outer level, we also need a synchronization node
- * after the write to ensure the data is saved to global memory
- * before the next iteration write to the same shared memory.
- * It also makes sure the data has arrived in global memory before
- * it is read in a subsequent iteration.
- */
-static __isl_give isl_schedule_node *add_copies_group_shared(
-	struct ppcg_kernel *kernel, struct gpu_array_ref_group *group,
-	__isl_take isl_schedule_node *node, int read)
-{
-	struct gpu_array_tile *tile;
-	isl_union_map *access;
-	isl_union_set *domain;
-	isl_multi_aff *ma;
-	isl_multi_aff *from_access;
-	isl_multi_pw_aff *mpa;
-	isl_multi_union_pw_aff *mupa;
-	isl_schedule_node *graft;
-	isl_union_set *filter;
-	int skip;
-	int kernel_depth;
-	int empty;
-
-	tile = gpu_array_ref_group_tile(group);
-	kernel_depth = isl_schedule_node_get_schedule_depth(node);
-	node = gpu_tree_move_down_to_depth(node, tile->depth, kernel->core);
-
-	access = anchored_non_local_accesses(kernel, group, node, read);
-	empty = isl_union_map_is_empty(access);
-	if (empty < 0 || empty) {
-		isl_union_map_free(access);
-		if (empty < 0)
-			return isl_schedule_node_free(node);
-		return gpu_tree_move_up_to_kernel(node);
-	}
-
-	group->array->global = 1;
-	group->local_array->global = 1;
-
-	from_access = create_from_access(kernel->ctx, group, read);
-
-	ma = isl_multi_aff_copy(tile->tiling);
-	ma = isl_multi_aff_pullback_multi_aff(ma,
-					    isl_multi_aff_copy(from_access));
-	mpa = isl_multi_pw_aff_from_multi_aff(ma);
-	mupa = isl_multi_union_pw_aff_from_multi_pw_aff(mpa);
-
-	domain = isl_union_map_range(access);
-
-	if (read && !gpu_array_is_scalar(group->array)) {
-		isl_map *map;
-		isl_union_set_free(domain);
-		map = group_tile(group);
-		domain = isl_union_set_from_set(isl_map_wrap(map));
-	}
-
-	domain = isl_union_set_preimage_multi_aff(domain, from_access);
-	access = isl_union_set_wrapped_domain_map(domain);
-	access = isl_union_map_reverse(access);
-	access = isl_union_map_coalesce(access);
-	graft = isl_schedule_node_from_extension(access);
-
-	graft = isl_schedule_node_child(graft, 0);
-
-	graft = isl_schedule_node_insert_partial_schedule(graft, mupa);
-	if (kernel->options->unroll_copy_shared)
-		graft = ppcg_set_schedule_node_type(graft, isl_ast_loop_unroll);
-
-	if (tile->n > kernel->n_block && kernel->n_block > 0) {
-		graft = isl_schedule_node_band_split(graft,
-						tile->n - kernel->n_block);
-		graft = isl_schedule_node_child(graft, 0);
-	}
-	if (tile->n < kernel->n_block)
-		skip = kernel->n_block - tile->n;
-	else
-		skip = 0;
-	filter = set_schedule_modulo(graft, kernel->thread_ids,
-					kernel->block_dim);
-	if (!kernel->options->wrap)
-		graft = snap_band_to_sizes(graft, kernel->block_dim + skip,
-			    kernel->options);
-	if (tile->n > kernel->n_block && kernel->n_block > 0)
-		graft = isl_schedule_node_parent(graft);
-	graft = isl_schedule_node_insert_filter(graft, filter);
-
-	while (graft && isl_schedule_node_has_parent(graft))
-		graft = isl_schedule_node_parent(graft);
-
-	if (read) {
-		if (kernel_depth < tile->depth)
-			node = gpu_tree_ensure_sync_after_core(node, kernel);
-		node = gpu_tree_move_left_to_sync(node, kernel);
-		node = isl_schedule_node_graft_before(node, graft);
-	} else {
-		node = gpu_tree_move_right_to_sync(node, kernel);
-		node = isl_schedule_node_graft_after(node, graft);
-		if (kernel_depth < tile->depth)
-			node = add_group_write_sync(node, kernel, group, 1);
-	}
-
-	node = gpu_tree_move_up_to_kernel(node);
-
-	return node;
-}
-
-/* Check whether the array reference group "group" is mapped to
- * private or shared memory and, if so,
- * add copy statements to the schedule tree of "node"
- * for reading from global memory to private or shared memory
- * (if "read" is set) or for writing back from private or shared memory
- * to global memory (if "read" is not set) for this group.
- * On input, "node" points to the kernel node, and it is moved
- * back there on output.
- */
-static __isl_give isl_schedule_node *add_copies_group(
-	struct ppcg_kernel *kernel, struct gpu_array_ref_group *group,
-	__isl_take isl_schedule_node *node, int read)
-{
-	enum ppcg_group_access_type type;
-
-	type = gpu_array_ref_group_type(group);
-	if (type == ppcg_access_private)
-		return add_copies_group_private(kernel, group, node, read);
-	if (type == ppcg_access_shared)
-		return add_copies_group_shared(kernel, group, node, read);
-	return node;
-}
-
-/* For each array reference group that is mapped to private or shared memory,
- * add copy statements to the schedule tree of "node"
- * for reading from global memory to private or shared memory
- * and for writing back.
- * On input, "node" points to the kernel node, and it is moved
- * back there on output.
- */
-static __isl_give isl_schedule_node *add_copies(struct ppcg_kernel *kernel,
-	__isl_take isl_schedule_node *node)
-{
-	int i, j;
-
-	for (i = 0; i < kernel->n_array; ++i) {
-		struct gpu_local_array_info *array = &kernel->array[i];
-
-		for (j = 0; j < array->n_group; ++j) {
-			struct gpu_array_ref_group *group = array->groups[j];
-
-			node = add_copies_group(kernel, group, node, 1);
-			if (!node)
-				return NULL;
-			node = add_copies_group(kernel, group, node, 0);
-			if (!node)
-				return NULL;
-		}
-	}
-
-	return node;
-}
-
-/* Mark all dimensions in the current band node atomic.
- */
-static __isl_give isl_schedule_node *atomic(__isl_take isl_schedule_node *node)
-{
-	return ppcg_set_schedule_node_type(node, isl_ast_loop_atomic);
-}
-
-/* Mark "node" atomic, if it is a band node.
- * Do the same for all ancestors.
- * Return a pointer to "node" (in the updated schedule tree).
- */
-static __isl_give isl_schedule_node *atomic_ancestors(
-	__isl_take isl_schedule_node *node)
-{
-	int pos;
-
-	if (!node)
-		return NULL;
-	if (!isl_schedule_node_has_parent(node))
-		return node;
-
-	pos = isl_schedule_node_get_child_position(node);
-	node = isl_schedule_node_parent(node);
-	if (isl_schedule_node_get_type(node) == isl_schedule_node_band)
-		node = atomic(node);
-	node = atomic_ancestors(node);
-	node = isl_schedule_node_child(node, pos);
-
-	return node;
-}
-
-/* Collect all write references that require synchronization.
- * "node" is assumed to point to the kernel node.
- * Each reference is represented by a universe set in a space
- *
- *	[S[i,j] -> R[]]
- *
- * with S[i,j] the statement instance space and R[] the array reference.
- *
- * This function should be called before block and thread filters are added.
- *
- * Synchronization is needed after a write if there is a subsequent read
- * within the same block that may not be performed by the same thread.
- * There should not be any dependences between different blocks,
- * so we start with the flow dependences within the same kernel invocation
- * and we subtract from these those dependences that are mapped
- * to the same iteration of the bands where synchronization is inserted.
- * We do not remove pairs of instances that are known to map to
- * the same thread across different iterations of the intermediate
- * bands because the read may be performed by a different thread
- * than the one that needs the value if shared memory is involved.
- *
- * We also consider all pairs of possible writes that access the same
- * memory location and that may be mapped to the same block but not
- * to the same iteration of the intermediate bands.
- * In theory, it would be possible for one thread to still be in
- * a previous iteration of a loop in these bands.
- * A write to global memory in this delayed thread could then overwrite
- * a write from another thread that has already moved on to
- * the next iteration.
- *
- * After computing the above writes paired off with reads or writes
- * that depend on them, we project onto the domain writes.
- * Sychronization is needed after writes to global memory
- * through these references.
- */
-static __isl_give isl_union_set *compute_sync_writes(
-	struct ppcg_kernel *kernel, __isl_keep isl_schedule_node *node)
-{
-	isl_union_map *local;
-	isl_union_map *may_writes, *shared_access;
-	isl_union_map *kernel_prefix, *thread_prefix;
-	isl_union_map *equal;
-	isl_union_set *wrap;
-	isl_union_set *domain;
-	isl_union_pw_multi_aff *contraction;
-
-	kernel_prefix = isl_schedule_node_get_prefix_schedule_union_map(node);
-	node = isl_schedule_node_copy(node);
-	node = gpu_tree_move_down_to_thread(node, kernel->core);
-	thread_prefix = isl_schedule_node_get_prefix_schedule_union_map(node);
-	isl_schedule_node_free(node);
-
-	contraction = kernel->contraction;
-	kernel_prefix = isl_union_map_preimage_domain_union_pw_multi_aff(
-		    kernel_prefix, isl_union_pw_multi_aff_copy(contraction));
-	thread_prefix = isl_union_map_preimage_domain_union_pw_multi_aff(
-		    thread_prefix, isl_union_pw_multi_aff_copy(contraction));
-	domain = isl_union_set_copy(kernel->expanded_domain);
-	domain = isl_union_set_universe(domain);
-
-	may_writes = isl_union_map_copy(kernel->prog->scop->tagged_may_writes);
-	may_writes = isl_union_map_curry(may_writes);
-	may_writes = isl_union_map_intersect_domain(may_writes, domain);
-	may_writes = isl_union_map_uncurry(may_writes);
-	shared_access = isl_union_map_copy(may_writes);
-	shared_access = isl_union_map_apply_range(shared_access,
-					isl_union_map_reverse(may_writes));
-
-	local = isl_union_map_copy(kernel->prog->scop->tagged_dep_flow);
-	local = isl_union_map_union(local, shared_access);
-	local = isl_union_map_zip(local);
-
-	equal = isl_union_map_apply_range(kernel_prefix,
-		    isl_union_map_reverse(isl_union_map_copy(kernel_prefix)));
-	wrap = isl_union_map_wrap(equal);
-	local = isl_union_map_intersect_domain(local, wrap);
-	equal = isl_union_map_apply_range(thread_prefix,
-		    isl_union_map_reverse(isl_union_map_copy(thread_prefix)));
-	wrap = isl_union_map_wrap(equal);
-	local = isl_union_map_subtract_domain(local, wrap);
-
-	local = isl_union_map_zip(local);
-	local = isl_union_map_universe(local);
-
-	return isl_union_map_domain(local);
-}
-
-/* Group the domain elements into a single space, named kernelX,
- * with X the kernel sequence number "kernel_id".
- */
-static __isl_give isl_schedule_node *group_statements(
-	__isl_take isl_schedule_node *node, int kernel_id)
-{
-	char buffer[20];
-	isl_id *id;
-
-	if (!node)
-		return NULL;
-
-	snprintf(buffer, sizeof(buffer), "kernel%d", kernel_id);
-	id = isl_id_alloc(isl_schedule_node_get_ctx(node), buffer, NULL);
-	return isl_schedule_node_group(node, id);
-}
-
-/* Create a ppcg_kernel representing the domain instances that reach "node"
- * and insert a mark node pointing to the ppcg_kernel before "node".
- * The band that "node" points to is the band that needs to be mapped
- * to block identifiers.  The band that needs to be mapped to thread
- * identifiers should be marked by a "thread" mark by the caller.
- * The linear branch between the current node and the "thread" mark
- * may also have a "shared" mark.  If present, the mapping to shared
- * memory is computed at that point.
- * Both marks are removed by this function.
- * If "scale" is set, then the band that "node" points to is scaled
- * by "sizes".
- *
- * Mark all outer band nodes as atomic to ensure each kernel is only
- * scheduled once.
- * If the domain elements that reach "node" live in more than one space,
- * then group the domain elements into a single space, named kernelX,
- * with X the kernel sequence number.
- *
- * Insert a guard node governing the kernel node to ensure that
- * no kernels with zero blocks are launched.
- *
- * Insert a context node describing the block and thread
- * identifiers inside the kernel mark.
- * The context node needs to be inserted after the effective block size
- * has been determined such that the bounds on the thread identifiers
- * would reflect the effective block size.
- * Insert a filter node inside the context node mapping the statement
- * instances to block identifiers.  In particular, the block identifiers
- * are equated to the partial schedule of band that was marked for mapping
- * to blocks modulo the grid size.
- * Insert a filter node inside the "thread" mark mapping the statement
- * instances to thread identifiers.  In particular, the thread identifiers
- * are equated to the partial schedule of band that was marked for mapping
- * to threads modulo the block size.
- *
- * Compute array reference groups for all arrays, set the local
- * array bounds based on the set of domain instances that reach
- * the kernel node, check the total amount of shared memory used
- * and compute all group tilings.
- * The array reference groups are computed after the block filter
- * has been inserted because it affects the mapping to shared or
- * private memory.  This computation also requires the thread filter
- * (in the ppcg_kernel object), but this thread filter should not
- * have been added to the schedule tree yet since the computation
- * requires the schedule of the band that needs to be mapped to
- * threads before the privatization is applied.
- *
- * If any array reference group requires the band mapped to threads
- * to be unrolled, then we perform the required unrolling.
- *
- * We save a copy of the schedule that may influence the mappings
- * to shared or private memory in kernel->copy_schedule.
- *
- * Finally, we add synchronization and copy statements to the schedule tree,
- * remove the "thread" mark and create representations for the local
- * variables in the kernel.
- *
- * We keep a copy of the isl_id that points to the kernel to ensure
- * that the kernel does not get destroyed if the schedule node
- * is freed due to some error condition.
- */
-__isl_give isl_schedule_node *gpu_create_kernel(struct gpu_gen *gen,
-	__isl_take isl_schedule_node *node, int scale,
-	__isl_keep isl_multi_val *sizes)
-{
-	struct ppcg_kernel *kernel;
-	isl_id *id;
-	isl_schedule_node *node_thread;
-	isl_union_map *host_schedule;
-	isl_union_pw_multi_aff *contraction;
-	isl_set *host_domain;
-	isl_union_set *domain, *expanded;
-	int single_statement;
-
-	node = gpu_tree_insert_shared_before_thread(node);
-	if (!node)
-		return NULL;
-
-	kernel = isl_calloc_type(gen->ctx, struct ppcg_kernel);
-	kernel = ppcg_kernel_create_local_arrays(kernel, gen->prog);
-	if (!kernel)
-		return isl_schedule_node_free(node);
-
-	domain = isl_schedule_node_get_domain(node);
-	single_statement = isl_union_set_n_set(domain) == 1;
-
-	kernel->ctx = gen->ctx;
-	kernel->prog = gen->prog;
-	kernel->options = gen->options;
-	kernel->context = extract_context(node, gen->prog);
-	kernel->core = isl_union_set_universe(isl_union_set_copy(domain));
-	contraction = isl_schedule_node_get_subtree_contraction(node);
-	kernel->contraction = isl_union_pw_multi_aff_copy(contraction);
-	expanded = isl_union_set_copy(domain);
-	expanded = isl_union_set_preimage_union_pw_multi_aff(expanded,
-						contraction);
-	kernel->expanded_domain = isl_union_set_copy(expanded);
-	kernel->arrays = accessed_by_domain(expanded, gen->prog);
-	kernel->n_grid = n_outer_coincidence(node);
-	node_thread = isl_schedule_node_copy(node);
-	node_thread = gpu_tree_move_down_to_thread(node_thread, kernel->core);
-	node_thread = isl_schedule_node_child(node_thread, 0);
-	kernel->n_block = n_outer_coincidence(node_thread);
-	isl_schedule_node_free(node_thread);
-	kernel->id = gen->kernel_id++;
-	read_grid_and_block_sizes(kernel, gen);
-
-	kernel->sync_writes = compute_sync_writes(kernel, node);
-
-	host_schedule = isl_schedule_node_get_prefix_schedule_union_map(node);
-	host_domain = isl_set_from_union_set(isl_union_map_range(
-								host_schedule));
-
-	node = atomic_ancestors(node);
-
-	id = isl_id_alloc(gen->ctx, "kernel", kernel);
-	id = isl_id_set_free_user(id, &ppcg_kernel_free_wrap);
-	node = isl_schedule_node_insert_mark(node, isl_id_copy(id));
-
-	if (!single_statement)
-		node = group_statements(node, kernel->id);
-
-	node = isl_schedule_node_child(node, 0);
-	node = split_band(node, kernel->n_grid);
-	kernel->block_ids = ppcg_scop_generate_names(gen->prog->scop,
-						kernel->n_grid, "b");
-	kernel->block_filter = set_schedule_modulo(node, kernel->block_ids,
-						kernel->grid_dim);
-	kernel->grid_size = extract_grid_size(kernel,
-						isl_union_set_copy(domain));
-	if (!kernel->options->wrap)
-		node = snap_band_to_sizes(node, kernel->grid_dim,
-						kernel->options);
-	if (scale)
-		node = scale_band(node, isl_multi_val_copy(sizes));
-	node = isl_schedule_node_parent(node);
-	if (!single_statement)
-		node = isl_schedule_node_parent(node);
-	node = insert_guard(node, kernel->context, kernel->grid_size,
-				gen->prog->scop);
-	node = gpu_tree_move_down_to_thread(node, kernel->core);
-	node = isl_schedule_node_child(node, 0);
-	node = split_band(node, kernel->n_block);
-	kernel->thread_ids = ppcg_scop_generate_names(gen->prog->scop,
-						kernel->n_block, "t");
-	kernel->thread_filter = set_schedule_modulo(node, kernel->thread_ids,
-						kernel->block_dim);
-	if (extract_block_size(kernel, domain) < 0)
-		node = isl_schedule_node_free(node);
-
-	node = gpu_tree_move_up_to_kernel(node);
-	node = isl_schedule_node_child(node, 0);
-	node = insert_context(kernel, node);
-	node = isl_schedule_node_child(node, 0);
-	node = isl_schedule_node_insert_filter(node,
-				    isl_union_set_copy(kernel->block_filter));
-
-	node = gpu_tree_move_up_to_kernel(node);
-
-	if (gpu_group_references(kernel, node) < 0)
-		node = isl_schedule_node_free(node);
-	localize_bounds(kernel, host_domain);
-	isl_set_free(host_domain);
-
-	check_shared_memory_bound(kernel);
-	mark_global_arrays(kernel);
-	compute_group_tilings(kernel);
-
-	node = gpu_tree_move_down_to_thread(node, kernel->core);
-	node = isl_schedule_node_child(node, 0);
-	if (!kernel->options->wrap)
-		node = snap_band_to_sizes(node, kernel->block_dim,
-						kernel->options);
-	node = isl_schedule_node_insert_filter(node,
-				    isl_union_set_copy(kernel->thread_filter));
-	if (kernel_requires_unroll(kernel)) {
-		node = isl_schedule_node_child(node, 0);
-		node = unroll(node);
-	}
-
-	node = gpu_tree_move_up_to_thread(node);
-	kernel->copy_schedule_dim = isl_schedule_node_get_schedule_depth(node);
-	kernel->copy_schedule =
-		isl_schedule_node_get_prefix_schedule_union_pw_multi_aff(node);
-	contraction = isl_union_pw_multi_aff_copy(kernel->contraction);
-	kernel->copy_schedule =
-		isl_union_pw_multi_aff_pullback_union_pw_multi_aff(
-					    kernel->copy_schedule, contraction);
-
-	node = gpu_tree_move_up_to_kernel(node);
-
-	node = add_sync(kernel, node);
-	node = add_copies(kernel, node);
-
-	node = gpu_tree_move_down_to_shared(node, kernel->core);
-	node = isl_schedule_node_delete(node);
-
-	node = gpu_tree_move_down_to_thread(node, kernel->core);
-	node = isl_schedule_node_delete(node);
-
-	node = gpu_tree_move_up_to_kernel(node);
-
-	if (create_kernel_vars(kernel) < 0)
-		node = isl_schedule_node_free(node);
-
-	if (!single_statement)
-		node = isl_schedule_node_parent(node);
-	node = isl_schedule_node_parent(node);
-
-	isl_id_free(id);
-	return node;
-}
-
-/* Insert a zero-dimensional permutable band at "node".
- */
-static __isl_give isl_schedule_node *insert_empty_permutable_band(
-	__isl_take isl_schedule_node *node)
-{
-	isl_space *space;
-	isl_schedule *schedule;
-	isl_union_set *domain;
-	isl_multi_union_pw_aff *mupa;
-
-	schedule = isl_schedule_node_get_schedule(node);
-	domain = isl_schedule_get_domain(schedule);
-	space = isl_union_set_get_space(domain);
-	isl_union_set_free(domain);
-	isl_schedule_free(schedule);
-
-	space = isl_space_set_from_params(space);
-	mupa = isl_multi_union_pw_aff_zero(space);
-	node = isl_schedule_node_insert_partial_schedule(node, mupa);
-	node = isl_schedule_node_band_set_permutable(node, 1);
-
-	return node;
-}
-
-/* See if hybrid tiling can be performed on "node" and its parent.
- * If so, apply hybrid tiling and return the updated schedule tree.
- * If not, return the original schedule tree.
- * Return NULL on error.
- *
- * First check if "node", together with its parent, meets
- * the basic requirements for hybrid tiling.
- * If so, compute the relative dependence distances of "node"
- * with respect to its parent and check if they are sufficiently bounded.
- * If so, apply hybrid tiling using user specified tile sizes.
- *
- * The tile sizes are read before the dependence distance bounds are
- * computed, because the user may have specified fewer dimensions
- * than are available.  In this case, the remaining schedule dimensions
- * are split off and the dependence distances should be computed
- * after these dimensions have been split off.
- */
-static __isl_give isl_schedule_node *try_hybrid_tile(struct gpu_gen *gen,
-	__isl_take isl_schedule_node *node)
-{
-	int tile_len;
-	int *tile_size;
-	isl_bool ok;
-	isl_schedule_node *orig = node;
-	ppcg_ht_bounds *bounds;
-
-	ok = ppcg_ht_parent_has_input_pattern(node);
-	if (ok < 0)
-		return isl_schedule_node_free(node);
-	if (!ok)
-		return orig;
-
-	tile_len = 1 + isl_schedule_node_band_n_member(node);
-	tile_size = read_tile_sizes(gen, &tile_len);
-	if (!tile_size)
-		return isl_schedule_node_free(node);
-
-	node = isl_schedule_node_copy(node);
-	node = split_band(node, tile_len - 1);
-	node = isl_schedule_node_parent(node);
-	bounds = ppcg_ht_compute_bounds(gen->prog->scop, node);
-	node = isl_schedule_node_child(node, 0);
-
-	ok = ppcg_ht_bounds_is_valid(bounds);
-	if (ok >= 0 && ok)
-		node = gpu_hybrid_tile(gen, node, bounds, tile_size);
-	else
-		ppcg_ht_bounds_free(bounds);
-	free(tile_size);
-
-	if (ok >= 0 && !ok) {
-		isl_schedule_node_free(node);
-		return orig;
-	}
-	isl_schedule_node_free(orig);
-	if (ok < 0)
-		return isl_schedule_node_free(node);
-	return node;
-}
-
-/* If "node" is the outermost permutable band that can be mapped to block and
- * thread identifiers in its branch (or the root of a subtree with
- * no such outer bands),
- * then mark the band as such, attaching a ppcg_kernel to the mark.
- *
- * If hybrid tiling is allowed, then first try and apply it
- * to "node" and its parent.
- *
- * If "node" is the root of a subtree without permutable bands,
- * then insert a zero-dimensional permutable band such that
- * we can assume that "node" always points to a band node.
- * This includes the case where "node" already points to a band node,
- * but one without any coincident dimension.  In this case,
- * the extra node ensures that this original node does not get tiled.
- *
- * Tile "node" using user specified tile sizes, after splitting the band
- * if the number of specified tile sizes is smaller than the dimension
- * of the band.  Mark the point band of this tiling as the band that
- * needs to be mapped to threads and instruct the AST generator to unroll
- * the band if the "unroll_gpu_tile" option is set.
- * Create a kernel representing the domain instances that reach "node" and
- * insert a mark node pointing to the ppcg_kernel before the band node.
- */
-static __isl_give isl_schedule_node *mark_outer_permutable(
-	__isl_take isl_schedule_node *node, void *user)
-{
-	struct gpu_gen *gen = user;
-	int outer;
-	int scale;
-	int tile_len;
-	int *tile_size;
-	isl_id *id;
-	isl_multi_val *sizes;
-
-	outer = is_outer_tilable(node);
-	if (outer < 0)
-		return isl_schedule_node_free(node);
-	if (!outer)
-		return node;
-
-	if (gen->options->hybrid) {
-		isl_schedule_node *saved = isl_schedule_node_copy(node);
-		node = try_hybrid_tile(gen, node);
-		isl_schedule_node_free(saved);
-		if (node != saved)
-			return node;
-	}
-
-	if (isl_schedule_node_get_type(node) != isl_schedule_node_band ||
-	    !isl_schedule_node_band_member_get_coincident(node, 0))
-		node = insert_empty_permutable_band(node);
-
-	tile_len = isl_schedule_node_band_n_member(node);
-	tile_size = read_tile_sizes(gen, &tile_len);
-	if (!tile_size)
-		return isl_schedule_node_free(node);
-	if (tile_len < isl_schedule_node_band_n_member(node))
-		node = isl_schedule_node_band_split(node, tile_len);
-	sizes = construct_band_tiles_sizes(node, tile_size);
-	node = tile_band(node, isl_multi_val_copy(sizes));
-	node = isl_schedule_node_child(node, 0);
-	if (gen->options->unroll_gpu_tile)
-		node = ppcg_set_schedule_node_type(node, isl_ast_loop_unroll);
-	id = isl_id_alloc(gen->ctx, "thread", NULL);
-	node = isl_schedule_node_insert_mark(node, id);
-	node = isl_schedule_node_parent(node);
-
-	scale = gen->options->scale_tile_loops;
-	node = gpu_create_kernel(gen, node, scale, sizes);
-	isl_multi_val_free(sizes);
-	free(tile_size);
-
-	return node;
-}
-
-/* Given a set or sequence node, return the union the filters of either all
- * (if "only_initial" is not set) or the initial (if "only_initial" is set)
- * direct subtrees that do not contain any suitably permutable bands
- * (according to subtree_has_permutable_bands).
- */
-static __isl_give isl_union_set *get_non_parallel_subtree_filters(
-	__isl_keep isl_schedule_node *node, int only_initial)
-{
-	isl_space *space;
-	isl_union_set *filter;
-	int i, n;
-
-	n = isl_schedule_node_n_children(node);
-	if (n < 0)
-		return NULL;
-
-	node = isl_schedule_node_copy(node);
-	node = isl_schedule_node_child(node, 0);
-	filter = isl_schedule_node_filter_get_filter(node);
-	node = isl_schedule_node_parent(node);
-	space = isl_union_set_get_space(filter);
-	isl_union_set_free(filter);
-	filter = isl_union_set_empty(space);
-
-	for (i = 0; i < n; ++i) {
-		int parallelism;
-
-		node = isl_schedule_node_child(node, i);
-		parallelism = subtree_has_permutable_bands(node);
-		if (parallelism < 0) {
-			filter = isl_union_set_free(filter);
-		} else if (!parallelism) {
-			isl_union_set *filter_i;
-			filter_i = isl_schedule_node_filter_get_filter(node);
-			filter = isl_union_set_union(filter, filter_i);
-		} else if (only_initial)
-			break;
-		node = isl_schedule_node_parent(node);
-	}
-
-	isl_schedule_node_free(node);
-
-	return filter;
-}
-
-/* Given a set or sequence node, return the union of the filters of
- * the direct subtrees that do not contain any suitably permutable bands
- * (according to subtree_has_permutable_bands).
- */
-static __isl_give isl_union_set *get_all_non_parallel_subtree_filters(
-	__isl_keep isl_schedule_node *node)
-{
-	return get_non_parallel_subtree_filters(node, 0);
-}
-
-/* Given a set or sequence node, return the union of the filters of
- * the initial direct subtrees that do not contain any suitably permutable
- * bands (according to subtree_has_permutable_bands).
- */
-static __isl_give isl_union_set *get_initial_non_parallel_subtree_filters(
-	__isl_keep isl_schedule_node *node)
-{
-	return get_non_parallel_subtree_filters(node, 1);
-}
-
-/* Mark all variables that are accessed by the statement instances in "domain"
- * and that are local to "prog" as requiring a declaration in the host code.
- * The statement instances in "domain" correspond to (a subset of)
- * the active instances at "node".
- * "node" is not modified by this function, except that NULL is returned
- * in case of error.
- */
-static __isl_give isl_schedule_node *declare_accessed_local_variables(
-	__isl_take isl_schedule_node *node, struct gpu_prog *prog,
-	__isl_keep isl_union_set *domain)
-{
-	isl_union_pw_multi_aff *contraction;
-	isl_union_set *arrays;
-	int i;
-
-	if (!ppcg_scop_any_hidden_declarations(prog->scop))
-		return node;
-	contraction = isl_schedule_node_get_subtree_contraction(node);
-	domain = isl_union_set_copy(domain);
-	domain = isl_union_set_preimage_union_pw_multi_aff(domain, contraction);
-	arrays = accessed_by_domain(domain, prog);
-
-	for (i = 0; i < prog->n_array; ++i) {
-		isl_space *space;
-		isl_set *set;
-		int empty;
-
-		if (!prog->array[i].local)
-			continue;
-		space = isl_set_get_space(prog->array[i].extent);
-		set = isl_union_set_extract_set(arrays, space);
-		empty = isl_set_plain_is_empty(set);
-		isl_set_free(set);
-		if (empty < 0)
-			goto error;
-		if (!empty)
-			prog->array[i].declare_local = 1;
-	}
-
-	isl_union_set_free(arrays);
-	return node;
-error:
-	isl_union_set_free(arrays);
-	return isl_schedule_node_free(node);
-}
-
-/* If "node" points to a set node, then separate its children
- * into subtrees that have suitably permutable bands and
- * those that do not.
- * Adjust the schedule tree in order to execute the second group
- * after the first group and return a pointer to the first group,
- * assuming there are any such subtrees.
- * If "node" points to a sequence node, then separate the initial
- * children that do not have suitably permutable bands and
- * return a pointer to the subsequence of children that do have such bands,
- * assuming there are any such subtrees.
- *
- * In both cases, mark all local variables in "prog" that are accessed by
- * the group without permutable bands as requiring a declaration on the host.
- */
-static __isl_give isl_schedule_node *isolate_permutable_subtrees(
-	__isl_take isl_schedule_node *node, struct gpu_prog *prog)
-{
-	isl_union_set *filter;
-	enum isl_schedule_node_type type;
-
-	if (!node)
-		return NULL;
-	type = isl_schedule_node_get_type(node);
-	if (type == isl_schedule_node_set) {
-		filter = get_all_non_parallel_subtree_filters(node);
-		node = declare_accessed_local_variables(node, prog, filter);
-		node = isl_schedule_node_order_after(node, filter);
-	} else if (type == isl_schedule_node_sequence) {
-		filter = get_initial_non_parallel_subtree_filters(node);
-		node = declare_accessed_local_variables(node, prog, filter);
-		node = isl_schedule_node_order_before(node, filter);
-	}
-
-	return node;
-}
-
-/* Replace any reference to an array element in the range of "copy"
- * by a reference to all array elements (defined by the extent of the array).
- */
-static __isl_give isl_union_map *approximate_copy_out(
-	__isl_take isl_union_map *copy, struct gpu_prog *prog)
-{
-	int i;
-	isl_union_map *res;
-
-	res = isl_union_map_empty(isl_union_map_get_space(copy));
-
-	for (i = 0; i < prog->n_array; ++i) {
-		isl_space *space;
-		isl_set *set;
-		isl_union_map *copy_i;
-		isl_union_set *extent, *domain;
-
-		space = isl_space_copy(prog->array[i].space);
-		extent = isl_union_set_from_set(isl_set_universe(space));
-		copy_i = isl_union_map_copy(copy);
-		copy_i = isl_union_map_intersect_range(copy_i, extent);
-		set = isl_set_copy(prog->array[i].extent);
-		extent = isl_union_set_from_set(set);
-		domain = isl_union_map_domain(copy_i);
-		copy_i = isl_union_map_from_domain_and_range(domain, extent);
-		res = isl_union_map_union(res, copy_i);
-	}
-
-	isl_union_map_free(copy);
-
-	return res;
-}
-
-/* Insert "kernel" marks that point to a ppcg_kernel structure
- * in front of all outermost tilable band that (by construction)
- * have at least one parallel loop.
- */
-static __isl_give isl_schedule_node *mark_kernels(struct gpu_gen *gen,
-	__isl_take isl_schedule_node *node)
-{
-	return isl_schedule_node_map_descendant_bottom_up(node,
-						&mark_outer_permutable, gen);
-}
-
-/* Construct schedule constraints from the dependences in prog->scop and
- * the array order dependences in prog->array_order.
- *
- * If live range reordering is allowed, then we need to make sure
- * that live ranges on arrays are not run in parallel since doing
- * so would require array expansion.  We therefore add the array
- * order dependences to the coincidence dependences.  Non-zero array
- * order dependences will then prevent a schedule dimension from being
- * considered parallel.
- * Live ranges derived from scalars are allowed to be run in parallel
- * since we force the scalars to be mapped to private memory in
- * check_scalar_live_ranges.
- * If live range reordering is allowed, then the false dependences
- * are not added to the validity constraints as that would prevent
- * reordering.  Instead, the external false dependences that enforce that reads
- * from potentially live-in data precede any later write and
- * that writes of potentially live-out data follow any other earlier write
- * are added to the validity and the coincidence constraints.
- * The false dependences are still added to the proximity constraints
- * for consistency with the case where live range reordering is not allowed.
- * The coincidence constraints then consist of flow dependences,
- * external false dependences and array order dependences.
- * The independences can be filtered out from the first two sets.
- * They have already been filtered out from the array order dependences
- * on a per array basis in collect_order_dependences.
- * There is no need for a per array handling of the other two sets
- * as there should be no flow or external false dependence on local
- * variables that can be filtered out.
- */
-static __isl_give isl_schedule_constraints *construct_schedule_constraints(
-	struct gpu_prog *prog)
-{
-	isl_union_set *domain;
-	isl_union_map *dep_raw, *dep;
-	isl_union_map *validity, *proximity, *coincidence;
-	isl_schedule_constraints *sc;
-
-	domain = isl_union_set_copy(prog->scop->domain);
-	sc = isl_schedule_constraints_on_domain(domain);
-	sc = isl_schedule_constraints_set_context(sc,
-				isl_set_copy(prog->scop->context));
-	if (prog->scop->options->live_range_reordering) {
-		sc = isl_schedule_constraints_set_conditional_validity(sc,
-			isl_union_map_copy(prog->scop->tagged_dep_flow),
-			isl_union_map_copy(prog->scop->tagged_dep_order));
-		proximity = isl_union_map_copy(prog->scop->dep_flow);
-		validity = isl_union_map_copy(proximity);
-		validity = isl_union_map_union(validity,
-			    isl_union_map_copy(prog->scop->dep_forced));
-		proximity = isl_union_map_union(proximity,
-			    isl_union_map_copy(prog->scop->dep_false));
-		coincidence = isl_union_map_copy(validity);
-		coincidence = isl_union_map_subtract(coincidence,
-			isl_union_map_copy(prog->scop->independence));
-		coincidence = isl_union_map_union(coincidence,
-				isl_union_map_copy(prog->array_order));
-	} else {
-		dep_raw = isl_union_map_copy(prog->scop->dep_flow);
-		dep = isl_union_map_copy(prog->scop->dep_false);
-		dep = isl_union_map_union(dep, dep_raw);
-		dep = isl_union_map_coalesce(dep);
-		proximity = isl_union_map_copy(dep);
-		coincidence = isl_union_map_copy(dep);
-		validity = dep;
-	}
-	sc = isl_schedule_constraints_set_validity(sc, validity);
-	sc = isl_schedule_constraints_set_coincidence(sc, coincidence);
-	sc = isl_schedule_constraints_set_proximity(sc, proximity);
-
-	if (prog->scop->options->debug->dump_schedule_constraints)
-		isl_schedule_constraints_dump(sc);
-	return sc;
-}
-
-/* Compute an appropriate schedule based on the accesses in
- * gen->read and gen->write.
- *
- * We derive schedule constraints from the dependences in gen->prog->scop
- * and then use isl to compute a schedule that has a parallel loop
- * in each tilable band.
- * During the schedule construction, some statement instances
- * may be grouped first based on the input schedule.
- */
-static __isl_give isl_schedule *compute_schedule(struct gpu_gen *gen)
-{
-	isl_schedule_constraints *sc;
-	isl_schedule *schedule;
-
-	sc = construct_schedule_constraints(gen->prog);
-	schedule = gen->prog->scop->schedule;
-	schedule = ppcg_compute_schedule(sc, schedule, gen->options);
-
-	return schedule;
-}
-
-/* If the band node "node" has exactly one member then mark it permutable.
- */
-static __isl_give isl_schedule_node *band_set_permutable(
-	__isl_take isl_schedule_node *node,
-	__isl_keep isl_schedule_constraints *sc)
-{
-	if (isl_schedule_node_band_n_member(node) == 1)
-		node = isl_schedule_node_band_set_permutable(node, 1);
-
-	return node;
-}
-
-/* Return the coincidence constraints between pairs of instances
- * that are scheduled together by the ancestors of "node".
- * That is, select those coincidence constraints that relate
- * pairs of instances that have the same value for the prefix schedule.
- * If the schedule depth is zero, then the prefix schedule does not
- * contain any information, so we intersect domain and range
- * of the schedule constraints with the reaching domain elements instead.
- */
-static __isl_give isl_union_map *get_local_coincidence(
-	__isl_keep isl_schedule_node *node,
-	__isl_keep isl_schedule_constraints *sc)
-{
-	isl_union_map *coincidence;
-	isl_multi_union_pw_aff *prefix;
-	isl_union_pw_multi_aff *contraction;
-
-	coincidence = isl_schedule_constraints_get_coincidence(sc);
-	contraction = isl_schedule_node_get_subtree_contraction(node);
-	if (isl_schedule_node_get_schedule_depth(node) == 0) {
-		isl_union_set *domain;
-
-		domain = isl_schedule_node_get_domain(node);
-		domain = isl_union_set_preimage_union_pw_multi_aff(domain,
-						    contraction);
-		coincidence = isl_union_map_intersect_domain(coincidence,
-						    isl_union_set_copy(domain));
-		coincidence = isl_union_map_intersect_range(coincidence,
-						    domain);
-		return coincidence;
-	}
-
-	prefix = isl_schedule_node_get_prefix_schedule_multi_union_pw_aff(node);
-	prefix = isl_multi_union_pw_aff_pullback_union_pw_multi_aff(prefix,
-								contraction);
-	return isl_union_map_eq_at_multi_union_pw_aff(coincidence, prefix);
-}
-
-/* For each member in the band node "node", determine whether
- * it is coincident with respect to the outer nodes and mark
- * it accordingly.
- *
- * That is, for each coincidence constraint between pairs
- * of instances that are scheduled together by the outer nodes,
- * check that domain and range are assigned the same value
- * by the band member.  This test is performed by checking
- * that imposing the same value for the band member does not
- * remove any elements from the set of coincidence constraints.
- */
-static __isl_give isl_schedule_node *band_set_coincident(
-	__isl_take isl_schedule_node *node,
-	__isl_keep isl_schedule_constraints *sc)
-{
-	isl_union_map *coincidence;
-	isl_union_pw_multi_aff *contraction;
-	isl_multi_union_pw_aff *partial;
-	int i, n;
-
-	coincidence = get_local_coincidence(node, sc);
-
-	partial = isl_schedule_node_band_get_partial_schedule(node);
-	contraction = isl_schedule_node_get_subtree_contraction(node);
-	partial = isl_multi_union_pw_aff_pullback_union_pw_multi_aff(partial,
-								contraction);
-	n = isl_schedule_node_band_n_member(node);
-	for (i = 0; i < n; ++i) {
-		isl_union_map *coincidence_i;
-		isl_union_pw_aff *upa;
-		isl_multi_union_pw_aff *partial_i;
-		int subset;
-
-		upa = isl_multi_union_pw_aff_get_union_pw_aff(partial, i);
-		partial_i = isl_multi_union_pw_aff_from_union_pw_aff(upa);
-		coincidence_i = isl_union_map_copy(coincidence);
-		coincidence_i = isl_union_map_eq_at_multi_union_pw_aff(
-						    coincidence_i, partial_i);
-		subset = isl_union_map_is_subset(coincidence, coincidence_i);
-		isl_union_map_free(coincidence_i);
-
-		if (subset < 0)
-			break;
-		node = isl_schedule_node_band_member_set_coincident(node, i,
-								    subset);
-	}
-	if (i < n)
-		node = isl_schedule_node_free(node);
-	isl_multi_union_pw_aff_free(partial);
-	isl_union_map_free(coincidence);
-
-	return node;
-}
-
-/* If "node" is a band, then set its properties.
- *
- * In particular, if the band has exactly one member, then mark it permutable.
- * Mark the band member coincident based on the coincidence constraints
- * of "sc".
- */
-static __isl_give isl_schedule_node *set_band_properties(
-	__isl_take isl_schedule_node *node, void *user)
-{
-	isl_schedule_constraints *sc = user;
-
-	if (isl_schedule_node_get_type(node) != isl_schedule_node_band)
-		return node;
-	if (isl_schedule_node_band_n_member(node) == 0)
-		return node;
-
-	node = band_set_permutable(node, sc);
-	node = band_set_coincident(node, sc);
-
-	return node;
-}
-
-/* Return the original schedule with all bands marked permutable and
- * all band members marked coincident based on the coincidence constraints.
- * The bands are explicitly marked permutable so that they will be considered
- * by mark_outer_permutable.
- */
-static __isl_give isl_schedule *determine_properties_original_schedule(
-	struct gpu_gen *gen)
-{
-	isl_schedule *schedule;
-	isl_schedule_constraints *sc;
-
-	schedule = isl_schedule_copy(gen->prog->scop->schedule);
-	sc = construct_schedule_constraints(gen->prog);
-	schedule = isl_schedule_map_schedule_node_bottom_up(schedule,
-						    &set_band_properties, sc);
-	isl_schedule_constraints_free(sc);
-
-	return schedule;
-}
-
-/* Compute a schedule or determine the properties of the original schedule
- * depending on the value of the "reschedule" option.
- */
-static __isl_give isl_schedule *compute_or_set_properties(void *user)
-{
-	struct gpu_gen *gen = user;
-
-	if (gen->options->reschedule)
-		return compute_schedule(gen);
-	else
-		return determine_properties_original_schedule(gen);
-}
-
-/* Obtain a schedule for the scop, by reading it from
- * a file, by computing one or by determining the properties
- * of the original schedule.
- */
-__isl_give isl_schedule *get_schedule(struct gpu_gen *gen)
-{
-	return ppcg_get_schedule(gen->ctx, gen->options,
-				&compute_or_set_properties, gen);
-}
-
-/* Construct the string "<a>_<b>".
- */
-static char *concat(isl_ctx *ctx, const char *a, const char *b)
-{
-	isl_printer *p;
-	char *s;
-
-	p = isl_printer_to_str(ctx);
-	p = isl_printer_print_str(p, a);
-	p = isl_printer_print_str(p, "_");
-	p = isl_printer_print_str(p, b);
-	s = isl_printer_get_str(p);
-	isl_printer_free(p);
-
-	return s;
-}
-
-/* For each array in "prog" of which an element appears in "accessed" and
- * that is not a read only scalar, create a zero-dimensional universe set
- * of which the tuple id has name "<prefix>_<name of array>" and a user
- * pointer pointing to the array (gpu_array_info).
- *
- * If the array is local to "prog", then make sure it will be declared
- * in the host code.
- *
- * Return the list of these universe sets.
- */
-static __isl_give isl_union_set_list *create_copy_filters(struct gpu_prog *prog,
-	const char *prefix, __isl_take isl_union_set *accessed)
-{
-	int i;
-	isl_ctx *ctx;
-	isl_union_set_list *filters;
-
-	ctx = prog->ctx;
-	filters = isl_union_set_list_alloc(ctx, 0);
-	for (i = 0; i < prog->n_array; ++i) {
-		struct gpu_array_info *array = &prog->array[i];
-		isl_space *space;
-		isl_set *accessed_i;
-		int empty;
-		char *name;
-		isl_id *id;
-		isl_union_set *uset;
-
-		if (gpu_array_is_read_only_scalar(array))
-			continue;
-
-		space = isl_space_copy(array->space);
-		accessed_i = isl_union_set_extract_set(accessed, space);
-		empty = isl_set_plain_is_empty(accessed_i);
-		isl_set_free(accessed_i);
-		if (empty < 0) {
-			filters = isl_union_set_list_free(filters);
-			break;
-		}
-		if (empty)
-			continue;
-
-		array->global = 1;
-		if (array->local)
-			array->declare_local = 1;
-
-		name = concat(ctx, prefix, array->name);
-		id = name ? isl_id_alloc(ctx, name, array) : NULL;
-		free(name);
-		space = isl_space_set_alloc(ctx, 0, 0);
-		space = isl_space_set_tuple_id(space, isl_dim_set, id);
-		uset = isl_union_set_from_set(isl_set_universe(space));
-
-		filters = isl_union_set_list_add(filters, uset);
-	}
-	isl_union_set_free(accessed);
-
-	return filters;
-}
-
-/* Make sure that code for the statements in "filters" that
- * copy arrays to or from the device is only generated when
- * the size of the corresponding array is positive.
- * That is, add a set node underneath "graft" with "filters" as children
- * and for each child add a guard that the selects the parameter
- * values for which the corresponding array has a positive size.
- * The array is available in the user pointer of the statement identifier.
- * "depth" is the schedule depth of the position where "graft"
- * will be added.
- */
-static __isl_give isl_schedule_node *insert_positive_size_guards(
-	__isl_take isl_schedule_node *graft,
-	__isl_take isl_union_set_list *filters, int depth)
-{
-	int i, n;
-
-	graft = isl_schedule_node_child(graft, 0);
-	graft = isl_schedule_node_insert_set(graft, filters);
-	n = isl_schedule_node_n_children(graft);
-	for (i = 0; i < n; ++i) {
-		isl_union_set *filter;
-		isl_set *domain, *guard;
-		isl_id *id;
-		struct gpu_array_info *array;
-
-		graft = isl_schedule_node_child(graft, i);
-		filter = isl_schedule_node_filter_get_filter(graft);
-		domain = isl_set_from_union_set(filter);
-		id = isl_set_get_tuple_id(domain);
-		array = isl_id_get_user(id);
-		isl_id_free(id);
-		isl_set_free(domain);
-		guard = gpu_array_positive_size_guard(array);
-		guard = isl_set_from_params(guard);
-		guard = isl_set_add_dims(guard, isl_dim_set, depth);
-		graft = isl_schedule_node_child(graft, 0);
-		graft = isl_schedule_node_insert_guard(graft, guard);
-		graft = isl_schedule_node_parent(graft);
-		graft = isl_schedule_node_parent(graft);
-	}
-	graft = isl_schedule_node_parent(graft);
-
-	return graft;
-}
-
-/* Create a graft for copying arrays to or from the device,
- * whenever the size of the array is strictly positive.
- * Each statement is called "<prefix>_<name of array>" and
- * the identifier has a user pointer pointing to the array.
- * The graft will be added at the position specified by "node".
- * "copy" contains the array elements that need to be copied.
- * Only arrays of which some elements need to be copied
- * will have a corresponding statement in the graph.
- * Note though that each such statement will copy the entire array.
- */
-static __isl_give isl_schedule_node *create_copy_device(struct gpu_prog *prog,
-	__isl_keep isl_schedule_node *node, const char *prefix,
-	__isl_take isl_union_set *copy)
-{
-	int depth;
-	isl_ctx *ctx;
-	isl_space *space;
-	isl_union_set *all, *domain;
-	isl_union_set_list *filters;
-	isl_union_map *extension;
-	isl_schedule_node *graft;
-
-	ctx = prog->ctx;
-	depth = isl_schedule_node_get_schedule_depth(node);
-	filters = create_copy_filters(prog, prefix, copy);
-	all = isl_union_set_list_union(isl_union_set_list_copy(filters));
-
-	space = depth < 0 ? NULL : isl_space_set_alloc(ctx, 0, depth);
-	domain = isl_union_set_from_set(isl_set_universe(space));
-	extension = isl_union_map_from_domain_and_range(domain, all);
-	graft = isl_schedule_node_from_extension(extension);
-
-	if (!filters)
-		return isl_schedule_node_free(graft);
-	if (isl_union_set_list_n_union_set(filters) == 0) {
-		isl_union_set_list_free(filters);
-		return graft;
-	}
-
-	return insert_positive_size_guards(graft, filters, depth);
-}
-
-/* Return (the universe spaces of) the arrays that are declared
- * inside the scop corresponding to "prog" and for which all
- * potential writes inside the scop form a subset of "domain".
- */
-static __isl_give isl_union_set *extract_local_accesses(struct gpu_prog *prog,
-	__isl_keep isl_union_set *domain)
-{
-	int i;
-	isl_union_set *local;
-
-	local = isl_union_set_empty(isl_union_set_get_space(domain));
-
-	for (i = 0; i < prog->n_array; ++i) {
-		isl_set *set;
-		isl_union_map *to_outer;
-		isl_union_map *may_write;
-		isl_union_set *write_domain;
-		isl_union_set *fields;
-		int subset;
-
-		if (!prog->array[i].local)
-			continue;
-
-		set = isl_set_universe(isl_space_copy(prog->array[i].space));
-		to_outer = isl_union_map_copy(prog->to_outer);
-		to_outer = isl_union_map_intersect_range(to_outer,
-				    isl_union_set_from_set(isl_set_copy(set)));
-		fields = isl_union_map_domain(to_outer);
-		may_write = isl_union_map_copy(prog->may_write);
-		may_write = isl_union_map_intersect_range(may_write, fields);
-		write_domain = isl_union_map_domain(may_write);
-		subset = isl_union_set_is_subset(write_domain, domain);
-		isl_union_set_free(write_domain);
-
-		if (subset < 0) {
-			isl_set_free(set);
-			return isl_union_set_free(local);
-		} else if (subset) {
-			local = isl_union_set_add_set(local, set);
-		} else {
-			isl_set_free(set);
-		}
-	}
-
-	return local;
-}
-
-/* Internal data structure for node_may_persist.
- *
- * "tagger" maps tagged iteration domains to the corresponding untagged
- *	iteration domain.
- *
- * "may_persist_flow" is the set of all tagged dataflow dependences
- * with those dependences removed that either precede or follow
- * the kernel launch in a sequence.
- * "inner_band_flow" is the set of all tagged dataflow dependences
- * that are local to a given iteration of the outer band nodes
- * with respect to the current node.
- * "local_flow" is equal to "inner_band_flow", except that the domain
- * and the range have been intersected with intermediate filters
- * on children of sets or sequences.
- */
-struct ppcg_may_persist_data {
-	isl_union_pw_multi_aff *tagger;
-
-	isl_union_map *local_flow;
-	isl_union_map *inner_band_flow;
-	isl_union_map *may_persist_flow;
-};
-
-/* Update the information in "data" based on the band ancestor "node".
- *
- * In particular, we restrict the dependences in data->local_flow
- * to those dependence where the source and the sink occur in
- * the same iteration of the given band node.
- * We also update data->inner_band_flow to the new value of
- * data->local_flow.
- */
-static int update_may_persist_at_band(__isl_keep isl_schedule_node *node,
-	struct ppcg_may_persist_data *data)
-{
-	isl_multi_union_pw_aff *partial;
-	isl_union_pw_multi_aff *contraction;
-	isl_union_map *flow;
-
-	if (isl_schedule_node_band_n_member(node) == 0)
-		return 0;
-
-	partial = isl_schedule_node_band_get_partial_schedule(node);
-	contraction = isl_schedule_node_get_subtree_contraction(node);
-	partial = isl_multi_union_pw_aff_pullback_union_pw_multi_aff(partial,
-								contraction);
-	partial = isl_multi_union_pw_aff_pullback_union_pw_multi_aff(partial,
-				isl_union_pw_multi_aff_copy(data->tagger));
-
-	flow = data->local_flow;
-	flow = isl_union_map_eq_at_multi_union_pw_aff(flow, partial);
-	data->local_flow = flow;
-
-	isl_union_map_free(data->inner_band_flow);
-	data->inner_band_flow = isl_union_map_copy(data->local_flow);
-
-	return 0;
-}
-
-/* Given a set of local reaching domain elements "domain",
- * expand them to the corresponding leaf domain elements using "contraction"
- * and insert the array references tags using data->tagger.
- */
-static __isl_give isl_union_set *expand_and_tag(
-	__isl_take isl_union_set *domain,
-	__isl_take isl_union_pw_multi_aff *contraction,
-	struct ppcg_may_persist_data *data)
-{
-	domain = isl_union_set_preimage_union_pw_multi_aff(domain,
-			    contraction);
-	domain = isl_union_set_preimage_union_pw_multi_aff(domain,
-			    isl_union_pw_multi_aff_copy(data->tagger));
-	return domain;
-}
-
-/* Given a filter node that is the child of a set or sequence node,
- * restrict data->local_flow to refer only to those elements
- * in the filter of the node.
- * "contraction" maps the leaf domain elements of the schedule tree
- * to the corresponding domain elements at (the parent of) "node".
- */
-static int filter_flow(__isl_keep isl_schedule_node *node,
-	struct ppcg_may_persist_data *data,
-	__isl_take isl_union_pw_multi_aff *contraction)
-{
-	isl_union_set *filter;
-	isl_union_map *flow;
-
-	flow = data->local_flow;
-	filter = isl_schedule_node_filter_get_filter(node);
-	filter = expand_and_tag(filter, contraction, data);
-	flow = isl_union_map_intersect_domain(flow, isl_union_set_copy(filter));
-	flow = isl_union_map_intersect_range(flow, filter);
-	data->local_flow = flow;
-
-	return 0;
-}
-
-/* Given a filter node "node", collect the filters on all preceding siblings
- * (which are also filter nodes), add them to "filters" and return the result.
- */
-static __isl_give isl_union_set *add_previous_filters(
-	__isl_take isl_union_set *filters, __isl_keep isl_schedule_node *node)
-{
-	isl_schedule_node *sibling;
-
-	sibling = isl_schedule_node_copy(node);
-	while (sibling && isl_schedule_node_has_previous_sibling(sibling)) {
-		isl_union_set *filter;
-
-		sibling = isl_schedule_node_previous_sibling(sibling);
-		filter = isl_schedule_node_filter_get_filter(sibling);
-		filters = isl_union_set_union(filters, filter);
-	}
-	isl_schedule_node_free(sibling);
-	if (!sibling)
-		return isl_union_set_free(filters);
-
-	return filters;
-}
-
-/* Given a filter node "node", collect the filters on all following siblings
- * (which are also filter nodes), add them to "filters" and return the result.
- */
-static __isl_give isl_union_set *add_next_filters(
-	__isl_take isl_union_set *filters, __isl_keep isl_schedule_node *node)
-{
-	isl_schedule_node *sibling;
-
-	sibling = isl_schedule_node_copy(node);
-	while (sibling && isl_schedule_node_has_next_sibling(sibling)) {
-		isl_union_set *filter;
-
-		sibling = isl_schedule_node_next_sibling(sibling);
-		filter = isl_schedule_node_filter_get_filter(sibling);
-		filters = isl_union_set_union(filters, filter);
-	}
-	isl_schedule_node_free(sibling);
-	if (!sibling)
-		return isl_union_set_free(filters);
-
-	return filters;
-}
-
-/* Remove those flow dependences from data->may_persist_flow
- * that flow between elements of "domain" within the same iteration
- * of all outer band nodes.
- * "contraction" maps the leaf domain elements of the schedule tree
- * to the corresponding elements "domain".
- */
-static void remove_external_flow(struct ppcg_may_persist_data *data,
-	__isl_take isl_union_set *domain,
-	__isl_keep isl_union_pw_multi_aff *contraction)
-{
-	isl_union_map *flow;
-
-	contraction = isl_union_pw_multi_aff_copy(contraction);
-	domain = expand_and_tag(domain, contraction, data);
-	flow = isl_union_map_copy(data->local_flow);
-	flow = isl_union_map_intersect_domain(flow, isl_union_set_copy(domain));
-	flow = isl_union_map_intersect_range(flow, domain);
-
-	data->may_persist_flow = isl_union_map_subtract(data->may_persist_flow,
-							flow);
-}
-
-/* Update the information in "data" based on the filter ancestor "node".
- * We only need to modify anything if the filter is the child
- * of a set or sequence node.
- *
- * In the case of a sequence, we remove the dependences between
- * statement instances that are both executed either before or
- * after the subtree that will be mapped to a kernel, within
- * the same iteration of outer bands.
- *
- * In both cases, we restrict data->local_flow to the current child.
- */
-static int update_may_persist_at_filter(__isl_keep isl_schedule_node *node,
-	struct ppcg_may_persist_data *data)
-{
-	enum isl_schedule_node_type type;
-	isl_schedule_node *parent;
-	isl_space *space;
-	isl_union_pw_multi_aff *contraction;
-	isl_union_set *before, *after, *filter;
-
-	type = isl_schedule_node_get_parent_type(node);
-	if (type != isl_schedule_node_sequence && type != isl_schedule_node_set)
-		return 0;
-
-	parent = isl_schedule_node_copy(node);
-	parent = isl_schedule_node_parent(parent);
-	contraction = isl_schedule_node_get_subtree_contraction(parent);
-	isl_schedule_node_free(parent);
-
-	if (type == isl_schedule_node_set)
-		return filter_flow(node, data, contraction);
-
-	filter = isl_schedule_node_filter_get_filter(node);
-	space = isl_union_set_get_space(filter);
-	isl_union_set_free(filter);
-	before = isl_union_set_empty(space);
-	after = isl_union_set_copy(before);
-	before = add_previous_filters(before, node);
-	after = add_next_filters(after, node);
-
-	remove_external_flow(data, before, contraction);
-	remove_external_flow(data, after, contraction);
-
-	return filter_flow(node, data, contraction);
-}
-
-/* Update the information in "data" based on the ancestor "node".
- */
-static isl_stat update_may_persist_at(__isl_keep isl_schedule_node *node,
-	void *user)
-{
-	struct ppcg_may_persist_data *data = user;
-
-	switch (isl_schedule_node_get_type(node)) {
-	case isl_schedule_node_error:
-		return isl_stat_error;
-	case isl_schedule_node_context:
-	case isl_schedule_node_domain:
-	case isl_schedule_node_expansion:
-	case isl_schedule_node_extension:
-	case isl_schedule_node_guard:
-	case isl_schedule_node_leaf:
-	case isl_schedule_node_mark:
-	case isl_schedule_node_sequence:
-	case isl_schedule_node_set:
-		break;
-	case isl_schedule_node_band:
-		if (update_may_persist_at_band(node, data) < 0)
-			return isl_stat_error;
-		break;
-	case isl_schedule_node_filter:
-		if (update_may_persist_at_filter(node, data) < 0)
-			return isl_stat_error;
-		break;
-	}
-
-	return isl_stat_ok;
-}
-
-/* Determine the set of array elements that may need to be perserved
- * by a kernel constructed from the subtree at "node".
- * This includes the set of array elements that may need to be preserved
- * by the entire scop (prog->may_persist) and the elements for which
- * there is a potential flow dependence that may cross a kernel launch.
- *
- * To determine the second set, we start from all flow dependences.
- * From this set of dependences, we remove those that cannot possibly
- * require data to be preserved by a kernel launch.
- * In particular, we consider the following sets of dependences.
- * - dependences of which the write occurs inside the kernel.
- *   If the data is needed outside the kernel, then it will
- *   be copied out immediately after the kernel launch, so there
- *   is no need for any special care.
- * - dependences of which the read occurs inside the kernel and the
- *   corresponding write occurs inside the same iteration of the
- *   outer band nodes.  This means that the data is needed in
- *   the first kernel launch after the write, which is already
- *   taken care of by the standard copy-in.  That is, the data
- *   do not need to be preserved by any intermediate call to
- *   the same kernel.
- * - dependences of which the write and the read either both occur
- *   before the kernel launch or both occur after the kernel launch,
- *   within the same iteration of the outer band nodes with respect
- *   to the sequence that determines the ordering of the dependence
- *   and the kernel launch.  Such flow dependences cannot cross
- *   any kernel launch.
- *
- * For the remaining (tagged) dependences, we take the domain
- * (i.e., the tagged writes) and apply the tagged access relation
- * to obtain the accessed data elements.
- * These are then combined with the elements that may need to be
- * preserved by the entire scop.
- */
-static __isl_give isl_union_set *node_may_persist(
-	__isl_keep isl_schedule_node *node, struct gpu_prog *prog)
-{
-	struct ppcg_may_persist_data data;
-	isl_union_pw_multi_aff *contraction;
-	isl_union_set *domain;
-	isl_union_set *persist;
-	isl_union_map *flow, *local_flow;
-
-	data.tagger = prog->scop->tagger;
-
-	flow = isl_union_map_copy(prog->scop->tagged_dep_flow);
-	data.local_flow = isl_union_map_copy(flow);
-	data.inner_band_flow = isl_union_map_copy(flow);
-	data.may_persist_flow = flow;
-	if (isl_schedule_node_foreach_ancestor_top_down(node,
-					&update_may_persist_at, &data) < 0)
-		data.may_persist_flow =
-				    isl_union_map_free(data.may_persist_flow);
-	flow = data.may_persist_flow;
-	isl_union_map_free(data.local_flow);
-
-	domain = isl_schedule_node_get_domain(node);
-	contraction = isl_schedule_node_get_subtree_contraction(node);
-	domain = isl_union_set_preimage_union_pw_multi_aff(domain,
-				    contraction);
-	domain = isl_union_set_preimage_union_pw_multi_aff(domain,
-				    isl_union_pw_multi_aff_copy(data.tagger));
-	flow = isl_union_map_subtract_domain(flow, isl_union_set_copy(domain));
-	local_flow = data.inner_band_flow;
-	local_flow = isl_union_map_intersect_range(local_flow, domain);
-	flow = isl_union_map_subtract(flow, local_flow);
-
-	persist = isl_union_map_domain(flow);
-	persist = isl_union_set_apply(persist,
-			isl_union_map_copy(prog->scop->tagged_may_writes));
-	persist = isl_union_set_union(persist,
-			isl_union_set_copy(prog->may_persist));
-
-	return persist;
-}
-
-/* Add nodes for copying outer arrays in and out of the device
- * before and after the subtree "node", which contains one or more kernels.
- * "domain" contains the original statement instances, i.e.,
- * those that correspond to the domains of the access relations in "prog".
- * In particular, the domain has not been contracted in any way.
- * "prefix" contains the prefix schedule at that point, in terms
- * of the same original statement instances.
- *
- * We first compute the sets of outer array elements that need
- * to be copied in and out and then graft in the nodes for
- * performing this copying.
- *
- * In particular, for each array that is possibly written anywhere in
- * the subtree "node" and that may be used after "node"
- * or that may be visible outside the corresponding scop,
- * we copy out its entire extent.
- *
- * Any array elements that is read without first being written inside
- * the subtree "node" needs to be copied in.
- * Furthermore, if there are any array elements that
- * are copied out, but that may not be written inside "node, then
- * they also need to be copied in to ensure that the value after execution
- * is the same as the value before execution, at least for those array
- * elements that may have their values preserved by the scop or that
- * may be written before "node" and read after "node".
- * In case the array elements are structures, we need to take into
- * account that all members of the structures need to be written
- * by "node" before we can avoid copying the data structure in.
- *
- * Note that the may_write relation is intersected with the domain,
- * which has been intersected with the context.
- * This helps in those cases where the arrays are declared with a fixed size,
- * while the accesses are parametric and the context assigns a fixed value
- * to the parameters.
- *
- * If an element from a local array is read without first being written,
- * then there is no point in copying it in since it cannot have been
- * written prior to the scop.  Warn about the uninitialized read instead.
- */
-static __isl_give isl_schedule_node *add_to_from_device(
-	__isl_take isl_schedule_node *node, __isl_take isl_union_set *domain,
-	__isl_take isl_union_map *prefix, struct gpu_prog *prog)
-{
-	isl_union_set *local;
-	isl_union_set *may_persist;
-	isl_union_map *may_write, *must_write, *copy_out, *not_written;
-	isl_union_map *read, *copy_in;
-	isl_union_map *tagged;
-	isl_union_map *local_uninitialized;
-	isl_schedule_node *graft;
-
-	tagged = isl_union_map_copy(prog->scop->tagged_reads);
-	tagged = isl_union_map_union(tagged,
-			    isl_union_map_copy(prog->scop->tagged_may_writes));
-
-	may_write = isl_union_map_copy(prog->may_write);
-	may_write = isl_union_map_intersect_domain(may_write,
-					isl_union_set_copy(domain));
-	may_write = remove_local_accesses(prog,
-					isl_union_map_copy(tagged), may_write,
-					isl_union_map_copy(prefix), 0);
-	may_write = isl_union_map_apply_range(may_write,
-					isl_union_map_copy(prog->to_outer));
-	may_write = isl_union_map_apply_domain(may_write,
-					isl_union_map_copy(prefix));
-	may_write = approximate_copy_out(may_write, prog);
-	copy_out = isl_union_map_copy(may_write);
-	may_write = isl_union_map_apply_range(may_write,
-					isl_union_map_copy(prog->to_inner));
-	must_write = isl_union_map_copy(prog->must_write);
-	must_write = isl_union_map_apply_domain(must_write,
-					isl_union_map_copy(prefix));
-	may_persist = node_may_persist(node, prog);
-	may_write = isl_union_map_intersect_range(may_write, may_persist);
-	not_written = isl_union_map_subtract(may_write, must_write);
-
-	local = extract_local_accesses(prog, domain);
-	read = isl_union_map_copy(prog->read);
-	read = isl_union_map_intersect_domain(read, domain);
-	read = remove_local_accesses(prog, tagged, read,
-					isl_union_map_copy(prefix), 1);
-	local = isl_union_set_apply(local, isl_union_map_copy(prog->to_inner));
-	local_uninitialized = isl_union_map_copy(prog->scop->live_in);
-	local_uninitialized = isl_union_map_intersect_range(local_uninitialized,
-							    local);
-	local_uninitialized = isl_union_map_intersect(local_uninitialized,
-						    isl_union_map_copy(read));
-	if (!isl_union_map_is_empty(local_uninitialized)) {
-		fprintf(stderr,
-			"possibly uninitialized reads (not copied in):\n");
-		isl_union_map_dump(local_uninitialized);
-	}
-	read = isl_union_map_subtract(read, local_uninitialized);
-	read = isl_union_map_apply_domain(read, prefix);
-	copy_in = isl_union_map_union(read, not_written);
-	copy_in = isl_union_map_apply_range(copy_in,
-				    isl_union_map_copy(prog->to_outer));
-
-	graft = create_copy_device(prog, node, "to_device",
-						isl_union_map_range(copy_in));
-	node = isl_schedule_node_graft_before(node, graft);
-	graft = create_copy_device(prog, node, "from_device",
-						isl_union_map_range(copy_out));
-	node = isl_schedule_node_graft_after(node, graft);
-
-	return node;
-}
-
-/* Add nodes for initializing ("init_device") and clearing ("clear_device")
- * the device before and after "node".
- */
-static __isl_give isl_schedule_node *add_init_clear_device(
-	__isl_take isl_schedule_node *node)
-{
-	isl_ctx *ctx;
-	isl_space *space;
-	isl_union_set *domain;
-	isl_schedule_node *graft;
-
-	ctx = isl_schedule_node_get_ctx(node);
-
-	space = isl_space_set_alloc(ctx, 0, 0);
-	space = isl_space_set_tuple_name(space, isl_dim_set, "init_device");
-	domain = isl_union_set_from_set(isl_set_universe(space));
-	graft = isl_schedule_node_from_domain(domain);
-
-	node = isl_schedule_node_graft_before(node, graft);
-
-	space = isl_space_set_alloc(ctx, 0, 0);
-	space = isl_space_set_tuple_name(space, isl_dim_set, "clear_device");
-	domain = isl_union_set_from_set(isl_set_universe(space));
-	graft = isl_schedule_node_from_domain(domain);
-
-	node = isl_schedule_node_graft_after(node, graft);
-
-	return node;
-}
-
-/* Update "schedule" for mapping to a GPU device.
- *
- * In particular, insert a context node, create kernels for
- * each outermost tilable band and introduce nodes for copying arrays
- * in and out of the device and for initializing and clearing the device.
- * If the child of the initial root points to a set node,
- * then children of this node that do not contain any tilable bands
- * are separated from the other children and are not mapped to
- * the device.
- *
- * The GPU code is generated in a context where at least one
- * statement instance is executed.  The corresponding guard is inserted
- * around the entire schedule.
- */
-__isl_give isl_schedule *map_to_device(struct gpu_gen *gen,
-	__isl_take isl_schedule *schedule, int to_from_device)
-{
-	isl_schedule_node *node;
-	isl_set *context;
-	isl_set *guard;
-	isl_union_set *domain;
-	isl_union_map *prefix;
-	isl_union_pw_multi_aff *contraction;
-	struct gpu_prog *prog;
-
-	context = isl_set_copy(gen->prog->context);
-	context = isl_set_from_params(context);
-	schedule = isl_schedule_insert_context(schedule, context);
-
-	prog = gen->prog;
-	guard = isl_union_set_params(isl_union_set_copy(prog->scop->domain));
-	prog->context = isl_set_intersect(prog->context, isl_set_copy(guard));
-	guard = isl_set_from_params(guard);
-
-	node = isl_schedule_get_root(schedule);
-	isl_schedule_free(schedule);
-	node = isl_schedule_node_child(node, 0);
-	node = isl_schedule_node_child(node, 0);
-	node = isolate_permutable_subtrees(node, gen->prog);
-	domain = isl_schedule_node_get_domain(node);
-	contraction = isl_schedule_node_get_subtree_contraction(node);
-	domain = isl_union_set_preimage_union_pw_multi_aff(domain,
-				    isl_union_pw_multi_aff_copy(contraction));
-	prefix = isl_schedule_node_get_prefix_schedule_union_map(node);
-	prefix = isl_union_map_preimage_domain_union_pw_multi_aff(prefix,
-				    contraction);
-	node = mark_kernels(gen, node);
-	if (to_from_device) {
-		node = add_to_from_device(node, domain, prefix, gen->prog);
-	} else {
-		isl_union_set_free(domain);
-		isl_union_map_free(prefix);
-	}
-	node = isl_schedule_node_root(node);
-	node = isl_schedule_node_child(node, 0);
-	node = isl_schedule_node_child(node, 0);
-	node = isl_schedule_node_insert_guard(node, guard);
-	node = isl_schedule_node_child(node, 0);
-	node = add_init_clear_device(node);
-	schedule = isl_schedule_node_get_schedule(node);
-	isl_schedule_node_free(node);
-
-	return schedule;
-}
-
-/* Internal data structure for extract_access.
- * "next_access" points to the end of a linked list that is extended
- * by extract_access.
- * "single_expression" is set if the access expressions belong to
- * an expression statement (i.e., a statement without internal control).
- * "any_to_outer" maps all intermediate arrays to their outer arrays.
- */
-struct ppcg_extract_access_data {
-	struct gpu_stmt_access **next_access;
-	int single_expression;
-	isl_union_map *any_to_outer;
-};
-
-/* Given a tagged access relation to a single array "tagged", extract it
- * as a map, taking into account that the input may be empty.
- * If the access relation is empty, then it does not contain
- * any space information, so we try to recover it from the index
- * expression.
- * The space of the index expression is of the form I -> A,
- * with I the statement instances and A the array, or [I -> F] -> A,
- * with F the filters corresponding to arguments.
- * We first drop F, if present, obtaining I -> A.
- * Then we construct I -> R, with R the reference tag,
- * combine the two into I -> [R -> A] and uncurry to obtain
- * the final result [I -> R] -> A.
- * Note that the index expression may have a lower dimension
- * than that of the array, but this dimension is not used
- * if the access relation is empty.
- */
-static __isl_give isl_map *extract_single_tagged_access(
-	__isl_take isl_union_map *tagged, __isl_keep pet_expr *expr)
-{
-	int empty;
-	isl_id *id;
-	isl_space *space, *space2;
-	isl_multi_pw_aff *index;
-
-	empty = isl_union_map_is_empty(tagged);
-	if (empty < 0)
-		goto error;
-	if (!empty)
-		return isl_map_from_union_map(tagged);
-	isl_union_map_free(tagged);
-
-	index = pet_expr_access_get_index(expr);
-	space = isl_multi_pw_aff_get_space(index);
-	isl_multi_pw_aff_free(index);
-	if (isl_space_domain_is_wrapping(space))
-		space = isl_space_domain_factor_domain(space);
-	space2 = isl_space_copy(space);
-	space2 = isl_space_from_domain(isl_space_domain(space));
-	id = pet_expr_access_get_ref_id(expr);
-	space2 = isl_space_set_tuple_id(space2, isl_dim_out, id);
-	space = isl_space_range_product(space2, space);
-	space = isl_space_uncurry(space);
-
-	return isl_map_empty(space);
-error:
-	isl_union_map_free(tagged);
-	return NULL;
-}
-
-/* Does the index expression "index" of "expr" represent an access
- * to a single element?
- * That is, is "index" completely specified?
- *
- * If "expr" accesses elements from different spaces (i.e., fields
- * of a structure), then it does not access a single element.
- * Otherwise, if the single space of the access matches the space
- * of "index", then the index expression is completely specified
- * (no pointer to a lower-dimensional slice of the accessed array)
- * and a single element is being accessed.
- */
-static isl_bool complete_index(__isl_keep pet_expr *expr,
-	__isl_keep isl_multi_pw_aff *index)
-{
-	isl_union_map *read, *write, *all;
-	isl_map *map;
-	isl_space *space1, *space2;
-	isl_bool complete;
-
-	read = pet_expr_access_get_may_read(expr);
-	write = pet_expr_access_get_may_write(expr);
-	all = isl_union_map_union(read, write);
-	if (!all)
-		return isl_bool_error;
-	if (isl_union_map_n_map(all) != 1) {
-		isl_union_map_free(all);
-		return isl_bool_false;
-	}
-	map = isl_map_from_union_map(all);
-	space1 = isl_map_get_space(map);
-	isl_map_free(map);
-	space2 = isl_multi_pw_aff_get_space(index);
-	complete = isl_space_tuple_is_equal(space1, isl_dim_out,
-					    space2, isl_dim_out);
-	isl_space_free(space1);
-	isl_space_free(space2);
-
-	return complete;
-}
-
-/* Does "expr" access a single, fixed element (independently of the statement
- * instance)?
- * That is, does it have a completely specified constant index expression?
- *
- * Note that it is not sufficient for the index expression to be
- * piecewise constant.  isl_multi_pw_aff_is_cst can therefore not be used.
- */
-static isl_bool accesses_fixed_element(__isl_keep pet_expr *expr)
-{
-	int i, n;
-	isl_multi_pw_aff *index;
-	isl_bool fixed = isl_bool_true;
-
-	index = pet_expr_access_get_index(expr);
-	if (index < 0)
-		return isl_bool_error;
-	n = isl_multi_pw_aff_dim(index, isl_dim_out);
-	for (i = 0; i < n; ++i) {
-		isl_pw_aff *pa;
-
-		pa = isl_multi_pw_aff_get_pw_aff(index, 0);
-		fixed = isl_pw_aff_n_piece(pa) == 1;
-		if (fixed)
-			fixed = isl_pw_aff_is_cst(pa);
-		isl_pw_aff_free(pa);
-		if (fixed < 0 || !fixed)
-			break;
-	}
-	if (fixed >= 0 && fixed)
-		fixed = complete_index(expr, index);
-	isl_multi_pw_aff_free(index);
-
-	return fixed;
-}
-
-/* Extract a gpu_stmt_access from "expr", append it to the list
- * that ends in *data->next_access and update the end of the list.
- * If the access expression performs a write, then it is considered
- * exact only if it appears in a single expression statement and
- * if its may access relation is equal to its must access relation.
- *
- * The combined set of may accesses may be a union if member accesses
- * are involved, but the entire set is derived from a single reference and
- * therefore from a single index expression.  These accesses therefore
- * all map to the same outer array.
- */
-static int extract_access(__isl_keep pet_expr *expr, void *user)
-{
-	struct ppcg_extract_access_data *data = user;
-	isl_union_map *tagged;
-	struct gpu_stmt_access *access;
-	isl_ctx *ctx = pet_expr_get_ctx(expr);
-	isl_multi_pw_aff *index;
-
-	access = isl_alloc_type(ctx, struct gpu_stmt_access);
-	assert(access);
-	access->next = NULL;
-	access->read = pet_expr_access_is_read(expr);
-	access->write = pet_expr_access_is_write(expr);
-	tagged = pet_expr_access_get_tagged_may_read(expr);
-	tagged = isl_union_map_union(tagged,
-				pet_expr_access_get_tagged_may_write(expr));
-	tagged = isl_union_map_apply_range(tagged,
-					isl_union_map_copy(data->any_to_outer));
-	if (!access->write) {
-		access->exact_write = 1;
-	} else if (!data->single_expression) {
-		access->exact_write = 0;
-	} else {
-		isl_union_map *must, *may;
-		may = isl_union_map_copy(tagged);
-		may = isl_union_map_domain_factor_domain(may);
-		must = pet_expr_access_get_must_write(expr);
-		access->exact_write = isl_union_map_is_equal(must, may);
-		isl_union_map_free(must);
-		isl_union_map_free(may);
-	}
-	index = pet_expr_access_get_index(expr);
-	access->n_index = isl_multi_pw_aff_dim(index, isl_dim_out);
-	isl_multi_pw_aff_free(index);
-	access->ref_id = pet_expr_access_get_ref_id(expr);
-	access->tagged_access = extract_single_tagged_access(tagged, expr);
-	access->access = isl_map_copy(access->tagged_access);
-	access->access = isl_map_domain_factor_domain(access->access);
-	access->fixed_element = accesses_fixed_element(expr);
-
-	*data->next_access = access;
-	data->next_access = &(*data->next_access)->next;
-
-	if (!access->access || access->fixed_element < 0)
-		return -1;
-
-	return 0;
-}
-
-/* Construct a linked list of gpu_stmt_access objects,
- * one for each access expression in the statement body.
- * "any_to_outer" maps all intermediate arrays to their outer arrays.
- */
-static int pet_stmt_extract_accesses(struct gpu_stmt *stmt,
-	__isl_keep isl_union_map *any_to_outer)
-{
-	struct ppcg_extract_access_data data;
-
-	stmt->accesses = NULL;
-	data.next_access = &stmt->accesses;
-	data.single_expression =
-		pet_tree_get_type(stmt->stmt->body) == pet_tree_expr;
-	data.any_to_outer = any_to_outer;
-	return pet_tree_foreach_access_expr(stmt->stmt->body,
-						&extract_access, &data);
-}
-
-/* Has statement "stmt" been killed from "scop"?
- * That is, is the instance set of "scop" free from any
- * instances of "stmt"?
- */
-static isl_bool is_stmt_killed(struct ppcg_scop *scop, struct pet_stmt *stmt)
-{
-	isl_space *space;
-	isl_set *left;
-	isl_bool empty;
-
-	if (!scop || !stmt)
-		return isl_bool_error;
-	space = isl_set_get_space(stmt->domain);
-	left = isl_union_set_extract_set(scop->domain, space);
-	empty = isl_set_plain_is_empty(left);
-	isl_set_free(left);
-
-	return empty;
-}
-
-/* Return an array of gpu_stmt representing the statements in "scop".
- * Do not collect array accesses for statements that have been killed.
- */
-static struct gpu_stmt *extract_stmts(isl_ctx *ctx, struct ppcg_scop *scop,
-	__isl_keep isl_union_map *any_to_outer)
-{
-	int i;
-	struct gpu_stmt *stmts;
-
-	stmts = isl_calloc_array(ctx, struct gpu_stmt, scop->pet->n_stmt);
-	if (!stmts)
-		return NULL;
-
-	for (i = 0; i < scop->pet->n_stmt; ++i) {
-		struct gpu_stmt *s = &stmts[i];
-		isl_bool killed;
-
-		s->id = isl_set_get_tuple_id(scop->pet->stmts[i]->domain);
-		s->stmt = scop->pet->stmts[i];
-		killed = is_stmt_killed(scop, scop->pet->stmts[i]);
-		if (killed < 0)
-			return free_stmts(stmts, i + 1);
-		if (killed)
-			continue;
-		if (pet_stmt_extract_accesses(s, any_to_outer) < 0)
-			return free_stmts(stmts, i + 1);
-	}
-
-	return stmts;
-}
-
-/* Generate CUDA code for "scop" and print it to "p".
- * After generating an AST for the transformed scop as explained below,
- * we call "gen->print" to print the AST in the desired output format
- * to "p".
- *
- * If it turns out that it does not make sense to generate GPU code,
- * then we generate CPU code instead.
- *
- * The declarations of the arrays that are visible outside of the scop
- * are printed outside of the code generated from the schedule,
- * because the generated code may involve a guard around the entire code.
- *
- * We first compute a schedule that respects the dependences
- * of the original program and select the outermost bands
- * of tilable dimensions that have at least one parallel loop.
- * If the --load-schedule is specified, then the loaded schedule
- * is used instead of a computed schedule.
- *
- * Each of these bands B is then tiled according to "tile" sizes, resulting
- * in two nested bands, with a kernel marker on top
- *
- *		K
- *		|
- *		T
- *		|
- *		P
- *
- * We then split off at most 2 parallel dimensions from the T band and
- * at most 3 parallel dimension from the P band
- *
- *		K
- *		|
- *		T
- *		T1
- *		|
- *		T2
- *		|
- *		P1
- *		|
- *		P2
- *
- * A filter is introduced in front of T1 that maps the domain instances
- * to block identifiers.  Similarly, a filter is introduced in front of P1
- * that maps the domain instances to thread identifiers.
- *
- * For each iteration of the T2 band and for each array, we compute
- * the array elements accessed by that iteration, construct a rectangular
- * box around it and shift it to the origin.  The result is used
- * as shared memory for the array.
- *
- * Copying and synchronization statements are added to this schedule tree.
- * In principle, these are added in front of the P1 band, but some of
- * them may get hoisted up to higher levels.
- *
- * The entire AST is then generated from the single resulting schedule tree.
- * During the generation the subtrees at kernel nodes (K) are saved
- * aside and replaced by kernel calls.  The result is printed as host code
- * while the saved subtrees are printed as device code.
- */
-static __isl_give isl_printer *generate(__isl_take isl_printer *p,
-	struct gpu_gen *gen, struct ppcg_scop *scop,
-	struct ppcg_options *options)
-{
-	struct gpu_prog *prog;
-	isl_ctx *ctx;
-	isl_schedule *schedule;
-	int any_permutable;
-
-	if (!scop)
-		return isl_printer_free(p);
-
-	ctx = isl_printer_get_ctx(p);
-	prog = gpu_prog_alloc(ctx, scop);
-	if (!prog)
-		return isl_printer_free(p);
-
-	gen->prog = prog;
-	schedule = get_schedule(gen);
-
-	any_permutable = has_any_permutable_node(schedule);
-	if (any_permutable < 0 || !any_permutable) {
-		if (any_permutable < 0)
-			p = isl_printer_free(p);
-		else
-			p = print_cpu(p, scop, options);
-		isl_schedule_free(schedule);
-	} else {
-		const int create_to_from_device = 1;
-		schedule = map_to_device(gen, schedule, create_to_from_device);
-		gen->tree = generate_code(gen, schedule);
-		p = ppcg_set_macro_names(p);
-		p = ppcg_print_exposed_declarations(p, prog->scop);
-		p = gen->print(p, gen->prog, gen->tree, &gen->types,
-				    gen->print_user);
-		isl_ast_node_free(gen->tree);
-	}
-
-	gpu_prog_free(prog);
-
-	return p;
-}
-
-/* Wrapper around generate for use as a ppcg_transform callback.
- */
-static __isl_give isl_printer *generate_wrap(__isl_take isl_printer *p,
-	struct ppcg_scop *scop, void *user)
-{
-	struct gpu_gen *gen = user;
-
-	return generate(p, gen, scop, gen->options);
-}
-
-/* Transform the code in the file called "input" by replacing
- * all scops by corresponding GPU code and write the results to "out".
- */
-int generate_gpu(isl_ctx *ctx, const char *input, FILE *out,
-	struct ppcg_options *options,
-	__isl_give isl_printer *(*print)(__isl_take isl_printer *p,
-		struct gpu_prog *prog, __isl_keep isl_ast_node *tree,
-		struct gpu_types *types, void *user), void *user)
-{
-	struct gpu_gen gen;
-	int r;
-	int i;
-
-	gen.ctx = ctx;
-	gen.sizes = extract_sizes_from_str(ctx, options->sizes);
-	gen.options = options;
-	gen.kernel_id = 0;
-	gen.print = print;
-	gen.print_user = user;
-	gen.types.n = 0;
-	gen.types.name = NULL;
-
-	if (options->debug->dump_sizes) {
-		isl_space *space = isl_space_params_alloc(ctx, 0);
-		gen.used_sizes = isl_union_map_empty(space);
-	}
-
-	r = ppcg_transform(ctx, input, out, options, &generate_wrap, &gen);
-
-	if (options->debug->dump_sizes) {
-		isl_union_map_dump(gen.used_sizes);
-		isl_union_map_free(gen.used_sizes);
-	}
-
-	isl_union_map_free(gen.sizes);
-	for (i = 0; i < gen.types.n; ++i)
-		free(gen.types.name[i]);
-	free(gen.types.name);
-
-	return r;
-}
-
-/* Compute the set of inner array elements that may have their values
- * preserved by "prog".  In particular, collect the array elements of
- * arrays that are not local to "prog" and remove those elements that
- * are definitely killed or definitely written by "prog".
- */
-__isl_give isl_union_set *compute_may_persist(struct gpu_prog *prog)
-{
-	int i;
-	isl_union_set *may_persist, *killed;
-	isl_union_map *must_kill;
-
-	may_persist = isl_union_set_empty(isl_set_get_space(prog->context));
-	for (i = 0; i < prog->n_array; ++i) {
-		isl_set *extent;
-
-		if (prog->array[i].local)
-			continue;
-
-		extent = isl_set_copy(prog->array[i].extent);
-		may_persist = isl_union_set_add_set(may_persist, extent);
-	}
-
-	may_persist = isl_union_set_intersect_params(may_persist,
-						isl_set_copy(prog->context));
-	may_persist = isl_union_set_apply(may_persist,
-					isl_union_map_copy(prog->to_inner));
-	must_kill = isl_union_map_copy(prog->tagged_must_kill);
-	killed = isl_union_map_range(must_kill);
-	must_kill = isl_union_map_copy(prog->must_write);
-	killed = isl_union_set_union(killed, isl_union_map_range(must_kill));
-
-	may_persist = isl_union_set_subtract(may_persist, killed);
-	return may_persist;
-}
-
-struct gpu_prog *gpu_prog_alloc(isl_ctx *ctx, struct ppcg_scop *scop)
-{
-	struct gpu_prog *prog;
-	isl_space *space;
-	isl_map *id;
-
-	if (!scop)
-		return NULL;
-
-	prog = isl_calloc_type(ctx, struct gpu_prog);
-	assert(prog);
-
-	prog->ctx = ctx;
-	prog->scop = scop;
-	prog->context = isl_set_copy(scop->context);
-	prog->n_stmts = scop->pet->n_stmt;
-	prog->any_to_outer = pet_scop_compute_outer_to_any(scop->pet);
-	prog->any_to_outer = isl_union_map_reverse(prog->any_to_outer);
-	space = isl_union_map_get_space(prog->any_to_outer);
-	space = isl_space_set_from_params(space);
-	space = isl_space_add_dims(space, isl_dim_set, 1);
-	space = isl_space_map_from_set(space);
-	id = isl_map_identity(space);
-	prog->any_to_outer = isl_union_map_add_map(prog->any_to_outer, id);
-	prog->stmts = extract_stmts(ctx, scop, prog->any_to_outer);
-	prog->read = isl_union_map_copy(scop->reads);
-	prog->may_write = isl_union_map_copy(scop->may_writes);
-	prog->must_write = isl_union_map_copy(scop->must_writes);
-	prog->tagged_must_kill = isl_union_map_copy(scop->tagged_must_kills);
-	prog->to_inner = pet_scop_compute_outer_to_inner(scop->pet);
-	prog->to_outer = isl_union_map_copy(prog->to_inner);
-	prog->to_outer = isl_union_map_reverse(prog->to_outer);
-
-	if (!prog->stmts)
-		return gpu_prog_free(prog);
-
-	if (collect_array_info(prog) < 0)
-		return gpu_prog_free(prog);
-	prog->may_persist = compute_may_persist(prog);
-
-	return prog;
-}
-
-void *gpu_prog_free(struct gpu_prog *prog)
-{
-	if (!prog)
-		return NULL;
-	free_array_info(prog);
-	free_stmts(prog->stmts, prog->n_stmts);
-	isl_union_map_free(prog->any_to_outer);
-	isl_union_map_free(prog->to_outer);
-	isl_union_map_free(prog->to_inner);
-	isl_union_map_free(prog->read);
-	isl_union_map_free(prog->may_write);
-	isl_union_map_free(prog->must_write);
-	isl_union_map_free(prog->tagged_must_kill);
-	isl_union_map_free(prog->array_order);
-	isl_union_set_free(prog->may_persist);
-	isl_set_free(prog->context);
-	free(prog);
-	return NULL;
-}
diff --git a/polly/lib/External/ppcg/gpu.h b/polly/lib/External/ppcg/gpu.h
deleted file mode 100644
index 62fb0f741bbe2..0000000000000
--- a/polly/lib/External/ppcg/gpu.h
+++ /dev/null
@@ -1,459 +0,0 @@
-#ifndef _GPU_H
-#define _GPU_H
-
-#include <isl/ast.h>
-#include <isl/id.h>
-#include <isl/id_to_ast_expr.h>
-
-#include <pet.h>
-
-#include "ppcg.h"
-#include "ppcg_options.h"
-
-/* An access to an outer array element or an iterator.
- * Accesses to iterators have an access relation that maps to an unnamed space.
- * An access may be both read and write.
- * If the access relation is empty, then the output dimension may
- * not be equal to the dimension of the corresponding array.
- */
-struct gpu_stmt_access {
-	/* Access reads elements */
-	int read;
-	/* Access writes elements */
-	int write;
-	/* All writes are definite writes. */
-	int exact_write;
-	/* Is a single, fixed element being accessed? */
-	isl_bool fixed_element;
-	/* The number of index expressions specified in the access. */
-	int n_index;
-
-	/* May access relation */
-	isl_map *access;
-	/* May access relation with as domain a mapping from iteration domain
-	 * to a reference identifier.
-	 */
-	isl_map *tagged_access;
-	/* The reference id of the corresponding pet_expr. */
-	isl_id *ref_id;
-
-	struct gpu_stmt_access *next;
-};
-
-/* A representation of a user statement.
- * "stmt" points to the corresponding pet statement.
- * "id" is the identifier of the instance set of the statement.
- * "accesses" is a linked list of accesses performed by the statement.
- * If the statement has been killed, i.e., if it will not be scheduled,
- * then this linked list may be empty even if the actual statement does
- * perform accesses.
- */
-struct gpu_stmt {
-	isl_id *id;
-	struct pet_stmt *stmt;
-
-	struct gpu_stmt_access *accesses;
-};
-
-/* Represents an outer array possibly accessed by a gpu_prog.
- */
-struct gpu_array_info {
-	/* The array data space. */
-	isl_space *space;
-	/* Element type. */
-	char *type;
-	/* Element size. */
-	int size;
-	/* Name of the array. */
-	char *name;
-	/* Declared extent of original array. */
-	isl_set *declared_extent;
-	/* AST expression for declared size of original array. */
-	isl_ast_expr *declared_size;
-	/* Extent of the array that needs to be copied. */
-	isl_set *extent;
-	/* Number of indices. */
-	unsigned n_index;
-	/* For each index, a bound on "extent" in that direction. */
-	isl_multi_pw_aff *bound;
-	/* The corresponding access AST expression, if the array needs
-	 * to be allocated on the device.
-	 */
-	isl_ast_expr *bound_expr;
-
-	/* All references to this array; point to elements of a linked list. */
-	int n_ref;
-	struct gpu_stmt_access **refs;
-
-	/* Is this array accessed at all by the program? */
-	int accessed;
-
-	/* Is this a scalar that is read-only within the entire program? */
-	int read_only_scalar;
-
-	/* Are the elements of the array structures? */
-	int has_compound_element;
-
-	/* Are the elements only accessed through constant index expressions? */
-	int only_fixed_element;
-
-	/* Is the array local to the scop? */
-	int local;
-	/* Is the array local and should it be declared on the host? */
-	int declare_local;
-
-	/* Is the corresponding global device memory accessed in any way? */
-	int global;
-
-	/* Should the array be linearized? */
-	int linearize;
-
-	/* Order dependences on this array.
-	 * Only used if live_range_reordering option is set.
-	 * It is set to NULL otherwise.
-	 */
-	isl_union_map *dep_order;
-
-    void *user;
-};
-
-/* Represents an outer array accessed by a ppcg_kernel, localized
- * to the context of this kernel.
- *
- * "array" points to the corresponding array in the gpu_prog.
- * The "n_group" "groups" are the reference groups associated to the array.
- * If "force_private" is set, then the array (in practice a scalar)
- * must be mapped to a register.
- * "global" is set if the global device memory corresponding
- * to this array is accessed by the kernel.
- * "bound" is equal to array->bound specialized to the current kernel.
- * "bound_expr" is the corresponding access AST expression.
- */
-struct gpu_local_array_info {
-	struct gpu_array_info *array;
-
-	int n_group;
-	struct gpu_array_ref_group **groups;
-
-	int force_private;
-	int global;
-
-	unsigned n_index;
-	isl_multi_pw_aff *bound;
-	isl_ast_expr *bound_expr;
-};
-
-__isl_give isl_ast_expr *gpu_local_array_info_linearize_index(
-	struct gpu_local_array_info *array, __isl_take isl_ast_expr *expr);
-
-/* A sequence of "n" names of types.
- */
-struct gpu_types {
-	int n;
-	char **name;
-};
-
-/* "read" and "write" contain the original access relations, possibly
- * involving member accesses.
- *
- * The elements of "array", as well as the ranges of "copy_in" and "copy_out"
- * only refer to the outer arrays of any possible member accesses.
- */
-struct gpu_prog {
-	isl_ctx *ctx;
-
-	struct ppcg_scop *scop;
-
-	/* Set of parameter values */
-	isl_set *context;
-
-	/* All potential read accesses in the entire program */
-	isl_union_map *read;
-
-	/* All potential write accesses in the entire program */
-	isl_union_map *may_write;
-	/* All definite write accesses in the entire program */
-	isl_union_map *must_write;
-	/* All tagged definite kills in the entire program */
-	isl_union_map *tagged_must_kill;
-
-	/* The set of inner array elements that may be preserved. */
-	isl_union_set *may_persist;
-
-	/* A mapping from all innermost arrays to their outer arrays. */
-	isl_union_map *to_outer;
-	/* A mapping from the outer arrays to all corresponding inner arrays. */
-	isl_union_map *to_inner;
-	/* A mapping from all intermediate arrays to their outer arrays,
-	 * including an identity mapping from the anonymous 1D space to itself.
-	 */
-	isl_union_map *any_to_outer;
-
-	/* Order dependences on non-scalars. */
-	isl_union_map *array_order;
-
-	/* Array of statements */
-	int n_stmts;
-	struct gpu_stmt *stmts;
-
-	int n_array;
-	struct gpu_array_info *array;
-};
-
-struct gpu_gen {
-	isl_ctx *ctx;
-	struct ppcg_options *options;
-
-	/* Callback for printing of AST in appropriate format. */
-	__isl_give isl_printer *(*print)(__isl_take isl_printer *p,
-		struct gpu_prog *prog, __isl_keep isl_ast_node *tree,
-		struct gpu_types *types, void *user);
-	void *print_user;
-
-    isl_id_to_ast_expr *(*build_ast_expr)(void *stmt,
-            isl_ast_build *build,
-            isl_multi_pw_aff *(*fn_index)(
-                __isl_take isl_multi_pw_aff *mpa, isl_id *id,
-                void *user),
-            void *user_index,
-            isl_ast_expr *(*fn_expr)(isl_ast_expr *expr,
-                isl_id *id, void *user),
-        void *user_expr);
-
-	struct gpu_prog *prog;
-	/* The generated AST. */
-	isl_ast_node *tree;
-
-	/* The sequence of types for which a definition has been printed. */
-	struct gpu_types types;
-
-	/* User specified tile, grid and block sizes for each kernel */
-	isl_union_map *sizes;
-
-	/* Effectively used tile, grid and block sizes for each kernel */
-	isl_union_map *used_sizes;
-
-	/* Identifier of the next kernel. */
-	int kernel_id;
-};
-
-enum ppcg_group_access_type {
-	ppcg_access_global,
-	ppcg_access_shared,
-	ppcg_access_private
-};
-
-enum ppcg_kernel_stmt_type {
-	ppcg_kernel_copy,
-	ppcg_kernel_domain,
-	ppcg_kernel_sync
-};
-
-/* Representation of special statements, in particular copy statements
- * and __syncthreads statements, inside a kernel.
- *
- * type represents the kind of statement
- *
- *
- * for ppcg_kernel_copy statements we have
- *
- * read is set if the statement should copy data from global memory
- * to shared memory or registers.
- *
- * index expresses an access to the array element that needs to be copied
- * local_index expresses the corresponding element in the tile
- *
- * array refers to the original array being copied
- * local_array is a pointer to the appropriate element in the "array"
- *	array of the ppcg_kernel to which this copy access belongs
- *
- *
- * for ppcg_kernel_domain statements we have
- *
- * stmt is the corresponding input statement
- *
- * n_access is the number of accesses in stmt
- * access is an array of local information about the accesses
- */
-struct ppcg_kernel_stmt {
-	enum ppcg_kernel_stmt_type type;
-
-	union {
-		struct {
-			int read;
-			isl_ast_expr *index;
-			isl_ast_expr *local_index;
-			struct gpu_array_info *array;
-			struct gpu_local_array_info *local_array;
-		} c;
-		struct {
-			struct gpu_stmt *stmt;
-			isl_id_to_ast_expr *ref2expr;
-		} d;
-	} u;
-};
-
-/* Representation of a local variable in a kernel.
- */
-struct ppcg_kernel_var {
-	struct gpu_array_info *array;
-	enum ppcg_group_access_type type;
-	char *name;
-	isl_vec *size;
-};
-
-/* Representation of a kernel.
- *
- * prog describes the original code from which the kernel is extracted.
- *
- * id is the sequence number of the kernel.
- *
- * block_ids contains the list of block identifiers for this kernel.
- * thread_ids contains the list of thread identifiers for this kernel.
- *
- * the first n_grid elements of grid_dim represent the specified size
- * of the grid.
- * the first n_block elements of block_dim represent the specified or
- * effective size of the block.
- * Note that in the input file, the sizes of the grid and the blocks
- * are specified in the order x, y, z, but internally, the sizes
- * are stored in reverse order, so that the last element always
- * refers to the x dimension.
- *
- * grid_size reflects the effective grid size.
- * grid_size_expr contains a corresponding access AST expression, built within
- * the context where the launch appears.
- *
- * context contains the values of the parameters and outer schedule dimensions
- * for which any statement instance in this kernel needs to be executed.
- *
- * n_sync is the number of synchronization operations that have
- * been introduced in the schedule tree corresponding to this kernel (so far).
- *
- * core contains the spaces of the statement domains that form
- * the core computation of the kernel.  It is used to navigate
- * the tree during the construction of the device part of the schedule
- * tree in gpu_create_kernel.
- *
- * expanded_domain contains the original statement instances,
- * i.e., those that appear in the domains of access relations,
- * that are involved in the kernel.
- * contraction maps those original statement instances to
- * the statement instances that are active at the point
- * in the schedule tree where the kernel is created.
- *
- * arrays is the set of possibly accessed outer array elements.
- *
- * space is the schedule space of the AST context.  That is, it represents
- * the loops of the generated host code containing the kernel launch.
- *
- * n_array is the total number of arrays in the input program and also
- * the number of element in the array array.
- * array contains information about each array that is local
- * to the current kernel.  If an array is not used in a kernel,
- * then the corresponding entry does not contain any information.
- *
- * any_force_private is set if any array in the kernel is marked force_private
- *
- * block_filter contains constraints on the domain elements in the kernel
- * that encode the mapping to block identifiers, where the block identifiers
- * are represented by "n_grid" parameters with as names the elements
- * of "block_ids".
- *
- * thread_filter contains constraints on the domain elements in the kernel
- * that encode the mapping to thread identifiers, where the thread identifiers
- * are represented by "n_block" parameters with as names the elements
- * of "thread_ids".
- *
- * copy_schedule corresponds to the schedule dimensions of
- * the (tiled) schedule for this kernel that have been taken into account
- * for computing private/shared memory tiles.
- * The domain corresponds to the original statement instances, i.e.,
- * those that appear in the leaves of the schedule tree.
- * copy_schedule_dim is the dimension of this schedule.
- *
- * sync_writes contains write references that require synchronization.
- * Each reference is represented by a universe set in a space [S[i,j] -> R[]]
- * with S[i,j] the statement instance space and R[] the array reference.
- */
-struct ppcg_kernel {
-	isl_ctx *ctx;
-	struct ppcg_options *options;
-
-	struct gpu_prog *prog;
-
-	int id;
-
-	isl_id_list *block_ids;
-	isl_id_list *thread_ids;
-
-	int n_grid;
-	int n_block;
-	int grid_dim[2];
-	int block_dim[3];
-
-	isl_multi_pw_aff *grid_size;
-	isl_ast_expr *grid_size_expr;
-	isl_set *context;
-
-	int n_sync;
-	isl_union_set *core;
-	isl_union_set *arrays;
-
-	isl_union_pw_multi_aff *contraction;
-	isl_union_set *expanded_domain;
-
-	isl_space *space;
-
-	int n_array;
-	struct gpu_local_array_info *array;
-
-	int n_var;
-	struct ppcg_kernel_var *var;
-
-	int any_force_private;
-
-	isl_union_set *block_filter;
-	isl_union_set *thread_filter;
-	isl_union_pw_multi_aff *copy_schedule;
-	int copy_schedule_dim;
-
-	isl_union_set *sync_writes;
-
-	isl_ast_node *tree;
-};
-
-int gpu_array_is_scalar(struct gpu_array_info *array);
-int gpu_array_is_read_only_scalar(struct gpu_array_info *array);
-int gpu_array_requires_device_allocation(struct gpu_array_info *array);
-__isl_give isl_set *gpu_array_positive_size_guard(struct gpu_array_info *array);
-isl_bool gpu_array_can_be_private(struct gpu_array_info *array);
-
-struct gpu_prog *gpu_prog_alloc(isl_ctx *ctx, struct ppcg_scop *scop);
-void *gpu_prog_free(struct gpu_prog *prog);
-
-int ppcg_kernel_requires_array_argument(struct ppcg_kernel *kernel, int i);
-
-int generate_gpu(isl_ctx *ctx, const char *input, FILE *out,
-	struct ppcg_options *options,
-	__isl_give isl_printer *(*print)(__isl_take isl_printer *p,
-		struct gpu_prog *prog, __isl_keep isl_ast_node *tree,
-		struct gpu_types *types, void *user), void *user);
-
-__isl_give isl_schedule_node *gpu_create_kernel(struct gpu_gen *gen,
-	__isl_take isl_schedule_node *node, int scale,
-	__isl_keep isl_multi_val *sizes);
-
-__isl_give isl_schedule *get_schedule(struct gpu_gen *gen);
-int has_any_permutable_node(__isl_keep isl_schedule *schedule);
-__isl_give isl_schedule *map_to_device(struct gpu_gen *gen,
-                                       __isl_take isl_schedule *schedule,
-                                      int to_from_device);
-__isl_give isl_ast_node *generate_code(struct gpu_gen *gen,
-                                       __isl_take isl_schedule *schedule);
-
-__isl_give isl_union_set *compute_may_persist(struct gpu_prog *prog);
-void collect_references(struct gpu_prog *prog, struct gpu_array_info *array);
-void collect_order_dependences(struct gpu_prog *prog);
-isl_bool only_fixed_element_accessed(struct gpu_array_info *array);
-#endif
diff --git a/polly/lib/External/ppcg/gpu_array_tile.c b/polly/lib/External/ppcg/gpu_array_tile.c
deleted file mode 100644
index b358f2b8e4a72..0000000000000
--- a/polly/lib/External/ppcg/gpu_array_tile.c
+++ /dev/null
@@ -1,71 +0,0 @@
-#include <isl/aff.h>
-#include <isl/map.h>
-
-#include "gpu_array_tile.h"
-
-struct gpu_array_tile *gpu_array_tile_free(struct gpu_array_tile *tile)
-{
-	int j;
-
-	if (!tile)
-		return NULL;
-
-	for (j = 0; j < tile->n; ++j) {
-		isl_val_free(tile->bound[j].size);
-		isl_val_free(tile->bound[j].stride);
-		isl_aff_free(tile->bound[j].lb);
-		isl_aff_free(tile->bound[j].shift);
-	}
-	free(tile->bound);
-	isl_multi_aff_free(tile->tiling);
-	free(tile);
-
-	return NULL;
-}
-
-/* Create a gpu_array_tile for an array of dimension "n_index".
- */
-struct gpu_array_tile *gpu_array_tile_create(isl_ctx *ctx, int n_index)
-{
-	int i;
-	struct gpu_array_tile *tile;
-
-	tile = isl_calloc_type(ctx, struct gpu_array_tile);
-	if (!tile)
-		return NULL;
-
-	tile->ctx = ctx;
-	tile->bound = isl_alloc_array(ctx, struct gpu_array_bound, n_index);
-	if (!tile->bound)
-		return gpu_array_tile_free(tile);
-
-	tile->n = n_index;
-
-	for (i = 0; i < n_index; ++i) {
-		tile->bound[i].size = NULL;
-		tile->bound[i].lb = NULL;
-		tile->bound[i].stride = NULL;
-		tile->bound[i].shift = NULL;
-	}
-
-	return tile;
-}
-
-/* Compute the size of the tile specified by "tile"
- * in number of elements and return the result.
- */
-__isl_give isl_val *gpu_array_tile_size(struct gpu_array_tile *tile)
-{
-	int i;
-	isl_val *size;
-
-	if (!tile)
-		return NULL;
-
-	size = isl_val_one(tile->ctx);
-
-	for (i = 0; i < tile->n; ++i)
-		size = isl_val_mul(size, isl_val_copy(tile->bound[i].size));
-
-	return size;
-}
diff --git a/polly/lib/External/ppcg/gpu_array_tile.h b/polly/lib/External/ppcg/gpu_array_tile.h
deleted file mode 100644
index 53b8e3db74c10..0000000000000
--- a/polly/lib/External/ppcg/gpu_array_tile.h
+++ /dev/null
@@ -1,59 +0,0 @@
-#ifndef GPU_ARRAY_TILE_H
-#define GPU_ARRAY_TILE_H
-
-#include <isl/aff_type.h>
-#include <isl/map_type.h>
-#include <isl/val.h>
-
-/* The fields stride and shift only contain valid information
- * if shift != NULL.
- * If so, they express that current index is such that if you add shift,
- * then the result is always a multiple of stride.
- * Let D represent the initial tile->depth dimensions of the computed schedule.
- * The spaces of "lb" and "shift" are of the form
- *
- *	D -> [b]
- */
-struct gpu_array_bound {
-	isl_val *size;
-	isl_aff *lb;
-
-	isl_val *stride;
-	isl_aff *shift;
-};
-
-/* A tile of an outer array.
- *
- * requires_unroll is set if the schedule dimensions that are mapped
- * to threads need to be unrolled for this (private) tile to be used.
- *
- * "depth" reflects the number of schedule dimensions that affect the tile.
- * The copying into and/or out of the tile is performed at that depth.
- *
- * n is the dimension of the array.
- * bound is an array of size "n" representing the lower bound
- *	and size for each index.
- *
- * tiling maps a tile in the global array to the corresponding
- * shared/private memory tile and is of the form
- *
- *	{ [D[i] -> A[a]] -> T[(a + shift(i))/stride - lb(i)] }
- *
- * where D represents the initial "depth" dimensions
- * of the computed schedule.
- */
-struct gpu_array_tile {
-	isl_ctx *ctx;
-	int requires_unroll;
-	int depth;
-	int n;
-	struct gpu_array_bound *bound;
-	isl_multi_aff *tiling;
-};
-
-struct gpu_array_tile *gpu_array_tile_create(isl_ctx *ctx, int n_index);
-struct gpu_array_tile *gpu_array_tile_free(struct gpu_array_tile *tile);
-
-__isl_give isl_val *gpu_array_tile_size(struct gpu_array_tile *tile);
-
-#endif
diff --git a/polly/lib/External/ppcg/gpu_group.c b/polly/lib/External/ppcg/gpu_group.c
deleted file mode 100644
index 7b869b978c4f1..0000000000000
--- a/polly/lib/External/ppcg/gpu_group.c
+++ /dev/null
@@ -1,1828 +0,0 @@
-/*
- * Copyright 2010-2011 INRIA Saclay
- * Copyright 2012-2014 Ecole Normale Superieure
- * Copyright 2015      Sven Verdoolaege
- *
- * Use of this software is governed by the MIT license
- *
- * Written by Sven Verdoolaege, INRIA Saclay - Ile-de-France,
- * Parc Club Orsay Universite, ZAC des vignes, 4 rue Jacques Monod,
- * 91893 Orsay, France
- * and Ecole Normale Superieure, 45 rue d'Ulm, 75230 Paris, France
- */
-
-#include <isl/constraint.h>
-#include <isl/ilp.h>
-
-#include "gpu_array_tile.h"
-#include "gpu_group.h"
-#include "gpu_tree.h"
-#include "schedule.h"
-
-/* Print the name of the local copy of a given group of array references.
- */
-__isl_give isl_printer *gpu_array_ref_group_print_name(
-	struct gpu_array_ref_group *group, __isl_take isl_printer *p)
-{
-	int global = 0;
-	enum ppcg_group_access_type type;
-
-	type = gpu_array_ref_group_type(group);
-	if (type == ppcg_access_private)
-		p = isl_printer_print_str(p, "private_");
-	else if (type == ppcg_access_shared)
-		p = isl_printer_print_str(p, "shared_");
-	else
-		global = 1;
-	p = isl_printer_print_str(p, group->array->name);
-	if (!global && group->local_array->n_group > 1) {
-		p = isl_printer_print_str(p, "_");
-		p = isl_printer_print_int(p, group->nr);
-	}
-
-	return p;
-}
-
-/* Return the union of all read (read = 1) and/or write (write = 1)
- * access relations in the group.
- */
-__isl_give isl_union_map *gpu_array_ref_group_access_relation(
-	struct gpu_array_ref_group *group, int read, int write)
-{
-	int i;
-	isl_union_map *access;
-
-	access = isl_union_map_empty(isl_map_get_space(group->access));
-	for (i = 0; i < group->n_ref; ++i) {
-		isl_map *map_i;
-
-		if (!((read && group->refs[i]->read) ||
-		     (write && group->refs[i]->write)))
-			continue;
-		map_i = isl_map_copy(group->refs[i]->access);
-		access = isl_union_map_union(access,
-					    isl_union_map_from_map(map_i));
-	}
-
-	return access;
-}
-
-/* Should this array reference group be mapped to private, shared or global
- * memory?
- * If we have computed both a private and a shared tile, then
- * the tile with the smallest depth is used.  If both have the same depth,
- * then the private tile is used.
- */
-enum ppcg_group_access_type gpu_array_ref_group_type(
-	struct gpu_array_ref_group *group)
-{
-	if (group->private_tile && group->shared_tile &&
-	    group->shared_tile->depth < group->private_tile->depth)
-		return ppcg_access_shared;
-	if (group->private_tile)
-		return ppcg_access_private;
-	if (group->shared_tile)
-		return ppcg_access_shared;
-	return ppcg_access_global;
-}
-
-
-/* Return the effective gpu_array_tile associated to "group" or
- * NULL if there is no such gpu_array_tile.
- */
-struct gpu_array_tile *gpu_array_ref_group_tile(
-	struct gpu_array_ref_group *group)
-{
-	switch (gpu_array_ref_group_type(group)) {
-	case ppcg_access_global:
-		return NULL;
-	case ppcg_access_shared:
-		return group->shared_tile;
-	case ppcg_access_private:
-		return group->private_tile;
-	}
-}
-
-/* Does the tile associated to "group" require unrolling of the schedule
- * dimensions mapped to threads?
- * Note that this can only happen for private tiles.
- */
-int gpu_array_ref_group_requires_unroll(struct gpu_array_ref_group *group)
-{
-	struct gpu_array_tile *tile;
-
-	tile = gpu_array_ref_group_tile(group);
-	if (!tile)
-		return 0;
-	return tile->requires_unroll;
-}
-
-/* Given a constraint
- *
- *		a(p,i) + j = g f(e)
- *
- * or -a(p,i) - j = g f(e) if sign < 0,
- * store a(p,i) in bound->shift and g (stride) in bound->stride.
- * a(p,i) is assumed to be an expression in only the parameters
- * and the input dimensions.
- */
-static void extract_stride(__isl_keep isl_constraint *c,
-	struct gpu_array_bound *bound, __isl_keep isl_val *stride, int sign)
-{
-	int i;
-	isl_val *v;
-	isl_space *space;
-	unsigned nparam;
-	unsigned nvar;
-	isl_aff *aff;
-
-	isl_val_free(bound->stride);
-	bound->stride = isl_val_copy(stride);
-
-	space = isl_constraint_get_space(c);
-	space = isl_space_domain(space);
-
-	nparam = isl_space_dim(space, isl_dim_param);
-	nvar = isl_space_dim(space, isl_dim_set);
-
-	v = isl_constraint_get_constant_val(c);
-	if (sign < 0)
-		v = isl_val_neg(v);
-	aff = isl_aff_zero_on_domain(isl_local_space_from_space(space));
-	aff = isl_aff_set_constant_val(aff, v);
-
-	for (i = 0; i < nparam; ++i) {
-		if (!isl_constraint_involves_dims(c, isl_dim_param, i, 1))
-			continue;
-		v = isl_constraint_get_coefficient_val(c, isl_dim_param, i);
-		if (sign < 0)
-			v = isl_val_neg(v);
-		aff = isl_aff_add_coefficient_val(aff, isl_dim_param, i, v);
-	}
-
-	for (i = 0; i < nvar; ++i) {
-		if (!isl_constraint_involves_dims(c, isl_dim_in, i, 1))
-			continue;
-		v = isl_constraint_get_coefficient_val(c, isl_dim_in, i);
-		if (sign < 0)
-			v = isl_val_neg(v);
-		aff = isl_aff_add_coefficient_val(aff, isl_dim_in, i, v);
-	}
-
-	bound->shift = aff;
-}
-
-/* Given an equality constraint of a map with a single output dimension j,
- * check if the constraint is of the form
- *
- *		a(p,i) + j = g f(e)
- *
- * with a(p,i) an expression in the parameters and input dimensions
- * and f(e) an expression in the existentially quantified variables.
- * If so, and if g is larger than any such g from a previously considered
- * constraint, then call extract_stride to record the stride information
- * in bound.
- */
-static isl_stat check_stride_constraint(__isl_take isl_constraint *c,
-	void *user)
-{
-	int i;
-	isl_ctx *ctx;
-	isl_val *v;
-	unsigned n_div;
-	struct gpu_array_bound *bound = user;
-
-	ctx = isl_constraint_get_ctx(c);
-	n_div = isl_constraint_dim(c, isl_dim_div);
-	v = isl_constraint_get_coefficient_val(c, isl_dim_out, 0);
-
-	if (n_div && (isl_val_is_one(v) || isl_val_is_negone(v))) {
-		int s = isl_val_sgn(v);
-		isl_val *stride = isl_val_zero(ctx);
-
-		isl_val_free(v);
-		for (i = 0; i < n_div; ++i) {
-			v = isl_constraint_get_coefficient_val(c,
-								isl_dim_div, i);
-			stride = isl_val_gcd(stride, v);
-		}
-		if (!isl_val_is_zero(stride) &&
-		    isl_val_gt(stride, bound->stride))
-			extract_stride(c, bound, stride, s);
-
-		isl_val_free(stride);
-	} else
-		isl_val_free(v);
-
-	isl_constraint_free(c);
-	return isl_stat_ok;
-}
-
-/* Given contraints on an array index i, check if we can find
- * a shift a(p) and a stride g such that
- *
- *	a(p) + i = 0 mod g
- *
- * If so, record the information in bound and apply the mapping
- * i -> (i + a(p))/g to the array index in bounds and return
- * the new constraints.
- * If not, simply return the original constraints.
- *
- * If bounds is a subset of the space
- *
- *	D -> i
- *
- * then the bound recorded in bound->shift is of the form
- *
- *	D -> s(D)
- *
- * with s(D) equal to a(p) above.
- * Next, we construct a mapping of the form
- *
- *	[D -> i] -> [D -> (i + S(D))/g]
- *
- * This mapping is computed as follows.
- * We first introduce "i" in the domain through precomposition
- * with [D -> i] -> D obtaining
- *
- *	[D -> i] -> s(D)
- *
- * Adding [D -> i] -> i produces
- *
- *	[D -> i] -> i + s(D)
- *
- * and the domain product with [D -> i] -> D yields
- *
- *	[D -> i] -> [D -> i + s(D)]
- *
- * Composition with [D -> i] -> [D -> i/g] gives the desired result.
- */
-static __isl_give isl_basic_map *check_stride(struct gpu_array_bound *bound,
-	__isl_take isl_basic_map *bounds)
-{
-	isl_space *space;
-	isl_basic_map *hull;
-	isl_basic_map *shift, *id, *bmap, *scale;
-	isl_basic_set *bset;
-	isl_aff *aff;
-
-	bound->stride = NULL;
-
-	hull = isl_basic_map_affine_hull(isl_basic_map_copy(bounds));
-
-	isl_basic_map_foreach_constraint(hull, &check_stride_constraint, bound);
-
-	isl_basic_map_free(hull);
-
-	if (!bound->stride)
-		return bounds;
-
-	shift = isl_basic_map_from_aff(isl_aff_copy(bound->shift));
-	space = isl_basic_map_get_space(bounds);
-	bmap = isl_basic_map_domain_map(isl_basic_map_universe(space));
-	shift = isl_basic_map_apply_range(bmap, shift);
-	space = isl_basic_map_get_space(bounds);
-	id = isl_basic_map_range_map(isl_basic_map_universe(space));
-	shift = isl_basic_map_sum(id, shift);
-	space = isl_basic_map_get_space(bounds);
-	id = isl_basic_map_domain_map(isl_basic_map_universe(space));
-	shift = isl_basic_map_range_product(id, shift);
-
-	space = isl_space_domain(isl_basic_map_get_space(bounds));
-	id = isl_basic_map_identity(isl_space_map_from_set(space));
-	space = isl_space_range(isl_basic_map_get_space(bounds));
-	aff = isl_aff_zero_on_domain(isl_local_space_from_space(space));
-	aff = isl_aff_add_coefficient_si(aff, isl_dim_in, 0, 1);
-	aff = isl_aff_scale_down_val(aff, isl_val_copy(bound->stride));
-	scale = isl_basic_map_from_aff(aff);
-	scale = isl_basic_map_product(id, scale);
-
-	bmap = isl_basic_map_apply_range(shift, scale);
-	bset = isl_basic_set_apply(isl_basic_map_wrap(bounds), bmap);
-	bounds = isl_basic_set_unwrap(bset);
-
-	return bounds;
-}
-
-/* Data used in compute_array_dim_size and compute_size_in_direction.
- *
- * pos is the position of the variable representing the array index,
- * i.e., the variable for which want to compute the size.  This variable
- * is also the last variable in the set.
- */
-struct gpu_size_info {
-	isl_basic_set *bset;
-	struct gpu_array_bound *bound;
-	int pos;
-};
-
-/* Given a constraint from the basic set describing the bounds on
- * an array index, check if it is a lower bound, say m i >= b(x), and,
- * if so, check whether the expression "i - ceil(b(x)/m) + 1" has a constant
- * upper bound.  If so, and if this bound is smaller than any bound
- * derived from earlier constraints, set the size to this bound on
- * the expression and the lower bound to ceil(b(x)/m).
- */
-static isl_stat compute_size_in_direction(__isl_take isl_constraint *c,
-	void *user)
-{
-	struct gpu_size_info *size = user;
-	unsigned nparam;
-	unsigned n_div;
-	isl_val *v;
-	isl_aff *aff;
-	isl_aff *lb;
-
-	nparam = isl_basic_set_dim(size->bset, isl_dim_param);
-	n_div = isl_constraint_dim(c, isl_dim_div);
-
-	if (isl_constraint_involves_dims(c, isl_dim_div, 0, n_div) ||
-	    !isl_constraint_is_lower_bound(c, isl_dim_set, size->pos)) {
-		isl_constraint_free(c);
-		return isl_stat_ok;
-	}
-
-	aff = isl_constraint_get_bound(c, isl_dim_set, size->pos);
-	aff = isl_aff_ceil(aff);
-
-	lb = isl_aff_copy(aff);
-
-	aff = isl_aff_neg(aff);
-	aff = isl_aff_add_coefficient_si(aff, isl_dim_in, size->pos, 1);
-
-	v = isl_basic_set_max_val(size->bset, aff);
-	isl_aff_free(aff);
-
-	if (isl_val_is_int(v)) {
-		v = isl_val_add_ui(v, 1);
-		if (!size->bound->size || isl_val_lt(v, size->bound->size)) {
-			isl_val_free(size->bound->size);
-			size->bound->size = isl_val_copy(v);
-			lb = isl_aff_drop_dims(lb, isl_dim_in, size->pos, 1);
-			isl_aff_free(size->bound->lb);
-			size->bound->lb = isl_aff_copy(lb);
-		}
-	}
-	isl_val_free(v);
-	isl_aff_free(lb);
-
-	isl_constraint_free(c);
-
-	return isl_stat_ok;
-}
-
-/* Given a basic map "bounds" that maps parameters and input dimensions
- * to a single output dimension, look for an expression in the parameters
- * and input dimensions such that the range of the output dimension shifted
- * by this expression is a constant.
- *
- * In particular, we currently only consider lower bounds on the output
- * dimension as candidate expressions.
- */
-static int compute_array_dim_size(struct gpu_array_bound *bound,
-	__isl_take isl_basic_map *bounds)
-{
-	struct gpu_size_info size;
-
-	bounds = isl_basic_map_detect_equalities(bounds);
-	bounds = check_stride(bound, bounds);
-
-	bound->size = NULL;
-	bound->lb = NULL;
-
-	size.bound = bound;
-	size.pos = isl_basic_map_dim(bounds, isl_dim_in);
-	size.bset = isl_basic_map_wrap(bounds);
-	size.bset = isl_basic_set_flatten(size.bset);
-	size.bset = isl_set_simple_hull(isl_basic_set_compute_divs(size.bset));
-	isl_basic_set_foreach_constraint(size.bset, &compute_size_in_direction,
-					&size);
-	isl_basic_set_free(size.bset);
-
-	return bound->size ? 0 : -1;
-}
-
-/* Check if we can find a memory tile for the given array
- * based on the given accesses, and if so, put the results in "tile".
- *
- * We project the accesses on each index in turn and look for a parametric
- * offset such that the size is constant.
- *
- * tile->depth is initialized to the input dimension of the computed bounds.
- */
-static int can_tile(__isl_keep isl_map *access, struct gpu_array_tile *tile)
-{
-	int i;
-
-	tile->depth = isl_map_dim(access, isl_dim_in);
-
-	for (i = 0; i < tile->n; ++i) {
-		isl_map *access_i;
-		isl_basic_map *hull;
-
-		access_i = isl_map_copy(access);
-		access_i = isl_map_project_out(access_i, isl_dim_out, 0, i);
-		access_i = isl_map_project_out(access_i, isl_dim_out,
-					    1, tile->n - (i + 1));
-		access_i = isl_map_compute_divs(access_i);
-		hull = isl_map_simple_hull(access_i);
-		if (compute_array_dim_size(&tile->bound[i], hull) < 0)
-			return 0;
-	}
-
-	return 1;
-}
-
-/* Internal data structure for gpu_group_references.
- *
- * scop represents the input scop.
- * kernel_depth is the schedule depth where the kernel launch will
- * be introduced, i.e., it is the depth of the band that is mapped
- * to blocks.
- * shared_depth is the schedule depth at which the copying to/from
- * shared memory is computed.  The copy operation may then
- * later be hoisted to a higher level.
- * thread_depth is the schedule depth where the thread mark is located,
- * i.e., it is the depth of the band that is mapped to threads and also
- * the schedule depth at which the copying to/from private memory
- * is computed.  The copy operation may then later be hoisted to
- * a higher level.
- * n_thread is the number of schedule dimensions in the band that
- * is mapped to threads.
- * privatization lives in the range of thread_sched (i.e., it is
- * of dimension thread_depth + n_thread) and encodes the mapping
- * to thread identifiers (as parameters).
- * host_sched contains the kernel_depth dimensions of the host schedule.
- * shared_sched contains the first shared_depth dimensions of the
- * kernel schedule.
- * copy_sched contains the first thread_depth dimensions of the
- * kernel schedule.
- * thread_sched contains the first (thread_depth + n_thread) dimensions
- * of the kernel schedule.
- * full_sched is a union_map representation of the entire kernel schedule.
- * The schedules are all formulated in terms of the original statement
- * instances, i.e., those that appear in the domains of the access
- * relations.
- */
-struct gpu_group_data {
-	struct ppcg_scop *scop;
-	int kernel_depth;
-	int shared_depth;
-	int thread_depth;
-	int n_thread;
-	isl_set *privatization;
-	isl_union_map *host_sched;
-	isl_union_map *shared_sched;
-	isl_union_map *copy_sched;
-	isl_union_map *thread_sched;
-	isl_union_map *full_sched;
-};
-
-/* Construct a map from domain_space to domain_space that increments
- * the dimension at position "pos" and leaves all other dimensions
- * constant.
- */
-static __isl_give isl_map *next(__isl_take isl_space *domain_space, int pos)
-{
-	isl_space *space;
-	isl_aff *aff;
-	isl_multi_aff *next;
-
-	space = isl_space_map_from_set(domain_space);
-	next = isl_multi_aff_identity(space);
-	aff = isl_multi_aff_get_aff(next, pos);
-	aff = isl_aff_add_constant_si(aff, 1);
-	next = isl_multi_aff_set_aff(next, pos, aff);
-
-	return isl_map_from_multi_aff(next);
-}
-
-/* Check if the given access is coalesced (or if there is no point
- * in trying to coalesce the access by mapping the array to shared memory).
- * That is, check whether incrementing the dimension that will get
- * wrapped over the last thread index results in incrementing
- * the last array index.
- *
- * If no two consecutive array elements are ever accessed by "access",
- * then mapping the corresponding array to shared memory will not
- * improve coalescing.  In fact, the copying will likely be performed
- * by a single thread.  Consider the access as coalesced such that
- * the caller will not try and map the array to shared memory just
- * to improve coalescing.
- *
- * This function is only called for access relations without reuse and
- * kernels with at least one thread identifier.
- */
-static int access_is_coalesced(struct gpu_group_data *data,
-	__isl_keep isl_union_map *access)
-{
-	int dim;
-	isl_space *space;
-	isl_set *accessed;
-	isl_map *access_map;
-	isl_map *next_thread_x;
-	isl_map *next_element;
-	isl_map *map;
-	int coalesced, empty;
-
-	access = isl_union_map_copy(access);
-	access = isl_union_map_apply_domain(access,
-				isl_union_map_copy(data->full_sched));
-	access_map = isl_map_from_union_map(access);
-
-	space = isl_map_get_space(access_map);
-	space = isl_space_range(space);
-	dim = isl_space_dim(space, isl_dim_set);
-	if (dim == 0)
-		next_element = isl_map_empty(isl_space_map_from_set(space));
-	else
-		next_element = next(space, dim - 1);
-
-	accessed = isl_map_range(isl_map_copy(access_map));
-	map = isl_map_copy(next_element);
-	map = isl_map_intersect_domain(map, isl_set_copy(accessed));
-	map = isl_map_intersect_range(map, accessed);
-	empty = isl_map_is_empty(map);
-	isl_map_free(map);
-
-	if (empty < 0 || empty) {
-		isl_map_free(next_element);
-		isl_map_free(access_map);
-		return empty;
-	}
-
-	space = isl_map_get_space(access_map);
-	space = isl_space_domain(space);
-	next_thread_x = next(space, data->thread_depth + data->n_thread - 1);
-
-	map = isl_map_apply_domain(next_thread_x, isl_map_copy(access_map));
-	map = isl_map_apply_range(map, access_map);
-
-	coalesced = isl_map_is_subset(map, next_element);
-
-	isl_map_free(next_element);
-	isl_map_free(map);
-
-	return coalesced;
-}
-
-/* Replace the host schedule dimensions in the access relation "access"
- * by parameters, so that they are treated as fixed when checking for reuse
- * (within a kernel) or whether two consecutive elements are accessed
- * (within a kernel).
- */
-static __isl_give isl_union_map *localize_access(struct gpu_group_data *data,
-	__isl_take isl_union_map *access)
-{
-	int n;
-	isl_space *space;
-	isl_set *param;
-	isl_union_map *umap;
-	isl_id_list *ids;
-
-	umap = isl_union_map_copy(data->host_sched);
-	space = isl_union_map_get_space(umap);
-	n = data->kernel_depth;
-	ids = ppcg_scop_generate_names(data->scop, n, "__ppcg_host_");
-	param = parametrization(space, n, 0, ids);
-	isl_id_list_free(ids);
-	umap = isl_union_map_intersect_range(umap,
-						isl_union_set_from_set(param));
-	access = isl_union_map_intersect_domain(access,
-						isl_union_map_domain(umap));
-
-	return access;
-}
-
-/* Given an access relation in terms of at least data->thread_depth initial
- * dimensions of the computed schedule, check if it is bijective for
- * fixed values of the first data->thread_depth dimensions.
- * We perform this check by equating these dimensions to parameters.
- */
-static int access_is_bijective(struct gpu_group_data *data,
-	__isl_keep isl_map *access)
-{
-	int res;
-	int dim;
-	isl_set *par;
-	isl_space *space;
-	isl_id_list *ids;
-
-	access = isl_map_copy(access);
-	space = isl_space_params(isl_map_get_space(access));
-	ids = ppcg_scop_generate_names(data->scop, data->thread_depth, "s");
-	dim = isl_map_dim(access, isl_dim_in);
-	par = parametrization(space, dim, 0, ids);
-	isl_id_list_free(ids);
-	access = isl_map_intersect_domain(access, par);
-	res = isl_map_is_bijective(access);
-	isl_map_free(access);
-
-	return res;
-}
-
-/* Compute the number of outer schedule tile dimensions that affect
- * the offset of "tile".
- * If there is no such dimension, then return the index
- * of the first kernel dimension, i.e., data->kernel_depth.
- */
-static int compute_tile_depth(struct gpu_group_data *data,
-	struct gpu_array_tile *tile)
-{
-	int i, j;
-
-	for (j = tile->depth - 1; j >= data->kernel_depth; --j) {
-		for (i = 0; i < tile->n; ++i) {
-			isl_aff *lb;
-			isl_aff *shift;
-
-			lb = tile->bound[i].lb;
-			if (isl_aff_involves_dims(lb, isl_dim_in, j, 1))
-				break;
-
-			shift = tile->bound[i].shift;
-			if (!shift)
-				continue;
-			if (isl_aff_involves_dims(shift, isl_dim_in, j, 1))
-				break;
-		}
-		if (i < tile->n)
-			break;
-	}
-
-	return ++j;
-}
-
-/* Return the lowest depth between data->kernel_depth and data->thread_depth
- * at which every array element accessed through "acc" is accessed
- * by a single thread.  The input dimension of "acc" is
- * data->thread_depth + data->n_thread, where the final data->n_thread
- * dimensions are those that will be mapped to threads.
- * If the values for these dimensions are uniquely determined
- * by the array index and a given number of outer dimensions, then
- * there is only one thread accessing that array element within those
- * outer dimensions.
- *
- * The input space of "acc" is first split up, such that it has the form
- *
- *	[O -> T] -> A
- *
- * with O the outer dimensions, T the dimensions that will be mapped to threads
- * and A the array index.
- *
- * Then the positions of T and A are interchanged to simplify the test
- * whether T uniquely depends on O and A.
- * In particular, the above access relation is first combined with
- *
- *	[O -> T] -> T
- *
- * to form
- *
- *	[O -> T] -> [A -> T]
- *
- * from which
- *
- *	O -> [A -> T]
- *
- * is extracted, which is then uncurried to
- *
- *	[O -> A] -> T
- *
- * Finally, the final dimensions of O are projected out one by one
- * until T is no longer uniquely determined by A and the remaining
- * dimensions in O.  The value returned is that of the last dimension
- * that was successfully projected out.
- * Note that there is no need to test whether [O -> A] -> T itself
- * is single-valued as that was already tested in access_is_bijective.
- */
-static int compute_accessed_by_single_thread_depth(struct gpu_group_data *data,
-	__isl_keep isl_map *acc)
-{
-	int i;
-	isl_space *space;
-	isl_map *map;
-	isl_bool sv;
-
-	if (data->thread_depth == data->kernel_depth)
-		return data->thread_depth;
-
-	acc = isl_map_copy(acc);
-
-	space = isl_map_get_space(acc);
-	space = isl_space_params(space);
-	space = isl_space_set_from_params(space);
-	space = isl_space_add_dims(space, isl_dim_set, data->thread_depth);
-	space = isl_space_from_domain(space);
-	space = isl_space_add_dims(space, isl_dim_out, data->n_thread);
-	space = isl_space_wrap(space);
-	map = isl_set_flatten_map(isl_set_universe(space));
-	acc = isl_map_apply_range(map, acc);
-
-	space = isl_space_domain(isl_map_get_space(acc));
-	map = isl_map_range_map(isl_map_universe(isl_space_unwrap(space)));
-	acc = isl_map_range_product(acc, map);
-	acc = isl_map_domain_factor_domain(acc);
-	acc = isl_map_uncurry(acc);
-
-	for (i = data->thread_depth - 1; i >= data->kernel_depth; --i) {
-		acc = isl_map_project_out(acc, isl_dim_in, i, 1);
-		sv = isl_map_is_single_valued(acc);
-		if (sv < 0)
-			return -1;
-		if (!sv)
-			break;
-	}
-
-	isl_map_free(acc);
-
-	return ++i;
-}
-
-/* Adjust the fields of "tile" to reflect the new input dimension "depth".
- * The dimension beyond "depth" are assumed not to affect the tile,
- * so they can simply be dropped.
- */
-static int tile_adjust_depth(struct gpu_array_tile *tile, int depth)
-{
-	int i;
-
-	if (tile->depth == depth)
-		return 0;
-
-	for (i = 0; i < tile->n; ++i) {
-		tile->bound[i].lb = isl_aff_drop_dims(tile->bound[i].lb,
-					isl_dim_in, depth, tile->depth - depth);
-		if (!tile->bound[i].lb)
-			return -1;
-		if (!tile->bound[i].shift)
-			continue;
-		tile->bound[i].shift = isl_aff_drop_dims(tile->bound[i].shift,
-					isl_dim_in, depth, tile->depth - depth);
-		if (!tile->bound[i].shift)
-			return -1;
-	}
-
-	tile->depth = depth;
-
-	return 0;
-}
-
-/* Determine the number of schedule dimensions that affect the offset of the
- * shared or private tile "tile" and store the result in tile->depth, with
- * a lower bound of data->kernel_depth.
- * Also adjust the fields of the tile to only refer to the tile->depth
- * outer schedule dimensions.
- */
-static isl_stat tile_set_depth(struct gpu_group_data *data,
-	struct gpu_array_tile *tile)
-{
-	if (tile_adjust_depth(tile, compute_tile_depth(data, tile)) < 0)
-		return isl_stat_error;
-
-	return isl_stat_ok;
-}
-
-/* Determine the number of schedule dimensions that affect the offset of the
- * shared tile and store the minimum of the private and shared tile depth
- * in group->min_depth, with a lower bound of data->kernel_depth.
- * If there is no tile defined on the array reference group,
- * then set group->min_depth to data->thread_depth.
- */
-static int set_depth(struct gpu_group_data *data,
-	struct gpu_array_ref_group *group)
-{
-	group->min_depth = data->thread_depth;
-
-	if (group->private_tile) {
-		if (group->private_tile->depth < group->min_depth)
-			group->min_depth = group->private_tile->depth;
-	}
-	if (group->shared_tile) {
-		if (tile_set_depth(data, group->shared_tile) < 0)
-			return -1;
-		if (group->shared_tile->depth < group->min_depth)
-			group->min_depth = group->shared_tile->depth;
-	}
-
-	return 0;
-}
-
-/* Fill up the groups array with singleton groups, i.e., one group
- * per reference, initializing the array, access, write, n_ref and refs fields.
- * In particular the access field is initialized to the scheduled
- * access relation of the array reference.
- *
- * Return the number of elements initialized, i.e., the number of
- * active references in the current kernel.
- */
-static int populate_array_references(struct gpu_local_array_info *local,
-	struct gpu_array_ref_group **groups, struct gpu_group_data *data)
-{
-	int i;
-	int n;
-	isl_ctx *ctx = isl_union_map_get_ctx(data->copy_sched);
-
-	n = 0;
-	for (i = 0; i < local->array->n_ref; ++i) {
-		isl_union_map *umap;
-		isl_map *map;
-		struct gpu_array_ref_group *group;
-		struct gpu_stmt_access *access = local->array->refs[i];
-
-		map = isl_map_copy(access->access);
-		umap = isl_union_map_from_map(map);
-		umap = isl_union_map_apply_domain(umap,
-				isl_union_map_copy(data->copy_sched));
-
-		if (isl_union_map_is_empty(umap)) {
-			isl_union_map_free(umap);
-			continue;
-		}
-
-		map = isl_map_from_union_map(umap);
-		map = isl_map_detect_equalities(map);
-
-		group = isl_calloc_type(ctx, struct gpu_array_ref_group);
-		if (!group)
-			return -1;
-		group->local_array = local;
-		group->array = local->array;
-		group->access = map;
-		group->write = access->write;
-		group->exact_write = access->exact_write;
-		group->slice = access->n_index < local->array->n_index;
-		group->refs = &local->array->refs[i];
-		group->n_ref = 1;
-
-		groups[n++] = group;
-	}
-
-	return n;
-}
-
-/* If group->n_ref == 1, then group->refs was set by
- * populate_array_references to point directly into
- * group->array->refs and should not be freed.
- * If group->n_ref > 1, then group->refs was set by join_groups
- * to point to a newly allocated array.
- */
-struct gpu_array_ref_group *gpu_array_ref_group_free(
-	struct gpu_array_ref_group *group)
-{
-	if (!group)
-		return NULL;
-	gpu_array_tile_free(group->shared_tile);
-	gpu_array_tile_free(group->private_tile);
-	isl_map_free(group->access);
-	if (group->n_ref > 1)
-		free(group->refs);
-	free(group);
-	return NULL;
-}
-
-/* Check if the access relations of group1 and group2 overlap within
- * copy_sched.
- */
-static int accesses_overlap(struct gpu_array_ref_group *group1,
-	struct gpu_array_ref_group *group2)
-{
-	int disjoint;
-
-	disjoint = isl_map_is_disjoint(group1->access, group2->access);
-	if (disjoint < 0)
-		return -1;
-
-	return !disjoint;
-}
-
-/* Combine the given two groups into a single group, containing
- * the references of both groups.
- */
-static struct gpu_array_ref_group *join_groups(
-	struct gpu_array_ref_group *group1,
-	struct gpu_array_ref_group *group2)
-{
-	int i;
-	isl_ctx *ctx;
-	struct gpu_array_ref_group *group;
-
-	if (!group1 || !group2)
-		return NULL;
-
-	ctx = isl_map_get_ctx(group1->access);
-	group = isl_calloc_type(ctx, struct gpu_array_ref_group);
-	if (!group)
-		return NULL;
-	group->local_array = group1->local_array;
-	group->array = group1->array;
-	group->access = isl_map_union(isl_map_copy(group1->access),
-					isl_map_copy(group2->access));
-	group->write = group1->write || group2->write;
-	group->exact_write = group1->exact_write && group2->exact_write;
-	group->slice = group1->slice || group2->slice;
-	group->n_ref = group1->n_ref + group2->n_ref;
-	group->refs = isl_alloc_array(ctx, struct gpu_stmt_access *,
-					group->n_ref);
-	if (!group->refs)
-		return gpu_array_ref_group_free(group);
-	for (i = 0; i < group1->n_ref; ++i)
-		group->refs[i] = group1->refs[i];
-	for (i = 0; i < group2->n_ref; ++i)
-		group->refs[group1->n_ref + i] = group2->refs[i];
-
-	return group;
-}
-
-/* Combine the given two groups into a single group and free
- * the original two groups.
- */
-static struct gpu_array_ref_group *join_groups_and_free(
-	struct gpu_array_ref_group *group1,
-	struct gpu_array_ref_group *group2)
-{
-	struct gpu_array_ref_group *group;
-
-	group = join_groups(group1, group2);
-	gpu_array_ref_group_free(group1);
-	gpu_array_ref_group_free(group2);
-	return group;
-}
-
-/* Report that the array reference group with the given access relation
- * is not mapped to shared memory in the given kernel because
- * it does not exhibit any reuse and is considered to be coalesced.
- */
-static void report_no_reuse_and_coalesced(struct ppcg_kernel *kernel,
-	__isl_keep isl_union_map *access)
-{
-	isl_ctx *ctx;
-	isl_printer *p;
-
-	ctx = isl_union_map_get_ctx(access);
-	p = isl_printer_to_file(ctx, stdout);
-	p = isl_printer_print_str(p, "Array reference group ");
-	p = isl_printer_print_union_map(p, access);
-	p = isl_printer_print_str(p,
-	    " not considered for mapping to shared memory in kernel");
-	p = isl_printer_print_int(p, kernel->id);
-	p = isl_printer_print_str(p,
-	    " because it exhibits no reuse and is considered to be coalesced");
-	p = isl_printer_end_line(p);
-	isl_printer_free(p);
-}
-
-/* Given an access relation in terms of the data->thread_depth initial
- * dimensions of the computed schedule and the thread identifiers
- * (as parameters), check if the use of the corresponding private tile
- * requires unrolling.
- *
- * If we are creating a private tile because we are forced to,
- * then no unrolling is required.
- * Otherwise we check if "access" is bijective and unrolling
- * is required if it is not.  Note that the access relation
- * has already been determined to be bijective before the introduction
- * of the thread identifiers and the removal of the schedule dimensions
- * that are mapped to these threads.  If the access relation is no longer
- * bijective, then this means that more than one value of one of those
- * schedule dimensions is mapped to the same thread and therefore
- * unrolling is required.
- */
-static int check_requires_unroll(struct gpu_group_data *data,
-	__isl_keep isl_map *access, int force_private)
-{
-	int bijective;
-
-	if (force_private)
-		return 0;
-	bijective = access_is_bijective(data, access);
-	if (bijective < 0)
-		return -1;
-	return !bijective;
-}
-
-/* Map the domain of "access" to the outer data->shared_depth
- * schedule dimensions.  When data->shared_depth is equal to
- * data->thread_depth, this result is already available in group->access.
- */
-static __isl_give isl_map *shared_access(struct gpu_array_ref_group *group,
-	__isl_keep isl_union_map *access, struct gpu_group_data *data)
-{
-	isl_union_map *shared;
-
-	if (data->shared_depth == data->thread_depth)
-		return isl_map_copy(group->access);
-
-	shared = isl_union_map_copy(access);
-	shared = isl_union_map_apply_domain(shared,
-			isl_union_map_copy(data->shared_sched));
-	return isl_map_from_union_map(shared);
-}
-
-/* Compute the private and/or shared memory tiles for the array
- * reference group "group" of array "array".
- * Return 0 on success and -1 on error.
- *
- * If the array is a read-only scalar or if the user requested
- * not to use shared or private memory, then we do not need to do anything.
- *
- * If any reference in the reference group accesses more than one element,
- * then we would have to make sure that the layout in shared memory
- * is the same as that in global memory.  Since we do not handle this yet
- * (and it may not even be possible), we refuse to map to private or
- * shared memory in such cases.
- *
- * If the array group involves any may writes (that are not must writes),
- * then we would have to make sure that we load the data into shared/private
- * memory first in case the data is not written by the kernel
- * (but still written back out to global memory).
- * Since we don't have any such mechanism at the moment, we don't
- * compute shared/private tiles for groups involving may writes.
- *
- * We only try to compute a shared memory tile if there is any reuse
- * or if the access is not coalesced.
- * Reuse and coalescing are checked within the given kernel.
- *
- * For computing a private memory tile, we also require that there is
- * some reuse.  Moreover, we require that the access is private
- * to the thread.  That is, we check that any given array element
- * is only accessed by a single thread.
- * We compute an access relation that maps the outer
- * data->thread_depth + data->n_thread schedule dimensions.
- * The latter data->n_thread will be mapped to thread identifiers.
- * We actually check that those iterators that will be wrapped
- * partition the array space.  This check is stricter than necessary
- * since several iterations may be mapped onto the same thread
- * and then they could be allowed to access the same memory elements,
- * but our check does not allow this situation.
- *
- * For private memory tiles, the number of schedule dimensions that
- * affect the offset is computed and stored in tile->depth, with
- * a lower bound of data->kernel_depth.  If this depth is smaller
- * than the minimal depth that still ensures that every element
- * is accessed by a single thread, then the depth is raised
- * to this minimal depth.
- * The fields of the tile are then adjusted to only refer to the tile->depth
- * outer schedule dimensions.
- *
- * We also check that the index expression only depends on parallel
- * loops.  That way, we can move those loops innermost and unroll them.
- * Again, we use a test that is stricter than necessary.
- * We actually check whether the index expression only depends
- * on the iterators that are wrapped over the threads.
- * These are necessarily parallel, but there may be more parallel loops.
- *
- * Combining the injectivity of the first test with the single-valuedness
- * of the second test, we simply test for bijectivity.
- *
- * If the use of the private tile requires unrolling, but some
- * of the other arrays are forcibly mapped to private memory,
- * then we do not allow the use of this private tile since
- * we cannot move the schedule dimensions that need to be unrolled down
- * without performing some kind of expansion on those arrays
- * that are forcibly mapped to private memory.
- *
- * If the array is marked force_private, then we bypass all checks
- * and assume we can (and should) use registers only.
- *
- * If it turns out we can (or have to) use registers, we compute
- * the private memory tile size using can_tile, after introducing a dependence
- * on the thread indices.
- */
-static int compute_group_bounds_core(struct ppcg_kernel *kernel,
-	struct gpu_array_ref_group *group, struct gpu_group_data *data)
-{
-	isl_ctx *ctx = isl_space_get_ctx(group->array->space);
-	isl_union_map *access, *local;
-	int n_index = group->array->n_index;
-	int no_reuse, coalesced;
-	isl_map *acc;
-	int force_private = group->local_array->force_private;
-	int use_shared = !force_private && kernel->options->use_shared_memory &&
-				data->n_thread > 0;
-	int use_private = force_private || kernel->options->use_private_memory;
-	int r = 0;
-	int requires_unroll;
-	int unique_depth;
-
-	if (!use_shared && !use_private)
-		return 0;
-	if (gpu_array_is_read_only_scalar(group->array))
-		return 0;
-	if (!force_private && !group->exact_write)
-		return 0;
-	if (group->slice)
-		return 0;
-
-	access = gpu_array_ref_group_access_relation(group, 1, 1);
-	local = localize_access(data, isl_union_map_copy(access));
-	no_reuse = isl_union_map_is_injective(local);
-	if (no_reuse < 0)
-		r = -1;
-	if (use_shared && no_reuse)
-		coalesced = access_is_coalesced(data, local);
-	isl_union_map_free(local);
-
-	if (r >= 0 && kernel->options->debug->verbose &&
-	    use_shared && no_reuse && coalesced)
-		report_no_reuse_and_coalesced(kernel, access);
-
-	if (use_shared && (!no_reuse || !coalesced)) {
-		group->shared_tile = gpu_array_tile_create(ctx,
-							group->array->n_index);
-		acc = shared_access(group, access, data);
-		if (!group->shared_tile)
-			r = -1;
-		else if (!can_tile(acc, group->shared_tile))
-			group->shared_tile =
-					gpu_array_tile_free(group->shared_tile);
-		isl_map_free(acc);
-	}
-
-	if (r < 0 || (!force_private && (!use_private || no_reuse))) {
-		isl_union_map_free(access);
-		return r;
-	}
-
-	access = isl_union_map_apply_domain(access,
-					isl_union_map_copy(data->thread_sched));
-
-	acc = isl_map_from_union_map(access);
-
-	if (!force_private && !access_is_bijective(data, acc)) {
-		isl_map_free(acc);
-		return 0;
-	}
-
-	unique_depth = compute_accessed_by_single_thread_depth(data, acc);
-
-	acc = isl_map_intersect_domain(acc, isl_set_copy(data->privatization));
-	acc = isl_map_project_out(acc, isl_dim_in, data->thread_depth,
-								data->n_thread);
-	requires_unroll = check_requires_unroll(data, acc, force_private);
-	if (unique_depth < 0 || requires_unroll < 0 ||
-	    (requires_unroll && kernel->any_force_private)) {
-		isl_map_free(acc);
-		return requires_unroll < 0 ? -1 : 0;
-	}
-
-	group->private_tile = gpu_array_tile_create(ctx, n_index);
-	if (!group->private_tile) {
-		isl_map_free(acc);
-		return -1;
-	}
-	group->private_tile->requires_unroll = requires_unroll;
-	if (!can_tile(acc, group->private_tile))
-		group->private_tile = gpu_array_tile_free(group->private_tile);
-
-	isl_map_free(acc);
-
-	if (group->private_tile) {
-		struct gpu_array_tile *tile = group->private_tile;
-		int tile_depth = compute_tile_depth(data, tile);
-		if (tile_depth < unique_depth)
-			tile_depth = unique_depth;
-		if (tile_adjust_depth(tile, tile_depth) < 0)
-			return -1;
-	}
-
-	if (force_private && !group->private_tile)
-		isl_die(ctx, isl_error_internal,
-			"unable to map array reference group to registers",
-			return -1);
-
-	return 0;
-}
-
-/* Compute the private and/or shared memory tiles for the array
- * reference group "group" of array "array" and set the tile depth.
- * Return 0 on success and -1 on error.
- */
-static int compute_group_bounds(struct ppcg_kernel *kernel,
-	struct gpu_array_ref_group *group, struct gpu_group_data *data)
-{
-	if (!group)
-		return -1;
-	if (compute_group_bounds_core(kernel, group, data) < 0)
-		return -1;
-	if (set_depth(data, group) < 0)
-		return -1;
-
-	return 0;
-}
-
-/* If two groups have overlapping access relations (as determined by
- * the "overlap" function) and if one of them involves a write,
- * then merge the two groups into one.
- * If "compute_bounds" is set, then call compute_group_bounds
- * on the merged groups.
- *
- * Return the updated number of groups.
- * Return -1 on error.
- */
-static int group_writes(struct ppcg_kernel *kernel,
-	int n, struct gpu_array_ref_group **groups,
-	int (*overlap)(struct gpu_array_ref_group *group1,
-		struct gpu_array_ref_group *group2), int compute_bounds,
-	struct gpu_group_data *data)
-{
-	int i, j;
-
-	for (i = 0; i < n; ++i) {
-		for (j = n - 1; j > i; --j) {
-			if (!groups[i]->write && !groups[j]->write)
-				continue;
-
-			if (!overlap(groups[i], groups[j]))
-				continue;
-
-			groups[i] = join_groups_and_free(groups[i], groups[j]);
-			if (j != n - 1)
-				groups[j] = groups[n - 1];
-			groups[n - 1] = NULL;
-			n--;
-
-			if (!groups[i])
-				return -1;
-			if (compute_bounds &&
-			    compute_group_bounds(kernel, groups[i], data) < 0)
-				return -1;
-		}
-	}
-
-	return n;
-}
-
-/* If two groups have overlapping access relations (within the innermost
- * loop) and if one of them involves a write, then merge the two groups
- * into one.
- *
- * Return the updated number of groups.
- */
-static int group_overlapping_writes(struct ppcg_kernel *kernel,
-	int n, struct gpu_array_ref_group **groups,
-	struct gpu_group_data *data)
-{
-	return group_writes(kernel, n, groups, &accesses_overlap, 0, data);
-}
-
-/* Check if the access relations of group1 and group2 overlap within
- * the outermost min(group1->min_depth, group2->min_depth) loops.
- */
-static int depth_accesses_overlap(struct gpu_array_ref_group *group1,
-	struct gpu_array_ref_group *group2)
-{
-	int depth;
-	int dim;
-	int empty;
-	isl_map *map_i, *map_j, *map;
-
-	depth = group1->min_depth;
-	if (group2->min_depth < depth)
-		depth = group2->min_depth;
-	map_i = isl_map_copy(group1->access);
-	dim = isl_map_dim(map_i, isl_dim_in);
-	map_i = isl_map_eliminate(map_i, isl_dim_in, depth, dim - depth);
-	map_j = isl_map_copy(group2->access);
-	map_j = isl_map_eliminate(map_j, isl_dim_in, depth, dim - depth);
-	map = isl_map_intersect(map_i, map_j);
-	empty = isl_map_is_empty(map);
-	isl_map_free(map);
-
-	return !empty;
-}
-
-/* If two groups have overlapping access relations (within the outer
- * depth loops) and if one of them involves a write,
- * then merge the two groups into one.
- *
- * Return the updated number of groups.
- */
-static int group_depth_overlapping_writes(struct ppcg_kernel *kernel,
-	int n, struct gpu_array_ref_group **groups, struct gpu_group_data *data)
-{
-	return group_writes(kernel, n, groups, &depth_accesses_overlap, 1,
-				data);
-}
-
-/* Is the size of the tile specified by "tile" smaller than the sum of
- * the sizes of the tiles specified by "tile1" and "tile2"?
- */
-static int smaller_tile(struct gpu_array_tile *tile,
-	struct gpu_array_tile *tile1, struct gpu_array_tile *tile2)
-{
-	int smaller;
-	isl_val *size, *size1, *size2;
-
-	size = gpu_array_tile_size(tile);
-	size1 = gpu_array_tile_size(tile1);
-	size2 = gpu_array_tile_size(tile2);
-
-	size = isl_val_sub(size, size1);
-	size = isl_val_sub(size, size2);
-	smaller = isl_val_is_neg(size);
-
-	isl_val_free(size);
-
-	return smaller;
-}
-
-/* Given an initial grouping of array references and shared memory tiles
- * for each group that allows for a shared memory tile, merge two groups
- * if both have a shared memory tile, the merged group also has
- * a shared memory tile and the size of the tile for the merge group
- * is smaller than the sum of the tile sizes of the individual groups.
- *
- * If merging two groups decreases the depth of the tile of
- * one or both of the two groups, then we need to check for overlapping
- * writes again.
- *
- * Return the number of groups after merging.
- * Return -1 on error.
- */
-static int group_common_shared_memory_tile(struct ppcg_kernel *kernel,
-	struct gpu_array_info *array, int n,
-	struct gpu_array_ref_group **groups, struct gpu_group_data *data)
-{
-	int i, j;
-	int recompute_overlap = 0;
-
-	for (i = 0; i < n; ++i) {
-		if (!groups[i]->shared_tile)
-			continue;
-		for (j = n - 1; j > i; --j) {
-			struct gpu_array_ref_group *group;
-
-			if (!groups[j]->shared_tile)
-				continue;
-
-			if (!depth_accesses_overlap(groups[i], groups[j]))
-				continue;
-
-			group = join_groups(groups[i], groups[j]);
-			if (compute_group_bounds(kernel, group, data) < 0) {
-				gpu_array_ref_group_free(group);
-				return -1;
-			}
-			if (!group->shared_tile ||
-			    !smaller_tile(group->shared_tile,
-					groups[i]->shared_tile,
-					groups[j]->shared_tile)) {
-				gpu_array_ref_group_free(group);
-				continue;
-			}
-
-			if (group->min_depth < groups[i]->min_depth ||
-			    group->min_depth < groups[j]->min_depth)
-				recompute_overlap = 1;
-			gpu_array_ref_group_free(groups[i]);
-			gpu_array_ref_group_free(groups[j]);
-			groups[i] = group;
-			if (j != n - 1)
-				groups[j] = groups[n - 1];
-			n--;
-		}
-	}
-
-	if (recompute_overlap)
-		n = group_depth_overlapping_writes(kernel, n, groups, data);
-	return n;
-}
-
-/* Set array->n_group and array->groups to n and groups.
- *
- * Additionally, set the "nr" field of each group.
- */
-static void set_array_groups(struct gpu_local_array_info *array,
-	int n, struct gpu_array_ref_group **groups)
-{
-	int i;
-
-	array->n_group = n;
-	array->groups = groups;
-
-	for (i = 0; i < n; ++i)
-		groups[i]->nr = i;
-}
-
-/* Combine all groups in "groups" into a single group and return
- * the new number of groups (1 or 0 if there were no groups to start with).
- */
-static int join_all_groups(int n, struct gpu_array_ref_group **groups)
-{
-	int i;
-
-	for (i = n - 1; i > 0; --i) {
-		groups[0] = join_groups_and_free(groups[0], groups[i]);
-		groups[i] = NULL;
-		n--;
-	}
-
-	return n;
-}
-
-/* Group array references that should be considered together when
- * deciding whether to access them from private, shared or global memory.
- * Return -1 on error.
- *
- * In particular, if two array references overlap and if one of them
- * is a write, then the two references are grouped together.
- * We first perform an initial grouping based only on the access relation.
- * After computing shared and private memory tiles, we check for
- * overlapping writes again, but this time taking into account
- * the depth of the effective tile.
- *
- * Furthermore, if two groups admit a shared memory tile and if the
- * combination of the two also admits a shared memory tile, we merge
- * the two groups.
- *
- * If the array contains structures, then we compute a single
- * reference group without trying to find any tiles
- * since we do not map such arrays to private or shared
- * memory.  The only exception is when those arrays of structures
- * are required to be mapped to private memory.
- */
-static int group_array_references(struct ppcg_kernel *kernel,
-	struct gpu_local_array_info *local, struct gpu_group_data *data)
-{
-	int i;
-	int n;
-	isl_ctx *ctx = isl_union_map_get_ctx(data->shared_sched);
-	struct gpu_array_ref_group **groups;
-
-	groups = isl_calloc_array(ctx, struct gpu_array_ref_group *,
-					local->array->n_ref);
-	if (!groups)
-		return -1;
-
-	n = populate_array_references(local, groups, data);
-
-	if (local->array->has_compound_element && !local->force_private) {
-		n = join_all_groups(n, groups);
-		set_array_groups(local, n, groups);
-		return 0;
-	}
-
-	n = group_overlapping_writes(kernel, n, groups, data);
-
-	for (i = 0; i < n; ++i)
-		if (compute_group_bounds(kernel, groups[i], data) < 0)
-			n = -1;
-
-	n = group_depth_overlapping_writes(kernel, n, groups, data);
-
-	n = group_common_shared_memory_tile(kernel, local->array,
-					    n, groups, data);
-
-	set_array_groups(local, n, groups);
-
-	if (n >= 0)
-		return 0;
-
-	for (i = 0; i < local->array->n_ref; ++i)
-		gpu_array_ref_group_free(groups[i]);
-	return -1;
-}
-
-/* For each array in the input program that can be mapped to private memory,
- * check if there are any order dependences active inside the current kernel,
- * within the same iteration of the host schedule, i.e., the prefix
- * schedule at "node".
- * If so, mark the array as force_private so that its reference groups will be
- * mapped to a registers.
- *
- * Note that the arrays that cannot be mapped to private memory have
- * had their order dependences added to prog->array_order and
- * subsequently to the coincidence constraints.
- */
-static void check_can_be_private_live_ranges(struct ppcg_kernel *kernel,
-	__isl_keep isl_schedule_node *node)
-{
-	int i;
-	isl_union_set *domain;
-	isl_multi_union_pw_aff *prefix;
-	isl_union_pw_multi_aff *contraction;
-
-	if (!kernel->options->live_range_reordering)
-		return;
-
-	kernel->any_force_private = 0;
-
-	prefix = isl_schedule_node_get_prefix_schedule_multi_union_pw_aff(node);
-	contraction = isl_union_pw_multi_aff_copy(kernel->contraction);
-	prefix = isl_multi_union_pw_aff_pullback_union_pw_multi_aff(prefix,
-								contraction);
-	domain = isl_union_set_copy(kernel->expanded_domain);
-	domain = isl_union_set_universe(domain);
-
-	for (i = 0; i < kernel->n_array; ++i) {
-		struct gpu_local_array_info *local = &kernel->array[i];
-		isl_union_map *order;
-
-		local->force_private = 0;
-		if (!gpu_array_can_be_private(local->array))
-			continue;
-		order = isl_union_map_copy(local->array->dep_order);
-		order = isl_union_map_intersect_domain(order,
-						    isl_union_set_copy(domain));
-		order = isl_union_map_intersect_range(order,
-						    isl_union_set_copy(domain));
-		order = isl_union_map_eq_at_multi_union_pw_aff(order,
-					isl_multi_union_pw_aff_copy(prefix));
-		if (!isl_union_map_is_empty(order)) {
-			local->force_private = 1;
-			kernel->any_force_private = 1;
-		}
-		isl_union_map_free(order);
-	}
-
-	isl_multi_union_pw_aff_free(prefix);
-	isl_union_set_free(domain);
-}
-
-/* Expand the domain of the schedule "s" by plugging in
- * the contraction "contraction" and return the result.
- */
-static __isl_give isl_union_map *expand(__isl_take isl_union_map *s,
-	__isl_keep isl_union_pw_multi_aff *contraction)
-{
-	contraction = isl_union_pw_multi_aff_copy(contraction);
-	s = isl_union_map_preimage_domain_union_pw_multi_aff(s, contraction);
-	return s;
-}
-
-/* Create a set of dimension data->thread_depth + data->n_thread
- * that equates the residue of the final data->n_thread dimensions
- * modulo the kernel->block_dim sizes to the thread identifiers.
- * Store the computed set in data->privatization.
- *
- * The construction starts with the space of kernel->thread_filter,
- * which is known to reference all thread identifiers.
- */
-static void compute_privatization(struct gpu_group_data *data,
-	struct ppcg_kernel *kernel)
-{
-	int i;
-	isl_ctx *ctx;
-	isl_space *space;
-	isl_local_space *ls;
-	isl_set *set;
-
-	ctx = isl_union_map_get_ctx(data->shared_sched);
-	space = isl_union_set_get_space(kernel->thread_filter);
-	space = isl_space_set_from_params(space);
-	space = isl_space_add_dims(space, isl_dim_set,
-				    data->thread_depth + data->n_thread);
-	set = isl_set_universe(space);
-	space = isl_set_get_space(set);
-	ls = isl_local_space_from_space(space);
-
-	for (i = 0; i < data->n_thread; ++i) {
-		isl_aff *aff, *aff2;
-		isl_constraint *c;
-		isl_val *v;
-		isl_id *id;
-		int pos;
-
-		aff = isl_aff_var_on_domain(isl_local_space_copy(ls),
-					isl_dim_set, data->thread_depth + i);
-		v = isl_val_int_from_si(ctx, kernel->block_dim[i]);
-		aff = isl_aff_mod_val(aff, v);
-		id = isl_id_list_get_id(kernel->thread_ids, i);
-		pos = isl_set_find_dim_by_id(set, isl_dim_param, id);
-		isl_id_free(id);
-		aff2 = isl_aff_var_on_domain(isl_local_space_copy(ls),
-					isl_dim_param, pos);
-		aff = isl_aff_sub(aff, aff2);
-		c = isl_equality_from_aff(aff);
-		set = isl_set_add_constraint(set, c);
-	}
-
-	isl_local_space_free(ls);
-	data->privatization = set;
-}
-
-/* Return the prefix schedule at "node" as a relation
- * between domain elements and schedule dimensions after detecting
- * equalities in this relation.
- */
-static __isl_give isl_union_map *prefix_with_equalities(
-	__isl_keep isl_schedule_node *node)
-{
-	isl_union_map *schedule;
-
-	schedule = isl_schedule_node_get_prefix_schedule_relation(node);
-	schedule = isl_union_map_detect_equalities(schedule);
-
-	return schedule;
-}
-
-/* Group references of all arrays in "kernel".
- * "node" points to the kernel mark.
- * The mapping to shared memory in computed at the "shared" mark.
- *
- * We first extract all required schedule information into
- * a gpu_group_data structure and then consider each array
- * in turn.
- */
-int gpu_group_references(struct ppcg_kernel *kernel,
-	__isl_keep isl_schedule_node *node)
-{
-	int i;
-	int r = 0;
-	isl_union_pw_multi_aff *contraction;
-	struct gpu_group_data data;
-
-	check_can_be_private_live_ranges(kernel, node);
-
-	data.scop = kernel->prog->scop;
-
-	data.kernel_depth = isl_schedule_node_get_schedule_depth(node);
-	data.host_sched = isl_schedule_node_get_prefix_schedule_relation(node);
-
-	node = isl_schedule_node_copy(node);
-	node = gpu_tree_move_down_to_shared(node, kernel->core);
-	data.shared_depth = isl_schedule_node_get_schedule_depth(node);
-	data.shared_sched = prefix_with_equalities(node);
-
-	node = gpu_tree_move_down_to_thread(node, kernel->core);
-	node = isl_schedule_node_child(node, 0);
-	data.thread_depth = isl_schedule_node_get_schedule_depth(node);
-	data.n_thread = isl_schedule_node_band_n_member(node);
-	if (data.thread_depth == data.shared_depth)
-		data.copy_sched = isl_union_map_copy(data.shared_sched);
-	else
-		data.copy_sched = prefix_with_equalities(node);
-	data.thread_sched = isl_union_map_copy(data.copy_sched);
-	data.thread_sched = isl_union_map_flat_range_product(data.thread_sched,
-		isl_schedule_node_band_get_partial_schedule_union_map(node));
-	data.thread_sched = isl_union_map_detect_equalities(data.thread_sched);
-
-	contraction = isl_union_pw_multi_aff_copy(kernel->contraction);
-	data.host_sched = expand(data.host_sched, contraction);
-	data.shared_sched = expand(data.shared_sched, contraction);
-	if (data.thread_depth == data.shared_depth) {
-		isl_union_map_free(data.copy_sched);
-		data.copy_sched = isl_union_map_copy(data.shared_sched);
-	} else {
-		data.copy_sched = expand(data.copy_sched, contraction);
-	}
-	data.thread_sched = expand(data.thread_sched, contraction);
-	isl_union_pw_multi_aff_free(contraction);
-
-	node = isl_schedule_node_child(node, 0);
-	data.full_sched = isl_union_map_copy(data.thread_sched);
-	data.full_sched = isl_union_map_flat_range_product(data.full_sched,
-		isl_schedule_node_get_subtree_schedule_union_map(node));
-	isl_schedule_node_free(node);
-
-	compute_privatization(&data, kernel);
-
-	for (i = 0; i < kernel->n_array; ++i) {
-		r = group_array_references(kernel, &kernel->array[i], &data);
-		if (r < 0)
-			break;
-	}
-
-	isl_union_map_free(data.host_sched);
-	isl_union_map_free(data.shared_sched);
-	isl_union_map_free(data.copy_sched);
-	isl_union_map_free(data.thread_sched);
-	isl_union_map_free(data.full_sched);
-	isl_set_free(data.privatization);
-
-	return r;
-}
-
-/* Given a description of an array tile "tile" and the "space"
- *
- *	{ D -> A }
- *
- * where D represents the first tile->depth schedule dimensions
- * and A represents the array, construct an isl_multi_aff
- *
- *	{ [D[i] -> A[a]] -> A'[a'] }
- *
- * with A' a scaled down copy of A according to the shifts and strides
- * in "tile".  In particular,
- *
- *	a' = (a + shift(i))/stride
- *
- * "insert_array" represents
- *
- *	{ [D -> A] -> D }
- *
- * and is used to insert A into the domain of functions that only
- * reference D.
- */
-static __isl_give isl_multi_aff *strided_tile(
-	struct gpu_array_tile *tile, __isl_keep isl_space *space,
-	__isl_keep isl_multi_aff *insert_array)
-{
-	int i;
-	isl_ctx *ctx;
-	isl_multi_aff *shift;
-	isl_multi_val *stride;
-	isl_space *space2;
-	isl_local_space *ls;
-	isl_multi_aff *tiling;
-
-	ctx = isl_space_get_ctx(space);
-	space2 = isl_space_domain(isl_space_copy(space));
-	ls = isl_local_space_from_space(space2);
-	space2 = isl_space_range(isl_space_copy(space));
-	stride = isl_multi_val_zero(space2);
-	shift = isl_multi_aff_zero(isl_space_copy(space));
-
-	for (i = 0; i < tile->n; ++i) {
-		struct gpu_array_bound *bound = &tile->bound[i];
-		isl_val *stride_i;
-		isl_aff *shift_i;
-
-		if (tile->bound[i].shift) {
-			stride_i = isl_val_copy(bound->stride);
-			shift_i = isl_aff_copy(bound->shift);
-		} else {
-			stride_i = isl_val_one(ctx);
-			shift_i = isl_aff_zero_on_domain(
-					isl_local_space_copy(ls));
-		}
-
-		stride = isl_multi_val_set_val(stride, i, stride_i);
-		shift = isl_multi_aff_set_aff(shift, i, shift_i);
-	}
-	isl_local_space_free(ls);
-
-	shift = isl_multi_aff_pullback_multi_aff(shift,
-				    isl_multi_aff_copy(insert_array));
-
-	tiling = isl_multi_aff_range_map(isl_space_copy(space));
-	tiling = isl_multi_aff_add(tiling, shift);
-	tiling = isl_multi_aff_scale_down_multi_val(tiling, stride);
-
-	return tiling;
-}
-
-/* Compute a tiling for the array reference group "group".
- *
- * The tiling is of the form
- *
- *	{ [D[i] -> A[a]] -> T[t] }
- *
- * where D represents the first tile->depth schedule dimensions,
- * A represents the global array and T represents the shared or
- * private memory tile.  The name of T is the name of the local
- * array.
- *
- * If there is any stride in the accesses, then the mapping is
- *
- *	t = (a + shift(i))/stride - lb(i)
- *
- * otherwise, it is simply
- *
- *	t = a - lb(i)
- */
-void gpu_array_ref_group_compute_tiling(struct gpu_array_ref_group *group)
-{
-	int i;
-	struct gpu_array_tile *tile;
-	isl_space *space;
-	isl_multi_aff *tiling, *lb, *insert_array;
-	isl_printer *p;
-	char *local_name;
-
-	tile = gpu_array_ref_group_tile(group);
-	if (!tile)
-		return;
-
-	space = isl_map_get_space(group->access);
-	space = isl_space_from_range(isl_space_range(space));
-	space = isl_space_add_dims(space, isl_dim_in, tile->depth);
-	insert_array = isl_multi_aff_domain_map(isl_space_copy(space));
-
-	for (i = 0; i < tile->n; ++i)
-		if (tile->bound[i].shift)
-			break;
-
-	if (i < tile->n)
-		tiling = strided_tile(tile, space, insert_array);
-	else
-		tiling = isl_multi_aff_range_map(isl_space_copy(space));
-
-	lb = isl_multi_aff_zero(space);
-	for (i = 0; i < tile->n; ++i) {
-		isl_aff *lb_i = isl_aff_copy(tile->bound[i].lb);
-		lb = isl_multi_aff_set_aff(lb, i, lb_i);
-	}
-	lb = isl_multi_aff_pullback_multi_aff(lb, insert_array);
-
-	tiling = isl_multi_aff_sub(tiling, lb);
-
-	p = isl_printer_to_str(isl_multi_aff_get_ctx(tiling));
-	p = gpu_array_ref_group_print_name(group, p);
-	local_name = isl_printer_get_str(p);
-	isl_printer_free(p);
-	tiling = isl_multi_aff_set_tuple_name(tiling, isl_dim_out, local_name);
-	free(local_name);
-
-	tile->tiling = tiling;
-}
diff --git a/polly/lib/External/ppcg/gpu_group.h b/polly/lib/External/ppcg/gpu_group.h
deleted file mode 100644
index c94812d39e263..0000000000000
--- a/polly/lib/External/ppcg/gpu_group.h
+++ /dev/null
@@ -1,65 +0,0 @@
-#ifndef GPU_GROUP_H
-#define GPU_GROUP_H
-
-#include <isl/schedule_node.h>
-#include "gpu.h"
-
-/* A group of array references in a kernel that should be handled together.
- * If private_tile is not NULL, then it is mapped to registers.
- * Otherwise, if shared_tile is not NULL, it is mapped to shared memory.
- * Otherwise, it is accessed from global memory.
- * Note that if both private_tile and shared_tile are set, then shared_tile
- * is only used inside group_common_shared_memory_tile.
- */
-struct gpu_array_ref_group {
-	/* The references in this group access this local array. */
-	struct gpu_local_array_info *local_array;
-	/* This is the corresponding array. */
-	struct gpu_array_info *array;
-	/* Position of this group in the list of reference groups of array. */
-	int nr;
-
-	/* The following fields are use during the construction of the groups.
-	 * access is the combined access relation relative to the private
-	 * memory tiling.  In particular, the domain of the map corresponds
-	 * to the first thread_depth dimensions of the kernel schedule.
-	 * write is set if any access in the group is a write.
-	 * exact_write is set if all writes are definite writes.
-	 * slice is set if there is at least one access in the group
-	 * that refers to more than one element
-	 * "min_depth" is the minimum of the tile depths and thread_depth.
-	 */
-	isl_map *access;
-	int write;
-	int exact_write;
-	int slice;
-	int min_depth;
-
-	/* The shared memory tile, NULL if none. */
-	struct gpu_array_tile *shared_tile;
-
-	/* The private memory tile, NULL if none. */
-	struct gpu_array_tile *private_tile;
-
-	/* References in this group; point to elements of a linked list. */
-	int n_ref;
-	struct gpu_stmt_access **refs;
-};
-
-int gpu_group_references(struct ppcg_kernel *kernel,
-	__isl_keep isl_schedule_node *node);
-
-__isl_give isl_printer *gpu_array_ref_group_print_name(
-	struct gpu_array_ref_group *group, __isl_take isl_printer *p);
-void gpu_array_ref_group_compute_tiling(struct gpu_array_ref_group *group);
-__isl_give isl_union_map *gpu_array_ref_group_access_relation(
-	struct gpu_array_ref_group *group, int read, int write);
-int gpu_array_ref_group_requires_unroll(struct gpu_array_ref_group *group);
-enum ppcg_group_access_type gpu_array_ref_group_type(
-	struct gpu_array_ref_group *group);
-struct gpu_array_tile *gpu_array_ref_group_tile(
-	struct gpu_array_ref_group *group);
-struct gpu_array_ref_group *gpu_array_ref_group_free(
-	struct gpu_array_ref_group *group);
-
-#endif
diff --git a/polly/lib/External/ppcg/gpu_hybrid.c b/polly/lib/External/ppcg/gpu_hybrid.c
deleted file mode 100644
index f1b8af3e50c4d..0000000000000
--- a/polly/lib/External/ppcg/gpu_hybrid.c
+++ /dev/null
@@ -1,146 +0,0 @@
-/*
- * Copyright 2013      Ecole Normale Superieure
- * Copyright 2015      Sven Verdoolaege
- *
- * Use of this software is governed by the MIT license
- *
- * Written by Sven Verdoolaege,
- * Ecole Normale Superieure, 45 rue d'Ulm, 75230 Paris, France
- */
-
-#include <string.h>
-
-#include <isl/val.h>
-#include <isl/space.h>
-#include <isl/union_set.h>
-#include <isl/schedule_node.h>
-
-#include "hybrid.h"
-#include "gpu_hybrid.h"
-#include "gpu_tree.h"
-#include "schedule.h"
-#include "util.h"
-
-/* Have all domain elements been filtered out before reaching
- * the "node" position in the schedule tree?
- */
-static isl_bool has_empty_domain(__isl_keep isl_schedule_node *node)
-{
-	isl_union_set *domain;
-	isl_bool empty;
-
-	domain = isl_schedule_node_get_domain(node);
-	empty = isl_union_set_is_empty(domain);
-	isl_union_set_free(domain);
-
-	return empty;
-}
-
-/* Given a pointer to a phase in the result of hybrid tiling,
- * map the phase to the device, provided the phase is non-empty.
- * Empty phases can occur if the input schedule domain can be
- * covered by a small number of hexagons that all belong to the same phase.
- *
- * The input has the following form:
- *
- *	M - CT - P - C - ...
- *
- * with M the phase marker, CT the space tiling, P the original
- * parent band and C the original child band.
- * The (outer dimensions of the) C band need to be mapped to threads.
- * The (outer dimension of the) CT band needs to be mapped to blocks.
- * The mapping to shared memory needs to be computed between the CT and
- * the P band.
- *
- * The C band is first shifted to start at zero.
- * Then the appropriate markers are introduced and a kernel is
- * created for the tree rooted at CT.
- * If the "unroll_gpu_tile" option is set, then the AST generator
- * is instructed to unroll the P and C bands.
- */
-static __isl_give isl_schedule_node *update_phase(
-	__isl_take isl_schedule_node *node, void *user)
-{
-	struct gpu_gen *gen = user;
-	int depth0, depth;
-	isl_ctx *ctx;
-	isl_id *id;
-	isl_bool empty_domain;
-	ppcg_ht_phase *phase;
-
-	empty_domain = has_empty_domain(node);
-	if (empty_domain < 0)
-		return isl_schedule_node_free(node);
-	if (empty_domain)
-		return node;
-
-	if (!node)
-		return NULL;
-	ctx = isl_schedule_node_get_ctx(node);
-
-	phase = ppcg_ht_phase_extract_from_mark(node);
-
-	depth0 = isl_schedule_node_get_tree_depth(node);
-
-	node = isl_schedule_node_child(node, 0);
-
-	node = isl_schedule_node_child(node, 0);
-	node = isl_schedule_node_child(node, 0);
-	node = ppcg_ht_phase_shift_space_point(phase, node);
-	if (gen->options->unroll_gpu_tile)
-		node = ppcg_set_schedule_node_type(node, isl_ast_loop_unroll);
-	id = isl_id_alloc(ctx, "thread", NULL);
-	node = isl_schedule_node_insert_mark(node, id);
-	node = isl_schedule_node_parent(node);
-	if (gen->options->unroll_gpu_tile)
-		node = ppcg_set_schedule_node_type(node, isl_ast_loop_unroll);
-	id = isl_id_alloc(ctx, "shared", NULL);
-	node = isl_schedule_node_insert_mark(node, id);
-	node = isl_schedule_node_parent(node);
-
-	node = gpu_create_kernel(gen, node, 0, NULL);
-
-	depth = isl_schedule_node_get_tree_depth(node);
-	node = isl_schedule_node_ancestor(node, depth - depth0);
-
-	return node;
-}
-
-/* Apply hybrid tiling on "node" and its parent based on the (valid)
- * bounds on the relative dependence distances "bounds" and
- * the tile sizes in "tile_sizes".
- * The number of elements in "tile_sizes" is at least as large
- * as the sum of the dimensions of the parent and the child node.
- *
- * Convert the tile_sizes to an isl_multi_val in the right space,
- * insert the hybrid tiling and then create a kernel inside each phase.
- * Finally, remove the phase marks.
- */
-__isl_give isl_schedule_node *gpu_hybrid_tile(struct gpu_gen *gen,
-	__isl_take isl_schedule_node *node, __isl_take ppcg_ht_bounds *bounds,
-	int *tile_sizes)
-{
-	isl_multi_val *mv;
-	isl_space *space, *space2;
-
-	if (!node || !bounds)
-		goto error;
-
-	space2 = isl_schedule_node_band_get_space(node);
-	node = isl_schedule_node_parent(node);
-	space = isl_schedule_node_band_get_space(node);
-	space = isl_space_product(space, space2);
-	mv = ppcg_multi_val_from_int_list(space, tile_sizes);
-
-	node = ppcg_ht_bounds_insert_tiling(bounds, mv, node, gen->options);
-
-	node = hybrid_tile_foreach_phase(node, &update_phase, gen);
-
-	node = hybrid_tile_drop_phase_marks(node);
-
-	return node;
-error:
-	isl_schedule_node_free(node);
-	ppcg_ht_bounds_free(bounds);
-	return NULL;
-}
diff --git a/polly/lib/External/ppcg/gpu_hybrid.h b/polly/lib/External/ppcg/gpu_hybrid.h
deleted file mode 100644
index 0348261586b3e..0000000000000
--- a/polly/lib/External/ppcg/gpu_hybrid.h
+++ /dev/null
@@ -1,13 +0,0 @@
-#ifndef GPU_HYBRID_H
-#define GPU_HYBRID_H
-
-#include <isl/schedule_node.h>
-
-#include "gpu.h"
-#include "hybrid.h"
-
-__isl_give isl_schedule_node *gpu_hybrid_tile(struct gpu_gen *gen,
-	__isl_take isl_schedule_node *node, __isl_take ppcg_ht_bounds *bounds,
-	int *tile_sizes);
-
-#endif
diff --git a/polly/lib/External/ppcg/gpu_print.c b/polly/lib/External/ppcg/gpu_print.c
deleted file mode 100644
index a1ae96830c628..0000000000000
--- a/polly/lib/External/ppcg/gpu_print.c
+++ /dev/null
@@ -1,310 +0,0 @@
-/*
- * Copyright 2012      Ecole Normale Superieure
- *
- * Use of this software is governed by the MIT license
- *
- * Written by Sven Verdoolaege,
- * Ecole Normale Superieure, 45 rue d’Ulm, 75230 Paris, France
- */
-
-#include <string.h>
-
-#include <isl/aff.h>
-
-#include "gpu_print.h"
-#include "print.h"
-#include "schedule.h"
-
-/* Print declarations to "p" for arrays that are local to "prog"
- * but that are used on the host and therefore require a declaration.
- */
-__isl_give isl_printer *gpu_print_local_declarations(__isl_take isl_printer *p,
-	struct gpu_prog *prog)
-{
-	int i;
-
-	if (!prog)
-		return isl_printer_free(p);
-
-	for (i = 0; i < prog->n_array; ++i) {
-		struct gpu_array_info *array = &prog->array[i];
-		isl_ast_expr *size;
-
-		if (!array->declare_local)
-			continue;
-		size = array->declared_size;
-		p = ppcg_print_declaration_with_size(p, array->type, size);
-	}
-
-	return p;
-}
-
-/* Print an expression for the size of "array" in bytes.
- */
-__isl_give isl_printer *gpu_array_info_print_size(__isl_take isl_printer *prn,
-	struct gpu_array_info *array)
-{
-	int i;
-
-	for (i = 0; i < array->n_index; ++i) {
-		isl_ast_expr *bound;
-
-		prn = isl_printer_print_str(prn, "(");
-		bound = isl_ast_expr_get_op_arg(array->bound_expr, 1 + i);
-		prn = isl_printer_print_ast_expr(prn, bound);
-		isl_ast_expr_free(bound);
-		prn = isl_printer_print_str(prn, ") * ");
-	}
-	prn = isl_printer_print_str(prn, "sizeof(");
-	prn = isl_printer_print_str(prn, array->type);
-	prn = isl_printer_print_str(prn, ")");
-
-	return prn;
-}
-
-/* Print the declaration of a non-linearized array argument.
- */
-static __isl_give isl_printer *print_non_linearized_declaration_argument(
-	__isl_take isl_printer *p, struct gpu_array_info *array)
-{
-	p = isl_printer_print_str(p, array->type);
-	p = isl_printer_print_str(p, " ");
-
-	p = isl_printer_print_ast_expr(p, array->bound_expr);
-
-	return p;
-}
-
-/* Print the declaration of an array argument.
- * "memory_space" allows to specify a memory space prefix.
- */
-__isl_give isl_printer *gpu_array_info_print_declaration_argument(
-	__isl_take isl_printer *p, struct gpu_array_info *array,
-	const char *memory_space)
-{
-	if (gpu_array_is_read_only_scalar(array)) {
-		p = isl_printer_print_str(p, array->type);
-		p = isl_printer_print_str(p, " ");
-		p = isl_printer_print_str(p, array->name);
-		return p;
-	}
-
-	if (memory_space) {
-		p = isl_printer_print_str(p, memory_space);
-		p = isl_printer_print_str(p, " ");
-	}
-
-	if (array->n_index != 0 && !array->linearize)
-		return print_non_linearized_declaration_argument(p, array);
-
-	p = isl_printer_print_str(p, array->type);
-	p = isl_printer_print_str(p, " ");
-	p = isl_printer_print_str(p, "*");
-	p = isl_printer_print_str(p, array->name);
-
-	return p;
-}
-
-/* Print the call of an array argument.
- */
-__isl_give isl_printer *gpu_array_info_print_call_argument(
-	__isl_take isl_printer *p, struct gpu_array_info *array)
-{
-	if (gpu_array_is_read_only_scalar(array))
-		return isl_printer_print_str(p, array->name);
-
-	p = isl_printer_print_str(p, "dev_");
-	p = isl_printer_print_str(p, array->name);
-
-	return p;
-}
-
-/* Print an access to the element in the private/shared memory copy
- * described by "stmt".  The index of the copy is recorded in
- * stmt->local_index as an access to the array.
- */
-static __isl_give isl_printer *stmt_print_local_index(__isl_take isl_printer *p,
-	struct ppcg_kernel_stmt *stmt)
-{
-	return isl_printer_print_ast_expr(p, stmt->u.c.local_index);
-}
-
-/* Print an access to the element in the global memory copy
- * described by "stmt".  The index of the copy is recorded in
- * stmt->index as an access to the array.
- */
-static __isl_give isl_printer *stmt_print_global_index(
-	__isl_take isl_printer *p, struct ppcg_kernel_stmt *stmt)
-{
-	struct gpu_array_info *array = stmt->u.c.array;
-	isl_ast_expr *index;
-
-	if (gpu_array_is_scalar(array)) {
-		if (!gpu_array_is_read_only_scalar(array))
-			p = isl_printer_print_str(p, "*");
-		p = isl_printer_print_str(p, array->name);
-		return p;
-	}
-
-	index = isl_ast_expr_copy(stmt->u.c.index);
-
-	p = isl_printer_print_ast_expr(p, index);
-	isl_ast_expr_free(index);
-
-	return p;
-}
-
-/* Print a copy statement.
- *
- * A read copy statement is printed as
- *
- *	local = global;
- *
- * while a write copy statement is printed as
- *
- *	global = local;
- */
-__isl_give isl_printer *ppcg_kernel_print_copy(__isl_take isl_printer *p,
-	struct ppcg_kernel_stmt *stmt)
-{
-	p = isl_printer_start_line(p);
-	if (stmt->u.c.read) {
-		p = stmt_print_local_index(p, stmt);
-		p = isl_printer_print_str(p, " = ");
-		p = stmt_print_global_index(p, stmt);
-	} else {
-		p = stmt_print_global_index(p, stmt);
-		p = isl_printer_print_str(p, " = ");
-		p = stmt_print_local_index(p, stmt);
-	}
-	p = isl_printer_print_str(p, ";");
-	p = isl_printer_end_line(p);
-
-	return p;
-}
-
-__isl_give isl_printer *ppcg_kernel_print_domain(__isl_take isl_printer *p,
-	struct ppcg_kernel_stmt *stmt)
-{
-	return pet_stmt_print_body(stmt->u.d.stmt->stmt, p, stmt->u.d.ref2expr);
-}
-
-/* This function is called for each node in a GPU AST.
- * In case of a user node, print the macro definitions required
- * for printing the AST expressions in the annotation, if any.
- * For other nodes, return true such that descendants are also
- * visited.
- *
- * In particular, for a kernel launch, print the macro definitions
- * needed for the grid size.
- * For a copy statement, print the macro definitions needed
- * for the two index expressions.
- * For an original user statement, print the macro definitions
- * needed for the substitutions.
- */
-static isl_bool at_node(__isl_keep isl_ast_node *node, void *user)
-{
-	const char *name;
-	isl_id *id;
-	int is_kernel;
-	struct ppcg_kernel *kernel;
-	struct ppcg_kernel_stmt *stmt;
-	isl_printer **p = user;
-
-	if (isl_ast_node_get_type(node) != isl_ast_node_user)
-		return isl_bool_true;
-
-	id = isl_ast_node_get_annotation(node);
-	if (!id)
-		return isl_bool_false;
-
-	name = isl_id_get_name(id);
-	if (!name)
-		return isl_bool_error;
-	is_kernel = !strcmp(name, "kernel");
-	kernel = is_kernel ? isl_id_get_user(id) : NULL;
-	stmt = is_kernel ? NULL : isl_id_get_user(id);
-	isl_id_free(id);
-
-	if ((is_kernel && !kernel) || (!is_kernel && !stmt))
-		return isl_bool_error;
-
-	if (is_kernel) {
-		*p = ppcg_ast_expr_print_macros(kernel->grid_size_expr, *p);
-	} else if (stmt->type == ppcg_kernel_copy) {
-		*p = ppcg_ast_expr_print_macros(stmt->u.c.index, *p);
-		*p = ppcg_ast_expr_print_macros(stmt->u.c.local_index, *p);
-	} else if (stmt->type == ppcg_kernel_domain) {
-		*p = ppcg_print_body_macros(*p, stmt->u.d.ref2expr);
-	}
-	if (!*p)
-		return isl_bool_error;
-
-	return isl_bool_false;
-}
-
-/* Print the required macros for the GPU AST "node" to "p",
- * including those needed for the user statements inside the AST.
- */
-__isl_give isl_printer *gpu_print_macros(__isl_take isl_printer *p,
-	__isl_keep isl_ast_node *node)
-{
-	if (isl_ast_node_foreach_descendant_top_down(node, &at_node, &p) < 0)
-		return isl_printer_free(p);
-	p = ppcg_print_macros(p, node);
-	return p;
-}
-
-/* Was the definition of "type" printed before?
- * That is, does its name appear in the list of printed types "types"?
- */
-static int already_printed(struct gpu_types *types,
-	struct pet_type *type)
-{
-	int i;
-
-	for (i = 0; i < types->n; ++i)
-		if (!strcmp(types->name[i], type->name))
-			return 1;
-
-	return 0;
-}
-
-/* Print the definitions of all types prog->scop that have not been
- * printed before (according to "types") on "p".
- * Extend the list of printed types "types" with the newly printed types.
- */
-__isl_give isl_printer *gpu_print_types(__isl_take isl_printer *p,
-	struct gpu_types *types, struct gpu_prog *prog)
-{
-	int i, n;
-	isl_ctx *ctx;
-	char **name;
-
-	n = prog->scop->pet->n_type;
-
-	if (n == 0)
-		return p;
-
-	ctx = isl_printer_get_ctx(p);
-	name = isl_realloc_array(ctx, types->name, char *, types->n + n);
-	if (!name)
-		return isl_printer_free(p);
-	types->name = name;
-
-	for (i = 0; i < n; ++i) {
-		struct pet_type *type = prog->scop->pet->types[i];
-
-		if (already_printed(types, type))
-			continue;
-
-		p = isl_printer_start_line(p);
-		p = isl_printer_print_str(p, type->definition);
-		p = isl_printer_print_str(p, ";");
-		p = isl_printer_end_line(p);
-
-		types->name[types->n++] = strdup(type->name);
-	}
-
-	return p;
-}
diff --git a/polly/lib/External/ppcg/gpu_print.h b/polly/lib/External/ppcg/gpu_print.h
deleted file mode 100644
index 7f07d3658696b..0000000000000
--- a/polly/lib/External/ppcg/gpu_print.h
+++ /dev/null
@@ -1,28 +0,0 @@
-#ifndef GPU_PRINT_H
-#define GPU_PRINT_H
-
-#include "gpu.h"
-
-__isl_give isl_printer *gpu_print_local_declarations(__isl_take isl_printer *p,
-	struct gpu_prog *prog);
-
-__isl_give isl_printer *gpu_print_types(__isl_take isl_printer *p,
-	struct gpu_types *types, struct gpu_prog *prog);
-
-__isl_give isl_printer *gpu_print_macros(__isl_take isl_printer *p,
-	__isl_keep isl_ast_node *node);
-
-__isl_give isl_printer *gpu_array_info_print_size(__isl_take isl_printer *prn,
-	struct gpu_array_info *array);
-__isl_give isl_printer *gpu_array_info_print_declaration_argument(
-	__isl_take isl_printer *p, struct gpu_array_info *array,
-	const char *memory_space);
-__isl_give isl_printer *gpu_array_info_print_call_argument(
-	__isl_take isl_printer *p, struct gpu_array_info *array);
-
-__isl_give isl_printer *ppcg_kernel_print_copy(__isl_take isl_printer *p,
-	struct ppcg_kernel_stmt *stmt);
-__isl_give isl_printer *ppcg_kernel_print_domain(__isl_take isl_printer *p,
-	struct ppcg_kernel_stmt *stmt);
-
-#endif
diff --git a/polly/lib/External/ppcg/gpu_tree.c b/polly/lib/External/ppcg/gpu_tree.c
deleted file mode 100644
index 0dcc7a144eef8..0000000000000
--- a/polly/lib/External/ppcg/gpu_tree.c
+++ /dev/null
@@ -1,640 +0,0 @@
-/*
- * Copyright 2013      Ecole Normale Superieure
- *
- * Use of this software is governed by the MIT license
- *
- * Written by Sven Verdoolaege,
- * Ecole Normale Superieure, 45 rue d'Ulm, 75230 Paris, France
- */
-
-#include <string.h>
-
-#include <isl/set.h>
-#include <isl/union_set.h>
-#include <isl/space.h>
-
-#include "gpu_tree.h"
-
-/* The functions in this file are used to navigate part of a schedule tree
- * that is mapped to blocks.  Initially, this part consists of a linear
- * branch segment with a mark node with name "kernel" on the outer end
- * and a mark node with name "thread" on the inner end.
- * During the mapping to blocks, branching may be introduced, but only
- * one of the elements in each sequence contains the "thread" mark.
- * The filter of this element (and only this filter) contains
- * domain elements identified by the "core" argument of the functions
- * that move down this tree.
- *
- * Synchronization statements have a name that starts with "sync" and
- * a user pointer pointing to the kernel that contains the synchronization.
- * The functions inserting or detecting synchronizations take a ppcg_kernel
- * argument to be able to create or identify such statements.
- * They may also use two fields in this structure, the "core" field
- * to move around in the tree and the "n_sync" field to make sure that
- * each synchronization has a different name (within the kernel).
- */
-
-/* Is "node" a mark node with an identifier called "name"?
- */
-static int is_marked(__isl_keep isl_schedule_node *node, const char *name)
-{
-	isl_id *mark;
-	int has_name;
-
-	if (!node)
-		return -1;
-
-	if (isl_schedule_node_get_type(node) != isl_schedule_node_mark)
-		return 0;
-
-	mark = isl_schedule_node_mark_get_id(node);
-	if (!mark)
-		return -1;
-
-	has_name = !strcmp(isl_id_get_name(mark), name);
-	isl_id_free(mark);
-
-	return has_name;
-}
-
-/* Is "node" a mark node with an identifier called "kernel"?
- */
-int gpu_tree_node_is_kernel(__isl_keep isl_schedule_node *node)
-{
-	return is_marked(node, "kernel");
-}
-
-/* Is "node" a mark node with an identifier called "shared"?
- */
-static int node_is_shared(__isl_keep isl_schedule_node *node)
-{
-	return is_marked(node, "shared");
-}
-
-/* Is "node" a mark node with an identifier called "thread"?
- */
-static int node_is_thread(__isl_keep isl_schedule_node *node)
-{
-	return is_marked(node, "thread");
-}
-
-/* Insert a mark node with identifier "shared" in front of "node".
- */
-static __isl_give isl_schedule_node *insert_shared(
-	__isl_take isl_schedule_node *node)
-{
-	isl_ctx *ctx;
-	isl_id *id;
-
-	ctx = isl_schedule_node_get_ctx(node);
-	id = isl_id_alloc(ctx, "shared", NULL);
-	node = isl_schedule_node_insert_mark(node, id);
-
-	return node;
-}
-
-/* Insert a "shared" mark in front of the "thread" mark
- * provided the linear branch between "node" and the "thread" mark
- * does not contain such a "shared" mark already.
- *
- * As a side effect, this function checks that the subtree at "node"
- * actually contains a "thread" mark and that there is no branching
- * in between "node" and this "thread" mark.
- */
-__isl_give isl_schedule_node *gpu_tree_insert_shared_before_thread(
-	__isl_take isl_schedule_node *node)
-{
-	int depth0, depth;
-	int any_shared = 0;
-
-	if (!node)
-		return NULL;
-
-	depth0 = isl_schedule_node_get_tree_depth(node);
-
-	for (;;) {
-		int is_thread;
-		int n;
-
-		if (!any_shared) {
-			any_shared = node_is_shared(node);
-			if (any_shared < 0)
-				return isl_schedule_node_free(node);
-		}
-		is_thread = node_is_thread(node);
-		if (is_thread < 0)
-			return isl_schedule_node_free(node);
-		if (is_thread)
-			break;
-		n = isl_schedule_node_n_children(node);
-		if (n == 0)
-			isl_die(isl_schedule_node_get_ctx(node),
-				isl_error_invalid,
-				"no thread marker found",
-				return isl_schedule_node_free(node));
-		if (n > 1)
-			isl_die(isl_schedule_node_get_ctx(node),
-				isl_error_invalid,
-				"expecting single thread marker",
-				return isl_schedule_node_free(node));
-
-		node = isl_schedule_node_child(node, 0);
-	}
-
-	if (!any_shared)
-		node = insert_shared(node);
-	depth = isl_schedule_node_get_tree_depth(node);
-	node = isl_schedule_node_ancestor(node, depth - depth0);
-
-	return node;
-}
-
-/* Assuming "node" is a filter node, does it correspond to the branch
- * that contains the "thread" mark, i.e., does it contain any elements
- * in "core"?
- */
-static int node_is_core(__isl_keep isl_schedule_node *node,
-	__isl_keep isl_union_set *core)
-{
-	int disjoint;
-	isl_union_set *filter;
-
-	filter = isl_schedule_node_filter_get_filter(node);
-	disjoint = isl_union_set_is_disjoint(filter, core);
-	isl_union_set_free(filter);
-	if (disjoint < 0)
-		return -1;
-
-	return !disjoint;
-}
-
-/* Move to the only child of "node" that has the "thread" mark as descendant,
- * where the branch containing this mark is identified by the domain elements
- * in "core".
- *
- * If "node" is not a sequence, then it only has one child and we move
- * to that single child.
- * Otherwise, we check each of the filters in the children, pick
- * the one that corresponds to "core" and return a pointer to the child
- * of the filter node.
- */
-static __isl_give isl_schedule_node *core_child(
-	__isl_take isl_schedule_node *node, __isl_keep isl_union_set *core)
-{
-	int i, n;
-
-	if (isl_schedule_node_get_type(node) != isl_schedule_node_sequence)
-		return isl_schedule_node_child(node, 0);
-
-	n = isl_schedule_node_n_children(node);
-	for (i = 0; i < n; ++i) {
-		int is_core;
-
-		node = isl_schedule_node_child(node, i);
-		is_core = node_is_core(node, core);
-
-		if (is_core < 0)
-			return isl_schedule_node_free(node);
-		if (is_core)
-			return isl_schedule_node_child(node, 0);
-
-		node = isl_schedule_node_parent(node);
-	}
-
-	isl_die(isl_schedule_node_get_ctx(node), isl_error_internal,
-		"core child not found", return isl_schedule_node_free(node));
-}
-
-/* Move down the branch between "kernel" and "thread" until
- * the "shared" mark is reached, where the branch containing the "shared"
- * mark is identified by the domain elements in "core".
- */
-__isl_give isl_schedule_node *gpu_tree_move_down_to_shared(
-	__isl_take isl_schedule_node *node, __isl_keep isl_union_set *core)
-{
-	int is_shared;
-
-	while ((is_shared = node_is_shared(node)) == 0)
-		node = core_child(node, core);
-	if (is_shared < 0)
-		node = isl_schedule_node_free(node);
-
-	return node;
-}
-
-/* Move down the branch between "kernel" and "thread" until
- * the "thread" mark is reached, where the branch containing the "thread"
- * mark is identified by the domain elements in "core".
- */
-__isl_give isl_schedule_node *gpu_tree_move_down_to_thread(
-	__isl_take isl_schedule_node *node, __isl_keep isl_union_set *core)
-{
-	int is_thread;
-
-	while ((is_thread = node_is_thread(node)) == 0)
-		node = core_child(node, core);
-	if (is_thread < 0)
-		node = isl_schedule_node_free(node);
-
-	return node;
-}
-
-/* Move up the tree underneath the "thread" mark until
- * the "thread" mark is reached.
- */
-__isl_give isl_schedule_node *gpu_tree_move_up_to_thread(
-	__isl_take isl_schedule_node *node)
-{
-	int is_thread;
-
-	while ((is_thread = node_is_thread(node)) == 0)
-		node = isl_schedule_node_parent(node);
-	if (is_thread < 0)
-		node = isl_schedule_node_free(node);
-
-	return node;
-}
-
-/* Move up the tree underneath the "kernel" mark until
- * the "kernel" mark is reached.
- */
-__isl_give isl_schedule_node *gpu_tree_move_up_to_kernel(
-	__isl_take isl_schedule_node *node)
-{
-	int is_kernel;
-
-	while ((is_kernel = gpu_tree_node_is_kernel(node)) == 0)
-		node = isl_schedule_node_parent(node);
-	if (is_kernel < 0)
-		node = isl_schedule_node_free(node);
-
-	return node;
-}
-
-/* Move down from the "kernel" mark (or at least a node with schedule
- * depth smaller than or equal to "depth") to a band node at schedule
- * depth "depth".  The "thread" mark is assumed to have a schedule
- * depth greater than or equal to "depth".  The branch containing the
- * "thread" mark is identified by the domain elements in "core".
- *
- * If the desired schedule depth is in the middle of band node,
- * then the band node is split into two pieces, the second piece
- * at the desired schedule depth.
- */
-__isl_give isl_schedule_node *gpu_tree_move_down_to_depth(
-	__isl_take isl_schedule_node *node, int depth,
-	__isl_keep isl_union_set *core)
-{
-	int is_shared;
-	int is_thread = 0;
-
-	while (node && isl_schedule_node_get_schedule_depth(node) < depth) {
-		if (isl_schedule_node_get_type(node) ==
-						    isl_schedule_node_band) {
-			int node_depth, node_dim;
-			node_depth = isl_schedule_node_get_schedule_depth(node);
-			node_dim = isl_schedule_node_band_n_member(node);
-			if (node_depth + node_dim > depth)
-				node = isl_schedule_node_band_split(node,
-							depth - node_depth);
-		}
-		node = core_child(node, core);
-	}
-	while ((is_shared = node_is_shared(node)) == 0 &&
-	    (is_thread = node_is_thread(node)) == 0 &&
-	    isl_schedule_node_get_type(node) != isl_schedule_node_band)
-		node = core_child(node, core);
-	if (is_shared < 0 || is_thread < 0)
-		node = isl_schedule_node_free(node);
-
-	return node;
-}
-
-/* Create a union set containing a single set with a tuple identifier
- * called "syncX" and user pointer equal to "kernel".
- */
-static __isl_give isl_union_set *create_sync_domain(struct ppcg_kernel *kernel)
-{
-	isl_space *space;
-	isl_id *id;
-	char name[40];
-
-	space = isl_space_set_alloc(kernel->ctx, 0, 0);
-	snprintf(name, sizeof(name), "sync%d", kernel->n_sync++);
-	id = isl_id_alloc(kernel->ctx, name, kernel);
-	space = isl_space_set_tuple_id(space, isl_dim_set, id);
-	return isl_union_set_from_set(isl_set_universe(space));
-}
-
-/* Is "id" the identifier of a synchronization statement inside "kernel"?
- * That is, does its name start with "sync" and does it point to "kernel"?
- */
-int gpu_tree_id_is_sync(__isl_keep isl_id *id, struct ppcg_kernel *kernel)
-{
-	const char *name;
-
-	name = isl_id_get_name(id);
-	if (!name)
-		return 0;
-	else if (strncmp(name, "sync", 4))
-		return 0;
-	return isl_id_get_user(id) == kernel;
-}
-
-/* Does "domain" consist of a single set with a tuple identifier
- * corresponding to a synchronization for "kernel"?
- */
-static int domain_is_sync(__isl_keep isl_union_set *domain,
-	struct ppcg_kernel *kernel)
-{
-	int is_sync;
-	isl_id *id;
-	isl_set *set;
-
-	if (isl_union_set_n_set(domain) != 1)
-		return 0;
-	set = isl_set_from_union_set(isl_union_set_copy(domain));
-	id = isl_set_get_tuple_id(set);
-	is_sync = gpu_tree_id_is_sync(id, kernel);
-	isl_id_free(id);
-	isl_set_free(set);
-
-	return is_sync;
-}
-
-/* Does "node" point to a filter selecting a synchronization statement
- * for "kernel"?
- */
-static int node_is_sync_filter(__isl_keep isl_schedule_node *node,
-	struct ppcg_kernel *kernel)
-{
-	int is_sync;
-	enum isl_schedule_node_type type;
-	isl_union_set *domain;
-
-	if (!node)
-		return -1;
-	type = isl_schedule_node_get_type(node);
-	if (type != isl_schedule_node_filter)
-		return 0;
-	domain = isl_schedule_node_filter_get_filter(node);
-	is_sync = domain_is_sync(domain, kernel);
-	isl_union_set_free(domain);
-
-	return is_sync;
-}
-
-/* Is "node" part of a sequence with a previous synchronization statement
- * for "kernel"?
- * That is, is the parent of "node" a filter such that there is
- * a previous filter that picks out exactly such a synchronization statement?
- */
-static int has_preceding_sync(__isl_keep isl_schedule_node *node,
-	struct ppcg_kernel *kernel)
-{
-	int found = 0;
-
-	node = isl_schedule_node_copy(node);
-	node = isl_schedule_node_parent(node);
-	while (!found && isl_schedule_node_has_previous_sibling(node)) {
-		node = isl_schedule_node_previous_sibling(node);
-		if (!node)
-			break;
-		found = node_is_sync_filter(node, kernel);
-	}
-	if (!node)
-		found = -1;
-	isl_schedule_node_free(node);
-
-	return found;
-}
-
-/* Is "node" part of a sequence with a subsequent synchronization statement
- * for "kernel"?
- * That is, is the parent of "node" a filter such that there is
- * a subsequent filter that picks out exactly such a synchronization statement?
- */
-static int has_following_sync(__isl_keep isl_schedule_node *node,
-	struct ppcg_kernel *kernel)
-{
-	int found = 0;
-
-	node = isl_schedule_node_copy(node);
-	node = isl_schedule_node_parent(node);
-	while (!found && isl_schedule_node_has_next_sibling(node)) {
-		node = isl_schedule_node_next_sibling(node);
-		if (!node)
-			break;
-		found = node_is_sync_filter(node, kernel);
-	}
-	if (!node)
-		found = -1;
-	isl_schedule_node_free(node);
-
-	return found;
-}
-
-/* Does the subtree rooted at "node" (which is a band node) contain
- * any synchronization statement for "kernel" that precedes
- * the core computation of "kernel" (identified by the elements
- * in kernel->core)?
- */
-static int has_sync_before_core(__isl_keep isl_schedule_node *node,
-	struct ppcg_kernel *kernel)
-{
-	int has_sync = 0;
-	int is_thread;
-
-	node = isl_schedule_node_copy(node);
-	while ((is_thread = node_is_thread(node)) == 0) {
-		node = core_child(node, kernel->core);
-		has_sync = has_preceding_sync(node, kernel);
-		if (has_sync < 0 || has_sync)
-			break;
-	}
-	if (is_thread < 0 || !node)
-		has_sync = -1;
-	isl_schedule_node_free(node);
-
-	return has_sync;
-}
-
-/* Does the subtree rooted at "node" (which is a band node) contain
- * any synchronization statement for "kernel" that follows
- * the core computation of "kernel" (identified by the elements
- * in kernel->core)?
- */
-static int has_sync_after_core(__isl_keep isl_schedule_node *node,
-	struct ppcg_kernel *kernel)
-{
-	int has_sync = 0;
-	int is_thread;
-
-	node = isl_schedule_node_copy(node);
-	while ((is_thread = node_is_thread(node)) == 0) {
-		node = core_child(node, kernel->core);
-		has_sync = has_following_sync(node, kernel);
-		if (has_sync < 0 || has_sync)
-			break;
-	}
-	if (is_thread < 0 || !node)
-		has_sync = -1;
-	isl_schedule_node_free(node);
-
-	return has_sync;
-}
-
-/* Insert (or extend) an extension on top of "node" that puts
- * a synchronization node for "kernel" before "node".
- * Return a pointer to the original node in the updated schedule tree.
- */
-static __isl_give isl_schedule_node *insert_sync_before(
-	__isl_take isl_schedule_node *node, struct ppcg_kernel *kernel)
-{
-	isl_union_set *domain;
-	isl_schedule_node *graft;
-
-	if (!node)
-		return NULL;
-
-	domain = create_sync_domain(kernel);
-	graft = isl_schedule_node_from_domain(domain);
-	node = isl_schedule_node_graft_before(node, graft);
-
-	return node;
-}
-
-/* Insert (or extend) an extension on top of "node" that puts
- * a synchronization node for "kernel" afater "node".
- * Return a pointer to the original node in the updated schedule tree.
- */
-static __isl_give isl_schedule_node *insert_sync_after(
-	__isl_take isl_schedule_node *node, struct ppcg_kernel *kernel)
-{
-	isl_union_set *domain;
-	isl_schedule_node *graft;
-
-	if (!node)
-		return NULL;
-
-	domain = create_sync_domain(kernel);
-	graft = isl_schedule_node_from_domain(domain);
-	node = isl_schedule_node_graft_after(node, graft);
-
-	return node;
-}
-
-/* Insert an extension on top of "node" that puts a synchronization node
- * for "kernel" before "node" unless there already is
- * such a synchronization node.
- */
-__isl_give isl_schedule_node *gpu_tree_ensure_preceding_sync(
-	__isl_take isl_schedule_node *node, struct ppcg_kernel *kernel)
-{
-	int has_sync;
-
-	has_sync = has_preceding_sync(node, kernel);
-	if (has_sync < 0)
-		return isl_schedule_node_free(node);
-	if (has_sync)
-		return node;
-	return insert_sync_before(node, kernel);
-}
-
-/* Insert an extension on top of "node" that puts a synchronization node
- * for "kernel" after "node" unless there already is
- * such a synchronization node.
- */
-__isl_give isl_schedule_node *gpu_tree_ensure_following_sync(
-	__isl_take isl_schedule_node *node, struct ppcg_kernel *kernel)
-{
-	int has_sync;
-
-	has_sync = has_following_sync(node, kernel);
-	if (has_sync < 0)
-		return isl_schedule_node_free(node);
-	if (has_sync)
-		return node;
-	return insert_sync_after(node, kernel);
-}
-
-/* Insert an extension on top of "node" that puts a synchronization node
- * for "kernel" after "node" unless there already is such a sync node or
- * "node" itself already * contains a synchronization node following
- * the core computation of "kernel".
- */
-__isl_give isl_schedule_node *gpu_tree_ensure_sync_after_core(
-	__isl_take isl_schedule_node *node, struct ppcg_kernel *kernel)
-{
-	int has_sync;
-
-	has_sync = has_sync_after_core(node, kernel);
-	if (has_sync < 0)
-		return isl_schedule_node_free(node);
-	if (has_sync)
-		return node;
-	has_sync = has_following_sync(node, kernel);
-	if (has_sync < 0)
-		return isl_schedule_node_free(node);
-	if (has_sync)
-		return node;
-	return insert_sync_after(node, kernel);
-}
-
-/* Move left in the sequence on top of "node" to a synchronization node
- * for "kernel".
- * If "node" itself contains a synchronization node preceding
- * the core computation of "kernel", then return "node" itself.
- * Otherwise, if "node" does not have a preceding synchronization node,
- * then create one first.
- */
-__isl_give isl_schedule_node *gpu_tree_move_left_to_sync(
-	__isl_take isl_schedule_node *node, struct ppcg_kernel *kernel)
-{
-	int has_sync;
-	int is_sync;
-
-	has_sync = has_sync_before_core(node, kernel);
-	if (has_sync < 0)
-		return isl_schedule_node_free(node);
-	if (has_sync)
-		return node;
-	node = gpu_tree_ensure_preceding_sync(node, kernel);
-	node = isl_schedule_node_parent(node);
-	while ((is_sync = node_is_sync_filter(node, kernel)) == 0)
-		node = isl_schedule_node_previous_sibling(node);
-	if (is_sync < 0)
-		node = isl_schedule_node_free(node);
-	node = isl_schedule_node_child(node, 0);
-
-	return node;
-}
-
-/* Move right in the sequence on top of "node" to a synchronization node
- * for "kernel".
- * If "node" itself contains a synchronization node following
- * the core computation of "kernel", then return "node" itself.
- * Otherwise, if "node" does not have a following synchronization node,
- * then create one first.
- */
-__isl_give isl_schedule_node *gpu_tree_move_right_to_sync(
-	__isl_take isl_schedule_node *node, struct ppcg_kernel *kernel)
-{
-	int has_sync;
-	int is_sync;
-
-	has_sync = has_sync_after_core(node, kernel);
-	if (has_sync < 0)
-		return isl_schedule_node_free(node);
-	if (has_sync)
-		return node;
-	node = gpu_tree_ensure_following_sync(node, kernel);
-	node = isl_schedule_node_parent(node);
-	while ((is_sync = node_is_sync_filter(node, kernel)) == 0)
-		node = isl_schedule_node_next_sibling(node);
-	if (is_sync < 0)
-		node = isl_schedule_node_free(node);
-	node = isl_schedule_node_child(node, 0);
-
-	return node;
-}
diff --git a/polly/lib/External/ppcg/gpu_tree.h b/polly/lib/External/ppcg/gpu_tree.h
deleted file mode 100644
index a2a0258437583..0000000000000
--- a/polly/lib/External/ppcg/gpu_tree.h
+++ /dev/null
@@ -1,33 +0,0 @@
-#ifndef GPU_TREE_H
-#define GPU_TREE_H
-
-#include <isl/schedule_node.h>
-
-#include "gpu.h"
-
-__isl_give isl_schedule_node *gpu_tree_insert_shared_before_thread(
-	__isl_take isl_schedule_node *node);
-int gpu_tree_node_is_kernel(__isl_keep isl_schedule_node *node);
-__isl_give isl_schedule_node *gpu_tree_move_down_to_shared(
-	__isl_take isl_schedule_node *node, __isl_keep isl_union_set *core);
-__isl_give isl_schedule_node *gpu_tree_move_up_to_thread(
-	__isl_take isl_schedule_node *node);
-__isl_give isl_schedule_node *gpu_tree_move_down_to_thread(
-	__isl_take isl_schedule_node *node, __isl_keep isl_union_set *core);
-__isl_give isl_schedule_node *gpu_tree_move_up_to_kernel(
-	__isl_take isl_schedule_node *node);
-__isl_give isl_schedule_node *gpu_tree_move_down_to_depth(
-	__isl_take isl_schedule_node *node, int depth,
-	__isl_keep isl_union_set *core);
-
-int gpu_tree_id_is_sync(__isl_keep isl_id *id, struct ppcg_kernel *kernel);
-__isl_give isl_schedule_node *gpu_tree_ensure_sync_after_core(
-	__isl_take isl_schedule_node *node, struct ppcg_kernel *kernel);
-__isl_give isl_schedule_node *gpu_tree_ensure_following_sync(
-	__isl_take isl_schedule_node *node, struct ppcg_kernel *kernel);
-__isl_give isl_schedule_node *gpu_tree_move_left_to_sync(
-	__isl_take isl_schedule_node *node, struct ppcg_kernel *kernel);
-__isl_give isl_schedule_node *gpu_tree_move_right_to_sync(
-	__isl_take isl_schedule_node *node, struct ppcg_kernel *kernel);
-
-#endif
diff --git a/polly/lib/External/ppcg/grouping.c b/polly/lib/External/ppcg/grouping.c
deleted file mode 100644
index 1c98bdbc6d0a4..0000000000000
--- a/polly/lib/External/ppcg/grouping.c
+++ /dev/null
@@ -1,684 +0,0 @@
-/*
- * Copyright 2016      Sven Verdoolaege
- *
- * Use of this software is governed by the MIT license
- *
- * Written by Sven Verdoolaege.
- */
-
-#include <isl/ctx.h>
-#include <isl/id.h>
-#include <isl/val.h>
-#include <isl/space.h>
-#include <isl/aff.h>
-#include <isl/set.h>
-#include <isl/map.h>
-#include <isl/union_set.h>
-#include <isl/union_map.h>
-#include <isl/schedule.h>
-#include <isl/schedule_node.h>
-
-#include "ppcg.h"
-
-/* Internal data structure for use during the detection of statements
- * that can be grouped.
- *
- * "sc" contains the original schedule constraints (not a copy).
- * "dep" contains the intersection of the validity and the proximity
- * constraints in "sc".  It may be NULL if it has not been computed yet.
- * "group_id" is the identifier for the next group that is extracted.
- *
- * "domain" is the set of statement instances that belong to any of the groups.
- * "contraction" maps the elements of "domain" to the corresponding group
- * instances.
- * "schedule" schedules the statements in each group relatively to each other.
- * These last three fields are NULL if no groups have been found so far.
- */
-struct ppcg_grouping {
-	isl_schedule_constraints *sc;
-
-	isl_union_map *dep;
-	int group_id;
-
-	isl_union_set *domain;
-	isl_union_pw_multi_aff *contraction;
-	isl_schedule *schedule;
-};
-
-/* Clear all memory allocated by "grouping".
- */
-static void ppcg_grouping_clear(struct ppcg_grouping *grouping)
-{
-	isl_union_map_free(grouping->dep);
-	isl_union_set_free(grouping->domain);
-	isl_union_pw_multi_aff_free(grouping->contraction);
-	isl_schedule_free(grouping->schedule);
-}
-
-/* Compute the intersection of the proximity and validity dependences
- * in grouping->sc and store the result in grouping->dep, unless
- * this intersection has been computed before.
- */
-static isl_stat ppcg_grouping_compute_dep(struct ppcg_grouping *grouping)
-{
-	isl_union_map *validity, *proximity;
-
-	if (grouping->dep)
-		return isl_stat_ok;
-
-	validity = isl_schedule_constraints_get_validity(grouping->sc);
-	proximity = isl_schedule_constraints_get_proximity(grouping->sc);
-	grouping->dep = isl_union_map_intersect(validity, proximity);
-
-	if (!grouping->dep)
-		return isl_stat_error;
-
-	return isl_stat_ok;
-}
-
-/* Information extracted from one or more consecutive leaves
- * in the input schedule.
- *
- * "list" contains the sets of statement instances in the leaves,
- * one element in the list for each original leaf.
- * "domain" contains the union of the sets in "list".
- * "prefix" contains the prefix schedule of these elements.
- */
-struct ppcg_grouping_leaf {
-	isl_union_set *domain;
-	isl_union_set_list *list;
-	isl_multi_union_pw_aff *prefix;
-};
-
-/* Free all memory allocated for "leaves".
- */
-static void ppcg_grouping_leaf_free(int n, struct ppcg_grouping_leaf leaves[])
-{
-	int i;
-
-	if (!leaves)
-		return;
-
-	for (i = 0; i < n; ++i) {
-		isl_union_set_free(leaves[i].domain);
-		isl_union_set_list_free(leaves[i].list);
-		isl_multi_union_pw_aff_free(leaves[i].prefix);
-	}
-
-	free(leaves);
-}
-
-/* Short-hand for retrieving the prefix schedule at "node"
- * in the form of an isl_multi_union_pw_aff.
- */
-static __isl_give isl_multi_union_pw_aff *get_prefix(
-	__isl_keep isl_schedule_node *node)
-{
-	return isl_schedule_node_get_prefix_schedule_multi_union_pw_aff(node);
-}
-
-/* Return an array of "n" elements with information extracted from
- * the "n" children of "node" starting at "first", all of which
- * are known to be filtered leaves.
- */
-struct ppcg_grouping_leaf *extract_leaves(__isl_keep isl_schedule_node *node,
-	int first, int n)
-{
-	int i;
-	isl_ctx *ctx;
-	struct ppcg_grouping_leaf *leaves;
-
-	if (!node)
-		return NULL;
-
-	ctx = isl_schedule_node_get_ctx(node);
-	leaves = isl_calloc_array(ctx, struct ppcg_grouping_leaf, n);
-	if (!leaves)
-		return NULL;
-
-	for (i = 0; i < n; ++i) {
-		isl_schedule_node *child;
-		isl_union_set *domain;
-
-		child = isl_schedule_node_get_child(node, first + i);
-		child = isl_schedule_node_child(child, 0);
-		domain = isl_schedule_node_get_domain(child);
-		leaves[i].domain = isl_union_set_copy(domain);
-		leaves[i].list = isl_union_set_list_from_union_set(domain);
-		leaves[i].prefix = get_prefix(child);
-		isl_schedule_node_free(child);
-	}
-
-	return leaves;
-}
-
-/* Internal data structure used by merge_leaves.
- *
- * "src" and "dst" point to the two consecutive leaves that are
- * under investigation for being merged.
- * "merge" is initially set to 0 and is set to 1 as soon as
- * it turns out that it is useful to merge the two leaves.
- */
-struct ppcg_merge_leaves_data {
-	int merge;
-	struct ppcg_grouping_leaf *src;
-	struct ppcg_grouping_leaf *dst;
-};
-
-/* Given a relation "map" between instances of two statements A and B,
- * does it relate every instance of A (according to the domain of "src")
- * to every instance of B (according to the domain of "dst")?
- */
-static isl_bool covers_src_and_dst(__isl_keep isl_map *map,
-	struct ppcg_grouping_leaf *src, struct ppcg_grouping_leaf *dst)
-{
-	isl_space *space;
-	isl_set *set1, *set2;
-	isl_bool is_subset;
-
-	space = isl_space_domain(isl_map_get_space(map));
-	set1 = isl_union_set_extract_set(src->domain, space);
-	set2 = isl_map_domain(isl_map_copy(map));
-	is_subset = isl_set_is_subset(set1, set2);
-	isl_set_free(set1);
-	isl_set_free(set2);
-	if (is_subset < 0 || !is_subset)
-		return is_subset;
-
-	space = isl_space_range(isl_map_get_space(map));
-	set1 = isl_union_set_extract_set(dst->domain, space);
-	set2 = isl_map_range(isl_map_copy(map));
-	is_subset = isl_set_is_subset(set1, set2);
-	isl_set_free(set1);
-	isl_set_free(set2);
-
-	return is_subset;
-}
-
-/* Given a relation "map" between instances of two statements A and B,
- * are pairs of related instances executed together in the input schedule?
- * That is, is each pair of instances assigned the same value
- * by the corresponding prefix schedules?
- *
- * In particular, select the subset of "map" that has pairs of elements
- * with the same value for the prefix schedules and then check
- * if "map" is still a subset of the result.
- */
-static isl_bool matches_prefix(__isl_keep isl_map *map,
-	struct ppcg_grouping_leaf *src, struct ppcg_grouping_leaf *dst)
-{
-	isl_union_map *umap, *equal;
-	isl_multi_union_pw_aff *src_prefix, *dst_prefix, *prefix;
-	isl_bool is_subset;
-
-	src_prefix = isl_multi_union_pw_aff_copy(src->prefix);
-	dst_prefix = isl_multi_union_pw_aff_copy(dst->prefix);
-	prefix = isl_multi_union_pw_aff_union_add(src_prefix, dst_prefix);
-
-	umap = isl_union_map_from_map(isl_map_copy(map));
-	equal = isl_union_map_copy(umap);
-	equal = isl_union_map_eq_at_multi_union_pw_aff(equal, prefix);
-
-	is_subset = isl_union_map_is_subset(umap, equal);
-
-	isl_union_map_free(umap);
-	isl_union_map_free(equal);
-
-	return is_subset;
-}
-
-/* Given a set of validity and proximity schedule constraints "map"
- * between statements in consecutive leaves in a valid schedule,
- * should the two leaves be merged into one?
- *
- * In particular, the two are merged if the constraints form
- * a bijection between every instance of the first statement and
- * every instance of the second statement.  Moreover, each
- * pair of such dependent instances needs to be executed consecutively
- * in the input schedule.  That is, they need to be assigned
- * the same value by their prefix schedules.
- *
- * What this means is that for each instance of the first statement
- * there is exactly one instance of the second statement that
- * is executed immediately after the instance of the first statement and
- * that, moreover, both depends on this statement instance and
- * should be brought as close as possible to this statement instance.
- * In other words, it is both possible to execute the two instances
- * together (according to the input schedule) and desirable to do so
- * (according to the validity and proximity schedule constraints).
- */
-static isl_stat check_merge(__isl_take isl_map *map, void *user)
-{
-	struct ppcg_merge_leaves_data *data = user;
-	isl_bool ok;
-
-	ok = covers_src_and_dst(map, data->src, data->dst);
-	if (ok >= 0 && ok)
-		ok = isl_map_is_bijective(map);
-	if (ok >= 0 && ok)
-		ok = matches_prefix(map, data->src, data->dst);
-
-	isl_map_free(map);
-
-	if (ok < 0)
-		return isl_stat_error;
-	if (!ok)
-		return isl_stat_ok;
-
-	data->merge = 1;
-	return isl_stat_error;
-}
-
-/* Merge the leaves at position "pos" and "pos + 1" in "leaves".
- */
-static isl_stat merge_pair(int n, struct ppcg_grouping_leaf leaves[], int pos)
-{
-	int i;
-
-	leaves[pos].domain = isl_union_set_union(leaves[pos].domain,
-						leaves[pos + 1].domain);
-	leaves[pos].list = isl_union_set_list_concat(leaves[pos].list,
-						leaves[pos + 1].list);
-	leaves[pos].prefix = isl_multi_union_pw_aff_union_add(
-				leaves[pos].prefix, leaves[pos + 1].prefix);
-	for (i = pos + 1; i + 1 < n; ++i)
-		leaves[i] = leaves[i + 1];
-	leaves[n - 1].domain = NULL;
-	leaves[n - 1].list = NULL;
-	leaves[n - 1].prefix = NULL;
-
-	if (!leaves[pos].domain || !leaves[pos].list || !leaves[pos].prefix)
-		return isl_stat_error;
-
-	return isl_stat_ok;
-}
-
-/* Merge pairs of consecutive leaves in "leaves" taking into account
- * the intersection of validity and proximity schedule constraints "dep".
- *
- * If a leaf has been merged with the next leaf, then the combination
- * is checked again for merging with the next leaf.
- * That is, if the leaves are A, B and C, then B may not have been
- * merged with C, but after merging A and B, it could still be useful
- * to merge the combination AB with C.
- *
- * Two leaves A and B are merged if there are instances of at least
- * one pair of statements, one statement in A and one B, such that
- * the validity and proximity schedule constraints between them
- * make them suitable for merging according to check_merge.
- *
- * Return the final number of leaves in the sequence, or -1 on error.
- */
-static int merge_leaves(int n, struct ppcg_grouping_leaf leaves[],
-	__isl_keep isl_union_map *dep)
-{
-	int i;
-	struct ppcg_merge_leaves_data data;
-
-	for (i = n - 1; i >= 0; --i) {
-		isl_union_map *dep_i;
-		isl_stat ok;
-
-		if (i + 1 >= n)
-			continue;
-
-		dep_i = isl_union_map_copy(dep);
-		dep_i = isl_union_map_intersect_domain(dep_i,
-				isl_union_set_copy(leaves[i].domain));
-		dep_i = isl_union_map_intersect_range(dep_i,
-				isl_union_set_copy(leaves[i + 1].domain));
-		data.merge = 0;
-		data.src = &leaves[i];
-		data.dst = &leaves[i + 1];
-		ok = isl_union_map_foreach_map(dep_i, &check_merge, &data);
-		isl_union_map_free(dep_i);
-		if (ok < 0 && !data.merge)
-			return -1;
-		if (!data.merge)
-			continue;
-		if (merge_pair(n, leaves, i) < 0)
-			return -1;
-		--n;
-		++i;
-	}
-
-	return n;
-}
-
-/* Construct a schedule with "domain" as domain, that executes
- * the elements of "list" in order (as a sequence).
- */
-static __isl_give isl_schedule *schedule_from_domain_and_list(
-	__isl_keep isl_union_set *domain, __isl_keep isl_union_set_list *list)
-{
-	isl_schedule *schedule;
-	isl_schedule_node *node;
-
-	schedule = isl_schedule_from_domain(isl_union_set_copy(domain));
-	node = isl_schedule_get_root(schedule);
-	isl_schedule_free(schedule);
-	node = isl_schedule_node_child(node, 0);
-	list = isl_union_set_list_copy(list);
-	node = isl_schedule_node_insert_sequence(node, list);
-	schedule = isl_schedule_node_get_schedule(node);
-	isl_schedule_node_free(node);
-
-	return schedule;
-}
-
-/* Construct a unique identifier for a group in "grouping".
- *
- * The name is of the form G_n, with n the first value starting at
- * grouping->group_id that does not result in an identifier
- * that is already in use in the domain of the original schedule
- * constraints.
- */
-static isl_id *construct_group_id(struct ppcg_grouping *grouping,
-	__isl_take isl_space *space)
-{
-	isl_ctx *ctx;
-	isl_id *id;
-	isl_bool empty;
-	isl_union_set *domain;
-
-	if (!space)
-		return NULL;
-
-	ctx = isl_space_get_ctx(space);
-	domain = isl_schedule_constraints_get_domain(grouping->sc);
-
-	do {
-		char buffer[20];
-		isl_id *id;
-		isl_set *set;
-
-		snprintf(buffer, sizeof(buffer), "G_%d", grouping->group_id);
-		grouping->group_id++;
-		id = isl_id_alloc(ctx, buffer, NULL);
-		space = isl_space_set_tuple_id(space, isl_dim_set, id);
-		set = isl_union_set_extract_set(domain, isl_space_copy(space));
-		empty = isl_set_plain_is_empty(set);
-		isl_set_free(set);
-	} while (empty >= 0 && !empty);
-
-	if (empty < 0)
-		space = isl_space_free(space);
-
-	id = isl_space_get_tuple_id(space, isl_dim_set);
-
-	isl_space_free(space);
-	isl_union_set_free(domain);
-
-	return id;
-}
-
-/* Construct a contraction from "prefix" and "domain" for a new group
- * in "grouping".
- *
- * The values of the prefix schedule "prefix" are used as instances
- * of the new group.  The identifier of the group is constructed
- * in such a way that it does not conflict with those of earlier
- * groups nor with statements in the domain of the original
- * schedule constraints.
- * The isl_multi_union_pw_aff "prefix" then simply needs to be
- * converted to an isl_union_pw_multi_aff.  However, this is not
- * possible if "prefix" is zero-dimensional, so in this case,
- * a contraction is constructed from "domain" instead.
- */
-static isl_union_pw_multi_aff *group_contraction_from_prefix_and_domain(
-	struct ppcg_grouping *grouping,
-	__isl_keep isl_multi_union_pw_aff *prefix,
-	__isl_keep isl_union_set *domain)
-{
-	isl_id *id;
-	isl_space *space;
-	int dim;
-
-	space = isl_multi_union_pw_aff_get_space(prefix);
-	if (!space)
-		return NULL;
-	dim = isl_space_dim(space, isl_dim_set);
-	id = construct_group_id(grouping, space);
-	if (dim == 0) {
-		isl_multi_val *mv;
-
-		space = isl_multi_union_pw_aff_get_space(prefix);
-		space = isl_space_set_tuple_id(space, isl_dim_set, id);
-		mv = isl_multi_val_zero(space);
-		domain = isl_union_set_copy(domain);
-		return isl_union_pw_multi_aff_multi_val_on_domain(domain, mv);
-	}
-	prefix = isl_multi_union_pw_aff_copy(prefix);
-	prefix = isl_multi_union_pw_aff_set_tuple_id(prefix, isl_dim_out, id);
-	return isl_union_pw_multi_aff_from_multi_union_pw_aff(prefix);
-}
-
-/* Extend "grouping" with groups corresponding to merged
- * leaves in the list of potentially merged leaves "leaves".
- *
- * The "list" field of each element in "leaves" contains a list
- * of the instances sets of the original leaves that have been
- * merged into this element.  If at least two of the original leaves
- * have been merged into a given element, then add the corresponding
- * group to "grouping".
- * In particular, the domain is extended with the statement instances
- * of the merged leaves, the contraction is extended with a mapping
- * of these statement instances to instances of a new group and
- * the schedule is extended with a schedule that executes
- * the statement instances according to the order of the leaves
- * in which they appear.
- * Since the instances of the groups should already be scheduled apart
- * in the schedule into which this schedule will be plugged in,
- * the schedules of the individual groups are combined independently
- * of each other (as a set).
- */
-static isl_stat add_groups(struct ppcg_grouping *grouping,
-	int n, struct ppcg_grouping_leaf leaves[])
-{
-	int i;
-
-	for (i = 0; i < n; ++i) {
-		int n_leaf;
-		isl_schedule *schedule;
-		isl_union_set *domain;
-		isl_union_pw_multi_aff *upma;
-
-		n_leaf = isl_union_set_list_n_union_set(leaves[i].list);
-		if (n_leaf < 0)
-			return isl_stat_error;
-		if (n_leaf <= 1)
-			continue;
-		schedule = schedule_from_domain_and_list(leaves[i].domain,
-							leaves[i].list);
-		upma = group_contraction_from_prefix_and_domain(grouping,
-					leaves[i].prefix, leaves[i].domain);
-
-		domain = isl_union_set_copy(leaves[i].domain);
-		if (grouping->domain) {
-			domain = isl_union_set_union(domain, grouping->domain);
-			upma = isl_union_pw_multi_aff_union_add(upma,
-						grouping->contraction);
-			schedule = isl_schedule_set(schedule,
-						grouping->schedule);
-		}
-		grouping->domain = domain;
-		grouping->contraction = upma;
-		grouping->schedule = schedule;
-
-		if (!grouping->domain || !grouping->contraction ||
-		    !grouping->schedule)
-			return isl_stat_error;
-	}
-
-	return isl_stat_ok;
-}
-
-/* Look for any pairs of consecutive leaves among the "n" children of "node"
- * starting at "first" that should be merged together.
- * Store the results in "grouping".
- *
- * First make sure the intersection of validity and proximity
- * schedule constraints is available and extract the required
- * information from the "n" leaves.
- * Then try and merge consecutive leaves based on the validity
- * and proximity constraints.
- * If any pairs were successfully merged, then add groups
- * corresponding to the merged leaves to "grouping".
- */
-static isl_stat group_subsequence(__isl_keep isl_schedule_node *node,
-	int first, int n, struct ppcg_grouping *grouping)
-{
-	int n_merge;
-	struct ppcg_grouping_leaf *leaves;
-
-	if (ppcg_grouping_compute_dep(grouping) < 0)
-		return isl_stat_error;
-
-	leaves = extract_leaves(node, first, n);
-	if (!leaves)
-		return isl_stat_error;
-
-	n_merge = merge_leaves(n, leaves, grouping->dep);
-	if (n_merge >= 0 && n_merge < n &&
-	    add_groups(grouping, n_merge, leaves) < 0)
-		return isl_stat_error;
-
-	ppcg_grouping_leaf_free(n, leaves);
-
-	return isl_stat_ok;
-}
-
-/* If "node" is a sequence, then check if it has any consecutive
- * leaves that should be merged together and store the results
- * in "grouping".
- *
- * In particular, call group_subsequence on each consecutive
- * sequence of (filtered) leaves among the children of "node".
- */
-static isl_bool detect_groups(__isl_keep isl_schedule_node *node, void *user)
-{
-	int i, n, first;
-	struct ppcg_grouping *grouping = user;
-
-	if (isl_schedule_node_get_type(node) != isl_schedule_node_sequence)
-		return isl_bool_true;
-
-	n = isl_schedule_node_n_children(node);
-	if (n < 0)
-		return isl_bool_error;
-
-	first = -1;
-	for (i = 0; i < n; ++i) {
-		isl_schedule_node *child;
-		enum isl_schedule_node_type type;
-
-		child = isl_schedule_node_get_child(node, i);
-		child = isl_schedule_node_child(child, 0);
-		type = isl_schedule_node_get_type(child);
-		isl_schedule_node_free(child);
-
-		if (first >= 0 && type != isl_schedule_node_leaf) {
-			if (group_subsequence(node, first, i - first,
-						grouping) < 0)
-				return isl_bool_error;
-			first = -1;
-		}
-		if (first < 0 && type == isl_schedule_node_leaf)
-			first = i;
-	}
-	if (first >= 0) {
-		if (group_subsequence(node, first, n - first, grouping) < 0)
-			return isl_bool_error;
-	}
-
-	return isl_bool_true;
-}
-
-/* Complete "grouping" to cover all statement instances in the domain
- * of grouping->sc.
- *
- * In particular, grouping->domain is set to the full set of statement
- * instances; group->contraction is extended with an identity
- * contraction on the additional instances and group->schedule
- * is extended with an independent schedule on those additional instances.
- * In the extension of group->contraction, the additional instances
- * are split into those belong to different statements and those
- * that belong to some of the same statements.  The first group
- * is replaced by its universe in order to simplify the contraction extension.
- */
-static void complete_grouping(struct ppcg_grouping *grouping)
-{
-	isl_union_set *domain, *left, *overlap;
-	isl_union_pw_multi_aff *upma;
-	isl_schedule *schedule;
-
-	domain = isl_schedule_constraints_get_domain(grouping->sc);
-	left = isl_union_set_subtract(isl_union_set_copy(domain),
-				    isl_union_set_copy(grouping->domain));
-	schedule = isl_schedule_from_domain(isl_union_set_copy(left));
-	schedule = isl_schedule_set(schedule, grouping->schedule);
-	grouping->schedule = schedule;
-
-	overlap = isl_union_set_universe(grouping->domain);
-	grouping->domain = domain;
-	overlap = isl_union_set_intersect(isl_union_set_copy(left), overlap);
-	left = isl_union_set_subtract(left, isl_union_set_copy(overlap));
-	left = isl_union_set_universe(left);
-	left = isl_union_set_union(left, overlap);
-	upma = isl_union_set_identity_union_pw_multi_aff(left);
-	upma = isl_union_pw_multi_aff_union_add(upma, grouping->contraction);
-	grouping->contraction = upma;
-}
-
-/* Compute a schedule on the domain of "sc" that respects the schedule
- * constraints in "sc".
- *
- * "schedule" is a known correct schedule that is used to combine
- * groups of statements if options->group_chains is set.
- * In particular, statements that are executed consecutively in a sequence
- * in this schedule and where all instances of the second depend on
- * the instance of the first that is executed in the same iteration
- * of outer band nodes are grouped together into a single statement.
- * The schedule constraints are then mapped to these groups of statements
- * and the resulting schedule is expanded again to refer to the original
- * statements.
- */
-__isl_give isl_schedule *ppcg_compute_schedule(
-	__isl_take isl_schedule_constraints *sc,
-	__isl_keep isl_schedule *schedule, struct ppcg_options *options)
-{
-	struct ppcg_grouping grouping = { sc };
-	isl_union_pw_multi_aff *contraction;
-	isl_union_map *umap;
-	isl_schedule *res, *expansion;
-
-	if (!options->group_chains)
-		return isl_schedule_constraints_compute_schedule(sc);
-
-	grouping.group_id = 0;
-	if (isl_schedule_foreach_schedule_node_top_down(schedule,
-			&detect_groups, &grouping) < 0)
-		goto error;
-	if (!grouping.contraction) {
-		ppcg_grouping_clear(&grouping);
-		return isl_schedule_constraints_compute_schedule(sc);
-	}
-	complete_grouping(&grouping);
-	contraction = isl_union_pw_multi_aff_copy(grouping.contraction);
-	umap = isl_union_map_from_union_pw_multi_aff(contraction);
-
-	sc = isl_schedule_constraints_apply(sc, umap);
-
-	res = isl_schedule_constraints_compute_schedule(sc);
-
-	contraction = isl_union_pw_multi_aff_copy(grouping.contraction);
-	expansion = isl_schedule_copy(grouping.schedule);
-	res = isl_schedule_expand(res, contraction, expansion);
-
-	ppcg_grouping_clear(&grouping);
-	return res;
-error:
-	ppcg_grouping_clear(&grouping);
-	isl_schedule_constraints_free(sc);
-	return NULL;
-}
diff --git a/polly/lib/External/ppcg/hybrid.c b/polly/lib/External/ppcg/hybrid.c
deleted file mode 100644
index b0a047cb5c928..0000000000000
--- a/polly/lib/External/ppcg/hybrid.c
+++ /dev/null
@@ -1,2242 +0,0 @@
-/*
- * Copyright 2013      Ecole Normale Superieure
- * Copyright 2015      Sven Verdoolaege
- *
- * Use of this software is governed by the MIT license
- *
- * Written by Sven Verdoolaege,
- * Ecole Normale Superieure, 45 rue d'Ulm, 75230 Paris, France
- */
-
-#include <string.h>
-
-#include <isl/space.h>
-#include <isl/constraint.h>
-#include <isl/val.h>
-#include <isl/aff.h>
-#include <isl/set.h>
-#include <isl/map.h>
-#include <isl/union_set.h>
-#include <isl/union_map.h>
-
-#include "hybrid.h"
-#include "schedule.h"
-
-/* The hybrid tiling implemented in this file is based on
- * Grosser et al., "Hybrid Hexagonal/Classical Tiling for GPUs".
- */
-
-/* Bounds on relative dependence distances in input to hybrid tiling.
- * upper is an upper bound on the relative dependence distances
- * in the first space dimension
- * -lower is a lower bound on the relative dependence distances
- * in all space dimensions.
- *
- * In particular,
- *
- *	d_i >= -lower_i d_0
- * and
- *	d_1 <= upper d_0
- *
- * for each dependence distance vector d, where d_1 is the component
- * corresponding to the first space dimension.
- *
- * upper and lower are always non-negative.
- * Some of the values may be NaN if no bound could be found.
- */
-struct ppcg_ht_bounds {
-	isl_val *upper;
-	isl_multi_val *lower;
-};
-
-/* Free "bounds" along with all its fields.
- */
-__isl_null ppcg_ht_bounds *ppcg_ht_bounds_free(
-	__isl_take ppcg_ht_bounds *bounds)
-{
-	if (!bounds)
-		return NULL;
-	isl_val_free(bounds->upper);
-	isl_multi_val_free(bounds->lower);
-	free(bounds);
-
-	return NULL;
-}
-
-/* Create a ppcg_ht_bounds object for a band living in "space".
- * The bounds are initialized to NaN.
- */
-__isl_give ppcg_ht_bounds *ppcg_ht_bounds_alloc(__isl_take isl_space *space)
-{
-	int i, n;
-	isl_ctx *ctx;
-	ppcg_ht_bounds *bounds;
-
-	if (!space)
-		return NULL;
-
-	ctx = isl_space_get_ctx(space);
-	bounds = isl_alloc_type(ctx, struct ppcg_ht_bounds);
-	if (!bounds)
-		goto error;
-	bounds->upper = isl_val_nan(ctx);
-	bounds->lower = isl_multi_val_zero(space);
-	n = isl_multi_val_dim(bounds->lower, isl_dim_set);
-	for (i = 0; i < n; ++i) {
-		isl_val *v = isl_val_copy(bounds->upper);
-		bounds->lower = isl_multi_val_set_val(bounds->lower, i, v);
-	}
-
-	if (!bounds->lower || !bounds->upper)
-		return ppcg_ht_bounds_free(bounds);
-
-	return bounds;
-error:
-	isl_space_free(space);
-	return NULL;
-}
-
-void ppcg_ht_bounds_dump(__isl_keep ppcg_ht_bounds *bounds)
-{
-	if (!bounds)
-		return;
-
-	fprintf(stderr, "lower: ");
-	isl_multi_val_dump(bounds->lower);
-	fprintf(stderr, "upper: ");
-	isl_val_dump(bounds->upper);
-}
-
-/* Return the upper bound on the relative dependence distances
- * in the first space dimension.
- */
-__isl_give isl_val *ppcg_ht_bounds_get_upper(__isl_keep ppcg_ht_bounds *bounds)
-{
-	if (!bounds)
-		return NULL;
-	return isl_val_copy(bounds->upper);
-}
-
-/* Replace the upper bound on the relative dependence distances
- * in the first space dimension by "upper".
- */
-__isl_give ppcg_ht_bounds *ppcg_ht_bounds_set_upper(
-	__isl_take ppcg_ht_bounds *bounds, __isl_take isl_val *upper)
-{
-	if (!bounds || !upper)
-		goto error;
-	isl_val_free(bounds->upper);
-	bounds->upper = upper;
-	return bounds;
-error:
-	ppcg_ht_bounds_free(bounds);
-	isl_val_free(upper);
-	return NULL;
-}
-
-/* Return the lower bound on the relative dependence distances
- * in space dimension "pos".
- */
-__isl_give isl_val *ppcg_ht_bounds_get_lower(__isl_keep ppcg_ht_bounds *bounds,
-	int pos)
-{
-	if (!bounds)
-		return NULL;
-	return isl_multi_val_get_val(bounds->lower, pos);
-}
-
-/* Replace the lower bound on the relative dependence distances
- * in space dimension "pos" by "lower".
- */
-__isl_give ppcg_ht_bounds *ppcg_ht_bounds_set_lower(
-	__isl_take ppcg_ht_bounds *bounds, int pos, __isl_take isl_val *lower)
-{
-	if (!bounds || !lower)
-		goto error;
-	bounds->lower = isl_multi_val_set_val(bounds->lower, pos, lower);
-	if (!bounds->lower)
-		return ppcg_ht_bounds_free(bounds);
-	return bounds;
-error:
-	ppcg_ht_bounds_free(bounds);
-	isl_val_free(lower);
-	return NULL;
-}
-
-/* Can the bounds on relative dependence distances recorded in "bounds"
- * be used to perform hybrid tiling?
- * In particular, have appropriate lower and upper bounds been found?
- * Any NaN indicates that no corresponding bound was found.
- */
-isl_bool ppcg_ht_bounds_is_valid(__isl_keep ppcg_ht_bounds *bounds)
-{
-	isl_bool is_nan;
-	int i, n;
-
-	if (!bounds)
-		return isl_bool_error;
-	is_nan = isl_val_is_nan(bounds->upper);
-	if (is_nan < 0)
-		return isl_bool_error;
-	if (is_nan)
-		return isl_bool_false;
-
-	n = isl_multi_val_dim(bounds->lower, isl_dim_set);
-	for (i = 0; i < n; ++i) {
-		isl_val *v;
-
-		v = isl_multi_val_get_val(bounds->lower, i);
-		is_nan = isl_val_is_nan(v);
-		if (is_nan < 0)
-			return isl_bool_error;
-		if (is_nan)
-			return isl_bool_false;
-		isl_val_free(v);
-	}
-
-	return isl_bool_true;
-}
-
-/* Structure that represents the basic hexagonal tiling,
- * along with information that is needed to perform the hybrid tiling.
- *
- * "bounds" are the bounds on the dependence distances that
- * define the hexagonal shape and the required skewing in the remaining
- * space dimensions.
- *
- * "input_node" points to the input pair of band nodes.
- * "input_schedule" is the partial schedule of this input pair of band nodes.
- * The space of this schedule is [P -> C], where P is the space
- * of the parent node and C is the space of the child node.
- *
- * "space_sizes" represent the total size of a tile for the space
- * dimensions, i.e., those corresponding to the child node.
- * The space of "space_sizes" is C.
- * If S_0 is the original tile size in the first space dimension,
- * then the first entry of "space_sizes" is equal to
- * W = 2*S_0 + floor(d_l h) + floor(d_u h).
- * The remaining entries are the same as in the original tile sizes.
- *
- * The basic hexagonal tiling "hex" is defined
- * in a "ts" (time-space) space and corresponds to the phase-1 tiles.
- * "time_tile" maps the "ts" space to outer time tile.
- * Is is equal to ts[t, s] -> floor(t/(2 * S_t)), with S_t the original tile
- * size corresponding to the parent node.
- * "local_time" maps the "ts" space to the time dimension inside each tile.
- * It is equal to ts[t, s] -> t mod (2 S_t), with S_t the original tile
- * size corresponding to the parent node.
- * "shift_space" shifts the tiles at time tile T = floor(t/(2 S_t))
- * in the space dimension such that they align to a multiple of W.
- * It is equal to ts[t, s] -> s + (-(2 * shift_s)*T) % W,
- * with shift_s = S_0 + floor(d_u h).
- * "shift_phase" is the shift taken to go from phase 0 to phase 1
- * It is equal to ts[t, s] -> ts[t + S_t, s + shift_s],
- * with shift_s = S_0 + floor(d_u h).
- *
- * "project_ts" projects the space of the input schedule to the ts-space.
- * It is equal to [P[t] -> C[s_0, ...]] -> ts[t, s_0].
- */
-struct ppcg_ht_tiling {
-	int ref;
-
-	ppcg_ht_bounds *bounds;
-	isl_schedule_node *input_node;
-	isl_multi_union_pw_aff *input_schedule;
-
-	isl_multi_val *space_sizes;
-
-	isl_aff *time_tile;
-	isl_aff *local_time;
-	isl_aff *shift_space;
-	isl_multi_aff *shift_phase;
-	isl_set *hex;
-
-	isl_multi_aff *project_ts;
-};
-typedef struct ppcg_ht_tiling ppcg_ht_tiling;
-
-/* Return the space of the pair of band nodes that form the input
- * to the hybrid tiling.
- * In particular, return the space [P -> C], where P is the space
- * of the parent node and C is the space of the child node.
- */
-__isl_give isl_space *ppcg_ht_tiling_get_input_space(
-	__isl_keep ppcg_ht_tiling *tile)
-{
-	if (!tile)
-		return NULL;
-
-	return isl_multi_union_pw_aff_get_space(tile->input_schedule);
-}
-
-/* Remove a reference to "tile" and free "tile" along with all its fields
- * as soon as the reference count drops to zero.
- */
-static __isl_null ppcg_ht_tiling *ppcg_ht_tiling_free(
-	__isl_take ppcg_ht_tiling *tiling)
-{
-	if (!tiling)
-		return NULL;
-	if (--tiling->ref > 0)
-		return NULL;
-
-	ppcg_ht_bounds_free(tiling->bounds);
-	isl_schedule_node_free(tiling->input_node);
-	isl_multi_union_pw_aff_free(tiling->input_schedule);
-	isl_multi_val_free(tiling->space_sizes);
-	isl_aff_free(tiling->time_tile);
-	isl_aff_free(tiling->local_time);
-	isl_aff_free(tiling->shift_space);
-	isl_multi_aff_free(tiling->shift_phase);
-	isl_set_free(tiling->hex);
-	isl_multi_aff_free(tiling->project_ts);
-	free(tiling);
-
-	return NULL;
-}
-
-/* Return a new reference to "tiling".
- */
-__isl_give ppcg_ht_tiling *ppcg_ht_tiling_copy(
-	__isl_keep ppcg_ht_tiling *tiling)
-{
-	if (!tiling)
-		return NULL;
-
-	tiling->ref++;
-	return tiling;
-}
-
-/* Return the isl_ctx to which "tiling" belongs.
- */
-isl_ctx *ppcg_ht_tiling_get_ctx(__isl_keep ppcg_ht_tiling *tiling)
-{
-	if (!tiling)
-		return NULL;
-
-	return isl_multi_union_pw_aff_get_ctx(tiling->input_schedule);
-}
-
-/* Representation of one of the two phases of hybrid tiling.
- *
- * "tiling" points to the shared tiling data.
- *
- * "time_tile", "local_time" and "shift_space" are equal to the corresponding
- * fields of "tiling", pulled back to the input space.
- * In case of phase 0, these expressions have also been moved
- * from phase 1 to phase 0.
- *
- * "domain" contains the hexagonal tiling of this phase.
- *
- * "space_shift" is the shift that should be added to the space band
- * in order to be able to apply rectangular tiling to the space.
- * For phase 1, it is equal to
- *
- *	[P[t] -> C[s_0, s_i]] -> C[(-(2 * shift_s)*T) % W, dl_i * u]
- *
- * with shift_s = S_0 + floor(d_u h),
- * T equal to "time_tile" and u equal to "local_time".
- * For phase 0, it is equal to
- *
- *	[P[t] -> C[s_0, s_i]] -> C[shift_s + (-(2 * shift_s)*T) % W, dl_i * u]
- *
- * "space_tile" is the space tiling.  It is equal to
- *
- *	[P[t] -> C[s]] -> C[floor((s + space_shift)/space_size]
- */
-struct ppcg_ht_phase {
-	ppcg_ht_tiling *tiling;
-
-	isl_aff *time_tile;
-	isl_aff *local_time;
-	isl_aff *shift_space;
-	isl_set *domain;
-
-	isl_multi_aff *space_shift;
-	isl_multi_aff *space_tile;
-};
-
-/* Free "phase" along with all its fields.
- */
-static __isl_null ppcg_ht_phase *ppcg_ht_phase_free(
-	__isl_take ppcg_ht_phase *phase)
-{
-	if (!phase)
-		return NULL;
-
-	ppcg_ht_tiling_free(phase->tiling);
-	isl_aff_free(phase->time_tile);
-	isl_aff_free(phase->local_time);
-	isl_aff_free(phase->shift_space);
-	isl_set_free(phase->domain);
-	isl_multi_aff_free(phase->space_shift);
-	isl_multi_aff_free(phase->space_tile);
-	free(phase);
-
-	return NULL;
-}
-
-/* Wrapper around ppcg_ht_phase_free for use as an argument
- * to isl_id_set_free_user.
- */
-static void ppcg_ht_phase_free_wrap(void *user)
-{
-	ppcg_ht_phase *phase = user;
-
-	ppcg_ht_phase_free(phase);
-}
-
-/* Return the domain of hybrid tiling phase "phase".
- */
-static __isl_give isl_set *ppcg_ht_phase_get_domain(ppcg_ht_phase *phase)
-{
-	if (!phase)
-		return NULL;
-
-	return isl_set_copy(phase->domain);
-}
-
-/* Return the space of the pair of band nodes that form the input
- * to the hybrid tiling of which "phase" is a phase.
- * In particular, return the space [P -> C], where P is the space
- * of the parent node and C is the space of the child node.
- */
-static __isl_give isl_space *ppcg_ht_phase_get_input_space(
-	__isl_keep ppcg_ht_phase *phase)
-{
-	if (!phase)
-		return NULL;
-
-	return ppcg_ht_tiling_get_input_space(phase->tiling);
-}
-
-/* Construct the lower left constraint of the hexagonal tile, i.e.,
- *
- *	du a - b <= (2h+1) du - duh
- *	-du a + b + (2h+1) du - duh >= 0
- *
- * where duh = floor(du * h).
- *
- * This constraint corresponds to (6) in
- * "Hybrid Hexagonal/Classical Tiling for GPUs".
- */
-static __isl_give isl_constraint *hex_lower_left(__isl_take isl_local_space *ls,
-	__isl_keep isl_val *h, __isl_keep isl_val *du, __isl_keep isl_val *duh)
-{
-	isl_val *v;
-	isl_aff *aff;
-
-	v = isl_val_add_ui(isl_val_mul_ui(isl_val_copy(h), 2), 1);
-	v = isl_val_mul(v, isl_val_copy(du));
-	v = isl_val_sub(v, isl_val_copy(duh));
-	aff = isl_aff_val_on_domain(ls, v);
-	v = isl_val_neg(isl_val_copy(du));
-	aff = isl_aff_set_coefficient_val(aff, isl_dim_in, 0, v);
-	aff = isl_aff_set_coefficient_si(aff, isl_dim_in, 1, 1);
-
-	return isl_inequality_from_aff(aff);
-}
-
-/* Construct the lower constraint of the hexagonal tile, i.e.,
- *
- *	a <= 2h+1
- *	-a + 2h+1 >= 0
- *
- * This constraint corresponds to (7) in
- * "Hybrid Hexagonal/Classical Tiling for GPUs".
- */
-static __isl_give isl_constraint *hex_lower(__isl_take isl_local_space *ls,
-	__isl_keep isl_val *h)
-{
-	isl_val *v;
-	isl_aff *aff;
-
-	v = isl_val_add_ui(isl_val_mul_ui(isl_val_copy(h), 2), 1);
-	aff = isl_aff_val_on_domain(ls, v);
-	aff = isl_aff_set_coefficient_si(aff, isl_dim_in, 0, -1);
-
-	return isl_inequality_from_aff(aff);
-}
-
-/* Construct the lower right constraint of the hexagonal tile, i.e.,
- *
- *	dl a + b <= (2h+1) dl + duh + (s0-1)
- *	-dl a - b + (2h+1) dl + duh + (s0-1) >= 0
- *
- * where duh = floor(du * h).
- *
- * This constraint corresponds to (8) in
- * "Hybrid Hexagonal/Classical Tiling for GPUs".
- */
-static __isl_give isl_constraint *hex_lower_right(
-	__isl_take isl_local_space *ls, __isl_keep isl_val *h,
-	__isl_keep isl_val *s0, __isl_keep isl_val *dl, __isl_keep isl_val *duh)
-{
-	isl_val *v;
-	isl_aff *aff;
-
-	v = isl_val_add_ui(isl_val_mul_ui(isl_val_copy(h), 2), 1);
-	v = isl_val_mul(v, isl_val_copy(dl));
-	v = isl_val_add(v, isl_val_copy(duh));
-	v = isl_val_add(v, isl_val_copy(s0));
-	v = isl_val_sub_ui(v, 1);
-	aff = isl_aff_val_on_domain(ls, v);
-	v = isl_val_neg(isl_val_copy(dl));
-	aff = isl_aff_set_coefficient_val(aff, isl_dim_in, 0, v);
-	aff = isl_aff_set_coefficient_si(aff, isl_dim_in, 1, -1);
-
-	return isl_inequality_from_aff(aff);
-}
-
-/* Construct the upper left constraint of the hexagonal tile, i.e.,
- *
- *	dl a + b >= h dl - (d - 1)/d				with d = den(dl)
- *	dl a + b - h dl + (d - 1)/d >= 0
- *
- * This constraint corresponds to (10) in
- * "Hybrid Hexagonal/Classical Tiling for GPUs".
- */
-static __isl_give isl_constraint *hex_upper_left(__isl_take isl_local_space *ls,
-	__isl_keep isl_val *h, __isl_keep isl_val *dl)
-{
-	isl_val *v, *d;
-	isl_aff *aff;
-
-	d = isl_val_get_den_val(dl);
-	v = isl_val_sub_ui(isl_val_copy(d), 1);
-	v = isl_val_div(v, d);
-	v = isl_val_sub(v, isl_val_mul(isl_val_copy(h), isl_val_copy(dl)));
-	aff = isl_aff_val_on_domain(ls, v);
-	aff = isl_aff_set_coefficient_val(aff, isl_dim_in, 0, isl_val_copy(dl));
-	aff = isl_aff_set_coefficient_si(aff, isl_dim_in, 1, 1);
-
-	return isl_inequality_from_aff(aff);
-}
-
-/* Construct the upper right constraint of the hexagonal tile, i.e.,
- *
- *	du a - b >= du h - duh - (s0-1) - dlh - (d - 1)/d	with d = den(du)
- *	du a - b - du h + duh + (s0-1) + dlh + (d - 1)/d >= 0
- *
- * where dlh = floor(dl * h) and duh = floor(du * h).
- *
- * This constraint corresponds to (12) in
- * "Hybrid Hexagonal/Classical Tiling for GPUs".
- */
-static __isl_give isl_constraint *hex_upper_right(
-	__isl_take isl_local_space *ls, __isl_keep isl_val *h,
-	__isl_keep isl_val *s0, __isl_keep isl_val *du,
-	__isl_keep isl_val *dlh, __isl_keep isl_val *duh)
-{
-	isl_val *v, *d;
-	isl_aff *aff;
-
-	d = isl_val_get_den_val(du);
-	v = isl_val_sub_ui(isl_val_copy(d), 1);
-	v = isl_val_div(v, d);
-	v = isl_val_sub(v, isl_val_mul(isl_val_copy(h), isl_val_copy(du)));
-	v = isl_val_add(v, isl_val_copy(duh));
-	v = isl_val_add(v, isl_val_copy(dlh));
-	v = isl_val_add(v, isl_val_copy(s0));
-	v = isl_val_sub_ui(v, 1);
-	aff = isl_aff_val_on_domain(ls, v);
-	aff = isl_aff_set_coefficient_val(aff, isl_dim_in, 0, isl_val_copy(du));
-	aff = isl_aff_set_coefficient_si(aff, isl_dim_in, 1, -1);
-
-	return isl_inequality_from_aff(aff);
-}
-
-/* Construct the uppper constraint of the hexagonal tile, i.e.,
- *
- *	a >= 0
- *
- * This constraint corresponds to (13) in
- * "Hybrid Hexagonal/Classical Tiling for GPUs".
- */
-static __isl_give isl_constraint *hex_upper(__isl_take isl_local_space *ls)
-{
-	isl_aff *aff;
-
-	aff = isl_aff_var_on_domain(ls, isl_dim_set, 0);
-
-	return isl_inequality_from_aff(aff);
-}
-
-/* Construct the basic hexagonal tile shape.
- * "space" is the 2D space in which the hexagon should be constructed.
- * h is st-1, with st the tile size in the time dimension
- * s0 is the tile size in the space dimension
- * dl is a bound on the negative relative dependence distances, i.e.,
- *
- *	d_s >= -dl d_t
- *
- * du is a bound on the positive relative dependence distances, i.e.,
- *
- *	d_s <= du d_t
- *
- * with (d_t,d_s) any dependence distance vector.
- * dlh = floor(dl * h)
- * duh = floor(du * h)
- *
- * The shape of the hexagon is as follows:
- *
- *		0 dlh   dlh+s0-1
- *		   ______                __
- * 0		  /      \_             /
- *		 /         \_          /
- * h		/            \ ______ /
- * h+1		\_           //      \\_
- *		  \_        //         \\_
- * 2h+1		    \______//            \\
- *		0   duh   duh+s0-1
- *		             duh+s0-1+dlh
- *		                  duh+s0-1+dlh+1+s0+1
- *
- * The next hexagon is shifted by duh + dlh + 2 * s0.
- *
- * The slope of the "/" constraints is dl.
- * The slope of the "\_" constraints is du.
- */
-static __isl_give isl_set *compute_hexagon(__isl_take isl_space *space,
-	__isl_keep isl_val *h, __isl_keep isl_val *s0,
-	__isl_keep isl_val *dl, __isl_keep isl_val *du,
-	__isl_keep isl_val *dlh, __isl_keep isl_val *duh)
-{
-	isl_local_space *ls;
-	isl_constraint *c;
-	isl_basic_set *bset;
-
-	ls = isl_local_space_from_space(space);
-
-	c = hex_lower_left(isl_local_space_copy(ls), h, du, duh);
-	bset = isl_basic_set_from_constraint(c);
-
-	c = hex_lower(isl_local_space_copy(ls), h);
-	bset = isl_basic_set_add_constraint(bset, c);
-
-	c = hex_lower_right(isl_local_space_copy(ls), h, s0, dl, duh);
-	bset = isl_basic_set_add_constraint(bset, c);
-
-	c = hex_upper_left(isl_local_space_copy(ls), h, dl);
-	bset = isl_basic_set_add_constraint(bset, c);
-
-	c = hex_upper_right(isl_local_space_copy(ls), h, s0, du, dlh, duh);
-	bset = isl_basic_set_add_constraint(bset, c);
-
-	c = hex_upper(ls);
-	bset = isl_basic_set_add_constraint(bset, c);
-
-	return isl_set_from_basic_set(bset);
-}
-
-/* Name of the ts-space.
- */
-static const char *ts_space_name = "ts";
-
-/* Construct and return the space ts[t, s].
- */
-static __isl_give isl_space *construct_ts_space(isl_ctx *ctx)
-{
-	isl_space *s;
-
-	s = isl_space_set_alloc(ctx, 0, 2);
-	s = isl_space_set_tuple_name(s, isl_dim_set, ts_space_name);
-
-	return s;
-}
-
-/* Name of the local ts-space.
- */
-static const char *local_ts_space_name = "local_ts";
-
-/* Construct and return the space local_ts[t, s].
- */
-static __isl_give isl_space *construct_local_ts_space(isl_ctx *ctx)
-{
-	isl_space *s;
-
-	s = isl_space_set_alloc(ctx, 0, 2);
-	s = isl_space_set_tuple_name(s, isl_dim_set, local_ts_space_name);
-
-	return s;
-}
-
-/* Compute the total size of a tile for the space dimensions,
- * i.e., those corresponding to the child node
- * of the input pattern.
- * If S_0 is the original tile size in the first space dimension,
- * then the first entry of "space_sizes" is equal to
- * W = 2*S_0 + floor(d_l h) + floor(d_u h).
- * The remaining entries are the same as in the original tile sizes.
- * "tile_sizes" contains the original tile sizes, including
- * the tile size corresponding to the parent node.
- * "dlh" is equal to floor(d_l h).
- * "duh" is equal to floor(d_u h).
- */
-static __isl_give isl_multi_val *compute_space_sizes(
-	__isl_keep isl_multi_val *tile_sizes,
-	__isl_keep isl_val *dlh, __isl_keep isl_val *duh)
-{
-	isl_val *size;
-	isl_multi_val *space_sizes;
-
-	space_sizes = isl_multi_val_copy(tile_sizes);
-	space_sizes = isl_multi_val_factor_range(space_sizes);
-	size = isl_multi_val_get_val(space_sizes, 0);
-	size = isl_val_mul_ui(size, 2);
-	size = isl_val_add(size, isl_val_copy(duh));
-	size = isl_val_add(size, isl_val_copy(dlh));
-	space_sizes = isl_multi_val_set_val(space_sizes, 0, size);
-
-	return space_sizes;
-}
-
-/* Compute the offset of phase 1 with respect to phase 0
- * in the ts-space ("space").
- * In particular, return
- *
- *	ts[st, s0 + duh]
- */
-static __isl_give isl_multi_val *compute_phase_shift(
-	__isl_keep isl_space *space, __isl_keep isl_val *st,
-	__isl_keep isl_val *s0, __isl_keep isl_val *duh)
-{
-	isl_val *v;
-	isl_multi_val *phase_shift;
-
-	phase_shift = isl_multi_val_zero(isl_space_copy(space));
-	phase_shift = isl_multi_val_set_val(phase_shift, 0, isl_val_copy(st));
-	v = isl_val_add(isl_val_copy(duh), isl_val_copy(s0));
-	phase_shift = isl_multi_val_set_val(phase_shift, 1, v);
-
-	return phase_shift;
-}
-
-/* Return the function
- *
- *	ts[t, s] -> floor(t/(2 * st))
- *
- * representing the time tile.
- * "space" is the space ts[t, s].
- */
-static __isl_give isl_aff *compute_time_tile(__isl_keep isl_space *space,
-	__isl_keep isl_val *st)
-{
-	isl_val *v;
-	isl_aff *t;
-	isl_local_space *ls;
-
-	ls = isl_local_space_from_space(isl_space_copy(space));
-	t = isl_aff_var_on_domain(ls, isl_dim_set, 0);
-	v = isl_val_mul_ui(isl_val_copy(st), 2);
-	t = isl_aff_floor(isl_aff_scale_down_val(t, v));
-
-	return t;
-}
-
-/* Compute a shift in the space dimension for tiles
- * at time tile T = floor(t/(2 * S_t))
- * such that they align to a multiple of the total space tile dimension W.
- * In particular, compute
- *
- *	ts[t, s] -> s + (-(2 * shift_s)*T) % W
- *
- * where shift_s is the shift of phase 1 with respect to phase 0
- * in the space dimension (the first element of "phase_shift").
- * W is stored in the first element of "space_sizes".
- * "time_tile" is the function
- *
- *	ts[t, s] -> floor(t/(2 * S_T))
- *
- * Since phase 1 is shifted by shift_s with respect to phase 0,
- * the next line of phase 0 (at T+1) is shifted by 2*shift_s
- * with respect to the previous line (at T).
- * A shift of -(2 * shift_s)*T therefore allows the basic pattern
- * (which starts at 0) to be applied.
- * However, this shift will be used to obtain the tile coordinate
- * in the first space dimension and if the original values
- * in the space dimension are non-negative, then the shift should
- * not make them negative.  Moreover, the shift should be as minimal
- * as possible.
- * Since the pattern repeats itself with a period of W in the space
- * dimension, the shift can be replaced by (-(2 * shift_s)*T) % W.
- */
-static __isl_give isl_aff *compute_shift_space(__isl_keep isl_aff *time_tile,
-	__isl_keep isl_multi_val *space_sizes,
-	__isl_keep isl_multi_val *phase_shift)
-{
-	isl_val *v;
-	isl_aff *s, *t;
-	isl_local_space *ls;
-
-	ls = isl_local_space_from_space(isl_aff_get_domain_space(time_tile));
-	t = isl_aff_copy(time_tile);
-	v = isl_val_mul_ui(isl_multi_val_get_val(phase_shift, 1), 2);
-	v = isl_val_neg(v);
-	t = isl_aff_scale_val(t, v);
-	v = isl_multi_val_get_val(space_sizes, 0);
-	t = isl_aff_mod_val(t, v);
-	s = isl_aff_var_on_domain(ls, isl_dim_set, 1);
-	s = isl_aff_add(s, t);
-
-	return s;
-}
-
-/* Give the phase_shift ts[S_t, S_0 + floor(d_u h)],
- * compute a function that applies the shift, i.e.,
- *
- *	ts[t, s] -> ts[t + S_t, s + S_0 + floor(d_u h)],
- */
-static __isl_give isl_multi_aff *compute_shift_phase(
-	__isl_keep isl_multi_val *phase_shift)
-{
-	isl_space *space;
-	isl_multi_aff *shift;
-
-	space = isl_multi_val_get_space(phase_shift);
-	shift = isl_multi_aff_multi_val_on_space(space,
-					isl_multi_val_copy(phase_shift));
-	space = isl_multi_aff_get_space(shift);
-	shift = isl_multi_aff_add(shift, isl_multi_aff_identity(space));
-
-	return shift;
-}
-
-/* Compute a mapping from the ts-space to the local coordinates
- * within each tile.  In particular, compute
- *
- *	ts[t, s] -> local_ts[t % (2 S_t), (s + (-(2 * shift_s)*T) % W) % W]
- *
- * "ts" is the space ts[t, s]
- * "local_ts" is the space local_ts[t, s]
- * "shift_space" is equal to ts[t, s] -> s + (-(2 * shift_s)*T) % W
- * "st" is the tile size in the time dimension S_t.
- * The first element of "space_sizes" is equal to W.
- */
-static __isl_give isl_multi_aff *compute_localize(
-	__isl_keep isl_space *local_ts, __isl_keep isl_aff *shift_space,
-	__isl_keep isl_val *st, __isl_keep isl_multi_val *space_sizes)
-{
-	isl_val *v;
-	isl_space *space;
-	isl_aff *s, *t;
-	isl_multi_aff *localize;
-
-	space = isl_aff_get_domain_space(shift_space);
-	local_ts = isl_space_copy(local_ts);
-	space = isl_space_map_from_domain_and_range(space, local_ts);
-	localize = isl_multi_aff_identity(space);
-	t = isl_multi_aff_get_aff(localize, 0);
-	v = isl_val_mul_ui(isl_val_copy(st), 2);
-	t = isl_aff_mod_val(t, v);
-	localize = isl_multi_aff_set_aff(localize, 0, t);
-	s = isl_aff_copy(shift_space);
-	v = isl_multi_val_get_val(space_sizes, 0);
-	s = isl_aff_mod_val(s, v);
-	localize = isl_multi_aff_set_aff(localize, 1, s);
-
-	return localize;
-}
-
-/* Set the project_ts field of "tiling".
- *
- * This field projects the space of the input schedule to the ts-space.
- * It is equal to [P[t] -> C[s_0, ...]] -> ts[t, s_0].
- */
-static __isl_give ppcg_ht_tiling *ppcg_ht_tiling_set_project_ts(
-	__isl_take ppcg_ht_tiling *tiling)
-{
-	int n;
-	isl_space *space;
-	isl_multi_aff *project;
-
-	if (!tiling)
-		return NULL;
-
-	space = ppcg_ht_tiling_get_input_space(tiling);
-	n = isl_space_dim(space, isl_dim_set);
-	project = isl_multi_aff_project_out_map(space, isl_dim_set, 2, n - 2);
-	project = isl_multi_aff_set_tuple_name(project,
-						isl_dim_out, ts_space_name);
-	if (!project)
-		return ppcg_ht_tiling_free(tiling);
-
-	tiling->project_ts = project;
-
-	return tiling;
-}
-
-/* Construct a hybrid tiling description from bounds on the dependence
- * distances "bounds".
- * "input_node" points to the original parent node.
- * "input_schedule" is the combined schedule of the parent and child
- * node in the input.
- * "tile_sizes" are the original, user specified tile sizes.
- */
-static __isl_give ppcg_ht_tiling *ppcg_ht_bounds_construct_tiling(
-	__isl_take ppcg_ht_bounds *bounds,
-	__isl_keep isl_schedule_node *input_node,
-	__isl_keep isl_multi_union_pw_aff *input_schedule,
-	__isl_keep isl_multi_val *tile_sizes)
-{
-	isl_ctx *ctx;
-	ppcg_ht_tiling *tiling;
-	isl_multi_val *space_sizes, *phase_shift;
-	isl_aff *time_tile, *shift_space;
-	isl_multi_aff *localize;
-	isl_val *h, *duh, *dlh;
-	isl_val *st, *s0, *du, *dl;
-	isl_space *ts, *local_ts;
-
-	if (!bounds || !input_node || !input_schedule || !tile_sizes)
-		goto error;
-
-	ctx = isl_multi_union_pw_aff_get_ctx(input_schedule);
-	tiling = isl_calloc_type(ctx, struct ppcg_ht_tiling);
-	if (!tiling)
-		goto error;
-	tiling->ref = 1;
-
-	st = isl_multi_val_get_val(tile_sizes, 0);
-	h = isl_val_sub_ui(isl_val_copy(st), 1);
-	s0 = isl_multi_val_get_val(tile_sizes, 1);
-	du = ppcg_ht_bounds_get_upper(bounds);
-	dl = ppcg_ht_bounds_get_lower(bounds, 0);
-
-	duh = isl_val_floor(isl_val_mul(isl_val_copy(du), isl_val_copy(h)));
-	dlh = isl_val_floor(isl_val_mul(isl_val_copy(dl), isl_val_copy(h)));
-
-	ts = construct_ts_space(ctx);
-	local_ts = construct_local_ts_space(ctx);
-
-	space_sizes = compute_space_sizes(tile_sizes, dlh, duh);
-	phase_shift = compute_phase_shift(ts, st, s0, duh);
-	time_tile = compute_time_tile(ts, st);
-	shift_space = compute_shift_space(time_tile, space_sizes, phase_shift);
-	localize = compute_localize(local_ts, shift_space, st, space_sizes);
-	isl_space_free(ts);
-
-	tiling->input_node = isl_schedule_node_copy(input_node);
-	tiling->input_schedule = isl_multi_union_pw_aff_copy(input_schedule);
-	tiling->space_sizes = space_sizes;
-	tiling->bounds = bounds;
-	tiling->local_time = isl_multi_aff_get_aff(localize, 0);
-	tiling->hex = compute_hexagon(local_ts, h, s0, dl, du, dlh, duh);
-	tiling->hex = isl_set_preimage_multi_aff(tiling->hex, localize);
-	tiling->time_tile = time_tile;
-	tiling->shift_space = shift_space;
-	tiling->shift_phase = compute_shift_phase(phase_shift);
-	isl_multi_val_free(phase_shift);
-
-	isl_val_free(duh);
-	isl_val_free(dlh);
-	isl_val_free(du);
-	isl_val_free(dl);
-	isl_val_free(s0);
-	isl_val_free(st);
-	isl_val_free(h);
-
-	if (!tiling->input_schedule || !tiling->local_time || !tiling->hex ||
-	    !tiling->shift_space || !tiling->shift_phase)
-		return ppcg_ht_tiling_free(tiling);
-
-	tiling = ppcg_ht_tiling_set_project_ts(tiling);
-
-	return tiling;
-error:
-	ppcg_ht_bounds_free(bounds);
-	return NULL;
-}
-
-/* Are all members of the band node "node" coincident?
- */
-static isl_bool all_coincident(__isl_keep isl_schedule_node *node)
-{
-	int i, n;
-
-	n = isl_schedule_node_band_n_member(node);
-	for (i = 0; i < n; ++i) {
-		isl_bool c;
-
-		c = isl_schedule_node_band_member_get_coincident(node, i);
-		if (c < 0 || !c)
-			return c;
-	}
-
-	return isl_bool_true;
-}
-
-/* Does "node" satisfy the properties of the inner node in the input
- * pattern for hybrid tiling?
- * That is, is it a band node with only coincident members, of which
- * there is at least one?
- */
-static isl_bool has_child_properties(__isl_keep isl_schedule_node *node)
-{
-	if (!node)
-		return isl_bool_error;
-	if (isl_schedule_node_get_type(node) != isl_schedule_node_band)
-		return isl_bool_false;
-	if (isl_schedule_node_band_n_member(node) < 1)
-		return isl_bool_false;
-	return all_coincident(node);
-}
-
-/* Does "node" satisfy the properties of the outer node in the input
- * pattern for hybrid tiling?
- * That is, is it a band node with a single member?
- */
-static isl_bool has_parent_properties(__isl_keep isl_schedule_node *node)
-{
-	if (!node)
-		return isl_bool_error;
-	if (isl_schedule_node_get_type(node) != isl_schedule_node_band)
-		return isl_bool_false;
-	if (isl_schedule_node_band_n_member(node) != 1)
-		return isl_bool_false;
-	return isl_bool_true;
-}
-
-/* Does the parent of "node" satisfy the input patttern for hybrid tiling?
- * That is, does "node" satisfy the properties of the inner node and
- * does the parent of "node" satisfy the properties of the outer node?
- */
-isl_bool ppcg_ht_parent_has_input_pattern(__isl_keep isl_schedule_node *node)
-{
-	isl_bool has_pattern;
-
-	has_pattern = has_child_properties(node);
-	if (has_pattern < 0 || !has_pattern)
-		return has_pattern;
-
-	node = isl_schedule_node_copy(node);
-	node = isl_schedule_node_parent(node);
-	has_pattern = has_parent_properties(node);
-	isl_schedule_node_free(node);
-
-	return has_pattern;
-}
-
-/* Does "node" satisfy the input patttern for hybrid tiling?
- * That is, does "node" satisfy the properties of the outer node and
- * does the child of "node" satisfy the properties of the inner node?
- */
-isl_bool ppcg_ht_has_input_pattern(__isl_keep isl_schedule_node *node)
-{
-	isl_bool has_pattern;
-
-	has_pattern = has_parent_properties(node);
-	if (has_pattern < 0 || !has_pattern)
-		return has_pattern;
-
-	node = isl_schedule_node_get_child(node, 0);
-	has_pattern = has_child_properties(node);
-	isl_schedule_node_free(node);
-
-	return has_pattern;
-}
-
-/* Check that "node" satisfies the input pattern for hybrid tiling.
- * Error out if it does not.
- */
-static isl_stat check_input_pattern(__isl_keep isl_schedule_node *node)
-{
-	isl_bool has_pattern;
-
-	has_pattern = ppcg_ht_has_input_pattern(node);
-	if (has_pattern < 0)
-		return isl_stat_error;
-	if (!has_pattern)
-		isl_die(isl_schedule_node_get_ctx(node), isl_error_invalid,
-			"invalid input pattern for hybrid tiling",
-			return isl_stat_error);
-
-	return isl_stat_ok;
-}
-
-/* Extract the input schedule from "node", i.e., the product
- * of the partial schedules of the parent and child nodes
- * in the input pattern.
- */
-static __isl_give isl_multi_union_pw_aff *extract_input_schedule(
-	__isl_keep isl_schedule_node *node)
-{
-	isl_multi_union_pw_aff *partial, *partial2;
-
-	partial = isl_schedule_node_band_get_partial_schedule(node);
-	node = isl_schedule_node_get_child(node, 0);
-	partial2 = isl_schedule_node_band_get_partial_schedule(node);
-	isl_schedule_node_free(node);
-
-	return isl_multi_union_pw_aff_range_product(partial, partial2);
-}
-
-/* Collect all dependences from "scop" that are relevant for performing
- * hybrid tiling on "node" and its child and map them to the schedule
- * space of this pair of nodes.
- *
- * In case live range reordering is not used,
- * the flow and the false dependences are collected.
- * In case live range reordering is used,
- * the flow and the forced dependences are collected, as well
- * as the order dependences that are adjacent to non-local
- * flow dependences.
- *
- * In all cases, only dependences that map to the same instance
- * of the outer part of the schedule are considered.
- */
-static __isl_give isl_map *collect_deps(struct ppcg_scop *scop,
-	__isl_keep isl_schedule_node *node)
-{
-	isl_space *space;
-	isl_multi_union_pw_aff *prefix, *partial;
-	isl_union_map *flow, *other, *dep, *umap;
-	isl_map *map;
-
-	prefix = isl_schedule_node_get_prefix_schedule_multi_union_pw_aff(node);
-	partial = extract_input_schedule(node);
-	space = isl_multi_union_pw_aff_get_space(partial);
-
-	flow = isl_union_map_copy(scop->dep_flow);
-	flow = isl_union_map_eq_at_multi_union_pw_aff(flow,
-					isl_multi_union_pw_aff_copy(prefix));
-	if (!scop->options->live_range_reordering) {
-		other = isl_union_map_copy(scop->dep_false);
-		other = isl_union_map_eq_at_multi_union_pw_aff(other, prefix);
-	} else {
-		isl_union_map *local, *non_local, *order, *adj;
-		isl_union_set *domain, *range;
-
-		other = isl_union_map_copy(scop->dep_forced);
-		other = isl_union_map_eq_at_multi_union_pw_aff(other,
-					isl_multi_union_pw_aff_copy(prefix));
-		local = isl_union_map_copy(flow);
-		local = isl_union_map_eq_at_multi_union_pw_aff(local,
-					isl_multi_union_pw_aff_copy(partial));
-		non_local = isl_union_map_copy(flow);
-		non_local = isl_union_map_subtract(non_local, local);
-
-		order = isl_union_map_copy(scop->dep_order);
-		order = isl_union_map_eq_at_multi_union_pw_aff(order, prefix);
-		adj = isl_union_map_copy(order);
-		domain = isl_union_map_domain(isl_union_map_copy(non_local));
-		domain = isl_union_set_coalesce(domain);
-		adj = isl_union_map_intersect_range(adj, domain);
-		other = isl_union_map_union(other, adj);
-
-		adj = order;
-		range = isl_union_map_range(non_local);
-		range = isl_union_set_coalesce(range);
-		adj = isl_union_map_intersect_domain(adj, range);
-		other = isl_union_map_union(other, adj);
-	}
-	dep = isl_union_map_union(flow, other);
-
-	umap = isl_union_map_from_multi_union_pw_aff(partial);
-	dep = isl_union_map_apply_domain(dep, isl_union_map_copy(umap));
-	dep = isl_union_map_apply_range(dep, umap);
-
-	space = isl_space_map_from_set(space);
-	map = isl_union_map_extract_map(dep, space);
-	isl_union_map_free(dep);
-
-	map = isl_map_coalesce(map);
-
-	return map;
-}
-
-/* Given a constraint of the form
- *
- *	a i_0 + b i_1 >= 0
- * or
- *	a i_0 + b i_1 = 0
- *
- * use it to update one or both of the non-negative bounds
- * in "list" = (min, max) such that
- *
- *	i_1 >= -min i_0
- * and
- *	i_1 <= max i_0
- *
- * If b = 0, then the constraint cannot be used.
- * Otherwise, the constraint is equivalent to
- *
- *	sgn(b) i_1 >= - a/abs(b) i_0
- * i.e.,
- *	i_1 >= - a/abs(b) i_0
- * or
- *	i_1 <= a/abs(b) i_0
- *
- * Set the first or second element of "list" to max(0, a/abs(b)),
- * according to the sign of "b".  Or set both in case the constraint
- * is an equality, taking into account the sign change.
- */
-static __isl_give isl_val_list *list_set_min_max(__isl_take isl_val_list *list,
-	__isl_keep isl_constraint *c)
-{
-	isl_val *a, *b;
-	int sign;
-	int pos;
-	isl_bool eq, is_zero, is_neg;
-
-	eq = isl_constraint_is_equality(c);
-	if (eq < 0)
-		return isl_val_list_free(list);
-
-	b = isl_constraint_get_coefficient_val(c, isl_dim_set, 1);
-	is_zero = isl_val_is_zero(b);
-	if (is_zero == isl_bool_true) {
-		isl_val_free(b);
-		return list;
-	}
-	a = isl_constraint_get_coefficient_val(c, isl_dim_set, 0);
-	sign = isl_val_sgn(b);
-	b = isl_val_abs(b);
-	a = isl_val_div(a, b);
-
-	if (eq)
-		b = isl_val_copy(a);
-
-	pos = sign > 0 ? 0 : 1;
-	is_neg = isl_val_is_neg(a);
-	if (is_neg == isl_bool_true)
-		a = isl_val_set_si(a, 0);
-	list = isl_val_list_set_val(list, pos, a);
-
-	if (!eq)
-		return is_neg < 0 ? isl_val_list_free(list) : list;
-
-	pos = 1 - pos;
-	a = isl_val_neg(b);
-	is_neg = isl_val_is_neg(a);
-	if (is_neg == isl_bool_true)
-		a = isl_val_set_si(a, 0);
-	list = isl_val_list_set_val(list, pos, a);
-
-	return is_neg < 0 ? isl_val_list_free(list) : list;
-}
-
-/* If constraint "c" passes through the origin, then try and use it
- * to update the non-negative bounds in "list" = (min, max) such that
- *
- *	i_1 >= -min i_0
- * and
- *	i_1 <= max i_0
- */
-static isl_stat set_min_max(__isl_take isl_constraint *c, void *user)
-{
-	isl_val *v;
-	isl_val_list **list = user;
-	isl_bool is_zero;
-
-	v = isl_constraint_get_constant_val(c);
-	is_zero = isl_val_is_zero(v);
-	isl_val_free(v);
-
-	if (is_zero == isl_bool_true)
-		*list = list_set_min_max(*list, c);
-
-	isl_constraint_free(c);
-	return is_zero < 0 ? isl_stat_error : isl_stat_ok;
-}
-
-/* Given a set of dependence distance vectors "dist", compute
- * pair of non-negative bounds min and max such that
- *
- *	d_pos >= -min d_0
- * and
- *	d_pos <= max d_0
- *
- * and return the pair (min, max).
- * If no bound can be found in either direction, then the bound
- * is replaced by NaN.
- *
- * The dependence distances are first projected onto the (d_0, d_pos).
- * Then the zero dependence distance is added and the convex hull is computed.
- * Finally, the bounds are extracted from the constraints of the convex hull
- * that pass through the origin.
- */
-static __isl_give isl_val_list *min_max_dist(__isl_keep isl_set *dist, int pos)
-{
-	isl_space *space;
-	isl_basic_set *hull;
-	int dim;
-	isl_ctx *ctx;
-	isl_val *nan;
-	isl_val_list *list;
-
-	ctx = isl_set_get_ctx(dist);
-	nan = isl_val_nan(ctx);
-	list = isl_val_list_alloc(ctx, 2);
-	list = isl_val_list_add(list, isl_val_copy(nan));
-	list = isl_val_list_add(list, nan);
-
-	dist = isl_set_copy(dist);
-	dim = isl_set_dim(dist, isl_dim_set);
-	if (dist && pos >= dim)
-		isl_die(ctx, isl_error_internal, "position out of bounds",
-			dist = isl_set_free(dist));
-	dist = isl_set_project_out(dist, isl_dim_set, pos + 1, dim - (pos + 1));
-	dist = isl_set_project_out(dist, isl_dim_set, 1, pos - 1);
-
-	space = isl_set_get_space(dist);
-	dist = isl_set_union(dist, isl_set_from_point(isl_point_zero(space)));
-	dist = isl_set_remove_divs(dist);
-	hull = isl_set_convex_hull(dist);
-
-	if (isl_basic_set_foreach_constraint(hull, &set_min_max, &list) < 0)
-		list = isl_val_list_free(list);
-	isl_basic_set_free(hull);
-
-	return list;
-}
-
-/* Given a schedule node "node" that, together with its child,
- * satisfies the input pattern for hybrid tiling, compute bounds
- * on the relative dependence distances of the child node with
- * respect to the parent node.  These bounds are needed to
- * construct a hybrid tiling.
- *
- * First all relevant dependences are collected and mapped
- * to the schedule space of the pair of nodes.  Then, the
- * dependence distances are computed in this space.
- *
- * These dependence distances are then projected onto a two-dimensional
- * space consisting of the single schedule dimension of the outer node
- * and one of the schedule dimensions of the inner node.
- * The maximal and minimal relative dependence distances are extracted
- * from these projections.
- * This process is repeated for each of the schedule dimensions
- * of the inner node.  For the first dimension, both minimal and
- * maximal relative dependence distances are stored in the result.
- * For the other dimensions, only the minimal relative dependence
- * distance is stored.
- */
-__isl_give ppcg_ht_bounds *ppcg_ht_compute_bounds(struct ppcg_scop *scop,
-	__isl_keep isl_schedule_node *node)
-{
-	ppcg_ht_bounds *bnd;
-	isl_space *space;
-	isl_map *map;
-	isl_set *dist;
-	isl_val_list *pair;
-	isl_schedule_node *child;
-	int n;
-	int i, dim;
-
-	if (!scop || !node || check_input_pattern(node) < 0)
-		return NULL;
-
-	child = isl_schedule_node_get_child(node, 0);
-	space = isl_schedule_node_band_get_space(child);
-	dim = isl_schedule_node_band_n_member(child);
-	isl_schedule_node_free(child);
-	bnd = ppcg_ht_bounds_alloc(space);
-	if (!bnd)
-		return NULL;
-
-	map = collect_deps(scop, node);
-
-	dist = isl_map_deltas(map);
-	n = isl_set_dim(dist, isl_dim_param);
-	dist = isl_set_project_out(dist, isl_dim_param, 0, n);
-
-	pair = min_max_dist(dist, 1);
-	bnd = ppcg_ht_bounds_set_lower(bnd, 0, isl_val_list_get_val(pair, 0));
-	bnd = ppcg_ht_bounds_set_upper(bnd, isl_val_list_get_val(pair, 1));
-	isl_val_list_free(pair);
-
-	for (i = 1; i < dim; ++i) {
-		pair = min_max_dist(dist, 1 + i);
-		bnd = ppcg_ht_bounds_set_lower(bnd, i,
-						isl_val_list_get_val(pair, 0));
-		isl_val_list_free(pair);
-	}
-
-	isl_set_free(dist);
-
-	return bnd;
-}
-
-/* Check if all the fields of "phase" are valid, freeing "phase"
- * if they are not.
- */
-static __isl_give ppcg_ht_phase *check_phase(__isl_take ppcg_ht_phase *phase)
-{
-	if (!phase)
-		return NULL;
-
-	if (!phase->tiling || !phase->local_time ||
-	    !phase->shift_space || !phase->domain)
-		return ppcg_ht_phase_free(phase);
-
-	return phase;
-}
-
-/* Construct a ppcg_ht_phase object, that simply copies
- * information from "tiling".
- * That is, the result is defined over the "ts" space and
- * corresponds to phase 1.
- */
-static __isl_give ppcg_ht_phase *construct_phase(
-	__isl_keep ppcg_ht_tiling *tiling)
-{
-	isl_ctx *ctx;
-	ppcg_ht_phase *phase;
-
-	if (!tiling)
-		return NULL;
-
-	ctx = ppcg_ht_tiling_get_ctx(tiling);
-	phase = isl_calloc_type(ctx, struct ppcg_ht_phase);
-	if (!phase)
-		return NULL;
-	phase->tiling = ppcg_ht_tiling_copy(tiling);
-	phase->time_tile = isl_aff_copy(tiling->time_tile);
-	phase->local_time = isl_aff_copy(tiling->local_time);
-	phase->shift_space = isl_aff_copy(tiling->shift_space);
-	phase->domain = isl_set_copy(tiling->hex);
-
-	return check_phase(phase);
-}
-
-/* Align the parameters of the elements of "phase" to those of "space".
- */
-static __isl_give ppcg_ht_phase *phase_align_params(
-	__isl_take ppcg_ht_phase *phase, __isl_take isl_space *space)
-{
-	if (!phase)
-		goto error;
-
-	phase->time_tile = isl_aff_align_params(phase->time_tile,
-							isl_space_copy(space));
-	phase->local_time = isl_aff_align_params(phase->local_time,
-							isl_space_copy(space));
-	phase->shift_space = isl_aff_align_params(phase->shift_space,
-							isl_space_copy(space));
-	phase->domain = isl_set_align_params(phase->domain, space);
-
-	return check_phase(phase);
-error:
-	isl_space_free(space);
-	return NULL;
-}
-
-/* Pull back "phase" over "ma".
- * That is, take a phase defined over the range of "ma" and
- * turn it into a phase defined over the domain of "ma".
- */
-static __isl_give ppcg_ht_phase *pullback_phase(__isl_take ppcg_ht_phase *phase,
-	__isl_take isl_multi_aff *ma)
-{
-	phase = phase_align_params(phase, isl_multi_aff_get_space(ma));
-	if (!phase)
-		goto error;
-
-	phase->time_tile = isl_aff_pullback_multi_aff(phase->time_tile,
-							isl_multi_aff_copy(ma));
-	phase->local_time = isl_aff_pullback_multi_aff(phase->local_time,
-							isl_multi_aff_copy(ma));
-	phase->shift_space = isl_aff_pullback_multi_aff(phase->shift_space,
-							isl_multi_aff_copy(ma));
-	phase->domain = isl_set_preimage_multi_aff(phase->domain, ma);
-
-	return check_phase(phase);
-error:
-	isl_multi_aff_free(ma);
-	return NULL;
-}
-
-/* Pullback "phase" over phase->tiling->shift_phase, which shifts
- * phase 0 to phase 1.  The pullback therefore takes a phase 1
- * description and turns it into a phase 0 description.
- */
-static __isl_give ppcg_ht_phase *shift_phase(__isl_take ppcg_ht_phase *phase)
-{
-	ppcg_ht_tiling *tiling;
-
-	if (!phase)
-		return NULL;
-
-	tiling = phase->tiling;
-	return pullback_phase(phase, isl_multi_aff_copy(tiling->shift_phase));
-}
-
-/* Take a "phase" defined over the ts-space and plug in the projection
- * from the input schedule space to the ts-space.
- * The result is then defined over this input schedule space.
- */
-static __isl_give ppcg_ht_phase *lift_phase(__isl_take ppcg_ht_phase *phase)
-{
-	ppcg_ht_tiling *tiling;
-
-	if (!phase)
-		return NULL;
-
-	tiling = phase->tiling;
-	return pullback_phase(phase, isl_multi_aff_copy(tiling->project_ts));
-}
-
-/* Compute the shift that should be added to the space band
- * in order to be able to apply rectangular tiling to the space.
- * Store the shift in phase->space_shift.
- *
- * In the first dimension, it is equal to shift_space - s.
- * For phase 1, this results in
- *
- *	(-(2 * shift_s)*T) % W
- *
- * In phase 0, the "s" in shift_space has been replaced by "s + shift_s",
- * so the result is
- *
- *	shift_s + (-(2 * shift_s)*T) % W
- *
- * In the other dimensions, the shift is equal to
- *
- *	dl_i * local_time.
- */
-static __isl_give ppcg_ht_phase *compute_space_shift(
-	__isl_take ppcg_ht_phase *phase)
-{
-	int i, n;
-	isl_space *space;
-	isl_local_space *ls;
-	isl_aff *aff, *s;
-	isl_multi_aff *space_shift;
-
-	if (!phase)
-		return NULL;
-
-	space = ppcg_ht_phase_get_input_space(phase);
-	space = isl_space_unwrap(space);
-	space = isl_space_range_map(space);
-
-	space_shift = isl_multi_aff_zero(space);
-	aff = isl_aff_copy(phase->shift_space);
-	ls = isl_local_space_from_space(isl_aff_get_domain_space(aff));
-	s = isl_aff_var_on_domain(ls, isl_dim_set, 1);
-	aff = isl_aff_sub(aff, s);
-	space_shift = isl_multi_aff_set_aff(space_shift, 0, aff);
-
-	n = isl_multi_aff_dim(space_shift, isl_dim_out);
-	for (i = 1; i < n; ++i) {
-		isl_val *v;
-		isl_aff *time;
-
-		v = ppcg_ht_bounds_get_lower(phase->tiling->bounds, i);
-		time = isl_aff_copy(phase->local_time);
-		time = isl_aff_scale_val(time, v);
-		space_shift = isl_multi_aff_set_aff(space_shift, i, time);
-	}
-
-	if (!space_shift)
-		return ppcg_ht_phase_free(phase);
-	phase->space_shift = space_shift;
-	return phase;
-}
-
-/* Compute the space tiling and store the result in phase->space_tile.
- * The space tiling is of the form
- *
- *	[P[t] -> C[s]] -> C[floor((s + space_shift)/space_size]
- */
-static __isl_give ppcg_ht_phase *compute_space_tile(
-	__isl_take ppcg_ht_phase *phase)
-{
-	isl_space *space;
-	isl_multi_val *space_sizes;
-	isl_multi_aff *space_shift;
-	isl_multi_aff *tile;
-
-	if (!phase)
-		return NULL;
-
-	space = ppcg_ht_phase_get_input_space(phase);
-	space = isl_space_unwrap(space);
-	tile = isl_multi_aff_range_map(space);
-	space_shift = isl_multi_aff_copy(phase->space_shift);
-	tile = isl_multi_aff_add(space_shift, tile);
-	space_sizes = isl_multi_val_copy(phase->tiling->space_sizes);
-	tile = isl_multi_aff_scale_down_multi_val(tile, space_sizes);
-	tile = isl_multi_aff_floor(tile);
-
-	if (!tile)
-		return ppcg_ht_phase_free(phase);
-	phase->space_tile = tile;
-	return phase;
-}
-
-/* Construct a representation for one of the two phase for hybrid tiling
- * "tiling".  If "shift" is not set, then the phase is constructed
- * directly from the hexagonal tile shape in "tiling", which represents
- * the phase-1 tiles.  If "shift" is set, then this tile shape is shifted
- * back over tiling->shift_phase to obtain the phase-0 tiles.
- *
- * First copy data from "tiling", then optionally shift the phase and
- * finally move the tiling from the "ts" space of "tiling" to
- * the space of the input pattern.
- *
- * After the basic phase has been computed, also compute
- * the corresponding space shift.
- */
-static __isl_give ppcg_ht_phase *ppcg_ht_tiling_compute_phase(
-	__isl_keep ppcg_ht_tiling *tiling, int shift)
-{
-	ppcg_ht_phase *phase;
-
-	phase = construct_phase(tiling);
-	if (shift)
-		phase = shift_phase(phase);
-	phase = lift_phase(phase);
-
-	phase = compute_space_shift(phase);
-	phase = compute_space_tile(phase);
-
-	return phase;
-}
-
-/* Consruct a function that is equal to the time tile of "phase0"
- * on the domain of "phase0" and equal to the time tile of "phase1"
- * on the domain of "phase1".
- * The two domains are assumed to form a partition of the input
- * schedule space.
- */
-static __isl_give isl_pw_multi_aff *combine_time_tile(
-	__isl_keep ppcg_ht_phase *phase0, __isl_keep ppcg_ht_phase *phase1)
-{
-	isl_aff *T;
-	isl_pw_aff *time, *time1;
-
-	if (!phase0 || !phase1)
-		return NULL;
-
-	T = isl_aff_copy(phase0->time_tile);
-	time = isl_pw_aff_alloc(ppcg_ht_phase_get_domain(phase0), T);
-
-	T = isl_aff_copy(phase1->time_tile);
-	time1 = isl_pw_aff_alloc(ppcg_ht_phase_get_domain(phase1), T);
-
-	time = isl_pw_aff_union_add(time, time1);
-
-	return isl_pw_multi_aff_from_pw_aff(time);
-}
-
-/* Name used in mark nodes that contain a pointer to a ppcg_ht_phase.
- */
-static char *ppcg_phase_name = "phase";
-
-/* Does "id" contain a pointer to a ppcg_ht_phase?
- * That is, is it called "phase"?
- */
-static isl_bool is_phase_id(__isl_keep isl_id *id)
-{
-	const char *name;
-
-	name = isl_id_get_name(id);
-	if (!name)
-		return isl_bool_error;
-
-	return !strcmp(name, ppcg_phase_name);
-}
-
-/* Given a mark node with an identifier that points to a ppcg_ht_phase,
- * extract this ppcg_ht_phase pointer.
- */
-__isl_keep ppcg_ht_phase *ppcg_ht_phase_extract_from_mark(
-	__isl_keep isl_schedule_node *node)
-{
-	isl_bool is_phase;
-	isl_id *id;
-	void *p;
-
-	if (!node)
-		return NULL;
-	if (isl_schedule_node_get_type(node) != isl_schedule_node_mark)
-		isl_die(isl_schedule_node_get_ctx(node), isl_error_internal,
-			"not a phase mark", return NULL);
-
-	id = isl_schedule_node_mark_get_id(node);
-	is_phase = is_phase_id(id);
-	p = isl_id_get_user(id);
-	isl_id_free(id);
-
-	if (is_phase < 0)
-		return NULL;
-	if (!is_phase)
-		isl_die(isl_schedule_node_get_ctx(node), isl_error_internal,
-			"not a phase mark", return NULL);
-
-	return p;
-}
-
-/* Insert a mark node at "node" holding a pointer to "phase".
- */
-static __isl_give isl_schedule_node *insert_phase(
-	__isl_take isl_schedule_node *node, __isl_take ppcg_ht_phase *phase)
-{
-	isl_ctx *ctx;
-	isl_id *id;
-
-	if (!node)
-		goto error;
-	ctx = isl_schedule_node_get_ctx(node);
-	id = isl_id_alloc(ctx, ppcg_phase_name, phase);
-	if (!id)
-		goto error;
-	id = isl_id_set_free_user(id, &ppcg_ht_phase_free_wrap);
-	node = isl_schedule_node_insert_mark(node, id);
-
-	return node;
-error:
-	ppcg_ht_phase_free(phase);
-	isl_schedule_node_free(node);
-	return NULL;
-}
-
-/* Construct a mapping from the elements of the original pair of bands
- * to which tiling was applied that belong to a tile of "phase"
- * to that tile, preserving the values for the outer bands.
- *
- * The mapping is of the form
- *
- *	[[outer] -> [P -> C]] -> [[outer] -> [tile]]
- *
- * where tile is defined by a concatenation of the time_tile and
- * the space_tile.
- */
-static __isl_give isl_map *construct_tile_map(__isl_keep ppcg_ht_phase *phase)
-{
-	int depth;
-	isl_space *space;
-	isl_multi_aff *ma;
-	isl_multi_aff *tiling;
-	isl_map *el2tile;
-
-	depth = isl_schedule_node_get_schedule_depth(
-						phase->tiling->input_node);
-	space = isl_aff_get_space(phase->time_tile);
-	space = isl_space_params(space);
-	space = isl_space_set_from_params(space);
-	space = isl_space_add_dims(space, isl_dim_set, depth);
-	space = isl_space_map_from_set(space);
-	ma = isl_multi_aff_identity(space);
-
-	tiling = isl_multi_aff_flat_range_product(
-		isl_multi_aff_from_aff(isl_aff_copy(phase->time_tile)),
-		isl_multi_aff_copy(phase->space_tile));
-	el2tile = isl_map_from_multi_aff(tiling);
-	el2tile = isl_map_intersect_domain(el2tile,
-						isl_set_copy(phase->domain));
-	el2tile = isl_map_product(isl_map_from_multi_aff(ma), el2tile);
-
-	return el2tile;
-}
-
-/* Return a description of the full tiles of "phase" at the point
- * in the original schedule tree where the tiling was applied.
- *
- * First construct a mapping from the input schedule dimensions
- * up to an including the original pair of bands to which hybrid tiling
- * was applied to schedule dimensions in which this original pair
- * has been replaced by the tiles.
- * This mapping is of the form
- *
- *	[[outer] -> [P -> C]] -> [[outer] -> [tile]]
- *
- * Apply this mapping to the set of all values for the input
- * schedule dimensions and then apply its inverse.
- * The result is the set of values for the input schedule dimensions
- * that would map to any of the tiles.  Subtracting from this set
- * the set of values that are actually executed produces the set
- * of values that belong to a tile but that are not executed.
- * Mapping these back to the tiles produces a description of
- * the partial tiles.  Subtracting these from the set of all tiles
- * produces a description of the full tiles in the form
- *
- *	[[outer] -> [tile]]
- */
-static __isl_give isl_set *compute_full_tile(__isl_keep ppcg_ht_phase *phase)
-{
-	isl_schedule_node *node;
-	isl_union_set *domain;
-	isl_union_map *prefix, *schedule;
-	isl_set *all, *partial, *all_el;
-	isl_map *tile2el, *el2tile;
-	isl_multi_union_pw_aff *mupa;
-
-	el2tile = construct_tile_map(phase);
-	tile2el = isl_map_reverse(isl_map_copy(el2tile));
-
-	node = phase->tiling->input_node;
-	prefix = isl_schedule_node_get_prefix_schedule_union_map(node);
-	domain = isl_schedule_node_get_domain(node);
-	mupa = isl_multi_union_pw_aff_copy(phase->tiling->input_schedule);
-	schedule = isl_union_map_from_multi_union_pw_aff(mupa);
-	schedule = isl_union_map_range_product(prefix, schedule);
-	all_el = isl_set_from_union_set(isl_union_set_apply(domain, schedule));
-	all_el = isl_set_coalesce(all_el);
-
-	all = isl_set_apply(isl_set_copy(all_el), isl_map_copy(el2tile));
-
-	partial = isl_set_copy(all);
-	partial = isl_set_apply(partial, tile2el);
-	partial = isl_set_subtract(partial, all_el);
-	partial = isl_set_apply(partial, el2tile);
-
-	return isl_set_subtract(all, partial);
-}
-
-/* Copy the AST loop types of the non-isolated part to those
- * of the isolated part.
- */
-static __isl_give isl_schedule_node *set_isolate_loop_type(
-	__isl_take isl_schedule_node *node)
-{
-	int i, n;
-
-	n = isl_schedule_node_band_n_member(node);
-	for (i = 0; i < n; ++i) {
-		enum isl_ast_loop_type type;
-
-		type = isl_schedule_node_band_member_get_ast_loop_type(node, i);
-		node = isl_schedule_node_band_member_set_isolate_ast_loop_type(
-								node, i, type);
-	}
-
-	return node;
-}
-
-/* If options->isolate_full_tiles is set, then mark the full tiles
- * in "node" for isolation.  The full tiles are derived from "phase".
- * "node" may point to a part of the tiling, e.g., the space tiling.
- *
- * The full tiles are originally computed in the form
- *
- *	[[outer] -> [tile]]
- *
- * However, the band that "node" points to may only contain
- * subset of the tile dimensions.
- * The description above is therefore treated as
- *
- *	[[outer] -> [before; this; after]]
- *
- * before is of size "pos"; this is of size "dim"; and
- * after is of size "out - pos - dim".
- * The after part is first project out.  Then the range is split
- * into a before and this part and finally the before part is moved
- * to the domain, resulting in
- *
- *	[[outer; before] -> [this]]
- *
- * This description is then used as the isolate option.
- *
- * The AST loop type for the isolated part is set to be the same
- * as that of the non-isolated part.
- */
-static __isl_give isl_schedule_node *ppcg_ht_phase_isolate_full_tile_node(
-	__isl_keep ppcg_ht_phase *phase, __isl_take isl_schedule_node *node,
-	struct ppcg_options *options)
-{
-	int in, out, pos, depth, dim;
-	isl_space *space;
-	isl_multi_aff *ma1, *ma2;
-	isl_set *tile;
-	isl_map *map;
-	isl_set *set;
-	isl_union_set *opt;
-
-	if (!options->isolate_full_tiles)
-		return node;
-
-	depth = isl_schedule_node_get_schedule_depth(node);
-	dim = isl_schedule_node_band_n_member(node);
-
-	tile = compute_full_tile(phase);
-	map = isl_set_unwrap(tile);
-	in = isl_map_dim(map, isl_dim_in);
-	out = isl_map_dim(map, isl_dim_out);
-	pos = depth - in;
-	map = isl_map_project_out(map, isl_dim_out, pos + dim,
-				out - (pos + dim));
-	space = isl_space_range(isl_map_get_space(map));
-	ma1 = isl_multi_aff_project_out_map(isl_space_copy(space),
-					   isl_dim_set, pos, dim);
-	ma2 = isl_multi_aff_project_out_map(space, isl_dim_set, 0, pos);
-	ma1 = isl_multi_aff_range_product(ma1, ma2);
-	map = isl_map_apply_range(map, isl_map_from_multi_aff(ma1));
-	map = isl_map_uncurry(map);
-	map = isl_map_flatten_domain(map);
-	set = isl_map_wrap(map);
-	set = isl_set_set_tuple_name(set, "isolate");
-
-	opt = isl_schedule_node_band_get_ast_build_options(node);
-	opt = isl_union_set_add_set(opt, set);
-	node = isl_schedule_node_band_set_ast_build_options(node, opt);
-	node = set_isolate_loop_type(node);
-
-	return node;
-}
-
-/* Insert a band node for performing the space tiling for "phase" at "node".
- * In particular, insert a band node with partial schedule
- *
- *	[P[t] -> C[s]] -> C[floor((s + space_shift)/space_size)]
- *
- * pulled back over the input schedule.
- * "options" determines whether full tiles should be separated
- * from partial tiles.
- *
- * The first tile dimension iterates over the hexagons in the same
- * phase, which are independent by construction.  The first dimension
- * is therefore marked coincident.
- * All dimensions are also marked for being generated as atomic loops
- * because separation is usually not desirable on tile loops.
- */
-static __isl_give isl_schedule_node *insert_space_tiling(
-	__isl_keep ppcg_ht_phase *phase, __isl_take isl_schedule_node *node,
-	struct ppcg_options *options)
-{
-	isl_multi_aff *space_tile;
-	isl_multi_union_pw_aff *mupa;
-
-	if (!phase)
-		return isl_schedule_node_free(node);
-
-	space_tile = isl_multi_aff_copy(phase->space_tile);
-	mupa = isl_multi_union_pw_aff_copy(phase->tiling->input_schedule);
-	mupa = isl_multi_union_pw_aff_apply_multi_aff(mupa, space_tile);
-	node = isl_schedule_node_insert_partial_schedule(node, mupa);
-	node = ppcg_set_schedule_node_type(node, isl_ast_loop_atomic);
-	node = ppcg_ht_phase_isolate_full_tile_node(phase, node, options);
-	node = isl_schedule_node_band_member_set_coincident(node, 0, 1);
-
-	return node;
-}
-
-/* Given a pointer "node" to (a copy of) the original child node
- * in the input pattern, adjust its partial schedule such that
- * it starts at zero within each tile.
- *
- * That is, replace "s" by (s + space_shift) % space_sizes.
- */
-__isl_give isl_schedule_node *ppcg_ht_phase_shift_space_point(
-	__isl_keep ppcg_ht_phase *phase, __isl_take isl_schedule_node *node)
-{
-	isl_multi_val *space_sizes;
-	isl_multi_aff *space_shift;
-	isl_multi_union_pw_aff *mupa;
-
-	space_shift = isl_multi_aff_copy(phase->space_shift);
-	mupa = isl_multi_union_pw_aff_copy(phase->tiling->input_schedule);
-	mupa = isl_multi_union_pw_aff_apply_multi_aff(mupa, space_shift);
-	node = isl_schedule_node_band_shift(node, mupa);
-	space_sizes = isl_multi_val_copy(phase->tiling->space_sizes);
-	node = isl_schedule_node_band_mod(node, space_sizes);
-
-	return node;
-}
-
-/* Does
- *
- *	s0 > delta + 2 * {delta * h} - 1
- *
- * hold?
- */
-static isl_bool wide_enough(__isl_keep isl_val *s0, __isl_keep isl_val *delta,
-	__isl_keep isl_val *h)
-{
-	isl_val *v, *v2;
-	isl_bool ok;
-
-	v = isl_val_mul(isl_val_copy(delta), isl_val_copy(h));
-	v2 = isl_val_floor(isl_val_copy(v));
-	v = isl_val_sub(v, v2);
-	v = isl_val_mul_ui(v, 2);
-	v = isl_val_add(v, isl_val_copy(delta));
-	v = isl_val_sub_ui(v, 1);
-	ok = isl_val_gt(s0, v);
-	isl_val_free(v);
-
-	return ok;
-}
-
-/* Is the tile size specified by "sizes" wide enough in the first space
- * dimension, i.e., the base of the hexagon?  This ensures that,
- * after hybrid tiling using "bounds" and these sizes,
- * neighboring hexagons in the same phase are far enough apart
- * that they do not depend on each other.
- * The test is only meaningful if the bounds are valid.
- *
- * Let st be (half) the size in the time dimension and s0 the base
- * size in the first space dimension.  Let delta be the dependence
- * distance in either positive or negative direction.  In principle,
- * it should be enough to have s0 + 1 > delta, i.e., s0 >= delta.
- * However, in case of fractional delta, the tile is not extended
- * with delta * (st - 1), but instead with floor(delta * (st - 1)).
- * The condition therefore needs to be adjusted to
- *
- *	s0 + 1 > delta + 2 {delta * (st - 1)}
- *
- * (with {} the fractional part) to account for the two slanted sides.
- * The condition in the paper "Hybrid Hexagonal/Classical Tiling for GPUs"
- * translates to
- *
- *	s0 >= delta + {delta * (st - 1)}
- *
- * Since 1 > frac(delta * (st - 1)), this condition implies
- * the condition above.
- *
- * The condition is checked for both directions.
- */
-isl_bool ppcg_ht_bounds_supports_sizes(__isl_keep ppcg_ht_bounds *bounds,
-	__isl_keep isl_multi_val *sizes)
-{
-	isl_val *s0, *h;
-	isl_val *delta;
-	isl_bool ok;
-
-	ok = ppcg_ht_bounds_is_valid(bounds);
-	if (ok < 0 || !ok)
-		return ok;
-
-	h = isl_val_sub_ui(isl_multi_val_get_val(sizes, 0), 1);
-	s0 = isl_multi_val_get_val(sizes, 1);
-
-	delta = ppcg_ht_bounds_get_lower(bounds, 0);
-	ok = wide_enough(s0, delta, h);
-	isl_val_free(delta);
-
-	delta = ppcg_ht_bounds_get_upper(bounds);
-	if (ok == isl_bool_true)
-		ok = wide_enough(s0, delta, h);
-	isl_val_free(delta);
-
-	isl_val_free(s0);
-	isl_val_free(h);
-
-	return ok;
-}
-
-/* Check that the tile will be wide enough in the first space
- * dimension, i.e., the base of the hexagon.  This ensures that
- * neighboring hexagons in the same phase are far enough apart
- * that they do not depend on each other.
- *
- * Error out if the condition fails to hold.
- */
-static isl_stat check_width(__isl_keep ppcg_ht_bounds *bounds,
-	__isl_keep isl_multi_val *sizes)
-{
-	isl_bool ok;
-
-	ok = ppcg_ht_bounds_supports_sizes(bounds, sizes);
-
-	if (ok < 0)
-		return isl_stat_error;
-	if (!ok)
-		isl_die(isl_multi_val_get_ctx(sizes), isl_error_invalid,
-			"base of hybrid tiling hexagon not sufficiently wide",
-			return isl_stat_error);
-
-	return isl_stat_ok;
-}
-
-/* Given valid bounds on the relative dependence distances for
- * the pair of nested nodes that "node" point to, as well as sufficiently
- * wide tile sizes "sizes", insert the corresponding time and space tiling
- * at "node", along with a pair of phase nodes that can be used
- * to make further changes.
- * The space of "sizes" should be the product of the spaces
- * of the schedules of the pair of parent and child nodes.
- * "options" determines whether full tiles should be separated
- * from partial tiles.
- *
- * In particular, given an input of the form
- *
- *	P - C - ...
- *
- * the output has the form
- *
- *	        /- F0 - M0 - CT0 - P - C - ...
- *	PT - seq
- *	        \- F1 - M1 - CT1 - P - C - ...
- *
- * PT is the global time tiling.  Within each of these tiles,
- * two phases are executed in order.  Within each phase, the schedule
- * space is further subdivided into tiles through CT0 and CT1.
- * The first dimension of each of these iterates over the hexagons
- * within a phase and these are independent by construction.
- * The F0 and F1 filters filter the statement instances that belong
- * to the corresponding phase.  The M0 and M1 marks contain a pointer
- * to a ppcg_ht_phase object that can be used to perform further changes.
- *
- * After checking that input satisfies the requirements,
- * a data structure is constructed that represents the tiling and
- * two additional data structures are constructed for the two phases
- * of the tiling.  These are then used to define the filters F0 and F1 and
- * combined to construct the time tiling PT.
- * Then the time tiling node PT is inserted, followed by
- * the sequence with the two filters, the CT space tiling nodes and
- * the phase markers M.
- */
-__isl_give isl_schedule_node *ppcg_ht_bounds_insert_tiling(
-	__isl_take ppcg_ht_bounds *bounds, __isl_take isl_multi_val *sizes,
-	__isl_take isl_schedule_node *node, struct ppcg_options *options)
-{
-	isl_ctx *ctx;
-	isl_union_set *phase0;
-	isl_union_set *phase1;
-	isl_multi_union_pw_aff *input, *dom_time;
-	isl_union_pw_multi_aff *upma;
-	isl_pw_multi_aff *time;
-	isl_union_set_list *phases;
-	ppcg_ht_tiling *tiling;
-	ppcg_ht_phase *phase_0;
-	ppcg_ht_phase *phase_1;
-
-	if (!node || !sizes || !bounds)
-		goto error;
-	if (check_input_pattern(node) < 0 || check_width(bounds, sizes) < 0)
-		goto error;
-
-	ctx = isl_schedule_node_get_ctx(node);
-
-	input = extract_input_schedule(node);
-
-	tiling = ppcg_ht_bounds_construct_tiling(bounds, node, input, sizes);
-	phase_0 = ppcg_ht_tiling_compute_phase(tiling, 1);
-	phase_1 = ppcg_ht_tiling_compute_phase(tiling, 0);
-	time = combine_time_tile(phase_0, phase_1);
-	ppcg_ht_tiling_free(tiling);
-
-	upma = isl_union_pw_multi_aff_from_multi_union_pw_aff(
-					isl_multi_union_pw_aff_copy(input));
-	phase0 = isl_union_set_from_set(ppcg_ht_phase_get_domain(phase_0));
-	phase0 = isl_union_set_preimage_union_pw_multi_aff(phase0,
-					isl_union_pw_multi_aff_copy(upma));
-	phase1 = isl_union_set_from_set(ppcg_ht_phase_get_domain(phase_1));
-	phase1 = isl_union_set_preimage_union_pw_multi_aff(phase1, upma);
-
-	phases = isl_union_set_list_alloc(ctx, 2);
-	phases = isl_union_set_list_add(phases, phase0);
-	phases = isl_union_set_list_add(phases, phase1);
-
-	dom_time = isl_multi_union_pw_aff_apply_pw_multi_aff(input, time);
-	node = isl_schedule_node_insert_partial_schedule(node, dom_time);
-
-	node = isl_schedule_node_child(node, 0);
-
-	node = isl_schedule_node_insert_sequence(node, phases);
-	node = isl_schedule_node_child(node, 0);
-	node = isl_schedule_node_child(node, 0);
-	node = insert_space_tiling(phase_0, node, options);
-	node = insert_phase(node, phase_0);
-	node = isl_schedule_node_parent(node);
-	node = isl_schedule_node_next_sibling(node);
-	node = isl_schedule_node_child(node, 0);
-	node = insert_space_tiling(phase_1, node, options);
-	node = insert_phase(node, phase_1);
-	node = isl_schedule_node_parent(node);
-	node = isl_schedule_node_parent(node);
-
-	node = isl_schedule_node_parent(node);
-
-	isl_multi_val_free(sizes);
-	return node;
-error:
-	isl_multi_val_free(sizes);
-	isl_schedule_node_free(node);
-	ppcg_ht_bounds_free(bounds);
-	return NULL;
-}
-
-/* Given a branch "node" that contains a sequence node with two phases
- * of hybrid tiling as input, call "fn" on each of the two phase marker
- * nodes.
- *
- * That is, the input is as follows
- *
- *	         /- F0 - M0 - ...
- *	... - seq
- *	         \- F1 - M1 - ...
- *
- * and "fn" is called on M0 and on M1.
- */
-__isl_give isl_schedule_node *hybrid_tile_foreach_phase(
-	__isl_take isl_schedule_node *node,
-	__isl_give isl_schedule_node *(*fn)(__isl_take isl_schedule_node *node,
-		void *user), void *user)
-{
-	int depth0, depth;
-
-	depth0 = isl_schedule_node_get_tree_depth(node);
-
-	while (node &&
-	    isl_schedule_node_get_type(node) != isl_schedule_node_sequence)
-		node = isl_schedule_node_child(node, 0);
-
-	node = isl_schedule_node_child(node, 0);
-	node = isl_schedule_node_child(node, 0);
-	if (!node)
-		return NULL;
-	node = fn(node, user);
-	node = isl_schedule_node_parent(node);
-	node = isl_schedule_node_next_sibling(node);
-	node = isl_schedule_node_child(node, 0);
-	if (!node)
-		return NULL;
-	node = fn(node, user);
-	node = isl_schedule_node_parent(node);
-	node = isl_schedule_node_parent(node);
-
-	depth = isl_schedule_node_get_tree_depth(node);
-	node = isl_schedule_node_ancestor(node, depth - depth0);
-
-	return node;
-}
-
-/* This function is called on each of the two phase marks
- * in a hybrid tiling tree.
- * Drop the phase mark at "node".
- */
-static __isl_give isl_schedule_node *drop_phase_mark(
-	__isl_take isl_schedule_node *node, void *user)
-{
-	isl_id *id;
-	isl_bool is_phase;
-
-	if (isl_schedule_node_get_type(node) != isl_schedule_node_mark)
-		return node;
-
-	id = isl_schedule_node_mark_get_id(node);
-	is_phase = is_phase_id(id);
-	isl_id_free(id);
-
-	if (is_phase < 0)
-		return isl_schedule_node_free(node);
-	if (is_phase)
-		node = isl_schedule_node_delete(node);
-
-	return node;
-}
-
-/* Given a branch "node" that contains a sequence node with two phases
- * of hybrid tiling as input, remove the two phase marker nodes.
- *
- * That is, the input is as follows
- *
- *	         /- F0 - M0 - ...
- *	... - seq
- *	         \- F1 - M1 - ...
- *
- * and the output is
- *
- *	         /- F0 - ...
- *	... - seq
- *	         \- F1 - ...
- */
-__isl_give isl_schedule_node *hybrid_tile_drop_phase_marks(
-	__isl_take isl_schedule_node *node)
-{
-	return hybrid_tile_foreach_phase(node, &drop_phase_mark, NULL);
-}
diff --git a/polly/lib/External/ppcg/hybrid.h b/polly/lib/External/ppcg/hybrid.h
deleted file mode 100644
index 71136c6779eee..0000000000000
--- a/polly/lib/External/ppcg/hybrid.h
+++ /dev/null
@@ -1,41 +0,0 @@
-#ifndef HYBRID_H
-#define HYBRID_H
-
-#include <isl/val.h>
-#include <isl/schedule_node.h>
-
-#include "ppcg.h"
-
-struct ppcg_ht_bounds;
-typedef struct ppcg_ht_bounds ppcg_ht_bounds;
-
-struct ppcg_ht_phase;
-typedef struct ppcg_ht_phase ppcg_ht_phase;
-
-isl_bool ppcg_ht_has_input_pattern(__isl_keep isl_schedule_node *node);
-isl_bool ppcg_ht_parent_has_input_pattern(__isl_keep isl_schedule_node *node);
-
-__isl_give ppcg_ht_bounds *ppcg_ht_compute_bounds(struct ppcg_scop *scop,
-	__isl_keep isl_schedule_node *node);
-void ppcg_ht_bounds_dump(__isl_keep ppcg_ht_bounds *bounds);
-isl_bool ppcg_ht_bounds_is_valid(__isl_keep ppcg_ht_bounds *bounds);
-isl_bool ppcg_ht_bounds_supports_sizes(__isl_keep ppcg_ht_bounds *bounds,
-	__isl_keep isl_multi_val *sizes);
-__isl_give isl_schedule_node *ppcg_ht_bounds_insert_tiling(
-	__isl_take ppcg_ht_bounds *bounds, __isl_take isl_multi_val *sizes,
-	__isl_take isl_schedule_node *node, struct ppcg_options *options);
-__isl_null ppcg_ht_bounds *ppcg_ht_bounds_free(
-	__isl_take ppcg_ht_bounds *bounds);
-
-__isl_keep ppcg_ht_phase *ppcg_ht_phase_extract_from_mark(
-	__isl_keep isl_schedule_node *node);
-__isl_give isl_schedule_node *ppcg_ht_phase_shift_space_point(
-	__isl_keep ppcg_ht_phase *phase, __isl_take isl_schedule_node *node);
-__isl_give isl_schedule_node *hybrid_tile_foreach_phase(
-	__isl_take isl_schedule_node *node,
-	__isl_give isl_schedule_node *(*fn)(__isl_take isl_schedule_node *node,
-		void *user), void *user);
-__isl_give isl_schedule_node *hybrid_tile_drop_phase_marks(
-	__isl_take isl_schedule_node *node);
-
-#endif
diff --git a/polly/lib/External/ppcg/ocl_utilities.c b/polly/lib/External/ppcg/ocl_utilities.c
deleted file mode 100644
index c7a215674c12d..0000000000000
--- a/polly/lib/External/ppcg/ocl_utilities.c
+++ /dev/null
@@ -1,174 +0,0 @@
-#include <stdio.h>
-#include <stdlib.h>
-#include "ocl_utilities.h"
-
-/* Return the OpenCL error string for a given error number.
- */
-const char *opencl_error_string(cl_int error)
-{
-	int errorCount;
-	int index;
-
-	static const char *errorString[] = {
-		[CL_SUCCESS] = "CL_SUCCESS",
-		[-CL_DEVICE_NOT_FOUND] = "CL_DEVICE_NOT_FOUND",
-		[-CL_DEVICE_NOT_AVAILABLE] = "CL_DEVICE_NOT_AVAILABLE",
-		[-CL_COMPILER_NOT_AVAILABLE] = "CL_COMPILER_NOT_AVAILABLE",
-		[-CL_MEM_OBJECT_ALLOCATION_FAILURE] =
-			"CL_MEM_OBJECT_ALLOCATION_FAILURE",
-		[-CL_OUT_OF_RESOURCES] = "CL_OUT_OF_RESOURCES",
-		[-CL_OUT_OF_HOST_MEMORY] = "CL_OUT_OF_HOST_MEMORY",
-		[-CL_PROFILING_INFO_NOT_AVAILABLE] =
-			"CL_PROFILING_INFO_NOT_AVAILABLE",
-		[-CL_MEM_COPY_OVERLAP] = "CL_MEM_COPY_OVERLAP",
-		[-CL_IMAGE_FORMAT_MISMATCH] = "CL_IMAGE_FORMAT_MISMATCH",
-		[-CL_IMAGE_FORMAT_NOT_SUPPORTED] =
-			"CL_IMAGE_FORMAT_NOT_SUPPORTED",
-		[-CL_BUILD_PROGRAM_FAILURE] = "CL_BUILD_PROGRAM_FAILURE",
-		[-CL_MAP_FAILURE] = "CL_MAP_FAILURE",
-		[-CL_INVALID_VALUE] = "CL_INVALID_VALUE",
-		[-CL_INVALID_DEVICE_TYPE] = "CL_INVALID_DEVICE_TYPE",
-		[-CL_INVALID_PLATFORM] = "CL_INVALID_PLATFORM",
-		[-CL_INVALID_DEVICE] = "CL_INVALID_DEVICE",
-		[-CL_INVALID_CONTEXT] = "CL_INVALID_CONTEXT",
-		[-CL_INVALID_QUEUE_PROPERTIES] = "CL_INVALID_QUEUE_PROPERTIES",
-		[-CL_INVALID_COMMAND_QUEUE] = "CL_INVALID_COMMAND_QUEUE",
-		[-CL_INVALID_HOST_PTR] = "CL_INVALID_HOST_PTR",
-		[-CL_INVALID_MEM_OBJECT] = "CL_INVALID_MEM_OBJECT",
-		[-CL_INVALID_IMAGE_FORMAT_DESCRIPTOR] =
-			"CL_INVALID_IMAGE_FORMAT_DESCRIPTOR",
-		[-CL_INVALID_IMAGE_SIZE] = "CL_INVALID_IMAGE_SIZE",
-		[-CL_INVALID_SAMPLER] = "CL_INVALID_SAMPLER",
-		[-CL_INVALID_BINARY] = "CL_INVALID_BINARY",
-		[-CL_INVALID_BUILD_OPTIONS] = "CL_INVALID_BUILD_OPTIONS",
-		[-CL_INVALID_PROGRAM] = "CL_INVALID_PROGRAM",
-		[-CL_INVALID_PROGRAM_EXECUTABLE] =
-			"CL_INVALID_PROGRAM_EXECUTABLE",
-		[-CL_INVALID_KERNEL_NAME] = "CL_INVALID_KERNEL_NAME",
-		[-CL_INVALID_KERNEL_DEFINITION] =
-			"CL_INVALID_KERNEL_DEFINITION",
-		[-CL_INVALID_KERNEL] = "CL_INVALID_KERNEL",
-		[-CL_INVALID_ARG_INDEX] = "CL_INVALID_ARG_INDEX",
-		[-CL_INVALID_ARG_VALUE] = "CL_INVALID_ARG_VALUE",
-		[-CL_INVALID_ARG_SIZE] = "CL_INVALID_ARG_SIZE",
-		[-CL_INVALID_KERNEL_ARGS] = "CL_INVALID_KERNEL_ARGS",
-		[-CL_INVALID_WORK_DIMENSION] = "CL_INVALID_WORK_DIMENSION",
-		[-CL_INVALID_WORK_GROUP_SIZE] = "CL_INVALID_WORK_GROUP_SIZE",
-		[-CL_INVALID_WORK_ITEM_SIZE] = "CL_INVALID_WORK_ITEM_SIZE",
-		[-CL_INVALID_GLOBAL_OFFSET] = "CL_INVALID_GLOBAL_OFFSET",
-		[-CL_INVALID_EVENT_WAIT_LIST] = "CL_INVALID_EVENT_WAIT_LIST",
-		[-CL_INVALID_EVENT] = "CL_INVALID_EVENT",
-		[-CL_INVALID_OPERATION] = "CL_INVALID_OPERATION",
-		[-CL_INVALID_GL_OBJECT] = "CL_INVALID_GL_OBJECT",
-		[-CL_INVALID_BUFFER_SIZE] = "CL_INVALID_BUFFER_SIZE",
-		[-CL_INVALID_MIP_LEVEL] = "CL_INVALID_MIP_LEVEL",
-		[-CL_INVALID_GLOBAL_WORK_SIZE] = "CL_INVALID_GLOBAL_WORK_SIZE",
-		[-CL_INVALID_PROPERTY] = "CL_INVALID_PROPERTY"
-	};
-
-	errorCount = sizeof(errorString) / sizeof(errorString[0]);
-	index = -error;
-
-	return (index >= 0 && index < errorCount) ?
-		errorString[index] : "Unspecified Error";
-}
-
-/* Find a GPU or a CPU associated with the first available platform.
- * If use_gpu is set, then this function first tries to look for a GPU
- * in the first available platform.
- * If this fails or if use_gpu is not set, then it tries to use the CPU.
- */
-cl_device_id opencl_create_device(int use_gpu)
-{
-	cl_platform_id platform;
-	cl_device_id dev;
-	int err;
-
-	err = clGetPlatformIDs(1, &platform, NULL);
-	if (err < 0) {
-		fprintf(stderr, "Error %s while looking for a platform.\n",
-				opencl_error_string(err));
-		exit(1);
-	}
-
-	err = CL_DEVICE_NOT_FOUND;
-	if (use_gpu)
-		err = clGetDeviceIDs(platform, CL_DEVICE_TYPE_GPU, 1, &dev,
-				NULL);
-	if (err == CL_DEVICE_NOT_FOUND)
-		err = clGetDeviceIDs(platform, CL_DEVICE_TYPE_CPU, 1, &dev,
-				NULL);
-	if (err < 0) {
-		fprintf(stderr, "Error %s while looking for a device.\n",
-				opencl_error_string(err));
-		exit(1);
-	}
-	return dev;
-}
-
-/* Create an OpenCL program from a string and compile it.
- */
-cl_program opencl_build_program_from_string(cl_context ctx, cl_device_id dev,
-	const char *program_source, size_t program_size,
-	const char *opencl_options)
-{
-	int err;
-	cl_program program;
-	char *program_log;
-	size_t log_size;
-
-	program = clCreateProgramWithSource(ctx, 1,
-			&program_source, &program_size, &err);
-	if (err < 0) {
-		fprintf(stderr, "Could not create the program\n");
-		exit(1);
-	}
-	err = clBuildProgram(program, 0, NULL, opencl_options, NULL, NULL);
-	if (err < 0) {
-		fprintf(stderr, "Could not build the program.\n");
-		clGetProgramBuildInfo(program, dev, CL_PROGRAM_BUILD_LOG, 0,
-				NULL, &log_size);
-		program_log = (char *) malloc(log_size + 1);
-		program_log[log_size] = '\0';
-		clGetProgramBuildInfo(program, dev, CL_PROGRAM_BUILD_LOG,
-				log_size + 1, program_log, NULL);
-		fprintf(stderr, "%s\n", program_log);
-		free(program_log);
-		exit(1);
-	}
-	return program;
-}
-
-/* Create an OpenCL program from a source file and compile it.
- */
-cl_program opencl_build_program_from_file(cl_context ctx, cl_device_id dev,
-	const char* filename, const char* opencl_options)
-{
-	cl_program program;
-	FILE *program_file;
-	char *program_source;
-	size_t program_size, read;
-
-	program_file = fopen(filename, "r");
-	if (program_file == NULL) {
-		fprintf(stderr, "Could not find the source file.\n");
-		exit(1);
-	}
-	fseek(program_file, 0, SEEK_END);
-	program_size = ftell(program_file);
-	rewind(program_file);
-	program_source = (char *) malloc(program_size + 1);
-	program_source[program_size] = '\0';
-	read = fread(program_source, sizeof(char), program_size, program_file);
-	if (read != program_size) {
-		fprintf(stderr, "Error while reading the kernel.\n");
-		exit(1);
-	}
-	fclose(program_file);
-
-	program = opencl_build_program_from_string(ctx, dev, program_source,
-						program_size, opencl_options);
-	free(program_source);
-
-	return program;
-}
diff --git a/polly/lib/External/ppcg/ocl_utilities.h b/polly/lib/External/ppcg/ocl_utilities.h
deleted file mode 100644
index 34a987e9a9a77..0000000000000
--- a/polly/lib/External/ppcg/ocl_utilities.h
+++ /dev/null
@@ -1,32 +0,0 @@
-#ifndef OCL_UTILITIES_H
-#define OCL_UTILITIES_H
-
-#if defined(__APPLE__)
-#include <OpenCL/opencl.h>
-#else
-#include <CL/opencl.h>
-#endif
-
-/* Return the OpenCL error string for a given error number.
- */
-const char *opencl_error_string(cl_int error);
-
-/* Find a GPU or a CPU associated with the first available platform.
- * If use_gpu is set, then this function first tries to look for a GPU
- * in the first available platform.
- * If this fails or if use_gpu is not set, then it tries to use the CPU.
- */
-cl_device_id opencl_create_device(int use_gpu);
-
-/* Create an OpenCL program from a string and compile it.
- */
-cl_program opencl_build_program_from_string(cl_context ctx, cl_device_id dev,
-	const char *program_source, size_t program_size,
-	const char *opencl_options);
-
-/* Create an OpenCL program from a source file and compile it.
- */
-cl_program opencl_build_program_from_file(cl_context ctx, cl_device_id dev,
-	const char* filename, const char* opencl_options);
-
-#endif
diff --git a/polly/lib/External/ppcg/opencl.h b/polly/lib/External/ppcg/opencl.h
deleted file mode 100644
index bf10f95cbf641..0000000000000
--- a/polly/lib/External/ppcg/opencl.h
+++ /dev/null
@@ -1,11 +0,0 @@
-#ifndef _OPENCL_H
-#define _OPENCL_H
-
-#include <pet.h>
-#include "ppcg_options.h"
-#include "ppcg.h"
-
-int generate_opencl(isl_ctx *ctx, struct ppcg_options *options,
-	const char *input, const char *output);
-
-#endif
diff --git a/polly/lib/External/ppcg/opencl_test.sh.in b/polly/lib/External/ppcg/opencl_test.sh.in
deleted file mode 100644
index b974d002e339c..0000000000000
--- a/polly/lib/External/ppcg/opencl_test.sh.in
+++ /dev/null
@@ -1,78 +0,0 @@
-#!/bin/sh
-
-keep=no
-
-for option; do
-	case "$option" in
-		--keep)
-			keep=yes
-			;;
-	esac
-done
-
-EXEEXT=@EXEEXT@
-VERSION=@GIT_HEAD_VERSION@
-CC="@CC@"
-CFLAGS="--std=gnu99"
-srcdir="@srcdir@"
-
-if [ $keep = "yes" ]; then
-	OUTDIR="opencl_test.$VERSION"
-	mkdir "$OUTDIR" || exit 1
-else
-	if test "x$TMPDIR" = "x"; then
-		TMPDIR=/tmp
-	fi
-	OUTDIR=`mktemp -d $TMPDIR/ppcg.XXXXXXXXXX` || exit 1
-fi
-
-run_tests () {
-	subdir=$1
-	ppcg_options=$2
-
-	echo Test with PPCG options \'$ppcg_options\'
-	mkdir ${OUTDIR}/${subdir} || exit 1
-	for i in $srcdir/tests/*.c; do
-		echo $i
-		name=`basename $i`
-		name="${name%.c}"
-		out_c="${OUTDIR}/${subdir}/$name.ppcg.c"
-		out="${OUTDIR}/${subdir}/$name.ppcg$EXEEXT"
-		options="--target=opencl --opencl-no-use-gpu $ppcg_options"
-		functions="$srcdir/tests/${name}_opencl_functions.cl"
-		if test -f $functions; then
-			options="$options --opencl-include-file=$functions"
-			options="$options --opencl-compiler-options=-I."
-		fi
-		./ppcg$EXEEXT $options $i -o "$out_c" || exit
-		$CC $CFLAGS -I "$srcdir" "$srcdir/ocl_utilities.c" -lOpenCL \
-			-I. "$out_c" -o "$out" || exit
-		$out || exit
-	done
-}
-
-run_tests default
-run_tests embed --opencl-embed-kernel-code
-
-for i in $srcdir/examples/*.c; do
-	echo $i
-	name=`basename $i`
-	name="${name%.c}"
-	exe_ref="${OUTDIR}/$name.ref$EXEEXT"
-	gen_ocl="${OUTDIR}/$name.ppcg.c"
-	exe_ocl="${OUTDIR}/$name.ppcg$EXEEXT"
-	output_ref="${OUTDIR}/$name.ref.out"
-	output_ocl="${OUTDIR}/$name.ppcg.out"
-	$CC $CFLAGS $i -o $exe_ref || exit
-	./ppcg$EXEEXT --target=opencl --opencl-no-use-gpu $i -o "$gen_ocl" || \
-		exit
-	$CC $CFLAGS -I "$srcdir" "$srcdir/ocl_utilities.c" -lOpenCL \
-		"$gen_ocl" -o "$exe_ocl" || exit
-	$exe_ref > $output_ref || exit
-	$exe_ocl > $output_ocl || exit
-	cmp $output_ref $output_ocl || exit
-done
-
-if [ $keep = "no" ]; then
-	rm -r "${OUTDIR}"
-fi
diff --git a/polly/lib/External/ppcg/polybench_test.sh.in b/polly/lib/External/ppcg/polybench_test.sh.in
deleted file mode 100644
index 4d54cb9211862..0000000000000
--- a/polly/lib/External/ppcg/polybench_test.sh.in
+++ /dev/null
@@ -1,109 +0,0 @@
-#!/bin/sh
-
-keep=no
-verbose=no
-
-for option; do
-	case "$option" in
-		--keep)
-			keep=yes
-			;;
-		--verbose)
-			verbose=yes
-			;;
-	esac
-done
-
-EXEEXT=@EXEEXT@
-DIR=@POLYBENCH_DIR@
-VERSION=@GIT_HEAD_VERSION@
-SIZE=-DMINI_DATASET
-CC="@CC@"
-HAVE_OPENCL=@HAVE_OPENCL@
-HAVE_OPENMP=@HAVE_OPENMP@
-srcdir="@srcdir@"
-if [ $keep = "yes" ]; then
-	OUTDIR="out.$VERSION"
-	mkdir "$OUTDIR" || exit 1
-else
-	if test "x$TMPDIR" = "x"; then
-		TMPDIR=/tmp
-	fi
-	OUTDIR=`mktemp -d $TMPDIR/ppcg.XXXXXXXXXX` || exit 1
-fi
-CPPFLAGS="-DPOLYBENCH_USE_C99_PROTO -DPOLYBENCH_DUMP_ARRAYS"
-CPPFLAGS="$CPPFLAGS $SIZE -I $DIR/utilities"
-CFLAGS="-lm --std=gnu99"
-
-echo "Running tests in folder ${OUTDIR}"
-
-run_tests () {
-	ext=$1
-
-	ppcg_options=$2
-	cc_options=$3
-
-	if [ "x$ppcg_options" = "x" ]; then
-		ppcg_option_str="none"
-	else
-		ppcg_option_str=$ppcg_options
-	fi
-
-	if [ "x$cc_options" = "x" ]; then
-		cc_option_str="none"
-	else
-		cc_option_str=$cc_options
-	fi
-
-	echo Test: $ext, ppcg options: $ppcg_option_str, CC options: $cc_option_str
-	for i in `cat $DIR/utilities/benchmark_list`; do
-		echo $i
-		name=`basename $i`
-		name=${name%.c}
-		source_opt="${OUTDIR}/$name.$ext.c"
-		prog_orig=${OUTDIR}/$name.orig${EXEEXT}
-		prog_opt=${OUTDIR}/$name.$ext${EXEEXT}
-		output_orig=${OUTDIR}/$name.orig.out
-		output_opt=${OUTDIR}/$name.$ext.out
-		dir=`dirname $i`
-		if [ $verbose = "yes" ]; then
-			echo ./ppcg$EXEEXT -I $DIR/$dir $DIR/$i \
-				$CPPFLAGS -o $source_opt $ppcg_options
-		fi
-		./ppcg$EXEEXT -I $DIR/$dir $DIR/$i $CPPFLAGS \
-			-o $source_opt $ppcg_options || exit
-		$CC -I $DIR/$dir $CPPFLAGS $DIR/$i -o $prog_orig \
-			$DIR/utilities/polybench.c $CFLAGS
-		$prog_orig 2> $output_orig
-		if [ $verbose = "yes" ]; then
-			echo $CC -I $DIR/$dir $CPPFLAGS $source_opt \
-				-o $prog_opt $DIR/utilities/polybench.c \
-				$CFLAGS $cc_options
-		fi
-		$CC -I $DIR/$dir $CPPFLAGS $source_opt -o $prog_opt \
-			$DIR/utilities/polybench.c $CFLAGS $cc_options || exit
-
-		$prog_opt 2> $output_opt
-		cmp $output_orig $output_opt || exit
-	done
-}
-
-run_tests ppcg "--target=c --tile"
-run_tests ppcg_live "--target=c --no-live-range-reordering --tile"
-
-# Test OpenMP code, if compiler supports openmp
-if [ $HAVE_OPENMP = "yes" ]; then
-	run_tests ppcg_omp "--target=c --openmp" -fopenmp
-	echo Introduced `grep -R 'omp parallel' "${OUTDIR}" | wc -l` '"pragma omp parallel for"'
-else
-	echo Compiler does not support OpenMP. Skipping OpenMP tests.
-fi
-
-if [ $HAVE_OPENCL = "yes" ]; then
-	run_tests ppcg_opencl "--target=opencl --opencl-no-use-gpu" \
-				"-I $srcdir $srcdir/ocl_utilities.c -lOpenCL"
-fi
-
-if [ $keep = "no" ]; then
-	rm -r "${OUTDIR}"
-fi
diff --git a/polly/lib/External/ppcg/ppcg.c b/polly/lib/External/ppcg/ppcg.c
deleted file mode 100644
index ed5aceca46abd..0000000000000
--- a/polly/lib/External/ppcg/ppcg.c
+++ /dev/null
@@ -1,1067 +0,0 @@
-/*
- * Copyright 2011      INRIA Saclay
- * Copyright 2013      Ecole Normale Superieure
- * Copyright 2015      Sven Verdoolaege
- *
- * Use of this software is governed by the MIT license
- *
- * Written by Sven Verdoolaege, INRIA Saclay - Ile-de-France,
- * Parc Club Orsay Universite, ZAC des vignes, 4 rue Jacques Monod,
- * 91893 Orsay, France
- * and Ecole Normale Superieure, 45 rue d'Ulm, 75230 Paris, France
- */
-
-#include <assert.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <isl/ctx.h>
-#include <isl/id.h>
-#include <isl/val.h>
-#include <isl/set.h>
-#include <isl/union_set.h>
-#include <isl/union_map.h>
-#include <isl/aff.h>
-#include <isl/flow.h>
-#include <isl/options.h>
-#include <isl/schedule.h>
-#include <isl/ast.h>
-#include <isl/id_to_ast_expr.h>
-#include <isl/ast_build.h>
-#include <isl/schedule.h>
-#include <pet.h>
-#include "ppcg.h"
-#include "ppcg_options.h"
-#include "cuda.h"
-#include "opencl.h"
-#include "cpu.h"
-
-struct options {
-	struct pet_options *pet;
-	struct ppcg_options *ppcg;
-	char *input;
-	char *output;
-};
-
-const char *ppcg_version(void);
-static void print_version(void)
-{
-	printf("%s", ppcg_version());
-}
-
-ISL_ARGS_START(struct options, options_args)
-ISL_ARG_CHILD(struct options, pet, "pet", &pet_options_args, "pet options")
-ISL_ARG_CHILD(struct options, ppcg, NULL, &ppcg_options_args, "ppcg options")
-ISL_ARG_STR(struct options, output, 'o', NULL,
-	"filename", NULL, "output filename (c and opencl targets)")
-ISL_ARG_ARG(struct options, input, "input", NULL)
-ISL_ARG_VERSION(print_version)
-ISL_ARGS_END
-
-ISL_ARG_DEF(options, struct options, options_args)
-
-/* Return a pointer to the final path component of "filename" or
- * to "filename" itself if it does not contain any components.
- */
-const char *ppcg_base_name(const char *filename)
-{
-	const char *base;
-
-	base = strrchr(filename, '/');
-	if (base)
-		return ++base;
-	else
-		return filename;
-}
-
-/* Copy the base name of "input" to "name" and return its length.
- * "name" is not NULL terminated.
- *
- * In particular, remove all leading directory components and
- * the final extension, if any.
- */
-int ppcg_extract_base_name(char *name, const char *input)
-{
-	const char *base;
-	const char *ext;
-	int len;
-
-	base = ppcg_base_name(input);
-	ext = strrchr(base, '.');
-	len = ext ? ext - base : strlen(base);
-
-	memcpy(name, base, len);
-
-	return len;
-}
-
-/* Does "scop" refer to any arrays that are declared, but not
- * exposed to the code after the scop?
- */
-int ppcg_scop_any_hidden_declarations(struct ppcg_scop *scop)
-{
-	int i;
-
-	if (!scop)
-		return 0;
-
-    // This is a pet feature not available in Polly.
-    return 0;
-
-	for (i = 0; i < scop->pet->n_array; ++i)
-		if (scop->pet->arrays[i]->declared &&
-		    !scop->pet->arrays[i]->exposed)
-			return 1;
-
-	return 0;
-}
-
-/* Collect all variable names that are in use in "scop".
- * In particular, collect all parameters in the context and
- * all the array names.
- * Store these names in an isl_id_to_ast_expr by mapping
- * them to a dummy value (0).
- */
-static __isl_give isl_id_to_ast_expr *collect_names(struct pet_scop *scop)
-{
-	int i, n;
-	isl_ctx *ctx;
-	isl_ast_expr *zero;
-	isl_id_to_ast_expr *names;
-
-	ctx = isl_set_get_ctx(scop->context);
-
-	n = isl_set_dim(scop->context, isl_dim_param);
-
-	names = isl_id_to_ast_expr_alloc(ctx, n + scop->n_array);
-	zero = isl_ast_expr_from_val(isl_val_zero(ctx));
-
-	for (i = 0; i < n; ++i) {
-		isl_id *id;
-
-		id = isl_set_get_dim_id(scop->context, isl_dim_param, i);
-		names = isl_id_to_ast_expr_set(names,
-						id, isl_ast_expr_copy(zero));
-	}
-
-	for (i = 0; i < scop->n_array; ++i) {
-		struct pet_array *array = scop->arrays[i];
-		isl_id *id;
-
-		id = isl_set_get_tuple_id(array->extent);
-		names = isl_id_to_ast_expr_set(names,
-						id, isl_ast_expr_copy(zero));
-	}
-
-	isl_ast_expr_free(zero);
-
-	return names;
-}
-
-/* Return an isl_id called "prefix%d", with "%d" set to "i".
- * If an isl_id with such a name already appears among the variable names
- * of "scop", then adjust the name to "prefix%d_%d".
- */
-static __isl_give isl_id *generate_name(struct ppcg_scop *scop,
-	const char *prefix, int i)
-{
-	int j;
-	char name[16];
-	isl_ctx *ctx;
-	isl_id *id;
-	int has_name;
-
-	ctx = isl_set_get_ctx(scop->context);
-	snprintf(name, sizeof(name), "%s%d", prefix, i);
-	id = isl_id_alloc(ctx, name, NULL);
-
-	j = 0;
-	while ((has_name = isl_id_to_ast_expr_has(scop->names, id)) == 1) {
-		isl_id_free(id);
-		snprintf(name, sizeof(name), "%s%d_%d", prefix, i, j++);
-		id = isl_id_alloc(ctx, name, NULL);
-	}
-
-	return has_name < 0 ? isl_id_free(id) : id;
-}
-
-/* Return a list of "n" isl_ids of the form "prefix%d".
- * If an isl_id with such a name already appears among the variable names
- * of "scop", then adjust the name to "prefix%d_%d".
- */
-__isl_give isl_id_list *ppcg_scop_generate_names(struct ppcg_scop *scop,
-	int n, const char *prefix)
-{
-	int i;
-	isl_ctx *ctx;
-	isl_id_list *names;
-
-	ctx = isl_set_get_ctx(scop->context);
-	names = isl_id_list_alloc(ctx, n);
-	for (i = 0; i < n; ++i) {
-		isl_id *id;
-
-		id = generate_name(scop, prefix, i);
-		names = isl_id_list_add(names, id);
-	}
-
-	return names;
-}
-
-/* Is "stmt" not a kill statement?
- */
-static int is_not_kill(struct pet_stmt *stmt)
-{
-	return !pet_stmt_is_kill(stmt);
-}
-
-/* Collect the iteration domains of the statements in "scop" that
- * satisfy "pred".
- */
-static __isl_give isl_union_set *collect_domains(struct pet_scop *scop,
-	int (*pred)(struct pet_stmt *stmt))
-{
-	int i;
-	isl_set *domain_i;
-	isl_union_set *domain;
-
-	if (!scop)
-		return NULL;
-
-	domain = isl_union_set_empty(isl_set_get_space(scop->context));
-
-	for (i = 0; i < scop->n_stmt; ++i) {
-		struct pet_stmt *stmt = scop->stmts[i];
-
-		if (!pred(stmt))
-			continue;
-
-		if (stmt->n_arg > 0)
-			isl_die(isl_union_set_get_ctx(domain),
-				isl_error_unsupported,
-				"data dependent conditions not supported",
-				return isl_union_set_free(domain));
-
-		domain_i = isl_set_copy(scop->stmts[i]->domain);
-		domain = isl_union_set_add_set(domain, domain_i);
-	}
-
-	return domain;
-}
-
-/* Collect the iteration domains of the statements in "scop",
- * skipping kill statements.
- */
-static __isl_give isl_union_set *collect_non_kill_domains(struct pet_scop *scop)
-{
-	return collect_domains(scop, &is_not_kill);
-}
-
-/* This function is used as a callback to pet_expr_foreach_call_expr
- * to detect if there is any call expression in the input expression.
- * Assign the value 1 to the integer that "user" points to and
- * abort the search since we have found what we were looking for.
- */
-static int set_has_call(__isl_keep pet_expr *expr, void *user)
-{
-	int *has_call = user;
-
-	*has_call = 1;
-
-	return -1;
-}
-
-/* Does "expr" contain any call expressions?
- */
-static int expr_has_call(__isl_keep pet_expr *expr)
-{
-	int has_call = 0;
-
-	if (pet_expr_foreach_call_expr(expr, &set_has_call, &has_call) < 0 &&
-	    !has_call)
-		return -1;
-
-	return has_call;
-}
-
-/* This function is a callback for pet_tree_foreach_expr.
- * If "expr" contains any call (sub)expressions, then set *has_call
- * and abort the search.
- */
-static int check_call(__isl_keep pet_expr *expr, void *user)
-{
-	int *has_call = user;
-
-	if (expr_has_call(expr))
-		*has_call = 1;
-
-	return *has_call ? -1 : 0;
-}
-
-/* Does "stmt" contain any call expressions?
- */
-static int has_call(struct pet_stmt *stmt)
-{
-	int has_call = 0;
-
-	if (pet_tree_foreach_expr(stmt->body, &check_call, &has_call) < 0 &&
-	    !has_call)
-		return -1;
-
-	return has_call;
-}
-
-/* Collect the iteration domains of the statements in "scop"
- * that contain a call expression.
- */
-static __isl_give isl_union_set *collect_call_domains(struct pet_scop *scop)
-{
-	return collect_domains(scop, &has_call);
-}
-
-/* Given a union of "tagged" access relations of the form
- *
- *	[S_i[...] -> R_j[]] -> A_k[...]
- *
- * project out the "tags" (R_j[]).
- * That is, return a union of relations of the form
- *
- *	S_i[...] -> A_k[...]
- */
-static __isl_give isl_union_map *project_out_tags(
-	__isl_take isl_union_map *umap)
-{
-	return isl_union_map_domain_factor_domain(umap);
-}
-
-/* Construct a function from tagged iteration domains to the corresponding
- * untagged iteration domains with as range of the wrapped map in the domain
- * the reference tags that appear in any of the reads, writes or kills.
- * Store the result in ps->tagger.
- *
- * For example, if the statement with iteration space S[i,j]
- * contains two array references R_1[] and R_2[], then ps->tagger will contain
- *
- *	{ [S[i,j] -> R_1[]] -> S[i,j]; [S[i,j] -> R_2[]] -> S[i,j] }
- */
-void compute_tagger(struct ppcg_scop *ps)
-{
-	isl_union_map *tagged;
-	isl_union_pw_multi_aff *tagger;
-
-	tagged = isl_union_map_copy(ps->tagged_reads);
-	tagged = isl_union_map_union(tagged,
-				isl_union_map_copy(ps->tagged_may_writes));
-	tagged = isl_union_map_union(tagged,
-				isl_union_map_copy(ps->tagged_must_kills));
-	tagged = isl_union_map_universe(tagged);
-	tagged = isl_union_set_unwrap(isl_union_map_domain(tagged));
-
-	tagger = isl_union_map_domain_map_union_pw_multi_aff(tagged);
-
-	ps->tagger = tagger;
-}
-
-/* Compute the live out accesses, i.e., the writes that are
- * potentially not killed by any kills or any other writes, and
- * store them in ps->live_out.
- *
- * We compute the "dependence" of any "kill" (an explicit kill
- * or a must write) on any may write.
- * The elements accessed by the may writes with a "depending" kill
- * also accessing the element are definitely killed.
- * The remaining may writes can potentially be live out.
- *
- * The result of the dependence analysis is
- *
- *	{ IW -> [IK -> A] }
- *
- * with IW the instance of the write statement, IK the instance of kill
- * statement and A the element that was killed.
- * The range factor range is
- *
- *	{ IW -> A }
- *
- * containing all such pairs for which there is a kill statement instance,
- * i.e., all pairs that have been killed.
- */
-static void compute_live_out(struct ppcg_scop *ps)
-{
-	isl_schedule *schedule;
-	isl_union_map *kills;
-	isl_union_map *exposed;
-	isl_union_map *covering;
-	isl_union_access_info *access;
-	isl_union_flow *flow;
-
-	schedule = isl_schedule_copy(ps->schedule);
-	kills = isl_union_map_union(isl_union_map_copy(ps->must_writes),
-				    isl_union_map_copy(ps->must_kills));
-	access = isl_union_access_info_from_sink(kills);
-	access = isl_union_access_info_set_may_source(access,
-				    isl_union_map_copy(ps->may_writes));
-	access = isl_union_access_info_set_schedule(access, schedule);
-	flow = isl_union_access_info_compute_flow(access);
-	covering = isl_union_flow_get_full_may_dependence(flow);
-	isl_union_flow_free(flow);
-
-	covering = isl_union_map_range_factor_range(covering);
-	exposed = isl_union_map_copy(ps->may_writes);
-	exposed = isl_union_map_subtract(exposed, covering);
-	ps->live_out = exposed;
-}
-
-/* Compute the tagged flow dependences and the live_in accesses and store
- * the results in ps->tagged_dep_flow and ps->live_in.
- *
- * We allow both the must writes and the must kills to serve as
- * definite sources such that a subsequent read would not depend
- * on any earlier write.  The resulting flow dependences with
- * a must kill as source reflect possibly uninitialized reads.
- * No dependences need to be introduced to protect such reads
- * (other than those imposed by potential flows from may writes
- * that follow the kill).  We therefore remove those flow dependences.
- * This is also useful for the dead code elimination, which assumes
- * the flow sources are non-kill instances.
- */
-static void compute_tagged_flow_dep_only(struct ppcg_scop *ps)
-{
-	isl_union_pw_multi_aff *tagger;
-	isl_schedule *schedule;
-	isl_union_map *live_in;
-	isl_union_access_info *access;
-	isl_union_flow *flow;
-	isl_union_map *must_source;
-	isl_union_map *kills;
-	isl_union_map *tagged_flow;
-
-	tagger = isl_union_pw_multi_aff_copy(ps->tagger);
-	schedule = isl_schedule_copy(ps->schedule);
-	schedule = isl_schedule_pullback_union_pw_multi_aff(schedule, tagger);
-	kills = isl_union_map_copy(ps->tagged_must_kills);
-	must_source = isl_union_map_copy(ps->tagged_must_writes);
-	must_source = isl_union_map_union(must_source,
-				isl_union_map_copy(kills));
-	access = isl_union_access_info_from_sink(
-				isl_union_map_copy(ps->tagged_reads));
-	access = isl_union_access_info_set_must_source(access, must_source);
-	access = isl_union_access_info_set_may_source(access,
-				isl_union_map_copy(ps->tagged_may_writes));
-	access = isl_union_access_info_set_schedule(access, schedule);
-	flow = isl_union_access_info_compute_flow(access);
-	tagged_flow = isl_union_flow_get_may_dependence(flow);
-	tagged_flow = isl_union_map_subtract_domain(tagged_flow,
-				isl_union_map_domain(kills));
-	ps->tagged_dep_flow = tagged_flow;
-	live_in = isl_union_flow_get_may_no_source(flow);
-	ps->live_in = project_out_tags(live_in);
-	isl_union_flow_free(flow);
-}
-
-/* Compute ps->dep_flow from ps->tagged_dep_flow
- * by projecting out the reference tags.
- */
-static void derive_flow_dep_from_tagged_flow_dep(struct ppcg_scop *ps)
-{
-	ps->dep_flow = isl_union_map_copy(ps->tagged_dep_flow);
-	ps->dep_flow = isl_union_map_factor_domain(ps->dep_flow);
-}
-
-/* Compute the flow dependences and the live_in accesses and store
- * the results in ps->dep_flow and ps->live_in.
- * A copy of the flow dependences, tagged with the reference tags
- * is stored in ps->tagged_dep_flow.
- *
- * We first compute ps->tagged_dep_flow, i.e., the tagged flow dependences
- * and then project out the tags.
- */
-static void compute_tagged_flow_dep(struct ppcg_scop *ps)
-{
-	compute_tagged_flow_dep_only(ps);
-	derive_flow_dep_from_tagged_flow_dep(ps);
-}
-
-/* Compute the order dependences that prevent the potential live ranges
- * from overlapping.
- *
- * In particular, construct a union of relations
- *
- *	[R[...] -> R_1[]] -> [W[...] -> R_2[]]
- *
- * where [R[...] -> R_1[]] is the range of one or more live ranges
- * (i.e., a read) and [W[...] -> R_2[]] is the domain of one or more
- * live ranges (i.e., a write).  Moreover, the read and the write
- * access the same memory element and the read occurs before the write
- * in the original schedule.
- * The scheduler allows some of these dependences to be violated, provided
- * the adjacent live ranges are all local (i.e., their domain and range
- * are mapped to the same point by the current schedule band).
- *
- * Note that if a live range is not local, then we need to make
- * sure it does not overlap with _any_ other live range, and not
- * just with the "previous" and/or the "next" live range.
- * We therefore add order dependences between reads and
- * _any_ later potential write.
- *
- * We also need to be careful about writes without a corresponding read.
- * They are already prevented from moving past non-local preceding
- * intervals, but we also need to prevent them from moving past non-local
- * following intervals.  We therefore also add order dependences from
- * potential writes that do not appear in any intervals
- * to all later potential writes.
- * Note that dead code elimination should have removed most of these
- * dead writes, but the dead code elimination may not remove all dead writes,
- * so we need to consider them to be safe.
- *
- * The order dependences are computed by computing the "dataflow"
- * from the above unmatched writes and the reads to the may writes.
- * The unmatched writes and the reads are treated as may sources
- * such that they would not kill order dependences from earlier
- * such writes and reads.
- */
-static void compute_order_dependences(struct ppcg_scop *ps)
-{
-	isl_union_map *reads;
-	isl_union_map *shared_access;
-	isl_union_set *matched;
-	isl_union_map *unmatched;
-	isl_union_pw_multi_aff *tagger;
-	isl_schedule *schedule;
-	isl_union_access_info *access;
-	isl_union_flow *flow;
-
-	tagger = isl_union_pw_multi_aff_copy(ps->tagger);
-	schedule = isl_schedule_copy(ps->schedule);
-	schedule = isl_schedule_pullback_union_pw_multi_aff(schedule, tagger);
-	reads = isl_union_map_copy(ps->tagged_reads);
-	matched = isl_union_map_domain(isl_union_map_copy(ps->tagged_dep_flow));
-	unmatched = isl_union_map_copy(ps->tagged_may_writes);
-	unmatched = isl_union_map_subtract_domain(unmatched, matched);
-	reads = isl_union_map_union(reads, unmatched);
-	access = isl_union_access_info_from_sink(
-				isl_union_map_copy(ps->tagged_may_writes));
-	access = isl_union_access_info_set_may_source(access, reads);
-	access = isl_union_access_info_set_schedule(access, schedule);
-	flow = isl_union_access_info_compute_flow(access);
-	shared_access = isl_union_flow_get_may_dependence(flow);
-	isl_union_flow_free(flow);
-
-	ps->tagged_dep_order = isl_union_map_copy(shared_access);
-	ps->dep_order = isl_union_map_factor_domain(shared_access);
-}
-
-/* Compute those validity dependences of the program represented by "scop"
- * that should be unconditionally enforced even when live-range reordering
- * is used.
- *
- * In particular, compute the external false dependences
- * as well as order dependences between sources with the same sink.
- * The anti-dependences are already taken care of by the order dependences.
- * The external false dependences are only used to ensure that live-in and
- * live-out data is not overwritten by any writes inside the scop.
- * The independences are removed from the external false dependences,
- * but not from the order dependences between sources with the same sink.
- *
- * In particular, the reads from live-in data need to precede any
- * later write to the same memory element.
- * As to live-out data, the last writes need to remain the last writes.
- * That is, any earlier write in the original schedule needs to precede
- * the last write to the same memory element in the computed schedule.
- * The possible last writes have been computed by compute_live_out.
- * They may include kills, but if the last access is a kill,
- * then the corresponding dependences will effectively be ignored
- * since we do not schedule any kill statements.
- *
- * Note that the set of live-in and live-out accesses may be
- * an overapproximation.  There may therefore be potential writes
- * before a live-in access and after a live-out access.
- *
- * In the presence of may-writes, there may be multiple live-ranges
- * with the same sink, accessing the same memory element.
- * The sources of these live-ranges need to be executed
- * in the same relative order as in the original program
- * since we do not know which of the may-writes will actually
- * perform a write.  Consider all sources that share a sink and
- * that may write to the same memory element and compute
- * the order dependences among them.
- */
-static void compute_forced_dependences(struct ppcg_scop *ps)
-{
-	isl_union_map *shared_access;
-	isl_union_map *exposed;
-	isl_union_map *live_in;
-	isl_union_map *sink_access;
-	isl_union_map *shared_sink;
-	isl_union_access_info *access;
-	isl_union_flow *flow;
-	isl_schedule *schedule;
-
-	exposed = isl_union_map_copy(ps->live_out);
-	schedule = isl_schedule_copy(ps->schedule);
-	access = isl_union_access_info_from_sink(exposed);
-	access = isl_union_access_info_set_may_source(access,
-				isl_union_map_copy(ps->may_writes));
-	access = isl_union_access_info_set_schedule(access, schedule);
-	flow = isl_union_access_info_compute_flow(access);
-	shared_access = isl_union_flow_get_may_dependence(flow);
-	isl_union_flow_free(flow);
-	ps->dep_forced = shared_access;
-
-	schedule = isl_schedule_copy(ps->schedule);
-	access = isl_union_access_info_from_sink(
-				isl_union_map_copy(ps->may_writes));
-	access = isl_union_access_info_set_may_source(access,
-				isl_union_map_copy(ps->live_in));
-	access = isl_union_access_info_set_schedule(access, schedule);
-	flow = isl_union_access_info_compute_flow(access);
-	live_in = isl_union_flow_get_may_dependence(flow);
-	isl_union_flow_free(flow);
-
-	ps->dep_forced = isl_union_map_union(ps->dep_forced, live_in);
-	ps->dep_forced = isl_union_map_subtract(ps->dep_forced,
-				isl_union_map_copy(ps->independence));
-
-	schedule = isl_schedule_copy(ps->schedule);
-	sink_access = isl_union_map_copy(ps->tagged_dep_flow);
-	sink_access = isl_union_map_range_product(sink_access,
-				isl_union_map_copy(ps->tagged_may_writes));
-	sink_access = isl_union_map_domain_factor_domain(sink_access);
-	access = isl_union_access_info_from_sink(
-				isl_union_map_copy(sink_access));
-	access = isl_union_access_info_set_may_source(access, sink_access);
-	access = isl_union_access_info_set_schedule(access, schedule);
-	flow = isl_union_access_info_compute_flow(access);
-	shared_sink = isl_union_flow_get_may_dependence(flow);
-	isl_union_flow_free(flow);
-	ps->dep_forced = isl_union_map_union(ps->dep_forced, shared_sink);
-}
-
-/* Remove independence from the tagged flow dependences.
- * Since the user has guaranteed that source and sink of an independence
- * can be executed in any order, there cannot be a flow dependence
- * between them, so they can be removed from the set of flow dependences.
- * However, if the source of such a flow dependence is a must write,
- * then it may have killed other potential sources, which would have
- * to be recovered if we were to remove those flow dependences.
- * We therefore keep the flow dependences that originate in a must write,
- * even if it corresponds to a known independence.
- */
-static void remove_independences_from_tagged_flow(struct ppcg_scop *ps)
-{
-	isl_union_map *tf;
-	isl_union_set *indep;
-	isl_union_set *mw;
-
-	tf = isl_union_map_copy(ps->tagged_dep_flow);
-	tf = isl_union_map_zip(tf);
-	indep = isl_union_map_wrap(isl_union_map_copy(ps->independence));
-	tf = isl_union_map_intersect_domain(tf, indep);
-	tf = isl_union_map_zip(tf);
-	mw = isl_union_map_domain(isl_union_map_copy(ps->tagged_must_writes));
-	tf = isl_union_map_subtract_domain(tf, mw);
-	ps->tagged_dep_flow = isl_union_map_subtract(ps->tagged_dep_flow, tf);
-}
-
-/* Compute the dependences of the program represented by "scop"
- * in case live range reordering is allowed.
- *
- * We compute the actual live ranges and the corresponding order
- * false dependences.
- *
- * The independences are removed from the flow dependences
- * (provided the source is not a must-write) as well as
- * from the external false dependences (by compute_forced_dependences).
- */
-static void compute_live_range_reordering_dependences(struct ppcg_scop *ps)
-{
-	compute_tagged_flow_dep_only(ps);
-	remove_independences_from_tagged_flow(ps);
-	derive_flow_dep_from_tagged_flow_dep(ps);
-	compute_order_dependences(ps);
-	compute_forced_dependences(ps);
-}
-
-/* Compute the potential flow dependences and the potential live in
- * accesses.
- */
-static void compute_flow_dep(struct ppcg_scop *ps)
-{
-	isl_union_access_info *access;
-	isl_union_flow *flow;
-
-	access = isl_union_access_info_from_sink(isl_union_map_copy(ps->reads));
-	access = isl_union_access_info_set_must_source(access,
-				isl_union_map_copy(ps->must_writes));
-	access = isl_union_access_info_set_may_source(access,
-				isl_union_map_copy(ps->may_writes));
-	access = isl_union_access_info_set_schedule(access,
-				isl_schedule_copy(ps->schedule));
-	flow = isl_union_access_info_compute_flow(access);
-
-	ps->dep_flow = isl_union_flow_get_may_dependence(flow);
-	ps->live_in = isl_union_flow_get_may_no_source(flow);
-	isl_union_flow_free(flow);
-}
-
-/* Compute the dependences of the program represented by "scop".
- * Store the computed potential flow dependences
- * in scop->dep_flow and the reads with potentially no corresponding writes in
- * scop->live_in.
- * Store the potential live out accesses in scop->live_out.
- * Store the potential false (anti and output) dependences in scop->dep_false.
- *
- * If live range reordering is allowed, then we compute a separate
- * set of order dependences and a set of external false dependences
- * in compute_live_range_reordering_dependences.
- */
-void compute_dependences(struct ppcg_scop *scop)
-{
-	isl_union_map *may_source;
-	isl_union_access_info *access;
-	isl_union_flow *flow;
-
-	if (!scop)
-		return;
-
-	compute_live_out(scop);
-
-	if (scop->options->live_range_reordering)
-		compute_live_range_reordering_dependences(scop);
-	else if (scop->options->target != PPCG_TARGET_C)
-		compute_tagged_flow_dep(scop);
-	else
-		compute_flow_dep(scop);
-
-	may_source = isl_union_map_union(isl_union_map_copy(scop->may_writes),
-					isl_union_map_copy(scop->reads));
-	access = isl_union_access_info_from_sink(
-				isl_union_map_copy(scop->may_writes));
-	access = isl_union_access_info_set_must_source(access,
-				isl_union_map_copy(scop->must_writes));
-	access = isl_union_access_info_set_may_source(access, may_source);
-	access = isl_union_access_info_set_schedule(access,
-				isl_schedule_copy(scop->schedule));
-	flow = isl_union_access_info_compute_flow(access);
-
-	scop->dep_false = isl_union_flow_get_may_dependence(flow);
-	scop->dep_false = isl_union_map_coalesce(scop->dep_false);
-	isl_union_flow_free(flow);
-}
-
-/* Eliminate dead code from ps->domain.
- *
- * In particular, intersect both ps->domain and the domain of
- * ps->schedule with the (parts of) iteration
- * domains that are needed to produce the output or for statement
- * iterations that call functions.
- * Also intersect the range of the dataflow dependences with
- * this domain such that the removed instances will no longer
- * be considered as targets of dataflow.
- *
- * We start with the iteration domains that call functions
- * and the set of iterations that last write to an array
- * (except those that are later killed).
- *
- * Then we add those statement iterations that produce
- * something needed by the "live" statements iterations.
- * We keep doing this until no more statement iterations can be added.
- * To ensure that the procedure terminates, we compute the affine
- * hull of the live iterations (bounded to the original iteration
- * domains) each time we have added extra iterations.
- */
-void eliminate_dead_code(struct ppcg_scop *ps)
-{
-	isl_union_set *live;
-	isl_union_map *dep;
-	isl_union_pw_multi_aff *tagger;
-
-	live = isl_union_map_domain(isl_union_map_copy(ps->live_out));
-	if (!isl_union_set_is_empty(ps->call)) {
-		live = isl_union_set_union(live, isl_union_set_copy(ps->call));
-		live = isl_union_set_coalesce(live);
-	}
-
-	dep = isl_union_map_copy(ps->dep_flow);
-	dep = isl_union_map_reverse(dep);
-
-	for (;;) {
-		isl_union_set *extra;
-
-		extra = isl_union_set_apply(isl_union_set_copy(live),
-					    isl_union_map_copy(dep));
-		if (isl_union_set_is_subset(extra, live)) {
-			isl_union_set_free(extra);
-			break;
-		}
-
-		live = isl_union_set_union(live, extra);
-		live = isl_union_set_affine_hull(live);
-		live = isl_union_set_intersect(live,
-					    isl_union_set_copy(ps->domain));
-	}
-
-	isl_union_map_free(dep);
-
-	ps->domain = isl_union_set_intersect(ps->domain,
-						isl_union_set_copy(live));
-	ps->schedule = isl_schedule_intersect_domain(ps->schedule,
-						isl_union_set_copy(live));
-	ps->dep_flow = isl_union_map_intersect_range(ps->dep_flow,
-						isl_union_set_copy(live));
-	tagger = isl_union_pw_multi_aff_copy(ps->tagger);
-	live = isl_union_set_preimage_union_pw_multi_aff(live, tagger);
-	ps->tagged_dep_flow = isl_union_map_intersect_range(ps->tagged_dep_flow,
-						live);
-}
-
-/* Intersect "set" with the set described by "str", taking the NULL
- * string to represent the universal set.
- */
-static __isl_give isl_set *set_intersect_str(__isl_take isl_set *set,
-	const char *str)
-{
-	isl_ctx *ctx;
-	isl_set *set2;
-
-	if (!str)
-		return set;
-
-	ctx = isl_set_get_ctx(set);
-	set2 = isl_set_read_from_str(ctx, str);
-	set = isl_set_intersect(set, set2);
-
-	return set;
-}
-
-void *ppcg_scop_free(struct ppcg_scop *ps)
-{
-	if (!ps)
-		return NULL;
-
-	isl_set_free(ps->context);
-	isl_union_set_free(ps->domain);
-	isl_union_set_free(ps->call);
-	isl_union_map_free(ps->tagged_reads);
-	isl_union_map_free(ps->reads);
-	isl_union_map_free(ps->live_in);
-	isl_union_map_free(ps->tagged_may_writes);
-	isl_union_map_free(ps->tagged_must_writes);
-	isl_union_map_free(ps->may_writes);
-	isl_union_map_free(ps->must_writes);
-	isl_union_map_free(ps->live_out);
-	isl_union_map_free(ps->tagged_must_kills);
-	isl_union_map_free(ps->must_kills);
-	isl_union_map_free(ps->tagged_dep_flow);
-	isl_union_map_free(ps->dep_flow);
-	isl_union_map_free(ps->dep_false);
-	isl_union_map_free(ps->dep_forced);
-	isl_union_map_free(ps->tagged_dep_order);
-	isl_union_map_free(ps->dep_order);
-	isl_schedule_free(ps->schedule);
-	isl_union_pw_multi_aff_free(ps->tagger);
-	isl_union_map_free(ps->independence);
-	isl_id_to_ast_expr_free(ps->names);
-
-	free(ps);
-
-	return NULL;
-}
-
-/* Extract a ppcg_scop from a pet_scop.
- *
- * The constructed ppcg_scop refers to elements from the pet_scop
- * so the pet_scop should not be freed before the ppcg_scop.
- */
-static struct ppcg_scop *ppcg_scop_from_pet_scop(struct pet_scop *scop,
-	struct ppcg_options *options)
-{
-	int i;
-	isl_ctx *ctx;
-	struct ppcg_scop *ps;
-
-	if (!scop)
-		return NULL;
-
-	ctx = isl_set_get_ctx(scop->context);
-
-	ps = isl_calloc_type(ctx, struct ppcg_scop);
-	if (!ps)
-		return NULL;
-
-	ps->names = collect_names(scop);
-	ps->options = options;
-	ps->start = pet_loc_get_start(scop->loc);
-	ps->end = pet_loc_get_end(scop->loc);
-	ps->context = isl_set_copy(scop->context);
-	ps->context = set_intersect_str(ps->context, options->ctx);
-	if (options->non_negative_parameters) {
-		isl_space *space = isl_set_get_space(ps->context);
-		isl_set *nn = isl_set_nat_universe(space);
-		ps->context = isl_set_intersect(ps->context, nn);
-	}
-	ps->domain = collect_non_kill_domains(scop);
-	ps->call = collect_call_domains(scop);
-	ps->tagged_reads = pet_scop_get_tagged_may_reads(scop);
-	ps->reads = pet_scop_get_may_reads(scop);
-	ps->tagged_may_writes = pet_scop_get_tagged_may_writes(scop);
-	ps->may_writes = pet_scop_get_may_writes(scop);
-	ps->tagged_must_writes = pet_scop_get_tagged_must_writes(scop);
-	ps->must_writes = pet_scop_get_must_writes(scop);
-	ps->tagged_must_kills = pet_scop_get_tagged_must_kills(scop);
-	ps->must_kills = pet_scop_get_must_kills(scop);
-	ps->schedule = isl_schedule_copy(scop->schedule);
-	ps->pet = scop;
-	ps->independence = isl_union_map_empty(isl_set_get_space(ps->context));
-	for (i = 0; i < scop->n_independence; ++i)
-		ps->independence = isl_union_map_union(ps->independence,
-			isl_union_map_copy(scop->independences[i]->filter));
-
-	compute_tagger(ps);
-	compute_dependences(ps);
-	eliminate_dead_code(ps);
-
-	if (!ps->context || !ps->domain || !ps->call || !ps->reads ||
-	    !ps->may_writes || !ps->must_writes || !ps->tagged_must_kills ||
-	    !ps->must_kills || !ps->schedule || !ps->independence || !ps->names)
-		return ppcg_scop_free(ps);
-
-	return ps;
-}
-
-/* Internal data structure for ppcg_transform.
- */
-struct ppcg_transform_data {
-	struct ppcg_options *options;
-	__isl_give isl_printer *(*transform)(__isl_take isl_printer *p,
-		struct ppcg_scop *scop, void *user);
-	void *user;
-};
-
-/* Should we print the original code?
- * That is, does "scop" involve any data dependent conditions or
- * nested expressions that cannot be handled by pet_stmt_build_ast_exprs?
- */
-static int print_original(struct pet_scop *scop, struct ppcg_options *options)
-{
-	if (!pet_scop_can_build_ast_exprs(scop)) {
-		if (options->debug->verbose)
-			fprintf(stdout, "Printing original code because "
-				"some index expressions cannot currently "
-				"be printed\n");
-		return 1;
-	}
-
-	if (pet_scop_has_data_dependent_conditions(scop)) {
-		if (options->debug->verbose)
-			fprintf(stdout, "Printing original code because "
-				"input involves data dependent conditions\n");
-		return 1;
-	}
-
-	return 0;
-}
-
-/* Callback for pet_transform_C_source that transforms
- * the given pet_scop to a ppcg_scop before calling the
- * ppcg_transform callback.
- *
- * If "scop" contains any data dependent conditions or if we may
- * not be able to print the transformed program, then just print
- * the original code.
- */
-static __isl_give isl_printer *transform(__isl_take isl_printer *p,
-	struct pet_scop *scop, void *user)
-{
-	struct ppcg_transform_data *data = user;
-	struct ppcg_scop *ps;
-
-	if (print_original(scop, data->options)) {
-		p = pet_scop_print_original(scop, p);
-		pet_scop_free(scop);
-		return p;
-	}
-
-	scop = pet_scop_align_params(scop);
-	ps = ppcg_scop_from_pet_scop(scop, data->options);
-
-	p = data->transform(p, ps, data->user);
-
-	ppcg_scop_free(ps);
-	pet_scop_free(scop);
-
-	return p;
-}
-
-/* Transform the C source file "input" by rewriting each scop
- * through a call to "transform".
- * The transformed C code is written to "out".
- *
- * This is a wrapper around pet_transform_C_source that transforms
- * the pet_scop to a ppcg_scop before calling "fn".
- */
-int ppcg_transform(isl_ctx *ctx, const char *input, FILE *out,
-	struct ppcg_options *options,
-	__isl_give isl_printer *(*fn)(__isl_take isl_printer *p,
-		struct ppcg_scop *scop, void *user), void *user)
-{
-	struct ppcg_transform_data data = { options, fn, user };
-	return pet_transform_C_source(ctx, input, out, &transform, &data);
-}
-
-/* Check consistency of options.
- *
- * Return -1 on error.
- */
-static int check_options(isl_ctx *ctx)
-{
-	struct options *options;
-
-	options = isl_ctx_peek_options(ctx, &options_args);
-	if (!options)
-		isl_die(ctx, isl_error_internal,
-			"unable to find options", return -1);
-
-	if (options->ppcg->openmp &&
-	    !isl_options_get_ast_build_atomic_upper_bound(ctx))
-		isl_die(ctx, isl_error_invalid,
-			"OpenMP requires atomic bounds", return -1);
-
-	return 0;
-}
-
-#if 0
-int main(int argc, char **argv)
-{
-	int r;
-	isl_ctx *ctx;
-	struct options *options;
-
-	options = options_new_with_defaults();
-	assert(options);
-
-	ctx = isl_ctx_alloc_with_options(&options_args, options);
-	ppcg_options_set_target_defaults(options->ppcg);
-	isl_options_set_ast_build_detect_min_max(ctx, 1);
-	isl_options_set_ast_print_macro_once(ctx, 1);
-	isl_options_set_schedule_whole_component(ctx, 0);
-	isl_options_set_schedule_maximize_band_depth(ctx, 1);
-	isl_options_set_schedule_maximize_coincidence(ctx, 1);
-	pet_options_set_encapsulate_dynamic_control(ctx, 1);
-	argc = options_parse(options, argc, argv, ISL_ARG_ALL);
-
-	if (check_options(ctx) < 0)
-		r = EXIT_FAILURE;
-	else if (options->ppcg->target == PPCG_TARGET_CUDA)
-		r = generate_cuda(ctx, options->ppcg, options->input);
-	else if (options->ppcg->target == PPCG_TARGET_OPENCL)
-		r = generate_opencl(ctx, options->ppcg, options->input,
-				options->output);
-	else
-		r = generate_cpu(ctx, options->ppcg, options->input,
-				options->output);
-
-	isl_ctx_free(ctx);
-
-	return r;
-}
-#endif
diff --git a/polly/lib/External/ppcg/ppcg.h b/polly/lib/External/ppcg/ppcg.h
deleted file mode 100644
index ef89683dc1f2b..0000000000000
--- a/polly/lib/External/ppcg/ppcg.h
+++ /dev/null
@@ -1,128 +0,0 @@
-#ifndef PPCG_H
-#define PPCG_H
-
-#include <isl/schedule.h>
-#include <isl/set.h>
-#include <isl/union_set.h>
-#include <isl/union_map.h>
-#include <isl/id_to_ast_expr.h>
-#include <pet.h>
-
-#include "ppcg_options.h"
-
-const char *ppcg_base_name(const char *filename);
-int ppcg_extract_base_name(char *name, const char *input);
-
-/* Representation of the scop for use inside PPCG.
- *
- * "options" are the options specified by the user.
- * Some fields in this structure may depend on some of the options.
- *
- * "start" and "end" are file offsets of the corresponding program text.
- * "context" represents constraints on the parameters.
- * "domain" is the union of all iteration domains.
- * "call" contains the iteration domains of statements with a call expression.
- * "reads" contains all potential read accesses.
- * "tagged_reads" is the same as "reads", except that the domain is a wrapped
- *	relation mapping an iteration domain to a reference identifier
- * "live_in" contains the potential read accesses that potentially
- *	have no corresponding writes in the scop.
- * "may_writes" contains all potential write accesses.
- * "tagged_may_writes" is the same as "may_writes", except that the domain
- *	is a wrapped relation mapping an iteration domain
- *	to a reference identifier
- * "must_writes" contains all definite write accesses.
- * "tagged_must_writes" is the same as "must_writes", except that the domain
- *	is a wrapped relation mapping an iteration domain
- *	to a reference identifier
- * "live_out" contains the potential write accesses that are potentially
- *	not killed by any kills or any other writes.
- * "must_kills" contains all definite kill accesses.
- * "tagged_must_kills" is the same as "must_kills", except that the domain
- *	is a wrapped relation mapping an iteration domain
- *	to a reference identifier.
- *
- * "tagger" maps tagged iteration domains to the corresponding untagged
- *	iteration domain.
- *
- * "independence" is the union of all independence filters.
- *
- * "dep_flow" represents the potential flow dependences.
- * "tagged_dep_flow" is the same as "dep_flow", except that both domain and
- *	range are wrapped relations mapping an iteration domain to
- *	a reference identifier.  May be NULL if not computed.
- * "dep_false" represents the potential false (anti and output) dependences.
- * "dep_forced" represents the validity constraints that should be enforced
- *	even when live-range reordering is used.
- *	In particular, these constraints ensure that all live-in
- *	accesses remain live-in and that all live-out accesses remain live-out
- *	and that multiple potential sources for the same read are
- *	executed in the original order.
- * "dep_order"/"tagged_dep_order" represents the order dependences between
- *	the live range intervals in "dep_flow"/"tagged_dep_flow".
- *	It is only used if the live_range_reordering
- *	option is set.  Otherwise it is NULL.
- *	If "dep_order" is used, then "dep_false" only contains a limited
- *	set of anti and output dependences.
- * "schedule" represents the (original) schedule.
- *
- * "names" contains all variable names that are in use by the scop.
- * The names are mapped to a dummy value.
- *
- * "pet" is the original pet_scop.
- */
-struct ppcg_scop {
-	struct ppcg_options *options;
-
-	unsigned start;
-	unsigned end;
-
-	isl_set *context;
-	isl_union_set *domain;
-	isl_union_set *call;
-	isl_union_map *tagged_reads;
-	isl_union_map *reads;
-	isl_union_map *live_in;
-	isl_union_map *tagged_may_writes;
-	isl_union_map *may_writes;
-	isl_union_map *tagged_must_writes;
-	isl_union_map *must_writes;
-	isl_union_map *live_out;
-	isl_union_map *tagged_must_kills;
-	isl_union_map *must_kills;
-
-	isl_union_pw_multi_aff *tagger;
-
-	isl_union_map *independence;
-
-	isl_union_map *dep_flow;
-	isl_union_map *tagged_dep_flow;
-	isl_union_map *dep_false;
-	isl_union_map *dep_forced;
-	isl_union_map *dep_order;
-	isl_union_map *tagged_dep_order;
-	isl_schedule *schedule;
-
-	isl_id_to_ast_expr *names;
-
-	struct pet_scop *pet;
-};
-
-int ppcg_scop_any_hidden_declarations(struct ppcg_scop *scop);
-__isl_give isl_id_list *ppcg_scop_generate_names(struct ppcg_scop *scop,
-	int n, const char *prefix);
-
-int ppcg_transform(isl_ctx *ctx, const char *input, FILE *out,
-	struct ppcg_options *options,
-	__isl_give isl_printer *(*fn)(__isl_take isl_printer *p,
-		struct ppcg_scop *scop, void *user), void *user);
-
-__isl_give isl_schedule *ppcg_compute_schedule(
-	__isl_take isl_schedule_constraints *sc,
-	__isl_keep isl_schedule *schedule, struct ppcg_options *options);
-
-void compute_tagger(struct ppcg_scop *ps);
-void compute_dependences(struct ppcg_scop *scop);
-void eliminate_dead_code(struct ppcg_scop *ps);
-void *ppcg_scop_free(struct ppcg_scop *ps);
-#endif
diff --git a/polly/lib/External/ppcg/ppcg_options.c b/polly/lib/External/ppcg/ppcg_options.c
deleted file mode 100644
index 77b0f29de016f..0000000000000
--- a/polly/lib/External/ppcg/ppcg_options.c
+++ /dev/null
@@ -1,136 +0,0 @@
-/*
- * Copyright 2010-2011 INRIA Saclay
- *
- * Use of this software is governed by the MIT license
- *
- * Written by Sven Verdoolaege, INRIA Saclay - Ile-de-France,
- * Parc Club Orsay Universite, ZAC des vignes, 4 rue Jacques Monod,
- * 91893 Orsay, France
- */
-
-#include "ppcg_options.h"
-
-static struct isl_arg_choice target[] = {
-	{"c",		PPCG_TARGET_C},
-	{"cuda",	PPCG_TARGET_CUDA},
-	{"opencl",      PPCG_TARGET_OPENCL},
-	{0}
-};
-
-/* Set defaults that depend on the target.
- * In particular, set --schedule-outer-coincidence iff target is a GPU.
- */
-void ppcg_options_set_target_defaults(struct ppcg_options *options)
-{
-	char *argv[2] = { NULL };
-
-	argv[0] = "ppcg_options_set_target_defaults";
-	if (options->target == PPCG_TARGET_C)
-		argv[1] = "--no-schedule-outer-coincidence";
-	else
-		argv[1] = "--schedule-outer-coincidence";
-
-	isl_options_parse(options->isl, 2, argv, ISL_ARG_ALL);
-}
-
-/* Callback that is called whenever the "target" option is set (to "val").
- * The callback is called after target has been updated.
- *
- * Call ppcg_options_set_target_defaults to reset the target-dependent options.
- */
-static int set_target(void *opt, unsigned val)
-{
-	struct ppcg_options *options = opt;
-
-	ppcg_options_set_target_defaults(options);
-
-	return 0;
-}
-
-ISL_ARGS_START(struct ppcg_debug_options, ppcg_debug_options_args)
-ISL_ARG_BOOL(struct ppcg_debug_options, dump_schedule_constraints, 0,
-	"dump-schedule-constraints", 0, "dump schedule constraints")
-ISL_ARG_BOOL(struct ppcg_debug_options, dump_schedule, 0,
-	"dump-schedule", 0, "dump isl computed schedule")
-ISL_ARG_BOOL(struct ppcg_debug_options, dump_final_schedule, 0,
-	"dump-final-schedule", 0, "dump PPCG computed schedule")
-ISL_ARG_BOOL(struct ppcg_debug_options, dump_sizes, 0,
-	"dump-sizes", 0,
-	"dump effectively used per kernel tile, grid and block sizes")
-ISL_ARG_BOOL(struct ppcg_debug_options, verbose, 'v', "verbose", 0, NULL)
-ISL_ARGS_END
-
-ISL_ARGS_START(struct ppcg_options, ppcg_opencl_options_args)
-ISL_ARG_STR(struct ppcg_options, opencl_compiler_options, 0, "compiler-options",
-	"options", NULL, "options to pass to the OpenCL compiler")
-ISL_ARG_BOOL(struct ppcg_options, opencl_use_gpu, 0, "use-gpu", 1,
-	"use GPU device (if available)")
-ISL_ARG_STR_LIST(struct ppcg_options, opencl_n_include_file,
-	opencl_include_files, 0, "include-file", "filename",
-	"file to #include in generated OpenCL code")
-ISL_ARG_BOOL(struct ppcg_options, opencl_print_kernel_types, 0,
-	"print-kernel-types", 1,
-	"print definitions of types in the kernel file")
-ISL_ARG_BOOL(struct ppcg_options, opencl_embed_kernel_code, 0,
-	"embed-kernel-code", 0, "embed kernel code into host code")
-ISL_ARGS_END
-
-ISL_ARGS_START(struct ppcg_options, ppcg_options_args)
-ISL_ARG_CHILD(struct ppcg_options, isl, "isl", &isl_options_args, "isl options")
-ISL_ARG_CHILD(struct ppcg_options, debug, NULL, &ppcg_debug_options_args,
-	"debugging options")
-ISL_ARG_BOOL(struct ppcg_options, group_chains, 0, "group-chains", 1,
-	"group chains of interdependent statements that are executed "
-	"consecutively in the original schedule before scheduling")
-ISL_ARG_BOOL(struct ppcg_options, reschedule, 0, "reschedule", 1,
-	"replace original schedule by isl computed schedule")
-ISL_ARG_BOOL(struct ppcg_options, scale_tile_loops, 0,
-	"scale-tile-loops", 1, NULL)
-ISL_ARG_BOOL(struct ppcg_options, wrap, 0, "wrap", 1, NULL)
-ISL_ARG_BOOL(struct ppcg_options, use_shared_memory, 0, "shared-memory", 1,
-	"use shared memory in kernel code")
-ISL_ARG_BOOL(struct ppcg_options, use_private_memory, 0, "private-memory", 1,
-	"use private memory in kernel code")
-ISL_ARG_STR(struct ppcg_options, ctx, 0, "ctx", "context", NULL,
-    "Constraints on parameters")
-ISL_ARG_BOOL(struct ppcg_options, non_negative_parameters, 0,
-	"assume-non-negative-parameters", 0,
-	"assume all parameters are non-negative)")
-ISL_ARG_BOOL(struct ppcg_options, tile, 0, "tile", 0,
-	"perform tiling (C target)")
-ISL_ARG_INT(struct ppcg_options, tile_size, 'S', "tile-size", "size", 32, NULL)
-ISL_ARG_BOOL(struct ppcg_options, isolate_full_tiles, 0, "isolate-full-tiles",
-	0, "isolate full tiles from partial tiles (hybrid tiling)")
-ISL_ARG_STR(struct ppcg_options, sizes, 0, "sizes", "sizes", NULL,
-	"Per kernel tile, grid and block sizes")
-ISL_ARG_INT(struct ppcg_options, max_shared_memory, 0,
-	"max-shared-memory", "size", 8192, "maximal amount of shared memory")
-ISL_ARG_BOOL(struct ppcg_options, openmp, 0, "openmp", 0,
-	"Generate OpenMP macros (only for C target)")
-ISL_ARG_USER_OPT_CHOICE(struct ppcg_options, target, 0, "target", target,
-	&set_target, PPCG_TARGET_CUDA, PPCG_TARGET_CUDA,
-	"the target to generate code for")
-ISL_ARG_BOOL(struct ppcg_options, linearize_device_arrays, 0,
-	"linearize-device-arrays", 1,
-	"linearize all device arrays, even those of fixed size")
-ISL_ARG_BOOL(struct ppcg_options, allow_gnu_extensions, 0,
-	"allow-gnu-extensions", 1,
-	"allow the use of GNU extensions in generated code")
-ISL_ARG_BOOL(struct ppcg_options, live_range_reordering, 0,
-	"live-range-reordering", 1,
-	"allow successive live ranges on the same memory element "
-	"to be reordered")
-ISL_ARG_BOOL(struct ppcg_options, hybrid, 0, "hybrid", 0,
-	"apply hybrid tiling whenever a suitable input pattern is found "
-	"(GPU targets)")
-ISL_ARG_BOOL(struct ppcg_options, unroll_copy_shared, 0, "unroll-copy-shared",
-	0, "unroll code for copying to/from shared memory")
-ISL_ARG_BOOL(struct ppcg_options, unroll_gpu_tile, 0, "unroll-gpu-tile", 0,
-	"unroll code inside tile on GPU targets")
-ISL_ARG_GROUP("opencl", &ppcg_opencl_options_args, "OpenCL options")
-ISL_ARG_STR(struct ppcg_options, save_schedule_file, 0, "save-schedule",
-	"file", NULL, "save isl computed schedule to <file>")
-ISL_ARG_STR(struct ppcg_options, load_schedule_file, 0, "load-schedule",
-	"file", NULL, "load schedule from <file>, "
-	"using it instead of an isl computed schedule")
-ISL_ARGS_END
diff --git a/polly/lib/External/ppcg/ppcg_options.h b/polly/lib/External/ppcg/ppcg_options.h
deleted file mode 100644
index 56aa7cb781a52..0000000000000
--- a/polly/lib/External/ppcg/ppcg_options.h
+++ /dev/null
@@ -1,100 +0,0 @@
-#ifndef PPCG_OPTIONS_H
-#define PPCG_OPTIONS_H
-
-#include <isl/arg.h>
-#include <isl/options.h>
-
-struct ppcg_debug_options {
-	int dump_schedule_constraints;
-	int dump_schedule;
-	int dump_final_schedule;
-	int dump_sizes;
-	int verbose;
-};
-
-struct ppcg_options {
-	struct isl_options *isl;
-	struct ppcg_debug_options *debug;
-
-	/* Group chains of consecutive statements before scheduling. */
-	int group_chains;
-
-	/* Use isl to compute a schedule replacing the original schedule. */
-	int reschedule;
-	int scale_tile_loops;
-	int wrap;
-
-	/* Assume all parameters are non-negative. */
-	int non_negative_parameters;
-	char *ctx;
-	char *sizes;
-
-	/* Perform tiling (C target). */
-	int tile;
-	int tile_size;
-
-	/* Isolate full tiles from partial tiles. */
-	int isolate_full_tiles;
-
-	/* Take advantage of private memory. */
-	int use_private_memory;
-
-	/* Take advantage of shared memory. */
-	int use_shared_memory;
-
-	/* Maximal amount of shared memory. */
-	int max_shared_memory;
-
-	/* The target we generate code for. */
-	int target;
-
-	/* Generate OpenMP macros (C target only). */
-	int openmp;
-
-	/* Linearize all device arrays. */
-	int linearize_device_arrays;
-
-	/* Allow the use of GNU extensions in generated code. */
-	int allow_gnu_extensions;
-
-	/* Allow live range to be reordered. */
-	int live_range_reordering;
-
-	/* Allow hybrid tiling whenever a suitable input pattern is found. */
-	int hybrid;
-
-	/* Unroll the code for copying to/from shared memory. */
-	int unroll_copy_shared;
-	/* Unroll code inside tile on GPU targets. */
-	int unroll_gpu_tile;
-
-	/* Options to pass to the OpenCL compiler.  */
-	char *opencl_compiler_options;
-	/* Prefer GPU device over CPU. */
-	int opencl_use_gpu;
-	/* Number of files to include. */
-	int opencl_n_include_file;
-	/* Files to include. */
-	const char **opencl_include_files;
-	/* Print definitions of types in kernels. */
-	int opencl_print_kernel_types;
-	/* Embed OpenCL kernel code in host code. */
-	int opencl_embed_kernel_code;
-
-	/* Name of file for saving isl computed schedule or NULL. */
-	char *save_schedule_file;
-	/* Name of file for loading schedule or NULL. */
-	char *load_schedule_file;
-};
-
-ISL_ARG_DECL(ppcg_debug_options, struct ppcg_debug_options,
-	ppcg_debug_options_args)
-ISL_ARG_DECL(ppcg_options, struct ppcg_options, ppcg_options_args)
-
-#define		PPCG_TARGET_C		0
-#define		PPCG_TARGET_CUDA	1
-#define		PPCG_TARGET_OPENCL      2
-
-void ppcg_options_set_target_defaults(struct ppcg_options *options);
-
-#endif
diff --git a/polly/lib/External/ppcg/print.c b/polly/lib/External/ppcg/print.c
deleted file mode 100644
index dd839e48e51ba..0000000000000
--- a/polly/lib/External/ppcg/print.c
+++ /dev/null
@@ -1,461 +0,0 @@
-/*
- * Copyright 2012-2013 Ecole Normale Superieure
- *
- * Use of this software is governed by the MIT license
- *
- * Written by Sven Verdoolaege,
- * Ecole Normale Superieure, 45 rue d’Ulm, 75230 Paris, France
- */
-
-#include <isl/aff.h>
-#include <isl/ast_build.h>
-#include <isl/id.h>
-
-#include "print.h"
-#include "util.h"
-
-__isl_give isl_printer *ppcg_start_block(__isl_take isl_printer *p)
-{
-	p = isl_printer_start_line(p);
-	p = isl_printer_print_str(p, "{");
-	p = isl_printer_end_line(p);
-	p = isl_printer_indent(p, 2);
-	return p;
-}
-
-__isl_give isl_printer *ppcg_end_block(__isl_take isl_printer *p)
-{
-	p = isl_printer_indent(p, -2);
-	p = isl_printer_start_line(p);
-	p = isl_printer_print_str(p, "}");
-	p = isl_printer_end_line(p);
-	return p;
-}
-
-/* Names of notes that keep track of whether min/max
- * macro definitions have already been printed.
- */
-static const char *ppcg_max_printed = "ppcg_max_printed";
-static const char *ppcg_min_printed = "ppcg_min_printed";
-
-/* Has the macro definition corresponding to "note_name" been printed
- * to "p" before?
- * That is, does "p" have an associated "note_name" note?
- */
-static isl_bool printed_before(__isl_keep isl_printer *p, const char *note_name)
-{
-	isl_ctx *ctx;
-	isl_id *id;
-	isl_bool printed;
-
-	if (!p)
-		return isl_bool_error;
-
-	ctx = isl_printer_get_ctx(p);
-	id = isl_id_alloc(ctx, note_name, NULL);
-	printed = isl_printer_has_note(p, id);
-	isl_id_free(id);
-
-	return printed;
-}
-
-/* Keep track of the fact that the macro definition corresponding
- * to "note_name" has been printed to "p" by attaching a note with
- * that name.  The value of the note is of no importance, but it
- * has to be a valid isl_id, so the note identifier is reused
- * as the note.
- */
-static __isl_give isl_printer *mark_printed(__isl_take isl_printer *p,
-	const char *note_name)
-{
-	isl_ctx *ctx;
-	isl_id *id;
-
-	if (!p)
-		return NULL;
-
-	ctx = isl_printer_get_ctx(p);
-	id = isl_id_alloc(ctx, note_name, NULL);
-	return isl_printer_set_note(p, id, isl_id_copy(id));
-}
-
-/* Print a macro definition "def" for the macro "name" to "p",
- * unless such a macro definition has been printed to "p" before.
- * "note_name" is used as the name of the note that keeps track
- * of whether this printing has happened.
- */
-static __isl_give isl_printer *print_ppcg_macro(__isl_take isl_printer *p,
-	const char *name, const char *def, const char *note_name)
-{
-	isl_bool printed;
-
-	printed = printed_before(p, note_name);
-	if (printed < 0)
-		return isl_printer_free(p);
-	if (printed)
-		return p;
-
-	p = isl_printer_start_line(p);
-	p = isl_printer_print_str(p, "#define ");
-	p = isl_printer_print_str(p, name);
-	p = isl_printer_print_str(p, def);
-	p = isl_printer_end_line(p);
-
-	p = mark_printed(p, note_name);
-
-	return p;
-}
-
-/* Structure for keeping track of definitions of some macros.
- */
-struct ppcg_macros {
-	const char *min;
-	const char *max;
-};
-
-/* Free the memory allocated by a struct ppcg_macros.
- */
-static void ppcg_macros_free(void *user)
-{
-	free(user);
-}
-
-/* Default macro definitions (when GNU extensions are allowed).
- */
-struct ppcg_macros ppcg_macros_default = {
-	.min = "(x,y)    "
-		"({ __typeof__(x) _x = (x); __typeof__(y) _y = (y); "
-		"_x < _y ? _x : _y; })",
-	.max = "(x,y)    "
-		"({ __typeof__(x) _x = (x); __typeof__(y) _y = (y); "
-		"_x > _y ? _x : _y; })",
-};
-
-/* Name used for the note that keeps track of macro definitions.
- */
-static const char *ppcg_macros = "ppcg_macros";
-
-/* Set the macro definitions for isl_ast_op_min and isl_ast_op_max
- * to "min" and "max" and store them in "p".
- *
- * In particular, create a ppcg_macros object and attach it
- * as a note to the printer.
- */
-__isl_give isl_printer *ppcg_set_macros(__isl_take isl_printer *p,
-	const char *min, const char *max)
-{
-	isl_ctx *ctx;
-	isl_id *id, *macros_id;
-	struct ppcg_macros *macros;
-
-	if (!p)
-		return NULL;
-
-	ctx = isl_printer_get_ctx(p);
-	macros = isl_alloc_type(ctx, struct ppcg_macros);
-	if (!macros)
-		return isl_printer_free(p);
-	macros->min = min;
-	macros->max = max;
-	id = isl_id_alloc(ctx, ppcg_macros, NULL);
-	macros_id = isl_id_alloc(ctx, NULL, macros);
-	if (!macros_id)
-		ppcg_macros_free(macros);
-	else
-		macros_id = isl_id_set_free_user(macros_id, &ppcg_macros_free);
-
-	p = isl_printer_set_note(p, id, macros_id);
-
-	return p;
-}
-
-/* Return the ppcg_macros object that holds the currently active
- * macro definitions in "p".
- * If "p" has a note with macro definitions, then return those.
- * Otherwise, return the default macro definitions.
- */
-static struct ppcg_macros *get_macros(__isl_keep isl_printer *p)
-{
-	isl_id *id;
-	isl_bool has_macros;
-	struct ppcg_macros *macros;
-
-	id = isl_id_alloc(isl_printer_get_ctx(p), ppcg_macros, NULL);
-	has_macros = isl_printer_has_note(p, id);
-	if (has_macros < 0 || !has_macros) {
-		isl_id_free(id);
-		if (has_macros < 0)
-			return NULL;
-		return &ppcg_macros_default;
-	}
-	id = isl_printer_get_note(p, id);
-	macros = isl_id_get_user(id);
-	isl_id_free(id);
-
-	return macros;
-}
-
-/* Print the currently active macro definition for ppcg_max.
- */
-static __isl_give isl_printer *print_max(__isl_take isl_printer *p)
-{
-	struct ppcg_macros *macros;
-
-	macros = get_macros(p);
-	if (!macros)
-		return isl_printer_free(p);
-	return print_ppcg_macro(p, ppcg_max, macros->max, ppcg_max_printed);
-}
-
-/* Print the currently active macro definition for ppcg_min.
- */
-static __isl_give isl_printer *print_min(__isl_take isl_printer *p)
-{
-	struct ppcg_macros *macros;
-
-	macros = get_macros(p);
-	if (!macros)
-		return isl_printer_free(p);
-	return print_ppcg_macro(p, ppcg_min, macros->min, ppcg_min_printed);
-}
-
-/* Print a macro definition for "type" to "p".
- * If GNU extensions are allowed, then print a specialized definition
- * for isl_ast_op_min and isl_ast_op_max.
- * Otherwise, use the default isl definition.
- */
-__isl_give isl_printer *ppcg_print_macro(enum isl_ast_op_type type,
-	__isl_take isl_printer *p)
-{
-	isl_ctx *ctx;
-	struct ppcg_options *options;
-
-	if (!p)
-		return NULL;
-
-	ctx = isl_printer_get_ctx(p);
-	options = isl_ctx_peek_options(ctx, &ppcg_options_args);
-	if (!options || !options->allow_gnu_extensions)
-		return isl_ast_op_type_print_macro(type, p);
-
-	switch (type) {
-	case isl_ast_op_max:
-		return print_max(p);
-	case isl_ast_op_min:
-		return print_min(p);
-	default:
-		return isl_ast_op_type_print_macro(type, p);
-	}
-}
-
-/* isl_ast_expr_foreach_ast_op_type or isl_ast_node_foreach_ast_op_type
- * callback that prints a macro definition for "type".
- */
-static isl_stat print_macro(enum isl_ast_op_type type, void *user)
-{
-	isl_printer **p = user;
-
-	*p = ppcg_print_macro(type, *p);
-	if (!*p)
-		return isl_stat_error;
-
-	return isl_stat_ok;
-}
-
-/* Print the required macros for "expr".
- */
-__isl_give isl_printer *ppcg_ast_expr_print_macros(
-	__isl_keep isl_ast_expr *expr, __isl_take isl_printer *p)
-{
-	if (isl_ast_expr_foreach_ast_op_type(expr, &print_macro, &p) < 0)
-		return isl_printer_free(p);
-	return p;
-}
-
-/* isl_id_to_ast_expr_foreach callback that prints the required
- * macro definitions for "val".
- */
-static isl_stat print_expr_macros(__isl_take isl_id *key,
-	__isl_take isl_ast_expr *val, void *user)
-{
-	isl_printer **p = user;
-
-	*p = ppcg_ast_expr_print_macros(val, *p);
-	isl_id_free(key);
-	isl_ast_expr_free(val);
-
-	if (!*p)
-		return isl_stat_error;
-	return isl_stat_ok;
-}
-
-/* Print the required macro definitions for the body of a statement in which
- * the access expressions are replaced by the isl_ast_expr objects
- * in "ref2expr".
- */
-__isl_give isl_printer *ppcg_print_body_macros(__isl_take isl_printer *p,
-	__isl_keep isl_id_to_ast_expr *ref2expr)
-{
-	if (isl_id_to_ast_expr_foreach(ref2expr, &print_expr_macros, &p) < 0)
-		return isl_printer_free(p);
-	return p;
-}
-
-/* Print the required macros for "node".
- */
-__isl_give isl_printer *ppcg_print_macros(__isl_take isl_printer *p,
-	__isl_keep isl_ast_node *node)
-{
-	if (isl_ast_node_foreach_ast_op_type(node, &print_macro, &p) < 0)
-		return isl_printer_free(p);
-	return p;
-}
-
-/* Names used for the macros that may appear in a printed isl AST.
- */
-const char *ppcg_min = "ppcg_min";
-const char *ppcg_max = "ppcg_max";
-const char *ppcg_fdiv_q = "ppcg_fdiv_q";
-
-/* Set the names of the macros that may appear in a printed isl AST.
- */
-__isl_give isl_printer *ppcg_set_macro_names(__isl_take isl_printer *p)
-{
-	p = isl_ast_op_type_set_print_name(p, isl_ast_op_min, ppcg_min);
-	p = isl_ast_op_type_set_print_name(p, isl_ast_op_max, ppcg_max);
-	p = isl_ast_op_type_set_print_name(p, isl_ast_op_fdiv_q, ppcg_fdiv_q);
-
-	return p;
-}
-
-/* Given a multi affine expression "mpa" without domain, modify it to have
- * the schedule space of "build" as domain.
- *
- * If the schedule space of "build" is a parameter space, then nothing
- * needs to be done.
- * Otherwise, "mpa" is first given a 0D domain and then it is combined
- * with a mapping from the schedule space of "build" to the same 0D domain.
- */
-__isl_give isl_multi_pw_aff *ppcg_attach_multi_pw_aff(
-	__isl_take isl_multi_pw_aff *mpa, __isl_keep isl_ast_build *build)
-{
-	isl_bool params;
-	isl_space *space;
-	isl_multi_aff *ma;
-
-	space = isl_ast_build_get_schedule_space(build);
-	params = isl_space_is_params(space);
-	if (params < 0 || params) {
-		isl_space_free(space);
-		if (params < 0)
-			return isl_multi_pw_aff_free(mpa);
-		return mpa;
-	}
-	space = isl_space_from_domain(space);
-	ma = isl_multi_aff_zero(space);
-	mpa = isl_multi_pw_aff_from_range(mpa);
-	mpa = isl_multi_pw_aff_pullback_multi_aff(mpa, ma);
-
-	return mpa;
-}
-
-/* Build an access AST expression from "size" using "build".
- * "size" does not have a domain, but "build" may have a proper schedule space.
- * First modify "size" to have that schedule space as domain.
- */
-__isl_give isl_ast_expr *ppcg_build_size_expr(__isl_take isl_multi_pw_aff *size,
-	__isl_keep isl_ast_build *build)
-{
-	size = ppcg_attach_multi_pw_aff(size, build);
-	return isl_ast_build_access_from_multi_pw_aff(build, size);
-}
-
-/* Print a declaration for an array with element type "base_type" and
- * size "size" to "p".
- */
-__isl_give isl_printer *ppcg_print_declaration_with_size(
-	__isl_take isl_printer *p, const char *base_type,
-	__isl_keep isl_ast_expr *size)
-{
-	if (!base_type || !size)
-		return isl_printer_free(p);
-
-	p = ppcg_ast_expr_print_macros(size, p);
-	p = isl_printer_start_line(p);
-	p = isl_printer_print_str(p, base_type);
-	p = isl_printer_print_str(p, " ");
-	p = isl_printer_print_ast_expr(p, size);
-	p = isl_printer_print_str(p, ";");
-	p = isl_printer_end_line(p);
-
-	return p;
-}
-
-/* Print a declaration for array "array" to "p", using "build"
- * to simplify any size expressions.
- *
- * The size is computed from the extent of the array and is
- * subsequently converted to an "access expression" by "build".
- */
-__isl_give isl_printer *ppcg_print_declaration(__isl_take isl_printer *p,
-	struct pet_array *array, __isl_keep isl_ast_build *build)
-{
-	isl_multi_pw_aff *size;
-	isl_ast_expr *expr;
-
-	if (!array)
-		return isl_printer_free(p);
-
-	size = ppcg_size_from_extent(isl_set_copy(array->extent));
-	expr = isl_ast_build_access_from_multi_pw_aff(build, size);
-	p = ppcg_print_declaration_with_size(p, array->element_type, expr);
-	isl_ast_expr_free(expr);
-
-	return p;
-}
-
-/* Print declarations for the arrays in "scop" that are declared
- * and that are exposed (if exposed == 1) or not exposed (if exposed == 0).
- */
-static __isl_give isl_printer *print_declarations(__isl_take isl_printer *p,
-	struct ppcg_scop *scop, int exposed)
-{
-	int i;
-	isl_ast_build *build;
-
-	if (!scop)
-		return isl_printer_free(p);
-
-	build = isl_ast_build_from_context(isl_set_copy(scop->context));
-	for (i = 0; i < scop->pet->n_array; ++i) {
-		struct pet_array *array = scop->pet->arrays[i];
-
-		if (!array->declared)
-			continue;
-		if (array->exposed != exposed)
-			continue;
-
-		p = ppcg_print_declaration(p, array, build);
-	}
-	isl_ast_build_free(build);
-
-	return p;
-}
-
-/* Print declarations for the arrays in "scop" that are declared
- * and exposed to the code after the scop.
- */
-__isl_give isl_printer *ppcg_print_exposed_declarations(
-	__isl_take isl_printer *p, struct ppcg_scop *scop)
-{
-	return print_declarations(p, scop, 1);
-}
-
-/* Print declarations for the arrays in "scop" that are declared,
- * but not exposed to the code after the scop.
- */
-__isl_give isl_printer *ppcg_print_hidden_declarations(
-	__isl_take isl_printer *p, struct ppcg_scop *scop)
-{
-	return print_declarations(p, scop, 0);
-}
diff --git a/polly/lib/External/ppcg/print.h b/polly/lib/External/ppcg/print.h
deleted file mode 100644
index 3b33b0ce32ac9..0000000000000
--- a/polly/lib/External/ppcg/print.h
+++ /dev/null
@@ -1,40 +0,0 @@
-#ifndef PRINT_H
-#define PRINT_H
-
-#include <isl/ast.h>
-
-#include "ppcg.h"
-
-extern const char *ppcg_min;
-extern const char *ppcg_max;
-extern const char *ppcg_fdiv_q;
-
-__isl_give isl_printer *ppcg_start_block(__isl_take isl_printer *p);
-__isl_give isl_printer *ppcg_end_block(__isl_take isl_printer *p);
-
-__isl_give isl_printer *ppcg_set_macro_names(__isl_take isl_printer *p);
-__isl_give isl_printer *ppcg_set_macros(__isl_take isl_printer *p,
-	const char *min, const char *max);
-__isl_give isl_printer *ppcg_print_macro(enum isl_ast_op_type type,
-	__isl_take isl_printer *p);
-__isl_give isl_printer *ppcg_ast_expr_print_macros(
-	__isl_keep isl_ast_expr *expr, __isl_take isl_printer *p);
-__isl_give isl_printer *ppcg_print_body_macros(__isl_take isl_printer *p,
-	__isl_keep isl_id_to_ast_expr *ref2expr);
-__isl_give isl_printer *ppcg_print_macros(__isl_take isl_printer *p,
-	__isl_keep isl_ast_node *node);
-
-__isl_give isl_ast_expr *ppcg_build_size_expr(__isl_take isl_multi_pw_aff *size,
-	__isl_keep isl_ast_build *build);
-
-__isl_give isl_printer *ppcg_print_declaration_with_size(
-	__isl_take isl_printer *p, const char *base_type,
-	__isl_keep isl_ast_expr *size);
-__isl_give isl_printer *ppcg_print_declaration(__isl_take isl_printer *p,
-	struct pet_array *array, __isl_keep isl_ast_build *build);
-__isl_give isl_printer *ppcg_print_exposed_declarations(
-	__isl_take isl_printer *p, struct ppcg_scop *scop);
-__isl_give isl_printer *ppcg_print_hidden_declarations(
-	__isl_take isl_printer *p, struct ppcg_scop *scop);
-
-#endif
diff --git a/polly/lib/External/ppcg/schedule.c b/polly/lib/External/ppcg/schedule.c
deleted file mode 100644
index 7e65c31391bb9..0000000000000
--- a/polly/lib/External/ppcg/schedule.c
+++ /dev/null
@@ -1,165 +0,0 @@
-/*
- * Copyright 2010-2011 INRIA Saclay
- *
- * Use of this software is governed by the MIT license
- *
- * Written by Sven Verdoolaege, INRIA Saclay - Ile-de-France,
- * Parc Club Orsay Universite, ZAC des vignes, 4 rue Jacques Monod,
- * 91893 Orsay, France
- */
-
-#include <assert.h>
-#include <ctype.h>
-#include <stdio.h>
-#include <string.h>
-
-#include <isl/set.h>
-#include <isl/map.h>
-#include <isl/constraint.h>
-
-#include "schedule.h"
-
-/* Add parameters with identifiers "ids" to "set".
- */
-static __isl_give isl_set *add_params(__isl_take isl_set *set,
-	__isl_keep isl_id_list *ids)
-{
-	int i, n;
-	unsigned nparam;
-
-	n = isl_id_list_n_id(ids);
-
-	nparam = isl_set_dim(set, isl_dim_param);
-	set = isl_set_add_dims(set, isl_dim_param, n);
-
-	for (i = 0; i < n; ++i) {
-		isl_id *id;
-
-		id = isl_id_list_get_id(ids, i);
-		set = isl_set_set_dim_id(set, isl_dim_param, nparam + i, id);
-	}
-
-	return set;
-}
-
-/* Equate the dimensions of "set" starting at "first" to
- * freshly created parameters with identifiers "ids".
- * The number of equated dimensions is equal to the number of elements in "ids".
- */
-static __isl_give isl_set *parametrize(__isl_take isl_set *set,
-	int first, __isl_keep isl_id_list *ids)
-{
-	int i, n;
-	unsigned nparam;
-
-	nparam = isl_set_dim(set, isl_dim_param);
-
-	set = add_params(set, ids);
-
-	n = isl_id_list_n_id(ids);
-	for (i = 0; i < n; ++i)
-		set = isl_set_equate(set, isl_dim_param, nparam + i,
-					isl_dim_set, first + i);
-
-	return set;
-}
-
-/* Given a parameter space "space", create a set of dimension "len"
- * of which the dimensions starting at "first" are equated to
- * freshly created parameters with identifiers "ids".
- */
-__isl_give isl_set *parametrization(__isl_take isl_space *space,
-	int len, int first, __isl_keep isl_id_list *ids)
-{
-	isl_set *set;
-
-	space = isl_space_set_from_params(space);
-	space = isl_space_add_dims(space, isl_dim_set, len);
-	set = isl_set_universe(space);
-
-	return parametrize(set, first, ids);
-}
-
-/* Load and return a schedule from a file called "filename".
- */
-static __isl_give isl_schedule *load_schedule(isl_ctx *ctx,
-	const char *filename)
-{
-	FILE *file;
-	isl_schedule *schedule;
-
-	file = fopen(filename, "r");
-	if (!file) {
-		fprintf(stderr, "Unable to open '%s' for reading\n", filename);
-		return NULL;
-	}
-	schedule = isl_schedule_read_from_file(ctx, file);
-	fclose(file);
-
-	return schedule;
-}
-
-/* Save the schedule "schedule" to a file called "filename".
- * The schedule is printed in block style.
- */
-static void save_schedule(__isl_keep isl_schedule *schedule,
-	const char *filename)
-{
-	FILE *file;
-	isl_ctx *ctx;
-	isl_printer *p;
-
-	if (!schedule)
-		return;
-
-	file = fopen(filename, "w");
-	if (!file) {
-		fprintf(stderr, "Unable to open '%s' for writing\n", filename);
-		return;
-	}
-	ctx = isl_schedule_get_ctx(schedule);
-	p = isl_printer_to_file(ctx, file);
-	p = isl_printer_set_yaml_style(p, ISL_YAML_STYLE_BLOCK);
-	p = isl_printer_print_schedule(p, schedule);
-	isl_printer_free(p);
-	fclose(file);
-}
-
-/* Obtain a schedule, either by reading it form a file
- * or by computing it using "compute".
- * Also take care of saving the computed schedule and/or
- * dumping the obtained schedule if requested by the user.
- */
-__isl_give isl_schedule *ppcg_get_schedule(isl_ctx *ctx,
-	struct ppcg_options *options,
-	__isl_give isl_schedule *(*compute)(void *user), void *user)
-{
-	isl_schedule *schedule;
-
-	if (options->load_schedule_file) {
-		schedule = load_schedule(ctx, options->load_schedule_file);
-	} else {
-		schedule = compute(user);
-		if (options->save_schedule_file)
-			save_schedule(schedule, options->save_schedule_file);
-	}
-	if (options->debug->dump_schedule)
-		isl_schedule_dump(schedule);
-
-	return schedule;
-}
-
-/* Mark all dimensions in the band node "node" to be of "type".
- */
-__isl_give isl_schedule_node *ppcg_set_schedule_node_type(
-	__isl_take isl_schedule_node *node, enum isl_ast_loop_type type)
-{
-	int i, n;
-
-	n = isl_schedule_node_band_n_member(node);
-	for (i = 0; i < n; ++i)
-		node = isl_schedule_node_band_member_set_ast_loop_type(node, i,
-							type);
-
-	return node;
-}
diff --git a/polly/lib/External/ppcg/schedule.h b/polly/lib/External/ppcg/schedule.h
deleted file mode 100644
index 771466558b395..0000000000000
--- a/polly/lib/External/ppcg/schedule.h
+++ /dev/null
@@ -1,21 +0,0 @@
-#ifndef _SCHEDULE_H
-#define _SCHEDULE_H
-
-#include <isl/id.h>
-#include <isl/space.h>
-#include <isl/schedule.h>
-#include <isl/schedule_node.h>
-
-#include "ppcg_options.h"
-
-__isl_give isl_set *parametrization(__isl_take isl_space *space,
-	int len, int first, __isl_keep isl_id_list *names);
-
-__isl_give isl_schedule *ppcg_get_schedule(isl_ctx *ctx,
-	struct ppcg_options *options,
-	__isl_give isl_schedule *(*compute)(void *user), void *user);
-
-__isl_give isl_schedule_node *ppcg_set_schedule_node_type(
-	__isl_take isl_schedule_node *node, enum isl_ast_loop_type type);
-
-#endif
diff --git a/polly/lib/External/ppcg/tests/allow-sparse-copy-in.c b/polly/lib/External/ppcg/tests/allow-sparse-copy-in.c
deleted file mode 100644
index 620d8254a060a..0000000000000
--- a/polly/lib/External/ppcg/tests/allow-sparse-copy-in.c
+++ /dev/null
@@ -1,49 +0,0 @@
-#include <stdlib.h>
-
-int main()
-{
-	int A[2][1000][1000];
-	int B[2][1000][1000];
-
-#pragma scop
-	{
-		for (int i = 0; i < 256; ++i)
-			for (int j = 0; j < 256; ++j)
-				if (j % 8 <= 2 || j % 8 >= 6)
-					A[1][i][j] = B[1][j][i];
-	}
-#pragma endscop
-
-/* 
-
-When compiled with:
-
-./ppcg tests/allow-sparse-copy-in.c --no-linearize-device-arrays
-	--on-error=abort --sizes='{kernel[i]->tile[8,8]; kernel[i]->block[1,8]}'
-	--max-shared-memory=-1  --unroll-copy-shared
-
-this originally resulted in the following copy-in code:
-
-      shared_B[0][0][t1] = B[1][8 * b1][8 * b0 + t1];
-      shared_B[0][1][t1] = B[1][8 * b1 + 1][8 * b0 + t1];
-      shared_B[0][2][t1] = B[1][8 * b1 + 2][8 * b0 + t1];
-      shared_B[0][3][t1] = B[1][8 * b1 + 3][8 * b0 + t1];
-      shared_B[0][4][t1] = B[1][8 * b1 + 4][8 * b0 + t1];
-      shared_B[0][5][t1] = B[1][8 * b1 + 5][8 * b0 + t1];
-      shared_B[0][6][t1] = B[1][8 * b1 + 6][8 * b0 + t1];
-      shared_B[0][7][t1] = B[1][8 * b1 + 7][8 * b0 + t1];
-
-whereas we only want to only perform copies that are actually needed:
-
-      shared_B[0][0][t1] = B[1][8 * b1][8 * b0 + t1];
-      shared_B[0][1][t1] = B[1][8 * b1 + 1][8 * b0 + t1];
-      shared_B[0][2][t1] = B[1][8 * b1 + 2][8 * b0 + t1];
-      shared_B[0][6][t1] = B[1][8 * b1 + 6][8 * b0 + t1];
-      shared_B[0][7][t1] = B[1][8 * b1 + 7][8 * b0 + t1];
-*/
-	for (int i = 0; i < 100; ++i)
-		if (A[1][0][i] != i)
-			return EXIT_FAILURE;
-
-	return EXIT_SUCCESS;
-}
diff --git a/polly/lib/External/ppcg/tests/call.c b/polly/lib/External/ppcg/tests/call.c
deleted file mode 100644
index 7f4f7c4a24f7f..0000000000000
--- a/polly/lib/External/ppcg/tests/call.c
+++ /dev/null
@@ -1,29 +0,0 @@
-#include <stdlib.h>
-
-void copy_summary(int b[1000], int a[1000], int pos)
-{
-	b[pos] = 0;
-	int c = a[pos];
-}
-
-#ifdef pencil_access
-__attribute__((pencil_access(copy_summary)))
-#endif
-void copy(int b[1000], int a[1000], int pos);
-
-int main()
-{
-	int a[1000], b[1000];
-
-	for (int i = 0; i < 1000; ++i)
-		a[i] = i;
-#pragma scop
-	for (int i = 0; i < 1000; ++i)
-		copy(b, a, i);
-#pragma endscop
-	for (int i = 0; i < 1000; ++i)
-		if (b[i] != a[i])
-			return EXIT_FAILURE;
-
-	return EXIT_SUCCESS;
-}
diff --git a/polly/lib/External/ppcg/tests/call2.c b/polly/lib/External/ppcg/tests/call2.c
deleted file mode 100644
index 1cfc85cdf0d2e..0000000000000
--- a/polly/lib/External/ppcg/tests/call2.c
+++ /dev/null
@@ -1,29 +0,0 @@
-#include <stdlib.h>
-
-void copy_summary(int b[1000], int a[1000], int pos)
-{
-	b[pos] = 0;
-	int c = a[pos];
-}
-
-#ifdef pencil_access
-__attribute__((pencil_access(copy_summary)))
-#endif
-void copy(int b[1000], int a[1000], int pos);
-
-int main()
-{
-	int a[2][1000];
-
-	for (int i = 0; i < 1000; ++i)
-		a[0][i] = i;
-#pragma scop
-	for (int i = 0; i < 1000; ++i)
-		copy(a[1], a[0], i);
-#pragma endscop
-	for (int i = 0; i < 1000; ++i)
-		if (a[1][i] != a[0][i])
-			return EXIT_FAILURE;
-
-	return EXIT_SUCCESS;
-}
diff --git a/polly/lib/External/ppcg/tests/call2_opencl_functions.cl b/polly/lib/External/ppcg/tests/call2_opencl_functions.cl
deleted file mode 100644
index 6f72aea34e87f..0000000000000
--- a/polly/lib/External/ppcg/tests/call2_opencl_functions.cl
+++ /dev/null
@@ -1,4 +0,0 @@
-void copy(__global int b[1000], __global int a[1000], int pos)
-{
-	b[pos] = a[pos];
-}
diff --git a/polly/lib/External/ppcg/tests/call3.c b/polly/lib/External/ppcg/tests/call3.c
deleted file mode 100644
index 51b1e290ce4f4..0000000000000
--- a/polly/lib/External/ppcg/tests/call3.c
+++ /dev/null
@@ -1,32 +0,0 @@
-#include <stdlib.h>
-
-void copy_summary(int b[100], int a[100])
-{
-	for (int i = 0; i < 100; ++i) {
-		b[i] = 0;
-		int c = a[i];
-	}
-}
-
-#ifdef pencil_access
-__attribute__((pencil_access(copy_summary)))
-#endif
-void copy(int b[100], int a[100]);
-
-int main()
-{
-	int A[100][100], B[100];
-
-	for (int i = 0; i < 100; ++i)
-		B[i] = i;
-#pragma scop
-	for (int i = 0; i < 100; ++i)
-		copy(A[i], B);
-#pragma endscop
-	for (int i = 0; i < 100; ++i)
-		for (int j = 0; j < 100; ++j)
-			if (A[j][i] != B[i])
-				return EXIT_FAILURE;
-
-	return EXIT_SUCCESS;
-}
diff --git a/polly/lib/External/ppcg/tests/call3_opencl_functions.cl b/polly/lib/External/ppcg/tests/call3_opencl_functions.cl
deleted file mode 100644
index 35c87a705352f..0000000000000
--- a/polly/lib/External/ppcg/tests/call3_opencl_functions.cl
+++ /dev/null
@@ -1,5 +0,0 @@
-void copy(__global int b[100], __global int a[100])
-{
-	for (int i = 0; i < 100; ++i)
-		b[i] = a[i];
-}
diff --git a/polly/lib/External/ppcg/tests/call_opencl_functions.cl b/polly/lib/External/ppcg/tests/call_opencl_functions.cl
deleted file mode 100644
index 6f72aea34e87f..0000000000000
--- a/polly/lib/External/ppcg/tests/call_opencl_functions.cl
+++ /dev/null
@@ -1,4 +0,0 @@
-void copy(__global int b[1000], __global int a[1000], int pos)
-{
-	b[pos] = a[pos];
-}
diff --git a/polly/lib/External/ppcg/tests/dead.c b/polly/lib/External/ppcg/tests/dead.c
deleted file mode 100644
index 3f70a338262de..0000000000000
--- a/polly/lib/External/ppcg/tests/dead.c
+++ /dev/null
@@ -1,23 +0,0 @@
-#include <stdlib.h>
-
-int main()
-{
-	int a[1000], b[1000];
-
-	for (int i = 0; i < 1000; ++i)
-		a[i] = i;
-#pragma scop
-	for (int i = 0; i < 1000; ++i) {
-		int c;
-		int d;
-		c = a[i];
-		d = c;
-		b[i] = c;
-	}
-#pragma endscop
-	for (int i = 0; i < 1000; ++i)
-		if (b[i] != a[i])
-			return EXIT_FAILURE;
-
-	return EXIT_SUCCESS;
-}
diff --git a/polly/lib/External/ppcg/tests/iterator.c b/polly/lib/External/ppcg/tests/iterator.c
deleted file mode 100644
index 651f530660dde..0000000000000
--- a/polly/lib/External/ppcg/tests/iterator.c
+++ /dev/null
@@ -1,18 +0,0 @@
-#include <stdlib.h>
-
-int main()
-{
-	int i;
-	int a[101];
-
-	i = 0;
-#pragma scop
-	for (i = 0; i < 100; ++i)
-		a[i] = i;
-	a[i] = i;
-#pragma endscop
-	if (a[100] != 100)
-		return EXIT_FAILURE;
-
-	return EXIT_SUCCESS;
-}
diff --git a/polly/lib/External/ppcg/tests/live_out.c b/polly/lib/External/ppcg/tests/live_out.c
deleted file mode 100644
index c13965ea29c39..0000000000000
--- a/polly/lib/External/ppcg/tests/live_out.c
+++ /dev/null
@@ -1,22 +0,0 @@
-#include <stdlib.h>
-
-/* Check that a write access is not removed from the live-out
- * accesses only because a strict subset of the (potentially)
- * accessed elements are killed by a later write.
- */
-int main()
-{
-	int A[10];
-
-	A[1] = 0;
-#pragma scop
-	int i = 1;
-	i = i * i;
-	A[i] = 1;
-	A[0] = 0;
-#pragma endscop
-	if (A[1] != 1)
-		return EXIT_FAILURE;
-
-	return EXIT_SUCCESS;
-}
diff --git a/polly/lib/External/ppcg/tests/local.c b/polly/lib/External/ppcg/tests/local.c
deleted file mode 100644
index e8b3894f019aa..0000000000000
--- a/polly/lib/External/ppcg/tests/local.c
+++ /dev/null
@@ -1,22 +0,0 @@
-#include <stdlib.h>
-
-int main()
-{
-	int A[100];
-
-#pragma scop
-	{
-		int B[100];
-		B[0] = 0;
-		for (int i = 1; i < 100; ++i)
-			B[i] = B[i - 1] + 1;
-		for (int i = 0; i < 100; ++i)
-			A[i] = B[i];
-	}
-#pragma endscop
-	for (int i = 0; i < 100; ++i)
-		if (A[i] != i)
-			return EXIT_FAILURE;
-
-	return EXIT_SUCCESS;
-}
diff --git a/polly/lib/External/ppcg/tests/loop.c b/polly/lib/External/ppcg/tests/loop.c
deleted file mode 100644
index 910425b403e3d..0000000000000
--- a/polly/lib/External/ppcg/tests/loop.c
+++ /dev/null
@@ -1,18 +0,0 @@
-#include <stdlib.h>
-
-int main()
-{
-	int a[1000], b[1000];
-
-	for (int i = 0; i < 1000; ++i)
-		a[i] = i;
-#pragma scop
-	for (int i = 0; i < 1000; ++i)
-		b[i] = a[i];
-#pragma endscop
-	for (int i = 0; i < 1000; ++i)
-		if (b[i] != a[i])
-			return EXIT_FAILURE;
-
-	return EXIT_SUCCESS;
-}
diff --git a/polly/lib/External/ppcg/tests/not_accessed.c b/polly/lib/External/ppcg/tests/not_accessed.c
deleted file mode 100644
index e5fb89236b83f..0000000000000
--- a/polly/lib/External/ppcg/tests/not_accessed.c
+++ /dev/null
@@ -1,29 +0,0 @@
-#include <stdlib.h>
-
-void copy_summary(int b[1000], int a[1000], int pos, int c[1000])
-{
-	b[pos] = 0;
-	int d = a[pos];
-}
-
-#ifdef pencil_access
-__attribute__((pencil_access(copy_summary)))
-#endif
-void copy(int b[1000], int a[1000], int pos, int c[1000]);
-
-int main()
-{
-	int a[1000], b[1000], c[1000];
-
-	for (int i = 0; i < 1000; ++i)
-		a[i] = i;
-#pragma scop
-	for (int i = 0; i < 1000; ++i)
-		copy(b, a, i, c);
-#pragma endscop
-	for (int i = 0; i < 1000; ++i)
-		if (b[i] != a[i])
-			return EXIT_FAILURE;
-
-	return EXIT_SUCCESS;
-}
diff --git a/polly/lib/External/ppcg/tests/not_accessed_opencl_functions.cl b/polly/lib/External/ppcg/tests/not_accessed_opencl_functions.cl
deleted file mode 100644
index e1588e051501a..0000000000000
--- a/polly/lib/External/ppcg/tests/not_accessed_opencl_functions.cl
+++ /dev/null
@@ -1,5 +0,0 @@
-void copy(__global int b[1000], __global int a[1000], int pos,
-	__global int c[1000])
-{
-	b[pos] = a[pos];
-}
diff --git a/polly/lib/External/ppcg/tests/scalar.c b/polly/lib/External/ppcg/tests/scalar.c
deleted file mode 100644
index ffd779798c042..0000000000000
--- a/polly/lib/External/ppcg/tests/scalar.c
+++ /dev/null
@@ -1,13 +0,0 @@
-#include <stdlib.h>
-
-int main()
-{
-	int a;
-#pragma scop
-	a = 1;
-#pragma endscop
-	if (a != 1)
-		return EXIT_FAILURE;
-
-	return EXIT_SUCCESS;
-}
diff --git a/polly/lib/External/ppcg/tests/shared_sink.c b/polly/lib/External/ppcg/tests/shared_sink.c
deleted file mode 100644
index ae3fac2fc1e93..0000000000000
--- a/polly/lib/External/ppcg/tests/shared_sink.c
+++ /dev/null
@@ -1,25 +0,0 @@
-#include <stdlib.h>
-
-/* Check that the sources of live ranges with the same sink
- * are executed in order.
- */
-int main()
-{
-	int A[128];
-	int n = 128;
-
-	A[0] = 0;
-#pragma scop
-	for (int i = 0; i < n; ++i) {
-		int set = 0;
-		if (A[i] < 2)
-			set = 1;
-		if (set)
-			A[i] = 2;
-	}
-#pragma endscop
-	if (A[0] != 2)
-		return EXIT_FAILURE;
-
-	return EXIT_SUCCESS;
-}
diff --git a/polly/lib/External/ppcg/tests/struct.c b/polly/lib/External/ppcg/tests/struct.c
deleted file mode 100644
index c67b7bd5619a7..0000000000000
--- a/polly/lib/External/ppcg/tests/struct.c
+++ /dev/null
@@ -1,31 +0,0 @@
-#include <stdlib.h>
-
-struct s {
-	int c[10][10];
-};
-
-int main()
-{
-	struct s a[10][10], b[10][10];
-
-	for (int i = 0; i < 10; ++i)
-		for (int j = 0; j < 10; ++j)
-			for (int k = 0; k < 10; ++k)
-				for (int l = 0; l < 10; ++l)
-					a[i][j].c[k][l] = i + j + k + l;
-#pragma scop
-	for (int i = 0; i < 10; ++i)
-		for (int j = 0; j < 10; ++j)
-			for (int k = 0; k < 10; ++k)
-				for (int l = 0; l < 10; ++l)
-					b[i][j].c[k][l] = i + j + k + l;
-#pragma endscop
-	for (int i = 0; i < 10; ++i)
-		for (int j = 0; j < 10; ++j)
-			for (int k = 0; k < 10; ++k)
-				for (int l = 0; l < 10; ++l)
-					if (b[i][j].c[k][l] != a[i][j].c[k][l])
-						return EXIT_FAILURE;
-
-	return EXIT_SUCCESS;
-}
diff --git a/polly/lib/External/ppcg/tests/struct2.c b/polly/lib/External/ppcg/tests/struct2.c
deleted file mode 100644
index d7d6a9fc91ff6..0000000000000
--- a/polly/lib/External/ppcg/tests/struct2.c
+++ /dev/null
@@ -1,21 +0,0 @@
-#include <stdlib.h>
-
-struct s {
-	int a;
-};
-
-int main()
-{
-	struct s a, b[10];
-
-#pragma scop
-	a.a = 42;
-	for (int i = 0; i < 10; ++i)
-		b[i].a = a.a;
-#pragma endscop
-	for (int i = 0; i < 10; ++i)
-		if (b[i].a != 42)
-			return EXIT_FAILURE;
-
-	return EXIT_SUCCESS;
-}
diff --git a/polly/lib/External/ppcg/tests/struct3.c b/polly/lib/External/ppcg/tests/struct3.c
deleted file mode 100644
index de383b65cdf2a..0000000000000
--- a/polly/lib/External/ppcg/tests/struct3.c
+++ /dev/null
@@ -1,25 +0,0 @@
-#include <stdlib.h>
-
-struct s {
-	int a;
-	int b;
-};
-
-int main()
-{
-	struct s a, b[10];
-
-	a.b = 57;
-#pragma scop
-	a.a = 42;
-	for (int i = 0; i < 10; ++i)
-		b[i] = a;
-#pragma endscop
-	for (int i = 0; i < 10; ++i)
-		if (b[i].a != 42)
-			return EXIT_FAILURE;
-	if (a.b != 57)
-		return EXIT_FAILURE;
-
-	return EXIT_SUCCESS;
-}
diff --git a/polly/lib/External/ppcg/tests/struct4.c b/polly/lib/External/ppcg/tests/struct4.c
deleted file mode 100644
index d249ff4271616..0000000000000
--- a/polly/lib/External/ppcg/tests/struct4.c
+++ /dev/null
@@ -1,27 +0,0 @@
-#include <stdlib.h>
-
-struct s {
-	int a;
-	int b;
-};
-
-int main()
-{
-	int a[10];
-
-	for (int i = 0; i < 10; ++i)
-		a[i] = 0;
-#pragma scop
-	for (int i = 0; i < 10; ++i) {
-		struct s b;
-		b.a = 1;
-		b.b = i;
-		a[i] = b.a + b.b;
-	}
-#pragma endscop
-	for (int i = 0; i < 10; ++i)
-		if (a[i] != 1 + i)
-			return EXIT_FAILURE;
-
-	return EXIT_SUCCESS;
-}
diff --git a/polly/lib/External/ppcg/util.c b/polly/lib/External/ppcg/util.c
deleted file mode 100644
index 0a67bc34d47a0..0000000000000
--- a/polly/lib/External/ppcg/util.c
+++ /dev/null
@@ -1,105 +0,0 @@
-/*
- * Copyright 2012-2013 Ecole Normale Superieure
- *
- * Use of this software is governed by the MIT license
- *
- * Written by Sven Verdoolaege,
- * Ecole Normale Superieure, 45 rue d'Ulm, 75230 Paris, France
- */
-
-#include <isl/space.h>
-#include <isl/val.h>
-#include <isl/aff.h>
-#include <isl/set.h>
-
-#include "util.h"
-
-/* Construct an isl_multi_val living in "space" with all values equal to "val".
- */
-__isl_give isl_multi_val *ppcg_multi_val_from_int(__isl_take isl_space *space,
-	int val)
-{
-	int i, n;
-	isl_ctx *ctx;
-	isl_val *v;
-	isl_multi_val *mv;
-
-	if (!space)
-		return NULL;
-
-	ctx = isl_space_get_ctx(space);
-	n = isl_space_dim(space, isl_dim_set);
-	mv = isl_multi_val_zero(space);
-	v = isl_val_int_from_si(ctx, val);
-	for (i = 0; i < n; ++i)
-		mv = isl_multi_val_set_val(mv, i, isl_val_copy(v));
-	isl_val_free(v);
-
-	return mv;
-}
-
-/* Construct an isl_multi_val living in "space" with values specified
- * by "list".  "list" is assumed to have at least as many entries
- * as the set dimension of "space".
- */
-__isl_give isl_multi_val *ppcg_multi_val_from_int_list(
-	__isl_take isl_space *space, int *list)
-{
-	int i, n;
-	isl_ctx *ctx;
-	isl_multi_val *mv;
-
-	if (!space)
-		return NULL;
-
-	ctx = isl_space_get_ctx(space);
-	n = isl_space_dim(space, isl_dim_set);
-	mv = isl_multi_val_zero(space);
-	for (i = 0; i < n; ++i) {
-		isl_val *v;
-
-		v = isl_val_int_from_si(ctx, list[i]);
-		mv = isl_multi_val_set_val(mv, i, v);
-	}
-
-	return mv;
-}
-
-/* Compute the size of a bounding box around the origin and "set",
- * where "set" is assumed to contain only non-negative elements.
- * In particular, compute the maximal value of "set" in each direction
- * and add one.
- */
-__isl_give isl_multi_pw_aff *ppcg_size_from_extent(__isl_take isl_set *set)
-{
-	int i, n;
-	isl_multi_pw_aff *mpa;
-
-	n = isl_set_dim(set, isl_dim_set);
-	mpa = isl_multi_pw_aff_zero(isl_set_get_space(set));
-	for (i = 0; i < n; ++i) {
-		isl_space *space;
-		isl_aff *one;
-		isl_pw_aff *bound;
-
-		if (!isl_set_dim_has_upper_bound(set, isl_dim_set, i)) {
-			const char *name;
-			name = isl_set_get_tuple_name(set);
-			if (!name)
-				name = "";
-			fprintf(stderr, "unable to determine extent of '%s' "
-				"in dimension %d\n", name, i);
-			set = isl_set_free(set);
-		}
-		bound = isl_set_dim_max(isl_set_copy(set), i);
-
-		space = isl_pw_aff_get_domain_space(bound);
-		one = isl_aff_zero_on_domain(isl_local_space_from_space(space));
-		one = isl_aff_add_constant_si(one, 1);
-		bound = isl_pw_aff_add(bound, isl_pw_aff_from_aff(one));
-		mpa = isl_multi_pw_aff_set_pw_aff(mpa, i, bound);
-	}
-	isl_set_free(set);
-
-	return mpa;
-}
diff --git a/polly/lib/External/ppcg/util.h b/polly/lib/External/ppcg/util.h
deleted file mode 100644
index 544d6edc82d3e..0000000000000
--- a/polly/lib/External/ppcg/util.h
+++ /dev/null
@@ -1,22 +0,0 @@
-#ifndef UTIL_H
-#define UTIL_H
-
-#include <string.h>
-
-#include <isl/space.h>
-#include <isl/val.h>
-
-/* Compare the prefix of "s" to "prefix" up to the length of "prefix".
- */
-static inline int prefixcmp(const char *s, const char *prefix)
-{
-	return strncmp(s, prefix, strlen(prefix));
-}
-
-__isl_give isl_multi_val *ppcg_multi_val_from_int(__isl_take isl_space *space,
-	int val);
-__isl_give isl_multi_val *ppcg_multi_val_from_int_list(
-	__isl_take isl_space *space, int *list);
-__isl_give isl_multi_pw_aff *ppcg_size_from_extent(__isl_take isl_set *set);
-
-#endif
diff --git a/polly/lib/External/ppcg/version.c b/polly/lib/External/ppcg/version.c
deleted file mode 100644
index cc19514605714..0000000000000
--- a/polly/lib/External/ppcg/version.c
+++ /dev/null
@@ -1,6 +0,0 @@
-#include "gitversion.h"
-
-const char *ppcg_version(void)
-{
-	return GIT_HEAD_ID"\n";
-}
diff --git a/polly/lib/Support/RegisterPasses.cpp b/polly/lib/Support/RegisterPasses.cpp
index a00c962f295e2..34570f0c83c5d 100644
--- a/polly/lib/Support/RegisterPasses.cpp
+++ b/polly/lib/Support/RegisterPasses.cpp
@@ -48,7 +48,6 @@
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/TargetSelect.h"
 #include "llvm/Transforms/IPO.h"
-#include "llvm/Transforms/IPO/PassManagerBuilder.h"
 
 namespace cl = llvm::cl;
 
@@ -103,7 +102,6 @@ static cl::opt<VectorizerChoice, true> Vectorizer(
     "polly-vectorizer", cl::desc("Select the vectorization strategy"),
     cl::values(
         clEnumValN(VECTORIZER_NONE, "none", "No Vectorization"),
-        clEnumValN(VECTORIZER_POLLY, "polly", "Polly internal vectorizer"),
         clEnumValN(
             VECTORIZER_STRIPMINE, "stripmine",
             "Strip-mine outer loops for the loop-vectorizer to trigger")),
@@ -218,14 +216,6 @@ static StaticInitializer InitializeEverything;
 void initializePollyPasses(llvm::PassRegistry &Registry) {
   initializeCodeGenerationPass(Registry);
 
-#ifdef GPU_CODEGEN
-  initializePPCGCodeGenerationPass(Registry);
-  initializeManagedMemoryRewritePassPass(Registry);
-  LLVMInitializeNVPTXTarget();
-  LLVMInitializeNVPTXTargetInfo();
-  LLVMInitializeNVPTXTargetMC();
-  LLVMInitializeNVPTXAsmPrinter();
-#endif
   initializeCodePreparationPass(Registry);
   initializeDeadCodeElimWrapperPassPass(Registry);
   initializeDependenceInfoPass(Registry);
diff --git a/polly/lib/Transform/Canonicalization.cpp b/polly/lib/Transform/Canonicalization.cpp
index 8cb235f946018..901299d8f02cd 100644
--- a/polly/lib/Transform/Canonicalization.cpp
+++ b/polly/lib/Transform/Canonicalization.cpp
@@ -56,7 +56,6 @@ void polly::registerCanonicalicationPasses(llvm::legacy::PassManagerBase &PM) {
     PM.add(createBarrierNoopPass());
   }
   PM.add(llvm::createInstructionCombiningPass());
-  PM.add(llvm::createIndVarSimplifyPass());
 }
 
 /// Adapted from llvm::PassBuilder::buildInlinerPipeline
diff --git a/polly/lib/Transform/ScheduleOptimizer.cpp b/polly/lib/Transform/ScheduleOptimizer.cpp
index 94d96d9a9dfd4..35a0a4def0403 100644
--- a/polly/lib/Transform/ScheduleOptimizer.cpp
+++ b/polly/lib/Transform/ScheduleOptimizer.cpp
@@ -711,11 +711,6 @@ static void runIslScheduleOptimizer(
     function_ref<const Dependences &(Dependences::AnalysisLevel)> GetDeps,
     TargetTransformInfo *TTI, OptimizationRemarkEmitter *ORE,
     isl::schedule &LastSchedule, bool &DepsChanged) {
-
-  // Skip SCoPs in case they're already optimised by PPCGCodeGeneration
-  if (S.isToBeSkipped())
-    return;
-
   // Skip empty SCoPs but still allow code generation as it will delete the
   // loops present but not needed.
   if (S.getSize() == 0) {
diff --git a/polly/test/CodeGen/20130211-getNumberOfIterations.ll b/polly/test/CodeGen/20130211-getNumberOfIterations.ll
deleted file mode 100644
index cef0dd559e957..0000000000000
--- a/polly/test/CodeGen/20130211-getNumberOfIterations.ll
+++ /dev/null
@@ -1,27 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen -polly-vectorizer=polly < %s
-
-; This test case checks that the polly vectorizer does not crash when
-; calculating the number of iterations.
-target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
-
-@b = external global [2048 x i64], align 16
-
-define void @foo(i64 %n) {
-entry:
-  br label %for.cond
-
-for.cond:                                         ; preds = %for.body, %entry
-  %indvar = phi i64 [ 0, %entry ], [ %inc, %for.body ]
-  %cmp = icmp slt i64 %indvar, %n
-  br i1 %cmp, label %for.body, label %for.end
-
-for.body:                                         ; preds = %for.cond
-  %arrayidx = getelementptr inbounds [2048 x i64], ptr @b, i64 0, i64 %indvar
-  store i64 1, ptr %arrayidx
-  %inc = add nsw i64 %indvar, 1
-  br label %for.cond
-
-for.end:                                          ; preds = %for.cond
-  ret void
-}
-
diff --git a/polly/test/CodeGen/LoopParallelMD/single_loop_param_parallel.ll b/polly/test/CodeGen/LoopParallelMD/single_loop_param_parallel.ll
index 3351d2f9feb2f..ec927acb1ec73 100644
--- a/polly/test/CodeGen/LoopParallelMD/single_loop_param_parallel.ll
+++ b/polly/test/CodeGen/LoopParallelMD/single_loop_param_parallel.ll
@@ -1,5 +1,5 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -S < %s | FileCheck %s -check-prefix=SEQUENTIAL
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -polly-ast-detect-parallel -S < %s | FileCheck %s -check-prefix=PARALLEL
+; RUN: opt %loadPolly -polly-codegen -S < %s | FileCheck %s -check-prefix=SEQUENTIAL
+; RUN: opt %loadPolly -polly-codegen -polly-ast-detect-parallel -S < %s | FileCheck %s -check-prefix=PARALLEL
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
 
 ; This is a trivially parallel loop. We just use it to ensure that we actually
@@ -20,8 +20,8 @@ loop.header:
   br i1 %exitcond, label %loop.body, label %ret
 
 loop.body:
-  %scevgep = getelementptr [1024 x i32], [1024 x i32]* @A, i64 0, i64 %i
-  store i32 1, i32* %scevgep
+  %scevgep = getelementptr [1024 x i32], ptr @A, i64 0, i64 %i
+  store i32 1, ptr %scevgep
   br label %loop.backedge
 
 loop.backedge:
@@ -39,7 +39,7 @@ ret:
 ; SEQUENTIAL-NOT: !llvm.loop
 
 ; PARALLEL: @test-one
-; PARALLEL: store i32 1, i32* %scevgep1, {{[ ._!,a-zA-Z0-9]*}}, !llvm.access.group ![[GROUPID3:[0-9]+]]
+; PARALLEL: store i32 1, ptr %scevgep1, {{[ ._!,a-zA-Z0-9]*}}, !llvm.access.group ![[GROUPID3:[0-9]+]]
 ; PARALLEL:  br i1 %polly.loop_cond, label %polly.loop_header, label %polly.loop_exit, !llvm.loop ![[LoopID4:[0-9]+]]
 
 
@@ -62,11 +62,11 @@ loop.header:
 loop.body:
   %loadoffset1 = mul nsw i64 %i, 2
   %loadoffset2 = add nsw i64 %loadoffset1, 1
-  %scevgepload = getelementptr [1024 x i32], [1024 x i32]* @A, i64 0, i64 %loadoffset2
-  %val = load i32, i32* %scevgepload
+  %scevgepload = getelementptr [1024 x i32], ptr @A, i64 0, i64 %loadoffset2
+  %val = load i32, ptr %scevgepload
   %storeoffset = mul i64 %i, 2
-  %scevgepstore = getelementptr [1024 x i32], [1024 x i32]* @A, i64 0, i64 %storeoffset
-  store i32 %val, i32* %scevgepstore
+  %scevgepstore = getelementptr [1024 x i32], ptr @A, i64 0, i64 %storeoffset
+  store i32 %val, ptr %scevgepstore
   br label %loop.backedge
 
 loop.backedge:
@@ -84,8 +84,8 @@ ret:
 ; SEQUENTIAL-NOT: !llvm.loop
 
 ; PARALLEL: @test-two
-; PARALLEL: %val_p_scalar_ = load i32, i32* %scevgep, {{[ ._!,a-zA-Z0-9]*}}, !llvm.access.group ![[GROUPID8:[0-9]*]]
-; PARALLEL: store i32 %val_p_scalar_, i32* %scevgep1, {{[ ._!,a-zA-Z0-9]*}}, !llvm.access.group ![[GROUPID8]]
+; PARALLEL: %val_p_scalar_ = load i32, ptr %scevgep, {{[ ._!,a-zA-Z0-9]*}}, !llvm.access.group ![[GROUPID8:[0-9]*]]
+; PARALLEL: store i32 %val_p_scalar_, ptr %scevgep1, {{[ ._!,a-zA-Z0-9]*}}, !llvm.access.group ![[GROUPID8]]
 ; PARALLEL:  br i1 %polly.loop_cond, label %polly.loop_header, label %polly.loop_exit, !llvm.loop ![[LoopID9:[0-9]*]]
 
 
diff --git a/polly/test/CodeGen/MemAccess/codegen_address_space.ll b/polly/test/CodeGen/MemAccess/codegen_address_space.ll
index f7e12e72e27ac..76bcc40f0639a 100644
--- a/polly/test/CodeGen/MemAccess/codegen_address_space.ll
+++ b/polly/test/CodeGen/MemAccess/codegen_address_space.ll
@@ -1,4 +1,4 @@
-;RUN: opt -opaque-pointers=0 %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -instnamer < %s -S | FileCheck %s
+;RUN: opt %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -instnamer < %s -S | FileCheck %s
 
 ;int A[100];
 ;
@@ -11,24 +11,24 @@
 
 target datalayout = "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:32:64-v64:64:64-v128:128:128-a0:0:64-f80:32:32-n8:16:32"
 
-define i32 @codegen_simple([100 x i32] addrspace(5)* %A) nounwind {
+define i32 @codegen_simple(ptr addrspace(5) %A) nounwind {
 entry:
-  %arrayidx.write = getelementptr [100 x i32], [100 x i32] addrspace(5)* %A, i32 0, i32 13
+  %arrayidx.write = getelementptr [100 x i32], ptr addrspace(5) %A, i32 0, i32 13
   br label %for.cond
 
 for.cond:                                         ; preds = %for.inc, %entry
   %tmp1 = phi i32 [ 0, %entry ], [ %inc, %for.inc ]
   %tmp = add i32 %tmp1, -1
-  %arrayidx4 = getelementptr [100 x i32], [100 x i32] addrspace(5)* %A, i32 0, i32 %tmp
-  %arrayidx = getelementptr [100 x i32], [100 x i32] addrspace(5)* %A, i32 0, i32 %tmp1
+  %arrayidx4 = getelementptr [100 x i32], ptr addrspace(5) %A, i32 0, i32 %tmp
+  %arrayidx = getelementptr [100 x i32], ptr addrspace(5) %A, i32 0, i32 %tmp1
   %exitcond = icmp ne i32 %tmp1, 12
   br i1 %exitcond, label %for.body, label %for.end
 
 for.body:                                         ; preds = %for.cond
-  %tmp2 = load i32, i32 addrspace(5)* %arrayidx, align 4
-  %tmp5 = load i32, i32 addrspace(5)* %arrayidx4, align 4
+  %tmp2 = load i32, ptr addrspace(5) %arrayidx, align 4
+  %tmp5 = load i32, ptr addrspace(5) %arrayidx4, align 4
   %add = add nsw i32 %tmp2, %tmp5
-  store i32 %add, i32 addrspace(5)* %arrayidx.write
+  store i32 %add, ptr addrspace(5) %arrayidx.write
   br label %for.inc
 
 for.inc:                                          ; preds = %for.body
@@ -39,6 +39,5 @@ for.end:                                          ; preds = %for.cond
   ret i32 0
 }
 
-; CHECK: %polly.access.cast.A = bitcast [100 x i32] addrspace(5)* %A to i32 addrspace(5)*
-; CHECK: %polly.access.A = getelementptr i32, i32 addrspace(5)* %polly.access.cast.A, i64 0
-; CHECK: %tmp2_p_scalar_ = load i32, i32 addrspace(5)* %polly.access.A, align 4, !alias.scope !0, !noalias !3
+; CHECK: %polly.access.A = getelementptr i32, ptr addrspace(5) %A, i64 0
+; CHECK: %tmp2_p_scalar_ = load i32, ptr addrspace(5) %polly.access.A, align 4, !alias.scope !0, !noalias !3
diff --git a/polly/test/CodeGen/MemAccess/codegen_constant_offset.ll b/polly/test/CodeGen/MemAccess/codegen_constant_offset.ll
index 2a934504fa210..ff12fe77f4698 100644
--- a/polly/test/CodeGen/MemAccess/codegen_constant_offset.ll
+++ b/polly/test/CodeGen/MemAccess/codegen_constant_offset.ll
@@ -1,4 +1,4 @@
-;RUN: opt -opaque-pointers=0 %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -instnamer < %s -S | FileCheck %s
+;RUN: opt %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -instnamer < %s -S | FileCheck %s
 
 ;int A[100];
 ;
@@ -20,16 +20,16 @@ entry:
 for.cond:                                         ; preds = %for.inc, %entry
   %tmp1 = phi i32 [ 0, %entry ], [ %inc, %for.inc ]
   %tmp = add i32 %tmp1, -1
-  %arrayidx4 = getelementptr [100 x i32], [100 x i32]* @A, i32 0, i32 %tmp
-  %arrayidx = getelementptr [100 x i32], [100 x i32]* @A, i32 0, i32 %tmp1
+  %arrayidx4 = getelementptr [100 x i32], ptr @A, i32 0, i32 %tmp
+  %arrayidx = getelementptr [100 x i32], ptr @A, i32 0, i32 %tmp1
   %exitcond = icmp ne i32 %tmp1, 12
   br i1 %exitcond, label %for.body, label %for.end
 
 for.body:                                         ; preds = %for.cond
-  %tmp2 = load i32, i32* %arrayidx, align 4
-  %tmp5 = load i32, i32* %arrayidx4, align 4
+  %tmp2 = load i32, ptr %arrayidx, align 4
+  %tmp5 = load i32, ptr %arrayidx4, align 4
   %add = add nsw i32 %tmp2, %tmp5
-  store i32 %add, i32* getelementptr inbounds ([100 x i32], [100 x i32]* @A, i32 0, i32 13), align 4
+  store i32 %add, ptr getelementptr inbounds ([100 x i32], ptr @A, i32 0, i32 13), align 4
   br label %for.inc
 
 for.inc:                                          ; preds = %for.body
@@ -39,4 +39,4 @@ for.inc:                                          ; preds = %for.body
 for.end:                                          ; preds = %for.cond
   ret i32 0
 }
-; CHECK: load i32, i32* getelementptr inbounds ([100 x i32], [100 x i32]* @A, i{{(32|64)}} 0, i{{(32|64)}} 10)
+; CHECK: load i32, ptr getelementptr (i32, ptr @A, i{{(32|64)}} 10)
diff --git a/polly/test/CodeGen/MemAccess/codegen_simple.ll b/polly/test/CodeGen/MemAccess/codegen_simple.ll
index 21d741fb87886..0e64a6b6fbb8e 100644
--- a/polly/test/CodeGen/MemAccess/codegen_simple.ll
+++ b/polly/test/CodeGen/MemAccess/codegen_simple.ll
@@ -1,4 +1,4 @@
-;RUN: opt -opaque-pointers=0 %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -instnamer < %s -S | FileCheck %s
+;RUN: opt %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -instnamer < %s -S | FileCheck %s
 
 ;int A[100];
 ;
@@ -20,16 +20,16 @@ entry:
 for.cond:                                         ; preds = %for.inc, %entry
   %tmp1 = phi i32 [ 0, %entry ], [ %inc, %for.inc ]
   %tmp = add i32 %tmp1, -1
-  %arrayidx4 = getelementptr [100 x i32], [100 x i32]* @A, i32 0, i32 %tmp
-  %arrayidx = getelementptr [100 x i32], [100 x i32]* @A, i32 0, i32 %tmp1
+  %arrayidx4 = getelementptr [100 x i32], ptr @A, i32 0, i32 %tmp
+  %arrayidx = getelementptr [100 x i32], ptr @A, i32 0, i32 %tmp1
   %exitcond = icmp ne i32 %tmp1, 12
   br i1 %exitcond, label %for.body, label %for.end
 
 for.body:                                         ; preds = %for.cond
-  %tmp2 = load i32, i32* %arrayidx, align 4
-  %tmp5 = load i32, i32* %arrayidx4, align 4
+  %tmp2 = load i32, ptr %arrayidx, align 4
+  %tmp5 = load i32, ptr %arrayidx4, align 4
   %add = add nsw i32 %tmp2, %tmp5
-  store i32 %add, i32* getelementptr inbounds ([100 x i32], [100 x i32]* @A, i32 0, i32 13), align 4
+  store i32 %add, ptr getelementptr inbounds ([100 x i32], ptr @A, i32 0, i32 13), align 4
   br label %for.inc
 
 for.inc:                                          ; preds = %for.body
@@ -39,4 +39,4 @@ for.inc:                                          ; preds = %for.body
 for.end:                                          ; preds = %for.cond
   ret i32 0
 }
-; CHECK: load i32, i32* getelementptr inbounds ([100 x i32], [100 x i32]* @A, i{{(32|64)}} 0, i{{(32|64)}} 0)
+; CHECK: load i32, ptr @A
diff --git a/polly/test/CodeGen/MemAccess/codegen_simple_float.ll b/polly/test/CodeGen/MemAccess/codegen_simple_float.ll
index b1a46e7b80f4f..65a2d3b9067bb 100644
--- a/polly/test/CodeGen/MemAccess/codegen_simple_float.ll
+++ b/polly/test/CodeGen/MemAccess/codegen_simple_float.ll
@@ -1,4 +1,4 @@
-;RUN: opt -opaque-pointers=0 %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -instnamer < %s -S | FileCheck %s
+;RUN: opt %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -instnamer < %s -S | FileCheck %s
 ;
 ;float A[100];
 ;
@@ -19,16 +19,16 @@ entry:
 for.cond:                                         ; preds = %for.inc, %entry
   %tmp1 = phi i32 [ 0, %entry ], [ %inc, %for.inc ]
   %tmp = add i32 %tmp1, -1
-  %arrayidx4 = getelementptr [100 x float], [100 x float]* @A, i32 0, i32 %tmp
-  %arrayidx = getelementptr [100 x float], [100 x float]* @A, i32 0, i32 %tmp1
+  %arrayidx4 = getelementptr [100 x float], ptr @A, i32 0, i32 %tmp
+  %arrayidx = getelementptr [100 x float], ptr @A, i32 0, i32 %tmp1
   %exitcond = icmp ne i32 %tmp1, 12
   br i1 %exitcond, label %for.body, label %for.end
 
 for.body:                                         ; preds = %for.cond
-  %tmp2 = load float, float* %arrayidx, align 4
-  %tmp5 = load float, float* %arrayidx4, align 4
+  %tmp2 = load float, ptr %arrayidx, align 4
+  %tmp5 = load float, ptr %arrayidx4, align 4
   %add = fadd float %tmp2, %tmp5
-  store float %add, float* getelementptr inbounds ([100 x float], [100 x float]* @A, i32 0, i32 13), align 4
+  store float %add, ptr getelementptr inbounds ([100 x float], ptr @A, i32 0, i32 13), align 4
   br label %for.inc
 
 for.inc:                                          ; preds = %for.body
@@ -38,4 +38,4 @@ for.inc:                                          ; preds = %for.body
 for.end:                                          ; preds = %for.cond
   ret i32 0
 }
-; CHECK: load float, float* getelementptr inbounds ([100 x float], [100 x float]* @A, i{{(32|64)}} 0, i{{(32|64)}} 0)
+; CHECK: load float, ptr @A
diff --git a/polly/test/CodeGen/MemAccess/codegen_simple_md.ll b/polly/test/CodeGen/MemAccess/codegen_simple_md.ll
index 987f113007dda..e4afcc8d22439 100644
--- a/polly/test/CodeGen/MemAccess/codegen_simple_md.ll
+++ b/polly/test/CodeGen/MemAccess/codegen_simple_md.ll
@@ -1,5 +1,5 @@
-;RUN: opt -opaque-pointers=0 %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed+withconst -polly-codegen < %s -S | FileCheck -check-prefix=WITHCONST %s
-;RUN: opt -opaque-pointers=0 %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed+withoutconst -polly-codegen < %s -S | FileCheck -check-prefix=WITHOUTCONST %s
+;RUN: opt %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed+withconst -polly-codegen < %s -S | FileCheck -check-prefix=WITHCONST %s
+;RUN: opt %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed+withoutconst -polly-codegen < %s -S | FileCheck -check-prefix=WITHOUTCONST %s
 
 ;int A[1040];
 ;
@@ -35,8 +35,8 @@ for.cond1:                                        ; preds = %for.inc, %for.body
 for.body3:                                        ; preds = %for.cond1
   %mul = shl nsw i32 %i.0, 5
   %add = add nsw i32 %mul, %j.0
-  %arrayidx = getelementptr inbounds [1040 x i32], [1040 x i32]* @A, i32 0, i32 %add
-  store i32 100, i32* %arrayidx, align 4
+  %arrayidx = getelementptr inbounds [1040 x i32], ptr @A, i32 0, i32 %add
+  store i32 100, ptr %arrayidx, align 4
   br label %for.inc
 
 for.inc:                                          ; preds = %for.body3
@@ -60,13 +60,13 @@ for.end6:                                         ; preds = %for.cond
 ; WITHCONST:  %[[MUL2:[._a-zA-Z0-9]+]] = mul nsw i64 2, %[[IVIn]]
 ; WITHCONST:  %[[SUM1:[._a-zA-Z0-9]+]] = add nsw i64 %[[MUL1]], %[[MUL2]]
 ; WITHCONST:  %[[SUM2:[._a-zA-Z0-9]+]] = add nsw i64 %[[SUM1]], 5
-; WITHCONST:  %[[ACC:[._a-zA-Z0-9]*]] = getelementptr i32, i32* getelementptr inbounds ([1040 x i32], [1040 x i32]* @A, i{{(32|64)}} 0, i{{(32|64)}} 0), i64 %[[SUM2]]
-; WITHCONST:  store i32 100, i32* %[[ACC]]
+; WITHCONST:  %[[ACC:[._a-zA-Z0-9]*]] = getelementptr i32, ptr @A, i64 %[[SUM2]]
+; WITHCONST:  store i32 100, ptr %[[ACC]]
 
 ; WITHOUTCONST:  %[[IVOut:polly.indvar[0-9]*]] = phi i64 [ 0, %polly.loop_preheader{{[0-9]*}} ], [ %polly.indvar_next{{[0-9]*}}, %polly.{{[._a-zA-Z0-9]*}} ]
 ; WITHOUTCONST:  %[[IVIn:polly.indvar[0-9]*]] = phi i64 [ 0, %polly.loop_preheader{{[0-9]*}} ], [ %polly.indvar_next{{[0-9]*}}, %polly.{{[._a-zA-Z0-9]*}} ]
 ; WITHOUTCONST:  %[[MUL1:[._a-zA-Z0-9]+]] = mul nsw i64 16, %[[IVOut]]
 ; WITHOUTCONST:  %[[MUL2:[._a-zA-Z0-9]+]] = mul nsw i64 2, %[[IVIn]]
 ; WITHOUTCONST:  %[[SUM1:[._a-zA-Z0-9]+]] = add nsw i64 %[[MUL1]], %[[MUL2]]
-; WITHOUTCONST:  %[[ACC:[._a-zA-Z0-9]*]] = getelementptr i32, i32* getelementptr inbounds ([1040 x i32], [1040 x i32]* @A, i{{(32|64)}} 0, i{{(32|64)}} 0), i64 %[[SUM1]]
-; WITHOUTCONST:  store i32 100, i32* %[[ACC]]
+; WITHOUTCONST:  %[[ACC:[._a-zA-Z0-9]*]] = getelementptr i32, ptr @A, i64 %[[SUM1]]
+; WITHOUTCONST:  store i32 100, ptr %[[ACC]]
diff --git a/polly/test/CodeGen/MemAccess/codegen_simple_md_float.ll b/polly/test/CodeGen/MemAccess/codegen_simple_md_float.ll
index fb8ee4eaebc45..c9913f3ed8733 100644
--- a/polly/test/CodeGen/MemAccess/codegen_simple_md_float.ll
+++ b/polly/test/CodeGen/MemAccess/codegen_simple_md_float.ll
@@ -1,5 +1,5 @@
-;RUN: opt -opaque-pointers=0 %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed+withconst -polly-codegen < %s -S | FileCheck -check-prefix=WITHCONST %s
-;RUN: opt -opaque-pointers=0 %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed+withoutconst -polly-codegen < %s -S | FileCheck -check-prefix=WITHOUTCONST %s
+;RUN: opt %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed+withconst -polly-codegen < %s -S | FileCheck -check-prefix=WITHCONST %s
+;RUN: opt %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed+withoutconst -polly-codegen < %s -S | FileCheck -check-prefix=WITHOUTCONST %s
 ;
 ;float A[1040];
 ;
@@ -32,8 +32,8 @@ for.cond1:                                        ; preds = %for.inc, %for.body
 for.body3:                                        ; preds = %for.cond1
   %mul = shl nsw i32 %i.0, 5
   %add = add nsw i32 %mul, %j.0
-  %arrayidx = getelementptr inbounds [1040 x float], [1040 x float]* @A, i32 0, i32 %add
-  store float 100.0, float* %arrayidx, align 4
+  %arrayidx = getelementptr inbounds [1040 x float], ptr @A, i32 0, i32 %add
+  store float 100.0, ptr %arrayidx, align 4
   br label %for.inc
 
 for.inc:                                          ; preds = %for.body3
@@ -57,13 +57,13 @@ for.end6:                                         ; preds = %for.cond
 ; WITHCONST:  %[[MUL2:[._a-zA-Z0-9]+]] = mul nsw i64 2, %[[IVIn]]
 ; WITHCONST:  %[[SUM1:[._a-zA-Z0-9]+]] = add nsw i64 %[[MUL1]], %[[MUL2]]
 ; WITHCONST:  %[[SUM2:[._a-zA-Z0-9]+]] = add nsw i64 %[[SUM1]], 5
-; WITHCONST:  %[[ACC:[._a-zA-Z0-9]*]] = getelementptr float, float* getelementptr inbounds ([1040 x float], [1040 x float]* @A, i{{(32|64)}} 0, i{{(32|64)}} 0), i64 %[[SUM2]]
-; WITHCONST:  store float 1.000000e+02, float* %[[ACC]]
+; WITHCONST:  %[[ACC:[._a-zA-Z0-9]*]] = getelementptr float, ptr @A, i64 %[[SUM2]]
+; WITHCONST:  store float 1.000000e+02, ptr %[[ACC]]
 
 ; WITHOUTCONST:  %[[IVOut:polly.indvar[0-9]*]] = phi i64 [ 0, %polly.loop_preheader{{[0-9]*}} ], [ %polly.indvar_next{{[0-9]*}}, %polly.{{[._a-zA-Z0-9]*}} ]
 ; WITHOUTCONST:  %[[IVIn:polly.indvar[0-9]*]] = phi i64 [ 0, %polly.loop_preheader{{[0-9]*}} ], [ %polly.indvar_next{{[0-9]*}}, %polly.{{[._a-zA-Z0-9]*}} ]
 ; WITHOUTCONST:  %[[MUL1:[._a-zA-Z0-9]+]] = mul nsw i64 16, %[[IVOut]]
 ; WITHOUTCONST:  %[[MUL2:[._a-zA-Z0-9]+]] = mul nsw i64 2, %[[IVIn]]
 ; WITHOUTCONST:  %[[SUM1:[._a-zA-Z0-9]+]] = add nsw i64 %[[MUL1]], %[[MUL2]]
-; WITHOUTCONST:  %[[ACC:[._a-zA-Z0-9]*]] = getelementptr float, float* getelementptr inbounds ([1040 x float], [1040 x float]* @A, i{{(32|64)}} 0, i{{(32|64)}} 0), i64 %[[SUM1]]
-; WITHOUTCONST:  store float 1.000000e+02, float* %[[ACC]]
+; WITHOUTCONST:  %[[ACC:[._a-zA-Z0-9]*]] = getelementptr float, ptr @A, i64 %[[SUM1]]
+; WITHOUTCONST:  store float 1.000000e+02, ptr %[[ACC]]
diff --git a/polly/test/CodeGen/MemAccess/create_arrays.ll b/polly/test/CodeGen/MemAccess/create_arrays.ll
index 07afc7f230d11..b454e028b982c 100644
--- a/polly/test/CodeGen/MemAccess/create_arrays.ll
+++ b/polly/test/CodeGen/MemAccess/create_arrays.ll
@@ -1,5 +1,5 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-print-scops -polly-print-import-jscop -polly-import-jscop-postfix=transformed -disable-output < %s 2>&1 | FileCheck %s
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -S < %s 2>&1 | FileCheck %s --check-prefix=CODEGEN
+; RUN: opt %loadPolly -polly-print-scops -polly-print-import-jscop -polly-import-jscop-postfix=transformed -disable-output < %s 2>&1 | FileCheck %s
+; RUN: opt %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -S < %s 2>&1 | FileCheck %s --check-prefix=CODEGEN
 ;
 ;  for (i = 0; i < _PB_NI; i++)
 ;    for (j = 0; j < _PB_NJ; j++)
@@ -17,7 +17,7 @@
 ;
 ; CHECK:New access function '{ Stmt_bb12[i0, i1, i2] -> E[i2, i0] }' detected in JSCOP file
 ;
-; CODEGEN:define internal void @create_arrays(i32 %arg, i32 %arg1, i32 %arg2, double %arg3, double %beta, [1056 x double]* %A, [1024 x double]* %B, [1056 x double]* %arg7) #0 {
+; CODEGEN:define internal void @create_arrays(i32 %arg, i32 %arg1, i32 %arg2, double %arg3, double %beta, ptr %A, ptr %B, ptr %arg7) #0 {
 ; CODEGEN:bb:
 ; CODEGEN:  %beta.s2a = alloca double
 ; CODEGEN:  %D = alloca [270336 x double]
@@ -25,12 +25,11 @@
 ; CODEGEN:  %F = alloca [270336 x i64]
 ; CODEGEN:  br label %bb8
 ;
-; CODEGEN:  %beta.s2a.reload = load double, double* %beta.s2a
-; CODEGEN:  %polly.access.cast.E = bitcast [270336 x [200000 x double]]* %E to double*
-; CODEGEN:  %polly.access.mul.E = mul nsw i64 %polly.indvar33, 200000
+; CODEGEN:  %beta.s2a.reload = load double, ptr %beta.s2a
+; CODEGEN:  %polly.access.mul.E = mul nsw i64 %polly.indvar31, 200000
 ; CODEGEN:  %polly.access.add.E = add nsw i64 %polly.access.mul.E, %polly.indvar
-; CODEGEN:  {{%.*}} = load double, double* %polly.access.E, align 8, !alias.scope !0, !noalias !3
-; CODEGEN:  store double {{%.*}}, double* %scevgep36, align 8, !alias.scope !8, !noalias !9
+; CODEGEN:  {{%.*}} = load double, ptr %polly.access.E, align 8, !alias.scope !0, !noalias !3
+; CODEGEN:  store double {{%.*}}, ptr %scevgep34, align 8, !alias.scope !8, !noalias !9
 ;
 ; CODEGEN: !0 = !{!1}
 ; CODEGEN: !1 = distinct !{!1, !2, !"polly.alias.scope.E"}
@@ -47,7 +46,7 @@ target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-unknown"
 
 ; Function Attrs: nounwind uwtable
-define internal void @create_arrays(i32 %arg, i32 %arg1, i32 %arg2, double %arg3, double %beta, [1056 x double]* %A, [1024 x double]* %B, [1056 x double]* %arg7) #0 {
+define internal void @create_arrays(i32 %arg, i32 %arg1, i32 %arg2, double %arg3, double %beta, ptr %A, ptr %B, ptr %arg7) #0 {
 bb:
   br label %bb8
 
@@ -64,11 +63,11 @@ bb10:                                             ; preds = %bb20, %bb9
 
 bb12:                                             ; preds = %bb12, %bb10
   %tmp13 = phi i64 [ 0, %bb10 ], [ %tmp18, %bb12 ]
-  %tmp14 = getelementptr inbounds [1024 x double], [1024 x double]* %B, i64 %tmp, i64 %tmp13
-  %tmp15 = load double, double* %tmp14, align 8
+  %tmp14 = getelementptr inbounds [1024 x double], ptr %B, i64 %tmp, i64 %tmp13
+  %tmp15 = load double, ptr %tmp14, align 8
   %tmp16 = fmul double %tmp15, %beta
-  %tmp17 = getelementptr inbounds [1056 x double], [1056 x double]* %A, i64 %tmp, i64 %tmp11
-  store double %tmp16, double* %tmp17, align 8
+  %tmp17 = getelementptr inbounds [1056 x double], ptr %A, i64 %tmp, i64 %tmp11
+  store double %tmp16, ptr %tmp17, align 8
   %tmp18 = add nuw nsw i64 %tmp13, 1
   %tmp19 = icmp ne i64 %tmp18, 1024
   br i1 %tmp19, label %bb12, label %bb20
diff --git a/polly/test/CodeGen/MemAccess/create_arrays_heap.ll b/polly/test/CodeGen/MemAccess/create_arrays_heap.ll
index ff500d0ea7b70..ada9bd98d7617 100644
--- a/polly/test/CodeGen/MemAccess/create_arrays_heap.ll
+++ b/polly/test/CodeGen/MemAccess/create_arrays_heap.ll
@@ -1,5 +1,5 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-stmt-granularity=bb -polly-print-scops -polly-print-import-jscop -polly-import-jscop-postfix=transformed -disable-output < %s | FileCheck %s
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-stmt-granularity=bb -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -S < %s | FileCheck %s --check-prefix=CODEGEN
+; RUN: opt %loadPolly -polly-stmt-granularity=bb -polly-print-scops -polly-print-import-jscop -polly-import-jscop-postfix=transformed -disable-output < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-stmt-granularity=bb -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -S < %s | FileCheck %s --check-prefix=CODEGEN
 ;
 ; #define Ni 1056
 ; #define Nj 1056
@@ -31,27 +31,21 @@
 ;	%E : size(E) = 270336*200000*8 = 432537600000 cast to double*
 ; 	%F : size(F) = 270336*8 = 2162688 cast to i64*
 ; CODEGEN: polly.start:
-; CODEGEN: %malloccall = tail call i8* @malloc(i64 2162688)
-; CODEGEN: %D = bitcast i8* %malloccall to double*
-; CODEGEN: %malloccall1 = tail call i8* @malloc(i64 432537600000)
-; CODEGEN: %E = bitcast i8* %malloccall1 to double*
-; CODEGEN: %malloccall2 = tail call i8* @malloc(i64 2162688)
-; CODEGEN: %F = bitcast i8* %malloccall2 to i64*
+; CODEGEN: %malloccall = tail call ptr @malloc(i64 2162688)
+; CODEGEN: %malloccall1 = tail call ptr @malloc(i64 432537600000)
+; CODEGEN: %malloccall2 = tail call ptr @malloc(i64 2162688)
 ;
 ; Check if there are the 3 expected malloc calls with the right parameters at polly.exiting.
 ; 	Cast to i8* before freeing because malloc give us a i8 and free is waiting for a i8*
 ; CODEGEN: polly.exiting:
-; CODEGEN: %12 = bitcast double* %D to i8*
-; CODEGEN: tail call void @free(i8* %12)
-; CODEGEN: %13 = bitcast double* %E to i8*
-; CODEGEN: tail call void @free(i8* %13)
-; CODEGEN: %14 = bitcast i64* %F to i8*
-; CODEGEN: tail call void @free(i8* %14)
+; CODEGEN: tail call void @free(ptr %malloccall)
+; CODEGEN: tail call void @free(ptr %malloccall1)
+; CODEGEN: tail call void @free(ptr %malloccall2)
 ;
 ; Check if the new access for array E is present.
-; CODEGEN: %polly.access.mul.E = mul nsw i64 %polly.indvar, 200000
-; CODEGEN: %polly.access.add.E = add nsw i64 %polly.access.mul.E, %
-; CODEGEN: %polly.access.E = getelementptr double, double* %E, i64 %polly.access.add.E
+; CODEGEN: %polly.access.mul.{{.*}} = mul nsw i64 %polly.indvar, 200000
+; CODEGEN: %polly.access.add.{{.*}} = add nsw i64 %polly.access.mul.{{.*}}, %
+; CODEGEN: %polly.access.{{.*}} = getelementptr double, ptr %malloccall1, i64 %polly.access.add.{{.*}}
 ;
 ; ModuleID = 'create_arrays_heap.ll'
 ;
@@ -59,7 +53,7 @@ target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; Function Attrs: nounwind uwtable
-define void @create_arrays_heap(double %beta, [1024 x double]* nocapture readonly %A, [1056 x double]* nocapture %B) local_unnamed_addr {
+define void @create_arrays_heap(double %beta, ptr nocapture readonly %A, ptr nocapture %B) local_unnamed_addr {
 entry:
   br label %for.cond1.preheader
 
@@ -69,30 +63,30 @@ for.cond1.preheader:                              ; preds = %for.inc16, %entry
 
 for.cond4.preheader:                              ; preds = %for.inc13, %for.cond1.preheader
   %indvars.iv32 = phi i64 [ 0, %for.cond1.preheader ], [ %indvars.iv.next33, %for.inc13 ]
-  %arrayidx12 = getelementptr inbounds [1056 x double], [1056 x double]* %B, i64 %indvars.iv35, i64 %indvars.iv32
+  %arrayidx12 = getelementptr inbounds [1056 x double], ptr %B, i64 %indvars.iv35, i64 %indvars.iv32
   br label %for.body6
 
 for.body6:                                        ; preds = %for.body6, %for.cond4.preheader
   %indvars.iv = phi i64 [ 0, %for.cond4.preheader ], [ %indvars.iv.next.3, %for.body6 ]
-  %arrayidx8 = getelementptr inbounds [1024 x double], [1024 x double]* %A, i64 %indvars.iv35, i64 %indvars.iv
-  %0 = load double, double* %arrayidx8, align 8
+  %arrayidx8 = getelementptr inbounds [1024 x double], ptr %A, i64 %indvars.iv35, i64 %indvars.iv
+  %0 = load double, ptr %arrayidx8, align 8
   %mul = fmul double %0, %beta
-  store double %mul, double* %arrayidx12, align 8
+  store double %mul, ptr %arrayidx12, align 8
   %indvars.iv.next = or i64 %indvars.iv, 1
-  %arrayidx8.1 = getelementptr inbounds [1024 x double], [1024 x double]* %A, i64 %indvars.iv35, i64 %indvars.iv.next
-  %1 = load double, double* %arrayidx8.1, align 8
+  %arrayidx8.1 = getelementptr inbounds [1024 x double], ptr %A, i64 %indvars.iv35, i64 %indvars.iv.next
+  %1 = load double, ptr %arrayidx8.1, align 8
   %mul.1 = fmul double %1, %beta
-  store double %mul.1, double* %arrayidx12, align 8
+  store double %mul.1, ptr %arrayidx12, align 8
   %indvars.iv.next.1 = or i64 %indvars.iv, 2
-  %arrayidx8.2 = getelementptr inbounds [1024 x double], [1024 x double]* %A, i64 %indvars.iv35, i64 %indvars.iv.next.1
-  %2 = load double, double* %arrayidx8.2, align 8
+  %arrayidx8.2 = getelementptr inbounds [1024 x double], ptr %A, i64 %indvars.iv35, i64 %indvars.iv.next.1
+  %2 = load double, ptr %arrayidx8.2, align 8
   %mul.2 = fmul double %2, %beta
-  store double %mul.2, double* %arrayidx12, align 8
+  store double %mul.2, ptr %arrayidx12, align 8
   %indvars.iv.next.2 = or i64 %indvars.iv, 3
-  %arrayidx8.3 = getelementptr inbounds [1024 x double], [1024 x double]* %A, i64 %indvars.iv35, i64 %indvars.iv.next.2
-  %3 = load double, double* %arrayidx8.3, align 8
+  %arrayidx8.3 = getelementptr inbounds [1024 x double], ptr %A, i64 %indvars.iv35, i64 %indvars.iv.next.2
+  %3 = load double, ptr %arrayidx8.3, align 8
   %mul.3 = fmul double %3, %beta
-  store double %mul.3, double* %arrayidx12, align 8
+  store double %mul.3, ptr %arrayidx12, align 8
   %indvars.iv.next.3 = add nsw i64 %indvars.iv, 4
   %exitcond.3 = icmp eq i64 %indvars.iv.next.3, 1024
   br i1 %exitcond.3, label %for.inc13, label %for.body6
diff --git a/polly/test/CodeGen/MemAccess/different_types.ll b/polly/test/CodeGen/MemAccess/different_types.ll
index 91bafcd11984f..624de62911ff6 100644
--- a/polly/test/CodeGen/MemAccess/different_types.ll
+++ b/polly/test/CodeGen/MemAccess/different_types.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-import-jscop \
+; RUN: opt %loadPolly -polly-import-jscop \
 ; RUN: \
 ; RUN: -polly-codegen -S < %s | FileCheck %s
 ;
@@ -9,16 +9,14 @@
 ;        A[i] += 10;
 ;    }
 
-; CHECK: %polly.access.cast.A14 = bitcast float* %A to i32*
-; CHECK: %[[R1:[._0-9]*]] = sub nsw i64 0, %polly.indvar11
+; CHECK: %[[R1:[._0-9]*]] = sub nsw i64 0, %polly.indvar7
 ; CHECK: %[[R2:[._0-9]*]] = add nsw i64 %[[R1]], 99
-; CHECK: %polly.access.A15 = getelementptr i32, i32* %polly.access.cast.A14, i64 %[[R2]]
-; CHECK: %[[R3:[._0-9]*]] = bitcast i32* %polly.access.A15 to float*
-; CHECK: %tmp14_p_scalar_ = load float, float* %[[R3]], align 4, !alias.scope !3, !noalias !0
+; CHECK: %polly.access.A10 = getelementptr i32, ptr %A, i64 %[[R2]]
+; CHECK: %tmp14_p_scalar_ = load float, ptr %polly.access.A10, align 4, !alias.scope !3, !noalias !0
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @different_types(float* %A, float* %B) {
+define void @different_types(ptr %A, ptr %B) {
 bb:
   br label %bb2
 
@@ -28,12 +26,10 @@ bb2:                                              ; preds = %bb8, %bb
   br i1 %exitcond1, label %bb3, label %bb10
 
 bb3:                                              ; preds = %bb2
-  %tmp = getelementptr inbounds float, float* %B, i64 %i.0
-  %tmp4 = bitcast float* %tmp to i32*
-  %tmp5 = load i32, i32* %tmp4, align 4
-  %tmp6 = getelementptr inbounds float, float* %A, i64 %i.0
-  %tmp7 = bitcast float* %tmp6 to i32*
-  store i32 %tmp5, i32* %tmp7, align 4
+  %tmp = getelementptr inbounds float, ptr %B, i64 %i.0
+  %tmp5 = load i32, ptr %tmp, align 4
+  %tmp6 = getelementptr inbounds float, ptr %A, i64 %i.0
+  store i32 %tmp5, ptr %tmp6, align 4
   br label %bb8
 
 bb8:                                              ; preds = %bb3
@@ -49,10 +45,10 @@ bb11:                                             ; preds = %bb16, %bb10
   br i1 %exitcond, label %bb12, label %bb18
 
 bb12:                                             ; preds = %bb11
-  %tmp13 = getelementptr inbounds float, float* %A, i64 %i1.0
-  %tmp14 = load float, float* %tmp13, align 4
+  %tmp13 = getelementptr inbounds float, ptr %A, i64 %i1.0
+  %tmp14 = load float, ptr %tmp13, align 4
   %tmp15 = fadd float %tmp14, 1.000000e+01
-  store float %tmp15, float* %tmp13, align 4
+  store float %tmp15, ptr %tmp13, align 4
   br label %bb16
 
 bb16:                                             ; preds = %bb12
diff --git a/polly/test/CodeGen/MemAccess/generate-all.ll b/polly/test/CodeGen/MemAccess/generate-all.ll
index 8da2b0ed12eb8..6f92ba13587e3 100644
--- a/polly/test/CodeGen/MemAccess/generate-all.ll
+++ b/polly/test/CodeGen/MemAccess/generate-all.ll
@@ -1,6 +1,6 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -polly-codegen-generate-expressions=false \
+; RUN: opt %loadPolly -polly-codegen -polly-codegen-generate-expressions=false \
 ; RUN:     -S < %s | FileCheck %s -check-prefix=SCEV
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -polly-codegen-generate-expressions=true \
+; RUN: opt %loadPolly -polly-codegen -polly-codegen-generate-expressions=true \
 ; RUN:     -S < %s | FileCheck %s -check-prefix=ASTEXPR
 ;
 ;    void foo(float A[]) {
@@ -11,29 +11,30 @@
 ; SCEV:      polly.stmt.bb2:                                   ; preds = %polly.loop_header
 ; SCEV-NEXT:   %0 = trunc i64 %polly.indvar to i2
 ; SCEV-NEXT:   %1 = zext i2 %0 to i64
-; SCEV-NEXT:   %scevgep = getelementptr float, float* %A, i64 %1
-; SCEV-NEXT:   %tmp4_p_scalar_ = load float, float* %scevgep, align 4, !alias.scope !0, !noalias !3
+; SCEV-NEXT:   %2 = shl nuw nsw i64 %1, 2
+; SCEV-NEXT:   %scevgep = getelementptr i8, ptr %A, i64 %2
+; SCEV-NEXT:   %tmp4_p_scalar_ = load float, ptr %scevgep, align 4, !alias.scope !0, !noalias !3
 ; SCEV-NEXT:   %p_tmp5 = fadd float %tmp4_p_scalar_, 1.000000e+01
-; SCEV-NEXT:   store float %p_tmp5, float* %scevgep, align 4, !alias.scope !0, !noalias !3
+; SCEV-NEXT:   store float %p_tmp5, ptr %scevgep, align 4, !alias.scope !0, !noalias !3
 ; SCEV-NEXT:   %polly.indvar_next = add nsw i64 %polly.indvar, 1
 ; SCEV-NEXT:   %polly.loop_cond = icmp sle i64 %polly.indvar_next, 99
 ; SCEV-NEXT:   br i1 %polly.loop_cond, label %polly.loop_header, label %polly.loop_exit
 
 ; ASTEXPR: 	polly.stmt.bb2:                                   ; preds = %polly.loop_header
 ; ASTEXPR-NEXT:   %pexp.pdiv_r = urem i64 %polly.indvar, 4
-; ASTEXPR-NEXT:   %polly.access.A = getelementptr float, float* %A, i64 %pexp.pdiv_r
-; ASTEXPR-NEXT:   %tmp4_p_scalar_ = load float, float* %polly.access.A, align 4, !alias.scope !0, !noalias !3
+; ASTEXPR-NEXT:   %polly.access.A = getelementptr float, ptr %A, i64 %pexp.pdiv_r
+; ASTEXPR-NEXT:   %tmp4_p_scalar_ = load float, ptr %polly.access.A, align 4, !alias.scope !0, !noalias !3
 ; ASTEXPR-NEXT:   %p_tmp5 = fadd float %tmp4_p_scalar_, 1.000000e+01
 ; ASTEXPR-NEXT:   %pexp.pdiv_r1 = urem i64 %polly.indvar, 4
-; ASTEXPR-NEXT:   %polly.access.A2 = getelementptr float, float* %A, i64 %pexp.pdiv_r1
-; ASTEXPR-NEXT:   store float %p_tmp5, float* %polly.access.A2, align 4, !alias.scope !0, !noalias !3
+; ASTEXPR-NEXT:   %polly.access.A2 = getelementptr float, ptr %A, i64 %pexp.pdiv_r1
+; ASTEXPR-NEXT:   store float %p_tmp5, ptr %polly.access.A2, align 4, !alias.scope !0, !noalias !3
 ; ASTEXPR-NEXT:   %polly.indvar_next = add nsw i64 %polly.indvar, 1
 ; ASTEXPR-NEXT:   %polly.loop_cond = icmp sle i64 %polly.indvar_next, 99
 ; ASTEXPR-NEXT:   br i1 %polly.loop_cond, label %polly.loop_header, label %polly.loop_exit
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @foo(float* %A) {
+define void @foo(ptr %A) {
 bb:
   br label %bb1
 
@@ -44,10 +45,10 @@ bb1:                                              ; preds = %bb6, %bb
 
 bb2:                                              ; preds = %bb1
   %tmp = srem i64 %i.0, 4
-  %tmp3 = getelementptr inbounds float, float* %A, i64 %tmp
-  %tmp4 = load float, float* %tmp3, align 4
+  %tmp3 = getelementptr inbounds float, ptr %A, i64 %tmp
+  %tmp4 = load float, ptr %tmp3, align 4
   %tmp5 = fadd float %tmp4, 1.000000e+01
-  store float %tmp5, float* %tmp3, align 4
+  store float %tmp5, ptr %tmp3, align 4
   br label %bb6
 
 bb6:                                              ; preds = %bb2
diff --git a/polly/test/CodeGen/MemAccess/invariant_base_ptr.ll b/polly/test/CodeGen/MemAccess/invariant_base_ptr.ll
index 124a11849b071..a6d1de0aac637 100644
--- a/polly/test/CodeGen/MemAccess/invariant_base_ptr.ll
+++ b/polly/test/CodeGen/MemAccess/invariant_base_ptr.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-import-jscop \
+; RUN: opt %loadPolly -polly-import-jscop \
 ; RUN:   -polly-codegen -polly-invariant-load-hoisting -S \
 ; RUN:   2>&1 < %s | FileCheck %s
 
@@ -7,10 +7,10 @@
 ; generated when invariant load hoisting is disabled. This test case checks
 ; that in case invariant load hoisting is enabled, we generate correct code.
 
-; CHECK: %polly.access.polly.access.X.load = getelementptr float, float* %polly.access.X.load, i64 %polly.indvar
+; CHECK: %polly.access.polly.access.X.load = getelementptr float, ptr %polly.access.X.load, i64 %polly.indvar
 
-define void @invariant_base_ptr(float* noalias %Array, float** noalias %X,
-                                float* noalias %C) {
+define void @invariant_base_ptr(ptr noalias %Array, ptr noalias %X,
+                                ptr noalias %C) {
 
 start:
   br label %loop
@@ -22,15 +22,15 @@ loop:
   br i1 %cmp, label %body, label %exit
 
 body:
-  %gep= getelementptr float, float* %Array, i64 %indvar
-  store float 42.0, float* %gep
+  %gep= getelementptr float, ptr %Array, i64 %indvar
+  store float 42.0, ptr %gep
   br label %body2
 
 body2:
-  %Base = load float*, float** %X
-  %gep2 = getelementptr float, float* %Base, i64 %indvar
-  %val2 = load float, float* %gep2
-  store float %val2, float* %C
+  %Base = load ptr, ptr %X
+  %gep2 = getelementptr float, ptr %Base, i64 %indvar
+  %val2 = load float, ptr %gep2
+  store float %val2, ptr %C
   br label %latch
 
 latch:
diff --git a/polly/test/CodeGen/MemAccess/invariant_base_ptr___%loop---%exit.jscop b/polly/test/CodeGen/MemAccess/invariant_base_ptr___%loop---%exit.jscop
index 94bfb2f59e34c..42ca0de578cab 100644
--- a/polly/test/CodeGen/MemAccess/invariant_base_ptr___%loop---%exit.jscop
+++ b/polly/test/CodeGen/MemAccess/invariant_base_ptr___%loop---%exit.jscop
@@ -8,7 +8,7 @@
       {
          "name" : "MemRef_X",
          "sizes" : [ "*" ],
-         "type" : "float*"
+         "type" : "ptr"
       },
       {
          "name" : "MemRef_Base",
diff --git a/polly/test/CodeGen/MemAccess/map_scalar_access.ll b/polly/test/CodeGen/MemAccess/map_scalar_access.ll
index 4c9865797a9f5..7c845d4a004f4 100644
--- a/polly/test/CodeGen/MemAccess/map_scalar_access.ll
+++ b/polly/test/CodeGen/MemAccess/map_scalar_access.ll
@@ -1,7 +1,7 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-stmt-granularity=bb -polly-import-jscop-postfix=transformed -polly-print-import-jscop -disable-output < %s | FileCheck %s
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-stmt-granularity=bb -polly-import-jscop-postfix=transformed -polly-import-jscop -polly-codegen -S < %s | FileCheck %s --check-prefix=CODEGEN
+; RUN: opt %loadPolly -polly-stmt-granularity=bb -polly-import-jscop-postfix=transformed -polly-print-import-jscop -disable-output < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-stmt-granularity=bb -polly-import-jscop-postfix=transformed -polly-import-jscop -polly-codegen -S < %s | FileCheck %s --check-prefix=CODEGEN
 
-define void @map_scalar_access(double* noalias nonnull %A) {
+define void @map_scalar_access(ptr noalias nonnull %A) {
 entry:
   br label %outer.for
 
@@ -26,8 +26,8 @@ outer.for:
       br label %reduction.for
 
     reduction.exit:
-      %A_idx = getelementptr inbounds double, double* %A, i32 %j
-      store double %phi, double* %A_idx
+      %A_idx = getelementptr inbounds double, ptr %A, i32 %j
+      store double %phi, ptr %A_idx
       br label %outer.inc
 
 
@@ -115,35 +115,35 @@ return:
 ; CHECK-NEXT: New access function '{ Stmt_reduction_exit[i0] -> MemRef_A[i0] }' detected in JSCOP file
 
 ; CODEGEN:      polly.stmt.outer.for:
-; CODEGEN-NEXT:   %polly.access.A[[R0:[0-9]*]] = getelementptr double, double* %A, i64 %polly.indvar
-; CODEGEN-NEXT:   store double 0.000000e+00, double* %polly.access.A[[R0]]
+; CODEGEN-NEXT:   %polly.access.A[[R0:[0-9]*]] = getelementptr double, ptr %A, i64 %polly.indvar
+; CODEGEN-NEXT:   store double 0.000000e+00, ptr %polly.access.A[[R0]]
 ; CODEGEN-NEXT:   br label %polly.cond
 
 ; CODEGEN:      polly.stmt.reduction.exit:
-; CODEGEN-NEXT:   %polly.access.A[[R1:[0-9]*]] = getelementptr double, double* %A, i64 0
-; CODEGEN-NEXT:   %polly.access.A[[R1]].reload = load double, double* %polly.access.A[[R1]]
-; CODEGEN-NEXT:   %polly.access.A[[R2:[0-9]*]] = getelementptr double, double* %A, i64 0
-; CODEGEN-NEXT:   store double %polly.access.A[[R1]].reload, double* %polly.access.A[[R2]]
+; CODEGEN-NEXT:   %polly.access.A[[R1:[0-9]*]] = getelementptr double, ptr %A, i64 0
+; CODEGEN-NEXT:   %polly.access.A[[R1]].reload = load double, ptr %polly.access.A[[R1]]
+; CODEGEN-NEXT:   %polly.access.A[[R2:[0-9]*]] = getelementptr double, ptr %A, i64 0
+; CODEGEN-NEXT:   store double %polly.access.A[[R1]].reload, ptr %polly.access.A[[R2]]
 ; CODEGEN-NEXT:   br label %polly.merge
 
 ; CODEGEN:      polly.stmt.reduction.for:
-; CODEGEN-NEXT:   %polly.access.A[[R3:[0-9]*]] = getelementptr double, double* %A, i64 0
-; CODEGEN-NEXT:   %polly.access.A[[R3]].reload = load double, double* %polly.access.A[[R3]]
-; CODEGEN-NEXT:   %polly.access.A[[R4:[0-9]*]] = getelementptr double, double* %A, i64 0
-; CODEGEN-NEXT:   store double %polly.access.A[[R3]].reload, double* %polly.access.A[[R4]]
+; CODEGEN-NEXT:   %polly.access.A[[R3:[0-9]*]] = getelementptr double, ptr %A, i64 0
+; CODEGEN-NEXT:   %polly.access.A[[R3]].reload = load double, ptr %polly.access.A[[R3]]
+; CODEGEN-NEXT:   %polly.access.A[[R4:[0-9]*]] = getelementptr double, ptr %A, i64 0
+; CODEGEN-NEXT:   store double %polly.access.A[[R3]].reload, ptr %polly.access.A[[R4]]
 ; CODEGEN-NEXT:   br label %polly.cond9
 
 ; CODEGEN:      polly.stmt.body:
-; CODEGEN-NEXT:   %polly.access.A[[R5:[0-9]*]] = getelementptr double, double* %A, i64 0
-; CODEGEN-NEXT:   %polly.access.A[[R5]].reload = load double, double* %polly.access.A[[R5]]
+; CODEGEN-NEXT:   %polly.access.A[[R5:[0-9]*]] = getelementptr double, ptr %A, i64 0
+; CODEGEN-NEXT:   %polly.access.A[[R5]].reload = load double, ptr %polly.access.A[[R5]]
 ; CODEGEN-NEXT:   %p_add = fadd double %polly.access.A13.reload, 4.200000e+00
-; CODEGEN-NEXT:   %polly.access.A[[R6:[0-9]*]] = getelementptr double, double* %A, i64 0
-; CODEGEN-NEXT:   store double %p_add, double* %polly.access.A[[R6]]
+; CODEGEN-NEXT:   %polly.access.A[[R6:[0-9]*]] = getelementptr double, ptr %A, i64 0
+; CODEGEN-NEXT:   store double %p_add, ptr %polly.access.A[[R6]]
 ; CODEGEN-NEXT:   br label %polly.stmt.reduction.inc
 
 ; CODEGEN:      polly.stmt.reduction.inc:
-; CODEGEN-NEXT:   %polly.access.A[[R7:[0-9]*]] = getelementptr double, double* %A, i64 0
-; CODEGEN-NEXT:   %polly.access.A[[R7]].reload = load double, double* %polly.access.A[[R7]]
-; CODEGEN-NEXT:   %polly.access.A[[R8:[0-9]*]] = getelementptr double, double* %A, i64 0
-; CODEGEN-NEXT:   store double %polly.access.A[[R7]].reload, double* %polly.access.A[[R8]]
+; CODEGEN-NEXT:   %polly.access.A[[R7:[0-9]*]] = getelementptr double, ptr %A, i64 0
+; CODEGEN-NEXT:   %polly.access.A[[R7]].reload = load double, ptr %polly.access.A[[R7]]
+; CODEGEN-NEXT:   %polly.access.A[[R8:[0-9]*]] = getelementptr double, ptr %A, i64 0
+; CODEGEN-NEXT:   store double %polly.access.A[[R7]].reload, ptr %polly.access.A[[R8]]
 ; CODEGEN-NEXT:   br label %polly.merge10
diff --git a/polly/test/CodeGen/MemAccess/multiple_types.ll b/polly/test/CodeGen/MemAccess/multiple_types.ll
index 36aa94ede5d41..1793bd30fc5bf 100644
--- a/polly/test/CodeGen/MemAccess/multiple_types.ll
+++ b/polly/test/CodeGen/MemAccess/multiple_types.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-stmt-granularity=bb -polly-import-jscop \
+; RUN: opt %loadPolly -polly-stmt-granularity=bb -polly-import-jscop \
 ; RUN: -polly-allow-differing-element-types \
 ; RUN:   -polly-codegen -S    < %s | FileCheck %s
 ;
@@ -12,24 +12,21 @@
 ;    }
 
 ; Short[0]
-; CHECK: %polly.access.Short10 = getelementptr i8, i8* %Short, i64 0
-; CHECK: %24 = bitcast i8* %polly.access.Short10 to i16*
-; CHECK: %tmp5_p_scalar_ = load i16, i16* %24
+; CHECK: %polly.access.Short10 = getelementptr i8, ptr %Short, i64 0
+; CHECK: %tmp5_p_scalar_ = load i16, ptr %polly.access.Short10
 
 ; Float[8 * i]
-; CHECK: %25 = mul nsw i64 8, %polly.indvar
-; CHECK: %polly.access.Float11 = getelementptr i8, i8* %Float, i64 %25
-; CHECK: %26 = bitcast i8* %polly.access.Float11 to float*
-; CHECK: %tmp11_p_scalar_ = load float, float* %26
+; CHECK: %24 = mul nsw i64 8, %polly.indvar
+; CHECK: %polly.access.Float11 = getelementptr i8, ptr %Float, i64 %24
+; CHECK: %tmp11_p_scalar_ = load float, ptr %polly.access.Float11
 
 ; Double[8]
-; CHECK: %polly.access.Double13 = getelementptr i8, i8* %Double, i64 8
-; CHECK: %27 = bitcast i8* %polly.access.Double13 to double*
-; CHECK: %tmp17_p_scalar_ = load double, double* %27
+; CHECK: %polly.access.Double13 = getelementptr i8, ptr %Double, i64 8
+; CHECK: %tmp17_p_scalar_ = load double, ptr %polly.access.Double13
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @multiple_types(i8* %Short, i8* %Float, i8* %Double) {
+define void @multiple_types(ptr %Short, ptr %Float, ptr %Double) {
 bb:
   br label %bb1
 
@@ -40,26 +37,23 @@ bb1:                                              ; preds = %bb20, %bb
 
 bb2:                                              ; preds = %bb1
   %tmp = shl nsw i64 %i.0, 1
-  %tmp3 = getelementptr inbounds i8, i8* %Short, i64 %tmp
-  %tmp4 = bitcast i8* %tmp3 to i16*
-  %tmp5 = load i16, i16* %tmp4, align 2
+  %tmp3 = getelementptr inbounds i8, ptr %Short, i64 %tmp
+  %tmp5 = load i16, ptr %tmp3, align 2
   %tmp6 = trunc i16 %tmp5 to i8
-  %tmp7 = getelementptr inbounds i8, i8* %Short, i64 %i.0
-  store i8 %tmp6, i8* %tmp7, align 1
+  %tmp7 = getelementptr inbounds i8, ptr %Short, i64 %i.0
+  store i8 %tmp6, ptr %tmp7, align 1
   %tmp8 = shl nsw i64 %i.0, 2
-  %tmp9 = getelementptr inbounds i8, i8* %Float, i64 %tmp8
-  %tmp10 = bitcast i8* %tmp9 to float*
-  %tmp11 = load float, float* %tmp10, align 4
+  %tmp9 = getelementptr inbounds i8, ptr %Float, i64 %tmp8
+  %tmp11 = load float, ptr %tmp9, align 4
   %tmp12 = fptosi float %tmp11 to i8
-  %tmp13 = getelementptr inbounds i8, i8* %Float, i64 %i.0
-  store i8 %tmp12, i8* %tmp13, align 1
+  %tmp13 = getelementptr inbounds i8, ptr %Float, i64 %i.0
+  store i8 %tmp12, ptr %tmp13, align 1
   %tmp14 = shl nsw i64 %i.0, 3
-  %tmp15 = getelementptr inbounds i8, i8* %Double, i64 %tmp14
-  %tmp16 = bitcast i8* %tmp15 to double*
-  %tmp17 = load double, double* %tmp16, align 8
+  %tmp15 = getelementptr inbounds i8, ptr %Double, i64 %tmp14
+  %tmp17 = load double, ptr %tmp15, align 8
   %tmp18 = fptosi double %tmp17 to i8
-  %tmp19 = getelementptr inbounds i8, i8* %Double, i64 %i.0
-  store i8 %tmp18, i8* %tmp19, align 1
+  %tmp19 = getelementptr inbounds i8, ptr %Double, i64 %i.0
+  store i8 %tmp18, ptr %tmp19, align 1
   br label %bb20
 
 bb20:                                             ; preds = %bb2
diff --git a/polly/test/CodeGen/MemAccess/simple_analyze.ll b/polly/test/CodeGen/MemAccess/simple_analyze.ll
index 00c90822d6565..143651b565aff 100644
--- a/polly/test/CodeGen/MemAccess/simple_analyze.ll
+++ b/polly/test/CodeGen/MemAccess/simple_analyze.ll
@@ -1,5 +1,4 @@
 ;RUN: opt %loadPolly -polly-print-import-jscop -polly-import-jscop-postfix=transformed -disable-output < %s | FileCheck %s
-;RUN: opt %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -polly-vectorizer=polly -S < %s | FileCheck %s --check-prefix=JSCOPVEC
 target datalayout = "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:32:64-v64:64:64-v128:128:128-a0:0:64-f80:32:32-n8:16:32"
 
 @A = common global [100 x i32] zeroinitializer, align 4
@@ -46,31 +45,3 @@ for.end14:                                        ; preds = %for.cond4
 }
 ; CHECK-DAG: New access function '{ Stmt_for_body7[i0] -> MemRef_B[0] }' detected in JSCOP file
 ; CHECK-DAG: New access function '{ Stmt_for_body[i0] -> MemRef_A[0] }' detected in JSCOP file
-
-; Verify that the new access function (see above) is actually used during vector code generation.
-
-; JSCOPVEC:  store i32 0, ptr @B
-; JSCOPVEC:  store i32 1, ptr @B
-; JSCOPVEC:  store i32 2, ptr @B
-; JSCOPVEC:  store i32 3, ptr @B
-; JSCOPVEC:  store i32 4, ptr @B
-; JSCOPVEC:  store i32 5, ptr @B
-; JSCOPVEC:  store i32 6, ptr @B
-; JSCOPVEC:  store i32 7, ptr @B
-; JSCOPVEC:  store i32 8, ptr @B
-; JSCOPVEC:  store i32 9, ptr @B
-; JSCOPVEC:  store i32 10, ptr @B
-; JSCOPVEC:  store i32 11, ptr @B
-
-; JSCOPVEC:  store i32 0, ptr @A
-; JSCOPVEC:  store i32 1, ptr @A
-; JSCOPVEC:  store i32 2, ptr @A
-; JSCOPVEC:  store i32 3, ptr @A
-; JSCOPVEC:  store i32 4, ptr @A
-; JSCOPVEC:  store i32 5, ptr @A
-; JSCOPVEC:  store i32 6, ptr @A
-; JSCOPVEC:  store i32 7, ptr @A
-; JSCOPVEC:  store i32 8, ptr @A
-; JSCOPVEC:  store i32 9, ptr @A
-; JSCOPVEC:  store i32 10, ptr @A
-; JSCOPVEC:  store i32 11, ptr @A
diff --git a/polly/test/CodeGen/MemAccess/simple_stride_test.ll b/polly/test/CodeGen/MemAccess/simple_stride_test.ll
deleted file mode 100644
index 8ba72a59bc42c..0000000000000
--- a/polly/test/CodeGen/MemAccess/simple_stride_test.ll
+++ /dev/null
@@ -1,47 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -basic-aa -polly-import-jscop -polly-codegen -polly-vectorizer=polly -S < %s | FileCheck %s
-;
-; Check that we use the correct __new__ strides:
-;    stride zero for B
-;    stride one for A
-;
-; CHECK:  %polly.access.B = getelementptr i32, i32* %B, i64 0
-; CHECK:  %[[BC:[._a-zA-Z0-9]*]] = bitcast i32* %polly.access.B to <1 x i32>*
-; CHECK:  %[[LD:[._a-zA-Z0-9]*]] = load <1 x i32>, <1 x i32>* %[[BC]], align 8
-; CHECK:  %[[SV:[._a-zA-Z0-9]*]] = shufflevector <1 x i32> %[[LD]], <1 x i32> %[[LD]], <16 x i32> zeroinitializer
-;
-; CHECK:  %polly.access.A = getelementptr i32, i32* %A, i64 0
-; CHECK:  %[[VP:[._a-zA-Z0-9]*]] = bitcast i32* %polly.access.A to <16 x i32>*
-; CHECK:  store <16 x i32> %[[SV]], <16 x i32>* %[[VP]], align 8
-;
-;    void simple_stride(int *restrict A, int *restrict B) {
-;      for (int i = 0; i < 16; i++)
-;        A[i * 2] = B[i * 2];
-;    }
-;
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @simple_stride(i32* noalias %A, i32* noalias %B) {
-entry:
-  br label %for.cond
-
-for.cond:                                         ; preds = %for.inc, %entry
-  %indvars.iv = phi i64 [ %indvars.iv.next, %for.inc ], [ 0, %entry ]
-  %exitcond = icmp ne i64 %indvars.iv, 16
-  br i1 %exitcond, label %for.body, label %for.end
-
-for.body:                                         ; preds = %for.cond
-  %tmp = shl nsw i64 %indvars.iv, 1
-  %arrayidx = getelementptr inbounds i32, i32* %B, i64 %tmp
-  %tmp4 = load i32, i32* %arrayidx, align 4
-  %tmp5 = shl nsw i64 %indvars.iv, 1
-  %arrayidx3 = getelementptr inbounds i32, i32* %A, i64 %tmp5
-  store i32 %tmp4, i32* %arrayidx3, align 4
-  br label %for.inc
-
-for.inc:                                          ; preds = %for.body
-  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
-  br label %for.cond
-
-for.end:                                          ; preds = %for.cond
-  ret void
-}
diff --git a/polly/test/CodeGen/OpenMP/alias-metadata.ll b/polly/test/CodeGen/OpenMP/alias-metadata.ll
index 4b378c39727e7..07d79631b2cb3 100644
--- a/polly/test/CodeGen/OpenMP/alias-metadata.ll
+++ b/polly/test/CodeGen/OpenMP/alias-metadata.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -polly-parallel -S < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-codegen -polly-parallel -S < %s | FileCheck %s
 ;
 ;    void foo(float *A, float *B) {
 ;      for (long i = 0; i < 1000; i++)
@@ -10,7 +10,7 @@ target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
 ; CHECK: define internal void @foo_polly_subfn
 
-define void @foo(float* %A, float* %B) {
+define void @foo(ptr %A, ptr %B) {
 bb:
   br label %bb2
 
@@ -28,12 +28,12 @@ bb4:                                              ; preds = %bb8, %bb3
   br i1 %exitcond, label %bb5, label %bb10
 
 bb5:                                              ; preds = %bb4
-  %tmp = getelementptr inbounds float, float* %B, i64 %i.0
-  %tmp7 = getelementptr inbounds float, float* %A, i64 %i.0
-  %tmp6 = load float, float* %tmp, align 4
-  store float %tmp6, float* %tmp7, align 4
-; CHECK: %tmp6_p_scalar_ = load float, float* %scevgep, align 4, !alias.scope !0, !noalias !3
-; CHECK: store float %tmp6_p_scalar_, float* %scevgep8, align 4, !alias.scope !3, !noalias !0
+  %tmp = getelementptr inbounds float, ptr %B, i64 %i.0
+  %tmp7 = getelementptr inbounds float, ptr %A, i64 %i.0
+  %tmp6 = load float, ptr %tmp, align 4
+  store float %tmp6, ptr %tmp7, align 4
+; CHECK: %tmp6_p_scalar_ = load float, ptr %scevgep, align 4, !alias.scope !0, !noalias !3
+; CHECK: store float %tmp6_p_scalar_, ptr %scevgep7, align 4, !alias.scope !3, !noalias !0
   br label %bb8
 
 bb8:                                              ; preds = %bb5
diff --git a/polly/test/CodeGen/OpenMP/loop-body-references-outer-values.ll b/polly/test/CodeGen/OpenMP/loop-body-references-outer-values.ll
index a4e1570e7c1a6..f9612d77533d5 100644
--- a/polly/test/CodeGen/OpenMP/loop-body-references-outer-values.ll
+++ b/polly/test/CodeGen/OpenMP/loop-body-references-outer-values.ll
@@ -1,5 +1,5 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-parallel -polly-parallel-force -polly-print-ast -disable-output < %s | FileCheck %s -check-prefix=AST
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-parallel -polly-parallel-force -polly-codegen -S < %s | FileCheck %s -check-prefix=IR
+; RUN: opt %loadPolly -polly-parallel -polly-parallel-force -polly-print-ast -disable-output < %s | FileCheck %s -check-prefix=AST
+; RUN: opt %loadPolly -polly-parallel -polly-parallel-force -polly-codegen -S < %s | FileCheck %s -check-prefix=IR
 
 ; Make sure we correctly forward the reference to 'A' to the OpenMP subfunction.
 ;
@@ -15,13 +15,12 @@
 ; AST:   Stmt_for_body(c0);
 
 ; IR-LABEL: polly.parallel.for:
-; IR-NEXT:  %polly.subfn.storeaddr.A = getelementptr inbounds { float* }, { float* }* %polly.par.userContext, i32 0, i32 0
-; IR-NEXT:  store float* %A, float** %polly.subfn.storeaddr.A
-; IR-NEXT:  %polly.par.userContext1 = bitcast { float* }* %polly.par.userContext to i8*
+; IR-NEXT:  %polly.subfn.storeaddr.A = getelementptr inbounds { ptr }, ptr %polly.par.userContext, i32 0, i32 0
+; IR-NEXT:  store ptr %A, ptr %polly.subfn.storeaddr.A
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @loop_references_outer_ids(float* %A) {
+define void @loop_references_outer_ids(ptr %A) {
 entry:
   br label %for.cond
 
@@ -32,8 +31,8 @@ for.cond:                                         ; preds = %for.inc, %entry
 
 for.body:                                         ; preds = %for.cond
   %conv = sitofp i64 %i.0 to float
-  %arrayidx = getelementptr inbounds float, float* %A, i64 %i.0
-  store float %conv, float* %arrayidx, align 4
+  %arrayidx = getelementptr inbounds float, ptr %A, i64 %i.0
+  store float %conv, ptr %arrayidx, align 4
   br label %for.inc
 
 for.inc:                                          ; preds = %for.body
diff --git a/polly/test/CodeGen/OpenMP/loop-bounds-reference-outer-ids.ll b/polly/test/CodeGen/OpenMP/loop-bounds-reference-outer-ids.ll
index edf9bbcecc4a2..da9da18c89b21 100644
--- a/polly/test/CodeGen/OpenMP/loop-bounds-reference-outer-ids.ll
+++ b/polly/test/CodeGen/OpenMP/loop-bounds-reference-outer-ids.ll
@@ -1,5 +1,5 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-parallel -polly-print-ast -disable-output < %s | FileCheck %s -check-prefix=AST
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-parallel -polly-codegen -S < %s | FileCheck %s -check-prefix=IR
+; RUN: opt %loadPolly -polly-parallel -polly-print-ast -disable-output < %s | FileCheck %s -check-prefix=AST
+; RUN: opt %loadPolly -polly-parallel -polly-codegen -S < %s | FileCheck %s -check-prefix=IR
 ;
 ; float A[100];
 ;
@@ -23,18 +23,16 @@
 ; AST:       Stmt_for_body6(c0, c1, c2);
 
 ; IR:      %polly.par.userContext = alloca { i64, i64 }
-; IR: %[[R1:[0-9a-z.]+]] = getelementptr inbounds { i64, i64 }, { i64, i64 }* %polly.par.userContext, i32 0, i32 0
-; IR-NEXT: store i64 %n, i64* %[[R1]]
-; IR-NEXT: %[[R2:[0-9a-z.]+]] = getelementptr inbounds { i64, i64 }, { i64, i64 }* %polly.par.userContext, i32 0, i32 1
-; IR-NEXT: store i64 %polly.indvar, i64* %[[R2]]
-; IR-NEXT: %polly.par.userContext1 = bitcast { i64, i64 }* %polly.par.userContext to i8*
-
-; IR-LABEL: @loop_references_outer_ids_polly_subfn(i8* %polly.par.userContext)
-; IR:       %polly.par.userContext1 = bitcast i8* %polly.par.userContext to { i64, i64 }*
-; IR-NEXT:  %[[R3:[0-9a-z.]+]] = getelementptr inbounds { i64, i64 }, { i64, i64 }* %polly.par.userContext1, i32 0, i32 0
-; IR-NEXT:  %[[R4:[0-9a-z.]+]] = load i64, i64* %[[R3]]
-; IR-NEXT:  %[[R5:[0-9a-z.]+]] = getelementptr inbounds { i64, i64 }, { i64, i64 }* %polly.par.userContext1, i32 0, i32 1
-; IR-NEXT:  %[[R6:[0-9a-z.]+]] = load i64, i64* %[[R5]]
+; IR: %[[R1:[0-9a-z.]+]] = getelementptr inbounds { i64, i64 }, ptr %polly.par.userContext, i32 0, i32 0
+; IR-NEXT: store i64 %n, ptr %[[R1]]
+; IR-NEXT: %[[R2:[0-9a-z.]+]] = getelementptr inbounds { i64, i64 }, ptr %polly.par.userContext, i32 0, i32 1
+; IR-NEXT: store i64 %polly.indvar, ptr %[[R2]]
+
+; IR-LABEL: @loop_references_outer_ids_polly_subfn(ptr %polly.par.userContext)
+; IR:  %[[R3:[0-9a-z.]+]] = getelementptr inbounds { i64, i64 }, ptr %polly.par.userContext, i32 0, i32 0
+; IR-NEXT:  %[[R4:[0-9a-z.]+]] = load i64, ptr %[[R3]]
+; IR-NEXT:  %[[R5:[0-9a-z.]+]] = getelementptr inbounds { i64, i64 }, ptr %polly.par.userContext, i32 0, i32 1
+; IR-NEXT:  %[[R6:[0-9a-z.]+]] = load i64, ptr %[[R5]]
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
@@ -70,10 +68,10 @@ for.body6:                                        ; preds = %for.cond4
   %add7 = add nsw i64 %i.0, %j.0
   %add8 = add nsw i64 %add7, %k.0
   %conv = sitofp i64 %add8 to float
-  %arrayidx = getelementptr inbounds [100 x float], [100 x float]* @A, i64 0, i64 %j.0
-  %tmp = load float, float* %arrayidx, align 4
+  %arrayidx = getelementptr inbounds [100 x float], ptr @A, i64 0, i64 %j.0
+  %tmp = load float, ptr %arrayidx, align 4
   %add9 = fadd float %tmp, %conv
-  store float %add9, float* %arrayidx, align 4
+  store float %add9, ptr %arrayidx, align 4
   br label %for.inc
 
 for.inc:                                          ; preds = %for.body6
diff --git a/polly/test/CodeGen/OpenMP/mapped-phi-access.ll b/polly/test/CodeGen/OpenMP/mapped-phi-access.ll
index f64391fabea08..1b8433693abfe 100644
--- a/polly/test/CodeGen/OpenMP/mapped-phi-access.ll
+++ b/polly/test/CodeGen/OpenMP/mapped-phi-access.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-stmt-granularity=bb -polly-parallel -polly-delicm -polly-codegen -S < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-stmt-granularity=bb -polly-parallel -polly-delicm -polly-codegen -S < %s | FileCheck %s
 ;
 ; Verify that -polly-parallel can handle mapped scalar MemoryAccesses.
 ;
@@ -7,23 +7,21 @@ target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 ; Function Attrs: nounwind uwtable
 define void @main() local_unnamed_addr #0 {
 entry:
-  %0 = load i8*, i8** undef, align 8, !tbaa !1
-  %1 = load i8*, i8** undef, align 8, !tbaa !1
-  %arraydecay16 = bitcast i8* %1 to double*
-  %arraydecay20 = bitcast i8* %0 to [4000 x double]*
+  %0 = load ptr, ptr undef, align 8, !tbaa !1
+  %1 = load ptr, ptr undef, align 8, !tbaa !1
   br label %for.body65.i226
 
 for.body65.i226:                                  ; preds = %for.inc85.i238, %entry
   %indvars.iv8.i223 = phi i64 [ 0, %entry ], [ %indvars.iv.next9.i236, %for.inc85.i238 ]
-  %arrayidx70.i224 = getelementptr inbounds double, double* %arraydecay16, i64 %indvars.iv8.i223
+  %arrayidx70.i224 = getelementptr inbounds double, ptr %1, i64 %indvars.iv8.i223
   br label %for.body68.i235
 
 for.body68.i235:                                  ; preds = %for.body68.i235, %for.body65.i226
   %2 = phi double [ undef, %for.body65.i226 ], [ undef, %for.body68.i235 ]
   %indvars.iv.i227 = phi i64 [ 0, %for.body65.i226 ], [ %indvars.iv.next.i233, %for.body68.i235 ]
-  %arrayidx74.i228 = getelementptr inbounds [4000 x double], [4000 x double]* %arraydecay20, i64 %indvars.iv8.i223, i64 %indvars.iv.i227
-  %3 = load double, double* %arrayidx74.i228, align 8, !tbaa !5
-  store double undef, double* %arrayidx70.i224, align 8, !tbaa !5
+  %arrayidx74.i228 = getelementptr inbounds [4000 x double], ptr %0, i64 %indvars.iv8.i223, i64 %indvars.iv.i227
+  %3 = load double, ptr %arrayidx74.i228, align 8, !tbaa !5
+  store double undef, ptr %arrayidx70.i224, align 8, !tbaa !5
   %indvars.iv.next.i233 = add nuw nsw i64 %indvars.iv.i227, 1
   %exitcond.i234 = icmp eq i64 %indvars.iv.next.i233, 4000
   br i1 %exitcond.i234, label %for.inc85.i238, label %for.body68.i235
@@ -50,9 +48,8 @@ attributes #0 = { nounwind uwtable "correctly-rounded-divide-sqrt-fp-math"="fals
 !6 = !{!"double", !3, i64 0}
 
 
-; CHECK-LABEL: define internal void @main_polly_subfn(i8* %polly.par.userContext)
+; CHECK-LABEL: define internal void @main_polly_subfn(ptr %polly.par.userContext)
 ;
 ; CHECK:       polly.stmt.for.body65.i226:
-; CHECK-NEXT:    %polly.access.cast.polly.subfunc.arg.[[R0:[0-9]*]] = bitcast i8* %polly.subfunc.arg.{{[0-9]*}} to double*
-; CHECK-NEXT:    %polly.access.polly.subfunc.arg.[[R1:[0-9]*]] = getelementptr double, double* %polly.access.cast.polly.subfunc.arg.[[R0]], i64 %polly.indvar
-; CHECK-NEXT:    store double undef, double* %polly.access.polly.subfunc.arg.[[R1]]
+; CHECK-NEXT:    %polly.access.polly.subfunc.arg.[[R1:[0-9]*]] = getelementptr double, ptr %polly.subfunc.arg.[[R0:[0-9]*]], i64 %polly.indvar
+; CHECK-NEXT:    store double undef, ptr %polly.access.polly.subfunc.arg.[[R1]]
diff --git a/polly/test/CodeGen/OpenMP/new_multidim_access.ll b/polly/test/CodeGen/OpenMP/new_multidim_access.ll
index 77f7aaac35ac3..1b2912ba9ef46 100644
--- a/polly/test/CodeGen/OpenMP/new_multidim_access.ll
+++ b/polly/test/CodeGen/OpenMP/new_multidim_access.ll
@@ -1,7 +1,7 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-print-import-jscop \
+; RUN: opt %loadPolly -polly-print-import-jscop \
 ; RUN:                -disable-output < %s | FileCheck %s
 
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-import-jscop \
+; RUN: opt %loadPolly -polly-import-jscop \
 ; RUN:                -polly-codegen -S < %s \
 ; RUN:                -polly-parallel \
 ; RUN:                | FileCheck %s -check-prefix=IR
@@ -20,19 +20,19 @@
 ; CHECK:    new: [n, m] -> { Stmt_bb4[i0, i1] -> MemRef_A[i0, 43 + i1] };
 
 ; IR: %polly.access.mul.polly.subfunc.arg.A = mul nsw i64 %polly.indvar, %polly.subfunc.arg.m
-; IR: %6 = add nsw i64 %polly.indvar5, 13
+; IR: %6 = add nsw i64 %polly.indvar4, 13
 ; IR: %polly.access.add.polly.subfunc.arg.A = add nsw i64 %polly.access.mul.polly.subfunc.arg.A, %6
-; IR: %polly.access.polly.subfunc.arg.A = getelementptr float, float* %polly.subfunc.arg.A, i64 %polly.access.add.polly.subfunc.arg.A
-; IR: %tmp10_p_scalar_ = load float, float* %polly.access.polly.subfunc.arg.A, align 4, !alias.scope !0, !noalias !3, !llvm.access.group !4
+; IR: %polly.access.polly.subfunc.arg.A = getelementptr float, ptr %polly.subfunc.arg.A, i64 %polly.access.add.polly.subfunc.arg.A
+; IR: %tmp10_p_scalar_ = load float, ptr %polly.access.polly.subfunc.arg.A, align 4, !alias.scope !0, !noalias !3, !llvm.access.group !4
 
-; IR: %polly.access.mul.polly.subfunc.arg.A8 = mul nsw i64 %polly.indvar, %polly.subfunc.arg.m
-; IR: %7 = add nsw i64 %polly.indvar5, 43
-; IR: %polly.access.add.polly.subfunc.arg.A9 = add nsw i64 %polly.access.mul.polly.subfunc.arg.A8, %7
-; IR: %polly.access.polly.subfunc.arg.A10 = getelementptr float, float* %polly.subfunc.arg.A, i64 %polly.access.add.polly.subfunc.arg.A9
-; IR: store float %p_tmp11, float* %polly.access.polly.subfunc.arg.A10, align 4, !alias.scope !0, !noalias !3, !llvm.access.group !4
+; IR: %polly.access.mul.polly.subfunc.arg.A7 = mul nsw i64 %polly.indvar, %polly.subfunc.arg.m
+; IR: %7 = add nsw i64 %polly.indvar4, 43
+; IR: %polly.access.add.polly.subfunc.arg.A8 = add nsw i64 %polly.access.mul.polly.subfunc.arg.A7, %7
+; IR: %polly.access.polly.subfunc.arg.A9 = getelementptr float, ptr %polly.subfunc.arg.A, i64 %polly.access.add.polly.subfunc.arg.A8
+; IR: store float %p_tmp11, ptr %polly.access.polly.subfunc.arg.A9, align 4, !alias.scope !0, !noalias !3, !llvm.access.group !4
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @new_multidim_access(i64 %n, i64 %m, float* %A) {
+define void @new_multidim_access(i64 %n, i64 %m, ptr %A) {
 bb:
   br label %bb1
 
@@ -55,10 +55,10 @@ bb4:                                              ; preds = %bb3
   %tmp7 = shl nsw i64 %j.0, 1
   %tmp8 = mul nsw i64 %i.0, %m
   %.sum = add i64 %tmp8, %tmp7
-  %tmp9 = getelementptr inbounds float, float* %A, i64 %.sum
-  %tmp10 = load float, float* %tmp9, align 4
+  %tmp9 = getelementptr inbounds float, ptr %A, i64 %.sum
+  %tmp10 = load float, ptr %tmp9, align 4
   %tmp11 = fadd float %tmp10, %tmp6
-  store float %tmp11, float* %tmp9, align 4
+  store float %tmp11, ptr %tmp9, align 4
   br label %bb12
 
 bb12:                                             ; preds = %bb4
diff --git a/polly/test/CodeGen/OpenMP/recomputed-srem.ll b/polly/test/CodeGen/OpenMP/recomputed-srem.ll
index f09da47d37156..cfae8e943cf14 100644
--- a/polly/test/CodeGen/OpenMP/recomputed-srem.ll
+++ b/polly/test/CodeGen/OpenMP/recomputed-srem.ll
@@ -1,15 +1,15 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-stmt-granularity=bb -polly-codegen -polly-parallel \
+; RUN: opt %loadPolly -polly-stmt-granularity=bb -polly-codegen -polly-parallel \
 ; RUN: -polly-parallel-force -S < %s | FileCheck %s
 ;
 ; Test to verify that we pass %rem96 to the parallel subfunction.
 ;
-; CHECK:       %[[R:[0-9]*]] = getelementptr inbounds { i32, i32, i64, float*, float*, i32 }, { i32, i32, i64, float*, float*, i32 }* %polly.par.userContext1, i32 0, i32 5
-; CHECK-NEXT:  %polly.subfunc.arg.rem96 = load i32, i32* %[[R]]
+; CHECK:       %[[R:[0-9]*]] = getelementptr inbounds { i32, i32, i64, ptr, ptr, i32 }, ptr %polly.par.userContext, i32 0, i32 5
+; CHECK-NEXT:  %polly.subfunc.arg.rem96 = load i32, ptr %[[R]]
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
 ; Function Attrs: nounwind uwtable
-define void @dmxpy(i32 %n1, float* %y, i32 %n2, float* %x) #0 {
+define void @dmxpy(i32 %n1, ptr %y, i32 %n2, ptr %x) #0 {
 entry:
   %rem96 = srem i32 %n2, 16
   %0 = sext i32 %rem96 to i64
@@ -23,11 +23,11 @@ for.cond195.preheader:                            ; preds = %for.inc363, %entry
 
 for.body197:                                      ; preds = %for.body197, %for.cond195.preheader
   %indvars.iv = phi i64 [ %indvars.iv.next, %for.body197 ], [ 0, %for.cond195.preheader ]
-  %arrayidx199 = getelementptr inbounds float, float* %y, i64 %indvars.iv
+  %arrayidx199 = getelementptr inbounds float, ptr %y, i64 %indvars.iv
   %2 = add nsw i64 %indvars.iv262, -6
-  %arrayidx292 = getelementptr inbounds float, float* %x, i64 %2
-  %3 = load float, float* %arrayidx292, align 4
-  store float undef, float* %arrayidx199, align 4
+  %arrayidx292 = getelementptr inbounds float, ptr %x, i64 %2
+  %3 = load float, ptr %arrayidx292, align 4
+  store float undef, ptr %arrayidx199, align 4
   %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
   %lftr.wideiv = trunc i64 %indvars.iv.next to i32
   %exitcond = icmp ne i32 %lftr.wideiv, %n1
diff --git a/polly/test/CodeGen/OpenMP/scev-rewriting.ll b/polly/test/CodeGen/OpenMP/scev-rewriting.ll
index 0f6ca904e2592..1b229fc19d259 100644
--- a/polly/test/CodeGen/OpenMP/scev-rewriting.ll
+++ b/polly/test/CodeGen/OpenMP/scev-rewriting.ll
@@ -1,4 +1,4 @@
-; RUN: opt %loadPolly < %s -polly-vectorizer=polly -polly-parallel -polly-parallel-force -polly-process-unprofitable -polly-codegen -S | FileCheck %s
+; RUN: opt %loadPolly < %s -polly-vectorizer=stripmine -polly-parallel -polly-parallel-force -polly-process-unprofitable -polly-codegen -S | FileCheck %s
 ; CHECK: define internal void @DoStringSort_polly_subfn
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
 target triple = "aarch64-unknown-linux-gnueabi"
diff --git a/polly/test/CodeGen/OpenMP/single_loop.ll b/polly/test/CodeGen/OpenMP/single_loop.ll
index b160cf58b3194..f79653a08d211 100644
--- a/polly/test/CodeGen/OpenMP/single_loop.ll
+++ b/polly/test/CodeGen/OpenMP/single_loop.ll
@@ -1,14 +1,14 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-parallel -polly-parallel-force -polly-print-ast -disable-output < %s | FileCheck %s -check-prefix=AST
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-parallel -polly-parallel-force -polly-codegen -S -verify-dom-info < %s | FileCheck %s -check-prefix=IR
+; RUN: opt %loadPolly -polly-parallel -polly-parallel-force -polly-print-ast -disable-output < %s | FileCheck %s -check-prefix=AST
+; RUN: opt %loadPolly -polly-parallel -polly-parallel-force -polly-codegen -S -verify-dom-info < %s | FileCheck %s -check-prefix=IR
 
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-parallel -polly-parallel-force -polly-import-jscop -polly-print-ast -disable-output < %s | FileCheck %s -check-prefix=AST-STRIDE4
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-parallel -polly-parallel-force -polly-import-jscop -polly-codegen -S < %s | FileCheck %s -check-prefix=IR-STRIDE4
+; RUN: opt %loadPolly -polly-parallel -polly-parallel-force -polly-import-jscop -polly-print-ast -disable-output < %s | FileCheck %s -check-prefix=AST-STRIDE4
+; RUN: opt %loadPolly -polly-parallel -polly-parallel-force -polly-import-jscop -polly-codegen -S < %s | FileCheck %s -check-prefix=IR-STRIDE4
 
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-parallel -polly-parallel-force -polly-codegen -polly-omp-backend=LLVM -polly-scheduling=static -polly-scheduling-chunksize=43 -S -verify-dom-info < %s | FileCheck %s -check-prefix=LIBOMP-IR-STATIC-CHUNKED
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-parallel -polly-parallel-force -polly-codegen -polly-omp-backend=LLVM -polly-scheduling=static -S -verify-dom-info < %s | FileCheck %s -check-prefix=LIBOMP-IR-STATIC
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-parallel -polly-parallel-force -polly-codegen -polly-omp-backend=LLVM -polly-scheduling=dynamic -S -verify-dom-info < %s | FileCheck %s -check-prefix=LIBOMP-IR-DYNAMIC
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-parallel -polly-parallel-force -polly-codegen -polly-omp-backend=LLVM -polly-scheduling=dynamic -polly-scheduling-chunksize=4 -S -verify-dom-info < %s | FileCheck %s -check-prefix=LIBOMP-IR-DYNAMIC-FOUR
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-parallel -polly-parallel-force -polly-import-jscop -polly-codegen -polly-omp-backend=LLVM -S < %s | FileCheck %s -check-prefix=LIBOMP-IR-STRIDE4
+; RUN: opt %loadPolly -polly-parallel -polly-parallel-force -polly-codegen -polly-omp-backend=LLVM -polly-scheduling=static -polly-scheduling-chunksize=43 -S -verify-dom-info < %s | FileCheck %s -check-prefix=LIBOMP-IR-STATIC-CHUNKED
+; RUN: opt %loadPolly -polly-parallel -polly-parallel-force -polly-codegen -polly-omp-backend=LLVM -polly-scheduling=static -S -verify-dom-info < %s | FileCheck %s -check-prefix=LIBOMP-IR-STATIC
+; RUN: opt %loadPolly -polly-parallel -polly-parallel-force -polly-codegen -polly-omp-backend=LLVM -polly-scheduling=dynamic -S -verify-dom-info < %s | FileCheck %s -check-prefix=LIBOMP-IR-DYNAMIC
+; RUN: opt %loadPolly -polly-parallel -polly-parallel-force -polly-codegen -polly-omp-backend=LLVM -polly-scheduling=dynamic -polly-scheduling-chunksize=4 -S -verify-dom-info < %s | FileCheck %s -check-prefix=LIBOMP-IR-DYNAMIC-FOUR
+; RUN: opt %loadPolly -polly-parallel -polly-parallel-force -polly-import-jscop -polly-codegen -polly-omp-backend=LLVM -S < %s | FileCheck %s -check-prefix=LIBOMP-IR-STRIDE4
 
 ; This extensive test case tests the creation of the full set of OpenMP calls
 ; as well as the subfunction creation using a trivial loop as example.
@@ -37,17 +37,15 @@
 ; IR-NEXT:   %polly.par.userContext = alloca
 
 ; IR-LABEL: polly.parallel.for:
-; IR-NEXT:   %polly.par.userContext1 = bitcast {}* %polly.par.userContext to i8*
-; IR-NEXT:   call void @GOMP_parallel_loop_runtime_start(void (i8*)* @single_parallel_loop_polly_subfn, i8* %polly.par.userContext1, i32 0, i64 0, i64 1024, i64 1)
-; IR-NEXT:   call void @single_parallel_loop_polly_subfn(i8* %polly.par.userContext1)
+; IR-NEXT:   call void @GOMP_parallel_loop_runtime_start(ptr @single_parallel_loop_polly_subfn, ptr %polly.par.userContext, i32 0, i64 0, i64 1024, i64 1)
+; IR-NEXT:   call void @single_parallel_loop_polly_subfn(ptr %polly.par.userContext)
 ; IR-NEXT:   call void @GOMP_parallel_end()
 ; IR-NEXT:   br label %polly.exiting
 
-; IR: define internal void @single_parallel_loop_polly_subfn(i8* %polly.par.userContext) #1
+; IR: define internal void @single_parallel_loop_polly_subfn(ptr %polly.par.userContext) #1
 ; IR-LABEL: polly.par.setup:
 ; IR-NEXT:   %polly.par.LBPtr = alloca i64
 ; IR-NEXT:   %polly.par.UBPtr = alloca i64
-; IR-NEXT:   %polly.par.userContext1 =
 ; IR:   br label %polly.par.checkNext
 
 ; IR-LABEL: polly.par.exit:
@@ -55,13 +53,13 @@
 ; IR-NEXT:   ret void
 
 ; IR-LABEL: polly.par.checkNext:
-; IR-NEXT:   %[[parnext:[._a-zA-Z0-9]*]] = call i8 @GOMP_loop_runtime_next(i64* %polly.par.LBPtr, i64* %polly.par.UBPtr)
+; IR-NEXT:   %[[parnext:[._a-zA-Z0-9]*]] = call i8 @GOMP_loop_runtime_next(ptr %polly.par.LBPtr, ptr %polly.par.UBPtr)
 ; IR-NEXT:   %[[cmp:[._a-zA-Z0-9]*]] = icmp ne i8 %[[parnext]], 0
 ; IR-NEXT:   br i1 %[[cmp]], label %polly.par.loadIVBounds, label %polly.par.exit
 
 ; IR-LABEL: polly.par.loadIVBounds:
-; IR-NEXT:   %polly.par.LB = load i64, i64* %polly.par.LBPtr
-; IR-NEXT:   %polly.par.UB = load i64, i64* %polly.par.UBPtr
+; IR-NEXT:   %polly.par.LB = load i64, ptr %polly.par.LBPtr
+; IR-NEXT:   %polly.par.UB = load i64, ptr %polly.par.UBPtr
 ; IR-NEXT:   %polly.par.UBAdjusted = sub i64 %polly.par.UB, 1
 ; IR-NEXT:   br label %polly.loop_preheader
 
@@ -73,8 +71,9 @@
 ; IR-NEXT:   br label %polly.stmt.S
 
 ; IR-LABEL: polly.stmt.S:
-; IR-NEXT:   %[[gep:[._a-zA-Z0-9]*]] = getelementptr [1024 x float], [1024 x float]* {{.*}}, i64 0, i64 %polly.indvar
-; IR-NEXT:   store float 1.000000e+00, float* %[[gep]]
+; IR-NEXT:   %[[offset:[0-9]+]] = shl i64 %polly.indvar, 2
+; IR-NEXT:   %[[gep:[._a-zA-Z0-9]*]] = getelementptr i8, ptr {{.*}}, i64 %[[offset]]
+; IR-NEXT:   store float 1.000000e+00, ptr %[[gep]]
 ; IR-NEXT:   %polly.indvar_next = add nsw i64 %polly.indvar, 1
 ; IR-NEXT:   %polly.loop_cond = icmp sle i64 %polly.indvar_next, %polly.par.UBAdjusted
 ; IR-NEXT:   br i1 %polly.loop_cond, label %polly.loop_header, label %polly.loop_exit
@@ -84,48 +83,46 @@
 
 ; IR: attributes #1 = { "polly.skip.fn" }
 
-; IR-STRIDE4:   call void @GOMP_parallel_loop_runtime_start(void (i8*)* @single_parallel_loop_polly_subfn, i8* %polly.par.userContext1, i32 0, i64 0, i64 1024, i64 4)
+; IR-STRIDE4:   call void @GOMP_parallel_loop_runtime_start(ptr @single_parallel_loop_polly_subfn, ptr %polly.par.userContext, i32 0, i64 0, i64 1024, i64 4)
 ; IR-STRIDE4:  add nsw i64 %polly.indvar, 3
 ; IR-STRIDE4:  %polly.indvar_next = add nsw i64 %polly.indvar, 4
 ; IR-STRIDE4   %polly.adjust_ub = sub i64 %polly.par.UBAdjusted, 4
 
-; LIBOMP-IR-STATIC-CHUNKED: %struct.ident_t = type { i32, i32, i32, i32, i8* }
+; LIBOMP-IR-STATIC-CHUNKED: %struct.ident_t = type { i32, i32, i32, i32, ptr }
 
 ; LIBOMP-IR-STATIC-CHUNKED-LABEL: single_parallel_loop()
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT: entry
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.par.userContext = alloca
 
 ; LIBOMP-IR-STATIC-CHUNKED-LABEL: polly.parallel.for:
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.par.userContext1 = bitcast {}* %polly.par.userContext to i8*
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   call void (%struct.ident_t*, i32, void (i32*, i32*, ...)*, ...) @__kmpc_fork_call(%struct.ident_t* @.loc.dummy, i32 4, void (i32*, i32*, ...)* bitcast (void (i32*, i32*, i64, i64, i64, i8*)* @single_parallel_loop_polly_subfn to void (i32*, i32*, ...)*), i64 0, i64 1024, i64 1, i8* %polly.par.userContext1)
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   call void (ptr, i32, ptr, ...) @__kmpc_fork_call(ptr @.loc.dummy, i32 4, ptr @single_parallel_loop_polly_subfn, i64 0, i64 1024, i64 1, ptr %polly.par.userContext)
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   br label %polly.exiting
 
-; LIBOMP-IR-STATIC-CHUNKED: define internal void @single_parallel_loop_polly_subfn(i32* %polly.kmpc.global_tid, i32* %polly.kmpc.bound_tid, i64 %polly.kmpc.lb, i64 %polly.kmpc.ub, i64 %polly.kmpc.inc, i8* %polly.kmpc.shared)
+; LIBOMP-IR-STATIC-CHUNKED: define internal void @single_parallel_loop_polly_subfn(ptr %polly.kmpc.global_tid, ptr %polly.kmpc.bound_tid, i64 %polly.kmpc.lb, i64 %polly.kmpc.ub, i64 %polly.kmpc.inc, ptr %polly.kmpc.shared)
 ; LIBOMP-IR-STATIC-CHUNKED-LABEL: polly.par.setup:
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.par.LBPtr = alloca i64
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.par.UBPtr = alloca i64
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.par.lastIterPtr = alloca i32
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.par.StridePtr = alloca i64
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.par.userContext = bitcast i8* %polly.kmpc.shared
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.par.global_tid = load i32, i32* %polly.kmpc.global_tid
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   store i64 %polly.kmpc.lb, i64* %polly.par.LBPtr
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   store i64 %polly.kmpc.ub, i64* %polly.par.UBPtr
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   store i32 0, i32* %polly.par.lastIterPtr
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   store i64 %polly.kmpc.inc, i64* %polly.par.StridePtr
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.par.global_tid = load i32, ptr %polly.kmpc.global_tid
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   store i64 %polly.kmpc.lb, ptr %polly.par.LBPtr
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   store i64 %polly.kmpc.ub, ptr %polly.par.UBPtr
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   store i32 0, ptr %polly.par.lastIterPtr
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   store i64 %polly.kmpc.inc, ptr %polly.par.StridePtr
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.indvar.UBAdjusted = add i64 %polly.kmpc.ub, -1
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   store i64 %polly.indvar.UBAdjusted, i64* %polly.par.UBPtr, align 8
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   call void @__kmpc_for_static_init_{{[4|8]}}(%struct.ident_t* @.loc.dummy{{[.0-9]*}}, i32 %polly.par.global_tid, i32 33, i32* %polly.par.lastIterPtr, i64* %polly.par.LBPtr, i64* %polly.par.UBPtr, i64* %polly.par.StridePtr, i64 1, i64 43)
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.kmpc.stride = load i64, i64* %polly.par.StridePtr, align 8
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.indvar.LB = load i64, i64* %polly.par.LBPtr
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.indvar.UB.temp = load i64, i64* %polly.par.UBPtr
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   store i64 %polly.indvar.UBAdjusted, ptr %polly.par.UBPtr, align 8
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   call void @__kmpc_for_static_init_{{[4|8]}}(ptr @.loc.dummy{{[.0-9]*}}, i32 %polly.par.global_tid, i32 33, ptr %polly.par.lastIterPtr, ptr %polly.par.LBPtr, ptr %polly.par.UBPtr, ptr %polly.par.StridePtr, i64 1, i64 43)
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.kmpc.stride = load i64, ptr %polly.par.StridePtr, align 8
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.indvar.LB = load i64, ptr %polly.par.LBPtr
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.indvar.UB.temp = load i64, ptr %polly.par.UBPtr
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.indvar.UB.inRange = icmp sle i64 %polly.indvar.UB.temp, %polly.indvar.UBAdjusted
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.indvar.UB = select i1 %polly.indvar.UB.inRange, i64 %polly.indvar.UB.temp, i64 %polly.indvar.UBAdjusted
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   store i64 %polly.indvar.UB, i64* %polly.par.UBPtr, align 8
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   store i64 %polly.indvar.UB, ptr %polly.par.UBPtr, align 8
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.hasIteration = icmp sle i64 %polly.indvar.LB, %polly.indvar.UB
 ; LIBOMP-IR-STATIC-CHUNKED:   br i1 %polly.hasIteration, label %polly.par.loadIVBounds, label %polly.par.exit
 
 ; LIBOMP-IR-STATIC-CHUNKED-LABEL: polly.par.exit:
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   call void @__kmpc_for_static_fini(%struct.ident_t* @.loc.dummy, i32 %polly.par.global_tid)
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   call void @__kmpc_for_static_fini(ptr @.loc.dummy, i32 %polly.par.global_tid)
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   ret void
 
 ; LIBOMP-IR-STATIC-CHUNKED-LABEL: polly.par.checkNext:
@@ -133,14 +130,14 @@
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %{{[0-9]+}} = add i64 %polly.indvar.UB.entry, %polly.kmpc.stride
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.indvar.nextUB.outOfBounds = icmp sgt i64 %{{[0-9]+}}, %polly.indvar.UBAdjusted
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.indvar.nextUB = select i1 %polly.indvar.nextUB.outOfBounds, i64 %polly.indvar.UBAdjusted, i64 %{{[0-9]+}}
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   store i64 %polly.indvar.nextLB, i64* %polly.par.LBPtr, align 8
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   store i64 %polly.indvar.nextUB, i64* %polly.par.UBPtr, align 8
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   store i64 %polly.indvar.nextLB, ptr %polly.par.LBPtr, align 8
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   store i64 %polly.indvar.nextUB, ptr %polly.par.UBPtr, align 8
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.hasWork = icmp sle i64 %polly.indvar.nextLB, %polly.indvar.UBAdjusted
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   br i1 %polly.hasWork, label %polly.par.loadIVBounds, label %polly.par.exit
 
 ; LIBOMP-IR-STATIC-CHUNKED-LABEL: polly.par.loadIVBounds:
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.indvar.LB.entry = load i64, i64* %polly.par.LBPtr, align 8
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.indvar.UB.entry = load i64, i64* %polly.par.UBPtr, align 8
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.indvar.LB.entry = load i64, ptr %polly.par.LBPtr, align 8
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.indvar.UB.entry = load i64, ptr %polly.par.UBPtr, align 8
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   br label %polly.loop_preheader
 
 ; LIBOMP-IR-STATIC-CHUNKED-LABEL: polly.loop_exit:
@@ -151,8 +148,9 @@
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   br label %polly.stmt.S
 
 ; LIBOMP-IR-STATIC-CHUNKED-LABEL: polly.stmt.S:
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %[[gep:[._a-zA-Z0-9]*]] = getelementptr [1024 x float], [1024 x float]* {{.*}}, i64 0, i64 %polly.indvar
-; LIBOMP-IR-STATIC-CHUNKED-NEXT:   store float 1.000000e+00, float* %[[gep]]
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %[[offset:[0-9]+]] = shl i64 %polly.indvar, 2
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %[[gep:[._a-zA-Z0-9]*]] = getelementptr i8, ptr {{.*}}, i64 %[[offset]]
+; LIBOMP-IR-STATIC-CHUNKED-NEXT:   store float 1.000000e+00, ptr %[[gep]]
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.indvar_next = add nsw i64 %polly.indvar, 1
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   %polly.loop_cond = icmp sle i64 %polly.indvar_next, %polly.indvar.UB.entry
 ; LIBOMP-IR-STATIC-CHUNKED-NEXT:   br i1 %polly.loop_cond, label %polly.loop_header, label %polly.loop_exit
@@ -162,13 +160,13 @@
 
 ; LIBOMP-IR-STATIC-CHUNKED: attributes #1 = { "polly.skip.fn" }
 
-; LIBOMP-IR-STATIC: define internal void @single_parallel_loop_polly_subfn(i32* %polly.kmpc.global_tid, i32* %polly.kmpc.bound_tid, i64 %polly.kmpc.lb, i64 %polly.kmpc.ub, i64 %polly.kmpc.inc, i8* %polly.kmpc.shared)
+; LIBOMP-IR-STATIC: define internal void @single_parallel_loop_polly_subfn(ptr %polly.kmpc.global_tid, ptr %polly.kmpc.bound_tid, i64 %polly.kmpc.lb, i64 %polly.kmpc.ub, i64 %polly.kmpc.inc, ptr %polly.kmpc.shared)
 ; LIBOMP-IR-STATIC-LABEL: polly.par.setup:
-; LIBOMP-IR-STATIC:   call void @__kmpc_for_static_init_{{[4|8]}}(%struct.ident_t* @.loc.dummy{{[.0-9]*}}, i32 %polly.par.global_tid, i32 34, i32* %polly.par.lastIterPtr, i64* %polly.par.LBPtr, i64* %polly.par.UBPtr, i64* %polly.par.StridePtr, i64 1, i64 1)
+; LIBOMP-IR-STATIC:   call void @__kmpc_for_static_init_{{[4|8]}}(ptr @.loc.dummy{{[.0-9]*}}, i32 %polly.par.global_tid, i32 34, ptr %polly.par.lastIterPtr, ptr %polly.par.LBPtr, ptr %polly.par.UBPtr, ptr %polly.par.StridePtr, i64 1, i64 1)
 ; LIBOMP-IR-STATIC:   br i1 %polly.hasIteration, label %polly.par.loadIVBounds, label %polly.par.exit
 
 ; LIBOMP-IR-STATIC-LABEL: polly.par.exit:
-; LIBOMP-IR-STATIC-NEXT:   call void @__kmpc_for_static_fini(%struct.ident_t* @.loc.dummy, i32 %polly.par.global_tid)
+; LIBOMP-IR-STATIC-NEXT:   call void @__kmpc_for_static_fini(ptr @.loc.dummy, i32 %polly.par.global_tid)
 ; LIBOMP-IR-STATIC-NEXT:   ret void
 
 ; LIBOMP-IR-STATIC-LABEL: polly.par.checkNext:
@@ -185,8 +183,9 @@
 ; LIBOMP-IR-STATIC-NEXT:   br label %polly.stmt.S
 
 ; LIBOMP-IR-STATIC-LABEL: polly.stmt.S:
-; LIBOMP-IR-STATIC-NEXT:   %[[gep:[._a-zA-Z0-9]*]] = getelementptr [1024 x float], [1024 x float]* {{.*}}, i64 0, i64 %polly.indvar
-; LIBOMP-IR-STATIC-NEXT:   store float 1.000000e+00, float* %[[gep]]
+; LIBOMP-IR-STATIC-NEXT:   %[[offset:[0-9]+]] = shl i64 %polly.indvar, 2
+; LIBOMP-IR-STATIC-NEXT:   %[[gep:[._a-zA-Z0-9]*]] = getelementptr i8, ptr {{.*}}, i64 %[[offset]]
+; LIBOMP-IR-STATIC-NEXT:   store float 1.000000e+00, ptr %[[gep]]
 ; LIBOMP-IR-STATIC-NEXT:   %polly.indvar_next = add nsw i64 %polly.indvar, 1
 ; LIBOMP-IR-STATIC-NEXT:   %polly.loop_cond = icmp sle i64 %polly.indvar_next, %polly.indvar.UB
 ; LIBOMP-IR-STATIC-NEXT:   br i1 %polly.loop_cond, label %polly.loop_header, label %polly.loop_exit
@@ -194,8 +193,8 @@
 ; LIBOMP-IR-STATIC-LABEL: polly.loop_preheader:
 ; LIBOMP-IR-STATIC-NEXT:   br label %polly.loop_header
 
-; LIBOMP-IR-DYNAMIC:   call void @__kmpc_dispatch_init_{{[4|8]}}(%struct.ident_t* @.loc.dummy, i32 %polly.par.global_tid, i32 35, i64 %polly.kmpc.lb, i64 %polly.indvar.UBAdjusted, i64 %polly.kmpc.inc, i64 1)
-; LIBOMP-IR-DYNAMIC-NEXT:   %{{[0-9]+}} = call i32 @__kmpc_dispatch_next_{{[4|8]}}(%struct.ident_t* @.loc.dummy, i32 %polly.par.global_tid, i32* %polly.par.lastIterPtr, i64* %polly.par.LBPtr, i64* %polly.par.UBPtr, i64* %polly.par.StridePtr)
+; LIBOMP-IR-DYNAMIC:   call void @__kmpc_dispatch_init_{{[4|8]}}(ptr @.loc.dummy, i32 %polly.par.global_tid, i32 35, i64 %polly.kmpc.lb, i64 %polly.indvar.UBAdjusted, i64 %polly.kmpc.inc, i64 1)
+; LIBOMP-IR-DYNAMIC-NEXT:   %{{[0-9]+}} = call i32 @__kmpc_dispatch_next_{{[4|8]}}(ptr @.loc.dummy, i32 %polly.par.global_tid, ptr %polly.par.lastIterPtr, ptr %polly.par.LBPtr, ptr %polly.par.UBPtr, ptr %polly.par.StridePtr)
 ; LIBOMP-IR-DYNAMIC-NEXT:   %polly.hasIteration = icmp eq i32 %{{[0-9]+}}, 1
 ; LIBOMP-IR-DYNAMIC-NEXT:   br i1 %polly.hasIteration, label %polly.par.loadIVBounds, label %polly.par.exit
 
@@ -203,18 +202,18 @@
 ; LIBOMP-IR-DYNAMIC-NEXT:   ret void
 
 ; LIBOMP-IR-DYNAMIC-LABEL: polly.par.checkNext:
-; LIBOMP-IR-DYNAMIC-NEXT:   %{{[0-9]+}} = call i32 @__kmpc_dispatch_next_{{[4|8]}}(%struct.ident_t* @.loc.dummy, i32 %polly.par.global_tid, i32* %polly.par.lastIterPtr, i64* %polly.par.LBPtr, i64* %polly.par.UBPtr, i64* %polly.par.StridePtr)
+; LIBOMP-IR-DYNAMIC-NEXT:   %{{[0-9]+}} = call i32 @__kmpc_dispatch_next_{{[4|8]}}(ptr @.loc.dummy, i32 %polly.par.global_tid, ptr %polly.par.lastIterPtr, ptr %polly.par.LBPtr, ptr %polly.par.UBPtr, ptr %polly.par.StridePtr)
 ; LIBOMP-IR-DYNAMIC-NEXT:   %polly.hasWork = icmp eq i32 %{{[0-9]+}}, 1
 ; LIBOMP-IR-DYNAMIC-NEXT:   br i1 %polly.hasWork, label %polly.par.loadIVBounds, label %polly.par.exit
 
 ; LIBOMP-IR-DYNAMIC-LABEL: polly.par.loadIVBounds:
-; LIBOMP-IR-DYNAMIC-NEXT:   %polly.indvar.LB = load i64, i64* %polly.par.LBPtr
-; LIBOMP-IR-DYNAMIC-NEXT:   %polly.indvar.UB = load i64, i64* %polly.par.UBPtr
+; LIBOMP-IR-DYNAMIC-NEXT:   %polly.indvar.LB = load i64, ptr %polly.par.LBPtr
+; LIBOMP-IR-DYNAMIC-NEXT:   %polly.indvar.UB = load i64, ptr %polly.par.UBPtr
 ; LIBOMP-IR-DYNAMIC-NEXT:   br label %polly.loop_preheader
 
-; LIBOMP-IR-DYNAMIC-FOUR:   call void @__kmpc_dispatch_init_{{[4|8]}}(%struct.ident_t* @.loc.dummy, i32 %polly.par.global_tid, i32 35, i64 %polly.kmpc.lb, i64 %polly.indvar.UBAdjusted, i64 %polly.kmpc.inc, i64 4)
+; LIBOMP-IR-DYNAMIC-FOUR:   call void @__kmpc_dispatch_init_{{[4|8]}}(ptr @.loc.dummy, i32 %polly.par.global_tid, i32 35, i64 %polly.kmpc.lb, i64 %polly.indvar.UBAdjusted, i64 %polly.kmpc.inc, i64 4)
 
-; LIBOMP-IR-STRIDE4:     call void (%struct.ident_t*, i32, void (i32*, i32*, ...)*, ...) @__kmpc_fork_call(%struct.ident_t* @.loc.dummy, i32 4, void (i32*, i32*, ...)* bitcast (void (i32*, i32*, i64, i64, i64, i8*)* @single_parallel_loop_polly_subfn to void (i32*, i32*, ...)*), i64 0, i64 1024, i64 4, i8* %polly.par.userContext1)
+; LIBOMP-IR-STRIDE4:     call void (ptr, i32, ptr, ...) @__kmpc_fork_call(ptr @.loc.dummy, i32 4, ptr @single_parallel_loop_polly_subfn, i64 0, i64 1024, i64 4, ptr %polly.par.userContext)
 
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
 
@@ -226,12 +225,12 @@ entry:
 
 for.i:
   %indvar = phi i64 [ %indvar.next, %for.inc], [ 0, %entry ]
-  %scevgep = getelementptr [1024 x float], [1024 x float]* @A, i64 0, i64 %indvar
+  %scevgep = getelementptr [1024 x float], ptr @A, i64 0, i64 %indvar
   %exitcond = icmp ne i64 %indvar, 1024
   br i1 %exitcond, label %S, label %exit
 
 S:
-  store float 1.0, float* %scevgep
+  store float 1.0, ptr %scevgep
   br label %for.inc
 
 for.inc:
diff --git a/polly/test/CodeGen/RuntimeDebugBuilder/combine_different_values.ll b/polly/test/CodeGen/RuntimeDebugBuilder/combine_different_values.ll
index f904b2a652b46..84827dd260496 100644
--- a/polly/test/CodeGen/RuntimeDebugBuilder/combine_different_values.ll
+++ b/polly/test/CodeGen/RuntimeDebugBuilder/combine_different_values.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -S \
+; RUN: opt %loadPolly -polly-codegen -S \
 ; RUN: -polly-codegen-add-debug-printing \
 ; RUN: -polly-ignore-aliasing < %s | FileCheck %s
 
@@ -51,49 +51,55 @@
 ; CHECK: @22 = private unnamed_addr addrspace(4) constant [2 x i8] c"\0A\00"
 ; CHECK: @23 = private unnamed_addr constant [12 x i8] c"%s%ld%s%f%s\00"
 
-; CHECK: %0 = ptrtoint double* %scevgep to i64
-; CHECK: %1 = call i32 (...) @printf(i8* getelementptr inbounds ([12 x i8], [12 x i8]* @3, i32 0, i32 0), i8 addrspace(4)* getelementptr inbounds ([11 x i8], [11 x i8] addrspace(4)* @0, i32 0, i32 0), i64 %0, i8 addrspace(4)* getelementptr inbounds ([3 x i8], [3 x i8] addrspace(4)* @1, i32 0, i32 0), double %tmp3_p_scalar_, i8 addrspace(4)* getelementptr inbounds ([2 x i8], [2 x i8] addrspace(4)* @2, i32 0, i32 0))
-; CHECK: %2 = call i32 @fflush(i8* null)
-; CHECK: %scevgep1 = getelementptr i8, i8* %C, i64 %polly.indvar
-; CHECK: %tmp5_p_scalar_ = load i8, i8* %scevgep1
-; CHECK: %3 = ptrtoint i8* %scevgep1 to i64
-; CHECK: %4 = sext i8 %tmp5_p_scalar_ to i64
-; CHECK: %5 = call i32 (...) @printf(i8* getelementptr inbounds ([13 x i8], [13 x i8]* @7, i32 0, i32 0), i8 addrspace(4)* getelementptr inbounds ([11 x i8], [11 x i8] addrspace(4)* @4, i32 0, i32 0), i64 %3, i8 addrspace(4)* getelementptr inbounds ([3 x i8], [3 x i8] addrspace(4)* @5, i32 0, i32 0), i64 %4, i8 addrspace(4)* getelementptr inbounds ([2 x i8], [2 x i8] addrspace(4)* @6, i32 0, i32 0))
-; CHECK: %6 = call i32 @fflush(i8* null)
+; CHECK: %0 = shl nuw nsw i64 %polly.indvar, 3
+; CHECK: %scevgep = getelementptr i8, ptr %B, i64 %0
+; CHECK: %tmp3_p_scalar_ = load double, ptr %scevgep, align 8, !alias.scope !0, !noalias !3
+; CHECK: %1 = ptrtoint ptr %scevgep to i64
+; CHECK: %2 = call i32 (...) @printf(ptr @3, ptr addrspace(4) @0, i64 %1, ptr addrspace(4) @1, double %tmp3_p_scalar_, ptr addrspace(4) @2)
+; CHECK: %3 = call i32 @fflush(ptr null)
+; CHECK: %scevgep1 = getelementptr i8, ptr %C, i64 %polly.indvar
+; CHECK: %tmp5_p_scalar_ = load i8, ptr %scevgep1, align 1, !alias.scope !8, !noalias !9
+; CHECK: %4 = ptrtoint ptr %scevgep1 to i64
+; CHECK: %5 = sext i8 %tmp5_p_scalar_ to i64
+; CHECK: %6 = call i32 (...) @printf(ptr @7, ptr addrspace(4) @4, i64 %4, ptr addrspace(4) @5, i64 %5, ptr addrspace(4) @6)
+; CHECK: %7 = call i32 @fflush(ptr null)
 ; CHECK: %p_tmp6 = sitofp i8 %tmp5_p_scalar_ to double
 ; CHECK: %p_tmp7 = fadd double %tmp3_p_scalar_, %p_tmp6
-; CHECK: %scevgep2 = getelementptr i32, i32* %D, i64 %polly.indvar
-; CHECK: %tmp9_p_scalar_ = load i32, i32* %scevgep2
-; CHECK: %7 = ptrtoint i32* %scevgep2 to i64
-; CHECK: %8 = sext i32 %tmp9_p_scalar_ to i64
-; CHECK: %9 = call i32 (...) @printf(i8* getelementptr inbounds ([13 x i8], [13 x i8]* @11, i32 0, i32 0), i8 addrspace(4)* getelementptr inbounds ([11 x i8], [11 x i8] addrspace(4)* @8, i32 0, i32 0), i64 %7, i8 addrspace(4)* getelementptr inbounds ([3 x i8], [3 x i8] addrspace(4)* @9, i32 0, i32 0), i64 %8, i8 addrspace(4)* getelementptr inbounds ([2 x i8], [2 x i8] addrspace(4)* @10, i32 0, i32 0))
-; CHECK: %10 = call i32 @fflush(i8* null)
+; CHECK: %8 = shl nuw nsw i64 %polly.indvar, 2
+; CHECK: %scevgep2 = getelementptr i8, ptr %D, i64 %8
+; CHECK: %tmp9_p_scalar_ = load i32, ptr %scevgep2, align 4, !alias.scope !10, !noalias !11
+; CHECK: %9 = ptrtoint ptr %scevgep2 to i64
+; CHECK: %10 = sext i32 %tmp9_p_scalar_ to i64
+; CHECK: %11 = call i32 (...) @printf(ptr @11, ptr addrspace(4) @8, i64 %9, ptr addrspace(4) @9, i64 %10, ptr addrspace(4) @10)
+; CHECK: %12 = call i32 @fflush(ptr null)
 ; CHECK: %p_tmp10 = sitofp i32 %tmp9_p_scalar_ to double
 ; CHECK: %p_tmp11 = fadd double %p_tmp7, %p_tmp10
-; CHECK: %scevgep3 = getelementptr i64, i64* %E, i64 %polly.indvar
-; CHECK: %tmp13_p_scalar_ = load i64, i64* %scevgep3
-; CHECK: %11 = ptrtoint i64* %scevgep3 to i64
-; CHECK: %12 = call i32 (...) @printf(i8* getelementptr inbounds ([13 x i8], [13 x i8]* @15, i32 0, i32 0), i8 addrspace(4)* getelementptr inbounds ([11 x i8], [11 x i8] addrspace(4)* @12, i32 0, i32 0), i64 %11, i8 addrspace(4)* getelementptr inbounds ([3 x i8], [3 x i8] addrspace(4)* @13, i32 0, i32 0), i64 %tmp13_p_scalar_, i8 addrspace(4)* getelementptr inbounds ([2 x i8], [2 x i8] addrspace(4)* @14, i32 0, i32 0))
-; CHECK: %13 = call i32 @fflush(i8* null)
+; CHECK: %13 = shl nuw nsw i64 %polly.indvar, 3
+; CHECK: %scevgep3 = getelementptr i8, ptr %E, i64 %13
+; CHECK: %tmp13_p_scalar_ = load i64, ptr %scevgep3, align 8, !alias.scope !12, !noalias !13
+; CHECK: %14 = ptrtoint ptr %scevgep3 to i64
+; CHECK: %15 = call i32 (...) @printf(ptr @15, ptr addrspace(4) @12, i64 %14, ptr addrspace(4) @13, i64 %tmp13_p_scalar_, ptr addrspace(4) @14)
+; CHECK: %16 = call i32 @fflush(ptr null)
 ; CHECK: %p_tmp14 = sitofp i64 %tmp13_p_scalar_ to double
 ; CHECK: %p_tmp15 = fadd double %p_tmp11, %p_tmp14
-; CHECK: %scevgep4 = getelementptr float, float* %A, i64 %polly.indvar
-; CHECK: %tmp17_p_scalar_ = load float, float* %scevgep4
-; CHECK: %14 = ptrtoint float* %scevgep4 to i64
-; CHECK: %15 = fpext float %tmp17_p_scalar_ to double
-; CHECK: %16 = call i32 (...) @printf(i8* getelementptr inbounds ([12 x i8], [12 x i8]* @19, i32 0, i32 0), i8 addrspace(4)* getelementptr inbounds ([11 x i8], [11 x i8] addrspace(4)* @16, i32 0, i32 0), i64 %14, i8 addrspace(4)* getelementptr inbounds ([3 x i8], [3 x i8] addrspace(4)* @17, i32 0, i32 0), double %15, i8 addrspace(4)* getelementptr inbounds ([2 x i8], [2 x i8] addrspace(4)* @18, i32 0, i32 0))
-; CHECK: %17 = call i32 @fflush(i8* null)
+; CHECK: %17 = shl nuw nsw i64 %polly.indvar, 2
+; CHECK: %scevgep4 = getelementptr i8, ptr %A, i64 %17
+; CHECK: %tmp17_p_scalar_ = load float, ptr %scevgep4, align 4, !alias.scope !14, !noalias !15
+; CHECK: %18 = ptrtoint ptr %scevgep4 to i64
+; CHECK: %19 = fpext float %tmp17_p_scalar_ to double
+; CHECK: %20 = call i32 (...) @printf(ptr @19, ptr addrspace(4) @16, i64 %18, ptr addrspace(4) @17, double %19, ptr addrspace(4) @18)
+; CHECK: %21 = call i32 @fflush(ptr null)
 ; CHECK: %p_tmp18 = fpext float %tmp17_p_scalar_ to double
 ; CHECK: %p_tmp19 = fadd double %p_tmp18, %p_tmp15
 ; CHECK: %p_tmp20 = fptrunc double %p_tmp19 to float
-; CHECK: %18 = ptrtoint float* %scevgep4 to i64
-; CHECK: %19 = fpext float %p_tmp20 to double
-; CHECK: %20 = call i32 (...) @printf(i8* getelementptr inbounds ([12 x i8], [12 x i8]* @23, i32 0, i32 0), i8 addrspace(4)* getelementptr inbounds ([11 x i8], [11 x i8] addrspace(4)* @20, i32 0, i32 0), i64 %18, i8 addrspace(4)* getelementptr inbounds ([3 x i8], [3 x i8] addrspace(4)* @21, i32 0, i32 0), double %19, i8 addrspace(4)* getelementptr inbounds ([2 x i8], [2 x i8] addrspace(4)* @22, i32 0, i32 0))
-; CHECK: %21 = call i32 @fflush(i8* null)
+; CHECK: %22 = ptrtoint ptr %scevgep4 to i64
+; CHECK: %23 = fpext float %p_tmp20 to double
+; CHECK: %24 = call i32 (...) @printf(ptr @23, ptr addrspace(4) @20, i64 %22, ptr addrspace(4) @21, double %23, ptr addrspace(4) @22)
+; CHECK: %25 = call i32 @fflush(ptr null)
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @foo(float* noalias %A, double* noalias %B, i8* noalias %C, i32* noalias %D, i64* noalias %E) {
+define void @foo(ptr noalias %A, ptr noalias %B, ptr noalias %C, ptr noalias %D, ptr noalias %E) {
 bb:
   br label %bb1
 
@@ -103,26 +109,26 @@ bb1:                                              ; preds = %bb21, %bb
   br i1 %exitcond, label %bb2, label %bb23
 
 bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds double, double* %B, i64 %i.0
-  %tmp3 = load double, double* %tmp, align 8
-  %tmp4 = getelementptr inbounds i8, i8* %C, i64 %i.0
-  %tmp5 = load i8, i8* %tmp4, align 1
+  %tmp = getelementptr inbounds double, ptr %B, i64 %i.0
+  %tmp3 = load double, ptr %tmp, align 8
+  %tmp4 = getelementptr inbounds i8, ptr %C, i64 %i.0
+  %tmp5 = load i8, ptr %tmp4, align 1
   %tmp6 = sitofp i8 %tmp5 to double
   %tmp7 = fadd double %tmp3, %tmp6
-  %tmp8 = getelementptr inbounds i32, i32* %D, i64 %i.0
-  %tmp9 = load i32, i32* %tmp8, align 4
+  %tmp8 = getelementptr inbounds i32, ptr %D, i64 %i.0
+  %tmp9 = load i32, ptr %tmp8, align 4
   %tmp10 = sitofp i32 %tmp9 to double
   %tmp11 = fadd double %tmp7, %tmp10
-  %tmp12 = getelementptr inbounds i64, i64* %E, i64 %i.0
-  %tmp13 = load i64, i64* %tmp12, align 8
+  %tmp12 = getelementptr inbounds i64, ptr %E, i64 %i.0
+  %tmp13 = load i64, ptr %tmp12, align 8
   %tmp14 = sitofp i64 %tmp13 to double
   %tmp15 = fadd double %tmp11, %tmp14
-  %tmp16 = getelementptr inbounds float, float* %A, i64 %i.0
-  %tmp17 = load float, float* %tmp16, align 4
+  %tmp16 = getelementptr inbounds float, ptr %A, i64 %i.0
+  %tmp17 = load float, ptr %tmp16, align 4
   %tmp18 = fpext float %tmp17 to double
   %tmp19 = fadd double %tmp18, %tmp15
   %tmp20 = fptrunc double %tmp19 to float
-  store float %tmp20, float* %tmp16, align 4
+  store float %tmp20, ptr %tmp16, align 4
   br label %bb21
 
 bb21:                                             ; preds = %bb2
@@ -149,16 +155,16 @@ bb1:                                              ; preds = %bb7, %bb
 
 bb2:                                              ; preds = %bb1
   fence seq_cst
-  %tmp = getelementptr inbounds [10 x i64], [10 x i64]* %E, i64 0, i64 %i.0
-  store i64 42, i64* %tmp, align 8
-  %tmp3 = getelementptr inbounds [10 x i32], [10 x i32]* %D, i64 0, i64 %i.0
-  store i32 42, i32* %tmp3, align 4
-  %tmp4 = getelementptr inbounds [10 x i8], [10 x i8]* %C, i64 0, i64 %i.0
-  store i8 42, i8* %tmp4, align 1
-  %tmp5 = getelementptr inbounds [10 x double], [10 x double]* %B, i64 0, i64 %i.0
-  store double 4.200000e+01, double* %tmp5, align 8
-  %tmp6 = getelementptr inbounds [10 x float], [10 x float]* %A, i64 0, i64 %i.0
-  store float 4.200000e+01, float* %tmp6, align 4
+  %tmp = getelementptr inbounds [10 x i64], ptr %E, i64 0, i64 %i.0
+  store i64 42, ptr %tmp, align 8
+  %tmp3 = getelementptr inbounds [10 x i32], ptr %D, i64 0, i64 %i.0
+  store i32 42, ptr %tmp3, align 4
+  %tmp4 = getelementptr inbounds [10 x i8], ptr %C, i64 0, i64 %i.0
+  store i8 42, ptr %tmp4, align 1
+  %tmp5 = getelementptr inbounds [10 x double], ptr %B, i64 0, i64 %i.0
+  store double 4.200000e+01, ptr %tmp5, align 8
+  %tmp6 = getelementptr inbounds [10 x float], ptr %A, i64 0, i64 %i.0
+  store float 4.200000e+01, ptr %tmp6, align 4
   br label %bb7
 
 bb7:                                              ; preds = %bb2
@@ -166,14 +172,14 @@ bb7:                                              ; preds = %bb2
   br label %bb1
 
 bb9:                                              ; preds = %bb1
-  %tmp10 = getelementptr inbounds [10 x float], [10 x float]* %A, i64 0, i64 0
-  %tmp11 = getelementptr inbounds [10 x double], [10 x double]* %B, i64 0, i64 0
-  %tmp12 = getelementptr inbounds [10 x i8], [10 x i8]* %C, i64 0, i64 0
-  %tmp13 = getelementptr inbounds [10 x i32], [10 x i32]* %D, i64 0, i64 0
-  %tmp14 = getelementptr inbounds [10 x i64], [10 x i64]* %E, i64 0, i64 0
-  call void @foo(float* %tmp10, double* %tmp11, i8* %tmp12, i32* %tmp13, i64* %tmp14)
-  %tmp15 = getelementptr inbounds [10 x float], [10 x float]* %A, i64 0, i64 8
-  %tmp16 = load float, float* %tmp15, align 16
+  %tmp10 = getelementptr inbounds [10 x float], ptr %A, i64 0, i64 0
+  %tmp11 = getelementptr inbounds [10 x double], ptr %B, i64 0, i64 0
+  %tmp12 = getelementptr inbounds [10 x i8], ptr %C, i64 0, i64 0
+  %tmp13 = getelementptr inbounds [10 x i32], ptr %D, i64 0, i64 0
+  %tmp14 = getelementptr inbounds [10 x i64], ptr %E, i64 0, i64 0
+  call void @foo(ptr %tmp10, ptr %tmp11, ptr %tmp12, ptr %tmp13, ptr %tmp14)
+  %tmp15 = getelementptr inbounds [10 x float], ptr %A, i64 0, i64 8
+  %tmp16 = load float, ptr %tmp15, align 16
   %tmp17 = fptosi float %tmp16 to i32
   ret i32 %tmp17
 }
diff --git a/polly/test/CodeGen/aliasing_different_base_and_access_type.ll b/polly/test/CodeGen/aliasing_different_base_and_access_type.ll
index ddcda888e15f6..a087414b84034 100644
--- a/polly/test/CodeGen/aliasing_different_base_and_access_type.ll
+++ b/polly/test/CodeGen/aliasing_different_base_and_access_type.ll
@@ -1,13 +1,12 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -S -polly-codegen < %s | FileCheck %s
+; RUN: opt %loadPolly -S -polly-codegen < %s | FileCheck %s
 ;
 ; We have to cast %B to "short *" before we create RTCs.
 ;
-; CHECK:   %polly.access.cast.B = bitcast i32* %B to i16*
-; CHECK-NEXT:   %polly.access.B = getelementptr i16, i16* %polly.access.cast.B, i64 1024
+; CHECK:   %polly.access.B = getelementptr i16, ptr %B, i64 1024
 ;
 ; We should never access %B as an i32 pointer:
 ;
-; CHECK-NOT: getelementptr i32, i32* %B
+; CHECK-NOT: getelementptr i32, ptr %B
 ;
 ;    void jd(int *A, int *B) {
 ;      for (int i = 0; i < 1024; i++)
@@ -16,7 +15,7 @@
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @jd(i32* %A, i32* %B) {
+define void @jd(ptr %A, ptr %B) {
 entry:
   br label %for.cond
 
@@ -26,12 +25,11 @@ for.cond:                                         ; preds = %for.inc, %entry
   br i1 %exitcond, label %for.body, label %for.end
 
 for.body:                                         ; preds = %for.cond
-  %tmp = bitcast i32* %B to i16*
-  %arrayidx = getelementptr inbounds i16, i16* %tmp, i64 %indvars.iv
-  %tmp1 = load i16, i16* %arrayidx, align 2
+  %arrayidx = getelementptr inbounds i16, ptr %B, i64 %indvars.iv
+  %tmp1 = load i16, ptr %arrayidx, align 2
   %conv = sext i16 %tmp1 to i32
-  %arrayidx2 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  store i32 %conv, i32* %arrayidx2, align 4
+  %arrayidx2 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  store i32 %conv, ptr %arrayidx2, align 4
   br label %for.inc
 
 for.inc:                                          ; preds = %for.body
diff --git a/polly/test/CodeGen/aliasing_different_pointer_types.ll b/polly/test/CodeGen/aliasing_different_pointer_types.ll
index 94582676a3f9b..91f5eab6b2a65 100644
--- a/polly/test/CodeGen/aliasing_different_pointer_types.ll
+++ b/polly/test/CodeGen/aliasing_different_pointer_types.ll
@@ -1,18 +1,18 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -S < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-codegen -S < %s | FileCheck %s
 ;
 ; Check that we cast the different pointer types correctly before we compare
 ; them in the RTC's. We use i8* as max pointer type.
 ;
 ; CHECK:   polly.split_new_and_old:
-; CHECK:   %polly.access.B = getelementptr float*, float** %B, i64 1024
-; CHECK:   %polly.access.A = getelementptr double*, double** %A, i64 0
-; CHECK:   %[[paBb:[._a-zA-Z0-9]]] = ptrtoint float** %polly.access.B to i64
-; CHECK:   %[[paAb:[._a-zA-Z0-9]]] = ptrtoint double** %polly.access.A to i64
+; CHECK:   %polly.access.B = getelementptr ptr, ptr %B, i64 1024
+; CHECK:   %polly.access.A = getelementptr ptr, ptr %A, i64 0
+; CHECK:   %[[paBb:[._a-zA-Z0-9]]] = ptrtoint ptr %polly.access.B to i64
+; CHECK:   %[[paAb:[._a-zA-Z0-9]]] = ptrtoint ptr %polly.access.A to i64
 ; CHECK:   %[[ALeB:[._a-zA-Z0-9]]] = icmp ule i64 %[[paBb]], %[[paAb]]
-; CHECK:   %polly.access.A1 = getelementptr double*, double** %A, i64 1024
-; CHECK:   %polly.access.B2 = getelementptr float*, float** %B, i64 0
-; CHECK:   %[[paA1b:[._a-zA-Z0-9]]] = ptrtoint double** %polly.access.A1 to i64
-; CHECK:   %[[paB2b:[._a-zA-Z0-9]]] = ptrtoint float** %polly.access.B2 to i64
+; CHECK:   %polly.access.A1 = getelementptr ptr, ptr %A, i64 1024
+; CHECK:   %polly.access.B2 = getelementptr ptr, ptr %B, i64 0
+; CHECK:   %[[paA1b:[._a-zA-Z0-9]]] = ptrtoint ptr %polly.access.A1 to i64
+; CHECK:   %[[paB2b:[._a-zA-Z0-9]]] = ptrtoint ptr %polly.access.B2 to i64
 ; CHECK:   %[[A1LeB2:[._a-zA-Z0-9]]] = icmp ule i64 %[[paA1b]], %[[paB2b]]
 ; CHECK:   %[[le1OrLe2:[._a-zA-Z0-9]]] = or i1 %[[ALeB]], %[[A1LeB2]]
 ; CHECK:   %[[orAndTrue:[._a-zA-Z0-9]]] = and i1 true, %[[le1OrLe2]]
@@ -24,7 +24,7 @@
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @jd(double** %A, float** %B) {
+define void @jd(ptr %A, ptr %B) {
 entry:
   br label %for.cond
 
@@ -34,11 +34,10 @@ for.cond:                                         ; preds = %for.inc, %entry
   br i1 %exitcond, label %for.body, label %for.end
 
 for.body:                                         ; preds = %for.cond
-  %arrayidx = getelementptr inbounds float*, float** %B, i64 %indvars.iv
-  %tmp = load float*, float** %arrayidx, align 8
-  %tmp1 = bitcast float* %tmp to double*
-  %arrayidx2 = getelementptr inbounds double*, double** %A, i64 %indvars.iv
-  store double* %tmp1, double** %arrayidx2, align 8
+  %arrayidx = getelementptr inbounds ptr, ptr %B, i64 %indvars.iv
+  %tmp = load ptr, ptr %arrayidx, align 8
+  %arrayidx2 = getelementptr inbounds ptr, ptr %A, i64 %indvars.iv
+  store ptr %tmp, ptr %arrayidx2, align 8
   br label %for.inc
 
 for.inc:                                          ; preds = %for.body
diff --git a/polly/test/CodeGen/aliasing_parametric_simple_1.ll b/polly/test/CodeGen/aliasing_parametric_simple_1.ll
index aa72882e909ed..5422da4426e97 100644
--- a/polly/test/CodeGen/aliasing_parametric_simple_1.ll
+++ b/polly/test/CodeGen/aliasing_parametric_simple_1.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -S < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-codegen -S < %s | FileCheck %s
 ;
 ;    void jd(int *A, int *B, int c) {
 ;      for (int i = 0; i < 1024; i++)
@@ -10,15 +10,15 @@
 ; CHECK:  %[[Cp1O:[._a-zA-Z0-9]*]] = extractvalue { i64, i1 } %[[Cp1]], 1
 ; CHECK:  %[[OS:[._a-zA-Z0-9]*]]   = or i1 false, %[[Cp1O]]
 ; CHECK:  %[[Cp1R:[._a-zA-Z0-9]*]] = extractvalue { i64, i1 } %[[Cp1]], 0
-; CHECK:  %[[BMax:[._a-zA-Z0-9]*]] = getelementptr i32, i32* %B, i64 %[[Cp1R]]
-; CHECK:  %[[AMin:[._a-zA-Z0-9]*]] = getelementptr i32, i32* %A, i64 0
-; CHECK:  %[[BMaxI:[._a-zA-Z0-9]*]] = ptrtoint i32* %[[BMax]] to i64
-; CHECK:  %[[AMinI:[._a-zA-Z0-9]*]] = ptrtoint i32* %[[AMin]] to i64
+; CHECK:  %[[BMax:[._a-zA-Z0-9]*]] = getelementptr i32, ptr %B, i64 %[[Cp1R]]
+; CHECK:  %[[AMin:[._a-zA-Z0-9]*]] = getelementptr i32, ptr %A, i64 0
+; CHECK:  %[[BMaxI:[._a-zA-Z0-9]*]] = ptrtoint ptr %[[BMax]] to i64
+; CHECK:  %[[AMinI:[._a-zA-Z0-9]*]] = ptrtoint ptr %[[AMin]] to i64
 ; CHECK:  %[[BltA:[._a-zA-Z0-9]*]] = icmp ule i64 %[[BMaxI]], %[[AMinI]]
-; CHECK:  %[[AMax:[._a-zA-Z0-9]*]] = getelementptr i32, i32* %A, i64 1024
-; CHECK:  %[[BMin:[._a-zA-Z0-9]*]] = getelementptr i32, i32* %B, i32 %c
-; CHECK:  %[[AMaxI:[._a-zA-Z0-9]*]] = ptrtoint i32* %[[AMax]] to i64
-; CHECK:  %[[BMinI:[._a-zA-Z0-9]*]] = ptrtoint i32* %[[BMin]] to i64
+; CHECK:  %[[AMax:[._a-zA-Z0-9]*]] = getelementptr i32, ptr %A, i64 1024
+; CHECK:  %[[BMin:[._a-zA-Z0-9]*]] = getelementptr i32, ptr %B, i32 %c
+; CHECK:  %[[AMaxI:[._a-zA-Z0-9]*]] = ptrtoint ptr %[[AMax]] to i64
+; CHECK:  %[[BMinI:[._a-zA-Z0-9]*]] = ptrtoint ptr %[[BMin]] to i64
 ; CHECK:  %[[AltB:[._a-zA-Z0-9]*]] = icmp ule i64 %[[AMaxI]], %[[BMinI]]
 ; CHECK:  %[[NoAlias:[._a-zA-Z0-9]*]] = or i1 %[[BltA]], %[[AltB]]
 ; CHECK:  %[[RTC:[._a-zA-Z0-9]*]] = and i1 true, %[[NoAlias]]
@@ -28,7 +28,7 @@
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @jd(i32* %A, i32* %B, i32 %c) {
+define void @jd(ptr %A, ptr %B, i32 %c) {
 entry:
   br label %for.cond
 
@@ -39,10 +39,10 @@ for.cond:                                         ; preds = %for.inc, %entry
 
 for.body:                                         ; preds = %for.cond
   %idxprom = sext i32 %c to i64
-  %arrayidx = getelementptr inbounds i32, i32* %B, i64 %idxprom
-  %tmp = load i32, i32* %arrayidx, align 4
-  %arrayidx2 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  store i32 %tmp, i32* %arrayidx2, align 4
+  %arrayidx = getelementptr inbounds i32, ptr %B, i64 %idxprom
+  %tmp = load i32, ptr %arrayidx, align 4
+  %arrayidx2 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  store i32 %tmp, ptr %arrayidx2, align 4
   br label %for.inc
 
 for.inc:                                          ; preds = %for.body
diff --git a/polly/test/CodeGen/aliasing_parametric_simple_2.ll b/polly/test/CodeGen/aliasing_parametric_simple_2.ll
index 510d6def7b03f..de945d403f922 100644
--- a/polly/test/CodeGen/aliasing_parametric_simple_2.ll
+++ b/polly/test/CodeGen/aliasing_parametric_simple_2.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -S < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-codegen -S < %s | FileCheck %s
 ;
 ;    void jd(int *A, int *B, int c) {
 ;      for (int i = 0; i < 1024; i++)
@@ -13,12 +13,12 @@
 ; CHECK-NEXT:  %[[M3R:[._a-zA-Z0-9]*]] = extractvalue { i64, i1 } %[[M3]], 0
 ; CHECK-NEXT:  %[[M1:[._a-zA-Z0-9]*]] = icmp sgt i64 6, %[[M3R]]
 ; CHECK-NEXT:  %[[M4:[._a-zA-Z0-9]*]] = select i1 %[[M1]], i64 6, i64 %[[M3R]]
-; CHECK-NEXT:  %[[BMax:[._a-zA-Z0-9]*]] = getelementptr i32, i32* %B, i64 %[[M4]]
-; CHECK-NEXT:  %[[AMin:[._a-zA-Z0-9]*]] = getelementptr i32, i32* %A, i64 0
-; CHECK-NEXT:  %[[BMaxI:[._a-zA-Z0-9]*]] = ptrtoint i32* %[[BMax]] to i64
-; CHECK-NEXT:  %[[AMinI:[._a-zA-Z0-9]*]] = ptrtoint i32* %[[AMin]] to i64
+; CHECK-NEXT:  %[[BMax:[._a-zA-Z0-9]*]] = getelementptr i32, ptr %B, i64 %[[M4]]
+; CHECK-NEXT:  %[[AMin:[._a-zA-Z0-9]*]] = getelementptr i32, ptr %A, i64 0
+; CHECK-NEXT:  %[[BMaxI:[._a-zA-Z0-9]*]] = ptrtoint ptr %[[BMax]] to i64
+; CHECK-NEXT:  %[[AMinI:[._a-zA-Z0-9]*]] = ptrtoint ptr %[[AMin]] to i64
 ; CHECK-NEXT:  %[[BltA:[._a-zA-Z0-9]*]] = icmp ule i64 %[[BMaxI]], %[[AMinI]]
-; CHECK-NEXT:  %[[AMax:[._a-zA-Z0-9]*]] = getelementptr i32, i32* %A, i64 1024
+; CHECK-NEXT:  %[[AMax:[._a-zA-Z0-9]*]] = getelementptr i32, ptr %A, i64 1024
 ; CHECK-NEXT:  %[[m0:[._a-zA-Z0-9]*]] = sext i32 %c to i64
 ; CHECK-NEXT:  %[[m3:[._a-zA-Z0-9]*]] = call { i64, i1 } @llvm.ssub.with.overflow.i64(i64 %[[m0]], i64 10)
 ; CHECK-NEXT:  %[[m3O:[._a-zA-Z0-9]*]] = extractvalue { i64, i1 } %[[m3]], 1
@@ -26,9 +26,9 @@
 ; CHECK-NEXT:  %[[m3R:[._a-zA-Z0-9]*]] = extractvalue { i64, i1 } %[[m3]], 0
 ; CHECK-NEXT:  %[[m1:[._a-zA-Z0-9]*]] = icmp slt i64 5, %[[m3R]]
 ; CHECK-NEXT:  %[[m4:[._a-zA-Z0-9]*]] = select i1 %[[m1]], i64 5, i64 %[[m3R]]
-; CHECK-NEXT:  %[[BMin:[._a-zA-Z0-9]*]] = getelementptr i32, i32* %B, i64 %[[m4]]
-; CHECK-NEXT:  %[[AMaxI:[._a-zA-Z0-9]*]] = ptrtoint i32* %[[AMax]] to i64
-; CHECK-NEXT:  %[[BMinI:[._a-zA-Z0-9]*]] = ptrtoint i32* %[[BMin]] to i64
+; CHECK-NEXT:  %[[BMin:[._a-zA-Z0-9]*]] = getelementptr i32, ptr %B, i64 %[[m4]]
+; CHECK-NEXT:  %[[AMaxI:[._a-zA-Z0-9]*]] = ptrtoint ptr %[[AMax]] to i64
+; CHECK-NEXT:  %[[BMinI:[._a-zA-Z0-9]*]] = ptrtoint ptr %[[BMin]] to i64
 ; CHECK-NEXT:  %[[AltB:[._a-zA-Z0-9]*]] = icmp ule i64 %[[AMaxI]], %[[BMinI]]
 ; CHECK-NEXT:  %[[NoAlias:[._a-zA-Z0-9]*]] = or i1 %[[BltA]], %[[AltB]]
 ; CHECK-NEXT:  %[[RTC:[._a-zA-Z0-9]*]] = and i1 %[[Ctx]], %[[NoAlias]]
@@ -38,7 +38,7 @@
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @jd(i32* %A, i32* %B, i32 %c) {
+define void @jd(ptr %A, ptr %B, i32 %c) {
 entry:
   br label %for.cond
 
@@ -50,13 +50,13 @@ for.cond:                                         ; preds = %for.inc, %entry
 for.body:                                         ; preds = %for.cond
   %sub = add nsw i32 %c, -10
   %idxprom = sext i32 %sub to i64
-  %arrayidx = getelementptr inbounds i32, i32* %B, i64 %idxprom
-  %tmp = load i32, i32* %arrayidx, align 4
-  %arrayidx1 = getelementptr inbounds i32, i32* %B, i64 5
-  %tmp1 = load i32, i32* %arrayidx1, align 4
+  %arrayidx = getelementptr inbounds i32, ptr %B, i64 %idxprom
+  %tmp = load i32, ptr %arrayidx, align 4
+  %arrayidx1 = getelementptr inbounds i32, ptr %B, i64 5
+  %tmp1 = load i32, ptr %arrayidx1, align 4
   %add = add nsw i32 %tmp, %tmp1
-  %arrayidx3 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  store i32 %add, i32* %arrayidx3, align 4
+  %arrayidx3 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  store i32 %add, ptr %arrayidx3, align 4
   br label %for.inc
 
 for.inc:                                          ; preds = %for.body
diff --git a/polly/test/CodeGen/annotated_alias_scopes.ll b/polly/test/CodeGen/annotated_alias_scopes.ll
index ae68af4e34335..f8d14cd34b621 100644
--- a/polly/test/CodeGen/annotated_alias_scopes.ll
+++ b/polly/test/CodeGen/annotated_alias_scopes.ll
@@ -1,14 +1,14 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -S < %s | FileCheck %s --check-prefix=SCOPES
+; RUN: opt %loadPolly -polly-codegen -S < %s | FileCheck %s --check-prefix=SCOPES
 ;
 ; Check that we create alias scopes that indicate the accesses to A, B and C cannot alias in any way.
 ;
 ; SCOPES-LABEL: polly.stmt.for.body:
-; SCOPES:      %[[BIdx:[._a-zA-Z0-9]*]] = getelementptr{{.*}} i32* %B, i64 %polly.indvar
-; SCOPES:      load i32, i32* %[[BIdx]], align 4, !alias.scope !0, !noalias !3
-; SCOPES:      %[[CIdx:[._a-zA-Z0-9]*]] = getelementptr{{.*}} float* %C, i64 %polly.indvar
-; SCOPES:      load float, float* %[[CIdx]], align 4, !alias.scope !6, !noalias !7
-; SCOPES:      %[[AIdx:[._a-zA-Z0-9]*]] = getelementptr{{.*}} i32* %A, i64 %polly.indvar
-; SCOPES:      store i32 %{{[._a-zA-Z0-9]*}}, i32* %[[AIdx]], align 4, !alias.scope !8, !noalias !9
+; SCOPES:      %[[BIdx:[._a-zA-Z0-9]*]] = getelementptr{{.*}} ptr %B, i64 %{{.*}}
+; SCOPES:      load i32, ptr %[[BIdx]], align 4, !alias.scope !0, !noalias !3
+; SCOPES:      %[[CIdx:[._a-zA-Z0-9]*]] = getelementptr{{.*}} ptr %C, i64 %{{.*}}
+; SCOPES:      load float, ptr %[[CIdx]], align 4, !alias.scope !6, !noalias !7
+; SCOPES:      %[[AIdx:[._a-zA-Z0-9]*]] = getelementptr{{.*}} ptr %A, i64 %{{.*}}
+; SCOPES:      store i32 %{{[._a-zA-Z0-9]*}}, ptr %[[AIdx]], align 4, !alias.scope !8, !noalias !9
 ;
 ; SCOPES: !0 = !{!1}
 ; SCOPES: !1 = distinct !{!1, !2, !"polly.alias.scope.MemRef_B"}
@@ -28,7 +28,7 @@
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @jd(i32* %A, i32* %B, float* %C) {
+define void @jd(ptr %A, ptr %B, ptr %C) {
 entry:
   br label %for.cond
 
@@ -38,15 +38,15 @@ for.cond:                                         ; preds = %for.inc, %entry
   br i1 %exitcond, label %for.body, label %for.end
 
 for.body:                                         ; preds = %for.cond
-  %arrayidx = getelementptr inbounds i32, i32* %B, i64 %indvars.iv
-  %tmp = load i32, i32* %arrayidx, align 4
+  %arrayidx = getelementptr inbounds i32, ptr %B, i64 %indvars.iv
+  %tmp = load i32, ptr %arrayidx, align 4
   %conv = sitofp i32 %tmp to float
-  %arrayidx2 = getelementptr inbounds float, float* %C, i64 %indvars.iv
-  %tmp1 = load float, float* %arrayidx2, align 4
+  %arrayidx2 = getelementptr inbounds float, ptr %C, i64 %indvars.iv
+  %tmp1 = load float, ptr %arrayidx2, align 4
   %add = fadd fast float %conv, %tmp1
   %conv3 = fptosi float %add to i32
-  %arrayidx5 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  store i32 %conv3, i32* %arrayidx5, align 4
+  %arrayidx5 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  store i32 %conv3, ptr %arrayidx5, align 4
   br label %for.inc
 
 for.inc:                                          ; preds = %for.body
diff --git a/polly/test/CodeGen/getNumberOfIterations.ll b/polly/test/CodeGen/getNumberOfIterations.ll
deleted file mode 100644
index e59a076e2efcf..0000000000000
--- a/polly/test/CodeGen/getNumberOfIterations.ll
+++ /dev/null
@@ -1,39 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-vectorizer=polly -polly-codegen \
-; RUN:      < %s -S | FileCheck %s
-
-; #pragma known-parallel
-; for (int c0 = 0; c0 <= min(15, N - 1); c0 += 1)
-;   Stmt_if_then(c0);
-
-; CHECK: polly.stmt.if.then:                               ; preds = %polly.loop_header
-; CHECK:   %p_conv = sitofp i64 %polly.indvar to float
-; CHECK:   %scevgep = getelementptr float, float* %A, i64 %polly.indvar
-; CHECK:   %_p_scalar_ = load float, float* %scevgep, align 4, !alias.scope !0, !noalias !3, !llvm.access.group !4
-; CHECK:   %p_add = fadd float %p_conv, %_p_scalar_
-; CHECK:   store float %p_add, float* %scevgep, align 4, !alias.scope !0, !noalias !3, !llvm.access.group !4
-
-define void @foo(float* %A, i64 %N) #0 {
-entry:
-  br label %for.body
-
-for.body:                                         ; preds = %entry, %for.inc
-  %i.02 = phi i64 [ 0, %entry ], [ %inc, %for.inc ]
-  %cmp1 = icmp slt i64 %i.02, %N
-  br i1 %cmp1, label %if.then, label %for.inc
-
-if.then:                                          ; preds = %for.body
-  %conv = sitofp i64 %i.02 to float
-  %arrayidx = getelementptr inbounds float, float* %A, i64 %i.02
-  %0 = load float, float* %arrayidx, align 4
-  %add = fadd float %conv, %0
-  store float %add, float* %arrayidx, align 4
-  br label %for.inc
-
-for.inc:                                          ; preds = %for.body, %if.then
-  %inc = add nuw nsw i64 %i.02, 1
-  %exitcond = icmp ne i64 %inc, 16
-  br i1 %exitcond, label %for.body, label %for.end
-
-for.end:                                          ; preds = %for.inc
-  ret void
-}
diff --git a/polly/test/CodeGen/if-conditions-in-vector-code.ll b/polly/test/CodeGen/if-conditions-in-vector-code.ll
deleted file mode 100644
index 0b1897e48c492..0000000000000
--- a/polly/test/CodeGen/if-conditions-in-vector-code.ll
+++ /dev/null
@@ -1,68 +0,0 @@
-; RUN: opt %loadPolly -polly-vectorizer=polly -polly-print-ast -disable-output < %s | FileCheck %s -check-prefix=AST
-; RUN: opt %loadPolly -polly-vectorizer=polly -polly-codegen -S < %s | FileCheck %s
-;
-;    void foo(float *A) {
-;      for (long i = 0; i < 16; i++) {
-;        if (i % 2)
-;          A[i] += 2;
-;        if (i % 3)
-;          A[i] += 3;
-;      }
-;    }
-;
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-; AST: #pragma simd
-; AST: #pragma known-parallel
-; AST: for (int c0 = 0; c0 <= 15; c0 += 1) {
-; AST:   if ((c0 + 1) % 2 == 0)
-; AST:     Stmt_bb4(c0);
-; AST:   if (c0 % 3 >= 1)
-; AST:     Stmt_bb11(c0);
-; AST: }
-
-; CHECK: polly.split_new_and_old
-
-define void @foo(ptr %A) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb16, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp17, %bb16 ]
-  %exitcond = icmp ne i64 %i.0, 16
-  br i1 %exitcond, label %bb2, label %bb18
-
-bb2:                                              ; preds = %bb1
-  %tmp = srem i64 %i.0, 2
-  %tmp3 = icmp eq i64 %tmp, 0
-  br i1 %tmp3, label %bb8, label %bb4
-
-bb4:                                              ; preds = %bb2
-  %tmp5 = getelementptr inbounds float, ptr %A, i64 %i.0
-  %tmp6 = load float, ptr %tmp5, align 4
-  %tmp7 = fadd float %tmp6, 2.000000e+00
-  store float %tmp7, ptr %tmp5, align 4
-  br label %bb8
-
-bb8:                                              ; preds = %bb2, %bb4
-  %tmp9 = srem i64 %i.0, 3
-  %tmp10 = icmp eq i64 %tmp9, 0
-  br i1 %tmp10, label %bb15, label %bb11
-
-bb11:                                             ; preds = %bb8
-  %tmp12 = getelementptr inbounds float, ptr %A, i64 %i.0
-  %tmp13 = load float, ptr %tmp12, align 4
-  %tmp14 = fadd float %tmp13, 3.000000e+00
-  store float %tmp14, ptr %tmp12, align 4
-  br label %bb15
-
-bb15:                                             ; preds = %bb8, %bb11
-  br label %bb16
-
-bb16:                                             ; preds = %bb15
-  %tmp17 = add nsw i64 %i.0, 1
-  br label %bb1
-
-bb18:                                             ; preds = %bb1
-  ret void
-}
diff --git a/polly/test/CodeGen/invariant_load.ll b/polly/test/CodeGen/invariant_load.ll
index 2414fc219846b..be3f7a32f35b7 100644
--- a/polly/test/CodeGen/invariant_load.ll
+++ b/polly/test/CodeGen/invariant_load.ll
@@ -1,12 +1,13 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -polly-invariant-load-hoisting=true -S < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-codegen -polly-invariant-load-hoisting=true -S < %s | FileCheck %s
 ;
 ; CHECK-LABEL: polly.preload.begin:
-; CHECK-NEXT:    %polly.access.B = getelementptr i32, i32* %B, i64 0
-; CHECK-NEXT:    %polly.access.B.load = load i32, i32* %polly.access.B
+; CHECK-NEXT:    %polly.access.B = getelementptr i32, ptr %B, i64 0
+; CHECK-NEXT:    %polly.access.B.load = load i32, ptr %polly.access.B
 ;
 ; CHECK-LABEL: polly.stmt.bb2:
-; CHECK-NEXT:    %scevgep = getelementptr i32, i32* %A, i64 %polly.indvar
-; CHECK-NEXT:    store i32 %polly.access.B.load, i32* %scevgep, align 4
+; CHECK-NEXT:    %[[offset:.*]] = shl nuw nsw i64 %polly.indvar, 2
+; CHECK-NEXT:    %scevgep = getelementptr i8, ptr %A, i64 %[[offset]]
+; CHECK-NEXT:    store i32 %polly.access.B.load, ptr %scevgep, align 4
 ;
 ;    void f(int *restrict A, int *restrict B) {
 ;      for (int i = 0; i < 1024; i++)
@@ -15,7 +16,7 @@
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @f(i32* noalias %A, i32* noalias %B) {
+define void @f(ptr noalias %A, ptr noalias %B) {
 bb:
   br label %bb1
 
@@ -25,9 +26,9 @@ bb1:                                              ; preds = %bb4, %bb
   br i1 %exitcond, label %bb2, label %bb5
 
 bb2:                                              ; preds = %bb1
-  %tmp = load i32, i32* %B, align 4
-  %tmp3 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  store i32 %tmp, i32* %tmp3, align 4
+  %tmp = load i32, ptr %B, align 4
+  %tmp3 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  store i32 %tmp, ptr %tmp3, align 4
   br label %bb4
 
 bb4:                                              ; preds = %bb2
diff --git a/polly/test/CodeGen/invariant_load_address_space.ll b/polly/test/CodeGen/invariant_load_address_space.ll
index 4aac1aaee0035..7c611ad3dd87b 100644
--- a/polly/test/CodeGen/invariant_load_address_space.ll
+++ b/polly/test/CodeGen/invariant_load_address_space.ll
@@ -1,13 +1,14 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -polly-invariant-load-hoisting=true -S < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-codegen -polly-invariant-load-hoisting=true -S < %s | FileCheck %s
 ;
 ; CHECK-LABEL: polly.preload.begin:
-; CHECK-NEXT:    %polly.access.B = getelementptr i32, i32 addrspace(1)* %B, i64 0
+; CHECK-NEXT:    %polly.access.B = getelementptr i32, ptr addrspace(1) %B, i64 0
 ; CHECK-NOT:     addrspacecast
-; CHECK-NEXT:    %polly.access.B.load = load i32, i32 addrspace(1)* %polly.access.B
+; CHECK-NEXT:    %polly.access.B.load = load i32, ptr addrspace(1) %polly.access.B
 ;
 ; CHECK-LABEL: polly.stmt.bb2:
-; CHECK-NEXT:    %scevgep = getelementptr i32, i32* %A, i64 %polly.indvar
-; CHECK-NEXT:    store i32 %polly.access.B.load, i32* %scevgep, align 4
+; CHECK-NEXT:    %[[offset:.*]] = shl nuw nsw i64 %polly.indvar, 2
+; CHECK-NEXT:    %scevgep = getelementptr i8, ptr %A, i64 %[[offset]]
+; CHECK-NEXT:    store i32 %polly.access.B.load, ptr %scevgep, align 4
 ;
 ;    void f(int *restrict A, int *restrict B) {
 ;      for (int i = 0; i < 1024; i++)
@@ -16,7 +17,7 @@
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @f(i32* noalias %A, i32 addrspace(1)* noalias %B) {
+define void @f(ptr noalias %A, ptr addrspace(1) noalias %B) {
 bb:
   br label %bb1
 
@@ -26,9 +27,9 @@ bb1:                                              ; preds = %bb4, %bb
   br i1 %exitcond, label %bb2, label %bb5
 
 bb2:                                              ; preds = %bb1
-  %tmp = load i32, i32 addrspace(1)* %B, align 4
-  %tmp3 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  store i32 %tmp, i32* %tmp3, align 4
+  %tmp = load i32, ptr addrspace(1) %B, align 4
+  %tmp3 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  store i32 %tmp, ptr %tmp3, align 4
   br label %bb4
 
 bb4:                                              ; preds = %bb2
diff --git a/polly/test/CodeGen/invariant_load_base_pointer.ll b/polly/test/CodeGen/invariant_load_base_pointer.ll
index 951d4cd9e3b24..eb07f8317b796 100644
--- a/polly/test/CodeGen/invariant_load_base_pointer.ll
+++ b/polly/test/CodeGen/invariant_load_base_pointer.ll
@@ -1,11 +1,12 @@
-; RUN: opt -opaque-pointers=0 %loadPolly  -polly-codegen -polly-invariant-load-hoisting=true -polly-ignore-aliasing -polly-process-unprofitable -S < %s | FileCheck %s
+; RUN: opt %loadPolly  -polly-codegen -polly-invariant-load-hoisting=true -polly-ignore-aliasing -polly-process-unprofitable -S < %s | FileCheck %s
 ;
 ; CHECK-LABEL: polly.preload.begin:
-; CHECK-NEXT:    %polly.access.BPLoc = getelementptr i32*, i32** %BPLoc, i64 0
-; CHECK-NEXT:    %polly.access.BPLoc.load = load i32*, i32** %polly.access.BPLoc
+; CHECK-NEXT:    %polly.access.BPLoc = getelementptr ptr, ptr %BPLoc, i64 0
+; CHECK-NEXT:    %polly.access.BPLoc.load = load ptr, ptr %polly.access.BPLoc
 ;
 ; CHECK-LABEL: polly.stmt.bb2:
-; CHECK-NEXT:    %scevgep = getelementptr i32, i32* %polly.access.BPLoc.load, i64 %polly.indvar
+; CHECK-NEXT:    %[[offset:.*]] = shl nuw nsw i64 %polly.indvar, 2
+; CHECK-NEXT:    %scevgep = getelementptr i8, ptr %polly.access.BPLoc.load, i64 %[[offset]]
 ;
 ;    void f(int **BPLoc) {
 ;      for (int i = 0; i < 1024; i++)
@@ -14,7 +15,7 @@
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @f(i32** %BPLoc) {
+define void @f(ptr %BPLoc) {
 bb:
   br label %bb1
 
@@ -24,9 +25,9 @@ bb1:                                              ; preds = %bb4, %bb
   br i1 %exitcond, label %bb2, label %bb5
 
 bb2:                                              ; preds = %bb1
-  %tmp = load i32*, i32** %BPLoc, align 8
-  %tmp3 = getelementptr inbounds i32, i32* %tmp, i64 %indvars.iv
-  store i32 0, i32* %tmp3, align 4
+  %tmp = load ptr, ptr %BPLoc, align 8
+  %tmp3 = getelementptr inbounds i32, ptr %tmp, i64 %indvars.iv
+  store i32 0, ptr %tmp3, align 4
   br label %bb4
 
 bb4:                                              ; preds = %bb2
diff --git a/polly/test/CodeGen/invariant_load_base_pointer_conditional.ll b/polly/test/CodeGen/invariant_load_base_pointer_conditional.ll
index 472d79d26022c..538077bb09e84 100644
--- a/polly/test/CodeGen/invariant_load_base_pointer_conditional.ll
+++ b/polly/test/CodeGen/invariant_load_base_pointer_conditional.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly  -polly-codegen -polly-invariant-load-hoisting=true -polly-ignore-aliasing -polly-process-unprofitable -S < %s | FileCheck %s
+; RUN: opt %loadPolly  -polly-codegen -polly-invariant-load-hoisting=true -polly-ignore-aliasing -polly-process-unprofitable -S < %s | FileCheck %s
 ;
 ; CHECK-LABEL: polly.preload.begin:
 ; CHECK-NEXT:    %0 = sext i32 %N to i64
@@ -10,10 +10,11 @@
 ; CHECK-NEXT:    br i1 %polly.preload.cond.result, label %polly.preload.exec, label %polly.preload.merge
 ;
 ; CHECK-LABEL: polly.preload.merge:
-; CHECK-NEXT:    %polly.preload.tmp6.merge = phi i32* [ %polly.access.BPLoc.load, %polly.preload.exec ], [ null, %polly.preload.cond ]
+; CHECK-NEXT:    %polly.preload.tmp6.merge = phi ptr [ %polly.access.BPLoc.load, %polly.preload.exec ], [ null, %polly.preload.cond ]
 ;
 ; CHECK-LABEL: polly.stmt.bb5:
-; CHECK-NEXT:    %scevgep9 = getelementptr i32, i32* %polly.preload.tmp6.merge, i64 %polly.indvar6
+; CHECK-NEXT:    %[[offset:.*]] = shl nuw nsw i64 %polly.indvar6, 2
+; CHECK-NEXT:    %{{.*}} = getelementptr i8, ptr %polly.preload.tmp6.merge, i64 %[[offset]]
 ;
 ;    void f(int **BPLoc, int *A, int N) {
 ;      for (int i = 0; i < N; i++)
@@ -25,7 +26,7 @@
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @f(i32** %BPLoc, i32* %A, i32 %N) {
+define void @f(ptr %BPLoc, ptr %A, i32 %N) {
 bb:
   %tmp = sext i32 %N to i64
   br label %bb1
@@ -40,14 +41,14 @@ bb3:                                              ; preds = %bb1
   br i1 %tmp4, label %bb5, label %bb8
 
 bb5:                                              ; preds = %bb3
-  %tmp6 = load i32*, i32** %BPLoc, align 8
-  %tmp7 = getelementptr inbounds i32, i32* %tmp6, i64 %indvars.iv
-  store i32 0, i32* %tmp7, align 4
+  %tmp6 = load ptr, ptr %BPLoc, align 8
+  %tmp7 = getelementptr inbounds i32, ptr %tmp6, i64 %indvars.iv
+  store i32 0, ptr %tmp7, align 4
   br label %bb10
 
 bb8:                                              ; preds = %bb3
-  %tmp9 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  store i32 0, i32* %tmp9, align 4
+  %tmp9 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  store i32 0, ptr %tmp9, align 4
   br label %bb10
 
 bb10:                                             ; preds = %bb8, %bb5
diff --git a/polly/test/CodeGen/invariant_load_base_pointer_conditional_2.ll b/polly/test/CodeGen/invariant_load_base_pointer_conditional_2.ll
index e8e2c29f211e7..7c2fb3ef97eda 100644
--- a/polly/test/CodeGen/invariant_load_base_pointer_conditional_2.ll
+++ b/polly/test/CodeGen/invariant_load_base_pointer_conditional_2.ll
@@ -1,6 +1,6 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-invariant-load-hoisting=true -polly-print-scops -disable-output < %s | FileCheck %s
-; RUN: opt -opaque-pointers=0 %loadPolly -S -polly-codegen -polly-invariant-load-hoisting=true < %s | FileCheck %s --check-prefix=IR
-; RUN: opt -opaque-pointers=0 %loadPolly -S -polly-codegen -polly-invariant-load-hoisting=true --polly-overflow-tracking=always < %s | FileCheck %s --check-prefix=IRA
+; RUN: opt %loadPolly -polly-invariant-load-hoisting=true -polly-print-scops -disable-output < %s | FileCheck %s
+; RUN: opt %loadPolly -S -polly-codegen -polly-invariant-load-hoisting=true < %s | FileCheck %s --check-prefix=IR
+; RUN: opt %loadPolly -S -polly-codegen -polly-invariant-load-hoisting=true --polly-overflow-tracking=always < %s | FileCheck %s --check-prefix=IRA
 ;
 ; As (p + q) can overflow we have to check that we load from
 ; I[p + q] only if it does not.
@@ -15,8 +15,8 @@
 ; CHECK-NEXT:    }
 ;
 ; IR:      polly.preload.merge:
-; IR-NEXT:   %polly.preload.tmp1.merge = phi i32* [ %polly.access.I.load, %polly.preload.exec ], [ null, %polly.preload.cond ]
-; IR-NEXT:   store i32* %polly.preload.tmp1.merge, i32** %tmp1.preload.s2a
+; IR-NEXT:   %polly.preload.tmp1.merge = phi ptr [ %polly.access.I.load, %polly.preload.exec ], [ null, %polly.preload.cond ]
+; IR-NEXT:   store ptr %polly.preload.tmp1.merge, ptr %tmp1.preload.s2a
 ; IR-NEXT:   %12 = sext i32 %N to i64
 ; IR-NEXT:   %13 = icmp sge i64 %12, 1
 ; IR-NEXT:   %14 = sext i32 %q to i64
@@ -43,12 +43,12 @@
 ; IR-NEXT:   br i1 %polly.preload.cond.result11
 ;
 ; IR:      polly.preload.exec14:
-; IR-NEXT:   %polly.access.polly.preload.tmp1.merge = getelementptr i32, i32* %polly.preload.tmp1.merge, i64 0
-; IR-NEXT:   %polly.access.polly.preload.tmp1.merge.load = load i32, i32* %polly.access.polly.preload.tmp1.merge, align 4
+; IR-NEXT:   %polly.access.polly.preload.tmp1.merge = getelementptr i32, ptr %polly.preload.tmp1.merge, i64 0
+; IR-NEXT:   %polly.access.polly.preload.tmp1.merge.load = load i32, ptr %polly.access.polly.preload.tmp1.merge, align 4
 ;
 ; IRA:      polly.preload.merge:
-; IRA-NEXT:   %polly.preload.tmp1.merge = phi i32* [ %polly.access.I.load, %polly.preload.exec ], [ null, %polly.preload.cond ]
-; IRA-NEXT:   store i32* %polly.preload.tmp1.merge, i32** %tmp1.preload.s2a
+; IRA-NEXT:   %polly.preload.tmp1.merge = phi ptr [ %polly.access.I.load, %polly.preload.exec ], [ null, %polly.preload.cond ]
+; IRA-NEXT:   store ptr %polly.preload.tmp1.merge, ptr %tmp1.preload.s2a
 ; IRA-NEXT:   %12 = sext i32 %N to i64
 ; IRA-NEXT:   %13 = icmp sge i64 %12, 1
 ; IRA-NEXT:   %14 = sext i32 %q to i64
@@ -73,8 +73,8 @@
 ; IRA-NEXT:   br i1 %polly.preload.cond.result10
 ;
 ; IRA:      polly.preload.exec13:
-; IRA-NEXT:   %polly.access.polly.preload.tmp1.merge = getelementptr i32, i32* %polly.preload.tmp1.merge, i64 0
-; IRA-NEXT:   %polly.access.polly.preload.tmp1.merge.load = load i32, i32* %polly.access.polly.preload.tmp1.merge, align 4
+; IRA-NEXT:   %polly.access.polly.preload.tmp1.merge = getelementptr i32, ptr %polly.preload.tmp1.merge, i64 0
+; IRA-NEXT:   %polly.access.polly.preload.tmp1.merge.load = load i32, ptr %polly.access.polly.preload.tmp1.merge, align 4
 ;
 ;    void f(int **I, int *A, int N, int p, int q) {
 ;      for (int i = 0; i < N; i++)
@@ -83,7 +83,7 @@
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @f(i32** %I, i32* %A, i32 %N, i32 %p, i32 %q) {
+define void @f(ptr %I, ptr %A, i32 %N, i32 %p, i32 %q) {
 entry:
   %tmp = sext i32 %N to i64
   br label %for.cond
@@ -96,11 +96,11 @@ for.cond:                                         ; preds = %for.inc, %entry
 for.body:                                         ; preds = %for.cond
   %add = add i32 %p, %q
   %idxprom = sext i32 %add to i64
-  %arrayidx = getelementptr inbounds i32*, i32** %I, i64 %idxprom
-  %tmp1 = load i32*, i32** %arrayidx, align 8
-  %tmp2 = load i32, i32* %tmp1, align 4
-  %arrayidx2 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  store i32 %tmp2, i32* %arrayidx2, align 4
+  %arrayidx = getelementptr inbounds ptr, ptr %I, i64 %idxprom
+  %tmp1 = load ptr, ptr %arrayidx, align 8
+  %tmp2 = load i32, ptr %tmp1, align 4
+  %arrayidx2 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  store i32 %tmp2, ptr %arrayidx2, align 4
   br label %for.inc
 
 for.inc:                                          ; preds = %for.body
diff --git a/polly/test/CodeGen/invariant_load_canonicalize_array_baseptrs.ll b/polly/test/CodeGen/invariant_load_canonicalize_array_baseptrs.ll
index 0609a71b41049..dc5a4c890381c 100644
--- a/polly/test/CodeGen/invariant_load_canonicalize_array_baseptrs.ll
+++ b/polly/test/CodeGen/invariant_load_canonicalize_array_baseptrs.ll
@@ -1,13 +1,13 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -S < %s \
+; RUN: opt %loadPolly -polly-codegen -S < %s \
 ; RUN:  -polly-invariant-load-hoisting \
 ; RUN:  | FileCheck %s
 
-; CHECK: %polly.access.A = getelementptr float*, float** %A, i64 0
-; CHECK: %polly.access.A.load = load float*, float** %polly.access.A
-; CHECK: store float 4.200000e+01, float* %polly.access.A.load
-; CHECK: store float 4.800000e+01, float* %polly.access.A.load
+; CHECK: %polly.access.A = getelementptr ptr, ptr %A, i64 0
+; CHECK: %polly.access.A.load = load ptr, ptr %polly.access.A
+; CHECK: store float 4.200000e+01, ptr %polly.access.A.load
+; CHECK: store float 4.800000e+01, ptr %polly.access.A.load
 
-define void @foo(float** %A) {
+define void @foo(ptr %A) {
 start:
   br label %loop
 
@@ -18,13 +18,13 @@ loop:
   br i1 %icmp, label %body1, label %exit
 
 body1:
-  %baseA = load float*, float** %A
-  store float 42.0, float* %baseA
+  %baseA = load ptr, ptr %A
+  store float 42.0, ptr %baseA
   br label %body2
 
 body2:
-  %baseB = load float*, float** %A
-  store float 48.0, float* %baseB
+  %baseB = load ptr, ptr %A
+  store float 48.0, ptr %baseB
   br label %latch
 
 latch:
diff --git a/polly/test/CodeGen/invariant_load_condition.ll b/polly/test/CodeGen/invariant_load_condition.ll
index 641765d2cc22d..edf0814d89835 100644
--- a/polly/test/CodeGen/invariant_load_condition.ll
+++ b/polly/test/CodeGen/invariant_load_condition.ll
@@ -1,9 +1,9 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-process-unprofitable -polly-codegen -polly-invariant-load-hoisting=true -S < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-process-unprofitable -polly-codegen -polly-invariant-load-hoisting=true -S < %s | FileCheck %s
 ;
 ; CHECK-LABEL: polly.preload.begin:
-; CHECK-NEXT:     %polly.access.C = getelementptr i32, i32* %C, i64 0
-; CHECK-NEXT:     %polly.access.C.load = load i32, i32* %polly.access.C
-; CHECK-NOT:      %polly.access.C.load = load i32, i32* %polly.access.C
+; CHECK-NEXT:     %polly.access.C = getelementptr i32, ptr %C, i64 0
+; CHECK-NEXT:     %polly.access.C.load = load i32, ptr %polly.access.C
+; CHECK-NOT:      %polly.access.C.load = load i32, ptr %polly.access.C
 ;
 ; CHECK-LABEL: polly.cond:
 ; CHECK-NEXT:   %[[R0:[0-9]*]] = sext i32 %polly.access.C.load to i64
@@ -23,7 +23,7 @@
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @f(i32* %A, i32* %C) {
+define void @f(ptr %A, ptr %C) {
 bb:
   br label %bb1
 
@@ -33,13 +33,13 @@ bb1:                                              ; preds = %bb7, %bb
   br i1 %exitcond, label %bb2, label %bb8
 
 bb2:                                              ; preds = %bb1
-  %tmp = load i32, i32* %C, align 4
+  %tmp = load i32, ptr %C, align 4
   %tmp3 = icmp eq i32 %tmp, 0
   br i1 %tmp3, label %bb6, label %bb4
 
 bb4:                                              ; preds = %bb2
-  %tmp5 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  store i32 0, i32* %tmp5, align 4
+  %tmp5 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  store i32 0, ptr %tmp5, align 4
   br label %bb6
 
 bb6:                                              ; preds = %bb2, %bb4
diff --git a/polly/test/CodeGen/invariant_load_escaping.ll b/polly/test/CodeGen/invariant_load_escaping.ll
index 80c0c7ff6e5c4..efccdf468a18d 100644
--- a/polly/test/CodeGen/invariant_load_escaping.ll
+++ b/polly/test/CodeGen/invariant_load_escaping.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -polly-invariant-load-hoisting=true -S < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-codegen -polly-invariant-load-hoisting=true -S < %s | FileCheck %s
 ;
 ;    int f(int *A, int *B) {
 ;      // Possible aliasing between A and B but if not then *B would be
@@ -16,9 +16,9 @@
 ;    }
 ;
 ; CHECK: polly.preload.begin:
-; CHECK:   %polly.access.B = getelementptr i32, i32* %B, i64 0
-; CHECK:   %polly.access.B.load = load i32, i32* %polly.access.B
-; CHECK:   store i32 %polly.access.B.load, i32* %tmp.preload.s2a
+; CHECK:   %polly.access.B = getelementptr i32, ptr %B, i64 0
+; CHECK:   %polly.access.B.load = load i32, ptr %polly.access.B
+; CHECK:   store i32 %polly.access.B.load, ptr %tmp.preload.s2a
 ;
 ; CHECK: polly.merge_new_and_old:
 ; CHECK:   %tmp.merge = phi i32 [ %tmp.final_reload, %polly.exiting ], [ %tmp, %do.cond ]
@@ -28,21 +28,21 @@
 ; CHECK:   ret i32 %tmp.merge
 ;
 ; CHECK: polly.loop_exit:
-; CHECK:   %tmp.final_reload = load i32, i32* %tmp.preload.s2a
+; CHECK:   %tmp.final_reload = load i32, ptr %tmp.preload.s2a
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define i32 @f(i32* %A, i32* %B) {
+define i32 @f(ptr %A, ptr %B) {
 entry:
   br label %do.body
 
 do.body:                                          ; preds = %do.cond, %entry
   %indvars.iv = phi i64 [ %indvars.iv.next, %do.cond ], [ 0, %entry ]
-  %tmp = load i32, i32* %B, align 4
-  %arrayidx = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  %tmp1 = load i32, i32* %arrayidx, align 4
+  %tmp = load i32, ptr %B, align 4
+  %arrayidx = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  %tmp1 = load i32, ptr %arrayidx, align 4
   %add = add nsw i32 %tmp1, %tmp
-  store i32 %add, i32* %arrayidx, align 4
+  store i32 %add, ptr %arrayidx, align 4
   br label %do.cond
 
 do.cond:                                          ; preds = %do.body
diff --git a/polly/test/CodeGen/invariant_load_hoist_alignment.ll b/polly/test/CodeGen/invariant_load_hoist_alignment.ll
deleted file mode 100644
index 0ae07f7d6ae0c..0000000000000
--- a/polly/test/CodeGen/invariant_load_hoist_alignment.ll
+++ /dev/null
@@ -1,31 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -basic-aa -polly-codegen -polly-vectorizer=polly -S \
-; RUN: -polly-invariant-load-hoisting=true < %s | FileCheck %s
-target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
-target triple = "x86_64-unknown-linux-gnu"
-
-@A = common global [1024 x i32] zeroinitializer, align 16
-@B = common global [1024 x i32] zeroinitializer, align 16
-
-declare i32 @foo(i32) readnone
-
-define void @force_alignment() nounwind {
-;CHECK: @force_alignment
-entry:
-  br label %body
-
-body:
-  %indvar = phi i64 [ 0, %entry ], [ %indvar_next, %body ]
-  %scevgep = getelementptr [1024 x i32], [1024 x i32]* @B, i64 0, i64 %indvar
-; CHECK: [[T2:%.load]] = load i32, i32* getelementptr inbounds ([1024 x i32], [1024 x i32]* @A, i32 0, i32 0), align 4
-; CHECK: %value_p.splatinsert = insertelement <4 x i32> poison, i32 [[T2]], i64 0
-  %value = load i32, i32* getelementptr inbounds ([1024 x i32], [1024 x i32]* @A, i64 0, i64 0), align 4
-  %result = tail call i32 @foo(i32 %value) nounwind
-  store i32 %result, i32* %scevgep, align 4
-  %indvar_next = add i64 %indvar, 1
-  %exitcond = icmp eq i64 %indvar_next, 4
-  br i1 %exitcond, label %return, label %body
-
-return:
-  ret void
-}
-
diff --git a/polly/test/CodeGen/invariant_load_ptr_ptr_noalias.ll b/polly/test/CodeGen/invariant_load_ptr_ptr_noalias.ll
index bf599acbc7de9..b4d4c55f0d9bc 100644
--- a/polly/test/CodeGen/invariant_load_ptr_ptr_noalias.ll
+++ b/polly/test/CodeGen/invariant_load_ptr_ptr_noalias.ll
@@ -1,14 +1,15 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-process-unprofitable -polly-codegen -polly-invariant-load-hoisting=true -polly-ignore-aliasing -S  < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-process-unprofitable -polly-codegen -polly-invariant-load-hoisting=true -polly-ignore-aliasing -S  < %s | FileCheck %s
 ;
 ; CHECK-LABEL: polly.preload.begin:
-; CHECK:   %polly.access.A = getelementptr i32**, i32*** %A, i64 42
-; CHECK:   %polly.access.A.load = load i32**, i32*** %polly.access.A
-; CHECK:   %polly.access.polly.access.A.load = getelementptr i32*, i32** %polly.access.A.load, i64 32
-; CHECK:   %polly.access.polly.access.A.load.load = load i32*, i32** %polly.access.polly.access.A.load
+; CHECK:   %polly.access.A = getelementptr ptr, ptr %A, i64 42
+; CHECK:   %polly.access.A.load = load ptr, ptr %polly.access.A
+; CHECK:   %polly.access.polly.access.A.load = getelementptr ptr, ptr %polly.access.A.load, i64 32
+; CHECK:   %polly.access.polly.access.A.load.load = load ptr, ptr %polly.access.polly.access.A.load
 ;
 ; CHECK: polly.stmt.bb2:
-; CHECK: %scevgep = getelementptr i32, i32* %polly.access.polly.access.A.load.load, i64 %polly.indvar
-; CHECK:   store i32 0, i32* %scevgep, align 4
+; CHECK:   %[[offset:.*]] = shl nuw nsw i64 %polly.indvar, 2
+; CHECK:   %scevgep = getelementptr i8, ptr %polly.access.polly.access.A.load.load, i64 %[[offset]]
+; CHECK:   store i32 0, ptr %scevgep, align 4
 ;
 ;    void f(int ***A) {
 ;      for (int i = 0; i < 1024; i++)
@@ -17,7 +18,7 @@
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @f(i32*** %A) {
+define void @f(ptr %A) {
 bb:
   br label %bb1
 
@@ -27,12 +28,12 @@ bb1:                                              ; preds = %bb7, %bb
   br i1 %exitcond, label %bb2, label %bb8
 
 bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds i32**, i32*** %A, i64 42
-  %tmp3 = load i32**, i32*** %tmp, align 8
-  %tmp4 = getelementptr inbounds i32*, i32** %tmp3, i64 32
-  %tmp5 = load i32*, i32** %tmp4, align 8
-  %tmp6 = getelementptr inbounds i32, i32* %tmp5, i64 %indvars.iv
-  store i32 0, i32* %tmp6, align 4
+  %tmp = getelementptr inbounds ptr, ptr %A, i64 42
+  %tmp3 = load ptr, ptr %tmp, align 8
+  %tmp4 = getelementptr inbounds ptr, ptr %tmp3, i64 32
+  %tmp5 = load ptr, ptr %tmp4, align 8
+  %tmp6 = getelementptr inbounds i32, ptr %tmp5, i64 %indvars.iv
+  store i32 0, ptr %tmp6, align 4
   br label %bb7
 
 bb7:                                              ; preds = %bb2
diff --git a/polly/test/CodeGen/invariant_load_scalar_dep.ll b/polly/test/CodeGen/invariant_load_scalar_dep.ll
index 6ab75cf546c13..05a40a4c47cc0 100644
--- a/polly/test/CodeGen/invariant_load_scalar_dep.ll
+++ b/polly/test/CodeGen/invariant_load_scalar_dep.ll
@@ -1,12 +1,13 @@
-; RUN: opt -opaque-pointers=0 %loadPolly  -polly-codegen -polly-invariant-load-hoisting=true -polly-ignore-aliasing -polly-process-unprofitable -S < %s | FileCheck %s
+; RUN: opt %loadPolly  -polly-codegen -polly-invariant-load-hoisting=true -polly-ignore-aliasing -polly-process-unprofitable -S < %s | FileCheck %s
 ;
 ; CHECK-LABEL: polly.preload.begin:
-; CHECK:    %polly.access.B = getelementptr i32, i32* %B, i64 0
-; CHECK:    %polly.access.B.load = load i32, i32* %polly.access.B
+; CHECK:    %polly.access.B = getelementptr i32, ptr %B, i64 0
+; CHECK:    %polly.access.B.load = load i32, ptr %polly.access.B
 ;
 ; CHECK-LABEL: polly.stmt.bb2.split:
-; CHECK:    %scevgep = getelementptr i32, i32* %A, i64 %polly.indvar
-; CHECK:    store i32 %polly.access.B.load, i32* %scevgep, align 4
+; CHECK-NEXT:    %[[offset:.*]] = shl nuw nsw i64 %polly.indvar, 2
+; CHECK-NEXT:    %scevgep = getelementptr i8, ptr %A, i64 %[[offset]]
+; CHECK:    store i32 %polly.access.B.load, ptr %scevgep, align 4
 ;
 ;    void f(int *restrict A, int *restrict B) {
 ;      for (int i = 0; i < 1024; i++)
@@ -17,7 +18,7 @@
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @f(i32* noalias %A, i32* noalias %B) {
+define void @f(ptr noalias %A, ptr noalias %B) {
 bb:
   br label %bb1
 
@@ -27,12 +28,12 @@ bb1:                                              ; preds = %bb4, %bb
   br i1 %exitcond, label %bb2, label %bb5
 
 bb2:                                              ; preds = %bb1
-  %tmp = load i32, i32* %B, align 4
+  %tmp = load i32, ptr %B, align 4
   br label %bb2.split
 
 bb2.split:
-  %tmp3 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  store i32 %tmp, i32* %tmp3, align 4
+  %tmp3 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  store i32 %tmp, ptr %tmp3, align 4
   br label %bb4
 
 bb4:                                              ; preds = %bb2
diff --git a/polly/test/CodeGen/load_subset_with_context.ll b/polly/test/CodeGen/load_subset_with_context.ll
index af8ba488970c5..ef0e051d5635f 100644
--- a/polly/test/CodeGen/load_subset_with_context.ll
+++ b/polly/test/CodeGen/load_subset_with_context.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -S < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -S < %s | FileCheck %s
 ;
 ; A load must provide a value for every statement instance.
 ; Statement instances not in the SCoP's context are irrelevant.
@@ -21,7 +21,7 @@ for.body:                                         ; preds = %for.cond33.preheade
   br label %for.cond7.preheader
 
 for.cond33.preheader:                             ; preds = %for.inc.3
-  %tmp175 = load float, float* undef, align 4
+  %tmp175 = load float, ptr undef, align 4
   %indvars.iv.next1000 = add nuw nsw i64 %indvars.iv999, 1
   %exitcond1002 = icmp eq i64 %indvars.iv.next1000, 17
   br i1 %exitcond1002, label %for.cond176.preheader, label %for.body
@@ -29,8 +29,8 @@ for.cond33.preheader:                             ; preds = %for.inc.3
 for.cond7.preheader:                              ; preds = %for.inc.3, %for.body
   %indvars.iv958 = phi i64 [ 0, %for.body ], [ %indvars.iv.next959, %for.inc.3 ]
   %tmp20 = add nuw nsw i64 %indvars.iv958, %tmp5
-  %arrayidx.2 = getelementptr inbounds [88 x float], [88 x float]* @ATH, i64 0, i64 0
-  %tmp157 = load float, float* %arrayidx.2, align 4
+  %arrayidx.2 = getelementptr inbounds [88 x float], ptr @ATH, i64 0, i64 0
+  %tmp157 = load float, ptr %arrayidx.2, align 4
   %tmp158 = add nuw nsw i64 %tmp20, 3
   %cmp12.3 = icmp ult i64 %tmp158, 88
   br i1 %cmp12.3, label %if.then.3, label %if.else.3
@@ -43,8 +43,8 @@ if.then.3:                                        ; preds = %for.cond7.preheader
 
 for.inc.3:                                        ; preds = %if.then.3, %if.else.3
   %min.1.3 = phi float [ undef, %if.then.3 ], [ %tmp157, %if.else.3 ]
-  %arrayidx29 = getelementptr inbounds [56 x float], [56 x float]* %ath, i64 0, i64 %indvars.iv958
-  store float %min.1.3, float* %arrayidx29, align 4
+  %arrayidx29 = getelementptr inbounds [56 x float], ptr %ath, i64 0, i64 %indvars.iv958
+  store float %min.1.3, ptr %arrayidx29, align 4
   %indvars.iv.next959 = add nuw nsw i64 %indvars.iv958, 1
   %exitcond961 = icmp eq i64 %indvars.iv.next959, 56
   br i1 %exitcond961, label %for.cond33.preheader, label %for.cond7.preheader
@@ -52,6 +52,5 @@ for.inc.3:                                        ; preds = %if.then.3, %if.else
 
 
 ; CHECK:      polly.stmt.if.else.3:
-; CHECK-NEXT:   %polly.access.cast.ath1 = bitcast [56 x float]* %ath to float*
-; CHECK-NEXT:   %polly.access.ath2 = getelementptr float, float* %polly.access.cast.ath1, i64 %polly.indvar
-; CHECK-NEXT:   %polly.access.ath2.reload = load float, float* %polly.access.ath2
+; CHECK-NEXT:   %polly.access.ath1 = getelementptr float, ptr %ath, i64 %polly.indvar
+; CHECK-NEXT:   %polly.access.ath1.reload = load float, ptr %polly.access.ath1
diff --git a/polly/test/CodeGen/loop_with_condition_nested.ll b/polly/test/CodeGen/loop_with_condition_nested.ll
index 72c5f8b4918df..24a49b47d9e67 100644
--- a/polly/test/CodeGen/loop_with_condition_nested.ll
+++ b/polly/test/CodeGen/loop_with_condition_nested.ll
@@ -1,5 +1,5 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -basic-aa -polly-print-ast -disable-output < %s | FileCheck %s
-; RUN: opt -opaque-pointers=0 %loadPolly -basic-aa -polly-codegen < %s | opt -opaque-pointers=0 -passes='print<loops>' -disable-output 2>&1 | FileCheck %s -check-prefix=LOOPS
+; RUN: opt %loadPolly -basic-aa -polly-print-ast -disable-output < %s | FileCheck %s
+; RUN: opt %loadPolly -basic-aa -polly-codegen < %s | opt -passes='print<loops>' -disable-output 2>&1 | FileCheck %s -check-prefix=LOOPS
 
 
 ;#include <string.h>
@@ -49,8 +49,8 @@
 
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
 
-@A = common global [1024 x i32] zeroinitializer, align 16 ; <[1024 x i32]*> [#uses=4]
-@B = common global [1024 x i32] zeroinitializer, align 16 ; <[1024 x i32]*> [#uses=4]
+@A = common global [1024 x i32] zeroinitializer, align 16 ; <ptr> [#uses=4]
+@B = common global [1024 x i32] zeroinitializer, align 16 ; <ptr> [#uses=4]
 
 define void @loop_with_condition() nounwind {
 bb0:
@@ -59,8 +59,8 @@ bb0:
 
 bb1:
   %indvar = phi i64 [ %indvar.next, %bb10 ], [ 0, %bb0 ] ; <i64> [#uses=5]
-  %scevgep = getelementptr [1024 x i32], [1024 x i32]* @A, i64 0, i64 %indvar ; <i32*> [#uses=2]
-  %scevgep1 = getelementptr [1024 x i32], [1024 x i32]* @B, i64 0, i64 %indvar ; <i32*> [#uses=1]
+  %scevgep = getelementptr [1024 x i32], ptr @A, i64 0, i64 %indvar ; <ptr> [#uses=2]
+  %scevgep1 = getelementptr [1024 x i32], ptr @B, i64 0, i64 %indvar ; <ptr> [#uses=1]
   %i.0 = trunc i64 %indvar to i32                 ; <i32> [#uses=2]
   %exitcond = icmp ne i64 %indvar, 1024           ; <i1> [#uses=1]
   br i1 %exitcond, label %bb2, label %bb11
@@ -74,18 +74,18 @@ bb4:
   br i1 %var5, label %bb6, label %bb7
 
 bb6:
-  store i32 1, i32* %scevgep
+  store i32 1, ptr %scevgep
   br label %bb8
 
 bb7:
-  store i32 2, i32* %scevgep
+  store i32 2, ptr %scevgep
   br label %bb8
 
 bb8:
   br label %bb9
 
 bb9:
-  store i32 3, i32* %scevgep1
+  store i32 3, ptr %scevgep1
   br label %bb10
 
 bb10:
@@ -99,20 +99,20 @@ bb11:
 
 define i32 @main() nounwind {
 ; <label>:0
-  call void @llvm.memset.p0i8.i64(i8* bitcast ([1024 x i32]* @A to i8*), i8 0, i64 4096, i32 1, i1 false)
-  call void @llvm.memset.p0i8.i64(i8* bitcast ([1024 x i32]* @B to i8*), i8 0, i64 4096, i32 1, i1 false)
+  call void @llvm.memset.p0.i64(ptr @A, i8 0, i64 4096, i32 1, i1 false)
+  call void @llvm.memset.p0.i64(ptr @B, i8 0, i64 4096, i32 1, i1 false)
   call void @loop_with_condition()
   br label %1
 
 ; <label>:1                                       ; preds = %8, %0
   %indvar1 = phi i64 [ %indvar.next2, %8 ], [ 0, %0 ] ; <i64> [#uses=3]
-  %scevgep3 = getelementptr [1024 x i32], [1024 x i32]* @B, i64 0, i64 %indvar1 ; <i32*> [#uses=1]
+  %scevgep3 = getelementptr [1024 x i32], ptr @B, i64 0, i64 %indvar1 ; <ptr> [#uses=1]
   %i.0 = trunc i64 %indvar1 to i32                ; <i32> [#uses=1]
   %2 = icmp slt i32 %i.0, 1024                    ; <i1> [#uses=1]
   br i1 %2, label %3, label %9
 
 ; <label>:3                                       ; preds = %1
-  %4 = load i32, i32* %scevgep3                        ; <i32> [#uses=1]
+  %4 = load i32, ptr %scevgep3                        ; <i32> [#uses=1]
   %5 = icmp ne i32 %4, 3                          ; <i1> [#uses=1]
   br i1 %5, label %6, label %7
 
@@ -131,7 +131,7 @@ define i32 @main() nounwind {
 
 ; <label>:10                                      ; preds = %37, %9
   %indvar = phi i64 [ %indvar.next, %37 ], [ 0, %9 ] ; <i64> [#uses=3]
-  %scevgep = getelementptr [1024 x i32], [1024 x i32]* @A, i64 0, i64 %indvar ; <i32*> [#uses=3]
+  %scevgep = getelementptr [1024 x i32], ptr @A, i64 0, i64 %indvar ; <ptr> [#uses=3]
   %i.1 = trunc i64 %indvar to i32                 ; <i32> [#uses=6]
   %11 = icmp slt i32 %i.1, 1024                   ; <i1> [#uses=1]
   br i1 %11, label %12, label %38
@@ -145,7 +145,7 @@ define i32 @main() nounwind {
   br i1 %15, label %16, label %20
 
 ; <label>:16                                      ; preds = %14
-  %17 = load i32, i32* %scevgep                        ; <i32> [#uses=1]
+  %17 = load i32, ptr %scevgep                        ; <i32> [#uses=1]
   %18 = icmp ne i32 %17, 1                        ; <i1> [#uses=1]
   br i1 %18, label %19, label %20
 
@@ -161,7 +161,7 @@ define i32 @main() nounwind {
   br i1 %23, label %24, label %28
 
 ; <label>:24                                      ; preds = %22
-  %25 = load i32, i32* %scevgep                        ; <i32> [#uses=1]
+  %25 = load i32, ptr %scevgep                        ; <i32> [#uses=1]
   %26 = icmp ne i32 %25, 2                        ; <i1> [#uses=1]
   br i1 %26, label %27, label %28
 
@@ -173,7 +173,7 @@ define i32 @main() nounwind {
   br i1 %29, label %30, label %34
 
 ; <label>:30                                      ; preds = %28
-  %31 = load i32, i32* %scevgep                        ; <i32> [#uses=1]
+  %31 = load i32, ptr %scevgep                        ; <i32> [#uses=1]
   %32 = icmp ne i32 %31, 0                        ; <i1> [#uses=1]
   br i1 %32, label %33, label %34
 
@@ -201,7 +201,7 @@ define i32 @main() nounwind {
   ret i32 %.0
 }
 
-declare void @llvm.memset.p0i8.i64(i8* nocapture, i8, i64, i32, i1) nounwind
+declare void @llvm.memset.p0.i64(ptr nocapture, i8, i64, i32, i1) nounwind
 
 ; CHECK: for (int c0 = 0; c0 <= 1023; c0 += 1) {
 ; CHECK:   if (c0 <= 20) {
diff --git a/polly/test/CodeGen/multiple-types-invariant-load.ll b/polly/test/CodeGen/multiple-types-invariant-load.ll
index e2a2d6aea891b..b1434679e3d10 100644
--- a/polly/test/CodeGen/multiple-types-invariant-load.ll
+++ b/polly/test/CodeGen/multiple-types-invariant-load.ll
@@ -1,29 +1,28 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-allow-differing-element-types -polly-codegen -S \
+; RUN: opt %loadPolly -polly-allow-differing-element-types -polly-codegen -S \
 ; RUN: -polly-invariant-load-hoisting=true < %s | FileCheck %s
 
-; CHECK: %polly.access.cast.global.load = bitcast %struct.hoge* %global.load to i32*
-; CHECK: %polly.access.global.load = getelementptr i32, i32* %polly.access.cast.global.load, i64 0
-; CHECK: %polly.access.global.load.load = load i32, i32* %polly.access.global.load
+; CHECK: %polly.access.global.load = getelementptr i32, ptr %global.load, i64 0
+; CHECK: %polly.access.global.load.load = load i32, ptr %polly.access.global.load
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 %struct.hoge = type { i32, double }
 
-@global = external global %struct.hoge*, align 8
+@global = external global ptr, align 8
 
 ; Function Attrs: nounwind uwtable
-define void @widget(double* %A) #0 {
+define void @widget(ptr %A) #0 {
 bb:
   br label %bb4
 
 bb4:
-  %tmp = load %struct.hoge*, %struct.hoge** @global
-  %tmp5 = getelementptr inbounds %struct.hoge, %struct.hoge* %tmp, i64 0, i32 0
-  %tmp6 = load i32, i32* %tmp5
-  %tmp7 = getelementptr inbounds %struct.hoge, %struct.hoge* %tmp, i64 0, i32 1
-  %tmp8 = load double, double* %tmp7
-  store double %tmp8, double* %A
+  %tmp = load ptr, ptr @global
+  %tmp5 = getelementptr inbounds %struct.hoge, ptr %tmp, i64 0, i32 0
+  %tmp6 = load i32, ptr %tmp5
+  %tmp7 = getelementptr inbounds %struct.hoge, ptr %tmp, i64 0, i32 1
+  %tmp8 = load double, ptr %tmp7
+  store double %tmp8, ptr %A
   br i1 false, label %bb11, label %bb12
 
 bb11:
diff --git a/polly/test/CodeGen/no-overflow-tracking.ll b/polly/test/CodeGen/no-overflow-tracking.ll
index 120751ce29874..f11e8927ddee2 100644
--- a/polly/test/CodeGen/no-overflow-tracking.ll
+++ b/polly/test/CodeGen/no-overflow-tracking.ll
@@ -1,5 +1,5 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-invariant-load-hoisting=true -polly-print-scops -disable-output < %s | FileCheck %s
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-invariant-load-hoisting=true -polly-overflow-tracking=never -polly-codegen -S < %s | FileCheck %s --check-prefix=IR
+; RUN: opt %loadPolly -polly-invariant-load-hoisting=true -polly-print-scops -disable-output < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-invariant-load-hoisting=true -polly-overflow-tracking=never -polly-codegen -S < %s | FileCheck %s --check-prefix=IR
 ;
 ; As (p + q) can overflow we have to check that we load from
 ; I[p + q] only if it does not.
@@ -14,8 +14,8 @@
 ; CHECK-NEXT:    }
 ;
 ; IR:      polly.preload.merge:
-; IR-NEXT:   %polly.preload.tmp1.merge = phi i32* [ %polly.access.I.load, %polly.preload.exec ], [ null, %polly.preload.cond ]
-; IR-NEXT:   store i32* %polly.preload.tmp1.merge, i32** %tmp1.preload.s2a
+; IR-NEXT:   %polly.preload.tmp1.merge = phi ptr [ %polly.access.I.load, %polly.preload.exec ], [ null, %polly.preload.cond ]
+; IR-NEXT:   store ptr %polly.preload.tmp1.merge, ptr %tmp1.preload.s2a
 ; IR-NEXT:   %12 = sext i32 %N to i64
 ; IR-NEXT:   %13 = icmp sge i64 %12, 1
 ; IR-NEXT:   %14 = sext i32 %q to i64
@@ -35,8 +35,8 @@
 ; IR-NEXT:   br i1 %polly.preload.cond.result1
 ;
 ; IR:      polly.preload.exec4:
-; IR-NEXT:   %polly.access.polly.preload.tmp1.merge = getelementptr i32, i32* %polly.preload.tmp1.merge, i64 0
-; IR-NEXT:   %polly.access.polly.preload.tmp1.merge.load = load i32, i32* %polly.access.polly.preload.tmp1.merge, align 4
+; IR-NEXT:   %polly.access.polly.preload.tmp1.merge = getelementptr i32, ptr %polly.preload.tmp1.merge, i64 0
+; IR-NEXT:   %polly.access.polly.preload.tmp1.merge.load = load i32, ptr %polly.access.polly.preload.tmp1.merge, align 4
 ;
 ;    void f(int **I, int *A, int N, int p, int q) {
 ;      for (int i = 0; i < N; i++)
@@ -45,7 +45,7 @@
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @f(i32** %I, i32* %A, i32 %N, i32 %p, i32 %q) {
+define void @f(ptr %I, ptr %A, i32 %N, i32 %p, i32 %q) {
 entry:
   %tmp = sext i32 %N to i64
   br label %for.cond
@@ -58,11 +58,11 @@ for.cond:                                         ; preds = %for.inc, %entry
 for.body:                                         ; preds = %for.cond
   %add = add i32 %p, %q
   %idxprom = sext i32 %add to i64
-  %arrayidx = getelementptr inbounds i32*, i32** %I, i64 %idxprom
-  %tmp1 = load i32*, i32** %arrayidx, align 8
-  %tmp2 = load i32, i32* %tmp1, align 4
-  %arrayidx2 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  store i32 %tmp2, i32* %arrayidx2, align 4
+  %arrayidx = getelementptr inbounds ptr, ptr %I, i64 %idxprom
+  %tmp1 = load ptr, ptr %arrayidx, align 8
+  %tmp2 = load i32, ptr %tmp1, align 4
+  %arrayidx2 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  store i32 %tmp2, ptr %arrayidx2, align 4
   br label %for.inc
 
 for.inc:                                          ; preds = %for.body
diff --git a/polly/test/CodeGen/non-affine-subregion-dominance-reuse.ll b/polly/test/CodeGen/non-affine-subregion-dominance-reuse.ll
index 4600ef0602956..6c749a404336b 100644
--- a/polly/test/CodeGen/non-affine-subregion-dominance-reuse.ll
+++ b/polly/test/CodeGen/non-affine-subregion-dominance-reuse.ll
@@ -1,11 +1,11 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -S -verify-dom-info \
+; RUN: opt %loadPolly -polly-codegen -S -verify-dom-info \
 ; RUN:     < %s | FileCheck %s
 ;
 ; Check that we do not reuse the B[i-1] GEP created in block S again in
 ; block Q. Hence, we create two GEPs for B[i-1]:
 ;
-; CHECK:  %scevgep{{.}} = getelementptr i32, i32* %B, i64 -1
-; CHECK:  %scevgep{{.}} = getelementptr i32, i32* %B, i64 -1
+; CHECK:  %scevgep{{.}} = getelementptr i8, ptr %B, i64 -4
+; CHECK:  %scevgep{{.}} = getelementptr i8, ptr %B, i64 -4
 ;
 ;    void f(int *A, int *B) {
 ;      int x = 0;
@@ -20,7 +20,7 @@
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @f(i32* %A, i32* %B) {
+define void @f(ptr %A, ptr %B) {
 bb:
   br label %bb1
 
@@ -30,8 +30,8 @@ bb1:                                              ; preds = %bb22, %bb
   br i1 %exitcond, label %bb2, label %bb23
 
 bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  %tmp3 = load i32, i32* %tmp, align 4
+  %tmp = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  %tmp3 = load i32, ptr %tmp, align 4
   %tmp4 = icmp eq i32 %tmp3, 0
   br i1 %tmp4, label %bb21, label %bb5
 
@@ -44,10 +44,10 @@ bb7:                                              ; preds = %bb5
 
 bb8:                                              ; preds = %bb7
   %tmp9 = add nsw i64 %indvars.iv, -1
-  %tmp10 = getelementptr inbounds i32, i32* %B, i64 %tmp9
-  %tmp11 = load i32, i32* %tmp10, align 4
-  %tmp12 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  store i32 %tmp11, i32* %tmp12, align 4
+  %tmp10 = getelementptr inbounds i32, ptr %B, i64 %tmp9
+  %tmp11 = load i32, ptr %tmp10, align 4
+  %tmp12 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  store i32 %tmp11, ptr %tmp12, align 4
   br label %bb13
 
 bb13:                                             ; preds = %bb8, %bb5
@@ -55,12 +55,12 @@ bb13:                                             ; preds = %bb8, %bb5
 
 bb14:                                             ; preds = %bb13
   %tmp15 = add nsw i64 %indvars.iv, -1
-  %tmp16 = getelementptr inbounds i32, i32* %B, i64 %tmp15
-  %tmp17 = load i32, i32* %tmp16, align 4
-  %tmp18 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  %tmp19 = load i32, i32* %tmp18, align 4
+  %tmp16 = getelementptr inbounds i32, ptr %B, i64 %tmp15
+  %tmp17 = load i32, ptr %tmp16, align 4
+  %tmp18 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  %tmp19 = load i32, ptr %tmp18, align 4
   %tmp20 = add nsw i32 %tmp19, %tmp17
-  store i32 %tmp20, i32* %tmp18, align 4
+  store i32 %tmp20, ptr %tmp18, align 4
   br label %bb21
 
 bb21:                                             ; preds = %bb2, %bb14
diff --git a/polly/test/CodeGen/non-affine-switch.ll b/polly/test/CodeGen/non-affine-switch.ll
index 11f88a62975c4..9c08b98700ae4 100644
--- a/polly/test/CodeGen/non-affine-switch.ll
+++ b/polly/test/CodeGen/non-affine-switch.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly \
+; RUN: opt %loadPolly \
 ; RUN: -S -polly-codegen < %s | FileCheck %s
 ;
 ;    void f(int *A, int N) {
@@ -14,8 +14,9 @@
 ;    }
 ;
 ; CHECK: polly.stmt.for.body:
-; CHECK:   %scevgep = getelementptr i32, i32* %A, i64 %polly.indvar
-; CHECK:   %tmp1_p_scalar_ = load i32, i32* %scevgep, align 4
+; CHECK:   %[[offset:.*]] = shl nuw nsw i64 %polly.indvar, 2
+; CHECK:   %scevgep = getelementptr i8, ptr %A, i64 %[[offset]]
+; CHECK:   %tmp1_p_scalar_ = load i32, ptr %scevgep, align 4
 ; CHECK:   switch i32 %tmp1_p_scalar_, label %polly.stmt.sw.epilog.exit [
 ; CHECK:     i32 0, label %polly.stmt.sw.bb
 ; CHECK:     i32 1, label %polly.stmt.sw.bb.3
@@ -23,7 +24,7 @@
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @f(i32* %A, i32 %N) {
+define void @f(ptr %A, i32 %N) {
 entry:
   %tmp = sext i32 %N to i64
   br label %for.cond
@@ -34,25 +35,25 @@ for.cond:                                         ; preds = %for.inc, %entry
   br i1 %cmp, label %for.body, label %for.end
 
 for.body:                                         ; preds = %for.cond
-  %arrayidx = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  %tmp1 = load i32, i32* %arrayidx, align 4
+  %arrayidx = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  %tmp1 = load i32, ptr %arrayidx, align 4
   switch i32 %tmp1, label %sw.epilog [
     i32 0, label %sw.bb
     i32 1, label %sw.bb.3
   ]
 
 sw.bb:                                            ; preds = %for.body
-  %arrayidx2 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  %tmp2 = load i32, i32* %arrayidx2, align 4
+  %arrayidx2 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  %tmp2 = load i32, ptr %arrayidx2, align 4
   %add = add nsw i32 %tmp2, 1
-  store i32 %add, i32* %arrayidx2, align 4
+  store i32 %add, ptr %arrayidx2, align 4
   br label %sw.epilog
 
 sw.bb.3:                                          ; preds = %for.body
-  %arrayidx5 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  %tmp3 = load i32, i32* %arrayidx5, align 4
+  %arrayidx5 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  %tmp3 = load i32, ptr %arrayidx5, align 4
   %add6 = add nsw i32 %tmp3, 2
-  store i32 %add6, i32* %arrayidx5, align 4
+  store i32 %add6, ptr %arrayidx5, align 4
   br label %sw.epilog
 
 sw.epilog:                                        ; preds = %sw.bb.3, %sw.bb, %for.body
diff --git a/polly/test/CodeGen/non-affine-synthesized-in-branch.ll b/polly/test/CodeGen/non-affine-synthesized-in-branch.ll
index e6fce90542480..cc0e60abcd09e 100644
--- a/polly/test/CodeGen/non-affine-synthesized-in-branch.ll
+++ b/polly/test/CodeGen/non-affine-synthesized-in-branch.ll
@@ -1,30 +1,30 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-process-unprofitable -polly-codegen -S < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-process-unprofitable -polly-codegen -S < %s | FileCheck %s
 ;
 ; llvm.org/PR25412
 ; %synthgep caused %gep to be synthesized in subregion_if which was reused for
 ; %retval in subregion_exit, even though it is not dominating subregion_exit.
 ;
 ; CHECK-LABEL: polly.stmt.polly.merge_new_and_old.exit:
-; CHECK:         %scevgep[[R1:[0-9]*]] = getelementptr %struct.hoge, %struct.hoge* %arg, i64 0, i32 2
-; CHECK:         store double* %scevgep[[R1]], double** %gep.s2a
+; CHECK:         %scevgep[[R1:[0-9]*]] = getelementptr i8, ptr %arg, i64 16
+; CHECK:         store ptr %scevgep[[R1]], ptr %gep.s2a
 ; CHECK:         br label
 
 %struct.hoge = type { double, double, double }
 
-define double @func(%struct.hoge* %arg) {
+define double @func(ptr %arg) {
 entry:
   br label %subregion_entry
 
 subregion_entry:
-  %gep = getelementptr inbounds %struct.hoge, %struct.hoge* %arg, i64 0, i32 2
+  %gep = getelementptr inbounds %struct.hoge, ptr %arg, i64 0, i32 2
   %cond = fcmp ogt double undef, undef
   br i1 %cond, label %subregion_if, label %subregion_exit
 
 subregion_if:
-  %synthgep = load double, double* %gep
+  %synthgep = load double, ptr %gep
   br label %subregion_exit
 
 subregion_exit:
-  %retval = load double, double* %gep
+  %retval = load double, ptr %gep
   ret double %retval
 }
diff --git a/polly/test/CodeGen/non-affine-update.ll b/polly/test/CodeGen/non-affine-update.ll
index 4c9007259dc37..d2b7fae75b23e 100644
--- a/polly/test/CodeGen/non-affine-update.ll
+++ b/polly/test/CodeGen/non-affine-update.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-import-jscop \
+; RUN: opt %loadPolly -polly-import-jscop \
 ; RUN:     -polly-codegen -S < %s | FileCheck %s
 ;
 ;    void non-affine-update(double A[], double C[], double B[]) {
@@ -15,20 +15,21 @@
 ; unique within non-affine scop statements.
 
 ; CHECK: polly.stmt.bb2:
-; CHECK:   %scevgep = getelementptr double, double* %A, i64 %polly.indvar
+; CHECK:   %[[offset:.*]] = shl nuw nsw i64 %polly.indvar, 3
+; CHECK:   %scevgep = getelementptr i8, ptr %A, i64 %[[offset]]
 
 ; CHECK: polly.stmt.bb9:
-; CHECK:   %polly.access.C{{.*}} = getelementptr double, double* %C, i64 42
-; CHECK:   %polly.access.C{{.*}} = getelementptr double, double* %C, i64 42
+; CHECK:   %polly.access.C{{.*}} = getelementptr double, ptr %C, i64 42
+; CHECK:   %polly.access.C{{.*}} = getelementptr double, ptr %C, i64 42
 
 ; CHECK: polly.stmt.bb5:
-; CHECK:   %polly.access.B{{.*}} = getelementptr double, double* %B, i64 113
-; CHECK:   %polly.access.B{{.*}} = getelementptr double, double* %B, i64 113
+; CHECK:   %polly.access.B{{.*}} = getelementptr double, ptr %B, i64 113
+; CHECK:   %polly.access.B{{.*}} = getelementptr double, ptr %B, i64 113
 
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @non-affine-update(double* %A, double* %C, double* %B) {
+define void @non-affine-update(ptr %A, ptr %C, ptr %B) {
 bb:
   br label %bb1
 
@@ -38,23 +39,23 @@ bb1:                                              ; preds = %bb14, %bb
   br i1 %exitcond, label %bb2, label %bb15
 
 bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds double, double* %A, i64 %indvars.iv
-  %tmp3 = load double, double* %tmp, align 8
+  %tmp = getelementptr inbounds double, ptr %A, i64 %indvars.iv
+  %tmp3 = load double, ptr %tmp, align 8
   %tmp4 = fcmp ult double %tmp3, 6.000000e+00
   br i1 %tmp4, label %bb9, label %bb5
 
 bb5:                                              ; preds = %bb2
-  %tmp6 = getelementptr inbounds double, double* %B, i64 %indvars.iv
-  %tmp7 = load double, double* %tmp6, align 8
+  %tmp6 = getelementptr inbounds double, ptr %B, i64 %indvars.iv
+  %tmp7 = load double, ptr %tmp6, align 8
   %tmp8 = fadd double %tmp7, 4.200000e+01
-  store double %tmp8, double* %tmp6, align 8
+  store double %tmp8, ptr %tmp6, align 8
   br label %bb13
 
 bb9:                                              ; preds = %bb2
-  %tmp10 = getelementptr inbounds double, double* %C, i64 %indvars.iv
-  %tmp11 = load double, double* %tmp10, align 8
+  %tmp10 = getelementptr inbounds double, ptr %C, i64 %indvars.iv
+  %tmp11 = load double, ptr %tmp10, align 8
   %tmp12 = fadd double %tmp11, 3.000000e+00
-  store double %tmp12, double* %tmp10, align 8
+  store double %tmp12, ptr %tmp10, align 8
   br label %bb13
 
 bb13:                                             ; preds = %bb9, %bb5
diff --git a/polly/test/CodeGen/non_affine_float_compare.ll b/polly/test/CodeGen/non_affine_float_compare.ll
index a4e1a5b556a9f..be310b5bf5ca6 100644
--- a/polly/test/CodeGen/non_affine_float_compare.ll
+++ b/polly/test/CodeGen/non_affine_float_compare.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen \
+; RUN: opt %loadPolly -polly-codegen \
 ; RUN:     -polly-allow-nonaffine-branches -S -verify-dom-info \
 ; RUN:     < %s | FileCheck %s
 ;
@@ -11,35 +11,39 @@
 ;
 ;
 ; CHECK: polly.stmt.bb2:
-; CHECK:   %scevgep[[R0:[0-9]*]] = getelementptr float, float* %A, i64 %polly.indvar
-; CHECK:   %tmp3_p_scalar_ = load float, float* %scevgep[[R0]], align 4, !alias.scope !0, !noalias !3
-; CHECK:   %scevgep[[R2:[0-9]*]] = getelementptr float, float* %scevgep{{[0-9]*}}, i64 %polly.indvar
-; CHECK:   %tmp6_p_scalar_ = load float, float* %scevgep[[R2]], align 4, !alias.scope !0, !noalias !3
+; CHECK:   %[[offset:.*]] = shl nuw nsw i64 %polly.indvar, 2
+; CHECK:   %scevgep[[R0:[0-9]*]] = getelementptr i8, ptr %A, i64 %[[offset]]
+; CHECK:   %tmp3_p_scalar_ = load float, ptr %scevgep[[R0]], align 4, !alias.scope !0, !noalias !3
+; CHECK:   %[[offset2:.*]] = shl nuw nsw i64 %polly.indvar, 2
+; CHECK:   %scevgep[[R2:[0-9]*]] = getelementptr i8, ptr %scevgep{{[0-9]*}}, i64 %[[offset2]]
+; CHECK:   %tmp6_p_scalar_ = load float, ptr %scevgep[[R2]], align 4, !alias.scope !0, !noalias !3
 ; CHECK:   %p_tmp7 = fcmp oeq float %tmp3_p_scalar_, %tmp6_p_scalar_
 ; CHECK:   br i1 %p_tmp7, label %polly.stmt.bb8, label %polly.stmt.bb12.[[R:[a-zA-Z_.0-9]*]]
 
 ; CHECK: polly.stmt.bb8:
-; CHECK:   %scevgep[[R3:[0-9]*]] = getelementptr float, float* %A, i64 %polly.indvar
-; CHECK:   %tmp10_p_scalar_ = load float, float* %scevgep[[R3]], align 4, !alias.scope !0, !noalias !3
+; CHECK:   %[[offset3:.*]] = shl nuw nsw i64 %polly.indvar, 2
+; CHECK:   %scevgep[[R3:[0-9]*]] = getelementptr i8, ptr %A, i64 %[[offset3]]
+; CHECK:   %tmp10_p_scalar_ = load float, ptr %scevgep[[R3]], align 4, !alias.scope !0, !noalias !3
 ; CHECK:   %p_tmp11 = fadd float %tmp10_p_scalar_, 1.000000e+00
-; CHECK:   store float %p_tmp11, float* %scevgep[[R3]], align 4, !alias.scope !0, !noalias !3
+; CHECK:   store float %p_tmp11, ptr %scevgep[[R3]], align 4, !alias.scope !0, !noalias !3
 ; CHECK:   br label %polly.stmt.bb12.[[R]]
 
 ; CHECK: polly.stmt.bb12.[[R]]:
 ; CHECK:   br label %polly.stmt.bb12
 
 ; CHECK: polly.stmt.bb12:
-; CHECK:   %scevgep[[R4:[0-9]*]] = getelementptr float, float* %A, i64 %polly.indvar
-; CHECK:   %tmp10b_p_scalar_ = load float, float* %scevgep[[R4]], align 4, !alias.scope !0, !noalias !3
+; CHECK:   %[[offset4:.*]] = shl nuw nsw i64 %polly.indvar, 2
+; CHECK:   %scevgep[[R4:[0-9]*]] = getelementptr i8, ptr %A, i64 %[[offset4]]
+; CHECK:   %tmp10b_p_scalar_ = load float, ptr %scevgep[[R4]], align 4, !alias.scope !0, !noalias !3
 ; CHECK:   %p_tmp11b = fadd float %tmp10b_p_scalar_, 1.000000e+00
-; CHECK:   store float %p_tmp11b, float* %scevgep[[R4]], align 4, !alias.scope !0, !noalias !3
+; CHECK:   store float %p_tmp11b, ptr %scevgep[[R4]], align 4, !alias.scope !0, !noalias !3
 ; CHECK:   %polly.indvar_next = add nsw i64 %polly.indvar, 1
 ; CHECK:   %polly.loop_cond = icmp sle i64 %polly.indvar_next, 1023
 ; CHECK:   br i1 %polly.loop_cond, label %polly.loop_header, label %polly.loop_exit
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @f(float* %A) {
+define void @f(ptr %A) {
 bb:
   br label %bb1
 
@@ -49,26 +53,26 @@ bb1:                                              ; preds = %bb13, %bb
   br i1 %exitcond, label %bb2, label %bb14
 
 bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds float, float* %A, i64 %indvars.iv
-  %tmp3 = load float, float* %tmp, align 4
+  %tmp = getelementptr inbounds float, ptr %A, i64 %indvars.iv
+  %tmp3 = load float, ptr %tmp, align 4
   %tmp4 = add nsw i64 %indvars.iv, -1
-  %tmp5 = getelementptr inbounds float, float* %A, i64 %tmp4
-  %tmp6 = load float, float* %tmp5, align 4
+  %tmp5 = getelementptr inbounds float, ptr %A, i64 %tmp4
+  %tmp6 = load float, ptr %tmp5, align 4
   %tmp7 = fcmp oeq float %tmp3, %tmp6
   br i1 %tmp7, label %bb8, label %bb12
 
 bb8:                                              ; preds = %bb2
-  %tmp9 = getelementptr inbounds float, float* %A, i64 %indvars.iv
-  %tmp10 = load float, float* %tmp9, align 4
+  %tmp9 = getelementptr inbounds float, ptr %A, i64 %indvars.iv
+  %tmp10 = load float, ptr %tmp9, align 4
   %tmp11 = fadd float %tmp10, 1.000000e+00
-  store float %tmp11, float* %tmp9, align 4
+  store float %tmp11, ptr %tmp9, align 4
   br label %bb12
 
 bb12:                                             ; preds = %bb8, %bb2
-  %tmp9b = getelementptr inbounds float, float* %A, i64 %indvars.iv
-  %tmp10b = load float, float* %tmp9b, align 4
+  %tmp9b = getelementptr inbounds float, ptr %A, i64 %indvars.iv
+  %tmp10b = load float, ptr %tmp9b, align 4
   %tmp11b = fadd float %tmp10b, 1.000000e+00
-  store float %tmp11b, float* %tmp9b, align 4
+  store float %tmp11b, ptr %tmp9b, align 4
   br label %bb13
 
 bb13:                                             ; preds = %bb12
diff --git a/polly/test/CodeGen/partial_write_array.ll b/polly/test/CodeGen/partial_write_array.ll
index 6305f512a32bd..6dc5550d82afe 100644
--- a/polly/test/CodeGen/partial_write_array.ll
+++ b/polly/test/CodeGen/partial_write_array.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -S < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -S < %s | FileCheck %s
 ;
 ; Partial write of an array access.
 ;
@@ -6,7 +6,7 @@
 ;   A[0] = 42.0
 ;
 
-define void @partial_write_array(i32 %n, double* noalias nonnull %A) {
+define void @partial_write_array(i32 %n, ptr noalias nonnull %A) {
 entry:
   br label %for
 
@@ -16,7 +16,7 @@ for:
   br i1 %j.cmp, label %body, label %exit
 
     body:
-      store double 42.0, double* %A
+      store double 42.0, ptr %A
       br label %inc
 
 inc:
@@ -37,8 +37,8 @@ return:
 ; CHECK-NEXT:   br i1 %polly.Stmt_body_Write0.cond, label %polly.stmt.body.Stmt_body_Write0.partial, label %polly.stmt.body.cont
 
 ; CHECK:      polly.stmt.body.Stmt_body_Write0.partial:
-; CHECK-NEXT:   %polly.access.A = getelementptr double, double* %A, i64 0
-; CHECK-NEXT:   store double 4.200000e+01, double* %polly.access.A, align 8, !alias.scope !0, !noalias !3
+; CHECK-NEXT:   %polly.access.A = getelementptr double, ptr %A, i64 0
+; CHECK-NEXT:   store double 4.200000e+01, ptr %polly.access.A, align 8, !alias.scope !0, !noalias !3
 ; CHECK-NEXT:   br label %polly.stmt.body.cont
 
 ; CHECK:      polly.stmt.body.cont:
diff --git a/polly/test/CodeGen/partial_write_full_write_that_appears_partial.ll b/polly/test/CodeGen/partial_write_full_write_that_appears_partial.ll
index 8c1d18c6ac633..18a809b305579 100644
--- a/polly/test/CodeGen/partial_write_full_write_that_appears_partial.ll
+++ b/polly/test/CodeGen/partial_write_full_write_that_appears_partial.ll
@@ -1,9 +1,7 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -S < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-codegen -S < %s | FileCheck %s
 
 ; CHECK:      polly.stmt.if.then81:                             ; preds = %polly.stmt.if.end75
-; CHECK-NEXT:   %scevgep = getelementptr [2 x %S], [2 x %S]* %tmp, i64 0, i64 %.147
-; CHECK-NEXT:   %scevgep1 = bitcast %S* %scevgep to float*
-; CHECK-NEXT:   store float undef, float* %scevgep1, align 4, !alias.scope !0, !noalias !3
+; CHECK-NEXT:   store float undef, ptr %fX64, align 4, !alias.scope !0, !noalias !3
 ; CHECK-NEXT:   br label %polly.stmt.if.end87.region_exiting
 
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
@@ -17,7 +15,7 @@ entry:
   %cmp52 = fcmp olt float undef, undef
   %not.cmp52 = xor i1 %cmp52, true
   %.147 = zext i1 %not.cmp52 to i64
-  %fX64 = getelementptr inbounds [2 x %S], [2 x %S]* %tmp, i64 0, i64 %.147, i32 0
+  %fX64 = getelementptr inbounds [2 x %S], ptr %tmp, i64 0, i64 %.147, i32 0
   br label %if.end75
 
 if.end75:
@@ -25,7 +23,7 @@ if.end75:
   br i1 %cmp80, label %if.then81, label %if.end87
 
 if.then81:
-  store float undef, float* %fX64, align 4
+  store float undef, ptr %fX64, align 4
   br label %if.end87
 
 if.end87:
diff --git a/polly/test/CodeGen/partial_write_impossible_restriction.ll b/polly/test/CodeGen/partial_write_impossible_restriction.ll
index 58afd0b6afdb8..178227fef8e54 100644
--- a/polly/test/CodeGen/partial_write_impossible_restriction.ll
+++ b/polly/test/CodeGen/partial_write_impossible_restriction.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-stmt-granularity=bb -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -S < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-stmt-granularity=bb -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-codegen -S < %s | FileCheck %s
 ;
 ; The isl scheduler isolates %cond.false into two instances.
 ; A partial write access in one of the instances was never executed,
@@ -7,7 +7,7 @@
 ;
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
 
-define void @partial_write_impossible_restriction(i32* %.pn) {
+define void @partial_write_impossible_restriction(ptr %.pn) {
 entry:
   br i1 undef, label %invoke.cont258, label %cond.true.i.i.i.i1007
 
@@ -18,8 +18,8 @@ invoke.cont258:
   br label %invoke.cont274
 
 invoke.cont274:                                   ; preds = %invoke.cont258
-  %tmp4 = load i32*, i32** undef
-  %tmp5 = load i32, i32* undef
+  %tmp4 = load ptr, ptr undef
+  %tmp5 = load i32, ptr undef
   %tmp6 = zext i32 %tmp5 to i64
   %tmp7 = sext i32 %tmp5 to i64
   br label %for.body344
@@ -31,14 +31,14 @@ for.body344:                                      ; preds = %cond.end, %invoke.c
   br i1 %cmp347, label %cond.end, label %cond.false
 
 cond.false:                                       ; preds = %for.body344
-  %add.ptr.i1128 = getelementptr inbounds i32, i32* %tmp4, i64 %indvars.iv.next1603
-  %cond.in.sroa.speculate.load.cond.false = load i32, i32* %add.ptr.i1128
+  %add.ptr.i1128 = getelementptr inbounds i32, ptr %tmp4, i64 %indvars.iv.next1603
+  %cond.in.sroa.speculate.load.cond.false = load i32, ptr %add.ptr.i1128
   br label %cond.end
 
 cond.end:                                         ; preds = %cond.false, %for.body344
   %cond.in.sroa.speculated = phi i32 [ %cond.in.sroa.speculate.load.cond.false, %cond.false ], [ undef, %for.body344 ]
-  %add.ptr.i1132 = getelementptr inbounds i32, i32* %.pn, i64 %indvars.iv1602
-  store i32 undef, i32* %add.ptr.i1132
+  %add.ptr.i1132 = getelementptr inbounds i32, ptr %.pn, i64 %indvars.iv1602
+  store i32 undef, ptr %add.ptr.i1132
   %cmp342 = icmp slt i64 %indvars.iv.next1603, %tmp7
   br i1 %cmp342, label %for.body344, label %if.then.i.i1141.loopexit
 
@@ -48,11 +48,11 @@ if.then.i.i1141.loopexit:                         ; preds = %cond.end
 
 
 ; CHECK-LABEL: polly.stmt.cond.false:
-; CHECK:         %polly.access..pn{{[0-9]*}} = getelementptr i32, i32* %.pn, i64 %polly.indvar
-; CHECK:         store i32 %cond.in.sroa.speculate.load.cond.false_p_scalar_, i32* %polly.access..pn{{[0-9]*}}, align 4, !alias.scope !0, !noalias !3
+; CHECK:         %polly.access..pn{{[0-9]*}} = getelementptr i32, ptr %.pn, i64 %polly.indvar
+; CHECK:         store i32 %cond.in.sroa.speculate.load.cond.false_p_scalar_, ptr %polly.access..pn{{[0-9]*}}, align 4, !alias.scope !0, !noalias !3
 ; CHECK:         br label %polly.merge
 
 ; CHECK-LABEL: polly.stmt.cond.false{{[0-9]*}}:
-; CHECK:         %polly.access..pn{{[0-9]*}} = getelementptr i32, i32* %.pn, i64 0
-; CHECK:         store i32 %cond.in.sroa.speculate.load.cond.false_p_scalar_{{[0-9]*}}, i32* %polly.access..pn{{[0-9]*}}, align 4, !alias.scope !0, !noalias !3
+; CHECK:         %polly.access..pn{{[0-9]*}} = getelementptr i32, ptr %.pn, i64 0
+; CHECK:         store i32 %cond.in.sroa.speculate.load.cond.false_p_scalar_{{[0-9]*}}, ptr %polly.access..pn{{[0-9]*}}, align 4, !alias.scope !0, !noalias !3
 ; CHECK:         br label %polly.stmt.cond.end{{[0-9]*}}
diff --git a/polly/test/CodeGen/partial_write_in_region.ll b/polly/test/CodeGen/partial_write_in_region.ll
index ce587a9583d19..d8f57b35d5855 100644
--- a/polly/test/CodeGen/partial_write_in_region.ll
+++ b/polly/test/CodeGen/partial_write_in_region.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-import-jscop \
+; RUN: opt %loadPolly -polly-import-jscop \
 ; RUN: -polly-import-jscop-postfix=transformed -polly-codegen \
 ; RUN: -verify-dom-info \
 ; RUN: -S < %s | FileCheck %s
@@ -13,16 +13,17 @@
 ;    }
 
 ; CHECK: polly.stmt.bb5:                                   ; preds = %polly.stmt.bb2
-; CHECK-NEXT:   %scevgep10 = getelementptr float, float* %B, i64 %polly.indvar
-; CHECK-NEXT:   %tmp7_p_scalar_ = load float, float* %scevgep10
+; CHECK-NEXT:   %[[offset:.*]] = shl nuw nsw i64 %polly.indvar, 2
+; CHECK-NEXT:   %scevgep10 = getelementptr i8, ptr %B, i64 %[[offset]]
+; CHECK-NEXT:   %tmp7_p_scalar_ = load float, ptr %scevgep10
 ; CHECK-NEXT:   %p_tmp8 = fadd float %tmp7_p_scalar_, 1.000000e+00
-; CHECK-NEXT:   %24 = icmp sle i64 %polly.indvar, 9
-; CHECK-NEXT:   %polly.Stmt_bb2__TO__bb9_MayWrite2.cond = icmp ne i1 %24, false
+; CHECK-NEXT:   %[[cmp:.*]] = icmp sle i64 %polly.indvar, 9
+; CHECK-NEXT:   %polly.Stmt_bb2__TO__bb9_MayWrite2.cond = icmp ne i1 %[[cmp]], false
 ; CHECK-NEXT:   br i1 %polly.Stmt_bb2__TO__bb9_MayWrite2.cond, label %polly.stmt.bb5.Stmt_bb2__TO__bb9_MayWrite2.partial, label %polly.stmt.bb5.cont
 
 ; CHECK: polly.stmt.bb5.Stmt_bb2__TO__bb9_MayWrite2.partial: ; preds = %polly.stmt.bb5
-; CHECK-NEXT:   %polly.access.B11 = getelementptr float, float* %B, i64 %polly.indvar
-; CHECK-NEXT:   store float %p_tmp8, float* %polly.access.B11
+; CHECK-NEXT:   %polly.access.B11 = getelementptr float, ptr %B, i64 %polly.indvar
+; CHECK-NEXT:   store float %p_tmp8, ptr %polly.access.B11
 ; CHECK-NEXT:   br label %polly.stmt.bb5.cont
 
 ; CHECK: polly.stmt.bb5.cont:                              ; preds = %polly.stmt.bb5, %polly.stmt.bb5.Stmt_bb2__TO__bb9_MayWrite2.partial
@@ -30,7 +31,7 @@
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @partial_write_in_region(i64* %A, float* %B, float* %C) {
+define void @partial_write_in_region(ptr %A, ptr %B, ptr %C) {
 bb:
   br label %bb1
 
@@ -40,20 +41,20 @@ bb1:                                              ; preds = %bb10, %bb
   br i1 %exitcond, label %bb2, label %bb12
 
 bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds i64, i64* %A, i64 %i.0
-  %tmp3 = load i64, i64* %tmp, align 8
+  %tmp = getelementptr inbounds i64, ptr %A, i64 %i.0
+  %tmp3 = load i64, ptr %tmp, align 8
   %tmp4 = icmp eq i64 %tmp3, 0
   br i1 %tmp4, label %bb9, label %bb5
 
 bb5:                                              ; preds = %bb2
-  %tmp6 = getelementptr inbounds float, float* %B, i64 %i.0
-  %tmp7 = load float, float* %tmp6, align 4
+  %tmp6 = getelementptr inbounds float, ptr %B, i64 %i.0
+  %tmp7 = load float, ptr %tmp6, align 4
   %tmp8 = fadd float %tmp7, 1.000000e+00
-  store float %tmp8, float* %tmp6, align 4
+  store float %tmp8, ptr %tmp6, align 4
   br label %bb9b
 
 bb9b:
-  store float 42.0, float* %C
+  store float 42.0, ptr %C
   br label %bb9
 
 bb9:                                              ; preds = %bb2, %bb5
diff --git a/polly/test/CodeGen/partial_write_mapped_vector.ll b/polly/test/CodeGen/partial_write_mapped_vector.ll
deleted file mode 100644
index 27d3a74bd6ba5..0000000000000
--- a/polly/test/CodeGen/partial_write_mapped_vector.ll
+++ /dev/null
@@ -1,57 +0,0 @@
-; RUN: opt %loadPolly -basic-aa -polly-stmt-granularity=bb -polly-import-jscop -polly-import-jscop-postfix=transformed -polly-vectorizer=polly -polly-opt-isl -polly-ast -polly-codegen -S < %s | FileCheck %s
-;
-; Polly's vectorizer does not support partial accesses.
-;
-; for (int j = 0; j < 4; j += 1) {
-;body:
-;   val = 21.0 + 21.0;
-;   if (j > 1)
-;user:
-;     A[0] = val;
-; }
-
-define void @partial_write_mapped_vector(ptr noalias nonnull %A) {
-entry:
-  br label %for
-
-for:
-  %j = phi i32 [0, %entry], [%j.inc, %inc]
-  %j.cmp = icmp slt i32 %j, 4
-  br i1 %j.cmp, label %body, label %exit
-
-    body:
-      %val = fadd double 21.0, 21.0
-      %if.cond = icmp sgt i32 %j, 1
-      br i1 %if.cond, label %user, label %inc
-
-    user:
-      %elt= getelementptr inbounds double, ptr %A, i32 %j
-      store double %val, ptr %elt
-      br label %inc
-
-inc:
-  %j.inc = add nuw nsw i32 %j, 1
-  br label %for
-
-exit:
-  br label %return
-
-return:
-  ret void
-}
-
-
-; CHECK-LABEL: polly.stmt.body:
-; CHECK-NEXT:    %p_val = fadd double 2.100000e+01, 2.100000e+01
-; CHECK-NEXT:    %0 = trunc i64 %polly.indvar to i32
-; CHECK-NEXT:    %p_if.cond = icmp sgt i32 %0, 1
-; CHECK-NEXT:    %1 = icmp sge i64 %polly.indvar, 2
-; CHECK-NEXT:    %polly.Stmt_body_Write0.cond = icmp ne i1 %1, false
-; CHECK-NEXT:    br i1 %polly.Stmt_body_Write0.cond, label %polly.stmt.body.Stmt_body_Write0.partial, label %polly.stmt.body.cont
-
-; CHECK-LABEL:  polly.stmt.body.Stmt_body_Write0.partial:
-; CHECK-NEXT:    %polly.access.A = getelementptr double, ptr %A, i64 1
-; CHECK-NEXT:    store double %p_val, ptr %polly.access.A
-; CHECK-NEXT:    br label %polly.stmt.body.cont
-
-; CHECK-LABEL:  polly.stmt.body.cont:
diff --git a/polly/test/CodeGen/partial_write_mapped_vector___%for---%return.jscop b/polly/test/CodeGen/partial_write_mapped_vector___%for---%return.jscop
deleted file mode 100644
index 2aadaf12d59ad..0000000000000
--- a/polly/test/CodeGen/partial_write_mapped_vector___%for---%return.jscop
+++ /dev/null
@@ -1,39 +0,0 @@
-{
-   "arrays" : [
-      {
-         "name" : "MemRef_A",
-         "sizes" : [ "*" ],
-         "type" : "double"
-      }
-   ],
-   "context" : "{  :  }",
-   "name" : "%for---%return",
-   "statements" : [
-      {
-         "accesses" : [
-            {
-               "kind" : "write",
-               "relation" : "{ Stmt_body[i0] -> MemRef_val[] }"
-            }
-         ],
-         "domain" : "{ Stmt_body[i0] : 0 <= i0 <= 3 }",
-         "name" : "Stmt_body",
-         "schedule" : "{ Stmt_body[i0] -> [i0, 0] }"
-      },
-      {
-         "accesses" : [
-            {
-               "kind" : "write",
-               "relation" : "{ Stmt_user[i0] -> MemRef_A[i0] }"
-            },
-            {
-               "kind" : "read",
-               "relation" : "{ Stmt_user[i0] -> MemRef_val[] }"
-            }
-         ],
-         "domain" : "{ Stmt_user[i0] : 2 <= i0 <= 3 }",
-         "name" : "Stmt_user",
-         "schedule" : "{ Stmt_user[i0] -> [i0, 1] }"
-      }
-   ]
-}
diff --git a/polly/test/CodeGen/partial_write_mapped_vector___%for---%return.jscop.transformed b/polly/test/CodeGen/partial_write_mapped_vector___%for---%return.jscop.transformed
deleted file mode 100644
index 39f97f0c292ad..0000000000000
--- a/polly/test/CodeGen/partial_write_mapped_vector___%for---%return.jscop.transformed
+++ /dev/null
@@ -1,39 +0,0 @@
-{
-   "arrays" : [
-      {
-         "name" : "MemRef_A",
-         "sizes" : [ "*" ],
-         "type" : "double"
-      }
-   ],
-   "context" : "{  :  }",
-   "name" : "%for---%return",
-   "statements" : [
-      {
-         "accesses" : [
-            {
-               "kind" : "write",
-               "relation" : "{ Stmt_body[j] -> MemRef_A[1] : j > 1 }"
-            }
-         ],
-         "domain" : "{ Stmt_body[i0] : 0 <= i0 <= 3 }",
-         "name" : "Stmt_body",
-         "schedule" : "{ Stmt_body[i0] -> [i0, 0] }"
-      },
-      {
-         "accesses" : [
-            {
-               "kind" : "write",
-               "relation" : "{ Stmt_user[i0] -> MemRef_A[i0] }"
-            },
-            {
-               "kind" : "read",
-               "relation" : "{ Stmt_user[j] -> MemRef_A[1] }"
-            }
-         ],
-         "domain" : "{ Stmt_user[i0] : 2 <= i0 <= 3 }",
-         "name" : "Stmt_user",
-         "schedule" : "{ Stmt_user[i0] -> [i0, 1] }"
-      }
-   ]
-}
diff --git a/polly/test/CodeGen/phi_conditional_simple_1.ll b/polly/test/CodeGen/phi_conditional_simple_1.ll
index 9def61a44094d..f1b93b540f70c 100644
--- a/polly/test/CodeGen/phi_conditional_simple_1.ll
+++ b/polly/test/CodeGen/phi_conditional_simple_1.ll
@@ -1,5 +1,5 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-print-ast -disable-output < %s | FileCheck %s --check-prefix=AST
-; RUN: opt -opaque-pointers=0 %loadPolly -S -polly-codegen < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-print-ast -disable-output < %s | FileCheck %s --check-prefix=AST
+; RUN: opt %loadPolly -S -polly-codegen < %s | FileCheck %s
 ;
 ;    void jd(int *A, int c) {
 ;      for (int i = 0; i < 1024; i++) {
@@ -21,19 +21,19 @@
 ; CHECK-LABEL:  entry:
 ; CHECK-NEXT:     %phi.phiops = alloca i32
 ; CHECK-LABEL:  polly.stmt.if.end:
-; CHECK-NEXT:     %phi.phiops.reload = load i32, i32* %phi.phiops
-; CHECK-NEXT:     %scevgep
-; CHECK-NEXT:     store i32 %phi.phiops.reload, i32*
+; CHECK-NEXT:     %phi.phiops.reload = load i32, ptr %phi.phiops
+; CHECK:          %scevgep
+; CHECK-NEXT:     store i32 %phi.phiops.reload, ptr
 ; CHECK-LABEL:  polly.stmt.if.then:
-; CHECK-NEXT:     store i32 1, i32* %phi.phiops
+; CHECK-NEXT:     store i32 1, ptr %phi.phiops
 ; CHECK-NEXT:     br label %polly.merge{{[.]?}}
 ; CHECK-LABEL:  polly.stmt.if.else:
-; CHECK-NEXT:     store i32 2, i32* %phi.phiops
+; CHECK-NEXT:     store i32 2, ptr %phi.phiops
 ; CHECK-NEXT:     br label %polly.merge{{[.]?}}
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @jd(i32* %A, i32 %c) {
+define void @jd(ptr %A, i32 %c) {
 entry:
   br label %for.cond
 
@@ -54,8 +54,8 @@ if.else:
 
 if.end:
   %phi = phi i32 [ 1, %if.then], [ 2, %if.else ]
-  %arrayidx = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  store i32 %phi, i32* %arrayidx, align 4
+  %arrayidx = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  store i32 %phi, ptr %arrayidx, align 4
   br label %for.inc
 
 for.inc:
diff --git a/polly/test/CodeGen/phi_loop_carried_float.ll b/polly/test/CodeGen/phi_loop_carried_float.ll
index 2e437bc38fff7..ca1870fb3a091 100644
--- a/polly/test/CodeGen/phi_loop_carried_float.ll
+++ b/polly/test/CodeGen/phi_loop_carried_float.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -S  -polly-codegen < %s | FileCheck %s
+; RUN: opt %loadPolly -S  -polly-codegen < %s | FileCheck %s
 ;
 ;    float f(float *A, int N) {
 ;      float tmp = 0;
@@ -17,29 +17,29 @@
 ; CHECK-NEXT:    ret
 
 ; CHECK-LABEL: polly.start:
-; CHECK-NEXT:    store float 0.000000e+00, float* %tmp.0.phiops
+; CHECK-NEXT:    store float 0.000000e+00, ptr %tmp.0.phiops
 ; CHECK-NEXT:    sext
 
 ; CHECK-LABEL: polly.exiting:
 ; CHECK-NEXT:    br label %polly.merge_new_and_old
 
 ; CHECK-LABEL: polly.stmt.bb1{{[0-9]*}}:
-; CHECK-NEXT:    %tmp.0.phiops.reload[[R1:[0-9]*]] = load float, float* %tmp.0.phiops
-; CHECK:         store float %tmp.0.phiops.reload[[R1]], float* %tmp.0.s2a
+; CHECK-NEXT:    %tmp.0.phiops.reload[[R1:[0-9]*]] = load float, ptr %tmp.0.phiops
+; CHECK:         store float %tmp.0.phiops.reload[[R1]], ptr %tmp.0.s2a
 
 ; CHECK-LABEL: polly.stmt.bb4:
-; CHECK:         %tmp.0.s2a.reload[[R3:[0-9]*]] = load float, float* %tmp.0.s2a
-; CHECK:         %tmp[[R5:[0-9]*]]_p_scalar_ = load float, float* %scevgep, align 4, !alias.scope !0, !noalias !3
+; CHECK:         %tmp.0.s2a.reload[[R3:[0-9]*]] = load float, ptr %tmp.0.s2a
+; CHECK:         %tmp[[R5:[0-9]*]]_p_scalar_ = load float, ptr %scevgep, align 4, !alias.scope !0, !noalias !3
 ; CHECK:         %p_tmp[[R4:[0-9]*]] = fadd float %tmp.0.s2a.reload[[R3]], %tmp[[R5]]_p_scalar_
-; CHECK:         store float %p_tmp[[R4]], float* %tmp.0.phiops
+; CHECK:         store float %p_tmp[[R4]], ptr %tmp.0.phiops
 
 ; CHECK-LABEL: polly.stmt.bb1{{[0-9]*}}:
-; CHECK-NEXT:    %tmp.0.phiops.reload[[R2:[0-9]*]] = load float, float* %tmp.0.phiops
-; CHECK:         store float %tmp.0.phiops.reload[[R2]], float* %tmp.0.s2a
+; CHECK-NEXT:    %tmp.0.phiops.reload[[R2:[0-9]*]] = load float, ptr %tmp.0.phiops
+; CHECK:         store float %tmp.0.phiops.reload[[R2]], ptr %tmp.0.s2a
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @f(float* %A, i32 %N) {
+define void @f(ptr %A, i32 %N) {
 bb:
   %tmp = sext i32 %N to i64
   br label %bb1
@@ -54,8 +54,8 @@ bb3:                                              ; preds = %bb1
   br label %bb4
 
 bb4:                                              ; preds = %bb3
-  %tmp5 = getelementptr inbounds float, float* %A, i64 %indvars.iv
-  %tmp6 = load float, float* %tmp5, align 4
+  %tmp5 = getelementptr inbounds float, ptr %A, i64 %indvars.iv
+  %tmp6 = load float, ptr %tmp5, align 4
   %tmp7 = fadd float %tmp.0, %tmp6
   %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
   br label %bb1
diff --git a/polly/test/CodeGen/phi_loop_carried_float_escape.ll b/polly/test/CodeGen/phi_loop_carried_float_escape.ll
index f3e8ed453e165..3b2ed01863b18 100644
--- a/polly/test/CodeGen/phi_loop_carried_float_escape.ll
+++ b/polly/test/CodeGen/phi_loop_carried_float_escape.ll
@@ -1,7 +1,7 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -S \
+; RUN: opt %loadPolly -S \
 ; RUN: -polly-analyze-read-only-scalars=false -polly-codegen < %s | FileCheck %s
 
-; RUN: opt -opaque-pointers=0 %loadPolly -S \
+; RUN: opt %loadPolly -S \
 ; RUN: -polly-analyze-read-only-scalars=true -polly-codegen < %s | FileCheck %s
 ;
 ;    float f(float *A, int N) {
@@ -16,30 +16,30 @@
 ; CHECK-NEXT:    br label %exit
 
 ; CHECK-LABEL: polly.start:
-; CHECK-NEXT:    store float 0.000000e+00, float* %tmp.0.phiops
+; CHECK-NEXT:    store float 0.000000e+00, ptr %tmp.0.phiops
 ; CHECK-NEXT:    sext
 
 ; CHECK-LABEL: polly.exiting:
-; CHECK-NEXT:    %tmp.0.final_reload = load float, float* %tmp.0.s2a
+; CHECK-NEXT:    %tmp.0.final_reload = load float, ptr %tmp.0.s2a
 ; CHECK-NEXT:    br label %polly.merge_new_and_old
 
 ; CHECK-LABEL: polly.stmt.bb1{{[0-9]*}}:
-; CHECK-NEXT:    %tmp.0.phiops.reload[[R1:[0-9]*]] = load float, float* %tmp.0.phiops
-; CHECK-:        store float %tmp.0.phiops.reload[[R1]], float* %tmp.0.s2a
+; CHECK-NEXT:    %tmp.0.phiops.reload[[R1:[0-9]*]] = load float, ptr %tmp.0.phiops
+; CHECK-:        store float %tmp.0.phiops.reload[[R1]], ptr %tmp.0.s2a
 
 ; CHECK-LABEL: polly.stmt.bb4:
-; CHECK:         %tmp.0.s2a.reload[[R3:[0-9]*]] = load float, float* %tmp.0.s2a
-; CHECK:         %tmp[[R5:[0-9]*]]_p_scalar_ = load float, float* %scevgep, align 4, !alias.scope !0, !noalias !3
+; CHECK:         %tmp.0.s2a.reload[[R3:[0-9]*]] = load float, ptr %tmp.0.s2a
+; CHECK:         %tmp[[R5:[0-9]*]]_p_scalar_ = load float, ptr %scevgep, align 4, !alias.scope !0, !noalias !3
 ; CHECK:         %p_tmp[[R4:[0-9]*]] = fadd float %tmp.0.s2a.reload[[R3]], %tmp[[R5]]_p_scalar_
-; CHECK:         store float %p_tmp[[R4]], float* %tmp.0.phiops
+; CHECK:         store float %p_tmp[[R4]], ptr %tmp.0.phiops
 
 ; CHECK-LABEL: polly.stmt.bb1{{[0-9]*}}:
-; CHECK-NEXT:    %tmp.0.phiops.reload[[R2:[0-9]*]] = load float, float* %tmp.0.phiops
-; CHECK:         store float %tmp.0.phiops.reload[[R2]], float* %tmp.0.s2a
+; CHECK-NEXT:    %tmp.0.phiops.reload[[R2:[0-9]*]] = load float, ptr %tmp.0.phiops
+; CHECK:         store float %tmp.0.phiops.reload[[R2]], ptr %tmp.0.s2a
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define float @f(float* %A, i32 %N) {
+define float @f(ptr %A, i32 %N) {
 bb:
   %tmp = sext i32 %N to i64
   br label %bb1
@@ -54,8 +54,8 @@ bb3:                                              ; preds = %bb1
   br label %bb4
 
 bb4:                                              ; preds = %bb3
-  %tmp5 = getelementptr inbounds float, float* %A, i64 %indvars.iv
-  %tmp6 = load float, float* %tmp5, align 4
+  %tmp5 = getelementptr inbounds float, ptr %A, i64 %indvars.iv
+  %tmp6 = load float, ptr %tmp5, align 4
   %tmp7 = fadd float %tmp.0, %tmp6
   %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
   br label %bb1
diff --git a/polly/test/CodeGen/region-with-instructions.ll b/polly/test/CodeGen/region-with-instructions.ll
index 02c0dfe05fc0b..28cabefbf68b4 100644
--- a/polly/test/CodeGen/region-with-instructions.ll
+++ b/polly/test/CodeGen/region-with-instructions.ll
@@ -1,16 +1,17 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -S < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-codegen -S < %s | FileCheck %s
 
 ; CHECK-LABEL:   polly.stmt.bb48:
-; CHECK-NEXT:   %scevgep = getelementptr i64, i64* %A, i64 %polly.indvar
-; CHECK-NEXT:   %tmp51_p_scalar_ = load i64, i64* %scevgep,
+; CHECK-NEXT:   %[[offset:.*]] = shl i64 %polly.indvar, 3
+; CHECK-NEXT:   %scevgep = getelementptr i8, ptr %A, i64 %[[offset]]
+; CHECK-NEXT:   %tmp51_p_scalar_ = load i64, ptr %scevgep,
 ; CHECK-NEXT:   %p_tmp52 = and i64 %tmp51_p_scalar_, %tmp26
 ; CHECK-NEXT:   %p_tmp53 = icmp eq i64 %p_tmp52, %tmp26
-; CHECK-NEXT:   store i64 42, i64* %scevgep, align 8
+; CHECK-NEXT:   store i64 42, ptr %scevgep, align 8
 ; CHECK-NEXT:   br i1 %p_tmp53, label %polly.stmt.bb54, label %polly.stmt.bb56.exit
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @quux(i32 %arg, i32 %arg1, i64* %A, i64 %tmp9, i64 %tmp24, i64 %tmp14, i64 %tmp22, i64 %tmp44) {
+define void @quux(i32 %arg, i32 %arg1, ptr %A, i64 %tmp9, i64 %tmp24, i64 %tmp14, i64 %tmp22, i64 %tmp44) {
 bb:
   %tmp26 = or i64 %tmp22, %tmp24
   br label %bb39
@@ -25,16 +26,16 @@ bb46:                                             ; preds = %bb39
 
 bb48:                                             ; preds = %bb56, %bb46
   %tmp49 = phi i64 [ 0, %bb46 ], [ %tmp57, %bb56 ]
-  %tmp50 = getelementptr inbounds i64, i64* %A, i64 %tmp49
-  %tmp51 = load i64, i64* %tmp50, align 8
+  %tmp50 = getelementptr inbounds i64, ptr %A, i64 %tmp49
+  %tmp51 = load i64, ptr %tmp50, align 8
   %tmp52 = and i64 %tmp51, %tmp26
   %tmp53 = icmp eq i64 %tmp52, %tmp26
-  store i64 42, i64* %tmp50, align 8
+  store i64 42, ptr %tmp50, align 8
   br i1 %tmp53, label %bb54, label %bb56
 
 bb54:                                             ; preds = %bb48
   %tmp55 = xor i64 %tmp51, %tmp47
-  store i64 %tmp55, i64* %tmp50, align 8
+  store i64 %tmp55, ptr %tmp50, align 8
   br label %bb56
 
 bb56:                                             ; preds = %bb54, %bb48
diff --git a/polly/test/CodeGen/scev-backedgetaken.ll b/polly/test/CodeGen/scev-backedgetaken.ll
index 43ebe1d1cf11f..15e12ee8b4511 100644
--- a/polly/test/CodeGen/scev-backedgetaken.ll
+++ b/polly/test/CodeGen/scev-backedgetaken.ll
@@ -1,11 +1,11 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -S < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-codegen -S < %s | FileCheck %s
 ;
 ; llvm.org/PR48422
 ; Use of ScalarEvolution in Codegen not possible because DominatorTree is not updated.
 ;
 target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
 
-define dso_local void @func(i1 %b, i1 %p3, [14 x i32]* %d) local_unnamed_addr {
+define dso_local void @func(i1 %b, i1 %p3, ptr %d) local_unnamed_addr {
 entry:
   %conv = zext i1 %b to i16
   %add = select i1 %p3, i32 21, i32 20
@@ -17,13 +17,13 @@ for.body.us.us:
   br i1 %b, label %omp.inner.for.body.us.us.us.preheader, label %omp.inner.for.body.us63.us.preheader
 
 omp.inner.for.body.us63.us.preheader:
-  %arrayidx25.us.le71.us = getelementptr inbounds [14 x i32], [14 x i32]* %d, i64 %idxprom.us.us, i64 0
-  %0 = load i32, i32* %arrayidx25.us.le71.us, align 4
+  %arrayidx25.us.le71.us = getelementptr inbounds [14 x i32], ptr %d, i64 %idxprom.us.us, i64 0
+  %0 = load i32, ptr %arrayidx25.us.le71.us, align 4
   br label %omp.inner.for.cond.simd.if.end.loopexit_crit_edge.us.us
 
 omp.inner.for.body.us.us.us.preheader:
-  %arrayidx25.us.le.us.us = getelementptr inbounds [14 x i32], [14 x i32]* %d, i64 %idxprom.us.us, i64 0
-  %1 = load i32, i32* %arrayidx25.us.le.us.us, align 4
+  %arrayidx25.us.le.us.us = getelementptr inbounds [14 x i32], ptr %d, i64 %idxprom.us.us, i64 0
+  %1 = load i32, ptr %arrayidx25.us.le.us.us, align 4
   %conv27.us.le.us.us = select i1 undef, i16 0, i16 undef
   br label %omp.inner.for.cond.simd.if.end.loopexit_crit_edge.us.us
 
@@ -41,7 +41,7 @@ for.cond.cleanup.loopexit:
 
 ; CHECK-LABEL: @func(
 ; CHECK:         polly.stmt.omp.inner.for.body.us.us.us.preheader:
-; CHECK:         load i32, i32* %scevgep, align 4, !alias.scope !0, !noalias !3
+; CHECK:         load i32, ptr %scevgep, align 4, !alias.scope !0, !noalias !3
 
 ; CHECK:       !0 = !{!1}
 ; CHECK:       !1 = distinct !{!1, !2, !"polly.alias.scope.MemRef_d"}
diff --git a/polly/test/CodeGen/scev_expansion_in_nonaffine.ll b/polly/test/CodeGen/scev_expansion_in_nonaffine.ll
index 4356f309bae6f..f61f21d4adb85 100644
--- a/polly/test/CodeGen/scev_expansion_in_nonaffine.ll
+++ b/polly/test/CodeGen/scev_expansion_in_nonaffine.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -S \
+; RUN: opt %loadPolly -polly-codegen -S \
 ; RUN: -polly-invariant-load-hoisting=true < %s | FileCheck %s
 
 ; bugpoint-reduced testcase of MiBench/consumer-lame/quantize-pvt.c from the
@@ -9,43 +9,43 @@
 ; whole subregion.
 
 ; CHECK-LABEL:  polly.stmt.if.then.110:
-; CHECK:          %[[R1_1:[0-9]*]] = mul nuw nsw i64 %polly.indvar[[R0_1:[0-9]*]], 30
-; CHECK:          %scevgep[[R1_2:[0-9]*]] = getelementptr i32, i32* %scevgep{{[0-9]*}}, i64 %[[R1_1]]
-; CHECK:          store i32 0, i32* %scevgep[[R1_2]], align 8
+; CHECK:          %[[R1_1:[0-9]*]] = mul nuw nsw i64 %polly.indvar[[R0_1:[0-9]*]], 120
+; CHECK:          %scevgep[[R1_2:[0-9]*]] = getelementptr i8, ptr %scevgep{{[0-9]*}}, i64 %[[R1_1]]
+; CHECK:          store i32 0, ptr %scevgep[[R1_2]], align 8
 
 ; CHECK-LABEL:  polly.stmt.if.else:
-; CHECK:          %[[R2_1:[0-9]*]] = mul nuw nsw i64 %polly.indvar[[R0_1]], 30
-; CHECK:          %scevgep[[R2_2:[0-9]*]] = getelementptr i32, i32* %scevgep{{[0-9]*}}, i64 %[[R2_1]]
-; CHECK:          store i32 21, i32* %scevgep[[R2_2]], align 8
+; CHECK:          %[[R2_1:[0-9]*]] = mul nuw nsw i64 %polly.indvar[[R0_1]], 120
+; CHECK:          %scevgep[[R2_2:[0-9]*]] = getelementptr i8, ptr %scevgep{{[0-9]*}}, i64 %[[R2_1]]
+; CHECK:          store i32 21, ptr %scevgep[[R2_2]], align 8
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
-%struct.gr_info.4.59.136.224.290 = type { i32, i32, i32, i32, i32, i32, i32, i32, [3 x i32], [3 x i32], i32, i32, i32, i32, i32, i32, i32, i32, i32, i32*, [4 x i32] }
+%struct.gr_info.4.59.136.224.290 = type { i32, i32, i32, i32, i32, i32, i32, i32, [3 x i32], [3 x i32], i32, i32, i32, i32, i32, i32, i32, i32, i32, ptr, [4 x i32] }
 %struct.gr_info_ss.5.60.137.225.291 = type { %struct.gr_info.4.59.136.224.290 }
 %struct.anon.6.61.138.226.292 = type { [2 x %struct.gr_info_ss.5.60.137.225.291] }
 %struct.III_side_info_t.7.62.139.227.293 = type { i32, i32, i32, [2 x [4 x i32]], [2 x %struct.anon.6.61.138.226.292] }
-%struct.lame_global_flags.3.58.135.223.289 = type { i64, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i8*, i8*, i32, i32, float, i32, i32, i32, i64, i64, i32, i32, i32, i32, i32, i32, i32, i32, float, i32, i32, i32, float, float, float, float, i32, i32, i32, i32, i32, i32, i32, i32 }
+%struct.lame_global_flags.3.58.135.223.289 = type { i64, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, ptr, ptr, i32, i32, float, i32, i32, i32, i64, i64, i32, i32, i32, i32, i32, i32, i32, i32, float, i32, i32, i32, float, float, float, float, i32, i32, i32, i32, i32, i32, i32, i32 }
 
 @convert_mdct = external global i32, align 4
 @reduce_sidechannel = external global i32, align 4
 
 ; Function Attrs: nounwind uwtable
-define void @iteration_init(%struct.lame_global_flags.3.58.135.223.289* %gfp, %struct.III_side_info_t.7.62.139.227.293* %l3_side, [2 x [576 x i32]]* %l3_enc) #0 {
+define void @iteration_init(ptr %gfp, ptr %l3_side, ptr %l3_enc) #0 {
 entry:
-  %resvDrain = getelementptr inbounds %struct.III_side_info_t.7.62.139.227.293, %struct.III_side_info_t.7.62.139.227.293* %l3_side, i64 0, i32 2
-  store i32 0, i32* %resvDrain, align 8
-  store i32 0, i32* @convert_mdct, align 4
-  store i32 0, i32* @reduce_sidechannel, align 4
-  %mode_gr = getelementptr inbounds %struct.lame_global_flags.3.58.135.223.289, %struct.lame_global_flags.3.58.135.223.289* %gfp, i64 0, i32 45
-  %0 = load i32, i32* %mode_gr, align 8
+  %resvDrain = getelementptr inbounds %struct.III_side_info_t.7.62.139.227.293, ptr %l3_side, i64 0, i32 2
+  store i32 0, ptr %resvDrain, align 8
+  store i32 0, ptr @convert_mdct, align 4
+  store i32 0, ptr @reduce_sidechannel, align 4
+  %mode_gr = getelementptr inbounds %struct.lame_global_flags.3.58.135.223.289, ptr %gfp, i64 0, i32 45
+  %0 = load i32, ptr %mode_gr, align 8
   %cmp95.145 = icmp sgt i32 %0, 0
   br i1 %cmp95.145, label %for.cond.98.preheader, label %for.cond.120.preheader
 
 for.cond.98.preheader:                            ; preds = %for.inc.117, %entry
   %indvars.iv157 = phi i64 [ %indvars.iv.next158, %for.inc.117 ], [ 0, %entry ]
-  %stereo = getelementptr inbounds %struct.lame_global_flags.3.58.135.223.289, %struct.lame_global_flags.3.58.135.223.289* %gfp, i64 0, i32 46
-  %1 = load i32, i32* %stereo, align 4
+  %stereo = getelementptr inbounds %struct.lame_global_flags.3.58.135.223.289, ptr %gfp, i64 0, i32 46
+  %1 = load i32, ptr %stereo, align 4
   %cmp99.143 = icmp sgt i32 %1, 0
   br i1 %cmp99.143, label %for.body.101, label %for.inc.117
 
@@ -54,34 +54,34 @@ for.cond.120.preheader:                           ; preds = %for.inc.117, %entry
 
 for.body.101:                                     ; preds = %for.inc.114, %for.cond.98.preheader
   %indvars.iv155 = phi i64 [ %indvars.iv.next156, %for.inc.114 ], [ 0, %for.cond.98.preheader ]
-  %block_type = getelementptr inbounds %struct.III_side_info_t.7.62.139.227.293, %struct.III_side_info_t.7.62.139.227.293* %l3_side, i64 0, i32 4, i64 %indvars.iv157, i32 0, i64 %indvars.iv155, i32 0, i32 6
-  %2 = load i32, i32* %block_type, align 8
+  %block_type = getelementptr inbounds %struct.III_side_info_t.7.62.139.227.293, ptr %l3_side, i64 0, i32 4, i64 %indvars.iv157, i32 0, i64 %indvars.iv155, i32 0, i32 6
+  %2 = load i32, ptr %block_type, align 8
   %cmp108 = icmp eq i32 %2, 2
-  %sfb_lmax = getelementptr inbounds %struct.III_side_info_t.7.62.139.227.293, %struct.III_side_info_t.7.62.139.227.293* %l3_side, i64 0, i32 4, i64 %indvars.iv157, i32 0, i64 %indvars.iv155, i32 0, i32 16
+  %sfb_lmax = getelementptr inbounds %struct.III_side_info_t.7.62.139.227.293, ptr %l3_side, i64 0, i32 4, i64 %indvars.iv157, i32 0, i64 %indvars.iv155, i32 0, i32 16
   br i1 %cmp108, label %if.then.110, label %if.else
 
 if.then.110:                                      ; preds = %for.body.101
-  store i32 0, i32* %sfb_lmax, align 8
-  %sfb_smax = getelementptr inbounds %struct.III_side_info_t.7.62.139.227.293, %struct.III_side_info_t.7.62.139.227.293* %l3_side, i64 0, i32 4, i64 %indvars.iv157, i32 0, i64 %indvars.iv155, i32 0, i32 17
-  store i32 0, i32* %sfb_smax, align 4
+  store i32 0, ptr %sfb_lmax, align 8
+  %sfb_smax = getelementptr inbounds %struct.III_side_info_t.7.62.139.227.293, ptr %l3_side, i64 0, i32 4, i64 %indvars.iv157, i32 0, i64 %indvars.iv155, i32 0, i32 17
+  store i32 0, ptr %sfb_smax, align 4
   br label %for.inc.114
 
 if.else:                                          ; preds = %for.body.101
-  store i32 21, i32* %sfb_lmax, align 8
-  %sfb_smax112 = getelementptr inbounds %struct.III_side_info_t.7.62.139.227.293, %struct.III_side_info_t.7.62.139.227.293* %l3_side, i64 0, i32 4, i64 %indvars.iv157, i32 0, i64 %indvars.iv155, i32 0, i32 17
-  store i32 12, i32* %sfb_smax112, align 4
+  store i32 21, ptr %sfb_lmax, align 8
+  %sfb_smax112 = getelementptr inbounds %struct.III_side_info_t.7.62.139.227.293, ptr %l3_side, i64 0, i32 4, i64 %indvars.iv157, i32 0, i64 %indvars.iv155, i32 0, i32 17
+  store i32 12, ptr %sfb_smax112, align 4
   br label %for.inc.114
 
 for.inc.114:                                      ; preds = %if.else, %if.then.110
   %indvars.iv.next156 = add nuw nsw i64 %indvars.iv155, 1
-  %3 = load i32, i32* %stereo, align 4
+  %3 = load i32, ptr %stereo, align 4
   %4 = sext i32 %3 to i64
   %cmp99 = icmp slt i64 %indvars.iv.next156, %4
   br i1 %cmp99, label %for.body.101, label %for.inc.117
 
 for.inc.117:                                      ; preds = %for.inc.114, %for.cond.98.preheader
   %indvars.iv.next158 = add nuw nsw i64 %indvars.iv157, 1
-  %5 = load i32, i32* %mode_gr, align 8
+  %5 = load i32, ptr %mode_gr, align 8
   %6 = sext i32 %5 to i64
   %cmp95 = icmp slt i64 %indvars.iv.next158, %6
   br i1 %cmp95, label %for.cond.98.preheader, label %for.cond.120.preheader
diff --git a/polly/test/CodeGen/scev_looking_through_bitcasts.ll b/polly/test/CodeGen/scev_looking_through_bitcasts.ll
index 6a1603d862fc6..c87d932479b7c 100644
--- a/polly/test/CodeGen/scev_looking_through_bitcasts.ll
+++ b/polly/test/CodeGen/scev_looking_through_bitcasts.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -S < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-codegen -S < %s | FileCheck %s
 ;
 ; Scalar write of bitcasted value. Instead of writing %b of type
 ; %structty, the SCEV expression looks through the bitcast such that
@@ -7,11 +7,11 @@
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-%structty = type { %structty*, %structty*, i32, [2 x i64] }
+%structty = type { ptr, ptr, i32, [2 x i64] }
 
 define void @bitmap_set_range() {
 entry:
-  %a = ptrtoint i8* undef to i64
+  %a = ptrtoint ptr undef to i64
   br label %cond.end32.i
 
 cond.end32.i:
@@ -21,19 +21,17 @@ cond.true67.i:
   br label %cond.end73.i
 
 cond.end73.i:
-  %add.ptr81.i = getelementptr inbounds i8, i8* null, i64 %a
-  %b = bitcast i8* %add.ptr81.i to %structty*
+  %add.ptr81.i = getelementptr inbounds i8, ptr null, i64 %a
   br label %bitmap_element_allocate.exit
 
 bitmap_element_allocate.exit:
-  %tobool43 = icmp eq %structty* %b, null
+  %tobool43 = icmp eq ptr %add.ptr81.i, null
   ret void
 }
 
 
 
 ; CHECK:      polly.stmt.cond.end73.i:
-; CHECK-NEXT:   %scevgep = getelementptr i8, i8* null, i64 %a
-; CHECK-NEXT:   %scevgep1 = bitcast i8* %scevgep to %structty*
-; CHECK-NEXT:   store %structty* %scevgep1, %structty** %b.s2a, align 8
+; CHECK-NEXT:   %scevgep = getelementptr i8, ptr null, i64 %a
+; CHECK-NEXT:   store ptr %scevgep, ptr %add.ptr81.i.s2a, align 8
 ; CHECK-NEXT:   br label %polly.exiting
diff --git a/polly/test/CodeGen/simple_vec_assign_scalar.ll b/polly/test/CodeGen/simple_vec_assign_scalar.ll
deleted file mode 100644
index ae8a75257ac66..0000000000000
--- a/polly/test/CodeGen/simple_vec_assign_scalar.ll
+++ /dev/null
@@ -1,64 +0,0 @@
-; RUN: opt %loadPolly -basic-aa -polly-codegen \
-; RUN: -polly-vectorizer=polly -dce -S < %s | FileCheck %s
-
-; RUN: opt %loadPolly -basic-aa -polly-codegen -polly-vectorizer=stripmine -dce -S < %s | FileCheck %s --check-prefix=STRIPMINE
-
-;#define N 1024
-;float A[N];
-;float B[N];
-;
-;void simple_vec_const(void) {
-;  int i;
-;
-;  for (i = 0; i < 4; i++)
-;    B[i] = A[i] + 1;
-;}
-;int main()
-;{
-;  simple_vec_const();
-;  return A[42];
-;}
-
-target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
-
-@A = common global [1024 x float] zeroinitializer, align 16
-@B = common global [1024 x float] zeroinitializer, align 16
-
-define void @simple_vec_const() nounwind {
-bb:
-  br label %bb2
-
-bb2:                                              ; preds = %bb5, %bb
-  %indvar = phi i64 [ %indvar.next, %bb5 ], [ 0, %bb ]
-  %scevgep = getelementptr [1024 x float], ptr @B, i64 0, i64 %indvar
-  %scevgep1 = getelementptr [1024 x float], ptr @A, i64 0, i64 %indvar
-  %exitcond = icmp ne i64 %indvar, 4
-  br i1 %exitcond, label %bb3, label %bb6
-
-bb3:                                              ; preds = %bb2
-  %tmp = load float, ptr %scevgep1, align 4
-  %tmp4 = fadd float %tmp, 1.000000e+00
-  store float %tmp4, ptr %scevgep, align 4
-  br label %bb5
-
-bb5:                                              ; preds = %bb3
-  %indvar.next = add i64 %indvar, 1
-  br label %bb2
-
-bb6:                                              ; preds = %bb2
-  ret void
-}
-
-define i32 @main() nounwind {
-bb:
-  call void @simple_vec_const()
-  %tmp = load float, ptr getelementptr inbounds ([1024 x float], ptr @A, i64 0, i64 42), align 8
-  %tmp1 = fptosi float %tmp to i32
-  ret i32 %tmp1
-}
-
-; STRIPMINE-NOT: <4 x float>
-
-; CHECK: %tmp_p_vec_full = load <4 x float>, ptr @A, align 8, !alias.scope !0, !noalias !3
-; CHECK: %tmp4p_vec = fadd <4 x float> %tmp_p_vec_full, <float 1.000000e+00, float 1.000000e+00, float 1.000000e+00, float 1.000000e+00>
-; CHECK: store <4 x float> %tmp4p_vec, ptr @B
diff --git a/polly/test/CodeGen/simple_vec_assign_scalar_2.ll b/polly/test/CodeGen/simple_vec_assign_scalar_2.ll
deleted file mode 100644
index f26720f95df3a..0000000000000
--- a/polly/test/CodeGen/simple_vec_assign_scalar_2.ll
+++ /dev/null
@@ -1,65 +0,0 @@
-; RUN: opt %loadPolly -basic-aa -polly-codegen -polly-vectorizer=polly -dce -S < %s | FileCheck %s
-
-;#define N 1024
-;float A[N];
-;float B[N];
-;
-;void simple_vec_const(void) {
-;  int i;
-;
-;  for (i = 0; i < 4; i++)
-;    B[i] = A[i] + i;
-;}
-;int main()
-;{
-;  simple_vec_const();
-;  return A[42];
-;}
-
-target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
-
-@A = common global [1024 x float] zeroinitializer, align 16
-@B = common global [1024 x float] zeroinitializer, align 16
-
-define void @simple_vec_const() nounwind {
-bb:
-  br label %bb2
-
-bb2:                                              ; preds = %bb6, %bb
-  %indvar = phi i64 [ %indvar.next, %bb6 ], [ 0, %bb ]
-  %scevgep = getelementptr [1024 x float], ptr @B, i64 0, i64 %indvar
-  %i.0 = trunc i64 %indvar to i32
-  %scevgep1 = getelementptr [1024 x float], ptr @A, i64 0, i64 %indvar
-  %exitcond = icmp ne i64 %indvar, 4
-  br i1 %exitcond, label %bb3, label %bb7
-
-bb3:                                              ; preds = %bb2
-  %tmp = load float, ptr %scevgep1, align 4
-  %tmp4 = sitofp i32 %i.0 to float
-  %tmp5 = fadd float %tmp, %tmp4
-  store float %tmp5, ptr %scevgep, align 4
-  br label %bb6
-
-bb6:                                              ; preds = %bb3
-  %indvar.next = add i64 %indvar, 1
-  br label %bb2
-
-bb7:                                              ; preds = %bb2
-  ret void
-}
-
-define i32 @main() nounwind {
-bb:
-  call void @simple_vec_const()
-  %tmp = load float, ptr getelementptr inbounds ([1024 x float], ptr @A, i64 0, i64 42), align 8
-  %tmp1 = fptosi float %tmp to i32
-  ret i32 %tmp1
-}
-
-
-; CHECK: insertelement <4 x float> undef, float %{{[^,]+}}, i32 0
-; CHECK: insertelement <4 x float> %0, float %{{[^,]+}}, i32 1
-; CHECK: insertelement <4 x float> %1, float %{{[^,]+}}, i32 2
-; CHECK: insertelement <4 x float> %2, float %{{[^,]+}}, i32 3
-; CHECK: fadd <4 x float> %tmp_p_vec_full, %3
-
diff --git a/polly/test/CodeGen/simple_vec_call.ll b/polly/test/CodeGen/simple_vec_call.ll
deleted file mode 100644
index e5f2c129f135c..0000000000000
--- a/polly/test/CodeGen/simple_vec_call.ll
+++ /dev/null
@@ -1,37 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -basic-aa -polly-codegen -polly-vectorizer=polly -S \
-; RUN: -polly-invariant-load-hoisting=true < %s | FileCheck %s
-target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
-
-@A = common global [1024 x float] zeroinitializer, align 16
-@B = common global [1024 x float] zeroinitializer, align 16
-
-declare float @foo(float) readnone
-
-define void @simple_vec_call() nounwind {
-entry:
-  br label %body
-
-body:
-  %indvar = phi i64 [ 0, %entry ], [ %indvar_next, %body ]
-  %scevgep = getelementptr [1024 x float], [1024 x float]* @B, i64 0, i64 %indvar
-  %value = load float, float* getelementptr inbounds ([1024 x float], [1024 x float]* @A, i64 0, i64 0), align 16
-  %result = tail call float @foo(float %value) nounwind
-  store float %result, float* %scevgep, align 4
-  %indvar_next = add i64 %indvar, 1
-  %exitcond = icmp eq i64 %indvar_next, 4
-  br i1 %exitcond, label %return, label %body
-
-return:
-  ret void
-}
-
-; CHECK: [[RES1:%[a-zA-Z0-9_]+]] = tail call float @foo(float %.load) [[NUW:#[0-9]+]]
-; CHECK: [[RES2:%[a-zA-Z0-9_]+]] = tail call float @foo(float %.load) [[NUW]]
-; CHECK: [[RES3:%[a-zA-Z0-9_]+]] = tail call float @foo(float %.load) [[NUW]]
-; CHECK: [[RES4:%[a-zA-Z0-9_]+]] = tail call float @foo(float %.load) [[NUW]]
-; CHECK: [[RES5:%[a-zA-Z0-9_]+]] = insertelement <4 x float> undef, float [[RES1]], i32 0
-; CHECK: [[RES6:%[a-zA-Z0-9_]+]] = insertelement <4 x float> [[RES5]], float [[RES2]], i32 1
-; CHECK: [[RES7:%[a-zA-Z0-9_]+]] = insertelement <4 x float> [[RES6]], float [[RES3]], i32 2
-; CHECK: [[RES8:%[a-zA-Z0-9_]+]] = insertelement <4 x float> [[RES7]], float [[RES4]], i32 3
-; CHECK:  store <4 x float> [[RES8]]
-; CHECK: attributes [[NUW]] = { nounwind }
diff --git a/polly/test/CodeGen/simple_vec_call_2.ll b/polly/test/CodeGen/simple_vec_call_2.ll
deleted file mode 100644
index b3767057dd65e..0000000000000
--- a/polly/test/CodeGen/simple_vec_call_2.ll
+++ /dev/null
@@ -1,37 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -basic-aa -polly-codegen -polly-vectorizer=polly -dce \
-; RUN: -polly-invariant-load-hoisting=true -S < %s | FileCheck %s
-target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
-
-@A = common global [1024 x float] zeroinitializer, align 16
-@B = common global [1024 x float**] zeroinitializer, align 16
-
-declare float** @foo(float) readnone
-
-define void @simple_vec_call() nounwind {
-entry:
-  br label %body
-
-body:
-  %indvar = phi i64 [ 0, %entry ], [ %indvar_next, %body ]
-  %scevgep = getelementptr [1024 x float**], [1024 x float**]* @B, i64 0, i64 %indvar
-  %value = load float, float* getelementptr inbounds ([1024 x float], [1024 x float]* @A, i64 0, i64 0), align 16
-  %result = tail call float** @foo(float %value) nounwind
-  store float** %result, float*** %scevgep, align 4
-  %indvar_next = add i64 %indvar, 1
-  %exitcond = icmp eq i64 %indvar_next, 4
-  br i1 %exitcond, label %return, label %body
-
-return:
-  ret void
-}
-
-; CHECK: [[RES1:%[a-zA-Z0-9_]+]] = tail call float** @foo(float %.load) [[NUW:#[0-9]+]]
-; CHECK: [[RES2:%[a-zA-Z0-9_]+]] = tail call float** @foo(float %.load) [[NUW]]
-; CHECK: [[RES3:%[a-zA-Z0-9_]+]] = tail call float** @foo(float %.load) [[NUW]]
-; CHECK: [[RES4:%[a-zA-Z0-9_]+]] = tail call float** @foo(float %.load) [[NUW]]
-; CHECK: %0 = insertelement <4 x float**> undef, float** %p_result, i32 0
-; CHECK: %1 = insertelement <4 x float**> %0, float** %p_result1, i32 1
-; CHECK: %2 = insertelement <4 x float**> %1, float** %p_result2, i32 2
-; CHECK: %3 = insertelement <4 x float**> %2, float** %p_result3, i32 3
-; CHECK: store <4 x float**> %3, <4 x float**>* bitcast ([1024 x float**]* @B to <4 x float**>*), align
-; CHECK: attributes [[NUW]] = { nounwind }
diff --git a/polly/test/CodeGen/simple_vec_cast.ll b/polly/test/CodeGen/simple_vec_cast.ll
deleted file mode 100644
index 7578663754ed7..0000000000000
--- a/polly/test/CodeGen/simple_vec_cast.ll
+++ /dev/null
@@ -1,38 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -basic-aa -polly-codegen -polly-vectorizer=polly \
-; RUN: -polly-invariant-load-hoisting=true -dce -S < %s | FileCheck %s
-target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
-
-@A = common global [1024 x float] zeroinitializer, align 16
-@B = common global [1024 x double] zeroinitializer, align 16
-
-define void @simple_vec_const() nounwind {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb3, %bb
-  %indvar = phi i64 [ %indvar.next, %bb3 ], [ 0, %bb ]
-  %scevgep = getelementptr [1024 x double], [1024 x double]* @B, i64 0, i64 %indvar
-  %exitcond = icmp ne i64 %indvar, 4
-  br i1 %exitcond, label %bb2, label %bb4
-
-bb2:                                              ; preds = %bb1
-  %tmp = load float, float* getelementptr inbounds ([1024 x float], [1024 x float]* @A, i64 0, i64 0), align 16
-  %tmp2 = fpext float %tmp to double
-  store double %tmp2, double* %scevgep, align 4
-  br label %bb3
-
-bb3:                                              ; preds = %bb2
-  %indvar.next = add i64 %indvar, 1
-  br label %bb1
-
-bb4:                                              ; preds = %bb1
-  ret void
-}
-
-; CHECK:   %.load = load float, float* getelementptr inbounds ([1024 x float], [1024 x float]* @A, i32 0, i32 0)
-
-; CHECK: polly.stmt.bb2:                                   ; preds = %polly.start
-; CHECK:   %tmp_p.splatinsert = insertelement <4 x float> poison, float %.load, i64 0
-; CHECK:   %tmp_p.splat = shufflevector <4 x float> %tmp_p.splatinsert, <4 x float> poison, <4 x i32> zeroinitializer
-; CHECK:   %0 = fpext <4 x float> %tmp_p.splat to <4 x double>
-; CHECK:   store <4 x double> %0, <4 x double>*
diff --git a/polly/test/CodeGen/simple_vec_const.ll b/polly/test/CodeGen/simple_vec_const.ll
deleted file mode 100644
index cb5edb2bcbd0f..0000000000000
--- a/polly/test/CodeGen/simple_vec_const.ll
+++ /dev/null
@@ -1,60 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -basic-aa -polly-codegen -polly-vectorizer=polly -S \
-; RUN: -polly-invariant-load-hoisting=true < %s | FileCheck %s
-
-;#define N 1024
-;float A[N];
-;float B[N];
-;
-;void simple_vec_const(void) {
-;  int i;
-;
-;  for (i = 0; i < 4; i++)
-;    B[i] = A[0];
-;}
-;int main()
-;{
-;  simple_vec_const();
-;  return A[42];
-;}
-
-target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
-
-@A = common global [1024 x float] zeroinitializer, align 16
-@B = common global [1024 x float] zeroinitializer, align 16
-
-define void @simple_vec_const() nounwind {
-; <label>:0
-  br label %1
-
-; <label>:1                                       ; preds = %4, %0
-  %indvar = phi i64 [ %indvar.next, %4 ], [ 0, %0 ]
-  %scevgep = getelementptr [1024 x float], [1024 x float]* @B, i64 0, i64 %indvar
-  %exitcond = icmp ne i64 %indvar, 4
-  br i1 %exitcond, label %2, label %5
-
-; <label>:2                                       ; preds = %1
-  %3 = load float, float* getelementptr inbounds ([1024 x float], [1024 x float]* @A, i64 0, i64 0), align 16
-  store float %3, float* %scevgep, align 4
-  br label %4
-
-; <label>:4                                       ; preds = %2
-  %indvar.next = add i64 %indvar, 1
-  br label %1
-
-; <label>:5                                       ; preds = %1
-  ret void
-}
-
-define i32 @main() nounwind {
-  call void @simple_vec_const()
-  %1 = load float, float* getelementptr inbounds ([1024 x float], [1024 x float]* @A, i64 0, i64 42), align 8
-  %2 = fptosi float %1 to i32
-  ret i32 %2
-}
-
-
-; CHECK:   %.load = load float, float* getelementptr inbounds ([1024 x float], [1024 x float]* @A, i32 0, i32 0)
-
-; CHECK: polly.stmt.:                                      ; preds = %polly.start
-; CHECK:   %_p.splatinsert = insertelement <4 x float> poison, float %.load, i64 0
-; CHECK:   %_p.splat = shufflevector <4 x float> %_p.splatinsert, <4 x float> poison, <4 x i32> zeroinitializer
diff --git a/polly/test/CodeGen/simple_vec_large_width.ll b/polly/test/CodeGen/simple_vec_large_width.ll
deleted file mode 100644
index 4ad3b89876165..0000000000000
--- a/polly/test/CodeGen/simple_vec_large_width.ll
+++ /dev/null
@@ -1,39 +0,0 @@
-; RUN: opt %loadPolly -basic-aa -polly-codegen -polly-vectorizer=polly -dce -S < %s | FileCheck %s
-target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
-
-@A = common global [1024 x float] zeroinitializer, align 16
-@B = common global [1024 x float] zeroinitializer, align 16
-
-define void @simple_vec_large_width() nounwind {
-; <label>:0
-  br label %1
-
-; <label>:1                                       ; preds = %4, %0
-  %indvar = phi i64 [ %indvar.next, %4 ], [ 0, %0 ]
-  %scevgep = getelementptr [1024 x float], ptr @B, i64 0, i64 %indvar
-  %scevgep1 = getelementptr [1024 x float], ptr @A, i64 0, i64 %indvar
-  %exitcond = icmp ne i64 %indvar, 15
-  br i1 %exitcond, label %2, label %5
-
-; <label>:2                                       ; preds = %1
-  %3 = load float, ptr %scevgep1, align 4
-  store float %3, ptr %scevgep, align 4
-  br label %4
-
-; <label>:4                                       ; preds = %2
-  %indvar.next = add i64 %indvar, 1
-  br label %1
-
-; <label>:5                                       ; preds = %1
-  ret void
-}
-
-define i32 @main() nounwind {
-  call void @simple_vec_large_width()
-  %1 = load float, ptr getelementptr inbounds ([1024 x float], ptr @A, i64 0, i64 42), align 8
-  %2 = fptosi float %1 to i32
-  ret i32 %2
-}
-
-; CHECK: [[VEC1:%[a-zA-Z0-9_]+_full]] = load <15 x float>, ptr
-; CHECK: store <15 x float> [[VEC1]]
diff --git a/polly/test/CodeGen/simple_vec_ptr_ptr_ty.ll b/polly/test/CodeGen/simple_vec_ptr_ptr_ty.ll
deleted file mode 100644
index 8e4b8bce6f8ff..0000000000000
--- a/polly/test/CodeGen/simple_vec_ptr_ptr_ty.ll
+++ /dev/null
@@ -1,31 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -basic-aa -polly-codegen -polly-vectorizer=polly -S \
-; RUN: -polly-invariant-load-hoisting=true < %s | FileCheck %s
-target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
-
-@A = common global [1024 x float**] zeroinitializer, align 16
-@B = common global [1024 x float**] zeroinitializer, align 16
-
-declare float @foo(float) readnone
-
-define void @simple_vec_call() nounwind {
-entry:
-  br label %body
-
-body:
-  %indvar = phi i64 [ 0, %entry ], [ %indvar_next, %body ]
-  %scevgep = getelementptr [1024 x float**], [1024 x float**]* @B, i64 0, i64 %indvar
-  %value = load float**, float*** getelementptr inbounds ([1024 x float**], [1024 x float**]* @A, i64 0, i64 0), align 16
-  store float** %value, float*** %scevgep, align 4
-  %indvar_next = add i64 %indvar, 1
-  %exitcond = icmp eq i64 %indvar_next, 4
-  br i1 %exitcond, label %return, label %body
-
-return:
-  ret void
-}
-; CHECK:   %.load = load float**, float*** getelementptr inbounds ([1024 x float**], [1024 x float**]* @A, i32 0, i32 0)
-
-; CHECK-NOT: load <1 x float**>
-; CHECK: %value_p.splatinsert = insertelement <4 x float**> poison, float** %.load, i64 0
-; CHECK: %value_p.splat = shufflevector <4 x float**> %value_p.splatinsert, <4 x float**> poison, <4 x i32> zeroinitializer
-; CHECK: store <4 x float**> %value_p.splat, <4 x float**>* bitcast ([1024 x float**]* @B to <4 x float**>*), align 8
diff --git a/polly/test/CodeGen/simple_vec_stride_negative_one.ll b/polly/test/CodeGen/simple_vec_stride_negative_one.ll
deleted file mode 100644
index 79ecf657d8902..0000000000000
--- a/polly/test/CodeGen/simple_vec_stride_negative_one.ll
+++ /dev/null
@@ -1,37 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen -polly-vectorizer=polly -S < %s | FileCheck %s
-
-; ModuleID = 'reverse.c'
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-;int A[100];
-;void foo() {
-;  for (int i=3; i >= 0; i--)
-;    A[i]+=1;
-;}
-
-
-@A = common global [100 x i32] zeroinitializer, align 16
-
-; Function Attrs: nounwind uwtable
-define void @foo() #0 {
-entry:
-  br label %for.body
-
-for.body:                                         ; preds = %entry, %for.body
-  %indvars.iv = phi i64 [ 3, %entry ], [ %indvars.iv.next, %for.body ]
-  %arrayidx = getelementptr inbounds [100 x i32], ptr @A, i64 0, i64 %indvars.iv
-  %0 = load i32, ptr %arrayidx, align 4
-  %add = add nsw i32 %0, 1
-  store i32 %add, ptr %arrayidx, align 4
-  %indvars.iv.next = add nsw i64 %indvars.iv, -1
-  %1 = trunc i64 %indvars.iv to i32
-  %cmp = icmp sgt i32 %1, 0
-  br i1 %cmp, label %for.body, label %for.end
-
-for.end:                                          ; preds = %for.body
-  ret void
-}
-
-; CHECK: @foo
-; CHECK: [[LOAD:%[a-zA-Z0-9_]+]] = load <4 x i32>, ptr
-; CHECK: [[REVERSE_LOAD:%[a-zA-Z0-9_]+reverse]] = shufflevector <4 x i32> [[LOAD]], <4 x i32> [[LOAD]], <4 x i32> <i32 3, i32 2, i32 1, i32 0>
diff --git a/polly/test/CodeGen/simple_vec_stride_one.ll b/polly/test/CodeGen/simple_vec_stride_one.ll
deleted file mode 100644
index 8250879283b19..0000000000000
--- a/polly/test/CodeGen/simple_vec_stride_one.ll
+++ /dev/null
@@ -1,37 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -polly-vectorizer=polly \
-; RUN:                 < %s -S | FileCheck %s
-
-; CHECK: store <4 x double> %val.s2a_p_splat, <4 x double>* %vector_ptr
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @update_access_functions(i64 %arg, double* %A, double* %B) {
-bb3:
-  br label %loop1
-
-loop1:
-  %indvar = phi i64 [ %indvar.next, %loop1 ], [ 0, %bb3 ]
-  %ptr1 = getelementptr inbounds double, double* %A, i64 %indvar
-  store double 42.0, double* %ptr1, align 8
-  %indvar.next = add nuw nsw i64 %indvar, 1
-  %cmp = icmp ne i64 %indvar.next, 4
-  br i1 %cmp, label %loop1, label %loop2
-
-loop2:
-  %indvar.2 = phi i64 [ %indvar.2.next, %loop2 ], [ 0, %loop1 ]
-  %ptr2 = getelementptr inbounds double, double* %A, i64 %indvar.2
-  %val = load double, double* %ptr2, align 8
-  %indvar.2.next = add nuw nsw i64 %indvar.2, 1
-  %cmp.2 = icmp ne i64 %indvar.2.next, 4
-  br i1 %cmp.2, label %loop2, label %loop3
-
-loop3:
-  %indvar.3 = phi i64 [ %indvar.3.next, %loop3 ], [ 0, %loop2 ]
-  %ptr3 = getelementptr inbounds double, double* %A, i64 %indvar.3
-  store double %val, double* %ptr3, align 8
-  %indvar.3.next = add nuw nsw i64 %indvar.3, 1
-  %cmp.3 = icmp ne i64 %indvar.3.next, 4
-  br i1 %cmp.3, label %loop3, label %exit
-
-exit:
-  ret void
-}
diff --git a/polly/test/CodeGen/simple_vec_stride_x.ll b/polly/test/CodeGen/simple_vec_stride_x.ll
deleted file mode 100644
index 20a8e8a3bbcc4..0000000000000
--- a/polly/test/CodeGen/simple_vec_stride_x.ll
+++ /dev/null
@@ -1,72 +0,0 @@
-; RUN: opt %loadPolly -basic-aa -polly-codegen -polly-vectorizer=polly  -dce -S < %s | FileCheck %s
-
-;#define N 1024
-;float A[N];
-;float B[N];
-;
-;void simple_vec_stride_x(void) {
-;  int i;
-;
-;  for (i = 0; i < 4; i++)
-;    B[2 * i] = A[2 * i];
-;}
-;int main()
-;{
-;  simple_vec_stride_x();
-;  return A[42];
-;}
-
-target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
-
-@A = common global [1024 x float] zeroinitializer, align 16
-@B = common global [1024 x float] zeroinitializer, align 16
-
-define void @simple_vec_stride_x() nounwind {
-bb:
-  br label %bb2
-
-bb2:                                              ; preds = %bb5, %bb
-  %indvar = phi i64 [ %indvar.next, %bb5 ], [ 0, %bb ]
-  %tmp = mul i64 %indvar, 2
-  %scevgep = getelementptr [1024 x float], ptr @B, i64 0, i64 %tmp
-  %scevgep1 = getelementptr [1024 x float], ptr @A, i64 0, i64 %tmp
-  %exitcond = icmp ne i64 %indvar, 4
-  br i1 %exitcond, label %bb3, label %bb6
-
-bb3:                                              ; preds = %bb2
-  %tmp4 = load float, ptr %scevgep1, align 8
-  store float %tmp4, ptr %scevgep, align 8
-  br label %bb5
-
-bb5:                                              ; preds = %bb3
-  %indvar.next = add i64 %indvar, 1
-  br label %bb2
-
-bb6:                                              ; preds = %bb2
-  ret void
-}
-
-define i32 @main() nounwind {
-bb:
-  call void @simple_vec_stride_x()
-  %tmp = load float, ptr getelementptr inbounds ([1024 x float], ptr @A, i64 0, i64 42), align 8
-  %tmp1 = fptosi float %tmp to i32
-  ret i32 %tmp1
-}
-
-; CHECK: [[LOAD1:%[a-zA-Z0-9_]+_scalar_]] = load float, ptr
-; CHECK: [[VEC1:%[a-zA-Z0-9_]+]] = insertelement <4 x float> undef, float [[LOAD1]], i32 0
-; CHECK: [[LOAD2:%[a-zA-Z0-9_]+]] = load float, ptr
-; CHECK: [[VEC2:%[a-zA-Z0-9_]+]] = insertelement <4 x float> [[VEC1]], float [[LOAD2]], i32 1
-; CHECK: [[LOAD3:%[a-zA-Z0-9_]+]] = load float, ptr
-; CHECK: [[VEC3:%[a-zA-Z0-9_]+]] = insertelement <4 x float> [[VEC2]], float [[LOAD3]], i32 2
-; CHECK: [[LOAD4:%[a-zA-Z0-9_]+]] = load float, ptr
-; CHECK: [[VEC4:%[a-zA-Z0-9_]+]] = insertelement <4 x float> [[VEC3]], float [[LOAD4]], i32 3
-; CHECK: [[EL1:%[a-zA-Z0-9_]+]] = extractelement <4 x float> [[VEC4]], i32 0
-; CHECK: store float [[EL1]]
-; CHECK: [[EL2:%[a-zA-Z0-9_]+]] = extractelement <4 x float> [[VEC4]], i32 1
-; CHECK: store float [[EL2]]
-; CHECK: [[EL3:%[a-zA-Z0-9_]+]] = extractelement <4 x float> [[VEC4]], i32 2
-; CHECK: store float [[EL3]]
-; CHECK: [[EL4:%[a-zA-Z0-9_]+]] = extractelement <4 x float> [[VEC4]], i32 3
-; CHECK: store float [[EL4]]
diff --git a/polly/test/CodeGen/simple_vec_strides_multidim.ll b/polly/test/CodeGen/simple_vec_strides_multidim.ll
deleted file mode 100644
index 98837af2fbc82..0000000000000
--- a/polly/test/CodeGen/simple_vec_strides_multidim.ll
+++ /dev/null
@@ -1,76 +0,0 @@
-; RUN: opt %loadPolly -polly-opt-isl -polly-codegen -polly-vectorizer=polly -polly-prevect-width=8 -S -dce < %s | FileCheck %s
-;
-;    void foo(long n, float A[restrict][n], float B[restrict][n],
-;             float C[restrict][n], float D[restrict][n]) {
-;      for (long i = 0; i < 8; i++)
-;        for (long j = 0; j < 8; j++)
-;          A[i][j] += B[i][0] + C[i][2 * j] + D[j][0];
-;    }
-;
-
-; CHECK: shufflevector
-; CHECK: insertelement
-; CHECK: insertelement
-; CHECK: insertelement
-; CHECK: insertelement
-; CHECK: insertelement
-; CHECK: insertelement
-; CHECK: insertelement
-; CHECK: insertelement
-; CHECK: store <8 x float>
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @foo(i64 %n, ptr noalias %A, ptr noalias %B, ptr noalias %C, ptr noalias %D) {
-bb:
-  br label %bb3
-
-bb3:                                              ; preds = %bb25, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp26, %bb25 ]
-  %exitcond2 = icmp ne i64 %i.0, 8
-  br i1 %exitcond2, label %bb4, label %bb27
-
-bb4:                                              ; preds = %bb3
-  br label %bb5
-
-bb5:                                              ; preds = %bb22, %bb4
-  %j.0 = phi i64 [ 0, %bb4 ], [ %tmp23, %bb22 ]
-  %exitcond = icmp ne i64 %j.0, 8
-  br i1 %exitcond, label %bb6, label %bb24
-
-bb6:                                              ; preds = %bb5
-  %tmp = mul nsw i64 %i.0, %n
-  %tmp7 = getelementptr inbounds float, ptr %B, i64 %tmp
-  %tmp8 = load float, ptr %tmp7, align 4
-  %tmp9 = shl nsw i64 %j.0, 1
-  %tmp10 = mul nsw i64 %i.0, %n
-  %.sum = add i64 %tmp10, %tmp9
-  %tmp11 = getelementptr inbounds float, ptr %C, i64 %.sum
-  %tmp12 = load float, ptr %tmp11, align 4
-  %tmp13 = fadd float %tmp8, %tmp12
-  %tmp14 = mul nsw i64 %j.0, %n
-  %tmp15 = getelementptr inbounds float, ptr %D, i64 %tmp14
-  %tmp16 = load float, ptr %tmp15, align 4
-  %tmp17 = fadd float %tmp13, %tmp16
-  %tmp18 = mul nsw i64 %i.0, %n
-  %.sum1 = add i64 %tmp18, %j.0
-  %tmp19 = getelementptr inbounds float, ptr %A, i64 %.sum1
-  %tmp20 = load float, ptr %tmp19, align 4
-  %tmp21 = fadd float %tmp20, %tmp17
-  store float %tmp21, ptr %tmp19, align 4
-  br label %bb22
-
-bb22:                                             ; preds = %bb6
-  %tmp23 = add nsw i64 %j.0, 1
-  br label %bb5
-
-bb24:                                             ; preds = %bb5
-  br label %bb25
-
-bb25:                                             ; preds = %bb24
-  %tmp26 = add nsw i64 %i.0, 1
-  br label %bb3
-
-bb27:                                             ; preds = %bb3
-  ret void
-}
diff --git a/polly/test/CodeGen/simple_vec_two_stmts.ll b/polly/test/CodeGen/simple_vec_two_stmts.ll
deleted file mode 100644
index 14c12c4887b53..0000000000000
--- a/polly/test/CodeGen/simple_vec_two_stmts.ll
+++ /dev/null
@@ -1,49 +0,0 @@
-; RUN: opt %loadPolly -basic-aa -polly-codegen -polly-vectorizer=polly -dce -S < %s | FileCheck %s
-target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
-
-@A = common global [1024 x float] zeroinitializer, align 16
-@B = common global [1024 x float] zeroinitializer, align 16
-@C = common global [1024 x float] zeroinitializer, align 16
-
-define void @simple_vec_stride_one() nounwind {
-bb0:
-  br label %bb1
-
-bb1:
-  %indvar = phi i64 [ %indvar.next, %bb4 ], [ 0, %bb0 ]
-  %scevgep = getelementptr [1024 x float], ptr @B, i64 0, i64 %indvar
-  %scevgep2 = getelementptr [1024 x float], ptr @C, i64 0, i64 %indvar
-  %scevgep1 = getelementptr [1024 x float], ptr @A, i64 0, i64 %indvar
-  %exitcond = icmp ne i64 %indvar, 4
-  br i1 %exitcond, label %bb2a, label %bb5
-
-bb2a:
-  %tmp1 = load float, ptr %scevgep1, align 4
-  store float %tmp1, ptr %scevgep, align 4
-  br label %bb2b
-
-bb2b:
-  %tmp2 = load float, ptr %scevgep1, align 4
-  store float %tmp2, ptr %scevgep2, align 4
-  br label %bb4
-
-bb4:
-  %indvar.next = add i64 %indvar, 1
-  br label %bb1
-
-bb5:
-  ret void
-}
-
-define i32 @main() nounwind {
-  call void @simple_vec_stride_one()
-  %1 = load float, ptr getelementptr inbounds ([1024 x float], ptr @A, i64 0, i64 42), align 8
-  %2 = fptosi float %1 to i32
-  ret i32 %2
-}
-
-; CHECK: [[LOAD1:%[a-zA-Z0-9_]+_full]] = load <4 x float>, ptr
-; CHECK: store <4 x float> [[LOAD1]]
-; CHECK: [[LOAD2:%[a-zA-Z0-9_]+_full]] = load <4 x float>, ptr
-; CHECK: store <4 x float> [[LOAD2]]
-
diff --git a/polly/test/CodeGen/single_loop_param_less_equal.ll b/polly/test/CodeGen/single_loop_param_less_equal.ll
index c6e7b986c8048..e63ee299a37cf 100644
--- a/polly/test/CodeGen/single_loop_param_less_equal.ll
+++ b/polly/test/CodeGen/single_loop_param_less_equal.ll
@@ -1,6 +1,6 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-print-ast -disable-output < %s | FileCheck %s
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen  -S < %s | FileCheck %s -check-prefix=CODEGEN
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen < %s | opt -opaque-pointers=0 -passes='print<loops>' -disable-output 2>&1 | FileCheck %s -check-prefix=LOOPS
+; RUN: opt %loadPolly -polly-print-ast -disable-output < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-codegen  -S < %s | FileCheck %s -check-prefix=CODEGEN
+; RUN: opt %loadPolly -polly-codegen < %s | opt -passes='print<loops>' -disable-output 2>&1 | FileCheck %s -check-prefix=LOOPS
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
 
 @A = common global [1024 x i32] zeroinitializer
@@ -13,12 +13,12 @@ start:
 
 loop.header:
   %i = phi i64 [ 0, %start ], [ %i.next, %loop.backedge ]
-  %scevgep = getelementptr [1024 x i32], [1024 x i32]* @A, i64 0, i64 %i
+  %scevgep = getelementptr [1024 x i32], ptr @A, i64 0, i64 %i
   %exitcond = icmp ne i64 %i, %n_plus_one
   br i1 %exitcond, label %loop.body, label %ret
 
 loop.body:
-  store i32 1, i32* %scevgep
+  store i32 1, ptr %scevgep
   br label %loop.backedge
 
 loop.backedge:
@@ -48,8 +48,9 @@ ret:
 ; CODEGEN:   br label %polly.stmt.loop.body
 
 ; CODEGEN: polly.stmt.loop.body:
-; CODEGEN:   [[PTR:%[a-zA-Z0-9_\.]+]] = getelementptr [1024 x i32], [1024 x i32]* @A, i64 0, i64 %polly.indvar
-; CODEGEN:   store i32 1, i32* [[PTR]]
+; CODEGEN:   %[[offset:.*]] = shl i64 %polly.indvar, 2
+; CODEGEN:   [[PTR:%[a-zA-Z0-9_\.]+]] = getelementptr i8, ptr @A, i64 %[[offset]]
+; CODEGEN:   store i32 1, ptr [[PTR]]
 ; CODEGEN:   %polly.indvar_next = add nsw i64 %polly.indvar, 1
 ; CODEGEN:   %polly.loop_cond = icmp sle i64 %polly.indvar_next, %n
 ; CODEGEN:   br i1 %polly.loop_cond, label %polly.loop_header, label %polly.loop_exit
diff --git a/polly/test/CodeGen/single_loop_param_less_than.ll b/polly/test/CodeGen/single_loop_param_less_than.ll
index fcd79d0a1734d..95130f9264501 100644
--- a/polly/test/CodeGen/single_loop_param_less_than.ll
+++ b/polly/test/CodeGen/single_loop_param_less_than.ll
@@ -1,5 +1,5 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-print-ast -disable-output < %s | FileCheck %s
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen  -S < %s | FileCheck %s -check-prefix=CODEGEN
+; RUN: opt %loadPolly -polly-print-ast -disable-output < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-codegen  -S < %s | FileCheck %s -check-prefix=CODEGEN
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
 
 @A = common global [1024 x i32] zeroinitializer
@@ -11,12 +11,12 @@ start:
 
 loop.header:
   %i = phi i64 [ 0, %start ], [ %i.next, %loop.backedge ]
-  %scevgep = getelementptr [1024 x i32], [1024 x i32]* @A, i64 0, i64 %i
+  %scevgep = getelementptr [1024 x i32], ptr @A, i64 0, i64 %i
   %exitcond = icmp ne i64 %i, %n
   br i1 %exitcond, label %loop.body, label %ret
 
 loop.body:
-  store i32 1, i32* %scevgep
+  store i32 1, ptr %scevgep
   br label %loop.backedge
 
 loop.backedge:
@@ -46,8 +46,9 @@ ret:
 ; CODEGEN:   br label %polly.stmt.loop.body
 
 ; CODEGEN: polly.stmt.loop.body:
-; CODEGEN:   [[PTR:%[a-zA-Z0-9_\.]+]] =  getelementptr [1024 x i32], [1024 x i32]* @A, i64 0, i64 %polly.indvar
-; CODEGEN:   store i32 1, i32* [[PTR]]
+; CODEGEN:   %[[offset:.*]] = shl i64 %polly.indvar, 2
+; CODEGEN:   [[PTR:%[a-zA-Z0-9_\.]+]] = getelementptr i8, ptr @A, i64 %[[offset]]
+; CODEGEN:   store i32 1, ptr [[PTR]]
 ; CODEGEN:   %polly.indvar_next = add nsw i64 %polly.indvar, 1
 ; CODEGEN:   %polly.loop_cond = icmp slt i64 %polly.indvar_next, %n
 ; CODEGEN:   br i1 %polly.loop_cond, label %polly.loop_header, label %polly.loop_exit
diff --git a/polly/test/CodeGen/stmt_split_no_dependence.ll b/polly/test/CodeGen/stmt_split_no_dependence.ll
index 5c6a0a362619d..a395aa14b4c8a 100644
--- a/polly/test/CodeGen/stmt_split_no_dependence.ll
+++ b/polly/test/CodeGen/stmt_split_no_dependence.ll
@@ -1,7 +1,7 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen -S < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-codegen -S < %s | FileCheck %s
 ;
-; CHECK:   store i32 %8, i32* %scevgep, align 4, !alias.scope !1, !noalias !4
-; CHECK:   store i32 %9, i32* %scevgep4, align 4, !alias.scope !4, !noalias !1
+; CHECK:   store i32 %9, ptr %scevgep, align 4, !alias.scope !1, !noalias !4
+; CHECK:   store i32 %11, ptr %scevgep4, align 4, !alias.scope !4, !noalias !1
 ;
 ;      void func(int *A, int *B){
 ;        for (int i = 0; i < 1024; i+=1) {
@@ -12,7 +12,7 @@
 ;      }
 ;
 ; Function Attrs: noinline nounwind uwtable
-define void @func(i32* %A, i32* %B) #0 {
+define void @func(ptr %A, ptr %B) #0 {
 entry:
   br label %for.cond
 
@@ -26,11 +26,11 @@ for.body: 					 ; preds = %for.cond
 
 Stmt: 						 ; preds = %for.body
   %idxprom = sext i32 %i.0 to i64
-  %arrayidx = getelementptr inbounds i32, i32* %A, i64 %idxprom
-  store i32 %i.0, i32* %arrayidx, align 4, !polly_split_after !0
+  %arrayidx = getelementptr inbounds i32, ptr %A, i64 %idxprom
+  store i32 %i.0, ptr %arrayidx, align 4, !polly_split_after !0
   %idxprom1 = sext i32 %i.0 to i64
-  %arrayidx2 = getelementptr inbounds i32, i32* %B, i64 %idxprom1
-  store i32 %i.0, i32* %arrayidx2, align 4
+  %arrayidx2 = getelementptr inbounds i32, ptr %B, i64 %idxprom1
+  store i32 %i.0, ptr %arrayidx2, align 4
   br label %for.inc
 
 for.inc: 					 ; preds = %Stmt
diff --git a/polly/test/CodeGen/stride_detection.ll b/polly/test/CodeGen/stride_detection.ll
deleted file mode 100644
index 3f9044baf599d..0000000000000
--- a/polly/test/CodeGen/stride_detection.ll
+++ /dev/null
@@ -1,56 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-opt-isl -polly-vectorizer=polly -polly-codegen < %s -S | FileCheck %s
-
-; #pragma known-parallel
-;   for (int c0 = 0; c0 <= 31; c0 += 1)
-;     for (int c1 = 0; c1 <= floord(nk - 1, 32); c1 += 1)
-;       for (int c2 = 0; c2 <= 7; c2 += 1)
-;         for (int c3 = 0; c3 <= min(31, nk - 32 * c1 - 1); c3 += 1)
-;           #pragma simd
-;           for (int c4 = 0; c4 <= 3; c4 += 1)
-;             Stmt_for_body_3(32 * c0 + 4 * c2 + c4, 32 * c1 + c3);
-
-; CHECK: polly.stmt.for.body.3:                            ; preds = %polly.loop_header18
-; CHECK:   %_p_splat_one = load <1 x double>, <1 x double>* %_p_vec_p, align 8, !alias.scope !3, !noalias !6, !llvm.access.group !2
-; CHECK:   %_p_vec_full = load <4 x double>, <4 x double>* %vector_ptr, align 8, !alias.scope !6, !noalias !3, !llvm.access.group !2
-; CHECK:   extractelement <4 x double> %addp_vec, i32 0
-; CHECK:   extractelement <4 x double> %addp_vec, i32 1
-; CHECK:   extractelement <4 x double> %addp_vec, i32 2
-; CHECK:   extractelement <4 x double> %addp_vec, i32 3
-; CHECK:   store <4 x double> %addp_vec, <4 x double>* {{.*}}, align 8, !alias.scope !6, !noalias !3, !llvm.access.group !2
-
-define void @kernel_gemm(i32 %ni, i32 %nj, i32 %nk, [1024 x double]* %C, [1024 x double]* %A) #0 {
-entry:
-  br label %for.cond.1.preheader
-
-for.cond.1.preheader:                             ; preds = %entry, %for.inc.10
-  %indvars.iv16 = phi i64 [ 0, %entry ], [ %indvars.iv.next17, %for.inc.10 ]
-  %cmp2.13 = icmp sgt i32 %nk, 0
-  br i1 %cmp2.13, label %for.body.3.lr.ph, label %for.inc.10
-
-for.body.3.lr.ph:                                 ; preds = %for.cond.1.preheader
-  br label %for.body.3
-
-for.body.3:                                       ; preds = %for.body.3.lr.ph, %for.body.3
-  %indvars.iv = phi i64 [ 0, %for.body.3.lr.ph ], [ %indvars.iv.next, %for.body.3 ]
-  %arrayidx5 = getelementptr inbounds [1024 x double], [1024 x double]* %A, i64 0, i64 %indvars.iv
-  %0 = load double, double* %arrayidx5, align 8
-  %arrayidx9 = getelementptr inbounds [1024 x double], [1024 x double]* %C, i64 0, i64 %indvars.iv16
-  %1 = load double, double* %arrayidx9, align 8
-  %add = fadd double %0, %1
-  store double %add, double* %arrayidx9, align 8
-  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
-  %lftr.wideiv = trunc i64 %indvars.iv.next to i32
-  %exitcond = icmp ne i32 %lftr.wideiv, %nk
-  br i1 %exitcond, label %for.body.3, label %for.cond.1.for.inc.10_crit_edge
-
-for.cond.1.for.inc.10_crit_edge:                  ; preds = %for.body.3
-  br label %for.inc.10
-
-for.inc.10:                                       ; preds = %for.cond.1.for.inc.10_crit_edge, %for.cond.1.preheader
-  %indvars.iv.next17 = add nuw nsw i64 %indvars.iv16, 1
-  %exitcond18 = icmp ne i64 %indvars.iv.next17, 1024
-  br i1 %exitcond18, label %for.cond.1.preheader, label %for.end.12
-
-for.end.12:                                       ; preds = %for.inc.10
-  ret void
-}
diff --git a/polly/test/CodeGen/switch-in-non-affine-region.ll b/polly/test/CodeGen/switch-in-non-affine-region.ll
index d9e26efbca7ee..8eeabd980846d 100644
--- a/polly/test/CodeGen/switch-in-non-affine-region.ll
+++ b/polly/test/CodeGen/switch-in-non-affine-region.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly \
+; RUN: opt %loadPolly \
 ; RUN: -S -polly-codegen < %s | FileCheck %s
 ;
 ;    void f(int *A, int N) {
@@ -15,8 +15,8 @@
 ;    }
 ;
 ; CHECK: polly.stmt.if.then:
-; CHECK:   %1 = trunc i64 %polly.indvar to i32
-; CHECK:   %p_rem = srem i32 %1, 4
+; CHECK:   %[[trunc:.*]] = trunc i64 %polly.indvar to i32
+; CHECK:   %p_rem = srem i32 %[[trunc]], 4
 ; CHECK:   switch i32 %p_rem, label %polly.stmt.sw.epilog [
 ; CHECK:     i32 0, label %polly.stmt.sw.bb
 ; CHECK:     i32 1, label %polly.stmt.sw.bb.3
@@ -24,7 +24,7 @@
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
-define void @f(i32* %A, i32 %N) {
+define void @f(ptr %A, i32 %N) {
 entry:
   %tmp = sext i32 %N to i64
   br label %for.cond
@@ -35,8 +35,8 @@ for.cond:                                         ; preds = %for.inc, %entry
   br i1 %cmp, label %for.body, label %for.end
 
 for.body:                                         ; preds = %for.cond
-  %arrayidx = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  %tmp1 = load i32, i32* %arrayidx, align 4
+  %arrayidx = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  %tmp1 = load i32, ptr %arrayidx, align 4
   %tobool = icmp eq i32 %tmp1, 0
   br i1 %tobool, label %if.end, label %if.then
 
@@ -49,17 +49,17 @@ if.then:                                          ; preds = %for.body
   ]
 
 sw.bb:                                            ; preds = %if.then
-  %arrayidx2 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  %tmp3 = load i32, i32* %arrayidx2, align 4
+  %arrayidx2 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  %tmp3 = load i32, ptr %arrayidx2, align 4
   %add = add nsw i32 %tmp3, 1
-  store i32 %add, i32* %arrayidx2, align 4
+  store i32 %add, ptr %arrayidx2, align 4
   br label %sw.epilog
 
 sw.bb.3:                                          ; preds = %if.then
-  %arrayidx5 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  %tmp4 = load i32, i32* %arrayidx5, align 4
+  %arrayidx5 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
+  %tmp4 = load i32, ptr %arrayidx5, align 4
   %add6 = add nsw i32 %tmp4, 2
-  store i32 %add6, i32* %arrayidx5, align 4
+  store i32 %add6, ptr %arrayidx5, align 4
   br label %sw.epilog
 
 sw.epilog:                                        ; preds = %sw.bb.3, %sw.bb, %if.then
diff --git a/polly/test/GPGPU/Inputs/libdevice-functions-copied-into-kernel_libdevice.ll b/polly/test/GPGPU/Inputs/libdevice-functions-copied-into-kernel_libdevice.ll
deleted file mode 100644
index 3f4c4a0aa6106..0000000000000
--- a/polly/test/GPGPU/Inputs/libdevice-functions-copied-into-kernel_libdevice.ll
+++ /dev/null
@@ -1,9 +0,0 @@
-define float @__nv_expf(float %a) {
-  ret float %a
-}
-define float @__nv_cosf(float %a) {
-  ret float %a
-}
-define float @__nv_logf(float %a) {
-  ret float %a
-}
diff --git a/polly/test/GPGPU/add-scalars-in-scop-to-kills.ll b/polly/test/GPGPU/add-scalars-in-scop-to-kills.ll
deleted file mode 100644
index 64b4cc4aa100b..0000000000000
--- a/polly/test/GPGPU/add-scalars-in-scop-to-kills.ll
+++ /dev/null
@@ -1,71 +0,0 @@
-; RUN: opt %loadPolly -polly-print-scops -disable-output < %s | FileCheck %s -check-prefix=SCOP
-; RUN: opt %loadPolly -S -polly-codegen-ppcg < %s | FileCheck %s -check-prefix=HOST-IR
-
-; REQUIRES: pollyacc
-
-; Check that we detect a scop.
-; SCOP:       Function: checkScalarKill
-; SCOP-NEXT: Region: %XLoopInit---%for.end
-; SCOP-NEXT: Max Loop Depth:  1
-
-; Check that we have a scalar that is not a phi node in the scop.
-; SCOP: i32 MemRef_x_0; // Element size 4
-
-; Check that kernel launch is generated in host IR.
-; the declare would not be generated unless a call to a kernel exists.
-; HOST-IR: declare void @polly_launchKernel(ptr, i32, i32, i32, i32, i32, ptr)
-
-; Check that we add variables that are local to a scop into the kills that we
-; pass to PPCG. This should enable PPCG to codegen this example.
-; void checkScalarKill(int A[], int B[], int C[], const int control1, int control2) {
-; int x;
-; #pragma scop
-;     for(int i = 0; i < 1000; i++) {
-; XLoopInit:        x = 0;
-;
-;         if (control1 > 2)
-;             C1Add: x += 10;
-;         if (control2 > 3)
-;             C2Add: x += A[i];
-;
-; BLoopAccumX:        B[i] += x;
-;     }
-;
-; #pragma endscop
-; }
-; ModuleID = 'test.ll'
-target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @checkScalarKill(ptr %A, ptr %B, ptr %C, i32 %control1, i32 %control2) {
-entry:
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  br label %XLoopInit
-
-XLoopInit:                                        ; preds = %entry.split, %BLoopAccumX
-  %indvars.iv = phi i64 [ 0, %entry.split ], [ %indvars.iv.next, %BLoopAccumX ]
-  %cmp1 = icmp sgt i32 %control1, 2
-  %x.0 = select i1 %cmp1, i32 10, i32 0
-  %cmp2 = icmp sgt i32 %control2, 3
-  br i1 %cmp2, label %C2Add, label %BLoopAccumX
-
-C2Add:                                            ; preds = %XLoopInit
-  %arrayidx = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
-  %tmp6 = load i32, ptr %arrayidx, align 4
-  %add4 = add nsw i32 %tmp6, %x.0
-  br label %BLoopAccumX
-
-BLoopAccumX:                                      ; preds = %XLoopInit, %C2Add
-  %x.1 = phi i32 [ %add4, %C2Add ], [ %x.0, %XLoopInit ]
-  %arrayidx7 = getelementptr inbounds i32, ptr %B, i64 %indvars.iv
-  %tmp11 = load i32, ptr %arrayidx7, align 4
-  %add8 = add nsw i32 %tmp11, %x.1
-  store i32 %add8, ptr %arrayidx7, align 4
-  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
-  %exitcond = icmp ne i64 %indvars.iv.next, 1000
-  br i1 %exitcond, label %XLoopInit, label %for.end
-
-for.end:                                          ; preds = %BLoopAccumX
-  ret void
-}
diff --git a/polly/test/GPGPU/align-params-in-schedule.ll b/polly/test/GPGPU/align-params-in-schedule.ll
deleted file mode 100644
index fa9a8f3eb4e5d..0000000000000
--- a/polly/test/GPGPU/align-params-in-schedule.ll
+++ /dev/null
@@ -1,53 +0,0 @@
-; RUN: opt %loadPolly -S -polly-process-unprofitable -polly-codegen-ppcg \
-; RUN: -polly-invariant-load-hoisting -polly-ignore-parameter-bounds < %s | \
-; RUN: FileCheck %s
-
-; REQUIRES: pollyacc
-
-; CHECK: polly_launchKernel
-
-; Verify that this program compiles. At some point, this compilation crashed
-; due to insufficient parameters being available.
-
-source_filename = "bugpoint-output-4d01492.bc"
-target datalayout = "e-p:64:64:64-S128-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f16:16:16-f32:32:32-f64:64:64-f128:128:128-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
-target triple = "x86_64-unknown-linux-gnu"
-
-%struct.barney = type { ptr, i64, i64, [2 x %struct.widget] }
-%struct.widget = type { i64, i64, i64 }
-
-@global = external unnamed_addr global %struct.barney, align 32
-
-; Function Attrs: nounwind uwtable
-define void @wobble(ptr noalias %arg) #0 {
-bb:
-  %tmp = load i32, ptr %arg, align 4
-  br label %bb1
-
-bb1:                                              ; preds = %bb13, %bb
-  %tmp2 = phi i32 [ %tmp15, %bb13 ], [ 1, %bb ]
-  br label %bb3
-
-bb3:                                              ; preds = %bb3, %bb1
-  %tmp4 = load ptr, ptr @global, align 32
-  %tmp5 = sext i32 %tmp2 to i64
-  %tmp6 = load i64, ptr getelementptr inbounds (%struct.barney, ptr @global, i64 0, i32 3, i64 1, i32 0), align 8
-  %tmp7 = mul i64 %tmp6, %tmp5
-  %tmp8 = add i64 %tmp7, 0
-  %tmp9 = load i64, ptr getelementptr inbounds (%struct.barney, ptr @global, i64 0, i32 1), align 8
-  %tmp10 = add i64 %tmp8, %tmp9
-  %tmp11 = getelementptr i32, ptr %tmp4, i64 %tmp10
-  store i32 undef, ptr %tmp11, align 4
-  %tmp12 = icmp eq i32 0, 0
-  br i1 %tmp12, label %bb13, label %bb3
-
-bb13:                                             ; preds = %bb3
-  %tmp14 = icmp eq i32 %tmp2, %tmp
-  %tmp15 = add i32 %tmp2, 1
-  br i1 %tmp14, label %bb16, label %bb1
-
-bb16:                                             ; preds = %bb13
-  ret void
-}
-
-attributes #0 = { nounwind uwtable }
diff --git a/polly/test/GPGPU/array-with-elem-type-smaller-than-byte.ll b/polly/test/GPGPU/array-with-elem-type-smaller-than-byte.ll
deleted file mode 100644
index 12b872d551923..0000000000000
--- a/polly/test/GPGPU/array-with-elem-type-smaller-than-byte.ll
+++ /dev/null
@@ -1,50 +0,0 @@
-; RUN: opt %loadPolly -S -polly-codegen-ppcg \
-; RUN: -polly-use-llvm-names < %s
-; ModuleID = 'test/GPGPU/zero-size-array.ll'
-
-; REQUIRES: pollyacc
-
-target datalayout = "e-p:64:64:64-S128-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f16:16:16-f32:32:32-f64:64:64-f128:128:128-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
-target triple = "x86_64-unknown-linux-gnu"
-
-
-; We used to divide the element size by 8 to arrive at the 'actual' size
-; of an array element. This used to cause arrays that have an element size
-; of less than 8 to collapse to size 0. This test makes sure that it does
-; not happen anymore.
-
-; f(int *niters_ptr, int *arr[0]) {
-;     const int inters = *niters_ptr;
-;     for(int i = 0; i < niters; i++) {
-;       arr[0][i + 1] = 0
-;     }
-; }
-
-; Function Attrs: nounwind uwtable
-define void @f(ptr noalias %niters.ptr, ptr noalias %arr) #0 {
-entry:
-  %niters = load i32, ptr %niters.ptr, align 4
-  br label %loop.body
-
-loop.body:                                             ; preds = %loop.body, %entry
-  %indvar = phi i32 [ %indvar.next, %loop.body ], [ 1, %entry ]
-  %indvar.sext = sext i32 %indvar to i64
-  %arr.slot = getelementptr [0 x i32], ptr %arr, i64 0, i64 %indvar.sext
-  store i32 0, ptr %arr.slot, align 4
-  %tmp8 = icmp eq i32 %indvar, %niters
-  %indvar.next = add i32 %indvar, 1
-  br i1 %tmp8, label %loop.exit, label %loop.body
-
-loop.exit:                                    ; preds = %loop.body
-  %tmp10 = icmp sgt i32 undef, 0
-  br label %auxiliary.loop
-
-auxiliary.loop:                                            ; preds = %"101", %loop.exit
-  %tmp11 = phi i1 [ %tmp10, %loop.exit ], [ undef, %auxiliary.loop ]
-  br i1 undef, label %auxiliary.loop, label %exit
-
-exit:                              ; preds = %auxiliary.loop
-  ret void
-}
-
-attributes #0 = { nounwind uwtable }
diff --git a/polly/test/GPGPU/bounds-construction-with-ignore-param-bounds.ll b/polly/test/GPGPU/bounds-construction-with-ignore-param-bounds.ll
deleted file mode 100644
index a60744289885e..0000000000000
--- a/polly/test/GPGPU/bounds-construction-with-ignore-param-bounds.ll
+++ /dev/null
@@ -1,55 +0,0 @@
-; RUN: opt %loadPolly -S -polly-codegen-ppcg \
-; RUN: -polly-ignore-parameter-bounds \
-; RUN: -polly-invariant-load-hoisting < %s| FileCheck %s -check-prefix=HOST-IR
-;
-; REQUIRES: pollyacc
-
-; When we have `-polly-ignore-parameter-bounds`, `Scop::Context` does not contain
-; all the parameters present in the program.
-;
-; The construction of the `isl_multi_pw_aff` requires all the indivisual `pw_aff`
-; to have the same parameter dimensions. To achieve this, we used to realign
-; every `pw_aff` with `Scop::Context`. However, in conjunction with
-; `-polly-ignore-parameter-bounds`, this is now incorrect, since `Scop::Context`
-; does not contain all parameters.
-;
-; We check that Polly does the right thing in this case and sets up the parameter
-; dimensions correctly.
-
-
-; Check that kernel launch is generated in host IR.
-; the declare would not be generated unless a call to a kernel exists.
-; HOST-IR: declare void @polly_launchKernel(ptr, i32, i32, i32, i32, i32, ptr)
-; ModuleID = 'test/GPGPU/bounds-construction-with-ignore-param-bounds.ll'
-
-; C pseudocode
-; ------------
-; void f(int *arr, long niters, long stride) {
-;     for(int i = 0; i < niters; i++) {
-;       arr[i * stride] = 1;
-;     }
-; }
-
-target datalayout = "e-p:64:64:64-S128-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f16:16:16-f32:32:32-f64:64:64-f128:128:128-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
-target triple = "x86_64-unknown-linux-gnu"
-
-; Function Attrs: nounwind uwtable
-define void @f(ptr %arr, i64 %niters, i64 %stride) unnamed_addr #1 {
-entry:
-  br label %loop
-
-loop:                                             ; preds = %loop, %entry
-  %indvar = phi i64 [ 0, %entry ], [ %indvar.next, %loop ]
-  %idx = mul nuw nsw i64 %indvar, %stride
-  %slot = getelementptr i32, ptr %arr, i64 %idx
-  store i32 1, ptr %slot, align 4
-  %indvar.next = add nuw nsw i64 %indvar, 1
-  %check = icmp sgt i64 %indvar.next, %niters
-  br i1 %check, label %exit, label %loop
-
-exit:                                             ; preds = %loop
-  ret void
-}
-
-attributes #0 = { nounwind }
-attributes #1 = { nounwind uwtable }
diff --git a/polly/test/GPGPU/cuda-annotations.ll b/polly/test/GPGPU/cuda-annotations.ll
deleted file mode 100644
index cbb0296d48ef6..0000000000000
--- a/polly/test/GPGPU/cuda-annotations.ll
+++ /dev/null
@@ -1,37 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-kernel-ir \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=KERNEL %s
-
-; REQUIRES: pollyacc
-
-; KERNEL: define ptx_kernel void @FUNC_foo_SCOP_0_KERNEL_0(ptr addrspace(1) %MemRef_A, i64 %n) #0 {
-
-; KERNEL: !nvvm.annotations = !{!0}
-
-; KERNEL: !0 = !{ptr @FUNC_foo_SCOP_0_KERNEL_0, !"maxntidx", i32 32, !"maxntidy", i32 1, !"maxntidz", i32 1}
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @foo(ptr %A, i64 %n) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb6, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp7, %bb6 ]
-  %tmp = icmp slt i64 %i.0, %n
-  br i1 %tmp, label %bb2, label %bb8
-
-bb2:                                              ; preds = %bb1
-  %tmp3 = getelementptr inbounds i64, ptr %A, i64 %i.0
-  %tmp4 = load i64, ptr %tmp3, align 8
-  %tmp5 = add nsw i64 %tmp4, 100
-  store i64 %tmp5, ptr %tmp3, align 8
-  br label %bb6
-
-bb6:                                              ; preds = %bb2
-  %tmp7 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb8:                                              ; preds = %bb1
-  ret void
-}
diff --git a/polly/test/GPGPU/cuda-managed-memory-simple.ll b/polly/test/GPGPU/cuda-managed-memory-simple.ll
deleted file mode 100644
index 8ef7e336cfad9..0000000000000
--- a/polly/test/GPGPU/cuda-managed-memory-simple.ll
+++ /dev/null
@@ -1,118 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -S  -polly-process-unprofitable -polly-acc-mincompute=0 -polly-codegen-ppcg -polly-acc-codegen-managed-memory < %s | \
-; RUN: FileCheck %s
-
-; REQUIRES: pollyacc
-
-;
-;    #include <cuda_runtime.h>
-;
-;    static const int N = 45;
-;
-;    void copy(int *R, int *A) {
-;      for (int i = 0; i < N; i++) {
-;        R[i] = A[i] * 10;
-;      }
-;    }
-;
-;    int main() {
-;      int *A, *R;
-;
-;      cudaMallocManaged((void **)(&A), sizeof(int) * N, cudaMemAttachGlobal);
-;      cudaMallocManaged((void **)(&R), sizeof(int) * N, cudaMemAttachGlobal);
-;
-;      for (int i = 0; i < N; i++) {
-;        A[i] = i;
-;        R[i] = 0;
-;      }
-;      copy(R, A);
-;
-;      return 0;
-;    }
-;
-
-; CHECK-NOT: polly_copyFromHostToDevice
-; CHECK-NOT: polly_copyFromDeviceToHost
-; CHECK-NOT: polly_freeDeviceMemory
-; CHECK-NOT: polly_allocateMemoryForDevice
-
-; CHECK:       %[[REGCTX:[0-9]+]] = call i8* @polly_initContextCUDA()
-; CHECK-NEXT:  %[[REGCA:[0-9]+]] = bitcast i32* %A to i8*
-; CHECK-NEXT:  %[[REGCR:[0-9]+]] = bitcast i32* %R to i8*
-; CHECK-NEXT:  %[[REGGEP0:[0-9]+]] = getelementptr [2 x i8*], [2 x i8*]* %polly_launch_0_params, i64 0, i64 0
-; CHECK-NEXT:  store i8* %[[REGCA]], i8** %polly_launch_0_param_0
-; CHECK-NEXT:  %[[REGCP0:[0-9]+]] = bitcast i8** %polly_launch_0_param_0 to i8*
-; CHECK-NEXT:  store i8* %[[REGCP0]], i8** %[[REGGEP0]]
-; CHECK-NEXT:  %[[REGGEP1:[0-9]+]] = getelementptr [2 x i8*], [2 x i8*]* %polly_launch_0_params, i64 0, i64 1
-; CHECK-NEXT:  store i8* %[[REGCR]], i8** %polly_launch_0_param_1
-; CHECK-NEXT:  %[[REGCP1:[0-9]+]] = bitcast i8** %polly_launch_0_param_1 to i8*
-; CHECK-NEXT:  store i8* %[[REGCP1]], i8** %[[REGGEP1]]
-; CHECK-NEXT:  %[[REGKERNEL:[0-9]+]] = call i8* @polly_getKernel(i8* getelementptr inbounds ([863 x i8], [863 x i8]* @FUNC_copy_SCOP_0_KERNEL_0, i32 0, i32 0), i8* getelementptr inbounds ([26 x i8], [26 x i8]* @FUNC_copy_SCOP_0_KERNEL_0_name, i32 0, i32 0))
-; CHECK-NEXT:  call void @polly_launchKernel(i8* %[[REGKERNEL]], i32 2, i32 1, i32 32, i32 1, i32 1, i8* %polly_launch_0_params_i8ptr)
-; CHECK-NEXT:  call void @polly_freeKernel(i8* %[[REGKERNEL]])
-; CHECK-NEXT:  call void @polly_synchronizeDevice()
-; CHECK-NEXT:  call void @polly_freeContext(i8* %[[REGCTX]])
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @copy(i32* %R, i32* %A) {
-entry:
-  br label %for.cond
-
-for.cond:                                         ; preds = %for.inc, %entry
-  %indvars.iv = phi i64 [ %indvars.iv.next, %for.inc ], [ 0, %entry ]
-  %exitcond = icmp ne i64 %indvars.iv, 45
-  br i1 %exitcond, label %for.body, label %for.end
-
-for.body:                                         ; preds = %for.cond
-  %arrayidx = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
-  %tmp = load i32, i32* %arrayidx, align 4
-  %mul = mul nsw i32 %tmp, 10
-  %arrayidx2 = getelementptr inbounds i32, i32* %R, i64 %indvars.iv
-  store i32 %mul, i32* %arrayidx2, align 4
-  br label %for.inc
-
-for.inc:                                          ; preds = %for.body
-  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
-  br label %for.cond
-
-for.end:                                          ; preds = %for.cond
-  ret void
-}
-
-define i32 @main() {
-entry:
-  %A = alloca i32*, align 8
-  %R = alloca i32*, align 8
-  %tmp = bitcast i32** %A to i8**
-  %call = call i32 @cudaMallocManaged(i8** nonnull %tmp, i64 180, i32 1) #2
-  %tmp1 = bitcast i32** %R to i8**
-  %call1 = call i32 @cudaMallocManaged(i8** nonnull %tmp1, i64 180, i32 1) #2
-  br label %for.cond
-
-for.cond:                                         ; preds = %for.inc, %entry
-  %indvars.iv = phi i64 [ %indvars.iv.next, %for.inc ], [ 0, %entry ]
-  %exitcond = icmp ne i64 %indvars.iv, 45
-  br i1 %exitcond, label %for.body, label %for.end
-
-for.body:                                         ; preds = %for.cond
-  %tmp2 = load i32*, i32** %A, align 8
-  %arrayidx = getelementptr inbounds i32, i32* %tmp2, i64 %indvars.iv
-  %tmp3 = trunc i64 %indvars.iv to i32
-  store i32 %tmp3, i32* %arrayidx, align 4
-  %tmp4 = load i32*, i32** %R, align 8
-  %arrayidx3 = getelementptr inbounds i32, i32* %tmp4, i64 %indvars.iv
-  store i32 0, i32* %arrayidx3, align 4
-  br label %for.inc
-
-for.inc:                                          ; preds = %for.body
-  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
-  br label %for.cond
-
-for.end:                                          ; preds = %for.cond
-  %tmp5 = load i32*, i32** %R, align 8
-  %tmp6 = load i32*, i32** %A, align 8
-  call void @copy(i32* %tmp5, i32* %tmp6)
-  ret i32 0
-}
-
-declare i32 @cudaMallocManaged(i8**, i64, i32) #1
diff --git a/polly/test/GPGPU/debug-metadata-leak.ll b/polly/test/GPGPU/debug-metadata-leak.ll
deleted file mode 100644
index c90926c318e81..0000000000000
--- a/polly/test/GPGPU/debug-metadata-leak.ll
+++ /dev/null
@@ -1,104 +0,0 @@
-; RUN: opt %loadPolly %s -polly-process-unprofitable -polly-codegen-ppcg -polly-acc-dump-kernel-ir \
-; RUN: | FileCheck --check-prefix=KERNEL-IR %s
-
-; REQUIRES: pollyacc
-
-; KERNEL-IR: define ptx_kernel void @FUNC_vec_add_1_SCOP_0_KERNEL_0(ptr addrspace(1) %MemRef_arr, i32 %N) #0 {
-
-; The instruction marked <<<LeakyInst>>> is copied into the GPUModule,
-; with changes only to the parameters to access data on the device instead of
-; the host, i.e., MemRef_arr becomes polly.access.cast.MemRef_arr. Since the
-; instruction is annotated with a DILocation, copying the instruction also copies
-; the metadata into the GPUModule. This stops codegenerating the ptx_kernel by
-; failing the verification of the Module in GPUNodeBuilder::finalize, due to the
-; copied DICompileUnit not being listed in a llvm.dbg.cu which was neither copied
-; nor created.
-;
-; https://reviews.llvm.org/D35630 removes this debug metadata before the
-; instruction is copied to the GPUModule.
-;
-; vec_add_1.c:
-;      void vec_add_1(int N, int arr[N]) {
-;        int i=0;
-;        for( i=0 ; i<N ; i++) arr[i] += 1;
-;      }
-;
-source_filename = "vec_add_1.c"
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-target triple = "x86_64-unknown-linux-gnu"
-
-define void @vec_add_1(i32 %N, ptr %arr) !dbg !7 {
-entry:
-  call void @llvm.dbg.value(metadata i32 %N, i64 0, metadata !13, metadata !16), !dbg !17
-  call void @llvm.dbg.value(metadata ptr %arr, i64 0, metadata !14, metadata !16), !dbg !18
-  call void @llvm.dbg.value(metadata i32 0, i64 0, metadata !15, metadata !16), !dbg !19
-  %tmp = sext i32 %N to i64, !dbg !20
-  br label %for.cond, !dbg !20
-
-for.cond:                                         ; preds = %for.inc, %entry
-  %indvars.iv = phi i64 [ %indvars.iv.next, %for.inc ], [ 0, %entry ]
-  call void @llvm.dbg.value(metadata i32 undef, i64 0, metadata !15, metadata !16), !dbg !19
-  %cmp = icmp slt i64 %indvars.iv, %tmp, !dbg !22
-  br i1 %cmp, label %for.body, label %for.end, !dbg !24
-
-for.body:                                         ; preds = %for.cond
-  %arrayidx = getelementptr inbounds i32, ptr %arr, i64 %indvars.iv, !dbg !25
-  %tmp1 = load i32, ptr %arrayidx, align 4, !dbg !26, !tbaa !27
-  %add = add nsw i32 %tmp1, 1, !dbg !26    ;   <<<LeakyInst>>>
-  store i32 %add, ptr %arrayidx, align 4, !dbg !26, !tbaa !27
-  br label %for.inc, !dbg !25
-
-for.inc:                                          ; preds = %for.body
-  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1, !dbg !31
-  call void @llvm.dbg.value(metadata !2, i64 0, metadata !15, metadata !16), !dbg !19
-  br label %for.cond, !dbg !32, !llvm.loop !33
-
-for.end:                                          ; preds = %for.cond
-  ret void, !dbg !35
-}
-
-declare void @llvm.dbg.declare(metadata, metadata, metadata)
-
-declare void @llvm.dbg.value(metadata, i64, metadata, metadata)
-
-
-!llvm.dbg.cu = !{!0}
-!llvm.module.flags = !{!3, !4, !5}
-!llvm.ident = !{!6}
-
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 5.0.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2)
-!1 = !DIFile(filename: "vec_add_1.c", directory: "/tmp")
-!2 = !{}
-!3 = !{i32 2, !"Dwarf Version", i32 4}
-!4 = !{i32 2, !"Debug Info Version", i32 3}
-!5 = !{i32 1, !"wchar_size", i32 4}
-!6 = !{!"clang version 5.0.0"}
-!7 = distinct !DISubprogram(name: "vec_add_1", scope: !1, file: !1, line: 1, type: !8, isLocal: false, isDefinition: true, scopeLine: 1, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !12)
-!8 = !DISubroutineType(types: !9)
-!9 = !{null, !10, !11}
-!10 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!11 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !10, size: 64)
-!12 = !{!13, !14, !15}
-!13 = !DILocalVariable(name: "N", arg: 1, scope: !7, file: !1, line: 1, type: !10)
-!14 = !DILocalVariable(name: "arr", arg: 2, scope: !7, file: !1, line: 1, type: !11)
-!15 = !DILocalVariable(name: "i", scope: !7, file: !1, line: 2, type: !10)
-!16 = !DIExpression()
-!17 = !DILocation(line: 1, column: 20, scope: !7)
-!18 = !DILocation(line: 1, column: 27, scope: !7)
-!19 = !DILocation(line: 2, column: 7, scope: !7)
-!20 = !DILocation(line: 3, column: 8, scope: !21)
-!21 = distinct !DILexicalBlock(scope: !7, file: !1, line: 3, column: 3)
-!22 = !DILocation(line: 3, column: 15, scope: !23)
-!23 = distinct !DILexicalBlock(scope: !21, file: !1, line: 3, column: 3)
-!24 = !DILocation(line: 3, column: 3, scope: !21)
-!25 = !DILocation(line: 3, column: 25, scope: !23)
-!26 = !DILocation(line: 3, column: 32, scope: !23)
-!27 = !{!28, !28, i64 0}
-!28 = !{!"int", !29, i64 0}
-!29 = !{!"omnipotent char", !30, i64 0}
-!30 = !{!"Simple C/C++ TBAA"}
-!31 = !DILocation(line: 3, column: 21, scope: !23)
-!32 = !DILocation(line: 3, column: 3, scope: !23)
-!33 = distinct !{!33, !24, !34}
-!34 = !DILocation(line: 3, column: 35, scope: !21)
-!35 = !DILocation(line: 4, column: 1, scope: !7)
diff --git a/polly/test/GPGPU/double-parallel-loop.ll b/polly/test/GPGPU/double-parallel-loop.ll
deleted file mode 100644
index 4aeee035a4070..0000000000000
--- a/polly/test/GPGPU/double-parallel-loop.ll
+++ /dev/null
@@ -1,254 +0,0 @@
-; RUN: opt %loadPolly -polly-print-scops -disable-output < %s | FileCheck %s
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-schedule \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=SCHED %s
-
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-
-; RUN: opt %loadPolly -polly-codegen-ppcg -S < %s | \
-; RUN: FileCheck %s -check-prefix=IR
-
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-kernel-ir \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck %s -check-prefix=KERNEL-IR
-
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-kernel-asm \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck %s -check-prefix=KERNEL-ASM
-
-; XFAIL: *
-
-; REQUIRES: pollyacc, target=nvptx{{.*}}
-
-; This fails today due to extensive output differences from when the test was written.
-
-; CHECK: Stmt_bb5
-; CHECK-NEXT:       Domain :=
-; CHECK-NEXT:           { Stmt_bb5[i0, i1] : 0 <= i0 <= 1023 and 0 <= i1 <= 1023 };
-; CHECK-NEXT:       Schedule :=
-; CHECK-NEXT:           { Stmt_bb5[i0, i1] -> [i0, i1] };
-; CHECK-NEXT:       ReadAccess :=       [Reduction Type: NONE] [Scalar: 0]
-; CHECK-NEXT:           { Stmt_bb5[i0, i1] -> MemRef_A[i0, i1] };
-; CHECK-NEXT:       MustWriteAccess :=  [Reduction Type: NONE] [Scalar: 0]
-; CHECK-NEXT:           { Stmt_bb5[i0, i1] -> MemRef_A[i0, i1] };
-
-; SCHED: domain: "{ Stmt_bb5[i0, i1] : 0 <= i0 <= 1023 and 0 <= i1 <= 1023 }"
-; SCHED-NEXT: child:
-; SCHED-NEXT:   context: "{ [] }"
-; SCHED-NEXT:   child:
-; SCHED-NEXT:     extension: "{ [] -> from_device_MemRef_A[]; [] -> to_device_MemRef_A[] }"
-; SCHED-NEXT:     child:
-; SCHED-NEXT:       sequence:
-; SCHED-NEXT:       - filter: "{ to_device_MemRef_A[] }"
-; SCHED-NEXT:         child:
-; SCHED-NEXT:           set:
-; SCHED-NEXT:           - filter: "{ to_device_MemRef_A[] }"
-; SCHED-NEXT:             child:
-; SCHED-NEXT:               guard: "{ [] }"
-; SCHED-NEXT:       - filter: "{ Stmt_bb5[i0, i1] }"
-; SCHED-NEXT:         child:
-; SCHED-NEXT:           guard: "{ [] }"
-; SCHED-NEXT:           child:
-; SCHED-NEXT:             mark: "kernel"
-; SCHED-NEXT:             child:
-; SCHED-NEXT:               context: "[b0, b1, t0, t1] -> { [] : 0 <= b0 <= 31 and 0 <= b1 <= 31 and 0 <= t0 <= 31 and 0 <= t1 <= 15 }"
-; SCHED-NEXT:               child:
-; SCHED-NEXT:                 filter: "[b0, b1] -> { Stmt_bb5[i0, i1] : -31 - 32b0 + i0 <= 8192*floor((i0)/8192) <= -32b0 + i0 and -31 - 32b1 + i1 <= 8192*floor((i1)/8192) <= -32b1 + i1 }"
-; SCHED-NEXT:                 child:
-; SCHED-NEXT:                   schedule: "[{ Stmt_bb5[i0, i1] -> [(floor((i0)/8192))] }, { Stmt_bb5[i0, i1] -> [(floor((i1)/8192))] }]"
-; SCHED-NEXT:                   permutable: 1
-; SCHED-NEXT:                   coincident: [ 1, 1 ]
-; SCHED-NEXT:                   child:
-; SCHED-NEXT:                     filter: "[t0, t1] -> { Stmt_bb5[i0, i1] : 32*floor((-t0 + i0)/32) = -t0 + i0 and 16*floor((-t1 + i1)/16) = -t1 + i1 and 0 <= t0 <= 31 and 0 <= t1 <= 15 }"
-; SCHED-NEXT:                     child:
-; SCHED-NEXT:                       schedule: "[{ Stmt_bb5[i0, i1] -> [(0)] }, { Stmt_bb5[i0, i1] -> [(floor((i1)/16) - 2*floor((i1)/32))] }]"
-; SCHED-NEXT:                       permutable: 1
-; SCHED-NEXT:                       coincident: [ 1, 1 ]
-; SCHED-NEXT:       - filter: "{ from_device_MemRef_A[] }"
-; SCHED-NEXT:         child:
-; SCHED-NEXT:           set:
-; SCHED-NEXT:           - filter: "{ from_device_MemRef_A[] }"
-; SCHED-NEXT:             child:
-; SCHED-NEXT:               guard: "{ [] }"
-
-; CODE: Code
-; CODE-NEXT: ====
-; CODE-NEXT: # host
-; CODE-NEXT: {
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(dev_MemRef_A, MemRef_A, (1024) * (1024) * sizeof(float), cudaMemcpyHostToDevice));
-; CODE-NEXT:   {
-; CODE-NEXT:     dim3 k0_dimBlock(16, 32);
-; CODE-NEXT:     dim3 k0_dimGrid(32, 32);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_A);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:   cudaCheckReturn(cudaMemcpy(MemRef_A, dev_MemRef_A, (1024) * (1024) * sizeof(float), cudaMemcpyDeviceToHost));
-; CODE-NEXT: }
-
-; CODE: # kernel0
-; CODE-NEXT: for (int c3 = 0; c3 <= 1; c3 += 1)
-; CODE-NEXT:   Stmt_bb5(32 * b0 + t0, 32 * b1 + t1 + 16 * c3);
-
-; IR: polly.split_new_and_old:
-; IR-NEXT:   %0 = call { i64, i1 } @llvm.smul.with.overflow.i64(i64 1, i64 1024)
-; IR-NEXT:   %.obit = extractvalue { i64, i1 } %0, 1
-; IR-NEXT:   %polly.overflow.state = or i1 false, %.obit
-; IR-NEXT:   %.res = extractvalue { i64, i1 } %0, 0
-; IR-NEXT:   %1 = call { i64, i1 } @llvm.smul.with.overflow.i64(i64 %.res, i64 1024)
-; IR-NEXT:   %.obit1 = extractvalue { i64, i1 } %1, 1
-; IR-NEXT:   %polly.overflow.state2 = or i1 %polly.overflow.state, %.obit1
-; IR-NEXT:   %.res3 = extractvalue { i64, i1 } %1, 0
-; IR-NEXT:   %2 = call { i64, i1 } @llvm.smul.with.overflow.i64(i64 7, i64 %.res3)
-; IR-NEXT:   %.obit4 = extractvalue { i64, i1 } %2, 1
-; IR-NEXT:   %polly.overflow.state5 = or i1 %polly.overflow.state2, %.obit4
-; IR-NEXT:   %.res6 = extractvalue { i64, i1 } %2, 0
-; IR-NEXT:   %3 = call { i64, i1 } @llvm.sadd.with.overflow.i64(i64 0, i64 %.res6)
-; IR-NEXT:   %.obit7 = extractvalue { i64, i1 } %3, 1
-; IR-NEXT:   %polly.overflow.state8 = or i1 %polly.overflow.state5, %.obit7
-; IR-NEXT:   %.res9 = extractvalue { i64, i1 } %3, 0
-; IR-NEXT:   %4 = icmp sge i64 %.res9, 2621440
-; IR-NEXT:   %5 = and i1 true, %4
-; IR-NEXT:   %polly.rtc.overflown = xor i1 %polly.overflow.state8, true
-; IR-NEXT:   %polly.rtc.result = and i1 %5, %polly.rtc.overflown
-; IR-NEXT:   br i1 %polly.rtc.result, label %polly.start, label %bb2
-
-; IR: polly.start:
-; IR-NEXT: br label %polly.acc.initialize
-
-; IR: polly.acc.initialize:
-; IR-NEXT:    [[GPUContext:%.*]] = call ptr @polly_initContext()
-; IR-NEXT:    %p_dev_array_MemRef_A = call ptr @polly_allocateMemoryForDevice(i64 4194304)
-; IR-NEXT:    call void @polly_copyFromHostToDevice(ptr %A, ptr %p_dev_array_MemRef_A, i64 4194304)
-; IR-NEXT:    [[DevPtr:%.*]]  = call ptr @polly_getDevicePtr(ptr %p_dev_array_MemRef_A)
-; IR-NEXT:    store ptr [[DevPtr]], ptr %polly_launch_0_param_0
-; IR-NEXT:    store ptr %polly_launch_0_param_0, ptr %polly_launch_0_params
-; IR-NEXT:    call ptr @polly_getKernel
-; IR-NEXT:    call void @polly_launchKernel(ptr %11, i32 32, i32 32, i32 32, i32 16, i32 1, ptr %polly_launch_0_params_i8ptr)
-; IR-NEXT:    call void @polly_freeKernel
-; IR-NEXT:    call void @polly_copyFromDeviceToHost(ptr %p_dev_array_MemRef_A, ptr %A, i64 4194304)
-; IR-NEXT:    call void @polly_freeDeviceMemory(ptr %p_dev_array_MemRef_A)
-; IR-NEXT:    call void @polly_freeContext(ptr [[GPUContext]])
-; IR-NEXT:    br label %polly.exiting
-
-; IR: polly.exiting:
-; IR-NEXT:    br label %polly.merge_new_and_old
-
-; KERNEL-IR-LABEL: define ptx_kernel void @kernel_0(ptr %MemRef_A) #0 {
-; KERNEL-IR-NEXT: entry:
-; KERNEL-IR-NEXT:   %0 = call i32 @llvm.nvvm.read.ptx.sreg.ctaid.x()
-; KERNEL-IR-NEXT:   %b0 = zext i32 %0 to i64
-; KERNEL-IR-NEXT:   %1 = call i32 @llvm.nvvm.read.ptx.sreg.ctaid.y()
-; KERNEL-IR-NEXT:   %b1 = zext i32 %1 to i64
-; KERNEL-IR-NEXT:   %2 = call i32 @llvm.nvvm.read.ptx.sreg.tid.x()
-; KERNEL-IR-NEXT:   %t0 = zext i32 %2 to i64
-; KERNEL-IR-NEXT:   %3 = call i32 @llvm.nvvm.read.ptx.sreg.tid.y()
-; KERNEL-IR-NEXT:   %t1 = zext i32 %3 to i64
-; KERNEL-IR-NEXT:   br label %polly.loop_preheader
-
-; KERNEL-IR-LABEL: polly.loop_exit:                                  ; preds = %polly.stmt.bb5
-; KERNEL-IR-NEXT:   ret void
-
-; KERNEL-IR-LABEL: polly.loop_header:                                ; preds = %polly.stmt.bb5, %polly.loop_preheader
-; KERNEL-IR-NEXT:   %polly.indvar = phi i64 [ 0, %polly.loop_preheader ], [ %polly.indvar_next, %polly.stmt.bb5 ]
-; KERNEL-IR-NEXT:   %4 = mul nsw i64 32, %b0
-; KERNEL-IR-NEXT:   %5 = add nsw i64 %4, %t0
-; KERNEL-IR-NEXT:   %6 = mul nsw i64 32, %b1
-; KERNEL-IR-NEXT:   %7 = add nsw i64 %6, %t1
-; KERNEL-IR-NEXT:   %8 = mul nsw i64 16, %polly.indvar
-; KERNEL-IR-NEXT:   %9 = add nsw i64 %7, %8
-; KERNEL-IR-NEXT:   br label %polly.stmt.bb5
-
-; KERNEL-IR-LABEL: polly.stmt.bb5:                                   ; preds = %polly.loop_header
-; KERNEL-IR-NEXT:   %10 = mul i64 %5, %9
-; KERNEL-IR-NEXT:   %p_tmp6 = sitofp i64 %10 to float
-; KERNEL-IR-NEXT:   %11 = mul nsw i64 32, %b0
-; KERNEL-IR-NEXT:   %12 = add nsw i64 %11, %t0
-; KERNEL-IR-NEXT:   %polly.access.mul.MemRef_A = mul nsw i64 %12, 1024
-; KERNEL-IR-NEXT:   %13 = mul nsw i64 32, %b1
-; KERNEL-IR-NEXT:   %14 = add nsw i64 %13, %t1
-; KERNEL-IR-NEXT:   %15 = mul nsw i64 16, %polly.indvar
-; KERNEL-IR-NEXT:   %16 = add nsw i64 %14, %15
-; KERNEL-IR-NEXT:   %polly.access.add.MemRef_A = add nsw i64 %polly.access.mul.MemRef_A, %16
-; KERNEL-IR-NEXT:   %polly.access.MemRef_A = getelementptr float, ptr %MemRef_A, i64 %polly.access.add.MemRef_A
-; KERNEL-IR-NEXT:   %tmp8_p_scalar_ = load float, ptr %polly.access.MemRef_A, align 4
-; KERNEL-IR-NEXT:   %p_tmp9 = fadd float %tmp8_p_scalar_, %p_tmp6
-; KERNEL-IR-NEXT:   %17 = mul nsw i64 32, %b0
-; KERNEL-IR-NEXT:   %18 = add nsw i64 %17, %t0
-; KERNEL-IR-NEXT:   %polly.access.mul.MemRef_A2 = mul nsw i64 %18, 1024
-; KERNEL-IR-NEXT:   %19 = mul nsw i64 32, %b1
-; KERNEL-IR-NEXT:   %20 = add nsw i64 %19, %t1
-; KERNEL-IR-NEXT:   %21 = mul nsw i64 16, %polly.indvar
-; KERNEL-IR-NEXT:   %22 = add nsw i64 %20, %21
-; KERNEL-IR-NEXT:   %polly.access.add.MemRef_A3 = add nsw i64 %polly.access.mul.MemRef_A2, %22
-; KERNEL-IR-NEXT:   %polly.access.MemRef_A4 = getelementptr float, ptr %MemRef_A, i64 %polly.access.add.MemRef_A3
-; KERNEL-IR-NEXT:   store float %p_tmp9, ptr %polly.access.MemRef_A4, align 4
-; KERNEL-IR-NEXT:   %polly.indvar_next = add nsw i64 %polly.indvar, 1
-; KERNEL-IR-NEXT:   %polly.loop_cond = icmp sle i64 %polly.indvar, 0
-; KERNEL-IR-NEXT:   br i1 %polly.loop_cond, label %polly.loop_header, label %polly.loop_exit
-
-; KERNEL-IR-LABEL: polly.loop_preheader:                             ; preds = %entry
-; KERNEL-IR-NEXT:   br label %polly.loop_header
-
-; KERNEL-IR: attributes #0 = { "polly.skip.fn" }
-
-; KERNEL-ASM: .version 3.2
-; KERNEL-ASM-NEXT: .target sm_30
-; KERNEL-ASM-NEXT: .address_size 64
-
-; KERNEL-ASM:   // .globl     kernel_0
-
-; KERNEL-ASM: .visible .entry kernel_0(
-; KERNEL-ASM-NEXT:   .param .u64 kernel_0_param_0
-; KERNEL-ASM-NEXT: )
-
-;    void double_parallel_loop(float A[][1024]) {
-;      for (long i = 0; i < 1024; i++)
-;        for (long j = 0; j < 1024; j++)
-;          A[i][j] += i * j;
-;    }
-;
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @double_parallel_loop(ptr %A) {
-bb:
-  br label %bb2
-
-bb2:                                              ; preds = %bb13, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp14, %bb13 ]
-  %exitcond1 = icmp ne i64 %i.0, 1024
-  br i1 %exitcond1, label %bb3, label %bb15
-
-bb3:                                              ; preds = %bb2
-  br label %bb4
-
-bb4:                                              ; preds = %bb10, %bb3
-  %j.0 = phi i64 [ 0, %bb3 ], [ %tmp11, %bb10 ]
-  %exitcond = icmp ne i64 %j.0, 1024
-  br i1 %exitcond, label %bb5, label %bb12
-
-bb5:                                              ; preds = %bb4
-  %tmp = mul nuw nsw i64 %i.0, %j.0
-  %tmp6 = sitofp i64 %tmp to float
-  %tmp7 = getelementptr inbounds [1024 x float], ptr %A, i64 %i.0, i64 %j.0
-  %tmp8 = load float, ptr %tmp7, align 4
-  %tmp9 = fadd float %tmp8, %tmp6
-  store float %tmp9, ptr %tmp7, align 4
-  br label %bb10
-
-bb10:                                             ; preds = %bb5
-  %tmp11 = add nuw nsw i64 %j.0, 1
-  br label %bb4
-
-bb12:                                             ; preds = %bb4
-  br label %bb13
-
-bb13:                                             ; preds = %bb12
-  %tmp14 = add nuw nsw i64 %i.0, 1
-  br label %bb2
-
-bb15:                                             ; preds = %bb2
-  ret void
-}
diff --git a/polly/test/GPGPU/failing-invariant-load-handling.ll b/polly/test/GPGPU/failing-invariant-load-handling.ll
deleted file mode 100644
index 70f88667bd609..0000000000000
--- a/polly/test/GPGPU/failing-invariant-load-handling.ll
+++ /dev/null
@@ -1,57 +0,0 @@
-; RUN: opt %loadPolly -polly-process-unprofitable -polly-invariant-load-hoisting -polly-print-scops -disable-output < %s | FileCheck %s -check-prefix=SCOPS
-; RUN: opt %loadPolly -S < %s -polly-codegen-ppcg -polly-process-unprofitable -polly-invariant-load-hoisting | FileCheck %s -check-prefix=CODEGEN
-
-; REQUIRES: pollyacc
-
-target datalayout = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n8:16:32-S64"
-
-%S = type { i32, i32, [12 x %L] }
-%L = type { i32, i32, double, i32, i32, i32, i32, i32 }
-
-define void @test(ptr %cpi, i1 %b) {
-; SCOPS-LABEL: Region: %if.then14---%exit
-; SCOPS:         Invariant Accesses: {
-; SCOPS-NEXT:            ReadAccess :=       [Reduction Type: NONE] [Scalar: 0]
-; SCOPS-NEXT:                [l2, l1] -> { Stmt_for_body_i[i0] -> MemRef_cpi[0, 0] };
-; SCOPS-NEXT:            Execution Context: [l2, l1] -> {  :  }
-; SCOPS-NEXT:            ReadAccess :=       [Reduction Type: NONE] [Scalar: 0]
-; SCOPS-NEXT:                [l2, l1] -> { Stmt_for_body_lr_ph_i[] -> MemRef_cpi[0, 1] };
-; SCOPS-NEXT:            Execution Context: [l2, l1] -> {  : l2 > 0 }
-; SCOPS-NEXT:    }
-; SCOPS:         Arrays {
-; SCOPS-NEXT:        i32 MemRef_cpi[*][(10 * %l1)]; // Element size 4
-; SCOPS-NEXT:    }
-
-; Check that we gracefully handle failing invariant loads.
-; This test case is taken from:
-; test/Isl/CodeGen/invariant-load-dimension.ll
-
-; FIXME: Figure out how to actually generate code for this loop.
-; CODEGEN-NOT: LLVM ERROR: preloading invariant loads failed in function
-
-entry:
-  %nt = getelementptr inbounds %S, ptr %cpi, i32 0, i32 1
-  br i1 %b, label %if.then14, label %exit
-
-if.then14:
-  %l0 = load i32, ptr %cpi, align 8
-  %cmp12.i = icmp sgt i32 %l0, 0
-  br i1 %cmp12.i, label %for.body.lr.ph.i, label %exit
-
-for.body.lr.ph.i:
-  %l1 = load i32, ptr %nt, align 4
-  br label %for.body.i
-
-for.body.i:
-  %phi = phi i32 [ 0, %for.body.lr.ph.i ], [ %inc, %for.body.i ]
-  %mul.i163 = mul nsw i32 %phi, %l1
-  %cv = getelementptr inbounds %S, ptr %cpi, i32 0, i32 2, i32 %mul.i163, i32 0
-  store i32 0, ptr %cv, align 8
-  %inc = add nuw nsw i32 %phi, 1
-  %l2 = load i32, ptr %cpi, align 8
-  %cmp.i164 = icmp slt i32 %inc, %l2
-  br i1 %cmp.i164, label %for.body.i, label %exit
-
-exit:
-  ret void
-}
diff --git a/polly/test/GPGPU/failing-invariant-load-hoisting.ll b/polly/test/GPGPU/failing-invariant-load-hoisting.ll
deleted file mode 100644
index aa62921e1af55..0000000000000
--- a/polly/test/GPGPU/failing-invariant-load-hoisting.ll
+++ /dev/null
@@ -1,41 +0,0 @@
-; RUN: opt %loadPolly -S < %s -polly-codegen-ppcg \
-; RUN: -polly-invariant-load-hoisting | FileCheck %s -check-prefix=CODEGEN
-
-; REQUIRES: pollyacc
-
-target datalayout = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n8:16:32-S64"
-
-%S = type { i32, i32, [12 x %L] }
-%L = type { i32, i32, double, i32, i32, i32, i32, i32 }
-
-define void @test(ptr %cpi, i1 %b) {
-; CODEGEN-LABEL: @test(
-; CODEGEN:    polly.preload.begin:
-; CODEGEN-NEXT:  br i1 false
-
-entry:
-  %nt = getelementptr inbounds %S, ptr %cpi, i32 0, i32 1
-  br i1 %b, label %if.then14, label %exit
-
-if.then14:
-  %l0 = load i32, ptr %cpi, align 8
-  %cmp12.i = icmp sgt i32 %l0, 0
-  br i1 %cmp12.i, label %for.body.lr.ph.i, label %exit
-
-for.body.lr.ph.i:
-  %l1 = load i32, ptr %nt, align 4
-  br label %for.body.i
-
-for.body.i:
-  %phi = phi i32 [ 0, %for.body.lr.ph.i ], [ %inc, %for.body.i ]
-  %mul.i163 = mul nsw i32 %phi, %l1
-  %cv = getelementptr inbounds %S, ptr %cpi, i32 0, i32 2, i32 %mul.i163, i32 0
-  store i32 0, ptr %cv, align 8
-  %inc = add nuw nsw i32 %phi, 1
-  %l2 = load i32, ptr %cpi, align 8
-  %cmp.i164 = icmp slt i32 %inc, %l2
-  br i1 %cmp.i164, label %for.body.i, label %exit
-
-exit:
-  ret void
-}
diff --git a/polly/test/GPGPU/host-control-flow.ll b/polly/test/GPGPU/host-control-flow.ll
deleted file mode 100644
index 5ba65d60819ce..0000000000000
--- a/polly/test/GPGPU/host-control-flow.ll
+++ /dev/null
@@ -1,176 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen-ppcg -disable-output \
-; RUN: -polly-acc-dump-code < %s | FileCheck %s -check-prefix=CODE
-
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen-ppcg -disable-output \
-; RUN: -polly-acc-dump-kernel-ir < %s | FileCheck %s -check-prefix=KERNEL-IR
-
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen-ppcg \
-; RUN: -S < %s | FileCheck %s -check-prefix=IR
-;    void foo(float A[2][100]) {
-;      for (long t = 0; t < 100; t++)
-;        for (long i = 1; i < 99; i++)
-;          A[(t + 1) % 2][i] += A[t % 2][i - 1] + A[t % 2][i] + A[t % 2][i + 1];
-;    }
-
-; REQUIRES: pollyacc
-
-; CODE:        cudaCheckReturn(cudaMemcpy(dev_MemRef_A, MemRef_A, (2) * (100) * sizeof(float), cudaMemcpyHostToDevice));
-; CODE-NEXT:   for (int c0 = 0; c0 <= 99; c0 += 1)
-; CODE-NEXT:     {
-; CODE-NEXT:       dim3 k0_dimBlock(32);
-; CODE-NEXT:       dim3 k0_dimGrid(4);
-; CODE-NEXT:       kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_A, c0);
-; CODE-NEXT:       cudaCheckKernel();
-; CODE-NEXT:     }
-
-; CODE:   cudaCheckReturn(cudaMemcpy(MemRef_A, dev_MemRef_A, (2) * (100) * sizeof(float), cudaMemcpyDeviceToHost));
-; CODE-NEXT: cudaCheckReturn(cudaFree(dev_MemRef_A));
-; CODE-NEXT: }
-
-; IR-LABEL: polly.loop_header:                                ; preds = %polly.loop_header, %polly.loop_preheader
-; IR-NEXT:   %polly.indvar = phi i64 [ 0, %polly.loop_preheader ], [ %polly.indvar_next, %polly.loop_header ]
-; ...
-; IR:  store i64 %polly.indvar, i64* %polly_launch_0_param_1
-; IR-NEXT:  [[REGA:%.+]] = getelementptr [2 x i8*], [2 x i8*]* %polly_launch_0_params, i64 0, i64 1
-; IR-NEXT:  [[REGB:%.+]] = bitcast i64* %polly_launch_0_param_1 to i8*
-; IR-NEXT:  store i8* [[REGB]], i8** [[REGA]]
-; IR: call i8* @polly_getKernel
-; ...
-; IR: call void @polly_freeKernel
-; IR-NEXT:   %polly.indvar_next = add nsw i64 %polly.indvar, 1
-; IR-NEXT:   %polly.loop_cond = icmp sle i64 %polly.indvar_next, 99
-; IR-NEXT:   br i1 %polly.loop_cond, label %polly.loop_header, label %polly.loop_exit
-
-; KERNEL-IR: define ptx_kernel void @FUNC_foo_SCOP_0_KERNEL_0(i8 addrspace(1)* %MemRef_A, i64 %c0)
-; KERNEL-IR-LABEL: entry:
-; KERNEL-IR-NEXT:   %0 = call i32 @llvm.nvvm.read.ptx.sreg.ctaid.x()
-; KERNEL-IR-NEXT:   %b0 = zext i32 %0 to i64
-; KERNEL-IR-NEXT:   %1 = call i32 @llvm.nvvm.read.ptx.sreg.tid.x()
-; KERNEL-IR-NEXT:   %t0 = zext i32 %1 to i64
-; KERNEL-IR-NEXT:   br label %polly.cond
-
-; KERNEL-IR-LABEL: polly.cond:                                       ; preds = %entry
-; KERNEL-IR-NEXT:   %2 = mul nsw i64 32, %b0
-; KERNEL-IR-NEXT:   %3 = add nsw i64 %2, %t0
-; KERNEL-IR-NEXT:   %4 = icmp sle i64 %3, 97
-; KERNEL-IR-NEXT:   br i1 %4, label %polly.then, label %polly.else
-
-; KERNEL-IR-LABEL: polly.merge:                                      ; preds = %polly.else, %polly.stmt.for.body3
-; KERNEL-IR-NEXT:   ret void
-
-; KERNEL-IR-LABEL: polly.then:                                       ; preds = %polly.cond
-; KERNEL-IR-NEXT:   %5 = mul nsw i64 32, %b0
-; KERNEL-IR-NEXT:   %6 = add nsw i64 %5, %t0
-; KERNEL-IR-NEXT:   br label %polly.stmt.for.body3
-
-; KERNEL-IR-LABEL: polly.stmt.for.body3:                             ; preds = %polly.then
-; KERNEL-IR-NEXT:   %polly.access.cast.MemRef_A = bitcast i8 addrspace(1)* %MemRef_A to float addrspace(1)*
-; KERNEL-IR-NEXT:   %pexp.pdiv_r = urem i64 %c0, 2
-; KERNEL-IR-NEXT:   %polly.access.mul.MemRef_A = mul nsw i64 %pexp.pdiv_r, 100
-; KERNEL-IR-NEXT:   %7 = mul nsw i64 32, %b0
-; KERNEL-IR-NEXT:   %8 = add nsw i64 %7, %t0
-; KERNEL-IR-NEXT:   %polly.access.add.MemRef_A = add nsw i64 %polly.access.mul.MemRef_A, %8
-; KERNEL-IR-NEXT:   %polly.access.MemRef_A = getelementptr float, float addrspace(1)* %polly.access.cast.MemRef_A, i64 %polly.access.add.MemRef_A
-; KERNEL-IR-NEXT:   %tmp_p_scalar_ = load float, float addrspace(1)* %polly.access.MemRef_A, align 4
-; KERNEL-IR-NEXT:   %polly.access.cast.MemRef_A1 = bitcast i8 addrspace(1)* %MemRef_A to float addrspace(1)*
-; KERNEL-IR-NEXT:   %pexp.pdiv_r2 = urem i64 %c0, 2
-; KERNEL-IR-NEXT:   %polly.access.mul.MemRef_A3 = mul nsw i64 %pexp.pdiv_r2, 100
-; KERNEL-IR-NEXT:   %9 = mul nsw i64 32, %b0
-; KERNEL-IR-NEXT:   %10 = add nsw i64 %9, %t0
-; KERNEL-IR-NEXT:   %11 = add nsw i64 %10, 1
-; KERNEL-IR-NEXT:   %polly.access.add.MemRef_A4 = add nsw i64 %polly.access.mul.MemRef_A3, %11
-; KERNEL-IR-NEXT:   %polly.access.MemRef_A5 = getelementptr float, float addrspace(1)* %polly.access.cast.MemRef_A1, i64 %polly.access.add.MemRef_A4
-; KERNEL-IR-NEXT:   %tmp2_p_scalar_ = load float, float addrspace(1)* %polly.access.MemRef_A5, align 4
-; KERNEL-IR-NEXT:   %p_add = fadd float %tmp_p_scalar_, %tmp2_p_scalar_
-; KERNEL-IR-NEXT:   %polly.access.cast.MemRef_A6 = bitcast i8 addrspace(1)* %MemRef_A to float addrspace(1)*
-; KERNEL-IR-NEXT:   %pexp.pdiv_r7 = urem i64 %c0, 2
-; KERNEL-IR-NEXT:   %polly.access.mul.MemRef_A8 = mul nsw i64 %pexp.pdiv_r7, 100
-; KERNEL-IR-NEXT:   %12 = mul nsw i64 32, %b0
-; KERNEL-IR-NEXT:   %13 = add nsw i64 %12, %t0
-; KERNEL-IR-NEXT:   %14 = add nsw i64 %13, 2
-; KERNEL-IR-NEXT:   %polly.access.add.MemRef_A9 = add nsw i64 %polly.access.mul.MemRef_A8, %14
-; KERNEL-IR-NEXT:   %polly.access.MemRef_A10 = getelementptr float, float addrspace(1)* %polly.access.cast.MemRef_A6, i64 %polly.access.add.MemRef_A9
-; KERNEL-IR-NEXT:   %tmp3_p_scalar_ = load float, float addrspace(1)* %polly.access.MemRef_A10, align 4
-; KERNEL-IR-NEXT:   %p_add12 = fadd float %p_add, %tmp3_p_scalar_
-; KERNEL-IR-NEXT:   %polly.access.cast.MemRef_A11 = bitcast i8 addrspace(1)* %MemRef_A to float addrspace(1)*
-; KERNEL-IR-NEXT:   %15 = add nsw i64 %c0, 1
-; KERNEL-IR-NEXT:   %pexp.pdiv_r12 = urem i64 %15, 2
-; KERNEL-IR-NEXT:   %polly.access.mul.MemRef_A13 = mul nsw i64 %pexp.pdiv_r12, 100
-; KERNEL-IR-NEXT:   %16 = mul nsw i64 32, %b0
-; KERNEL-IR-NEXT:   %17 = add nsw i64 %16, %t0
-; KERNEL-IR-NEXT:   %18 = add nsw i64 %17, 1
-; KERNEL-IR-NEXT:   %polly.access.add.MemRef_A14 = add nsw i64 %polly.access.mul.MemRef_A13, %18
-; KERNEL-IR-NEXT:   %polly.access.MemRef_A15 = getelementptr float, float addrspace(1)* %polly.access.cast.MemRef_A11, i64 %polly.access.add.MemRef_A14
-; KERNEL-IR-NEXT:   %tmp4_p_scalar_ = load float, float addrspace(1)* %polly.access.MemRef_A15, align 4
-; KERNEL-IR-NEXT:   %p_add17 = fadd float %tmp4_p_scalar_, %p_add12
-; KERNEL-IR-NEXT:   %polly.access.cast.MemRef_A16 = bitcast i8 addrspace(1)* %MemRef_A to float addrspace(1)*
-; KERNEL-IR-NEXT:   %19 = add nsw i64 %c0, 1
-; KERNEL-IR-NEXT:   %pexp.pdiv_r17 = urem i64 %19, 2
-; KERNEL-IR-NEXT:   %polly.access.mul.MemRef_A18 = mul nsw i64 %pexp.pdiv_r17, 100
-; KERNEL-IR-NEXT:   %20 = mul nsw i64 32, %b0
-; KERNEL-IR-NEXT:   %21 = add nsw i64 %20, %t0
-; KERNEL-IR-NEXT:   %22 = add nsw i64 %21, 1
-; KERNEL-IR-NEXT:   %polly.access.add.MemRef_A19 = add nsw i64 %polly.access.mul.MemRef_A18, %22
-; KERNEL-IR-NEXT:   %polly.access.MemRef_A20 = getelementptr float, float addrspace(1)* %polly.access.cast.MemRef_A16, i64 %polly.access.add.MemRef_A19
-; KERNEL-IR-NEXT:   store float %p_add17, float addrspace(1)* %polly.access.MemRef_A20, align 4
-; KERNEL-IR-NEXT:   br label %polly.merge
-
-; KERNEL-IR-LABEL: polly.else:                                       ; preds = %polly.cond
-; KERNEL-IR-NEXT:   br label %polly.merge
-; KERNEL-IR-NEXT: }
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @foo([100 x float]* %A) {
-entry:
-  br label %for.cond
-
-for.cond:                                         ; preds = %for.inc18, %entry
-  %t.0 = phi i64 [ 0, %entry ], [ %inc19, %for.inc18 ]
-  %exitcond1 = icmp ne i64 %t.0, 100
-  br i1 %exitcond1, label %for.body, label %for.end20
-
-for.body:                                         ; preds = %for.cond
-  br label %for.cond1
-
-for.cond1:                                        ; preds = %for.inc, %for.body
-  %i.0 = phi i64 [ 1, %for.body ], [ %inc, %for.inc ]
-  %exitcond = icmp ne i64 %i.0, 99
-  br i1 %exitcond, label %for.body3, label %for.end
-
-for.body3:                                        ; preds = %for.cond1
-  %sub = add nsw i64 %i.0, -1
-  %rem = srem i64 %t.0, 2
-  %arrayidx4 = getelementptr inbounds [100 x float], [100 x float]* %A, i64 %rem, i64 %sub
-  %tmp = load float, float* %arrayidx4, align 4
-  %rem5 = srem i64 %t.0, 2
-  %arrayidx7 = getelementptr inbounds [100 x float], [100 x float]* %A, i64 %rem5, i64 %i.0
-  %tmp2 = load float, float* %arrayidx7, align 4
-  %add = fadd float %tmp, %tmp2
-  %add8 = add nuw nsw i64 %i.0, 1
-  %rem9 = srem i64 %t.0, 2
-  %arrayidx11 = getelementptr inbounds [100 x float], [100 x float]* %A, i64 %rem9, i64 %add8
-  %tmp3 = load float, float* %arrayidx11, align 4
-  %add12 = fadd float %add, %tmp3
-  %add13 = add nuw nsw i64 %t.0, 1
-  %rem14 = srem i64 %add13, 2
-  %arrayidx16 = getelementptr inbounds [100 x float], [100 x float]* %A, i64 %rem14, i64 %i.0
-  %tmp4 = load float, float* %arrayidx16, align 4
-  %add17 = fadd float %tmp4, %add12
-  store float %add17, float* %arrayidx16, align 4
-  br label %for.inc
-
-for.inc:                                          ; preds = %for.body3
-  %inc = add nuw nsw i64 %i.0, 1
-  br label %for.cond1
-
-for.end:                                          ; preds = %for.cond1
-  br label %for.inc18
-
-for.inc18:                                        ; preds = %for.end
-  %inc19 = add nuw nsw i64 %t.0, 1
-  br label %for.cond
-
-for.end20:                                        ; preds = %for.cond
-  ret void
-}
diff --git a/polly/test/GPGPU/host-statement.ll b/polly/test/GPGPU/host-statement.ll
deleted file mode 100644
index d7232b2fa5380..0000000000000
--- a/polly/test/GPGPU/host-statement.ll
+++ /dev/null
@@ -1,204 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -polly-invariant-load-hoisting=false \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-kernel-ir \
-; RUN: -polly-invariant-load-hoisting=false \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=KERNEL-IR %s
-
-; REQUIRES: pollyacc
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-target triple = "x86_64-unknown-linux-gnu"
-
-declare void @llvm.lifetime.start(i64, ptr nocapture) #0
-
-; This test case tests that we can correctly handle a ScopStmt that is
-; scheduled on the host, instead of within a kernel.
-
-; CODE:        cudaCheckReturn(cudaMemcpy(dev_MemRef_A, MemRef_A, (512) * (512) * sizeof(double), cudaMemcpyHostToDevice));
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(dev_MemRef_R, MemRef_R, (p_0 + 1) * (512) * sizeof(double), cudaMemcpyHostToDevice));
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(dev_MemRef_Q, MemRef_Q, (512) * (512) * sizeof(double), cudaMemcpyHostToDevice));
-; CODE-NEXT:   {
-; CODE-NEXT:     dim3 k0_dimBlock(32);
-; CODE-NEXT:     dim3 k0_dimGrid(16);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_A, dev_MemRef_R, dev_MemRef_Q, p_0, p_1);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:   if (p_0 <= 510 && p_1 <= 510) {
-; CODE-NEXT:     {
-; CODE-NEXT:       dim3 k1_dimBlock(32);
-; CODE-NEXT:       dim3 k1_dimGrid(p_1 <= -1048034 ? 32768 : -p_1 + floord(31 * p_1 + 30, 32) + 16);
-; CODE-NEXT:       kernel1 <<<k1_dimGrid, k1_dimBlock>>> (dev_MemRef_A, dev_MemRef_R, dev_MemRef_Q, p_0, p_1);
-; CODE-NEXT:       cudaCheckKernel();
-; CODE-NEXT:     }
-
-; CODE:     {
-; CODE-NEXT:       dim3 k2_dimBlock(16, 32);
-; CODE-NEXT:       dim3 k2_dimGrid(16, p_1 <= -7650 ? 256 : -p_1 + floord(31 * p_1 + 30, 32) + 16);
-; CODE-NEXT:       kernel2 <<<k2_dimGrid, k2_dimBlock>>> (dev_MemRef_A, dev_MemRef_R, dev_MemRef_Q, p_0, p_1);
-; CODE-NEXT:       cudaCheckKernel();
-; CODE-NEXT:     }
-
-; CODE:   }
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(MemRef_A, dev_MemRef_A, (512) * (512) * sizeof(double), cudaMemcpyDeviceToHost));
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(MemRef_R, dev_MemRef_R, (p_0 + 1) * (512) * sizeof(double), cudaMemcpyDeviceToHost));
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(MemRef_Q, dev_MemRef_Q, (512) * (512) * sizeof(double), cudaMemcpyDeviceToHost));
-; CODE-NEXT:     Stmt_for_cond33_preheader_last();
-
-; CODE: }
-
-; CODE: # kernel0
-; CODE-NEXT: Stmt_for_body16(32 * b0 + t0);
-
-; CODE: # kernel1
-; CODE-NEXT: for (int c0 = 0; c0 <= (-p_1 - 32 * b0 + 510) / 1048576; c0 += 1)
-; CODE-NEXT:   for (int c1 = 0; c1 <= 15; c1 += 1) {
-; CODE-NEXT:     if (p_1 + 32 * b0 + t0 + 1048576 * c0 <= 510 && c1 == 0)
-; CODE-NEXT:       Stmt_for_body35(32 * b0 + t0 + 1048576 * c0);
-; CODE-NEXT:     if (p_1 + 32 * b0 + t0 + 1048576 * c0 <= 510)
-; CODE-NEXT:       for (int c3 = 0; c3 <= 31; c3 += 1)
-; CODE-NEXT:         Stmt_for_body42(32 * b0 + t0 + 1048576 * c0, 32 * c1 + c3);
-; CODE-NEXT:     sync0();
-; CODE-NEXT:   }
-
-; CODE: # kernel2
-; CODE-NEXT: for (int c0 = 0; c0 <= (-p_1 - 32 * b0 + 510) / 8192; c0 += 1)
-; CODE-NEXT:   if (p_1 + 32 * b0 + t0 + 8192 * c0 <= 510)
-; CODE-NEXT:     for (int c3 = 0; c3 <= 1; c3 += 1)
-; CODE-NEXT:       Stmt_for_body62(32 * b0 + t0 + 8192 * c0, 32 * b1 + t1 + 16 * c3);
-
-; KERNEL-IR: call void @llvm.nvvm.barrier0()
-
-; Function Attrs: nounwind uwtable
-define internal void @kernel_gramschmidt(i32 %ni, i32 %nj, ptr %A, ptr %R, ptr %Q) #1 {
-entry:
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  br label %for.cond1.preheader
-
-for.cond1.preheader:                              ; preds = %entry.split, %for.inc86
-  %indvars.iv24 = phi i64 [ 0, %entry.split ], [ %indvars.iv.next25, %for.inc86 ]
-  %indvars.iv19 = phi i64 [ 1, %entry.split ], [ %indvars.iv.next20, %for.inc86 ]
-  br label %for.inc
-
-for.inc:                                          ; preds = %for.cond1.preheader, %for.inc
-  %indvars.iv = phi i64 [ 0, %for.cond1.preheader ], [ %indvars.iv.next, %for.inc ]
-  %nrm.02 = phi double [ 0.000000e+00, %for.cond1.preheader ], [ %add, %for.inc ]
-  %arrayidx5 = getelementptr inbounds [512 x double], ptr %A, i64 %indvars.iv, i64 %indvars.iv24
-  %tmp = load double, ptr %arrayidx5, align 8, !tbaa !1
-  %arrayidx9 = getelementptr inbounds [512 x double], ptr %A, i64 %indvars.iv, i64 %indvars.iv24
-  %tmp27 = load double, ptr %arrayidx9, align 8, !tbaa !1
-  %mul = fmul double %tmp, %tmp27
-  %add = fadd double %nrm.02, %mul
-  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
-  %exitcond = icmp ne i64 %indvars.iv.next, 512
-  br i1 %exitcond, label %for.inc, label %for.end
-
-for.end:                                          ; preds = %for.inc
-  %add.lcssa = phi double [ %add, %for.inc ]
-  %call = tail call double @sqrt(double %add.lcssa) #2
-  %arrayidx13 = getelementptr inbounds [512 x double], ptr %R, i64 %indvars.iv24, i64 %indvars.iv24
-  store double %call, ptr %arrayidx13, align 8, !tbaa !1
-  br label %for.body16
-
-for.cond33.preheader:                             ; preds = %for.body16
-  %indvars.iv.next25 = add nuw nsw i64 %indvars.iv24, 1
-  %cmp347 = icmp slt i64 %indvars.iv.next25, 512
-  br i1 %cmp347, label %for.body35.lr.ph, label %for.inc86
-
-for.body35.lr.ph:                                 ; preds = %for.cond33.preheader
-  br label %for.body35
-
-for.body16:                                       ; preds = %for.end, %for.body16
-  %indvars.iv10 = phi i64 [ 0, %for.end ], [ %indvars.iv.next11, %for.body16 ]
-  %arrayidx20 = getelementptr inbounds [512 x double], ptr %A, i64 %indvars.iv10, i64 %indvars.iv24
-  %tmp28 = load double, ptr %arrayidx20, align 8, !tbaa !1
-  %arrayidx24 = getelementptr inbounds [512 x double], ptr %R, i64 %indvars.iv24, i64 %indvars.iv24
-  %tmp29 = load double, ptr %arrayidx24, align 8, !tbaa !1
-  %div = fdiv double %tmp28, %tmp29
-  %arrayidx28 = getelementptr inbounds [512 x double], ptr %Q, i64 %indvars.iv10, i64 %indvars.iv24
-  store double %div, ptr %arrayidx28, align 8, !tbaa !1
-  %indvars.iv.next11 = add nuw nsw i64 %indvars.iv10, 1
-  %exitcond12 = icmp ne i64 %indvars.iv.next11, 512
-  br i1 %exitcond12, label %for.body16, label %for.cond33.preheader
-
-for.cond33.loopexit:                              ; preds = %for.body62
-  %indvars.iv.next22 = add nuw nsw i64 %indvars.iv21, 1
-  %lftr.wideiv = trunc i64 %indvars.iv.next22 to i32
-  %exitcond23 = icmp ne i32 %lftr.wideiv, 512
-  br i1 %exitcond23, label %for.body35, label %for.cond33.for.inc86_crit_edge
-
-for.body35:                                       ; preds = %for.body35.lr.ph, %for.cond33.loopexit
-  %indvars.iv21 = phi i64 [ %indvars.iv19, %for.body35.lr.ph ], [ %indvars.iv.next22, %for.cond33.loopexit ]
-  %arrayidx39 = getelementptr inbounds [512 x double], ptr %R, i64 %indvars.iv24, i64 %indvars.iv21
-  store double 0.000000e+00, ptr %arrayidx39, align 8, !tbaa !1
-  br label %for.body42
-
-for.cond60.preheader:                             ; preds = %for.body42
-  br label %for.body62
-
-for.body42:                                       ; preds = %for.body35, %for.body42
-  %indvars.iv13 = phi i64 [ 0, %for.body35 ], [ %indvars.iv.next14, %for.body42 ]
-  %arrayidx46 = getelementptr inbounds [512 x double], ptr %Q, i64 %indvars.iv13, i64 %indvars.iv24
-  %tmp30 = load double, ptr %arrayidx46, align 8, !tbaa !1
-  %arrayidx50 = getelementptr inbounds [512 x double], ptr %A, i64 %indvars.iv13, i64 %indvars.iv21
-  %tmp31 = load double, ptr %arrayidx50, align 8, !tbaa !1
-  %mul51 = fmul double %tmp30, %tmp31
-  %arrayidx55 = getelementptr inbounds [512 x double], ptr %R, i64 %indvars.iv24, i64 %indvars.iv21
-  %tmp32 = load double, ptr %arrayidx55, align 8, !tbaa !1
-  %add56 = fadd double %tmp32, %mul51
-  store double %add56, ptr %arrayidx55, align 8, !tbaa !1
-  %indvars.iv.next14 = add nuw nsw i64 %indvars.iv13, 1
-  %exitcond15 = icmp ne i64 %indvars.iv.next14, 512
-  br i1 %exitcond15, label %for.body42, label %for.cond60.preheader
-
-for.body62:                                       ; preds = %for.cond60.preheader, %for.body62
-  %indvars.iv16 = phi i64 [ 0, %for.cond60.preheader ], [ %indvars.iv.next17, %for.body62 ]
-  %arrayidx66 = getelementptr inbounds [512 x double], ptr %A, i64 %indvars.iv16, i64 %indvars.iv21
-  %tmp33 = load double, ptr %arrayidx66, align 8, !tbaa !1
-  %arrayidx70 = getelementptr inbounds [512 x double], ptr %Q, i64 %indvars.iv16, i64 %indvars.iv24
-  %tmp34 = load double, ptr %arrayidx70, align 8, !tbaa !1
-  %arrayidx74 = getelementptr inbounds [512 x double], ptr %R, i64 %indvars.iv24, i64 %indvars.iv21
-  %tmp35 = load double, ptr %arrayidx74, align 8, !tbaa !1
-  %mul75 = fmul double %tmp34, %tmp35
-  %sub = fsub double %tmp33, %mul75
-  %arrayidx79 = getelementptr inbounds [512 x double], ptr %A, i64 %indvars.iv16, i64 %indvars.iv21
-  store double %sub, ptr %arrayidx79, align 8, !tbaa !1
-  %indvars.iv.next17 = add nuw nsw i64 %indvars.iv16, 1
-  %exitcond18 = icmp ne i64 %indvars.iv.next17, 512
-  br i1 %exitcond18, label %for.body62, label %for.cond33.loopexit
-
-for.cond33.for.inc86_crit_edge:                   ; preds = %for.cond33.loopexit
-  br label %for.inc86
-
-for.inc86:                                        ; preds = %for.cond33.for.inc86_crit_edge, %for.cond33.preheader
-  %indvars.iv.next20 = add nuw nsw i64 %indvars.iv19, 1
-  %exitcond26 = icmp ne i64 %indvars.iv.next25, 512
-  br i1 %exitcond26, label %for.cond1.preheader, label %for.end88
-
-for.end88:                                        ; preds = %for.inc86
-  ret void
-}
-
-; Function Attrs: argmemonly nounwind
-declare void @llvm.lifetime.end(i64, ptr nocapture) #0
-
-; Function Attrs: nounwind
-declare double @sqrt(double) #2
-
-attributes #0 = { argmemonly nounwind }
-attributes #1 = { nounwind uwtable "disable-tail-calls"="false" "less-precise-fpmad"="false" "frame-pointer"="none" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+fxsr,+mmx,+sse,+sse2,+x87" "unsafe-fp-math"="false" "use-soft-float"="false" }
-attributes #2 = { nounwind }
-
-!llvm.ident = !{!0}
-
-!0 = !{!"clang version 3.9.0 (trunk 275267) (llvm/trunk 275268)"}
-!1 = !{!2, !2, i64 0}
-!2 = !{!"double", !3, i64 0}
-!3 = !{!"omnipotent char", !4, i64 0}
-!4 = !{!"Simple C/C++ TBAA"}
diff --git a/polly/test/GPGPU/ignore-parameter-bounds.ll b/polly/test/GPGPU/ignore-parameter-bounds.ll
deleted file mode 100644
index 1d0b5482941e1..0000000000000
--- a/polly/test/GPGPU/ignore-parameter-bounds.ll
+++ /dev/null
@@ -1,41 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-
-; REQUIRES: pollyacc
-
-; CODE: Code
-; CODE: ====
-; CODE: No code generated
-
-source_filename = "bugpoint-output-83bcdeb.bc"
-target datalayout = "e-p:64:64:64-S128-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f16:16:16-f32:32:32-f64:64:64-f128:128:128-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
-target triple = "x86_64-unknown-linux-gnu"
-
-@__data_radiation_MOD_cobi = external global [168 x double], align 32
-
-; Function Attrs: nounwind uwtable
-define void @__radiation_rg_MOD_coe_so() #0 {
-entry:
-  %polly.access.kspec.load = load i32, ptr undef, align 4
-  %0 = or i1 undef, undef
-  br label %polly.preload.cond29
-
-polly.preload.cond29:                             ; preds = %entry
-  br i1 %0, label %polly.preload.exec31, label %polly.preload.merge30
-
-polly.preload.merge30:                            ; preds = %polly.preload.exec31, %polly.preload.cond29
-  %polly.preload..merge32 = phi double [ %polly.access.__data_radiation_MOD_cobi.load, %polly.preload.exec31 ], [ 0.000000e+00, %polly.preload.cond29 ]
-  ret void
-
-polly.preload.exec31:                             ; preds = %polly.preload.cond29
-  %1 = sext i32 %polly.access.kspec.load to i64
-  %2 = mul nsw i64 7, %1
-  %3 = add nsw i64 0, %2
-  %4 = add nsw i64 %3, 48
-  %polly.access.__data_radiation_MOD_cobi = getelementptr double, ptr @__data_radiation_MOD_cobi, i64 %4
-  %polly.access.__data_radiation_MOD_cobi.load = load double, ptr %polly.access.__data_radiation_MOD_cobi, align 8
-  br label %polly.preload.merge30
-}
-
-attributes #0 = { nounwind uwtable }
diff --git a/polly/test/GPGPU/intrinsic-copied-into-kernel.ll b/polly/test/GPGPU/intrinsic-copied-into-kernel.ll
deleted file mode 100644
index 7c1e3672abb5c..0000000000000
--- a/polly/test/GPGPU/intrinsic-copied-into-kernel.ll
+++ /dev/null
@@ -1,76 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-print-scops -disable-output < %s | FileCheck %s --check-prefix=SCOP
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen-ppcg -polly-acc-dump-kernel-ir -disable-output < %s | FileCheck %s --check-prefix=KERNEL-IR
-; RUN: opt -opaque-pointers=0 %loadPolly -S -polly-codegen-ppcg  < %s | FileCheck %s --check-prefix=HOST-IR
-
-; Test that we do recognise and codegen a kernel that has intrinsics.
-
-; REQUIRES: pollyacc
-
-; Check that we model the kernel as a scop.
-; SCOP:      Function: f
-; SCOP-NEXT:       Region: %entry.split---%for.end
-
-; Check that the intrinsic call is present in the kernel IR.
-; KERNEL-IR:   %p_sqrt = tail call float @llvm.sqrt.f32(float %A.arr.i.val_p_scalar_)
-; KERNEL-IR:   declare float @llvm.sqrt.f32(float)
-; KERNEL-IR:   declare float @llvm.fabs.f32(float)
-
-
-; Check that kernel launch is generated in host IR.
-; the declare would not be generated unless a call to a kernel exists.
-; HOST-IR: declare void @polly_launchKernel(i8*, i32, i32, i32, i32, i32, i8*)
-
-
-; void f(float *A, float *B, int N) {
-;   for(int i = 0; i < N; i++) {
-;       float tmp0 = A[i];
-;       float tmp1 = sqrt(tmp1);
-;       float tmp2 = fabs(tmp2);
-;       float tmp3 = copysignf(tmp1, tmp2);
-;       B[i] = tmp4;
-;   }
-; }
-
-target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @f(float* %A, float* %B, i32 %N) {
-entry:
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  %cmp1 = icmp sgt i32 %N, 0
-  br i1 %cmp1, label %for.body.lr.ph, label %for.end
-
-for.body.lr.ph:                                   ; preds = %entry.split
-  br label %for.body
-
-for.body:                                         ; preds = %for.body.lr.ph, %for.body
-  %indvars.iv = phi i64 [ 0, %for.body.lr.ph ], [ %indvars.iv.next, %for.body ]
-  %A.arr.i = getelementptr inbounds float, float* %A, i64 %indvars.iv
-  %A.arr.i.val = load float, float* %A.arr.i, align 4
-  ; Call to intrinsics that should be part of the kernel.
-  %sqrt = tail call float @llvm.sqrt.f32(float %A.arr.i.val)
-  %fabs = tail call float @llvm.fabs.f32(float %sqrt);
-  %copysign = tail call float @llvm.copysign.f32(float %sqrt, float %fabs);
-  %B.arr.i = getelementptr inbounds float, float* %B, i64 %indvars.iv
-  store float %copysign, float* %B.arr.i, align 4
-
-  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
-  %wide.trip.count = zext i32 %N to i64
-  %exitcond = icmp ne i64 %indvars.iv.next, %wide.trip.count
-  br i1 %exitcond, label %for.body, label %for.cond.for.end_crit_edge
-
-for.cond.for.end_crit_edge:                       ; preds = %for.body
-  br label %for.end
-
-for.end:                                          ; preds = %for.cond.for.end_crit_edge, %entry.split
-  ret void
-}
-
-; Function Attrs: nounwind readnone
-declare float @llvm.sqrt.f32(float) #0
-declare float @llvm.fabs.f32(float) #0
-declare float @llvm.copysign.f32(float, float) #0
-
-attributes #0 = { nounwind readnone }
-
diff --git a/polly/test/GPGPU/invalid-kernel-assert-verifymodule.ll b/polly/test/GPGPU/invalid-kernel-assert-verifymodule.ll
deleted file mode 100644
index 4b9139f0b44c8..0000000000000
--- a/polly/test/GPGPU/invalid-kernel-assert-verifymodule.ll
+++ /dev/null
@@ -1,47 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg  -polly-acc-fail-on-verify-module-failure \
-; RUN: -disable-output < %s
-
-; Make sure that if -polly-acc-fail-on-verify-module-failure is on, we actually
-; fail on an illegal module.
-
-; REQUIRES: pollyacc, asserts
-; XFAIL: *
-;
-;    void foo(long A[1024], long B[1024]) {
-;      for (long i = 0; i < 1024; i++)
-;        A[i] += (B[i] + (long)&B[i]);
-;    }
-
-
-; RUN: opt %loadPolly -polly-codegen-ppcg -S < %s
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @foo(ptr %A, ptr %B) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb10, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp11, %bb10 ]
-  %exitcond = icmp ne i64 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb12
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds i64, ptr %B, i64 %i.0
-  %tmp3 = load i64, ptr %tmp, align 8
-  %tmp4 = getelementptr inbounds i64, ptr %B, i64 %i.0
-  %tmp5 = ptrtoint ptr %tmp4 to i64
-  %tmp6 = add nsw i64 %tmp3, %tmp5
-  %tmp7 = getelementptr inbounds i64, ptr %A, i64 %i.0
-  %tmp8 = load i64, ptr %tmp7, align 8
-  %tmp9 = add nsw i64 %tmp8, %tmp6
-  store i64 %tmp9, ptr %tmp7, align 8
-  br label %bb10
-
-bb10:                                             ; preds = %bb2
-  %tmp11 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb12:                                             ; preds = %bb1
-  ret void
-}
diff --git a/polly/test/GPGPU/invalid-kernel.ll b/polly/test/GPGPU/invalid-kernel.ll
deleted file mode 100644
index 9dd32eac97c0c..0000000000000
--- a/polly/test/GPGPU/invalid-kernel.ll
+++ /dev/null
@@ -1,73 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-kernel-ir \
-; RUN: -disable-output < %s | \
-; RUN: not FileCheck %s -check-prefix=KERNEL-IR
-
-; RUN: opt %loadPolly -polly-codegen-ppcg -S < %s | \
-; RUN: FileCheck %s -check-prefix=IR
-
-; REQUIRES: pollyacc
-;
-;    void foo(long A[1024], long B[1024]) {
-;      for (long i = 0; i < 1024; i++)
-;        A[i] += (B[i] + (long)&B[i]);
-;    }
-
-; This kernel loads/stores a pointer address we model. This is a rare case,
-; were we still lack proper code-generation support. We check here that we
-; detect the invalid IR and bail out gracefully.
-
-; CODE:        cudaCheckReturn(cudaMemcpy(dev_MemRef_B, MemRef_B, (1024) * sizeof(i64), cudaMemcpyHostToDevice));
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(dev_MemRef_A, MemRef_A, (1024) * sizeof(i64), cudaMemcpyHostToDevice));
-; CODE-NEXT:   {
-; CODE-NEXT:     dim3 k0_dimBlock(32);
-; CODE-NEXT:     dim3 k0_dimGrid(32);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_B, dev_MemRef_A);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:   cudaCheckReturn(cudaMemcpy(MemRef_A, dev_MemRef_A, (1024) * sizeof(i64), cudaMemcpyDeviceToHost));
-
-; CODE: # kernel0
-; CODE-NEXT: Stmt_bb2(32 * b0 + t0);
-
-; RUN: opt %loadPolly -polly-codegen-ppcg -S < %s | \
-; RUN: FileCheck %s -check-prefix=IR
-
-; KERNEL-IR: kernel
-
-; IR: br i1 false, label %polly.start, label %bb1
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @foo(ptr %A, ptr %B) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb10, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp11, %bb10 ]
-  %exitcond = icmp ne i64 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb12
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds i64, ptr %B, i64 %i.0
-  %tmp3 = load i64, ptr %tmp, align 8
-  %tmp4 = getelementptr inbounds i64, ptr %B, i64 %i.0
-  %tmp5 = ptrtoint ptr %tmp4 to i64
-  %tmp6 = add nsw i64 %tmp3, %tmp5
-  %tmp7 = getelementptr inbounds i64, ptr %A, i64 %i.0
-  %tmp8 = load i64, ptr %tmp7, align 8
-  %tmp9 = add nsw i64 %tmp8, %tmp6
-  store i64 %tmp9, ptr %tmp7, align 8
-  br label %bb10
-
-bb10:                                             ; preds = %bb2
-  %tmp11 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb12:                                             ; preds = %bb1
-  ret void
-}
diff --git a/polly/test/GPGPU/invariant-load-array-access.ll b/polly/test/GPGPU/invariant-load-array-access.ll
deleted file mode 100644
index 02c0330a7e7e2..0000000000000
--- a/polly/test/GPGPU/invariant-load-array-access.ll
+++ /dev/null
@@ -1,70 +0,0 @@
-; RUN: opt %loadPolly -polly-invariant-load-hoisting -polly-print-scops -disable-output < %s | FileCheck %s -check-prefix=SCOP
-
-; RUN: opt %loadPolly -S -polly-codegen-ppcg \
-; RUN: -polly-invariant-load-hoisting < %s | FileCheck %s -check-prefix=HOST-IR
-
-
-; REQUIRES: pollyacc
-
-; Check that we detect a scop.
-; SCOP:      Function: f
-; SCOP-NEXT: Region: %for.body---%for.end
-; SCOP-NEXT: Max Loop Depth:  1
-; SCOP-NEXT: Invariant Accesses: {
-; SCOP-NEXT:         ReadAccess :=	[Reduction Type: NONE] [Scalar: 0]
-; SCOP-NEXT:             [tmp] -> { Stmt_for_body[i0] -> MemRef_control[0] };
-; SCOP-NEXT:         Execution Context: [tmp] -> {  :  }
-; SCOP-NEXT:         ReadAccess :=	[Reduction Type: NONE] [Scalar: 0]
-; SCOP-NEXT:             [tmp] -> { Stmt_if_then[i0] -> MemRef_readarr[0] };
-; SCOP-NEXT:         Execution Context: [tmp] -> {  : tmp >= 4 }
-; SCOP-NEXT: }
-
-; Check that kernel launch is generated in host IR.
-; the declare would not be generated unless a call to a kernel exists.
-; HOST-IR: declare void @polly_launchKernel(ptr, i32, i32, i32, i32, i32, ptr)
-
-; This test makes sure that such an access pattern is handled correctly
-; by PPCGCodeGeneration. It appears that not calling `preloadInvariantLoads`
-; was the main reason that caused this test case to crash.
-;
-; void f(int *arr, const int *control, const int *readarr) {
-;     for(int i = 0; i < 1000; i++) {
-;         int t = 0;
-;         if (*control > 3) {
-;             t += *readarr;
-;         }
-;         arr[i] = t;
-;     }
-; }
-
-
-target datalayout = "e-m:o-p:32:32-f64:32:64-f80:128-n8:16:32-S128"
-target triple = "i386-apple-macosx10.12.0"
-define void @f(ptr %arr, ptr %control, ptr %readarr) {
-entry:
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  br label %for.body
-
-for.body:                                         ; preds = %entry.split, %if.end
-  %i.01 = phi i32 [ 0, %entry.split ], [ %inc, %if.end ]
-  %tmp = load i32, ptr %control, align 4
-  %cmp1 = icmp sgt i32 %tmp, 3
-  br i1 %cmp1, label %if.then, label %if.end
-
-if.then:                                          ; preds = %for.body
-  %tmp1 = load i32, ptr %readarr, align 4
-  br label %if.end
-
-if.end:                                           ; preds = %if.then, %for.body
-  %t.0 = phi i32 [ %tmp1, %if.then ], [ 0, %for.body ]
-  %arrayidx = getelementptr inbounds i32, ptr %arr, i32 %i.01
-  store i32 %t.0, ptr %arrayidx, align 4
-  %inc = add nuw nsw i32 %i.01, 1
-  %exitcond = icmp eq i32 %inc, 1000
-  br i1 %exitcond, label %for.end, label %for.body
-
-for.end:                                          ; preds = %if.end
-  ret void
-}
diff --git a/polly/test/GPGPU/invariant-load-escaping-values.ll b/polly/test/GPGPU/invariant-load-escaping-values.ll
deleted file mode 100644
index 54f4b43fdb92a..0000000000000
--- a/polly/test/GPGPU/invariant-load-escaping-values.ll
+++ /dev/null
@@ -1,30 +0,0 @@
-; RUN: opt %loadPolly -S -polly-codegen-ppcg \
-; RUN: -polly-invariant-load-hoisting < %s | FileCheck %s
-
-; REQUIRES: pollyacc
-
-; CHECK: store i64 %polly.access.B.load, ptr %invariant.preload.s2a
-; CHECK: %invariant.final_reload = load i64, ptr %invariant.preload.s2a
-
-; Verify that the final reload of an invariant scalar memory access uses the
-; same stack slot that into which the invariant memory access was stored
-; originally. Earlier, this was broken as we introduce a new stack slot aside
-; of the preload stack slot, which remained uninitialized and caused our escaping
-; loads to contain garbage.
-
-define i64 @foo(ptr %A, ptr %B) {
-entry:
-  br label %loop
-
-loop:
-  %indvar = phi i64 [0, %entry], [%indvar.next, %loop]
-  %indvar.next = add nsw i64 %indvar, 1
-  %idx = getelementptr float, ptr %A, i64 %indvar
-  store float 42.0, ptr %idx
-  %invariant = load i64, ptr %B
-  %cmp = icmp sle i64 %indvar, 1024
-  br i1 %cmp, label %loop, label %exit
-
-exit:
-  ret i64 %invariant
-}
diff --git a/polly/test/GPGPU/invariant-load-hoisting-of-array.ll b/polly/test/GPGPU/invariant-load-hoisting-of-array.ll
deleted file mode 100644
index 015a3dacbe10f..0000000000000
--- a/polly/test/GPGPU/invariant-load-hoisting-of-array.ll
+++ /dev/null
@@ -1,101 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-invariant-load-hoisting -polly-print-scops -disable-output < %s | FileCheck %s -check-prefix=SCOP
-
-; RUN: opt -opaque-pointers=0 %loadPolly -S -polly-codegen-ppcg \
-; RUN: -polly-invariant-load-hoisting < %s | FileCheck %s -check-prefix=HOST-IR
-
-; REQUIRES: pollyacc
-
-; Entry: Contains (%loaded.ptr.preload.s2a = alloca double*) which is
-;   |    invariant load hoisted `%loaded.ptr`
-;   v
-; Run-time check --(failure branch)--> { old code - contains `%loaded.ptr` }
-;   |
-;  (success branch)
-;   |
-;   v
-; New Code: Should refer to `%loaded.ptr.preload.s2a`, which is
-;           the invariant load hoisted value, NOT `%loaded.ptr`.
-
-; In Polly, we preserve the old code and create a separate branch that executes
-; the GPU code if a run-time check succeeds.
-
-; We need to make sure that in the new branch, we pick up invariant load hoisted
-; values. The old values will belong to the old code branch.
-
-; In this case, we use to try to load the 'original' %loaded.ptr in the
-; 'New Code' branch,which is wrong. Check that this does not happen.
-
-; Check that we have a Scop with an invariant load of the array.
-; SCOP:       Function: f
-; SCOP-NEXT:  Region: %arrload---%for.exit
-; SCOP-NEXT:  Max Loop Depth:  1
-; SCOP-NEXT:  Invariant Accesses: {
-; SCOP-NEXT:          ReadAccess :=	[Reduction Type: NONE] [Scalar: 0]
-; SCOP-NEXT:              { Stmt_arrload[] -> MemRef_arr_of_ptrs[0] };
-
-
-
-; Check that we have the preloaded array.
-; HOST-IR: entry:
-; HOST-IR-NEXT:  %loaded.ptr.preload.s2a = alloca double*
-
-; Chek that we store the correct value in the preload.
-; polly.preload.begin:                              ; preds = %polly.split_new_and_old
-; HOST-IR: %polly.access.arr.of.ptrs = getelementptr double*, double** %arr.of.ptrs, i64 0
-; HOST-IR-NEXT: %polly.access.arr.of.ptrs.load = load double*, double** %polly.access.arr.of.ptrs
-; HOST-IR-NEXT: store double* %polly.access.arr.of.ptrs.load, double** %loaded.ptr.preload.s2a
-
-; Check that we get back data from the kernel.
-; HOST-IR: polly.acc.initialize:                             ; preds = %polly.start
-; HOST-IR: [[FIRSTINDEX:%.+]] = getelementptr double, double* %polly.access.arr.of.ptrs.load, i64 1
-; HOST-IR: [[BITCASTED:%.+]] = bitcast double* [[FIRSTINDEX]] to i8*
-; HOST-IR: call void @polly_copyFromDeviceToHost(i8* %p_dev_array_MemRef_loaded_ptr, i8* [[BITCASTED]], i64 800)
-
-; Check that the kernel launch is generated in the host IR.
-; This declaration would not have been generated unless a kernel launch exists.
-; HOST-IR: declare void @polly_launchKernel(i8*, i32, i32, i32, i32, i32, i8*)
-
-
-; C pseudocode equivalent
-; void f(double **arr_of_ptrs) {
-;     double *loaded_ptr = arr_of_ptrs[0];
-;     if (false) { return; }
-;     else {
-;         for(int i = 1; i < 100; i++) {
-;             loaded_ptr[i] = 42.0;
-;         }
-;     }
-; }
-
-
-target datalayout = "e-p:64:64:64-S128-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f16:16:16-f32:32:32-f64:64:64-f128:128:128-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
-target triple = "x86_64-unknown-linux-gnu"
-
-
-; Function Attrs: nounwind uwtable
-define void @f(double **%arr.of.ptrs) #0 {
-entry:
-  br label %arrload
-
-arrload:                                             ; preds = %"7"
-  %loaded.ptr = load double*, double** %arr.of.ptrs, align 8
-  br i1 false, label %"for.exit", label %"for.preheader"
-
-"for.preheader":                                       ; preds = %"51"
-  br label %"for.body"
-
-"for.body":                                             ; preds = %"53", %"53.lr.ph"
-  %indvar = phi i64 [ 1, %"for.preheader" ], [ %indvar.next, %"for.body" ]
-  %slot = getelementptr double, double* %loaded.ptr, i64 %indvar
-  store double 42.0, double* %slot, align 8
-
-  %indvar.next = add nuw nsw i64 %indvar, 1
-
-  %check = icmp sgt i64 %indvar.next, 100
-  br i1 %check, label %"for.exit", label %"for.body"
-
-"for.exit":                                             ; preds = %"52.54_crit_edge", %"51"
-    ret void
-}
-
-attributes #0 = { nounwind uwtable }
diff --git a/polly/test/GPGPU/invariant-load-hoisting-read-in-kernel.ll b/polly/test/GPGPU/invariant-load-hoisting-read-in-kernel.ll
deleted file mode 100644
index ad30ef6f9b24d..0000000000000
--- a/polly/test/GPGPU/invariant-load-hoisting-read-in-kernel.ll
+++ /dev/null
@@ -1,47 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-invariant-load-hoisting \
-; RUN: -S < %s | \
-; RUN: FileCheck -check-prefix=HOST-IR %s
-
-; RUN: opt %loadPolly -disable-output -polly-acc-dump-kernel-ir \
-; RUN: -polly-codegen-ppcg -polly-scops \
-; RUN: -polly-invariant-load-hoisting < %s | FileCheck -check-prefix=KERNEL-IR %s
-
-; REQUIRES: pollyacc
-
-; Verify that invariant loads used in a kernel statement are correctly forwarded
-; as subtree value to the GPU kernel.
-
-; HOST-IR: store float %polly.access.p.load, ptr %invariant.preload.s2a, align 4
-
-; KERNEL-IR:  define ptx_kernel void @FUNC_foo_SCOP_0_KERNEL_2({{.*}}ptr addrspace(1) %MemRef_indvar2f__phi{{.*}})
-; KERNEL-IR:   %indvar2f.phiops.reload = load float, ptr %indvar2f.phiops, align 4
-; KERNEL-IR:   store float %indvar2f.phiops.reload, ptr addrspace(1) %polly.access.MemRef_A, align 4
-
-; FIXME: store float %indvar2f.phiops.reload, ptr %indvar2f.phiops, align 4
-; For some reason the above instruction is emitted that stores back to the addess it was just loaded from.
-
-define void @foo(ptr %A, ptr %p) {
-entry:
-  br label %loop
-
-loop:
-  %indvar = phi i64 [0, %entry], [%indvar.next, %loop]
-  %indvar.next = add i64 %indvar, 1
-  %invariant = load float, ptr %p
-  %ptr = getelementptr float, ptr %A, i64 %indvar
-  store float 42.0, ptr %ptr
-  %cmp = icmp sle i64 %indvar, 1024
-  br i1 %cmp, label %loop, label %anotherloop
-
-anotherloop:
-  %indvar2 = phi i64 [0, %loop], [%indvar2.next, %anotherloop]
-  %indvar2f = phi float [%invariant, %loop], [%indvar2f, %anotherloop]
-  %indvar2.next = add i64 %indvar2, 1
-  store float %indvar2f, ptr %A
-  %cmp2 = icmp sle i64 %indvar2, 1024
-  br i1 %cmp2, label %anotherloop, label %end
-
-end:
-  ret void
-
-}
diff --git a/polly/test/GPGPU/invariant-load-hoisting-with-variable-bounds.ll b/polly/test/GPGPU/invariant-load-hoisting-with-variable-bounds.ll
deleted file mode 100644
index 7a650eeb22ee5..0000000000000
--- a/polly/test/GPGPU/invariant-load-hoisting-with-variable-bounds.ll
+++ /dev/null
@@ -1,62 +0,0 @@
-; RUN: opt %loadPolly -polly-invariant-load-hoisting -polly-print-scops -disable-output < %s | FileCheck %s -check-prefix=SCOP
-
-
-; RUN: opt %loadPolly -S -polly-use-llvm-names -polly-codegen-ppcg \
-; RUN: -polly-invariant-load-hoisting < %s | FileCheck %s -check-prefix=HOST-IR
-
-; REQUIRES: pollyacc
-
-; SCOP:      Function: f
-; SCOP-NEXT: Region: %entry.split---%for.end
-; SCOP-NEXT: Max Loop Depth:  1
-; SCOP-NEXT: Invariant Accesses: {
-; SCOP-NEXT:         ReadAccess :=	[Reduction Type: NONE] [Scalar: 0]
-; SCOP-NEXT:             [tmp1, tmp4] -> { Stmt_entry_split[] -> MemRef_begin[0] };
-; SCOP-NEXT:         Execution Context: [tmp1, tmp4] -> {  :  }
-; SCOP-NEXT:         ReadAccess :=	[Reduction Type: NONE] [Scalar: 0]
-; SCOP-NEXT:             [tmp1, tmp4] -> { Stmt_for_body[i0] -> MemRef_end[0] };
-; SCOP-NEXT:         Execution Context: [tmp1, tmp4] -> {  :  }
-; SCOP-NEXT: }
-
-
-; Check that the kernel launch is generated in the host IR.
-; This declaration would not have been generated unless a kernel launch exists.
-; HOST-IR: declare void @polly_launchKernel(ptr, i32, i32, i32, i32, i32, ptr)
-
-;    void f(int *begin, int *end, int *arr) {
-;      for (int i = *begin; i < *end; i++) {
-;        arr[i] = 0;
-;      }
-;    }
-;
-
-target datalayout = "e-m:o-p:32:32-f64:32:64-f80:128-n8:16:32-S128"
-
-define void @f(ptr %begin, ptr %end, ptr %arr) {
-entry:
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  %tmp1 = load i32, ptr %begin, align 4
-  %tmp41 = load i32, ptr %end, align 4
-  %cmp2 = icmp slt i32 %tmp1, %tmp41
-  br i1 %cmp2, label %for.body.lr.ph, label %for.end
-
-for.body.lr.ph:                                   ; preds = %entry.split
-  br label %for.body
-
-for.body:                                         ; preds = %for.body.lr.ph, %for.body
-  %i.03 = phi i32 [ %tmp1, %for.body.lr.ph ], [ %inc, %for.body ]
-  %arrayidx = getelementptr inbounds i32, ptr %arr, i32 %i.03
-  store i32 0, ptr %arrayidx, align 4
-  %inc = add nsw i32 %i.03, 1
-  %tmp4 = load i32, ptr %end, align 4
-  %cmp = icmp slt i32 %inc, %tmp4
-  br i1 %cmp, label %for.body, label %for.cond.for.end_crit_edge
-
-for.cond.for.end_crit_edge:                       ; preds = %for.body
-  br label %for.end
-
-for.end:                                          ; preds = %for.cond.for.end_crit_edge, %entry.split
-  ret void
-}
diff --git a/polly/test/GPGPU/invariant-load-hoisting-with-variable-lower-bound.ll b/polly/test/GPGPU/invariant-load-hoisting-with-variable-lower-bound.ll
deleted file mode 100644
index a637cc44c7a30..0000000000000
--- a/polly/test/GPGPU/invariant-load-hoisting-with-variable-lower-bound.ll
+++ /dev/null
@@ -1,56 +0,0 @@
-; RUN: opt %loadPolly -polly-invariant-load-hoisting -polly-print-scops -disable-output < %s | FileCheck %s -check-prefix=SCOP
-
-
-; RUN: opt %loadPolly -S -polly-use-llvm-names -polly-codegen-ppcg \
-; RUN: -polly-invariant-load-hoisting < %s | FileCheck %s -check-prefix=HOST-IR
-
-; REQUIRES: pollyacc
-
-; Check that we detect a scop with invariant accesses.
-; SCOP:      Function: f
-; SCOP-NEXT: Region: %entry.split---%for.end
-; SCOP-NEXT: Max Loop Depth:  1
-; SCOP-NEXT: Invariant Accesses: {
-; SCOP-NEXT:         ReadAccess :=	[Reduction Type: NONE] [Scalar: 0]
-; SCOP-NEXT:             [beginval] -> { Stmt_entry_split[] -> MemRef_begin[0] };
-; SCOP-NEXT:         Execution Context: [beginval] -> {  :  }
-; SCOP-NEXT: }
-
-; Check that the kernel launch is generated in the host IR.
-; This declaration would not have been generated unless a kernel launch exists.
-; HOST-IR: declare void @polly_launchKernel(ptr, i32, i32, i32, i32, i32, ptr)
-
-;
-; void f(int *begin, int *arr) {
-;     for (int i = *begin; i < 100; i++) {
-;         arr[i] = 0;
-;     }
-; }
-
-target datalayout = "e-m:o-p:32:32-f64:32:64-f80:128-n8:16:32-S128"
-
-define void @f(ptr %begin, ptr %arr) {
-entry:
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  %beginval = load i32, ptr %begin, align 4
-  %cmp1 = icmp slt i32 %beginval, 100
-  br i1 %cmp1, label %for.body, label %for.end
-
-
-
-for.body:                                         ; preds = %for.body.lr.ph, %for.body
-  %ival = phi i32 [ %beginval, %entry.split ], [ %inc, %for.body ]
-  %arrayidx = getelementptr inbounds i32, ptr %arr, i32 %ival
-  store i32 0, ptr %arrayidx, align 4
-  %inc = add nsw i32 %ival, 1
-  %cmp = icmp slt i32 %ival, 99
-  br i1 %cmp, label %for.body, label %for.cond.for.end_crit_edge
-
-for.cond.for.end_crit_edge:                       ; preds = %for.body
-  br label %for.end
-
-for.end:                                          ; preds = %for.cond.for.end_crit_edge, %entry.split
-  ret void
-}
diff --git a/polly/test/GPGPU/invariant-load-hoisting-with-variable-upper-bound.ll b/polly/test/GPGPU/invariant-load-hoisting-with-variable-upper-bound.ll
deleted file mode 100644
index 3c19a306734a7..0000000000000
--- a/polly/test/GPGPU/invariant-load-hoisting-with-variable-upper-bound.ll
+++ /dev/null
@@ -1,57 +0,0 @@
-; RUN: opt %loadPolly -polly-invariant-load-hoisting -polly-print-scops -disable-output < %s | FileCheck %s -check-prefix=SCOP
-; RUN: opt %loadPolly -S -polly-use-llvm-names -polly-codegen-ppcg -polly-invariant-load-hoisting < %s | FileCheck %s -check-prefix=HOST-IR
-
-; REQUIRES: pollyacc
-
-; Check that we detect a scop with invariant accesses.
-; SCOP:      Function: f
-; SCOP-NEXT: Region: %entry.split---%for.end
-; SCOP-NEXT: Max Loop Depth:  1
-; SCOP-NEXT: Invariant Accesses: {
-; SCOP-NEXT:         ReadAccess :=	[Reduction Type: NONE] [Scalar: 0]
-; SCOP-NEXT:             [tmp2] -> { Stmt_for_body[i0] -> MemRef_idx[0] };
-; SCOP-NEXT:         Execution Context: [tmp2] -> {  :  }
-; SCOP-NEXT: }
-
-; Check that kernel launch is generated in host IR.
-; the declare would not be generated unless a call to a kernel exists.
-; HOST-IR: declare void @polly_launchKernel(ptr, i32, i32, i32, i32, i32, ptr)
-
-; Check if we generate GPU code for simple loop with variable upper bound.
-; This always worked, but have this test to prevent regressions.
-;    void f(int *idx, int *arr) {
-;      for (int i = 0; i < *idx; i++) {
-;        arr[i] = 0;
-;      }
-;    }
-;
-target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @f(ptr %idx, ptr %arr) {
-entry:
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  %tmp21 = load i32, ptr %idx, align 4
-  %cmp2 = icmp sgt i32 %tmp21, 0
-  br i1 %cmp2, label %for.body.lr.ph, label %for.end
-
-for.body.lr.ph:                                   ; preds = %entry.split
-  br label %for.body
-
-for.body:                                         ; preds = %for.body.lr.ph, %for.body
-  %indvars.iv = phi i64 [ 0, %for.body.lr.ph ], [ %indvars.iv.next, %for.body ]
-  %arrayidx = getelementptr inbounds i32, ptr %arr, i64 %indvars.iv
-  store i32 0, ptr %arrayidx, align 4
-  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
-  %tmp2 = load i32, ptr %idx, align 4
-  %0 = sext i32 %tmp2 to i64
-  %cmp = icmp slt i64 %indvars.iv.next, %0
-  br i1 %cmp, label %for.body, label %for.cond.for.end_crit_edge
-
-for.cond.for.end_crit_edge:                       ; preds = %for.body
-  br label %for.end
-
-for.end:                                          ; preds = %for.cond.for.end_crit_edge, %entry.split
-  ret void
-}
diff --git a/polly/test/GPGPU/invariant-load-hoisting.ll b/polly/test/GPGPU/invariant-load-hoisting.ll
deleted file mode 100644
index 5ae1cfae255d7..0000000000000
--- a/polly/test/GPGPU/invariant-load-hoisting.ll
+++ /dev/null
@@ -1,116 +0,0 @@
-; RUN: opt %loadPolly -polly-invariant-load-hoisting -polly-print-scops -disable-output < %s | FileCheck %s -check-prefix=SCOP
-;
-; RUN: opt %loadPolly -polly-scops -S  -polly-invariant-load-hoisting \
-; RUN: -polly-codegen-ppcg < %s | FileCheck %s -check-prefix=HOST-IR
-;
-; RUN: opt %loadPolly -polly-invariant-load-hoisting -polly-codegen-ppcg -polly-acc-dump-kernel-ir -polly-print-scops -disable-output < %s | FileCheck %s -check-prefix=KERNEL-IR
-;
-; REQUIRES: pollyacc
-;
-; SCOP:       Function: f
-; SCOP-NEXT:  Region: %entry.split---%for.end26
-; SCOP-NEXT:  Max Loop Depth:  3
-; SCOP-NEXT:  Invariant Accesses: {
-; SCOP-NEXT:          ReadAccess :=	[Reduction Type: NONE] [Scalar: 0]
-; SCOP-NEXT:              [n, tmp12] -> { Stmt_for_body6[i0, i1, i2] -> MemRef_invariant[0] };
-; SCOP-NEXT:          Execution Context: [n, tmp12] -> {  : n > 0 }
-; SCOP-NEXT:  }
-; HOST-IR:      call void @polly_launchKernel(ptr %[[REGC:[0-9]+]], i32 %{{[0-9]+}}, i32 1, i32 32, i32 1, i32 1, ptr %polly_launch_0_params_i8ptr)
-; HOST-IR-NEXT: call void @polly_freeKernel(ptr %[[REGC]])
-
-; KERNEL-IR: define ptx_kernel void @FUNC_f_SCOP_0_KERNEL_0(ptr addrspace(1) %MemRef_B, ptr addrspace(1) %MemRef_A, i32 %n, i32 %tmp12, i32 %polly.preload.tmp21.merge)
-
-
-; Check that we generate correct GPU code in case of invariant load hoisting.
-;
-;
-;    static const int N = 3000;
-;
-;    void f(int A[N][N], int *invariant, int B[N][N], int n) {
-;      for (int i = 0; i < n; i++) {
-;        for (int j = 0; j < n; j++) {
-;          for (int k = 0; k < n; k++) {
-;
-;            A[*invariant][k] = B[k][k];
-;            A[k][*invariant] += B[k][k];
-;          }
-;        }
-;      }
-;    }
-;
-
-define void @f(ptr %A, ptr %invariant, ptr %B, i32 %n) {
-entry:
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  %cmp6 = icmp sgt i32 %n, 0
-  br i1 %cmp6, label %for.cond1.preheader.lr.ph, label %for.end26
-
-for.cond1.preheader.lr.ph:                        ; preds = %entry.split
-  br label %for.cond1.preheader
-
-for.cond1.preheader:                              ; preds = %for.cond1.preheader.lr.ph, %for.inc24
-  %i.07 = phi i32 [ 0, %for.cond1.preheader.lr.ph ], [ %inc25, %for.inc24 ]
-  %cmp23 = icmp sgt i32 %n, 0
-  br i1 %cmp23, label %for.cond4.preheader.lr.ph, label %for.inc24
-
-for.cond4.preheader.lr.ph:                        ; preds = %for.cond1.preheader
-  br label %for.cond4.preheader
-
-for.cond4.preheader:                              ; preds = %for.cond4.preheader.lr.ph, %for.inc21
-  %j.04 = phi i32 [ 0, %for.cond4.preheader.lr.ph ], [ %inc22, %for.inc21 ]
-  %cmp51 = icmp sgt i32 %n, 0
-  br i1 %cmp51, label %for.body6.lr.ph, label %for.inc21
-
-for.body6.lr.ph:                                  ; preds = %for.cond4.preheader
-  br label %for.body6
-
-for.body6:                                        ; preds = %for.body6.lr.ph, %for.body6
-  %k.02 = phi i32 [ 0, %for.body6.lr.ph ], [ %inc, %for.body6 ]
-  %idxprom = sext i32 %k.02 to i64
-  %idxprom7 = sext i32 %k.02 to i64
-  %arrayidx8 = getelementptr inbounds [3000 x i32], ptr %B, i64 %idxprom, i64 %idxprom7
-  %tmp9 = load i32, ptr %arrayidx8, align 4
-  %tmp12 = load i32, ptr %invariant, align 4
-  %idxprom9 = sext i32 %tmp12 to i64
-  %idxprom11 = sext i32 %k.02 to i64
-  %arrayidx12 = getelementptr inbounds [3000 x i32], ptr %A, i64 %idxprom9, i64 %idxprom11
-  store i32 %tmp9, ptr %arrayidx12, align 4
-  %idxprom13 = sext i32 %k.02 to i64
-  %idxprom15 = sext i32 %k.02 to i64
-  %arrayidx16 = getelementptr inbounds [3000 x i32], ptr %B, i64 %idxprom13, i64 %idxprom15
-  %tmp17 = load i32, ptr %arrayidx16, align 4
-  %idxprom17 = sext i32 %k.02 to i64
-  %tmp21 = load i32, ptr %invariant, align 4
-  %idxprom19 = sext i32 %tmp21 to i64
-  %arrayidx20 = getelementptr inbounds [3000 x i32], ptr %A, i64 %idxprom17, i64 %idxprom19
-  %tmp22 = load i32, ptr %arrayidx20, align 4
-  %add = add nsw i32 %tmp22, %tmp17
-  store i32 %add, ptr %arrayidx20, align 4
-  %inc = add nuw nsw i32 %k.02, 1
-  %cmp5 = icmp slt i32 %inc, %n
-  br i1 %cmp5, label %for.body6, label %for.cond4.for.inc21_crit_edge
-
-for.cond4.for.inc21_crit_edge:                    ; preds = %for.body6
-  br label %for.inc21
-
-for.inc21:                                        ; preds = %for.cond4.for.inc21_crit_edge, %for.cond4.preheader
-  %inc22 = add nuw nsw i32 %j.04, 1
-  %cmp2 = icmp slt i32 %inc22, %n
-  br i1 %cmp2, label %for.cond4.preheader, label %for.cond1.for.inc24_crit_edge
-
-for.cond1.for.inc24_crit_edge:                    ; preds = %for.inc21
-  br label %for.inc24
-
-for.inc24:                                        ; preds = %for.cond1.for.inc24_crit_edge, %for.cond1.preheader
-  %inc25 = add nuw nsw i32 %i.07, 1
-  %cmp = icmp slt i32 %inc25, %n
-  br i1 %cmp, label %for.cond1.preheader, label %for.cond.for.end26_crit_edge
-
-for.cond.for.end26_crit_edge:                     ; preds = %for.inc24
-  br label %for.end26
-
-for.end26:                                        ; preds = %for.cond.for.end26_crit_edge, %entry.split
-  ret void
-}
diff --git a/polly/test/GPGPU/invariant-load-of-scalar.ll b/polly/test/GPGPU/invariant-load-of-scalar.ll
deleted file mode 100644
index fbc1d4d7eceee..0000000000000
--- a/polly/test/GPGPU/invariant-load-of-scalar.ll
+++ /dev/null
@@ -1,81 +0,0 @@
-; RUN: opt %loadPolly -polly-invariant-load-hoisting -polly-print-scops -disable-output < %s | FileCheck -check-prefix=SCOP %s
-
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-invariant-load-hoisting \
-; RUN: -S < %s | \
-; RUN: FileCheck -check-prefix=HOST-IR %s
-
-
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-invariant-load-hoisting \
-; RUN: -disable-output -polly-acc-dump-kernel-ir < %s | \
-; RUN: FileCheck -check-prefix=KERNEL-IR %s
-
-; REQUIRES: pollyacc
-
-; Check that we offload invariant loads of scalars correctly.
-
-; Check that invariant loads are present.
-; SCOP:      Function: checkPrivatization
-; SCOP-NEXT: Region: %entry.split---%for.end
-; SCOP-NEXT: Max Loop Depth:  1
-; SCOP-NEXT: Invariant Accesses: {
-; SCOP-NEXT:         ReadAccess :=	[Reduction Type: NONE] [Scalar: 0]
-; SCOP-NEXT:             [tmp, tmp2] -> { Stmt_entry_split[] -> MemRef_begin[0] };
-; SCOP-NEXT:         Execution Context: [tmp, tmp2] -> {  :  }
-; SCOP-NEXT:         ReadAccess :=	[Reduction Type: NONE] [Scalar: 0]
-; SCOP-NEXT:             [tmp, tmp2] -> { Stmt_for_body[i0] -> MemRef_end[0] };
-; SCOP-NEXT:         Execution Context: [tmp, tmp2] -> {  :  }
-; SCOP-NEXT: }
-;
-
-; Check that we do not actually allocate arrays for %begin, %end, since they are
-; invariant load hoisted.
-; HOST-IR: %p_dev_array_MemRef_A = call ptr @polly_allocateMemoryForDevice
-; HOST-IR-NOT: call ptr @polly_allocateMemoryForDevice
-
-; Check that we send the invariant loaded scalars as parameters to the
-; kernel function.
-; KERNEL-IR: define ptx_kernel void @FUNC_checkPrivatization_SCOP_0_KERNEL_0
-; KERNEL-IR-SAME: (ptr addrspace(1) %MemRef_A, i32 %tmp,
-; KERNEL-IR-SAME: i32 %tmp2, i32 %polly.access.begin.load)
-
-
-; void checkScalarPointerOffload(int A[], int *begin, int *end) {
-;     for(int i = *begin; i < *end; i++) {
-;         A[i] = 10;
-;     }
-; }
-
-target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
-target triple = "x86_64-apple-macosx10.12.0"
-
-define void @checkPrivatization(ptr %A, ptr %begin, ptr %end) {
-entry:
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  %tmp = load i32, ptr %begin, align 4
-  %tmp21 = load i32, ptr %end, align 4
-  %cmp3 = icmp slt i32 %tmp, %tmp21
-  br i1 %cmp3, label %for.body.lr.ph, label %for.end
-
-for.body.lr.ph:                                   ; preds = %entry.split
-  %tmp1 = sext i32 %tmp to i64
-  br label %for.body
-
-for.body:                                         ; preds = %for.body.lr.ph, %for.body
-  %indvars.iv4 = phi i64 [ %tmp1, %for.body.lr.ph ], [ %indvars.iv.next, %for.body ]
-  %arrayidx = getelementptr inbounds i32, ptr %A, i64 %indvars.iv4
-  store i32 10, ptr %arrayidx, align 4
-  %indvars.iv.next = add i64 %indvars.iv4, 1
-  %tmp2 = load i32, ptr %end, align 4
-  %tmp3 = sext i32 %tmp2 to i64
-  %cmp = icmp slt i64 %indvars.iv.next, %tmp3
-  br i1 %cmp, label %for.body, label %for.cond.for.end_crit_edge
-
-for.cond.for.end_crit_edge:                       ; preds = %for.body
-  br label %for.end
-
-for.end:                                          ; preds = %for.cond.for.end_crit_edge, %entry.split
-  ret void
-}
-
diff --git a/polly/test/GPGPU/kernel-params-only-some-arrays.ll b/polly/test/GPGPU/kernel-params-only-some-arrays.ll
deleted file mode 100644
index 87ae470e29bc8..0000000000000
--- a/polly/test/GPGPU/kernel-params-only-some-arrays.ll
+++ /dev/null
@@ -1,106 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen-ppcg -polly-acc-dump-kernel-ir \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=KERNEL %s
-
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen-ppcg \
-; RUN: -S < %s | \
-; RUN: FileCheck -check-prefix=IR %s
-
-; REQUIRES: pollyacc
-;
-;    void kernel_params_only_some_arrays(float A[], float B[]) {
-;      for (long i = 0; i < 32; i++)
-;        A[i] += 42;
-;
-;      for (long i = 0; i < 32; i++)
-;        B[i] += 42;
-;    }
-
-; KERNEL: ; ModuleID = 'FUNC_kernel_params_only_some_arrays_SCOP_0_KERNEL_0'
-; KERNEL-NEXT: source_filename = "FUNC_kernel_params_only_some_arrays_SCOP_0_KERNEL_0"
-; KERNEL-NEXT: target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-i128:128:128-f32:32:32-f64:64:64-v16:16:16-v32:32:32-v64:64:64-v128:128:128-n16:32:64"
-; KERNEL-NEXT: target triple = "nvptx64-nvidia-cuda"
-
-; KERNEL: define ptx_kernel void @FUNC_kernel_params_only_some_arrays_SCOP_0_KERNEL_0(i8 addrspace(1)* %MemRef_B)
-; KERNEL-NEXT:   entry:
-; KERNEL-NEXT:     %0 = call i32 @llvm.nvvm.read.ptx.sreg.ctaid.x()
-; KERNEL-NEXT:     %b0 = zext i32 %0 to i64
-; KERNEL-NEXT:     %1 = call i32 @llvm.nvvm.read.ptx.sreg.tid.x()
-; KERNEL-NEXT:     %t0 = zext i32 %1 to i64
-
-; KERNEL:     ret void
-; KERNEL-NEXT: }
-
-; KERNEL: ; ModuleID = 'FUNC_kernel_params_only_some_arrays_SCOP_0_KERNEL_1'
-; KERNEL-NEXT: source_filename = "FUNC_kernel_params_only_some_arrays_SCOP_0_KERNEL_1"
-; KERNEL-NEXT: target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-i128:128:128-f32:32:32-f64:64:64-v16:16:16-v32:32:32-v64:64:64-v128:128:128-n16:32:64"
-; KERNEL-NEXT: target triple = "nvptx64-nvidia-cuda"
-
-; KERNEL: define ptx_kernel void @FUNC_kernel_params_only_some_arrays_SCOP_0_KERNEL_1(i8 addrspace(1)* %MemRef_A)
-; KERNEL-NEXT:   entry:
-; KERNEL-NEXT:     %0 = call i32 @llvm.nvvm.read.ptx.sreg.ctaid.x()
-; KERNEL-NEXT:     %b0 = zext i32 %0 to i64
-; KERNEL-NEXT:     %1 = call i32 @llvm.nvvm.read.ptx.sreg.tid.x()
-; KERNEL-NEXT:     %t0 = zext i32 %1 to i64
-
-; KERNEL:     ret void
-; KERNEL-NEXT: }
-
-
-; IR:       [[DEVPTR:%.*]] = call i8* @polly_getDevicePtr(i8* %p_dev_array_MemRef_B)
-; IR-NEXT:  [[SLOT:%.*]] = getelementptr [1 x i8*], [1 x i8*]* %polly_launch_0_params, i64 0, i64 0
-; IR-NEXT:  store i8* [[DEVPTR]], i8** %polly_launch_0_param_0
-; IR-NEXT:  [[DATA:%.*]] = bitcast i8** %polly_launch_0_param_0 to i8*
-; IR-NEXT:  store i8* [[DATA]], i8** [[SLOT]]
-
-; IR:       [[DEVPTR:%.*]] = call i8* @polly_getDevicePtr(i8* %p_dev_array_MemRef_A)
-; IR-NEXT:  [[SLOT:%.*]] = getelementptr [1 x i8*], [1 x i8*]* %polly_launch_1_params, i64 0, i64 0
-; IR-NEXT:  store i8* [[DEVPTR]], i8** %polly_launch_1_param_0
-; IR-NEXT:  [[DATA:%.*]] = bitcast i8** %polly_launch_1_param_0 to i8*
-; IR-NEXT:  store i8* [[DATA]], i8** [[SLOT]]
-
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @kernel_params_only_some_arrays(float* %A, float* %B) {
-entry:
-  br label %for.cond
-
-for.cond:                                         ; preds = %for.inc, %entry
-  %i.0 = phi i64 [ 0, %entry ], [ %inc, %for.inc ]
-  %exitcond1 = icmp ne i64 %i.0, 32
-  br i1 %exitcond1, label %for.body, label %for.end
-
-for.body:                                         ; preds = %for.cond
-  %arrayidx = getelementptr inbounds float, float* %A, i64 %i.0
-  %tmp = load float, float* %arrayidx, align 4
-  %add = fadd float %tmp, 4.200000e+01
-  store float %add, float* %arrayidx, align 4
-  br label %for.inc
-
-for.inc:                                          ; preds = %for.body
-  %inc = add nuw nsw i64 %i.0, 1
-  br label %for.cond
-
-for.end:                                          ; preds = %for.cond
-  br label %for.cond2
-
-for.cond2:                                        ; preds = %for.inc7, %for.end
-  %i1.0 = phi i64 [ 0, %for.end ], [ %inc8, %for.inc7 ]
-  %exitcond = icmp ne i64 %i1.0, 32
-  br i1 %exitcond, label %for.body4, label %for.end9
-
-for.body4:                                        ; preds = %for.cond2
-  %arrayidx5 = getelementptr inbounds float, float* %B, i64 %i1.0
-  %tmp2 = load float, float* %arrayidx5, align 4
-  %add6 = fadd float %tmp2, 4.200000e+01
-  store float %add6, float* %arrayidx5, align 4
-  br label %for.inc7
-
-for.inc7:                                         ; preds = %for.body4
-  %inc8 = add nuw nsw i64 %i1.0, 1
-  br label %for.cond2
-
-for.end9:                                         ; preds = %for.cond2
-  ret void
-}
diff --git a/polly/test/GPGPU/kernel-params-scop-parameter.ll b/polly/test/GPGPU/kernel-params-scop-parameter.ll
deleted file mode 100644
index 527492bfd5fba..0000000000000
--- a/polly/test/GPGPU/kernel-params-scop-parameter.ll
+++ /dev/null
@@ -1,38 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-kernel-ir \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=KERNEL-IR %s
-
-; REQUIRES: pollyacc
-
-;    void kernel_params_scop_parameter(float A[], long n) {
-;      for (long i = 0; i < n; i++)
-;        A[i] += 42;
-;    }
-
-; KERNEL-IR: define ptx_kernel void @FUNC_kernel_params_scop_parameter_SCOP_0_KERNEL_0(ptr addrspace(1) %MemRef_A, i64 %n)
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @kernel_params_scop_parameter(ptr %A, i64 %n) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb6, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp7, %bb6 ]
-  %tmp = icmp slt i64 %i.0, %n
-  br i1 %tmp, label %bb2, label %bb8
-
-bb2:                                              ; preds = %bb1
-  %tmp3 = getelementptr inbounds float, ptr %A, i64 %i.0
-  %tmp4 = load float, ptr %tmp3, align 4
-  %tmp5 = fadd float %tmp4, 4.200000e+01
-  store float %tmp5, ptr %tmp3, align 4
-  br label %bb6
-
-bb6:                                              ; preds = %bb2
-  %tmp7 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb8:                                              ; preds = %bb1
-  ret void
-}
diff --git a/polly/test/GPGPU/kernels-names-across-scops-funcs.ll b/polly/test/GPGPU/kernels-names-across-scops-funcs.ll
deleted file mode 100644
index 57fe70ec0d9b3..0000000000000
--- a/polly/test/GPGPU/kernels-names-across-scops-funcs.ll
+++ /dev/null
@@ -1,124 +0,0 @@
-; RUN: opt %loadPolly -polly-process-unprofitable -polly-codegen-ppcg \
-; RUN: -polly-acc-dump-kernel-ir -disable-output < %s | \
-; RUN: FileCheck -check-prefix=KERNEL %s
-
-; REQUIRES: pollyacc
-
-; KERNEL: define ptx_kernel void @FUNC_foo_SCOP_0_KERNEL_0(ptr addrspace(1) %MemRef_arg1, i32 %arg) #0 {
-; KERNEL: define ptx_kernel void @FUNC_foo_SCOP_1_KERNEL_0(ptr addrspace(1) %MemRef_arg1, i32 %arg) #0 {
-; KERNEL: define ptx_kernel void @FUNC_foo2_SCOP_0_KERNEL_0(ptr addrspace(1) %MemRef_arg1, i32 %arg) #0 {
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-; Function Attrs: nounwind uwtable
-define void @foo(i32 %arg, ptr %arg1) #0 {
-bb:
-  br label %bb2
-
-bb2:                                              ; preds = %bb
-  %tmp = icmp sgt i32 %arg, 0
-  br i1 %tmp, label %bb3, label %bb13
-
-bb3:                                              ; preds = %bb2
-  br label %bb4
-
-bb4:                                              ; preds = %bb4, %bb3
-  %tmp5 = phi i64 [ 0, %bb3 ], [ %tmp9, %bb4 ]
-  %tmp6 = getelementptr inbounds i32, ptr %arg1, i64 %tmp5
-  %tmp7 = load i32, ptr %tmp6, align 4, !tbaa !2
-  %tmp8 = add nsw i32 %tmp7, 1
-  store i32 %tmp8, ptr %tmp6, align 4, !tbaa !2
-  %tmp9 = add nuw nsw i64 %tmp5, 1
-  %tmp10 = zext i32 %arg to i64
-  %tmp11 = icmp ne i64 %tmp9, %tmp10
-  br i1 %tmp11, label %bb4, label %bb12
-
-bb12:                                             ; preds = %bb4
-  br label %bb13
-
-bb13:                                             ; preds = %bb12, %bb2
-  %tmp14 = tail call i64 @clock() #3
-  %tmp15 = icmp eq i64 %tmp14, 0
-  br i1 %tmp15, label %bb16, label %bb29
-
-bb16:                                             ; preds = %bb13
-  %tmp17 = icmp sgt i32 %arg, 0
-  br i1 %tmp17, label %bb18, label %bb28
-
-bb18:                                             ; preds = %bb16
-  br label %bb19
-
-bb19:                                             ; preds = %bb19, %bb18
-  %tmp20 = phi i64 [ 0, %bb18 ], [ %tmp24, %bb19 ]
-  %tmp21 = getelementptr inbounds i32, ptr %arg1, i64 %tmp20
-  %tmp22 = load i32, ptr %tmp21, align 4, !tbaa !2
-  %tmp23 = add nsw i32 %tmp22, 1
-  store i32 %tmp23, ptr %tmp21, align 4, !tbaa !2
-  %tmp24 = add nuw nsw i64 %tmp20, 1
-  %tmp25 = zext i32 %arg to i64
-  %tmp26 = icmp ne i64 %tmp24, %tmp25
-  br i1 %tmp26, label %bb19, label %bb27
-
-bb27:                                             ; preds = %bb19
-  br label %bb28
-
-bb28:                                             ; preds = %bb27, %bb16
-  br label %bb29
-
-bb29:                                             ; preds = %bb28, %bb13
-  ret void
-}
-
-; Function Attrs: argmemonly nounwind
-declare void @llvm.lifetime.start.p0(i64, ptr nocapture) #1
-
-; Function Attrs: nounwind
-declare i64 @clock() #2
-
-; Function Attrs: argmemonly nounwind
-declare void @llvm.lifetime.end.p0(i64, ptr nocapture) #1
-
-; Function Attrs: nounwind uwtable
-define void @foo2(i32 %arg, ptr %arg1) #0 {
-bb:
-  br label %bb2
-
-bb2:                                              ; preds = %bb
-  %tmp = icmp sgt i32 %arg, 0
-  br i1 %tmp, label %bb3, label %bb13
-
-bb3:                                              ; preds = %bb2
-  br label %bb4
-
-bb4:                                              ; preds = %bb4, %bb3
-  %tmp5 = phi i64 [ 0, %bb3 ], [ %tmp9, %bb4 ]
-  %tmp6 = getelementptr inbounds i32, ptr %arg1, i64 %tmp5
-  %tmp7 = load i32, ptr %tmp6, align 4, !tbaa !2
-  %tmp8 = add nsw i32 %tmp7, 1
-  store i32 %tmp8, ptr %tmp6, align 4, !tbaa !2
-  %tmp9 = add nuw nsw i64 %tmp5, 1
-  %tmp10 = zext i32 %arg to i64
-  %tmp11 = icmp ne i64 %tmp9, %tmp10
-  br i1 %tmp11, label %bb4, label %bb12
-
-bb12:                                             ; preds = %bb4
-  br label %bb13
-
-bb13:                                             ; preds = %bb12, %bb2
-  ret void
-}
-
-attributes #0 = { nounwind uwtable "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "less-precise-fpmad"="false" "frame-pointer"="none" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+fxsr,+mmx,+sse,+sse2,+x87" "unsafe-fp-math"="false" "use-soft-float"="false" }
-attributes #1 = { argmemonly nounwind }
-attributes #2 = { nounwind "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "less-precise-fpmad"="false" "frame-pointer"="none" "no-infs-fp-math"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+fxsr,+mmx,+sse,+sse2,+x87" "unsafe-fp-math"="false" "use-soft-float"="false" }
-attributes #3 = { nounwind }
-
-!llvm.module.flags = !{!0}
-!llvm.ident = !{!1}
-
-!0 = !{i32 1, !"wchar_size", i32 4}
-!1 = !{!"clang version 5.0.0"}
-!2 = !{!3, !3, i64 0}
-!3 = !{!"int", !4, i64 0}
-!4 = !{!"omnipotent char", !5, i64 0}
-!5 = !{!"Simple C/C++ TBAA"}
diff --git a/polly/test/GPGPU/libdevice-functions-copied-into-kernel.ll b/polly/test/GPGPU/libdevice-functions-copied-into-kernel.ll
deleted file mode 100644
index 0f8405dad7e8d..0000000000000
--- a/polly/test/GPGPU/libdevice-functions-copied-into-kernel.ll
+++ /dev/null
@@ -1,89 +0,0 @@
-; RUN: opt %loadPolly -polly-acc-libdevice=%S/Inputs/libdevice-functions-copied-into-kernel_libdevice.ll -polly-print-scops -disable-output < %s | FileCheck %s --check-prefix=SCOP
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-kernel-ir -polly-acc-libdevice=%S/Inputs/libdevice-functions-copied-into-kernel_libdevice.ll -disable-output < %s | FileCheck %s --check-prefix=KERNEL-IR
-; RUN: opt %loadPolly -S -polly-codegen-ppcg  < %s \
-; RUN: -polly-acc-libdevice=%S/Inputs/libdevice-functions-copied-into-kernel_libdevice.ll \
-; RUN:     | FileCheck %s --check-prefix=HOST-IR
-
-; Test that we do recognise and codegen a kernel that has functions that can
-; be mapped to NVIDIA's libdevice
-
-; REQUIRES: pollyacc
-
-; Check that we model the kernel as a scop.
-; SCOP:      Function: f
-; SCOP-NEXT:       Region: %entry.split---%for.end
-
-; Check that the intrinsic call is present in the kernel IR.
-; KERNEL-IR:   %p_expf = tail call float @__nv_expf(float %A.arr.i.val_p_scalar_)
-; KERNEL-IR:   %p_cosf = tail call float @__nv_cosf(float %p_expf)
-; KERNEL-IR:   %p_logf = tail call float @__nv_logf(float %p_cosf)
-
-; Powi and exp cannot be lowered directly. Rather, we expect them to be
-; lowered by libdevice.
-; KERNEL-IR: %p_powi = tail call float @__nv_powif(float %p_logf, i32 2)
-; KERNEL-IR: %p_exp = tail call float @__nv_expf(float %p_powi)
-
-; Check that kernel launch is generated in host IR.
-; the declare would not be generated unless a call to a kernel exists.
-; HOST-IR: declare void @polly_launchKernel(ptr, i32, i32, i32, i32, i32, ptr)
-
-
-; void f(float *A, float *B, int N) {
-;   for(int i = 0; i < N; i++) {
-;       float tmp0 = A[i];
-;       float expf  = expf(tmp1);
-;       cosf = cosf(expf);
-;       logf = logf(cosf);
-;       powi = powi(logf, 2);
-;       exp = exp(powi);
-;       B[i] = logf;
-;   }
-; }
-
-target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @f(ptr %A, ptr %B, i32 %N) {
-entry:
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  %cmp1 = icmp sgt i32 %N, 0
-  br i1 %cmp1, label %for.body.lr.ph, label %for.end
-
-for.body.lr.ph:                                   ; preds = %entry.split
-  br label %for.body
-
-for.body:                                         ; preds = %for.body.lr.ph, %for.body
-  %indvars.iv = phi i64 [ 0, %for.body.lr.ph ], [ %indvars.iv.next, %for.body ]
-  %A.arr.i = getelementptr inbounds float, ptr %A, i64 %indvars.iv
-  %A.arr.i.val = load float, ptr %A.arr.i, align 4
-  ; Call to intrinsics that should be part of the kernel.
-  %expf = tail call float @expf(float %A.arr.i.val)
-  %cosf = tail call float @cosf(float %expf)
-  %logf = tail call float @logf(float %cosf)
-  %powi = tail call float @llvm.powi.f32.i32(float %logf, i32 2)
-  %exp = tail call float @llvm.exp.f32(float %powi)
-  %B.arr.i = getelementptr inbounds float, ptr %B, i64 %indvars.iv
-  store float %exp, ptr %B.arr.i, align 4
-
-  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
-  %wide.trip.count = zext i32 %N to i64
-  %exitcond = icmp ne i64 %indvars.iv.next, %wide.trip.count
-  br i1 %exitcond, label %for.body, label %for.cond.for.end_crit_edge
-
-for.cond.for.end_crit_edge:                       ; preds = %for.body
-  br label %for.end
-
-for.end:                                          ; preds = %for.cond.for.end_crit_edge, %entry.split
-  ret void
-}
-
-; Function Attrs: nounwind readnone
-declare float @expf(float) #0
-declare float @cosf(float) #0
-declare float @logf(float) #0
-declare float @llvm.powi.f32.i32(float, i32) #0
-declare float @llvm.exp.f32(float) #0
-
-attributes #0 = { nounwind readnone }
-
diff --git a/polly/test/GPGPU/live-range-reordering-with-privatization.ll b/polly/test/GPGPU/live-range-reordering-with-privatization.ll
deleted file mode 100644
index 3b047fd557ff8..0000000000000
--- a/polly/test/GPGPU/live-range-reordering-with-privatization.ll
+++ /dev/null
@@ -1,78 +0,0 @@
-  ; RUN: opt %loadPolly -polly-use-llvm-names -polly-scops \
-; RUN: -polly-invariant-load-hoisting -polly-codegen-ppcg \
-; RUN: -polly-acc-dump-code -disable-output \
-; RUN:   < %s | FileCheck %s -check-prefix=CODE
-
-; RUN: opt %loadPolly -polly-use-llvm-names -polly-scops \
-; RUN: -polly-invariant-load-hoisting -polly-codegen-ppcg \
-; RUN: -polly-acc-dump-kernel-ir -disable-output \
-; RUN:   < %s | FileCheck %s -check-prefix=KERNELIR
-
-; REQUIRES: pollyacc
-
-;    void f(const int *end, int *arr, const int *control, const int *readarr) {
-;      for (int i = 0; i < *end; i++) {
-;        int t = 0;
-;        if (*control > 3) {
-;          t += readarr[i];
-;        }
-;        arr[i] = t;
-;      }
-;    }
-
-; This test case tests the ability to infer that `t` is local to each loop
-; iteration, and can therefore be privatized.
-
-; CODE: # kernel0
-; CODE-NEXT: for (int c0 = 0; c0 <= (tmp - 32 * b0 - 1) / 1048576; c0 += 1)
-; CODE-NEXT:   if (tmp >= 32 * b0 + t0 + 1048576 * c0 + 1) {
-; CODE-NEXT:     Stmt_for_body_last(32 * b0 + t0 + 1048576 * c0);
-; CODE-NEXT:     if (tmp1 >= 4)
-; CODE-NEXT:       Stmt_if_then(32 * b0 + t0 + 1048576 * c0);
-; CODE-NEXT:     Stmt_if_end(32 * b0 + t0 + 1048576 * c0);
-; CODE-NEXT:   }
-
-; KERNELIR: %private_array = alloca i32
-
-target datalayout = "e-m:o-p:32:32-f64:32:64-f80:128-n8:16:32-S128"
-target triple = "i386-apple-macosx10.12.0"
-
-define void @f(ptr %end, ptr %arr, ptr %control, ptr %readarr) {
-entry:
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  %tmp3 = load i32, ptr %end, align 4
-  %cmp4 = icmp sgt i32 %tmp3, 0
-  br i1 %cmp4, label %for.body.lr.ph, label %for.end
-
-for.body.lr.ph:                                   ; preds = %entry.split
-  br label %for.body
-
-for.body:                                         ; preds = %for.body.lr.ph, %if.end
-  %i.05 = phi i32 [ 0, %for.body.lr.ph ], [ %inc, %if.end ]
-  %tmp1 = load i32, ptr %control, align 4
-  %cmp1 = icmp sgt i32 %tmp1, 3
-  br i1 %cmp1, label %if.then, label %if.end
-
-if.then:                                          ; preds = %for.body
-  %arrayidx = getelementptr inbounds i32, ptr %readarr, i32 %i.05
-  %tmp2 = load i32, ptr %arrayidx, align 4
-  br label %if.end
-
-if.end:                                           ; preds = %if.then, %for.body
-  %t.0 = phi i32 [ %tmp2, %if.then ], [ 0, %for.body ]
-  %arrayidx2 = getelementptr inbounds i32, ptr %arr, i32 %i.05
-  store i32 %t.0, ptr %arrayidx2, align 4
-  %inc = add nuw nsw i32 %i.05, 1
-  %tmp = load i32, ptr %end, align 4
-  %cmp = icmp slt i32 %inc, %tmp
-  br i1 %cmp, label %for.body, label %for.cond.for.end_crit_edge
-
-for.cond.for.end_crit_edge:                       ; preds = %if.end
-  br label %for.end
-
-for.end:                                          ; preds = %for.cond.for.end_crit_edge, %entry.split
-  ret void
-}
-
diff --git a/polly/test/GPGPU/loops-outside-scop.ll b/polly/test/GPGPU/loops-outside-scop.ll
deleted file mode 100644
index 36b3a706338a5..0000000000000
--- a/polly/test/GPGPU/loops-outside-scop.ll
+++ /dev/null
@@ -1,67 +0,0 @@
-; RUN: opt %loadPolly -polly-print-scops -disable-output < %s | FileCheck %s -check-prefix=SCOP
-
-; There is no FileCheck because we want to make sure that this doesn't crash.
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-fail-on-verify-module-failure \
-; RUN: -disable-output < %s
-
-; REQUIRES: pollyacc
-
-; Due to the existence of the `fence` call, We can only detect the inner loop
-; and not the outer loop. PPCGCodeGeneration had not implemented this case.
-; The fix was to pull the implementation from `IslNodeBuilder.
-
-; Make sure that we only capture the inner loop
-; SCOP:      Function: f
-; SCOP-NEXT: Region: %for2.body---%for2.body.fence
-; SCOP-NEXT: Max Loop Depth:  1
-
-target datalayout = "e-p:64:64:64-S128-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f16:16:16-f32:32:32-f64:64:64-f128:128:128-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
-target triple = "x86_64-unknown-linux-gnu"
-
-declare void @fn_to_fence(ptr %val)
-
-; void f(int *arr, bool shouldcont) {
-;     for(int i = 0; ; i++) {
-;         for(int j = 0; j < 10; j++) {
-;             arr[j] = i;
-;         }
-;         fence(arr);
-;         if (!shouldcont) break;
-;     }
-; }
-
-
-; Function Attrs: nounwind uwtable
-define void @f(ptr %arr, i1 %shouldcont) #1 {
-entry:
-  br label %for.init
-
-for.init:                                             ; preds = %for.end, %entry.split
-  %i = phi i32 [ %i.next, %for.end ], [ 0, %entry ]
-  br label %for2.body
-
-for2.body:                                             ; preds = %"65", %"64"
-  %j = phi i32 [ %j.next, %for2.body ], [ 0, %for.init ]
-  %j.sext = sext i32 %j to i64
-  %arr.slot = getelementptr i32, ptr %arr, i64 %j.sext
-  store i32 %i, ptr %arr.slot, align 4
-  %exitcond = icmp eq i32 %j, 10
-  %j.next = add i32 %j, 1
-  br i1 %exitcond, label %for2.body.fence, label %for2.body
-
-for2.body.fence:                                             ; preds = %"65"
-  call void @fn_to_fence(ptr %arr) #2
-  br i1 %shouldcont, label %for.end, label %exit
-for.end:                                             ; preds = %"69"
-  %i.next = add i32 %i, 1
-  br label %for.init
-
-exit:                                             ; preds = %"69"
-  ret void
-
-}
-
-
-attributes #0 = { argmemonly nounwind }
-attributes #1 = { nounwind uwtable }
-attributes #2 = { nounwind }
diff --git a/polly/test/GPGPU/managed-memory-rewrite-alloca.ll b/polly/test/GPGPU/managed-memory-rewrite-alloca.ll
deleted file mode 100644
index 6dbd87db5eb5e..0000000000000
--- a/polly/test/GPGPU/managed-memory-rewrite-alloca.ll
+++ /dev/null
@@ -1,60 +0,0 @@
-; RUN: opt %loadPolly -polly-print-scops -disable-output < %s | FileCheck %s --check-prefix=SCOP
-
-; RUN: opt %loadPolly -S  -polly-process-unprofitable -polly-acc-mincompute=0 \
-; RUN: -polly-codegen-ppcg -polly-acc-codegen-managed-memory \
-; RUN: -polly-acc-rewrite-managed-memory  -polly-acc-rewrite-allocas < %s | FileCheck %s --check-prefix=HOST-IR
-
-; REQUIRES: pollyacc
-
-; SCOP:      Function: f
-; SCOP-NEXT: Region: %for.body---%for.end
-; SCOP-NEXT: Max Loop Depth:  1
-; SCOP: i32 MemRef_arr[*];
-
-; Check that we generate a constructor call for @A.toptr
-; HOST-IR-NOT:   %arr = alloca [100 x i32]
-
-source_filename = "test.c"
-target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
-target triple = "x86_64-apple-macosx10.12.0"
-
-
-define void @f() {
-entry:
-  %arr = alloca [100 x i32]
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  br label %for.body
-
-for.body:                                         ; preds = %entry.split, %for.body
-  %indvars.iv1 = phi i64 [ 0, %entry.split ], [ %indvars.iv.next, %for.body ]
-  %arrayidx = getelementptr inbounds [100 x i32], ptr %arr, i64 0, i64 %indvars.iv1
-  store i32 42, ptr %arrayidx, align 4, !tbaa !3
-  %indvars.iv.next = add nuw nsw i64 %indvars.iv1, 1
-  %exitcond = icmp eq i64 %indvars.iv.next, 100
-  br i1 %exitcond, label %for.end, label %for.body
-
-for.end:                                          ; preds = %for.body
-  ret void
-}
-
-; Function Attrs: argmemonly nounwind
-declare void @llvm.lifetime.start.p0(i64, ptr nocapture) #0
-
-
-; Function Attrs: argmemonly nounwind
-declare void @llvm.lifetime.end.p0(i64, ptr nocapture) #0
-
-attributes #0 = { argmemonly nounwind }
-
-!llvm.module.flags = !{!0, !1}
-!llvm.ident = !{!2}
-
-!0 = !{i32 1, !"wchar_size", i32 4}
-!1 = !{i32 7, !"PIC Level", i32 2}
-!2 = !{!"clang version 6.0.0"}
-!3 = !{!4, !4, i64 0}
-!4 = !{!"int", !5, i64 0}
-!5 = !{!"omnipotent char", !6, i64 0}
-!6 = !{!"Simple C/C++ TBAA"}
diff --git a/polly/test/GPGPU/managed-memory-rewrite-malloc-free-inside-constexpr.ll b/polly/test/GPGPU/managed-memory-rewrite-malloc-free-inside-constexpr.ll
deleted file mode 100644
index 946da40919ec8..0000000000000
--- a/polly/test/GPGPU/managed-memory-rewrite-malloc-free-inside-constexpr.ll
+++ /dev/null
@@ -1,93 +0,0 @@
-; RUN: opt %loadPolly -polly-print-scops -disable-output < %s | FileCheck %s --check-prefix=SCOP
-
-; RUN: opt %loadPolly -polly-codegen-ppcg \
-; RUN: -S -polly-acc-codegen-managed-memory \
-; RUN: -polly-acc-rewrite-managed-memory < %s | FileCheck %s --check-prefix=HOST-IR
-;
-; REQUIRES: pollyacc
-;
-; Check that we can correctly rewrite `malloc` to `polly_mallocManaged`, and
-; `free` to `polly_freeManaged` with the `polly-acc-rewrite-managed-memory`
-; pass, even inside `constantExpr`. This is necessary because a cookie cutter
-; Inst->replaceUsesOfWith(...) call does not actually work, because this does
-; not replace the instruction within a ConstantExpr.
-;
-; #include <memory.h>
-;
-; static const int N = 100;
-; int* f(int *ToFree) {
-;     free(ToFree);
-;     int *A = (int *)malloc(sizeof(int) * N);
-;     for(int i = 0; i < N; i++) {
-;         A[i] = 42;
-;     }
-;     return A;
-;
-; }
-
-; SCOP:      Function: f
-; SCOP-NEXT: Region: %for.body---%for.end
-; SCOP-NEXT: Max Loop Depth:  1
-
-; SCOP:      Arrays {
-; SCOP-NEXT:     i32 MemRef_tmp[*]; // Element size 4
-; SCOP-NEXT: }
-
-; // Check that polly_mallocManaged is declared and used correctly.
-; HOST-IR: declare ptr @polly_mallocManaged(i64)
-
-; // Check that polly_freeManaged is declared and used correctly.
-; HOST-IR  call void @polly_freeManaged(i8* %toFree)
-; HOST-IR: declare void @polly_freeManaged(ptr)
-
-; // Check that we remove the original malloc,free
-; HOST-IR-NOT: declare ptr @malloc(i64)
-; HOST-IR-NOT: declare void @free(ptr)
-
-target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
-target triple = "x86_64-apple-macosx10.12.0"
-
-define ptr @f(ptr %toFree) {
-entry:
-  ; Free inside bitcast
-  call void @free (ptr %toFree)
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  ; malloc inside bitcast.
-  %tmp = call ptr @malloc (i64 400)
-  br label %for.body
-
-for.body:                                         ; preds = %entry.split, %for.body
-  %indvars.iv1 = phi i64 [ 0, %entry.split ], [ %indvars.iv.next, %for.body ]
-  %arrayidx = getelementptr inbounds i32, ptr %tmp, i64 %indvars.iv1
-  store i32 42, ptr %arrayidx, align 4, !tbaa !3
-  %indvars.iv.next = add nuw nsw i64 %indvars.iv1, 1
-  %exitcond = icmp eq i64 %indvars.iv.next, 100
-  br i1 %exitcond, label %for.end, label %for.body
-
-for.end:                                          ; preds = %for.body
-  ret ptr %tmp
-}
-
-; Function Attrs: argmemonly nounwind
-declare void @llvm.lifetime.start.p0(i64, ptr nocapture) #0
-
-declare ptr @malloc(i64)
-declare void @free(ptr)
-
-; Function Attrs: argmemonly nounwind
-declare void @llvm.lifetime.end.p0(i64, ptr nocapture) #0
-
-attributes #0 = { argmemonly nounwind }
-
-!llvm.module.flags = !{!0, !1}
-!llvm.ident = !{!2}
-
-!0 = !{i32 1, !"wchar_size", i32 4}
-!1 = !{i32 7, !"PIC Level", i32 2}
-!2 = !{!"clang version 6.0.0"}
-!3 = !{!4, !4, i64 0}
-!4 = !{!"int", !5, i64 0}
-!5 = !{!"omnipotent char", !6, i64 0}
-!6 = !{!"Simple C/C++ TBAA"}
diff --git a/polly/test/GPGPU/managed-memory-rewrite-malloc-free.ll b/polly/test/GPGPU/managed-memory-rewrite-malloc-free.ll
deleted file mode 100644
index 8e456127b1274..0000000000000
--- a/polly/test/GPGPU/managed-memory-rewrite-malloc-free.ll
+++ /dev/null
@@ -1,91 +0,0 @@
-; RUN: opt %loadPolly -polly-print-scops -disable-output < %s | FileCheck %s --check-prefix=SCOP
-
-; RUN: opt %loadPolly -polly-codegen-ppcg \
-; RUN: -S -polly-acc-codegen-managed-memory \
-; RUN: -polly-acc-rewrite-managed-memory < %s | FileCheck %s --check-prefix=HOST-IR
-;
-; REQUIRES: pollyacc
-;
-; Check that we can correctly rewrite `malloc` to `polly_mallocManaged`, and
-; `free` to `polly_freeManaged` with the `polly-acc-rewrite-managed-memory`
-; pass.
-;
-; #include <memory.h>
-;
-; static const int N = 100;
-; int* f(int *ToFree) {
-;     free(ToFree);
-;     int *A = (int *)malloc(sizeof(int) * N);
-;     for(int i = 0; i < N; i++) {
-;         A[i] = 42;
-;     }
-;     return A;
-;
-; }
-
-; SCOP:      Function: f
-; SCOP-NEXT: Region: %for.body---%for.end
-; SCOP-NEXT: Max Loop Depth:  1
-
-; SCOP:      Arrays {
-; SCOP-NEXT:     i32 MemRef_call[*]; // Element size 4
-; SCOP-NEXT: }
-
-; // Check that polly_mallocManaged is declared and used correctly.
-; HOST-IR: %call = tail call ptr @polly_mallocManaged(i64 400)
-; HOST-IR: declare ptr @polly_mallocManaged(i64)
-
-; // Check that polly_freeManaged is declared and used correctly.
-; HOST-IR  %toFreeBitcast = bitcast i32* %toFree to i8*
-; HOST-IR  call void @polly_freeManaged(i8* %toFreeBitcast)
-; HOST-IR: declare void @polly_freeManaged(ptr)
-
-; // Check that we remove the original malloc,free
-; HOST-IR-NOT: declare ptr @malloc(i64)
-; HOST-IR-NOT: declare void @free(ptr)
-
-target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
-target triple = "x86_64-apple-macosx10.12.0"
-
-define ptr @f(ptr %toFree) {
-entry:
-  call void @free(ptr %toFree)
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  %call = tail call ptr @malloc(i64 400)
-  br label %for.body
-
-for.body:                                         ; preds = %entry.split, %for.body
-  %indvars.iv1 = phi i64 [ 0, %entry.split ], [ %indvars.iv.next, %for.body ]
-  %arrayidx = getelementptr inbounds i32, ptr %call, i64 %indvars.iv1
-  store i32 42, ptr %arrayidx, align 4, !tbaa !3
-  %indvars.iv.next = add nuw nsw i64 %indvars.iv1, 1
-  %exitcond = icmp eq i64 %indvars.iv.next, 100
-  br i1 %exitcond, label %for.end, label %for.body
-
-for.end:                                          ; preds = %for.body
-  ret ptr %call
-}
-
-; Function Attrs: argmemonly nounwind
-declare void @llvm.lifetime.start.p0(i64, ptr nocapture) #0
-
-declare ptr @malloc(i64)
-declare void @free(ptr)
-
-; Function Attrs: argmemonly nounwind
-declare void @llvm.lifetime.end.p0(i64, ptr nocapture) #0
-
-attributes #0 = { argmemonly nounwind }
-
-!llvm.module.flags = !{!0, !1}
-!llvm.ident = !{!2}
-
-!0 = !{i32 1, !"wchar_size", i32 4}
-!1 = !{i32 7, !"PIC Level", i32 2}
-!2 = !{!"clang version 6.0.0"}
-!3 = !{!4, !4, i64 0}
-!4 = !{!"int", !5, i64 0}
-!5 = !{!"omnipotent char", !6, i64 0}
-!6 = !{!"Simple C/C++ TBAA"}
diff --git a/polly/test/GPGPU/memory-only-referenced-from-access.ll b/polly/test/GPGPU/memory-only-referenced-from-access.ll
deleted file mode 100644
index b3828950324a6..0000000000000
--- a/polly/test/GPGPU/memory-only-referenced-from-access.ll
+++ /dev/null
@@ -1,44 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-kernel-ir \
-; RUN: -polly-invariant-load-hoisting -polly-ignore-aliasing \
-; RUN: -polly-process-unprofitable -polly-ignore-parameter-bounds \
-; RUN: -polly-acc-fail-on-verify-module-failure \
-; RUN: -polly-acc-codegen-managed-memory \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck %s
-
-; REQUIRES: pollyacc
-
-; Verify that we correctly generate a kernel even if certain invariant load
-; hoisted parameters appear only in memory accesses, but not domain elements.
-
-; CHECK: @FUNC_quux_SCOP_0_KERNEL_0(ptr addrspace(1) %MemRef_tmp4, i32 %tmp3, i32 %tmp, i32 %tmp31, i32 %tmp2)
-
-target datalayout = "e-p:64:64:64-S128-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f16:16:16-f32:32:32-f64:64:64-f128:128:128-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
-target triple = "x86_64-unknown-linux-gnu"
-
-%struct.hoge = type { ptr, i64, i64, [1 x %struct.widget] }
-%struct.widget = type { i64, i64, i64 }
-
-@global = external unnamed_addr global %struct.hoge, align 32
-
-define void @quux(ptr noalias %arg, ptr noalias %arg1) {
-bb:
-  %tmp = load i32, ptr %arg, align 4
-  %tmp2 = sext i32 %tmp to i64
-  %tmp3 = load i32, ptr %arg1, align 4
-  %tmp4 = load ptr, ptr @global, align 32
-  br label %bb5
-
-bb5:                                              ; preds = %bb5, %bb
-  %tmp6 = phi i32 [ %tmp11, %bb5 ], [ 0, %bb ]
-  %tmp7 = sext i32 %tmp6 to i64
-  %tmp8 = sub nsw i64 %tmp7, %tmp2
-  %tmp9 = getelementptr [0 x double], ptr %tmp4, i64 0, i64 %tmp8
-  store double undef, ptr %tmp9, align 8
-  %tmp10 = icmp eq i32 %tmp6, %tmp3
-  %tmp11 = add i32 %tmp6, 1
-  br i1 %tmp10, label %bb12, label %bb5
-
-bb12:                                             ; preds = %bb5
-  ret void
-}
diff --git a/polly/test/GPGPU/mostly-sequential.ll b/polly/test/GPGPU/mostly-sequential.ll
deleted file mode 100644
index c42c24482a38d..0000000000000
--- a/polly/test/GPGPU/mostly-sequential.ll
+++ /dev/null
@@ -1,105 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-
-; REQUIRES: pollyacc
-
-;    void foo(float A[]) {
-;      for (long i = 0; i < 128; i++)
-;        A[i] += i;
-;
-;      for (long i = 0; i < 128; i++)
-;        for (long j = 0; j < 128; j++)
-;          A[42] += i + j;
-;    }
-
-; CODE:        cudaCheckReturn(cudaMemcpy(dev_MemRef_A, MemRef_A, (128) * sizeof(float), cudaMemcpyHostToDevice));
-; CODE-NEXT:   {
-; CODE-NEXT:     dim3 k0_dimBlock(32);
-; CODE-NEXT:     dim3 k0_dimGrid(4);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_A);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:            {
-; CODE-NEXT:         dim3 k1_dimBlock;
-; CODE-NEXT:         dim3 k1_dimGrid;
-; CODE-NEXT:         kernel1 <<<k1_dimGrid, k1_dimBlock>>> (dev_MemRef_A);
-; CODE-NEXT:         cudaCheckKernel();
-; CODE-NEXT:       }
-
-; CODE:   cudaCheckReturn(cudaMemcpy(MemRef_A, dev_MemRef_A, (128) * sizeof(float), cudaMemcpyDeviceToHost));
-; CODE-NEXT: cudaCheckReturn(cudaFree(dev_MemRef_A));
-; CODE-NEXT: }
-
-; CODE: # kernel0
-; CODE-NEXT: Stmt_bb4(32 * b0 + t0);
-
-; CODE: # kernel1
-; CODE-NEXT: for (int c0 = 0; c0 <= 127; c0 += 1)
-; CODE-NEXT:   for (int c1 = 0; c1 <= 127; c1 += 1)
-; CODE-NEXT:     Stmt_bb14(c0, c1);
-
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @foo(ptr %A) {
-bb:
-  br label %bb3
-
-bb3:                                              ; preds = %bb8, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp9, %bb8 ]
-  %exitcond2 = icmp ne i64 %i.0, 128
-  br i1 %exitcond2, label %bb4, label %bb10
-
-bb4:                                              ; preds = %bb3
-  %tmp = sitofp i64 %i.0 to float
-  %tmp5 = getelementptr inbounds float, ptr %A, i64 %i.0
-  %tmp6 = load float, ptr %tmp5, align 4
-  %tmp7 = fadd float %tmp6, %tmp
-  store float %tmp7, ptr %tmp5, align 4
-  br label %bb8
-
-bb8:                                              ; preds = %bb4
-  %tmp9 = add nuw nsw i64 %i.0, 1
-  br label %bb3
-
-bb10:                                             ; preds = %bb3
-  br label %bb11
-
-bb11:                                             ; preds = %bb23, %bb10
-  %i1.0 = phi i64 [ 0, %bb10 ], [ %tmp24, %bb23 ]
-  %exitcond1 = icmp ne i64 %i1.0, 128
-  br i1 %exitcond1, label %bb12, label %bb25
-
-bb12:                                             ; preds = %bb11
-  br label %bb13
-
-bb13:                                             ; preds = %bb20, %bb12
-  %j.0 = phi i64 [ 0, %bb12 ], [ %tmp21, %bb20 ]
-  %exitcond = icmp ne i64 %j.0, 128
-  br i1 %exitcond, label %bb14, label %bb22
-
-bb14:                                             ; preds = %bb13
-  %tmp15 = add nuw nsw i64 %i1.0, %j.0
-  %tmp16 = sitofp i64 %tmp15 to float
-  %tmp17 = getelementptr inbounds float, ptr %A, i64 42
-  %tmp18 = load float, ptr %tmp17, align 4
-  %tmp19 = fadd float %tmp18, %tmp16
-  store float %tmp19, ptr %tmp17, align 4
-  br label %bb20
-
-bb20:                                             ; preds = %bb14
-  %tmp21 = add nuw nsw i64 %j.0, 1
-  br label %bb13
-
-bb22:                                             ; preds = %bb13
-  br label %bb23
-
-bb23:                                             ; preds = %bb22
-  %tmp24 = add nuw nsw i64 %i1.0, 1
-  br label %bb11
-
-bb25:                                             ; preds = %bb11
-  ret void
-}
diff --git a/polly/test/GPGPU/non-read-only-scalars.ll b/polly/test/GPGPU/non-read-only-scalars.ll
deleted file mode 100644
index 1ce6e0991ebbc..0000000000000
--- a/polly/test/GPGPU/non-read-only-scalars.ll
+++ /dev/null
@@ -1,168 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-kernel-ir \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck %s -check-prefix=KERNEL-IR
-;
-; REQUIRES: pollyacc
-;
-; #include <stdio.h>
-;
-; float foo(float A[]) {
-;   float sum = 0;
-;
-;   for (long i = 0; i < 32; i++)
-;     A[i] = i;
-;
-;   for (long i = 0; i < 32; i++)
-;     A[i] += i;
-;
-;   for (long i = 0; i < 32; i++)
-;     sum += A[i];
-;
-;   return sum;
-; }
-;
-; int main() {
-;   float A[32];
-;   float sum = foo(A);
-;   printf("%f\n", sum);
-; }
-
-; CODE:          dim3 k0_dimBlock(32);
-; CODE-NEXT:     dim3 k0_dimGrid(1);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_A);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:   {
-; CODE-NEXT:     dim3 k1_dimBlock;
-; CODE-NEXT:     dim3 k1_dimGrid;
-; CODE-NEXT:     kernel1 <<<k1_dimGrid, k1_dimBlock>>> (dev_MemRef_sum_0__phi);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:          {
-; CODE-NEXT:       dim3 k2_dimBlock;
-; CODE-NEXT:       dim3 k2_dimGrid;
-; CODE-NEXT:       kernel2 <<<k2_dimGrid, k2_dimBlock>>> (dev_MemRef_A, dev_MemRef_sum_0__phi, dev_MemRef_sum_0);
-; CODE-NEXT:       cudaCheckKernel();
-; CODE-NEXT:     }
-
-; CODE:        cudaCheckReturn(cudaMemcpy(MemRef_A, dev_MemRef_A, (32) * sizeof(float), cudaMemcpyDeviceToHost));
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(&MemRef_sum_0, dev_MemRef_sum_0, sizeof(float), cudaMemcpyDeviceToHost));
-; CODE-NEXT:   cudaCheckReturn(cudaFree(dev_MemRef_A));
-; CODE-NEXT:   cudaCheckReturn(cudaFree(dev_MemRef_sum_0__phi));
-; CODE-NEXT:   cudaCheckReturn(cudaFree(dev_MemRef_sum_0));
-; CODE-NEXT: }
-
-; CODE: # kernel0
-; CODE-NEXT: {
-; CODE-NEXT:   Stmt_bb4(t0);
-; CODE-NEXT:   Stmt_bb10(t0);
-; CODE-NEXT: }
-
-; CODE: # kernel1
-; CODE-NEXT: Stmt_bb17();
-
-; CODE: # kernel2
-; TODO-NEXT: {
-; TODO-NEXT:   read();
-; TODO-NEXT:   for (int c0 = 0; c0 <= 32; c0 += 1) {
-; TODO-NEXT:     Stmt_bb18(c0);
-; TODO-NEXT:     if (c0 <= 31)
-; TODO-NEXT:       Stmt_bb20(c0);
-; TODO-NEXT:   }
-; TODO-NEXT:   write();
-; TODO-NEXT: }
-
-
-; KERNEL-IR: define ptx_kernel void @FUNC_foo_SCOP_0_KERNEL_1(ptr addrspace(1) %MemRef_sum_0__phi)
-; KERNEL-IR:  store float 0.000000e+00, ptr %sum.0.phiops
-; KERNEL-IR:  [[REGA:%.+]] = addrspacecast ptr addrspace(1) %MemRef_sum_0__phi to ptr
-; KERNEL-IR:  [[REGB:%.+]] = load float, ptr %sum.0.phiops
-; KERNEL-IR:  store float [[REGB]], ptr [[REGA]]
-
-; KERNEL-IR: define ptx_kernel void @FUNC_foo_SCOP_0_KERNEL_2(ptr addrspace(1) %MemRef_A, ptr addrspace(1) %MemRef_sum_0__phi, ptr addrspace(1) %MemRef_sum_0)
-
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-@.str = private unnamed_addr constant [4 x i8] c"%f\0A\00", align 1
-
-define float @foo(ptr %A) {
-bb:
-  br label %bb3
-
-bb3:                                              ; preds = %bb6, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp7, %bb6 ]
-  %exitcond2 = icmp ne i64 %i.0, 32
-  br i1 %exitcond2, label %bb4, label %bb8
-
-bb4:                                              ; preds = %bb3
-  %tmp = sitofp i64 %i.0 to float
-  %tmp5 = getelementptr inbounds float, ptr %A, i64 %i.0
-  store float %tmp, ptr %tmp5, align 4
-  br label %bb6
-
-bb6:                                              ; preds = %bb4
-  %tmp7 = add nuw nsw i64 %i.0, 1
-  br label %bb3
-
-bb8:                                              ; preds = %bb3
-  br label %bb9
-
-bb9:                                              ; preds = %bb15, %bb8
-  %i1.0 = phi i64 [ 0, %bb8 ], [ %tmp16, %bb15 ]
-  %exitcond1 = icmp ne i64 %i1.0, 32
-  br i1 %exitcond1, label %bb10, label %bb17
-
-bb10:                                             ; preds = %bb9
-  %tmp11 = sitofp i64 %i1.0 to float
-  %tmp12 = getelementptr inbounds float, ptr %A, i64 %i1.0
-  %tmp13 = load float, ptr %tmp12, align 4
-  %tmp14 = fadd float %tmp13, %tmp11
-  store float %tmp14, ptr %tmp12, align 4
-  br label %bb15
-
-bb15:                                             ; preds = %bb10
-  %tmp16 = add nuw nsw i64 %i1.0, 1
-  br label %bb9
-
-bb17:                                             ; preds = %bb9
-  br label %bb18
-
-bb18:                                             ; preds = %bb20, %bb17
-  %sum.0 = phi float [ 0.000000e+00, %bb17 ], [ %tmp23, %bb20 ]
-  %i2.0 = phi i64 [ 0, %bb17 ], [ %tmp24, %bb20 ]
-  %exitcond = icmp ne i64 %i2.0, 32
-  br i1 %exitcond, label %bb19, label %bb25
-
-bb19:                                             ; preds = %bb18
-  br label %bb20
-
-bb20:                                             ; preds = %bb19
-  %tmp21 = getelementptr inbounds float, ptr %A, i64 %i2.0
-  %tmp22 = load float, ptr %tmp21, align 4
-  %tmp23 = fadd float %sum.0, %tmp22
-  %tmp24 = add nuw nsw i64 %i2.0, 1
-  br label %bb18
-
-bb25:                                             ; preds = %bb18
-  %sum.0.lcssa = phi float [ %sum.0, %bb18 ]
-  ret float %sum.0.lcssa
-}
-
-define i32 @main() {
-bb:
-  %A = alloca [32 x float], align 16
-  %tmp1 = call float @foo(ptr %A)
-  %tmp2 = fpext float %tmp1 to double
-  %tmp3 = call i32 (ptr, ...) @printf(ptr @.str, double %tmp2) #2
-  ret i32 0
-}
-
-declare i32 @printf(ptr, ...) #1
-
diff --git a/polly/test/GPGPU/non-zero-array-offset.ll b/polly/test/GPGPU/non-zero-array-offset.ll
deleted file mode 100644
index f18f6828a47fe..0000000000000
--- a/polly/test/GPGPU/non-zero-array-offset.ll
+++ /dev/null
@@ -1,116 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-
-; RUN: opt %loadPolly -polly-codegen-ppcg -S < %s | \
-; RUN: FileCheck %s -check-prefix=IR
-;
-; REQUIRES: pollyacc
-
-; CODE:      cudaCheckReturn(cudaMemcpy(dev_MemRef_B, MemRef_B, (16) * sizeof(float), cudaMemcpyHostToDevice));
-; CODE-NEXT: cudaCheckReturn(cudaMemcpy(dev_MemRef_A, MemRef_A, (8) * sizeof(float), cudaMemcpyHostToDevice));
-
-; CODE:          dim3 k0_dimBlock(8);
-; CODE-NEXT:     dim3 k0_dimGrid(1);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_B);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:        {
-; CODE-NEXT:     dim3 k1_dimBlock(8);
-; CODE-NEXT:     dim3 k1_dimGrid(1);
-; CODE-NEXT:     kernel1 <<<k1_dimGrid, k1_dimBlock>>> (dev_MemRef_A);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:   cudaCheckReturn(cudaMemcpy(MemRef_B, dev_MemRef_B, (16) * sizeof(float), cudaMemcpyDeviceToHost));
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(MemRef_A, dev_MemRef_A, (8) * sizeof(float), cudaMemcpyDeviceToHost));
-
-; CODE: # kernel0
-; CODE-NEXT: Stmt_bb3(t0);
-
-; CODE: # kernel1
-; CODE-NEXT: Stmt_bb11(t0);
-
-; IR:       %p_dev_array_MemRef_B = call ptr @polly_allocateMemoryForDevice(i64 32)
-; IR-NEXT:  %p_dev_array_MemRef_A = call ptr @polly_allocateMemoryForDevice(i64 32)
-; IR-NEXT:  [[REG0:%.+]] = getelementptr float, ptr %B, i64 8
-; IR-NEXT:  call void @polly_copyFromHostToDevice(ptr [[REG0]], ptr %p_dev_array_MemRef_B, i64 32)
-
-; IR:      [[REGA:%.+]] = call ptr @polly_getDevicePtr(ptr %p_dev_array_MemRef_B)
-; IR-NEXT: [[REGC:%.+]]  = getelementptr float, ptr [[REGA]], i64 -8
-
-;    void foo(float A[], float B[]) {
-;      for (long i = 0; i < 8; i++)
-;        B[i + 8] *= 4;
-;
-;      for (long i = 0; i < 8; i++)
-;        A[i] *= 12;
-;    }
-;
-;    #ifdef OUTPUT
-;    int main() {
-;      float A[16];
-;
-;      for (long i = 0; i < 16; i++) {
-;        __sync_synchronize();
-;        A[i] = i;
-;      }
-;
-;      foo(A, A);
-;
-;      float sum = 0;
-;      for (long i = 0; i < 16; i++) {
-;        __sync_synchronize();
-;        sum += A[i];
-;      }
-;
-;      printf("%f\n", sum);
-;    }
-;    #endif
-;
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @foo(ptr %A, ptr %B) {
-bb:
-  br label %bb2
-
-bb2:                                              ; preds = %bb7, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp8, %bb7 ]
-  %exitcond1 = icmp ne i64 %i.0, 8
-  br i1 %exitcond1, label %bb3, label %bb9
-
-bb3:                                              ; preds = %bb2
-  %tmp = add nuw nsw i64 %i.0, 8
-  %tmp4 = getelementptr inbounds float, ptr %B, i64 %tmp
-  %tmp5 = load float, ptr %tmp4, align 4
-  %tmp6 = fmul float %tmp5, 4.000000e+00
-  store float %tmp6, ptr %tmp4, align 4
-  br label %bb7
-
-bb7:                                              ; preds = %bb3
-  %tmp8 = add nuw nsw i64 %i.0, 1
-  br label %bb2
-
-bb9:                                              ; preds = %bb2
-  br label %bb10
-
-bb10:                                             ; preds = %bb15, %bb9
-  %i1.0 = phi i64 [ 0, %bb9 ], [ %tmp16, %bb15 ]
-  %exitcond = icmp ne i64 %i1.0, 8
-  br i1 %exitcond, label %bb11, label %bb17
-
-bb11:                                             ; preds = %bb10
-  %tmp12 = getelementptr inbounds float, ptr %A, i64 %i1.0
-  %tmp13 = load float, ptr %tmp12, align 4
-  %tmp14 = fmul float %tmp13, 1.200000e+01
-  store float %tmp14, ptr %tmp12, align 4
-  br label %bb15
-
-bb15:                                             ; preds = %bb11
-  %tmp16 = add nuw nsw i64 %i1.0, 1
-  br label %bb10
-
-bb17:                                             ; preds = %bb10
-  ret void
-}
diff --git a/polly/test/GPGPU/only-part-of-array-modified.ll b/polly/test/GPGPU/only-part-of-array-modified.ll
deleted file mode 100644
index abc380badfb6d..0000000000000
--- a/polly/test/GPGPU/only-part-of-array-modified.ll
+++ /dev/null
@@ -1,40 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-;
-; REQUIRES: pollyacc
-;
-;    void foo(float A[], float B[]) {
-;      for (long i = 0; i < 1024; i++)
-;        A[2 * i] = B[i];
-;    }
-
-; CODE: cudaCheckReturn(cudaMemcpy(dev_MemRef_B, MemRef_B, (1024) * sizeof(i32), cudaMemcpyHostToDevice));
-; CODE-NEXT: cudaCheckReturn(cudaMemcpy(dev_MemRef_A, MemRef_A, (2047) * sizeof(i32), cudaMemcpyHostToDevice));
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @foo(ptr %A, ptr %B) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb8, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp9, %bb8 ]
-  %exitcond = icmp ne i64 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb10
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds float, ptr %B, i64 %i.0
-  %tmp4 = load i32, ptr %tmp, align 4
-  %tmp5 = shl nsw i64 %i.0, 1
-  %tmp6 = getelementptr inbounds float, ptr %A, i64 %tmp5
-  store i32 %tmp4, ptr %tmp6, align 4
-  br label %bb8
-
-bb8:                                              ; preds = %bb2
-  %tmp9 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb10:                                             ; preds = %bb1
-  ret void
-}
diff --git a/polly/test/GPGPU/parametric-loop-bound.ll b/polly/test/GPGPU/parametric-loop-bound.ll
deleted file mode 100644
index e436bd663a4a4..0000000000000
--- a/polly/test/GPGPU/parametric-loop-bound.ll
+++ /dev/null
@@ -1,62 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-
-; RUN: opt %loadPolly -polly-codegen-ppcg \
-; RUN: -S < %s | \
-; RUN: FileCheck -check-prefix=IR %s
-
-; REQUIRES: pollyacc
-
-;    void foo(long A[], long n) {
-;      for (long i = 0; i < n; i++)
-;        A[i] += 100;
-;    }
-
-; CODE: if (n >= 1) {
-; CODE:        cudaCheckReturn(cudaMemcpy(dev_MemRef_A, MemRef_A, (n) * sizeof(i64), cudaMemcpyHostToDevice));
-; CODE-NEXT:   {
-; CODE-NEXT:     dim3 k0_dimBlock(32);
-; CODE-NEXT:     dim3 k0_dimGrid(n >= 1048545 ? 32768 : (n + 31) / 32);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_A, n);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:        cudaCheckReturn(cudaMemcpy(MemRef_A, dev_MemRef_A, (n) * sizeof(i64), cudaMemcpyDeviceToHost));
-; CODE-NEXT:   cudaCheckReturn(cudaFree(dev_MemRef_A));
-; CODE-NEXT: }
-
-; CODE: # kernel0
-; CODE-NEXT: for (int c0 = 0; c0 <= (n - 32 * b0 - 1) / 1048576; c0 += 1)
-; CODE-NEXT:   if (n >= 32 * b0 + t0 + 1048576 * c0 + 1)
-; CODE-NEXT:     Stmt_bb2(32 * b0 + t0 + 1048576 * c0);
-
-; IR: store i64 %n, ptr %polly_launch_0_param_1
-; IR-NEXT: [[REGA:%.+]] = getelementptr [2 x ptr], ptr %polly_launch_0_params, i64 0, i64 1
-; IR-NEXT: store ptr %polly_launch_0_param_1, ptr [[REGA]]
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @foo(ptr %A, i64 %n) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb6, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp7, %bb6 ]
-  %tmp = icmp slt i64 %i.0, %n
-  br i1 %tmp, label %bb2, label %bb8
-
-bb2:                                              ; preds = %bb1
-  %tmp3 = getelementptr inbounds i64, ptr %A, i64 %i.0
-  %tmp4 = load i64, ptr %tmp3, align 8
-  %tmp5 = add nsw i64 %tmp4, 100
-  store i64 %tmp5, ptr %tmp3, align 8
-  br label %bb6
-
-bb6:                                              ; preds = %bb2
-  %tmp7 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb8:                                              ; preds = %bb1
-  ret void
-}
diff --git a/polly/test/GPGPU/partial_writes.ll b/polly/test/GPGPU/partial_writes.ll
deleted file mode 100644
index c3df624df7ac1..0000000000000
--- a/polly/test/GPGPU/partial_writes.ll
+++ /dev/null
@@ -1,49 +0,0 @@
-; RUN: opt %loadPolly -polly-import-jscop -polly-codegen-ppcg -polly-stmt-granularity=bb -S < %s \
-; RUN: | FileCheck %s
-
-; REQUIRES: pollyacc
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-target triple = "x86_64-unknown-linux-gnu"
-
-; CHECK: polly_launchKernel
-
-; Function Attrs: nounwind uwtable
-define void @partial_writes() {
-bb:
-  %tmp = tail call ptr @wibble() #2
-  br label %bb2
-
-bb2:                                              ; preds = %bb11, %bb
-  %tmp3 = phi i64 [ 0, %bb ], [ %tmp12, %bb11 ]
-  %tmp4 = getelementptr inbounds [1200 x double], ptr %tmp, i64 0, i64 %tmp3
-  %tmp5 = load double, ptr %tmp4, align 8, !tbaa !1
-  br label %bb6
-
-bb6:                                              ; preds = %bb6, %bb2
-  %tmp7 = phi double [ undef, %bb2 ], [ undef, %bb6 ]
-  %tmp8 = phi i64 [ 0, %bb2 ], [ %tmp9, %bb6 ]
-  store double undef, ptr %tmp4, align 8, !tbaa !1
-  %tmp9 = add nuw nsw i64 %tmp8, 1
-  %tmp10 = icmp eq i64 %tmp9, 900
-  br i1 %tmp10, label %bb11, label %bb6
-
-bb11:                                             ; preds = %bb6
-  %tmp12 = add nuw nsw i64 %tmp3, 1
-  %tmp13 = icmp eq i64 %tmp12, 1200
-  br i1 %tmp13, label %bb14, label %bb2
-
-bb14:                                             ; preds = %bb11
-  ret void
-}
-
-declare ptr @wibble()
-
-
-!llvm.ident = !{!0}
-
-!0 = !{!"clang version 6.0.0 (trunk 309912) (llvm/trunk 309933)"}
-!1 = !{!2, !2, i64 0}
-!2 = !{!"double", !3, i64 0}
-!3 = !{!"omnipotent char", !4, i64 0}
-!4 = !{!"Simple C/C++ TBAA"}
diff --git a/polly/test/GPGPU/partial_writes___%bb2---%bb14.jscop b/polly/test/GPGPU/partial_writes___%bb2---%bb14.jscop
deleted file mode 100644
index d5b537ee1f059..0000000000000
--- a/polly/test/GPGPU/partial_writes___%bb2---%bb14.jscop
+++ /dev/null
@@ -1,47 +0,0 @@
-{
-   "arrays" : [
-      {
-         "name" : "MemRef_tmp",
-         "sizes" : [ "*" ],
-         "type" : "double"
-      }
-   ],
-   "context" : "{  :  }",
-   "name" : "%bb2---%bb14",
-   "statements" : [
-      {
-         "accesses" : [
-            {
-               "kind" : "read",
-               "relation" : "{ Stmt_bb2[i0] -> MemRef_tmp[i0] }"
-            },
-            {
-               "kind" : "write",
-               "relation" : "{ Stmt_bb2[i0] -> MemRef_tmp[i0] }"
-            }
-         ],
-         "domain" : "{ Stmt_bb2[i0] : 0 <= i0 <= 1199 }",
-         "name" : "Stmt_bb2",
-         "schedule" : "{ Stmt_bb2[i0] -> [i0, 0, 0] }"
-      },
-      {
-         "accesses" : [
-            {
-               "kind" : "write",
-               "relation" : "{ Stmt_bb6[i0, i1] -> MemRef_tmp[i0] : i1 <= 898 }"
-            },
-            {
-               "kind" : "read",
-               "relation" : "{ Stmt_bb6[i0, i1] -> MemRef_tmp[i0] }"
-            },
-            {
-               "kind" : "write",
-               "relation" : "{ Stmt_bb6[i0, i1] -> MemRef_tmp[i0] }"
-            }
-         ],
-         "domain" : "{ Stmt_bb6[i0, i1] : 0 <= i0 <= 1199 and 0 <= i1 <= 899 }",
-         "name" : "Stmt_bb6",
-         "schedule" : "{ Stmt_bb6[i0, i1] -> [i0, 1, i1] }"
-      }
-   ]
-}
diff --git a/polly/test/GPGPU/phi-nodes-in-kernel.ll b/polly/test/GPGPU/phi-nodes-in-kernel.ll
deleted file mode 100644
index acb1f2c4e0e25..0000000000000
--- a/polly/test/GPGPU/phi-nodes-in-kernel.ll
+++ /dev/null
@@ -1,86 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen-ppcg -S < %s | \
-; RUN: FileCheck %s -check-prefix=IR
-
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen-ppcg -polly-acc-dump-kernel-ir \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck %s -check-prefix=KERNEL-IR
-
-; REQUIRES: pollyacc
-
-; Approximate C source:
-; void kernel_dynprog(int c[50]) {
-;     int iter = 0;
-;     int outl = 0;
-;
-;      while(1) {
-;         for(int indvar = 1 ; indvar <= 49; indvar++) {
-;             c[indvar] = undef;
-;         }
-;         add78 = c[49] + outl;
-;         inc80 = iter + 1;
-;
-;         if (true) break;
-;
-;         outl = add78;
-;         iter = inc80;
-;      }
-;}
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-target triple = "x86_64-unknown-linux-gnu"
-
-; CODE:       cudaCheckReturn(cudaMalloc((void **) &dev_MemRef_c, (50) * sizeof(i32)));
-
-; CODE:       {
-; CODE-NEXT:    dim3 k0_dimBlock(32);
-; CODE-NEXT:    dim3 k0_dimGrid(2);
-; CODE-NEXT:    kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_c);
-; CODE-NEXT:    cudaCheckKernel();
-; CODE-NEXT:  }
-
-; CODE:       cudaCheckReturn(cudaMemcpy(MemRef_c, dev_MemRef_c, (50) * sizeof(i32), cudaMemcpyDeviceToHost));
-; CODE-NEXT:  cudaCheckReturn(cudaFree(dev_MemRef_c));
-
-; CODE: # kernel0
-; CODE-NEXT: if (32 * b0 + t0 <= 48)
-; CODE-NEXT:     Stmt_for_body17(0, 32 * b0 + t0);
-
-; IR-LABEL: call void @polly_freeKernel
-; IR:       [[REGC:%.+]] =   bitcast i32* %{{[0-9]+}} to i8*
-; IR-NEXT:  call void @polly_copyFromDeviceToHost(i8* %p_dev_array_MemRef_c, i8* [[REGC]], i64 196)
-
-; KERNEL-IR: define ptx_kernel void @FUNC_kernel_dynprog_SCOP_0_KERNEL_0(i8 addrspace(1)* %MemRef_c) #0 {
-; KERNEL-IR: %polly.access.MemRef_c = getelementptr i32, i32 addrspace(1)* %polly.access.cast.MemRef_c, i64 %9
-; KERNEL-IR-NEXT: store i32 422, i32 addrspace(1)* %polly.access.MemRef_c, align 4
-
-define void @kernel_dynprog([50 x i32]* %c) {
-entry:
-  %arrayidx77 = getelementptr inbounds [50 x i32], [50 x i32]* %c, i64 0, i64 49
-  br label %for.cond1.preheader
-
-for.cond1.preheader:                              ; preds = %for.cond15.for.cond12.loopexit_crit_edge, %entry
-  %out_l.055 = phi i32 [ 0, %entry ], [ %add78, %for.cond15.for.cond12.loopexit_crit_edge ]
-  %iter.054 = phi i32 [ 0, %entry ], [ %inc80, %for.cond15.for.cond12.loopexit_crit_edge ]
-  br label %for.body17
-
-for.cond15.for.cond12.loopexit_crit_edge:         ; preds = %for.body17
-  %tmp = load i32, i32* %arrayidx77, align 4
-  %add78 = add nsw i32 %tmp, %out_l.055
-  %inc80 = add nuw nsw i32 %iter.054, 1
-  br i1 false, label %for.cond1.preheader, label %for.end81
-
-for.body17:                                       ; preds = %for.body17, %for.cond1.preheader
-  %indvars.iv71 = phi i64 [ 1, %for.cond1.preheader ], [ %indvars.iv.next72, %for.body17 ]
-  %arrayidx69 = getelementptr inbounds [50 x i32], [50 x i32]* %c, i64 0, i64 %indvars.iv71
-  store i32 422, i32* %arrayidx69, align 4
-  %indvars.iv.next72 = add nuw nsw i64 %indvars.iv71, 1
-  %lftr.wideiv74 = trunc i64 %indvars.iv.next72 to i32
-  %exitcond75 = icmp ne i32 %lftr.wideiv74, 50
-  br i1 %exitcond75, label %for.body17, label %for.cond15.for.cond12.loopexit_crit_edge
-
-for.end81:                                        ; preds = %for.cond15.for.cond12.loopexit_crit_edge
-  ret void
-}
diff --git a/polly/test/GPGPU/private-memory.ll b/polly/test/GPGPU/private-memory.ll
deleted file mode 100644
index d4ba9fa19b395..0000000000000
--- a/polly/test/GPGPU/private-memory.ll
+++ /dev/null
@@ -1,82 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -polly-acc-use-private \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen-ppcg \
-; RUN: -polly-acc-use-private \
-; RUN: -disable-output -polly-acc-dump-kernel-ir < %s | \
-; RUN: FileCheck -check-prefix=KERNEL %s
-
-; REQUIRES: pollyacc
-
-;    void add(float *A) {
-;      for (long i = 0; i < 32; i++)
-;        for (long j = 0; j < 10; j++)
-;          A[i] += 1;
-;    }
-
-; CODE: # kernel0
-; CODE: {
-; CODE:     read(t0);
-; CODE:     for (int c3 = 0; c3 <= 9; c3 += 1)
-; CODE:       Stmt_bb5(t0, c3);
-; CODE:     write(t0);
-; CODE: }
-
-; KERNEL: %private_array = alloca [1 x float]
-
-; KERNEL:   %polly.access.cast.private_array = bitcast [1 x float]* %private_array to float*
-; KERNEL-NEXT:   %polly.access.private_array = getelementptr float, float* %polly.access.cast.private_array, i64 0
-; KERNEL-NEXT:   %polly.access.cast.MemRef_A = bitcast i8 addrspace(1)* %MemRef_A to float addrspace(1)*
-; KERNEL-NEXT:   %polly.access.MemRef_A = getelementptr float, float addrspace(1)* %polly.access.cast.MemRef_A, i64 %t0
-; KERNEL-NEXT:   %shared.read = load float, float addrspace(1)* %polly.access.MemRef_A
-; KERNEL-NEXT:   store float %shared.read, float* %polly.access.private_array
-
-; KERNEL:   %polly.access.cast.private_array5 = bitcast [1 x float]* %private_array to float*
-; KERNEL-NEXT:   %polly.access.private_array6 = getelementptr float, float* %polly.access.cast.private_array5, i64 0
-; KERNEL-NEXT:   %polly.access.cast.MemRef_A7 = bitcast i8 addrspace(1)* %MemRef_A to float addrspace(1)*
-; KERNEL-NEXT:   %polly.access.MemRef_A8 = getelementptr float, float addrspace(1)* %polly.access.cast.MemRef_A7, i64 %t0
-; KERNEL-NEXT:   %shared.write = load float, float* %polly.access.private_array6
-; KERNEL-NEXT:   store float %shared.write, float addrspace(1)* %polly.access.MemRef_A8
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @add(float* %A) {
-bb:
-  br label %bb2
-
-bb2:                                              ; preds = %bb11, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp12, %bb11 ]
-  %exitcond1 = icmp ne i64 %i.0, 32
-  br i1 %exitcond1, label %bb3, label %bb13
-
-bb3:                                              ; preds = %bb2
-  br label %bb4
-
-bb4:                                              ; preds = %bb8, %bb3
-  %j.0 = phi i64 [ 0, %bb3 ], [ %tmp9, %bb8 ]
-  %exitcond = icmp ne i64 %j.0, 10
-  br i1 %exitcond, label %bb5, label %bb10
-
-bb5:                                              ; preds = %bb4
-  %tmp = getelementptr inbounds float, float* %A, i64 %i.0
-  %tmp6 = load float, float* %tmp, align 4
-  %tmp7 = fadd float %tmp6, 1.000000e+00
-  store float %tmp7, float* %tmp, align 4
-  br label %bb8
-
-bb8:                                              ; preds = %bb5
-  %tmp9 = add nuw nsw i64 %j.0, 1
-  br label %bb4
-
-bb10:                                             ; preds = %bb4
-  br label %bb11
-
-bb11:                                             ; preds = %bb10
-  %tmp12 = add nuw nsw i64 %i.0, 1
-  br label %bb2
-
-bb13:                                             ; preds = %bb2
-  ret void
-}
diff --git a/polly/test/GPGPU/privatization-simple.ll b/polly/test/GPGPU/privatization-simple.ll
deleted file mode 100644
index c715b8e77b677..0000000000000
--- a/polly/test/GPGPU/privatization-simple.ll
+++ /dev/null
@@ -1,58 +0,0 @@
-; RUN: opt %loadPolly -polly-print-scops -disable-output < %s | FileCheck %s -check-prefix=SCOP
-; RUN: opt %loadPolly -S -polly-codegen-ppcg < %s | FileCheck %s -check-prefix=HOST-IR
-
-; REQUIRES: pollyacc
-
-; SCOP:      Function: f
-; SCOP-NEXT: Region: %for.body---%for.end
-; SCOP-NEXT: Max Loop Depth:  1
-
-; Check that kernel launch is generated in host IR.
-; the declare would not be generated unless a call to a kernel exists.
-; HOST-IR: declare void @polly_launchKernel(ptr, i32, i32, i32, i32, i32, ptr)
-
-; void f(int A[], int B[], int control, int C[]) {
-;     int x;
-; #pragma scop
-;     for(int i = 0; i < 1000; i ++) {
-;         x = 0;
-;         if(control) x = C[i];
-;         B[i] = x * A[i];
-;
-;     }
-; #pragma endscop
-; }
-
-target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @f(ptr %A, ptr %B, i32 %control, ptr %C) {
-entry:
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  br label %for.body
-
-for.body:                                         ; preds = %entry.split, %if.end
-  %indvars.iv = phi i64 [ 0, %entry.split ], [ %indvars.iv.next, %if.end ]
-  %tobool = icmp eq i32 %control, 0
-  br i1 %tobool, label %if.end, label %if.then
-
-if.then:                                          ; preds = %for.body
-  %arrayidx = getelementptr inbounds i32, ptr %C, i64 %indvars.iv
-  %tmp4 = load i32, ptr %arrayidx, align 4
-  br label %if.end
-
-if.end:                                           ; preds = %for.body, %if.then
-  %x.0 = phi i32 [ %tmp4, %if.then ], [ 0, %for.body ]
-  %arrayidx2 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
-  %tmp8 = load i32, ptr %arrayidx2, align 4
-  %mul = mul nsw i32 %tmp8, %x.0
-  %arrayidx4 = getelementptr inbounds i32, ptr %B, i64 %indvars.iv
-  store i32 %mul, ptr %arrayidx4, align 4
-  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
-  %exitcond = icmp ne i64 %indvars.iv.next, 1000
-  br i1 %exitcond, label %for.body, label %for.end
-
-for.end:                                          ; preds = %if.end
-  ret void
-}
diff --git a/polly/test/GPGPU/privatization.ll b/polly/test/GPGPU/privatization.ll
deleted file mode 100644
index fbb291575146c..0000000000000
--- a/polly/test/GPGPU/privatization.ll
+++ /dev/null
@@ -1,62 +0,0 @@
-; RUN: opt %loadPolly -polly-print-scops -disable-output < %s | FileCheck %s -check-prefix=SCOP
-; RUN: opt %loadPolly -S -polly-codegen-ppcg < %s | FileCheck %s -check-prefix=HOST-IR
-
-; REQUIRES: pollyacc
-
-; SCOP:      Function: checkPrivatization
-; SCOP-NEXT: Region: %for.body---%for.end
-; SCOP-NEXT: Max Loop Depth:  1
-
-
-; Check that kernel launch is generated in host IR.
-; the declare would not be generated unless a call to a kernel exists.
-; HOST-IR: declare void @polly_launchKernel(ptr, i32, i32, i32, i32, i32, ptr)
-
-;
-;
-;    void checkPrivatization(int A[], int B[], int C[], int control) {
-;      int x;
-;    #pragma scop
-;      for (int i = 0; i < 1000; i++) {
-;        x = 0;
-;        if (control)
-;          x += C[i];
-;
-;        B[i] = x * A[i];
-;      }
-;    #pragma endscop
-;    }
-;
-target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @checkPrivatization(ptr %A, ptr %B, ptr %C, i32 %control) {
-entry:
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  br label %for.body
-
-for.body:                                         ; preds = %entry.split, %if.end
-  %indvars.iv = phi i64 [ 0, %entry.split ], [ %indvars.iv.next, %if.end ]
-  %tobool = icmp eq i32 %control, 0
-  br i1 %tobool, label %if.end, label %if.then
-
-if.then:                                          ; preds = %for.body
-  %arrayidx = getelementptr inbounds i32, ptr %C, i64 %indvars.iv
-  %tmp4 = load i32, ptr %arrayidx, align 4
-  br label %if.end
-
-if.end:                                           ; preds = %for.body, %if.then
-  %x.0 = phi i32 [ %tmp4, %if.then ], [ 0, %for.body ]
-  %arrayidx2 = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
-  %tmp9 = load i32, ptr %arrayidx2, align 4
-  %mul = mul nsw i32 %tmp9, %x.0
-  %arrayidx4 = getelementptr inbounds i32, ptr %B, i64 %indvars.iv
-  store i32 %mul, ptr %arrayidx4, align 4
-  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
-  %exitcond = icmp ne i64 %indvars.iv.next, 1000
-  br i1 %exitcond, label %for.body, label %for.end
-
-for.end:                                          ; preds = %if.end
-  ret void
-}
diff --git a/polly/test/GPGPU/region-stmt.ll b/polly/test/GPGPU/region-stmt.ll
deleted file mode 100644
index 8e392fb300626..0000000000000
--- a/polly/test/GPGPU/region-stmt.ll
+++ /dev/null
@@ -1,81 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-
-; RUN: opt %loadPolly -polly-codegen-ppcg -S < %s | \
-; RUN: FileCheck %s -check-prefix=IR
-
-; CODE:        cudaCheckReturn(cudaMemcpy(dev_MemRef_A, MemRef_A, (128) * sizeof(float), cudaMemcpyHostToDevice));
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(dev_MemRef_B, MemRef_B, (128) * sizeof(float), cudaMemcpyHostToDevice));
-; CODE-NEXT:   {
-; CODE-NEXT:     dim3 k0_dimBlock(32);
-; CODE-NEXT:     dim3 k0_dimGrid(4);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_A, dev_MemRef_B);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:   cudaCheckReturn(cudaMemcpy(MemRef_B, dev_MemRef_B, (128) * sizeof(float), cudaMemcpyDeviceToHost));
-
-; CODE: # kernel0
-; CODE-NEXT: Stmt_for_body__TO__if_end(32 * b0 + t0);
-
-; IR: @polly_initContext
-
-; KERNEL-IR: kernel_0
-
-; REQUIRES: pollyacc
-
-;    void foo(float A[], float B[]) {
-;      for (long i = 0; i < 128; i++)
-;        if (A[i] == 42)
-;          B[i] += 2 * i;
-;        else
-;          B[i] += 4 * i;
-;    }
-;
-source_filename = "/tmp/test.c"
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @foo(ptr %A, ptr %B) {
-entry:
-  br label %for.cond
-
-for.cond:                                         ; preds = %for.inc, %entry
-  %i.0 = phi i64 [ 0, %entry ], [ %inc, %for.inc ]
-  %exitcond = icmp ne i64 %i.0, 128
-  br i1 %exitcond, label %for.body, label %for.end
-
-for.body:                                         ; preds = %for.cond
-  %arrayidx = getelementptr inbounds float, ptr %A, i64 %i.0
-  %tmp = load float, ptr %arrayidx, align 4
-  %cmp1 = fcmp oeq float %tmp, 4.200000e+01
-  br i1 %cmp1, label %if.then, label %if.else
-
-if.then:                                          ; preds = %for.body
-  %mul = shl nsw i64 %i.0, 1
-  %conv = sitofp i64 %mul to float
-  %arrayidx2 = getelementptr inbounds float, ptr %B, i64 %i.0
-  %tmp1 = load float, ptr %arrayidx2, align 4
-  %add = fadd float %tmp1, %conv
-  store float %add, ptr %arrayidx2, align 4
-  br label %if.end
-
-if.else:                                          ; preds = %for.body
-  %mul3 = shl nsw i64 %i.0, 2
-  %conv4 = sitofp i64 %mul3 to float
-  %arrayidx5 = getelementptr inbounds float, ptr %B, i64 %i.0
-  %tmp2 = load float, ptr %arrayidx5, align 4
-  %add6 = fadd float %tmp2, %conv4
-  store float %add6, ptr %arrayidx5, align 4
-  br label %if.end
-
-if.end:                                           ; preds = %if.else, %if.then
-  br label %for.inc
-
-for.inc:                                          ; preds = %if.end
-  %inc = add nuw nsw i64 %i.0, 1
-  br label %for.cond
-
-for.end:                                          ; preds = %for.cond
-  ret void
-}
diff --git a/polly/test/GPGPU/remove-dead-instructions-in-stmt-2.ll b/polly/test/GPGPU/remove-dead-instructions-in-stmt-2.ll
deleted file mode 100644
index 326236cf92fde..0000000000000
--- a/polly/test/GPGPU/remove-dead-instructions-in-stmt-2.ll
+++ /dev/null
@@ -1,39 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-kernel-ir \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck %s -check-prefix=KERNEL-IR
-
-; REQUIRES: pollyacc
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-target triple = "x86_64-unknown-linux-gnu"
-
-; KERNEL-IR: store i32 0, ptr addrspace(1) %polly.access.MemRef_sum_c, align 4
-; KERNEL-IR-NEXT: br label %polly.merge
-
-define void @kernel_dynprog(ptr %sum_c) {
-entry:
-  br label %for.cond1.preheader
-
-for.cond1.preheader:                              ; preds = %entry
-  br label %for.body3
-
-for.cond1.loopexit:                               ; preds = %for.end
-  %indvars.iv.next49 = add nuw nsw i64 %indvars.iv48, 1
-  %exitcond57 = icmp ne i64 %indvars.iv.next56, 49
-  br i1 %exitcond57, label %for.body3, label %for.inc55
-
-for.body3:                                        ; preds = %for.cond1.loopexit, %for.cond1.preheader
-  %indvars.iv55 = phi i64 [ 0, %for.cond1.preheader ], [ %indvars.iv.next56, %for.cond1.loopexit ]
-  %indvars.iv48 = phi i64 [ 1, %for.cond1.preheader ], [ %indvars.iv.next49, %for.cond1.loopexit ]
-  %indvars.iv.next56 = add nuw nsw i64 %indvars.iv55, 1
-  %arrayidx10 = getelementptr inbounds [50 x [50 x i32]], ptr %sum_c, i64 %indvars.iv55, i64 %indvars.iv48, i64 %indvars.iv55
-  store i32 0, ptr %arrayidx10, align 4
-  %cmp1334 = icmp slt i64 %indvars.iv.next56, %indvars.iv48
-  br label %for.end
-
-for.end:                                          ; preds = %for.body3
-  br label %for.cond1.loopexit
-
-for.inc55:                                        ; preds = %for.cond1.loopexit
-  ret void
-}
diff --git a/polly/test/GPGPU/remove-dead-instructions-in-stmt.ll b/polly/test/GPGPU/remove-dead-instructions-in-stmt.ll
deleted file mode 100644
index 2024f006c53af..0000000000000
--- a/polly/test/GPGPU/remove-dead-instructions-in-stmt.ll
+++ /dev/null
@@ -1,62 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-kernel-ir \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck %s -check-prefix=KERNEL-IR
-
-; REQUIRES: pollyacc
-
-; Ensure that no dead instructions are emitted between the store and the
-; branch instruction of the ScopStmt. At some point, our dead-code-elimination
-; did not remove code that was inserted to compute the old (unused) branch
-; condition. This code referred to CPU registers and consequently resulted
-; in invalid bitcode.
-
-; KERNEL-IR: store i32 0, ptr addrspace(1) %polly.access.MemRef_sum_c, align 4
-; KERNEL-IR-NEXT: br label %polly.merge
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-target triple = "x86_64-unknown-linux-gnu"
-
-define void @kernel_dynprog(ptr %sum_c) {
-entry:
-  br label %for.cond1.preheader
-
-for.cond1.preheader:                              ; preds = %entry
-  br label %for.body3
-
-for.cond4.for.cond1.loopexit_crit_edge:           ; preds = %for.end
-  br label %for.cond1.loopexit
-
-for.cond1.loopexit:                               ; preds = %for.cond4.for.cond1.loopexit_crit_edge
-  br i1 undef, label %for.body3, label %for.inc55
-
-for.body3:                                        ; preds = %for.cond1.loopexit, %for.cond1.preheader
-  %indvars.iv55 = phi i64 [ 0, %for.cond1.preheader ], [ %indvars.iv.next56, %for.cond1.loopexit ]
-  %indvars.iv.next56 = add nuw nsw i64 %indvars.iv55, 1
-  br label %for.body6
-
-for.body6:                                        ; preds = %for.end, %for.body3
-  %indvars.iv50 = phi i64 [ 0, %for.body3 ], [ %indvars.iv.next51, %for.end ]
-  %arrayidx10 = getelementptr inbounds [50 x [50 x i32]], ptr %sum_c, i64 %indvars.iv55, i64 %indvars.iv50, i64 %indvars.iv55
-  store i32 0, ptr %arrayidx10, align 4
-  %cmp1334 = icmp slt i64 %indvars.iv.next56, %indvars.iv50
-  br i1 %cmp1334, label %for.body14.lr.ph, label %for.end
-
-for.body14.lr.ph:                                 ; preds = %for.body6
-  br label %for.body14
-
-for.body14:                                       ; preds = %for.body14, %for.body14.lr.ph
-  %arrayidx32 = getelementptr inbounds [50 x [50 x i32]], ptr %sum_c, i64 %indvars.iv55, i64 %indvars.iv50, i64 0
-  br i1 false, label %for.body14, label %for.cond12.for.end_crit_edge
-
-for.cond12.for.end_crit_edge:                     ; preds = %for.body14
-  br label %for.end
-
-for.end:                                          ; preds = %for.cond12.for.end_crit_edge, %for.body6
-  %indvars.iv.next51 = add nuw nsw i64 %indvars.iv50, 1
-  %lftr.wideiv53 = trunc i64 %indvars.iv.next51 to i32
-  %exitcond54 = icmp ne i32 %lftr.wideiv53, 50
-  br i1 %exitcond54, label %for.body6, label %for.cond4.for.cond1.loopexit_crit_edge
-
-for.inc55:                                        ; preds = %for.cond1.loopexit
-  unreachable
-}
diff --git a/polly/test/GPGPU/run-time-check.ll b/polly/test/GPGPU/run-time-check.ll
deleted file mode 100644
index 3b04c3e015935..0000000000000
--- a/polly/test/GPGPU/run-time-check.ll
+++ /dev/null
@@ -1,58 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -S < %s | \
-; RUN: FileCheck %s -check-prefix=IR
-;
-; REQUIRES: pollyacc
-;
-;    void foo(long n, float A[][32]) {
-;      for (long i = 0; i < n; i++)
-;        for (long j = 0; j < n; j++)
-;          A[i][j] += A[i + 1][j + 1];
-;    }
-
-; IR:       %tmp = icmp slt i64 %i.0, %n
-; IR-NEXT:  br i1 %tmp, label %bb2, label %polly.merge_new_and_old
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @foo(i64 %n, ptr %A) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb15, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp16, %bb15 ]
-  %tmp = icmp slt i64 %i.0, %n
-  br i1 %tmp, label %bb2, label %bb17
-
-bb2:                                              ; preds = %bb1
-  br label %bb3
-
-bb3:                                              ; preds = %bb12, %bb2
-  %j.0 = phi i64 [ 0, %bb2 ], [ %tmp13, %bb12 ]
-  %exitcond = icmp ne i64 %j.0, %n
-  br i1 %exitcond, label %bb4, label %bb14
-
-bb4:                                              ; preds = %bb3
-  %tmp5 = add nuw nsw i64 %j.0, 1
-  %tmp6 = add nuw nsw i64 %i.0, 1
-  %tmp7 = getelementptr inbounds [32 x float], ptr %A, i64 %tmp6, i64 %tmp5
-  %tmp8 = load float, ptr %tmp7, align 4
-  %tmp9 = getelementptr inbounds [32 x float], ptr %A, i64 %i.0, i64 %j.0
-  %tmp10 = load float, ptr %tmp9, align 4
-  %tmp11 = fadd float %tmp10, %tmp8
-  store float %tmp11, ptr %tmp9, align 4
-  br label %bb12
-
-bb12:                                             ; preds = %bb4
-  %tmp13 = add nuw nsw i64 %j.0, 1
-  br label %bb3
-
-bb14:                                             ; preds = %bb3
-  br label %bb15
-
-bb15:                                             ; preds = %bb14
-  %tmp16 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb17:                                             ; preds = %bb1
-  ret void
-}
diff --git a/polly/test/GPGPU/scalar-param-and-value-32-bit.ll b/polly/test/GPGPU/scalar-param-and-value-32-bit.ll
deleted file mode 100644
index 0313d64e976cd..0000000000000
--- a/polly/test/GPGPU/scalar-param-and-value-32-bit.ll
+++ /dev/null
@@ -1,41 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-kernel-ir \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck %s
-
-; REQUIRES: pollyacc, target=nvptx{{.*}}
-;
-;    void foo(float A[], int n) {
-;      for (long j = 0; j < n; j++)
-;        A[j + n] += 42;
-;    }
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-; CHECK: define ptx_kernel void @FUNC_foo_SCOP_0_KERNEL_0(ptr addrspace(1) %MemRef_A, i32 %n)
-
-define void @foo(ptr %A, i32 %n) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb9, %bb
-  %j.0 = phi i64 [ 0, %bb ], [ %tmp10, %bb9 ]
-  %tmp = sext i32 %n to i64
-  %tmp2 = icmp slt i64 %j.0, %tmp
-  br i1 %tmp2, label %bb3, label %bb11
-
-bb3:                                              ; preds = %bb1
-  %tmp4 = sext i32 %n to i64
-  %tmp5 = add nsw i64 %j.0, %tmp4
-  %tmp6 = getelementptr inbounds float, ptr %A, i64 %tmp5
-  %tmp7 = load float, ptr %tmp6, align 4
-  %tmp8 = fadd float %tmp7, 4.200000e+01
-  store float %tmp8, ptr %tmp6, align 4
-  br label %bb9
-
-bb9:                                              ; preds = %bb3
-  %tmp10 = add nuw nsw i64 %j.0, 1
-  br label %bb1
-
-bb11:                                             ; preds = %bb1
-  ret void
-}
diff --git a/polly/test/GPGPU/scalar-param-and-value-use.ll b/polly/test/GPGPU/scalar-param-and-value-use.ll
deleted file mode 100644
index 0301d88e16ac3..0000000000000
--- a/polly/test/GPGPU/scalar-param-and-value-use.ll
+++ /dev/null
@@ -1,67 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-kernel-ir \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=IR %s
-
-; REQUIRES: pollyacc, target=nvptx{{.*}}
-
-;    void foo(long n, float A[][n]) {
-;      for (long i = 0; i < 32; i++)
-;        for (long j = 0; j < 32; j++)
-;          A[i][j] += A[i + 1][j + 1];
-;    }
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-; This test case failed at some point as %n was only available in this kernel
-; when referenced through an isl_id in an isl ast expression, but not when
-; it was referenced from a SCEV  or instruction that not part of any loop
-; bound.
-
-; IR: %polly.access.mul.MemRef_A = mul nsw i64 {{.*}}, %n
-
-define void @foo(i64 %n, ptr %A) {
-bb:
-  br label %bb2
-
-bb2:                                              ; preds = %bb19, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp20, %bb19 ]
-  %exitcond1 = icmp ne i64 %i.0, 32
-  br i1 %exitcond1, label %bb3, label %bb21
-
-bb3:                                              ; preds = %bb2
-  br label %bb4
-
-bb4:                                              ; preds = %bb16, %bb3
-  %j.0 = phi i64 [ 0, %bb3 ], [ %tmp17, %bb16 ]
-  %exitcond = icmp ne i64 %j.0, 32
-  br i1 %exitcond, label %bb5, label %bb18
-
-bb5:                                              ; preds = %bb4
-  %tmp = add nuw nsw i64 %j.0, 1
-  %tmp6 = add nuw nsw i64 %i.0, 1
-  %tmp7 = mul nsw i64 %tmp6, %n
-  %tmp8 = getelementptr inbounds float, ptr %A, i64 %tmp7
-  %tmp9 = getelementptr inbounds float, ptr %tmp8, i64 %tmp
-  %tmp10 = load float, ptr %tmp9, align 4
-  %tmp11 = mul nsw i64 %i.0, %n
-  %tmp12 = getelementptr inbounds float, ptr %A, i64 %tmp11
-  %tmp13 = getelementptr inbounds float, ptr %tmp12, i64 %j.0
-  %tmp14 = load float, ptr %tmp13, align 4
-  %tmp15 = fadd float %tmp14, %tmp10
-  store float %tmp15, ptr %tmp13, align 4
-  br label %bb16
-
-bb16:                                             ; preds = %bb5
-  %tmp17 = add nuw nsw i64 %j.0, 1
-  br label %bb4
-
-bb18:                                             ; preds = %bb4
-  br label %bb19
-
-bb19:                                             ; preds = %bb18
-  %tmp20 = add nuw nsw i64 %i.0, 1
-  br label %bb2
-
-bb21:                                             ; preds = %bb2
-  ret void
-}
diff --git a/polly/test/GPGPU/scalar-parameter-fp128.ll b/polly/test/GPGPU/scalar-parameter-fp128.ll
deleted file mode 100644
index f20a809c7c83b..0000000000000
--- a/polly/test/GPGPU/scalar-parameter-fp128.ll
+++ /dev/null
@@ -1,39 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code -disable-output %s
-
-; XFAIL: *
-
-; REQUIRES: pollyacc, target=nvptx{{.*}}
-
-; This fails today with "LowerFormalArguments didn't emit the correct number of values!"
-
-;    void foo(fp128 A[], fp128 b) {
-;      for (long i = 0; i < 1024; i++)
-;        A[i] += b;
-;    }
-;
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @fp128(ptr %A, fp128 %b) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb5, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp6, %bb5 ]
-  %exitcond = icmp ne i64 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb7
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds fp128, ptr %A, i64 %i.0
-  %tmp3 = load fp128, ptr %tmp, align 4
-  %tmp4 = fadd fp128 %tmp3, %b
-  store fp128 %tmp4, ptr %tmp, align 4
-  br label %bb5
-
-bb5:                                              ; preds = %bb2
-  %tmp6 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb7:                                              ; preds = %bb1
-  ret void
-}
-
diff --git a/polly/test/GPGPU/scalar-parameter-half.ll b/polly/test/GPGPU/scalar-parameter-half.ll
deleted file mode 100644
index 1270962568124..0000000000000
--- a/polly/test/GPGPU/scalar-parameter-half.ll
+++ /dev/null
@@ -1,35 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code -disable-output %s
-
-; REQUIRES: pollyacc, target=nvptx{{.*}}
-
-;    void foo(half A[], half b) {
-;      for (long i = 0; i < 1024; i++)
-;        A[i] += b;
-;    }
-;
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @half(ptr %A, half %b) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb5, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp6, %bb5 ]
-  %exitcond = icmp ne i64 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb7
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds half, ptr %A, i64 %i.0
-  %tmp3 = load half, ptr %tmp, align 4
-  %tmp4 = fadd half %tmp3, %b
-  store half %tmp4, ptr %tmp, align 4
-  br label %bb5
-
-bb5:                                              ; preds = %bb2
-  %tmp6 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb7:                                              ; preds = %bb1
-  ret void
-}
-
diff --git a/polly/test/GPGPU/scalar-parameter-i120.ll b/polly/test/GPGPU/scalar-parameter-i120.ll
deleted file mode 100644
index 06fb46dd917ed..0000000000000
--- a/polly/test/GPGPU/scalar-parameter-i120.ll
+++ /dev/null
@@ -1,39 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code -disable-output %s
-
-; XFAIL: *
-
-; REQUIRES: pollyacc, target=nvptx{{.*}}
-
-; This fails today with "Promotion is not suitable for scalars of size larger than 64-bits"
-
-;    void foo(i120 A[], i120 b) {
-;      for (long i = 0; i < 1024; i++)
-;        A[i] += b;
-;    }
-;
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @i120(ptr %A, i120 %b) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb5, %bb
-  %i.0 = phi i120 [ 0, %bb ], [ %tmp6, %bb5 ]
-  %exitcond = icmp ne i120 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb7
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds i120, ptr %A, i120 %i.0
-  %tmp3 = load i120, ptr %tmp, align 4
-  %tmp4 = add i120 %tmp3, %b
-  store i120 %tmp4, ptr %tmp, align 4
-  br label %bb5
-
-bb5:                                              ; preds = %bb2
-  %tmp6 = add nuw nsw i120 %i.0, 1
-  br label %bb1
-
-bb7:                                              ; preds = %bb1
-  ret void
-}
-
diff --git a/polly/test/GPGPU/scalar-parameter-i128.ll b/polly/test/GPGPU/scalar-parameter-i128.ll
deleted file mode 100644
index 8e54cf4636d4e..0000000000000
--- a/polly/test/GPGPU/scalar-parameter-i128.ll
+++ /dev/null
@@ -1,34 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code -disable-output %s
-
-; REQUIRES: pollyacc, target=nvptx{{.*}}
-
-;    void foo(i128 A[], i128 b) {
-;      for (long i = 0; i < 1024; i++)
-;        A[i] += b;
-;    }
-;
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @i128(ptr %A, i128 %b) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb5, %bb
-  %i.0 = phi i128 [ 0, %bb ], [ %tmp6, %bb5 ]
-  %exitcond = icmp ne i128 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb7
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds i128, ptr %A, i128 %i.0
-  %tmp3 = load i128, ptr %tmp, align 4
-  %tmp4 = add i128 %tmp3, %b
-  store i128 %tmp4, ptr %tmp, align 4
-  br label %bb5
-
-bb5:                                              ; preds = %bb2
-  %tmp6 = add nuw nsw i128 %i.0, 1
-  br label %bb1
-
-bb7:                                              ; preds = %bb1
-  ret void
-}
diff --git a/polly/test/GPGPU/scalar-parameter-i3000.ll b/polly/test/GPGPU/scalar-parameter-i3000.ll
deleted file mode 100644
index 5c36b3fd62cb8..0000000000000
--- a/polly/test/GPGPU/scalar-parameter-i3000.ll
+++ /dev/null
@@ -1,38 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code -disable-output %s
-
-; XFAIL: *
-
-; REQUIRES: pollyacc, target=nvptx{{.*}}
-
-; This fails today with "Promotion is not suitable for scalars of size larger than 64-bits"
-
-;    void foo(i3000 A[], i3000 b) {
-;      for (long i = 0; i < 1024; i++)
-;        A[i] += b;
-;    }
-;
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @i3000(ptr %A, i3000 %b) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb5, %bb
-  %i.0 = phi i3000 [ 0, %bb ], [ %tmp6, %bb5 ]
-  %exitcond = icmp ne i3000 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb7
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds i3000, ptr %A, i3000 %i.0
-  %tmp3 = load i3000, ptr %tmp, align 4
-  %tmp4 = add i3000 %tmp3, %b
-  store i3000 %tmp4, ptr %tmp, align 4
-  br label %bb5
-
-bb5:                                              ; preds = %bb2
-  %tmp6 = add nuw nsw i3000 %i.0, 1
-  br label %bb1
-
-bb7:                                              ; preds = %bb1
-  ret void
-}
diff --git a/polly/test/GPGPU/scalar-parameter-i80.ll b/polly/test/GPGPU/scalar-parameter-i80.ll
deleted file mode 100644
index a672cd5c1cdc5..0000000000000
--- a/polly/test/GPGPU/scalar-parameter-i80.ll
+++ /dev/null
@@ -1,39 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code -disable-output %s
-
-; XFAIL: *
-
-; REQUIRES: pollyacc, target=nvptx{{.*}}
-
-; This fails today with "Promotion is not suitable for scalars of size larger than 64-bits"
-
-;    void foo(i80 A[], i80 b) {
-;      for (long i = 0; i < 1024; i++)
-;        A[i] += b;
-;    }
-;
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @i80(ptr %A, i80 %b) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb5, %bb
-  %i.0 = phi i80 [ 0, %bb ], [ %tmp6, %bb5 ]
-  %exitcond = icmp ne i80 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb7
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds i80, ptr %A, i80 %i.0
-  %tmp3 = load i80, ptr %tmp, align 4
-  %tmp4 = add i80 %tmp3, %b
-  store i80 %tmp4, ptr %tmp, align 4
-  br label %bb5
-
-bb5:                                              ; preds = %bb2
-  %tmp6 = add nuw nsw i80 %i.0, 1
-  br label %bb1
-
-bb7:                                              ; preds = %bb1
-  ret void
-}
-
diff --git a/polly/test/GPGPU/scalar-parameter-ppc_fp128.ll b/polly/test/GPGPU/scalar-parameter-ppc_fp128.ll
deleted file mode 100644
index 11dfd68ede9b3..0000000000000
--- a/polly/test/GPGPU/scalar-parameter-ppc_fp128.ll
+++ /dev/null
@@ -1,38 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code -disable-output %s
-
-; XFAIL: *
-
-; REQUIRES: pollyacc, target=nvptx{{.*}}
-
-; This fails today with "LowerFormalArguments didn't emit the correct number of values!"
-
-;    void foo(fp128 A[], fp128 b) {
-;      for (long i = 0; i < 1024; i++)
-;        A[i] += b;
-;    }
-;
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @ppc_fp128(ptr %A, ppc_fp128 %b) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb5, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp6, %bb5 ]
-  %exitcond = icmp ne i64 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb7
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds ppc_fp128, ptr %A, i64 %i.0
-  %tmp3 = load ppc_fp128, ptr %tmp, align 4
-  %tmp4 = fadd ppc_fp128 %tmp3, %b
-  store ppc_fp128 %tmp4, ptr %tmp, align 4
-  br label %bb5
-
-bb5:                                              ; preds = %bb2
-  %tmp6 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb7:                                              ; preds = %bb1
-  ret void
-}
diff --git a/polly/test/GPGPU/scalar-parameter-x86_fp80.ll b/polly/test/GPGPU/scalar-parameter-x86_fp80.ll
deleted file mode 100644
index f20a809c7c83b..0000000000000
--- a/polly/test/GPGPU/scalar-parameter-x86_fp80.ll
+++ /dev/null
@@ -1,39 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code -disable-output %s
-
-; XFAIL: *
-
-; REQUIRES: pollyacc, target=nvptx{{.*}}
-
-; This fails today with "LowerFormalArguments didn't emit the correct number of values!"
-
-;    void foo(fp128 A[], fp128 b) {
-;      for (long i = 0; i < 1024; i++)
-;        A[i] += b;
-;    }
-;
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @fp128(ptr %A, fp128 %b) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb5, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp6, %bb5 ]
-  %exitcond = icmp ne i64 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb7
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds fp128, ptr %A, i64 %i.0
-  %tmp3 = load fp128, ptr %tmp, align 4
-  %tmp4 = fadd fp128 %tmp3, %b
-  store fp128 %tmp4, ptr %tmp, align 4
-  br label %bb5
-
-bb5:                                              ; preds = %bb2
-  %tmp6 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb7:                                              ; preds = %bb1
-  ret void
-}
-
diff --git a/polly/test/GPGPU/scalar-parameter.ll b/polly/test/GPGPU/scalar-parameter.ll
deleted file mode 100644
index e416c93211d56..0000000000000
--- a/polly/test/GPGPU/scalar-parameter.ll
+++ /dev/null
@@ -1,411 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-
-; RUN: opt %loadPolly -polly-codegen-ppcg \
-; RUN: -S < %s | \
-; RUN: FileCheck -check-prefix=IR %s
-
-; RUN: opt %loadPolly -polly-codegen-ppcg \
-; RUN: -disable-output -polly-acc-dump-kernel-ir < %s | \
-; RUN: FileCheck -check-prefix=KERNEL %s
-
-; XFAIL: *
-
-; REQUIRES: pollyacc, target=nvptx{{.*}}
-
-; This fails today due to extensive output differences from when the test was written.
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-; KERNEL: define ptx_kernel void @kernel_0(ptr %MemRef_A, float %MemRef_b)
-
-; CODE: Code
-; CODE-NEXT: ====
-; CODE-NEXT: # host
-; CODE-NEXT: {
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(dev_MemRef_A, MemRef_A, (1024) * sizeof(float), cudaMemcpyHostToDevice));
-; CODE-NEXT:   {
-; CODE-NEXT:     dim3 k0_dimBlock(32);
-; CODE-NEXT:     dim3 k0_dimGrid(32);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_A, MemRef_b);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:   cudaCheckReturn(cudaMemcpy(MemRef_A, dev_MemRef_A, (1024) * sizeof(float), cudaMemcpyDeviceToHost));
-; CODE-NEXT: }
-
-; CODE: # kernel0
-; CODE-NEXT: Stmt_bb2(32 * b0 + t0);
-
-;    void foo(float A[], float b) {
-;      for (long i = 0; i < 1024; i++)
-;        A[i] += b;
-;    }
-;
-define void @float(ptr %A, float %b) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb5, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp6, %bb5 ]
-  %exitcond = icmp ne i64 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb7
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds float, ptr %A, i64 %i.0
-  %tmp3 = load float, ptr %tmp, align 4
-  %tmp4 = fadd float %tmp3, %b
-  store float %tmp4, ptr %tmp, align 4
-  br label %bb5
-
-bb5:                                              ; preds = %bb2
-  %tmp6 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb7:                                              ; preds = %bb1
-  ret void
-}
-
-; KERNEL: define ptx_kernel void @kernel_0(ptr %MemRef_A, double %MemRef_b)
-; KERNEL-NEXT: entry:
-; KERNEL-NEXT:   %b.s2a = alloca double
-; KERNEL-NEXT:   store double %MemRef_b, ptr %b.s2a
-
-; CODE: Code
-; CODE-NEXT: ====
-; CODE-NEXT: # host
-; CODE-NEXT: {
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(dev_MemRef_A, MemRef_A, (1024) * sizeof(double), cudaMemcpyHostToDevice));
-; CODE-NEXT:   {
-; CODE-NEXT:     dim3 k0_dimBlock(32);
-; CODE-NEXT:     dim3 k0_dimGrid(32);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_A, MemRef_b);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:   cudaCheckReturn(cudaMemcpy(MemRef_A, dev_MemRef_A, (1024) * sizeof(double), cudaMemcpyDeviceToHost));
-; CODE-NEXT: }
-
-; CODE: # kernel0
-; CODE-NEXT: Stmt_bb2(32 * b0 + t0);
-
-;    void foo(double A[], double b) {
-;      for (long i = 0; i < 1024; i++)
-;        A[i] += b;
-;    }
-;
-define void @double(ptr %A, double %b) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb5, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp6, %bb5 ]
-  %exitcond = icmp ne i64 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb7
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds double, ptr %A, i64 %i.0
-  %tmp3 = load double, ptr %tmp, align 4
-  %tmp4 = fadd double %tmp3, %b
-  store double %tmp4, ptr %tmp, align 4
-  br label %bb5
-
-bb5:                                              ; preds = %bb2
-  %tmp6 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb7:                                              ; preds = %bb1
-  ret void
-}
-
-; CODE: Code
-; CODE-NEXT: ====
-; CODE-NEXT: # host
-; CODE-NEXT: {
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(dev_MemRef_A, MemRef_A, (1024) * sizeof(i1), cudaMemcpyHostToDevice));
-; CODE-NEXT:   {
-; CODE-NEXT:     dim3 k0_dimBlock(32);
-; CODE-NEXT:     dim3 k0_dimGrid(32);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_A);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:   cudaCheckReturn(cudaMemcpy(MemRef_A, dev_MemRef_A, (1024) * sizeof(i1), cudaMemcpyDeviceToHost));
-; CODE-NEXT: }
-
-; CODE: # kernel0
-; CODE-NEXT: Stmt_bb2(32 * b0 + t0);
-
-;    void foo(i1 A[], i1 b) {
-;      for (long i = 0; i < 1024; i++)
-;        A[i] += b;
-;    }
-;
-define void @i1(ptr %A, i1 %b) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb5, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp6, %bb5 ]
-  %exitcond = icmp ne i64 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb7
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds i1, ptr %A, i64 %i.0
-  %tmp3 = load i1, ptr %tmp, align 4
-  %tmp4 = add i1 %tmp3, %b
-  store i1 %tmp4, ptr %tmp, align 4
-  br label %bb5
-
-bb5:                                              ; preds = %bb2
-  %tmp6 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb7:                                              ; preds = %bb1
-  ret void
-}
-
-; CODE: Code
-; CODE-NEXT: ====
-; CODE-NEXT: # host
-; CODE-NEXT: {
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(dev_MemRef_A, MemRef_A, (1024) * sizeof(i3), cudaMemcpyHostToDevice));
-; CODE-NEXT:   {
-; CODE-NEXT:     dim3 k0_dimBlock(32);
-; CODE-NEXT:     dim3 k0_dimGrid(32);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_A);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:   cudaCheckReturn(cudaMemcpy(MemRef_A, dev_MemRef_A, (1024) * sizeof(i3), cudaMemcpyDeviceToHost));
-; CODE-NEXT: }
-
-; CODE: # kernel0
-; CODE-NEXT: Stmt_bb2(32 * b0 + t0);
-
-;    void foo(i3 A[], i3 b) {
-;      for (long i = 0; i < 1024; i++)
-;        A[i] += b;
-;    }
-;
-define void @i3(ptr %A, i3 %b) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb5, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp6, %bb5 ]
-  %exitcond = icmp ne i64 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb7
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds i3, ptr %A, i64 %i.0
-  %tmp3 = load i3, ptr %tmp, align 4
-  %tmp4 = add i3 %tmp3, %b
-  store i3 %tmp4, ptr %tmp, align 4
-  br label %bb5
-
-bb5:                                              ; preds = %bb2
-  %tmp6 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb7:                                              ; preds = %bb1
-  ret void
-}
-
-; CODE: Code
-; CODE-NEXT: ====
-; CODE-NEXT: # host
-; CODE-NEXT: {
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(dev_MemRef_A, MemRef_A, (1024) * sizeof(i8), cudaMemcpyHostToDevice));
-; CODE-NEXT:   {
-; CODE-NEXT:     dim3 k0_dimBlock(32);
-; CODE-NEXT:     dim3 k0_dimGrid(32);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_A);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:   cudaCheckReturn(cudaMemcpy(MemRef_A, dev_MemRef_A, (1024) * sizeof(i8), cudaMemcpyDeviceToHost));
-; CODE-NEXT: }
-
-; CODE: # kernel0
-; CODE-NEXT: Stmt_bb2(32 * b0 + t0);
-
-;    void foo(i8 A[], i32 b) {
-;      for (long i = 0; i < 1024; i++)
-;        A[i] += b;
-;    }
-;
-define void @i8(ptr %A, i8 %b) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb5, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp6, %bb5 ]
-  %exitcond = icmp ne i64 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb7
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds i8, ptr %A, i64 %i.0
-  %tmp3 = load i8, ptr %tmp, align 4
-  %tmp4 = add i8 %tmp3, %b
-  store i8 %tmp4, ptr %tmp, align 4
-  br label %bb5
-
-bb5:                                              ; preds = %bb2
-  %tmp6 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb7:                                              ; preds = %bb1
-  ret void
-}
-
-; IR-LABEL: @i8
-
-; IR: [[REGA:%.+]] = call ptr @polly_getDevicePtr(ptr %p_dev_array_MemRef_A)
-; IR-NEXT: store ptr [[REGA:%.+]], ptr %polly_launch_0_param_0
-; IR-NEXT: store ptr %polly_launch_0_param_0, ptr %polly_launch_0_params
-; IR-NEXT: store i8 %b, ptr %polly_launch_0_param_1
-; IR-NEXT: [[REGD:%.+]] = getelementptr [2 x ptr], ptr %polly_launch_0_params, i64 0, i64 1
-; IR-NEXT: store ptr %polly_launch_0_param_1, ptr [[REGD]]
-
-; CODE: Code
-; CODE-NEXT: ====
-; CODE-NEXT: # host
-; CODE-NEXT: {
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(dev_MemRef_A, MemRef_A, (1024) * sizeof(i32), cudaMemcpyHostToDevice));
-; CODE-NEXT:   {
-; CODE-NEXT:     dim3 k0_dimBlock(32);
-; CODE-NEXT:     dim3 k0_dimGrid(32);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_A);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:   cudaCheckReturn(cudaMemcpy(MemRef_A, dev_MemRef_A, (1024) * sizeof(i32), cudaMemcpyDeviceToHost));
-; CODE-NEXT: }
-
-; CODE: # kernel0
-; CODE-NEXT: Stmt_bb2(32 * b0 + t0);
-
-;    void foo(i32 A[], i32 b) {
-;      for (long i = 0; i < 1024; i++)
-;        A[i] += b;
-;    }
-;
-define void @i32(ptr %A, i32 %b) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb5, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp6, %bb5 ]
-  %exitcond = icmp ne i64 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb7
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds i32, ptr %A, i64 %i.0
-  %tmp3 = load i32, ptr %tmp, align 4
-  %tmp4 = add i32 %tmp3, %b
-  store i32 %tmp4, ptr %tmp, align 4
-  br label %bb5
-
-bb5:                                              ; preds = %bb2
-  %tmp6 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb7:                                              ; preds = %bb1
-  ret void
-}
-
-; CODE: Code
-; CODE-NEXT: ====
-; CODE-NEXT: # host
-; CODE-NEXT: {
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(dev_MemRef_A, MemRef_A, (1024) * sizeof(i60), cudaMemcpyHostToDevice));
-; CODE-NEXT:   {
-; CODE-NEXT:     dim3 k0_dimBlock(32);
-; CODE-NEXT:     dim3 k0_dimGrid(32);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_A);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:   cudaCheckReturn(cudaMemcpy(MemRef_A, dev_MemRef_A, (1024) * sizeof(i60), cudaMemcpyDeviceToHost));
-; CODE-NEXT: }
-
-; CODE: # kernel0
-; CODE-NEXT: Stmt_bb2(32 * b0 + t0);
-
-;    void foo(i60 A[], i60 b) {
-;      for (long i = 0; i < 1024; i++)
-;        A[i] += b;
-;    }
-;
-define void @i60(ptr %A, i60 %b) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb5, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp6, %bb5 ]
-  %exitcond = icmp ne i64 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb7
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds i60, ptr %A, i64 %i.0
-  %tmp3 = load i60, ptr %tmp, align 4
-  %tmp4 = add i60 %tmp3, %b
-  store i60 %tmp4, ptr %tmp, align 4
-  br label %bb5
-
-bb5:                                              ; preds = %bb2
-  %tmp6 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb7:                                              ; preds = %bb1
-  ret void
-}
-
-; CODE: Code
-; CODE-NEXT: ====
-; CODE-NEXT: # host
-; CODE-NEXT: {
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(dev_MemRef_A, MemRef_A, (1024) * sizeof(i64), cudaMemcpyHostToDevice));
-; CODE-NEXT:   {
-; CODE-NEXT:     dim3 k0_dimBlock(32);
-; CODE-NEXT:     dim3 k0_dimGrid(32);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_A);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:   cudaCheckReturn(cudaMemcpy(MemRef_A, dev_MemRef_A, (1024) * sizeof(i64), cudaMemcpyDeviceToHost));
-; CODE-NEXT: }
-
-; CODE: # kernel0
-; CODE-NEXT: Stmt_bb2(32 * b0 + t0);
-
-;    void foo(i64 A[], i64 b) {
-;      for (long i = 0; i < 1024; i++)
-;        A[i] += b;
-;    }
-;
-define void @i64(ptr %A, i64 %b) {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb5, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp6, %bb5 ]
-  %exitcond = icmp ne i64 %i.0, 1024
-  br i1 %exitcond, label %bb2, label %bb7
-
-bb2:                                              ; preds = %bb1
-  %tmp = getelementptr inbounds i64, ptr %A, i64 %i.0
-  %tmp3 = load i64, ptr %tmp, align 4
-  %tmp4 = add i64 %tmp3, %b
-  store i64 %tmp4, ptr %tmp, align 4
-  br label %bb5
-
-bb5:                                              ; preds = %bb2
-  %tmp6 = add nuw nsw i64 %i.0, 1
-  br label %bb1
-
-bb7:                                              ; preds = %bb1
-  ret void
-}
diff --git a/polly/test/GPGPU/scalar-writes-in-scop-requires-abort.ll b/polly/test/GPGPU/scalar-writes-in-scop-requires-abort.ll
deleted file mode 100644
index 31110437fdca5..0000000000000
--- a/polly/test/GPGPU/scalar-writes-in-scop-requires-abort.ll
+++ /dev/null
@@ -1,65 +0,0 @@
-; RUN: opt %loadPolly -polly-acc-dump-code -polly-invariant-load-hoisting -polly-print-scops -disable-output < %s | FileCheck %s -check-prefix=SCOP
-
-; RUN: opt %loadPolly -S -polly-use-llvm-names -polly-codegen-ppcg \
-; RUN: -polly-acc-dump-code -polly-stmt-granularity=bb \
-; RUN: -polly-invariant-load-hoisting < %s | FileCheck %s -check-prefix=CODE
-
-; RUN: opt %loadPolly -S -polly-use-llvm-names -polly-codegen-ppcg \
-; RUN: -polly-invariant-load-hoisting -polly-stmt-granularity=bb < %s \
-; RUN: | FileCheck %s -check-prefix=HOST-IR
-
-; REQUIRES: pollyacc
-
-; SCOP:      Invariant Accesses: {
-; SCOP-NEXT:         ReadAccess :=	[Reduction Type: NONE] [Scalar: 0]
-; SCOP-NEXT:             { Stmt_loop_a[i0] -> MemRef_p[0] };
-; SCOP-NEXT:         Execution Context: {  :  }
-; SCOP-NEXT: }
-
-; CODE: # kernel0
-; CODE-NEXT: {
-; CODE-NEXT:   if (32 * b0 + t0 <= 1025) {
-; CODE-NEXT:     Stmt_loop(32 * b0 + t0);
-; CODE-NEXT:     write(0);
-; CODE-NEXT:   }
-; CODE-NEXT:   sync0();
-; CODE-NEXT: }
-
-; Check that we generate a correct "always false" branch.
-; HOST-IR:  br i1 false, label %polly.start, label %loop.pre_entry_bb
-
-; This test case checks that we generate correct code if PPCGCodeGeneration
-; decides a build is unsuccessful with invariant load hoisting enabled.
-;
-; There is a conditional branch which switches between the original code and
-; the new code. We try to set this conditional branch to branch on false.
-; However, invariant load hoisting changes the structure of the scop, so we
-; need to change the way we *locate* this instruction.
-
-target datalayout = "e-m:o-p:32:32-f64:32:64-f80:128-n8:16:32-S128"
-target triple = "i386-apple-macosx10.12.0"
-
-define void @foo(ptr %A, ptr %p) {
-entry:
-  br label %loop
-
-loop:
-  %indvar = phi i64 [0, %entry], [%indvar.next, %loop]
-  %indvar.next = add i64 %indvar, 1
-  %invariant = load float, ptr %p
-  %ptr = getelementptr float, ptr %A, i64 %indvar
-  store float 42.0, ptr %ptr
-  %cmp = icmp sle i64 %indvar, 1024
-  br i1 %cmp, label %loop, label %loop2
-
-loop2:
-  %indvar2 = phi i64 [0, %loop], [%indvar2.next, %loop2]
-  %indvar2f = phi float [%invariant, %loop], [%indvar2f, %loop2]
-  %indvar2.next = add i64 %indvar2, 1
-  store float %indvar2f, ptr %A
-  %cmp2 = icmp sle i64 %indvar2, 1024
-  br i1 %cmp2, label %loop2, label %end
-
-end:
-  ret void
-}
diff --git a/polly/test/GPGPU/scheduler-timeout.ll b/polly/test/GPGPU/scheduler-timeout.ll
deleted file mode 100644
index 4a49c53d66c74..0000000000000
--- a/polly/test/GPGPU/scheduler-timeout.ll
+++ /dev/null
@@ -1,174 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-
-; REQUIRES: pollyacc
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-target triple = "x86_64-unknown-linux-gnu"
-
-; This test case took at some point forever to schedule, as the isl scheduler
-; seems to have problems if domain constraints appear in the dependences
-; provided to the scheduler.
-
-;   /* D := alpha*A*B*C + beta*D */
-;   for (i = 0; i < _PB_NI; i++)
-;     for (j = 0; j < _PB_NJ; j++)
-;       {
-;   tmp[i][j] = 0;
-;   for (k = 0; k < _PB_NK; ++k)
-;     tmp[i][j] += alpha * A[i][k] * B[k][j];
-;       }
-;   for (i = 0; i < _PB_NI; i++)
-;     for (j = 0; j < _PB_NL; j++)
-;       {
-;   D[i][j] *= beta;
-;   for (k = 0; k < _PB_NJ; ++k)
-;     D[i][j] += tmp[i][k] * C[k][j];
-;       }
-
-; CODE:        cudaCheckReturn(cudaMemcpy(dev_MemRef_A, MemRef_A, (4096) * (4096) * sizeof(float), cudaMemcpyHostToDevice));
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(dev_MemRef_B, MemRef_B, (4096) * (4096) * sizeof(float), cudaMemcpyHostToDevice));
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(dev_MemRef_D, MemRef_D, (4096) * (4096) * sizeof(float), cudaMemcpyHostToDevice));
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(dev_MemRef_C, MemRef_C, (4096) * (4096) * sizeof(float), cudaMemcpyHostToDevice));
-; CODE-NEXT:   {
-; CODE-NEXT:     dim3 k0_dimBlock(16, 32);
-; CODE-NEXT:     dim3 k0_dimGrid(128, 128);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_tmp, dev_MemRef_A, MemRef_alpha, dev_MemRef_B);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:   {
-; CODE-NEXT:     dim3 k1_dimBlock(16, 32);
-; CODE-NEXT:     dim3 k1_dimGrid(128, 128);
-; CODE-NEXT:     kernel1 <<<k1_dimGrid, k1_dimBlock>>> (dev_MemRef_tmp, dev_MemRef_D, MemRef_beta, dev_MemRef_C);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:   cudaCheckReturn(cudaMemcpy(MemRef_tmp, dev_MemRef_tmp, (4096) * (4096) * sizeof(float), cudaMemcpyDeviceToHost));
-; CODE-NEXT:   cudaCheckReturn(cudaMemcpy(MemRef_D, dev_MemRef_D, (4096) * (4096) * sizeof(float), cudaMemcpyDeviceToHost));
-
-; CODE: # kernel0
-; CODE-NEXT: for (int c2 = 0; c2 <= 127; c2 += 1)
-; CODE-NEXT:   for (int c4 = 0; c4 <= 1; c4 += 1) {
-; CODE-NEXT:     if (c2 == 0)
-; CODE-NEXT:       Stmt_for_body6(32 * b0 + t0, 32 * b1 + t1 + 16 * c4);
-; CODE-NEXT:     for (int c5 = 0; c5 <= 31; c5 += 1)
-; CODE-NEXT:       Stmt_for_body11(32 * b0 + t0, 32 * b1 + t1 + 16 * c4, 32 * c2 + c5);
-; CODE-NEXT:   }
-
-; CODE: # kernel1
-; CODE-NEXT: for (int c2 = 0; c2 <= 127; c2 += 1)
-; CODE-NEXT:   for (int c4 = 0; c4 <= 1; c4 += 1) {
-; CODE-NEXT:     if (c2 == 0)
-; CODE-NEXT:       Stmt_for_body36(32 * b0 + t0, 32 * b1 + t1 + 16 * c4);
-; CODE-NEXT:     for (int c5 = 0; c5 <= 31; c5 += 1)
-; CODE-NEXT:       Stmt_for_body44(32 * b0 + t0, 32 * b1 + t1 + 16 * c4, 32 * c2 + c5);
-; CODE-NEXT:   }
-
-
-
-; Function Attrs: argmemonly nounwind
-declare void @llvm.lifetime.start(i64, ptr nocapture) #0
-
-; Function Attrs: nounwind uwtable
-define internal void @kernel_2mm(i32 %ni, i32 %nj, i32 %nk, i32 %nl, float %alpha, float %beta, ptr %tmp, ptr %A, ptr %B, ptr %C, ptr %D) #1 {
-entry:
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  br label %for.cond4.preheader
-
-for.cond4.preheader:                              ; preds = %entry.split, %for.inc28
-  %indvars.iv19 = phi i64 [ 0, %entry.split ], [ %indvars.iv.next20, %for.inc28 ]
-  br label %for.body6
-
-for.cond31.preheader:                             ; preds = %for.inc28
-  br label %for.cond34.preheader
-
-for.body6:                                        ; preds = %for.cond4.preheader, %for.inc25
-  %indvars.iv16 = phi i64 [ 0, %for.cond4.preheader ], [ %indvars.iv.next17, %for.inc25 ]
-  %arrayidx8 = getelementptr inbounds [4096 x float], ptr %tmp, i64 %indvars.iv19, i64 %indvars.iv16
-  store float 0.000000e+00, ptr %arrayidx8, align 4, !tbaa !1
-  br label %for.body11
-
-for.body11:                                       ; preds = %for.body6, %for.body11
-  %indvars.iv13 = phi i64 [ 0, %for.body6 ], [ %indvars.iv.next14, %for.body11 ]
-  %arrayidx15 = getelementptr inbounds [4096 x float], ptr %A, i64 %indvars.iv19, i64 %indvars.iv13
-  %tmp22 = load float, ptr %arrayidx15, align 4, !tbaa !1
-  %mul = fmul float %tmp22, %alpha
-  %arrayidx19 = getelementptr inbounds [4096 x float], ptr %B, i64 %indvars.iv13, i64 %indvars.iv16
-  %tmp23 = load float, ptr %arrayidx19, align 4, !tbaa !1
-  %mul20 = fmul float %mul, %tmp23
-  %arrayidx24 = getelementptr inbounds [4096 x float], ptr %tmp, i64 %indvars.iv19, i64 %indvars.iv16
-  %tmp24 = load float, ptr %arrayidx24, align 4, !tbaa !1
-  %add = fadd float %tmp24, %mul20
-  store float %add, ptr %arrayidx24, align 4, !tbaa !1
-  %indvars.iv.next14 = add nuw nsw i64 %indvars.iv13, 1
-  %exitcond15 = icmp ne i64 %indvars.iv.next14, 4096
-  br i1 %exitcond15, label %for.body11, label %for.inc25
-
-for.inc25:                                        ; preds = %for.body11
-  %indvars.iv.next17 = add nuw nsw i64 %indvars.iv16, 1
-  %exitcond18 = icmp ne i64 %indvars.iv.next17, 4096
-  br i1 %exitcond18, label %for.body6, label %for.inc28
-
-for.inc28:                                        ; preds = %for.inc25
-  %indvars.iv.next20 = add nuw nsw i64 %indvars.iv19, 1
-  %exitcond21 = icmp ne i64 %indvars.iv.next20, 4096
-  br i1 %exitcond21, label %for.cond4.preheader, label %for.cond31.preheader
-
-for.cond34.preheader:                             ; preds = %for.cond31.preheader, %for.inc65
-  %indvars.iv10 = phi i64 [ 0, %for.cond31.preheader ], [ %indvars.iv.next11, %for.inc65 ]
-  br label %for.body36
-
-for.body36:                                       ; preds = %for.cond34.preheader, %for.inc62
-  %indvars.iv7 = phi i64 [ 0, %for.cond34.preheader ], [ %indvars.iv.next8, %for.inc62 ]
-  %arrayidx40 = getelementptr inbounds [4096 x float], ptr %D, i64 %indvars.iv10, i64 %indvars.iv7
-  %tmp25 = load float, ptr %arrayidx40, align 4, !tbaa !1
-  %mul41 = fmul float %tmp25, %beta
-  store float %mul41, ptr %arrayidx40, align 4, !tbaa !1
-  br label %for.body44
-
-for.body44:                                       ; preds = %for.body36, %for.body44
-  %indvars.iv = phi i64 [ 0, %for.body36 ], [ %indvars.iv.next, %for.body44 ]
-  %arrayidx48 = getelementptr inbounds [4096 x float], ptr %tmp, i64 %indvars.iv10, i64 %indvars.iv
-  %tmp26 = load float, ptr %arrayidx48, align 4, !tbaa !1
-  %arrayidx52 = getelementptr inbounds [4096 x float], ptr %C, i64 %indvars.iv, i64 %indvars.iv7
-  %tmp27 = load float, ptr %arrayidx52, align 4, !tbaa !1
-  %mul53 = fmul float %tmp26, %tmp27
-  %arrayidx57 = getelementptr inbounds [4096 x float], ptr %D, i64 %indvars.iv10, i64 %indvars.iv7
-  %tmp28 = load float, ptr %arrayidx57, align 4, !tbaa !1
-  %add58 = fadd float %tmp28, %mul53
-  store float %add58, ptr %arrayidx57, align 4, !tbaa !1
-  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
-  %exitcond = icmp ne i64 %indvars.iv.next, 4096
-  br i1 %exitcond, label %for.body44, label %for.inc62
-
-for.inc62:                                        ; preds = %for.body44
-  %indvars.iv.next8 = add nuw nsw i64 %indvars.iv7, 1
-  %exitcond9 = icmp ne i64 %indvars.iv.next8, 4096
-  br i1 %exitcond9, label %for.body36, label %for.inc65
-
-for.inc65:                                        ; preds = %for.inc62
-  %indvars.iv.next11 = add nuw nsw i64 %indvars.iv10, 1
-  %exitcond12 = icmp ne i64 %indvars.iv.next11, 4096
-  br i1 %exitcond12, label %for.cond34.preheader, label %for.end67
-
-for.end67:                                        ; preds = %for.inc65
-  ret void
-}
-
-; Function Attrs: argmemonly nounwind
-declare void @llvm.lifetime.end(i64, ptr nocapture) #0
-
-attributes #0 = { argmemonly nounwind }
-attributes #1 = { nounwind uwtable "disable-tail-calls"="false" "less-precise-fpmad"="false" "frame-pointer"="none" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+fxsr,+mmx,+sse,+sse2,+x87" "unsafe-fp-math"="false" "use-soft-float"="false" }
-
-!llvm.ident = !{!0}
-
-!0 = !{!"clang version 3.9.0 (trunk 275267) (llvm/trunk 275268)"}
-!1 = !{!2, !2, i64 0}
-!2 = !{!"float", !3, i64 0}
-!3 = !{!"omnipotent char", !4, i64 0}
-!4 = !{!"Simple C/C++ TBAA"}
diff --git a/polly/test/GPGPU/shared-memory-scalar.ll b/polly/test/GPGPU/shared-memory-scalar.ll
deleted file mode 100644
index cd2b1705a388f..0000000000000
--- a/polly/test/GPGPU/shared-memory-scalar.ll
+++ /dev/null
@@ -1,65 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -polly-acc-use-shared \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-
-; REQUIRES: pollyacc
-
-;    void add(float *A, float alpha) {
-;      for (long i = 0; i < 32; i++)
-;        for (long j = 0; j < 10; j++)
-;          A[i] += alpha;
-;    }
-
-; CODE:       read(t0);
-; CODE-NEXT:  sync0();
-; CODE-NEXT:  for (int c3 = 0; c3 <= 9; c3 += 1)
-; CODE-NEXT:    Stmt_bb5(t0, c3);
-; CODE-NEXT:  sync1();
-; CODE-NEXT:  write(t0);
-
-; This test case was intended to test code generation for scalars stored
-; in shared memory. However, after properly marking the scalar as read-only
-; the scalar is not stored any more in shared memory. We still leave this
-; test case as documentation if we every forget to mark scalars as read-only.
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @add(ptr %A, float %alpha) {
-bb:
-  br label %bb2
-
-bb2:                                              ; preds = %bb11, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp12, %bb11 ]
-  %exitcond1 = icmp ne i64 %i.0, 32
-  br i1 %exitcond1, label %bb3, label %bb13
-
-bb3:                                              ; preds = %bb2
-  br label %bb4
-
-bb4:                                              ; preds = %bb8, %bb3
-  %j.0 = phi i64 [ 0, %bb3 ], [ %tmp9, %bb8 ]
-  %exitcond = icmp ne i64 %j.0, 10
-  br i1 %exitcond, label %bb5, label %bb10
-
-bb5:                                              ; preds = %bb4
-  %tmp = getelementptr inbounds float, ptr %A, i64 %i.0
-  %tmp6 = load float, ptr %tmp, align 4
-  %tmp7 = fadd float %tmp6, %alpha
-  store float %tmp7, ptr %tmp, align 4
-  br label %bb8
-
-bb8:                                              ; preds = %bb5
-  %tmp9 = add nuw nsw i64 %j.0, 1
-  br label %bb4
-
-bb10:                                             ; preds = %bb4
-  br label %bb11
-
-bb11:                                             ; preds = %bb10
-  %tmp12 = add nuw nsw i64 %i.0, 1
-  br label %bb2
-
-bb13:                                             ; preds = %bb2
-  ret void
-}
diff --git a/polly/test/GPGPU/shared-memory-two-dimensional.ll b/polly/test/GPGPU/shared-memory-two-dimensional.ll
deleted file mode 100644
index 6ee51650295f8..0000000000000
--- a/polly/test/GPGPU/shared-memory-two-dimensional.ll
+++ /dev/null
@@ -1,103 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -polly-acc-use-shared \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen-ppcg \
-; RUN: -polly-acc-use-shared \
-; RUN: -disable-output -polly-acc-dump-kernel-ir < %s | \
-; RUN: FileCheck -check-prefix=KERNEL %s
-
-; REQUIRES: pollyacc
-
-;    void foo(float A[], float b[][8]) {
-;      for (long i = 0; i < 32; i++)
-;        for (long j = 0; j < 16; j++)
-;          for (long k = 0; k < 8; k++)
-;            A[i] += j * k * b[j][k];
-;    }
-
-
-; CODE:      # kernel0
-; CODE-NEXT: {
-; CODE-NEXT:   if (t0 <= 7)
-; CODE-NEXT:     for (int c0 = 0; c0 <= 15; c0 += 1)
-; CODE-NEXT:       read(c0, t0);
-; CODE-NEXT:   read(t0);
-; CODE-NEXT:   sync0();
-; CODE-NEXT:   for (int c3 = 0; c3 <= 15; c3 += 1)
-; CODE-NEXT:     for (int c4 = 0; c4 <= 7; c4 += 1)
-; CODE-NEXT:       Stmt_bb8(t0, c3, c4);
-; CODE-NEXT:   sync1();
-; CODE-NEXT:   write(t0);
-; CODE-NEXT: }
-
-; KERNEL: @shared_MemRef_b = internal addrspace(3) global [16 x [8 x float]] zeroinitializer, align 4
-
-; KERNEL:        %polly.access.mul.MemRef_b = mul nsw i64 %polly.indvar, 8
-; KERNEL-NEXT:   %polly.access.add.MemRef_b = add nsw i64 %polly.access.mul.MemRef_b, %t0
-; KERNEL-NEXT:   %polly.access.MemRef_b = getelementptr float, float addrspace(1)* %polly.access.cast.MemRef_b, i64 %polly.access.add.MemRef_b
-; KERNEL-NEXT:   %shared.read = load float, float addrspace(1)* %polly.access.MemRef_b
-; KERNEL-NEXT:   store float %shared.read, float addrspace(3)* %polly.access.shared_MemRef_b
-
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @foo(float* %A, [8 x float]* %b) {
-bb:
-  br label %bb3
-
-bb3:                                              ; preds = %bb22, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp23, %bb22 ]
-  %exitcond2 = icmp ne i64 %i.0, 32
-  br i1 %exitcond2, label %bb4, label %bb24
-
-bb4:                                              ; preds = %bb3
-  br label %bb5
-
-bb5:                                              ; preds = %bb19, %bb4
-  %j.0 = phi i64 [ 0, %bb4 ], [ %tmp20, %bb19 ]
-  %exitcond1 = icmp ne i64 %j.0, 16
-  br i1 %exitcond1, label %bb6, label %bb21
-
-bb6:                                              ; preds = %bb5
-  br label %bb7
-
-bb7:                                              ; preds = %bb16, %bb6
-  %k.0 = phi i64 [ 0, %bb6 ], [ %tmp17, %bb16 ]
-  %exitcond = icmp ne i64 %k.0, 8
-  br i1 %exitcond, label %bb8, label %bb18
-
-bb8:                                              ; preds = %bb7
-  %tmp = mul nuw nsw i64 %j.0, %k.0
-  %tmp9 = sitofp i64 %tmp to float
-  %tmp10 = getelementptr inbounds [8 x float], [8 x float]* %b, i64 %j.0, i64 %k.0
-  %tmp11 = load float, float* %tmp10, align 4
-  %tmp12 = fmul float %tmp9, %tmp11
-  %tmp13 = getelementptr inbounds float, float* %A, i64 %i.0
-  %tmp14 = load float, float* %tmp13, align 4
-  %tmp15 = fadd float %tmp14, %tmp12
-  store float %tmp15, float* %tmp13, align 4
-  br label %bb16
-
-bb16:                                             ; preds = %bb8
-  %tmp17 = add nuw nsw i64 %k.0, 1
-  br label %bb7
-
-bb18:                                             ; preds = %bb7
-  br label %bb19
-
-bb19:                                             ; preds = %bb18
-  %tmp20 = add nuw nsw i64 %j.0, 1
-  br label %bb5
-
-bb21:                                             ; preds = %bb5
-  br label %bb22
-
-bb22:                                             ; preds = %bb21
-  %tmp23 = add nuw nsw i64 %i.0, 1
-  br label %bb3
-
-bb24:                                             ; preds = %bb3
-  ret void
-}
diff --git a/polly/test/GPGPU/shared-memory.ll b/polly/test/GPGPU/shared-memory.ll
deleted file mode 100644
index 920db0d371279..0000000000000
--- a/polly/test/GPGPU/shared-memory.ll
+++ /dev/null
@@ -1,83 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -polly-acc-use-shared \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen-ppcg \
-; RUN: -polly-acc-use-shared \
-; RUN: -disable-output -polly-acc-dump-kernel-ir < %s | \
-; RUN: FileCheck -check-prefix=KERNEL %s
-
-; REQUIRES: pollyacc
-
-;    void add(float *A) {
-;      for (long i = 0; i < 32; i++)
-;        for (long j = 0; j < 10; j++)
-;          A[i] += 1;
-;    }
-
-; CODE: # kernel0
-; CODE: {
-; CODE:   read(t0);
-; CODE:   sync0();
-; CODE:   for (int c3 = 0; c3 <= 9; c3 += 1)
-; CODE:     Stmt_bb5(t0, c3);
-; CODE:   sync1();
-; CODE:   write(t0);
-; CODE: }
-
-; KERNEL: @shared_MemRef_A = internal addrspace(3) global [32 x float] zeroinitializer, align 4
-
-; KERNEL:   %polly.access.shared_MemRef_A = getelementptr float, float addrspace(3)* getelementptr inbounds ([32 x float], [32 x float] addrspace(3)* @shared_MemRef_A, i32 0, i32 0), i64 %t0
-; KERNEL-NEXT:   %polly.access.cast.MemRef_A = bitcast i8 addrspace(1)* %MemRef_A to float addrspace(1)*
-; KERNEL-NEXT:   %polly.access.MemRef_A = getelementptr float, float addrspace(1)* %polly.access.cast.MemRef_A, i64 %t0
-; KERNEL-NEXT:   %shared.read = load float, float addrspace(1)* %polly.access.MemRef_A
-; KERNEL-NEXT:   store float %shared.read, float addrspace(3)* %polly.access.shared_MemRef_A
-
-; KERNEL:   %polly.access.shared_MemRef_A3 = getelementptr float, float addrspace(3)* getelementptr inbounds ([32 x float], [32 x float] addrspace(3)* @shared_MemRef_A, i32 0, i32 0), i64 %t0
-; KERNEL-NEXT:   %polly.access.cast.MemRef_A4 = bitcast i8 addrspace(1)* %MemRef_A to float addrspace(1)*
-; KERNEL-NEXT:   %polly.access.MemRef_A5 = getelementptr float, float addrspace(1)* %polly.access.cast.MemRef_A4, i64 %t0
-; KERNEL-NEXT:   %shared.write = load float, float addrspace(3)* %polly.access.shared_MemRef_A3
-; KERNEL-NEXT:   store float %shared.write, float addrspace(1)* %polly.access.MemRef_A5
-
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @add(float* %A) {
-bb:
-  br label %bb2
-
-bb2:                                              ; preds = %bb11, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp12, %bb11 ]
-  %exitcond1 = icmp ne i64 %i.0, 32
-  br i1 %exitcond1, label %bb3, label %bb13
-
-bb3:                                              ; preds = %bb2
-  br label %bb4
-
-bb4:                                              ; preds = %bb8, %bb3
-  %j.0 = phi i64 [ 0, %bb3 ], [ %tmp9, %bb8 ]
-  %exitcond = icmp ne i64 %j.0, 10
-  br i1 %exitcond, label %bb5, label %bb10
-
-bb5:                                              ; preds = %bb4
-  %tmp = getelementptr inbounds float, float* %A, i64 %i.0
-  %tmp6 = load float, float* %tmp, align 4
-  %tmp7 = fadd float %tmp6, 1.000000e+00
-  store float %tmp7, float* %tmp, align 4
-  br label %bb8
-
-bb8:                                              ; preds = %bb5
-  %tmp9 = add nuw nsw i64 %j.0, 1
-  br label %bb4
-
-bb10:                                             ; preds = %bb4
-  br label %bb11
-
-bb11:                                             ; preds = %bb10
-  %tmp12 = add nuw nsw i64 %i.0, 1
-  br label %bb2
-
-bb13:                                             ; preds = %bb2
-  ret void
-}
diff --git a/polly/test/GPGPU/simple-managed-memory-rewrite.ll b/polly/test/GPGPU/simple-managed-memory-rewrite.ll
deleted file mode 100644
index d8c5b320e2b04..0000000000000
--- a/polly/test/GPGPU/simple-managed-memory-rewrite.ll
+++ /dev/null
@@ -1,71 +0,0 @@
-; RUN: opt %loadPolly -polly-print-scops -disable-output < %s |  FileCheck %s --check-prefix=SCOP
-
-; RUN: opt %loadPolly -S  -polly-process-unprofitable -polly-acc-mincompute=0 \
-; RUN: -polly-codegen-ppcg -polly-acc-codegen-managed-memory \
-; RUN: -polly-acc-rewrite-managed-memory < %s | FileCheck %s --check-prefix=HOST-IR
-
-; REQUIRES: pollyacc
-
-; SCOP:      Function: f
-; SCOP-NEXT: Region: %for.body---%for.end
-; SCOP-NEXT: Max Loop Depth:  1
-; SCOP: i32 MemRef_A[*];
-
-; Check that we generate a constructor call for @A.toptr
-; HOST-IR: @llvm.global_ctors = appending global [1 x { i32, ptr, ptr }] [{ i32, ptr, ptr } { i32 0, ptr {{.*}}, ptr @A.toptr }]
-
-; Check that we generate a constructor
-; 4 bytes * 100 = 400
-; HOST-IR: define void {{.*}}constructor() {
-; HOST-IR-NEXT: entry:
-; HOST-IR-NEXT:   %mem.raw = call ptr @polly_mallocManaged(i64 400)
-; HOST-IR-NEXT:   store ptr %mem.raw, ptr @A.toptr
-; HOST-IR-NEXT:   ret void
-; HOST-IR-NEXT: }
-
-; HOST-IR-NOT: @A
-
-source_filename = "test.c"
-target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
-target triple = "x86_64-apple-macosx10.12.0"
-
-@A = internal global [100 x i32] zeroinitializer, align 16
-
-define void @f() {
-entry:
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  br label %for.body
-
-for.body:                                         ; preds = %entry.split, %for.body
-  %indvars.iv1 = phi i64 [ 0, %entry.split ], [ %indvars.iv.next, %for.body ]
-  %arrayidx = getelementptr inbounds [100 x i32], ptr @A, i64 0, i64 %indvars.iv1
-  store i32 42, ptr %arrayidx, align 4, !tbaa !3
-  %indvars.iv.next = add nuw nsw i64 %indvars.iv1, 1
-  %exitcond = icmp eq i64 %indvars.iv.next, 100
-  br i1 %exitcond, label %for.end, label %for.body
-
-for.end:                                          ; preds = %for.body
-  ret void
-}
-
-; Function Attrs: argmemonly nounwind
-declare void @llvm.lifetime.start.p0(i64, ptr nocapture) #0
-
-
-; Function Attrs: argmemonly nounwind
-declare void @llvm.lifetime.end.p0(i64, ptr nocapture) #0
-
-attributes #0 = { argmemonly nounwind }
-
-!llvm.module.flags = !{!0, !1}
-!llvm.ident = !{!2}
-
-!0 = !{i32 1, !"wchar_size", i32 4}
-!1 = !{i32 7, !"PIC Level", i32 2}
-!2 = !{!"clang version 6.0.0"}
-!3 = !{!4, !4, i64 0}
-!4 = !{!"int", !5, i64 0}
-!5 = !{!"omnipotent char", !6, i64 0}
-!6 = !{!"Simple C/C++ TBAA"}
diff --git a/polly/test/GPGPU/size-cast.ll b/polly/test/GPGPU/size-cast.ll
deleted file mode 100644
index 5e2c85de42513..0000000000000
--- a/polly/test/GPGPU/size-cast.ll
+++ /dev/null
@@ -1,63 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-
-; RUN: opt %loadPolly -polly-codegen-ppcg -S < %s | \
-; RUN: FileCheck %s -check-prefix=IR
-
-; REQUIRES: pollyacc
-
-; This test case ensures that we properly sign-extend the types we are using.
-
-; CODE:      if (arg >= 1 && arg1 == 0) {
-; CODE:        cudaCheckReturn(cudaMemcpy(dev_MemRef_arg3, MemRef_arg3, (arg) * sizeof(double), cudaMemcpyHostToDevice));
-; CODE-NEXT:   {
-; CODE-NEXT:     dim3 k0_dimBlock(32);
-; CODE-NEXT:     dim3 k0_dimGrid(arg >= 1048545 ? 32768 : (arg + 31) / 32);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_arg3, dev_MemRef_arg2, arg, arg1);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:   cudaCheckReturn(cudaMemcpy(MemRef_arg2, dev_MemRef_arg2, (arg) * sizeof(double), cudaMemcpyDeviceToHost));
-; CODE-NEXT  cudaCheckReturn(cudaFree(dev_MemRef_arg3));
-; CODE-NEXT  cudaCheckReturn(cudaFree(dev_MemRef_arg2));
-
-; CODE: # kernel0
-; CODE-NEXT: for (int c0 = 0; c0 <= (arg - 32 * b0 - 1) / 1048576; c0 += 1)
-; CODE-NEXT:   if (arg >= 32 * b0 + t0 + 1048576 * c0 + 1)
-; CODE-NEXT:     Stmt_bb6(0, 32 * b0 + t0 + 1048576 * c0);
-
-; IR-LABEL:  call ptr @polly_initContextCUDA()
-; IR:        sext i32 %arg to i64
-; IR-NEXT:   mul i64
-; IR-NEXT:   @polly_allocateMemoryForDevice
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-target triple = "x86_64-unknown-linux-gnu"
-
-define void @hoge(i32 %arg, i32 %arg1, ptr %arg2, ptr %arg3) {
-bb:
-  br label %bb4
-
-bb4:                                              ; preds = %bb13, %bb
-  br label %bb6
-
-bb5:                                              ; preds = %bb13
-  ret void
-
-bb6:                                              ; preds = %bb6, %bb4
-  %tmp = phi i64 [ 0, %bb4 ], [ %tmp10, %bb6 ]
-  %tmp7 = getelementptr inbounds double, ptr %arg3, i64 %tmp
-  %tmp8 = load double, ptr %tmp7, align 8
-  %tmp9 = getelementptr inbounds [1000 x double], ptr %arg2, i64 0, i64 %tmp
-  store double %tmp8, ptr %tmp9, align 8
-  %tmp10 = add nuw nsw i64 %tmp, 1
-  %tmp11 = zext i32 %arg to i64
-  %tmp12 = icmp ne i64 %tmp10, %tmp11
-  br i1 %tmp12, label %bb6, label %bb13
-
-bb13:                                             ; preds = %bb6
-  %tmp14 = zext i32 %arg1 to i64
-  %tmp15 = icmp ne i64 0, %tmp14
-  br i1 %tmp15, label %bb4, label %bb5
-}
diff --git a/polly/test/GPGPU/spir-codegen.ll b/polly/test/GPGPU/spir-codegen.ll
deleted file mode 100644
index 3715e1ec4427a..0000000000000
--- a/polly/test/GPGPU/spir-codegen.ll
+++ /dev/null
@@ -1,118 +0,0 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-codegen-ppcg \
-; RUN: -polly-gpu-arch=spir32 \
-; RUN: -polly-acc-dump-kernel-ir -polly-process-unprofitable -disable-output < %s | \
-; RUN: FileCheck %s
-
-; REQUIRES: pollyacc
-
-; CHECK:      target datalayout = "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-i128:128:128-f32:32:32-f64:64:64-v16:16:16-v24:32:32-v32:32:32-v48:64:64-v64:64:64-v96:128:128-v128:128:128-v192:256:256-v256:256:256-v512:512:512-v1024:1024:1024"
-; CHECK-NEXT: target triple = "spir-unknown-unknown"
-
-; CHECK-LABEL: define spir_kernel void @FUNC_double_parallel_loop_SCOP_0_KERNEL_0(i8 addrspace(1)* %MemRef_A) #0 !kernel_arg_addr_space !0 !kernel_arg_name !1 !kernel_arg_access_qual !1 !kernel_arg_type !1 !kernel_arg_type_qual !1 !kernel_arg_base_type !1 {
-; CHECK-NEXT: entry:
-; CHECK-NEXT:   %0 = call i32 @__gen_ocl_get_group_id0()
-; CHECK-NEXT:   %__gen_ocl_get_group_id0 = zext i32 %0 to i64
-; CHECK-NEXT:   %1 = call i32 @__gen_ocl_get_group_id1()
-; CHECK-NEXT:   %__gen_ocl_get_group_id1 = zext i32 %1 to i64
-; CHECK-NEXT:   %2 = call i32 @__gen_ocl_get_local_id0()
-; CHECK-NEXT:   %__gen_ocl_get_local_id0 = zext i32 %2 to i64
-; CHECK-NEXT:   %3 = call i32 @__gen_ocl_get_local_id1()
-; CHECK-NEXT:   %__gen_ocl_get_local_id1 = zext i32 %3 to i64
-; CHECK-NEXT:   br label %polly.loop_preheader
-
-; CHECK-LABEL: polly.loop_exit:                                  ; preds = %polly.stmt.bb5
-; CHECK-NEXT:   ret void
-
-; CHECK-LABEL: polly.loop_header:                                ; preds = %polly.stmt.bb5, %polly.loop_preheader
-; CHECK-NEXT:   %polly.indvar = phi i64 [ 0, %polly.loop_preheader ], [ %polly.indvar_next, %polly.stmt.bb5 ]
-; CHECK-NEXT:   %4 = mul nsw i64 32, %__gen_ocl_get_group_id0
-; CHECK-NEXT:   %5 = add nsw i64 %4, %__gen_ocl_get_local_id0
-; CHECK-NEXT:   %6 = mul nsw i64 32, %__gen_ocl_get_group_id1
-; CHECK-NEXT:   %7 = add nsw i64 %6, %__gen_ocl_get_local_id1
-; CHECK-NEXT:   %8 = mul nsw i64 16, %polly.indvar
-; CHECK-NEXT:   %9 = add nsw i64 %7, %8
-; CHECK-NEXT:   br label %polly.stmt.bb5
-
-; CHECK-LABEL: polly.stmt.bb5:                                   ; preds = %polly.loop_header
-; CHECK-NEXT:   %10 = mul i64 %5, %9
-; CHECK-NEXT:   %p_tmp6 = sitofp i64 %10 to float
-; CHECK-NEXT:   %polly.access.cast.MemRef_A = bitcast i8 addrspace(1)* %MemRef_A to float addrspace(1)*
-; CHECK-NEXT:   %11 = mul nsw i64 32, %__gen_ocl_get_group_id0
-; CHECK-NEXT:   %12 = add nsw i64 %11, %__gen_ocl_get_local_id0
-; CHECK-NEXT:   %polly.access.mul.MemRef_A = mul nsw i64 %12, 1024
-; CHECK-NEXT:   %13 = mul nsw i64 32, %__gen_ocl_get_group_id1
-; CHECK-NEXT:   %14 = add nsw i64 %13, %__gen_ocl_get_local_id1
-; CHECK-NEXT:   %15 = mul nsw i64 16, %polly.indvar
-; CHECK-NEXT:   %16 = add nsw i64 %14, %15
-; CHECK-NEXT:   %polly.access.add.MemRef_A = add nsw i64 %polly.access.mul.MemRef_A, %16
-; CHECK-NEXT:   %polly.access.MemRef_A = getelementptr float, float addrspace(1)* %polly.access.cast.MemRef_A, i64 %polly.access.add.MemRef_A
-; CHECK-NEXT:   %tmp8_p_scalar_ = load float, float addrspace(1)* %polly.access.MemRef_A, align 4
-; CHECK-NEXT:   %p_tmp9 = fadd float %tmp8_p_scalar_, %p_tmp6
-; CHECK-NEXT:   %polly.access.cast.MemRef_A1 = bitcast i8 addrspace(1)* %MemRef_A to float addrspace(1)*
-; CHECK-NEXT:   %17 = mul nsw i64 32, %__gen_ocl_get_group_id0
-; CHECK-NEXT:   %18 = add nsw i64 %17, %__gen_ocl_get_local_id0
-; CHECK-NEXT:   %polly.access.mul.MemRef_A2 = mul nsw i64 %18, 1024
-; CHECK-NEXT:   %19 = mul nsw i64 32, %__gen_ocl_get_group_id1
-; CHECK-NEXT:   %20 = add nsw i64 %19, %__gen_ocl_get_local_id1
-; CHECK-NEXT:   %21 = mul nsw i64 16, %polly.indvar
-; CHECK-NEXT:   %22 = add nsw i64 %20, %21
-; CHECK-NEXT:   %polly.access.add.MemRef_A3 = add nsw i64 %polly.access.mul.MemRef_A2, %22
-; CHECK-NEXT:   %polly.access.MemRef_A4 = getelementptr float, float addrspace(1)* %polly.access.cast.MemRef_A1, i64 %polly.access.add.MemRef_A3
-; CHECK-NEXT:   store float %p_tmp9, float addrspace(1)* %polly.access.MemRef_A4, align 4
-; CHECK-NEXT:   %polly.indvar_next = add nsw i64 %polly.indvar, 1
-; CHECK-NEXT:   %polly.loop_cond = icmp sle i64 %polly.indvar_next, 1
-; CHECK-NEXT:   br i1 %polly.loop_cond, label %polly.loop_header, label %polly.loop_exit
-
-; CHECK-LABEL: polly.loop_preheader:                             ; preds = %entry
-; CHECK-NEXT:   br label %polly.loop_header
-
-; CHECK: attributes #0 = { "polly.skip.fn" }
-
-;    void double_parallel_loop(float A[][1024]) {
-;      for (long i = 0; i < 1024; i++)
-;        for (long j = 0; j < 1024; j++)
-;          A[i][j] += i * j;
-;    }
-;
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @double_parallel_loop([1024 x float]* %A) {
-bb:
-  br label %bb2
-
-bb2:                                              ; preds = %bb13, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp14, %bb13 ]
-  %exitcond1 = icmp ne i64 %i.0, 1024
-  br i1 %exitcond1, label %bb3, label %bb15
-
-bb3:                                              ; preds = %bb2
-  br label %bb4
-
-bb4:                                              ; preds = %bb10, %bb3
-  %j.0 = phi i64 [ 0, %bb3 ], [ %tmp11, %bb10 ]
-  %exitcond = icmp ne i64 %j.0, 1024
-  br i1 %exitcond, label %bb5, label %bb12
-
-bb5:                                              ; preds = %bb4
-  %tmp = mul nuw nsw i64 %i.0, %j.0
-  %tmp6 = sitofp i64 %tmp to float
-  %tmp7 = getelementptr inbounds [1024 x float], [1024 x float]* %A, i64 %i.0, i64 %j.0
-  %tmp8 = load float, float* %tmp7, align 4
-  %tmp9 = fadd float %tmp8, %tmp6
-  store float %tmp9, float* %tmp7, align 4
-  br label %bb10
-
-bb10:                                             ; preds = %bb5
-  %tmp11 = add nuw nsw i64 %j.0, 1
-  br label %bb4
-
-bb12:                                             ; preds = %bb4
-  br label %bb13
-
-bb13:                                             ; preds = %bb12
-  %tmp14 = add nuw nsw i64 %i.0, 1
-  br label %bb2
-
-bb15:                                             ; preds = %bb2
-  ret void
-}
diff --git a/polly/test/GPGPU/spir-typesize.ll b/polly/test/GPGPU/spir-typesize.ll
deleted file mode 100644
index fce17c54e6e9a..0000000000000
--- a/polly/test/GPGPU/spir-typesize.ll
+++ /dev/null
@@ -1,90 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg \
-; RUN: -polly-gpu-arch=spir64 \
-; RUN: -polly-acc-dump-kernel-ir -polly-process-unprofitable -disable-output < %s | \
-; RUN: FileCheck -check-prefix=I64 %s
-
-; RUN: opt %loadPolly -polly-codegen-ppcg \
-; RUN: -polly-gpu-arch=spir32 \
-; RUN: -polly-acc-dump-kernel-ir -polly-process-unprofitable -disable-output < %s | \
-; RUN: FileCheck -check-prefix=I32 %s
-
-; REQUIRES: pollyacc
-
-; This test case checks whether the openCl runtime functions (get_local_id/get_group_id) return the right types for 32 and 64bit devices.
-
-; I32:      target datalayout = "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-i128:128:128-f32:32:32-f64:64:64-v16:16:16-v24:32:32-v32:32:32-v48:64:64-v64:64:64-v96:128:128-v128:128:128-v192:256:256-v256:256:256-v512:512:512-v1024:1024:1024"
-; I32-NEXT: target triple = "spir-unknown-unknown"
-
-; I32-LABEL: define spir_kernel void @FUNC_double_parallel_loop_SCOP_0_KERNEL_0(ptr addrspace(1) %MemRef_A) #0 !kernel_arg_addr_space !0 !kernel_arg_name !1 !kernel_arg_access_qual !1 !kernel_arg_type !1 !kernel_arg_type_qual !1 !kernel_arg_base_type !1 {
-; I32-NEXT: entry:
-; I32-NEXT:   %0 = call i32 @__gen_ocl_get_group_id0()
-; I32-NEXT:   %__gen_ocl_get_group_id0 = zext i32 %0 to i64
-; I32-NEXT:   %1 = call i32 @__gen_ocl_get_group_id1()
-; I32-NEXT:   %__gen_ocl_get_group_id1 = zext i32 %1 to i64
-; I32-NEXT:   %2 = call i32 @__gen_ocl_get_local_id0()
-; I32-NEXT:   %__gen_ocl_get_local_id0 = zext i32 %2 to i64
-; I32-NEXT:   %3 = call i32 @__gen_ocl_get_local_id1()
-; I32-NEXT:   %__gen_ocl_get_local_id1 = zext i32 %3 to i64
-; I32-NEXT:   br label %polly.loop_preheader
-
-; I64:       target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-i128:128:128-f32:32:32-f64:64:64-v16:16:16-v24:32:32-v32:32:32-v48:64:64-v64:64:64-v96:128:128-v128:128:128-v192:256:256-v256:256:256-v512:512:512-v1024:1024:1024"
-; I64-next:  target triple = "spir64-unknown-unknown"
-
-; I64-LABEL: define spir_kernel void @FUNC_double_parallel_loop_SCOP_0_KERNEL_0(ptr addrspace(1) %MemRef_A) #0 !kernel_arg_addr_space !0 !kernel_arg_name !1 !kernel_arg_access_qual !1 !kernel_arg_type !1 !kernel_arg_type_qual !1 !kernel_arg_base_type !1 {
-; I64-NEXT: entry:
-; I64-NEXT:   %0 = call i64 @__gen_ocl_get_group_id0()
-; I64-NEXT:   %1 = call i64 @__gen_ocl_get_group_id1()
-; I64-NEXT:   %2 = call i64 @__gen_ocl_get_local_id0()
-; I64-NEXT:   %3 = call i64 @__gen_ocl_get_local_id1()
-; I64-NEXT:   br label %polly.loop_preheader
-
-
-;    void double_parallel_loop(float A[][1024]) {
-;      for (long i = 0; i < 1024; i++)
-;        for (long j = 0; j < 1024; j++)
-;          A[i][j] += i * j;
-;    }
-;
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-
-define void @double_parallel_loop(ptr %A) {
-bb:
-  br label %bb2
-
-bb2:                                              ; preds = %bb13, %bb
-  %i.0 = phi i64 [ 0, %bb ], [ %tmp14, %bb13 ]
-  %exitcond1 = icmp ne i64 %i.0, 1024
-  br i1 %exitcond1, label %bb3, label %bb15
-
-bb3:                                              ; preds = %bb2
-  br label %bb4
-
-bb4:                                              ; preds = %bb10, %bb3
-  %j.0 = phi i64 [ 0, %bb3 ], [ %tmp11, %bb10 ]
-  %exitcond = icmp ne i64 %j.0, 1024
-  br i1 %exitcond, label %bb5, label %bb12
-
-bb5:                                              ; preds = %bb4
-  %tmp = mul nuw nsw i64 %i.0, %j.0
-  %tmp6 = sitofp i64 %tmp to float
-  %tmp7 = getelementptr inbounds [1024 x float], ptr %A, i64 %i.0, i64 %j.0
-  %tmp8 = load float, ptr %tmp7, align 4
-  %tmp9 = fadd float %tmp8, %tmp6
-  store float %tmp9, ptr %tmp7, align 4
-  br label %bb10
-
-bb10:                                             ; preds = %bb5
-  %tmp11 = add nuw nsw i64 %j.0, 1
-  br label %bb4
-
-bb12:                                             ; preds = %bb4
-  br label %bb13
-
-bb13:                                             ; preds = %bb12
-  %tmp14 = add nuw nsw i64 %i.0, 1
-  br label %bb2
-
-bb15:                                             ; preds = %bb2
-  ret void
-}
diff --git a/polly/test/GPGPU/unknown-fn-call-not-copied-into-kernel.ll b/polly/test/GPGPU/unknown-fn-call-not-copied-into-kernel.ll
deleted file mode 100644
index 6fd14cbfbcd1b..0000000000000
--- a/polly/test/GPGPU/unknown-fn-call-not-copied-into-kernel.ll
+++ /dev/null
@@ -1,82 +0,0 @@
-; RUN: opt %loadPolly -polly-print-scops -disable-output < %s | FileCheck %s --check-prefix=SCOP
-; RUN: opt %loadPolly -S -polly-codegen-ppcg < %s | FileCheck %s
-
-; Check that we do not create a kernel if there is an
-; unknown function call in a candidate kernel.
-
-; Check that we model the kernel as a scop.
-; SCOP:      Function: f
-; SCOP-NEXT:     Region: %entry.split---%for.end13
-
-; If a kernel were generated, then this code would have been part of the kernel
-; and not the `.ll` file that is generated.
-; CHECK:       %conv = fpext float %0 to double
-; CHECK-NEXT:  %1 = tail call double @extern.fn(double %conv)
-; CHECK-NEXT:  %conv6 = fptrunc double %1 to float
-
-; REQUIRES: pollyacc
-
-; static const int N = 1000;
-; void f(float A[N][N], int n, float B[N][N]) {
-;   for(int i = 0; i < n; i++) {
-;     for(int j = 0; j < n; j++) {
-;       B[i][j] = extern_fn(A[i][j], 3);
-;     }
-;
-;   }
-; }
-
-target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
-target triple = "x86_64-apple-macosx10.11.0"
-
-define void @f(ptr %A, i32 %n, ptr %B) {
-entry:
-  br label %entry.split
-
-entry.split:                                      ; preds = %entry
-  %cmp3 = icmp sgt i32 %n, 0
-  br i1 %cmp3, label %for.cond1.preheader.lr.ph, label %for.end13
-
-for.cond1.preheader.lr.ph:                        ; preds = %entry.split
-  br label %for.cond1.preheader
-
-for.cond1.preheader:                              ; preds = %for.cond1.preheader.lr.ph, %for.inc11
-  %indvars.iv5 = phi i64 [ 0, %for.cond1.preheader.lr.ph ], [ %indvars.iv.next6, %for.inc11 ]
-  %cmp21 = icmp sgt i32 %n, 0
-  br i1 %cmp21, label %for.body3.lr.ph, label %for.inc11
-
-for.body3.lr.ph:                                  ; preds = %for.cond1.preheader
-  br label %for.body3
-
-for.body3:                                        ; preds = %for.body3.lr.ph, %for.body3
-  %indvars.iv = phi i64 [ 0, %for.body3.lr.ph ], [ %indvars.iv.next, %for.body3 ]
-  %arrayidx5 = getelementptr inbounds [1000 x float], ptr %A, i64 %indvars.iv5, i64 %indvars.iv
-  %0 = load float, ptr %arrayidx5, align 4
-  %conv = fpext float %0 to double
-  %1 = tail call double @extern.fn(double %conv)
-  %conv6 = fptrunc double %1 to float
-  %arrayidx10 = getelementptr inbounds [1000 x float], ptr %B, i64 %indvars.iv5, i64 %indvars.iv
-  store float %conv6, ptr %arrayidx10, align 4
-  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
-  %wide.trip.count = zext i32 %n to i64
-  %exitcond = icmp ne i64 %indvars.iv.next, %wide.trip.count
-  br i1 %exitcond, label %for.body3, label %for.cond1.for.inc11_crit_edge
-
-for.cond1.for.inc11_crit_edge:                    ; preds = %for.body3
-  br label %for.inc11
-
-for.inc11:                                        ; preds = %for.cond1.for.inc11_crit_edge, %for.cond1.preheader
-  %indvars.iv.next6 = add nuw nsw i64 %indvars.iv5, 1
-  %wide.trip.count7 = zext i32 %n to i64
-  %exitcond8 = icmp ne i64 %indvars.iv.next6, %wide.trip.count7
-  br i1 %exitcond8, label %for.cond1.preheader, label %for.cond.for.end13_crit_edge
-
-for.cond.for.end13_crit_edge:                     ; preds = %for.inc11
-  br label %for.end13
-
-for.end13:                                        ; preds = %for.cond.for.end13_crit_edge, %entry.split
-  ret void
-}
-
-declare double @extern.fn(double) #0
-attributes #0 = { readnone }
diff --git a/polly/test/GPGPU/untouched-arrays.ll b/polly/test/GPGPU/untouched-arrays.ll
deleted file mode 100644
index 5c7e0c7b543b9..0000000000000
--- a/polly/test/GPGPU/untouched-arrays.ll
+++ /dev/null
@@ -1,270 +0,0 @@
-; RUN: opt %loadPolly -polly-codegen-ppcg -polly-acc-dump-code \
-; RUN: -disable-output < %s | \
-; RUN: FileCheck -check-prefix=CODE %s
-
-; REQUIRES: pollyacc
-
-; CODE:        cudaCheckReturn(cudaMemcpy(dev_MemRef_global_1, MemRef_global_1, (142) * sizeof(i32), cudaMemcpyHostToDevice));
-; CODE-NEXT:   {
-; CODE-NEXT:     dim3 k0_dimBlock(10);
-; CODE-NEXT:     dim3 k0_dimGrid(1);
-; CODE-NEXT:     kernel0 <<<k0_dimGrid, k0_dimBlock>>> (dev_MemRef_global_1);
-; CODE-NEXT:     cudaCheckKernel();
-; CODE-NEXT:   }
-
-; CODE:   cudaCheckReturn(cudaMemcpy(MemRef_global_1, dev_MemRef_global_1, (142) * sizeof(i32), cudaMemcpyDeviceToHost));
-; CODE:   cudaCheckReturn(cudaFree(dev_MemRef_global_1));
-; CODE-NEXT: }
-
-; CODE: # kernel0
-; CODE-NEXT: Stmt_bb33(t0, 0);
-
-
-target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
-target triple = "x86_64-unknown-linux-gnu"
-
-%struct.hoge = type { [23 x i16], [22 x i16], [14 x i16], [13 x i16] }
-
-@global = external global [9 x %struct.hoge], align 16
-@global.1 = external global [9 x [152 x i32]], align 16
-
-; Function Attrs: nounwind uwtable
-define void @widget() #0 {
-bb:
-  br label %bb1
-
-bb1:                                              ; preds = %bb1, %bb
-  br i1 undef, label %bb1, label %bb2
-
-bb2:                                              ; preds = %bb2, %bb1
-  br i1 undef, label %bb2, label %bb3
-
-bb3:                                              ; preds = %bb3, %bb2
-  br i1 undef, label %bb3, label %bb4
-
-bb4:                                              ; preds = %bb4, %bb3
-  br i1 undef, label %bb4, label %bb5
-
-bb5:                                              ; preds = %bb5, %bb4
-  br i1 undef, label %bb5, label %bb6
-
-bb6:                                              ; preds = %bb6, %bb5
-  br i1 undef, label %bb6, label %bb7
-
-bb7:                                              ; preds = %bb7, %bb6
-  br i1 undef, label %bb7, label %bb8
-
-bb8:                                              ; preds = %bb8, %bb7
-  br i1 undef, label %bb8, label %bb9
-
-bb9:                                              ; preds = %bb8
-  br label %bb10
-
-bb10:                                             ; preds = %bb12, %bb9
-  br label %bb11
-
-bb11:                                             ; preds = %bb11, %bb10
-  br i1 undef, label %bb11, label %bb12
-
-bb12:                                             ; preds = %bb11
-  br i1 undef, label %bb10, label %bb13
-
-bb13:                                             ; preds = %bb18, %bb12
-  br i1 undef, label %bb16, label %bb14
-
-bb14:                                             ; preds = %bb16, %bb13
-  br i1 undef, label %bb15, label %bb18
-
-bb15:                                             ; preds = %bb14
-  br label %bb17
-
-bb16:                                             ; preds = %bb16, %bb13
-  br i1 undef, label %bb16, label %bb14
-
-bb17:                                             ; preds = %bb17, %bb15
-  br i1 undef, label %bb17, label %bb18
-
-bb18:                                             ; preds = %bb17, %bb14
-  br i1 undef, label %bb13, label %bb19
-
-bb19:                                             ; preds = %bb25, %bb18
-  br label %bb20
-
-bb20:                                             ; preds = %bb24, %bb19
-  br i1 undef, label %bb21, label %bb24
-
-bb21:                                             ; preds = %bb20
-  br i1 undef, label %bb23, label %bb22
-
-bb22:                                             ; preds = %bb21
-  br label %bb24
-
-bb23:                                             ; preds = %bb21
-  br label %bb24
-
-bb24:                                             ; preds = %bb23, %bb22, %bb20
-  br i1 undef, label %bb20, label %bb25
-
-bb25:                                             ; preds = %bb24
-  br i1 undef, label %bb19, label %bb26
-
-bb26:                                             ; preds = %bb56, %bb25
-  %tmp = phi ptr [ undef, %bb56 ], [ getelementptr inbounds ([9 x [152 x i32]], ptr @global.1, i64 0, i64 0, i64 32), %bb25 ]
-  br label %bb27
-
-bb27:                                             ; preds = %bb27, %bb26
-  br i1 undef, label %bb27, label %bb28
-
-bb28:                                             ; preds = %bb27
-  br label %bb30
-
-bb30:                                             ; preds = %bb38, %bb28
-  %tmp31 = phi i32 [ 3, %bb28 ], [ %tmp40, %bb38 ]
-  %tmp32 = phi ptr [ %tmp, %bb28 ], [ %tmp39, %bb38 ]
-  br label %bb33
-
-bb33:                                             ; preds = %bb33, %bb30
-  %tmp34 = phi i32 [ 0, %bb30 ], [ %tmp37, %bb33 ]
-  %tmp35 = phi ptr [ %tmp32, %bb30 ], [ undef, %bb33 ]
-  %tmp36 = getelementptr inbounds i32, ptr %tmp35, i64 1
-  store i32 undef, ptr %tmp36, align 4, !tbaa !1
-  %tmp37 = add nuw nsw i32 %tmp34, 1
-  br i1 false, label %bb33, label %bb38
-
-bb38:                                             ; preds = %bb33
-  %tmp39 = getelementptr i32, ptr %tmp32, i64 12
-  %tmp40 = add nuw nsw i32 %tmp31, 1
-  %tmp41 = icmp ne i32 %tmp40, 13
-  br i1 %tmp41, label %bb30, label %bb42
-
-bb42:                                             ; preds = %bb38
-  %tmp43 = getelementptr inbounds [9 x %struct.hoge], ptr @global, i64 0, i64 0, i32 3, i64 0
-  br label %bb44
-
-bb44:                                             ; preds = %bb51, %bb42
-  %tmp45 = phi i32 [ 0, %bb42 ], [ %tmp52, %bb51 ]
-  %tmp46 = phi ptr [ %tmp43, %bb42 ], [ undef, %bb51 ]
-  %tmp47 = load i16, ptr %tmp46, align 2, !tbaa !5
-  br label %bb48
-
-bb48:                                             ; preds = %bb48, %bb44
-  %tmp49 = phi i32 [ 0, %bb44 ], [ %tmp50, %bb48 ]
-  %tmp50 = add nuw nsw i32 %tmp49, 1
-  br i1 false, label %bb48, label %bb51
-
-bb51:                                             ; preds = %bb48
-  %tmp52 = add nuw nsw i32 %tmp45, 1
-  %tmp53 = icmp ne i32 %tmp52, 13
-  br i1 %tmp53, label %bb44, label %bb54
-
-bb54:                                             ; preds = %bb51
-  br label %bb55
-
-bb55:                                             ; preds = %bb55, %bb54
-  br i1 undef, label %bb55, label %bb56
-
-bb56:                                             ; preds = %bb55
-  br i1 undef, label %bb26, label %bb57
-
-bb57:                                             ; preds = %bb60, %bb56
-  br label %bb58
-
-bb58:                                             ; preds = %bb58, %bb57
-  br i1 undef, label %bb58, label %bb59
-
-bb59:                                             ; preds = %bb59, %bb58
-  br i1 undef, label %bb59, label %bb60
-
-bb60:                                             ; preds = %bb59
-  br i1 undef, label %bb57, label %bb61
-
-bb61:                                             ; preds = %bb65, %bb60
-  br label %bb62
-
-bb62:                                             ; preds = %bb64, %bb61
-  br label %bb63
-
-bb63:                                             ; preds = %bb63, %bb62
-  br i1 undef, label %bb63, label %bb64
-
-bb64:                                             ; preds = %bb63
-  br i1 undef, label %bb62, label %bb65
-
-bb65:                                             ; preds = %bb64
-  br i1 undef, label %bb61, label %bb66
-
-bb66:                                             ; preds = %bb70, %bb65
-  br label %bb67
-
-bb67:                                             ; preds = %bb69, %bb66
-  br label %bb68
-
-bb68:                                             ; preds = %bb68, %bb67
-  br i1 undef, label %bb68, label %bb69
-
-bb69:                                             ; preds = %bb68
-  br i1 undef, label %bb67, label %bb70
-
-bb70:                                             ; preds = %bb69
-  br i1 undef, label %bb66, label %bb71
-
-bb71:                                             ; preds = %bb73, %bb70
-  br label %bb72
-
-bb72:                                             ; preds = %bb72, %bb71
-  br i1 undef, label %bb72, label %bb73
-
-bb73:                                             ; preds = %bb72
-  br i1 undef, label %bb71, label %bb74
-
-bb74:                                             ; preds = %bb80, %bb73
-  br label %bb75
-
-bb75:                                             ; preds = %bb79, %bb74
-  br label %bb76
-
-bb76:                                             ; preds = %bb78, %bb75
-  br label %bb77
-
-bb77:                                             ; preds = %bb77, %bb76
-  br i1 undef, label %bb77, label %bb78
-
-bb78:                                             ; preds = %bb77
-  br i1 undef, label %bb76, label %bb79
-
-bb79:                                             ; preds = %bb78
-  br i1 undef, label %bb75, label %bb80
-
-bb80:                                             ; preds = %bb79
-  br i1 undef, label %bb74, label %bb81
-
-bb81:                                             ; preds = %bb85, %bb80
-  br label %bb82
-
-bb82:                                             ; preds = %bb84, %bb81
-  br label %bb83
-
-bb83:                                             ; preds = %bb83, %bb82
-  br i1 undef, label %bb83, label %bb84
-
-bb84:                                             ; preds = %bb83
-  br i1 undef, label %bb82, label %bb85
-
-bb85:                                             ; preds = %bb84
-  br i1 undef, label %bb81, label %bb86
-
-bb86:                                             ; preds = %bb85
-  ret void
-}
-
-attributes #0 = { nounwind uwtable "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "less-precise-fpmad"="false" "frame-pointer"="none" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+fxsr,+mmx,+sse,+sse2,+x87" "unsafe-fp-math"="false" "use-soft-float"="false" }
-
-!llvm.ident = !{!0}
-
-!0 = !{!"clang version 4.0.0"}
-!1 = !{!2, !2, i64 0}
-!2 = !{!"int", !3, i64 0}
-!3 = !{!"omnipotent char", !4, i64 0}
-!4 = !{!"Simple C/C++ TBAA"}
-!5 = !{!6, !6, i64 0}
-!6 = !{!"short", !3, i64 0}
diff --git a/polly/test/ScheduleOptimizer/2012-04-16-Trivially-vectorizable-loops.ll b/polly/test/ScheduleOptimizer/2012-04-16-Trivially-vectorizable-loops.ll
deleted file mode 100644
index 8379da290769d..0000000000000
--- a/polly/test/ScheduleOptimizer/2012-04-16-Trivially-vectorizable-loops.ll
+++ /dev/null
@@ -1,204 +0,0 @@
-; RUN: opt %loadPolly -basic-aa -polly-opt-isl -polly-vectorizer=polly -disable-output < %s
-target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
-
-%struct._IO_FILE = type { i32, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, i32, i32, i64, i16, i8, [1 x i8], ptr, i64, ptr, ptr, ptr, ptr, i64, i32, [20 x i8] }
-%struct._IO_marker = type { ptr, ptr, i32 }
-
-@A = common global [1536 x [1536 x float]] zeroinitializer, align 16
-@B = common global [1536 x [1536 x float]] zeroinitializer, align 16
-@stdout = external global ptr
-@.str = private unnamed_addr constant [5 x i8] c"%lf \00", align 1
-@C = common global [1536 x [1536 x float]] zeroinitializer, align 16
-@.str1 = private unnamed_addr constant [2 x i8] c"\0A\00", align 1
-
-define void @init_array() nounwind uwtable {
-entry:
-  br label %for.cond
-
-for.cond:                                         ; preds = %for.inc17, %entry
-  %i.0 = phi i32 [ 0, %entry ], [ %inc18, %for.inc17 ]
-  %cmp = icmp slt i32 %i.0, 1536
-  br i1 %cmp, label %for.body, label %for.end19
-
-for.body:                                         ; preds = %for.cond
-  br label %for.cond1
-
-for.cond1:                                        ; preds = %for.inc, %for.body
-  %j.0 = phi i32 [ 0, %for.body ], [ %inc, %for.inc ]
-  %cmp2 = icmp slt i32 %j.0, 1536
-  br i1 %cmp2, label %for.body3, label %for.end
-
-for.body3:                                        ; preds = %for.cond1
-  %mul = mul nsw i32 %i.0, %j.0
-  %rem = srem i32 %mul, 1024
-  %add = add nsw i32 1, %rem
-  %conv = sitofp i32 %add to double
-  %div = fdiv double %conv, 2.000000e+00
-  %conv4 = fptrunc double %div to float
-  %idxprom = sext i32 %j.0 to i64
-  %idxprom5 = sext i32 %i.0 to i64
-  %arrayidx = getelementptr inbounds [1536 x [1536 x float]], ptr @A, i32 0, i64 %idxprom5
-  %arrayidx6 = getelementptr inbounds [1536 x float], ptr %arrayidx, i32 0, i64 %idxprom
-  store float %conv4, ptr %arrayidx6, align 4
-  %mul7 = mul nsw i32 %i.0, %j.0
-  %rem8 = srem i32 %mul7, 1024
-  %add9 = add nsw i32 1, %rem8
-  %conv10 = sitofp i32 %add9 to double
-  %div11 = fdiv double %conv10, 2.000000e+00
-  %conv12 = fptrunc double %div11 to float
-  %idxprom13 = sext i32 %j.0 to i64
-  %idxprom14 = sext i32 %i.0 to i64
-  %arrayidx15 = getelementptr inbounds [1536 x [1536 x float]], ptr @B, i32 0, i64 %idxprom14
-  %arrayidx16 = getelementptr inbounds [1536 x float], ptr %arrayidx15, i32 0, i64 %idxprom13
-  store float %conv12, ptr %arrayidx16, align 4
-  br label %for.inc
-
-for.inc:                                          ; preds = %for.body3
-  %inc = add nsw i32 %j.0, 1
-  br label %for.cond1
-
-for.end:                                          ; preds = %for.cond1
-  br label %for.inc17
-
-for.inc17:                                        ; preds = %for.end
-  %inc18 = add nsw i32 %i.0, 1
-  br label %for.cond
-
-for.end19:                                        ; preds = %for.cond
-  ret void
-}
-
-define void @print_array() nounwind uwtable {
-entry:
-  br label %for.cond
-
-for.cond:                                         ; preds = %for.inc10, %entry
-  %i.0 = phi i32 [ 0, %entry ], [ %inc11, %for.inc10 ]
-  %cmp = icmp slt i32 %i.0, 1536
-  br i1 %cmp, label %for.body, label %for.end12
-
-for.body:                                         ; preds = %for.cond
-  br label %for.cond1
-
-for.cond1:                                        ; preds = %for.inc, %for.body
-  %j.0 = phi i32 [ 0, %for.body ], [ %inc, %for.inc ]
-  %cmp2 = icmp slt i32 %j.0, 1536
-  br i1 %cmp2, label %for.body3, label %for.end
-
-for.body3:                                        ; preds = %for.cond1
-  %0 = load ptr, ptr @stdout, align 8
-  %idxprom = sext i32 %j.0 to i64
-  %idxprom4 = sext i32 %i.0 to i64
-  %arrayidx = getelementptr inbounds [1536 x [1536 x float]], ptr @C, i32 0, i64 %idxprom4
-  %arrayidx5 = getelementptr inbounds [1536 x float], ptr %arrayidx, i32 0, i64 %idxprom
-  %1 = load float, ptr %arrayidx5, align 4
-  %conv = fpext float %1 to double
-  %call = call i32 (ptr, ptr, ...) @fprintf(ptr %0, ptr @.str, double %conv)
-  %rem = srem i32 %j.0, 80
-  %cmp6 = icmp eq i32 %rem, 79
-  br i1 %cmp6, label %if.then, label %if.end
-
-if.then:                                          ; preds = %for.body3
-  %2 = load ptr, ptr @stdout, align 8
-  %call8 = call i32 (ptr, ptr, ...) @fprintf(ptr %2, ptr @.str1)
-  br label %if.end
-
-if.end:                                           ; preds = %if.then, %for.body3
-  br label %for.inc
-
-for.inc:                                          ; preds = %if.end
-  %inc = add nsw i32 %j.0, 1
-  br label %for.cond1
-
-for.end:                                          ; preds = %for.cond1
-  %3 = load ptr, ptr @stdout, align 8
-  %call9 = call i32 (ptr, ptr, ...) @fprintf(ptr %3, ptr @.str1)
-  br label %for.inc10
-
-for.inc10:                                        ; preds = %for.end
-  %inc11 = add nsw i32 %i.0, 1
-  br label %for.cond
-
-for.end12:                                        ; preds = %for.cond
-  ret void
-}
-
-declare i32 @fprintf(ptr, ptr, ...)
-
-define i32 @main() nounwind uwtable {
-entry:
-  call void @init_array()
-  br label %for.cond
-
-for.cond:                                         ; preds = %for.inc28, %entry
-  %i.0 = phi i32 [ 0, %entry ], [ %inc29, %for.inc28 ]
-  %cmp = icmp slt i32 %i.0, 1536
-  br i1 %cmp, label %for.body, label %for.end30
-
-for.body:                                         ; preds = %for.cond
-  br label %for.cond1
-
-for.cond1:                                        ; preds = %for.inc25, %for.body
-  %j.0 = phi i32 [ 0, %for.body ], [ %inc26, %for.inc25 ]
-  %cmp2 = icmp slt i32 %j.0, 1536
-  br i1 %cmp2, label %for.body3, label %for.end27
-
-for.body3:                                        ; preds = %for.cond1
-  %idxprom = sext i32 %j.0 to i64
-  %idxprom4 = sext i32 %i.0 to i64
-  %arrayidx = getelementptr inbounds [1536 x [1536 x float]], ptr @C, i32 0, i64 %idxprom4
-  %arrayidx5 = getelementptr inbounds [1536 x float], ptr %arrayidx, i32 0, i64 %idxprom
-  store float 0.000000e+00, ptr %arrayidx5, align 4
-  br label %for.cond6
-
-for.cond6:                                        ; preds = %for.inc, %for.body3
-  %k.0 = phi i32 [ 0, %for.body3 ], [ %inc, %for.inc ]
-  %cmp7 = icmp slt i32 %k.0, 1536
-  br i1 %cmp7, label %for.body8, label %for.end
-
-for.body8:                                        ; preds = %for.cond6
-  %idxprom9 = sext i32 %j.0 to i64
-  %idxprom10 = sext i32 %i.0 to i64
-  %arrayidx11 = getelementptr inbounds [1536 x [1536 x float]], ptr @C, i32 0, i64 %idxprom10
-  %arrayidx12 = getelementptr inbounds [1536 x float], ptr %arrayidx11, i32 0, i64 %idxprom9
-  %0 = load float, ptr %arrayidx12, align 4
-  %idxprom13 = sext i32 %k.0 to i64
-  %idxprom14 = sext i32 %i.0 to i64
-  %arrayidx15 = getelementptr inbounds [1536 x [1536 x float]], ptr @A, i32 0, i64 %idxprom14
-  %arrayidx16 = getelementptr inbounds [1536 x float], ptr %arrayidx15, i32 0, i64 %idxprom13
-  %1 = load float, ptr %arrayidx16, align 4
-  %idxprom17 = sext i32 %j.0 to i64
-  %idxprom18 = sext i32 %k.0 to i64
-  %arrayidx19 = getelementptr inbounds [1536 x [1536 x float]], ptr @B, i32 0, i64 %idxprom18
-  %arrayidx20 = getelementptr inbounds [1536 x float], ptr %arrayidx19, i32 0, i64 %idxprom17
-  %2 = load float, ptr %arrayidx20, align 4
-  %mul = fmul float %1, %2
-  %add = fadd float %0, %mul
-  %idxprom21 = sext i32 %j.0 to i64
-  %idxprom22 = sext i32 %i.0 to i64
-  %arrayidx23 = getelementptr inbounds [1536 x [1536 x float]], ptr @C, i32 0, i64 %idxprom22
-  %arrayidx24 = getelementptr inbounds [1536 x float], ptr %arrayidx23, i32 0, i64 %idxprom21
-  store float %add, ptr %arrayidx24, align 4
-  br label %for.inc
-
-for.inc:                                          ; preds = %for.body8
-  %inc = add nsw i32 %k.0, 1
-  br label %for.cond6
-
-for.end:                                          ; preds = %for.cond6
-  br label %for.inc25
-
-for.inc25:                                        ; preds = %for.end
-  %inc26 = add nsw i32 %j.0, 1
-  br label %for.cond1
-
-for.end27:                                        ; preds = %for.cond1
-  br label %for.inc28
-
-for.inc28:                                        ; preds = %for.end27
-  %inc29 = add nsw i32 %i.0, 1
-  br label %for.cond
-
-for.end30:                                        ; preds = %for.cond
-  ret i32 0
-}
diff --git a/polly/test/ScheduleOptimizer/prevectorization-without-tiling.ll b/polly/test/ScheduleOptimizer/prevectorization-without-tiling.ll
index 9089d45c1320e..fea2155b1e4ed 100644
--- a/polly/test/ScheduleOptimizer/prevectorization-without-tiling.ll
+++ b/polly/test/ScheduleOptimizer/prevectorization-without-tiling.ll
@@ -1,4 +1,4 @@
-; RUN: opt -S %loadPolly -basic-aa -polly-tiling=false -polly-pattern-matching-based-opts=false -polly-vectorizer=polly -polly-opt-isl -polly-print-ast -disable-output < %s | FileCheck %s
+; RUN: opt -S %loadPolly -basic-aa -polly-tiling=false -polly-pattern-matching-based-opts=false -polly-vectorizer=stripmine -polly-opt-isl -polly-print-ast -disable-output < %s | FileCheck %s
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
 @C = common global [1536 x [1536 x float]] zeroinitializer, align 16
diff --git a/polly/test/ScheduleOptimizer/prevectorization.ll b/polly/test/ScheduleOptimizer/prevectorization.ll
index 55943d6daf736..385ebf14712ab 100644
--- a/polly/test/ScheduleOptimizer/prevectorization.ll
+++ b/polly/test/ScheduleOptimizer/prevectorization.ll
@@ -1,6 +1,5 @@
-; RUN: opt -S %loadPolly -basic-aa -polly-pattern-matching-based-opts=false -polly-vectorizer=polly                         -polly-opt-isl -polly-print-ast -disable-output < %s | FileCheck %s
-; RUN: opt -S %loadPolly -basic-aa -polly-pattern-matching-based-opts=false -polly-vectorizer=stripmine                     -polly-opt-isl -polly-print-ast -disable-output < %s | FileCheck %s
-; RUN: opt -S %loadPolly -basic-aa -polly-pattern-matching-based-opts=false -polly-vectorizer=polly -polly-prevect-width=16 -polly-opt-isl -polly-print-ast -disable-output < %s | FileCheck %s -check-prefix=VEC16
+; RUN: opt -S %loadPolly -basic-aa -polly-pattern-matching-based-opts=false -polly-vectorizer=stripmine                         -polly-opt-isl -polly-print-ast -disable-output < %s | FileCheck %s
+; RUN: opt -S %loadPolly -basic-aa -polly-pattern-matching-based-opts=false -polly-vectorizer=stripmine -polly-prevect-width=16 -polly-opt-isl -polly-print-ast -disable-output < %s | FileCheck %s -check-prefix=VEC16
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
diff --git a/polly/test/ScheduleOptimizer/rectangular-tiling.ll b/polly/test/ScheduleOptimizer/rectangular-tiling.ll
index c254e4f367c4d..b527255ab5f7b 100644
--- a/polly/test/ScheduleOptimizer/rectangular-tiling.ll
+++ b/polly/test/ScheduleOptimizer/rectangular-tiling.ll
@@ -2,7 +2,6 @@
 ; RUN: opt %loadPolly -polly-tile-sizes=256,16 -polly-tiling=false                                                                                                                    -polly-opt-isl -polly-print-ast -disable-output < %s | FileCheck %s --check-prefix=NOTILING
 ; RUN: opt %loadPolly -polly-tile-sizes=256,16 -polly-2nd-level-tiling -polly-2nd-level-tile-sizes=16,8                                                                               -polly-opt-isl -polly-print-ast -disable-output < %s | FileCheck %s --check-prefix=TWOLEVEL
 ; RUN: opt %loadPolly -polly-tile-sizes=256,16 -polly-2nd-level-tiling -polly-2nd-level-tile-sizes=16,8 -polly-register-tiling                                                        -polly-opt-isl -polly-print-ast -disable-output < %s | FileCheck %s --check-prefix=TWO-PLUS-REGISTER
-; RUN: opt %loadPolly -polly-tile-sizes=256,16 -polly-2nd-level-tiling -polly-2nd-level-tile-sizes=16,8 -polly-register-tiling -polly-register-tile-sizes=2,4 -polly-vectorizer=polly -polly-opt-isl -polly-print-ast -disable-output < %s | FileCheck %s --check-prefix=TWO-PLUS-REGISTER-PLUS-VECTORIZATION
 
 ; CHECK: // 1st level tiling - Tiles
 ; CHECK: for (int c0 = 0; c0 <= 3; c0 += 1)
@@ -49,21 +48,6 @@
 ; TWO-PLUS-REGISTER:               Stmt_for_body3(256 * c0 + 16 * c2 + 2 * c4 + 1, 16 * c1 + 8 * c3 + 2 * c5 + 1);
 ; TWO-PLUS-REGISTER:             }
 
-; TWO-PLUS-REGISTER-PLUS-VECTORIZATION: #pragma known-parallel
-; TWO-PLUS-REGISTER-PLUS-VECTORIZATION: for (int c0 = 0; c0 <= 3; c0 += 1)
-; TWO-PLUS-REGISTER-PLUS-VECTORIZATION:   for (int c1 = 0; c1 <= 31; c1 += 1)
-; TWO-PLUS-REGISTER-PLUS-VECTORIZATION:     for (int c2 = 0; c2 <= 15; c2 += 1)
-; TWO-PLUS-REGISTER-PLUS-VECTORIZATION:       for (int c3 = 0; c3 <= 1; c3 += 1)
-; TWO-PLUS-REGISTER-PLUS-VECTORIZATION:         for (int c4 = 0; c4 <= 7; c4 += 1)
-; TWO-PLUS-REGISTER-PLUS-VECTORIZATION:           for (int c5 = 0; c5 <= 1; c5 += 1) {
-; TWO-PLUS-REGISTER-PLUS-VECTORIZATION:             // SIMD
-; TWO-PLUS-REGISTER-PLUS-VECTORIZATION:             for (int c8 = 0; c8 <= 3; c8 += 1)
-; TWO-PLUS-REGISTER-PLUS-VECTORIZATION:               Stmt_for_body3(256 * c0 + 16 * c2 + 2 * c4, 16 * c1 + 8 * c3 + 4 * c5 + c8);
-; TWO-PLUS-REGISTER-PLUS-VECTORIZATION:             // SIMD
-; TWO-PLUS-REGISTER-PLUS-VECTORIZATION:             for (int c8 = 0; c8 <= 3; c8 += 1)
-; TWO-PLUS-REGISTER-PLUS-VECTORIZATION:               Stmt_for_body3(256 * c0 + 16 * c2 + 2 * c4 + 1, 16 * c1 + 8 * c3 + 4 * c5 + c8);
-; TWO-PLUS-REGISTER-PLUS-VECTORIZATION:           }
-
 target datalayout = "e-m:e-p:32:32-i64:64-v128:64:128-n32-S64"
 
 ; Function Attrs: nounwind
diff --git a/polly/test/ScheduleOptimizer/vec-addr-space.ll b/polly/test/ScheduleOptimizer/vec-addr-space.ll
deleted file mode 100644
index 563c37f256ca0..0000000000000
--- a/polly/test/ScheduleOptimizer/vec-addr-space.ll
+++ /dev/null
@@ -1,85 +0,0 @@
-; RUN: opt %loadPolly -polly-vectorizer=polly -polly-opt-isl -polly-codegen -S < %s | FileCheck %s
-;
-; Polly crashed during codegen with an assertion error while trying to generate
-; a pointer bitcast from a pointer having an address space to one without
-;
-; CHECK-LABEL: entry:
-; CHECK: load <4 x float>, ptr addrspace(4)
-;
-; ModuleID = '/tmp/lud.bc'
-source_filename = "lud.c"
-; This datalayout was for a 32-bit ARC processor with 512-bit vector extension
-target datalayout = "e-m:e-p:32:32-p1:32:32-p3:32:32-p5:32:32-i64:32-f64:32-v64:32-v128:32-a:0:32-v256:32-v512:32-n8:16:32"
-; Specify x86 because the ARC backend is still experimental and not built by default
-target triple = "x86_64-unknown-unknown"
-
-; Function Attrs: noinline nounwind
-define void @LU_decomp_kij_opt(i32 %n, i32 %lda, ptr addrspace(4) %A, ptr addrspace(4) %scratch) #0 {
-entry:
-  %cmp34 = icmp sgt i32 %n, 0
-  br i1 %cmp34, label %for.body.lr.ph, label %for.end34
-
-for.body.lr.ph:                                   ; preds = %entry
-  %0 = add nsw i32 %n, -1
-  br label %for.body
-
-for.body:                                         ; preds = %for.inc32, %for.body.lr.ph
-  %k.035 = phi i32 [ 0, %for.body.lr.ph ], [ %add2, %for.inc32 ]
-  %mul = mul nsw i32 %k.035, %lda
-  %add = add nsw i32 %mul, %k.035
-  %arrayidx = getelementptr inbounds float, ptr addrspace(4) %A, i32 %add
-  %1 = load float, ptr addrspace(4) %arrayidx, align 4
-  %conv1 = fdiv arcp float 1.000000e+00, %1
-  %add2 = add nuw nsw i32 %k.035, 1
-  %exitcond37 = icmp eq i32 %k.035, %0
-  br i1 %exitcond37, label %for.end34, label %for.body6.lr.ph
-
-for.body6.lr.ph:                                  ; preds = %for.body
-  br label %for.body6
-
-for.body6:                                        ; preds = %for.inc29, %for.body6.lr.ph
-  %i.033 = phi i32 [ %add2, %for.body6.lr.ph ], [ %inc30, %for.inc29 ]
-  %mul7 = mul nsw i32 %i.033, %lda
-  %add8 = add nsw i32 %mul7, %k.035
-  %arrayidx9 = getelementptr inbounds float, ptr addrspace(4) %A, i32 %add8
-  %2 = load float, ptr addrspace(4) %arrayidx9, align 4
-  %mul10 = fmul arcp contract float %conv1, %2
-  store float %mul10, ptr addrspace(4) %arrayidx9, align 4
-  br label %for.body18
-
-for.body18:                                       ; preds = %for.body18, %for.body6
-  %j.031 = phi i32 [ %add2, %for.body6 ], [ %inc, %for.body18 ]
-  %3 = load float, ptr addrspace(4) %arrayidx9, align 4
-  %add23 = add nsw i32 %j.031, %mul
-  %arrayidx24 = getelementptr inbounds float, ptr addrspace(4) %A, i32 %add23
-  %4 = load float, ptr addrspace(4) %arrayidx24, align 4
-  %mul25 = fmul arcp contract float %3, %4
-  %add27 = add nsw i32 %j.031, %mul7
-  %arrayidx28 = getelementptr inbounds float, ptr addrspace(4) %A, i32 %add27
-  %5 = load float, ptr addrspace(4) %arrayidx28, align 4
-  %sub = fsub arcp contract float %5, %mul25
-  store float %sub, ptr addrspace(4) %arrayidx28, align 4
-  %inc = add nuw nsw i32 %j.031, 1
-  %exitcond = icmp eq i32 %inc, %n
-  br i1 %exitcond, label %for.inc29, label %for.body18
-
-for.inc29:                                        ; preds = %for.body18
-  %inc30 = add nuw nsw i32 %i.033, 1
-  %exitcond36 = icmp eq i32 %inc30, %n
-  br i1 %exitcond36, label %for.inc32, label %for.body6
-
-for.inc32:                                        ; preds = %for.inc29
-  br label %for.body
-
-for.end34:                                        ; preds = %for.body, %entry
-  ret void
-}
-
-attributes #0 = { noinline nounwind "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "frame-pointer"="none" "less-precise-fpmad"="false" "min-legal-vector-width"="0" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "unsafe-fp-math"="true" "use-soft-float"="false" }
-
-!llvm.module.flags = !{!0, !1}
-!llvm.ident = !{!2}
-
-!0 = !{i32 1, !"ArcIntrinsicCheck", i32 18224056}
-!1 = !{i32 1, !"wchar_size", i32 2}
-!2 = !{!"clang version 10.0.1 "}
diff --git a/polly/test/ScopDetect/base_pointer_load_is_inst_inside_invariant_1___%for.i---%exit.jscop b/polly/test/ScopDetect/base_pointer_load_is_inst_inside_invariant_1___%for.i---%exit.jscop
index bcc3c301cd814..f8c49fcb1e4f3 100644
--- a/polly/test/ScopDetect/base_pointer_load_is_inst_inside_invariant_1___%for.i---%exit.jscop
+++ b/polly/test/ScopDetect/base_pointer_load_is_inst_inside_invariant_1___%for.i---%exit.jscop
@@ -3,7 +3,7 @@
       {
          "name" : "MemRef_A",
          "sizes" : [ "*" ],
-         "type" : "float*"
+         "type" : "ptr"
       },
       {
          "name" : "MemRef_ptr",
diff --git a/polly/test/ScopDetect/base_pointer_load_setNewAccessRelation.ll b/polly/test/ScopDetect/base_pointer_load_setNewAccessRelation.ll
index 7d88bbc1d3908..0411aed6ae045 100644
--- a/polly/test/ScopDetect/base_pointer_load_setNewAccessRelation.ll
+++ b/polly/test/ScopDetect/base_pointer_load_setNewAccessRelation.ll
@@ -1,4 +1,4 @@
-; RUN: opt -opaque-pointers=0 %loadPolly -polly-ignore-aliasing -polly-invariant-load-hoisting=true -polly-scops -polly-print-import-jscop -polly-codegen -disable-output < %s | FileCheck %s
+; RUN: opt %loadPolly -polly-ignore-aliasing -polly-invariant-load-hoisting=true -polly-scops -polly-print-import-jscop -polly-codegen -disable-output < %s | FileCheck %s
 ;
 ; This violated an assertion in setNewAccessRelation that assumed base pointers
 ; to be load-hoisted. Without this assertion, it codegen would generate invalid
@@ -6,7 +6,7 @@
 ;
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128"
 
-define void @base_pointer_load_is_inst_inside_invariant_1(i64 %n, float** %A) {
+define void @base_pointer_load_is_inst_inside_invariant_1(i64 %n, ptr %A) {
 entry:
   br label %for.i
 
@@ -15,10 +15,10 @@ for.i:
   br label %S1
 
 S1:
-  %ptr = load float*, float** %A
+  %ptr = load ptr, ptr %A
   %conv = sitofp i64 %indvar.i to float
-  %arrayidx5 = getelementptr float, float* %ptr, i64 %indvar.i
-  store float %conv, float* %arrayidx5, align 4
+  %arrayidx5 = getelementptr float, ptr %ptr, i64 %indvar.i
+  store float %conv, ptr %arrayidx5, align 4
   br label %for.i.inc
 
 for.i.inc:
diff --git a/polly/test/ScopInfo/20111108-Parameter-not-detected.ll b/polly/test/ScopInfo/20111108-Parameter-not-detected.ll
index f295c8a57dcac..f80177cb90e77 100644
--- a/polly/test/ScopInfo/20111108-Parameter-not-detected.ll
+++ b/polly/test/ScopInfo/20111108-Parameter-not-detected.ll
@@ -50,7 +50,7 @@ for.end7:                                         ; preds = %for.cond
   ret i32 0
 }
 
-; CHECK:      p0: {0,+,1}<%for.cond>
+; CHECK:      p0: {0,+,1}<nuw><nsw><%for.cond>
 ;
 ; CHECK:      Statements {
 ; CHECK-NEXT:     Stmt_if_then
diff --git a/polly/test/ScopInfo/int2ptr_ptr2int.ll b/polly/test/ScopInfo/int2ptr_ptr2int.ll
index 7cd6ec4c239fe..2e7bfbe5eaee3 100644
--- a/polly/test/ScopInfo/int2ptr_ptr2int.ll
+++ b/polly/test/ScopInfo/int2ptr_ptr2int.ll
@@ -25,9 +25,9 @@
 ; IR-NEXT:   %p_tmp2 = ptrtoint ptr %p_add.ptr2 to i64
 ; IR-NEXT:   %p_arrayidx = getelementptr inbounds i64, ptr %A, i64 %p_tmp2
 ; IR-NEXT:   %tmp3_p_scalar_ = load i64, ptr %p_arrayidx, align 8, !alias.scope !0, !noalias !3
-; IR-NEXT:   %tmp4_p_scalar_ = load i64, ptr %uglygep, align 8, !alias.scope !0, !noalias !3
+; IR-NEXT:   %tmp4_p_scalar_ = load i64, ptr %scevgep, align 8, !alias.scope !0, !noalias !3
 ; IR-NEXT:   %p_add4 = add nsw i64 %tmp4_p_scalar_, %tmp3_p_scalar_
-; IR-NEXT:   store i64 %p_add4, ptr %uglygep, align 8, !alias.scope !0, !noalias !3
+; IR-NEXT:   store i64 %p_add4, ptr %scevgep, align 8, !alias.scope !0, !noalias !3
 ; IR-NEXT:   %polly.indvar_next = add nsw i64 %polly.indvar, 1
 ; IR-NEXT:   %polly.loop_cond = icmp sle i64 %polly.indvar_next, 99
 ; IR-NEXT:   br i1 %polly.loop_cond, label %polly.loop_header, label %polly.loop_exit
@@ -36,7 +36,7 @@
 ; IR-NEXT:   %14 = add i64 %val, 1
 ; IR-NEXT:   %15 = shl i32 %ptr1, 3
 ; IR-NEXT:   %16 = add i32 %15, 72
-; IR-NEXT:   %uglygep = getelementptr i8, ptr %A, i32 %16
+; IR-NEXT:   %scevgep = getelementptr i8, ptr %A, i32 %16
 ; IR-NEXT:   br label %polly.loop_header
 
 ;
diff --git a/polly/test/ScopInfo/int2ptr_ptr2int_2.ll b/polly/test/ScopInfo/int2ptr_ptr2int_2.ll
index a9c3431270fb8..97878f7091b12 100644
--- a/polly/test/ScopInfo/int2ptr_ptr2int_2.ll
+++ b/polly/test/ScopInfo/int2ptr_ptr2int_2.ll
@@ -24,9 +24,9 @@
 ; IR-NEXT:   %ptr13 = ptrtoint ptr %ptr to i16
 ;
 ; IR:      polly.stmt.for.body:
-; IR-NEXT:   %tmp4_p_scalar_ = load i64, ptr %uglygep, align 8, !alias.scope !3, !noalias !0
+; IR-NEXT:   %tmp4_p_scalar_ = load i64, ptr %scevgep, align 8, !alias.scope !3, !noalias !0
 ; IR-NEXT:   %p_add4 = add nsw i64 %tmp4_p_scalar_, %polly.preload.tmp3.merge
-; IR-NEXT:   store i64 %p_add4, ptr %uglygep, align 8, !alias.scope !3, !noalias !0
+; IR-NEXT:   store i64 %p_add4, ptr %scevgep, align 8, !alias.scope !3, !noalias !0
 ; IR-NEXT:   %polly.indvar_next = add nsw i64 %polly.indvar, 1
 ; IR-NEXT:   %polly.loop_cond = icmp sle i64 %polly.indvar_next, 99
 ; IR-NEXT:   br i1 %polly.loop_cond, label %polly.loop_header, label %polly.loop_exit
@@ -35,7 +35,7 @@
 ; IR-NEXT:   %41 = add i16 %val, 1
 ; IR-NEXT:   %42 = shl i16 %ptr13, 3
 ; IR-NEXT:   %43 = add i16 %42, 72
-; IR-NEXT:   %uglygep = getelementptr i8, ptr %A, i16 %43
+; IR-NEXT:   %scevgep = getelementptr i8, ptr %A, i16 %43
 ; IR-NEXT:   br label %polly.loop_header
 ;
 target datalayout = "e-p:16:16:16-m:e-i64:64-f80:128-n8:16:16:64-S128"
diff --git a/polly/test/ScopInfo/licm_load.ll b/polly/test/ScopInfo/licm_load.ll
index 1ba6f9135deab..ade640976d007 100644
--- a/polly/test/ScopInfo/licm_load.ll
+++ b/polly/test/ScopInfo/licm_load.ll
@@ -1,8 +1,6 @@
-; RUN: opt %loadPolly -basic-aa -loop-rotate -indvars       -polly-prepare \
-; RUN: -polly-invariant-load-hoisting=true -polly-print-scops -disable-output < %s \
+; RUN: opt %loadNPMPolly -passes='loop(loop-rotate,indvars),polly-prepare,print<polly-function-scops>' -polly-invariant-load-hoisting=true -disable-output < %s 2>&1 \
 ; RUN: | FileCheck %s
-; RUN: opt %loadPolly -basic-aa -loop-rotate -indvars -licm -polly-prepare \
-; RUN: -polly-invariant-load-hoisting=true -polly-print-scops --disable-output< %s \
+; RUN: opt %loadNPMPolly -passes='loop-mssa(loop-rotate,indvars,licm),polly-prepare,print<polly-function-scops>' -polly-invariant-load-hoisting=true -disable-output < %s 2>&1 \
 ; RUN: | FileCheck %s
 ;
 ;    void foo(int n, float A[static const restrict n],
diff --git a/polly/test/ScopInfo/licm_potential_store.ll b/polly/test/ScopInfo/licm_potential_store.ll
index 19aeeb605e403..8a36ee84313a2 100644
--- a/polly/test/ScopInfo/licm_potential_store.ll
+++ b/polly/test/ScopInfo/licm_potential_store.ll
@@ -1,11 +1,9 @@
-; RUN: opt %loadPolly -basic-aa -sroa -instcombine -simplifycfg -tailcallopt \
-; RUN:    -simplifycfg -reassociate -loop-rotate -instcombine -indvars \
-; RUN:    -polly-prepare -polly-print-scops -disable-output < %s \
+; RUN: opt %loadNPMPolly -passes='sroa,instcombine,simplifycfg,reassociate,loop(loop-rotate),instcombine,indvars,polly-prepare,print<polly-function-scops>' \
+; RUN:    -tailcallopt -disable-output < %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=NOLICM
 
-; RUN: opt %loadPolly -basic-aa -sroa -instcombine -simplifycfg -tailcallopt \
-; RUN:    -simplifycfg -reassociate -loop-rotate -instcombine -indvars -licm \
-; RUN:    -polly-prepare -polly-print-scops -disable-output < %s \
+; RUN: opt %loadNPMPolly -passes='sroa,instcombine,simplifycfg,reassociate,loop(loop-rotate),instcombine,indvars,loop-mssa(licm),polly-prepare,print<polly-function-scops>' \
+; RUN:    -tailcallopt -disable-output < %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=LICM
 
 ;    void foo(int n, float A[static const restrict n], float x) {
diff --git a/polly/test/Unit/lit.site.cfg.in b/polly/test/Unit/lit.site.cfg.in
index 2aeaf197f06cf..a93b8b7a527ba 100644
--- a/polly/test/Unit/lit.site.cfg.in
+++ b/polly/test/Unit/lit.site.cfg.in
@@ -11,7 +11,6 @@ config.polly_obj_root = "@POLLY_BINARY_DIR@"
 config.polly_lib_dir = "@POLLY_LIB_DIR@"
 config.shlibdir = "@SHLIBDIR@"
 config.target_triple = "@LLVM_TARGET_TRIPLE@"
-config.enable_gpgpu_codegen = "@GPU_CODEGEN@"
 config.llvm_polly_link_into_tools = "@LLVM_POLLY_LINK_INTO_TOOLS@"
 config.has_unittests = @POLLY_GTEST_AVAIL@
 
diff --git a/polly/test/lit.cfg b/polly/test/lit.cfg
index 41e3a589c61e6..0943507ebe502 100644
--- a/polly/test/lit.cfg
+++ b/polly/test/lit.cfg
@@ -70,6 +70,4 @@ except OSError:
     print("Could not find llvm-config in " + config.llvm_tools_dir)
     exit(42)
 
-if re.search(r'NVPTX', llvm_config_cmd.stdout.read().decode('ascii')):
-    config.available_features.add('nvptx-registered-target')
 llvm_config_cmd.wait()
diff --git a/polly/test/lit.site.cfg.in b/polly/test/lit.site.cfg.in
index 4aed9875c3fb1..b440612608340 100644
--- a/polly/test/lit.site.cfg.in
+++ b/polly/test/lit.site.cfg.in
@@ -7,7 +7,6 @@ config.llvm_libs_dir = lit_config.substitute("@LLVM_LIBS_DIR@")
 config.polly_obj_root = "@POLLY_BINARY_DIR@"
 config.polly_lib_dir = "@POLLY_LIB_DIR@"
 config.target_triple = "@LLVM_TARGET_TRIPLE@"
-config.enable_gpgpu_codegen = "@GPU_CODEGEN@"
 config.llvm_polly_link_into_tools = "@LLVM_POLLY_LINK_INTO_TOOLS@"
 config.targets_to_build = "@TARGETS_TO_BUILD@"
 config.extra_paths = "@POLLY_TEST_EXTRA_PATHS@".split(";")
@@ -50,9 +49,6 @@ else:
     config.substitutions.append(('%loadNPMPolly', commonOpts ))
 
 
-if config.enable_gpgpu_codegen == 'TRUE' :
-    config.available_features.add('pollyacc')
-
 import lit.llvm
 lit.llvm.initialize(lit_config, config)
 
diff --git a/polly/tools/CMakeLists.txt b/polly/tools/CMakeLists.txt
deleted file mode 100644
index 7c89ed549421a..0000000000000
--- a/polly/tools/CMakeLists.txt
+++ /dev/null
@@ -1,5 +0,0 @@
-if (CUDA_FOUND OR OpenCL_FOUND)
-  add_subdirectory(GPURuntime)
-endif (CUDA_FOUND OR OpenCL_FOUND)
-
-set(LLVM_COMMON_DEPENDS ${LLVM_COMMON_DEPENDS} PARENT_SCOPE)
diff --git a/polly/tools/GPURuntime/CMakeLists.txt b/polly/tools/GPURuntime/CMakeLists.txt
deleted file mode 100644
index 4e17c276a11ba..0000000000000
--- a/polly/tools/GPURuntime/CMakeLists.txt
+++ /dev/null
@@ -1,19 +0,0 @@
-set(MODULE TRUE)
-set(LLVM_NO_RTTI 1)
-
-add_polly_library(GPURuntime
-  GPUJIT.c
-  )
-
-set_target_properties(GPURuntime
-  PROPERTIES
-  LINKER_LANGUAGE C
-  PREFIX "lib"
-  )
-
-set_property(TARGET GPURuntime PROPERTY C_STANDARD 99)
-
-set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fvisibility=default ")
-if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "Clang")
-  set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fno-sanitize=all ")
-endif()
diff --git a/polly/tools/GPURuntime/GPUJIT.c b/polly/tools/GPURuntime/GPUJIT.c
deleted file mode 100644
index 4de9626b135e7..0000000000000
--- a/polly/tools/GPURuntime/GPUJIT.c
+++ /dev/null
@@ -1,1856 +0,0 @@
-/******************** GPUJIT.c - GPUJIT Execution Engine **********************/
-/*                                                                            */
-/* Part of the LLVM Project, under the Apache License v2.0 with LLVM          */
-/* Exceptions.                                                                */
-/* See https://llvm.org/LICENSE.txt for license information.                  */
-/* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception                    */
-/*                                                                            */
-/******************************************************************************/
-/*                                                                            */
-/*  This file implements GPUJIT, a ptx string execution engine for GPU.       */
-/*                                                                            */
-/******************************************************************************/
-
-#include "GPUJIT.h"
-
-#ifdef HAS_LIBCUDART
-#include <cuda.h>
-#include <cuda_runtime.h>
-#endif /* HAS_LIBCUDART */
-
-#ifdef HAS_LIBOPENCL
-#ifdef __APPLE__
-#include <OpenCL/opencl.h>
-#else
-#include <CL/cl.h>
-#endif /* __APPLE__ */
-#endif /* HAS_LIBOPENCL */
-
-#include <assert.h>
-#include <dlfcn.h>
-#include <stdarg.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <unistd.h>
-
-static int DebugMode;
-static int CacheMode;
-#define max(x, y) ((x) > (y) ? (x) : (y))
-
-static PollyGPURuntime Runtime = RUNTIME_NONE;
-
-static void debug_print(const char *format, ...) {
-  if (!DebugMode)
-    return;
-
-  va_list args;
-  va_start(args, format);
-  vfprintf(stderr, format, args);
-  va_end(args);
-}
-#define dump_function() debug_print("-> %s\n", __func__)
-
-#define KERNEL_CACHE_SIZE 10
-
-static void err_runtime() __attribute__((noreturn));
-static void err_runtime() {
-  fprintf(stderr, "Runtime not correctly initialized.\n");
-  exit(-1);
-}
-
-struct PollyGPUContextT {
-  void *Context;
-};
-
-struct PollyGPUFunctionT {
-  void *Kernel;
-};
-
-struct PollyGPUDevicePtrT {
-  void *DevicePtr;
-};
-
-/******************************************************************************/
-/*                                  OpenCL                                    */
-/******************************************************************************/
-#ifdef HAS_LIBOPENCL
-
-struct OpenCLContextT {
-  cl_context Context;
-  cl_command_queue CommandQueue;
-};
-
-struct OpenCLKernelT {
-  cl_kernel Kernel;
-  cl_program Program;
-  const char *BinaryString;
-};
-
-struct OpenCLDevicePtrT {
-  cl_mem MemObj;
-};
-
-/* Dynamic library handles for the OpenCL runtime library. */
-static void *HandleOpenCL;
-static void *HandleOpenCLBeignet;
-
-/* Type-defines of function pointer to OpenCL Runtime API. */
-typedef cl_int clGetPlatformIDsFcnTy(cl_uint NumEntries,
-                                     cl_platform_id *Platforms,
-                                     cl_uint *NumPlatforms);
-static clGetPlatformIDsFcnTy *clGetPlatformIDsFcnPtr;
-
-typedef cl_int clGetDeviceIDsFcnTy(cl_platform_id Platform,
-                                   cl_device_type DeviceType,
-                                   cl_uint NumEntries, cl_device_id *Devices,
-                                   cl_uint *NumDevices);
-static clGetDeviceIDsFcnTy *clGetDeviceIDsFcnPtr;
-
-typedef cl_int clGetDeviceInfoFcnTy(cl_device_id Device,
-                                    cl_device_info ParamName,
-                                    size_t ParamValueSize, void *ParamValue,
-                                    size_t *ParamValueSizeRet);
-static clGetDeviceInfoFcnTy *clGetDeviceInfoFcnPtr;
-
-typedef cl_int clGetKernelInfoFcnTy(cl_kernel Kernel, cl_kernel_info ParamName,
-                                    size_t ParamValueSize, void *ParamValue,
-                                    size_t *ParamValueSizeRet);
-static clGetKernelInfoFcnTy *clGetKernelInfoFcnPtr;
-
-typedef cl_context clCreateContextFcnTy(
-    const cl_context_properties *Properties, cl_uint NumDevices,
-    const cl_device_id *Devices,
-    void CL_CALLBACK *pfn_notify(const char *Errinfo, const void *PrivateInfo,
-                                 size_t CB, void *UserData),
-    void *UserData, cl_int *ErrcodeRet);
-static clCreateContextFcnTy *clCreateContextFcnPtr;
-
-typedef cl_command_queue
-clCreateCommandQueueFcnTy(cl_context Context, cl_device_id Device,
-                          cl_command_queue_properties Properties,
-                          cl_int *ErrcodeRet);
-static clCreateCommandQueueFcnTy *clCreateCommandQueueFcnPtr;
-
-typedef cl_mem clCreateBufferFcnTy(cl_context Context, cl_mem_flags Flags,
-                                   size_t Size, void *HostPtr,
-                                   cl_int *ErrcodeRet);
-static clCreateBufferFcnTy *clCreateBufferFcnPtr;
-
-typedef cl_int
-clEnqueueWriteBufferFcnTy(cl_command_queue CommandQueue, cl_mem Buffer,
-                          cl_bool BlockingWrite, size_t Offset, size_t Size,
-                          const void *Ptr, cl_uint NumEventsInWaitList,
-                          const cl_event *EventWaitList, cl_event *Event);
-static clEnqueueWriteBufferFcnTy *clEnqueueWriteBufferFcnPtr;
-
-typedef cl_program
-clCreateProgramWithLLVMIntelFcnTy(cl_context Context, cl_uint NumDevices,
-                                  const cl_device_id *DeviceList,
-                                  const char *Filename, cl_int *ErrcodeRet);
-static clCreateProgramWithLLVMIntelFcnTy *clCreateProgramWithLLVMIntelFcnPtr;
-
-typedef cl_program clCreateProgramWithBinaryFcnTy(
-    cl_context Context, cl_uint NumDevices, const cl_device_id *DeviceList,
-    const size_t *Lengths, const unsigned char **Binaries, cl_int *BinaryStatus,
-    cl_int *ErrcodeRet);
-static clCreateProgramWithBinaryFcnTy *clCreateProgramWithBinaryFcnPtr;
-
-typedef cl_int clBuildProgramFcnTy(
-    cl_program Program, cl_uint NumDevices, const cl_device_id *DeviceList,
-    const char *Options,
-    void(CL_CALLBACK *pfn_notify)(cl_program Program, void *UserData),
-    void *UserData);
-static clBuildProgramFcnTy *clBuildProgramFcnPtr;
-
-typedef cl_kernel clCreateKernelFcnTy(cl_program Program,
-                                      const char *KernelName,
-                                      cl_int *ErrcodeRet);
-static clCreateKernelFcnTy *clCreateKernelFcnPtr;
-
-typedef cl_int clSetKernelArgFcnTy(cl_kernel Kernel, cl_uint ArgIndex,
-                                   size_t ArgSize, const void *ArgValue);
-static clSetKernelArgFcnTy *clSetKernelArgFcnPtr;
-
-typedef cl_int clEnqueueNDRangeKernelFcnTy(
-    cl_command_queue CommandQueue, cl_kernel Kernel, cl_uint WorkDim,
-    const size_t *GlobalWorkOffset, const size_t *GlobalWorkSize,
-    const size_t *LocalWorkSize, cl_uint NumEventsInWaitList,
-    const cl_event *EventWaitList, cl_event *Event);
-static clEnqueueNDRangeKernelFcnTy *clEnqueueNDRangeKernelFcnPtr;
-
-typedef cl_int clEnqueueReadBufferFcnTy(cl_command_queue CommandQueue,
-                                        cl_mem Buffer, cl_bool BlockingRead,
-                                        size_t Offset, size_t Size, void *Ptr,
-                                        cl_uint NumEventsInWaitList,
-                                        const cl_event *EventWaitList,
-                                        cl_event *Event);
-static clEnqueueReadBufferFcnTy *clEnqueueReadBufferFcnPtr;
-
-typedef cl_int clFlushFcnTy(cl_command_queue CommandQueue);
-static clFlushFcnTy *clFlushFcnPtr;
-
-typedef cl_int clFinishFcnTy(cl_command_queue CommandQueue);
-static clFinishFcnTy *clFinishFcnPtr;
-
-typedef cl_int clReleaseKernelFcnTy(cl_kernel Kernel);
-static clReleaseKernelFcnTy *clReleaseKernelFcnPtr;
-
-typedef cl_int clReleaseProgramFcnTy(cl_program Program);
-static clReleaseProgramFcnTy *clReleaseProgramFcnPtr;
-
-typedef cl_int clReleaseMemObjectFcnTy(cl_mem Memobject);
-static clReleaseMemObjectFcnTy *clReleaseMemObjectFcnPtr;
-
-typedef cl_int clReleaseCommandQueueFcnTy(cl_command_queue CommandQueue);
-static clReleaseCommandQueueFcnTy *clReleaseCommandQueueFcnPtr;
-
-typedef cl_int clReleaseContextFcnTy(cl_context Context);
-static clReleaseContextFcnTy *clReleaseContextFcnPtr;
-
-static void *getAPIHandleCL(void *Handle, const char *FuncName) {
-  char *Err;
-  void *FuncPtr;
-  dlerror();
-  FuncPtr = dlsym(Handle, FuncName);
-  if ((Err = dlerror()) != 0) {
-    fprintf(stderr, "Load OpenCL Runtime API failed: %s. \n", Err);
-    return 0;
-  }
-  return FuncPtr;
-}
-
-static int initialDeviceAPILibrariesCL() {
-  HandleOpenCLBeignet = dlopen("/usr/local/lib/beignet/libcl.so", RTLD_LAZY);
-  HandleOpenCL = dlopen("libOpenCL.so", RTLD_LAZY);
-  if (!HandleOpenCL) {
-    fprintf(stderr, "Cannot open library: %s. \n", dlerror());
-    return 0;
-  }
-  return 1;
-}
-
-/* Get function pointer to OpenCL Runtime API.
- *
- * Note that compilers conforming to the ISO C standard are required to
- * generate a warning if a conversion from a void * pointer to a function
- * pointer is attempted as in the following statements. The warning
- * of this kind of cast may not be emitted by clang and new versions of gcc
- * as it is valid on POSIX 2008. For compilers required to generate a warning,
- * we temporarily disable -Wpedantic, to avoid bloating the output with
- * unnecessary warnings.
- *
- * Reference:
- * http://pubs.opengroup.org/onlinepubs/9699919799/functions/dlsym.html
- */
-#pragma GCC diagnostic push
-#pragma GCC diagnostic ignored "-Wpedantic"
-static int initialDeviceAPIsCL() {
-  if (initialDeviceAPILibrariesCL() == 0)
-    return 0;
-
-  // FIXME: We are now always selecting the Intel Beignet driver if it is
-  // available on the system, instead of a possible NVIDIA or AMD OpenCL
-  // API. This selection should occurr based on the target architecture
-  // chosen when compiling.
-  void *Handle =
-      (HandleOpenCLBeignet != NULL ? HandleOpenCLBeignet : HandleOpenCL);
-
-  clGetPlatformIDsFcnPtr =
-      (clGetPlatformIDsFcnTy *)getAPIHandleCL(Handle, "clGetPlatformIDs");
-
-  clGetDeviceIDsFcnPtr =
-      (clGetDeviceIDsFcnTy *)getAPIHandleCL(Handle, "clGetDeviceIDs");
-
-  clGetDeviceInfoFcnPtr =
-      (clGetDeviceInfoFcnTy *)getAPIHandleCL(Handle, "clGetDeviceInfo");
-
-  clGetKernelInfoFcnPtr =
-      (clGetKernelInfoFcnTy *)getAPIHandleCL(Handle, "clGetKernelInfo");
-
-  clCreateContextFcnPtr =
-      (clCreateContextFcnTy *)getAPIHandleCL(Handle, "clCreateContext");
-
-  clCreateCommandQueueFcnPtr = (clCreateCommandQueueFcnTy *)getAPIHandleCL(
-      Handle, "clCreateCommandQueue");
-
-  clCreateBufferFcnPtr =
-      (clCreateBufferFcnTy *)getAPIHandleCL(Handle, "clCreateBuffer");
-
-  clEnqueueWriteBufferFcnPtr = (clEnqueueWriteBufferFcnTy *)getAPIHandleCL(
-      Handle, "clEnqueueWriteBuffer");
-
-  if (HandleOpenCLBeignet)
-    clCreateProgramWithLLVMIntelFcnPtr =
-        (clCreateProgramWithLLVMIntelFcnTy *)getAPIHandleCL(
-            Handle, "clCreateProgramWithLLVMIntel");
-
-  clCreateProgramWithBinaryFcnPtr =
-      (clCreateProgramWithBinaryFcnTy *)getAPIHandleCL(
-          Handle, "clCreateProgramWithBinary");
-
-  clBuildProgramFcnPtr =
-      (clBuildProgramFcnTy *)getAPIHandleCL(Handle, "clBuildProgram");
-
-  clCreateKernelFcnPtr =
-      (clCreateKernelFcnTy *)getAPIHandleCL(Handle, "clCreateKernel");
-
-  clSetKernelArgFcnPtr =
-      (clSetKernelArgFcnTy *)getAPIHandleCL(Handle, "clSetKernelArg");
-
-  clEnqueueNDRangeKernelFcnPtr = (clEnqueueNDRangeKernelFcnTy *)getAPIHandleCL(
-      Handle, "clEnqueueNDRangeKernel");
-
-  clEnqueueReadBufferFcnPtr =
-      (clEnqueueReadBufferFcnTy *)getAPIHandleCL(Handle, "clEnqueueReadBuffer");
-
-  clFlushFcnPtr = (clFlushFcnTy *)getAPIHandleCL(Handle, "clFlush");
-
-  clFinishFcnPtr = (clFinishFcnTy *)getAPIHandleCL(Handle, "clFinish");
-
-  clReleaseKernelFcnPtr =
-      (clReleaseKernelFcnTy *)getAPIHandleCL(Handle, "clReleaseKernel");
-
-  clReleaseProgramFcnPtr =
-      (clReleaseProgramFcnTy *)getAPIHandleCL(Handle, "clReleaseProgram");
-
-  clReleaseMemObjectFcnPtr =
-      (clReleaseMemObjectFcnTy *)getAPIHandleCL(Handle, "clReleaseMemObject");
-
-  clReleaseCommandQueueFcnPtr = (clReleaseCommandQueueFcnTy *)getAPIHandleCL(
-      Handle, "clReleaseCommandQueue");
-
-  clReleaseContextFcnPtr =
-      (clReleaseContextFcnTy *)getAPIHandleCL(Handle, "clReleaseContext");
-
-  return 1;
-}
-#pragma GCC diagnostic pop
-
-/* Context and Device. */
-static PollyGPUContext *GlobalContext = NULL;
-static cl_device_id GlobalDeviceID = NULL;
-
-/* Fd-Decl: Print out OpenCL Error codes to human readable strings. */
-static void printOpenCLError(int Error);
-
-static void checkOpenCLError(int Ret, const char *format, ...) {
-  if (Ret == CL_SUCCESS)
-    return;
-
-  printOpenCLError(Ret);
-  va_list args;
-  va_start(args, format);
-  vfprintf(stderr, format, args);
-  va_end(args);
-  exit(-1);
-}
-
-static PollyGPUContext *initContextCL() {
-  dump_function();
-
-  PollyGPUContext *Context;
-
-  cl_platform_id PlatformID = NULL;
-  cl_device_id DeviceID = NULL;
-  cl_uint NumDevicesRet;
-  cl_int Ret;
-
-  char DeviceRevision[256];
-  char DeviceName[256];
-  size_t DeviceRevisionRetSize, DeviceNameRetSize;
-
-  static __thread PollyGPUContext *CurrentContext = NULL;
-
-  if (CurrentContext)
-    return CurrentContext;
-
-  /* Get API handles. */
-  if (initialDeviceAPIsCL() == 0) {
-    fprintf(stderr, "Getting the \"handle\" for the OpenCL Runtime failed.\n");
-    exit(-1);
-  }
-
-  /* Get number of devices that support OpenCL. */
-  static const int NumberOfPlatforms = 1;
-  Ret = clGetPlatformIDsFcnPtr(NumberOfPlatforms, &PlatformID, NULL);
-  checkOpenCLError(Ret, "Failed to get platform IDs.\n");
-  // TODO: Extend to CL_DEVICE_TYPE_ALL?
-  static const int NumberOfDevices = 1;
-  Ret = clGetDeviceIDsFcnPtr(PlatformID, CL_DEVICE_TYPE_GPU, NumberOfDevices,
-                             &DeviceID, &NumDevicesRet);
-  checkOpenCLError(Ret, "Failed to get device IDs.\n");
-
-  GlobalDeviceID = DeviceID;
-  if (NumDevicesRet == 0) {
-    fprintf(stderr, "There is no device supporting OpenCL.\n");
-    exit(-1);
-  }
-
-  /* Get device revision. */
-  Ret =
-      clGetDeviceInfoFcnPtr(DeviceID, CL_DEVICE_VERSION, sizeof(DeviceRevision),
-                            DeviceRevision, &DeviceRevisionRetSize);
-  checkOpenCLError(Ret, "Failed to fetch device revision.\n");
-
-  /* Get device name. */
-  Ret = clGetDeviceInfoFcnPtr(DeviceID, CL_DEVICE_NAME, sizeof(DeviceName),
-                              DeviceName, &DeviceNameRetSize);
-  checkOpenCLError(Ret, "Failed to fetch device name.\n");
-
-  debug_print("> Running on GPU device %d : %s.\n", DeviceID, DeviceName);
-
-  /* Create context on the device. */
-  Context = (PollyGPUContext *)malloc(sizeof(PollyGPUContext));
-  if (Context == 0) {
-    fprintf(stderr, "Allocate memory for Polly GPU context failed.\n");
-    exit(-1);
-  }
-  Context->Context = (OpenCLContext *)malloc(sizeof(OpenCLContext));
-  if (Context->Context == 0) {
-    fprintf(stderr, "Allocate memory for Polly OpenCL context failed.\n");
-    exit(-1);
-  }
-  ((OpenCLContext *)Context->Context)->Context =
-      clCreateContextFcnPtr(NULL, NumDevicesRet, &DeviceID, NULL, NULL, &Ret);
-  checkOpenCLError(Ret, "Failed to create context.\n");
-
-  static const int ExtraProperties = 0;
-  ((OpenCLContext *)Context->Context)->CommandQueue =
-      clCreateCommandQueueFcnPtr(((OpenCLContext *)Context->Context)->Context,
-                                 DeviceID, ExtraProperties, &Ret);
-  checkOpenCLError(Ret, "Failed to create command queue.\n");
-
-  if (CacheMode)
-    CurrentContext = Context;
-
-  GlobalContext = Context;
-  return Context;
-}
-
-static void freeKernelCL(PollyGPUFunction *Kernel) {
-  dump_function();
-
-  if (CacheMode)
-    return;
-
-  if (!GlobalContext) {
-    fprintf(stderr, "GPGPU-code generation not correctly initialized.\n");
-    exit(-1);
-  }
-
-  cl_int Ret;
-  Ret = clFlushFcnPtr(((OpenCLContext *)GlobalContext->Context)->CommandQueue);
-  checkOpenCLError(Ret, "Failed to flush command queue.\n");
-  Ret = clFinishFcnPtr(((OpenCLContext *)GlobalContext->Context)->CommandQueue);
-  checkOpenCLError(Ret, "Failed to finish command queue.\n");
-
-  if (((OpenCLKernel *)Kernel->Kernel)->Kernel) {
-    cl_int Ret =
-        clReleaseKernelFcnPtr(((OpenCLKernel *)Kernel->Kernel)->Kernel);
-    checkOpenCLError(Ret, "Failed to release kernel.\n");
-  }
-
-  if (((OpenCLKernel *)Kernel->Kernel)->Program) {
-    cl_int Ret =
-        clReleaseProgramFcnPtr(((OpenCLKernel *)Kernel->Kernel)->Program);
-    checkOpenCLError(Ret, "Failed to release program.\n");
-  }
-
-  if (Kernel->Kernel)
-    free((OpenCLKernel *)Kernel->Kernel);
-
-  if (Kernel)
-    free(Kernel);
-}
-
-static PollyGPUFunction *getKernelCL(const char *BinaryBuffer,
-                                     const char *KernelName) {
-  dump_function();
-
-  if (!GlobalContext) {
-    fprintf(stderr, "GPGPU-code generation not correctly initialized.\n");
-    exit(-1);
-  }
-
-  static __thread PollyGPUFunction *KernelCache[KERNEL_CACHE_SIZE];
-  static __thread int NextCacheItem = 0;
-
-  for (long i = 0; i < KERNEL_CACHE_SIZE; i++) {
-    // We exploit here the property that all Polly-ACC kernels are allocated
-    // as global constants, hence a pointer comparision is sufficient to
-    // determin equality.
-    if (KernelCache[i] &&
-        ((OpenCLKernel *)KernelCache[i]->Kernel)->BinaryString ==
-            BinaryBuffer) {
-      debug_print("  -> using cached kernel\n");
-      return KernelCache[i];
-    }
-  }
-
-  PollyGPUFunction *Function = malloc(sizeof(PollyGPUFunction));
-  if (Function == 0) {
-    fprintf(stderr, "Allocate memory for Polly GPU function failed.\n");
-    exit(-1);
-  }
-  Function->Kernel = (OpenCLKernel *)malloc(sizeof(OpenCLKernel));
-  if (Function->Kernel == 0) {
-    fprintf(stderr, "Allocate memory for Polly OpenCL kernel failed.\n");
-    exit(-1);
-  }
-
-  if (!GlobalDeviceID) {
-    fprintf(stderr, "GPGPU-code generation not initialized correctly.\n");
-    exit(-1);
-  }
-
-  cl_int Ret;
-
-  if (HandleOpenCLBeignet) {
-    // This is a workaround, since clCreateProgramWithLLVMIntel only
-    // accepts a filename to a valid llvm-ir file as an argument, instead
-    // of accepting the BinaryBuffer directly.
-    char FileName[] = "/tmp/polly_kernelXXXXXX";
-    int File = mkstemp(FileName);
-    write(File, BinaryBuffer, strlen(BinaryBuffer));
-
-    ((OpenCLKernel *)Function->Kernel)->Program =
-        clCreateProgramWithLLVMIntelFcnPtr(
-            ((OpenCLContext *)GlobalContext->Context)->Context, 1,
-            &GlobalDeviceID, FileName, &Ret);
-    checkOpenCLError(Ret, "Failed to create program from llvm.\n");
-    close(File);
-    unlink(FileName);
-  } else {
-    size_t BinarySize = strlen(BinaryBuffer);
-    ((OpenCLKernel *)Function->Kernel)->Program =
-        clCreateProgramWithBinaryFcnPtr(
-            ((OpenCLContext *)GlobalContext->Context)->Context, 1,
-            &GlobalDeviceID, (const size_t *)&BinarySize,
-            (const unsigned char **)&BinaryBuffer, NULL, &Ret);
-    checkOpenCLError(Ret, "Failed to create program from binary.\n");
-  }
-
-  Ret = clBuildProgramFcnPtr(((OpenCLKernel *)Function->Kernel)->Program, 1,
-                             &GlobalDeviceID, NULL, NULL, NULL);
-  checkOpenCLError(Ret, "Failed to build program.\n");
-
-  ((OpenCLKernel *)Function->Kernel)->Kernel = clCreateKernelFcnPtr(
-      ((OpenCLKernel *)Function->Kernel)->Program, KernelName, &Ret);
-  checkOpenCLError(Ret, "Failed to create kernel.\n");
-
-  ((OpenCLKernel *)Function->Kernel)->BinaryString = BinaryBuffer;
-
-  if (CacheMode) {
-    if (KernelCache[NextCacheItem])
-      freeKernelCL(KernelCache[NextCacheItem]);
-
-    KernelCache[NextCacheItem] = Function;
-
-    NextCacheItem = (NextCacheItem + 1) % KERNEL_CACHE_SIZE;
-  }
-
-  return Function;
-}
-
-static void copyFromHostToDeviceCL(void *HostData, PollyGPUDevicePtr *DevData,
-                                   long MemSize) {
-  dump_function();
-
-  if (!GlobalContext) {
-    fprintf(stderr, "GPGPU-code generation not correctly initialized.\n");
-    exit(-1);
-  }
-
-  cl_int Ret;
-  Ret = clEnqueueWriteBufferFcnPtr(
-      ((OpenCLContext *)GlobalContext->Context)->CommandQueue,
-      ((OpenCLDevicePtr *)DevData->DevicePtr)->MemObj, CL_TRUE, 0, MemSize,
-      HostData, 0, NULL, NULL);
-  checkOpenCLError(Ret, "Copying data from host memory to device failed.\n");
-}
-
-static void copyFromDeviceToHostCL(PollyGPUDevicePtr *DevData, void *HostData,
-                                   long MemSize) {
-  dump_function();
-
-  if (!GlobalContext) {
-    fprintf(stderr, "GPGPU-code generation not correctly initialized.\n");
-    exit(-1);
-  }
-
-  cl_int Ret;
-  Ret = clEnqueueReadBufferFcnPtr(
-      ((OpenCLContext *)GlobalContext->Context)->CommandQueue,
-      ((OpenCLDevicePtr *)DevData->DevicePtr)->MemObj, CL_TRUE, 0, MemSize,
-      HostData, 0, NULL, NULL);
-  checkOpenCLError(Ret, "Copying results from device to host memory failed.\n");
-}
-
-static void launchKernelCL(PollyGPUFunction *Kernel, unsigned int GridDimX,
-                           unsigned int GridDimY, unsigned int BlockDimX,
-                           unsigned int BlockDimY, unsigned int BlockDimZ,
-                           void **Parameters) {
-  dump_function();
-
-  cl_int Ret;
-  cl_uint NumArgs;
-
-  if (!GlobalContext) {
-    fprintf(stderr, "GPGPU-code generation not correctly initialized.\n");
-    exit(-1);
-  }
-
-  OpenCLKernel *CLKernel = (OpenCLKernel *)Kernel->Kernel;
-  Ret = clGetKernelInfoFcnPtr(CLKernel->Kernel, CL_KERNEL_NUM_ARGS,
-                              sizeof(cl_uint), &NumArgs, NULL);
-  checkOpenCLError(Ret, "Failed to get number of kernel arguments.\n");
-
-  /* Argument sizes are stored at the end of the Parameters array. */
-  for (cl_uint i = 0; i < NumArgs; i++) {
-    Ret = clSetKernelArgFcnPtr(CLKernel->Kernel, i,
-                               *((int *)Parameters[NumArgs + i]),
-                               (void *)Parameters[i]);
-    checkOpenCLError(Ret, "Failed to set Kernel argument %d.\n", i);
-  }
-
-  unsigned int GridDimZ = 1;
-  size_t GlobalWorkSize[3] = {BlockDimX * GridDimX, BlockDimY * GridDimY,
-                              BlockDimZ * GridDimZ};
-  size_t LocalWorkSize[3] = {BlockDimX, BlockDimY, BlockDimZ};
-
-  static const int WorkDim = 3;
-  OpenCLContext *CLContext = (OpenCLContext *)GlobalContext->Context;
-  Ret = clEnqueueNDRangeKernelFcnPtr(CLContext->CommandQueue, CLKernel->Kernel,
-                                     WorkDim, NULL, GlobalWorkSize,
-                                     LocalWorkSize, 0, NULL, NULL);
-  checkOpenCLError(Ret, "Launching OpenCL kernel failed.\n");
-}
-
-static void freeDeviceMemoryCL(PollyGPUDevicePtr *Allocation) {
-  dump_function();
-
-  OpenCLDevicePtr *DevPtr = (OpenCLDevicePtr *)Allocation->DevicePtr;
-  cl_int Ret = clReleaseMemObjectFcnPtr((cl_mem)DevPtr->MemObj);
-  checkOpenCLError(Ret, "Failed to free device memory.\n");
-
-  free(DevPtr);
-  free(Allocation);
-}
-
-static PollyGPUDevicePtr *allocateMemoryForDeviceCL(long MemSize) {
-  dump_function();
-
-  if (!GlobalContext) {
-    fprintf(stderr, "GPGPU-code generation not correctly initialized.\n");
-    exit(-1);
-  }
-
-  PollyGPUDevicePtr *DevData = malloc(sizeof(PollyGPUDevicePtr));
-  if (DevData == 0) {
-    fprintf(stderr, "Allocate memory for GPU device memory pointer failed.\n");
-    exit(-1);
-  }
-  DevData->DevicePtr = (OpenCLDevicePtr *)malloc(sizeof(OpenCLDevicePtr));
-  if (DevData->DevicePtr == 0) {
-    fprintf(stderr, "Allocate memory for GPU device memory pointer failed.\n");
-    exit(-1);
-  }
-
-  cl_int Ret;
-  ((OpenCLDevicePtr *)DevData->DevicePtr)->MemObj =
-      clCreateBufferFcnPtr(((OpenCLContext *)GlobalContext->Context)->Context,
-                           CL_MEM_READ_WRITE, MemSize, NULL, &Ret);
-  checkOpenCLError(Ret,
-                   "Allocate memory for GPU device memory pointer failed.\n");
-
-  return DevData;
-}
-
-static void *getDevicePtrCL(PollyGPUDevicePtr *Allocation) {
-  dump_function();
-
-  OpenCLDevicePtr *DevPtr = (OpenCLDevicePtr *)Allocation->DevicePtr;
-  return (void *)DevPtr->MemObj;
-}
-
-static void synchronizeDeviceCL() {
-  dump_function();
-
-  if (!GlobalContext) {
-    fprintf(stderr, "GPGPU-code generation not correctly initialized.\n");
-    exit(-1);
-  }
-
-  if (clFinishFcnPtr(((OpenCLContext *)GlobalContext->Context)->CommandQueue) !=
-      CL_SUCCESS) {
-    fprintf(stderr, "Synchronizing device and host memory failed.\n");
-    exit(-1);
-  }
-}
-
-static void freeContextCL(PollyGPUContext *Context) {
-  dump_function();
-
-  cl_int Ret;
-
-  GlobalContext = NULL;
-
-  OpenCLContext *Ctx = (OpenCLContext *)Context->Context;
-  if (Ctx->CommandQueue) {
-    Ret = clReleaseCommandQueueFcnPtr(Ctx->CommandQueue);
-    checkOpenCLError(Ret, "Could not release command queue.\n");
-  }
-
-  if (Ctx->Context) {
-    Ret = clReleaseContextFcnPtr(Ctx->Context);
-    checkOpenCLError(Ret, "Could not release context.\n");
-  }
-
-  free(Ctx);
-  free(Context);
-}
-
-static void printOpenCLError(int Error) {
-
-  switch (Error) {
-  case CL_SUCCESS:
-    // Success, don't print an error.
-    break;
-
-  // JIT/Runtime errors.
-  case CL_DEVICE_NOT_FOUND:
-    fprintf(stderr, "Device not found.\n");
-    break;
-  case CL_DEVICE_NOT_AVAILABLE:
-    fprintf(stderr, "Device not available.\n");
-    break;
-  case CL_COMPILER_NOT_AVAILABLE:
-    fprintf(stderr, "Compiler not available.\n");
-    break;
-  case CL_MEM_OBJECT_ALLOCATION_FAILURE:
-    fprintf(stderr, "Mem object allocation failure.\n");
-    break;
-  case CL_OUT_OF_RESOURCES:
-    fprintf(stderr, "Out of resources.\n");
-    break;
-  case CL_OUT_OF_HOST_MEMORY:
-    fprintf(stderr, "Out of host memory.\n");
-    break;
-  case CL_PROFILING_INFO_NOT_AVAILABLE:
-    fprintf(stderr, "Profiling info not available.\n");
-    break;
-  case CL_MEM_COPY_OVERLAP:
-    fprintf(stderr, "Mem copy overlap.\n");
-    break;
-  case CL_IMAGE_FORMAT_MISMATCH:
-    fprintf(stderr, "Image format mismatch.\n");
-    break;
-  case CL_IMAGE_FORMAT_NOT_SUPPORTED:
-    fprintf(stderr, "Image format not supported.\n");
-    break;
-  case CL_BUILD_PROGRAM_FAILURE:
-    fprintf(stderr, "Build program failure.\n");
-    break;
-  case CL_MAP_FAILURE:
-    fprintf(stderr, "Map failure.\n");
-    break;
-  case CL_MISALIGNED_SUB_BUFFER_OFFSET:
-    fprintf(stderr, "Misaligned sub buffer offset.\n");
-    break;
-  case CL_EXEC_STATUS_ERROR_FOR_EVENTS_IN_WAIT_LIST:
-    fprintf(stderr, "Exec status error for events in wait list.\n");
-    break;
-  case CL_COMPILE_PROGRAM_FAILURE:
-    fprintf(stderr, "Compile program failure.\n");
-    break;
-  case CL_LINKER_NOT_AVAILABLE:
-    fprintf(stderr, "Linker not available.\n");
-    break;
-  case CL_LINK_PROGRAM_FAILURE:
-    fprintf(stderr, "Link program failure.\n");
-    break;
-  case CL_DEVICE_PARTITION_FAILED:
-    fprintf(stderr, "Device partition failed.\n");
-    break;
-  case CL_KERNEL_ARG_INFO_NOT_AVAILABLE:
-    fprintf(stderr, "Kernel arg info not available.\n");
-    break;
-
-  // Compiler errors.
-  case CL_INVALID_VALUE:
-    fprintf(stderr, "Invalid value.\n");
-    break;
-  case CL_INVALID_DEVICE_TYPE:
-    fprintf(stderr, "Invalid device type.\n");
-    break;
-  case CL_INVALID_PLATFORM:
-    fprintf(stderr, "Invalid platform.\n");
-    break;
-  case CL_INVALID_DEVICE:
-    fprintf(stderr, "Invalid device.\n");
-    break;
-  case CL_INVALID_CONTEXT:
-    fprintf(stderr, "Invalid context.\n");
-    break;
-  case CL_INVALID_QUEUE_PROPERTIES:
-    fprintf(stderr, "Invalid queue properties.\n");
-    break;
-  case CL_INVALID_COMMAND_QUEUE:
-    fprintf(stderr, "Invalid command queue.\n");
-    break;
-  case CL_INVALID_HOST_PTR:
-    fprintf(stderr, "Invalid host pointer.\n");
-    break;
-  case CL_INVALID_MEM_OBJECT:
-    fprintf(stderr, "Invalid memory object.\n");
-    break;
-  case CL_INVALID_IMAGE_FORMAT_DESCRIPTOR:
-    fprintf(stderr, "Invalid image format descriptor.\n");
-    break;
-  case CL_INVALID_IMAGE_SIZE:
-    fprintf(stderr, "Invalid image size.\n");
-    break;
-  case CL_INVALID_SAMPLER:
-    fprintf(stderr, "Invalid sampler.\n");
-    break;
-  case CL_INVALID_BINARY:
-    fprintf(stderr, "Invalid binary.\n");
-    break;
-  case CL_INVALID_BUILD_OPTIONS:
-    fprintf(stderr, "Invalid build options.\n");
-    break;
-  case CL_INVALID_PROGRAM:
-    fprintf(stderr, "Invalid program.\n");
-    break;
-  case CL_INVALID_PROGRAM_EXECUTABLE:
-    fprintf(stderr, "Invalid program executable.\n");
-    break;
-  case CL_INVALID_KERNEL_NAME:
-    fprintf(stderr, "Invalid kernel name.\n");
-    break;
-  case CL_INVALID_KERNEL_DEFINITION:
-    fprintf(stderr, "Invalid kernel definition.\n");
-    break;
-  case CL_INVALID_KERNEL:
-    fprintf(stderr, "Invalid kernel.\n");
-    break;
-  case CL_INVALID_ARG_INDEX:
-    fprintf(stderr, "Invalid arg index.\n");
-    break;
-  case CL_INVALID_ARG_VALUE:
-    fprintf(stderr, "Invalid arg value.\n");
-    break;
-  case CL_INVALID_ARG_SIZE:
-    fprintf(stderr, "Invalid arg size.\n");
-    break;
-  case CL_INVALID_KERNEL_ARGS:
-    fprintf(stderr, "Invalid kernel args.\n");
-    break;
-  case CL_INVALID_WORK_DIMENSION:
-    fprintf(stderr, "Invalid work dimension.\n");
-    break;
-  case CL_INVALID_WORK_GROUP_SIZE:
-    fprintf(stderr, "Invalid work group size.\n");
-    break;
-  case CL_INVALID_WORK_ITEM_SIZE:
-    fprintf(stderr, "Invalid work item size.\n");
-    break;
-  case CL_INVALID_GLOBAL_OFFSET:
-    fprintf(stderr, "Invalid global offset.\n");
-    break;
-  case CL_INVALID_EVENT_WAIT_LIST:
-    fprintf(stderr, "Invalid event wait list.\n");
-    break;
-  case CL_INVALID_EVENT:
-    fprintf(stderr, "Invalid event.\n");
-    break;
-  case CL_INVALID_OPERATION:
-    fprintf(stderr, "Invalid operation.\n");
-    break;
-  case CL_INVALID_GL_OBJECT:
-    fprintf(stderr, "Invalid GL object.\n");
-    break;
-  case CL_INVALID_BUFFER_SIZE:
-    fprintf(stderr, "Invalid buffer size.\n");
-    break;
-  case CL_INVALID_MIP_LEVEL:
-    fprintf(stderr, "Invalid mip level.\n");
-    break;
-  case CL_INVALID_GLOBAL_WORK_SIZE:
-    fprintf(stderr, "Invalid global work size.\n");
-    break;
-  case CL_INVALID_PROPERTY:
-    fprintf(stderr, "Invalid property.\n");
-    break;
-  case CL_INVALID_IMAGE_DESCRIPTOR:
-    fprintf(stderr, "Invalid image descriptor.\n");
-    break;
-  case CL_INVALID_COMPILER_OPTIONS:
-    fprintf(stderr, "Invalid compiler options.\n");
-    break;
-  case CL_INVALID_LINKER_OPTIONS:
-    fprintf(stderr, "Invalid linker options.\n");
-    break;
-  case CL_INVALID_DEVICE_PARTITION_COUNT:
-    fprintf(stderr, "Invalid device partition count.\n");
-    break;
-  case -69: // OpenCL 2.0 Code for CL_INVALID_PIPE_SIZE
-    fprintf(stderr, "Invalid pipe size.\n");
-    break;
-  case -70: // OpenCL 2.0 Code for CL_INVALID_DEVICE_QUEUE
-    fprintf(stderr, "Invalid device queue.\n");
-    break;
-
-  // NVIDIA specific error.
-  case -9999:
-    fprintf(stderr, "NVIDIA invalid read or write buffer.\n");
-    break;
-
-  default:
-    fprintf(stderr, "Unknown error code!\n");
-    break;
-  }
-}
-
-#endif /* HAS_LIBOPENCL */
-/******************************************************************************/
-/*                                   CUDA                                     */
-/******************************************************************************/
-#ifdef HAS_LIBCUDART
-
-struct CUDAContextT {
-  CUcontext Cuda;
-};
-
-struct CUDAKernelT {
-  CUfunction Cuda;
-  CUmodule CudaModule;
-  const char *BinaryString;
-};
-
-struct CUDADevicePtrT {
-  CUdeviceptr Cuda;
-};
-
-/* Dynamic library handles for the CUDA and CUDA runtime library. */
-static void *HandleCuda;
-static void *HandleCudaRT;
-
-/* Type-defines of function pointer to CUDA driver APIs. */
-typedef CUresult CUDAAPI CuMemAllocFcnTy(CUdeviceptr *, size_t);
-static CuMemAllocFcnTy *CuMemAllocFcnPtr;
-
-typedef CUresult CUDAAPI CuMemAllocManagedFcnTy(CUdeviceptr *, size_t,
-                                                unsigned int);
-static CuMemAllocManagedFcnTy *CuMemAllocManagedFcnPtr;
-
-typedef CUresult CUDAAPI CuLaunchKernelFcnTy(
-    CUfunction F, unsigned int GridDimX, unsigned int GridDimY,
-    unsigned int gridDimZ, unsigned int blockDimX, unsigned int BlockDimY,
-    unsigned int BlockDimZ, unsigned int SharedMemBytes, CUstream HStream,
-    void **KernelParams, void **Extra);
-static CuLaunchKernelFcnTy *CuLaunchKernelFcnPtr;
-
-typedef CUresult CUDAAPI CuMemcpyDtoHFcnTy(void *, CUdeviceptr, size_t);
-static CuMemcpyDtoHFcnTy *CuMemcpyDtoHFcnPtr;
-
-typedef CUresult CUDAAPI CuMemcpyHtoDFcnTy(CUdeviceptr, const void *, size_t);
-static CuMemcpyHtoDFcnTy *CuMemcpyHtoDFcnPtr;
-
-typedef CUresult CUDAAPI CuMemFreeFcnTy(CUdeviceptr);
-static CuMemFreeFcnTy *CuMemFreeFcnPtr;
-
-typedef CUresult CUDAAPI CuModuleUnloadFcnTy(CUmodule);
-static CuModuleUnloadFcnTy *CuModuleUnloadFcnPtr;
-
-typedef CUresult CUDAAPI CuProfilerStopFcnTy();
-static CuProfilerStopFcnTy *CuProfilerStopFcnPtr;
-
-typedef CUresult CUDAAPI CuCtxDestroyFcnTy(CUcontext);
-static CuCtxDestroyFcnTy *CuCtxDestroyFcnPtr;
-
-typedef CUresult CUDAAPI CuInitFcnTy(unsigned int);
-static CuInitFcnTy *CuInitFcnPtr;
-
-typedef CUresult CUDAAPI CuDeviceGetCountFcnTy(int *);
-static CuDeviceGetCountFcnTy *CuDeviceGetCountFcnPtr;
-
-typedef CUresult CUDAAPI CuCtxCreateFcnTy(CUcontext *, unsigned int, CUdevice);
-static CuCtxCreateFcnTy *CuCtxCreateFcnPtr;
-
-typedef CUresult CUDAAPI CuCtxGetCurrentFcnTy(CUcontext *);
-static CuCtxGetCurrentFcnTy *CuCtxGetCurrentFcnPtr;
-
-typedef CUresult CUDAAPI CuDeviceGetFcnTy(CUdevice *, int);
-static CuDeviceGetFcnTy *CuDeviceGetFcnPtr;
-
-typedef CUresult CUDAAPI CuModuleLoadDataExFcnTy(CUmodule *, const void *,
-                                                 unsigned int, CUjit_option *,
-                                                 void **);
-static CuModuleLoadDataExFcnTy *CuModuleLoadDataExFcnPtr;
-
-typedef CUresult CUDAAPI CuModuleLoadDataFcnTy(CUmodule *Module,
-                                               const void *Image);
-static CuModuleLoadDataFcnTy *CuModuleLoadDataFcnPtr;
-
-typedef CUresult CUDAAPI CuModuleGetFunctionFcnTy(CUfunction *, CUmodule,
-                                                  const char *);
-static CuModuleGetFunctionFcnTy *CuModuleGetFunctionFcnPtr;
-
-typedef CUresult CUDAAPI CuDeviceComputeCapabilityFcnTy(int *, int *, CUdevice);
-static CuDeviceComputeCapabilityFcnTy *CuDeviceComputeCapabilityFcnPtr;
-
-typedef CUresult CUDAAPI CuDeviceGetNameFcnTy(char *, int, CUdevice);
-static CuDeviceGetNameFcnTy *CuDeviceGetNameFcnPtr;
-
-typedef CUresult CUDAAPI CuLinkAddDataFcnTy(CUlinkState State,
-                                            CUjitInputType Type, void *Data,
-                                            size_t Size, const char *Name,
-                                            unsigned int NumOptions,
-                                            CUjit_option *Options,
-                                            void **OptionValues);
-static CuLinkAddDataFcnTy *CuLinkAddDataFcnPtr;
-
-typedef CUresult CUDAAPI CuLinkCreateFcnTy(unsigned int NumOptions,
-                                           CUjit_option *Options,
-                                           void **OptionValues,
-                                           CUlinkState *StateOut);
-static CuLinkCreateFcnTy *CuLinkCreateFcnPtr;
-
-typedef CUresult CUDAAPI CuLinkCompleteFcnTy(CUlinkState State, void **CubinOut,
-                                             size_t *SizeOut);
-static CuLinkCompleteFcnTy *CuLinkCompleteFcnPtr;
-
-typedef CUresult CUDAAPI CuLinkDestroyFcnTy(CUlinkState State);
-static CuLinkDestroyFcnTy *CuLinkDestroyFcnPtr;
-
-typedef CUresult CUDAAPI CuCtxSynchronizeFcnTy();
-static CuCtxSynchronizeFcnTy *CuCtxSynchronizeFcnPtr;
-
-/* Type-defines of function pointer ot CUDA runtime APIs. */
-typedef cudaError_t CUDARTAPI CudaThreadSynchronizeFcnTy(void);
-static CudaThreadSynchronizeFcnTy *CudaThreadSynchronizeFcnPtr;
-
-static void *getAPIHandleCUDA(void *Handle, const char *FuncName) {
-  char *Err;
-  void *FuncPtr;
-  dlerror();
-  FuncPtr = dlsym(Handle, FuncName);
-  if ((Err = dlerror()) != 0) {
-    fprintf(stderr, "Load CUDA driver API failed: %s. \n", Err);
-    return 0;
-  }
-  return FuncPtr;
-}
-
-static int initialDeviceAPILibrariesCUDA() {
-  HandleCuda = dlopen("libcuda.so", RTLD_LAZY);
-  if (!HandleCuda) {
-    fprintf(stderr, "Cannot open library: %s. \n", dlerror());
-    return 0;
-  }
-
-  HandleCudaRT = dlopen("libcudart.so", RTLD_LAZY);
-  if (!HandleCudaRT) {
-    fprintf(stderr, "Cannot open library: %s. \n", dlerror());
-    return 0;
-  }
-
-  return 1;
-}
-
-/* Get function pointer to CUDA Driver APIs.
- *
- * Note that compilers conforming to the ISO C standard are required to
- * generate a warning if a conversion from a void * pointer to a function
- * pointer is attempted as in the following statements. The warning
- * of this kind of cast may not be emitted by clang and new versions of gcc
- * as it is valid on POSIX 2008. For compilers required to generate a warning,
- * we temporarily disable -Wpedantic, to avoid bloating the output with
- * unnecessary warnings.
- *
- * Reference:
- * http://pubs.opengroup.org/onlinepubs/9699919799/functions/dlsym.html
- */
-#pragma GCC diagnostic push
-#pragma GCC diagnostic ignored "-Wpedantic"
-static int initialDeviceAPIsCUDA() {
-  if (initialDeviceAPILibrariesCUDA() == 0)
-    return 0;
-
-  CuLaunchKernelFcnPtr =
-      (CuLaunchKernelFcnTy *)getAPIHandleCUDA(HandleCuda, "cuLaunchKernel");
-
-  CuMemAllocFcnPtr =
-      (CuMemAllocFcnTy *)getAPIHandleCUDA(HandleCuda, "cuMemAlloc_v2");
-
-  CuMemAllocManagedFcnPtr = (CuMemAllocManagedFcnTy *)getAPIHandleCUDA(
-      HandleCuda, "cuMemAllocManaged");
-
-  CuMemFreeFcnPtr =
-      (CuMemFreeFcnTy *)getAPIHandleCUDA(HandleCuda, "cuMemFree_v2");
-
-  CuMemcpyDtoHFcnPtr =
-      (CuMemcpyDtoHFcnTy *)getAPIHandleCUDA(HandleCuda, "cuMemcpyDtoH_v2");
-
-  CuMemcpyHtoDFcnPtr =
-      (CuMemcpyHtoDFcnTy *)getAPIHandleCUDA(HandleCuda, "cuMemcpyHtoD_v2");
-
-  CuModuleUnloadFcnPtr =
-      (CuModuleUnloadFcnTy *)getAPIHandleCUDA(HandleCuda, "cuModuleUnload");
-
-  CuProfilerStopFcnPtr =
-      (CuProfilerStopFcnTy *)getAPIHandleCUDA(HandleCuda, "cuProfilerStop");
-
-  CuCtxDestroyFcnPtr =
-      (CuCtxDestroyFcnTy *)getAPIHandleCUDA(HandleCuda, "cuCtxDestroy");
-
-  CuInitFcnPtr = (CuInitFcnTy *)getAPIHandleCUDA(HandleCuda, "cuInit");
-
-  CuDeviceGetCountFcnPtr =
-      (CuDeviceGetCountFcnTy *)getAPIHandleCUDA(HandleCuda, "cuDeviceGetCount");
-
-  CuDeviceGetFcnPtr =
-      (CuDeviceGetFcnTy *)getAPIHandleCUDA(HandleCuda, "cuDeviceGet");
-
-  CuCtxCreateFcnPtr =
-      (CuCtxCreateFcnTy *)getAPIHandleCUDA(HandleCuda, "cuCtxCreate_v2");
-
-  CuCtxGetCurrentFcnPtr =
-      (CuCtxGetCurrentFcnTy *)getAPIHandleCUDA(HandleCuda, "cuCtxGetCurrent");
-
-  CuModuleLoadDataExFcnPtr = (CuModuleLoadDataExFcnTy *)getAPIHandleCUDA(
-      HandleCuda, "cuModuleLoadDataEx");
-
-  CuModuleLoadDataFcnPtr =
-      (CuModuleLoadDataFcnTy *)getAPIHandleCUDA(HandleCuda, "cuModuleLoadData");
-
-  CuModuleGetFunctionFcnPtr = (CuModuleGetFunctionFcnTy *)getAPIHandleCUDA(
-      HandleCuda, "cuModuleGetFunction");
-
-  CuDeviceComputeCapabilityFcnPtr =
-      (CuDeviceComputeCapabilityFcnTy *)getAPIHandleCUDA(
-          HandleCuda, "cuDeviceComputeCapability");
-
-  CuDeviceGetNameFcnPtr =
-      (CuDeviceGetNameFcnTy *)getAPIHandleCUDA(HandleCuda, "cuDeviceGetName");
-
-  CuLinkAddDataFcnPtr =
-      (CuLinkAddDataFcnTy *)getAPIHandleCUDA(HandleCuda, "cuLinkAddData");
-
-  CuLinkCreateFcnPtr =
-      (CuLinkCreateFcnTy *)getAPIHandleCUDA(HandleCuda, "cuLinkCreate");
-
-  CuLinkCompleteFcnPtr =
-      (CuLinkCompleteFcnTy *)getAPIHandleCUDA(HandleCuda, "cuLinkComplete");
-
-  CuLinkDestroyFcnPtr =
-      (CuLinkDestroyFcnTy *)getAPIHandleCUDA(HandleCuda, "cuLinkDestroy");
-
-  CuCtxSynchronizeFcnPtr =
-      (CuCtxSynchronizeFcnTy *)getAPIHandleCUDA(HandleCuda, "cuCtxSynchronize");
-
-  /* Get function pointer to CUDA Runtime APIs. */
-  CudaThreadSynchronizeFcnPtr = (CudaThreadSynchronizeFcnTy *)getAPIHandleCUDA(
-      HandleCudaRT, "cudaThreadSynchronize");
-
-  return 1;
-}
-#pragma GCC diagnostic pop
-
-static PollyGPUContext *initContextCUDA() {
-  dump_function();
-  PollyGPUContext *Context;
-  CUdevice Device;
-
-  int Major = 0, Minor = 0, DeviceID = 0;
-  char DeviceName[256];
-  int DeviceCount = 0;
-
-  static __thread PollyGPUContext *CurrentContext = NULL;
-
-  if (CurrentContext)
-    return CurrentContext;
-
-  /* Get API handles. */
-  if (initialDeviceAPIsCUDA() == 0) {
-    fprintf(stderr, "Getting the \"handle\" for the CUDA driver API failed.\n");
-    exit(-1);
-  }
-
-  if (CuInitFcnPtr(0) != CUDA_SUCCESS) {
-    fprintf(stderr, "Initializing the CUDA driver API failed.\n");
-    exit(-1);
-  }
-
-  /* Get number of devices that supports CUDA. */
-  CuDeviceGetCountFcnPtr(&DeviceCount);
-  if (DeviceCount == 0) {
-    fprintf(stderr, "There is no device supporting CUDA.\n");
-    exit(-1);
-  }
-
-  CuDeviceGetFcnPtr(&Device, 0);
-
-  /* Get compute capabilities and the device name. */
-  CuDeviceComputeCapabilityFcnPtr(&Major, &Minor, Device);
-  CuDeviceGetNameFcnPtr(DeviceName, 256, Device);
-  debug_print("> Running on GPU device %d : %s.\n", DeviceID, DeviceName);
-
-  /* Create context on the device. */
-  Context = (PollyGPUContext *)malloc(sizeof(PollyGPUContext));
-  if (Context == 0) {
-    fprintf(stderr, "Allocate memory for Polly GPU context failed.\n");
-    exit(-1);
-  }
-  Context->Context = malloc(sizeof(CUDAContext));
-  if (Context->Context == 0) {
-    fprintf(stderr, "Allocate memory for Polly CUDA context failed.\n");
-    exit(-1);
-  }
-
-  // In cases where managed memory is used, it is quite likely that
-  // `cudaMallocManaged` / `polly_mallocManaged` was called before
-  // `polly_initContext` was called.
-  //
-  // If `polly_initContext` calls `CuCtxCreate` when there already was a
-  // pre-existing context created by the runtime API, this causes code running
-  // on P100 to hang. So, we query for a pre-existing context to try and use.
-  // If there is no pre-existing context, we create a new context
-
-  // The possible pre-existing context from previous runtime API calls.
-  CUcontext MaybeRuntimeAPIContext;
-  if (CuCtxGetCurrentFcnPtr(&MaybeRuntimeAPIContext) != CUDA_SUCCESS) {
-    fprintf(stderr, "cuCtxGetCurrent failed.\n");
-    exit(-1);
-  }
-
-  // There was no previous context, initialise it.
-  if (MaybeRuntimeAPIContext == NULL) {
-    if (CuCtxCreateFcnPtr(&(((CUDAContext *)Context->Context)->Cuda), 0,
-                          Device) != CUDA_SUCCESS) {
-      fprintf(stderr, "cuCtxCreateFcnPtr failed.\n");
-      exit(-1);
-    }
-  } else {
-    ((CUDAContext *)Context->Context)->Cuda = MaybeRuntimeAPIContext;
-  }
-
-  if (CacheMode)
-    CurrentContext = Context;
-
-  return Context;
-}
-
-static void freeKernelCUDA(PollyGPUFunction *Kernel) {
-  dump_function();
-
-  if (CacheMode)
-    return;
-
-  if (((CUDAKernel *)Kernel->Kernel)->CudaModule)
-    CuModuleUnloadFcnPtr(((CUDAKernel *)Kernel->Kernel)->CudaModule);
-
-  if (Kernel->Kernel)
-    free((CUDAKernel *)Kernel->Kernel);
-
-  if (Kernel)
-    free(Kernel);
-}
-
-static PollyGPUFunction *getKernelCUDA(const char *BinaryBuffer,
-                                       const char *KernelName) {
-  dump_function();
-
-  static __thread PollyGPUFunction *KernelCache[KERNEL_CACHE_SIZE];
-  static __thread int NextCacheItem = 0;
-
-  for (long i = 0; i < KERNEL_CACHE_SIZE; i++) {
-    // We exploit here the property that all Polly-ACC kernels are allocated
-    // as global constants, hence a pointer comparision is sufficient to
-    // determin equality.
-    if (KernelCache[i] &&
-        ((CUDAKernel *)KernelCache[i]->Kernel)->BinaryString == BinaryBuffer) {
-      debug_print("  -> using cached kernel\n");
-      return KernelCache[i];
-    }
-  }
-
-  PollyGPUFunction *Function = malloc(sizeof(PollyGPUFunction));
-  if (Function == 0) {
-    fprintf(stderr, "Allocate memory for Polly GPU function failed.\n");
-    exit(-1);
-  }
-  Function->Kernel = (CUDAKernel *)malloc(sizeof(CUDAKernel));
-  if (Function->Kernel == 0) {
-    fprintf(stderr, "Allocate memory for Polly CUDA function failed.\n");
-    exit(-1);
-  }
-
-  CUresult Res;
-  CUlinkState LState;
-  CUjit_option Options[6];
-  void *OptionVals[6];
-  float Walltime = 0;
-  unsigned long LogSize = 8192;
-  char ErrorLog[8192], InfoLog[8192];
-  void *CuOut;
-  size_t OutSize;
-
-  // Setup linker options
-  // Return walltime from JIT compilation
-  Options[0] = CU_JIT_WALL_TIME;
-  OptionVals[0] = (void *)&Walltime;
-  // Pass a buffer for info messages
-  Options[1] = CU_JIT_INFO_LOG_BUFFER;
-  OptionVals[1] = (void *)InfoLog;
-  // Pass the size of the info buffer
-  Options[2] = CU_JIT_INFO_LOG_BUFFER_SIZE_BYTES;
-  OptionVals[2] = (void *)LogSize;
-  // Pass a buffer for error message
-  Options[3] = CU_JIT_ERROR_LOG_BUFFER;
-  OptionVals[3] = (void *)ErrorLog;
-  // Pass the size of the error buffer
-  Options[4] = CU_JIT_ERROR_LOG_BUFFER_SIZE_BYTES;
-  OptionVals[4] = (void *)LogSize;
-  // Make the linker verbose
-  Options[5] = CU_JIT_LOG_VERBOSE;
-  OptionVals[5] = (void *)1;
-
-  memset(ErrorLog, 0, sizeof(ErrorLog));
-
-  CuLinkCreateFcnPtr(6, Options, OptionVals, &LState);
-  Res = CuLinkAddDataFcnPtr(LState, CU_JIT_INPUT_PTX, (void *)BinaryBuffer,
-                            strlen(BinaryBuffer) + 1, 0, 0, 0, 0);
-  if (Res != CUDA_SUCCESS) {
-    fprintf(stderr, "PTX Linker Error:\n%s\n%s", ErrorLog, InfoLog);
-    exit(-1);
-  }
-
-  Res = CuLinkCompleteFcnPtr(LState, &CuOut, &OutSize);
-  if (Res != CUDA_SUCCESS) {
-    fprintf(stderr, "Complete ptx linker step failed.\n");
-    fprintf(stderr, "\n%s\n", ErrorLog);
-    exit(-1);
-  }
-
-  debug_print("CUDA Link Completed in %fms. Linker Output:\n%s\n", Walltime,
-              InfoLog);
-
-  Res = CuModuleLoadDataFcnPtr(&(((CUDAKernel *)Function->Kernel)->CudaModule),
-                               CuOut);
-  if (Res != CUDA_SUCCESS) {
-    fprintf(stderr, "Loading ptx assembly text failed.\n");
-    exit(-1);
-  }
-
-  Res = CuModuleGetFunctionFcnPtr(&(((CUDAKernel *)Function->Kernel)->Cuda),
-                                  ((CUDAKernel *)Function->Kernel)->CudaModule,
-                                  KernelName);
-  if (Res != CUDA_SUCCESS) {
-    fprintf(stderr, "Loading kernel function failed.\n");
-    exit(-1);
-  }
-
-  CuLinkDestroyFcnPtr(LState);
-
-  ((CUDAKernel *)Function->Kernel)->BinaryString = BinaryBuffer;
-
-  if (CacheMode) {
-    if (KernelCache[NextCacheItem])
-      freeKernelCUDA(KernelCache[NextCacheItem]);
-
-    KernelCache[NextCacheItem] = Function;
-
-    NextCacheItem = (NextCacheItem + 1) % KERNEL_CACHE_SIZE;
-  }
-
-  return Function;
-}
-
-static void synchronizeDeviceCUDA() {
-  dump_function();
-  if (CuCtxSynchronizeFcnPtr() != CUDA_SUCCESS) {
-    fprintf(stderr, "Synchronizing device and host memory failed.\n");
-    exit(-1);
-  }
-}
-
-static void copyFromHostToDeviceCUDA(void *HostData, PollyGPUDevicePtr *DevData,
-                                     long MemSize) {
-  dump_function();
-
-  CUdeviceptr CuDevData = ((CUDADevicePtr *)DevData->DevicePtr)->Cuda;
-  CuMemcpyHtoDFcnPtr(CuDevData, HostData, MemSize);
-}
-
-static void copyFromDeviceToHostCUDA(PollyGPUDevicePtr *DevData, void *HostData,
-                                     long MemSize) {
-  dump_function();
-
-  if (CuMemcpyDtoHFcnPtr(HostData, ((CUDADevicePtr *)DevData->DevicePtr)->Cuda,
-                         MemSize) != CUDA_SUCCESS) {
-    fprintf(stderr, "Copying results from device to host memory failed.\n");
-    exit(-1);
-  }
-}
-
-static void launchKernelCUDA(PollyGPUFunction *Kernel, unsigned int GridDimX,
-                             unsigned int GridDimY, unsigned int BlockDimX,
-                             unsigned int BlockDimY, unsigned int BlockDimZ,
-                             void **Parameters) {
-  dump_function();
-
-  unsigned GridDimZ = 1;
-  unsigned int SharedMemBytes = CU_SHARED_MEM_CONFIG_DEFAULT_BANK_SIZE;
-  CUstream Stream = 0;
-  void **Extra = 0;
-
-  CUresult Res;
-  Res =
-      CuLaunchKernelFcnPtr(((CUDAKernel *)Kernel->Kernel)->Cuda, GridDimX,
-                           GridDimY, GridDimZ, BlockDimX, BlockDimY, BlockDimZ,
-                           SharedMemBytes, Stream, Parameters, Extra);
-  if (Res != CUDA_SUCCESS) {
-    fprintf(stderr, "Launching CUDA kernel failed.\n");
-    exit(-1);
-  }
-}
-
-// Maximum number of managed memory pointers.
-#define DEFAULT_MAX_POINTERS 4000
-// For the rationale behing a list of free pointers, see `polly_freeManaged`.
-void **g_managedptrs;
-unsigned long long g_nmanagedptrs = 0;
-unsigned long long g_maxmanagedptrs = 0;
-
-__attribute__((constructor)) static void initManagedPtrsBuffer() {
-  g_maxmanagedptrs = DEFAULT_MAX_POINTERS;
-  const char *maxManagedPointersString = getenv("POLLY_MAX_MANAGED_POINTERS");
-  if (maxManagedPointersString)
-    g_maxmanagedptrs = atoll(maxManagedPointersString);
-
-  g_managedptrs = (void **)malloc(sizeof(void *) * g_maxmanagedptrs);
-}
-
-// Add a pointer as being allocated by cuMallocManaged
-void addManagedPtr(void *mem) {
-  assert(g_maxmanagedptrs > 0 && "g_maxmanagedptrs was set to 0!");
-  assert(g_nmanagedptrs < g_maxmanagedptrs &&
-         "We have hit the maximum number of "
-         "managed pointers allowed. Set the "
-         "POLLY_MAX_MANAGED_POINTERS environment variable. ");
-  g_managedptrs[g_nmanagedptrs++] = mem;
-}
-
-int isManagedPtr(void *mem) {
-  for (unsigned long long i = 0; i < g_nmanagedptrs; i++) {
-    if (g_managedptrs[i] == mem)
-      return 1;
-  }
-  return 0;
-}
-
-void freeManagedCUDA(void *mem) {
-  dump_function();
-
-  // In a real-world program this was used (COSMO), there were more `free`
-  // calls in the original source than `malloc` calls. Hence, replacing all
-  // `free`s with `cudaFree` does not work, since we would try to free
-  // 'illegal' memory.
-  // As a quick fix, we keep a free list and check if `mem` is a managed memory
-  // pointer. If it is, we call `cudaFree`.
-  // If not, we pass it along to the underlying allocator.
-  // This is a hack, and can be removed if the underlying issue is fixed.
-  if (isManagedPtr(mem)) {
-    if (CuMemFreeFcnPtr((size_t)mem) != CUDA_SUCCESS) {
-      fprintf(stderr, "cudaFree failed.\n");
-      exit(-1);
-    }
-    return;
-  } else {
-    free(mem);
-  }
-}
-
-void *mallocManagedCUDA(size_t size) {
-  // Note: [Size 0 allocations]
-  // Sometimes, some runtime computation of size could create a size of 0
-  // for an allocation. In these cases, we do not wish to fail.
-  // The CUDA API fails on size 0 allocations.
-  // So, we allocate size a minimum of size 1.
-  if (!size && DebugMode)
-    fprintf(stderr, "cudaMallocManaged called with size 0. "
-                    "Promoting to size 1");
-  size = max(size, 1);
-  PollyGPUContext *_ = polly_initContextCUDA();
-  assert(_ && "polly_initContextCUDA failed");
-
-  void *newMemPtr;
-  const CUresult Res = CuMemAllocManagedFcnPtr((CUdeviceptr *)&newMemPtr, size,
-                                               CU_MEM_ATTACH_GLOBAL);
-  if (Res != CUDA_SUCCESS) {
-    fprintf(stderr, "cudaMallocManaged failed for size: %zu\n", size);
-    exit(-1);
-  }
-  addManagedPtr(newMemPtr);
-  return newMemPtr;
-}
-
-static void freeDeviceMemoryCUDA(PollyGPUDevicePtr *Allocation) {
-  dump_function();
-  CUDADevicePtr *DevPtr = (CUDADevicePtr *)Allocation->DevicePtr;
-  CuMemFreeFcnPtr((CUdeviceptr)DevPtr->Cuda);
-  free(DevPtr);
-  free(Allocation);
-}
-
-static PollyGPUDevicePtr *allocateMemoryForDeviceCUDA(long MemSize) {
-  if (!MemSize && DebugMode)
-    fprintf(stderr, "allocateMemoryForDeviceCUDA called with size 0. "
-                    "Promoting to size 1");
-  // see: [Size 0 allocations]
-  MemSize = max(MemSize, 1);
-  dump_function();
-
-  PollyGPUDevicePtr *DevData = malloc(sizeof(PollyGPUDevicePtr));
-  if (DevData == 0) {
-    fprintf(stderr,
-            "Allocate memory for GPU device memory pointer failed."
-            " Line: %d | Size: %ld\n",
-            __LINE__, MemSize);
-    exit(-1);
-  }
-  DevData->DevicePtr = (CUDADevicePtr *)malloc(sizeof(CUDADevicePtr));
-  if (DevData->DevicePtr == 0) {
-    fprintf(stderr,
-            "Allocate memory for GPU device memory pointer failed."
-            " Line: %d | Size: %ld\n",
-            __LINE__, MemSize);
-    exit(-1);
-  }
-
-  CUresult Res =
-      CuMemAllocFcnPtr(&(((CUDADevicePtr *)DevData->DevicePtr)->Cuda), MemSize);
-
-  if (Res != CUDA_SUCCESS) {
-    fprintf(stderr,
-            "Allocate memory for GPU device memory pointer failed."
-            " Line: %d | Size: %ld\n",
-            __LINE__, MemSize);
-    exit(-1);
-  }
-
-  return DevData;
-}
-
-static void *getDevicePtrCUDA(PollyGPUDevicePtr *Allocation) {
-  dump_function();
-
-  CUDADevicePtr *DevPtr = (CUDADevicePtr *)Allocation->DevicePtr;
-  return (void *)DevPtr->Cuda;
-}
-
-static void freeContextCUDA(PollyGPUContext *Context) {
-  dump_function();
-
-  CUDAContext *Ctx = (CUDAContext *)Context->Context;
-  if (Ctx->Cuda) {
-    CuProfilerStopFcnPtr();
-    CuCtxDestroyFcnPtr(Ctx->Cuda);
-    free(Ctx);
-    free(Context);
-  }
-
-  dlclose(HandleCuda);
-  dlclose(HandleCudaRT);
-}
-
-#endif /* HAS_LIBCUDART */
-/******************************************************************************/
-/*                                    API                                     */
-/******************************************************************************/
-
-PollyGPUContext *polly_initContext() {
-  DebugMode = getenv("POLLY_DEBUG") != 0;
-  CacheMode = getenv("POLLY_NOCACHE") == 0;
-
-  dump_function();
-
-  PollyGPUContext *Context;
-
-  switch (Runtime) {
-#ifdef HAS_LIBCUDART
-  case RUNTIME_CUDA:
-    Context = initContextCUDA();
-    break;
-#endif /* HAS_LIBCUDART */
-#ifdef HAS_LIBOPENCL
-  case RUNTIME_CL:
-    Context = initContextCL();
-    break;
-#endif /* HAS_LIBOPENCL */
-  default:
-    err_runtime();
-  }
-
-  return Context;
-}
-
-void polly_freeKernel(PollyGPUFunction *Kernel) {
-  dump_function();
-
-  switch (Runtime) {
-#ifdef HAS_LIBCUDART
-  case RUNTIME_CUDA:
-    freeKernelCUDA(Kernel);
-    break;
-#endif /* HAS_LIBCUDART */
-#ifdef HAS_LIBOPENCL
-  case RUNTIME_CL:
-    freeKernelCL(Kernel);
-    break;
-#endif /* HAS_LIBOPENCL */
-  default:
-    err_runtime();
-  }
-}
-
-PollyGPUFunction *polly_getKernel(const char *BinaryBuffer,
-                                  const char *KernelName) {
-  dump_function();
-
-  PollyGPUFunction *Function;
-
-  switch (Runtime) {
-#ifdef HAS_LIBCUDART
-  case RUNTIME_CUDA:
-    Function = getKernelCUDA(BinaryBuffer, KernelName);
-    break;
-#endif /* HAS_LIBCUDART */
-#ifdef HAS_LIBOPENCL
-  case RUNTIME_CL:
-    Function = getKernelCL(BinaryBuffer, KernelName);
-    break;
-#endif /* HAS_LIBOPENCL */
-  default:
-    err_runtime();
-  }
-
-  return Function;
-}
-
-void polly_copyFromHostToDevice(void *HostData, PollyGPUDevicePtr *DevData,
-                                long MemSize) {
-  dump_function();
-
-  switch (Runtime) {
-#ifdef HAS_LIBCUDART
-  case RUNTIME_CUDA:
-    copyFromHostToDeviceCUDA(HostData, DevData, MemSize);
-    break;
-#endif /* HAS_LIBCUDART */
-#ifdef HAS_LIBOPENCL
-  case RUNTIME_CL:
-    copyFromHostToDeviceCL(HostData, DevData, MemSize);
-    break;
-#endif /* HAS_LIBOPENCL */
-  default:
-    err_runtime();
-  }
-}
-
-void polly_copyFromDeviceToHost(PollyGPUDevicePtr *DevData, void *HostData,
-                                long MemSize) {
-  dump_function();
-
-  switch (Runtime) {
-#ifdef HAS_LIBCUDART
-  case RUNTIME_CUDA:
-    copyFromDeviceToHostCUDA(DevData, HostData, MemSize);
-    break;
-#endif /* HAS_LIBCUDART */
-#ifdef HAS_LIBOPENCL
-  case RUNTIME_CL:
-    copyFromDeviceToHostCL(DevData, HostData, MemSize);
-    break;
-#endif /* HAS_LIBOPENCL */
-  default:
-    err_runtime();
-  }
-}
-
-void polly_launchKernel(PollyGPUFunction *Kernel, unsigned int GridDimX,
-                        unsigned int GridDimY, unsigned int BlockDimX,
-                        unsigned int BlockDimY, unsigned int BlockDimZ,
-                        void **Parameters) {
-  dump_function();
-
-  switch (Runtime) {
-#ifdef HAS_LIBCUDART
-  case RUNTIME_CUDA:
-    launchKernelCUDA(Kernel, GridDimX, GridDimY, BlockDimX, BlockDimY,
-                     BlockDimZ, Parameters);
-    break;
-#endif /* HAS_LIBCUDART */
-#ifdef HAS_LIBOPENCL
-  case RUNTIME_CL:
-    launchKernelCL(Kernel, GridDimX, GridDimY, BlockDimX, BlockDimY, BlockDimZ,
-                   Parameters);
-    break;
-#endif /* HAS_LIBOPENCL */
-  default:
-    err_runtime();
-  }
-}
-
-void polly_freeDeviceMemory(PollyGPUDevicePtr *Allocation) {
-  dump_function();
-
-  switch (Runtime) {
-#ifdef HAS_LIBCUDART
-  case RUNTIME_CUDA:
-    freeDeviceMemoryCUDA(Allocation);
-    break;
-#endif /* HAS_LIBCUDART */
-#ifdef HAS_LIBOPENCL
-  case RUNTIME_CL:
-    freeDeviceMemoryCL(Allocation);
-    break;
-#endif /* HAS_LIBOPENCL */
-  default:
-    err_runtime();
-  }
-}
-
-PollyGPUDevicePtr *polly_allocateMemoryForDevice(long MemSize) {
-  dump_function();
-
-  PollyGPUDevicePtr *DevData;
-
-  switch (Runtime) {
-#ifdef HAS_LIBCUDART
-  case RUNTIME_CUDA:
-    DevData = allocateMemoryForDeviceCUDA(MemSize);
-    break;
-#endif /* HAS_LIBCUDART */
-#ifdef HAS_LIBOPENCL
-  case RUNTIME_CL:
-    DevData = allocateMemoryForDeviceCL(MemSize);
-    break;
-#endif /* HAS_LIBOPENCL */
-  default:
-    err_runtime();
-  }
-
-  return DevData;
-}
-
-void *polly_getDevicePtr(PollyGPUDevicePtr *Allocation) {
-  dump_function();
-
-  void *DevPtr;
-
-  switch (Runtime) {
-#ifdef HAS_LIBCUDART
-  case RUNTIME_CUDA:
-    DevPtr = getDevicePtrCUDA(Allocation);
-    break;
-#endif /* HAS_LIBCUDART */
-#ifdef HAS_LIBOPENCL
-  case RUNTIME_CL:
-    DevPtr = getDevicePtrCL(Allocation);
-    break;
-#endif /* HAS_LIBOPENCL */
-  default:
-    err_runtime();
-  }
-
-  return DevPtr;
-}
-
-void polly_synchronizeDevice() {
-  dump_function();
-
-  switch (Runtime) {
-#ifdef HAS_LIBCUDART
-  case RUNTIME_CUDA:
-    synchronizeDeviceCUDA();
-    break;
-#endif /* HAS_LIBCUDART */
-#ifdef HAS_LIBOPENCL
-  case RUNTIME_CL:
-    synchronizeDeviceCL();
-    break;
-#endif /* HAS_LIBOPENCL */
-  default:
-    err_runtime();
-  }
-}
-
-void polly_freeContext(PollyGPUContext *Context) {
-  dump_function();
-
-  if (CacheMode)
-    return;
-
-  switch (Runtime) {
-#ifdef HAS_LIBCUDART
-  case RUNTIME_CUDA:
-    freeContextCUDA(Context);
-    break;
-#endif /* HAS_LIBCUDART */
-#ifdef HAS_LIBOPENCL
-  case RUNTIME_CL:
-    freeContextCL(Context);
-    break;
-#endif /* HAS_LIBOPENCL */
-  default:
-    err_runtime();
-  }
-}
-
-void polly_freeManaged(void *mem) {
-  dump_function();
-
-#ifdef HAS_LIBCUDART
-  freeManagedCUDA(mem);
-#else
-  fprintf(stderr, "No CUDA Runtime. Managed memory only supported by CUDA\n");
-  exit(-1);
-#endif
-}
-
-void *polly_mallocManaged(size_t size) {
-  dump_function();
-
-#ifdef HAS_LIBCUDART
-  return mallocManagedCUDA(size);
-#else
-  fprintf(stderr, "No CUDA Runtime. Managed memory only supported by CUDA\n");
-  exit(-1);
-#endif
-}
-
-/* Initialize GPUJIT with CUDA as runtime library. */
-PollyGPUContext *polly_initContextCUDA() {
-#ifdef HAS_LIBCUDART
-  Runtime = RUNTIME_CUDA;
-  return polly_initContext();
-#else
-  fprintf(stderr, "GPU Runtime was built without CUDA support.\n");
-  exit(-1);
-#endif /* HAS_LIBCUDART */
-}
-
-/* Initialize GPUJIT with OpenCL as runtime library. */
-PollyGPUContext *polly_initContextCL() {
-#ifdef HAS_LIBOPENCL
-  Runtime = RUNTIME_CL;
-  return polly_initContext();
-#else
-  fprintf(stderr, "GPU Runtime was built without OpenCL support.\n");
-  exit(-1);
-#endif /* HAS_LIBOPENCL */
-}
diff --git a/polly/tools/GPURuntime/GPUJIT.h b/polly/tools/GPURuntime/GPUJIT.h
deleted file mode 100644
index 61591f06b74a3..0000000000000
--- a/polly/tools/GPURuntime/GPUJIT.h
+++ /dev/null
@@ -1,123 +0,0 @@
-/******************************************************************************/
-/*                                                                            */
-/* Part of the LLVM Project, under the Apache License v2.0 with LLVM          */
-/* Exceptions.                                                                */
-/* See https://llvm.org/LICENSE.txt for license information.                  */
-/* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception                    */
-/*                                                                            */
-/******************************************************************************/
-/*                                                                            */
-/*  This file defines GPUJIT.                                                 */
-/*                                                                            */
-/******************************************************************************/
-
-#ifndef GPUJIT_H_
-#define GPUJIT_H_
-#include "stddef.h"
-
-/*
- * The following demonstrates how we can use the GPURuntime library to
- * execute a GPU kernel.
- *
- * char KernelString[] = "\n\
- *   .version 1.4\n\
- *   .target sm_10, map_f64_to_f32\n\
- *   .entry _Z8myKernelPi (\n\
- *   .param .u64 __cudaparm__Z8myKernelPi_data)\n\
- *   {\n\
- *     .reg .u16 %rh<4>;\n\
- *     .reg .u32 %r<5>;\n\
- *     .reg .u64 %rd<6>;\n\
- *     cvt.u32.u16     %r1, %tid.x;\n\
- *     mov.u16         %rh1, %ctaid.x;\n\
- *     mov.u16         %rh2, %ntid.x;\n\
- *     mul.wide.u16    %r2, %rh1, %rh2;\n\
- *     add.u32         %r3, %r1, %r2;\n\
- *     ld.param.u64    %rd1, [__cudaparm__Z8myKernelPi_data];\n\
- *     cvt.s64.s32     %rd2, %r3;\n\
- *     mul.wide.s32    %rd3, %r3, 4;\n\
- *     add.u64         %rd4, %rd1, %rd3;\n\
- *     st.global.s32   [%rd4+0], %r3;\n\
- *     exit;\n\
- *   }\n\
- * ";
- *
- * const char *Entry = "_Z8myKernelPi";
- *
- * int main() {
- *   PollyGPUFunction *Kernel;
- *   PollyGPUContext *Context;
- *   PollyGPUDevicePtr *DevArray;
- *   int *HostData;
- *   int MemSize;
- *
- *   int GridX = 8;
- *   int GridY = 8;
- *
- *   int BlockX = 16;
- *   int BlockY = 16;
- *   int BlockZ = 1;
- *
- *   MemSize = 256*64*sizeof(int);
- *   Context = polly_initContext();
- *   DevArray = polly_allocateMemoryForDevice(MemSize);
- *   Kernel = polly_getKernel(KernelString, KernelName);
- *
- *   void *Params[1];
- *   void *DevPtr = polly_getDevicePtr(DevArray)
- *   Params[0] = &DevPtr;
- *
- *   polly_launchKernel(Kernel, GridX, GridY, BlockX, BlockY, BlockZ, Params);
- *
- *   polly_copyFromDeviceToHost(HostData, DevData, MemSize);
- *   polly_freeKernel(Kernel);
- *   polly_freeDeviceMemory(DevArray);
- *   polly_freeContext(Context);
- * }
- *
- */
-
-typedef enum PollyGPURuntimeT {
-  RUNTIME_NONE,
-  RUNTIME_CUDA,
-  RUNTIME_CL
-} PollyGPURuntime;
-
-typedef struct PollyGPUContextT PollyGPUContext;
-typedef struct PollyGPUFunctionT PollyGPUFunction;
-typedef struct PollyGPUDevicePtrT PollyGPUDevicePtr;
-
-typedef struct OpenCLContextT OpenCLContext;
-typedef struct OpenCLKernelT OpenCLKernel;
-typedef struct OpenCLDevicePtrT OpenCLDevicePtr;
-
-typedef struct CUDAContextT CUDAContext;
-typedef struct CUDAKernelT CUDAKernel;
-typedef struct CUDADevicePtrT CUDADevicePtr;
-
-PollyGPUContext *polly_initContextCUDA();
-PollyGPUContext *polly_initContextCL();
-PollyGPUFunction *polly_getKernel(const char *BinaryBuffer,
-                                  const char *KernelName);
-void polly_freeKernel(PollyGPUFunction *Kernel);
-void polly_copyFromHostToDevice(void *HostData, PollyGPUDevicePtr *DevData,
-                                long MemSize);
-void polly_copyFromDeviceToHost(PollyGPUDevicePtr *DevData, void *HostData,
-                                long MemSize);
-void polly_synchronizeDevice();
-void polly_launchKernel(PollyGPUFunction *Kernel, unsigned int GridDimX,
-                        unsigned int GridDimY, unsigned int BlockSizeX,
-                        unsigned int BlockSizeY, unsigned int BlockSizeZ,
-                        void **Parameters);
-void polly_freeDeviceMemory(PollyGPUDevicePtr *Allocation);
-void polly_freeContext(PollyGPUContext *Context);
-
-// Note that polly_{malloc/free}Managed are currently not used by Polly.
-// We use them in COSMO by replacing all malloc with polly_mallocManaged and all
-// frees with cudaFree, so we can get managed memory "automatically".
-// Needless to say, this is a hack.
-// Please make sure that this code is not present in Polly when 2018 rolls in.
-// If this is still present, ping Siddharth Bhat <siddu.druid@gmail.com>
-void *polly_mallocManaged(size_t size);
-void polly_freeManaged(void *mem);
-#endif /* GPUJIT_H_ */
diff --git a/polly/tools/GPURuntime/LICENSE.TXT b/polly/tools/GPURuntime/LICENSE.TXT
deleted file mode 100644
index a556c3a996cfc..0000000000000
--- a/polly/tools/GPURuntime/LICENSE.TXT
+++ /dev/null
@@ -1,310 +0,0 @@
-==============================================================================
-The LLVM Project is under the Apache License v2.0 with LLVM Exceptions:
-==============================================================================
-
-                                 Apache License
-                           Version 2.0, January 2004
-                        http://www.apache.org/licenses/
-
-    TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
-    1. Definitions.
-
-      "License" shall mean the terms and conditions for use, reproduction,
-      and distribution as defined by Sections 1 through 9 of this document.
-
-      "Licensor" shall mean the copyright owner or entity authorized by
-      the copyright owner that is granting the License.
-
-      "Legal Entity" shall mean the union of the acting entity and all
-      other entities that control, are controlled by, or are under common
-      control with that entity. For the purposes of this definition,
-      "control" means (i) the power, direct or indirect, to cause the
-      direction or management of such entity, whether by contract or
-      otherwise, or (ii) ownership of fifty percent (50%) or more of the
-      outstanding shares, or (iii) beneficial ownership of such entity.
-
-      "You" (or "Your") shall mean an individual or Legal Entity
-      exercising permissions granted by this License.
-
-      "Source" form shall mean the preferred form for making modifications,
-      including but not limited to software source code, documentation
-      source, and configuration files.
-
-      "Object" form shall mean any form resulting from mechanical
-      transformation or translation of a Source form, including but
-      not limited to compiled object code, generated documentation,
-      and conversions to other media types.
-
-      "Work" shall mean the work of authorship, whether in Source or
-      Object form, made available under the License, as indicated by a
-      copyright notice that is included in or attached to the work
-      (an example is provided in the Appendix below).
-
-      "Derivative Works" shall mean any work, whether in Source or Object
-      form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other modifications
-      represent, as a whole, an original work of authorship. For the purposes
-      of this License, Derivative Works shall not include works that remain
-      separable from, or merely link (or bind by name) to the interfaces of,
-      the Work and Derivative Works thereof.
-
-      "Contribution" shall mean any work of authorship, including
-      the original version of the Work and any modifications or additions
-      to that Work or Derivative Works thereof, that is intentionally
-      submitted to Licensor for inclusion in the Work by the copyright owner
-      or by an individual or Legal Entity authorized to submit on behalf of
-      the copyright owner. For the purposes of this definition, "submitted"
-      means any form of electronic, verbal, or written communication sent
-      to the Licensor or its representatives, including but not limited to
-      communication on electronic mailing lists, source code control systems,
-      and issue tracking systems that are managed by, or on behalf of, the
-      Licensor for the purpose of discussing and improving the Work, but
-      excluding communication that is conspicuously marked or otherwise
-      designated in writing by the copyright owner as "Not a Contribution."
-
-      "Contributor" shall mean Licensor and any individual or Legal Entity
-      on behalf of whom a Contribution has been received by Licensor and
-      subsequently incorporated within the Work.
-
-    2. Grant of Copyright License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      copyright license to reproduce, prepare Derivative Works of,
-      publicly display, publicly perform, sublicense, and distribute the
-      Work and such Derivative Works in Source or Object form.
-
-    3. Grant of Patent License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      (except as stated in this section) patent license to make, have made,
-      use, offer to sell, sell, import, and otherwise transfer the Work,
-      where such license applies only to those patent claims licensable
-      by such Contributor that are necessarily infringed by their
-      Contribution(s) alone or by combination of their Contribution(s)
-      with the Work to which such Contribution(s) was submitted. If You
-      institute patent litigation against any entity (including a
-      cross-claim or counterclaim in a lawsuit) alleging that the Work
-      or a Contribution incorporated within the Work constitutes direct
-      or contributory patent infringement, then any patent licenses
-      granted to You under this License for that Work shall terminate
-      as of the date such litigation is filed.
-
-    4. Redistribution. You may reproduce and distribute copies of the
-      Work or Derivative Works thereof in any medium, with or without
-      modifications, and in Source or Object form, provided that You
-      meet the following conditions:
-
-      (a) You must give any other recipients of the Work or
-          Derivative Works a copy of this License; and
-
-      (b) You must cause any modified files to carry prominent notices
-          stating that You changed the files; and
-
-      (c) You must retain, in the Source form of any Derivative Works
-          that You distribute, all copyright, patent, trademark, and
-          attribution notices from the Source form of the Work,
-          excluding those notices that do not pertain to any part of
-          the Derivative Works; and
-
-      (d) If the Work includes a "NOTICE" text file as part of its
-          distribution, then any Derivative Works that You distribute must
-          include a readable copy of the attribution notices contained
-          within such NOTICE file, excluding those notices that do not
-          pertain to any part of the Derivative Works, in at least one
-          of the following places: within a NOTICE text file distributed
-          as part of the Derivative Works; within the Source form or
-          documentation, if provided along with the Derivative Works; or,
-          within a display generated by the Derivative Works, if and
-          wherever such third-party notices normally appear. The contents
-          of the NOTICE file are for informational purposes only and
-          do not modify the License. You may add Your own attribution
-          notices within Derivative Works that You distribute, alongside
-          or as an addendum to the NOTICE text from the Work, provided
-          that such additional attribution notices cannot be construed
-          as modifying the License.
-
-      You may add Your own copyright statement to Your modifications and
-      may provide additional or different license terms and conditions
-      for use, reproduction, or distribution of Your modifications, or
-      for any such Derivative Works as a whole, provided Your use,
-      reproduction, and distribution of the Work otherwise complies with
-      the conditions stated in this License.
-
-    5. Submission of Contributions. Unless You explicitly state otherwise,
-      any Contribution intentionally submitted for inclusion in the Work
-      by You to the Licensor shall be under the terms and conditions of
-      this License, without any additional terms or conditions.
-      Notwithstanding the above, nothing herein shall supersede or modify
-      the terms of any separate license agreement you may have executed
-      with Licensor regarding such Contributions.
-
-    6. Trademarks. This License does not grant permission to use the trade
-      names, trademarks, service marks, or product names of the Licensor,
-      except as required for reasonable and customary use in describing the
-      origin of the Work and reproducing the content of the NOTICE file.
-
-    7. Disclaimer of Warranty. Unless required by applicable law or
-      agreed to in writing, Licensor provides the Work (and each
-      Contributor provides its Contributions) on an "AS IS" BASIS,
-      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or conditions
-      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
-      PARTICULAR PURPOSE. You are solely responsible for determining the
-      appropriateness of using or redistributing the Work and assume any
-      risks associated with Your exercise of permissions under this License.
-
-    8. Limitation of Liability. In no event and under no legal theory,
-      whether in tort (including negligence), contract, or otherwise,
-      unless required by applicable law (such as deliberate and grossly
-      negligent acts) or agreed to in writing, shall any Contributor be
-      liable to You for damages, including any direct, indirect, special,
-      incidental, or consequential damages of any character arising as a
-      result of this License or out of the use or inability to use the
-      Work (including but not limited to damages for loss of goodwill,
-      work stoppage, computer failure or malfunction, or any and all
-      other commercial damages or losses), even if such Contributor
-      has been advised of the possibility of such damages.
-
-    9. Accepting Warranty or Additional Liability. While redistributing
-      the Work or Derivative Works thereof, You may choose to offer,
-      and charge a fee for, acceptance of support, warranty, indemnity,
-      or other liability obligations and/or rights consistent with this
-      License. However, in accepting such obligations, You may act only
-      on Your own behalf and on Your sole responsibility, not on behalf
-      of any other Contributor, and only if You agree to indemnify,
-      defend, and hold each Contributor harmless for any liability
-      incurred by, or claims asserted against, such Contributor by reason
-      of your accepting any such warranty or additional liability.
-
-    END OF TERMS AND CONDITIONS
-
-    APPENDIX: How to apply the Apache License to your work.
-
-      To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "[]"
-      replaced with your own identifying information. (Don't include
-      the brackets!)  The text should be enclosed in the appropriate
-      comment syntax for the file format. We also recommend that a
-      file or class name and description of purpose be included on the
-      same "printed page" as the copyright notice for easier
-      identification within third-party archives.
-
-    Copyright [yyyy] [name of copyright owner]
-
-    Licensed under the Apache License, Version 2.0 (the "License");
-    you may not use this file except in compliance with the License.
-    You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-    Unless required by applicable law or agreed to in writing, software
-    distributed under the License is distributed on an "AS IS" BASIS,
-    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-    See the License for the specific language governing permissions and
-    limitations under the License.
-
-
----- LLVM Exceptions to the Apache 2.0 License ----
-
-As an exception, if, as a result of your compiling your source code, portions
-of this Software are embedded into an Object form of such source code, you
-may redistribute such embedded portions in such Object form without complying
-with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
-
-In addition, if you combine or link compiled forms of this Software with
-software that is licensed under the GPLv2 ("Combined Software") and if a
-court of competent jurisdiction determines that the patent provision (Section
-3), the indemnity provision (Section 9) or other Section of the License
-conflicts with the conditions of the GPLv2, you may retroactively and
-prospectively choose to deem waived or otherwise exclude such Section(s) of
-the License, but only in their entirety and only with respect to the Combined
-Software.
-
-==============================================================================
-Software from third parties included in the LLVM Project:
-==============================================================================
-The LLVM Project contains third party software which is under different license
-terms. All such code will be identified clearly using at least one of two
-mechanisms:
-1) It will be in a separate directory tree with its own `LICENSE.txt` or
-   `LICENSE` file at the top containing the specific license and restrictions
-   which apply to that software, or
-2) It will contain specific license and restriction terms at the top of every
-   file.
-
-==============================================================================
-Legacy LLVM License (https://llvm.org/docs/DeveloperPolicy.html#legacy):
-==============================================================================
-
-The GPURuntime library is dual licensed under both the University of Illinois
-"BSD-Like" license and the MIT license.  As a user of this code you may choose
-to use it under either license.  As a contributor, you agree to allow your code
-to be used under both.
-
-Full text of the relevant licenses is included below.
-
-==============================================================================
-
-University of Illinois/NCSA
-Open Source License
-
-Copyright (c) 2009-2019 by the contributors listed in CREDITS.TXT
-
-All rights reserved.
-
-Developed by:
-
-    Polly Team
-
-    http://polly.llvm.org
-
-Permission is hereby granted, free of charge, to any person obtaining a copy of
-this software and associated documentation files (the "Software"), to deal with
-the Software without restriction, including without limitation the rights to
-use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
-of the Software, and to permit persons to whom the Software is furnished to do
-so, subject to the following conditions:
-
-    * Redistributions of source code must retain the above copyright notice,
-      this list of conditions and the following disclaimers.
-
-    * Redistributions in binary form must reproduce the above copyright notice,
-      this list of conditions and the following disclaimers in the
-      documentation and/or other materials provided with the distribution.
-
-    * Neither the names of the LLVM Team, University of Illinois at
-      Urbana-Champaign, nor the names of its contributors may be used to
-      endorse or promote products derived from this Software without specific
-      prior written permission.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
-FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
-CONTRIBUTORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS WITH THE
-SOFTWARE.
-
-==============================================================================
-
-Copyright (c) 2009-2016 by the contributors listed in CREDITS.TXT
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in
-all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.
-
diff --git a/polly/www/documentation/gpgpucodegen.html b/polly/www/documentation/gpgpucodegen.html
deleted file mode 100644
index 94038cf48aa95..0000000000000
--- a/polly/www/documentation/gpgpucodegen.html
+++ /dev/null
@@ -1,229 +0,0 @@
-<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
-        "http://www.w3.org/TR/html4/strict.dtd">
-<!-- Material used from: HTML 4.01 specs: http://www.w3.org/TR/html401/ -->
-<html>
-<head>
-  <META http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
-  <title>Polly - GPGPU Code Generation</title>
-  <link type="text/css" rel="stylesheet" href="../menu.css">
-  <link type="text/css" rel="stylesheet" href="../content.css">
-</head>
-<body>
-<div id="box">
-<!--#include virtual="../menu.html.incl"-->
-<div id="content">
-  <!--*********************************************************************-->
-  <h1>Polly - GPGPU Code Generation</h1>
-  <!--*********************************************************************-->
-<p><em>WARNING: This project was part of the Google Summer of Code 2012.
-It is currently not finished, but it is in the design and implementation stage.
-The ideas/plans described here may not yet be implemented in Polly and may
-change later on.</em></p>
-
-This project adds GPGPU code generation feature to Polly.
-
-<h2>Objective</h2>
-<p>The overall objective of this GSoC project is to create a preliminary
-   implementation of GPGPU code generation for Polly. With this addition, users
-   can parallelize some perfectly nested loops with Polly to execute on a
-   heterogeneous platform, composed of CPU and GPU.</p>
-<p>There are several successful projects about automatic source-to-source gpu
-   code transformation. C-to-CUDA[1] uses the standard Pluto algorithms for
-   computing an affine schedule and then applies a wavefront transformation to
-   obtain one sequential and n-1 parallel loops. The parallel loops are then
-   mapped onto the blocks and threads of GPU. PPCG[2] introduces some advanced
-   algorithms which can expose much more parallelism than other methods . And It
-   also introduces affine partition heuristics and code generation algorithms
-   for locality enhancement in the registers and shared memory.</p>
-<p>Since automatic GPGPU code generation is quite a complex problem and what we
-   target is a low-level intermediate representation, LLVM IR, rather than a
-   high-level language source, it is important for us to set a proper objective
-   as a start step to give a complete solution to GPGPU code generation for LLVM
-   IR.</p>
-<p>Firstly, we plan to target two kinds of relatively simple test cases. One is
-   comprised of pure parallel and perfectly nested loops, like the following
-   code.</p>
-<pre>
-parfor(int i=0 to M)
-  parfor(int j=0 to N)
-    LoopBody(i, j);
-</pre>
-<p>Another one is that all the loops in it are parallel except the inner-most
-   one, just like this:</p>
-<pre>
-parfor(int i=0 to M)
-  parfor(int j=0 to N)
-    non-parfor(int k=0 to K)
-      LoopBody(i, j, k);
-</pre>
-<p>The LoopBody part should be limited to instructions or functions calls
-   (intrinsics) which can be handled by LLVM's NVPTX backend.</p>
-<p>On the other hand, we focus on building a preliminary and scalable framework
-   of GPGPU code generation for polly. Thus we plan to employ relatively simple
-   tiling and mapping algorithms and optimize them later.</p>
-<h2>Work Flow</h2>
-<h3>GPGPU Code Generation In General</h3>
-<p>C-to-CUDA[1] and PPCG[2] propose similar steps to solve the automatic GPGPU
-   code generation problem.</p>
-<li>Look for parallel loops.</li>
-<li>Create a polyhedral model from the loops.</li>
-<li>Tile and map the loops to GPU blocks and threads.</li>
-<li>Determine where to place the data.</li>
-<h3>What has been done in Polly</h3>
-<p>Polly has implemented the 1st, 2nd and part of the 3rd of the above steps and
-   many other analysis and transformation passes.</p>
-<h3>What to do in Polly</h3>
-<p>Unlike many source-to-source optimizers such as C-to-CUDA and PPCG, Polly is
-   a low-level optimizer, which means we can't use a source-level compiler
-   (e.g. NVCC) to generate the final assembly for the device. We need manually
-   insert device driver API calls to execute the generated kernel assembly
-   text.</p>
-<p>In this project, we assume that the device driver library has provided an
-   interface to launch kernels in the form of assembly text. Fortunately, most
-   of the mainstream GPU vendors provide such a feature in their products (see
-   ptxjit of NVIDIA GPUs and CAL of AMD GPUs). Generally speaking, what we
-   are going to do in Polly is:</p>
-<li>Find a way to tile the parallel loops.</li>
-<li>Find a way to extract the loop body and transform it into thread-centric
-    parallel code.</li>
-<li>Find a way to store/load the thread-centric code into/from a device module.
-<li>Find a way to pass the target machine information and generate code of the
-    device module for the target.
-<li>Find a way to map the tiled loop to GPU blocks and threads.</li>
-<li>Find a way to insert CUDA synchronization operations on-demand.
-<li>Find a way to generate the memory copy operations between a host and a
-    device.</li>
-<li>Implement/Wrap a runtime library to serve as the execution engine for the
-    generated device code.</li>
-
-<h3>The Work Flow</h3>
-<p>In this section, we assume that the host cpu is X86 and the device is NVIDIA
-   CUDA-compatible. we will use the following test case to describe our work
-   flow.</p>
-<pre>
-for(i = 0; i &lt; 128; i++)
-      for(j = 0; j &lt; 128; j++)
-              A[i][j] = i*128 + j;
-</pre>
-<p>The work flow of our code generator is as follows.</p>
-<p>1.We first use Polly's jscop file importer to get a wanted 4-level parallel
-   tiled code.</p>
-The "schedule" part of the pre-optimization jscop file is as the following:
-<pre>
-"schedule" : "{ Stmt_for_body3[i0, i1] -&gt; schedule[0, i0, 0, i1, 0] }"
-</pre>
-The jscop file describing the tiling transformation is:
-<pre>
-"schedule" : "{ Stmt_for_body3[i0, i1] -&gt; schedule[0, o0, o1, o2, o3]:
-              o0 &gt;= 0 and o0 &lt;= 7 and o1 &gt;= 0 and o1 &lt;= 15 and
-              o2 &gt;= 0 and o2 &lt;= 7 and o3 &gt;= 0 and o3 &lt;= 15 and
-              i0 = 16o0 + o1 and i1 = 16o2 + o3 }"
-</pre>
-We can test the schedule with the following command line.
-<pre>
-opt -load /path/to/polly/build/LLVMPolly.so -basic-aa -polly-import-jscop
-    -polly-ast -analyze -q ./test.ll
-    -polly-import-jscop-postfix=transformed+gpu
-</pre>
-The output of this schedule is:
-<pre>
-for (c2=0;c2&lt;=7;c2++) {
-  for (c3=0;c3&lt;=15;c3++) {
-    for (c4=0;c4&lt;=7;c4++) {
-      for (c5=0;c5&lt;=15;c5++) {
-        Stmt_for_body3(16*c2+c3,16*c4+c5);
-      }
-    }
-  }
-}
-</pre>
-Now we get a 4-dimensional parallel loops with a single SCoP statement in it.
-<p>2.We then extract the loop body (or the inner-most non-parallel loop) into a
-   LLVM function, tagging it with PTX_Kernel call convention.</p>
-<p>3.We extract the PTX_kernel function into a temporary module, set the target
-   triple (e.g. nvptx64-unknown-linux) for the module, transform the temporary
-   module into a string, store it in the original module and erase the
-   PTX_kernel function.</p>
-<p>4.We replace the loops with their GPGPU counterpart. The GPGPU part of code
-   is composed of a call to the llvm.codegen intrinsic and function calls to our
-   GPU runtime library.</p>
-<p>5.Finally, we generate the executable program with <em>llc</em> or run the
-   optimized LLVM IRs with a JIT compiler like  <em>lli</em>.</p>
-<h2>Usage</h2>
-<p>1. Apply the llvm.codegen intrinsic patch to LLVM code base.</p>
-<pre>cd /path/to/llvm/source
-git am /path/to/polly/source/utils/0001-Add-llvm.codegen-intrinsic.patch</pre>
-<p>2. Build the test case.</p>
-<pre>/path/to/polly/source/test/create_ll.sh test.c</pre>
-<p>3. Get and edit the jscop file (take function "gpu_codegen" as an example).
-</p>
-<pre>opt -load /path/to/polly/build/lib/LLVMPolly.so -basic-aa
-    -polly-export-jscop ./test.ll
-cp gpu_codegen___%for.cond---%for.end8.jscop
-   gpu_codegen___%for.cond---%for.end8.jscop.transformed+gpu
-vi gpu_codegen___%for.cond---%for.end8.jscop.transformed+gpu</pre>
-<p><em>(Please refer to section "The Work Flow" on how to edit the "schedule"
-       part of a statement)</em></p>
-<p>4. Optimize the code with GPGPU code generation.</p>
-<pre>opt -load /path/to/polly/build/lib/LLVMPolly.so -basic-aa
-    -polly-import-jscop-postfix=transformed+gpu -enable-polly-gpgpu
-    -polly-gpgpu-triple=nvptx64-unknown-unknown -polly-codegen ./test.ll -S
-    -o test.gpued.ll</pre>
-<p>5. Build the final assembly and executable.</p>
-<pre>llc test.gpued.ll -o test.s
-gcc test.s -lGPURuntime -o test</pre>
-<p><em>(Please make sure that LD_LIBRARY_PATH is set properly so that
-        /path/to/polly/build/lib/libGPURuntime.so is visible to gcc.)</em></p>
-<h2>TODO List</h2>
-
-<table class="wikitable" cellpadding="2">
-<tbody>
-<tr style="background: rgb(239, 239, 239)">
-  <th width="400px"> Tasks</th>
-  <th width="150px"> Status </th>
-  <th> Owner </th>
-</tr>
-<tr>
-<th align="left">Tiling the Parallel Loops with An External Jscop File</th>
-<td align="center" class='open'>Open, In Design</td>
-<td>Yabin Hu</td>
-</tr>
-<tr>
-<th align="left">GPU Runtime Library Implementation</th>
-<td align="center" class='inprogress'>Coding Finished, In Reviewing</td>
-<td></td>
-</tr>
-<tr>
-<th align="left">llvm.codegen Intrinsic Implementation</th>
-<td align="center" class='inprogress'>Coding Finished, To Be Reviewed</td>
-<td></td>
-</tr>
-<tr>
-<th align="left">Code Generation For Host</th>
-<td align="center" class='inprogress'>50% Done</td>
-<td></td>
-</tr>
-
-</tbody></table>
-
-<h2>References</h2>
-<li type="1" value="1">
-<em>Automatic C-to-CUDA Code Generation for Affine Programs. </em><br />
-    Muthu Manikandan Baskaran, J. Ramanujam and P. Sadayappan.<br />
-    International Conference on Compiler Construction (CC) 2010.<br />
-</li>
-<li type="1"><em>PPCG Project</em><br />
-<a href="http://freecode.com/projects/ppcg">http://freecode.com/projects/ppcg
-</a></li>
-<li type="1">
-<em>Where is the Data? Why You Cannot Debate GPU vs. CPU Performance Without the
-    Answer. </em><br />
-  Chris Gregg and Kim Hazelwood<br />
-  International Symposium on Performance Analysis of Systems and Software
-  (ISPASS) 2011.
-</li>
-<p></p>
-</div>
-</div>
-</body>
-</html>
diff --git a/polly/www/index.html b/polly/www/index.html
index 6c583004a1cae..45bafd6157972 100644
--- a/polly/www/index.html
+++ b/polly/www/index.html
@@ -28,8 +28,7 @@ <h1>About Polly</h1>
   on integer polyhedra to analyze and optimize the memory access pattern of a
   program. We currently perform classical loop transformations, especially
   tiling and loop fusion to improve data-locality. Polly can also exploit
-  OpenMP level parallelism, expose SIMDization opportunities. Work has also be
-  done in the area of automatic GPU code generation.</p>
+  OpenMP level parallelism, expose SIMDization opportunities.</p>
 
   For many users, however, it's not the existing optimizations in Polly that are
   of most interest, but the new analyses and optimizations enabled by the Polly
diff --git a/polly/www/todo.html b/polly/www/todo.html
index 4ada733e07b28..8cc69c6cd51e4 100644
--- a/polly/www/todo.html
+++ b/polly/www/todo.html
@@ -342,14 +342,6 @@ <h3 id="phase3"> Phase 3 - Improve Robustness, Interoperability and
 </td><td>Johannes
 <tr>
 <th align="left"> <a
-href="https://polly.llvm.org/documentation/gpgpucodegen.html">GPGPU Code
-Generation</a>
-</th><td class="niceinprogress">in progress
-</td><td>
-Yabin
-</td></tr>
-<tr>
-<th align="left"> <a
 href="https://polly.llvm.org/documentation/memaccess.html">Allow optimizers to
 change memory access functions</a>
 </th><td class="done"> Done
diff --git a/runtimes/cmake/Modules/WarningFlags.cmake b/runtimes/cmake/Modules/WarningFlags.cmake
new file mode 100644
index 0000000000000..d06409841dc9d
--- /dev/null
+++ b/runtimes/cmake/Modules/WarningFlags.cmake
@@ -0,0 +1,77 @@
+include(HandleFlags)
+
+# Warning flags ===============================================================
+function(cxx_add_warning_flags target enable_werror enable_pedantic)
+  target_compile_definitions(${target} PUBLIC -D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
+  if (MSVC)
+    # -W4 is the cl.exe/clang-cl equivalent of -Wall. (In cl.exe and clang-cl,
+    # -Wall is equivalent to -Weverything in GCC style compiler drivers.)
+    target_add_compile_flags_if_supported(${target} PRIVATE -W4)
+  else()
+    target_add_compile_flags_if_supported(${target} PRIVATE -Wall)
+  endif()
+  # TODO: Should -Wconversion be enabled?
+  target_add_compile_flags_if_supported(${target} PRIVATE
+      -Wextra
+      -Wnewline-eof
+      -Wshadow
+      -Wwrite-strings
+      -Wno-unused-parameter
+      -Wno-long-long
+      -Werror=return-type
+      -Wextra-semi
+      -Wundef
+      -Wunused-template
+      -Wformat-nonliteral
+      )
+
+  if ("${CMAKE_CXX_COMPILER_ID}" MATCHES "Clang")
+    target_add_compile_flags_if_supported(${target} PRIVATE
+      -Wno-user-defined-literals
+      -Wno-covered-switch-default
+      -Wno-suggest-override
+    )
+    if (LIBCXX_TARGETING_CLANG_CL)
+      target_add_compile_flags_if_supported(${target} PRIVATE
+        -Wno-c++98-compat
+        -Wno-c++98-compat-pedantic
+        -Wno-c++11-compat
+        -Wno-undef
+        -Wno-reserved-id-macro
+        -Wno-gnu-include-next
+        -Wno-gcc-compat # For ignoring "'diagnose_if' is a clang extension" warnings
+        -Wno-zero-as-null-pointer-constant # FIXME: Remove this and fix all occurrences.
+        -Wno-deprecated-dynamic-exception-spec # For auto_ptr
+        -Wno-sign-conversion
+        -Wno-old-style-cast
+        -Wno-deprecated # FIXME: Remove this and fix all occurrences.
+        -Wno-shift-sign-overflow # FIXME: Why do we need this with clang-cl but not clang?
+        -Wno-double-promotion # FIXME: remove me
+      )
+    endif()
+
+  elseif("${CMAKE_CXX_COMPILER_ID}" MATCHES "GNU")
+
+    target_add_compile_flags_if_supported(${target} PRIVATE
+      -Wstrict-aliasing=2
+      -Wstrict-overflow=4
+      -Wno-attributes
+      -Wno-literal-suffix
+      -Wno-c++14-compat
+      -Wno-noexcept-type
+      -Wno-suggest-override
+      )
+
+  endif()
+  if (${enable_werror})
+    target_add_compile_flags_if_supported(${target} PRIVATE -Werror)
+    target_add_compile_flags_if_supported(${target} PRIVATE -WX)
+  else()
+    # TODO(EricWF) Remove this. We shouldn't be suppressing errors when -Werror is
+    # added elsewhere.
+    target_add_compile_flags_if_supported(${target} PRIVATE -Wno-error)
+  endif()
+  if (${enable_pedantic})
+    target_add_compile_flags_if_supported(${target} PRIVATE -pedantic)
+  endif()
+endfunction()
diff --git a/third-party/unittest/CMakeLists.txt b/third-party/unittest/CMakeLists.txt
index 302e9e0e84407..74a523b6dfcc0 100644
--- a/third-party/unittest/CMakeLists.txt
+++ b/third-party/unittest/CMakeLists.txt
@@ -40,7 +40,13 @@ endif()
 
 # Do not build unittest libraries automatically, they will be pulled in
 # by unittests if these are built.
+
+set(BUILDTREE_ONLY BUILDTREE_ONLY)
 set(EXCLUDE_FROM_ALL ON)
+if (LLVM_INSTALL_GTEST)
+  set(EXCLUDE_FROM_ALL OFF)
+  set(BUILDTREE_ONLY "")
+endif ()
 
 add_llvm_library(llvm_gtest
   googletest/src/gtest-all.cc
@@ -53,7 +59,7 @@ add_llvm_library(llvm_gtest
   Support # Depends on llvm::raw_ostream
 
   # This is a library meant only for the build tree.
-  BUILDTREE_ONLY
+  ${BUILDTREE_ONLY}
 )
 
 # The googletest and googlemock sources don't presently use the 'override'
@@ -69,12 +75,22 @@ if (NOT LLVM_ENABLE_THREADS)
 endif ()
 
 target_include_directories(llvm_gtest
-  PUBLIC googletest/include googlemock/include
+  PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/googletest/include>
+         $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/googlemock/include>
+         $<INSTALL_INTERFACE:include/llvm-gtest/>
+         $<INSTALL_INTERFACE:include/llvm-gmock/>
   PRIVATE googletest googlemock
   )
 
 add_subdirectory(UnitTestMain)
 
+if (LLVM_INSTALL_GTEST)
+  install(TARGETS llvm_gtest llvm_gtest_main LLVMTestingSupport LLVMTestingAnnotations
+	  ARCHIVE DESTINATION "lib${LLVM_LIBDIR_SUFFIX}" COMPONENT llvm_gtest)
+  install(DIRECTORY googletest/include/gtest/ DESTINATION include/llvm-gtest/gtest/ COMPONENT llvm_gtest)
+  install(DIRECTORY googlemock/include/gmock/ DESTINATION include/llvm-gmock/gmock/ COMPONENT llvm_gtest)
+endif()
+
 # When LLVM_LINK_LLVM_DYLIB is enabled, libLLVM.so is added to the interface
 # link libraries for gtest and gtest_main.  This means that any target, like
 # unittests for example, that links against gtest will be forced to link
diff --git a/third-party/unittest/UnitTestMain/CMakeLists.txt b/third-party/unittest/UnitTestMain/CMakeLists.txt
index 7a3ac8a0f43e6..729ea7e3fa7e2 100644
--- a/third-party/unittest/UnitTestMain/CMakeLists.txt
+++ b/third-party/unittest/UnitTestMain/CMakeLists.txt
@@ -1,3 +1,8 @@
+set(BUILDTREE_ONLY BUILDTREE_ONLY)
+if (LLVM_INSTALL_GTEST)
+  set(BUILDTREE_ONLY "")
+endif ()
+
 add_llvm_library(llvm_gtest_main
   TestMain.cpp
 
@@ -7,6 +12,5 @@ add_llvm_library(llvm_gtest_main
   LINK_COMPONENTS
   Support # Depends on llvm::cl
 
-  # This library is not meant to be in the install tree, only the build tree.
-  BUILDTREE_ONLY
+  ${BUILDTREE_ONLY}
   )
diff --git a/third-party/unittest/googletest/include/gtest/gtest-matchers.h b/third-party/unittest/googletest/include/gtest/gtest-matchers.h
index 667d16e6e7276..12584b70763f4 100644
--- a/third-party/unittest/googletest/include/gtest/gtest-matchers.h
+++ b/third-party/unittest/googletest/include/gtest/gtest-matchers.h
@@ -32,7 +32,7 @@
 // This file implements just enough of the matcher interface to allow
 // EXPECT_DEATH and friends to accept a matcher argument.
 
-// // IWYU pragma: private, include "gtest/gtest.h"
+// IWYU pragma: private, include "gtest/gtest.h"
 // IWYU pragma: friend gtest/.*
 // IWYU pragma: friend gmock/.*
 
diff --git a/utils/bazel/.bazelrc b/utils/bazel/.bazelrc
index b2b472b1e2d54..b0f137f9d1b43 100644
--- a/utils/bazel/.bazelrc
+++ b/utils/bazel/.bazelrc
@@ -22,6 +22,9 @@ build --dynamic_mode=off
 # based on compilation_mode.
 build --strip=never
 
+# Add layering check to all projects.
+build --features=layering_check
+
 ###############################################################################
 # Options to select different strategies for linking potential dependent
 # libraries. The default leaves it disabled.
diff --git a/utils/bazel/llvm-project-overlay/clang-tools-extra/clang-tidy/defs.bzl b/utils/bazel/llvm-project-overlay/clang-tools-extra/clang-tidy/defs.bzl
index 4234f363878b9..41c03aad871c7 100644
--- a/utils/bazel/llvm-project-overlay/clang-tools-extra/clang-tidy/defs.bzl
+++ b/utils/bazel/llvm-project-overlay/clang-tools-extra/clang-tidy/defs.bzl
@@ -21,6 +21,5 @@ def clang_tidy_library(name, **kwargs):
     kwargs["deps"] = kwargs.get("deps", []) + _common_library_deps
     cc_library(
         name = name,
-        alwayslink = True,
         **kwargs
     )
diff --git a/utils/bazel/llvm-project-overlay/clang-tools-extra/unittests/BUILD.bazel b/utils/bazel/llvm-project-overlay/clang-tools-extra/unittests/BUILD.bazel
index 54d607a6067a6..6cc88e172c840 100644
--- a/utils/bazel/llvm-project-overlay/clang-tools-extra/unittests/BUILD.bazel
+++ b/utils/bazel/llvm-project-overlay/clang-tools-extra/unittests/BUILD.bazel
@@ -29,6 +29,7 @@ cc_test(
         "//clang:frontend",
         "//clang:lex",
         "//clang:serialization",
+        "//clang:testing",
         "//clang:tooling",
         "//clang:tooling_core",
         "//clang:transformer",
diff --git a/utils/bazel/llvm-project-overlay/clang/BUILD.bazel b/utils/bazel/llvm-project-overlay/clang/BUILD.bazel
index 15c3cb4e21924..a7800ac4ea66a 100644
--- a/utils/bazel/llvm-project-overlay/clang/BUILD.bazel
+++ b/utils/bazel/llvm-project-overlay/clang/BUILD.bazel
@@ -1026,6 +1026,7 @@ cc_library(
     deps = [
         ":basic",
         ":frontend",
+        "//llvm:MC",
         "//llvm:Support",
         "//third-party/unittest:gtest",
     ],
diff --git a/utils/bazel/llvm-project-overlay/libc/BUILD.bazel b/utils/bazel/llvm-project-overlay/libc/BUILD.bazel
index 0a140038c6e0f..23c288db31ada 100644
--- a/utils/bazel/llvm-project-overlay/libc/BUILD.bazel
+++ b/utils/bazel/llvm-project-overlay/libc/BUILD.bazel
@@ -209,6 +209,7 @@ libc_support_library(
     hdrs = ["src/__support/CPP/atomic.h"],
     deps = [
         ":__support_cpp_type_traits",
+        ":__support_macros_attributes",
         ":libc_root",
     ],
 )
@@ -380,6 +381,7 @@ libc_support_library(
         ":__support_macros_attributes",
         ":__support_macros_properties_architectures",
         ":__support_macros_sanitizer",
+        ":errno",
         ":libc_root",
     ],
 )
@@ -761,6 +763,7 @@ libc_support_library(
         "__support_cpp_bit",
         "__support_cpp_type_traits",
         ":__support_common",
+        ":errno",
         ":libc_root",
     ],
 )
@@ -1484,16 +1487,18 @@ libc_support_library(
         "src/string/memory_utils/utils.h",
     ],
     textual_hdrs = [
+        "src/string/memory_utils/aarch64/memcmp_implementations.h",
+        "src/string/memory_utils/aarch64/memcpy_implementations.h",
         "src/string/memory_utils/bcmp_implementations.h",
         "src/string/memory_utils/bzero_implementations.h",
         "src/string/memory_utils/memcmp_implementations.h",
         "src/string/memory_utils/memcpy_implementations.h",
-        "src/string/memory_utils/aarch64/memcpy_implementations.h",
-        "src/string/memory_utils/x86_64/memcpy_implementations.h",
         "src/string/memory_utils/memmove_implementations.h",
         "src/string/memory_utils/memset_implementations.h",
         "src/string/memory_utils/strcmp_implementations.h",
         "src/string/memory_utils/strstr_implementations.h",
+        "src/string/memory_utils/x86_64/memcmp_implementations.h",
+        "src/string/memory_utils/x86_64/memcpy_implementations.h",
     ],
     deps = [
         ":__support_common",
diff --git a/utils/bazel/llvm-project-overlay/libc/test/libc_test_rules.bzl b/utils/bazel/llvm-project-overlay/libc/test/libc_test_rules.bzl
index 302fdfabc3172..b4e473d932fdd 100644
--- a/utils/bazel/llvm-project-overlay/libc/test/libc_test_rules.bzl
+++ b/utils/bazel/llvm-project-overlay/libc/test/libc_test_rules.bzl
@@ -24,10 +24,11 @@ def libc_test(name, srcs, libc_function_deps, deps = [], **kwargs):
       deps: The list of other libraries to be linked in to the test target.
       **kwargs: Attributes relevant for a cc_test. For example, name, srcs.
     """
+    all_function_deps = libc_function_deps + ["//libc:errno"]
     native.cc_test(
         name = name,
         srcs = srcs,
-        deps = [d + INTERNAL_SUFFIX for d in libc_function_deps] + [
+        deps = [d + INTERNAL_SUFFIX for d in all_function_deps] + [
             "//libc:libc_root",
             "//libc/test/UnitTest:LibcUnitTest",
         ] + deps,
diff --git a/utils/bazel/llvm-project-overlay/libc/test/src/math/libc_math_test_rules.bzl b/utils/bazel/llvm-project-overlay/libc/test/src/math/libc_math_test_rules.bzl
index 233bc15de2473..f8c3d9d92fc88 100644
--- a/utils/bazel/llvm-project-overlay/libc/test/src/math/libc_math_test_rules.bzl
+++ b/utils/bazel/llvm-project-overlay/libc/test/src/math/libc_math_test_rules.bzl
@@ -25,14 +25,14 @@ def math_test(name, hdrs = [], deps = [], **kwargs):
         srcs = [test_name + ".cpp"] + hdrs,
         libc_function_deps = ["//libc:func_name".replace("func_name", name)],
         deps = [
-            "//libc:__support_fputil_basic_operations",
             "//libc:__support_builtin_wrappers",
+            "//libc:__support_fputil_basic_operations",
             "//libc:__support_fputil_fenv_impl",
             "//libc:__support_fputil_float_properties",
             "//libc:__support_fputil_fp_bits",
-            "//libc:__support_uint128",
             "//libc:__support_fputil_manipulation_functions",
             "//libc:__support_fputil_nearest_integer_operations",
+            "//libc:__support_uint128",
             "//libc/test/UnitTest:fp_test_helpers",
         ] + deps,
         **kwargs
diff --git a/utils/bazel/llvm-project-overlay/libc/test/src/stdlib/BUILD.bazel b/utils/bazel/llvm-project-overlay/libc/test/src/stdlib/BUILD.bazel
index 23088e363401c..ca9887b8798f8 100644
--- a/utils/bazel/llvm-project-overlay/libc/test/src/stdlib/BUILD.bazel
+++ b/utils/bazel/llvm-project-overlay/libc/test/src/stdlib/BUILD.bazel
@@ -54,7 +54,6 @@ libc_test(
     ],
     deps = [
         "//libc:__support_fputil_fp_bits",
-        "//libc:errno",
     ],
 )
 
@@ -80,7 +79,7 @@ cc_library(
     deps = [
         "//libc:__support_cpp_limits",
         "//libc:__support_cpp_type_traits",
-        "//libc:errno",
+        "//libc:errno.__internal__",
         "//libc/test/UnitTest:LibcUnitTest",
     ],
 )
@@ -129,7 +128,6 @@ libc_test(
     ],
     deps = [
         "//libc:__support_fputil_fp_bits",
-        "//libc:errno",
         "//libc/utils/testutils:libc_test_utils",
     ],
 )
@@ -142,7 +140,6 @@ libc_test(
     ],
     deps = [
         "//libc:__support_fputil_fp_bits",
-        "//libc:errno",
         "//libc/utils/testutils:libc_test_utils",
     ],
 )
@@ -156,6 +153,5 @@ libc_test(
     deps = [
         "//libc:__support_fputil_fp_bits",
         "//libc:__support_uint128",
-        "//libc:errno",
     ],
 )
diff --git a/utils/bazel/llvm-project-overlay/libc/utils/MPFRWrapper/BUILD.bazel b/utils/bazel/llvm-project-overlay/libc/utils/MPFRWrapper/BUILD.bazel
index 3cd50bf5307d2..09d914449f000 100644
--- a/utils/bazel/llvm-project-overlay/libc/utils/MPFRWrapper/BUILD.bazel
+++ b/utils/bazel/llvm-project-overlay/libc/utils/MPFRWrapper/BUILD.bazel
@@ -27,6 +27,13 @@ cc_library(
     name = "mpfr_wrapper",
     srcs = ["MPFRUtils.cpp"],
     hdrs = ["MPFRUtils.h"],
+    # Disable layering check when using mpfr_system.
+    features = select(
+        {
+            "//conditions:default": [],
+            "//libc:mpfr_system": ["-layering_check"],
+        },
+    ),
     deps = [
         "//libc:__support_common",
         "//libc:__support_cpp_bit",
@@ -36,9 +43,9 @@ cc_library(
         "//libc:__support_fputil_fp_bits",
         "//libc:__support_fputil_platform_defs",
         "//libc:libc_root",
-        "//libc/utils/MPFRWrapper:mpfr_impl",
         "//libc/test/UnitTest:LibcUnitTest",
         "//libc/test/UnitTest:fp_test_helpers",
+        "//libc/utils/MPFRWrapper:mpfr_impl",
         "//libc/utils/testutils:libc_test_utils",
     ],
 )
diff --git a/utils/bazel/llvm-project-overlay/llvm/BUILD.bazel b/utils/bazel/llvm-project-overlay/llvm/BUILD.bazel
index 157e6c4cd671a..9707d1296debc 100644
--- a/utils/bazel/llvm-project-overlay/llvm/BUILD.bazel
+++ b/utils/bazel/llvm-project-overlay/llvm/BUILD.bazel
@@ -572,6 +572,7 @@ cc_library(
     name = "tblgen",
     srcs = glob([
         "utils/TableGen/*.cpp",
+        "utils/TableGen/*.inc",
         "utils/TableGen/GlobalISel/*.cpp",
 
         # Some tablegen sources include headers from MC, so these have to be
@@ -1527,7 +1528,6 @@ cc_library(
     hdrs = glob([
         "include/llvm/Transforms/IPO/*.h",
     ]) + [
-        "include/llvm-c/Transforms/PassManagerBuilder.h",
         "include/llvm/Transforms/IPO.h",
     ],
     copts = llvm_copts,
diff --git a/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel b/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel
index 7259cbead1371..869d7a9fe7c4a 100644
--- a/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel
+++ b/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel
@@ -5,6 +5,7 @@
 # Description:
 #   The MLIR "Multi-Level Intermediate Representation" Compiler Infrastructure
 
+load("@bazel_skylib//rules:write_file.bzl", "write_file")
 load(":tblgen.bzl", "gentbl_cc_library", "td_library")
 load(":linalggen.bzl", "genlinalg")
 load(
@@ -2223,6 +2224,7 @@ cc_library(
     deps = [
         ":AffineDialect",
         ":ArithDialect",
+        ":ArithUtils",
         ":BufferizationDialect",
         ":BufferizationTransforms",
         ":ComplexDialect",
@@ -2695,7 +2697,10 @@ cc_library(
         "lib/Dialect/Affine/Transforms/*.cpp",
         "lib/Dialect/Affine/Transforms/*.h",
     ]),
-    hdrs = ["include/mlir/Dialect/Affine/Passes.h"],
+    hdrs = [
+        "include/mlir/Dialect/Affine/Passes.h",
+        "include/mlir/Dialect/Affine/Transforms/Transforms.h",
+    ],
     includes = ["include"],
     deps = [
         ":AffineAnalysis",
@@ -3556,6 +3561,35 @@ cc_library(
     ],
 )
 
+cc_library(
+    name = "Debug",
+    srcs = glob([
+        "lib/Debug/*.cpp",
+        "lib/Debug/*.h",
+    ]),
+    hdrs = glob(["include/mlir/Debug/*.h"]),
+    includes = ["include"],
+    deps = [
+        ":IR",
+        ":Support",
+        "//llvm:Support",
+    ],
+)
+
+cc_library(
+    name = "BreakpointManagers",
+    srcs = glob([
+        "lib/Debug/BreakpointManagers/*.cpp",
+        "lib/Debug/BreakpointManagers/*.h",
+    ]),
+    hdrs = glob(["include/mlir/Debug/BreakpointManagers/*.h"]),
+    deps = [
+        ":Debug",
+        ":IR",
+        "//llvm:Support",
+    ],
+)
+
 cc_library(
     name = "MlirLspServerSupportLib",
     srcs = glob(
@@ -3998,6 +4032,9 @@ cc_library(
             "lib/Dialect/GPU/Transforms/*.cpp",
             "lib/Dialect/GPU/Transforms/*.h",
         ],
+        exclude = [
+            "lib/Dialect/GPU/Transforms/SerializeToCubin.cpp",
+        ],
     ),
     hdrs = glob(["include/mlir/Dialect/GPU/Transforms/*.h"]),
     includes = ["include"],
@@ -4017,6 +4054,7 @@ cc_library(
         ":ROCDLToLLVMIRTranslation",
         ":SCFDialect",
         ":FuncDialect",
+        ":SerializeToCubin_stub",
         ":SideEffectInterfaces",
         ":Support",
         ":Transforms",
@@ -4027,15 +4065,61 @@ cc_library(
         "//llvm:Support",
         "//llvm:Target",
     ] + if_cuda_available([
-        # Dependencies for SerializeToCubin.cpp with
-        # -DMLIR_GPU_TO_CUBIN_PASS_ENABLE
         ":NVVMToLLVMIRTranslation",
         "//llvm:NVPTXCodeGen",
+    ]),
+)
+
+cc_library(
+    name = "SerializeToCubin",
+    srcs = [
+        "lib/Dialect/GPU/Transforms/SerializeToCubin.cpp",
+    ],
+    local_defines = if_cuda_available(["MLIR_GPU_TO_CUBIN_PASS_ENABLE"]),
+    deps = [
+        ":NVVMToLLVMIRTranslation",
+        ":ToLLVMIRTranslation",
+        ":GPUDialect",
+        ":GPUPassIncGen",
+        ":GPUTransforms",
+        ":Pass",
+        ":Support",
+        "//llvm:Support",
+    ] + if_cuda_available([
         "@cuda//:cuda_headers",
         "@cuda//:libcuda",
     ]),
 )
 
+write_file(
+    name = "SerializeToCubin_stub_cc",
+    out = "SerializeToCubin_stub.cc",
+    content = [
+        """
+#include "mlir/Dialect/GPU/Transforms/Passes.h"
+
+// Provide a weak registration stub in case the real SerializeToCubin is not
+// linked in.
+
+__attribute__((weak)) void mlir::registerGpuSerializeToCubinPass() {}
+""",
+    ],
+)
+
+cc_library(
+    name = "SerializeToCubin_stub",
+    srcs = [":SerializeToCubin_stub_cc"],
+    hdrs = glob(["include/mlir/Dialect/GPU/Transforms/*.h"]),
+    includes = ["include"],
+    deps = [
+        ":GPUDialect",
+        ":GPUPassIncGen",
+        ":Pass",
+        ":Support",
+        "//llvm:Support",
+    ],
+)
+
 td_library(
     name = "GPUTransformOpsTdFiles",
     srcs = [
@@ -4078,9 +4162,12 @@ cc_library(
     ],
     includes = ["include"],
     deps = [
+        ":AffineDialect",
         ":ArithDialect",
         ":AsmParser",
         ":ControlFlowDialect",
+        ":DialectUtils",
+        ":FuncDialect",
         ":GPUDialect",
         ":GPUTransformOpsIncGen",
         ":GPUTransforms",
@@ -4089,6 +4176,7 @@ cc_library(
         ":Parser",
         ":SCFDialect",
         ":SideEffectInterfaces",
+        ":Support",
         ":TransformDialect",
         ":TransformUtils",
         "//llvm:Support",
@@ -4100,10 +4188,10 @@ td_library(
     srcs = [
         "include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td",
         "include/mlir/Dialect/LLVMIR/LLVMEnums.td",
+        "include/mlir/Dialect/LLVMIR/LLVMInterfaces.td",
         "include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td",
         "include/mlir/Dialect/LLVMIR/LLVMOpBase.td",
         "include/mlir/Dialect/LLVMIR/LLVMOps.td",
-        "include/mlir/Dialect/LLVMIR/LLVMInterfaces.td",
         "include/mlir/Dialect/LLVMIR/LLVMTypes.td",
     ],
     includes = ["include"],
@@ -4248,11 +4336,13 @@ cc_library(
     ]),
     includes = ["include"],
     deps = [
+        ":ArithDialect",
         ":ConversionPassIncGen",
         ":IR",
         ":Pass",
         ":SPIRVConversion",
         ":SPIRVDialect",
+        ":Support",
         ":Transforms",
         ":VectorDialect",
         "//llvm:Support",
@@ -5146,6 +5236,7 @@ cc_library(
     hdrs = ["lib/Conversion/SPIRVCommon/Pattern.h"],
     includes = ["include"],
     deps = [
+        ":IR",
         ":SPIRVDialect",
         ":Transforms",
         "//llvm:Support",
@@ -6836,6 +6927,7 @@ cc_library(
     deps = [
         ":BytecodeReader",
         ":BytecodeWriter",
+        ":Debug",
         ":IR",
         ":Parser",
         ":Pass",
@@ -7025,6 +7117,7 @@ cc_binary(
         ":Pass",
         ":QuantOps",
         ":SCFToGPU",
+        ":SerializeToCubin",
         ":Support",
         ":Transforms",
         "//llvm:AllTargetsCodeGens",
@@ -7061,6 +7154,7 @@ cc_binary(
         "//mlir/test:TestTransforms",
         "//mlir/test:TestTypeDialect",
         "//mlir/test:TestVector",
+        "//mlir/test:TestVectorToSPIRV",
     ],
 )
 
@@ -8764,6 +8858,7 @@ cc_library(
         ":Support",
         ":Transforms",
         ":VectorDialect",
+        ":VectorTransforms",
         ":VectorUtils",
         "//llvm:Core",
         "//llvm:Support",
@@ -9915,6 +10010,7 @@ cc_library(
         ":ControlFlowDialect",
         ":DialectUtils",
         ":FuncDialect",
+        ":GPUDialect",
         ":IR",
         ":InferTypeOpInterface",
         ":LoopLikeInterface",
diff --git a/utils/bazel/llvm-project-overlay/mlir/test/BUILD.bazel b/utils/bazel/llvm-project-overlay/mlir/test/BUILD.bazel
index 2364995d00fc9..5fbf4d994a624 100644
--- a/utils/bazel/llvm-project-overlay/mlir/test/BUILD.bazel
+++ b/utils/bazel/llvm-project-overlay/mlir/test/BUILD.bazel
@@ -507,6 +507,20 @@ cc_library(
     ],
 )
 
+cc_library(
+    name = "TestVectorToSPIRV",
+    srcs = glob(["lib/Conversion/VectorToSPIRV/*.cpp"]),
+    deps = [
+        "//mlir:ArithDialect",
+        "//mlir:FuncDialect",
+        "//mlir:Pass",
+        "//mlir:SPIRVDialect",
+        "//mlir:Transforms",
+        "//mlir:VectorDialect",
+        "//mlir:VectorToSPIRV",
+    ],
+)
+
 cc_library(
     name = "TestAffine",
     srcs = glob([
@@ -826,6 +840,7 @@ cc_library(
     deps = [
         "//mlir:AffineDialect",
         "//mlir:Analysis",
+        "//mlir:ArithDialect",
         "//mlir:FuncDialect",
         "//mlir:GPUDialect",
         "//mlir:LLVMDialect",
@@ -835,6 +850,7 @@ cc_library(
         "//mlir:Pass",
         "//mlir:SCFDialect",
         "//mlir:Support",
+        "//mlir:TensorDialect",
         "//mlir:TransformUtils",
         "//mlir:VectorDialect",
         "//mlir:VectorToSCF",
diff --git a/utils/bazel/llvm-project-overlay/mlir/unittests/BUILD.bazel b/utils/bazel/llvm-project-overlay/mlir/unittests/BUILD.bazel
index 551af89a21658..9091f856c8874 100644
--- a/utils/bazel/llvm-project-overlay/mlir/unittests/BUILD.bazel
+++ b/utils/bazel/llvm-project-overlay/mlir/unittests/BUILD.bazel
@@ -11,6 +11,25 @@ package(
 
 licenses(["notice"])
 
+cc_test(
+    name = "debug_tests",
+    size = "small",
+    srcs = glob([
+        "Debug/*.cpp",
+        "Debug/*.h",
+    ]),
+    deps = [
+        "//llvm:Support",
+        "//llvm:TestingSupport",
+        "//mlir:BreakpointManagers",
+        "//mlir:Debug",
+        "//mlir:Support",
+        "//third-party/unittest:gmock",
+        "//third-party/unittest:gtest",
+        "//third-party/unittest:gtest_main",
+    ],
+)
+
 cc_test(
     name = "ir_tests",
     size = "small",
@@ -21,6 +40,7 @@ cc_test(
     deps = [
         "//llvm:Support",
         "//mlir:IR",
+        "//mlir:Support",
         "//mlir/test:TestDialect",
         "//third-party/unittest:gmock",
         "//third-party/unittest:gtest",